muteval — eval coverage report
27% effective coverage
27%raw score
11–52%95% CI
4/15killed / evaluated
0inert (excluded)
3high-severity survivors
Survivors (11)
HIGH
weaken_modals
#0
weakened "Do not" -> "try not to" (near: user did a great work. Do not use markdown for the fe)
fix: add an eval for the rule near "user did a great work. Do not use markdown for the …" (e.g. a checks.llm_judge for that behavior)
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You've made a good effort, but there are areas to improve. Focus on the differences between 'ser' and 'estar' as they are crucial in Spanish. Additionally, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
}
HIGH
flip_negation
#1
inverted "Do not" -> "do" (near: user did a great work. Do not use markdown for the fe)
fix: add an eval for the rule near "user did a great work. Do not use markdown for the …" (e.g. a checks.llm_judge for that behavior)
--- baseline
+++ mutant
@@ -1,3 +1,5 @@
+```json
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "Hello! You've made a good effort, but there are a couple of areas to focus on for improvement. First, pay attention to the difference between 'ser' and 'estar' as they are used in different contexts. Secondly, make sure that nouns and adjectives agree in gender. Keep practicing, and you'll see progress!"
}
+```
HIGH
delete_sentences
#2
deleted sentence: "Do not use markdown for the feedback message."
fix: add checks.llm_judge("the reply still follows: Do not use markdown for the feedback message.")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You've made a good effort, but there are areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Additionally, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
}
MEDIUM
drop_instruction_lines
#3
dropped line: "You are a friendly language tutor."
fix: add checks.llm_judge("the reply still follows: You are a friendly language tutor.")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You need to focus on the differences between 'ser' and 'estar' as well as ensuring gender agreement in your sentences. Practice these areas to improve your understanding and usage of the language."
}
MEDIUM
drop_instruction_lines
#4
dropped line: "student in English so they can improve in learning the language."
fix: add checks.llm_judge("the reply still follows: student in English so they can improve in learning …")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You need to work on the difference between 'ser' and 'estar' as well as ensuring gender agreement in your sentences. Keep practicing, and you'll improve!"
}
MEDIUM
drop_instruction_lines
#5
dropped line: "feedback message. The feedback should be concise and direct."
fix: add checks.llm_judge("the reply still follows: feedback message. The feedback should be concise an…")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You've made a good effort, but there are a couple of areas to focus on for improvement. First, try to practice the difference between 'ser' and 'estar', as they are used in different contexts. Additionally, pay attention to gender agreement in your sentences; make sure that adjectives and articles match the nouns they describe. Keep practicing, and you'll see progress!"
}
MEDIUM
drop_instruction_lines
#6
dropped line: "Grading:"
fix: add checks.llm_judge("the reply still follows: Grading:")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You've made some progress, but there are areas to improve. Focus on understanding the difference between 'ser' and 'estar', as well as ensuring gender agreement in your sentences. Practice these concepts to enhance your skills."
}
MEDIUM
delete_sentences
#7
deleted sentence: "You are a friendly language tutor."
fix: add checks.llm_judge("the reply still follows: You are a friendly language tutor.")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You need to focus on the differences between 'ser' and 'estar' as they are crucial in Spanish. Additionally, pay attention to gender agreement in nouns and adjectives. Practice these areas to improve your understanding and usage of the language."
}
MEDIUM
delete_sentences
#8
deleted sentence: "Based on the following grading in {{language}}, provide constructive …"
fix: add checks.llm_judge("the reply still follows: Based on the following grading in {{language}}, pro…")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You made some mistakes with ser vs estar and gender agreement. Keep practicing!"
}
MEDIUM
delete_sentences
#9
deleted sentence: "If the mark is 10, just say the user did a great work."
fix: add checks.llm_judge("the reply still follows: If the mark is 10, just say the user did a great wo…")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "Great job on your progress! To improve further, focus on the differences between 'ser' and 'estar' as they are used in different contexts. Additionally, pay attention to gender agreement in nouns and adjectives, ensuring they match correctly. Keep practicing, and you'll see improvement!"
}
MEDIUM
delete_sentences
#10
deleted sentence: "The feedback should be concise and direct."
fix: add checks.llm_judge("the reply still follows: The feedback should be concise and direct.")
--- baseline
+++ mutant
@@ -1,3 +1,3 @@
{
- "feedback_message": "You've made a good effort, but there are some areas to improve. Focus on the difference between 'ser' and 'estar' as they are used in different contexts. Also, pay attention to gender agreement in nouns and adjectives. Practice these concepts to enhance your understanding and fluency."
+ "feedback_message": "You've made a good effort, but there are some areas to improve. Pay attention to the difference between 'ser' and 'estar', as they are used in different contexts. Additionally, make sure to check the gender agreement between nouns and adjectives. Practicing these concepts will help you communicate more clearly in Spanish. Keep up the hard work!"
}
Generated by muteval. Each survivor is an output change your evals did not catch — write an eval that fails on it, then re-run.