Instruction following in Armenian: 884 multi-turn conversations from IFEval and Multi-IF, translated with human-expert review or culturally adapted by human experts. Models are ranked by Overall — higher is better. 📚 Dataset
IFEval & Multi-IF hy is an Armenian extension of Multi-IF, the benchmark for multi-turn, multilingual instruction following (itself an extension of IFEval). Culturally neutral conversations were translated with an agentic translation and review pipeline combining LLM-based processing with human-expert review; culturally sensitive conversations were adapted manually by human experts. 884 conversations: 235 culturally adapted, 649 translated directly.
| Metric | Meaning |
|---|---|
| P-str | Prompt-level, strict: fraction of conversations where every instruction was followed, checked exactly as written. |
| I-str | Instruction-level, strict: fraction of individual instructions followed, checked exactly as written. |
| P-loo | Prompt-level, loose: same as P-str, but forgiving of minor formatting. |
| I-loo | Instruction-level, loose: same as I-str, but forgiving of minor formatting. |
| Overall | Average of the four scores above — the headline number. |
| T1 / T2 / T3 | Overall, computed separately on turn 1, 2 and 3 — shows how accuracy degrades as the conversation goes on. |
results.csv → Edit.Culturally Adapted, Culturally Neutral and Merged — with the eight scores (0–1).The maintainers review the numbers and merge; the leaderboard updates automatically.
model,split,p_str,i_str,p_loo,i_loo,overall,t1,t2,t3 My Model (think on),Culturally Adapted,0.700,0.860,0.730,0.880,0.793,0.870,0.780,0.730 My Model (think on),Culturally Neutral,0.690,0.850,0.720,0.870,0.783,0.860,0.770,0.720 My Model (think on),Merged,0.693,0.853,0.723,0.873,0.786,0.863,0.773,0.723