Why Average Scores Fail Code-Switched Benchmarks
An average score can hide how sharply a model's performance drops on complex code-switched text. Evaluating by complexity helps us see where it fails.
When we evaluate NLP models on code-switched text, we often report one score: 85% accuracy, for example, or an F1 score of 0.72. Without accounting for document complexity, that number tells us little about which kinds of text the model can handle.
What the score leaves out
Consider two sentences:
"I'm going to the store later."
"Oya, make we go market before traffic go choke, you know say Lagos no dey smile for weekend."
Both are comprehensible to a bilingual speaker. The second mixes English, Yoruba, and Nigerian Pidgin, and its cultural references require contextual knowledge. Getting that sentence right asks more of a model. Should the two successes count the same in an evaluation?
Current benchmarks count them equally. I think that makes the score harder to interpret.
Different kinds of complexity
Code-switched documents vary in complexity. A speaker might insert one word from another language, or combine grammatical structures from several languages within a sentence. Understanding the text may also require cultural knowledge.
Warning
A model could score 80% on simple code-switched sentences and 40% on complex ones. An average of 60% would hide how much its performance drops as the text becomes more complex.
An aggregate metric hides these differences. It becomes harder to identify where the model fails and decide what needs improving.
Evaluating by complexity
Benchmarks could group code-switched text by complexity and report performance for each group:
- Lexical switching involves inserting a single word into an otherwise monolingual sentence.
- Phrasal switching embeds a phrase from another language in a matrix language.
- Intra-sentential complexity involves grammatical integration between languages within a sentence.
- Cultural pragmatics concerns sentences that require cultural or contextual knowledge.
For example, a report of "75% accuracy" could become "95% on lexical switches, 60% on phrasal switches, 35% on complex intra-sentential switches." Those results tell us where to look for failures.
My research
My research explores evaluation frameworks that account for complexity in Nigerian Pidgin and other code-switched contexts. I want the results to show which kinds of language a model understands and where that understanding breaks down.