SK Telecom's A.X K2 Lifts Korean Benchmark Accuracy From 51% to 73%

SK Telecom’s newsroom has published a guest essay by Shin Hyopil, a professor of linguistics at Seoul National University, timed to Hangul Day, the Korean holiday that commemorates the creation of the Korean alphabet. The essay includes Korean-language scores for SK Telecom’s AI model A.X K2. On KoBALT, a benchmark that measures Korean comprehension, A.X K2 scored 73.0% accuracy. The previous model, A.X K1, scored 51.0%. That is a gain of 22.0 percentage points.

How Korean ability was measured

KoBALT is a Korean-language performance benchmark built by the Natural Language Processing Lab at Seoul National University. It uses 700 difficult items selected by experts, covering five areas: sentence structure, meaning, conversational context, phonological rules, and word formation. Shin explained that the project grew out of a concern that there was no common yardstick for precisely measuring the intricate rules of Korean.

When KoBALT was released in 2025, Claude 3.7 Sonnet posted overall accuracy of 61%. It scored only 36% in the category covering word formation and inflection, and only 31% in the category covering phonological rules. Shin read the results as evidence of a gap between a model’s ability to grasp the surface meaning of Korean expressions and its command of fine-grained rules.

Why honorifics are hard for AI

Honorifics are not simply a matter of getting the grammar right. Shin explained that using them requires weighing not only the subject of the sentence but also the speaker’s relationship with the listener. Even if a sentence uses honorific forms for a professor’s actions, he noted, when the speaker is relaying that matter politely to someone senior, the word used for oneself also has to shift from the plain na (나) to the humble jeo (저) for the sentence to sound natural.

The ability to recover what has been left out of a conversation matters just as much. Korean drops subjects and objects whenever context makes them clear. On KoBALT’s tasks involving such omissions, most overseas models scored below 55%. Shin pointed out that errors in identifying an omitted referent can distort answers in business, legal, and medical settings.

A model also has to recognize that a word carries the same meaning even when its form changes, as when the Korean verb dopda (돕다, “to help”) becomes dowa (도와). Shin argued that English-oriented methods of breaking text into small units fail to adequately capture Korean particles, verb endings, and irregular inflection. The implication is that training data and text-processing tools tailored to the characteristics of Korean are needed.

What changed in A.X K2

A.X K2 is designed to select and draw on the most relevant information within long passages. It applies SGA, an architecture developed in-house by SK Telecom. On KMMLU-Pro, which tests Korean-language knowledge, it scored 80.5, and on CLIcK, which tests understanding of Korean culture, it scored 91.6.

Shin stressed that an AI built independently for the Korean language and culture, often called sovereign AI, must be able to adjust its tone according to the conversational partner and the situation. He also pointed out problems with existing Korean evaluations: some mix in translated items, and some overlap with training data. In other words, scores can be raised through memorization rather than genuine language ability. For that reason, he argued, the characteristics of Korean need to be built in from the development stage.

For Korean users, the challenges are clear in practice: using appropriate honorifics in workplace conversation, and accurately identifying omitted referents and unstated intentions. Shin emphasized that models should be advanced by validating them both on linguistics benchmarks and in real service environments.