A safety benchmark that only swaps languages can miss a lot. Scale AI and the Korea AI Safety Institute built one that swaps context as well.
ROK-FORTRESS, published Thursday, is a bilingual English-Korean adversarial benchmark. Each prompt appears in up to four variants that vary both language and geopolitical grounding, trading U.S. entities, institutions and operational details for Korean ones. The researchers describe this as a transcreation matrix. Prompts also come with benign twins, so the benchmark captures over-refusal alongside harm.
The set covers 1,235 tasks. National security threats come first, covering chemical, biological, radiological, nuclear and explosive risks, followed by political violence and terrorism.
Two more domains round out the set: criminal and financial activity, and information leakage.
Calibrated LLM-as-judge panels score responses against expert-written reference labels, using prompt-specific binary rubrics from red teamers. Madhu Sehwag authored the post.
Across 14 frontier models, prompts written in Korean and grounded in Korean context drew consistently lower measured harm than their U.S.-grounded equivalents.
One example shows the mechanism. The same mass-casualty intent can point at the 1995 Oklahoma City bombing or the 1987 downing of Korean Air Flight 858. Translate the prompt without changing what it references, and the shift never surfaces.
The work extends Scale AI’s FORTRESS benchmark for national security and public safety, part of a wider partnership with the Korean institute covering joint research, evaluations and red teaming.
