Can an LLM code in a language it was not trained for?

Spoiler: kind of. It is not very fast and not very efficient, but if you give the LLM a way to test its work, it does OK.
See the generated code in an obscure Soviet language at https://github.com/ikriv/rapira-samples.

Background

My friends voiced an opinion that LLMs may starve innovation, because if no one is coding by hand anymore, how can LLMs learn to code using new paradigms? Then I watched Michael Xing’s video AI vs freshman CS where the author makes the following claim:

It seems that as soon as you hand the LLMs a new programming language it’s never seen before, no matter how in depth is the documentation you provided, its lack of training data causes it to fall apart.

I decided to give it a try, and my experience was slightly better than Michael’s, but admittedly it’s not exactly the same experiment.

The Stage

Rapira is a programming language designed for educational purposes in the Soviet Union, in my native Siberia around the year 1980. Consider it a Soviet Pascal. This was my first programming language and I have very fond memories of it.

I downloaded the original Rapira specification (in Russian), and also installed the Rapture, a Rapira interpreter written in Kotlin.

I also downloaded a few Rapira coding examples.

The Task

First I asked Cursor IDE, equipped with the Claude Opus 4.6 High model, to create a program that prints Pascal’s Triangle. The samples from the Internet already included the code for the Fibonacci sequence and finding the prime numbers, so Pascal’s Triangle seemed like a natural choice. Claude did it in a couple of minutes, but it had to do a few iterations before the program actually worked.

Then I was up for a more challenging task. ChatGPT suggested a chess evaluation engine, but it felt too involved, so we settled on a markdown-to-HTML converter.

Mr. Claude was not super enthusiastic, claiming that it is a rather ambitious task for a fossil Jurassic language like Rapira, but took up the challenge anyway and produced a working result in 15-20 minutes. This included about 770 lines of Rapira code for the converter itself, and another 780-ish lines for the tests.

In the process, it found that the dialect that the Rapture interpreter implements differs quite a bit from the original spec I provided, but it was able to figure it out nonetheless.

How much did Claude know up front?

I asked Claude Opus 4.6 to tell me what it knew about Rapira without using tools. To my astonishment, it knew quite a bit, but mostly in general terms. E.g. it wrote an approximate code snippet for iterating over a loop, but admitted that it is quite fuzzy on the exact syntax.

ДЛЯ и ОТ 1 ДО ДЛИН(перем) ДЕЛАТЬ
  ВЫВОД: перем[и]
КОНЕЦ ДЛЯ

Indeed, the actual syntax is similar, but not the same:

ДЛЯ к ОТ 1 ДО #перем ЦИКЛ
  ВЫВОД: перем[к]
КЦ

Namely, the array length operator is different, some of the keywords are different, and you cannot use “и” as a variable name, because it is also a keyword, meaning “and” (akin to Spanish ‘y’).

So, unlike the CritterLang, Rapira was not a completely unfamiliar language, but Claude’s information about it was rather vague. Besides, the spec that I gave it did not match what the interpreter actually did, which made the task much more interesting.

The Results

Obviously, it took Claude much longer to generate correct Rapira code than it would have for Java or Python. Another problem is that most of this knowledge is lost when I start another chat. E.g. I started two chats, and both wasted time by naming a loop index variable “и”, and then discovering that this is not allowed. This can be partially mitigated by skills and what-not, but one way or another, you’d have to put exact knowledge of Rapira into the context window every time you want to use it. It is not baked into the model.

Still, Claude was pretty successful at creating a working Rapira program mdconv.rap in minutes, despite the fact that my spec and my interpreter did not agree with each other on many fundamental things, e.g. what the assignment operator looks like, or how the name scope is defined.

One interesting observation: initially Claude emulated an actual programming style of the 1970s and early 1980s with short but cryptic variable names like init_zz and constructs like if d or s=='c' then.... But then I asked it to come up with better identifiers, and it obliged.

Conclusion

To answer the original question, LLMs seem to be versatile enough to handle medium amounts of new knowledge on-the-fly. Granted, Claude already knew something about Rapira, and Rapira is very similar in structure to existing languages like Pascal and Algol, but still, it had to figure out the details, some of which were rather peculiar, especially the name scoping rules. It seems to me that if the rise of new knowledge is incremental (which it usually is), existing LLMs will be able to absorb it on the fly well enough until it is baked into the permanent memory via training.