AI Linguistics: Language Models Master Metalinguistics

In a recent study, researchers show that OpenAI’s o1 reasoning model is able to recognize, map out, and even build upon one of the most complex phenomena of human language, a concept called linguistic recursion. Recursion involves nesting one element within another element in a sentence; it’s analogous to a lake on an island in a lake. The results were published on 3 June in IEEE Transactions on Artificial Intelligence.

Study coauthor Gašper Beguš is an associate professor of linguistics at the University of California, Berkeley, with a deep interest in language and intelligence. His research compares machine and human forms of learning to understand their differences and strengths, and also to understand the limits of AI from a safety and regulatory standpoint.

Can LLMs Do Metalinguistics?

In the new study, Beguš and his collaborators examined the metalinguistic abilities of four large language models (LLMs): OpenAI’s GPT-3.5 Turbo, GPT-4, and o1, as well as Meta’s Llama 3.1. While many studies have explored how well such models can produce language, this study looked specifically at the models’ ability to analyze language—their ability to perform metalinguistics.

For example, when a sentence has multiple meanings, are language models able to map out and “understand” correctly all the various meanings? Beguš provides a simple one-word example of this challenge. “Unlockable has two meanings, right? Either you cannot unlock it, or you can unlock it,” he explains.

In their study, the researchers tested the AI models with difficult complete sentences that could have multiple meanings, called ambiguous structures. For example: “Eliza wanted her cast out.”

The sentence could be expressing Eliza’s desire to have a person be cast out of a group, or to have her medical cast removed. Whereas all four language models correctly identified the sentence as having ambiguous structure, only o1 was able to correctly map out the different meanings the sentence could potentially contain.

LLMs’ Recursive Abilities

Beguš emphasizes that the most important advance reported in this study was o1’s ability to successfully engage in linguistic recursion. An example of a recursive element within a sentence is shown in brackets in the following sentence: “The worldview [that the prose Nietzsche wrote expressed] was unprecedented.” In fact, like Russian nesting dolls, the sentence contains a recursion within a recursion: “The worldview [that the prose [Nietzsche wrote] expressed] was unprecedented.”

In the linguistic recursion experiment, the researchers asked the language models to determine whether a given sentence is recursive,…