Mysterious ‘gpt2-chatbot’ emerges as users claim it to be a top AI model rival

A mysterious new AI chatbot named “gpt2-chatbot” has emerged, following its release on a prominent large language model benchmarking site, LMSYS Org.

Speculation suggests that gpt2-chatbot boasts capabilities comparable to OpenAI’s GPT-4, placing it among a select few AI models that have been able to achieve this.

Ethan Mollick, a Professor researching artificial intelligence at the Wharton School of the University of Pennsylvania, wrote in a social media post: “No one knows who made it or what it is, but I have been playing with it a little and it appears to be in the same rough ability level as GPT-4. A mysterious GPT-4 class model?”

There is a mysterious new model called gpt2-chatbot accessible from a major LLM benchmarking site. No one knows who made it or what it is, but I have been playing with it a little and it appears to be in the same rough ability level as GPT-4. A mysterious GPT-4 class model? Neat! pic.twitter.com/1s2iEreaiT

— Ethan Mollick (@emollick) April 29, 2024

Access to the new model is currently restricted to the Chatbot Arena website, albeit in a restricted capacity. Within the site’s “side-by-side” arena mode, where users deliberately choose the model, gpt2-chatbot is subject to a rate limit of eight queries per day, limiting users’ ability to thoroughly test it.

A post on X from the organization later confirmed that the chatbot had been temporarily removed “due to unexpectedly high traffic.” Nevertheless, LMSYS advises staying tuned for its wider releases.

Thanks for the incredible enthusiasm from our community! We really didn’t see this coming.

Just a couple of things to clear up:

– In line with our policy, we’ve worked with several model developers in the past to offer community access to unreleased models/checkpoints (e.g.,…

— lmsys.org (@lmsysorg) April 30, 2024

“Just to clarify, following our policy, we’ve partnered with several model developers to bring their new models to our platform for community preview testing,” said LMSYS Org on X, responding to a thread about gpt2-chatbot. “These models are strictly for testing and won’t be listed on the leaderboard until they go public.”

hi @simonw, thanks a ton! We really value your feedback.

Just to clarify, following our policy, we’ve partnered with several model developers to bring their new models to our platform for community preview testing. These models are strictly for testing and won’t be listed on the…

— lmsys.org (@lmsysorg) April 29, 2024

How has gpt2-chatbot been received?

The LLM was even tested by OpenAI CEO Sam Altman, who said he had a “soft spot” for it. However, there has been no confirmation regarding whether this is a model for ChatGPT-4.5 or ChatGPT-5.

i do have a soft spot for gpt2

— Sam Altman (@sama) April 30, 2024

Another user said that it “definitely feels like GPT4.5/GPT5 for me. Gave it some hard prompts where I barely get a right answer in Claude/ GPT4 and it aced it.”

Definitely feels like GPT4.5/GPT5 for me. Gave it some hard prompts where I barely get a right answer in Claude/ GPT4 and it aced it.

— Torsten Jacobi (@jacobi_torsten) April 29, 2024

Initial mentions of the model surfaced on 4chan before gaining traction on social media platforms like X. Subsequently, Reddit threads emerged, claiming the new model’s enhanced capabilities, surpassing all other language learning models in the field.

Featured image: Canva