Testing details and interesting observations

The prompt used was the following:

List every airliner family (20 or more passenger capacity and operated by at least one airline) that has been in production at any point during and after the year 2000. Group broadly by name; for example, "Boeing 737" should be considered one family.

All models were tested with web search turned off for fairness. For GPT-5 and o3, the latest ChatGPT version was used. For Gemini, Google AI Studio was used.

Graph looks squashed because Kimi K2 was added last-minute.

Notably, thinking does not significantly increase performance in the two open-weight models (in fact it seems like somehow it decreases performance), unlike GPT-5 which benefits greatly from thinking. Though, even in the two open-weight models, the number of incorrect answers do seem to decrease when thinking is enabled.

I was under the impression that Qwen was the one without much world knowledge and Deepseek knows more, but it seems Deepseek suffers from the same issue as well.

Surprisingly, GPT-5 Instant (along with Qwen 3 without reasoning) had the highest rate of incorrect answers. From experience, I've also felt that GPT-5 gets a lot of things just plain wrong when it doesn't use reasoning. GPT-5 being the model the vast majority of casual AI chatbot users around the world rely on for information, this seems somewhat problematic.

I think the most notable thing isn't that the open-weight models are bad at this, but rather that the open-weight models just don't see that much improvement when thinking is enabled. GPT-5 also does pretty badly when thinking is disabled, but the performance shoots up when you allow it to think.

GPT-5 Thinking was the only model to mention the An-38 and C-212.

Qwen, both when thinking was enabled and disabled, always put the MA700 (an in-development airliner family not yet in service) in its list but always left out the MA60/600 (currently in production and in service). The implications of this are vague and confusing.

The MC-21, an airliner family in production but not yet in service, was kind of a gray zone. The majority of responses actually did mention it, but I did not include it in the list of 39 because the prompt specifically asks for airliners operated by at least one airline. This means it was counted as an incorrect answer.

The following are the 39 airliner families that were tested for. I am fairly sure I did not miss any. What constitutes a "family" is arbitrary; this is just the way I classified them, and most of the time it would match how the LLM classified them. The results were not calculated naïvely from the number of families it listed; instead, they are the number of airliner families in this list that the response checks off. For example, if it listed E-jets and E-jets E2 as one family but still specified that it includes the E2 family, it would be counted as two.

A220 A300 A320 A330 A340 A350 A380 717 737 747 757 767 777 787 MD-90 MD-11 CRJ E-jet E-jetE2 ERJ EMB-120 Dornier328 AvroRJ ATR 42/72 Dash8 C-212 CN-235 An-148/158 An-140 An-38 C909 C919 MA60/600 SSJ100 Il-96 Il-114 Tu-154 Tu-204/214 Yak-42