Performance gains of the Python 3.14 tail-call interpreter were largely due to benchmark errors

Posted by Ok_Fox_8448@reddit | Python | View on Reddit | 32 comments

I was really surprised and confused by last month's claims of a 15% speedup for the new interpreter. It turned out it was an error in the benchmark setup, caused by a bug in LLVM 19.

See https://blog.nelhage.com/post/cpython-tail-call/ and the correction in https://docs.python.org/3.14/whatsnew/3.14.html#whatsnew314-tail-call

A 5% speedup is still nice though!

[-]

alcalde@reddit

A 5% speedup for something not particularly fast really isn't a very nice speedup. Early on we were promised much better....

Earlier in 2021, Python author Guido van Rossum was rehired by Microsoft to continue work on CPython, and they have proposed a faster-cpython project to improve the performance of CPython by a factor of 5 over 4 years.

We're getting to the end of that and we're not even at 2x speed over 2021, never mind 5x.
https://archive.is/K2x3j

[-]

kenjin4096@reddit

I'm the PR author for the tail-calling interpreter. I published a personal blog post to apologise to the Python community: https://fidget-spinner.github.io/posts/apology-tail-call.html

Nelson was great to work with and spent a lot of time digging into this, they deserve all the kudos for finding the bug!

We're really sorry that we didn't notice the compiler bug and reported inaccurate numbers for those 3 weeks. A compiler bug was the last thing I expected.

[-]

theturtlemafiamusic@reddit

I think if you're (un)lucky enough to experience a true compiler bug, you have nothing to apologize for. That's like apologizing for being late to work because you saw a unicorn on the way =P

Thank you for helping to improve python, and Nelson's blog post is really illuminating about how modern compilers handle a switch and/or jump table, taught me a lot. So that's one loss and two wins.

[-]

Bunslow@reddit

I take full personal responsibility for the oversight that led to it.

Honestly I'm not even sure that it is your fault. Well idk maybe 20% but certainly nowhere near 100% as best I can figure. Honestly I'd suggest toning it down a little, being overly apologetic can have its own downsides

[-]

ashvy@reddit

All the more reason to believe software is a team and community effort, propping each other up

[-]

Ok_Fox_8448@reddit (OP)

I'm really surprised you felt any need to apologise to the Python community, 5% speedup is amazing!

[-]

DontBeLikeBoeing@reddit

That kind of stuff happens, and you did everything right with the information you had, both before and after the discovery. All good ;-)

[-]

totheendandbackagain@reddit

5% is huge! Thanks for the correction. We get better together!

[-]

_MicroWave_@reddit

Your post is very humble and well written.

I don't think you have much to apologise for. These kind of things happen. We move on older and wiser.

Thank you for your hard work maintaining cPython.

[-]

MeroLegend4@reddit

👍

[-]

john0201@reddit

Thank you for your work.

[-]

ArabicLawrence@reddit

I strongly believe you do not need to apologise nor be sorry. Maybe the community can think of a way to prevent this in the future, but thanks for the massive work and congratulations. 5% speed increase is still a lot!

[-]

Bunslow@reddit

id like to say that saying "1.09x slower" and "1.01x faster" in the same table is a diabolically bad way to present relative performance data

(why on earth not simply say "0.91x" and "1.01x"???)

[-]

russellvt@reddit

(why on earth not simply say "0.91x" and "1.01x"???)

Because technically, it's more like 92% slower (91.7), not just 91. The "1.09" looks... cleaner or more consistent, maybe? /s

[-]

serjester4@reddit

It becomes much harder to compare. 2x faster and 0.5x slower are the same thing but sound different. It gets even worse if it’s 100x faster vs 0.01X slower.

[-]

Bunslow@reddit

those are also silly ways to say it.

good way to say it: "2.0x speed vs 0.5x speed vs 100x speed vs 0.01x speed". even just using "faster" or "slower" is automatically bad. state the number, and never state the adjective.

[-]

ambidextr_us@reddit

It also makes mental math easier, like the video speed controller extension for youtube, you set the speed to 0.9x or 1.3x etc, and it comes out to just the raw percentage change.

[-]

JanEric1@reddit

I think that is the default that hyperfine (and probably other benchmarking programs) spit out?

[-]

HommeMusical@reddit

I just want to say what a civilized and well-written article this is.

There's a solid summary, and then there's another level of detail, and then a third, you can stop reading at many points and still get one level of the picture.

The problem is put in perspective and the article explains how this would slip past even very conscientious reviewers.

[-]

Ok_Fox_8448@reddit (OP)

Just to clarify, I'm not the author of the article

[-]

ArabicLawrence@reddit

I read the links, but I still cannot understand why the Python speed benchmark did not notice the compiler regression immediately. As far as I know, CPython tests this kind of impacts.

[-]

daredevil82@reddit

this is specific to the CLANG compiler, and cypython is built with GCC, IIRC. Why would a benchmark notice a code regression?

and

those benchmarks were accurate, in that they were accurately measuring (as far as I know) the performance difference between those builds.

[-]

ArabicLawrence@reddit

Forgive me for being dense, but if the issue is on CLANG and CPython is built with GCC, I still do not understand the cause of the "wrong" benchmark. The performance gain was reported to be on CLANG See A new tail-calling interpreter for significantly better interpreter performance · Issue #128563 · python/cpython stating (emphasis mine):

TLDR (all results are pyperformance, clang-19, with PGO + ThinLTO unless stated otherwise):

[-]

kenjin4096@reddit

To get meaningful results on whether Python sped up, we try to hold all other things constant. This includes the compiler. If we benchmarked GCC vs Clang, we would have no clue whether the speedup was due to a change in the compiler, or something we did.

Unfortunately, this is one of those cases where that turned out to be bad. So I'm sorry for the oversight.

[-]

ArabicLawrence@reddit

But is this because gcc does not have a tail-call optimizer?

[-]

kenjin4096@reddit

They do. However, what we needed was not just tail-call optimization, but guaranteed tail calls and a special calling convention.

GCC 15 has the guaranteed tail call, but not the special calling convention. So we couldn't do a comparison to it.

[-]

Bunslow@reddit

I note that some associated PRs have been merged as of just today, right around the time of this thread being submitted:

https://github.com/llvm/llvm-project/issues/106846 https://github.com/llvm/llvm-project/pull/114990

So this is definitely going to be fixed in Clang 20, and I see hints of it being backported into Clang 19?

[-]

Bunslow@reddit

well i suppose good on this guy for doing the digging, and good on cpython for immediately recognizing the good work and including it in the relevant notes.

overall, this is a great example of why it's really important to have two independent compilers, and compiler projects, and it's also a great example of collective open source engineering and cooperative contributions.

[-]