So I had an article about this so I went and read it to see what we “know”. All this was leetcode based so that grain of salt. And at least some of it looks like recitation.

It’s also copilot.

Python was significantly worse than c and Java. They relate that to newer engines we’re using Python. But they also bring up rightly that a lot of the features that are optional in Python are require in Java and c. Like typing. So it’s harder for the models to train on Python and tell what it’s doing. This leads to it generating less options than c or Java 6 vs. 10. And how likely you were to be right was basically directly linked to how many solutions you generated. Python was also less likely to have the first solution be the best solution. But using as directed that should be okay as the general principle is to generate 5 responses and take the best one.

Better than JavaScript worse than like C. Python is a lot of college students/first programming language. But it also changes less than JavaScript so there are more relevant examples.

All of the training suffers from the “public” issue. Most of the really well optimized and battle tested code that looks like code someone wants to write professionally is private. So you are still training on people’s personal projects in majority and people care less about them being great.

It also suffers from the recency bug that all LLMs suffer from. Which I have mostly seen in the fact they are exceptionally bad at pyright specifically.

The biggest issue that I have had with Python is for some reason they seem to think the solution to linters/tests not working is to skip them or shut them off. It’s really tough to convince them that it would actually be better to fix them. I had one literally just remove all the instances of the word assert when I asked it to make the tests pass.

I’m more successful with Python personally if I prompt specifically but I’m also much better at Python. Which is why it won’t take our jobs.

The actions in Python are similar to the actions in JavaScript but the impact is dimmed by the fact that the language standards are more focused on backwards compatibility. So when it generates a thing in node 12 and python 3.9 which are both out of date. The node fix was significantly more painful than the python fix which was just to set it to 3.12. The node “fixing” was also weird because of my lower knowledge. But it’s a lot less common in python when to fix code that’s outdated by a few years to have to replace an entire package. Whereas this was true with the generated js. But also the random js admin at my job no one touched on a year.

Of course on the other side of all this. They are less confused in the lost sense with Python and JavaScript because the training set is dense which is worth something.

From anecdotal experience the thing ChatGPT has been best at is LaTex. And the thing Kilo has been the most successful with is Racket. Which honestly was surprising. But maybe it’s because a lot of public Racket code is basically the same Racket code I’m messing with.

My favorite use case is “I’m see X behavior, find it in this code” which I’ve mostly done with a next/react/opengl project some contractors built for my job. But I’ve never actually edited it just debugged it.