It sucks because it also doesn't say I don't know. You get in infinite loop and it keeps saying I'm sorry try this instead of realizing it's the same answer it gave 2 questions ago that didn't work.
But it knows what it answered in the same conversation. By looking at its own answers it shouldn't repeat an answer that didn't work. I'm OK with it giving a wrong answer. I'm not ok with it getting stuck in a loop.
On Reddit and other online platforms, people don’t tend to respond “I don’t know.” If something is unknown, they don’t respond or they speculate and bullshit.
Wikipedia doesn’t have long articles explaining the scope of what we don’t know about difficult topics: if it’s unknown or poorly understood, there’s either no article, a stub, or a low quality article.
We inadvertently trained it not just on human language, but also on human behavior. But we shouldn’t really want an LLM AI to behave like a human. Or like a Redditor, or even like a Wikipedia writer. That’s not the use case.
Yeah, the predisposition of LLMs to always generate an answer is aggravating. I don't find it useful to have a tool that is confidently incorrect most of the time.
Even if it wasn't, the problems you want to find in code reviews are often tied to business logic or other external knowledge/dependencies that can't fit its context window.
AI code review sounds like a bad idea, even if AI were good at it:
Half the purpose of code review is to keep the team up-to-date with changes to the codebase. That doesn't happen with AI.
The reviewer has a chance to learn (e.g. patterns) as much as the reviewee. Again, doesn't happen with AI.
Automated "code review" should happen while writing, same way as a linter which can run on the dev machine. What's the point of calling it "code review" if it's just AI linting?
Except when you are a solo dev, and you would still like someone to review your code and point out potential issues/blindspots.
I am one, it's useful to me, even if it isn't always right/gives immediately actionable advice.
Have to disagree here. I’ve started used an ai code review tool and it has already found 2 serious bugs overlooked by me and reviewer. Granted there is also a lot of noise, but it’s worth it despite the false positives
Agreed. I’m a founder of a popular LLM powered code review tool funded by Y Combinator and we see dozens of sign ups every day.
People just want a second pair of eyes looking for stupid mistakes. It doesn’t replace people, but if it does catch 1 issue/week it’s worth the $20/month
The horse meant to be contained by that particular barn door bolted about fifty years ago and has since lived to a ripe old age and died surrounded by loving grandchildren in Nepal. That ship has sailed, struck an iceberg, sunk to the bottom of the Atlantic, and become the subject of a James Cameron documentary. Not only is the genie out of the bottle, but the bottle has been shattered into a million pieces and recycled into iPhone screens.
Well the word artificial just means made by humans, although I understand it has come to carry the connotation of being inferior to its "natural" counterpart.
Haha you're right although that's an older term which has its own meaning. I would say LLMs are marketed with the term "AI" in a way that implies they are AGI.
AGI is like a very powerful form of AI. So both LLMs and AGI (whatever the implementation) would be in the bucket of "AI."
He has ceased to be! ‘E’s expired and gone to meet ‘is maker! ‘E’s a stiff! Bereft of life, ‘e rests in peace. ‘E’s off the twig! ‘E’s kicked the bucket, ‘e’s shuffled off ‘is mortal coil, run down the curtain and joined the bleedin’ choir invisible
When people think AI, the hype has somehow convinced people it's AGI... The reality of an LLM is nowhere near that, it's very process intensive pattern matching, that's it.
Chess engines are AI. LLMs are a subcategory or form of AI. But LLMs aren't the extent of AI. So it's wrong to say "AI" when you specifically mean LLMs.
The use of the word "AI" in this context is so vague that you might as well say "algorithms."
1) Comment your code, it provides the context a LLM needs to be better at it's job.
2) Obviously don't take it too seriously, it's a tool not a person.
Personally, I have a tool for flagging code at compile time. This tool can also print all the changes for a feature including some surrounding context. It can generate prompts out of this code context + templates + project information + feature information, and those prompts generate me really good documentation. Like good enough that everyone who sees it is fairly happy with it, because it's better than we get out of humans.
We also have an AI code reviewer, and while it's not the worlds best reviewer by a long shot, it does point out legitimate issues. It also doesn't give stamps and we still require human reviewers, so I don't really see the problem with that.
Do you have reading comprehension issues? 99% of the output of AI for any nontrivial task is absolute garbage. A real junior can at least learn in the review process to start writing acceptable code, but AI outputs stuff worse than the average junior most of the time, and that code has to be rewritten.
No that's you having a skill issue, jealousy runs really deep for those like you who have 1% of the talent needed in order to make proper use of AI for nontrivial tasks and can only produce garbage.
You should not use AI to do code reviews, and you should not have to rewrite all of the output of AI if you put some time and effort into gaining experience using AI for coding.
I don't need to tell myself anything, I get paid a lot of money, and my peers (other Sr devs), and our CTO, have so far never had any problems with my code reviews, It says everything for me.
My point is that that's completely worth it. You're getting a code review in 2 seconds that would take a human 10 minutes. It then takes you 30 seconds to read and ignore the stupid things. 9m30s gain.
I mean it's almost as if, beyond simplistic cases, giving pre-determined answers Y and Z to specific input X based on what was said on the web or elsewhere before doesn't work as you don't actually know whether that's a good answer for this input. But the LLM can't possibly know that, because before, it was a good answer.
And yeah it's almost as if that is obvious, and the whole point of a (good) code review is to not do this. LLMs for code review are just Sonarcube, but in bad because they aren't deterministic.
I don’t think this is a universal statement. Human reviewers working for corporations are only as good as the business values good code. Which agile typically does not have the development cycle to have rigorous reviews, testing and/or refactoring. Plenty of code reviews take place without even running the code in question..
I have plenty of lines of bad code in production because as the application changed over time those pieces of bad code never fell in line with the current sprints stores. So it will sit in perpetuity until it’s sunset altogether.
Could I just go and refactor it? Sure, but I don’t have a business case to do so, or at least not one that would matter enough for the business to allow it.
Almost all things mentioned as where AI shines, are still things you really want in your IDE instead. Great tools, but they should be real-time, not after the fact in code review.
If someone told me in 2001 that in 2024 people would simply assume that a randomly generated computer program that can generate passably good English is automatically capable of doing code reviews, I’d have put my time into learning outdoor survival skills.
"isn't guaranteed to spot all problems or issues...and sometimes it will make mistakes", well to be fair, this also accurately describes human developers doing code reviews
I_Hate_Traffic@reddit
It sucks because it also doesn't say I don't know. You get in infinite loop and it keeps saying I'm sorry try this instead of realizing it's the same answer it gave 2 questions ago that didn't work.
BitterArtichoke8975@reddit
CoPilot for Security sucks more
ForgettableUsername@reddit
It doesn’t know how to not know something.
I_Hate_Traffic@reddit
But it knows what it answered in the same conversation. By looking at its own answers it shouldn't repeat an answer that didn't work. I'm OK with it giving a wrong answer. I'm not ok with it getting stuck in a loop.
ForgettableUsername@reddit
Does it, though?
I_Hate_Traffic@reddit
Yes it keeps track of conversations for sure. Even in a new conversation it knows what you talked about in other tabs.
ForgettableUsername@reddit
If keeping track of something was the same as knowing it, my conversations with the project manager at work would be a lot less frustrating.
mfitzp@reddit
Given it was trained on internet comments this makes a lot of sense.
ForgettableUsername@reddit
On Reddit and other online platforms, people don’t tend to respond “I don’t know.” If something is unknown, they don’t respond or they speculate and bullshit.
Wikipedia doesn’t have long articles explaining the scope of what we don’t know about difficult topics: if it’s unknown or poorly understood, there’s either no article, a stub, or a low quality article.
We inadvertently trained it not just on human language, but also on human behavior. But we shouldn’t really want an LLM AI to behave like a human. Or like a Redditor, or even like a Wikipedia writer. That’s not the use case.
PandaMoniumHUN@reddit
Trained on comments from reddit experts. You're gonna love it
tcpukl@reddit
It doesn't know anything
LeRealSir@reddit
in fact, it doesn't know anything.
moreVCAs@reddit
It doesn’t know how to know something either
Carighan@reddit
Yeah, it's a chinese room.
fishling@reddit
Yeah, the predisposition of LLMs to always generate an answer is aggravating. I don't find it useful to have a tool that is confidently incorrect most of the time.
fluffy_in_california@reddit
Worse, sometimes it start trying to 'fix' it's proposed change and makes it even worse
Wonderful-Wind-5736@reddit
Yup, I stopped asking for corrections. Either it works on first try or I let it write a few tests and fix the rest myself.
relent0r@reddit
So much this, if you don't already know the subject matter your going down a deep rabbit hole of frustration.
Harzer-Zwerg@reddit
you notice immediately when "AI" like chatGPT has not been "trained" with enough data beforehand: then only rubbish comes out.
Basically, it is just a tool, but by no means a replacement for a dev.
OffbeatDrizzle@reddit
Even if it is trained with enough data it's all junior dev crap anyway
theeth@reddit
Even if it wasn't, the problems you want to find in code reviews are often tied to business logic or other external knowledge/dependencies that can't fit its context window.
Harzer-Zwerg@reddit
yes and often with errors/nonsense... I actually only use it as a more convenient alternative to googling...
MassimoCairo@reddit
AI code review sounds like a bad idea, even if AI were good at it:
gabrielmuriens@reddit
Except when you are a solo dev, and you would still like someone to review your code and point out potential issues/blindspots.
I am one, it's useful to me, even if it isn't always right/gives immediately actionable advice.
oclafloptson@reddit
I'm less trusting of AI because I'm a solo dev and have no other sets of eyes to catch its errors. Adding a layer of AI just makes my job harder
robhaswell@reddit
So you're saying it's sometimes better than nothing.
Anyway, you should give reviewing your own code a try. It's surprisingly useful.
oalbrecht@reddit
The first part could be solved by the AI having a daily standup where it lets everyone know what they missed. /s
Beneficial-Ad-104@reddit
Have to disagree here. I’ve started used an ai code review tool and it has already found 2 serious bugs overlooked by me and reviewer. Granted there is also a lot of noise, but it’s worth it despite the false positives
Man_of_Math@reddit
Agreed. I’m a founder of a popular LLM powered code review tool funded by Y Combinator and we see dozens of sign ups every day.
People just want a second pair of eyes looking for stupid mistakes. It doesn’t replace people, but if it does catch 1 issue/week it’s worth the $20/month
Realistic_Income4586@reddit
What was the tool's name? Can you use a local llm?
gareththegeek@reddit
And yet everyone demands I stop writing code and start reviewing the AIs code. It can dish it out, but it can't take it.
Newguyiswinning_@reddit
Why would it be good at them? You all do realize AI cant solve problems right? If you think AI can do a code review, you should not be using AI
billie_parker@reddit
We should stop saying "AI" and start saying "LLMs." AI is a very general term co-opted by marketing hype.
ForgettableUsername@reddit
The horse meant to be contained by that particular barn door bolted about fifty years ago and has since lived to a ripe old age and died surrounded by loving grandchildren in Nepal. That ship has sailed, struck an iceberg, sunk to the bottom of the Atlantic, and become the subject of a James Cameron documentary. Not only is the genie out of the bottle, but the bottle has been shattered into a million pieces and recycled into iPhone screens.
uJumpiJump@reddit
High quality comment
billie_parker@reddit
I guess we'll have to start saying "literally AI" or something similar. Or invent a new term. Thanks a lot, idiots
Enerbane@reddit
Well, probably for the best, because I think it might be rude to call something artificial intelligence if it actually is artificial intelligence.
billie_parker@reddit
I don't know what you mean by that
Enerbane@reddit
It's a joke. If something is genuinely "intelligent" then it's probably going to be rude to call it artificial.
billie_parker@reddit
Well the word artificial just means made by humans, although I understand it has come to carry the connotation of being inferior to its "natural" counterpart.
ForgettableUsername@reddit
That’s awfully anthropocentric. Perhaps artificial intelligences will take pride in being artificial.
AngryHoosky@reddit
The new term is Artificial General Intelligence (AGI).
tcpukl@reddit
That's not a new term at all. It's been around decades.
billie_parker@reddit
Haha you're right although that's an older term which has its own meaning. I would say LLMs are marketed with the term "AI" in a way that implies they are AGI.
AGI is like a very powerful form of AI. So both LLMs and AGI (whatever the implementation) would be in the bucket of "AI."
overlordmouse@reddit
He has ceased to be! ‘E’s expired and gone to meet ‘is maker! ‘E’s a stiff! Bereft of life, ‘e rests in peace. ‘E’s off the twig! ‘E’s kicked the bucket, ‘e’s shuffled off ‘is mortal coil, run down the curtain and joined the bleedin’ choir invisible
overlordmouse@reddit
r/suddenlymontypython
_senpo_@reddit
lmao I laughed at those phrases
mb194dc@reddit
When people think AI, the hype has somehow convinced people it's AGI... The reality of an LLM is nowhere near that, it's very process intensive pattern matching, that's it.
abraxasnl@reddit
LLM is an implementation detail. LLMs are one way to implement AI. I will die on this hill (insert Wilhelm scream).
fishling@reddit
Why would that be a hill anyone would choose to die on?
jambonetoeufs@reddit
The problem I see is AI means different things to different people. There’s not a common understanding of what’s “AI”.
TwentyCharactersShor@reddit
Eh? Aye. In Yorkshire means: "I'm sorry, I didn't hear you. Please repeat....oh yes.".
Wonderful-Wind-5736@reddit
If you want to be that picky, at least be accurate. You could describe GPT er. al. functionally autoregressive text completion models.
tcpukl@reddit
I just call them glorified pattern matchers.
billie_parker@reddit
Chess engines are AI. LLMs are a subcategory or form of AI. But LLMs aren't the extent of AI. So it's wrong to say "AI" when you specifically mean LLMs.
The use of the word "AI" in this context is so vague that you might as well say "algorithms."
TrevorPace@reddit
Most modern chess engines heavily rely on LLMs actually.
jdm1891@reddit
Do you mean transformers? Or just the concept of a weighted network?
Either way they're not LLMs
PiotrDz@reddit
Where did you get that info from
Dragdu@reddit
And this is why you don't LLMs write your comments for you.
billie_parker@reddit
You're horribly misinformed
TrevorPace@reddit
No, I got your point, but you are right I was misinformed.
Bodine12@reddit
Or even better, “Text predictors.”
Bodine12@reddit
Or even better, “Text predictors.”
danted002@reddit
I’ve been doing it for months now. It’s an LLM not an AI
kuzux@reddit
I'd just like to interject for a moment. What you're refering to as AI, is in fact, LLMs/AI, or as I've recently taken to calling it, LLMs plus AI.
7heblackwolf@reddit
But who tf call them LLM? Not even the very people involved in the development.
billie_parker@reddit
In my experience that's not true, but to the extent that it is - that's just emblematic of the problem
st4rdr0id@reddit
If by AI you mean fake generative chatbots, then yes, they suck at pretty much everything except mimicking their training set.
HaMMeReD@reddit
I'm going to counter this in a few ways.
1) Comment your code, it provides the context a LLM needs to be better at it's job.
2) Obviously don't take it too seriously, it's a tool not a person.
Personally, I have a tool for flagging code at compile time. This tool can also print all the changes for a feature including some surrounding context. It can generate prompts out of this code context + templates + project information + feature information, and those prompts generate me really good documentation. Like good enough that everyone who sees it is fairly happy with it, because it's better than we get out of humans.
We also have an AI code reviewer, and while it's not the worlds best reviewer by a long shot, it does point out legitimate issues. It also doesn't give stamps and we still require human reviewers, so I don't really see the problem with that.
mb194dc@reddit
LLMs are not AI, that's your problem right there.
AGI would be able to do code reviews, no problem.
notjshua@reddit
of course.. this is why we have to code review the output of AI
hopefully in the future a smarter model will be able to do it all :>
ForgettableUsername@reddit
That’s just code review with extra steps!
notjshua@reddit
what extra step?
cummer_420@reddit
Rewriting the dogshit output of a shitty LLM trained seemingly on primarily first projects.
notjshua@reddit
you think code reviews means rewriting everything they wrote? that's not really how it works. I didn't write its training data.
cummer_420@reddit
Do you have reading comprehension issues? 99% of the output of AI for any nontrivial task is absolute garbage. A real junior can at least learn in the review process to start writing acceptable code, but AI outputs stuff worse than the average junior most of the time, and that code has to be rewritten.
notjshua@reddit
No that's you having a skill issue, jealousy runs really deep for those like you who have 1% of the talent needed in order to make proper use of AI for nontrivial tasks and can only produce garbage.
You should not use AI to do code reviews, and you should not have to rewrite all of the output of AI if you put some time and effort into gaining experience using AI for coding.
cummer_420@reddit
Lmao, keep telling yourself you've got "good" output from AI. Something merely working is not good.
notjshua@reddit
I don't need to tell myself anything, I get paid a lot of money, and my peers (other Sr devs), and our CTO, have so far never had any problems with my code reviews, It says everything for me.
ClownPFart@reddit
Title is 3 words too long
PeepingSparrow@reddit
"AI is... le bad!" is such a tired topic on this sub. Preaching to the choir
-a-non-a-mouse-@reddit
Nah, AI (at least Claude 3.5) is pretty damn good at code reviews as long as you actually read what it says.
Are half of the things it says nonsense? Yep. Ignore those. The other half aren't.
Ihavenocluelad@reddit
Same here. Half of the time its complaining about stupid stuff, other half it actually catches something important and helps a lot. So semi worth it?
-a-non-a-mouse-@reddit
My point is that that's completely worth it. You're getting a code review in 2 seconds that would take a human 10 minutes. It then takes you 30 seconds to read and ignore the stupid things. 9m30s gain.
Ihavenocluelad@reddit
Yeah I agree. I actually developed an internal tool for our company that does exactly this so im already sold :P
SoInsightful@reddit
AI Sucks at Writing This Article
illuminatedtiger@reddit
If you're going to use an LLM for code reviews there's little to no benefit in doing code reviews at all.
blizzacane85@reddit
Al is better at scoring 4 touchdowns in a single game for Polk High during the city championship in 1966
trekit_@reddit
Is cursor any better? Thinking to add it as a ide plugin
7heblackwolf@reddit
AI sucks at PR because humans sucks at PR.
Pharisaeus@reddit
Or maybe just because there simply isn't much code review on the internet to teach the model on?
Carighan@reddit
I mean it's almost as if, beyond simplistic cases, giving pre-determined answers Y and Z to specific input X based on what was said on the web or elsewhere before doesn't work as you don't actually know whether that's a good answer for this input. But the LLM can't possibly know that, because before, it was a good answer.
And yeah it's almost as if that is obvious, and the whole point of a (good) code review is to not do this. LLMs for code review are just Sonarcube, but in bad because they aren't deterministic.
Sp33dy2@reddit
Humans are not good at code review either.
Pure-Huckleberry-484@reddit
I don’t think this is a universal statement. Human reviewers working for corporations are only as good as the business values good code. Which agile typically does not have the development cycle to have rigorous reviews, testing and/or refactoring. Plenty of code reviews take place without even running the code in question..
I have plenty of lines of bad code in production because as the application changed over time those pieces of bad code never fell in line with the current sprints stores. So it will sit in perpetuity until it’s sunset altogether.
Could I just go and refactor it? Sure, but I don’t have a business case to do so, or at least not one that would matter enough for the business to allow it.
danger_falling_ice@reddit
PREACH!
brotatowolf@reddit
So do my coworkers
jimbojsb@reddit
I mean think of what it was trained on. Of course it does.
poop-machine@reddit
I'm doing my part!
Terribleturtleharm@reddit
Yes, it's becoming more human every day.
MainEditor0@reddit
Only for now
abraxasnl@reddit
Almost all things mentioned as where AI shines, are still things you really want in your IDE instead. Great tools, but they should be real-time, not after the fact in code review.
topological_rabbit@reddit
AI Sucks ~~at Code Reviews~~
huyvanbin@reddit
If someone told me in 2001 that in 2024 people would simply assume that a randomly generated computer program that can generate passably good English is automatically capable of doing code reviews, I’d have put my time into learning outdoor survival skills.
mgntom@reddit
"isn't guaranteed to spot all problems or issues...and sometimes it will make mistakes", well to be fair, this also accurately describes human developers doing code reviews