Is Claude from Anthropic the best AI Code Assist in the market?

[-]

Odd-Environment-7193@reddit

Can I make a suggestion. Try out mistral large. You can signup to their platform and get 150 USD per month free. The models are very capable. Mistral loves to spit out super long pieces and it is very useful.

I built my own chatbot, so my flow is sometimes, start with 01 mini for brainstorming -> then work through the code with gemini or claude or gpt4. Then when I want the final answer I always switch to mistral and it spits long unbroken code really well.

The larger LLMS are great problem solvers but they all have their pros and cons.

Claude is a good one to subscribe to. But If you like to do stuff for free, try out gemini studio aka makersuite. Then try out mistral. You can even hook these free models up to something like a continue plugin.

Also a good free model for tabbing and autocompletions is codestral. It's also free by mistral.

Go see what tickles your fancy.

[-]

Agile-Seesaw8541@reddit

Hi Mistral models are really bad when compared to claude 3.5 sonnet. Especially for any structured output generation. They hallucinate on anything specific. I tried to build a text2 sql application and somehow it refused to return the specific columns.
But this was just my experience. Happy to learn more

[-]

brokester@reddit

Any specific prompts for sonnet 3.5 ? I'm not that impressed honestly, talks too much

[-]

Odd-Environment-7193@reddit

Why are people downvoting your opinion lol? Some people have befriended these tools and can't take any criticism.

Here are some system prompts you can try:

Try this prompt for longer more elaborate responses:

All questions should be answered comprehensively with details, unless the user requests a concise response specifically. Respond in the same language as the query.

Try this prompt for shorter more concise responses:

"All questions should be answered concisely, unless the user requests a comprehensive response specifically. Respond in the same language as the query."

The type of reflective talking behavior from claude is supposed to lead to better answers. But it can be very annoying. I prefer to use base models that do not have that behavior, then add a special reflect/chain of thought style system instruction if needed.

[-]

arm2armreddit@reddit

exactly the same impression with mistral! moving back to Sonnet. sometimes, I need to change gpt4o, back, and force, but mostly 3.5 sonnet

[-]

ainz-sama619@reddit

Mistral unfortunately isn't even in the same league as Sonnet 3.5. the gap is huge in my experience

[-]

Odd-Environment-7193@reddit

Well, you should be using structured outputs If you want JSON responses(if not, a couple of shots with a JSON parser on the retry). Claude is one of the top models on the market; it definitely has an edge. I built an AI studio where I can access about 11 top models and switch between them mid-conversation. I rarely use Claude for anything anymore. Like I said, I usually work through the problems and then get mistral large (latest) to give me the final outputs. It tends to spit out thousands of tokens in one shot until it exhausts itself, then continues. Which is the behavior I am after. Every other model, including Claude, has issues with this. Claudes quality has massively gone down and the chat (20$) per month is ridiculously restricted, after a few messages you cap out. I cannot work like that. When models start deleting (for whatever purpose), I quickly lose interest in them and start shopping around again.

I currently use these:
gemini-1.5-pro-exp-0827, gemini-1.5-pro-002, gpt-4o, gpt-4o-mini, o1-preview-2024-09-12, o1-mini-2024-09-12, mistral-large-latest, mistral-small-latest, open-mistral-nemo, pixtral-12b-2409, gemini-1.5-flash-002, gemini-1.5-flash-8b-exp-0827, claude-3-opus-20240229, claude-3-5-sonnet-20240620, claude-3-haiku-20240307, meta-llama/Llama-3.2-11B-Vision-Instruct-Turbo, meta-llama/Llama-3.2-90B-Vision-Instruct-Turbo

It's a lot of fun to bounce around according to your needs. Lots of people swear by Claude and use It for everything. It's a great model. Just not a fan of their system overall(chatbot side).

[-]

kyazoglu@reddit

I did my own testing last week on the newest LeetCode problems. Be aware that being able to solve these kinds of complex questions actually requires a high level of reasoning. So I wasn't just testing coding abilities, but a combination of coding and reasoning. (I actually came to this conclusion after seeing the results.)

The results were shocking to me. o1-preview (and mini) was off the charts, doubling the score of the model in 2nd place. And surprisingly, Sonnet was even behind GPT-4o, in 3rd place. This got me thinking: either I was doing something really wrong, or this benchmark was not just about coding, and I believe the latter is more accurate.

I'll share the results once I have them automated. Since I did it manually, there was probably some human error involved. But one thing is for sure: o1 is extremely good at complex coding (that involves advanced reasoning).

That being said, if you are prompting to get the skeleton code of a new project, or something that can be written easily using the documentation of a framework/language, that's where probably Sonnet shines.

[-]

Tramagust@reddit

people keep praising o1 but my testing hasn't gone that well. Claude 3.5 sonnet still does a slightly better job overall.

[-]

diligentgrasshopper@reddit

In my experience o1 is only good for some very specific hard problems that Sonnet couldn't handle. On average difficulty coding it's either on par or slightly worse than Sonnet.

The worst part about o1 though is that after spending a long time thinking through the problem it can still make mistakes and we still need to prompt iteratively. At that point you might as well use a conventional model.

[-]

Odd-Environment-7193@reddit

It's good. It can solve some things I can never get right on claude. Then I throw it in 01 and it kills it. It is not a all the time model though. Rather just use it for brainstorming or when you really hit a wall with your other models and can't seem to find a solution. That's when it really shines through.

[-]

ObnoxiouslyVivid@reddit

I'm curious, can you give an example of problems that claude hits a wall with? I wonder if it can be solved with prompting or more of a fundamental limitation of the model.

[-]

Environmental-Metal9@reddit

Parsing YAML from scratch without a library using only C#. I had to go to o1 for that specifically, and then bring a solution to Claude to continue.

[-]

notathrowacc@reddit

I'm currently using it to build a web app with their Projects. Usually Claude got it 90% right but there are some weird bugs or decisions especially on the frontend. I've exchanged debug messages and some suggestions to fix but it kept sending weirder or roundabout way to fix it. At this time I usually opened a new prompt for a fresh start with the first outputted code, and it worked like 50% of the time.

After o1, I just sent the buggy outputted code to o1 and it magically got everything right every time, plus a few optimizations I didn't even think about.

[-]

olenjan@reddit

I have seen o1-mini do amazing when describing a complex problem with requirements if starting from scratch, but struggle if you provide code or ask changes.

3.5 sonnet does well when asking for changes and follows instructions better.

[-]

ObnoxiouslyVivid@reddit

I've seen this a lot as well. Seems like it was trained for single-shot and breaks down when requirements change or you need modifications.

I imagine most complex problems are about modifying an already existing system. They don't exist in a vacuum that o1 was trained on.

[-]

Jesus359@reddit

Mine is a hit or miss. Right now most time Claude will get it right but now there have more instances where GPT is more helpful coding than Claude only because I can self correct GPT quicker with a semi working code from Claude if needed.

Claude did go down in quality though. I really hope they’re not trying to be the same as the other LLMs and focus on their strengths with Claude.

[-]

brucebay@reddit

interesting I have not seen the quality decrease. but I only use it for python and mostly data analytics and machine learning. I noticed it is harder to hit usage limits so nay be they are using some tag or quantizations.

[-]

Jesus359@reddit

Yeah I’m focusing all my code on python since it’s the most versatile for a bunch of things. I’ve hit my limit with Claude once or twice and my 4o limit a couple of times but yeah the $20 are going to OpenAI so far.

[-]

West-Code4642@reddit

I like a combination of both

[-]

arnoopt@reddit

Its integration within Cursor is amazing indeed. Coding is pleasant once again!

[-]

TempWanderer101@reddit

It's worth mentioning that Cursor only allows 500 calls to Sonnet per month, which may not be enough for some people.

[-]

arnoopt@reddit

It’s 500 fast requests, the other might just be slower at rush hour.

[-]

TempWanderer101@reddit

Thanks for clarifying. How slow is slow, exactly?

[-]

arnoopt@reddit

What are fast and slow uses? Fast uses of premium models are given first priority by our backend. On Pro, once you hit your fast usage limit, you can still use premium models, but your requests may be queued behind others at times of high load.

https://www.cursor.com/pricing

[-]

Environmental-Metal9@reddit

Can’t you also provide your own API keys for either of those and bypass the wait, at API call cost? (Admittedly quite high for o1 models in comparison, but pretty doable for Claude)

[-]

Remote_Top181@reddit

You can buy 500 more for $20.

[-]

TechnoTherapist@reddit

Claude (Sonnet 3.5) is your best daily driver.

When you need extra horse power, you can use o1-preview.

Also since you're coding, give Cursor and/or Aider a shot. (with Sonnet 3.5). Prepare to be amazed. :)

[-]

Environmental-Metal9@reddit

Heck, get cursor and go for both Claude AND o1-preview. Use o1-preview to start a project from scratch, say, by writing the README.md of the project with as many details as you can think of, and use that as a reference (built in RAG) to prompt o1-preview to write the files necessary for this project, and then switch to Claude to continue the work. This workflow has allowed me to experiment with tens of ideas that I would never even have tried before, and some poc projects even turned out to work really really well! And I even learned enough Go in the process to use sans llm

[-]

GortKlaatu_@reddit

I have found that Claude 3.5 Sonnet provides me with hallucinated libraries and long deprecated function calls. I wish internally it would use a tool to fact check against the current documentation for the libraries it believes it need to use for coding examples.

It's no doubt more time consuming for the initial answer, but it'd be infinitely more usable.

Additionally, I've seen Claude completely ignore specific details from the initial prompt. Such as if I ask for a langgraph example using llama3.2 through ollama, it'll import libraries which don't exist and it'll switch it to llama2 because that's what existed when Claude was trained... even though I just told it exactly what to use.

I'm thinking about cancelling my Anthropic subscription and either switching to OpenAI and/or replacing it all with agents to give me a better first pass for code.

[-]

kesor@reddit

I am working (its slow) on an agentic swarm for myself that will have a google/tavili search and wolframalpha and other "tools" that it can use to get better informed about updated features. I even created a custom search that includes just the python libraries that I need/want to search though in my project.

The python docs search is here: https://cse.google.com/cse?cx=61ab9b46e7a3f4dd4 . At the moment, I told it to only search through 22 python documentation websites. And I already have a piece of code that can programmatically from python send a request to this search, and scrape the links returned to the result.

docs.pydantic.dev
*.mypy-lang.org
docs.pytest.org
flask.palletsprojects.com
*.realpython.com
mlflow.org/docs/
docs.ray.io/
fastapi.tiangolo.com/
*.keras.io/
www.tensorflow.org/api_docs/python/tf
pytorch.org/docs/stable/
docs.scipy.org/doc/scipy/
numpy.org/doc/
www.llama.com/docs/
plotly.com/python/
scikit-learn.org/stable/
pandas.pydata.org/docs/
huggingface.co/docs/
docs.python-guide.org/
docs.djangoproject.com/
docs.python.org/
*.readthedocs.io/

[-]

Environmental-Metal9@reddit

Do you have a GitHub, blog, newsletter, or something that I could follow progress on that? I’ve been toying with the idea of either writing a simple ComfyUI for LLMs, or write some custom nodes for regular comfyui, and your idea lets itself really well to the notion of nodes that perform specific actions.

[-]

brucebay@reddit

it almost always makes mistakes at the beginning but quickly recovers. I have seen both gpt4 and gemini repeating the same mistakes like a loop.

[-]

TempWanderer101@reddit

This is the main issue I had with Claude as well, although the issue usually occurs with less popular libraries. The issue is not specific to Claude though.

[-]

brandall10@reddit

I recommend using the API rather than a sub. With continue.dev I push 3.5 pretty hard, and with occasional 4o use only end up spending about $15/month total.

[-]

vhthc@reddit

I have the same issues with chatgpt 4o. outdated/deprecated/hallucinated libraries, especially for python and rust

[-]

ph33rlus@reddit

I just used GPT to modify some code. Claude couldn’t handle it. Not enough context? Not sure if it’s because I’m using the free version though

[-]

ObnoxiouslyVivid@reddit

Claude + aider is an amazing combo, and you don't even need to pay a subscription for it.

Claude really excels at only touching the important parts and keeping the rest as is. OpenAI models really struggle to keep my code as it was, they always try to rewrite unrelated parts or remove random bits like comments, etc.

[-]

lolzinventor@reddit

When you really start pushing the limit of what AI code assist can do, subtle things start to matter such as the approach taken by the AI to solve a problem. Claude 3.5 and o1 sometimes approach a problem in different ways, so being able to easily switch between them helps give you this perspective. Most of the time I use Claude, but if it gets stuck I give o1 a try, and on at least one occasion it has solved the problem in a different way, that worked... I initially switch to Claude because of gpt4 laziness. Claude is lazy, but slightly less lazy imo. Telling it that trying to save tokens by only giving half an answer will waste more tokens because I will only re-request the full response seems to help. In both cases I think you are better of using the API than using a subscription.

[-]

AMGraduate564@reddit (OP)

How do we use the Claude API?

[-]

AbleSugar@reddit

For my use case I use Codestral until it can't do what I need and then I ask Claude (both in Continue.dev). 99% of the time Codestral does what I need. Work pays for Co-pilot, but I've found that Codestral + Claude is better for my use cases (even if I'm paying for Claude API)

[-]

ThaisaGuilford@reddit

If they're open source, yes.

[-]

ainz-sama619@reddit

Claude isn't open source, and all open source models suck ass for coding

[-]

ThaisaGuilford@reddit

Yeah that's why I'm in r/localLlama

[-]

brucebay@reddit

yes it is. i have pro subscription to openai, claude and gemini. Claude beats them everytime. don't get me wrong, it makes mistakes, almost any initial code. but quickly fixes them. knows how to debug its own code. sometimes I just upload others code, and ask it to change the way I want it to work, recent example, somebody put a local gui for joy caption alpha two. it expected either 4 bit model for btb or full model. couldn't load in 8 bit because of the code was written. I asked Claude to convert the code to cli with 8bit load support. after 4-5 iteration with debug statements it put, Claude made it work. overall context was pretty long due to side distractions but I didn't hit the usage limits. if debugging was not impressive, going back to 10 response earlier and making a change on that code was.

sometimes Claude generate a solution that is overly complicated, so I just point the right direction and it changes accordingly. I still use other providers to ask an architectural solution, then decide what will I do, and ask Claude to implement it.

it also keeps exactly the code in unchanged code sections at each iteration. the last time I checked chatgpt was rewriting almost everytime which is annoying if you make some local changes.

the only issue is when I diverge from original problem, and ask some questions, then go back to original problem, it may still generate code that is relevant the side question. for example, I ask it to generate some dataframe, and later ask Claude modify it on my local code for another purpose,. after that if I ask it a slight change in the original code it would integrate the same dataframe changes I don't want yet.

and no o1 never produced better code for me. sometimes I wonder if openai is using chatbots to create fake positive reviews.

[-]

angry_queef_master@reddit

Claude overcomplicating things is so annoying. I have to constantly tell it not to add anything extra.

[-]

brucebay@reddit

Interesting. It never adds anything extra for me, but the way it implements the solution could be over complicated. I don't have any recent example but just to illustrate my point it would be on par with writing a for loop and using loc function to update a column in pandas, instead of using apply function.

[-]

synn89@reddit

Sonnet 3.5 has been the most reliable for me. Though I've heard great things regarding Qwen 2.5 72B for coding.

[-]

MrWeirdoFace@reddit

I found claude 3.5 sonnet best for larger multi-script projects, but gpt 4o best for single scripts under 200 lines.

I say this as someone who's not an actual coder, so I've just had better luck getting working scripts under these circumstances.

[-]

ExhibitQ@reddit

O1 and sonnet 3.5 are neck and neck depending on use - case.

Sonnet because of limits is more usable.

[-]

cmdr-William-Riker@reddit

I switched to anthropic last month and don't regret it. The artifacts interface and project system is useful enough to me to justify the price right now, not to mention they tend to follow through with what they say they're going to do whereas openai always seems to have some massive internal drama and a hype train going that usually just leads to more disappointment

[-]

Super_Spot3712@reddit

claude 3.5 sonnet is my go to. good for general tasks and fast/efficient

[-]

kesor@reddit

I have been writing a piece of code with GPT-4o+Canvas yesterday. It did an okay job, but eventually it gets to this point where it just doesn't pull through. So I copy-pasted the thing into GPT-o1-preview, and holy macaroni! That thing ripped apart into new concerns, fixes, and niche library features that I would never have thought of or noticed myself. But then again, O1 is pretty bad at iterating.

On the other side of the spectrum, I tried to use Cursor IDE with my local Llama3.1-8B (also Llama3.2-3B) models. And these models simply don't have the diff-editing capabilities that Sonnet does. Having a model that can do diff-editing and not just whole-file-editing is a huge feature that has major importance. I wouldn't mind having models that can also do chunk editing instead of only diffs on huge files. But that is just me and my fantasies.

[-]

Agile-Seesaw8541@reddit

And claude does not have a code interpretter yet. Once it gets that, it will shatter all the benchmarks that o1 currently has.

[-]

brucebay@reddit

not sure what do you mean by code interpreter. it understands others code, and once I saw it actually execute its own code to validate something like gpt4 analytics does. I believe it was not hallucinating because if I remember correctly gui was showing the progress and not the response section in the chat.

[-]

Jesus359@reddit

Have you tried qwen? Qwen is supposedly trained with Claude so it might be better with coding than the rest.

[-]

Competitive-Dark5729@reddit

I have qwen coder 2.5 running locally. It sometimes fixes things o1-preview messed up. I didn’t try it enough to say how good it is for from scratch coding. I’d say for now a combination of models is needed.

For best results for specific tasks, you can fine tune a model with library code and code examples you find online. Need a lot though.

[-]

Qual_@reddit

ive tried it and... sometimes it's good... sometimes it's really dog shit, like magnitude worse than GPT 3.5.

I have really good hope for the 35b version tho"

[-]

Competitive-Dark5729@reddit

Yeah. That was my first impression too. Can’t really tell the difference, for small code blocks it seems not bad

[-]

kesor@reddit

I have it, but have not tried coding with it all that much. My go to is still GPT-4o, especially with the Canvas, and when it starts messing up I copy paste into O1.

[-]

Jesus359@reddit

I mean we’re taking little models here compared to however big those companies are who probably have a whole room. That’s pretty good especially how early it is for all of this is.

[-]

kesor@reddit

I am working on an agentic "swarm" kind of thing that hopefully will work by using just small models and a lot of tools. So I will definitely be giving qwen a deeper look and evaluation, and compare that to llama3.2, gemma, deepseek, phi, and the like. I suspect that a key point for the smaller models is to give them small tasks. For example, tool calling should certainly work fine. Doing a ReAct cycle on fixing a single unit test should also work fine. But these are not human-interaction kind of activities, you need to give them some autonomy and a goal to reach and let them do their thing for a while before they'll be able to reach that goal. At least this is how I understand it.

[-]

Odd-Environment-7193@reddit

This is also my experience. When you hit a wall 01 can give you that breakthrough or solve that issue you just can't seem to get right. Love it for that.

[-]

AMGraduate564@reddit (OP)

So the final verdict is still in favor of Claude?

[-]

Informal_Warning_703@reddit

If you want an LLM to write code “from scratch”, because you are just starting a project, o1-mini is the best choice because the damn thing won’t shut up and spits out a ton of code no matter what.

If you are trying to refactor or debug complex code, o1-preview because it’s noticeable better than anything else on the market.

If you’re doing standard refactoring or adding features, Claude can handle these better than o1-mini because it stays “focused” on the instructions and doesn’t waste a ton of tokens. It tends to not go for ridiculously over-engineered solutions, which is the main problem I’ve had with o1-preview.

Overall, o1-preview is best, and many times by a significant margin. But after presenting my code and the problem, I often have to tell it “No, don’t try to solve the problem by refactoring these three files, just pass the argument through such and such.” Claude, when it can find a solution, usually finds one that’s good enough.

[-]

TempWanderer101@reddit

It depends on the language too. Both GPT-4o and Claude can trip up in some languages or frameworks.

Another factor to consider is whether an IDE integration would work better for you. Claude Pro does not provide API access, so you can't use it in an IDE.

[-]

TheConnectionist@reddit

It depends.

If you've got a small preexisting code base and want to make specific modifications o1-mini has performed best for me. Though you can occasionally run into the old "# Implement The Thing Here." snippets that everyone hates.

If you've got a larger code base or want to build something from a research paper Claude is definitely better if you're using a good CoT prompt.

[-]

bunk3rk1ng@reddit

Someone will release something better in a couple months and Claude will suddenly become trash and everyone knew it the whole time and nobody should even use it.

[-]

kingp1ng@reddit

I like o1-mini for brainstorming and debugging. Sometimes it goes off on a random tangent.

I like Claude 3.5 for brute force boilerplate. It's fairly concise and stays focused to the prompt.

[-]

Agile-Seesaw8541@reddit

There are slightly different but I think the best thing about Claude 3.5 is that it really follows instructions if you provide an unordered list. I have had no real complaints with Sonnet 3.5. It is the best model out there.

[-]

AleccioIsland@reddit

Claude is amazing, I built a little Python game with it going over several interations and was much Impressed by its capability of adding my additional ideas to the code

[-]

VulpineFPV@reddit

My programming circle hate GPT-all of em, dislike Claude but it’s better, and seemingly they want me running things like Star coder, or other localized models from my rig with API access to them.

At first they didn’t think hallucinations existed, and some believe the art models are sentient, but I just load the models and they connect to em. They like my RAG use cases.

They run the front ends and have recently asked me to host them online for them, they’d pay the GPU and server costs if I can get it up for them.

Deepseek also seems like a great option in their use. I just help nudge the AI and they do the coding. They like this one the most.

[-]

kexibis@reddit

I find Claude for some to have an advantage, but GPT-4o for other things.. don't know

[-]

North-Income8928@reddit

Claude's insanely low limits kill it for me. O1 is almost as good with a significantly higher limit.

[-]

Additional_Ad_7718@reddit

One challenge I have when trying to answer this question is that OpenAI continues to update their models like 4o without any official announcement. So judging closed source models is a continuous effort.

I would say experiment every month or two and reevaluate. Right now the latest 4o is mind-blowing for the things I use it for, it is the best for me.

[-]

gabe_dos_santos@reddit

I would not use the word "best" I'd say it is less worse. But I think Claude is still superior, 8 months later. I do not know if it is a good or bad thing, I think we are reaching a plateau where we need more data. Or we will never reach the point where anyone can code. Anyone can make simple applications but when bugs start to appear we have to turn to our knowledge.

[-]

Ylsid@reddit

It's certainly the best price performance (but I ain't paying), as much as I want to love Qwen the huggingface chat UI is awful and Claude has better domain knowledge

[-]

_stevencasteel_@reddit

Claude 3.5 Sonnet has been generally the best at everything since it came out in June.

It built my website for me. I purposely kept the site bare bones for now to not exceed the context length in the free tier.

[-]

Cold-Cake9495@reddit

Deepseek 2.5 is great for coding