Improving development processes with AI

Posted by ajbrun86@reddit | ExperiencedDevs | View on Reddit | 33 comments

The company I work for is very pro AI. As a result, we're expected to embrace it in whatever ways we can. I have no direct influence over product features, but I definitely can direct development processes.

My stance is, I'm happy to work with AI tools, but they must be beneficial - I wouldn't want to introduce something just to check an AI box. We've provided all developers access to GitHub copilot however I'm not yet aware of anything else that would have real impact.

The overall goal would be to increase team efficiency whilst still being able to demonstrate quality of output.

I'm looking for anything else which may be useful. What sort of tools involving AI are out there people are using successfully? Or perhaps can GitHub copilot be used in a way I've not considered...?

[-]

binarycow@reddit

The main limiting factor for me using AI is accuracy.

If I have to spend time checking the AI's work, then why use it at all?

The more complicated the task, the more I have to check it, because the higher liklihood of mistakes.

When I write the code myself, I know it inside and out. Sure, I may forget stuff about it months or years later, but it'll come back to me. I can articulate why I made every decision. If someone else on the team has questions, I can answer them. If an AI writes it, there is no one with first hand knowledge. You become a reviewer.

When I write the code myself, I grow as a developer. I learn, from experience, what works well and what doesn't. If AI writes the code, I don't learn from those experiences.

I typically have a higher standard for my code than others do. And since AI is trained on other people's code, it is going to do things "wrong". An example is C#'s nullable reference types. I use them, always. I am very strict about following it too - I do not bypass it. (The null forgiving operator is only used when I have no other option). But AI that has been trained using code that doesn't use nullable reference types? Well, it's going to give me shit code.

===

The other limiting factor is that the AI doesn't consider the full context (Or, it requires me to put it into the prompt each time). This leads to generation of code that is technically correct (so, accurate, no errors, etc), but a poor choice of techniques, or code that doesn't use the newest features.

So now I have to spend a bunch of time modernizing that code, or end up writing it myself anyway because the AI didn't consider X, Y, and Z.

===

So, if I were to use AI, I would have to be able to tweak every "knob" that exists. I should be able to tell it that I'm using C#, which version of C#, which version of .NET, what my code style rules are, what the general context is, etc. And then it should only consider training material that adheres to those guidelines. If I say I'm using C# 12 on .NET 8, then it should suggest using records, switch expressions, collection expressions, etc. It shouldn't even consider code that was written in C# 5. If I say that I am writing a recursive descent parser, it shouldn't even consider parsers that use lex/yacc.

[-]

cleancodecrew@reddit

Have you looked at AI code chat tools? This is where you get a chat session with a code repo / branch in a web browser. I've had a lot of success with something like this that we used at my workplace for onboarding team members to legacy code bases. It'll let you pin the repo and versions for your contexts, not exactly what you are suggesting, but it does the trick in terms of code exploration.

[-]

binarycow@reddit

Have you looked at AI code chat tools

No.

I have had such bad experiences with any of the AI chat tools that immediately disable it as soon as I see it. If I can't disable it, I discontinue using the product.

As an example of my bad experiences:

I type my search query into a search box
The AI response area takes up half of the real estate
I have to wait, as the AI types it's response, word by word
The AI response is not what I want, at all, or it took so damn long to type or out that I scrolled past it.
The very first search result is what I want, but it was so far down the screen that it was too fucking annoying.

I used to use Edge and Bing on mobile. They introduced AI crap, and I couldn't figure out how to disable it. So now I use Firefox and DuckDuckGo.

When I search for "list of countries by population", this is what I want to see. A list of search results. That is all.

When I use Edge, I get this.

The first result is a summarization of multiple sources. Well, population of a country has different measurement standards (e.g., what do we count as part of the country, etc). If the search result summarizes multiple sources, how do I know that the AI only used sources that have the same criteria? I don't!
The next result is summarizations of questions I didn't ask! Why the fuck do I care? I would have asked those questions if I had that question.
When I end up getting to the actual first search result (rather than a stupid AI response), it takes up so much space because they also give me "see also" and "related links". Did I ask for that? No. So stop showing it.

If I use the "Copilot" search, it's even worse.

First, I have a popup, asking if I want to use a copilot app. No, I don't want to use the copilot app. If I wanted to use it, I would have opened it. So, continue in browser.
Now I get only the summarization. It doesn't even give me the link to the source. It doesn't tell me the measurement standards. It doesn't tell me the "as of" date for those measurements. I just get five results, every single one is an approximation. To get more information, I have to ask.
What's even more terrible, is if I repeat the same query, I get different results. Cool. Now I get ten countries. And it changed the "flavor text" for India from "nipping at China's heels" to "inching closer to China every day". At least the numbers are the same.
If I search two more times, I get 15 results. So, it pays attention, and thinks that if I keep searching, I want to see more countries. Why can't it just give me a fucking link? That is the standard behavior of any search engine. I shouldn't have to ask it to provide a link. I shouldn't have to ask it to provide sources to confirm what it's saying.
If I search two more times, I get an even worse response. Now they stopped giving me any information at all, and now just tells me that countries have different population density. Uhhh yeah. Duh.

With experiences like that, for a simple search query, why the fuck would I even want to try using AI for programming, where my livelihood depends on me doing my job correctly? Fuck that.

[-]

cleancodecrew@reddit

u/ajbrun86 IDE based copilots are typically considered dumb from a Sr. Dev perspective. Though I find Github copilot useful, we have also tried Codeium.ai , Cursor.com and TuringMind.ai

We didn't notice significant time saving with IDE based copilots - the time savings are negated by the bugs introduced by the copilots. Overall my junior devs love them, our Senior devs hate em.

The area where we gained the most advantage and I believe had real success with AI is in the complex but necessary process of onboarding developers to new projects. We tried TuringMind.ai and were able to use it to onboard developers, by letting them interact with the code repositories. I was most impressed by its ability to answer very complex questions, something that would have required a senior to answer during the onboarding process.

As always, the measure for success with using AI for productivity gains or time savings - copilots haven't been very popular among experienced folks on my team, but we have seen progress in other areas such as app sec and onboarding.

[-]

ramenAtMidnight@reddit

Couple good things I’ve heard from other people: - Use copilot to explain a class or a component on what they do. Great for first time maintainers - Generate documents, including API endpoints, specs etc. - Mass refactoring. Requires solid test suite of course. - File/data manipulation. Good for complicated stuff that vim couldn’t do and too much of a hassle to deal using bash or python

Personally I don’t get to code much lately to attest for those points, but they’re from good engineers of all levels. Keep in mind we also have an epidemic of bad code generated by junior devs, so take this with a grain of salt.

[-]

Resident-Trouble-574@reddit

Mass refactoring. Requires solid test suite of course.

OMG please, don't do that...

[-]

ramenAtMidnight@reddit

Hah, got a bad experience already? Care to share the story? Conceptually imo it seems fine. Works exactly like a vim macro and from what I heard, takes a bit less time.

[-]

Resident-Trouble-574@reddit

A vim macro is deterministic. AI is not.

Also, if you use AI to same the little time that will save you compared to traditional automatic refactoring tools, I seriously doubt that you've taken the time to write good tests.

[-]

ramenAtMidnight@reddit

A vim macro is deterministic. AI is not.

This seems a good premise but would be nicer to expand a bit, or give an example. Tbh I was a bit disappointed when the bulk of your response is some judgement that has nothing to do with the argument. I like to think we’re in the same boat here, finding the best use of a tool, detached from hype, and also, mindless aversion.

[-]

bart007345@reddit

As someone who is currently giving training on copilot let me tell you in can absolutely deliver development gains if you know what it can do beyond autocomplete.

[-]

ajbrun86@reddit (OP)

Just to confirm - we are using it and I'm certainly finding it useful. My goal here is to either:

Introduce something new to the team Show the team how to improve their use of copilot

[-]

bart007345@reddit

DM me if you're company is interested in getting some training.

[-]

Incorrect_ASSertion@reddit

I'm part of a team that evaluates usefullness of copilot within my company and I agree, it is sometimes mad useful. Most of the times it isn't or just plainly provides wrony answers. Automcomplete is best disabled tho.

[-]

Eric848448@reddit

AI is autocomplete with a superiority complex.

Change my view.

[-]

bart007345@reddit

Give copilot two inputs, one is the user story and the other is the code skill guidelines.

Both are in markdown format. All copilot to write the code.

[-]

sexytortuga@reddit

Check out Atlassian Rovo if y’all are a JIRA/Confluence shop

[-]

Vitrio85@reddit

Copilot is a double edged sword.

Senior developers can do their implementation faster.

Juniors will implement Copilots solution without thinking. Or take whatever it says instead of reading documentation.

I encourage juniors not to use IA.

[-]

qlut@reddit

You should check out AI tools that generate front-end code from Figma designs or mockups. Could be a huge timesaver if your team does a lot of UI work!

[-]

RastaBambi@reddit

You can't just say "Here's AI. Find a use case for it."

If you want to improve your company's efficiency, I'd suggest you first establish a baseline for metrics like:

customer satisfaction scores
amount of bugs and "turnaround time"
amount of customer service calls
code quality
test coverage
team velocity
etc.

Basically track whatever is meaningful to your business. I'm not going to tell you which metric is meaningful because that depends on your goals.

So let's say you have measured stuff for three months without intervention. Congratulations. You now have a baseline.

From there you identify your pain points and bottlenecks, then discuss the bottlenecks and find targeted solutions for the areas that you've identified and want to improve.

Can AI help improve certain areas like turnaround time? Maybe. But what if it also increases the amount of bugs in production? Then your apparent productivity gains will be canceled out by a decrease in quality, which drives up the amount of customer service calls and you're going back to the drawing board.

[-]

Greenawayer@reddit

Maybe stop hounding Devs to use these tools...?

If they are useful then Devs will use them. However if Management forces a tool on them it will be a waste of money.

Listen to your Developers about Development tools they need, not the Management.

[-]

gowithflow192@reddit

If they can increase output then I can see companies making it standard and raising expectations. Resisting AI would make you look like a Luddite.

[-]

Greenawayer@reddit

If they can increase output then I can see companies making it standard and raising expectations.

The problem is Management usually get fed bullshit expectations by snake-oil "AI" salespeople. Devs are then expected to match this pack of lies.

Resisting AI would make you look like a Luddite.

Lol. Pointing out something is not the magic bullet is made out to be is "being a luddite".

I keep forgetting. Actual skills and knowledge don't count for much these days. It's how well you agree with what other people have said.

[-]

gowithflow192@reddit

So tell me, you don't think AI can improve your speed of delivery (while keeping all other factors the same)?

If you believe it can, then little wonder management wants to force the tool on you.

If you believe it can't, you must be deluded.

[-]

lampshadish2@reddit

The amount of time it saves me from typing matches the amount of time I have to spend verifying the code it generated is correct and appropriate.

[-]

Greenawayer@reddit

Can using StackOverflow improve speed of delivery...?

Can a decent IDE improve speed of delivery...?

Etc.

These "AI" tools are useful but limited.

[-]

Ok-Yogurt2360@reddit

We should make a blood sacrifice to the omnisiah before using AI.

If you can't see how this will make you a better developer you must be deluded.

[-]

iamnewtopcgaming@reddit

Here’s one of the biggest gains that I haven’t seen talked about when I use Cursor (w Claude) with full access to the codebase. It can point you to the files / locations of where to make changes. Even if it gets the exact changes wrong (it does get close enough fairly often to just make some edits), you can get up to speed on what you’re trying to do faster. I would easily switch to Cursor at work if it was approved.

[-]

ajbrun86@reddit (OP)

Actual Cursor has been something I've been meaning to look into. So long as a private GPT (or similar) key can be used for it, it should be viable.

As with all of this, security compliance must be bet which is often a barrier to trying new AI technologies.

[-]

iamnewtopcgaming@reddit

I know it has api key support for a few models, it might work for you. I’ve actually been thinking about how my next job better let me use it or they’re not moving fast enough.

[-]

b1e@reddit

Companies like this are doomed. It’s focusing on the wrong thing.

Some people find copilot useful as a fancy autocomplete. Some find the chat useful for generating some basic code.

You can offer people the tools but forcing them to use it is silly.

[-]

ajbrun86@reddit (OP)

Yes - that's not my purpose.

My goal is to either offer a new tool that isn't yet used or known about and demonstrate it's advantages or show people The possibilities of copilot that they may not yet be benefiting from.

There is no mechanism to force people to use anything. If a developer can legitimately be more productive without these tools, I don't think anyone would think it right to enforce usage. It's personal preference but also about making sure the tools and knowledge are out there.

[-]

softgripper@reddit

FWIW, I turned off copilot, and now avoid chatGPT.

Those tools, in addition to giving you bullshit wrong answers the vast majority of the time, slowly take away your ability to problem solve on your own. I can't help but feel like they are detrimental quality, at least in their current form.

[-]

Greenawayer@reddit

FWIW, I turned off copilot, and avoid ChatGPT. I used both for about a year (give or take a few months).

Yep. Both are useful only in a limited number of situations. If you're a decent coder then they are inconvenience and waste of time.