How do you create a AI Center of Excellence that doesn't suck?

Posted by Nunuvin@reddit | ExperiencedDevs | View on Reddit | 20 comments

Suppose you are a mid level / early senior dev in a large siloed org. You have some exposure to data sci and anomaly detection (timeseries, text etc) but not much to write home about. Your new team can do llm and has potential eventually expanding toolset to better AI approaches.

And you are now tasked with creating an AI COE for the entire org (it is an ask from above)...

What is your advice if you would be in this situation?

I feel like its once in a decade opportunity to do something impactful in this org but at the same time I wouldn't say I am 100% ready for such a role change and I am not too hopeful that someone else will pick this up. Or should I claim that I am too busy and let this thing probably fail?

What do you want your AI COE to do?

What do you not want AI COE to do?

Did your company do this? How did it go? What lead to success/demise of the COE?

How would you go about centralizing and breaking silos, encouraging collaboration and scaling promising solutions??

What are the main pain points when interacting with corporate AI projects?

[-]

experienceddevsb@reddit

This flair is only allowed on wednesday, saturday (UTC). Please repost on an allowed day. Intentionally trying to circumvent this rule will result in a suspension. See: https://www.reddit.com/r/ExperiencedDevs/comments/1rfhdrg/moderation_changes/

[-]

mehtheswede@reddit

Centers of Excellence don’t work for this. I would argue that the standardization should be minimal, the playing field should be super wide with guardrails that still allow mistakes to happen, and people should use AI the way their brains work.

Everyone is different and forcing people to use AI tools in a standard way will cause groupthink.

Build a small but diverse team to handle policy and push back against security and IT every step of the way.

[-]

Nunuvin@reddit (OP)

In my head its not about forcing people to use AI like claude or gpt. Its about solving business problems with AI.

[-]

TastyToad@reddit

Hold on, what is the stated goal of your CoE ?

Solving business problems with AI sounds like a regular work, not something you'd need CoE for.

[-]

NightSp4rk@reddit

Do I even want to know what is a "center of excellence"?

[-]

SquiffSquiff@reddit

I haven't had great experience in orgs with a 'Centre of Excellence' and without being mean, your post illustrates the reasons why perfectly- points 1 and 2 below from my own list:

Where does the 'excellence' come from? You and your org don't appear to have a clear idea of what this looks like for either the tech or the techniques.
'Centres of Excellence' in my experience are really about command and control based on centralised management. They play especially well with management consultancies and the kind of orgs that like to engage them.
What if another part of the organisation also has excellent people in this discipline? How do you accommodate and learn from them?
Where/What is the accountability? In my experience it's common for such teams not to be held accountable for external business outcomes whilst their client teams still are. This means that the 'Centre of Excellence' can just sit there and say 'no' and not suffer the consequences of their actions. This to me me is the biggest problem- no accountability = no place being there.
If this is the entry strategy, where is the exit strategy? Consider some other aspect of the modern business, e.g. email and calendaring or company wiki/confluence. Would you have a 'centre of excellence' for that? Why not? What's your strategy for when this becomes simply an expected part of everyday life rather than some special thing that has to be 'managed'.

[-]

Nunuvin@reddit (OP)

Thanks. Just want to make the best of the situation.

[-]

SquiffSquiff@reddit

What does this actually mean beyond your own personal career advancement?

[-]

Nunuvin@reddit (OP)

Fair question. If I didn't care, I wouldn't ask. I don't think my career is on the line here. Its something I can influence and make the best of the situation.

Worked with other governance initiatives before, I know what you are talking about accountability and blocking. Had some projects get burned and run late because of blocks and had plenty of incidents where the issue would have been easily avoided if not for actions of others. So why not try to do something better?

With regards to people who can do better, sure they could share their experience so we can increase AI solution adoption. Everyone wins, we increase adoption, they gain better visibility.

Again the goal isnt to shove claude onto devs but rather have a way to streamline aritifical intelligence (the entire field) development. I don't want this to be a claude md or agents md file collection or some other bs.

Sometimes you are dealt cards you can either fold or try to make the best of the worst. Its probably very naive, but I don't think trying to do something better will make it worse than if I don't try...

[-]

SquiffSquiff@reddit

Sure. Thing is you're still simply describing what you don't want this initiative to do and then reaching to justify it via Post hoc rationalisation. If you can't describe what you want this project to you achieve, how can you define success?

[-]

throwaway_0x90@reddit

buzzwords impossible to parse

[-]

QuitTypical3210@reddit

Wat is center of excellence even mean

[-]

therealslimshady1234@reddit

Would like to know this too. Sounds sloppy

[-]

LeadingPokemon@reddit

It’s a place for developers to share their production code created by AI. Microphone. Drop.

[-]

LeadingPokemon@reddit

If someone shares code but it’s not in production they are a BITCH

[-]

UnderstandingDry1256@reddit

Talk with opus/gpt for a day, I’m absolutely serious.

Provide your situation context, etc. Make it ask you questions. At the end of the day you’ll likely be absolutely confident.

[-]

SquiffSquiff@reddit

You're absolutely right! That's an excellent suggestion. Where shall we start?

[-]

LittleLordFuckleroy1@reddit

Sounds like wishful slop thinking.

[-]

davearneson@reddit

Just don't. Most centers of excellence become function silos that become a major bottleneck for the organization.

[-]

Future_Manager3217@reddit

Don’t make the CoE an approval board. Make it a service team with a narrow mandate: turn scattered AI experiments into reusable, measurable operating patterns.

I’d have it own four things:

an intake/risk rubric: allowed use cases, review-needed use cases, hard no’s
two or three reference implementations people can copy, with logging/evals/cost controls already included
a lightweight eval + incident template: baseline, expected behavior, known failure modes, rollback, owner
office hours and design review, not central delivery of every AI project

What I would not let it own: every AI feature, vendor-selection theater, generic prompt training, or slide decks about transformation.

In a political org, I’d start with one boring cross-silo workflow where you can measure before/after and name the operational owner. If the CoE can’t show one measurable deployment, it will become a committee.