AI CoE · Dedicated AI Organization
AI Center of Excellence
What is an AI CoE?
- A team that helps the business use AI, not one that builds it
- Five areas: standards, tools, training, project support, and measurement
- Becoming just an approval desk drives more shadow AI
When people hear about creating a dedicated AI organization, they tend to picture gathering developers to build AI services in-house. But what most companies need first is not a team that builds, but a team that clears the way for the business to use AI. The CoE’s first job is to sort out the situation where each department buys its own tools, asks the security team the same questions again and again, and runs overlapping projects. A CoE’s results show up not in its own deliverables but in how well business departments use AI.
Its work falls into five areas: managing usage guidelines and security standards; selecting, contracting, and allocating tools and accounts; running the training roadmap and the AI Champion program; identifying department projects and supporting PoCs; and measuring impact and reporting to executives. Teams often start with three or four dedicated members plus part-time members. Staffing should not be IT alone: a mix of people from business units, training and HR, and security and legal keeps the standards realistic. Because tool features and pricing change often, the team also reviews contract terms regularly.
You choose from three models: a centralized model where the center holds both standards and execution; a decentralized model where each department acts on its own; and a hub-and-spoke model, where the center handles standards and support while department champions handle execution. The usual path is to set standards with a centralized model early on, then move to hub-and-spoke as usage spreads. Whichever model you pick, the reporting line should reach an executive who can set budget and priorities.
The most common failure in practice is the CoE hardening into an approval desk. When every tool and project must go through CoE review and answers take weeks, the business stops waiting and works from personal accounts. The organization that meant to control ends up feeding shadow AI. Conversely, a CoE created in name only, with no authority or budget, stays a team that publishes newsletters. From the start, put in writing what the CoE decides, what departments handle on their own, and the deadline for answering inquiries.
John KimInstructor John Kim, author of B2B Marketing Design and CEO of Nanum Consulting
Seolyeong ParkGenerative AI and AX training specialist with 17 years of hands-on experience and 200+ lectures, helping non-technical staff put AI to work right away| Category | Centralized | Hub-and-spoke | Decentralized |
|---|---|---|---|
| Setting standards | Central team | Central team | By department |
| Project execution | Central team | Departments (Champions) | By department |
| Strengths | Standards align quickly | Balances consistency with speed on the ground | Moves fast on the front line |
| Watch-outs | Prone to bottlenecks | Needs a champion network to sustain it | Duplicate spending and uneven security |
A center of excellence is an organizational model originally set up to spread standards and best practices for a specific technology or method across a company. The term has no single founder; it took hold in management and IT practice, and as AI adoption has grown, the same model is now applied to AI.
Consider a hypothetical case. Company G, an auto parts maker, has 1,200 employees. Five business units had each contracted for four AI tools on their own, the information security team received over 70 identical usage inquiries a month, and three similar PoCs were running without knowing about each other.
- The head of Management Support built an AI CoE with 4 dedicated and 6 part-time members. The 4 dedicated members came one each from production engineering, HR and training, IT, and information security, so no single viewpoint would dominate.
- In its first month, the CoE cut four tools down to two and issued a one-page usage guideline. Bundling contracts lowered the per-account price, and annual account costs fell 18%.
- When the head of procurement asked, “Do we have to ask permission every time?”, the CoE leader answered that use within the guidelines is up to each department, and only projects involving customer data are subject to review.
- They also set a rule to give a first response within 24 hours to every review request. The three duplicate PoCs were merged into one and run jointly by two business units.
- One year later, usage inquiries to the security team fell from 70 to 15 per month. With 40 champions across the business units handling execution, active projects grew from 6 to 31.