Meet Once, Not Twice: Think Your Data and AI Organization Together

Most organizations start with data and AI as two projects. The data initiative comes first, with a Data Mesh, a central platform, governance standards, an enabling team and a handful of decentralized domain teams. Some time later the AI initiative is added, usually with its own budget, its own team and its own operating model. So you end up with two initiatives, two roadmaps and two org charts.

What costs the most in this setup is that both initiatives have to talk to the same cross-cutting teams. The data initiative meets Security, Legal & Compliance, Data Governance, Enterprise Architecture and Portfolio Management to find out which data may be collected, how it has to be classified, where it may be stored, and how the whole effort ranks against everything else the company could fund. Then the AI initiative meets the very same teams to find out which data may be processed with a model, where inference may run, which jurisdiction applies, and how its effort ranks. The same people answer more or less the same questions twice.

You probably know the result: duplicated meetings compete for the same few reviewers, standards drift apart as they have been decided in two different rooms, and time-to-market grows, as every cross-cutting concern is negotiated twice. Yet, for any given use case the data flows and the AI on top of them are steps in one value stream. If you split them into two governance tracks, you add overhead and, in addition, separate decisions which belong together. This is what the two setups look like:

Let’s look at why the two can share one organization, and how far I would go with this.

Why Data and AI Can Share One Organization

That both initiatives meet the same teams is no coincidence. The two disciplines have a very similar structure, as they developed in a similar way.

Data organizations have been moving back and forth between centralized and decentralized approaches for about two decades. Analysis started inside the operational systems and then moved into a central Data Warehouse, to take load off the transactional databases and to have everything in one place. The central approach was good for consistency, but it had a problem with knowledge, as the central team never understood the data of a domain as well as the domain itself did. So things moved back towards the domains and, with Data Mesh, ended up as a structured hybrid: domains own their data products, a central platform team offers self-serve infrastructure, and a federated governance forum defines the rules everyone shares.

AI operating models went the same way, only faster. A central ML/AI Center of Excellence played the role of the Data Warehouse: a few specialists, separated from the domains, built models which never fully contained the knowledge of the domains. With hub-and-spoke these specialists were then embedded into the business. And when generative AI became something the business asked for, instead of something IT offered, the central hub stopped being a gatekeeper and turned into a platform and enablement function. The central data team did the same when it became a platform team.

So both fields ended up with the same answer independently of each other: execution moves to the domains, the guardrails stay central as platform and policy, and a federated forum balances the two. This is the reason why one organization can carry both.

Yet, having the same structure doesn’t mean you should merge everything. The rule I propose in this article is rather narrow: thinking together is mandatory, while implementing together is a choice you make per use case. You think the data organization and the AI operating model through as one, so that you find the synergies. The biggest synergy, which you can have from day one no matter how mature your organization is, is the shared table. (By table I mean the wooden one you sit around, and not the one with rows and columns.)

The Integrated Model

In the integrated model one decentralized organization carries both data and AI. It consists of four parts:

• Domain teams, which are your existing business domains, own their data products as well as their AI use cases. In hub-and-spoke terms they are the spokes.
• A platform team runs the shared infrastructure, which means the data platform and the AI platform side by side.
• An enabling team lends data and AI specialists to the domains which need them, and tries to make itself unnecessary over time.
• A federated governance forum is the place where domains, platform, enabling and the cross-cutting teams agree on the rules together.

The AI hub-and-spoke model can be mapped to these four parts almost one-to-one, which means you don’t need to build a second organization. The four-layer structure deserves an article of its own. For now it is enough to see that one shared table serves both. The integrated model looks like this:

One Table

Let’s get back to the two calendars from the beginning. In the integrated organization there’s one governance cadence instead of two. Security, Legal & Compliance, Data Governance, Enterprise Architecture and Portfolio Management sit at the same table as the domain, platform and enabling teams, and the end-to-end question is asked once: which data may I collect, which of it may I process with a model, and where does it have to stay?

This is a data governance question and an AI governance question at the same time. If one body answers it once, the answer is consistent. If it’s answered in two rooms, chances are high that the answers contradict each other, and someone spends the next quarter reconciling them.

Let’s say a data team asks Compliance whether the way it plans to handle certain data within a use case complies with the GDPR, and Compliance gives its okay. Then an AI team adds a step to the same use case, in which the data is classified with an LLM. The LLM they use runs in the US, and Compliance doesn’t allow this, as personal data must not flow to companies outside Europe. So the end-to-end use case doesn’t work, although the data team had already received an okay for its part. If all teams had discussed the use case in one meeting, the options would have been clear much earlier.

I see the following benefits:

• Decisions are taken once and stay consistent, so there is no AI policy which contradicts the data classification standard.
• The cross-cutting teams see the whole value stream instead of half of it, so their review is based on what the use case does end to end.
• Portfolio management can rank a use case which needs data and AI as one effort against everything else, instead of adding up two halves which are hard to compare.
• Reviewers who are hard to get are booked once. A good data protection lawyer or security architect is a bottleneck, and you don’t want to book them twice for the same question.
• Time-to-market shrinks, as in most use cases the approvals take longer than the engineering.

For all of this the data people and the AI people don’t have to become one team, as long as they share one table.

For some organizations one seat at this table is much more important than the others: when the usage data itself can endanger the people it describes, security cannot be a review step anymore and needs to be involved in the design. I’ve written about this case, and what it does to an architecture, in Security by Design for Analytics Platforms.

Objections

The integration doesn’t come for free, and there are objections you should take seriously. An integrated model is harder to explain than two separate initiatives. Data and AI often move at a different pace: building a solid data foundation is a governance marathon, while an AI win is often a quick sprint, and forcing both into the same rhythm can frustrate both sides. In addition the two disciplines come with different cultures, skills and expectations regarding speed, so putting them in one room can create friction instead of removing it.

All of these are valid arguments against implementing everything together. I do not think they are arguments against thinking together or against the shared table, though. So you keep one governance forum and one portfolio view, and decide per use case whether a piece of data and AI work is built by one team or by two. Where the rhythms don’t fit, you separate the delivery. Where they fit, you use the synergy. The table stays shared in both cases.

Check Regularly

As this is a balance which changes over time, you should review it regularly. Every few months, ask the same three questions:

• Where does thinking together clearly pay off, and should we integrate more tightly?
• Where does it create friction, and should some delivery run on separate tracks?
• Where have the preconditions changed, e.g. because of a new AI initiative or a domain which has matured, so that a separation which made sense yesterday isn’t needed anymore?

These are questions for the governance forum, too. The same table which answers the use case questions also decides how integrated you want to be.

Conclusion

There is nothing wrong with choosing Data Mesh for your data organization and hub-and-spoke for AI. It only becomes expensive when you run them as two organizations which both meet the rest of the company separately. Both have the same roots and a similar structure, both depend on the same cross-cutting teams, and both serve the same end-to-end value streams.

Therefore I would give them one organization and, most importantly, one table, so that everyone has to answer the important questions only once.