AI knowledge base: a consultancy case study

Mid-2025. A senior consultancy of about thirty people, several decades of expertise in demanding regulated domains. All the firm's value lives where value lives in any senior consultancy: in the heads of the partners and a few seniors. Producing one RFP response: several dozen days. A recurring frustration: seniors spend their time fixing the juniors' formatting, and the juniors never learn the trade. A first AI mission, handed to a generalist provider, had delivered solutions that were too basic. That is the situation Catalia walked into.

The friction every senior consultancy lives with

The most common story in senior consulting firms (law, actuarial audit, financial advisory, regulatory consulting, M&A, premium accounting) is the same everywhere. And it is rarely told out loud.

A partner in a senior consultancy spends several days orchestrating one RFP response of 200 to 300 pages. The partner does not write all of it; they delegate to juniors and seniors. But they spend most of their time correcting: the formatting, the firm's editorial rules, the quality of the substance.

Instead of passing trade knowledge down to the juniors, the partner does intensive proofreading. The juniors, for their part, are frustrated: they make no real progress on substance, because substance is not what is asked of them. What is asked is that they got the container right.

The structural consequence: the firm's tacit knowledge, its real value accumulated over several decades, stays locked in a few heads. It does not get passed on. It does not get industrialized. It does not get capitalized.

That is the situation the client was in when they called Catalia in mid-2025, after a disappointing first AI experience with a provider who lacked command of the subject and had delivered tools too basic to produce any real transformation.

Why Catalia, after a first external AI mission had failed

The firm's founding partner named the cause of the first failure plainly: they had trusted a provider who had command of the surface of generative AI, not the depth. The deliverables produced were assemblies of Copilot and templates. No measurable benefit, no change in practice.

For the second mission, the firm was looking for three things at once. Serious RAG technical depth, because in demanding regulated contexts precision and sourcing are not optional. A real understanding of the senior consulting trade, where value is not measured in raw productivity but in passing tacit knowledge down. And a pedagogy of AI critical thinking, to teach users to question the tool correctly, to understand its limits, to know when to dig and when a piece of information comes in one shot.

Catalia holds all three requirements at once. That is what justified the founding partner's decision.

Mid-2025: short audit and framing of the target architecture

The AI audit lasted one week, run by a small team. Three key acts:

1. Mapping the historical RFPs available to work from: typology, length, the firm's editorial constraints, the recurring senior/junior friction points.
2. Identifying the firm's strict editorial rules: the in-house formalism that defines its visual and structural signature.
3. Spotting internal skills: which of the 30 people had the appetite to become a beta tester of the system, and who could carry the transformation internally after rollout.

The audit was led by Matthieu Sabourin, founder of Catalia, in the posture of an expert consultant on strategic framing. A senior partner on the client side ran the mission internally.

Mid-2025 → end 2025: building the tools and rolling them out

Three months of building, across two complementary tools.

Tool 1. Proprietary domain-specific RAG. Indexing a corpus of historical RFP responses, each 200 to 300 pages, on subjects in highly regulated domains. Technical stack used: LangGraph for orchestrating the reasoning chains, LangChain for the connectors and the LLM abstraction layer, and several models tested for generation (notably Anthropic's Claude Opus) to identify the combination that delivers the best trade-off of quality, cost, and sourcing reliability against the client's trade requirements.

The real RAG implementation techniques used are proprietary to Catalia and remain confidential at the client's request.

Tool 2. Automated visual standardization engine. Generating output that conforms to the firm's strict formalism, by combining two complementary logics:

• a strong AI part for the writing and editorial coherence,
• a strong automation part, coding the firm's deterministic rules in VBA (layout, structure, formats).

This separation is deliberate. Generative AI is not the right answer to every problem. Some editorial rules are better handled with classic automation, at an execution cost close to zero. Knowing how to make that call case by case is what Catalia has been doing for more than 10,000 cumulative hours of use on complex RAG applications.

Tool 3 (cross-cutting). AI training in critical thinking. A piece often underestimated in AI missions, central at Catalia. About twenty internal users were trained to:

• frame productive questions (and recognize when a question is badly posed),
• understand that AI does not have the ability to answer 100% correctly every time, and that competent use runs through critical thinking,
• tell the difference between a piece of information that is too hard to get in one shot (and demands several back-and-forths to dig progressively) and one that is available in a single query.

Without this cognitive training, the technical tools are not used to their full potential. With it, users become autonomous: they iterate on the RAG themselves, adjust their prompts, and get the trade quality they are after.

The moment that shows what the mission really produces

The proprietary Catalia methodology that sets this project apart from any standard RAG mission on the market is applied during the validation and integration phase: the combination of LLM-as-Judge and multi-beta-tester trade tests.

LLM-as-Judge is an automated evaluation system where a language model plays the role of evaluator on the quality of the responses produced by another model. But used alone, it introduces its own biases. That is why Catalia pairs it with a panel of internal beta testers at the client, recruited across the trade roles (partners, seniors, juniors), who run real trade tests on the RAG in parallel with the automated evaluation.

This dual validation, automated and human across roles, eliminates what neither one can eliminate alone: the biases specific to a single person in the evaluation, and the biases specific to a single model in the scoring. The system gains industrial robustness, not pure technical elegance.

That is what sets this mission apart from a project delivered by a lone RAG freelancer or a standard SaaS vendor. And it is what Catalia keeps capitalizing on in its Scientific Lab.

The Catalia team on the mission

Three people took turns on the mission, in line with the Catalia model of mobilizing by expertise:

• Matthieu Sabourin, founder of Catalia, on strategic steering and guidance for the firm's leadership, in the posture of an expert consultant on the target architecture and change management.
• Sami Mhidia, CTO of WeAreCasus, a startup that indexes more than two million tax and legislative sources for professional contexts where error is not tolerated. Sami brings a rare expertise on RAG architectures in demanding regulatory environments: rigor of sourcing, reliability against huge data volumes, legal precision. That dual hat is rare and decisive for this kind of mission.
• Yassine Doghmi, project lead on the mission, in continuous contact with the client.

Three distinct roles, one coherence. It is the direct application of Catalia's freelance community by vertical model: not a firm with a fixed headcount mobilizing an oversized team by default, but the right expert at the right moment, on the right subject, on a scoped mission.

For the leaders of French senior consultancies

The lesson of this mission transfers directly to every French senior consultancy in high-value regulated professions: business lawyers, premium accounting, M&A advisory, actuarial audit, banking or insurance regulatory consulting, financial investment advisory, intellectual property firms.

They all live with the same friction. The partners' tacit knowledge is their central asset, but it is not accessible without them. That caps growth, weakens succession, frustrates the juniors, and burns out the seniors. The RAG tools on the market, for their part, do not hold the precision, the sourcing, and the trade compliance that these regulated contexts demand. And when an AI mission is handed to a provider who has command of the surface but not the depth, the firm ends up with tools too basic and concludes, wrongly, that AI does not work in their trade.

Catalia holds all three requirements at once: RAG technical depth, a proprietary validation method, and training users in critical thinking. On top of that comes something harder to put into words: a feel for how senior consulting actually works, which helps separate what AI can augment from what has to stay a human call.

Catalia works with several other law firms and players in finance and the legal professions on the same challenges.

What this case says about Catalia

This firm came to Catalia after a first external AI mission had failed. Four months later, its tacit knowledge, locked for several decades in a few heads, had become an asset the whole team can query. Plus about twenty autonomous users able to question the system, produce with it, and iterate with no dependence on a provider.

The lesson is not in the technical stack. LangGraph, LangChain, Claude Opus, and VBA exist and are within reach for many. It is in the rare combination of RAG technical depth, a proprietary validation method, and a pedagogy of AI critical thinking. The combination that knows how to build an industrial RAG asset, validate it without bias, and pass it on to users who become autonomous with it.

That is what Catalia is structuring in its Scientific Lab. It is what, tomorrow, makes this firm able to enrich the system without depending on any provider, Catalia included.

At Catalia, we don't teach you to produce with AI. We teach you to think with it.