The core challenges

In compliance assessments, finding a relevant regulation is only where the work starts. The real time sink is judgment: Does this regulation apply to the product in question? How has the company handled similar cases in the past? What conditions must be satisfied, and when should a case be escalated to a human expert?

Meta's difficulties in the compliance space come down to three problem areas.

First, experts know how to make a judgment, but the documentation does not fully explain why.

A lot of tacit knowledge is procedural: what an expert looks at first, which factors carry more weight, when an old conclusion still holds, and when it's time to stop extrapolating and bring in another human. Experts often become so practiced at these steps that they no longer think to explain them, leaving only the final conclusion behind in the historical record.

For example, "limited trial approved" is a conclusion. It does not tell the next person who the trial is for, what data it uses, or whether it can be rolled back if something goes wrong. The following teaching rules make these hidden conditions explicit: A request proceeds to standard trial review only when all three conditions are met: internal testing, simulated data, and a rollback option. Other cases require specialized human review or additional information. These are fictional rules for illustration, not Meta policy.

Changing the number of test users might not change the review path. But swap simulated data for real user information and, even with the same number of people, the request could demand a different kind of scrutiny. This is exactly what expert experience encodes: what is only superficially similar, and what actually changes the judgment.

Writing this experience down is also not the same as summarizing an old document. A summary may still just say "project approved for a trial." Reusable knowledge captures the applicable conditions, exceptions, the reasoning behind them, and open questions. The post describes an offline process that extracts knowledge from documents and then refines it with expert feedback, but it does not release a full extraction algorithm. If an insight never appears in any document and no expert has ever voiced it, the system should not be assumed to know it.

Second, large models have plenty of general knowledge but no idea which standards this particular company has chosen.

An industry might support several valid approaches to the same kind of work. A given company, though, has already settled on one, based on its history, its business direction, and how much risk it's willing to carry.

A general-purpose model can lay out those options, but it doesn't know which choice the company has made. Its recommendations might be perfectly sensible, yet an internal expert will still have to redo the judgment and correct the output.

What's missing is the organization's own position: which interpretation do we adopt for this problem, and what boundaries do we refuse to cross?

Third, RAG can find the right documents, but it can't save you from reconstructing the full chain of reasoning every single time.

RAG, or retrieval-augmented generation, can pull in relevant documents before generating an answer. But the retrieved results might include historical project records, current policies, and exceptions that only apply in very specific situations, all at once.

Without some prior structure, the AI has to redo judgment work on every interaction: which document wins? Which conditions apply here? How do these sources relate to each other?

That consumes time and invites inconsistent conclusions for similar problems. If an expert corrects the system once and that correction only lives in a chat log, never reaching the knowledge base or the workflow, the next analysis can easily walk down the same wrong path.

So Meta saw two capabilities it needed to build: first, spell out in advance the judgment conditions experts use; second, actually feed later corrections back into the system, then verify those updates work.

The core architecture: four pillars

Meta's system is split into four layers, giving each one a distinct responsibility: reasoning standards, execution steps, acceptance criteria, and feedback integration.

The knowledge layer holds the company's adopted judgment standards. The reasoning layer defines how an analysis should be carried out. The evaluation layer checks whether a fix works and whether it causes side effects. And the improvement loop turns expert feedback into proposed changes that still have to be validated.

The four layers are interdependent. Knowledge files need clearly defined boundaries so they can be edited with precision. Analyses need explicit steps so failures can be traced back to their source. Evaluation decides whether a proposed modification passes muster, and the improvement loop feeds the result back into both knowledge and workflow.

Knowledge layer: write down what experts explain every time, once

Meta first processes the original documents offline, distilling organizational judgments into more than 200 structured text files.

Those files fall into four categories:

• Position files: which interpretation the company adopts for a given issue, under what conditions, and with what constraints.
• Taxonomy and vocabulary files: what terms and phrases mean inside the team, to stop different people, or systems, loading the same word with different meanings.
• Routing index: uses the features of the current task to point to applicable rules and analysis workflows, rather than relying only on text similarity.
• Entry-check files: determine whether the current task falls inside a given specialty's scope at all, so the wrong playbook doesn't get applied right out of the gate.

The generalized example above places analysis steps in a directory called recipe and positions plus facts in a directory called knowledge. In the file headers on the right, triggers says when the file should be loaded, depends_on records which sources it relies on, and referenced_by lists which workflows or rules will use it. Those structured fields are written in YAML.

The entry check answers one question first: does this task fall within the scope of this analysis track? Only then does routing decide which rules apply. A piece of text might look very similar to the current problem without actually matching it. Explicit trigger conditions give the system something defensible to point to when it chooses a standard.

Sticking with the trial example, a usable knowledge card could look like this:

Organizational position: Internal testing, simulated data, and a rollback option are all required for standard trial review.
Scope: Feature trial requests.
Exclusions: External access, real user information, or no rollback option.
Insufficient information: Gather the missing information first; do not assume the conditions are met.
Basis: Expert clarification in this teaching example.
Used by: Trial review workflow and supporting-material checklist.

That text is a teaching example. The directories and field values in the original diagram are also marked as generalized. What matters here is the structure: the rule, its boundaries, the underlying rationale, and the intended consumers can each be examined on their own.

Not everything needs to live in a wiki, either. Meta separates material by information density and how often it gets used. Information density comes down to a practical question: how much of a given text directly helps make the current call? Frequently used distilled rules sit in the wiki, while specific product specs and historical detail are pulled in on demand.

One long historical document can serve both purposes at once. The general judgment it teaches gets extracted into the core knowledge, while the raw details stay behind as evidence to fetch when needed. Frequently used does not mean the entire document needs to be resident in context all the time.

Reasoning layer: turn the expert's method into steps

Even with the right knowledge on hand, the AI can still use it in the wrong order, skip a required check, or jump to a conclusion too soon.

So Meta built a second layer of structure it calls Recipes—prescribed analysis flows that say what gets checked first, where each piece of knowledge gets loaded, and what counts as completing the analysis.

Knowledge files say what standard to apply; recipe files say how to apply those standards to get this particular task done. Recipes cite the knowledge files rather than copying the domain facts into their own steps. Knowledge files state a position; they do not prescribe a procedure.

A top-level workflow dispatches to different subworkflows depending on what the task requires, with each subworkflow handling one phase of the analysis. When the organization adopts a new position, the main edits land in the knowledge and routing layers. When the problem is in how the analysis is being performed, the recipe files are the ones that change.

Because execution happens in stages, each step only pulls in the instructions and source material it needs at that moment. Following the teaching case: the entry step first checks whether this is a trial request at all. When consulting standards, it brings in the position file. When verifying conditions, it pulls up the data scope and rollback materials for this specific product.

This gives knowledge changes and workflow changes distinct targets. If a review standard changes, update the position file. If the standard is clear but the assistant misses the data-scope check, change the verification step. References connect the two, reducing the risk of copying one standard into multiple workflows and overlooking a copy when it changes.

Human oversight: stop clearly where an expert needs to step in

Meta's design keeps two mechanisms for human intervention, both woven through the analysis flow.

Checkpoints sit at predetermined points in the analysis. The AI shows its intermediate reasoning, and the expert confirms the direction before the process continues, or corrects it on the spot.

Escalations are for cases that cannot be judged reliably, like when a required piece of information is missing or the evidence supports more than one reasonable interpretation. A human expert then decides what happens next.

Expert involvement does double duty: it steers the current task, and it leaves behind feedback the system can learn from. By watching the intermediate reasoning and the spots where the model gets uncertain, the expert also gets a read on which parts of the process can be trusted to the system as it matures.

The standout mechanism: how the "self-evolving flywheel" spins

The real learning happens in what comes after an expert correction.

A simple "that conclusion is wrong" cannot be dropped straight into the knowledge base as a reliable update. Files depend on each other; changing one rule can shadow or break another. The system has to find the root cause, identify where the fix goes, and then prove the change actually solves the problem.

Doing that dependency mapping, impact analysis, and fix validation by hand can take weeks, according to Meta.

Meta frames this maintenance work as a kind of "compilation": translating natural-language feedback into precise, reviewable file edits. The overall process runs through four stages.

1. Root cause: is knowledge missing, or was it used wrong?

The team's first attempts sorted corrections by how the expert phrased them: new information meant a knowledge gap, a redirected answer meant a workflow problem. That approach failed because one and the same correction can point at very different root causes.

Now the system first pulls the substance of the expert's feedback out of the conversation, then reconstructs what the AI actually had loaded at the time, when it loaded it, and how it used it. It reads the contents of those files too.

Then it asks a pivotal question: given the material the AI had access to at that moment, could it have reached the right answer on its own?

Back to the trial example: if the loaded rule never said "simulated data only," the fix is new knowledge. If that condition was already in the sources but the AI never verified the data type, the fix is in the workflow. If the expert community has not yet settled how real user information should be handled, that is a topic for people to decide first.

Note that this analysis uses the sources actually consulted during the run. Searching the full knowledge base for whether some sentence exists somewhere is not enough to explain a specific failure.

The system also weighs problems by how often they occur and how severe their impact is, which helps decide what gets fixed first.

This step stops every error from being treated as a need to add one more rule.

2. Surgical edits and adversarial review: keep changes small and check their impact

Once the cause is found, the system converts the problem into concrete file edits.

Multiple specialized subagents each check a different concern: whether the change conflicts with an older position, whether any reference relationships are broken, whether content is duplicated, whether the file's budget is being exceeded, and whether tests cover the surrounding cases.

Then a separate review agent, running in an entirely fresh context, inspects the diff. It does not receive the proposer's explanation. Its job is strictly to hunt for contradictions and broken edge cases.

Deterministic checkers run alongside these agents to catch structural problems such as dangling references, identifier collisions, files that exceed a size limit, and dependency cycles.

Independent review checks meaning; automated checks handle format and structure.

3. Two-stage testing: fix this case without breaking others

The first phase is a blind replay of the original case.

The system reruns the scenario that originally triggered the expert correction. The AI agent carrying out the task does not know it is being evaluated. An independent judge, unaware of what was changed, scores the new output against the original expert feedback — a setup meant to reduce bias toward favorable ratings based on knowledge of the intended fix.

The second phase is regression testing.

The system runs existing benchmark cases from the domain to check whether previously sound judgments have degraded. For analytical tasks where multiple reasonable answers exist, evaluation is based on predefined criteria rather than exact wording matches.

The two tests address different concerns: the original-case replay verifies that this particular error is fixed, while the regression test checks for side effects of the change. Failure in either phase sends the work back to the revision step. If regression fails, the system returns the regressed case, the original problem, and the attempted patch together as material for rework.

4. Save the update and grow the test suite: turn one fix into an ongoing check

Once validation passes, the system generates a change request (PR) containing specific diffs and audit records for expert review.

Only after the expert approves does the knowledge or workflow get formally updated. At the same time, the original failure scenario and the verified correct answer are automatically added to the regression test suite.

Each correction thus leaves behind two deliverables:

• Updated knowledge or workflow, available for future tasks.
• A new test case, used to verify that later modifications do not break this confirmed behavior.

The teaching example below shows what this looks like as a concrete change. The old workflow verified only the target audience and rollback plan, missing the data type; after expert correction, the patch should add data verification — not escalate every trial to full special review.

A patch that routes everything through special review might fix this particular false approval, but it also upgrades tasks that legitimately meet routine criteria. Regression cases exist precisely to catch that kind of side effect. What gets saved after approval is the updated workflow and the test; model weights are not changed by the correction, and future tasks draw on the experience by reading those saved files.

Results in practice

Meta reports that after three development iterations over six weeks, the system achieved the following results:

• Single evaluations dropped from days to minutes. The AI handles most of the analysis, freeing experts to focus on ambiguous judgment calls.
• Output usability improved markedly. Experts found results useful in nearly all cases, a clear step up from earlier stages that often required substantial rework.
• Token consumption per round fell by roughly 80%. Tokens are the unit by which the model processes text. The savings came from staged, on-demand loading that cut irrelevant material entering each round.
• Knowledge maintenance accelerated as well. The automated improvement loop produced validated knowledge edits, work that previously required a full engineering development cycle.
• No regressions were observed during the improvement period described. Each fix also expands the test suite, adding more checks for future updates.

These are Meta's reported results for a specific internal domain. The "zero regression" finding applies only to the checks that actually ran, not a guarantee that every future task will be error-free. Other organizations will still need to validate the approach with their own cases.

What it means for enterprise AI adoption

This practice points enterprises in a concrete direction: teaching AI to make expert judgments requires storing the reasoning behind those judgments, the method for applying them, and the acceptance criteria for verifying them.

Documentation alone leaves the AI unsure how to use it; a workflow alone provides no basis for handling organization-specific problems; correction logs without updates or tests let similar mistakes keep recurring.

Meta connects these pieces so every piece of feedback has a clear destination: which knowledge needs supplementing, which step needs changing, how to prove the fix works, and who approves it.

Knowledge and workflows live in text files readable by both humans and AI, so every update can be diffed, versioned, and rolled back if needed. A complex improvement cycle ultimately leaves behind a concrete, reviewable change.

This approach is especially valuable for work that concentrates specialized expertise, requires consistent evaluation across similar cases, generates more volume than experts can handle alone, and demands analytical quality that general-purpose models cannot yet reach. Compliance, financial risk, security review, and engineering standards assessment may all fit that profile — though the specific rules and tests still have to be built by each domain.

Adopting it requires a knowledge base with clear boundaries and linkages, an analytical workflow kept separate from that knowledge, a steadily growing evaluation set, and human checkpoints proportional to domain risk.

The value of an organizational second brain is this: after an expert spends time correcting a mistake once, subsequent tasks can use that experience — and the system keeps checking that it was preserved correctly.