Working Memo · AI Governance
Most AI governance frameworks converge on a similar model: providers self-assess, sometimes supported by pre-deployment evaluations from an external party. This memo argues that pre-deployment evaluation is not sufficient — the more consequential gap is at the output layer, where a model's response becomes an action or decision that affects someone. Drawing on two years of building and deploying a third-party auditing system for a values-sensitive domain, I argue for a specific, narrow expansion of what independent auditors should be able to observe and, in some cases, block.
Current AI governance is dominated by what I'll call the evaluation paradigm: run a model against a benchmark or red-team suite before deployment, get a score, decide whether to ship. This is true whether the evaluator is the lab itself, a government body, or an independent auditor under something like the EU AI Act's conformity assessment regime.
Evaluation-only oversight has a structural weakness that's well understood in security engineering but underdiscussed in AI governance: a model that passes evaluation on Monday is not the same model in production on Tuesday. Fine-tuning, prompt injection, adversarial drift, RAG pipeline changes, and simple distribution shift in real user queries all mean the evaluated artifact and the deployed behavior diverge over time. This is why financial auditing, food safety, and industrial safety regimes all pair pre-approval with continuous, in-production monitoring — and why AI governance, still largely stuck at the pre-approval stage, is under-built by comparison.
The EU AI Act gestures at this with its post-market monitoring requirements (Articles 9, 17, and related provisions), but the mechanics of what a monitor actually observes, and what authority it has when it observes a violation, remain vague in practice. Most "monitoring" solutions in the market today are statistical drift detectors — they tell you that something changed, not whether the output itself was admissible. That distinction matters more than it might first appear.
There are two separable questions that current governance tooling collapses into one:
Drift detection is necessary but domain-agnostic — a Population Stability Index shift tells you the input or output distribution moved, not whether the new distribution is permissible. Admissibility is domain-specific and requires an explicit, checkable rule set: this output violates a regulatory constraint, a safety constraint, or — in the domain I work in — a cultural or religious constraint that a purely statistical system has no way to encode.
For the last two years I've built and operated a production system — MaqasidAI/MACI — that performs exactly this kind of output-level admissibility checking, in a domain chosen specifically because it has hard, well-documented rules that most general-purpose AI systems get wrong: Islamic financial and religious content. The domain was useful precisely because it forces the tooling to be concrete rather than aspirational. "Don't be harmful" is not implementable; "flag any output attributing a specific ruling to a scholar or school of jurisprudence that did not issue it" is.
The system works in five layers: a fine-tuned multilingual classifier flags candidate violations across eight categories (fabricated religious rulings, misattributed scholarly opinions, financial structures resembling prohibited interest-bearing instruments, and others); a retrieval layer checks flagged outputs against an indexed corpus of verified source rulings; a madhab-aware (jurisprudential-school-aware) policy layer resolves cases where the "correct" answer depends on which tradition is asking; and every decision produces a cryptographically hashed, replayable audit receipt — so a regulator, a client, or an internal reviewer can reconstruct exactly why an output was allowed or blocked, after the fact.
This maps onto a distinction I think AI governance needs more of: the difference between a model explaining itself and a model being governed by something outside itself. Explainability tools (SHAP, LIME, counterfactual explanations) answer "why did the model do this" — valuable for debugging and for building trust, but they don't answer "was the model allowed to do this," and they don't stop it from happening again. An admissibility layer sits at the boundary between model output and real-world consequence, and it can say no.
I want to be careful not to overclaim from a single deployed system in a narrow vertical. But the architecture generalizes in a fairly mechanical way, and I think that's the more important claim than any specific result from my own system:
If the argument above is right, there are a few concrete implications worth pursuing further:
I've spent the last two years building the technical side of this argument in one domain because I think domain-specific, deployed, auditable systems are more useful evidence for AI governance debates than general position papers — including this one. But the technical work alone doesn't answer the harder governance question of where auditing authority should sit, how it should be standardized across domains, or how regulators should think about the tradeoff between auditor independence and provider cooperation. That's the research I want to do next, and it's why a governance-focused research position, rather than a purely technical one, is the right next step for this work.