Building a model inventory from scratch

Why inventories rot

The usual story: someone is asked for a list of the models in production. They spend two weeks interviewing teams and produce a spreadsheet. It is accurate on the day it is finished. Six months later three models have been retrained, one has been retired, two new ones shipped, and the person who owned the spreadsheet has changed teams.

The failure is structural, not personal. The inventory was a document maintained by memory, and memory is not a control. Any governance artifact that depends on someone remembering to update it will be out of date exactly when you need it.

The fix is not discipline. It is making the inventory a by-product of shipping, so that the act of deploying a model is what updates it.

The minimum viable field set

Frameworks list dozens of possible fields. Start with these nine. Every one of them answers a question somebody will actually ask you.

Resist adding more at the start. A register with nine populated fields is worth more than one with thirty fields where half are empty. Empty fields teach people the register is optional.

Where the data already lives

Almost none of this needs to be collected by asking people. It exists in systems you already run.

   model registry          deploy manifests          CI pipeline
   (version, artifact)     (what is live, where)     (eval results)
          │                        │                       │
          └────────────┬───────────┴───────────┬───────────┘
                       ▼                       ▼
                 ┌──────────────────────────────────┐
                 │        MODEL INVENTORY           │
                 │  id · purpose · tier · owner     │
                 │  version · data · reviewed       │
                 └──────────────────────────────────┘
                       │                       │
                       ▼                       ▼
                risk register           audit evidence
                (what could go          (what you show
                 wrong, mitigation)      a regulator)
            

Concretely:

• Version and artifact — your model registry, or the tag on the container that serves it.
• What is actually live — deploy manifests. This is the field most often wrong when maintained by hand, because staging and production drift.
• Evaluation results — CI. If evals run in the pipeline, the numbers are already recorded somewhere addressable.
• Owner — whatever maps services to teams. A CODEOWNERS file is usually closer to the truth than any document.

Only two fields genuinely need a human: purpose and risk_tier. Those are judgements. Everything else is a join across systems of record.

The shadow inventory problem

The models you know about are the easy half. Every organisation past a certain size has systems nobody classified as AI: a scoring rule in a stored procedure, a spreadsheet macro that orders a queue, a vendor product with a model inside it that arrived through procurement rather than engineering.

These matter because the regulation follows the decision, not the technology stack. A hand-tuned rules engine deciding credit limits carries the same obligations as a neural network doing the same job.

Two searches find most of them. Grep your deployment configs for calls to external inference endpoints — that surfaces vendor models. Then ask each team one question: what decisions does your service make automatically that used to be made by a person? That phrasing finds the stored procedures, because it does not use the word "model" at all.

Keeping it current without a mandate

You will not get compliance by asking. Three mechanisms that work because they attach to something people already do:

Fail the build on an unregistered model

A pre-commit hook or CI check: if a deploy manifest references a model with no inventory entry, the build fails with a link to the form. This converts the register from a favour into a dependency. It is roughly thirty lines of code and does more than any policy document.

Make staleness visible

A weekly job that lists entries where last_reviewed is older than the tier requires, posted to the owning team's channel. Not escalated — just visible. Most teams fix their own row when they can see it is the stale one.

Review at deploy, not on a calendar

Annual reviews produce a scramble every twelve months. Reviewing when a model changes means the review is small, relevant, and done by someone with the context loaded.

A worked example

A mid-sized fintech, five models. This is the whole register — it fits on a screen, and that is the point.

The fourth row is the useful one. It has no owner, has never been reviewed, and its tier is a question mark. That is not a gap in the register — it is the register doing its job. Somebody built a CV ranker, it probably falls in the high-risk tier, and nobody owns it.

A register that only contains settled rows is a register that is hiding something. Leave the uncertainty visible: an unanswered question you can see beats a confident answer that is wrong.

What to do with the flagged row

The résumé ranker is the useful case to work through, because it is the shape most unowned systems take. Somebody in recruitment asked for it, an engineer built it in a sprint, and it quietly became part of the hiring process without anyone deciding it should be.

The sequence is: find who runs the process it feeds, not who wrote the code. Ownership follows the decision, and the recruiting team owns hiring outcomes whether or not they can read Python. Then answer the tier question properly — CV ranking is named in the high-risk category in most readings, so the honest default is high until someone argues otherwise in writing.

If nobody will own it, that is also an answer. An automated decision affecting people's employment, which no team is willing to put their name against, should be turned off. That conversation is much easier to have when the register makes the situation visible rather than leaving it as folklore.