AI Supply Chain and Model Provenance Security

Data is the first layer. Datasets arrive from vendors, scrapes, partnerships and internal systems, each with different licensing, consent basis and quality. A dataset without provenance cannot be audited for bias, licence compliance or poisoning, and it cannot be withdrawn if a problem emerges later.

Models are the second layer. Base weights, fine-tunes, adapters and quantised variants are distributed through hubs, vendors and internal registries. The risks are malicious artefacts, unsafe serialisation formats, modified weights and unclear licences. Verification requires hashes, signatures and format review.

Frameworks and libraries are the third layer, and they are the most familiar. Tokenizers, inference engines, training libraries and CUDA dependencies are conventional software supply chain risk, amplified by fast-moving projects with frequent breaking changes and broad installation privileges.

Providers are the fourth layer and the hardest to verify. A hosted model is a dependency you cannot inspect and may not be told about changing. The controls that work here are contractual, behavioural and procedural: terms that define change notice, version pinning where offered, and golden-input testing to detect behaviour shifts.

Integrity verification starts with content hashing. Every artefact should have a recorded digest at the point of acquisition, stored in your inventory, and re-verified at load. This detects modification in transit or in storage and gives an authoritative identifier for incident response.

Origin verification uses signatures and attestations. Where a publisher signs artefacts, verify the signature against a pinned public key and check the signing identity against your approved list. Where no signature exists, record that gap explicitly and compensate with behavioural testing and sandboxing.

Format review is essential and frequently neglected. Python pickle-based formats execute arbitrary code on load. Prefer safe formats such as safetensors, and where a legacy format is unavoidable, load it only in an isolated environment with no credentials and no network access.

Reproducibility closes the loop. Record the exact versions of base model, adapter, tokenizer, library and configuration used to produce each served model, so that a rebuild produces the same artefact or a documented difference. Without this, rollback and forensics are guesswork.

Automate the checks in the pipeline. Verification at load and in CI, with a build failure on mismatch, converts a policy into a control. Manual verification spreadsheets are accurate on the day they are written and wrong a month later.

An ML-BOM is the practical provenance artefact. For each model it records base model and version, adapters, datasets with version and licence, training configuration, evaluation results, known limitations, owner and review status. It should be machine readable and generated from the pipeline rather than written by hand.

Vendor assurance is a separate track for hosted providers. Ask for change notification commitments, version pinning options, data use and retention terms, subprocessor disclosure, incident notification timelines and whether the provider offers an evaluation or red-team summary. Record the answers and review them annually.

Licence compliance deserves explicit attention. Model and dataset licences vary widely on commercial use, redistribution and derivative works, and some carry use restrictions that conflict with regulated industries. Track licences per artefact and gate deployment on an approved list rather than relying on an engineer reading a readme.

Response must be designed in advance. When an upstream component is reported compromised, you need to know instantly which served models depend on it, which customers are affected and how to roll back. That requires the ML-BOM, versioned artefacts and a rehearsal, not a search through chat history.

Finally, monitor behaviour continuously. Provenance establishes what you deployed; behavioural canaries establish what is actually running. Together they catch both supply chain substitution and silent provider updates, which is the only way to know your verification still describes reality.

AI Supply Chain and Model Provenance Security is best defended by understanding exactly how it fails. The attacks below are not theoretical constructs: each has been demonstrated against production systems, and each leaves a detection signal that a mature team can instrument. For every technique, the security question is not only "can it happen" but "what is the cheapest observable evidence that it is happening now".

Malicious model or adapter upload. An attacker publishes a plausible model or adapter to a public hub, hoping it is pulled into a training or serving pipeline where its behaviour benefits them. The observable signal is: Unsigned or unverified artefacts, low-download publishers, behaviour differing from the claimed baseline on canary inputs. The primary countermeasure is: Approved registries, signature verification, behavioural canaries before promotion and sandboxed first execution.

Unsafe deserialisation in model files. A pickle-based model file executes arbitrary code when loaded, giving the attacker code execution in the training or serving environment. The observable signal is: Legacy serialisation formats in the pipeline; unexpected network or filesystem activity during model load. The primary countermeasure is: Prefer safetensors, scan and convert formats, and load untrusted artefacts only in isolated environments without credentials or egress.

Typosquatting and dependency confusion. A package name similar to a popular library, or a public package shadowing an internal one, is installed by a training or inference script. The observable signal is: New package names appearing in lockfiles; installs resolving to public registries instead of internal ones. The primary countermeasure is: Pinned lockfiles, internal registry preference, allowlisted dependencies and install-time policy checks.

Dataset licence and origin violation. Training data with incompatible licensing or unclear consent is used, creating legal exposure and potential removal obligations. The observable signal is: Datasets without recorded licence metadata; provenance gaps in the ML-BOM. The primary countermeasure is: Licence metadata per dataset, an approved licence list and a deployment gate on provenance completeness.

Silent provider version change. A hosted model is updated by the provider, changing behaviour in production without any change in your code or configuration. The observable signal is: Golden-input regression failures; drift in refusal rate, output length or formatting; unexplained quality complaints. The primary countermeasure is: Version pinning where available, canary testing on every run, change notification terms and rapid rollback to a pinned version.

Compromised build or registry credential. An attacker with access to the model registry or CI credentials replaces an artefact with a tampered version that passes routine checks. The observable signal is: Artefact hash mismatch at load; unexpected registry writes; builds by unfamiliar identities. The primary countermeasure is: Immutable registries, signed artefacts, separation of build and publish permissions and audit of registry writes.

Because AI attacks rarely arrive one at a time, threat models must describe chains rather than isolated techniques. A reconnaissance step (enumerating endpoints and model behaviour) enables an initial access attempt (a crafted input or poisoned artefact), which enables privilege escalation through excessive agency, which enables impact such as data exfiltration or a fraudulent transaction. Teams that model the chain, and place a control at each link, degrade an adversary's success probability far more effectively than hardening a single step.

Detection design should assume partial failure. Every control in this course is evaluated on the assumption that some percentage of attempts bypass it, which is why least privilege, approval gates, egress restriction and immutable audit trails appear repeatedly. The goal of an AI security programme is not a system that cannot be attacked; it is a system where a successful attack is bounded, observed, attributed and recoverable.

A defensible ai supply chain and model provenance security posture layers preventive, detective and corrective controls so that no single component carries the entire trust burden. In practice this means securing four planes: the model and artefact plane (weights, adapters, prompts, versions), the data plane (training data, retrieval corpora, logs), the control plane (identity, policy, configuration, secrets) and the action plane (tools, APIs, outbound integrations). Attacks cross planes, so controls must be designed at the crossing points.

Approved model and dataset registries. Artefacts are consumed only from internal or approved external registries with access control and immutable versioning. This control maps to ISO/IEC 27001 A.5.19 · SLSA and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Artefact hashing and signature verification. Every artefact has a recorded digest, verified at load; signatures and signing identity checked against an approved list where available. This control maps to SLSA · ISO/IEC 27001 A.8.30 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Safe serialisation policy. safetensors or equivalent safe formats by default; legacy formats loaded only in isolated environments without credentials or egress. This control maps to OWASP ML Top 10 · ISO/IEC 27001 A.8.28 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

ML-BOM generation. Machine-readable bill of materials per model covering base, adapters, datasets, configurations, evaluations and licences. This control maps to NIST AI RMF MAP 3 · ISO/IEC 42001 A.7 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Dependency pinning and registry policy. Lockfiles, internal registry preference, allowlisted packages and install-time policy that blocks unknown sources. This control maps to ISO/IEC 27001 A.8.28 · SLSA and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Licence and consent gating. Licence metadata per dataset and model with an approved list, gating deployment on provenance completeness. This control maps to GDPR Art.6 · DPDP Act 2023 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Behavioural canaries and version pinning. Golden-input testing on every run plus pinned provider versions where offered, detecting silent upstream change. This control maps to EU AI Act Art.15 · NIST AI RMF MEASURE 2 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Supply chain incident runbook. A rehearsed process to identify dependent models, roll back artefacts and notify affected parties when an upstream component is compromised. This control maps to ISO/IEC 27001 A.5.26 · NIST AI RMF MANAGE 4 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.

Reference architectures should make the secure path the default path. Wrap model access behind an internal gateway that enforces authentication, quota, policy evaluation and logging; make the raw provider endpoint unreachable from application subnets; issue short-lived scoped credentials from a secret manager rather than embedding keys; and separate the orchestrator's identity from the tool's identity so that authorisation can be evaluated per hop. When the platform enforces these properties, individual teams cannot accidentally ship an unprotected integration.

Standards anchor the architecture in language regulators and auditors accept. SLSA, CycloneDX ML-BOM and in-toto attestations, OWASP Machine Learning Security Top 10, EU AI Act and ISO/IEC 42001 provide the vocabulary of governance, risk and assurance. Map each control to at least one standard clause and to one internal policy. Where a standard is silent on an AI-specific issue, record that gap explicitly — auditors increasingly reward documented reasoning over false completeness.

Finally, defence in depth must survive component failure. Define what happens when the guardrail service is unavailable, when the policy engine times out, when the model provider degrades, and when an on-call engineer is the only responder at 03:00. Fail-closed on high-impact actions (payments, deletions, external communications, privilege changes) and fail-open only where the risk of blocking exceeds the risk of allowing, with an explicit record of that decision.

Security claims about ai supply chain and model provenance security decay quickly. Models are retrained, prompts change, connectors are added and providers ship silent updates. Assurance therefore has to be continuous, measured and tied to thresholds that trigger action. A control without a metric is an intention; a metric without a threshold is a dashboard nobody acts on.

Artefacts with verified signatures. Target: >= 95% of production models. Signature coverage is the clearest measure of whether origin is actually established. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

ML-BOM completeness. Target: 100% of production models. Without it, dependency questions during an incident cannot be answered. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Unsafe format count in pipeline. Target: 0 outside sandboxes. Legacy serialisation is the most direct code execution path in ML infrastructure. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Golden-set regression failures. Target: all triaged within 24 hours. The primary signal of a silent provider or artefact change. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Time to identify dependents of a compromised artefact. Target: < 1 hour. Determines how quickly you can scope and contain a supply chain incident. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.

Testing should combine four modes. Static review validates configuration, model format, dependency provenance and policy definitions before deployment. Automated adversarial suites replay a maintained corpus of known attacks on every release to catch regressions. Manual red-team exercises explore reasoning failures, multi-turn escalation and cross-component chains that automation misses. Finally, production monitoring measures real behaviour — refusal rates, extraction attempt volumes, tool-call anomalies and cost per user — because production traffic is the only place where the true attack distribution is visible.

Lab — Verify an artefact end to end. Objective: Implement hash and signature checks at load. Record a digest at acquisition, verify at load, simulate a tampered file and demonstrate the pipeline failing closed with evidence. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.

Lab — Generate an ML-BOM. Objective: Produce a machine-readable provenance record from a pipeline. Emit base model, adapters, datasets, licences and configuration for a small model, then use the record to answer a hypothetical dependency question. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.

Lab — Build a supply chain drill. Objective: Scope the impact of a compromised dependency in under an hour. Choose a component used by two models, use the ML-BOM to identify dependents, roll back one and document the notification and evidence trail. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.

Reporting is part of assurance. A finding that cannot be understood by a risk committee will not be funded, and a control that cannot be explained to an auditor will not be trusted. Write findings in business language first — what an attacker gains, which data or operations are affected, how many users or records, what the detection time would be — and only then in technical detail. Attach reproduction steps, but never attach live credentials, personal data or restricted content to a report; use redacted transcripts.

Implementing ai supply chain and model provenance security in a large organisation is an operating-model problem as much as a technical one. Three roles must be explicit: an accountable owner for the AI system (usually the business product owner), a control owner for each security capability (platform security, identity, data protection, SOC), and an assurance function that independently verifies claims. Where these roles are ambiguous, controls quietly rot after the launch milestone.

Start with an inventory. You cannot secure models, agents or datasets you have not catalogued, and most enterprises discover on their first red-team engagement that shadow AI usage exceeds sanctioned usage. Build an AI system register capturing purpose, data classification, model and provider, tools and integrations, human oversight model, risk tier and control status. Tie registration to procurement and to cloud project creation so new AI workloads cannot appear outside the register.

Define runbooks before incidents, not during them. At minimum: how to revoke a model or agent credential within minutes; how to disable a single tool or MCP server without taking the product down; how to roll back to a previous model and prompt bundle; how to isolate a poisoned index or dataset; and how to notify customers and regulators. Rehearse each runbook at least twice a year — including one exercise without the engineer who wrote it.

Tooling should reduce toil rather than add dashboards. Sigstore / cosign and in-toto, safetensors and format converters, Trivy / Grype / dependency scanners, Model registries (MLflow, cloud registries, OCI artefacts), CycloneDX ML-BOM or in-toto attestations each solve part of the problem, but integration matters more than breadth: policy decisions must reach the gateway, telemetry must reach the SIEM, and findings must reach the backlog engineers actually work from. A single well-integrated pipeline with clear ownership outperforms five disconnected best-of-breed tools.

Governance closes the loop. Report control coverage, open findings by severity, mean time to remediate, adversarial test pass rates and incidents to the AI risk committee quarterly, and to the board annually in business language. Where a residual risk is accepted — for example a model that cannot yet meet a robustness threshold — record the acceptance, the compensating controls, the owner and the expiry date. Time-bound acceptance is the difference between governance and wishful thinking.

Finally, embed security in delivery. Threat model each new AI feature at design time, add adversarial regression tests to the definition of done, require provenance checks in the deployment pipeline, and give engineers a paved road — approved model gateways, SDKs with guardrails built in, and pre-cleared tool integrations — so the secure path is also the fastest path.

Financial services ML platform (Banking). The problem: Nobody could determine which production models depended on a widely used open-source adapter when a vulnerability was reported in its publisher. The response: Generated an ML-BOM per model from the pipeline, linked artefacts to registry versions and rehearsed a supply chain response exercise. The outcome: Dependency scoping time dropped from days to under an hour, and rollback became a routine operation The transferable lesson: The incident response benefit of provenance is often larger than the preventive one For teams working on ai supply chain and model provenance security, the pattern to copy is the sequence — measure the exposure first, fix the highest-leverage control, then prove the improvement with the same measurement.

Enterprise SaaS provider (Software). The problem: A hosted model provider changed a model version silently, degrading extraction quality and refusal behaviour in a customer-facing feature. The response: Added golden-input regression on every run, pinned provider versions where available and negotiated change notification terms. The outcome: Silent changes were detected within a day rather than through customer complaints weeks later The transferable lesson: For hosted models, behavioural verification is the only provenance you control For teams working on ai supply chain and model provenance security, the pattern to copy is the sequence — measure the exposure first, fix the highest-leverage control, then prove the improvement with the same measurement.