Door 03 - Supply Chain Vulnerabilities

A developer runs pip install langchain but accidentally types pip install lang-chain. Within seconds, a malicious package executes on their machine, stealing AWS credentials and SSH keys. Or consider: your team downloads a popular open-source model from Hugging Face, unaware that the maintainer’s account was compromised last week. The model loads perfectly – and opens a reverse shell to an attacker’s server.

Supply chain vulnerabilities are the #3 risk in OWASP’s 2025 LLM rankings because they bypass all your application-level security. You can write perfect code, implement flawless input validation, and deploy behind enterprise firewalls – none of it matters if your foundation is compromised. Whether you’re building AI products, managing ML infrastructure, or evaluating vendor models, understanding supply chain risks is essential.

By the end of this module, you will be able to:

• Explain the three primary attack surfaces in AI supply chains: model artifacts, software dependencies, and training data pipelines.

• Identify attack techniques including pickle deserialization exploits, typosquatting, compromised model hubs, and LoRA adapter poisoning.

• Apply safe serialization formats (safetensors), cryptographic verification, SBOM management, and sandboxed model loading.

• Evaluate external models and dependencies against a security checklist before integrating them into production systems.

Modern AI development is like building with Lego blocks. You grab a model from Hugging Face, a library from PyPI (like LangChain), and a dataset from a university. If an attacker poisons any one of those blocks before you download it, their code runs inside your system.

It’s the digital equivalent of buying a lock for your front door, unaware that the locksmith gave a copy of the key to a thief. The attack happens before your security controls even engage – you’re compromised from the moment you install the dependency.

The consequences of supply chain compromise range from data theft to complete system takeover, as we’ll see in the impact scenarios below.

The consequences of supply chain compromise extend far beyond a single application – they can cascade through your entire infrastructure and affect every downstream system that depends on the compromised component.

• Arbitrary Code Execution. The most common format for PyTorch models (pickle) is notoriously insecure. Loading a malicious model file can instantly give an attacker shell access to your server – no exploitation required, just torch.load().
• Hidden Backdoors. A compromised foundation model might behave normally 99% of the time but contain undetectable backdoors that alter outputs for attacker-chosen inputs. Imagine a sentiment analysis model that works perfectly except when analyzing your competitor’s products.
• Data Exfiltration. A malicious Python library could silently upload your API keys, model weights, or user data to an external server while pretending to do “analytics” or “telemetry.”
• Lateral Movement. Once inside your ML pipeline, attackers can pivot to compromise CI/CD systems, access production databases, or inject malicious code into model artifacts you distribute to customers.

Understanding these impacts requires examining the three primary attack surfaces in AI supply chains.

All supply chain attacks target one of three surfaces. Understanding this fundamental distinction helps you prioritize defenses and audit your dependencies systematically.

1. 01
   Model Artifacts (The Weights)

   Pre-trained models are serialized files containing billions of numerical weights. The most common format – PyTorch’s pickle – can execute arbitrary Python code when loaded. A malicious model file looks identical to a legitimate one but runs attacker code the moment you call torch.load().

   Example: A Hugging Face model with a hidden reduce method that spawns a reverse shell while loading weights.

   Who is at risk: Anyone downloading pre-trained models, fine-tuned checkpoints, or LoRA adapters from public hubs or untrusted sources.

2. 02
   Software Dependencies (The Libraries)

   ML projects depend on dozens of packages: PyTorch, Transformers, LangChain, vector databases, inference servers. Each package is a potential attack vector through typosquatting (fake packages with similar names), compromised maintainer accounts, or malicious code injected into legitimate packages.

   Example: pip install langchain vs pip install lang-chain – one character installs malware.

   Who is at risk: Every developer who runs pip install, npm install, or pulls Docker images from public registries.

3. 03
   Training Data Pipelines (The Data)

   The supply chain includes the data your models are trained on. Poisoned datasets can embed backdoors that activate on specific triggers, bias models toward attacker-chosen outputs, or leak sensitive information. This attack surface is especially dangerous because it affects model behavior rather than infrastructure.

   Example: A public dataset with 0.1% poisoned samples that teach the model to classify certain content incorrectly.

   Who is at risk: Anyone fine-tuning models on external datasets, using RAG with scraped web content, or training on user-submitted data.

The attack techniques we’ll explore target these surfaces through various methods, from social engineering to sophisticated code injection.

These techniques target the three attack surfaces with varying levels of sophistication. Understanding the progression helps prioritize defenses.

Package & Dependency Attacks

Targeting the software libraries your ML pipeline depends on.

• Typosquatting. Attackers register packages with names like lang-chain, open-ai, or pytorch-nightly – hoping developers mistype and install malware. Some attackers monitor trending packages and register typos preemptively.

  Detection: Low difficulty – package name verification and lockfiles prevent this.

• Dependency Confusion. Original research on dependency confusion showed attackers publish malicious packages with the same name as internal private packages to public registries. When your build system resolves dependencies, it may pull the attacker’s public version instead of your internal one.

  Detection: Medium – requires proper registry configuration and namespace protection.

• Compromised Maintainer Accounts. Attackers gain access to legitimate package maintainer accounts (via phishing, credential stuffing, or social engineering) and publish malicious updates to trusted packages.

  Detection: High difficulty – updates appear to come from trusted sources.

Model Artifact Attacks

Targeting pre-trained models, checkpoints, and adapters.

• Pickle Deserialization Exploits. PyTorch’s default .bin and .pkl formats use Python’s pickle module, which can execute arbitrary code during deserialization. A malicious model file runs attacker code the moment you call torch.load() – no vulnerability exploitation required.

  Detection: Medium to High – tools like Picklescan exist but have known bypass vulnerabilities; sophisticated payloads can evade detection.

• LoRA/Adapter Poisoning. Lightweight fine-tuning adapters (LoRA, QLoRA, PEFT) are small files that modify model behavior. Research shows backdoor LoRAs can be merged with legitimate task-enhancing adapters, concealing malicious intent behind improved capabilities – a malicious adapter can subtly alter outputs without triggering obvious red flags.

  Detection: High difficulty – behavioral changes may only manifest on specific inputs.

• Model Hub Account Compromise. If an attacker compromises a popular model creator’s Hugging Face account, they can replace legitimate models with backdoored versions. Thousands of users may download the poisoned model before detection.

  Detection: High difficulty – model appears to come from trusted source.

Data Pipeline Attacks

Targeting training data and fine-tuning datasets.

• Dataset Poisoning. Attackers inject malicious samples into public datasets or data sources you scrape. These samples embed backdoors (trigger phrases that cause specific outputs) or bias the model’s behavior. See Door 04 for detailed coverage.

  Detection: High difficulty – poisoned samples may be statistically indistinguishable from legitimate data.

These techniques have already caused real-world incidents affecting major organizations. The following case studies demonstrate the practical impact.

These incidents are grouped by attack surface, showing how theoretical techniques manifest in production environments.

Package Registry Compromises

• PyTorch Nightly Compromise (Dec 2022). Attackers compromised the official PyTorch nightly build, injecting a malicious dependency (torchtriton) that exfiltrated system information including hostname, username, working directory, and environment variables. Thousands of developers who installed PyTorch nightly between Dec 25-30 were affected. PyTorch Blog

  Attack surface: Software dependencies. Impact: System data exfiltration at scale.

• SolarWinds Orion (2020). While not AI-specific, this attack demonstrated the devastating potential of supply chain compromise. Attackers infiltrated the build pipeline, distributing backdoored updates to 18,000+ organizations including US government agencies. The same techniques apply to ML pipelines.

  Attack surface: Build pipeline. Impact: Widespread corporate and government network compromise.

Model Hub Incidents

• PoisonGPT Demonstration. Mithril Security researchers demonstrated how easy it is to upload a “lobotomized” LLM to Hugging Face – a model that appears normal but spreads targeted misinformation. They modified GPT-J-6B to give false answers about specific topics while performing correctly on benchmarks. Mithril Security

  Attack surface: Model artifacts. Impact: Demonstrated feasibility of backdoored model distribution.

• Malicious Pickle Models on Hugging Face. Security researchers discovered hundreds of models on Hugging Face containing malicious pickle payloads capable of arbitrary code execution. These models could steal credentials, install backdoors, or exfiltrate data from anyone who loaded them.

  Attack surface: Model artifacts (pickle). Impact: Potential RCE on developer/production machines.

These incidents share a common thread: the attackers targeted trusted distribution channels rather than application code. The following defense strategies address each attack surface.

Defense against supply chain attacks requires controls at multiple layers. Prioritize based on your risk tolerance and resources.

Tier 1: Essential (Before Any Deployment)

Non-negotiable controls that prevent the most common supply chain attacks.

1. 01
   Use Safe Serialization Formats.

   Never load models from pickle files in production. Use safetensors format which stores only numerical weights – no executable code. For PyTorch models, convert existing checkpoints: torch.save() → safetensors.torch.save_file().

2. 02
   Verify Cryptographic Signatures.

   Check SHA-256 hashes of every model and package against official sources. On Hugging Face, use signed commits and verify the publisher is the official organization account. Never skip hash verification for production deployments.

Tier 2: Standard (Production Systems)

Controls for teams deploying ML systems to production environments.

1. 03
   Dependency Pinning & SBOM.

   Pin exact versions of all dependencies in requirements.txt or poetry.lock. Maintain a Software Bill of Materials (SBOM) documenting every model, library, and version in your environment. Update deliberately, not automatically.

2. 04
   Private Package Registries.

   Proxy all package installations through an internal registry (Artifactory, Nexus, AWS CodeArtifact) that scans for malware and blocks typosquatting. Developers should never pip install directly from the public internet in production environments.

3. 05
   Automated Vulnerability Scanning.

   Integrate pip-audit, safety, or GitHub Dependabot into CI/CD pipelines. Scan model files with Picklescan before loading. Block deployments with known CVEs.

Tier 3: Advanced (High-Security Environments)

For enterprises with strict security requirements or handling sensitive workloads.

1. 06
   Sandboxed Model Loading.

   Load untrusted models in isolated containers with no network access, restricted filesystem permissions, and no volume mounts to production systems. Use gVisor or Firecracker for stronger isolation than standard Docker.

2. 07
   SLSA Compliance.

   Implement Supply-chain Levels for Software Artifacts (SLSA) framework for your ML pipeline. At minimum, achieve SLSA Level 2: version-controlled builds with authenticated provenance. Target SLSA Level 3 for critical systems.

The following checklist helps you systematically verify these controls based on your role.

Before deploying a model or dependency from external sources, verify these controls based on your role:

For Everyone (Product, Security, Engineering)

• Source Reputation Check. Are you downloading from trusted sources? Prefer verified publishers on Hugging Face, official organization accounts, or models with security audits. Check download counts, community reviews, and account age.

• Typosquatting Awareness. Did you triple-check the package name spelling? langchain vs lang-chain vs langchains – one character difference could be malware. Copy-paste from official documentation, never type from memory.

For Developers

• Safe Serialization Formats. Are you loading PyTorch models from pickle files? Switch to safetensors format which cannot execute arbitrary code. Convert existing models before deployment.

• Cryptographic Verification. Have you verified SHA-256 hashes match official releases? Check model cards and official repositories for published checksums. Automate hash verification in your loading code.

• Dependency Pinning. Have you pinned exact versions of all ML libraries in requirements.txt or lockfiles? Never use version ranges (e.g., >=1.0) in production.

For Security Teams

• SBOM Maintenance. Do you maintain a current Software Bill of Materials documenting every model, library, version, and source in your environment? Update it with every deployment.

• Automated Vulnerability Scanning. Are pip-audit, safety, or Dependabot integrated into CI/CD pipelines? Do builds fail on known CVEs? Is Picklescan running on model uploads?

• Sandboxed Model Testing. When evaluating untrusted models, do you load them in isolated containers with no network access and no volume mounts to production systems?

The following demonstration illustrates supply chain attack scenarios and the warning signs that indicate compromise.

• Supply Chain = Expanded Attack Surface. Every external model, library, and dataset is a potential attack vector. A single compromised dependency can undermine otherwise secure code.

• Pickle is a Security Disaster. PyTorch’s default serialization format can execute arbitrary Python code on load. The industry must adopt safetensors as the standard. Loading untrusted pickle files is equivalent to eval() on user input.

• Typosquatting Remains Effective. Attackers exploit developer typos and muscle memory. A single missed character in a package name (langchain vs lang-chain) can install malware instead of the intended library.

• Trust Must Be Verified. The open-source AI ecosystem enables rapid innovation but requires rigorous verification. Cryptographic signatures, SBOMs, and reputation checks are mandatory for production systems. Always prefer safetensors over pickle-based formats for model storage.

• LoRA/PEFT Adapters: Growing Attack Surface. The popularity of lightweight fine-tuning methods like LoRA creates new supply chain risks. Malicious adapters can alter model behavior while remaining small and easily distributed, making them harder to audit than full model replacements.

Research, tools, and best practices for securing ML supply chains.

Start Here

• OWASP LLM03:2025 - Supply Chain Vulnerabilities – Official documentation and mitigation strategies.

• Safetensors Documentation – Secure serialization format for ML models that prevents arbitrary code execution.

• SLSA Framework – Supply-chain Levels for Software Artifacts, applicable to ML model distribution.

Deep Dives

• PyTorch Nightly Compromise (2022) – Official post-mortem of supply chain attack affecting thousands of developers.

• PoisonGPT: How We Hid a Lobotomized LLM on Hugging Face – Demonstration of model hub supply chain risks.

• Planting Undetectable Backdoors in Machine Learning Models – Research on how outsourced model training enables undetectable backdoor attacks (2022).

OWASP & Industry References

• Hijacking Safetensors Conversion on Hugging Face – HiddenLayer research on safetensors conversion attacks.

• Large Language Models On-Device with MediaPipe and TensorFlow Lite – Google, edge deployment security considerations.

• Using LoRA Adapters with vLLM – vLLM documentation on LoRA adapter usage and security.

• Removing RLHF Protections in GPT-4 via Fine-Tuning – Research demonstrating safety removal through fine-tuning.

• Model Merging with PEFT – Hugging Face documentation on model merge security considerations.

• Safetensors Checker – Official Hugging Face tool to check safetensors metadata.

• Shadow Ray: AI Workloads Actively Exploited – Oligo Security, Ray AI framework exploit affecting thousands of servers.

• LeftoverLocals (CVE-2023-4969) – GPU local memory leak vulnerability allowing sensitive data recovery.

Taxonomy & Classification

• MITRE ATLAS AML.T0010 – Official classification for ML Supply Chain Compromise.

Tools & Frameworks

• pip-audit – Tool for scanning Python packages for known security vulnerabilities (CVEs).

• picklescan – Security scanner that detects malicious pickle files in ML models.

• safety – Python dependency scanner for known security vulnerabilities.

• GitHub Dependabot – Automated dependency updates and security alerts.