Your architecture diagram shows your cloud, your identity provider and your SIEM. It almost certainly does not show the six or seven places where prompts, documents and API keys leave the company for a model that somebody else operates. That layer is now a supplier relationship in everything but name, and the documented incidents around it are not about exotic model attacks. They are about stolen keys, exposed endpoints, typosquatted packages, extensions that read your chats, and a resale market for inference capacity that you pay for.
Five provider relationships, not one
Third-party LLM risk gets discussed as if there were one kind of provider. There are five, and they fail differently, so they need different controls.
| Relationship | What you have given them | How it fails |
|---|---|---|
| Contracted vendor (frontier or enterprise API) | Prompts, documents, outputs and a payment relationship, governed by terms you accepted | Provider-side incident or breach becomes your notification duty; retention and training terms decide what you gave away; behaviour changes on their release schedule. |
| Aggregator, router or gateway | Keys and content passing through an intermediary, often with logging you cannot inspect | You inherit a supplier you did not audit, at a price model that rewards volume; abuse quotas and terms enforcement land on your account. |
| Criminal service | Nothing voluntarily, but the consequences arrive regardless | Stolen cloud or API credentials consumed and resold; uncensored models sold by subscription to the people targeting you. |
| Fake or impersonating provider | Keys, prompts, chat histories and, sometimes, money | Typosquatted API sites harvesting credentials, packages and SDKs with infostealers, browser extensions and MCP servers that exist to collect data. |
| Your own exposed endpoint | Compute, model access and whatever the endpoint can reach | A self-hosted model reachable from the internet turns you into an unwitting provider: someone else’s usage, your data and your bill. |
Figure 1. The provider relationship you have is not always the one you think you signed up for.
Stolen inference capacity: the criminal provider economy
The clearest documented threat in this space is the theft of AI credentials, a technique Sysdig named LLMjacking in May 2024. The first documented campaign exploited CVE-2021-3129, a remote code execution flaw in Laravel Ignition debug mode, used it to reach cloud credentials embedded on the host, and then invoked AI services across ten different cloud providers from those credentials [1][2].
The economics are the point. Sysdig estimated that running a Claude 2.x-class model at full capacity could cost the victim around 46,000 US dollars per day, and by the time higher-capability models were in scope the estimate rose to over 100,000 US dollars per day [2][3]. The Cloud Security Alliance’s analysis of the same telemetry reports a 376% increase in credential theft targeting AI services between the fourth quarter of 2025 and the first quarter of 2026, and documents what it calls the first fully commercial LLMjacking marketplace: automated scanning, credential validation and resale through a dedicated storefront, generating 35,000 attack sessions across more than 30 LLM providers in roughly 36 days [2].
Two details matter more than the headline numbers. First, the resale price: stolen AI credentials trade at 40 to 60 percent below legitimate pricing, which tells you the demand is steady and the supply is not scarce [2]. Second, the detection path: victims typically discover the abuse from an unexplained cloud bill or from their provider revoking the key for terms violations, not from a security alert [2]. Sysdig also warned that disabling models in a platform such as Amazon Bedrock is not a security measure, because attackers will simply enable them on your behalf once they hold the credentials [3].
Exposed inference: when you are the provider
The same abuse applies in reverse. In September 2025 Censys identified 10,600 high-confidence Ollama instances exposed to the internet, and demonstrated that 1,500 of them answered direct prompts through the exposed API, with 1,122 responding to both of two test prompts [4]. A joint investigation by SentinelOne SentinelLABS and Censys in January 2026 counted 175,000 unique exposed Ollama hosts across 130 countries, describing an unmanaged and publicly accessible layer of AI compute infrastructure that runs outside the guardrails platform providers apply by default [5].
If your organisation runs self-hosted inference, ask the question in the other direction: what would someone do with prompt access to your model and whatever network position it holds? The exposure is not only the model weights, it is the ability to use your compute, reach your data through retrieval, and appear in your logs as legitimate internal traffic.
Fakes and impersonators: providers that exist to take keys
Fake AI services have been a phishing theme since the first wave of ChatGPT hype. Check Point Research found in May 2023 that of newly registered domains related to ChatGPT or OpenAI, one in every 25 was malicious or potentially malicious [11]. The refinement since then is what gets taken: not passwords, but API keys, because a valid key converts directly into someone else’s compute.
GitGuardian’s 2026 State of Secrets Sprawl report quantifies the supply. In 2025, 28.65 million new hardcoded secrets were added to public GitHub commits, a 34% year-on-year increase and the largest single-year jump the report has recorded. AI service secrets reached 1,275,105, up 81% year on year, with 113,000 leaked DeepSeek API keys cited as one example. Eight of the ten fastest-growing secret detectors tracked by the report were tied to AI services, and LLM infrastructure such as orchestration, retrieval and vector storage leaked five times faster than core model providers. The same report counted 24,008 unique secrets exposed in MCP configuration files on public GitHub, of which 2,117 were valid credentials [6].
The scale of what sits in public AI artefacts is larger still. Truffle Security scanned every public dataset on Hugging Face, 7.6 petabytes across roughly 187 million files and some 815,000 dataset repositories, and found 221,303 live unique credentials sitting in 6,003 datasets, including one secret that had access to 393 GB of data [7]. A model or dataset you pull is not code you reviewed.
Malicious packages are the delivery mechanism. In July 2026, an advisory was published for the PyPI package openai-mcp: a typosquat published from a compromised account containing an embedded infostealer that exfiltrated environment variables and cloud tokens, linked to the wider Shai-Hulud related PyPI campaign wave [10]. The name is designed to look like the thing you were already going to install.
The provider inside your workflow
The most consequential third-party components are not models, they are connectors and agents that hold permissions. In September 2025 the npm package postmark-mcp, an MCP server that lets assistants send email through Postmark, was modified to add a blind carbon copy to an external address so that every outbound email was silently forwarded to the attacker. The behaviour appeared around version 1.0.16, the package was downloaded roughly 1,500 times a week, and it is regarded as the first widely covered case of a malicious MCP server used against real workflows. The relevant structural point from the analysis: MCP servers run with high trust and broad permissions inside agent toolchains, so whatever they handle, including password resets and invoices, passes through them [8].
Browser-side, Microsoft’s Defender research reported in March 2026 that malicious Chromium extensions impersonating legitimate AI assistant tools had reached approximately 900,000 installs, with activity confirmed across more than 20,000 enterprise tenants. The extensions collected full URLs and AI chat content from platforms including ChatGPT and DeepSeek, exposing proprietary code, internal workflows and confidential discussions [9]. No exploit was required, only an install.
Dark LLM shops: providers built for your attacker
There is also a provider market where the customer is the attacker. FraudGPT appeared on Telegram channels from 22 July 2023, sold for offensive use with advertised capabilities including malicious code, undetectable malware, carding and phishing pages, at subscription prices from 200 US dollars per month to 1,700 US dollars per year, with 3,000 or more confirmed sales and reviews at the time of the research. The same actor launched WormGPT on 13 July 2023 [12]. KELA’s analysis of cybercrime channels reported a 219% increase in mentions of malicious AI tools and a 52% increase in jailbreak discussion during 2024, which is the trend line that matters: the tooling is normalising, not declining [13].
Your provider’s incident is your incident
Provider security is not an abstraction. In March 2023 OpenAI disclosed that a bug in the open-source redis-py client library allowed some users to see other users’ chat history titles, and that the same bug may have caused payment-related information of 1.2% of ChatGPT Plus subscribers active in a specific nine-hour window to be unintentionally visible: first and last name, email address, payment address, card type, the last four digits of the card and the expiry date, with no full card numbers exposed [15].
The lesson is not that this vendor is unsafe. It is that you cannot patch a supplier. The controls you hold are what you send them, what they are permitted to retain, what they must tell you and when, and how quickly you can change providers. Those are contract and architecture decisions, and they have to be made before the incident rather than after it.
Why normal supply chain review misses this
- The provider holds your data by design. Prompts are the input and outputs are the product, so data minimisation is the only durable control [2][6].
- The attack is measured in money. Stolen inference looks like usage, and abuse surfaces in billing before it surfaces in a security tool [2][3].
- The credential is the asset. Keys resell at a discount and are harvested from repositories, notebooks, MCP configs and datasets [2][6][7].
- Agents convert content into action. A connector that can send, write or fetch turns a provider response into an operation [8].
- Secrets leak faster than review cycles. AI service secrets grew 81% in a single year, and eight of the ten fastest-growing secret types were AI related [6].
- Adoption is invisible. Extensions at 900,000 installs, packages pulled into build pipelines, MCP servers pasted into agent configs [8][9][10].
The provider review questions that matter
| Question | Why it matters | What good looks like |
|---|---|---|
| Does our data train or improve their models, and can that be turned off in writing? | This is the single largest difference between vendors at similar prices | Contractual no-training commitment, honoured by configuration, testable on your own tenant |
| What is retained, where, and for how long? | Determines your exposure window and your notification burden | Documented retention schedule, regional processing options, deletion on request |
| Who are the subprocessors? | Your data reaches parties you have never assessed | Published subprocessor list, notice of changes, flow-down obligations |
| What are the security credentials and the incident notification terms? | You inherit their timeline in your own regulatory reporting | Independent attestation (ISO 27001 or SOC 2), notification measured in hours, clear escalation contact |
| What happens to keys and access on exit? | Offboarding is where standing access hides | Revocation and deletion within a defined period, evidence on request |
| Can we get usage and audit logs? | Without them, abuse is invisible and investigation is impossible | Invocation logs, per-key usage, exportable audit trail |
| How is abuse handled, and how are we notified? | Terms enforcement can take your production feature offline without warning | Documented abuse process, advance notice where lawful, appeal route |
| Is the provider reselling capacity or hosting others? | Aggregators add a party you did not audit and cannot see | Transparent routing, named subprocessors, no logging beyond what is disclosed |
Controls that reduce the blast radius
Figure 2. Credentials and money are the two layers most often missed, and the two that detect LLMjacking earliest.
| Control | Attacks it blunts | Implementation |
|---|---|---|
| Per-workload, scoped, expiring credentials | Credential theft and resale; silent lateral use | One key per service, least-privilege scopes, short lifetimes, rotation in a secret manager, no keys in notebooks, configs or MCP files [6][7] |
| Hard quotas and budget alarms on every key | Inference theft and cost attacks | Provider usage dashboards plus platform cost anomaly detection; alarm thresholds that page someone; caps that stop the spend rather than report it [2][3] |
| Inference and API invocation logging | Undetectable abuse, unattributable incidents | Turn on model invocation logging and ship it to the SIEM; alert on new geography, new ASN, unusual token volume and models enabled without a change record [2][3] |
| Provider output treated as untrusted input | Response-side prompt injection into agents | Constrain what consumes the text: no direct shell or database execution, human confirmation for irreversible actions, sandboxed browsing [14] |
| Egress allow-listing for AI workloads | Exfiltration and unauthorised provider use | Only named provider endpoints reachable from agent and inference subnets; block consumer assistants on managed devices or supply an approved alternative [9] |
| Secret scanning across code, notebooks and MCP configs | Key leakage feeding the resale market | Pre-commit and repository scanning for model provider key formats, including agent and MCP configuration files, where valid credentials were found in the tens of thousands [6] |
| Dependency and connector review | Malicious packages, extensions and MCP servers | Pin versions, review diffs on update, treat new connectors as new suppliers, inventory extensions in the browser estate [8][10][17] |
| Provenance checks on models and datasets | Tampered or malicious artefacts pulled from public hubs | Verify hashes and signatures, prefer safe serialisation formats, use private mirrors, and scan datasets before use [7][16][18] |
Detection signals worth alerting on
Two of the platform features this guidance depends on are documented and available today: Amazon Bedrock model invocation logging, which captures the prompts and responses your account sends to models, and CloudWatch anomaly detection for the spend that stolen capacity produces. Both are configuration decisions rather than projects [19].
| Signal | What it usually means | Where it shows |
|---|---|---|
| Spend or token volume climbing without a matching feature release | Stolen credentials consuming your capacity | Cloud billing, provider usage dashboards, cost anomaly detection [2][3] |
| A key used from a new country, ASN or user agent | Key leaked or resold | Provider audit logs, model invocation logs, cloud trail [2] |
| Models enabled or quotas raised without a change ticket | Attacker preparing capacity | Provider audit logs; note that model activation is not gated by default [3] |
| Provider terms enforcement or key revocation notices | Abuse originating from your account | Provider email, support tickets |
| Unexpected outbound destinations from agent or inference subnets | Exfiltration or unauthorised provider use | Egress logs, DNS logging, MCP server logs [8][9] |
| A dependency, extension or MCP server updating without review | Silent malicious behaviour introduced by a supplier | Lockfile diffs, browser extension inventory, package advisories [8][10] |
| Chat or prompt data appearing in datasets or public repositories | Leakage through development and training workflows | Secret scanning, dataset scanning, exposure monitoring [6][7] |
When it goes wrong, in order
- Identify the credential, what it can reach and who else could have used it, including any agent holding the same key.
- Revoke and rotate first, then investigate: standing keys keep working while you analyse.
- Pull the provider and platform logs for the period of exposure to see what was queried, from where and at what volume.
- Assess what left: prompts, retrieved documents, generated output and any data the credential could reach directly [2][3].
- Check the contractual and regulatory clock: notification duties to regulators and customers, and the provider’s obligations to you [15].
- Remove the design flaw that allowed it: replace long-lived shared keys with scoped, expiring credentials and quotas, then re-test detection with the abuse case you just lived through.
This is also where established secure development guidance lands. The NCSC’s joint guidelines with international partners place dependency and supply chain management inside secure development, which is exactly where provider review belongs: before the integration ships, not after the invoice arrives [20].
What to take away
- There are five provider relationships in a typical business, not one, and the invisible ones carry the most risk [4][8][9].
- The dominant documented threat is credential theft leading to resold inference capacity, with victims discovering it from their bill [1][2][3].
- Exposed self-hosted inference is the same problem in reverse, at a scale of tens of thousands of endpoints [4][5].
- The supply of keys is large and growing: AI service secrets up 81% in a year, with valid credentials sitting in MCP configs and public datasets [6][7].
- Provider incidents reach you regardless of fault, so retention terms, notification clauses and exit plans are security controls, not procurement paperwork [15].
Sources
- Sysdig, “LLMjacking: Stolen Cloud Credentials Used in New AI Attack” (6 May 2024), which coined the term and documented the first campaign · https://www.sysdig.com/blog/llmjacking-stolen-cloud-credentials-used-in-new-ai-attack
- Cloud Security Alliance research note, “LLMjacking Evolved: Stolen AI Compute as Offensive Infrastructure”, analysing Sysdig and Pillar Security telemetry including Operation Bizarre Bazaar · https://labs.cloudsecurityalliance.org/research/csa-research-note-llmjacking-offensive-agentic-infrastructur/
- CSO Online, reporting Sysdig’s LLMjacking research and the cost estimates per model class (24 April 2025) · https://www.csoonline.com/article/3535433/llmjacking-how-attackers-use-stolen-aws-credentials-to-enable-llms-and-rack-up-costs-for-victims.html
- Censys, “Ollama Drama: Investigating the Prevalence of Ollama Open Instances” (24 September 2025) · https://censys.com/blog/ollama-drama-investigating-the-prevalence-of-ollama-open-instances-with-censys
- The Hacker News, reporting the SentinelOne SentinelLABS and Censys joint investigation into 175,000 exposed Ollama hosts (29 January 2026) · https://thehackernews.com/2026/01/researchers-find-175000-publicly.html
- GitGuardian, The State of Secrets Sprawl 2026: 28.65 million secrets on public GitHub, AI service secrets up 81%, and secrets in MCP configuration files · https://blog.gitguardian.com/the-state-of-secrets-sprawl-2026/
- Truffle Security, “Scanning 7.6 Petabytes of HuggingFace Training Data for Secrets” (1 June 2026) · https://trufflesecurity.com/blog/scanning-7-6-petabytes-of-ai-training-data-for-secrets
- Snyk, “Malicious MCP Server on npm postmark-mcp Harvests Emails” (25 September 2025) · https://snyk.io/blog/malicious-mcp-server-on-npm-postmark-mcp-harvests-emails/
- Microsoft Security Blog, “Malicious AI Assistant Extensions Harvest LLM Chat Histories” (5 March 2026) · https://www.microsoft.com/en-us/security/blog/2026/03/05/malicious-ai-assistant-extensions-harvest-llm-chat-histories/
- GitHub Advisory Database, GHSA-h7rx-63j6-3w83, malicious code in the PyPI package openai-mcp (21 July 2026) · https://github.com/advisories/GHSA-h7rx-63j6-3w83
- Check Point Research, fake websites impersonating association with ChatGPT (2 May 2023) · https://blog.checkpoint.com/research/fake-websites-impersonating-association-to-chatgpt-poses-high-risk-warns-check-point-research/
- Netenrich, “FraudGPT: The Villain Avatar of ChatGPT” (25 July 2023), documenting FraudGPT pricing, capabilities and the launch of WormGPT on 13 July 2023 · https://netenrich.com/blog/fraudgpt-the-villain-avatar-of-chatgpt
- Infosecurity Magazine, reporting KELA’s analysis of the 2024 surge in malicious AI tool mentions and jailbreak discussion across cybercrime channels · https://www.infosecurity-magazine.com/news/dark-web-mentions-malicious-ai/
- OWASP Top 10 for LLM Applications 2025, including LLM03 Supply Chain, LLM05 Improper Output Handling and LLM10 Unbounded Consumption · https://genai.owasp.org/llm-top-10/
- OpenAI, “March 20 ChatGPT outage: Here’s what happened” (24 March 2023) · https://openai.com/index/march-20-chatgpt-outage/
- MITRE ATLAS, knowledge base of adversary tactics and techniques against AI systems, including ML supply chain compromise · https://atlas.mitre.org/
- Model Context Protocol, the open standard for connecting assistants to tools and data sources whose configuration files were scanned by GitGuardian · https://modelcontextprotocol.io/
- NSA, CISA and international partners, “Deploying AI Systems Securely” (15 April 2024), on validating artefacts and monitoring model usage · https://www.cisa.gov/news-events/alerts/2024/04/15/joint-guidance-deploying-ai-systems-securely
- AWS documentation for Amazon Bedrock model invocation logging and CloudWatch anomaly detection, the two platform features referenced in the detection guidance · https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
- UK National Cyber Security Centre with international partners, “Guidelines for secure AI system development”, whose secure development guidance covers supply chain and dependency management · https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
Where this leaves you
The practical shift is to treat model access the way you treat administrative access: something that has to be named, scoped, monitored and revocable. If you cannot say which keys your workloads hold, which provider each one reaches, what that provider retains and what would happen if the key were sold tomorrow, the gap is in the inventory rather than in the model. We run two engagements that close it directly: an AI and third-party provider review that produces the inventory, the contract gaps and the key-by-key exposure picture, and a detection build for AI telemetry so that stolen inference shows up as an alert rather than an invoice.
Published for defensive education. Figures are quoted from the sources listed and are reproduced with their dates. Verify current terms and platform features with your own providers before acting.