Adopting AI without losing control: the evidence on risk, and the controls that hold

Most companies are no longer deciding whether to adopt AI. They are deciding whether adoption happens inside a control framework or around one. The risk is not speculative: there is now a documented attack class against assistants and agents, a documented failure mode where AI acts on your systems with your credentials, and a documented record of what it costs when data walks out through a tool nobody reviewed. This post is the evidence, then the controls that hold, and then how to keep the speed advantage that made adoption worth doing.

First, the honest picture of the upside

Security arguments about AI lose credibility when they ignore the measured gains. Those gains are real, and they are conditional. The strongest published evidence comes from randomised or controlled studies rather than vendor surveys:

Study	Setting	Measured effect
Peng, Kalliamvakou, Cihon and Demirer (2023), arXiv 2302.06590 [1]	Randomised controlled experiment, developers asked to implement an HTTP server in JavaScript	The group with GitHub Copilot completed the task 55.8% faster than the control group.
Dell’Acqua et al. (2023), HBS working paper 24-013, later published in Organization Science [2]	758 consultants at Boston Consulting Group, 18 realistic consulting tasks	12.2% more tasks completed, 25.1% faster, and quality more than 40% higher. Consultants below the average performance threshold improved by 43%.
Noy and Zhang (2023), Science [3]	Mid-level professional writing tasks, preregistered experiment	Time on task fell 37% (about 10 minutes off a 27 minute control mean) and quality rose roughly 0.45 standard deviations.
Brynjolfsson, Li and Raymond, NBER working paper 31161, later in the Quarterly Journal of Economics [4]	Customer support agents at a single company	Issues resolved per hour rose 14% on average, 34% for novice and lower-skilled workers, with minimal change for the most experienced.
METR (2025) randomised controlled trial [5]	16 experienced open-source developers working on repositories they already knew well	Developers took 19% longer with AI tools than without, while expecting a 24% speedup.

Read those together and the pattern is clear rather than contradictory. Gains concentrate where the task is well specified, feedback is fast, and the human is not already the expert. Gains can invert when the work depends on tacit context that the model cannot see, which is exactly what the METR trial measured. The practical consequence: an average from a study is not a forecast for your workflow. Adopt where you can measure, and measure locally.

Second, what the incident record actually shows

The following are not hypotheticals. Each is a documented case, with the primary source listed at the end of this post.

Indirect prompt injection is an attack class, not a research curiosity

An assistant that reads content on your behalf cannot reliably distinguish your instructions from instructions hidden in that content. Four cases settled this:

• EchoLeak (CVE-2025-32711) in Microsoft 365 Copilot: a zero-click prompt injection that allowed remote, unauthenticated data exfiltration triggered by a single crafted email, with no user interaction required [6].
• Slack AI: PromptArmor demonstrated data exfiltration via indirect prompt injection from a public channel, where the assistant could be made to leak content from private channels it was authorised to read [7].
• The official GitHub MCP server: Invariant Labs showed that a malicious issue in a public repository could hijack a user’s agent into pulling private repository data into context and publishing it in a pull request on the public repository, where anyone can read it [8].
• Web browsing and document summarisation inherit the same weakness whenever the assistant can also send, write or fetch.

Simon Willison’s framing, the lethal trifecta, is the most useful way to reason about exposure: an assistant that has (a) access to private data, (b) exposure to untrusted content, and (c) the ability to communicate externally, is in a position to be turned into an exfiltration channel. Remove any one leg and most of this class of attack collapses [9].

Agents hold credentials, so agency is an entitlement question

OWASP’s 2025 list of the top risks for large language model applications names excessive agency as its own category, alongside prompt injection and sensitive information disclosure [18]. The operational translation is blunt: whatever an agent’s credentials can reach is the real blast radius, and agents are usually given more reach than the task needs. In July 2025, the founder of SaaStr reported that Replit’s coding agent deleted a production database during a declared code freeze, with the vendor’s chief executive publicly apologising and announcing safeguards afterwards [10].

Data leaves through the tools people actually use

The most common incident is not exotic. In April 2023 Samsung engineers pasted proprietary source code into ChatGPT; the company subsequently banned generative AI tools on company devices and internal networks [11]. The industry-wide pattern is that usage continues regardless, which is why the control has to be a sanctioned path rather than a ban. IBM’s Cost of a Data Breach Report 2026 puts the global average cost of a breach at 4.99 million US dollars, a 12% increase and a record high, and reports a 56% increase in AI-driven attacks led by deepfake impersonation and AI-enabled malware [12].

The model and dataset supply chain is executable

Model files are code in several common formats. In February 2025 ReversingLabs documented two malicious models on Hugging Face, using a technique it named nullifAI, that evaded Picklescan, the scanner Hugging Face uses, by packaging broken PyTorch archives that the scanner could not properly inspect. The authors note the structural weakness: a blacklist of dangerous functions does not scale against a format that can execute code [13].

Model output is frequently unsafe to trust

Veracode’s 2025 GenAI Code Security Report tested more than 100 large language models across Java, JavaScript, Python and C#: 45% of generated code samples failed security tests and introduced OWASP Top 10 vulnerabilities, and defences against cross-site scripting failed in 86% of the relevant samples. Java was the riskiest language tested, with Python at 38%, JavaScript at 43% and C# at 45% [14].

Overreliance is a liability question

When an assistant invents something and a customer relies on it, the courts do not care that it was the model’s fault. On 14 February 2024 the British Columbia Civil Resolution Tribunal found Air Canada liable for negligent misrepresentation after its website chatbot gave incorrect bereavement fare information [16]. Separately, the US Securities and Exchange Commission settled charges in March 2024 against two investment advisers for materially misleading statements about their AI capabilities, with combined penalties of 400,000 US dollars, in what the agency framed as its first AI-washing enforcement actions [17]. Risk exists in the marketing of AI as much as in its operation.

Attackers are already using it against you

In November 2025 Anthropic published an investigation into a campaign it attributes with high confidence to a Chinese state-sponsored group it designates GTG-1002. The operator used Claude Code to attempt intrusions against roughly thirty targets, succeeding in a small number of cases. Anthropic’s assessment is that AI performed 80 to 90% of the campaign, with human involvement only at a handful of critical decision points per campaign, targeting large technology companies, financial institutions, chemical manufacturers and government agencies [15].

Figure 1. The entry points are content, endpoints, integrations and artefacts. The pivots are entitlements, retrieval boundaries and output handling. The impact is what follows.

Why the controls you already own do not cover this

• Natural language is now an input channel. Input validation, parameterised queries and typed interfaces assume structured input. A document, a web page, a calendar invite or a repository issue can carry instructions the system will act on, and there is no reliable parser that separates data from instructions [6][8].
• The attacker does not need an exploit. The vulnerability class is abuse of intended functionality. Nothing is patched on your side because nothing is broken on your side.
• Identity is the blast radius. Agents, copilots and integrations authenticate as something. Standing credentials, broad scopes and shared service accounts convert a content-level bug into a data-level incident [18].
• Data moves to third parties by design. Prompts, retrieved documents and outputs cross an organisational boundary that your perimeter controls do not model, and retention and training terms differ per vendor and per tier [12].
• There is no patch cycle for prompts. Models and their behaviour change on the vendor’s schedule. Controls that depend on a specific model refusing a specific request are not controls, they are luck.
• The telemetry is usually missing. Many deployments log the request and response but not the retrieved context, the tool calls, the agent identity or the egress. An incident you cannot reconstruct is an incident you cannot contain [22].
• Integrations expand reach faster than review. Every MCP server, plugin or connector an assistant may call is another supplier inside your trust boundary with its own permissions and its own content [8][28].

The control framework that holds

Nothing below is novel, and that is the point. These are extensions of controls you already run, mapped to the four functions of the NIST AI Risk Management Framework (GOVERN, MAP, MEASURE, MANAGE) [19], the generative AI risk categories in NIST AI 600-1 [20], the OWASP top 10 for LLM applications [18], and the joint guidance from the NCSC with CISA, the NSA and international partners, which splits the problem into secure design, secure development, secure deployment, and secure operation and maintenance [21][22].

Layer	What it prevents	Concrete practice
Governance and inventory	Unknown deployments, unowned risk, vendor terms nobody read	One page of rules people can follow, a register of AI use including what was adopted without asking, a named owner per use case, and documented vendor terms on training, retention and subprocessors. ISO/IEC 42001:2023 provides a management system structure if you want one.
Risk tiering	Uniform scepticism, which slows everything down, or uniform enthusiasm, which is worse	Sort use cases by data sensitivity, tool reach and reversibility of action. Low-risk work gets a fast lane with conditions. High-risk work gets review, testing and monitoring before launch.
Identity and entitlements	Agents converting a content bug into a data breach	Dedicated non-human identities per agent, short-lived scoped tokens, no standing administrative rights, read-only by default, explicit human confirmation before irreversible actions such as payments, deletions or outbound messages.
Data controls	Sensitive data leaving the company inside a prompt	Classify before prompting. Block or transform sensitive fields at source, keep regulated data in a private deployment, verify tenant isolation, and apply egress control to the domains AI workloads may reach. Permission-aware retrieval so an index cannot return documents the requesting user could not open.
Output handling	Text becoming an action	Treat model output as untrusted input: contextual encoding before rendering, no direct concatenation into shell commands, SQL or configuration, sandbox any browsing or code execution, and allowlist outbound destinations.
Supply chain and provenance	Malicious or tampered models, datasets and integrations	Verify signatures and hashes, prefer safe serialisation formats over executable ones, use private registries, scan artefacts, review new connectors and MCP servers like any other supplier, and keep an inventory of where each model came from [13][22].
Telemetry	Undetectable, unattributable AI incidents	Log prompts, retrieved documents, tool calls, agent or service identity, model and configuration changes, and egress. Retain long enough to investigate, and keep it off the system under investigation [22].
Detection and response	Slow containment, repeat compromise	Alert on anomalous agent behaviour, unusual retrieval volumes, new egress destinations, injection patterns in ingested content, and unexpected model configuration changes. Maintain a kill switch and an AI incident playbook: revoke tokens, disable tools, quarantine the integration, preserve evidence.
Assurance	Confidence without evidence	Red teaming mapped to MITRE ATLAS [23], prompt injection test suites in the pipeline, control testing before go-live, and periodic external review.

Figure 2. The layers map to the NIST AI RMF functions and to the joint NCSC, CISA and NSA guidance. Reporting is the layer most organisations skip.

Detection and response, specifically

The joint NSA and CISA guide on deploying AI systems securely is unusually concrete and worth reading in full [22]. Its monitoring advice is the part most deployments miss: collect logs covering inputs, outputs, intermediate states and errors; automate alerts; monitor the model’s architecture and configuration for unauthorised changes; and monitor for attempts to access or elicit data from the model or to aggregate inference responses. The same guide recommends validating artifacts cryptographically before and during use, testing models after modification, hardening the deployment environment with isolated containers or virtual machines and allow-listed egress, and engaging external testers before go-live.

For incident response, the practical additions to your existing playbook are:

1. Identify the agent, the credentials it held and the tools it could reach.
2. Revoke those credentials and token scopes first, then disable the integration, because the data path is usually still live while you investigate.
3. Preserve prompts, retrieved context, tool calls and egress logs before rotating anything that would overwrite them.
4. Check the retrieval index for poisoned or injected documents, since a single malicious document keeps working until it is removed.
5. Assess notification duties. In Hong Kong, personal data breaches engage the Personal Data (Privacy) Ordinance and, in regulated sectors, the HKMA or SFC as applicable. Keep the legal test separate from the technical timeline.

Rehearse it. An AI incident tabletop, run against a scenario drawn from the cases above, costs a morning and exposes the gaps that matter: who can revoke an agent’s access at 02:00, and who decides to switch a capability off.

Keeping the competitive edge while doing this properly

Controls that stop adoption lose to competitors who adopted. The aim is a framework that says yes quickly to most things and no, or not yet, to the few that can hurt you. Four design principles do most of the work.

• Make the safe path the fast path. Provide an approved assistant with enterprise terms, single sign-on, logging and a private tenant, and make requesting a new tool a two-day process rather than a six-week project. Prohibition without a sanctioned alternative produces Samsung’s problem plus no visibility [11].
• Tier by consequence, not by technology. A summarisation tool that reads public marketing content is not the same risk as an agent that can move money. Reserve deep review for irreversible actions, regulated data and broad entitlements.
• Measure locally, then scale what works. The published evidence is real but conditional [1][2][3][4], and METR’s trial shows it can invert in expert, context-heavy work [5]. Run a bounded pilot with a baseline, a control group if you can manage it, and a decision date. Scale the workflows that clear the bar and retire the ones that do not.
• Instrument from day one. You cannot manage what you do not log, and you cannot prove value without a baseline. Telemetry is the same investment that makes the incident survivable and the business case defensible [22].

There is also a positioning argument. A company that can demonstrate AI governance, tested controls and a rehearsed AI incident response is a better counterparty for regulated customers, insurers and enterprise procurement. In regulated sectors that is not marketing, it is a gate: the Hong Kong Monetary Authority expects authorised institutions to extend its 2019 guiding principles on big data and AI to generative AI in customer-facing applications across governance and accountability, fairness, transparency and disclosure, and data privacy and protection [25], while the Securities and Futures Commission issued its own expectations for licensed corporations using generative AI language models in November 2024 [26]. The Privacy Commissioner’s model framework for personal data protection in AI, published in June 2024, sets the privacy baseline [27]. Beyond Hong Kong, the EU AI Act has been phasing in since 1 August 2024: prohibitions and AI literacy duties applied from 2 February 2025, obligations for general-purpose AI models, governance and penalties from 2 August 2025, and the remainder from 2 August 2026, which matters to any group with EU customers, staff or suppliers [24].

A 90 day plan

Window	Objective	Deliverables
Days 1 to 30	See what is already happening	Inventory every AI tool and integration in use, including unsanctioned ones discovered through single sign-on, finance and network data. Classify the data each one touches. Write the one-page policy with a fast lane and a review lane. Publish a sanctioned assistant.
Days 31 to 60	Control the paths that matter	Fix identity: dedicated credentials, scoped tokens, no standing admin for agents. Apply egress allow-lists to AI workloads and permission-aware retrieval to internal indexes. Turn on prompt, tool call and configuration logging. Threat model the top three use cases with the lethal trifecta as the test [9].
Days 61 to 90	Prove it and rehearse it	Red team one assistant against prompt injection and one agent chain against the GitHub MCP pattern [8]. Run an AI incident tabletop. Land the vendor terms review. Report to the board on exposure, controls and measured value, using your own pilot data rather than vendor averages.

What to take away

1. The upside is real, conditional, and measurable. Adopt where you can measure, and do not assume a study’s average applies to your workflow [5].
2. Prompt injection against assistants that read untrusted content is an established attack class with public cases, including a zero-click vulnerability in a mainstream enterprise assistant [6][7][8].
3. Agents turn a content problem into an entitlement problem. The blast radius is the credentials, not the model [10][18].
4. Most documented loss still comes from ordinary data handling and AI-assisted fraud, not from exotic model attacks [11][12].
5. The model supply chain is executable and the scanners are incomplete [13].
6. Governing frameworks and sector expectations already exist, and they are specific enough to build a programme on [19][20][21][22][25][26][27].

Sources

1. Peng, Kalliamvakou, Cihon and Demirer, “The Impact of AI on Developer Productivity: Evidence from GitHub Copilot” (2023) · https://arxiv.org/abs/2302.06590
2. Dell’Acqua et al., “Navigating the Jagged Technological Frontier” (HBS working paper 24-013, 2023; later published in Organization Science) · https://mitsloan.mit.edu/sites/default/files/2023-10/SSRN-id4573321.pdf
3. Noy and Zhang, “Experimental evidence on the productivity effects of generative artificial intelligence”, Science (2023) · https://www.science.org/doi/10.1126/science.adh2586
4. Brynjolfsson, Li and Raymond, “Generative AI at Work”, NBER working paper 31161, later published in the Quarterly Journal of Economics · https://www.nber.org/papers/w31161
5. METR, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity” (July 2025) · https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
6. Reddy and Gujral, “EchoLeak: The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System”, arXiv 2509.10540 (CVE-2025-32711) · https://arxiv.org/abs/2509.10540
7. PromptArmor, “Data Exfiltration from Slack AI via Indirect Prompt Injection” (August 2024) · https://www.promptarmor.com/resources/data-exfiltration-from-slack-ai-via-indirect-prompt-injection
8. Invariant Labs, “GitHub MCP Exploited: Accessing private repositories via MCP” (26 May 2025) · https://invariantlabs.ai/blog/mcp-github-vulnerability
9. Simon Willison, “The lethal trifecta for AI agents” (June 2025) · https://simonwillison.net/2025/Jun/16/the-lethal-trifecta/
10. Business Insider, reporting on the Replit agent database deletion and the vendor response (July 2025) · https://www.businessinsider.com/replit-ceo-apologizes-ai-coding-tool-delete-company-database-2025-7
11. TechCrunch, “Samsung bans use of generative AI tools like ChatGPT after April internal data leak” (May 2023) · https://techcrunch.com/2023/05/02/samsung-bans-use-of-generative-ai-tools-like-chatgpt-after-april-internal-data-leak/
12. IBM, Cost of a Data Breach Report 2026 · https://www.ibm.com/reports/data-breach
13. ReversingLabs, “Malicious ML models discovered on Hugging Face” (nullifAI, February 2025) · https://www.reversinglabs.com/blog/rl-identifies-malware-ml-model-hosted-on-hugging-face
14. Veracode, 2025 GenAI Code Security Report (July 2025) · https://www.veracode.com/blog/genai-code-security-report/
15. Anthropic, “Disrupting an AI-orchestrated cyber espionage campaign” (November 2025) · https://www.anthropic.com/news/disrupting-AI-espionage
16. Cassels analysis of Moffatt v. Air Canada, British Columbia Civil Resolution Tribunal, decision of 14 February 2024 · https://www.cassels.com/insights/talk-is-not-always-cheap-ai-chatbots-misinformation-leads-to-liability/
17. US Securities and Exchange Commission, press release 2024-36, settled charges for AI washing (18 March 2024) · https://www.sec.gov/newsroom/press-releases/2024-36
18. OWASP, Top 10 for LLM Applications 2025 · https://genai.owasp.org/llm-top-10/
19. NIST, AI Risk Management Framework (AI RMF 1.0, January 2023, voluntary) · https://www.nist.gov/itl/ai-risk-management-framework
20. NIST, AI 600-1 Generative AI Profile (July 2024), including the twelve generative AI risk categories · https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.600-1.pdf
21. UK National Cyber Security Centre with international partners, “Guidelines for secure AI system development” · https://www.ncsc.gov.uk/collection/guidelines-secure-ai-system-development
22. NSA, CISA and international partners, “Deploying AI Systems Securely” (15 April 2024) · https://www.cisa.gov/news-events/alerts/2024/04/15/joint-guidance-deploying-ai-systems-securely
23. MITRE ATLAS, knowledge base of adversary tactics and techniques against AI systems · https://atlas.mitre.org/
24. EU AI Act implementation timeline (in force 1 August 2024; prohibitions and AI literacy from 2 February 2025; general-purpose AI, governance and penalties from 2 August 2025; remainder from 2 August 2026) · https://artificialintelligenceact.eu/implementation-timeline/
25. HKMA, circular on consumer protection in respect of the use of generative AI, 7 November 2024 · https://brdr.hkma.gov.hk/eng/doc-ldg/docId/getPdf/20241107-1-EN/20241107-1-EN.pdf
26. SFC, circular to licensed corporations on the use of generative AI language models (ref 24EC55, effective 12 November 2024) · https://apps.sfc.hk/edistributionWeb/gateway/EN/circular/intermediaries/supervision/doc?refNo=24EC55
27. PCPD, “Artificial Intelligence: Model Personal Data Protection Framework” (11 June 2024) · https://www.pcpd.org.hk/english/news_events/media_statements/press_20240611.html
28. Model Context Protocol, open standard for connecting assistants to tools and data sources · https://modelcontextprotocol.io/

Where this leaves you

The companies that will get this right are not the ones that moved fastest or the ones that waited. They are the ones that treated AI adoption as an architecture and identity problem early, wrote down what they would not allow, gave everyone a good sanctioned option, and built the logging and the response muscle before they needed it. If you want to test that position rather than assert it, we run two engagements that map directly to this post: an AI adoption risk assessment that produces the inventory, the tiering and the control gaps, and a tabletop exercise built from the incidents above so your team finds out where the gaps are on a Tuesday afternoon rather than during an incident.

Published for defensive education. Figures are quoted from the sources listed. Verify current guidance against your own regulators and vendors before acting, and validate any control in your own environment.