Master the security engineering required to deploy large language models and generative AI safely in production. Learn the OWASP LLM Top 10 in depth, design secure inference architectures, defend against data exfiltration and unsafe output, and build an assurance programme that survives enterprise audit.
Large language models moved from prototypes to production systems that read customer data, call internal APIs, write code and take decisions. The security model that grew around them is still catching up. Traditional application security assumes deterministic code and a clear boundary between trusted logic and untrusted input. A generative system collapses that boundary: instructions and data share the same channel, outputs are probabilistic, and the same request can produce a safe answer today and an unsafe one tomorrow.
This course teaches security engineering for that environment. It begins with the OWASP Top 10 for LLM Applications and the MITRE ATLAS adversary model, then works through the architecture of a defensible inference stack: authenticated gateways, tenant isolation, prompt and context segregation, output handling, tool scoping, telemetry and evaluation. Each module pairs an attack with the control that contains it and the metric that proves the control works.
The course is intentionally adversarial. Learners reproduce representative attacks in a sandbox, observe what a successful injection actually looks like in logs, and then design detection and containment. By the end, you should be able to review an LLM feature specification and state, with evidence, which risks are mitigated, which are accepted and which remain unknown.
Generative AI security is now an audit topic. Enterprise customers ask for OWASP LLM coverage in security questionnaires, insurers ask for guardrail evidence, and regulators ask for documented risk management under frameworks such as the EU AI Act and India DPDP Act. The practices in this course are designed to produce evidence, not just assurances.
Scope. This course covers llm and generative ai security across the model, application, data and operational layers. It is deliberately practical: every module ends with a control you can implement, a test you can run and a metric you can report. Adjacent topics in this category — prompt injection, retrieval security, agent tooling, runtime guardrails, supply chain provenance and AI security operations — are referenced where they interact, because production AI security is a system property rather than a feature of one component.
Who it is for. AI engineers, application security architects, cloud security leads, model platform owners and technology risk officers responsible for generative AI deployments. The material assumes working knowledge of application security fundamentals, cloud architecture and LLM application development, and it is written for professionals who must both design and defend the resulting system in front of engineers, auditors and executive stakeholders.
How it is assessed. Learners complete the lab set in this page, then the category assessment of 40 multiple-choice questions. A score of 80 percent or above unlocks the verifiable AIVANA BRAYNOR category certificate, which carries an ID and a public verification link for recruiters and enterprise procurement teams.
Estimated effort. 8-10 weeks of structured study, including approximately 30 to 40 hours of reading, labs and adversarial exercises. Difficulty level: Advanced.
The reference taxonomy for generative AI risk: prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain, sensitive information disclosure, insecure plugin design, excessive agency, overreliance and model theft.
Explicit separation between untrusted content (user input, retrieved documents, tool output) and trusted instructions, enforced by architecture rather than prompt wording.
The failure mode where model output is passed into a browser, shell, SQL client or tool without validation, turning a text generator into an injection vector.
Giving a model or agent more permissions, tools and autonomy than its task requires, so an injected instruction becomes a privileged action.
Leakage of secrets, personal data or proprietary content through prompts, retrieval context, logs, embeddings or model memorisation.
Resource exhaustion driven by oversized prompts, recursive tool loops or adversarial inputs, expressed financially as inference spend spikes.
A production LLM application is a pipeline, not a model call. The request path typically includes identity resolution, rate limiting, prompt template selection, context assembly (system instructions, conversation history, retrieved documents, user input), optional model routing, output post-processing, citation binding and downstream action execution. Each of these stages is a trust boundary, and the security question at every boundary is the same: what data crosses here, and what is allowed to influence what.
The critical architectural principle is that untrusted content must never be positioned where the model can interpret it as instruction. In practice this means delimiters and structural separation (content passed as data, not concatenated into the instruction block), explicit labelling of provenance, and — most importantly — controls outside the model. Models obey precedence only probabilistically; applications must enforce it.
Consider the difference between a chat assistant and a document workflow. In a chat assistant the attacker is usually the user, so the impact is limited to what that user is already authorised to see. In a document workflow the attacker can be a third party who placed text in a PDF, an email, a web page or a ticket comment. That content reaches the model with the privileges of the assisting user, which is the definition of an indirect injection risk.
Output handling deserves equal architectural attention. Model output is untrusted data. If it is rendered as HTML, executed as code, interpolated into SQL or passed as a shell argument, the model becomes an injection engine. Teams that separate "generate text" from "perform action" through a typed, validated action interface remove an entire class of failure.
Finally, memory and state make reasoning about any single request insufficient. Conversation history, cached summaries, long-term memory and shared context can carry a payload across sessions and users. Where memory exists, it must be scoped to a principal, filtered on write, and expired on a schedule.
Most reported generative AI incidents are not exotic jailbreaks. They are disclosures produced by mundane design shortcuts: an embedding index built from a shared document store with no entitlement filter, a debug mode that echoes the full prompt into an error message, a conversation transcript stored without access control, or an assistant that summarises any file URL it is given.
Retrieval is the highest-risk component. If the vector index is global and retrieval filters are applied after the search, the model may already have seen the restricted passage, and even a refusal leaks its existence. The correct design applies the caller's entitlements as a mandatory filter inside the search query, using the identity of the end user rather than the identity of the service.
Secrets require separate treatment. API keys, connection strings and internal hostnames should never be reachable by the model, because a model that can see a secret can be persuaded to reveal it. Keep secrets in a broker accessible only to the tool runtime, inject them at execution time, and never place them in prompts, logs or evaluation datasets.
Personal data introduces legal obligations. Prompt and response logging is valuable for forensics but becomes a processing activity under DPDP and GDPR, requiring purpose limitation, retention limits and redaction. A pragmatic pattern is to store structured metadata and hashed content by default, with full transcripts retained only for a short window or for flagged sessions.
Model memorisation is the residual risk that cannot be engineered away completely. If sensitive content was in training data, it may surface in outputs. The control is upstream: data curation, deduplication and contractual constraints on provider training, plus periodic extraction testing against representative sensitive patterns.
The moment a model can call tools, the security posture is determined by the union of every tool's permissions. Teams routinely grant a single service credential to the whole assistant, which means an injected instruction in a retrieved document inherits write access to the CRM, the ticketing system and the email account. The model did not escalate privilege; the architecture did.
Least agency is the discipline of minimising four dimensions: which tools are available in a given context, what each tool can do, how many steps the agent may take, and whose authority it exercises. A support assistant answering a product question needs a read-only search tool and nothing else. A scheduling assistant needs calendar free/busy, not full mailbox access.
Irreversible actions require gates. Payments, deletions, external communications, privilege changes and production modifications should require explicit human confirmation, with the confirmation showing the exact action and arguments the agent intends to take. The gate must be enforced by the tool runtime, not requested in the prompt.
Delegation matters for accountability. When an agent acts on behalf of a user, the tool should receive a token that identifies both the agent and the user, with the intersection of their permissions. Without this, audit trails record "the assistant did it" — an answer that fails every serious investigation.
Finally, instrument the action boundary. Every tool call should emit a structured event with principal, tool, arguments hash, decision, latency and result code. This single control converts agents from opaque actors into auditable systems, and it is the foundation of the incident-response playbooks covered later in this category.
LLM and Generative AI Security is best defended by understanding exactly how it fails. The attacks below are not theoretical constructs: each has been demonstrated against production systems, and each leaves a detection signal that a mature team can instrument. For every technique, the security question is not only "can it happen" but "what is the cheapest observable evidence that it is happening now".
Indirect prompt injection via retrieved content. An attacker places instructions inside a document, web page, email or ticket that the assistant later ingests, causing the model to follow attacker intent with the victim user's privileges. The observable signal is: Retrieved chunks containing imperative phrasing, unexpected tool invocations immediately after retrieval, or answers that reference instructions never issued by the user. The primary countermeasure is: Treat retrieved text as data with explicit provenance labels; restrict tool availability during retrieval-grounded turns; require approval for any action triggered after ingestion.
Insecure output handling to a downstream sink. Model output containing HTML, SQL fragments, shell commands or URLs is rendered or executed without validation, producing XSS, SQL injection, SSRF or command execution. The observable signal is: Output containing markup or query syntax in a field that should be plain text; outbound requests to unexpected destinations shortly after a generation. The primary countermeasure is: Type and validate all output before it reaches a sink; render as text by default; enforce an egress allowlist for any URL the model can influence.
Sensitive information disclosure through retrieval or logs. Missing entitlement filters, over-broad indexes, verbose error messages or unprotected transcript stores expose restricted content to an unauthorised user. The observable signal is: Cross-tenant or cross-department retrieval results; prompt content appearing in application errors; access to transcript endpoints without authorisation. The primary countermeasure is: Authorisation filters inside the search query; redacted logging with retention limits; deny-by-default access to transcript and debug endpoints.
Excessive agency exploitation. An injected instruction drives an over-privileged tool — sending email, updating records, deleting data or moving money — because the assistant holds broad standing credentials. The observable signal is: Tool calls with unusual arguments, bursts of actions per session, or actions on resources outside the user's normal scope. The primary countermeasure is: Least agency, per-user delegated tokens, action allowlists and human approval gates for irreversible operations.
Model denial of service and cost exhaustion. Oversized prompts, recursive tool loops or adversarial inputs saturate inference capacity or inflate spend, degrading service or causing direct financial loss. The observable signal is: Token-per-request outliers, repeated identical calls, tool recursion depth anomalies and spend velocity above baseline. The primary countermeasure is: Per-user and per-tenant quotas, input length limits, recursion caps, spend alerts and model routing that degrades gracefully under load.
Training data and supply chain poisoning. Tampered fine-tuning data, unpinned model artefacts or a compromised provider update changes model behaviour in ways that bypass existing guardrails. The observable signal is: Behavioural drift on golden prompts after a model or adapter update; unverified artefact hashes; unexpected system-prompt content. The primary countermeasure is: Signed artefacts with verification at load, pinned model versions, golden-prompt regression suites and post-update behavioural comparison.
Because AI attacks rarely arrive one at a time, threat models must describe chains rather than isolated techniques. A reconnaissance step (enumerating endpoints and model behaviour) enables an initial access attempt (a crafted input or poisoned artefact), which enables privilege escalation through excessive agency, which enables impact such as data exfiltration or a fraudulent transaction. Teams that model the chain, and place a control at each link, degrade an adversary's success probability far more effectively than hardening a single step.
Detection design should assume partial failure. Every control in this course is evaluated on the assumption that some percentage of attempts bypass it, which is why least privilege, approval gates, egress restriction and immutable audit trails appear repeatedly. The goal of an AI security programme is not a system that cannot be attacked; it is a system where a successful attack is bounded, observed, attributed and recoverable.
A defensible llm and generative ai security posture layers preventive, detective and corrective controls so that no single component carries the entire trust burden. In practice this means securing four planes: the model and artefact plane (weights, adapters, prompts, versions), the data plane (training data, retrieval corpora, logs), the control plane (identity, policy, configuration, secrets) and the action plane (tools, APIs, outbound integrations). Attacks cross planes, so controls must be designed at the crossing points.
Authenticated inference gateway. A single internal entry point for model calls that enforces identity, quota, policy, logging and provider abstraction, and that is the only network path to a provider. This control maps to NIST AI RMF GOVERN 1.2 · ISO/IEC 27001 A.8.20 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Prompt and context segregation. Structural separation of system instructions, developer configuration, user content and retrieved data, with provenance labels and no string concatenation of untrusted text into instructions. This control maps to OWASP LLM01 · NIST AI 600-1 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Output type enforcement. Validating and typing model output before it reaches any sink, with text rendering by default and allowlisted structured schemas for actions. This control maps to OWASP LLM02 · OWASP ASVS 5.3 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Retrieval authorisation. Enforcing end-user entitlements inside the vector search itself so restricted passages never enter the context window. This control maps to OWASP LLM06 · DPDP Act 2023 s.8 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Least agency and approval gates. Minimising tools, permissions and steps, and requiring human confirmation, enforced at the runtime, for irreversible actions. This control maps to OWASP LLM08 · ISO/IEC 42001 A.7 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Decision telemetry and retention. Structured, tamper-evident logs of prompts metadata, retrieval decisions, guardrail verdicts, tool calls and outcomes with redaction and bounded retention. This control maps to EU AI Act Art.12 · ISO/IEC 42001 A.8 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Adversarial regression suite. A maintained corpus of known attacks executed on every release, with pass-rate thresholds that block promotion when regressions appear. This control maps to NIST AI 100-2 · OWASP LLM Top 10 (2025) and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Quota and cost governance. Per-principal rate limits, token ceilings, recursion caps and spend alerts that bound the financial impact of abuse. This control maps to ISO/IEC 27001 A.8.6 · NIST AI RMF MANAGE 2.3 and should be evidenced rather than merely asserted: policy document, configuration export, test result and monitoring screenshot constitute the minimum acceptable evidence set during an audit.
Reference architectures should make the secure path the default path. Wrap model access behind an internal gateway that enforces authentication, quota, policy evaluation and logging; make the raw provider endpoint unreachable from application subnets; issue short-lived scoped credentials from a secret manager rather than embedding keys; and separate the orchestrator's identity from the tool's identity so that authorisation can be evaluated per hop. When the platform enforces these properties, individual teams cannot accidentally ship an unprotected integration.
Standards anchor the architecture in language regulators and auditors accept. OWASP Top 10 for LLM Applications (2025), NIST AI Risk Management Framework 1.0 and AI 600-1, ISO/IEC 42001:2023, EU AI Act and India DPDP Act 2023 provide the vocabulary of governance, risk and assurance. Map each control to at least one standard clause and to one internal policy. Where a standard is silent on an AI-specific issue, record that gap explicitly — auditors increasingly reward documented reasoning over false completeness.
Finally, defence in depth must survive component failure. Define what happens when the guardrail service is unavailable, when the policy engine times out, when the model provider degrades, and when an on-call engineer is the only responder at 03:00. Fail-closed on high-impact actions (payments, deletions, external communications, privilege changes) and fail-open only where the risk of blocking exceeds the risk of allowing, with an explicit record of that decision.
Security claims about llm and generative ai security decay quickly. Models are retrained, prompts change, connectors are added and providers ship silent updates. Assurance therefore has to be continuous, measured and tied to thresholds that trigger action. A control without a metric is an intention; a metric without a threshold is a dashboard nobody acts on.
Adversarial corpus pass rate. Target: >= 95% per release. Measures whether known attacks are still blocked after prompt, model or guardrail changes. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.
Retrieval authorisation violations. Target: 0 per 100,000 queries. A single violation is a data-protection incident; sampling catches filter regressions early. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.
Tool-call approval coverage. Target: 100% of irreversible actions. Proves that no path exists from model output to a destructive action without human confirmation. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.
AI incident mean time to detect. Target: < 30 minutes for leakage events. Detection latency dominates the cost of an AI data incident and the regulatory exposure. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.
Guardrail false-positive rate. Target: < 3% of legitimate requests. Unusable guardrails get switched off; precision protects adoption as well as security. Instrument this metric at the platform layer so it is computed identically across teams, and review it on a fixed cadence with named owners rather than in ad hoc reviews.
Testing should combine four modes. Static review validates configuration, model format, dependency provenance and policy definitions before deployment. Automated adversarial suites replay a maintained corpus of known attacks on every release to catch regressions. Manual red-team exercises explore reasoning failures, multi-turn escalation and cross-component chains that automation misses. Finally, production monitoring measures real behaviour — refusal rates, extraction attempt volumes, tool-call anomalies and cost per user — because production traffic is the only place where the true attack distribution is visible.
Lab — Threat model an LLM assistant. Objective: Produce a complete threat model for a retrieval-backed internal assistant. Enumerate trust boundaries, list assets, model three attack chains, and specify one preventive and one detective control per chain with the evidence each produces. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.
Lab — Execute an indirect injection in a sandbox. Objective: Demonstrate injection through a retrieved document and observe the telemetry. Place an instruction inside a test PDF, retrieve it through the assistant, and capture the prompt assembly, tool call and guardrail verdict logs produced when the instruction executes. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.
Lab — Build an adversarial regression suite. Objective: Convert five known attacks into automated tests with thresholds. Use an open-source harness to run the corpus against the assistant and report pass rates; then change a prompt and demonstrate the regression is caught. Each lab is assessed on evidence produced, not on the answer reached: test transcripts, metrics before and after, and a written residual-risk statement are the deliverables.
Reporting is part of assurance. A finding that cannot be understood by a risk committee will not be funded, and a control that cannot be explained to an auditor will not be trusted. Write findings in business language first — what an attacker gains, which data or operations are affected, how many users or records, what the detection time would be — and only then in technical detail. Attach reproduction steps, but never attach live credentials, personal data or restricted content to a report; use redacted transcripts.
Implementing llm and generative ai security in a large organisation is an operating-model problem as much as a technical one. Three roles must be explicit: an accountable owner for the AI system (usually the business product owner), a control owner for each security capability (platform security, identity, data protection, SOC), and an assurance function that independently verifies claims. Where these roles are ambiguous, controls quietly rot after the launch milestone.
Start with an inventory. You cannot secure models, agents or datasets you have not catalogued, and most enterprises discover on their first red-team engagement that shadow AI usage exceeds sanctioned usage. Build an AI system register capturing purpose, data classification, model and provider, tools and integrations, human oversight model, risk tier and control status. Tie registration to procurement and to cloud project creation so new AI workloads cannot appear outside the register.
Define runbooks before incidents, not during them. At minimum: how to revoke a model or agent credential within minutes; how to disable a single tool or MCP server without taking the product down; how to roll back to a previous model and prompt bundle; how to isolate a poisoned index or dataset; and how to notify customers and regulators. Rehearse each runbook at least twice a year — including one exercise without the engineer who wrote it.
Tooling should reduce toil rather than add dashboards. OWASP LLM Top 10 and LLMGuard, MITRE ATLAS, Promptfoo / Garak / PyRIT, Model provider safety APIs and vendor firewalls, LangSmith / Arize / OpenTelemetry GenAI conventions each solve part of the problem, but integration matters more than breadth: policy decisions must reach the gateway, telemetry must reach the SIEM, and findings must reach the backlog engineers actually work from. A single well-integrated pipeline with clear ownership outperforms five disconnected best-of-breed tools.
Governance closes the loop. Report control coverage, open findings by severity, mean time to remediate, adversarial test pass rates and incidents to the AI risk committee quarterly, and to the board annually in business language. Where a residual risk is accepted — for example a model that cannot yet meet a robustness threshold — record the acceptance, the compensating controls, the owner and the expiry date. Time-bound acceptance is the difference between governance and wishful thinking.
Finally, embed security in delivery. Threat model each new AI feature at design time, add adversarial regression tests to the definition of done, require provenance checks in the deployment pipeline, and give engineers a paved road — approved model gateways, SDKs with guardrails built in, and pre-cleared tool integrations — so the secure path is also the fastest path.
JPMorgan Chase (Banking). The problem: Restricting employee use of public LLMs without an approved alternative drove usage underground. The response: Deployed an internal LLM suite with data-boundary controls, restricted external tool access and monitored usage patterns for sensitive-data egress attempts. The outcome: Brought sanctioned usage into a controlled environment and reduced the risk of customer data entering external providers The transferable lesson: Blocking alone fails; the control must be paired with an approved, usable alternative For teams working on llm and generative ai security, the pattern to copy is the sequence — measure the exposure first, fix the highest-leverage control, then prove the improvement with the same measurement.
Air Canada (Aviation). The problem: A customer-facing chatbot gave incorrect policy information that the airline was held responsible for, exposing a governance gap between model output and authoritative content. The response: Constrained the assistant to authoritative policy retrieval with grounding requirements and human escalation for edge cases. The outcome: Illustrated that overreliance and ungrounded output are commercial liabilities, driving tighter grounding and accountability controls across the industry The transferable lesson: Grounding plus escalation is cheaper than disputing a committed answer in a tribunal For teams working on llm and generative ai security, the pattern to copy is the sequence — measure the exposure first, fix the highest-leverage control, then prove the improvement with the same measurement.
Generative AI is the fastest-adopted enterprise technology in modern IT history, and adoption has outpaced control. Within eighteen months most large organisations moved from a pilot to dozens of production assistants, copilots and document workflows, often built by business units outside the security review process. The result is a large, semi-visible attack surface where prompts, retrieval corpora and tool credentials have never been threat modelled.
The economics of attack have shifted. Historically, discovering an injection flaw required specialist skill. With LLM systems, an attacker can simply converse. Prompt injection payloads are written in natural language, they travel inside documents and emails, and they do not require binary exploitation knowledge. The barrier to entry has fallen while the potential impact — access to internal systems through an over-privileged agent — has risen.
Data leakage is the failure executives understand best. A retrieval-backed assistant that fails to enforce entitlements at query time will happily summarise a colleague's salary review, a legal hold memo or an unreleased financial result. These are not exotic exploits; they are design omissions, and they are the most common cause of real, reported AI incidents.
Cost and availability are security concerns in generative AI. Adversarial prompts that force long outputs, recursive tool loops or expensive model routing can convert a denial-of-service attempt into a direct financial loss. Security teams now monitor inference spend as a control signal, in the same way they monitor failed logins.
Finally, accountability is becoming explicit. Boards are asked to attest that AI risk is managed; auditors ask who approved a model, what data it saw, what it can do and how that is evidenced. Professionals who can answer those questions with documentation, telemetry and test results are scarce, and this scarcity is the career opportunity.
The market has converged on a reference stack: an application layer that assembles prompts and context, a gateway that handles authentication, quota and routing, one or more model providers, a retrieval layer, a tool layer and an observability layer. Security maturity correlates less with which model is used and more with how strictly those layers are separated, authenticated and logged.
Attack tooling has matured faster than defence tooling. Public adversarial prompt corpora, automated jailbreak search, and agent-focused exploitation frameworks are freely available. Defence has responded with classifier-based guardrails, model firewalls and prompt-hardening techniques, but every one of those is probabilistic and must be combined with deterministic controls such as scope limits and approval gates.
Regulatory direction is now clear even where rules are not final. The EU AI Act imposes risk management, logging and human oversight obligations on general-purpose and high-risk systems. India's DPDP Act and sectoral regulators in financial services and healthcare require purpose limitation and data minimisation that directly constrain what can be placed in a prompt. Enterprise procurement has effectively become the fastest enforcement mechanism: security questionnaires now routinely demand OWASP LLM coverage and evidence of red-team testing.
Talent supply lags demand sharply. Organisations report difficulty hiring people who understand both machine learning behaviour and security engineering. Most practitioners arrive from one side and learn the other on the job, which is precisely why structured, adversarial, evidence-producing training like this course shortens the path.
Techniques an adversary applies against this layer, with the observable evidence and the primary countermeasure.
| Technique | Mechanism | Detection signal | Primary control |
|---|---|---|---|
| Indirect prompt injection via retrieved content | An attacker places instructions inside a document, web page, email or ticket that the assistant later ingests, causing the model to follow attacker intent with the victim user's privileges. | Retrieved chunks containing imperative phrasing, unexpected tool invocations immediately after retrieval, or answers that reference instructions never issued by the user. | Treat retrieved text as data with explicit provenance labels; restrict tool availability during retrieval-grounded turns; require approval for any action triggered after ingestion. |
| Insecure output handling to a downstream sink | Model output containing HTML, SQL fragments, shell commands or URLs is rendered or executed without validation, producing XSS, SQL injection, SSRF or command execution. | Output containing markup or query syntax in a field that should be plain text; outbound requests to unexpected destinations shortly after a generation. | Type and validate all output before it reaches a sink; render as text by default; enforce an egress allowlist for any URL the model can influence. |
| Sensitive information disclosure through retrieval or logs | Missing entitlement filters, over-broad indexes, verbose error messages or unprotected transcript stores expose restricted content to an unauthorised user. | Cross-tenant or cross-department retrieval results; prompt content appearing in application errors; access to transcript endpoints without authorisation. | Authorisation filters inside the search query; redacted logging with retention limits; deny-by-default access to transcript and debug endpoints. |
| Excessive agency exploitation | An injected instruction drives an over-privileged tool — sending email, updating records, deleting data or moving money — because the assistant holds broad standing credentials. | Tool calls with unusual arguments, bursts of actions per session, or actions on resources outside the user's normal scope. | Least agency, per-user delegated tokens, action allowlists and human approval gates for irreversible operations. |
| Model denial of service and cost exhaustion | Oversized prompts, recursive tool loops or adversarial inputs saturate inference capacity or inflate spend, degrading service or causing direct financial loss. | Token-per-request outliers, repeated identical calls, tool recursion depth anomalies and spend velocity above baseline. | Per-user and per-tenant quotas, input length limits, recursion caps, spend alerts and model routing that degrades gracefully under load. |
| Training data and supply chain poisoning | Tampered fine-tuning data, unpinned model artefacts or a compromised provider update changes model behaviour in ways that bypass existing guardrails. | Behavioural drift on golden prompts after a model or adapter update; unverified artefact hashes; unexpected system-prompt content. | Signed artefacts with verification at load, pinned model versions, golden-prompt regression suites and post-update behavioural comparison. |
Source: Synthesised from NIST AI 100-2, OWASP Top 10 for LLM Applications (2025) and MITRE ATLAS
Preventive, detective and corrective controls with the standard each one evidences.
| Control | What it does | Standard evidence |
|---|---|---|
| Authenticated inference gateway | A single internal entry point for model calls that enforces identity, quota, policy, logging and provider abstraction, and that is the only network path to a provider. | NIST AI RMF GOVERN 1.2 · ISO/IEC 27001 A.8.20 |
| Prompt and context segregation | Structural separation of system instructions, developer configuration, user content and retrieved data, with provenance labels and no string concatenation of untrusted text into instructions. | OWASP LLM01 · NIST AI 600-1 |
| Output type enforcement | Validating and typing model output before it reaches any sink, with text rendering by default and allowlisted structured schemas for actions. | OWASP LLM02 · OWASP ASVS 5.3 |
| Retrieval authorisation | Enforcing end-user entitlements inside the vector search itself so restricted passages never enter the context window. | OWASP LLM06 · DPDP Act 2023 s.8 |
| Least agency and approval gates | Minimising tools, permissions and steps, and requiring human confirmation, enforced at the runtime, for irreversible actions. | OWASP LLM08 · ISO/IEC 42001 A.7 |
| Decision telemetry and retention | Structured, tamper-evident logs of prompts metadata, retrieval decisions, guardrail verdicts, tool calls and outcomes with redaction and bounded retention. | EU AI Act Art.12 · ISO/IEC 42001 A.8 |
| Adversarial regression suite | A maintained corpus of known attacks executed on every release, with pass-rate thresholds that block promotion when regressions appear. | NIST AI 100-2 · OWASP LLM Top 10 (2025) |
| Quota and cost governance | Per-principal rate limits, token ceilings, recursion caps and spend alerts that bound the financial impact of abuse. | ISO/IEC 27001 A.8.6 · NIST AI RMF MANAGE 2.3 |
Source: ISO/IEC 42001, NIST AI RMF 1.0, OWASP ASVS and vendor security documentation
Measurable targets that make the controls auditable rather than aspirational.
| Metric | Target | Why it matters |
|---|---|---|
| Adversarial corpus pass rate | >= 95% per release | Measures whether known attacks are still blocked after prompt, model or guardrail changes. |
| Retrieval authorisation violations | 0 per 100,000 queries | A single violation is a data-protection incident; sampling catches filter regressions early. |
| Tool-call approval coverage | 100% of irreversible actions | Proves that no path exists from model output to a destructive action without human confirmation. |
| AI incident mean time to detect | < 30 minutes for leakage events | Detection latency dominates the cost of an AI data incident and the regulatory exposure. |
| Guardrail false-positive rate | < 3% of legitimate requests | Unusable guardrails get switched off; precision protects adoption as well as security. |
Source: AIVANA BRAYNOR AI security measurement framework, 2026
Representative platforms and frameworks, their role, and the limitation to plan around.
| Tool or framework | Purpose | Typical use | Limitation |
|---|---|---|---|
| OWASP LLM Top 10 and LLMGuard | Reference risk taxonomy and open-source input/output scanning | Baseline guardrails, evaluation harnesses and security review checklists | Classifier-based detection is probabilistic and evadable; must be layered with architectural controls |
| MITRE ATLAS | Adversary tactics and techniques knowledge base for AI systems | Threat modelling and mapping detections to attacker behaviour | Coverage of emerging agent techniques is still catching up with the field |
| Promptfoo / Garak / PyRIT | Automated adversarial testing and red-team orchestration | Regression suites in CI, systematic jailbreak and injection testing | Coverage is only as good as the corpus; novel multi-turn chains still need manual testers |
| Model provider safety APIs and vendor firewalls | Managed content filtering and prompt-attack detection | First-pass screening of untrusted input and unsafe output | Vendor-dependent thresholds, regional differences and no visibility into the classifier internals |
| LangSmith / Arize / OpenTelemetry GenAI conventions | LLM observability, tracing and evaluation | Decision telemetry, latency and cost attribution, regression comparison | Telemetry often captures sensitive content — redaction must be configured explicitly |
Source: Vendor documentation and open-source project documentation, reviewed October 2026
Publicly reported programmes and the measurable result each achieved.
| Organisation | Sector | Challenge | Result |
|---|---|---|---|
| National Australia Bank | Financial services | Rapid internal adoption of generative assistants created unmanaged prompt, data and tool access across business units. | Consolidated shadow AI usage into a governed platform, enabled audit-ready logging, reduced unmanaged model spend and shortened security review from weeks to days |
| JPMorgan Chase | Banking | Restricting employee use of public LLMs without an approved alternative drove usage underground. | Brought sanctioned usage into a controlled environment and reduced the risk of customer data entering external providers |
| Air Canada | Aviation | A customer-facing chatbot gave incorrect policy information that the airline was held responsible for, exposing a governance gap between model output and authoritative content. | Illustrated that overreliance and ungrounded output are commercial liabilities, driving tighter grounding and accountability controls across the industry |
Source: Public case reports, regulator publications and vendor disclosures
Level, role, expected skills, experience and responsibilities for professionals working in this discipline.
| Level | Role | Skills expected | Experience | Responsibilities |
|---|---|---|---|---|
| Entry | AI Security Analyst | Prompt testing, OWASP LLM literacy, basic Python, log analysis | 0-2 years | Run adversarial test suites, triage findings, maintain the attack corpus and support reviews |
| Intermediate | AI Security Engineer | Threat modelling, guardrail engineering, API and identity security | 2-5 years | Design and implement controls, build regression pipelines, own detection content for AI systems |
| Senior | AI Security Architect | Reference architecture, policy design, risk quantification, standards mapping | 5-10 years | Set the control standard, review high-risk designs, lead red-team programmes and vendor assessments |
| Leadership | Head of AI Security / AI Risk | Programme strategy, regulatory engagement, board communication, operating model | 10+ years | Own the AI security programme, risk acceptance decisions, regulator and board reporting |
Source: AIVANA BRAYNOR AI security career framework, 2026
Rapid internal adoption of generative assistants created unmanaged prompt, data and tool access across business units.
Centralised an internal LLM gateway with identity-bound access, prompt logging, tool scoping and a formal AI risk assessment before any assistant could reach production.
Consolidated shadow AI usage into a governed platform, enabled audit-ready logging, reduced unmanaged model spend and shortened security review from weeks to days
A paved road that is easier than the workaround is the most effective AI security control ever deployed
A single internal entry point for model calls that enforces identity, quota, policy, logging and provider abstraction, and that is the only network path to a provider.
Structural separation of system instructions, developer configuration, user content and retrieved data, with provenance labels and no string concatenation of untrusted text into instructions.
Validating and typing model output before it reaches any sink, with text rendering by default and allowlisted structured schemas for actions.
Enforcing end-user entitlements inside the vector search itself so restricted passages never enter the context window.
Minimising tools, permissions and steps, and requiring human confirmation, enforced at the runtime, for irreversible actions.
Structured, tamper-evident logs of prompts metadata, retrieval decisions, guardrail verdicts, tool calls and outcomes with redaction and bounded retention.
A maintained corpus of known attacks executed on every release, with pass-rate thresholds that block promotion when regressions appear.
Per-principal rate limits, token ceilings, recursion caps and spend alerts that bound the financial impact of abuse.
Treat model, prompt and policy changes as production changes: version them, review them, and gate deployment on adversarial regression results.
Assume at least one control will be bypassed and ensure the resulting impact is limited, observable and reversible.
Payments, deletions, external communications and privilege changes always need an approval gate with a named approver.
Keep transcripts of decisions and tool calls so incidents can be reconstructed months later without guesswork.
Report pass rates against real attacks, detection latency, false-positive cost and mean time to remediate rather than control counts.
A phased path from first inventory to continuous assurance for llm and generative ai security.
Catalogue every model, prompt, dataset, tool and integration in scope. Classify data and autonomy. Produce a threat model that names the adversary, the asset and the attack path for llm and generative ai security, and agree the risk tier with the accountable owner.
Implement the minimum viable control set: authenticated gateway, scoped credentials, least agency, provenance verification and decision logging. Where a control cannot be implemented immediately, record a time-bound acceptance with a compensating control instead of leaving it undocumented.
Run the lab exercises as a first-party adversarial test. Establish pass rates against the maintained attack corpus, measure detection latency and false-positive rate, and convert every bypass into a regression test.
Wire guardrail decisions, tool calls and access anomalies into the SIEM, set thresholds with named owners, publish the metric set, and rehearse the incident runbooks including credential revocation and model rollback.
Re-run adversarial suites on every release, refresh the threat model quarterly, review residual risk acceptance with the risk committee and report control coverage and incident trends to executive leadership.
LLM and Generative AI Security skills are in demand across three hiring markets: AI platform teams inside large enterprises, security engineering functions expanding into AI, and specialist consultancies running AI assurance engagements. Roles are being written now around exactly the capabilities this course builds — threat modelling for generative systems, adversarial testing, guardrail engineering and AI incident response.
The career progression table on this page sets out the route from analyst to leadership, including the skills expected and the responsibilities that distinguish each level. In the Indian market, the same skills support global capability centre roles with international exposure; in North America and Europe they are increasingly written into regulated financial and healthcare programmes.
Roles this opens
Skills you build
The next phase of generative AI security is about bounded autonomy. Models will increasingly act rather than answer, which moves the control problem from content filtering to authority management. Expect policy decision points for AI actions, per-agent identity with cryptographic attestation, and runtime enforcement that looks far more like cloud IAM than like a content moderator.
Detection will shift from prompt patterns to behavioural baselines. Attackers will use legitimate-sounding language, varied phrasing and multi-turn escalation, so signature-style detection will decay. The durable approach is behavioural: what does this user, agent or tenant normally do, and does this action deviate enough to warrant review.
Regulation will converge on evidence. Reporting obligations already require organisations to demonstrate logging, oversight and incident response rather than merely describing intent. Security teams that invest in evidence pipelines now will find compliance becomes a reporting task rather than a remediation crisis.
Finally, expect consolidation of the tooling market around gateways and control planes rather than point products. Organisations are tired of five overlapping scanners and want one enforcement point with a policy language. Professionals who understand both the attacks and the operating model will define those platforms.
The baseline risk taxonomy used across enterprise AI security questionnaires.
Govern, Map, Measure, Manage functions with generative AI specific actions.
Technique-level adversary model for AI systems, used for detection mapping.
Certifiable management system standard for AI governance and control.
Binding obligations for logging, oversight, transparency and post-market monitoring.
Indian personal data obligations that constrain prompt content, logging and retention.
Authoritative taxonomy for evasion, poisoning, extraction and inversion.
Vendor control frameworks useful for benchmarking internal practice.
Practice for LLM and Generative AI Security. These checks provide learning feedback only; certification requires the separate server-scored category assessment.
A model reproduces a customer phone number that appeared in its fine-tuning data. Which OWASP LLM risk class does this represent?
Which control most directly reduces excessive agency in an LLM feature that can issue refunds?
Why can system-prompt instructions never fully prevent prompt injection?
Where must tenant isolation be enforced in a multi-tenant retrieval-augmented assistant?
Improper output handling becomes dangerous because downstream systems typically do what?
What is the main security value of routing all model calls through an internal gateway?
LLM and Generative AI Security is assessed as part of the Advanced AI Security, Adversarial ML & AI Red Teaming category assessment, not in isolation. Work through the practice questions and the labs on this page before sitting the online assessment from the Courses page. A score of 80 percent or above on the 40-question assessment unlocks the verifiable AIVANA BRAYNOR category certificate, issued with a unique certificate ID and a public verification link for recruiters and procurement teams.
Related topics and courses to explore next
Learner outcomes
Reported outcomes from professionals across 170+ categories.