Temario del curso
Module 1 — How AI Apps Break
Lab: none — architecture walkthrough & discussion
A builder’s mental model of the attack surface.
Topics:
- LLM, RAG, and agent architectures from the developer’s side
- the request/response lifecycle of an AI feature
- prompt flow: system, developer, user, and tool messages
- where untrusted data enters (and re-enters) the model
- the trust boundaries a developer owns vs. inherits
- why AI attacks are semantic, not syntactic
- mapping the OWASP LLM Top 10 to code you write
Key insight: Every place untrusted text reaches the model — or model output reaches your code — is a boundary you own.
Module 2 — Prompt Injection for Builders
Lab: Lab 01 — 01-Prompt-Injection
The “SQL injection moment” for AI — but you can’t fully escape it.
Topics:
- direct vs. indirect prompt injection
- hidden instructions in documents, web pages, tool output
- jailbreaks and role-confusion
- why instruction/data separation matters
- defensive prompt design (delimiters, structure, minimal authority)
- why prevention is partial — design for containment
Hands-on:
- attack your own chatbot
- bypass a naïve filter
- restructure the prompt to shrink the blast radius
Module 3 — Treating Model Output as Untrusted
Lab: Lab 02 — 02-Output-Handling
The bug class developers underestimate most.
Topics:
- model output as untrusted input to the rest of the app
- insecure output handling (LLM02): XSS, SSRF, command/SQL injection downstream
- never eval/exec/render raw model output
- structured outputs and schema validation
- output encoding and allowlists
- safe rendering in web/UI contexts
Hands-on:
- find and fix an insecure-output-handling vulnerability
- enforce a JSON schema on model responses
Module 4 — RAG Security
Lab: Lab 03 — 03-RAG-Security
One of the biggest new attack surfaces — and it’s yours to build.
Topics:
- vector DB and retrieval threats
- ingestion sanitization
- document provenance and trust scoring
- retrieval scoping and metadata isolation
- hidden instructions in retrieved content (indirect injection)
- data exfiltration via retrieval
Hands-on: - poison a RAG pipeline with a malicious document - add ingestion sanitization and retrieval scoping to defend it
Module 5 — Agent & Tool Safety
Lab: Lab 04 — 04-Agent-Safety
Where a bug becomes an action.
Topics:
- excessive agency (LLM06) and tool abuse
- least privilege for agents
- tool allowlists and argument validation
- approval gates and human-in-the-loop
- sandboxing tool execution
- scoped, short-lived credentials for agents
- limiting autonomous loops and chaining
Hands-on:
- lock down an over-permissioned agent
- add an allowlist + approval gate to a dangerous tool
Module 6 — Secrets, Identity & Cost
Lab: Lab 05 — 05-Secrets-and-Cost
The operational mistakes that hurt fastest.
Topics:
- API key and secret management (never in prompts, code, or logs)
- per-user authentication and authorization for AI features
- propagating user identity to tools and retrieval
- denial-of-wallet: unbounded token/cost consumption
- rate limits, token budgets, and timeouts
- logging without leaking secrets or PII
Hands-on:
- get secrets out of the prompt/code path
- add per-user rate limits and a token/cost budget
Module 7 — Guardrail Libraries
Lab: Lab 06 — 06-Guardrails
Buy vs. build for input/output safety.
Topics:
- what guardrail frameworks do (and don’t)
- input guardrails: injection/PII/topic classifiers
- output guardrails: validation, filtering, grounding checks
- when a guardrail is appropriate vs. your own deterministic check
- layering guardrails with the controls from earlier modules
- performance, false positives, and failure modes
Hands-on:
- add an input/output guardrail layer to an AI feature
- measure what it catches and what it misses
Module 8 — Red-Teaming Your Own App
Lab: Lab 07 — 07-Red-Teaming
Ship it like an attacker already has it.
Topics:
- building an abuse/test suite for AI features
- automated prompt-injection and jailbreak tests
- regression-testing guardrails and policies
- running AI security checks in CI
- model and dependency supply chain (provenance, pinning)
- a pre-ship security checklist for AI features
Hands-on:
- write automated red-team tests for an AI feature
- wire them into a CI check
Module 9 — Scoring AI Security: The SAIS-100 Framework
Lab: none — scoring exercise (uses the Capstone app)
Turn everything you’ve built into a repeatable score.
Topics:
- the AI Security Hexagon: six questions instead of “is it secure?”
- the six scored categories (Data, Prompt, Agent, Supply Chain, Detection, Governance)
- the 100-point rubric and its weightings
- verdict bands and the single-category override rule
- the Elephant Scale Secure AI Score (SAIS-100) as a branded, re-runnable framework
- scoring before/after hardening as a metric
Hands-on:
- score the Capstone app on the 100-point scale
- name the single change that most raises the score
Key insight: The three highest-weighted categories map to the trust boundaries a developer owns — so the score measures exactly what this course taught.
Capstone
Students harden a deliberately vulnerable AI application end-to-end.
The starter app contains:
- an injectable prompt
- insecure output handling
- an unscoped RAG pipeline
- an over-permissioned agent
- secrets in the prompt path
- no cost limits
Students apply the course:
- restructure prompts for containment
- validate and encode model output
- sanitize and scope retrieval
- apply least privilege and approval gates to the agent
- move secrets out and add cost/rate limits
- add guardrails and automated red-team tests
Deliverable: a hardened app plus a short OWASP LLM Top 10 self-assessment.
Module - Lab map
Labs run in lab order, which follows module order. The course has 9 modules and 7 labs: Module 1 is an architecture walkthrough/discussion and Module 9 is a scoring exercise, so neither has its own lab folder.
- Lab 01 - 01-Prompt-Injection: Attack your chatbot & design for containment (Module 2)
- Lab 02 - 02-Output-Handling: Fix an insecure-output-handling bug (Module 3)
- Lab 03 - 03-RAG-Security: Poison then defend a RAG pipeline (Module 4)
- Lab 04 - 04-Agent-Safety: Lock down an over-permissioned agent (Module 5)
- Lab 05 - 05-Secrets-and-Cost: Secure keys + add cost guardrails (Module 6)
- Lab 06 - 06-Guardrails: Add an input/output guardrail layer (Module 7)
- Lab 07 - 07-Red-Teaming: Automated red-team tests in CI (Module 8)
Module 1 (How AI Apps Break) has no lab — it runs as an architecture walkthrough and discussion. Module 9 (Scoring AI Security) has no lab folder — it runs as a scoring exercise against the Capstone app.
Requerimientos
- Nivel de habilidad: Intermedio.
- Los estudiantes deben estar cómodos con: construir y consumir APIs REST, un lenguaje de scripting (las prácticas usan Python), autenticación básica de aplicaciones, git y la línea de comandos (CLI).
- No se requiere conocimiento previo de aprendizaje automático (machine learning); este es un curso de seguridad de aplicación para personas que construyen con LLMs, no las que los entrenan.
Público Objetivo
- Ingenieros de software / backend que construyen funciones con LLM
- Desarrolladores full-stack y de API
- Ingenieros de aplicaciones de IA/ML
- Ingenieros de plataforma que entregan copilotos y agentes
- Líderes técnicos e ingenieros senior responsables de funciones de IA
Testimonios (2)
Realmente disfruté aprender sobre los ataques de IA y las herramientas disponibles para comenzar a practicar y utilizarlas activamente en pruebas de seguridad. Adquirí muchos conocimientos que no tenía al inicio, y el curso cumplió con lo que esperaba. Mi parte favorita del entrenamiento fue el navegador Comet, y quedé impresionado por lo que podía hacer. Sin duda seguiré explorándolo más. En general, fue un excelente curso y disfruté aprender sobre los Top 10 de OWASP para GenAI.
Patrick Collins - Optum
Curso - OWASP GenAI Security
Traducción Automática
El conocimiento profesional y la forma en que lo presentó ante nosotros
Miroslav Nachev - PUBLIC COURSE
Curso - Cybersecurity in AI Systems
Traducción Automática
