AI Security

AI systems bring new kinds of security risk. A chatbot can be manipulated through carefully crafted text. A model can learn the wrong behavior if attackers tamper with its training data. An AI agent with access to email and files can be tricked into sending confidential information outside the organization. A model's weights, prompts and data can be stolen. At the same time, AI systems run on traditional infrastructure that must still be secured. AI security covers all of these threats, combining established cybersecurity practices with defenses designed for how AI systems work. It applies to AI the organization builds and to AI it buys.

The field has developed quickly. The OWASP Top 10 for Large Language Model Applications, first published in 2023, lists the most critical security risks for applications built on language models. MITRE ATLAS catalogs real-world tactics used to attack AI systems, and NIST has published guidance on adversarial machine learning and a draft profile applying its Cybersecurity Framework to AI. For security, AI, data and risk teams, AI security is now a core part of protecting the organization.

Definition of AI Security

AI security is the set of practices, controls and technologies that protect the confidentiality, integrity and availability of AI systems and the data they use, and that prevent AI systems from being manipulated or misused to cause harm, across the AI lifecycle from data collection and training to deployment and use.

AI security is a core part of AI TRiSM, defends against threats such as prompt injection and builds on the NIST Cybersecurity Framework.

Common AI Security Threats

Prompt injection manipulates a model through crafted inputs, either typed directly or hidden in content the model reads, so it ignores its instructions.

Sensitive information disclosure occurs when models reveal confidential data, personal information or their own hidden instructions.

Data and model poisoning corrupt training or fine-tuning data so that a model learns harmful or hidden behaviors. Poisoned behavior may only appear when a specific trigger is present, making it hard to detect.

Supply chain risks arise from third-party models, datasets, libraries and plugins that may be compromised or poorly secured. Downloaded models can even contain code that runs when they are loaded.

Excessive agency occurs when AI agents have more permissions than they need, so manipulation can lead to damaging actions. Limiting permissions is one of the most effective defenses.

Model theft and extraction involve copying a model's weights or reconstructing its behavior through repeated queries. Valuable proprietary models deserve the same protection as other intellectual property.

Denial of service and cost attacks overload AI systems or drive up usage costs. Rate limits and quotas help contain both.

AI Security and Traditional Security

Traditional controls still apply, including identity and access management, network security, encryption, logging and vulnerability management. Many AI incidents begin with basic security failures, such as exposed credentials.

AI adds new attack surfaces, because natural language inputs can carry instructions and models behave probabilistically rather than following fixed rules.

Many AI attacks cannot be fully prevented by patches, so defense relies on layers of controls, limited permissions and monitoring. The goal is to limit what an attacker can achieve even if one control fails.

AI security teams therefore combine security engineering, machine learning expertise and an understanding of how AI is used in the business.

Importance of AI Security in AI Governance

AI security protects data. It reduces the risk that AI systems expose confidential, personal or regulated information. This matters most for assistants connected to internal documents and systems.

It protects operations. Securing AI agents and applications prevents attackers from using them to take actions, move through systems or disrupt services. Agents with broad access are particularly attractive targets.

It also supports compliance and trust. Security is required for high-risk AI under the EU AI Act and is a central concern of customers evaluating AI vendors.

AI Security and Financial Management

AI-related data breaches carry the same costs as other breaches, including investigation, notification, fines, legal claims and lost business.

Cost attacks and runaway agents can drive up AI usage bills, so rate limits and spending caps protect budgets as well as systems.

Security reviews of AI vendors and models help avoid costly dependence on insecure products.

Investment should be focused on AI systems that handle sensitive data, connect to critical systems or act autonomously.

Common Challenges

AI threats evolve quickly, and defenses that work today may be bypassed tomorrow. Regular testing and threat intelligence help keep defenses current.

Security teams may lack AI expertise, while AI teams may lack security experience. Joint reviews and training help close the gap.

AI features are often added to existing software by vendors, sometimes without the security team's knowledge. Vendor reviews and contract terms should require notice of new AI features.