An AI agent that spots suspicious activity can help a security team respond faster. But if the same agent can revoke credentials or isolate a device, a mistaken or manipulated decision can also cause harm. Security therefore concerns both using AI to defend systems and protecting AI systems from misuse. section 8.7 section 8.25
Large language models can be manipulated through prompts. Jailbreaks and prompt injections attempt to bypass restrictions and draw out responses the model would otherwise withhold. Such attacks are one part of a wider misuse problem shaped by attackers’ goals, not just technical weaknesses. Safeguards must extend beyond the model alone. section 5.1
AI agents can support several stages of cybersecurity work:
- Detection and investigation: summarize threat information, prioritize alerts, correlate events, and flag activity for human review. section 8.7
- Testing: follow multistep penetration-testing procedures to find weaknesses before malicious actors do. section 8.7
- Response: carry out predefined containment steps, such as revoking credentials or blocking malicious traffic. section 8.7
More autonomy makes oversight more important. Sandboxed tool access, behavioral monitoring, user training, and policy guardrails can help limit an agent’s actions and their side effects. Human analysts still need to check outputs, manage false positives, and judge whether a response is appropriate. section 8.25 section 8.7
Security also depends on governance. Privacy, data protection, supervision, vulnerability reporting, and cybersecurity safeguards appear among proposed responsibilities for AI developers. Voluntary commitments may signal priorities, but vague requirements without enforcement leave open how reliably they will be met. section 10.33 section 10.29
