A recommendation feature can expose customer information even when the team’s goal is simply to make better recommendations. Security protects an AI system’s confidentiality, integrity, and availability. It must address threats to data, models, and production usage—not just familiar cybersecurity risks. AI adds particular difficulty: poisoned training data or harmful prompt instructions can corrupt outputs, while unstructured results and third-party components can make attacks harder to detect or control. section 11.1 section 11.1.1
Use the three-level view to check where security needs attention: data, intelligence, and user experience. section 11.1
When confidential data reaches a recommendation feature
At DataMax, malicious users exfiltrated proprietary information through an AI recommendation feature. The team discovered the problem in prompt logs only after it had gone unnoticed for weeks; information might also have reached users who had no malicious intent. The incident exposed a basic governance gap: sensitive inputs had not been adequately restricted. Exfiltration threatened confidentiality, while data poisoning was a separate threat to integrity. section 11.1.1
A practical response starts with the data’s path through the product:
- Classify data by confidentiality, then trace its movement through training, evaluation, and production use. section 11.1.1
- Remove or anonymize confidential data where it need not flow; use encryption and role-based access controls where authorized access remains necessary. section 11.1.1
- Monitor continuously, arrange regular third-party audits, and train the team on a policy covering privacy, compliance, and accountability. section 11.1.1
- Validate training data and vet third-party libraries and software for threats to data and intellectual property. section 11.6
Security is also a delivery decision. Internal AI tools may handle personally identifiable information, so product teams need IT and compliance partners to secure data pipelines. Treating governance as an early design concern helps protect customers and preserve trust rather than waiting for an incident. section 2.1.4 section 11.0
