A content team wants AI drafts to include client cost reductions from internal case studies. Today, staff search Confluence, Salesforce, and Google Drive by hand, then add those details to generic drafts. Giving an AI system access to that knowledge could save time and make content more useful—but access to proprietary information also creates a responsibility to control where it goes and who can see it. section 7.1.1
Sensitive information includes confidential customer data, proprietary business knowledge, and personal information that should not be available for unrestricted use. It can enter an AI system through retrieved documents, training data, user inputs, or production workflows. The question is not only whether a model can use it, but whether that use is authorized and protected. section 7.1.1 section 11.2.1 section 11.1.1
Before connecting sensitive information to an AI feature, review its path through the system:
- Classify data by confidentiality and map where it moves during training, evaluation, and production use. section 11.1.1
- Check a model’s documented training-data composition and a provider’s retention policy; documentation may be incomplete or vague. section 11.2.1 section 5.1.1
- Limit what is sent to models and who can interact with them. For third-party LLMs, encrypt data in transit and at rest, and audit provider privacy commitments. section 11.2.1
When a useful feature becomes a disclosure
A recommendation system included confidential enterprise-client data in its model-training process. A review of prompt logs showed that malicious users had extracted some of it; harmless users might also have received it. The failure was not simply poor model performance: sensitive inputs had been allowed to flow through the system without adequate restrictions. section 11.1.1
Privacy is also a product-trust issue. A 2023 ChatGPT bug exposed some users’ chat histories and billing details to others, illustrating how failures outside the model itself can disclose information. Regulatory duties vary by data and jurisdiction, while customer confidence depends on people believing their information is handled securely and with appropriate control. section 11.2.3
