A grocery store collects more purchase records than staff can inspect one by one. Data mining uses algorithms to find useful patterns, relationships, anomalies, and rules in large datasets. Those findings can inform decisions about stocking fresh produce, arranging products, setting prices, and offering promotions. The goal is not simply to collect more data, but to turn it into insights people can use. section 3.1
Different questions call for different techniques. Classification assigns records to categories; prediction estimates outcomes; clustering finds groups; and market basket analysis looks for products bought together. These approaches draw on statistical, probabilistic, and mathematical methods. They offer a way to search datasets too large for a person to analyze unaided. section 3.1
From receipts to a purchasing pattern
Suppose one million receipts include 20,000 purchases of bread and cheese, and 5,000 purchases of bread, cheese, and olives. Among the receipts containing bread and cheese, one in four also contains olives. Association analysis can identify that pattern; deciding whether to change a store layout still calls for a business decision. section 3.2 section 3.1
For fraud prevention, a decision tree can use transaction attributes to classify a purchase. Useful inputs may also be created from the records, such as the number of transactions in the previous 15 minutes or hour. A tree’s logic can sometimes be expressed as if-then-else rules, making its decisions easier to inspect, although a full tree may contain thousands of nodes. section 3.2
Figure 3.1 The first few levels of a decision tree. In a real-world application, a full tree may contain thousands of nodes.
The first few levels show how a decision tree branches; a deployed tree may be much larger. section 3.2
A data-mining project is iterative rather than a single algorithm run:
- Define the application goal, gather data, and integrate records from different sources. Labeling data can be costly. section 3.2
- Clean inconsistencies and consider new attributes that could make the data more useful. section 3.2
- Separate training and testing data, select an algorithm, build and evaluate the model, and check its scalability and resilience. section 3.2
- Repeat as needed, then deploy the model and integrate it into operations. section 3.2
Patterns alone do not make a complete AI system. In a proposed fraud-prevention approach, data mining identifies signs of suspicious behavior while techniques such as case-based reasoning and rule-based systems contribute other capabilities. Choose techniques for the question at hand, test their performance, and account for the work required to prepare the data. section 11.7 section 3.2
