Raw data can be irrelevant, sensitive, or difficult to search. Preprocessing turns it into data suited to testing, training, or retrieval. section 4.4 section 6.3.2
A practical workflow may include these steps:
- For test data, sort utterances into usable and irrelevant groups, scrub personal information when needed, annotate them, and convert them into sets for a testing tool. section 4.4
- For RAG, convert documents such as PDFs or tables into processable text, add metadata if useful, divide content into coherent chunks, generate embeddings, and store them in a vector database. section 6.3.2
Figure 4.7 Once you obtain some data, each utterance should first be sorted into buckets to identify potential candidates; this will separate the good, usable test data from the bad or irrelevant user inputs. The data may also need to be scrubbed to fix problems like personal identifiable information (PII). After that, the data will need to be annotated (for classifiers, it will need to be labeled with the correct intent; for generative AI, it will need to be associated with an ideal output response). Finally, the data will need to be converted into one or more sets that can be consumed by your testing tool.
Shows how utterances are filtered, scrubbed, annotated, and converted into test sets. section 4.4
