A data ingestion API is the write-facing interface of a dataset management service. It lets data engineers create datasets, append collected and labeled data, and query dataset summaries and status before downstream training retrieves the data. section 2.2.2
A reference ingestion layer exposes four operations:
- Create a dataset from metadata, a dataset type, tags, and a data location. section 2.2.3
- Update an existing dataset by adding new data as a commit. section 2.2.3
- Retrieve one dataset’s summary and history. section 2.2.3
- List summaries for all datasets. section 2.2.3
Instead of streaming files through the API, clients provide downloadable data URLs, such as cloud object-storage links. The service downloads the data, selects a dataset transformer, parses and converts it, stores the result, and returns a summary. This reduces file-transfer bandwidth and avoids the complexity of highly available large-file streaming. section 2.2.3
Creating a dataset
A typical flow is: upload data to object storage; obtain its URL; call CreateDataset with the URL, name, description, type, and tags; let the service transform and store the data; then receive the dataset ID, commit history, and statistics. Subsequent updates create another commit rather than replacing the existing one. section 2.2.3
Figure 2.9 A high-level overview of the seven steps to creating a new dataset: (1) upload data to the cloud object storage; (2) get a data link; (3) call createDataset API with a data link as the payload; (4) DM first downloads data from the data link and then finds the right dataset transformer (IntentTextTransformer) to do data parsing and conversion; (5) DM saves the transformed data; and (6 and 7) DM returns the dataset summary (ID, commit history, data statistics) to the user.
Follow the end-to-end creation workflow from object-storage upload through transformation, persistence, and the returned dataset summary. section 2.2.3
