A data scientist translates a business problem into a machine learning problem and develops a solution against production data rather than standard benchmarks. The role connects product needs, data preparation, model experimentation, engineering, and operational monitoring. section 1.1.2
Typical responsibilities include:
- Explore available data, determine what is useful, and decide how to preprocess it for training. section 1.1.2
- Prototype alternative approaches, combine neural-network architectures or research techniques, and apply additional machine learning methods when appropriate. section 1.1.2
- Convert experimental code into production code, automate workflows, and follow the engineering process to ship models. section 1.1.2
- Monitor data and model performance in production, iterate on additional data needs, and troubleshoot model degradation. section 1.1.2
Training an intent-classification model
A data scientist retrieves a training dataset through a dataset-fetching API and trains an intent-classification model, typically using PyTorch or TensorFlow. A data engineer supplies and labels data through ingestion interfaces. section 2.2.2
Figure 2.6 System overview of the sample dataset management service. The sample service contains three main components, a data ingestion API, internal storage, and a dataset fetching API.
A dataset management architecture shows how data ingestion, internal storage, and dataset fetching support model training and separate engineering responsibilities. section 2.2.2
