Chapter 4: LLM-driven data science
The introduction explains how LLMs can turn business questions into data-analysis notebooks while emphasizing that plausible-looking results still require human verification.
After this chapter you will be able to
- LLM-assisted analysis workflow. Describe how answering a business question starts with choosing a suitable dataset and preparing it through cleaning, parsing, and normalization. Explain that an LLM can translate the question into a runnable notebook, letting a person with basic programming knowledge focus on the data and business domain rather than Python.
- Plausible results and verification. Explain why errors in LLM-assisted data science can be harder to notice than errors in ordinary coding, and describe the resulting trust-but-verify approach.
- Dataset suitability screening. Explain how the section screens candidate datasets for download availability, record count, language, and suitability for the planned analysis, rather than accepting an LLM recommendation without review.
- Real versus synthetic data. Justify the choice of the Indian Medicine Dataset over the Kaggle dataset by contrasting real product data with easier-to-use synthetic data, including the consequences for analysis and model results.
- Isolated Conda environment. Explain why the chapter uses Conda before running the analysis code: it manages packages in an isolated environment so installed libraries do not interfere with other projects.
- Activating and launching Jupyter. Describe how to enter the prepared environment and open Jupyter: activate `drug_analysis`, then run `jupyter notebook` from the `manning_vibe_engineering` directory to open its interface in a browser.
- LLM-guided exploratory analysis. Explain how to start analyzing the unfamiliar medicine dataset without specifying questions in advance: ask the LLM to suggest analyses and create a Jupyter notebook, then inspect the results.
- Question Driven Notebook Prompt. Describe how to turn specific market questions into a prompt for an LLM-generated Jupyter notebook, including the dataset location and the requested analyses.
- Focused Versus Open Analysis. Distinguish asking AI to explore a dataset freely from giving it known questions and boundaries, including the benefit and limitation of each approach.
- Ingredient field selection. Explain why short_composition is used as the primary source of ingredient information rather than salt_composition.
- Structured drug feature extraction. Describe how the script derives a multi-salt indicator and separates an ingredient's name, dosage value, and dosage unit from short_composition.
- Parsing output review. Explain how inspecting the script's output revealed a missed dosage and how a concrete example prompted an improved parser.
- Drug Price Prediction Inputs. Describe the factors and data columns proposed for predicting a drug’s price, including why manufacturer name is included.
- Features, Label, and Row Alignment. Explain that the model uses drug features to predict price: X contains the feature columns, while y contains the price label. Describe why each row in X must correspond to the same drug's row in y.
- Held-Out Price Evaluation. Explain why some records must be kept out of training and how their known prices are used to check a trained model’s predictions.
- Training and Test Sets. Describe the section’s 80/20 split of the 250,000 drug records and the resulting feature and label datasets.
- Drug Price Prediction Pipeline. Trace how the drug dataset is split, used to train a price-prediction model, and used to assess its predictions.
- Training data preparation. Explain why the pipeline loads the engineered-feature CSV, removes rows missing required feature values, and excludes exceptionally high prices before training.
- Top N one-hot encoding. Trace how the pipeline converts a categorical salt name into numeric feature columns, including the top-50 grouping and Other_Salt category.
- Numeric price model training. Explain how the prepared features and price label are used to train the example’s regression model.
- Complementary test-set metrics. Distinguish what R², MAE, and RMSE reveal when evaluating the price predictor on the test set.
- Price Prediction Fit. Explain how the predicted-versus-actual chart and R² value show the model’s fit, and why that fit is insufficient for a fully trusted price estimate.
- Average Error And Large Misses. Distinguish what MAE and RMSE reveal about this model’s price errors, including why their difference matters.
- Feature Importance Interpretation. Explain what Random Forest feature-importance values indicate, what they cannot indicate, and which features stand out in this model.
- Baseline feedback for improvement. Explain how the previous model’s evaluation results were fed back to the LLM to request an improved approach, and why the resulting code was tried without testing each suggested change separately.
- Mixed evaluation metric results. Describe how the revised model’s R², MAE, and RMSE compared with the baseline, and explain the narrower sense in which the section calls it an improvement.
- Price prediction deployment limits. Explain why the improved price model might support rough estimates but is not suitable for production, and give the section’s suggested reason for its remaining error.
- Medication request risk classification. Explain that the proposed website classifies a client's text about a medication request as LOW RISK or HIGH RISK, then involves a human pharmacist for high-risk requests but not low-risk ones.
- Drug descriptions and risk labels. Explain that the model uses a medicine’s long, human-written description as its input and predicts whether the drug is high risk or low risk, with the label determined by the side_effects column.
- Shared Vocabulary and Binary Vectors. Trace how the example converts medicine descriptions into numeric vectors: build one dictionary from all descriptions, create a zero-filled vector for each record, then mark the dictionary positions of words present in that record with 1.
- Guardian Angel notebook prompt. Describe how the prompt asks AI to build the classifier: derive a high-risk flag from drug-interaction text, combine medicine descriptions and side effects as a text feature, use deep learning, and report precision and recall for the high-risk class.
- High-risk classification reliability. Distinguish precision from recall for high-risk medicines and explain what the reported confusion-matrix results indicate.
- Drug risk inference helper. Trace how predict_high_risk() turns a caller’s text into a LOW RISK or HIGH RISK result.
- Realistic query checks. Explain what the example predictions reveal about testing the classifier with inputs resembling actual user queries.
- Risk classification threshold trade-off. Explain how changing the probability threshold changes which medicines are labeled high risk and the trade-off between false alarms and missed risks.
- Threshold evaluation and selection. Describe how the section uses a precision-and-recall-versus-threshold plot, F1 score, and five-fold cross-validation to evaluate a cutoff; state why approximately 0.60 was selected in this example.
- Limits of strong validation results. Explain why the strong plot and consistent cross-validation scores do not rule out data leakage or shortcuts in this medicine dataset.
- Text Tokenization and Padding. Describe how medical descriptions become fixed-length numeric inputs for the classifier.
- Embedding and Dropout Layers. Explain what the embedding and dropout layers contribute to the classifier.
- Sequence Reading and Probability Output. Trace how the GRU and final dense layer turn the word sequence into a risk output.
- Class Imbalance Training Weights. Explain why the classifier weights high-risk training cases more heavily.
Section by section
Introduction
The introduction explains how LLMs can turn business questions into data-analysis notebooks while emphasizing that plausible-looking results still require human verification.
- LLM-assisted analysis workflow: Describe how answering a business question starts with choosing a suitable dataset and preparing it through cleaning, parsing, and normalization. Explain that an LLM can translate the question into a runnable notebook, letting a person with basic programming knowledge focus on the data and business domain rather than Python.
Why it matters: Generating a notebook is useful only if it addresses the right question with suitable, prepared data. - Plausible results and verification: Explain why errors in LLM-assisted data science can be harder to notice than errors in ordinary coding, and describe the resulting trust-but-verify approach.
Why it matters: A notebook that runs without errors can still lead you to a wrong business conclusion. - Educational medical model limits: State the boundary the introduction places on the chapter's pharmaceutical models.
Why it matters: The chapter's examples should not be mistaken for dependable medical advice.
Key terms: dataset, data cleaning, parsing, normalization, analytics notebook, business domain, trust but verify, business context, pharmaceutical data, strictly educational
4.1 Choosing the proper dataset
The section uses LLM-assisted discovery and comparison to choose a pharmaceutical dataset large and realistic enough for meaningful analysis and machine learning.
- Dataset suitability screening: Explain how the section screens candidate datasets for download availability, record count, language, and suitability for the planned analysis, rather than accepting an LLM recommendation without review.
Why it matters: A promising search result is not necessarily a usable dataset for the project. - Real versus synthetic data: Justify the choice of the Indian Medicine Dataset over the Kaggle dataset by contrasting real product data with easier-to-use synthetic data, including the consequences for analysis and model results.
Why it matters: The easiest dataset to model may not produce findings or models that hold up against actual products.
Key terms: availability, cardinality, records, translation, Indian Medicine Dataset, Kaggle Medicine Dataset, real-world data, synthetic datasets, outliers, malformed records
4.2 Set up the environment
Explains how to prepare an isolated Conda environment, launch Jupyter Notebook, and run Python code in notebook cells.
- Isolated Conda environment: Explain why the chapter uses Conda before running the analysis code: it manages packages in an isolated environment so installed libraries do not interfere with other projects.
Why it matters: Isolation lets you install what the analysis needs without disrupting your other work. - Activating and launching Jupyter: Describe how to enter the prepared environment and open Jupyter: activate `drug_analysis`, then run `jupyter notebook` from the `manning_vibe_engineering` directory to open its interface in a browser.
Why it matters: These steps put the notebook in the intended environment and open the interface for the analysis. - Interactive notebook cells: Describe a Jupyter Notebook as an interactive document of cells where Python code can be entered and run, using Shift + Enter or the Run button.
Why it matters: Running cells is how you will interact with the analysis code in Jupyter.
Key terms: Conda, isolated "sandbox", Miniconda, conda activate drug_analysis, jupyter notebook, manning_vibe_engineering, cells, Shift + Enter, Run
4.3 Exploratory data analysis
The section shows how to begin exploring an unfamiliar medicine dataset by asking an LLM to suggest analyses and create a Jupyter notebook.
- LLM-guided exploratory analysis: Explain how to start analyzing the unfamiliar medicine dataset without specifying questions in advance: ask the LLM to suggest analyses and create a Jupyter notebook, then inspect the results.
Why it matters: This gives you a way to find useful starting questions when you do not yet know the dataset. - Notebook results and data path: Describe how the generated notebook is used in the example, including the repository-relative data path and what the manufacturer analysis answers.
Why it matters: The notebook needs the right file path before its generated analyses can run on the provided data.
Key terms: exploratory, LLM, Jupyter notebook, file_path, Manufacturer analysis
4.3.1 Data analysis when we know the questions
The section shows how to turn known questions about a drug dataset into a prompt for a runnable analysis notebook, then contrasts that focused approach with open-ended AI analysis.
- Question Driven Notebook Prompt: Describe how to turn specific market questions into a prompt for an LLM-generated Jupyter notebook, including the dataset location and the requested analyses.
Why it matters: A precise prompt makes the generated notebook answer the questions you actually have and find the data it needs. - Focused Versus Open Analysis: Distinguish asking AI to explore a dataset freely from giving it known questions and boundaries, including the benefit and limitation of each approach.
Why it matters: Choosing how much to guide the AI determines whether the analysis favors discovery or answers to specific questions. - Interpreting Generated Analysis Results: Describe how the notebook's displayed results answer the requested questions, including what the single medicine-type bar reveals about this dataset.
Why it matters: Reading the outputs against the questions helps distinguish a dataset limitation from a missing chart result.
Key terms: LLM prompt, Jupyter Notebook, data/updated_indian_medicine_data.csv, delegating all analysis, questions, expectations and boundaries, manufacturer's analysis, price per drug type, summary DataFrame
4.4 Applying feature engineering
The section uses LLM-assisted feature engineering to turn drug composition text into structured fields, then checks and improves the extracted data.
- Ingredient field selection: Explain why short_composition is used as the primary source of ingredient information rather than salt_composition.
Why it matters: Choosing the more structured source makes the subsequent feature extraction more useful for analysis. - Structured drug feature extraction: Describe how the script derives a multi-salt indicator and separates an ingredient's name, dosage value, and dosage unit from short_composition.
Why it matters: The resulting columns provide reusable, structured inputs for later analysis and modeling. - Parsing output review: Explain how inspecting the script's output revealed a missed dosage and how a concrete example prompted an improved parser.
Why it matters: A quick review of real rows can expose fixable gaps that an apparently successful first script conceals.
Key terms: short_composition, salt_composition, Active Pharmaceutical Ingredient (API), Is_Multi_Salt, Salt_Name, Dosage_Value, Dosage_Unit, features_engineered_medicine_data.csv, Products with Dosage Value, nan
4.5 Predicting the price of drugs with machine learning techniques
This section motivates drug-price prediction and identifies the data columns the proposed model will use.
- Drug Price Prediction Inputs: Describe the factors and data columns proposed for predicting a drug’s price, including why manufacturer name is included.
Why it matters: These columns define what information the proposed price predictor will draw on. - Price Setting Tradeoffs: Explain why setting a product’s price too high or too low can harm a company.
Why it matters: The tradeoff explains why a useful price suggestion matters to the business.
Key terms: Dosage_Value, Is_Multi_Salt, Salt_Name, manufacturer_name, Dosage_Unit, margin, pricing
4.5.1 Supervised learning for predicting a label
This section defines the feature data and price label used to train a drug-price prediction model.
- Features, Label, and Row Alignment: Explain that the model uses drug features to predict price: X contains the feature columns, while y contains the price label. Describe why each row in X must correspond to the same drug's row in y.
Why it matters: Matching each drug's features to its price gives the model the examples it needs for price prediction.
Key terms: label, features, X, y, price
4.5.2 The train/test data split
This section explains how to shuffle and split the drug-price dataset so a model can train on one portion and have its predictions checked against held-out prices.
- Held-Out Price Evaluation: Explain why some records must be kept out of training and how their known prices are used to check a trained model’s predictions.
Why it matters: Keeping prices aside gives you a way to check predictions rather than merely train the model. - Training and Test Sets: Describe the section’s 80/20 split of the 250,000 drug records and the resulting feature and label datasets.
Why it matters: The split and its four datasets show which records teach the model and which support the accuracy check. - Shuffling Before the Split: Justify shuffling the records before dividing them into training and test sets.
Why it matters: Mixing the rows first helps prevent their original order from distorting what the model trains on.
Key terms: features, label, predicted price, accuracy, training set, test set, X, y, labels, shuffle
4.5.3 The full predict-the-price ML pipeline
This section brings together the supervised learning steps for training and evaluating a drug-price prediction model.
- Drug Price Prediction Pipeline: Trace how the drug dataset is split, used to train a price-prediction model, and used to assess its predictions.
Why it matters: The pipeline shows how to check price predictions against answers the model was not trained on.
Key terms: X training dataset, y training dataset, X test set, y-test dataset, predict method, accuracy
4.5.4 Vibe engineer the price prediction
The section walks through an LLM-assisted drug price prediction pipeline, from preparing engineered features and encoding categories to training and evaluating a regression model.
- ML engineer prompt framing: Describe how the example prompts the LLM to act as an ML engineer and show the evolution of a drug price prediction solution rather than only its final result.
Why it matters: The prompt frames the generated work as an explainable process, not just a finished model. - Training data preparation: Explain why the pipeline loads the engineered-feature CSV, removes rows missing required feature values, and excludes exceptionally high prices before training.
Why it matters: The model depends on usable engineered features and on prices that do not distort what it learns. - Top N one-hot encoding: Trace how the pipeline converts a categorical salt name into numeric feature columns, including the top-50 grouping and Other_Salt category.
Why it matters: Encoding lets a numeric model use text categories without creating a column for every distinct salt name. - Numeric price model training: Explain how the prepared features and price label are used to train the example’s regression model.
Why it matters: The choice of regression follows from predicting a numeric price rather than a category. - Complementary test-set metrics: Distinguish what R², MAE, and RMSE reveal when evaluating the price predictor on the test set.
Why it matters: The three metrics show different aspects of predictive performance rather than reducing quality to one number.
Key terms: ML engineer, Role & Context, features_engineered_medicine_data.csv, Dosage_Value, Salt_Name, 99th percentile, outliers, X, y, categorical features
4.5.5 Accessing the results
This section interprets a drug-price model’s prediction chart, accuracy metrics, and feature-importance output to show what the model captures and where it falls short.
- Price Prediction Fit: Explain how the predicted-versus-actual chart and R² value show the model’s fit, and why that fit is insufficient for a fully trusted price estimate.
Why it matters: A model can capture some pricing patterns without being reliable enough to set a final price. - Average Error And Large Misses: Distinguish what MAE and RMSE reveal about this model’s price errors, including why their difference matters.
Why it matters: The average error alone can conceal predictions that miss the real price by a great deal. - Feature Importance Interpretation: Explain what Random Forest feature-importance values indicate, what they cannot indicate, and which features stand out in this model.
Why it matters: Reading importance as relevance rather than direction prevents unsupported claims about what makes prices rise or fall.
Key terms: Predicted vs. Actual Drug Price, R-squared (R²), Actual Price, predicted Price, Mean Absolute Error (MAE), Root Mean Square Error (RMSE), INR, Random Forest, feature importance, Meropenem salt name
4.5.6 Improving the model
This section uses baseline evaluation results to prompt an LLM for a revised price-prediction model, then weighs the mixed metric changes against the model’s practical limits.
- Baseline feedback for improvement: Explain how the previous model’s evaluation results were fed back to the LLM to request an improved approach, and why the resulting code was tried without testing each suggested change separately.
Why it matters: Giving the LLM measured results makes the request for improvement specific, while trying the changes together limits what can be attributed to any one change. - Mixed evaluation metric results: Describe how the revised model’s R², MAE, and RMSE compared with the baseline, and explain the narrower sense in which the section calls it an improvement.
Why it matters: A lower average error is useful, but it does not mean every measure of model quality improved. - Price prediction deployment limits: Explain why the improved price model might support rough estimates but is not suitable for production, and give the section’s suggested reason for its remaining error.
Why it matters: An improved metric does not by itself make a prediction model reliable enough to deploy.
Key terms: model evaluation, dosage, ablation study, drop-in replacement, Dosage_Bin, R², MAE, RMSE, rough price estimation, production environment
4.6 Building a digital guardian angel
Introduces a text-based risk classifier for medication requests and shows how its two classifications determine whether a human pharmacist is involved.
- Medication request risk classification: Explain that the proposed website classifies a client's text about a medication request as LOW RISK or HIGH RISK, then involves a human pharmacist for high-risk requests but not low-risk ones.
Why it matters: The classification matters because it determines when a request needs human attention.
Key terms: classification system, LOW RISK, HIGH RISK, human pharmacist
4.6.1 Correlating long text with predicted class
This section sets up a drug-risk classification task using long medicine descriptions as input and risk labels derived from reported side effects.
- Drug descriptions and risk labels: Explain that the model uses a medicine’s long, human-written description as its input and predicts whether the drug is high risk or low risk, with the label determined by the side_effects column.
Why it matters: It distinguishes the text the model reads from the side-effect information used to assign the answer. - Training and test set roles: Distinguish the role of the training set from the test set in this classification task.
Why it matters: The split keeps model training and accuracy evaluation in distinct parts of the pipeline.
Key terms: medicine description, side_effects column, high risk, low risk, train set, test set, accuracy
4.6.2 Tokenization of medicine description
This section shows how to turn medicine descriptions into binary word-presence vectors and explains why practical tokenizers and vocabulary size matter.
- Shared Vocabulary and Binary Vectors: Trace how the example converts medicine descriptions into numeric vectors: build one dictionary from all descriptions, create a zero-filled vector for each record, then mark the dictionary positions of words present in that record with 1.
Why it matters: A shared word-to-index mapping makes descriptions usable as numeric inputs. - Practical Tokenizer Handling: Explain why the section recommends a tokenizer library rather than relying on its simplest word-to-vector example.
Why it matters: The simple example explains the representation, while a library handles routine text-cleaning complications. - Vocabulary Size and Input Layer: State how the size of the built dictionary relates to the deep-learning model's input layer.
Why it matters: Choosing the dictionary also determines the size of the model's input.
Key terms: medicine_description, dictionary, vector, tokenization, word normalization, tokenizer, input layer, neurons
4.6.3 Deep learning based classification with vibe engineering
This section uses an AI-generated notebook to build a deep-learning drug-risk classifier, then checks its usable data, training output, and classification results.
- Guardian Angel notebook prompt: Describe how the prompt asks AI to build the classifier: derive a high-risk flag from drug-interaction text, combine medicine descriptions and side effects as a text feature, use deep learning, and report precision and recall for the high-risk class.
Why it matters: The prompt makes clear what the model should predict, what evidence it should use, and how its results should be checked. - Usable records and missing fields: Explain why the classification training data is much smaller than the 250,000-record regression dataset and how the notebook checks the available records.
Why it matters: Checking which fields are present explains how much data is actually available to train this classifier. - Training output interpretation: Describe what the notebook's training output shows about progress and when training finishes.
Why it matters: The training output provides a basic check that the generated notebook is improving rather than merely running. - High-risk classification reliability: Distinguish precision from recall for high-risk medicines and explain what the reported confusion-matrix results indicate.
Why it matters: A safety warning must be judged both by whether its alerts are credible and by whether it misses dangerous cases.
Key terms: Target Engineering, NLP Feature Engineering, High Risk, medicine_desc, side_effects, drug_interactions, class distribution, non-null fields, df.count(), epochs
4.6.4 How to use the model?
The section shows how to call the drug-risk model and why its predictions should be checked with queries resembling real user input.
- Drug risk inference helper: Trace how predict_high_risk() turns a caller’s text into a LOW RISK or HIGH RISK result.
Why it matters: The probability and threshold explain how a model output becomes a risk label for a caller. - Realistic query checks: Explain what the example predictions reveal about testing the classifier with inputs resembling actual user queries.
Why it matters: Trying plausible user queries can reveal behavior that detailed test-set descriptions do not represent.
Key terms: predict_high_risk(), medicine_description, side_effects, threshold, LOW RISK, HIGH RISK, test set description, user query, predicted probability, anxiety
4.6.5 Picking the proper threshold
The section shows how to choose a high-risk classification threshold by examining precision and recall, checking the choice with cross-validation and F1 score, and questioning unusually strong results.
- Risk classification threshold trade-off: Explain how changing the probability threshold changes which medicines are labeled high risk and the trade-off between false alarms and missed risks.
Why it matters: The cutoff determines what the model warns users about, not just how its score is displayed. - Threshold evaluation and selection: Describe how the section uses a precision-and-recall-versus-threshold plot, F1 score, and five-fold cross-validation to evaluate a cutoff; state why approximately 0.60 was selected in this example.
Why it matters: A measured cutoff is more defensible than automatically using 0.5 or choosing one by guesswork. - Limits of strong validation results: Explain why the strong plot and consistent cross-validation scores do not rule out data leakage or shortcuts in this medicine dataset.
Why it matters: High validation scores should not be mistaken for proof that warnings will be reliable on unfamiliar medicines.
Key terms: threshold, High Risk, Low Risk, false positives, false negatives, Precision/Recall vs. Threshold plot, precision, recall, F1-Score, K-Fold Cross-Validation
4.6.6 Understanding the architecture
Explains how the drug-risk classifier turns medical descriptions into a probability and compensates for rare high-risk cases during training.
- Text Tokenization and Padding: Describe how medical descriptions become fixed-length numeric inputs for the classifier.
Why it matters: This step makes descriptions of different lengths usable by the same classifier. - Embedding and Dropout Layers: Explain what the embedding and dropout layers contribute to the classifier.
Why it matters: These layers help the model represent word meaning without depending too heavily on particular words. - Sequence Reading and Probability Output: Trace how the GRU and final dense layer turn the word sequence into a risk output.
Why it matters: This connects the description’s word order and context to the classifier’s single probability output. - Class Imbalance Training Weights: Explain why the classifier weights high-risk training cases more heavily.
Why it matters: Weighting keeps a strong-looking accuracy score from masking missed high-risk cases.
Key terms: Text Tokenization and Padding, tokenizer, vocabulary, padding, Embedding Layer, 64-dimensional vector space, SpatialDropout1D, overfitting, Sequential, GRU (Gated Recurrent Unit)
4.7 Summary
Concepts explained in this chapter
- language modelA language model is a machine-learning model that processes and generates text by predicting sequences of tokens.
- feature engineeringA feature engineering pipeline is a sequence of transformations that creates model inputs from raw data.
- precisionPrecision is the proportion of predicted positive instances that are actually positive.
- recallRecall is the proportion of actual positive cases that a classifier correctly identifies.
- data cleaningData cleaning is the process of detecting, correcting, removing, or managing inaccurate, incomplete, inconsistent, or irrelevant data.
- thresholdA classification threshold is the score cutoff used to convert a model's continuous output into a discrete class decision.
- parsingString parsing is the process of interpreting character sequences and extracting their structured components.
- mean absolute errorMean absolute error is a regression metric equal to the average absolute difference between predicted and actual values.
- promptA prompt is an input instruction or request supplied to an AI model to elicit a response or action.
- data analysisData analysis, or analytics, is the systematic inspection, transformation, and interpretation of data to discover patterns, derive insights, and support decisions.
- labelA label is the target value associated with an example that a supervised machine-learning model learns to predict or uses for evaluation.
- overfittingOverfitting is a modeling failure in which learned patterns fit training data too closely and generalize poorly to unseen data.
- false positiveA false positive is a classification that incorrectly reports a positive or high-risk condition when it is absent.
- normalizationNormalization is the process of transforming data into a consistent representation or scale suitable for comparison and analysis.
- precision-recall curveA precision-recall curve plots a classifier's precision against its recall across different decision thresholds.
- root mean squared errorThe square root of the average squared difference between predicted and actual values, which penalizes larger errors more heavily.
- coefficient of determinationA regression metric, also called R-squared or R², measuring the proportion of target variance explained by a model.
- machine learningMachine learning is the field and set of computational methods in which models learn patterns from data to make predictions or decisions.
- cross-validationCross-validation is a model-evaluation technique that repeatedly trains and evaluates a model on different partitions of a dataset.
- data leakageData contamination is the leakage of evaluation answers or related held-out information into model inputs or training-like context, invalidating an evaluation.
- f-scoreThe F1 score is the harmonic mean of precision and recall.
- signalA feature is an input variable or observed attribute supplied to a machine-learning model for making a prediction.
- high-risk categoryA classification label indicating that a medication or medication request is considered high risk and requires heightened scrutiny or review.
- synthetic dataSynthetic data is data generated algorithmically rather than directly collected from the underlying real-world process.
- deep learningDeep learning is a machine-learning approach based on multilayer neural networks that learn hierarchical representations from data.
- machine learning workflowA machine-learning pipeline is an ordered workflow that prepares data, trains a model, and produces or evaluates predictions.
- manufacturer analysisManufacturer analysis is the examination of medicine records to determine which manufacturers produce the most medicines.
- real dataA real-world dataset is a collection of observations gathered from an actual domain or operational process rather than generated artificially.
- evaluationThe assessment of an artifact or system to determine its quality, correctness, or fitness against relevant criteria.
- classificationClassification is a supervised machine-learning technique that assigns inputs to one of a defined set of classes.
- evaluation datasetA held-out portion of data used to evaluate a trained machine-learning model on previously unseen inputs and targets.
- training datasetTraining data is the collection of examples used to fit or adapt a machine-learning model.
- one-hot encodingOne-hot encoding transforms a categorical value into a vector of binary indicator variables, with one indicator active for the selected category.
- random forestA random forest is an ensemble machine-learning model that combines predictions from many decision trees.
- known-question analysisKnown-question analysis is a data-analysis approach in which predefined questions determine the computations and outputs to produce.
- tradeoffA trade-off is a compromise in which improving one objective or property requires accepting a reduction in another.
- GRUA gated recurrent unit is a recurrent neural-network architecture that uses gating mechanisms to control information flow through a sequence.
- datasetA dataset is a structured collection of related data records or values used for analysis or machine learning.
- tokenizationThe process of converting text into discrete tokens suitable for computational processing.
- vibe engineeringA disciplined approach to AI-assisted software development that emphasizes clear intent, clean abstractions, verification, testing, quality, and maintainable ownership.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
