What you will learn in Starting Data Analytics with Generative AI and Python
Chapter by chapter
Chapter 3: Descriptive analysis and statistical inference supported by generative AI
What you learn in chapter 3 →Concepts explained in this book
- generative modelingA machine-learning model that generates new content, such as text, code, or explanations, from learned patterns and user instructions.
- ChatGPTA conversational generative-AI system developed by OpenAI that responds to user prompts with generated content.
- data frameA DataFrame is a two-dimensional tabular data structure whose rows and columns can hold labeled, potentially heterogeneous data.
- language modelA machine-learning model trained on large text collections to understand and generate natural language.
- heat mapA heatmap is a graphical display that represents numerical values through colors arranged in a grid.
- hallucinationA generative-AI behavior in which a model produces plausible but unsupported, invented, or inaccurate information.
- sentiment analysisSentiment analysis is the computational analysis of text to determine expressed opinions, emotions, or evaluative polarity.
- recallThe proportion of actual positive cases correctly classified as positive, equivalent to sensitivity.
- specificitySpecificity is the proportion of actual negative instances that a classification method correctly identifies.
- descriptive statisticsDescriptive analytics is the examination and summarization of observed data to characterize its distributions, patterns, and relationships.
- data cleaningA data cleaning protocol is a documented procedure for detecting, correcting, and documenting problems in data.
- GeminiA family of Google's generative AI models capable of processing and generating text and, in some variants, other modalities.
- lexical searchA text-mining technique that locates and extracts information from text collections by matching specified words or phrases.
- contractA defined software interface through which programs request data or operations from another system and receive responses.
- data qualityA data quality issue is a defect in data's accuracy, completeness, consistency, validity, or interpretation.
- accuracyClassification accuracy is the proportion of observations whose predicted class matches the actual class.
- confusion matrixA confusion matrix is a table comparing actual classes with predicted classes to summarize classification outcomes.
- visualizationData visualization is the graphical representation of data to communicate patterns, relationships, or distributions.
- promptAn input instruction or contextual payload that directs a generative AI system to produce a response.
- machine translationCode conversion is the transformation of source code from one programming language into equivalent code in another language.
- frequency analysisFrequency-based text analysis infers textual meaning or themes from the occurrence counts of words, phrases, or terms.
- linear regressionLinear regression is a statistical modeling technique that estimates a linear relationship between a dependent variable and one or more independent variables.
- pandaspandas is a Python library providing labeled data structures and tools for manipulating and analyzing tabular data.
- continuous improvementA feedback loop is a recurring cycle in which information about outcomes is communicated back to participants to guide improvements or decisions.
- vocabularyA vocabulary is the fixed collection of tokens that a language model can represent.
- performance optimizationCode optimization is the systematic modification of algorithms, code, or data-processing pipelines to improve execution speed, performance, or resource utilization.
- missing valueA missing value is an absent or unavailable data value in a record or variable.
- parallelismParallelized code is software structured so that multiple portions of a computation can execute concurrently.
- seasonality effectSeasonality is a recurring pattern in observations associated with particular calendar periods or intervals.
- summarizationText summarization is the generation of a shorter text that preserves the important information and key ideas from a source text.
- prompt engineeringPrompt engineering is the deliberate design and refinement of instructions supplied to a generative AI system.
- data analysisThe process of examining and interpreting data to produce insights for understanding or decision-making.
- distinct valueA unique value is a distinct value occurring in a field or variable, counted once regardless of its frequency.
- metadataMetadata is descriptive information about data, including its fields, meanings, units, formats, and provenance.
- portuguesePortuguese is a natural language spoken and written in Portuguese-speaking communities and supported by language-specific text-processing resources.
- post-processingPost-processing is the transformation, validation, or cleanup of generated or computed output after its initial production.
- revenueRevenue is the income generated from selling products or services before subtracting costs and expenses.
- volumeSales volume is the quantity of products or units sold during a specified period or across a specified population.
- review scoreA review score is a numerical rating assigned by a customer to express an evaluation of a product or purchase.
- bootstrappingBootstrapping is a resampling technique that estimates the variability of a statistic by repeatedly sampling observations with replacement from an observed dataset.
- pretrained modelA generative AI model that has undergone initial training before being adapted or used for a task.
- classifierA task that assigns observations or records to predefined categories.
- dictionary-based methodA dictionary-based method analyzes text by matching its terms against a predefined dictionary or lexicon of relevant words and expressions.
- lemmatizationLemmatization is the process of converting inflected words into their canonical dictionary forms using linguistic analysis.
- biasBias is a systematic preference or assumption that can distort the selection or interpretation of information, including generative-model outputs.
- context windowThe maximum amount of conversational or input-output context that a language model implementation can process in one exchange.
- coefficient of determinationR2 is the proportion of variance in a dependent variable explained by a regression model's independent variables.
- product attributeA product attribute is a measurable or descriptive characteristic of a product, such as its weight, dimensions, or volume.
- shipping costA shipping cost is the monetary charge for transporting a product or package from a sender to a destination.
- stop wordA stop word is a high-frequency function word that text-processing systems commonly exclude from analysis because it carries limited task-specific content.
- zero-shot learningA zero-shot learning model performs a task or makes predictions for categories without task-specific examples for those categories during inference.
- product categoryA product category is a classification grouping products that share relevant characteristics for analysis, organization, or commerce.
- human-in-the-loopOversight is the monitoring and supervision of an AI system's performance, impacts, and compliance by responsible people or groups.
- ordinal logistic regressionOrdinal logistic regression is a regression method for modeling an ordered categorical outcome using explanatory variables.
- probability distributionThe pattern of values and their frequencies for a variable or measurement.
- anomalyAn outlier is an observation that is unusually distant from the other values in a dataset or from an expected pattern.
- confidence intervalA confidence interval is an interval-estimation procedure that gives a range of plausible values for a population parameter at a stated confidence level.
- regression coefficientA regression coefficient is an estimated parameter expressing the change in a modeled outcome associated with a one-unit change in a predictor while other predictors are held constant.
- data explorationExploratory data analysis is the process of examining data with summaries, statistical methods, domain knowledge, and visualizations to discover structure, patterns, trends, and potential issues.
- AI delusionA user's unjustified belief that a generative AI system is highly capable, infallible, or authoritative, leading to insufficient skepticism toward its outputs.
- co-occurrence analysisCo-occurrence analysis is a text-analysis technique that measures how often terms appear near one another within a defined context window.
- customer satisfactionCustomer satisfaction is the degree to which customers view a product or service favorably in relation to their expectations.
- meanThe mean is the arithmetic average of a set of numerical values.
- multilingual supportA multilingual model is a machine-learning model designed to process or generate content in multiple human languages.
- data groupingA groupby operation partitions tabular rows by one or more key columns so that aggregations or transformations can be computed separately for each group.
- domain knowledgeSubject-matter understanding of the meanings, expected values, and valid relationships in an application's domain.
- initial inspectionInitial inspection is the preliminary examination of a dataset's structure, values, types, completeness, and distributions before analysis.
- Brazilian e-commerce public datasetThe Brazilian e-commerce public dataset is a collection of records describing online retail activity in Brazil.
- scalabilityScalability is the ability of a system or method to maintain or improve effective operation as workload or data volume increases.
- stacked bar chartA stacked bar plot displays multiple component values as segments accumulated within each bar.
- order statusAn order status is a categorical label representing the current or recorded state of an order, such as approved, shipped, or delivered.
- time seriesA time series is a sequence of observations indexed or ordered by time.
- token limitA token limit is a maximum number of tokens permitted for a model interaction or generated response.
- stemmingA text-normalization technique that reduces words to truncated root-like forms by applying linguistic rules.
- consistencyThe property of data values and representations conforming to the same rules and conventions across records or sources.
- data aggregationAggregation is the process of combining detailed records or categories into summarized groups or values.
- resamplingResampling is a statistical technique that repeatedly constructs samples from observed data to assess the behavior or uncertainty of an estimator.
- AI use policyAn AI use policy is a documented set of organizational rules and principles governing the ethical, responsible, and compliant use of artificial intelligence.
- mean absolute errorMean absolute error is the average of the absolute differences between predicted and actual values.
- mean squared errorMean squared error is the average of the squared differences between predicted and actual values.
- root mean squared errorRoot mean squared error is the square root of mean squared error and expresses prediction error in the dependent variable's units.
- grouped bar chartA grouped bar chart displays separate bars for categories or subgroups side by side for comparison.
- OpenAI APIA public programming interface that provides access to OpenAI artificial-intelligence models and services.
- machine learningMachine learning is the computational use of algorithms that learn patterns from data to make predictions, classifications, or decisions.
- validationValidation is the process of checking whether an analytical or model-generated result is accurate, relevant, plausible, and fit for its intended purpose.
- sycophancySycophancy is a model tendency to agree with or appease a user even when doing so conflicts with truth, reliability, or accuracy.
- correlationA statistical measure of the association between variables.
- area under the curveA metric equal to the area under a receiver operating characteristic curve that summarizes discrimination across thresholds.
- matthews correlation coefficientA binary-classification metric ranging from -1 to +1 that summarizes agreement between predicted and observed classes using all four confusion-matrix counts.
- receiver operating characteristicA plot of true positive rate against false positive rate across classification thresholds.
- fine-tuningFine-tuning is the further training of a pretrained model on targeted data to adapt its behavior or performance.
- scatter matrix plotA scatterplot matrix is a grid of scatterplots showing pairwise relationships among multiple numerical variables.
- analysis designA planned specification of the steps, methods, data, possible results, and interpretations needed to answer an analytical question.
- data tokenizationTokenization is the process of replacing sensitive data elements with unique tokens that have no intrinsic meaning outside a specific dataset.
- keywordA word or short phrase that represents a topic, property, or information need in text analysis or retrieval.
- bootstrap distributionA bootstrap distribution is the empirical distribution of a statistic computed across many resampled datasets.
- sampling with replacementSampling with replacement is a procedure in which each selected observation is returned to the source population and can be selected again.
- Gemini AdvancedA subscription conversational generative-AI service that provides access to Google's advanced language models.
- knowledge cutoffThe temporal boundary after which information is generally absent from a model's trained knowledge.
- preprocessingData preprocessing is the transformation and preparation of raw data into a form suitable for analysis or modeling.
- positive sentimentPositive sentiment is a favorable emotional or evaluative attitude expressed toward a product, service, or experience.
- CPUA CPU is a general-purpose processor designed to execute a broad range of sequential and parallel computational workloads.
- CUDACUDA is NVIDIA's parallel-computing platform and programming interface for executing general-purpose computations on compatible GPUs.
- CuPyCuPy is a Python array-computing library that provides GPU-accelerated operations with an interface modeled on NumPy.
- GPUA GPU is a massively parallel processor originally designed for graphics that is also used to accelerate general-purpose numerical and machine-learning workloads.
- multicoreA multicore architecture is a processor design containing multiple independent CPU cores capable of executing computations concurrently.
- parallelization overheadParallelization overhead is the extra computation and coordination cost incurred when dividing, scheduling, communicating, and recombining concurrent work.
- global interpreter lockThe Global Interpreter Lock is a Python runtime mechanism that restricts simultaneous execution of Python bytecode by multiple threads in one process.
- thread poolThreadPoolExecutor is a Python concurrency component that manages a pool of worker threads for executing callable tasks.
- TPUA TPU is a specialized accelerator designed for high-performance machine-learning computations, especially neural-network operations.
- bar chartA bar chart, or bar plot, represents values for discrete categories using rectangular bars whose lengths or heights encode magnitude.
- code generationThe automatic production of executable source code from prompts, specifications, or other program representations.
- Hugging FaceHugging Face is a platform that hosts and distributes machine-learning models, datasets, and related development resources.
- training from scratchTraining from scratch is the process of learning a model's parameters from initialized parameters using task-relevant training data.
- sales performanceSales performance is the measured level of sales activity or results for products, categories, regions, or periods.
- privacyThe protection and proper governance of information about identifiable people during its collection, processing, storage, and disclosure.
- LangChainLangChain is a software framework for developing applications that compose language models with prompts, tools, data sources, and conversation state.
- LlamaA family of large language models developed by Meta for generative-AI applications.
- sentiment scoreSentiment intensity is the degree or strength of favorable, unfavorable, or emotional expression in text.
- keyword-based extractionA technique that classifies or extracts text by matching predefined keywords associated with a target category.
- data analysis systemA system or workflow that collects, transforms, analyzes, and communicates insights from data.
- resource utilizationResource utilization is the amount or proportion of available computing resources consumed by a system or task.
- topic modelingTopic modeling is a statistical or machine-learning technique that discovers recurring abstract topics in a collection of documents.
- Code LLaMACode Llama is a large language model specialized for generating and understanding programming code.
- tokenA token is a unit of text processed by a language model, such as a word, word fragment, or punctuation mark.
- cyclical patternCyclicity is the recurrence of patterns or fluctuations at regular or approximately regular intervals.
- cyclicity detectionCyclicity detection is the identification of recurring patterns or periodic fluctuations in data over time.
- time series analysisTime series analysis is the analysis of observations ordered in time to identify trends, cycles, dependencies, and other temporal behavior.
- profilingProfiling is the systematic measurement of a program or system's resource usage and execution behavior to identify performance characteristics.
- stakeholder educationStakeholder education is the planned development of knowledge and practical understanding among people affected by or involved in an AI system.
- workshopA training workshop is an organized instructional session that develops participants’ knowledge or skills through teaching and practice.
- accountabilityAccountability is the assignment and acceptance of responsibility for the operation, effects, and corrective management of an AI system.
- sentimentCustomer sentiment is the overall favorable, unfavorable, or neutral attitude expressed by customers about a product, service, or experience.
- causationCausation is a relationship in which changes in one factor produce changes in another factor.
- risk managementRisk management is the process of identifying, assessing, and addressing potential threats to objectives or operations.
- abstractive summarizationAbstractive summarization generates a summary by interpreting source content and producing new wording rather than only selecting source sentences.
- extractive summarizationExtractive summarization creates a summary by selecting and combining important sentences, terms, or other passages from the source text.
- problem definitionA precise formulation of the real problem that an analysis is intended to address.
- pipelineA data pipeline is a sequence of processes that moves and transforms data between sources, storage, and analytical stages.
- transparencyTransparency is the degree to which a system’s decisions, processes, and relevant information are made understandable and inspectable.
- DaskDask is a Python library for parallel and distributed computing on datasets and computations that may exceed the resources of a single machine.
- text analysisText analysis is the computational examination of textual content to identify linguistic patterns, themes, frequencies, or other meaningful properties.
- reviewA product review is a user's written or rated evaluation of a product or service.
- client libraryA collection of programming tools, libraries, documentation, and other resources for building applications against a software platform or service.
- statistical parameterA numerical characteristic of an entire statistical population, such as its mean or proportion.
- business questionA business question is a decision-oriented question about an organization's goals, operations, performance, or opportunities.
- data dictionaryA data dictionary is a reference that describes a dataset's variables, types, scales, meanings, codes, and transformations.
- research questionA research question is a specific, answerable question that guides analysis of data in order to address a broader issue.
- false-positive rateThe proportion of actual negative cases incorrectly classified as positive.
- data harmonizationData harmonization is the process of integrating data from different sources into a consistent and coherent representation.
- dialogueAn interaction pattern in which multiple successive user and assistant messages form a single contextual conversation.
- Google Vertex AIGoogle Cloud's managed platform for developing, deploying, and using machine-learning and generative AI models.
- messages arrayAn ordered array of message objects representing the conversation input supplied to a language model.
- network bandwidthData transfer bandwidth is the maximum rate at which data can be transferred through a storage or communication channel.
- OpenAI API-compatible modelAn AI model exposed through an interface conforming to the conventions of the OpenAI API.
- system-role messageA conversation message that establishes background instructions or behavioral guidance for an assistant model.
- messageA structured record containing a conversational role and the corresponding message content.
- distribution testingDistribution analysis is the examination of observed values and frequencies to identify patterns, deviations, errors, or bias.
- context-awarenessThe ability of an AI system to account for relevant surrounding information when interpreting input and generating output.
- multicollinearityMulticollinearity is a condition in which two or more predictor variables are highly correlated, making coefficient estimates unstable or difficult to interpret.
- logitLog-odds are the natural logarithm of the odds of an event or ordered outcome category.
- ordinal dataAn ordinal variable is a categorical variable whose values have a meaningful order but not necessarily equal intervals.
- overfittingOverfitting is a modeling failure in which a model fits noise or sample-specific variation rather than generalizable underlying patterns.
- NLTKNLTK is a Python library for natural language processing, including tokenization, corpora, classification, and text analysis.
- spaCyspaCy is an open-source Python library for industrial-strength natural language processing.
- monitoringMonitoring is the ongoing collection and analysis of system measurements to track behavior and detect changes or problems.
- tokenizerA tokenizer is a software component that segments text into discrete tokens according to specified processing rules, often for a particular language.
- stop-word removalStopword removal is the text-preprocessing technique of excluding selected common function words from analyzed text.
- securityProtection of systems, data, and operations against unauthorized access, misuse, disruption, or compromise.
- fairnessThe property of treating relevant individuals or groups equitably and avoiding unjustified disparities in AI systems or their use.
- reliabilityReliability is the degree to which a system produces accurate and timely results consistently under varying conditions.
- text miningText mining is the computational extraction and organization of patterns, relationships, and insights from textual content.
- logistic regressionLogistic regression is a regression method that models the probability of a categorical outcome, commonly a binary outcome.
- part-of-speech tagA linguistic annotation identifying the grammatical category of a word, such as noun, verb, or adjective.
- anonymizationData anonymization is the irreversible modification or removal of identifying information so that individuals in a dataset can no longer be identified.
- scatter plotA scatter plot is a chart that represents observations as points positioned according to two quantitative or categorical variables.
- training datasetA training dataset is a collection of examples used to fit the parameters of a machine-learning model.
- histogramA histogram is a chart that represents the frequency or density of numeric values across contiguous intervals.
- inappropriate contentGenerated material that reflects unfair distortions or violates contextual expectations of acceptable content.
- input/output limitA maximum bound on the amount of prompt and response text a generative AI implementation can process.
- tokenizationAn algorithm that divides text into the token units consumed by a language model.
- verbosityThe tendency of generated output to contain more words or repeated phrasing than necessary.
- throughputThroughput is the rate at which a system processes data, commonly expressed as transactions or records per unit of time.
- correlation matrixA correlation matrix is a tabular array containing pairwise correlation values for multiple variables.
- missing data indicatorA missing value indicator is a boolean representation marking whether each data value is missing.
- populationA population is the complete set of individuals, units, or observations about which a statistical analysis seeks to draw conclusions.
- statistical sampleA sample is a subset of observations collected or selected to represent a larger population or data-generating process.
- cloud computingCloud computing is the delivery of computing resources and software services over the internet rather than from locally maintained hardware.
- mergeA merge function combines tabular data structures by matching rows on one or more key columns.
- validation ruleA validation rule is a defined condition that data must satisfy to be considered acceptable.
- foreign keyA foreign key is a column or set of columns whose values reference identifying values in another dataset's primary key.
- primary keyA primary key is a column or set of columns whose values uniquely identify records in a dataset.
- data typeA data type specifies the kind of value represented by a variable or column and the operations appropriate for that value.
- binomial distributionA binomial distribution gives the probabilities of the number of successes in a fixed number of independent Bernoulli trials with a common success probability.
- normal distributionA normal distribution is a continuous probability distribution that is symmetric about its mean and has a bell-shaped density curve.
- bias detectionBias detection is the process of identifying systematic distortions or unequal treatment in model behavior or outputs.
- sentiment classifierA sentiment classifier is a model that assigns text to sentiment categories such as positive, negative, or neutral.
- computer clusterA cluster is a group of interconnected computers that operates as a unified system for computational tasks.
- study designThe planning of an experiment or analytical investigation, including its procedures, possible outcomes, and interpretation of those outcomes.
- statistical significanceThe degree to which observed data provide evidence against a specified null hypothesis under a statistical testing procedure.
- ratioA ratio is a quantitative comparison of two values expressed through their quotient or relative size.
- personal dataPersonally identifiable information is information that can identify a specific individual directly or when combined with other information.
- bias mitigationBias mitigation is the application of measures intended to reduce systematic distortion or unfairness in model behavior or outputs.
- contextual ruleA rule describing how the interpretation or relevance of a lexical item changes according to its surrounding context.
- frequency-based summarizationFrequency-based summarization identifies frequently occurring terms, n-grams, or phrases and uses them to represent the main topics of a text.
- keyword extractionKeyword extraction identifies words or short phrases that best represent the topics or salient content of a text.
- rule-based summarizationRule-based summarization generates summaries according to predefined linguistic rules, heuristics, or templates.
- co-occurrence matrixA co-occurrence matrix is a table whose rows and columns represent unique items and whose values record how often pairs occur together.
- box-and-whisker plotA box plot is a statistical chart that summarizes a numerical distribution using quartiles, a median, and potential extremes.
- NLIA language-analysis task that determines whether a hypothesis is entailed by, contradicts, or is neutral with respect to a premise.
- sarcasmAn indirect or ironic form of expression in which the intended attitude differs from the literal wording, often conveying criticism or frustration.
- polarityA numerical value representing the favorable, unfavorable, or neutral sentiment orientation associated with text or a lexical item.
- benchmarkBenchmarking is the comparison of a system's performance with a reference implementation or established standard metrics.
- concurrencyConcurrency is the capacity of a system to handle multiple queries, processes, or tasks at the same time.
- uptimeAvailability is the likelihood that a system is operational and accessible when needed.
- stress testStress testing evaluates whether a system continues to perform adequately under workloads near or beyond its expected maximum.
- intellectual propertyCreations of the mind, such as inventions, works, designs, and proprietary knowledge, that may be legally protected or commercially valuable.
- content filteringOutput filtering is the screening or modification of generated responses to block content that violates safety or usage requirements.
- GDPRThe General Data Protection Regulation is a European Union regulation governing the processing and protection of personal data.
- relevanceRelevance is the data-quality property of being pertinent to the specific business question or problem being addressed.
- data integrityData integrity is the preservation of data's accuracy, consistency, completeness, and trustworthiness throughout its handling.
- timelinessTimeliness is the data-quality property of being current and appropriate for the period being analyzed.
- manual evaluationManual inspection is the direct examination of data or output by a person to identify unusual, incorrect, or inconsistent values or formatting.
- garbage in, garbage outGarbage in, garbage out is the principle that poor-quality inputs can produce unreliable outputs even when the processing or analysis is sophisticated.
- natural language processingNatural language processing is the field concerned with computational analysis and generation of human language.
- frequencyInformation describing how often a word or expression occurs in a language or a specified text collection.
- data driftDomain mismatch is the discrepancy between the data domain used to develop a model and the domain of the data to which it is applied.
- FinBERT-PT-BRFinBERT-PT-BR is a Portuguese sentiment-classification model trained on financial news.
- negative reviewA negative review is a customer review that expresses unfavorable evaluation or sentiment according to a chosen classification rule.
- lemmaA lemma is the canonical dictionary form representing a set of inflected word forms.
- temporal autocorrelationAutocorrelation is the correlation between a time series and lagged versions of itself.
- time differenceA time difference is the elapsed duration between two time-valued observations.
- risk acceptanceRisk acceptance is a policy decision defining the level and conditions of risk that may be tolerated without additional intervention.
- truncationTruncation is the removal of input tokens that exceed a model's permitted context length.
- data formatA defined representation and encoding scheme for storing or exchanging data.
- error handlingA technique for detecting, responding to, and recovering from software or service failures.
- retryA recovery procedure that repeats a failed operation according to defined retry conditions.
- source materialReference content supplied to a generative AI system as evidence or context for producing a response.
- rate limitA technique that restricts the frequency or volume of requests accepted by a service.
- compute-boundA CPU-bound task is a computation whose execution time is primarily limited by available processor computation capacity.
- I/O-bound workAn I/O-bound task is a computation whose execution time is primarily limited by input or output operations rather than processor computation.
- memory boundA bottleneck is a component or operation that limits the overall performance or throughput of a system.
- dataset biasBiased data is data whose composition or collection process systematically overrepresents, underrepresents, or favors particular groups or viewpoints.
- word cloudA visual representation of words in which visual prominence, such as font size, reflects their frequencies.
- virtual machineA virtual machine is a software-emulated computer that runs its own operating system within an isolated allocation of physical computing resources.
- lil_matrixlil_matrix is a SciPy sparse matrix representation optimized for incremental construction and modification.
- sparse representationA sparse matrix is a matrix data structure that stores primarily nonzero elements without allocating equivalent storage for the many zero elements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.