A dataset is the collection of data assembled for a particular analysis. It may contain numbers, text, or mixed data types; it may be clean or unclean, ordered or unordered. Its value comes from preparing and analyzing it to find patterns and produce actionable insights. section 1.2.1 section 1.0
Start with the business question, then choose data and attributes that describe it. An apparel retailer might combine revenue and invoice data in a master table and derive average transaction value as total revenue divided by the number of invoices. A telecom operator would instead need attributes such as subscriber usage, call rate, revenue, time on the network, and data usage. The dataset changes with the domain. section 2.6.2
Before modeling, check whether the dataset can support the intended decision:
- Confirm that the required data is accessible to the people and systems that need it, whether it resides in a database, server, cloud service, or file. section 11.3
- Check completeness, relevance, and coverage of business variability. Include enough history to capture effects such as seasonality when they matter. section 11.3
- Look for missing values and outliers, verify business rules, and plan how the data will be refreshed after deployment. section 11.3
Customer reviews as a dataset
For sentiment analysis, collect reviews from sources such as websites or surveys. Store review attributes—including rating, title, date, and body—in a pandas DataFrame, then save and reload the data as a CSV file. Inspect its shape and rating distribution before cleaning the review text and computing sentiment. Comparing star ratings with sentiment results helps put the outputs in context. section 7.15
# Create a dataframe here.
reviews_df = pd.DataFrame()
for review in reviews:
rating = review.xpath(xpath_rating)
title = review.xpath(xpath_title)
author = review.xpath(xpath_author)
date = review.xpath(xpath_date)
body = review.xpath(xpath_body)
helpful = review.xpath(xpath_helpful)
review_dict = {'rating': rating,
'title': title,
'author': author,
'date': date,
'body': body,
'helpful': helpful}
reviews_df = reviews_df.append(review_dict, ignore_index=True)
all_reviews = pd.DataFrame()
The Python listing shows how extracted review fields become rows in a pandas DataFrame. section 7.15
