5 Exploring, Tokenizing, and Visualizing Hugging Face Datasets

 

This chapter covers

  • What is Hugging Face Datasets?
  • How to programmatically download Hugging Face Datasets
  • What is tokenization and how you can apply it on Hugging Face datasets
  • How to perform data visualization on Hugging Face datasets

Hugging Face is an AI platform that develops, trains, and deploys cutting-edge open-source machine learning models. Alongside providing a hub for these trained models, Hugging Face also hosts a wide array of datasets (available at https://huggingface.co/datasets), which you can leverage for your own projects.

We will now guide you through accessing datasets from Hugging Face and show you how to programmatically download them to your local machine. You will also gain a deeper understanding of tokenization, including how to tokenize datasets and prepare your data for fine-tuning, which we will cover in chapter 6 we will explore how to visualize various datasets with Hugging Face.

5.1 What is Hugging Face Datasets?

5.1.1 Getting the list of datasets available

5.1.2 Validating the availability of a dataset

5.1.3 Downloading a dataset

5.1.4 Shuffling the Dataset

5.1.5 Streaming the Dataset

5.1.6 Getting the Parquet files of a dataset

5.2 Tokenization in NLP

5.2.1 Types of Tokenization Methods

5.2.2 Tokenizing Hugging Face Datasets

5.3 Visualizing the Datasets

5.3.1 Using the Twitter Financial News Topic dataset

5.3.2 Using the CIFAR-10 dataset

5.4 Summary