Splitting breaks text into small, individual units called tokens. It is a standard step in text preprocessing, alongside standardization and indexing, and prepares written language for conversion into numeric data. section 14.2 section 14.6
Split a sentence into words
Using stringr, define a function that detects word boundaries, preserves punctuation, removes whitespace, and returns tokens such as "The", "quick", "brown", and ".". The resulting tokens can then be matched against a vocabulary to obtain integer IDs. section 14.2
Choose the splitting unit according to the representation the model needs:
- Character splitting returns each character, including spaces, as a separate token. section 14.2
- Word splitting separates text at word boundaries; preserving punctuation requires retaining nonword elements and filtering out whitespace. section 14.2
- Subword splitting is another major tokenization type and is usually the most effective when models have sufficient data and capacity. section 14.6
- Regular expressions provide flexible character-pattern matching for more specialized splitting rules. section 14.2
library(stringr)
library(glue)
library(dplyr)
library(keras3)split_chars <- function(text) {
unlist(str_split(text, boundary("character")))
}
Runnable R code defines character splitting with stringr and prepares the packages used by the text-preprocessing example. section 14.2
