concept character in category python

This is an excerpt from Manning's book Tiny Python Projects: Learn coding and testing with puzzles and games. Login to get full access to this book.

The 1 is a string. It may look like a number to you, but it is actually the character '1'. That is a valid string value, so our program accepts it.

Figure 8.9 map() will apply a given function to each element of an iterable. A string will be processed as a list of characters.

When we change a character, what will we change it to? For this, we’ll use the string module. I encourage you to take a look at the documentation by importing the module and reading help(string): >>> import string >>> help(string)

One way to choose the characters to change would be to mimic method 1 in chapter 8. We could iterate through each of the characters in our text and select a random number to decide whether to keep the original character or change it to some randomly selected value. If our random number is less than or equal to our mutations setting, we should change the character:

new_text = ''                        #1
for char in args.text:               #2
    new_text += random.choice(alpha) if random.random() <= args.mutations else char                       #3
print(new_text)                      #4

copy

We used the random.choice() function in abuse.py in chapter 9 to randomly select one value from a list of choices. We can use it here to select a character from alpha if the random.random() value falls within the range of the args.mutation value (which we know is also a float).

The problem with this approach is that, by the end of the for loop, we are not guaranteed to have made exactly the correct number of changes. That is, we calculated that we should change 9 characters out of 44 when the mutation rate is 20%. We would expect to end up changing about 20% of the characters with this code, because a random value from a uniform distribution of values between 0 and 1 should be less than or equal to 0.2 about 20% of the time. Sometimes we might end up only changing 8 characters or other times we might change 10. Because of this uncertainty, this is approach would be considered non-deterministic.

This is an excerpt from Manning's book Natural Language Processing in Action: Understanding, analyzing, and generating text with Python. Login to get full access to this book.

We won’t stay in this confusing binary world of logic for long, but let’s imagine we’re famous World War II-era code-breaker Mavis Batey at Bletchley Park and we’ve just been handed that binary, Morse code message intercepted from communication between two German military officers. It could hold the key to winning the war. Where would we start? Well the first step in our analysis would be to do something statistical with that stream of bits to see if we can find patterns. We can first use the Morse code table (or ASCII table, in our case) to assign letters to each group of bits. Then, if the characters are gibberish to us, as they are to a computer or a cryptographer in WWII, we could start counting them up, looking up the short sequences in a dictionary of all the words we’ve seen before and putting a mark next to the entry every time it occurs. We might also make a mark in some other log book to indicate which message the word occurred in, creating an encyclopedic index to all the documents we’ve read before. This collection of documents is called a corpus, and the collection of words or sequences we’ve listed in our index is called a lexicon.

If we’re lucky, and we’re not at war, and the messages we’re looking at aren’t strongly encrypted, we’ll see patterns in those German word counts that mirror counts of English words used to communicate similar kinds of messages. Unlike a cryptographer trying to decipher German Morse code intercepts, we know that the symbols have consistent meaning and aren’t changed with every key click to try to confuse us. This tedious counting of characters and words is just the sort of thing a computer can do without thinking. And surprisingly, it’s nearly enough to make the machine appear to understand our language. It can even do math on these statistical vectors that coincides with our human understanding of those phrases and words. When we show you how to teach a machine our language using Word2Vec in later chapters, it may seem magical, but it’s not. It’s just math, computation.

Zipf’s law can help you predict the frequencies of all sorts of things, including words, characters, and people.

Listing 9.17. One-hot encoder for characters

>>> import numpy as np
 
>>> def onehot_encode(dataset, char_indices, maxlen=1500):
...     """
...     One-hot encode the tokens
...
...     Args:
...         dataset  list of lists of tokens
...         char_indices
...                  dictionary of {key=character,
...                                 value=index to use encoding vector}
...         maxlen  int  Length of each sample
...     Return:
...         np array of shape (samples, tokens, encoding length)
...     """
...     X = np.zeros((len(dataset), maxlen, len(char_indices.keys())))
...     for i, sentence in enumerate(dataset):
...         for t, char in enumerate(sentence):
...             X[i, t, char_indices[char]] = 1
...     return X                                 #1

copy

This is an excerpt from Manning's book Get Programming: Learn to code with Python. Login to get full access to this book.

To understand this, consider the following metaphor. There are two people: an author of fiction and a journalist. The author of fiction is someone who comes up with a plot, characters, dialogue, and interactions and then puts these ideas together in interesting ways by using the rules of the English language. The author writes a story for people to enjoy. The journalist doesn’t need to employ their creative side but rather hunts down stories based on fact. The journalist then puts the facts on paper, also using the rules of the English language, for people to be informed.

Working with sequences of characters is common. These sequences are known as strings, and you can store any sequence of characters inside a string object: your name, your phone number, your address including the new lines, and so on. Storing information in a string format is useful. You can do many operations after you have data represented in a string. For example, if you and a friend are both doing research for a project, you can take notes on separate concepts and then combine your findings. If you’re writing an essay and discover that you overused a word, you can remove all instances of that word or replace some instances with another word. If you discover that you accidentally had Caps Lock on, you can convert the entire text to lowercase instead of rewriting.

Figure 7.2. Three examples of slicing into the string “Python rules!”. The numbered circles on each row indicate the order in which Python retrieves the characters from the string to form a new substring from the slice.

7.1. Strings as sequences of characters

In lesson 4, you learned that a string is a sequence of characters and that all characters in that string are denoted inside quotation marks. You can use either " or ' as long as you’re consistent for one string. The type of a string in Python is str. The following are examples of string objects:

Listing 17.2. A for loop that iterates over each index in a string

my_string = "Python is fun so far!"                        #1
len_s = len(my_string)                                     #1
for i in range(len_s):                                     #2
    print("the character is", my_string[i])                #3

copy

This is an excerpt from Manning's book Deep Learning with Python. Login to get full access to this book.

A dataset of tweets, where we encode each tweet as a sequence of 280 characters out of an alphabet of 128 unique characters. In this setting, each character can be encoded as a binary vector of size 128 (an all-zeros vector except for a 1 entry at the index corresponding to the character). Then each tweet can be encoded as a 2D tensor of shape (280, 128), and a dataset of 1 million tweets can be stored in a tensor of shape (1000000, 280, 128).

This chapter explores deep-learning models that can process text (understood as sequences of words or sequences of characters), timeseries, and sequence data in general. The two fundamental deep-learning algorithms for sequence processing are recurrent neural networks and 1D convnets, the one-dimensional version of the 2D convnets that we covered in the previous chapters. We’ll discuss both of these approaches in this chapter.

Of course, one-hot encoding can be done at the character level, as well. To unambiguously drive home what one-hot encoding is and how to implement it, listings 6.1 and 6.2 show two toy examples: one for words, the other for characters.

Figure 8.1. The process of character-by-character text generation using a language model

Listing 8.3. Vectorizing sequences of characters

maxlen = 60 #1 step = 3 #2 sentences = [] #3 next_chars = [] #4 for i in range(0, len(text) - maxlen, step): sentences.append(text[i: i + maxlen]) next_chars.append(text[i + maxlen]) print('Number of sequences:', len(sentences)) chars = sorted(list(set(text))) #5 print('Unique characters:', len(chars)) char_indices = dict((char, chars.index(char)) for char in chars) #6 print('Vectorization...') x = np.zeros((len(sentences), maxlen, len(chars)), dtype=np.bool) #7 y = np.zeros((len(sentences), len(chars)), dtype=np.bool) #7 for i, sentence in enumerate(sentences): #7 for t, char in enumerate(sentence): #7 x[i, t, char_indices[char]] = 1 #7 y[i, char_indices[next_chars[i]]] = 1 #7 #1 - You’ll extract sequences of 60 characters. #2 - You’ll sample a new sequence every three characters. #3 - Holds the extracted sequences #4 - Holds the targets (the follow-up characters) #5 - List of unique characters in the corpus #6 - Dictionary that maps unique characters to their index in the list “chars” #7 - One-hot encodes the characters into binary arrays

copy

This is an excerpt from Manning's book Hello App Inventor!: Android programming for kids and the rest of us. Login to get full access to this book.

Throughout the book, when you come to a tutorial, we’ll tell you whether its functionality works fully on the emulator or the phone only (using the Phones Only Android symbol from “How to Use this Book”). For example, one of the games you’re going to program lets you move characters by physically tilting the phone from side to side—which isn’t going to work on a computer. Believe us, picking up your computer screen and tilting doesn’t work.

Learning Point: Characters and bytes

Have you noticed that Gentoo and the Yeti face forward rather than left or right? That helps you because, as you saw in the Hungry Spider app, if you want sprites that can face left and right, you have to produce two sets of images and write blocks of code that can switch between them depending on which direction the sprite is travelling. Having the characters face the player means you cut down on a lot of coding! If you’re familiar with MIT’s Scratch programming interface (http://scratch.mit.edu/), you know there are sprite controls to do this hard work automatically—hopefully we’ll see something similar in App Inventor very soon.