liveBook · Manning

Sentence complexity

As explained in Learn AI-Assisted Python Programming, Second Edition

Sentence complexity is a text measure calculated from the average number of punctuation-delimited phrases per sentence.

Sentence complexity measures how many punctuation-delimited phrases sentences contain on average. It helps describe an author’s writing style as part of a larger text signature. section 11.1 section 11.3.1

Calculate the measure with a simple rule of thumb:

A short calculation

Suppose two sentences contain three phrases altogether. The sentence complexity is 3/2, or 1.5. A sentence without commas, semicolons, or colons counts as one phrase. section 11.3.1

Listing 11.11 Average number of phrases per sentence
def average_sentence_complexity(text):
    '''
    text is a string of text.
    
    Return the average number of phrases per sentence in text.
    
    >>> average_sentence_complexity('A pearl! Pearl! Lustrous \
pearl! Rare. What a nice find.')
    1.0
    >>> average_sentence_complexity('A pearl! Pearl! Lustrous \
pearl! Rare, what a nice find.')
    1.25                            #1
    '''
    sentences = get_sentences(text)     #2
    total = 0
    for sentence in sentences:               #3
        phrases = get_phrases(sentence)       #4
        total += len(phrases)             #5
    return total / len(sentences)      #6

The implementation separates sentence detection, phrase detection, counting, and averaging, making each step easier to inspect and test. section 11.5.9

How Learn AI-Assisted Python Programming, Second Edition covers it

How deep it goes
Defines a punctuation-based heuristic, works through a calculation, and implements the averaging function without pursuing accurate linguistic parsing.
Examples
Demonstrates three phrases across two sentences and tests texts with punctuation changes that produce averages of 1.0 and 1.25.
What it assumes
Assumes only that readers can recognize sentences and punctuation-delimited phrases; supplies the calculation and supporting function structure.
What you can do after
Build it from scratch by detecting sentences, extracting phrases, counting them, and returning their average.
How it connects
Connects sentence complexity to authorship identification, machine learning, supervised learning, and text signatures.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Learn AI-Assisted Python Programming, Second Edition. The section links point to the parts of the book it draws on.