liveBook · Manning

Extractive question answering

As explained in AI-Powered Search

Extractive question answering selects and returns relevant words, sentences, or passages directly from source documents.

A searcher asking a question may want an answer, not a list of documents to read. Extractive question answering finds relevant words, sentences, or passages in search results and returns them directly. It searches and ranks answers within documents, making it more targeted than returning documents alone. section 1.3.5 section 14.0

Unlike abstractive question answering, extraction does not synthesize a new response: the answer is taken from the searched documents. Its usefulness therefore depends on finding context that contains an answer; the returned span is a likely answer, not a guarantee. section 1.3.5 section 14.1.1

A model can select a short answer span from a question and a retrieved context:

  • Tokenize the question and context so the model can work with their constituent tokens. section 14.1.1
  • Estimate the probability that each token starts the answer and the probability that each token ends it. These are two probability mass functions over discrete tokens. section 14.1.1
  • Choose start and end positions and extract the corresponding tokens as the answer. The demonstrated implementation identifies the strongest normalized start and end positions. section 14.1.1

Finding a definition

For the question “What are minimalist shoes?”, a retrieved passage describes several features of the shoes. The model can select the shorter span “shoes intended to closely approximate barefoot running conditions” rather than returning the entire passage. section 14.1.1

Listing 14.1 Loading the tokenizer and model
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
model_name = "deepset/roberta-base-squad2"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForQuestionAnswering.from_pretrained(model_name)

Load a pretrained question-answering model and its tokenizer with the Transformers library before processing a question and context. section 14.1.1

Listing 14.3 Identifying an answer span from a tokenized context
start_tokens = []
end_tokens = []
terms = tokenizer.convert_ids_to_tokens(input_ids)
start_token_id = 0
end_token_id = len(terms)
for i, term in enumerate(terms):
  start_tokens.append(stylize(term, [0, 127, 255], start_logits_norm[0][i]))
  end_tokens.append(stylize(term, [255, 0, 255], end_logits_norm[0][i]))
  if start_logits_norm[0][i] == 1.0:
    start_token_id = i
  if end_logits_norm[0][i] == 1.0:
    end_token_id = i + 1

answer = terms[start_token_id:end_token_id]
display(HTML(f'<h3>{clean_token(" ".join(answer))}</h3>'))  #1
display(HTML(f'<pre>{" ".join(start_tokens)}</pre>'))  #2
display(HTML(f'<pre>{" ".join(end_tokens)}</pre>'))  #3

Inspect the example Python code for identifying start and end tokens and extracting the answer span. section 14.1.1

How AI-Powered Search covers it

How deep it goes
Explains token-level start and end probabilities and demonstrates selecting an answer span with Python.
Examples
Uses a minimalist-shoes question and a retrieved definition to illustrate answer extraction.
What it assumes
Builds on Transformer-based semantic search while explaining answer-span probabilities in terms of discrete tokens.
What you can do after
Apply it via the Transformers library to select an answer span from a question and context.
How it connects
Connects search ranking and Transformer-based question answering with abstractive question answering and generative search.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI-Powered Search. The section links point to the parts of the book it draws on.