CommitmentBank is a natural-language inference dataset of 1,200 natural discourses. Each discourse ends with a sentence containing a clause-embedding predicate beneath an entailment-canceling operator. section 8.2
Each example supplies:
- Premise and hypothesis strings for paired-sentence classification. section 8.2
- An integer index and a label: entailment (0), contradiction (1), or neutral (2). section 8.2
- Operators such as questions, modals, negation, and conditional antecedents that cancel entailment. section 8.2
Listing 8.1 Using Ray data with Hugging Face
datasets = load_dataset("super_glue", "cb") #1
ray_datasets = { #2
"train": ray.data.from_items(hf_dataset["train"].to_list()),
"validation": ray.data.from_items(hf_dataset["validation"].to_list()),
"test": ray.data.from_items(hf_dataset["test"].to_list()),
}
Load the SuperGLUE cb configuration, then convert its train, validation, and test splits into Ray-native datasets for distributed processing. section 8.2
Listing 8.2 Tokenizing input sentences
def tokenize_fn(samples):
outputs = tokenizer(
samples["premise"],
samples["hypothesis"],
truncation=True,
padding="longest",
return_tensors="pt"
)
outputs["labels"] = torch.tensor(samples["label"], dtype=torch.long) #1
outputs = {key: value.to('cuda') for key, value in outputs.items()} #2
return outputs
Tokenize premise–hypothesis pairs, attach integer labels, and move the resulting tensors to CUDA for supervised model training. section 8.2
