liveBook · Manning

TABLET benchmark

As explained in Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI

A benchmark for evaluating large language models on prediction tasks involving tabular data.

The TABLET benchmark evaluates how well large language models perform prediction tasks on tabular data, especially tables combining textual and numerical inputs. section 1.5

LLMs versus XGBoost

Using four-shot examples, ChatGPT averaged an F1 score of 0.68 and Flan-T5 11b averaged 0.66. XGBoost, trained on the entire dataset, averaged 0.94. section 1.5

The benchmark highlights three practical conclusions:

  • Fully supervised tabular models can substantially outperform prompted LLMs on prediction tasks. section 1.5
  • LLMs remain stronger at instruction following and producing textual outputs than at extracting additional predictive information from tabular data. section 1.5
  • Operational concerns include GPU cost, scalability, latency, and throughput. section 1.5

How Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI covers it

How deep it goes
Reports comparative F1 scores using four-shot prompting and full-data XGBoost, but does not detail benchmark construction.
Examples
Compares ChatGPT, Flan-T5 11b, and XGBoost on prediction tasks involving textual and numerical data.
What it assumes
Assumes familiarity with tabular prediction, supervised models, prompting, and F1-score evaluation.
What you can do after
Evaluate whether prompted LLMs are suitable for tabular prediction relative to supervised baselines.
How it connects
Connects large language models with tabular data, supervised learning, XGBoost, prompting, and F1 score.
Cover of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
liveBookmanning.com
Read Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI. The section links point to the parts of the book it draws on.