‹ Culture 3.14 · What we ask ourselves
Question · Decisions
When does a tabular model deliver more value than a language model?
On tabular data with labels and a known cost of error, gradient boosting still wins on accuracy, cost and the ability to analyse the result.
The question often arrives in the form of a proposal: “a language model could solve this”. Sometimes it can. In classification and prioritisation over tabular data, in our tests, almost never.
We set up a comparison over a tabular set with historical labels and a known cost of error. A gradient boosting model trained on the available variables, against a language model handed the same row serialised and a carefully written instruction.
The tabular model performed better at a cost per inference several orders of magnitude lower, and above all it made it possible to analyse why. Variable importance, calibration by segment and a cost matrix are tools we do not have in the other case.
The language model was better at one thing: the cases with free text that no variable captured. There it contributed real signal, and we built it in as one more variable instead of replacing the model.
The conclusion we take away is not about CatBoost. It is about the order of the questions: first which decision has to improve and what information is available when it is taken; then the technique. When labels exist, the task is stable and the cost of the error is known, a model learned on that data is still hard to beat.