Rippling Tested 15 AI Models on Real Payroll Data. The Cheapest Tied the Priciest.

Rippling President and CPO Matt MacInnis ran roughly 2,100 graded attempts per model — 15 models in total — against real production payroll and HR data, under strict pass/fail scoring where an unfinished attempt counted as a failure. Jason Lemkin’s writeup of the results collapses a crowded model market to three realistic choices. Opus 4.6 essentially tied GPT-5.5 medium once five months of Opus-specific tuning is factored in — 91.0% pass at $1,453 versus 89.5% pass at $1,435. GPT-5.5 low was fastest at 130 seconds slowest-decile latency, still hitting 88.8% pass at $1,308. GLM 5.2 undercut both on price at the same accuracy tier: 88.7% pass for just $621. Quality flattened across the top seven models — all landing between 88.5% and 89.5% pass rate — while price varied three to seven times over. Two results stood out as warnings against assuming newer means better: Fable 5 matched GPT-5.5 low’s accuracy at 3.3 times the price and double the wait, and re-running the same test set on Grok 4.6 versus Grok 4.5 actually showed accuracy drop while latency nearly doubled.

The finding echoes guidance Anthropic itself has been giving model buyers: its own model-selection guide argues cost-per-task is often lower for more capable models despite higher per-token pricing, and cites an “advisor strategy” — Sonnet 5 paired with a Fable 5 advisor lands within 10% of Fable 5’s own SWE-bench Pro score at 63% of the price. Rippling’s payroll benchmark is the sales-tooling world’s version of the same lesson, run on a workload sales orgs actually care about: don’t assume a frontier-tier price buys frontier-tier accuracy, and don’t assume last quarter’s benchmark holds for this quarter’s model release. MacInnis’s practical takeaway, echoed by Lemkin: re-run your own benchmark on your own data every time a new model ships, because the price-to-accuracy curve moves faster than most procurement cycles do.