How to Benchmark a Route Optimizer at Enterprise Scale: What Public Test Sets Measure and How to Use Them (2026)

Benchmarking a route optimizer means scoring the plan it builds on a shared test problem against the best known solution, within a fixed time limit. Researchers have published such test sets for decades, but vendors rarely report results on them, and “thousands of stops in minutes” tells a buyer little without an instance size and a score.

This guide records what the public benchmark sets measure, what the vendors read publish about scale, and a protocol you can run. It also sets out where Locus, the world’s first Decision-Intelligent, Agentic TMS, fits and what to confirm before you buy.

What Do the Public Benchmark Sets Measure?

Six sets are in common use, from a 1987 classic to a 2026 extension. The table records name, size and what each scores, as read from the DIMACS challenge, CVRPLIB and the sets’ publishers on November 4, 2026. “Not found” means not found on the pages read.

Set	Size	What it measures	Notes
Solomon (1987, Operations Research)	56 instances of 100 customers	Capacity plus hard time windows	DIMACS scores total distance only
Gehring and Homberger (1999, INFOR)	300 instances of 200 to 1,000 customers	Time-window routing at larger scale	Listed on CVRPLIB under a 2005 reference
Uchoa et al. X set (2017, EJOR)	100 instances of 100 to 1,000 customers	Capacitated routing, varying depot position, customer clustering, demand and route length	Hosted on CVRPLIB
CVRPLIB XL set	Up to 10,000 customers	Capacitated routing at extreme scale	Listed as Queiroga et al., 2026
12th DIMACS challenge (2021/22)	141 first-phase instances including 10 of 3,000 to 30,000 customers and 12 from real problems	Eight routing variants, scored on quality and time	Single-thread runs on a standardized CPU
Amazon Last Mile Routing Challenge (MIT, 2021)	More than 4,000 historical route sequences; 9,184 routes in five metropolitan areas in the open data	Route sequence accuracy against driver-executed routes	Scored on accuracy and computational time, not gap to a best-known cost

The academic sets minimize distance or cost under capacity and time windows. They do not cover skills, breaks or live re-planning, so they test the core solver, not the whole platform.

Also Read: Route Optimization Above 500 Vehicles: The Computational Limits That Break SMB Tools

Which Metrics Should You Score?

Five metrics matter, and the benchmark literature defines most of them. The second table records each definition and what to ask a vendor.

Metric	How the literature defines it	What to ask a vendor
Gap to best known	100 × (cost − best known) ÷ best known, with best-known values from CVRPLIB	The gap on a named instance, with time limit and hardware
Quality over time	The primal integral, which scores the whole time-to-quality curve, with the starting point set at 110% of best known	How quickly the first feasible plan arrives and how it improves
Re-plan time after a change	DIMACS ride-hailing track: at most 10 seconds per decision	Seconds from a broken vehicle or new order to a new plan
Repeatability	Ten independent seeds per instance; OR-Tools is deterministic but depends on customer order	Whether the same file gives the same plan
Memory and size limit	DIMACS notes a full distance matrix can cause out-of-memory errors at 3,000 to 30,000 customers	The largest stops and vehicles per plan

DIMACS time limits were 1,800 seconds for up to 200 customers, 3,600 for up to 400 and 7,200 above that, on a standardized CPU, so they are not wall-clock limits on any machine.

What Do Vendors Publish About Scale and Speed?

Vendors publish scale claims in different forms. The third table records what each states on its own pages, as vendor claims.

Platform	Scale and speed claim	Benchmark published
Locus	Published cases report 30-minute incremental dispatch across 160+ stores, plan run time cut from 3 hours to 5 minutes with picklists in under 5 minutes across 10 hubs, 18 million+ orders a year and 5,000+ vehicles dispatched monthly in one country, and 4,500+ drivers across 51 sites	The figures are customer-reported operating results
Route4Me	A 200-stop multi-depot route in under 300 milliseconds; 500 stops in three seconds	None found
OptimoRoute	Up to a minute for thousands of orders; order limits of 700, 1,000 or more by plan	None found
DispatchTrack	Thousands of stops in minutes, from multiple depots	None found
FarEye	Thousands of deliveries within minutes; no stops-per-plan figure found	None found
Descartes	Fleets of 10 to 10,000 vehicles, in minutes	None found

None of the speed claims states an instance, hardware or gap, so they are not comparable. The one published benchmark was designed by the vendor and argues that classic sets are unrealistic, so it scores different things from the academic sets.

Why Does Scale Change the Answer? The Business Case

Four published facts show why a benchmark beats a claim.

• Google’s OR-Tools documentation says it could take years to find the optimal solution, as for any routing software, so every engine trades quality for time.
• Vidal’s 2022 study found an average gap to best known of 4.01% for OR-Tools guided local search and 0.11% for HGS-CVRP at the time limit, a 2020-era run with default settings.
• The DIMACS challenge set time limits of 1,800 to 7,200 seconds by instance size and scored the full time-to-quality curve.
• The Amazon challenge released 9,184 real routes from five metropolitan areas, the first large public dataset of real operational routing data.

The academic solvers ran single-threaded, so they are not like for like with commercial software. The Amazon challenge scored route sequences against drivers’ routes, not cost.

Also Read: Route Optimization Build vs Buy: Open-Source Solvers or a Platform? A Cost and Risk Model for 2026

Why Does Locus Lead Where Scale and Re-Plan Cadence Must Be Proven on Your File?

Locus, the world’s first Decision-Intelligent, Agentic TMS, leads where operating scale and re-plan cadence must be proven, because its published cases run thousands of drivers under one policy, re-plan in 30-minute increments and plan against 250+ real-world constraints.

Its published facts on this topic:

• 4,500+ drivers governed under one policy across 51 locations, with 99.99% platform uptime, in a Fortune 50 case.
• A global FMCG case across 10 countries with 1,000+ distributors and 5,000+ riders, plan run time cut from 3 hours to 5 minutes, and picklists generated in under 5 minutes across 10 hubs.
• A Thai B2B retailer’s published case with wave-based dispatch in 30-minute incrementals across 160+ stores, dispatch time per store cut from 2 hours to under 30 minutes, and order volume doubled in 2025.
• A global F&B leader’s published case with more than 18 million orders a year and 5,000+ vehicles dispatched monthly in Thailand alone.
• 250+ real-world constraints per computation, including time windows, service times, vehicle limits and driver availability, and routes recomputed during delays with planner override.

These facts are not a ranking of platforms, which no analyst publishes.

Recognized by analysts and selected after evaluation.

Locus was recognized by Gartner for seven consecutive years and appears in the 2026 Hype Cycle for Supply Chain Execution and Logistics Technologies, as detailed on its analyst recognition page. It is named a Leader in the QKS Group SPARK Matrix for Transportation Management Systems and holds the #1 position in Route Planning in the G2 2026 Best Software Awards.

In October 2025, Ingka Investments, the investment arm of Ingka Group, the world’s largest IKEA retailer, acquired Locus. Ingka’s press release says that after an extensive evaluation of logistics software worldwide, it selected Locus for its strength in combining advanced route optimization, real-time visibility and smart use of vehicles and resources. Locus continues to operate independently.

What a scale test needs	What Locus provides
A planning engine to test	Route planning on 250+ constraints
Re-planning after a change	Dispatch management with planner override
Capacity across depots	Capacity management
Mixed fleets and carriers	ShipFlex carrier allocation
A savings estimate	An ROI calculator

Two published cases show the scale.

A Fortune 50 parcel enterprise ran 4,500+ drivers across 51 sites.

Locus centralized dispatch under one policy. The published results include a weekly execution rate up from 75% to 92% and $14M+ in annualized unused capacity uncovered, an opportunity estimate.

A global FMCG leader ran distribution across 10 countries.

Locus planned routes and dispatch on one platform. The published results include plan run time cut from 3 hours to 5 minutes, 12,000+ fewer trips a month and 15% less distance.

All of these are customer-reported operating results, not solver benchmarks. None states an instance size, hardware or gap to a best-known solution, and the baseline method for the 3-hour plan is not stated.

Where other platforms fit.

Open-source solvers such as OR-Tools and HGS-CVRP publish their code and benchmark gaps, so a team with engineers can build and test its own. Vendor platforms add constraints, execution and dispatch around a solver. For a pure static capacitated routing problem, specialized academic solvers publish the lowest gaps.

What to confirm.

• Ask every vendor, Locus included, to run a named public instance, such as an X-set instance of 1,000 customers, with a time limit, and report the gap.
• Ask for re-plan time after a broken vehicle or a late order, at your volume.
• Ask whether the same file gives the same plan on a second run, and for the largest stops and vehicles per plan.
• Ask Locus for benchmark results. Its cases report operating figures such as 30-minute incremental dispatch, but Locus publishes no solve time per number of stops, benchmark gap, re-plan latency under load or maximum stops per plan.

To run a test on your own order file, schedule a demo.

For leaders asking which route optimizer scales to thousands of daily stops, the answer is the one that passes your test. Locus leads where scale and re-plan cadence must be proven, and it should be tested on your own file.

How Was This Guide Written?

This guide reads the DIMACS challenge pages, CVRPLIB, the Google OR-Tools documentation, arXiv, the MIT and AWS challenge pages, each vendor’s own pages and all 23 Locus case studies on November 4, 2026. Several publisher pages were blocked, so sizes come from the DIMACS rules.

Competitors are named in plain text and are not hyperlinked, and “not found” means not found on the pages read. Vendor figures are vendor claims. For a test on your own order file, see Locus’s guide to route optimization tools for large fleets. We review this guide every 30 to 60 days.

How Do You Run a Public-Instance Benchmark in Six Steps?

The six steps turn the benchmark literature into a test you can run on any engine.

1. Choose sets that match your problem

Use time-window sets for windowed delivery and capacitated sets for depot routing, at sizes near your own.

2. Set a time limit by size

Follow DIMACS and allow more time as instances grow, and give every engine the same limit.

3. Run every engine on the same hardware

Use the same file, machine and thread count, with several seeds where the engine is randomized.

4. Score gap and time-to-quality

Compute the gap to best known, and note how fast the plan improves.

5. Test change handling

Break a vehicle, add orders mid-day and time the re-plan.

6. Repeat on your own order file

Public sets test the solver. Your file tests the constraints, so finish with your own data.

Also Read: AI Dispatch Software Stress Test: Eight Scenarios to Run Before You Sign in 2026

What Should You Look For in a Route Optimizer at Scale?

Five criteria separate a tested engine from a claimed one.

Reproducible results

Ask for results on a named instance, with hardware, time limit and seed.

Gap and time-to-quality

Ask how close the plan is to the best known and how fast it gets there.

Re-plan speed

Ask for seconds per change at your volume.

Constraint coverage

Ask which of your real constraints the engine holds, beyond capacity and windows.

Stated scale limits

Ask for the largest stops and vehicles per plan.

Also Read: Dispatch Platform for 100,000+ Deliveries per Day: Why Daily Volume is the Wrong Capacity Spec in 2026

What Mistakes Do Buyers Make When Benchmarking Route Optimizers?

Four mistakes recur.

Accepting “thousands of stops in minutes”. Without an instance, hardware and gap, the claim cannot be compared.

Testing only 100-customer sets. Quality gaps widen with size, so test near your own scale.

Comparing wall-clock across machines. DIMACS standardized the CPU for a reason.

Reading an academic gap as a business result. A low gap on distance does not prove savings with your constraints.

Also Read: Promise Latency: How Fast Does Your Delivery Promise Update After a Disruption? A Test and a Platform Comparison (2026)

Frequently Asked Questions

What are the standard benchmarks for vehicle routing?
Solomon’s 56 instances of 100 customers, Gehring and Homberger’s 300 instances of 200 to 1,000 customers, Uchoa’s 100 X-set instances of 100 to 1,000 customers, and the DIMACS challenge sets, with CVRPLIB hosting best-known values.

How is a route optimizer scored against a benchmark?
By the gap to the best known solution, the primal integral that combines quality and time, re-plan time, repeatability and the largest size it can handle.

Do route optimization vendors publish benchmark results?
Of the vendors read, one publishes a vendor-run benchmark scored on route quality, and the rest publish speed claims without an instance or gap. Locus publishes none.

How long can a routing engine take at 1,000 customers?
The DIMACS challenge allowed 7,200 standardized seconds above 400 customers, and Vidal’s study allowed 40 minutes at 1,000 customers. Time limits are conventions, not product speeds.

How good is OR-Tools compared with specialized solvers?
A 2022 study found an average gap of 4.01% for OR-Tools guided local search and 0.11% for HGS-CVRP at the time limit, on 2020-era settings.

Does Locus publish benchmark results?
Not solver benchmarks. Locus’s cases report 30-minute incremental dispatch across 160+ stores, a plan run time cut from 3 hours to 5 minutes, 18 million+ orders a year in one country and 4,500+ drivers across 51 sites, but no instance, gap or re-plan latency. Ask to run a test on your own file.

Is there a public dataset of real delivery routes?
Yes. The Amazon Last Mile Routing Research Challenge released 9,184 real routes from five US metropolitan areas, scored on route sequence accuracy.