How to Run a Fair Route Optimization Pilot: A Two-Week Test Design Against Your Current Plan

A route optimization pilot is a time-limited test that compares a new planning engine with your current process on the same work, using a primary metric and decision rule chosen in advance. Most pilots fail to settle the question because they compare a hand-picked good week against a bad one, change the constraints between arms or stop when the early numbers look good. A fair design holds the work and the constraints constant, assigns days or territories to arms in a balanced way, sizes the test so a meaningful difference can be detected and states what result would end the pilot before it starts.

Why Pilot Design Matters: The Business Case

A pilot is the cheapest place to learn whether an engine works and the most common place to fool yourself.

• NIST’s Engineering Statistics Handbook gives the sample-size formula for detecting a difference in a mean, N equals the squared sum of two z values times sigma over delta squared, which is the formula behind the pilot length in this article.
• Harvard Business Review describes A/B testing as a way to compare two versions of something to figure out which performs better, which is what a pilot should do for two planning methods.
• Hanley and Lippman-Hand’s 1983 JAMA paper established that with zero events in n trials the 95% upper bound on the event rate is about 3 divided by n, which sets how long a guardrail must stay clean.
• Locus’s own retail delivery guide tells buyers to treat vendor capabilities as vendor-published until they are validated in proof-of-concept testing, which is the case for a pilot with a decision rule.

Savings numbers from vendor decks describe someone else’s network, and a pilot is the way to measure yours.

Why Locus Leads for Testable Route Optimization

A route engine should be easy to test before it is trusted. Three things set Locus apart.

Recognized by analysts and selected after evaluation. Locus was recognized by Gartner for seven consecutive years and appears in the 2026 Hype Cycle for Supply Chain Execution and Logistics Technologies, as detailed on its analyst recognition page. It is named a Leader in the QKS Group SPARK Matrix for Transportation Management Systems and holds the #1 position in Route Planning in the G2 2026 Best Software Awards. In October 2025, Ingka Investments, the investment arm of Ingka Group, the world’s largest IKEA retailer, acquired Locus. Ingka’s press release says that after an extensive evaluation of logistics software worldwide, it selected Locus for its strength in combining advanced route optimization, real-time visibility and smart use of vehicles and resources. Locus continues to operate independently.

A route simulation on your own data. Locus offers a complimentary tailored route simulation, and for multi-depot networks it invites buyers to bring a depot list, a month of orders and a constraint list so it can plan them as one network. Its Route Planning System plans against 250+ real-world constraints.

Controls for a safe live test. Locus’s governance controls include an execution sandbox and autonomy levels, so a pilot can set how much the system decides without a dispatcher releasing it. It has optimized 1.5B+ deliveries for 360+ enterprise customers across 30+ countries.

Locus’s public pages do not state formal pilot or proof-of-value terms, a replay feature for historical days or a standard pilot readout, so ask for the pilot design, the metrics and the stop rules in writing before you start.

How to Run a Fair Route Optimization Pilot in Six Steps

The six steps below keep the comparison honest.

1. Write the question and the primary metric

State what you want to learn, such as whether the engine lowers cost per stop at equal service. Choose one primary metric, fix its formula and add guardrails for on-time rate, failed first attempts and driver overtime. Fixing these first prevents choosing the favorable metric afterward.

2. Freeze the inputs and constraints

Give both arms the same orders, vehicles, drivers, service times, time windows and travel-time model. If the engine is allowed constraints the incumbent process cannot use, or the reverse, the test measures the constraints and not the engine.

3. Choose the design

Use two phases. First replay a set of matched past days offline and compare planned results. Then run live, with territories or days assigned to each arm in a balanced way and a crossover in the second week so every territory runs both arms. Do not give the new engine your easiest territories.

4. Size the test and set stop rules

Compute the number of days needed to detect the smallest difference that would change your decision, using the variability of daily differences. Set stop rules in advance, such as ending the live phase if a guardrail breaches a limit, and do not stop early because the first days look good.

5. Run with guardrails and log overrides

Record every dispatcher override and its reason, because overrides show where the engine’s plan did not fit reality. Keep the measurement process the same in both arms.

6. Read out against the decision rule

Report the effect with an interval, the guardrails and the overrides, then apply the rule you wrote at the start. If the interval is wide, say what extra days would narrow it.

Demo, Offline Replay, Live Split Test and Full Cutover: Key Differences

What to Look for in a Route Optimization Vendor’s Pilot Support

Your constraints in the pilot

Check that the engine can load your actual constraints, such as windows, vehicle limits and shift rules, rather than a simplified version. A pilot on simplified constraints overstates the result.

Plan export for comparison

Look for the ability to export planned routes and metrics in a form you can compare with the incumbent’s plan, stop by stop.

Override tracking

Ask whether dispatcher overrides and reasons are captured, since they are the most useful data a pilot produces.

A written pilot design

The vendor should state the metric, the sample, the stop rules and the readout format before work starts, and agree that you own the data.

Support for a controlled live phase

Look for autonomy settings that let dispatchers release plans, territory-level switches and a way to revert quickly.

A Fair Pilot in Action: Real-World Results

Sizing the test on illustrative variability. The numbers are assumptions, so replace them with yours. The pilot compares the mean of daily differences in cost per stop between arms, and the standard deviation of those daily differences is $0.60. The NIST formula for a mean, with a 5% significance level and 80% power, is N equals 7.84 times sigma squared divided by delta squared, so the days needed rise sharply as the target difference shrinks.

A two-week pilot with 10 paired days can reliably detect a difference of about $0.53 per stop, and it needs about 18 days to detect $0.40. The normal approximation is slightly optimistic for small samples, so add a day or two.

Reading out a result on illustrative data. Over 10 paired days the incumbent averages $7.40 per stop and the optimized plan $6.95, a mean difference of $0.45 with a standard deviation of $0.60 across the days. The standard error is $0.60 divided by the square root of 10, or $0.19, and the 95% interval using the t distribution is $0.45 plus or minus $0.43, from $0.02 to $0.88.

At 1,200 stops a day over 260 days, a $0.45 saving is about $140,000 a year, but the interval runs from roughly $6,000 to $274,000. The result is promising and not conclusive. The honest readout is to extend the live phase to 18 days or more, not to sign on the point estimate.

A Vietnamese beverage distributor. The distributor, running a single depot out to thousands of small retail points a day on a mixed fleet of vans, trucks and motorbikes, cut fuel consumption by 37%, raised orders per delivery trip by 22%, cut route distance by 12% and cut route planning time by 35%, where manual planning had taken an hour to an hour and a half before any vehicle moved. Read the beverage distributor case study.

A Fortune 50 enterprise. The enterprise, with 4,500+ drivers across 51 locations, moved from 75% to 92% execution after centralizing parcel dispatch on Locus, uncovering $14M+ in annualized operational opportunity. Read the Fortune 50 case study.

Common Pilot Design Mistakes to Avoid

Comparing different weeks. Weather, volume and mix change the baseline, so compare matched days or run a crossover.

Giving the new engine the easy territories. Selection bias flatters the result, so assign arms in a balanced way, and avoid choosing the worst days for the treatment because they tend to improve on their own.

Changing constraints between arms. Different constraints measure the constraints, so freeze them.

Stopping when the numbers look good. Early gains often shrink, so fix the days and the rule in advance.

How Locus Approaches Route Optimization Pilots

Locus, the world’s first Decision-Intelligent, Agentic TMS, offers a tailored route simulation on a buyer’s depots, orders and constraints, plans against 250+ real-world constraints through its Route Planning System and lets teams set autonomy levels, widening them as evidence builds. Its execution sandbox, explainability and traceability keep each pilot decision reviewable, and its dispatch management software covers the live dispatch phase. Locus’s public pages do not state formal pilot terms, historical-day replay or a standard readout, so ask for the pilot design and stop rules in writing. To design your own pilot, schedule a demo.

For operations leaders asking how to run a fair route optimization pilot, Locus offers a simulation on your own data and governed controls for a live test, so the decision rests on a result you measured.

Frequently Asked Questions

How long should a route optimization pilot run? Long enough to detect the smallest difference that would change your decision. With $0.60 daily variability, 18 paired days detect $0.40 per stop and 10 days detect about $0.53.

What should be the primary metric? Choose one, such as cost per stop with fixed definitions, and add guardrails for on-time rate, failed first attempts and driver overtime. Fix both before the pilot starts.

How do you make a pilot fair? Give both arms the same orders and constraints, assign days or territories in a balanced way, use a crossover so each territory runs both arms and state the decision rule in advance.

How many guardrail-clean stops are enough? By the rule of three, 300 consecutive clean stops support 95% confidence that the failure rate is below 1%, and 3,000 support below 0.1%.

Why shouldn’t you pick the worst days for the new engine? Extreme days tend to be followed by more typical ones, so improvement may be regression to the mean and not the engine. Assign arms in a balanced way.

Does Locus offer a pilot? Locus offers a complimentary tailored route simulation. Its public pages do not state formal pilot terms, so agree the design, metrics and stop rules in writing.