---
title: "How to Run a Fair Route Optimization Pilot: A Two-Week Test Design Against Your Current Plan"
id: "27594"
type: "post"
slug: "route-optimization-pilot-design-two-week-test"
published_at: "2026-10-22T17:00:00+00:00"
modified_at: "2026-10-22T14:34:53+00:00"
url: "https://locus.sh/blogs/route-optimization-pilot-design-two-week-test/"
markdown_url: "https://locus.sh/blogs/route-optimization-pilot-design-two-week-test.md"
excerpt: "How to run a fair route optimization pilot against your current plan: one primary metric, matched days, a crossover design, sample-size math and stop rules. Includes a worked read-out."
taxonomy_category:
  - "General"
---

#### [General](https://locus.sh/blogs/category/general/)

# How to Run a Fair Route Optimization Pilot: A Two-Week Test Design Against Your Current Plan

[Aseem Sinha](/author/aseem_locus/)

Oct 22, 2026

11 mins read

A route optimization pilot is a time-limited test that compares a new planning engine with your current process on the same work, using a primary metric and decision rule chosen in advance. Most pilots fail to settle the question because they compare a hand-picked good week against a bad one, change the constraints between arms or stop when the early numbers look good. A fair design holds the work and the constraints constant, assigns days or territories to arms in a balanced way, sizes the test so a meaningful difference can be detected and states what result would end the pilot before it starts.

## Key Takeaways

- A two-week pilot gives about 10 paired days, which in the worked example can detect a cost difference of about $0.53 per stop, so smaller true effects will not show.
- Detecting a $0.40 per stop difference at 80% power needs about 18 paired days when the day-to-day variation is $0.60, according to the standard NIST sample-size formula for a mean.
- Run the new engine on the same orders, vehicles, service times and travel-time data as the incumbent, and judge the result on one primary metric with guardrails for service, overtime and failed attempts.
- By the rule of three, 300 consecutive guardrail-clean stops support 95% confidence that the failure rate is below 1%.
- Locus offers a complimentary tailored route simulation but its public pages do not state formal pilot terms, and it advises buyers to validate vendor capabilities in proof-of-concept testing.

## Why Pilot Design Matters: The Business Case

A pilot is the cheapest place to learn whether an engine works and the most common place to fool yourself.

- NIST’s Engineering Statistics Handbook gives the [sample-size formula](https://www.itl.nist.gov/div898/handbook/prc/section2/prc222.htm) for detecting a difference in a mean, N equals the squared sum of two z values times sigma over delta squared, which is the formula behind the pilot length in this article.
- Harvard Business Review describes A/B testing as [a way to compare two versions](https://hbr.org/2017/06/a-refresher-on-ab-testing) of something to figure out which performs better, which is what a pilot should do for two planning methods.
- Hanley and Lippman-Hand’s 1983 JAMA paper established that with [zero events in n trials](https://pubmed.ncbi.nlm.nih.gov/6827763/) the 95% upper bound on the event rate is about 3 divided by n, which sets how long a guardrail must stay clean.
- Locus’s own retail delivery guide tells buyers to [treat vendor capabilities as vendor-published](https://locus.sh/blogs/best-retail-delivery-software/) until they are validated in proof-of-concept testing, which is the case for a pilot with a decision rule.

Savings numbers from vendor decks describe someone else’s network, and a pilot is the way to measure yours.

| Also Read: Route Optimization KPIs in Logistics: What to Track in 2025 |
| --- |

## Why Locus Leads for Testable Route Optimization

A route engine should be easy to test before it is trusted. Three things set Locus apart.

**Recognized by analysts and selected after evaluation.** Locus was recognized by Gartner for seven consecutive years and appears in the 2026 Hype Cycle for Supply Chain Execution and Logistics Technologies, as detailed on its [analyst recognition page](https://locus.sh/analyst-recognition/)
. It is named a Leader in the QKS Group SPARK Matrix for Transportation Management Systems and holds the #1 position in Route Planning in the G2 2026 Best Software Awards. In October 2025, Ingka Investments, the investment arm of Ingka Group, the world’s largest IKEA retailer, acquired Locus. Ingka’s [press release](https://locus.sh/press-releases/ingka-group-acquires-locus/)
 says that after an extensive evaluation of logistics software worldwide, it selected Locus for its strength in combining advanced route optimization, real-time visibility and smart use of vehicles and resources. Locus continues to operate independently.

**A route simulation on your own data.** Locus offers a complimentary tailored route simulation, and for multi-depot networks it invites buyers to bring a depot list, a month of orders and a constraint list so it can plan them as one network. Its [Route Planning System](https://locus.sh/route-planning-system/)
 plans against 250+ real-world constraints.

**Controls for a safe live test.** Locus’s governance controls include an execution sandbox and autonomy levels, so a pilot can set how much the system decides without a dispatcher releasing it. It has optimized 1.5B+ deliveries for 360+ enterprise customers across 30+ countries.

Locus’s public pages do not state formal pilot or proof-of-value terms, a replay feature for historical days or a standard pilot readout, so ask for the pilot design, the metrics and the stop rules in writing before you start.

| Also Read: Top 5 KPIs Your Route Optimization Engine Must Track |
| --- |

## How to Run a Fair Route Optimization Pilot in Six Steps

The six steps below keep the comparison honest.

### 1. Write the question and the primary metric

State what you want to learn, such as whether the engine lowers cost per stop at equal service. Choose one primary metric, fix its formula and add guardrails for on-time rate, failed first attempts and driver overtime. Fixing these first prevents choosing the favorable metric afterward.

### 2. Freeze the inputs and constraints

Give both arms the same orders, vehicles, drivers, service times, time windows and travel-time model. If the engine is allowed constraints the incumbent process cannot use, or the reverse, the test measures the constraints and not the engine.

### 3. Choose the design

Use two phases. First replay a set of matched past days offline and compare planned results. Then run live, with territories or days assigned to each arm in a balanced way and a crossover in the second week so every territory runs both arms. Do not give the new engine your easiest territories.

### 4. Size the test and set stop rules

Compute the number of days needed to detect the smallest difference that would change your decision, using the variability of daily differences. Set stop rules in advance, such as ending the live phase if a guardrail breaches a limit, and do not stop early because the first days look good.

### 5. Run with guardrails and log overrides

Record every dispatcher override and its reason, because overrides show where the engine’s plan did not fit reality. Keep the measurement process the same in both arms.

### 6. Read out against the decision rule

Report the effect with an interval, the guardrails and the overrides, then apply the rule you wrote at the start. If the interval is wide, say what extra days would narrow it.

| Also Read: AI vs. Rule-Based Route Optimization: 2026 Benchmark |
| --- |

## Demo, Offline Replay, Live Split Test and Full Cutover: Key Differences

| Approach | What it proves | What it misses | Cost and risk |
| --- | --- | --- | --- |
| Vendor demo on sample data | The engine can produce plans | Fit to your constraints and data | Low cost, low evidence |
| Offline replay of matched days | Plan quality on your orders and constraints | Execution behavior, driver response | Low risk, moderate evidence |
| Live split or crossover test | Results on real execution with a control | Rare events outside the test window | Moderate cost, strong evidence |
| Full cutover | Everything at once | Nothing can be compared | High risk, no control |

## What to Look for in a Route Optimization Vendor’s Pilot Support

### Your constraints in the pilot

Check that the engine can load your actual constraints, such as windows, vehicle limits and shift rules, rather than a simplified version. A pilot on simplified constraints overstates the result.

### Plan export for comparison

Look for the ability to export planned routes and metrics in a form you can compare with the incumbent’s plan, stop by stop.

### Override tracking

Ask whether dispatcher overrides and reasons are captured, since they are the most useful data a pilot produces.

### A written pilot design

The vendor should state the metric, the sample, the stop rules and the readout format before work starts, and agree that you own the data.

### Support for a controlled live phase

Look for autonomy settings that let dispatchers release plans, territory-level switches and a way to revert quickly.

| Also Read: Peak Season Dispatch Cutover: A Go/No-Go Framework |
| --- |

## A Fair Pilot in Action: Real-World Results

**Sizing the test on illustrative variability.** The numbers are assumptions, so replace them with yours. The pilot compares the mean of daily differences in cost per stop between arms, and the standard deviation of those daily differences is $0.60. The NIST formula for a mean, with a 5% significance level and 80% power, is N equals 7.84 times sigma squared divided by delta squared, so the days needed rise sharply as the target difference shrinks.

| Smallest difference to detect | Paired days needed |
| --- | --- |
| $0.75 per stop | 6 |
| $0.53 per stop | 10 to 11 |
| $0.40 per stop | 18 |
| $0.30 per stop | 32 |

A two-week pilot with 10 paired days can reliably detect a difference of about $0.53 per stop, and it needs about 18 days to detect $0.40. The normal approximation is slightly optimistic for small samples, so add a day or two.

**Reading out a result on illustrative data.** Over 10 paired days the incumbent averages $7.40 per stop and the optimized plan $6.95, a mean difference of $0.45 with a standard deviation of $0.60 across the days. The standard error is $0.60 divided by the square root of 10, or $0.19, and the 95% interval using the t distribution is $0.45 plus or minus $0.43, from $0.02 to $0.88.

| Measure | Incumbent | Optimized | Difference |
| --- | --- | --- | --- |
| Cost per stop | $7.40 | $6.95 | $0.45 saved |
| 95% interval for the saving |  |  | $0.02 to $0.88 |
| On-time rate | 96.1% | 96.4% | +0.3 points |
| Failed first attempts | 4.0% | 3.9% | -0.1 points |

At 1,200 stops a day over 260 days, a $0.45 saving is about $140,000 a year, but the interval runs from roughly $6,000 to $274,000. The result is promising and not conclusive. The honest readout is to extend the live phase to 18 days or more, not to sign on the point estimate.

**A Vietnamese beverage distributor.** The distributor, running a single depot out to thousands of small retail points a day on a mixed fleet of vans, trucks and motorbikes, cut fuel consumption by 37%, raised orders per delivery trip by 22%, cut route distance by 12% and cut route planning time by 35%, where manual planning had taken an hour to an hour and a half before any vehicle moved. [Read the beverage distributor case study](https://locus.sh/case-studies/beverage-distributor-route-planning-dispatch/)
.

**A Fortune 50 enterprise.** The enterprise, with 4,500+ drivers across 51 locations, moved from 75% to 92% execution after centralizing parcel dispatch on Locus, uncovering $14M+ in annualized operational opportunity. [Read the Fortune 50 case study](https://locus.sh/case-studies/fortune-50-parcel-centralized-dispatch/)
.

## Common Pilot Design Mistakes to Avoid

**Comparing different weeks.** Weather, volume and mix change the baseline, so compare matched days or run a crossover.

**Giving the new engine the easy territories.** Selection bias flatters the result, so assign arms in a balanced way, and avoid choosing the worst days for the treatment because they tend to improve on their own.

**Changing constraints between arms.** Different constraints measure the constraints, so freeze them.

**Stopping when the numbers look good.** Early gains often shrink, so fix the days and the rule in advance.

## How Locus Approaches Route Optimization Pilots

Locus, the world’s first Decision-Intelligent, Agentic TMS, offers a tailored route simulation on a buyer’s depots, orders and constraints, plans against 250+ real-world constraints through its [Route Planning System](https://locus.sh/route-planning-system/)
 and lets teams set autonomy levels, widening them as evidence builds. Its execution sandbox, explainability and traceability keep each pilot decision reviewable, and its [dispatch management software](https://locus.sh/dispatch-management-software/)
 covers the live dispatch phase. Locus’s public pages do not state formal pilot terms, historical-day replay or a standard readout, so ask for the pilot design and stop rules in writing. To design your own pilot, [schedule a demo](https://locus.sh/schedule-demo/)
.

For operations leaders asking how to run a fair route optimization pilot, Locus offers a simulation on your own data and governed controls for a live test, so the decision rests on a result you measured.

## Frequently Asked Questions

**How long should a route optimization pilot run?** Long enough to detect the smallest difference that would change your decision. With $0.60 daily variability, 18 paired days detect $0.40 per stop and 10 days detect about $0.53.

**What should be the primary metric?** Choose one, such as cost per stop with fixed definitions, and add guardrails for on-time rate, failed first attempts and driver overtime. Fix both before the pilot starts.

**How do you make a pilot fair?** Give both arms the same orders and constraints, assign days or territories in a balanced way, use a crossover so each territory runs both arms and state the decision rule in advance.

**How many guardrail-clean stops are enough?** By the rule of three, 300 consecutive clean stops support 95% confidence that the failure rate is below 1%, and 3,000 support below 0.1%.

**Why shouldn’t you pick the worst days for the new engine?** Extreme days tend to be followed by more typical ones, so improvement may be regression to the mean and not the engine. Assign arms in a balanced way.

**Does Locus offer a pilot?** Locus offers a complimentary tailored route simulation. Its public pages do not state formal pilot terms, so agree the design, metrics and stop rules in writing.

MEET THE AUTHOR

Aseem Sinha

Vice President - Marketing

Aseem, leads Marketing at Locus. He has more than two decades of experience in executing global brand, product, and growth marketing strategies across the US, Europe, SEA, MEA, and India.

### Related Tags:

[https://locus.sh/blogs/drivers-per-dispatcher-span-of-control-exception-load/](https://locus.sh/blogs/drivers-per-dispatcher-span-of-control-exception-load/)
#### [General](https://locus.sh/blogs/category/general/)

## [How Many Drivers per Dispatcher? Span of Control and Exception Load in Last-Mile Operations](https://locus.sh/blogs/drivers-per-dispatcher-span-of-control-exception-load/)

[Ishan Bhattacharya](https://locus.sh/blogs/author/ishan_locus/)

Oct 22, 2026

No benchmark exists for drivers per dispatcher. Use this model: exception rate, handling time and target utilization set the span of control. Includes worked scenarios and queueing limits.

[Read more](https://locus.sh/blogs/drivers-per-dispatcher-span-of-control-exception-load/)

[https://locus.sh/blogs/order-cut-off-time-design-route-cost/](https://locus.sh/blogs/order-cut-off-time-design-route-cost/)
#### [General](https://locus.sh/blogs/category/general/)

## [Order Cut-Off Time Design: How the Cut-Off Hour Changes Route Quality and Cost](https://locus.sh/blogs/order-cut-off-time-design-route-cost/)

[Anas T](https://locus.sh/blogs/author/anas_locus/)

Oct 22, 2026

How to choose an order cut-off time: match it to pick capacity, price the marginal order, and compare daily cut-offs with dispatch waves. Includes a worked capacity model.

[Read more](https://locus.sh/blogs/order-cut-off-time-design-route-cost/)

## How to Run a Fair Route Optimization Pilot: A Two-Week Test Design Against Your Current Plan

- Share
- [Print](javascript:window.print())
- [Download](#)
- [Schedule a Demo](https://locus.sh/schedule-demo/)

### Is your team spending more time on fixing logistics plan than running the operation?

- Agentic transportation management from order intake to freight settlement
- Route optimization built on 250+ real-world constraints
- AI-driven dispatch with automatic execution handling

20%Cost Reduction

66%Faster Planning Cycles

[Schedule a demo](/schedule-demo/)

Insights Worth Your Time

#### [General](https://locus.sh/blogs/category/general/)

## [Locus 2026 UK Consumer Survey: Why Returns Visibility is Now the Conversion Engine for AI-Driven Shopping in UK Retail](https://locus.sh/blogs/returns-visibility-conversion-engine-ai-shopping-uk-retail-locus-q2-2026-consumer-survey/)

[Aseem Sinha](https://locus.sh/blogs/author/aseem_locus/)

May 29, 2026

#### [General](https://locus.sh/blogs/category/general/)

## [Locus 2026 US Consumer Survey: Generative AI isn’t Just Changing How Consumers Shop, it’s Breaking the Demand Patterns US Retail Was Built On](https://locus.sh/blogs/generative-ai-shopping-effect-retail-fulfillment-operations-locus-q2-2026-consumer-survey/)

[Ishan Bhattacharya](https://locus.sh/blogs/author/ishan_locus/)

May 29, 2026

#### [General](https://locus.sh/blogs/category/general/)

## [Embedded vs Bolted-On AI: The Architecture Question European Logistics Buyers Are Asking](https://locus.sh/blogs/embedded-vs-bolted-on-ai-european-logistics-platform-architecture-business-benefits/)

[Aseem Sinha](https://locus.sh/blogs/author/aseem_locus/)

May 21, 2026

#### [General](https://locus.sh/blogs/category/general/)

## [Hybrid Fleet Management: How Owned, 3PL, Gig, ICE, and EV Capacity Actually Operate at Most Enterprises](https://locus.sh/blogs/three-workforce-fleet-reality-owned-3pl-gig-drivers/)

[Aseem Sinha](https://locus.sh/blogs/author/aseem_locus/)

May 7, 2026

#### [General](https://locus.sh/blogs/category/general/)

## [US Returns Hit $850 Billion in 2025: Why US Retailers Are Restructuring Reverse Logistics in 2026](https://locus.sh/blogs/850-billion-us-returns-ai-routing-reverse-logistics-2026/)

[Ishan Bhattacharya](https://locus.sh/blogs/author/ishan_locus/)

May 7, 2026