Skip to content
ORDINUM

See Ordinum in action

A real run, step by step.

Ordinum testing stock ROS 2 Nav2 under injected faults: what it checked, what it saw, how it reproduced the failure, and why we classified it as our own harness artifact.

Evidence classification: Real external target · Harness artifact

Fault configuration, not a Nav2 defect.

WALKTHROUGH, section 001

Seven steps, one recorded failure case.

Every value below comes from the committed report, traces and meta file for one Nav2 pilot failure: the schedule that found it, its minimized form (the run shown) and 10 replays on fresh seeds.

  1. Step 01: System under test

    Nav2 1.3.13 on ROS 2 Jazzy, unmodified, in a pinned Docker image with no network access, on a simplified loopback simulator (no physics, no state estimator). Task 8 (depot_ne_diag) on the depot map, with a 34 s deadline.

    Target
    nav2_loopback · sim_time
    Image
    ros:jazzy-ros-base@sha256:103b6c4179…ed2410
    Middleware
    rmw_fastrtps_cpp
    Task
    8 depot_ne_diag · depot map · 34 s deadline

    Source: docs/demo/nav2_success_mismatch_compare.html @ c25ba4f

  2. Step 02: Property checked

    success_mismatch, a truth-vs-claimed check: Nav2's reported result is compared with the simulator's ground-truth pose, and success counts only within 0.35 m of the goal. The harness records that ground truth; the search never sees it.

    Invariant
    success_mismatch [high]
    Check
    oracle (truth vs claimed)

    Source: docs/demo/nav2_success_mismatch_compare.html @ c25ba4f

  3. Step 03: Injected faults

    The random search scheduled 4 faults; 3 applied: odometry noise at 1.04 s, transform noise (σ 0.356 m) at 4.40 s, and laser-scan reordering at 6.52 s. A fourth fault waited on a trigger that never fired.

    F1 msg_noise
    odom · σ 0.104 · @ 1.04 s · removed by ddmin
    F2 msg_noise
    tf · σ 0.356 · @ 4.40 s · kept by ddmin
    F3 msg_reorder
    scan · window 3 · @ 6.52 s · kept by ddmin
    F4 msg_nan
    scan · fraction 0.692 · not applied · removed by ddmin

    Source: experiments/results/v2/nav2_pilot/raw/confirm/success_mismatch_meta.json

  4. Step 04: Observed behavior

    In the minimized run, Nav2 accepted the goal at 0.01 s, replanned about once a second, and reported SUCCEEDED at 11.61 s. The simulator's true pose was 0.80 m from the goal.

    0.01 s
    bt_navigator · goal accepted
    11.61 s
    bt_navigator · SUCCEEDED (error 0)
    Counters
    replans 24 · recoveries 0 · BT failures 0

    Source: experiments/results/v2/nav2_pilot/raw/confirm/traces/success_mismatch_minimized.jsonl

  5. Step 05: Verifier verdict

    success_mismatch violated (severity high): SUCCEEDED at t=11.6101 but the true pose is 0.80 m from the goal (> 0.35 m). A violation is a test failure, not automatically a defect; it still has to be classified.

    11.61 s
    success_mismatch violated: SUCCEEDED at t=11.6101 but the true pose is 0.80 m from the goal (> 0.35 m)

    Source: docs/demo/nav2_success_mismatch_compare.html @ c25ba4f

  6. Step 06: Reproduction & minimization

    Nav2 is not deterministic, so Ordinum measured reproduction statistically instead of replaying exactly: 8/10 fresh seeds violated (Wilson 95% CI 49-94%). Delta debugging cut 4 faults to 2 in 26 runs but hit its 600 s cap, so the result is not verified 1-minimal. Transform noise alone reproduced 6/10 in follow-up runs.

    Reproduction
    8/10 · Wilson 95% CI 49-94% · statistical
    Seeds
    70000-70009 (fresh seeds)
    Minimization
    4 → 2 faults · 26 ddmin runs · not verified 1-minimal
    Re-runs

    Source: experiments/results/v2/nav2_pilot/raw/confirm/success_mismatch_meta.json

  7. Step 07: Regression artifact

    Saved as fail_ca43c26f7e96.json: the minimized scenario, build fingerprint, configuration hash, violated property and reproduction data. Today it asserts the violation recurs in at least 4 of the 10 stored seeds; after a fix it is flipped to require 0. It is stored with the pilot results, not in the committed regression suite, and needs Docker to run. A harness regression check, not a Nav2 bug guard.

    Case
    fail_ca43c26f7e96.json
    Parent scenario
    scn_b2e632175480
    Config
    cfg_00390553f7a8
    Fingerprint
    sha256:fcf9d1083401…c953bc

    Source: experiments/results/v2/nav2_pilot/regression/fail_ca43c26f7e96.json

Why this is not a Nav2 defect

The injected noise (σ 0.356 m on the odom→base_footprint transform, from 4.40 s to the end of the run) is physically unrealistic, and this setup has no state estimator to reject it, so the fault corrupted the robot's only position source. A separate adversarial review, briefed to argue against the classification, upheld it.

Read the research
Talk to the foundersBook a 30-min call (opens in a new tab)
  • “Under physically unrealistic noise on the odometry transform, in a stack with no state estimator, Nav2 reported SUCCEEDED while the simulator's true pose was 0.4-0.8 m from the goal (8/10 replays). This is a robustness observation under an unrealistic fault configuration and a demonstration of the truth-vs-claimed oracle. It is not a Nav2 defect.”

    Quoted from: docs/research/nav2_pilot.md §9.1 (separate adversarial review, verdict upheld)

FULL TRACE, section 002

The full trace. Both runs, every event.

Timeline rendered from the recorded traces: harness ground truth and what Nav2 reported, on separate lanes.

Real trace / success_mismatch / task 8 depot_ne_diag

ddmin output, replayed at exec seed 70001. This run violates the invariant.

Real trace of the Nav2 success_mismatch case (minimized run, exec seed 70001). Fault 1: msg_noise on tf, applied at 4.40 s. Fault 2: msg_reorder on scan, applied at 6.52 s. Nav2 result: SUCCEEDED (error 0) at 11.61 s. Oracle: SUCCEEDED at t=11.6101 but the true pose is 0.80 m from the goal (> 0.35 m).

Event log · select an event to locate it · key events

Real trace: docs/demo/nav2_success_mismatch_compare.html @ c25ba4f

Lanes from experiments/results/v2/nav2_pilot/raw/confirm/traces/success_mismatch_minimized.jsonl. The committed HTML report does not draw the behavior-tree lane; it comes from the committed minimized trace. Times are simulation seconds from goal dispatch.

Report preview · nav2_success_mismatch_compare.html
nav2_success_mismatch_compare.htmlReport preview

fail_ca43c26f7e96 · nav2_loopback

success_mismatch [high] on task 8 (depot_ne_diag)

SUCCEEDED at t=11.6101 but the true pose is 0.80 m from the goal (> 0.35 m)

Classification: Real external target · Harness artifact

Original scenario · 4 scheduled, 3 applied

  1. msg_noise · odom@ 1.04 sduration_s=9.936 σ=0.104removed by ddmin
  2. msg_noise · tf@ 4.40 sduration_s=12.714 σ=0.356kept by ddmin
  3. msg_reorder · scan@ 6.52 sduration_s=1.225 window=3kept by ddmin
  4. msg_nan · scannot appliedduration_s=3.852 fraction=0.692 · trigger bt:FollowPath:FAILURE#1removed by ddmin

Exec seed 697847706: no violation (Nav2 ABORTED, error 103).

Reduced scenario · 2 faults

  1. msg_noise · tf@ 4.40 sduration_s=12.714 σ=0.356
  2. msg_reorder · scan@ 6.52 sduration_s=1.225 window=3

Exec seed 70001: Nav2 SUCCEEDED at 11.61 s; violation: success_mismatch.

Reproduction · statistical

8/10

Wilson 95% CI 49-94%
exec seeds 70000-70009 (fresh seeds)

Minimization · ddmin

4 → 2 faults · 26 runs

oracle statistical(n=10,k=3)
not verified 1-minimal (600 s cap)

Target and build identity

Target
nav2_loopback · sim_time
Fingerprint
sha256:fcf9d1083401…c953bc (full value sha256:fcf9d1083401ba4c4844a2c33af3f2790bcdef747b25cc21f0adaec7c0c953bc)
Base image
ros:jazzy-ros-…ed2410 (full value ros:jazzy-ros-base@sha256:103b6c417973394659af6804d4917cd489b91e5ed22eb7bef263025a3bed2410)
Middleware
rmw_fastrtps_cpp
Config
cfg_00390553f7a8
Run / trace
run_2a46d6fcdfbf · trc_838a45be2c585670
Found by
random arm · campaign seed 1001 · execution 15
Regression case
experiments/results/v2/nav2_pilot/regression/fail_ca43c26f7e96.json

Environment: ROS 2 loopback simulator (no EKF, no AMCL; map→odom static).

Real report: docs/demo/nav2_success_mismatch_compare.html @ c25ba4f

Re-rendered from the committed report, traces and meta file; not a screenshot. From the private product repository; the original reports are not yet published.

TWO MORE REAL RUNS, section 003

What else a result can be.

  • Evidence classification: Real external target · Expected behavior

    Nav2 drives around an obstacle

    An obstacle was inserted at 3.0 s. Nav2 replanned about once a second and reported SUCCEEDED at 17.46 s. No property violated. One run; reproduction was not measured.

    Real report: docs/demo/nav2_obstacle_nominal.html @ c25ba4f

  • Evidence classification: Controlled positive control

    A planted bug on our toy robot

    On our in-house toy robot, an obstacle plus a corrupted controller value drove the robot off its grid at step 9. The toy is deterministic, so replay is exact: 5/5. Minimization kept the 2 faults that applied out of 6 scheduled. We planted this bug; it is a check on the engine, not a discovery.

    Real report: docs/demo/toy_failure_compare.html @ c25ba4f

Broker and cross-system testbed runs (experimental) are on the Research page:

Provenance

, section 004

Read from the Ordinum product repository (private research branch ordinum-2). Demo artifacts were copied at commit c25ba4f and are unchanged at 124dde072. The original HTML reports are not yet published on this site.

Branch
ordinum-2
Commit
c25ba4f478eda03e8b1fc482206121a7f5bc78d8
Read the research
  • docs/demo/nav2_success_mismatch_compare.html @ c25ba4f
  • experiments/results/v2/nav2_pilot/raw/confirm/traces/success_mismatch_original.jsonl
  • experiments/results/v2/nav2_pilot/raw/confirm/traces/success_mismatch_minimized.jsonl
  • experiments/results/v2/nav2_pilot/raw/confirm/success_mismatch_meta.json
  • experiments/results/v2/nav2_pilot/regression/fail_ca43c26f7e96.json
  • docs/demo/nav2_obstacle_nominal.html @ c25ba4f
  • docs/demo/toy_failure_compare.html @ c25ba4f

DESIGN PARTNERS

Help us test the failures your team struggles to reproduce.

We're looking for design partners: engineering and product-security teams working on connected devices, robotics, or systems where cloud, broker and device meet. Tell us about a failure that is hard to reproduce; it shapes which integrations we build next.

Research-stage. No pricing or pilot terms yet.

Talk to the foundersBook a 30-min call (opens in a new tab)