Skip to content
ORDINUM

Research, section 03

Measured, with the caveats. Early results from the prototype.

Feedback-guided fuzzing against random fault injection and a scripted suite, on a controlled simulated robot. What it shows, what it does not, and what we are testing next.

01 /What we have measured so far

ENGINEERING EVIDENCE, section 001

A measurable starting point. More to prove.

We built the engine and measured it against random fault injection and a scripted test suite on a controlled simulated target. Feedback guidance reaches substantially more behavior. It does not yet find more failure classes than random. We report both.

Status

Research preview
  • BUILT

    Zero-dependency core engine with 10 fault kinds and 7 invariants

  • BUILT

    Automatic replay, ddmin minimization and regression-test emission

  • NEXT

    ROS2 and MuJoCo adapters, independent validation on systems we did not build, then design-partner pilots

  • Behavioral coverage

    vs. random

    +35%

    Feedback-guided vs. random fault injection. 139.1 vs. 103.1 coverage keys.

    CONTROLLED SIMULATED TARGET / 8 SEEDS / 6,000 EXECUTIONS PER CAMPAIGN

  • Unique trajectories

    vs. random

    +23%

    Feedback-guided vs. random fault injection. 996.6 vs. 807.8 trajectories.

    CONTROLLED SIMULATED TARGET / 8 SEEDS / 6,000 EXECUTIONS PER CAMPAIGN

  • Failure classes found

    Tie with random

    6/7

    Feedback-guided and random tie at six of seven. The scripted suite finds four.

    NO DISCOVERY LEAD OVER RANDOM YET / CONTROLLED SIMULATED TARGET

Benchmarks

8 seeds × 6,000 executions / campaign

Behavioral coverage keys

Mean ± standard deviation across 8 seeds. Higher is better.
Behavioral coverage keys
StrategyMeanStandard deviation
Feedback-guided139.12.0
Random103.14.2
Scripted suite51not reported

Unique trajectories

Mean ± standard deviation across 8 seeds. Scripted suite: n/a.
Unique trajectories
StrategyMeanStandard deviation
Feedback-guided996.669.7
Random807.814.8
Scripted suitenot applicablenot applicable

Worked example

One failure, start to finish

A blind collision found once in 4,000 executions. ddmin removed one unnecessary fault, leaving five. It replays deterministically every time and is saved as a regression test.

Violates no_collision

Minimization trace / no_collisionIllustrative / representative faults
  1. 01 / Found6 faults
  2. 02 / ddmin1 removed
  3. 03 / ReplayIdentical trace
  4. 04 / RegressionEmitted
Faults
6→5
Replay
Pending
Regression test / pending
Illustrative composition / fault kinds and timings are representative.

Illustrative injected faults by tick (kinds and timings representative). ddmin removed 1 of 6; 5 remain.

  1. t=3: env_obstacle, kept
  2. t=4: sensor_noise, kept
  3. t=6: obs_delay, kept
  4. t=8: controller_retry, removed by ddmin
  5. t=10: env_obstacle, kept
  6. t=11: actuator_fail, kept
  7. t=12: no_collision violated

Replay: 100% deterministic replay. Emitted as regression_0001.

Methodology

Controlled, deterministic simulated 2D grid robot: perception → A* planner → state machine → controller → actuator. 10 fault kinds, 7 invariants. Feedback-guided fuzzing and random fault injection were each run in two experiments across 8 seeds at 6,000 executions per campaign: 192,000 executions in total. The scripted test suite is a fixed, hand-written baseline.

Caveats

  1. Random reaches its first failure sooner than feedback-guided.
  2. This is a small, controlled simulated target we built. It is not yet an external system, ROS2 stack or real robot.

Building autonomous systems? Let's break them safely.

Get a demoBook a call (opens in a new tab)