LIRRA  1.0.0
Loading...
Searching...
No Matches
Case Studies

All case studies use the patrol task of the NAHS interval-STL runtime-assurance study (Baird, Morales-Cuadrado and Coogan, 2026) on three platforms of increasing fidelity: a double integrator, the MuJoCo Menagerie Skydio X2 quadrotor, and a MuJoCo model of Georgia Tech's Miniature Autonomous Blimp (GT-MAB). We evaluate eight specifications: the two NAHS formulas, on the original planar square and on a three-dimensional cube, each with exact and with uncertain boundaries. Videos of every vehicle and specification, and a 6-minute showreel of the 3-D experiments, are on the Videos page.

The Patrol Task

A nominal PD controller patrols between waypoints just outside a restricted region \(S\), which the vehicle may enter only briefly. Its straight legs cut through \(S\) on every lap, so the runtime-assurance layer has to bend the path around \(S\) while staying close to the patrol schedule.

Two temporal rules (formulas (24) and (25) of the NAHS study, bounds in seconds):

\[ \vartheta = (x \in S) \Rightarrow \mathbf{F}_{[0,4]}(x \notin S), \qquad \varphi = (x \in S) \Rightarrow \mathbf{F}_{[0,3]}\mathbf{G}_{[0,1]}(x \notin S). \]

After entering \(S\) the vehicle must leave within 4 s ( \(\vartheta\)), or leave within 3 s and stay out for 1 s ( \(\varphi\)). Both are enforced on every window.

Two shapes of the region:

  • Square (the NAHS task): \(S = \{|p_x|, |p_y| < 1.41\}\) m, four faces, waypoints \((0, \pm 1.52)\) and \((\pm 1.52, 0)\) m.
  • Cube (our 3-D extension): \(S = \{|p_x|, |p_y|, |p_z| < 1.41\}\) m, six faces, centered 2.75 m above the floor, with waypoints at \(\pm 1.52\) m along each axis visited in the order \(+y, +x, +z, -y, -x, -z\). Consecutive faces are adjacent, so every leg cuts an edge region.

Two kinds of boundary:

  • Exact: every face is a known plane, e.g. \(p_x - 1.41 \ge 0\).
  • Uncertain: the true region has a half-width in \(1.41 \pm 0.05\) m and is rotated about its center by at most 0.05 rad (about any axis in 3-D). Each face becomes an interval predicate \([a]^\top y - [b] \ge 0\) that covers every possible true face, and I-STL checks its worst case. Each episode is scored against a true region drawn from this set.
Specification Region Boundaries Paper symbol
θ, square square exact – (NAHS (24))
φ, square square exact – (NAHS (25))
θ, cube cube exact \(\vartheta\)
φ, cube cube exact \(\varphi\)
θ, square, uncertain square uncertain –
φ, square, uncertain square uncertain –
θ, cube, uncertain cube uncertain \(\vartheta^+\)
φ, cube, uncertain cube uncertain \(\varphi^+\)

The paper focuses on the four cube specifications, the harder setting: on the blimp, even full synthesis has no solution from some cube initial states near \(S\), whereas every planar initial state admits one.

Benchmarks

System Simulator Planning model \(\Delta t\) [s] \(h\) (θ / φ) \(N\), \(d\) Uncertainty levels
Double integrator \(f_{\mathrm{syn}} + Bw\) 2-D double integrator 0.25 15 / 15 24, 2 \(\bar w \in \{0, 0.02, 0.04\}\)
Skydio X2 MuJoCo, body-rate tracker 3-D double integrator 0.2 19 / 19 30, 1 \(\varepsilon \in \{0.05, 0.10, 0.15\}\) m
Blimp MuJoCo, tracker linearization, 8 states 0.5 7 / 7 12, 1 \(\varepsilon \in \{0.05, 0.10, 0.15\}\) m

\(h\): formula horizon in steps; \(N\): plan length; \(d\): solve budget in planning steps.

  • Double integrator. The disturbance enters the true dynamics inside \(\mathcal{W}\) and the tube comes from the feedback gain \(K\), so every assumption of the theorem holds.
  • Skydio X2. The Menagerie MJCF flown through a collective-thrust and body-rate interface: an outer position loop tracks the zero-order-hold interpolation of the plan, and an inner rate loop with motor allocation flies the vehicle. The planning model is a 3-D double integrator with a constant planar tube \(\varepsilon\).
  • Blimp. A MuJoCo model built from the parameters of the NAHS blimp simulator (Fossen's model about the center of buoyancy, rigid-body mass 0.1249 kg, added mass as armature, center of gravity 5 cm and thrust 0.295 m below the center of buoyancy, six gondola thrusters). The planning model is the NAHS linearization about hover restricted to its four decoupled blocks; MuJoCo keeps the swing coupling the linearization drops. Because the blimp's PD patrol saturates its thrust most of the time, its reference is flown with inputs capped at 80% of the bounds.

Loops compared. All loops share the program, the solver (Gurobi, one thread, time limit \(0.8\,d\,\Delta t\)), the initial solve at \(t = 0^-\), the initial states, the uncertainty levels and the disturbance sequences.

  • STL solver only (always-on): solve whenever no solve is pending. This is the NAHS filter with delay compensation and verified terminal sets.
  • First stage, checked: LIRRA with a proposer from behavior cloning and PPO on the original expert data.
  • Full pipeline, checked: LIRRA with the proposer from behavior cloning, DAgger, margin-weighted cloning and PPO.
  • RL only, no check: the full-pipeline proposer flown unchecked.

Results

All eight specifications

Always-on I-STL (synthesis at every planning step) against LIRRA (full-pipeline PPO proposer, checked), 30 episodes per row: 10 start states × 3 uncertainty levels \(\varepsilon \in \{0.05, 0.10, 0.15\}\) m, with a solve budget of one planning step. Fallbacks are synthesis requests per planning step (always-on I-STL: 1 by definition). Time is solve + inference + check per planning step. Waypoints are counted in patrol order; for the cube, the order-free count of visits is in parentheses. Violations are LIRRA episodes violating the specification (for uncertain boundaries, the true region).

System Specification LIRRA fallbacks / step Time I-STL [ms] Time LIRRA [ms] Speed-up Waypoints I-STL Waypoints LIRRA Violations
Skydio X2 θ, square .003 133.0 1.4 95× 6.0 6.0 0/30
φ, square .012 143.0 2.7 53× 6.0 6.0 0/30
θ, cube .006 140.9 1.7 83× 6.0 (6.1) 6.0 (6.0) 0/30
φ, cube .007 151.5 2.0 76× 5.6 (6.0) 6.0 (6.1) 0/30
θ, square, uncertain .003 144.1 1.6 90× 6.0 6.0 0/30
φ, square, uncertain .019 145.6 2.9 50× 6.0 6.0 0/30
θ, cube, uncertain .013 158.8 3.1 51× 6.0 (6.1) 6.0 (6.0) 0/30
φ, cube, uncertain .011 151.2 3.0 50× 6.0 (6.1) 5.8 (6.0) 0/30
Blimp θ, square .074 12.7 2.0 6.4× 9.5 6.3 0/30
φ, square .043 17.9 1.9 9.4× 10.2 4.1 0/30
θ, cube .062 10.9 1.7 6.4× 3.0 (6.9) 3.4 (7.0) 0/30
φ, cube .045 12.1 1.8 6.7× 3.2 (6.8) 2.6 (6.6) 0/29
θ, square, uncertain .046 12.1 1.9 6.4× 6.0 2.5 0/30
φ, square, uncertain .045 15.6 2.0 7.8× 6.5 4.1 0/30
θ, cube, uncertain .077 11.7 2.2 5.3× 2.0 (5.7) 3.2 (6.8) 0/30
φ, cube, uncertain .041 14.0 1.9 7.4× 2.0 (5.8) 1.4 (4.5) 0/23

Blimp φ (cube) and φ (cube, uncertain) have 29 and 23 episodes: from the remaining starts, no method can begin, because the initial synthesis problem itself has no solution.

What the table shows:

  • No violations anywhere. All 472 LIRRA episodes satisfied their specification, including every uncertain-boundary episode against its true region.
  • Synthesis becomes rare. The X2 needs fallback synthesis on 0.3–1.9% of planning steps and the blimp on 4–8%, against 100% for always-on I-STL. Mean computation per step falls 50–95× on the X2 and 5–9× on the blimp.
  • The harder specifications cost little. Fallback rates on the cube and with uncertain boundaries are close to those on the original square.
  • Mission progress is mixed on the blimp. On the X2, LIRRA keeps the six-waypoint pace (and beats I-STL on φ, cube, where I-STL's solver sometimes times out without a plan). On the blimp LIRRA reaches fewer waypoints on most specifications but more on θ (cube) and θ (cube, uncertain).

CPU energy

CPU energy per planning step on the four cube specifications, measured with Intel RAPL on a laptop (Intel Core Ultra 7 265U), one evaluation at a time on an otherwise idle machine. Energy is what the processor package consumed while computing (solves, proposer inference and checks), divided by the number of planning steps, the same definition as the computation time. Inference and checks are shorter than RAPL's update interval, so their energy per call is measured over many repetitions. Times in this table are from the same runs, on the laptop.

System Spec. Energy I-STL [mJ/step] Energy LIRRA [mJ/step] Energy reduction Time I-STL [ms/step] Time LIRRA [ms/step]
Skydio X2 θ, cube 1848 13.7 135× 119.5 1.3
φ, cube 2269 18.2 124× 136.6 1.8
θ, cube, uncertain 2038 51.7 39× 135.7 2.7
φ, cube, uncertain 2063 38.5 54× 137.3 2.1
Blimp θ, cube 148 12.0 12× 7.0 0.9
φ, cube 166 11.9 14× 9.6 0.9
θ, cube, uncertain 146 18.2 8.0× 9.2 1.2
φ, cube, uncertain 175 15.8 11× 12.0 1.1

The processor draws 14–24 W while solving in every run, so energy follows computation time: LIRRA reduces energy per step 39–135× on the X2 and 8–14× on the blimp, against 51–92× and 8–11× for computation time on this machine.

Why uncertain boundaries need interval predicates

Each uncertain-boundary episode draws a true region (half-width within ±0.05 m, rotation within 0.05 rad). The ablation plans and checks with the nominal faces instead, and is scored against the true region. Episodes that entered the true region:

System Specification I-STL, interval LIRRA, interval I-STL, nominal faces
Skydio X2 θ, square, uncertain 0/30 0/30 4/30
φ, square, uncertain 0/30 0/30 12/30
θ, cube, uncertain 0/30 0/30 7/30
φ, cube, uncertain 0/30 0/30 12/30
Blimp θ, square, uncertain 0/30 0/30 15/30
φ, square, uncertain 0/30 0/30 16/30
θ, cube, uncertain 0/30 0/30 17/30
φ, cube, uncertain 0/23 0/23 19/29

Planning against the nominal faces looks safe to the planner and still enters the real region in up to two thirds of the episodes.

What the check and the training pipeline each contribute

θ and φ on the square, pooled (60 episodes per row: 10 start states × 3 uncertainty levels × 2 formulas), comparing four loops: always-on synthesis; LIRRA with a first-stage proposer (behavior cloning and PPO); LIRRA with the full-pipeline proposer; and the full-pipeline proposer flown without the check.

Platform Loop Solves / episode Per solve [ms] RL + check [ms] Compute / step [ms] Waypoints Violated episodes Min ρ
Double integrator STL solver only 120.0 121.9 – 60.94 6.0 0 0.000
RL only, no check 0.0 – 0.73 0.73 6.0 0 0.008
First stage, checked 2.8 91.4 1.34 2.39 6.0 0 0.021
Full pipeline, checked 0.2 71.9 1.12 1.19 6.0 0 0.035
Skydio X2 STL solver only 300.0 138.0 – 137.97 6.0 0 0.003
RL only, no check 0.0 – 0.84 0.84 6.0 0 0.011
First stage, checked 8.4 150.3 1.11 5.29 6.0 0 0.003
Full pipeline, checked 2.2 135.3 1.10 2.08 6.0 0 0.020
Blimp STL solver only 240.0 15.3 – 15.29 9.8 0 0.001
RL only, no check 0.0 – 1.04 1.04 5.8 16 −0.054
First stage, checked 59.4 18.4 2.00 6.05 6.0 0 0.000
Full pipeline, checked 19.9 16.9 2.33 3.54 5.5 0 0.001

The check is what keeps the blimp safe: unchecked, the same policy violated the specification in 16 of 60 episodes; checked, in none. On the cube, the unchecked final cloned and PPO proposers violated θ or φ in 57 of 238 episodes, including 48 of 60 blimp θ episodes.

Ablations on the cube

Each row removes or weakens one part of LIRRA on the four cube specifications (θ and φ, with exact and uncertain boundaries), with the same full-pipeline proposers and start states as the results table. Episodes are pooled over the four specifications; the blimp has 112 because 8 of its starts have no initial solution for any method.

System Loop Episodes Violated episodes Fallback solves Solves with no solution Missed deadlines
Skydio X2 LIRRA (full check) 120 0 332 0 0
Nominal robustness trigger 120 0 44 0 0
No delay compensation 120 2 427 134 0
LIRRA, measured solver latency 120 0 328 0 0
Always-on I-STL, measured solver latency 120 0 35,992 231 7
Blimp LIRRA (full check) 112 0 1,541 3 0
Nominal robustness trigger 112 0 460 90 0
No delay compensation 112 2 1,671 437 0
LIRRA, measured solver latency 112 0 1,541 3 0
Always-on I-STL, measured solver latency 112 0 26,880 44 0

What each row shows:

  • Nominal robustness trigger. Proposals are accepted when their nominal robustness exceeds δ, without the tracking-error margin, the input bounds or the terminal set. This weaker test accepts more proposals, so fewer solves are needed, and there were still no violations. But on the blimp, 90 of 460 fallback solves found no solution (3 of 1,541 under the full check). The accepted proposals had led the vehicle to states from which no verified plan exists, and it was left flying an unverified cached plan. Robustness alone shows a proposal is satisfying now; the full check also keeps a verified way forward.
  • No delay compensation. The solver starts from the state measured when it is called, not the state the vehicle will have reached when the solution arrives. By then the plan no longer starts where the vehicle is: about a third of the X2's solves and a quarter of the blimp's found no solution, and two episodes on each vehicle violated the specification (X2: θ and θ with uncertain boundaries, minimum robustness −0.039; blimp: θ and φ with uncertain boundaries, −0.033 and −0.012).
  • Measured solver latency. Instead of assuming every solve takes one planning step, the real solve time counts. Always-on I-STL on the X2 missed 7 deadlines, and late results were discarded with the cached plan kept in force. LIRRA solves so rarely that it missed none.

Latency and deadlines

With real (measured) solver times, always-on synthesis on the X2 missed 3 (θ) and 5 (φ) of about 9,000 solve deadlines, with worst solves of 0.96 s and 0.93 s against a 0.2 s planning period. A missed deadline leaves the cached plan in force, so none of these led to a violation. With the full-pipeline proposer the X2 issued 23 (θ) and 103 (φ) requests in total, with no missed deadlines.

Is the X2 tracking-error assumption realistic?

The guarantee on the MuJoCo vehicles assumes that the tracking error stays inside the assumed tube \(\varepsilon\) (assumption (A3) in Theory). Checked loops on the X2, by tube size:

Tube \(\varepsilon\) [m] Episodes Violated Warm-start failures / requests Max planar error [m]
0.05 340 7 42 / 12,984 0.112
0.10 340 0 0 / 13,068 0.118
0.15 340 0 0 / 14,161 0.119

The X2 tracker's error reaches about 0.12 m. A 0.05 m tube is therefore too small, the assumption fails, and both the warm start and the specification can fail with it. From 0.10 m the error stays (almost always) inside the tube, and both counts drop to zero. On the blimp, the largest planar deviation from the cached plan at planning instants was 4 cm, inside \(\varepsilon = 0.10\) m.

Reproduce

The full pipeline for one system and specification, here the X2 with θ on the cube (see Installation & Usage for setup, a Gurobi licence and the names of all eight specifications):

# 1. expert dataset: synthesis solutions along expert, noisy-expert and nominal-patrol trajectories
uv run python experiments/patrol/dataset.py --system x2 --spec theta_plus --episodes 600
# 2. behavior cloning
uv run python experiments/patrol/train_bc.py --system x2 --spec theta_plus \
--data runs/patrol_dataset/x2/<run>/dataset.npz
# 3. DAgger rounds, margin-weighted cloning and PPO (BC and PPO use the GPU if there is one)
cluster/dagger_ppo_pipeline.sh x2 theta_plus runs/patrol_bc/x2/<run>/policy.npz \
runs/patrol_dataset/x2/<run>/dataset.npz
# 4. evaluate standard I-STL, LIRRA, and the unchecked proposer on the same 10 starts x 3 levels
uv run python experiments/patrol/evaluate.py --system x2 --spec theta_plus --seeds 10 \
--pairs always_on shield policy_only --reference clock \
--policy runs/patrol_ppo/x2/<run>/policy.npz
# 5. render an episode (top-down, MuJoCo, four-panel composite) as a GIF
MUJOCO_GL=egl uv run --no-sync --with imageio-ffmpeg python experiments/patrol/render_episode.py \
--system x2 --spec theta_plus --policy runs/patrol_ppo/x2/<run>/policy.npz --level 2 \
--view composite --gif

For an uncertain-boundary specification, use theta_plus_u (or any _u name) throughout, and add --pairs always_on --nominal-predicates to step 4 for the nominal-boundary comparison. experiments/patrol/main_table.py then builds the results tables on this page from the evaluation runs.