|
LIRRA
1.0.0
|
All case studies use the patrol task of the NAHS interval-STL runtime-assurance study (Baird, Morales-Cuadrado and Coogan, 2026) on three platforms of increasing fidelity: a double integrator, the MuJoCo Menagerie Skydio X2 quadrotor, and a MuJoCo model of Georgia Tech's Miniature Autonomous Blimp (GT-MAB). We evaluate eight specifications: the two NAHS formulas, on the original planar square and on a three-dimensional cube, each with exact and with uncertain boundaries. Videos of every vehicle and specification, and a 6-minute showreel of the 3-D experiments, are on the Videos page.
A nominal PD controller patrols between waypoints just outside a restricted region \(S\), which the vehicle may enter only briefly. Its straight legs cut through \(S\) on every lap, so the runtime-assurance layer has to bend the path around \(S\) while staying close to the patrol schedule.
Two temporal rules (formulas (24) and (25) of the NAHS study, bounds in seconds):
\[ \vartheta = (x \in S) \Rightarrow \mathbf{F}_{[0,4]}(x \notin S), \qquad \varphi = (x \in S) \Rightarrow \mathbf{F}_{[0,3]}\mathbf{G}_{[0,1]}(x \notin S). \]
After entering \(S\) the vehicle must leave within 4 s ( \(\vartheta\)), or leave within 3 s and stay out for 1 s ( \(\varphi\)). Both are enforced on every window.
Two shapes of the region:
Two kinds of boundary:
| Specification | Region | Boundaries | Paper symbol |
|---|---|---|---|
| θ, square | square | exact | – (NAHS (24)) |
| φ, square | square | exact | – (NAHS (25)) |
| θ, cube | cube | exact | \(\vartheta\) |
| φ, cube | cube | exact | \(\varphi\) |
| θ, square, uncertain | square | uncertain | – |
| φ, square, uncertain | square | uncertain | – |
| θ, cube, uncertain | cube | uncertain | \(\vartheta^+\) |
| φ, cube, uncertain | cube | uncertain | \(\varphi^+\) |
The paper focuses on the four cube specifications, the harder setting: on the blimp, even full synthesis has no solution from some cube initial states near \(S\), whereas every planar initial state admits one.
| System | Simulator | Planning model | \(\Delta t\) [s] | \(h\) (θ / φ) | \(N\), \(d\) | Uncertainty levels |
|---|---|---|---|---|---|---|
| Double integrator | \(f_{\mathrm{syn}} + Bw\) | 2-D double integrator | 0.25 | 15 / 15 | 24, 2 | \(\bar w \in \{0, 0.02, 0.04\}\) |
| Skydio X2 | MuJoCo, body-rate tracker | 3-D double integrator | 0.2 | 19 / 19 | 30, 1 | \(\varepsilon \in \{0.05, 0.10, 0.15\}\) m |
| Blimp | MuJoCo, tracker | linearization, 8 states | 0.5 | 7 / 7 | 12, 1 | \(\varepsilon \in \{0.05, 0.10, 0.15\}\) m |
\(h\): formula horizon in steps; \(N\): plan length; \(d\): solve budget in planning steps.
Loops compared. All loops share the program, the solver (Gurobi, one thread, time limit \(0.8\,d\,\Delta t\)), the initial solve at \(t = 0^-\), the initial states, the uncertainty levels and the disturbance sequences.
Always-on I-STL (synthesis at every planning step) against LIRRA (full-pipeline PPO proposer, checked), 30 episodes per row: 10 start states × 3 uncertainty levels \(\varepsilon \in \{0.05, 0.10, 0.15\}\) m, with a solve budget of one planning step. Fallbacks are synthesis requests per planning step (always-on I-STL: 1 by definition). Time is solve + inference + check per planning step. Waypoints are counted in patrol order; for the cube, the order-free count of visits is in parentheses. Violations are LIRRA episodes violating the specification (for uncertain boundaries, the true region).
| System | Specification | LIRRA fallbacks / step | Time I-STL [ms] | Time LIRRA [ms] | Speed-up | Waypoints I-STL | Waypoints LIRRA | Violations |
|---|---|---|---|---|---|---|---|---|
| Skydio X2 | θ, square | .003 | 133.0 | 1.4 | 95× | 6.0 | 6.0 | 0/30 |
| φ, square | .012 | 143.0 | 2.7 | 53× | 6.0 | 6.0 | 0/30 | |
| θ, cube | .006 | 140.9 | 1.7 | 83× | 6.0 (6.1) | 6.0 (6.0) | 0/30 | |
| φ, cube | .007 | 151.5 | 2.0 | 76× | 5.6 (6.0) | 6.0 (6.1) | 0/30 | |
| θ, square, uncertain | .003 | 144.1 | 1.6 | 90× | 6.0 | 6.0 | 0/30 | |
| φ, square, uncertain | .019 | 145.6 | 2.9 | 50× | 6.0 | 6.0 | 0/30 | |
| θ, cube, uncertain | .013 | 158.8 | 3.1 | 51× | 6.0 (6.1) | 6.0 (6.0) | 0/30 | |
| φ, cube, uncertain | .011 | 151.2 | 3.0 | 50× | 6.0 (6.1) | 5.8 (6.0) | 0/30 | |
| Blimp | θ, square | .074 | 12.7 | 2.0 | 6.4× | 9.5 | 6.3 | 0/30 |
| φ, square | .043 | 17.9 | 1.9 | 9.4× | 10.2 | 4.1 | 0/30 | |
| θ, cube | .062 | 10.9 | 1.7 | 6.4× | 3.0 (6.9) | 3.4 (7.0) | 0/30 | |
| φ, cube | .045 | 12.1 | 1.8 | 6.7× | 3.2 (6.8) | 2.6 (6.6) | 0/29 | |
| θ, square, uncertain | .046 | 12.1 | 1.9 | 6.4× | 6.0 | 2.5 | 0/30 | |
| φ, square, uncertain | .045 | 15.6 | 2.0 | 7.8× | 6.5 | 4.1 | 0/30 | |
| θ, cube, uncertain | .077 | 11.7 | 2.2 | 5.3× | 2.0 (5.7) | 3.2 (6.8) | 0/30 | |
| φ, cube, uncertain | .041 | 14.0 | 1.9 | 7.4× | 2.0 (5.8) | 1.4 (4.5) | 0/23 |
Blimp φ (cube) and φ (cube, uncertain) have 29 and 23 episodes: from the remaining starts, no method can begin, because the initial synthesis problem itself has no solution.
What the table shows:
CPU energy per planning step on the four cube specifications, measured with Intel RAPL on a laptop (Intel Core Ultra 7 265U), one evaluation at a time on an otherwise idle machine. Energy is what the processor package consumed while computing (solves, proposer inference and checks), divided by the number of planning steps, the same definition as the computation time. Inference and checks are shorter than RAPL's update interval, so their energy per call is measured over many repetitions. Times in this table are from the same runs, on the laptop.
| System | Spec. | Energy I-STL [mJ/step] | Energy LIRRA [mJ/step] | Energy reduction | Time I-STL [ms/step] | Time LIRRA [ms/step] |
|---|---|---|---|---|---|---|
| Skydio X2 | θ, cube | 1848 | 13.7 | 135× | 119.5 | 1.3 |
| φ, cube | 2269 | 18.2 | 124× | 136.6 | 1.8 | |
| θ, cube, uncertain | 2038 | 51.7 | 39× | 135.7 | 2.7 | |
| φ, cube, uncertain | 2063 | 38.5 | 54× | 137.3 | 2.1 | |
| Blimp | θ, cube | 148 | 12.0 | 12× | 7.0 | 0.9 |
| φ, cube | 166 | 11.9 | 14× | 9.6 | 0.9 | |
| θ, cube, uncertain | 146 | 18.2 | 8.0× | 9.2 | 1.2 | |
| φ, cube, uncertain | 175 | 15.8 | 11× | 12.0 | 1.1 |
The processor draws 14–24 W while solving in every run, so energy follows computation time: LIRRA reduces energy per step 39–135× on the X2 and 8–14× on the blimp, against 51–92× and 8–11× for computation time on this machine.
Each uncertain-boundary episode draws a true region (half-width within ±0.05 m, rotation within 0.05 rad). The ablation plans and checks with the nominal faces instead, and is scored against the true region. Episodes that entered the true region:
| System | Specification | I-STL, interval | LIRRA, interval | I-STL, nominal faces |
|---|---|---|---|---|
| Skydio X2 | θ, square, uncertain | 0/30 | 0/30 | 4/30 |
| φ, square, uncertain | 0/30 | 0/30 | 12/30 | |
| θ, cube, uncertain | 0/30 | 0/30 | 7/30 | |
| φ, cube, uncertain | 0/30 | 0/30 | 12/30 | |
| Blimp | θ, square, uncertain | 0/30 | 0/30 | 15/30 |
| φ, square, uncertain | 0/30 | 0/30 | 16/30 | |
| θ, cube, uncertain | 0/30 | 0/30 | 17/30 | |
| φ, cube, uncertain | 0/23 | 0/23 | 19/29 |
Planning against the nominal faces looks safe to the planner and still enters the real region in up to two thirds of the episodes.
θ and φ on the square, pooled (60 episodes per row: 10 start states × 3 uncertainty levels × 2 formulas), comparing four loops: always-on synthesis; LIRRA with a first-stage proposer (behavior cloning and PPO); LIRRA with the full-pipeline proposer; and the full-pipeline proposer flown without the check.
| Platform | Loop | Solves / episode | Per solve [ms] | RL + check [ms] | Compute / step [ms] | Waypoints | Violated episodes | Min ρ |
|---|---|---|---|---|---|---|---|---|
| Double integrator | STL solver only | 120.0 | 121.9 | – | 60.94 | 6.0 | 0 | 0.000 |
| RL only, no check | 0.0 | – | 0.73 | 0.73 | 6.0 | 0 | 0.008 | |
| First stage, checked | 2.8 | 91.4 | 1.34 | 2.39 | 6.0 | 0 | 0.021 | |
| Full pipeline, checked | 0.2 | 71.9 | 1.12 | 1.19 | 6.0 | 0 | 0.035 | |
| Skydio X2 | STL solver only | 300.0 | 138.0 | – | 137.97 | 6.0 | 0 | 0.003 |
| RL only, no check | 0.0 | – | 0.84 | 0.84 | 6.0 | 0 | 0.011 | |
| First stage, checked | 8.4 | 150.3 | 1.11 | 5.29 | 6.0 | 0 | 0.003 | |
| Full pipeline, checked | 2.2 | 135.3 | 1.10 | 2.08 | 6.0 | 0 | 0.020 | |
| Blimp | STL solver only | 240.0 | 15.3 | – | 15.29 | 9.8 | 0 | 0.001 |
| RL only, no check | 0.0 | – | 1.04 | 1.04 | 5.8 | 16 | −0.054 | |
| First stage, checked | 59.4 | 18.4 | 2.00 | 6.05 | 6.0 | 0 | 0.000 | |
| Full pipeline, checked | 19.9 | 16.9 | 2.33 | 3.54 | 5.5 | 0 | 0.001 |
The check is what keeps the blimp safe: unchecked, the same policy violated the specification in 16 of 60 episodes; checked, in none. On the cube, the unchecked final cloned and PPO proposers violated θ or φ in 57 of 238 episodes, including 48 of 60 blimp θ episodes.
Each row removes or weakens one part of LIRRA on the four cube specifications (θ and φ, with exact and uncertain boundaries), with the same full-pipeline proposers and start states as the results table. Episodes are pooled over the four specifications; the blimp has 112 because 8 of its starts have no initial solution for any method.
| System | Loop | Episodes | Violated episodes | Fallback solves | Solves with no solution | Missed deadlines |
|---|---|---|---|---|---|---|
| Skydio X2 | LIRRA (full check) | 120 | 0 | 332 | 0 | 0 |
| Nominal robustness trigger | 120 | 0 | 44 | 0 | 0 | |
| No delay compensation | 120 | 2 | 427 | 134 | 0 | |
| LIRRA, measured solver latency | 120 | 0 | 328 | 0 | 0 | |
| Always-on I-STL, measured solver latency | 120 | 0 | 35,992 | 231 | 7 | |
| Blimp | LIRRA (full check) | 112 | 0 | 1,541 | 3 | 0 |
| Nominal robustness trigger | 112 | 0 | 460 | 90 | 0 | |
| No delay compensation | 112 | 2 | 1,671 | 437 | 0 | |
| LIRRA, measured solver latency | 112 | 0 | 1,541 | 3 | 0 | |
| Always-on I-STL, measured solver latency | 112 | 0 | 26,880 | 44 | 0 |
What each row shows:
With real (measured) solver times, always-on synthesis on the X2 missed 3 (θ) and 5 (φ) of about 9,000 solve deadlines, with worst solves of 0.96 s and 0.93 s against a 0.2 s planning period. A missed deadline leaves the cached plan in force, so none of these led to a violation. With the full-pipeline proposer the X2 issued 23 (θ) and 103 (φ) requests in total, with no missed deadlines.
The guarantee on the MuJoCo vehicles assumes that the tracking error stays inside the assumed tube \(\varepsilon\) (assumption (A3) in Theory). Checked loops on the X2, by tube size:
| Tube \(\varepsilon\) [m] | Episodes | Violated | Warm-start failures / requests | Max planar error [m] |
|---|---|---|---|---|
| 0.05 | 340 | 7 | 42 / 12,984 | 0.112 |
| 0.10 | 340 | 0 | 0 / 13,068 | 0.118 |
| 0.15 | 340 | 0 | 0 / 14,161 | 0.119 |
The X2 tracker's error reaches about 0.12 m. A 0.05 m tube is therefore too small, the assumption fails, and both the warm start and the specification can fail with it. From 0.10 m the error stays (almost always) inside the tube, and both counts drop to zero. On the blimp, the largest planar deviation from the cached plan at planning instants was 4 cm, inside \(\varepsilon = 0.10\) m.
The full pipeline for one system and specification, here the X2 with θ on the cube (see Installation & Usage for setup, a Gurobi licence and the names of all eight specifications):
For an uncertain-boundary specification, use theta_plus_u (or any _u name) throughout, and add --pairs always_on --nominal-predicates to step 4 for the nominal-boundary comparison. experiments/patrol/main_table.py then builds the results tables on this page from the evaluation runs.