Planning-level patrol environment for PPO fine-tuning of the segment proposer (pure JAX).
One environment step is one planning step ``dt``. The action is a whole proposal: ``N``
synthesis-model inputs in ``(-1, 1)`` (Brax's tanh-Gaussian policy), mapped into the tightened
input box exactly as at runtime (:func:`rl_stl.policies.segment.squash` semantics). The reward
scores the *proposal*, so PPO optimises what the runtime check and the synthesis objective
reward, and no solver is ever called inside the JIT-compiled batch:
r = - c_cost * J(plan; nominal) - c_cert * max(0, eps - cert(plan)) + c_ok * 1[cert >= eps]
- c_syn * 1[cert < 0]
``J`` is the synthesis objective (discounted L1 deviation from the nominal patrol reference plus
input effort), ``cert`` the differentiable runtime check (:func:`patrol_certificate`). The
first planned input is then applied to ``f_syn`` with uncertainty inside the level's bound:
* double integrator: ``x+ = A x + B (u + w)``, ``w`` uniform in ``[-w_bar, w_bar]``;
* constant-tube vehicles (X2, blimp): the measured state is the planned one plus a position
tracking error uniform in ``[-eps, eps]`` on every signal axis (planar, or x, y, z for the cube
specs; the tube assumption; MuJoCo is not in the loop).
Initial states: positions on a ring (2-D) or a spherical shell (cube specs) of radius ``start_r``
around the centre of ``S``, uniform velocities on the patrol axes.
The reference is the one the tables were built for (``LevelTables.reference``): ``rerooted``
rolls the nominal out from the current state, with the waypoint index advanced on the realised
trajectory; ``clock`` rolls the nominal patrol out once at reset, from the episode's initial
state (inputs saturated at ``LevelTables.ref_input_scale`` times the bounds), and the plan
rooted at step ``k`` tracks ``x_nom(k..k+N)``. The ``waypoints`` metric
counts waypoints the vehicle itself reaches (tolerance as in the evaluation), whichever
reference the reward uses.
Optional path-accuracy term (``c_wp > 0``; off by default): the executed state's progress along the
closed patrol polyline through the waypoints (the projection of :func:`rl_stl.evaluation.patrol.
patrol_progress`) is tracked, and a step on which it passes a waypoint is scored as in
:func:`~rl_stl.evaluation.patrol.waypoint_miss`: with ``miss`` the distance to the waypoint at the
(linearly interpolated) crossing,
r_wp = c_wp * ( 1[forward pass] * max(0, 1 - miss / wp_radius) - 1[backward pass] )
so a precise pass earns up to ``c_wp``, a pass ``wp_radius`` away nothing, and crossing a waypoint
backwards costs the full bonus (no gain from oscillating across a waypoint). With
``wp_target = "plan"`` the same score is summed over the steps of the proposed segment ``X`` (its
``f_syn`` rollout from the current state) instead of the executed step, so the term rates the action
itself at every planning step whose segment passes a waypoint. It only shapes the
reward; the runtime check, the certificate terms and the fallback dynamics are unchanged.
With ``fallback`` the env models the checked loop: a proposal that fails the runtime check
(``cert < 0``) is not flown; the vehicle follows the cached plan (the last accepted proposal),
reaching its next nominal state up to the tube error as the tracker does (constant tube) or
applying its next input plus ``w`` (disturbance), and once the cache is exhausted a stopping
input (PD on velocity, regulated axes to their set-points). Before the first proposal passes
there is no certified plan (the real loop solves at ``t = 0-``), so proposals are flown.
The solver is not modelled: its plan is approximated by the shifted cached plan, which is a
feasible point of the request. ``c_syn`` prices the synthesis call that each rejection triggers
in the real loop. Without ``fallback`` every proposal's first input is flown, as before.
Observations are the runtime features (:func:`features`) normalised with fixed statistics
(those of the behaviour-cloning data) so a BC network maps onto Brax's policy unchanged.