Enhancing Safety and Robustness of Vision-Based Controllers via Reachability Analysis

Audio Overview (generated using NotebookLM)

Abstract

Autonomous systems, such as self-driving cars and drones, have made significant strides in recent years by leveraging visual inputs and machine learning for decision-making and control. Despite their impressive performance, these vision-based controllers can make erroneous predictions when faced with novel or out-of-distribution inputs. Such errors can cascade into catastrophic system failures and compromise system safety. In this work, we compute Neural Reachable Tubes, which act as parameterized approximations of Backward Reachable Tubes to stress-test the vision-based controllers and mine their failure modes. The identified failures are then used to enhance the system safety through both offline and online methods. The online approach involves training a classifier as a run-time failure monitor to detect closed-loop, system-level failures, subsequently triggering a fallback controller that robustly handles these de- tected failures to preserve system safety. For the offline approach, we improve the original controller via incremental training using a carefully augmented failure dataset, resulting in a more robust controller that is resistant to the known failure modes. In either approach, the system is safeguarded against shortcomings that transcend the vision-based controller and pertain to the safety of the overall system. We validate the proposed approaches on an autonomous aircraft taxiing task that involves using a vision- based controller to guide the aircraft towards the centerline of the runway. Our results show the efficacy of the algorithms in identifying and handling system-level failures, outperforming methods that rely on controller prediction error or uncertainty quantification for identifying system failures.

Failure Mining : A Reachability Approach

We leverage Hamilton-Jacobi Reachability Theory to compute the Neural Reachable Tubes (NRTs), a parameterized version of the Backward Reachable Tubes (BRTs) of the system over varying environmental conditions (parameterized by 'd'). Analyzing the variations in NRTs help us to identify specific failure modes of the vision-based controller. Here we present the NRTs for an autonomous aircraft taxiing system (TaxiNet).

Fig. NRT slices with variation of parameters (top row) and a sample image (bottom row) (a) d1 = morning, d2 = clear, (b) d1 = evening, d2 = clear, (c) d1 = night, d2 = clear and (d) d1 = morning, d2 = overcast, for a fixed initial starting py = 110m.
Analyzing the NRTs in general and across varying different parameters, like time of day and cloud conidtions here, we can identify the failures of underlying vision-based controller. Some of the interesting semantic failure modes of TaxiNet controller are:

Aircraft undergoes failure when starting close to the runway boundary as it assumes it to be the centerline.
Aircraft fails to identify runway centerline in night time, causing failure.
Aircraft assumes the runway markings as centerline and deviate from the runway.

Fig. (a) The overlaid NRTs for d1 =night (blue) on d1 =morning (cyan) for py = 110m. The state of interest, shown with a yellow star, is only contained in the morning NRT and not in the night NRT. (b) Top-view of the runway in the morning. The trajectory, “A” to “C”, followed by the aircraft under the CNN policy (cyan line), takes it off the runway in the morning. The trajectory (blue line) from “A” to “B” is followed at night. (c) The runway marking in the image, which acts as a failure mode, can be vividly seen by the CNN at location “A” in the morning cannot be seen clearly at night (d) due to poor illumination. (e) The overlaid NRTs for d1 =morning (cyan) on d1 =night (blue) for py = 190m. The state, shown with a yellow star, is only included in the night NRT. (f) Top view of the runway. In the morning, the CNN policy accomplishes the taxiing task by taking the cyan trajectory from “A” (yellow star in (a)) to “C.” At night, the policy takes the aircraft outside the runway along the blue trajectory from “A” to “B”. (g) The centreline in the image can be seen clearly by the CNN at location “A” in the morning, whereas it cannot be seen at night (h) due to poor illumination..

Online Failure Detection

Leveraging the knowledge of failure modes of a vision-based controller, we can construct a dataset of visual inputs which can be automatically labelled as "safe" or "unsafe" depending on if they can potentially cause a system failure or not, which is verified through NRTs. This dataset can be then used to train a binary classifier that can detect such potentially unsafe visual inputs and trigger a fallback controller to maintain systme safety.

We use an EfficientNet-B0 model fine-tuned on safe and unsafe visual inputs across multiple runways and environmental conditions. Further, we provide proper certificates on its performance for reliable safety maintenance using Conformal Prediction Theory.

We implemented Class-Conditioned Conformal Prediction method to obtain a new decision threshold for predicted probabilities, providing coverage guarantees, which we further prove to be a guarantee on the Recall Metric of the classifier. Thus, by choosing an appropriate decision threshold, we can obtain a failure detector with Safety-First behaviour with a quantified confidence.
(Refer the paper for implementation details on proof on recall guarantess)

Table : Performance comparison of trained classifier in testing conditions, unseen during training, after using conformal prediction.
(Refer the paper for further comparison on calibration the decision threshold online using privileged testing data).

Fallback Mechanisms for Online Failure Mitigation

We present two possible fallback mechanisms for the TaxiNet controller.

Obtaining a noisy state-estimate from additional sensors like GPS.
Modifying high-level control inputs to take the aircraft back to safe region.

Fig. (a) NRT under GPS-based state estimation fallback mechanism (yellow) and NRT under TaxiNet controller (blue). (b) System NRT under default TaxiNet controller (blue) and NRT under the safety pipeline (yellow). The NRT obtained using the FD and the fallback controller is appreciably smaller than the NRT under TaxiNet. (c, d) Trajectory followed by the aircraft under the TaxiNet controller (dashed blue line) and the safety pipeline (yellow line). The color shift in the yellow curve shows velocity variation due to the fallback controller.

Incremental Training

Another interesting approach to robustify vision-based controllers is to incrementally re-train them on newly-encountered unsafe visual inputs. It will keep collecting new data and perform continual training. However, the re-training has to be performed in a way to prevent catastrophic forgetting of the controller model. Here we show a single iteration of incremental training. We noticed 20% reduction with re-trained controller and 10% reduction in its prediction error. However, we also noticed some cases when the performance degrades compared to the original network.

Fig. (a) NRT slices for py = 110m in morning on KMWH runway (training dataset), with the incremental controller NRT (green) overlaid on the default controller NRT (blue). Trajectories on the KMWH runway under (b) default controller and (c) incrementally trained controller. The incrementally trained controller shows significantly fewer failures than the default version. (d) NRT slices for py = 110m in morning on KATL runway, with the incremental controller NRT (green) overlaid on the default controller NRT (blue). (I) marks an area where the incremental controller does better than the default version, while (II) marks an area where the default controller is better. (e) Trajectories on the KATL runway with default controller show multiple failure trajectories. (f) Trajectories on the KATL runway with incrementally trained controller showing failure due to effects not seen in the augmented dataset (eg. another runway merging from the right).

BibTeX


    @misc{chakraborty2024enhancingsafetyrobustnessvisionbased,
    title={Enhancing Safety and Robustness of Vision-Based Controllers via Reachability Analysis}, 
    author={Kaustav Chakraborty and Aryaman Gupta and Somil Bansal},
    year={2024},
    eprint={2410.21736},
    archivePrefix={arXiv},
    primaryClass={cs.RO},
    url={https://arxiv.org/abs/2410.21736}}