Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries

“Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries”

Authors: Øystein Kaarstad Helgesen, Emil H. Thyri, Edmund Brekke, Annette Stahl and Morten Breivik,
Affiliation: NTNU, Department of Engineering Cybernetics and Zeabuz AS
Reference: 2023, Vol 44, No 2, pp. 55-68.

Keywords: Maritime autonomy, target tracking, collision avoidance, daylight cameras, full-scale experiments, autonomous surface vehicle, autonomous urban passenger ferries

Abstract: Maritime collision avoidance systems rely on accurate state estimates of other objects in the environment from a tracking system. Traditionally, this understanding is generated using one or more active sensors such as radars and lidars. Imaging sensors such as daylight cameras have recently become a popular addition to these sensor suites due to their low cost and high resolution. However, most tracking systems still rely exclusively on active sensors or a fusion of active and passive sensors. In this work, we present a complete collision avoidance system relying solely on camera tracking. The viability of this autonomous navigation system is verified through a real-world, closed-loop collision avoidance experiment with a single target in Trondheim, Norway in December 2022. Accurate tracking was established in all scenarios and the collision avoidance system took appropriate actions to avoid collisions.

PDF

PDF (3291 Kb) DOI: 10.4173/mic.2023.2.2

DOI forward links to this article:

[1] Mathias Thoresen Paasche, Oystein Kaarstad Helgesen and Edmund Forland Brekke (2023), doi:10.1088/1742-6596/2618/1/012009

[2] Yibo Li, Guobin Song, Tsz-Leung Yip and Gi-Tae Yeo (2024), doi:10.3390/jmse12050717

[3] Trym A. Nygard, Nicholas Dalhaug, Rudolf Mester, Edmund Brekke and Annette Stahl (2024), doi:10.4173/mic.2024.2.2

[4] Henrik Hilmarsen, Nicholas Dalhaug, Trym Anthonsen Nygard, Edmund Forland Brekke, Rudolf Mester and Annette Stahl (2024), doi:10.23919/FUSION59988.2024.10706307

References:
[1] Bitar, G., Eriksen, B.-O.H., Lekkas, A.M., and Breivik, M. (2019). Energy-Optimized Hybrid Collision Avoidance for ASVs, In 18th European Control Conference (ECC). pages 2522--2529, 2019. doi:10.23919/ECC.2019.8795645
[2] Bochkovskiy, A., Wang, C., and Liao, H.M. (2020). YOLOv4: Optimal Speed and Accuracy of Object Detection, CoRR. abs/2004.10934. https://arxiv.org/abs/2004.10934.
[3] Cormack, D., Schlangen, I., Hopgood, J.R., and Clark, D.E. (2020). Joint Registration and Fusion of an Infrared Camera and Scanning Radar in a Maritime Context, IEEE Transactions on Aerospace and Electronic Systems. 56(2):1357--1369. doi:10.1109/TAES.2019.2929974
[4] Eriksen, B.-O.H. (2019). Collision avoidance and motion control for autonomous surface vehicles , Ph.D. thesis, Norwegian University of Science and Technology. https://ntnuopen.ntnu.no/ntnu-xmlui/handle/11250/2616394.
[5] Eriksen, B.-O.H., Breivik, M., Wilthil, E.F., Flåten, A.L., and Brekke, E.F. (2019). The branching-course model predictive control algorithm for maritime collision avoidance, Journal of Field Robotics. 36(7):1222--1249. doi:https://doi.org/10.1002/rob.21900
[6] Fossen, T. (2021). Handbook of Marine Craft Hydrodynamics and Motion Control, Wiley. https://books.google.no/books?id=tCQqEAAAQBAJ.
[7] Fowdur, J.S., Baum, M., and Heymann, F. (2021). Real-World Marine Radar Datasets for Evaluating Target Tracking Methods, Sensors. 21(14):4641. doi:https://doi.org/10.3390/s21144641
[8] Gaglione, D., Braca, P., and Soldi, G. (2018). Belief Propagation Based AIS/Radar Data Fusion for Multi - Target Tracking, In 21st International Conference on Information Fusion (FUSION). pages 2143--2150. doi:10.23919/ICIF.2018.8455217
[9] Helgesen, O.K., Stahl, A., and Brekke, E.F. (2023). Maritime tracking with georeferenced multi-camera fusion, IEEE Access. 11:30340--30359. doi:10.1109/ACCESS.2023.3261556
[10] Helgesen, O.K., Vasstein, K., Brekke, E.F., and Stahl, A. (2022). Heterogeneous multi-sensor tracking for an autonomous surface vehicle in a littoral environment, Ocean Engineering. 252:111168. doi:https://doi.org/10.1016/j.oceaneng.2022.111168
[11] Kuwata, Y., Wolf, M.T., Zarzhitsky, D., and Huntsberger, T.L. (2014). Safe Maritime Autonomous Navigation with COLREGS, Using Velocity Obstacles, IEEE Journal of Oceanic Engineering. 39(1):110--119. doi:10.1109/JOE.2013.2254214
[12] Kuznietsov, Y., Stückler, J., and Leibe, B. (2017). Semi-Supervised Deep Learning for Monocular Depth Map Prediction, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pages 2215--2223. doi:10.1109/CVPR.2017.238
[13] Musicki, D., Evans, R., and Stankovic, S. (1992). Integrated probabilistic data association (IPDA), In 31st IEEE Conference on Decision and Control. pages 3796--3798 vol.4. doi:10.1109/CDC.1992.370951
[14] Schuster, M., Blaich, M., and Reuter, J. (2014). Collision Avoidance for Vessels using a Low-Cost Radar Sensor, 19th IFAC World Congress. 47(3):9673--9678. doi:https://doi.org/10.3182/20140824-6-ZA-1003.01872
[15] Schöller, F., Blanke, M., Plenge-Feidenhans’, M., and Nalpantidis, L. (2020). Vision-based Object Tracking in Marine Environments using Features from Neural Network Detections, 21th IFAC World Congress. 53(2):14517--14523. 21st IFAC World Congress. doi:https://doi.org/10.1016/j.ifacol.2020.12.1455
[16] Thyri, E.H., Breivik, M., and Lekkas, A.M. (2020). A Path-Velocity Decomposition Approach to Collision Avoidance for Autonomous Passenger Ferries in Confined Waters, 21th IFAC World Congress. 53(2):14628--14635. doi:https://doi.org/10.1016/j.ifacol.2020.12.1472
[17] Wilthil, E.F., Flåten, A.L., and Brekke, E.F. (2017). A target tracking system for ASV collision avoidance based on the PDAF, In P.Fossen and Nijmeijer, editors, Sensing and Control for Autonomous Vehicles, volume 474, pages 269--288. Springer, Alesund, Norway, 2017. http://dx.doi.org/10.1007/978-3-319-55372-6_13.
[18] Wolf, M.T., Assad, C., Kuwata, Y., Howard, A., Aghazarian, H., Zhu, D., Lu, T., Trebi-Ollennu, A., and Huntsberger, T. (2010). 360-degree visual detection and target tracking on an autonomous surface vehicle, Journal of Field Robotics. 27(6):819--833. doi:https://doi.org/10.1002/rob.20371
[19] Zhang, Z. (2000). A flexible new technique for camera calibration, IEEE Transactions on Pattern Analysis and Machine Intelligence. 22(11):1330--1334. doi:10.1109/34.888718

BibTeX:
@article{MIC-2023-2-2,
  title={{Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries}},
  author={Helgesen, Øystein Kaarstad and Thyri, Emil H. and Brekke, Edmund and Stahl, Annette and Breivik, Morten},
  journal={Modeling, Identification and Control},
  volume={44},
  number={2},
  pages={55--68},
  year={2023},
  doi={10.4173/mic.2023.2.2},
  publisher={Norwegian Society of Automatic Control}
};

Article

“Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries”