“Stereo Camera-based Free Space Estimation for Docking in Urban Waters”
Authors: Trym A. Nygård, Nicholas Dalhaug, Rudolf Mester, Edmund Brekke and Annette Stahl,Affiliation: NTNU, Department of Engineering Cybernetics and NTNU
Reference: 2024, Vol 45, No 2, pp. 51-63.
Keywords: Maritime autonomy, Free space, Safe navigation, Stereo camera, Docking
Abstract: Operating in urban waters with an autonomous vessel can be challenging. The autonomous vessel must be able to react quickly and detect obstacles to avoid collisions and risky maneuvers. Exteroceptive sensors such as LiDAR and RADAR have typically been used with great success in the maritime domain, but the measurements are often too sparse to represent smaller obstacles during docking and other maritime operations. However, other sensor modalities, such as stereo cameras, can provide both appearance and dense depth information. In this paper, we present a stereo camera-based free space estimation method for the maritime domain. The mapping of navigable areas is crucial for path planning and collision avoidance systems. To robustly estimate the free space, we use vertically oriented rectangular segments known as stixels. We utilized both stereo correspondences and a recent image segmentation network trained on a large, generalized dataset to create the stixels. To validate our approach, we analyzed the estimated free space, evaluating both the accuracy and consistency in the estimated depth over time. We demonstrate the approach using a real dataset recorded with a stereo camera mounted on an autonomous ferry and compare its accuracy against measurements from a LiDAR.
PDF (21918 Kb)
DOI: 10.4173/mic.2024.2.2References:
[1] Badino, H., Franke, U., and Mester, R. (2007). Free Space Computation Using Stochastic Occupancy Grids and Dynamic Programming, In Workshop on Dynamical Vision, ICCV, Rio de Janeiro, Brazil, volume20. page73.
[2] Badino, H., Franke, U., and Pfeiffer, D. (2009). The Stixel World - A Compact Medium Level Representation of the 3D-World, In J.Denzler, G.Notni, and H.Süße, editors, Pattern Recognition, Lecture Notes in Computer Science. Springer, Berlin, Heidelberg, pages 51--60. doi:10.1007/978-3-642-03798-6_6
[3] Bovcon, B. and Kristan, M. (2022). WaSR—A Water Segmentation and Refinement Maritime Obstacle Detection Network, IEEE Transactions on Cybernetics. 52(12):12661--12674. doi:10.1109/TCYB.2021.3085856
[4] Elfes, A. (1989). Using occupancy grids for mobile robot perception and navigation, Computer. 22(6):46--57. doi:10.1109/2.30720
[5] Fischler, M.A. and Bolles, R.C. (1987). Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography, In M.A. Fischler and O.Firschein, editors, Readings in Computer Vision, pages 726--740. Morgan Kaufmann, San Francisco (CA). doi:10.1016/B978-0-08-051581-6.50070-2
[6] Griesser, D., Umlauf, G., and Franz, M.O. (2023). Visual Pitch and Roll Estimation For Inland Water Vessels, In 2023 IEEE International Conference on Robotics and Automation (ICRA). pages 1961--1967. doi:10.1109/ICRA48891.2023.10160460
[7] Helgesen, O.K., Thyri, E.H., Brekke, E., Stahl, A., and Breivik, M. (2023). Experimental validation of camera-based maritime collision avoidance for autonomous urban passenger ferries, Modeling, Identification and Control: A Norwegian Research Bulletin. 44(2):55--68. doi:10.4173/mic.2023.2.2
[8] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.-Y., Dollár, P., and Girshick, R. (2023). Segment Anything, 2023. doi:10.48550/arXiv.2304.02643
[9] Muhovič, J., Mandeljc, R., Bovcon, B., Kristan, M., and Perš, J. (2020). Obstacle Tracking for Unmanned Surface Vessels Using 3-D Point Cloud, IEEE Journal of Oceanic Engineering. 45(3):786--798. doi:10.1109/JOE.2019.2909507
[10] Pfeiffer, D. and Franke, U. (2010). Efficient representation of traffic scenes by means of dynamic stixels, In 2010 IEEE Intelligent Vehicles Symposium. pages 217--224. doi:10.1109/IVS.2010.5548114
[11] Pfeiffer, D. and Franke, U. (2011). Towards a Global Optimal Multi-Layer Stixel Representation of Dense 3D Data, In the British Machine Vision Conference. British Machine Vision Association, Dundee, pages 51.1--51.12. doi:10.5244/C.25.51
[12] Pinggera, P., Ramos, S., Gehrig, S., Franke, U., Rother, C., and Mester, R. (2016). Lost and Found: detecting small road hazards for self-driving vehicles, In 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pages 1099--1106. doi:10.1109/IROS.2016.7759186
[13] Plenge-Feidenhans'l, M. (2023). Robust Free Area Mapping for Autonomous Harbour Navigation, Ph.D. thesis, Technical University of Denmark.
[14] Plenge-Feidenhans’l, M.K. and Blanke, M. (2021). Open Water Detection for Autonomous In-harbor Navigation Using a Classification Network, IFAC-PapersOnLine. 54(16):30--36. doi:10.1016/j.ifacol.2021.10.069
[15] Schneider, L., Cordts, M., Rehfeld, T., Pfeiffer, D., Enzweiler, M., Franke, U., Pollefeys, M., and Roth, S. (2016). Semantic Stixels: Depth is not enough, In 2016 IEEE Intelligent Vehicles Symposium (IV). pages 110--117. doi:10.1109/IVS.2016.7535373
[16] Soquet, N., Perrollaz, M., Labayrade, R., and Aubert, D. (2007). Free Space Estimation for Autonomous Navigation, International Conference on Computer Vision Systems. doi:10.2390/biecoll-icvs2007-30
[17] Volden, O., Stahl, A., and Fossen, T.I. (2022). Vision-based positioning system for auto-docking of unmanned surface vehicles (USVs), International Journal of Intelligent Robotics and Applications. 6(1):86--103. doi:10.1007/s41315-021-00193-0
[18] Wan, E. and Van DerMerwe, R. (2000). The unscented Kalman filter for nonlinear estimation, In IEEE Adaptive Systems for Signal Processing, Communications, and Control Symposium. pages 153--158. doi:10.1109/ASSPCC.2000.882463
[19] Yao, J., Ramalingam, S., Taguchi, Y., Miki, Y., and Urtasun, R. (2015). Estimating Drivable Collision-Free Space from Monocular Video, In IEEE Winter Conference on Applications of Computer Vision. pages 420--427. doi:10.1109/WACV.2015.62
[20] Zhao, X., Ding, W., An, Y., Du, Y., Yu, T., Li, M., Tang, M., and Wang, J. (2023). Fast Segment Anything, 2023. doi:10.48550/arXiv.2306.12156
BibTeX:
@article{MIC-2024-2-2,
title={{Stereo Camera-based Free Space Estimation for Docking in Urban Waters}},
author={Nygård, Trym A. and Dalhaug, Nicholas and Mester, Rudolf and Brekke, Edmund and Stahl, Annette},
journal={Modeling, Identification and Control},
volume={45},
number={2},
pages={51--63},
year={2024},
doi={10.4173/mic.2024.2.2},
publisher={Norwegian Society of Automatic Control}
};