“Evaluation of Subjective and Objective Performance Metrics for Haptically Controlled Robotic Systems”
Authors: Cong Dung Pham, Huynh Nhat Trinh Phan and Pål J. From,Affiliation: Norwegian University of Life Sciences
Reference: 2014, Vol 35, No 3, pp. 147-157.
Keywords: Performance evaluation, controller design, human-robot interface, mobile manipulation
Abstract: This paper studies in detail how different evaluation methods perform when it comes to describing the performance of haptically controlled mobile manipulators. Particularly, we investigate how well subjective metrics perform compared to objective metrics. To find the best metrics to describe the performance of a control scheme is challenging when human operators are involved; how the user perceives the performance of the controller does not necessarily correspond to the directly measurable metrics normally used in controller evaluation. It is therefore important to study whether there is any correspondence between how the user perceives the performance of a controller, and how it performs in terms of directly measurable metrics such as the time used to perform a task, number of errors, accuracy, and so on. To perform these tests we choose a system that consists of a mobile manipulator that is controlled by an operator through a haptic device. This is a good system for studying different performance metrics as the performance can be determined by subjective metrics based on feedback from the users, and also as objective and directly measurable metrics. The system consists of a robotic arm which provides for interaction and manipulation, which is mounted on a mobile base which extends the workspace of the arm. The operator thus needs to perform both interaction and locomotion using a single haptic device. While the position of the on-board camera is determined by the base motion, the principal control objective is the motion of the manipulator arm. This calls for intelligent control allocation between the base and the manipulator arm in order to obtain intuitive control of both the camera and the arm. We implement three different approaches to the control allocation problem, i.e., whether the vehicle or manipulator arm actuation is applied to generate the desired motion. The performance of the different control schemes is evaluated, and our findings strongly suggest that objective metrics better describe the performance of the controller, even though there is a clear correlation between subjective and objective performance metrics.
PDF (715 Kb)
DOI: 10.4173/mic.2014.3.2DOI forward links to this article:
| [1] M. Talha, E. A. M. Ghalamzan, C. Takahashi, J. Kuo, W. Ingamells and R. Stolkin (2016), doi:10.1109/SSRR.2016.7784294 |
| [2] Iina Aaltonen, Susanna Aromaa, Kaj Helin and Ali Muhammad (2018), doi:10.1007/978-3-319-60384-1_32 |
| [3] Sam Van Damme, Maria Torres Vega and Filip De Turck (2020), doi:10.1109/NetSoft48620.2020.9165335 |
[1] Adams, J. and Kaymaz-Keskinpala, H. (2004). Analysis of perceived workload when using a pda for mobile robot teleoperation, In Proceedings of IEEE International Conference on Robotics and Automation, volume4. pages 4128--4133 Vol.4. doi:10.1109/ROBOT.2004.1308919
[2] Farkhatdinov, I. and Ryu, J.H. (2008). Switching of control signals in teleoperation systems: Formalization and application, In Advanced Intelligent Mechatronics. AIM 2008. IEEE/ASME International Conference on. pages 353--358. doi:10.1109/AIM.2008.4601686
[3] Franken, M., Stramigioli, S., Misra, S., Secchi, C., and Macchelli, A. (2011). Bilateral telemanipulation with time delays: A two-layer approach combining passivity and transparency, Robotics, IEEE Transactions on. 27(4):741--756. doi:10.1109/TRO.2011.2142430
[4] From, P.J., Duindam, V., Pettersen, K.Y., Gravdahl, J.T., and Sastry, S. (2010). Singularity-free dynamic equations of vehicle-manipulator systems, Simulation Modelling Practice and Theory. 18(6):712--731.
[5] From, P.J., Pettersen, K.Y., and Gravdahl., J.T. (2014). Vehicle-manipulator systems - modeling for simulation, analysis, and control, Springer Verlag, London, UK.
[6] Goodrich, M.A., Boer, E.R., Crandall, J.W., Ricks, R.W., and Quigley, M.L. (2004). Behavioral entropy in human-robot interaction, Technical report, DTIC Document.
[7] Goodrich, M.A. and Schultz, A.C. (2007). Human-robot interaction: a survey, Foundations and trends in human-computer interaction. 1(3):203--275.
[8] Grane, C. and Bengtsson, P. (2005). Menu selection with a rotary device founded on haptic and/or graphic information, In Eurohaptics Conference, 2005 and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics 2005. First Joint. pages 475--476.
[9] Hannaford, B. (1989). A design framework for teleoperators with kinesthetic feedback, Robotics and Automation, IEEE Transactions on. 5(4):426--434. doi:10.1109/70.88057
[10] Hokayem, P.F. and Spong, M.W. (2006). Bilateral teleoperation: An historical survey, Automatica. 42(12):2035--2057. doi:10.1016/j.automatica.2006.06.027
[11] Kaber, D.B., Onal, E., and Endsley, M.R. (2000). Design of automation for telerobots and the effect on performance, operator situation awareness, and subjective workload, Human Factors and Ergonomics in Manufacturing, 2000. 10(4):409--430. doi:10.1002/1520-6564(200023)10:4<409::AID-HFM4>3.0.CO;2-V
[12] Kaber, D.B., Riley, J.M., Zhou, R., and Draper, J. (2000). Effects of visual interface design, and control mode and latency on performance, telepresence and workload in a teleoperation task, In Proceedings of the Human Factors and Ergonomics Society Annual Meeting, volume44. SAGE Publications, pages 503--506, 2000.
[13] Kiselev, A. and Loutfi, A. (2012). Using a mental workload index as a measure of usability of a user interface for social robotic telepresence, In Workshop in Social Robotics Telepresence. 2012.
[14] Park, J. and Khatib, O. (2006). Robust haptic teleoperation of a mobile manipulation platform, 2006.
[15] Pham, C.D. and From, P.J. (2013). Control allocation for mobile manipulators with on-board cameras, In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pages 5002--5008. doi:10.1109/IROS.2013.6697079
[16] Rook, A. and Hogema, J. (2005). Effects of human-machine interface design for intelligent speed adaptation on driving behavior and acceptance, Transportation Research Record: Journal of the Transportation Research Board. 1937(-1):79--86.
[17] Ross, T. and Burnett, G. (2001). Evaluating the human–-machine interface to vehicle navigation systems as an example of ubiquitous computing, International Journal of Human-Computer Studies. 55(4):661 -- 674. doi:10.1006/ijhc.2001.0495
[18] Rubio, S., Diaz, E., Martin, J., and Puente, J.M. (2004). Evaluation of subjective mental workload: A comparison of swat, nasa-tlx, and workload profile methods, Applied Psychology. 53(1):61--86. doi:10.1111/j.1464-0597.2004.00161.x
[19] Ryu, J.H., Kim, Y.S., and Hannaford, B. (2004). Sampled- and continuous-time passivity and stability of virtual environments, Robotics, IEEE Transactions on, 2004. 20(4):772--776. doi:10.1109/TRO.2004.829453
[20] Ryu, J.H., Kwon, D.S., and Hannaford, B. (2004). Stable teleoperation with time-domain passivity control, Robotics and Automation, IEEE Transactions on, 2004. 20(2):365--373. doi:10.1109/TRA.2004.824689
[21] Seraji, H. (1998). A unified approach to motion control of mobile manipulators, International Journal of Robotics Research. 17(2):107--118. doiprefixdoi10.1177/027836499801700201. Seraji, H doi:10.1177/027836499801700201 ewblock Seraji, H.
[22] Stefanidis, D., Wang, F., KorndorfferJr, J.R., Dunne, J.B., and Scott, D.J. (2010). Robotic assistance improves intracorporeal suturing performance and safety in the operating room while decreasing operator workload, Surgical endoscopy. 24(2):377--382. doi:10.1007/s00464-009-0578-0
[23] Steinfeld, A., Fong, T., Kaber, D., Lewis, M., Scholtz, J., Schultz, A., and Goodrich, M. (2006). Common metrics for human-robot interaction, In Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction. ACM, pages 33--40.
[24] Wrock, M.R. and Nokleby, S.B. (2011). Decoupled teleoperation of a holonomic mobile-manipulator system using automatic switching, In Electrical and Computer Engineering (CCECE), 2011 24th Canadian Conference on. pages 001164--00116.
BibTeX:
@article{MIC-2014-3-2,
title={{Evaluation of Subjective and Objective Performance Metrics for Haptically Controlled Robotic Systems}},
author={Pham, Cong Dung and Phan, Huynh Nhat Trinh and From, Pål J.},
journal={Modeling, Identification and Control},
volume={35},
number={3},
pages={147--157},
year={2014},
doi={10.4173/mic.2014.3.2},
publisher={Norwegian Society of Automatic Control}
};