User interfaces for human-robot interaction: Application on a semi-autonomous agricultural robot sprayer

Abstract

This dissertation focuses on the usability of user interfaces for tele-operated and telemanipulated mobile robots, with an application on a semi-autonomous agricultural robot sprayer. Semi-autonomous operation of agricultural robots is proposed including a framework for its levels of autonomy. In this case, the robot, in addition to whatever pre-programmed operation can do, is in communication with a human operator (farmer), who intervenes when needed. The farmer does not need to be present in the field; for reasons of occupational comfort and safety (as in the case of spraying which is the example discussed here) as well as for reasons of efficiency (as in the case of operating multiple robots in tandem which is not discussed here), the farmer is assumed to be “away”. The objective of this dissertation is to study the design and evaluation aspects of a user interface that supports human-robot interaction, for semi-autonomous agricultural spraying robots. Various aspects related to the user interface design and evaluation that can enhance human-robot interaction are investigated within this thesis including: 1) custom transformation of a robotic platform into a piece of agricultural machinery, 2) proposing a framework for semiautonomous robot modes of operation, 3) a taxonomy of user interface guidelines / heuristics for tele-operated field robots, 4) studies and experiments with the design aspects of user interfaces for robot tele-operation and tele-manipulation for the specific tasks of navigation, target identification and spraying, and 5) development and evaluation of suitable interfaces with enhanced human-robot interaction awareness to the farmer to effectively tele-operate a semiautonomous vineyard robotic sprayer. Specifically, this dissertation starts with the methodological approach followed to transform an existing robotic platform to a semi-autonomous agricultural robot sprayer (AgriRobot). This is followed by the proposed levels of autonomy. The semi-autonomous mode is the mode of operation where one or more operations are in manual mode and one or more operations are in autonomous mode. The robot has operations both in manual and in autonomous modes, concurrently. This formal framework brings out human-robot interaction theoretical issues of human-robot interaction and more practical issues specific to the user interface design framework. This is followed by a systematic approach to develop a taxonomy of design guidelines for robot teleoperation developed from a focused literature review of robot teleoperation. A list of user interface design guidelines was assembled, open card sorting and a focus group were used to classify them, and closed card sorting was employed to validate and further refine the proposed taxonomy. The initially obtained set of 70 guidelines is grouped into eight categories: platform architecture and scalability, error prevention and recovery, visual design, information presentation, robot state awareness, interaction effectiveness and efficiency, robot environment/surroundings awareness, and cognitive factors. The semi-autonomous agricultural robot sprayer constructed was used as an application case study for implementation and field evaluation. The proposed guidelines taxonomy was used heuristically to evaluate the usability of existing user interfaces of the teleoperated agricultural robot sprayer. In terms of experimentation, the first step was to determine how to begin work in this research area. Initially, without the resources to experiment in the field, as a first step we used an effective test-bed - a simulation experiment in a lab – to evaluate the usability of three different input devices. The goal was to evaluate the selection input device (Mouse vs Wiimote vs Digital pen) for marking the targets (grape clusters). Results indicated usability preference for the mouse and the digital pen. Later, in a field experiment, the usability of different interaction modes for agricultural robot teleoperation was also investigated. Specifically, two different types of peripheral vision support mechanism, two different types of control input devices, two different types of output devices and the overall influence of the user interface on observed and perceived usability of a teleoperated agricultural sprayer were examined. Specific recommendations for mobile field robot teleoperation to improve HRI awareness for the agricultural spraying task were drawn. A value-added from this dissertation is the placing of a camera on top of the end-effector sprayer to provide accurate target identification and spraying verification, thus improving activity awareness. Similarly, placing a camera at the back-top of the robot provides peripheral vision and enables the operator to locate obstacles around the robot wheels, thus improving location and surroundings awareness. Regarding the input/output devices, the PC keyboard and monitor were preferred over the PS3 gamepad and the head mounted display. The dissertation concludes with a discussion on the research findings and suggestions for future research directions. In sum, this work described aspects of how a robotic system should be designed (i.e. asking users how they expect the robot to perform tasks), defining levels of autonomy (including levels and type of communication), using heuristics and design guidelines (gathered from a large body of literature specific for mobile field robots) to develop and evaluate the user interface. In terms of future research directions, these include the robotization of a tractor. In this case, the tractor can be used for several agricultural tasks which could enhance its financial feasibility. In the case of a new robot with a robotic arm installed and additional sensor capabilities (e.g. laser and LIDAR scanners), a new user interface should be developed, following the taxonomy guidelines, and experiment with other teleoperation equipment. In terms of user interface technologies, with the emergence of new sensor technologies and 3D cameras improvements, it would be worthwhile to develop user interfaces with augmented reality capabilities to investigate their effect on situational awareness of operators when using tele-robotics. Finally, it would be interesting to apply the proposed framework of the levels of autonomy to other related work in human-robot collaboration research (i.e. search and rescue robotics) including switching between collaboration levels.