Teleoperation Interfaces: Challenges and guidelines for Autonomous Vehicles Research Summary

DriveU.auto helps accelerate the deployment of autonomous vehicles with teleoperations. Our connectivity platform provides 4k video, audio, data and control commands with very low latency and ultra-high reliability, using cellular bonding and dynamic encoding.

It is widely accepted that teleoperation plays a critical role in deploying large-scale autonomous fleets. It is also widely acknowledged that remote operation or supervision is not a trivial task. In this article, we provide highlights of an in-depth study on the human interface challenges of teleoperation: “Driving from a Distance: Challenges and Guidelines for Autonomous Vehicle Teleoperation Interfaces.” The research, recently presented at the CHI22 Conference on Human-Computer Interaction, was conducted by Felix Tener and Joel Lanir from the University of Haifa and was backed by the Andromeda Consortium, proudly led by DriveU.auto.

The Six Challenges of AV Teleoperation Interfaces

The authors relied on real-life observation and in-depth interviews with remote operators to arrive at conclusions. Research like this holds enormous practical value for AV developers, companies, and policymakers. The article will highlight the paper’s key points and guidelines. They have identified the following main challenges:

Figure 1: Categories of teleoperation challenges. The numbers indicate how many times themes in this category appeared in the data. Source: Driving from a Distance

Lack of physical sensing

In a remote driving station, an operator receives mostly visual feedback. Much physical feedback that assists in-vehicle drivers—bodily sensing of linear and angular accelerations, speed, road inclinations, etc.—is lacking. This is particularly important for vehicle maneuvering during specific actions such as making turns. One tele-driving study participant said: “The issue of turns is very difficult. I didn’t see a [remote] driver who didn’t ask his passengers ‘How was the turn?’ [after completing one].” The lack of physical connection extends to the controls: lack of feedback from the steering wheel or pedals makes it difficult for remote drivers to gauge just how far they turned the wheel or how hard they hit the brakes. The lack of physical feedback may also lead to negative physical effects on the drivers themselves. Some people report dizziness or nausea after a remote driving session.

Human cognition and perception

This is a complex topic comprised of several cognition and perceptions factors related to how humans create an understanding of their environment, and how to act in it. In-vehicle drivers are immersed in the vehicle’s environment, and become effortlessly aware of surrounding factors. Remote operators must actively concentrate to obtain and understand information from the driving interface, which can be exhausting. Factors in successfully controlling a remote vehicle include situational awareness, cognitive load, depth perception, spatial awareness, and mental models.

Video and communication quality

In teleoperation, the lack of or small amount of audio and haptic (sensory) data means remote operators rely almost entirely on visual information provided by video. Video quality is the most obvious issue, where even small reductions in video quality can create problems. Changes in frames per second (FPS), resolution, and latency can all affect the remote driver’s ability to correctly process information. For example, five out of eight study participants said increased latency caused over- and under-steering and even impacted the use of gas and brake pedals. One participant said: “If the latency is below 200-250ms, one can do teleoperation, but more than that, it’s impossible… [Also,] if it changes significantly all the time, it is very tiring for a remote operator (RO) [to the point of dizziness].” Other video-related issues include how videos from different sources are combined into one coherent picture, issues with field of view, and depth of vision.

Field of view due to camera angle — Image 1: camera angle and image sharpness. Source: Driving from a Distance

Interaction with humans

Autonomous vehicles are designed to serve the humans that either ride in them or receive a service. They operate in environments populated by human drivers, pedestrians, law enforcement agents, and more. There will inevitably be scenarios where interaction with humans is needed, namely emergency situations such as a passenger not feeling well, handling complex driving situations such as a four-way stop; or, it could be something as simple as entering a closed facility, or attempting a pizza delivery to a confusing address.

Impaired visibility

As noted, the already important role of visual representation in driving is heightened in remote driving. An RO’s view of the environment comes via the AV’s installed cameras. Distortions to the visual field can impair the operator’s ability to see and react to things. For example, compressing a 180-degree field of view from the cameras to a narrow monitor may slow down response times. Peripheral vision is also a challenge. One participant said that “Everything that peripheral vision gives me in a real vehicle, is lacking in remote operation. In driving, the effect of peripheral vision is very important.” Other challenges are lighting changes or the inability to improve the viewpoint. You have all probably leaned forwards and stretched your neck when something obstructs your field of vision. That is simply not possible when your “eyes” are a camera thousands of miles away.

Lack of sound

The lack of sounds from outside the car takes away valuable non-visual information. There are several types of sound: Sounds by other entities: Sirens, horns, voices, sounds that are a result of the vehicles’ interaction with the environment such as road surface, leaves on the road, wind; There’s also vehicle sounds, such as engine sounds; and lastly, weather-related sounds.

User interface design suggestions

The researchers provided the following suggestions on how to mitigate the above challenges:

Add UI cues to bridge the physical disconnect

Visual cues in the UI can indicate physical forces, like speed, acceleration, pressure, etc. For example, colored UI elements that change with force applied to pedals and to passengers inside the car can help ROs construct accurate mental models.

Cues in the interface — Image 2: cues to indicate the force passengers feel. Source: Driving from a Distance

Emphasize the intervention reason

A simple message in the UI, or virtual layers on the video feed indicating the reason for intervention can help ROs grasp the issue and remedy it as soon as possible.

Add contextual road information

In some of the teleoperation use cases, ROs assume driving responsibilities in the middle of a driving session. They need to be provided with critical information that is not immediately obvious—speed limit, off-limits detours, etc. Information from other road users or infrastructure (cooperative perception) and advanced alerts can improve situational awareness and performance.

Integrate AI suggestions in the UI

Sophisticated AI algorithms in AVs can assist intervening humans. Even when they can’t pinpoint the right action, their insights can reduce cognitive load and help remote drivers navigate tricky situations.

Visualize AVs direction based on the position of the steering wheel

Indicating the AV’s trajectory to the RO based on the steering wheel’s position aids situational awareness. However, UI elements representing the wheel increase cognitive load. It is preferable to instead project the trajectory onto the screen, as with curved turning lines on the road.

Add depth perception cues

Lack of depth perception can be remedied with special depth cues. For example, the UI may apply color to indicate distance, a technique used in medical visualizations. A car’s color or saturation, for instance, may change as it approaches or moves away from the AV.

Calibrate all video cameras and stitch images from overlapping streams

Calibrated AV cameras can prevent confusion related to distance, etc. Additionally, image stitching, despite being an expensive, complex process, provides a clear and accurate image of the road and reduces cognitive demand.

Visualize network and video quality

Visual cues for network quality, latency, and frame rate in the UI itself give ROs a heads-up on technical issues and boost situational awareness.

Research highlights challenges and opportunities for teleoperation in wide-scale AV adoption

Transportation is undergoing major transformation, and new driverless models are emerging to support the safe and efficient transportation of people and goods. Teleoperation plays an important role in driverless use cases, whether in full remote operation models, in taking over on-demand in edge cases, or by providing information and high-level commands. All of these cases share some common challenges that need to be overcome. This research, supported by DriveU.auto and the Andromeda Consortium sheds light on the key challenges and provides initial suggestions on how to overcome them. This work lays a primary foundation for future research and development by designers and engineers.

Get superior connectivity for autonomous vehicle teleoperation

Learn More

Accelerate your autonomous vehicle deployment

Schedule a meeting

Sign up for news updates

By clicking subscribe I agree to the DriveU.auto Terms and conditions and Privacy policy, and to receive emails from DriveU.auto. I understand that I may opt out of riveU.auto subscriptions at any time.