Teleoperation Interfaces: Challenges and guidelines for Autonomous Vehicles Research Summary
Written by: DriveU.auto
About DriveU.auto
DriveU.auto helps accelerate the deployment of autonomous vehicles with teleoperations. Our connectivity platform provides 4k video, audio, data and control commands with very low latency and ultra-high reliability, using cellular bonding and dynamic encoding.
The authors relied on real-life observation and in-depth interviews with remote operators to arrive at conclusions. Research like this holds enormous practical value for AV developers, companies, and policymakers. The article will highlight the paper’s key points and guidelines. They have identified the following main challenges: Figure 1: Categories of teleoperation challenges. The numbers indicate how many times themes in this category appeared in the data. Source: Driving from a Distance
Lack of physical sensing
In a remote driving station, an operator receives mostly visual feedback. Much physical feedback that assists in-vehicle drivers—bodily sensing of linear and angular accelerations, speed, road inclinations, etc.—is lacking. This is particularly important for vehicle maneuvering during specific actions such as making turns. One tele-driving study participant said: “The issue of turns is very difficult. I didn’t see a [remote] driver who didn’t ask his passengers ‘How was the turn?’ [after completing one].”The lack of physical connection extends to the controls: lack of feedback from the steering wheel or pedals makes it difficult for remote drivers to gauge just how far they turned the wheel or how hard they hit the brakes. The lack of physical feedback may also lead to negative physical effects on the drivers themselves. Some people report dizziness or nausea after a remote driving session.
Human cognition and perception
This is a complex topic comprised of several cognition and perceptions factors related to how humans create an understanding of their environment, and how to act in it. In-vehicle drivers are immersed in the vehicle’s environment, and become effortlessly aware of surrounding factors. Remote operators must actively concentrate to obtain and understand information from the driving interface, which can be exhausting. Factors in successfully controlling a remote vehicle include situational awareness, cognitive load, depth perception, spatial awareness, and mental models.
Video and communication quality
In teleoperation, the lack of or small amount of audio and haptic (sensory) data means remote operators rely almost entirely on visual information provided by video. Video quality is the most obvious issue, where even small reductions in video quality can create problems. Changes in frames per second (FPS), resolution, and latency can all affect the remote driver’s ability to correctly process information. For example, five out of eight study participants said increased latency caused over- and under-steering and even impacted the use of gas and brake pedals. One participant said: “If the latency is below 200-250ms, one can do teleoperation, but more than that, it’s impossible… [Also,] if it changes significantly all the time, it is very tiring for a remote operator (RO) [to the point of dizziness].”Other video-related issues include how videos from different sources are combined into one coherent picture, issues with field of view, and depth of vision.Image 1: camera angle and image sharpness. Source: Driving from a Distance
Interaction with humans
Autonomous vehicles are designed to serve the humans that either ride in them or receive a service. They operate in environments populated by human drivers, pedestrians, law enforcement agents, and more. There will inevitably be scenarios where interaction with humans is needed, namely emergency situations such as a passenger not feeling well, handling complex driving situations such as a four-way stop; or, it could be something as simple as entering a closed facility, or attempting a pizza delivery to a confusing address.
Impaired visibility
As noted, the already important role of visual representation in driving is heightened in remote driving. An RO’s view of the environment comes via the AV’s installed cameras. Distortions to the visual field can impair the operator’s ability to see and react to things. For example, compressing a 180-degree field of view from the cameras to a narrow monitor may slow down response times. Peripheral vision is also a challenge. One participant said that “Everything that peripheral vision gives me in a real vehicle, is lacking in remote operation. In driving, the effect of peripheral vision is very important.”Other challenges are lighting changes or the inability to improve the viewpoint. You have all probably leaned forwards and stretched your neck when something obstructs your field of vision. That is simply not possible when your “eyes” are a camera thousands of miles away.
Lack of sound
The lack of sounds from outside the car takes away valuable non-visual information. There are several types of sound:Sounds by other entities: Sirens, horns, voices, sounds that are a result of the vehicles’ interaction with the environment such as road surface, leaves on the road, wind; There’s also vehicle sounds, such as engine sounds; and lastly, weather-related sounds.
User interface design suggestions
The researchers provided the following suggestions on how to mitigate the above challenges:
Add UI cues to bridge the physical disconnect
Visual cues in the UI can indicate physical forces, like speed, acceleration, pressure, etc. For example, colored UI elements that change with force applied to pedals and to passengers inside the car can help ROs construct accurate mental models. Image 2: cues to indicate the force passengers feel. Source: Driving from a Distance
Emphasize the intervention reason
A simple message in the UI, or virtual layers on the video feed indicating the reason for intervention can help ROs grasp the issue and remedy it as soon as possible.
Add contextual road information
In some of the teleoperation use cases, ROs assume driving responsibilities in the middle of a driving session. They need to be provided with critical information that is not immediately obvious—speed limit, off-limits detours, etc. Information from other road users or infrastructure (cooperative perception) and advanced alerts can improve situational awareness and performance.
Integrate AI suggestions in the UI
Sophisticated AI algorithms in AVs can assist intervening humans. Even when they can’t pinpoint the right action, their insights can reduce cognitive load and help remote drivers navigate tricky situations.
Visualize AVs direction based on the position of the steering wheel
Indicating the AV’s trajectory to the RO based on the steering wheel’s position aids situational awareness. However, UI elements representing the wheel increase cognitive load. It is preferable to instead project the trajectory onto the screen, as with curved turning lines on the road.
Add depth perception cues
Lack of depth perception can be remedied with special depth cues. For example, the UI may apply color to indicate distance, a technique used in medical visualizations. A car’s color or saturation, for instance, may change as it approaches or moves away from the AV.
Calibrate all video cameras and stitch images from overlapping streams
Calibrated AV cameras can prevent confusion related to distance, etc. Additionally, image stitching, despite being an expensive, complex process, provides a clear and accurate image of the road and reduces cognitive demand.
Visualize network and video quality
Visual cues for network quality, latency, and frame rate in the UI itself give ROs a heads-up on technical issues and boost situational awareness.
Research highlights challenges and opportunities for teleoperation in wide-scale AV adoption
Transportation is undergoing major transformation, and new driverless models are emerging to support the safe and efficient transportation of people and goods. Teleoperation plays an important role in driverless use cases, whether in full remote operation models, in taking over on-demand in edge cases, or by providing information and high-level commands.All of these cases share some common challenges that need to be overcome. This research, supported by DriveU.auto and the Andromeda Consortium sheds light on the key challenges and provides initial suggestions on how to overcome them. This work lays a primary foundation for future research and development by designers and engineers.
By clicking subscribe I agree to the DriveU.auto Terms and conditions and Privacy policy, and to receive emails from DriveU.auto. I understand that I may opt out of riveU.auto subscriptions at any time.
From AGVs to Autonomous Terminal Trucks – what can container terminal operators learn from intralogistics? Container terminal operators face significant challenges, with growing freight volumes,
In this installment in our series about how Autonomous Vehicles (AVs) are affecting whole industries, we look at agriculture to find out that AVs drive
The global transportation and logistics industry is facing a multitude of interconnected challenges that lead to rising volatility and costs across the supply chain. One
The initial hype surrounding the self-driving car industry has matured significantly since its inception. While the vision of autonomous vehicles (AVs) seamlessly navigating city streets
Teleoperation for autonomous and driverless vehicles requires reliable, low-latency connectivity. With the growing traction of 5G networks, we dive deeper into the technology to analyze
With autonomous vehicle deployment growing around the globe, government officials are becoming increasingly confident in self-driving cars. This is evidenced by the recent spate of
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
