What’s teleoperation got to do with it? The complete guide to autonomous vehicle teleoperation

Remote truck driving. Teleoperation.

Try and think for a moment – were those terms you were familiar with a year ago?

How about now?

Headline after headline keep announcing implementations and use cases involving one form of teleoperation or another. Einride trucks, Voyage shuttles and Postmates delivery robots are just a few examples.

Why the sudden trend? What exactly does teleoperation have to do with autonomous vehicles? Why is this a good sign for the Autonomous Vehicle domain?

Much ink has been spent over the past year lamenting the fact that the autonomous car revolution is not happening as quickly as some had predicted. Yet at the same time, news of extensive trials, initial commercial deployments, and new use cases keep pouring in. Teleoperation and remote driving have a lot to do with it.

One of the signs of the maturity of the autonomous vehicle industry is that players are embracing the fact that full autonomy will be achieved through a myriad of narrowly defined use-cases rather than a zero-to-one global deployment. In addition, the industry has accepted that beyond the AI driving the vehicles, there is a host of underlying and complementary technologies that are essential for making widespread, commercially viable autonomous driving a reality.

What is Autonomous Vehicle Teleoperation?

Teleoperation is a technical term that describes the operation of machines, systems or robots from afar. It  has been around for quite some time, and is used across a variety of different industries. Simply put, it gives humans (or computers) the ability to control (steer, accelerate, break) a vehicle remotely, i.e., from outside the vehicle. To perform this task, teleoperation relies on sensors, high quality cameras, audio, cellular connectivity and advanced command & control centers.

Teleoperation can be broken down into a few implementation options and categories, including how the vehicle is controlled, the extent of control, the level of control, the use case, and more.

Teleoperation uses

Here are the most common ways that teleoperation is used in the autonomous vehicle industry:

• Remote driving. From autonomous mining trucks developed by the likes of Caterpillar, to last mile delivery robots and self-driving trucks that handle last mile deliveries from pickup to handover, teleoperation is already on the front lines of the autonomous revolution.

• Autonomous vehicle (AV) fallback mechanism. One of the major obstacles delaying the widespread launch of AV is that the decision-making AI engines need to be ‘trained’ on how to respond to countless scenarios and edge cases. Covering every possible real world scenario is practically impossible. To bridge this gap, teleoperators are used to guide AVs when they encounter new scenarios or unsolvable situations, so that the AVs can safely complete their missions.
• Safety mechanism during AV trials. Whether mandated by law or self-imposed, the overwhelming majority of AV trials are conducted with a safety driver along for the ride (either remotely or physically). Safety drivers are there as backup to the AI engine, ready to intervene as needed.

Just like sitting in the driver’s seat, but remotely? Think again!

When people think of teleoperation, they envision a driver sitting in a driving station with a steering wheel, and driving the vehicle as if they were inside it. While that may be true in some cases, that is not the case for the majority of use-cases.

Teleoperation can be conducted using three distinct mechanisms — using direct control (remote driving), providing commands or providing additional information.

The distinction is who makes the decisions, and who executes them – the teleoperator or the vehicle intelligence.

Direct control, AKA remote driving, is a term that describes instances when full control of the AV is given to the human teleoperator. Using advanced remote control technology, the teleoperator performs all aspects of driving including steering, acceleration and braking. In essence, in the direct control scenario the teleoperator is the one making all the decisions and executing them from afar.

Using high-level and low-level commands, human teleoperators are in charge of the decision making and direct the AV to execute them.

• With high-level commands (which are more relevant to the AV fallback use case), remote teleoperators give direct orders to the AV that regulate its behavior. These orders can include directives like changing lanes, continue driving straight or slowing down to 20MPH, among many others. However, these commands are just that: the AV decides for itself how it will go about performing the actual operations that facilitate the commands.
• Low-level commands are much more direct. They deliver explicit instructions regarding how to facilitate the operations, for example: steer 10 degrees left and accelerate speed by 10%.

Providing the vehicle with additional information may be considered a type of command, but in fact leaves some of the decision making and the execution to the vehicle. In this scenario, the AI asks the human teleoperator how it should react to specific situations it does not know how to deal with — for example, whether it’s okay to overtake a stalled moving truck that is blocking a lane.

As you probably gathered, different modes of teleoperations are suited for different uses and different vehicle types.

Technological considerations in teleoperations

After covering how teleoperation works, different uses and modes of operation, let’s address a few important technical aspects of teleoperations. The three most important elements for safe teleoperation are connectivity, connectivity and, you guessed it, connectivity.

Connectivity is the linchpin that holds the entire AV teleoperation together. The presentation of 4k video, audio and sensor data (like speed) to the remote driver in a timely manner relies on high speed, reliable, connectivity. Without connectivity, external signals cannot be received. Uninterrupted, high capacity, ultra-low latency connectivity is a basic requirement for teleoperation.

Uninterrupted, low latency connectivity is especially critical when it comes to video quality, which teleoperators need at high levels to ensure effective operations. If connectivity isn’t stable, teleoperators would have to wait for the video to render, which is simply unacceptable. Depending on the encoding mechanism, high quality video requires at the very least 4-10 megabits per second to function properly. The highly unstable, dynamic nature of cellular networks makes achieving this level of performance from a moving vehicle  on a consistent basis quite a challenge.

System architecture is another technical aspect of integrating teleoperation hardware and software into a vehicle, and it is no small feat. Audio, visual, in-vehicle and motion data need to be captured from the vehicle’s sensors, cameras and systems. Control interfaces to handle steering, throttling and braking must be implemented. Advanced system architecture to orchestrate the interactions between hardware and software and ensure resource utilization and system response times should be developed and integrated — all while keeping the addition of new hardware at a minimum to deliver lower SWAP (space, weight and power) that is key to ensuring reduced cost and integration time.

The remote driver environment can be just as challenging. Empowering safe, effective teleoperation requires extremely low levels of latency to ensure real time situational awareness. Our experience at DriveU.auto has shown us that surprisingly, less is more — only the relevant, critical information should be presented. Remote teleoperators can be overwhelmed in a deluge of unnecessary data that can sometimes do more harm than good — the rule of thumb is that if it’s not mandatory, don’t display it. When it comes to the amount of data presented to the remote teleoperator, a case can definitely be made against ‘too much information’.

Teleoperation is a critical component of widespread AV deployment

It is now widely acknowledged by industry leaders and government entities such as NIST (US National Institute of Standards and Technology) that AV teleoperations is needed to support widespread AV adoption. Its use augments a plethora of use cases and serves to solve different issues along the evolution of autonomous driving.

Connectivity is a critical element in safe teleoperation, yet one network is not enough to deliver the level of connectivity required. More on that in our next blog.