Teleoperation: accelerating autonomous vehicle deployment

Autonomous Vehicles – are we there yet?

According to the US Department of Transportation, autonomous vehicles could potentially eliminate up to 94% of traffic fatalities. Research conducted at the University of Ohio estimates that a 60% reduction in harmful emissions can be achieved with autonomous vehicles. KPMG estimates that AVs could reduce travel time by 40%, saving US workers 80 billion hours of commute time and $1.3 trillion dollars a year. AVs are a fundamental component of a paradigm shift the automotive industry is undergoing – the move towards Connected Autonomous, Shared and Electric (CASE, or ACES).

There seems to be little doubt that Autonomous Vehicles, deployed within a wide range of use cases – including robotaxis, trucks, delivery robots, agriculture, yard and port automation, to name a few – will become a reality in the coming years.

But it’s hard work. As in, extremely difficult.

Building an autonomous vehicle is very capital intensive. The Information estimated that a total of $16 Billion was invested by the top companies engaged in this space over recent years. And yet, even with so much hype and cash flowing into autonomous vehicles, experts say it could take at least a few more years before robotaxis becomes a common site and a decade before private consumers can get their very own self-driving car.

AI is at the core of AV

Autonomous driving technology is usually broken down into four main components: Sensing, Sensor Fusion/Perception, Path/Trajectory Planning, and Drive Control. The most challenging part is perception, where the vehicle needs to ‘understand’ the situation and predict several potential future scenarios, including an evaluation of most likely and most dangerous.

Perception is based on Artificial Intelligence (AI). Using machine learning and neural networks, the AV software stack learns how to respond to scenarios and edge cases in a manner similar to humans. The operative word in that sentence is “learns”: The AI models need to be trained and “taught”. A key challenge in training AVs is the seemingly endless number of different scenarios and edge cases to contend with. For example, how should the AV respond in a scenario where soapy water has spilled on the road? Can it differentiate between a flock of birds crossing the street (brake) and a bunch of leaves flying in the wind (ignore)?

It is impossible to teach an AV how to deal with every single possible case – in the real world or through simulation. The AI is instilled with decision making capabilities that can “learn” from other relevant cases, so as to make the correct decision when encountering a new event it has not been trained for.

If the AV’s AI is properly trained over a large number of scenarios and combinations of edge cases, and has robust machine learning capabilities, it will be able to take the right action under any circumstance, even if it had not encountered it in the past.

But with practically infinite potential scenarios to contend with and dozens of AI engine models to test, scaling up real-world training and testing is simply not feasible; even with extensive use of computer-based simulation there is always the risk of a novel scenario. In addition, there is always the risk of encountering a situation that’s outside of the AV’s Operational Design Domain (ODD).

All we need is (a little) human intervention

Autonomous vehicles have made great leaps forward. Examples of hours of uninterrupted AV driving have been released by Mobileye, Voyage, Waymo, Cruise and others. Yet there are, and probably always will be, cases where the AI does not ‘know’ what to do, or its level of confidence in selecting the appropriate action is too low – such as an unprotected left turn – instances in which the AV requires human assistance for mission accomplishment.

In light of the above, and as has been widely acknowledged by industry leaders and government entities such as NIST (US National Institute of Standards and Technology), AV teleoperations is needed. Teleoperation is the technical term for the operation of an unmanned machine, system, or robot from a distance. Teleoperation is needed in order to ensure AVs are able to complete their mission under any circumstances. Some even see teleoperations as the stop-gap between driven and driverless vehicles.

In teleoperations, remote operators residing in a command & control center are ready to be called on by an AV in need, prepared to guide the AV on how to overcome a specific obstacle, or take over the entire driving operation until the challenge is dealt with.

There are two main ways teleoperators can assist an AV in mission accomplishment:

• Remote Control or Remote Driving: full control of the vehicle is handed over to the teleoperator, who performs all the different aspects of driving: decision-making, steering, acceleration, and braking. They are driving the vehicle, only remotely.
• High-Level Commands: in this mode, teleoperators don’t actually drive the vehicle; rather, they assess the situation that prevents mission accomplishment and assist the AV by providing instructions, or approving or correcting the vehicle’s suggested path. The remote operator does not drive, but rather provides additional input that allows the AI to complete its mission.

How can teleoperation speed up AV deployment?

DriveU.auto sees the following framework as governing the implementation of teleoperations as part of Autonomous Vehicle deployments:

Teleoperation during AV testing and training

During testing and training, human oversight is mandatory. This is both required by regulation and self-imposed by AV developers. Human oversight can be done either by a safety driver in the vehicle, or, as is now the case for Cruise in California, outside of the vehicle, in a command and control center.

Teleoperation after commercial AV deployment

As the market moves to commercial deployments, AV developers will ensure that the AI is safe and the vehicle can always execute a safe Minimal Risk Maneuver (MRM). This means that if a time-critical safety issue occurs, such as a child running into the street, the AI will always be able to react in the safest manner.

In situations where the AI is not able to accomplish the AV’s mission because of encountering a circumstance beyond its ability, the AI will execute the MRM and hand off control to a remote human operator. Once the AI is able to resume safe operations, the remote human driver will hand off control back to the AI.

Teleoperation can also be used when the vehicle needs to operate outside the Operational Design Domain (ODD). Such cases can be outside of a specified geographical area, under certain weather or visibility conditions, sensor fault, and more. In these cases, a remote human operator enables the AV’s operation outside of the AV’s ODD, through remote driving.

An additional teleoperation benefit is that in all cases, the actions or instructions of the remote human assistant can be used as additional training data for the AI, and further improve it.

Parting words

The large-scale deployment of fully autonomous vehicles is sure to benefit society on many significant fronts, and there has been significant progress toward that vision. Yet there is still much ground to cover on the route to full, unconstrained autonomy and mass adoption. Our ability to train and test AI’s capability to correctly react to every possible scenario and edge case is a major obstacle. Teleoperation can speed up AV deployment by providing a trustworthy fallback mechanism both during testing and after commercial launch.