Figure 1: Plan of American Centre for Mobility, Ann Arbor, MI, USA
Figure 1 depicts the plan of the American Centre for Mobility (ACM) in Ann Arbor, Michigan, USA. ACM has urban and rural roads (different kinds of material, lane markings, and road conditions like dirt roads or unpaved roads), user-defined area (can be customized to create any scenario), parking lots, bike and pedestrian corridors, a highway stretch, tunnels, a 6-lane intersection, a 6-lane boulevard, an urban canyon, and buildings with private spaces meant to be rented out for AV development/testing work.
Why test in AVPGs?
In the world of autonomous vehicles, the AI software stack, the underlying processing hardware and the sensors together are responsible for its navigation and other autonomous decision-making. This AI software is undergoing continuous development and each updated version is being tested. AVPG fits in the right tier between simulation testing and limited and public road testing. The combination of Simulation testing and closed track testing (in AVPGs, for example) is referred to as Offline testing. While most companies in their early years cannot indulge in public testing (for cost as well as regulatory reasons), AVPGs provide them the right platform for zero-risk, real-world testing inside closed doors.
There are newer versions of the hardware and software available that need to be tested either as a result of an optimization or an upgrade owing to a design fix in the previous version. All these versions need to be tested for first, integration stability, and second, because the updated versions may have improved AI modules as compared to any previous versions. AVPGs thus provide the right indoor environment for progressive testing.
AV testing standardization
New Car Assessment Programme (NCAP) is a regulation that governs the crash worthiness of a car. There are over 10 NCAP(s) worldwide. An organization called Global NCAP is bringing together all these NCAP bodies under one umbrella. ANCAP is aligning on test and assessment protocols with Euro NCAP. NCAP is more like a Safety rating authority; the more the rating, the safer the cars, though NCAP is not a legal binding by itself. Amongst the 10 odd NCAP authorities across the world, Euro NCAP is the most exhaustive for active safety. ISO26262 Functional Safety is defined for Automotive in general and requires additions for a fully autonomous vehicle. UL 4600 is the first standard for safety in Autonomous vehicles.
Having said that, as far as standardization of AV testing is concerned, over and above the Automotive testing methodology, the focus area is always safety. However, there is no standard test suite that is followed religiously for AV testing. This essentially means more the number of public road miles, more the number of edge-case scenarios that the vehicle is exposed to, and better the accuracy of decision-making. However, the proportion of scenarios in which an AV is definitely going to take the right decision hasn’t been documented by anyone yet.
The only closest standardization of edge-case scenarios is disengagement scenarios. These are basically where the safety driver has had to take manual control of the vehicle. These scenarios minus the ones where an unintended change in the operating conditions happens are the edge-case scenarios. US NHTSA describes terms like ODD, OEDR, and ConOps to detail out the operating conditions and possible responses from AV under a plethora of scenarios.
The point here being that testing in AVPGs should cover these edge-case scenarios. But, the very first time these disengagement scenarios are encountered is during public road testing because that is what truly creates unanticipated situations. If simulators provide a rich virtual environment, i.e. a digital replica of specific locations in actual cities that can model complex traffic, then these edge-case scenarios can be used to create multiple synthetic scenarios with some tweaking. Disengagement scenarios are indispensable for making any scenario library exhaustive from a safety perspective. However, disengagement scenarios without public road testing can be obtained only through proprietary data sources.
The situation is depicted in figures 2 and 3 below.