November | 2020
Introduction to an AVPG
An Autonomous Vehicle Proving Ground is a test track for self-driving cars[1]. Since this is meant to test the automotive engineering bit as well as the brain or the AI software in the car, an AVPG has additional elements on top of an ordinary test track. These elements are meant to comprehensively test the decision-making capabilities of an autonomous vehicle. These tracks are custom built with elements like parking lots, traffic management systems at intersections, cloverleaf intersections, 8-lane roads, etc. The intent behind having these traffic entities or the mobility infrastructure is to expose the AVs to as many commonly encountered and not-so-common edge-case driving situations as possible.
Figure 1: Plan of American Centre for Mobility, Ann Arbor, MI, USA[2]
Figure 1 depicts the plan of the American Centre for Mobility (ACM) in Ann Arbor, Michigan, USA. ACM has urban and rural roads (different kinds of material, lane markings, and road conditions like dirt roads or unpaved roads), user-defined area (can be customized to create any scenario), parking lots, bike and pedestrian corridors, a highway stretch, tunnels, a 6-lane intersection, a 6-lane boulevard, an urban canyon, and buildings with private spaces meant to be rented out for AV development/testing work.
In the world of autonomous vehicles, the AI software stack, the underlying processing hardware and the sensors together are responsible for its navigation and other autonomous decision-making. This AI software is undergoing continuous development and each updated version is being tested. AVPG fits in the right tier between simulation testing and limited and public road testing. The combination of Simulation testing and closed track testing (in AVPGs, for example) is referred to as Offline testing[3]. While most companies in their early years cannot indulge in public testing (for cost as well as regulatory reasons), AVPGs provide them the right platform for zero-risk, real-world testing inside closed doors.
There are newer versions of the hardware and software available that need to be tested either as a result of an optimization or an upgrade owing to a design fix in the previous version. All these versions need to be tested for first, integration stability, and second, because the updated versions may have improved AI modules as compared to any previous versions. AVPGs thus provide the right indoor environment for progressive testing.
New Car Assessment Programme (NCAP) is a regulation that governs the crash worthiness of a car. There are over 10 NCAP(s) worldwide. An organization called Global NCAP is bringing together all these NCAP bodies under one umbrella. ANCAP is aligning on test and assessment protocols with Euro NCAP. NCAP is more like a Safety rating authority; the more the rating, the safer the cars, though NCAP is not a legal binding by itself.[4] Amongst the 10 odd NCAP authorities across the world, Euro NCAP is the most exhaustive for active safety.[5] ISO26262 Functional Safety is defined for Automotive in general and requires additions for a fully autonomous vehicle. UL 4600 is the first standard for safety in Autonomous vehicles.[6]
Having said that, as far as standardization of AV testing is concerned, over and above the Automotive testing methodology, the focus area is always safety. However, there is no standard test suite that is followed religiously for AV testing. This essentially means more the number of public road miles, more the number of edge-case scenarios that the vehicle is exposed to, and better the accuracy of decision-making. However, the proportion of scenarios in which an AV is definitely going to take the right decision hasn’t been documented by anyone yet.
The only closest standardization of edge-case scenarios is disengagement scenarios[7]. These are basically where the safety driver has had to take manual control of the vehicle. These scenarios minus the ones where an unintended change in the operating conditions happens are the edge-case scenarios. US NHTSA describes terms like ODD, OEDR, and ConOps to detail out the operating conditions and possible responses from AV under a plethora of scenarios.
The point here being that testing in AVPGs should cover these edge-case scenarios. But, the very first time these disengagement scenarios are encountered is during public road testing because that is what truly creates unanticipated situations. If simulators provide a rich virtual environment, i.e. a digital replica of specific locations in actual cities that can model complex traffic, then these edge-case scenarios can be used to create multiple synthetic scenarios with some tweaking. Disengagement scenarios are indispensable for making any scenario library exhaustive from a safety perspective. However, disengagement scenarios without public road testing can be obtained only through proprietary data sources.
The situation is depicted in figures 2 and 3 below.
AVPG contribution to AV safety standards
One approach to ensure the safety of an autonomous vehicle is to ensure that the hardware platform, sensors, compute engines, and software have all individually gone through safety analysis post integration as well. Vehicle testing, integration testing, and system and sub-system level testing are conducted to ensure safety. Principles like DFA (Dependent Failure Analysis) to find any common failure causes or cascaded failures apply to AVs as well, just as they apply in automotive safety compliance (ISO26262), Safety of the Intended Functionality (SOTIF). Guaranteeing completeness of the safety analysis of the autonomous vehicle system is not easy. Till date, any safety analysis has been measured on the basis of disengagement situations per thousand miles, and this number hasn’t hit zero, though it is on the decline.
Software upgrades or updated hardware platform/units:
Each flaw in the decision taken by an autonomous vehicle is followed by a system software or hardware upgrade or both. The testing then has to be repeated. First, in a simulated environment to repeat the major chunk of test cases. Then, HIL tests can be run with recorded data, and after there is some confidence, this upgraded unit is again tested on public roads.
Every software update or new hardware installation should be followed by identification of the impacted subsystems, and they should be rigorously tested again. Even with respect to the much talked about OTA, there are concerns around the overall stability of the system post an upgrade, let alone the possibility of a major safety fiasco.
The continuous development of the AV towards flawless decision-making involves multiple iterative changes to the blocks within the vehicle technology architecture. This has a couple of major implications. First, as stated earlier, the need for a scalable and low-cost testing solution and second, reassessment of the safety aspects with each iteration.
Figure 4 below shows the steps involved in testing, at the vehicle level:
Figure 4: AV level, complete system level testing for safety analysis
AVPG(s) can serve as testbeds for shared mobility:
Presently, when smart mobility and multi-modal transport is a pressing need, AVPG(s) possibly have roles to play beyond just testing and verifying AVs. They can play the role of a testing ground for emulating smart modes of transport in a modern city like infrastructure. And, the presence of these infrastructural entities will make the testing ground conducive for autonomous shuttle companies to find the right kind of scenarios.
AVPG contribution to norms around interaction with Vulnerable Road Users (VRU(s)) like pedestrians, bicyclists:
AVPG is a Global phenomenon:
AVPGs are spread across the world. Their locations align closely with the locations of the centres of autonomous technology development worldwide, as can be observed in figure 5.
Figure 5: Autonomous Vehicle Proving Ground locations across the globe
AVPG technology readiness:
AVPG(s) are testing grounds for autonomous vehicles, connected vehicles, electric vehicles, and basically all new modes of transport. They are more than just an automotive test track. Consequently, technology-wise, they need to provide the necessary backend infrastructure to autonomous vehicle companies for seamless, efficient, and exhaustive testing. Following are the necessary technology elements expected in and around an AVPG:
Possible consulting services to AVPG:
Other than enabling 5G network and helping introduce the right geography-specific infrastructure complexities, the two major services that Wipro can offer to AVPG(s) are as follows:
Scenario matrix or library to an AVPG: Creates the superset of scenarios by using elements of the ODD for which the ADS is intended. This scenario library is the input to the Simulator. The scenario library will actually be a combination of logged and synthetic test cases, as stated earlier.
Simulator with Digital twin of the AVPG: For SIL testing of AI algorithms and finding tricky scenarios, a good approach is to run all possible scenarios in the digital twin of the AVPG in a Simulator. Further, this simulator will throw up situations in which the response of the AV was not ideal. These can be tested physically in the AVPG.
AVPG– challenges
AVPGs occupy fairly large areas, implying that they are a heavy investment. Most AVPGs today are co-owned by government authorities related to the transportation sector or automotive & private bodies. AVPGs are generally located in the outskirts of cities, not close to any public roads. A specific case is that of ACM in which the highway stretch passes from above the US 12 highway. The geography in which AVPG is located is also crucial, since a good mix of ambient light conditions is needed for Autonomous vehicles (AV) testing. Also, AVPG needs to expose AV to all kinds of weather conditions, so the geography plays a key role. AVPGs need to have the necessary IT and technology infrastructure needed for a company to establish a temporary or even permanent lab for development and testing of its AV technology. This also means that the labs need to be of the scale needed to fit even large commercial tractor-trailer combinations through their doors.
Conclusion
Irrespective of the time horizons associated with the mass deployment of autonomous Level 5 systems, testing is a continuous process and all major players in this space are continuing to invest toward improvement of the technology. The investments have taken newer forms like creating consortiums to pool in efforts. Even though continuous testing is indispensable, clearly, there are multiple bottlenecks with public road testing. It is neither very scalable nor affordable and not even easy to kick-start owing to the self-safety assessment report submission guideline prescribed by the NHTSA in the US. An optimal combination of public road testing with Simulator testing and AVPG testing (offline modes of testing) is needed. This point is better illustrated by Waymo, where for every mile of public road testing, 1,000 miles are covered in the Simulator environment and in between the two is the AVPG testing. [10]
References:
Balaji Sunil Kumar
Sunil currently leads the AI Algo Stack team for Wipro’s Autonomous Systems and Robotics practice. He is involved in developing Algorithms stack related to Automated Guided Vehicle systems. Sunil is also a Senior Member of Wipro’s DMTS community and has around 22 years of experience working primarily with Embedded Systems across a variety of industries. He can be reached at balaji.kumar@wipro.com.
Garima Jain
Garima is a Global Business Manager in Wipro’s Global 100 leadership program, working across functions and business units in India and the US, with rotations in pre-sales, sales, delivery, and domain consulting. Prior to completing her MBA from the Indian Institute of Management in Bangalore, she worked as an engineer on automotive core chip design for three years. Garima is passionate about robotics and all things related to autonomous and connected vehicles. She can be reached at garima.jain5@wipro.com.
Locations
