Augmented reality (AR) is an interactive experience of a real-world environment where the objects that reside in the real world are enhanced by computer generated perceptual information.
With the help of advanced AR technologies (for example, adding computer vision, incorporating AR cameras into smartphone applications and object detection), the information about the surrounding real world of the user becomes interactive and digitally manipulated. With the advent of AR technology and improvements in AR research, enterprise customers are more interested in finding new ways to deploy AR to improve the efficiency of their technicians by providing AR guidance during live troubleshooting of the equipment. Implementation of AR is an emerging technology, and each AR implementation uncovers challenges that must be addressed to have smooth project implementation.
Among these challenges are data collection, annotation, model training, object detection, app performance in AR-based guidance solutions. The following sections enumerate some of the challenges.
Data Collection Challenges
Data collection requires close collaboration between the customer subject matter expert and the implementation data collection team. The visual information gathering for a particular scene is labor intensive and requires multiple runs.
Digital Assets - Many enterprises are not ready with the visual digital assets of the equipment, so photos and videos must be shot of the actual equipment for each object of interest.
Subject Matter Expert (SME) Availability - The customer SME is mandatory during digital asset capture. The SME is required to spend a good amount of time reproducing the troubleshooting steps while another person digitally captures the SME performing the steps.
Video Quality Digital - Assets are the base for AR, and a significant quantity of high-quality data is critical for the success of object detection of these assets. Many pictures/videos are captured from all possible angles, both zoom-in/zoom-out and with good resolution for each object of interest. This can be a time-consuming, tedious activity.
Annotation is a manual process where objects of interest are highlighted efficiently in the images such that the models created using them are accurate. Object detection is tightly associated with the number of images trained. This is an iterative process that continues until the model converges.
Model Training challenges
The efficacy of object detection by the trained model is difficult to predict in some cases. Model testing reveals weaknesses in the detection of certain objects. This requires the capture and training of additional images for the specific object of interest. This is an iterative process which continues until sufficient accuracy levels are reached. The detection is often constrained by tiny objects, similar objects, occlusion objects, wires, tubes, etc.
Object Detection Challenges
Object detection is a computer technology that deals with detecting instances of objects of a certain class (such as humans, buildings or cars) in digital images and videos through machine learning algorithms.
Tiny Objects - A machine-learning algorithm may not be able to detect tiny objects. In some cases, this can be addressed by enlarging the object of interest area.
Internal Parts - The machine-learning algorithm may not be able to detect internal parts due to low visibility and camera reachability. In such situations, AR detection is substituted with video to explain the particular process.
Object Occlusion - The machine-learning algorithm may not be able to differentiate the object of interest from occluding objects like wire, tubes, etc. The workaround for such issues may be to remove or sidestep the occlusion
Similar Objects - The machine-learning algorithm may get confused if more than one object has the same features.
Color Differentiation - The machine-learning algorithm may not be able to differentiate objects with the same features but different colors. This requires algorithm enhancement to distinguish the base colors.
App Performance Challenges
Size of Procedures - A large number of procedures take more memory in the smartphone app since it has a motion digital asset. On demand procedure availability works around such challenges.
Network Bandwidth - AR object detection relies on the central GPU for processing against the trained model. In such a scenario, a stable bandwidth is required for the app to perform smoothly. In such cases, an app with offline capability would help.
Customer Restrictions - In restricted or highly secure zones, the customer may not approve the use of camera phones. In such situations, the app can’t perform.
Infrastructure - Server location affects app performance due to latency.
As outlined in this article, data collection and annotation are time-consuming, repetitive tasks that impact project schedules and cost. Tools will help address some of these implementation challenges.
Video Recording Tool - While taking video, a guidance tool suggests the required angles and ensures the video is taken with consistency across objects. Helpful tools include:
- Video capture in the left, right, top, and bottom directions with at least 30 degrees of angle variation in each direction
- Zoom in and out
- Automatic change (increase or decrease) in-camera brightness based on the object visibility
- Warnings when the app identifies occlusion objects, or identical objects with the same or different colors
- Suggestions like minimum duration, video backup, etc.
- Auto-uploading of the recorded video to the AR system
Image Segregation Tool - This tool extracts images from videos for the given frames per second and removes unwanted blurry, shaky, or unclear images.
Image Augmentation Tool - This tool auto-generates additional images based on input parameters such as flip, rotation, blurriness, etc.
Annotation Automation Tool - This tool automatically annotates additional images based on the baselined images and balancing labels.
Auto Model Retraining Tool - For objects that are not detected properly, the entire process needs to be repeated; the tool auto-generates additional images, auto-annotates and auto initiates model training.
Integrated Development Environment (IDE) - An IDE that integrates the tools would help in building the application faster. Continuous Improvement During live usage, there can be cases where object detection may not happen properly. Apps capable of capturing nondeductible objects would help in adding more images to the learning repository.
On the technology front, AR is maturing with algorithms and tools. AR solutions are evolving as the learning from challenges redefines the workarounds for specific situations. This can be achieved by extending existing data, continuous improvements such as auto-balancing labels, auto-annotation and auto-training to improve object detection. As more research yields results in AR technologies, these outcomes will uncover better and more efficient ways to address some of the implementation challenges of AR. This will also improve the process of AR project implementation and more effective use cases.
For more information, email us at askIX.firstname.lastname@example.org