User tests for an aeronautical parts inspection mixed-reality software
- Jan 23, 2021
- 5 min read
Updated: Aug 19, 2024
Problem statement
Aciturri, one of Airbus' main suppliers of aeronautical parts, aims to streamline and enhance the reliability of its parts inspection process. Currently, aeronautical engineers perform this process manually, but there is a need to ensure its accuracy while also saving time. To address this, Aciturri sought to develop software that would facilitate inspection tasks.
Sopra Steria’s development team created a beta version of "Scoop," a software application that operates through mixed reality glasses (Microsoft Hololens). Scoop allows users to superimpose digital designs of aeronautical parts onto real aircraft components. The software not only catalogs all existing parts but also provides step-by-step instructions for their inspection.
The client now wants to test the software to identify any potential adverse effects from prolonged use of the mixed reality glasses and to assess any issues related to the user experience.
Project goals
Test and refine the user interface and experience of Scoop, making sure it is intuitive and efficient for aeronautical engineers.
Ensure that the new software improves the consistency and reliability of the inspection process compared to the current manual methods.
Assess the potential adverse effects of prolonged use of mixed reality glasses on the users, such as eye strain, fatigue, or other health-related issues.
Verify that the software correctly compiles all relevant aeronautical parts and provides accurate, detailed inspection instructions for each one.
Collect user feedback on the beta version of Scoop to identify areas for improvement and inform further development before a full-scale rollout.
Team structure
1 Product manager
1 UX Researcher
1 Developer
UXR Method:
User Testing
1 - Objectives
Assess the efficiency of the inspection process using Scoop compared to the manual method.
Evaluate the reliability and accuracy of inspections performed with Scoop.
Gather feedback on the user experience, including the software interface and ease of use.
Identify any adverse effects from prolonged use of mixed reality glasses.
Verify the completeness and accuracy of the part cataloging and inspection instructions.
2 - Participants
6 aeronautical engineers who regularly perform inspections.
Engineers with different levels of experience and familiarity with mixed reality technology.
3 - Process
Participants were given a pre test survey to collect baseline data on their experience with manual inspections and familiarity with mixed reality technology.
Each participant was assigned 3 inspection tasks using both the manual method and the Scoop software. Time taken was recorded and difficulties encountered registered. They were required to follow the inspection instructions provided by Scoop and evaluate the clarity and usefulness of the guidance on a scale from 1 to 5, being 1 extremely unclear/extremely useless and 5 extremely clear/useful.
All sessions were recorded to monitor afterwards their accuracy in part identification and inspection using the software compared to manual methods.
Participants were interviewed after session completion to gather qualitative feedback on the user experience, and their health and comfort were assessed by using a self-reported survey to capture data on eye strain, discomfort, or fatigue.
4 - Results:
Time Comparison (Manual vs. Scoop):
P1: Manual method was faster (Avg: 239 sec) than Scoop (Avg: 276 sec).
P2: Manual method was faster (Avg: 262 sec) compared to Scoop (Avg: 341 sec).
P3: Manual method was faster (Avg: 330 sec) compared to Scoop (Avg: 452 sec).
P4: Manual method was slower (Avg: 235 sec) than Scoop (Avg: 247 sec).
P5: Manual method was slightly faster (Avg: 298 sec) compared to Scoop (Avg: 300 sec).
P6: Scoop was slightly faster (Avg: 240 sec) compared to Manual (Avg: 251 sec).
Individual Task Performance:
Task 1: For all participants except P6, the Manual method was faster. P6 had almost the same performance for both methods.
Task 2: The Scoop method was generally faster for most participants, except P3.
Task 3: In most cases, the Manual method was slightly faster or had comparable performance with Scoop.
Average Time Trends:
In general, the Manual method tends to be faster for most participants across the tasks, except for P4 and P6, where Scoop showed slight improvements or similar times.
5 - Analysis
Manual vs. Scoop Efficiency:
Manual Method: On average, it seems more efficient across most participants and tasks. The consistent performance across different tasks suggests that participants might be more familiar with manual procedures.
Scoop Method: Though designed to augment the inspection process, Scoop appears to take longer to complete tasks for most participants. However, it is more efficient in specific tasks for some participants (e.g., Task 2 for P1, P4, P5, and P6).
Learning Curve/Tool Familiarity:
The slower performance using Scoop could indicate a learning curve or lack of familiarity with the tool. Participants might require more practice or training to use Scoop more efficiently.
Potential for Improvement:
The close time averages for P4 and P6 suggest that with further training or improvements in the Scoop software, it could become more competitive or even surpass the manual method in efficiency.
6 - Conclusion
The data suggests that the Manual method currently leads to quicker task completion times for most participants, indicating that Scoop might need further optimization or user training to enhance its effectiveness. This analysis highlights the importance of considering user experience and familiarity when introducing new technology into established workflows.
The first task showed very long execution times across all participants because Scoop consistently detected that the device was not correctly calibrated. This technical issue significantly impacted performance, leading to extended completion times.
As expected, the three participants who had previously used augmented reality glasses (P1, P5, and P6) demonstrated more noticeable time savings when using Scoop. Their familiarity with AR technology likely contributed to their ability to adapt more quickly and efficiently to the Scoop tool, resulting in better performance.
Some users found the ‘Create Space’ and ‘Load Space’ options in the start menu to be confusing, as they seemed similar in function.
Participants easily identified and activated the ‘Bounding Box’ feature. However, when the bounding box was enabled, its visibility was problematic due to the dimensions of the object. This required users to step back to properly manipulate the object.
Recommendation: A warning or explanatory note could help prevent this confusion.
“I thought the Bounding Box didn’t appear. Since you're practically inside the piece, the only way I could see it was to move a few steps away.” - Participant
Users encountered difficulties rotating or moving objects after reducing their size. This issue was not present when manipulating objects at their normal size.
Due to limited experience with the device, participants reported slight challenges with Hololens' interaction methods, including Air Taps, object manipulation, clicks, and the use of the holographic keyboard.
Recommendation: Users should gain sufficient experience with the device before attempting to perform everyday tasks.
Participants recognized the value of Scoop in supporting quality control, particularly during final checks, inspections of large parts, or lengthy inspection procedures.
The level of immersion was rated as low, partly due to the intensity of surrounding stimuli. The working environment’s characteristics should be considered if voice commands are to be enabled in the future.
“Mixed reality is not as immersive as virtual reality, so it’s difficult to isolate yourself from your reality, especially with all this external noise...” - Participant
In the initial interview, some participants mentioned experiencing mild myopia and/or eye fatigue after prolonged screen exposure in their daily work. These symptoms were infrequent and did not significantly impact their ability to perform work tasks.
After the user tests, no participant reported any physical discomfort, such as dizziness, visual issues, vestibular disturbances, or motor problems.
“It can be used for one-off tasks, but I don’t see us using it for a whole day.” - Participant
These findings highlight the potential for Scoop to be highly effective, particularly for users with prior experience in augmented reality. With targeted training and further software optimization, Scoop could become a more efficient tool for a broader range of users, ultimately improving task performance and reducing the need for manual processes.
7 - Suggestions
Implement anchored pop-ups for information display.
Introduce an option to save sessions for later use.
Provide a summary of inspections with the possibility of automated report generation.
Include options for generating various types of inspection reports.
Comments