Overview
Sound plays a crucial role in shaping immersive experiences, yet its impact on outdoor AR tourism remains under explored. Augmented Reality (AR) has been widely adopted in tourism to enhance visitor experiences by overlaying digital information onto the physical world.
My Master’s Thesis explores how auditory elements influence outdoor AR tourism experiences, aiming to uncover the role of sound in shaping user engagement, immersion, and information retention.
Why Auditory Modality?
If we look at mobile AR in the industry, most AR applications are dominated by visual modality. Examples include Google Map AR Navigation, Pokémon, and even the latest Apple Vision Pro.
However, human interactions are inherently multimodal, involving multiple senses beyond just vision. AR experiences can be multisensory as well.
Research Question
The driving question behind this research was how auditory augmentation shapes user experiences in outdoor AR tourism. Rather than looking at this broadly, I broke it down into three specific questions:
1. How does auditory modality influence user experience during navigation tasks?
2. What is its impact on user experience during exploration tasks?
3. How does auditory AR affect immersion in both navigation and exploration?
2. What is its impact on user experience during exploration tasks?
3. How does auditory AR affect immersion in both navigation and exploration?
To explore these aspects, I examined different types of auditory cues—such as ambient sounds, voice narrations, and spatial audio—to understand how they alter user perception and interaction with AR environments. By focusing on these three dimensions, the research aimed to uncover both the benefits and limitations of integrating sound into AR tourism applications.
Methodology
To explore the relationship between sound and AR tourism experiences, I adopted a structured approach that combined theoretical research with practical experimentation. I began with an extensive literature review, diving into existing studies on AR Tourism, Auditory AR, and AR User Experience.
Building on these insights, I designed three scenarios for the user study and that can help me to answer these three research questions:
Q1: What is the impact of auditory modality on user experience during navigation tasks?
To answer this question, two navigation conditions were designed:
Spatial Auditory Navigation
Visual Augmented Navigation
In the Spatial Auditory Navigation condition, people can hear a piece of music emitting from their destination, direction and distance can be judged through the characteristics of spatial sound. In the Visual Augmented Navigation condition, people can see the direction and distance to their destination right on the screen through augmented visual cues.
📈 Evaluation metrics: Qualitative (UEQ-S questionnaire) and Quantitative (semi-structured interview)
Q2: What is the impact of auditory modality on user experience during exploration tasks?
To answer this question, two types of AR objects at tourism sights were designed: Auditory AR Objects and Textual AR Objects.
Auditory AR Objects
Textual AR Objects
For Auditory AR objects, users can get tourism information through auditory media. For Textual AR objects, users can get tourism information through texts. Both types of AR objects are interactive, users can move them around and scale them up and down.
📈 Evaluation metrics: Quantitative (UEQ+ questionnaire) and Qualitative (semi-structured interview)
Q3: How does the auditory AR impact the factors of immersion during navigation and exploration tasks?
To answer this question, an additional condition was designed: users are guided through spatial auditory navigation, in the meantime they can set markers on their way to the destination in case of getting lost.
Set markers during navigation
📈 Evaluation metrics: Quantitative (ARI questionnaire) and Qualitative (semi-structured interview)
Augmented Reality Immersion (ARI) is an instrument for assessing immersion in location-aware AR applications
Prototyping
To be able to run the test in a real outdoor environment, I developed a prototype using Unity with an Android AR setup. The prototype incorporated various auditory elements into an AR environment, integrating ARCore for Android functionality and leveraging Unity AR Foundation to enable device tracking, camera functionality, and plane detection. Some key technologies were applied to achieve the requirements of desired user study:
Spatial Auditory Navigation
• Google ARCore Geospatial API + FMOD plugin
• Geospatial API: more accurately detects the user’s device location by GPS and Visual Positioning System (VPS)
• FMOD plugin: a sound object is attached to the location of the destination (GPS), and can be heard in the activation zone
3D Sound Working Mechanism
Augmented Visual Navigation
• Mapbox API: generates direction and distance to the destination based on GPS data
AR Objects Interaction
• Unity XR Interaction Toolkit: provides a framework that makes the interaction tasks like hovering, selecting, and grabbing AR objects available
User Study
For conducting the user study, the mobile App prototype was tailored to meet the specific tourism needs of Tiergarten, Berlin. Three tourist sites the Goethe Monument, Beethoven Memorial, and Amazone zu Pferde were selected as the areas of user study.
Location setting of the user study
I conducted user study in real-world outdoor environments that closely mirrored tourism settings:
• 📍 Tiergarten, Berlin onsite.
• 📍 Tiergarten, Berlin onsite.
• 📚 Within-subjects design and tasks-based assessment, allowing participants to partake in all conditions.
• 👥 A total of 20 participants were recruited to complete a standardized set of tasks, 45 mins for each.
• 📊 Quantitative data & qualitative data were collected.
The user study protocol is as follows:
Results
Finally, I performed a data analysis, combining qualitative insights with quantitative metrics to assess engagement levels, immersion, and information retention.
Demographics of 20 participants
• 9 males and 11 females
• Age between 18 and 54
• Participants are from Germany, China, Britain, Vietnam, Netherlands, Sweden, Azerbaijan
Spatial Auditory Navigation vs Augmented Visual Navigation
Quantitative results of Spatial Auditory Navigation and Augmented Visual Navigation
Higher Hedonic Quality value represents a more exciting, interesting, inventive and leading edge user experience, higher Pragmatic Quality value represents a more supportive, easy, efficient and clear user experience.
The results indicate that:
• Spatial Auditory Navigation has significantly higher Hedonic Quality
• Pragmatic and Overall are not significantly different between the two navigation modalities
AR Objects with Auditory Information and Textual Information
Quantitative results of Auditory Information and Textual Information AR Objects
AR objects with Auditory Information Display tended to show advantages in terms of Motivating, while AR objects with Textual Information Display showed advantages in terms of Intuitive Use. However, no significant differences were found between the two types of AR objects.
The Factors of Immersion
Quantitative results of the factors of immersion
The results showed that:
• The mean value of Engagement was significantly higher than that of Total Immersion
• The mean value of Engrossment was significantly higher than that of Total Immersion
• No statistically significant difference between Engagement and Engrossment
Qualitative Results from Interview
Here is a summary of qualitative results:
Conclusion
The study revealed several important insights about the role of auditory augmentation in outdoor AR tourism. In conclusion, the study shows that:
1. Spatial auditory modality enhances the Hedonic Quality of navigation tasks by offering a more exciting, interesting, and novel user experience.
2. AR objects with information displayed through auditory modality have the potential to attract users' interest and motivate interactivity, which can help to lead an enhanced user experience during exploration tasks.
3. Auditory AR design contributes significantly to the factors of Engagement and Engrossment, demonstrating potential for immersive user experience.
Outlook on Future Research
The study revealed the potential impact of combining multiple modalities and the influence factors on user experience. In the future, some research topics concerning auditory modality can be considered:
1. How would the combination of auditory and visual modalities perform during outdoor AR experience?
2. How much information level is needed during outdoor AR tourism exploration?
I hope by further exploring these research topics, the understanding of auditory modality’s role in outdoor AR tourism experiences can be expanded, leading to more informed and optimized design strategies in the future. AR is benefiting our life experience, but as a UX designer, I would like to focus on how to use the technology rather than AR itself. Hence, there is still a long course to go.
Thank you for reading:)