Team
This project was a student team collaboration conducted at Technische Universität Berlin’s Quality and Usability Lab. The team consisted of three members: me as the design lead, another student managed project coordination, and a third member handled development. Our goal was to explore how AR could enhance performing arts engagement through immersive digital experiences.
Overview
Performing arts, including theater and live performances, often struggle to engage modern audiences who are accustomed to interactive digital experiences. This project explores how Augmented Reality (AR) can bridge the gap between traditional performing arts and digital interactivity, offering new ways for audiences to engage with performances before, during, and after a show.
Project Scope
This project aimed to design an AR application that enhances the performing arts experience by providing contextual, interactive, and immersive content. The focus was on exploring how digital overlays, real-time interactions, and extended storytelling through AR can enrich the audience’s connection with the performance.
Research & Insights
Research was conducted in the following areas to understand the potential of AR in performing arts:
• Audience expectations: Investigated how modern audiences engage with performances and what digital enhancements could improve their experience.
• Current AR applications in entertainment: Analyzed existing AR implementations in museums, exhibitions, and live events to identify best practices.
• Technical feasibility: Explored available AR frameworks and technologies that could be implemented in a performing arts setting.
Key insights revealed that audiences are looking for more interactive and personalized experiences. They also prefer content that complements rather than distracts from the performance.
User Scenarios
To illustrate how users interact with the AR application, two user scenarios were created that depict different engagement moments within the performing arts experience.
Scenario 1
Scenario 2
Scenario 1: Pre-Show Engagement
A user sits comfortably at home, using the AR application on their tablet to explore behind-the-scenes content before attending the performance. Scanning promotional materials provides exclusive insights into the play's background, the cast, and interactive previews of key scenes.
Scenario 2: Live Performance Interaction During the Show
The user holds up their tablet, where AR elements overlay onto the live performance, enriching the storytelling experience. Additional contextual information, character insights, or interactive elements appear seamlessly to enhance audience engagement without disrupting the immersion.
Design & Sketches
To support the idea, a mobile application was designed with the following core functionalities:
• Display of static and dynamic objects:
The application allowed for the integration of both fixed and animated AR elements to enhance the performance experience.
• Display of subtitles synchronously to spoken words:
A real-time subtitle feature was incorporated to provide accessible text overlays synchronized with dialogue.
• Interaction with virtual objects:
Users could interact with AR elements during the performance, engaging with story-relevant objects.
• Integration of story-relevant elements:
Contextually relevant digital overlays were added to enrich the narrative experience.
• Virtual extension of real objects in 3D space:
AR technology was used to enhance real-world props and stage elements, adding depth and interactivity to the performance.
Some sketches are shown below:
Prototype & Development
To bring the concept to life, a mobile prototype was built using AR development platforms, demonstrating key AR features such as real-time subtitle synchronization with spoken words, interaction with virtual objects, and the integration of story-relevant digital overlays.
Here are some technology details:
⚙️ Development: Unity AR Foundation (Plugin ARCoreXR for android), Vuforia SDK (Scene switch).
📱 Testing devices: Android phone or tablet with android version > 7.0.
🎨 Content production: Figma (Concept drawings, UI design), Adobe Photoshop, Adobe Premiere Pro.
Some features of the application are listed here in .GIF. You can find the full demo video here.
User Testing & Feedback
To evaluate the effectiveness of the AR application, a user study was conducted with a group of 20 participants in a controlled setting over a specified period. The study involved testing various AR features and collecting feedback through a tailored questionnaire based on the UES, SUS, and UEQ frameworks. In the study, participants experienced different AR features in a controlled setting, and answered a tailored questionnaire.
Feedback highlighted the following:
• Enhanced engagement: Users found that AR added depth to their understanding of the performance.
• Balance is key: While AR improved the experience, it needed to be carefully integrated to avoid disrupting the immersive nature of live performances.
• Technical considerations: Factors such as lighting conditions and device compatibility influenced the overall usability of the AR features.
Conclusion & Next Steps
This project demonstrated the potential of AR in enhancing performing arts experiences by providing interactive and immersive digital content. The findings emphasized the importance of seamless integration, ensuring that technology enhances rather than detracts from the artistic expression of live performances.
Future developments could explore:
• Real-time audience participation through AR-based voting or interactive storytelling.
• Expansion to different performance genres, including dance and opera.
• Collaborations with theaters and production companies to implement AR in live performances.
Thanks for reading:)