Investigating the Impact of Responsive Feedback on the Experience of Learning to Conduct with Sympathetic Orchestra
Bob Tianqi Wei, Shm Almeda, Ethan Tam, and Dor Abrahamson.
CHI 2025, PAPER UNDER REVIEW

ABSTRACT
Learning to conduct a symphony orchestra effectively requires reflexive understanding of music and personalized expression, but conducting students often practice with recordings, which lack the real-time feedback necessary for developing these critical skills. In this work, we developed Sympathetic Orchestra, an interactive hand-tracking system designed to offer a dynamically responsive audio playback experience to support the development of tacit knowledge for conducting. Our research aims to identify specific features of responsive orchestral conducting interfaces that show potential for enhancing conducting education compared to static recordings. An empirical study analyzed the behaviors of novice conducting students learning to conduct with Sympathetic Orchestra. Analyses focused on how aspects of system design interacted with students’ behavior as they developed conducting skills like phrasing, modulation, and expressive interpretation. Our findings provide insights into the design of future interactive learning technologies to support the development of tacit musical knowledge.
‍

Demonstration of Sympathetic Orchestra: An Interactive Conducting Education System for Responsive, Tacit Skill Development
Bob Tianqi Wei, Shm Almeda, Ethan Tam, and Dor Abrahamson.
UIST 2024

ABSTRACT
Students learning musical conducting often practice along to static recordings, which do not provide real-time feedback similar to that of a live orchestra during rehearsals. Novice conductors need better solutions for practicing with feedback that mimics the experience of conducting a live orchestra. We can leverage emergent multimodal and spatial interaction technologies to support a “virtual orchestra” practice experience that allows students to develop tacit, live-practice knowledge. Through formative interviews with conducting experts and students, we designed and developed a dynamic, multimodal interaction system that targets key goals held by students developing their orchestral conducting skills, and that traditional practicing methods lack support for. Sympathetic Orchestra is an interactive virtual orchestra system that uses Google AI edge-powered Computer Vision hand and face tracking on webcam data to responsively interact with dynamic audio music playback and develop tacit practicing experiences.