Lyraflo
A VR music theory learning experience, developed in Unity.
Skills: Rapid Prototyping, Transformational Design, Playtesting
Lyraflo: Music Theory in VR, 2021 https://projects.etc.cmu.edu/lyraflo/
Game Designer, Sound Designer, Composer
Project Overview
Lyraflo was a 14-week project which aimed to explore how the unique properties of VR could be utilized to convey music theory concepts to music beginners. We delivered 3 prototypes which explored different ways of leveraging VR to teach different aspects of music theory, ranging from pitches, cadences, to tonality.
On this page, I will go over how I contributed to this project through Rapid Prototyping.
My Role: Game Designer, Sound Designer, Composer
Skills: Rapid Prototyping, Playtesting, Transformational Design
Project Trailer
Rapid Prototyping
As the field of music education in VR is very much unexplored territory, there were a lot of things our team had to figure out by ourselves. We had a lot of ideas, but not enough time to fully prototype and test them out. One of my key responsibilities was coming up with ways to test our ideas in the fast, efficient ways.
Here is an example of how I rapidly prototyped for one of our ideas.
The Major Minor Prototype
I built 4 Rapid 'Paper' Prototypes over the course of 2 weeks to quickly test and gather data on the feasibility on a idea, before our team started prototyping in VR.
The Idea
Use immersive VR environments to help teach music beginners the difference between major and minor tonalities.
The idea was to create an audio 'playground', where the player could freely change the tonality of a piece of music between major and minor. As they do so, the surrounding VR environment would dynamically shift to match the mood and atmosphere of the music, so music beginners could better feel and understand the differences between tonalities.
*Tonality in music is how pitches and chords are arranged into a hierarchy of relationships between each other, and can drastically affect how people perceive a piece of music.
A demo of a VR prototype we built for this idea
Identifying Key Questions
I first began by coming up with the various problems and questions we had concerning our idea. Would visuals actually be helpful? Can music beginners even hear the difference between major and minor tonalities? I wonder how they perceive major and minor when they hear it?
I distilled these thoughts into isolated questions I could test and answer one at a time.
The 4 Rapid Prototypes: Overview
*Click on the Prototype Titles to go to a more detailed explanation section
Key Question:
Can music beginners hear the difference between major and minor tonalities?
How We Tested:
Conducted two variations of a listening test with music that could shift between major and minor tonalities.
The Result:
Music Beginners can KIND OF hear the difference between major and minor
Key Question:
Do music beginners make differing visual associations to different tonalities?
How We Tested:
Observed how music beginners matched images with music in different tonalities.
The Result:
Music Beginners do make differing visual associations for different tonalities
Key Question:
Is there a consistent trend or theme to how musical beginners make visual associations to certain musical tonalities?
How We Tested:
Sent out a survey with a collection of images, and two music files in major and minor, and had participants choose all images they thought matched with each music file, and as well as write out adjectives to describe each music file.
The Answer:
Yes! There is a noticeable trend to the visual associations music beginners make to each tonality!
Key Question:
Do supplemental visuals help music beginners grasp the difference between major and minor tonalities more clearly?
How We Tested:
Conducted a playtest with two different groups of testers. One group got to listen to the two tonalities of music along with supplemental visuals, while another group, the control, only got to listen to the music, for 5 minutes total. Then we had each participant take a listening test with only music, which gauged how well they were able to detect differences between music in different tonalities.
The Result:
Visuals do help music beginners grasp the difference between major and minor tonalities!
Conclusion:
Through the 4 rapid prototypes I created over 2 weeks, our team was able to learn that not only does our idea work, but we also how we could design our music and visuals to be the most helpful for our players.
-
We used data from Prototype 1 to inform how we composed our music so the shifts between tonalities would be easier to understand for music beginners. To quickly go over the general music theory of this, major and minor tonality have a lot of notes/sounds in common, with just 2 or 3 notes that are slightly different between the two. By composing our music so it avoids the notes in common between the two tonalities, and incorporates a lot of those 2 or 3 notes that slightly changes between the two, we could make shifts in tonality more easily recognizable for our players.
-
We used data from Prototype 2 + 3 to design our visuals in a way that best helps supplement our music. We took the most commonly made visual associations to each tonality, and incorporated those traits into the environment of our VR prototypes.
A trailer for one of the VR Prototypes
In our final VR prototype, 20 out of 22 testers, who had no musical knowledge, learned to be able to hear and spot the differences between major and minor music by ear alone after playing our prototype for 20 minutes!
The Rapid Prototypes: Details
Rapid Prototype 1: Music in Major and Minor
Key Question:
Can music beginners hear the difference between major and minor tonalities?
We first had to test if music beginners could hear the difference between music in major and minor at all.
How We Tested:
To test this out, I composed a test track of music that was designed to shift between major and minor seamlessly.
We had our testers listen to the tracks on loop. We first had them listen to the major version, then the minor version, and then asked if they heard a difference between the two.
Then, we would play the track, but shift seamlessly between major and minor variations without any cues, asking the testers to raise their hand when they thought the tonality had shifted.
The purpose of this test was to first see if testers could hear differences between major and minor, then test how well they could distinguish between the two in a live test.
The examples below are composed in the same key, tempo, and instruments, but with different tonalities. Can you hear the difference?
The Result:
Music Beginners can KIND OF hear the difference between major and minor
After bringing in around 20 people to listen to the test tracks, we found that music beginners can hear a difference between the two tonalities to a limited degree.
If we played the tracks one at a time, almost all of the testers said they could hear the difference, although they struggled to put into words how they felt different.
When we played the tracks and shifted between the tonalities, the testers had a much harder time pinpointing when the shifts occurred, and only 6 people managed to do it on some sort of consistent basis.
This gave us valuable feedback that although music beginners could hear some differences between tonalities, they had a hard time really grasping onto the changes without more help.
Rapid Prototype 2: Tonality and Images
Key Question:
Do music beginners make differing visual associations to different tonalities?
Now, knowing that music beginners could kind of hear the differences between tonalities, we wanted to test if visuals could help them better grasp this concept.
We now aimed to test if music beginners make any sort of visual association to different tonalities.
How We Tested:
Those familiar with music often describe major as happier, and minor as sadder, so we chose a variety of brighter and darker colored images, and put them onto a PowerPoint.
The concept was simple. We play the music in each tonality one at a time, and let the tester go through the PowerPoint of images, until they found an image that they thought fit the music best.
Try listening to the examples again, which image do you think fits for each example?
The Result:
Music Beginners do make differing visual associations for different tonalities
After around a dozen playtests, we found that testers chose a brighter images for major, and darker images for minor. We also found that testers were able to better word what they perceived as different between each example track with the help of the images.
Now, we needed to dig deeper, and analyze what kind of visual associations players were making with each tonality.
Rapid Prototype 3: Tonality and Image Association Survey
Key Question:
Is there a consistent trend or theme to how musical beginners make visual associations to certain musical tonalities?
In order to create a compelling VR environment that would help music beginners better grasp tonalities, we needed to learn more about if there were certain recurring key traits to the visual associations being made to major and minor music.
How We Tested:
For this question, we needed to gather a lot of data in a short amount of time in order to find significant trends or consistent patterns, so I decided to make a autonomous 'prototype' of sorts to help us out.
I created a survey with a collection of images, along with 2 music files in major and minor tonalities. Participants were asked to choose all the images they thought corresponded to each music file, and also write adjectives to describe each music file.
The aim of this test was to gather large amounts of data on visual associations for both tonalities, as well as to understand the thought process behind how the participants were making these associations.
Some of the images in the survey
The Answer:
Yes! There is a noticeable trend to the visual associations music beginners make to each tonality!
We managed to gather around 60 responses across 2 days, and found a strong trend in the visual associations being made.
Almost all of the responses matched major with brighter, colorful images, and described it with words like lively, hopeful, joyful, energetic, etc.
The responses for minor were slightly more varied, but still coalesced around darker, more ominous images, and words like dangerous, mysterious, and sad.
We now knew that there were consistent trends to how music beginners made visual associations to differing tonalities.
Now, we wanted to know if we could use this data to come up with visuals that could help music beginners better grasp the difference between major and minor.
Rapid Prototype 4: Music with Visuals Test
Key Question:
Do supplemental visuals help music beginners grasp the difference between major and minor tonalities more clearly?
For the final prototype, I wanted to test if our idea of using visuals to help convey tonality differences could actually work, without having to set up an time consuming prototype in VR.
How We Tested:
Using the data from Rapid Prototype 3, my team and I chose an image to go with the major and minor variations of the test track.
The image for minor: Dark, Mysterious, Danger
The image for major: Bright, Hopeful, Relaxed
Part 1:
We had two groups of testers.
Group 1: Test
We played the test tracks for our testers, along with its corresponding image. Testers could freely switch between the major variation and its image, and the minor variation of the track and its image . The testers were allowed to do this for up to 5 minutes. The purpose of this part of the test was to get testers familiarized with the two tracks, with the help of the images.
Group 2: Control
We allowed the testers to freely listen to the two tracks, without images, for up to 5 minutes.
Part 2:
We then took away the images from Group 2, and played the tracks, switching between major and minor without telling the tester when. We had our testers raise their hands when they thought the tonality switched.
The Result:
Visuals do help music beginners grasp the difference between major and minor tonalities!
The results were surprisingly successful for us.
6 out of 8 testers in Group 1 (Test) were successful in identifying when the tonality switches occurred in part 2 of the test.
2 out of 8 testers in Group 2 (Control) were successful in identifying when the tonality switches occurred in part 2 of the test.
It seemed like being able to explore the music in major and minor, while also receiving visual feedback in the form of the images, helped the music beginners better understand and grasp the changes between tonalities, to the point where they could detect the music shifting in tonalities just by ear much more successfully.
Thanks for tuning in to Lyraflo!