Understanding Music Visually - Feedback and Feedforward

Human emotions are expressed through musical performances for the musician and their audience (Gabrielsson & Juslin, 1996). Musical performances typically utilize instruments that are played and vocals that are sung by people. Traditional instruments take form of different physical artefacts of varied shapes and sizes that are used to play by striking, strumming, and blowing to produce different sounds. A performance usually comprises of multiple instruments that are played in harmony. While playing a number of traditional instruments such as the guitar or a piano require skill, technology looks to blur the gaps by transforming traditional instruments into its digital equivalents.

Technology has enabled people to create music that is generated by traditional instruments using computers and even hand held devices in the form of applications. In terms of HCI, technology relied on traditional interaction paradigms that of ‘WMIP’ and ‘Mobile’. The evolution of technology has given rise to more novel ways of interacting with it forming newer paradigm. NIME (New Interfaces for Musical Expression) looks to produce a new paradigm shift to create music in unique interactive ways. These new interfaces are developed in varied forms and have no strict platform. Reactable (Jordà et al., 2017), is an example of using physical blocks and table to interactively create music. Another example is a project created by Cinimod studios (Cinimod Studio, 2017), called as ‘Aurora’ that utilizes gestures without any physical artefacts to create music while providing a dynamic visual experience to people.

SoundScape is a new musical interface created for people to interact with Lycra® as an interface to create music by pushing and gesturing on it while the gestures produce visualizations based upon the extent to which a push or a gesture is produced. Soundscape is an iteration of the concept SoundGarden that was created for coursework studies for the course Physical Computing in the first half of 2017 for the University of Queensland.

Traditional instruments keep us constantly aware of what we can do next in terms of the next key we want to press, the next fret we want to push, and the next string we want to strike by maintaining a sense of visible awareness. Playing traditional instruments produce organic sounds and haptic feedback, while newer forms of technology, that of NIME, fail to reciprocate haptics as accurate as traditional instruments (Jordà, 2003). While feedback mechanisms are an challenge with NIME, there are also challenges with respect to feedforward mechanisms (Dobrian & Koppelman, 2006) in terms of understanding what one can do next. Dobrian (2006) highlights this by explaining the visual information that is traditionally used to show people ‘what can be done’ and ‘what just happened’. Jordi (2003) illustrates highlights the value of visualization in performances as means to enhance the musician’s and audience’s experiences. It was also found that visualizations would positively improve experiences when displayed in accordance with emotions (Chen, Weng, Jeng, & Chuang, 2008).

This report will illustrate a few examples of existing projects, compare them to SoundScape, and explain their feedback and feedforward mechanisms in terms of visualizations. People’s feedback will also illustrate the success to which SoundScape implemented both the mechanisms.

Fire & Ice by Cinimod

Fire and Ice is an installation created by Cinimod (2017) to allow people to use gestures as means to create dynamic visualisations. The sound that are created in the installation is to depict fire and ice. A maximum of two people can simultaneously take the role of fire and ice respectively to move their arms and create dynamic patterns out of the visualizations. Fire and Ice uses a Kinect sensor to monitor people’s arm movement and manipulate visualizations while the visualization themselves is displayed on a flat LED screen. The main aspect of this installation is to provide dynamic visualizations to people who create and who watch them. In order to attain maximum system efficiency, it was important to produce visualizations with minimum latency.

The feedback mechanism of this system is audio-visual but is predominantly visual as people gave more value to what they saw to interact with the installation. Research has identified people’s reliance on visual information to make acumen to interact (Tsay, 2013). Limitation of Fire & Ice as an installation was the lack of cues on how to interact with the system.

Wood wall for inside Rolls-Royce by Cinimod

The wood wall for inside rolls Royce is an interactive installation which was created by Cinimod for Rolls-Royce to portray the wooden interiors of Rolls-Royce. The interface itself comprised of a flexible screen that could be pushed and gestured to produce a multitude of visual patterns. The installation could afford more than two people interacting on the same screen to produce different patterns based upon the number of simultaneous interactions. The visuals that were produced were mapped using a Kinect sensor behind the screen and a projector was used to create the visualizations. In addition to the visualization, the system also afforded an orchestral soundtrack to allow people to manipulate the sounds while interacting with the visuals.

The feedback the installation provided was a little ambiguous due to the nature of the unpredictable patterns generated due to pushes. However, the objective of the design itself could be to make people to interact and explore newer patterns. In terms of limitations, the system did not allow people to clearly understand what they could do, or what they did.

Aurora by Cinimod

Aurora is an installation developed by Cinimod to allow people to experience music creation in a gestural format. While the objective of this installation was to create an open-ended playful experience for people, Aurora only affords one person to interact with the installation at a time where the person stands in the centre of the installation and moves their arms around to create music. The visuals react to their arm movements to give them a sense of feedback. Simultaneously, along with the visualizations, the music generated are also manipulated.

The feedback to people are audio-visual but visual cues supersede the audio cues as people use the visualization to understand ‘what they did’. While the interaction mode of this installation is instructing, exploring, and manipulating, the installation does little when it comes to show ‘what possibly can be done’ highlighting its limitation when it comes to feedforward mechanisms.

Reactable

A more famous example of a NIME that was more successfully implemented is the Reactable. The Reactable is a tangible interface that allows people to control music with the use of physical artefacts on a table. Placing a physical object on the table creates a visualization and sound. Connections can be visually created by placing multiple objects on the table to create compound sounds, they are visually shown in the form of different waveforms to differentiate between connections, and these connections can also visually be manipulated to change. In order to manipulate the music, people use the visual cues and change them with use of the objects. The success of Reactable has resulted in it being recreated for more commercial purposes in the form of applications for iPads and iPhones.

The affordance created by using tangible objects to create and use visualizations to adjust aspects of sound is a good example of good feedback and feedforward. Placing an object creates a ring of controls around the object, further, turning the object results in adjusting the different filters, levels and sound frequencies. People can maintain a sense of control over the system with the support of visualizations.

Whilst visualizations are the premise to provide feedback for people in different ways, they all enhance the overall experience for the people composing the soundtrack. However, the success of SoundScape depends upon people’s feedback. For the purpose, feedback was obtained from people to understand how well these mechanisms have aided them to produce music that is coherent in terms of music generation. The feedback obtained is sectioned into pre-performance and post-performance feedback to detail how feedback mechanisms evolved through the process, its constraints, their success factors and limitations.