An Interactive Audiovisual Experience

In a world dominated by opaque algorithms and vast social networks, we have become accustomed to being constantly exposed to strangers: their opinions, jokes, and personalities begin to blend with our own until it seems everything just blurs into one breathing system of interconnectivity.

Subjectivity is an interactive audiovisual installation built in Unity, with ChucK for audio processing and OpenCV for facial detection. Inspired by Josef Albers' 'Homage to the Square', Subjectivity invites us to consider these blurred boundaries between the self and the world, offering an experience that can be enjoyed by strangers, friends, and loners alike.


This project began with the prompt to create an interactive audiovisual experience in Unity and ChucK to display at a student expo at the end of the class.

My initial brainstorm was immediately inspired by 'Homage to the Square', a collection of conceptual artworks by Bauhaus professor Josef Albers in the 1960s.

I was touched by his commentary on the malleability of perception and the importance of context: the exact same color could appear softer or warmer to us based on what colors surrounded it. I drew a parallel to the importance of context in our own lives- the way our surroundings inform our personalities and opinions.

As I developed my first prototype, I decided to forgo keyboard and mouse input and instead focus on webcam input- blurring the boundary between user and screen to help facilitate a seamless experience.

This initial webcam functionality was relatively simple: Each square was colored based on the brightness of a pixel from the webcam using a harmonious 15-color scale.

Next, working to further complement the concept of blurred boundaries, I adjusted my squares to experience a constant fluid color change and for each layer in a square to change colors faster if a face was closer to the screen and slower if it was farther away, as seen above.

At the core, I wanted my installation to be collaborative, to invite people to consider the blurring not only within themself or between self and world but also between self and others.

To bring this to my project, I implemented an OpenCV plugin for facial detection, seen above simply as I logged the number of faces brought into the webcam. Later, I added a smoothing algorithm to adjust for a more stable number.

Finally, I added a narrative to the experience that walks users through the core principles and questions of the experience to help them better connect to the ideas and to better understand how the interaction works.

Users are encouraged to bring more people into the camera's view, to move closer and farther away, and to reflect on questions of harmony, collaboration, control, and (of course) subjectivity.

Squares are generated on screen based on the number of faces detected, and each face controls a subset of the squares on screen. While the user objectively has the same amount of control regardless of the number of squares, it becomes increasingly difficult to determine the boundaries of control as more people join the picture and play their part. This blurring boundary from an individualistic to a collectivistic mindset asks us to consider where exactly the separation lies at all.

Each individual square, while made of the same basic colors, brings people together in a microcosm of the greater system. It's these unique, ever-changing combinations that make each square special and allows each one to tap into a distinct range of emotion.  

Paired with the visuals is a constant background chord that pulses almost like a heartbeat. As more faces are detected, more notes are played and the harmony grows stronger, reinforcing a sense of joyous teamwork and collaboration.