Out of all of the midnight experiments I did, here's the one that was my favorite. I struggled a little with getting ChucK to output the sound I wanted, so I decided to focus more on the paint application and what different drawings would look like. Enjoy!

EXPERIMENTS

Experiment 1: Trackpad

For my first experiment, I found an obscure GitHub repo from Lingdong Huang that sends Mac trackpad data through OSC, which was perfect for the purposes of Wekinator. Unfortunately, however, the data is sent as an XML message, which Wekinator can't read, so I had to do some manual adjustments in the Processing visualizer starter code so that it could parse the XML message for meaningful values (position) and send them through a new OSC port to Wekinator for training (Note: It supports a maximum of 5 fingers and data is padded with 0s to ensure it's always the same size). I trained on about 7 gestures with random sounds and combined with Ge's starter code for receiving Wekinator outputs in ChucK, then trained Wekinator on the examples to create the sound control that you see in the video.

In total, this experiment runs 4 programs at once: Trackpad OSC (1) sends data to Processing (2), which visualizes it and sends it to Wekinator (3), which then sends outputs to ChucK (4) to generate the sound.

Experiment 2: Hand Gestures

The next obscure GitHub repo I raided was by Frederick (faaip), and sends computer vision hand gesture data from the webcam through OSC. Then I discovered it doesn't work on Mac, and I saw Terry from class had commented on the repo with the same problem, so I decided to try something else. Consider this experiment closed.

Experiment 3: Color Sensing

Next, I looked into "breaking free of the screen" and hooked up my laptop to my Arduino to see what I could get out of it. My main problems here were troubleshooting serial communication between Arduino and Processing, which I originally tried to fix by adding a Python middleman but ended up sticking with Processing. I added an Adafruit TCS34725 and adapted the starter code to send data to Processing and display the color that the sensor sees (since I learned quickly that you have to hold an object straight up to the sensor for it to see it properly), which is displayed in the small colored square. Like before, I trained on a handful of colorful examples around my room then trained in Wekinator.

In total, this experiment runs 4 programs at once: The Arduino (1) sends data to Processing (2) over serial port, which displays the color and sends it to Wekinator (3) via OSC, which then sends outputs via OSC to ChucK (4) to generate the sound.

The Arduino setup with my colorful objects of choice

Experiment 4: Emotions

The next GitHub repo I looked at was a deep vision library for Processing. I chose an example that opens up webcam input, tracks faces, and classifies the faces with an emotion. I thought it had some fun capabilities for controlling the sound with your face without having to train all the different facial features myself. I added an emoji classifier in the corner to show the emotion found by the program. I added OSC generation to send face classification info to Wekinator, where I trained on different emotions to generate sound using the starter code.

In total, this experiment runs 3 programs at once: Processing (1) detects faces and sends data to Wekinator (2) via OSC, which then relays sound information to ChucK (3) to generate sound.