CS470 Final Project

Part A

Full video with sound>

As I reflected on my work this quarter, I found myself still... haunted by our first etude using Word2Vec to create poetry. As I wrote in that initial reflection, trying to write poetry using such a primitive NLP tool was uninspiring at best and frustrating at worst and I began to feel sorry for the fact that I was asking a computer program to accomplish something so fundamentally distinct from its capabilities. Poetry (and any kind of art for that matter) only feels successful when imbued with some kind of reflection on the nature of humanity, and Word2Vec's nonsensical mumbling felt like I was watching someone go crazy.

For one of the poetry toys, beauty.ck, I was thinking about what it would mean for a computer to be able to perceive beauty in nature, which feels like an important building block in poetic and artistic movements. I imagined my program in an endless field, picking flowers and reflecting on what each different kind and color represented. Of course, the "beauty" is all fake and the sounds and text quickly become annoying and repetitive, breaking whatever illusion might have been there and reminding us of the fundamental nature of the program.

This reflection on my unintentional personification led me to my ideas for the rest of the project!

Part B

In addition to the reflection on the first etude, there were a handful of other things that led me to my final project.

Poets of Sound and Time: We started the class with an etude using word2vec and I was immediately struck by how much I resonated with the program running in VSCode. I knew it didn't have any emotion, but it felt somehow cruel to run this program over and over waiting to gain any meaningful emotion out of something fundamentally not made for that.

My conversation with Dave Wilner (Head of Trust and Safety at OpenAI). It went something along the lines of:

Dave: See, Bing didn't train their model as well as we did, which is why it's telling people it loves them. At OpenAI, we invented RLHF, so being "good" is baked into our model rather than being tacked on at the end of learning.
Me: I feel like every time I listen to these conversations I'm struck by how all the language seems to be referring to children and animals. "Learning," "training," etc. I hear that and I think: yeah, you can ~train~ a dog, but there's still a risk that it'll-
Dave: -bite you? Exactly. That's why I take my job seriously.

If Dave is scared. I should definitely be scared too. Wtf.

Replika, a company specializing in AI chatbots that learn from talking to you and that act as companions, with the paid version including being able to talk to your chatbot as if they were your girlfriend or wife. Yikes. Recently, they lost a bunch of customers from the paid plan because they banned the erotic roleplay (ERP) features.

Essentially, this all left me thinking about AI as a digital companion, or some kind of digital wild animal that we've conveniently domesticated as a pet but that can and maybe actually will maul all of humanity.

The only logical path, then, is to look at the original digital pets. Tamagotchi.

Vimeo Link>

For the final project, I made a little virtual tamagotchi bunny that responds to gesture input to appear, turn its head, and open its mouth. There's a gentle chord melody in the background, and when you open your puppet hand, the bunny opens its mouth to "sing." A small health bar at the top of the screen indicates the bunny's health: if you go too long without letting her sing, her health declines, her song starts to slow down, and she appears more sad. Eventually, without proper care, she dies :(

The words of the bunny's song are generated by Word2Vec starting from an initial choice of some lyrical words. While the notes are randomly generated, there's some small ChucK code that counts (guesstimates) the syllables in the words so she can sing them properly. The more you let her sing, the faster she goes!

Finally, the buttons on the bottom allow you to add some ambient sounds, which reflect on the screen in little drawings to add to her environment. I liked the idea of providing "enrichment" to the virtual pet in a way that changed the sound without distracting from the idea of singing.

The data starts in VisionOSC (1), where the OSC gesture input data is sent to Processing (2), where it's parsed and processed before being sent to Wekinator (3), where the hand data is used to classify the mouth position and head direction labels, which are sent back to Processing (4), where the correct animation frame is rendered on the Tamagotchi and the Wekinator data is passed again to ChucK (5), where the melody is generated based on Word2Vec and finally these lyrics are sent back to Processing (6) to be rendered on the screen along with the health bar. There's a lot going on but it's all worth it!

Overall, I'm pretty happy with this final project! I wish I could've made it a little more existentialist or dystopian, but the idea of a virtual pet/companion was really fun to play around with and I got a lot better at coding music in ChucK! Plus, I think it's a super cute little aesthetic.