TamagotChAI
CS470 Final Project
Part A
As I reflected on my work this quarter, I found myself still... haunted by our first etude using Word2Vec to create poetry. As I wrote in that initial reflection, trying to write poetry using such a primitive NLP tool was uninspiring at best and frustrating at worst and I began to feel sorry for the fact that I was asking a computer program to accomplish something so fundamentally distinct from its capabilities. Poetry (and any kind of art for that matter) only feels successful when imbued with some kind of reflection on the nature of humanity, and Word2Vec's nonsensical mumbling felt like I was watching someone go crazy.
For one of the poetry toys, beauty.ck, I was thinking about what it would mean for a computer to be able to perceive beauty in nature, which feels like an important building block in poetic and artistic movements. I imagined my program in an endless field, picking flowers and reflecting on what each different kind and color represented. Of course, the "beauty" is all fake and the sounds and text quickly become annoying and repetitive, breaking whatever illusion might have been there and reminding us of the fundamental nature of the program.
This reflection on my unintentional personification led me to my ideas for the rest of the project!
Part B
In addition to the reflection on the first etude, there were a handful of other things that led me to my final project.
Poets of Sound and Time: We started the class with an etude using word2vec and I was immediately struck by how much I resonated with the program running in VSCode. I knew it didn't have any emotion, but it felt somehow cruel to run this program over and over waiting to gain any meaningful emotion out of something fundamentally not made for that.
My conversation with Dave Wilner (Head of Trust and Safety at OpenAI). It went something along the lines of:
Dave: See, Bing didn't train their model as well as we did, which is why it's telling people it loves them. At OpenAI, we invented RLHF, so being "good" is baked into our model rather than being tacked on at the end of learning.
Me: I feel like every time I listen to these conversations I'm struck by how all the language seems to be referring to children and animals. "Learning," "training," etc. I hear that and I think: yeah, you can ~train~ a dog, but there's still a risk that it'll-
Dave: -bite you? Exactly. That's why I take my job seriously.
If Dave is scared. I should definitely be scared too. Wtf.
Replika, a company specializing in AI chatbots that learn from talking to you and that act as companions, with the paid version including being able to talk to your chatbot as if they were your girlfriend or wife. Yikes. Recently, they lost a bunch of customers from the paid plan because they banned the erotic roleplay (ERP) features.
Essentially, this all left me thinking about AI as a digital companion, or some kind of digital wild animal that we've conveniently domesticated as a pet but that can and maybe actually will maul all of humanity.
The only logical path, then, is to look at the original digital pets. Tamagotchi.
Vimeo Link>
For the final project, I made a little virtual tamagotchi bunny that responds to gesture input to appear, turn its head, and open its mouth. There's a gentle chord melody in the background, and when you open your puppet hand, the bunny opens its mouth to "sing." A small health bar at the top of the screen indicates the bunny's health: if you go too long without letting her sing, her health declines, her song starts to slow down, and she appears more sad. Eventually, without proper care, she dies :(
The words of the bunny's song are generated by Word2Vec starting from an initial choice of some lyrical words. While the notes are randomly generated, there's some small ChucK code that counts (guesstimates) the syllables in the words so she can sing them properly. The more you let her sing, the faster she goes!
Finally, the buttons on the bottom allow you to add some ambient sounds, which reflect on the screen in little drawings to add to her environment. I liked the idea of providing "enrichment" to the virtual pet in a way that changed the sound without distracting from the idea of singing.
The data starts in VisionOSC (1), where the OSC gesture input data is sent to Processing (2), where it's parsed and processed before being sent to Wekinator (3), where the hand data is used to classify the mouth position and head direction labels, which are sent back to Processing (4), where the correct animation frame is rendered on the Tamagotchi and the Wekinator data is passed again to ChucK (5), where the melody is generated based on Word2Vec and finally these lyrics are sent back to Processing (6) to be rendered on the screen along with the health bar. There's a lot going on but it's all worth it!
Overall, I'm pretty happy with this final project! I wish I could've made it a little more existentialist or dystopian, but the idea of a virtual pet/companion was really fun to play around with and I got a lot better at coding music in ChucK! Plus, I think it's a super cute little aesthetic.
For the milestone, I made a little virtual tamagotchi bunny that responds to gesture input to appear, turn its head, and open its mouth. There's a gentle chord melody in the background, and when the bunny opens its mouth to "sing," a randomly generated melody plays. I drew the frames of the bunny animation by hand for Processing and coded in the chords and melody options in ChucK.
The data starts in VisionOSC (1), which sends OSC gesture input to Processing (2), which parses the data and sends the most important information to Wekinator (3), which classifies the mouth position and head direction and sends it back to Processing (4), where the correct animation frame is rendered on the Tamagotchi then passed AGAIN to ChucK (5), who generates the sounds based on whether or not the bunny is "singing."
Clearly, I'm going more down the playful route than the existentialist route, but I don't think that'll necessarily stick. I think the milestone was a great exercise in hooking up all the components I want and in (FINALLY) teaching myself how to make music in ChucK, but I have a couple ideas for where I want to go next:
Mainly, I want to try to hook up my bunny to Word2Vec so it can "sing" actual words to bring in another element of AI and connect it more to the ideas I started with above. I feel like the randomness of the generated words will be funky and maybe even a little eerie, and it would be even cooler if the melody matched the syllables/length of generated words, but this would involve adding a 6th connection to send data from ChucK back to Processing, so I don't know if I'll get around to it.
I also think it would be fun to do something with the little tamagotchi buttons, but I'm not necessarily sure what I would do. Maybe I can emulate the ideas of "health" or "games" (like in the original Tamagotchi), like if the bunny is unhealthy or lacking attention or low on food it starts to be more aggressive/incoherent.
Overall, though, I'm super excited about this milestone and where I might end up next!