Power to the People
Feb 20, 2023
Part 1
I think my weakness is that all I do is read and hear and think about things and everything I read or hear or think about reminds me of some other thing I've read or heard or thought about. So here's some things I've read or heard or thought about relating to interactive AI systems.
On Friday, I attended an incredible panel speaking event with Dr. Margaret Mitchell (Chief Ethics Scientist at Hugging Face), Dr. Pattie Maes (Professor at MIT Media Lab), and Dave Willner (Head of Trust & Safety at OpenAI). At the talk, they discussed their thoughts on the promises and perils of AI and how we might mitigate potential risks, each offering a different perspective unique to their background and line of work.
One idea discussed in the talk was what Dr. Maes called the Turing Trap: the notion that the Turing Test had fundamentally sent AI researchers down the wrong path since the very beginning of machine learning. She suggested that the Turing Test established the goal of AI research to create machines that outperformed humans. Thus, researchers are perpetually taking tasks that humans are good at (chess, driving, making art) and making models that meet and surpass our own performance. By following this path, we find ourselves optimizing against our own interests as we build systems that fundamentally aim to replace humans rather than empower humans. Instead, she proposed that we try to optimize for what people + AI would be able to accomplish rather than just AI alone. Furthermore, she suggested that we should evaluate AI systems not just in their accuracy or bias, properties that are easily quantifiable, but also in their secondary qualitative effects (i.e. how people respond to and interact with the AI). To do this, she suggested that AI researchers test with users before they've even made the system (using Wizard-of-Oz techniques), then use the information about how people interacted to the system to ensure that what they build responds properly to the people it's meant to serve. These are all ideas that stick out to me as important when designing interfaces: enhance the performance of humans with AI, and consider the secondary effects of human behaviour.
When I think about designing interactive AI systems in general, one idea that I always come back to is "escaping the screen." I read a fantastic article from Bret Victor about what he called the "Pictures Under Glass" phenomenon: designing "revolutionary" technological advantages that lose all the magic of our 3D, tactile, texture-rich world and collapse interfaces to a glassy facade. He had a great quote about sandwiches: "The next time you make a sandwich, pay attention to your hands. Seriously! Notice the myriad little tricks your fingers have for manipulating the ingredients and the utensils and all the other objects involved in this enterprise. Then compare your experience to sliding around Pictures Under Glass. Are we really going to accept an Interface Of The Future that is less expressive than a sandwich?" While I'm not really sure exactly what it would look like, I'm excited by the potential of interfaces that invite us to poke and prod and hold and grab and move and squish and bend and slide and spin and twist and play and explore and every other tactile verb we can think of.
A favorite interaction designer of mine right now is Linus Lee, a researcher in creative AI who makes a lot of his own software tools exploring HCAI and who writes a blog filled with thoughts on designing interfaces. He writes about how "for every new capability of computers, the first generation of software interfaces tend to expose the most direct and raw interface to that capability," for example chatbot interfaces directly exposing the feature of text completion to the user. He talks a lot about play as well: how interactive system with tight, immediate feedback loops lend themselves much more to a playful mindset than a 'work' mindset, and how we can use these feedback loops to make work feel more like play. He is also a big proponent of making your own workflow tools (and his personal AI projects/tools are incredibly powerful), which inspires me to think about AI interfaces opening up a wider breadth of personalization: tools and toys that bend around us instead of forcing us to bend around them.
And finally, Ge Wang's Humans in the Loop article that wraps together all these thoughts on interactive AI systems, now more specifically with regard to "artistic" AI tools/interfaces. He writes about how, in artistic domains, a piece of art is about more than just the product/outcome, but about the process and journey as well. A video from SPACE10 on the future of creativity in the age of AI explains how the invention and dissemination of photography seemed like it bore the end of painting as an artistic discipline. In response, however, we see the an explosion of artistic creativity: abstract expressionism and surrealism and conceptualism and entire artistic movements that poke at the definition of art and that invite criticism and focus, like Ge suggests, on both the process and the product.
In general, I find myself thinking a lot about how automation poses a threat to intentionality and thoughtfulness by optimizing away that kind of delayed gratification that forces us to think more deeply about what we're doing. In the Design of Everyday Things, I remember Don Norman talking about how the invention of the e-mail promised widespread connectivity, how we would be able to instantaneously communicate with people on the other side of the world. And while that might be true with every text and e-mail and tweet and Instagram post, there's something still to be said for the fact that none of these will ever compare, neither for the sender nor the receiver, to the feeling of sending or receiving a handwritten letter that trekked its way across the planet to you. A letter, with all its texture and realness and intentionality and effort covered in signs that "I was here," that a human was here and a human made this and no one else could have created this same thing. Maybe it's cheesy but I think that's magical. AI offers similar promises: beyond instantaneous communication and on to instantaneous knowledge, instantaneous art, instantaneous music, instantaneous productivity. As we design the next iteration of technological progress with AI tools, I think it's important to continue valuing and encouraging meaningful interaction, collaboration, and that special human touch that comes from the process.
.jpeg)
Part 2:
Generate 10 activities or tasks (in different areas, musical or otherwise) that could benefit from an interactive AI approach/mindset. For each of these, with as much as specificity as you can muster in a single sentence, describe how it would work
I had ChatGPT generate some ideas, and I'll comment a little on how each idea might work or how it could be adjusted to better align with some of the ideas I outlined above: how it could be made more interactive, explore interfaces beyond a screen, enhance human + AI performance, employ tight feedback loops, and emphasize the process to keep a human touch.
ChatGPT: Language learning: An interactive AI language learning application could analyze the learner's progress and customize lessons to focus on their areas of weakness, providing interactive exercises and conversations with virtual language partners.
GS: I've tried to learn many languages with varying degrees of success: Spanish, French, Dutch, Italian, and Japanese, to name a few. I think one thing I struggle with on Duolingo is when a vocabulary word is on the tip of my tongue or when I can remember it in a different language. I think conversational language learning assistants is a great idea, but maybe it could be combined with some interactive parameters to adjust whether you want new content or revision/practice or some kind of interface that offers you new vocabulary words to learn or maybe dynamically displays words/grammar that you frequently mess up on.
ChatGPT: Music production: An AI music producer could help musicians and producers to generate new ideas, suggest chord progressions, and provide real-time feedback on song structure, melody, and rhythm.
GS: In imagining different kinds of screen-free interactivity, I can totally imagine this kind of music AI assistant acting more on the process/brainstorming side as some kind of attachment to a guitar (like those small digital tuning clips) that listens to you playing chords and generates a melody that you can then begin to play along with. Maybe it switches up the melody/tempo periodically to encourage exploring new ideas or you can manually change it by using your foot or voice activation or some other part of your body.
ChatGPT: Healthcare: An AI-powered virtual assistant could help patients manage their health, reminding them to take medications, suggesting exercise routines, and providing real-time feedback on vital signs such as heart rate and blood pressure.
GS: When I'm working out, something I always struggle with is playing the right music. I like doing those automatic treadmill routines where it automatically sets up run/walk intervals and it would be sick to have an AI that uses your walking speed to play music that you would like at a BPM matches your pace. Constant changing would be annoying, so maybe it only adjusts if you change your pace significantly (say between a run/walk interval). Another struggle I find with working out is wondering whether or not I have the proper form when doing strength training, so it would be awesome if AI could help gently correct my posture or form (either through voice activation in my headphones or some kind of wearable that uses haptic feedback).
ChatGPT: Retail: An AI-powered personal shopping assistant could help customers find products based on their preferences, suggest complementary items, and provide real-time feedback on sizing and fit.
GS: Shopping I find tricky to adjust because I honestly think we have a massive problem in the US with overconsumption and shopping addictions and mass waste production that's all produced unethically or all ends up in landfills (@ Shein). Rather than helping customers buy more, I'd be interested in an AI that helps us consume less: maybe by suggesting more ethical retailers, reminding you that you already have similar clothing before you buy more, helping thrift shops with resale of gently used clothing, or even suggesting how to revamp a piece of clothing you never wear (restyling or tailoring).
ChatGPT: Education: An interactive AI-based learning platform could assess students' knowledge and adapt their learning paths based on their progress, providing interactive exercises, feedback, and personalized learning materials.
GS: In education, I'm again skeptical of any AI tools that put students in front of more screens than they probably already see when using phones and tablets and TVs at home. I went to a Montessori school when I was little and really valued the independence, collaboration mindset, and freedom that it encouraged. I'd be interested in AI tools that facilitate these ideas: maybe encouraging collaboration by pairing students together that are interested in a similar activity and generating activities that they could do together. As far as limiting reliance on screens, maybe AI in education could be more of a tool for teachers to gauge class progress.
Travel: An interactive AI travel agent could suggest destinations based on customers' interests, preferences, and budget, providing real-time feedback on travel options, local culture, and attractions.
GS: Honestly, my problem with traveling isn't as much coming up with places to go as much as it is how I get there. I'd definitely be interested in some kind of AI interface that helps users get from point A to point B in a new country, maybe with detailed live information about how the public transit system works (Can I realistically buy a ticket there or do I have to plan way in advance? Is it expensive? Should I sign up for a pass for discounts? Which metro passes lead to which places?) or with information on which areas are safe/sketchy, maybe with adjustable parameters based on whether I care most about safety or about cost or about timeliness.
ChatGPT: Cooking: An AI-powered recipe assistant could provide personalized recipe recommendations based on dietary restrictions, ingredient preferences, and cooking skill level, as well as real-time feedback on cooking techniques and timing.
GS: Cooking feels like an especially process-important task: people likely cook because they enjoy the process of creating rather than simply wanting food. Something I struggle with in cooking is multitasking, so maybe an AI would be helpful in taking in what I want to make, offering suggestions, and helping me figure out which parts I should start in what order (so I remember to start the rice cooker at the beginning rather than forgetting until the end). Maybe you can adjust parameters based on whether you like things spicy, sweet, crunchy, flavorful, well-done, etc, and the AI system could adapt a recipe you're interested in to include different ingredients/substitutes to best fit your preferences.
Gaming: An AI game companion could analyze players' gameplay and provide suggestions for strategies, tips, and customized challenges to enhance the gaming experience.
GS: I think CPU players offer a great opportunity to play video games and develop your skills even if you don't have a real person to play the game with. More than just filling up the game arena, I'd be interested in AI co-players that function more like real people- maybe you can talk to it through your headphones just like you would a normal person to generate and carry out a certain strategy (to maintain the ideas of agency and collaboration that come with playing with real people).
Writing: An interactive AI writing assistant could analyze the writer's style and provide real-time feedback on grammar, sentence structure, and writing flow, suggesting synonyms and providing prompts to overcome writer's block.
GS: Something I struggle with in writing is making sure I'm signposting enough or making sure my ideas are coming through to the reader. My process is usually just to start word vomiting onto the page then later refine and rearrange until I have something coherent. I'd be interested in AI systems that maybe encourage signposting or that help you recognize recurrent themes in your writing, maybe you can give the AI your rubric or prompt and as you write it can ask you informative open-ended questions like a peer editor might do about how you might strengthen a certain point or develop a certain idea, offering a kind of bigger-picture feedback that falls between the granularity of Grammarly and the do-the-whole-thing-for-you of ChatGPT.
ChatGPT: Art: An interactive AI art instructor could provide real-time feedback on technique, composition, and style, as well as suggest creative prompts and provide inspiration for new projects.
GS: Again, I'm intrigued more by the idea of interactive AI art systems that act as a peer editor rather than trying to replace you and create all the art for you. For artists, I'd be interested in an AI system that asks you questions about your work and progress almost as if you're at a critique: maybe you can interact with it by asking it questions or clarifying what you want feedback on (How could I improve the color balance in the piece? I don't like x aspect, could you help me brainstorm how I might incorporate that?). For art consumers, I'd be interested in AI artist assistant maybe that provides contextual feedback on a piece and asks you questions: maybe it remembers past pieces you've seen in a museum/online or pieces that you've liked and encourages you to make connections between artist's techniques/ideas or historical context. Alternatively, maybe it asks you how you feel about the artist's technique or what you think the message of a piece is to help you make the most out of observing art (generally presenting open-ended questions back to the user rather than presenting assertive truths about it's interpretation).