CMU Researchers Build an AI Robot That Paints
Researchers at Carnegie Mellon University’s Robotics Institute have developed a device called FRIDA, which is a robotic arm with a brush attached to it. The tool leverages artificial intelligence (AI) to work with humans on art projects.
The team will present a research entitled “FRIDA: Collaborative Robot Painter With Differentiable Real2Sim2Real Planning Environmentsat the IEEE 2023 International Conference on Robotics and Automation in May.
Peter Schaldenbrand is a Ph.D. student at the Robotics Institute in the School of Computer Science. He works with FRIDA and explores AI and creativity.
“There is one painting of a frog ballerina that I think turned out to be very good,” he said. “It’s totally silly and fun, and I think the surprise of what FRIDA came up with based on my input was really fun to see.”
FRIDA stands for Framework and Robotics Initiative for Creating Arts. It was named after Frida Kahlo.
This research was led by Schalderbrand, together with RI faculty members Jean Oh and Jim McCaam, and has attracted students and researchers from across CMU.
Non-Artist Collaboration Tool
Users can guide FRIDA by entering a text description, submitting other artwork to inspire her style, or uploading a photo and asking her to paint a representation of it. The team also tested other inputs, such as audio.
“FRIDA is a robotic painting system, but FRIDA is not an artist,” continues Schalderbrand. “FRIDA did not come up with the idea of communicating. FRIDA is a system that artists can collaborate with. Artists can set high-level goals for FRIDA and then FRIDA can execute them.
To paint pictures, the robot uses an AI model comparable to that of ChatGPT OpenAI and DALL-E 2, which generates text or images in response to commands. FRIDA simulates how to paint a picture with brush strokes and uses machine learning to assess its progress as you work.
FRIDA’s final product is whimsical and impressionistic. The brush strokes are thick and lack the precision one often seeks in robotic endeavors.
“FRIDA is a project that explores the intersection between human and robotic creativity,” added McCann. “Frida uses the kinds of AI models that have been developed to do things like draw text and understand scene content and apply that to these artistic generative problems.”
FRIDA uses AI and machine learning several times during its art creation process. First, he spends an hour or so learning how to use his brush. Then, it uses a visual language model that has been trained on a large dataset that pairs text and images pulled from the internet, such as OpenAI Contrastive Image-Language Pre-Training (CLIP), to understand the input.
One of the most significant technical challenges in producing a physical image is reducing the simulation-to-real gap, which is the difference between what FRIDA creates in the simulation and what it paints on canvas. FRIDA uses an idea known as real2sim2real, in which actual robotic brushstrokes are used to train a simulator to mirror and mimic the robot’s physical abilities and painting materials.
The FRIDA team now aims to overcome some of the limitations in the current big vision language model by continuing to refine the one they use. They modeled the headlines of news articles to give them a sense of what’s going on in the world and coached them further on images and text that are more representative of diverse cultures to avoid American or Western bias.
