Innovative AI Tool Transforms Ideas into Physical Objects Using Lego Bricks
A groundbreaking development from Carnegie Mellon University’s School of Computer Science may revolutionize the way we build and manufacture objects. This new AI-driven tool, known as BrickGPT, leverages text prompts to guide both humans and robots in constructing models with Lego bricks. By inputting a word like “guitar,” users receive a step-by-step guide to assembling a stable model of the object.
According to Jun-Yan Zhu, Michael B. Donohue Assistant Professor of Computer Science and Robotics, “This research paves the way toward generative manufacturing, which is when people can use a generative model to design everyday objects they can build themselves.” This process allows users to create items such as furniture and toys, showcasing a novel application of generative models beyond digital media.
The integration of AI and robotics in this tool aims to streamline the design and construction processes. Associate Professor Changliu Liu mentioned, “It takes a long time to turn ideas into a physical design and prototype. But if you can integrate generative AI into the process, it can significantly improve efficiency and reduce the roadblocks to kicking off projects.”
Currently, BrickGPT can generate construction guides for 21 different Lego models, including items like a birdhouse and a piano. Users type a word into the system, which then produces a 3D model. An algorithm further converts this into a brick-by-brick guide, ensuring the stability of the final structure, which can be assembled by either people or robotic arms.
The development of BrickGPT involved creating a dataset called StableText2Brick, which includes more than 47,000 brick structures derived from over 28,000 unique 3D objects. These objects were voxelized from an existing dataset, ShapeNetCore, and used to train an autoregressive large language model (LLM). This model predicts the placement of each brick to maintain structural stability, rolling back any unstable configurations.
Alongside Liu and Zhu, the project team includes Ava Pun, Kangle Deng, Ruixuan Liu, and Deva Ramanan. Ava Pun explained, “If a structure is unstable, there’s a rollback process. During that step, the model determines which bricks were wrong or unstable and we roll back to the point before that. We detect instability with our physics reasoning algorithm, which generates a stability score for each brick in the structure.”
The team aims to expand BrickGPT’s capabilities beyond the current 21 models, enhancing the diversity and complexity of designs. This research was supported by CMU’s Manufacturing Futures Institute and conducted at Mill19. For further exploration and a demo of BrickGPT, visit the research website.
Read More Here
