Imagine this: Our super-smart AI can write poems, translate languages, and even create stunning images, but it struggles with something incredibly basic – tying a knot! That's the surprising reality revealed by new research from Cornell University, which puts AI's spatial reasoning skills to the test.
Researchers discovered that while today's AI models excel at many tasks, they're surprisingly inept when it comes to the 3D world. They can untangle simple knots, but tying a knot from loops or transforming one knot into another? That's where they get tangled up.
The study's lead author, Zoe (Zizhao) Chen, a doctoral student at Cornell Tech, points out a critical limitation: "With current AI, it works great with big blocks of text. Once moved to reason in the 3D world, AI breaks." This suggests that despite its impressive abilities, AI still has a long way to go in understanding and manipulating objects in space. This is a crucial skill for the future of robotics and other AI-powered fields.
To explore this, Chen and Yoav Artzi, an associate professor at Cornell Tech, created KnotGym, a 3D simulator designed to challenge various AI models, including advanced large language models (LLMs) like GPT-4. KnotGym presents AI agents with visual representations of loops and knots, asking them to untie, tie, or convert knots. It also features a "generalization ladder" to gradually increase the difficulty of the knot-tying tasks.
And this is the part most people miss... AI agents achieved a roughly 90% success rate in untying knots with up to four crossings, including a standard shoelace knot (which has three crossings). However, the difficulty skyrocketed when it came to tying knots. Simple two-crossing knots were tied with an 83% success rate, but this plummeted to 16% for knots with three crossings. Knots with more than three crossings completely stumped the AI. The success rate for knot conversions mirrored the struggles with tying knots.
Chen explains that AI currently lacks the ability to play, explore, and learn in the same way humans do. "When kids play with this,” she says, referencing a Rubik’s Cube, “they just start fiddling around and eventually figure out a sequence of actions that get them to certain configurations without messing up the rest. They explore. They reuse lessons, build off previous knowledge, and work toward a grander goal. That’s an ability we want to see with AI, but it’s not there yet.”
But here's where it gets controversial... The research suggests that AI's current reliance on text-based reasoning is a significant limitation. While AI can process and generate text with impressive fluency, it struggles to translate this knowledge into practical, real-world actions involving spatial understanding.
In future work, Chen plans to enhance KnotGym by utilizing Graphics Processing Units (GPUs) to improve evaluation speed.
What do you think? Do you believe that the ability to understand and manipulate objects in 3D is essential for true AI? What other areas do you think AI is currently struggling with? Share your thoughts in the comments below!
This research was supported by the National Science Foundation, Open Philanthropy, Nvidia Academic Grant, and the National Artificial Intelligence Research Resource (NAIRR) Pilot.
This article was written by Louis DiPietro, a writer for the Cornell Ann S. Bowers College of Computing and Information Science.
