InstructMesh helps repair SI-generated models before 3D printing

The research tool lets users highlight parts, describe defects and approve changes. Its novice evaluation suggests potential, but does not establish general reliability.

The research tool lets users highlight parts, describe defects and approve changes. Its novice evaluation suggests potential, but does not establish general reliability.
InstructMesh lets users repair and customize SI-generated 3D designs by highlighting parts and describing what needs to change before printing. Researchers at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Google and Northeastern University developed the tool, which MIT News described on October 1.
The problem it addresses is practical: a generated object can look right without working as intended. A mug, for example, might resemble a cup but fail to hold liquid. Fixing such designs can require skills in specialist 3D-modeling software, making repairs difficult for beginners.
InstructMesh pairs Microsoft’s TRELLIS, which generates 3D models from text and images, with GPT-4, a large language model that works with language and images. TRELLIS supplies the 3D-generation capability; GPT-4 helps interpret what the user wants to change.
Users highlight a part of the design and describe the problem in ordinary language. According to lead author Faraz Faruqi, InstructMesh makes changes through the generator’s internal representation of the model. It then presents the altered geometry—the object’s shape and structure—for the user to evaluate and approve.
Sliders provide finer control over particular adjustments. Users can, for example, enlarge a selected part or extrude it, extending it outward rather than relying only on a written instruction.
The team used TRELLIS to recreate popular designs from Thingiverse, a platform for sharing 3D-printable models. Researchers then asked novices to find and repair problems in those generated models using InstructMesh.
| Measure | Reported result |
|---|---|
| TRELLIS recreations with structural flaws | Nearly 80% |
| Novices identifying and fixing issues, as reviewed by an expert | Around 90% of the time |
The participant count and detailed criteria for assessing repairs remain unclear. These researcher-reported results are not an independent comparison with another approach and do not establish how reliably InstructMesh would work across other designs or users.
MIT News also reported physical demonstrations, including a shell-shaped whistle, glasses with butterfly wings above the lenses and a shrimp-shaped robot enclosure containing a motor. Researchers fabricated a denim-looking knee brace too, but the reported results do not establish medical-device safety.
Physics simulations are a possible future addition. They could help assess how a design responds to use—for example, whether a bowl breaks when dropped—and which materials might suit it. Those checks are distinct from the fabrication demonstrations already described.
The October 1 report also said Faruqi was working at Google and might incorporate InstructMesh into an augmented reality platform, which would connect digital content with a user’s physical surroundings. The proposed use would let people describe an object in the context of nearby items; it was presented as a future possibility, not an available feature.
Google participated in the research and, alongside the MIT-HPI Collaborative Research Program, partly funded it. MIT News said the team would present the work at the ACM Symposium on User Interface Software and Technology in November.

An MIT theoretical study separates repeated rejection from reduced discovery and finds that averaging several algorithms can sometimes improve simulated hiring results.
3 min

The macOS and Windows preview lets Copilot use on-screen controls. Removing a saved approval does not revoke access already granted in a running session.
3 min

Nordic Makers invested as co-founder Lasse Finderup took majority ownership. The new tool connects recordings with earlier interview material, according to the company.
2 min