> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # LLM-driven world and model authoring > Using large language models to generate Gazebo SDF/Xacro world files and 3D models from natural language descriptions, with prompt patterns, validation workflows, and literature review. Writing Gazebo world files and 3D model descriptions by hand is tedious and error-prone. Large language models (LLMs) can generate valid SDF XML from plain English descriptions, dramatically accelerating the world-building process. This page covers the techniques, prompt patterns, and tools for LLM-assisted Gazebo asset authoring. This page builds on [Gazebo worlds and the SDF/Xacro format](/aiml-common/lectures/simulation/gazebo-worlds), which covers world-file anatomy, the SDF and Xacro schema, the `model.config` and `model.sdf` structure, and the AWS RoboMaker Small House world that the examples below extend. Read that page first if you are new to SDF authoring. *** ## Why use LLMs for simulation authoring? A typical Gazebo model requires coordinating several files: `model.config`, `model.sdf`, mesh references, material scripts, and placement poses in the world file. Writing this by hand demands: * Knowledge of SDF schema and element nesting rules * Correct inertia tensor values for realistic physics * Consistent URI conventions (`model://name/meshes/...`) * Collision mesh simplification decisions LLMs encode knowledge of the SDF schema from training data and can produce syntactically valid XML for common object types in seconds. The human role shifts from *author* to *reviewer and corrector*. *** ## Literature review Several research threads are directly relevant to NLP-to-simulation authoring: ### Procedural and generative world creation Early work on procedural content generation (PCG) for simulation used template engines and grammar-based methods. LLMs represent a qualitative shift: they can interpret underspecified natural language and infer reasonable defaults. **WorldGen / SceneX-type approaches**, papers such as *SceneX* (CVPR 2024) and *Holodeck* (CVPR 2024) demonstrate end-to-end pipelines that convert a text description of a room ("a modern living room with a grey sofa facing a TV") into a 3D scene with placed furniture. These systems query LLMs to select object categories and spatial relationships, then look up 3D assets from a database (Objaverse, ShapeNet) and assign poses. The core LLM task is **structured output generation**, producing a JSON or XML scene graph from free text. **Code-as-interface**, *ProgPrompt* (ICLR 2023) and *Code as Policies* (ICRA 2023) show that LLMs are effective at generating robot task programs when given a library of available primitives. The same principle applies to SDF authoring: if the LLM is told what XML elements are available and what each does, it generates structurally correct documents. The SDF schema serves as the "primitive library." **Language-to-simulation bridges**, *ChatSim* (CVPR 2024) specifically targets autonomous driving simulation, converting natural language commands into scene edits in a CARLA/nuScenes-style environment. The authors use a multi-agent LLM pipeline where one agent interprets the command, another places objects, and a third checks physical plausibility. ### Prompt engineering for structured output Getting an LLM to produce valid XML consistently requires careful prompt design: 1. **Schema injection**, include the relevant portion of the SDF specification in the system prompt. Modern long-context models (Claude, GPT-4o) can hold the full SDF 1.6 schema and still produce coherent output. 2. **Few-shot examples**, provide one or two complete model.sdf examples before the request. LLMs are highly sensitive to format exemplars. 3. **Validation feedback loop**, pipe the generated XML through `gz sdf --check` and feed errors back as user messages for iterative correction. 4. **Decomposition**, generate the collision mesh description and visual mesh description separately, then combine. Monolithic generation of complex models degrades quality. ### Mesh generation SDF model files reference `.DAE` (COLLADA) or `.OBJ` meshes for geometry. LLMs generate the *description* of the model, but mesh files must come from one of: * **3D asset libraries**, [Google Poly](https://poly.pizza/), [Sketchfab](https://sketchfab.com/), [AWS RoboMaker models](https://github.com/aws-robotics) * **Procedural geometry**, for simple shapes (boxes, cylinders, L-shaped objects), SDF primitive geometry suffices and requires no mesh file * **Text-to-3D models**, *Shap-E* (OpenAI, 2023) and *One-2-3-45* generate meshes from text or images; quality is improving rapidly but remains lower than hand-authored assets for precise collision models For classroom exercises, **SDF primitive geometry** (boxes, cylinders, spheres) is the recommended starting point. It requires no mesh files and the LLM can generate the complete SDF without any external assets. *** ## Prompt patterns ### Pattern 1: Generate a model from a description Use this pattern to create a new `model.sdf` for a simple object using primitive geometry. **System prompt:** ``` You are an expert in Gazebo SDF 1.6 model authoring. Generate syntactically valid SDF 1.6 model files. Use primitive geometry (, , ) unless a mesh URI is provided. Always include realistic mass and inertia values. Set false for movable objects. Respond with only the XML, no explanation. ``` **User prompt:** ``` Create a model.sdf for a rectangular wooden coffee table. Dimensions: 1.2 m long, 0.6 m wide, 0.45 m tall. Top surface thickness: 0.03 m. Four legs, each 0.05 m square cross-section. Mass: 12 kg. The table is a static piece of furniture. ``` **Example LLM output:** ```xml theme={null} true {/* Table top */} 9.0 0.279 1.082 1.350 000 0 0 0.435 0 0 0 1.2 0.6 0.03 0 0 0.435 0 0 0 1.2 0.6 0.03 0.55 0.35 0.15 1 0.55 0.35 0.15 1 {/* Front-left leg */} 0.75 0.0130.0130.0002 000 0.575 0.275 0.21 0 0 0 0.05 0.05 0.42 0.575 0.275 0.21 0 0 0 0.05 0.05 0.42 0.55 0.35 0.15 1 0.55 0.35 0.15 1 {/* (front-right, back-left, back-right legs follow same pattern) */} top leg_fl ``` ### Pattern 2: Place models in a world file Use this pattern to generate the `` placement blocks for a room layout. **User prompt:** ``` I have a 6 m × 4 m living room in a Gazebo world. The origin is at the center of the room. Generate SDF placement blocks for: - A sofa along the south wall (y = -1.8), facing north, model URI: aws_robomaker_residential_Sofa_01 - A coffee table in front of the sofa, model URI: aws_robomaker_residential_CoffeeTable_01 - Two armchairs facing the sofa, model URI: aws_robomaker_residential_ChairA_01 - A TV stand on the north wall (y = 1.8), facing south, model URI: aws_robomaker_residential_TVStand_01 All models are static. Use realistic spacing. Output only the XML blocks. ``` **Example LLM output:** ```xml theme={null} true model://aws_robomaker_residential_Sofa_01 0 -1.5 0 0 0 0 true model://aws_robomaker_residential_CoffeeTable_01 0 -0.6 0 0 0 0 true model://aws_robomaker_residential_ChairA_01 -1.2 -0.3 0 0 0 -1.5708 true model://aws_robomaker_residential_ChairA_01 1.2 -0.3 0 0 0 1.5708 true model://aws_robomaker_residential_TVStand_01 0 1.6 0 0 0 3.1416 ``` ### Pattern 3: Iterative correction with validation feedback After generating SDF, validate it with the Gazebo command-line tool and feed errors back to the LLM: ```bash theme={null} gz sdf --check model.sdf ``` If validation fails, append the error to the conversation: ``` The SDF validator returned this error: Error: Missing required element [inertia] in [inertial] at line 12 in model.sdf Please fix the model.sdf to resolve this error. ``` The LLM will correct the specific element while preserving the rest of the file. This loop typically converges in 1–3 iterations for straightforward models. *** ## Worked example: generating a new appliance model The AWS Small House world includes air conditioners (`AirconditionerA_01`) but no ceiling fan. Here is how to generate one using the LLM workflow. ### Step 1: Describe the object ``` Create a model.sdf for a ceiling fan. The fan consists of: - A central hub: cylinder, 0.15 m diameter, 0.05 m tall, mass 1.5 kg, mounted at z = 0 (fan hangs from ceiling) - Four blades: each a flat box, 0.45 m × 0.12 m × 0.008 m, mass 0.2 kg each Blades are arranged 90° apart, centered 0.25 m from hub center - The fan is static (ceiling-mounted, no spinning in simulation) Name: ceiling_fan ``` ### Step 2: Generate Send the prompt to your preferred LLM (Claude, GPT-4o). The response will be a complete `model.sdf` using `` and `` primitives. ### Step 3: Create model directory ```bash theme={null} cd tb_worlds/models mkdir ceiling_fan ``` Create `model.config`: ```xml theme={null} ceiling_fan 1.0 model.sdf Generated by LLM Ceiling fan with four blades, static ``` Save the generated SDF as `model.sdf` in the same directory. ### Step 4: Validate ```bash theme={null} gz sdf --check tb_worlds/models/ceiling_fan/model.sdf ``` Fix any errors by feeding them back to the LLM. ### Step 5: Place in the world Add a placement block to `house_world.sdf.xacro`: ```xml theme={null} true model://ceiling_fan {/* Hang from kitchen ceiling at z = 2.5 m */} 1.5 0.5 2.5 0 0 0 ``` ### Step 6: Launch and inspect ```bash theme={null} docker compose up demo-world-house ``` Open Gazebo and verify the fan appears at the correct location. If the position is wrong, adjust the `` values and restart. *** ## Common failure modes and fixes | Failure | Cause | Fix | | ------------------------------------ | ----------------------------------------------- | ------------------------------------------------------------------------------ | | Model spawns underground | Z pose too low for model origin | Increase z in `` to half the model height | | Model floats above floor | Z pose too high | Decrease z | | Physics instability (model explodes) | Unrealistic inertia tensor (off-diagonal terms) | Set all off-diagonal terms to 0; use `true` for furniture | | Collision mesh blocks navigation | Collision geometry too large | Scale collision down, or use `true` with simplified collision | | `gz sdf --check` URI error | Model name in URI doesn't match directory name | Ensure `model://name` matches the directory name exactly | | Model invisible in Gazebo | Missing `` element | LLM sometimes omits visual for collision-only models; add it | *** ## Integration with the TurtleBot demo Adding new models to the house world affects robot navigation in two ways: **1. Obstacle map**, If a model has a `` element, Nav2 will detect it as an obstacle via the LiDAR scan. Static furniture that the robot should navigate around must have collision geometry. **2. Camera perception**, New objects may be detected by the YOLOv8 or HSV vision pipeline. If you add a bright red object to the scene, the HSV detector (targeting `TARGET_COLOR=red`) will report a detection when the robot looks at it. After modifying the world, re-run SLAM to generate an updated map: ```bash theme={null} # Terminal 1: house world with new furniture docker compose up demo-world-house # Terminal 2: run GMapping SLAM (if using 2D SLAM) docker compose run overlay ros2 launch nav2_bringup slam_launch.py # Save the map docker compose run overlay ros2 run nav2_map_server map_saver_cli \ -f tb_worlds/maps/house_world_map ``` *** ## Further reading * [Gazebo worlds and the SDF/Xacro format](/aiml-common/lectures/simulation/gazebo-worlds), the manual world-authoring foundation this page extends * [Gazebo SDF specification](http://sdformat.org/spec), complete schema reference * [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://yueyang1996.github.io/holodeck/) (CVPR 2024) * [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://arxiv.org/abs/2209.11302) (ICLR 2023) * [Code as Policies: Language Model Programs for Embodied Control](https://arxiv.org/abs/2209.07753) (ICRA 2023) * [ChatSim: Editable Scene Simulation for Autonomous Driving](https://arxiv.org/abs/2402.05746) (CVPR 2024) * [AWS RoboMaker Small House World](https://github.com/aws-robotics/aws-robomaker-small-house-world), open-source residential models *** [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/simulation/llm-world-authoring.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).