> ## Documentation Index > Fetch the complete documentation index at: https://dimensional-feat-ivan-pure-modules.mintlify.site/llms.txt > Use this file to discover all available pages before exploring further. # Manipulation Motion planning and teleoperation for robotic manipulators. Drake remains the default world backend, RoboPlan is available as an optional planning backend, and manipulation visualization supports Meshcat or Viser. ## Quick Start Recent addition: the A-750 keyboard teleop blueprint is now available via: ```bash theme={null} dimos run keyboard-teleop-a750 ``` ### Keyboard Teleop (single command) Each blueprint launches the full stack — keyboard UI, mock controller, IK solver, and Drake visualization: ```bash theme={null} dimos run keyboard-teleop-a750 # A-750 6-DOF dimos run keyboard-teleop-piper # Piper 6-DOF dimos run keyboard-teleop-xarm6 # XArm6 6-DOF dimos run keyboard-teleop-xarm7 # XArm7 7-DOF ``` Open the Meshcat URL printed in the terminal (default `http://localhost:7000`) to see the robot. Keyboard controls: | Key | Action | | --- | -------------------- | | W/S | +X/-X (forward/back) | | A/D | +Y/-Y (left/right) | | Q/E | +Z/-Z (up/down) | | R/F | +Roll/-Roll | | T/G | +Pitch/-Pitch | | Y/H | +Yaw/-Yaw | | ESC | Quit | ### Motion Planning (two terminals) ```bash theme={null} # Terminal 1: Mock coordinator dimos run coordinator-mock # Terminal 2: Planner with Drake visualization dimos run xarm7-planner-coordinator ``` Pink IK is the default solver. Tune it with nested module config overrides: ```bash theme={null} dimos run xarm7-planner-coordinator \ -o manipulationmodule.kinematics.backend=pink \ -o manipulationmodule.kinematics.max_iterations=100 \ -o manipulationmodule.kinematics.dt=0.02 ``` For blueprints that instantiate `PickAndPlaceModule`, use the corresponding module prefix: ```bash theme={null} dimos run xarm-perception-sim \ -o pickandplacemodule.kinematics.backend=pink ``` Then use the IPython client: ```bash theme={null} python -m dimos.manipulation.planning.examples.manipulation_client ``` ```python skip theme={null} joints() # Get current joints plan([0.1] * 7) # Plan to target preview() # Preview in Meshcat execute() # Execute via coordinator ``` ### Planning backend selection Manipulation planning separates the world backend from the planner algorithm: * `world_backend` selects the robot/world/collision representation. * `planner_name` selects the path-planning algorithm. * `kinematics.backend` selects the IK backend. The legacy `kinematics_name` field remains available as a compatibility shim. Drake remains the default: ```bash theme={null} dimos run xarm7-planner-coordinator ``` RoboPlan is available as an optional backend for evaluating a non-Drake world implementation. Select it explicitly with module options: ```bash theme={null} dimos run xarm7-planner-coordinator \ -o manipulationmodule.world_backend=roboplan \ -o manipulationmodule.planner_name=rrt_connect ``` Valid combinations: | `world_backend` | `planner_name` | `kinematics.backend` | Status | | --------------- | -------------- | -------------------- | -------------------------------------------------------- | | `drake` | `rrt_connect` | `pink` | Default path | | `drake` | `rrt_connect` | `jacobian` | Legacy Jacobian IK | | `drake` | `rrt_connect` | `drake_optimization` | Drake-only IK | | `roboplan` | `rrt_connect` | `pink` or `jacobian` | Generic RRT over RoboPlan collision checks | | `roboplan` | `roboplan` | `pink` or `jacobian` | RoboPlan-native planner, using the RoboPlan world object | Invalid combinations fail during startup instead of waiting for the first plan request. For example, `planner_name=roboplan` requires `world_backend=roboplan`, and `kinematics.backend=drake_optimization` requires `world_backend=drake`. Install the manipulation dependencies: ```bash theme={null} uv sync --extra manipulation --inexact ``` The `manipulation` extra includes RoboPlan via `roboplan` from PyPI. The `--inexact` flag preserves other extras already installed in your current environment. Safety behavior for unsupported RoboPlan features: * Planning-critical unsupported inputs fail loudly before planning. Examples include unsupported obstacle geometry, unavailable robot loading APIs, or unavailable collision query APIs. RoboPlan worlds generate a minimal SRDF from the DimOS robot config, including configured collision-exclusion pairs. * Unverified non-critical query methods raise explicit `NotImplementedError`. In particular, signed minimum-distance semantics are not implemented for RoboPlan until a safe equivalent is verified. * Embedded Meshcat visualization requires a world implementing `VisualizationSpec`; use Viser or `none` with the RoboPlan backend. ### Planning Visualization Manipulation visualization is configured on `ManipulationModuleConfig.visualization`. It is independent from the global Rerun stream viewer in `docs/usage/visualization.md`. Backend choices: * `meshcat`: embedded Drake/Meshcat visualizer. The planning world must be created with embedded visualization enabled, so this is selected through the visualization config. * `viser`: in-process Viser visualizer. It renders pushed current robot state, target controls, transient preview ghosts, synchronized trajectory previews, and optional panel controls. * `none`: no manipulation planning visualization. CLI example: ```bash theme={null} uv run dimos run xarm7-planner-coordinator \ -o manipulationmodule.visualization.backend=viser ``` Blueprint example: ```python skip theme={null} from dimos.manipulation.manipulation_module import ManipulationModule, ManipulationModuleConfig manipulation = ManipulationModule.blueprint( config=ManipulationModuleConfig( robots=[...], visualization={ "backend": "viser", "host": "127.0.0.1", "port": 8095, "open_browser": True, "panel_enabled": True, # default; set False for scene-only Viser }, ) ) ``` Viser support is included in the `manipulation` extra: ```bash theme={null} uv sync --extra manipulation --inexact ``` The Viser panel talks to the concrete `ManipulationOperator` bound into its `VisualizationSession`. GUI callbacks enqueue operations through that operator for target evaluation, planning, preview, execution, cancellation, reset, and clear-plan actions. The panel owns only target drafts, selection state, and callback generations; it does not touch `WorldSpec`, IK, planner objects, `ManipulationModule`, `WorldMonitor`, or live Drake contexts directly. External manipulation visualizers are initialized from a backend-neutral `VisualizationSession` after the planning world has added its robots. The session contains static `PlanningSceneInfo` metadata: world robot IDs, `RobotModelConfig` values, and resolved planning groups. Runtime joint state is then pushed through `VisualizationStateFrame` updates so renderers do not poll world/module state or own freshness policy. Embedded Meshcat visualization does not need extra setup because it observes the Drake world directly. Previews use the stored synchronized `JointTrajectory` from the generated plan. Viser projects the globally named trajectory into robot-local preview ghosts and plays the stored timestamped points directly; optional preview duration only scales the stored delays. Execute freshness is enforced by the manipulation module/operator immediately before dispatch, not by Viser-side telemetry snapshots. ### Perception + Agent ```bash theme={null} # Coordinator + perception + manipulation + LLM agent (single command) XARM7_IP= dimos run coordinator-xarm7 xarm-perception-agent ``` For a simulation walkthrough, see [Agentic xArm simulation](/docs/capabilities/manipulation/agentic.md). ## Architecture ``` KeyboardTeleopModule ──→ ControlCoordinator ──→ ManipulationModule (pygame UI) (100Hz tick loop) (WorldSpec backend) │ │ │ TwistStamped EEFTwistTask RRT planner spatial EEF twist (Pinocchio FK/IK) JacobianIK │ DrakeWorld JointState ────────────→ (visualization) ``` * **KeyboardTeleopModule** — Pygame UI publishing routed spatial EEF twist intent * **ControlCoordinator** — 100Hz control loop with mock or real hardware adapters * **ManipulationModule** — world backend, optional visualization, RRT motion planning, obstacle management Internally, planning code depends on `WorldSpec` for world, collision, and kinematics behavior. Meshcat preview and publishing are exposed separately through `VisualizationSpec`, so non-visual planning paths do not require a visualization backend. ## Blueprints | Blueprint | Description | | ------------------------------------------------------------------------- | --------------------------------------------- | | `keyboard-teleop-a750` | A750 6-DOF keyboard teleop with Drake viz | | `keyboard-teleop-piper` | Piper 6-DOF keyboard teleop with Drake viz | | `keyboard-teleop-xarm6` | XArm6 6-DOF keyboard teleop with Drake viz | | `keyboard-teleop-xarm7` | XArm7 7-DOF keyboard teleop with Drake viz | | `xarm6-planner-only` | XArm6 standalone planner (no coordinator) | | `xarm7-planner-coordinator` | XArm7 planner with coordinator integration | | `dual-xarm6-planner` | Dual XArm6 planning | | `xarm-perception` | XArm7 + RealSense camera for perception | | `xarm-perception-agent` | XArm7 perception + LLM agent | | `xarm-perception-sim` | XArm7 simulation perception stack | | [`xarm-perception-sim-agent`](/docs/capabilities/manipulation/agentic.md) | XArm7 simulation perception stack + LLM agent | ## Supported Robots | Robot | DOF | Teleop | Planning | Perception | | ------------------------------------------------ | --- | ------ | -------- | ---------- | | [A-750](/docs/capabilities/manipulation/a750.md) | 6 | Y | Y | — | | Piper | 6 | Y | Y | — | | XArm6 | 6 | Y | Y | — | | XArm7 | 7 | Y | Y | Y | ## Adding a Custom Arm [guide is here](/docs/capabilities/manipulation/adding_a_custom_arm.md) ## Key Files | File | Description | | ------------------------------------------------------------------------------------------------------------- | --------------------------------------------- | | [`manipulation_module.py`](/dimos/manipulation/manipulation_module.py) | Main module (RPC interface, state machine) | | [`robot/manipulators/common/blueprints.py`](/dimos/robot/manipulators/common/blueprints.py) | Shared coordinator, planner, and task helpers | | [`robot/manipulators/a750/config.py`](/dimos/robot/manipulators/a750/config.py) | A-750 model and hardware config | | [`robot/manipulators/a750/blueprints/teleop.py`](/dimos/robot/manipulators/a750/blueprints/teleop.py) | A-750 keyboard teleop blueprint | | [`robot/manipulators/piper/blueprints/basic.py`](/dimos/robot/manipulators/piper/blueprints/basic.py) | Piper coordinator blueprint | | [`robot/manipulators/piper/blueprints/teleop.py`](/dimos/robot/manipulators/piper/blueprints/teleop.py) | Piper teleop blueprints | | [`robot/manipulators/xarm/blueprints/basic.py`](/dimos/robot/manipulators/xarm/blueprints/basic.py) | XArm coordinator and planner blueprints | | [`robot/manipulators/xarm/blueprints/perception.py`](/dimos/robot/manipulators/xarm/blueprints/perception.py) | XArm perception blueprint | | [`teleop/keyboard/keyboard_teleop_module.py`](/dimos/teleop/keyboard/keyboard_teleop_module.py) | Keyboard teleop module | | [`planning/world/drake_world.py`](/dimos/manipulation/planning/world/drake_world.py) | Drake physics backend | | [`planning/planners/rrt_planner.py`](/dimos/manipulation/planning/planners/rrt_planner.py) | RRT-Connect motion planner |