OpenAMRobot 2.0: open-source dual-arm mobile manipulator for imitation learning (OpenArm 2.0 on a lift)

Hi everyone,

We are building OpenAMRobot 2.0, a general-purpose open-source mobile manipulator for embodied AI: a mobile base, a 350 mm lift and two OpenArm 2.0 arms (7 DoF each). The goal is that people who know the work teach the robot by demonstration (show, correct, judge) instead of writing robot code.

Today we merged the combined robot model: base, lift and both arms in one URDF, running with MoveIt and Gazebo, with the shoulder at 1.00 to 1.35 m. The lift is next as a planned joint.

On the learning side, the stack records whole-body demonstrations in ROS 2, with an ACT baseline and evaluation now and VLA models next, all on a single Jetson Orin NX.

Repo: GitHub - openAMRobot/openamr-upperbody-sw: Open-source mobile base with lift for OpenArm 2.0 dual arms: URDF, MoveIt, Gazebo · GitHub
3D model in the browser: OpenAMRobot 2.0 Robot

Timeline: rc.1 on 20 November, v2.0.0 on 18 December after physical integration in Berlin. Software MIT, hardware CERN-OHL-P-2.0, docs CC BY 4.0.

Feedback we are looking for:

  • Recording demonstrations for a mobile base plus lift plus two arms in LeRobot dataset format: how do you handle the base and the lift in the action space?
  • Experience running ACT or a small VLA next to MoveIt 2 on Orin NX

If your team needs a mobile base for OpenArm or help integrating one, we offer a fixed-price integration sprint, just reply here. Stars and contributors are very welcome.

Alex

I’m not very familiar with physical AI, but… For now, I tried a small dataset conversion:


A possible starting route for demonstrations and ACT

Your aim of letting people teach a general-purpose robot through demonstrations, corrections, and evaluation seems worth preserving. I would not make a single learned controller for every joint, wheel, and actuator a prerequisite for that. In particular, your November 2 update already says that the mobile base, 350 mm lift, and two OpenArm 2.0 arms are in one URDF/Xacro, with the lift represented as a planned joint in MoveIt and tested in the simulation/CI path. The remaining question for imitation learning is less about adding the lift to the robot model and more about what gets recorded and what a policy is allowed to command.

For your two questions, my short answers would be:

  1. Demonstration format: Keep the two seven-joint arms and grippers, lift, and differential-drive base as named, typed components. Their action representation should follow the real control interfaces. If ACT drives the base, record forward velocity and yaw rate; if Nav2 owns travel, keep navigation goals and phase changes separate. Under your published lift safety baseline, lift positioning is also a distinct phase. Preserve observed state, requested/dispatched commands where available, timestamps, and units. Then export one verified LeRobotDataset v3 episode.
  2. ACT/small VLA plus MoveIt 2 on Orin NX: I cannot give a measured yes/no for this exact configuration. A model’s isolated FPS is not the camera-to-controller latency with ROS 2, MoveIt, recording, and multiple cameras active. A small ACT baseline on the actual NX, compared with the same pipeline under the MoveIt/ROS workload, is a low-cost way to make the decision. The right metrics are tail latency and missed deadlines as well as memory and thermal headroom.

There is a useful upstream starting point for the first question. The Enactic OpenArm Dataset format already records separate action and observation streams for two arms and an optional lifter, and its conversion API supports LeRobot v3. I ran a small CPU conversion on its real test fixture to check the exact output names and dimensions.

What the small CPU test established

I used Enactic’s dataset_0.4.0_qpos fixture, one episode, 90 output frames at 30 FPS, with cameras and smoothing disabled. The tested v3 export produced:

Variant action observation.state Result
Both arms, no declared lifter float32[16] float32[16] Passed
Both arms, recorded lifter float32[17] float32[17] Passed
Lifter declared in metadata but its streams intentionally removed No completed export No completed export KeyError: 'lifter/elevation'

For the successful lifter case, the 17 entries were ordered as right joints 1-7, right gripper, left joints 1-7, left gripper, elevation.pos. The first two sampled rows preserved the same 16 arm values across the successful variants; the last field was the added lifter value.

The missing-stream case was an intentionally inconsistent input, not an OpenAMRobot failure. It shows only that the tested exporter expects a declared component to have recorded values. Also, the upstream OpenArm Cell lift uses millimeters from the bottom while the exported meta/info.json did not retain a unit label beside elevation.pos. That is a reason to carry units explicitly when adapting the pattern to your different DOLD lift, not a reason to assume its device API is the same.

This was a CPU format experiment, not an OpenAMRobot integration test. I did not test base commands, ROS 2 topic mapping, cameras, ACT training, real lift motion, or Orin NX timing. The detail section below gives the exact provenance and limits.

The default path I would try

  1. Define an action contract before counting dimensions. For each subsystem, specify its measured state, desired target, unit, frame, update rate, command limits, and current command owner. In particular, distinguish lift height targets from lift velocity commands, and direct base twist from Nav2 navigation goals.
  2. Retain raw demonstrations. Record the relevant ROS 2 streams, images, source stamps, episode boundaries, and interventions in replayable rosbag2/MCAP or an equivalent native format. Use a small importer or a ROS-backed LeRobot driver to create named, fixed-FPS training examples. The original recording remains useful if you change the policy later.
  3. Stage lift positioning and manipulation. Your published lift-control baseline specifies lift motion only with the base stopped and arms stowed/carry-safe. So a natural first episode has base travel, safe lift adjustment, then fixed-lift arm/gripper ACT manipulation. Keep the mode boundaries in the recording. Simultaneous whole-body learning remains a separate research branch subject to the actual validated control constraints.
  4. Inspect one short conversion before scaling collection. Confirm ordered names, actual vector shapes, source units, command versus feedback, lift variation, image freshness, and interpolation/hold rules.
  5. Measure the complete NX loop. Run the intended cameras and small policy alone, then the same configuration with the relevant ROS 2/MoveIt load. Choose local, reduced, staged, or remote inference from the resulting latency and memory trace.

The most important branch is who drives the base

Task/control choice A reasonable first policy Important separate contract
Navigate, adjust lift, then manipulate Arm/gripper ACT at a held lift height; separately controlled lift phase Nav2 stop, stowed arms during lift motion, mode handoff
Learn continuous base/manipulator coordination Joint/gripper commands plus direct base twist, lift fixed under the published baseline Frames, limits, synchronization, validated allowed motions
Command end-effector goals and coordinate the whole robot Cartesian task-space policy plus a solver Frame transforms, reachability, whole-body controller

For a conventional differential-drive velocity interface, the two base actions would be body-frame forward linear.x (m/s) and yaw angular.z (rad/s), as in the ROS 2 Jazzy diff_drive_controller documentation. Those are illustrative action semantics, not a claim that your STM32 accepts exactly that interface.

If both arms/grippers contribute 16 targets, a position-controlled lift contributes one, and the policy also emits two base velocities, that is 19 scalar actions. This is dimensional arithmetic, not a canonical LeRobot layout, OpenAMRobot schema, or permission to move all axes simultaneously. The published lift baseline instead suggests mode-gated actions; velocity-controlled lift or Cartesian arms would need different fields.

For the NX, the decisive check is the same camera-to-command pipeline with and without MoveIt/ROS load, with p95, missed deadlines and memory measured. Remote inference remains a conditional alternative with explicit controller-hold requirements.

What the small OpenArm conversion actually tested

Exact CPU conversion evidence

This test used Enactic’s official tests/fixture/dataset_0.4.0_qpos data, not synthesized robot motion and not an OpenAMRobot recording. The fixture files and conversion helpers came from revision 58af735f600bd4f907fb12ac6ca30c76c8524329 of enactic/openarm_dataset. Specifically, the selection logic was the pinned lerobot_v21.py, and the writer was the pinned lerobot_v30.py.

The runtime used the installed PyPI openarm-dataset==0.4.0 core, whose dataset.py differed from that pinned snapshot. This was a mixed-source execution, not a full pinned-checkout validation. It ran on a Colab CPU with Python 3.13.18, openarm-control==0.5.0 and pyarrow==23.0.1; the earlier Windows Python 3.9 runner only retrieved files.

The export selected state=qpos, fps=30, one 90-frame episode, camera_names=[] and smoothing_cutoff=0. The genuine source Parquet streams were fetched; no synthetic fallback was used.

Both successful exports recorded action and observation.state with these names:

 0 right_joint1.pos       8 left_joint1.pos
 1 right_joint2.pos       9 left_joint2.pos
 2 right_joint3.pos      10 left_joint3.pos
 3 right_joint4.pos      11 left_joint4.pos
 4 right_joint5.pos      12 left_joint5.pos
 5 right_joint6.pos      13 left_joint6.pos
 6 right_joint7.pos      14 left_joint7.pos
 7 right_gripper.pos     15 left_gripper.pos
                         16 elevation.pos   # with lifter only

In the 17-component export, elevation.pos was the sole added terminal name. The experiment inspected the resulting meta/info.json and sampled the first two rows of the exported Parquet, verifying lengths and equality of the first 16 arm entries between successful cases. It did not independently compare all 90 rows in a second execution environment. Also, the fact that action and state had equal dimensions here is a property of this tested mapping, not a general rule for mobile manipulation datasets.

The third case deliberately retained the lifter declaration in metadata.yaml while removing both obs/lifter/elevation.parquet and action/lifter/elevation.parquet from a local test copy. The helper enumerated expected keys from metadata, then raised KeyError: 'lifter/elevation' while assembling a stream that did not exist. This is an informative consistency test; it must not be described as an observed failure of your build. For real recordings, the options would be to require the declared lift data, make a consistent arms-only variant, or implement a clearly documented missing-data policy. Mixing 16D and 17D episodes without a feature contract would be difficult to interpret.

The OpenArm native format describes 8-element qpos vectors per arm (seven joints plus gripper), and optional obs/lifter/elevation.parquet and action/lifter/elevation.parquet with one lift value and a source timestamp. Its OpenArm Cell elevation is in mm relative to the bottom. The v3 export in this test named it elevation.pos but provided no corresponding unit annotation in meta/info.json, and the pinned helper did not make an explicit mm-to-m conversion. Thus a dataset should preserve a separate unit/normalization manifest, and DOLD-specific units must be checked against DOLD’s actual driver.

The exporter constructs fixed-rate episode timestamps using i / fps rather than preserving raw acquisition times as frame timestamps. This does not establish source synchronization. Cameras, motors, base, inference and NX timing were untested. The narrow result is a working nearby two-arm-plus-lift v3 format pattern; the OpenArm training guide documents its intended use.

Action semantics and the main architecture branches

Let the controller interface determine the action

For a mobile manipulator, the same floating-point vector can describe radically different things. I would specify each action component by name, coordinate frame, unit, target versus velocity mode, rate, limits and owner, then decide how to concatenate it for ACT. The measured observation should be defined separately.

Direct learned base control: A policy can emit body-frame forward velocity and yaw rate. The Jazzy diff_drive_controller uses these two twist components, not lateral twist; the STM32 contract still needs verification. Base odometry and localization can remain observations. Coordinated base/arm behavior is a possible research branch, but it does not override the published restriction on lift motion.

Nav2-owned travel: A NavigateToPose-style goal belongs to a task/phase layer. Stop the base, move the lift only with arms in the specified safe pose, then hold it during the arm/gripper ACT phase. Base pose, lift height and context can remain observations. This baseline is simpler to audit without discarding whole-body recordings.

Cartesian or whole-body goals: A policy could instead output wrist poses/gripper state and request motion through MoveIt Servo or another whole-body solver. That requires a declared pose frame, valid transforms at the current lift elevation, reachability and a documented solver/controller interface. It is not equivalent to renaming the joint-position vector. MoveIt Servo is one relevant interface, not an automatic codec for an arbitrary learned action.

One illustrative direct-control vector is:

right joints 1..7 + right gripper      8
left joints 1..7 + left gripper        8
lift target height, if position-based  1
base forward velocity and yaw rate     2
                                      --
total                                 19

That example mixes positions and velocities; units and normalization must stay group-specific. Measured lift height is an observation, while the lift action could be a height target, velocity or one of the project’s calibrated stops. The published lift-control decision record calls for the base to be stopped and arms safely stowed during lift motion. A policy spanning multiple phases could use a phase label/action mask, rather than assuming every predicted coordinate may execute concurrently. Do not label velocity .pos merely to match Enactic’s export. Gripper conventions, MoveIt/policy ownership, and mode transitions likewise belong in the recorded contract.

The cited precedents are useful for comparing representations, but the actual OpenAMRobot command contract should come from its own controllers and demonstrations, not another project’s vector length.

ROS 2 demonstrations and a small LeRobotDataset v3 export

Record once; make the policy-specific dataset deliberately

I see two workable routes, rather than one required integration:

  • Bag first: record ROS 2 topics with their original timestamps and episode/task markers, then map a bounded subset into named LeRobotDataset features at the selected FPS. This is convenient for replaying, auditing and changing the action definition later.
  • LeRobot first: implement a ROS-backed LeRobot Robot/teleoperator that declares its features and records through the LeRobot loop. This can simplify online record/playback if the driver, Python environment and scheduling already fit the target stack.

A hybrid is possible: raw rosbag2/MCAP for provenance plus LeRobot episodes for training. I would lean toward this when the hardware interfaces are still evolving. The LeRobot ROS 2 integration RFC #4368 surveys related architectures. In issue #3139, maintainer guidance for rosbag2/MCAP conversion was to explicitly map topics to features and align frames using the dataset API; closing that issue did not establish a universal ready-made importer for OpenAMRobot.

For example, the mapping could be specified in a little table before any exporter code is written. The following names are illustrative, not verified OpenAMRobot topic names:

Source semantics Dataset destination Notes
Right/left measured JointState observation.state arm components Preserve joint order and gripper convention
Requested arm/gripper setpoints action arm components, if they are the policy’s target Do not silently substitute measurements
Measured lift elevation Observation lift feature Source unit, home, stamp, validity and age
Requested lift height or velocity Action lift feature in the allowed control phase DOLD mode, limits, stopped base and safe arm pose
Requested Twist.linear.x / angular.z Two base action components if directly policy-owned m/s, rad/s, body frame
Measured odometry/pose/twist Observation base features Frames and localization quality when needed
Camera frames Named image observations Device capture stamp, calibration, resolution
Nav2 goal and active-controller phase Task/context or sidecar metadata Not equivalent to two continuous base velocities

The source-topic-to-feature mapping table is a conceptual contract, not executable configuration. If Nav2 owns the wheels, omit continuous base action from that policy while retaining base logs. If the lift accepts velocity, record velocity with its real unit. Action and observation need not have matching dimensions; the 17/17 fixture did so only because of its converter.

Alignment is more than choosing fps=30

There are at least three different times in this pipeline:

  1. Acquisition time: when the camera exposed the frame or a sensor measured the state. Firmware may use its own clock.
  2. ROS transport/record time: when middleware sent/received the message or the bag recorded it. This can include queueing delay and is not necessarily acquisition time.
  3. Policy frame time: the regular output timeline after sampling, interpolation or hold. It may use i/fps even when the source timestamps were irregular.

The ROS 2 time design distinguishes system, steady and ROS/simulated time; /clock and use_sim_time matter in Gazebo. rosbag2 has its own recording/playback clock and QoS concerns. The LeRobotDataset v3 format describes stored frames, not a guarantee that upstream cameras and motors were synchronized correctly. Recording both the raw source stamps and the output sampling policy makes latency and image freshness diagnosable.

For a five-to-ten-second episode, I would check only the high-information items: exact feature order and shape in meta/info.json against the actual Parquet row; lift present when declared; source units; per-topic first/last stamps and inter-sample gaps; camera frame age at each policy tick; action versus feedback; and any dropped/held/interpolated samples. This catches a surprising amount without requiring a trained model or an instrumented physical robot.

What counts as an action after controller limits?

The LeRobot Robot API describes send_action() returning what was actually sent, which may be limited. LeRobot issue #4679 documents a recorder path that retained pre-send requested action rather than the returned action. This is a version/code-path-specific concern, not something observed in your robot or the Enactic conversion test.

Which of the first two is the learning target depends on the intended policy interface; the important thing is to avoid labeling one as the other. If the target happens to use Jazzy diff_drive_controller, its optional cmd_vel_out is a possible view of limited base commands. Other embedded/arm/lift controllers may expose different feedback.

For code to inspect before writing a new bridge, ngres/leros2 offers community rosbag conversion with explicit FPS/clock choices, while sacovo/lerobot_ros illustrates ROS-node-based recording and control. These are community options with version and interface constraints, not adapters I have verified on OpenAMRobot. Current LeRobot Dataset API and the installed tag should be matched when implementing create/add_frame/save_episode/finalize. The first-party OpenArm v3 exporter is another starting point for the arm/lift subset, not a complete mobile-base ROS adapter.

Jetson Orin NX: what would make a co-run result meaningful

Measure the camera-to-command loop, not model FPS alone

The proposed architecture already separates high-level Orin computation from an STM32H723/micro-ROS base-control layer. That can be helpful, because ACT and MoveIt planning do not need to execute inside each motor servo cycle. It does not, by itself, establish that the whole high-level pipeline meets the action deadline.

For a useful first answer about one Orin NX, I would compare the following with the same actual checkpoint, cameras, resolution, model precision, action horizon and desired policy rate:

Trial Workload Main diagnostic
A: vision/policy baseline Camera acquisition, decode, preprocess, inference, action postprocess and command serialization Does this input/model fit the NX without competition?
B: co-run Same pipeline with relevant ROS 2 graph, MoveIt/ros2_control and recording active Additional latency, memory and scheduling pressure
C: optional B plus Nav2/localization and representative base activity Only necessary if the task requires simultaneous navigation

Track at least:

  • End-to-end age of a command: exposure/capture timestamp through policy queue, preprocessing, inference, decoding, ROS publishing and controller acceptance where observable.
  • Tail behavior: p50 and p95 latency, p99 if sample count supports it; missed action/refill deadlines and whether stalls cluster.
  • Resources: peak RAM/GPU memory and CPU utilization, plus sustained power, clock and thermal readings from tegrastats or comparable tools.
  • Configuration: NX 8GB/16GB variant, JetPack/L4T, ROS 2 and LeRobot revisions, camera count and image sizes, checkpoint and chunk/temporal-ensembling settings.

The desired deadline depends on the task and whether the policy emits one action or a chunk. A model can have acceptable average inference time but still generate commands from stale images after a camera or planner stall. Conversely, a controller can continue at its own rate while policy action chunks refill, provided the hold/stop behavior is explicitly supported. I would not invent a universal millisecond threshold without a chosen control contract.

If the co-run has comfortable margin, local ACT is a straightforward baseline with fewer network failure modes. If it does not, choose the smallest remedy that the trace supports: fewer/lower-resolution policy image streams, a smaller validated checkpoint, lower policy query rate with bounded chunk playback, better scheduling of MoveIt planning, or task-level separation between navigation and manipulation. Each has a tradeoff in observation detail, responsiveness, or task coverage. The larger future VLA should be evaluated separately rather than assumed to inherit the ACT result.

Offboard inference is an option, not a transparent substitution

The current LeRobot main remote-inference guide describes robot-side rollout and a policy server, but its rollout contract requires a robot driver that can hold the last valid position target while waiting for the next actions. The companion hardware support table explicitly excludes standard mobile-base velocity drivers by default; OpenArm/bimanual support depends on configuration. This is an execution-contract restriction, not proof that OpenAMRobot can never use remote computation.

A conditional design would keep the velocity base under a local, timeout-aware controller and send only compatible arm position targets through a supported remote path, or implement and test a custom whole-body remote control/hold scheme. Network failure, stale action chunks and ownership transitions must still have defined behavior.

Version matters: current main uses lerobot[remote], lerobot-policy-server and lerobot-rollout rather than older lerobot[async]/PolicyServer examples. The installed tag decides which commands apply. Real-Time Chunking has a distinct contract; ACT temporal ensembling and driver hold support must be checked before a remote rollout.

Controller boundaries, lift-dependent frames, and stop behavior

How I would connect the policy to the existing stack

The author’s combined model and simulation update already includes the planned lift joint; the upper-body software repository provides the project-specific foundation. I would preserve that model and make the learned-policy command authority explicit at runtime.

Subsystem Possible command owner Transition/timeout to define
Base Nav2, supervised teleop, or learned direct twist Who may send velocity; how does stale velocity stop?
Each arm MoveIt trajectory, teleop, or learned joint targets Previous trajectory cancellation, mode switch, valid hold
Lift Planned motion, supervised driver, or learned command Homing reference, allowed mode, limits, hold/brake
Grippers Teleop or learned command Position/force convention, what persists on pause

In a ros2_control configuration, the Jazzy controller-manager documentation explains controller lifecycle, command-resource ownership and mode-switching interfaces. I am not claiming to know how your STM32 firmware and supervisors currently arbitrate those commands. The design aim is simply that MoveIt and an ACT stream do not both believe they own the same joint, and Nav2 and a direct learned cmd_vel stream do not compete in the same phase.

Stopping the base and holding an arm or lift are different operations. A base velocity command should expire and lead to a locally enforced stop; a position-controlled arm may hold the last valid target; the lift’s brake/hold/mode behavior is actuator-specific. Jazzy diff_drive_controller documents a cmd_vel_timeout, but that does not prove the OpenAMRobot custom base path implements that exact setting. A lost model process or remote connection must not be mistaken for a fresh velocity command.

The lift also changes geometry: shoulder transforms, reachability and potentially head-camera frames. Cartesian targets need the correct lift-dependent transform from the integrated model and TF; Enactic’s OpenArm pose representation does not automatically account for any external lift. More specifically, the OpenAMRobot upper-body firmware decision record calls for publishing lift position with validity and age, and treating stale position as invalidating the head-camera transform. It also restricts lift motion to a stopped base and stowed/carry-safe arms. The README explicitly says these are planned decisions, not implemented safety firmware; this is not a diagnosed TF fault.

Your published architecture already discusses E-stops, brakes, a safety layer and sensor limits, so I would not suggest those are missing. Software collision limiting, such as Nav2 Collision Monitor, is useful in an appropriately ordered velocity path but is not a hardware safety certification and does not cover all lift/arm hazards. The dataset/ACT test, ROS controller test and later physical acceptance test should be described as different evidence.

Relevant examples and source references

What I would borrow, and what I would not

  • Dual arms with learned base velocity: Mobile ALOHA supports the formulation of direct mobile actions alongside bimanual manipulation, but has no lift. Its paper-level vector and any individual converted dataset’s schema are not necessarily identical.
  • Bimanual platform with lift: YOR is a more physically similar reference, but its omnidirectional base, pose-oriented actions, and use of remote computation differ from this differential-drive local-NX proposal. Its 19D label should not dictate this robot’s 19D example.
  • Staged reachability/visibility: MoMaGen is relevant to selecting base/torso positions before manipulation. MoMa-Teleop illustrates separating some base control from the human’s end-effector motion task. These can reduce demonstration burden without precluding a later whole-body policy.
  • Broader task framing: BEHAVIOR Robot Suite is useful background for bimanual wheeled tasks, not validation of the specific DOLD lift or target compute budget.

The primary OpenArm Dataset, LeRobot, ROS 2 and controller references are linked at the points above where their specific claims matter. None is a verified complete OpenAMRobot mobile-manipulation adapter.

A practical next step

My default would be Nav2/teleop travel, lift adjustment under the published safe-pose conditions, then fixed-lift ACT manipulation, preserving the raw base/lift records for later learning. The upstream 17-feature result is a format reference, not a DOLD test. For compute, one matching pair of NX policy-only/co-run traces is the next useful evidence. If a short recording or feature manifest exists, I would be happy to check its names, units and phase conventions. I hope this supports your wider goal without prescribing one architecture.

Thank you, this is very useful, and the KeyError test is a good catch.

I’m passing your findings to our team, and we’re building them into OpenAMRobot 2.0:

  • Arms, grippers, lift and base as separately named, typed components in the dataset.
  • The lift as its own phase, matching our safety baseline (base stopped, arms stowed during lift motion).
  • Base velocity commands recorded separately from Nav2 goals.
  • The lift recorded in metres, with units written into the dataset metadata. Millimetre data is converted on import.
  • Raw ROS 2 bags first, converted to LeRobot format afterwards. For remote inference, the controller holds the last valid target.
  • An ACT benchmark on the real Orin NX, with and without ROS 2 and MoveIt 2, tracking tail latency, missed deadlines, memory and thermal headroom. We’ll post the numbers here.

We’ll share a short Gazebo sample recording here before rc.1 on 20 November. It would be great if you could check its names, units and phases.

We’d also be glad to have you work with us more closely: in our GitHub (OpenAMRobot · GitHub), in the team Slack, or in whatever form suits you. If you’re interested, send me a message here or at alex@botshare.ai and I’ll send you an invite.