I’m not very familiar with physical AI, but… For now, I tried a small dataset conversion:
A possible starting route for demonstrations and ACT
Your aim of letting people teach a general-purpose robot through demonstrations, corrections, and evaluation seems worth preserving. I would not make a single learned controller for every joint, wheel, and actuator a prerequisite for that. In particular, your November 2 update already says that the mobile base, 350 mm lift, and two OpenArm 2.0 arms are in one URDF/Xacro, with the lift represented as a planned joint in MoveIt and tested in the simulation/CI path. The remaining question for imitation learning is less about adding the lift to the robot model and more about what gets recorded and what a policy is allowed to command.
For your two questions, my short answers would be:
- Demonstration format: Keep the two seven-joint arms and grippers, lift, and differential-drive base as named, typed components. Their action representation should follow the real control interfaces. If ACT drives the base, record forward velocity and yaw rate; if Nav2 owns travel, keep navigation goals and phase changes separate. Under your published lift safety baseline, lift positioning is also a distinct phase. Preserve observed state, requested/dispatched commands where available, timestamps, and units. Then export one verified LeRobotDataset v3 episode.
- ACT/small VLA plus MoveIt 2 on Orin NX: I cannot give a measured yes/no for this exact configuration. A model’s isolated FPS is not the camera-to-controller latency with ROS 2, MoveIt, recording, and multiple cameras active. A small ACT baseline on the actual NX, compared with the same pipeline under the MoveIt/ROS workload, is a low-cost way to make the decision. The right metrics are tail latency and missed deadlines as well as memory and thermal headroom.
There is a useful upstream starting point for the first question. The Enactic OpenArm Dataset format already records separate action and observation streams for two arms and an optional lifter, and its conversion API supports LeRobot v3. I ran a small CPU conversion on its real test fixture to check the exact output names and dimensions.
What the small CPU test established
I used Enactic’s dataset_0.4.0_qpos fixture, one episode, 90 output frames at 30 FPS, with cameras and smoothing disabled. The tested v3 export produced:
| Variant |
action |
observation.state |
Result |
| Both arms, no declared lifter |
float32[16] |
float32[16] |
Passed |
| Both arms, recorded lifter |
float32[17] |
float32[17] |
Passed |
| Lifter declared in metadata but its streams intentionally removed |
No completed export |
No completed export |
KeyError: 'lifter/elevation' |
For the successful lifter case, the 17 entries were ordered as right joints 1-7, right gripper, left joints 1-7, left gripper, elevation.pos. The first two sampled rows preserved the same 16 arm values across the successful variants; the last field was the added lifter value.
The missing-stream case was an intentionally inconsistent input, not an OpenAMRobot failure. It shows only that the tested exporter expects a declared component to have recorded values. Also, the upstream OpenArm Cell lift uses millimeters from the bottom while the exported meta/info.json did not retain a unit label beside elevation.pos. That is a reason to carry units explicitly when adapting the pattern to your different DOLD lift, not a reason to assume its device API is the same.
This was a CPU format experiment, not an OpenAMRobot integration test. I did not test base commands, ROS 2 topic mapping, cameras, ACT training, real lift motion, or Orin NX timing. The detail section below gives the exact provenance and limits.
The default path I would try
- Define an action contract before counting dimensions. For each subsystem, specify its measured state, desired target, unit, frame, update rate, command limits, and current command owner. In particular, distinguish lift height targets from lift velocity commands, and direct base twist from Nav2 navigation goals.
- Retain raw demonstrations. Record the relevant ROS 2 streams, images, source stamps, episode boundaries, and interventions in replayable rosbag2/MCAP or an equivalent native format. Use a small importer or a ROS-backed LeRobot driver to create named, fixed-FPS training examples. The original recording remains useful if you change the policy later.
- Stage lift positioning and manipulation. Your published lift-control baseline specifies lift motion only with the base stopped and arms stowed/carry-safe. So a natural first episode has base travel, safe lift adjustment, then fixed-lift arm/gripper ACT manipulation. Keep the mode boundaries in the recording. Simultaneous whole-body learning remains a separate research branch subject to the actual validated control constraints.
- Inspect one short conversion before scaling collection. Confirm ordered names, actual vector shapes, source units, command versus feedback, lift variation, image freshness, and interpolation/hold rules.
- Measure the complete NX loop. Run the intended cameras and small policy alone, then the same configuration with the relevant ROS 2/MoveIt load. Choose local, reduced, staged, or remote inference from the resulting latency and memory trace.
The most important branch is who drives the base
| Task/control choice |
A reasonable first policy |
Important separate contract |
| Navigate, adjust lift, then manipulate |
Arm/gripper ACT at a held lift height; separately controlled lift phase |
Nav2 stop, stowed arms during lift motion, mode handoff |
| Learn continuous base/manipulator coordination |
Joint/gripper commands plus direct base twist, lift fixed under the published baseline |
Frames, limits, synchronization, validated allowed motions |
| Command end-effector goals and coordinate the whole robot |
Cartesian task-space policy plus a solver |
Frame transforms, reachability, whole-body controller |
For a conventional differential-drive velocity interface, the two base actions would be body-frame forward linear.x (m/s) and yaw angular.z (rad/s), as in the ROS 2 Jazzy diff_drive_controller documentation. Those are illustrative action semantics, not a claim that your STM32 accepts exactly that interface.
If both arms/grippers contribute 16 targets, a position-controlled lift contributes one, and the policy also emits two base velocities, that is 19 scalar actions. This is dimensional arithmetic, not a canonical LeRobot layout, OpenAMRobot schema, or permission to move all axes simultaneously. The published lift baseline instead suggests mode-gated actions; velocity-controlled lift or Cartesian arms would need different fields.
For the NX, the decisive check is the same camera-to-command pipeline with and without MoveIt/ROS load, with p95, missed deadlines and memory measured. Remote inference remains a conditional alternative with explicit controller-hold requirements.
What the small OpenArm conversion actually tested
Exact CPU conversion evidence
This test used Enactic’s official tests/fixture/dataset_0.4.0_qpos data, not synthesized robot motion and not an OpenAMRobot recording. The fixture files and conversion helpers came from revision 58af735f600bd4f907fb12ac6ca30c76c8524329 of enactic/openarm_dataset. Specifically, the selection logic was the pinned lerobot_v21.py, and the writer was the pinned lerobot_v30.py.
The runtime used the installed PyPI openarm-dataset==0.4.0 core, whose dataset.py differed from that pinned snapshot. This was a mixed-source execution, not a full pinned-checkout validation. It ran on a Colab CPU with Python 3.13.18, openarm-control==0.5.0 and pyarrow==23.0.1; the earlier Windows Python 3.9 runner only retrieved files.
The export selected state=qpos, fps=30, one 90-frame episode, camera_names=[] and smoothing_cutoff=0. The genuine source Parquet streams were fetched; no synthetic fallback was used.
Both successful exports recorded action and observation.state with these names:
0 right_joint1.pos 8 left_joint1.pos
1 right_joint2.pos 9 left_joint2.pos
2 right_joint3.pos 10 left_joint3.pos
3 right_joint4.pos 11 left_joint4.pos
4 right_joint5.pos 12 left_joint5.pos
5 right_joint6.pos 13 left_joint6.pos
6 right_joint7.pos 14 left_joint7.pos
7 right_gripper.pos 15 left_gripper.pos
16 elevation.pos # with lifter only
In the 17-component export, elevation.pos was the sole added terminal name. The experiment inspected the resulting meta/info.json and sampled the first two rows of the exported Parquet, verifying lengths and equality of the first 16 arm entries between successful cases. It did not independently compare all 90 rows in a second execution environment. Also, the fact that action and state had equal dimensions here is a property of this tested mapping, not a general rule for mobile manipulation datasets.
The third case deliberately retained the lifter declaration in metadata.yaml while removing both obs/lifter/elevation.parquet and action/lifter/elevation.parquet from a local test copy. The helper enumerated expected keys from metadata, then raised KeyError: 'lifter/elevation' while assembling a stream that did not exist. This is an informative consistency test; it must not be described as an observed failure of your build. For real recordings, the options would be to require the declared lift data, make a consistent arms-only variant, or implement a clearly documented missing-data policy. Mixing 16D and 17D episodes without a feature contract would be difficult to interpret.
The OpenArm native format describes 8-element qpos vectors per arm (seven joints plus gripper), and optional obs/lifter/elevation.parquet and action/lifter/elevation.parquet with one lift value and a source timestamp. Its OpenArm Cell elevation is in mm relative to the bottom. The v3 export in this test named it elevation.pos but provided no corresponding unit annotation in meta/info.json, and the pinned helper did not make an explicit mm-to-m conversion. Thus a dataset should preserve a separate unit/normalization manifest, and DOLD-specific units must be checked against DOLD’s actual driver.
The exporter constructs fixed-rate episode timestamps using i / fps rather than preserving raw acquisition times as frame timestamps. This does not establish source synchronization. Cameras, motors, base, inference and NX timing were untested. The narrow result is a working nearby two-arm-plus-lift v3 format pattern; the OpenArm training guide documents its intended use.
Action semantics and the main architecture branches
Let the controller interface determine the action
For a mobile manipulator, the same floating-point vector can describe radically different things. I would specify each action component by name, coordinate frame, unit, target versus velocity mode, rate, limits and owner, then decide how to concatenate it for ACT. The measured observation should be defined separately.
Direct learned base control: A policy can emit body-frame forward velocity and yaw rate. The Jazzy diff_drive_controller uses these two twist components, not lateral twist; the STM32 contract still needs verification. Base odometry and localization can remain observations. Coordinated base/arm behavior is a possible research branch, but it does not override the published restriction on lift motion.
Nav2-owned travel: A NavigateToPose-style goal belongs to a task/phase layer. Stop the base, move the lift only with arms in the specified safe pose, then hold it during the arm/gripper ACT phase. Base pose, lift height and context can remain observations. This baseline is simpler to audit without discarding whole-body recordings.
Cartesian or whole-body goals: A policy could instead output wrist poses/gripper state and request motion through MoveIt Servo or another whole-body solver. That requires a declared pose frame, valid transforms at the current lift elevation, reachability and a documented solver/controller interface. It is not equivalent to renaming the joint-position vector. MoveIt Servo is one relevant interface, not an automatic codec for an arbitrary learned action.
One illustrative direct-control vector is:
right joints 1..7 + right gripper 8
left joints 1..7 + left gripper 8
lift target height, if position-based 1
base forward velocity and yaw rate 2
--
total 19
That example mixes positions and velocities; units and normalization must stay group-specific. Measured lift height is an observation, while the lift action could be a height target, velocity or one of the project’s calibrated stops. The published lift-control decision record calls for the base to be stopped and arms safely stowed during lift motion. A policy spanning multiple phases could use a phase label/action mask, rather than assuming every predicted coordinate may execute concurrently. Do not label velocity .pos merely to match Enactic’s export. Gripper conventions, MoveIt/policy ownership, and mode transitions likewise belong in the recorded contract.
The cited precedents are useful for comparing representations, but the actual OpenAMRobot command contract should come from its own controllers and demonstrations, not another project’s vector length.
ROS 2 demonstrations and a small LeRobotDataset v3 export
Record once; make the policy-specific dataset deliberately
I see two workable routes, rather than one required integration:
- Bag first: record ROS 2 topics with their original timestamps and episode/task markers, then map a bounded subset into named LeRobotDataset features at the selected FPS. This is convenient for replaying, auditing and changing the action definition later.
- LeRobot first: implement a ROS-backed LeRobot
Robot/teleoperator that declares its features and records through the LeRobot loop. This can simplify online record/playback if the driver, Python environment and scheduling already fit the target stack.
A hybrid is possible: raw rosbag2/MCAP for provenance plus LeRobot episodes for training. I would lean toward this when the hardware interfaces are still evolving. The LeRobot ROS 2 integration RFC #4368 surveys related architectures. In issue #3139, maintainer guidance for rosbag2/MCAP conversion was to explicitly map topics to features and align frames using the dataset API; closing that issue did not establish a universal ready-made importer for OpenAMRobot.
For example, the mapping could be specified in a little table before any exporter code is written. The following names are illustrative, not verified OpenAMRobot topic names:
| Source semantics |
Dataset destination |
Notes |
Right/left measured JointState |
observation.state arm components |
Preserve joint order and gripper convention |
| Requested arm/gripper setpoints |
action arm components, if they are the policy’s target |
Do not silently substitute measurements |
| Measured lift elevation |
Observation lift feature |
Source unit, home, stamp, validity and age |
| Requested lift height or velocity |
Action lift feature in the allowed control phase |
DOLD mode, limits, stopped base and safe arm pose |
Requested Twist.linear.x / angular.z |
Two base action components if directly policy-owned |
m/s, rad/s, body frame |
| Measured odometry/pose/twist |
Observation base features |
Frames and localization quality when needed |
| Camera frames |
Named image observations |
Device capture stamp, calibration, resolution |
| Nav2 goal and active-controller phase |
Task/context or sidecar metadata |
Not equivalent to two continuous base velocities |
The source-topic-to-feature mapping table is a conceptual contract, not executable configuration. If Nav2 owns the wheels, omit continuous base action from that policy while retaining base logs. If the lift accepts velocity, record velocity with its real unit. Action and observation need not have matching dimensions; the 17/17 fixture did so only because of its converter.
Alignment is more than choosing fps=30
There are at least three different times in this pipeline:
- Acquisition time: when the camera exposed the frame or a sensor measured the state. Firmware may use its own clock.
- ROS transport/record time: when middleware sent/received the message or the bag recorded it. This can include queueing delay and is not necessarily acquisition time.
- Policy frame time: the regular output timeline after sampling, interpolation or hold. It may use
i/fps even when the source timestamps were irregular.
The ROS 2 time design distinguishes system, steady and ROS/simulated time; /clock and use_sim_time matter in Gazebo. rosbag2 has its own recording/playback clock and QoS concerns. The LeRobotDataset v3 format describes stored frames, not a guarantee that upstream cameras and motors were synchronized correctly. Recording both the raw source stamps and the output sampling policy makes latency and image freshness diagnosable.
For a five-to-ten-second episode, I would check only the high-information items: exact feature order and shape in meta/info.json against the actual Parquet row; lift present when declared; source units; per-topic first/last stamps and inter-sample gaps; camera frame age at each policy tick; action versus feedback; and any dropped/held/interpolated samples. This catches a surprising amount without requiring a trained model or an instrumented physical robot.
What counts as an action after controller limits?
The LeRobot Robot API describes send_action() returning what was actually sent, which may be limited. LeRobot issue #4679 documents a recorder path that retained pre-send requested action rather than the returned action. This is a version/code-path-specific concern, not something observed in your robot or the Enactic conversion test.
Which of the first two is the learning target depends on the intended policy interface; the important thing is to avoid labeling one as the other. If the target happens to use Jazzy diff_drive_controller, its optional cmd_vel_out is a possible view of limited base commands. Other embedded/arm/lift controllers may expose different feedback.
For code to inspect before writing a new bridge, ngres/leros2 offers community rosbag conversion with explicit FPS/clock choices, while sacovo/lerobot_ros illustrates ROS-node-based recording and control. These are community options with version and interface constraints, not adapters I have verified on OpenAMRobot. Current LeRobot Dataset API and the installed tag should be matched when implementing create/add_frame/save_episode/finalize. The first-party OpenArm v3 exporter is another starting point for the arm/lift subset, not a complete mobile-base ROS adapter.
Jetson Orin NX: what would make a co-run result meaningful
Measure the camera-to-command loop, not model FPS alone
The proposed architecture already separates high-level Orin computation from an STM32H723/micro-ROS base-control layer. That can be helpful, because ACT and MoveIt planning do not need to execute inside each motor servo cycle. It does not, by itself, establish that the whole high-level pipeline meets the action deadline.
For a useful first answer about one Orin NX, I would compare the following with the same actual checkpoint, cameras, resolution, model precision, action horizon and desired policy rate:
| Trial |
Workload |
Main diagnostic |
| A: vision/policy baseline |
Camera acquisition, decode, preprocess, inference, action postprocess and command serialization |
Does this input/model fit the NX without competition? |
| B: co-run |
Same pipeline with relevant ROS 2 graph, MoveIt/ros2_control and recording active |
Additional latency, memory and scheduling pressure |
| C: optional |
B plus Nav2/localization and representative base activity |
Only necessary if the task requires simultaneous navigation |
Track at least:
- End-to-end age of a command: exposure/capture timestamp through policy queue, preprocessing, inference, decoding, ROS publishing and controller acceptance where observable.
- Tail behavior:
p50 and p95 latency, p99 if sample count supports it; missed action/refill deadlines and whether stalls cluster.
- Resources: peak RAM/GPU memory and CPU utilization, plus sustained power, clock and thermal readings from
tegrastats or comparable tools.
- Configuration: NX 8GB/16GB variant, JetPack/L4T, ROS 2 and LeRobot revisions, camera count and image sizes, checkpoint and chunk/temporal-ensembling settings.
The desired deadline depends on the task and whether the policy emits one action or a chunk. A model can have acceptable average inference time but still generate commands from stale images after a camera or planner stall. Conversely, a controller can continue at its own rate while policy action chunks refill, provided the hold/stop behavior is explicitly supported. I would not invent a universal millisecond threshold without a chosen control contract.
If the co-run has comfortable margin, local ACT is a straightforward baseline with fewer network failure modes. If it does not, choose the smallest remedy that the trace supports: fewer/lower-resolution policy image streams, a smaller validated checkpoint, lower policy query rate with bounded chunk playback, better scheduling of MoveIt planning, or task-level separation between navigation and manipulation. Each has a tradeoff in observation detail, responsiveness, or task coverage. The larger future VLA should be evaluated separately rather than assumed to inherit the ACT result.
Offboard inference is an option, not a transparent substitution
The current LeRobot main remote-inference guide describes robot-side rollout and a policy server, but its rollout contract requires a robot driver that can hold the last valid position target while waiting for the next actions. The companion hardware support table explicitly excludes standard mobile-base velocity drivers by default; OpenArm/bimanual support depends on configuration. This is an execution-contract restriction, not proof that OpenAMRobot can never use remote computation.
A conditional design would keep the velocity base under a local, timeout-aware controller and send only compatible arm position targets through a supported remote path, or implement and test a custom whole-body remote control/hold scheme. Network failure, stale action chunks and ownership transitions must still have defined behavior.
Version matters: current main uses lerobot[remote], lerobot-policy-server and lerobot-rollout rather than older lerobot[async]/PolicyServer examples. The installed tag decides which commands apply. Real-Time Chunking has a distinct contract; ACT temporal ensembling and driver hold support must be checked before a remote rollout.
Controller boundaries, lift-dependent frames, and stop behavior
How I would connect the policy to the existing stack
The author’s combined model and simulation update already includes the planned lift joint; the upper-body software repository provides the project-specific foundation. I would preserve that model and make the learned-policy command authority explicit at runtime.
| Subsystem |
Possible command owner |
Transition/timeout to define |
| Base |
Nav2, supervised teleop, or learned direct twist |
Who may send velocity; how does stale velocity stop? |
| Each arm |
MoveIt trajectory, teleop, or learned joint targets |
Previous trajectory cancellation, mode switch, valid hold |
| Lift |
Planned motion, supervised driver, or learned command |
Homing reference, allowed mode, limits, hold/brake |
| Grippers |
Teleop or learned command |
Position/force convention, what persists on pause |
In a ros2_control configuration, the Jazzy controller-manager documentation explains controller lifecycle, command-resource ownership and mode-switching interfaces. I am not claiming to know how your STM32 firmware and supervisors currently arbitrate those commands. The design aim is simply that MoveIt and an ACT stream do not both believe they own the same joint, and Nav2 and a direct learned cmd_vel stream do not compete in the same phase.
Stopping the base and holding an arm or lift are different operations. A base velocity command should expire and lead to a locally enforced stop; a position-controlled arm may hold the last valid target; the lift’s brake/hold/mode behavior is actuator-specific. Jazzy diff_drive_controller documents a cmd_vel_timeout, but that does not prove the OpenAMRobot custom base path implements that exact setting. A lost model process or remote connection must not be mistaken for a fresh velocity command.
The lift also changes geometry: shoulder transforms, reachability and potentially head-camera frames. Cartesian targets need the correct lift-dependent transform from the integrated model and TF; Enactic’s OpenArm pose representation does not automatically account for any external lift. More specifically, the OpenAMRobot upper-body firmware decision record calls for publishing lift position with validity and age, and treating stale position as invalidating the head-camera transform. It also restricts lift motion to a stopped base and stowed/carry-safe arms. The README explicitly says these are planned decisions, not implemented safety firmware; this is not a diagnosed TF fault.
Your published architecture already discusses E-stops, brakes, a safety layer and sensor limits, so I would not suggest those are missing. Software collision limiting, such as Nav2 Collision Monitor, is useful in an appropriately ordered velocity path but is not a hardware safety certification and does not cover all lift/arm hazards. The dataset/ACT test, ROS controller test and later physical acceptance test should be described as different evidence.
Relevant examples and source references
What I would borrow, and what I would not
- Dual arms with learned base velocity: Mobile ALOHA supports the formulation of direct mobile actions alongside bimanual manipulation, but has no lift. Its paper-level vector and any individual converted dataset’s schema are not necessarily identical.
- Bimanual platform with lift: YOR is a more physically similar reference, but its omnidirectional base, pose-oriented actions, and use of remote computation differ from this differential-drive local-NX proposal. Its 19D label should not dictate this robot’s 19D example.
- Staged reachability/visibility: MoMaGen is relevant to selecting base/torso positions before manipulation. MoMa-Teleop illustrates separating some base control from the human’s end-effector motion task. These can reduce demonstration burden without precluding a later whole-body policy.
- Broader task framing: BEHAVIOR Robot Suite is useful background for bimanual wheeled tasks, not validation of the specific DOLD lift or target compute budget.
The primary OpenArm Dataset, LeRobot, ROS 2 and controller references are linked at the points above where their specific claims matter. None is a verified complete OpenAMRobot mobile-manipulation adapter.
A practical next step
My default would be Nav2/teleop travel, lift adjustment under the published safe-pose conditions, then fixed-lift ACT manipulation, preserving the raw base/lift records for later learning. The upstream 17-feature result is a format reference, not a DOLD test. For compute, one matching pair of NX policy-only/co-run traces is the next useful evidence. If a short recording or feature manifest exists, I would be happy to check its names, units and phase conventions. I hope this supports your wider goal without prescribing one architecture.