Hmm… I think you’d probably get a more definitive answer by asking in the LeRobot Discord (linked from lerobot (LeRobot)), but for now:
I think the main distinction here is that the 79-D states vector and LeRobot’s 8-D observation.state are probably two different kinds of state, rather than the latter being a subset or dimensionality reduction of the former.
In the standard LIBERO data path, the HDF5 states array is used as a flattened MuJoCo simulator state. LIBERO’s own create_dataset.py restores that simulator state, replays the trajectory, and separately constructs proprioceptive observations such as:
ee_state = np.hstack(
(
obs["robot0_eef_pos"], # 3
T.quat2axisangle(obs["robot0_eef_quat"]), # 3
)
)
gripper_state = obs["robot0_gripper_qpos"] # 2
That gives:
3 (EEF position)
+ 3 (EEF orientation as axis-angle)
+ 2 (gripper qpos)
= 8
This is also the representation currently documented by LeRobot’s LIBERO integration: observation.state is 8-D EEF position + axis-angle orientation + gripper qpos.
So, if your custom HDF5 follows the usual LIBERO layout, I would not start by selecting eight indices out of the 79-D vector. I would first check whether the HDF5 already contains the proprio fields used to construct the 8-D state.
A practical decision tree would be:
Does demo_0/obs contain ee_states (6) and gripper_states (2)?
│
├─ Yes
│ └─ observation.state = concat(ee_states, gripper_states)
│
└─ No
│
├─ Do you have eef_pos + eef_quat + gripper_qpos?
│ └─ Build 3 + quat→axis-angle(3) + 2
│
└─ Only the simulator states are available
│
├─ Can the LIBERO environment be reconstructed?
│ └─ Restore each simulator state and regenerate observations
│
└─ The 79-D layout is custom / unknown
└─ Check the exporter/schema first; don't guess fixed indices
The any4lerobot/libero2lerobot converter is a useful concrete example here: its LeRobot state is 8-D, while it also preserves richer native LIBERO state information separately. In other words, making a training-compatible observation.state does not require throwing the original simulator/robot state away.
For features / meta/info.json
I would treat the feature schema as the output contract of the conversion, not as the thing that determines how the 79-D source vector should be interpreted.
Conceptually:
source HDF5
↓
interpret source semantics
↓
construct observation.state / action / images
↓
declare those outputs in LeRobot `features`
↓
write the LeRobot dataset
With current LeRobot v3, I would normally define the semantic features when creating the dataset rather than manually constructing the whole meta/info.json first.
A simplified skeleton is roughly:
features = {
"observation.state": {
"dtype": "float32",
"shape": (8,),
"names": {
"motors": [
"x", "y", "z",
"axis_angle1", "axis_angle2", "axis_angle3",
"gripper_left", "gripper_right",
]
},
},
"action": {
"dtype": "float32",
"shape": (7,),
# Add names that match your actual action semantics.
},
"observation.images.image": {
"dtype": "video", # or "image", depending on your choice
"shape": (HEIGHT, WIDTH, 3),
"names": ["height", "width", "rgb"],
},
"observation.images.image2": {
"dtype": "video",
"shape": (HEIGHT, WIDTH, 3),
"names": ["height", "width", "rgb"],
},
}
dataset = LeRobotDataset.create(
repo_id="your-name/your-dataset",
fps=SOURCE_FPS,
features=features,
# root=...,
)
for episode in episodes:
for frame in episode:
dataset.add_frame(
{
"observation.state": state8,
"action": action7,
"observation.images.image": main_image,
"observation.images.image2": wrist_image,
"task": task_description,
}
)
dataset.save_episode()
dataset.finalize()
The exact camera keys and shapes should match your source data and the policy you intend to train.
Current LeRobot’s writer automatically handles standard bookkeeping such as frame indices and timestamps; add_frame() computes timestamp from frame_index / fps. Also, with v3 you should call finalize() after recording, because it closes/flushed the incremental Parquet writers and finalizes metadata.
So I would not copy another converter’s FEATURES constant verbatim. Use it as a template, but derive:
- state shape and meaning from your source observations,
- action shape and meaning from your controller/exporter,
- image keys/shapes from your actual cameras,
- FPS from the rate represented by your data.
Before converting everything, I would do only one or two episodes, call finalize(), reload the resulting LeRobotDataset, and inspect a few samples. That usually catches schema mistakes much more cheaply than converting the whole dataset first.
For example, verify at least:
observation.state -> (8,), float32
action -> (7,), float32
observation.images.* -> expected camera keys/shapes
timestamp -> consistent with the source FPS
task -> present
If you are targeting a particular pretrained LeRobot policy/checkpoint rather than training completely from scratch, I would additionally verify that its expected input feature names as well as shapes match your dataset.
Why I would not slice the 79-D state directly
LIBERO’s terminology can be confusing because several different representations can reasonably be called “state”.
There are at least three useful layers here:
1. simulator state
MuJoCo state sufficient to restore the simulator
2. robot/environment observations
eef position, quaternion, joint positions, gripper qpos, images, ...
3. policy representation
the subset / representation exposed to the learning policy
The 8-D state used by current LeRobot’s LIBERO integration is layer 3.
The states stored in standard LIBERO demonstration HDF5 files are much closer to layer 1. LIBERO’s own script literally describes them as flattened MuJoCo states and restores them using set_state_from_flattened().
This distinction also explains why it is risky to infer semantics only from the number of dimensions. Other LIBERO users have encountered simulator-state dimensions such as 47 or 92 as well; see for example:
Those issues are useful evidence that this distinction is easy to run into, but I would use the actual LIBERO source code rather than the issue discussions as the authoritative description.
Also, 8-D is not a universal definition of “the LIBERO state”. LIBERO itself can train using other low-dimensional observations; for example its default data configuration uses joint_states + gripper_states. The 8-D EEF representation is specifically the representation currently used by the LeRobot LIBERO interface.
So if you need the original 79-D simulator state for analysis, reproducibility, or future policies, you can keep it as an additional feature rather than forcing it into observation.state.
If your HDF5 really has only simulator states
If the useful proprio fields are missing, you may still be able to regenerate them rather than reverse-engineering indices from the flattened state.
LIBERO exposes regenerate_obs_from_state(), which essentially does:
set simulator state
→ sim.forward()
→ refresh observables
→ obtain observations
That provides a fairly clean route if you have enough metadata to reconstruct the original environment.
LIBERO’s dataset processing code also demonstrates another route: restore the initial state and replay the recorded actions. Importantly, the script compares the replayed MuJoCo state with the recorded next state and warns if they diverge.
So for a custom dataset I would prefer:
A. direct state → observation regeneration
when the environment can be reconstructed cleanly, or:
B. short action replay + divergence check
before relying on a full replay.
This matters because simulator/environment/library-version differences can make action replay drift even when the nominal action sequence is the same.
If neither the proprio observations nor sufficient environment/export metadata are available, then the right mapping depends on what your custom exporter put into those 79 values. In that case there isn’t a safe generic "indices [a:b]" answer—the exporter or simulator-state definition is the thing to inspect.
A very small diagnostic is usually enough. You do not need to upload the whole dataset; even the keys/shapes/attributes for one episode would distinguish most branches:
data/demo_0/states
data/demo_0/actions
data/demo_0/obs/*
data/demo_0 attrs
data attrs
In particular I would look for:
ee_states
gripper_states
robot0_eef_pos
robot0_eef_quat
robot0_gripper_qpos
model_file
env_args / BDDL-related metadata
A few secondary compatibility checks
Once the state mapping is resolved, there are several separate things worth checking before calling the dataset fully training-compatible.
1. Action semantics
A 7-D action vector having the expected shape does not guarantee that it has the expected meaning.
For LIBERO-like EEF control, questions include:
translation: delta or absolute?
rotation: axis-angle / Euler / something else?
reference frame: world / base / end-effector?
gripper: sign/range/open-close convention?
Current LeRobot’s LIBERO documentation describes a 7-D continuous action and supports relative/absolute control modes. Your converted dataset should preserve the semantics of the controller that generated the demonstrations, or explicitly transform them to the semantics expected by the target policy.
This is another reason the existing libero2lerobot converter is useful as a reference implementation, but not necessarily something to copy field-for-field for a custom exporter.
2. FPS / timestamps
Do not infer FPS solely from another LIBERO conversion.
LeRobot uses the dataset FPS to generate timestamps, and different public LIBERO-derived datasets/converters have used different rates. Use the rate represented by your own recorded trajectories.
A simple check is:
episode duration ≈ number_of_frames / fps
and then compare that to the original recording/control timing if available.
3. Camera orientation and preprocessing
Some existing LIBERO/VLA conversion pipelines rotate or otherwise preprocess images. Those steps are often specific to the lineage of that dataset or evaluation pipeline.
For a custom dataset, visualize a few converted frames before adopting such transformations.
4. Observation/action alignment
If your exporter is custom, also check what one row means temporally:
(state_t, action_t)
versus
(action_t, resulting_state_{t+1})
LIBERO’s own post-processing/replay path performs an action and then obtains the resulting observations, so this is worth verifying from the exporter rather than assuming that identical array indices necessarily describe the same instant.
5. Policy compatibility is more than file-format compatibility
A valid LeRobot v3 dataset can contain many legitimate state representations. Therefore:
valid LeRobot dataset
does not automatically imply:
compatible with every LeRobot policy/checkpoint
If your final goal is a specific policy, compare its expected:
feature keys
state dimension/semantics
action dimension/semantics
camera keys
normalization expectations
against the dataset after conversion.
So for your specific three questions, my current answer would be:
-
State mapping: probably do not select eight elements from the 79-D simulator state. If obs/ee_states and obs/gripper_states exist, concatenating those is the most direct candidate. Otherwise reconstruct those observations from the raw EEF/gripper fields or from the simulator.
-
Features configuration: define features according to the converted semantic outputs, e.g. an 8-D float observation.state, 7-D action, and your actual image features. Let the current LeRobot writer generate the corresponding v3 metadata rather than treating meta/info.json as the starting point.
-
Known pattern/script: any4lerobot/libero2lerobot is probably the closest reference implementation for LIBERO-style HDF5 → LeRobot v3. I would compare its source mapping with your HDF5 keys, rather than just copying its FEATURES constant.
If your file already has obs/ee_states and obs/gripper_states, you may actually be quite close—the 79-D vector might not need to participate in the LeRobot policy state conversion at all.