How to use the `lerobot_eval` command

Beginners
1

I am using SO101 for imitation learning. The model I am using is SmolVLA. When I used the ‘lerobot_eval’ command to evaluate the training data, the camera resolution automatically changed from 640×480 to 1280×800, and I am now unable to adjust the resolution. The FPS value has also become incorrect.
The command I used is as follows.
What could be the cause of this?

lerobot-eval
–env.type=pusht
–eval.n_episodes=10
–eval.batch_size=10
–policy.path=outputs/train/smolvla_base_3camera_dataset/smolvla_20260602_ep50_random_001_filtered/checkpoints/last/pretrained_model
–policy.device=cuda
–policy.use_amp=false

2

Since this is Physical AI, I would still recommend asking in the LeRobot Discord for a definitive answer. But at first glance, it looks like there may be a few common traps here:

Short version

I would not debug this as a camera-resolution bug first.

The first thing I would check is whether you are mixing two different LeRobot paths:

  1. Simulation/environment evaluation
    Example shape: lerobot-eval --env.type=pusht

  2. Real SO101 + SmolVLA + physical cameras
    Example shape: lerobot-rollout or older lerobot-record, with explicit --robot.type, --robot.port, --robot.cameras, and --policy.path

Your description sounds like a real SO101 + SmolVLA + camera setup, but the command you showed uses:

--env.type=pusht

That is the first thing I would separate.

Decision tree

Question If yes If no
Are you intentionally evaluating in the PushT simulation environment? lerobot-eval --env.type=pusht may be the right direction. Debug the PushT env / policy / eval config. Go to the next question.
Are you trying to run a policy on a real SO101 with physical cameras? Do not start from --env.type=pusht. Use the real-robot rollout/record path and pass camera settings through --robot.cameras. The answer depends on the actual environment you want to evaluate.
Are you already using the real-robot path and explicitly passing camera width/height/fps? Then debug camera backend / device identity / FPS / USB bandwidth / V4L2 vs FFMPEG. First fix the command path and camera config.
Does the policy expect 3 cameras, but the robot/dataset provides different camera names/count/order? Check SmolVLA image features, dataset metadata, camera keys, and rename maps. Then focus on the camera backend/device layer.

Why --env.type=pusht looks suspicious here

LeRobot examples often use lerobot-eval with an environment such as pusht or libero when evaluating a policy in a benchmark/simulation-style environment.

For example, the LeRobot README shows evaluation with an environment argument like:

lerobot-eval \
  --policy.path=lerobot/pi0_libero_finetuned \
  --env.type=libero \
  --env.task=libero_object \
  --eval.n_episodes=10

See: LeRobot README — Inference & Evaluation

There are also PushT examples/tests using:

--env.type=pusht

See: lerobot_eval.py

That does not look like the same layer as configuring real SO101 cameras.

For a real robot, the important object is not only an env; it is the robot and its camera config.

Real SO101 / physical camera path

For real robot inference, current LeRobot docs point to lerobot-rollout.

See: LeRobot Policy Deployment / lerobot-rollout

The cheat sheet also says:

Inference means running the trained policy/model on a robot. For that we use lerobot-rollout.

It also notes that camera configuration needs to match what was used when collecting the dataset.

See: LeRobot Cheat Sheet

So for a real SO101 setup, I would expect the command to be closer to this shape:

lerobot-rollout \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=<your_robot_id> \
  --robot.cameras='{
    front: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30},
    wrist: {type: opencv, index_or_path: 2, width: 640, height: 480, fps: 30}
  }' \
  --policy.path=<your_policy_path_or_hub_repo> \
  --task="<your task>"

Exact arguments may differ by LeRobot version, so check your installed version first.

Version check

Before changing too much, I would collect:

lerobot-info
pip show lerobot
lerobot-eval --help
lerobot-rollout --help
lerobot-record --help

This matters because LeRobot CLI names and argument names have changed across versions. Some older examples use record.py, control_robot.py, or lerobot-record where newer docs use lerobot-rollout.

For example, there are real SO101 + SmolVLA issue examples using lerobot-record with explicit robot and camera config:

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{camera1: {type: opencv, index_or_path: 0, width: 640, height: 480, fps: 30}, ...}" \
  --policy.path=...

See: Inference lerobot/smolvla_base with SO-101 failed #2210

So I would not assume that a command from one LeRobot version maps cleanly to another version.

If the real-robot path is already correct, then check camera backend

After confirming that you are using the real SO101 robot path, then the camera issue becomes very plausible.

There is a relevant LeRobot issue where Linux/OpenCV camera settings did not apply correctly with the default backend:

  • backend=ANY auto-selected FFMPEG
  • requested fourcc="MJPG" did not apply
  • width/height/fps setting failed or behaved incorrectly
  • forcing backend=V4L2 made the same settings work

See: OpenCVCameraConfig backend=ANY vs V4L2 issue #3198

There is also a Hugging Face forum thread from an SO101 user where forcing Video4Linux2 was used as a temporary fix for camera backend problems:

See: Lerobot Camera Backend Issues

So if you are on Linux with USB/OpenCV cameras, the camera backend is a strong second-level suspect.

Camera discovery

I would also run:

lerobot-find-cameras opencv

LeRobot’s camera docs say OpenCV and RealSense cameras support auto-discovery, and that identifiers may change after rebooting or re-plugging cameras.

See: LeRobot Cameras docs

Please check:

Item Why it matters
/dev/video* path or camera index It may change after unplug/reboot.
backend API FFMPEG vs V4L2 can matter on Linux.
default stream profile The camera may default to unexpected width/FPS.
supported modes Some resolutions only work at lower FPS.
fourcc / MJPG support MJPEG is often needed for higher resolution/FPS USB camera modes.

SmolVLA-specific trap: camera keys/count/order

SmolVLA uses multiple camera views plus robot state and language instruction.

See: SmolVLA docs

So after fixing the command path and camera backend, another possible issue is that the policy expects different image features than your robot/dataset provides.

Things to check:

policy expected image keys
dataset image keys
robot camera names
camera order
number of cameras
image resolution
FPS
normalization/statistics

There are related issues where users hit camera feature/name/count problems with SmolVLA or SO101:

This may not be the first cause of your resolution/FPS symptom, but it is worth checking if the policy runs but behaves badly.

What I would ask you to post

For a useful LeRobot Discord/GitHub/forum question, I would include:

lerobot-info
pip show lerobot
lerobot-eval --help
lerobot-rollout --help
lerobot-record --help
lerobot-find-cameras opencv

Also include:

OS:
LeRobot version:
Install method:
Robot type:
Camera model(s):
Camera backend:
Full command:
Policy path:
Dataset repo_id:
Number of cameras used during training:
Number of cameras connected during evaluation:
Expected width/height/fps:
Actual width/height/fps:

Practical triage order

I would debug in this order:

  1. Decide whether this is PushT simulation evaluation or real SO101 robot rollout.
  2. If it is real SO101, switch to the real-robot path rather than --env.type=pusht.
  3. Pass cameras explicitly through --robot.cameras.
  4. Run lerobot-find-cameras opencv.
  5. If width/FPS still changes, check OpenCV backend: FFMPEG vs V4L2, MJPG, supported modes, and USB bandwidth.
  6. If the policy runs but behaves badly, check SmolVLA camera keys/count/order and dataset metadata.

My best guess

Based only on your command and description, the most likely first issue is not SmolVLA itself and not necessarily OpenCV yet.

The most likely first issue is:

The command is using a PushT environment-evaluation path,
while the problem description sounds like real SO101 robot-camera evaluation.

After that is fixed, then the OpenCV/V4L2/FPS issue becomes the next thing to check.

3

First, when I ran the eval command, I got an error indicating that some dependencies were missing. I don’t remember the exact details of the error—sorry about that.

After that, the eval command became unusable.

Then I tried using the record and train commands, and I noticed that the camera settings were incorrect. Even after changing the resolution setting to “640×480,” the changes were not applied.

4

Hmm… In that case, seems this may not be a single issue:

Short version

I would split this into two different layers:

  1. The lerobot-eval problem

    • This may be an installation / optional-dependency issue.
    • lerobot-eval can require extras such as lerobot[evaluation], the policy extra, and, for PushT simulation, the pusht environment extra.

  2. The camera resolution/FPS problem in record / train

    • This sounds like a real camera configuration/backend problem.
    • If 640x480 is not being applied even in record / train, I would not treat that as only an eval problem.
    • On Linux/OpenCV cameras, I would specifically check whether OpenCV is using FFMPEG instead of V4L2, and whether MJPG / width / height / FPS settings are actually being applied.

So my current guess is:

Issue A: `lerobot-eval` became unusable because the required optional dependencies were missing or the environment got into a partially-installed state.

Issue B: the camera settings not applying in `record` / `train` is probably a separate real-camera backend/configuration issue.

Why I would separate these

Your latest detail changes the triage.

You said:

First, when I ran the eval command, I got an error indicating that some dependencies were missing.
After that, the eval command became unusable.

Then I tried using the record and train commands, and I noticed that the camera settings were incorrect.
Even after changing the resolution setting to 640x480, the changes were not applied.

That sounds less like one single SmolVLA problem, and more like two failures that happened close together:

Symptom Likely layer First thing to check
lerobot-eval says dependencies are missing install / optional extras / environment state lerobot[evaluation], lerobot[pusht], policy extras, clean environment
record / train camera settings are wrong real camera config / OpenCV backend / device mode lerobot-find-cameras opencv, V4L2 vs FFMPEG, MJPG, supported camera modes
640x480 is configured but not applied camera backend or camera mode negotiation v4l2-ctl, OpenCV backend, fourcc, actual stream profile
SmolVLA runs but behaves badly policy/dataset/robot observation mismatch camera keys, camera count, camera order, dataset metadata

Layer 1: lerobot-eval probably needs optional extras

The lerobot-eval script itself says it evaluates a policy on an environment, and its docstring lists requirements like:

pip install 'lerobot[evaluation]'

plus the policy extra, and the environment extra such as:

lerobot[pusht]

when evaluating in simulation.

Reference:

The installation docs also say that the base PyPI package:

pip install lerobot

installs only the default dependencies, and that additional functionality uses extras such as:

pip install 'lerobot[all]'
pip install 'lerobot[aloha,pusht]'
pip install 'lerobot[feetech]'

Reference:

So if the exact missing-dependency error is gone, I would still suspect this area first.

Minimal checks for the eval side

which python
python --version
which lerobot-eval
lerobot-info
pip show lerobot
pip freeze | grep -E "lerobot|gym|gymnasium|pusht|torch|opencv|av|ffmpeg"
lerobot-eval --help

If you are trying to evaluate PushT specifically, I would also check whether the PushT extra is installed in the same environment where lerobot-eval is being run.

A clean reinstall may be easier than repairing a partially modified environment:

conda create -n lerobot-clean python=3.12 -y
conda activate lerobot-clean

# choose the install form that matches the docs/version you are using
pip install 'lerobot[pusht]'
# or, from source:
# git clone https://github.com/huggingface/lerobot.git
# cd lerobot
# pip install -e ".[pusht]"

If lerobot-eval still fails, the most useful thing to post is the full error traceback, not just the summary.

Layer 2: simulation eval vs real robot rollout

I would also keep the command-path distinction explicit.

There are at least three different-looking paths:

Goal Typical command family Main config object
PushT / simulation benchmark evaluation lerobot-eval --env.type=pusht environment config
Real SO101 policy deployment, current docs lerobot-rollout robot config + policy config
Real SO101 policy execution in older examples/releases lerobot-record --policy.path ... robot config + camera config + policy config

--env.type=pusht points toward the simulation/environment-evaluation path. That can be valid if you are evaluating a PushT policy in the PushT environment.

But if your real goal is:

SO101 + SmolVLA + real physical cameras

then I would not use --env.type=pusht as the main mental model.

Current docs describe lerobot-rollout as the CLI for deploying trained policies on real robots:

The cheat sheet also says that inference means running the trained policy on a robot, and notes that older v0.5.1 workflows use lerobot-record instead of lerobot-rollout:

So I would first decide which branch you are in:

A. I want PushT simulation evaluation
   -> use `lerobot-eval --env.type=pusht`
   -> install the evaluation + pusht extras
   -> debug environment/policy compatibility

B. I want real SO101 robot evaluation
   -> use the real-robot path
   -> current docs: `lerobot-rollout`
   -> older release examples: `lerobot-record --policy.path ...`
   -> pass cameras through `--robot.cameras`

C. I am training/recording and the camera setting is wrong
   -> this is probably not an `eval` issue
   -> debug camera backend / camera mode / device path

Layer 3: camera resolution/FPS not applying

The camera part sounds like a separate issue because you said the wrong settings also appear in record and train.

For that, I would start with:

lerobot-find-cameras opencv

Then compare:

expected width/height/fps
actual width/height/fps
backend API
camera path or index
default stream profile

LeRobot camera docs:

If you are on Linux with USB/OpenCV cameras, I would also check the actual camera modes through V4L2:

v4l2-ctl --list-devices
v4l2-ctl --list-formats-ext -d /dev/video0
v4l2-ctl --list-formats-ext -d /dev/video2

Replace /dev/video0 and /dev/video2 with your actual camera devices.

Why V4L2 / FFMPEG / MJPG may matter

There is a very relevant LeRobot issue:

The reported pattern is very close to a “resolution/FPS setting does not apply” situation:

backend=ANY
-> OpenCV selects FFMPEG
-> fourcc="MJPG" does not apply
-> width/height/fps setting fails or is not reliably applied

backend=V4L2 + fourcc="MJPG"
-> the same requested camera settings work

There is also a related Hugging Face forum post from an SO101 user:

That user saw errors like:

OpenCVCamera(/dev/video0) failed to set fourcc=MJPG
OpenCVCamera(/dev/video0) failed to set capture_width=640

and temporarily fixed the issue by forcing OpenCV to use Video4Linux2:

cv2.VideoCapture(self.index_or_path, cv2.CAP_V4L2)

I am not saying this is definitely your exact bug, but it is a strong candidate if:

- you are on Linux
- you use OpenCV cameras
- `lerobot-find-cameras opencv` reports FFMPEG backend
- width/height/fps settings do not apply
- 640x480 works in another camera tool but not through LeRobot

Camera config shape to verify

For real SO101 camera configuration, I would expect something like this shape, adjusted to your installed LeRobot version:

lerobot-rollout \
  --strategy.type=base \
  --policy.path=<your_policy_path_or_hub_repo> \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.cameras="{ front: {type: opencv, index_or_path: /dev/video0, width: 640, height: 480, fps: 30}, wrist: {type: opencv, index_or_path: /dev/video2, width: 640, height: 480, fps: 30}}" \
  --task="<your_task>" \
  --duration=60

For older versions, the equivalent may use lerobot-record instead of lerobot-rollout.

The important point is not this exact command. The important point is:

real robot camera settings should be passed through the robot/camera config path,
not through `--env.type=pusht`.

SmolVLA-specific thing to keep in mind

If the camera settings are eventually applied correctly but the policy still behaves badly, then I would check a different layer:

Do the camera names/count/order match what the SmolVLA policy expects?

SmolVLA uses camera observations, robot state, and language instruction. So a mismatch in image keys can matter even when the camera itself works.

Reference:

Related issues/examples:

This is probably not the first explanation for “640x480 is not applied,” but it may become relevant after the camera backend/config issue is fixed.

What I would post next for debugging

If you ask in the LeRobot Discord or continue here, I would include this exact information:

OS:
LeRobot version:
Install method: pip / source / uv / conda
Python version:
Policy path:
Robot type:
Camera type: OpenCV / RealSense / other
Camera model:
Full command used for eval:
Full command used for record:
Full command used for train:
Expected resolution/FPS:
Actual resolution/FPS:

And these outputs:

lerobot-info
pip show lerobot
pip freeze | grep -E "lerobot|opencv|av|ffmpeg|torch|gym|gymnasium|pusht"
lerobot-eval --help
lerobot-record --help
lerobot-rollout --help
lerobot-find-cameras opencv
v4l2-ctl --list-devices
v4l2-ctl --list-formats-ext -d /dev/video0

If you have multiple cameras, also run:

v4l2-ctl --list-formats-ext -d /dev/video2
v4l2-ctl --list-formats-ext -d /dev/video4

using your actual devices.

Practical triage order

I would debug in this order:

Step Question Why
1 Is lerobot-eval failing because optional dependencies are missing? The original eval error was a dependency error.
2 Are you trying to evaluate PushT simulation or a real SO101 robot? --env.type=pusht and real robot rollout are different paths.
3 Which LeRobot version are you using? CLI names changed; current docs use lerobot-rollout, older examples may use lerobot-record.
4 Are camera settings explicitly passed through --robot.cameras? Real camera settings need to be in the robot/camera config.
5 What does lerobot-find-cameras opencv report? This shows actual camera backend and default profile.
6 Does V4L2 list 640x480 at the desired FPS? The camera may not support the requested mode in the current format/backend.
7 Is OpenCV using FFMPEG instead of V4L2? This is a known suspect on Linux/OpenCV webcam setups.
8 Do policy/dataset/robot camera keys match? SmolVLA can fail or behave badly if observation keys/count/order differ.

My best current guess

Based on the new details, I would not assume there is one root cause.

My best guess is:

1. `lerobot-eval` failed because the environment did not have the required optional dependencies for evaluation / PushT simulation.

2. The camera resolution/FPS problem in `record` / `train` is a separate real-camera backend/config issue, likely involving OpenCV backend selection, V4L2 vs FFMPEG, MJPG, or device-mode negotiation.

3. If the camera backend is fixed but the SmolVLA policy still behaves incorrectly, then check camera names/count/order and dataset-policy observation compatibility.

So I would first restore a clean LeRobot environment with the correct extras, then independently debug the camera mode with lerobot-find-cameras opencv and v4l2-ctl.