Genesis: a mobile manipulator, registers to VLA

Abstract

Genesis Bot is a mobile manipulator I keep improving to learn robotics end to end, from ESP32 registers to learned policies. It started as an Ackermann-steering RC car that I repurposed for robotics. It is now an omnidirectional base with two arms on a lift column. I use it to explore autonomous behavior. I am currently working on:

• A behavior tree orchestrator that combines classical planning with learned policies.
• End-to-end models such as VLAs.
• A custom multimodal keypoint-based model, which I am evaluating.

Introduction

This is a personal project. I set my own deadlines, which leaves time to explore, learn and take on problems I'm interested in, or challenge myself in areas where I think I am lacking. Lately I have focused on researching state-of-the-art approaches to robot autonomy and control, following my work on other projects.

Although this is a low-cost platform, it is still a good place to evaluate approaches to autonomous behavior. Its limits are outweighed by how accessible it is and how easy it is to test and improve.

The sections below cover the platform, motion planning, perception, autonomy and learning.

The platform

• Base. Four mecanum wheels. ESP32 firmware with register-level PWM, hardware encoders and per-wheel PID.
• Arms. SO-101 5-DOF arms. One on the single-arm variant, two on a lead-screw lift column on the dual-arm variant.
• Compute. A Raspberry Pi 5 and two ESP32s running ROS 2 Jazzy. A Zenoh bridge links the robot to the dev machine.
• Power. Two 4S 18650 packs with a BMS, buck converters and USB-C.

Motion planning

The whole-body planner moves the base, the lift and the arms together as one 11 + 3 DOF system. It runs on CPU. It uses masked, coarse collision search, vectorization and parallelization to produce efficient, collision-free whole-body motion in under a second. Pinocchio and Coal compute the kinematics and collision checks. OMPL searches for a collision-free path. TOPPRA then re-times the path to respect torque limits.

Three choices keep planning fast on CPU:

• Collision checks run on batches of configurations with SIMD, not one at a time.Toggle noteSIMD (single instruction, multiple data) evaluates several configurations with one CPU instruction.
• IK is solved in batches and is not tied to one robot.
• A coarse check prunes candidates before the fine check. An edge is dropped as soon as one configuration collides.

A dual-arm plan takes about 600 ms.

Semantic perception

Perception has multiple consumers. They are high-level functions built on lower-level processes that read and process data from the hardware or the shared tooling. Dependency injection keeps each consumer fully abstracted. Consumers share the same interfaces and ROS 2 topics for input and output, so one can easily be swapped for another. The first consumer tested on hardware used classical PBVS: feature detection, depth estimated with a pinhole camera model, and base control through PID. That resulted in what I called Good Boy mode.

I am currently developing a learning-based perception stack. It works in three steps:

• Features. A pretrained DINOv3 ViT-S/16 extracts features from the camera image. Its weights stay frozen. Features from layers 6, 9 and 12 are combined into one dense feature map.
• Keypoints. Small networks trained on top of that map predict 2D keypoints for each task, such as grasp contact points.
• Pose. PnP matches the 2D keypoints to 3D points, using the floor plane or a second camera view. The result is the object's 6-DoF pose.

The policy also receives the feature map directly. It can then react to what the camera sees during the grasp, instead of relying on one pose estimate.

Early results are promising. I will update this page once I have full results.

Autonomy

The stack is split into decoupled layers. Each layer can change without affecting the others:

• Platform abstraction. Low-level interfaces to the hardware, so the layers above do not depend on the robot.
• Tooling. Shared utilities used across the stack.
• Skills. High-level functions: pick, place, explore, approach and policy (RL, SmolVLA and others).

The behavior tree calls skills as its leaves. This makes long-horizon tasks reliable. Tasks are defined in YAML. Macros expand into whole-body goals at runtime. Each goal has an on_failure branch, so the behavior tree can recover instead of stopping.

Learning

The planner also works as a scripted expert. It generates demonstrations without teleoperation. I currently train ACT and SmolVLA on these demonstrations. A trained policy runs as one step in the behavior tree and sends commands through the same interface as the planner.

I am currently working on leader-arm teleoperation. It will collect demonstrations for future dexterous tasks.

Implementation status

Subsystem	Feature	Status
Hardware	Custom servo driver PCB (KiCad)	✅
Hardware	Chassis, arm mounts and lift column (Fusion 360)	✅
Hardware	Dual SO-101 arms on the lift column	🧪
Hardware	Power system restructure	🧪
Firmware	Register-level PWM + PCNT encoder interface	✅
Firmware	Wheel PID (anti-windup, derivative filter)	✅
Firmware	ESP32 ↔ RPi5 serial communication protocol	✅
Mobile Base	Mecanum kinematics (FK + IK)	✅
Mobile Base	Good Boy autonomous ball retrieval	✅
Mobile Base	Wheel odometry + TF2 broadcast	🧪
Mobile Base	IMU integration + EKF sensor fusion	🧪
Mobile Base	Frontier exploration + SLAM	🧪
Mobile Base	A* / RRT* autonomous navigation	🧪
Manipulator	Analytical, Newton-Raphson and batched IK	✅
Manipulator	IBVS + PBVS visual servoing	✅
Manipulator	Grasp planning	🔧
Motion Planning	Pinocchio + Coal kinematics and collision queries	✅
Motion Planning	OMPL collision-free path search	✅
Motion Planning	SIMD batch collision checking + early exit	✅
Motion Planning	TOPPRA time-optimal re-timing	✅
Motion Planning	Whole-body coordination (base + arms)	✅
Motion Planning	Dual-arm whole-body solve ~600 ms on CPU	✅
Perception	Camera intrinsic + hand-eye calibration	✅
Perception	LiDAR integration + scan filtering	✅
Perception	Frozen DINOv3 ViT-S/16 feature extraction	✅
Perception	2D keypoint heatmap heads	✅
Perception	6-DOF pose via PnP + floor / multi-view	✅
Perception	Synthetic keypoint data (Replicator, BlenderProc)	✅
Perception	Depth camera pipeline	🗓️
Autonomy	Hierarchical YAML task definitions	✅
Autonomy	Behavior tree with on_failure recovery	✅
Autonomy	Dynamic frame resolution (semantic targets)	✅
Autonomy	Unified executor protocol (sim ↔ real)	✅
Teleoperation	Gamepad teleop (base + arm)	✅
Teleoperation	MediaMTX live video stream	✅
Teleoperation	Custom handheld controller (6-DOF + screen)	🗓️
Teleoperation	Leader-follower arm teleoperation	🗓️
System	Zenoh distributed compute (RPi5 ↔ dev machine)	🧪
System	rosbag2 logging + telemetry pipeline	🗓️
Learning	LeRobot / VLA data collection pipeline	✅
Learning	Autonomous demonstration generation	✅
Learning	ACT imitation policy	✅
Learning	SmolVLA fine-tuning	✅
Learning	Keypoint-conditioned policies	🧪
Learning	HIL-SERL + reward classifiers	🧪
Learning	Dataset management + versioning	🧪
Learning	Sim-to-real transfer	🔧
Learning	RL dexterous manipulation	🔧
Learning	pi0 / GR00T export	🗓️
Simulation	Gazebo + ROS2 integration (URDF, sensors)	✅
Simulation	MuJoCo digital twin (MjSpec from URDF at boot)	✅
Simulation	Isaac Sim rendering + domain randomization	🔧
Simulation	Isaac Lab / MJX parallel environments	🔧
Simulation	RL policy training (PPO / SAC)	🔧

✅ Complete · 🧪 Testing · 🔧 In Progress · 🗓️ Planned

Limits and next steps

• Hardware trails simulation. The dual-arm variant is tested in simulation first.
• Keypoint-conditioned policies are in training. They are not yet tested on the real arms.
• The power system and hardware interfacing are being reworked around a custom-designed PCB.
• Leader-arm teleoperation is in progress.