Project 2 / Build note
Where a VLM Belongs in a Robot Control Loop
On paper, cable insertion is one motion: guide the gripper to a clip and press. On the robot, the cable bends as it is held and routed, and the force that seats it changes with the contact. I wanted a vision-language model to help the system recover from those differences. My first design gave it too much control.
I tested that boundary on an emergency lighting unit, where the system had to insert two cables into clips. The first version let the VLM construct a complete behavior tree from low-level primitives. When that proved unreliable, I moved the model up a level and kept execution deterministic.
The task was easy to describe and hard to specify
A cable is not a rigid object with one pose waiting to be estimated. Its shape changes with how it is held, where it is routed, and what forces are applied. From a fixed viewpoint, the full state is not observable, and the possible configurations are effectively unlimited.
That makes cable insertion a contact-rich manipulation problem. The gripper must apply enough stiffness to seat the cable in the clip, but not so much that it damages the cable or the lighting unit. There was no force value that could be fixed once and reused: the useful operating range varied across clips and cable positions.
The project objective was to let the system discover those manipulation parameters through iterative testing, without expert demonstrations or manual tuning for each configuration.
First attempt: let the VLM write the whole tree
The first design treated the VLM as a planner all the way down. It received the task description and a set of low-level primitives, then constructed a complete behavior tree from scratch.
That boundary did not hold. Early testing showed that the VLM could reason about the task structure at a high level, but it did not reliably produce trees that were both syntactically valid and semantically correct when it had to compose unfamiliar low-level primitives. A tree can look plausible in language and still fail as an executable manipulation sequence.
The VLM still had a useful role; the problem was asking it to make decisions at the level where small mistakes became robot actions.
The revised boundary: semantic subtrees
I replaced the open-ended primitive vocabulary with predefined behavior-tree subtrees. Each subtree represented a meaningful manipulation phase, such as approaching a waypoint, moving the gripper to a contact position, or applying a specified contact force.
Each subtree also had a natural-language description of what it accomplished in the context of the full manipulation sequence. That context mattered. “Move here” is a mechanical instruction; “reach the contact position before applying insertion force” describes a role in the task.
The VLM now composed those subtrees into a valid tree for the task. After an attempt, it could adjust parameters within the tree or select an alternative subtree combination. The primitive action layer stayed deterministic and verifiable.
This was a more useful division of labor. The VLM handled the parts of the problem that benefited from semantic reasoning. The executor handled the parts where repeatability mattered more than flexibility.
The system became a closed loop
Once the low-level execution path was fixed, the system could operate as a real adaptation loop:
- execute the selected behavior-tree subtrees;
- observe the contact and insertion outcome;
- turn that attempt into an analysis report;
- ask the VLM to adjust parameters or restructure the subtree composition; and
- retry with deterministic execution.
The VLM was not issuing an unbounded stream of motor-level instructions. It was choosing what the next structured attempt should be.
Binary contact feedback hid the decision
The first state representation was a binary contact signal. It told the adaptation loop whether contact had occurred, but that answer was too coarse for the next decision.
A near-success might need a small force increase. Another failed insertion might need a trajectory correction. Both could arrive at the adaptation loop as the same kind of binary outcome, even though they called for different changes.
I added a time-series summary of force-torque readings during the insertion phase to the analysis reports. The VLM could then reason over how the contact signal evolved instead of seeing only the final bit.
That richer feedback helped the model distinguish the failure mode and recommend the appropriate adjustment. It gave the VLM a contact history to interpret and a basis for choosing the next adjustment.
The result was a working insertion loop
The system successfully inserted two cables into clips on the emergency lighting unit.
The engineering result I would carry forward is the boundary that made that possible: the VLM composed and adapted semantic behavior-tree subtrees, while deterministic execution preserved the low-level behavior. The feedback representation was part of that design. Choosing what the VLM could see was inseparable from choosing what it could change.
What I would test next
The next experiments should test whether the same loop can become more expressive without giving up its control boundary.
- Larger subtree vocabularies. Add more contact and routing behaviors, then test whether the loop transfers to a wider range of deformable-object tasks without restructuring the architecture.
- Graduated success signals. Replace the binary outcome with a continuous measure such as final insertion depth, so the VLM can make finer adjustments.
- Representation ablations. Compare low-level primitives against semantic subtrees, and binary feedback against force-torque summaries. The point would be to isolate which abstraction boundary and which feedback representation produce the better adaptation behavior.
Those tests would turn the design lesson into an evaluation question: how much structure should a robot give the model, and how much of the system’s state must be represented before the model can adapt usefully?