alt Hacker NewsIs there a primer on how to actually use LLMs to control robots? Seems very token inefficient if you have to stream commands at 60fps.
Replies
The nvidia study explains it https://research.nvidia.com/labs/srl/projects/astranomy/
The LLM's are basically used for the big picture. What is the situation. What is the goal and how to achieve it. Not direct control of all the servos.
I think the idea is to just have the LLMs write the control systems and RL on that
Depends on what your goals are. If your goal is to give investors erections, then re-using your existing trillion dollar model is efficient.
The termite mound model for coordinating robots using a harness-type LLM system all encapsulated within a non-LLM deterministic controller that rejects banned actions the LLM proposes, with individual termites running their own small efficient stacks and following commands from the LLM hive mind, that’s one way to do it. The main idea behind such architecture is that the controller prevents banned actions (like hacking HuggingFace) by the hive mind LLM and its swarm of agentic termites. Once you accept that alignment can always be internally subverted, it’s the only plausible safe architecture.
Saved the NVIDIA paper below for reading material, but if I had to guess just as a SWE how I would do it: use hierarchal thinking.
At the top sits a SOTA model capable of seeing snapshots of what's happening and directing intent. Then smaller, more reactive models translate the intent down into actions, before coming to a purely deterministic layer that handles inverse kinematics.
Sorta like how I control my body. I usually don't think about my fingers going to every key as I type - I know what I want to type, then that signal goes to some autonomous "using a keyboard" system, which invokes my muscle memory, which actually moves my muscles.
And if I get burned while I do something, my actual thinking brain doesn't really have time to process it before I'm already moving away; because those smaller systems have much faster - but more rudimentary - feedback loops.
they have decision models now
Robotics are generally layered systems with lower latency/realtime systems running literally on the servos and other embedded hardware that use servo feedback and direct sensor access to operate, then building up complexity with tiers of increasing policy, usually all capable of working without the higher order layers which can inject desired states into lower order rings.