What is the scenario you're thinking about? This is kinda what happens with VLAs, the latent knowledge of an LLM can help in action generation pipelines, but it's also kinda slow.