Sure, if you assume the agent will be hostile on you. I thought it's just so the agent doesn't accidentally rm -rf / on you
There are documented instances of LLMs casually using LPEs in order to achieve an objective: https://xcancel.com/sluongng/status/2060746160558543217
And that's without anything like prompt injection happening.
They do try privilege escalation unprompted.
The agent might install hostile software, e.g. a npm package. Unfortunately, very common problem nowadays.