I do want to setup home assistant at some point, but my main usecase for a smartphone assistant is what I mentioned. Things like setting reminders/todo list items, creating calendar events - even sending quick messages to contacts. Basically, things I can already do on my phone, but can do quicker through natural language.
I just have an always on VPN so can tunnel back to my home assistant server; which can handle todos/reminders/calendar events. Quick messages to contacts I haven't considered.
Yeah HA isn't the best choice. It can do some of that stuff, but will require lots of customization to do so. Plus I think you would find things it can't do at all.