I actually tried this workflow with some ebay'ed microcassette recorders - even the really compact ones are bulkier (though not necessarily heavier) than a modern cellphone, plus being non-connected you basically have to set aside some time to transcribe and erase (doable, but if you're specifically using it as a memory assist then having to remember this isn't great.) They did do a good job of "one press and they're operating, and you can confidently believe it" (but you have to pick them up in the right orientation for that to work.)