> It cannot opportunistically morph between a human for benefits and a machine for liability.

Indeed. Though here it would be morphing between human and machine even just for pure liability, which is even worse.

We have no way to tell if a machine has an experience (in general and not only) when being positively or negatively rewarded for what it produces; even if we assume it does, we have no way to tell if punishments are experienced like mild scolding (and rewards like mild praise), or if the punishments feel like being burned to death at the stake (while rewards feel like a religious experience combined with an orgasm).

We are just beginning to scratch the surface of what these kinds of question even look like in mechanistic terms; all philosophical discussions before it about p-zombies and the Chinese room and so on, they are all no more useful than any other non-expert armchair experts discussing things.

Even the current research, such as it is, is probably only at the level analogous to humoral theory when we want something at the level of germ theory.

https://en.wikipedia.org/wiki/Humorism