"You can just choose not to use it", sure, until signing a consent form to use ChatGPT becomes mandatory for a doctor visit, just like all kinds of other technology (like having a cell phone to verify SMS, for example) is basically essential now to function in society.
Doctors in Europe already use LLMs to treat you.