Unfortunately, anti-androgens have myriad effects beyond basic T suppression.

There are two primary drivers behind why anti-androgens would cause loss of libido beyond effect on T:

1) AA's cause androgen receptor blockade systemically. This blocks action at the AR that would be residual across systems from adrenal production. Most important for libido are the AR activity that occurs inside of neurons and astrocytes in the brain

2) AA's have a two-punch effects on the Prolactin/Dopamine system + Progesterone system. Chronically elevated prolactin causes down-regulation of dopamine, which by itself is enough to kill libido. Progestins modulate GABA, which can cause "flat affect" and "emotional flatlining".

The combo punch of neuronal/adrenal AR blockade + Prolactin/Dopamine dysregulation + GABA dysregulation would require a miracle to have preserved libido on.