Both coffee and tea have thousands of other constituent parts which could play a role.

The decaffeination process also removes more parts than just the caffeine.