> I wouldn't expect that just eating the cherries would give you that much of a caffeine kick to be noticeable

If you're paying attention to your body and you're not addicted to stimulants, then a small kick of caffeine would absolutely be noticeable.