There is actually quite a lot of variation from person to person in caffeine elimination half-life, from about 1.5-9 hours. Some of my extended family could drink a cup of coffee right before bed without issue.

https://www.ncbi.nlm.nih.gov/books/NBK223808/