_We_ are still traveling forwards in time. If you followed a cloud of antiparticles backwards in time along its direction of travel, then you would see entropy decrease yes, though regular particles would do the same if we followed them back in time.