These seems like an inherent challenge to recommending based on stars. Stars are very sparse, so there’s little “didn’t star this” signal, and there’s no “thumbs down”.

So you’re left with things you “should” star, but there very well could be a reason you didn’t.