To me the interesting question about a job like this is "How can you tell if you're doing it well?" It involves such high-stakes, high-uncertainty and highly variability that it has to be nearly impossible to know. I mean you're predicting distant outcomes from creative pursuits which must first survive a gauntlet of wicked complexity and randomness.

Only a few percent of your judgements are ever tested (by surviving being optioned, produced and released) and, of the ones that are, at best you only get a small sampling of false positives over a sea of potential false negatives. I imagine he's incredibly interested in the fate of any titles he didn't recommend which end up being produced (perhaps by another studio). Having filled a similar role in a different industry with similar high-stakes 'unknowables', I thought a lot about this. It was pretty obvious what practically mattered was how much my output "felt right" to downstream decision-makers vs actually being right.

While my stakeholders were quite happy with my work, actually targeting such ephemeral and uncorrelated feedback felt unproductive and dumb. Eventually, I settled on making the evaluation process fully transparent and consistent. I ensured all objective criteria were documented and each subjective judgement had clear confidence intervals. This was more challenging than it sounds. In the end, it was still hard to know if I was really improving year to year. For that, I still had to rely on my own, mostly subjective, self-assessment but at least I had some objective tracking data to calibrate on. That at least helped me feel like I was executing with diligence and integrity. It also increased my confidence no one else in the industry was doing it any better.