alt Hacker NewsForecaster of the Confounding Company here :)
You're right but, if the time series looks like music then you can apply your music domain knowledge. You don't have to only apply knowledge that's general across every domain, part of the magic of large neural nets is that you can do recognition and prediction at the same time thereby using (statistically) things known from training about how to identify a sequence and then also how to continue it.
Sure adding outside info helps even more, but with enough data it should become unnecessary (or so goes the theory).
But why would you throw away that data at all? These are tiny models compared to domain-specific audio/music/image models, and any capacity spent on recognition is capacity that could have been used for prediction.
Is there a reason you don't throw in something like cross-attention on an embedding of the sequence description/data labels? I've trained numerous generative models for audio and images and it's always the case that the unconditional model performs worse, trains slower, etc. than the conditional one (with the best performance from a hybrid using classifier-free guidance or similar scheme).
Again I don't have experience with the domain of time-series forecasting directly, but I would think there are generalities between autoregressive generative models and time-series forecasting, given that both involve predicting the next sample(s) given N regularly sampled past inputs.