If you frame it like so:

  <noob> Where do birds go when it rains?
  <expert> They

then GPT-2 generally doesn't write more questions.