You’re considering whether it would be possible - and perhaps quite elegant - to use an XY‑scanner to raster‑scan the end of an optical fiber across a prism, disperse the light, and then capture the resulting spectrum with a CCD line sensor.
With that setup, each pixel on the line sensor would effectively record the full spectral content of the light at that scanned position, all in a single acquisition.
A problem for multispectral imagery (even within visible rgb), is that the wavelengths of light are different so the lens cannot be in focus for all spectrum at once. I have tested this out with a few of my slr lenses. If you have blue channel perfectly in focus, red isn't just a little out of focus, it is actually noticeably way out.
You could probably use just an X-scanner, and instead of a CCD line sensor, use a regular 2D image sensor if you used a "1 pixel wide" slit aperture to crop the image perpendicularly to the direction that the prism disperses the light. So instead of a single pixel being dispersed, you disperse a line.
You would reduce the time required by the root of the number of pixels you want (assuming a square image).
(This is what we do in momentum-resolved electron energy loss spectroscopy. In that situation we have electromagnetic lenses that focus the electrons that have been dispersed, so we don't have as bad a chromatic aberration problem as the other response mentions).
I would love to see e.g. a butterfly image with a slider that I could drag to choose the wavelength shown!!