alt Hacker NewsOne way I handle the 'simple case that Fourier has trouble with' is I add the 'simple' bases to the catalogue of Fourier basis functions and then orthogonalise the resulting basis system.
Depending on specific 'simple' cases this can be messy but effective.
The same deal happens for decision trees. (Axis parallel) Trees have a hard time fitting linear functions, requiring many modes. One can use essentially the same solution.
BTW very informative commentary. The things that you bring up really needed to be brought out.
I’ll keep asking my ignorant questions since I found informed people: what about a basis of wavelets or some heterogeneous bag of functions (which could include linear, polynomial times Gaussian, and why not some harmonic functions too)? At the end of the day a functional basis might not need to be “well formed” like Fourier or some polynomial family. It just needs to do the job.