alt Hacker NewsWhat is the advantage of using Numexpr instead of say Polars, Numba or Taichi?
Numexpr seems to sit in a sort of odd niche of having to do relatively simple arithmetic on in-core matrix data fast. For anything more complex, Polars seems more powerful and yet easier to understand, Numba and Taichi are both much more flexible in that they can be used to implement much more complex arithmetic (at the cost of writing lower level python code).
Numexpr basically evaluates raw strings, which makes any sort of heavy usage basically immune to linting, code inspection and refactoring.
Pandas has the eval() method on the Dataframe that uses numexpr as backend, but we generally never use it because of the upper mentioned maintenance issues and the availability of better alternatives.
numexpr is for different use cases than Polars or Taichi, which themselves are quite different from each other. numexpr is more akin to numba - it speeds up numpy usage.
numexpr speeds up different kinds of usage than numba does. numba is best at speeding up non-vectorizable usage of numpy like repeated operations on arrays inside of for-loops. numexpr speeds up regular numpy expressions, like `5 * x + 7` where x is an array, by avoiding intermediate allocations. It calculates the entire expression for each cell, rather than doing each individual operation into intermediate arrays. It uses strings for expressing calculation so that Python will not break down the expression and hand it off to numexpr one operator at a time like it does with numpy.