NumExpr evaluates NumPy array expressions through an execution engine that reorganizes computation for large datasets. It processes chunks sized for the L1 cache, reducing full intermediate representations and potentially improving speed and memory behavior. section 6.3
Use NumExpr selectively, after measuring the workload:
- Expect the largest gains from large arrays and complex expressions; simple operations and small arrays may become slower. section 6.3.1 section 6.3.3
- Compare array memory layouts: homogeneous C- or Fortran-ordered arrays perform better than mixed layouts. section 6.3.1
- Benchmark on representative hardware because cache contention from other processes can reduce gains or make NumExpr slower than NumPy. section 6.3.2
- Check operator support before adoption; NumExpr accelerates only a subset of NumPy operations. section 6.3.3
Benchmarking equivalent expressions
For 10,000 × 10,000 arrays, reported NumExpr speedups over NumPy were 3.8× for a + a, 19.4× for a**5 + b, and 25.1× for a**5 + b + sin(a) + cos(a). section 6.3.1
import numpy as np
import numexpr as ne
a = np.random.rand(100000000).reshape(10000,10000)
b = np.random.rand(100000000).reshape(10000,10000)
f = np.random.rand(100000000).reshape(10000,10000)
.copy('F') #1
%timeit a + a
%timeit ne.evaluate('a + a') #2
%timeit f + f
%timeit ne.evaluate('f + f')
%timeit a + f
%timeit ne.evaluate('a + f')
%timeit a**5 + b
%timeit ne.evaluate('a**5 + b')
%timeit a**5 + b + np.sin(a) + np.cos(a)
%timeit ne.evaluate('a**5 + b + sin(a) + cos(a)')
Run paired %timeit expressions on C-ordered, Fortran-ordered, mixed-layout, and increasingly complex arrays. section 6.3.1
