• 9

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191


File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

name Punditsdkoslkdosdkoskdo

Pandas mask / where methods versus NumPy np.where

I often use Pandas mask and where methods for cleaner logic when updating values in a series conditionally. However, for relatively performance-critical code I notice a significant performance drop relative to numpy.where.

While I'm happy to accept this for specific cases, I'm interested to know:

  1. Do Pandas mask / where methods offer any additional functionality, apart from inplace / errors / try-cast parameters? I understand those 3 parameters but rarely use them. For example, I have no idea what the level parameter refers to.
  2. Is there any non-trivial counter-example where mask / where outperforms numpy.where? If such an example exists, it could influence how I choose appropriate methods going forwards.

For reference, here's some benchmarking on Pandas 0.19.2 / Python 3.6.0:


n = 10000000
df = pd.DataFrame(np.random.random(n))

assert (df[0].mask(df[0] > 0.5, 1).values == np.where(df[0] > 0.5, 1, df[0])).all()

%timeit df[0].mask(df[0] > 0.5, 1)       # 145 ms per loop
%timeit np.where(df[0] > 0.5, 1, df[0])  # 113 ms per loop

The performance appears to diverge further for non-scalar values:

%timeit df[0].mask(df[0] > 0.5, df[0]*2)       # 338 ms per loop
%timeit np.where(df[0] > 0.5, df[0]*2, df[0])  # 153 ms per loop
      • 1
    • @ead, This is interesting. An answer, in that case, may comment on what's changed (and in which version?). I haven't seen any mention of implementation change. Also, do you see equivalent performance for both my examples?
      • 2
    • I have to search the sourcecode for a sophisticated answer. But looking at the memory consumption it looks like df[0].mask does quite a lot more temporary memory allocation which np.where does not. I implemented also a parallelized version in Numba, inplace its 8 times faster than np.where, out of place its only 3 times faster. I assume the np.where is a simple compiled for,if,else loop like my Numba solution,while pandas creates a temporary mask array.

I'm using pandas 0.23.3 and Python 3.6, so I can see a real difference in running time only for your second example.

But let's investigate a slightly different version of your second example (so we get2*df[0] out of the way). Here is our baseline on my machine:

twice = df[0]*2
mask = df[0] > 0.5
%timeit np.where(mask, twice, df[0])  
# 61.4 ms ± 1.51 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit df[0].mask(mask, twice)
# 143 ms ± 5.27 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Numpy's version is about 2.3 times faster than pandas.

So let's profile both functions to see the difference - profiling is a good way to get the big picture when one isn't very familiar with the code basis: it is faster than debugging and less error-prone than trying to figure out what's going on just by reading the code.

I'm on Linux and use perf. For the numpy's version we get (for the listing see appendix A):

>>> perf record python np_where.py
>>> perf report

Overhead  Command  Shared Object                                Symbol                              
  68,50%  python   multiarray.cpython-36m-x86_64-linux-gnu.so   [.] PyArray_Where
   8,96%  python   [unknown]                                    [k] 0xffffffff8140290c
   1,57%  python   mtrand.cpython-36m-x86_64-linux-gnu.so       [.] rk_random

As we can see, the lion's share of the time is spent in PyArray_Where - about 69%. The unknown symbol is a kernel function (as matter of fact clear_page) - I run without root privileges so the symbol is not resolved.

And for pandas we get (see Appendix B for code):

>>> perf record python pd_mask.py
>>> perf report

Overhead  Command  Shared Object                                Symbol                                                                                               
  37,12%  python   interpreter.cpython-36m-x86_64-linux-gnu.so  [.] vm_engine_iter_task
  23,36%  python   libc-2.23.so                                 [.] __memmove_ssse3_back
  19,78%  python   [unknown]                                    [k] 0xffffffff8140290c
   3,32%  python   umath.cpython-36m-x86_64-linux-gnu.so        [.] DOUBLE_isnan
   1,48%  python   umath.cpython-36m-x86_64-linux-gnu.so        [.] BOOL_logical_not

Quite a different situation:

  • pandas doesn't use PyArray_Where under the hood - the most prominent time-consumer is vm_engine_iter_task, which is numexpr-functionality.
  • there is some heavy memory-copying going on - __memmove_ssse3_back uses about 25% of time! Probably some of the kernel's functions are also connected to memory-accesses.

Actually, pandas-0.19 used PyArray_Where under the hood, for the older version the perf-report would look like:

Overhead  Command        Shared Object                     Symbol                                                                                                     
  32,42%  python         multiarray.so                     [.] PyArray_Where
  30,25%  python         libc-2.23.so                      [.] __memmove_ssse3_back
  21,31%  python         [kernel.kallsyms]                 [k] clear_page
   1,72%  python         [kernel.kallsyms]                 [k] __schedule

So basically it would use np.where under the hood + some overhead (all above data-copying, see __memmove_ssse3_back) back then.

I see no scenario where pandas could become faster than numpy in pandas' version 0.19 - it just adds overhead to numpy's functionality. Pandas' version 0.23.3 is an entirely different story - here numexpr-module is used, it is very possible that there are scenarios for which pandas' version is (at least slightly) faster.

I'm not sure this memory-copying is really called for/necessary - maybe one even could call it performance-bug, but I just don't know enough to be certain.

We could help pandas not to copy, by peeling away some indirections (passing np.array instead of pd.Series). For example:

%timeit df[0].mask(mask.values > 0.5, twice.values)
# 75.7 ms ± 1.5 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Now, pandas is only 25% slower. The perf says:

Overhead  Command  Shared Object                                Symbol                                                                                                
  50,81%  python   interpreter.cpython-36m-x86_64-linux-gnu.so  [.] vm_engine_iter_task
  14,12%  python   [unknown]                                    [k] 0xffffffff8140290c
   9,93%  python   libc-2.23.so                                 [.] __memmove_ssse3_back
   4,61%  python   umath.cpython-36m-x86_64-linux-gnu.so        [.] DOUBLE_isnan
   2,01%  python   umath.cpython-36m-x86_64-linux-gnu.so        [.] BOOL_logical_not

Much less data-copying, but still more than in the numpy's version which is mostly responsible for the overhead.

My key take-aways from it:

  • pandas has the potential to be at least slightly faster than numpy (because it is possible to be faster). However, pandas' somewhat opaque handling of data-copying makes it hard to predict when this potential is overshadowed by (unnecessary) data copying.

  • when the performance of where/mask is the bottleneck, I would use numba/cython to improve the performance - see my rather naive tries to use numba and cython further below.

The idea is to take

np.where(df[0] > 0.5, df[0]*2, df[0])

version and to eliminate the need to create a temporary - i.e, df[0]*2.

As proposed by @max9111, using numba:

import numba as nb
def nb_where(df):
    n = len(df)
    output = np.empty(n, dtype=np.float64)
    for i in range(n):
        if df[i]>0.5:
            output[i] = 2.0*df[i]
            output[i] = df[i]
    return output

assert(np.where(df[0] > 0.5, twice, df[0])==nb_where(df[0].values)).all()
%timeit np.where(df[0] > 0.5, df[0]*2, df[0])
# 85.1 ms ± 1.61 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

%timeit nb_where(df[0].values)
# 17.4 ms ± 673 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Which is about factor 5 faster than the numpy's version!

And here is my by far less successful try to improve the performance with help of Cython:

%%cython -a
cimport numpy as np
import numpy as np
cimport cython

def cy_where(double[::1] df):
    cdef int i
    cdef int n = len(df)
    cdef np.ndarray[np.float64_t] output = np.empty(n, dtype=np.float64)
    for i in range(n):
        if df[i]>0.5:
            output[i] = 2.0*df[i]
            output[i] = df[i]
    return output

assert (df[0].mask(df[0] > 0.5, 2*df[0]).values == cy_where(df[0].values)).all()

%timeit cy_where(df[0].values)
# 66.7± 753 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

gives 25% speed-up. Not sure, why cython is so much slower than numba though.


A: np_where.py:

import pandas as pd
import numpy as np


n = 10000000
df = pd.DataFrame(np.random.random(n))

twice = df[0]*2
for _ in range(50):
      np.where(df[0] > 0.5, twice, df[0])  

B: pd_mask.py:

import pandas as pd
import numpy as np


n = 10000000
df = pd.DataFrame(np.random.random(n))

twice = df[0]*2
mask = df[0] > 0.5
for _ in range(50):
      df[0].mask(mask, twice)
  • 24
Reply Report
    • @jpp as my understanding goes, level is used (together with axis-argument) if the other-argument has another number (i.e. less) of dimensions than df[0] in order to control (at least to some degree) the way the alignment is performed. I'm pretty sure, it hasn't any impact on the performance in your case and not sure it should be a part of this answer.
    • OK, will play around with different dimensions to figure it out. Sure, no need to update your answer. It's great as it is.
      • 2
    • I guess Numba performs very good because of SIMD-vectorization of the loop. You can check this with llvmlite.binding as llvm llvm.set_option('', '--debug-only=loop-vectorize') .With the right C-Compiler settings Cython might also perform the same. (The equivivalent would be (-O3, -march=native) with the Clang Compiler)

Trending Tags