• 3
name

A PHP Error was encountered

Severity: Notice

Message: Undefined index: userid

Filename: views/question.php

Line Number: 191

Backtrace:

File: /home/prodcxja/public_html/questions/application/views/question.php
Line: 191
Function: _error_handler

File: /home/prodcxja/public_html/questions/application/controllers/Questions.php
Line: 433
Function: view

File: /home/prodcxja/public_html/questions/index.php
Line: 315
Function: require_once

name Punditsdkoslkdosdkoskdo

Supplying NumPy site.cfg arguments to pip

I'm using NumPy built against Intel's Math Kernel Library. I use virtualenv, and typically use pip to install packages.

However, in order for NumPy to find the MKL libraries, it's necessary to create a site.cfg file in the NumPy source directory prior to compiling it, then manually build and install. I could script this whole process, but I was hoping for a simpler solution.

I have a standard site.cfg file that can be used for this purpose under version control. Are there any pip command line options that will tell it to copy a particular file to the source directory before building a package?

Alternatively, are there any environment variables that can be set instead of supplying the library paths in a site.cfg file? Here is the site.cfg file that I use. It was taken almost verbatim from Intel's site.

[mkl]
library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64
include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/include
mkl_libs = mkl_rt
lapack_libs =

For reference, I'm running Ubuntu, Python 2.7, and NumPy 1.6.

From the source (https://github.com/numpy/numpy/blob/master/site.cfg.example):

To assist automatic installation like easy_install, the user's home directory will also be checked for the file ~/.numpy-site.cfg .

Is that a workable solution? You'd still need to preload the home directories with the global .numpy-site.cfg, but you wouldn't have to muck with the build or installation after that.

  • 30
Reply Report

I ended up putting together a script to automate this. Here it is, in case it can help someone else. I've tested it in Python 2.7, but it should work elsewhere without significant modifications.

from __future__ import unicode_literals

import io
import os.path
import re
import subprocess
import urllib2

# This downloads, builds, and installs NumPy against the MKL in the
# currently active virtualenv

file_name = 'numpy-1.6.2.tar.gz'
url = ('http://sourceforge.net/projects/numpy/files/NumPy/1.6.2/'
       'numpy-1.6.2.tar.gz/download')

def main():

    # download NumPy and unpack it
    file_data = urllib2.urlopen(url).read()
    with io.open(file_name, 'wb') as fobj:
        fobj.write(file_data)
    subprocess.check_call('tar -xvf {0}'.format(file_name), shell=True)
    base_name = re.search(r'(.*)\.tar\.gz$', file_name).group(1)
    os.chdir(base_name)

    # write out a site.cfg file in the build directory
    site_cfg = (
        '[mkl]\n'
        'library_dirs = /opt/intel/composer_xe_2013.1.117/mkl/lib/intel64\n'
        'include_dirs = /opt/intel/composer_xe_2013.1.117/mkl/include\n'
        'mkl_libs = mkl_rt\n'
        'lapack_libs =\n')
    with io.open('site.cfg', 'wt', encoding='UTF-8') as fobj:
        fobj.write(site_cfg)

    # build and install NumPy
    subprocess.check_call('python setup.py build', shell=True)
    subprocess.check_call('python setup.py install', shell=True)


if __name__ == '__main__':
    main()
  • 6
Reply Report

Your goal of installing NumPy to use Intel's Math Kernel Library is now much easier since Intel created pips to install MKL + NumPy:

pip uninstall numpy -y  # if the standard numpy is present
pip install intel-numpy

as well as intel-scipy, intel-scikit-learn, pydaal, tbb4py, mkl_fft, mkl_random, and the lower level packages if you need just them. Again, you must first uninstall the standard packages if they're already installed in your virtualenv.

NOTE:

If standard NumPy, SciPy and Scikit-Learn packages are already installed, the packages must be uninstalled before installing the Intel® variants of these packages(intel-numpy etc) to avoid any conflicts. As mentioned earlier, pydaal uses intel-numpy, hence it is important to first remove the standard Numpy library (if installed) and then install pydaal.

  • 3
Reply Report

To your question of how to configure NumPy (e.g. to use OpenBLAS):

  1. Download https://github.com/numpy/numpy/blob/master/site.cfg.example
  2. Edit the relevant lines, e.g.
[openblas]
libraries = openblas
library_dirs = /opt/OpenBLAS/lib
include_dirs = /opt/OpenBLAS/include
  1. Save it as ~/.numpy-site.cfg
  2. Install numpy from source without manually downloading it (--force-reinstall will let it replace an existing package):

    pip install numpy --no-binary numpy --force-reinstall

  3. Bonus: The same file ~/.numpy-site.cfg works for installing scipy on the OpenBLAS:

    pip install scipy --no-binary scipy

    or install them together:

    pip install numpy scipy --no-binary numpy,scipy --force-reinstall

Sept. 2019: If you're still using Python 2.7, install numpy then install scipy. Attempting to install them together will:

  • invoke a SciPy easy_install installer that requests NumPy,
  • load the latest NumPy installer (even if you specifically asked pip to install numpy==1.14.6 scipy==1.0.1 --no-binary numpy,scipy), then
  • fail with RuntimeError: Python version >= 3.5 required because the latest NumPy does not support Python 2.7.
  • 0
Reply Report

Trending Tags