The jobs array is getting initialized in O(compiled cpus^2) complexity. Distros and people with bigger systems will use pretty high values (128 or 256 or more) for this value, leading to interesting bubbles in performance. Baseline (single threaded performance) gets roughly 13 - 15 multiplications per cycle in the interesting range (threading kicks in at 65x65 mult by 65x65). The hardware is capable of 32 multiplications per cycle theoretically. Matrix SGEMM cycles MPC DGEMM cycles MPC 48 x 48 10703.9 10.6 0.0% 17990.6 6.3 0.0% 64 x 64 20778.4 12.8 0.0% 40629.2 6.5 0.0% 65 x 65 26869.9 10.3 0.0% 52545.7 5.3 0.0% 80 x 80 38104.5 13.5 0.0% 72492.7 7.1 0.0% 96 x 96 61626.4 14.4 0.0% 113983.8 7.8 0.0% 112 x 112 91803.8 15.3 0.0% 180987.3 7.8 0.0% 128 x 128 133161.4 15.8 0.0% 258374.3 8.1 0.0% When threading is turned on TARGET=SKYLAKEX F_COMPILER=GFORTRAN SHARED=1 DYNAMIC_THREADS=1 USE_OPENMP=0 NUM_THREADS=128 Matrix SGEMM cycles MPC DGEMM cycles MPC 48 x 48 10725.9 10.5 -0.2% 18134.9 6.2 -0.8% 64 x 64 20500.6 12.9 1.3% 40929.1 6.5 -0.7% 65 x 65 2040832.1 0.1 -7495.2% 2097633.6 0.1 -3892.0% 80 x 80 2063129.1 0.2 -5314.4% 2119925.2 0.2 -2824.3% 96 x 96 2070374.5 0.4 -3259.6% 2173604.4 0.4 -1806.9% 112 x 112 2111721.5 0.7 -2169.6% 2263330.8 0.6 -1170.0% 128 x 128 2276181.5 0.9 -1609.3% 2377228.9 0.9 -820.1% There is a deep deep cliff once you hit 65x65 With this patch Matrix SGEMM cycles MPC DGEMM cycles MPC 48 x 48 10630.0 10.6 0.7% 18112.8 6.2 -0.7% 64 x 64 20374.8 13.0 1.9% 40487.0 6.5 0.4% 65 x 65 141955.2 1.9 -428.3% 146708.8 1.9 -179.2% 80 x 80 178921.1 2.9 -369.6% 186032.7 2.8 -156.6% 96 x 96 205436.2 4.3 -233.4% 224513.1 3.9 -97.0% 112 x 112 244408.2 5.8 -162.7% 262158.7 5.4 -47.1% 128 x 128 321334.5 6.5 -141.3% 333829.0 6.3 -29.2% The cliff is very significantly reduced. (more to follow) |
||
|---|---|---|
| benchmark | ||
| cmake | ||
| ctest | ||
| driver | ||
| exports | ||
| interface | ||
| kernel | ||
| lapack | ||
| lapack-netlib | ||
| reference | ||
| relapack | ||
| test | ||
| utest | ||
| .gitignore | ||
| .travis.yml | ||
| BACKERS.md | ||
| CMakeLists.txt | ||
| CONTRIBUTORS.md | ||
| Changelog.txt | ||
| GotoBLAS_00License.txt | ||
| GotoBLAS_01Readme.txt | ||
| GotoBLAS_02QuickInstall.txt | ||
| GotoBLAS_03FAQ.txt | ||
| GotoBLAS_04FAQ.txt | ||
| GotoBLAS_05LargePage.txt | ||
| GotoBLAS_06WeirdPerformance.txt | ||
| LICENSE | ||
| Makefile | ||
| Makefile.alpha | ||
| Makefile.arm | ||
| Makefile.arm64 | ||
| Makefile.generic | ||
| Makefile.ia64 | ||
| Makefile.install | ||
| Makefile.mips | ||
| Makefile.mips64 | ||
| Makefile.power | ||
| Makefile.prebuild | ||
| Makefile.rule | ||
| Makefile.sparc | ||
| Makefile.system | ||
| Makefile.tail | ||
| Makefile.x86 | ||
| Makefile.x86_64 | ||
| Makefile.zarch | ||
| README.md | ||
| TargetList.txt | ||
| USAGE.md | ||
| appveyor.yml | ||
| c_check | ||
| cblas.h | ||
| common.h | ||
| common_alpha.h | ||
| common_arm.h | ||
| common_arm64.h | ||
| common_c.h | ||
| common_d.h | ||
| common_ia64.h | ||
| common_interface.h | ||
| common_lapack.h | ||
| common_level1.h | ||
| common_level2.h | ||
| common_level3.h | ||
| common_linux.h | ||
| common_macro.h | ||
| common_mips.h | ||
| common_mips64.h | ||
| common_param.h | ||
| common_power.h | ||
| common_q.h | ||
| common_reference.h | ||
| common_s.h | ||
| common_sparc.h | ||
| common_stackalloc.h | ||
| common_thread.h | ||
| common_x.h | ||
| common_x86.h | ||
| common_x86_64.h | ||
| common_z.h | ||
| common_zarch.h | ||
| cpuid.S | ||
| cpuid.h | ||
| cpuid_alpha.c | ||
| cpuid_arm.c | ||
| cpuid_arm64.c | ||
| cpuid_ia64.c | ||
| cpuid_mips.c | ||
| cpuid_mips64.c | ||
| cpuid_power.c | ||
| cpuid_sparc.c | ||
| cpuid_x86.c | ||
| cpuid_zarch.c | ||
| ctest.c | ||
| ctest1.c | ||
| ctest2.c | ||
| f_check | ||
| ftest.f | ||
| ftest2.f | ||
| ftest3.f | ||
| gen_config_h.c | ||
| getarch.c | ||
| getarch_2nd.c | ||
| l1param.h | ||
| l2param.h | ||
| make.inc | ||
| openblas.pc.in | ||
| openblas_config_template.h | ||
| param.h | ||
| quickbuild.32bit | ||
| quickbuild.64bit | ||
| quickbuild.win32 | ||
| quickbuild.win64 | ||
| segfaults.patch | ||
| symcopy.h | ||
| version.h | ||
README.md
OpenBLAS
Introduction
OpenBLAS is an optimized BLAS library based on GotoBLAS2 1.13 BSD version.
Please read the documentation on the OpenBLAS wiki pages: http://github.com/xianyi/OpenBLAS/wiki.
Binary Packages
We provide official binary packages for the following platform:
- Windows x86/x86_64
You can download them from file hosting on sourceforge.net.
Installation from Source
Download from project homepage, http://xianyi.github.com/OpenBLAS/, or check out the code using Git from https://github.com/xianyi/OpenBLAS.git.
Dependencies
Building OpenBLAS requires the following to be installed:
- GNU Make
- A C compiler, e.g. GCC or Clang
- A Fortran compiler (optional, for LAPACK)
- IBM MASS (optional, see below)
Normal compile
Simply invoking make (or gmake on BSD) will detect the CPU automatically.
To set a specific target CPU, use make TARGET=xxx, e.g. make TARGET=NEHALEM.
The full target list is in the file TargetList.txt.
Cross compile
Set CC and FC to point to the cross toolchains, and set HOSTCC to your host C compiler.
The target must be specified explicitly when cross compiling.
Examples:
-
On an x86 box, compile this library for a loongson3a CPU:
make BINARY=64 CC=mips64el-unknown-linux-gnu-gcc FC=mips64el-unknown-linux-gnu-gfortran HOSTCC=gcc TARGET=LOONGSON3A -
On an x86 box, compile this library for a loongson3a CPU with loongcc (based on Open64) compiler:
make CC=loongcc FC=loongf95 HOSTCC=gcc TARGET=LOONGSON3A CROSS=1 CROSS_SUFFIX=mips64el-st-linux-gnu- NO_LAPACKE=1 NO_SHARED=1 BINARY=32
Debug version
A debug version can be built using make DEBUG=1.
Compile with MASS support on Power CPU (optional)
The IBM MASS library consists of a set of mathematical functions for C, C++, and Fortran applications that are are tuned for optimum performance on POWER architectures. OpenBLAS with MASS requires a 64-bit, little-endian OS on POWER. The library can be installed as shown:
-
On Ubuntu:
wget -q http://public.dhe.ibm.com/software/server/POWER/Linux/xl-compiler/eval/ppc64le/ubuntu/public.gpg -O- | sudo apt-key add - echo "deb http://public.dhe.ibm.com/software/server/POWER/Linux/xl-compiler/eval/ppc64le/ubuntu/ trusty main" | sudo tee /etc/apt/sources.list.d/ibm-xl-compiler-eval.list sudo apt-get update sudo apt-get install libxlmass-devel.8.1.5 -
On RHEL/CentOS:
wget http://public.dhe.ibm.com/software/server/POWER/Linux/xl-compiler/eval/ppc64le/rhel7/repodata/repomd.xml.key sudo rpm --import repomd.xml.key wget http://public.dhe.ibm.com/software/server/POWER/Linux/xl-compiler/eval/ppc64le/rhel7/ibm-xl-compiler-eval.repo sudo cp ibm-xl-compiler-eval.repo /etc/yum.repos.d/ sudo yum install libxlmass-devel.8.1.5
After installing the MASS library, compile OpenBLAS with USE_MASS=1.
For example, to compile on Power8 with MASS support: make USE_MASS=1 TARGET=POWER8.
Install to a specific directory (optional)
Use PREFIX= when invoking make, for example
make install PREFIX=your_installation_directory
The default installation directory is /opt/OpenBLAS.
Supported CPUs and Operating Systems
Please read GotoBLAS_01Readme.txt.
Additional supported CPUs
x86/x86-64
- Intel Xeon 56xx (Westmere): Used GotoBLAS2 Nehalem codes.
- Intel Sandy Bridge: Optimized Level-3 and Level-2 BLAS with AVX on x86-64.
- Intel Haswell: Optimized Level-3 and Level-2 BLAS with AVX2 and FMA on x86-64.
- AMD Bobcat: Used GotoBLAS2 Barcelona codes.
- AMD Bulldozer: x86-64 ?GEMM FMA4 kernels. (Thanks to Werner Saar)
- AMD PILEDRIVER: Uses Bulldozer codes with some optimizations.
- AMD STEAMROLLER: Uses Bulldozer codes with some optimizations.
MIPS64
- ICT Loongson 3A: Optimized Level-3 BLAS and the part of Level-1,2.
- ICT Loongson 3B: Experimental
ARM
- ARMv6: Optimized BLAS for vfpv2 and vfpv3-d16 (e.g. BCM2835, Cortex M0+)
- ARMv7: Optimized BLAS for vfpv3-d32 (e.g. Cortex A8, A9 and A15)
ARM64
- ARMv8: Experimental
- ARM Cortex-A57: Experimental
PPC/PPC64
- POWER8: Optmized Level-3 BLAS and some Level-1, only with
USE_OPENMP=1
IBM zEnterprise System
- Z13: Optimized Level-3 BLAS and Level-1,2 (double precision)
Supported OS
- GNU/Linux
- MinGW or Visual Studio (CMake)/Windows: Please read https://github.com/xianyi/OpenBLAS/wiki/How-to-use-OpenBLAS-in-Microsoft-Visual-Studio.
- Darwin/macOS: Experimental. Although GotoBLAS2 supports Darwin, we are not macOS experts.
- FreeBSD: Supported by the community. We don't actively test the library on this OS.
- OpenBSD: Supported by the community. We don't actively test the library on this OS.
- DragonFly BSD: Supported by the community. We don't actively test the library on this OS.
- Android: Supported by the community. Please read https://github.com/xianyi/OpenBLAS/wiki/How-to-build-OpenBLAS-for-Android.
Usage
Statically link with libopenblas.a or dynamically link with -lopenblas if OpenBLAS was
compiled as a shared library.
Setting the number of threads using environment variables
Environment variables are used to specify a maximum number of threads. For example,
export OPENBLAS_NUM_THREADS=4
export GOTO_NUM_THREADS=4
export OMP_NUM_THREADS=4
The priorities are OPENBLAS_NUM_THREADS > GOTO_NUM_THREADS > OMP_NUM_THREADS.
If you compile this library with USE_OPENMP=1, you should set the OMP_NUM_THREADS
environment variable; OpenBLAS ignores OPENBLAS_NUM_THREADS and GOTO_NUM_THREADS when
compiled with USE_OPENMP=1.
Setting the number of threads at runtime
We provide the following functions to control the number of threads at runtime:
void goto_set_num_threads(int num_threads);
void openblas_set_num_threads(int num_threads);
If you compile this library with USE_OPENMP=1, you should use the above functions too.
Reporting bugs
Please submit an issue in https://github.com/xianyi/OpenBLAS/issues.
Contact
- OpenBLAS users mailing list: https://groups.google.com/forum/#!forum/openblas-users
- OpenBLAS developers mailing list: https://groups.google.com/forum/#!forum/openblas-dev
Change log
Please see Changelog.txt to view the differences between OpenBLAS and GotoBLAS2 1.13 BSD version.
Troubleshooting
- Please read the FAQ first.
- Please use GCC version 4.6 and above to compile Sandy Bridge AVX kernels on Linux/MinGW/BSD.
- Please use Clang version 3.1 and above to compile the library on Sandy Bridge microarchitecture. Clang 3.0 will generate the wrong AVX binary code.
- The number of CPUs/cores should less than or equal to 256. On Linux
x86_64(amd64), there is experimental support for up to 1024 CPUs/cores and 128 numa nodes if you build the library withBIGNUMA=1. - OpenBLAS does not set processor affinity by default.
On Linux, you can enable processor affinity by commenting out the line
NO_AFFINITY=1in Makefile.rule. However, note that this may cause a conflict with R parallel. - On Loongson 3A,
make testmay fail with apthread_createerror (EAGAIN). However, it will be okay when you run the same test case on the shell.
Contributing
- Check for open issues or open a fresh issue to start a discussion around a feature idea or a bug.
- Fork the OpenBLAS repository to start making your changes.
- Write a test which shows that the bug was fixed or that the feature works as expected.
- Send a pull request. Make sure to add yourself to
CONTRIBUTORS.md.
Donation
Please read this wiki page.