7b294a99fd
Move common.h back to the top of the file so that SKYLAKEX (from config.h) is defined in time
2021-03-18 21:28:19 +01:00
0934568d9c
Move includes under the ifdef for compilers w/o intrinsics support
2021-03-12 12:42:05 +01:00
09d47af2c0
Optimize zscal function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-03-10 17:15:33 -06:00
ef0238ba2b
Merge pull request #3130 from martin-frbg/issue3128
...
Replace spurious AVX512 requirement in the Haswell srot microkernel with an AVX2/FMA3 guard
2021-03-06 19:15:53 +01:00
a9f6f7ad39
Remove spurious AVX512 requirement and add AVX2/FMA3 guard
2021-03-06 14:35:49 +01:00
41646ed006
Optimize s/dasum function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-03-05 16:22:36 -06:00
0571c3187b
POWER10: Rename mma builtins
...
The LLVM and GCC teams agreed to rename the __builtin_mma_assemble_pair and
__builtin_mma_disassemble_pair built-ins to __builtin_vsx_assemble_pair and
__builtin_vsx_disassemble_pair respectively. This patch is to make
corresponding changes in dgemm kernel. Also made changes in
inputs to those builtins to avoid some potential typecasting issues.
Reference gcc commit id:77ef995c1fbcab76a2a69b9f4700bcfd005d8e62
2021-02-26 20:56:34 -06:00
292d1af1a0
Update omatcopy_rt.c
2021-02-24 09:34:14 +01:00
325b398e3c
Update omatcopy_rt.c
2021-02-24 09:13:12 +01:00
6f5667b4d4
Enable optimized S/D OMATCOPY_RT
2021-02-24 09:03:41 +01:00
cceeee7806
Add optimized omatcopy_rt
2021-02-24 09:00:54 +01:00
0a4546b742
Typo fix
2021-02-23 13:14:35 +01:00
b1eed27a54
Replace naive omatcopy_rt with 4x4 blocked implementation
...
as suggested by MigMuc in issue 2532
2021-02-22 21:35:42 +01:00
47691c031f
Use Haswell optimizations for Zen as well
2021-02-11 09:26:15 +01:00
ce7ddd8921
Use Haswell optimizations for Zen as well
2021-02-11 09:25:36 +01:00
950c047b49
Use Haswell optimizations for Zen as well
2021-02-11 09:24:51 +01:00
46509953a9
Use Haswell optimizations for Zen as well
2021-02-11 09:24:16 +01:00
db348dcff2
Enable optimized srot/drot kernels from Haswell
2021-02-11 09:23:05 +01:00
2056ffc227
Optimize cscal function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-01-29 13:51:43 -06:00
3ede843d50
Optimize s/dscal function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-01-24 07:48:28 -06:00
69a5558203
Merge pull request #3059 from Guobing-Chen/BF16_gemm
...
Initial code for Cooperlake BF16 GEMM kernel
2021-01-23 19:08:05 +01:00
d6905403e3
Merge pull request #3068 from alexhenrie/scan-build
...
scan-build fixes
2021-01-23 19:06:29 +01:00
439b93f6d2
Optimize s/drot function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-01-21 13:24:45 -06:00
eff7c9166e
Optimize cdot function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-01-15 13:40:34 -06:00
202fc9e8ed
Fix uninitialized argument value in dasum_k
2021-01-14 19:40:31 -07:00
e378b24487
Merge pull request #3067 from albertziegenhagel/fix-generic-cmake
...
Fix building "generic" TRMM kernel with CMake
2021-01-14 21:35:19 +01:00
e3f4063683
Fix building "generic" TRMM kernel with CMake
...
The CMake "TARGET_CORE" variables stores the "generic" target name in all lowercase letters, but gets compared to an all uppercase string, which results in the wrong TRMM kernel being selected.
This commit converts the TARGET_CORE to all uppercase before comparing its value to make sure case mismatches are not an issue in the future anymore.
2021-01-14 10:00:49 +01:00
b716c0ef01
Add workaround for NVIDIA HPC
2021-01-12 16:51:35 +01:00
2efa3b70dc
Add workaround for NVIDIA HPC
2021-01-12 16:49:39 +01:00
49959d4f1c
Add workaround for NVIDIA HPC
2021-01-12 16:47:15 +01:00
0f27a03607
Add workaround for NVIDIA HPC mishandling of the asm DOT kernels
2021-01-12 16:39:35 +01:00
c2a8ebfe69
Add workaround for NVIDIA HPC mishandling of the asm DOT kernels
2021-01-12 16:38:51 +01:00
43aac5bacc
Support NVIDIA HPC compiler
2021-01-12 16:36:12 +01:00
b0beb0b1ca
Initial code for Cooperlake BF16 GEMM kernel
2021-01-11 02:15:21 +08:00
601b711c78
Optimize swap function for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2021-01-08 08:01:36 -06:00
1b2508362b
arm64: Fix nrm2 for input vectors with Inf
...
Fix double precision nrm2 kernels returning NaN when the
input vectors contain Inf/-Inf.
2021-01-01 02:49:37 -08:00
3559c5d7a2
Merge pull request #3048 from martin-frbg/issue2998
...
Temporarily revert to the old NRM2 kernels for ThunderX2/3 and NeoverseN1
2020-12-21 13:30:08 +01:00
8631e2976a
Temporarily revert to the old nrm2 kernels
2020-12-21 07:45:13 +01:00
2768bc1764
Temporarily revert to the old nrm2 kernels
2020-12-21 07:42:51 +01:00
6f4698ee1f
Temporarily revert to the old nrm2 kernel
2020-12-21 07:41:18 +01:00
114eb159a4
Disable FMA intrinsics in the srot kernel when the compiler is PGI/NVIDIA
2020-12-19 22:15:58 +01:00
005cce5507
Amend SkylakeX options to support the NVIDIA compiler
2020-12-19 22:11:49 +01:00
c73d8ee40d
Conditionally add -mfma to compiler options where needed
2020-12-17 11:34:05 +01:00
2fb11f873b
POWER10: Improve copy performance
...
This patch aligns the stores to 32 byte boundary for scopy and dcopy
before entering into vector pair loop. For ccopy, changed the store
instructions to stxv to improve performance of unaligned cases.
2020-12-13 10:41:45 -06:00
043128cbe5
Merge pull request #3029 from RajalakshmiSR/axpyp10
...
POWER10: Improve axpy performance
2020-12-10 22:49:28 +01:00
3331ca492d
Merge pull request #3021 from austinpagan/trsm_p10
...
POWER: Added special unrolled vectorized versions of "Solve" for specific si…
2020-12-10 19:42:54 +01:00
346e30a46a
POWER10: Improve axpy performance
...
This patch aligns the stores to 32 byte boundary for saxpy and daxpy
before entering into vector pair loop. Fox caxpy, changed the store
instructions to stxv to improve performance of unaligned cases.
2020-12-10 11:51:42 -06:00
4b548857d6
Add msa support for loongson
...
1. Using core loongson3r3 and loongson3r4 for loongson
2. Add DYNAMIC_ARCH for loongson
Change-Id: I1c6b54dbeca3a0cc31d1222af36a7e9bd6ab54c1
2020-12-09 10:28:46 +08:00
7f11e33e8d
Merge pull request #3025 from TiredNotTear/develop
...
MIPS: Fix two bugs
2020-12-08 09:39:27 +01:00
53e0837809
Merge pull request #3022 from jinboson/develop
...
Fix test errors reported by cblas_cgemm & cblas_ctrmm
2020-12-07 08:09:11 +01:00
ad38bd0e89
Fix failed cgemv and zgemv test case after using msa optimization
...
The cgemv and zgemv test case will call cgemv_n/t_msa.c zgemv_n/t_msa.c files in MIPS environment.
When the macro CONJ is defined, the calculation result will be wrong due to the wrong definition of OP2.
This patch updates the value of OP2 and passes the corresponding test.
2020-12-07 10:25:01 +08:00
47b639cc9b
Fix failed sswap and dswap case by using msa optimization
...
The swap test case will call sswap_msa.c and dswap_msa.c files in MIPS environmnet.
When inc_x or inc_y is equal to zero, the calculation result of the two functions will be wrong.
This patch adds the processing of inc_x or inc_y equal to zero, and the swap test case has passed.
2020-12-07 10:24:49 +08:00
b660008c7e
Work around DOT and SWAP test failures
2020-12-06 19:15:37 +01:00
f8346603cf
Fix compilation with SolarisStudio
2020-12-06 19:14:16 +01:00
65de6f5957
Fix test errors reported by cblas_cgemm & cblas_ctrmm
...
The file cgemm_kernel_8x4_msa.c holds the MSA optimization
codes of cblas_cgemm and cblas_ctrmm. It defines two
macros: CGEMM_SCALE_1X2 and CGEMM_TRMM_SCALE_1X2. The pc1
array index in the two macros should be 0 and 1.
2020-12-05 15:08:17 +08:00
213c0e7abb
Added special unrolled vectorized versions of "Solve" for specific sizes,
...
in DTRSM and STRSM, to improve performance in Power9 and Power10.
2020-12-04 17:07:06 -06:00
441c08c9ff
Merge pull request #3016 from xiegengxin/complex-asum
...
Improve the performance of zasum and casum with AVX512 intrinsic
2020-12-04 22:07:16 +01:00
0cb7a403b2
fix error declare function blas_level1_thread_with_return_value
2020-12-02 09:51:52 +08:00
b766c1e9bb
Improve the performance of zasum and casum with AVX512 intrinsic
2020-12-01 16:49:26 +08:00
7d46e31de1
POWER10: Optimize dgemv_n
...
Handling as 4x8 with vector pairs gives better performance than
existing code in POWER10.
2020-11-29 15:28:28 -06:00
f1bf040b25
Merge pull request #2988 from xiegengxin/smp-asum
...
Improve the performance of dasum and sasum when SMP is defined
2020-11-22 12:24:13 +01:00
7037849498
Merge branch 'develop' into risc-v
2020-11-22 16:04:50 +08:00
7e9cb39a25
Merge pull request #2981 from Qiyu8/fix-sum
...
Fix sum optimize issues
2020-11-16 08:40:46 +01:00
d6e7e05bb3
Improve the performance of dasum and sasum when SMP is defined
2020-11-13 14:20:52 +08:00
ae0b1dea19
modify system.cmake to enable fma flag
2020-11-13 10:20:24 +08:00
e0dac6b53b
fix the CI failure of target specific option mismatch
2020-11-12 20:31:03 +08:00
e5c2ceb675
fix the CI failure of lack the head
2020-11-12 17:35:17 +08:00
a87e537b8c
modify macro
2020-11-11 15:53:48 +08:00
5bc0a7583f
only FMA3 and vector larger than 128 have positive effects.
2020-11-11 15:18:01 +08:00
8c0b206d4c
Optimize the performance of rot by using universal intrinsics
2020-11-11 14:33:12 +08:00
c4c591ac5a
fix sum optimize issues
2020-11-10 16:16:38 +08:00
fc35b72ae1
Refs #2899
...
Merge branch 'openblas-open-910' of git://github.com/damonyu1989/OpenBLAS into damonyu1989-openblas-open-910
2020-11-10 09:38:04 +08:00
913cc9a4ca
Merge branch 'develop' into risc-v
2020-11-10 09:18:25 +08:00
ff16329cb7
Merge pull request #2972 from xiegengxin/rot-intrinsic
...
Improve the performance of rot by using AVX512 and AVX2 intrinsic
2020-11-08 22:43:00 +01:00
110c7a6de0
Merge pull request #2979 from RajalakshmiSR/dot_power10
...
Optimize sdot/ddot for POWER10
2020-11-08 10:19:34 +01:00
6e364981a8
Optimize sdot/ddot for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2020-11-07 15:21:58 -06:00
b976a0bf40
Remove previous workaround for compiler flags related to cpu capabilities in x86_64 DYNAMIC_ARCH builds
2020-11-07 20:39:56 +01:00
ff74319ea5
Merge pull request #2977 from martin-frbg/issue2976
...
Fix macro name used in ifdef for POWERPC/PGI
2020-11-07 14:41:34 +01:00
28d2dfe2b3
Fix macro name used in ifdef
2020-11-07 12:17:49 +01:00
725ffbf041
fix typo
2020-11-05 16:25:17 +08:00
d9ba49165a
Improve the performance of rot by using AVX512 and AVX2 intrinsic
2020-11-05 15:12:36 +08:00
dd7a9cc5bf
POWER10: Change dgemm unroll factors
...
Changing the unroll factors for dgemm to 8 shows improved performance with
POWER10 MMA feature. Also made some minor changes in sgemm for edge cases.
2020-10-31 18:28:57 -05:00
b435491885
Optimize caxpy for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2020-10-29 14:57:51 -05:00
a7b1f9b1bb
Implementation of BF16 based gemv
...
1. Add a new API -- sbgemv to support bfloat16 based gemv
2. Implement a generic kernel for sbgemv
3. Implement an avx512-bf16 based kernel for sbgemv
Signed-off-by: Chen, Guobing <guobing.chen@intel.com>
2020-10-29 02:08:23 +08:00
67f39ad813
Merge pull request #2939 from thrasibule/Makefile_cleanup
...
reuse variables defined in Makefile.system
2020-10-28 09:38:40 +01:00
c24ba8b1dd
Optimize saxpy for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores.
2020-10-26 13:24:59 -05:00
6f9460f0f6
Merge pull request #2937 from martin-frbg/pwr-buffersz
...
Increase and unify BUFFERSIZE on POWER;fix gcc inline warning
2020-10-23 07:15:32 +02:00
1917a4e7b8
reuse variables defined in Makefile.system
2020-10-22 22:04:25 -04:00
34c3c407ef
label always_inline function as inline to silence a gcc warning
2020-10-22 22:14:26 +02:00
2e48d560ba
Fix compiler version check
2020-10-22 16:23:29 +02:00
ad745c0bae
Optimize scopy/ccopy for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores. Also reorganized all variants of copy functions
to make use of same kernel.
2020-10-21 09:53:45 -05:00
4a1d00f589
Fix build with -Werror=return-type
...
dgemm_tcopy_16_skylakex.c CNAME function should return an int, add a
return 0 similar to other files.
2020-10-21 08:43:39 +02:00
b073d759d0
x86_64: clobber all xmm registers after vzeroupper
...
As observed using GCC 10 using -march=native -ftree-vectorize
on Knights Landing, it is now smart enough to find clobbers inside
non-inlined static functions.
In particular, sgemv counted on a kernel to preserve the whole
%ymm2 register (since it was not in the clobber list), but the top
part was destroyed by vzeroupper. This caused many tests to fail.
This patch makes sure all xmm (and ymm/zmm by extension) registers
are listed as clobbered to avoid this happening, as most kernels
already did correctly in fact.
2020-10-20 02:16:47 +00:00
dc6e44c3f8
Merge pull request #2916 from martin-frbg/issue2911
...
Clean up duplicate definitions in POWER8 kernels and fix power10 option passing
2020-10-19 23:33:31 +02:00
a61c086408
Fix spurious trailing whitespace in comment
2020-10-19 09:12:12 +02:00
03e781b766
sgemm_direct_skylakex: fix 75eeb26 regression.
...
The
`#if defined(SKYLAKEX) || defined (COOPERLAKE)`
from that commit was before #include "common.h" so caused the
compiled function to be empty, returning garbage results for
qualifying sgemm's on those architectures.
Closes #2914
2020-10-18 19:58:07 +00:00
f1a4071d8c
Clean up STACKSIZE redefinition
2020-10-18 19:41:43 +02:00
97cf10062f
Clean up STACKSIZE redefinition
2020-10-18 19:39:18 +02:00
17e288e18d
Clean up STACKSIZE redefinition
2020-10-18 19:37:04 +02:00
c1422f3e46
Clean up STACKSIZE redefinition
2020-10-18 19:31:01 +02:00
d85b24e103
Clean up STACKSIZE redefinition
2020-10-18 19:29:45 +02:00
d7ba7679b6
Merge branch 'develop' into risc-v
2020-10-16 23:27:38 +08:00
df70667043
fix core list for sse/sse2
2020-10-16 09:55:48 +02:00
f071d1207a
add sse2
2020-10-15 22:10:32 +02:00
dc6cefd2f5
Expressly enable -msse for 32bit DYNAMIC_ARCH kernels
2020-10-15 20:16:15 +02:00
c339c40c01
Silence a redefinition warning
2020-10-15 19:08:12 +02:00
10379fc83b
Use ifdef instead of if
2020-10-15 19:05:37 +02:00
4c25910da0
Merge pull request #2896 from martin-frbg/intrin-double
...
Add compiler flag for SSE4 where available
2020-10-15 11:12:35 +02:00
ef8e7d0279
Add the support for RISC-V Vector.
...
Change-Id: Iae7800a32f5af3903c330882cdf6f292d885f266
2020-10-15 16:09:02 +08:00
ae6ac83991
Revert "add double precision SSE"
2020-10-15 08:37:02 +02:00
4fac91ef37
adapt arm platform
2020-10-15 11:08:10 +08:00
bfdf4b56da
Add double precision universal intrinsics for X86/ARM
2020-10-15 10:29:42 +08:00
ebf0470fc2
add sse4.1 for DYNAMIC_ARCH kernels
2020-10-14 20:34:33 +02:00
c9c3ae07af
Add double precision operations
2020-10-14 18:10:45 +02:00
756802df61
Merge pull request #2890 from martin-frbg/s-d-sum
...
Revert special handling of Windows xNRM2 and enable C+intrinsics kern…
2020-10-14 09:02:03 +02:00
0826d68f93
POWER10: Change the packing format for bfloat16
...
As the new MMA instructions need the inputs in 4x2 order for bfloat16,
changing the format in copy/packing code. This avoids permute instructions
in the gemm kernel inner loop.
2020-10-13 16:05:10 -05:00
b5d30b390d
Fix build issues with bfloat16
...
This patch fixes compilation errors due to recent renaming from SH to SB
with BUILD_BFLOAT16.
2020-10-13 11:00:22 -05:00
fecedc9c69
Add -mssse3
2020-10-13 11:55:41 +02:00
0eacbca85f
Add Haswell and Zen to temporary sse3 whitelist
2020-10-13 11:42:39 +02:00
6999086a2b
whitelist SANDYBRIDGE for SSE3
2020-10-13 10:32:19 +02:00
8d2df7d066
Revert special handling of Windows xNRM2 and enable C+intrinsics kernel for SSUM/DSUM
2020-10-13 00:14:29 +02:00
08929430cd
Merge pull request #2886 from martin-frbg/issue_2767
...
Rename "HALF" precision functions (sh prefix) to "BFLOAT16" with "sb" prefix
2020-10-13 00:04:35 +02:00
0c84ffe05f
Merge pull request #2881 from mattip/fninit
...
add fninit to reset fpu registers before assembler routines
2020-10-12 23:50:41 +02:00
403eb513a0
use emms instead, add WIN guards
2020-10-12 18:15:01 +03:00
0ed1f07660
Optimize the performance of sum by using universal intrinsics
2020-10-12 19:48:53 +08:00
3aecafad80
Change "HALF" and "sh" to "BFLOAT16" and "sb"
2020-10-12 00:00:55 +02:00
756062afa5
Rename "HALF" and "sh" to "BFLOAT16" and "sb"
2020-10-11 23:56:17 +02:00
2061f7fdff
Rename "HALF" and "sh" to "BFLOAT16" and "sb"
2020-10-11 23:54:53 +02:00
dc8a1afa63
Rename "HALF" and "sh" to "BFLOAT16" and "sb"
2020-10-11 23:53:50 +02:00
fd94236042
Rename "HALF" and "sh" to "BFLOAT16" and "sb"
2020-10-11 23:42:07 +02:00
68ce719fac
Rename shdot_microk_cooperlake.c to sbdot_microk_cooperlake.c
2020-10-11 23:41:13 +02:00
d7dd9b396c
Rename shdot.c to sbdot.c
2020-10-11 23:40:43 +02:00
9ae80490e0
rename "HALF" and "sh" to "BFLOAT16" and "sb"
2020-10-11 23:39:42 +02:00
d314d1f49f
Rename shgemm_kernel_power10.c to sbgemm_kernel_power10.c
2020-10-11 23:37:38 +02:00
c589c3e2a1
Merge pull request #2882 from martin-frbg/issue2709
...
Use generic C for (D/Z)NRM2 on Windows x86_64
2020-10-11 22:22:30 +02:00
ec638a82bf
Merge pull request #2852 from martin-frbg/issue2588-cmake
...
Support building only a subset of variable types
2020-10-11 22:21:33 +02:00
6b6adf8a4a
Allow compiling only a subset of kernels for specific variable types
2020-10-11 14:52:09 +02:00
ac653c94f3
Merge branch 'develop' into issue2588-cmake
2020-10-11 13:57:07 +02:00
7a53128481
Add whitelist of DYNAMIC_ARCH kernels for which -msse3 needs to be enabled
2020-10-11 01:06:46 +02:00
e1b7123bbe
Merge pull request #2867 from Qiyu8/usimd-floatdot
...
Optimize the performance of dot by using universal intrinsics in X86/ARM
2020-10-10 12:10:25 +02:00
f32d34a015
add sse3 compiler flag
2020-10-10 10:36:15 +08:00
7812486091
Use generic C for D/Z nrm2 kernels on Windows to work around fpu exception bug
2020-10-06 21:33:16 +02:00
a5b164946c
add fninit to reset fpu registers before assembler routines
2020-10-05 22:13:25 +03:00
d2333e7842
aarch64 fix std=c18 compilation
2020-10-03 18:00:34 +03:00
60e6c68e38
Adapt ARM architect
2020-09-29 16:36:14 +08:00
1b1a757f5f
Optimize the performance of dot by using universal intrinsics in X86/ARM
2020-09-28 20:36:53 +08:00
2df4235e00
Optimize dcopy/zcopy for POWER10
...
This patch makes use of new POWER10 vector pair instructions for
loads and stores. Tested in simulator and no new failures.
2020-09-27 21:42:32 -05:00
dfbc62ef7e
Support building only a subset of types
2020-09-22 23:25:59 +02:00
14f7dad3b7
performance improved
2020-09-22 16:52:15 +08:00
325b539c26
Optimize the performance of daxpy by using universal intrinsics
2020-09-22 10:38:35 +08:00
Martin Kroeker
Rajalakshmi Srinivasaraghavan
Alex Henrie
Albert Ziegenhagel
Chen, Guobing
Ashwin Sekhar T K
gxw
Hao Chen
Jin Bo
Gordon Fossum
Gengxin Xie
Xianyi Zhang
Qiyu8
Guillaume Horel
İsmail Dönmez
Bart Oldeman
Zhang Xianyi
damonyu
Matti Picus
User User-User