66b39b835c
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:44:45 +02:00
bb6d6735bf
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:44:15 +02:00
d18efaed20
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:43:43 +02:00
99f6d31ed5
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:42:55 +02:00
7de9335c56
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:42:09 +02:00
437c0bf2b4
Merge pull request #3843 from Mousius/switch-ratio
...
Propagate SWITCH_RATIO to DYNAMIC_ARCH builds
2023-04-19 11:51:54 +02:00
ec334e69dc
Use SVE kernel for SGEMM/DGEMM on Arm(R) Neoverse(TM) V1
...
This re-spins #3869 with some additional copy unrolling which helps maintain SYRK performance.
After #3868 , the SVE kernels represent a pretty good boost.
This re-uses ARMV8SVE as a base and I'm going to incrementally move everything to use ARMV8SVE in additional patches (as well as fix up anything that's not already in ARMV8SVE).
2023-04-17 17:38:42 +01:00
32f2fafde7
Propagate SWITCH_RATIO to DYNAMIC_ARCH builds
...
Previously dynamic builds were either using the default SWITCH_RATIO
or one from the higher level architecture; this patch ensures the
dynamic builds can use this parameter as well.
2023-04-17 15:34:12 +01:00
44164e3a3d
revert "move alpha out of register 18" (out of PR scope, no SVE on Apple hw)
2023-04-17 14:23:13 +02:00
8be68fa7f4
move declaration of sca to really keep the compiler from throwing it out (for now)
2023-04-15 12:02:39 +02:00
3727672a74
Improve workaround and keep compilers from optimizing it out
2023-04-13 18:07:52 +02:00
108a21e47a
Move ALPHA out of register 18 (reserved on OSX)
2023-04-13 18:05:14 +02:00
0b1acb0ba3
Move ALPHA_I out of register 18 (reserved on OSX)
2023-04-13 18:03:35 +02:00
c7bbad09ad
Move ALPHA_I out of register 18 (reserved on OSX)
2023-04-13 18:00:47 +02:00
cda29633a3
move ALPHA_I out of register 18 (reserved on OSX)
2023-04-13 17:59:48 +02:00
09ace3cf23
Merge pull request #3846 from lilh9598/sbgemm_opt
...
Improve the performance of sbgemm_tcopy on neoversen2
2023-03-26 19:04:57 +02:00
1374a2d08b
This PR adapts latest spec changes
...
Add prefix (_riscv) for all riscv intrinsics
Update some intrinsics' parameter, like vfredxxxx, vmerge
2023-03-19 23:59:03 -07:00
19f17c8bc6
Merge pull request #3893 from HellerZheng/develop
...
add riscv level3 C,Z kernel functions.
2023-03-15 10:17:13 +08:00
cb0a70e0e2
dot.c early bail fix
2023-03-02 09:51:10 +00:00
9b61be4545
factoring riscv64/dot.c fix into separate PR as requested
2023-03-01 17:40:42 +00:00
2406958629
* update intrinsics to match latest spec at https://github.com/riscv-non-isa/rvv-intrinsic-doc (in particular, __riscv_ prefixes for rvv intrinsics)
...
* fix multiple numerical stability and corner case issues
* add a script to generate arbitrary gemm kernel shapes
* add a generic zvl256b target to demonstrate large gemm kernel unrolls
2023-02-24 10:45:03 +00:00
38d6fb4225
Fix dependencies in builds with specified subsets of precision types
2023-02-23 23:12:06 +01:00
e412bee313
fix GEMM kernel dependencies in builds that use only a subset of precisions
2023-02-22 00:37:14 +01:00
d80adf253e
make SSYMV available to BUILD_DOUBLE-only builds
2023-02-22 00:30:20 +01:00
5481c328e8
fix DYNAMIC_ARCH builds that use only a subset of precisions
2023-02-22 00:28:25 +01:00
63cf4d0166
add riscv level3 C,Z kernel functions.
2023-02-01 19:13:44 -08:00
c19dff0a31
Fix T-Head RVV intrinsic API changes.
2023-01-25 19:33:32 +08:00
5a9cd87794
Merge pull request #3868 from Mousius/sve-prefetch
...
Remove prefetches from SVE kernels
2022-12-24 10:52:29 +01:00
1361229291
Remove prefetches from SVE kernels
...
This is a precursor to enabling the SVE kernels for Arm(R) Neoverse(TM)
V1 which has 256-bit SVE. Testing revealed that the SVE kernel was
actually worse in some cases than the existing kernel which seemed odd -
removing these prefetches the underlying architecture seems to do a better job
😸
2022-12-16 14:43:09 +00:00
60e49b851c
Fix typo in clobber list, should be xmm14 instead of ymm14.
2022-12-06 16:30:46 -05:00
4afe1439a1
Fix skylake fallback kernel name for old compilers.
2022-12-06 16:09:54 -05:00
5ceca1a4d8
Add sscal.c + microkernels for Haswell, Zen, Skylake and newer.
...
Unlike [dcz]scal, sscal still used the original GotoBLAS SSE code from scal_sse.S.
This code follows dscal as closely as possible, except for the inc_x > 1 code
for which a plain C loop is used much like the one in cscal.c, instead of an
adaptation of the SSE2 asm code of dscal.c (I tried but the performance wasn't
better than the plain C loop).
2022-12-06 14:05:49 -05:00
729af6406f
bugfix for sbgemm_ncopy_8_neoversen2
2022-12-05 05:10:18 -05:00
042e3c0e7c
Merge pull request #3848 from bartoldeman/dscal-haswell-ymm
...
dscal: use ymm registers in Haswell microkernel
2022-12-05 08:56:08 +01:00
e5313f53d5
Merge branch 'develop' of https://github.com/HellerZheng/OpenBLAS_riscv_x280 into HellerZheng-develop
2022-12-03 12:00:52 +08:00
5c3169ecd8
dscal: use ymm registers in Haswell microkernel
...
Using 256-bit registers in dscal makes this microkernel consistent with
cscal and zscal, and generally doubles performance if the vector fits in
L1 cache.
2022-12-01 07:48:05 -05:00
eea006a688
Wrap SVE header with __has_include check
2022-12-01 12:07:55 +00:00
fd4f52c797
Add SVE implementation for sdot/ddot
...
This adds an SVE implementation to sdot/ddot when available, falling back to the previous Advanced SIMD kernel where there's no SVE implementation for the kernel.
All the targets were essentially treating `dot_thunderx2t99.c` as the Advanced SIMD implementation so I've renamed it to better fit with the feature detection.
2022-12-01 12:07:50 +00:00
fdac8a97c1
Add sbgemm_ncopy_8 and sbgemm_tcopy_4
2022-11-29 04:46:14 -05:00
135718eafc
Improve the performance of sbgemm_tcopy on neoversen2
2022-11-28 04:17:54 -05:00
4f7b77e08a
Remove unnecessary instructions from Advanced SIMD dot
...
The existing kernel was issuing extra instructions to organise the arguments into the same registers they would usually be in and similarly to put the result into the appropriate register.
This has an impact on smaller sized dots and seemed like a quick fix
2022-11-25 16:19:03 +00:00
3918d8504e
nrm2 simple optimization
2022-11-21 19:06:07 -08:00
943372bdf5
Merge branch 'develop' into develop
2022-11-18 10:12:46 +08:00
f73cfb7e2c
change line endings from CRLF to LF
2022-11-17 09:39:56 +01:00
1688c7da43
change line endings from CRLF to LF
2022-11-16 22:24:01 +01:00
5d0d1c5551
Remove redundant files
2022-11-15 18:22:21 -08:00
bef47917bd
Initial version for riscv sifive x280
2022-11-15 00:06:25 -08:00
6c1043eb41
Add [cz]scal microkernels for SKYLAKEX
...
These are as similar to dscal_microk_skylakex-2.c as possible
for consistency.
Note that before this change SKYLAKEX+ uses generic C functions for
cscal/zscal via commit 2271c350#2610 (which is masked by
commit 086d87a30#3799 disables FMAs (in turn enabled
by `-march=skylake-avx512`) in the plain C code which fixes excessive
LAPACK test failures more nicely.
2022-11-09 08:57:03 -05:00
c9d78dc3b2
Remove excess initializer (leftover from rework of PR 3793)
2022-10-31 16:57:03 +01:00
65338a9493
Merge pull request #3799 from bartoldeman/cscal-zscal-no-fma
...
x86_64: prevent GCC and Clang from generating FMAs in cscal/zscal.
2022-10-30 18:56:10 +01:00
79066b6bf3
Change file name to match the norm and delete useless code.
2022-10-28 17:09:39 +08:00
e7e3aa2948
x86_64: prevent GCC and Clang from generating FMAs in cscal/zscal.
...
If e.g. -march=haswell is set in CFLAGS, GCC generates FMAs by default, which
is inconsistent with the microkernels, none of which use FMAs. These
inconsistencies cause a few failures in the LAPACK testcases, where
eigenvalue results with/without eigenvectors are compared.
Moreover using FMAs for multiplication of complex numbers can give surprising
results, see 22aa81f22aa81f
2022-10-27 18:16:43 -04:00
4989e039a5
Define SBGEMM_ALIGN_K for DYNAMIC_ARCH build
2022-10-27 14:10:26 +08:00
843e9fd0b9
Fix typo error
2022-10-26 17:06:33 +08:00
b00d5b9746
New sbgemm implementation for Neoverse N2
...
1. Use UZP instructions but not gather load and scatter store instructions to get lower latency.
2. Padding k to a power of 4.
2022-10-26 15:09:41 +08:00
f6f35a4288
fix copyobj declarations to work with DYNAMIC_ARCH
2022-09-29 08:47:14 +02:00
b1d69fb3ac
Add MIPS64_GENERIC as a copy of GENERIC
2022-09-17 23:52:32 +02:00
edea1bcfaf
MIPS64: Fixed failed utest dsdot:dsdot_n_1 when TARGET=I6500
2022-09-17 16:43:22 +08:00
101a2c77c3
Fix warnings
2022-09-15 09:19:19 +02:00
23d59baaf1
Add -mfma to -mavx2 for Apple clang, and set AVX2 options for Zen as well
2022-09-13 22:39:27 +02:00
365936ae1b
MIPS64: Using the macro MTC rather than MTC1
2022-09-13 16:39:40 +08:00
739c3c44a7
Work around windows/osx gcc12 x86_64 tree-optimizer problem and add an osx/gcc12 build to Azure CI ( #3745 )
...
Add pragma to disable the gcc tree-optimizer for some x86_64 S and Z kernels with gcc12 on OSX or Windows
2022-09-03 15:01:22 +02:00
bd30120ba7
Merge pull request #3720 from FlyGoat/mips64
...
Make it work on general MIPS64 processors
2022-08-19 20:24:27 +02:00
a50b29c540
Provide a fallback MIPS64_GENERIC target
...
It is really dangerous to fallback to Loongson core on other
MIPS64 processors.
Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
2022-08-12 13:13:28 +01:00
50c4eeb97d
alpha: Remove include of version.h
...
It will be defined by preprocessor argument.
Signed-off-by: Jiaxun Yang <jiaxun.yang@flygoat.com>
2022-08-11 15:02:58 +01:00
802e71bf05
Add const attribute to lsame
2022-08-08 15:15:52 +02:00
fbfe1daf6e
LoongArch64: Add DYNAMIC_ARCH support
2022-07-28 14:28:45 +08:00
cd8e57040c
Merge pull request #3691 from martin-frbg/issue3679-sparc
...
SPARC: fix DNRM2 returning INF instead of zero due to intermediate overflow
2022-07-25 15:41:15 +02:00
6c118b7977
Fix DNRM2 returning INF instead of zero due to intermediate overflow
2022-07-24 17:42:31 +02:00
c43ec53bdd
Merge pull request #3690 from RajalakshmiSR/cdotp10
...
POWER: Fix complex dot function failures
2022-07-19 13:59:16 +02:00
b7c65d08cb
Merge pull request #3689 from RajalakshmiSR/dgemvgcc10
...
POWER10: dgemv builtin rename
2022-07-19 10:25:01 +02:00
06ef015234
fix DNRM2 returning INF instead of zero due to intermediate overflow
2022-07-19 10:19:27 +02:00
a612e78a97
POWER: Fix complex dot function failures
...
There are some test failures in complex dot functions when compiling with gcc12.
The machine constraints used now do not update all the four elements in the
expected result array. Fixing this with a reduced level of optimization.
This is not changing any performance numbers but will be converted to C code in future.
2022-07-18 14:48:43 -05:00
432fd99445
POWER10: dgemv builtin rename
...
Add check to use correct builtin name for older versions
of gcc10 compilers.
2022-07-18 09:48:01 -05:00
4dd05e526b
LoongArch64: Fix dnrm2_tiny testcase failure
2022-07-15 11:18:59 +08:00
cce4b1d956
MIPS64: Fix dnrm2_tiny testcase failure
2022-07-11 19:18:38 +08:00
e12d474780
Eliminate uses of CREAL on left-hand side of assignments
2022-07-05 00:01:09 +02:00
9e29598575
workaround fault with ssq=inf,scale=0
2022-07-02 23:47:17 +02:00
123e0dfb62
Neoverse N2 sbgemm:
...
1. Modify the algorithm to resolve multithreading failures
2. No memory allocation in sbgemm kernel
3. Optimize when alpha == 1.0f
2022-06-29 10:14:21 +08:00
bc3728475f
format code
2022-06-29 10:14:21 +08:00
55d686d41e
neoverse n2 sbgemm:
...
implement ncopy tcopy kernel_8x4
2022-06-29 10:14:21 +08:00
04593bb27c
neoverse n2 sbgemm: init file
2022-06-29 10:14:21 +08:00
be5500e704
Merge pull request #3669 from VFerrari/fix_small_matrix_kernel
...
POWER: fix issues with the small matrix kernel
2022-06-28 16:09:36 +02:00
92275a7902
Merge pull request #3642 from nursik/develop
...
Add ARM64 support for Windows
2022-06-28 16:05:11 +02:00
cac634fce3
POWER10: Fix multithreading check when USE_THREAD=0
...
This patch fixes an issue when OpenBLAS is compiled for TARGET=POWER10
and the flag USE_THREAD is set to 0.
The function `num_cpu_avail` is only available when USE_THREAD=1,
so SMP is defined.
2022-06-25 03:46:46 -03:00
9283c7c0b5
Merge pull request #3655 from RajalakshmiSR/zgemmasmp10
...
POWER10: Fix ZGEMM testcase failures
2022-06-18 20:52:26 +02:00
f191bc652b
POWER10: Fix ZGEMM testcase failures
...
This patch fixes storing and restoring non volatile registers
in zgemm POWER10 kernel.
2022-06-17 08:18:08 -05:00
8419d538ff
POWER10: convert dgemv inline assembly
...
This patch makes use of compiler builtins and matches with assembly
performance. Tested with clang14 and gcc12.
2022-06-09 10:42:57 -05:00
5e9a912591
Merge branch 'develop' into risc-v
2022-06-06 14:12:09 +08:00
968e1f51d8
Update RISC-V Intrinsic API.
2022-06-06 13:52:21 +08:00
1bb7993a97
Fix MSVC ARM64 build. Add generic kernel for ARM64
2022-06-02 16:53:54 +02:00
dc49edd4e6
Revert "roll back DGEMM kernel ... for DYNAMIC_ARCH"
2022-05-20 11:23:30 +02:00
b62173c5a0
POWER10: Changing store instructions for Level1 functions
...
This patch changes 32 bytes stores to two 16 bytes stores
to fix a recent degradation due to 32 bytes stores.
2022-05-12 11:17:33 -05:00
84cb58b7fb
Fix generator rules for ?laswp_ncopy and ?neg_tcopy
2022-04-30 15:28:38 +02:00
05dcfa176e
fix undefined prefetchsizes
2022-04-16 10:04:27 +02:00
2bbb9f05c7
fix undefined prefetchsize
2022-04-16 10:00:10 +02:00
115bc9b98f
CortexX1 is ARMV8 like A7x
2022-03-28 17:28:29 +02:00
b3b4672c30
Add initial support for Phytium FT2000 series and ARMV9 Cortex 510/710/X1/X2
2022-03-27 15:29:20 +02:00
40302558ed
Remove extraneous (and wrong) definition of sbgemm_r on x86_64
2022-03-23 20:05:32 +01:00
5cc1111383
fix unsafe read of Y in assembly kernel
2022-03-11 11:56:33 -06:00
45786b05da
Merge branch 'develop' into risc-v
2022-02-28 11:48:02 +08:00
225683218c
Small Matrix: use proper inline asm input constraint for AVX512 mask
2022-02-28 03:22:31 +00:00
9c626e466e
really fix definition of SHUFFLE_MAGIC_NO
2022-02-25 15:36:02 +01:00
0698212c8c
Remove stray $
2022-02-25 15:33:02 +01:00
9d7429406f
Declare SHUFFLE_MAGIC_NO as const to placate clang
2022-02-25 10:05:36 +01:00
d9894f45d3
Define sbgemm_r to fix DYNAMIC_ARCH builds
2022-02-25 10:04:00 +01:00
522f809825
Merge pull request #3542 from martin-frbg/issue3540
...
Fix compilation for CooperLake on Windows/clang
2022-02-24 00:00:00 +01:00
abbc947edb
Fix compilation of Skylake AVX512 kernels with GCC 6
2022-02-23 22:51:59 +00:00
c62f8e2c01
Prevent compiler attempts to use k0 as mask register
2022-02-23 20:12:20 +01:00
80eb581c83
Fix non-portable u_int64_t
2022-02-23 20:10:59 +01:00
73ffabe6ba
Guard uses of _mm512_reduce_add_p?
2022-02-23 20:06:14 +01:00
7656aba00e
Merge pull request #3493 from martin-frbg/casts+cleanup
...
WIP casts and cleanups
2022-02-06 23:55:06 +01:00
addc2a7aaa
Add proper defaults for IMIN/IMAX
2022-01-27 19:56:32 +01:00
299d4d70a3
Add default KERNEL file for Elbrus E2K arch
2022-01-22 18:59:36 +01:00
3492bea602
Create Makefile
2022-01-22 18:57:28 +01:00
898cf5faf3
Add Elbrus e2k architecture support
2022-01-22 18:55:10 +01:00
c1c0d5ce1d
Merge pull request #3492 from binebrank/arm_sve_zgemm
...
SVE zgemm&cgemm (and other BLAS 3 complex)
2022-01-18 21:36:33 +01:00
19d435b1b3
update armv8sve + contributors
2022-01-18 08:28:31 +01:00
f158d59087
adapt CMake
2022-01-17 22:36:48 +01:00
b6a445cfd8
adapt Makefile for SVE trsm
2022-01-16 21:40:56 +01:00
0fb6cc07bf
fix ztrsm lt/ut copy
2022-01-16 21:39:57 +01:00
f1315288a8
add sve ztrsm
2022-01-15 22:27:25 +01:00
aaa2b1a861
fix sve dtrsm kernels
2022-01-15 21:02:14 +01:00
8071e179f1
add remaining sve trsm copy kernels
2022-01-11 21:16:38 +01:00
f87468ac91
trsm_lncopy_sve
2022-01-10 21:45:37 +01:00
e8939b3d30
sve trsmRN and trsmRT
2022-01-10 20:42:20 +01:00
098672b51b
add trsm_kernel_LT_sve
2022-01-09 20:11:47 +01:00
be7e55880c
sve trsm_kernel_LN
2022-01-09 19:40:04 +01:00
b6b024232d
Merge pull request #3508 from snadampal/v1_n2
...
OpenBLAS: aarch64: Add neoverse-v1/n2 architecture specifics
2022-01-09 14:50:26 +01:00
19c8f615dc
OpenBLAS: aarch64: Add neoverse-v1/n2 architecture specifics
2022-01-07 00:28:17 +00:00
bb33446b40
fix makefile.L3
2022-01-06 10:26:11 +01:00
f33543d029
combine zchemm into single file
2022-01-05 14:42:37 +01:00
0c91d043ae
adapt CMake for SVE
2022-01-05 14:36:39 +01:00
39ab219704
sve copy functions for cgemm chemm zsymm
2022-01-05 09:12:22 +01:00
18102ae8c3
add cgemm ctrmm sve kernels
2022-01-05 09:09:18 +01:00
87537b8c55
modify sve zgemmcopy kernels
2022-01-05 09:07:28 +01:00
d30157d891
update configuration of kernels for A64FX and ARMV8SVE
2022-01-05 09:00:54 +01:00
07fa6fa3b1
configure Makefile for sve
2022-01-05 08:57:51 +01:00
2e2c02b762
fix sve ztrmm kernel
2022-01-04 14:42:07 +01:00
68c414d3a6
ztrmm sve copy functions
2022-01-04 14:40:59 +01:00
ce329ab686
add sve zhemm copy routines
2022-01-03 15:56:05 +01:00
0140373802
add sve ztrmm
2022-01-02 19:15:33 +01:00
f7b6912868
ztrmm sve copy kernels
2021-12-30 21:00:16 +01:00
40b14e4957
fix zgemm kernel
2021-12-29 11:42:04 +01:00
6ec4aab875
zgemm sve copy routines
2021-12-26 17:05:46 +01:00
878064f394
sve zgemm kernel
2021-12-26 08:44:05 +01:00
683a7548bf
added macros for sve zgemm kernels
2021-12-25 11:46:41 +01:00
7b146e590c
fix function typecast
2021-12-24 20:01:52 +01:00
e9a0e52201
fix function typecast
2021-12-24 20:00:50 +01:00
d1ee6ff73f
fix function typecasts
2021-12-21 18:45:28 +01:00
e3c9947c0f
prepare kernel for sve zgemm
2021-12-21 11:19:27 +01:00
8d9b9c6b2a
loongarch64: Optimize dgemm_kernel
2021-12-21 09:33:06 +08:00
92b7b949dd
fix bug in zscal function
...
memset can not be used in zscal because of
the stride parameters.
Signed-off-by: Wu Zhigang <zhigang.wu@starfivetech.com>
2021-12-15 01:23:30 -08:00
b0a590f4fe
Merge pull request #3475 from wjc404/optimize-A53-dgemm
...
optimize cgemm on ARM cortex A53 & cortex A55
2021-12-12 19:09:08 +01:00
f4d1f0333b
Merge pull request #3474 from rafaelcfsousa/rafael/cmake_power
...
Add CMake support for Power
2021-12-12 19:08:27 +01:00
b610d2de37
optimize cgemm on ARM cortex A53 & cortex A55
2021-12-12 17:22:52 +08:00
697e2752d7
Merge pull request #3464 from binebrank/arm_sve_sgemm
...
Add sgemm part for Arm SVE
2021-12-11 20:35:22 +01:00
a8f62a347b
fix UNROLL_MN and add to targets for SVE
2021-12-11 16:37:23 +01:00
774267fdac
adjust Makefile.L3 for SVE
2021-12-11 16:35:08 +01:00
23a7561353
Fix error cmake (small kernels)
2021-12-09 09:57:39 -06:00
5378046abd
roll back DGEMM kernels to 4x8 when compiling for DYNAMIC_ARCH
2021-12-06 19:43:54 +01:00
a1fea1fe2a
sgemm v2x8 SVE kernel
2021-12-05 18:47:29 +01:00
abe1ce3434
strmm sve v1x8 kernel
2021-12-05 14:03:08 +01:00
54d321d742
Merge pull request #3466 from rafaelcfsousa/rafael/small_matrix_p10
...
[POWER] Add small matrix for sgemm/dgemm on Power10
2021-12-03 12:12:20 +01:00
0882db30a2
Merge pull request #3455 from cenewcombe/develop
...
Fix unsafe read during final iteration of zsymv_L_sse2.S
2021-12-03 10:01:20 +01:00
0de36f7b5c
trmm sve copy fucntions for single precision
2021-11-29 21:25:05 +01:00
c78fdcc80d
[POWER] Add support for SMALL_MATRIX_OPT
2021-11-28 12:41:16 -06:00
86ae89bf33
add sgemm kernel and copy functions for sgemm and ssymm
2021-11-28 18:12:47 +01:00
454edd741c
Merge pull request #3425 from binebrank/arm_sve_dgemm
...
Add dgemm kernel for arm64 SVE
2021-11-26 16:14:55 +01:00
bcfbdc81b2
Merge pull request #3459 from rafaelcfsousa/fix_cmake
...
Fix issues when building OpenBLAS with cmake
2021-11-26 15:19:24 +01:00
1af73ce38e
Adapt CMake for SVE
2021-11-26 10:35:01 +01:00
e7fca060db
Merge pull request #3457 from wjc404/optimize-A53-dgemm
...
MOD: optimize zgemm on cortex-A53/cortex-A55
2021-11-26 10:30:47 +01:00
5c1cd5e0c2
MOD: add comments to a53 zgemm kernel
2021-11-25 22:48:48 +08:00
d5c9353f1b
Modify the order that cmake set the KERNEL variables (generic now is fallback)
2021-11-24 20:08:35 -06:00
9f59b19fcd
MOD: optimize zgemm on cortex-A53/cortex-A55
2021-11-24 21:51:45 +08:00
531a28b6a0
removed unused code (compiler warnings)
2021-11-22 10:12:34 +01:00
9b9cb90bb1
modify Makefile for SVE copy
2021-11-22 09:54:20 +01:00
9388f05a3c
configure SVE Makefile
2021-11-21 18:33:43 +01:00
b58d4f31ab
some clean-up & commentary
2021-11-21 14:56:27 +01:00
b7df500106
Add generic mips32 target
2021-11-20 17:31:51 +01:00
e6ed4be02e
symm SVE copy rutines
2021-11-20 16:35:29 +01:00
feeb8283a5
Fix unsafe read during final iteration of zsymv_L_sse2.S
2021-11-19 14:29:32 -06:00
302f22693a
MOD: optimize normal DGEMM on ARMV8 cortex-A53 & cortex-A55
2021-11-18 21:14:43 +08:00
3c7eed0e53
add remaining trmm copy rutines for SVE
2021-11-14 16:00:10 +01:00
7d996b1c36
dtrmm_utcopy sve function
2021-11-13 18:48:53 +01:00
ab7917910d
add v2x8 kernel + fix sve dtrmm
2021-11-07 20:37:51 +01:00
7093372e32
add ARMV8SVE target
2021-11-01 22:53:21 +01:00
a8fbdbac34
fix sve dgemm kernel + sve dtrmm
2021-10-31 10:24:25 +01:00
746b4f0f17
added SVE ncopy and tcopy
2021-10-30 12:11:44 +02:00
1a10d3e09d
add sve dgemm prototype
2021-10-27 16:37:18 +02:00
22bf5c27ba
Add basic support for the Fujitsu A64FX ( #3415 )
...
* Add initial support for Fujitsu A64FX as generic ARMV8
2021-10-18 15:00:19 +02:00
63a103ba6e
sbgemm: spr: disable small matrix path by default
2021-10-17 19:08:03 -07:00
82194ea9d2
sbgemm: spr: implement otcopy_16
2021-10-17 19:08:03 -07:00
8632380a96
sbgemm: spr: reuse ncopy_16 from cooperlake as incopy
2021-10-17 19:08:03 -07:00
6bc8204ce5
sbgemm: spr: optimization for tmp_c buffer
2021-10-17 19:08:03 -07:00
f018aa342a
sbgemm: spr: kernel handle alpha != 1.0
2021-10-17 19:08:03 -07:00
a52456b168
sbgemm: spr: oncopy: use tile load/store instead
2021-10-17 19:08:03 -07:00
f2485352a6
sbgemm: spr: only load A once in tail_k handling
2021-10-17 19:08:03 -07:00
9ab33228bb
sbgemm: spr: process k2 and odd k at the same time
2021-10-17 19:08:03 -07:00
10d52646e2
sbgemm: spr: oncopy: avoid handling too much pointer at a time
2021-10-17 19:08:03 -07:00
88154ed02d
sbgemm: spr: reduce tile conf loading by seperate tail k handling
2021-10-17 19:08:03 -07:00
a70bfb52d5
sbgemm: spr: kernel works for NN case when alpha is 1.0
2021-10-17 19:08:03 -07:00
6051c86741
sbgemm: spr: kernel works for m32 in NN case
2021-10-17 19:08:03 -07:00
d0b253ac6e
sbgemm: spr: implement oncopy_16
2021-10-17 19:08:03 -07:00
1d48b7cb16
sbgemm: spr: add dummy source files
2021-10-17 19:08:03 -07:00
3dc6052c7e
initial support for Sapphire Rapids platform
2021-10-12 01:30:40 -07:00
8c20ca345a
Use Neoverse's current mix of ThunderX2 kernels for Vortex as well
2021-10-06 11:06:43 +02:00
8e4c209002
Merge pull request #3398 from kavanabhat/aix_p10_gnuas
...
Big Endian Changes for Power10 kernels
2021-10-05 18:59:47 +02:00
9cc95e5657
AIX changes for P10 with GNU Compiler
2021-10-01 05:18:35 -05:00
fe3c778c51
AIX changes for P10 with GNU Compiler
2021-09-30 06:06:27 -05:00
ee5ca8a328
x86_64: BFLOAT16: fix build warning
2021-09-28 18:30:06 +08:00
90cc944625
Move alphaI to x22 to leave x18 unused (reserved on OSX)
2021-09-17 09:53:18 +02:00
590fbff06e
move alpha to x19/x20 to leave x18 unused for OSX
2021-09-17 09:42:17 +02:00
380940271b
Move temp to x21 to leave x18 unused (reserved on OSX)
2021-09-17 09:28:19 +02:00
7d75177446
Move temp to x21 to leave x18 unused (reserved on OSX)
2021-09-17 09:24:11 +02:00
0a4ac4b585
Use x21 for I to leave x18 unused (reserved on OSX)
2021-09-17 09:19:51 +02:00
7d4a221579
Remove unused TEMP2 and reshuffle to leave x18 unused (reserved on OSX)
2021-09-17 09:18:25 +02:00
d3a9c7ef7f
Merge pull request #3382 from rafaelcfsousa/rafael/cwarnings
...
[POWER] Remove unused variable warnings.
2021-09-17 09:15:16 +02:00
8dfa61a61c
Initialize abs_mask1 with itself to silence a gcc warning
2021-09-15 22:11:35 +02:00
99aa10b3ff
Initialize abs_mask1 with itself to silence a gcc warning
...
actual initialization is via the _mm_cmpeq_ep18, which I've seen claimed to be the fastest way to set an xmm register to all 1s
2021-09-15 22:10:43 +02:00
b751edf624
Fix unused variable warnings on Power
2021-09-15 13:36:07 -05:00
80346b8813
Merge pull request #3379 from martin-frbg/issue3369-2
...
Add casts to fix compiler warnings for SkylakeX sasum/dasum
2021-09-15 07:18:57 +02:00
ce036a2fc0
Add casts
2021-09-14 21:41:53 +02:00
ddf106f769
Add dedicated entries for BFLOAT16 kernels
2021-09-14 16:17:18 +02:00
af8843875a
Merge pull request #3376 from martin-frbg/issue3370
...
Fix a few harmless compiler warnings
2021-09-12 00:01:31 +02:00
0925dfe2c9
One instance of kernel_4x1 is used even on SKX
2021-09-11 15:30:19 +02:00
7d873a329f
Add ifdefs around conditionally used functions
2021-09-11 14:38:47 +02:00
ef24712030
Move a conditionally used variable
2021-09-11 14:37:44 +02:00
d17238599b
Add casts
2021-09-11 13:38:28 +02:00
59a1114d03
sbgemm: cooperlake: tuning for small matrix
2021-09-07 21:30:46 +08:00
682d66555d
sbgemm: cooperlake: implement ncopy_16
2021-09-07 21:30:46 +08:00
beccb83b16
sbgemm: cooperlake: add n24 kernel for tcopy_4
2021-09-07 21:30:46 +08:00
5fcacad32b
sbgemm: cooperlake: implement tcopy_4
2021-09-07 21:30:46 +08:00
bb1c4fa5bd
sbgemm: cooperlake: prefetch A & B
2021-09-07 21:30:46 +08:00
7a2d1601ec
sbgemm: cooperlake: unroll core loop by 2
2021-09-07 21:30:46 +08:00
45fdf951b6
sbgemm: cooperlake: reorder ptr increase for performance
2021-09-07 21:30:46 +08:00
cece3541ab
sbgemm: cooperlake: fix bug in m64n12
2021-09-07 21:30:46 +08:00
9df0953cde
sbgemm: cooperlake: kernel works for NN
2021-09-07 21:30:45 +08:00
2ec9f3a8aa
sbgemm: cooperlake: change kernel size to 16x4
2021-09-07 21:30:45 +08:00
ef8f5fecc8
sbgemm: cooperlake: implement sbgemm_tcopy_32
2021-09-07 21:30:45 +08:00
4c294336e6
sbgemm: cooperlake: add dummy source files
2021-09-07 21:30:45 +08:00
f1e3305974
Add workaround for Windows10 macro name clash
2021-09-01 21:36:50 +02:00
619588fbab
sbgemm: remove unnecessary b0 files
2021-08-30 17:55:01 +08:00
f39301935c
sbgemm: cooperlake: make sure hot buffer aligned to 64
2021-08-30 17:40:30 +08:00
7d27b182fc
sbgemm: cooperlake: enable SBGEMM by small matrix path
2021-08-30 17:40:30 +08:00
1d83ca4bca
Small Matrix: support BFLOAT16 data type
2021-08-30 17:40:20 +08:00
bec9d9f63d
Merge pull request #3335 from guowangy/small-matrix-latest
...
Add GEMM optimization for small matrix and single/double kernel for skylakex
2021-08-29 22:33:33 +02:00
dbbb39199f
sgemv: skylakex: fix build warning
2021-08-25 07:13:00 +00:00
e9acb46431
sgemv: skylakex: bug fix for sgemv_t kernel in corner case
2021-08-25 07:07:27 +00:00
f9dba63c28
Small Matrix: skylakex: remove unnecessary b0 source files
2021-08-13 03:28:44 +00:00
989e6bbdd3
Small Matrix: reduce generic kernel source files
2021-08-13 03:17:38 +00:00
04255be948
Merge pull request #3344 from gxw-loongson/develop
...
Delete the macro instruction "li" and use "li.d" instead
2021-08-12 15:16:46 +02:00
a7bc8ec1f1
Delete the macro instruction "li" and use "li.d" instead
...
Change-Id: Icff7981e2eb7df29ba5af1f8eb5be8443c67450f
2021-08-12 17:02:54 +08:00
b06880c2cd
POWER10: Improving dasum performance
...
Unrolling a loop in dasum micro code to help in improving
POWER10 performance.
2021-08-10 22:06:04 -05:00
44d0032f3b
Small Matrix: skylakex: fix build error in old compiler
2021-08-05 04:43:47 +00:00
5d86becdae
Add all SBGEMM kernels for IA AVX512-BF16 based platforms
...
Added all SBGEMM kernels including NN/NT/TN/TT for both ColMajor and
RowMajor, based on AVX512-BF16 ISA set on IA.
Signed-off-by: Chen, Guobing <guobing.chen@intel.com>
2021-08-05 11:11:29 +08:00
fee5abd84b
Small Matrix: support cmake build
2021-08-04 08:50:15 +00:00
478d1086c1
Small Matrix: support DYNAMIC_ARCH build
2021-08-04 03:12:41 +00:00
6b58bca18b
Small Matrix: disable low performance default kernel
2021-08-03 06:49:03 +00:00
fa777f5517
Small Matrix: skylakex: add DGEMM_SMALL_M_PERMIT and tune for TN kernel
2021-08-02 07:06:54 +00:00
8592c21af4
Small Matrix: skylakex: dgemm nn: fix typo in idx load
2021-08-02 07:06:54 +00:00
3e79f6d89a
Small Matrix: skylakex: add dgemm tn kernel
2021-08-02 07:06:54 +00:00
323d7da4f7
Small Matrix: skylakex: add dgemm tt kernel
2021-08-02 07:06:54 +00:00
f57fc932ac
Small Matrix: skylakex: add dgemm nt kernel
2021-08-02 07:06:54 +00:00
91ec21202b
Small Matrix: skylakex: add dgemm nn kernel
2021-08-02 07:06:54 +00:00
72e070539c
Small Matrix: skylakex: add sgemm tt kernel
2021-08-02 07:06:54 +00:00
02c6e764f2
Small Matrix: skylakex: add SGEMM_SMALL_M_PERMIT and tune for TN kernel
2021-08-02 07:06:54 +00:00
5dc7c3c8e5
Small Matrix: add GEMM_SMALL_MATRIX_PERMIT to tune small matrics case
2021-08-02 07:06:54 +00:00
642c393879
Small Matrix: skylakex: add sgemm tn kernel
2021-08-02 07:06:54 +00:00
ae3f5c737c
Small Matrix: skylakex: sgemm nt: optimize for M < 12
2021-08-02 07:06:54 +00:00
0d72d75bf9
Small Matrix: skylakex: add sgemm nt kernel
2021-08-02 07:06:54 +00:00
ca7682e3a3
Small Matrix: skylakex: sgemm nn: fix n6 conflicts with n4
2021-08-02 07:06:54 +00:00
9967e61abb
Small Matrix: skylakex: sgemm nn: fix error when beta not zero
2021-08-02 07:06:54 +00:00
a87736346f
Small Matrix: skylakex: sgemm nn: add n6 to improve performance
2021-08-02 07:06:54 +00:00
4c9d9940fd
Small Matrix: skylakex: sgemm nn: reduce store 4 N at a time
2021-08-02 07:06:54 +00:00
13b32f69b7
Small Matrix: skylakex: sgemm nn: reduce store 4 M at a time
2021-08-02 07:06:54 +00:00
3d8c6d9607
Small Matrix: skylakex: sgemm nn: clean up unused code
2021-08-02 07:06:54 +00:00
49b61a3f30
Small Matrix: skylakex: sgemm_nn: optimize for M <= 8
2021-08-02 07:06:54 +00:00
f88470323b
Optimize M < 16 using AVX512 mask
2021-08-02 07:06:54 +00:00
9186456a12
small matrix: SkylakeX: add SGEMM NN kernel
2021-08-02 07:06:54 +00:00
6022e5629c
Refs #2587 fix small matrix c/zgemm bug.
2021-08-02 07:06:54 +00:00
57ed58cefe
Refs #2587 Add small matrix optimization reference kernel for c/zgemm.
2021-08-02 07:06:54 +00:00
17d32a4a82
Change a1b0 gemm to b0 gemm.
2021-08-02 07:06:54 +00:00
59cb5de46b
Refs #2587 Fix typos.
2021-08-02 07:06:54 +00:00
be3349405d
Add alpha=1.0 beta=0.0 for small gemm.
2021-08-02 07:01:47 +00:00
0a2077901c
Add small marix optimization kernel interface.
...
make SMALL_MATRIX_OPT=1
2021-08-02 07:01:47 +00:00
0b8f7c8c10
Add cmake support for LOONGARCH64
2021-08-02 10:00:41 +08:00
af0a69f355
Add support for LOONGARCH64
2021-07-27 15:29:12 +08:00
49bbf330ca
Empirical workaround for numpy SVD NaN problem from issue 3318
2021-07-18 22:19:19 +02:00
5b4b385ecf
Temporarily disable the SkylakeX sgemv_t microkernel due to LAPACK testsuite failures
2021-07-14 20:50:14 +02:00
39ef0880ae
copy conf
2021-06-19 21:49:58 +02:00
c4b464cac6
Merge pull request #3273 from austinpagan/sbgemm_gcc10_fix
...
Power10: Fix for SBGEMM
2021-06-15 22:58:48 +02:00
e6dd44d989
Power10: Fix for SBGEMM
...
While testing bfloat16 sbgemm kernel, there are some failures for odd value inputs due to updating result for
additional bytes.
2021-06-15 13:07:47 -05:00
9d292d37b2
arm64: add the missing d9 register to the clobber list
...
Refs. numpy/numpy#18422
Signed-off-by: Gilles Gouaillardet <gilles@rist.or.jp>
2021-06-14 17:01:28 +09:00
2e8ff4a781
Merge pull request #3266 from martin-frbg/powerparam
...
Remove spurious casts from PPC parameters and fix compilation for older targets
2021-06-10 18:05:47 +02:00
dbba381dc3
Merge pull request #3260 from intelmy/sgemv_t_opt
...
Optimized sgemv_t for small N based on AVX512
2021-06-10 16:08:24 +02:00
efdbdd8f82
Add prefetch values for power3
2021-06-10 11:20:29 +02:00
3906ef3b0f
Add prefetch values for power3
2021-06-10 11:19:40 +02:00
8adf0971d8
Add prefetch values for power3
2021-06-10 11:18:22 +02:00
08e2e60762
Add prefetch values for power3
2021-06-10 11:17:33 +02:00
Martin Kroeker
Chris Sidebottom
Heller Zheng
Zhang Xianyi
Sergei Lewis
Xianyi Zhang
Bart Oldeman
lilianhuang
HellerZheng
Bart Oldeman
Honglin Zhu
gxw
Jiaxun Yang
Ivan Pribec
Rajalakshmi Srinivasaraghavan
VFerrari
Nursultan Zarlyk
Caroline Newcombe
Wangyang Guo
Mosè Giordano
Bine Brank
Sunita Nadampalli
Wu Zhigang
Jia-Chen
Rafael Cardoso Fernandes Sousa
kavanabhat
gxw
Chen, Guobing
User User-User
Gordon Fossum
Gilles Gouaillardet