f8ad5344c2
Fix casum fallback kernel.
...
This kernel is only used on Skylake+ if the kernel with AVX512
intrinsics can't be used, but used the variable x1 incorrectly
in the tail end of the loop, as it is still at the initial
value instead of where x points to.
This caused 55 "other error"s in the LAPACK tests
(https://github.com/OpenMathLib/OpenBLAS/issues/4282 )
This change makes casum.c as similar as possible as zasum.c,
because zasum.c does this correctly.
2023-11-17 23:53:56 +00:00
04bc801999
(Re)apply fixes for supporting only a subset of precision types from PR 3915
2023-11-04 23:48:59 +01:00
9019bc4945
Use SkylakeX ?ASUM microkernel for Cooperlake/Sapphirerapids as well
2023-11-04 22:10:06 +01:00
3bfa4d4dcc
Fix outdated SVE kernel definitions for Cortex cpus by aliasing to ARMV8SVE
2023-11-03 14:55:31 +01:00
980f702f72
POWER: AIX: Make use of power10 optimization
...
POWER10 optimizations are disabled when using default AIX assembler.
As we have fixed many issues recently, enabling optimization path
for default assembler.
2023-10-19 18:48:19 -05:00
9f42570e33
POWER: Increase macro size limit for AIX
...
This patch increases the macro size limit from 4096 to 16384 to
allow compiling larger assembly files in AIX.
Tested with GCC and IBM Open XL C.
2023-10-12 12:37:40 -05:00
9f49aef91b
Merge pull request #4255 from RajalakshmiSR/AIX-P10
...
POWER10: Fix compilation issues with Open XL C
2023-10-12 18:59:17 +02:00
e7d05402e0
Fix up S/D GEMM copy function definitions after #4009
2023-10-12 14:24:53 +02:00
71d733e5f7
POWER: Avoid m4 conversions for C files
...
This patch removes intermediate m4 conversions used in sbgemm
compilation as it is not needed for .c files.
Tested on AIX with gcc and IBM Open XL C.
2023-10-11 17:18:42 -05:00
82fc29a57a
POWER10: Fallback to POWER8 functions
...
As cgemm and zgemm kernels are not optimized for big endian falling
back to POWER8 versions. Tested on AIX using gcc and Open XL C.
2023-10-11 17:04:42 -05:00
db0805906b
powerpc: Fix build errors with Open XL C
...
This patch fixes errors when using Open XL C compiler on AIX.
Tested with gcc/xlf and ibm-clang/xlf compiler combinations.
2023-10-04 14:04:03 -05:00
675cd551da
fix improper function prototypes (empty parentheses)
2023-09-30 12:56:38 +02:00
d15e0a055c
LoongArch64: Fixed compilation issues when enable DYNAMIC_ARCH
2023-09-27 10:05:27 +08:00
4670eb1462
LoongArch64: Add dtrsm kernel
2023-09-26 15:45:14 +08:00
f2cf929374
LoongArch64: Add sgemv kernel
2023-09-04 14:28:37 +08:00
8e6d93359d
Merge pull request #4196 from TiborGY/obsolete_inlines
...
Modernize obsolete inline order
2023-09-03 14:12:42 +02:00
394a1fd1bf
LoongArch64: Compatible with early internal toolchain
...
__loongarch_grlen and __loongarch_frlen were introduced in gcc version 8.3.0
(Loongnix 8.3.0-6.lnd.vec.31) internally within Loongson to standardize the
general and floating-point register widths. However, previous versions did
not have them, requiring additional checks to be added.
2023-08-31 16:55:29 +08:00
9c4ae4d4fb
Merge pull request #4206 from martin-frbg/issue4201-2
...
Work around miscompilation of zdot_thunderx2t99 by the current NVIDIA HPC compiler
2023-08-26 10:17:27 +02:00
88435104c8
Merge pull request #4204 from martin-frbg/llvm17-2
...
Work around LLVM17 miscompiling the AVX512 microkernels for CASUM/ZASUM
2023-08-26 00:32:18 +02:00
fc8894dd98
Workaround miscompilation by NVIDIA nvc
2023-08-26 00:30:17 +02:00
7a6203ffa1
restore default Neoverse SVE build instructions for non-NVIDIA compilers
2023-08-25 18:25:51 +02:00
2c3034ff7f
Disable the C/ZASUM AVX512 microkernels when compiling with LLVM17 as well
2023-08-25 17:22:51 +02:00
8794544b43
Add support for compiling the Neoverse SVE kernels with the NVIDIA HPC compiler
2023-08-25 16:47:32 +02:00
553cc1372f
LoongArch64: Add sgemm_kernel
2023-08-23 16:08:43 +08:00
12ede72ab7
Merge pull request #4192 from imciner2/im/clangfix
...
Fix cooperlake and sapphire rapids march flags on clang
2023-08-21 15:46:35 +02:00
79c15db348
Fix power10 gcc intrinsic check
...
__builtin_vsx_assemble_pair was only in GCC 10-11.2 and was replaced by
__builtin_vsx_build_pair thereafter.
2023-08-17 15:05:29 +01:00
b5ba95a6c0
Modernize obsolete inline order
2023-08-16 00:48:40 +02:00
8a8a8479be
Fix cooperlake and sapphire rapids march flags on clang
...
The march=cooperlake and march=sapphirerapids flags were never getting
added when building with Clang targetting those architectures. Instead
it was falling back to the skylake AVX512 implementation.
Clang added support for these two architectures in Clang 9 and Clang 12,
so introduce new checks for those versions to enable the appropriate
march flag, and fallback to skylake otherwise.
2023-08-14 16:12:35 +01:00
34da1a067d
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 17:01:50 +02:00
07e32c4cb8
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 17:00:18 +02:00
c211da0688
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:58:57 +02:00
a34a0a7abc
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:56:52 +02:00
54d3246fc6
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:55:17 +02:00
7dd441d5db
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:53:33 +02:00
f692178792
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:52:09 +02:00
d15ffb7fdf
Allow negative INCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:50:44 +02:00
a2d867f4d1
Allow negative iNCX (API change from version 3.10 of the reference implementation)
2023-08-10 16:49:05 +02:00
afdc56a421
Merge pull request #4158 from XiWeiGu/loongarch64_update_dgemm_kernel
...
LoongArch64: Update dgemm kernel
2023-08-07 12:44:09 +02:00
e8b571d245
LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S V2
2023-08-07 11:20:42 +08:00
71fcee6eef
LoongArch64: Update dgemm kernel
2023-08-07 11:06:52 +08:00
0f521ece25
Merge pull request #4183 from martin-frbg/issue4181
...
Apply USE_TRMM to MIPS64_GENERIC as to GENERIC in gmake builds
2023-08-06 18:59:50 +02:00
41c31bc1d4
Revert "LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S"
2023-08-06 16:00:03 +02:00
61d803547a
Apply USE_TRMM to MIPS64_GENERIC as to GENERIC
2023-08-06 15:17:38 +02:00
f8ee309402
Merge pull request #4153 from XiWeiGu/dgemv
...
LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S
2023-08-06 08:49:16 +02:00
ec1e96aac8
LoongArch64: Add dgemv_t_8_lasx.S and dgemv_n_8_lasx.S
2023-08-05 10:24:17 +08:00
d46772e037
LoongArch64: Add compiler feature checks
2023-08-05 10:21:43 +08:00
4664b57e6e
use shortcut only when both incx and incy are zero
2023-08-04 12:25:34 +02:00
09131f79a6
Merge pull request #4164 from martin-frbg/issue4162
...
Enable use of AVX512 microkernels with NVIDIA HPC from version 22.3
2023-07-29 15:07:20 +02:00
6a428b5629
Update casum_microk_skylakex-2.c
2023-07-29 12:24:30 +02:00
ebb447e32e
Update zasum_microk_skylakex-2.c
2023-07-29 12:23:57 +02:00
9f6847583a
nvc currently miscompiles this, hopefully fixed in release 23.09
2023-07-29 11:50:16 +02:00
fe54ee3d15
nvc currently miscompiles this, hopefully fixed in release 23.09
2023-07-29 11:48:38 +02:00
5720fa02c5
Merge pull request #4168 from Mousius/sve-zgemm-cgemm
...
Use SVE zgemm/cgemm on Arm(R) Neoverse(TM) V1 core
2023-07-27 17:41:45 +02:00
84a268b6ca
Use SVE zgemm/cgemm on Arm(R) Neoverse(TM) V1 core
...
This patch removes the prefetches from cgemm/zgemm which improves the performance similar to sgemm/dgemm did in #3868 , this means I'm happy to enable this on any applicable cores.
I also replicated the unrolling the copies from sgemm and dgemm.
2023-07-27 14:12:20 +01:00
730ca04b48
Fix ZHEMM copy for SVE
...
Whilst disambiguating whilelt, I inadvertantly used the wrong datatype
for offsets, which can be negative. This rectifies that.
2023-07-27 13:27:28 +01:00
2a62d2df96
Enable use of AVX512 microkernels with NVIDIA HPC from version 22.3
2023-07-26 19:39:11 +02:00
849c8806b8
Merge pull request #4161 from Mousius/non-sve-kernels
...
Use latest non-SVE kernels in ARMV8SVE
2023-07-26 15:49:40 +02:00
24586bc4ff
Disambiguate whilelt
2023-07-25 20:15:44 +01:00
aea2a4622b
Use latest non-SVE kernels in ARMV8SVE
...
These are generally better and, in some cases, include threading which helps in the cores we're targeting here.
2023-07-25 14:12:26 +01:00
7976deff80
Fix file permissions (issue 4095)
2023-07-23 20:37:07 +02:00
76ef1672f8
Override DSDOT with generic code to get rid of qemu precision error
2023-07-19 22:31:07 +02:00
49077e7bde
Merge pull request #4145 from martin-frbg/issue4144
...
Restore zero-initialization of variables in generic ztrsm_utcopy
2023-07-14 12:44:05 +02:00
3d31191b0f
Work around Clang failing to disambiguate SVE intrinsics and add AppleClang crossbuild to MacOS/arm64 DYNAMIC_ARCH in AzureCI ( #4140 )
...
* Add AppleClang crossbuild to MacOS/arm64 DYNAMIC_ARCH
* add casts to disambiguate svwhilelt for clang
2023-07-14 11:06:48 +02:00
cfa0a80664
Restore initialization of data variables
2023-07-13 23:23:12 +02:00
9567305e4c
Restore initialization of data01,data02
2023-07-13 23:21:18 +02:00
e14a025bb1
Temporily walk around zaxpy vector kernel bug.
2023-06-28 11:17:38 +00:00
772b0cc715
Fix early bailout
2023-06-27 16:12:27 +02:00
d6be5036d7
Fix IDAMAX
2023-06-26 21:19:33 +02:00
1fe96f8da7
Fix failures to handle increments of zero
2023-06-25 22:36:57 +02:00
73b30b1dec
Fix VLEV_FLOAT/VSEV_FLOAT macros to compile with t-head 2.6.1
2023-06-18 17:46:29 +02:00
c3a2d407a0
Merge pull request #4048 from imzhuhl/spr_sbgemm_fix
...
Sapphire Rapids sbgemm fix
2023-06-17 20:47:09 +02:00
58b88aa5f0
POWER10: Fix compiler warnings
...
This patch removes the warning messages related to unused variables in
sbgemm_kernel_power10.c.
Signed-off-by: Manjul Mohan <manjul@linux.vnet.ibm.com>
2023-06-12 01:08:59 -04:00
9e80a194d6
Fix dynamic_list build and gcc version check error
2023-05-21 19:52:58 +08:00
a76afdc047
Compatible with older version of GNU make
2023-05-20 13:58:23 +08:00
90f041e348
Invoke the syscall to allow the use of amx tiles
2023-05-19 10:48:18 +08:00
0b83088887
spr dynamic arch support
2023-05-19 10:48:18 +08:00
f249ccb741
Fix spr sbgemm error
2023-05-19 10:48:18 +08:00
e9a8d5b45f
Merge pull request #4015 from martin-frbg/issue4013-2
...
[WIP] Disable gcc's tree-vectorizer for x86_64 CGEMV
2023-04-23 18:51:12 +02:00
72caceb324
Merge pull request #4009 from Mousius/sve-gemm
...
Use SVE kernel for SGEMM/DGEMM on Arm(R) Neoverse(TM) V1
2023-04-22 13:56:45 +02:00
84bcf6639f
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-20 23:24:52 +02:00
c9174ae8d7
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:45:44 +02:00
c2fe9cb91f
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:45:14 +02:00
66b39b835c
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:44:45 +02:00
bb6d6735bf
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:44:15 +02:00
d18efaed20
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:43:43 +02:00
99f6d31ed5
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:42:55 +02:00
7de9335c56
Disable gcc's tree-vectorizer pass on all operating systems
2023-04-19 23:42:09 +02:00
437c0bf2b4
Merge pull request #3843 from Mousius/switch-ratio
...
Propagate SWITCH_RATIO to DYNAMIC_ARCH builds
2023-04-19 11:51:54 +02:00
ec334e69dc
Use SVE kernel for SGEMM/DGEMM on Arm(R) Neoverse(TM) V1
...
This re-spins #3869 with some additional copy unrolling which helps maintain SYRK performance.
After #3868 , the SVE kernels represent a pretty good boost.
This re-uses ARMV8SVE as a base and I'm going to incrementally move everything to use ARMV8SVE in additional patches (as well as fix up anything that's not already in ARMV8SVE).
2023-04-17 17:38:42 +01:00
32f2fafde7
Propagate SWITCH_RATIO to DYNAMIC_ARCH builds
...
Previously dynamic builds were either using the default SWITCH_RATIO
or one from the higher level architecture; this patch ensures the
dynamic builds can use this parameter as well.
2023-04-17 15:34:12 +01:00
44164e3a3d
revert "move alpha out of register 18" (out of PR scope, no SVE on Apple hw)
2023-04-17 14:23:13 +02:00
8be68fa7f4
move declaration of sca to really keep the compiler from throwing it out (for now)
2023-04-15 12:02:39 +02:00
3727672a74
Improve workaround and keep compilers from optimizing it out
2023-04-13 18:07:52 +02:00
108a21e47a
Move ALPHA out of register 18 (reserved on OSX)
2023-04-13 18:05:14 +02:00
0b1acb0ba3
Move ALPHA_I out of register 18 (reserved on OSX)
2023-04-13 18:03:35 +02:00
c7bbad09ad
Move ALPHA_I out of register 18 (reserved on OSX)
2023-04-13 18:00:47 +02:00
cda29633a3
move ALPHA_I out of register 18 (reserved on OSX)
2023-04-13 17:59:48 +02:00
09ace3cf23
Merge pull request #3846 from lilh9598/sbgemm_opt
...
Improve the performance of sbgemm_tcopy on neoversen2
2023-03-26 19:04:57 +02:00
cb0a70e0e2
dot.c early bail fix
2023-03-02 09:51:10 +00:00
38d6fb4225
Fix dependencies in builds with specified subsets of precision types
2023-02-23 23:12:06 +01:00
Bart Oldeman
Martin Kroeker
Rajalakshmi Srinivasaraghavan
gxw
Ian McInerney
TGY
Chris Sidebottom
Xianyi Zhang
Manjul Mohan
Honglin Zhu
Sergei Lewis