xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch '3.0' into feature/stream

This commit is contained in:
Liu Jicong 2022-10-14 15:38:58 +08:00
parent a57a305035 cef365749c
commit 50896c3a01
72 changed files with 3533 additions and 1152 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
README-CN.md
View File

@ -107,7 +107,7 @@ sudo yum config-manager --set-enabled Powertools
### macOS ### macOS
``` ```
sudo brew install argp-standalone pkgconfig brew install argp-standalone pkgconfig
``` ```
### 设置 golang 开发环境 ### 设置 golang 开发环境
@ -276,7 +276,7 @@ sudo make install
安装成功后，可以在应用程序中双击 TDengine 图标启动服务，或者在终端中启动 TDengine 服务： 安装成功后，可以在应用程序中双击 TDengine 图标启动服务，或者在终端中启动 TDengine 服务：
```bash ```bash
launchctl start taosd launchctl start com.tdengine.taosd
``` ```
用户可以使用 TDengine CLI 来连接 TDengine 服务，在终端中，输入： 用户可以使用 TDengine CLI 来连接 TDengine 服务，在终端中，输入：

4
README.md
View File

@ -108,7 +108,7 @@ sudo yum config-manager --set-enabled powertools
### macOS ### macOS
``` ```
sudo brew install argp-standalone pkgconfig brew install argp-standalone pkgconfig
``` ```
### Setup golang environment ### Setup golang environment
@ -280,7 +280,7 @@ Installing from source code will also configure service management for TDengine.
To start the service after installation, double-click the /applications/TDengine to start the program, or in a terminal, use: To start the service after installation, double-click the /applications/TDengine to start the program, or in a terminal, use:
```bash ```bash
launchctl start taosd launchctl start com.tdengine.taosd
``` ```
Then users can use the TDengine CLI to connect the TDengine server. In a terminal, use: Then users can use the TDengine CLI to connect the TDengine server. In a terminal, use:

13
cmake/cmake.platform
View File

@ -45,10 +45,19 @@ IF (${CMAKE_SYSTEM_NAME} MATCHES "Linux" OR ${CMAKE_SYSTEM_NAME} MATCHES "Darwin
ADD_DEFINITIONS("-DDARWIN -Wno-tautological-pointer-compare") ADD_DEFINITIONS("-DDARWIN -Wno-tautological-pointer-compare")
MESSAGE("Current system processor is ${CMAKE_SYSTEM_PROCESSOR}.") MESSAGE("Current system processor is ${CMAKE_SYSTEM_PROCESSOR}.")
IF (${CMAKE_SYSTEM_PROCESSOR} MATCHES "arm64" OR ${CMAKE_SYSTEM_PROCESSOR} MATCHES "x86_64") IF (${CMAKE_SYSTEM_PROCESSOR} MATCHES "arm64")
MESSAGE("Current system arch is 64") MESSAGE("Current system arch is arm64")
SET(TD_DARWIN_64 TRUE) SET(TD_DARWIN_64 TRUE)
SET(TD_DARWIN_ARM64 TRUE)
ADD_DEFINITIONS("-D_TD_DARWIN_64") ADD_DEFINITIONS("-D_TD_DARWIN_64")
ADD_DEFINITIONS("-D_TD_DARWIN_ARM64")
ENDIF ()
IF (${CMAKE_SYSTEM_PROCESSOR} MATCHES "x86_64")
MESSAGE("Current system arch is x86_64")
SET(TD_DARWIN_64 TRUE)
SET(TD_DARWIN_X64 TRUE)
ADD_DEFINITIONS("-D_TD_DARWIN_64")
ADD_DEFINITIONS("-D_TD_DARWIN_X64")
ENDIF () ENDIF ()
ADD_DEFINITIONS("-DHAVE_UNISTD_H") ADD_DEFINITIONS("-DHAVE_UNISTD_H")

2
contrib/CMakeLists.txt
View File

@ -270,7 +270,7 @@ if(${JEMALLOC_ENABLED})
PREFIX "jemalloc" PREFIX "jemalloc"
SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/jemalloc SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/jemalloc
BUILD_IN_SOURCE 1 BUILD_IN_SOURCE 1
CONFIGURE_COMMAND ./autogen.sh COMMAND ./configure --prefix=${CMAKE_BINARY_DIR}/build/ --disable-initial-exec-tls CONFIGURE_COMMAND ./autogen.sh COMMAND ./configure --prefix=${CMAKE_BINARY_DIR}/build/ --disable-initial-exec-tls --with-malloc-conf='background_thread:true,metadata_thp:auto'
BUILD_COMMAND ${MAKE} BUILD_COMMAND ${MAKE}
) )
INCLUDE_DIRECTORIES(${CMAKE_BINARY_DIR}/build/include) INCLUDE_DIRECTORIES(${CMAKE_BINARY_DIR}/build/include)

6
docs/en/05-get-started/03-package.md
View File

@ -189,13 +189,13 @@ After the installation is complete, run `C:\TDengine\taosd.exe` to start TDengin
<TabItem label="macOS" value="macos"> <TabItem label="macOS" value="macos">
After the installation is complete, double-click the /applications/TDengine to start the program, or run `launchctl start taosd` to start TDengine Server. After the installation is complete, double-click the /applications/TDengine to start the program, or run `launchctl start com.tdengine.taosd` to start TDengine Server.
The following `launchctl` commands can help you manage TDengine service: The following `launchctl` commands can help you manage TDengine service:
- Start TDengine Server: `launchctl start taosd` - Start TDengine Server: `launchctl start com.tdengine.taosd`
- Stop TDengine Server: `launchctl stop taosd` - Stop TDengine Server: `launchctl stop com.tdengine.taosd`
- Check TDengine Server status: `launchctl list | grep taosd` - Check TDengine Server status: `launchctl list | grep taosd`

4
docs/en/12-taos-sql/28-recovery.md
View File

@ -16,10 +16,10 @@ You can use the SHOW CONNECTIONS statement to find the conn_id.
## Terminate a Query ## Terminate a Query
```sql ```sql
SHOW QUERY query_id; KILL QUERY kill_id;
``` ```
You can use the SHOW QUERIES statement to find the query_id. You can use the SHOW QUERIES statement to find the kill_id.
## Terminate a Transaction ## Terminate a Transaction

2
docs/en/14-reference/12-config/index.md
View File

@ -675,7 +675,7 @@ To prevent system resource from being exhausted by multiple concurrent streams,
| Meaning | Whether to generate core file when server crashes | | Meaning | Whether to generate core file when server crashes |
| Value Range | 0: false, 1: true | | Value Range | 0: false, 1: true |
| Default Value | 1 | | Default Value | 1 |
| Note | The core file is generated under root directory `systemctl/launchctl start taosd` is used to start, or under the working directory if `taosd` is started directly on Linux/macOS Shell. | | Note | The core file is generated under root directory `systemctl start taosd`/`launchctl start com.tdengine.taosd` is used to start, or under the working directory if `taosd` is started directly on Linux/macOS Shell. |
### udf ### udf

2
docs/en/21-tdinternal/01-arch.md
View File

@ -181,7 +181,7 @@ To make full use of the characteristics of time-series data, TDengine splits the
For time-series data, there is generally a retention policy, which is determined by the system configuration parameter `keep`. Data files exceeding this set number of days will be automatically deleted by the system to free up storage space. For time-series data, there is generally a retention policy, which is determined by the system configuration parameter `keep`. Data files exceeding this set number of days will be automatically deleted by the system to free up storage space.
Given `duration` and `keep` parameters, the total number of data files in a vnode is: keep/duration. The total number of data files should not be too large or too small. 10 to 100 is appropriate. Based on this principle, reasonable `duration` can be set. In the current version, parameter `keep` can be modified, but parameter `duration` cannot be modified once it is set. Given `duration` and `keep` parameters, the total number of data files in a vnode is: round up of (keep/duration+1). The total number of data files should not be too large or too small. 10 to 100 is appropriate. Based on this principle, reasonable `duration` can be set. In the current version, parameter `keep` can be modified, but parameter `duration` cannot be modified once it is set.
In each data file, the data of a table is stored in blocks. A table can have one or more data file blocks. In a file block, data is stored in columns, occupying a continuous storage space, thus greatly improving the reading speed. The size of file block is determined by the system parameter `maxRows` (the maximum number of records per block), and the default value is 4096. This value should not be too large or too small. If it is too large, data location for queries will take a longer time. If it is too small, the index of data block is too large, and the compression efficiency will be low with slower reading speed. In each data file, the data of a table is stored in blocks. A table can have one or more data file blocks. In a file block, data is stored in columns, occupying a continuous storage space, thus greatly improving the reading speed. The size of file block is determined by the system parameter `maxRows` (the maximum number of records per block), and the default value is 4096. This value should not be too large or too small. If it is too large, data location for queries will take a longer time. If it is too small, the index of data block is too large, and the compression efficiency will be low with slower reading speed.

6
docs/zh/05-get-started/03-package.md
View File

@ -188,13 +188,13 @@ Active: inactive (dead)
<TabItem label="macOS 系统" value="macos"> <TabItem label="macOS 系统" value="macos">
安装后，在应用程序目录下，双击 TDengine 图标来启动程序，也可以运行 `launchctl start taosd` 来启动 TDengine 服务进程。 安装后，在应用程序目录下，双击 TDengine 图标来启动程序，也可以运行 `launchctl start com.tdengine.taosd` 来启动 TDengine 服务进程。
如下 `launchctl` 命令可以帮助你管理 TDengine 服务： 如下 `launchctl` 命令可以帮助你管理 TDengine 服务：
- 启动服务进程：`launchctl start taosd` - 启动服务进程：`launchctl start com.tdengine.taosd`
- 停止服务进程：`launchctl stop taosd` - 停止服务进程：`launchctl stop com.tdengine.taosd`
- 查看服务状态：`launchctl list | grep taosd` - 查看服务状态：`launchctl list | grep taosd`

4
docs/zh/12-taos-sql/28-recovery.md
View File

@ -17,10 +17,10 @@ conn_id 可以通过 `SHOW CONNECTIONS` 获取。
## 终止查询 ## 终止查询
```sql ```sql
SHOW QUERY query_id; KILL QUERY kill_id;
``` ```
query_id 可以通过 `SHOW QUERIES` 获取。 kill_id 可以通过 `SHOW QUERIES` 获取。
## 终止事务 ## 终止事务

2
include/libs/qcom/query.h
View File

@ -230,7 +230,7 @@ int32_t cloneTableMeta(STableMeta* pSrc, STableMeta** pDst);
int32_t cloneDbVgInfo(SDBVgInfo* pSrc, SDBVgInfo** pDst); int32_t cloneDbVgInfo(SDBVgInfo* pSrc, SDBVgInfo** pDst);
extern int32_t (*queryBuildMsg[TDMT_MAX])(void* input, char** msg, int32_t msgSize, int32_t* msgLen, extern int32_t (*queryBuildMsg[TDMT_MAX])(void* input, char** msg, int32_t msgSize, int32_t* msgLen,
void* (*mallocFp)(int32_t)); void* (*mallocFp)(int64_t));
extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t msgSize); extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t msgSize);
#define SET_META_TYPE_NULL(t) (t) = META_TYPE_NULL_TABLE #define SET_META_TYPE_NULL(t) (t) = META_TYPE_NULL_TABLE

2
include/libs/scalar/scalar.h
View File

@ -43,7 +43,7 @@ int32_t scalarGetOperatorParamNum(EOperatorType type);
int32_t scalarGenerateSetFromList(void **data, void *pNode, uint32_t type); int32_t scalarGenerateSetFromList(void **data, void *pNode, uint32_t type);
int32_t vectorGetConvertType(int32_t type1, int32_t type2); int32_t vectorGetConvertType(int32_t type1, int32_t type2);
int32_t vectorConvertImpl(const SScalarParam *pIn, SScalarParam *pOut, int32_t *overflow); int32_t vectorConvertSingleColImpl(const SScalarParam *pIn, SScalarParam *pOut, int32_t *overflow, int32_t startIndex, int32_t numOfRows);
/* Math functions */ /* Math functions */
int32_t absFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t absFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);

4
include/libs/transport/trpc.h
View File

@ -123,9 +123,9 @@ void rpcCleanup();
void *rpcOpen(const SRpcInit *pRpc); void *rpcOpen(const SRpcInit *pRpc);
void rpcClose(void *); void rpcClose(void *);
void rpcCloseImpl(void *); void rpcCloseImpl(void *);
void *rpcMallocCont(int32_t contLen); void *rpcMallocCont(int64_t contLen);
void rpcFreeCont(void *pCont); void rpcFreeCont(void *pCont);
void *rpcReallocCont(void *ptr, int32_t contLen); void *rpcReallocCont(void *ptr, int64_t contLen);
// Because taosd supports multi-process mode // Because taosd supports multi-process mode
// These functions should not be used on the server side // These functions should not be used on the server side

8
include/os/osMemory.h
View File

@ -29,12 +29,12 @@ extern "C" {
#define free FREE_FUNC_TAOS_FORBID #define free FREE_FUNC_TAOS_FORBID
#endif #endif
void *taosMemoryMalloc(int32_t size); void *taosMemoryMalloc(int64_t size);
void *taosMemoryCalloc(int32_t num, int32_t size); void *taosMemoryCalloc(int64_t num, int64_t size);
void *taosMemoryRealloc(void *ptr, int32_t size); void *taosMemoryRealloc(void *ptr, int64_t size);
void *taosMemoryStrDup(const char *ptr); void *taosMemoryStrDup(const char *ptr);
void taosMemoryFree(void *ptr); void taosMemoryFree(void *ptr);
int32_t taosMemorySize(void *ptr); int64_t taosMemorySize(void *ptr);
void taosPrintBackTrace(); void taosPrintBackTrace();
#define taosMemoryFreeClear(ptr) \ #define taosMemoryFreeClear(ptr) \

6
include/util/trbtree.h
View File

@ -26,7 +26,7 @@ typedef struct SRBTree SRBTree;
typedef struct SRBTreeNode SRBTreeNode; typedef struct SRBTreeNode SRBTreeNode;
typedef struct SRBTreeIter SRBTreeIter; typedef struct SRBTreeIter SRBTreeIter;
typedef int32_t (*tRBTreeCmprFn)(const void *, const void *); typedef int32_t (*tRBTreeCmprFn)(const SRBTreeNode *, const SRBTreeNode *);
// SRBTree ============================================= // SRBTree =============================================
#define tRBTreeMin(T) ((T)->min == ((T)->NIL) ? NULL : (T)->min) #define tRBTreeMin(T) ((T)->min == ((T)->NIL) ? NULL : (T)->min)
@ -36,7 +36,7 @@ void tRBTreeCreate(SRBTree *pTree, tRBTreeCmprFn cmprFn);
SRBTreeNode *tRBTreePut(SRBTree *pTree, SRBTreeNode *z); SRBTreeNode *tRBTreePut(SRBTree *pTree, SRBTreeNode *z);
void tRBTreeDrop(SRBTree *pTree, SRBTreeNode *z); void tRBTreeDrop(SRBTree *pTree, SRBTreeNode *z);
SRBTreeNode *tRBTreeDropByKey(SRBTree *pTree, void *pKey); SRBTreeNode *tRBTreeDropByKey(SRBTree *pTree, void *pKey);
SRBTreeNode *tRBTreeGet(SRBTree *pTree, void *pKey); SRBTreeNode *tRBTreeGet(SRBTree *pTree, const SRBTreeNode *pKeyNode);
// SRBTreeIter ============================================= // SRBTreeIter =============================================
#define tRBTreeIterCreate(tree, ascend) \ #define tRBTreeIterCreate(tree, ascend) \
@ -53,8 +53,6 @@ struct SRBTreeNode {
SRBTreeNode *right; SRBTreeNode *right;
}; };
#define RBTREE_NODE_PAYLOAD(N) ((const void *)&(N)[1])
struct SRBTree { struct SRBTree {
tRBTreeCmprFn cmprFn; tRBTreeCmprFn cmprFn;
int64_t n; int64_t n;

4
packaging/release.sh
View File

@ -260,7 +260,7 @@ if [ "$osType" != "Darwin" ]; then
if [[ "$pagMode" == "full" ]]; then if [[ "$pagMode" == "full" ]]; then
if [ -d ${top_dir}/tools/taos-tools/packaging/deb ]; then if [ -d ${top_dir}/tools/taos-tools/packaging/deb ]; then
cd ${top_dir}/tools/taos-tools/packaging/deb cd ${top_dir}/tools/taos-tools/packaging/deb
taos_tools_ver=$(git describe --tags | sed -e 's/ver-//g' | awk -F '-' '{print $1}') taos_tools_ver=$(git tag |grep -v taos | sort | tail -1)
[ -z "$taos_tools_ver" ] && taos_tools_ver="0.1.0" [ -z "$taos_tools_ver" ] && taos_tools_ver="0.1.0"
${csudo}./make-taos-tools-deb.sh ${top_dir} \ ${csudo}./make-taos-tools-deb.sh ${top_dir} \
@ -285,7 +285,7 @@ if [ "$osType" != "Darwin" ]; then
if [[ "$pagMode" == "full" ]]; then if [[ "$pagMode" == "full" ]]; then
if [ -d ${top_dir}/tools/taos-tools/packaging/rpm ]; then if [ -d ${top_dir}/tools/taos-tools/packaging/rpm ]; then
cd ${top_dir}/tools/taos-tools/packaging/rpm cd ${top_dir}/tools/taos-tools/packaging/rpm
taos_tools_ver=$(git describe --tags | sed -e 's/ver-//g' | awk -F '-' '{print $1}' | sed -e 's/-/_/g') taos_tools_ver=$(git tag |grep -v taos | sort | tail -1)
[ -z "$taos_tools_ver" ] && taos_tools_ver="0.1.0" [ -z "$taos_tools_ver" ] && taos_tools_ver="0.1.0"
${csudo}./make-taos-tools-rpm.sh ${top_dir} \ ${csudo}./make-taos-tools-rpm.sh ${top_dir} \

4
packaging/tools/TDengine
View File

@ -17,12 +17,12 @@ EOF
taosd_status=`Launchctl list | grep taosd | head -n 1 | awk '{print $1}'` taosd_status=`Launchctl list | grep taosd | head -n 1 | awk '{print $1}'`
if [ "$taosd_status"x = "-"x ]; then if [ "$taosd_status"x = "-"x ]; then
launchctl start taosd launchctl start com.tdengine.taosd
showAlertMessage "Taosd is running!" "TDengine" "ok" "note" showAlertMessage "Taosd is running!" "TDengine" "ok" "note"
else else
choose_result=`showAlertMessage "Taosd is running!\nDo you want to close it?" "TDengine" "yes,cancel" "stop"` choose_result=`showAlertMessage "Taosd is running!\nDo you want to close it?" "TDengine" "yes,cancel" "stop"`
if [ "$choose_result"x = "button returned:yes"x ]; then if [ "$choose_result"x = "button returned:yes"x ]; then
launchctl stop taosd launchctl stop com.tdengine.taosd
fi fi
fi fi

882
packaging/tools/TDengine.pkgproj Normal file
View File

@ -0,0 +1,882 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>PACKAGES</key>
<array>
<dict>
<key>MUST-CLOSE-APPLICATION-ITEMS</key>
<array/>
<key>MUST-CLOSE-APPLICATIONS</key>
<false/>
<key>PACKAGE_FILES</key>
<dict>
<key>DEFAULT_INSTALL_LOCATION</key>
<string>/</string>
<key>HIERARCHY</key>
<dict>
<key>CHILDREN</key>
<array>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>80</integer>
<key>PATH</key>
<string>Applications</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>509</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>80</integer>
<key>PATH</key>
<string>Application Support</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Automator</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Documentation</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Extensions</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Filesystems</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Frameworks</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Input Methods</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Internet Plug-Ins</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Keyboard Layouts</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>LaunchAgents</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>LaunchDaemons</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>PreferencePanes</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Preferences</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>80</integer>
<key>PATH</key>
<string>Printers</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>PrivilegedHelperTools</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>1005</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>QuickLook</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>QuickTime</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Screen Savers</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Scripts</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Services</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Widgets</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
</array>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Library</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<dict>
<key>CHILDREN</key>
<array>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>Shared</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>1023</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
</array>
<key>GID</key>
<integer>80</integer>
<key>PATH</key>
<string>Users</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
</array>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>/</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>1</integer>
<key>UID</key>
<integer>0</integer>
</dict>
<key>PAYLOAD_TYPE</key>
<integer>0</integer>
<key>PRESERVE_EXTENDED_ATTRIBUTES</key>
<false/>
<key>SHOW_INVISIBLE</key>
<false/>
<key>SPLIT_FORKS</key>
<true/>
<key>TREAT_MISSING_FILES_AS_WARNING</key>
<false/>
<key>VERSION</key>
<integer>5</integer>
</dict>
<key>PACKAGE_SCRIPTS</key>
<dict>
<key>POSTINSTALL_PATH</key>
<dict>
<key>PATH</key>
<string>/opt/taos/TDengine/packaging/tools/post.sh</string>
<key>PATH_TYPE</key>
<integer>0</integer>
</dict>
<key>PREINSTALL_PATH</key>
<dict>
<key>PATH_TYPE</key>
<integer>0</integer>
</dict>
<key>RESOURCES</key>
<array>
<dict>
<key>CHILDREN</key>
<array/>
<key>GID</key>
<integer>0</integer>
<key>PATH</key>
<string>/opt/tdengine</string>
<key>PATH_TYPE</key>
<integer>0</integer>
<key>PERMISSIONS</key>
<integer>493</integer>
<key>TYPE</key>
<integer>3</integer>
<key>UID</key>
<integer>0</integer>
</dict>
</array>
</dict>
<key>PACKAGE_SETTINGS</key>
<dict>
<key>AUTHENTICATION</key>
<integer>1</integer>
<key>CONCLUSION_ACTION</key>
<integer>0</integer>
<key>FOLLOW_SYMBOLIC_LINKS</key>
<false/>
<key>IDENTIFIER</key>
<string>com.taosdata.pkg.pkgtaos</string>
<key>LOCATION</key>
<integer>0</integer>
<key>NAME</key>
<string>TDengine</string>
<key>OVERWRITE_PERMISSIONS</key>
<true/>
<key>PAYLOAD_SIZE</key>
<integer>-1</integer>
<key>REFERENCE_PATH</key>
<string></string>
<key>RELOCATABLE</key>
<false/>
<key>USE_HFS+_COMPRESSION</key>
<false/>
<key>VERSION</key>
<string>3.0.1.4</string>
</dict>
<key>TYPE</key>
<integer>0</integer>
<key>UUID</key>
<string>48F6B249-AF56-46E3-B75A-AEED5858A764</string>
</dict>
</array>
<key>PROJECT</key>
<dict>
<key>PROJECT_COMMENTS</key>
<dict>
<key>NOTES</key>
<data>
</data>
</dict>
<key>PROJECT_PRESENTATION</key>
<dict>
<key>BACKGROUND</key>
<dict>
<key>APPAREANCES</key>
<dict>
<key>DARK_AQUA</key>
<dict/>
<key>LIGHT_AQUA</key>
<dict/>
</dict>
<key>SHARED_SETTINGS_FOR_ALL_APPAREANCES</key>
<true/>
</dict>
<key>INSTALLATION TYPE</key>
<dict>
<key>HIERARCHIES</key>
<dict>
<key>INSTALLER</key>
<dict>
<key>LIST</key>
<array>
<dict>
<key>CHILDREN</key>
<array/>
<key>DESCRIPTION</key>
<array/>
<key>OPTIONS</key>
<dict>
<key>HIDDEN</key>
<false/>
<key>STATE</key>
<integer>1</integer>
</dict>
<key>PACKAGE_UUID</key>
<string>48F6B249-AF56-46E3-B75A-AEED5858A764</string>
<key>TITLE</key>
<array/>
<key>TYPE</key>
<integer>0</integer>
<key>UUID</key>
<string>7ED88D2C-D55C-46FF-99CB-73313ACAD73D</string>
</dict>
</array>
<key>REMOVED</key>
<dict/>
</dict>
</dict>
<key>MODE</key>
<integer>0</integer>
</dict>
<key>INSTALLATION_STEPS</key>
<array>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewIntroductionController</string>
<key>INSTALLER_PLUGIN</key>
<string>Introduction</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewReadMeController</string>
<key>INSTALLER_PLUGIN</key>
<string>ReadMe</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewLicenseController</string>
<key>INSTALLER_PLUGIN</key>
<string>License</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewDestinationSelectController</string>
<key>INSTALLER_PLUGIN</key>
<string>TargetSelect</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewInstallationTypeController</string>
<key>INSTALLER_PLUGIN</key>
<string>PackageSelection</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewInstallationController</string>
<key>INSTALLER_PLUGIN</key>
<string>Install</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
<dict>
<key>ICPRESENTATION_CHAPTER_VIEW_CONTROLLER_CLASS</key>
<string>ICPresentationViewSummaryController</string>
<key>INSTALLER_PLUGIN</key>
<string>Summary</string>
<key>LIST_TITLE_KEY</key>
<string>InstallerSectionTitle</string>
</dict>
</array>
<key>INTRODUCTION</key>
<dict>
<key>LOCALIZATIONS</key>
<array>
<dict>
<key>LANGUAGE</key>
<string>English</string>
<key>VALUE</key>
<dict>
<key>PATH</key>
<string>/opt/taos/TDengine/packaging/tools/mac_before_install.txt</string>
<key>PATH_TYPE</key>
<integer>0</integer>
</dict>
</dict>
</array>
</dict>
<key>LICENSE</key>
<dict>
<key>LOCALIZATIONS</key>
<array/>
<key>MODE</key>
<integer>0</integer>
</dict>
<key>README</key>
<dict>
<key>LOCALIZATIONS</key>
<array/>
</dict>
<key>SUMMARY</key>
<dict>
<key>LOCALIZATIONS</key>
<array/>
</dict>
<key>TITLE</key>
<dict>
<key>LOCALIZATIONS</key>
<array>
<dict>
<key>LANGUAGE</key>
<string>English</string>
<key>VALUE</key>
<string>TDengine</string>
</dict>
</array>
</dict>
</dict>
<key>PROJECT_REQUIREMENTS</key>
<dict>
<key>LIST</key>
<array/>
<key>RESOURCES</key>
<array/>
<key>ROOT_VOLUME_ONLY</key>
<false/>
</dict>
<key>PROJECT_SETTINGS</key>
<dict>
<key>BUILD_FORMAT</key>
<integer>0</integer>
<key>BUILD_PATH</key>
<dict>
<key>PATH</key>
<string>/opt/taos/TDengine/release</string>
<key>PATH_TYPE</key>
<integer>0</integer>
</dict>
<key>EXCLUDED_FILES</key>
<array>
<dict>
<key>PATTERNS_ARRAY</key>
<array>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.DS_Store</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
</array>
<key>PROTECTED</key>
<true/>
<key>PROXY_NAME</key>
<string>Remove .DS_Store files</string>
<key>PROXY_TOOLTIP</key>
<string>Remove ".DS_Store" files created by the Finder.</string>
<key>STATE</key>
<true/>
</dict>
<dict>
<key>PATTERNS_ARRAY</key>
<array>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.pbdevelopment</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
</array>
<key>PROTECTED</key>
<true/>
<key>PROXY_NAME</key>
<string>Remove .pbdevelopment files</string>
<key>PROXY_TOOLTIP</key>
<string>Remove ".pbdevelopment" files created by ProjectBuilder or Xcode.</string>
<key>STATE</key>
<true/>
</dict>
<dict>
<key>PATTERNS_ARRAY</key>
<array>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>CVS</string>
<key>TYPE</key>
<integer>1</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.cvsignore</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.cvspass</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.svn</string>
<key>TYPE</key>
<integer>1</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.git</string>
<key>TYPE</key>
<integer>1</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>.gitignore</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
</array>
<key>PROTECTED</key>
<true/>
<key>PROXY_NAME</key>
<string>Remove SCM metadata</string>
<key>PROXY_TOOLTIP</key>
<string>Remove helper files and folders used by the CVS, SVN or Git Source Code Management systems.</string>
<key>STATE</key>
<true/>
</dict>
<dict>
<key>PATTERNS_ARRAY</key>
<array>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>classes.nib</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>designable.db</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>info.nib</string>
<key>TYPE</key>
<integer>0</integer>
</dict>
</array>
<key>PROTECTED</key>
<true/>
<key>PROXY_NAME</key>
<string>Optimize nib files</string>
<key>PROXY_TOOLTIP</key>
<string>Remove "classes.nib", "info.nib" and "designable.nib" files within .nib bundles.</string>
<key>STATE</key>
<true/>
</dict>
<dict>
<key>PATTERNS_ARRAY</key>
<array>
<dict>
<key>REGULAR_EXPRESSION</key>
<false/>
<key>STRING</key>
<string>Resources Disabled</string>
<key>TYPE</key>
<integer>1</integer>
</dict>
</array>
<key>PROTECTED</key>
<true/>
<key>PROXY_NAME</key>
<string>Remove Resources Disabled folders</string>
<key>PROXY_TOOLTIP</key>
<string>Remove "Resources Disabled" folders.</string>
<key>STATE</key>
<true/>
</dict>
<dict>
<key>SEPARATOR</key>
<true/>
</dict>
</array>
<key>NAME</key>
<string>TDengine-client-3.0.1.4-macOS-arm64</string>
<key>PAYLOAD_ONLY</key>
<false/>
<key>TREAT_MISSING_PRESENTATION_DOCUMENTS_AS_WARNING</key>
<false/>
</dict>
</dict>
<key>TYPE</key>
<integer>0</integer>
<key>VERSION</key>
<integer>2</integer>
</dict>
</plist>

2
packaging/tools/com.taosdata.taosd.plist
View File

@ -3,7 +3,7 @@
<plist version="1.0"> <plist version="1.0">
<dict> <dict>
<key>Label</key> <key>Label</key>
<string>taosd</string> <string>com.tdengine.taosd</string>
<key>ProgramArguments</key> <key>ProgramArguments</key>
<array> <array>
<string>/usr/local/bin/taosd</string> <string>/usr/local/bin/taosd</string>

2
packaging/tools/mac_before_install.txt
View File

@ -1,5 +1,5 @@
TDengine is a high-efficient, scalable, high-available distributed time-series database, which makes a lot of optimizations on inserting and querying data, which is far more efficient than normal regular databases. So TDengine can meet the high requirements of IOT and other areas on storing and querying a large amount of data. TDengine is a high-efficient, scalable, high-available distributed time-series database, which makes a lot of optimizations on inserting and querying data, which is far more efficient than normal regular databases. So TDengine can meet the high requirements of IOT and other areas on storing and querying a large amount of data.
To configure TDengine : edit /etc/taos/taos.cfg To configure TDengine : edit /etc/taos/taos.cfg
To start service : launchctl start taosd To start service : launchctl start com.tdengine.taosd
To access TDengine : use taos in shell To access TDengine : use taos in shell

4
packaging/tools/make_install.sh
View File

@ -615,7 +615,7 @@ function update_TDengine() {
if [ "$osType" != "Darwin" ]; then if [ "$osType" != "Darwin" ]; then
echo -e "${GREEN_DARK}To start ${productName} ${NC}: ${serverName}${NC}" echo -e "${GREEN_DARK}To start ${productName} ${NC}: ${serverName}${NC}"
else else
echo -e "${GREEN_DARK}To start service ${NC}: launchctl start ${serverName}${NC}" echo -e "${GREEN_DARK}To start service ${NC}: launchctl start com.tdengine.taosd${NC}"
fi fi
[ -f ${installDir}/bin/taosadapter ] && \ [ -f ${installDir}/bin/taosadapter ] && \
echo -e "${GREEN_DARK}To start Taos Adapter ${NC}: taosadapter &${NC}" echo -e "${GREEN_DARK}To start Taos Adapter ${NC}: taosadapter &${NC}"
@ -666,7 +666,7 @@ function install_TDengine() {
if [ "$osType" != "Darwin" ]; then if [ "$osType" != "Darwin" ]; then
echo -e "${GREEN_DARK}To start ${productName} ${NC}: ${serverName}${NC}" echo -e "${GREEN_DARK}To start ${productName} ${NC}: ${serverName}${NC}"
else else
echo -e "${GREEN_DARK}To start service ${NC}: launchctl start ${serverName}${NC}" echo -e "${GREEN_DARK}To start service ${NC}: launchctl start com.tdengine.taosd${NC}"
fi fi
[ -f ${installDir}/bin/taosadapter ] && \ [ -f ${installDir}/bin/taosadapter ] && \
echo -e "${GREEN_DARK}To start Taos Adapter ${NC}: taosadapter &${NC}" echo -e "${GREEN_DARK}To start Taos Adapter ${NC}: taosadapter &${NC}"

12
packaging/tools/post.sh
View File

@ -18,6 +18,7 @@ if [ "$osType" != "Darwin" ]; then
script_dir=$(dirname $(readlink -f "$0")) script_dir=$(dirname $(readlink -f "$0"))
verNumber="" verNumber=""
lib_file_ext="so" lib_file_ext="so"
lib_file_ext_1="so.1"
bin_link_dir="/usr/bin" bin_link_dir="/usr/bin"
lib_link_dir="/usr/lib" lib_link_dir="/usr/lib"
@ -29,6 +30,7 @@ else
script_dir=${source_dir}/packaging/tools script_dir=${source_dir}/packaging/tools
verNumber=`ls tdengine/driver | grep -E "libtaos\.[0-9]\.[0-9]" | sed "s/libtaos.//g" | sed "s/.dylib//g" | head -n 1` verNumber=`ls tdengine/driver | grep -E "libtaos\.[0-9]\.[0-9]" | sed "s/libtaos.//g" | sed "s/.dylib//g" | head -n 1`
lib_file_ext="dylib" lib_file_ext="dylib"
lib_file_ext_1="1.dylib"
bin_link_dir="/usr/local/bin" bin_link_dir="/usr/local/bin"
lib_link_dir="/usr/local/lib" lib_link_dir="/usr/local/lib"
@ -134,14 +136,14 @@ function install_lib() {
[ -f ${lib_link_dir}/libtaosws.${lib_file_ext} ] && ${csudo}rm -f ${lib_link_dir}/libtaosws.${lib_file_ext} || : [ -f ${lib_link_dir}/libtaosws.${lib_file_ext} ] && ${csudo}rm -f ${lib_link_dir}/libtaosws.${lib_file_ext} || :
[ -f ${lib64_link_dir}/libtaosws.${lib_file_ext} ] && ${csudo}rm -f ${lib64_link_dir}/libtaosws.${lib_file_ext} || : [ -f ${lib64_link_dir}/libtaosws.${lib_file_ext} ] && ${csudo}rm -f ${lib64_link_dir}/libtaosws.${lib_file_ext} || :
${csudo}ln -s ${lib_dir}/libtaos.* ${lib_link_dir}/libtaos.so.1 ${csudo}ln -s ${lib_dir}/libtaos.* ${lib_link_dir}/libtaos.${lib_file_ext_1}
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so ${csudo}ln -s ${lib_link_dir}/libtaos.${lib_file_ext_1} ${lib_link_dir}/libtaos.${lib_file_ext}
[ -f ${lib_dir}/libtaosws.${lib_file_ext} ] && ${csudo}ln -sf ${lib_dir}/libtaosws.${lib_file_ext} ${lib_link_dir}/libtaosws.${lib_file_ext} ||: [ -f ${lib_dir}/libtaosws.${lib_file_ext} ] && ${csudo}ln -sf ${lib_dir}/libtaosws.${lib_file_ext} ${lib_link_dir}/libtaosws.${lib_file_ext} ||:
if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.so ]]; then if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.${lib_file_ext} ]]; then
${csudo}ln -s ${lib_dir}/libtaos.* ${lib64_link_dir}/libtaos.so.1 || : ${csudo}ln -s ${lib_dir}/libtaos.* ${lib64_link_dir}/libtaos.${lib_file_ext_1} || :
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || : ${csudo}ln -s ${lib64_link_dir}/libtaos.${lib_file_ext_1} ${lib64_link_dir}/libtaos.${lib_file_ext} || :
[ -f ${lib_dir}/libtaosws.${lib_file_ext} ] && ${csudo}ln -sf ${lib_dir}/libtaosws.${lib_file_ext} ${lib64_link_dir}/libtaosws.${lib_file_ext} || : [ -f ${lib_dir}/libtaosws.${lib_file_ext} ] && ${csudo}ln -sf ${lib_dir}/libtaosws.${lib_file_ext} ${lib64_link_dir}/libtaosws.${lib_file_ext} || :
fi fi

2
packaging/tools/remove.sh
View File

@ -117,7 +117,7 @@ function clean_local_bin() {
function clean_lib() { function clean_lib() {
# Remove link # Remove link
${csudo}rm -f ${lib_link_dir}/libtaos.* || : ${csudo}rm -f ${lib_link_dir}/libtaos.* || :
[ -f ${lib_link_dir}/libtaosws.so ] && ${csudo}rm -f ${lib_link_dir}/libtaosws.so || : [ -f ${lib_link_dir}/libtaosws.* ] && ${csudo}rm -f ${lib_link_dir}/libtaosws.* || :
${csudo}rm -f ${lib64_link_dir}/libtaos.* || : ${csudo}rm -f ${lib64_link_dir}/libtaos.* || :
[ -f ${lib64_link_dir}/libtaosws.* ] && ${csudo}rm -f ${lib64_link_dir}/libtaosws.* || : [ -f ${lib64_link_dir}/libtaosws.* ] && ${csudo}rm -f ${lib64_link_dir}/libtaosws.* || :

7
source/client/CMakeLists.txt
View File

@ -19,6 +19,13 @@ target_link_libraries(
PRIVATE os util common transport nodes parser command planner catalog scheduler function qcom PRIVATE os util common transport nodes parser command planner catalog scheduler function qcom
) )
if(TD_DARWIN_ARM64)
target_link_libraries(
taos
PRIVATE "-arch x86_64"
)
endif()
if(TD_WINDOWS) if(TD_WINDOWS)
INCLUDE_DIRECTORIES(jni/windows) INCLUDE_DIRECTORIES(jni/windows)
INCLUDE_DIRECTORIES(jni/windows/win32) INCLUDE_DIRECTORIES(jni/windows/win32)

5
source/client/src/clientSml.c
View File

@ -2477,11 +2477,12 @@ static void smlInsertCallback(void *param, void *res, int32_t code) {
} else { } else {
pParam->request->body.resInfo.numOfRows += info->affectedRows; pParam->request->body.resInfo.numOfRows += info->affectedRows;
} }
// unlock
taosThreadSpinUnlock(&pParam->lock);
if (pParam->cnt == pParam->total) { if (pParam->cnt == pParam->total) {
tsem_post(&pParam->sem); tsem_post(&pParam->sem);
} }
taosThreadSpinUnlock(&pParam->lock);
// unlock
uDebug("SML:0x%" PRIx64 " insert finished, code: %d, rows: %d, total: %d", info->id, code, rows, info->affectedRows); uDebug("SML:0x%" PRIx64 " insert finished, code: %d, rows: %d, total: %d", info->id, code, rows, info->affectedRows);
info->cost.endTime = taosGetTimestampUs(); info->cost.endTime = taosGetTimestampUs();
info->cost.code = code; info->cost.code = code;

23
source/dnode/vnode/src/tsdb/tsdbCache.c
View File

@ -429,7 +429,8 @@ typedef struct {
SDataFReader *pDataFReader; SDataFReader *pDataFReader;
TSDBROW row; TSDBROW row;
SMergeTree mergeTree; SMergeTree mergeTree;
SMergeTree *pMergeTree;
} SFSLastNextRowIter; } SFSLastNextRowIter;
static int32_t getNextRowFromFSLast(void *iter, TSDBROW **ppRow) { static int32_t getNextRowFromFSLast(void *iter, TSDBROW **ppRow) {
@ -444,11 +445,14 @@ static int32_t getNextRowFromFSLast(void *iter, TSDBROW **ppRow) {
case SFSLASTNEXTROW_FILESET: { case SFSLASTNEXTROW_FILESET: {
SDFileSet *pFileSet = NULL; SDFileSet *pFileSet = NULL;
_next_fileset: _next_fileset:
if (state->pMergeTree != NULL) {
tMergeTreeClose(state->pMergeTree);
state->pMergeTree = NULL;
}
if (--state->iFileSet >= 0) { if (--state->iFileSet >= 0) {
pFileSet = (SDFileSet *)taosArrayGet(state->aDFileSet, state->iFileSet); pFileSet = (SDFileSet *)taosArrayGet(state->aDFileSet, state->iFileSet);
} else { } else {
tMergeTreeClose(&state->mergeTree);
*ppRow = NULL; *ppRow = NULL;
return code; return code;
} }
@ -460,10 +464,10 @@ static int32_t getNextRowFromFSLast(void *iter, TSDBROW **ppRow) {
tMergeTreeOpen(&state->mergeTree, 1, state->pDataFReader, state->suid, state->uid, tMergeTreeOpen(&state->mergeTree, 1, state->pDataFReader, state->suid, state->uid,
&(STimeWindow){.skey = TSKEY_MIN, .ekey = TSKEY_MAX}, &(STimeWindow){.skey = TSKEY_MIN, .ekey = TSKEY_MAX},
&(SVersionRange){.minVer = 0, .maxVer = UINT64_MAX}, pLoadInfo, true, NULL); &(SVersionRange){.minVer = 0, .maxVer = UINT64_MAX}, pLoadInfo, true, NULL);
state->pMergeTree = &state->mergeTree;
bool hasVal = tMergeTreeNext(&state->mergeTree); bool hasVal = tMergeTreeNext(&state->mergeTree);
if (!hasVal) { if (!hasVal) {
state->state = SFSLASTNEXTROW_FILESET; state->state = SFSLASTNEXTROW_FILESET;
tMergeTreeClose(&state->mergeTree);
goto _next_fileset; goto _next_fileset;
} }
state->state = SFSLASTNEXTROW_BLOCKROW; state->state = SFSLASTNEXTROW_BLOCKROW;
@ -475,6 +479,7 @@ static int32_t getNextRowFromFSLast(void *iter, TSDBROW **ppRow) {
if (!hasVal) { if (!hasVal) {
state->state = SFSLASTNEXTROW_FILESET; state->state = SFSLASTNEXTROW_FILESET;
} }
return code; return code;
default: default:
ASSERT(0); ASSERT(0);
@ -486,6 +491,11 @@ _err:
tsdbDataFReaderClose(&state->pDataFReader); tsdbDataFReaderClose(&state->pDataFReader);
state->pDataFReader = NULL; state->pDataFReader = NULL;
} }
if (state->pMergeTree != NULL) {
tMergeTreeClose(state->pMergeTree);
state->pMergeTree = NULL;
}
*ppRow = NULL; *ppRow = NULL;
return code; return code;
@ -504,6 +514,11 @@ int32_t clearNextRowFromFSLast(void *iter) {
state->pDataFReader = NULL; state->pDataFReader = NULL;
} }
if (state->pMergeTree != NULL) {
tMergeTreeClose(state->pMergeTree);
state->pMergeTree = NULL;
}
return code; return code;
} }

9
source/dnode/vnode/src/tsdb/tsdbCommit.c
View File

@ -396,12 +396,19 @@ _exit:
return code; return code;
} }
static int32_t tDataIterCmprFn(const SRBTreeNode *n1, const SRBTreeNode *n2) {
SDataIter *pIter1 = (SDataIter *)((uint8_t *)n1 - offsetof(SDataIter, n));
SDataIter *pIter2 = (SDataIter *)((uint8_t *)n2 - offsetof(SDataIter, n));
return tRowInfoCmprFn(&pIter1->r, &pIter2->r);
}
static int32_t tsdbOpenCommitIter(SCommitter *pCommitter) { static int32_t tsdbOpenCommitIter(SCommitter *pCommitter) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0; int32_t lino = 0;
pCommitter->pIter = NULL; pCommitter->pIter = NULL;
tRBTreeCreate(&pCommitter->rbt, tRowInfoCmprFn); tRBTreeCreate(&pCommitter->rbt, tDataIterCmprFn);
// memory // memory
TSDBKEY tKey = {.ts = pCommitter->minKey, .version = VERSION_MIN}; TSDBKEY tKey = {.ts = pCommitter->minKey, .version = VERSION_MIN};

9
source/dnode/vnode/src/tsdb/tsdbFS.c
View File

@ -610,9 +610,6 @@ int32_t tsdbFSRollback(STsdbFS *pFS) {
ASSERT(0); ASSERT(0);
return code; return code;
_err:
return code;
} }
int32_t tsdbFSUpsertDelFile(STsdbFS *pFS, SDelFile *pDelFile) { int32_t tsdbFSUpsertDelFile(STsdbFS *pFS, SDelFile *pDelFile) {
@ -866,7 +863,7 @@ int32_t tsdbFSCommit2(STsdb *pTsdb, STsdbFS *pFSNew) {
nRef = atomic_sub_fetch_32(&fSet.pSmaF->nRef, 1); nRef = atomic_sub_fetch_32(&fSet.pSmaF->nRef, 1);
if (nRef == 0) { if (nRef == 0) {
tsdbSmaFileName(pTsdb, pSetOld->diskId, pSetOld->fid, fSet.pSmaF, fname); tsdbSmaFileName(pTsdb, pSetOld->diskId, pSetOld->fid, fSet.pSmaF, fname);
taosRemoveFile(fname); (void)taosRemoveFile(fname);
taosMemoryFree(fSet.pSmaF); taosMemoryFree(fSet.pSmaF);
} }
} else { } else {
@ -877,7 +874,7 @@ int32_t tsdbFSCommit2(STsdb *pTsdb, STsdbFS *pFSNew) {
// stt // stt
if (sameDisk) { if (sameDisk) {
if (pSetNew->nSttF > pSetOld->nSttF) { if (pSetNew->nSttF > pSetOld->nSttF) {
ASSERT(pSetNew->nSttF = pSetOld->nSttF + 1); ASSERT(pSetNew->nSttF == pSetOld->nSttF + 1);
pSetOld->aSttF[pSetOld->nSttF] = (SSttFile *)taosMemoryMalloc(sizeof(SSttFile)); pSetOld->aSttF[pSetOld->nSttF] = (SSttFile *)taosMemoryMalloc(sizeof(SSttFile));
if (pSetOld->aSttF[pSetOld->nSttF] == NULL) { if (pSetOld->aSttF[pSetOld->nSttF] == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
@ -1104,7 +1101,7 @@ void tsdbFSUnref(STsdb *pTsdb, STsdbFS *pFS) {
ASSERT(nRef >= 0); ASSERT(nRef >= 0);
if (nRef == 0) { if (nRef == 0) {
tsdbDelFileName(pTsdb, pFS->pDelFile, fname); tsdbDelFileName(pTsdb, pFS->pDelFile, fname);
taosRemoveFile(fname); (void)taosRemoveFile(fname);
taosMemoryFree(pFS->pDelFile); taosMemoryFree(pFS->pDelFile);
} }
} }

2
source/dnode/vnode/src/tsdb/tsdbMemTable.c
View File

@ -559,6 +559,8 @@ static int32_t tsdbInsertTableDataImpl(SMemTable *pMemTable, STbData *pTbData, i
// backward put first data // backward put first data
row.pTSRow = tGetSubmitBlkNext(&blkIter); row.pTSRow = tGetSubmitBlkNext(&blkIter);
if (row.pTSRow == NULL) return code;
key.ts = row.pTSRow->ts; key.ts = row.pTSRow->ts;
nRow++; nRow++;
tbDataMovePosTo(pTbData, pos, &key, SL_MOVE_BACKWARD); tbDataMovePosTo(pTbData, pos, &key, SL_MOVE_BACKWARD);

28
source/dnode/vnode/src/tsdb/tsdbMergeTree.c
View File

@ -101,8 +101,8 @@ void *destroyLastBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo) {
static SBlockData *loadLastBlock(SLDataIter *pIter, const char *idStr) { static SBlockData *loadLastBlock(SLDataIter *pIter, const char *idStr) {
int32_t code = 0; int32_t code = 0;
SSttBlockLoadInfo* pInfo = pIter->pBlockLoadInfo; SSttBlockLoadInfo *pInfo = pIter->pBlockLoadInfo;
if (pInfo->blockIndex[0] == pIter->iSttBlk) { if (pInfo->blockIndex[0] == pIter->iSttBlk) {
if (pInfo->currentLoadBlockIndex != 0) { if (pInfo->currentLoadBlockIndex != 0) {
tsdbDebug("current load index is set to 0, block index:%d, file index:%d, due to uid:%" PRIu64 ", load data, %s", tsdbDebug("current load index is set to 0, block index:%d, file index:%d, due to uid:%" PRIu64 ", load data, %s",
pIter->iSttBlk, pIter->iStt, pIter->uid, idStr); pIter->iSttBlk, pIter->iStt, pIter->uid, idStr);
@ -113,7 +113,7 @@ static SBlockData *loadLastBlock(SLDataIter *pIter, const char *idStr) {
if (pInfo->blockIndex[1] == pIter->iSttBlk) { if (pInfo->blockIndex[1] == pIter->iSttBlk) {
if (pInfo->currentLoadBlockIndex != 1) { if (pInfo->currentLoadBlockIndex != 1) {
tsdbDebug("current load index is set to 1, block index:%d, file index:%d, due to uid:%"PRIu64", load data, %s", tsdbDebug("current load index is set to 1, block index:%d, file index:%d, due to uid:%" PRIu64 ", load data, %s",
pIter->iSttBlk, pIter->iStt, pIter->uid, idStr); pIter->iSttBlk, pIter->iStt, pIter->uid, idStr);
pInfo->currentLoadBlockIndex = 1; pInfo->currentLoadBlockIndex = 1;
} }
@ -140,8 +140,10 @@ static SBlockData *loadLastBlock(SLDataIter *pIter, const char *idStr) {
pInfo->elapsedTime += el; pInfo->elapsedTime += el;
pInfo->loadBlocks += 1; pInfo->loadBlocks += 1;
tsdbDebug("read last block, total load:%d, trigger by uid:%"PRIu64", last file index:%d, last block index:%d, entry:%d, %p, elapsed time:%.2f ms, %s", tsdbDebug("read last block, total load:%d, trigger by uid:%" PRIu64
pInfo->loadBlocks, pIter->uid, pIter->iStt, pIter->iSttBlk, pInfo->currentLoadBlockIndex, pBlock, el, idStr); ", last file index:%d, last block index:%d, entry:%d, %p, elapsed time:%.2f ms, %s",
pInfo->loadBlocks, pIter->uid, pIter->iStt, pIter->iSttBlk, pInfo->currentLoadBlockIndex, pBlock, el,
idStr);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
goto _exit; goto _exit;
} }
@ -336,7 +338,7 @@ _exit:
void tLDataIterClose(SLDataIter *pIter) { taosMemoryFree(pIter); } void tLDataIterClose(SLDataIter *pIter) { taosMemoryFree(pIter); }
void tLDataIterNextBlock(SLDataIter *pIter, const char* idStr) { void tLDataIterNextBlock(SLDataIter *pIter, const char *idStr) {
int32_t step = pIter->backward ? -1 : 1; int32_t step = pIter->backward ? -1 : 1;
int32_t oldIndex = pIter->iSttBlk; int32_t oldIndex = pIter->iSttBlk;
@ -386,10 +388,10 @@ void tLDataIterNextBlock(SLDataIter *pIter, const char* idStr) {
if (index != -1) { if (index != -1) {
pIter->iSttBlk = index; pIter->iSttBlk = index;
pIter->pSttBlk = (SSttBlk *)taosArrayGet(pIter->pBlockLoadInfo->aSttBlk, pIter->iSttBlk); pIter->pSttBlk = (SSttBlk *)taosArrayGet(pIter->pBlockLoadInfo->aSttBlk, pIter->iSttBlk);
tsdbDebug("try next last file block:%d from %d, trigger by uid:%"PRIu64", file index:%d, %s", pIter->iSttBlk, oldIndex, pIter->uid, pIter->iStt, tsdbDebug("try next last file block:%d from %d, trigger by uid:%" PRIu64 ", file index:%d, %s", pIter->iSttBlk,
idStr); oldIndex, pIter->uid, pIter->iStt, idStr);
} else { } else {
tsdbDebug("no more last block qualified, uid:%"PRIu64", file index::%d, %s", pIter->uid, oldIndex, idStr); tsdbDebug("no more last block qualified, uid:%" PRIu64 ", file index::%d, %s", pIter->uid, oldIndex, idStr);
} }
} }
@ -502,9 +504,9 @@ _exit:
SRowInfo *tLDataIterGet(SLDataIter *pIter) { return &pIter->rInfo; } SRowInfo *tLDataIterGet(SLDataIter *pIter) { return &pIter->rInfo; }
// SMergeTree ================================================= // SMergeTree =================================================
static FORCE_INLINE int32_t tLDataIterCmprFn(const void *p1, const void *p2) { static FORCE_INLINE int32_t tLDataIterCmprFn(const SRBTreeNode *p1, const SRBTreeNode *p2) {
SLDataIter *pIter1 = (SLDataIter *)(((uint8_t *)p1) - sizeof(SRBTreeNode)); SLDataIter *pIter1 = (SLDataIter *)(((uint8_t *)p1) - offsetof(SLDataIter, node));
SLDataIter *pIter2 = (SLDataIter *)(((uint8_t *)p2) - sizeof(SRBTreeNode)); SLDataIter *pIter2 = (SLDataIter *)(((uint8_t *)p2) - offsetof(SLDataIter, node));
TSDBKEY key1 = TSDBROW_KEY(&pIter1->rInfo.row); TSDBKEY key1 = TSDBROW_KEY(&pIter1->rInfo.row);
TSDBKEY key2 = TSDBROW_KEY(&pIter2->rInfo.row); TSDBKEY key2 = TSDBROW_KEY(&pIter2->rInfo.row);
@ -581,7 +583,7 @@ bool tMergeTreeNext(SMergeTree *pMTree) {
// compare with min in RB Tree // compare with min in RB Tree
pIter = (SLDataIter *)tRBTreeMin(&pMTree->rbt); pIter = (SLDataIter *)tRBTreeMin(&pMTree->rbt);
if (pMTree->pIter && pIter) { if (pMTree->pIter && pIter) {
int32_t c = pMTree->rbt.cmprFn(RBTREE_NODE_PAYLOAD(&pMTree->pIter->node), RBTREE_NODE_PAYLOAD(&pIter->node)); int32_t c = pMTree->rbt.cmprFn(&pMTree->pIter->node, &pIter->node);
if (c > 0) { if (c > 0) {
tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pMTree->pIter); tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pMTree->pIter);
pMTree->pIter = NULL; pMTree->pIter = NULL;

6
source/dnode/vnode/src/tsdb/tsdbOpen.c
View File

@ -87,9 +87,13 @@ _err:
int tsdbClose(STsdb **pTsdb) { int tsdbClose(STsdb **pTsdb) {
if (*pTsdb) { if (*pTsdb) {
taosThreadRwlockDestroy(&(*pTsdb)->rwLock); taosThreadRwlockWrlock(&(*pTsdb)->rwLock);
tsdbMemTableDestroy((*pTsdb)->mem); tsdbMemTableDestroy((*pTsdb)->mem);
(*pTsdb)->mem = NULL; (*pTsdb)->mem = NULL;
taosThreadRwlockUnlock(&(*pTsdb)->rwLock);
taosThreadRwlockDestroy(&(*pTsdb)->rwLock);
tsdbFSClose(*pTsdb); tsdbFSClose(*pTsdb);
tsdbCloseCache(*pTsdb); tsdbCloseCache(*pTsdb);
taosMemoryFreeClear(*pTsdb); taosMemoryFreeClear(*pTsdb);

2
source/dnode/vnode/src/tsdb/tsdbRead.c
View File

@ -1645,7 +1645,7 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
STSRow* pTSRow = NULL; STSRow* pTSRow = NULL;
SRowMerger merge = {0}; SRowMerger merge = {0};
TSDBROW fRow = tMergeTreeGetRow(&pLastBlockReader->mergeTree); TSDBROW fRow = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
tsdbTrace("fRow ptr:%p, %d, uid:%"PRIu64", %s", fRow.pBlockData, fRow.iRow, pLastBlockReader->uid, pReader->idStr); tsdbTrace("fRow ptr:%p, %d, uid:%" PRIu64 ", %s", fRow.pBlockData, fRow.iRow, pLastBlockReader->uid, pReader->idStr);
// only last block exists // only last block exists
if ((!mergeBlockData) || (tsLastBlock != pBlockData->aTSKEY[pDumpInfo->rowIndex])) { if ((!mergeBlockData) || (tsLastBlock != pBlockData->aTSKEY[pDumpInfo->rowIndex])) {

73
source/dnode/vnode/src/tsdb/tsdbReaderWriter.c
View File

@ -18,7 +18,7 @@
// =============== PAGE-WISE FILE =============== // =============== PAGE-WISE FILE ===============
static int32_t tsdbOpenFile(const char *path, int32_t szPage, int32_t flag, STsdbFD **ppFD) { static int32_t tsdbOpenFile(const char *path, int32_t szPage, int32_t flag, STsdbFD **ppFD) {
int32_t code = 0; int32_t code = 0;
STsdbFD *pFD; STsdbFD *pFD = NULL;
*ppFD = NULL; *ppFD = NULL;
@ -35,6 +35,7 @@ static int32_t tsdbOpenFile(const char *path, int32_t szPage, int32_t flag, STsd
pFD->pFD = taosOpenFile(path, flag); pFD->pFD = taosOpenFile(path, flag);
if (pFD->pFD == NULL) { if (pFD->pFD == NULL) {
code = TAOS_SYSTEM_ERROR(errno); code = TAOS_SYSTEM_ERROR(errno);
taosMemoryFree(pFD);
goto _exit; goto _exit;
} }
pFD->szPage = szPage; pFD->szPage = szPage;
@ -42,11 +43,15 @@ static int32_t tsdbOpenFile(const char *path, int32_t szPage, int32_t flag, STsd
pFD->pBuf = taosMemoryCalloc(1, szPage); pFD->pBuf = taosMemoryCalloc(1, szPage);
if (pFD->pBuf == NULL) { if (pFD->pBuf == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
taosCloseFile(&pFD->pFD);
taosMemoryFree(pFD); taosMemoryFree(pFD);
goto _exit; goto _exit;
} }
if (taosStatFile(path, &pFD->szFile, NULL) < 0) { if (taosStatFile(path, &pFD->szFile, NULL) < 0) {
code = TAOS_SYSTEM_ERROR(errno); code = TAOS_SYSTEM_ERROR(errno);
taosMemoryFree(pFD->pBuf);
taosCloseFile(&pFD->pFD);
taosMemoryFree(pFD);
goto _exit; goto _exit;
} }
ASSERT(pFD->szFile % szPage == 0); ASSERT(pFD->szFile % szPage == 0);
@ -59,10 +64,12 @@ _exit:
static void tsdbCloseFile(STsdbFD **ppFD) { static void tsdbCloseFile(STsdbFD **ppFD) {
STsdbFD *pFD = *ppFD; STsdbFD *pFD = *ppFD;
taosMemoryFree(pFD->pBuf); if (pFD) {
taosCloseFile(&pFD->pFD); taosMemoryFree(pFD->pBuf);
taosMemoryFree(pFD); taosCloseFile(&pFD->pFD);
*ppFD = NULL; taosMemoryFree(pFD);
*ppFD = NULL;
}
} }
static int32_t tsdbWriteFilePage(STsdbFD *pFD) { static int32_t tsdbWriteFilePage(STsdbFD *pFD) {
@ -443,7 +450,7 @@ int32_t tsdbWriteDataBlk(SDataFWriter *pWriter, SMapData *mDataBlk, SBlockIdx *p
pBlockIdx->size = size; pBlockIdx->size = size;
pHeadFile->size += size; pHeadFile->size += size;
tsdbTrace("vgId:%d, write block, file ID:%d commit ID:%d suid:%" PRId64 " uid:%" PRId64 " offset:%" PRId64 tsdbTrace("vgId:%d, write block, file ID:%d commit ID:%" PRId64 " suid:%" PRId64 " uid:%" PRId64 " offset:%" PRId64
" size:%" PRId64 " nItem:%d", " size:%" PRId64 " nItem:%d",
TD_VID(pWriter->pTsdb->pVnode), pWriter->wSet.fid, pHeadFile->commitID, pBlockIdx->suid, pBlockIdx->uid, TD_VID(pWriter->pTsdb->pVnode), pWriter->wSet.fid, pHeadFile->commitID, pBlockIdx->suid, pBlockIdx->uid,
pBlockIdx->offset, pBlockIdx->size, mDataBlk->nItem); pBlockIdx->offset, pBlockIdx->size, mDataBlk->nItem);
@ -457,7 +464,7 @@ _err:
int32_t tsdbWriteSttBlk(SDataFWriter *pWriter, SArray *aSttBlk) { int32_t tsdbWriteSttBlk(SDataFWriter *pWriter, SArray *aSttBlk) {
int32_t code = 0; int32_t code = 0;
SSttFile *pSttFile = &pWriter->fStt[pWriter->wSet.nSttF - 1]; SSttFile *pSttFile = &pWriter->fStt[pWriter->wSet.nSttF - 1];
int64_t size; int64_t size = 0;
int64_t n; int64_t n;
// check // check
@ -906,10 +913,6 @@ int32_t tsdbDataFReaderClose(SDataFReader **ppReader) {
taosMemoryFree(*ppReader); taosMemoryFree(*ppReader);
*ppReader = NULL; *ppReader = NULL;
return code; return code;
_err:
tsdbError("vgId:%d, data file reader close failed since %s", TD_VID((*ppReader)->pTsdb->pVnode), tstrerror(code));
return code;
} }
int32_t tsdbReadBlockIdx(SDataFReader *pReader, SArray *aBlockIdx) { int32_t tsdbReadBlockIdx(SDataFReader *pReader, SArray *aBlockIdx) {
@ -1289,16 +1292,17 @@ _exit:
// SDelFWriter ==================================================== // SDelFWriter ====================================================
int32_t tsdbDelFWriterOpen(SDelFWriter **ppWriter, SDelFile *pFile, STsdb *pTsdb) { int32_t tsdbDelFWriterOpen(SDelFWriter **ppWriter, SDelFile *pFile, STsdb *pTsdb) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0;
char fname[TSDB_FILENAME_LEN]; char fname[TSDB_FILENAME_LEN];
uint8_t hdr[TSDB_FHDR_SIZE] = {0}; uint8_t hdr[TSDB_FHDR_SIZE] = {0};
SDelFWriter *pDelFWriter; SDelFWriter *pDelFWriter = NULL;
int64_t n; int64_t n;
// alloc // alloc
pDelFWriter = (SDelFWriter *)taosMemoryCalloc(1, sizeof(*pDelFWriter)); pDelFWriter = (SDelFWriter *)taosMemoryCalloc(1, sizeof(*pDelFWriter));
if (pDelFWriter == NULL) { if (pDelFWriter == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
pDelFWriter->pTsdb = pTsdb; pDelFWriter->pTsdb = pTsdb;
pDelFWriter->fDel = *pFile; pDelFWriter->fDel = *pFile;
@ -1306,21 +1310,28 @@ int32_t tsdbDelFWriterOpen(SDelFWriter **ppWriter, SDelFile *pFile, STsdb *pTsdb
tsdbDelFileName(pTsdb, pFile, fname); tsdbDelFileName(pTsdb, pFile, fname);
code = tsdbOpenFile(fname, pTsdb->pVnode->config.tsdbPageSize, TD_FILE_READ | TD_FILE_WRITE | TD_FILE_CREATE, code = tsdbOpenFile(fname, pTsdb->pVnode->config.tsdbPageSize, TD_FILE_READ | TD_FILE_WRITE | TD_FILE_CREATE,
&pDelFWriter->pWriteH); &pDelFWriter->pWriteH);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
// update header // update header
code = tsdbWriteFile(pDelFWriter->pWriteH, 0, hdr, TSDB_FHDR_SIZE); code = tsdbWriteFile(pDelFWriter->pWriteH, 0, hdr, TSDB_FHDR_SIZE);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
pDelFWriter->fDel.size = TSDB_FHDR_SIZE; pDelFWriter->fDel.size = TSDB_FHDR_SIZE;
pDelFWriter->fDel.offset = 0; pDelFWriter->fDel.offset = 0;
*ppWriter = pDelFWriter; *ppWriter = pDelFWriter;
return code;
_err: _exit:
tsdbError("vgId:%d, failed to open del file writer since %s", TD_VID(pTsdb->pVnode), tstrerror(code)); if (code) {
*ppWriter = NULL; if (pDelFWriter) {
taosMemoryFree(pDelFWriter);
tsdbCloseFile(&pDelFWriter->pWriteH);
}
*ppWriter = NULL;
tsdbError("vgId:%d %s failed at line %d since %s", TD_VID(pTsdb->pVnode), __func__, lino, tstrerror(errno));
} else {
*ppWriter = pDelFWriter;
}
return code; return code;
} }
@ -1456,15 +1467,15 @@ struct SDelFReader {
int32_t tsdbDelFReaderOpen(SDelFReader **ppReader, SDelFile *pFile, STsdb *pTsdb) { int32_t tsdbDelFReaderOpen(SDelFReader **ppReader, SDelFile *pFile, STsdb *pTsdb) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0;
char fname[TSDB_FILENAME_LEN]; char fname[TSDB_FILENAME_LEN];
SDelFReader *pDelFReader; SDelFReader *pDelFReader = NULL;
int64_t n;
// alloc // alloc
pDelFReader = (SDelFReader *)taosMemoryCalloc(1, sizeof(*pDelFReader)); pDelFReader = (SDelFReader *)taosMemoryCalloc(1, sizeof(*pDelFReader));
if (pDelFReader == NULL) { if (pDelFReader == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; goto _exit;
} }
// open impl // open impl
@ -1473,14 +1484,18 @@ int32_t tsdbDelFReaderOpen(SDelFReader **ppReader, SDelFile *pFile, STsdb *pTsdb
tsdbDelFileName(pTsdb, pFile, fname); tsdbDelFileName(pTsdb, pFile, fname);
code = tsdbOpenFile(fname, pTsdb->pVnode->config.tsdbPageSize, TD_FILE_READ, &pDelFReader->pReadH); code = tsdbOpenFile(fname, pTsdb->pVnode->config.tsdbPageSize, TD_FILE_READ, &pDelFReader->pReadH);
if (code) goto _err; if (code) {
taosMemoryFree(pDelFReader);
goto _exit;
}
*ppReader = pDelFReader; _exit:
return code; if (code) {
*ppReader = NULL;
_err: tsdbError("vgId:%d %s failed at %d since %s", TD_VID(pTsdb->pVnode), __func__, lino, tstrerror(code));
tsdbError("vgId:%d, del file reader open failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code)); } else {
*ppReader = NULL; *ppReader = pDelFReader;
}
return code; return code;
} }

68
source/dnode/vnode/src/tsdb/tsdbSnapshot.c
View File

@ -67,6 +67,13 @@ extern int32_t tRowInfoCmprFn(const void* p1, const void* p2);
extern int32_t tsdbReadDataBlockEx(SDataFReader* pReader, SDataBlk* pDataBlk, SBlockData* pBlockData); extern int32_t tsdbReadDataBlockEx(SDataFReader* pReader, SDataBlk* pDataBlk, SBlockData* pBlockData);
extern int32_t tsdbUpdateTableSchema(SMeta* pMeta, int64_t suid, int64_t uid, SSkmInfo* pSkmInfo); extern int32_t tsdbUpdateTableSchema(SMeta* pMeta, int64_t suid, int64_t uid, SSkmInfo* pSkmInfo);
static int32_t tFDataIterCmprFn(const SRBTreeNode* pNode1, const SRBTreeNode* pNode2) {
SFDataIter* pIter1 = (SFDataIter*)(((uint8_t*)pNode1) - offsetof(SFDataIter, n));
SFDataIter* pIter2 = (SFDataIter*)(((uint8_t*)pNode2) - offsetof(SFDataIter, n));
return tRowInfoCmprFn(&pIter1->rInfo, &pIter2->rInfo);
}
static int32_t tsdbSnapReadOpenFile(STsdbSnapReader* pReader) { static int32_t tsdbSnapReadOpenFile(STsdbSnapReader* pReader) {
int32_t code = 0; int32_t code = 0;
@ -79,7 +86,7 @@ static int32_t tsdbSnapReadOpenFile(STsdbSnapReader* pReader) {
if (code) goto _err; if (code) goto _err;
pReader->pIter = NULL; pReader->pIter = NULL;
tRBTreeCreate(&pReader->rbt, tRowInfoCmprFn); tRBTreeCreate(&pReader->rbt, tFDataIterCmprFn);
// .data file // .data file
SFDataIter* pIter = &pReader->aFDataIter[0]; SFDataIter* pIter = &pReader->aFDataIter[0];
@ -421,7 +428,7 @@ static int32_t tsdbSnapReadDel(STsdbSnapReader* pReader, uint8_t** ppData) {
n += tPutDelData((*ppData) + n, pDelData); n += tPutDelData((*ppData) + n, pDelData);
} }
tsdbInfo("vgId:%d, vnode snapshot tsdb read del data for %s, suid:%" PRId64 " uid:%d" PRId64 " size:%d", tsdbInfo("vgId:%d, vnode snapshot tsdb read del data for %s, suid:%" PRId64 " uid:%" PRId64 " size:%d",
TD_VID(pTsdb->pVnode), pTsdb->path, pDelIdx->suid, pDelIdx->uid, size); TD_VID(pTsdb->pVnode), pTsdb->path, pDelIdx->suid, pDelIdx->uid, size);
break; break;
@ -431,7 +438,7 @@ _exit:
return code; return code;
_err: _err:
tsdbError("vgId:%d, vnode snapshot tsdb read del for %s failed since %s", TD_VID(pTsdb->pVnode), pTsdb->pVnode, tsdbError("vgId:%d, vnode snapshot tsdb read del for %s failed since %s", TD_VID(pTsdb->pVnode), pTsdb->path,
tstrerror(code)); tstrerror(code));
return code; return code;
} }
@ -1247,20 +1254,21 @@ _err:
// APIs // APIs
int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWriter** ppWriter) { int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWriter** ppWriter) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0;
STsdbSnapWriter* pWriter = NULL; STsdbSnapWriter* pWriter = NULL;
// alloc // alloc
pWriter = (STsdbSnapWriter*)taosMemoryCalloc(1, sizeof(*pWriter)); pWriter = (STsdbSnapWriter*)taosMemoryCalloc(1, sizeof(*pWriter));
if (pWriter == NULL) { if (pWriter == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
pWriter->pTsdb = pTsdb; pWriter->pTsdb = pTsdb;
pWriter->sver = sver; pWriter->sver = sver;
pWriter->ever = ever; pWriter->ever = ever;
code = tsdbFSCopy(pTsdb, &pWriter->fs); code = tsdbFSCopy(pTsdb, &pWriter->fs);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
// config // config
pWriter->minutes = pTsdb->keepCfg.days; pWriter->minutes = pTsdb->keepCfg.days;
@ -1272,7 +1280,7 @@ int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWr
// SNAP_DATA_TSDB // SNAP_DATA_TSDB
code = tBlockDataCreate(&pWriter->bData); code = tBlockDataCreate(&pWriter->bData);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
pWriter->fid = INT32_MIN; pWriter->fid = INT32_MIN;
pWriter->id = (TABLEID){0}; pWriter->id = (TABLEID){0};
@ -1280,53 +1288,67 @@ int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWr
pWriter->dReader.aBlockIdx = taosArrayInit(0, sizeof(SBlockIdx)); pWriter->dReader.aBlockIdx = taosArrayInit(0, sizeof(SBlockIdx));
if (pWriter->dReader.aBlockIdx == NULL) { if (pWriter->dReader.aBlockIdx == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
code = tBlockDataCreate(&pWriter->dReader.bData); code = tBlockDataCreate(&pWriter->dReader.bData);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
// Writer // Writer
pWriter->dWriter.aBlockIdx = taosArrayInit(0, sizeof(SBlockIdx)); pWriter->dWriter.aBlockIdx = taosArrayInit(0, sizeof(SBlockIdx));
if (pWriter->dWriter.aBlockIdx == NULL) { if (pWriter->dWriter.aBlockIdx == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
pWriter->dWriter.aSttBlk = taosArrayInit(0, sizeof(SSttBlk)); pWriter->dWriter.aSttBlk = taosArrayInit(0, sizeof(SSttBlk));
if (pWriter->dWriter.aSttBlk == NULL) { if (pWriter->dWriter.aSttBlk == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
code = tBlockDataCreate(&pWriter->dWriter.bData); code = tBlockDataCreate(&pWriter->dWriter.bData);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
code = tBlockDataCreate(&pWriter->dWriter.sData); code = tBlockDataCreate(&pWriter->dWriter.sData);
if (code) goto _err; TSDB_CHECK_CODE(code, lino, _exit);
// SNAP_DATA_DEL // SNAP_DATA_DEL
pWriter->aDelIdxR = taosArrayInit(0, sizeof(SDelIdx)); pWriter->aDelIdxR = taosArrayInit(0, sizeof(SDelIdx));
if (pWriter->aDelIdxR == NULL) { if (pWriter->aDelIdxR == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
pWriter->aDelData = taosArrayInit(0, sizeof(SDelData)); pWriter->aDelData = taosArrayInit(0, sizeof(SDelData));
if (pWriter->aDelData == NULL) { if (pWriter->aDelData == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
pWriter->aDelIdxW = taosArrayInit(0, sizeof(SDelIdx)); pWriter->aDelIdxW = taosArrayInit(0, sizeof(SDelIdx));
if (pWriter->aDelIdxW == NULL) { if (pWriter->aDelIdxW == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY; code = TSDB_CODE_OUT_OF_MEMORY;
goto _err; TSDB_CHECK_CODE(code, lino, _exit);
} }
*ppWriter = pWriter; _exit:
if (code) {
tsdbError("vgId:%d %s failed at line %d since %s", TD_VID(pTsdb->pVnode), __func__, lino, tstrerror(code));
*ppWriter = NULL;
tsdbInfo("vgId:%d, tsdb snapshot writer open for %s succeed", TD_VID(pTsdb->pVnode), pTsdb->path); if (pWriter) {
return code; if (pWriter->aDelIdxW) taosArrayDestroy(pWriter->aDelIdxW);
if (pWriter->aDelData) taosArrayDestroy(pWriter->aDelData);
_err: if (pWriter->aDelIdxR) taosArrayDestroy(pWriter->aDelIdxR);
tsdbError("vgId:%d, tsdb snapshot writer open for %s failed since %s", TD_VID(pTsdb->pVnode), pTsdb->path, tBlockDataDestroy(&pWriter->dWriter.sData, 1);
tstrerror(code)); tBlockDataDestroy(&pWriter->dWriter.bData, 1);
*ppWriter = NULL; if (pWriter->dWriter.aSttBlk) taosArrayDestroy(pWriter->dWriter.aSttBlk);
if (pWriter->dWriter.aBlockIdx) taosArrayDestroy(pWriter->dWriter.aBlockIdx);
tBlockDataDestroy(&pWriter->dReader.bData, 1);
if (pWriter->dReader.aBlockIdx) taosArrayDestroy(pWriter->dReader.aBlockIdx);
tBlockDataDestroy(&pWriter->bData, 1);
tsdbFSDestroy(&pWriter->fs);
taosMemoryFree(pWriter);
}
} else {
tsdbDebug("vgId:%d, tsdb snapshot writer open for %s succeed", TD_VID(pTsdb->pVnode), pTsdb->path);
*ppWriter = pWriter;
}
return code; return code;
} }

4
source/dnode/vnode/src/tsdb/tsdbWrite.c
View File

@ -34,7 +34,9 @@ int tsdbInsertData(STsdb *pTsdb, int64_t version, SSubmitReq *pMsg, SSubmitRsp *
} }
// loop to insert // loop to insert
tInitSubmitMsgIter(pMsg, &msgIter); if (tInitSubmitMsgIter(pMsg, &msgIter) < 0) {
return -1;
}
while (true) { while (true) {
SSubmitBlkRsp r = {0}; SSubmitBlkRsp r = {0};
tGetSubmitMsgNext(&msgIter, &pBlock); tGetSubmitMsgNext(&msgIter, &pBlock);

18
source/dnode/vnode/src/vnd/vnodeBufPool.c
View File

@ -141,17 +141,17 @@ void *vnodeBufPoolMalloc(SVBufPool *pPool, int size) {
} }
void vnodeBufPoolFree(SVBufPool *pPool, void *p) { void vnodeBufPoolFree(SVBufPool *pPool, void *p) {
uint8_t *ptr = (uint8_t *)p; // uint8_t *ptr = (uint8_t *)p;
SVBufPoolNode *pNode; // SVBufPoolNode *pNode;
if (ptr < pPool->node.data || ptr >= pPool->node.data + pPool->node.size) { // if (ptr < pPool->node.data || ptr >= pPool->node.data + pPool->node.size) {
pNode = &((SVBufPoolNode *)p)[-1]; // pNode = &((SVBufPoolNode *)p)[-1];
*pNode->pnext = pNode->prev; // *pNode->pnext = pNode->prev;
pNode->prev->pnext = pNode->pnext; // pNode->prev->pnext = pNode->pnext;
pPool->size = pPool->size - sizeof(*pNode) - pNode->size; // pPool->size = pPool->size - sizeof(*pNode) - pNode->size;
taosMemoryFree(pNode); // taosMemoryFree(pNode);
} // }
} }
void vnodeBufPoolRef(SVBufPool *pPool) { void vnodeBufPoolRef(SVBufPool *pPool) {

11
source/dnode/vnode/src/vnd/vnodeModule.c
View File

@ -46,11 +46,17 @@ int vnodeInit(int nthreads) {
return 0; return 0;
} }
vnodeGlobal.stop = 0; taosThreadMutexInit(&vnodeGlobal.mutex, NULL);
taosThreadCondInit(&vnodeGlobal.hasTask, NULL);
taosThreadMutexLock(&vnodeGlobal.mutex);
vnodeGlobal.stop = 0;
vnodeGlobal.queue.next = &vnodeGlobal.queue; vnodeGlobal.queue.next = &vnodeGlobal.queue;
vnodeGlobal.queue.prev = &vnodeGlobal.queue; vnodeGlobal.queue.prev = &vnodeGlobal.queue;
taosThreadMutexUnlock(&(vnodeGlobal.mutex));
vnodeGlobal.nthreads = nthreads; vnodeGlobal.nthreads = nthreads;
vnodeGlobal.threads = taosMemoryCalloc(nthreads, sizeof(TdThread)); vnodeGlobal.threads = taosMemoryCalloc(nthreads, sizeof(TdThread));
if (vnodeGlobal.threads == NULL) { if (vnodeGlobal.threads == NULL) {
@ -59,9 +65,6 @@ int vnodeInit(int nthreads) {
return -1; return -1;
} }
taosThreadMutexInit(&vnodeGlobal.mutex, NULL);
taosThreadCondInit(&vnodeGlobal.hasTask, NULL);
for (int i = 0; i < nthreads; i++) { for (int i = 0; i < nthreads; i++) {
taosThreadCreate(&(vnodeGlobal.threads[i]), NULL, loop, NULL); taosThreadCreate(&(vnodeGlobal.threads[i]), NULL, loop, NULL);
} }

4
source/dnode/vnode/src/vnd/vnodeOpen.c
View File

@ -38,7 +38,7 @@ int vnodeCreate(const char *path, SVnodeCfg *pCfg, STfs *pTfs) {
if (taosMkDir(path)) { if (taosMkDir(path)) {
return TAOS_SYSTEM_ERROR(errno); return TAOS_SYSTEM_ERROR(errno);
} }
strcpy(dir, path); snprintf(dir, TSDB_FILENAME_LEN, "%s", path);
} }
if (pCfg) { if (pCfg) {
@ -51,7 +51,7 @@ int vnodeCreate(const char *path, SVnodeCfg *pCfg, STfs *pTfs) {
info.state.commitID = 0; info.state.commitID = 0;
if (vnodeSaveInfo(dir, &info) < 0 || vnodeCommitInfo(dir, &info) < 0) { if (vnodeSaveInfo(dir, &info) < 0 || vnodeCommitInfo(dir, &info) < 0) {
vError("vgId:%d, failed to save vnode config since %s", pCfg->vgId, tstrerror(terrno)); vError("vgId:%d, failed to save vnode config since %s", pCfg ? pCfg->vgId : 0, tstrerror(terrno));
return -1; return -1;
} }

9
source/dnode/vnode/src/vnd/vnodeSnapshot.c
View File

@ -166,7 +166,7 @@ int32_t vnodeSnapRead(SVSnapReader *pReader, uint8_t **ppData, uint32_t *nData)
if (*ppData) { if (*ppData) {
goto _exit; goto _exit;
} else { } else {
pReader->tqHandleDone = 1; pReader->tqOffsetDone = 1;
code = tqOffsetReaderClose(&pReader->pTqOffsetReader); code = tqOffsetReaderClose(&pReader->pTqOffsetReader);
if (code) goto _err; if (code) goto _err;
} }
@ -219,7 +219,7 @@ _exit:
return code; return code;
_err: _err:
vError("vgId:% vnode snapshot read failed since %s", TD_VID(pReader->pVnode), tstrerror(code)); vError("vgId:%d vnode snapshot read failed since %s", TD_VID(pReader->pVnode), tstrerror(code));
return code; return code;
} }
@ -260,7 +260,10 @@ int32_t vnodeSnapWriterOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapWr
// commit it // commit it
code = vnodeCommit(pVnode); code = vnodeCommit(pVnode);
if (code) goto _err; if (code) {
taosMemoryFree(pWriter);
goto _err;
}
// inc commit ID // inc commit ID
pVnode->state.commitID++; pVnode->state.commitID++;

5
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -68,7 +68,10 @@ int32_t vnodePreProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int64_t ctime = taosGetTimestampMs(); int64_t ctime = taosGetTimestampMs();
tb_uid_t uid; tb_uid_t uid;
tInitSubmitMsgIter(pSubmitReq, &msgIter); if (tInitSubmitMsgIter(pSubmitReq, &msgIter) < 0) {
code = terrno;
goto _err;
}
for (;;) { for (;;) {
tGetSubmitMsgNext(&msgIter, &pBlock); tGetSubmitMsgNext(&msgIter, &pBlock);

26
source/libs/catalog/src/ctgRemote.c
View File

@ -638,7 +638,7 @@ int32_t ctgGetQnodeListFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, SArray
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_QNODE_LIST; int32_t reqType = TDMT_MND_QNODE_LIST;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get qnode list from mnode, mgmtEpInUse:%d", pConn->mgmtEps.inUse); ctgDebug("try to get qnode list from mnode, mgmtEpInUse:%d", pConn->mgmtEps.inUse);
@ -692,7 +692,7 @@ int32_t ctgGetDnodeListFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, SArray
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_DNODE_LIST; int32_t reqType = TDMT_MND_DNODE_LIST;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get dnode list from mnode, mgmtEpInUse:%d", pConn->mgmtEps.inUse); ctgDebug("try to get dnode list from mnode, mgmtEpInUse:%d", pConn->mgmtEps.inUse);
@ -743,7 +743,7 @@ int32_t ctgGetDBVgInfoFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, SBuildU
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_USE_DB; int32_t reqType = TDMT_MND_USE_DB;
SCtgTask* pTask = tReq ? tReq->pTask : NULL; SCtgTask* pTask = tReq ? tReq->pTask : NULL;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get db vgInfo from mnode, dbFName:%s", input->db); ctgDebug("try to get db vgInfo from mnode, dbFName:%s", input->db);
@ -795,7 +795,7 @@ int32_t ctgGetDBCfgFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, const char
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_GET_DB_CFG; int32_t reqType = TDMT_MND_GET_DB_CFG;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get db cfg from mnode, dbFName:%s", dbFName); ctgDebug("try to get db cfg from mnode, dbFName:%s", dbFName);
@ -850,7 +850,7 @@ int32_t ctgGetIndexInfoFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, const
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_GET_INDEX; int32_t reqType = TDMT_MND_GET_INDEX;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get index from mnode, indexName:%s", indexName); ctgDebug("try to get index from mnode, indexName:%s", indexName);
@ -905,7 +905,7 @@ int32_t ctgGetTbIndexFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, SName* n
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_GET_TABLE_INDEX; int32_t reqType = TDMT_MND_GET_TABLE_INDEX;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
char tbFName[TSDB_TABLE_FNAME_LEN]; char tbFName[TSDB_TABLE_FNAME_LEN];
tNameExtractFullName(name, tbFName); tNameExtractFullName(name, tbFName);
@ -962,7 +962,7 @@ int32_t ctgGetUdfInfoFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, const ch
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_RETRIEVE_FUNC; int32_t reqType = TDMT_MND_RETRIEVE_FUNC;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get udf info from mnode, funcName:%s", funcName); ctgDebug("try to get udf info from mnode, funcName:%s", funcName);
@ -1017,7 +1017,7 @@ int32_t ctgGetUserDbAuthFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, const
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_GET_USER_AUTH; int32_t reqType = TDMT_MND_GET_USER_AUTH;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get user auth from mnode, user:%s", user); ctgDebug("try to get user auth from mnode, user:%s", user);
@ -1077,7 +1077,7 @@ int32_t ctgGetTbMetaFromMnodeImpl(SCatalog* pCtg, SRequestConnInfo* pConn, char*
int32_t reqType = TDMT_MND_TABLE_META; int32_t reqType = TDMT_MND_TABLE_META;
char tbFName[TSDB_TABLE_FNAME_LEN]; char tbFName[TSDB_TABLE_FNAME_LEN];
sprintf(tbFName, "%s.%s", dbFName, tbName); sprintf(tbFName, "%s.%s", dbFName, tbName);
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
ctgDebug("try to get table meta from mnode, tbFName:%s", tbFName); ctgDebug("try to get table meta from mnode, tbFName:%s", tbFName);
@ -1140,7 +1140,7 @@ int32_t ctgGetTbMetaFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, const SNa
int32_t reqType = TDMT_VND_TABLE_META; int32_t reqType = TDMT_VND_TABLE_META;
char tbFName[TSDB_TABLE_FNAME_LEN]; char tbFName[TSDB_TABLE_FNAME_LEN];
sprintf(tbFName, "%s.%s", dbFName, pTableName->tname); sprintf(tbFName, "%s.%s", dbFName, pTableName->tname);
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
SEp* pEp = &vgroupInfo->epSet.eps[vgroupInfo->epSet.inUse]; SEp* pEp = &vgroupInfo->epSet.eps[vgroupInfo->epSet.inUse];
ctgDebug("try to get table meta from vnode, vgId:%d, ep num:%d, ep %s:%d, tbFName:%s", vgroupInfo->vgId, ctgDebug("try to get table meta from vnode, vgId:%d, ep num:%d, ep %s:%d, tbFName:%s", vgroupInfo->vgId,
@ -1209,7 +1209,7 @@ int32_t ctgGetTableCfgFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, const S
int32_t reqType = TDMT_VND_TABLE_CFG; int32_t reqType = TDMT_VND_TABLE_CFG;
char tbFName[TSDB_TABLE_FNAME_LEN]; char tbFName[TSDB_TABLE_FNAME_LEN];
tNameExtractFullName(pTableName, tbFName); tNameExtractFullName(pTableName, tbFName);
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
char dbFName[TSDB_DB_FNAME_LEN]; char dbFName[TSDB_DB_FNAME_LEN];
tNameGetFullDbName(pTableName, dbFName); tNameGetFullDbName(pTableName, dbFName);
SBuildTableInput bInput = {.vgId = vgroupInfo->vgId, .dbFName = dbFName, .tbName = (char*)pTableName->tname}; SBuildTableInput bInput = {.vgId = vgroupInfo->vgId, .dbFName = dbFName, .tbName = (char*)pTableName->tname};
@ -1274,7 +1274,7 @@ int32_t ctgGetTableCfgFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, const S
int32_t reqType = TDMT_MND_TABLE_CFG; int32_t reqType = TDMT_MND_TABLE_CFG;
char tbFName[TSDB_TABLE_FNAME_LEN]; char tbFName[TSDB_TABLE_FNAME_LEN];
tNameExtractFullName(pTableName, tbFName); tNameExtractFullName(pTableName, tbFName);
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
char dbFName[TSDB_DB_FNAME_LEN]; char dbFName[TSDB_DB_FNAME_LEN];
tNameGetFullDbName(pTableName, dbFName); tNameGetFullDbName(pTableName, dbFName);
SBuildTableInput bInput = {.vgId = 0, .dbFName = dbFName, .tbName = (char*)pTableName->tname}; SBuildTableInput bInput = {.vgId = 0, .dbFName = dbFName, .tbName = (char*)pTableName->tname};
@ -1326,7 +1326,7 @@ int32_t ctgGetSvrVerFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, char** ou
char* msg = NULL; char* msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
int32_t reqType = TDMT_MND_SERVER_VERSION; int32_t reqType = TDMT_MND_SERVER_VERSION;
void* (*mallocFp)(int32_t) = pTask ? taosMemoryMalloc : rpcMallocCont; void* (*mallocFp)(int64_t) = pTask ? taosMemoryMalloc : rpcMallocCont;
qDebug("try to get svr ver from mnode"); qDebug("try to get svr ver from mnode");

11
source/libs/executor/src/executil.c
View File

@ -1250,6 +1250,17 @@ void createExprFromOneNode(SExprInfo* pExp, SNode* pNode, int16_t slotId) {
pExp->base.resSchema = pExp->base.resSchema =
createResSchema(pType->type, pType->bytes, slotId, pType->scale, pType->precision, pOpNode->node.aliasName); createResSchema(pType->type, pType->bytes, slotId, pType->scale, pType->precision, pOpNode->node.aliasName);
pExp->pExpr->_optrRoot.pRootNode = pNode; pExp->pExpr->_optrRoot.pRootNode = pNode;
} else if (type == QUERY_NODE_CASE_WHEN) {
pExp->pExpr->nodeType = QUERY_NODE_OPERATOR;
SCaseWhenNode* pCaseNode = (SCaseWhenNode*)pNode;
pExp->base.pParam = taosMemoryCalloc(1, sizeof(SFunctParam));
pExp->base.numOfParams = 1;
SDataType* pType = &pCaseNode->node.resType;
pExp->base.resSchema = createResSchema(pType->type, pType->bytes, slotId, pType->scale,
pType->precision, pCaseNode->node.aliasName);
pExp->pExpr->_optrRoot.pRootNode = pNode;
} else { } else {
ASSERT(0); ASSERT(0);
} }

2
source/libs/nodes/src/nodesCloneFuncs.c
View File

@ -131,7 +131,7 @@ static int32_t valueNodeCopy(const SValueNode* pSrc, SValueNode* pDst) {
COPY_SCALAR_FIELD(placeholderNo); COPY_SCALAR_FIELD(placeholderNo);
COPY_SCALAR_FIELD(typeData); COPY_SCALAR_FIELD(typeData);
COPY_SCALAR_FIELD(unit); COPY_SCALAR_FIELD(unit);
if (!pSrc->translate) { if (!pSrc->translate || pSrc->isNull) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
switch (pSrc->node.resType.type) { switch (pSrc->node.resType.type) {

4
source/libs/nodes/src/nodesCodeFuncs.c
View File

@ -3011,7 +3011,7 @@ static int32_t valueNodeToJson(const void* pObj, SJson* pJson) {
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkValueIsNull, pNode->isNull); code = tjsonAddBoolToObject(pJson, jkValueIsNull, pNode->isNull);
} }
if (TSDB_CODE_SUCCESS == code && pNode->translate) { if (TSDB_CODE_SUCCESS == code && pNode->translate && !pNode->isNull) {
code = datumToJson(pNode, pJson); code = datumToJson(pNode, pJson);
} }
@ -3161,7 +3161,7 @@ static int32_t jsonToValueNode(const SJson* pJson, void* pObj) {
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkValueIsNull, &pNode->isNull); code = tjsonGetBoolValue(pJson, jkValueIsNull, &pNode->isNull);
} }
if (TSDB_CODE_SUCCESS == code && pNode->translate) { if (TSDB_CODE_SUCCESS == code && pNode->translate && !pNode->isNull) {
code = jsonToDatum(pJson, pNode); code = jsonToDatum(pJson, pNode);
} }

2
source/libs/nodes/src/nodesUtilFuncs.c
View File

@ -1604,7 +1604,7 @@ char* nodesGetStrValueFromNode(SValueNode* pNode) {
bool nodesIsExprNode(const SNode* pNode) { bool nodesIsExprNode(const SNode* pNode) {
ENodeType type = nodeType(pNode); ENodeType type = nodeType(pNode);
return (QUERY_NODE_COLUMN == type || QUERY_NODE_VALUE == type || QUERY_NODE_OPERATOR == type || return (QUERY_NODE_COLUMN == type || QUERY_NODE_VALUE == type || QUERY_NODE_OPERATOR == type ||
QUERY_NODE_FUNCTION == type || QUERY_NODE_LOGIC_CONDITION == type); QUERY_NODE_FUNCTION == type || QUERY_NODE_LOGIC_CONDITION == type || QUERY_NODE_CASE_WHEN == type);
} }
bool nodesIsUnaryOp(const SOperatorNode* pOp) { bool nodesIsUnaryOp(const SOperatorNode* pOp) {

1
source/libs/parser/src/parTranslater.c
View File

@ -1888,6 +1888,7 @@ static EDealRes translateCaseWhen(STranslateContext* pCxt, SCaseWhenNode* pCaseW
pWhenThen->pWhen = pIsTrue; pWhenThen->pWhen = pIsTrue;
} }
if (first) { if (first) {
first = false;
pCaseWhen->node.resType = ((SExprNode*)pNode)->resType; pCaseWhen->node.resType = ((SExprNode*)pNode)->resType;
} else if (!dataTypeEqual(&pCaseWhen->node.resType, &((SExprNode*)pNode)->resType)) { } else if (!dataTypeEqual(&pCaseWhen->node.resType, &((SExprNode*)pNode)->resType)) {
SNode* pCastFunc = NULL; SNode* pCastFunc = NULL;

23
source/libs/planner/src/planLogicCreater.c
View File

@ -66,7 +66,8 @@ static EDealRes doRewriteExpr(SNode** pNode, void* pContext) {
switch (nodeType(*pNode)) { switch (nodeType(*pNode)) {
case QUERY_NODE_OPERATOR: case QUERY_NODE_OPERATOR:
case QUERY_NODE_LOGIC_CONDITION: case QUERY_NODE_LOGIC_CONDITION:
case QUERY_NODE_FUNCTION: { case QUERY_NODE_FUNCTION:
case QUERY_NODE_CASE_WHEN: {
SRewriteExprCxt* pCxt = (SRewriteExprCxt*)pContext; SRewriteExprCxt* pCxt = (SRewriteExprCxt*)pContext;
SNode* pExpr; SNode* pExpr;
int32_t index = 0; int32_t index = 0;
@ -118,6 +119,17 @@ static EDealRes doNameExpr(SNode* pNode, void* pContext) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
static int32_t rewriteExprForSelect(SNode* pExpr, SSelectStmt* pSelect, ESqlClause clause) {
nodesWalkExpr(pExpr, doNameExpr, NULL);
SRewriteExprCxt cxt = {.errCode = TSDB_CODE_SUCCESS, .pExprs = NULL};
cxt.errCode = nodesListMakeAppend(&cxt.pExprs, pExpr);
if (TSDB_CODE_SUCCESS == cxt.errCode) {
nodesRewriteSelectStmt(pSelect, clause, doRewriteExpr, &cxt);
nodesClearList(cxt.pExprs);
}
return cxt.errCode;
}
static int32_t rewriteExprsForSelect(SNodeList* pExprs, SSelectStmt* pSelect, ESqlClause clause) { static int32_t rewriteExprsForSelect(SNodeList* pExprs, SSelectStmt* pSelect, ESqlClause clause) {
nodesWalkExprs(pExprs, doNameExpr, NULL); nodesWalkExprs(pExprs, doNameExpr, NULL);
SRewriteExprCxt cxt = {.errCode = TSDB_CODE_SUCCESS, .pExprs = pExprs}; SRewriteExprCxt cxt = {.errCode = TSDB_CODE_SUCCESS, .pExprs = pExprs};
@ -711,8 +723,13 @@ static int32_t createWindowLogicNodeByState(SLogicPlanContext* pCxt, SStateWindo
nodesDestroyNode((SNode*)pWindow); nodesDestroyNode((SNode*)pWindow);
return TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_OUT_OF_MEMORY;
} }
// rewrite the expression in subsequent clauses
return createWindowLogicNodeFinalize(pCxt, pSelect, pWindow, pLogicNode); int32_t code = rewriteExprForSelect(pWindow->pStateExpr, pSelect, SQL_CLAUSE_WINDOW);
if (TSDB_CODE_SUCCESS == code) {
code = createWindowLogicNodeFinalize(pCxt, pSelect, pWindow, pLogicNode);
}
return code;
} }
static int32_t createWindowLogicNodeBySession(SLogicPlanContext* pCxt, SSessionWindowNode* pSession, static int32_t createWindowLogicNodeBySession(SLogicPlanContext* pCxt, SSessionWindowNode* pSession,

3
source/libs/planner/src/planPhysiCreater.c
View File

@ -761,7 +761,8 @@ static EDealRes doRewritePrecalcExprs(SNode** pNode, void* pContext) {
return collectAndRewrite(pCxt, pNode); return collectAndRewrite(pCxt, pNode);
} }
case QUERY_NODE_OPERATOR: case QUERY_NODE_OPERATOR:
case QUERY_NODE_LOGIC_CONDITION: { case QUERY_NODE_LOGIC_CONDITION:
case QUERY_NODE_CASE_WHEN: {
return collectAndRewrite(pCxt, pNode); return collectAndRewrite(pCxt, pNode);
} }
case QUERY_NODE_FUNCTION: { case QUERY_NODE_FUNCTION: {

3
source/libs/planner/src/planUtil.c
View File

@ -53,7 +53,8 @@ static EDealRes doCreateColumn(SNode* pNode, void* pContext) {
} }
case QUERY_NODE_OPERATOR: case QUERY_NODE_OPERATOR:
case QUERY_NODE_LOGIC_CONDITION: case QUERY_NODE_LOGIC_CONDITION:
case QUERY_NODE_FUNCTION: { case QUERY_NODE_FUNCTION:
case QUERY_NODE_CASE_WHEN: {
SExprNode* pExpr = (SExprNode*)pNode; SExprNode* pExpr = (SExprNode*)pNode;
SColumnNode* pCol = (SColumnNode*)nodesMakeNode(QUERY_NODE_COLUMN); SColumnNode* pCol = (SColumnNode*)nodesMakeNode(QUERY_NODE_COLUMN);
if (NULL == pCol) { if (NULL == pCol) {

30
source/libs/qcom/src/querymsg.c
View File

@ -24,8 +24,8 @@
#pragma GCC diagnostic ignored "-Wformat-truncation" #pragma GCC diagnostic ignored "-Wformat-truncation"
#endif #endif
int32_t (*queryBuildMsg[TDMT_MAX])(void *input, char **msg, int32_t msgSize, int32_t *msgLen, int32_t (*queryBuildMsg[TDMT_MAX])(void *input, char **msg, int32_t msgSize, int32_t *msgLen,
void *(*mallocFp)(int32_t)) = {0}; void *(*mallocFp)(int64_t)) = {0};
int32_t (*queryProcessMsgRsp[TDMT_MAX])(void *output, char *msg, int32_t msgSize) = {0}; int32_t (*queryProcessMsgRsp[TDMT_MAX])(void *output, char *msg, int32_t msgSize) = {0};
int32_t queryBuildUseDbOutput(SUseDbOutput *pOut, SUseDbRsp *usedbRsp) { int32_t queryBuildUseDbOutput(SUseDbOutput *pOut, SUseDbRsp *usedbRsp) {
@ -67,8 +67,8 @@ int32_t queryBuildUseDbOutput(SUseDbOutput *pOut, SUseDbRsp *usedbRsp) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildTableMetaReqMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, int32_t queryBuildTableMetaReqMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen,
void *(*mallcFp)(int32_t)) { void *(*mallcFp)(int64_t)) {
SBuildTableInput *pInput = input; SBuildTableInput *pInput = input;
if (NULL == input || NULL == msg || NULL == msgLen) { if (NULL == input || NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
@ -91,7 +91,7 @@ int32_t queryBuildTableMetaReqMsg(void *input, char **msg, int32_t msgSize, int3
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildUseDbMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildUseDbMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
SBuildUseDBInput *pInput = input; SBuildUseDBInput *pInput = input;
if (NULL == pInput || NULL == msg || NULL == msgLen) { if (NULL == pInput || NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
@ -114,7 +114,7 @@ int32_t queryBuildUseDbMsg(void *input, char **msg, int32_t msgSize, int32_t *ms
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildQnodeListMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildQnodeListMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -132,7 +132,7 @@ int32_t queryBuildQnodeListMsg(void *input, char **msg, int32_t msgSize, int32_t
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildDnodeListMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildDnodeListMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -150,7 +150,7 @@ int32_t queryBuildDnodeListMsg(void *input, char **msg, int32_t msgSize, int32_t
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildGetSerVerMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildGetSerVerMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -167,7 +167,7 @@ int32_t queryBuildGetSerVerMsg(void *input, char **msg, int32_t msgSize, int32_t
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildGetDBCfgMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildGetDBCfgMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -185,7 +185,7 @@ int32_t queryBuildGetDBCfgMsg(void *input, char **msg, int32_t msgSize, int32_t
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildGetIndexMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildGetIndexMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -203,8 +203,8 @@ int32_t queryBuildGetIndexMsg(void *input, char **msg, int32_t msgSize, int32_t
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildRetrieveFuncMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, int32_t queryBuildRetrieveFuncMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen,
void *(*mallcFp)(int32_t)) { void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -227,7 +227,7 @@ int32_t queryBuildRetrieveFuncMsg(void *input, char **msg, int32_t msgSize, int3
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildGetUserAuthMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildGetUserAuthMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -245,7 +245,7 @@ int32_t queryBuildGetUserAuthMsg(void *input, char **msg, int32_t msgSize, int32
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildGetTbIndexMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildGetTbIndexMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }
@ -263,7 +263,7 @@ int32_t queryBuildGetTbIndexMsg(void *input, char **msg, int32_t msgSize, int32_
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t queryBuildGetTbCfgMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int32_t)) { int32_t queryBuildGetTbCfgMsg(void *input, char **msg, int32_t msgSize, int32_t *msgLen, void *(*mallcFp)(int64_t)) {
if (NULL == msg || NULL == msgLen) { if (NULL == msg || NULL == msgLen) {
return TSDB_CODE_TSC_INVALID_INPUT; return TSDB_CODE_TSC_INVALID_INPUT;
} }

7
source/libs/scalar/inc/sclInt.h
View File

@ -86,7 +86,7 @@ typedef struct SScalarCtx {
} \ } \
} while (0) } while (0)
int32_t doConvertDataType(SValueNode* pValueNode, SScalarParam* out, int32_t* overflow); int32_t sclConvertValueToSclParam(SValueNode* pValueNode, SScalarParam* out, int32_t* overflow);
int32_t sclCreateColumnInfoData(SDataType* pType, int32_t numOfRows, SScalarParam* pParam); int32_t sclCreateColumnInfoData(SDataType* pType, int32_t numOfRows, SScalarParam* pParam);
int32_t sclConvertToTsValueNode(int8_t precision, SValueNode* valueNode); int32_t sclConvertToTsValueNode(int8_t precision, SValueNode* valueNode);
@ -95,6 +95,11 @@ int32_t sclConvertToTsValueNode(int8_t precision, SValueNode* valueNode);
#define GET_PARAM_PRECISON(_c) ((_c)->columnData->info.precision) #define GET_PARAM_PRECISON(_c) ((_c)->columnData->info.precision)
void sclFreeParam(SScalarParam* param); void sclFreeParam(SScalarParam* param);
void doVectorCompare(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t startIndex, int32_t numOfRows,
int32_t _ord, int32_t optr);
void vectorCompareImpl(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t startIndex, int32_t numOfRows,
int32_t _ord, int32_t optr);
void vectorCompare(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t _ord, int32_t optr);
#ifdef __cplusplus #ifdef __cplusplus
} }

9
source/libs/scalar/inc/sclvector.h
View File

@ -20,6 +20,15 @@
extern "C" { extern "C" {
#endif #endif
typedef struct SSclVectorConvCtx {
const SScalarParam* pIn;
SScalarParam* pOut;
int32_t startIndex;
int32_t endIndex;
int16_t inType;
int16_t outType;
} SSclVectorConvCtx;
typedef double (*_getDoubleValue_fn_t)(void *src, int32_t index); typedef double (*_getDoubleValue_fn_t)(void *src, int32_t index);
static FORCE_INLINE double getVectorDoubleValue_TINYINT(void *src, int32_t index) { static FORCE_INLINE double getVectorDoubleValue_TINYINT(void *src, int32_t index) {

9
source/libs/scalar/src/filter.c
View File

@ -1165,7 +1165,7 @@ int32_t fltAddGroupUnitFromNode(SFilterInfo *info, SNode *tree, SArray *group) {
SValueNode *valueNode = (SValueNode *)cell->pNode; SValueNode *valueNode = (SValueNode *)cell->pNode;
if (valueNode->node.resType.type != type) { if (valueNode->node.resType.type != type) {
int32_t overflow = 0; int32_t overflow = 0;
code = doConvertDataType(valueNode, &out, &overflow); code = sclConvertValueToSclParam(valueNode, &out, &overflow);
if (code) { if (code) {
// fltError("convert from %d to %d failed", in.type, out.type); // fltError("convert from %d to %d failed", in.type, out.type);
FLT_ERR_RET(code); FLT_ERR_RET(code);
@ -1973,7 +1973,7 @@ int32_t fltInitValFieldData(SFilterInfo *info) {
} }
// todo refactor the convert // todo refactor the convert
int32_t code = doConvertDataType(var, &out, NULL); int32_t code = sclConvertValueToSclParam(var, &out, NULL);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
qError("convert value to type[%d] failed", type); qError("convert value to type[%d] failed", type);
return TSDB_CODE_TSC_INVALID_OPERATION; return TSDB_CODE_TSC_INVALID_OPERATION;
@ -3792,6 +3792,11 @@ EDealRes fltReviseRewriter(SNode **pNode, void *pContext) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
if (QUERY_NODE_CASE_WHEN == nodeType(*pNode) || QUERY_NODE_WHEN_THEN == nodeType(*pNode)) {
stat->scalarMode = true;
return DEAL_RES_CONTINUE;
}
if (QUERY_NODE_OPERATOR == nodeType(*pNode)) { if (QUERY_NODE_OPERATOR == nodeType(*pNode)) {
SOperatorNode *node = (SOperatorNode *)*pNode; SOperatorNode *node = (SOperatorNode *)*pNode;
if (!FLT_IS_COMPARISON_OPERATOR(node->opType)) { if (!FLT_IS_COMPARISON_OPERATOR(node->opType)) {

372
source/libs/scalar/src/scalar.c
View File

@ -61,7 +61,7 @@ int32_t sclCreateColumnInfoData(SDataType *pType, int32_t numOfRows, SScalarPara
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t doConvertDataType(SValueNode *pValueNode, SScalarParam *out, int32_t *overflow) { int32_t sclConvertValueToSclParam(SValueNode* pValueNode, SScalarParam* out, int32_t* overflow) {
SScalarParam in = {.numOfRows = 1}; SScalarParam in = {.numOfRows = 1};
int32_t code = sclCreateColumnInfoData(&pValueNode->node.resType, 1, &in); int32_t code = sclCreateColumnInfoData(&pValueNode->node.resType, 1, &in);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
@ -71,12 +71,34 @@ int32_t doConvertDataType(SValueNode *pValueNode, SScalarParam *out, int32_t *ov
colDataAppend(in.columnData, 0, nodesGetValueFromNode(pValueNode), false); colDataAppend(in.columnData, 0, nodesGetValueFromNode(pValueNode), false);
colInfoDataEnsureCapacity(out->columnData, 1); colInfoDataEnsureCapacity(out->columnData, 1);
code = vectorConvertImpl(&in, out, overflow); code = vectorConvertSingleColImpl(&in, out, overflow, -1, -1);
sclFreeParam(&in); sclFreeParam(&in);
return code; return code;
} }
int32_t sclExtendResRows(SScalarParam *pDst, SScalarParam *pSrc, SArray *pBlockList) {
SSDataBlock* pb = taosArrayGetP(pBlockList, 0);
SScalarParam *pLeft = taosMemoryCalloc(1, sizeof(SScalarParam));
if (NULL == pLeft) {
sclError("calloc %d failed", (int32_t)sizeof(SScalarParam));
SCL_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
pLeft->numOfRows = pb->info.rows;
if (pDst->numOfRows < pb->info.rows) {
colInfoDataEnsureCapacity(pDst->columnData, pb->info.rows);
}
_bin_scalar_fn_t OperatorFn = getBinScalarOperatorFn(OP_TYPE_ASSIGN);
OperatorFn(pLeft, pSrc, pDst, TSDB_ORDER_ASC);
taosMemoryFree(pLeft);
return TSDB_CODE_SUCCESS;
}
int32_t scalarGenerateSetFromList(void **data, void *pNode, uint32_t type) { int32_t scalarGenerateSetFromList(void **data, void *pNode, uint32_t type) {
SHashObj *pObj = taosHashInit(256, taosGetDefaultHashFunction(type), true, false); SHashObj *pObj = taosHashInit(256, taosGetDefaultHashFunction(type), true, false);
if (NULL == pObj) { if (NULL == pObj) {
@ -110,7 +132,7 @@ int32_t scalarGenerateSetFromList(void **data, void *pNode, uint32_t type) {
} }
int32_t overflow = 0; int32_t overflow = 0;
code = doConvertDataType(valueNode, &out, &overflow); code = sclConvertValueToSclParam(valueNode, &out, &overflow);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
// sclError("convert data from %d to %d failed", in.type, out.type); // sclError("convert data from %d to %d failed", in.type, out.type);
SCL_ERR_JRET(code); SCL_ERR_JRET(code);
@ -178,7 +200,7 @@ void sclFreeRes(SHashObj *res) {
} }
void sclFreeParam(SScalarParam *param) { void sclFreeParam(SScalarParam *param) {
if (!param->colAlloced) { if (NULL == param || !param->colAlloced) {
return; return;
} }
@ -386,7 +408,8 @@ int32_t sclInitParam(SNode *node, SScalarParam *param, SScalarCtx *ctx, int32_t
} }
case QUERY_NODE_FUNCTION: case QUERY_NODE_FUNCTION:
case QUERY_NODE_OPERATOR: case QUERY_NODE_OPERATOR:
case QUERY_NODE_LOGIC_CONDITION: { case QUERY_NODE_LOGIC_CONDITION:
case QUERY_NODE_CASE_WHEN: {
SScalarParam *res = (SScalarParam *)taosHashGet(ctx->pRes, &node, POINTER_BYTES); SScalarParam *res = (SScalarParam *)taosHashGet(ctx->pRes, &node, POINTER_BYTES);
if (NULL == res) { if (NULL == res) {
sclError("no result for node, type:%d, node:%p", nodeType(node), node); sclError("no result for node, type:%d, node:%p", nodeType(node), node);
@ -540,6 +563,135 @@ _return:
SCL_RET(code); SCL_RET(code);
} }
int32_t sclGetNodeRes(SNode* node, SScalarCtx *ctx, SScalarParam **res) {
if (NULL == node) {
return TSDB_CODE_SUCCESS;
}
int32_t rowNum = 0;
*res = taosMemoryCalloc(1, sizeof(**res));
if (NULL == *res) {
SCL_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
}
SCL_ERR_RET(sclInitParam(node, *res, ctx, &rowNum));
return TSDB_CODE_SUCCESS;
}
int32_t sclWalkCaseWhenList(SScalarCtx *ctx, SNodeList* pList, struct SListCell* pCell, SScalarParam *pCase, SScalarParam *pElse, SScalarParam *pComp, SScalarParam *output, int32_t rowIdx, int32_t totalRows, bool *complete) {
SNode *node = NULL;
SWhenThenNode* pWhenThen = NULL;
SScalarParam *pWhen = NULL;
SScalarParam *pThen = NULL;
int32_t code = 0;
for (SListCell* cell = pCell; (NULL != cell ? (node = cell->pNode, true) : (node = NULL, false)); cell = cell->pNext) {
pWhenThen = (SWhenThenNode*)node;
SCL_ERR_RET(sclGetNodeRes(pWhenThen->pWhen, ctx, &pWhen));
SCL_ERR_RET(sclGetNodeRes(pWhenThen->pThen, ctx, &pThen));
vectorCompareImpl(pCase, pWhen, pComp, rowIdx, 1, TSDB_ORDER_ASC, OP_TYPE_EQUAL);
bool *equal = (bool*)colDataGetData(pComp->columnData, rowIdx);
if (*equal) {
colDataAppend(output->columnData, rowIdx, colDataGetData(pThen->columnData, (pThen->numOfRows > 1 ? rowIdx : 0)), colDataIsNull_s(pThen->columnData, (pThen->numOfRows > 1 ? rowIdx : 0)));
if (0 == rowIdx && 1 == pCase->numOfRows && 1 == pWhen->numOfRows && 1 == pThen->numOfRows && totalRows > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
*complete = true;
}
goto _return;
}
}
if (pElse) {
colDataAppend(output->columnData, rowIdx, colDataGetData(pElse->columnData, (pElse->numOfRows > 1 ? rowIdx : 0)), colDataIsNull_s(pElse->columnData, (pElse->numOfRows > 1 ? rowIdx : 0)));
if (0 == rowIdx && 1 == pCase->numOfRows && 1 == pElse->numOfRows && totalRows > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
*complete = true;
}
goto _return;
}
colDataAppend(output->columnData, rowIdx, NULL, true);
if (0 == rowIdx && 1 == pCase->numOfRows && totalRows > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
*complete = true;
}
_return:
sclFreeParam(pWhen);
sclFreeParam(pThen);
SCL_RET(code);
}
int32_t sclWalkWhenList(SScalarCtx *ctx, SNodeList* pList, struct SListCell* pCell, SScalarParam *pElse, SScalarParam *output,
int32_t rowIdx, int32_t totalRows, bool *complete, bool preSingle) {
SNode *node = NULL;
SWhenThenNode* pWhenThen = NULL;
SScalarParam *pWhen = NULL;
SScalarParam *pThen = NULL;
int32_t code = 0;
for (SListCell* cell = pCell; (NULL != cell ? (node = cell->pNode, true) : (node = NULL, false)); cell = cell->pNext) {
pWhenThen = (SWhenThenNode*)node;
pWhen = NULL;
pThen = NULL;
SCL_ERR_JRET(sclGetNodeRes(pWhenThen->pWhen, ctx, &pWhen));
SCL_ERR_JRET(sclGetNodeRes(pWhenThen->pThen, ctx, &pThen));
bool *whenValue = (bool*)colDataGetData(pWhen->columnData, (pWhen->numOfRows > 1 ? rowIdx : 0));
if (*whenValue) {
colDataAppend(output->columnData, rowIdx, colDataGetData(pThen->columnData, (pThen->numOfRows > 1 ? rowIdx : 0)), colDataIsNull_s(pThen->columnData, (pThen->numOfRows > 1 ? rowIdx : 0)));
if (preSingle && 0 == rowIdx && 1 == pWhen->numOfRows && 1 == pThen->numOfRows && totalRows > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
*complete = true;
}
goto _return;
}
sclFreeParam(pWhen);
sclFreeParam(pThen);
}
if (pElse) {
colDataAppend(output->columnData, rowIdx, colDataGetData(pElse->columnData, (pElse->numOfRows > 1 ? rowIdx : 0)), colDataIsNull_s(pElse->columnData, (pElse->numOfRows > 1 ? rowIdx : 0)));
if (preSingle && 0 == rowIdx && 1 == pElse->numOfRows && totalRows > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
*complete = true;
}
goto _return;
}
colDataAppend(output->columnData, rowIdx, NULL, true);
if (preSingle && 0 == rowIdx && totalRows > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
*complete = true;
}
_return:
sclFreeParam(pWhen);
sclFreeParam(pThen);
SCL_RET(code);
}
int32_t sclExecFunction(SFunctionNode *node, SScalarCtx *ctx, SScalarParam *output) { int32_t sclExecFunction(SFunctionNode *node, SScalarCtx *ctx, SScalarParam *output) {
SScalarParam *params = NULL; SScalarParam *params = NULL;
int32_t rowNum = 0; int32_t rowNum = 0;
@ -698,6 +850,101 @@ _return:
SCL_RET(code); SCL_RET(code);
} }
int32_t sclExecCaseWhen(SCaseWhenNode *node, SScalarCtx *ctx, SScalarParam *output) {
int32_t code = 0;
SScalarParam *pCase = NULL;
SScalarParam *pElse = NULL;
SScalarParam *pWhen = NULL;
SScalarParam *pThen = NULL;
SScalarParam comp = {0};
int32_t rowNum = 1;
bool complete = false;
if (NULL == node->pWhenThenList || node->pWhenThenList->length <= 0) {
sclError("invalid whenThen list");
SCL_ERR_RET(TSDB_CODE_INVALID_PARA);
}
if (ctx->pBlockList) {
SSDataBlock* pb = taosArrayGetP(ctx->pBlockList, 0);
rowNum = pb->info.rows;
output->numOfRows = pb->info.rows;
}
SCL_ERR_JRET(sclCreateColumnInfoData(&node->node.resType, rowNum, output));
SCL_ERR_JRET(sclGetNodeRes(node->pCase, ctx, &pCase));
SCL_ERR_JRET(sclGetNodeRes(node->pElse, ctx, &pElse));
SDataType compType = {0};
compType.type = TSDB_DATA_TYPE_BOOL;
compType.bytes = tDataTypes[compType.type].bytes;
SCL_ERR_JRET(sclCreateColumnInfoData(&compType, rowNum, &comp));
SNode* tnode = NULL;
SWhenThenNode* pWhenThen = (SWhenThenNode*)node->pWhenThenList->pHead->pNode;
SCL_ERR_JRET(sclGetNodeRes(pWhenThen->pWhen, ctx, &pWhen));
SCL_ERR_JRET(sclGetNodeRes(pWhenThen->pThen, ctx, &pThen));
if (pCase) {
vectorCompare(pCase, pWhen, &comp, TSDB_ORDER_ASC, OP_TYPE_EQUAL);
for (int32_t i = 0; i < rowNum; ++i) {
bool *equal = (bool*)colDataGetData(comp.columnData, (comp.numOfRows > 1 ? i : 0));
if (*equal) {
colDataAppend(output->columnData, i, colDataGetData(pThen->columnData, (pThen->numOfRows > 1 ? i : 0)), colDataIsNull_s(pThen->columnData, (pThen->numOfRows > 1 ? i : 0)));
if (0 == i && 1 == pCase->numOfRows && 1 == pWhen->numOfRows && 1 == pThen->numOfRows && rowNum > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
break;
}
} else {
SCL_ERR_JRET(sclWalkCaseWhenList(ctx, node->pWhenThenList, node->pWhenThenList->pHead->pNext, pCase, pElse, &comp, output, i, rowNum, &complete));
if (complete) {
break;
}
}
}
} else {
for (int32_t i = 0; i < rowNum; ++i) {
bool *whenValue = (bool*)colDataGetData(pWhen->columnData, (pWhen->numOfRows > 1 ? i : 0));
if (*whenValue) {
colDataAppend(output->columnData, i, colDataGetData(pThen->columnData, (pThen->numOfRows > 1 ? i : 0)), colDataIsNull_s(pThen->columnData, (pThen->numOfRows > 1 ? i : 0)));
if (0 == i && 1 == pWhen->numOfRows && 1 == pThen->numOfRows && rowNum > 1) {
SCL_ERR_JRET(sclExtendResRows(output, output, ctx->pBlockList));
break;
}
} else {
SCL_ERR_JRET(sclWalkWhenList(ctx, node->pWhenThenList, node->pWhenThenList->pHead->pNext, pElse, output, i, rowNum, &complete, (pWhen->numOfRows == 1 && pThen->numOfRows == 1)));
if (complete) {
break;
}
}
}
}
sclFreeParam(pCase);
sclFreeParam(pElse);
sclFreeParam(&comp);
sclFreeParam(pWhen);
sclFreeParam(pThen);
return TSDB_CODE_SUCCESS;
_return:
sclFreeParam(pCase);
sclFreeParam(pElse);
sclFreeParam(&comp);
sclFreeParam(pWhen);
sclFreeParam(pThen);
sclFreeParam(output);
SCL_RET(code);
}
EDealRes sclRewriteNullInOptr(SNode **pNode, SScalarCtx *ctx, EOperatorType opType) { EDealRes sclRewriteNullInOptr(SNode **pNode, SScalarCtx *ctx, EOperatorType opType) {
if (opType <= OP_TYPE_CALC_MAX) { if (opType <= OP_TYPE_CALC_MAX) {
SValueNode *res = (SValueNode *)nodesMakeNode(QUERY_NODE_VALUE); SValueNode *res = (SValueNode *)nodesMakeNode(QUERY_NODE_VALUE);
@ -960,9 +1207,66 @@ EDealRes sclRewriteOperator(SNode **pNode, SScalarCtx *ctx) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
EDealRes sclRewriteCaseWhen(SNode** pNode, SScalarCtx *ctx) {
SCaseWhenNode *node = (SCaseWhenNode *)*pNode;
if ((!SCL_IS_CONST_NODE(node->pCase)) || (!SCL_IS_CONST_NODE(node->pElse))) {
return DEAL_RES_CONTINUE;
}
SNode* tnode = NULL;
FOREACH(tnode, node->pWhenThenList) {
SWhenThenNode* pWhenThen = (SWhenThenNode*)tnode;
if (!SCL_IS_CONST_NODE(pWhenThen->pWhen) || !SCL_IS_CONST_NODE(pWhenThen->pThen)) {
return DEAL_RES_CONTINUE;
}
}
SScalarParam output = {0};
ctx->code = sclExecCaseWhen(node, ctx, &output);
if (ctx->code) {
return DEAL_RES_ERROR;
}
SValueNode *res = (SValueNode *)nodesMakeNode(QUERY_NODE_VALUE);
if (NULL == res) {
sclError("make value node failed");
sclFreeParam(&output);
ctx->code = TSDB_CODE_QRY_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
}
res->translate = true;
res->node.resType = node->node.resType;
if (colDataIsNull_s(output.columnData, 0)) {
res->isNull = true;
res->node.resType = node->node.resType;
} else {
int32_t type = output.columnData->info.type;
if (IS_VAR_DATA_TYPE(type)) { // todo refactor
res->datum.p = output.columnData->pData;
output.columnData->pData = NULL;
} else {
nodesSetValueNodeValue(res, output.columnData->pData);
}
}
nodesDestroyNode(*pNode);
*pNode = (SNode*)res;
sclFreeParam(&output);
return DEAL_RES_CONTINUE;
}
EDealRes sclConstantsRewriter(SNode **pNode, void *pContext) { EDealRes sclConstantsRewriter(SNode **pNode, void *pContext) {
SScalarCtx *ctx = (SScalarCtx *)pContext; SScalarCtx *ctx = (SScalarCtx *)pContext;
if (QUERY_NODE_OPERATOR == nodeType(*pNode)) {
return sclRewriteOperator(pNode, ctx);
}
if (QUERY_NODE_FUNCTION == nodeType(*pNode)) { if (QUERY_NODE_FUNCTION == nodeType(*pNode)) {
return sclRewriteFunction(pNode, ctx); return sclRewriteFunction(pNode, ctx);
} }
@ -971,8 +1275,8 @@ EDealRes sclConstantsRewriter(SNode **pNode, void *pContext) {
return sclRewriteLogic(pNode, ctx); return sclRewriteLogic(pNode, ctx);
} }
if (QUERY_NODE_OPERATOR == nodeType(*pNode)) { if (QUERY_NODE_CASE_WHEN == nodeType(*pNode)) {
return sclRewriteOperator(pNode, ctx); return sclRewriteCaseWhen(pNode, ctx);
} }
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
@ -1082,13 +1386,36 @@ EDealRes sclWalkTarget(SNode *pNode, SScalarCtx *ctx) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
EDealRes sclWalkCaseWhen(SNode* pNode, SScalarCtx *ctx) {
SCaseWhenNode *node = (SCaseWhenNode *)pNode;
SScalarParam output = {0};
ctx->code = sclExecCaseWhen(node, ctx, &output);
if (ctx->code) {
return DEAL_RES_ERROR;
}
if (taosHashPut(ctx->pRes, &pNode, POINTER_BYTES, &output, sizeof(output))) {
ctx->code = TSDB_CODE_QRY_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
}
return DEAL_RES_CONTINUE;
}
EDealRes sclCalcWalker(SNode *pNode, void *pContext) { EDealRes sclCalcWalker(SNode *pNode, void *pContext) {
if (QUERY_NODE_VALUE == nodeType(pNode) || QUERY_NODE_NODE_LIST == nodeType(pNode) || if (QUERY_NODE_VALUE == nodeType(pNode) || QUERY_NODE_NODE_LIST == nodeType(pNode)
QUERY_NODE_COLUMN == nodeType(pNode) || QUERY_NODE_LEFT_VALUE == nodeType(pNode)) { || QUERY_NODE_COLUMN == nodeType(pNode) || QUERY_NODE_LEFT_VALUE == nodeType(pNode)
|| QUERY_NODE_WHEN_THEN == nodeType(pNode)) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
SScalarCtx *ctx = (SScalarCtx *)pContext; SScalarCtx *ctx = (SScalarCtx *)pContext;
if (QUERY_NODE_OPERATOR == nodeType(pNode)) {
return sclWalkOperator(pNode, ctx);
}
if (QUERY_NODE_FUNCTION == nodeType(pNode)) { if (QUERY_NODE_FUNCTION == nodeType(pNode)) {
return sclWalkFunction(pNode, ctx); return sclWalkFunction(pNode, ctx);
} }
@ -1097,38 +1424,19 @@ EDealRes sclCalcWalker(SNode *pNode, void *pContext) {
return sclWalkLogic(pNode, ctx); return sclWalkLogic(pNode, ctx);
} }
if (QUERY_NODE_OPERATOR == nodeType(pNode)) {
return sclWalkOperator(pNode, ctx);
}
if (QUERY_NODE_TARGET == nodeType(pNode)) { if (QUERY_NODE_TARGET == nodeType(pNode)) {
return sclWalkTarget(pNode, ctx); return sclWalkTarget(pNode, ctx);
} }
if (QUERY_NODE_CASE_WHEN == nodeType(pNode)) {
return sclWalkCaseWhen(pNode, ctx);
}
sclError("invalid node type for scalar calculating, type:%d", nodeType(pNode)); sclError("invalid node type for scalar calculating, type:%d", nodeType(pNode));
ctx->code = TSDB_CODE_QRY_INVALID_INPUT; ctx->code = TSDB_CODE_QRY_INVALID_INPUT;
return DEAL_RES_ERROR; return DEAL_RES_ERROR;
} }
int32_t sclExtendResRows(SScalarParam *pDst, SScalarParam *pSrc, SArray *pBlockList) {
SSDataBlock *pb = taosArrayGetP(pBlockList, 0);
SScalarParam *pLeft = taosMemoryCalloc(1, sizeof(SScalarParam));
if (NULL == pLeft) {
sclError("calloc %d failed", (int32_t)sizeof(SScalarParam));
SCL_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
pLeft->numOfRows = pb->info.rows;
colInfoDataEnsureCapacity(pDst->columnData, pb->info.rows);
_bin_scalar_fn_t OperatorFn = getBinScalarOperatorFn(OP_TYPE_ASSIGN);
OperatorFn(pLeft, pSrc, pDst, TSDB_ORDER_ASC);
taosMemoryFree(pLeft);
return TSDB_CODE_SUCCESS;
}
int32_t sclCalcConstants(SNode *pNode, bool dual, SNode **pRes) { int32_t sclCalcConstants(SNode *pNode, bool dual, SNode **pRes) {
if (NULL == pNode) { if (NULL == pNode) {
SCL_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT); SCL_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);

362
source/libs/scalar/src/sclvector.c
View File

@ -367,46 +367,45 @@ static FORCE_INLINE void ncharToVar(char *buf, SScalarParam *pOut, int32_t rowIn
taosMemoryFree(t); taosMemoryFree(t);
} }
// TODO opt performance, tmp is not needed. //TODO opt performance, tmp is not needed.
int32_t vectorConvertFromVarData(const SScalarParam *pIn, SScalarParam *pOut, int32_t inType, int32_t outType, int32_t vectorConvertFromVarData(SSclVectorConvCtx *pCtx, int32_t* overflow) {
int32_t *overflow) {
bool vton = false; bool vton = false;
_bufConverteFunc func = NULL; _bufConverteFunc func = NULL;
if (TSDB_DATA_TYPE_BOOL == outType) { if (TSDB_DATA_TYPE_BOOL == pCtx->outType) {
func = varToBool; func = varToBool;
} else if (IS_SIGNED_NUMERIC_TYPE(outType)) { } else if (IS_SIGNED_NUMERIC_TYPE(pCtx->outType)) {
func = varToSigned; func = varToSigned;
} else if (IS_UNSIGNED_NUMERIC_TYPE(outType)) { } else if (IS_UNSIGNED_NUMERIC_TYPE(pCtx->outType)) {
func = varToUnsigned; func = varToUnsigned;
} else if (IS_FLOAT_TYPE(outType)) { } else if (IS_FLOAT_TYPE(pCtx->outType)) {
func = varToFloat; func = varToFloat;
} else if (outType == TSDB_DATA_TYPE_BINARY) { // nchar -> binary } else if (pCtx->outType == TSDB_DATA_TYPE_BINARY) { // nchar -> binary
ASSERT(inType == TSDB_DATA_TYPE_NCHAR); ASSERT(pCtx->inType == TSDB_DATA_TYPE_NCHAR);
func = ncharToVar; func = ncharToVar;
vton = true; vton = true;
} else if (outType == TSDB_DATA_TYPE_NCHAR) { // binary -> nchar } else if (pCtx->outType == TSDB_DATA_TYPE_NCHAR) { // binary -> nchar
ASSERT(inType == TSDB_DATA_TYPE_VARCHAR); ASSERT(pCtx->inType == TSDB_DATA_TYPE_VARCHAR);
func = varToNchar; func = varToNchar;
vton = true; vton = true;
} else if (TSDB_DATA_TYPE_TIMESTAMP == outType) { } else if (TSDB_DATA_TYPE_TIMESTAMP == pCtx->outType) {
func = varToTimestamp; func = varToTimestamp;
} else { } else {
sclError("invalid convert outType:%d", outType); sclError("invalid convert outType:%d", pCtx->outType);
return TSDB_CODE_QRY_APP_ERROR; return TSDB_CODE_QRY_APP_ERROR;
} }
pOut->numOfRows = pIn->numOfRows; pCtx->pOut->numOfRows = pCtx->pIn->numOfRows;
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = pCtx->startIndex; i <= pCtx->endIndex; ++i) {
if (IS_HELPER_NULL(pIn->columnData, i)) { if (IS_HELPER_NULL(pCtx->pIn->columnData, i)) {
colDataAppendNULL(pOut->columnData, i); colDataAppendNULL(pCtx->pOut->columnData, i);
continue; continue;
} }
char *data = colDataGetVarData(pIn->columnData, i); char* data = colDataGetVarData(pCtx->pIn->columnData, i);
int32_t convertType = inType; int32_t convertType = pCtx->inType;
if (inType == TSDB_DATA_TYPE_JSON) { if(pCtx->inType == TSDB_DATA_TYPE_JSON){
if (*data == TSDB_DATA_TYPE_NULL) { if(*data == TSDB_DATA_TYPE_NULL) {
ASSERT(0); ASSERT(0);
} else if (*data == TSDB_DATA_TYPE_NCHAR) { } else if (*data == TSDB_DATA_TYPE_NCHAR) {
data += CHAR_BYTES; data += CHAR_BYTES;
@ -415,13 +414,13 @@ int32_t vectorConvertFromVarData(const SScalarParam *pIn, SScalarParam *pOut, in
terrno = TSDB_CODE_QRY_JSON_NOT_SUPPORT_ERROR; terrno = TSDB_CODE_QRY_JSON_NOT_SUPPORT_ERROR;
return terrno; return terrno;
} else { } else {
convertNumberToNumber(data + CHAR_BYTES, colDataGetNumData(pOut->columnData, i), *data, outType); convertNumberToNumber(data+CHAR_BYTES, colDataGetNumData(pCtx->pOut->columnData, i), *data, pCtx->outType);
continue; continue;
} }
} }
int32_t bufSize = pIn->columnData->info.bytes; int32_t bufSize = pCtx->pIn->columnData->info.bytes;
char *tmp = taosMemoryMalloc(varDataTLen(data)); char *tmp = taosMemoryMalloc(varDataTLen(data));
if (!tmp) { if(!tmp){
sclError("out of memory in vectorConvertFromVarData"); sclError("out of memory in vectorConvertFromVarData");
return TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_OUT_OF_MEMORY;
} }
@ -444,8 +443,8 @@ int32_t vectorConvertFromVarData(const SScalarParam *pIn, SScalarParam *pOut, in
tmp[len] = 0; tmp[len] = 0;
} }
} }
(*func)(tmp, pOut, i, overflow); (*func)(tmp, pCtx->pOut, i, overflow);
taosMemoryFreeClear(tmp); taosMemoryFreeClear(tmp);
} }
@ -581,64 +580,64 @@ bool convertJsonValue(__compar_fn_t *fp, int32_t optr, int8_t typeLeft, int8_t t
return true; return true;
} }
int32_t vectorConvertToVarData(const SScalarParam *pIn, SScalarParam *pOut, int16_t inType, int16_t outType) { int32_t vectorConvertToVarData(SSclVectorConvCtx *pCtx) {
SColumnInfoData *pInputCol = pIn->columnData; SColumnInfoData* pInputCol = pCtx->pIn->columnData;
SColumnInfoData *pOutputCol = pOut->columnData; SColumnInfoData* pOutputCol = pCtx->pOut->columnData;
char tmp[128] = {0}; char tmp[128] = {0};
if (IS_SIGNED_NUMERIC_TYPE(inType) || inType == TSDB_DATA_TYPE_BOOL || inType == TSDB_DATA_TYPE_TIMESTAMP) { if (IS_SIGNED_NUMERIC_TYPE(pCtx->inType) || pCtx->inType == TSDB_DATA_TYPE_BOOL || pCtx->inType == TSDB_DATA_TYPE_TIMESTAMP) {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = pCtx->startIndex; i <= pCtx->endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
int64_t value = 0; int64_t value = 0;
GET_TYPED_DATA(value, int64_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, int64_t, pCtx->inType, colDataGetData(pInputCol, i));
int32_t len = sprintf(varDataVal(tmp), "%" PRId64, value); int32_t len = sprintf(varDataVal(tmp), "%" PRId64, value);
varDataLen(tmp) = len; varDataLen(tmp) = len;
if (outType == TSDB_DATA_TYPE_NCHAR) { if (pCtx->outType == TSDB_DATA_TYPE_NCHAR) {
varToNchar(tmp, pOut, i, NULL); varToNchar(tmp, pCtx->pOut, i, NULL);
} else { } else {
colDataAppend(pOutputCol, i, (char *)tmp, false); colDataAppend(pOutputCol, i, (char *)tmp, false);
} }
} }
} else if (IS_UNSIGNED_NUMERIC_TYPE(inType)) { } else if (IS_UNSIGNED_NUMERIC_TYPE(pCtx->inType)) {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = pCtx->startIndex; i <= pCtx->endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
uint64_t value = 0; uint64_t value = 0;
GET_TYPED_DATA(value, uint64_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, uint64_t, pCtx->inType, colDataGetData(pInputCol, i));
int32_t len = sprintf(varDataVal(tmp), "%" PRIu64, value); int32_t len = sprintf(varDataVal(tmp), "%" PRIu64, value);
varDataLen(tmp) = len; varDataLen(tmp) = len;
if (outType == TSDB_DATA_TYPE_NCHAR) { if (pCtx->outType == TSDB_DATA_TYPE_NCHAR) {
varToNchar(tmp, pOut, i, NULL); varToNchar(tmp, pCtx->pOut, i, NULL);
} else { } else {
colDataAppend(pOutputCol, i, (char *)tmp, false); colDataAppend(pOutputCol, i, (char *)tmp, false);
} }
} }
} else if (IS_FLOAT_TYPE(inType)) { } else if (IS_FLOAT_TYPE(pCtx->inType)) {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = pCtx->startIndex; i <= pCtx->endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
double value = 0; double value = 0;
GET_TYPED_DATA(value, double, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, double, pCtx->inType, colDataGetData(pInputCol, i));
int32_t len = sprintf(varDataVal(tmp), "%lf", value); int32_t len = sprintf(varDataVal(tmp), "%lf", value);
varDataLen(tmp) = len; varDataLen(tmp) = len;
if (outType == TSDB_DATA_TYPE_NCHAR) { if (pCtx->outType == TSDB_DATA_TYPE_NCHAR) {
varToNchar(tmp, pOut, i, NULL); varToNchar(tmp, pCtx->pOut, i, NULL);
} else { } else {
colDataAppend(pOutputCol, i, (char *)tmp, false); colDataAppend(pOutputCol, i, (char *)tmp, false);
} }
} }
} else { } else {
sclError("not supported input type:%d", inType); sclError("not supported input type:%d", pCtx->inType);
return TSDB_CODE_QRY_APP_ERROR; return TSDB_CODE_QRY_APP_ERROR;
} }
@ -646,34 +645,35 @@ int32_t vectorConvertToVarData(const SScalarParam *pIn, SScalarParam *pOut, int1
} }
// TODO opt performance // TODO opt performance
int32_t vectorConvertImpl(const SScalarParam *pIn, SScalarParam *pOut, int32_t *overflow) { int32_t vectorConvertSingleColImpl(const SScalarParam* pIn, SScalarParam* pOut, int32_t* overflow, int32_t startIndex, int32_t numOfRows) {
SColumnInfoData *pInputCol = pIn->columnData; SColumnInfoData* pInputCol = pIn->columnData;
SColumnInfoData *pOutputCol = pOut->columnData; SColumnInfoData* pOutputCol = pOut->columnData;
if (NULL == pInputCol) { if (NULL == pInputCol) {
sclError("input column is NULL, hashFilter %p", pIn->pHashFilter); sclError("input column is NULL, hashFilter %p", pIn->pHashFilter);
return TSDB_CODE_APP_ERROR; return TSDB_CODE_APP_ERROR;
} }
int16_t inType = pInputCol->info.type; int32_t rstart = startIndex >= 0 ? startIndex : 0;
int16_t outType = pOutputCol->info.type; int32_t rend = numOfRows > 0 ? rstart + numOfRows - 1 : rstart + pIn->numOfRows - 1;
SSclVectorConvCtx cCtx = {pIn, pOut, rstart, rend, pInputCol->info.type, pOutputCol->info.type};
if (IS_VAR_DATA_TYPE(inType)) { if (IS_VAR_DATA_TYPE(cCtx.inType)) {
return vectorConvertFromVarData(pIn, pOut, inType, outType, overflow); return vectorConvertFromVarData(&cCtx, overflow);
} }
if (overflow) { if (overflow) {
ASSERT(1 == pIn->numOfRows); ASSERT(1 == pIn->numOfRows);
pOut->numOfRows = 0; pOut->numOfRows = 0;
if (IS_SIGNED_NUMERIC_TYPE(outType)) { if (IS_SIGNED_NUMERIC_TYPE(cCtx.outType)) {
int64_t minValue = tDataTypes[outType].minValue; int64_t minValue = tDataTypes[cCtx.outType].minValue;
int64_t maxValue = tDataTypes[outType].maxValue; int64_t maxValue = tDataTypes[cCtx.outType].maxValue;
double value = 0; double value = 0;
GET_TYPED_DATA(value, double, inType, colDataGetData(pInputCol, 0)); GET_TYPED_DATA(value, double, cCtx.inType, colDataGetData(pInputCol, 0));
if (value > maxValue) { if (value > maxValue) {
*overflow = 1; *overflow = 1;
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -683,13 +683,13 @@ int32_t vectorConvertImpl(const SScalarParam *pIn, SScalarParam *pOut, int32_t *
} else { } else {
*overflow = 0; *overflow = 0;
} }
} else if (IS_UNSIGNED_NUMERIC_TYPE(outType)) { } else if (IS_UNSIGNED_NUMERIC_TYPE(cCtx.outType)) {
uint64_t minValue = (uint64_t)tDataTypes[outType].minValue; uint64_t minValue = (uint64_t)tDataTypes[cCtx.outType].minValue;
uint64_t maxValue = (uint64_t)tDataTypes[outType].maxValue; uint64_t maxValue = (uint64_t)tDataTypes[cCtx.outType].maxValue;
double value = 0; double value = 0;
GET_TYPED_DATA(value, double, inType, colDataGetData(pInputCol, 0)); GET_TYPED_DATA(value, double, cCtx.inType, colDataGetData(pInputCol, 0));
if (value > maxValue) { if (value > maxValue) {
*overflow = 1; *overflow = 1;
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -703,157 +703,157 @@ int32_t vectorConvertImpl(const SScalarParam *pIn, SScalarParam *pOut, int32_t *
} }
pOut->numOfRows = pIn->numOfRows; pOut->numOfRows = pIn->numOfRows;
switch (outType) { switch (cCtx.outType) {
case TSDB_DATA_TYPE_BOOL: { case TSDB_DATA_TYPE_BOOL: {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
bool value = 0; bool value = 0;
GET_TYPED_DATA(value, bool, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, bool, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt8(pOutputCol, i, (int8_t *)&value); colDataAppendInt8(pOutputCol, i, (int8_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_TINYINT: { case TSDB_DATA_TYPE_TINYINT: {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
int8_t value = 0; int8_t value = 0;
GET_TYPED_DATA(value, int8_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, int8_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt8(pOutputCol, i, (int8_t *)&value); colDataAppendInt8(pOutputCol, i, (int8_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_SMALLINT: { case TSDB_DATA_TYPE_SMALLINT:{
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
int16_t value = 0; int16_t value = 0;
GET_TYPED_DATA(value, int16_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, int16_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt16(pOutputCol, i, (int16_t *)&value); colDataAppendInt16(pOutputCol, i, (int16_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_INT: { case TSDB_DATA_TYPE_INT:{
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
int32_t value = 0; int32_t value = 0;
GET_TYPED_DATA(value, int32_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, int32_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt32(pOutputCol, i, (int32_t *)&value); colDataAppendInt32(pOutputCol, i, (int32_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_BIGINT: case TSDB_DATA_TYPE_BIGINT:
case TSDB_DATA_TYPE_TIMESTAMP: { case TSDB_DATA_TYPE_TIMESTAMP: {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
int64_t value = 0; int64_t value = 0;
GET_TYPED_DATA(value, int64_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, int64_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt64(pOutputCol, i, (int64_t *)&value); colDataAppendInt64(pOutputCol, i, (int64_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_UTINYINT: { case TSDB_DATA_TYPE_UTINYINT:{
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
uint8_t value = 0; uint8_t value = 0;
GET_TYPED_DATA(value, uint8_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, uint8_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt8(pOutputCol, i, (int8_t *)&value); colDataAppendInt8(pOutputCol, i, (int8_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_USMALLINT: { case TSDB_DATA_TYPE_USMALLINT:{
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
uint16_t value = 0; uint16_t value = 0;
GET_TYPED_DATA(value, uint16_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, uint16_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt16(pOutputCol, i, (int16_t *)&value); colDataAppendInt16(pOutputCol, i, (int16_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_UINT: { case TSDB_DATA_TYPE_UINT:{
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
uint32_t value = 0; uint32_t value = 0;
GET_TYPED_DATA(value, uint32_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, uint32_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt32(pOutputCol, i, (int32_t *)&value); colDataAppendInt32(pOutputCol, i, (int32_t *)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_UBIGINT: { case TSDB_DATA_TYPE_UBIGINT: {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
uint64_t value = 0; uint64_t value = 0;
GET_TYPED_DATA(value, uint64_t, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, uint64_t, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendInt64(pOutputCol, i, (int64_t *)&value); colDataAppendInt64(pOutputCol, i, (int64_t*)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_FLOAT: { case TSDB_DATA_TYPE_FLOAT:{
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
float value = 0; float value = 0;
GET_TYPED_DATA(value, float, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, float, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendFloat(pOutputCol, i, (float *)&value); colDataAppendFloat(pOutputCol, i, (float*)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_DOUBLE: { case TSDB_DATA_TYPE_DOUBLE: {
for (int32_t i = 0; i < pIn->numOfRows; ++i) { for (int32_t i = cCtx.startIndex; i <= cCtx.endIndex; ++i) {
if (colDataIsNull_f(pInputCol->nullbitmap, i)) { if (colDataIsNull_f(pInputCol->nullbitmap, i)) {
colDataAppendNULL(pOutputCol, i); colDataAppendNULL(pOutputCol, i);
continue; continue;
} }
double value = 0; double value = 0;
GET_TYPED_DATA(value, double, inType, colDataGetData(pInputCol, i)); GET_TYPED_DATA(value, double, cCtx.inType, colDataGetData(pInputCol, i));
colDataAppendDouble(pOutputCol, i, (double *)&value); colDataAppendDouble(pOutputCol, i, (double*)&value);
} }
break; break;
} }
case TSDB_DATA_TYPE_BINARY: case TSDB_DATA_TYPE_BINARY:
case TSDB_DATA_TYPE_NCHAR: { case TSDB_DATA_TYPE_NCHAR: {
return vectorConvertToVarData(pIn, pOut, inType, outType); return vectorConvertToVarData(&cCtx);
} }
default: default:
sclError("invalid convert output type:%d", outType); sclError("invalid convert output type:%d", cCtx.outType);
return TSDB_CODE_QRY_APP_ERROR; return TSDB_CODE_QRY_APP_ERROR;
} }
@ -894,7 +894,7 @@ int32_t vectorGetConvertType(int32_t type1, int32_t type2) {
return gConvertTypes[type2][type1]; return gConvertTypes[type2][type1];
} }
int32_t vectorConvertScalarParam(SScalarParam *input, SScalarParam *output, int32_t type) { int32_t vectorConvertSingleCol(SScalarParam *input, SScalarParam *output, int32_t type, int32_t startIndex, int32_t numOfRows) {
SDataType t = {.type = type, .bytes = tDataTypes[type].bytes}; SDataType t = {.type = type, .bytes = tDataTypes[type].bytes};
output->numOfRows = input->numOfRows; output->numOfRows = input->numOfRows;
@ -903,17 +903,16 @@ int32_t vectorConvertScalarParam(SScalarParam *input, SScalarParam *output, int3
return TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_OUT_OF_MEMORY;
} }
code = vectorConvertImpl(input, output, NULL); code = vectorConvertSingleColImpl(input, output, NULL, startIndex, numOfRows);
if (code) { if (code) {
// taosMemoryFreeClear(paramOut1->data);
return code; return code;
} }
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t vectorConvert(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pLeftOut, SScalarParam *pRightOut) { int32_t vectorConvertCols(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam* pLeftOut, SScalarParam* pRightOut, int32_t startIndex, int32_t numOfRows) {
int32_t leftType = GET_PARAM_TYPE(pLeft); int32_t leftType = GET_PARAM_TYPE(pLeft);
int32_t rightType = GET_PARAM_TYPE(pRight); int32_t rightType = GET_PARAM_TYPE(pRight);
if (leftType == rightType) { if (leftType == rightType) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -941,14 +940,14 @@ int32_t vectorConvert(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *p
} }
if (type != GET_PARAM_TYPE(param1)) { if (type != GET_PARAM_TYPE(param1)) {
code = vectorConvertScalarParam(param1, paramOut1, type); code = vectorConvertSingleCol(param1, paramOut1, type, startIndex, numOfRows);
if (code) { if (code) {
return code; return code;
} }
} }
if (type != GET_PARAM_TYPE(param2)) { if (type != GET_PARAM_TYPE(param2)) {
code = vectorConvertScalarParam(param2, paramOut2, type); code = vectorConvertSingleCol(param2, paramOut2, type, startIndex, numOfRows);
if (code) { if (code) {
return code; return code;
} }
@ -962,31 +961,6 @@ enum {
VECTOR_UN_CONVERT = 0x2, VECTOR_UN_CONVERT = 0x2,
}; };
static int32_t doConvertHelper(SScalarParam *pDest, int32_t *convert, const SScalarParam *pParam, int32_t type) {
SColumnInfoData *pCol = pParam->columnData;
if (IS_VAR_DATA_TYPE(pCol->info.type) && pCol->info.type != TSDB_DATA_TYPE_JSON) {
pDest->numOfRows = pParam->numOfRows;
SDataType t = {.type = type, .bytes = tDataTypes[type].bytes};
int32_t code = sclCreateColumnInfoData(&t, pParam->numOfRows, pDest);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
code = vectorConvertImpl(pParam, pDest, NULL);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
*convert = VECTOR_DO_CONVERT;
} else {
*convert = VECTOR_UN_CONVERT;
}
return TSDB_CODE_SUCCESS;
}
// TODO not correct for descending order scan // TODO not correct for descending order scan
static void vectorMathAddHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol, static void vectorMathAddHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t i) { int32_t numOfRows, int32_t step, int32_t i) {
@ -1030,20 +1004,25 @@ static void vectorMathTsAddHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pR
} }
} }
static SColumnInfoData *doVectorConvert(SScalarParam *pInput, int32_t *doConvert) { static SColumnInfoData* vectorConvertVarToDouble(SScalarParam* pInput, int32_t* converted) {
SScalarParam convertParam = {0}; SScalarParam output = {0};
SColumnInfoData* pCol = pInput->columnData;
int32_t code = doConvertHelper(&convertParam, doConvert, pInput, TSDB_DATA_TYPE_DOUBLE); if (IS_VAR_DATA_TYPE(pCol->info.type) && pCol->info.type != TSDB_DATA_TYPE_JSON) {
if (code != TSDB_CODE_SUCCESS) { int32_t code = vectorConvertSingleCol(pInput, &output, TSDB_DATA_TYPE_DOUBLE, -1, -1);
terrno = code; if (code != TSDB_CODE_SUCCESS) {
return NULL; terrno = code;
return NULL;
}
*converted = VECTOR_DO_CONVERT;
return output.columnData;
} }
if (*doConvert == VECTOR_DO_CONVERT) { *converted = VECTOR_UN_CONVERT;
return convertParam.columnData;
} else { return pInput->columnData;
return pInput->columnData;
}
} }
static void doReleaseVec(SColumnInfoData *pCol, int32_t type) { static void doReleaseVec(SColumnInfoData *pCol, int32_t type) {
@ -1061,9 +1040,9 @@ void vectorMathAdd(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut
pOut->numOfRows = TMAX(pLeft->numOfRows, pRight->numOfRows); pOut->numOfRows = TMAX(pLeft->numOfRows, pRight->numOfRows);
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
if ((GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_TIMESTAMP && IS_INTEGER_TYPE(GET_PARAM_TYPE(pRight))) || if ((GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_TIMESTAMP && IS_INTEGER_TYPE(GET_PARAM_TYPE(pRight))) ||
(GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_TIMESTAMP && IS_INTEGER_TYPE(GET_PARAM_TYPE(pLeft))) || (GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_TIMESTAMP && IS_INTEGER_TYPE(GET_PARAM_TYPE(pLeft))) ||
@ -1168,9 +1147,9 @@ void vectorMathSub(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
if ((GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_TIMESTAMP && GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_BIGINT) || if ((GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_TIMESTAMP && GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_BIGINT) ||
(GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_TIMESTAMP && (GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_TIMESTAMP &&
@ -1246,9 +1225,9 @@ void vectorMathMultiply(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
@ -1279,9 +1258,9 @@ void vectorMathDivide(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *p
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
@ -1333,9 +1312,9 @@ void vectorMathRemainder(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
@ -1412,8 +1391,8 @@ void vectorMathMinus(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pO
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : (pLeft->numOfRows - 1); int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : (pLeft->numOfRows - 1);
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0; int32_t leftConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type); _getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
@ -1525,9 +1504,9 @@ void vectorBitAnd(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut,
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type); _getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type); _getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
@ -1579,9 +1558,9 @@ void vectorBitOr(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut,
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t leftConvert = 0, rightConvert = 0; int32_t leftConvert = 0, rightConvert = 0;
SColumnInfoData *pLeftCol = doVectorConvert(pLeft, &leftConvert); SColumnInfoData *pLeftCol = vectorConvertVarToDouble(pLeft, &leftConvert);
SColumnInfoData *pRightCol = doVectorConvert(pRight, &rightConvert); SColumnInfoData *pRightCol = vectorConvertVarToDouble(pRight, &rightConvert);
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type); _getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type); _getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
@ -1605,8 +1584,8 @@ void vectorBitOr(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut,
doReleaseVec(pRightCol, rightConvert); doReleaseVec(pRightCol, rightConvert);
} }
int32_t doVectorCompareImpl(int32_t numOfRows, SScalarParam *pOut, int32_t startIndex, int32_t step, __compar_fn_t fp, int32_t doVectorCompareImpl(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut, int32_t startIndex, int32_t numOfRows,
SScalarParam *pLeft, SScalarParam *pRight, int32_t optr) { int32_t step, __compar_fn_t fp, int32_t optr) {
int32_t num = 0; int32_t num = 0;
for (int32_t i = startIndex; i < numOfRows && i >= 0; i += step) { for (int32_t i = startIndex; i < numOfRows && i >= 0; i += step) {
@ -1659,20 +1638,29 @@ int32_t doVectorCompareImpl(int32_t numOfRows, SScalarParam *pOut, int32_t start
return num; return num;
} }
void vectorCompareImpl(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut, int32_t _ord, int32_t optr) { void doVectorCompare(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t startIndex, int32_t numOfRows,
int32_t i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1; int32_t _ord, int32_t optr) {
int32_t i = 0;
int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1; int32_t step = ((_ord) == TSDB_ORDER_ASC) ? 1 : -1;
int32_t lType = GET_PARAM_TYPE(pLeft); int32_t lType = GET_PARAM_TYPE(pLeft);
int32_t rType = GET_PARAM_TYPE(pRight); int32_t rType = GET_PARAM_TYPE(pRight);
__compar_fn_t fp = NULL; __compar_fn_t fp = NULL;
int32_t compRows = 0;
if (lType == rType) { if (lType == rType) {
fp = filterGetCompFunc(lType, optr); fp = filterGetCompFunc(lType, optr);
} else { } else {
fp = filterGetCompFuncEx(lType, rType, optr); fp = filterGetCompFuncEx(lType, rType, optr);
} }
pOut->numOfRows = TMAX(pLeft->numOfRows, pRight->numOfRows); if (startIndex < 0) {
i = ((_ord) == TSDB_ORDER_ASC) ? 0 : TMAX(pLeft->numOfRows, pRight->numOfRows) - 1;
pOut->numOfRows = TMAX(pLeft->numOfRows, pRight->numOfRows);
compRows = pOut->numOfRows;
} else {
compRows = startIndex + numOfRows;
i = startIndex;
}
if (pRight->pHashFilter != NULL) { if (pRight->pHashFilter != NULL) {
for (; i >= 0 && i < pLeft->numOfRows; i += step) { for (; i >= 0 && i < pLeft->numOfRows; i += step) {
@ -1690,13 +1678,14 @@ void vectorCompareImpl(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *
} }
} }
} else { // normal compare } else { // normal compare
pOut->numOfQualified = doVectorCompareImpl(pOut->numOfRows, pOut, i, step, fp, pLeft, pRight, optr); pOut->numOfQualified = doVectorCompareImpl(pLeft, pRight, pOut, i, compRows, step, fp, optr);
} }
} }
void vectorCompare(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut, int32_t _ord, int32_t optr) { void vectorCompareImpl(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t startIndex, int32_t numOfRows,
SScalarParam pLeftOut = {0}; int32_t _ord, int32_t optr) {
SScalarParam pRightOut = {0}; SScalarParam pLeftOut = {0};
SScalarParam pRightOut = {0};
SScalarParam *param1 = NULL; SScalarParam *param1 = NULL;
SScalarParam *param2 = NULL; SScalarParam *param2 = NULL;
@ -1704,7 +1693,7 @@ void vectorCompare(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut
param1 = pLeft; param1 = pLeft;
param2 = pRight; param2 = pRight;
} else { } else {
vectorConvert(pLeft, pRight, &pLeftOut, &pRightOut); vectorConvertCols(pLeft, pRight, &pLeftOut, &pRightOut, startIndex, numOfRows);
if (pLeftOut.columnData != NULL) { if (pLeftOut.columnData != NULL) {
param1 = &pLeftOut; param1 = &pLeftOut;
@ -1719,12 +1708,17 @@ void vectorCompare(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut
} }
} }
vectorCompareImpl(param1, param2, pOut, _ord, optr); doVectorCompare(param1, param2, pOut, startIndex, numOfRows, _ord, optr);
sclFreeParam(&pLeftOut); sclFreeParam(&pLeftOut);
sclFreeParam(&pRightOut); sclFreeParam(&pRightOut);
} }
void vectorGreater(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut, int32_t _ord) { void vectorCompare(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t _ord, int32_t optr) {
vectorCompareImpl(pLeft, pRight, pOut, -1, -1, _ord, optr);
}
void vectorGreater(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t _ord) {
vectorCompare(pLeft, pRight, pOut, _ord, OP_TYPE_GREATER_THAN); vectorCompare(pLeft, pRight, pOut, _ord, OP_TYPE_GREATER_THAN);
} }
@ -1788,10 +1782,10 @@ void vectorNotNull(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut
pOut->numOfRows = pLeft->numOfRows; pOut->numOfRows = pLeft->numOfRows;
} }
void vectorIsTrue(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOut, int32_t _ord) { void vectorIsTrue(SScalarParam* pLeft, SScalarParam* pRight, SScalarParam *pOut, int32_t _ord) {
vectorConvertImpl(pLeft, pOut, NULL); vectorConvertSingleColImpl(pLeft, pOut, NULL, -1, -1);
for (int32_t i = 0; i < pOut->numOfRows; ++i) { for(int32_t i = 0; i < pOut->numOfRows; ++i) {
if (colDataIsNull_s(pOut->columnData, i)) { if(colDataIsNull_s(pOut->columnData, i)) {
int8_t v = 0; int8_t v = 0;
colDataAppendInt8(pOut->columnData, i, &v); colDataAppendInt8(pOut->columnData, i, &v);
colDataSetNotNull_f(pOut->columnData->nullbitmap, i); colDataSetNotNull_f(pOut->columnData->nullbitmap, i);

627
source/libs/scheduler/inc/schInt.h
View File

@ -20,13 +20,13 @@
extern "C" { extern "C" {
#endif #endif
#include "command.h"
#include "os.h" #include "os.h"
#include "tarray.h"
#include "planner.h" #include "planner.h"
#include "scheduler.h" #include "scheduler.h"
#include "tarray.h"
#include "thash.h" #include "thash.h"
#include "trpc.h" #include "trpc.h"
#include "command.h"
enum { enum {
SCH_READ = 1, SCH_READ = 1,
@ -51,20 +51,20 @@ typedef enum {
SCH_ALL, SCH_ALL,
} SCH_POLICY; } SCH_POLICY;
#define SCHEDULE_DEFAULT_MAX_JOB_NUM 1000 #define SCHEDULE_DEFAULT_MAX_JOB_NUM 1000
#define SCHEDULE_DEFAULT_MAX_TASK_NUM 1000 #define SCHEDULE_DEFAULT_MAX_TASK_NUM 1000
#define SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM 200 // unit is TSDB_TABLE_NUM_UNIT #define SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM 200 // unit is TSDB_TABLE_NUM_UNIT
#define SCHEDULE_DEFAULT_POLICY SCH_LOAD_SEQ #define SCHEDULE_DEFAULT_POLICY SCH_LOAD_SEQ
#define SCHEDULE_DEFAULT_MAX_NODE_NUM 20 #define SCHEDULE_DEFAULT_MAX_NODE_NUM 20
#define SCH_DEFAULT_TASK_TIMEOUT_USEC 10000000 #define SCH_DEFAULT_TASK_TIMEOUT_USEC 10000000
#define SCH_MAX_TASK_TIMEOUT_USEC 60000000 #define SCH_MAX_TASK_TIMEOUT_USEC 60000000
#define SCH_DEFAULT_MAX_RETRY_NUM 6 #define SCH_DEFAULT_MAX_RETRY_NUM 6
#define SCH_MIN_AYSNC_EXEC_NUM 3 #define SCH_MIN_AYSNC_EXEC_NUM 3
typedef struct SSchDebug { typedef struct SSchDebug {
bool lockEnable; bool lockEnable;
bool apiEnable; bool apiEnable;
} SSchDebug; } SSchDebug;
typedef struct SSchTrans { typedef struct SSchTrans {
@ -80,7 +80,6 @@ typedef struct SSchHbTrans {
} SSchHbTrans; } SSchHbTrans;
typedef struct SSchApiStat { typedef struct SSchApiStat {
#if defined(WINDOWS) || defined(_TD_DARWIN_64) #if defined(WINDOWS) || defined(_TD_DARWIN_64)
size_t avoidCompilationErrors; size_t avoidCompilationErrors;
#endif #endif
@ -88,7 +87,6 @@ typedef struct SSchApiStat {
} SSchApiStat; } SSchApiStat;
typedef struct SSchRuntimeStat { typedef struct SSchRuntimeStat {
#if defined(WINDOWS) || defined(_TD_DARWIN_64) #if defined(WINDOWS) || defined(_TD_DARWIN_64)
size_t avoidCompilationErrors; size_t avoidCompilationErrors;
#endif #endif
@ -96,7 +94,6 @@ typedef struct SSchRuntimeStat {
} SSchRuntimeStat; } SSchRuntimeStat;
typedef struct SSchJobStat { typedef struct SSchJobStat {
#if defined(WINDOWS) || defined(_TD_DARWIN_64) #if defined(WINDOWS) || defined(_TD_DARWIN_64)
size_t avoidCompilationErrors; size_t avoidCompilationErrors;
#endif #endif
@ -104,17 +101,17 @@ typedef struct SSchJobStat {
} SSchJobStat; } SSchJobStat;
typedef struct SSchStat { typedef struct SSchStat {
SSchApiStat api; SSchApiStat api;
SSchRuntimeStat runtime; SSchRuntimeStat runtime;
SSchJobStat job; SSchJobStat job;
} SSchStat; } SSchStat;
typedef struct SSchResInfo { typedef struct SSchResInfo {
SExecResult* execRes; SExecResult *execRes;
void** fetchRes; void **fetchRes;
schedulerExecFp execFp; schedulerExecFp execFp;
schedulerFetchFp fetchFp; schedulerFetchFp fetchFp;
void* cbParam; void *cbParam;
} SSchResInfo; } SSchResInfo;
typedef struct SSchOpEvent { typedef struct SSchOpEvent {
@ -123,9 +120,9 @@ typedef struct SSchOpEvent {
SSchedulerReq *pReq; SSchedulerReq *pReq;
} SSchOpEvent; } SSchOpEvent;
typedef int32_t (*schStatusEnterFp)(void* pHandle, void* pParam); typedef int32_t (*schStatusEnterFp)(void *pHandle, void *pParam);
typedef int32_t (*schStatusLeaveFp)(void* pHandle, void* pParam); typedef int32_t (*schStatusLeaveFp)(void *pHandle, void *pParam);
typedef int32_t (*schStatusEventFp)(void* pHandle, void* pParam, void* pEvent); typedef int32_t (*schStatusEventFp)(void *pHandle, void *pParam, void *pEvent);
typedef struct SSchStatusFps { typedef struct SSchStatusFps {
EJobTaskType status; EJobTaskType status;
@ -142,16 +139,16 @@ typedef struct SSchedulerCfg {
} SSchedulerCfg; } SSchedulerCfg;
typedef struct SSchedulerMgmt { typedef struct SSchedulerMgmt {
uint64_t taskId; // sequential taksId uint64_t taskId; // sequential taksId
uint64_t sId; // schedulerId uint64_t sId; // schedulerId
SSchedulerCfg cfg; SSchedulerCfg cfg;
bool exit; bool exit;
int32_t jobRef; int32_t jobRef;
int32_t jobNum; int32_t jobNum;
SSchStat stat; SSchStat stat;
SRWLatch hbLock; SRWLatch hbLock;
SHashObj *hbConnections; SHashObj *hbConnections;
void *queryMgmt; void *queryMgmt;
} SSchedulerMgmt; } SSchedulerMgmt;
typedef struct SSchCallbackParamHeader { typedef struct SSchCallbackParamHeader {
@ -174,11 +171,11 @@ typedef struct SSchHbCallbackParam {
} SSchHbCallbackParam; } SSchHbCallbackParam;
typedef struct SSchFlowControl { typedef struct SSchFlowControl {
SRWLatch lock; SRWLatch lock;
bool sorted; bool sorted;
int32_t tableNumSum; int32_t tableNumSum;
uint32_t execTaskNum; uint32_t execTaskNum;
SArray *taskList; // Element is SSchTask* SArray *taskList; // Element is SSchTask*
} SSchFlowControl; } SSchFlowControl;
typedef struct SSchNodeInfo { typedef struct SSchNodeInfo {
@ -187,50 +184,50 @@ typedef struct SSchNodeInfo {
} SSchNodeInfo; } SSchNodeInfo;
typedef struct SSchLevel { typedef struct SSchLevel {
int32_t level; int32_t level;
int8_t status; int8_t status;
SRWLatch lock; SRWLatch lock;
int32_t taskFailed; int32_t taskFailed;
int32_t taskSucceed; int32_t taskSucceed;
int32_t taskNum; int32_t taskNum;
int32_t taskLaunchedNum; int32_t taskLaunchedNum;
int32_t taskDoneNum; int32_t taskDoneNum;
SArray *subTasks; // Element is SSchTask SArray *subTasks; // Element is SSchTask
} SSchLevel; } SSchLevel;
typedef struct SSchTaskProfile { typedef struct SSchTaskProfile {
int64_t startTs; int64_t startTs;
SArray* execTime; SArray *execTime;
int64_t waitTime; int64_t waitTime;
int64_t endTs; int64_t endTs;
} SSchTaskProfile; } SSchTaskProfile;
typedef struct SSchTask { typedef struct SSchTask {
uint64_t taskId; // task id uint64_t taskId; // task id
SRWLatch lock; // task reentrant lock SRWLatch lock; // task reentrant lock
int32_t maxExecTimes; // task max exec times int32_t maxExecTimes; // task max exec times
int32_t maxRetryTimes; // task max retry times int32_t maxRetryTimes; // task max retry times
int32_t retryTimes; // task retry times int32_t retryTimes; // task retry times
bool waitRetry; // wait for retry bool waitRetry; // wait for retry
int32_t execId; // task current execute index int32_t execId; // task current execute index
SSchLevel *level; // level SSchLevel *level; // level
SRWLatch planLock; // task update plan lock SRWLatch planLock; // task update plan lock
SSubplan *plan; // subplan SSubplan *plan; // subplan
char *msg; // operator tree char *msg; // operator tree
int32_t msgLen; // msg length int32_t msgLen; // msg length
int8_t status; // task status int8_t status; // task status
int32_t lastMsgType; // last sent msg type int32_t lastMsgType; // last sent msg type
int64_t timeoutUsec; // task timeout useconds before reschedule int64_t timeoutUsec; // task timeout useconds before reschedule
SQueryNodeAddr succeedAddr; // task executed success node address SQueryNodeAddr succeedAddr; // task executed success node address
int8_t candidateIdx; // current try condidation index int8_t candidateIdx; // current try condidation index
SArray *candidateAddrs; // condidate node addresses, element is SQueryNodeAddr SArray *candidateAddrs; // condidate node addresses, element is SQueryNodeAddr
SHashObj *execNodes; // all tried node for current task, element is SSchNodeInfo SHashObj *execNodes; // all tried node for current task, element is SSchNodeInfo
SSchTaskProfile profile; // task execution profile SSchTaskProfile profile; // task execution profile
int32_t childReady; // child task ready number int32_t childReady; // child task ready number
SArray *children; // the datasource tasks,from which to fetch the result, element is SQueryTask* SArray *children; // the datasource tasks,from which to fetch the result, element is SQueryTask*
SArray *parents; // the data destination tasks, get data from current task, element is SQueryTask* SArray *parents; // the data destination tasks, get data from current task, element is SQueryTask*
void* handle; // task send handle void *handle; // task send handle
bool registerdHb; // registered in hb bool registerdHb; // registered in hb
} SSchTask; } SSchTask;
typedef struct SSchJobAttr { typedef struct SSchJobAttr {
@ -243,46 +240,46 @@ typedef struct SSchJobAttr {
} SSchJobAttr; } SSchJobAttr;
typedef struct { typedef struct {
int32_t op; int32_t op;
SRWLatch lock; SRWLatch lock;
bool syncReq; bool syncReq;
} SSchOpStatus; } SSchOpStatus;
typedef struct SSchJob { typedef struct SSchJob {
int64_t refId; int64_t refId;
uint64_t queryId; uint64_t queryId;
SSchJobAttr attr; SSchJobAttr attr;
int32_t levelNum; int32_t levelNum;
int32_t taskNum; int32_t taskNum;
SRequestConnInfo conn; SRequestConnInfo conn;
SArray *nodeList; // qnode/vnode list, SArray<SQueryNodeLoad> SArray *nodeList; // qnode/vnode list, SArray<SQueryNodeLoad>
SArray *levels; // starting from 0. SArray<SSchLevel> SArray *levels; // starting from 0. SArray<SSchLevel>
SQueryPlan *pDag; SQueryPlan *pDag;
int64_t allocatorRefId; int64_t allocatorRefId;
SArray *dataSrcTasks; // SArray<SQueryTask*> SArray *dataSrcTasks; // SArray<SQueryTask*>
int32_t levelIdx; int32_t levelIdx;
SEpSet dataSrcEps; SEpSet dataSrcEps;
SHashObj *taskList; SHashObj *taskList;
SHashObj *execTasks; // executing and executed tasks, key:taskid, value:SQueryTask* SHashObj *execTasks; // executing and executed tasks, key:taskid, value:SQueryTask*
SHashObj *flowCtrl; // key is ep, element is SSchFlowControl SHashObj *flowCtrl; // key is ep, element is SSchFlowControl
SExplainCtx *explainCtx; SExplainCtx *explainCtx;
int8_t status; int8_t status;
SQueryNodeAddr resNode; SQueryNodeAddr resNode;
tsem_t rspSem; tsem_t rspSem;
SSchOpStatus opStatus; SSchOpStatus opStatus;
schedulerChkKillFp chkKillFp; schedulerChkKillFp chkKillFp;
void* chkKillParam; void *chkKillParam;
SSchTask *fetchTask; SSchTask *fetchTask;
int32_t errCode; int32_t errCode;
SRWLatch resLock; SRWLatch resLock;
SExecResult execRes; SExecResult execRes;
void *fetchRes; //TODO free it or not void *fetchRes; // TODO free it or not
bool fetched; bool fetched;
int32_t resNumOfRows; int32_t resNumOfRows;
SSchResInfo userRes; SSchResInfo userRes;
char *sql; char *sql;
SQueryProfileSummary summary; SQueryProfileSummary summary;
} SSchJob; } SSchJob;
@ -294,226 +291,282 @@ typedef struct SSchTaskCtx {
extern SSchedulerMgmt schMgmt; extern SSchedulerMgmt schMgmt;
#define SCH_TASK_TIMEOUT(_task) ((taosGetTimestampUs() - *(int64_t*)taosArrayGet((_task)->profile.execTime, (_task)->execId)) > (_task)->timeoutUsec) #define SCH_TASK_TIMEOUT(_task) \
((taosGetTimestampUs() - *(int64_t *)taosArrayGet((_task)->profile.execTime, (_task)->execId)) > (_task)->timeoutUsec)
#define SCH_TASK_READY_FOR_LAUNCH(readyNum, task) ((readyNum) >= taosArrayGetSize((task)->children)) #define SCH_TASK_READY_FOR_LAUNCH(readyNum, task) ((readyNum) >= taosArrayGetSize((task)->children))
#define SCH_LOCK_TASK(_task) SCH_LOCK(SCH_WRITE, &(_task)->lock) #define SCH_LOCK_TASK(_task) SCH_LOCK(SCH_WRITE, &(_task)->lock)
#define SCH_UNLOCK_TASK(_task) SCH_UNLOCK(SCH_WRITE, &(_task)->lock) #define SCH_UNLOCK_TASK(_task) SCH_UNLOCK(SCH_WRITE, &(_task)->lock)
#define SCH_TASK_ID(_task) ((_task) ? (_task)->taskId : -1) #define SCH_TASK_ID(_task) ((_task) ? (_task)->taskId : -1)
#define SCH_TASK_EID(_task) ((_task) ? (_task)->execId : -1) #define SCH_TASK_EID(_task) ((_task) ? (_task)->execId : -1)
#define SCH_IS_DATA_BIND_QRY_TASK(task) ((task)->plan->subplanType == SUBPLAN_TYPE_SCAN) #define SCH_IS_DATA_BIND_QRY_TASK(task) ((task)->plan->subplanType == SUBPLAN_TYPE_SCAN)
#define SCH_IS_DATA_BIND_TASK(task) (((task)->plan->subplanType == SUBPLAN_TYPE_SCAN) || ((task)->plan->subplanType == SUBPLAN_TYPE_MODIFY)) #define SCH_IS_DATA_BIND_TASK(task) \
(((task)->plan->subplanType == SUBPLAN_TYPE_SCAN) || ((task)->plan->subplanType == SUBPLAN_TYPE_MODIFY))
#define SCH_IS_LEAF_TASK(_job, _task) (((_task)->level->level + 1) == (_job)->levelNum) #define SCH_IS_LEAF_TASK(_job, _task) (((_task)->level->level + 1) == (_job)->levelNum)
#define SCH_IS_DATA_MERGE_TASK(task) (!SCH_IS_DATA_BIND_TASK(task)) #define SCH_IS_DATA_MERGE_TASK(task) (!SCH_IS_DATA_BIND_TASK(task))
#define SCH_IS_LOCAL_EXEC_TASK(_job, _task) ((_job)->attr.localExec && SCH_IS_QUERY_JOB(_job) && (!SCH_IS_INSERT_JOB(_job)) && (!SCH_IS_DATA_BIND_QRY_TASK(_task))) #define SCH_IS_LOCAL_EXEC_TASK(_job, _task) \
((_job)->attr.localExec && SCH_IS_QUERY_JOB(_job) && (!SCH_IS_INSERT_JOB(_job)) && \
(!SCH_IS_DATA_BIND_QRY_TASK(_task)))
#define SCH_SET_TASK_STATUS(task, st) atomic_store_8(&(task)->status, st) #define SCH_SET_TASK_STATUS(task, st) atomic_store_8(&(task)->status, st)
#define SCH_GET_TASK_STATUS(task) atomic_load_8(&(task)->status) #define SCH_GET_TASK_STATUS(task) atomic_load_8(&(task)->status)
#define SCH_GET_TASK_STATUS_STR(task) jobTaskStatusStr(SCH_GET_TASK_STATUS(task)) #define SCH_GET_TASK_STATUS_STR(task) jobTaskStatusStr(SCH_GET_TASK_STATUS(task))
#define SCH_GET_TASK_HANDLE(_task) ((_task) ? (_task)->handle : NULL) #define SCH_GET_TASK_HANDLE(_task) ((_task) ? (_task)->handle : NULL)
#define SCH_SET_TASK_HANDLE(_task, _handle) ((_task)->handle = (_handle)) #define SCH_SET_TASK_HANDLE(_task, _handle) ((_task)->handle = (_handle))
#define SCH_SET_JOB_STATUS(job, st) atomic_store_8(&(job)->status, st) #define SCH_SET_JOB_STATUS(job, st) atomic_store_8(&(job)->status, st)
#define SCH_GET_JOB_STATUS(job) atomic_load_8(&(job)->status) #define SCH_GET_JOB_STATUS(job) atomic_load_8(&(job)->status)
#define SCH_GET_JOB_STATUS_STR(job) jobTaskStatusStr(SCH_GET_JOB_STATUS(job)) #define SCH_GET_JOB_STATUS_STR(job) jobTaskStatusStr(SCH_GET_JOB_STATUS(job))
#define SCH_JOB_IN_SYNC_OP(job) ((job)->opStatus.op && (job)->opStatus.syncReq) #define SCH_JOB_IN_SYNC_OP(job) ((job)->opStatus.op && (job)->opStatus.syncReq)
#define SCH_JOB_IN_ASYNC_EXEC_OP(job) ((SCH_OP_EXEC == atomic_val_compare_exchange_32(&(job)->opStatus.op, SCH_OP_EXEC, SCH_OP_NULL)) && (!(job)->opStatus.syncReq)) #define SCH_JOB_IN_ASYNC_EXEC_OP(job) \
#define SCH_JOB_IN_ASYNC_FETCH_OP(job) ((SCH_OP_FETCH == atomic_val_compare_exchange_32(&(job)->opStatus.op, SCH_OP_FETCH, SCH_OP_NULL)) && (!(job)->opStatus.syncReq)) ((SCH_OP_EXEC == atomic_val_compare_exchange_32(&(job)->opStatus.op, SCH_OP_EXEC, SCH_OP_NULL)) && \
(!(job)->opStatus.syncReq))
#define SCH_JOB_IN_ASYNC_FETCH_OP(job) \
((SCH_OP_FETCH == atomic_val_compare_exchange_32(&(job)->opStatus.op, SCH_OP_FETCH, SCH_OP_NULL)) && \
(!(job)->opStatus.syncReq))
#define SCH_SET_JOB_NEED_FLOW_CTRL(_job) (_job)->attr.needFlowCtrl = true #define SCH_SET_JOB_NEED_FLOW_CTRL(_job) (_job)->attr.needFlowCtrl = true
#define SCH_JOB_NEED_FLOW_CTRL(_job) ((_job)->attr.needFlowCtrl) #define SCH_JOB_NEED_FLOW_CTRL(_job) ((_job)->attr.needFlowCtrl)
#define SCH_TASK_NEED_FLOW_CTRL(_job, _task) (SCH_IS_DATA_BIND_QRY_TASK(_task) && SCH_JOB_NEED_FLOW_CTRL(_job) && SCH_IS_LEVEL_UNFINISHED((_task)->level)) #define SCH_TASK_NEED_FLOW_CTRL(_job, _task) \
#define SCH_FETCH_TYPE(_pSrcTask) (SCH_IS_DATA_BIND_QRY_TASK(_pSrcTask) ? TDMT_SCH_FETCH : TDMT_SCH_MERGE_FETCH) (SCH_IS_DATA_BIND_QRY_TASK(_task) && SCH_JOB_NEED_FLOW_CTRL(_job) && SCH_IS_LEVEL_UNFINISHED((_task)->level))
#define SCH_FETCH_TYPE(_pSrcTask) (SCH_IS_DATA_BIND_QRY_TASK(_pSrcTask) ? TDMT_SCH_FETCH : TDMT_SCH_MERGE_FETCH)
#define SCH_TASK_NEED_FETCH(_task) ((_task)->plan->subplanType != SUBPLAN_TYPE_MODIFY) #define SCH_TASK_NEED_FETCH(_task) ((_task)->plan->subplanType != SUBPLAN_TYPE_MODIFY)
#define SCH_SET_JOB_TYPE(_job, type) do { if ((type) != SUBPLAN_TYPE_MODIFY) { (_job)->attr.queryJob = true; } else { (_job)->attr.insertJob = true; } } while (0) #define SCH_SET_JOB_TYPE(_job, type) \
#define SCH_IS_QUERY_JOB(_job) ((_job)->attr.queryJob) do { \
#define SCH_IS_INSERT_JOB(_job) ((_job)->attr.insertJob) if ((type) != SUBPLAN_TYPE_MODIFY) { \
(_job)->attr.queryJob = true; \
} else { \
(_job)->attr.insertJob = true; \
} \
} while (0)
#define SCH_IS_QUERY_JOB(_job) ((_job)->attr.queryJob)
#define SCH_IS_INSERT_JOB(_job) ((_job)->attr.insertJob)
#define SCH_JOB_NEED_FETCH(_job) ((_job)->attr.needFetch) #define SCH_JOB_NEED_FETCH(_job) ((_job)->attr.needFetch)
#define SCH_JOB_NEED_WAIT(_job) (!SCH_IS_QUERY_JOB(_job)) #define SCH_JOB_NEED_WAIT(_job) (!SCH_IS_QUERY_JOB(_job))
#define SCH_JOB_NEED_DROP(_job) (SCH_IS_QUERY_JOB(_job)) #define SCH_JOB_NEED_DROP(_job) (SCH_IS_QUERY_JOB(_job))
#define SCH_IS_EXPLAIN_JOB(_job) (EXPLAIN_MODE_ANALYZE == (_job)->attr.explainMode) #define SCH_IS_EXPLAIN_JOB(_job) (EXPLAIN_MODE_ANALYZE == (_job)->attr.explainMode)
#define SCH_NETWORK_ERR(_code) ((_code) == TSDB_CODE_RPC_BROKEN_LINK || (_code) == TSDB_CODE_RPC_NETWORK_UNAVAIL) #define SCH_NETWORK_ERR(_code) ((_code) == TSDB_CODE_RPC_BROKEN_LINK || (_code) == TSDB_CODE_RPC_NETWORK_UNAVAIL)
#define SCH_MERGE_TASK_NETWORK_ERR(_task, _code, _len) (SCH_NETWORK_ERR(_code) && (((_len) > 0) || (!SCH_IS_DATA_BIND_TASK(_task)))) #define SCH_MERGE_TASK_NETWORK_ERR(_task, _code, _len) \
#define SCH_REDIRECT_MSGTYPE(_msgType) ((_msgType) == TDMT_SCH_LINK_BROKEN || (_msgType) == TDMT_SCH_QUERY || (_msgType) == TDMT_SCH_MERGE_QUERY || (_msgType) == TDMT_SCH_FETCH || (_msgType) == TDMT_SCH_MERGE_FETCH) (SCH_NETWORK_ERR(_code) && (((_len) > 0) || (!SCH_IS_DATA_BIND_TASK(_task))))
#define SCH_TASK_NEED_REDIRECT(_task, _msgType, _code, _rspLen) (SCH_REDIRECT_MSGTYPE(_msgType) && (NEED_SCHEDULER_REDIRECT_ERROR(_code) || SCH_MERGE_TASK_NETWORK_ERR((_task), (_code), (_rspLen)))) #define SCH_REDIRECT_MSGTYPE(_msgType) \
#define SCH_NEED_RETRY(_msgType, _code) ((SCH_NETWORK_ERR(_code) && SCH_REDIRECT_MSGTYPE(_msgType)) || (_code) == TSDB_CODE_SCH_TIMEOUT_ERROR) ((_msgType) == TDMT_SCH_LINK_BROKEN || (_msgType) == TDMT_SCH_QUERY || (_msgType) == TDMT_SCH_MERGE_QUERY || \
(_msgType) == TDMT_SCH_FETCH || (_msgType) == TDMT_SCH_MERGE_FETCH)
#define SCH_TASK_NEED_REDIRECT(_task, _msgType, _code, _rspLen) \
(SCH_REDIRECT_MSGTYPE(_msgType) && \
(NEED_SCHEDULER_REDIRECT_ERROR(_code) || SCH_MERGE_TASK_NETWORK_ERR((_task), (_code), (_rspLen))))
#define SCH_NEED_RETRY(_msgType, _code) \
((SCH_NETWORK_ERR(_code) && SCH_REDIRECT_MSGTYPE(_msgType)) || (_code) == TSDB_CODE_SCH_TIMEOUT_ERROR)
#define SCH_IS_LEVEL_UNFINISHED(_level) ((_level)->taskLaunchedNum < (_level)->taskNum) #define SCH_IS_LEVEL_UNFINISHED(_level) ((_level)->taskLaunchedNum < (_level)->taskNum)
#define SCH_GET_CUR_EP(_addr) (&(_addr)->epSet.eps[(_addr)->epSet.inUse]) #define SCH_GET_CUR_EP(_addr) (&(_addr)->epSet.eps[(_addr)->epSet.inUse])
#define SCH_SWITCH_EPSET(_addr) ((_addr)->epSet.inUse = ((_addr)->epSet.inUse + 1) % (_addr)->epSet.numOfEps) #define SCH_SWITCH_EPSET(_addr) ((_addr)->epSet.inUse = ((_addr)->epSet.inUse + 1) % (_addr)->epSet.numOfEps)
#define SCH_TASK_NUM_OF_EPS(_addr) ((_addr)->epSet.numOfEps) #define SCH_TASK_NUM_OF_EPS(_addr) ((_addr)->epSet.numOfEps)
#define SCH_LOG_TASK_START_TS(_task) \ #define SCH_LOG_TASK_START_TS(_task) \
do { \ do { \
int64_t us = taosGetTimestampUs(); \ int64_t us = taosGetTimestampUs(); \
taosArrayPush((_task)->profile.execTime, &us); \ taosArrayPush((_task)->profile.execTime, &us); \
if (0 == (_task)->execId) { \ if (0 == (_task)->execId) { \
(_task)->profile.startTs = us; \ (_task)->profile.startTs = us; \
} \ } \
} while (0) } while (0)
#define SCH_LOG_TASK_WAIT_TS(_task) \ #define SCH_LOG_TASK_WAIT_TS(_task) \
do { \ do { \
int64_t us = taosGetTimestampUs(); \ int64_t us = taosGetTimestampUs(); \
(_task)->profile.waitTime += us - *(int64_t*)taosArrayGet((_task)->profile.execTime, (_task)->execId); \ (_task)->profile.waitTime += us - *(int64_t *)taosArrayGet((_task)->profile.execTime, (_task)->execId); \
} while (0) } while (0)
#define SCH_LOG_TASK_END_TS(_task) \
#define SCH_LOG_TASK_END_TS(_task) \ do { \
do { \ int64_t us = taosGetTimestampUs(); \
int64_t us = taosGetTimestampUs(); \ int32_t idx = (_task)->execId % (_task)->maxExecTimes; \
int32_t idx = (_task)->execId % (_task)->maxExecTimes; \
int64_t *startts = taosArrayGet((_task)->profile.execTime, (_task)->execId); \ int64_t *startts = taosArrayGet((_task)->profile.execTime, (_task)->execId); \
*startts = us - *startts; \ *startts = us - *startts; \
(_task)->profile.endTs = us; \ (_task)->profile.endTs = us; \
} while (0) } while (0)
#define SCH_JOB_ELOG(param, ...) qError("QID:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__) #define SCH_JOB_ELOG(param, ...) qError("QID:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__)
#define SCH_JOB_DLOG(param, ...) qDebug("QID:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__) #define SCH_JOB_DLOG(param, ...) qDebug("QID:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__)
#define SCH_TASK_ELOG(param, ...) \ #define SCH_TASK_ELOG(param, ...) \
qError("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask),__VA_ARGS__) qError("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \
#define SCH_TASK_DLOG(param, ...) \ __VA_ARGS__)
qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask),__VA_ARGS__) #define SCH_TASK_DLOG(param, ...) \
#define SCH_TASK_TLOG(param, ...) \ qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \
qTrace("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask),__VA_ARGS__) __VA_ARGS__)
#define SCH_TASK_DLOGL(param, ...) \ #define SCH_TASK_TLOG(param, ...) \
qDebugL("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask),__VA_ARGS__) qTrace("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \
#define SCH_TASK_WLOG(param, ...) \ __VA_ARGS__)
qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask),__VA_ARGS__) #define SCH_TASK_DLOGL(param, ...) \
qDebugL("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \
__VA_ARGS__)
#define SCH_TASK_WLOG(param, ...) \
qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \
__VA_ARGS__)
#define SCH_SET_ERRNO(_err) do { if (TSDB_CODE_SCH_IGNORE_ERROR != (_err)) { terrno = (_err); } } while (0) #define SCH_SET_ERRNO(_err) \
#define SCH_ERR_RET(c) do { int32_t _code = c; if (_code != TSDB_CODE_SUCCESS) { SCH_SET_ERRNO(_code); return _code; } } while (0) do { \
#define SCH_RET(c) do { int32_t _code = c; if (_code != TSDB_CODE_SUCCESS) { SCH_SET_ERRNO(_code); } return _code; } while (0) if (TSDB_CODE_SCH_IGNORE_ERROR != (_err)) { \
#define SCH_ERR_JRET(c) do { code = c; if (code != TSDB_CODE_SUCCESS) { SCH_SET_ERRNO(code); goto _return; } } while (0) terrno = (_err); \
} \
} while (0)
#define SCH_ERR_RET(c) \
do { \
int32_t _code = c; \
if (_code != TSDB_CODE_SUCCESS) { \
SCH_SET_ERRNO(_code); \
return _code; \
} \
} while (0)
#define SCH_RET(c) \
do { \
int32_t _code = c; \
if (_code != TSDB_CODE_SUCCESS) { \
SCH_SET_ERRNO(_code); \
} \
return _code; \
} while (0)
#define SCH_ERR_JRET(c) \
do { \
code = c; \
if (code != TSDB_CODE_SUCCESS) { \
SCH_SET_ERRNO(code); \
goto _return; \
} \
} while (0)
#define SCH_LOCK_DEBUG(...) do { if (gSCHDebug.lockEnable) { qDebug(__VA_ARGS__); } } while (0) #define SCH_LOCK_DEBUG(...) \
do { \
if (gSCHDebug.lockEnable) { \
qDebug(__VA_ARGS__); \
} \
} while (0)
#define TD_RWLATCH_WRITE_FLAG_COPY 0x40000000 #define TD_RWLATCH_WRITE_FLAG_COPY 0x40000000
#define SCH_LOCK(type, _lock) do { \ #define SCH_LOCK(type, _lock) \
if (SCH_READ == (type)) { \ do { \
assert(atomic_load_32(_lock) >= 0); \ if (SCH_READ == (type)) { \
SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ assert(atomic_load_32(_lock) >= 0); \
taosRLockLatch(_lock); \ SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosRLockLatch(_lock); \
assert(atomic_load_32(_lock) > 0); \ SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
} else { \ assert(atomic_load_32(_lock) > 0); \
assert(atomic_load_32(_lock) >= 0); \ } else { \
SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ assert(atomic_load_32(_lock) >= 0); \
taosWLockLatch(_lock); \ SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosWLockLatch(_lock); \
assert(atomic_load_32(_lock) == TD_RWLATCH_WRITE_FLAG_COPY); \ SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
} \ assert(atomic_load_32(_lock) == TD_RWLATCH_WRITE_FLAG_COPY); \
} while (0) } \
} while (0)
#define SCH_UNLOCK(type, _lock) do { \ #define SCH_UNLOCK(type, _lock) \
if (SCH_READ == (type)) { \ do { \
assert(atomic_load_32((_lock)) > 0); \ if (SCH_READ == (type)) { \
SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ assert(atomic_load_32((_lock)) > 0); \
taosRUnLockLatch(_lock); \ SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosRUnLockLatch(_lock); \
assert(atomic_load_32((_lock)) >= 0); \ SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
} else { \ assert(atomic_load_32((_lock)) >= 0); \
assert(atomic_load_32((_lock)) & TD_RWLATCH_WRITE_FLAG_COPY); \ } else { \
SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ assert(atomic_load_32((_lock)) & TD_RWLATCH_WRITE_FLAG_COPY); \
taosWUnLockLatch(_lock); \ SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosWUnLockLatch(_lock); \
assert(atomic_load_32((_lock)) >= 0); \ SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
} \ assert(atomic_load_32((_lock)) >= 0); \
} while (0) } \
} while (0)
void schDeregisterTaskHb(SSchJob *pJob, SSchTask *pTask);
void schDeregisterTaskHb(SSchJob *pJob, SSchTask *pTask); void schCleanClusterHb(void *pTrans);
void schCleanClusterHb(void* pTrans); int32_t schLaunchTask(SSchJob *job, SSchTask *task);
int32_t schLaunchTask(SSchJob *job, SSchTask *task); int32_t schBuildAndSendMsg(SSchJob *job, SSchTask *task, SQueryNodeAddr *addr, int32_t msgType);
int32_t schBuildAndSendMsg(SSchJob *job, SSchTask *task, SQueryNodeAddr *addr, int32_t msgType);
SSchJob *schAcquireJob(int64_t refId); SSchJob *schAcquireJob(int64_t refId);
int32_t schReleaseJob(int64_t refId); int32_t schReleaseJob(int64_t refId);
void schFreeFlowCtrl(SSchJob *pJob); void schFreeFlowCtrl(SSchJob *pJob);
int32_t schChkJobNeedFlowCtrl(SSchJob *pJob, SSchLevel *pLevel); int32_t schChkJobNeedFlowCtrl(SSchJob *pJob, SSchLevel *pLevel);
int32_t schDecTaskFlowQuota(SSchJob *pJob, SSchTask *pTask); int32_t schDecTaskFlowQuota(SSchJob *pJob, SSchTask *pTask);
int32_t schCheckIncTaskFlowQuota(SSchJob *pJob, SSchTask *pTask, bool *enough); int32_t schCheckIncTaskFlowQuota(SSchJob *pJob, SSchTask *pTask, bool *enough);
int32_t schLaunchTasksInFlowCtrlList(SSchJob *pJob, SSchTask *pTask); int32_t schLaunchTasksInFlowCtrlList(SSchJob *pJob, SSchTask *pTask);
int32_t schAsyncLaunchTaskImpl(SSchJob *pJob, SSchTask *pTask); int32_t schAsyncLaunchTaskImpl(SSchJob *pJob, SSchTask *pTask);
int32_t schLaunchFetchTask(SSchJob *pJob); int32_t schLaunchFetchTask(SSchJob *pJob);
int32_t schProcessOnTaskFailure(SSchJob *pJob, SSchTask *pTask, int32_t errCode); int32_t schProcessOnTaskFailure(SSchJob *pJob, SSchTask *pTask, int32_t errCode);
int32_t schBuildAndSendHbMsg(SQueryNodeEpId *nodeEpId, SArray* taskAction); int32_t schBuildAndSendHbMsg(SQueryNodeEpId *nodeEpId, SArray *taskAction);
int32_t schCloneSMsgSendInfo(void *src, void **dst); int32_t schCloneSMsgSendInfo(void *src, void **dst);
int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob); int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob);
void schFreeJobImpl(void *job); void schFreeJobImpl(void *job);
int32_t schMakeHbRpcCtx(SSchJob *pJob, SSchTask *pTask, SRpcCtx *pCtx); int32_t schMakeHbRpcCtx(SSchJob *pJob, SSchTask *pTask, SRpcCtx *pCtx);
int32_t schEnsureHbConnection(SSchJob *pJob, SSchTask *pTask); int32_t schEnsureHbConnection(SSchJob *pJob, SSchTask *pTask);
int32_t schUpdateHbConnection(SQueryNodeEpId *epId, SSchTrans *trans); int32_t schUpdateHbConnection(SQueryNodeEpId *epId, SSchTrans *trans);
int32_t schHandleHbCallback(void *param, SDataBuf *pMsg, int32_t code); int32_t schHandleHbCallback(void *param, SDataBuf *pMsg, int32_t code);
void schFreeRpcCtx(SRpcCtx *pCtx); void schFreeRpcCtx(SRpcCtx *pCtx);
int32_t schGetCallbackFp(int32_t msgType, __async_send_cb_fn_t *fp); int32_t schGetCallbackFp(int32_t msgType, __async_send_cb_fn_t *fp);
bool schJobNeedToStop(SSchJob *pJob, int8_t *pStatus); bool schJobNeedToStop(SSchJob *pJob, int8_t *pStatus);
int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask); int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask);
int32_t schSaveJobExecRes(SSchJob *pJob, SQueryTableRsp *rsp); int32_t schSaveJobExecRes(SSchJob *pJob, SQueryTableRsp *rsp);
int32_t schProcessOnExplainDone(SSchJob *pJob, SSchTask *pTask, SRetrieveTableRsp *pRsp); int32_t schProcessOnExplainDone(SSchJob *pJob, SSchTask *pTask, SRetrieveTableRsp *pRsp);
void schProcessOnDataFetched(SSchJob *job); void schProcessOnDataFetched(SSchJob *job);
int32_t schGetTaskInJob(SSchJob *pJob, uint64_t taskId, SSchTask **pTask); int32_t schGetTaskInJob(SSchJob *pJob, uint64_t taskId, SSchTask **pTask);
void schFreeRpcCtxVal(const void *arg); void schFreeRpcCtxVal(const void *arg);
int32_t schMakeBrokenLinkVal(SSchJob *pJob, SSchTask *pTask, SRpcBrokenlinkVal *brokenVal, bool isHb); int32_t schMakeBrokenLinkVal(SSchJob *pJob, SSchTask *pTask, SRpcBrokenlinkVal *brokenVal, bool isHb);
int32_t schAppendTaskExecNode(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr, int32_t execId); int32_t schAppendTaskExecNode(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr, int32_t execId);
int32_t schExecStaticExplainJob(SSchedulerReq *pReq, int64_t *job, bool sync); int32_t schExecStaticExplainJob(SSchedulerReq *pReq, int64_t *job, bool sync);
int32_t schUpdateJobStatus(SSchJob *pJob, int8_t newStatus); int32_t schUpdateJobStatus(SSchJob *pJob, int8_t newStatus);
int32_t schCancelJob(SSchJob *pJob); int32_t schCancelJob(SSchJob *pJob);
int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode); int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode);
uint64_t schGenTaskId(void); uint64_t schGenTaskId(void);
void schCloseJobRef(void); void schCloseJobRef(void);
int32_t schAsyncExecJob(SSchedulerReq *pReq, int64_t *pJob); int32_t schAsyncExecJob(SSchedulerReq *pReq, int64_t *pJob);
int32_t schJobFetchRows(SSchJob *pJob); int32_t schJobFetchRows(SSchJob *pJob);
int32_t schJobFetchRowsA(SSchJob *pJob); int32_t schJobFetchRowsA(SSchJob *pJob);
int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, void *handle, int32_t execId); int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, void *handle, int32_t execId);
int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId* pEpId, SArray* pStatusList); int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId *pEpId, SArray *pStatusList);
char* schGetOpStr(SCH_OP_TYPE type); char *schGetOpStr(SCH_OP_TYPE type);
int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync); int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync);
int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq); int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq);
int32_t schExecJob(SSchJob *pJob, SSchedulerReq *pReq); int32_t schExecJob(SSchJob *pJob, SSchedulerReq *pReq);
int32_t schDumpJobExecRes(SSchJob* pJob, SExecResult* pRes); int32_t schDumpJobExecRes(SSchJob *pJob, SExecResult *pRes);
int32_t schUpdateTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask, SEpSet* pEpSet); int32_t schUpdateTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask, SEpSet *pEpSet);
int32_t schHandleRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf* pData, int32_t rspCode); int32_t schHandleRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf *pData, int32_t rspCode);
void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int32_t errCode); void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq *pReq, int32_t errCode);
int32_t schProcessOnOpBegin(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq); int32_t schProcessOnOpBegin(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq *pReq);
void schProcessOnCbEnd(SSchJob *pJob, SSchTask *pTask, int32_t errCode); void schProcessOnCbEnd(SSchJob *pJob, SSchTask *pTask, int32_t errCode);
int32_t schProcessOnCbBegin(SSchJob** job, SSchTask** task, uint64_t qId, int64_t rId, uint64_t tId); int32_t schProcessOnCbBegin(SSchJob **job, SSchTask **task, uint64_t qId, int64_t rId, uint64_t tId);
void schDropTaskOnExecNode(SSchJob *pJob, SSchTask *pTask); void schDropTaskOnExecNode(SSchJob *pJob, SSchTask *pTask);
bool schJobDone(SSchJob *pJob); bool schJobDone(SSchJob *pJob);
int32_t schRemoveTaskFromExecList(SSchJob *pJob, SSchTask *pTask); int32_t schRemoveTaskFromExecList(SSchJob *pJob, SSchTask *pTask);
int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask); int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask);
int32_t schSwitchJobStatus(SSchJob* pJob, int32_t status, void* param); int32_t schSwitchJobStatus(SSchJob *pJob, int32_t status, void *param);
int32_t schHandleOpBeginEvent(int64_t jobId, SSchJob** job, SCH_OP_TYPE type, SSchedulerReq* pReq); int32_t schHandleOpBeginEvent(int64_t jobId, SSchJob **job, SCH_OP_TYPE type, SSchedulerReq *pReq);
int32_t schHandleOpEndEvent(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int32_t errCode); int32_t schHandleOpEndEvent(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq *pReq, int32_t errCode);
int32_t schHandleTaskRetry(SSchJob *pJob, SSchTask *pTask); int32_t schHandleTaskRetry(SSchJob *pJob, SSchTask *pTask);
void schUpdateJobErrCode(SSchJob *pJob, int32_t errCode); void schUpdateJobErrCode(SSchJob *pJob, int32_t errCode);
int32_t schTaskCheckSetRetry(SSchJob *pJob, SSchTask *pTask, int32_t errCode, bool *needRetry); int32_t schTaskCheckSetRetry(SSchJob *pJob, SSchTask *pTask, int32_t errCode, bool *needRetry);
int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode); int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode);
int32_t schProcessOnJobPartialSuccess(SSchJob *pJob); int32_t schProcessOnJobPartialSuccess(SSchJob *pJob);
void schFreeTask(SSchJob *pJob, SSchTask *pTask); void schFreeTask(SSchJob *pJob, SSchTask *pTask);
void schDropTaskInHashList(SSchJob *pJob, SHashObj *list); void schDropTaskInHashList(SSchJob *pJob, SHashObj *list);
int32_t schLaunchLevelTasks(SSchJob *pJob, SSchLevel *level); int32_t schLaunchLevelTasks(SSchJob *pJob, SSchLevel *level);
int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTask); int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTask);
int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel); int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel);
int32_t schSwitchTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask); int32_t schSwitchTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask);
void schDirectPostJobRes(SSchedulerReq* pReq, int32_t errCode); void schDirectPostJobRes(SSchedulerReq *pReq, int32_t errCode);
int32_t schHandleJobFailure(SSchJob *pJob, int32_t errCode); int32_t schHandleJobFailure(SSchJob *pJob, int32_t errCode);
int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode); int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode);
bool schChkCurrentOp(SSchJob *pJob, int32_t op, int8_t sync); bool schChkCurrentOp(SSchJob *pJob, int32_t op, int8_t sync);
int32_t schProcessFetchRsp(SSchJob *pJob, SSchTask *pTask, char *msg, int32_t rspCode); int32_t schProcessFetchRsp(SSchJob *pJob, SSchTask *pTask, char *msg, int32_t rspCode);
int32_t schProcessExplainRsp(SSchJob *pJob, SSchTask *pTask, SExplainRsp *rsp); int32_t schProcessExplainRsp(SSchJob *pJob, SSchTask *pTask, SExplainRsp *rsp);
extern SSchDebug gSCHDebug; extern SSchDebug gSCHDebug;
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif

6
source/libs/scheduler/src/schDbg.c
View File

@ -16,19 +16,17 @@
#include "query.h" #include "query.h"
#include "schInt.h" #include "schInt.h"
tsem_t schdRspSem; tsem_t schdRspSem;
SSchDebug gSCHDebug = {0}; SSchDebug gSCHDebug = {0};
void schdExecCallback(SExecResult* pResult, void* param, int32_t code) { void schdExecCallback(SExecResult* pResult, void* param, int32_t code) {
if (code) { if (code) {
pResult->code = code; pResult->code = code;
} }
*(SExecResult*)param = *pResult; *(SExecResult*)param = *pResult;
taosMemoryFree(pResult); taosMemoryFree(pResult);
tsem_post(&schdRspSem); tsem_post(&schdRspSem);
} }

121
source/libs/scheduler/src/schFlowCtrl.c
View File

@ -13,10 +13,10 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>. * along with this program. If not, see <http://www.gnu.org/licenses/>.
*/ */
#include "catalog.h"
#include "query.h"
#include "schInt.h" #include "schInt.h"
#include "tmsg.h" #include "tmsg.h"
#include "query.h"
#include "catalog.h"
#include "tref.h" #include "tref.h"
void schFreeFlowCtrl(SSchJob *pJob) { void schFreeFlowCtrl(SSchJob *pJob) {
@ -25,14 +25,14 @@ void schFreeFlowCtrl(SSchJob *pJob) {
} }
SSchFlowControl *ctrl = NULL; SSchFlowControl *ctrl = NULL;
void *pIter = taosHashIterate(pJob->flowCtrl, NULL); void *pIter = taosHashIterate(pJob->flowCtrl, NULL);
while (pIter) { while (pIter) {
ctrl = (SSchFlowControl *)pIter; ctrl = (SSchFlowControl *)pIter;
if (ctrl->taskList) { if (ctrl->taskList) {
taosArrayDestroy(ctrl->taskList); taosArrayDestroy(ctrl->taskList);
} }
pIter = taosHashIterate(pJob->flowCtrl, pIter); pIter = taosHashIterate(pJob->flowCtrl, pIter);
} }
@ -59,7 +59,8 @@ int32_t schChkJobNeedFlowCtrl(SSchJob *pJob, SSchLevel *pLevel) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
pJob->flowCtrl = taosHashInit(pJob->taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK); pJob->flowCtrl =
taosHashInit(pJob->taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK);
if (NULL == pJob->flowCtrl) { if (NULL == pJob->flowCtrl) {
SCH_JOB_ELOG("taosHashInit %d flowCtrl failed", pJob->taskNum); SCH_JOB_ELOG("taosHashInit %d flowCtrl failed", pJob->taskNum);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
@ -73,17 +74,17 @@ int32_t schChkJobNeedFlowCtrl(SSchJob *pJob, SSchLevel *pLevel) {
} }
int32_t schDecTaskFlowQuota(SSchJob *pJob, SSchTask *pTask) { int32_t schDecTaskFlowQuota(SSchJob *pJob, SSchTask *pTask) {
SSchLevel *pLevel = pTask->level; SSchLevel *pLevel = pTask->level;
SSchFlowControl *ctrl = NULL; SSchFlowControl *ctrl = NULL;
int32_t code = 0; int32_t code = 0;
SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode); SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode);
ctrl = (SSchFlowControl *)taosHashGet(pJob->flowCtrl, ep, sizeof(SEp)); ctrl = (SSchFlowControl *)taosHashGet(pJob->flowCtrl, ep, sizeof(SEp));
if (NULL == ctrl) { if (NULL == ctrl) {
SCH_TASK_ELOG("taosHashGet node from flowCtrl failed, fqdn:%s, port:%d", ep->fqdn, ep->port); SCH_TASK_ELOG("taosHashGet node from flowCtrl failed, fqdn:%s, port:%d", ep->fqdn, ep->port);
SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR);
} }
SCH_LOCK(SCH_WRITE, &ctrl->lock); SCH_LOCK(SCH_WRITE, &ctrl->lock);
if (ctrl->execTaskNum <= 0) { if (ctrl->execTaskNum <= 0) {
SCH_TASK_ELOG("taosHashGet node from flowCtrl failed, fqdn:%s, port:%d", ep->fqdn, ep->port); SCH_TASK_ELOG("taosHashGet node from flowCtrl failed, fqdn:%s, port:%d", ep->fqdn, ep->port);
@ -93,8 +94,8 @@ int32_t schDecTaskFlowQuota(SSchJob *pJob, SSchTask *pTask) {
--ctrl->execTaskNum; --ctrl->execTaskNum;
ctrl->tableNumSum -= pTask->plan->execNodeStat.tableNum; ctrl->tableNumSum -= pTask->plan->execNodeStat.tableNum;
SCH_TASK_DLOG("task quota removed, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", SCH_TASK_DLOG("task quota removed, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", ep->fqdn,
ep->fqdn, ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum); ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum);
_return: _return:
@ -104,11 +105,11 @@ _return:
} }
int32_t schCheckIncTaskFlowQuota(SSchJob *pJob, SSchTask *pTask, bool *enough) { int32_t schCheckIncTaskFlowQuota(SSchJob *pJob, SSchTask *pTask, bool *enough) {
SSchLevel *pLevel = pTask->level; SSchLevel *pLevel = pTask->level;
int32_t code = 0; int32_t code = 0;
SSchFlowControl *ctrl = NULL; SSchFlowControl *ctrl = NULL;
SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode); SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode);
do { do {
ctrl = (SSchFlowControl *)taosHashGet(pJob->flowCtrl, ep, sizeof(SEp)); ctrl = (SSchFlowControl *)taosHashGet(pJob->flowCtrl, ep, sizeof(SEp));
if (NULL == ctrl) { if (NULL == ctrl) {
@ -119,34 +120,34 @@ int32_t schCheckIncTaskFlowQuota(SSchJob *pJob, SSchTask *pTask, bool *enough) {
if (HASH_NODE_EXIST(code)) { if (HASH_NODE_EXIST(code)) {
continue; continue;
} }
SCH_TASK_ELOG("taosHashPut flowCtrl failed, size:%d", (int32_t)sizeof(nctrl)); SCH_TASK_ELOG("taosHashPut flowCtrl failed, size:%d", (int32_t)sizeof(nctrl));
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
} }
SCH_TASK_DLOG("task quota added, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", SCH_TASK_DLOG("task quota added, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", ep->fqdn,
ep->fqdn, ep->port, pTask->plan->execNodeStat.tableNum, nctrl.tableNumSum, nctrl.execTaskNum); ep->port, pTask->plan->execNodeStat.tableNum, nctrl.tableNumSum, nctrl.execTaskNum);
*enough = true; *enough = true;
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
SCH_LOCK(SCH_WRITE, &ctrl->lock); SCH_LOCK(SCH_WRITE, &ctrl->lock);
if (0 == ctrl->execTaskNum) { if (0 == ctrl->execTaskNum) {
ctrl->tableNumSum = pTask->plan->execNodeStat.tableNum; ctrl->tableNumSum = pTask->plan->execNodeStat.tableNum;
++ctrl->execTaskNum; ++ctrl->execTaskNum;
*enough = true; *enough = true;
break; break;
} }
int32_t sum = pTask->plan->execNodeStat.tableNum + ctrl->tableNumSum; int32_t sum = pTask->plan->execNodeStat.tableNum + ctrl->tableNumSum;
if (sum <= schMgmt.cfg.maxNodeTableNum) { if (sum <= schMgmt.cfg.maxNodeTableNum) {
ctrl->tableNumSum = sum; ctrl->tableNumSum = sum;
++ctrl->execTaskNum; ++ctrl->execTaskNum;
*enough = true; *enough = true;
break; break;
} }
@ -166,24 +167,25 @@ int32_t schCheckIncTaskFlowQuota(SSchJob *pJob, SSchTask *pTask, bool *enough) {
*enough = false; *enough = false;
ctrl->sorted = false; ctrl->sorted = false;
break; break;
} while (true); } while (true);
_return: _return:
SCH_TASK_DLOG("task quota %s added, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", SCH_TASK_DLOG("task quota %s added, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d",
((*enough)?"":"NOT"), ep->fqdn, ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum); ((*enough) ? "" : "NOT"), ep->fqdn, ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum,
ctrl->execTaskNum);
SCH_UNLOCK(SCH_WRITE, &ctrl->lock); SCH_UNLOCK(SCH_WRITE, &ctrl->lock);
SCH_RET(code); SCH_RET(code);
} }
int32_t schTaskTableNumCompare(const void* key1, const void* key2) { int32_t schTaskTableNumCompare(const void *key1, const void *key2) {
SSchTask *pTask1 = *(SSchTask **)key1; SSchTask *pTask1 = *(SSchTask **)key1;
SSchTask *pTask2 = *(SSchTask **)key2; SSchTask *pTask2 = *(SSchTask **)key2;
if (pTask1->plan->execNodeStat.tableNum < pTask2->plan->execNodeStat.tableNum) { if (pTask1->plan->execNodeStat.tableNum < pTask2->plan->execNodeStat.tableNum) {
return 1; return 1;
} else if (pTask1->plan->execNodeStat.tableNum > pTask2->plan->execNodeStat.tableNum) { } else if (pTask1->plan->execNodeStat.tableNum > pTask2->plan->execNodeStat.tableNum) {
@ -193,22 +195,21 @@ int32_t schTaskTableNumCompare(const void* key1, const void* key2) {
} }
} }
int32_t schLaunchTasksInFlowCtrlListImpl(SSchJob *pJob, SSchFlowControl *ctrl) { int32_t schLaunchTasksInFlowCtrlListImpl(SSchJob *pJob, SSchFlowControl *ctrl) {
SCH_LOCK(SCH_WRITE, &ctrl->lock); SCH_LOCK(SCH_WRITE, &ctrl->lock);
if (NULL == ctrl->taskList || taosArrayGetSize(ctrl->taskList) <= 0) { if (NULL == ctrl->taskList || taosArrayGetSize(ctrl->taskList) <= 0) {
SCH_UNLOCK(SCH_WRITE, &ctrl->lock); SCH_UNLOCK(SCH_WRITE, &ctrl->lock);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t remainNum = schMgmt.cfg.maxNodeTableNum - ctrl->tableNumSum; int32_t remainNum = schMgmt.cfg.maxNodeTableNum - ctrl->tableNumSum;
int32_t taskNum = taosArrayGetSize(ctrl->taskList); int32_t taskNum = taosArrayGetSize(ctrl->taskList);
int32_t code = 0; int32_t code = 0;
SSchTask *pTask = NULL; SSchTask *pTask = NULL;
if (taskNum > 1 && !ctrl->sorted) { if (taskNum > 1 && !ctrl->sorted) {
taosArraySort(ctrl->taskList, schTaskTableNumCompare); // desc order taosArraySort(ctrl->taskList, schTaskTableNumCompare); // desc order
} }
for (int32_t i = 0; i < taskNum; ++i) { for (int32_t i = 0; i < taskNum; ++i) {
@ -216,36 +217,36 @@ int32_t schLaunchTasksInFlowCtrlListImpl(SSchJob *pJob, SSchFlowControl *ctrl) {
SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode); SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode);
if (pTask->plan->execNodeStat.tableNum > remainNum && ctrl->execTaskNum > 0) { if (pTask->plan->execNodeStat.tableNum > remainNum && ctrl->execTaskNum > 0) {
SCH_TASK_DLOG("task NOT to launch, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", SCH_TASK_DLOG("task NOT to launch, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", ep->fqdn,
ep->fqdn, ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum); ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum);
continue; continue;
} }
ctrl->tableNumSum += pTask->plan->execNodeStat.tableNum; ctrl->tableNumSum += pTask->plan->execNodeStat.tableNum;
++ctrl->execTaskNum; ++ctrl->execTaskNum;
taosArrayRemove(ctrl->taskList, i); taosArrayRemove(ctrl->taskList, i);
SCH_TASK_DLOG("task to launch, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", SCH_TASK_DLOG("task to launch, fqdn:%s, port:%d, tableNum:%d, remainNum:%d, remainExecTaskNum:%d", ep->fqdn,
ep->fqdn, ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum); ep->port, pTask->plan->execNodeStat.tableNum, ctrl->tableNumSum, ctrl->execTaskNum);
SCH_ERR_JRET(schAsyncLaunchTaskImpl(pJob, pTask)); SCH_ERR_JRET(schAsyncLaunchTaskImpl(pJob, pTask));
remainNum -= pTask->plan->execNodeStat.tableNum; remainNum -= pTask->plan->execNodeStat.tableNum;
if (remainNum <= 0) { if (remainNum <= 0) {
SCH_TASK_DLOG("no more task to launch, fqdn:%s, port:%d, remainNum:%d, remainExecTaskNum:%d", SCH_TASK_DLOG("no more task to launch, fqdn:%s, port:%d, remainNum:%d, remainExecTaskNum:%d", ep->fqdn, ep->port,
ep->fqdn, ep->port, ctrl->tableNumSum, ctrl->execTaskNum); ctrl->tableNumSum, ctrl->execTaskNum);
break; break;
} }
if (i < (taskNum - 1)) { if (i < (taskNum - 1)) {
SSchTask *pLastTask = *(SSchTask **)taosArrayGetLast(ctrl->taskList); SSchTask *pLastTask = *(SSchTask **)taosArrayGetLast(ctrl->taskList);
if (remainNum < pLastTask->plan->execNodeStat.tableNum) { if (remainNum < pLastTask->plan->execNodeStat.tableNum) {
SCH_TASK_DLOG("no more task to launch, fqdn:%s, port:%d, remainNum:%d, remainExecTaskNum:%d, smallestInList:%d", SCH_TASK_DLOG("no more task to launch, fqdn:%s, port:%d, remainNum:%d, remainExecTaskNum:%d, smallestInList:%d",
ep->fqdn, ep->port, ctrl->tableNumSum, ctrl->execTaskNum, pLastTask->plan->execNodeStat.tableNum); ep->fqdn, ep->port, ctrl->tableNumSum, ctrl->execTaskNum, pLastTask->plan->execNodeStat.tableNum);
break; break;
} }
} }
@ -253,7 +254,7 @@ int32_t schLaunchTasksInFlowCtrlListImpl(SSchJob *pJob, SSchFlowControl *ctrl) {
--i; --i;
--taskNum; --taskNum;
} }
_return: _return:
SCH_UNLOCK(SCH_WRITE, &ctrl->lock); SCH_UNLOCK(SCH_WRITE, &ctrl->lock);
@ -261,11 +262,10 @@ _return:
if (code) { if (code) {
code = schProcessOnTaskFailure(pJob, pTask, code); code = schProcessOnTaskFailure(pJob, pTask, code);
} }
SCH_RET(code); SCH_RET(code);
} }
int32_t schLaunchTasksInFlowCtrlList(SSchJob *pJob, SSchTask *pTask) { int32_t schLaunchTasksInFlowCtrlList(SSchJob *pJob, SSchTask *pTask) {
if (!SCH_TASK_NEED_FLOW_CTRL(pJob, pTask)) { if (!SCH_TASK_NEED_FLOW_CTRL(pJob, pTask)) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -274,17 +274,16 @@ int32_t schLaunchTasksInFlowCtrlList(SSchJob *pJob, SSchTask *pTask) {
SCH_ERR_RET(schDecTaskFlowQuota(pJob, pTask)); SCH_ERR_RET(schDecTaskFlowQuota(pJob, pTask));
SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode); SEp *ep = SCH_GET_CUR_EP(&pTask->plan->execNode);
SSchFlowControl *ctrl = (SSchFlowControl *)taosHashGet(pJob->flowCtrl, ep, sizeof(SEp)); SSchFlowControl *ctrl = (SSchFlowControl *)taosHashGet(pJob->flowCtrl, ep, sizeof(SEp));
if (NULL == ctrl) { if (NULL == ctrl) {
SCH_TASK_ELOG("taosHashGet node from flowCtrl failed, fqdn:%s, port:%d", ep->fqdn, ep->port); SCH_TASK_ELOG("taosHashGet node from flowCtrl failed, fqdn:%s, port:%d", ep->fqdn, ep->port);
SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR);
} }
int32_t code = schLaunchTasksInFlowCtrlListImpl(pJob, ctrl);; int32_t code = schLaunchTasksInFlowCtrlListImpl(pJob, ctrl);
;
SCH_ERR_RET(code); SCH_ERR_RET(code);
return code; // to avoid compiler error return code; // to avoid compiler error
} }

151
source/libs/scheduler/src/schJob.c
View File

@ -52,9 +52,8 @@ _return:
bool schJobDone(SSchJob *pJob) { bool schJobDone(SSchJob *pJob) {
int8_t status = SCH_GET_JOB_STATUS(pJob); int8_t status = SCH_GET_JOB_STATUS(pJob);
return (status == JOB_TASK_STATUS_FAIL || status == JOB_TASK_STATUS_DROP || return (status == JOB_TASK_STATUS_FAIL || status == JOB_TASK_STATUS_DROP || status == JOB_TASK_STATUS_SUCC);
status == JOB_TASK_STATUS_SUCC);
} }
FORCE_INLINE bool schJobNeedToStop(SSchJob *pJob, int8_t *pStatus) { FORCE_INLINE bool schJobNeedToStop(SSchJob *pJob, int8_t *pStatus) {
@ -221,7 +220,7 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) {
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY); SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
} }
SCH_TASK_DLOG("parents info, the %d parent TID 0x%" PRIx64, n, (*parentTask)->taskId); SCH_TASK_DLOG("parents info, the %d parent TID 0x%" PRIx64, n, (*parentTask)->taskId);
} }
SCH_TASK_DLOG("level:%d, parentNum:%d, childNum:%d", i, parentNum, childNum); SCH_TASK_DLOG("level:%d, parentNum:%d, childNum:%d", i, parentNum, childNum);
@ -235,7 +234,7 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) {
SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR);
} }
SSchTask* pTask = taosArrayGet(pLevel->subTasks, 0); SSchTask *pTask = taosArrayGet(pLevel->subTasks, 0);
if (SUBPLAN_TYPE_MODIFY != pTask->plan->subplanType) { if (SUBPLAN_TYPE_MODIFY != pTask->plan->subplanType) {
pJob->attr.needFetch = true; pJob->attr.needFetch = true;
} }
@ -244,7 +243,6 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schAppendJobDataSrc(SSchJob *pJob, SSchTask *pTask) { int32_t schAppendJobDataSrc(SSchJob *pJob, SSchTask *pTask) {
if (!SCH_IS_DATA_BIND_QRY_TASK(pTask)) { if (!SCH_IS_DATA_BIND_QRY_TASK(pTask)) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -255,7 +253,6 @@ int32_t schAppendJobDataSrc(SSchJob *pJob, SSchTask *pTask) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob) { int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob) {
int32_t code = 0; int32_t code = 0;
pJob->queryId = pDag->queryId; pJob->queryId = pDag->queryId;
@ -365,7 +362,7 @@ int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob) {
SCH_ERR_JRET(schBuildTaskRalation(pJob, planToTask)); SCH_ERR_JRET(schBuildTaskRalation(pJob, planToTask));
_return: _return:
if (planToTask) { if (planToTask) {
taosHashCleanup(planToTask); taosHashCleanup(planToTask);
} }
@ -373,8 +370,7 @@ _return:
SCH_RET(code); SCH_RET(code);
} }
int32_t schDumpJobExecRes(SSchJob *pJob, SExecResult *pRes) {
int32_t schDumpJobExecRes(SSchJob* pJob, SExecResult* pRes) {
pRes->code = atomic_load_32(&pJob->errCode); pRes->code = atomic_load_32(&pJob->errCode);
pRes->numOfRows = pJob->resNumOfRows; pRes->numOfRows = pJob->resNumOfRows;
pRes->res = pJob->execRes.res; pRes->res = pJob->execRes.res;
@ -387,13 +383,13 @@ int32_t schDumpJobExecRes(SSchJob* pJob, SExecResult* pRes) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schDumpJobFetchRes(SSchJob* pJob, void** pData) { int32_t schDumpJobFetchRes(SSchJob *pJob, void **pData) {
int32_t code = 0; int32_t code = 0;
SCH_LOCK(SCH_WRITE, &pJob->resLock); SCH_LOCK(SCH_WRITE, &pJob->resLock);
pJob->fetched = true; pJob->fetched = true;
if (pJob->fetchRes && ((SRetrieveTableRsp *)pJob->fetchRes)->completed) { if (pJob->fetchRes && ((SRetrieveTableRsp *)pJob->fetchRes)->completed) {
SCH_ERR_JRET(schSwitchJobStatus(pJob, JOB_TASK_STATUS_SUCC, NULL)); SCH_ERR_JRET(schSwitchJobStatus(pJob, JOB_TASK_STATUS_SUCC, NULL));
} }
@ -422,12 +418,12 @@ int32_t schDumpJobFetchRes(SSchJob* pJob, void** pData) {
_return: _return:
SCH_UNLOCK(SCH_WRITE, &pJob->resLock); SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
return code; return code;
} }
int32_t schNotifyUserExecRes(SSchJob* pJob) { int32_t schNotifyUserExecRes(SSchJob *pJob) {
SExecResult* pRes = taosMemoryCalloc(1, sizeof(SExecResult)); SExecResult *pRes = taosMemoryCalloc(1, sizeof(SExecResult));
if (pRes) { if (pRes) {
schDumpJobExecRes(pJob, pRes); schDumpJobExecRes(pJob, pRes);
} }
@ -439,9 +435,9 @@ int32_t schNotifyUserExecRes(SSchJob* pJob) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schNotifyUserFetchRes(SSchJob* pJob) { int32_t schNotifyUserFetchRes(SSchJob *pJob) {
void* pRes = NULL; void *pRes = NULL;
schDumpJobFetchRes(pJob, &pRes); schDumpJobFetchRes(pJob, &pRes);
SCH_JOB_DLOG("sch start to invoke fetch cb, code: %s", tstrerror(pJob->errCode)); SCH_JOB_DLOG("sch start to invoke fetch cb, code: %s", tstrerror(pJob->errCode));
@ -453,17 +449,17 @@ int32_t schNotifyUserFetchRes(SSchJob* pJob) {
void schPostJobRes(SSchJob *pJob, SCH_OP_TYPE op) { void schPostJobRes(SSchJob *pJob, SCH_OP_TYPE op) {
SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock);
if (SCH_OP_NULL == pJob->opStatus.op) { if (SCH_OP_NULL == pJob->opStatus.op) {
SCH_JOB_DLOG("job not in any operation, no need to post job res, status:%s", jobTaskStatusStr(pJob->status)); SCH_JOB_DLOG("job not in any operation, no need to post job res, status:%s", jobTaskStatusStr(pJob->status));
goto _return; goto _return;
} }
if (op && pJob->opStatus.op != op) { if (op && pJob->opStatus.op != op) {
SCH_JOB_ELOG("job in operation %s mis-match with expected %s", schGetOpStr(pJob->opStatus.op), schGetOpStr(op)); SCH_JOB_ELOG("job in operation %s mis-match with expected %s", schGetOpStr(pJob->opStatus.op), schGetOpStr(op));
goto _return; goto _return;
} }
if (SCH_JOB_IN_SYNC_OP(pJob)) { if (SCH_JOB_IN_SYNC_OP(pJob)) {
SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock);
tsem_post(&pJob->rspSem); tsem_post(&pJob->rspSem);
@ -487,7 +483,7 @@ _return:
int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode) { int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode) {
schUpdateJobErrCode(pJob, errCode); schUpdateJobErrCode(pJob, errCode);
int32_t code = atomic_load_32(&pJob->errCode); int32_t code = atomic_load_32(&pJob->errCode);
if (code) { if (code) {
SCH_JOB_DLOG("job failed with error %s", tstrerror(code)); SCH_JOB_DLOG("job failed with error %s", tstrerror(code));
@ -507,9 +503,7 @@ int32_t schHandleJobFailure(SSchJob *pJob, int32_t errCode) {
return TSDB_CODE_SCH_IGNORE_ERROR; return TSDB_CODE_SCH_IGNORE_ERROR;
} }
int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode) { int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode) { SCH_RET(schProcessOnJobFailure(pJob, errCode)); }
SCH_RET(schProcessOnJobFailure(pJob, errCode));
}
int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode) { int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode) {
if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) { if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) {
@ -520,8 +514,7 @@ int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode) {
return TSDB_CODE_SCH_IGNORE_ERROR; return TSDB_CODE_SCH_IGNORE_ERROR;
} }
int32_t schProcessOnJobPartialSuccess(SSchJob *pJob) {
int32_t schProcessOnJobPartialSuccess(SSchJob *pJob) {
if (schChkCurrentOp(pJob, SCH_OP_FETCH, -1)) { if (schChkCurrentOp(pJob, SCH_OP_FETCH, -1)) {
SCH_ERR_RET(schLaunchFetchTask(pJob)); SCH_ERR_RET(schLaunchFetchTask(pJob));
} else { } else {
@ -531,9 +524,7 @@ int32_t schProcessOnJobPartialSuccess(SSchJob *pJob) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
void schProcessOnDataFetched(SSchJob *pJob) { void schProcessOnDataFetched(SSchJob *pJob) { schPostJobRes(pJob, SCH_OP_FETCH); }
schPostJobRes(pJob, SCH_OP_FETCH);
}
int32_t schProcessOnExplainDone(SSchJob *pJob, SSchTask *pTask, SRetrieveTableRsp *pRsp) { int32_t schProcessOnExplainDone(SSchJob *pJob, SSchTask *pTask, SRetrieveTableRsp *pRsp) {
SCH_TASK_DLOG("got explain rsp, rows:%d, complete:%d", htonl(pRsp->numOfRows), pRsp->completed); SCH_TASK_DLOG("got explain rsp, rows:%d, complete:%d", htonl(pRsp->numOfRows), pRsp->completed);
@ -548,14 +539,13 @@ int32_t schProcessOnExplainDone(SSchJob *pJob, SSchTask *pTask, SRetrieveTableRs
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask) { int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask) {
if (!SCH_IS_QUERY_JOB(pJob)) { if (!SCH_IS_QUERY_JOB(pJob)) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
SSchLevel *pLevel = pTask->level; SSchLevel *pLevel = pTask->level;
int32_t doneNum = atomic_add_fetch_32(&pLevel->taskDoneNum, 1); int32_t doneNum = atomic_add_fetch_32(&pLevel->taskDoneNum, 1);
if (doneNum == pLevel->taskNum) { if (doneNum == pLevel->taskNum) {
pJob->levelIdx--; pJob->levelIdx--;
@ -566,7 +556,7 @@ int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask) {
if (pTask->children && taosArrayGetSize(pTask->children) > 0) { if (pTask->children && taosArrayGetSize(pTask->children) > 0) {
continue; continue;
} }
SCH_ERR_RET(schLaunchTask(pJob, pTask)); SCH_ERR_RET(schLaunchTask(pJob, pTask));
} }
} }
@ -577,11 +567,11 @@ int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask) {
int32_t schSaveJobExecRes(SSchJob *pJob, SQueryTableRsp *rsp) { int32_t schSaveJobExecRes(SSchJob *pJob, SQueryTableRsp *rsp) {
if (rsp->tbFName[0]) { if (rsp->tbFName[0]) {
SCH_LOCK(SCH_WRITE, &pJob->resLock); SCH_LOCK(SCH_WRITE, &pJob->resLock);
if (NULL == pJob->execRes.res) { if (NULL == pJob->execRes.res) {
pJob->execRes.res = taosArrayInit(pJob->taskNum, sizeof(STbVerInfo)); pJob->execRes.res = taosArrayInit(pJob->taskNum, sizeof(STbVerInfo));
if (NULL == pJob->execRes.res) { if (NULL == pJob->execRes.res) {
SCH_UNLOCK(SCH_WRITE, &pJob->resLock); SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
SCH_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); SCH_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
@ -610,7 +600,6 @@ int32_t schGetTaskInJob(SSchJob *pJob, uint64_t taskId, SSchTask **pTask) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schLaunchJob(SSchJob *pJob) { int32_t schLaunchJob(SSchJob *pJob) {
if (EXPLAIN_MODE_STATIC == pJob->attr.explainMode) { if (EXPLAIN_MODE_STATIC == pJob->attr.explainMode) {
SCH_ERR_RET(qExecStaticExplain(pJob->pDag, (SRetrieveTableRsp **)&pJob->fetchRes)); SCH_ERR_RET(qExecStaticExplain(pJob->pDag, (SRetrieveTableRsp **)&pJob->fetchRes));
@ -623,11 +612,10 @@ int32_t schLaunchJob(SSchJob *pJob) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
void schDropJobAllTasks(SSchJob *pJob) { void schDropJobAllTasks(SSchJob *pJob) {
schDropTaskInHashList(pJob, pJob->execTasks); schDropTaskInHashList(pJob, pJob->execTasks);
// schDropTaskInHashList(pJob, pJob->succTasks); // schDropTaskInHashList(pJob, pJob->succTasks);
// schDropTaskInHashList(pJob, pJob->failTasks); // schDropTaskInHashList(pJob, pJob->failTasks);
} }
void schFreeJobImpl(void *job) { void schFreeJobImpl(void *job) {
@ -659,10 +647,10 @@ void schFreeJobImpl(void *job) {
schFreeFlowCtrl(pJob); schFreeFlowCtrl(pJob);
taosHashCleanup(pJob->execTasks); taosHashCleanup(pJob->execTasks);
// taosHashCleanup(pJob->failTasks); // taosHashCleanup(pJob->failTasks);
// taosHashCleanup(pJob->succTasks); // taosHashCleanup(pJob->succTasks);
taosHashCleanup(pJob->taskList); taosHashCleanup(pJob->taskList);
taosArrayDestroy(pJob->levels); taosArrayDestroy(pJob->levels);
taosArrayDestroy(pJob->nodeList); taosArrayDestroy(pJob->nodeList);
taosArrayDestroy(pJob->dataSrcTasks); taosArrayDestroy(pJob->dataSrcTasks);
@ -688,19 +676,19 @@ void schFreeJobImpl(void *job) {
} }
int32_t schJobFetchRows(SSchJob *pJob) { int32_t schJobFetchRows(SSchJob *pJob) {
int32_t code = 0; int32_t code = 0;
if (!(pJob->attr.explainMode == EXPLAIN_MODE_STATIC)) { if (!(pJob->attr.explainMode == EXPLAIN_MODE_STATIC)) {
SCH_ERR_RET(schLaunchFetchTask(pJob)); SCH_ERR_RET(schLaunchFetchTask(pJob));
if (schChkCurrentOp(pJob, SCH_OP_FETCH, true)) { if (schChkCurrentOp(pJob, SCH_OP_FETCH, true)) {
SCH_JOB_DLOG("sync wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob)); SCH_JOB_DLOG("sync wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob));
tsem_wait(&pJob->rspSem); tsem_wait(&pJob->rspSem);
SCH_RET(schDumpJobFetchRes(pJob, pJob->userRes.fetchRes)); SCH_RET(schDumpJobFetchRes(pJob, pJob->userRes.fetchRes));
} }
} else { } else {
if (schChkCurrentOp(pJob, SCH_OP_FETCH, true)) { if (schChkCurrentOp(pJob, SCH_OP_FETCH, true)) {
SCH_RET(schDumpJobFetchRes(pJob, pJob->userRes.fetchRes)); SCH_RET(schDumpJobFetchRes(pJob, pJob->userRes.fetchRes));
} else { } else {
schPostJobRes(pJob, SCH_OP_FETCH); schPostJobRes(pJob, SCH_OP_FETCH);
} }
@ -736,9 +724,9 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) {
} else { } else {
pJob->nodeList = taosArrayDup(pReq->pNodeList); pJob->nodeList = taosArrayDup(pReq->pNodeList);
} }
pJob->taskList = pJob->taskList = taosHashInit(pReq->pDag->numOfSubplans, taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT), false,
taosHashInit(pReq->pDag->numOfSubplans, taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT), false, HASH_ENTRY_LOCK); HASH_ENTRY_LOCK);
if (NULL == pJob->taskList) { if (NULL == pJob->taskList) {
SCH_JOB_ELOG("taosHashInit %d taskList failed", pReq->pDag->numOfSubplans); SCH_JOB_ELOG("taosHashInit %d taskList failed", pReq->pDag->numOfSubplans);
SCH_ERR_JRET(TSDB_CODE_QRY_OUT_OF_MEMORY); SCH_ERR_JRET(TSDB_CODE_QRY_OUT_OF_MEMORY);
@ -750,8 +738,8 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) {
SCH_ERR_JRET(qExecExplainBegin(pReq->pDag, &pJob->explainCtx, pReq->startTs)); SCH_ERR_JRET(qExecExplainBegin(pReq->pDag, &pJob->explainCtx, pReq->startTs));
} }
pJob->execTasks = pJob->execTasks = taosHashInit(pReq->pDag->numOfSubplans, taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT), false,
taosHashInit(pReq->pDag->numOfSubplans, taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT), false, HASH_ENTRY_LOCK); HASH_ENTRY_LOCK);
if (NULL == pJob->execTasks) { if (NULL == pJob->execTasks) {
SCH_JOB_ELOG("taosHashInit %d execTasks failed", pReq->pDag->numOfSubplans); SCH_JOB_ELOG("taosHashInit %d execTasks failed", pReq->pDag->numOfSubplans);
SCH_ERR_JRET(TSDB_CODE_QRY_OUT_OF_MEMORY); SCH_ERR_JRET(TSDB_CODE_QRY_OUT_OF_MEMORY);
@ -769,7 +757,7 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) {
*pJobId = pJob->refId; *pJobId = pJob->refId;
SCH_JOB_DLOG("job refId:0x%" PRIx64" created", pJob->refId); SCH_JOB_DLOG("job refId:0x%" PRIx64 " created", pJob->refId);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -782,31 +770,31 @@ _return:
} else { } else {
taosRemoveRef(schMgmt.jobRef, pJob->refId); taosRemoveRef(schMgmt.jobRef, pJob->refId);
} }
SCH_RET(code); SCH_RET(code);
} }
int32_t schExecJob(SSchJob *pJob, SSchedulerReq *pReq) { int32_t schExecJob(SSchJob *pJob, SSchedulerReq *pReq) {
int32_t code = 0; int32_t code = 0;
qDebug("QID:0x%" PRIx64 " sch job refId 0x%"PRIx64 " started", pReq->pDag->queryId, pJob->refId); qDebug("QID:0x%" PRIx64 " sch job refId 0x%" PRIx64 " started", pReq->pDag->queryId, pJob->refId);
SCH_ERR_RET(schLaunchJob(pJob)); SCH_ERR_RET(schLaunchJob(pJob));
if (pReq->syncReq) { if (pReq->syncReq) {
SCH_JOB_DLOG("sync wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob)); SCH_JOB_DLOG("sync wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob));
tsem_wait(&pJob->rspSem); tsem_wait(&pJob->rspSem);
} }
SCH_JOB_DLOG("job exec done, job status:%s, jobId:0x%" PRIx64, SCH_GET_JOB_STATUS_STR(pJob), pJob->refId); SCH_JOB_DLOG("job exec done, job status:%s, jobId:0x%" PRIx64, SCH_GET_JOB_STATUS_STR(pJob), pJob->refId);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
void schDirectPostJobRes(SSchedulerReq* pReq, int32_t errCode) { void schDirectPostJobRes(SSchedulerReq *pReq, int32_t errCode) {
if (NULL == pReq || pReq->syncReq) { if (NULL == pReq || pReq->syncReq) {
return; return;
} }
if (pReq->execFp) { if (pReq->execFp) {
(*pReq->execFp)(NULL, pReq->cbParam, errCode); (*pReq->execFp)(NULL, pReq->cbParam, errCode);
} else if (pReq->fetchFp) { } else if (pReq->fetchFp) {
@ -827,16 +815,17 @@ bool schChkCurrentOp(SSchJob *pJob, int32_t op, int8_t sync) {
return r; return r;
} }
void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int32_t errCode) { void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq *pReq, int32_t errCode) {
int32_t op = 0; int32_t op = 0;
switch (type) { switch (type) {
case SCH_OP_EXEC: case SCH_OP_EXEC:
if (pReq && pReq->syncReq) { if (pReq && pReq->syncReq) {
SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock);
op = atomic_val_compare_exchange_32(&pJob->opStatus.op, type, SCH_OP_NULL); op = atomic_val_compare_exchange_32(&pJob->opStatus.op, type, SCH_OP_NULL);
if (SCH_OP_NULL == op || op != type) { if (SCH_OP_NULL == op || op != type) {
SCH_JOB_ELOG("job not in %s operation, op:%s, status:%s", schGetOpStr(type), schGetOpStr(op), jobTaskStatusStr(pJob->status)); SCH_JOB_ELOG("job not in %s operation, op:%s, status:%s", schGetOpStr(type), schGetOpStr(op),
jobTaskStatusStr(pJob->status));
} }
SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock);
schDumpJobExecRes(pJob, pReq->pExecRes); schDumpJobExecRes(pJob, pReq->pExecRes);
@ -847,7 +836,8 @@ void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int
SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock);
op = atomic_val_compare_exchange_32(&pJob->opStatus.op, type, SCH_OP_NULL); op = atomic_val_compare_exchange_32(&pJob->opStatus.op, type, SCH_OP_NULL);
if (SCH_OP_NULL == op || op != type) { if (SCH_OP_NULL == op || op != type) {
SCH_JOB_ELOG("job not in %s operation, op:%s, status:%s", schGetOpStr(type), schGetOpStr(op), jobTaskStatusStr(pJob->status)); SCH_JOB_ELOG("job not in %s operation, op:%s, status:%s", schGetOpStr(type), schGetOpStr(op),
jobTaskStatusStr(pJob->status));
} }
SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock);
} }
@ -866,10 +856,10 @@ void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int
SCH_JOB_DLOG("job end %s operation with code %s", schGetOpStr(type), tstrerror(errCode)); SCH_JOB_DLOG("job end %s operation with code %s", schGetOpStr(type), tstrerror(errCode));
} }
int32_t schProcessOnOpBegin(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq) { int32_t schProcessOnOpBegin(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq *pReq) {
int32_t code = 0; int32_t code = 0;
int8_t status = SCH_GET_JOB_STATUS(pJob); int8_t status = SCH_GET_JOB_STATUS(pJob);
switch (type) { switch (type) {
case SCH_OP_EXEC: case SCH_OP_EXEC:
SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_LOCK(SCH_WRITE, &pJob->opStatus.lock);
@ -879,9 +869,9 @@ int32_t schProcessOnOpBegin(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq
schDirectPostJobRes(pReq, TSDB_CODE_TSC_APP_ERROR); schDirectPostJobRes(pReq, TSDB_CODE_TSC_APP_ERROR);
SCH_ERR_RET(TSDB_CODE_TSC_APP_ERROR); SCH_ERR_RET(TSDB_CODE_TSC_APP_ERROR);
} }
SCH_JOB_DLOG("job start %s operation", schGetOpStr(pJob->opStatus.op)); SCH_JOB_DLOG("job start %s operation", schGetOpStr(pJob->opStatus.op));
pJob->opStatus.syncReq = pReq->syncReq; pJob->opStatus.syncReq = pReq->syncReq;
SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock);
break; break;
@ -893,16 +883,16 @@ int32_t schProcessOnOpBegin(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq
schDirectPostJobRes(pReq, TSDB_CODE_TSC_APP_ERROR); schDirectPostJobRes(pReq, TSDB_CODE_TSC_APP_ERROR);
SCH_ERR_RET(TSDB_CODE_TSC_APP_ERROR); SCH_ERR_RET(TSDB_CODE_TSC_APP_ERROR);
} }
SCH_JOB_DLOG("job start %s operation", schGetOpStr(pJob->opStatus.op)); SCH_JOB_DLOG("job start %s operation", schGetOpStr(pJob->opStatus.op));
pJob->userRes.fetchRes = pReq->pFetchRes; pJob->userRes.fetchRes = pReq->pFetchRes;
pJob->userRes.fetchFp = pReq->fetchFp; pJob->userRes.fetchFp = pReq->fetchFp;
pJob->userRes.cbParam = pReq->cbParam; pJob->userRes.cbParam = pReq->cbParam;
pJob->opStatus.syncReq = pReq->syncReq; pJob->opStatus.syncReq = pReq->syncReq;
SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock); SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock);
if (!SCH_JOB_NEED_FETCH(pJob)) { if (!SCH_JOB_NEED_FETCH(pJob)) {
SCH_JOB_ELOG("no need to fetch data, status:%s", SCH_GET_JOB_STATUS_STR(pJob)); SCH_JOB_ELOG("no need to fetch data, status:%s", SCH_GET_JOB_STATUS_STR(pJob));
SCH_ERR_RET(TSDB_CODE_QRY_APP_ERROR); SCH_ERR_RET(TSDB_CODE_QRY_APP_ERROR);
@ -912,7 +902,7 @@ int32_t schProcessOnOpBegin(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq
SCH_JOB_ELOG("job status error for fetch, status:%s", jobTaskStatusStr(status)); SCH_JOB_ELOG("job status error for fetch, status:%s", jobTaskStatusStr(status));
SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR);
} }
break; break;
case SCH_OP_GET_STATUS: case SCH_OP_GET_STATUS:
if (pJob->status < JOB_TASK_STATUS_INIT || pJob->levelNum <= 0 || NULL == pJob->levels) { if (pJob->status < JOB_TASK_STATUS_INIT || pJob->levelNum <= 0 || NULL == pJob->levels) {
@ -941,23 +931,23 @@ void schProcessOnCbEnd(SSchJob *pJob, SSchTask *pTask, int32_t errCode) {
if (errCode) { if (errCode) {
schHandleJobFailure(pJob, errCode); schHandleJobFailure(pJob, errCode);
} }
if (pJob) { if (pJob) {
schReleaseJob(pJob->refId); schReleaseJob(pJob->refId);
} }
} }
int32_t schProcessOnCbBegin(SSchJob** job, SSchTask** task, uint64_t qId, int64_t rId, uint64_t tId) { int32_t schProcessOnCbBegin(SSchJob **job, SSchTask **task, uint64_t qId, int64_t rId, uint64_t tId) {
int32_t code = 0; int32_t code = 0;
int8_t status = 0; int8_t status = 0;
SSchTask *pTask = NULL; SSchTask *pTask = NULL;
SSchJob *pJob = schAcquireJob(rId); SSchJob *pJob = schAcquireJob(rId);
if (NULL == pJob) { if (NULL == pJob) {
qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, qId, tId, rId); qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, qId, tId, rId);
SCH_ERR_RET(TSDB_CODE_QRY_JOB_NOT_EXIST); SCH_ERR_RET(TSDB_CODE_QRY_JOB_NOT_EXIST);
} }
if (schJobNeedToStop(pJob, &status)) { if (schJobNeedToStop(pJob, &status)) {
SCH_TASK_DLOG("will not do further processing cause of job status %s", jobTaskStatusStr(status)); SCH_TASK_DLOG("will not do further processing cause of job status %s", jobTaskStatusStr(status));
SCH_ERR_JRET(TSDB_CODE_SCH_IGNORE_ERROR); SCH_ERR_JRET(TSDB_CODE_SCH_IGNORE_ERROR);
@ -980,9 +970,6 @@ _return:
if (pJob) { if (pJob) {
schReleaseJob(rId); schReleaseJob(rId);
} }
SCH_RET(code); SCH_RET(code);
} }

16
source/libs/scheduler/src/schStatus.c
View File

@ -29,25 +29,25 @@ int32_t schSwitchJobStatus(SSchJob* pJob, int32_t status, void* param) {
case JOB_TASK_STATUS_INIT: case JOB_TASK_STATUS_INIT:
break; break;
case JOB_TASK_STATUS_EXEC: case JOB_TASK_STATUS_EXEC:
SCH_ERR_JRET(schExecJob(pJob, (SSchedulerReq*)param)); SCH_ERR_JRET(schExecJob(pJob, (SSchedulerReq*)param));
break; break;
case JOB_TASK_STATUS_PART_SUCC: case JOB_TASK_STATUS_PART_SUCC:
SCH_ERR_JRET(schProcessOnJobPartialSuccess(pJob)); SCH_ERR_JRET(schProcessOnJobPartialSuccess(pJob));
break; break;
case JOB_TASK_STATUS_SUCC: case JOB_TASK_STATUS_SUCC:
break; break;
case JOB_TASK_STATUS_FAIL: case JOB_TASK_STATUS_FAIL:
SCH_RET(schProcessOnJobFailure(pJob, (param ? *(int32_t*)param : 0))); SCH_RET(schProcessOnJobFailure(pJob, (param ? *(int32_t*)param : 0)));
break; break;
case JOB_TASK_STATUS_DROP: case JOB_TASK_STATUS_DROP:
schProcessOnJobDropped(pJob, *(int32_t*)param); schProcessOnJobDropped(pJob, *(int32_t*)param);
if (taosRemoveRef(schMgmt.jobRef, pJob->refId)) { if (taosRemoveRef(schMgmt.jobRef, pJob->refId)) {
SCH_JOB_ELOG("remove job from job list failed, refId:0x%" PRIx64, pJob->refId); SCH_JOB_ELOG("remove job from job list failed, refId:0x%" PRIx64, pJob->refId);
} else { } else {
SCH_JOB_DLOG("job removed from jobRef list, refId:0x%" PRIx64, pJob->refId); SCH_JOB_DLOG("job removed from jobRef list, refId:0x%" PRIx64, pJob->refId);
} }
break; break;
default: { default: {
SCH_JOB_ELOG("unknown job status %d", status); SCH_JOB_ELOG("unknown job status %d", status);
SCH_RET(TSDB_CODE_SCH_STATUS_ERROR); SCH_RET(TSDB_CODE_SCH_STATUS_ERROR);
@ -62,7 +62,7 @@ _return:
} }
int32_t schHandleOpBeginEvent(int64_t jobId, SSchJob** job, SCH_OP_TYPE type, SSchedulerReq* pReq) { int32_t schHandleOpBeginEvent(int64_t jobId, SSchJob** job, SCH_OP_TYPE type, SSchedulerReq* pReq) {
SSchJob *pJob = schAcquireJob(jobId); SSchJob* pJob = schAcquireJob(jobId);
if (NULL == pJob) { if (NULL == pJob) {
qWarn("Acquire sch job failed, may be dropped, jobId:0x%" PRIx64, jobId); qWarn("Acquire sch job failed, may be dropped, jobId:0x%" PRIx64, jobId);
SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR); SCH_ERR_RET(TSDB_CODE_SCH_STATUS_ERROR);
@ -75,12 +75,12 @@ int32_t schHandleOpBeginEvent(int64_t jobId, SSchJob** job, SCH_OP_TYPE type, SS
int32_t schHandleOpEndEvent(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int32_t errCode) { int32_t schHandleOpEndEvent(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int32_t errCode) {
int32_t code = errCode; int32_t code = errCode;
if (NULL == pJob) { if (NULL == pJob) {
schDirectPostJobRes(pReq, errCode); schDirectPostJobRes(pReq, errCode);
SCH_RET(code); SCH_RET(code);
} }
schProcessOnOpEnd(pJob, type, pReq, errCode); schProcessOnOpEnd(pJob, type, pReq, errCode);
if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) { if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) {
@ -91,5 +91,3 @@ int32_t schHandleOpEndEvent(SSchJob* pJob, SCH_OP_TYPE type, SSchedulerReq* pReq
return code; return code;
} }

63
source/libs/scheduler/src/schTask.c
View File

@ -16,12 +16,12 @@
#include "catalog.h" #include "catalog.h"
#include "command.h" #include "command.h"
#include "query.h" #include "query.h"
#include "qworker.h"
#include "schInt.h" #include "schInt.h"
#include "tglobal.h"
#include "tmsg.h" #include "tmsg.h"
#include "tref.h" #include "tref.h"
#include "trpc.h" #include "trpc.h"
#include "qworker.h"
#include "tglobal.h"
void schFreeTask(SSchJob *pJob, SSchTask *pTask) { void schFreeTask(SSchJob *pJob, SSchTask *pTask) {
schDeregisterTaskHb(pJob, pTask); schDeregisterTaskHb(pJob, pTask);
@ -94,7 +94,7 @@ int32_t schRecordTaskSucceedNode(SSchJob *pJob, SSchTask *pTask) {
if (SCH_IS_LOCAL_EXEC_TASK(pJob, pTask)) { if (SCH_IS_LOCAL_EXEC_TASK(pJob, pTask)) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx); SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx);
if (NULL == addr) { if (NULL == addr) {
SCH_TASK_ELOG("taosArrayGet candidate addr failed, idx:%d, size:%d", pTask->candidateIdx, SCH_TASK_ELOG("taosArrayGet candidate addr failed, idx:%d, size:%d", pTask->candidateIdx,
@ -162,14 +162,15 @@ int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, v
if (dropExecNode) { if (dropExecNode) {
SCH_RET(schDropTaskExecNode(pJob, pTask, handle, execId)); SCH_RET(schDropTaskExecNode(pJob, pTask, handle, execId));
} }
schUpdateTaskExecNode(pJob, pTask, handle, execId); schUpdateTaskExecNode(pJob, pTask, handle, execId);
if ((execId != pTask->execId) || pTask->waitRetry) { // ignore it if ((execId != pTask->execId) || pTask->waitRetry) { // ignore it
SCH_TASK_DLOG("handle not updated since execId %d is already not current execId %d, waitRetry %d", execId, pTask->execId, pTask->waitRetry); SCH_TASK_DLOG("handle not updated since execId %d is already not current execId %d, waitRetry %d", execId,
pTask->execId, pTask->waitRetry);
SCH_ERR_RET(TSDB_CODE_SCH_IGNORE_ERROR); SCH_ERR_RET(TSDB_CODE_SCH_IGNORE_ERROR);
} }
SCH_SET_TASK_HANDLE(pTask, handle); SCH_SET_TASK_HANDLE(pTask, handle);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -837,17 +838,18 @@ int32_t schHandleExplainRes(SArray *pExplainRes) {
goto _return; goto _return;
} }
SSchTask *pTask = NULL; SSchTask *pTask = NULL;
SSchJob *pJob = NULL; SSchJob *pJob = NULL;
for (int32_t i = 0; i < resNum; ++i) { for (int32_t i = 0; i < resNum; ++i) {
SExplainLocalRsp* localRsp = taosArrayGet(pExplainRes, i); SExplainLocalRsp *localRsp = taosArrayGet(pExplainRes, i);
qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ", begin to handle LOCAL explain rsp msg", localRsp->qId, localRsp->tId); qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ", begin to handle LOCAL explain rsp msg", localRsp->qId, localRsp->tId);
pJob = schAcquireJob(localRsp->rId); pJob = schAcquireJob(localRsp->rId);
if (NULL == pJob) { if (NULL == pJob) {
qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, localRsp->qId, localRsp->tId, localRsp->rId); qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, localRsp->qId,
localRsp->tId, localRsp->rId);
SCH_ERR_JRET(TSDB_CODE_QRY_JOB_NOT_EXIST); SCH_ERR_JRET(TSDB_CODE_QRY_JOB_NOT_EXIST);
} }
@ -857,16 +859,17 @@ int32_t schHandleExplainRes(SArray *pExplainRes) {
schReleaseJob(pJob->refId); schReleaseJob(pJob->refId);
SCH_ERR_JRET(TSDB_CODE_SCH_IGNORE_ERROR); SCH_ERR_JRET(TSDB_CODE_SCH_IGNORE_ERROR);
} }
code = schGetTaskInJob(pJob, localRsp->tId, &pTask); code = schGetTaskInJob(pJob, localRsp->tId, &pTask);
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
code = schProcessExplainRsp(pJob, pTask, &localRsp->rsp); code = schProcessExplainRsp(pJob, pTask, &localRsp->rsp);
} }
schReleaseJob(pJob->refId); schReleaseJob(pJob->refId);
qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ", end to handle LOCAL explain rsp msg, code:%x", localRsp->qId, localRsp->tId, code); qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ", end to handle LOCAL explain rsp msg, code:%x", localRsp->qId,
localRsp->tId, code);
SCH_ERR_JRET(code); SCH_ERR_JRET(code);
@ -879,7 +882,7 @@ int32_t schHandleExplainRes(SArray *pExplainRes) {
_return: _return:
for (int32_t i = 0; i < resNum; ++i) { for (int32_t i = 0; i < resNum; ++i) {
SExplainLocalRsp* localRsp = taosArrayGet(pExplainRes, i); SExplainLocalRsp *localRsp = taosArrayGet(pExplainRes, i);
tFreeSExplainRsp(&localRsp->rsp); tFreeSExplainRsp(&localRsp->rsp);
} }
@ -890,7 +893,7 @@ _return:
int32_t schLaunchRemoteTask(SSchJob *pJob, SSchTask *pTask) { int32_t schLaunchRemoteTask(SSchJob *pJob, SSchTask *pTask) {
SSubplan *plan = pTask->plan; SSubplan *plan = pTask->plan;
int32_t code = 0; int32_t code = 0;
if (NULL == pTask->msg) { // TODO add more detailed reason for failure if (NULL == pTask->msg) { // TODO add more detailed reason for failure
code = qSubPlanToMsg(plan, &pTask->msg, &pTask->msgLen); code = qSubPlanToMsg(plan, &pTask->msg, &pTask->msgLen);
@ -899,7 +902,7 @@ int32_t schLaunchRemoteTask(SSchJob *pJob, SSchTask *pTask) {
pTask->msgLen); pTask->msgLen);
SCH_ERR_RET(code); SCH_ERR_RET(code);
} else if (tsQueryPlannerTrace) { } else if (tsQueryPlannerTrace) {
char *msg = NULL; char *msg = NULL;
int32_t msgLen = 0; int32_t msgLen = 0;
qSubPlanToString(plan, &msg, &msgLen); qSubPlanToString(plan, &msg, &msgLen);
SCH_TASK_DLOGL("physical plan len:%d, %s", msgLen, msg); SCH_TASK_DLOGL("physical plan len:%d, %s", msgLen, msg);
@ -912,18 +915,18 @@ int32_t schLaunchRemoteTask(SSchJob *pJob, SSchTask *pTask) {
if (SCH_IS_QUERY_JOB(pJob)) { if (SCH_IS_QUERY_JOB(pJob)) {
SCH_ERR_RET(schEnsureHbConnection(pJob, pTask)); SCH_ERR_RET(schEnsureHbConnection(pJob, pTask));
} }
SCH_RET(schBuildAndSendMsg(pJob, pTask, NULL, plan->msgType)); SCH_RET(schBuildAndSendMsg(pJob, pTask, NULL, plan->msgType));
} }
int32_t schLaunchLocalTask(SSchJob *pJob, SSchTask *pTask) { int32_t schLaunchLocalTask(SSchJob *pJob, SSchTask *pTask) {
//SCH_ERR_JRET(schSetTaskCandidateAddrs(pJob, pTask)); // SCH_ERR_JRET(schSetTaskCandidateAddrs(pJob, pTask));
if (NULL == schMgmt.queryMgmt) { if (NULL == schMgmt.queryMgmt) {
SCH_ERR_RET(qWorkerInit(NODE_TYPE_CLIENT, CLIENT_HANDLE, (void **)&schMgmt.queryMgmt, NULL)); SCH_ERR_RET(qWorkerInit(NODE_TYPE_CLIENT, CLIENT_HANDLE, (void **)&schMgmt.queryMgmt, NULL));
} }
SArray *explainRes = NULL; SArray *explainRes = NULL;
SQWMsg qwMsg = {0}; SQWMsg qwMsg = {0};
qwMsg.msgInfo.taskType = TASK_TYPE_TEMP; qwMsg.msgInfo.taskType = TASK_TYPE_TEMP;
qwMsg.msgInfo.explain = SCH_IS_EXPLAIN_JOB(pJob); qwMsg.msgInfo.explain = SCH_IS_EXPLAIN_JOB(pJob);
qwMsg.msgInfo.needFetch = SCH_TASK_NEED_FETCH(pTask); qwMsg.msgInfo.needFetch = SCH_TASK_NEED_FETCH(pTask);
@ -934,8 +937,9 @@ int32_t schLaunchLocalTask(SSchJob *pJob, SSchTask *pTask) {
if (SCH_IS_EXPLAIN_JOB(pJob)) { if (SCH_IS_EXPLAIN_JOB(pJob)) {
explainRes = taosArrayInit(pJob->taskNum, sizeof(SExplainLocalRsp)); explainRes = taosArrayInit(pJob->taskNum, sizeof(SExplainLocalRsp));
} }
SCH_ERR_RET(qWorkerProcessLocalQuery(schMgmt.queryMgmt, schMgmt.sId, pJob->queryId, pTask->taskId, pJob->refId, pTask->execId, &qwMsg, explainRes)); SCH_ERR_RET(qWorkerProcessLocalQuery(schMgmt.queryMgmt, schMgmt.sId, pJob->queryId, pTask->taskId, pJob->refId,
pTask->execId, &qwMsg, explainRes));
if (SCH_IS_EXPLAIN_JOB(pJob)) { if (SCH_IS_EXPLAIN_JOB(pJob)) {
SCH_ERR_RET(schHandleExplainRes(explainRes)); SCH_ERR_RET(schHandleExplainRes(explainRes));
@ -958,17 +962,17 @@ int32_t schLaunchTaskImpl(void *param) {
if (pCtx->asyncLaunch) { if (pCtx->asyncLaunch) {
SCH_LOCK_TASK(pTask); SCH_LOCK_TASK(pTask);
} }
int8_t status = 0; int8_t status = 0;
int32_t code = 0; int32_t code = 0;
atomic_add_fetch_32(&pTask->level->taskLaunchedNum, 1); atomic_add_fetch_32(&pTask->level->taskLaunchedNum, 1);
pTask->execId++; pTask->execId++;
pTask->retryTimes++; pTask->retryTimes++;
pTask->waitRetry = false; pTask->waitRetry = false;
SCH_TASK_DLOG("start to launch %s task, execId %d, retry %d", SCH_IS_LOCAL_EXEC_TASK(pJob, pTask) ? "LOCAL" : "REMOTE", SCH_TASK_DLOG("start to launch %s task, execId %d, retry %d",
pTask->execId, pTask->retryTimes); SCH_IS_LOCAL_EXEC_TASK(pJob, pTask) ? "LOCAL" : "REMOTE", pTask->execId, pTask->retryTimes);
SCH_LOG_TASK_START_TS(pTask); SCH_LOG_TASK_START_TS(pTask);
@ -1086,19 +1090,20 @@ int32_t schExecRemoteFetch(SSchJob *pJob, SSchTask *pTask) {
} }
int32_t schExecLocalFetch(SSchJob *pJob, SSchTask *pTask) { int32_t schExecLocalFetch(SSchJob *pJob, SSchTask *pTask) {
void *pRsp = NULL; void *pRsp = NULL;
SArray *explainRes = NULL; SArray *explainRes = NULL;
if (SCH_IS_EXPLAIN_JOB(pJob)) { if (SCH_IS_EXPLAIN_JOB(pJob)) {
explainRes = taosArrayInit(pJob->taskNum, sizeof(SExplainLocalRsp)); explainRes = taosArrayInit(pJob->taskNum, sizeof(SExplainLocalRsp));
} }
SCH_ERR_RET(qWorkerProcessLocalFetch(schMgmt.queryMgmt, schMgmt.sId, pJob->queryId, pTask->taskId, pJob->refId, pTask->execId, &pRsp, explainRes)); SCH_ERR_RET(qWorkerProcessLocalFetch(schMgmt.queryMgmt, schMgmt.sId, pJob->queryId, pTask->taskId, pJob->refId,
pTask->execId, &pRsp, explainRes));
if (SCH_IS_EXPLAIN_JOB(pJob)) { if (SCH_IS_EXPLAIN_JOB(pJob)) {
SCH_ERR_RET(schHandleExplainRes(explainRes)); SCH_ERR_RET(schHandleExplainRes(explainRes));
} }
SCH_ERR_RET(schProcessFetchRsp(pJob, pTask, pRsp, TSDB_CODE_SUCCESS)); SCH_ERR_RET(schProcessFetchRsp(pJob, pTask, pRsp, TSDB_CODE_SUCCESS));
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;

53
source/libs/scheduler/src/schUtil.c
View File

@ -21,21 +21,21 @@
#include "tref.h" #include "tref.h"
#include "trpc.h" #include "trpc.h"
FORCE_INLINE SSchJob *schAcquireJob(int64_t refId) { FORCE_INLINE SSchJob *schAcquireJob(int64_t refId) {
qDebug("sch acquire jobId:0x%"PRIx64, refId); qDebug("sch acquire jobId:0x%" PRIx64, refId);
return (SSchJob *)taosAcquireRef(schMgmt.jobRef, refId); return (SSchJob *)taosAcquireRef(schMgmt.jobRef, refId);
} }
FORCE_INLINE int32_t schReleaseJob(int64_t refId) { FORCE_INLINE int32_t schReleaseJob(int64_t refId) {
if (0 == refId) { if (0 == refId) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
qDebug("sch release jobId:0x%"PRIx64, refId); qDebug("sch release jobId:0x%" PRIx64, refId);
return taosReleaseRef(schMgmt.jobRef, refId); return taosReleaseRef(schMgmt.jobRef, refId);
} }
char* schGetOpStr(SCH_OP_TYPE type) { char *schGetOpStr(SCH_OP_TYPE type) {
switch (type) { switch (type) {
case SCH_OP_NULL: case SCH_OP_NULL:
return "NULL"; return "NULL";
@ -53,28 +53,28 @@ char* schGetOpStr(SCH_OP_TYPE type) {
void schFreeHbTrans(SSchHbTrans *pTrans) { void schFreeHbTrans(SSchHbTrans *pTrans) {
rpcReleaseHandle(pTrans->trans.pHandle, TAOS_CONN_CLIENT); rpcReleaseHandle(pTrans->trans.pHandle, TAOS_CONN_CLIENT);
schFreeRpcCtx(&pTrans->rpcCtx); schFreeRpcCtx(&pTrans->rpcCtx);
} }
void schCleanClusterHb(void* pTrans) { void schCleanClusterHb(void *pTrans) {
SCH_LOCK(SCH_WRITE, &schMgmt.hbLock); SCH_LOCK(SCH_WRITE, &schMgmt.hbLock);
SSchHbTrans *hb = taosHashIterate(schMgmt.hbConnections, NULL); SSchHbTrans *hb = taosHashIterate(schMgmt.hbConnections, NULL);
while (hb) { while (hb) {
if (hb->trans.pTrans == pTrans) { if (hb->trans.pTrans == pTrans) {
SQueryNodeEpId* pEpId = taosHashGetKey(hb, NULL); SQueryNodeEpId *pEpId = taosHashGetKey(hb, NULL);
schFreeHbTrans(hb); schFreeHbTrans(hb);
taosHashRemove(schMgmt.hbConnections, pEpId, sizeof(SQueryNodeEpId)); taosHashRemove(schMgmt.hbConnections, pEpId, sizeof(SQueryNodeEpId));
} }
hb = taosHashIterate(schMgmt.hbConnections, hb); hb = taosHashIterate(schMgmt.hbConnections, hb);
} }
SCH_UNLOCK(SCH_WRITE, &schMgmt.hbLock); SCH_UNLOCK(SCH_WRITE, &schMgmt.hbLock);
} }
int32_t schRemoveHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *epId) { int32_t schRemoveHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *epId) {
int32_t code = 0; int32_t code = 0;
SCH_LOCK(SCH_WRITE, &schMgmt.hbLock); SCH_LOCK(SCH_WRITE, &schMgmt.hbLock);
SSchHbTrans *hb = taosHashGet(schMgmt.hbConnections, epId, sizeof(SQueryNodeEpId)); SSchHbTrans *hb = taosHashGet(schMgmt.hbConnections, epId, sizeof(SQueryNodeEpId));
@ -94,7 +94,6 @@ int32_t schRemoveHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *ep
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t schAddHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *epId, bool *exist) { int32_t schAddHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *epId, bool *exist) {
int32_t code = 0; int32_t code = 0;
SSchHbTrans hb = {0}; SSchHbTrans hb = {0};
@ -155,13 +154,13 @@ void schDeregisterTaskHb(SSchJob *pJob, SSchTask *pTask) {
if (!pTask->registerdHb) { if (!pTask->registerdHb) {
return; return;
} }
SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx); SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx);
SQueryNodeEpId epId = {0}; SQueryNodeEpId epId = {0};
epId.nodeId = addr->nodeId; epId.nodeId = addr->nodeId;
SEp* pEp = SCH_GET_CUR_EP(addr); SEp *pEp = SCH_GET_CUR_EP(addr);
strcpy(epId.ep.fqdn, pEp->fqdn); strcpy(epId.ep.fqdn, pEp->fqdn);
epId.ep.port = pEp->port; epId.ep.port = pEp->port;
@ -180,24 +179,22 @@ void schDeregisterTaskHb(SSchJob *pJob, SSchTask *pTask) {
} else { } else {
SCH_UNLOCK(SCH_READ, &schMgmt.hbLock); SCH_UNLOCK(SCH_READ, &schMgmt.hbLock);
} }
pTask->registerdHb = false; pTask->registerdHb = false;
} }
int32_t schEnsureHbConnection(SSchJob *pJob, SSchTask *pTask) { int32_t schEnsureHbConnection(SSchJob *pJob, SSchTask *pTask) {
SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx); SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx);
SQueryNodeEpId epId = {0}; SQueryNodeEpId epId = {0};
epId.nodeId = addr->nodeId; epId.nodeId = addr->nodeId;
SEp* pEp = SCH_GET_CUR_EP(addr); SEp *pEp = SCH_GET_CUR_EP(addr);
strcpy(epId.ep.fqdn, pEp->fqdn); strcpy(epId.ep.fqdn, pEp->fqdn);
epId.ep.port = pEp->port; epId.ep.port = pEp->port;
SCH_ERR_RET(schRegisterHbConnection(pJob, pTask, &epId)); SCH_ERR_RET(schRegisterHbConnection(pJob, pTask, &epId));
pTask->registerdHb = true; pTask->registerdHb = true;
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -226,7 +223,6 @@ int32_t schUpdateHbConnection(SQueryNodeEpId *epId, SSchTrans *trans) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
void schCloseJobRef(void) { void schCloseJobRef(void) {
if (!atomic_load_8((int8_t *)&schMgmt.exit)) { if (!atomic_load_8((int8_t *)&schMgmt.exit)) {
return; return;
@ -242,7 +238,7 @@ uint64_t schGenTaskId(void) { return atomic_add_fetch_64(&schMgmt.taskId, 1); }
uint64_t schGenUUID(void) { uint64_t schGenUUID(void) {
static uint64_t hashId = 0; static uint64_t hashId = 0;
static int32_t requestSerialId = 0; static int32_t requestSerialId = 0;
if (hashId == 0) { if (hashId == 0) {
char uid[64] = {0}; char uid[64] = {0};
@ -254,15 +250,14 @@ uint64_t schGenUUID(void) {
} }
} }
int64_t ts = taosGetTimestampMs(); int64_t ts = taosGetTimestampMs();
uint64_t pid = taosGetPId(); uint64_t pid = taosGetPId();
int32_t val = atomic_add_fetch_32(&requestSerialId, 1); int32_t val = atomic_add_fetch_32(&requestSerialId, 1);
uint64_t id = ((hashId & 0x0FFF) << 52) | ((pid & 0x0FFF) << 40) | ((ts & 0xFFFFFF) << 16) | (val & 0xFFFF); uint64_t id = ((hashId & 0x0FFF) << 52) | ((pid & 0x0FFF) << 40) | ((ts & 0xFFFFFF) << 16) | (val & 0xFFFF);
return id; return id;
} }
void schFreeRpcCtxVal(const void *arg) { void schFreeRpcCtxVal(const void *arg) {
if (NULL == arg) { if (NULL == arg) {
return; return;
@ -307,5 +302,3 @@ int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTas
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }

12
source/libs/scheduler/src/scheduler.c
View File

@ -14,10 +14,10 @@
*/ */
#include "query.h" #include "query.h"
#include "qworker.h"
#include "schInt.h" #include "schInt.h"
#include "tmsg.h" #include "tmsg.h"
#include "tref.h" #include "tref.h"
#include "qworker.h"
SSchedulerMgmt schMgmt = { SSchedulerMgmt schMgmt = {
.jobRef = -1, .jobRef = -1,
@ -35,7 +35,7 @@ int32_t schedulerInit() {
schMgmt.cfg.enableReSchedule = true; schMgmt.cfg.enableReSchedule = true;
qDebug("schedule policy init to %d", schMgmt.cfg.schPolicy); qDebug("schedule policy init to %d", schMgmt.cfg.schPolicy);
schMgmt.jobRef = taosOpenRef(schMgmt.cfg.maxJobNum, schFreeJobImpl); schMgmt.jobRef = taosOpenRef(schMgmt.cfg.maxJobNum, schFreeJobImpl);
if (schMgmt.jobRef < 0) { if (schMgmt.jobRef < 0) {
qError("init schduler jobRef failed, num:%u", schMgmt.cfg.maxJobNum); qError("init schduler jobRef failed, num:%u", schMgmt.cfg.maxJobNum);
@ -61,7 +61,7 @@ int32_t schedulerInit() {
int32_t schedulerExecJob(SSchedulerReq *pReq, int64_t *pJobId) { int32_t schedulerExecJob(SSchedulerReq *pReq, int64_t *pJobId) {
qDebug("scheduler %s exec job start", pReq->syncReq ? "SYNC" : "ASYNC"); qDebug("scheduler %s exec job start", pReq->syncReq ? "SYNC" : "ASYNC");
int32_t code = 0; int32_t code = 0;
SSchJob *pJob = NULL; SSchJob *pJob = NULL;
SCH_ERR_JRET(schInitJob(pJobId, pReq)); SCH_ERR_JRET(schInitJob(pJobId, pReq));
@ -73,7 +73,7 @@ int32_t schedulerExecJob(SSchedulerReq *pReq, int64_t *pJobId) {
SCH_ERR_JRET(schSwitchJobStatus(pJob, JOB_TASK_STATUS_EXEC, pReq)); SCH_ERR_JRET(schSwitchJobStatus(pJob, JOB_TASK_STATUS_EXEC, pReq));
_return: _return:
SCH_RET(schHandleOpEndEvent(pJob, SCH_OP_EXEC, pReq, code)); SCH_RET(schHandleOpEndEvent(pJob, SCH_OP_EXEC, pReq, code));
} }
@ -144,7 +144,7 @@ int32_t schedulerEnableReSchedule(bool enableResche) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
void schedulerFreeJob(int64_t* jobId, int32_t errCode) { void schedulerFreeJob(int64_t *jobId, int32_t errCode) {
if (0 == *jobId) { if (0 == *jobId) {
return; return;
} }
@ -158,7 +158,7 @@ void schedulerFreeJob(int64_t* jobId, int32_t errCode) {
SCH_JOB_DLOG("start to free job 0x%" PRIx64 ", errCode:0x%x", *jobId, errCode); SCH_JOB_DLOG("start to free job 0x%" PRIx64 ", errCode:0x%x", *jobId, errCode);
schHandleJobDrop(pJob, errCode); schHandleJobDrop(pJob, errCode);
schReleaseJob(*jobId); schReleaseJob(*jobId);
*jobId = 0; *jobId = 0;
} }

382
source/libs/scheduler/test/schedulerTests.cpp
View File

@ -30,15 +30,12 @@
#endif #endif
#include "os.h" #include "os.h"
#include "tglobal.h"
#include "taos.h"
#include "tdef.h"
#include "tvariant.h"
#include "catalog.h" #include "catalog.h"
#include "scheduler.h" #include "scheduler.h"
#include "taos.h" #include "taos.h"
#include "tdatablock.h" #include "tdatablock.h"
#include "tdef.h" #include "tdef.h"
#include "tglobal.h"
#include "trpc.h" #include "trpc.h"
#include "tvariant.h" #include "tvariant.h"
@ -56,8 +53,9 @@
namespace { namespace {
extern "C" int32_t schHandleResponseMsg(SSchJob *job, SSchTask *task, int32_t msgType, char *msg, int32_t msgSize, int32_t rspCode); extern "C" int32_t schHandleResponseMsg(SSchJob *job, SSchTask *task, int32_t msgType, char *msg, int32_t msgSize,
extern "C" int32_t schHandleCallback(void* param, const SDataBuf* pMsg, int32_t msgType, int32_t rspCode); int32_t rspCode);
extern "C" int32_t schHandleCallback(void *param, const SDataBuf *pMsg, int32_t msgType, int32_t rspCode);
int64_t insertJobRefId = 0; int64_t insertJobRefId = 0;
int64_t queryJobRefId = 0; int64_t queryJobRefId = 0;
@ -66,16 +64,15 @@ uint64_t schtMergeTemplateId = 0x4;
uint64_t schtFetchTaskId = 0; uint64_t schtFetchTaskId = 0;
uint64_t schtQueryId = 1; uint64_t schtQueryId = 1;
bool schtTestStop = false; bool schtTestStop = false;
bool schtTestDeadLoop = false; bool schtTestDeadLoop = false;
int32_t schtTestMTRunSec = 10; int32_t schtTestMTRunSec = 10;
int32_t schtTestPrintNum = 1000; int32_t schtTestPrintNum = 1000;
int32_t schtStartFetch = 0; int32_t schtStartFetch = 0;
void schtInitLogFile() { void schtInitLogFile() {
const char *defaultLogFileNamePrefix = "taoslog"; const char *defaultLogFileNamePrefix = "taoslog";
const int32_t maxLogFileNum = 10; const int32_t maxLogFileNum = 10;
tsAsyncLog = 0; tsAsyncLog = 0;
qDebugFlag = 159; qDebugFlag = 159;
@ -84,24 +81,22 @@ void schtInitLogFile() {
if (taosInitLog(defaultLogFileNamePrefix, maxLogFileNum) < 0) { if (taosInitLog(defaultLogFileNamePrefix, maxLogFileNum) < 0) {
printf("failed to open log file in directory:%s\n", tsLogDir); printf("failed to open log file in directory:%s\n", tsLogDir);
} }
} }
void schtQueryCb(SExecResult* pResult, void* param, int32_t code) { void schtQueryCb(SExecResult *pResult, void *param, int32_t code) {
assert(TSDB_CODE_SUCCESS == code); assert(TSDB_CODE_SUCCESS == code);
*(int32_t*)param = 1; *(int32_t *)param = 1;
} }
void schtBuildQueryDag(SQueryPlan *dag) { void schtBuildQueryDag(SQueryPlan *dag) {
uint64_t qId = schtQueryId; uint64_t qId = schtQueryId;
dag->queryId = qId; dag->queryId = qId;
dag->numOfSubplans = 2; dag->numOfSubplans = 2;
dag->pSubplans = nodesMakeList(); dag->pSubplans = nodesMakeList();
SNodeListNode *scan = (SNodeListNode*)nodesMakeNode(QUERY_NODE_NODE_LIST); SNodeListNode *scan = (SNodeListNode *)nodesMakeNode(QUERY_NODE_NODE_LIST);
SNodeListNode *merge = (SNodeListNode*)nodesMakeNode(QUERY_NODE_NODE_LIST); SNodeListNode *merge = (SNodeListNode *)nodesMakeNode(QUERY_NODE_NODE_LIST);
SSubplan *scanPlan = (SSubplan *)taosMemoryCalloc(1, sizeof(SSubplan)); SSubplan *scanPlan = (SSubplan *)taosMemoryCalloc(1, sizeof(SSubplan));
SSubplan *mergePlan = (SSubplan *)taosMemoryCalloc(1, sizeof(SSubplan)); SSubplan *mergePlan = (SSubplan *)taosMemoryCalloc(1, sizeof(SSubplan));
@ -117,7 +112,7 @@ void schtBuildQueryDag(SQueryPlan *dag) {
scanPlan->pChildren = NULL; scanPlan->pChildren = NULL;
scanPlan->level = 1; scanPlan->level = 1;
scanPlan->pParents = nodesMakeList(); scanPlan->pParents = nodesMakeList();
scanPlan->pNode = (SPhysiNode*)taosMemoryCalloc(1, sizeof(SPhysiNode)); scanPlan->pNode = (SPhysiNode *)taosMemoryCalloc(1, sizeof(SPhysiNode));
scanPlan->msgType = TDMT_SCH_QUERY; scanPlan->msgType = TDMT_SCH_QUERY;
mergePlan->id.queryId = qId; mergePlan->id.queryId = qId;
@ -129,32 +124,32 @@ void schtBuildQueryDag(SQueryPlan *dag) {
mergePlan->pChildren = nodesMakeList(); mergePlan->pChildren = nodesMakeList();
mergePlan->pParents = NULL; mergePlan->pParents = NULL;
mergePlan->pNode = (SPhysiNode*)taosMemoryCalloc(1, sizeof(SPhysiNode)); mergePlan->pNode = (SPhysiNode *)taosMemoryCalloc(1, sizeof(SPhysiNode));
mergePlan->msgType = TDMT_SCH_QUERY; mergePlan->msgType = TDMT_SCH_QUERY;
merge->pNodeList = nodesMakeList(); merge->pNodeList = nodesMakeList();
scan->pNodeList = nodesMakeList(); scan->pNodeList = nodesMakeList();
nodesListAppend(merge->pNodeList, (SNode*)mergePlan); nodesListAppend(merge->pNodeList, (SNode *)mergePlan);
nodesListAppend(scan->pNodeList, (SNode*)scanPlan); nodesListAppend(scan->pNodeList, (SNode *)scanPlan);
nodesListAppend(mergePlan->pChildren, (SNode*)scanPlan); nodesListAppend(mergePlan->pChildren, (SNode *)scanPlan);
nodesListAppend(scanPlan->pParents, (SNode*)mergePlan); nodesListAppend(scanPlan->pParents, (SNode *)mergePlan);
nodesListAppend(dag->pSubplans, (SNode*)merge); nodesListAppend(dag->pSubplans, (SNode *)merge);
nodesListAppend(dag->pSubplans, (SNode*)scan); nodesListAppend(dag->pSubplans, (SNode *)scan);
} }
void schtBuildQueryFlowCtrlDag(SQueryPlan *dag) { void schtBuildQueryFlowCtrlDag(SQueryPlan *dag) {
uint64_t qId = schtQueryId; uint64_t qId = schtQueryId;
int32_t scanPlanNum = 20; int32_t scanPlanNum = 20;
dag->queryId = qId; dag->queryId = qId;
dag->numOfSubplans = 2; dag->numOfSubplans = 2;
dag->pSubplans = nodesMakeList(); dag->pSubplans = nodesMakeList();
SNodeListNode *scan = (SNodeListNode*)nodesMakeNode(QUERY_NODE_NODE_LIST); SNodeListNode *scan = (SNodeListNode *)nodesMakeNode(QUERY_NODE_NODE_LIST);
SNodeListNode *merge = (SNodeListNode*)nodesMakeNode(QUERY_NODE_NODE_LIST); SNodeListNode *merge = (SNodeListNode *)nodesMakeNode(QUERY_NODE_NODE_LIST);
SSubplan *scanPlan = (SSubplan *)taosMemoryCalloc(scanPlanNum, sizeof(SSubplan)); SSubplan *scanPlan = (SSubplan *)taosMemoryCalloc(scanPlanNum, sizeof(SSubplan));
SSubplan *mergePlan = (SSubplan *)taosMemoryCalloc(1, sizeof(SSubplan)); SSubplan *mergePlan = (SSubplan *)taosMemoryCalloc(1, sizeof(SSubplan));
@ -180,13 +175,13 @@ void schtBuildQueryFlowCtrlDag(SQueryPlan *dag) {
scanPlan[i].pChildren = NULL; scanPlan[i].pChildren = NULL;
scanPlan[i].level = 1; scanPlan[i].level = 1;
scanPlan[i].pParents = nodesMakeList(); scanPlan[i].pParents = nodesMakeList();
scanPlan[i].pNode = (SPhysiNode*)taosMemoryCalloc(1, sizeof(SPhysiNode)); scanPlan[i].pNode = (SPhysiNode *)taosMemoryCalloc(1, sizeof(SPhysiNode));
scanPlan[i].msgType = TDMT_SCH_QUERY; scanPlan[i].msgType = TDMT_SCH_QUERY;
nodesListAppend(scanPlan[i].pParents, (SNode*)mergePlan); nodesListAppend(scanPlan[i].pParents, (SNode *)mergePlan);
nodesListAppend(mergePlan->pChildren, (SNode*)(scanPlan + i)); nodesListAppend(mergePlan->pChildren, (SNode *)(scanPlan + i));
nodesListAppend(scan->pNodeList, (SNode*)(scanPlan + i)); nodesListAppend(scan->pNodeList, (SNode *)(scanPlan + i));
} }
mergePlan->id.queryId = qId; mergePlan->id.queryId = qId;
@ -197,29 +192,25 @@ void schtBuildQueryFlowCtrlDag(SQueryPlan *dag) {
mergePlan->execNode.epSet.numOfEps = 0; mergePlan->execNode.epSet.numOfEps = 0;
mergePlan->pParents = NULL; mergePlan->pParents = NULL;
mergePlan->pNode = (SPhysiNode*)taosMemoryCalloc(1, sizeof(SPhysiNode)); mergePlan->pNode = (SPhysiNode *)taosMemoryCalloc(1, sizeof(SPhysiNode));
mergePlan->msgType = TDMT_SCH_QUERY; mergePlan->msgType = TDMT_SCH_QUERY;
nodesListAppend(merge->pNodeList, (SNode*)mergePlan); nodesListAppend(merge->pNodeList, (SNode *)mergePlan);
nodesListAppend(dag->pSubplans, (SNode*)merge);
nodesListAppend(dag->pSubplans, (SNode*)scan);
}
void schtFreeQueryDag(SQueryPlan *dag) {
nodesListAppend(dag->pSubplans, (SNode *)merge);
nodesListAppend(dag->pSubplans, (SNode *)scan);
} }
void schtFreeQueryDag(SQueryPlan *dag) {}
void schtBuildInsertDag(SQueryPlan *dag) { void schtBuildInsertDag(SQueryPlan *dag) {
uint64_t qId = 0x0000000000000002; uint64_t qId = 0x0000000000000002;
dag->queryId = qId; dag->queryId = qId;
dag->numOfSubplans = 2; dag->numOfSubplans = 2;
dag->pSubplans = nodesMakeList(); dag->pSubplans = nodesMakeList();
SNodeListNode *inserta = (SNodeListNode*)nodesMakeNode(QUERY_NODE_NODE_LIST); SNodeListNode *inserta = (SNodeListNode *)nodesMakeNode(QUERY_NODE_NODE_LIST);
SSubplan *insertPlan = (SSubplan *)taosMemoryCalloc(2, sizeof(SSubplan)); SSubplan *insertPlan = (SSubplan *)taosMemoryCalloc(2, sizeof(SSubplan));
insertPlan[0].id.queryId = qId; insertPlan[0].id.queryId = qId;
@ -235,7 +226,7 @@ void schtBuildInsertDag(SQueryPlan *dag) {
insertPlan[0].pChildren = NULL; insertPlan[0].pChildren = NULL;
insertPlan[0].pParents = NULL; insertPlan[0].pParents = NULL;
insertPlan[0].pNode = NULL; insertPlan[0].pNode = NULL;
insertPlan[0].pDataSink = (SDataSinkNode*)taosMemoryCalloc(1, sizeof(SDataSinkNode)); insertPlan[0].pDataSink = (SDataSinkNode *)taosMemoryCalloc(1, sizeof(SDataSinkNode));
insertPlan[0].msgType = TDMT_VND_SUBMIT; insertPlan[0].msgType = TDMT_VND_SUBMIT;
insertPlan[1].id.queryId = qId; insertPlan[1].id.queryId = qId;
@ -251,48 +242,43 @@ void schtBuildInsertDag(SQueryPlan *dag) {
insertPlan[1].pChildren = NULL; insertPlan[1].pChildren = NULL;
insertPlan[1].pParents = NULL; insertPlan[1].pParents = NULL;
insertPlan[1].pNode = NULL; insertPlan[1].pNode = NULL;
insertPlan[1].pDataSink = (SDataSinkNode*)taosMemoryCalloc(1, sizeof(SDataSinkNode)); insertPlan[1].pDataSink = (SDataSinkNode *)taosMemoryCalloc(1, sizeof(SDataSinkNode));
insertPlan[1].msgType = TDMT_VND_SUBMIT; insertPlan[1].msgType = TDMT_VND_SUBMIT;
inserta->pNodeList = nodesMakeList(); inserta->pNodeList = nodesMakeList();
nodesListAppend(inserta->pNodeList, (SNode*)insertPlan); nodesListAppend(inserta->pNodeList, (SNode *)insertPlan);
insertPlan += 1; insertPlan += 1;
nodesListAppend(inserta->pNodeList, (SNode*)insertPlan); nodesListAppend(inserta->pNodeList, (SNode *)insertPlan);
nodesListAppend(dag->pSubplans, (SNode*)inserta); nodesListAppend(dag->pSubplans, (SNode *)inserta);
} }
int32_t schtPlanToString(const SSubplan *subplan, char **str, int32_t *len) {
int32_t schtPlanToString(const SSubplan *subplan, char** str, int32_t* len) {
*str = (char *)taosMemoryCalloc(1, 20); *str = (char *)taosMemoryCalloc(1, 20);
*len = 20; *len = 20;
return 0; return 0;
} }
void schtExecNode(SSubplan* subplan, uint64_t groupId, SQueryNodeAddr* ep) { void schtExecNode(SSubplan *subplan, uint64_t groupId, SQueryNodeAddr *ep) {}
} void schtRpcSendRequest(void *shandle, const SEpSet *pEpSet, SRpcMsg *pMsg, int64_t *pRid) {}
void schtRpcSendRequest(void *shandle, const SEpSet *pEpSet, SRpcMsg *pMsg, int64_t *pRid) {
}
void schtSetPlanToString() { void schtSetPlanToString() {
static Stub stub; static Stub stub;
stub.set(qSubPlanToString, schtPlanToString); stub.set(qSubPlanToString, schtPlanToString);
{ {
#ifdef WINDOWS #ifdef WINDOWS
AddrAny any; AddrAny any;
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_func_addr("qSubPlanToString", result); any.get_func_addr("qSubPlanToString", result);
#endif #endif
#ifdef LINUX #ifdef LINUX
AddrAny any("libplanner.so"); AddrAny any("libplanner.so");
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_global_func_addr_dynsym("^qSubPlanToString$", result); any.get_global_func_addr_dynsym("^qSubPlanToString$", result);
#endif #endif
for (const auto& f : result) { for (const auto &f : result) {
stub.set(f.second, schtPlanToString); stub.set(f.second, schtPlanToString);
} }
} }
@ -303,16 +289,16 @@ void schtSetExecNode() {
stub.set(qSetSubplanExecutionNode, schtExecNode); stub.set(qSetSubplanExecutionNode, schtExecNode);
{ {
#ifdef WINDOWS #ifdef WINDOWS
AddrAny any; AddrAny any;
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_func_addr("qSetSubplanExecutionNode", result); any.get_func_addr("qSetSubplanExecutionNode", result);
#endif #endif
#ifdef LINUX #ifdef LINUX
AddrAny any("libplanner.so"); AddrAny any("libplanner.so");
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_global_func_addr_dynsym("^qSetSubplanExecutionNode$", result); any.get_global_func_addr_dynsym("^qSetSubplanExecutionNode$", result);
#endif #endif
for (const auto& f : result) { for (const auto &f : result) {
stub.set(f.second, schtExecNode); stub.set(f.second, schtExecNode);
} }
} }
@ -323,22 +309,22 @@ void schtSetRpcSendRequest() {
stub.set(rpcSendRequest, schtRpcSendRequest); stub.set(rpcSendRequest, schtRpcSendRequest);
{ {
#ifdef WINDOWS #ifdef WINDOWS
AddrAny any; AddrAny any;
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_func_addr("rpcSendRequest", result); any.get_func_addr("rpcSendRequest", result);
#endif #endif
#ifdef LINUX #ifdef LINUX
AddrAny any("libtransport.so"); AddrAny any("libtransport.so");
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_global_func_addr_dynsym("^rpcSendRequest$", result); any.get_global_func_addr_dynsym("^rpcSendRequest$", result);
#endif #endif
for (const auto& f : result) { for (const auto &f : result) {
stub.set(f.second, schtRpcSendRequest); stub.set(f.second, schtRpcSendRequest);
} }
} }
} }
int32_t schtAsyncSendMsgToServer(void *pTransporter, SEpSet* epSet, int64_t* pTransporterId, SMsgSendInfo* pInfo) { int32_t schtAsyncSendMsgToServer(void *pTransporter, SEpSet *epSet, int64_t *pTransporterId, SMsgSendInfo *pInfo) {
if (pInfo) { if (pInfo) {
taosMemoryFreeClear(pInfo->param); taosMemoryFreeClear(pInfo->param);
taosMemoryFreeClear(pInfo->msgInfo.pData); taosMemoryFreeClear(pInfo->msgInfo.pData);
@ -347,32 +333,30 @@ int32_t schtAsyncSendMsgToServer(void *pTransporter, SEpSet* epSet, int64_t* pTr
return 0; return 0;
} }
void schtSetAsyncSendMsgToServer() { void schtSetAsyncSendMsgToServer() {
static Stub stub; static Stub stub;
stub.set(asyncSendMsgToServer, schtAsyncSendMsgToServer); stub.set(asyncSendMsgToServer, schtAsyncSendMsgToServer);
{ {
#ifdef WINDOWS #ifdef WINDOWS
AddrAny any; AddrAny any;
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_func_addr("asyncSendMsgToServer", result); any.get_func_addr("asyncSendMsgToServer", result);
#endif #endif
#ifdef LINUX #ifdef LINUX
AddrAny any("libtransport.so"); AddrAny any("libtransport.so");
std::map<std::string,void*> result; std::map<std::string, void *> result;
any.get_global_func_addr_dynsym("^asyncSendMsgToServer$", result); any.get_global_func_addr_dynsym("^asyncSendMsgToServer$", result);
#endif #endif
for (const auto& f : result) { for (const auto &f : result) {
stub.set(f.second, schtAsyncSendMsgToServer); stub.set(f.second, schtAsyncSendMsgToServer);
} }
} }
} }
void *schtSendRsp(void *param) { void *schtSendRsp(void *param) {
SSchJob *pJob = NULL; SSchJob *pJob = NULL;
int64_t job = 0; int64_t job = 0;
int32_t code = 0; int32_t code = 0;
while (true) { while (true) {
job = *(int64_t *)param; job = *(int64_t *)param;
@ -384,7 +368,7 @@ void *schtSendRsp(void *param) {
} }
pJob = schAcquireJob(job); pJob = schAcquireJob(job);
void *pIter = taosHashIterate(pJob->execTasks, NULL); void *pIter = taosHashIterate(pJob->execTasks, NULL);
while (pIter) { while (pIter) {
SSchTask *task = *(SSchTask **)pIter; SSchTask *task = *(SSchTask **)pIter;
@ -392,9 +376,9 @@ void *schtSendRsp(void *param) {
SSubmitRsp rsp = {0}; SSubmitRsp rsp = {0};
rsp.affectedRows = 10; rsp.affectedRows = 10;
schHandleResponseMsg(pJob, task, TDMT_VND_SUBMIT_RSP, (char *)&rsp, sizeof(rsp), 0); schHandleResponseMsg(pJob, task, TDMT_VND_SUBMIT_RSP, (char *)&rsp, sizeof(rsp), 0);
pIter = taosHashIterate(pJob->execTasks, pIter); pIter = taosHashIterate(pJob->execTasks, pIter);
} }
schReleaseJob(job); schReleaseJob(job);
@ -402,28 +386,27 @@ void *schtSendRsp(void *param) {
} }
void *schtCreateFetchRspThread(void *param) { void *schtCreateFetchRspThread(void *param) {
int64_t job = *(int64_t *)param; int64_t job = *(int64_t *)param;
SSchJob* pJob = schAcquireJob(job); SSchJob *pJob = schAcquireJob(job);
taosSsleep(1); taosSsleep(1);
int32_t code = 0; int32_t code = 0;
SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, sizeof(SRetrieveTableRsp)); SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, sizeof(SRetrieveTableRsp));
rsp->completed = 1; rsp->completed = 1;
rsp->numOfRows = 10; rsp->numOfRows = 10;
code = schHandleResponseMsg(pJob, pJob->fetchTask, TDMT_SCH_FETCH_RSP, (char *)rsp, sizeof(*rsp), 0); code = schHandleResponseMsg(pJob, pJob->fetchTask, TDMT_SCH_FETCH_RSP, (char *)rsp, sizeof(*rsp), 0);
schReleaseJob(job); schReleaseJob(job);
assert(code == 0); assert(code == 0);
return NULL; return NULL;
} }
void *schtFetchRspThread(void *aa) { void *schtFetchRspThread(void *aa) {
SDataBuf dataBuf = {0}; SDataBuf dataBuf = {0};
SSchTaskCallbackParam* param = NULL; SSchTaskCallbackParam *param = NULL;
while (!schtTestStop) { while (!schtTestStop) {
if (0 == atomic_val_compare_exchange_32(&schtStartFetch, 1, 0)) { if (0 == atomic_val_compare_exchange_32(&schtStartFetch, 1, 0)) {
@ -431,13 +414,13 @@ void *schtFetchRspThread(void *aa) {
} }
taosUsleep(1); taosUsleep(1);
param = (SSchTaskCallbackParam *)taosMemoryCalloc(1, sizeof(*param)); param = (SSchTaskCallbackParam *)taosMemoryCalloc(1, sizeof(*param));
param->queryId = schtQueryId; param->queryId = schtQueryId;
param->taskId = schtFetchTaskId; param->taskId = schtFetchTaskId;
int32_t code = 0; int32_t code = 0;
SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, sizeof(SRetrieveTableRsp)); SRetrieveTableRsp *rsp = (SRetrieveTableRsp *)taosMemoryCalloc(1, sizeof(SRetrieveTableRsp));
rsp->completed = 1; rsp->completed = 1;
rsp->numOfRows = 10; rsp->numOfRows = 10;
@ -446,7 +429,7 @@ void *schtFetchRspThread(void *aa) {
dataBuf.len = sizeof(*rsp); dataBuf.len = sizeof(*rsp);
code = schHandleCallback(param, &dataBuf, TDMT_SCH_FETCH_RSP, 0); code = schHandleCallback(param, &dataBuf, TDMT_SCH_FETCH_RSP, 0);
assert(code == 0 || code); assert(code == 0 || code);
} }
return NULL; return NULL;
@ -454,8 +437,8 @@ void *schtFetchRspThread(void *aa) {
void schtFreeQueryJob(int32_t freeThread) { void schtFreeQueryJob(int32_t freeThread) {
static uint32_t freeNum = 0; static uint32_t freeNum = 0;
int64_t job = queryJobRefId; int64_t job = queryJobRefId;
if (job && atomic_val_compare_exchange_64(&queryJobRefId, job, 0)) { if (job && atomic_val_compare_exchange_64(&queryJobRefId, job, 0)) {
schedulerFreeJob(&job, 0); schedulerFreeJob(&job, 0);
if (freeThread) { if (freeThread) {
@ -466,31 +449,29 @@ void schtFreeQueryJob(int32_t freeThread) {
} }
} }
void* schtRunJobThread(void *aa) { void *schtRunJobThread(void *aa) {
void *mockPointer = (void *)0x1; void *mockPointer = (void *)0x1;
char *clusterId = "cluster1"; char *clusterId = "cluster1";
char *dbname = "1.db1"; char *dbname = "1.db1";
char *tablename = "table1"; char *tablename = "table1";
SVgroupInfo vgInfo = {0}; SVgroupInfo vgInfo = {0};
SQueryPlan dag; SQueryPlan dag;
schtInitLogFile(); schtInitLogFile();
int32_t code = schedulerInit(); int32_t code = schedulerInit();
assert(code == 0); assert(code == 0);
schtSetPlanToString(); schtSetPlanToString();
schtSetExecNode(); schtSetExecNode();
schtSetAsyncSendMsgToServer(); schtSetAsyncSendMsgToServer();
SSchJob *pJob = NULL; SSchJob *pJob = NULL;
SSchTaskCallbackParam *param = NULL; SSchTaskCallbackParam *param = NULL;
SHashObj *execTasks = NULL; SHashObj *execTasks = NULL;
SDataBuf dataBuf = {0}; SDataBuf dataBuf = {0};
uint32_t jobFinished = 0; uint32_t jobFinished = 0;
int32_t queryDone = 0; int32_t queryDone = 0;
while (!schtTestStop) { while (!schtTestStop) {
schtBuildQueryDag(&dag); schtBuildQueryDag(&dag);
@ -503,10 +484,10 @@ void* schtRunJobThread(void *aa) {
taosArrayPush(qnodeList, &qnodeAddr); taosArrayPush(qnodeList, &qnodeAddr);
queryDone = 0; queryDone = 0;
SRequestConnInfo conn = {0}; SRequestConnInfo conn = {0};
conn.pTrans = mockPointer; conn.pTrans = mockPointer;
SSchedulerReq req = {0}; SSchedulerReq req = {0};
req.syncReq = false; req.syncReq = false;
req.pConn = &conn; req.pConn = &conn;
req.pNodeList = qnodeList; req.pNodeList = qnodeList;
@ -514,8 +495,8 @@ void* schtRunJobThread(void *aa) {
req.sql = "select * from tb"; req.sql = "select * from tb";
req.execFp = schtQueryCb; req.execFp = schtQueryCb;
req.cbParam = &queryDone; req.cbParam = &queryDone;
code = schedulerExecJob(&req, &queryJobRefId); code = schedulerExecJob(&req, &queryJobRefId);
assert(code == 0); assert(code == 0);
pJob = schAcquireJob(queryJobRefId); pJob = schAcquireJob(queryJobRefId);
@ -524,20 +505,20 @@ void* schtRunJobThread(void *aa) {
schtFreeQueryDag(&dag); schtFreeQueryDag(&dag);
continue; continue;
} }
execTasks = taosHashInit(5, taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT), false, HASH_ENTRY_LOCK); execTasks = taosHashInit(5, taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT), false, HASH_ENTRY_LOCK);
void *pIter = taosHashIterate(pJob->execTasks, NULL); void *pIter = taosHashIterate(pJob->execTasks, NULL);
while (pIter) { while (pIter) {
SSchTask *task = *(SSchTask **)pIter; SSchTask *task = *(SSchTask **)pIter;
schtFetchTaskId = task->taskId - 1; schtFetchTaskId = task->taskId - 1;
taosHashPut(execTasks, &task->taskId, sizeof(task->taskId), task, sizeof(*task)); taosHashPut(execTasks, &task->taskId, sizeof(task->taskId), task, sizeof(*task));
pIter = taosHashIterate(pJob->execTasks, pIter); pIter = taosHashIterate(pJob->execTasks, pIter);
} }
param = (SSchTaskCallbackParam *)taosMemoryCalloc(1, sizeof(*param)); param = (SSchTaskCallbackParam *)taosMemoryCalloc(1, sizeof(*param));
param->refId = queryJobRefId; param->refId = queryJobRefId;
param->queryId = pJob->queryId; param->queryId = pJob->queryId;
pIter = taosHashIterate(execTasks, NULL); pIter = taosHashIterate(execTasks, NULL);
while (pIter) { while (pIter) {
@ -547,16 +528,16 @@ void* schtRunJobThread(void *aa) {
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
dataBuf.pData = &rsp; dataBuf.pData = &rsp;
dataBuf.len = sizeof(rsp); dataBuf.len = sizeof(rsp);
code = schHandleCallback(param, &dataBuf, TDMT_SCH_QUERY_RSP, 0); code = schHandleCallback(param, &dataBuf, TDMT_SCH_QUERY_RSP, 0);
assert(code == 0 || code); assert(code == 0 || code);
pIter = taosHashIterate(execTasks, pIter); pIter = taosHashIterate(execTasks, pIter);
} }
param = (SSchTaskCallbackParam *)taosMemoryCalloc(1, sizeof(*param)); param = (SSchTaskCallbackParam *)taosMemoryCalloc(1, sizeof(*param));
param->refId = queryJobRefId; param->refId = queryJobRefId;
param->queryId = pJob->queryId; param->queryId = pJob->queryId;
pIter = taosHashIterate(execTasks, NULL); pIter = taosHashIterate(execTasks, NULL);
while (pIter) { while (pIter) {
@ -566,28 +547,27 @@ void* schtRunJobThread(void *aa) {
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
dataBuf.pData = &rsp; dataBuf.pData = &rsp;
dataBuf.len = sizeof(rsp); dataBuf.len = sizeof(rsp);
code = schHandleCallback(param, &dataBuf, TDMT_SCH_QUERY_RSP, 0); code = schHandleCallback(param, &dataBuf, TDMT_SCH_QUERY_RSP, 0);
assert(code == 0 || code); assert(code == 0 || code);
pIter = taosHashIterate(execTasks, pIter);
}
pIter = taosHashIterate(execTasks, pIter);
}
while (true) { while (true) {
if (queryDone) { if (queryDone) {
break; break;
} }
taosUsleep(10000); taosUsleep(10000);
} }
atomic_store_32(&schtStartFetch, 1); atomic_store_32(&schtStartFetch, 1);
void *data = NULL; void *data = NULL;
req.syncReq = true; req.syncReq = true;
req.pFetchRes = &data; req.pFetchRes = &data;
code = schedulerFetchRows(queryJobRefId, &req); code = schedulerFetchRows(queryJobRefId, &req);
assert(code == 0 || code); assert(code == 0 || code);
@ -600,7 +580,7 @@ void* schtRunJobThread(void *aa) {
data = NULL; data = NULL;
code = schedulerFetchRows(queryJobRefId, &req); code = schedulerFetchRows(queryJobRefId, &req);
assert(code == 0 || code); assert(code == 0 || code);
schtFreeQueryJob(0); schtFreeQueryJob(0);
taosHashCleanup(execTasks); taosHashCleanup(execTasks);
@ -620,7 +600,7 @@ void* schtRunJobThread(void *aa) {
return NULL; return NULL;
} }
void* schtFreeJobThread(void *aa) { void *schtFreeJobThread(void *aa) {
while (!schtTestStop) { while (!schtTestStop) {
taosUsleep(taosRand() % 100); taosUsleep(taosRand() % 100);
schtFreeQueryJob(1); schtFreeQueryJob(1);
@ -628,17 +608,16 @@ void* schtFreeJobThread(void *aa) {
return NULL; return NULL;
} }
} // namespace
}
TEST(queryTest, normalCase) { TEST(queryTest, normalCase) {
void *mockPointer = (void *)0x1; void *mockPointer = (void *)0x1;
char *clusterId = "cluster1"; char *clusterId = "cluster1";
char *dbname = "1.db1"; char *dbname = "1.db1";
char *tablename = "table1"; char *tablename = "table1";
SVgroupInfo vgInfo = {0}; SVgroupInfo vgInfo = {0};
int64_t job = 0; int64_t job = 0;
SQueryPlan dag; SQueryPlan dag;
memset(&dag, 0, sizeof(dag)); memset(&dag, 0, sizeof(dag));
@ -648,7 +627,7 @@ TEST(queryTest, normalCase) {
strcpy(qnodeAddr.fqdn, "qnode0.ep"); strcpy(qnodeAddr.fqdn, "qnode0.ep");
qnodeAddr.port = 6031; qnodeAddr.port = 6031;
taosArrayPush(qnodeList, &qnodeAddr); taosArrayPush(qnodeList, &qnodeAddr);
int32_t code = schedulerInit(); int32_t code = schedulerInit();
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
@ -662,30 +641,29 @@ TEST(queryTest, normalCase) {
SRequestConnInfo conn = {0}; SRequestConnInfo conn = {0};
conn.pTrans = mockPointer; conn.pTrans = mockPointer;
SSchedulerReq req = {0}; SSchedulerReq req = {0};
req.pConn = &conn; req.pConn = &conn;
req.pNodeList = qnodeList; req.pNodeList = qnodeList;
req.pDag = &dag; req.pDag = &dag;
req.sql = "select * from tb"; req.sql = "select * from tb";
req.execFp = schtQueryCb; req.execFp = schtQueryCb;
req.cbParam = &queryDone; req.cbParam = &queryDone;
code = schedulerExecJob(&req, &job); code = schedulerExecJob(&req, &job);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
SSchJob *pJob = schAcquireJob(job); SSchJob *pJob = schAcquireJob(job);
void *pIter = taosHashIterate(pJob->execTasks, NULL); void *pIter = taosHashIterate(pJob->execTasks, NULL);
while (pIter) { while (pIter) {
SSchTask *task = *(SSchTask **)pIter; SSchTask *task = *(SSchTask **)pIter;
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0); code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
pIter = taosHashIterate(pJob->execTasks, pIter); pIter = taosHashIterate(pJob->execTasks, pIter);
} }
pIter = taosHashIterate(pJob->execTasks, NULL); pIter = taosHashIterate(pJob->execTasks, NULL);
while (pIter) { while (pIter) {
@ -693,10 +671,10 @@ TEST(queryTest, normalCase) {
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0); code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
pIter = taosHashIterate(pJob->execTasks, pIter); pIter = taosHashIterate(pJob->execTasks, pIter);
} }
while (true) { while (true) {
if (queryDone) { if (queryDone) {
@ -705,14 +683,14 @@ TEST(queryTest, normalCase) {
taosUsleep(10000); taosUsleep(10000);
} }
TdThreadAttr thattr; TdThreadAttr thattr;
taosThreadAttrInit(&thattr); taosThreadAttrInit(&thattr);
TdThread thread1; TdThread thread1;
taosThreadCreate(&(thread1), &thattr, schtCreateFetchRspThread, &job); taosThreadCreate(&(thread1), &thattr, schtCreateFetchRspThread, &job);
void *data = NULL; void *data = NULL;
req.syncReq = true; req.syncReq = true;
req.pFetchRes = &data; req.pFetchRes = &data;
@ -739,13 +717,13 @@ TEST(queryTest, normalCase) {
} }
TEST(queryTest, readyFirstCase) { TEST(queryTest, readyFirstCase) {
void *mockPointer = (void *)0x1; void *mockPointer = (void *)0x1;
char *clusterId = "cluster1"; char *clusterId = "cluster1";
char *dbname = "1.db1"; char *dbname = "1.db1";
char *tablename = "table1"; char *tablename = "table1";
SVgroupInfo vgInfo = {0}; SVgroupInfo vgInfo = {0};
int64_t job = 0; int64_t job = 0;
SQueryPlan dag; SQueryPlan dag;
memset(&dag, 0, sizeof(dag)); memset(&dag, 0, sizeof(dag));
@ -755,7 +733,7 @@ TEST(queryTest, readyFirstCase) {
strcpy(qnodeAddr.fqdn, "qnode0.ep"); strcpy(qnodeAddr.fqdn, "qnode0.ep");
qnodeAddr.port = 6031; qnodeAddr.port = 6031;
taosArrayPush(qnodeList, &qnodeAddr); taosArrayPush(qnodeList, &qnodeAddr);
int32_t code = schedulerInit(); int32_t code = schedulerInit();
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
@ -765,11 +743,11 @@ TEST(queryTest, readyFirstCase) {
schtSetExecNode(); schtSetExecNode();
schtSetAsyncSendMsgToServer(); schtSetAsyncSendMsgToServer();
int32_t queryDone = 0; int32_t queryDone = 0;
SRequestConnInfo conn = {0}; SRequestConnInfo conn = {0};
conn.pTrans = mockPointer; conn.pTrans = mockPointer;
SSchedulerReq req = {0}; SSchedulerReq req = {0};
req.pConn = &conn; req.pConn = &conn;
req.pNodeList = qnodeList; req.pNodeList = qnodeList;
req.pDag = &dag; req.pDag = &dag;
@ -779,19 +757,18 @@ TEST(queryTest, readyFirstCase) {
code = schedulerExecJob(&req, &job); code = schedulerExecJob(&req, &job);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
SSchJob *pJob = schAcquireJob(job); SSchJob *pJob = schAcquireJob(job);
void *pIter = taosHashIterate(pJob->execTasks, NULL); void *pIter = taosHashIterate(pJob->execTasks, NULL);
while (pIter) { while (pIter) {
SSchTask *task = *(SSchTask **)pIter; SSchTask *task = *(SSchTask **)pIter;
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0); code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
pIter = taosHashIterate(pJob->execTasks, pIter); pIter = taosHashIterate(pJob->execTasks, pIter);
} }
pIter = taosHashIterate(pJob->execTasks, NULL); pIter = taosHashIterate(pJob->execTasks, NULL);
while (pIter) { while (pIter) {
@ -799,10 +776,10 @@ TEST(queryTest, readyFirstCase) {
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0); code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
pIter = taosHashIterate(pJob->execTasks, pIter); pIter = taosHashIterate(pJob->execTasks, pIter);
} }
while (true) { while (true) {
if (queryDone) { if (queryDone) {
@ -812,14 +789,13 @@ TEST(queryTest, readyFirstCase) {
taosUsleep(10000); taosUsleep(10000);
} }
TdThreadAttr thattr; TdThreadAttr thattr;
taosThreadAttrInit(&thattr); taosThreadAttrInit(&thattr);
TdThread thread1; TdThread thread1;
taosThreadCreate(&(thread1), &thattr, schtCreateFetchRspThread, &job); taosThreadCreate(&(thread1), &thattr, schtCreateFetchRspThread, &job);
void *data = NULL; void *data = NULL;
req.syncReq = true; req.syncReq = true;
req.pFetchRes = &data; req.pFetchRes = &data;
code = schedulerFetchRows(job, &req); code = schedulerFetchRows(job, &req);
@ -844,28 +820,26 @@ TEST(queryTest, readyFirstCase) {
schedulerDestroy(); schedulerDestroy();
} }
TEST(queryTest, flowCtrlCase) { TEST(queryTest, flowCtrlCase) {
void *mockPointer = (void *)0x1; void *mockPointer = (void *)0x1;
char *clusterId = "cluster1"; char *clusterId = "cluster1";
char *dbname = "1.db1"; char *dbname = "1.db1";
char *tablename = "table1"; char *tablename = "table1";
SVgroupInfo vgInfo = {0}; SVgroupInfo vgInfo = {0};
int64_t job = 0; int64_t job = 0;
SQueryPlan dag; SQueryPlan dag;
schtInitLogFile(); schtInitLogFile();
taosSeedRand(taosGetTimestampSec()); taosSeedRand(taosGetTimestampSec());
SArray *qnodeList = taosArrayInit(1, sizeof(SEp)); SArray *qnodeList = taosArrayInit(1, sizeof(SEp));
SEp qnodeAddr = {0}; SEp qnodeAddr = {0};
strcpy(qnodeAddr.fqdn, "qnode0.ep"); strcpy(qnodeAddr.fqdn, "qnode0.ep");
qnodeAddr.port = 6031; qnodeAddr.port = 6031;
taosArrayPush(qnodeList, &qnodeAddr); taosArrayPush(qnodeList, &qnodeAddr);
int32_t code = schedulerInit(); int32_t code = schedulerInit();
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
@ -875,10 +849,10 @@ TEST(queryTest, flowCtrlCase) {
schtSetExecNode(); schtSetExecNode();
schtSetAsyncSendMsgToServer(); schtSetAsyncSendMsgToServer();
int32_t queryDone = 0; int32_t queryDone = 0;
SRequestConnInfo conn = {0}; SRequestConnInfo conn = {0};
conn.pTrans = mockPointer; conn.pTrans = mockPointer;
SSchedulerReq req = {0}; SSchedulerReq req = {0};
req.pConn = &conn; req.pConn = &conn;
req.pNodeList = qnodeList; req.pNodeList = qnodeList;
req.pDag = &dag; req.pDag = &dag;
@ -889,17 +863,16 @@ TEST(queryTest, flowCtrlCase) {
code = schedulerExecJob(&req, &job); code = schedulerExecJob(&req, &job);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
SSchJob *pJob = schAcquireJob(job); SSchJob *pJob = schAcquireJob(job);
bool qDone = false; bool qDone = false;
while (!qDone) { while (!qDone) {
void *pIter = taosHashIterate(pJob->execTasks, NULL); void *pIter = taosHashIterate(pJob->execTasks, NULL);
if (NULL == pIter) { if (NULL == pIter) {
break; break;
} }
while (pIter) { while (pIter) {
SSchTask *task = *(SSchTask **)pIter; SSchTask *task = *(SSchTask **)pIter;
@ -908,15 +881,15 @@ TEST(queryTest, flowCtrlCase) {
if (task->lastMsgType == TDMT_SCH_QUERY) { if (task->lastMsgType == TDMT_SCH_QUERY) {
SQueryTableRsp rsp = {0}; SQueryTableRsp rsp = {0};
code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0); code = schHandleResponseMsg(pJob, task, TDMT_SCH_QUERY_RSP, (char *)&rsp, sizeof(rsp), 0);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
} else { } else {
qDone = true; qDone = true;
break; break;
} }
pIter = NULL; pIter = NULL;
} }
} }
while (true) { while (true) {
@ -933,7 +906,7 @@ TEST(queryTest, flowCtrlCase) {
TdThread thread1; TdThread thread1;
taosThreadCreate(&(thread1), &thattr, schtCreateFetchRspThread, &job); taosThreadCreate(&(thread1), &thattr, schtCreateFetchRspThread, &job);
void *data = NULL; void *data = NULL;
req.syncReq = true; req.syncReq = true;
req.pFetchRes = &data; req.pFetchRes = &data;
code = schedulerFetchRows(job, &req); code = schedulerFetchRows(job, &req);
@ -958,15 +931,14 @@ TEST(queryTest, flowCtrlCase) {
schedulerDestroy(); schedulerDestroy();
} }
TEST(insertTest, normalCase) { TEST(insertTest, normalCase) {
void *mockPointer = (void *)0x1; void *mockPointer = (void *)0x1;
char *clusterId = "cluster1"; char *clusterId = "cluster1";
char *dbname = "1.db1"; char *dbname = "1.db1";
char *tablename = "table1"; char *tablename = "table1";
SVgroupInfo vgInfo = {0}; SVgroupInfo vgInfo = {0};
SQueryPlan dag; SQueryPlan dag;
uint64_t numOfRows = 0; uint64_t numOfRows = 0;
SArray *qnodeList = taosArrayInit(1, sizeof(SEp)); SArray *qnodeList = taosArrayInit(1, sizeof(SEp));
@ -974,7 +946,7 @@ TEST(insertTest, normalCase) {
strcpy(qnodeAddr.fqdn, "qnode0.ep"); strcpy(qnodeAddr.fqdn, "qnode0.ep");
qnodeAddr.port = 6031; qnodeAddr.port = 6031;
taosArrayPush(qnodeList, &qnodeAddr); taosArrayPush(qnodeList, &qnodeAddr);
int32_t code = schedulerInit(); int32_t code = schedulerInit();
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
@ -993,21 +965,21 @@ TEST(insertTest, normalCase) {
SRequestConnInfo conn = {0}; SRequestConnInfo conn = {0};
conn.pTrans = mockPointer; conn.pTrans = mockPointer;
SSchedulerReq req = {0}; SSchedulerReq req = {0};
req.pConn = &conn; req.pConn = &conn;
req.pNodeList = qnodeList; req.pNodeList = qnodeList;
req.pDag = &dag; req.pDag = &dag;
req.sql = "insert into tb values(now,1)"; req.sql = "insert into tb values(now,1)";
req.execFp = schtQueryCb; req.execFp = schtQueryCb;
req.cbParam = NULL; req.cbParam = NULL;
code = schedulerExecJob(&req, &insertJobRefId); code = schedulerExecJob(&req, &insertJobRefId);
ASSERT_EQ(code, 0); ASSERT_EQ(code, 0);
ASSERT_EQ(res.numOfRows, 20); ASSERT_EQ(res.numOfRows, 20);
schedulerFreeJob(&insertJobRefId, 0); schedulerFreeJob(&insertJobRefId, 0);
schedulerDestroy(); schedulerDestroy();
} }
TEST(multiThread, forceFree) { TEST(multiThread, forceFree) {
@ -1027,12 +999,12 @@ TEST(multiThread, forceFree) {
break; break;
} }
} }
schtTestStop = true; schtTestStop = true;
taosSsleep(3); taosSsleep(3);
} }
int main(int argc, char** argv) { int main(int argc, char **argv) {
taosSeedRand(taosGetTimestampSec()); taosSeedRand(taosGetTimestampSec());
testing::InitGoogleTest(&argc, argv); testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS(); return RUN_ALL_TESTS();

6
source/libs/tdb/src/db/tdbPager.c
View File

@ -34,9 +34,9 @@ static int tdbPagerInitPage(SPager *pPager, SPage *pPage, int (*initPage)(SPage
static int tdbPagerWritePageToJournal(SPager *pPager, SPage *pPage); static int tdbPagerWritePageToJournal(SPager *pPager, SPage *pPage);
static int tdbPagerWritePageToDB(SPager *pPager, SPage *pPage); static int tdbPagerWritePageToDB(SPager *pPager, SPage *pPage);
static FORCE_INLINE int32_t pageCmpFn(const void *lhs, const void *rhs) { static FORCE_INLINE int32_t pageCmpFn(const SRBTreeNode *lhs, const SRBTreeNode *rhs) {
SPage *pPageL = (SPage *)(((uint8_t *)lhs) - sizeof(SRBTreeNode)); SPage *pPageL = (SPage *)(((uint8_t *)lhs) - offsetof(SPage, node));
SPage *pPageR = (SPage *)(((uint8_t *)rhs) - sizeof(SRBTreeNode)); SPage *pPageR = (SPage *)(((uint8_t *)rhs) - offsetof(SPage, node));
SPgno pgnoL = TDB_PAGE_PGNO(pPageL); SPgno pgnoL = TDB_PAGE_PGNO(pPageL);
SPgno pgnoR = TDB_PAGE_PGNO(pPageR); SPgno pgnoR = TDB_PAGE_PGNO(pPageR);

8
source/libs/transport/src/trans.c
View File

@ -100,8 +100,8 @@ void rpcCloseImpl(void* arg) {
taosMemoryFree(pRpc); taosMemoryFree(pRpc);
} }
void* rpcMallocCont(int32_t contLen) { void* rpcMallocCont(int64_t contLen) {
int32_t size = contLen + TRANS_MSG_OVERHEAD; int64_t size = contLen + TRANS_MSG_OVERHEAD;
char* start = taosMemoryCalloc(1, size); char* start = taosMemoryCalloc(1, size);
if (start == NULL) { if (start == NULL) {
tError("failed to malloc msg, size:%d", size); tError("failed to malloc msg, size:%d", size);
@ -120,11 +120,11 @@ void rpcFreeCont(void* cont) {
tTrace("rpc free cont:%p", (char*)cont - TRANS_MSG_OVERHEAD); tTrace("rpc free cont:%p", (char*)cont - TRANS_MSG_OVERHEAD);
} }
void* rpcReallocCont(void* ptr, int32_t contLen) { void* rpcReallocCont(void* ptr, int64_t contLen) {
if (ptr == NULL) return rpcMallocCont(contLen); if (ptr == NULL) return rpcMallocCont(contLen);
char* st = (char*)ptr - TRANS_MSG_OVERHEAD; char* st = (char*)ptr - TRANS_MSG_OVERHEAD;
int32_t sz = contLen + TRANS_MSG_OVERHEAD; int64_t sz = contLen + TRANS_MSG_OVERHEAD;
st = taosMemoryRealloc(st, sz); st = taosMemoryRealloc(st, sz);
if (st == NULL) { if (st == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;

8
source/os/src/osMemory.c
View File

@ -228,7 +228,7 @@ void taosPrintBackTrace() {
void taosPrintBackTrace() { return; } void taosPrintBackTrace() { return; }
#endif #endif
void *taosMemoryMalloc(int32_t size) { void *taosMemoryMalloc(int64_t size) {
#ifdef USE_TD_MEMORY #ifdef USE_TD_MEMORY
void *tmp = malloc(size + sizeof(TdMemoryInfo)); void *tmp = malloc(size + sizeof(TdMemoryInfo));
if (tmp == NULL) return NULL; if (tmp == NULL) return NULL;
@ -244,7 +244,7 @@ void *taosMemoryMalloc(int32_t size) {
#endif #endif
} }
void *taosMemoryCalloc(int32_t num, int32_t size) { void *taosMemoryCalloc(int64_t num, int64_t size) {
#ifdef USE_TD_MEMORY #ifdef USE_TD_MEMORY
int32_t memorySize = num * size; int32_t memorySize = num * size;
char *tmp = calloc(memorySize + sizeof(TdMemoryInfo), 1); char *tmp = calloc(memorySize + sizeof(TdMemoryInfo), 1);
@ -261,7 +261,7 @@ void *taosMemoryCalloc(int32_t num, int32_t size) {
#endif #endif
} }
void *taosMemoryRealloc(void *ptr, int32_t size) { void *taosMemoryRealloc(void *ptr, int64_t size) {
#ifdef USE_TD_MEMORY #ifdef USE_TD_MEMORY
if (ptr == NULL) return taosMemoryMalloc(size); if (ptr == NULL) return taosMemoryMalloc(size);
@ -318,7 +318,7 @@ void taosMemoryFree(void *ptr) {
#endif #endif
} }
int32_t taosMemorySize(void *ptr) { int64_t taosMemorySize(void *ptr) {
if (ptr == NULL) return 0; if (ptr == NULL) return 0;
#ifdef USE_TD_MEMORY #ifdef USE_TD_MEMORY

12
source/util/src/trbtree.c
View File

@ -219,7 +219,7 @@ SRBTreeNode *tRBTreePut(SRBTree *pTree, SRBTreeNode *z) {
while (temp != pTree->NIL) { while (temp != pTree->NIL) {
y = temp; y = temp;
int32_t c = pTree->cmprFn(RBTREE_NODE_PAYLOAD(z), RBTREE_NODE_PAYLOAD(temp)); int32_t c = pTree->cmprFn(z, temp);
if (c < 0) { if (c < 0) {
temp = temp->left; temp = temp->left;
} else if (c > 0) { } else if (c > 0) {
@ -232,7 +232,7 @@ SRBTreeNode *tRBTreePut(SRBTree *pTree, SRBTreeNode *z) {
if (y == pTree->NIL) { if (y == pTree->NIL) {
pTree->root = z; pTree->root = z;
} else if (pTree->cmprFn(RBTREE_NODE_PAYLOAD(z), RBTREE_NODE_PAYLOAD(y)) < 0) { } else if (pTree->cmprFn(z, y) < 0) {
y->left = z; y->left = z;
} else { } else {
y->right = z; y->right = z;
@ -245,10 +245,10 @@ SRBTreeNode *tRBTreePut(SRBTree *pTree, SRBTreeNode *z) {
tRBTreePutFix(pTree, z); tRBTreePutFix(pTree, z);
// update min/max node // update min/max node
if (pTree->min == pTree->NIL || pTree->cmprFn(RBTREE_NODE_PAYLOAD(pTree->min), RBTREE_NODE_PAYLOAD(z)) > 0) { if (pTree->min == pTree->NIL || pTree->cmprFn(pTree->min, z) > 0) {
pTree->min = z; pTree->min = z;
} }
if (pTree->max == pTree->NIL || pTree->cmprFn(RBTREE_NODE_PAYLOAD(pTree->max), RBTREE_NODE_PAYLOAD(z)) < 0) { if (pTree->max == pTree->NIL || pTree->cmprFn(pTree->max, z) < 0) {
pTree->max = z; pTree->max = z;
} }
pTree->n++; pTree->n++;
@ -309,11 +309,11 @@ SRBTreeNode *tRBTreeDropByKey(SRBTree *pTree, void *pKey) {
return pNode; return pNode;
} }
SRBTreeNode *tRBTreeGet(SRBTree *pTree, void *pKey) { SRBTreeNode *tRBTreeGet(SRBTree *pTree, const SRBTreeNode *pKeyNode) {
SRBTreeNode *pNode = pTree->root; SRBTreeNode *pNode = pTree->root;
while (pNode != pTree->NIL) { while (pNode != pTree->NIL) {
int32_t c = pTree->cmprFn(pKey, RBTREE_NODE_PAYLOAD(pNode)); int32_t c = pTree->cmprFn(pKeyNode, pNode);
if (c < 0) { if (c < 0) {
pNode = pNode->left; pNode = pNode->left;

5
source/util/src/tskiplist.c
View File

@ -145,7 +145,10 @@ void tSkipListPutBatchByIter(SSkipList *pSkipList, void *iter, iter_next_fn_t it
tSkipListWLock(pSkipList); tSkipListWLock(pSkipList);
void *pData = iterate(iter); void *pData = iterate(iter);
if (pData == NULL) return; if (pData == NULL) {
tSkipListUnlock(pSkipList);
return;
}
// backward to put the first data // backward to put the first data
hasDup = tSkipListGetPosToPut(pSkipList, backward, pData); hasDup = tSkipListGetPosToPut(pSkipList, backward, pData);

1
tests/script/jenkins/basic.txt
View File

@ -352,6 +352,7 @@
./test.sh -f tsim/scalar/in.sim ./test.sh -f tsim/scalar/in.sim
./test.sh -f tsim/scalar/scalar.sim ./test.sh -f tsim/scalar/scalar.sim
./test.sh -f tsim/scalar/filter.sim ./test.sh -f tsim/scalar/filter.sim
./test.sh -f tsim/scalar/caseWhen.sim
# ---- alter ---- # ---- alter ----
./test.sh -f tsim/alter/cached_schema_after_alter.sim ./test.sh -f tsim/alter/cached_schema_after_alter.sim

1064
tests/script/tsim/scalar/caseWhen.sim Normal file
View File

File diff suppressed because it is too large Load Diff

22
tools/scripts/codeFormat.sh Normal file → Executable file
View File

@ -12,25 +12,7 @@ FORMAT_DIR_LIST=(
"${PRJ_ROOT_DIR}/source/os" "${PRJ_ROOT_DIR}/source/os"
"${PRJ_ROOT_DIR}/source/util" "${PRJ_ROOT_DIR}/source/util"
"${PRJ_ROOT_DIR}/source/common" "${PRJ_ROOT_DIR}/source/common"
"${PRJ_ROOT_DIR}/source/libs/cache" "${PRJ_ROOT_DIR}/source/libs"
"${PRJ_ROOT_DIR}/source/libs/catalog"
"${PRJ_ROOT_DIR}/source/libs/command"
"${PRJ_ROOT_DIR}/source/libs/executor"
"${PRJ_ROOT_DIR}/source/libs/function"
"${PRJ_ROOT_DIR}/source/libs/index"
"${PRJ_ROOT_DIR}/source/libs/monitor"
"${PRJ_ROOT_DIR}/source/libs/nodes"
# "${PRJ_ROOT_DIR}/source/libs/parser"
"${PRJ_ROOT_DIR}/source/libs/planner"
"${PRJ_ROOT_DIR}/source/libs/qcom"
"${PRJ_ROOT_DIR}/source/libs/qworker"
"${PRJ_ROOT_DIR}/source/libs/scalar"
"${PRJ_ROOT_DIR}/source/libs/stream"
"${PRJ_ROOT_DIR}/source/libs/sync"
"${PRJ_ROOT_DIR}/source/libs/tdb"
"${PRJ_ROOT_DIR}/source/libs/tfs"
"${PRJ_ROOT_DIR}/source/libs/transport"
"${PRJ_ROOT_DIR}/source/libs/wal"
"${PRJ_ROOT_DIR}/source/client/inc" "${PRJ_ROOT_DIR}/source/client/inc"
"${PRJ_ROOT_DIR}/source/client/src" "${PRJ_ROOT_DIR}/source/client/src"
"${PRJ_ROOT_DIR}/source/client/test" "${PRJ_ROOT_DIR}/source/client/test"
@ -45,7 +27,7 @@ EXCLUDE_FILE_LIST=(
) )
for d in ${FORMAT_DIR_LIST[@]}; do for d in ${FORMAT_DIR_LIST[@]}; do
for f in $(find $d -type f -regex '.*\.\(cpp\|hpp\|c\|h\)'); do for f in $(find $d -type f -not -name '*sql.c' -regex '.*\.\(cpp\|hpp\|c\|h\)'); do
${FORMAT_BIN} -i $f ${FORMAT_BIN} -i $f
done done
done done