xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch '3.0' of github.com:taosdata/TDengine into szhou/fixbugs

This commit is contained in:
slzhou 2022-11-22 22:59:06 +08:00
parent 272a161a61 8802981aac
commit 0161374a8b
103 changed files with 5317 additions and 1439 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

27
CONTRIBUTING-CN.md
View File

@ -7,25 +7,18 @@
- 任何用户都可以通过 **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)** 向我们报告错误。请您对所遇到的问题进行**详细描述**，最好提供重现错误的详细步骤。
- 欢迎提供包含由 Bug 生成的日志文件的附录。
## 需要强调的代码提交规则
## 代码提交规则
- 在提交代码之前，需要**同意贡献者许可协议（CLA）**。点击 [TaosData CLA](https://cla-assistant.io/taosdata/TDengine) 阅读并签署协议。如果您不接受该协议，请停止提交。
- 请在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中解决问题或添加注册功能。
- 如果在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中没有找到相应的问题或功能，请**创建一个新的 issue**。
- 将代码提交到我们的存储库时，请创建**包含问题编号的 PR**。
1. 在提交代码之前，需要**同意贡献者许可协议（CLA）**。点击 [TaosData CLA](https://cla-assistant.io/taosdata/TDengine) 阅读并签署协议。如果您不接受该协议，请停止提交。
2. 请在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中解决问题或添加注册功能。
如果在 [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) 中没有找到相应的问题或功能，请**创建一个新的 issue**。
将代码提交到我们的存储库时，请创建**包含问题编号的 PR**。
3. 将TDengine仓库库fork到自己的账户中并创建分支(branch)。
注意：默认分支`main`不能直接接受PR，请基于开发分支`3.0`创建自己的分支。
注意：修改文档的分支要以`docs/`为开头，以免进行不必要的测试。
4. 创建pull request，将自己的分支合并到开发分支`3.0`，我们开发团队将尽快审核。
## 贡献指南
1. 请用友好的语气书写。
2. **主动语态**总体上优于被动语态。主动语态中的句子会突出执行动作的人，而不是被动语态突出动作的接受者。
3. 文档写作建议
- 正确拼写产品名称 “TDengine”。 “TD” 用大写字母，“TD” 和 “engine” 之间没有空格 **（正确拼写：TDengine）**。
- 在句号或其他标点符号后只留一个空格。
4. 尽量**使用简单句**，而不是复杂句。
如遇任何问题，请添加官方微信TDengineECO。我们的团队会帮忙解决。
## 给贡献者的礼品

60
CONTRIBUTING.md
View File

@ -1,40 +1,36 @@
# Contributing
# Contributing to TDengine
We appreciate contributions from all developers. Feel free to follow us, fork the repository, report bugs, and even submit your code on GitHub. However, we would like developers to follow the guidelines in this document to ensure effective cooperation.
TDengine Community Edition is free, open-source software. Its development is led by the TDengine Team, but we welcome contributions from all community members and open-source developers. This document describes how you can contribute, no matter whether you're a user or a developer yourself.
## Reporting a bug
## Bug reports
- Any users can report bugs to us through the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. We would appreciate if you could provide **a detailed description** of the problem you encountered, including steps to reproduce it.
All users can report bugs to us through the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**. To ensure that the development team can locate and resolve the issue that you experienced, please include the following in your bug report:
- Attaching log files caused by the bug is really appreciated.
- A detailed description of the issue, including the steps to reproduce it.
- Any log files that may be relevant to the issue.
## Guidelines for committing code
## Code contributions
- You must agree to the **Contributor License Agreement(CLA) before submitting your code patch**. Follow the **[TAOSData CLA](https://cla-assistant.io/taosdata/TDengine)** link to read through and sign the agreement. If you do not accept the agreement, your contributions cannot be accepted.
Developers are encouraged to submit patches to the project, and all contributions, from minor documentation changes to bug fixes, are appreciated by our team. To ensure that your code can be merged successfully and improve the experience for other community members, we ask that you go through the following procedure before submitting a pull request:
- Please solve an issue or add a feature registered in the **[GitHub issue tracker](https://github.com/taosdata/TDengine/issues)**.
- If no corresponding issue or feature is found in the issue tracker, please **create one**.
- When submitting your code to our repository, please create a pull request with the **issue number** included.
1. Read and accept the terms of the TAOS Data Contributor License Agreement (CLA) located at [https://cla-assistant.io/taosdata/TDengine](https://cla-assistant.io/taosdata/TDengine).
## Guidelines for communicating
2. For bug fixes, search the [GitHub issue tracker](https://github.com/taosdata/TDengine/issues) to check whether the bug has already been filed.
- If the bug that you want to fix already exists in the issue tracker, review the previous discussion before submitting your patch.
- If the bug that you want to fix does not exist in the issue tracker, click **New issue** and file a report.
- Ensure that you note the issue number in your pull request when you submit your patch.
3. Fork our repository to your GitHub account and create a branch for your patch.
**Important:** The `main` branch is for stable versions and cannot accept patches directly. For all code and documentation changes, create your own branch from the development branch `3.0` and not from `main`.
Note: For a documentation change, ensure that the branch name starts with `docs/` so that the change can be merged without running tests.
4. Create a pull request to merge your changes into the development branch `3.0`, and our team members will review the request as soon as possible.
1. Please be **nice and polite** in the description.
2. **Active voice is better than passive voice in general**. Sentences in the active voice will highlight who is performing the action rather than the recipient of the action highlighted by the passive voice.
3. Documentation writing advice
If you encounter any difficulties or problems in contributing your code, you can join our [Discord server](https://discord.com/invite/VZdSuUg4pS) and receive assistance from members of the TDengine Team.
- Spell the product name "TDengine" correctly. "TD" is written in capital letters, and there is no space between "TD" and "engine" (**Correct spelling: TDengine**).
- Please **capitalize the first letter** of every sentence.
- Leave **only one space** after periods or other punctuation marks.
- Use **American spelling**.
- When possible, **use second person** rather than first person (e.g.“You are recommended to use a reverse proxy such as Nginx.” rather than “We recommend to use a reverse proxy such as Nginx.”).
## Expressing our thanks
5. Use **simple sentences**, rather than complex sentences.
## Gifts for the contributors
Developers, as long as you contribute to TDengine, whether it's code contributions to fix bugs or feature requests, or documentation changes, **you are eligible for a very special Contributor Souvenir Gift!**
**You can choose one of the following gifts:**
To thank community members for your support, we are offering a free gift to any developer who submits at least one contribution. You can choose one of the following items:
<p align="left">
<img
@ -53,12 +49,10 @@ Developers, as long as you contribute to TDengine, whether it's code contributio
width="200"
/>
The TDengine community is committed to making TDengine accepted and used by more developers.
If you would like to claim your gift, send an email to [developer@tdengine.com](mailto:developer@tdengine.com?subject=Claiming&20my%20developer%20gift) including the following information:
Just fill out the **Contributor Submission Form** to choose your desired gift.
- Your GitHub account name
- Your name and mailing address
- Your preferred gift
- [Contributor Submission Form](https://page.ma.scrmtech.com/form/index?pf_uid=27715_2095&id=12100)
## Contact us
If you have any problems or questions that need help from us, please feel free to add our WeChat account: TDengineECO.
Note: Limit one per person.

2
cmake/taostools_CMakeLists.txt.in
View File

@ -2,7 +2,7 @@
# taos-tools
ExternalProject_Add(taos-tools
GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
GIT_TAG 23e2b73
GIT_TAG e00ebd9
SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
BINARY_DIR ""
#BUILD_IN_SOURCE TRUE

2
cmake/taosws_CMakeLists.txt.in
View File

@ -2,7 +2,7 @@
# taosws-rs
ExternalProject_Add(taosws-rs
GIT_REPOSITORY https://github.com/taosdata/taos-connector-rust.git
GIT_TAG 9843872
GIT_TAG f406d51
SOURCE_DIR "${TD_SOURCE_DIR}/tools/taosws-rs"
BINARY_DIR ""
#BUILD_IN_SOURCE TRUE

13
docs/en/05-get-started/index.md
View File

@ -20,7 +20,18 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
<DocCardList items={useCurrentSidebarCategory().items}/>
```
### Join TDengine Community
## Study TDengine Knowledge Map
The TDengine Knowledge Map covers the various knowledge points of TDengine, revealing the invocation relationships and data flow between various conceptual entities. Learning and understanding the TDengine Knowledge Map will help you quickly master the TDengine knowledge system.
<figure>
<center>
<a href="pathname:///img/tdengine-map.svg" target="_blank"><img src="/img/tdengine-map.svg" width="80%" /></a>
<figcaption>Diagram 1. TDengine Knowledge Map</figcaption>
</center>
</figure>
## Join TDengine Community
<table width="100%">
<tr align="center" style={{border:0}}>

2
docs/en/12-taos-sql/06-select.md
View File

@ -301,7 +301,7 @@ SELECT TIMEZONE();
### Syntax
```txt
WHERE (column|tbname) **match/MATCH/nmatch/NMATCH** _regex_
WHERE (column|tbname) match/MATCH/nmatch/NMATCH _regex_
```
### Specification

15
docs/en/14-reference/03-connector/index.mdx
View File

@ -26,14 +26,13 @@ Using REST connection can support a broader range of operating systems as it doe
TDengine version updates often add new features, and the connector versions in the list are the best-fit versions of the connector.
| **TDengine Versions** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** |
| --------------------- | -------- | ---------- | ------------ | ------------- | --------------- | -------- |
| **3.0.0.0 and later** | 3.0.0 | current version | 3.0 branch | 3.0.0 | 3.0.0 | current version |
| **2.4.0.14 and up** | 2.0.38 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
| **2.4.0.6 and up** | 2.0.37 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
| **2.4.0.4 - 2.4.0.5** | 2.0.37 | current version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
| **2.2.x.x ** | 2.0.36 | current version | master branch | n/a | 2.0.7 - 2.0.9 | current version |
| **2.0.x.x ** | 2.0.34 | current version | master branch | n/a | 2.0.1 - 2.0.6 | current version |
| **TDengine Versions** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** |
| --------------------------- | -------------- | -------------- | -------------- | ------------- | --------------- | --------------- |
| **3.0.0.0 and later** | 3.0.2 + | latest version | 3.0 branch | 3.0.0 | 3.0.0 | current version |
| **2.4.0.14 and up ** | 2.0.38 | latest version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
| **2.4.0.4 - 2.4.0.13 ** | 2.0.37 | latest version | develop branch | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | current version |
| **2.2.x.x ** | 2.0.36 | latest version | master branch | n/a | 2.0.7 - 2.0.9 | current version |
| **2.0.x.x ** | 2.0.34 | latest version | master branch | n/a | 2.0.1 - 2.0.6 | current version |
## Functional Features

13
docs/zh/05-get-started/index.md
View File

@ -18,7 +18,18 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
<DocCardList items={useCurrentSidebarCategory().items}/>
```
### 加入 TDengine 官方社区
## 学习 TDengine 知识地图
TDengine 知识地图中涵盖了 TDengine 的各种知识点，揭示了各概念实体之间的调用关系和数据流向。学习和了解 TDengine 知识地图有助于你快速掌握 TDengine 的知识体系。
<figure>
<center>
<a href="pathname:///img/tdengine-map.svg" target="_blank"><img src="/img/tdengine-map.svg" width="80%" /></a>
<figcaption>图 1. TDengine 知识地图</figcaption>
</center>
</figure>
## 加入 TDengine 官方社区
微信扫描以下二维码，学习了解 TDengine 的最新技术，与大家共同交流物联网大数据技术应用、TDengine 使用问题和技巧等话题。

15
docs/zh/08-connector/index.md
View File

@ -26,14 +26,13 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速
TDengine 版本更新往往会增加新的功能特性，列表中的连接器版本为连接器最佳适配版本。
| **TDengine 版本** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** |
| --------------------- | -------- | ---------- | ------------ | ------------- | --------------- | -------- |
| **3.0.0.0 及以上** | 3.0.0 | 当前版本 | 3.0 分支 | 3.0.0 | 3.0.0 | 当前版本 |
| **2.4.0.14 及以上** | 2.0.38 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
| **2.4.0.6 及以上** | 2.0.37 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
| **2.4.0.4 - 2.4.0.5** | 2.0.37 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
| **2.2.x.x ** | 2.0.36 | 当前版本 | master 分支 | n/a | 2.0.7 - 2.0.9 | 当前版本 |
| **2.0.x.x ** | 2.0.34 | 当前版本 | master 分支 | n/a | 2.0.1 - 2.0.6 | 当前版本 |
| **TDengine 版本** | **Java** | **Python** | **Go** | **C#** | **Node.js** | **Rust** |
| ---------------------- | --------- | ---------- | ------------ | ------------- | --------------- | -------- |
| **3.0.0.0 及以上** | 3.0.2以上 | 当前版本 | 3.0 分支 | 3.0.0 | 3.0.0 | 当前版本 |
| **2.4.0.14 及以上** | 2.0.38 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
| **2.4.0.4 - 2.4.0.13** | 2.0.37 | 当前版本 | develop 分支 | 1.0.2 - 1.0.6 | 2.0.10 - 2.0.12 | 当前版本 |
| **2.2.x.x ** | 2.0.36 | 当前版本 | master 分支 | n/a | 2.0.7 - 2.0.9 | 当前版本 |
| **2.0.x.x ** | 2.0.34 | 当前版本 | master 分支 | n/a | 2.0.1 - 2.0.6 | 当前版本 |
## 功能特性

2
docs/zh/12-taos-sql/06-select.md
View File

@ -302,7 +302,7 @@ SELECT TIMEZONE();
### 语法
```txt
WHERE (column|tbname) **match/MATCH/nmatch/NMATCH** _regex_
WHERE (column|tbname) match/MATCH/nmatch/NMATCH _regex_
```
### 正则表达式规范

21
include/common/tmsg.h
View File

@ -1615,15 +1615,21 @@ typedef struct SSubQueryMsg {
uint64_t taskId;
int64_t refId;
int32_t execId;
int32_t msgMask;
int8_t taskType;
int8_t explain;
int8_t needFetch;
uint32_t sqlLen; // the query sql,
uint32_t phyLen;
int32_t msgMask;
char msg[];
uint32_t sqlLen;
char *sql;
uint32_t msgLen;
char *msg;
} SSubQueryMsg;
int32_t tSerializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq);
int32_t tDeserializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq);
void tFreeSSubQueryMsg(SSubQueryMsg *pReq);
typedef struct {
SMsgHead header;
uint64_t sId;
@ -1732,6 +1738,13 @@ typedef struct {
int32_t execId;
} STaskDropReq;
int32_t tSerializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq);
int32_t tDeserializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq);
int32_t tSerializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp);
int32_t tDeserializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp);
typedef struct {
int32_t code;
} STaskDropRsp;

11
include/libs/stream/streamState.h
View File

@ -27,8 +27,7 @@ typedef struct SStreamTask SStreamTask;
typedef bool (*state_key_cmpr_fn)(void* pKey1, void* pKey2);
// incremental state storage
typedef struct {
typedef struct STdbState {
SStreamTask* pOwner;
TDB* db;
TTB* pStateDb;
@ -37,7 +36,12 @@ typedef struct {
TTB* pSessionStateDb;
TTB* pParNameDb;
TXN txn;
int32_t number;
} STdbState;
// incremental state storage
typedef struct {
STdbState* pTdbState;
int32_t number;
} SStreamState;
SStreamState* streamStateOpen(char* path, SStreamTask* pTask, bool specPath, int32_t szPage, int32_t pages);
@ -45,6 +49,7 @@ void streamStateClose(SStreamState* pState);
int32_t streamStateBegin(SStreamState* pState);
int32_t streamStateCommit(SStreamState* pState);
int32_t streamStateAbort(SStreamState* pState);
void streamStateDestroy(SStreamState* pState);
typedef struct {
TBC* pCur;

2
include/libs/sync/sync.h
View File

@ -36,7 +36,7 @@ extern "C" {
#define SYNC_DEL_WAL_MS (1000 * 60)
#define SYNC_ADD_QUORUM_COUNT 3
#define SYNC_MNODE_LOG_RETENTION 10000
#define SYNC_VNODE_LOG_RETENTION 100
#define SYNC_VNODE_LOG_RETENTION 20
#define SNAPSHOT_MAX_CLOCK_SKEW_MS 1000 * 10
#define SNAPSHOT_WAIT_MS 1000 * 30

187
packaging/MPtestJenkinsfile
View File

@ -40,27 +40,32 @@ pipeline {
choice(
name: 'sourcePath',
choices: ['nas','web'],
description: 'choice which way to download the installation pacakge;web is Office Web and nas means taos nas server '
description: 'Choice which way to download the installation pacakge;web is Office Web and nas means taos nas server '
)
choice(
name: 'verMode',
choices: ['all','community','enterprise'],
description: 'Choice which types of package you want do check '
)
string (
name:'version',
defaultValue:'3.0.1.6',
description: 'release version number,eg: 3.0.0.1 or 3.0.0.'
defaultValue:'3.0.1.7',
description: 'Release version number,eg: 3.0.0.1 or 3.0.0.'
)
string (
name:'baseVersion',
defaultValue:'3.0.1.6',
description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1'
defaultValue:'3.0.1.7',
description: 'The number of baseVerison is generally not modified.Now it is 3.0.0.1'
)
string (
name:'toolsVersion',
defaultValue:'2.2.7',
description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1'
description: 'Release version number,eg:2.2.0'
)
string (
name:'toolsBaseVersion',
defaultValue:'2.1.2',
description: 'This number of baseVerison is generally not modified.Now it is 3.0.0.1'
description: 'The number of baseVerison is generally not modified.Now it is 2.1.2'
)
}
environment{
@ -68,10 +73,10 @@ pipeline {
TDINTERNAL_ROOT_DIR = '/var/lib/jenkins/workspace/TDinternal'
TDENGINE_ROOT_DIR = '/var/lib/jenkins/workspace/TDinternal/community'
BRANCH_NAME = '3.0'
TD_SERVER_TAR = "TDengine-server-${version}-Linux-x64.tar.gz"
TD_SERVER_TAR = "${preServerPackag}-${version}-Linux-x64.tar.gz"
BASE_TD_SERVER_TAR = "TDengine-server-${baseVersion}-Linux-x64.tar.gz"
TD_SERVER_ARM_TAR = "TDengine-server-${version}-Linux-arm64.tar.gz"
BASE_TD_SERVER_ARM_TAR = "TDengine-server-${baseVersion}-Linux-arm64.tar.gz"
@ -108,19 +113,28 @@ pipeline {
timeout(time: 30, unit: 'MINUTES'){
sync_source("${BRANCH_NAME}")
sh '''
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
'''
sh '''
bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_DEB} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t deb
python3 checkPackageRuning.py
'''
}
}
@ -131,22 +145,30 @@ pipeline {
timeout(time: 30, unit: 'MINUTES'){
sync_source("${BRANCH_NAME}")
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_DEB} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
dpkg -r tdengine
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t deb
python3 checkPackageRuning.py
dpkg -r tdengine
'''
}
}
}
@ -156,19 +178,28 @@ pipeline {
timeout(time: 30, unit: 'MINUTES'){
sync_source("${BRANCH_NAME}")
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_RPM} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t rpm
python3 checkPackageRuning.py
'''
}
}
@ -179,21 +210,30 @@ pipeline {
timeout(time: 30, unit: 'MINUTES'){
sync_source("${BRANCH_NAME}")
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_TAR} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_LITE_TAR} ${version} ${BASE_TD_SERVER_LITE_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
bash testpackage.sh -f server -m community -f server -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
'''
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_RPM} ${version} ${BASE_TD_SERVER_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
bash testpackage.sh -f server -m community -f server -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t rpm
python3 checkPackageRuning.py
sudo rpm -e tdengine
'''
'''
}
}
}
@ -203,9 +243,16 @@ pipeline {
timeout(time: 30, unit: 'MINUTES'){
sync_source("${BRANCH_NAME}")
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_SERVER_ARM_TAR} ${version} ${BASE_TD_SERVER_ARM_TAR} ${baseVersion} server ${sourcePath}
python3 checkPackageRuning.py
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m ${verModeSin} -f server -l false -c arm64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
'''
}
}
@ -219,8 +266,16 @@ pipeline {
steps {
timeout(time: 30, unit: 'MINUTES'){
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_CLIENT_TAR} ${version} ${BASE_TD_CLIENT_TAR} ${baseVersion} client ${sourcePath}
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m ${verModeSin} -f client -l false -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
python3 checkPackageRuning.py 192.168.0.21
'''
}
@ -231,8 +286,10 @@ pipeline {
steps {
timeout(time: 30, unit: 'MINUTES'){
sh '''
verModeList=community
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_CLIENT_LITE_TAR} ${version} ${BASE_TD_CLIENT_LITE_TAR} ${baseVersion} client ${sourcePath}
bash testpackage.sh -f server -m ${verModeSin} -f client -l true -c x64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
python3 checkPackageRuning.py 192.168.0.24
'''
}
@ -245,8 +302,16 @@ pipeline {
steps {
timeout(time: 30, unit: 'MINUTES'){
sh '''
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh ${TD_CLIENT_ARM_TAR} ${version} ${BASE_TD_CLIENT_ARM_TAR} ${baseVersion} client ${sourcePath}
if [ "${verMode}" = "all" ];then
verMode="community enterprise"
fi
verModeList=${verMode}
for verModeSin in ${verModeList}
do
cd ${TDENGINE_ROOT_DIR}/packaging
bash testpackage.sh -f server -m ${verModeSin} -f client -l false -c arm64 -v ${version} -o ${baseVersion} -s ${sourcePath} -t tar
python3 checkPackageRuning.py
done
python3 checkPackageRuning.py 192.168.0.21
'''
}

3
packaging/docker/DockerfileCloud
View File

@ -7,6 +7,9 @@ ARG dirName
ARG cpuType
RUN echo ${pkgFile} && echo ${dirName}
RUN apt update
RUN apt install -y curl
COPY ${pkgFile} /root/
ENV TINI_VERSION v0.19.0
ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-${cpuType} /tini

183
packaging/docker/run.sh Normal file → Executable file
View File

@ -1,16 +1,161 @@
#!/bin/bash
TAOS_RUN_TAOSBENCHMARK_TEST_ONCE=0
#ADMIN_URL=${ADMIN_URL:-http://172.26.10.84:10001}
TAOSD_STARTUP_TIMEOUT_SECOND=${TAOSD_STARTUP_TIMEOUT_SECOND:-160}
TAOS_TIMEOUT_SECOND=${TAOS_TIMEOUT_SECOND:-5}
BACKUP_CORE_FOLDER=/var/log/corefile
ALERT_URL=app/system/alert/add
echo "ADMIN_URL: ${ADMIN_URL}"
echo "TAOS_TIMEOUT_SECOND: ${TAOS_TIMEOUT_SECOND}"
function set_service_state() {
#echo "set service state: $1, $2"
service_state="$1"
service_msg="$2"
}
set_service_state "init" "ok"
app_name=`hostname |cut -d\- -f1`
function check_taosd() {
timeout $TAOS_TIMEOUT_SECOND taos -s "show databases;" >/dev/null
local ret=$?
if [ $ret -ne 0 ]; then
echo "`date` check taosd error $ret"
if [ "x$1" != "xignore" ]; then
set_service_state "error" "taos check failed $ret"
fi
else
set_service_state "ready" "ok"
fi
}
function post_error_msg() {
if [ ! -z "${ADMIN_URL}" ]; then
taos_version=`taos --version`
echo "app_name: ${app_name}"
echo "service_state: ${service_state}"
echo "`date` service_msg: ${service_msg}"
echo "${taos_version}"
curl --connect-timeout 10 --max-time 20 -X POST -H "Content-Type: application/json" \
-d"{\"appName\":\"${app_name}\",\
\"alertLevel\":\"${service_state}\",\
\"taosVersion\":\"${taos_version}\",\
\"alertMsg\":\"${service_msg}\"}" \
${ADMIN_URL}/${ALERT_URL}
fi
}
function check_taosd_exit_type() {
local core_pattern=`cat /proc/sys/kernel/core_pattern`
echo "$core_pattern" | grep -q "^/"
if [ $? -eq 0 ]; then
core_folder=`dirname $core_pattern`
core_prefix=`basename $core_pattern | sed "s/%.*//"`
else
core_folder=`pwd`
core_prefix="$core_pattern"
fi
local core_files=`ls $core_folder | grep "^${core_prefix}"`
if [ ! -z "$core_files" ]; then
# move core files to another folder
mkdir -p ${BACKUP_CORE_FOLDER}
cp ${core_folder}/${core_prefix}* ${BACKUP_CORE_FOLDER}/
rm -f ${core_folder}/${core_prefix}*
set_service_state "error" "taosd exit with core file"
else
set_service_state "error" "taosd exit without core file"
fi
}
disk_usage_level=(60 80 99)
current_disk_level=0
disk_state="ok"
disk_msg="ok"
get_usage_ok="yes"
function post_disk_error_msg() {
if [ ! -z "${ADMIN_URL}" ]; then
taos_version=`taos --version`
echo "app_name: ${app_name}"
echo "disk_state: ${disk_state}"
echo "`date` disk_msg: ${disk_msg}"
echo "${taos_version}"
curl --connect-timeout 10 --max-time 20 -X POST -H "Content-Type: application/json" \
-d"{\"appName\":\"${app_name}\",\
\"alertLevel\":\"${disk_state}\",\
\"taosVersion\":\"${taos_version}\",\
\"alertMsg\":\"${disk_msg}\"}" \
${ADMIN_URL}/${ALERT_URL}
fi
}
function check_disk() {
local folder=`cat /etc/taos/taos.cfg|grep -v "^#"|grep dataDir|awk '{print $NF}'`
if [ -z "$folder" ]; then
folder="/var/lib/taos"
fi
local mount_point="$folder"
local usage=""
while [ -z "$usage" ]; do
usage=`df -h|grep -w "${mount_point}"|awk '{print $5}'|grep -v Use|sed "s/%$//"`
if [ "x${mount_point}" = "x/" ]; then
break
fi
mount_point=`dirname ${mount_point}`
done
if [ -z "$usage" ]; then
disk_state="error"
disk_msg="cannot get disk usage"
if [ "$get_usage_ok" = "yes" ]; then
post_disk_error_msg
get_usage_ok="no"
fi
else
get_usage_ok="yes"
local current_level=0
for level in ${disk_usage_level[*]}; do
if [ ${usage} -ge ${level} ]; then
disk_state="error"
disk_msg="disk usage over ${level}%"
current_level=${level}
fi
done
if [ ${current_level} -gt ${current_disk_level} ]; then
post_disk_error_msg
elif [ ${current_level} -lt ${current_disk_level} ]; then
echo "disk usage reduced from ${current_disk_level} to ${current_level}"
fi
current_disk_level=${current_level}
fi
}
function run_taosd() {
taosd
set_service_state "error" "taosd exit"
# post error msg
# check crash or OOM
check_taosd_exit_type
post_error_msg
}
function print_service_state_change() {
if [ "x$1" != "x${service_state}" ]; then
echo "`date` service state: ${service_state}, ${service_msg}"
fi
}
taosd_start_time=`date +%s`
while ((1))
do
check_disk
# echo "outer loop: $a"
sleep 10
output=`taos -k`
status=${output:0:1}
output=`timeout $TAOS_TIMEOUT_SECOND taos -k`
if [ -z "${output}" ]; then
echo "`date` taos -k error"
status=""
else
status=${output:0:1}
fi
# echo $output
# echo $status
if [ "$status"x = "0"x ]
then
taosd &
# taosd_start_time=`date +%s`
run_taosd &
fi
# echo "$status"x "$TAOS_RUN_TAOSBENCHMARK_TEST"x "$TAOS_RUN_TAOSBENCHMARK_TEST_ONCE"x
if [ "$status"x = "2"x ] && [ "$TAOS_RUN_TAOSBENCHMARK_TEST"x = "1"x ] && [ "$TAOS_RUN_TAOSBENCHMARK_TEST_ONCE"x = "0"x ]
@ -24,13 +169,37 @@ do
taos -s "select stable_name from information_schema.ins_stables where db_name = 'test';"|grep -q -w meters
if [ $? -ne 0 ]; then
taosBenchmark -y -t 1000 -n 1000 -S 900000
taos -s "create user admin_user pass 'NDS65R6t' sysinfo 0;"
taos -s "GRANT ALL on test.* to admin_user;"
taos -s "create user admin_user pass 'NDS65R6t' sysinfo 0;"
taos -s "GRANT ALL on test.* to admin_user;"
fi
fi
# check taosd status
if [ "$service_state" = "ready" ]; then
# check taosd status
check_taosd
print_service_state_change "ready"
if [ "$service_state" = "error" ]; then
post_error_msg
fi
elif [ "$service_state" = "init" ]; then
check_taosd "ignore"
# check timeout
current_time=`date +%s`
time_elapsed=$(( current_time - taosd_start_time ))
if [ ${time_elapsed} -gt ${TAOSD_STARTUP_TIMEOUT_SECOND} ]; then
set_service_state "error" "taosd startup timeout"
post_error_msg
fi
print_service_state_change "init"
elif [ "$service_state" = "error" ]; then
# check taosd status
check_taosd
print_service_state_change "error"
fi
# check taosadapter
nc -z localhost 6041
if [ $? -ne 0 ]; then
taosadapter &
taosadapter &
fi
sleep 10
done

4
packaging/release.sh
View File

@ -221,12 +221,12 @@ if [[ "$cpuType" == "x64" ]] || [[ "$cpuType" == "aarch64" ]] || [[ "$cpuType" =
# community-version compile
cmake ../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DPAGMODE=${pagMode} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
elif [ "$verMode" == "cloud" ]; then
cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_CLOUD=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_TAOSX=true -DBUILD_CLOUD=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
elif [ "$verMode" == "cluster" ]; then
if [[ "$dbName" != "taos" ]]; then
replace_enterprise_$dbName
fi
cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
cmake ../../ -DCPUTYPE=${cpuType} -DWEBSOCKET=true -DBUILD_TAOSX=true -DOSTYPE=${osType} -DSOMODE=${soMode} -DDBNAME=${dbName} -DVERTYPE=${verType} -DVERDATE="${build_time}" -DGITINFO=${gitinfo} -DGITINFOI=${gitinfoOfInternal} -DVERNUMBER=${verNumber} -DVERCOMPATIBLE=${verNumberComp} -DBUILD_HTTP=${BUILD_HTTP} -DBUILD_TOOLS=${BUILD_TOOLS} ${allocator_macro}
fi
else
echo "input cpuType=${cpuType} error!!!"

107
packaging/testpackage.sh
View File

@ -1,15 +1,72 @@
#!/bin/sh
function usage() {
echo "$0"
echo -e "\t -f test file type,server/client/tools/"
echo -e "\t -m pacakage version Type,community/enterprise"
echo -e "\t -l package type,lite or not"
echo -e "\t -c operation type,x64/arm64"
echo -e "\t -v pacakage version,3.0.1.7"
echo -e "\t -o pacakage version,3.0.1.7"
echo -e "\t -s source Path,web/nas"
echo -e "\t -t package Type,tar/rpm/deb"
echo -e "\t -h help"
}
#parameter
scriptDir=$(dirname $(readlink -f $0))
packgeName=$1
version=$2
originPackageName=$3
originversion=$4
testFile=$5
# sourcePath:web/nas
sourcePath=$6
version="3.0.1.7"
originversion="3.0.1.7"
testFile="server"
verMode="communtity"
sourcePath="nas"
cpuType="x64"
lite="true"
packageType="tar"
subFile="taos.tar.gz"
while getopts "m:c:f:l:s:o:t:v:h" opt; do
case $opt in
m)
verMode=$OPTARG
;;
v)
version=$OPTARG
;;
f)
testFile=$OPTARG
;;
l)
lite=$OPTARG
;;
s)
sourcePath=$OPTARG
;;
o)
originversion=$OPTARG
;;
c)
cpuType=$OPTARG
;;
t)
packageType=$OPTARG
;;
h)
usage
exit 0
;;
?)
echo "Invalid option: -$OPTARG"
usage
exit 0
;;
esac
done
echo "testFile:${testFile},verMode:${verMode},lite:${lite},cpuType:${cpuType},packageType:${packageType},version-${version},originversion:${originversion},sourcePath:${sourcePath}"
# Color setting
RED='\033[41;30m'
GREEN='\033[1;32m'
@ -21,20 +78,40 @@ BLUE_DARK='\033[0;34m'
GREEN_UNDERLINE='\033[4;32m'
NC='\033[0m'
if [ ${testFile} = "server" ];then
tdPath="TDengine-server-${version}"
originTdpPath="TDengine-server-${originversion}"
if [[ ${verMode} = "enterprise" ]];then
prePackag="TDengine-enterprise-${testFile}"
elif [ ${verMode} = "community" ];then
prePackag="TDengine-${testFile}"
fi
if [ ${lite} = "true" ];then
packageLite="-Lite"
elif [ ${lite} = "false" ];then
packageLite=""
fi
if [[ "$packageType" = "tar" ]] ;then
packageType="tar.gz"
fi
tdPath="${prePackag}-${version}"
originTdpPath="${prePackag}-${originversion}"
packgeName="${tdPath}-Linux-${cpuType}${packageLite}.${packageType}"
originPackageName="${originTdpPath}-Linux-${cpuType}${packageLite}.${packageType}"
if [ "$testFile" == "server" ] ;then
installCmd="install.sh"
elif [ ${testFile} = "client" ];then
tdPath="TDengine-client-${version}"
originTdpPath="TDengine-client-${originversion}"
installCmd="install_client.sh"
elif [ ${testFile} = "tools" ];then
tdPath="taosTools-${version}"
originTdpPath="taosTools-${originversion}"
packgeName="${tdPath}-Linux-${cpuType}${packageLite}.${packageType}"
originPackageName="${originTdpPath}-Linux-${cpuType}${packageLite}.${packageType}"
installCmd="install-taostools.sh"
fi
echo "tdPath:${tdPath},originTdpPath:${originTdpPath},packgeName:${packgeName},originPackageName:${originPackageName}"
function cmdInstall {
command=$1
if command -v ${command} ;then
@ -76,16 +153,16 @@ file=$1
versionPath=$2
sourceP=$3
nasServerIP="192.168.1.131"
packagePath="/nas/TDengine/v${versionPath}/community"
packagePath="/nas/TDengine/v${versionPath}/${verMode}"
if [ -f ${file} ];then
echoColor YD "${file} already exists ,it will delete it and download it again "
rm -rf ${file}
fi
if [ ${sourceP} = 'web' ];then
if [[ ${sourceP} = 'web' ]];then
echoColor BD "====download====:wget https://www.taosdata.com/assets-download/3.0/${file}"
wget https://www.taosdata.com/assets-download/3.0/${file}
elif [ ${sourceP} = 'nas' ];then
elif [[ ${sourceP} = 'nas' ]];then
echoColor BD "====download====:scp root@${nasServerIP}:${packagePath}/${file} ."
scp root@${nasServerIP}:${packagePath}/${file} .
fi

3
source/common/src/tdatablock.c
View File

@ -36,7 +36,7 @@ int32_t colDataGetFullLength(const SColumnInfoData* pColumnInfoData, int32_t num
if (IS_VAR_DATA_TYPE(pColumnInfoData->info.type)) {
return pColumnInfoData->varmeta.length + sizeof(int32_t) * numOfRows;
} else {
return pColumnInfoData->info.bytes * numOfRows + BitmapLen(numOfRows);
return ((pColumnInfoData->info.type == TSDB_DATA_TYPE_NULL) ? 0 : pColumnInfoData->info.bytes * numOfRows) + BitmapLen(numOfRows);
}
}
@ -1962,6 +1962,7 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf)
memset(pBuf, 0, sizeof(pBuf));
char* pData = colDataGetVarData(pColInfoData, j);
int32_t dataSize = TMIN(sizeof(pBuf), varDataLen(pData));
dataSize = TMIN(dataSize, 50);
memcpy(pBuf, varDataVal(pData), dataSize);
len += snprintf(dumpBuf + len, size - len, " %15s |", pBuf);
if (len >= size - 1) return dumpBuf;

4
source/common/src/tglobal.c
View File

@ -277,7 +277,9 @@ static int32_t taosAddServerLogCfg(SConfig *pCfg) {
static int32_t taosAddClientCfg(SConfig *pCfg) {
char defaultFqdn[TSDB_FQDN_LEN] = {0};
int32_t defaultServerPort = 6030;
if (taosGetFqdn(defaultFqdn) != 0) return -1;
if (taosGetFqdn(defaultFqdn) != 0) {
strcpy(defaultFqdn, "localhost");
}
if (cfgAddString(pCfg, "firstEp", "", 1) != 0) return -1;
if (cfgAddString(pCfg, "secondEp", "", 1) != 0) return -1;

172
source/common/src/tmsg.c
View File

@ -4643,6 +4643,178 @@ int32_t tDeserializeSMqHbReq(void *buf, int32_t bufLen, SMqHbReq *pReq) {
return 0;
}
int32_t tSerializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq) {
int32_t headLen = sizeof(SMsgHead);
if (buf != NULL) {
buf = (char *)buf + headLen;
bufLen -= headLen;
}
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeU64(&encoder, pReq->sId) < 0) return -1;
if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1;
if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1;
if (tEncodeI64(&encoder, pReq->refId) < 0) return -1;
if (tEncodeI32(&encoder, pReq->execId) < 0) return -1;
if (tEncodeI32(&encoder, pReq->msgMask) < 0) return -1;
if (tEncodeI8(&encoder, pReq->taskType) < 0) return -1;
if (tEncodeI8(&encoder, pReq->explain) < 0) return -1;
if (tEncodeI8(&encoder, pReq->needFetch) < 0) return -1;
if (tEncodeU32(&encoder, pReq->sqlLen) < 0) return -1;
if (tEncodeCStrWithLen(&encoder, pReq->sql, pReq->sqlLen) < 0) return -1;
if (tEncodeU32(&encoder, pReq->msgLen) < 0) return -1;
if (tEncodeBinary(&encoder, (uint8_t*)pReq->msg, pReq->msgLen) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
if (buf != NULL) {
SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen);
pHead->vgId = htonl(pReq->header.vgId);
pHead->contLen = htonl(tlen + headLen);
}
return tlen + headLen;
}
int32_t tDeserializeSSubQueryMsg(void *buf, int32_t bufLen, SSubQueryMsg *pReq) {
int32_t headLen = sizeof(SMsgHead);
SMsgHead *pHead = buf;
pHead->vgId = pReq->header.vgId;
pHead->contLen = pReq->header.contLen;
SDecoder decoder = {0};
tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1;
if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1;
if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->refId) < 0) return -1;
if (tDecodeI32(&decoder, &pReq->execId) < 0) return -1;
if (tDecodeI32(&decoder, &pReq->msgMask) < 0) return -1;
if (tDecodeI8(&decoder, &pReq->taskType) < 0) return -1;
if (tDecodeI8(&decoder, &pReq->explain) < 0) return -1;
if (tDecodeI8(&decoder, &pReq->needFetch) < 0) return -1;
if (tDecodeU32(&decoder, &pReq->sqlLen) < 0) return -1;
if (tDecodeCStrAlloc(&decoder, &pReq->sql) < 0) return -1;
if (tDecodeU32(&decoder, &pReq->msgLen) < 0) return -1;
if (tDecodeBinaryAlloc(&decoder, (void**)&pReq->msg, NULL) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSSubQueryMsg(SSubQueryMsg *pReq) {
if (NULL == pReq) {
return;
}
taosMemoryFreeClear(pReq->sql);
taosMemoryFreeClear(pReq->msg);
}
int32_t tSerializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq) {
int32_t headLen = sizeof(SMsgHead);
if (buf != NULL) {
buf = (char *)buf + headLen;
bufLen -= headLen;
}
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeU64(&encoder, pReq->sId) < 0) return -1;
if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1;
if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1;
if (tEncodeI64(&encoder, pReq->refId) < 0) return -1;
if (tEncodeI32(&encoder, pReq->execId) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
if (buf != NULL) {
SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen);
pHead->vgId = htonl(pReq->header.vgId);
pHead->contLen = htonl(tlen + headLen);
}
return tlen + headLen;
}
int32_t tDeserializeSTaskDropReq(void *buf, int32_t bufLen, STaskDropReq *pReq) {
int32_t headLen = sizeof(SMsgHead);
SMsgHead *pHead = buf;
pHead->vgId = pReq->header.vgId;
pHead->contLen = pReq->header.contLen;
SDecoder decoder = {0};
tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1;
if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1;
if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->refId) < 0) return -1;
if (tDecodeI32(&decoder, &pReq->execId) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
int32_t tSerializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->code) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->tbFName) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->sversion) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->tversion) < 0) return -1;
if (tEncodeI64(&encoder, pRsp->affectedRows) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSQueryTableRsp(void *buf, int32_t bufLen, SQueryTableRsp *pRsp) {
SDecoder decoder = {0};
tDecoderInit(&decoder, (char *)buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->code) < 0) return -1;
if (tDecodeCStrTo(&decoder, pRsp->tbFName) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->sversion) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->tversion) < 0) return -1;
if (tDecodeI64(&decoder, &pRsp->affectedRows) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
int32_t tSerializeSSchedulerHbReq(void *buf, int32_t bufLen, SSchedulerHbReq *pReq) {
int32_t headLen = sizeof(SMsgHead);

4
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
View File

@ -148,8 +148,8 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId);
if (pVnode == NULL) {
dGError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s qtype:%d", pHead->vgId, pMsg, terrstr(),
TMSG_INFO(pMsg->msgType), qtype);
dGError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s qtype:%d contLen:%d", pHead->vgId, pMsg, terrstr(),
TMSG_INFO(pMsg->msgType), qtype, pHead->contLen);
return terrno != 0 ? terrno : -1;
}

1
source/dnode/vnode/inc/vnode.h
View File

@ -239,6 +239,7 @@ int32_t tqReaderSetDataMsg(STqReader *pReader, const SSubmitReq *pMsg, int64_t v
bool tqNextDataBlock(STqReader *pReader);
bool tqNextDataBlockFilterOut(STqReader *pReader, SHashObj *filterOutUids);
int32_t tqRetrieveDataBlock(SSDataBlock *pBlock, STqReader *pReader);
int32_t tqRetrieveTaosxBlock(STqReader *pReader, SArray *blocks, SArray *schemas);
int32_t vnodeEnqueueStreamMsg(SVnode *pVnode, SRpcMsg *pMsg);

5
source/dnode/vnode/src/sma/smaTimeRange.c
View File

@ -203,10 +203,13 @@ static int32_t tdProcessTSmaInsertImpl(SSma *pSma, int64_t indexUid, const char
goto _err;
}
SBatchDeleteReq deleteReq;
SBatchDeleteReq deleteReq = {0};
SSubmitReq *pSubmitReq =
tqBlockToSubmit(pSma->pVnode, (const SArray *)msg, pTsmaStat->pTSchema, &pTsmaStat->pTSma->schemaTag, true,
pTsmaStat->pTSma->dstTbUid, pTsmaStat->pTSma->dstTbName, &deleteReq);
// TODO deleteReq
taosArrayDestroy(deleteReq.deleteReqs);
if (!pSubmitReq) {
smaError("vgId:%d, failed to gen submit blk while tsma insert for smaIndex %" PRIi64 " since %s", SMA_VID(pSma),

1
source/dnode/vnode/src/tq/tq.c
View File

@ -671,7 +671,6 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg) {
SSubmitReq* pCont = (SSubmitReq*)&pHead->body;
if (tqTaosxScanLog(pTq, pHandle, pCont, &taosxRsp) < 0) {
/*ASSERT(0);*/
}
if (taosxRsp.blockNum > 0 /* threshold */) {
tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer);

86
source/dnode/vnode/src/tq/tqExec.c
View File

@ -44,7 +44,7 @@ static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, SMqDataRsp* pRs
return 0;
}
static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) {
static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, int32_t n) {
SMetaReader mr = {0};
metaReaderInit(&mr, pTq->pVnode->pMeta, 0);
// TODO add reference to gurantee success
@ -52,8 +52,10 @@ static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) {
metaReaderClear(&mr);
return -1;
}
char* tbName = strdup(mr.me.name);
taosArrayPush(pRsp->blockTbName, &tbName);
for (int32_t i = 0; i < n; i++) {
char* tbName = strdup(mr.me.name);
taosArrayPush(pRsp->blockTbName, &tbName);
}
metaReaderClear(&mr);
return 0;
}
@ -111,7 +113,7 @@ int32_t tqScanData(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, STqOffs
if (pRsp->withTbName) {
if (pRsp->rspOffset.type == TMQ_OFFSET__LOG) {
int64_t uid = pExec->pExecReader->msgIter.uid;
tqAddTbNameToRsp(pTq, uid, pRsp);
tqAddTbNameToRsp(pTq, uid, pRsp, 1);
} else {
pRsp->withTbName = false;
}
@ -155,7 +157,7 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, STaosxRsp* pRsp, SMqMeta
int64_t uid = 0;
if (pOffset->type == TMQ_OFFSET__LOG) {
uid = pExec->pExecReader->msgIter.uid;
if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) {
if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, 1) < 0) {
continue;
}
} else {
@ -225,18 +227,30 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp
STqExecHandle* pExec = &pHandle->execHandle;
ASSERT(pExec->subType != TOPIC_SUB_TYPE__COLUMN);
SArray* pBlocks = taosArrayInit(0, sizeof(SSDataBlock));
SArray* pSchemas = taosArrayInit(0, sizeof(void*));
if (pExec->subType == TOPIC_SUB_TYPE__TABLE) {
STqReader* pReader = pExec->pExecReader;
tqReaderSetDataMsg(pReader, pReq, 0);
while (tqNextDataBlock(pReader)) {
SSDataBlock block = {0};
if (tqRetrieveDataBlock(&block, pReader) < 0) {
/*SSDataBlock block = {0};*/
/*if (tqRetrieveDataBlock(&block, pReader) < 0) {*/
/*if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;*/
/*}*/
taosArrayClear(pBlocks);
taosArrayClear(pSchemas);
if (tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas) < 0) {
if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;
}
if (pRsp->withTbName) {
int64_t uid = pExec->pExecReader->msgIter.uid;
if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) {
blockDataFreeRes(&block);
if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlocks)) < 0) {
taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes);
taosArrayDestroyP(pSchemas, (FDelete)tDeleteSSchemaWrapper);
pBlocks = taosArrayInit(0, sizeof(SSDataBlock));
pSchemas = taosArrayInit(0, sizeof(void*));
continue;
}
}
@ -255,25 +269,37 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp
pRsp->createTableNum++;
}
}
tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),
pTq->pVnode->config.tsdbCfg.precision);
blockDataFreeRes(&block);
tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);
pRsp->blockNum++;
for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) {
SSDataBlock* pBlock = taosArrayGet(pBlocks, i);
tqAddBlockDataToRsp(pBlock, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlock->pDataBlock),
pTq->pVnode->config.tsdbCfg.precision);
blockDataFreeRes(pBlock);
SSchemaWrapper* pSW = taosArrayGetP(pSchemas, i);
taosArrayPush(pRsp->blockSchema, &pSW);
pRsp->blockNum++;
}
}
} else if (pExec->subType == TOPIC_SUB_TYPE__DB) {
STqReader* pReader = pExec->pExecReader;
tqReaderSetDataMsg(pReader, pReq, 0);
while (tqNextDataBlockFilterOut(pReader, pExec->execDb.pFilterOutTbUid)) {
SSDataBlock block = {0};
if (tqRetrieveDataBlock(&block, pReader) < 0) {
/*SSDataBlock block = {0};*/
/*if (tqRetrieveDataBlock(&block, pReader) < 0) {*/
/*if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;*/
/*}*/
taosArrayClear(pBlocks);
taosArrayClear(pSchemas);
if (tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas) < 0) {
if (terrno == TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND) continue;
}
if (pRsp->withTbName) {
int64_t uid = pExec->pExecReader->msgIter.uid;
if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp) < 0) {
blockDataFreeRes(&block);
continue;
if (tqAddTbNameToRsp(pTq, uid, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlocks)) < 0) {
taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes);
taosArrayDestroyP(pSchemas, (FDelete)tDeleteSSchemaWrapper);
pBlocks = taosArrayInit(0, sizeof(SSDataBlock));
pSchemas = taosArrayInit(0, sizeof(void*));
return -1;
}
}
if (pHandle->fetchMeta) {
@ -291,14 +317,26 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SSubmitReq* pReq, STaosxRsp
pRsp->createTableNum++;
}
}
tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),
pTq->pVnode->config.tsdbCfg.precision);
blockDataFreeRes(&block);
tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);
pRsp->blockNum++;
/*tqAddBlockDataToRsp(&block, (SMqDataRsp*)pRsp, taosArrayGetSize(block.pDataBlock),*/
/*pTq->pVnode->config.tsdbCfg.precision);*/
/*blockDataFreeRes(&block);*/
/*tqAddBlockSchemaToRsp(pExec, (SMqDataRsp*)pRsp);*/
/*pRsp->blockNum++;*/
for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) {
SSDataBlock* pBlock = taosArrayGet(pBlocks, i);
tqAddBlockDataToRsp(pBlock, (SMqDataRsp*)pRsp, taosArrayGetSize(pBlock->pDataBlock),
pTq->pVnode->config.tsdbCfg.precision);
blockDataFreeRes(pBlock);
SSchemaWrapper* pSW = taosArrayGetP(pSchemas, i);
taosArrayPush(pRsp->blockSchema, &pSW);
pRsp->blockNum++;
}
}
}
taosArrayDestroy(pBlocks);
taosArrayDestroy(pSchemas);
if (pRsp->blockNum == 0) {
return -1;
}

40
source/dnode/vnode/src/tq/tqRead.c
View File

@ -556,7 +556,7 @@ FAIL:
return -1;
}
int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* schemas) {
int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas) {
int32_t sversion = htonl(pReader->pBlock->sversion);
if (pReader->cachedSchemaSuid == 0 || pReader->cachedSchemaVer != sversion ||
@ -592,9 +592,10 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
int32_t colAtMost = pSchemaWrapper->nCols;
int32_t curRow = 0;
int32_t lastRow = 0;
char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols);
if (assigned) return -1;
if (assigned == NULL) return -1;
tInitSubmitBlkIter(&pReader->msgIter, pReader->pBlock, &pReader->blkIter);
STSRowIter iter = {0};
@ -605,11 +606,13 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
bool buildNew = false;
tdSTSRowIterReset(&iter, row);
tqDebug("vgId:%d, row of block %d", pReader->pWalReader->pWal->cfg.vgId, curRow);
for (int32_t i = 0; i < colAtMost; i++) {
SCellVal sVal = {0};
if (!tdSTSRowIterFetch(&iter, pSchemaWrapper->pSchema[i].colId, pSchemaWrapper->pSchema[i].type, &sVal)) {
break;
}
tqDebug("vgId:%d, %d col, type %d", pReader->pWalReader->pWal->cfg.vgId, i, sVal.valType);
if (curRow == 0) {
assigned[i] = sVal.valType != TD_VTYPE_NONE;
buildNew = true;
@ -623,27 +626,42 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
}
if (buildNew) {
SSDataBlock block;
SSchemaWrapper sw;
if (tqMaskBlock(&sw, &block, pSchemaWrapper, assigned) < 0) {
if (taosArrayGetSize(blocks) > 0) {
SSDataBlock* pLastBlock = taosArrayGetLast(blocks);
pLastBlock->info.rows = curRow - lastRow;
lastRow = curRow;
}
SSDataBlock* pBlock = createDataBlock();
SSchemaWrapper* pSW = taosMemoryCalloc(1, sizeof(SSchemaWrapper));
if (tqMaskBlock(pSW, pBlock, pSchemaWrapper, assigned) < 0) {
blockDataDestroy(pBlock);
goto FAIL;
}
SSDataBlock block = {0};
assignOneDataBlock(&block, pBlock);
blockDataDestroy(pBlock);
tqDebug("vgId:%d, build new block, col %d", pReader->pWalReader->pWal->cfg.vgId,
(int32_t)taosArrayGetSize(block.pDataBlock));
taosArrayPush(blocks, &block);
taosArrayPush(schemas, &sw);
taosArrayPush(schemas, &pSW);
}
SSDataBlock* pBlock = taosArrayGetLast(blocks);
pBlock->info.uid = pReader->msgIter.uid;
pBlock->info.rows = pReader->msgIter.numOfRows;
pBlock->info.rows = 0;
pBlock->info.version = pReader->pMsg->version;
tqDebug("vgId:%d, taosx scan, block num: %d", pReader->pWalReader->pWal->cfg.vgId,
(int32_t)taosArrayGetSize(blocks));
if (blockDataEnsureCapacity(pBlock, pReader->msgIter.numOfRows - curRow) < 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto FAIL;
}
tdSTSRowIterInit(&iter, pTschema);
tdSTSRowIterReset(&iter, row);
for (int32_t i = 0; i < taosArrayGetSize(pBlock->pDataBlock); i++) {
SColumnInfoData* pColData = taosArrayGet(pBlock->pDataBlock, i);
SCellVal sVal = {0};
@ -654,12 +672,16 @@ int32_t tqSplitRetrieveDataBlock(STqReader* pReader, SArray* blocks, SArray* sch
ASSERT(sVal.valType != TD_VTYPE_NONE);
if (colDataAppend(pColData, curRow, sVal.val, sVal.valType != TD_VTYPE_NORM) < 0) {
if (colDataAppend(pColData, curRow, sVal.val, sVal.valType == TD_VTYPE_NULL) < 0) {
goto FAIL;
}
tqDebug("vgId:%d, row %d col %d append %d", pReader->pWalReader->pWal->cfg.vgId, curRow, i,
sVal.valType == TD_VTYPE_NULL);
}
curRow++;
}
SSDataBlock* pLastBlock = taosArrayGetLast(blocks);
pLastBlock->info.rows = curRow - lastRow;
taosMemoryFree(assigned);
return 0;

21
source/dnode/vnode/src/tq/tqSink.c
View File

@ -236,6 +236,7 @@ SSubmitReq* tqBlockToSubmit(SVnode* pVnode, const SArray* pBlocks, const STSchem
SSDataBlock* pDataBlock = taosArrayGet(pBlocks, i);
if (pDataBlock->info.type == STREAM_DELETE_RESULT) {
pDeleteReq->suid = suid;
pDeleteReq->deleteReqs = taosArrayInit(0, sizeof(SSingleDeleteReq));
tqBuildDeleteReq(pVnode, stbFullName, pDataBlock, pDeleteReq);
continue;
}
@ -349,7 +350,6 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
.contLen = len + sizeof(SMsgHead),
};
if (tmsgPutToQueue(&pVnode->msgCb, WRITE_QUEUE, &msg) != 0) {
rpcFreeCont(serializedDeleteReq);
tqDebug("failed to put delete req into write-queue since %s", terrstr());
}
} else {
@ -476,12 +476,12 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
cap += sizeof(SSubmitBlk) + schemaLen + rows * maxLen;
SSubmitReq* ret = rpcMallocCont(cap);
ret->header.vgId = pVnode->config.vgId;
ret->length = sizeof(SSubmitReq);
ret->numOfBlocks = htonl(1);
SSubmitReq* pSubmit = rpcMallocCont(cap);
pSubmit->header.vgId = pVnode->config.vgId;
pSubmit->length = sizeof(SSubmitReq);
pSubmit->numOfBlocks = htonl(1);
SSubmitBlk* blkHead = POINTER_SHIFT(ret, sizeof(SSubmitReq));
SSubmitBlk* blkHead = POINTER_SHIFT(pSubmit, sizeof(SSubmitReq));
blkHead->numOfRows = htonl(pDataBlock->info.rows);
blkHead->sversion = htonl(pTSchema->version);
@ -531,17 +531,16 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
}
blkHead->dataLen = htonl(dataLen);
ret->length += sizeof(SSubmitBlk) + schemaLen + dataLen;
ret->length = htonl(ret->length);
pSubmit->length += sizeof(SSubmitBlk) + schemaLen + dataLen;
pSubmit->length = htonl(pSubmit->length);
SRpcMsg msg = {
.msgType = TDMT_VND_SUBMIT,
.pCont = ret,
.contLen = ntohl(ret->length),
.pCont = pSubmit,
.contLen = ntohl(pSubmit->length),
};
if (tmsgPutToQueue(&pVnode->msgCb, WRITE_QUEUE, &msg) != 0) {
rpcFreeCont(ret);
tqDebug("failed to put into write-queue since %s", terrstr());
}
}

15
source/dnode/vnode/src/tsdb/tsdbCacheRead.c
View File

@ -287,12 +287,17 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32
hasRes = true;
p->ts = pColVal->ts;
uint8_t* px = p->colVal.value.pData;
p->colVal = pColVal->colVal;
if (!IS_VAR_DATA_TYPE(pColVal->colVal.type)) {
p->colVal = pColVal->colVal;
} else {
if (COL_VAL_IS_VALUE(&pColVal->colVal)) {
memcpy(p->colVal.value.pData, pColVal->colVal.value.pData, pColVal->colVal.value.nData);
}
if (COL_VAL_IS_VALUE(&pColVal->colVal) && IS_VAR_DATA_TYPE(pColVal->colVal.type)) {
p->colVal.value.pData = px;
memcpy(px, pColVal->colVal.value.pData, pColVal->colVal.value.nData);
p->colVal.value.nData = pColVal->colVal.value.nData;
p->colVal.type = pColVal->colVal.type;
p->colVal.flag = pColVal->colVal.flag;
p->colVal.cid = pColVal->colVal.cid;
}
}
}

49
source/dnode/vnode/src/tsdb/tsdbRead.c
View File

@ -305,7 +305,7 @@ static void* getPosInBlockInfoBuf(SBlockInfoBuf* pBuf, int32_t index) {
}
// NOTE: speedup the whole processing by preparing the buffer for STableBlockScanInfo in batch model
static SHashObj* createDataBlockScanInfo(STsdbReader* pTsdbReader, const STableKeyInfo* idList, int32_t numOfTables) {
static SHashObj* createDataBlockScanInfo(STsdbReader* pTsdbReader, SBlockInfoBuf* pBuf, const STableKeyInfo* idList, int32_t numOfTables) {
// allocate buffer in order to load data blocks from file
// todo use simple hash instead, optimize the memory consumption
SHashObj* pTableMap =
@ -315,10 +315,10 @@ static SHashObj* createDataBlockScanInfo(STsdbReader* pTsdbReader, const STableK
}
int64_t st = taosGetTimestampUs();
initBlockScanInfoBuf(&pTsdbReader->blockInfoBuf, numOfTables);
initBlockScanInfoBuf(pBuf, numOfTables);
for (int32_t j = 0; j < numOfTables; ++j) {
STableBlockScanInfo* pScanInfo = getPosInBlockInfoBuf(&pTsdbReader->blockInfoBuf, j);
STableBlockScanInfo* pScanInfo = getPosInBlockInfoBuf(pBuf, j);
pScanInfo->uid = idList[j].uid;
if (ASCENDING_TRAVERSE(pTsdbReader->order)) {
int64_t skey = pTsdbReader->window.skey;
@ -329,20 +329,6 @@ static SHashObj* createDataBlockScanInfo(STsdbReader* pTsdbReader, const STableK
}
taosHashPut(pTableMap, &pScanInfo->uid, sizeof(uint64_t), &pScanInfo, POINTER_BYTES);
#if 0
// STableBlockScanInfo info = {.lastKey = 0, .uid = idList[j].uid};
if (ASCENDING_TRAVERSE(pTsdbReader->order)) {
int64_t skey = pTsdbReader->window.skey;
info.lastKey = (skey > INT64_MIN) ? (skey - 1) : skey;
} else {
int64_t ekey = pTsdbReader->window.ekey;
info.lastKey = (ekey < INT64_MAX) ? (ekey + 1) : ekey;
}
taosHashPut(pTableMap, &info.uid, sizeof(uint64_t), &info, sizeof(info));
#endif
tsdbTrace("%p check table uid:%" PRId64 " from lastKey:%" PRId64 " %s", pTsdbReader, pScanInfo->uid,
pScanInfo->lastKey, pTsdbReader->idStr);
}
@ -361,11 +347,17 @@ static void resetAllDataBlockScanInfo(SHashObj* pTableMap, int64_t ts) {
STableBlockScanInfo* pInfo = *(STableBlockScanInfo**)p;
pInfo->iterInit = false;
pInfo->iter.hasVal = false;
pInfo->iiter.hasVal = false;
if (pInfo->iter.iter != NULL) {
pInfo->iter.iter = tsdbTbDataIterDestroy(pInfo->iter.iter);
}
if (pInfo->iiter.iter != NULL) {
pInfo->iiter.iter = tsdbTbDataIterDestroy(pInfo->iiter.iter);
}
pInfo->delSkyline = taosArrayDestroy(pInfo->delSkyline);
pInfo->lastKey = ts;
}
@ -373,6 +365,8 @@ static void resetAllDataBlockScanInfo(SHashObj* pTableMap, int64_t ts) {
static void clearBlockScanInfo(STableBlockScanInfo* p) {
p->iterInit = false;
p->iter.hasVal = false;
p->iiter.hasVal = false;
if (p->iter.iter != NULL) {
@ -388,9 +382,9 @@ static void clearBlockScanInfo(STableBlockScanInfo* p) {
tMapDataClear(&p->mapData);
}
static void destroyAllBlockScanInfo(SHashObj* pTableMap, bool clearEntry) {
static void destroyAllBlockScanInfo(SHashObj* pTableMap) {
void* p = NULL;
while (clearEntry && ((p = taosHashIterate(pTableMap, p)) != NULL)) {
while ((p = taosHashIterate(pTableMap, p)) != NULL) {
clearBlockScanInfo(*(STableBlockScanInfo**)p);
}
@ -2226,6 +2220,7 @@ static int32_t initMemDataIterator(STableBlockScanInfo* pBlockScanInfo, STsdbRea
if (pReader->pReadSnap->pMem != NULL) {
d = tsdbGetTbDataFromMemTable(pReader->pReadSnap->pMem, pReader->suid, pBlockScanInfo->uid);
if (d != NULL) {
ASSERT(pBlockScanInfo->iter.iter == NULL);
code = tsdbTbDataIterCreate(d, &startKey, backward, &pBlockScanInfo->iter.iter);
if (code == TSDB_CODE_SUCCESS) {
pBlockScanInfo->iter.hasVal = (tsdbTbDataIterGet(pBlockScanInfo->iter.iter) != NULL);
@ -3790,11 +3785,10 @@ int32_t tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, void* pTableL
updateBlockSMAInfo(pReader->pSchema, &pReader->suppInfo);
}
STsdbReader* p = pReader->innerReader[0] != NULL ? pReader->innerReader[0] : pReader;
pReader->status.pTableMap = createDataBlockScanInfo(p, pTableList, numOfTables);
STsdbReader* p = (pReader->innerReader[0] != NULL)? pReader->innerReader[0]:pReader;
pReader->status.pTableMap = createDataBlockScanInfo(p, &pReader->blockInfoBuf, pTableList, numOfTables);
if (pReader->status.pTableMap == NULL) {
tsdbReaderClose(pReader);
tsdbReaderClose(p);
*ppReader = NULL;
code = TSDB_CODE_TDB_OUT_OF_MEMORY;
@ -3850,7 +3844,7 @@ void tsdbReaderClose(STsdbReader* pReader) {
}
{
if (pReader->innerReader[0] != NULL) {
if (pReader->innerReader[0] != NULL || pReader->innerReader[1] != NULL) {
STsdbReader* p = pReader->innerReader[0];
p->status.pTableMap = NULL;
@ -3888,9 +3882,12 @@ void tsdbReaderClose(STsdbReader* pReader) {
cleanupDataBlockIterator(&pReader->status.blockIter);
size_t numOfTables = taosHashGetSize(pReader->status.pTableMap);
destroyAllBlockScanInfo(pReader->status.pTableMap, (pReader->innerReader[0] == NULL) ? true : false);
if (pReader->status.pTableMap != NULL) {
destroyAllBlockScanInfo(pReader->status.pTableMap);
clearBlockScanInfoBuf(&pReader->blockInfoBuf);
}
blockDataDestroy(pReader->pResBlock);
clearBlockScanInfoBuf(&pReader->blockInfoBuf);
if (pReader->pFileReader != NULL) {
tsdbDataFReaderClose(&pReader->pFileReader);

2
source/dnode/vnode/src/tsdb/tsdbUtil.c
View File

@ -1698,7 +1698,7 @@ int32_t tsdbCmprColData(SColData *pColData, int8_t cmprAlg, SBlockCol *pBlockCol
size += pBlockCol->szBitmap;
// offset
if (IS_VAR_DATA_TYPE(pColData->type)) {
if (IS_VAR_DATA_TYPE(pColData->type) && pColData->flag != (HAS_NULL | HAS_NONE)) {
code = tsdbCmprData((uint8_t *)pColData->aOffset, sizeof(int32_t) * pColData->nVal, TSDB_DATA_TYPE_INT, cmprAlg,
ppOut, nOut + size, &pBlockCol->szOffset, ppBuf);
if (code) goto _exit;

1
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -609,6 +609,7 @@ static int32_t vnodeProcessCreateTbReq(SVnode *pVnode, int64_t version, void *pR
_exit:
for (int32_t iReq = 0; iReq < req.nReqs; iReq++) {
pCreateReq = req.pReqs + iReq;
taosMemoryFree(pCreateReq->comment);
taosArrayDestroy(pCreateReq->ctb.tagName);
}
taosArrayDestroyEx(rsp.pArray, tFreeSVCreateTbRsp);

19
source/libs/executor/inc/executorimpl.h
View File

@ -239,6 +239,7 @@ typedef struct SSourceDataInfo {
int32_t index;
SRetrieveTableRsp* pRsp;
uint64_t totalRows;
int64_t startTime;
int32_t code;
EX_SOURCE_STATUS status;
const char* taskId;
@ -753,6 +754,23 @@ typedef struct SStreamPartitionOperatorInfo {
SSDataBlock* pDelRes;
} SStreamPartitionOperatorInfo;
typedef struct SStreamFillSupporter {
int32_t type; // fill type
SInterval interval;
SResultRowData prev;
SResultRowData cur;
SResultRowData next;
SResultRowData nextNext;
SFillColInfo* pAllColInfo; // fill exprs and not fill exprs
SExprSupp notFillExprSup;
int32_t numOfAllCols; // number of all exprs, including the tags columns
int32_t numOfFillCols;
int32_t numOfNotFillCols;
int32_t rowSize;
SSHashObj* pResMap;
bool hasDelete;
} SStreamFillSupporter;
typedef struct SStreamFillOperatorInfo {
SStreamFillSupporter* pFillSup;
SSDataBlock* pRes;
@ -776,7 +794,6 @@ typedef struct STimeSliceOperatorInfo {
SArray* pPrevRow; // SArray<SGroupValue>
SArray* pNextRow; // SArray<SGroupValue>
SArray* pLinearInfo; // SArray<SFillLinearInfo>
bool fillLastPoint;
bool isPrevRowSet;
bool isNextRowSet;
int32_t fillType; // fill type

19
source/libs/executor/inc/tfill.h
View File

@ -36,7 +36,8 @@ typedef struct SFillColInfo {
typedef struct SFillLinearInfo {
SPoint start;
SPoint end;
bool hasNull;
bool isStartSet;
bool isEndSet;
int16_t type;
int32_t bytes;
} SFillLinearInfo;
@ -110,22 +111,6 @@ typedef struct SStreamFillInfo {
int32_t delIndex;
} SStreamFillInfo;
typedef struct SStreamFillSupporter {
int32_t type; // fill type
SInterval interval;
SResultRowData prev;
SResultRowData cur;
SResultRowData next;
SResultRowData nextNext;
SFillColInfo* pAllColInfo; // fill exprs and not fill exprs
int32_t numOfAllCols; // number of all exprs, including the tags columns
int32_t numOfFillCols;
int32_t numOfNotFillCols;
int32_t rowSize;
SSHashObj* pResMap;
bool hasDelete;
} SStreamFillSupporter;
int64_t getNumOfResultsAfterFillGap(SFillInfo* pFillInfo, int64_t ekey, int32_t maxNumOfRows);
void taosFillSetStartInfo(struct SFillInfo* pFillInfo, int32_t numOfRows, TSKEY endKey);

11
source/libs/executor/inc/tsort.h
View File

@ -36,12 +36,13 @@ typedef struct SMultiMergeSource {
typedef struct SSortSource {
SMultiMergeSource src;
union {
struct {
SArray* pageIdList;
int32_t pageIndex;
};
struct {
SArray* pageIdList;
int32_t pageIndex;
};
struct {
void* param;
bool onlyRef;
};
} SSortSource;

20
source/libs/executor/src/cachescanoperator.c
View File

@ -26,7 +26,7 @@
#include "ttypes.h"
static SSDataBlock* doScanCache(SOperatorInfo* pOperator);
static void destroyLastrowScanOperator(void* param);
static void destroyCacheScanOperator(void* param);
static int32_t extractCacheScanSlotId(const SArray* pColMatchInfo, SExecTaskInfo* pTaskInfo, int32_t** pSlotIds);
static int32_t removeRedundantTsCol(SLastRowScanPhysiNode* pScanNode, SColMatchInfo* pColMatchInfo);
@ -97,14 +97,14 @@ SOperatorInfo* createCacherowsScanOperator(SLastRowScanPhysiNode* pScanNode, SRe
pOperator->exprSupp.numOfExprs = taosArrayGetSize(pInfo->pRes->pDataBlock);
pOperator->fpSet =
createOperatorFpSet(operatorDummyOpenFn, doScanCache, NULL, destroyLastrowScanOperator, NULL);
createOperatorFpSet(operatorDummyOpenFn, doScanCache, NULL, destroyCacheScanOperator, NULL);
pOperator->cost.openCost = 0;
return pOperator;
_error:
pTaskInfo->code = code;
destroyLastrowScanOperator(pInfo);
destroyCacheScanOperator(pInfo);
taosMemoryFree(pOperator);
return NULL;
}
@ -157,9 +157,12 @@ SSDataBlock* doScanCache(SOperatorInfo* pOperator) {
SColumnInfoData* pSrc = taosArrayGet(pInfo->pBufferredRes->pDataBlock, slotId);
SColumnInfoData* pDst = taosArrayGet(pRes->pDataBlock, slotId);
char* p = colDataGetData(pSrc, pInfo->indexOfBufferedRes);
bool isNull = colDataIsNull_s(pSrc, pInfo->indexOfBufferedRes);
colDataAppend(pDst, 0, p, isNull);
if (colDataIsNull_s(pSrc, pInfo->indexOfBufferedRes)) {
colDataAppendNULL(pDst, 0);
} else {
char* p = colDataGetData(pSrc, pInfo->indexOfBufferedRes);
colDataAppend(pDst, 0, p, false);
}
}
pRes->info.uid = *(tb_uid_t*)taosArrayGet(pInfo->pUidList, pInfo->indexOfBufferedRes);
@ -226,6 +229,8 @@ SSDataBlock* doScanCache(SOperatorInfo* pOperator) {
pInfo->pLastrowReader = tsdbCacherowsReaderClose(pInfo->pLastrowReader);
return pInfo->pRes;
} else {
pInfo->pLastrowReader = tsdbCacherowsReaderClose(pInfo->pLastrowReader);
}
}
@ -234,7 +239,7 @@ SSDataBlock* doScanCache(SOperatorInfo* pOperator) {
}
}
void destroyLastrowScanOperator(void* param) {
void destroyCacheScanOperator(void* param) {
SLastrowScanInfo* pInfo = (SLastrowScanInfo*)param;
blockDataDestroy(pInfo->pRes);
blockDataDestroy(pInfo->pBufferredRes);
@ -246,6 +251,7 @@ void destroyLastrowScanOperator(void* param) {
pInfo->pLastrowReader = tsdbCacherowsReaderClose(pInfo->pLastrowReader);
}
cleanupExprSupp(&pInfo->pseudoExprSup);
taosMemoryFreeClear(param);
}

20
source/libs/executor/src/exchangeoperator.c
View File

@ -44,7 +44,7 @@ typedef struct SFetchRspHandleWrapper {
static void destroyExchangeOperatorInfo(void* param);
static void freeBlock(void* pParam);
static void freeSourceDataInfo(void* param);
static void* setAllSourcesCompleted(SOperatorInfo* pOperator, int64_t startTs);
static void* setAllSourcesCompleted(SOperatorInfo* pOperator);
static int32_t loadRemoteDataCallback(void* param, SDataBuf* pMsg, int32_t code);
static int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInfo* pTaskInfo, int32_t sourceIndex);
@ -59,7 +59,7 @@ static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeIn
size_t totalSources = taosArrayGetSize(pExchangeInfo->pSourceDataInfo);
int32_t completed = getCompletedSources(pExchangeInfo->pSourceDataInfo);
if (completed == totalSources) {
setAllSourcesCompleted(pOperator, pExchangeInfo->openedTs);
setAllSourcesCompleted(pOperator);
return;
}
@ -113,7 +113,8 @@ static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeIn
taosArrayPush(pExchangeInfo->pResultBlockList, &pb);
}
updateLoadRemoteInfo(pLoadInfo, pRetrieveRsp->numOfRows, pRetrieveRsp->compLen, pExchangeInfo->openedTs, pOperator);
updateLoadRemoteInfo(pLoadInfo, pRetrieveRsp->numOfRows, pRetrieveRsp->compLen, pDataInfo->startTime, pOperator);
pDataInfo->totalRows += pRetrieveRsp->numOfRows;
if (pRsp->completed == 1) {
pDataInfo->status = EX_SOURCE_DATA_EXHAUSTED;
@ -388,6 +389,7 @@ int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInfo* pTas
SDownstreamSourceNode* pSource = taosArrayGet(pExchangeInfo->pSources, sourceIndex);
SSourceDataInfo* pDataInfo = taosArrayGet(pExchangeInfo->pSourceDataInfo, sourceIndex);
pDataInfo->startTime = taosGetTimestampUs();
ASSERT(pDataInfo->status == EX_SOURCE_DATA_NOT_READY);
@ -493,18 +495,14 @@ int32_t extractDataBlockFromFetchRsp(SSDataBlock* pRes, char* pData, SArray* pCo
return TSDB_CODE_SUCCESS;
}
void* setAllSourcesCompleted(SOperatorInfo* pOperator, int64_t startTs) {
void* setAllSourcesCompleted(SOperatorInfo* pOperator) {
SExchangeInfo* pExchangeInfo = pOperator->info;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
int64_t el = taosGetTimestampUs() - startTs;
SLoadRemoteDataInfo* pLoadInfo = &pExchangeInfo->loadInfo;
pLoadInfo->totalElapsed += el;
size_t totalSources = taosArrayGetSize(pExchangeInfo->pSources);
qDebug("%s all %" PRIzu " sources are exhausted, total rows: %" PRIu64 " bytes:%" PRIu64 ", elapsed:%.2f ms",
GET_TASKID(pTaskInfo), totalSources, pLoadInfo->totalRows, pLoadInfo->totalSize,
qDebug("%s all %" PRIzu " sources are exhausted, total rows: %" PRIu64 ", %.2f Kb, elapsed:%.2f ms",
GET_TASKID(pTaskInfo), totalSources, pLoadInfo->totalRows, pLoadInfo->totalSize / 1024.0,
pLoadInfo->totalElapsed / 1000.0);
setOperatorCompleted(pOperator);
@ -566,7 +564,7 @@ int32_t seqLoadRemoteData(SOperatorInfo* pOperator) {
while (1) {
if (pExchangeInfo->current >= totalSources) {
setAllSourcesCompleted(pOperator, startTs);
setAllSourcesCompleted(pOperator);
return TSDB_CODE_SUCCESS;
}

42
source/libs/executor/src/executorimpl.c
View File

@ -20,7 +20,6 @@
#include "querynodes.h"
#include "tfill.h"
#include "tname.h"
#include "tref.h"
#include "tdatablock.h"
#include "tglobal.h"
@ -134,45 +133,6 @@ static int32_t doCopyToSDataBlock(SExecTaskInfo* pTaskInfo, SSDataBlock* pBlock,
static void initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size);
static void doSetTableGroupOutputBuf(SOperatorInfo* pOperator, int32_t numOfOutput, uint64_t groupId);
#if 0
static bool chkResultRowFromKey(STaskRuntimeEnv* pRuntimeEnv, SResultRowInfo* pResultRowInfo, char* pData,
int16_t bytes, bool masterscan, uint64_t uid) {
bool existed = false;
SET_RES_WINDOW_KEY(pRuntimeEnv->keyBuf, pData, bytes, uid);
SResultRow** p1 =
(SResultRow**)taosHashGet(pRuntimeEnv->pResultRowHashTable, pRuntimeEnv->keyBuf, GET_RES_WINDOW_KEY_LEN(bytes));
// in case of repeat scan/reverse scan, no new time window added.
if (QUERY_IS_INTERVAL_QUERY(pRuntimeEnv->pQueryAttr)) {
if (!masterscan) { // the *p1 may be NULL in case of sliding+offset exists.
return p1 != NULL;
}
if (p1 != NULL) {
if (pResultRowInfo->size == 0) {
existed = false;
} else if (pResultRowInfo->size == 1) {
// existed = (pResultRowInfo->pResult[0] == (*p1));
} else { // check if current pResultRowInfo contains the existed pResultRow
SET_RES_EXT_WINDOW_KEY(pRuntimeEnv->keyBuf, pData, bytes, uid, pResultRowInfo);
int64_t* index =
taosHashGet(pRuntimeEnv->pResultRowListSet, pRuntimeEnv->keyBuf, GET_RES_EXT_WINDOW_KEY_LEN(bytes));
if (index != NULL) {
existed = true;
} else {
existed = false;
}
}
}
return existed;
}
return p1 != NULL;
}
#endif
SResultRow* getNewResultRow(SDiskbasedBuf* pResultBuf, int32_t* currentPageId, int32_t interBufSize) {
SFilePage* pData = NULL;
@ -2096,6 +2056,8 @@ void destroyExprInfo(SExprInfo* pExpr, int32_t numOfExprs) {
for (int32_t j = 0; j < pExprInfo->base.numOfParams; ++j) {
if (pExprInfo->base.pParam[j].type == FUNC_PARAM_TYPE_COLUMN) {
taosMemoryFreeClear(pExprInfo->base.pParam[j].pCol);
} else if (pExprInfo->base.pParam[j].type == FUNC_PARAM_TYPE_VALUE) {
taosVariantDestroy(&pExprInfo->base.pParam[j].param);
}
}

1
source/libs/executor/src/scanoperator.c
View File

@ -4532,6 +4532,7 @@ int32_t startGroupTableMergeScan(SOperatorInfo* pOperator) {
SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource));
STableMergeScanSortSourceParam* param = taosArrayGet(pInfo->sortSourceParams, i);
ps->param = param;
ps->onlyRef = true;
tsortAddSource(pInfo->pSortHandle, ps);
}

12
source/libs/executor/src/sortoperator.c
View File

@ -176,10 +176,10 @@ int32_t doOpenSortOperator(SOperatorInfo* pOperator) {
SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource));
ps->param = pOperator->pDownstream[0];
ps->onlyRef = true;
tsortAddSource(pInfo->pSortHandle, ps);
int32_t code = tsortOpen(pInfo->pSortHandle);
taosMemoryFreeClear(ps);
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, terrno);
@ -377,10 +377,10 @@ int32_t beginSortGroup(SOperatorInfo* pOperator) {
param->childOpInfo = pOperator->pDownstream[0];
param->grpSortOpInfo = pInfo;
ps->param = param;
ps->onlyRef = false;
tsortAddSource(pInfo->pCurrSortHandle, ps);
int32_t code = tsortOpen(pInfo->pCurrSortHandle);
taosMemoryFreeClear(ps);
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, terrno);
@ -471,6 +471,9 @@ void destroyGroupSortOperatorInfo(void* param) {
taosArrayDestroy(pInfo->pSortInfo);
taosArrayDestroy(pInfo->matchInfo.pList);
tsortDestroySortHandle(pInfo->pCurrSortHandle);
pInfo->pCurrSortHandle = NULL;
taosMemoryFreeClear(param);
}
@ -563,6 +566,7 @@ int32_t doOpenMultiwayMergeOperator(SOperatorInfo* pOperator) {
for (int32_t i = 0; i < pOperator->numOfDownstream; ++i) {
SSortSource* ps = taosMemoryCalloc(1, sizeof(SSortSource));
ps->param = pOperator->pDownstream[i];
ps->onlyRef = true;
tsortAddSource(pInfo->pSortHandle, ps);
}
@ -690,6 +694,8 @@ SSDataBlock* doMultiwayMerge(SOperatorInfo* pOperator) {
T_LONG_JMP(pTaskInfo->env, code);
}
qDebug("start to merge final sorted rows, %s", GET_TASKID(pTaskInfo));
SSDataBlock* pBlock = getMultiwaySortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pInfo->matchInfo.pList, pOperator);
if (pBlock != NULL) {
pOperator->resultInfo.totalRows += pBlock->info.rows;
@ -754,7 +760,7 @@ SOperatorInfo* createMultiwayMergeOperatorInfo(SOperatorInfo** downStreams, size
SPhysiNode* pChildNode = (SPhysiNode*)nodesListGetNode(pPhyNode->pChildren, 0);
SSDataBlock* pInputBlock = createResDataBlock(pChildNode->pOutputDataBlockDesc);
initResultSizeInfo(&pOperator->resultInfo, 1024);
initResultSizeInfo(&pOperator->resultInfo, 4096);
blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity);
pInfo->groupSort = pMergePhyNode->groupSort;

31
source/libs/executor/src/tfill.c
View File

@ -709,6 +709,7 @@ void* destroyStreamFillSupporter(SStreamFillSupporter* pFillSup) {
pFillSup->pResMap = NULL;
releaseOutputBuf(NULL, NULL, (SResultRow*)pFillSup->cur.pRowVal);
pFillSup->cur.pRowVal = NULL;
cleanupExprSupp(&pFillSup->notFillExprSup);
taosMemoryFree(pFillSup);
return NULL;
@ -1417,25 +1418,13 @@ static void doApplyStreamScalarCalculation(SOperatorInfo* pOperator, SSDataBlock
blockDataEnsureCapacity(pDstBlock, pSrcBlock->info.rows);
setInputDataBlock(pSup, pSrcBlock, TSDB_ORDER_ASC, MAIN_SCAN, false);
projectApplyFunctions(pSup->pExprInfo, pDstBlock, pSrcBlock, pSup->pCtx, pSup->numOfExprs, NULL);
pDstBlock->info.rows = 0;
pSup = &pInfo->pFillSup->notFillExprSup;
setInputDataBlock(pSup, pSrcBlock, TSDB_ORDER_ASC, MAIN_SCAN, false);
projectApplyFunctions(pSup->pExprInfo, pDstBlock, pSrcBlock, pSup->pCtx, pSup->numOfExprs, NULL);
pDstBlock->info.groupId = pSrcBlock->info.groupId;
SColumnInfoData* pDst = taosArrayGet(pDstBlock->pDataBlock, pInfo->primaryTsCol);
SColumnInfoData* pSrc = taosArrayGet(pSrcBlock->pDataBlock, pInfo->primarySrcSlotId);
colDataAssign(pDst, pSrc, pDstBlock->info.rows, &pDstBlock->info);
int32_t numOfNotFill = pInfo->pFillSup->numOfAllCols - pInfo->pFillSup->numOfFillCols;
for (int32_t i = 0; i < numOfNotFill; ++i) {
SFillColInfo* pCol = &pInfo->pFillSup->pAllColInfo[i + pInfo->pFillSup->numOfFillCols];
ASSERT(pCol->notFillCol);
SExprInfo* pExpr = pCol->pExpr;
int32_t srcSlotId = pExpr->base.pParam[0].pCol->slotId;
int32_t dstSlotId = pExpr->base.resSchema.slotId;
SColumnInfoData* pDst1 = taosArrayGet(pDstBlock->pDataBlock, dstSlotId);
SColumnInfoData* pSrc1 = taosArrayGet(pSrcBlock->pDataBlock, srcSlotId);
colDataAssign(pDst1, pSrc1, pDstBlock->info.rows, &pDstBlock->info);
}
blockDataUpdateTsWindow(pDstBlock, pInfo->primaryTsCol);
}
@ -1577,6 +1566,14 @@ static SStreamFillSupporter* initStreamFillSup(SStreamFillPhysiNode* pPhyFillNod
destroyStreamFillSupporter(pFillSup);
return NULL;
}
SExprInfo* noFillExpr = createExprInfo(pPhyFillNode->pNotFillExprs, NULL, &numOfNotFillCols);
code = initExprSupp(&pFillSup->notFillExprSup, noFillExpr, numOfNotFillCols);
if (code != TSDB_CODE_SUCCESS) {
destroyStreamFillSupporter(pFillSup);
return NULL;
}
_hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY);
pFillSup->pResMap = tSimpleHashInit(16, hashFn);
pFillSup->hasDelete = false;

339
source/libs/executor/src/timewindowoperator.c
View File

@ -1618,6 +1618,7 @@ void destroyStreamFinalIntervalOperatorInfo(void* param) {
nodesDestroyNode((SNode*)pInfo->pPhyNode);
colDataDestroy(&pInfo->twAggSup.timeWindowData);
cleanupGroupResInfo(&pInfo->groupResInfo);
cleanupExprSupp(&pInfo->scalarSupp);
taosMemoryFreeClear(param);
}
@ -1942,10 +1943,8 @@ static void doKeepPrevRows(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlock
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
// null data should not be kept since it can not be used to perform interpolation
if (!colDataIsNull_s(pColInfoData, i)) {
SGroupKeys* pkey = taosArrayGet(pSliceInfo->pPrevRow, i);
SGroupKeys* pkey = taosArrayGet(pSliceInfo->pPrevRow, i);
if (!colDataIsNull_s(pColInfoData, rowIndex)) {
pkey->isNull = false;
char* val = colDataGetData(pColInfoData, rowIndex);
if (!IS_VAR_DATA_TYPE(pkey->type)) {
@ -1953,6 +1952,8 @@ static void doKeepPrevRows(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlock
} else {
memcpy(pkey->pData, val, varDataLen(val));
}
} else {
pkey->isNull = true;
}
}
@ -1964,10 +1965,8 @@ static void doKeepNextRows(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlock
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
// null data should not be kept since it can not be used to perform interpolation
if (!colDataIsNull_s(pColInfoData, i)) {
SGroupKeys* pkey = taosArrayGet(pSliceInfo->pNextRow, i);
SGroupKeys* pkey = taosArrayGet(pSliceInfo->pNextRow, i);
if (!colDataIsNull_s(pColInfoData, rowIndex)) {
pkey->isNull = false;
char* val = colDataGetData(pColInfoData, rowIndex);
if (!IS_VAR_DATA_TYPE(pkey->type)) {
@ -1975,50 +1974,51 @@ static void doKeepNextRows(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlock
} else {
memcpy(pkey->pData, val, varDataLen(val));
}
} else {
pkey->isNull = true;
}
}
pSliceInfo->isNextRowSet = true;
}
static void doKeepLinearInfo(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlock* pBlock, int32_t rowIndex,
bool isLastRow) {
static void doKeepLinearInfo(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlock* pBlock, int32_t rowIndex) {
int32_t numOfCols = taosArrayGetSize(pBlock->pDataBlock);
bool fillLastPoint = pSliceInfo->fillLastPoint;
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
SColumnInfoData* pTsCol = taosArrayGet(pBlock->pDataBlock, pSliceInfo->tsCol.slotId);
SFillLinearInfo* pLinearInfo = taosArrayGet(pSliceInfo->pLinearInfo, i);
// null data should not be kept since it can not be used to perform interpolation
if (!colDataIsNull_s(pColInfoData, i)) {
if (isLastRow) {
// null value is represented by using key = INT64_MIN for now.
// TODO: optimize to ignore null values for linear interpolation.
if (!pLinearInfo->isStartSet) {
if (!colDataIsNull_s(pColInfoData, rowIndex)) {
pLinearInfo->start.key = *(int64_t*)colDataGetData(pTsCol, rowIndex);
memcpy(pLinearInfo->start.val, colDataGetData(pColInfoData, rowIndex), pLinearInfo->bytes);
} else if (fillLastPoint) {
}
pLinearInfo->isStartSet = true;
} else if (!pLinearInfo->isEndSet) {
if (!colDataIsNull_s(pColInfoData, rowIndex)) {
pLinearInfo->end.key = *(int64_t*)colDataGetData(pTsCol, rowIndex);
memcpy(pLinearInfo->end.val, colDataGetData(pColInfoData, rowIndex), pLinearInfo->bytes);
}
pLinearInfo->isEndSet = true;
} else {
pLinearInfo->start.key = pLinearInfo->end.key;
memcpy(pLinearInfo->start.val, pLinearInfo->end.val, pLinearInfo->bytes);
if (!colDataIsNull_s(pColInfoData, rowIndex)) {
pLinearInfo->end.key = *(int64_t*)colDataGetData(pTsCol, rowIndex);
memcpy(pLinearInfo->end.val, colDataGetData(pColInfoData, rowIndex), pLinearInfo->bytes);
} else {
pLinearInfo->start.key = *(int64_t*)colDataGetData(pTsCol, rowIndex);
pLinearInfo->end.key = *(int64_t*)colDataGetData(pTsCol, rowIndex + 1);
char* val;
val = colDataGetData(pColInfoData, rowIndex);
memcpy(pLinearInfo->start.val, val, pLinearInfo->bytes);
val = colDataGetData(pColInfoData, rowIndex + 1);
memcpy(pLinearInfo->end.val, val, pLinearInfo->bytes);
pLinearInfo->end.key = INT64_MIN;
}
pLinearInfo->hasNull = false;
} else {
pLinearInfo->hasNull = true;
}
}
pSliceInfo->fillLastPoint = isLastRow;
}
static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp* pExprSup, SSDataBlock* pResBlock) {
static bool genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp* pExprSup, SSDataBlock* pResBlock, bool beforeTs) {
int32_t rows = pResBlock->info.rows;
blockDataEnsureCapacity(pResBlock, rows + 1);
// todo set the correct primary timestamp column
@ -2037,7 +2037,6 @@ static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp
}
int32_t srcSlot = pExprInfo->base.pParam[0].pCol->slotId;
// SColumnInfoData* pSrc = taosArrayGet(pBlock->pDataBlock, srcSlot);
switch (pSliceInfo->fillType) {
case TSDB_FILL_NULL: {
colDataAppendNULL(pDst, rows);
@ -2069,21 +2068,26 @@ static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp
SPoint start = pLinearInfo->start;
SPoint end = pLinearInfo->end;
SPoint current = {.key = pSliceInfo->current};
current.val = taosMemoryCalloc(pLinearInfo->bytes, 1);
// before interp range, do not fill
if (start.key == INT64_MIN || end.key == INT64_MAX) {
// do not interpolate before ts range, only increate pSliceInfo->current
if (beforeTs && !pLinearInfo->isEndSet) {
return true;
}
if (!pLinearInfo->isStartSet || !pLinearInfo->isEndSet) {
hasInterp = false;
break;
}
if (pLinearInfo->hasNull) {
if (start.key == INT64_MIN || end.key == INT64_MIN) {
colDataAppendNULL(pDst, rows);
} else {
taosGetLinearInterpolationVal(&current, pLinearInfo->type, &start, &end, pLinearInfo->type);
colDataAppend(pDst, rows, (char*)current.val, false);
break;
}
current.val = taosMemoryCalloc(pLinearInfo->bytes, 1);
taosGetLinearInterpolationVal(&current, pLinearInfo->type, &start, &end, pLinearInfo->type);
colDataAppend(pDst, rows, (char*)current.val, false);
taosMemoryFree(current.val);
break;
}
@ -2094,7 +2098,11 @@ static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp
}
SGroupKeys* pkey = taosArrayGet(pSliceInfo->pPrevRow, srcSlot);
colDataAppend(pDst, rows, pkey->pData, false);
if (pkey->isNull == false) {
colDataAppend(pDst, rows, pkey->pData, false);
} else {
colDataAppendNULL(pDst, rows);
}
break;
}
@ -2105,7 +2113,11 @@ static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp
}
SGroupKeys* pkey = taosArrayGet(pSliceInfo->pNextRow, srcSlot);
colDataAppend(pDst, rows, pkey->pData, false);
if (pkey->isNull == false) {
colDataAppend(pDst, rows, pkey->pData, false);
} else {
colDataAppendNULL(pDst, rows);
}
break;
}
@ -2118,8 +2130,40 @@ static void genInterpolationResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp
if (hasInterp) {
pResBlock->info.rows += 1;
}
return hasInterp;
}
static void addCurrentRowToResult(STimeSliceOperatorInfo* pSliceInfo, SExprSupp* pExprSup, SSDataBlock* pResBlock,
SSDataBlock* pSrcBlock, int32_t index) {
blockDataEnsureCapacity(pResBlock, pResBlock->info.rows + 1);
for (int32_t j = 0; j < pExprSup->numOfExprs; ++j) {
SExprInfo* pExprInfo = &pExprSup->pExprInfo[j];
int32_t dstSlot = pExprInfo->base.resSchema.slotId;
SColumnInfoData* pDst = taosArrayGet(pResBlock->pDataBlock, dstSlot);
if (IS_TIMESTAMP_TYPE(pExprInfo->base.resSchema.type)) {
colDataAppend(pDst, pResBlock->info.rows, (char*)&pSliceInfo->current, false);
} else {
int32_t srcSlot = pExprInfo->base.pParam[0].pCol->slotId;
SColumnInfoData* pSrc = taosArrayGet(pSrcBlock->pDataBlock, srcSlot);
if (colDataIsNull_s(pSrc, index)) {
colDataAppendNULL(pDst, pResBlock->info.rows);
continue;
}
char* v = colDataGetData(pSrc, index);
colDataAppend(pDst, pResBlock->info.rows, v, false);
}
}
pResBlock->info.rows += 1;
return;
}
static int32_t initPrevRowsKeeper(STimeSliceOperatorInfo* pInfo, SSDataBlock* pBlock) {
if (pInfo->pPrevRow != NULL) {
return TSDB_CODE_SUCCESS;
@ -2190,24 +2234,19 @@ static int32_t initFillLinearInfo(STimeSliceOperatorInfo* pInfo, SSDataBlock* pB
SFillLinearInfo linearInfo = {0};
linearInfo.start.key = INT64_MIN;
linearInfo.end.key = INT64_MAX;
linearInfo.end.key = INT64_MIN;
linearInfo.start.val = taosMemoryCalloc(1, pColInfo->info.bytes);
linearInfo.end.val = taosMemoryCalloc(1, pColInfo->info.bytes);
linearInfo.hasNull = false;
linearInfo.isStartSet = false;
linearInfo.isEndSet = false;
linearInfo.type = pColInfo->info.type;
linearInfo.bytes = pColInfo->info.bytes;
taosArrayPush(pInfo->pLinearInfo, &linearInfo);
}
pInfo->fillLastPoint = false;
return TSDB_CODE_SUCCESS;
}
static bool needToFillLastPoint(STimeSliceOperatorInfo* pSliceInfo) {
return (pSliceInfo->fillLastPoint == true && pSliceInfo->fillType == TSDB_FILL_LINEAR);
}
static int32_t initKeeperInfo(STimeSliceOperatorInfo* pInfo, SSDataBlock* pBlock) {
int32_t code;
code = initPrevRowsKeeper(pInfo, pBlock);
@ -2263,195 +2302,73 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) {
for (int32_t i = 0; i < pBlock->info.rows; ++i) {
int64_t ts = *(int64_t*)colDataGetData(pTsCol, i);
if (i == 0 && needToFillLastPoint(pSliceInfo)) { // first row in current block
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
while (pSliceInfo->current < ts && pSliceInfo->current <= pSliceInfo->win.ekey) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
}
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
if (ts == pSliceInfo->current) {
blockDataEnsureCapacity(pResBlock, pResBlock->info.rows + 1);
for (int32_t j = 0; j < pOperator->exprSupp.numOfExprs; ++j) {
SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[j];
addCurrentRowToResult(pSliceInfo, &pOperator->exprSupp, pResBlock, pBlock, i);
int32_t dstSlot = pExprInfo->base.resSchema.slotId;
SColumnInfoData* pDst = taosArrayGet(pResBlock->pDataBlock, dstSlot);
if (IS_TIMESTAMP_TYPE(pExprInfo->base.resSchema.type)) {
colDataAppend(pDst, pResBlock->info.rows, (char*)&pSliceInfo->current, false);
} else {
int32_t srcSlot = pExprInfo->base.pParam[0].pCol->slotId;
SColumnInfoData* pSrc = taosArrayGet(pBlock->pDataBlock, srcSlot);
if (colDataIsNull_s(pSrc, i)) {
colDataAppendNULL(pDst, pResBlock->info.rows);
continue;
}
char* v = colDataGetData(pSrc, i);
colDataAppend(pDst, pResBlock->info.rows, v, false);
}
}
pResBlock->info.rows += 1;
doKeepPrevRows(pSliceInfo, pBlock, i);
doKeepLinearInfo(pSliceInfo, pBlock, i);
// for linear interpolation, always fill value between this and next points;
// if its the first point in data block, also fill values between previous(if there's any) and this point;
// if its the last point in data block, no need to fill, but reserve this point as the start value and do
// the interpolation when processing next data block.
if (pSliceInfo->fillType == TSDB_FILL_LINEAR) {
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (i < pBlock->info.rows - 1) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
pSliceInfo->current = taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit,
pInterval->precision);
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
}
} else { // it is the last row of current block
// store ts value as start, and calculate interp value when processing next block
doKeepLinearInfo(pSliceInfo, pBlock, i, true);
}
} else { // non-linear interpolation
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
} else if (ts < pSliceInfo->current) {
// in case of interpolation window starts and ends between two datapoints, fill(prev) need to interpolate
doKeepPrevRows(pSliceInfo, pBlock, i);
doKeepLinearInfo(pSliceInfo, pBlock, i);
if (pSliceInfo->fillType == TSDB_FILL_LINEAR) {
// no need to increate pSliceInfo->current here
// pSliceInfo->current =
// taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (i < pBlock->info.rows - 1) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
pSliceInfo->current = taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit,
pInterval->precision);
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
if (i < pBlock->info.rows - 1) {
// in case of interpolation window starts and ends between two datapoints, fill(next) need to interpolate
doKeepNextRows(pSliceInfo, pBlock, i + 1);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
if (!genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock, false) && pSliceInfo->fillType == TSDB_FILL_LINEAR) {
break;
} else {
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
}
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
} else {
// store ts value as start, and calculate interp value when processing next block
doKeepLinearInfo(pSliceInfo, pBlock, i, true);
}
} else { // non-linear interpolation
if (i < pBlock->info.rows - 1) {
// in case of interpolation window starts and ends between two datapoints, fill(next) need to interpolate
doKeepNextRows(pSliceInfo, pBlock, i + 1);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
pSliceInfo->current = taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit,
pInterval->precision);
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
} else {
// ignore current row, and do nothing
}
} else { // it is the last row of current block
doKeepPrevRows(pSliceInfo, pBlock, i);
// ignore current row, and do nothing
}
} else { // it is the last row of current block
doKeepPrevRows(pSliceInfo, pBlock, i);
}
} else { // ts > pSliceInfo->current
// in case of interpolation window starts and ends between two datapoints, fill(next) need to interpolate
doKeepNextRows(pSliceInfo, pBlock, i);
doKeepLinearInfo(pSliceInfo, pBlock, i);
while (pSliceInfo->current < ts && pSliceInfo->current <= pSliceInfo->win.ekey) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (!genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock, true) && pSliceInfo->fillType == TSDB_FILL_LINEAR) {
break;
} else {
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
}
}
// add current row if timestamp match
if (ts == pSliceInfo->current && pSliceInfo->current <= pSliceInfo->win.ekey) {
blockDataEnsureCapacity(pResBlock, pResBlock->info.rows + 1);
for (int32_t j = 0; j < pOperator->exprSupp.numOfExprs; ++j) {
SExprInfo* pExprInfo = &pOperator->exprSupp.pExprInfo[j];
int32_t dstSlot = pExprInfo->base.resSchema.slotId;
SColumnInfoData* pDst = taosArrayGet(pResBlock->pDataBlock, dstSlot);
if (IS_TIMESTAMP_TYPE(pExprInfo->base.resSchema.type)) {
colDataAppend(pDst, pResBlock->info.rows, (char*)&pSliceInfo->current, false);
} else {
int32_t srcSlot = pExprInfo->base.pParam[0].pCol->slotId;
SColumnInfoData* pSrc = taosArrayGet(pBlock->pDataBlock, srcSlot);
if (colDataIsNull_s(pSrc, i)) {
colDataAppendNULL(pDst, pResBlock->info.rows);
continue;
}
char* v = colDataGetData(pSrc, i);
colDataAppend(pDst, pResBlock->info.rows, v, false);
}
}
pResBlock->info.rows += 1;
addCurrentRowToResult(pSliceInfo, &pOperator->exprSupp, pResBlock, pBlock, i);
doKeepPrevRows(pSliceInfo, pBlock, i);
if (pSliceInfo->fillType == TSDB_FILL_LINEAR) {
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
if (i < pBlock->info.rows - 1) {
doKeepLinearInfo(pSliceInfo, pBlock, i, false);
int64_t nextTs = *(int64_t*)colDataGetData(pTsCol, i + 1);
if (nextTs > pSliceInfo->current) {
while (pSliceInfo->current < nextTs && pSliceInfo->current <= pSliceInfo->win.ekey) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
pSliceInfo->current = taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit,
pInterval->precision);
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
setOperatorCompleted(pOperator);
break;
}
}
} else { // it is the last row of current block
// store ts value as start, and calculate interp value when processing next block
doKeepLinearInfo(pSliceInfo, pBlock, i, true);
}
} else { // non-linear interpolation
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
}
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
}
if (pSliceInfo->current > pSliceInfo->win.ekey) {
@ -2466,7 +2383,7 @@ static SSDataBlock* doTimeslice(SOperatorInfo* pOperator) {
// except for fill(next), fill(linear)
while (pSliceInfo->current <= pSliceInfo->win.ekey && pSliceInfo->fillType != TSDB_FILL_NEXT &&
pSliceInfo->fillType != TSDB_FILL_LINEAR) {
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock);
genInterpolationResult(pSliceInfo, &pOperator->exprSupp, pResBlock, false);
pSliceInfo->current =
taosTimeAdd(pSliceInfo->current, pInterval->interval, pInterval->intervalUnit, pInterval->precision);
}
@ -2787,6 +2704,7 @@ static void rebuildIntervalWindow(SOperatorInfo* pOperator, SArray* pWinArray, S
pChildSup->rowEntryInfoOffset, &pChInfo->aggSup);
updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &parentWin, true);
compactFunctions(pSup->pCtx, pChildSup->pCtx, numOfOutput, pTaskInfo, &pInfo->twAggSup.timeWindowData);
releaseOutputBuf(pChInfo->pState, pWinRes, pChResult);
}
if (num > 0 && pUpdatedMap) {
saveWinResultInfo(pCurResult->win.skey, pWinRes->groupId, pUpdatedMap);
@ -5446,15 +5364,6 @@ SOperatorInfo* createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhys
pInfo->primaryTsIndex = ((SColumnNode*)pIntervalPhyNode->window.pTspk)->slotId;
initResultSizeInfo(&pOperator->resultInfo, 4096);
if (pIntervalPhyNode->window.pExprs != NULL) {
int32_t numOfScalar = 0;
SExprInfo* pScalarExprInfo = createExprInfo(pIntervalPhyNode->window.pExprs, NULL, &numOfScalar);
code = initExprSupp(&pInfo->scalarSupp, pScalarExprInfo, numOfScalar);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
}
size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES;
code = initAggInfo(pSup, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str);
if (code != TSDB_CODE_SUCCESS) {

44
source/libs/executor/src/tsort.c
View File

@ -110,6 +110,22 @@ static int32_t sortComparCleanup(SMsortComparParam* cmpParam) {
return TSDB_CODE_SUCCESS;
}
void tsortClearOrderdSource(SArray *pOrderedSource) {
for (size_t i = 0; i < taosArrayGetSize(pOrderedSource); i++) {
SSortSource** pSource = taosArrayGet(pOrderedSource, i);
if (NULL == *pSource) {
continue;
}
if ((*pSource)->param && !(*pSource)->onlyRef) {
taosMemoryFree((*pSource)->param);
}
taosMemoryFreeClear(*pSource);
}
taosArrayClear(pOrderedSource);
}
void tsortDestroySortHandle(SSortHandle* pSortHandle) {
if (pSortHandle == NULL) {
return;
@ -123,10 +139,8 @@ void tsortDestroySortHandle(SSortHandle* pSortHandle) {
destroyDiskbasedBuf(pSortHandle->pBuf);
taosMemoryFreeClear(pSortHandle->idStr);
blockDataDestroy(pSortHandle->pDataBlock);
for (size_t i = 0; i < taosArrayGetSize(pSortHandle->pOrderedSource); i++) {
SSortSource** pSource = taosArrayGet(pSortHandle->pOrderedSource, i);
taosMemoryFreeClear(*pSource);
}
tsortClearOrderdSource(pSortHandle->pOrderedSource);
taosArrayDestroy(pSortHandle->pOrderedSource);
taosMemoryFreeClear(pSortHandle);
}
@ -561,7 +575,7 @@ static int32_t doInternalMergeSort(SSortHandle* pHandle) {
}
}
taosArrayClear(pHandle->pOrderedSource);
tsortClearOrderdSource(pHandle->pOrderedSource);
taosArrayAddAll(pHandle->pOrderedSource, pResList);
taosArrayDestroy(pResList);
@ -598,8 +612,11 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
size_t sortBufSize = pHandle->numOfPages * pHandle->pageSize;
if (pHandle->type == SORT_SINGLESOURCE_SORT) {
SSortSource* source = taosArrayGetP(pHandle->pOrderedSource, 0);
taosArrayClear(pHandle->pOrderedSource);
SSortSource** pSource = taosArrayGet(pHandle->pOrderedSource, 0);
SSortSource* source = *pSource;
*pSource = NULL;
tsortClearOrderdSource(pHandle->pOrderedSource);
while (1) {
SSDataBlock* pBlock = pHandle->fetchfp(source->param);
@ -623,6 +640,10 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
int32_t code = blockDataMerge(pHandle->pDataBlock, pBlock);
if (code != 0) {
if (source->param && !source->onlyRef) {
taosMemoryFree(source->param);
}
taosMemoryFree(source);
return code;
}
@ -632,6 +653,10 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
int64_t p = taosGetTimestampUs();
code = blockDataSort(pHandle->pDataBlock, pHandle->pSortInfo);
if (code != 0) {
if (source->param && !source->onlyRef) {
taosMemoryFree(source->param);
}
taosMemoryFree(source);
return code;
}
@ -642,6 +667,11 @@ static int32_t createInitialSources(SSortHandle* pHandle) {
}
}
if (source->param && !source->onlyRef) {
taosMemoryFree(source->param);
}
taosMemoryFree(source);
if (pHandle->pDataBlock != NULL && pHandle->pDataBlock->info.rows > 0) {
size_t size = blockDataGetSize(pHandle->pDataBlock);

4
source/libs/function/src/builtinsimpl.c
View File

@ -6568,7 +6568,9 @@ int32_t cachedLastRowFunction(SqlFunctionCtx* pCtx) {
for (int32_t i = pInput->numOfRows + pInput->startRowIndex - 1; i >= pInput->startRowIndex; --i) {
numOfElems++;
char* data = colDataGetData(pInputCol, i);
bool isNull = colDataIsNull(pInputCol, pInput->numOfRows, i, NULL);
char* data = isNull ? NULL : colDataGetData(pInputCol, i);
TSKEY cts = getRowPTs(pInput->pPTS, i);
if (pResInfo->numOfRes == 0 || pInfo->ts < cts) {
doSaveLastrow(pCtx, data, i, cts, pInfo);

227
source/libs/qworker/src/qwMsg.c
View File

@ -65,19 +65,37 @@ int32_t qwBuildAndSendErrorRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t c
int32_t qwBuildAndSendQueryRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t code, SQWTaskCtx *ctx) {
STbVerInfo *tbInfo = ctx ? &ctx->tbInfo : NULL;
int64_t affectedRows = ctx ? ctx->affectedRows : 0;
SQueryTableRsp *pRsp = (SQueryTableRsp *)rpcMallocCont(sizeof(SQueryTableRsp));
pRsp->code = htonl(code);
pRsp->affectedRows = htobe64(affectedRows);
SQueryTableRsp rsp = {0};
rsp.code = code;
rsp.affectedRows = affectedRows;
if (tbInfo) {
strcpy(pRsp->tbFName, tbInfo->tbFName);
pRsp->sversion = htonl(tbInfo->sversion);
pRsp->tversion = htonl(tbInfo->tversion);
strcpy(rsp.tbFName, tbInfo->tbFName);
rsp.sversion = tbInfo->sversion;
rsp.tversion = tbInfo->tversion;
}
int32_t msgSize = tSerializeSQueryTableRsp(NULL, 0, &rsp);
if (msgSize < 0) {
qError("tSerializeSQueryTableRsp failed");
QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
void *pRsp = rpcMallocCont(msgSize);
if (NULL == pRsp) {
qError("rpcMallocCont %d failed", msgSize);
QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
if (tSerializeSQueryTableRsp(pRsp, msgSize, &rsp) < 0) {
qError("tSerializeSQueryTableRsp %d failed", msgSize);
QW_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
SRpcMsg rpcRsp = {
.msgType = rspType,
.pCont = pRsp,
.contLen = sizeof(*pRsp),
.contLen = msgSize,
.code = code,
.info = *pConn,
};
@ -182,23 +200,37 @@ int32_t qwBuildAndSendDropRsp(SRpcHandleInfo *pConn, int32_t code) {
#endif
int32_t qwBuildAndSendDropMsg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) {
STaskDropReq *req = (STaskDropReq *)rpcMallocCont(sizeof(STaskDropReq));
if (NULL == req) {
QW_SCH_TASK_ELOG("rpcMallocCont %d failed", (int32_t)sizeof(STaskDropReq));
STaskDropReq qMsg;
qMsg.header.vgId = mgmt->nodeId;
qMsg.header.contLen = 0;
qMsg.sId = sId;
qMsg.queryId = qId;
qMsg.taskId = tId;
qMsg.refId = rId;
qMsg.execId = eId;
int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg);
if (msgSize < 0) {
QW_SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize);
QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
void *msg = rpcMallocCont(msgSize);
if (NULL == msg) {
QW_SCH_TASK_ELOG("rpcMallocCont %d failed", msgSize);
QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) {
QW_SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize);
rpcFreeCont(msg);
QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
req->header.vgId = mgmt->nodeId;
req->sId = sId;
req->queryId = qId;
req->taskId = tId;
req->refId = rId;
req->execId = eId;
SRpcMsg pNewMsg = {
.msgType = TDMT_SCH_DROP_TASK,
.pCont = req,
.contLen = sizeof(STaskDropReq),
.pCont = msg,
.contLen = msgSize,
.code = 0,
.info = *pConn,
};
@ -247,22 +279,37 @@ int32_t qwBuildAndSendCQueryMsg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) {
}
int32_t qwRegisterQueryBrokenLinkArg(QW_FPARAMS_DEF, SRpcHandleInfo *pConn) {
STaskDropReq *req = (STaskDropReq *)rpcMallocCont(sizeof(STaskDropReq));
if (NULL == req) {
QW_SCH_TASK_ELOG("rpcMallocCont %d failed", (int32_t)sizeof(STaskDropReq));
STaskDropReq qMsg;
qMsg.header.vgId = mgmt->nodeId;
qMsg.header.contLen = 0;
qMsg.sId = sId;
qMsg.queryId = qId;
qMsg.taskId = tId;
qMsg.refId = rId;
qMsg.execId = eId;
int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg);
if (msgSize < 0) {
QW_SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize);
QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
void *msg = rpcMallocCont(msgSize);
if (NULL == msg) {
QW_SCH_TASK_ELOG("rpcMallocCont %d failed", msgSize);
QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) {
QW_SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize);
rpcFreeCont(msg);
QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
req->header.vgId = htonl(mgmt->nodeId);
req->sId = htobe64(sId);
req->queryId = htobe64(qId);
req->taskId = htobe64(tId);
req->refId = htobe64(rId);
SRpcMsg brokenMsg = {
.msgType = TDMT_SCH_DROP_TASK,
.pCont = req,
.contLen = sizeof(STaskDropReq),
.pCont = msg,
.contLen = msgSize,
.code = TSDB_CODE_RPC_BROKEN_LINK,
.info = *pConn,
};
@ -312,40 +359,33 @@ int32_t qWorkerPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg, bool chkGran
}
int32_t code = 0;
SSubQueryMsg *msg = pMsg->pCont;
SQWorker *mgmt = (SQWorker *)qWorkerMgmt;
if (NULL == msg || pMsg->contLen <= sizeof(*msg)) {
QW_ELOG("invalid query msg, msg:%p, msgLen:%d", msg, pMsg->contLen);
SSubQueryMsg msg = {0};
if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) {
QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen);
QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
}
msg->sId = be64toh(msg->sId);
msg->queryId = be64toh(msg->queryId);
msg->taskId = be64toh(msg->taskId);
msg->refId = be64toh(msg->refId);
msg->execId = ntohl(msg->execId);
msg->phyLen = ntohl(msg->phyLen);
msg->sqlLen = ntohl(msg->sqlLen);
msg->msgMask = ntohl(msg->msgMask);
if (chkGrant && (!TEST_SHOW_REWRITE_MASK(msg->msgMask)) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
QW_ELOG("query failed cause of grant expired, msgMask:%d", msg->msgMask);
if (chkGrant && (!TEST_SHOW_REWRITE_MASK(msg.msgMask)) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
QW_ELOG("query failed cause of grant expired, msgMask:%d", msg.msgMask);
tFreeSSubQueryMsg(&msg);
QW_ERR_RET(TSDB_CODE_GRANT_EXPIRED);
}
uint64_t sId = msg->sId;
uint64_t qId = msg->queryId;
uint64_t tId = msg->taskId;
int64_t rId = msg->refId;
int32_t eId = msg->execId;
uint64_t sId = msg.sId;
uint64_t qId = msg.queryId;
uint64_t tId = msg.taskId;
int64_t rId = msg.refId;
int32_t eId = msg.execId;
SQWMsg qwMsg = {
.msgType = pMsg->msgType, .msg = msg->msg + msg->sqlLen, .msgLen = msg->phyLen, .connInfo = pMsg->info};
.msgType = pMsg->msgType, .msg = msg.msg, .msgLen = msg.msgLen, .connInfo = pMsg->info};
QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p", pMsg->info.handle);
QW_ERR_RET(qwPreprocessQuery(QW_FPARAMS(), &qwMsg));
QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p", pMsg->info.handle);
QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p, SQL:%s", pMsg->info.handle, msg.sql);
code = qwPreprocessQuery(QW_FPARAMS(), &qwMsg);
QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p, code:%x", pMsg->info.handle, code);
tFreeSSubQueryMsg(&msg);
return TSDB_CODE_SUCCESS;
}
@ -355,19 +395,25 @@ int32_t qWorkerAbortPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg) {
QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
}
SSubQueryMsg *msg = pMsg->pCont;
SQWorker *mgmt = (SQWorker *)qWorkerMgmt;
SSubQueryMsg msg = {0};
if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) {
QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen);
QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
}
uint64_t sId = msg->sId;
uint64_t qId = msg->queryId;
uint64_t tId = msg->taskId;
int64_t rId = msg->refId;
int32_t eId = msg->execId;
uint64_t sId = msg.sId;
uint64_t qId = msg.queryId;
uint64_t tId = msg.taskId;
int64_t rId = msg.refId;
int32_t eId = msg.execId;
QW_SCH_TASK_DLOG("Abort prerocessQuery start, handle:%p", pMsg->info.handle);
qwAbortPrerocessQuery(QW_FPARAMS());
QW_SCH_TASK_DLOG("Abort prerocessQuery end, handle:%p", pMsg->info.handle);
tFreeSSubQueryMsg(&msg);
return TSDB_CODE_SUCCESS;
}
@ -377,42 +423,41 @@ int32_t qWorkerProcessQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int
}
int32_t code = 0;
SSubQueryMsg *msg = pMsg->pCont;
SQWorker *mgmt = (SQWorker *)qWorkerMgmt;
qwUpdateTimeInQueue(mgmt, ts, QUERY_QUEUE);
QW_STAT_INC(mgmt->stat.msgStat.queryProcessed, 1);
if (NULL == msg || pMsg->contLen <= sizeof(*msg)) {
QW_ELOG("invalid query msg, msg:%p, msgLen:%d", msg, pMsg->contLen);
SSubQueryMsg msg = {0};
if (tDeserializeSSubQueryMsg(pMsg->pCont, pMsg->contLen, &msg) < 0) {
QW_ELOG("tDeserializeSSubQueryMsg failed, contLen:%d", pMsg->contLen);
QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
}
uint64_t sId = msg->sId;
uint64_t qId = msg->queryId;
uint64_t tId = msg->taskId;
int64_t rId = msg->refId;
int32_t eId = msg->execId;
uint64_t sId = msg.sId;
uint64_t qId = msg.queryId;
uint64_t tId = msg.taskId;
int64_t rId = msg.refId;
int32_t eId = msg.execId;
SQWMsg qwMsg = {.node = node,
.msg = msg->msg + msg->sqlLen,
.msgLen = msg->phyLen,
.msg = msg.msg,
.msgLen = msg.msgLen,
.connInfo = pMsg->info,
.msgType = pMsg->msgType};
qwMsg.msgInfo.explain = msg->explain;
qwMsg.msgInfo.taskType = msg->taskType;
qwMsg.msgInfo.needFetch = msg->needFetch;
qwMsg.msgInfo.explain = msg.explain;
qwMsg.msgInfo.taskType = msg.taskType;
qwMsg.msgInfo.needFetch = msg.needFetch;
char *sql = strndup(msg->msg, msg->sqlLen);
QW_SCH_TASK_DLOG("processQuery start, node:%p, type:%s, handle:%p, SQL:%s", node, TMSG_INFO(pMsg->msgType),
pMsg->info.handle, sql);
QW_ERR_JRET(qwProcessQuery(QW_FPARAMS(), &qwMsg, sql));
pMsg->info.handle, msg.sql);
code = qwProcessQuery(QW_FPARAMS(), &qwMsg, msg.sql);
msg.sql = NULL;
QW_SCH_TASK_DLOG("processQuery end, node:%p, code:%x", node, code);
_return:
tFreeSSubQueryMsg(&msg);
QW_SCH_TASK_DLOG("processQuery end, node:%p, code:%d", node, code);
return code;
return TSDB_CODE_SUCCESS;
}
int32_t qWorkerProcessCQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts) {
@ -548,28 +593,22 @@ int32_t qWorkerProcessDropMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int6
}
int32_t code = 0;
STaskDropReq *msg = pMsg->pCont;
SQWorker *mgmt = (SQWorker *)qWorkerMgmt;
qwUpdateTimeInQueue(mgmt, ts, FETCH_QUEUE);
QW_STAT_INC(mgmt->stat.msgStat.dropProcessed, 1);
if (NULL == msg || pMsg->contLen < sizeof(*msg)) {
QW_ELOG("invalid task drop msg, msg:%p, msgLen:%d", msg, pMsg->contLen);
STaskDropReq msg = {0};
if (tDeserializeSTaskDropReq(pMsg->pCont, pMsg->contLen, &msg) < 0) {
QW_ELOG("tDeserializeSTaskDropReq failed, contLen:%d", pMsg->contLen);
QW_ERR_RET(TSDB_CODE_QRY_INVALID_INPUT);
}
msg->sId = be64toh(msg->sId);
msg->queryId = be64toh(msg->queryId);
msg->taskId = be64toh(msg->taskId);
msg->refId = be64toh(msg->refId);
msg->execId = ntohl(msg->execId);
uint64_t sId = msg->sId;
uint64_t qId = msg->queryId;
uint64_t tId = msg->taskId;
int64_t rId = msg->refId;
int32_t eId = msg->execId;
uint64_t sId = msg.sId;
uint64_t qId = msg.queryId;
uint64_t tId = msg.taskId;
int64_t rId = msg.refId;
int32_t eId = msg.execId;
SQWMsg qwMsg = {.node = node, .msg = NULL, .msgLen = 0, .code = pMsg->code, .connInfo = pMsg->info};

29
source/libs/qworker/test/qworkerTests.cpp
View File

@ -114,7 +114,7 @@ void qwtBuildQueryReqMsg(SRpcMsg *queryRpc) {
qwtqueryMsg.queryId = htobe64(atomic_add_fetch_64(&qwtTestQueryId, 1));
qwtqueryMsg.sId = htobe64(1);
qwtqueryMsg.taskId = htobe64(1);
qwtqueryMsg.phyLen = htonl(100);
qwtqueryMsg.msgLen = htonl(100);
qwtqueryMsg.sqlLen = 0;
queryRpc->msgType = TDMT_SCH_QUERY;
queryRpc->pCont = &qwtqueryMsg;
@ -131,12 +131,29 @@ void qwtBuildFetchReqMsg(SResFetchReq *fetchMsg, SRpcMsg *fetchRpc) {
}
void qwtBuildDropReqMsg(STaskDropReq *dropMsg, SRpcMsg *dropRpc) {
dropMsg->sId = htobe64(1);
dropMsg->queryId = htobe64(atomic_load_64(&qwtTestQueryId));
dropMsg->taskId = htobe64(1);
dropMsg->sId = 1;
dropMsg->queryId = atomic_load_64(&qwtTestQueryId);
dropMsg->taskId = 1;
int32_t msgSize = tSerializeSTaskDropReq(NULL, 0, dropMsg);
if (msgSize < 0) {
return;
}
char *msg = (char*)taosMemoryCalloc(1, msgSize);
if (NULL == msg) {
return;
}
if (tSerializeSTaskDropReq(msg, msgSize, dropMsg) < 0) {
taosMemoryFree(msg);
return;
}
dropRpc->msgType = TDMT_SCH_DROP_TASK;
dropRpc->pCont = dropMsg;
dropRpc->contLen = sizeof(STaskDropReq);
dropRpc->pCont = msg;
dropRpc->contLen = msgSize;
}
int32_t qwtStringToPlan(const char *str, SSubplan **subplan) {

2
source/libs/scalar/src/sclfunc.c
View File

@ -24,6 +24,8 @@ static double tlog2(double v, double base) {
return a;
} else if (isnan(b) || isinf(b)) {
return b;
} else if (b == 0) {
return INFINITY;
} else {
return a / b;
}

94
source/libs/scheduler/src/schRemote.c
View File

@ -334,17 +334,17 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t execId, SDa
SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_INPUT);
}
SQueryTableRsp *rsp = (SQueryTableRsp *)msg;
rsp->code = ntohl(rsp->code);
rsp->sversion = ntohl(rsp->sversion);
rsp->tversion = ntohl(rsp->tversion);
rsp->affectedRows = be64toh(rsp->affectedRows);
SQueryTableRsp rsp = {0};
if (tDeserializeSQueryTableRsp(msg, msgSize, &rsp) < 0) {
SCH_TASK_ELOG("tDeserializeSQueryTableRsp failed, msgSize:%d", msgSize);
SCH_ERR_JRET(TSDB_CODE_QRY_INVALID_MSG);
}
SCH_ERR_JRET(rsp.code);
SCH_ERR_JRET(rsp->code);
SCH_ERR_JRET(schSaveJobExecRes(pJob, &rsp));
SCH_ERR_JRET(schSaveJobExecRes(pJob, rsp));
atomic_add_fetch_32(&pJob->resNumOfRows, rsp->affectedRows);
atomic_add_fetch_32(&pJob->resNumOfRows, rsp.affectedRows);
taosMemoryFreeClear(msg);
@ -1042,30 +1042,40 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr,
case TDMT_SCH_MERGE_QUERY: {
SCH_ERR_RET(schMakeQueryRpcCtx(pJob, pTask, &rpcCtx));
uint32_t len = strlen(pJob->sql);
msgSize = sizeof(SSubQueryMsg) + pTask->msgLen + len;
SSubQueryMsg qMsg;
qMsg.header.vgId = addr->nodeId;
qMsg.header.contLen = 0;
qMsg.sId = schMgmt.sId;
qMsg.queryId = pJob->queryId;
qMsg.taskId = pTask->taskId;
qMsg.refId = pJob->refId;
qMsg.execId = pTask->execId;
qMsg.msgMask = (pTask->plan->showRewrite) ? QUERY_MSG_MASK_SHOW_REWRITE() : 0;
qMsg.taskType = TASK_TYPE_TEMP;
qMsg.explain = SCH_IS_EXPLAIN_JOB(pJob);
qMsg.needFetch = SCH_TASK_NEED_FETCH(pTask);
qMsg.sqlLen = strlen(pJob->sql);
qMsg.sql = pJob->sql;
qMsg.msgLen = pTask->msgLen;
qMsg.msg = pTask->msg;
msgSize = tSerializeSSubQueryMsg(NULL, 0, &qMsg);
if (msgSize < 0) {
SCH_TASK_ELOG("tSerializeSSubQueryMsg get size, msgSize:%d", msgSize);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
msg = taosMemoryCalloc(1, msgSize);
if (NULL == msg) {
SCH_TASK_ELOG("calloc %d failed", msgSize);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
SSubQueryMsg *pMsg = msg;
pMsg->header.vgId = htonl(addr->nodeId);
pMsg->sId = htobe64(schMgmt.sId);
pMsg->queryId = htobe64(pJob->queryId);
pMsg->taskId = htobe64(pTask->taskId);
pMsg->refId = htobe64(pJob->refId);
pMsg->execId = htonl(pTask->execId);
pMsg->taskType = TASK_TYPE_TEMP;
pMsg->explain = SCH_IS_EXPLAIN_JOB(pJob);
pMsg->needFetch = SCH_TASK_NEED_FETCH(pTask);
pMsg->phyLen = htonl(pTask->msgLen);
pMsg->sqlLen = htonl(len);
pMsg->msgMask = htonl((pTask->plan->showRewrite) ? QUERY_MSG_MASK_SHOW_REWRITE() : 0);
memcpy(pMsg->msg, pJob->sql, len);
memcpy(pMsg->msg + len, pTask->msg, pTask->msgLen);
if (tSerializeSSubQueryMsg(msg, msgSize, &qMsg) < 0) {
SCH_TASK_ELOG("tSerializeSSubQueryMsg failed, msgSize:%d", msgSize);
taosMemoryFree(msg);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
persistHandle = true;
SCH_SET_TASK_HANDLE(pTask, rpcAllocHandle());
@ -1092,22 +1102,32 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr,
break;
}
case TDMT_SCH_DROP_TASK: {
msgSize = sizeof(STaskDropReq);
STaskDropReq qMsg;
qMsg.header.vgId = addr->nodeId;
qMsg.header.contLen = 0;
qMsg.sId = schMgmt.sId;
qMsg.queryId = pJob->queryId;
qMsg.taskId = pTask->taskId;
qMsg.refId = pJob->refId;
qMsg.execId = pTask->execId;
msgSize = tSerializeSTaskDropReq(NULL, 0, &qMsg);
if (msgSize < 0) {
SCH_TASK_ELOG("tSerializeSTaskDropReq get size, msgSize:%d", msgSize);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
msg = taosMemoryCalloc(1, msgSize);
if (NULL == msg) {
SCH_TASK_ELOG("calloc %d failed", msgSize);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
STaskDropReq *pMsg = msg;
pMsg->header.vgId = htonl(addr->nodeId);
pMsg->sId = htobe64(schMgmt.sId);
pMsg->queryId = htobe64(pJob->queryId);
pMsg->taskId = htobe64(pTask->taskId);
pMsg->refId = htobe64(pJob->refId);
pMsg->execId = htonl(pTask->execId);
if (tSerializeSTaskDropReq(msg, msgSize, &qMsg) < 0) {
SCH_TASK_ELOG("tSerializeSTaskDropReq failed, msgSize:%d", msgSize);
taosMemoryFree(msg);
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
}
break;
}
case TDMT_SCH_QUERY_HEARTBEAT: {

143
source/libs/stream/src/streamState.c
View File

@ -114,6 +114,12 @@ SStreamState* streamStateOpen(char* path, SStreamTask* pTask, bool specPath, int
terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL;
}
pState->pTdbState = taosMemoryCalloc(1, sizeof(STdbState));
if (pState->pTdbState == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
streamStateDestroy(pState);
return NULL;
}
char statePath[1024];
if (!specPath) {
@ -122,31 +128,34 @@ SStreamState* streamStateOpen(char* path, SStreamTask* pTask, bool specPath, int
memset(statePath, 0, 1024);
tstrncpy(statePath, path, 1024);
}
if (tdbOpen(statePath, szPage, pages, &pState->db, 0) < 0) {
if (tdbOpen(statePath, szPage, pages, &pState->pTdbState->db, 0) < 0) {
goto _err;
}
// open state storage backend
if (tdbTbOpen("state.db", sizeof(SStateKey), -1, stateKeyCmpr, pState->db, &pState->pStateDb, 0) < 0) {
if (tdbTbOpen("state.db", sizeof(SStateKey), -1, stateKeyCmpr, pState->pTdbState->db, &pState->pTdbState->pStateDb,
0) < 0) {
goto _err;
}
// todo refactor
if (tdbTbOpen("fill.state.db", sizeof(SWinKey), -1, winKeyCmpr, pState->db, &pState->pFillStateDb, 0) < 0) {
if (tdbTbOpen("fill.state.db", sizeof(SWinKey), -1, winKeyCmpr, pState->pTdbState->db,
&pState->pTdbState->pFillStateDb, 0) < 0) {
goto _err;
}
if (tdbTbOpen("session.state.db", sizeof(SStateSessionKey), -1, stateSessionKeyCmpr, pState->db,
&pState->pSessionStateDb, 0) < 0) {
if (tdbTbOpen("session.state.db", sizeof(SStateSessionKey), -1, stateSessionKeyCmpr, pState->pTdbState->db,
&pState->pTdbState->pSessionStateDb, 0) < 0) {
goto _err;
}
if (tdbTbOpen("func.state.db", sizeof(STupleKey), -1, STupleKeyCmpr, pState->db, &pState->pFuncStateDb, 0) < 0) {
if (tdbTbOpen("func.state.db", sizeof(STupleKey), -1, STupleKeyCmpr, pState->pTdbState->db,
&pState->pTdbState->pFuncStateDb, 0) < 0) {
goto _err;
}
if (tdbTbOpen("parname.state.db", sizeof(int64_t), TSDB_TABLE_NAME_LEN, NULL, pState->db, &pState->pParNameDb, 0) <
0) {
if (tdbTbOpen("parname.state.db", sizeof(int64_t), TSDB_TABLE_NAME_LEN, NULL, pState->pTdbState->db,
&pState->pTdbState->pParNameDb, 0) < 0) {
goto _err;
}
@ -154,117 +163,117 @@ SStreamState* streamStateOpen(char* path, SStreamTask* pTask, bool specPath, int
goto _err;
}
pState->pOwner = pTask;
pState->pTdbState->pOwner = pTask;
return pState;
_err:
tdbTbClose(pState->pStateDb);
tdbTbClose(pState->pFuncStateDb);
tdbTbClose(pState->pFillStateDb);
tdbTbClose(pState->pSessionStateDb);
tdbTbClose(pState->pParNameDb);
tdbClose(pState->db);
taosMemoryFree(pState);
tdbTbClose(pState->pTdbState->pStateDb);
tdbTbClose(pState->pTdbState->pFuncStateDb);
tdbTbClose(pState->pTdbState->pFillStateDb);
tdbTbClose(pState->pTdbState->pSessionStateDb);
tdbTbClose(pState->pTdbState->pParNameDb);
tdbClose(pState->pTdbState->db);
streamStateDestroy(pState);
return NULL;
}
void streamStateClose(SStreamState* pState) {
tdbCommit(pState->db, &pState->txn);
tdbPostCommit(pState->db, &pState->txn);
tdbTbClose(pState->pStateDb);
tdbTbClose(pState->pFuncStateDb);
tdbTbClose(pState->pFillStateDb);
tdbTbClose(pState->pSessionStateDb);
tdbTbClose(pState->pParNameDb);
tdbClose(pState->db);
tdbCommit(pState->pTdbState->db, &pState->pTdbState->txn);
tdbPostCommit(pState->pTdbState->db, &pState->pTdbState->txn);
tdbTbClose(pState->pTdbState->pStateDb);
tdbTbClose(pState->pTdbState->pFuncStateDb);
tdbTbClose(pState->pTdbState->pFillStateDb);
tdbTbClose(pState->pTdbState->pSessionStateDb);
tdbTbClose(pState->pTdbState->pParNameDb);
tdbClose(pState->pTdbState->db);
taosMemoryFree(pState);
streamStateDestroy(pState);
}
int32_t streamStateBegin(SStreamState* pState) {
if (tdbTxnOpen(&pState->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) <
0) {
if (tdbTxnOpen(&pState->pTdbState->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL,
TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) < 0) {
return -1;
}
if (tdbBegin(pState->db, &pState->txn) < 0) {
tdbTxnClose(&pState->txn);
if (tdbBegin(pState->pTdbState->db, &pState->pTdbState->txn) < 0) {
tdbTxnClose(&pState->pTdbState->txn);
return -1;
}
return 0;
}
int32_t streamStateCommit(SStreamState* pState) {
if (tdbCommit(pState->db, &pState->txn) < 0) {
if (tdbCommit(pState->pTdbState->db, &pState->pTdbState->txn) < 0) {
return -1;
}
if (tdbPostCommit(pState->db, &pState->txn) < 0) {
if (tdbPostCommit(pState->pTdbState->db, &pState->pTdbState->txn) < 0) {
return -1;
}
memset(&pState->txn, 0, sizeof(TXN));
if (tdbTxnOpen(&pState->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) <
0) {
memset(&pState->pTdbState->txn, 0, sizeof(TXN));
if (tdbTxnOpen(&pState->pTdbState->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL,
TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) < 0) {
return -1;
}
if (tdbBegin(pState->db, &pState->txn) < 0) {
if (tdbBegin(pState->pTdbState->db, &pState->pTdbState->txn) < 0) {
return -1;
}
return 0;
}
int32_t streamStateAbort(SStreamState* pState) {
if (tdbAbort(pState->db, &pState->txn) < 0) {
if (tdbAbort(pState->pTdbState->db, &pState->pTdbState->txn) < 0) {
return -1;
}
memset(&pState->txn, 0, sizeof(TXN));
if (tdbTxnOpen(&pState->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) <
0) {
memset(&pState->pTdbState->txn, 0, sizeof(TXN));
if (tdbTxnOpen(&pState->pTdbState->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL,
TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) < 0) {
return -1;
}
if (tdbBegin(pState->db, &pState->txn) < 0) {
if (tdbBegin(pState->pTdbState->db, &pState->pTdbState->txn) < 0) {
return -1;
}
return 0;
}
int32_t streamStateFuncPut(SStreamState* pState, const STupleKey* key, const void* value, int32_t vLen) {
return tdbTbUpsert(pState->pFuncStateDb, key, sizeof(STupleKey), value, vLen, &pState->txn);
return tdbTbUpsert(pState->pTdbState->pFuncStateDb, key, sizeof(STupleKey), value, vLen, &pState->pTdbState->txn);
}
int32_t streamStateFuncGet(SStreamState* pState, const STupleKey* key, void** pVal, int32_t* pVLen) {
return tdbTbGet(pState->pFuncStateDb, key, sizeof(STupleKey), pVal, pVLen);
return tdbTbGet(pState->pTdbState->pFuncStateDb, key, sizeof(STupleKey), pVal, pVLen);
}
int32_t streamStateFuncDel(SStreamState* pState, const STupleKey* key) {
return tdbTbDelete(pState->pFuncStateDb, key, sizeof(STupleKey), &pState->txn);
return tdbTbDelete(pState->pTdbState->pFuncStateDb, key, sizeof(STupleKey), &pState->pTdbState->txn);
}
// todo refactor
int32_t streamStatePut(SStreamState* pState, const SWinKey* key, const void* value, int32_t vLen) {
SStateKey sKey = {.key = *key, .opNum = pState->number};
return tdbTbUpsert(pState->pStateDb, &sKey, sizeof(SStateKey), value, vLen, &pState->txn);
return tdbTbUpsert(pState->pTdbState->pStateDb, &sKey, sizeof(SStateKey), value, vLen, &pState->pTdbState->txn);
}
// todo refactor
int32_t streamStateFillPut(SStreamState* pState, const SWinKey* key, const void* value, int32_t vLen) {
return tdbTbUpsert(pState->pFillStateDb, key, sizeof(SWinKey), value, vLen, &pState->txn);
return tdbTbUpsert(pState->pTdbState->pFillStateDb, key, sizeof(SWinKey), value, vLen, &pState->pTdbState->txn);
}
// todo refactor
int32_t streamStateGet(SStreamState* pState, const SWinKey* key, void** pVal, int32_t* pVLen) {
SStateKey sKey = {.key = *key, .opNum = pState->number};
return tdbTbGet(pState->pStateDb, &sKey, sizeof(SStateKey), pVal, pVLen);
return tdbTbGet(pState->pTdbState->pStateDb, &sKey, sizeof(SStateKey), pVal, pVLen);
}
// todo refactor
int32_t streamStateFillGet(SStreamState* pState, const SWinKey* key, void** pVal, int32_t* pVLen) {
return tdbTbGet(pState->pFillStateDb, key, sizeof(SWinKey), pVal, pVLen);
return tdbTbGet(pState->pTdbState->pFillStateDb, key, sizeof(SWinKey), pVal, pVLen);
}
// todo refactor
int32_t streamStateDel(SStreamState* pState, const SWinKey* key) {
SStateKey sKey = {.key = *key, .opNum = pState->number};
return tdbTbDelete(pState->pStateDb, &sKey, sizeof(SStateKey), &pState->txn);
return tdbTbDelete(pState->pTdbState->pStateDb, &sKey, sizeof(SStateKey), &pState->pTdbState->txn);
}
int32_t streamStateClear(SStreamState* pState) {
@ -288,7 +297,7 @@ void streamStateSetNumber(SStreamState* pState, int32_t number) { pState->number
// todo refactor
int32_t streamStateFillDel(SStreamState* pState, const SWinKey* key) {
return tdbTbDelete(pState->pFillStateDb, key, sizeof(SWinKey), &pState->txn);
return tdbTbDelete(pState->pTdbState->pFillStateDb, key, sizeof(SWinKey), &pState->pTdbState->txn);
}
int32_t streamStateAddIfNotExist(SStreamState* pState, const SWinKey* key, void** pVal, int32_t* pVLen) {
@ -314,7 +323,7 @@ int32_t streamStateReleaseBuf(SStreamState* pState, const SWinKey* key, void* pV
SStreamStateCur* streamStateGetCur(SStreamState* pState, const SWinKey* key) {
SStreamStateCur* pCur = taosMemoryCalloc(1, sizeof(SStreamStateCur));
if (pCur == NULL) return NULL;
tdbTbcOpen(pState->pStateDb, &pCur->pCur, NULL);
tdbTbcOpen(pState->pTdbState->pStateDb, &pCur->pCur, NULL);
int32_t c = 0;
SStateKey sKey = {.key = *key, .opNum = pState->number};
@ -330,7 +339,7 @@ SStreamStateCur* streamStateGetCur(SStreamState* pState, const SWinKey* key) {
SStreamStateCur* streamStateFillGetCur(SStreamState* pState, const SWinKey* key) {
SStreamStateCur* pCur = taosMemoryCalloc(1, sizeof(SStreamStateCur));
if (pCur == NULL) return NULL;
tdbTbcOpen(pState->pFillStateDb, &pCur->pCur, NULL);
tdbTbcOpen(pState->pTdbState->pFillStateDb, &pCur->pCur, NULL);
int32_t c = 0;
tdbTbcMoveTo(pCur->pCur, key, sizeof(SWinKey), &c);
@ -402,6 +411,7 @@ int32_t streamStateGetFirst(SStreamState* pState, SWinKey* key) {
streamStatePut(pState, &tmp, NULL, 0);
SStreamStateCur* pCur = streamStateSeekKeyNext(pState, &tmp);
int32_t code = streamStateGetKVByCur(pCur, key, NULL, 0);
streamStateFreeCur(pCur);
streamStateDel(pState, &tmp);
return code;
}
@ -422,7 +432,7 @@ SStreamStateCur* streamStateSeekKeyNext(SStreamState* pState, const SWinKey* key
return NULL;
}
pCur->number = pState->number;
if (tdbTbcOpen(pState->pStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}
@ -448,7 +458,7 @@ SStreamStateCur* streamStateFillSeekKeyNext(SStreamState* pState, const SWinKey*
if (!pCur) {
return NULL;
}
if (tdbTbcOpen(pState->pFillStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pFillStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}
@ -473,7 +483,7 @@ SStreamStateCur* streamStateFillSeekKeyPrev(SStreamState* pState, const SWinKey*
if (pCur == NULL) {
return NULL;
}
if (tdbTbcOpen(pState->pFillStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pFillStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}
@ -520,7 +530,8 @@ void streamFreeVal(void* val) { tdbFree(val); }
int32_t streamStateSessionPut(SStreamState* pState, const SSessionKey* key, const void* value, int32_t vLen) {
SStateSessionKey sKey = {.key = *key, .opNum = pState->number};
return tdbTbUpsert(pState->pSessionStateDb, &sKey, sizeof(SStateSessionKey), value, vLen, &pState->txn);
return tdbTbUpsert(pState->pTdbState->pSessionStateDb, &sKey, sizeof(SStateSessionKey), value, vLen,
&pState->pTdbState->txn);
}
int32_t streamStateSessionGet(SStreamState* pState, SSessionKey* key, void** pVal, int32_t* pVLen) {
@ -543,7 +554,7 @@ int32_t streamStateSessionGet(SStreamState* pState, SSessionKey* key, void** pVa
int32_t streamStateSessionDel(SStreamState* pState, const SSessionKey* key) {
SStateSessionKey sKey = {.key = *key, .opNum = pState->number};
return tdbTbDelete(pState->pSessionStateDb, &sKey, sizeof(SStateSessionKey), &pState->txn);
return tdbTbDelete(pState->pTdbState->pSessionStateDb, &sKey, sizeof(SStateSessionKey), &pState->pTdbState->txn);
}
SStreamStateCur* streamStateSessionSeekKeyCurrentPrev(SStreamState* pState, const SSessionKey* key) {
@ -552,7 +563,7 @@ SStreamStateCur* streamStateSessionSeekKeyCurrentPrev(SStreamState* pState, cons
return NULL;
}
pCur->number = pState->number;
if (tdbTbcOpen(pState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}
@ -579,7 +590,7 @@ SStreamStateCur* streamStateSessionSeekKeyCurrentNext(SStreamState* pState, cons
return NULL;
}
pCur->number = pState->number;
if (tdbTbcOpen(pState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}
@ -607,7 +618,7 @@ SStreamStateCur* streamStateSessionSeekKeyNext(SStreamState* pState, const SSess
return NULL;
}
pCur->number = pState->number;
if (tdbTbcOpen(pState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}
@ -674,7 +685,7 @@ int32_t streamStateSessionGetKeyByRange(SStreamState* pState, const SSessionKey*
return -1;
}
pCur->number = pState->number;
if (tdbTbcOpen(pState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return -1;
}
@ -821,13 +832,19 @@ _end:
}
int32_t streamStatePutParName(SStreamState* pState, int64_t groupId, const char tbname[TSDB_TABLE_NAME_LEN]) {
tdbTbUpsert(pState->pParNameDb, &groupId, sizeof(int64_t), tbname, TSDB_TABLE_NAME_LEN, &pState->txn);
tdbTbUpsert(pState->pTdbState->pParNameDb, &groupId, sizeof(int64_t), tbname, TSDB_TABLE_NAME_LEN,
&pState->pTdbState->txn);
return 0;
}
int32_t streamStateGetParName(SStreamState* pState, int64_t groupId, void** pVal) {
int32_t len;
return tdbTbGet(pState->pParNameDb, &groupId, sizeof(int64_t), pVal, &len);
return tdbTbGet(pState->pTdbState->pParNameDb, &groupId, sizeof(int64_t), pVal, &len);
}
void streamStateDestroy(SStreamState* pState) {
taosMemoryFreeClear(pState->pTdbState);
taosMemoryFreeClear(pState);
}
#if 0
@ -837,7 +854,7 @@ char* streamStateSessionDump(SStreamState* pState) {
return NULL;
}
pCur->number = pState->number;
if (tdbTbcOpen(pState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
if (tdbTbcOpen(pState->pTdbState->pSessionStateDb, &pCur->pCur, NULL) < 0) {
streamStateFreeCur(pCur);
return NULL;
}

3
source/libs/sync/src/syncAppendEntriesReply.c
View File

@ -67,6 +67,9 @@ int32_t syncNodeOnAppendEntriesReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
if (pMsg->matchIndex > oldMatchIndex) {
syncIndexMgrSetIndex(ths->pMatchIndex, &(pMsg->srcId), pMsg->matchIndex);
syncMaybeAdvanceCommitIndex(ths);
// maybe update minMatchIndex
ths->minMatchIndex = syncMinMatchIndex(ths);
}
syncIndexMgrSetIndex(ths->pNextIndex, &(pMsg->srcId), pMsg->matchIndex + 1);

49
source/libs/sync/src/syncMain.c
View File

@ -243,6 +243,18 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) {
goto _DEL_WAL;
} else {
lastApplyIndex -= SYNC_VNODE_LOG_RETENTION;
SyncIndex beginIndex = pSyncNode->pLogStore->syncLogBeginIndex(pSyncNode->pLogStore);
SyncIndex endIndex = pSyncNode->pLogStore->syncLogEndIndex(pSyncNode->pLogStore);
bool isEmpty = pSyncNode->pLogStore->syncLogIsEmpty(pSyncNode->pLogStore);
if (isEmpty || !(lastApplyIndex >= beginIndex && lastApplyIndex <= endIndex)) {
sNTrace(pSyncNode, "new-snapshot-index:%" PRId64 ", empty:%d, do not delete wal", lastApplyIndex, isEmpty);
syncNodeRelease(pSyncNode);
return 0;
}
// vnode
if (pSyncNode->replicaNum > 1) {
// multi replicas
@ -300,26 +312,31 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) {
_DEL_WAL:
do {
SyncIndex snapshottingIndex = atomic_load_64(&pSyncNode->snapshottingIndex);
SSyncLogStoreData* pData = pSyncNode->pLogStore->data;
SyncIndex snapshotVer = walGetSnapshotVer(pData->pWal);
SyncIndex walCommitVer = walGetCommittedVer(pData->pWal);
SyncIndex wallastVer = walGetLastVer(pData->pWal);
if (lastApplyIndex <= walCommitVer) {
SyncIndex snapshottingIndex = atomic_load_64(&pSyncNode->snapshottingIndex);
if (snapshottingIndex == SYNC_INDEX_INVALID) {
atomic_store_64(&pSyncNode->snapshottingIndex, lastApplyIndex);
pSyncNode->snapshottingTime = taosGetTimestampMs();
if (snapshottingIndex == SYNC_INDEX_INVALID) {
atomic_store_64(&pSyncNode->snapshottingIndex, lastApplyIndex);
pSyncNode->snapshottingTime = taosGetTimestampMs();
code = walBeginSnapshot(pData->pWal, lastApplyIndex);
if (code == 0) {
sNTrace(pSyncNode, "wal snapshot begin, index:%" PRId64 ", last apply index:%" PRId64,
pSyncNode->snapshottingIndex, lastApplyIndex);
} else {
sNError(pSyncNode, "wal snapshot begin error since:%s, index:%" PRId64 ", last apply index:%" PRId64,
terrstr(terrno), pSyncNode->snapshottingIndex, lastApplyIndex);
atomic_store_64(&pSyncNode->snapshottingIndex, SYNC_INDEX_INVALID);
}
SSyncLogStoreData* pData = pSyncNode->pLogStore->data;
code = walBeginSnapshot(pData->pWal, lastApplyIndex);
if (code == 0) {
sNTrace(pSyncNode, "wal snapshot begin, index:%" PRId64 ", last apply index:%" PRId64,
pSyncNode->snapshottingIndex, lastApplyIndex);
} else {
sNError(pSyncNode, "wal snapshot begin error since:%s, index:%" PRId64 ", last apply index:%" PRId64,
terrstr(terrno), pSyncNode->snapshottingIndex, lastApplyIndex);
atomic_store_64(&pSyncNode->snapshottingIndex, SYNC_INDEX_INVALID);
sNTrace(pSyncNode, "snapshotting for %" PRId64 ", do not delete wal for new-snapshot-index:%" PRId64,
snapshottingIndex, lastApplyIndex);
}
} else {
sNTrace(pSyncNode, "snapshotting for %" PRId64 ", do not delete wal for new-snapshot-index:%" PRId64,
snapshottingIndex, lastApplyIndex);
}
} while (0);

12
source/libs/sync/src/syncRaftLog.c
View File

@ -375,7 +375,17 @@ static int32_t raftLogGetLastEntry(SSyncLogStore* pLogStore, SSyncRaftEntry** pp
int32_t raftLogUpdateCommitIndex(SSyncLogStore* pLogStore, SyncIndex index) {
SSyncLogStoreData* pData = pLogStore->data;
SWal* pWal = pData->pWal;
// ASSERT(walCommit(pWal, index) == 0);
// need not update
SyncIndex snapshotVer = walGetSnapshotVer(pWal);
SyncIndex walCommitVer = walGetCommittedVer(pWal);
SyncIndex wallastVer = walGetLastVer(pWal);
if (index < snapshotVer || index > wallastVer) {
// ignore
return 0;
}
int32_t code = walCommit(pWal, index);
if (code != 0) {
int32_t err = terrno;

24
source/libs/sync/src/syncTimeout.c
View File

@ -62,18 +62,20 @@ static int32_t syncNodeTimerRoutine(SSyncNode* ths) {
syncNodeCleanConfigIndex(ths);
}
// end timeout wal snapshot
int64_t timeNow = taosGetTimestampMs();
if (timeNow - ths->snapshottingIndex > SYNC_DEL_WAL_MS &&
atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) {
SSyncLogStoreData* pData = ths->pLogStore->data;
int32_t code = walEndSnapshot(pData->pWal);
if (code != 0) {
sNError(ths, "timer wal snapshot end error since:%s", terrstr());
return -1;
} else {
sNTrace(ths, "wal snapshot end, index:%" PRId64, atomic_load_64(&ths->snapshottingIndex));
atomic_store_64(&ths->snapshottingIndex, SYNC_INDEX_INVALID);
if (atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) {
// end timeout wal snapshot
if (timeNow - ths->snapshottingTime > SYNC_DEL_WAL_MS &&
atomic_load_64(&ths->snapshottingIndex) != SYNC_INDEX_INVALID) {
SSyncLogStoreData* pData = ths->pLogStore->data;
int32_t code = walEndSnapshot(pData->pWal);
if (code != 0) {
sNError(ths, "timer wal snapshot end error since:%s", terrstr());
return -1;
} else {
sNTrace(ths, "wal snapshot end, index:%" PRId64, atomic_load_64(&ths->snapshottingIndex));
atomic_store_64(&ths->snapshottingIndex, SYNC_INDEX_INVALID);
}
}
}

4
source/libs/sync/src/syncUtil.c
View File

@ -239,11 +239,11 @@ void syncPrintNodeLog(const char* flags, ELogLevel level, int32_t dflag, SSyncNo
"vgId:%d, sync %s "
"%s"
", tm:%" PRIu64 ", cmt:%" PRId64 ", fst:%" PRId64 ", lst:%" PRId64 ", min:%" PRId64 ", snap:%" PRId64
", snap-tm:%" PRIu64 ", sby:%d, aq:%d, bch:%d, r-num:%d, lcfg:%" PRId64
", snap-tm:%" PRIu64 ", sby:%d, aq:%d, snaping:%" PRId64 ", r-num:%d, lcfg:%" PRId64
", chging:%d, rsto:%d, dquorum:%d, elt:%" PRId64 ", hb:%" PRId64 ", %s, %s",
pNode->vgId, syncStr(pNode->state), eventLog, currentTerm, pNode->commitIndex, logBeginIndex,
logLastIndex, pNode->minMatchIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
pNode->pRaftCfg->isStandBy, aqItems, pNode->pRaftCfg->batchSize, pNode->replicaNum,
pNode->pRaftCfg->isStandBy, aqItems, pNode->snapshottingIndex, pNode->replicaNum,
pNode->pRaftCfg->lastConfigIndex, pNode->changing, pNode->restoreFinish, quorum,
pNode->electTimerLogicClock, pNode->heartbeatTimerLogicClockUser, peerStr, cfgStr);
}

12
source/libs/transport/src/transCli.c
View File

@ -791,6 +791,7 @@ void cliSend(SCliConn* pConn) {
int msgLen = transMsgLenFromCont(pMsg->contLen);
STransMsgHead* pHead = transHeadFromCont(pMsg->pCont);
pHead->ahandle = pCtx != NULL ? (uint64_t)pCtx->ahandle : 0;
pHead->noResp = REQUEST_NO_RESP(pMsg) ? 1 : 0;
pHead->persist = REQUEST_PERSIS_HANDLE(pMsg) ? 1 : 0;
@ -820,10 +821,15 @@ void cliSend(SCliConn* pConn) {
uv_timer_start((uv_timer_t*)pConn->timer, cliReadTimeoutCb, TRANS_READ_TIMEOUT, 0);
}
if (pTransInst->compressSize != -1 && pTransInst->compressSize < pMsg->contLen) {
msgLen = transCompressMsg(pMsg->pCont, pMsg->contLen) + sizeof(STransMsgHead);
pHead->msgLen = (int32_t)htonl((uint32_t)msgLen);
if (pHead->comp == 0) {
if (pTransInst->compressSize != -1 && pTransInst->compressSize < pMsg->contLen) {
msgLen = transCompressMsg(pMsg->pCont, pMsg->contLen) + sizeof(STransMsgHead);
pHead->msgLen = (int32_t)htonl((uint32_t)msgLen);
}
} else {
msgLen = (int32_t)ntohl((uint32_t)(pHead->msgLen));
}
tGDebug("%s conn %p %s is sent to %s, local info %s, len:%d", CONN_GET_INST_LABEL(pConn), pConn,
TMSG_INFO(pHead->msgType), pConn->dst, pConn->src, msgLen);

2
source/libs/transport/src/transSvr.c
View File

@ -398,7 +398,7 @@ static int uvPrepareSendData(SSvrMsg* smsg, uv_buf_t* wb) {
pHead->magicNum = htonl(TRANS_MAGIC_NUM);
// handle invalid drop_task resp, TD-20098
if (pMsg->msgType == TDMT_SCH_DROP_TASK && pMsg->code == TSDB_CODE_VND_INVALID_VGROUP_ID) {
if (pConn->inType == TDMT_SCH_DROP_TASK && pMsg->code == TSDB_CODE_VND_INVALID_VGROUP_ID) {
transQueuePop(&pConn->srvMsgs);
destroySmsg(smsg);
return -1;

23
source/libs/wal/src/walWrite.c
View File

@ -324,23 +324,36 @@ int32_t walEndSnapshot(SWal *pWal) {
// find files safe to delete
SWalFileInfo *pInfo = taosArraySearch(pWal->fileInfoSet, &tmp, compareWalFileInfo, TD_LE);
if (pInfo) {
SWalFileInfo *pLastFileInfo = taosArrayGetLast(pWal->fileInfoSet);
wDebug("vgId:%d, wal search found file info: first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer,
pInfo->lastVer);
if (ver >= pInfo->lastVer) {
pInfo--;
pInfo++;
wDebug("vgId:%d, wal remove advance one file: first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer,
pInfo->lastVer);
}
if (POINTER_DISTANCE(pInfo, pWal->fileInfoSet->pData) > 0) {
wDebug("vgId:%d, wal end remove for %" PRId64, pWal->cfg.vgId, pInfo->firstVer);
if (pInfo <= pLastFileInfo) {
wDebug("vgId:%d, wal end remove for first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, pInfo->firstVer,
pInfo->lastVer);
} else {
wDebug("vgId:%d, wal no remove", pWal->cfg.vgId);
}
// iterate files, until the searched result
for (SWalFileInfo *iter = pWal->fileInfoSet->pData; iter < pInfo; iter++) {
if ((pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize) ||
(pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts)) {
wDebug("vgId:%d, wal check remove file %" PRId64 "(file size %" PRId64 " close ts %" PRId64
"), new tot size %" PRId64,
pWal->cfg.vgId, iter->firstVer, iter->fileSize, iter->closeTs, newTotSize);
if (((pWal->cfg.retentionSize == 0) || (pWal->cfg.retentionSize != -1 && newTotSize > pWal->cfg.retentionSize)) ||
((pWal->cfg.retentionPeriod == 0) ||
(pWal->cfg.retentionPeriod != -1 && iter->closeTs + pWal->cfg.retentionPeriod > ts))) {
// delete according to file size or close time
wDebug("vgId:%d, check pass", pWal->cfg.vgId);
deleteCnt++;
newTotSize -= iter->fileSize;
}
wDebug("vgId:%d, check not pass", pWal->cfg.vgId);
}
wDebug("vgId:%d, wal should delete %d files", pWal->cfg.vgId, deleteCnt);
int32_t actualDelete = 0;
char fnameStr[WAL_FILE_LEN];
// remove file

2
source/util/src/tpagedbuf.c
View File

@ -107,7 +107,7 @@ static uint64_t allocatePositionInFile(SDiskbasedBuf* pBuf, size_t size) {
static void setPageNotInBuf(SPageInfo* pPageInfo) { pPageInfo->pData = NULL; }
static FORCE_INLINE size_t getAllocPageSize(int32_t pageSize) { return pageSize + POINTER_BYTES + 2; }
static FORCE_INLINE size_t getAllocPageSize(int32_t pageSize) { return pageSize + POINTER_BYTES + sizeof(SFilePage); }
/**
* +--------------------------+-------------------+--------------+

699
tests/parallel_test/cases.task
View File

@ -231,7 +231,7 @@
,,y,script,./test.sh -f tsim/stream/windowClose.sim
,,y,script,./test.sh -f tsim/stream/ignoreExpiredData.sim
,,y,script,./test.sh -f tsim/stream/sliding.sim
,,y,script,./test.sh -f tsim/stream/partitionbyColumnInterval.sim
,,n,script,./test.sh -f tsim/stream/partitionbyColumnInterval.sim
,,y,script,./test.sh -f tsim/stream/partitionbyColumnSession.sim
,,y,script,./test.sh -f tsim/stream/partitionbyColumnState.sim
,,y,script,./test.sh -f tsim/stream/deleteInterval.sim
@ -279,6 +279,7 @@
,,y,script,./test.sh -f tsim/stable/vnode3.sim
,,y,script,./test.sh -f tsim/stable/metrics_idx.sim
,,n,script,./test.sh -f tsim/sma/drop_sma.sim
,,n,script,./test.sh -f tsim/sma/sma_leak.sim
,,y,script,./test.sh -f tsim/sma/tsmaCreateInsertQuery.sim
,,y,script,./test.sh -f tsim/sma/rsmaCreateInsertQuery.sim
,,y,script,./test.sh -f tsim/sma/rsmaPersistenceRecovery.sim
@ -415,208 +416,208 @@
,,,system-test,python3 ./test.py -f 0-others/sysinfo.py
,,,system-test,python3 ./test.py -f 0-others/user_control.py
,,,system-test,python3 ./test.py -f 0-others/fsync.py
,,,system-test,python3 ./test.py -f 0-others/compatibility.py
,,,system-test,python3 ./test.py -f 0-others/compatibility.py
,,,system-test,python3 ./test.py -f 1-insert/alter_database.py
,,,system-test,python3 ./test.py -f 1-insert/influxdb_line_taosc_insert.py
,,,system-test,python3 ./test.py -f 1-insert/opentsdb_telnet_line_taosc_insert.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/opentsdb_telnet_line_taosc_insert.py
,,,system-test,python3 ./test.py -f 1-insert/opentsdb_json_taosc_insert.py
,,,system-test,python3 ./test.py -f 1-insert/test_stmt_muti_insert_query.py
,,,system-test,python3 ./test.py -f 1-insert/test_stmt_set_tbname_tag.py
,,,system-test,python3 ./test.py -f 1-insert/alter_stable.py
,,,system-test,python3 ./test.py -f 1-insert/alter_table.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/alter_stable.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/alter_table.py
,,,system-test,python3 ./test.py -f 1-insert/boundary.py
,,,system-test,python3 ./test.py -f 1-insert/insertWithMoreVgroup.py
,,,system-test,python3 ./test.py -f 1-insert/table_comment.py
,,,system-test,python3 ./test.py -f 1-insert/time_range_wise.py
,,,system-test,python3 ./test.py -f 1-insert/block_wise.py
,,,system-test,python3 ./test.py -f 1-insert/create_retentions.py
,,,system-test,python3 ./test.py -f 1-insert/table_param_ttl.py
,,,system-test,python3 ./test.py -f 1-insert/mutil_stage.py
,,,system-test,python3 ./test.py -f 1-insert/table_param_ttl.py -R
,,,system-test,python3 ./test.py -f 1-insert/update_data_muti_rows.py
,,,system-test,python3 ./test.py -f 1-insert/db_tb_name_check.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/table_param_ttl.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/table_param_ttl.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/update_data_muti_rows.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/db_tb_name_check.py
,,,system-test,python3 ./test.py -f 1-insert/database_pre_suf.py
,,,system-test,python3 ./test.py -f 1-insert/InsertFuturets.py
,,,system-test,python3 ./test.py -f 0-others/show.py
,,,system-test,python3 ./test.py -f 2-query/abs.py
,,,system-test,python3 ./test.py -f 2-query/abs.py -R
,,,system-test,python3 ./test.py -f 2-query/and_or_for_byte.py
,,,system-test,python3 ./test.py -f 2-query/and_or_for_byte.py -R
,,,system-test,python3 ./test.py -f 2-query/apercentile.py
,,,system-test,python3 ./test.py -f 2-query/apercentile.py -R
,,,system-test,python3 ./test.py -f 2-query/arccos.py
,,,system-test,python3 ./test.py -f 2-query/arccos.py -R
,,,system-test,python3 ./test.py -f 2-query/arcsin.py
,,,system-test,python3 ./test.py -f 2-query/arcsin.py -R
,,,system-test,python3 ./test.py -f 2-query/arctan.py
,,,system-test,python3 ./test.py -f 2-query/arctan.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/show.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/and_or_for_byte.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/and_or_for_byte.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -R
,,,system-test,python3 ./test.py -f 2-query/avg.py
,,,system-test,python3 ./test.py -f 2-query/avg.py -R
,,,system-test,python3 ./test.py -f 2-query/between.py
,,,system-test,python3 ./test.py -f 2-query/between.py -R
,,,system-test,python3 ./test.py -f 2-query/bottom.py
,,,system-test,python3 ./test.py -f 2-query/bottom.py -R
,,,system-test,python3 ./test.py -f 2-query/cast.py
,,,system-test,python3 ./test.py -f 2-query/cast.py -R
,,,system-test,python3 ./test.py -f 2-query/ceil.py
,,,system-test,python3 ./test.py -f 2-query/ceil.py -R
,,,system-test,python3 ./test.py -f 2-query/char_length.py
,,,system-test,python3 ./test.py -f 2-query/char_length.py -R
,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py
,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -R
,,,system-test,python3 ./test.py -f 2-query/concat.py
,,,system-test,python3 ./test.py -f 2-query/concat.py -R
,,,system-test,python3 ./test.py -f 2-query/concat_ws.py
,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -R
,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py
,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -R
,,,system-test,python3 ./test.py -f 2-query/cos.py
,,,system-test,python3 ./test.py -f 2-query/cos.py -R
,,,system-test,python3 ./test.py -f 2-query/count_partition.py
,,,system-test,python3 ./test.py -f 2-query/count_partition.py -R
,,,system-test,python3 ./test.py -f 2-query/count.py
,,,system-test,python3 ./test.py -f 2-query/count.py -R
,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py
,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -R
,,,system-test,python3 ./test.py -f 2-query/db.py
,,,system-test,python3 ./test.py -f 2-query/db.py -R
,,,system-test,python3 ./test.py -f 2-query/diff.py
,,,system-test,python3 ./test.py -f 2-query/diff.py -R
,,,system-test,python3 ./test.py -f 2-query/distinct.py
,,,system-test,python3 ./test.py -f 2-query/distinct.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -R
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -R
,,,system-test,python3 ./test.py -f 2-query/explain.py
,,,system-test,python3 ./test.py -f 2-query/explain.py -R
,,,system-test,python3 ./test.py -f 2-query/first.py
,,,system-test,python3 ./test.py -f 2-query/first.py -R
,,,system-test,python3 ./test.py -f 2-query/floor.py
,,,system-test,python3 ./test.py -f 2-query/floor.py -R
,,,system-test,python3 ./test.py -f 2-query/function_null.py
,,,system-test,python3 ./test.py -f 2-query/function_null.py -R
,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py
,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -R
,,,system-test,python3 ./test.py -f 2-query/histogram.py
,,,system-test,python3 ./test.py -f 2-query/histogram.py -R
,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py
,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/db.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/db.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_stateduration.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_stateduration.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/histogram.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/histogram.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -R
,,,system-test,python3 ./test.py -f 2-query/interp.py
,,,system-test,python3 ./test.py -f 2-query/interp.py -R
,,,system-test,python3 ./test.py -f 2-query/irate.py
,,,system-test,python3 ./test.py -f 2-query/irate.py -R
,,,system-test,python3 ./test.py -f 2-query/join.py
,,,system-test,python3 ./test.py -f 2-query/join.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -R
,,,system-test,python3 ./test.py -f 2-query/last_row.py
,,,system-test,python3 ./test.py -f 2-query/last_row.py -R
,,,system-test,python3 ./test.py -f 2-query/last.py
,,,system-test,python3 ./test.py -f 2-query/last.py -R
,,,system-test,python3 ./test.py -f 2-query/leastsquares.py
,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -R
,,,system-test,python3 ./test.py -f 2-query/length.py
,,,system-test,python3 ./test.py -f 2-query/length.py -R
,,,system-test,python3 ./test.py -f 2-query/log.py
,,,system-test,python3 ./test.py -f 2-query/log.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -R
,,,system-test,python3 ./test.py -f 2-query/lower.py
,,,system-test,python3 ./test.py -f 2-query/lower.py -R
,,,system-test,python3 ./test.py -f 2-query/ltrim.py
,,,system-test,python3 ./test.py -f 2-query/ltrim.py -R
,,,system-test,python3 ./test.py -f 2-query/mavg.py
,,,system-test,python3 ./test.py -f 2-query/mavg.py -R
,,,system-test,python3 ./test.py -f 2-query/max_partition.py
,,,system-test,python3 ./test.py -f 2-query/max_partition.py -R
,,,system-test,python3 ./test.py -f 2-query/max.py
,,,system-test,python3 ./test.py -f 2-query/max.py -R
,,,system-test,python3 ./test.py -f 2-query/min.py
,,,system-test,python3 ./test.py -f 2-query/min.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -R
,,,system-test,python3 ./test.py -f 2-query/mode.py
,,,system-test,python3 ./test.py -f 2-query/mode.py -R
,,,system-test,python3 ./test.py -f 2-query/Now.py
,,,system-test,python3 ./test.py -f 2-query/Now.py -R
,,,system-test,python3 ./test.py -f 2-query/percentile.py
,,,system-test,python3 ./test.py -f 2-query/percentile.py -R
,,,system-test,python3 ./test.py -f 2-query/pow.py
,,,system-test,python3 ./test.py -f 2-query/pow.py -R
,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py
,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -R
,,,system-test,python3 ./test.py -f 2-query/round.py
,,,system-test,python3 ./test.py -f 2-query/round.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -R
,,,system-test,python3 ./test.py -f 2-query/rtrim.py
,,,system-test,python3 ./test.py -f 2-query/rtrim.py -R
,,,system-test,python3 ./test.py -f 2-query/sample.py
,,,system-test,python3 ./test.py -f 2-query/sample.py -R
,,,system-test,python3 ./test.py -f 2-query/sin.py
,,,system-test,python3 ./test.py -f 2-query/sin.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sample.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sample.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -R
,,,system-test,python3 ./test.py -f 2-query/smaTest.py
,,,system-test,python3 ./test.py -f 2-query/smaTest.py -R
,,,system-test,python3 ./test.py -f 2-query/sml.py
,,,system-test,python3 ./test.py -f 2-query/sml.py -R
,,,system-test,python3 ./test.py -f 2-query/spread.py
,,,system-test,python3 ./test.py -f 2-query/spread.py -R
,,,system-test,python3 ./test.py -f 2-query/sqrt.py
,,,system-test,python3 ./test.py -f 2-query/sqrt.py -R
,,,system-test,python3 ./test.py -f 2-query/statecount.py
,,,system-test,python3 ./test.py -f 2-query/statecount.py -R
,,,system-test,python3 ./test.py -f 2-query/stateduration.py
,,,system-test,python3 ./test.py -f 2-query/stateduration.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/statecount.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/statecount.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -R
,,,system-test,python3 ./test.py -f 2-query/substr.py
,,,system-test,python3 ./test.py -f 2-query/substr.py -R
,,,system-test,python3 ./test.py -f 2-query/sum.py
,,,system-test,python3 ./test.py -f 2-query/sum.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sum.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sum.py -R
,,,system-test,python3 ./test.py -f 2-query/tail.py
,,,system-test,python3 ./test.py -f 2-query/tail.py -R
,,,system-test,python3 ./test.py -f 2-query/tan.py
,,,system-test,python3 ./test.py -f 2-query/tan.py -R
,,,system-test,python3 ./test.py -f 2-query/Timediff.py
,,,system-test,python3 ./test.py -f 2-query/Timediff.py -R
,,,system-test,python3 ./test.py -f 2-query/timetruncate.py
,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -R
,,,system-test,python3 ./test.py -f 2-query/timezone.py
,,,system-test,python3 ./test.py -f 2-query/timezone.py -R
,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py
,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -R
,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py
,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -R
,,,system-test,python3 ./test.py -f 2-query/Today.py
,,,system-test,python3 ./test.py -f 2-query/Today.py -R
,,,system-test,python3 ./test.py -f 2-query/top.py
,,,system-test,python3 ./test.py -f 2-query/top.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -R
,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py
,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -R
,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py
,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -R
,,,system-test,python3 ./test.py -f 2-query/twa.py
,,,system-test,python3 ./test.py -f 2-query/twa.py -R
,,,system-test,python3 ./test.py -f 2-query/union.py
,,,system-test,python3 ./test.py -f 2-query/union.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/union.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/union.py -R
,,,system-test,python3 ./test.py -f 2-query/unique.py
,,,system-test,python3 ./test.py -f 2-query/unique.py -R
,,,system-test,python3 ./test.py -f 2-query/upper.py
,,,system-test,python3 ./test.py -f 2-query/upper.py -R
,,,system-test,python3 ./test.py -f 2-query/varchar.py
,,,system-test,python3 ./test.py -f 2-query/varchar.py -R
,,,system-test,python3 ./test.py -f 2-query/case_when.py
,,,system-test,python3 ./test.py -f 2-query/case_when.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -R
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -R
,,,system-test,python3 ./test.py -f 1-insert/update_data.py
,,,system-test,python3 ./test.py -f 1-insert/tb_100w_data_order.py
,,,system-test,python3 ./test.py -f 1-insert/delete_stable.py
,,,system-test,python3 ./test.py -f 1-insert/delete_childtable.py
,,,system-test,python3 ./test.py -f 1-insert/delete_normaltable.py
,,,system-test,python3 ./test.py -f 1-insert/keep_expired.py
,,,system-test,python3 ./test.py -f 2-query/join2.py
,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/keep_expired.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py
,,,system-test,python3 ./test.py -f 2-query/union1.py
,,,system-test,python3 ./test.py -f 2-query/concat2.py
,,,system-test,python3 ./test.py -f 2-query/json_tag.py
@ -626,9 +627,9 @@
,,,system-test,python3 ./test.py -f 2-query/nestedQuery_time.py
,,,system-test,python3 ./test.py -f 2-query/stablity.py
,,,system-test,python3 ./test.py -f 2-query/stablity_1.py
,,,system-test,python3 ./test.py -f 2-query/elapsed.py
,,,system-test,python3 ./test.py -f 2-query/csum.py
,,,system-test,python3 ./test.py -f 2-query/function_diff.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/elapsed.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/csum.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_diff.py
,,,system-test,python3 ./test.py -f 2-query/queryQnode.py
,,,system-test,python3 ./test.py -f 6-cluster/5dnode1mnode.py
,,,system-test,python3 ./test.py -f 6-cluster/5dnode2mnode.py -N 5
@ -665,7 +666,7 @@
,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_insertdatas.py -N 4 -M 1
,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_insertdatas_querys.py -N 4 -M 1
,,,system-test,python3 test.py -f 6-cluster/vnode/4dnode1mnode_basic_replica3_vgroups.py -N 4 -M 1
,,,system-test,python3 ./test.py -f 7-tmq/create_wrong_topic.py
,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/create_wrong_topic.py
,,,system-test,python3 ./test.py -f 7-tmq/dropDbR3ConflictTransaction.py -N 3
,,,system-test,python3 ./test.py -f 7-tmq/basic5.py
,,,system-test,python3 ./test.py -f 7-tmq/subscribeDb.py
@ -703,6 +704,7 @@
,,,system-test,python3 ./test.py -f 7-tmq/tmqConsFromTsdb1-mutilVg-mutilCtb.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqAutoCreateTbl.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqDnodeRestart.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqDnodeRestart1.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdate-1ctb.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdateWithConsume.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqUpdate-multiCtb-snapshot0.py
@ -723,18 +725,18 @@
,,,system-test,python3 ./test.py -f 7-tmq/stbTagFilter-multiCtb.py
,,,system-test,python3 ./test.py -f 99-TDcase/TD-19201.py
,,,system-test,python3 ./test.py -f 7-tmq/tmqSubscribeStb-r3.py -N 5
,,,system-test,python3 ./test.py -f 2-query/between.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/length.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/join.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/cast.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/union.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/union1.py -Q 2
@ -742,45 +744,45 @@
,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/max.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/min.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/spread.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/count.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/last.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/first.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/top.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/round.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/log.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 2
@ -791,44 +793,44 @@
,,,system-test,python3 ./test.py -f 2-query/avg.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/csum.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/sample.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/unique.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/tail.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/twa.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/irate.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 2
,,,system-test,python3 ./test.py -f 2-query/between.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 2
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/length.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/join.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/cast.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join2.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/union.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/union1.py -Q 3
@ -836,45 +838,45 @@
,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/max.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/min.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/count.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/last.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/first.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/top.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/round.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/log.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/nestedQuery_math.py -Q 3
@ -884,43 +886,43 @@
,,,system-test,python3 ./test.py -f 2-query/avg.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/csum.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/sample.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/unique.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/tail.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/twa.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/irate.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 3
,,,system-test,python3 ./test.py -f 2-query/between.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distinct.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/varchar.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 3
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/between.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distinct.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/varchar.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/ltrim.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/rtrim.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/length.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/char_length.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/upper.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/lower.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/join.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/length.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/char_length.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/upper.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/lower.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/join.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/join2.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/substr.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/union.py -Q 4
@ -929,45 +931,45 @@
,,,system-test,python3 ./test.py -f 2-query/concat2.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/concat_ws.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/concat_ws2.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/check_tsdb.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/check_tsdb.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/spread.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/hyperloglog.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/explain.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/leastsquares.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/timezone.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/Now.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/Today.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/max.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/min.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/hyperloglog.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/explain.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/leastsquares.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timezone.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Now.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Today.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/min.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/mode.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/count.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/last.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/first.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/To_iso8601.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/timetruncate.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/diff.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/Timediff.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/json_tag.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/top.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/bottom.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/percentile.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/apercentile.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/abs.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/ceil.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/floor.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/round.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/log.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/pow.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/sqrt.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/sin.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/cos.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/tan.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/arcsin.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/arccos.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/arctan.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/countAlwaysReturnValue.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/last.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/first.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_iso8601.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/To_unixtimestamp.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/timetruncate.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/diff.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/Timediff.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/json_tag.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/top.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/bottom.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/percentile.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/apercentile.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/abs.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ceil.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/floor.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/round.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/log.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/pow.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sqrt.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sin.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cos.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/tan.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arcsin.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arccos.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/arctan.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/query_cols_tags_and_or.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/nestedQuery.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/nestedQuery_str.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/nestedQuery_math.py -Q 4
@ -977,33 +979,34 @@
,,,system-test,python3 ./test.py -f 2-query/avg.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/elapsed.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/csum.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/mavg.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/mavg.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/sample.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cast.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/function_diff.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/unique.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/stateduration.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/stateduration.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/function_stateduration.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/statecount.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/tail.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/ttl_comment.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_count.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_max.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_min.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/twa.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/irate.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/function_null.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/count_partition.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/max_partition.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ttl_comment.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_count.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_max.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_min.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_sum.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_spread.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_apercentile.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_avg.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/distribute_agg_stddev.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/twa.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/irate.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/function_null.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/count_partition.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/max_partition.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/last_row.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/tsbsQuery.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/sml.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/sml.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/interp.py -Q 4
,,,system-test,python3 ./test.py -f 2-query/case_when.py -Q 4
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/case_when.py -Q 4
#develop test
,,,develop-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/auto_create_table_json.py

365
tests/pytest/auto_crash_gen.py Executable file
View File

@ -0,0 +1,365 @@
import os
import socket
import requests
# -*- coding: utf-8 -*-
import os ,sys
import random
import argparse
import subprocess
import time
import platform
# valgrind mode ?
valgrind_mode = False
msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" }
# formal
hostname = socket.gethostname()
group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9'
def get_msg(text):
return {
"msg_type": "post",
"content": {
"post": {
"zh_cn": {
"title": "Crash_gen Monitor",
"content": [
[{
"tag": "text",
"text": text
}
]]
}
}
}
}
def send_msg(json):
headers = {
'Content-Type': 'application/json'
}
req = requests.post(url=group_url, headers=headers, json=json)
inf = req.json()
if "StatusCode" in inf and inf["StatusCode"] == 0:
pass
else:
print(inf)
# set path about run instance
core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
core_path = "/".join(core_path.split("/")[:-1])
print(" ======= core path is %s ======== " %core_path)
if not os.path.exists(core_path):
os.mkdir(core_path)
base_dir = os.path.dirname(os.path.realpath(__file__))
if base_dir.find("community")>0:
repo = "community"
elif base_dir.find("TDengine")>0:
repo = "TDengine"
else:
repo ="TDengine"
print("base_dir:",base_dir)
home_dir = base_dir[:base_dir.find(repo)]
print("home_dir:",home_dir)
run_dir = os.path.join(home_dir,'run_dir')
run_dir = os.path.abspath(run_dir)
print("run dir is *** :",run_dir)
if not os.path.exists(run_dir):
os.mkdir(run_dir)
run_log_file = run_dir+'/crash_gen_run.log'
crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh')
exit_status_logs = os.path.join(run_dir, 'crash_exit.log')
def get_path():
buildPath=''
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
projPath = selfPath[:selfPath.find("community")]
else:
projPath = selfPath[:selfPath.find("tests")]
for root, dirs, files in os.walk(projPath):
if ("taosd" in files):
rootRealPath = os.path.dirname(os.path.realpath(root))
if ("packaging" not in rootRealPath):
buildPath = root[:len(root) - len("/build/bin")]
break
return buildPath
# generate crash_gen start script randomly
def random_args(args_list):
nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments
bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
"--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"
] # record bool type arguments
strs_args_list = ["--connector-type"] # record str type arguments
args_list["--auto-start-service"]= False
args_list["--continue-on-exception"]=True
# connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test"
connect_types=['native']
# args_list["--connector-type"]=connect_types[random.randint(0,2)]
args_list["--connector-type"]= connect_types[0]
args_list["--max-dbs"]= random.randint(1,10)
# dnodes = [1,3] # set single dnodes;
# args_list["--num-dnodes"]= random.sample(dnodes,1)[0]
# args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"])
args_list["--debug"]=False
args_list["--per-thread-db-connection"]=True
args_list["--track-memory-leaks"]=False
args_list["--max-steps"]=random.randint(500,2000)
# args_list["--ignore-errors"]=[] ## can add error codes for detail
args_list["--run-tdengine"]= False
args_list["--use-shadow-db"]= False
args_list["--dynamic-db-table-names"]= True
args_list["--verify-data"]= False
args_list["--record-ops"] = False
for key in bools_args_list:
set_bool_value = [True,False]
if key == "--auto-start-service" :
continue
elif key =="--run-tdengine":
continue
elif key == "--ignore-errors":
continue
elif key == "--debug":
continue
elif key == "--per-thread-db-connection":
continue
elif key == "--continue-on-exception":
continue
elif key == "--use-shadow-db":
continue
elif key =="--track-memory-leaks":
continue
elif key == "--dynamic-db-table-names":
continue
elif key == "--verify-data":
continue
elif key == "--record-ops":
continue
else:
args_list[key]=set_bool_value[random.randint(0,1)]
if args_list["--larger-data"]:
threads = [16,32]
else:
threads = [32,64,128,256]
args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug
return args_list
def limits(args_list):
if args_list["--use-shadow-db"]==True:
if args_list["--max-dbs"] > 1:
print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1")
args_list["--max-dbs"]=1
else:
pass
# env is start by test frame , not crash_gen instance
# elif args_list["--num-replicas"]==0:
# print(" make sure num-replicas is at least 1 ")
# args_list["--num-replicas"]=1
# elif args_list["--num-replicas"]==1:
# pass
# elif args_list["--num-replicas"]>1:
# if not args_list["--auto-start-service"]:
# print("it should be deployed by crash_gen auto-start-service for multi replicas")
# else:
# pass
return args_list
def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode):
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
"--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"]
arguments = ""
for k ,v in args_list.items():
if k == "--ignore-errors":
if v:
arguments+=(k+"="+str(v)+" ")
else:
arguments+=""
elif k in bools_args_list and v==True:
arguments+=(k+" ")
elif k in bools_args_list and v==False:
arguments+=""
else:
arguments+=(k+"="+str(v)+" ")
if valgrind :
crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203 '%(crash_gen_path ,arguments)
else:
crash_gen_cmd = 'cd %s && ./crash_gen.sh %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203'%(crash_gen_path ,arguments)
return crash_gen_cmd
def start_taosd():
build_path = get_path()
if repo == "community":
start_path = build_path[:-5]+"community/tests/system-test/"
elif repo == "TDengine":
start_path = build_path[:-5]+"/tests/system-test/"
else:
pass
start_cmd = 'cd %s && python3 test.py >>/dev/null '%(start_path)
os.system(start_cmd)
def get_cmds(args_list):
# build_path = get_path()
# if repo == "community":
# crash_gen_path = build_path[:-5]+"community/tests/pytest/"
# elif repo == "TDengine":
# crash_gen_path = build_path[:-5]+"/tests/pytest/"
# else:
# pass
# crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550 '%(crash_gen_path)
crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode)
return crash_gen_cmd
def run_crash_gen(crash_cmds):
# prepare env of taosd
start_taosd()
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
result_file = os.path.join(crash_gen_path, 'valgrind.out')
# run crash_gen and back logs
os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file))
os.system("%s >>%s "%(crash_cmds,result_file))
def check_status():
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
result_file = os.path.join(crash_gen_path, 'valgrind.out')
run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs))
core_check = subprocess.Popen('ls -l %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if int(core_check.strip().rstrip()) > 0:
# it means core files has occured
return 3
if "Crash_Gen is now exiting with status code: 1" in run_code:
return 1
elif "Crash_Gen is now exiting with status code: 0" in run_code:
return 0
else:
return 2
def main():
args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[],
"--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False,
"--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False ,
"--continue-on-exception":False }
args = random_args(args_list)
args = limits(args)
build_path = get_path()
os.system("pip install git+https://github.com/taosdata/taos-connector-python.git")
if repo =="community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo =="TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
if os.path.exists(crash_gen_path+"crash_gen.sh"):
print(" make sure crash_gen.sh is ready")
else:
print( " crash_gen.sh is not exists ")
sys.exit(1)
git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16]
# crash_cmds = get_cmds()
crash_cmds = get_cmds(args)
# clean run_dir
os.system('rm -rf %s'%run_dir )
if not os.path.exists(run_dir):
os.mkdir(run_dir)
print(crash_cmds)
run_crash_gen(crash_cmds)
status = check_status()
print("exit status : ", status)
if status ==4:
print('======== crash_gen found memory bugs ========')
if status ==5:
print('======== crash_gen found memory errors ========')
if status >0:
print('======== crash_gen run failed and not exit as expected ========')
else:
print('======== crash_gen run sucess and exit as expected ========')
if status!=0 :
try:
text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n " + f" and git commit : {git_commit}"
send_msg(get_msg(text))
except Exception as e:
print("exception:", e)
exit(status)
if __name__ == '__main__':
main()

399
tests/pytest/auto_crash_gen_valgrind.py Executable file
View File

@ -0,0 +1,399 @@
#!/usr/bin/python3
import os
import socket
import requests
# -*- coding: utf-8 -*-
import os ,sys
import random
import argparse
import subprocess
import time
import platform
# valgrind mode ?
valgrind_mode = True
msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" }
# formal
hostname = socket.gethostname()
group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9'
def get_msg(text):
return {
"msg_type": "post",
"content": {
"post": {
"zh_cn": {
"title": "Crash_gen Monitor",
"content": [
[{
"tag": "text",
"text": text
}
]]
}
}
}
}
def send_msg(json):
headers = {
'Content-Type': 'application/json'
}
req = requests.post(url=group_url, headers=headers, json=json)
inf = req.json()
if "StatusCode" in inf and inf["StatusCode"] == 0:
pass
else:
print(inf)
# set path about run instance
core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
core_path = "/".join(core_path.split("/")[:-1])
print(" ======= core path is %s ======== " %core_path)
if not os.path.exists(core_path):
os.mkdir(core_path)
base_dir = os.path.dirname(os.path.realpath(__file__))
if base_dir.find("community")>0:
repo = "community"
elif base_dir.find("TDengine")>0:
repo = "TDengine"
else:
repo ="TDengine"
print("base_dir:",base_dir)
home_dir = base_dir[:base_dir.find(repo)]
print("home_dir:",home_dir)
run_dir = os.path.join(home_dir,'run_dir')
run_dir = os.path.abspath(run_dir)
print("run dir is *** :",run_dir)
if not os.path.exists(run_dir):
os.mkdir(run_dir)
run_log_file = run_dir+'/crash_gen_run.log'
crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh')
exit_status_logs = os.path.join(run_dir, 'crash_exit.log')
def get_path():
buildPath=''
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
projPath = selfPath[:selfPath.find("community")]
else:
projPath = selfPath[:selfPath.find("tests")]
for root, dirs, files in os.walk(projPath):
if ("taosd" in files):
rootRealPath = os.path.dirname(os.path.realpath(root))
if ("packaging" not in rootRealPath):
buildPath = root[:len(root) - len("/build/bin")]
break
return buildPath
# generate crash_gen start script randomly
def random_args(args_list):
nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments
bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
"--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"
] # record bool type arguments
strs_args_list = ["--connector-type"] # record str type arguments
args_list["--auto-start-service"]= False
args_list["--continue-on-exception"]=True
# connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test"
connect_types=['native']
# args_list["--connector-type"]=connect_types[random.randint(0,2)]
args_list["--connector-type"]= connect_types[0]
args_list["--max-dbs"]= random.randint(1,10)
# dnodes = [1,3] # set single dnodes;
# args_list["--num-dnodes"]= random.sample(dnodes,1)[0]
# args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"])
args_list["--debug"]=False
args_list["--per-thread-db-connection"]=True
args_list["--track-memory-leaks"]=False
args_list["--max-steps"]=random.randint(200,500)
threads = [16,32]
args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug
# args_list["--ignore-errors"]=[] ## can add error codes for detail
args_list["--run-tdengine"]= False
args_list["--use-shadow-db"]= False
args_list["--dynamic-db-table-names"]= True
args_list["--verify-data"]= False
args_list["--record-ops"] = False
for key in bools_args_list:
set_bool_value = [True,False]
if key == "--auto-start-service" :
continue
elif key =="--run-tdengine":
continue
elif key == "--ignore-errors":
continue
elif key == "--debug":
continue
elif key == "--per-thread-db-connection":
continue
elif key == "--continue-on-exception":
continue
elif key == "--use-shadow-db":
continue
elif key =="--track-memory-leaks":
continue
elif key == "--dynamic-db-table-names":
continue
elif key == "--verify-data":
continue
elif key == "--record-ops":
continue
elif key == "--larger-data":
continue
else:
args_list[key]=set_bool_value[random.randint(0,1)]
return args_list
def limits(args_list):
if args_list["--use-shadow-db"]==True:
if args_list["--max-dbs"] > 1:
print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1")
args_list["--max-dbs"]=1
else:
pass
# env is start by test frame , not crash_gen instance
# elif args_list["--num-replicas"]==0:
# print(" make sure num-replicas is at least 1 ")
# args_list["--num-replicas"]=1
# elif args_list["--num-replicas"]==1:
# pass
# elif args_list["--num-replicas"]>1:
# if not args_list["--auto-start-service"]:
# print("it should be deployed by crash_gen auto-start-service for multi replicas")
# else:
# pass
return args_list
def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode):
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
"--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"]
arguments = ""
for k ,v in args_list.items():
if k == "--ignore-errors":
if v:
arguments+=(k+"="+str(v)+" ")
else:
arguments+=""
elif k in bools_args_list and v==True:
arguments+=(k+" ")
elif k in bools_args_list and v==False:
arguments+=""
else:
arguments+=(k+"="+str(v)+" ")
if valgrind :
crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203 '%(crash_gen_path ,arguments)
else:
crash_gen_cmd = 'cd %s && ./crash_gen.sh %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0203'%(crash_gen_path ,arguments)
return crash_gen_cmd
def start_taosd():
build_path = get_path()
if repo == "community":
start_path = build_path[:-5]+"community/tests/system-test/"
elif repo == "TDengine":
start_path = build_path[:-5]+"/tests/system-test/"
else:
pass
start_cmd = 'cd %s && python3 test.py '%(start_path)
os.system(start_cmd +">>/dev/null")
def get_cmds(args_list):
# build_path = get_path()
# if repo == "community":
# crash_gen_path = build_path[:-5]+"community/tests/pytest/"
# elif repo == "TDengine":
# crash_gen_path = build_path[:-5]+"/tests/pytest/"
# else:
# pass
# crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550 '%(crash_gen_path)
crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode)
return crash_gen_cmd
def run_crash_gen(crash_cmds):
# prepare env of taosd
start_taosd()
# run crash_gen and back logs
os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file))
# os.system("cp %s %s"%(crash_gen_cmds_file, core_path))
os.system("%s"%(crash_cmds))
def check_status():
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
result_file = os.path.join(crash_gen_path, 'valgrind.out')
run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs))
core_check = subprocess.Popen('ls -l %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if int(core_check.strip().rstrip()) > 0:
# it means core files has occured
return 3
mem_status = check_memory()
if mem_status >0:
return mem_status
if "Crash_Gen is now exiting with status code: 1" in run_code:
return 1
elif "Crash_Gen is now exiting with status code: 0" in run_code:
return 0
else:
return 2
def check_memory():
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
'''
invalid read, invalid write
'''
back_path = os.path.join(core_path,"valgrind_report")
if not os.path.exists(back_path):
os.mkdir(back_path)
stderr_file = os.path.join(crash_gen_path , "valgrind.err")
status = 0
grep_res = subprocess.Popen("grep -i 'Invalid read' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if grep_res:
# os.system("cp %s %s"%(stderr_file , back_path))
status = 4
grep_res = subprocess.Popen("grep -i 'Invalid write' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if grep_res:
# os.system("cp %s %s"%(stderr_file , back_path))
status = 4
grep_res = subprocess.Popen("grep -i 'taosMemoryMalloc' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if grep_res:
# os.system("cp %s %s"%(stderr_file , back_path))
status = 5
return status
def main():
args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[],
"--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False,
"--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False ,
"--continue-on-exception":False }
args = random_args(args_list)
args = limits(args)
build_path = get_path()
os.system("pip install git+https://github.com/taosdata/taos-connector-python.git >>/dev/null")
if repo =="community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo =="TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
if os.path.exists(crash_gen_path+"crash_gen.sh"):
print(" make sure crash_gen.sh is ready")
else:
print( " crash_gen.sh is not exists ")
sys.exit(1)
git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16]
# crash_cmds = get_cmds()
crash_cmds = get_cmds(args)
# clean run_dir
os.system('rm -rf %s'%run_dir )
if not os.path.exists(run_dir):
os.mkdir(run_dir)
print(crash_cmds)
run_crash_gen(crash_cmds)
status = check_status()
# back_path = os.path.join(core_path,"valgrind_report")
print("exit status : ", status)
if status ==4:
print('======== crash_gen found memory bugs ========')
if status ==5:
print('======== crash_gen found memory errors ========')
if status >0:
print('======== crash_gen run failed and not exit as expected ========')
else:
print('======== crash_gen run sucess and exit as expected ========')
if status!=0 :
try:
text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n " + f" and git commit : {git_commit}"
send_msg(get_msg(text))
except Exception as e:
print("exception:", e)
exit(status)
if __name__ == '__main__':
main()

399
tests/pytest/auto_crash_gen_valgrind_cluster.py Executable file
View File

@ -0,0 +1,399 @@
#!/usr/bin/python3
import os
import socket
import requests
# -*- coding: utf-8 -*-
import os ,sys
import random
import argparse
import subprocess
import time
import platform
# valgrind mode ?
valgrind_mode = True
msg_dict = {0:"success" , 1:"failed" , 2:"other errors" , 3:"crash occured" , 4:"Invalid read/write" , 5:"memory leak" }
# formal
hostname = socket.gethostname()
group_url = 'https://open.feishu.cn/open-apis/bot/v2/hook/56c333b5-eae9-4c18-b0b6-7e4b7174f5c9'
def get_msg(text):
return {
"msg_type": "post",
"content": {
"post": {
"zh_cn": {
"title": "Crash_gen Monitor",
"content": [
[{
"tag": "text",
"text": text
}
]]
}
}
}
}
def send_msg(json):
headers = {
'Content-Type': 'application/json'
}
req = requests.post(url=group_url, headers=headers, json=json)
inf = req.json()
if "StatusCode" in inf and inf["StatusCode"] == 0:
pass
else:
print(inf)
# set path about run instance
core_path = subprocess.Popen("cat /proc/sys/kernel/core_pattern", shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
core_path = "/".join(core_path.split("/")[:-1])
print(" ======= core path is %s ======== " %core_path)
if not os.path.exists(core_path):
os.mkdir(core_path)
base_dir = os.path.dirname(os.path.realpath(__file__))
if base_dir.find("community")>0:
repo = "community"
elif base_dir.find("TDengine")>0:
repo = "TDengine"
else:
repo ="TDengine"
print("base_dir:",base_dir)
home_dir = base_dir[:base_dir.find(repo)]
print("home_dir:",home_dir)
run_dir = os.path.join(home_dir,'run_dir')
run_dir = os.path.abspath(run_dir)
print("run dir is *** :",run_dir)
if not os.path.exists(run_dir):
os.mkdir(run_dir)
run_log_file = run_dir+'/crash_gen_run.log'
crash_gen_cmds_file = os.path.join(run_dir, 'crash_gen_cmds.sh')
exit_status_logs = os.path.join(run_dir, 'crash_exit.log')
def get_path():
buildPath=''
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
projPath = selfPath[:selfPath.find("community")]
else:
projPath = selfPath[:selfPath.find("tests")]
for root, dirs, files in os.walk(projPath):
if ("taosd" in files):
rootRealPath = os.path.dirname(os.path.realpath(root))
if ("packaging" not in rootRealPath):
buildPath = root[:len(root) - len("/build/bin")]
break
return buildPath
# generate crash_gen start script randomly
def random_args(args_list):
nums_args_list = ["--max-dbs","--num-replicas","--num-dnodes","--max-steps","--num-threads",] # record int type arguments
bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
"--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"
] # record bool type arguments
strs_args_list = ["--connector-type"] # record str type arguments
args_list["--auto-start-service"]= False
args_list["--continue-on-exception"]=True
# connect_types=['native','rest','mixed'] # restful interface has change ,we should trans dbnames to connection or change sql such as "db.test"
connect_types=['native']
# args_list["--connector-type"]=connect_types[random.randint(0,2)]
args_list["--connector-type"]= connect_types[0]
args_list["--max-dbs"]= random.randint(1,10)
# dnodes = [1,3] # set single dnodes;
# args_list["--num-dnodes"]= random.sample(dnodes,1)[0]
# args_list["--num-replicas"]= random.randint(1,args_list["--num-dnodes"])
args_list["--debug"]=False
args_list["--per-thread-db-connection"]=True
args_list["--track-memory-leaks"]=False
args_list["--max-steps"]=random.randint(200,500)
threads = [16,32]
args_list["--num-threads"]=random.sample(threads,1)[0] #$ debug
# args_list["--ignore-errors"]=[] ## can add error codes for detail
args_list["--run-tdengine"]= False
args_list["--use-shadow-db"]= False
args_list["--dynamic-db-table-names"]= True
args_list["--verify-data"]= False
args_list["--record-ops"] = False
for key in bools_args_list:
set_bool_value = [True,False]
if key == "--auto-start-service" :
continue
elif key =="--run-tdengine":
continue
elif key == "--ignore-errors":
continue
elif key == "--debug":
continue
elif key == "--per-thread-db-connection":
continue
elif key == "--continue-on-exception":
continue
elif key == "--use-shadow-db":
continue
elif key =="--track-memory-leaks":
continue
elif key == "--dynamic-db-table-names":
continue
elif key == "--verify-data":
continue
elif key == "--record-ops":
continue
elif key == "--larger-data":
continue
else:
args_list[key]=set_bool_value[random.randint(0,1)]
return args_list
def limits(args_list):
if args_list["--use-shadow-db"]==True:
if args_list["--max-dbs"] > 1:
print("Cannot combine use-shadow-db with max-dbs of more than 1 ,set max-dbs=1")
args_list["--max-dbs"]=1
else:
pass
# env is start by test frame , not crash_gen instance
# elif args_list["--num-replicas"]==0:
# print(" make sure num-replicas is at least 1 ")
# args_list["--num-replicas"]=1
# elif args_list["--num-replicas"]==1:
# pass
# elif args_list["--num-replicas"]>1:
# if not args_list["--auto-start-service"]:
# print("it should be deployed by crash_gen auto-start-service for multi replicas")
# else:
# pass
return args_list
def get_auto_mix_cmds(args_list ,valgrind=valgrind_mode):
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
bools_args_list = ["--auto-start-service" , "--debug","--run-tdengine","--ignore-errors","--track-memory-leaks","--larger-data","--mix-oos-data","--dynamic-db-table-names",
"--per-thread-db-connection","--record-ops","--verify-data","--use-shadow-db","--continue-on-exception"]
arguments = ""
for k ,v in args_list.items():
if k == "--ignore-errors":
if v:
arguments+=(k+"="+str(v)+" ")
else:
arguments+=""
elif k in bools_args_list and v==True:
arguments+=(k+" ")
elif k in bools_args_list and v==False:
arguments+=""
else:
arguments+=(k+"="+str(v)+" ")
if valgrind :
crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -i 3 %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0707,0x0203 '%(crash_gen_path ,arguments)
else:
crash_gen_cmd = 'cd %s && ./crash_gen.sh -i 3 %s -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550,0x0014,0x0707,0x0203'%(crash_gen_path ,arguments)
return crash_gen_cmd
def start_taosd():
build_path = get_path()
if repo == "community":
start_path = build_path[:-5]+"community/tests/system-test/"
elif repo == "TDengine":
start_path = build_path[:-5]+"/tests/system-test/"
else:
pass
start_cmd = 'cd %s && python3 test.py -N 4 -M 1 '%(start_path)
os.system(start_cmd +">>/dev/null")
def get_cmds(args_list):
# build_path = get_path()
# if repo == "community":
# crash_gen_path = build_path[:-5]+"community/tests/pytest/"
# elif repo == "TDengine":
# crash_gen_path = build_path[:-5]+"/tests/pytest/"
# else:
# pass
# crash_gen_cmd = 'cd %s && ./crash_gen.sh --valgrind -p -t 10 -s 1000 -g 0x32c,0x32d,0x3d3,0x18,0x2501,0x369,0x388,0x061a,0x2550 '%(crash_gen_path)
crash_gen_cmd = get_auto_mix_cmds(args_list,valgrind=valgrind_mode)
return crash_gen_cmd
def run_crash_gen(crash_cmds):
# prepare env of taosd
start_taosd()
# run crash_gen and back logs
os.system('echo "%s">>%s'%(crash_cmds,crash_gen_cmds_file))
# os.system("cp %s %s"%(crash_gen_cmds_file, core_path))
os.system("%s"%(crash_cmds))
def check_status():
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
result_file = os.path.join(crash_gen_path, 'valgrind.out')
run_code = subprocess.Popen("tail -n 50 %s"%result_file, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
os.system("tail -n 50 %s>>%s"%(result_file,exit_status_logs))
core_check = subprocess.Popen('ls -l %s | grep "^-" | wc -l'%core_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if int(core_check.strip().rstrip()) > 0:
# it means core files has occured
return 3
mem_status = check_memory()
if mem_status >0:
return mem_status
if "Crash_Gen is now exiting with status code: 1" in run_code:
return 1
elif "Crash_Gen is now exiting with status code: 0" in run_code:
return 0
else:
return 2
def check_memory():
build_path = get_path()
if repo == "community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo == "TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
'''
invalid read, invalid write
'''
back_path = os.path.join(core_path,"valgrind_report")
if not os.path.exists(back_path):
os.mkdir(back_path)
stderr_file = os.path.join(crash_gen_path , "valgrind.err")
status = 0
grep_res = subprocess.Popen("grep -i 'Invalid read' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if grep_res:
# os.system("cp %s %s"%(stderr_file , back_path))
status = 4
grep_res = subprocess.Popen("grep -i 'Invalid write' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if grep_res:
# os.system("cp %s %s"%(stderr_file , back_path))
status = 4
grep_res = subprocess.Popen("grep -i 'taosMemoryMalloc' %s "%stderr_file , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")
if grep_res:
# os.system("cp %s %s"%(stderr_file , back_path))
status = 5
return status
def main():
args_list = {"--auto-start-service":False ,"--max-dbs":0,"--connector-type":"native","--debug":False,"--run-tdengine":False,"--ignore-errors":[],
"--track-memory-leaks":False , "--larger-data":False, "--mix-oos-data":False, "--dynamic-db-table-names":False,
"--per-thread-db-connection":False , "--record-ops":False , "--max-steps":100, "--num-threads":10, "--verify-data":False,"--use-shadow-db":False ,
"--continue-on-exception":False }
args = random_args(args_list)
args = limits(args)
build_path = get_path()
os.system("pip install git+https://github.com/taosdata/taos-connector-python.git >>/dev/null")
if repo =="community":
crash_gen_path = build_path[:-5]+"community/tests/pytest/"
elif repo =="TDengine":
crash_gen_path = build_path[:-5]+"/tests/pytest/"
else:
pass
if os.path.exists(crash_gen_path+"crash_gen.sh"):
print(" make sure crash_gen.sh is ready")
else:
print( " crash_gen.sh is not exists ")
sys.exit(1)
git_commit = subprocess.Popen("cd %s && git log | head -n1"%crash_gen_path, shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8")[8:16]
# crash_cmds = get_cmds()
crash_cmds = get_cmds(args)
# clean run_dir
os.system('rm -rf %s'%run_dir )
if not os.path.exists(run_dir):
os.mkdir(run_dir)
print(crash_cmds)
run_crash_gen(crash_cmds)
status = check_status()
# back_path = os.path.join(core_path,"valgrind_report")
print("exit status : ", status)
if status ==4:
print('======== crash_gen found memory bugs ========')
if status ==5:
print('======== crash_gen found memory errors ========')
if status >0:
print('======== crash_gen run failed and not exit as expected ========')
else:
print('======== crash_gen run sucess and exit as expected ========')
if status!=0 :
try:
text = f"crash_gen instance exit status of docker [ {hostname} ] is : {msg_dict[status]}\n " + f" and git commit : {git_commit}"
send_msg(get_msg(text))
except Exception as e:
print("exception:", e)
exit(status)
if __name__ == '__main__':
main()

11
tests/pytest/auto_run_regular.sh Executable file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# set LD_LIBRARY_PATH
export PATH=$PATH:/home/TDengine/debug/build/bin
export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib
ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null
ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null
ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null
# run crash_gen auto script
python3 /home/TDengine/tests/pytest/auto_crash_gen.py

11
tests/pytest/auto_run_valgrind.sh Executable file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# set LD_LIBRARY_PATH
export PATH=$PATH:/home/TDengine/debug/build/bin
export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib
ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null
ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null
ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null
# run crash_gen auto script
python3 /home/TDengine/tests/pytest/auto_crash_gen_valgrind.py

11
tests/pytest/auto_run_valgrind_cluster.sh Executable file
View File

@ -0,0 +1,11 @@
#!/bin/bash
# set LD_LIBRARY_PATH
export PATH=$PATH:/home/TDengine/debug/build/bin
export LD_LIBRARY_PATH=/home/TDengine/debug/build/lib
ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so 2>/dev/null
ln -s /home/TDengine/debug/build/lib/libtaos.so /usr/lib/libtaos.so.1 2>/dev/null
ln -s /home/TDengine/include/client/taos.h /usr/include/taos.h 2>/dev/null
# run crash_gen auto script
python3 /home/TDengine/tests/pytest/auto_crash_gen_valgrind_cluster.py

2
tests/pytest/crash_gen.sh
View File

@ -45,7 +45,7 @@ fi
# Now getting ready to execute Python
# The following is the default of our standard dev env (Ubuntu 20.04), modify/adjust at your own risk
PYTHON_EXEC=python3.8
PYTHON_EXEC=python3
# First we need to set up a path for Python to find our own TAOS modules, so that "import" can work.
# export PYTHONPATH=$(pwd)/../../src/connector/python:$(pwd)

721
tests/pytest/crash_gen/crash_gen_main.py
View File

@ -37,6 +37,7 @@ import requests
# from guppy import hpy
import gc
import taos
from taos.tmq import *
from .shared.types import TdColumns, TdTags
@ -419,10 +420,12 @@ class ThreadCoordinator:
except threading.BrokenBarrierError as err:
self._execStats.registerFailure("Aborted due to worker thread timeout")
Logging.error("\n")
Logging.error("Main loop aborted, caused by worker thread(s) time-out of {} seconds".format(
ThreadCoordinator.WORKER_THREAD_TIMEOUT))
Logging.error("TAOS related threads blocked at (stack frames top-to-bottom):")
ts = ThreadStacks()
ts.record_current_time(time.time()) # record thread exit time at current moment
ts.print(filterInternal=True)
workerTimeout = True
@ -546,7 +549,12 @@ class ThreadCoordinator:
# pick a task type for current state
db = self.pickDatabase()
taskType = db.getStateMachine().pickTaskType() # dynamic name of class
if Dice.throw(2)==1:
taskType = db.getStateMachine().pickTaskType() # dynamic name of class
else:
taskType = db.getStateMachine().balance_pickTaskType() # and an method can get balance task types
pass
return taskType(self._execStats, db) # create a task from it
def resetExecutedTasks(self):
@ -674,9 +682,15 @@ class AnyState:
# only "under normal circumstances", as we may override it with the -b option
CAN_DROP_DB = 2
CAN_CREATE_FIXED_SUPER_TABLE = 3
CAN_CREATE_STREAM = 3 # super table must exists
CAN_CREATE_TOPIC = 3 # super table must exists
CAN_CREATE_CONSUMERS = 3
CAN_DROP_FIXED_SUPER_TABLE = 4
CAN_DROP_TOPIC = 4
CAN_DROP_STREAM = 4
CAN_ADD_DATA = 5
CAN_READ_DATA = 6
CAN_DELETE_DATA = 6
def __init__(self):
self._info = self.getInfo()
@ -727,12 +741,30 @@ class AnyState:
return False
return self._info[self.CAN_DROP_FIXED_SUPER_TABLE]
def canCreateTopic(self):
return self._info[self.CAN_CREATE_TOPIC]
def canDropTopic(self):
return self._info[self.CAN_DROP_TOPIC]
def canCreateConsumers(self):
return self._info[self.CAN_CREATE_CONSUMERS]
def canCreateStreams(self):
return self._info[self.CAN_CREATE_STREAM]
def canDropStream(self):
return self._info[self.CAN_DROP_STREAM]
def canAddData(self):
return self._info[self.CAN_ADD_DATA]
def canReadData(self):
return self._info[self.CAN_READ_DATA]
def canDeleteData(self):
return self._info[self.CAN_DELETE_DATA]
def assertAtMostOneSuccess(self, tasks, cls):
sCnt = 0
for task in tasks:
@ -902,7 +934,7 @@ class StateHasData(AnyState):
): # only if we didn't create one
# we shouldn't have dropped it
self.assertNoTask(tasks, TaskDropDb)
if (not self.hasTask(tasks, TaskCreateSuperTable)
if not( self.hasTask(tasks, TaskCreateSuperTable)
): # if we didn't create the table
# we should not have a task that drops it
self.assertNoTask(tasks, TaskDropSuperTable)
@ -974,14 +1006,21 @@ class StateMechine:
# did not do this when openning connection, and this is NOT the worker
# thread, which does this on their own
dbc.use(dbName)
if not dbc.hasTables(): # no tables
Logging.debug("[STT] DB_ONLY found, between {} and {}".format(ts, time.time()))
return StateDbOnly()
# For sure we have tables, which means we must have the super table. # TODO: are we sure?
sTable = self._db.getFixedSuperTable()
if sTable.hasRegTables(dbc): # no regular tables
if sTable.hasRegTables(dbc): # no regular tables
# print("debug=====*\n"*100)
Logging.debug("[STT] SUPER_TABLE_ONLY found, between {} and {}".format(ts, time.time()))
return StateSuperTableOnly()
else: # has actual tables
Logging.debug("[STT] HAS_DATA found, between {} and {}".format(ts, time.time()))
@ -1051,6 +1090,28 @@ class StateMechine:
# Logging.debug(" (weighted random:{}/{}) ".format(i, len(taskTypes)))
return taskTypes[i]
def balance_pickTaskType(self):
# all the task types we can choose from at curent state
BasicTypes = self.getTaskTypes()
weightsTypes = BasicTypes.copy()
# this matrixs can balance the Frequency of TaskTypes
balance_TaskType_matrixs = {'TaskDropDb': 5 , 'TaskDropTopics': 20 , 'TaskDropStreams':10 , 'TaskDropStreamTables':10 ,
'TaskReadData':50 , 'TaskDropSuperTable':5 , 'TaskAlterTags':3 , 'TaskAddData':10,
'TaskDeleteData':10 , 'TaskCreateDb':10 , 'TaskCreateStream': 3, 'TaskCreateTopic' :3,
'TaskCreateConsumers':10, 'TaskCreateSuperTable': 10 } # TaskType : balance_matrixs of task
for task , weights in balance_TaskType_matrixs.items():
for basicType in BasicTypes:
if basicType.__name__ == task:
for _ in range(weights):
weightsTypes.append(basicType)
task = random.sample(weightsTypes,1)
return task[0]
# ref:
# https://eli.thegreenplace.net/2010/01/22/weighted-random-generation-in-python/
def _weighted_choice_sub(self, weights) -> int:
@ -1109,6 +1170,7 @@ class Database:
return "fs_table"
def getFixedSuperTable(self) -> TdSuperTable:
return TdSuperTable(self.getFixedSuperTableName(), self.getName())
# We aim to create a starting time tick, such that, whenever we run our test here once
@ -1342,6 +1404,19 @@ class Task():
0x2603, # Table does not exist, replaced by 2662 below
0x260d, # Tags number not matched
0x2662, # Table does not exist #TODO: what about 2603 above?
0x2600, # database not specified, SQL: show stables , database droped , and show tables
0x032C, # Object is creating
0x032D, # Object is dropping
0x03D3, # Conflict transaction not completed
0x0707, # Query not ready , it always occur at replica 3
0x707, # Query not ready
0x396, # Database in creating status
0x386, # Database in droping status
0x03E1, # failed on tmq_subscribe ,topic not exist
0x03ed , # Topic must be dropped first, SQL: drop database db_0
0x0203 , # Invalid value
0x03f0 , # Stream already exist , topic already exists
@ -1638,9 +1713,12 @@ class TaskCreateDb(StateTransitionTask):
# numReplica = Dice.throw(Settings.getConfig().max_replicas) + 1 # 1,2 ... N
numReplica = Config.getConfig().num_replicas # fixed, always
repStr = "replica {}".format(numReplica)
updatePostfix = "update 1" if Config.getConfig().verify_data else "" # allow update only when "verify data" is active
updatePostfix = "" if Config.getConfig().verify_data else "" # allow update only when "verify data" is active , 3.0 version default is update 1
vg_nums = random.randint(1,8)
cache_model = Dice.choice(['none' , 'last_row' , 'last_value' , 'both'])
buffer = random.randint(3,128)
dbName = self._db.getName()
self.execWtSql(wt, "create database {} {} {} ".format(dbName, repStr, updatePostfix ) )
self.execWtSql(wt, "create database {} {} {} vgroups {} cachemodel '{}' buffer {} ".format(dbName, repStr, updatePostfix, vg_nums, cache_model,buffer ) )
if dbName == "db_0" and Config.getConfig().use_shadow_db:
self.execWtSql(wt, "create database {} {} {} ".format("db_s", repStr, updatePostfix ) )
@ -1654,9 +1732,211 @@ class TaskDropDb(StateTransitionTask):
return state.canDropDb()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
self.execWtSql(wt, "drop database {}".format(self._db.getName()))
try:
self.queryWtSql(wt, "drop database {}".format(self._db.getName())) # drop database maybe failed ,because topic exists
except taos.error.ProgrammingError as err:
errno = Helper.convertErrno(err.errno)
if errno in [0x0203]: # drop maybe failed
pass
Logging.debug("[OPS] database dropped at {}".format(time.time()))
class TaskCreateStream(StateTransitionTask):
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canCreateStreams()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
dbname = self._db.getName()
sub_stream_name = dbname+ '_sub_stream'
sub_stream_tb_name = 'stream_tb_sub'
super_stream_name = dbname+ '_super_stream'
super_stream_tb_name = 'stream_tb_super'
if not self._db.exists(wt.getDbConn()):
Logging.debug("Skipping task, no DB yet")
return
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
# wt.execSql("use db") # should always be in place
stbname =sTable.getName()
sub_tables = sTable.getRegTables(wt.getDbConn())
aggExpr = Dice.choice([
'count(*)', 'avg(speed)', 'sum(speed)', 'stddev(speed)','min(speed)', 'max(speed)', 'first(speed)', 'last(speed)',
'apercentile(speed, 10)', 'last_row(*)', 'twa(speed)'])
stream_sql = '' # set default value
if sub_tables:
sub_tbname = sub_tables[0]
# create stream with query above sub_table
stream_sql = 'create stream {} into {}.{} as select {}, avg(speed) FROM {}.{} PARTITION BY tbname INTERVAL(5s) SLIDING(3s) '.\
format(sub_stream_name,dbname,sub_stream_tb_name ,aggExpr,dbname,sub_tbname)
else:
stream_sql = 'create stream {} into {}.{} as select {}, avg(speed) FROM {}.{} PARTITION BY tbname INTERVAL(5s) SLIDING(3s) '.\
format(super_stream_name,dbname,super_stream_tb_name,aggExpr, dbname,stbname)
self.execWtSql(wt, stream_sql)
Logging.debug("[OPS] stream is creating at {}".format(time.time()))
class TaskCreateTopic(StateTransitionTask):
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canCreateTopic()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
dbname = self._db.getName()
sub_topic_name = dbname+ '_sub_topic'
super_topic_name = dbname+ '_super_topic'
stable_topic = dbname+ '_stable_topic'
db_topic = 'database_' + dbname+ '_topics'
if not self._db.exists(wt.getDbConn()):
Logging.debug("Skipping task, no DB yet")
return
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
# wt.execSql("use db") # should always be in place
# create topic if not exists topic_ctb_column as select ts, c1, c2, c3 from stb1;
stbname =sTable.getName()
sub_tables = sTable.getRegTables(wt.getDbConn())
scalarExpr = Dice.choice([ '*','speed','color','abs(speed)','acos(speed)','asin(speed)','atan(speed)','ceil(speed)','cos(speed)','cos(speed)',
'floor(speed)','log(speed,2)','pow(speed,2)','round(speed)','sin(speed)','sqrt(speed)','char_length(color)','concat(color,color)',
'concat_ws(" ", color,color," ")','length(color)', 'lower(color)', 'ltrim(color)','substr(color , 2)','upper(color)','cast(speed as double)',
'cast(ts as bigint)'])
topic_sql = '' # set default value
if Dice.throw(3)==0: # create topic : source data from sub query
if sub_tables: # if not empty
sub_tbname = sub_tables[0]
# create topic : source data from sub query of sub stable
topic_sql = 'create topic {} as select {} FROM {}.{} ; '.format(sub_topic_name,scalarExpr,dbname,sub_tbname)
else: # create topic : source data from sub query of stable
topic_sql = 'create topic {} as select {} FROM {}.{} '.format(super_topic_name,scalarExpr, dbname,stbname)
elif Dice.throw(3)==1: # create topic : source data from super table
topic_sql = 'create topic {} AS STABLE {}.{} '.format(stable_topic,dbname,stbname)
elif Dice.throw(3)==2: # create topic : source data from whole database
topic_sql = 'create topic {} AS DATABASE {} '.format(db_topic,dbname)
else:
pass
# exec create topics
self.execWtSql(wt, "use {}".format(dbname))
self.execWtSql(wt, topic_sql)
Logging.debug("[OPS] db topic is creating at {}".format(time.time()))
class TaskDropTopics(StateTransitionTask):
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canDropTopic()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
dbname = self._db.getName()
if not self._db.exists(wt.getDbConn()):
Logging.debug("Skipping task, no DB yet")
return
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
# wt.execSql("use db") # should always be in place
tblName = sTable.getName()
if sTable.hasTopics(wt.getDbConn()):
sTable.dropTopics(wt.getDbConn(),dbname,None) # drop topics of database
sTable.dropTopics(wt.getDbConn(),dbname,tblName) # drop topics of stable
class TaskDropStreams(StateTransitionTask):
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canDropStream()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
# dbname = self._db.getName()
if not self._db.exists(wt.getDbConn()):
Logging.debug("Skipping task, no DB yet")
return
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
# wt.execSql("use db") # should always be in place
# tblName = sTable.getName()
if sTable.hasStreams(wt.getDbConn()):
sTable.dropStreams(wt.getDbConn()) # drop stream of database
class TaskDropStreamTables(StateTransitionTask):
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canDropStream()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
# dbname = self._db.getName()
if not self._db.exists(wt.getDbConn()):
Logging.debug("Skipping task, no DB yet")
return
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
wt.execSql("use db") # should always be in place
# tblName = sTable.getName()
if sTable.hasStreamTables(wt.getDbConn()):
sTable.dropStreamTables(wt.getDbConn()) # drop stream tables
class TaskCreateConsumers(StateTransitionTask):
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canCreateConsumers()
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
if Config.getConfig().connector_type == 'native':
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
# wt.execSql("use db") # should always be in place
if sTable.hasTopics(wt.getDbConn()):
sTable.createConsumer(wt.getDbConn(),random.randint(1,10))
pass
else:
print(" restful not support tmq consumers")
return
class TaskCreateSuperTable(StateTransitionTask):
@classmethod
def getEndState(cls):
@ -1673,7 +1953,7 @@ class TaskCreateSuperTable(StateTransitionTask):
sTable = self._db.getFixedSuperTable() # type: TdSuperTable
# wt.execSql("use db") # should always be in place
sTable.create(wt.getDbConn(),
{'ts': TdDataType.TIMESTAMP, 'speed': TdDataType.INT, 'color': TdDataType.BINARY16}, {
'b': TdDataType.BINARY200, 'f': TdDataType.FLOAT},
@ -1688,14 +1968,17 @@ class TdSuperTable:
def __init__(self, stName, dbName):
self._stName = stName
self._dbName = dbName
self._consumerLists = {}
self._ConsumerInsts = []
def getName(self):
return self._stName
def drop(self, dbc, skipCheck = False):
dbName = self._dbName
if self.exists(dbc) : # if myself exists
fullTableName = dbName + '.' + self._stName
fullTableName = dbName + '.' + self._stName
dbc.execute("DROP TABLE {}".format(fullTableName))
else:
if not skipCheck:
@ -1711,10 +1994,12 @@ class TdSuperTable:
dbName = self._dbName
dbc.execute("USE " + dbName)
fullTableName = dbName + '.' + self._stName
fullTableName = dbName + '.' + self._stName
if dbc.existsSuperTable(self._stName):
if dropIfExists:
dbc.execute("DROP TABLE {}".format(fullTableName))
if dropIfExists:
dbc.execute("DROP TABLE {}".format(fullTableName))
else: # error
raise CrashGenError("Cannot create super table, already exists: {}".format(self._stName))
@ -1728,12 +2013,61 @@ class TdSuperTable:
)
else:
sql += " TAGS (dummy int) "
dbc.execute(sql)
dbc.execute(sql)
def createConsumer(self, dbc,Consumer_nums):
def generateConsumer(current_topic_list):
conf = TaosTmqConf()
conf.set("group.id", "tg2")
conf.set("td.connect.user", "root")
conf.set("td.connect.pass", "taosdata")
conf.set("enable.auto.commit", "true")
def tmq_commit_cb_print(tmq, resp, offset, param=None):
print(f"commit: {resp}, tmq: {tmq}, offset: {offset}, param: {param}")
conf.set_auto_commit_cb(tmq_commit_cb_print, None)
consumer = conf.new_consumer()
topic_list = TaosTmqList()
for topic in current_topic_list:
topic_list.append(topic)
try:
consumer.subscribe(topic_list)
except TmqError as e :
pass
# consumer with random work life
time_start = time.time()
while 1:
res = consumer.poll(1000)
if time.time() - time_start >random.randint(5,50) :
break
try:
consumer.unsubscribe()
except TmqError as e :
pass
return
# mulit Consumer
current_topic_list = self.getTopicLists(dbc)
for i in range(Consumer_nums):
consumer_inst = threading.Thread(target=generateConsumer, args=(current_topic_list,))
self._ConsumerInsts.append(consumer_inst)
for ConsumerInst in self._ConsumerInsts:
ConsumerInst.start()
for ConsumerInst in self._ConsumerInsts:
ConsumerInst.join()
def getTopicLists(self, dbc: DbConn):
dbc.query("show topics ")
topics = dbc.getQueryResult()
topicLists = [v[0] for v in topics]
return topicLists
def getRegTables(self, dbc: DbConn):
dbName = self._dbName
try:
dbc.query("select TBNAME from {}.{}".format(dbName, self._stName)) # TODO: analyze result set later
dbc.query("select distinct TBNAME from {}.{}".format(dbName, self._stName)) # TODO: analyze result set later
except taos.error.ProgrammingError as err:
errno2 = Helper.convertErrno(err.errno)
Logging.debug("[=] Failed to get tables from super table: errno=0x{:X}, msg: {}".format(errno2, err))
@ -1743,7 +2077,75 @@ class TdSuperTable:
return [v[0] for v in qr] # list transformation, ref: https://stackoverflow.com/questions/643823/python-list-transformation
def hasRegTables(self, dbc: DbConn):
return dbc.query("SELECT * FROM {}.{}".format(self._dbName, self._stName)) > 0
if dbc.existsSuperTable(self._stName):
return dbc.query("SELECT * FROM {}.{}".format(self._dbName, self._stName)) > 0
else:
return False
def hasStreamTables(self,dbc: DbConn):
return dbc.query("show {}.stables like 'stream_tb%'".format(self._dbName)) > 0
def hasStreams(self,dbc: DbConn):
return dbc.query("show streams") > 0
def hasTopics(self,dbc: DbConn):
return dbc.query("show topics") > 0
def dropTopics(self,dbc: DbConn , dbname=None,stb_name=None):
dbc.query("show topics ")
topics = dbc.getQueryResult()
if dbname !=None and stb_name == None :
for topic in topics:
if dbname in topic[0] and topic[0].startswith("database"):
try:
dbc.execute('drop topic {}'.format(topic[0]))
Logging.debug("[OPS] topic {} is droping at {}".format(topic,time.time()))
except taos.error.ProgrammingError as err:
errno = Helper.convertErrno(err.errno)
if errno in [0x03EB]: # Topic subscribed cannot be dropped
pass
# for subsript in subscriptions:
else:
pass
pass
return True
elif dbname !=None and stb_name!= None:
for topic in topics:
if topic[0].startswith(self._dbName) and topic[0].endswith('topic'):
dbc.execute('drop topic {}'.format(topic[0]))
Logging.debug("[OPS] topic {} is droping at {}".format(topic,time.time()))
return True
else:
return True
pass
def dropStreams(self,dbc:DbConn):
dbc.query("show streams ")
Streams = dbc.getQueryResult()
for Stream in Streams:
if Stream[0].startswith(self._dbName):
dbc.execute('drop stream {}'.format(Stream[0]))
return not dbc.query("show streams ") > 0
def dropStreamTables(self, dbc: DbConn):
dbc.query("show {}.stables like 'stream_tb%'".format(self._dbName))
StreamTables = dbc.getQueryResult()
for StreamTable in StreamTables:
if self.dropStreams(dbc):
dbc.execute('drop table {}.{}'.format(self._dbName,StreamTable[0]))
return not dbc.query("show {}.stables like 'stream_tb%'".format(self._dbName))
def ensureRegTable(self, task: Optional[Task], dbc: DbConn, regTableName: str):
'''
@ -1838,10 +2240,46 @@ class TdSuperTable:
# Run the query against the regular table first
doAggr = (Dice.throw(2) == 0) # 1 in 2 chance
if not doAggr: # don't do aggregate query, just simple one
commonExpr = Dice.choice([
'*',
'abs(speed)',
'acos(speed)',
'asin(speed)',
'atan(speed)',
'ceil(speed)',
'cos(speed)',
'cos(speed)',
'floor(speed)',
'log(speed,2)',
'pow(speed,2)',
'round(speed)',
'sin(speed)',
'sqrt(speed)',
'char_length(color)',
'concat(color,color)',
'concat_ws(" ", color,color," ")',
'length(color)',
'lower(color)',
'ltrim(color)',
'substr(color , 2)',
'upper(color)',
'cast(speed as double)',
'cast(ts as bigint)',
# 'TO_ISO8601(color)',
# 'TO_UNIXTIMESTAMP(ts)',
'now()',
'timediff(ts,now)',
'timezone()',
'TIMETRUNCATE(ts,1s)',
'TIMEZONE()',
'TODAY()',
'distinct(color)'
]
)
ret.append(SqlQuery( # reg table
"select {} from {}.{}".format('*', self._dbName, rTbName)))
"select {} from {}.{}".format(commonExpr, self._dbName, rTbName)))
ret.append(SqlQuery( # super table
"select {} from {}.{}".format('*', self._dbName, self.getName())))
"select {} from {}.{}".format(commonExpr, self._dbName, self.getName())))
else: # Aggregate query
aggExpr = Dice.choice([
'count(*)',
@ -1857,17 +2295,34 @@ class TdSuperTable:
'top(speed, 50)', # TODO: not supported?
'bottom(speed, 50)', # TODO: not supported?
'apercentile(speed, 10)', # TODO: TD-1316
# 'last_row(speed)', # TODO: commented out per TD-3231, we should re-create
'last_row(*)', # TODO: commented out per TD-3231, we should re-create
# Transformation Functions
# 'diff(speed)', # TODO: no supported?!
'spread(speed)'
'spread(speed)',
'elapsed(ts)',
'mode(speed)',
'bottom(speed,1)',
'top(speed,1)',
'tail(speed,1)',
'unique(color)',
'csum(speed)',
'DERIVATIVE(speed,1s,1)',
'diff(speed,1)',
'irate(speed)',
'mavg(speed,3)',
'sample(speed,5)',
'STATECOUNT(speed,"LT",1)',
'STATEDURATION(speed,"LT",1)',
'twa(speed)'
]) # TODO: add more from 'top'
# if aggExpr not in ['stddev(speed)']: # STDDEV not valid for super tables?! (Done in TD-1049)
sql = "select {} from {}.{}".format(aggExpr, self._dbName, self.getName())
if Dice.throw(3) == 0: # 1 in X chance
sql = sql + ' GROUP BY color'
partion_expr = Dice.choice(['color','tbname'])
sql = sql + ' partition BY ' + partion_expr + ' order by ' + partion_expr
Progress.emit(Progress.QUERY_GROUP_BY)
# Logging.info("Executing GROUP-BY query: " + sql)
ret.append(SqlQuery(sql))
@ -1974,6 +2429,7 @@ class TaskDropSuperTable(StateTransitionTask):
isSuccess = False
Logging.debug("[DB] Acceptable error when dropping a table")
continue # try to delete next regular table
if (not tickOutput):
tickOutput = True # Print only one time
@ -1985,6 +2441,8 @@ class TaskDropSuperTable(StateTransitionTask):
# Drop the super table itself
tblName = self._db.getFixedSuperTableName()
self.execWtSql(wt, "drop table {}.{}".format(self._db.getName(), tblName))
class TaskAlterTags(StateTransitionTask):
@ -2234,6 +2692,220 @@ class TaskAddData(StateTransitionTask):
self.activeTable.discard(i) # not raising an error, unlike remove
class TaskDeleteData(StateTransitionTask):
# Track which table is being actively worked on
activeTable: Set[int] = set()
# We use these two files to record operations to DB, useful for power-off tests
fAddLogReady = None # type: Optional[io.TextIOWrapper]
fAddLogDone = None # type: Optional[io.TextIOWrapper]
@classmethod
def prepToRecordOps(cls):
if Config.getConfig().record_ops:
if (cls.fAddLogReady is None):
Logging.info(
"Recording in a file operations to be performed...")
cls.fAddLogReady = open("add_log_ready.txt", "w")
if (cls.fAddLogDone is None):
Logging.info("Recording in a file operations completed...")
cls.fAddLogDone = open("add_log_done.txt", "w")
@classmethod
def getEndState(cls):
return StateHasData()
@classmethod
def canBeginFrom(cls, state: AnyState):
return state.canDeleteData()
def _lockTableIfNeeded(self, fullTableName, extraMsg = ''):
if Config.getConfig().verify_data:
# Logging.info("Locking table: {}".format(fullTableName))
self.lockTable(fullTableName)
# Logging.info("Table locked {}: {}".format(extraMsg, fullTableName))
# print("_w" + str(nextInt % 100), end="", flush=True) # Trace what was written
else:
# Logging.info("Skipping locking table")
pass
def _unlockTableIfNeeded(self, fullTableName):
if Config.getConfig().verify_data:
# Logging.info("Unlocking table: {}".format(fullTableName))
self.unlockTable(fullTableName)
# Logging.info("Table unlocked: {}".format(fullTableName))
else:
pass
# Logging.info("Skipping unlocking table")
def _deleteData(self, db: Database, dbc, regTableName, te: TaskExecutor): # implied: NOT in batches
numRecords = self.LARGE_NUMBER_OF_RECORDS if Config.getConfig().larger_data else self.SMALL_NUMBER_OF_RECORDS
del_Records = int(numRecords/5)
if Dice.throw(2) == 0:
for j in range(del_Records): # number of records per table
intToWrite = db.getNextInt()
nextTick = db.getNextTick()
# nextColor = db.getNextColor()
if Config.getConfig().record_ops:
self.prepToRecordOps()
if self.fAddLogReady is None:
raise CrashGenError("Unexpected empty fAddLogReady")
self.fAddLogReady.write("Ready to delete {} to {}\n".format(intToWrite, regTableName))
self.fAddLogReady.flush()
os.fsync(self.fAddLogReady.fileno())
# TODO: too ugly trying to lock the table reliably, refactor...
fullTableName = db.getName() + '.' + regTableName
self._lockTableIfNeeded(fullTableName) # so that we are verify read-back. TODO: deal with exceptions before unlock
try:
sql = "delete from {} where ts = '{}' ;".format( # removed: tags ('{}', {})
fullTableName,
# ds.getFixedSuperTableName(),
# ds.getNextBinary(), ds.getNextFloat(),
nextTick)
# print(sql)
# Logging.info("Adding data: {}".format(sql))
dbc.execute(sql)
# Logging.info("Data added: {}".format(sql))
intWrote = intToWrite
# Quick hack, attach an update statement here. TODO: create an "update" task
if (not Config.getConfig().use_shadow_db) and Dice.throw(5) == 0: # 1 in N chance, plus not using shaddow DB
intToUpdate = db.getNextInt() # Updated， but should not succeed
# nextColor = db.getNextColor()
sql = "delete from {} where ts = '{}' ;".format( # "INSERt" means "update" here
fullTableName,
nextTick)
# sql = "UPDATE {} set speed={}, color='{}' WHERE ts='{}'".format(
# fullTableName, db.getNextInt(), db.getNextColor(), nextTick)
dbc.execute(sql)
intWrote = intToUpdate # We updated, seems TDengine non-cluster accepts this.
except: # Any exception at all
self._unlockTableIfNeeded(fullTableName)
raise
# Now read it back and verify, we might encounter an error if table is dropped
if Config.getConfig().verify_data: # only if command line asks for it
try:
dbc.query("SELECT * from {}.{} WHERE ts='{}'".
format(db.getName(), regTableName, nextTick))
result = dbc.getQueryResult()
if len(result)==0:
# means data has been delete
print("D1",end="") # DF means delete failed
else:
print("DF",end="") # DF means delete failed
except taos.error.ProgrammingError as err:
errno = Helper.convertErrno(err.errno)
# if errno == CrashGenError.INVALID_EMPTY_RESULT: # empty result
# print("D1",end="") # D1 means delete data success and only 1 record
if errno in [0x218, 0x362,0x2662]: # table doesn't exist
# do nothing
pass
else:
# Re-throw otherwise
raise
finally:
self._unlockTableIfNeeded(fullTableName) # Quite ugly, refactor lock/unlock
# Done with read-back verification, unlock the table now
# Successfully wrote the data into the DB, let's record it somehow
te.recordDataMark(intWrote)
else:
# delete all datas and verify datas ,expected table is empty
if Config.getConfig().record_ops:
self.prepToRecordOps()
if self.fAddLogReady is None:
raise CrashGenError("Unexpected empty fAddLogReady")
self.fAddLogReady.write("Ready to delete {} to {}\n".format(intToWrite, regTableName))
self.fAddLogReady.flush()
os.fsync(self.fAddLogReady.fileno())
# TODO: too ugly trying to lock the table reliably, refactor...
fullTableName = db.getName() + '.' + regTableName
self._lockTableIfNeeded(fullTableName) # so that we are verify read-back. TODO: deal with exceptions before unlock
try:
sql = "delete from {} ;".format( # removed: tags ('{}', {})
fullTableName)
# Logging.info("Adding data: {}".format(sql))
dbc.execute(sql)
# Logging.info("Data added: {}".format(sql))
# Quick hack, attach an update statement here. TODO: create an "update" task
if (not Config.getConfig().use_shadow_db) and Dice.throw(5) == 0: # 1 in N chance, plus not using shaddow DB
sql = "delete from {} ;".format( # "INSERt" means "update" here
fullTableName)
dbc.execute(sql)
except: # Any exception at all
self._unlockTableIfNeeded(fullTableName)
raise
# Now read it back and verify, we might encounter an error if table is dropped
if Config.getConfig().verify_data: # only if command line asks for it
try:
dbc.query("SELECT * from {}.{} WHERE ts='{}'".
format(db.getName(), regTableName, nextTick))
result = dbc.getQueryResult()
if len(result)==0:
# means data has been delete
print("DA",end="")
else:
print("DF",end="") # DF means delete failed
except taos.error.ProgrammingError as err:
errno = Helper.convertErrno(err.errno)
# if errno == CrashGenError.INVALID_EMPTY_RESULT: # empty result
# print("Da",end="") # Da means delete data success and for all datas
if errno in [0x218, 0x362,0x2662]: # table doesn't exist
# do nothing
pass
else:
# Re-throw otherwise
raise
finally:
self._unlockTableIfNeeded(fullTableName) # Quite ugly, refactor lock/unlock
# Done with read-back verification, unlock the table now
if Config.getConfig().record_ops:
if self.fAddLogDone is None:
raise CrashGenError("Unexpected empty fAddLogDone")
self.fAddLogDone.write("Wrote {} to {}\n".format(intWrote, regTableName))
self.fAddLogDone.flush()
os.fsync(self.fAddLogDone.fileno())
def _executeInternal(self, te: TaskExecutor, wt: WorkerThread):
# ds = self._dbManager # Quite DANGEROUS here, may result in multi-thread client access
db = self._db
dbc = wt.getDbConn()
numTables = self.LARGE_NUMBER_OF_TABLES if Config.getConfig().larger_data else self.SMALL_NUMBER_OF_TABLES
numRecords = self.LARGE_NUMBER_OF_RECORDS if Config.getConfig().larger_data else self.SMALL_NUMBER_OF_RECORDS
tblSeq = list(range(numTables ))
random.shuffle(tblSeq) # now we have random sequence
for i in tblSeq:
if (i in self.activeTable): # wow already active
# print("x", end="", flush=True) # concurrent insertion
Progress.emit(Progress.CONCURRENT_INSERTION)
else:
self.activeTable.add(i) # marking it active
dbName = db.getName()
sTable = db.getFixedSuperTable()
regTableName = self.getRegTableName(i) # "db.reg_table_{}".format(i)
fullTableName = dbName + '.' + regTableName
# self._lockTable(fullTableName) # "create table" below. Stop it if the table is "locked"
sTable.ensureRegTable(self, wt.getDbConn(), regTableName) # Ensure the table exists
# self._unlockTable(fullTableName)
self._deleteData(db, dbc, regTableName, te)
self.activeTable.discard(i) # not raising an error, unlike remove
class ThreadStacks: # stack info for all threads
def __init__(self):
@ -2244,6 +2916,9 @@ class ThreadStacks: # stack info for all threads
shortTid = th.native_id % 10000 #type: ignore
self._allStacks[shortTid] = stack # Was using th.native_id
def record_current_time(self,current_time):
self.current_time = current_time
def print(self, filteredEndName = None, filterInternal = False):
for shortTid, stack in self._allStacks.items(): # for each thread, stack frames top to bottom
lastFrame = stack[-1]
@ -2258,8 +2933,11 @@ class ThreadStacks: # stack info for all threads
continue # ignore
# Now print
print("\n<----- Thread Info for LWP/ID: {} (most recent call last) <-----".format(shortTid))
lastSqlForThread = DbConn.fetchSqlForThread(shortTid)
print("Last SQL statement attempted from thread {} is: {}".format(shortTid, lastSqlForThread))
last_sql_commit_time = DbConn.get_save_sql_time(shortTid)
# time_cost = DbConn.get_time_cost()
print("Last SQL statement attempted from thread {} ({:.4f} sec ago) is: {}".format(shortTid, self.current_time-last_sql_commit_time ,lastSqlForThread))
stackFrame = 0
for frame in stack: # was using: reversed(stack)
# print(frame)
@ -2268,6 +2946,8 @@ class ThreadStacks: # stack info for all threads
print(" {}".format(frame.line))
stackFrame += 1
print("-----> End of Thread Info ----->\n")
if self.current_time-last_sql_commit_time >100: # dead lock occured
print("maybe dead locked of thread {} ".format(shortTid))
class ClientManager:
def __init__(self):
@ -2631,4 +3311,3 @@ class Container():
return
self._verifyValidProperty(name)
self._cargo[name] = value

79
tests/pytest/crash_gen/shared/db.py
View File

@ -26,10 +26,13 @@ class DbConn:
TYPE_NATIVE = "native-c"
TYPE_REST = "rest-api"
TYPE_INVALID = "invalid"
# class variables
lastSqlFromThreads : dict[int, str] = {} # stored by thread id, obtained from threading.current_thread().ident%10000
spendThreads : dict[int, float] = {} # stored by thread id, obtained from threading.current_thread().ident%10000
current_time : dict[int, float] = {} # save current time
@classmethod
def saveSqlForCurrentThread(cls, sql: str):
'''
@ -37,15 +40,56 @@ class DbConn:
run into a dead-lock situation, we can pick out the deadlocked thread, and use
that information to find what what SQL statement is stuck.
'''
th = threading.current_thread()
shortTid = th.native_id % 10000 #type: ignore
cls.lastSqlFromThreads[shortTid] = sql # Save this for later
cls.record_save_sql_time()
@classmethod
def fetchSqlForThread(cls, shortTid : int) -> str :
def fetchSqlForThread(cls, shortTid : int) -> str :
print("=======================")
if shortTid not in cls.lastSqlFromThreads:
raise CrashGenError("No last-attempted-SQL found for thread id: {}".format(shortTid))
return cls.lastSqlFromThreads[shortTid]
return cls.lastSqlFromThreads[shortTid]
@classmethod
def get_save_sql_time(cls, shortTid : int):
'''
Let us save the last SQL statement on a per-thread basis, so that when later we
run into a dead-lock situation, we can pick out the deadlocked thread, and use
that information to find what what SQL statement is stuck.
'''
return cls.current_time[shortTid]
@classmethod
def record_save_sql_time(cls):
'''
Let us save the last SQL statement on a per-thread basis, so that when later we
run into a dead-lock situation, we can pick out the deadlocked thread, and use
that information to find what what SQL statement is stuck.
'''
th = threading.current_thread()
shortTid = th.native_id % 10000 #type: ignore
cls.current_time[shortTid] = float(time.time()) # Save this for later
@classmethod
def sql_exec_spend(cls, cost: float):
'''
Let us save the last SQL statement on a per-thread basis, so that when later we
run into a dead-lock situation, we can pick out the deadlocked thread, and use
that information to find what what SQL statement is stuck.
'''
th = threading.current_thread()
shortTid = th.native_id % 10000 #type: ignore
cls.spendThreads[shortTid] = cost # Save this for later
@classmethod
def get_time_cost(cls) ->float:
th = threading.current_thread()
shortTid = th.native_id % 10000 #type: ignore
return cls.spendThreads.get(shortTid)
@classmethod
def create(cls, connType, dbTarget):
@ -61,6 +105,7 @@ class DbConn:
def createNative(cls, dbTarget) -> DbConn:
return cls.create(cls.TYPE_NATIVE, dbTarget)
@classmethod
def createRest(cls, dbTarget) -> DbConn:
return cls.create(cls.TYPE_REST, dbTarget)
@ -75,6 +120,7 @@ class DbConn:
return "[DbConn: type={}, target={}]".format(self._type, self._dbTarget)
def getLastSql(self):
return self._lastSql
def open(self):
@ -184,13 +230,19 @@ class DbConnRest(DbConn):
def _doSql(self, sql):
self._lastSql = sql # remember this, last SQL attempted
self.saveSqlForCurrentThread(sql) # Save in global structure too. #TODO: combine with above
try:
time_cost = -1
time_start = time.time()
try:
r = requests.post(self._url,
data = sql,
auth = HTTPBasicAuth('root', 'taosdata'))
auth = HTTPBasicAuth('root', 'taosdata'))
except:
print("REST API Failure (TODO: more info here)")
self.sql_exec_spend(-2)
raise
finally:
time_cost = time.time()- time_start
self.sql_exec_spend(time_cost)
rj = r.json()
# Sanity check for the "Json Result"
if ('status' not in rj):
@ -223,6 +275,8 @@ class DbConnRest(DbConn):
"[SQL-REST] Execution Result, nRows = {}, SQL = {}".format(nRows, sql))
return nRows
def query(self, sql): # return rows affected
return self.execute(sql)
@ -336,6 +390,7 @@ class MyTDSql:
raise
return self.affectedRows
class DbTarget:
def __init__(self, cfgPath, hostAddr, port):
self.cfgPath = cfgPath
@ -355,6 +410,7 @@ class DbConnNative(DbConn):
# _connInfoDisplayed = False # TODO: find another way to display this
totalConnections = 0 # Not private
totalRequests = 0
time_cost = -1
def __init__(self, dbTarget):
super().__init__(dbTarget)
@ -413,8 +469,18 @@ class DbConnNative(DbConn):
"Cannot exec SQL unless db connection is open", CrashGenError.DB_CONNECTION_NOT_OPEN)
Logging.debug("[SQL] Executing SQL: {}".format(sql))
self._lastSql = sql
time_cost = -1
nRows = 0
time_start = time.time()
self.saveSqlForCurrentThread(sql) # Save in global structure too. #TODO: combine with above
nRows = self._tdSql.execute(sql)
try:
nRows= self._tdSql.execute(sql)
except Exception as e:
self.sql_exec_spend(-2)
finally:
time_cost = time.time() - time_start
self.sql_exec_spend(time_cost)
cls = self.__class__
cls.totalRequests += 1
Logging.debug(
@ -494,4 +560,3 @@ class DbManager():
self._dbConn.close()
self._dbConn = None
Logging.debug("DbManager closed DB connection...")

31
tests/pytest/docker_exec_service.sh Normal file
View File

@ -0,0 +1,31 @@
#!/bin/bash
mkdir -p /data/wz/crash_gen_logs/
logdir='/data/wz/crash_gen_logs/'
date_tag=`date +%Y%m%d-%H%M%S`
hostname='vm_valgrind_'
for i in {1..50}
do
echo $i
# create docker and start crash_gen
log_dir=${logdir}${hostname}${date_tag}_${i}
docker run -d --hostname=${hostname}${date_tag}_${i} --name ${hostname}${date_tag}_${i} --privileged -v ${log_dir}:/corefile/ -- crash_gen:v1.0 sleep 99999999999999
echo create docker ${hostname}${date_tag}_${i}
docker exec -d ${hostname}${date_tag}_${i} sh -c 'rm -rf /home/taos-connector-python'
docker cp /data/wz/TDengine ${hostname}${date_tag}_${i}:/home/TDengine
docker cp /data/wz/taos-connector-python ${hostname}${date_tag}_${i}:/home/taos-connector-python
echo copy TDengine in container done!
docker exec ${hostname}${date_tag}_${i} sh -c 'sh /home/TDengine/tests/pytest/auto_run_valgrind.sh '
if [ $? -eq 0 ]
then
echo crash_gen exit as expect , run success
# # clear docker which success
docker stop ${hostname}${date_tag}_${i}
docker rm -f ${hostname}${date_tag}_${i}
else
docker stop ${hostname}${date_tag}_${i}
echo crash_gen exit error , run failed
fi
done

77
tests/pytest/util/dnodes.py
View File

@ -116,6 +116,7 @@ class TDDnode:
self.deployed = 0
self.testCluster = False
self.valgrind = 0
self.asan = False
self.remoteIP = ""
self.cfgDict = {
"monitor": "0",
@ -158,6 +159,15 @@ class TDDnode:
def setValgrind(self, value):
self.valgrind = value
def setAsan(self, value):
self.asan = value
if value:
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
self.execPath = os.path.abspath(self.path + "/community/tests/script/sh/exec.sh")
else:
self.execPath = os.path.abspath(self.path + "/tests/script/sh/exec.sh")
def getDataSize(self):
totalSize = 0
@ -383,8 +393,14 @@ class TDDnode:
cmd = "mintty -h never %s -c %s" % (
binPath, self.cfgDir)
else:
cmd = "nohup %s -c %s > /dev/null 2>&1 & " % (
binPath, self.cfgDir)
if self.asan:
asanDir = "%s/sim/asan/dnode%d.asan" % (
self.path, self.index)
cmd = "nohup %s -c %s > /dev/null 2> %s & " % (
binPath, self.cfgDir, asanDir)
else:
cmd = "nohup %s -c %s > /dev/null 2>&1 & " % (
binPath, self.cfgDir)
else:
valgrindCmdline = "valgrind --log-file=\"%s/../log/valgrind.log\" --tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all -v --workaround-gcc296-bugs=yes"%self.cfgDir
@ -444,8 +460,14 @@ class TDDnode:
tdLog.exit("dnode:%d is not deployed" % (self.index))
if self.valgrind == 0:
cmd = "nohup %s -c %s > /dev/null 2>&1 & " % (
binPath, self.cfgDir)
if self.asan:
asanDir = "%s/sim/asan/dnode%d.asan" % (
self.path, self.index)
cmd = "nohup %s -c %s > /dev/null 2> %s & " % (
binPath, self.cfgDir, asanDir)
else:
cmd = "nohup %s -c %s > /dev/null 2>&1 & " % (
binPath, self.cfgDir)
else:
valgrindCmdline = "valgrind --log-file=\"%s/../log/valgrind.log\" --tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all -v --workaround-gcc296-bugs=yes"%self.cfgDir
@ -464,6 +486,12 @@ class TDDnode:
tdLog.debug("dnode:%d is running with %s " % (self.index, cmd))
def stop(self):
if self.asan:
stopCmd = "%s -s stop -n dnode%d" % (self.execPath, self.index)
tdLog.info("execute script: " + stopCmd)
os.system(stopCmd)
return
if (not self.remoteIP == ""):
self.remoteExec(self.cfgDict, "tdDnodes.dnodes[%d].running=1\ntdDnodes.dnodes[%d].stop()"%(self.index-1,self.index-1))
tdLog.info("stop dnode%d"%self.index)
@ -501,6 +529,12 @@ class TDDnode:
def stoptaosd(self):
if self.asan:
stopCmd = "%s -s stop -n dnode%d" % (self.execPath, self.index)
tdLog.info("execute script: " + stopCmd)
os.system(stopCmd)
return
if (not self.remoteIP == ""):
self.remoteExec(self.cfgDict, "tdDnodes.dnodes[%d].running=1\ntdDnodes.dnodes[%d].stop()"%(self.index-1,self.index-1))
tdLog.info("stop dnode%d"%self.index)
@ -534,6 +568,13 @@ class TDDnode:
tdLog.debug("dnode:%d is stopped by kill -INT" % (self.index))
def forcestop(self):
if self.asan:
stopCmd = "%s -s stop -n dnode%d -x SIGKILL" + \
(self.execPath, self.index)
tdLog.info("execute script: " + stopCmd)
os.system(stopCmd)
return
if (not self.remoteIP == ""):
self.remoteExec(self.cfgDict, "tdDnodes.dnodes[%d].running=1\ntdDnodes.dnodes[%d].forcestop()"%(self.index-1,self.index-1))
return
@ -606,6 +647,7 @@ class TDDnodes:
self.simDeployed = False
self.testCluster = False
self.valgrind = 0
self.asan = False
self.killValgrind = 1
def init(self, path, remoteIP = ""):
@ -629,6 +671,18 @@ class TDDnodes:
def setValgrind(self, value):
self.valgrind = value
def setAsan(self, value):
self.asan = value
if value:
selfPath = os.path.dirname(os.path.realpath(__file__))
if ("community" in selfPath):
self.stopDnodesPath = os.path.abspath(self.path + "/community/tests/script/sh/stop_dnodes.sh")
self.stopDnodesSigintPath = os.path.abspath(self.path + "/community/tests/script/sh/sigint_stop_dnodes.sh")
else:
self.stopDnodesPath = os.path.abspath(self.path + "/tests/script/sh/stop_dnodes.sh")
self.stopDnodesSigintPath = os.path.abspath(self.path + "/tests/script/sh/sigint_stop_dnodes.sh")
tdLog.info("run in address sanitizer mode")
def setKillValgrind(self, value):
self.killValgrind = value
@ -642,6 +696,7 @@ class TDDnodes:
self.check(index)
self.dnodes[index - 1].setTestCluster(self.testCluster)
self.dnodes[index - 1].setValgrind(self.valgrind)
self.dnodes[index - 1].setAsan(self.asan)
self.dnodes[index - 1].deploy(updatecfgDict)
def cfg(self, index, option, value):
@ -692,8 +747,22 @@ class TDDnodes:
if index < 1 or index > 10:
tdLog.exit("index:%d should on a scale of [1, 10]" % (index))
def StopAllSigint(self):
tdLog.info("stop all dnodes sigint")
if self.asan:
tdLog.info("execute script: %s" % self.stopDnodesSigintPath)
os.system(self.stopDnodesSigintPath)
tdLog.info("execute finished")
return
def stopAll(self):
tdLog.info("stop all dnodes")
if self.asan:
tdLog.info("execute script: %s" % self.stopDnodesPath)
os.system(self.stopDnodesPath)
tdLog.info("execute finished")
return
if (not self.dnodes[0].remoteIP == ""):
self.dnodes[0].remoteExec(self.dnodes[0].cfgDict, "for i in range(len(tdDnodes.dnodes)):\n tdDnodes.dnodes[i].running=1\ntdDnodes.stopAll()")
return

4
tests/pytest/util/log.py
View File

@ -36,10 +36,10 @@ class TDLog:
printf("\033[1;32m%s %s\033[0m" % (datetime.datetime.now(), info))
def notice(self, err):
printf("\033[1;33m%s %s\033[0m" % (datetime.datetime.now(), err))
print("\033[1;33m%s %s\033[0m" % (datetime.datetime.now(), err))
def exit(self, err):
printf("\033[1;31m%s %s\033[0m" % (datetime.datetime.now(), err))
print("\033[1;31m%s %s\033[0m" % (datetime.datetime.now(), err))
sys.exit(1)
def printNoPrefix(self, info):

19
tests/script/sh/checkAsan.sh
View File

@ -3,6 +3,7 @@
set +e
#set -x
unset LD_PRELOAD
SCRIPT_DIR=`dirname $0`
cd $SCRIPT_DIR/../
SCRIPT_DIR=`pwd`
@ -15,25 +16,37 @@ else
fi
TAOS_DIR=`pwd`
LOG_DIR=$TAOS_DIR/sim/tsim/asan
LOG_DIR=$TAOS_DIR/sim/asan
error_num=`cat ${LOG_DIR}/*.asan | grep "ERROR" | wc -l`
memory_leak=`cat ${LOG_DIR}/*.asan | grep "Direct leak" | wc -l`
indirect_leak=`cat ${LOG_DIR}/*.asan | grep "Indirect leak" | wc -l`
runtime_error=`cat ${LOG_DIR}/*.asan | grep "runtime error" | grep -v "trees.c:873" | wc -l`
# ignore
# /root/TDengine/source/libs/scalar/src/sclfunc.c:735:11: runtime error: 4.75783e+11 is outside the range of representable values of type 'signed char'
# /root/TDengine/source/libs/scalar/src/sclfunc.c:790:11: runtime error: 3.4e+38 is outside the range of representable values of type 'long int'
# /root/TDengine/source/libs/scalar/src/sclfunc.c:772:11: runtime error: 3.52344e+09 is outside the range of representable values of type 'int'
# /root/TDengine/source/libs/scalar/src/sclfunc.c:753:11: runtime error: 4.75783e+11 is outside the range of representable values of type 'short int'
runtime_error=`cat ${LOG_DIR}/*.asan | grep "runtime error" | grep -v "trees.c:873" | grep -v "sclfunc.c.*outside the range of representable values of type" | wc -l`
python_error=`cat ${LOG_DIR}/*.info | grep -w "stack" | wc -l`
echo -e "\033[44;32;1m"asan error_num: $error_num"\033[0m"
echo -e "\033[44;32;1m"asan memory_leak: $memory_leak"\033[0m"
echo -e "\033[44;32;1m"asan indirect_leak: $indirect_leak"\033[0m"
echo -e "\033[44;32;1m"asan runtime error: $runtime_error"\033[0m"
echo -e "\033[44;32;1m"asan python error: $python_error"\033[0m"
let "errors=$error_num+$memory_leak+$indirect_leak+$runtime_error"
let "errors=$error_num+$memory_leak+$indirect_leak+$runtime_error+$python_error"
if [ $errors -eq 0 ]; then
echo -e "\033[44;32;1m"no asan errors"\033[0m"
exit 0
else
echo -e "\033[44;31;1m"asan total errors: $errors"\033[0m"
if [ $python_error -ne 0 ]; then
cat ${LOG_DIR}/*.info
fi
cat ${LOG_DIR}/*.asan
exit 1
fi

3
tests/script/sh/exec.sh
View File

@ -11,6 +11,7 @@
set +e
#set -x
unset LD_PRELOAD
UNAME_BIN=`which uname`
OS_TYPE=`$UNAME_BIN`
@ -80,7 +81,7 @@ LOG_DIR=$NODE_DIR/log
DATA_DIR=$NODE_DIR/data
MGMT_DIR=$NODE_DIR/data/mgmt
TSDB_DIR=$NODE_DIR/data/tsdb
ASAN_DIR=$SIM_DIR/tsim/asan
ASAN_DIR=$SIM_DIR/asan
TAOS_CFG=$NODE_DIR/cfg/taos.cfg
echo ------------ $EXEC_OPTON $NODE_NAME

16
tests/script/sh/sigint_stop_dnodes.sh Executable file
View File

@ -0,0 +1,16 @@
#!/bin/sh
set +e
#set -x
unset LD_PRELOAD
UNAME_BIN=`which uname`
OS_TYPE=`$UNAME_BIN`
PID=`ps -ef|grep -w taosd | grep -v grep | awk '{print $2}'`
echo "Killing taosd processes " $PID
while [ -n "$PID" ]; do
#echo "Killing taosd processes " $PID
kill $PID
PID=`ps -ef|grep -w taosd | grep -v grep | awk '{print $2}'`
done

17
tests/script/sh/stop_dnodes.sh
View File

@ -1,5 +1,9 @@
#!/bin/sh
set +e
#set -x
unset LD_PRELOAD
UNAME_BIN=`which uname`
OS_TYPE=`$UNAME_BIN`
@ -22,16 +26,3 @@ while [ -n "$PID" ]; do
fi
PID=`ps -ef|grep -w taosd | grep -v grep | awk '{print $2}'`
done
PID=`ps -ef|grep -w tarbitrator | grep -v grep | awk '{print $2}'`
while [ -n "$PID" ]; do
echo kill -9 $PID
pkill -9 tarbitrator
if [ "$OS_TYPE" != "Darwin" ]; then
fuser -k -n tcp 6040
else
lsof -nti:6040 | xargs kill -9
fi
PID=`ps -ef|grep -w tarbitrator | grep -v grep | awk '{print $2}'`
done

11
tests/script/test.sh
View File

@ -66,16 +66,13 @@ else
fi
declare -x BUILD_DIR=$TOP_DIR/$BIN_DIR
declare -x SIM_DIR=$TOP_DIR/sim
PROGRAM=$BUILD_DIR/build/bin/tsim
PRG_DIR=$SIM_DIR/tsim
CFG_DIR=$PRG_DIR/cfg
LOG_DIR=$PRG_DIR/log
DATA_DIR=$PRG_DIR/data
ASAN_DIR=$PRG_DIR/asan
ASAN_DIR=$SIM_DIR/asan
chmod -R 777 $PRG_DIR
echo "------------------------------------------------------------------------"
@ -141,11 +138,15 @@ if [ -n "$FILE_NAME" ]; then
echo "AsanDir:" $ASAN_DIR/tsim.asan
eval $PROGRAM -c $CFG_DIR -f $FILE_NAME 2> $ASAN_DIR/tsim.asan
result=$?
echo "Execute result: " $result
echo "Execute result:" $result
if [ $result -eq 0 ]; then
$CODE_DIR/sh/sigint_stop_dnodes.sh
$CODE_DIR/sh/checkAsan.sh
else
echo "TSIM has asan errors"
sleep 1
$CODE_DIR/sh/checkAsan.sh
exit 1
fi
fi

154
tests/script/tsim/sma/sma_leak.sim Normal file
View File

@ -0,0 +1,154 @@
system sh/stop_dnodes.sh
system sh/deploy.sh -n dnode1 -i 1
system sh/deploy.sh -n dnode2 -i 2
system sh/deploy.sh -n dnode3 -i 3
system sh/cfg.sh -n dnode1 -c supportVnodes -v 0
system sh/cfg.sh -n dnode2 -c supportVnodes -v 4
system sh/cfg.sh -n dnode3 -c supportVnodes -v 4
print ========== step1
system sh/exec.sh -n dnode1 -s start
sql connect
print ========== step2
sql create dnode $hostname port 7200
sql create dnode $hostname port 7300
system sh/exec.sh -n dnode2 -s start
system sh/exec.sh -n dnode3 -s start
$x = 0
step2:
$x = $x + 1
sleep 1000
if $x == 10 then
print ====> dnode not ready!
return -1
endi
sql select * from information_schema.ins_dnodes
print ===> $data00 $data01 $data02 $data03 $data04 $data05
print ===> $data10 $data11 $data12 $data13 $data14 $data15
if $rows != 3 then
return -1
endi
if $data(1)[4] != ready then
goto step2
endi
if $data(2)[4] != ready then
goto step2
endi
if $data(3)[4] != ready then
goto step2
endi
print ========== step3
sql create database d1 vgroups 1
sql use d1;
print --> create stb
sql create table if not exists stb (ts timestamp, c1 int, c2 float, c3 double) tags (t1 int unsigned);
print --> create sma
sql create sma index sma_index_name1 on stb function(max(c1),max(c2),min(c1)) interval(6m,10s) sliding(6m);
return
print --> show sma
sql show indexes from stb from d1;
if $rows != 1 then
return -1
endi
if $data[0][0] != sma_index_name1 then
return -1
endi
if $data[0][1] != d1 then
return -1
endi
if $data[0][2] != stb then
return -1
endi
print --> drop stb
sql drop table stb;
print ========== step4 repeat
print --> create stb
sql create table if not exists stb (ts timestamp, c1 int, c2 float, c3 double) tags (t1 int unsigned);
print --> create sma
sql create sma index sma_index_name1 on stb function(max(c1),max(c2),min(c1)) interval(6m,10s) sliding(6m);
print --> show sma
sql show indexes from stb from d1;
if $rows != 1 then
return -1
endi
if $data[0][0] != sma_index_name1 then
return -1
endi
if $data[0][1] != d1 then
return -1
endi
if $data[0][2] != stb then
return -1
endi
print --> drop stb
sql drop table stb;
print ========== step5
sql drop database if exists db;
sql create database db duration 300;
sql use db;
sql create table stb1(ts timestamp, c_int int, c_bint bigint, c_sint smallint, c_tint tinyint, c_float float, c_double double, c_bool bool, c_binary binary(16), c_nchar nchar(32), c_ts timestamp, c_tint_un tinyint unsigned, c_sint_un smallint unsigned, c_int_un int unsigned, c_bint_un bigint unsigned) tags (t_int int);
sql CREATE SMA INDEX sma_index_1 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) sliding(6m) watermark 5s;
print ========== step6 repeat
sql drop database if exists db;
sql create database db duration 300;
sql use db;
sql create table stb1(ts timestamp, c_int int, c_bint bigint ) tags (t_int int);
sql CREATE SMA INDEX sma_index_1 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) sliding(6m) watermark 5s;
print ========== step7
sql drop database if exists db;
sql create database db duration 300;
sql use db;
sql create table stb1(ts timestamp, c_int int, c_bint bigint, c_sint smallint, c_tint tinyint,c_float float, c_double double, c_bool bool,c_binary binary(16), c_nchar nchar(32), c_ts timestamp,c_tint_un tinyint unsigned, c_sint_un smallint unsigned,c_int_un int unsigned, c_bint_un bigint unsigned) tags (t_int int);
sql create table ct1 using stb1 tags ( 1 );
sql create table ct2 using stb1 tags ( 2 );
sql create table ct3 using stb1 tags ( 3 );
sql create table ct4 using stb1 tags ( 4 );
sql CREATE SMA INDEX sma_index_1 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) sliding(6m) watermark 5s;
sql CREATE SMA INDEX sma_index_2 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) sliding(6m) max_delay 6m;
sql CREATE SMA INDEX sma_index_3 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) watermark 5s max_delay 6m;
sql DROP INDEX sma_index_1 ;
sql DROP INDEX sma_index_2 ;
sql DROP INDEX sma_index_3 ;
print ========== step8
sql drop database if exists db;
sql create database db duration 300;
sql use db;
sql create table stb1(ts timestamp, c_int int, c_bint bigint, c_sint smallint, c_tint tinyint,c_float float, c_double double, c_bool bool,c_binary binary(16), c_nchar nchar(32), c_ts timestamp,c_tint_un tinyint unsigned, c_sint_un smallint unsigned,c_int_un int unsigned, c_bint_un bigint unsigned) tags (t_int int);
sql create table ct1 using stb1 tags ( 1 );
sql create table ct2 using stb1 tags ( 2 );
sql create table ct3 using stb1 tags ( 3 );
sql create table ct4 using stb1 tags ( 4 );
sql CREATE SMA INDEX sma_index_1 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) sliding(6m) watermark 5s;
sql CREATE SMA INDEX sma_index_2 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) sliding(6m) max_delay 6m;
sql CREATE SMA INDEX sma_index_3 ON stb1 function(min(c_int), max(c_int)) interval(6m, 10s) watermark 5s max_delay 6m;
sql DROP INDEX sma_index_1 ;
sql DROP INDEX sma_index_2 ;
sql DROP INDEX sma_index_3 ;
system sh/exec.sh -n dnode1 -s stop -x SIGINT
system sh/exec.sh -n dnode2 -s stop -x SIGINT
system sh/exec.sh -n dnode3 -s stop -x SIGINT
system sh/exec.sh -n dnode4 -s stop -x SIGINT

22
tests/system-test/2-query/abs.py
View File

@ -204,18 +204,12 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(abs_query)
for row_index, row in enumerate(abs_result):
for col_index, elem in enumerate(row):
if auto_result[row_index][col_index] != elem:
check_status = False
if not check_status:
tdLog.notice(
"abs function value has not as expected , sql is \"%s\" " % abs_query)
sys.exit(1)
else:
tdLog.info(
"abs value check pass , it work as expected ,sql is \"%s\" " % abs_query)
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self):
dbname = "testdb"
@ -466,19 +460,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto(f"select c1, c2, c3 , c4, c5 ,c6 from {dbname}.sub1_bound ",
f"select abs(c1), abs(c2) ,abs(c3), abs(c4), abs(c5) ,abs(c6) from {dbname}.sub1_bound")

8
tests/system-test/2-query/and_or_for_byte.py
View File

@ -426,19 +426,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_function("&", False , f"{dbname}.sub1_bound" ,"c1","c2","c3","c4","c5","c6" )
self.check_function("&", False , f"{dbname}.sub1_bound","abs(c1)","abs(c2)","abs(c3)","abs(c4)","abs(c5)","abs(c6)" )

25
tests/system-test/2-query/arccos.py
View File

@ -86,21 +86,12 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(pow_query)
for row_index , row in enumerate(pow_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] == None and not (auto_result[row_index][col_index] == None and elem == None):
check_status = False
elif auto_result[row_index][col_index] != None and (auto_result[row_index][col_index] - elem > 0.00000001):
check_status = False
else:
pass
if not check_status:
tdLog.notice("acos function value has not as expected , sql is \"%s\" "%pow_query )
sys.exit(1)
else:
tdLog.info("acos value check pass , it work as expected ,sql is \"%s\" "%pow_query )
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
@ -414,19 +405,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_acos( f"select abs(c1), abs(c2), abs(c3) , abs(c4), abs(c5) from {dbname}.sub1_bound ", f"select acos(abs(c1)), acos(abs(c2)) ,acos(abs(c3)), acos(abs(c4)), acos(abs(c5)) from {dbname}.sub1_bound")

26
tests/system-test/2-query/arcsin.py
View File

@ -86,21 +86,13 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(pow_query)
for row_index , row in enumerate(pow_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] == None and not (auto_result[row_index][col_index] == None and elem == None):
check_status = False
elif auto_result[row_index][col_index] != None and (auto_result[row_index][col_index] - elem > 0.00000001):
check_status = False
else:
pass
if not check_status:
tdLog.notice("asin function value has not as expected , sql is \"%s\" "%pow_query )
sys.exit(1)
else:
tdLog.info("asin value check pass , it work as expected ,sql is \"%s\" "%pow_query )
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
@ -414,19 +406,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_asin( f"select abs(c1), abs(c2), abs(c3) , abs(c4), abs(c5) from {dbname}.sub1_bound ", f"select asin(abs(c1)), asin(abs(c2)) ,asin(abs(c3)), asin(abs(c4)), asin(abs(c5)) from {dbname}.sub1_bound")

24
tests/system-test/2-query/arctan.py
View File

@ -84,22 +84,12 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(pow_query)
for row_index , row in enumerate(pow_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] == None and elem:
check_status = False
elif auto_result[row_index][col_index] != None and (auto_result[row_index][col_index] - elem > 0.00000001):
check_status = False
else:
pass
if not check_status:
tdLog.notice("atan function value has not as expected , sql is \"%s\" "%pow_query )
sys.exit(1)
else:
tdLog.info("atan value check pass , it work as expected ,sql is \"%s\" "%pow_query )
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
f"select atan from {dbname}.t1",
@ -412,19 +402,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_atan( f"select abs(c1), abs(c2), abs(c3) , abs(c4), abs(c5) from {dbname}.sub1_bound ", f"select atan(abs(c1)), atan(abs(c2)) ,atan(abs(c3)), atan(abs(c4)), atan(abs(c5)) from {dbname}.sub1_bound")

24
tests/system-test/2-query/avg.py
View File

@ -114,16 +114,10 @@ class TDTestCase:
avg_result = tdSql.getResult(origin_query)
origin_result = tdSql.getResult(check_query)
check_status = True
tdSql.query(origin_query)
for row_index , row in enumerate(avg_result):
for col_index , elem in enumerate(row):
if avg_result[row_index][col_index] != origin_result[row_index][col_index]:
check_status = False
if not check_status:
tdLog.notice("avg function value has not as expected , sql is \"%s\" "%origin_query )
sys.exit(1)
else:
tdLog.info("avg value check pass , it work as expected ,sql is \"%s\" "%check_query )
tdSql.checkData(row_index,col_index,origin_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
@ -378,33 +372,33 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483645, 9223372036854775805, 32765, 125, 3.40E+37, 1.7e+307, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483645, 9223372036854775805, 32765, 125, 3.40E+37, 1.7e+307, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483644, 9223372036854775804, 32764, 124, 3.40E+37, 1.7e+307, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483644, 9223372036854775804, 32764, 124, 3.40E+37, 1.7e+307, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+20s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
#self.check_avg(f"select avg(c1), avg(c2), avg(c3) , avg(c4), avg(c5) ,avg(c6) from {dbname}.sub1_bound " , f" select sum(c1)/count(c1), sum(c2)/count(c2) ,sum(c3)/count(c3), sum(c4)/count(c4), sum(c5)/count(c5) ,sum(c6)/count(c6) from {dbname}.sub1_bound ")

21
tests/system-test/2-query/ceil.py
View File

@ -85,16 +85,11 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(ceil_query)
for row_index , row in enumerate(ceil_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] != elem:
check_status = False
if not check_status:
tdLog.notice("ceil function value has not as expected , sql is \"%s\" "%ceil_query )
sys.exit(1)
else:
tdLog.info("ceil value check pass , it work as expected ,sql is \"%s\" "%ceil_query )
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
@ -377,10 +372,10 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
@ -388,15 +383,15 @@ class TDTestCase:
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto( f"select c1, c2, c3 , c4, c5 ,c6 from {dbname}.sub1_bound ", f"select ceil(c1), ceil(c2) ,ceil(c3), ceil(c4), ceil(c5) ,ceil(c6) from {dbname}.sub1_bound")
self.check_result_auto( f"select c1, c2, c3 , c3, c2 ,c1 from {dbname}.sub1_bound ", f"select ceil(c1), ceil(c2) ,ceil(c3), ceil(c3), ceil(c2) ,ceil(c1) from {dbname}.sub1_bound")

26
tests/system-test/2-query/cos.py
View File

@ -84,26 +84,10 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
print("========",pow_query, origin_query )
tdSql.query(pow_query)
for row_index , row in enumerate(pow_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] == None and elem:
check_status = False
elif auto_result[row_index][col_index] != None and ((auto_result[row_index][col_index] != elem) and (str(auto_result[row_index][col_index])[:6] != str(elem)[:6] )):
# elif auto_result[row_index][col_index] != None and (abs(auto_result[row_index][col_index] - elem) > 0.000001):
print("=====")
print(row_index, col_index)
print(auto_result[row_index][col_index], elem, origin_result[row_index][col_index])
check_status = False
else:
pass
if not check_status:
tdLog.notice("cos function value has not as expected , sql is \"%s\" "%pow_query )
sys.exit(1)
else:
tdLog.info("cos value check pass , it work as expected ,sql is \"%s\" "%pow_query )
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
@ -413,16 +397,16 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
# self.check_result_auto_cos( f"select abs(c1), abs(c2), abs(c3) , abs(c4), abs(c5) from {dbname}.sub1_bound ", f"select cos(abs(c1)), cos(abs(c2)) ,cos(abs(c3)), cos(abs(c4)), cos(abs(c5)) from {dbname}.sub1_bound")

21
tests/system-test/2-query/floor.py
View File

@ -85,16 +85,11 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(floor_query)
for row_index , row in enumerate(floor_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] != elem:
check_status = False
if not check_status:
tdLog.notice("floor function value has not as expected , sql is \"%s\" "%floor_query )
sys.exit(1)
else:
tdLog.info("floor value check pass , it work as expected ,sql is \"%s\" "%floor_query )
tdSql.checkData(row_index,col_index,auto_result[row_index][col_index])
def test_errors(self, dbname=DBNAME):
error_sql_lists = [
@ -388,10 +383,10 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
@ -399,15 +394,15 @@ class TDTestCase:
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto( f"select c1, c2, c3 , c4, c5 ,c6 from {dbname}.sub1_bound ", f"select floor(c1), floor(c2) ,floor(c3), floor(c4), floor(c5) ,floor(c6) from {dbname}.sub1_bound")
self.check_result_auto( f"select c1, c2, c3 , c3, c2 ,c1 from {dbname}.sub1_bound ", f"select floor(c1), floor(c2) ,floor(c3), floor(c3), floor(c2) ,floor(c1) from {dbname}.sub1_bound")

10
tests/system-test/2-query/function_stateduration.py
View File

@ -364,10 +364,10 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
@ -375,15 +375,15 @@ class TDTestCase:
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.query(f"select stateduration(c1,'GT',1,1s) from {dbname}.sub1_bound")

964
tests/system-test/2-query/interp.py
View File

File diff suppressed because it is too large Load Diff

8
tests/system-test/2-query/last_row.py
View File

@ -572,19 +572,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
# check basic elem for table per row

9
tests/system-test/2-query/log.py
View File

@ -91,7 +91,6 @@ class TDTestCase:
elem = math.log(elem , base)
elif elem <=0:
elem = None
row_check.append(elem)
auto_result.append(row_check)
@ -519,19 +518,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_log(None , f"select c1, c2, c3 , c4, c5 ,c6 from {dbname}.sub1_bound ", f"select log(c1), log(c2) ,log(c3), log(c4), log(c5) ,log(c6) from {dbname}.sub1_bound")
self.check_result_auto_log( 2 , f"select c1, c2, c3 , c4, c5 ,c6 from {dbname}.sub1_bound ", f"select log(c1,2), log(c2,2) ,log(c3,2), log(c4,2), log(c5,2) ,log(c6,2) from {dbname}.sub1_bound")

8
tests/system-test/2-query/pow.py
View File

@ -507,19 +507,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_pow(2, f"select c1, c3 , c4, c5 from {dbname}.sub1_bound ", f"select pow(c1,2), pow(c3,2), pow(c4,2), pow(c5,2) from {dbname}.sub1_bound")
self.check_result_auto_pow(3, f"select c1, c3 , c4, c5 from {dbname}.sub1_bound ", f"select pow(c1,3), pow(c3,3), pow(c4,3), pow(c5,3) from {dbname}.sub1_bound")

22
tests/system-test/2-query/round.py
View File

@ -81,16 +81,12 @@ class TDTestCase:
row_check.append(elem)
auto_result.append(row_check)
check_status = True
tdSql.query(round_query)
for row_index , row in enumerate(round_result):
for col_index , elem in enumerate(row):
if auto_result[row_index][col_index] != elem:
check_status = False
if not check_status:
tdLog.notice("round function value has not as expected , sql is \"%s\" "%round_query )
sys.exit(1)
else:
tdLog.info("round value check pass , it work as expected ,sql is \"%s\" "%round_query )
tdSql.checkData(row_index , col_index ,auto_result[row_index][col_index])
def test_errors(self, dbname="db"):
error_sql_lists = [
@ -388,10 +384,10 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
@ -399,15 +395,15 @@ class TDTestCase:
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto( f"select c1, c2, c3 , c4, c5 ,c6 from {dbname}.sub1_bound ", f"select round(c1), round(c2) ,round(c3), round(c4), round(c5) ,round(c6) from {dbname}.sub1_bound")
self.check_result_auto( f"select c1, c2, c3 , c3, c2 ,c1 from {dbname}.sub1_bound ", f"select round(c1), round(c2) ,round(c3), round(c3), round(c2) ,round(c1) from {dbname}.sub1_bound")

8
tests/system-test/2-query/sin.py
View File

@ -394,19 +394,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_sin( f"select abs(c1), abs(c2), abs(c3) , abs(c4) from {dbname}.sub1_bound ", f"select sin(abs(c1)), sin(abs(c2)) ,sin(abs(c3)), sin(abs(c4)) from {dbname}.sub1_bound")

8
tests/system-test/2-query/sqrt.py
View File

@ -443,19 +443,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_sqrt( f"select abs(c1), abs(c2), abs(c3) , abs(c4), abs(c5) from {dbname}.sub1_bound ", f"select sqrt(abs(c1)), sqrt(abs(c2)) ,sqrt(abs(c3)), sqrt(abs(c4)), sqrt(abs(c5)) from {dbname}.sub1_bound")

10
tests/system-test/2-query/statecount.py
View File

@ -451,10 +451,10 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
@ -462,15 +462,15 @@ class TDTestCase:
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.query(f"select statecount(c1,'GT',1) from {dbname}.sub1_bound")

16
tests/system-test/2-query/tail.py
View File

@ -421,10 +421,10 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
@ -432,20 +432,20 @@ class TDTestCase:
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+15s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.query(f"select tail(c2,2) from {dbname}.sub1_bound order by 1 desc")
tdSql.checkRows(2)
tdSql.checkData(0,0,9223372036854775803)
tdSql.query(f"select tail(c2,1) from {dbname}.sub1_bound order by 1 desc")
tdSql.checkRows(1)
tdSql.checkData(0,0,-9223372036854775803)
def run(self): # sourcery skip: extract-duplicate-method, remove-redundant-fstring
tdSql.prepare()

8
tests/system-test/2-query/tan.py
View File

@ -391,19 +391,19 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
self.check_result_auto_tan( f"select abs(c1), abs(c2), abs(c3) , abs(c4) from {dbname}.sub1_bound ", f"select tan(abs(c1)), tan(abs(c2)) ,tan(abs(c3)), tan(abs(c4)) from {dbname}.sub1_bound")

40
tests/system-test/2-query/txt.txt Normal file
View File

@ -0,0 +1,40 @@
abs.py: def check_boundary_values(self):
abs.py: self.check_boundary_values()
and_or_for_byte.py: def check_boundary_values(self, dbname="bound_test"):
and_or_for_byte.py: self.check_boundary_values()
arccos.py: def check_boundary_values(self, dbname="bound_test"):
arccos.py: self.check_boundary_values()
arcsin.py: def check_boundary_values(self, dbname="bound_test"):
arcsin.py: self.check_boundary_values()
arctan.py: def check_boundary_values(self, dbname="bound_test"):
arctan.py: self.check_boundary_values()
avg.py: def check_boundary_values(self, dbname="bound_test"):
avg.py: self.check_boundary_values()
ceil.py: def check_boundary_values(self, dbname="bound_test"):
ceil.py: self.check_boundary_values()
cos.py: def check_boundary_values(self, dbname="bound_test"):
cos.py: self.check_boundary_values()
floor.py: def check_boundary_values(self, dbname="bound_test"):
floor.py: self.check_boundary_values()
function_stateduration.py: def check_boundary_values(self, dbname="bound_test"):
function_stateduration.py: self.check_boundary_values()
last_row.py: def check_boundary_values(self, dbname="bound_test"):
last_row.py: self.check_boundary_values()
log.py: def check_boundary_values(self, dbname="bound_test"):
log.py: self.check_boundary_values()
pow.py: def check_boundary_values(self, dbname="bound_test"):
pow.py: self.check_boundary_values()
round.py: def check_boundary_values(self, dbname="bound_test"):
round.py: self.check_boundary_values()
sin.py: def check_boundary_values(self, dbname="testdb"):
sin.py: self.check_boundary_values()
sqrt.py: def check_boundary_values(self, dbname="bound_test"):
sqrt.py: self.check_boundary_values()
statecount.py: def check_boundary_values(self, dbname="bound_test"):
statecount.py: self.check_boundary_values()
tail.py: def check_boundary_values(self, dbname="bound_test"):
tail.py: self.check_boundary_values()
tan.py: def check_boundary_values(self, dbname="bound_test"):
tan.py: self.check_boundary_values()
unique.py: def check_boundary_values(self, dbname="bound_test"):
unique.py: self.check_boundary_values()

12
tests/system-test/2-query/unique.py
View File

@ -451,26 +451,26 @@ class TDTestCase:
)
tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )')
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()-1s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()-5s, 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()+1s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()+2s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483643, 9223372036854775803, 32763, 123, 3.39E+38, 1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.execute(
f"insert into {dbname}.sub1_bound values ( now()+3s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+20s, -2147483643, -9223372036854775803, -32763, -123, -3.39E+38, -1.69e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.error(
f"insert into {dbname}.sub1_bound values ( now()+1s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
f"insert into {dbname}.sub1_bound values ( now()+30s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )"
)
tdSql.query(f"select unique(c2) from {dbname}.sub1_bound order by 1 desc")

10
tests/system-test/7-tmq/tmqDnodeRestart1.py
View File

@ -149,6 +149,8 @@ class TDTestCase:
tmqCom.waitSubscriptionExit(tdSql, topicFromStb)
tdSql.query("drop topic %s"%topicFromStb)
tmqCom.stopTmqSimProcess(processorName="tmq_sim")
tdLog.printNoPrefix("======== test case 1 end ...... ")
@ -178,6 +180,8 @@ class TDTestCase:
paraDict['vgroups'] = self.vgroups
paraDict['ctbNum'] = self.ctbNum
paraDict['rowsPerTbl'] = self.rowsPerTbl
tmqCom.initConsumerTable()
tdLog.info("create topics from stb")
topicFromDb = 'topic_db'
@ -203,10 +207,10 @@ class TDTestCase:
tmqCom.getStartCommitNotifyFromTmqsim('cdb',1)
tdLog.info("create some new child table and insert data for latest mode")
paraDict["batchNum"] = 100
paraDict["batchNum"] = 10
paraDict["ctbPrefix"] = 'newCtb'
paraDict["ctbNum"] = 10
paraDict["rowsPerTbl"] = 10
paraDict["ctbNum"] = 100
paraDict["rowsPerTbl"] = 100
tmqCom.insert_data_with_autoCreateTbl(tdSql,paraDict["dbName"],paraDict["stbName"],paraDict["ctbPrefix"],paraDict["ctbNum"],paraDict["rowsPerTbl"],paraDict["batchNum"])
tdLog.info("================= restart dnode ===========================")

Some files were not shown because too many files have changed in this diff Show More