xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/adj_assert

This commit is contained in:
54liuyao 2024-08-20 19:35:36 +08:00
parent 1f8e712a64 c1df349d4e
commit bd37547994
76 changed files with 1104 additions and 587 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -134,3 +134,4 @@ tags
.clangd
*CMakeCache*
*CMakeFiles*
.history/

0
docs/zh/14-reference/03-taos-sql/25-grant.md → docs/zh/14-reference/03-taos-sql/25-user.md
View File

168
docs/zh/14-reference/03-taos-sql/26-grant.md Normal file
View File

@ -0,0 +1,168 @@
---
toc_max_heading_level: 4
title: 权限管理
---
TDengine 中的权限管理分为[用户管理](../user)、数据库授权管理以及消息订阅授权管理，本节重点说明数据库授权和订阅授权。
## 数据库访问授权
系统管理员可以根据业务需要对系统中的每个用户针对每个数据库进行特定的授权，以防止业务数据被不恰当的用户读取或修改。对某个用户进行数据库访问授权的语法如下：
```sql
GRANT privileges ON priv_level TO user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
```
对数据库的访问权限包含读和写两种权限，它们可以被分别授予，也可以被同时授予。
说明
- priv_level 格式中 "." 之前为数据库名称， "." 之后为表名称，意思为表级别的授权控制。如果 "." 之后为 "\*" ，意为 "." 前所指定的数据库中的所有表
- "dbname.\*" 意思是名为 "dbname" 的数据库中的所有表
- "\*.\*" 意思是所有数据库名中的所有表
### 数据库权限说明
对 root 用户和普通用户的权限的说明如下表
| 用户 | 描述 | 权限说明 |
| -------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 超级用户 | 只有 root 是超级用户 | DB 外部 所有操作权限，例如user、dnode、udf、qnode等的CRUD DB 权限，包括 创建 删除 更新，例如修改 Option，移动 Vgruop等 读 写 Enable/Disable 用户 |
| 普通用户 | 除 root 以外的其它用户均为普通用户 | 在可读的 DB 中，普通用户可以进行读操作 select describe show subscribe 在可写 DB 的内部，用户可以进行写操作： 创建、删除、修改 超级表 创建、删除、修改 子表 创建、删除、修改 topic 写入数据 被限制系统信息时，不可进行如下操作 show dnode、mnode、vgroups、qnode、snode 修改用户包括自身密码 show db时只能看到自己的db，并且不能看到vgroups、副本、cache等信息 无论是否被限制系统信息，都可以 管理 udf 可以创建 DB 自己创建的 DB 具备所有权限 非自己创建的 DB ，参照读、写列表中的权限 |
## 消息订阅授权
任意用户都可以在自己拥有读权限的数据库上创建 topic。超级用户 root 可以在任意数据库上创建 topic。每个 topic 的订阅权限都可以被独立授权给任何用户，不管该用户是否拥有该数据库的访问权限。删除 topic 只能由 root 用户或者该 topic 的创建者进行。topic 只能由超级用户、topic的创建者或者被显式授予 subscribe 权限的用户订阅。
具体的 SQL 语法如下：
```sql
GRANT SUBSCRIBE ON topic_name TO user_name
REVOKE SUBSCRIBE ON topic_name FROM user_name
```
## 基于标签的授权（表级授权）
从 TDengine 3.0.5.0 开始，我们支持按标签授权某个超级表中部分特定的子表。具体的 SQL 语法如下。
```sql
GRANT privileges ON priv_level [WITH tag_condition] TO user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
REVOKE privileges ON priv_level [WITH tag_condition] FROM user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
```
上面 SQL 的语义为：
- 用户可以通过 dbname.tbname 来为指定的表（包括超级表和普通表）授予或回收其读写权限，不支持直接对子表授予或回收权限。
- 用户可以通过 dbname.tbname 和 WITH 子句来为符合条件的所有子表授予或回收其读写权限。使用 WITH 子句时，权限级别必须为超级表。
## 表级权限和数据库权限的关系
下表列出了在不同的数据库授权和表级授权的组合下产生的实际权限。
| | **表无授权** | **表读授权** | **表读授权有标签条件** | **表写授权** | **表写授权有标签条件** |
| ---------------- | ---------------- | ---------------------------------------- | ------------------------------------------------------------ | ---------------------------------------- | ---------------------------------------------------------- |
| **数据库无授权** | 无授权 | 对此表有读权限，对数据库下的其他表无权限 | 对此表符合标签权限的子表有读权限，对数据库下的其他表无权限 | 对此表有写权限，对数据库下的其他表无权限 | 对此表符合标签权限的子表有写权限，对数据库下的其他表无权限 |
| **数据库读授权** | 对所有表有读权限 | 对所有表有读权限 | 对此表符合标签权限的子表有读权限，对数据库下的其他表有读权限 | 对此表有写权限，对所有表有读权限 | 对此表符合标签权限的子表有写权限，所有表有读权限 |
| **数据库写授权** | 对所有表有写权限 | 对此表有读权限，对所有表有写权限 | 对此表符合标签权限的子表有读权限，对所有表有写权限 | 对所有表有写权限 | 对此表符合标签权限的子表有写权限，数据库下的其他表有写权限 |
## 查看用户授权
使用下面的命令可以显示一个用户所拥有的授权：
```sql
show user privileges
```
## 撤销授权
1. 撤销数据库访问的授权
```sql
REVOKE privileges ON priv_level FROM user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
```
2. 撤销数据订阅的授权
```sql
REVOKE privileges ON priv_level FROM user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
SUBSCRIBE
}
priv_level : {
topic_name
}
```

0
docs/zh/14-reference/03-taos-sql/26-udf.md → docs/zh/14-reference/03-taos-sql/27-udf.md
View File

0
docs/zh/14-reference/03-taos-sql/27-indexing.md → docs/zh/14-reference/03-taos-sql/28-index.md
View File

0
docs/zh/14-reference/03-taos-sql/28-recovery.md → docs/zh/14-reference/03-taos-sql/29-recovery.md
View File

0
docs/zh/14-reference/03-taos-sql/29-changes.md → docs/zh/14-reference/03-taos-sql/30-changes.md
View File

0
docs/zh/14-reference/03-taos-sql/30-join.md → docs/zh/14-reference/03-taos-sql/31-join.md
View File

0
docs/zh/14-reference/03-taos-sql/31-compress.md → docs/zh/14-reference/03-taos-sql/32-compress.md
View File

0
docs/zh/14-reference/03-taos-sql/32-view.md → docs/zh/14-reference/03-taos-sql/33-view.md
View File

20
docs/zh/20-third-party/01-collection/11-kafka.md vendored
View File

@ -27,7 +27,7 @@ TDengine Source Connector 用于把数据实时地从 TDengine 读出来发送
## 安装 Kafka
在任意目录下执行：
- 在任意目录下执行：
```shell
curl -O https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
@ -35,13 +35,12 @@ tar xzf kafka_2.13-3.4.0.tgz -C /opt/
ln -s /opt/kafka_2.13-3.4.0 /opt/kafka
```
然后需要把 `$KAFKA_HOME/bin` 目录加入 PATH。
- 然后需要把 `$KAFKA_HOME/bin` 目录加入 PATH。
```title=".profile"
export KAFKA_HOME=/opt/kafka
export PATH=$PATH:$KAFKA_HOME/bin
```
以上脚本可以追加到当前用户的 profile 文件（~/.profile 或 ~/.bash_profile）
## 安装 TDengine Connector 插件
@ -325,6 +324,21 @@ curl -X DELETE http://localhost:8083/connectors/TDengineSinkConnector
curl -X DELETE http://localhost:8083/connectors/TDengineSourceConnector
```
### 性能调优
如果在从 TDengine 同步数据到 Kafka 的过程中发现性能不达预期，可以尝试使用如下参数提升 Kafka 的写入吞吐量。
1. 打开 KAFKA_HOME/config/producer.properties 配置文件。
2. 参数说明及配置建议如下：
| **参数** | **参数说明** | **设置建议** |
| --------| --------------------------------- | -------------- |
| producer.type | 此参数用于设置消息的发送方式，默认值为 `sync` 表示同步发送，`async` 表示异步发送。采用异步发送能够提升消息发送的吞吐量。 | async |
| request.required.acks | 参数用于配置生产者发送消息后需要等待的确认数量。当设置为1时，表示只要领导者副本成功写入消息就会给生产者发送确认，而无需等待集群中的其他副本写入成功。这种设置可以在一定程度上保证消息的可靠性，同时也能保证一定的吞吐量。因为不需要等待所有副本都写入成功，所以可以减少生产者的等待时间，提高发送消息的效率。|1|
| max.request.size| 该参数决定了生产者在一次请求中可以发送的最大数据量。其默认值为 1048576，也就是 1M。如果设置得太小，可能会导致频繁的网络请求，降低吞吐量。如果设置得太大，可能会导致内存占用过高，或者在网络状况不佳时增加请求失败的概率。建议设置为 100M。|104857600|
|batch.size| 此参数用于设定 batch 的大小，默认值为 16384，即 16KB。在消息发送过程中，发送到 Kafka 缓冲区中的消息会被划分成一个个的 batch。故而减小 batch 大小有助于降低消息延迟，而增大 batch 大小则有利于提升吞吐量，可根据实际的数据量大小进行合理配置。可根据实际情况进行调整，建议设置为 512K。|524288|
| buffer.memory| 此参数用于设置生产者缓冲待发送消息的内存总量。较大的缓冲区可以允许生产者积累更多的消息后批量发送，提高吞吐量，但也会增加延迟和内存使用。可根据机器资源来配置，建议配置为 1G。|1073741824|
## 配置参考
### 通用配置

5
include/util/taoserror.h
View File

@ -604,6 +604,9 @@ int32_t taosGetErrSize();
#define TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x0732)
#define TSDB_CODE_QRY_INVALID_JOIN_CONDITION TAOS_DEF_ERROR_CODE(0, 0x0733)
#define TSDB_CODE_QRY_FILTER_NOT_SUPPORT_TYPE TAOS_DEF_ERROR_CODE(0, 0x0734)
#define TSDB_CODE_QRY_FILTER_WRONG_OPTR_TYPE TAOS_DEF_ERROR_CODE(0, 0x0735)
#define TSDB_CODE_QRY_FILTER_RANGE_ERROR TAOS_DEF_ERROR_CODE(0, 0x0736)
#define TSDB_CODE_QRY_FILTER_INVALID_TYPE TAOS_DEF_ERROR_CODE(0, 0x0737)
// grant
#define TSDB_CODE_GRANT_EXPIRED TAOS_DEF_ERROR_CODE(0, 0x0800)
@ -878,6 +881,8 @@ int32_t taosGetErrSize();
#define TSDB_CODE_FUNC_INVALID_VALUE_RANGE TAOS_DEF_ERROR_CODE(0, 0x280C)
#define TSDB_CODE_FUNC_SETUP_ERROR TAOS_DEF_ERROR_CODE(0, 0x280D)
#define TSDB_CODE_FUNC_INVALID_RES_LENGTH TAOS_DEF_ERROR_CODE(0, 0x280E)
#define TSDB_CODE_FUNC_HISTOGRAM_ERROR TAOS_DEF_ERROR_CODE(0, 0x280F)
#define TSDB_CODE_FUNC_PERCENTILE_ERROR TAOS_DEF_ERROR_CODE(0, 0x2810)
//udf

3
include/util/tcoding.h
View File

@ -213,7 +213,6 @@ static FORCE_INLINE int32_t taosEncodeVariantU16(void **buf, uint16_t value) {
if (buf != NULL) ((uint8_t *)(*buf))[i] = (uint8_t)(value | ENCODE_LIMIT);
value >>= 7;
i++;
ASSERT(i < 3);
}
if (buf != NULL) {
@ -261,7 +260,6 @@ static FORCE_INLINE int32_t taosEncodeVariantU32(void **buf, uint32_t value) {
if (buf != NULL) ((uint8_t *)(*buf))[i] = (value | ENCODE_LIMIT);
value >>= 7;
i++;
ASSERT(i < 5);
}
if (buf != NULL) {
@ -309,7 +307,6 @@ static FORCE_INLINE int32_t taosEncodeVariantU64(void **buf, uint64_t value) {
if (buf != NULL) ((uint8_t *)(*buf))[i] = (uint8_t)(value | ENCODE_LIMIT);
value >>= 7;
i++;
ASSERT(i < 10);
}
if (buf != NULL) {

3
include/util/tref.h
View File

@ -29,8 +29,7 @@ typedef void (*RefFp)(void *);
int32_t taosOpenRef(int32_t max, RefFp fp);
// close the reference set, refId is the return value by taosOpenRef
// return 0 if success. On error, -1 is returned, and terrno is set appropriately
int32_t taosCloseRef(int32_t rsetId);
void taosCloseRef(int32_t rsetId);
// add ref, p is the pointer to resource or pointer ID
// return Reference ID(rid) allocated. On error, -1 is returned, and terrno is set appropriately

2
source/client/src/clientImpl.c
View File

@ -1135,8 +1135,6 @@ void schedulerExecCb(SExecResult* pResult, void* param, int32_t code) {
(void)atomic_add_fetch_64((int64_t*)&pActivity->numOfInsertRows, pResult->numOfRows);
}
}
schedulerFreeJob(&pRequest->body.queryJob, 0);
}
taosMemoryFree(pResult);

8
source/client/src/clientMain.c
View File

@ -74,15 +74,11 @@ void taos_cleanup(void) {
int32_t id = clientReqRefPool;
clientReqRefPool = -1;
if (TSDB_CODE_SUCCESS != taosCloseRef(id)) {
tscWarn("failed to close clientReqRefPool");
}
taosCloseRef(id);
id = clientConnRefPool;
clientConnRefPool = -1;
if (TSDB_CODE_SUCCESS != taosCloseRef(id)) {
tscWarn("failed to close clientReqRefPool");
}
taosCloseRef(id);
nodesDestroyAllocatorSet();
cleanupAppInfo();

2
source/client/src/clientTmq.c
View File

@ -1184,7 +1184,7 @@ void tmqMgmtClose(void) {
}
if (tmqMgmt.rsetId >= 0) {
(void)taosCloseRef(tmqMgmt.rsetId);
taosCloseRef(tmqMgmt.rsetId);
tmqMgmt.rsetId = -1;
}
}

42
source/common/src/cos.c
View File

@ -12,7 +12,7 @@ extern char tsS3AccessKeySecret[][TSDB_FQDN_LEN];
extern char tsS3BucketName[TSDB_FQDN_LEN];
extern char tsS3AppId[][TSDB_FQDN_LEN];
extern char tsS3Hostname[][TSDB_FQDN_LEN];
extern int8_t tsS3Https;
extern int8_t tsS3Https[];
static int32_t s3ListBucketByEp(char const *bucketname, int8_t epIndex);
static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *object_name, int64_t offset, int64_t size,
@ -33,13 +33,13 @@ static int verifyPeerG = 0;
static const char *awsRegionG = NULL;
static int forceG = 0;
static int showResponsePropertiesG = 0;
static S3Protocol protocolG = S3ProtocolHTTPS;
static S3Protocol protocolG[TSDB_MAX_EP_NUM] = {S3ProtocolHTTPS};
// static S3Protocol protocolG = S3ProtocolHTTP;
static S3UriStyle uriStyleG = S3UriStylePath;
static S3UriStyle uriStyleG[TSDB_MAX_EP_NUM] = {S3UriStylePath};
static int retriesG = 5;
static int timeoutMsG = 0;
extern int8_t tsS3Oss;
extern int8_t tsS3Oss[];
int32_t s3Begin() {
S3Status status;
@ -55,9 +55,9 @@ int32_t s3Begin() {
TAOS_RETURN(TSDB_CODE_FAILED);
}
protocolG = !tsS3Https;
if (tsS3Oss) {
uriStyleG = S3UriStyleVirtualHost;
for (int i = 0; i < tsS3EpNum; i++) {
protocolG[i] = tsS3Https[i] ? S3ProtocolHTTPS : S3ProtocolHTTP;
uriStyleG[i] = tsS3Oss[i] ? S3UriStyleVirtualHost : S3UriStylePath;
}
TAOS_RETURN(TSDB_CODE_SUCCESS);
@ -976,8 +976,8 @@ int32_t s3PutObjectFromFile2ByEp(const char *file, const char *object_name, int8
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1059,8 +1059,8 @@ static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *objec
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1155,8 +1155,8 @@ static void s3FreeObjectKey(void *pItem) {
static SArray *getListByPrefixByEp(const char *prefix, int8_t epIndex) {
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1223,8 +1223,8 @@ static int32_t s3DeleteObjectsByEp(const char *object_name[], int nobject, int8_
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1299,8 +1299,8 @@ static int32_t s3GetObjectBlockByEp(const char *object_name, int64_t offset, int
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1372,8 +1372,8 @@ static int32_t s3GetObjectToFileByEp(const char *object_name, const char *fileNa
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1449,8 +1449,8 @@ static long s3SizeByEp(const char *object_name, int8_t epIndex) {
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,

9
source/common/src/tglobal.c
View File

@ -303,10 +303,10 @@ char tsS3BucketName[TSDB_FQDN_LEN] = "<bucketname>";
char tsS3AppId[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<appid>"};
int8_t tsS3Enabled = false;
int8_t tsS3EnabledCfg = false;
int8_t tsS3Oss = false;
int8_t tsS3Oss[TSDB_MAX_EP_NUM] = {false};
int8_t tsS3StreamEnabled = false;
int8_t tsS3Https = true;
int8_t tsS3Https[TSDB_MAX_EP_NUM] = {true};
char tsS3Hostname[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<hostname>"};
int32_t tsS3BlockSize = -1; // number of tsdb pages (4096)
@ -431,11 +431,10 @@ int32_t taosSetS3Cfg(SConfig *pCfg) {
tstrncpy(tsS3AppId[i], appid + 1, TSDB_FQDN_LEN);
}
}
tsS3Https[i] = (strstr(tsS3Endpoint[i], "https://") != NULL);
tsS3Oss[i] = (strstr(tsS3Endpoint[i], "aliyuncs.") != NULL);
}
tsS3Https = (strstr(tsS3Endpoint[0], "https://") != NULL);
tsS3Oss = (strstr(tsS3Endpoint[0], "aliyuncs.") != NULL);
if (tsS3BucketName[0] != '<') {
#if defined(USE_COS) || defined(USE_S3)
#ifdef TD_ENTERPRISE

2
source/dnode/mnode/impl/src/mndSync.c
View File

@ -486,7 +486,7 @@ int32_t mndInitSync(SMnode *pMnode) {
int32_t code = 0;
(void)tsem_init(&pMgmt->syncSem, 0, 0);
pMgmt->sync = syncOpen(&syncInfo, true);
pMgmt->sync = syncOpen(&syncInfo, 1); // always check
if (pMgmt->sync <= 0) {
if (terrno != 0) code = terrno;
mError("failed to open sync since %s", tstrerror(code));

2
source/dnode/vnode/src/inc/tsdb.h
View File

@ -898,7 +898,7 @@ typedef struct SSttDataInfoForTable {
int32_t tMergeTreeOpen2(SMergeTree *pMTree, SMergeTreeConf *pConf, SSttDataInfoForTable *pTableInfo);
void tMergeTreeAddIter(SMergeTree *pMTree, SLDataIter *pIter);
bool tMergeTreeNext(SMergeTree *pMTree);
int32_t tMergeTreeNext(SMergeTree *pMTree, bool* pHasNext);
void tMergeTreePinSttBlock(SMergeTree *pMTree);
void tMergeTreeUnpinSttBlock(SMergeTree *pMTree);
bool tMergeTreeIgnoreEarlierTs(SMergeTree *pMTree);

6
source/dnode/vnode/src/meta/metaQuery.c
View File

@ -694,9 +694,13 @@ int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sv
SSchemaWrapper *pSchemaWrapper = &schema;
tDecoderInit(&dc, pData, nData);
(void)tDecodeSSchemaWrapper(&dc, pSchemaWrapper);
code = tDecodeSSchemaWrapper(&dc, pSchemaWrapper);
tDecoderClear(&dc);
tdbFree(pData);
if (TSDB_CODE_SUCCESS != code) {
taosMemoryFree(pSchemaWrapper->pSchema);
goto _exit;
}
// convert
STSchema *pTSchema = tBuildTSchema(pSchemaWrapper->pSchema, pSchemaWrapper->nCols, pSchemaWrapper->version);

35
source/dnode/vnode/src/meta/metaTtl.c
View File

@ -254,6 +254,16 @@ static int32_t ttlMgrFindExpiredOneEntry(const void *pKey, int keyLen, const voi
return c;
}
// static int32_t ttlMgrDumpOneEntry(const void *pKey, int keyLen, const void *pVal, int valLen, void *pDumpCtx) {
// STtlIdxKeyV1 *ttlKey = (STtlIdxKeyV1 *)pKey;
// int64_t *ttlDays = (int64_t *)pVal;
// metaInfo("ttlMgr dump, ttl: %" PRId64 ", ctime: %" PRId64 ", uid: %" PRId64, *ttlDays, ttlKey->deleteTimeMs,
// ttlKey->uid);
// TAOS_RETURN(TSDB_CODE_SUCCESS);
// }
static int ttlMgrConvert(TTB *pOldTtlIdx, TTB *pNewTtlIdx, void *pMeta) {
SMeta *meta = pMeta;
@ -402,15 +412,20 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) {
int32_t code = TSDB_CODE_SUCCESS;
void *pIter = taosHashIterate(pTtlMgr->pDirtyUids, NULL);
while (pIter != NULL) {
void *pIter = NULL;
while ((pIter = taosHashIterate(pTtlMgr->pDirtyUids, pIter)) != NULL) {
STtlDirtyEntry *pEntry = (STtlDirtyEntry *)pIter;
tb_uid_t *pUid = taosHashGetKey(pIter, NULL);
STtlCacheEntry *cacheEntry = taosHashGet(pTtlMgr->pTtlCache, pUid, sizeof(*pUid));
if (cacheEntry == NULL) {
metaError("%s, ttlMgr flush failed to get ttl cache, uid: %" PRId64 ", type: %d", pTtlMgr->logPrefix, *pUid,
metaInfo("%s, ttlMgr flush failed to get ttl cache, uid: %" PRId64 ", type: %d", pTtlMgr->logPrefix, *pUid,
pEntry->type);
code = taosHashRemove(pTtlMgr->pDirtyUids, pUid, sizeof(*pUid));
if (TSDB_CODE_SUCCESS != code) {
metaError("%s, ttlMgr flush failed to remove dirty uid since %s", pTtlMgr->logPrefix, tstrerror(code));
goto _out;
}
continue;
}
@ -449,9 +464,15 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) {
goto _out;
}
void *pIterTmp = pIter;
pIter = taosHashIterate(pTtlMgr->pDirtyUids, pIterTmp);
(void)taosHashRemove(pTtlMgr->pDirtyUids, pUid, sizeof(tb_uid_t));
metaDebug("isdel:%d", pEntry->type == ENTRY_TYPE_DELETE);
metaDebug("ttlkey:%" PRId64 ", uid:%" PRId64, ttlKey.deleteTimeMs, ttlKey.uid);
metaDebug("ttlkeyDirty:%" PRId64 ", uid:%" PRId64, ttlKeyDirty.deleteTimeMs, ttlKeyDirty.uid);
code = taosHashRemove(pTtlMgr->pDirtyUids, pUid, sizeof(*pUid));
if (TSDB_CODE_SUCCESS != code) {
metaError("%s, ttlMgr flush failed to remove dirty uid since %s", pTtlMgr->logPrefix, tstrerror(code));
goto _out;
}
}
taosHashClear(pTtlMgr->pDirtyUids);
@ -459,6 +480,8 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) {
code = TSDB_CODE_SUCCESS;
_out:
taosHashCancelIterate(pTtlMgr->pDirtyUids, pIter);
endNs = taosGetTimestampNs();
metaTrace("%s, ttl mgr flush end, time consumed: %" PRId64 " ns", pTtlMgr->logPrefix, endNs - startNs);

4
source/dnode/vnode/src/sma/smaEnv.c
View File

@ -69,7 +69,7 @@ int32_t smaInit() {
if (!smaMgmt.refHash || !smaMgmt.tmrHandle) {
code = terrno;
(void)taosCloseRef(smaMgmt.rsetId);
taosCloseRef(smaMgmt.rsetId);
if (smaMgmt.refHash) {
taosHashCleanup(smaMgmt.refHash);
smaMgmt.refHash = NULL;
@ -103,7 +103,7 @@ void smaCleanUp() {
}
if (old == 1) {
(void)taosCloseRef(smaMgmt.rsetId);
taosCloseRef(smaMgmt.rsetId);
taosHashCleanup(smaMgmt.refHash);
smaMgmt.refHash = NULL;
taosTmrCleanUp(smaMgmt.tmrHandle);

4
source/dnode/vnode/src/tq/tq.c
View File

@ -1204,11 +1204,11 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
}
if (req.mndTrigger) {
qInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 ", transId:%d, ", pTask->id.idStr,
tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 ", transId:%d, ", pTask->id.idStr,
vgId, pTask->info.taskLevel, req.checkpointId, req.transId);
} else {
const char* pPrevStatus = streamTaskGetStatusStr(streamTaskGetPrevStatus(pTask));
qInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64
tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64
", transId:%d after transfer-state, prev status:%s",
pTask->id.idStr, vgId, pTask->info.taskLevel, req.checkpointId, req.transId, pPrevStatus);
}

13
source/dnode/vnode/src/tsdb/tsdbCache.c
View File

@ -2245,17 +2245,20 @@ static int32_t lastIterClose(SFSLastIter **iter) {
}
static int32_t lastIterNext(SFSLastIter *iter, TSDBROW **ppRow) {
int32_t code = 0;
bool hasVal = tMergeTreeNext(iter->pMergeTree);
if (!hasVal) {
bool hasVal = false;
*ppRow = NULL;
int32_t code = tMergeTreeNext(iter->pMergeTree, &hasVal);
if (code != 0) {
return code;
}
if (!hasVal) {
*ppRow = NULL;
TAOS_RETURN(code);
}
*ppRow = tMergeTreeGetRow(iter->pMergeTree);
TAOS_RETURN(code);
}

25
source/dnode/vnode/src/tsdb/tsdbMergeTree.c
View File

@ -115,10 +115,7 @@ void destroySttBlockReader(SArray *pLDataIterArray, SSttBlockLoadCostInfo *pLoad
SArray *pList = taosArrayGetP(pLDataIterArray, i);
for (int32_t j = 0; j < taosArrayGetSize(pList); ++j) {
SLDataIter *pIter = taosArrayGetP(pList, j);
if (pIter->pBlockLoadInfo == NULL) {
continue;
}
if (pIter->pBlockLoadInfo != NULL) {
SSttBlockLoadCostInfo *pCost = &pIter->pBlockLoadInfo->cost;
if (pLoadCost != NULL) {
pLoadCost->loadBlocks += pCost->loadBlocks;
@ -126,6 +123,7 @@ void destroySttBlockReader(SArray *pLDataIterArray, SSttBlockLoadCostInfo *pLoad
pLoadCost->blockElapsedTime += pCost->blockElapsedTime;
pLoadCost->statisElapsedTime += pCost->statisElapsedTime;
}
}
destroyLDataIter(pIter);
}
@ -903,6 +901,7 @@ int32_t tLDataIterNextRow(SLDataIter *pIter, const char *idStr, bool* hasNext) {
pIter->rInfo.row = tsdbRowFromBlockData(pBlockData, pIter->iRow);
_exit:
tsdbError("failed to exec stt-file nextIter, lino:%d, code:%s, %s", lino, tstrerror(code), idStr);
*hasNext = (code == TSDB_CODE_SUCCESS) && (pIter->pSttBlk != NULL) && (pBlockData != NULL);
return code;
}
@ -1102,13 +1101,22 @@ void tMergeTreeUnpinSttBlock(SMergeTree *pMTree) {
tLDataIterUnpinSttBlock(pIter, pMTree->idStr);
}
bool tMergeTreeNext(SMergeTree *pMTree) {
int32_t tMergeTreeNext(SMergeTree *pMTree, bool *pHasNext) {
int32_t code = 0;
if (pHasNext == NULL) {
return TSDB_CODE_INVALID_PARA;
}
*pHasNext = false;
if (pMTree->pIter) {
SLDataIter *pIter = pMTree->pIter;
bool hasVal = false;
int32_t code = tLDataIterNextRow(pIter, pMTree->idStr, &hasVal);
code = tLDataIterNextRow(pIter, pMTree->idStr, &hasVal);
if (!hasVal || (code != 0)) {
if (code == TSDB_CODE_FILE_CORRUPTED) {
code = 0; // suppress the file corrupt error to enable all queries within this cluster can run without failed.
}
pMTree->pIter = NULL;
}
@ -1132,7 +1140,8 @@ bool tMergeTreeNext(SMergeTree *pMTree) {
}
}
return pMTree->pIter != NULL;
*pHasNext = (pMTree->pIter != NULL);
return code;
}
void tMergeTreeClose(SMergeTree *pMTree) {

90
source/dnode/vnode/src/tsdb/tsdbRead2.c
View File

@ -1383,7 +1383,6 @@ static int32_t copyBlockDataToSDataBlock(STsdbReader* pReader, SRowKey* pLastPro
static FORCE_INLINE STSchema* getTableSchemaImpl(STsdbReader* pReader, uint64_t uid) {
ASSERT(pReader->info.pSchema == NULL);
int32_t code = metaGetTbTSchemaEx(pReader->pTsdb->pVnode->pMeta, pReader->info.suid, uid, -1, &pReader->info.pSchema);
if (code != TSDB_CODE_SUCCESS || pReader->info.pSchema == NULL) {
terrno = code;
@ -1414,7 +1413,9 @@ static int32_t doLoadFileBlockData(STsdbReader* pReader, SDataBlockIter* pBlockI
if (pReader->info.pSchema == NULL) {
pSchema = getTableSchemaImpl(pReader, uid);
if (pSchema == NULL) {
tsdbDebug("%p table uid:%" PRIu64 " has been dropped, no data existed, %s", pReader, uid, pReader->idStr);
code = terrno;
tsdbError("%p table uid:%" PRIu64 " failed to get tableschema, code:%s, %s", pReader, uid, tstrerror(code),
pReader->idStr);
return code;
}
}
@ -1449,7 +1450,7 @@ static int32_t doLoadFileBlockData(STsdbReader* pReader, SDataBlockIter* pBlockI
pReader->cost.blockLoadTime += elapsedTime;
pDumpInfo->allDumped = false;
return TSDB_CODE_SUCCESS;
return code;
}
/**
@ -1759,14 +1760,22 @@ static bool tryCopyDistinctRowFromFileBlock(STsdbReader* pReader, SBlockData* pB
return code;
}
static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, int32_t pkSrcSlot,
static int32_t nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, int32_t pkSrcSlot,
SVersionRange* pVerRange) {
int32_t code = 0;
int32_t order = pSttBlockReader->order;
int32_t step = ASCENDING_TRAVERSE(order) ? 1 : -1;
SRowKey* pNextProc = &pScanInfo->sttKeyInfo.nextProcKey;
while (1) {
bool hasVal = tMergeTreeNext(&pSttBlockReader->mergeTree);
bool hasVal = false;
code = tMergeTreeNext(&pSttBlockReader->mergeTree, &hasVal);
if (code) {
tsdbError("failed to iter the next row in stt-file merge tree, code:%s, %s", tstrerror(code),
pSttBlockReader->mergeTree.idStr);
return code;
}
if (!hasVal) { // the next value will be the accessed key in stt
pScanInfo->sttKeyInfo.status = STT_FILE_NO_DATA;
@ -1779,7 +1788,7 @@ static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockSc
memset(pNextProc->pks[0].pData, 0, pNextProc->pks[0].nData);
}
}
return false;
return code;
}
TSDBROW* pRow = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
@ -1798,13 +1807,15 @@ static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockSc
if (!hasBeenDropped(pScanInfo->delSkyline, &pScanInfo->sttBlockDelIndex, key, ver, order, pVerRange,
pSttBlockReader->numOfPks > 0)) {
pScanInfo->sttKeyInfo.status = STT_FILE_HAS_DATA;
return true;
return code;
}
} else {
pScanInfo->sttKeyInfo.status = STT_FILE_HAS_DATA;
return true;
return code;
}
}
return code;
}
static void doPinSttBlock(SSttBlockReader* pSttBlockReader) { tMergeTreePinSttBlock(&pSttBlockReader->mergeTree); }
@ -1819,9 +1830,14 @@ static bool tryCopyDistinctRowFromSttBlock(TSDBROW* fRow, SSttBlockReader* pSttB
// avoid the fetch next row replace the referenced stt block in buffer
doPinSttBlock(pSttBlockReader);
bool hasVal = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
doUnpinSttBlock(pSttBlockReader);
if (hasVal) {
if (code) {
return code;
}
if (hasDataInSttBlock(pScanInfo)) {
SRowKey* pNext = getCurrentKeyInSttBlock(pSttBlockReader);
if (pkCompEx(pSttKey, pNext) != 0) {
code = doAppendRowFromFileBlock(pReader->resBlockInfo.pResBlock, pReader, fRow->pBlockData, fRow->iRow);
@ -2125,7 +2141,7 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
if (piRow->type == TSDBROW_ROW_FMT) {
piSchema = doGetSchemaForTSRow(TSDBROW_SVERSION(piRow), pReader, pBlockScanInfo->uid);
if (piSchema == NULL) {
return code;
return terrno;
}
}
@ -2380,14 +2396,13 @@ static bool isValidFileBlockRow(SBlockData* pBlockData, int32_t rowIndex, STable
return true;
}
static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, STsdbReader* pReader) {
bool hasData = true;
static void initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, STsdbReader* pReader) {
int32_t order = pReader->info.order;
bool asc = ASCENDING_TRAVERSE(order);
// the stt block reader has been initialized for this table.
if (pSttBlockReader->uid == pScanInfo->uid) {
return hasDataInSttBlock(pScanInfo);
return;
}
if (pSttBlockReader->uid != 0) {
@ -2396,9 +2411,14 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
pSttBlockReader->uid = pScanInfo->uid;
// second time init stt block reader
// second or third time init stt block reader
if (pScanInfo->cleanSttBlocks && (pReader->info.execMode == READER_EXEC_ROWS)) {
return !pScanInfo->sttBlockReturned;
// only allowed to retrieve clean stt blocks for count once
if (pScanInfo->sttBlockReturned) {
pScanInfo->sttKeyInfo.status = STT_FILE_NO_DATA;
tsdbDebug("uid:%" PRIu64 " set no stt-file data after stt-block retrieved, %s", pScanInfo->uid, pReader->idStr);
}
return;
}
STimeWindow w = pSttBlockReader->window;
@ -2435,28 +2455,28 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
SSttDataInfoForTable info = {.pKeyRangeList = taosArrayInit(4, sizeof(SSttKeyRange))};
if (info.pKeyRangeList == NULL) {
pReader->code = terrno;
return false;
return;
}
int32_t code = tMergeTreeOpen2(&pSttBlockReader->mergeTree, &conf, &info);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(info.pKeyRangeList);
pReader->code = code;
return false;
return;
}
code = initMemDataIterator(pScanInfo, pReader);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(info.pKeyRangeList);
pReader->code = code;
return false;
return;
}
code = initDelSkylineIterator(pScanInfo, pReader->info.order, &pReader->cost);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(info.pKeyRangeList);
pReader->code = code;
return false;
return;
}
if (conf.rspRows) {
@ -2484,27 +2504,26 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
SRowKey* p = asc ? &pScanInfo->sttRange.skey : &pScanInfo->sttRange.ekey;
tRowKeyAssign(&pScanInfo->sttKeyInfo.nextProcKey, p);
hasData = (pScanInfo->sttKeyInfo.status == STT_FILE_HAS_DATA);
} else { // not clean stt blocks
INIT_KEYRANGE(&pScanInfo->sttRange); // reset the time window
pScanInfo->sttBlockReturned = false;
hasData = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
}
} else {
pScanInfo->cleanSttBlocks = false;
INIT_KEYRANGE(&pScanInfo->sttRange); // reset the time window
pScanInfo->sttBlockReturned = false;
hasData = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
}
pScanInfo->sttBlockReturned = false;
taosArrayDestroy(info.pKeyRangeList);
int64_t el = taosGetTimestampUs() - st;
pReader->cost.initSttBlockReader += (el / 1000.0);
tsdbDebug("init stt block reader completed, elapsed time:%" PRId64 "us %s", el, pReader->idStr);
return hasData;
if (code != 0) {
pReader->code = code;
}
}
static bool hasDataInSttBlock(STableBlockScanInfo* pInfo) { return pInfo->sttKeyInfo.status == STT_FILE_HAS_DATA; }
@ -2772,7 +2791,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
}
SBlockData* pBlockData = &pReader->status.fileBlockData;
(void) initSttBlockReader(pSttBlockReader, pBlockScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pBlockScanInfo, pReader);
if (pReader->code != 0) {
code = pReader->code;
goto _end;
@ -3190,12 +3209,12 @@ static int32_t doLoadSttBlockSequentially(STsdbReader* pReader) {
continue;
}
bool hasDataInSttFile = initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
if (pReader->code != TSDB_CODE_SUCCESS) {
return pReader->code;
}
if (!hasDataInSttFile) {
if (!hasDataInSttBlock(pScanInfo)) {
bool hasNexTable = moveToNextTable(pUidList, pStatus);
if (!hasNexTable) {
return TSDB_CODE_SUCCESS;
@ -3287,7 +3306,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
}
if (pScanInfo->sttKeyInfo.status == STT_FILE_READER_UNINIT) {
(void) initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
if (pReader->code != 0) {
return pReader->code;
}
@ -3331,7 +3350,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
int64_t st = taosGetTimestampUs();
// let's load data from stt files, make sure clear the cleanStt block flag before load the data from stt files
(void) initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
if (pReader->code != 0) {
return pReader->code;
}
@ -4087,7 +4106,12 @@ int32_t doMergeRowsInSttBlock(SSttBlockReader* pSttBlockReader, STableBlockScanI
SRowKey* pRowKey = &pScanInfo->lastProcKey;
int32_t code = TSDB_CODE_SUCCESS;
while (nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pkSrcSlot, pVerRange)) {
while (1) {
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pkSrcSlot, pVerRange);
if (code || (!hasDataInSttBlock(pScanInfo))) {
return code;
}
SRowKey* pNextKey = getCurrentKeyInSttBlock(pSttBlockReader);
int32_t ret = pkCompEx(pRowKey, pNextKey);

36
source/dnode/vnode/src/tsdb/tsdbSttFileRW.c
View File

@ -61,7 +61,9 @@ int32_t tsdbSttFileReaderOpen(const char *fname, const SSttFileReaderConfig *con
// // open each segment reader
int64_t offset = config->file->size - sizeof(SSttFooter);
ASSERT(offset >= TSDB_FHDR_SIZE);
if (offset < TSDB_FHDR_SIZE) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
int32_t encryptAlgoirthm = config->tsdb->pVnode->config.tsdbCfg.encryptAlgorithm;
char *encryptKey = config->tsdb->pVnode->config.tsdbCfg.encryptKey;
@ -115,7 +117,9 @@ int32_t tsdbSttFileReaderClose(SSttFileReader **reader) {
int32_t tsdbSttFileReadStatisBlk(SSttFileReader *reader, const TStatisBlkArray **statisBlkArray) {
if (!reader->ctx->statisBlkLoaded) {
if (reader->footer->statisBlkPtr->size > 0) {
ASSERT(reader->footer->statisBlkPtr->size % sizeof(SStatisBlk) == 0);
if (reader->footer->statisBlkPtr->size % sizeof(SStatisBlk) != 0) {
return TSDB_CODE_FILE_CORRUPTED;
}
int32_t size = reader->footer->statisBlkPtr->size / sizeof(SStatisBlk);
void *data = taosMemoryMalloc(reader->footer->statisBlkPtr->size);
@ -147,7 +151,9 @@ int32_t tsdbSttFileReadStatisBlk(SSttFileReader *reader, const TStatisBlkArray *
int32_t tsdbSttFileReadTombBlk(SSttFileReader *reader, const TTombBlkArray **tombBlkArray) {
if (!reader->ctx->tombBlkLoaded) {
if (reader->footer->tombBlkPtr->size > 0) {
ASSERT(reader->footer->tombBlkPtr->size % sizeof(STombBlk) == 0);
if (reader->footer->tombBlkPtr->size % sizeof(STombBlk) != 0) {
return TSDB_CODE_FILE_CORRUPTED;
}
int32_t size = reader->footer->tombBlkPtr->size / sizeof(STombBlk);
void *data = taosMemoryMalloc(reader->footer->tombBlkPtr->size);
@ -179,7 +185,9 @@ int32_t tsdbSttFileReadTombBlk(SSttFileReader *reader, const TTombBlkArray **tom
int32_t tsdbSttFileReadSttBlk(SSttFileReader *reader, const TSttBlkArray **sttBlkArray) {
if (!reader->ctx->sttBlkLoaded) {
if (reader->footer->sttBlkPtr->size > 0) {
ASSERT(reader->footer->sttBlkPtr->size % sizeof(SSttBlk) == 0);
if (reader->footer->sttBlkPtr->size % sizeof(SSttBlk) != 0) {
return TSDB_CODE_FILE_CORRUPTED;
}
int32_t size = reader->footer->sttBlkPtr->size / sizeof(SSttBlk);
void *data = taosMemoryMalloc(reader->footer->sttBlkPtr->size);
@ -256,7 +264,9 @@ int32_t tsdbSttFileReadBlockDataByColumn(SSttFileReader *reader, const SSttBlk *
SBufferReader br = BUFFER_READER_INITIALIZER(0, buffer0);
TAOS_CHECK_GOTO(tGetDiskDataHdr(&br, &hdr), &lino, _exit);
ASSERT(hdr.delimiter == TSDB_FILE_DLMT);
if (hdr.delimiter != TSDB_FILE_DLMT) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
// set data container
tBlockDataReset(bData);
@ -266,7 +276,9 @@ int32_t tsdbSttFileReadBlockDataByColumn(SSttFileReader *reader, const SSttBlk *
// key part
TAOS_CHECK_GOTO(tBlockDataDecompressKeyPart(&hdr, &br, bData, assist), &lino, _exit);
ASSERT(br.offset == buffer0->size);
if (br.offset != buffer0->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
bool loadExtra = false;
for (int i = 0; i < ncid; i++) {
@ -376,7 +388,10 @@ int32_t tsdbSttFileReadTombBlock(SSttFileReader *reader, const STombBlk *tombBlk
br.offset += tombBlk->size[i];
}
ASSERT(br.offset == tombBlk->dp->size);
if (br.offset != tombBlk->dp->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(reader->config->tsdb->pVnode), __func__, __FILE__, lino,
@ -444,7 +459,9 @@ int32_t tsdbSttFileReadStatisBlock(SSttFileReader *reader, const SStatisBlk *sta
}
}
ASSERT(br.offset == buffer0->size);
if (br.offset != buffer0->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -814,8 +831,6 @@ _exit:
}
static void tsdbSttFWriterDoClose(SSttFileWriter *writer) {
ASSERT(writer->fd == NULL);
for (int32_t i = 0; i < ARRAY_SIZE(writer->local); ++i) {
tBufferDestroy(writer->local + i);
}
@ -854,7 +869,6 @@ static int32_t tsdbSttFWriterCloseCommit(SSttFileWriter *writer, TFileOpArray *o
tsdbCloseFile(&writer->fd);
ASSERT(writer->file->size > 0);
STFileOp op = (STFileOp){
.optype = TSDB_FOP_CREATE,
.fid = writer->config->fid,

2
source/dnode/vnode/src/tsdb/tsdbUtil2.c
View File

@ -171,7 +171,7 @@ static int32_t tStatisBlockUpdate(STbStatisBlock *block, SRowInfo *row) {
TAOS_CHECK_RETURN(tBufferPutAt(&block->counts, (block->numOfRecords - 1) * sizeof(record.count), &record.count,
sizeof(record.count)));
} else {
ASSERT(0);
return TSDB_CODE_INVALID_PARA;
}
return 0;

15
source/dnode/vnode/src/vnd/vnodeAsync.c
View File

@ -165,9 +165,7 @@ static int32_t vnodeAsyncTaskDone(SVAsync *async, SVATask *task) {
}
ret = vHashDrop(async->taskTable, task);
if (ret != 0) {
ASSERT(0);
}
TAOS_UNUSED(ret);
async->numTasks--;
if (task->numWait == 0) {
@ -403,7 +401,6 @@ static int32_t vnodeAsyncDestroy(SVAsync **async) {
}
(void)taosThreadJoin((*async)->workers[i].thread, NULL);
ASSERT((*async)->workers[i].state == EVA_WORKER_STATE_STOP);
(*async)->workers[i].state = EVA_WORKER_STATE_UINIT;
}
@ -413,18 +410,11 @@ static int32_t vnodeAsyncDestroy(SVAsync **async) {
channel->prev->next = channel->next;
int32_t ret = vHashDrop((*async)->channelTable, channel);
if (ret) {
ASSERT(0);
}
TAOS_UNUSED(ret);
(*async)->numChannels--;
taosMemoryFree(channel);
}
ASSERT((*async)->numLaunchWorkers == 0);
ASSERT((*async)->numIdleWorkers == 0);
ASSERT((*async)->numChannels == 0);
ASSERT((*async)->numTasks == 0);
(void)taosThreadMutexDestroy(&(*async)->mutex);
(void)taosThreadCondDestroy(&(*async)->hasTask);
@ -438,7 +428,6 @@ static int32_t vnodeAsyncDestroy(SVAsync **async) {
static int32_t vnodeAsyncLaunchWorker(SVAsync *async) {
for (int32_t i = 0; i < async->numWorkers; i++) {
ASSERT(async->workers[i].state != EVA_WORKER_STATE_IDLE);
if (async->workers[i].state == EVA_WORKER_STATE_ACTIVE) {
continue;
} else if (async->workers[i].state == EVA_WORKER_STATE_STOP) {

4
source/dnode/vnode/src/vnd/vnodeCfg.c
View File

@ -375,11 +375,11 @@ int vnodeDecodeConfig(const SJson *pJson, void *pObj) {
}
tjsonGetNumberValue(pJson, "s3ChunkSize", pCfg->s3ChunkSize, code);
if (code < 0) {
if (code < 0 || pCfg->s3ChunkSize < TSDB_MIN_S3_CHUNK_SIZE) {
pCfg->s3ChunkSize = TSDB_DEFAULT_S3_CHUNK_SIZE;
}
tjsonGetNumberValue(pJson, "s3KeepLocal", pCfg->s3KeepLocal, code);
if (code < 0) {
if (code < 0 || pCfg->s3KeepLocal < TSDB_MIN_S3_KEEP_LOCAL) {
pCfg->s3KeepLocal = TSDB_DEFAULT_S3_KEEP_LOCAL;
}
tjsonGetNumberValue(pJson, "s3Compact", pCfg->s3Compact, code);

3
source/dnode/vnode/src/vnd/vnodeCommit.c
View File

@ -302,7 +302,6 @@ static int32_t vnodePrepareCommit(SVnode *pVnode, SCommitInfo *pInfo) {
TSDB_CHECK_CODE(code, lino, _exit);
(void)taosThreadMutexLock(&pVnode->mutex);
ASSERT(pVnode->onCommit == NULL);
pVnode->onCommit = pVnode->inUse;
pVnode->inUse = NULL;
(void)taosThreadMutexUnlock(&pVnode->mutex);
@ -339,7 +338,7 @@ static void vnodeReturnBufPool(SVnode *pVnode) {
pVnode->recycleTail = pPool;
}
} else {
ASSERT(0);
vError("vgId:%d, buffer pool %p of id %d nRef:%d", TD_VID(pVnode), pPool, pPool->id, nRef);
}
(void)taosThreadMutexUnlock(&pVnode->mutex);

1
source/dnode/vnode/src/vnd/vnodeHash.c
View File

@ -77,7 +77,6 @@ int32_t vHashDestroy(SVHashTable** ht) {
}
if (*ht) {
ASSERT((*ht)->numEntries == 0);
taosMemoryFree((*ht)->buckets);
taosMemoryFree(*ht);
(*ht) = NULL;

4
source/dnode/vnode/src/vnd/vnodeOpen.c
View File

@ -558,7 +558,9 @@ void vnodeClose(SVnode *pVnode) {
// start the sync timer after the queue is ready
int32_t vnodeStart(SVnode *pVnode) {
ASSERT(pVnode);
if (pVnode == NULL) {
return TSDB_CODE_INVALID_PARA;
}
return vnodeSyncStart(pVnode);
}

4
source/libs/executor/src/dataDispatcher.c
View File

@ -318,6 +318,7 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pD
dispatcher->sink.fGetCacheSize = getCacheSize;
dispatcher->pManager = pManager;
pManager = NULL;
dispatcher->pSchema = pDataSink->pInputDataBlockDesc;
dispatcher->status = DS_BUF_EMPTY;
dispatcher->queryEnd = false;
@ -336,6 +337,9 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pD
return TSDB_CODE_SUCCESS;
_return:
taosMemoryFree(pManager);
if (dispatcher) {
dsDestroyDataSinker(dispatcher);
}

2
source/libs/executor/src/dynqueryctrloperator.c
View File

@ -320,7 +320,7 @@ static int32_t buildMergeJoinOperatorParam(SOperatorParam** ppRes, bool initPara
return code;
}
(*ppRes)->pChildren = taosArrayInit(2, POINTER_BYTES);
if (NULL == *ppRes) {
if (NULL == (*ppRes)->pChildren) {
code = terrno;
freeOperatorParam(pChild0, OP_GET_PARAM);
freeOperatorParam(pChild1, OP_GET_PARAM);

5
source/libs/executor/src/exchangeoperator.c
View File

@ -225,7 +225,10 @@ static SSDataBlock* doLoadRemoteDataImpl(SOperatorInfo* pOperator) {
} else {
concurrentlyLoadRemoteDataImpl(pOperator, pExchangeInfo, pTaskInfo);
}
if (TSDB_CODE_SUCCESS != pOperator->pTaskInfo->code) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
T_LONG_JMP(pTaskInfo->env, pOperator->pTaskInfo->code);
}
if (taosArrayGetSize(pExchangeInfo->pResultBlockList) == 0) {
return NULL;
} else {

6
source/libs/executor/src/executor.c
View File

@ -31,7 +31,7 @@ int32_t exchangeObjRefPool = -1;
static void cleanupRefPool() {
int32_t ref = atomic_val_compare_exchange_32(&exchangeObjRefPool, exchangeObjRefPool, 0);
(void)taosCloseRef(ref);
taosCloseRef(ref);
}
static void initRefPool() {
@ -667,7 +667,7 @@ int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bo
if (isTaskKilled(pTaskInfo)) {
atomic_store_64(&pTaskInfo->owner, 0);
qDebug("%s already killed, abort", GET_TASKID(pTaskInfo));
return TSDB_CODE_SUCCESS;
return pTaskInfo->code;
}
// error occurs, record the error code and return to client
@ -790,7 +790,7 @@ int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pRes, uint64_t* useconds) {
qDebug("%s already killed, abort", GET_TASKID(pTaskInfo));
taosRUnLockLatch(&pTaskInfo->lock);
return TSDB_CODE_SUCCESS;
return pTaskInfo->code;
}
if (pTaskInfo->owner != 0) {

3
source/libs/executor/src/mergejoinoperator.c
View File

@ -1746,6 +1746,9 @@ void destroyGrpArray(void* ppArray) {
}
void destroyMergeJoinTableCtx(SMJoinTableCtx* pTable) {
if (NULL == pTable) {
return;
}
mJoinDestroyCreatedBlks(pTable->createdBlks);
taosArrayDestroy(pTable->createdBlks);
tSimpleHashCleanup(pTable->pGrpHash);

7
source/libs/executor/src/scanoperator.c
View File

@ -464,15 +464,10 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableSca
SSDataBlock* p = NULL;
code = pAPI->tsdReader.tsdReaderRetrieveDataBlock(pTableScanInfo->dataReader, &p, NULL);
if (p == NULL || code != TSDB_CODE_SUCCESS) {
if (p == NULL || code != TSDB_CODE_SUCCESS || p != pBlock) {
return code;
}
if(p != pBlock) {
qError("[loadDataBlock] p != pBlock");
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
code = doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, pBlock->info.rows);
if (code) {
return code;

7
source/libs/executor/src/streamtimewindowoperator.c
View File

@ -438,6 +438,11 @@ void destroyFlusedPos(void* pRes) {
}
}
void destroyFlusedppPos(void* ppRes) {
void *pRes = *(void **)ppRes;
destroyFlusedPos(pRes);
}
void clearGroupResInfo(SGroupResInfo* pGroupResInfo) {
if (pGroupResInfo->freeItem) {
int32_t size = taosArrayGetSize(pGroupResInfo->pRows);
@ -1920,6 +1925,7 @@ int32_t createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiN
code = initAggSup(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str,
pInfo->pState, &pTaskInfo->storageAPI.functionStore);
QUERY_CHECK_CODE(code, lino, _error);
tSimpleHashSetFreeFp(pInfo->aggSup.pResultRowHashTable, destroyFlusedppPos);
code = initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window);
QUERY_CHECK_CODE(code, lino, _error);
@ -5286,6 +5292,7 @@ int32_t createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiNode*
code = initAggSup(pSup, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str, pInfo->pState,
&pTaskInfo->storageAPI.functionStore);
QUERY_CHECK_CODE(code, lino, _error);
tSimpleHashSetFreeFp(pInfo->aggSup.pResultRowHashTable, destroyFlusedppPos);
if (pIntervalPhyNode->window.pExprs != NULL) {
int32_t numOfScalar = 0;

2
source/libs/function/inc/thistogram.h
View File

@ -59,7 +59,7 @@ int32_t tHistogramCreate(int32_t numOfEntries, SHistogramInfo** pHisto);
SHistogramInfo* tHistogramCreateFrom(void* pBuf, int32_t numOfBins);
int32_t tHistogramAdd(SHistogramInfo** pHisto, double val);
int64_t tHistogramSum(SHistogramInfo* pHisto, double v);
int32_t tHistogramSum(SHistogramInfo* pHisto, double v, int64_t *res);
int32_t tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num, double** pVal);
int32_t tHistogramMerge(SHistogramInfo* pHisto1, SHistogramInfo* pHisto2, int32_t numOfEntries,

2
source/libs/function/inc/tpercentile.h
View File

@ -47,7 +47,7 @@ typedef struct tMemBucketSlot {
} tMemBucketSlot;
struct tMemBucket;
typedef int32_t (*__perc_hash_func_t)(struct tMemBucket *pBucket, const void *value);
typedef int32_t (*__perc_hash_func_t)(struct tMemBucket *pBucket, const void *value, int32_t *index);
typedef struct tMemBucket {
int16_t numOfSlots;

4
source/libs/function/src/builtinsimpl.c
View File

@ -4018,7 +4018,9 @@ int32_t saveTupleData(SqlFunctionCtx* pCtx, int32_t rowIndex, const SSDataBlock*
if (NULL == pColInfo) {
return TSDB_CODE_OUT_OF_RANGE;
}
ASSERT(pColInfo->info.type == TSDB_DATA_TYPE_TIMESTAMP);
if (pColInfo->info.type != TSDB_DATA_TYPE_TIMESTAMP) {
return TSDB_CODE_FUNC_FUNTION_PARA_TYPE;
}
key.groupId = pSrcBlock->info.id.groupId;
key.ts = *(int64_t*)colDataGetData(pColInfo, rowIndex);
}

89
source/libs/function/src/thistogram.c
View File

@ -14,6 +14,7 @@
*/
#include "os.h"
#include "query.h"
#include "taosdef.h"
#include "thistogram.h"
#include "tlosertree.h"
@ -81,9 +82,9 @@ int32_t tHistogramAdd(SHistogramInfo** pHisto, double val) {
#if defined(USE_ARRAYLIST)
int32_t idx = histoBinarySearch((*pHisto)->elems, (*pHisto)->numOfEntries, val);
if (ASSERTS(idx >= 0 && idx <= (*pHisto)->maxEntries && (*pHisto)->elems != NULL, "tHistogramAdd Error, idx:%d, maxEntries:%d, elems:%p",
idx, (*pHisto)->maxEntries, (*pHisto)->elems)) {
return TSDB_CODE_FAILED;
if (idx < 0 || idx > (*pHisto)->maxEntries || (*pHisto)->elems == NULL) {
qError("tHistogramAdd Error, idx:%d, maxEntries:%d, elems:%p", idx, (*pHisto)->maxEntries, (*pHisto)->elems);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
if ((*pHisto)->elems[idx].val == val && idx >= 0) {
@ -95,21 +96,19 @@ int32_t tHistogramAdd(SHistogramInfo** pHisto, double val) {
} else { /* insert a new slot */
if ((*pHisto)->numOfElems >= 1 && idx < (*pHisto)->numOfEntries) {
if (idx > 0) {
if (ASSERTS((*pHisto)->elems[idx - 1].val <= val, "tHistogramAdd Error, elems[%d].val:%lf, val:%lf",
idx - 1, (*pHisto)->elems[idx - 1].val, val)) {
return TSDB_CODE_FAILED;
if ((*pHisto)->elems[idx - 1].val > val) {
qError("tHistogramAdd Error, elems[%d].val:%lf, val:%lf", idx - 1, (*pHisto)->elems[idx - 1].val, val);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
} else {
if (ASSERTS((*pHisto)->elems[idx].val > val, "tHistogramAdd Error, elems[%d].val:%lf, val:%lf",
idx, (*pHisto)->elems[idx].val, val)) {
return TSDB_CODE_FAILED;
if ((*pHisto)->elems[idx].val <= val) {
qError("tHistogramAdd Error, elems[%d].val:%lf, val:%lf", idx, (*pHisto)->elems[idx].val, val);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
}
} else if ((*pHisto)->numOfElems > 0) {
if (ASSERTS((*pHisto)->elems[(*pHisto)->numOfEntries].val <= val, "tHistogramAdd Error, elems[%d].val:%lf, val:%lf",
(*pHisto)->numOfEntries, (*pHisto)->elems[idx].val, val)) {
return TSDB_CODE_FAILED;
}
} else if ((*pHisto)->numOfElems > 0 && (*pHisto)->elems[(*pHisto)->numOfEntries].val > val) {
qError("tHistogramAdd Error, elems[%d].val:%lf, val:%lf", (*pHisto)->numOfEntries, (*pHisto)->elems[idx].val, val);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
code = histogramCreateBin(*pHisto, idx, val);
@ -225,9 +224,9 @@ int32_t tHistogramAdd(SHistogramInfo** pHisto, double val) {
tSkipListNode* pNext = pNode->pForward[0];
SHistBin* pNextEntry = (SHistBin*)pNext->pData;
if (ASSERTS(pNextEntry->val - pEntry->val == pEntry->delta, "tHistogramAdd Error, pNextEntry->val:%lf, pEntry->val:%lf, pEntry->delta:%lf",
pNextEntry->val, pEntry->val, pEntry->delta)) {
return -1;
if (pNextEntry->val - pEntry->val != pEntry->delta) {
qError("tHistogramAdd Error, pNextEntry->val:%lf, pEntry->val:%lf, pEntry->delta:%lf", pNextEntry->val, pEntry->val, pEntry->delta);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
double newVal = (pEntry->val * pEntry->num + pNextEntry->val * pNextEntry->num) / (pEntry->num + pNextEntry->num);
@ -278,8 +277,9 @@ int32_t tHistogramAdd(SHistogramInfo** pHisto, double val) {
} else {
SHistBin* pEntry = (SHistBin*)pResNode->pData;
if (ASSERTS(pEntry->val == val, "tHistogramAdd Error, pEntry->val:%lf, val:%lf")) {
return -1;
if (pEntry->val != val) {
qError("tHistogramAdd Error, pEntry->val:%lf, val:%lf", pEntry->val, val);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
pEntry->num += 1;
}
@ -356,9 +356,9 @@ int32_t histogramCreateBin(SHistogramInfo* pHisto, int32_t index, double val) {
(void)memmove(&pHisto->elems[index + 1], &pHisto->elems[index], sizeof(SHistBin) * remain);
}
if (ASSERTS(index >= 0 && index <= pHisto->maxEntries, "histogramCreateBin Error, index:%d, maxEntries:%d",
index, pHisto->maxEntries)) {
return TSDB_CODE_FAILED;
if (index < 0 || index > pHisto->maxEntries) {
qError("histogramCreateBin Error, index:%d, maxEntries:%d", index, pHisto->maxEntries);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
pHisto->elems[index].num = 1;
@ -373,9 +373,9 @@ int32_t histogramCreateBin(SHistogramInfo* pHisto, int32_t index, double val) {
pHisto->elems[pHisto->maxEntries].num = 0;
}
#endif
if (ASSERTS(pHisto->numOfEntries <= pHisto->maxEntries, "histogramCreateBin Error, numOfEntries:%d, maxEntries:%d",
pHisto->numOfEntries, pHisto->maxEntries)) {
return TSDB_CODE_FAILED;
if (pHisto->numOfEntries > pHisto->maxEntries) {
qError("histogramCreateBin Error, numOfEntries:%d, maxEntries:%d", pHisto->numOfEntries, pHisto->maxEntries);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
return TSDB_CODE_SUCCESS;
@ -411,8 +411,9 @@ void tHistogramPrint(SHistogramInfo* pHisto) {
* Estimated number of points in the interval (−inf,b].
* @param pHisto
* @param v
* @param res
*/
int64_t tHistogramSum(SHistogramInfo* pHisto, double v) {
int32_t tHistogramSum(SHistogramInfo* pHisto, double v, int64_t *res) {
#if defined(USE_ARRAYLIST)
int32_t slotIdx = histoBinarySearch(pHisto->elems, pHisto->numOfEntries, v);
if (pHisto->elems[slotIdx].val != v) {
@ -420,14 +421,18 @@ int64_t tHistogramSum(SHistogramInfo* pHisto, double v) {
if (slotIdx < 0) {
slotIdx = 0;
ASSERTS(v <= pHisto->elems[slotIdx].val, "tHistogramSum Error, elems[%d].val:%lf, v:%lf",
slotIdx, pHisto->elems[slotIdx].val, v);
if (v > pHisto->elems[slotIdx].val) {
qError("tHistogramSum Error, elems[%d].val:%lf, v:%lf", slotIdx, pHisto->elems[slotIdx].val, v);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
} else {
ASSERTS(v >= pHisto->elems[slotIdx].val, "tHistogramSum Error, elems[%d].val:%lf, v:%lf",
slotIdx, pHisto->elems[slotIdx].val, v);
if (slotIdx + 1 < pHisto->numOfEntries) {
ASSERTS(v < pHisto->elems[slotIdx + 1].val, "tHistogramSum Error, elems[%d].val:%lf, v:%lf",
slotIdx + 1, pHisto->elems[slotIdx + 1].val, v);
if (v < pHisto->elems[slotIdx].val) {
qError("tHistogramSum Error, elems[%d].val:%lf, v:%lf", slotIdx, pHisto->elems[slotIdx].val, v);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
if (slotIdx + 1 < pHisto->numOfEntries && v >= pHisto->elems[slotIdx + 1].val) {
qError("tHistogramSum Error, elems[%d].val:%lf, v:%lf", slotIdx + 1, pHisto->elems[slotIdx + 1].val, v);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
}
}
@ -447,8 +452,9 @@ int64_t tHistogramSum(SHistogramInfo* pHisto, double v) {
s1 = s1 + m1 / 2;
return (int64_t)s1;
*res = (int64_t)s1;
#endif
return TSDB_CODE_SUCCESS;
}
int32_t tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num, double** pVal) {
@ -484,9 +490,11 @@ int32_t tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num, do
j += 1;
}
ASSERTS(total <= numOfElem && total + pHisto->elems[j + 1].num > numOfElem,
"tHistogramUniform Error, total:%ld, numOfElem:%ld, elems[%d].num:%ld",
total, (int64_t)numOfElem, j + 1, pHisto->elems[j + 1].num);
if (total > numOfElem || total + pHisto->elems[j + 1].num <= numOfElem) {
qError("tHistogramUniform Error, total:%d, numOfElem:%d, elems[%d].num:%d",
(int32_t)total, (int32_t)numOfElem, j + 1, (int32_t)pHisto->elems[j + 1].num);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
double delta = numOfElem - total;
if (fabs(delta) < FLT_EPSILON) {
@ -545,9 +553,10 @@ int32_t tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num, do
j += 1;
}
ASSERTS(total <= numOfElem && total + pEntry->num > numOfElem,
"tHistogramUniform Error, total:%d, numOfElem:%d, pEntry->num:%d",
total, numOfElem, pEntry->num);
if (total > numOfElem || total + pEntry->num <= numOfElem) {
qError("tHistogramUniform Error, total:%d, numOfElem:%d, pEntry->num:%d", (int32_t)total, (int32_t)numOfElem, (int32_t)pEntry->num);
return TSDB_CODE_FUNC_HISTOGRAM_ERROR;
}
double delta = numOfElem - total;
if (fabs(delta) < FLT_EPSILON) {

126
source/libs/function/src/tpercentile.c
View File

@ -13,6 +13,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "query.h"
#include "taoserror.h"
#include "tcompare.h"
#include "tglobal.h"
@ -107,7 +108,10 @@ static void resetPosInfo(SSlotInfo *pInfo) {
}
int32_t findOnlyResult(tMemBucket *pMemBucket, double *result) {
ASSERT(pMemBucket->total == 1);
if (pMemBucket->total != 1) {
qError("MemBucket:%p, total:%d, but only one element is expected", pMemBucket, pMemBucket->total);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
terrno = 0;
for (int32_t i = 0; i < pMemBucket->numOfSlots; ++i) {
@ -120,7 +124,10 @@ int32_t findOnlyResult(tMemBucket *pMemBucket, double *result) {
SArray **pList = taosHashGet(pMemBucket->groupPagesMap, &groupId, sizeof(groupId));
if (pList != NULL) {
SArray *list = *pList;
ASSERT(list->size == 1);
if (list->size != 1) {
qError("list:%p, total list size:%zu, but only one element is expected", list, list->size);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
int32_t *pageId = taosArrayGet(list, 0);
if (NULL == pageId) {
@ -130,7 +137,11 @@ int32_t findOnlyResult(tMemBucket *pMemBucket, double *result) {
if (pPage == NULL) {
return terrno;
}
ASSERT(pPage->num == 1);
if (pPage->num != 1) {
qError("page:%p, total num:%d, but only one element is expected", pPage, pPage->num);
releaseBufPage(pMemBucket->pBuffer, pPage);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
GET_TYPED_DATA(*result, double, pMemBucket->type, pPage->data);
return TSDB_CODE_SUCCESS;
@ -141,64 +152,69 @@ int32_t findOnlyResult(tMemBucket *pMemBucket, double *result) {
return TSDB_CODE_SUCCESS;
}
int32_t tBucketIntHash(tMemBucket *pBucket, const void *value) {
int32_t tBucketIntHash(tMemBucket *pBucket, const void *value, int32_t *index) {
int64_t v = 0;
GET_TYPED_DATA(v, int64_t, pBucket->type, value);
int32_t index = -1;
*index = -1;
if (v > pBucket->range.dMaxVal || v < pBucket->range.dMinVal) {
return index;
return TSDB_CODE_SUCCESS;
}
// divide the value range into 1024 buckets
uint64_t span = pBucket->range.dMaxVal - pBucket->range.dMinVal;
if (span < pBucket->numOfSlots) {
int64_t delta = v - pBucket->range.dMinVal;
index = (delta % pBucket->numOfSlots);
*index = (delta % pBucket->numOfSlots);
} else {
double slotSpan = ((double)span) / pBucket->numOfSlots;
uint64_t delta = (uint64_t)(v - pBucket->range.dMinVal);
index = delta / slotSpan;
if (v == pBucket->range.dMaxVal || index == pBucket->numOfSlots) {
index -= 1;
*index = delta / slotSpan;
if (v == pBucket->range.dMaxVal || *index == pBucket->numOfSlots) {
*index -= 1;
}
}
ASSERTS(index >= 0 && index < pBucket->numOfSlots, "tBucketIntHash Error, index:%d, numOfSlots:%d",
index, pBucket->numOfSlots);
return index;
if (*index < 0 || *index >= pBucket->numOfSlots) {
qError("tBucketIntHash Error, index:%d, numOfSlots:%d", *index, pBucket->numOfSlots);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
return TSDB_CODE_SUCCESS;
}
int32_t tBucketUintHash(tMemBucket *pBucket, const void *value) {
int32_t tBucketUintHash(tMemBucket *pBucket, const void *value, int32_t *index) {
int64_t v = 0;
GET_TYPED_DATA(v, uint64_t, pBucket->type, value);
int32_t index = -1;
*index = -1;
if (v > pBucket->range.u64MaxVal || v < pBucket->range.u64MinVal) {
return index;
return TSDB_CODE_SUCCESS;
}
// divide the value range into 1024 buckets
uint64_t span = pBucket->range.u64MaxVal - pBucket->range.u64MinVal;
if (span < pBucket->numOfSlots) {
int64_t delta = v - pBucket->range.u64MinVal;
index = (int32_t)(delta % pBucket->numOfSlots);
*index = (int32_t)(delta % pBucket->numOfSlots);
} else {
double slotSpan = (double)span / pBucket->numOfSlots;
index = (int32_t)((v - pBucket->range.u64MinVal) / slotSpan);
*index = (int32_t)((v - pBucket->range.u64MinVal) / slotSpan);
if (v == pBucket->range.u64MaxVal) {
index -= 1;
*index -= 1;
}
}
ASSERT(index >= 0 && index < pBucket->numOfSlots);
return index;
if (*index < 0 || *index >= pBucket->numOfSlots) {
qError("tBucketUintHash Error, index:%d, numOfSlots:%d", *index, pBucket->numOfSlots);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
return TSDB_CODE_SUCCESS;
}
int32_t tBucketDoubleHash(tMemBucket *pBucket, const void *value) {
int32_t tBucketDoubleHash(tMemBucket *pBucket, const void *value, int32_t *index) {
double v = 0;
if (pBucket->type == TSDB_DATA_TYPE_FLOAT) {
v = GET_FLOAT_VAL(value);
@ -206,27 +222,30 @@ int32_t tBucketDoubleHash(tMemBucket *pBucket, const void *value) {
v = GET_DOUBLE_VAL(value);
}
int32_t index = -1;
*index = -1;
if (v > pBucket->range.dMaxVal || v < pBucket->range.dMinVal) {
return index;
return TSDB_CODE_SUCCESS;
}
// divide a range of [dMinVal, dMaxVal] into 1024 buckets
double span = pBucket->range.dMaxVal - pBucket->range.dMinVal;
if (span < pBucket->numOfSlots) {
int32_t delta = (int32_t)(v - pBucket->range.dMinVal);
index = (delta % pBucket->numOfSlots);
*index = (delta % pBucket->numOfSlots);
} else {
double slotSpan = span / pBucket->numOfSlots;
index = (int32_t)((v - pBucket->range.dMinVal) / slotSpan);
*index = (int32_t)((v - pBucket->range.dMinVal) / slotSpan);
if (v == pBucket->range.dMaxVal) {
index -= 1;
*index -= 1;
}
}
ASSERT(index >= 0 && index < pBucket->numOfSlots);
return index;
if (*index < 0 || *index >= pBucket->numOfSlots) {
qError("tBucketDoubleHash Error, index:%d, numOfSlots:%d", *index, pBucket->numOfSlots);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
return TSDB_CODE_SUCCESS;
}
static __perc_hash_func_t getHashFunc(int32_t type) {
@ -333,7 +352,7 @@ void tMemBucketDestroy(tMemBucket *pBucket) {
taosMemoryFreeClear(pBucket);
}
void tMemBucketUpdateBoundingBox(MinMaxEntry *r, const char *data, int32_t dataType) {
int32_t tMemBucketUpdateBoundingBox(MinMaxEntry *r, const char *data, int32_t dataType) {
if (IS_SIGNED_NUMERIC_TYPE(dataType)) {
int64_t v = 0;
GET_TYPED_DATA(v, int64_t, dataType, data);
@ -368,8 +387,10 @@ void tMemBucketUpdateBoundingBox(MinMaxEntry *r, const char *data, int32_t dataT
r->dMaxVal = v;
}
} else {
ASSERT(0);
qError("tMemBucketUpdateBoundingBox Error, invalid data type:%d", dataType);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
return TSDB_CODE_SUCCESS;
}
/*
@ -378,9 +399,14 @@ void tMemBucketUpdateBoundingBox(MinMaxEntry *r, const char *data, int32_t dataT
int32_t tMemBucketPut(tMemBucket *pBucket, const void *data, size_t size) {
int32_t count = 0;
int32_t bytes = pBucket->bytes;
int32_t code = TSDB_CODE_SUCCESS;
for (int32_t i = 0; i < size; ++i) {
char *d = (char *)data + i * bytes;
int32_t index = (pBucket->hashFunc)(pBucket, d);
int32_t index = -1;
code = (pBucket->hashFunc)(pBucket, d, &index);
if (TSDB_CODE_SUCCESS != code) {
return code;
}
if (index < 0) {
continue;
}
@ -388,7 +414,10 @@ int32_t tMemBucketPut(tMemBucket *pBucket, const void *data, size_t size) {
count += 1;
tMemBucketSlot *pSlot = &pBucket->pSlots[index];
tMemBucketUpdateBoundingBox(&pSlot->range, d, pBucket->type);
code = tMemBucketUpdateBoundingBox(&pSlot->range, d, pBucket->type);
if (TSDB_CODE_SUCCESS != code) {
return code;
}
// ensure available memory pages to allocate
int32_t groupId = getGroupId(pBucket->numOfSlots, index, pBucket->times);
@ -396,7 +425,11 @@ int32_t tMemBucketPut(tMemBucket *pBucket, const void *data, size_t size) {
if (pSlot->info.data == NULL || pSlot->info.data->num >= pBucket->elemPerPage) {
if (pSlot->info.data != NULL) {
ASSERT(pSlot->info.data->num >= pBucket->elemPerPage && pSlot->info.size > 0);
if (pSlot->info.data->num < pBucket->elemPerPage || pSlot->info.size <= 0) {
qError("tMemBucketPut failed since wrong pSLot info dataNum : %d, size : %d",
pSlot->info.data->num, pSlot->info.size);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
// keep the pointer in memory
setBufPageDirty(pSlot->info.data, true);
@ -411,7 +444,7 @@ int32_t tMemBucketPut(tMemBucket *pBucket, const void *data, size_t size) {
if (NULL == pPageIdList) {
return terrno;
}
int32_t code = taosHashPut(pBucket->groupPagesMap, &groupId, sizeof(groupId), &pPageIdList, POINTER_BYTES);
code = taosHashPut(pBucket->groupPagesMap, &groupId, sizeof(groupId), &pPageIdList, POINTER_BYTES);
if (TSDB_CODE_SUCCESS != code) {
taosArrayDestroy(pPageIdList);
return code;
@ -449,21 +482,23 @@ int32_t tMemBucketPut(tMemBucket *pBucket, const void *data, size_t size) {
* j is the last slot of current segment, we need to get the first
* slot of the next segment.
*/
static MinMaxEntry getMinMaxEntryOfNextSlotWithData(tMemBucket *pMemBucket, int32_t slotIdx) {
static int32_t getMinMaxEntryOfNextSlotWithData(tMemBucket *pMemBucket, int32_t slotIdx, MinMaxEntry *next) {
int32_t j = slotIdx + 1;
while (j < pMemBucket->numOfSlots && (pMemBucket->pSlots[j].info.size == 0)) {
++j;
}
ASSERT(j < pMemBucket->numOfSlots);
return pMemBucket->pSlots[j].range;
if (j >= pMemBucket->numOfSlots) {
qError("getMinMaxEntryOfNextSlotWithData can not get valid slot, start with slotIdx:%d", slotIdx);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
*next = pMemBucket->pSlots[j].range;
return TSDB_CODE_SUCCESS;
}
static bool isIdenticalData(tMemBucket *pMemBucket, int32_t index);
static double getIdenticalDataVal(tMemBucket *pMemBucket, int32_t slotIndex) {
ASSERT(isIdenticalData(pMemBucket, slotIndex));
tMemBucketSlot *pSlot = &pMemBucket->pSlots[slotIndex];
double finalResult = 0.0;
@ -494,7 +529,11 @@ int32_t getPercentileImpl(tMemBucket *pMemBucket, int32_t count, double fraction
* now, we need to find the minimum value of the next slot for interpolating the percentile value
* j is the last slot of current segment, we need to get the first slot of the next segment.
*/
MinMaxEntry next = getMinMaxEntryOfNextSlotWithData(pMemBucket, i);
MinMaxEntry next;
int32_t code = getMinMaxEntryOfNextSlotWithData(pMemBucket, i, &next);
if (TSDB_CODE_SUCCESS != code) {
return code;
}
double maxOfThisSlot = 0;
double minOfNextSlot = 0;
@ -509,7 +548,10 @@ int32_t getPercentileImpl(tMemBucket *pMemBucket, int32_t count, double fraction
minOfNextSlot = (double)next.dMinVal;
}
ASSERT(minOfNextSlot > maxOfThisSlot);
if (minOfNextSlot <= maxOfThisSlot) {
qError("getPercentileImpl get minOfNextSlot : %f less equal than maxOfThisSlot : %f", minOfNextSlot, maxOfThisSlot);
return TSDB_CODE_FUNC_PERCENTILE_ERROR;
}
*result = (1 - fraction) * maxOfThisSlot + fraction * minOfNextSlot;
return TSDB_CODE_SUCCESS;

6
source/libs/geometry/src/geomFunc.c
View File

@ -131,12 +131,12 @@ _exit:
int32_t executeMakePointFunc(SColumnInfoData *pInputData[], int32_t iLeft, int32_t iRight,
SColumnInfoData *pOutputData) {
int32_t code = TSDB_CODE_FAILED;
unsigned char *output = NULL;
_getDoubleValue_fn_t getDoubleValueFn[2];
getDoubleValueFn[0] = getVectorDoubleValueFn(pInputData[0]->info.type);
getDoubleValueFn[1] = getVectorDoubleValueFn(pInputData[1]->info.type);
TAOS_CHECK_GOTO(getVectorDoubleValueFn(pInputData[0]->info.type, &getDoubleValueFn[0]), NULL, _exit);
TAOS_CHECK_GOTO(getVectorDoubleValueFn(pInputData[1]->info.type, &getDoubleValueFn[1]), NULL, _exit);
unsigned char *output = NULL;
double leftRes = 0;
double rightRes = 0;

2
source/libs/index/src/index.c
View File

@ -74,7 +74,7 @@ void indexCleanup() {
// refacto later
taosCleanUpScheduler(indexQhandle);
taosMemoryFreeClear(indexQhandle);
(void)taosCloseRef(indexRefMgt);
taosCloseRef(indexRefMgt);
}
typedef struct SIdxColInfo {

2
source/libs/nodes/src/nodesUtilFuncs.c
View File

@ -238,7 +238,7 @@ void nodesDestroyAllocatorSet() {
(void)taosRemoveRef(g_allocatorReqRefPool, refId);
pAllocator = taosIterateRef(g_allocatorReqRefPool, refId);
}
(void)taosCloseRef(g_allocatorReqRefPool);
taosCloseRef(g_allocatorReqRefPool);
}
}

56
source/libs/parser/src/parTranslater.c
View File

@ -12628,6 +12628,27 @@ static int32_t createParOperatorNode(EOperatorType opType, const char* pLeftCol,
return TSDB_CODE_SUCCESS;
}
static int32_t createIsOperatorNode(EOperatorType opType, const char* pColName, SNode** pOp) {
SOperatorNode* pOper = NULL;
int32_t code = nodesMakeNode(QUERY_NODE_OPERATOR, (SNode**)&pOper);
if (NULL == pOper) {
return code;
}
pOper->opType = opType;
code = nodesMakeNode(QUERY_NODE_COLUMN, (SNode**)&pOper->pLeft);
if (TSDB_CODE_SUCCESS != code) {
nodesDestroyNode((SNode*)pOper);
return code;
}
pOper->pRight = NULL;
snprintf(((SColumnNode*)pOper->pLeft)->colName, sizeof(((SColumnNode*)pOper->pLeft)->colName), "%s", pColName);
*pOp = (SNode*)pOper;
return TSDB_CODE_SUCCESS;
}
static const char* getTbNameColName(ENodeType type) {
const char* colName;
switch (type) {
@ -15190,23 +15211,42 @@ static int32_t rewriteShowAliveStmt(STranslateContext* pCxt, SQuery* pQuery) {
return code;
}
// pSubSelect, pTemp1, pTempVal need to free
pThen = NULL;
code = nodesMakeValueNodeFromInt32(1, &pThen);
SNode* pCondIsNULL = NULL;
code = createIsOperatorNode(OP_TYPE_IS_NULL, pSumColAlias, &pCondIsNULL);
if (TSDB_CODE_SUCCESS != code) {
nodesDestroyNode((SNode*)pSubSelect);
nodesDestroyNode(pTemp1);
nodesDestroyNode(pTempVal);
return code;
}
// pSubSelect, pTemp1, pThen, pTempVal need to free
pWhenThen = NULL;
code = createParWhenThenNode(pTemp1, pThen, &pWhenThen);
SNode* pCondFull1 = NULL;
code = createLogicCondNode(&pTemp1, &pCondIsNULL, &pCondFull1, LOGIC_COND_TYPE_OR);
if (TSDB_CODE_SUCCESS != code) {
nodesDestroyNode((SNode*)pSubSelect);
nodesDestroyNode(pTemp1);
nodesDestroyNode(pTempVal);
nodesDestroyNode(pCondIsNULL);
return code;
}
// pSubSelect, pCondFull1, pTempVal need to free
pThen = NULL;
code = nodesMakeValueNodeFromInt32(1, &pThen);
if (TSDB_CODE_SUCCESS != code) {
nodesDestroyNode((SNode*)pSubSelect);
nodesDestroyNode(pCondFull1);
nodesDestroyNode(pTempVal);
return code;
}
// pSubSelect, pCondFull1, pThen, pTempVal need to free
pWhenThen = NULL;
code = createParWhenThenNode(pCondFull1, pThen, &pWhenThen);
if (TSDB_CODE_SUCCESS != code) {
nodesDestroyNode((SNode*)pSubSelect);
nodesDestroyNode(pCondFull1);
nodesDestroyNode(pThen);
nodesDestroyNode(pTempVal);
return code;
@ -15289,7 +15329,7 @@ static int32_t rewriteShowAliveStmt(STranslateContext* pCxt, SQuery* pQuery) {
}
// pSubSelect, pWhenThenlist need to free
// case when leader_col = count_col and count_col > 0 then 1 when leader_col < count_col and count_col > 0 then 2 else
// case when leader_col = count_col and leader_col > 0 then 1 when leader_col < count_col and leader_col > 0 then 2 else
// 0 end as status
pElse = NULL;
code = nodesMakeValueNodeFromInt32(0, &pElse);

2
source/libs/qworker/src/qwUtil.c
View File

@ -564,7 +564,7 @@ int32_t qwSaveTbVersionInfo(qTaskInfo_t pTaskInfo, SQWTaskCtx *ctx) {
void qwCloseRef(void) {
taosWLockLatch(&gQwMgmt.lock);
if (atomic_load_32(&gQwMgmt.qwNum) <= 0 && gQwMgmt.qwRef >= 0) {
(void)taosCloseRef(gQwMgmt.qwRef); // ignore error
taosCloseRef(gQwMgmt.qwRef); // ignore error
gQwMgmt.qwRef = -1;
}
taosWUnLockLatch(&gQwMgmt.lock);

37
source/libs/scalar/inc/sclvector.h
View File

@ -78,40 +78,41 @@ static FORCE_INLINE int32_t getVectorDoubleValue_BOOL(void *src, int32_t index,
int32_t getVectorDoubleValue_JSON(void *src, int32_t index, double *out);
static FORCE_INLINE _getDoubleValue_fn_t getVectorDoubleValueFn(int32_t srcType) {
_getDoubleValue_fn_t p = NULL;
static FORCE_INLINE int32_t getVectorDoubleValueFn(int32_t srcType, _getDoubleValue_fn_t *p) {
*p = NULL;
if (srcType == TSDB_DATA_TYPE_TINYINT) {
p = getVectorDoubleValue_TINYINT;
*p = getVectorDoubleValue_TINYINT;
} else if (srcType == TSDB_DATA_TYPE_UTINYINT) {
p = getVectorDoubleValue_UTINYINT;
*p = getVectorDoubleValue_UTINYINT;
} else if (srcType == TSDB_DATA_TYPE_SMALLINT) {
p = getVectorDoubleValue_SMALLINT;
*p = getVectorDoubleValue_SMALLINT;
} else if (srcType == TSDB_DATA_TYPE_USMALLINT) {
p = getVectorDoubleValue_USMALLINT;
*p = getVectorDoubleValue_USMALLINT;
} else if (srcType == TSDB_DATA_TYPE_INT) {
p = getVectorDoubleValue_INT;
*p = getVectorDoubleValue_INT;
} else if (srcType == TSDB_DATA_TYPE_UINT) {
p = getVectorDoubleValue_UINT;
*p = getVectorDoubleValue_UINT;
} else if (srcType == TSDB_DATA_TYPE_BIGINT) {
p = getVectorDoubleValue_BIGINT;
*p = getVectorDoubleValue_BIGINT;
} else if (srcType == TSDB_DATA_TYPE_UBIGINT) {
p = getVectorDoubleValue_UBIGINT;
*p = getVectorDoubleValue_UBIGINT;
} else if (srcType == TSDB_DATA_TYPE_FLOAT) {
p = getVectorDoubleValue_FLOAT;
*p = getVectorDoubleValue_FLOAT;
} else if (srcType == TSDB_DATA_TYPE_DOUBLE) {
p = getVectorDoubleValue_DOUBLE;
*p = getVectorDoubleValue_DOUBLE;
} else if (srcType == TSDB_DATA_TYPE_TIMESTAMP) {
p = getVectorDoubleValue_BIGINT;
*p = getVectorDoubleValue_BIGINT;
} else if (srcType == TSDB_DATA_TYPE_JSON) {
p = getVectorDoubleValue_JSON;
*p = getVectorDoubleValue_JSON;
} else if (srcType == TSDB_DATA_TYPE_BOOL) {
p = getVectorDoubleValue_BOOL;
*p = getVectorDoubleValue_BOOL;
} else if (srcType == TSDB_DATA_TYPE_NULL) {
p = NULL;
*p = NULL;
} else {
ASSERT(0);
*p = NULL;
return TSDB_CODE_SCALAR_CONVERT_ERROR;
}
return p;
return TSDB_CODE_SUCCESS;
}
typedef int32_t (*_bufConverteFunc)(char *buf, SScalarParam *pOut, int32_t outType, int32_t *overflow);

183
source/libs/scalar/src/filter.c
View File

@ -92,7 +92,9 @@ rangeCompFunc gRangeCompare[] = {filterRangeCompee, filterRangeCompei, filterRan
int8_t filterGetRangeCompFuncFromOptrs(uint8_t optr, uint8_t optr2) {
if (optr2) {
ASSERT(optr2 == OP_TYPE_LOWER_THAN || optr2 == OP_TYPE_LOWER_EQUAL);
if (optr2 != OP_TYPE_LOWER_THAN && optr2 != OP_TYPE_LOWER_EQUAL) {
return -1;
}
if (optr == OP_TYPE_GREATER_THAN) {
if (optr2 == OP_TYPE_LOWER_THAN) {
@ -763,7 +765,10 @@ int32_t filterAddRangeCtx(void *dst, void *src, int32_t optr) {
SFilterRangeCtx *dctx = (SFilterRangeCtx *)dst;
SFilterRangeCtx *sctx = (SFilterRangeCtx *)src;
ASSERT(optr == LOGIC_COND_TYPE_OR);
if (optr != LOGIC_COND_TYPE_OR) {
fltError("filterAddRangeCtx get invalid optr:%d", optr);
return TSDB_CODE_QRY_FILTER_WRONG_OPTR_TYPE;
}
if (sctx->rs == NULL) {
return TSDB_CODE_SUCCESS;
@ -1204,7 +1209,10 @@ int32_t filterAddUnitImpl(SFilterInfo *info, uint8_t optr, SFilterFieldId *left,
if (u->right.type == FLD_TYPE_VALUE) {
SFilterField *val = FILTER_UNIT_RIGHT_FIELD(info, u);
ASSERT(FILTER_GET_FLAG(val->flag, FLD_TYPE_VALUE));
if (!FILTER_GET_FLAG(val->flag, FLD_TYPE_VALUE)) {
fltError("filterAddUnitImpl get invalid flag : %d in val", val->flag);
return TSDB_CODE_APP_ERROR;
}
} else {
int32_t paramNum = scalarGetOperatorParamNum(optr);
if (1 != paramNum) {
@ -1214,7 +1222,10 @@ int32_t filterAddUnitImpl(SFilterInfo *info, uint8_t optr, SFilterFieldId *left,
}
SFilterField *col = FILTER_UNIT_LEFT_FIELD(info, u);
ASSERT(FILTER_GET_FLAG(col->flag, FLD_TYPE_COLUMN));
if (!FILTER_GET_FLAG(col->flag, FLD_TYPE_COLUMN)) {
fltError("filterAddUnitImpl get invalid flag : %d in col", col->flag);
return TSDB_CODE_APP_ERROR;
}
info->units[info->unitNum].compare.type = FILTER_GET_COL_FIELD_TYPE(col);
info->units[info->unitNum].compare.precision = FILTER_GET_COL_FIELD_PRECISION(col);
@ -1398,29 +1409,48 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
if (optr == LOGIC_COND_TYPE_AND) {
if (ctx->isnull) {
ASSERT(ctx->notnull == false && ctx->isrange == false);
if (ctx->notnull || ctx->isrange) {
fltError("filterAddGroupUnitFromCtx get invalid ctx : isnull %d, notnull %d, isrange %d",
ctx->isnull, ctx->notnull, ctx->isrange);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
FLT_ERR_RET(filterAddUnit(dst, OP_TYPE_IS_NULL, &left, NULL, &uidx));
FLT_ERR_RET(filterAddUnitToGroup(g, uidx));
return TSDB_CODE_SUCCESS;
}
if (ctx->notnull) {
ASSERT(ctx->isnull == false && ctx->isrange == false);
if (ctx->isnull || ctx->isrange) {
fltError("filterAddGroupUnitFromCtx get invalid ctx : isnull %d, notnull %d, isrange %d",
ctx->isnull, ctx->notnull, ctx->isrange);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
FLT_ERR_RET(filterAddUnit(dst, OP_TYPE_IS_NOT_NULL, &left, NULL, &uidx));
FLT_ERR_RET(filterAddUnitToGroup(g, uidx));
return TSDB_CODE_SUCCESS;
}
if (!ctx->isrange) {
ASSERT(ctx->isnull || ctx->notnull);
if (!ctx->isnull && !ctx->notnull) {
fltError("filterAddGroupUnitFromCtx get invalid ctx : isnull %d, notnull %d, isrange %d",
ctx->isnull, ctx->notnull, ctx->isrange);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
return TSDB_CODE_SUCCESS;
}
ASSERT(ctx->rs && ctx->rs->next == NULL);
if (!ctx->rs || ctx->rs->next != NULL) {
fltError("filterAddGroupUnitFromCtx get invalid range node with rs:%p", ctx->rs);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
SFilterRange *ra = &ctx->rs->ra;
ASSERT(!((FILTER_GET_FLAG(ra->sflag, RANGE_FLG_NULL)) && (FILTER_GET_FLAG(ra->eflag, RANGE_FLG_NULL))));
if (((FILTER_GET_FLAG(ra->sflag, RANGE_FLG_NULL)) && (FILTER_GET_FLAG(ra->eflag, RANGE_FLG_NULL)))) {
fltError("filterAddGroupUnitFromCtx get invalid range with sflag:%d, eflag:%d",
FILTER_GET_FLAG(ra->sflag, RANGE_FLG_NULL), FILTER_GET_FLAG(ra->eflag, RANGE_FLG_NULL));
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
if ((!FILTER_GET_FLAG(ra->sflag, RANGE_FLG_NULL)) && (!FILTER_GET_FLAG(ra->eflag, RANGE_FLG_NULL))) {
__compar_fn_t func = getComparFunc(type, 0);
@ -1489,7 +1519,11 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
SFilterGroup ng = {0};
g = &ng;
ASSERT(ctx->isnull || ctx->notnull || ctx->isrange);
if (!ctx->isnull && !ctx->notnull && !ctx->isrange) {
fltError("filterAddGroupUnitFromCtx get invalid ctx : isnull %d, notnull %d, isrange %d",
ctx->isnull, ctx->notnull, ctx->isrange);
FLT_ERR_RET(TSDB_CODE_APP_ERROR);
}
if (ctx->isnull) {
FLT_ERR_RET(filterAddUnit(dst, OP_TYPE_IS_NULL, &left, NULL, &uidx));
@ -1500,7 +1534,11 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
}
if (ctx->notnull) {
ASSERT(!ctx->isrange);
if (ctx->isrange) {
fltError("filterAddGroupUnitFromCtx get invalid ctx : isnull %d, notnull %d, isrange %d",
ctx->isnull, ctx->notnull, ctx->isrange);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
(void)memset(g, 0, sizeof(*g));
FLT_ERR_RET(filterAddUnit(dst, OP_TYPE_IS_NOT_NULL, &left, NULL, &uidx));
@ -1511,7 +1549,11 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
}
if (!ctx->isrange) {
ASSERT(ctx->isnull || ctx->notnull);
if (!ctx->isnull && !ctx->notnull) {
fltError("filterAddGroupUnitFromCtx get invalid ctx : isnull %d, notnull %d, isrange %d",
ctx->isnull, ctx->notnull, ctx->isrange);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
g->unitNum = 0;
return TSDB_CODE_SUCCESS;
}
@ -1586,7 +1628,10 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
FLT_ERR_RET(filterAddUnitToGroup(g, uidx));
}
ASSERT(g->unitNum > 0);
if (g->unitNum <= 0) {
fltError("filterAddGroupUnitFromCtx get invalid filter group unit num %d", g->unitNum);
FLT_ERR_RET(TSDB_CODE_QRY_FILTER_RANGE_ERROR);
}
if (NULL == taosArrayPush(res,g)) {
FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
@ -2066,7 +2111,10 @@ void filterFreeInfo(SFilterInfo *info) {
}
int32_t filterHandleValueExtInfo(SFilterUnit *unit, char extInfo) {
ASSERT(extInfo > 0 || extInfo < 0);
if (extInfo == 0) {
fltError("filterHandleValueExtInfo get invalid extInfo : %d", extInfo);
return TSDB_CODE_APP_ERROR;
}
uint8_t optr = FILTER_UNIT_OPTR(unit);
switch (optr) {
@ -2093,13 +2141,20 @@ int32_t fltInitValFieldData(SFilterInfo *info) {
for (uint32_t i = 0; i < info->unitNum; ++i) {
SFilterUnit *unit = &info->units[i];
if (unit->right.type != FLD_TYPE_VALUE) {
ASSERT(unit->compare.optr == FILTER_DUMMY_EMPTY_OPTR || scalarGetOperatorParamNum(unit->compare.optr) == 1);
if (unit->compare.optr != FILTER_DUMMY_EMPTY_OPTR && scalarGetOperatorParamNum(unit->compare.optr) != 1) {
fltError("filterInitValFieldData get invalid operator param num : %d and invalid compare optr %d",
scalarGetOperatorParamNum(unit->compare.optr), unit->compare.optr);
return TSDB_CODE_APP_ERROR;
}
continue;
}
SFilterField *right = FILTER_UNIT_RIGHT_FIELD(info, unit);
ASSERT(FILTER_GET_FLAG(right->flag, FLD_TYPE_VALUE));
if (!FILTER_GET_FLAG(right->flag, FLD_TYPE_VALUE)) {
fltError("filterInitValFieldData get invalid field flag : %d", right->flag);
return TSDB_CODE_APP_ERROR;
}
uint32_t type = FILTER_UNIT_DATA_TYPE(unit);
int8_t precision = FILTER_UNIT_DATA_PRECISION(unit);
@ -2107,7 +2162,10 @@ int32_t fltInitValFieldData(SFilterInfo *info) {
SValueNode *var = (SValueNode *)fi->desc;
if (var == NULL) {
ASSERT(fi->data != NULL);
if (!fi->data) {
fltError("filterInitValFieldData get invalid field data : NULL");
return TSDB_CODE_APP_ERROR;
}
continue;
}
@ -2257,7 +2315,10 @@ int32_t filterAddUnitRange(SFilterInfo *info, SFilterUnit *u, SFilterRangeCtx *c
FILTER_SET_FLAG(ra.sflag, RANGE_FLG_NULL);
break;
case OP_TYPE_NOT_EQUAL:
ASSERT(type == TSDB_DATA_TYPE_BOOL);
if (type != TSDB_DATA_TYPE_BOOL) {
fltError("filterAddUnitRange get invalid type : %d", type);
return TSDB_CODE_QRY_FILTER_INVALID_TYPE;
}
if (GET_INT8_VAL(val)) {
SIMPLE_COPY_VALUES(&ra.s, &tmp);
SIMPLE_COPY_VALUES(&ra.e, &tmp);
@ -2273,7 +2334,7 @@ int32_t filterAddUnitRange(SFilterInfo *info, SFilterUnit *u, SFilterRangeCtx *c
break;
default:
fltError("unsupported operator type");
return TSDB_CODE_APP_ERROR;
return TSDB_CODE_QRY_FILTER_NOT_SUPPORT_TYPE;
}
FLT_ERR_RET(filterAddRange(ctx, &ra, optr));
@ -2372,6 +2433,7 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t
}
gRes[gResIdx]->colInfo = taosMemoryCalloc(info->fields[FLD_TYPE_COLUMN].num, sizeof(SFilterColInfo));
if (gRes[gResIdx]->colInfo == NULL) {
filterFreeGroupCtx(gRes[gResIdx]);
FLT_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY);
}
colIdxi = 0;
@ -2384,6 +2446,7 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t
if (gRes[gResIdx]->colInfo[cidx].info == NULL) {
gRes[gResIdx]->colInfo[cidx].info = (SArray *)taosArrayInit(4, POINTER_BYTES);
if (gRes[gResIdx]->colInfo[cidx].info == NULL) {
filterFreeGroupCtx(gRes[gResIdx]);
FLT_ERR_JRET(terrno);
}
colIdx[colIdxi++] = cidx;
@ -2408,7 +2471,11 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t
continue;
}
FLT_ERR_JRET(filterMergeUnits(info, gRes[gResIdx], colIdx[l], &empty));
code = filterMergeUnits(info, gRes[gResIdx], colIdx[l], &empty);
if (TSDB_CODE_SUCCESS != code) {
filterFreeGroupCtx(gRes[gResIdx]);
SCL_ERR_JRET(code);
}
if (empty) {
break;
@ -2426,9 +2493,8 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t
gRes[gResIdx]->colNum = colIdxi;
FILTER_COPY_IDX(&gRes[gResIdx]->colIdx, colIdx, colIdxi);
++gResIdx;
}
*gResNum = gResIdx;
}
if (gResIdx == 0) {
FILTER_SET_FLAG(info->status, FI_STATUS_EMPTY);
@ -2542,8 +2608,11 @@ int32_t filterMergeTwoGroupsImpl(SFilterInfo *info, SFilterRangeCtx **ctx, int32
FLT_ERR_RET(filterReuseRangeCtx(*ctx, type, 0));
}
ASSERT(gRes2->colInfo[cidx].type == RANGE_TYPE_MR_CTX);
ASSERT(gRes1->colInfo[cidx].type == RANGE_TYPE_MR_CTX);
if (gRes2->colInfo[cidx].type != RANGE_TYPE_MR_CTX || gRes1->colInfo[cidx].type != RANGE_TYPE_MR_CTX) {
fltError("filterMergeTwoGroupsImpl get invalid col type : %d and %d",
gRes2->colInfo[cidx].type, gRes1->colInfo[cidx].type);
return TSDB_CODE_QRY_FILTER_NOT_SUPPORT_TYPE;
}
FLT_ERR_RET(filterCopyRangeCtx(*ctx, gRes2->colInfo[cidx].info));
FLT_ERR_RET(filterSourceRangeFromCtx(*ctx, gRes1->colInfo[cidx].info, optr, empty, all));
@ -2583,7 +2652,10 @@ int32_t filterMergeTwoGroups(SFilterInfo *info, SFilterGroupCtx **gRes1, SFilter
continue;
}
ASSERT(idx1 == idx2);
if (idx1 != idx2) {
fltError("filterMergeTwoGroups get invalid idx : %d and %d", idx1, idx2);
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
++merNum;
@ -2639,16 +2711,19 @@ int32_t filterMergeTwoGroups(SFilterInfo *info, SFilterGroupCtx **gRes1, SFilter
}
}
ASSERT(merNum > 0);
SFilterColInfo *colInfo = NULL;
ASSERT(merNum == equal1 || merNum == equal2);
if (merNum == 0 || (equal1 != merNum && equal2 != merNum)) {
fltError("filterMergeTwoGroups get invalid merge num : %d, equal1 : %d, equal2 : %d", merNum, equal1, equal2);
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
filterFreeGroupCtx(*gRes2);
*gRes2 = NULL;
ASSERT(colCtxs && taosArrayGetSize(colCtxs) > 0);
if (!colCtxs || taosArrayGetSize(colCtxs) <= 0) {
fltError("filterMergeTwoGroups get invalid colCtxs with size %zu", taosArrayGetSize(colCtxs));
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
SFilterColInfo *colInfo = NULL;
int32_t ctxSize = (int32_t)taosArrayGetSize(colCtxs);
SFilterColCtx *pctx = NULL;
@ -2708,7 +2783,10 @@ int32_t filterMergeGroups(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t *gR
if (pColNum > 0) {
for (int32_t m = 0; m <= pEnd; ++m) {
for (int32_t n = cStart; n <= cEnd; ++n) {
ASSERT(m < n);
if (m >= n) {
fltError("filterMergeGroups get invalid m : %d and n : %d", m, n);
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
FLT_ERR_JRET(filterMergeTwoGroups(info, &gRes[m], &gRes[n], &all));
FLT_CHK_JMP(all);
@ -2729,7 +2807,10 @@ int32_t filterMergeGroups(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t *gR
for (int32_t m = cStart; m < cEnd; ++m) {
for (int32_t n = m + 1; n <= cEnd; ++n) {
ASSERT(m < n);
if (m >= n) {
fltError("filterMergeGroups get invalid m : %d and n : %d", m, n);
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
FLT_ERR_JRET(filterMergeTwoGroups(info, &gRes[m], &gRes[n], &all));
FLT_CHK_JMP(all);
@ -2839,7 +2920,10 @@ int32_t filterRewrite(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t gResNum
for (uint32_t m = 0; m < res->colNum; ++m) {
colInfo = &res->colInfo[res->colIdx[m]];
if (FILTER_NO_MERGE_DATA_TYPE(colInfo->dataType)) {
ASSERT(colInfo->type == RANGE_TYPE_UNIT);
if (colInfo->type != RANGE_TYPE_UNIT) {
fltError("filterRewrite get invalid col type : %d", colInfo->type);
FLT_ERR_JRET(TSDB_CODE_QRY_FILTER_INVALID_TYPE);
}
int32_t usize = (int32_t)taosArrayGetSize((SArray *)colInfo->info);
for (int32_t n = 0; n < usize; ++n) {
@ -2854,7 +2938,10 @@ int32_t filterRewrite(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t gResNum
continue;
}
ASSERT(colInfo->type == RANGE_TYPE_MR_CTX);
if (colInfo->type != RANGE_TYPE_MR_CTX) {
fltError("filterRewrite get invalid col type : %d", colInfo->type);
FLT_ERR_JRET(TSDB_CODE_QRY_FILTER_INVALID_TYPE);
}
FLT_ERR_JRET(filterAddGroupUnitFromCtx(info, &oinfo, colInfo->info, res->colIdx[m], &ng, optr, group));
}
@ -2900,7 +2987,10 @@ int32_t filterGenerateColRange(SFilterInfo *info, SFilterGroupCtx **gRes, int32_
continue;
}
ASSERT(idxNum[i] == gResNum);
if (idxNum[i] != gResNum) {
fltError("filterGenerateColRange get invalid idxNum : %d and gResNum : %d", idxNum[i], gResNum);
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
if (idxs == NULL) {
idxs = taosMemoryCalloc(info->fields[FLD_TYPE_COLUMN].num, sizeof(*idxs));
@ -2931,7 +3021,10 @@ int32_t filterGenerateColRange(SFilterInfo *info, SFilterGroupCtx **gRes, int32_
continue;
}
ASSERT(res->colIdx[n] == idxs[m]);
if (res->colIdx[n] != idxs[m]) {
fltError("filterGenerateColRange get invalid colIdx : %d and idxs : %d", res->colIdx[n], idxs[m]);
SCL_ERR_JRET(TSDB_CODE_APP_ERROR);
}
SFilterColInfo *colInfo = &res->colInfo[res->colIdx[n]];
if (info->colRange[m] == NULL) {
@ -2940,7 +3033,10 @@ int32_t filterGenerateColRange(SFilterInfo *info, SFilterGroupCtx **gRes, int32_
info->colRange[m]->colId = FILTER_GET_COL_FIELD_ID(fi);
}
ASSERT(colInfo->type == RANGE_TYPE_MR_CTX);
if (colInfo->type != RANGE_TYPE_MR_CTX) {
fltError("filterGenerateColRange get invalid col type : %d", colInfo->type);
FLT_ERR_JRET(TSDB_CODE_QRY_FILTER_INVALID_TYPE);
}
bool all = false;
FLT_ERR_JRET(filterSourceRangeFromCtx(info->colRange[m], colInfo->info, LOGIC_COND_TYPE_OR, NULL, &all));
@ -3190,7 +3286,10 @@ int32_t filterRmUnitByRange(SFilterInfo *info, SColumnDataAgg *pDataStatis, int3
unitIdx = pGroupIdx;
--info->blkGroupNum;
ASSERT(empty || all);
if (!empty && !all) {
fltError("filterRmUnitByRange get invalid empty and all : %d and %d", empty, all);
FLT_ERR_RET(TSDB_CODE_APP_ERROR);
}
if (empty) {
FILTER_SET_FLAG(info->blkFlag, FI_STATUS_BLK_EMPTY);
@ -3298,7 +3397,10 @@ int32_t filterExecuteBasedOnStatis(SFilterInfo *info, int32_t numOfRows, SColumn
goto _return;
}
ASSERT(info->unitNum > 1);
if (info->unitNum <= 1) {
fltError("filterExecuteBasedOnStatis get invalid unit num : %d", info->unitNum);
FLT_ERR_JRET(TSDB_CODE_APP_ERROR);
}
*all = filterExecuteBasedOnStatisImpl(info, numOfRows, p, statis, numOfCols);
goto _return;
@ -5156,7 +5258,6 @@ int32_t filterExecute(SFilterInfo *info, SSDataBlock *pSrc, SColumnInfoData **p,
return TSDB_CODE_SUCCESS;
}
ASSERT(false == info->scalarMode);
*p = output.columnData;
output.numOfRows = pSrc->info.rows;

7
source/libs/scalar/src/sclfunc.c
View File

@ -297,7 +297,8 @@ static int32_t doScalarFunctionUnique(SScalarParam *pInput, int32_t inputNum, SS
SColumnInfoData *pInputData = pInput->columnData;
SColumnInfoData *pOutputData = pOutput->columnData;
_getDoubleValue_fn_t getValueFn = getVectorDoubleValueFn(type);
_getDoubleValue_fn_t getValueFn;
SCL_ERR_RET(getVectorDoubleValueFn(type, &getValueFn));
double *out = (double *)pOutputData->pData;
@ -328,7 +329,7 @@ static int32_t doScalarFunctionUnique2(SScalarParam *pInput, int32_t inputNum, S
for (int32_t i = 0; i < inputNum; ++i) {
pInputData[i] = pInput[i].columnData;
getValueFn[i] = getVectorDoubleValueFn(GET_PARAM_TYPE(&pInput[i]));
SCL_ERR_RET(getVectorDoubleValueFn(GET_PARAM_TYPE(&pInput[i]), &getValueFn[i]));
}
double *out = (double *)pOutputData->pData;
@ -2918,7 +2919,7 @@ static int32_t doScalarFunction2(SScalarParam *pInput, int32_t inputNum, SScalar
for (int32_t i = 0; i < inputNum; ++i) {
pInputData[i] = pInput[i].columnData;
getValueFn[i] = getVectorDoubleValueFn(GET_PARAM_TYPE(&pInput[i]));
SCL_ERR_RET(getVectorDoubleValueFn(GET_PARAM_TYPE(&pInput[i]), &getValueFn[i]));
}
bool hasNullType = (IS_NULL_TYPE(GET_PARAM_TYPE(&pInput[0])) || IS_NULL_TYPE(GET_PARAM_TYPE(&pInput[1])));

168
source/libs/scalar/src/sclvector.c
View File

@ -43,7 +43,7 @@ bool noConvertBeforeCompare(int32_t leftType, int32_t rightType, int32_t optr) {
(optr >= OP_TYPE_GREATER_THAN && optr <= OP_TYPE_NOT_EQUAL);
}
void convertNumberToNumber(const void *inData, void *outData, int8_t inType, int8_t outType) {
int32_t convertNumberToNumber(const void *inData, void *outData, int8_t inType, int8_t outType) {
switch (outType) {
case TSDB_DATA_TYPE_BOOL: {
GET_TYPED_DATA(*((bool *)outData), bool, inType, inData);
@ -91,9 +91,10 @@ void convertNumberToNumber(const void *inData, void *outData, int8_t inType, int
break;
}
default: {
ASSERT(0);
return TSDB_CODE_SCALAR_CONVERT_ERROR;
}
}
return TSDB_CODE_SUCCESS;
}
int32_t convertNcharToDouble(const void *inData, void *outData) {
@ -180,7 +181,10 @@ int32_t getVectorBigintValue_BOOL(void *src, int32_t index, int64_t *res) {
}
int32_t getVectorBigintValue_JSON(void *src, int32_t index, int64_t *res) {
ASSERT(!colDataIsNull_var(((SColumnInfoData *)src), index));
if (colDataIsNull_var(((SColumnInfoData *)src), index)) {
sclError("getVectorBigintValue_JSON get json data null with index %d", index);
SCL_ERR_RET(TSDB_CODE_SCALAR_CONVERT_ERROR);
}
char *data = colDataGetVarData((SColumnInfoData *)src, index);
double out = 0;
if (*data == TSDB_DATA_TYPE_NULL) {
@ -192,46 +196,47 @@ int32_t getVectorBigintValue_JSON(void *src, int32_t index, int64_t *res) {
*res = 0;
SCL_ERR_RET(TSDB_CODE_QRY_JSON_NOT_SUPPORT_ERROR);
} else {
convertNumberToNumber(data + CHAR_BYTES, &out, *data, TSDB_DATA_TYPE_DOUBLE);
SCL_ERR_RET(convertNumberToNumber(data + CHAR_BYTES, &out, *data, TSDB_DATA_TYPE_DOUBLE));
}
*res = (int64_t)out;
SCL_RET(TSDB_CODE_SUCCESS);
}
_getBigintValue_fn_t getVectorBigintValueFn(int32_t srcType) {
_getBigintValue_fn_t p = NULL;
int32_t getVectorBigintValueFn(int32_t srcType, _getBigintValue_fn_t *p) {
*p = NULL;
if (srcType == TSDB_DATA_TYPE_TINYINT) {
p = getVectorBigintValue_TINYINT;
*p = getVectorBigintValue_TINYINT;
} else if (srcType == TSDB_DATA_TYPE_UTINYINT) {
p = getVectorBigintValue_UTINYINT;
*p = getVectorBigintValue_UTINYINT;
} else if (srcType == TSDB_DATA_TYPE_SMALLINT) {
p = getVectorBigintValue_SMALLINT;
*p = getVectorBigintValue_SMALLINT;
} else if (srcType == TSDB_DATA_TYPE_USMALLINT) {
p = getVectorBigintValue_USMALLINT;
*p = getVectorBigintValue_USMALLINT;
} else if (srcType == TSDB_DATA_TYPE_INT) {
p = getVectorBigintValue_INT;
*p = getVectorBigintValue_INT;
} else if (srcType == TSDB_DATA_TYPE_UINT) {
p = getVectorBigintValue_UINT;
*p = getVectorBigintValue_UINT;
} else if (srcType == TSDB_DATA_TYPE_BIGINT) {
p = getVectorBigintValue_BIGINT;
*p = getVectorBigintValue_BIGINT;
} else if (srcType == TSDB_DATA_TYPE_UBIGINT) {
p = getVectorBigintValue_UBIGINT;
*p = getVectorBigintValue_UBIGINT;
} else if (srcType == TSDB_DATA_TYPE_FLOAT) {
p = getVectorBigintValue_FLOAT;
*p = getVectorBigintValue_FLOAT;
} else if (srcType == TSDB_DATA_TYPE_DOUBLE) {
p = getVectorBigintValue_DOUBLE;
*p = getVectorBigintValue_DOUBLE;
} else if (srcType == TSDB_DATA_TYPE_TIMESTAMP) {
p = getVectorBigintValue_BIGINT;
*p = getVectorBigintValue_BIGINT;
} else if (srcType == TSDB_DATA_TYPE_BOOL) {
p = getVectorBigintValue_BOOL;
*p = getVectorBigintValue_BOOL;
} else if (srcType == TSDB_DATA_TYPE_JSON) {
p = getVectorBigintValue_JSON;
*p = getVectorBigintValue_JSON;
} else if (srcType == TSDB_DATA_TYPE_NULL) {
p = NULL;
*p = NULL;
} else {
ASSERT(0);
sclError("getVectorBigintValueFn invalid srcType : %d", srcType);
return TSDB_CODE_SCALAR_CONVERT_ERROR;
}
return p;
return TSDB_CODE_SUCCESS;
}
static FORCE_INLINE int32_t varToTimestamp(char *buf, SScalarParam *pOut, int32_t rowIndex, int32_t *overflow) {
@ -467,7 +472,7 @@ static FORCE_INLINE int32_t varToGeometry(char *buf, SScalarParam *pOut, int32_t
_return:
taosMemoryFree(output);
geosFreeBuffer(t);
ASSERT(t == NULL && len == 0);
t = NULL;
VarDataLenT dummyHeader = 0;
SCL_ERR_RET(colDataSetVal(pOut->columnData, rowIndex, (const char *)&dummyHeader, false));
SCL_RET(code);
@ -525,7 +530,7 @@ int32_t vectorConvertFromVarData(SSclVectorConvCtx *pCtx, int32_t *overflow) {
} else if (tTagIsJson(data) || *data == TSDB_DATA_TYPE_NULL) {
SCL_ERR_JRET(TSDB_CODE_QRY_JSON_NOT_SUPPORT_ERROR);
} else {
convertNumberToNumber(data + CHAR_BYTES, colDataGetNumData(pCtx->pOut->columnData, i), *data, pCtx->outType);
SCL_ERR_JRET(convertNumberToNumber(data + CHAR_BYTES, colDataGetNumData(pCtx->pOut->columnData, i), *data, pCtx->outType));
continue;
}
}
@ -582,7 +587,7 @@ int32_t getVectorDoubleValue_JSON(void *src, int32_t index, double *out) {
} else if (tTagIsJson(data)) {
SCL_ERR_RET(TSDB_CODE_QRY_JSON_NOT_SUPPORT_ERROR);
} else {
convertNumberToNumber(data + CHAR_BYTES, out, *data, TSDB_DATA_TYPE_DOUBLE);
SCL_ERR_RET(convertNumberToNumber(data + CHAR_BYTES, out, *data, TSDB_DATA_TYPE_DOUBLE));
}
SCL_RET(TSDB_CODE_SUCCESS);
}
@ -673,7 +678,7 @@ int32_t convertJsonValue(__compar_fn_t *fp, int32_t optr, int8_t typeLeft, int8_
*result = false;
return TSDB_CODE_SUCCESS;
} else if (typeLeft != type) {
convertNumberToNumber(*pLeftData, pLeftOut, typeLeft, type);
SCL_ERR_RET(convertNumberToNumber(*pLeftData, pLeftOut, typeLeft, type));
*pLeftData = pLeftOut;
}
@ -683,7 +688,7 @@ int32_t convertJsonValue(__compar_fn_t *fp, int32_t optr, int8_t typeLeft, int8_
*result = false;
return TSDB_CODE_SUCCESS;
} else if (typeRight != type) {
convertNumberToNumber(*pRightData, pRightOut, typeRight, type);
SCL_ERR_RET(convertNumberToNumber(*pRightData, pRightOut, typeRight, type));
*pRightData = pRightOut;
}
} else if (type == TSDB_DATA_TYPE_BINARY ||
@ -1130,8 +1135,10 @@ enum {
// TODO not correct for descending order scan
static int32_t vectorMathAddHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t i) {
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_RET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_RET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
double *output = (double *)pOutputCol->pData;
@ -1155,9 +1162,10 @@ static int32_t vectorMathAddHelper(SColumnInfoData *pLeftCol, SColumnInfoData *p
static int32_t vectorMathTsAddHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t i) {
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_RET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_RET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
int64_t *output = (int64_t *)pOutputCol->pData;
if (IS_HELPER_NULL(pRightCol, 0)) { // Set pLeft->numOfRows NULL value
@ -1230,8 +1238,10 @@ int32_t vectorMathAdd(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *p
(GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_TIMESTAMP &&
GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_BOOL)) { // timestamp plus duration
int64_t *output = (int64_t *)pOutputCol->pData;
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_JRET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_JRET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
if (pLeft->numOfRows == 1 && pRight->numOfRows == 1) {
if (GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_TIMESTAMP) {
@ -1258,9 +1268,10 @@ int32_t vectorMathAdd(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *p
}
} else {
double *output = (double *)pOutputCol->pData;
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_JRET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_JRET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
if (pLeft->numOfRows == pRight->numOfRows) {
for (; i < pRight->numOfRows && i >= 0; i += step, output += 1) {
if (IS_NULL) {
@ -1289,8 +1300,10 @@ _return:
// TODO not correct for descending order scan
static int32_t vectorMathSubHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t factor, int32_t i) {
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_RET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_RET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
double *output = (double *)pOutputCol->pData;
@ -1314,8 +1327,10 @@ static int32_t vectorMathSubHelper(SColumnInfoData *pLeftCol, SColumnInfoData *p
static int32_t vectorMathTsSubHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t factor, int32_t i) {
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_RET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_RET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
int64_t *output = (int64_t *)pOutputCol->pData;
@ -1357,8 +1372,10 @@ int32_t vectorMathSub(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *p
(GET_PARAM_TYPE(pRight) == TSDB_DATA_TYPE_TIMESTAMP &&
GET_PARAM_TYPE(pLeft) == TSDB_DATA_TYPE_BIGINT)) { // timestamp minus duration
int64_t *output = (int64_t *)pOutputCol->pData;
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_JRET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_JRET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
if (pLeft->numOfRows == 1 && pRight->numOfRows == 1) {
SCL_ERR_JRET(vectorMathTsSubHelper(pLeftCol, pRightCol, pOutputCol, pLeft->numOfRows, step, 1, i));
@ -1381,8 +1398,10 @@ int32_t vectorMathSub(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *p
}
} else {
double *output = (double *)pOutputCol->pData;
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_JRET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_JRET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
if (pLeft->numOfRows == pRight->numOfRows) {
for (; i < pRight->numOfRows && i >= 0; i += step, output += 1) {
@ -1412,8 +1431,10 @@ _return:
// TODO not correct for descending order scan
static int32_t vectorMathMultiplyHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t i) {
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_RET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_RET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
double *output = (double *)pOutputCol->pData;
@ -1449,8 +1470,10 @@ int32_t vectorMathMultiply(SScalarParam *pLeft, SScalarParam *pRight, SScalarPar
SCL_ERR_JRET(vectorConvertVarToDouble(pLeft, &leftConvert, &pLeftCol));
SCL_ERR_JRET(vectorConvertVarToDouble(pRight, &rightConvert, &pRightCol));
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_JRET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_JRET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
double *output = (double *)pOutputCol->pData;
if (pLeft->numOfRows == pRight->numOfRows) {
@ -1491,8 +1514,10 @@ int32_t vectorMathDivide(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam
SCL_ERR_JRET(vectorConvertVarToDouble(pLeft, &leftConvert, &pLeftCol));
SCL_ERR_JRET(vectorConvertVarToDouble(pRight, &rightConvert, &pRightCol));
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_JRET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_JRET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
double *output = (double *)pOutputCol->pData;
if (pLeft->numOfRows == pRight->numOfRows) {
@ -1573,8 +1598,10 @@ int32_t vectorMathRemainder(SScalarParam *pLeft, SScalarParam *pRight, SScalarPa
SCL_ERR_JRET(vectorConvertVarToDouble(pLeft, &leftConvert, &pLeftCol));
SCL_ERR_JRET(vectorConvertVarToDouble(pRight, &rightConvert, &pRightCol));
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnRight = getVectorDoubleValueFn(pRightCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
_getDoubleValue_fn_t getVectorDoubleValueFnRight;
SCL_ERR_JRET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
SCL_ERR_JRET(getVectorDoubleValueFn(pRightCol->info.type, &getVectorDoubleValueFnRight));
double *output = (double *)pOutputCol->pData;
@ -1661,7 +1688,8 @@ int32_t vectorMathMinus(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam
SColumnInfoData *pLeftCol = NULL;
SCL_ERR_JRET(vectorConvertVarToDouble(pLeft, &leftConvert, &pLeftCol));
_getDoubleValue_fn_t getVectorDoubleValueFnLeft = getVectorDoubleValueFn(pLeftCol->info.type);
_getDoubleValue_fn_t getVectorDoubleValueFnLeft;
SCL_ERR_JRET(getVectorDoubleValueFn(pLeftCol->info.type, &getVectorDoubleValueFnLeft));
double *output = (double *)pOutputCol->pData;
for (; i < pLeft->numOfRows && i >= 0; i += step, output += 1) {
@ -1692,15 +1720,20 @@ int32_t vectorAssign(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pO
}
}
ASSERT(pRight->numOfQualified == 1 || pRight->numOfQualified == 0);
if (pRight->numOfQualified != 1 && pRight->numOfQualified != 0) {
sclError("vectorAssign: invalid qualified number %d", pRight->numOfQualified);
SCL_ERR_RET(TSDB_CODE_APP_ERROR);
}
pOut->numOfQualified = pRight->numOfQualified * pOut->numOfRows;
return TSDB_CODE_SUCCESS;
}
static int32_t vectorBitAndHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t i) {
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_RET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_RET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
int64_t *output = (int64_t *)pOutputCol->pData;
@ -1736,8 +1769,10 @@ int32_t vectorBitAnd(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pO
SCL_ERR_JRET(vectorConvertVarToDouble(pLeft, &leftConvert, &pLeftCol));
SCL_ERR_JRET(vectorConvertVarToDouble(pRight, &rightConvert, &pRightCol));
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_JRET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_JRET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
int64_t *output = (int64_t *)pOutputCol->pData;
if (pLeft->numOfRows == pRight->numOfRows) {
@ -1766,8 +1801,10 @@ _return:
static int32_t vectorBitOrHelper(SColumnInfoData *pLeftCol, SColumnInfoData *pRightCol, SColumnInfoData *pOutputCol,
int32_t numOfRows, int32_t step, int32_t i) {
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_RET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_RET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
int64_t *output = (int64_t *)pOutputCol->pData;
@ -1803,8 +1840,10 @@ int32_t vectorBitOr(SScalarParam *pLeft, SScalarParam *pRight, SScalarParam *pOu
SCL_ERR_JRET(vectorConvertVarToDouble(pLeft, &leftConvert, &pLeftCol));
SCL_ERR_JRET(vectorConvertVarToDouble(pRight, &rightConvert, &pRightCol));
_getBigintValue_fn_t getVectorBigintValueFnLeft = getVectorBigintValueFn(pLeftCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnRight = getVectorBigintValueFn(pRightCol->info.type);
_getBigintValue_fn_t getVectorBigintValueFnLeft;
_getBigintValue_fn_t getVectorBigintValueFnRight;
SCL_ERR_JRET(getVectorBigintValueFn(pLeftCol->info.type, &getVectorBigintValueFnLeft));
SCL_ERR_JRET(getVectorBigintValueFn(pRightCol->info.type, &getVectorBigintValueFnRight));
int64_t *output = (int64_t *)pOutputCol->pData;
if (pLeft->numOfRows == pRight->numOfRows) {
@ -1892,7 +1931,8 @@ int32_t doVectorCompareImpl(SScalarParam *pLeft, SScalarParam *pRight, SScalarPa
&leftOut, &rightOut, &isJsonnull, &freeLeft, &freeRight, &result));
if (isJsonnull) {
ASSERT(0);
sclError("doVectorCompareImpl: invalid json null value");
SCL_ERR_RET(TSDB_CODE_APP_ERROR);
}
if (!pLeftData || !pRightData) {

2
source/libs/scalar/test/filter/CMakeLists.txt
View File

@ -1,7 +1,7 @@
MESSAGE(STATUS "build filter unit test")
IF(NOT TD_DARWIN)
IF(TD_DARWIN)
# GoogleTest requires at least C++11
SET(CMAKE_CXX_STANDARD 11)
AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)

3
source/libs/scalar/test/filter/filterTests.cpp
View File

@ -113,8 +113,7 @@ int32_t flttMakeColumnNode(SNode **pNode, SSDataBlock **block, int32_t dataType,
if (NULL == *block) {
SSDataBlock *res = NULL;
int32_t code = createDataBlock(&res);
ASSERT(code == 0);
FLT_ERR_RET(createDataBlock(&res));
for (int32_t i = 0; i < 2; ++i) {
SColumnInfoData idata = createColumnInfoData(TSDB_DATA_TYPE_NULL, 10, 1 + i);

14
source/libs/scalar/test/scalar/scalarTests.cpp
View File

@ -115,7 +115,10 @@ int32_t scltAppendReservedSlot(SArray *pBlockList, int16_t *dataBlockId, int16_t
res->info.capacity = rows;
res->info.rows = rows;
SColumnInfoData *p = static_cast<SColumnInfoData *>(taosArrayGet(res->pDataBlock, 0));
ASSERT(p->pData != NULL && p->nullbitmap != NULL);
if (p->pData == NULL || p->nullbitmap == NULL) {
sclError("data block is not initialized since pData or nullbitmap is NULL");
SCL_ERR_RET(TSDB_CODE_APP_ERROR);
}
(void)taosArrayPush(pBlockList, &res);
*dataBlockId = taosArrayGetSize(pBlockList) - 1;
@ -189,8 +192,7 @@ int32_t scltMakeColumnNode(SNode **pNode, SSDataBlock **block, int32_t dataType,
if (NULL == *block) {
SSDataBlock *res = NULL;
int32_t code = createDataBlock(&res);
ASSERT(code == 0);
SCL_ERR_RET(createDataBlock(&res));
for (int32_t i = 0; i < 2; ++i) {
SColumnInfoData idata = createColumnInfoData(TSDB_DATA_TYPE_INT, 10, i + 1);
@ -1422,8 +1424,10 @@ int32_t makeCalculate(void *json, void *key, int32_t rightType, void *rightData,
opType == OP_TYPE_NMATCH) {
(void)printf("op:%s,3result:%d,except:%f\n", operatorTypeStr(opType), *((bool *)colDataGetData(column, 0)),
exceptValue);
assert(*(bool *)colDataGetData(column, 0) == exceptValue);
// ASSERT_EQ((int) *((bool *)colDataGetData(column, 0)), (int)exceptValue);
if(*(bool *)colDataGetData(column, 0) != exceptValue) {
(void)printf("expect value %d, but got %d\n", *((bool *)colDataGetData(column, 0)), exceptValue);
SCL_ERR_RET(TSDB_CODE_FAILED);
}
}
taosArrayDestroyEx(blockList, scltFreeDataBlock);

2
source/libs/scheduler/src/schUtil.c
View File

@ -273,7 +273,7 @@ void schCloseJobRef(void) {
}
if (schMgmt.jobRef >= 0) {
(void)taosCloseRef(schMgmt.jobRef);
taosCloseRef(schMgmt.jobRef);
schMgmt.jobRef = -1;
}
}

10
source/libs/stream/src/streamMeta.c
View File

@ -65,9 +65,9 @@ static void streamMetaEnvInit() {
void streamMetaInit() { (void)taosThreadOnce(&streamMetaModuleInit, streamMetaEnvInit); }
void streamMetaCleanup() {
(void)taosCloseRef(streamBackendId);
(void)taosCloseRef(streamBackendCfWrapperId);
(void)taosCloseRef(streamMetaId);
taosCloseRef(streamBackendId);
taosCloseRef(streamBackendCfWrapperId);
taosCloseRef(streamMetaId);
metaRefMgtCleanup();
streamTimerCleanUp();
@ -1260,7 +1260,9 @@ void streamMetaUpdateStageRole(SStreamMeta* pMeta, int64_t stage, bool isLeader)
pMeta->stage = stage;
// mark the sign to send msg before close all tasks
if ((!isLeader) && (pMeta->role == NODE_ROLE_LEADER)) {
// 1. for leader vnode, always send msg before closing
// 2. for follower vnode, if it's is changed from leader, also sending msg before closing.
if (pMeta->role == NODE_ROLE_LEADER) {
pMeta->sendMsgBeforeClosing = true;
}

17
source/libs/sync/inc/syncEnv.h
View File

@ -23,29 +23,12 @@ extern "C" {
#include "syncInt.h"
#define TIMER_MAX_MS 0x7FFFFFFF
#define ENV_TICK_TIMER_MS 1000
#define PING_TIMER_MS 5000
#define ELECT_TIMER_MS_MIN 2500
#define HEARTBEAT_TIMER_MS 1000
#define HEARTBEAT_TICK_NUM 20
typedef struct SSyncEnv {
uint8_t isStart;
// tick timer
tmr_h pEnvTickTimer;
int32_t envTickTimerMS;
uint64_t envTickTimerLogicClock; // if use queue, should pass logic clock into queue item
uint64_t envTickTimerLogicClockUser;
TAOS_TMR_CALLBACK FpEnvTickTimer; // Timer Fp
uint64_t envTickTimerCounter;
// timer manager
tmr_h pTimerManager;
// other resources shared by SyncNodes
// ...
} SSyncEnv;
SSyncEnv* syncEnv();

1
source/libs/sync/inc/syncRaftCfg.h
View File

@ -24,7 +24,6 @@ extern "C" {
int32_t syncWriteCfgFile(SSyncNode *pNode);
int32_t syncReadCfgFile(SSyncNode *pNode);
int32_t syncAddCfgIndex(SSyncNode *pNode, SyncIndex cfgIndex);
#ifdef __cplusplus
}

4
source/libs/sync/inc/syncUtil.h
View File

@ -75,8 +75,6 @@ int32_t syncUtilElectRandomMS(int32_t min, int32_t max);
int32_t syncUtilQuorum(int32_t replicaNum);
const char* syncStr(ESyncState state);
void syncUtilMsgHtoN(void* msg);
bool syncUtilUserPreCommit(tmsg_t msgType);
bool syncUtilUserRollback(tmsg_t msgType);
void syncUtilGenerateArbToken(int32_t nodeId, int32_t groupId, char* buf);
@ -109,7 +107,7 @@ void syncLogRecvAppendEntries(SSyncNode* pSyncNode, const SyncAppendEntries* pMs
void syncLogSendAppendEntries(SSyncNode* pSyncNode, const SyncAppendEntries* pMsg, const char* s);
void syncLogRecvRequestVote(SSyncNode* pSyncNode, const SyncRequestVote* pMsg, int32_t voteGranted, const char* s);
void syncLogSendRequestVote(SSyncNode* pNode, const SyncRequestVote* pMsg, const char* s);
void syncLogSendRequestVote(SSyncNode* pSyncNode, const SyncRequestVote* pMsg, const char* s);
void syncLogRecvRequestVoteReply(SSyncNode* pSyncNode, const SyncRequestVoteReply* pMsg, const char* s);
void syncLogSendRequestVoteReply(SSyncNode* pSyncNode, const SyncRequestVoteReply* pMsg, const char* s);

101
source/libs/sync/src/syncEnv.c
View File

@ -21,7 +21,6 @@
static SSyncEnv gSyncEnv = {0};
static int32_t gNodeRefId = -1;
static int32_t gHbDataRefId = -1;
static void syncEnvTick(void *param, void *tmrId);
SSyncEnv *syncEnv() { return &gSyncEnv; }
@ -33,67 +32,69 @@ int32_t syncInit() {
uint32_t seed = (uint32_t)(taosGetTimestampNs() & 0x00000000FFFFFFFF);
taosSeedRand(seed);
memset(&gSyncEnv, 0, sizeof(SSyncEnv));
gSyncEnv.envTickTimerCounter = 0;
gSyncEnv.envTickTimerMS = ENV_TICK_TIMER_MS;
gSyncEnv.FpEnvTickTimer = syncEnvTick;
atomic_store_64(&gSyncEnv.envTickTimerLogicClock, 0);
atomic_store_64(&gSyncEnv.envTickTimerLogicClockUser, 0);
// start tmr thread
(void)memset(&gSyncEnv, 0, sizeof(SSyncEnv));
gSyncEnv.pTimerManager = taosTmrInit(1000, 50, 10000, "SYNC-ENV");
atomic_store_8(&gSyncEnv.isStart, 1);
gNodeRefId = taosOpenRef(200, (RefFp)syncNodeClose);
if (gNodeRefId < 0) {
sError("failed to init node ref");
sError("failed to init node rset");
syncCleanUp();
return TSDB_CODE_SYN_WRONG_REF;
}
sDebug("sync node rset is open, rsetId:%d", gNodeRefId);
gHbDataRefId = taosOpenRef(200, (RefFp)syncHbTimerDataFree);
if (gHbDataRefId < 0) {
sError("failed to init hb-data ref");
sError("failed to init hbdata rset");
syncCleanUp();
return TSDB_CODE_SYN_WRONG_REF;
}
sDebug("sync rsetId:%d is open", gNodeRefId);
sDebug("sync hbdata rset is open, rsetId:%d", gHbDataRefId);
atomic_store_8(&gSyncEnv.isStart, 1);
return 0;
}
void syncCleanUp() {
atomic_store_8(&gSyncEnv.isStart, 0);
taosTmrCleanUp(gSyncEnv.pTimerManager);
memset(&gSyncEnv, 0, sizeof(SSyncEnv));
(void)memset(&gSyncEnv, 0, sizeof(SSyncEnv));
if (gNodeRefId != -1) {
sDebug("sync rsetId:%d is closed", gNodeRefId);
(void)taosCloseRef(gNodeRefId);
sDebug("sync node rset is closed, rsetId:%d", gNodeRefId);
taosCloseRef(gNodeRefId);
gNodeRefId = -1;
}
if (gHbDataRefId != -1) {
sDebug("sync rsetId:%d is closed", gHbDataRefId);
(void)taosCloseRef(gHbDataRefId);
sDebug("sync hbdata rset is closed, rsetId:%d", gHbDataRefId);
taosCloseRef(gHbDataRefId);
gHbDataRefId = -1;
}
}
int64_t syncNodeAdd(SSyncNode *pNode) {
pNode->rid = taosAddRef(gNodeRefId, pNode);
if (pNode->rid < 0) return -1;
if (pNode->rid < 0) {
return terrno = TSDB_CODE_SYN_WRONG_REF;
}
sDebug("vgId:%d, sync rid:%" PRId64 " is added to rsetId:%d", pNode->vgId, pNode->rid, gNodeRefId);
sDebug("vgId:%d, sync node refId:%" PRId64 " is added to rsetId:%d", pNode->vgId, pNode->rid, gNodeRefId);
return pNode->rid;
}
void syncNodeRemove(int64_t rid) { (void)taosRemoveRef(gNodeRefId, rid); }
void syncNodeRemove(int64_t rid) {
sDebug("sync node refId:%" PRId64 " is removed from rsetId:%d", rid, gNodeRefId);
if (rid > 0) {
(void)taosRemoveRef(gNodeRefId, rid);
}
}
SSyncNode *syncNodeAcquire(int64_t rid) {
SSyncNode *pNode = taosAcquireRef(gNodeRefId, rid);
if (pNode == NULL) {
sError("failed to acquire node from refId:%" PRId64, rid);
sError("failed to acquire sync node from refId:%" PRId64 ", rsetId:%d", rid, gNodeRefId);
terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
}
@ -101,62 +102,38 @@ SSyncNode *syncNodeAcquire(int64_t rid) {
}
void syncNodeRelease(SSyncNode *pNode) {
if (pNode) (void)taosReleaseRef(gNodeRefId, pNode->rid);
if (pNode) {
(void)taosReleaseRef(gNodeRefId, pNode->rid);
}
}
int64_t syncHbTimerDataAdd(SSyncHbTimerData *pData) {
pData->rid = taosAddRef(gHbDataRefId, pData);
if (pData->rid < 0) return TSDB_CODE_SYN_WRONG_REF;
if (pData->rid < 0) {
return terrno = TSDB_CODE_SYN_WRONG_REF;
}
return pData->rid;
}
void syncHbTimerDataRemove(int64_t rid) { (void)taosRemoveRef(gHbDataRefId, rid); }
void syncHbTimerDataRemove(int64_t rid) {
if (rid > 0) {
(void)taosRemoveRef(gHbDataRefId, rid);
}
}
SSyncHbTimerData *syncHbTimerDataAcquire(int64_t rid) {
SSyncHbTimerData *pData = taosAcquireRef(gHbDataRefId, rid);
if (pData == NULL && rid > 0) {
sInfo("failed to acquire hb-timer-data from refId:%" PRId64, rid);
sInfo("failed to acquire hbdata from refId:%" PRId64 ", rsetId:%d", rid, gHbDataRefId);
terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
}
return pData;
}
void syncHbTimerDataRelease(SSyncHbTimerData *pData) { (void)taosReleaseRef(gHbDataRefId, pData->rid); }
#if 0
void syncEnvStartTimer() {
taosTmrReset(gSyncEnv.FpEnvTickTimer, gSyncEnv.envTickTimerMS, &gSyncEnv, gSyncEnv.pTimerManager,
&gSyncEnv.pEnvTickTimer);
atomic_store_64(&gSyncEnv.envTickTimerLogicClock, gSyncEnv.envTickTimerLogicClockUser);
void syncHbTimerDataRelease(SSyncHbTimerData *pData) {
if (pData) {
(void)taosReleaseRef(gHbDataRefId, pData->rid);
}
void syncEnvStopTimer() {
int32_t ret = 0;
atomic_add_fetch_64(&gSyncEnv.envTickTimerLogicClockUser, 1);
taosTmrStop(gSyncEnv.pEnvTickTimer);
gSyncEnv.pEnvTickTimer = NULL;
return ret;
}
#endif
static void syncEnvTick(void *param, void *tmrId) {
#if 0
SSyncEnv *pSyncEnv = param;
if (atomic_load_64(&gSyncEnv.envTickTimerLogicClockUser) <= atomic_load_64(&gSyncEnv.envTickTimerLogicClock)) {
gSyncEnv.envTickTimerCounter++;
sTrace("syncEnvTick do ... envTickTimerLogicClockUser:%" PRIu64 ", envTickTimerLogicClock:%" PRIu64
", envTickTimerCounter:%" PRIu64 ", envTickTimerMS:%d, tmrId:%p",
gSyncEnv.envTickTimerLogicClockUser, gSyncEnv.envTickTimerLogicClock, gSyncEnv.envTickTimerCounter,
gSyncEnv.envTickTimerMS, tmrId);
// do something, tick ...
taosTmrReset(syncEnvTick, gSyncEnv.envTickTimerMS, pSyncEnv, gSyncEnv.pTimerManager, &gSyncEnv.pEnvTickTimer);
} else {
sTrace("syncEnvTick pass ... envTickTimerLogicClockUser:%" PRIu64 ", envTickTimerLogicClock:%" PRIu64
", envTickTimerCounter:%" PRIu64 ", envTickTimerMS:%d, tmrId:%p",
gSyncEnv.envTickTimerLogicClockUser, gSyncEnv.envTickTimerLogicClock, gSyncEnv.envTickTimerCounter,
gSyncEnv.envTickTimerMS, tmrId);
}
#endif
}

78
source/libs/sync/src/syncMain.c
View File

@ -50,7 +50,7 @@ static int32_t syncHbTimerStart(SSyncNode* pSyncNode, SSyncTimer* pSyncTimer);
static int32_t syncHbTimerStop(SSyncNode* pSyncNode, SSyncTimer* pSyncTimer);
static int32_t syncNodeUpdateNewConfigIndex(SSyncNode* ths, SSyncCfg* pNewCfg);
static bool syncNodeInConfig(SSyncNode* pSyncNode, const SSyncCfg* config);
static void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* newConfig, SyncIndex lastConfigChangeIndex);
static int32_t syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* newConfig, SyncIndex lastConfigChangeIndex);
static bool syncNodeIsOptimizedOneReplica(SSyncNode* ths, SRpcMsg* pMsg);
static bool syncNodeCanChange(SSyncNode* pSyncNode);
@ -182,7 +182,12 @@ int32_t syncReconfig(int64_t rid, SSyncCfg* pNewCfg) {
}
TAOS_CHECK_RETURN(syncNodeUpdateNewConfigIndex(pSyncNode, pNewCfg));
syncNodeDoConfigChange(pSyncNode, pNewCfg, pNewCfg->lastIndex);
if (syncNodeDoConfigChange(pSyncNode, pNewCfg, pNewCfg->lastIndex) != 0) {
code = TSDB_CODE_SYN_NEW_CONFIG_ERROR;
sError("vgId:%d, failed to reconfig since do change error", pSyncNode->vgId);
TAOS_RETURN(code);
}
if (pSyncNode->state == TAOS_SYNC_STATE_LEADER || pSyncNode->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
// TODO check return value
@ -1015,7 +1020,7 @@ SSyncNode* syncNodeOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion) {
if (!taosDirExist((char*)(pSyncInfo->path))) {
if (taosMkDir(pSyncInfo->path) != 0) {
terrno = TAOS_SYSTEM_ERROR(errno);
sError("failed to create dir:%s since %s", pSyncInfo->path, terrstr());
sError("vgId:%d, failed to create dir:%s since %s", pSyncInfo->vgId, pSyncInfo->path, terrstr());
goto _error;
}
}
@ -1108,37 +1113,6 @@ SSyncNode* syncNodeOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion) {
goto _error;
}
// init internal
pSyncNode->myNodeInfo = pSyncNode->raftCfg.cfg.nodeInfo[pSyncNode->raftCfg.cfg.myIndex];
if (!syncUtilNodeInfo2RaftId(&pSyncNode->myNodeInfo, pSyncNode->vgId, &pSyncNode->myRaftId)) {
terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
sError("vgId:%d, failed to determine my raft member id", pSyncNode->vgId);
goto _error;
}
pSyncNode->arbTerm = -1;
(void)taosThreadMutexInit(&pSyncNode->arbTokenMutex, NULL);
syncUtilGenerateArbToken(pSyncNode->myNodeInfo.nodeId, pSyncInfo->vgId, pSyncNode->arbToken);
sInfo("vgId:%d, arb token:%s", pSyncNode->vgId, pSyncNode->arbToken);
// init peersNum, peers, peersId
pSyncNode->peersNum = pSyncNode->raftCfg.cfg.totalReplicaNum - 1;
int32_t j = 0;
for (int32_t i = 0; i < pSyncNode->raftCfg.cfg.totalReplicaNum; ++i) {
if (i != pSyncNode->raftCfg.cfg.myIndex) {
pSyncNode->peersNodeInfo[j] = pSyncNode->raftCfg.cfg.nodeInfo[i];
syncUtilNodeInfo2EpSet(&pSyncNode->peersNodeInfo[j], &pSyncNode->peersEpset[j]);
j++;
}
}
for (int32_t i = 0; i < pSyncNode->peersNum; ++i) {
if (!syncUtilNodeInfo2RaftId(&pSyncNode->peersNodeInfo[i], pSyncNode->vgId, &pSyncNode->peersId[i])) {
terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
sError("vgId:%d, failed to determine raft member id, peer:%d", pSyncNode->vgId, i);
goto _error;
}
}
// init replicaNum, replicasId
pSyncNode->replicaNum = pSyncNode->raftCfg.cfg.replicaNum;
pSyncNode->totalReplicaNum = pSyncNode->raftCfg.cfg.totalReplicaNum;
@ -1150,6 +1124,27 @@ SSyncNode* syncNodeOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion) {
}
}
// init internal
pSyncNode->myNodeInfo = pSyncNode->raftCfg.cfg.nodeInfo[pSyncNode->raftCfg.cfg.myIndex];
pSyncNode->myRaftId = pSyncNode->replicasId[pSyncNode->raftCfg.cfg.myIndex];
// init peersNum, peers, peersId
pSyncNode->peersNum = pSyncNode->raftCfg.cfg.totalReplicaNum - 1;
int32_t j = 0;
for (int32_t i = 0; i < pSyncNode->raftCfg.cfg.totalReplicaNum; ++i) {
if (i != pSyncNode->raftCfg.cfg.myIndex) {
pSyncNode->peersNodeInfo[j] = pSyncNode->raftCfg.cfg.nodeInfo[i];
pSyncNode->peersId[j] = pSyncNode->replicasId[i];
syncUtilNodeInfo2EpSet(&pSyncNode->peersNodeInfo[j], &pSyncNode->peersEpset[j]);
j++;
}
}
pSyncNode->arbTerm = -1;
(void)taosThreadMutexInit(&pSyncNode->arbTokenMutex, NULL);
syncUtilGenerateArbToken(pSyncNode->myNodeInfo.nodeId, pSyncInfo->vgId, pSyncNode->arbToken);
sInfo("vgId:%d, generate arb token:%s", pSyncNode->vgId, pSyncNode->arbToken);
// init raft algorithm
pSyncNode->pFsm = pSyncInfo->pFsm;
pSyncInfo->pFsm = NULL;
@ -1766,11 +1761,11 @@ static bool syncIsConfigChanged(const SSyncCfg* pOldCfg, const SSyncCfg* pNewCfg
return false;
}
void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncIndex lastConfigChangeIndex) {
int32_t syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncIndex lastConfigChangeIndex) {
SSyncCfg oldConfig = pSyncNode->raftCfg.cfg;
if (!syncIsConfigChanged(&oldConfig, pNewConfig)) {
sInfo("vgId:1, sync not reconfig since not changed");
return;
return 0;
}
pSyncNode->raftCfg.cfg = *pNewConfig;
@ -1809,7 +1804,15 @@ void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncInde
}
// add last config index
(void)syncAddCfgIndex(pSyncNode, lastConfigChangeIndex);
SRaftCfg* pCfg = &pSyncNode->raftCfg;
if (pCfg->configIndexCount >= MAX_CONFIG_INDEX_COUNT) {
sNError(pSyncNode, "failed to add cfg index:%d since out of range", pCfg->configIndexCount);
terrno = TSDB_CODE_OUT_OF_RANGE;
return -1;
}
pCfg->configIndexArr[pCfg->configIndexCount] = lastConfigChangeIndex;
pCfg->configIndexCount++;
if (IamInNew) {
//-----------------------------------------
@ -1924,6 +1927,7 @@ void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncInde
_END:
// log end config change
sNInfo(pSyncNode, "end do config change, from %d to %d", oldConfig.totalReplicaNum, pNewConfig->totalReplicaNum);
return 0;
}
// raft state change --------------

42
source/libs/sync/src/syncRaftCfg.c
View File

@ -18,7 +18,7 @@
#include "syncUtil.h"
#include "tjson.h"
const char *syncRoleToStr(ESyncRole role) {
static const char *syncRoleToStr(ESyncRole role) {
switch (role) {
case TAOS_SYNC_ROLE_VOTER:
return "true";
@ -29,16 +29,15 @@ const char *syncRoleToStr(ESyncRole role) {
}
}
const ESyncRole syncStrToRole(char *str) {
static const ESyncRole syncStrToRole(char *str) {
if (strcmp(str, "true") == 0) {
return TAOS_SYNC_ROLE_VOTER;
}
if (strcmp(str, "false") == 0) {
} else if (strcmp(str, "false") == 0) {
return TAOS_SYNC_ROLE_LEARNER;
}
} else {
return TAOS_SYNC_ROLE_ERROR;
}
}
static int32_t syncEncodeSyncCfg(const void *pObj, SJson *pJson) {
SSyncCfg *pCfg = (SSyncCfg *)pObj;
@ -52,10 +51,12 @@ static int32_t syncEncodeSyncCfg(const void *pObj, SJson *pJson) {
if (nodeInfo == NULL) {
TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY);
}
if ((code = tjsonAddItemToObject(pJson, "nodeInfo", nodeInfo)) < 0) {
tjsonDelete(nodeInfo);
TAOS_CHECK_EXIT(code);
}
for (int32_t i = 0; i < pCfg->totalReplicaNum; ++i) {
SJson *info = tjsonCreateObject();
if (info == NULL) {
@ -68,20 +69,25 @@ static int32_t syncEncodeSyncCfg(const void *pObj, SJson *pJson) {
TAOS_CHECK_GOTO(tjsonAddStringToObject(info, "isReplica", syncRoleToStr(pCfg->nodeInfo[i].nodeRole)), NULL, _err);
TAOS_CHECK_GOTO(tjsonAddItemToArray(nodeInfo, info), NULL, _err);
continue;
_err:
tjsonDelete(info);
break;
}
_exit:
if (code < 0) {
sError("failed to encode sync cfg at line %d since %s", lino, tstrerror(code));
}
TAOS_RETURN(code);
}
static int32_t syncEncodeRaftCfg(const void *pObj, SJson *pJson) {
SRaftCfg *pCfg = (SRaftCfg *)pObj;
int32_t code = 0, lino = 0;
int32_t code = 0;
int32_t lino = 0;
TAOS_CHECK_EXIT(tjsonAddObject(pJson, "SSyncCfg", syncEncodeSyncCfg, (void *)&pCfg->cfg));
TAOS_CHECK_EXIT(tjsonAddDoubleToObject(pJson, "isStandBy", pCfg->isStandBy));
TAOS_CHECK_EXIT(tjsonAddDoubleToObject(pJson, "snapshotStrategy", pCfg->snapshotStrategy));
@ -93,10 +99,12 @@ static int32_t syncEncodeRaftCfg(const void *pObj, SJson *pJson) {
if (configIndexArr == NULL) {
TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY);
}
if ((code = tjsonAddItemToObject(pJson, "configIndexArr", configIndexArr)) < 0) {
tjsonDelete(configIndexArr);
TAOS_CHECK_EXIT(code);
}
for (int32_t i = 0; i < pCfg->configIndexCount; ++i) {
SJson *configIndex = tjsonCreateObject();
if (configIndex == NULL) {
@ -105,14 +113,17 @@ static int32_t syncEncodeRaftCfg(const void *pObj, SJson *pJson) {
TAOS_CHECK_EXIT(tjsonAddIntegerToObject(configIndex, "index", pCfg->configIndexArr[i]));
TAOS_CHECK_EXIT(tjsonAddItemToArray(configIndexArr, configIndex));
continue;
_err:
tjsonDelete(configIndex);
break;
}
_exit:
if (code < 0) {
sError("failed to encode raft cfg at line %d since %s", lino, tstrerror(code));
}
TAOS_RETURN(code);
}
@ -124,11 +135,13 @@ int32_t syncWriteCfgFile(SSyncNode *pNode) {
const char *realfile = pNode->configPath;
SRaftCfg *pCfg = &pNode->raftCfg;
char file[PATH_MAX] = {0};
(void)snprintf(file, sizeof(file), "%s.bak", realfile);
if ((pJson = tjsonCreateObject()) == NULL) {
TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY);
}
TAOS_CHECK_EXIT(tjsonAddObject(pJson, "RaftCfg", syncEncodeRaftCfg, pCfg));
buffer = tjsonToString(pJson);
if (buffer == NULL) {
@ -145,6 +158,7 @@ int32_t syncWriteCfgFile(SSyncNode *pNode) {
if (taosWriteFile(pFile, buffer, len) <= 0) {
TAOS_CHECK_EXIT(TAOS_SYSTEM_ERROR(errno));
}
if (taosFsyncFile(pFile) < 0) {
TAOS_CHECK_EXIT(TAOS_SYSTEM_ERROR(errno));
}
@ -165,6 +179,7 @@ _exit:
if (code != 0) {
sError("vgId:%d, failed to write sync cfg file:%s since %s", pNode->vgId, realfile, tstrerror(code));
}
TAOS_RETURN(code);
}
@ -232,6 +247,7 @@ static int32_t syncDecodeRaftCfg(const SJson *pJson, void *pObj) {
tjsonGetNumberValue(configIndex, "index", pCfg->configIndexArr[i], code);
if (code < 0) return TSDB_CODE_INVALID_JSON_FORMAT;
}
return 0;
}
@ -292,16 +308,6 @@ _OVER:
if (code != 0) {
sError("vgId:%d, failed to read sync cfg file:%s since %s", pNode->vgId, file, tstrerror(code));
}
TAOS_RETURN(code);
}
int32_t syncAddCfgIndex(SSyncNode *pNode, SyncIndex cfgIndex) {
SRaftCfg *pCfg = &pNode->raftCfg;
if (pCfg->configIndexCount >= MAX_CONFIG_INDEX_COUNT) {
return TSDB_CODE_OUT_OF_RANGE;
}
pCfg->configIndexArr[pCfg->configIndexCount] = cfgIndex;
pCfg->configIndexCount++;
return 0;
}

39
source/libs/sync/src/syncUtil.c
View File

@ -23,7 +23,7 @@
#include "syncSnapshot.h"
#include "tglobal.h"
void syncCfg2SimpleStr(const SSyncCfg* pCfg, char* buf, int32_t bufLen) {
static void syncCfg2SimpleStr(const SSyncCfg* pCfg, char* buf, int32_t bufLen) {
int32_t len = snprintf(buf, bufLen, "{num:%d, as:%d, [", pCfg->replicaNum, pCfg->myIndex);
for (int32_t i = 0; i < pCfg->replicaNum; ++i) {
len += snprintf(buf + len, bufLen - len, "%s:%d", pCfg->nodeInfo[i].nodeFqdn, pCfg->nodeInfo[i].nodePort);
@ -43,14 +43,11 @@ void syncUtilNodeInfo2EpSet(const SNodeInfo* pInfo, SEpSet* pEpSet) {
bool syncUtilNodeInfo2RaftId(const SNodeInfo* pInfo, SyncGroupId vgId, SRaftId* raftId) {
uint32_t ipv4 = 0xFFFFFFFF;
sDebug(
"vgId:%d, start to resolve sync addr fqdn in %d seconds, "
"dnode:%d cluster:%" PRId64 " fqdn:%s port:%u ",
vgId, tsResolveFQDNRetryTime, pInfo->nodeId, pInfo->clusterId, pInfo->nodeFqdn, pInfo->nodePort);
for (int i = 0; i < tsResolveFQDNRetryTime; i++) {
sDebug("vgId:%d, resolve sync addr from fqdn, ep:%s:%u", vgId, pInfo->nodeFqdn, pInfo->nodePort);
for (int32_t i = 0; i < tsResolveFQDNRetryTime; i++) {
int32_t code = taosGetIpv4FromFqdn(pInfo->nodeFqdn, &ipv4);
if (code) {
sError("failed to resolve ipv4 addr, fqdn:%s, wait one second", pInfo->nodeFqdn);
sError("vgId:%d, failed to resolve sync addr, dnode:%d fqdn:%s, retry", vgId, pInfo->nodeId, pInfo->nodeFqdn);
taosSsleep(1);
} else {
break;
@ -58,7 +55,7 @@ bool syncUtilNodeInfo2RaftId(const SNodeInfo* pInfo, SyncGroupId vgId, SRaftId*
}
if (ipv4 == 0xFFFFFFFF || ipv4 == 1) {
sError("failed to resolve ipv4 addr, fqdn:%s", pInfo->nodeFqdn);
sError("vgId:%d, failed to resolve sync addr, dnode:%d fqdn:%s", vgId, pInfo->nodeId, pInfo->nodeFqdn);
terrno = TSDB_CODE_TSC_INVALID_FQDN;
return false;
}
@ -68,14 +65,20 @@ bool syncUtilNodeInfo2RaftId(const SNodeInfo* pInfo, SyncGroupId vgId, SRaftId*
raftId->addr = SYNC_ADDR(pInfo);
raftId->vgId = vgId;
sInfo("vgId:%d, sync addr:%" PRIu64 ", dnode:%d cluster:%" PRId64 " fqdn:%s ip:%s port:%u ipv4:%u", vgId,
raftId->addr, pInfo->nodeId, pInfo->clusterId, pInfo->nodeFqdn, ipbuf, pInfo->nodePort, ipv4);
sInfo("vgId:%d, sync addr:%" PRIu64 " is resolved, ep:%s:%u ip:%s ipv4:%u dnode:%d cluster:%" PRId64, vgId,
raftId->addr, pInfo->nodeFqdn, pInfo->nodePort, ipbuf, ipv4, pInfo->nodeId, pInfo->clusterId);
return true;
}
bool syncUtilSameId(const SRaftId* pId1, const SRaftId* pId2) {
if (pId1->addr == pId2->addr && pId1->vgId == pId2->vgId) return true;
if ((CID(pId1) == 0 || CID(pId2) == 0) && (DID(pId1) == DID(pId2)) && pId1->vgId == pId2->vgId) return true;
if (pId1->addr == pId2->addr && pId1->vgId == pId2->vgId) {
return true;
}
if ((CID(pId1) == 0 || CID(pId2) == 0) && (DID(pId1) == DID(pId2)) && pId1->vgId == pId2->vgId) {
return true;
}
return false;
}
@ -98,10 +101,6 @@ void syncUtilMsgHtoN(void* msg) {
pHead->vgId = htonl(pHead->vgId);
}
bool syncUtilUserPreCommit(tmsg_t msgType) { return msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_LEADER_TRANSFER; }
bool syncUtilUserRollback(tmsg_t msgType) { return msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_LEADER_TRANSFER; }
void syncUtilGenerateArbToken(int32_t nodeId, int32_t groupId, char* buf) {
(void)memset(buf, 0, TSDB_ARB_TOKEN_SIZE);
int32_t randVal = taosSafeRand() % 1000;
@ -142,18 +141,18 @@ static void syncLogBufferStates2Str(SSyncNode* pSyncNode, char* buf, int32_t buf
if (pBuf == NULL) {
return;
}
int len = 0;
int32_t len = 0;
len += snprintf(buf + len, bufLen - len, "[%" PRId64 " %" PRId64 " %" PRId64 ", %" PRId64 ")", pBuf->startIndex,
pBuf->commitIndex, pBuf->matchIndex, pBuf->endIndex);
}
static void syncLogReplStates2Str(SSyncNode* pSyncNode, char* buf, int32_t bufLen) {
int len = 0;
int32_t len = 0;
len += snprintf(buf + len, bufLen - len, "%s", "{");
for (int32_t i = 0; i < pSyncNode->replicaNum; i++) {
SSyncLogReplMgr* pMgr = pSyncNode->logReplMgrs[i];
if (pMgr == NULL) break;
len += snprintf(buf + len, bufLen - len, "%d:%d [%" PRId64 " %" PRId64 ", %" PRId64 ")", i, pMgr->restored,
len += snprintf(buf + len, bufLen - len, "%d:%d [%" PRId64 " %" PRId64 ", %" PRId64 "]", i, pMgr->restored,
pMgr->startIndex, pMgr->matchIndex, pMgr->endIndex);
if (i + 1 < pSyncNode->replicaNum) {
len += snprintf(buf + len, bufLen - len, "%s", ", ");
@ -280,7 +279,7 @@ void syncPrintSnapshotSenderLog(const char* flags, ELogLevel level, int32_t dfla
" end:%" PRId64 " last-index:%" PRId64 " last-term:%" PRId64 " last-cfg:%" PRId64
", seq:%d, ack:%d, "
" buf:[%" PRId64 " %" PRId64 ", %" PRId64
"), finish:%d, as:%d, to-dnode:%d}"
"], finish:%d, as:%d, to-dnode:%d}"
", term:%" PRIu64 ", commit-index:%" PRId64 ", firstver:%" PRId64 ", lastver:%" PRId64
", min-match:%" PRId64 ", snap:{last-index:%" PRId64 ", term:%" PRIu64
"}, standby:%d, batch-sz:%d, replicas:%d, last-cfg:%" PRId64

7
source/libs/transport/src/transCli.c
View File

@ -2957,6 +2957,7 @@ int32_t transSendRequest(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, S
STrans* pTransInst = (STrans*)transAcquireExHandle(transGetInstMgt(), (int64_t)shandle);
if (pTransInst == NULL) {
transFreeMsg(pReq->pCont);
pReq->pCont = NULL;
return TSDB_CODE_RPC_MODULE_QUIT;
}
int32_t code = 0;
@ -3008,6 +3009,7 @@ int32_t transSendRequest(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, S
_exception:
transFreeMsg(pReq->pCont);
pReq->pCont = NULL;
(void)transReleaseExHandle(transGetInstMgt(), (int64_t)shandle);
return code;
}
@ -3053,6 +3055,7 @@ int32_t transSendRequestWithId(void* shandle, const SEpSet* pEpSet, STransMsg* p
_exception:
transFreeMsg(pReq->pCont);
pReq->pCont = NULL;
(void)transReleaseExHandle(transGetInstMgt(), (int64_t)shandle);
return code;
}
@ -3061,6 +3064,7 @@ int32_t transSendRecv(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, STra
STrans* pTransInst = (STrans*)transAcquireExHandle(transGetInstMgt(), (int64_t)shandle);
if (pTransInst == NULL) {
transFreeMsg(pReq->pCont);
pReq->pCont = NULL;
return TSDB_CODE_RPC_MODULE_QUIT;
}
int32_t code = 0;
@ -3139,6 +3143,7 @@ _RETURN1:
(void)transReleaseExHandle(transGetInstMgt(), (int64_t)shandle);
taosMemoryFree(pTransRsp);
taosMemoryFree(pReq->pCont);
pReq->pCont = NULL;
return code;
}
int32_t transCreateSyncMsg(STransMsg* pTransMsg, int64_t* refId) {
@ -3183,6 +3188,7 @@ int32_t transSendRecvWithTimeout(void* shandle, SEpSet* pEpSet, STransMsg* pReq,
STrans* pTransInst = (STrans*)transAcquireExHandle(transGetInstMgt(), (int64_t)shandle);
if (pTransInst == NULL) {
transFreeMsg(pReq->pCont);
pReq->pCont = NULL;
return TSDB_CODE_RPC_MODULE_QUIT;
}
@ -3263,6 +3269,7 @@ _RETURN:
return code;
_RETURN2:
transFreeMsg(pReq->pCont);
pReq->pCont = NULL;
taosMemoryFree(pTransMsg);
(void)transReleaseExHandle(transGetInstMgt(), (int64_t)shandle);
return code;

2
source/libs/transport/src/transComm.c
View File

@ -736,7 +736,7 @@ int32_t transOpenRefMgt(int size, void (*func)(void*)) {
}
void transCloseRefMgt(int32_t mgt) {
// close ref
(void)taosCloseRef(mgt);
taosCloseRef(mgt);
}
int64_t transAddExHandle(int32_t refMgt, void* p) {
// acquire extern handle

2
source/libs/wal/src/walMgmt.c
View File

@ -75,7 +75,7 @@ void walCleanUp() {
if (old == 1) {
walStopThread();
TAOS_UNUSED(taosCloseRef(tsWal.refSetId));
taosCloseRef(tsWal.refSetId);
wInfo("wal module is cleaned up");
atomic_store_8(&tsWal.inited, 0);
}

5
source/util/src/terror.c
View File

@ -478,6 +478,9 @@ TAOS_DEFINE_ERROR(TSDB_CODE_QRY_INVALID_WINDOW_CONDITION, "The time pseudo colum
TAOS_DEFINE_ERROR(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR, "Executor internal error")
TAOS_DEFINE_ERROR(TSDB_CODE_QRY_INVALID_JOIN_CONDITION, "Not supported join on condition")
TAOS_DEFINE_ERROR(TSDB_CODE_QRY_FILTER_NOT_SUPPORT_TYPE, "Not supported range type")
TAOS_DEFINE_ERROR(TSDB_CODE_QRY_FILTER_WRONG_OPTR_TYPE, "Wrong operator type")
TAOS_DEFINE_ERROR(TSDB_CODE_QRY_FILTER_RANGE_ERROR, "Wrong filter range")
TAOS_DEFINE_ERROR(TSDB_CODE_QRY_FILTER_INVALID_TYPE, "Invalid filter type")
// grant
TAOS_DEFINE_ERROR(TSDB_CODE_GRANT_EXPIRED, "License expired")
@ -724,6 +727,8 @@ TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_TIME_UNIT_TOO_SMALL, "Function time unit c
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_INVALID_VALUE_RANGE, "Function got invalid value range")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_SETUP_ERROR, "Function set up failed")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_INVALID_RES_LENGTH, "Function result exceed max length")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_HISTOGRAM_ERROR, "Function failed to calculate histogram")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_PERCENTILE_ERROR, "Function failed to calculate percentile")
//udf
TAOS_DEFINE_ERROR(TSDB_CODE_UDF_STOPPING, "udf is stopping")

6
source/util/src/tref.c
View File

@ -110,13 +110,13 @@ int32_t taosOpenRef(int32_t max, RefFp fp) {
return rsetId;
}
int32_t taosCloseRef(int32_t rsetId) {
void taosCloseRef(int32_t rsetId) {
SRefSet *pSet;
int32_t deleted = 0;
if (rsetId < 0 || rsetId >= TSDB_REF_OBJECTS) {
uTrace("rsetId:%d is invalid, out of range", rsetId);
return terrno = TSDB_CODE_REF_INVALID_ID;
return;
}
pSet = tsRefSetList + rsetId;
@ -134,8 +134,6 @@ int32_t taosCloseRef(int32_t rsetId) {
(void)taosThreadMutexUnlock(&tsRefMutex);
if (deleted) taosDecRsetCount(pSet);
return 0;
}
int64_t taosAddRef(int32_t rsetId, void *p) {

3
source/util/src/tscalablebf.c
View File

@ -87,6 +87,7 @@ int32_t tScalableBfPutNoCheck(SScalableBf* pSBf, const void* keyBuf, uint32_t le
pSBf->status = SBF_INVALID;
if (code == TSDB_CODE_OUT_OF_BUFFER) {
code = TSDB_CODE_SUCCESS;
return code;
}
QUERY_CHECK_CODE(code, lino, _error);
}
@ -126,6 +127,8 @@ int32_t tScalableBfPut(SScalableBf* pSBf, const void* keyBuf, uint32_t len, int3
pSBf->status = SBF_INVALID;
if (code == TSDB_CODE_OUT_OF_BUFFER) {
code = TSDB_CODE_SUCCESS;
(*winRes) = TSDB_CODE_FAILED;
goto _end;
}
QUERY_CHECK_CODE(code, lino, _end);
}

42
tests/script/tsim/show/showalive.sim
View File

@ -21,6 +21,30 @@ sql create dnode $hostname4 port 7500
sleep 1000
$loop_count = 0
loop00:
sleep 1000
$loop_count = $loop_count + 1
if $loop_count == 20 then
return -1
endi
print 0 show cluster alive;
sql show cluster alive;
print res------------------------
print $data00 $data01
print $data10 $data11
if $data00 != 1 then
print =====data00=$data00
goto loop00
endi
print =============== create database, stable, table
sql create database test vgroups 6;
sql use test;
@ -46,6 +70,15 @@ endi
print show cluster alive;
sql show cluster alive;
print res------------------------
print $data00 $data01
print $data10 $data11
if $rows != 1 then
print =====rows=$rows
goto loop0
endi
if $data00 != 1 then
print =====data00=$data00
goto loop0
@ -54,6 +87,15 @@ endi
print show test.alive;
sql show test.alive;
print res------------------------
print $data00 $data01
print $data10 $data11
if $rows != 1 then
print =====rows=$rows
goto loop0
endi
if $data00 != 1 then
print =====data00=$data00
goto loop0