xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch '3.0' into enh/TD-31535/assert

This commit is contained in:
xinsheng Ren 2024-08-22 13:26:06 +08:00 committed by GitHub
parent 3f1fd2b0ee dacf36e433
commit 3b631ddac7
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
175 changed files with 2375 additions and 1604 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -134,3 +134,4 @@ tags
.clangd
*CMakeCache*
*CMakeFiles*
.history/

0
docs/zh/14-reference/03-taos-sql/25-grant.md → docs/zh/14-reference/03-taos-sql/25-user.md
View File

168
docs/zh/14-reference/03-taos-sql/26-grant.md Normal file
View File

@ -0,0 +1,168 @@
---
toc_max_heading_level: 4
title: 权限管理
---
TDengine 中的权限管理分为[用户管理](../user)、数据库授权管理以及消息订阅授权管理，本节重点说明数据库授权和订阅授权。
## 数据库访问授权
系统管理员可以根据业务需要对系统中的每个用户针对每个数据库进行特定的授权，以防止业务数据被不恰当的用户读取或修改。对某个用户进行数据库访问授权的语法如下：
```sql
GRANT privileges ON priv_level TO user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
```
对数据库的访问权限包含读和写两种权限，它们可以被分别授予，也可以被同时授予。
说明
- priv_level 格式中 "." 之前为数据库名称， "." 之后为表名称，意思为表级别的授权控制。如果 "." 之后为 "\*" ，意为 "." 前所指定的数据库中的所有表
- "dbname.\*" 意思是名为 "dbname" 的数据库中的所有表
- "\*.\*" 意思是所有数据库名中的所有表
### 数据库权限说明
对 root 用户和普通用户的权限的说明如下表
| 用户 | 描述 | 权限说明 |
| -------- | ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 超级用户 | 只有 root 是超级用户 | DB 外部 所有操作权限，例如user、dnode、udf、qnode等的CRUD DB 权限，包括 创建 删除 更新，例如修改 Option，移动 Vgruop等 读 写 Enable/Disable 用户 |
| 普通用户 | 除 root 以外的其它用户均为普通用户 | 在可读的 DB 中，普通用户可以进行读操作 select describe show subscribe 在可写 DB 的内部，用户可以进行写操作： 创建、删除、修改 超级表 创建、删除、修改 子表 创建、删除、修改 topic 写入数据 被限制系统信息时，不可进行如下操作 show dnode、mnode、vgroups、qnode、snode 修改用户包括自身密码 show db时只能看到自己的db，并且不能看到vgroups、副本、cache等信息 无论是否被限制系统信息，都可以 管理 udf 可以创建 DB 自己创建的 DB 具备所有权限 非自己创建的 DB ，参照读、写列表中的权限 |
## 消息订阅授权
任意用户都可以在自己拥有读权限的数据库上创建 topic。超级用户 root 可以在任意数据库上创建 topic。每个 topic 的订阅权限都可以被独立授权给任何用户，不管该用户是否拥有该数据库的访问权限。删除 topic 只能由 root 用户或者该 topic 的创建者进行。topic 只能由超级用户、topic的创建者或者被显式授予 subscribe 权限的用户订阅。
具体的 SQL 语法如下：
```sql
GRANT SUBSCRIBE ON topic_name TO user_name
REVOKE SUBSCRIBE ON topic_name FROM user_name
```
## 基于标签的授权（表级授权）
从 TDengine 3.0.5.0 开始，我们支持按标签授权某个超级表中部分特定的子表。具体的 SQL 语法如下。
```sql
GRANT privileges ON priv_level [WITH tag_condition] TO user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
REVOKE privileges ON priv_level [WITH tag_condition] FROM user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
```
上面 SQL 的语义为：
- 用户可以通过 dbname.tbname 来为指定的表（包括超级表和普通表）授予或回收其读写权限，不支持直接对子表授予或回收权限。
- 用户可以通过 dbname.tbname 和 WITH 子句来为符合条件的所有子表授予或回收其读写权限。使用 WITH 子句时，权限级别必须为超级表。
## 表级权限和数据库权限的关系
下表列出了在不同的数据库授权和表级授权的组合下产生的实际权限。
| | **表无授权** | **表读授权** | **表读授权有标签条件** | **表写授权** | **表写授权有标签条件** |
| ---------------- | ---------------- | ---------------------------------------- | ------------------------------------------------------------ | ---------------------------------------- | ---------------------------------------------------------- |
| **数据库无授权** | 无授权 | 对此表有读权限，对数据库下的其他表无权限 | 对此表符合标签权限的子表有读权限，对数据库下的其他表无权限 | 对此表有写权限，对数据库下的其他表无权限 | 对此表符合标签权限的子表有写权限，对数据库下的其他表无权限 |
| **数据库读授权** | 对所有表有读权限 | 对所有表有读权限 | 对此表符合标签权限的子表有读权限，对数据库下的其他表有读权限 | 对此表有写权限，对所有表有读权限 | 对此表符合标签权限的子表有写权限，所有表有读权限 |
| **数据库写授权** | 对所有表有写权限 | 对此表有读权限，对所有表有写权限 | 对此表符合标签权限的子表有读权限，对所有表有写权限 | 对所有表有写权限 | 对此表符合标签权限的子表有写权限，数据库下的其他表有写权限 |
## 查看用户授权
使用下面的命令可以显示一个用户所拥有的授权：
```sql
show user privileges
```
## 撤销授权
1. 撤销数据库访问的授权
```sql
REVOKE privileges ON priv_level FROM user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
READ
| WRITE
}
priv_level : {
dbname.tbname
| dbname.*
| *.*
}
```
2. 撤销数据订阅的授权
```sql
REVOKE privileges ON priv_level FROM user_name
privileges : {
ALL
| priv_type [, priv_type] ...
}
priv_type : {
SUBSCRIBE
}
priv_level : {
topic_name
}
```

0
docs/zh/14-reference/03-taos-sql/26-udf.md → docs/zh/14-reference/03-taos-sql/27-udf.md
View File

0
docs/zh/14-reference/03-taos-sql/27-indexing.md → docs/zh/14-reference/03-taos-sql/28-index.md
View File

0
docs/zh/14-reference/03-taos-sql/28-recovery.md → docs/zh/14-reference/03-taos-sql/29-recovery.md
View File

0
docs/zh/14-reference/03-taos-sql/29-changes.md → docs/zh/14-reference/03-taos-sql/30-changes.md
View File

0
docs/zh/14-reference/03-taos-sql/30-join.md → docs/zh/14-reference/03-taos-sql/31-join.md
View File

0
docs/zh/14-reference/03-taos-sql/31-compress.md → docs/zh/14-reference/03-taos-sql/32-compress.md
View File

0
docs/zh/14-reference/03-taos-sql/32-view.md → docs/zh/14-reference/03-taos-sql/33-view.md
View File

40
docs/zh/20-third-party/01-collection/11-kafka.md vendored
View File

@ -27,22 +27,21 @@ TDengine Source Connector 用于把数据实时地从 TDengine 读出来发送
## 安装 Kafka
在任意目录下执行：
- 在任意目录下执行：
```shell
curl -O https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
tar xzf kafka_2.13-3.4.0.tgz -C /opt/
ln -s /opt/kafka_2.13-3.4.0 /opt/kafka
```
```shell
curl -O https://downloads.apache.org/kafka/3.4.0/kafka_2.13-3.4.0.tgz
tar xzf kafka_2.13-3.4.0.tgz -C /opt/
ln -s /opt/kafka_2.13-3.4.0 /opt/kafka
```
然后需要把 `$KAFKA_HOME/bin` 目录加入 PATH。
- 然后需要把 `$KAFKA_HOME/bin` 目录加入 PATH。
```title=".profile"
export KAFKA_HOME=/opt/kafka
export PATH=$PATH:$KAFKA_HOME/bin
```
以上脚本可以追加到当前用户的 profile 文件（~/.profile 或 ~/.bash_profile）
```title=".profile"
export KAFKA_HOME=/opt/kafka
export PATH=$PATH:$KAFKA_HOME/bin
```
以上脚本可以追加到当前用户的 profile 文件（~/.profile 或 ~/.bash_profile）
## 安装 TDengine Connector 插件
@ -325,6 +324,21 @@ curl -X DELETE http://localhost:8083/connectors/TDengineSinkConnector
curl -X DELETE http://localhost:8083/connectors/TDengineSourceConnector
```
### 性能调优
如果在从 TDengine 同步数据到 Kafka 的过程中发现性能不达预期，可以尝试使用如下参数提升 Kafka 的写入吞吐量。
1. 打开 KAFKA_HOME/config/producer.properties 配置文件。
2. 参数说明及配置建议如下：
| **参数** | **参数说明** | **设置建议** |
| --------| --------------------------------- | -------------- |
| producer.type | 此参数用于设置消息的发送方式，默认值为 `sync` 表示同步发送，`async` 表示异步发送。采用异步发送能够提升消息发送的吞吐量。 | async |
| request.required.acks | 参数用于配置生产者发送消息后需要等待的确认数量。当设置为1时，表示只要领导者副本成功写入消息就会给生产者发送确认，而无需等待集群中的其他副本写入成功。这种设置可以在一定程度上保证消息的可靠性，同时也能保证一定的吞吐量。因为不需要等待所有副本都写入成功，所以可以减少生产者的等待时间，提高发送消息的效率。|1|
| max.request.size| 该参数决定了生产者在一次请求中可以发送的最大数据量。其默认值为 1048576，也就是 1M。如果设置得太小，可能会导致频繁的网络请求，降低吞吐量。如果设置得太大，可能会导致内存占用过高，或者在网络状况不佳时增加请求失败的概率。建议设置为 100M。|104857600|
|batch.size| 此参数用于设定 batch 的大小，默认值为 16384，即 16KB。在消息发送过程中，发送到 Kafka 缓冲区中的消息会被划分成一个个的 batch。故而减小 batch 大小有助于降低消息延迟，而增大 batch 大小则有利于提升吞吐量，可根据实际的数据量大小进行合理配置。可根据实际情况进行调整，建议设置为 512K。|524288|
| buffer.memory| 此参数用于设置生产者缓冲待发送消息的内存总量。较大的缓冲区可以允许生产者积累更多的消息后批量发送，提高吞吐量，但也会增加延迟和内存使用。可根据机器资源来配置，建议配置为 1G。|1073741824|
## 配置参考
### 通用配置

2
include/common/tdataformat.h
View File

@ -193,7 +193,7 @@ int32_t tColDataDecompress(void *input, SColDataCompressInfo *info, SColData *co
// for stmt bind
int32_t tColDataAddValueByBind(SColData *pColData, TAOS_MULTI_BIND *pBind, int32_t buffMaxLen);
void tColDataSortMerge(SArray *colDataArr);
int32_t tColDataSortMerge(SArray *colDataArr);
// for raw block
int32_t tColDataAddValueByDataBlock(SColData *pColData, int8_t type, int32_t bytes, int32_t nRows, char *lengthOrbitmap,

5
include/common/trow.h
View File

@ -213,7 +213,6 @@ static FORCE_INLINE SKvRowIdx *tdKvRowColIdxAt(STSRow *pRow, col_id_t idx) {
}
static FORCE_INLINE int16_t tdKvRowColIdAt(STSRow *pRow, col_id_t idx) {
ASSERT(idx >= 0);
if (idx == 0) {
return PRIMARYKEY_TIMESTAMP_COL_ID;
}
@ -288,9 +287,7 @@ int32_t tdGetBitmapValType(const void *pBitmap, int16_t colIdx, TDRowValT *pValT
*
*/
static FORCE_INLINE void tdSRowInit(SRowBuilder *pBuilder, int16_t sver) {
pBuilder->sver = sver;
}
static FORCE_INLINE void tdSRowInit(SRowBuilder *pBuilder, int16_t sver) { pBuilder->sver = sver; }
int32_t tdSRowSetInfo(SRowBuilder *pBuilder, int32_t nCols, int32_t nBoundCols, int32_t flen);
int32_t tdSRowSetTpInfo(SRowBuilder *pBuilder, int32_t nCols, int32_t flen);
int32_t tdSRowSetExtendedInfo(SRowBuilder *pBuilder, int32_t nCols, int32_t nBoundCols, int32_t flen,

4
include/common/tvariant.h
View File

@ -46,14 +46,14 @@ int32_t toUInteger(const char *z, int32_t n, int32_t base, uint64_t *value);
/**
* non floating point integers
*/
*/
int32_t toIntegerPure(const char *z, int32_t n, int32_t base, int64_t *value);
void taosVariantCreateFromBinary(SVariant *pVar, const char *pz, size_t len, uint32_t type);
void taosVariantDestroy(SVariant *pV);
void taosVariantAssign(SVariant *pDst, const SVariant *pSrc);
int32_t taosVariantAssign(SVariant *pDst, const SVariant *pSrc);
int32_t taosVariantCompare(const SVariant *p1, const SVariant *p2);

30
include/libs/sync/sync.h
View File

@ -156,11 +156,11 @@ typedef struct SSnapshotParam {
typedef struct SSnapshot {
int32_t type;
SSyncTLV* data;
SSyncTLV* data;
ESyncFsmState state;
SyncIndex lastApplyIndex;
SyncTerm lastApplyTerm;
SyncIndex lastConfigIndex;
SyncIndex lastApplyIndex;
SyncTerm lastApplyTerm;
SyncIndex lastConfigIndex;
} SSnapshot;
typedef struct SSnapshotMeta {
@ -263,16 +263,16 @@ typedef struct SSyncState {
int64_t startTimeMs;
} SSyncState;
int32_t syncInit();
void syncCleanUp();
int64_t syncOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion);
int32_t syncStart(int64_t rid);
void syncStop(int64_t rid);
void syncPreStop(int64_t rid);
void syncPostStop(int64_t rid);
int32_t syncPropose(int64_t rid, SRpcMsg* pMsg, bool isWeak, int64_t* seq);
int32_t syncCheckMember(int64_t rid);
int32_t syncIsCatchUp(int64_t rid);
int32_t syncInit();
void syncCleanUp();
int64_t syncOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion);
int32_t syncStart(int64_t rid);
void syncStop(int64_t rid);
void syncPreStop(int64_t rid);
void syncPostStop(int64_t rid);
int32_t syncPropose(int64_t rid, SRpcMsg* pMsg, bool isWeak, int64_t* seq);
int32_t syncCheckMember(int64_t rid);
int32_t syncIsCatchUp(int64_t rid);
ESyncRole syncGetRole(int64_t rid);
int64_t syncGetTerm(int64_t rid);
int32_t syncProcessMsg(int64_t rid, SRpcMsg* pMsg);
@ -296,7 +296,7 @@ int32_t syncGetAssignedLogSynced(int64_t rid);
void syncGetRetryEpSet(int64_t rid, SEpSet* pEpSet);
const char* syncStr(ESyncState state);
int32_t syncNodeGetConfig(int64_t rid, SSyncCfg *cfg);
int32_t syncNodeGetConfig(int64_t rid, SSyncCfg* cfg);
// util
int32_t syncSnapInfoDataRealloc(SSnapshot* pSnap, int32_t size);

12
include/util/taoserror.h
View File

@ -91,7 +91,7 @@ int32_t taosGetErrSize();
#define TSDB_CODE_RPC_NETWORK_BUSY TAOS_DEF_ERROR_CODE(0, 0x0024)
#define TSDB_CODE_HTTP_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0025)
#define TSDB_CODE_RPC_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0026)
#define TSDB_CODE_RPC_ASYNC_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0027)
#define TSDB_CODE_RPC_ASYNC_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0027)
@ -152,6 +152,7 @@ int32_t taosGetErrSize();
#define TSDB_CODE_FAILED_TO_CONNECT_S3 TAOS_DEF_ERROR_CODE(0, 0x0135)
#define TSDB_CODE_MSG_PREPROCESSED TAOS_DEF_ERROR_CODE(0, 0x0136) // internal
#define TSDB_CODE_OUT_OF_BUFFER TAOS_DEF_ERROR_CODE(0, 0x0137)
#define TSDB_CODE_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x0138)
//client
#define TSDB_CODE_TSC_INVALID_OPERATION TAOS_DEF_ERROR_CODE(0, 0x0200)
@ -604,6 +605,9 @@ int32_t taosGetErrSize();
#define TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x0732)
#define TSDB_CODE_QRY_INVALID_JOIN_CONDITION TAOS_DEF_ERROR_CODE(0, 0x0733)
#define TSDB_CODE_QRY_FILTER_NOT_SUPPORT_TYPE TAOS_DEF_ERROR_CODE(0, 0x0734)
#define TSDB_CODE_QRY_FILTER_WRONG_OPTR_TYPE TAOS_DEF_ERROR_CODE(0, 0x0735)
#define TSDB_CODE_QRY_FILTER_RANGE_ERROR TAOS_DEF_ERROR_CODE(0, 0x0736)
#define TSDB_CODE_QRY_FILTER_INVALID_TYPE TAOS_DEF_ERROR_CODE(0, 0x0737)
// grant
#define TSDB_CODE_GRANT_EXPIRED TAOS_DEF_ERROR_CODE(0, 0x0800)
@ -878,6 +882,8 @@ int32_t taosGetErrSize();
#define TSDB_CODE_FUNC_INVALID_VALUE_RANGE TAOS_DEF_ERROR_CODE(0, 0x280C)
#define TSDB_CODE_FUNC_SETUP_ERROR TAOS_DEF_ERROR_CODE(0, 0x280D)
#define TSDB_CODE_FUNC_INVALID_RES_LENGTH TAOS_DEF_ERROR_CODE(0, 0x280E)
#define TSDB_CODE_FUNC_HISTOGRAM_ERROR TAOS_DEF_ERROR_CODE(0, 0x280F)
#define TSDB_CODE_FUNC_PERCENTILE_ERROR TAOS_DEF_ERROR_CODE(0, 0x2810)
//udf
@ -892,6 +898,7 @@ int32_t taosGetErrSize();
#define TSDB_CODE_UDF_SCRIPT_NOT_SUPPORTED TAOS_DEF_ERROR_CODE(0, 0x2909)
#define TSDB_CODE_UDF_FUNC_EXEC_FAILURE TAOS_DEF_ERROR_CODE(0, 0x290A)
#define TSDB_CODE_UDF_UV_EXEC_FAILURE TAOS_DEF_ERROR_CODE(0, 0x290B)
#define TSDB_CODE_UDF_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x2920)
// sml
#define TSDB_CODE_SML_INVALID_PROTOCOL_TYPE TAOS_DEF_ERROR_CODE(0, 0x3000)
@ -958,7 +965,8 @@ int32_t taosGetErrSize();
#define TSDB_CODE_STREAM_INVALID_STATETRANS TAOS_DEF_ERROR_CODE(0, 0x4103)
#define TSDB_CODE_STREAM_TASK_IVLD_STATUS TAOS_DEF_ERROR_CODE(0, 0x4104)
#define TSDB_CODE_STREAM_NOT_LEADER TAOS_DEF_ERROR_CODE(0, 0x4105)
#define TSDB_CODE_STREAM_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x41FF)
#define TSDB_CODE_STREAM_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x4106)
// TDLite
#define TSDB_CODE_TDLITE_IVLD_OPEN_FLAGS TAOS_DEF_ERROR_CODE(0, 0x5100)

3
include/util/tcoding.h
View File

@ -213,7 +213,6 @@ static FORCE_INLINE int32_t taosEncodeVariantU16(void **buf, uint16_t value) {
if (buf != NULL) ((uint8_t *)(*buf))[i] = (uint8_t)(value | ENCODE_LIMIT);
value >>= 7;
i++;
ASSERT(i < 3);
}
if (buf != NULL) {
@ -261,7 +260,6 @@ static FORCE_INLINE int32_t taosEncodeVariantU32(void **buf, uint32_t value) {
if (buf != NULL) ((uint8_t *)(*buf))[i] = (value | ENCODE_LIMIT);
value >>= 7;
i++;
ASSERT(i < 5);
}
if (buf != NULL) {
@ -309,7 +307,6 @@ static FORCE_INLINE int32_t taosEncodeVariantU64(void **buf, uint64_t value) {
if (buf != NULL) ((uint8_t *)(*buf))[i] = (uint8_t)(value | ENCODE_LIMIT);
value >>= 7;
i++;
ASSERT(i < 10);
}
if (buf != NULL) {

3
include/util/tref.h
View File

@ -29,8 +29,7 @@ typedef void (*RefFp)(void *);
int32_t taosOpenRef(int32_t max, RefFp fp);
// close the reference set, refId is the return value by taosOpenRef
// return 0 if success. On error, -1 is returned, and terrno is set appropriately
int32_t taosCloseRef(int32_t rsetId);
void taosCloseRef(int32_t rsetId);
// add ref, p is the pointer to resource or pointer ID
// return Reference ID(rid) allocated. On error, -1 is returned, and terrno is set appropriately

7
include/util/tutil.h
View File

@ -139,6 +139,13 @@ static FORCE_INLINE int32_t taosGetTbHashVal(const char *tbname, int32_t tblen,
#define QUERY_CHECK_CODE TSDB_CHECK_CODE
#define QUERY_CHECK_CONDITION(condition, CODE, LINO, LABEL, ERRNO) \
if (!condition) { \
(CODE) = (ERRNO); \
(LINO) = __LINE__; \
goto LABEL; \
}
#define TSDB_CHECK_NULL(ptr, CODE, LINO, LABEL, ERRNO) \
if ((ptr) == NULL) { \
(CODE) = (ERRNO); \

5
packaging/cfg/taos.cfg
View File

@ -98,11 +98,6 @@
# enable/disable system monitor
# monitor 1
# enable/disable audit log
# audit 1
# enable/disable audit create table
# auditCreateTable 1
# The following parameter is used to limit the maximum number of lines in log files.
# max number of lines per log filters

32
source/client/src/clientImpl.c
View File

@ -166,7 +166,10 @@ int32_t taos_connect_internal(const char* ip, const char* user, const char* pass
pInst = &p;
} else {
ASSERTS((*pInst) && (*pInst)->pAppHbMgr, "*pInst:%p, pAppHgMgr:%p", *pInst, (*pInst) ? (*pInst)->pAppHbMgr : NULL);
if (NULL == *pInst || NULL == (*pInst)->pAppHbMgr) {
tscError("*pInst:%p, pAppHgMgr:%p", *pInst, (*pInst) ? (*pInst)->pAppHbMgr : NULL);
TSC_ERR_JRET(TSDB_CODE_TSC_INTERNAL_ERROR);
}
// reset to 0 in case of conn with duplicated user key but its user has ever been dropped.
atomic_store_8(&(*pInst)->pAppHbMgr->connHbFlag, 0);
}
@ -1135,8 +1138,6 @@ void schedulerExecCb(SExecResult* pResult, void* param, int32_t code) {
(void)atomic_add_fetch_64((int64_t*)&pActivity->numOfInsertRows, pResult->numOfRows);
}
}
schedulerFreeJob(&pRequest->body.queryJob, 0);
}
taosMemoryFree(pResult);
@ -2038,9 +2039,9 @@ static int32_t estimateJsonLen(SReqResultInfo* pResultInfo, int32_t numOfCols, i
// | version | total length | total rows | total columns | flag seg| block group id | column schema | each column
// length |
int32_t cols = *(int32_t*)(p + sizeof(int32_t) * 3);
if (ASSERT(numOfCols == cols)) {
if (numOfCols != cols) {
tscError("estimateJsonLen error: numOfCols:%d != cols:%d", numOfCols, cols);
return -1;
return TSDB_CODE_TSC_INTERNAL_ERROR;
}
int32_t len = getVersion1BlockMetaSize(p, numOfCols);
@ -2125,7 +2126,7 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
int32_t totalLen = 0;
int32_t cols = *(int32_t*)(p + sizeof(int32_t) * 3);
if (ASSERT(numOfCols == cols)) {
if (numOfCols != cols) {
tscError("doConvertJson error: numOfCols:%d != cols:%d", numOfCols, cols);
return TSDB_CODE_TSC_INTERNAL_ERROR;
}
@ -2150,7 +2151,7 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
for (int32_t i = 0; i < numOfCols; ++i) {
int32_t colLen = (blockVersion == BLOCK_VERSION_1) ? htonl(colLength[i]) : colLength[i];
int32_t colLen1 = (blockVersion == BLOCK_VERSION_1) ? htonl(colLength1[i]) : colLength1[i];
if (ASSERT(colLen < dataLen)) {
if (colLen >= dataLen) {
tscError("doConvertJson error: colLen:%d >= dataLen:%d", colLen, dataLen);
return TSDB_CODE_TSC_INTERNAL_ERROR;
}
@ -2238,7 +2239,7 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32_t numOfCols, int32_t numOfRows,
bool convertUcs4) {
if (ASSERT(numOfCols > 0 && pFields != NULL && pResultInfo != NULL)) {
if (numOfCols <= 0 || pFields == NULL || pResultInfo == NULL) {
tscError("setResultDataPtr paras error");
return TSDB_CODE_TSC_INTERNAL_ERROR;
}
@ -2271,7 +2272,7 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32
int32_t cols = *(int32_t*)p;
p += sizeof(int32_t);
if (ASSERT(rows == numOfRows && cols == numOfCols)) {
if (rows != numOfRows || cols != numOfCols) {
tscError("setResultDataPtr paras error:rows;%d numOfRows:%d cols:%d numOfCols:%d", rows, numOfRows, cols,
numOfCols);
return TSDB_CODE_TSC_INTERNAL_ERROR;
@ -2290,8 +2291,6 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32
int32_t bytes = *(int32_t*)p;
p += sizeof(int32_t);
/*ASSERT(type == pFields[i].type && bytes == pFields[i].bytes);*/
}
int32_t* colLength = (int32_t*)p;
@ -2413,18 +2412,23 @@ int32_t setQueryResultFromRsp(SReqResultInfo* pResultInfo, const SRetrieveTableR
if (pRsp->compressed && compLen < rawLen) {
int32_t len = tsDecompressString(pStart, compLen, 1, pResultInfo->decompBuf, rawLen, ONE_STAGE_COMP, NULL, 0);
ASSERT(len == rawLen);
if (len < 0) {
tscError("tsDecompressString failed");
return terrno ? terrno : TSDB_CODE_FAILED;
}
if (len != rawLen) {
tscError("tsDecompressString failed, len:%d != rawLen:%d", len, rawLen);
return TSDB_CODE_TSC_INTERNAL_ERROR;
}
pResultInfo->pData = pResultInfo->decompBuf;
pResultInfo->payloadLen = rawLen;
} else {
pResultInfo->pData = pStart;
pResultInfo->payloadLen = htonl(pRsp->compLen);
ASSERT(pRsp->compLen == pRsp->payloadLen);
if (pRsp->compLen != pRsp->payloadLen) {
tscError("pRsp->compLen:%d != pRsp->payloadLen:%d", pRsp->compLen, pRsp->payloadLen);
return TSDB_CODE_TSC_INTERNAL_ERROR;
}
}
}

35
source/client/src/clientMain.c
View File

@ -74,15 +74,11 @@ void taos_cleanup(void) {
int32_t id = clientReqRefPool;
clientReqRefPool = -1;
if (TSDB_CODE_SUCCESS != taosCloseRef(id)) {
tscWarn("failed to close clientReqRefPool");
}
taosCloseRef(id);
id = clientConnRefPool;
clientConnRefPool = -1;
if (TSDB_CODE_SUCCESS != taosCloseRef(id)) {
tscWarn("failed to close clientReqRefPool");
}
taosCloseRef(id);
nodesDestroyAllocatorSet();
cleanupAppInfo();
@ -481,7 +477,6 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) {
} else if (TD_RES_TMQ_META(res) || TD_RES_TMQ_BATCH_META(res)) {
return NULL;
} else {
// assert to avoid un-initialization error
tscError("invalid result passed to taos_fetch_row");
terrno = TSDB_CODE_TSC_INTERNAL_ERROR;
return NULL;
@ -561,12 +556,14 @@ int taos_print_row(char *str, TAOS_ROW row, TAOS_FIELD *fields, int num_fields)
case TSDB_DATA_TYPE_GEOMETRY: {
int32_t charLen = varDataLen((char *)row[i] - VARSTR_HEADER_SIZE);
if (fields[i].type == TSDB_DATA_TYPE_BINARY || fields[i].type == TSDB_DATA_TYPE_VARBINARY || fields[i].type == TSDB_DATA_TYPE_GEOMETRY) {
if (ASSERT(charLen <= fields[i].bytes && charLen >= 0)) {
if (charLen > fields[i].bytes || charLen < 0) {
tscError("taos_print_row error binary. charLen:%d, fields[i].bytes:%d", charLen, fields[i].bytes);
break;
}
} else {
if (ASSERT(charLen <= fields[i].bytes * TSDB_NCHAR_SIZE && charLen >= 0)) {
if (charLen > fields[i].bytes * TSDB_NCHAR_SIZE || charLen < 0) {
tscError("taos_print_row error. charLen:%d, fields[i].bytes:%d", charLen, fields[i].bytes);
break;
}
}
@ -1319,11 +1316,11 @@ void restartAsyncQuery(SRequestObj *pRequest, int32_t code) {
}
void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) {
if (ASSERT(res != NULL && fp != NULL)) {
if (res == NULL || fp == NULL) {
tscError("taos_fetch_rows_a invalid paras");
return;
}
if (ASSERT(TD_RES_QUERY(res))) {
if (!TD_RES_QUERY(res)) {
tscError("taos_fetch_rows_a res is NULL");
return;
}
@ -1338,12 +1335,12 @@ void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) {
}
void taos_fetch_raw_block_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) {
if (ASSERT(res != NULL && fp != NULL)) {
tscError("taos_fetch_rows_a invalid paras");
if (res == NULL || fp == NULL) {
tscError("taos_fetch_raw_block_a invalid paras");
return;
}
if (ASSERT(TD_RES_QUERY(res))) {
tscError("taos_fetch_rows_a res is NULL");
if (!TD_RES_QUERY(res)) {
tscError("taos_fetch_raw_block_a res is NULL");
return;
}
SRequestObj *pRequest = res;
@ -1357,12 +1354,12 @@ void taos_fetch_raw_block_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) {
}
const void *taos_get_raw_block(TAOS_RES *res) {
if (ASSERT(res != NULL)) {
tscError("taos_fetch_rows_a invalid paras");
if (res == NULL) {
tscError("taos_get_raw_block invalid paras");
return NULL;
}
if (ASSERT(TD_RES_QUERY(res))) {
tscError("taos_fetch_rows_a res is NULL");
if (!TD_RES_QUERY(res)) {
tscError("taos_get_raw_block res is NULL");
return NULL;
}
SRequestObj *pRequest = res;

31
source/client/src/clientMonitor.c
View File

@ -120,9 +120,13 @@ static int32_t monitorReportAsyncCB(void* param, SDataBuf* pMsg, int32_t code) {
.data = NULL};
if (monitorPutData2MonitorQueue(tmp) == 0) {
p->fileName = NULL;
} else {
if(taosCloseFile(&(p->pFile)) != 0) {
tscError("failed to close file:%p", p->pFile);
}
}
}
return code;
return TSDB_CODE_SUCCESS;
}
static int32_t sendReport(void* pTransporter, SEpSet* epSet, char* pCont, MONITOR_TYPE type, void* param) {
@ -164,7 +168,10 @@ static int32_t sendReport(void* pTransporter, SEpSet* epSet, char* pCont, MONITO
int64_t transporterId = 0;
return asyncSendMsgToServer(pTransporter, epSet, &transporterId, pInfo);
FAILED:
FAILED:
if (taosCloseFile(&(((MonitorSlowLogData*)param)->pFile)) != 0) {
tscError("failed to close file:%p", ((MonitorSlowLogData*)param)->pFile);
}
monitorFreeSlowLogDataEx(param);
return TAOS_GET_TERRNO(TSDB_CODE_TSC_INTERNAL_ERROR);
}
@ -315,7 +322,7 @@ void monitorCreateClientCounter(int64_t clusterId, const char* name, const char*
void monitorCounterInc(int64_t clusterId, const char* counterName, const char** label_values) {
taosWLockLatch(&monitorLock);
if (atomic_load_32(&monitorFlag) == 1) {
taosRUnLockLatch(&monitorLock);
taosWUnLockLatch(&monitorLock);
return;
}
@ -466,11 +473,17 @@ static int64_t getFileSize(char* path) {
static int32_t sendSlowLog(int64_t clusterId, char* data, TdFilePtr pFile, int64_t offset, SLOW_LOG_QUEUE_TYPE type,
char* fileName, void* pTransporter, SEpSet* epSet) {
if (data == NULL) {
if (taosCloseFile(&pFile) != 0) {
tscError("failed to close file:%p", pFile);
}
taosMemoryFree(fileName);
return TSDB_CODE_INVALID_PARA;
}
MonitorSlowLogData* pParam = taosMemoryMalloc(sizeof(MonitorSlowLogData));
if (pParam == NULL) {
if (taosCloseFile(&pFile) != 0) {
tscError("failed to close file:%p", pFile);
}
taosMemoryFree(data);
taosMemoryFree(fileName);
return terrno;
@ -489,6 +502,9 @@ static int32_t monitorReadSend(int64_t clusterId, TdFilePtr pFile, int64_t* offs
SAppInstInfo* pInst = getAppInstByClusterId(clusterId);
if (pInst == NULL) {
tscError("failed to get app instance by clusterId:%" PRId64, clusterId);
if (taosCloseFile(&pFile) != 0) {
tscError("failed to close file:%p", pFile);
}
taosMemoryFree(fileName);
return terrno;
}
@ -715,7 +731,7 @@ static void* monitorThreadFunc(void* param) {
MonitorSlowLogData* slowLogData = NULL;
(void)taosReadQitem(monitorQueue, (void**)&slowLogData);
if (slowLogData != NULL) {
if (slowLogData->type == SLOW_LOG_READ_BEGINNIG) {
if (slowLogData->type == SLOW_LOG_READ_BEGINNIG && quitCnt == 0) {
if (slowLogData->pFile != NULL) {
monitorSendSlowLogAtBeginning(slowLogData->clusterId, &(slowLogData->fileName), slowLogData->pFile,
slowLogData->offset);
@ -734,9 +750,9 @@ static void* monitorThreadFunc(void* param) {
break;
}
}
monitorFreeSlowLogData(slowLogData);
taosFreeQitem(slowLogData);
}
monitorFreeSlowLogData(slowLogData);
taosFreeQitem(slowLogData);
if (quitCnt == 0) {
monitorSendAllSlowLog();
@ -858,6 +874,9 @@ int32_t monitorPutData2MonitorQueue(MonitorSlowLogData data) {
if (taosWriteQitem(monitorQueue, slowLogData) == 0) {
(void)tsem2_post(&monitorSem);
} else {
if (taosCloseFile(&(slowLogData->pFile)) != 0) {
tscError("failed to close file:%p", slowLogData->pFile);
}
monitorFreeSlowLogData(slowLogData);
taosFreeQitem(slowLogData);
}

2
source/client/src/clientTmq.c
View File

@ -1184,7 +1184,7 @@ void tmqMgmtClose(void) {
}
if (tmqMgmt.rsetId >= 0) {
(void)taosCloseRef(tmqMgmt.rsetId);
taosCloseRef(tmqMgmt.rsetId);
tmqMgmt.rsetId = -1;
}
}

42
source/common/src/cos.c
View File

@ -12,7 +12,7 @@ extern char tsS3AccessKeySecret[][TSDB_FQDN_LEN];
extern char tsS3BucketName[TSDB_FQDN_LEN];
extern char tsS3AppId[][TSDB_FQDN_LEN];
extern char tsS3Hostname[][TSDB_FQDN_LEN];
extern int8_t tsS3Https;
extern int8_t tsS3Https[];
static int32_t s3ListBucketByEp(char const *bucketname, int8_t epIndex);
static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *object_name, int64_t offset, int64_t size,
@ -33,13 +33,13 @@ static int verifyPeerG = 0;
static const char *awsRegionG = NULL;
static int forceG = 0;
static int showResponsePropertiesG = 0;
static S3Protocol protocolG = S3ProtocolHTTPS;
static S3Protocol protocolG[TSDB_MAX_EP_NUM] = {S3ProtocolHTTPS};
// static S3Protocol protocolG = S3ProtocolHTTP;
static S3UriStyle uriStyleG = S3UriStylePath;
static S3UriStyle uriStyleG[TSDB_MAX_EP_NUM] = {S3UriStylePath};
static int retriesG = 5;
static int timeoutMsG = 0;
extern int8_t tsS3Oss;
extern int8_t tsS3Oss[];
int32_t s3Begin() {
S3Status status;
@ -55,9 +55,9 @@ int32_t s3Begin() {
TAOS_RETURN(TSDB_CODE_FAILED);
}
protocolG = !tsS3Https;
if (tsS3Oss) {
uriStyleG = S3UriStyleVirtualHost;
for (int i = 0; i < tsS3EpNum; i++) {
protocolG[i] = tsS3Https[i] ? S3ProtocolHTTPS : S3ProtocolHTTP;
uriStyleG[i] = tsS3Oss[i] ? S3UriStyleVirtualHost : S3UriStylePath;
}
TAOS_RETURN(TSDB_CODE_SUCCESS);
@ -976,8 +976,8 @@ int32_t s3PutObjectFromFile2ByEp(const char *file, const char *object_name, int8
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1059,8 +1059,8 @@ static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *objec
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1155,8 +1155,8 @@ static void s3FreeObjectKey(void *pItem) {
static SArray *getListByPrefixByEp(const char *prefix, int8_t epIndex) {
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1223,8 +1223,8 @@ static int32_t s3DeleteObjectsByEp(const char *object_name[], int nobject, int8_
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1299,8 +1299,8 @@ static int32_t s3GetObjectBlockByEp(const char *object_name, int64_t offset, int
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1372,8 +1372,8 @@ static int32_t s3GetObjectToFileByEp(const char *object_name, const char *fileNa
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
@ -1449,8 +1449,8 @@ static long s3SizeByEp(const char *object_name, int8_t epIndex) {
S3BucketContext bucketContext = {tsS3Hostname[epIndex],
tsS3BucketName,
protocolG,
uriStyleG,
protocolG[epIndex],
uriStyleG[epIndex],
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,

147
source/common/src/tdataformat.c
View File

@ -119,7 +119,6 @@ static FORCE_INLINE void tRowBuildScanAddValue(SRowBuildScanInfo *sinfo, SColVal
bool isPK = ((pTColumn->flags & COL_IS_KEY) != 0);
if (isPK) {
ASSERTS(sinfo->numOfPKs < TD_MAX_PK_COLS, "too many primary keys");
sinfo->tupleIndices[sinfo->numOfPKs].type = colVal->value.type;
sinfo->tupleIndices[sinfo->numOfPKs].offset =
IS_VAR_DATA_TYPE(pTColumn->type) ? sinfo->tupleVarSize + sinfo->tupleFixedSize : pTColumn->offset;
@ -149,9 +148,15 @@ static int32_t tRowBuildScan(SArray *colVals, const STSchema *schema, SRowBuildS
int32_t numOfColVals = TARRAY_SIZE(colVals);
SColVal *colValArray = (SColVal *)TARRAY_DATA(colVals);
ASSERT(numOfColVals > 0);
ASSERT(colValArray[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID);
ASSERT(colValArray[0].value.type == TSDB_DATA_TYPE_TIMESTAMP);
if (!(numOfColVals > 0)) {
return TSDB_CODE_INVALID_PARA;
}
if (!(colValArray[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID)) {
return TSDB_CODE_INVALID_PARA;
}
if (!(colValArray[0].value.type == TSDB_DATA_TYPE_TIMESTAMP)) {
return TSDB_CODE_INVALID_PARA;
}
*sinfo = (SRowBuildScanInfo){
.tupleFixedSize = schema->flen,
@ -166,7 +171,10 @@ static int32_t tRowBuildScan(SArray *colVals, const STSchema *schema, SRowBuildS
}
if (colValArray[colValIndex].cid == schema->columns[i].colId) {
ASSERT(colValArray[colValIndex].value.type == schema->columns[i].type);
if (!(colValArray[colValIndex].value.type == schema->columns[i].type)) {
code = TSDB_CODE_INVALID_PARA;
goto _exit;
}
if (COL_VAL_IS_VALUE(&colValArray[colValIndex])) {
tRowBuildScanAddValue(sinfo, &colValArray[colValIndex], schema->columns + i);
@ -272,7 +280,6 @@ static int32_t tRowBuildTupleRow(SArray *aColVal, const SRowBuildScanInfo *sinfo
(*ppRow)->ts = colValArray[0].value.val;
if (sinfo->tupleFlag == HAS_NONE || sinfo->tupleFlag == HAS_NULL) {
ASSERT(sinfo->tupleRowSize == sizeof(SRow));
return 0;
}
@ -285,7 +292,6 @@ static int32_t tRowBuildTupleRow(SArray *aColVal, const SRowBuildScanInfo *sinfo
for (int32_t i = 0; i < sinfo->numOfPKs; i++) {
primaryKeys += tPutPrimaryKeyIndex(primaryKeys, sinfo->tupleIndices + i);
}
ASSERT(primaryKeys == bitmap);
// bitmap + fixed + varlen
int32_t numOfColVals = TARRAY_SIZE(aColVal);
@ -356,7 +362,9 @@ static int32_t tRowBuildKVRow(SArray *aColVal, const SRowBuildScanInfo *sinfo, c
(*ppRow)->len = sinfo->kvRowSize;
(*ppRow)->ts = colValArray[0].value.val;
ASSERT(sinfo->flag != HAS_NONE && sinfo->flag != HAS_NULL);
if (!(sinfo->flag != HAS_NONE && sinfo->flag != HAS_NULL)) {
return TSDB_CODE_INVALID_PARA;
}
uint8_t *primaryKeys = (*ppRow)->data;
SKVIdx *indices = (SKVIdx *)(primaryKeys + sinfo->kvPKSize);
@ -367,7 +375,6 @@ static int32_t tRowBuildKVRow(SArray *aColVal, const SRowBuildScanInfo *sinfo, c
for (int32_t i = 0; i < sinfo->numOfPKs; i++) {
primaryKeys += tPutPrimaryKeyIndex(primaryKeys, sinfo->kvIndices + i);
}
ASSERT(primaryKeys == (uint8_t *)indices);
int32_t numOfColVals = TARRAY_SIZE(aColVal);
int32_t colValIndex = 1;
@ -504,8 +511,8 @@ _exit:
}
int32_t tRowGet(SRow *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal) {
ASSERT(iCol < pTSchema->numOfCols);
ASSERT(pRow->sver == pTSchema->version);
if (!(iCol < pTSchema->numOfCols)) return TSDB_CODE_INVALID_PARA;
if (!(pRow->sver == pTSchema->version)) return TSDB_CODE_INVALID_PARA;
STColumn *pTColumn = pTSchema->columns + iCol;
@ -786,7 +793,7 @@ struct SRowIter {
};
int32_t tRowIterOpen(SRow *pRow, STSchema *pTSchema, SRowIter **ppIter) {
ASSERT(pRow->sver == pTSchema->version);
if (!(pRow->sver == pTSchema->version)) return TSDB_CODE_INVALID_PARA;
int32_t code = 0;
@ -839,7 +846,6 @@ int32_t tRowIterOpen(SRow *pRow, STSchema *pTSchema, SRowIter **ppIter) {
pIter->pv = pIter->pf + pTSchema->flen;
break;
default:
ASSERT(0);
break;
}
}
@ -952,7 +958,6 @@ SColVal *tRowIterNext(SRowIter *pIter) {
bv = GET_BIT2(pIter->pb, pIter->iTColumn - 1);
break;
default:
ASSERT(0);
break;
}
@ -1070,14 +1075,15 @@ static int32_t tRowTupleUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *
pv = pf + pTSchema->flen;
break;
default:
ASSERTS(0, "Invalid row flag");
return TSDB_CODE_INVALID_DATA_FMT;
}
while (pColData) {
if (pTColumn) {
if (pTColumn->colId == pColData->cid) {
ASSERT(pTColumn->type == pColData->type);
if (!(pTColumn->type == pColData->type)) {
return TSDB_CODE_INVALID_PARA;
}
if (pb) {
uint8_t bv;
switch (pRow->flag) {
@ -1095,7 +1101,6 @@ static int32_t tRowTupleUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *
bv = GET_BIT2(pb, iTColumn - 1);
break;
default:
ASSERTS(0, "Invalid row flag");
return TSDB_CODE_INVALID_DATA_FMT;
}
@ -1178,7 +1183,7 @@ static int32_t tRowKVUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *aCo
} else if (pRow->flag & KV_FLG_BIG) {
pv = pKVIdx->idx + (pKVIdx->nCol << 2);
} else {
ASSERT(0);
return TSDB_CODE_INVALID_PARA;
}
while (pColData) {
@ -1193,7 +1198,6 @@ static int32_t tRowKVUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *aCo
} else if (pRow->flag & KV_FLG_BIG) {
pData = pv + ((uint32_t *)pKVIdx->idx)[iCol];
} else {
ASSERTS(0, "Invalid KV row format");
return TSDB_CODE_INVALID_DATA_FMT;
}
@ -1256,8 +1260,8 @@ _exit:
* flag < 0: backward update
*/
int32_t tRowUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *aColData, int32_t nColData, int32_t flag) {
ASSERT(pRow->sver == pTSchema->version);
ASSERT(nColData > 0);
if (!(pRow->sver == pTSchema->version)) return TSDB_CODE_INVALID_PARA;
if (!(nColData > 0)) return TSDB_CODE_INVALID_PARA;
if (pRow->flag == HAS_NONE) {
return tRowNoneUpsertColData(aColData, nColData, flag);
@ -1277,8 +1281,6 @@ void tRowGetPrimaryKey(SRow *row, SRowKey *key) {
return;
}
ASSERT(row->numOfPKs <= TD_MAX_PK_COLS);
SPrimaryKeyIndex indices[TD_MAX_PK_COLS];
uint8_t *data = row->data;
@ -1317,8 +1319,6 @@ void tRowGetPrimaryKey(SRow *row, SRowKey *key) {
} while (0)
int32_t tValueCompare(const SValue *tv1, const SValue *tv2) {
ASSERT(tv1->type == tv2->type);
switch (tv1->type) {
case TSDB_DATA_TYPE_BOOL:
case TSDB_DATA_TYPE_TINYINT:
@ -1476,7 +1476,6 @@ static void debugPrintTagVal(int8_t type, const void *val, int32_t vlen, const c
printf("%s:%d type:%d vlen:%d, val:%" PRIi8 "\n", tag, ln, (int32_t)type, vlen, *(int8_t *)val);
break;
default:
ASSERT(0);
break;
}
}
@ -1526,7 +1525,6 @@ static int32_t tPutTagVal(uint8_t *p, STagVal *pTagVal, int8_t isJson) {
n += tPutCStr(p ? p + n : p, pTagVal->pKey);
} else {
n += tPutI16v(p ? p + n : p, pTagVal->cid);
ASSERTS(pTagVal->cid > 0, "Invalid tag cid:%" PRIi16, pTagVal->cid);
}
// type
@ -1602,8 +1600,6 @@ int32_t tTagNew(SArray *pArray, int32_t version, int8_t isJson, STag **ppTag) {
isLarge = 1;
}
ASSERT(szTag <= INT16_MAX);
// build tag
(*ppTag) = (STag *)taosMemoryCalloc(szTag, 1);
if ((*ppTag) == NULL) {
@ -1806,8 +1802,14 @@ STSchema *tBuildTSchema(SSchema *aSchema, int32_t numOfCols, int32_t version) {
pTSchema->version = version;
// timestamp column
ASSERT(aSchema[0].type == TSDB_DATA_TYPE_TIMESTAMP);
ASSERT(aSchema[0].colId == PRIMARYKEY_TIMESTAMP_COL_ID);
if (!(aSchema[0].type == TSDB_DATA_TYPE_TIMESTAMP)) {
terrno = TSDB_CODE_INVALID_PARA;
return NULL;
}
if (!(aSchema[0].colId == PRIMARYKEY_TIMESTAMP_COL_ID)) {
terrno = TSDB_CODE_INVALID_PARA;
return NULL;
}
pTSchema->columns[0].colId = aSchema[0].colId;
pTSchema->columns[0].type = aSchema[0].type;
pTSchema->columns[0].flags = aSchema[0].flags;
@ -1910,7 +1912,9 @@ static FORCE_INLINE int32_t tColDataPutValue(SColData *pColData, uint8_t *pData,
pColData->nData += nData;
}
} else {
ASSERT(pColData->nData == tDataTypes[pColData->type].bytes * pColData->nVal);
if (!(pColData->nData == tDataTypes[pColData->type].bytes * pColData->nVal)) {
return TSDB_CODE_INVALID_PARA;
}
code = tRealloc(&pColData->pData, pColData->nData + tDataTypes[pColData->type].bytes);
if (code) goto _exit;
if (pData) {
@ -2259,7 +2263,9 @@ static int32_t (*tColDataAppendValueImpl[8][3])(SColData *pColData, uint8_t *pDa
// VALUE NONE NULL
};
int32_t tColDataAppendValue(SColData *pColData, SColVal *pColVal) {
ASSERT(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type);
if (!(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type)) {
return TSDB_CODE_INVALID_PARA;
}
return tColDataAppendValueImpl[pColData->flag][pColVal->flag](
pColData, IS_VAR_DATA_TYPE(pColData->type) ? pColVal->value.pData : (uint8_t *)&pColVal->value.val,
pColVal->value.nData);
@ -2569,8 +2575,8 @@ static int32_t (*tColDataUpdateValueImpl[8][3])(SColData *pColData, uint8_t *pDa
// VALUE NONE NULL
};
int32_t tColDataUpdateValue(SColData *pColData, SColVal *pColVal, bool forward) {
ASSERT(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type);
ASSERT(pColData->nVal > 0);
if (!(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type)) return TSDB_CODE_INVALID_PARA;
if (!(pColData->nVal > 0)) return TSDB_CODE_INVALID_PARA;
if (tColDataUpdateValueImpl[pColData->flag][pColVal->flag] == NULL) return 0;
@ -2687,7 +2693,6 @@ uint8_t tColDataGetBitValue(const SColData *pColData, int32_t iVal) {
case (HAS_VALUE | HAS_NULL | HAS_NONE):
return GET_BIT2(pColData->pBitMap, iVal);
default:
ASSERTS(0, "not possible");
return 0;
}
}
@ -3053,7 +3058,9 @@ int32_t tColDataAddValueByBind(SColData *pColData, TAOS_MULTI_BIND *pBind, int32
int32_t code = 0;
if (!(pBind->num == 1 && pBind->is_null && *pBind->is_null)) {
ASSERT(pColData->type == pBind->buffer_type);
if (!(pColData->type == pBind->buffer_type)) {
return TSDB_CODE_INVALID_PARA;
}
}
if (IS_VAR_DATA_TYPE(pColData->type)) { // var-length data type
@ -3521,15 +3528,21 @@ static void tColDataMerge(SColData *aColData, int32_t nColData) {
}
}
void tColDataSortMerge(SArray *colDataArr) {
int32_t tColDataSortMerge(SArray *colDataArr) {
int32_t nColData = TARRAY_SIZE(colDataArr);
SColData *aColData = (SColData *)TARRAY_DATA(colDataArr);
if (aColData[0].nVal <= 1) goto _exit;
if (!(aColData[0].type == TSDB_DATA_TYPE_TIMESTAMP)) {
return TSDB_CODE_INVALID_PARA;
}
if (!(aColData[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID)) {
return TSDB_CODE_INVALID_PARA;
}
if (!(aColData[0].flag == HAS_VALUE)) {
return TSDB_CODE_INVALID_PARA;
}
ASSERT(aColData[0].type == TSDB_DATA_TYPE_TIMESTAMP);
ASSERT(aColData[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID);
ASSERT(aColData[0].flag == HAS_VALUE);
if (aColData[0].nVal <= 1) goto _exit;
int8_t doSort = 0;
int8_t doMerge = 0;
@ -3578,7 +3591,7 @@ void tColDataSortMerge(SArray *colDataArr) {
}
_exit:
return;
return 0;
}
static int32_t tPutColDataVersion0(uint8_t *pBuf, SColData *pColData) {
@ -3685,8 +3698,7 @@ int32_t tPutColData(uint8_t version, uint8_t *pBuf, SColData *pColData) {
} else if (version == 1) {
return tPutColDataVersion1(pBuf, pColData);
} else {
ASSERT(0);
return -1;
return TSDB_CODE_INVALID_PARA;
}
}
@ -3696,8 +3708,7 @@ int32_t tGetColData(uint8_t version, uint8_t *pBuf, SColData *pColData) {
} else if (version == 1) {
return tGetColDataVersion1(pBuf, pColData);
} else {
ASSERT(0);
return -1;
return TSDB_CODE_INVALID_PARA;
}
}
@ -3733,7 +3744,6 @@ static FORCE_INLINE void tColDataCalcSMABool(SColData *pColData, int64_t *sum, i
CALC_SUM_MAX_MIN(*sum, *max, *min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3765,7 +3775,6 @@ static FORCE_INLINE void tColDataCalcSMATinyInt(SColData *pColData, int64_t *sum
CALC_SUM_MAX_MIN(*sum, *max, *min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3797,7 +3806,6 @@ static FORCE_INLINE void tColDataCalcSMATinySmallInt(SColData *pColData, int64_t
CALC_SUM_MAX_MIN(*sum, *max, *min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3829,7 +3837,6 @@ static FORCE_INLINE void tColDataCalcSMAInt(SColData *pColData, int64_t *sum, in
CALC_SUM_MAX_MIN(*sum, *max, *min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3861,7 +3868,6 @@ static FORCE_INLINE void tColDataCalcSMABigInt(SColData *pColData, int64_t *sum,
CALC_SUM_MAX_MIN(*sum, *max, *min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3893,7 +3899,6 @@ static FORCE_INLINE void tColDataCalcSMAFloat(SColData *pColData, int64_t *sum,
CALC_SUM_MAX_MIN(*(double *)sum, *(double *)max, *(double *)min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3925,7 +3930,6 @@ static FORCE_INLINE void tColDataCalcSMADouble(SColData *pColData, int64_t *sum,
CALC_SUM_MAX_MIN(*(double *)sum, *(double *)max, *(double *)min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3957,7 +3961,6 @@ static FORCE_INLINE void tColDataCalcSMAUTinyInt(SColData *pColData, int64_t *su
CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -3989,7 +3992,6 @@ static FORCE_INLINE void tColDataCalcSMATinyUSmallInt(SColData *pColData, int64_
CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -4021,7 +4023,6 @@ static FORCE_INLINE void tColDataCalcSMAUInt(SColData *pColData, int64_t *sum, i
CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -4053,7 +4054,6 @@ static FORCE_INLINE void tColDataCalcSMAUBigInt(SColData *pColData, int64_t *sum
CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val);
break;
default:
ASSERT(0);
break;
}
}
@ -4092,7 +4092,6 @@ static FORCE_INLINE void tColDataCalcSMAVarType(SColData *pColData, int64_t *sum
}
break;
default:
ASSERT(0);
break;
}
}
@ -4153,7 +4152,9 @@ int32_t tValueColumnAppend(SValueColumn *valCol, const SValue *value) {
valCol->type = value->type;
}
ASSERT(value->type == valCol->type);
if (!(value->type == valCol->type)) {
return TSDB_CODE_INVALID_PARA;
}
if (IS_VAR_DATA_TYPE(value->type)) {
if ((code = tBufferPutI32(&valCol->offsets, tBufferGetSize(&valCol->data)))) {
@ -4230,7 +4231,9 @@ int32_t tValueColumnGet(SValueColumn *valCol, int32_t idx, SValue *value) {
int32_t tValueColumnCompress(SValueColumn *valCol, SValueColumnCompressInfo *info, SBuffer *output, SBuffer *assist) {
int32_t code;
ASSERT(valCol->numOfValues > 0);
if (!(valCol->numOfValues > 0)) {
return TSDB_CODE_INVALID_PARA;
}
(*info) = (SValueColumnCompressInfo){
.cmprAlg = info->cmprAlg,
@ -4344,7 +4347,7 @@ int32_t tValueColumnCompressInfoDecode(SBufferReader *reader, SValueColumnCompre
if ((code = tBufferGetI32v(reader, &info->dataOriginalSize))) return code;
if ((code = tBufferGetI32v(reader, &info->dataCompressedSize))) return code;
} else {
ASSERT(0);
return TSDB_CODE_INVALID_PARA;
}
return 0;
@ -4360,7 +4363,9 @@ int32_t tCompressData(void *input, // input
int32_t code;
extraSizeNeeded = (info->cmprAlg == NO_COMPRESSION) ? info->originalSize : info->originalSize + COMP_OVERFLOW_BYTES;
ASSERT(outputSize >= extraSizeNeeded);
if (!(outputSize >= extraSizeNeeded)) {
return TSDB_CODE_INVALID_PARA;
}
if (info->cmprAlg == NO_COMPRESSION) {
(void)memcpy(output, input, info->originalSize);
@ -4442,10 +4447,14 @@ int32_t tDecompressData(void *input, // input
) {
int32_t code;
ASSERT(outputSize >= info->originalSize);
if (!(outputSize >= info->originalSize)) {
return TSDB_CODE_INVALID_PARA;
}
if (info->cmprAlg == NO_COMPRESSION) {
ASSERT(info->compressedSize == info->originalSize);
if (!(info->compressedSize == info->originalSize)) {
return TSDB_CODE_INVALID_PARA;
}
(void)memcpy(output, input, info->compressedSize);
} else if (info->cmprAlg == ONE_STAGE_COMP || info->cmprAlg == TWO_STAGE_COMP) {
SBuffer local;
@ -4478,7 +4487,9 @@ int32_t tDecompressData(void *input, // input
return TSDB_CODE_COMPRESS_ERROR;
}
ASSERT(decompressedSize == info->originalSize);
if (!(decompressedSize == info->originalSize)) {
return TSDB_CODE_COMPRESS_ERROR;
}
tBufferDestroy(&local);
} else {
DEFINE_VAR(info->cmprAlg);
@ -4512,7 +4523,9 @@ int32_t tDecompressData(void *input, // input
return TSDB_CODE_COMPRESS_ERROR;
}
ASSERT(decompressedSize == info->originalSize);
if (!(decompressedSize == info->originalSize)) {
return TSDB_CODE_COMPRESS_ERROR;
}
tBufferDestroy(&local);
}

15
source/common/src/tglobal.c
View File

@ -303,10 +303,10 @@ char tsS3BucketName[TSDB_FQDN_LEN] = "<bucketname>";
char tsS3AppId[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<appid>"};
int8_t tsS3Enabled = false;
int8_t tsS3EnabledCfg = false;
int8_t tsS3Oss = false;
int8_t tsS3Oss[TSDB_MAX_EP_NUM] = {false};
int8_t tsS3StreamEnabled = false;
int8_t tsS3Https = true;
int8_t tsS3Https[TSDB_MAX_EP_NUM] = {true};
char tsS3Hostname[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<hostname>"};
int32_t tsS3BlockSize = -1; // number of tsdb pages (4096)
@ -431,11 +431,10 @@ int32_t taosSetS3Cfg(SConfig *pCfg) {
tstrncpy(tsS3AppId[i], appid + 1, TSDB_FQDN_LEN);
}
}
tsS3Https[i] = (strstr(tsS3Endpoint[i], "https://") != NULL);
tsS3Oss[i] = (strstr(tsS3Endpoint[i], "aliyuncs.") != NULL);
}
tsS3Https = (strstr(tsS3Endpoint[0], "https://") != NULL);
tsS3Oss = (strstr(tsS3Endpoint[0], "aliyuncs.") != NULL);
if (tsS3BucketName[0] != '<') {
#if defined(USE_COS) || defined(USE_S3)
#ifdef TD_ENTERPRISE
@ -1600,6 +1599,12 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "minDiskFreeSize");
tsMinDiskFreeSize = pItem->i64;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3MigrateIntervalSec");
tsS3MigrateIntervalSec = pItem->i32;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3MigrateEnabled");
tsS3MigrateEnabled = (bool)pItem->bval;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3PageCacheSize");
tsS3PageCacheSize = pItem->i32;

27
source/common/src/tmsg.c
View File

@ -103,7 +103,9 @@ int32_t tInitSubmitMsgIter(const SSubmitReq *pMsg, SSubmitMsgIter *pIter) {
pIter->totalLen = htonl(pMsg->length);
pIter->numOfBlocks = htonl(pMsg->numOfBlocks);
ASSERT(pIter->totalLen > 0);
if (!(pIter->totalLen > 0)) {
return terrno = TSDB_CODE_TDB_SUBMIT_MSG_MSSED_UP;
}
pIter->len = 0;
pIter->pMsg = pMsg;
if (pIter->totalLen <= sizeof(SSubmitReq)) {
@ -114,17 +116,21 @@ int32_t tInitSubmitMsgIter(const SSubmitReq *pMsg, SSubmitMsgIter *pIter) {
}
int32_t tGetSubmitMsgNext(SSubmitMsgIter *pIter, SSubmitBlk **pPBlock) {
ASSERT(pIter->len >= 0);
if (!(pIter->len >= 0)) {
return terrno = TSDB_CODE_INVALID_MSG_LEN;
}
if (pIter->len == 0) {
pIter->len += sizeof(SSubmitReq);
} else {
if (pIter->len >= pIter->totalLen) {
ASSERT(0);
return terrno = TSDB_CODE_INVALID_MSG_LEN;
}
pIter->len += (sizeof(SSubmitBlk) + pIter->dataLen + pIter->schemaLen);
ASSERT(pIter->len > 0);
if (!(pIter->len > 0)) {
return terrno = TSDB_CODE_INVALID_MSG_LEN;
}
}
if (pIter->len > pIter->totalLen) {
@ -377,7 +383,9 @@ static int32_t tDeserializeSClientHbReq(SDecoder *pDecoder, SClientHbReq *pReq)
}
}
ASSERT(desc.subPlanNum == taosArrayGetSize(desc.subDesc));
if (!(desc.subPlanNum == taosArrayGetSize(desc.subDesc))) {
return TSDB_CODE_INVALID_MSG;
}
if (!taosArrayPush(pReq->query->queryDesc, &desc)) {
return terrno;
@ -8476,7 +8484,7 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) {
} else if (pReq->type == TSDB_NORMAL_TABLE) {
if (tEncodeSSchemaWrapper(pCoder, &pReq->ntb.schemaRow) < 0) return -1;
} else {
ASSERT(0);
return TSDB_CODE_INVALID_MSG;
}
// ENCODESQL
@ -8528,7 +8536,7 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) {
} else if (pReq->type == TSDB_NORMAL_TABLE) {
if (tDecodeSSchemaWrapperEx(pCoder, &pReq->ntb.schemaRow) < 0) return -1;
} else {
ASSERT(0);
return TSDB_CODE_INVALID_MSG;
}
// DECODESQL
@ -9311,7 +9319,6 @@ bool tOffsetEqual(const STqOffsetVal *pLeft, const STqOffsetVal *pRight) {
return pLeft->uid == pRight->uid;
} else {
uError("offset type:%d", pLeft->type);
ASSERT(0);
}
}
return false;
@ -9727,7 +9734,9 @@ static int32_t tEncodeSSubmitTbData(SEncoder *pCoder, const SSubmitTbData *pSubm
// auto create table
if (pSubmitTbData->flags & SUBMIT_REQ_AUTO_CREATE_TABLE) {
ASSERT(pSubmitTbData->pCreateTbReq);
if (!(pSubmitTbData->pCreateTbReq)) {
return TSDB_CODE_INVALID_MSG;
}
if (tEncodeSVCreateTbReq(pCoder, pSubmitTbData->pCreateTbReq) < 0) return -1;
}

6
source/common/src/tname.c
View File

@ -105,7 +105,6 @@ int32_t tNameExtractFullName(const SName* name, char* dst) {
size_t tnameLen = strlen(name->tname);
if (tnameLen > 0) {
/*ASSERT(name->type == TSDB_TABLE_NAME_T);*/
dst[len] = TS_PATH_DELIMITER[0];
memcpy(dst + len + 1, name->tname, tnameLen);
@ -160,7 +159,10 @@ int32_t tNameGetFullDbName(const SName* name, char* dst) {
bool tNameIsEmpty(const SName* name) { return name->type == 0 || name->acctId == 0; }
const char* tNameGetTableName(const SName* name) {
ASSERT(name != NULL && name->type == TSDB_TABLE_NAME_T);
if (!(name != NULL && name->type == TSDB_TABLE_NAME_T)) {
terrno = TSDB_CODE_INVALID_PARA;
return NULL;
}
return &name->tname[0];
}

9
source/common/src/tvariant.c
View File

@ -459,8 +459,8 @@ void taosVariantDestroy(SVariant *pVar) {
}
}
void taosVariantAssign(SVariant *pDst, const SVariant *pSrc) {
if (pSrc == NULL || pDst == NULL) return;
int32_t taosVariantAssign(SVariant *pDst, const SVariant *pSrc) {
if (pSrc == NULL || pDst == NULL) return 0;
pDst->nType = pSrc->nType;
if (pSrc->nType == TSDB_DATA_TYPE_BINARY || pSrc->nType == TSDB_DATA_TYPE_VARBINARY ||
@ -468,19 +468,20 @@ void taosVariantAssign(SVariant *pDst, const SVariant *pSrc) {
pSrc->nType == TSDB_DATA_TYPE_GEOMETRY) {
int32_t len = pSrc->nLen + TSDB_NCHAR_SIZE;
char *p = taosMemoryRealloc(pDst->pz, len);
ASSERT(p);
if (!p) return terrno;
(void)memset(p, 0, len);
pDst->pz = p;
(void)memcpy(pDst->pz, pSrc->pz, pSrc->nLen);
pDst->nLen = pSrc->nLen;
return;
return 0;
}
if (IS_NUMERIC_TYPE(pSrc->nType) || (pSrc->nType == TSDB_DATA_TYPE_BOOL)) {
pDst->i = pSrc->i;
}
return 0;
}
int32_t taosVariantCompare(const SVariant *p1, const SVariant *p2) {

2
source/dnode/mnode/impl/src/mndSync.c
View File

@ -486,7 +486,7 @@ int32_t mndInitSync(SMnode *pMnode) {
int32_t code = 0;
(void)tsem_init(&pMgmt->syncSem, 0, 0);
pMgmt->sync = syncOpen(&syncInfo, true);
pMgmt->sync = syncOpen(&syncInfo, 1); // always check
if (pMgmt->sync <= 0) {
if (terrno != 0) code = terrno;
mError("failed to open sync since %s", tstrerror(code));

2
source/dnode/vnode/src/inc/tsdb.h
View File

@ -898,7 +898,7 @@ typedef struct SSttDataInfoForTable {
int32_t tMergeTreeOpen2(SMergeTree *pMTree, SMergeTreeConf *pConf, SSttDataInfoForTable *pTableInfo);
void tMergeTreeAddIter(SMergeTree *pMTree, SLDataIter *pIter);
bool tMergeTreeNext(SMergeTree *pMTree);
int32_t tMergeTreeNext(SMergeTree *pMTree, bool* pHasNext);
void tMergeTreePinSttBlock(SMergeTree *pMTree);
void tMergeTreeUnpinSttBlock(SMergeTree *pMTree);
bool tMergeTreeIgnoreEarlierTs(SMergeTree *pMTree);

1
source/dnode/vnode/src/meta/metaCache.c
View File

@ -503,7 +503,6 @@ static int checkAllEntriesInCache(const STagFilterResEntry* pEntry, SArray* pInv
}
static FORCE_INLINE void setMD5DigestInKey(uint64_t* pBuf, const char* key, int32_t keyLen) {
// ASSERT(keyLen == sizeof(int64_t) * 2);
memcpy(&pBuf[2], key, keyLen);
}

4
source/dnode/vnode/src/meta/metaOpen.c
View File

@ -332,6 +332,10 @@ static int tagIdxKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kL
} else if (!pTagIdxKey1->isNull && !pTagIdxKey2->isNull) {
// all not NULL, compr tag vals
__compar_fn_t func = getComparFunc(pTagIdxKey1->type, 0);
if (func == NULL) {
metaError("meta/open: %s", terrstr());
return TSDB_CODE_FAILED;
}
c = func(pTagIdxKey1->data, pTagIdxKey2->data);
if (c) return c;
}

21
source/dnode/vnode/src/meta/metaQuery.c
View File

@ -694,9 +694,13 @@ int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sv
SSchemaWrapper *pSchemaWrapper = &schema;
tDecoderInit(&dc, pData, nData);
(void)tDecodeSSchemaWrapper(&dc, pSchemaWrapper);
code = tDecodeSSchemaWrapper(&dc, pSchemaWrapper);
tDecoderClear(&dc);
tdbFree(pData);
if (TSDB_CODE_SUCCESS != code) {
taosMemoryFree(pSchemaWrapper->pSchema);
goto _exit;
}
// convert
STSchema *pTSchema = tBuildTSchema(pSchemaWrapper->pSchema, pSchemaWrapper->nCols, pSchemaWrapper->version);
@ -1099,7 +1103,12 @@ int32_t metaFilterCreateTime(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
SBtimeIdxKey *p = entryKey;
if (count > TRY_ERROR_LIMIT) break;
terrno = TSDB_CODE_SUCCESS;
int32_t cmp = (*param->filterFunc)((void *)&p->btime, (void *)&pBtimeKey->btime, param->type);
if (terrno != TSDB_CODE_SUCCESS) {
ret = terrno;
break;
}
if (cmp == 0) {
if (taosArrayPush(pUids, &p->uid) == NULL) {
ret = terrno;
@ -1163,7 +1172,12 @@ int32_t metaFilterTableName(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
if (count > TRY_ERROR_LIMIT) break;
char *pTableKey = (char *)pEntryKey;
terrno = TSDB_CODE_SUCCESS;
cmp = (*param->filterFunc)(pTableKey, pName, pCursor->type);
if (terrno != TSDB_CODE_SUCCESS) {
ret = terrno;
goto END;
}
if (cmp == 0) {
tb_uid_t tuid = *(tb_uid_t *)pEntryVal;
if (taosArrayPush(pUids, &tuid) == NULL) {
@ -1357,7 +1371,12 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
}
}
terrno = TSDB_CODE_SUCCESS;
int32_t cmp = (*param->filterFunc)(p->data, pKey->data, pKey->type);
if (terrno != TSDB_CODE_SUCCESS) {
TAOS_CHECK_GOTO(terrno, NULL, END);
break;
}
if (cmp == 0) {
// match
tb_uid_t tuid = 0;

4
source/dnode/vnode/src/sma/smaEnv.c
View File

@ -69,7 +69,7 @@ int32_t smaInit() {
if (!smaMgmt.refHash || !smaMgmt.tmrHandle) {
code = terrno;
(void)taosCloseRef(smaMgmt.rsetId);
taosCloseRef(smaMgmt.rsetId);
if (smaMgmt.refHash) {
taosHashCleanup(smaMgmt.refHash);
smaMgmt.refHash = NULL;
@ -103,7 +103,7 @@ void smaCleanUp() {
}
if (old == 1) {
(void)taosCloseRef(smaMgmt.rsetId);
taosCloseRef(smaMgmt.rsetId);
taosHashCleanup(smaMgmt.refHash);
smaMgmt.refHash = NULL;
taosTmrCleanUp(smaMgmt.tmrHandle);

4
source/dnode/vnode/src/tq/tq.c
View File

@ -1216,11 +1216,11 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
}
if (req.mndTrigger) {
qInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 ", transId:%d, ", pTask->id.idStr,
tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 ", transId:%d, ", pTask->id.idStr,
vgId, pTask->info.taskLevel, req.checkpointId, req.transId);
} else {
const char* pPrevStatus = streamTaskGetStatusStr(streamTaskGetPrevStatus(pTask));
qInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64
tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64
", transId:%d after transfer-state, prev status:%s",
pTask->id.idStr, vgId, pTask->info.taskLevel, req.checkpointId, req.transId, pPrevStatus);
}

33
source/dnode/vnode/src/tsdb/tsdbCache.c
View File

@ -1413,7 +1413,7 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr
SIdxKey *idxKey = taosArrayGet(remainCols, 0);
if (idxKey->key.cid != PRIMARYKEY_TIMESTAMP_COL_ID) {
// ignore 'ts' loaded from cache and load it from tsdb
SLastCol* pLastCol = taosArrayGet(pLastArray, 0);
SLastCol *pLastCol = taosArrayGet(pLastArray, 0);
tsdbCacheUpdateLastColToNone(pLastCol, TSDB_LAST_CACHE_NO_CACHE);
SLastKey *key = &(SLastKey){.lflag = ltype, .uid = uid, .cid = PRIMARYKEY_TIMESTAMP_COL_ID};
@ -1745,12 +1745,12 @@ int32_t tsdbCacheGetBatch(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SCache
goto _exit;
}
}
if (taosArrayPush(remainCols, &(SIdxKey){i, key}) ==NULL) {
if (taosArrayPush(remainCols, &(SIdxKey){i, key}) == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
bool ignoreRocks = pLastCol ? (pLastCol->cacheStatus == TSDB_LAST_CACHE_NO_CACHE) : false;
if (taosArrayPush(ignoreFromRocks, &ignoreRocks) ==NULL) {
if (taosArrayPush(ignoreFromRocks, &ignoreRocks) == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
@ -1802,12 +1802,12 @@ int32_t tsdbCacheGetBatch(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SCache
}
_exit:
if (remainCols) {
taosArrayDestroy(remainCols);
}
if (ignoreFromRocks) {
taosArrayDestroy(ignoreFromRocks);
}
if (remainCols) {
taosArrayDestroy(remainCols);
}
if (ignoreFromRocks) {
taosArrayDestroy(ignoreFromRocks);
}
TAOS_RETURN(code);
}
@ -2245,17 +2245,20 @@ static int32_t lastIterClose(SFSLastIter **iter) {
}
static int32_t lastIterNext(SFSLastIter *iter, TSDBROW **ppRow) {
int32_t code = 0;
bool hasVal = false;
*ppRow = NULL;
int32_t code = tMergeTreeNext(iter->pMergeTree, &hasVal);
if (code != 0) {
return code;
}
bool hasVal = tMergeTreeNext(iter->pMergeTree);
if (!hasVal) {
*ppRow = NULL;
TAOS_RETURN(code);
}
*ppRow = tMergeTreeGetRow(iter->pMergeTree);
TAOS_RETURN(code);
}
@ -2365,6 +2368,9 @@ static int32_t getNextRowFromFS(void *iter, TSDBROW **ppRow, bool *pIgnoreEarlie
if (!state->pIndexList) {
state->pIndexList = taosArrayInit(1, sizeof(SBrinBlk));
if (!state->pIndexList) {
TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err);
}
} else {
taosArrayClear(state->pIndexList);
}
@ -2650,7 +2656,6 @@ static int32_t getNextRowFromMem(void *iter, TSDBROW **ppRow, bool *pIgnoreEarli
TAOS_RETURN(code);
}
default:
ASSERT(0);
break;
}

18
source/dnode/vnode/src/tsdb/tsdbCommit2.c
View File

@ -208,8 +208,6 @@ static int32_t tsdbCommitOpenReader(SCommitter2 *committer) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(TARRAY2_SIZE(committer->sttReaderArray) == 0);
if (committer->ctx->info->fset == NULL //
|| committer->sttTrigger > 1 //
|| TARRAY2_SIZE(committer->ctx->info->fset->lvlArr) == 0 //
@ -264,11 +262,6 @@ static int32_t tsdbCommitOpenIter(SCommitter2 *committer) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(TARRAY2_SIZE(committer->dataIterArray) == 0);
ASSERT(committer->dataIterMerger == NULL);
ASSERT(TARRAY2_SIZE(committer->tombIterArray) == 0);
ASSERT(committer->tombIterMerger == NULL);
STsdbIter *iter;
STsdbIterConfig config = {0};
@ -342,10 +335,6 @@ static int32_t tsdbCommitFileSetBegin(SCommitter2 *committer) {
committer->ctx->tbid->suid = 0;
committer->ctx->tbid->uid = 0;
ASSERT(TARRAY2_SIZE(committer->dataIterArray) == 0);
ASSERT(committer->dataIterMerger == NULL);
ASSERT(committer->writer == NULL);
TAOS_CHECK_GOTO(tsdbCommitOpenReader(committer), &lino, _exit);
TAOS_CHECK_GOTO(tsdbCommitOpenIter(committer), &lino, _exit);
TAOS_CHECK_GOTO(tsdbCommitOpenWriter(committer), &lino, _exit);
@ -637,13 +626,10 @@ static int32_t tsdbCloseCommitter(SCommitter2 *committer, int32_t eno) {
if (eno == 0) {
TAOS_CHECK_GOTO(tsdbFSEditBegin(committer->tsdb->pFS, committer->fopArray, TSDB_FEDIT_COMMIT), &lino, _exit);
} else {
// TODO
ASSERT(0);
tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(committer->tsdb->pVnode), __func__, __FILE__, lino,
tstrerror(eno));
}
ASSERT(committer->writer == NULL);
ASSERT(committer->dataIterMerger == NULL);
ASSERT(committer->tombIterMerger == NULL);
TARRAY2_DESTROY(committer->dataIterArray, NULL);
TARRAY2_DESTROY(committer->tombIterArray, NULL);
TARRAY2_DESTROY(committer->sttReaderArray, NULL);

4
source/dnode/vnode/src/tsdb/tsdbDataFileRAW.c
View File

@ -113,7 +113,7 @@ _exit:
}
static int32_t tsdbDataFileRAWWriterCloseAbort(SDataFileRAWWriter *writer) {
ASSERT(0);
tsdbError("vgId:%d %s failed since not implemented", TD_VID(writer->config->tsdb->pVnode), __func__);
return 0;
}
@ -122,8 +122,6 @@ static int32_t tsdbDataFileRAWWriterDoClose(SDataFileRAWWriter *writer) { return
static int32_t tsdbDataFileRAWWriterCloseCommit(SDataFileRAWWriter *writer, TFileOpArray *opArr) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(writer->ctx->offset <= writer->file.size);
ASSERT(writer->config->fid == writer->file.fid);
STFileOp op = (STFileOp){
.optype = TSDB_FOP_CREATE,

54
source/dnode/vnode/src/tsdb/tsdbDataFileRW.c
View File

@ -283,7 +283,9 @@ int32_t tsdbDataFileReadBrinBlock(SDataFileReader *reader, const SBrinBlk *brinB
}
}
ASSERT(br.offset == br.buffer->size);
if (br.offset != br.buffer->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -313,7 +315,10 @@ int32_t tsdbDataFileReadBlockData(SDataFileReader *reader, const SBrinRecord *re
// decompress
SBufferReader br = BUFFER_READER_INITIALIZER(0, buffer);
TAOS_CHECK_GOTO(tBlockDataDecompress(&br, bData, assist), &lino, _exit);
ASSERT(br.offset == buffer->size);
if (br.offset != buffer->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -345,7 +350,9 @@ int32_t tsdbDataFileReadBlockDataByColumn(SDataFileReader *reader, const SBrinRe
SBufferReader br = BUFFER_READER_INITIALIZER(0, buffer0);
TAOS_CHECK_GOTO(tGetDiskDataHdr(&br, &hdr), &lino, _exit);
ASSERT(hdr.delimiter == TSDB_FILE_DLMT);
if (hdr.delimiter != TSDB_FILE_DLMT) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
tBlockDataReset(bData);
bData->suid = hdr.suid;
@ -354,7 +361,9 @@ int32_t tsdbDataFileReadBlockDataByColumn(SDataFileReader *reader, const SBrinRe
// Key part
TAOS_CHECK_GOTO(tBlockDataDecompressKeyPart(&hdr, &br, bData, assist), &lino, _exit);
ASSERT(br.offset == buffer0->size);
if (br.offset != buffer0->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
int extraColIdx = -1;
for (int i = 0; i < ncid; i++) {
@ -526,7 +535,9 @@ int32_t tsdbDataFileReadBlockSma(SDataFileReader *reader, const SBrinRecord *rec
TAOS_CHECK_GOTO(tGetColumnDataAgg(&br, sma), &lino, _exit);
TAOS_CHECK_GOTO(TARRAY2_APPEND_PTR(columnDataAggArray, sma), &lino, _exit);
}
ASSERT(br.offset == record->smaSize);
if (br.offset != record->smaSize) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
}
_exit:
@ -661,7 +672,8 @@ struct SDataFileWriter {
};
static int32_t tsdbDataFileWriterCloseAbort(SDataFileWriter *writer) {
ASSERT(0);
tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(writer->config->tsdb->pVnode), __func__, __FILE__, __LINE__,
"not implemented");
return 0;
}
@ -980,7 +992,9 @@ static int32_t tsdbDataFileDoWriteBlockData(SDataFileWriter *writer, SBlockData
return 0;
}
ASSERT(bData->uid);
if (!bData->uid) {
return TSDB_CODE_INVALID_PARA;
}
int32_t code = 0;
int32_t lino = 0;
@ -1234,7 +1248,6 @@ static int32_t tsdbDataFileWriteTableDataEnd(SDataFileWriter *writer) {
if (writer->ctx->tbHasOldData) {
TAOS_CHECK_GOTO(tsdbDataFileDoWriteTableOldData(writer, NULL /* as the largest key */), &lino, _exit);
ASSERT(writer->ctx->tbHasOldData == false);
}
TAOS_CHECK_GOTO(tsdbDataFileDoWriteBlockData(writer, writer->blockData), &lino, _exit);
@ -1251,9 +1264,6 @@ static int32_t tsdbDataFileWriteTableDataBegin(SDataFileWriter *writer, const TA
int32_t code = 0;
int32_t lino = 0;
ASSERT(writer->ctx->blockDataIdx == writer->ctx->blockData->nRow);
ASSERT(writer->blockData->nRow == 0);
SMetaInfo info;
bool drop = false;
TABLEID tbid1[1];
@ -1451,7 +1461,9 @@ int32_t tsdbFileWriteTombBlk(STsdbFD *fd, const TTombBlkArray *tombBlkArray, SFD
}
static int32_t tsdbDataFileDoWriteTombBlk(SDataFileWriter *writer) {
ASSERT(TARRAY2_SIZE(writer->tombBlkArray) > 0);
if (TARRAY2_SIZE(writer->tombBlkArray) <= 0) {
return TSDB_CODE_INVALID_PARA;
}
int32_t code = 0;
int32_t lino = 0;
@ -1523,7 +1535,9 @@ static int32_t tsdbDataFileDoWriteTombRecord(SDataFileWriter *writer, const STom
TAOS_CHECK_GOTO(tsdbDataFileDoWriteTombBlock(writer), &lino, _exit);
}
} else {
ASSERT(0);
tsdbError("vgId:%d duplicate tomb record, cid:%" PRId64 ", suid:%" PRId64 ", uid:%" PRId64 ", version:%" PRId64,
TD_VID(writer->config->tsdb->pVnode), writer->config->cid, record->suid, record->uid,
record->version);
}
}
@ -1566,7 +1580,9 @@ _exit:
int32_t tsdbFileWriteBrinBlk(STsdbFD *fd, TBrinBlkArray *brinBlkArray, SFDataPtr *ptr, int64_t *fileSize,
int32_t encryptAlgorithm, char *encryptKey) {
ASSERT(TARRAY2_SIZE(brinBlkArray) > 0);
if (TARRAY2_SIZE(brinBlkArray) <= 0) {
return TSDB_CODE_INVALID_PARA;
}
ptr->offset = *fileSize;
ptr->size = TARRAY2_DATA_LEN(brinBlkArray);
@ -1852,7 +1868,9 @@ int32_t tsdbDataFileWriteBlockData(SDataFileWriter *writer, SBlockData *bData) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(bData->uid);
if (!bData->uid) {
return TSDB_CODE_INVALID_PARA;
}
if (!writer->ctx->opened) {
TAOS_CHECK_GOTO(tsdbDataFileWriterDoOpen(writer), &lino, _exit);
@ -1895,7 +1913,9 @@ _exit:
}
int32_t tsdbDataFileFlush(SDataFileWriter *writer) {
ASSERT(writer->ctx->opened);
if (!writer->ctx->opened) {
return TSDB_CODE_INVALID_PARA;
}
if (writer->blockData->nRow == 0) return 0;
if (writer->ctx->tbHasOldData) return 0;
@ -1910,8 +1930,6 @@ static int32_t tsdbDataFileWriterOpenTombFD(SDataFileWriter *writer) {
char fname[TSDB_FILENAME_LEN];
int32_t ftype = TSDB_FTYPE_TOMB;
ASSERT(writer->files[ftype].size == 0);
int32_t flag = (TD_FILE_READ | TD_FILE_WRITE | TD_FILE_CREATE | TD_FILE_TRUNC);
int32_t lcn = writer->files[ftype].lcn;

25
source/dnode/vnode/src/tsdb/tsdbFS.c
View File

@ -83,7 +83,10 @@ static int32_t tsdbBinaryToFS(uint8_t *pData, int64_t nData, STsdbFS *pFS) {
}
}
ASSERT(n + sizeof(TSCKSUM) == nData);
if (n + sizeof(TSCKSUM) != nData) {
code = TSDB_CODE_FILE_CORRUPTED;
goto _exit;
}
_exit:
return code;
@ -450,8 +453,9 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe
taosMemoryFree(pHeadF);
}
} else {
ASSERT(pHeadF->offset == pSetNew->pHeadF->offset);
ASSERT(pHeadF->size == pSetNew->pHeadF->size);
if (pHeadF->offset != pSetNew->pHeadF->offset || pHeadF->size != pSetNew->pHeadF->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
}
// data
@ -499,7 +503,6 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe
// stt
if (sameDisk) {
if (pSetNew->nSttF > pSetOld->nSttF) {
ASSERT(pSetNew->nSttF == pSetOld->nSttF + 1);
pSetOld->aSttF[pSetOld->nSttF] = (SSttFile *)taosMemoryMalloc(sizeof(SSttFile));
if (pSetOld->aSttF[pSetOld->nSttF] == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
@ -509,7 +512,6 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe
pSetOld->aSttF[pSetOld->nSttF]->nRef = 1;
pSetOld->nSttF++;
} else if (pSetNew->nSttF < pSetOld->nSttF) {
ASSERT(pSetNew->nSttF == 1);
for (int32_t iStt = 0; iStt < pSetOld->nSttF; iStt++) {
SSttFile *pSttFile = pSetOld->aSttF[iStt];
nRef = atomic_sub_fetch_32(&pSttFile->nRef, 1);
@ -548,8 +550,10 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe
*pSetOld->aSttF[iStt] = *pSetNew->aSttF[iStt];
pSetOld->aSttF[iStt]->nRef = 1;
} else {
ASSERT(pSetOld->aSttF[iStt]->size == pSetOld->aSttF[iStt]->size);
ASSERT(pSetOld->aSttF[iStt]->offset == pSetOld->aSttF[iStt]->offset);
if (pSetOld->aSttF[iStt]->size != pSetOld->aSttF[iStt]->size ||
pSetOld->aSttF[iStt]->offset == pSetOld->aSttF[iStt]->offset) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
}
}
}
@ -777,8 +781,6 @@ int32_t tsdbFSOpen(STsdb *pTsdb, int8_t rollback) {
// empty one
code = tsdbSaveFSToFile(&pTsdb->fs, current);
TSDB_CHECK_CODE(code, lino, _exit);
ASSERT(!rollback);
}
// scan and fix FS
@ -796,7 +798,6 @@ int32_t tsdbFSClose(STsdb *pTsdb) {
int32_t code = 0;
if (pTsdb->fs.pDelFile) {
ASSERT(pTsdb->fs.pDelFile->nRef == 1);
taosMemoryFree(pTsdb->fs.pDelFile);
}
@ -804,20 +805,16 @@ int32_t tsdbFSClose(STsdb *pTsdb) {
SDFileSet *pSet = (SDFileSet *)taosArrayGet(pTsdb->fs.aDFileSet, iSet);
// head
ASSERT(pSet->pHeadF->nRef == 1);
taosMemoryFree(pSet->pHeadF);
// data
ASSERT(pSet->pDataF->nRef == 1);
taosMemoryFree(pSet->pDataF);
// sma
ASSERT(pSet->pSmaF->nRef == 1);
taosMemoryFree(pSet->pSmaF);
// stt
for (int32_t iStt = 0; iStt < pSet->nSttF; iStt++) {
ASSERT(pSet->aSttF[iStt]->nRef == 1);
taosMemoryFree(pSet->aSttF[iStt]);
}
}

8
source/dnode/vnode/src/tsdb/tsdbFS2.c
View File

@ -208,7 +208,9 @@ static int32_t load_fs(STsdb *pTsdb, const char *fname, TFileSetArray *arr) {
/* fmtv */
item1 = cJSON_GetObjectItem(json, "fmtv");
if (cJSON_IsNumber(item1)) {
ASSERT(item1->valuedouble == 1);
if (item1->valuedouble != 1) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
} else {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
@ -892,7 +894,6 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) {
// commit
code = commit_edit(fs);
ASSERT(code == 0);
TSDB_CHECK_CODE(code, lino, _exit);
// schedule merge
@ -1050,7 +1051,6 @@ static SHashObj *tsdbFSetRangeArrayToHash(TFileSetRangeArray *pRanges) {
STFileSetRange *u = TARRAY2_GET(pRanges, i);
int32_t fid = u->fid;
int32_t code = taosHashPut(pHash, &fid, sizeof(fid), u, sizeof(*u));
ASSERT(code == 0);
tsdbDebug("range diff hash fid:%d, sver:%" PRId64 ", ever:%" PRId64, u->fid, u->sver, u->ever);
}
return pHash;
@ -1190,10 +1190,8 @@ int32_t tsdbBeginTaskOnFileSet(STsdb *tsdb, int32_t fid, STFileSet **fset) {
(void)taosThreadCondWait(&(*fset)->beginTask, &tsdb->mutex);
(void)tsdbFSGetFSet(tsdb->pFS, fid, fset);
ASSERT(fset != NULL);
(*fset)->numWaitTask--;
ASSERT((*fset)->numWaitTask >= 0);
} else {
(*fset)->taskRunning = true;
break;

13
source/dnode/vnode/src/tsdb/tsdbFSet2.c
View File

@ -106,7 +106,9 @@ static void tsdbSttLvlRemove(SSttLvl **lvl) {
static int32_t tsdbSttLvlApplyEdit(STsdb *pTsdb, const SSttLvl *lvl1, SSttLvl *lvl2) {
int32_t code = 0;
ASSERT(lvl1->level == lvl2->level);
if (lvl1->level != lvl2->level) {
return TSDB_CODE_INVALID_PARA;
}
int32_t i1 = 0, i2 = 0;
while (i1 < TARRAY2_SIZE(lvl1->fobjArr) || i2 < TARRAY2_SIZE(lvl2->fobjArr)) {
@ -331,29 +333,24 @@ int32_t tsdbTFileSetEdit(STsdb *pTsdb, STFileSet *fset, const STFileOp *op) {
code = TARRAY2_SORT_INSERT(lvl->fobjArr, fobj, tsdbTFileObjCmpr);
if (code) return code;
} else {
ASSERT(fset->farr[fobj->f->type] == NULL);
fset->farr[fobj->f->type] = fobj;
}
} else if (op->optype == TSDB_FOP_REMOVE) {
// delete a file
if (op->of.type == TSDB_FTYPE_STT) {
SSttLvl *lvl = tsdbTFileSetGetSttLvl(fset, op->of.stt->level);
ASSERT(lvl);
STFileObj tfobj = {.f[0] = {.cid = op->of.cid}};
STFileObj *tfobjp = &tfobj;
int32_t idx = TARRAY2_SEARCH_IDX(lvl->fobjArr, &tfobjp, tsdbTFileObjCmpr, TD_EQ);
ASSERT(idx >= 0);
TARRAY2_REMOVE(lvl->fobjArr, idx, tsdbSttLvlClearFObj);
} else {
ASSERT(tsdbIsSameTFile(&op->of, fset->farr[op->of.type]->f));
(void)tsdbTFileObjUnref(fset->farr[op->of.type]);
fset->farr[op->of.type] = NULL;
}
} else {
if (op->nf.type == TSDB_FTYPE_STT) {
SSttLvl *lvl = tsdbTFileSetGetSttLvl(fset, op->of.stt->level);
ASSERT(lvl);
STFileObj tfobj = {.f[0] = {.cid = op->of.cid}}, *tfobjp = &tfobj;
STFileObj **fobjPtr = TARRAY2_SEARCH(lvl->fobjArr, &tfobjp, tsdbTFileObjCmpr, TD_EQ);
@ -374,7 +371,9 @@ int32_t tsdbTFileSetEdit(STsdb *pTsdb, STFileSet *fset, const STFileOp *op) {
int32_t tsdbTFileSetApplyEdit(STsdb *pTsdb, const STFileSet *fset1, STFileSet *fset2) {
int32_t code = 0;
ASSERT(fset1->fid == fset2->fid);
if (fset1->fid != fset2->fid) {
return TSDB_CODE_INVALID_PARA;
}
for (tsdb_ftype_t ftype = TSDB_FTYPE_MIN; ftype < TSDB_FTYPE_MAX; ++ftype) {
if (!fset1->farr[ftype] && !fset2->farr[ftype]) continue;

7
source/dnode/vnode/src/tsdb/tsdbFSetRAW.c
View File

@ -154,13 +154,11 @@ _exit:
return code;
}
int32_t tsdbFSetRAWWriteBlockData(SFSetRAWWriter *writer, STsdbDataRAWBlockHeader *bHdr, int32_t encryptAlgorithm,
char* encryptKey) {
int32_t tsdbFSetRAWWriteBlockData(SFSetRAWWriter *writer, STsdbDataRAWBlockHeader *bHdr, int32_t encryptAlgorithm,
char *encryptKey) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(writer->ctx->offset >= 0 && writer->ctx->offset <= writer->ctx->file.size);
if (writer->ctx->offset == writer->ctx->file.size) {
code = tsdbFSetRAWWriteFileDataEnd(writer);
TSDB_CHECK_CODE(code, lino, _exit);
@ -173,7 +171,6 @@ int32_t tsdbFSetRAWWriteBlockData(SFSetRAWWriter *writer, STsdbDataRAWBlockHeade
TSDB_CHECK_CODE(code, lino, _exit);
writer->ctx->offset += bHdr->dataLength;
ASSERT(writer->ctx->offset == writer->dataWriter->ctx->offset);
_exit:
if (code) {

2
source/dnode/vnode/src/tsdb/tsdbFSetRW.c
View File

@ -73,8 +73,6 @@ static int32_t tsdbFSetWriteTableDataEnd(SFSetWriter *writer) {
int32_t numRow = ((writer->blockData[pidx].nRow + writer->blockData[cidx].nRow) >> 1);
if (writer->blockData[pidx].nRow > 0 && numRow >= writer->config->minRow) {
ASSERT(writer->blockData[pidx].nRow == writer->config->maxRow);
SRowInfo row = {
.suid = writer->ctx->tbid->suid,
.uid = writer->ctx->tbid->uid,

37
source/dnode/vnode/src/tsdb/tsdbFile2.c
View File

@ -242,19 +242,30 @@ int32_t tsdbTFileObjInit(STsdb *pTsdb, const STFile *f, STFileObj **fobj) {
int32_t tsdbTFileObjRef(STFileObj *fobj) {
int32_t nRef;
(void)(void)taosThreadMutexLock(&fobj->mutex);
ASSERT(fobj->ref > 0 && fobj->state == TSDB_FSTATE_LIVE);
(void)taosThreadMutexLock(&fobj->mutex);
if (fobj->ref <= 0 || fobj->state != TSDB_FSTATE_LIVE) {
(void)taosThreadMutexUnlock(&fobj->mutex);
tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, fobj->ref);
return TSDB_CODE_FAILED;
}
nRef = ++fobj->ref;
(void)(void)taosThreadMutexUnlock(&fobj->mutex);
(void)taosThreadMutexUnlock(&fobj->mutex);
tsdbTrace("ref file %s, fobj:%p ref %d", fobj->fname, fobj, nRef);
return 0;
}
int32_t tsdbTFileObjUnref(STFileObj *fobj) {
(void)(void)taosThreadMutexLock(&fobj->mutex);
(void)taosThreadMutexLock(&fobj->mutex);
int32_t nRef = --fobj->ref;
(void)(void)taosThreadMutexUnlock(&fobj->mutex);
ASSERT(nRef >= 0);
(void)taosThreadMutexUnlock(&fobj->mutex);
if (nRef < 0) {
tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, nRef);
return TSDB_CODE_FAILED;
}
tsdbTrace("unref file %s, fobj:%p ref %d", fobj->fname, fobj, nRef);
if (nRef == 0) {
if (fobj->state == TSDB_FSTATE_DEAD) {
@ -319,7 +330,11 @@ static void tsdbTFileObjRemoveLC(STFileObj *fobj, bool remove_all) {
int32_t tsdbTFileObjRemove(STFileObj *fobj) {
(void)taosThreadMutexLock(&fobj->mutex);
ASSERT(fobj->state == TSDB_FSTATE_LIVE && fobj->ref > 0);
if (fobj->state != TSDB_FSTATE_LIVE || fobj->ref <= 0) {
(void)taosThreadMutexUnlock(&fobj->mutex);
tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, fobj->ref);
return TSDB_CODE_FAILED;
}
fobj->state = TSDB_FSTATE_DEAD;
int32_t nRef = --fobj->ref;
(void)taosThreadMutexUnlock(&fobj->mutex);
@ -333,7 +348,13 @@ int32_t tsdbTFileObjRemove(STFileObj *fobj) {
int32_t tsdbTFileObjRemoveUpdateLC(STFileObj *fobj) {
(void)taosThreadMutexLock(&fobj->mutex);
ASSERT(fobj->state == TSDB_FSTATE_LIVE && fobj->ref > 0);
if (fobj->state != TSDB_FSTATE_LIVE || fobj->ref <= 0) {
(void)taosThreadMutexUnlock(&fobj->mutex);
tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, fobj->ref);
return TSDB_CODE_FAILED;
}
fobj->state = TSDB_FSTATE_DEAD;
int32_t nRef = --fobj->ref;
(void)taosThreadMutexUnlock(&fobj->mutex);

28
source/dnode/vnode/src/tsdb/tsdbIter.c
View File

@ -547,8 +547,7 @@ int32_t tsdbIterOpen(const STsdbIterConfig *config, STsdbIter **iter) {
code = tsdbMemTombIterOpen(iter[0]);
break;
default:
code = TSDB_CODE_INVALID_PARA;
ASSERTS(false, "Not implemented");
return TSDB_CODE_INVALID_PARA;
}
if (code) {
@ -588,7 +587,7 @@ int32_t tsdbIterClose(STsdbIter **iter) {
case TSDB_ITER_TYPE_MEMT_TOMB:
break;
default:
ASSERT(false);
return TSDB_CODE_INVALID_PARA;
}
taosMemoryFree(iter[0]);
iter[0] = NULL;
@ -610,7 +609,7 @@ int32_t tsdbIterNext(STsdbIter *iter) {
case TSDB_ITER_TYPE_MEMT_TOMB:
return tsdbMemTombIterNext(iter, NULL);
default:
ASSERT(false);
return TSDB_CODE_INVALID_PARA;
}
return 0;
}
@ -630,7 +629,7 @@ static int32_t tsdbIterSkipTableData(STsdbIter *iter, const TABLEID *tbid) {
case TSDB_ITER_TYPE_MEMT_TOMB:
return tsdbMemTombIterNext(iter, tbid);
default:
ASSERT(false);
return TSDB_CODE_INVALID_PARA;
}
return 0;
}
@ -691,7 +690,10 @@ int32_t tsdbIterMergerOpen(const TTsdbIterArray *iterArray, SIterMerger **merger
TARRAY2_FOREACH(iterArray, iter) {
if (iter->noMoreData) continue;
node = tRBTreePut(merger[0]->iterTree, iter->node);
ASSERT(node);
if (node == NULL) {
taosMemoryFree(merger[0]);
return TSDB_CODE_INVALID_PARA;
}
}
return tsdbIterMergerNext(merger[0]);
@ -718,10 +720,8 @@ int32_t tsdbIterMergerNext(SIterMerger *merger) {
merger->iter = NULL;
} else if ((node = tRBTreeMin(merger->iterTree))) {
c = merger->iterTree->cmprFn(merger->iter->node, node);
ASSERT(c);
if (c > 0) {
node = tRBTreePut(merger->iterTree, merger->iter->node);
ASSERT(node);
merger->iter = NULL;
}
}
@ -734,15 +734,9 @@ int32_t tsdbIterMergerNext(SIterMerger *merger) {
return 0;
}
SRowInfo *tsdbIterMergerGetData(SIterMerger *merger) {
ASSERT(!merger->isTomb);
return merger->iter ? merger->iter->row : NULL;
}
SRowInfo *tsdbIterMergerGetData(SIterMerger *merger) { return merger->iter ? merger->iter->row : NULL; }
STombRecord *tsdbIterMergerGetTombRecord(SIterMerger *merger) {
ASSERT(merger->isTomb);
return merger->iter ? merger->iter->record : NULL;
}
STombRecord *tsdbIterMergerGetTombRecord(SIterMerger *merger) { return merger->iter ? merger->iter->record : NULL; }
int32_t tsdbIterMergerSkipTableData(SIterMerger *merger, const TABLEID *tbid) {
int32_t code;
@ -757,10 +751,8 @@ int32_t tsdbIterMergerSkipTableData(SIterMerger *merger, const TABLEID *tbid) {
merger->iter = NULL;
} else if ((node = tRBTreeMin(merger->iterTree))) {
c = merger->iterTree->cmprFn(merger->iter->node, node);
ASSERT(c);
if (c > 0) {
node = tRBTreePut(merger->iterTree, merger->iter->node);
ASSERT(node);
merger->iter = NULL;
}
}

17
source/dnode/vnode/src/tsdb/tsdbMemTable.c
View File

@ -170,7 +170,6 @@ int32_t tsdbDeleteTableData(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid
goto _err;
}
ASSERT(pPool != NULL);
// do delete
SDelData *pDelData = (SDelData *)vnodeBufPoolMalloc(pPool, sizeof(*pDelData));
if (pDelData == NULL) {
@ -183,7 +182,6 @@ int32_t tsdbDeleteTableData(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid
pDelData->pNext = NULL;
taosWLockLatch(&pTbData->lock);
if (pTbData->pHead == NULL) {
ASSERT(pTbData->pTail == NULL);
pTbData->pHead = pTbData->pTail = pDelData;
} else {
pTbData->pTail->pNext = pDelData;
@ -263,8 +261,6 @@ void tsdbTbDataIterOpen(STbData *pTbData, STsdbRowKey *pFrom, int8_t backward, S
bool tsdbTbDataIterNext(STbDataIter *pIter) {
pIter->pRow = NULL;
if (pIter->backward) {
ASSERT(pIter->pNode != pIter->pTbData->sl.pTail);
if (pIter->pNode == pIter->pTbData->sl.pHead) {
return false;
}
@ -274,8 +270,6 @@ bool tsdbTbDataIterNext(STbDataIter *pIter) {
return false;
}
} else {
ASSERT(pIter->pNode != pIter->pTbData->sl.pHead);
if (pIter->pNode == pIter->pTbData->sl.pTail) {
return false;
}
@ -366,7 +360,6 @@ static int32_t tsdbGetOrCreateTbData(SMemTable *pMemTable, tb_uid_t suid, tb_uid
SVBufPool *pPool = pMemTable->pTsdb->pVnode->inUse;
int8_t maxLevel = pMemTable->pTsdb->pVnode->config.tsdbCfg.slLevel;
ASSERT(pPool != NULL);
pTbData = vnodeBufPoolMallocAligned(pPool, sizeof(*pTbData) + SL_NODE_SIZE(maxLevel) * 2);
if (pTbData == NULL) {
code = terrno;
@ -516,8 +509,6 @@ static int32_t tbDataDoPut(SMemTable *pMemTable, STbData *pTbData, SMemSkipListN
pNode = (SMemSkipListNode *)vnodeBufPoolMallocAligned(pPool, nSize + pRow->pTSRow->len);
} else if (pRow->type == TSDBROW_COL_FMT) {
pNode = (SMemSkipListNode *)vnodeBufPoolMallocAligned(pPool, nSize);
} else {
ASSERT(0);
}
if (pNode == NULL) {
code = terrno;
@ -582,10 +573,6 @@ static int32_t tsdbInsertColDataToTable(SMemTable *pMemTable, STbData *pTbData,
int32_t nColData = TARRAY_SIZE(pSubmitTbData->aCol);
SColData *aColData = (SColData *)TARRAY_DATA(pSubmitTbData->aCol);
ASSERT(aColData[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID);
ASSERT(aColData[0].type == TSDB_DATA_TYPE_TIMESTAMP);
ASSERT(aColData[0].flag == HAS_VALUE);
// copy and construct block data
SBlockData *pBlockData = vnodeBufPoolMalloc(pPool, sizeof(*pBlockData));
if (pBlockData == NULL) {
@ -740,7 +727,9 @@ int32_t tsdbRefMemTable(SMemTable *pMemTable, SQueryNode *pQNode) {
int32_t code = 0;
int32_t nRef = atomic_fetch_add_32(&pMemTable->nRef, 1);
ASSERT(nRef > 0);
if (nRef <= 0) {
tsdbError("vgId:%d, memtable ref count is invalid, ref:%d", TD_VID(pMemTable->pTsdb->pVnode), nRef);
}
(void)vnodeBufPoolRegisterQuery(pMemTable->pPool, pQNode);

16
source/dnode/vnode/src/tsdb/tsdbMerge.c
View File

@ -69,13 +69,6 @@ static int32_t tsdbMergerClose(SMerger *merger) {
int32_t lino = 0;
SVnode *pVnode = merger->tsdb->pVnode;
ASSERT(merger->writer == NULL);
ASSERT(merger->dataIterMerger == NULL);
ASSERT(merger->tombIterMerger == NULL);
ASSERT(TARRAY2_SIZE(merger->dataIterArr) == 0);
ASSERT(TARRAY2_SIZE(merger->tombIterArr) == 0);
ASSERT(TARRAY2_SIZE(merger->sttReaderArr) == 0);
// clear the merge
TARRAY2_DESTROY(merger->tombIterArr, NULL);
TARRAY2_DESTROY(merger->dataIterArr, NULL);
@ -156,8 +149,6 @@ static int32_t tsdbMergeFileSetBeginOpenReader(SMerger *merger) {
}
}
ASSERT(merger->ctx->level > 0);
if (merger->ctx->level <= TSDB_MAX_LEVEL) {
TARRAY2_FOREACH_REVERSE(merger->ctx->fset->lvlArr, lvl) {
if (TARRAY2_SIZE(lvl->fobjArr) == 0) {
@ -180,8 +171,6 @@ static int32_t tsdbMergeFileSetBeginOpenReader(SMerger *merger) {
numFile = numFile - TARRAY2_SIZE(lvl->fobjArr) * pow(merger->sttTrigger, lvl->level);
}
ASSERT(numFile >= 0);
// get file system operations
TARRAY2_FOREACH(merger->ctx->fset->lvlArr, lvl) {
if (lvl->level >= merger->ctx->level) {
@ -323,11 +312,6 @@ static int32_t tsdbMergeFileSetBegin(SMerger *merger) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(TARRAY2_SIZE(merger->sttReaderArr) == 0);
ASSERT(TARRAY2_SIZE(merger->dataIterArr) == 0);
ASSERT(merger->dataIterMerger == NULL);
ASSERT(merger->writer == NULL);
TARRAY2_CLEAR(merger->fopArr, NULL);
merger->ctx->tbid->suid = 0;

44
source/dnode/vnode/src/tsdb/tsdbMergeTree.c
View File

@ -115,16 +115,14 @@ void destroySttBlockReader(SArray *pLDataIterArray, SSttBlockLoadCostInfo *pLoad
SArray *pList = taosArrayGetP(pLDataIterArray, i);
for (int32_t j = 0; j < taosArrayGetSize(pList); ++j) {
SLDataIter *pIter = taosArrayGetP(pList, j);
if (pIter->pBlockLoadInfo == NULL) {
continue;
}
SSttBlockLoadCostInfo *pCost = &pIter->pBlockLoadInfo->cost;
if (pLoadCost != NULL) {
pLoadCost->loadBlocks += pCost->loadBlocks;
pLoadCost->loadStatisBlocks += pCost->loadStatisBlocks;
pLoadCost->blockElapsedTime += pCost->blockElapsedTime;
pLoadCost->statisElapsedTime += pCost->statisElapsedTime;
if (pIter->pBlockLoadInfo != NULL) {
SSttBlockLoadCostInfo *pCost = &pIter->pBlockLoadInfo->cost;
if (pLoadCost != NULL) {
pLoadCost->loadBlocks += pCost->loadBlocks;
pLoadCost->loadStatisBlocks += pCost->loadStatisBlocks;
pLoadCost->blockElapsedTime += pCost->blockElapsedTime;
pLoadCost->statisElapsedTime += pCost->statisElapsedTime;
}
}
destroyLDataIter(pIter);
@ -903,6 +901,10 @@ int32_t tLDataIterNextRow(SLDataIter *pIter, const char *idStr, bool* hasNext) {
pIter->rInfo.row = tsdbRowFromBlockData(pBlockData, pIter->iRow);
_exit:
if (code) {
tsdbError("failed to exec stt-file nextIter, lino:%d, code:%s, %s", lino, tstrerror(code), idStr);
}
*hasNext = (code == TSDB_CODE_SUCCESS) && (pIter->pSttBlk != NULL) && (pBlockData != NULL);
return code;
}
@ -1102,13 +1104,22 @@ void tMergeTreeUnpinSttBlock(SMergeTree *pMTree) {
tLDataIterUnpinSttBlock(pIter, pMTree->idStr);
}
bool tMergeTreeNext(SMergeTree *pMTree) {
int32_t tMergeTreeNext(SMergeTree *pMTree, bool *pHasNext) {
int32_t code = 0;
if (pHasNext == NULL) {
return TSDB_CODE_INVALID_PARA;
}
*pHasNext = false;
if (pMTree->pIter) {
SLDataIter *pIter = pMTree->pIter;
bool hasVal = false;
int32_t code = tLDataIterNextRow(pIter, pMTree->idStr, &hasVal);
bool hasVal = false;
code = tLDataIterNextRow(pIter, pMTree->idStr, &hasVal);
if (!hasVal || (code != 0)) {
if (code == TSDB_CODE_FILE_CORRUPTED) {
code = 0; // suppress the file corrupt error to enable all queries within this cluster can run without failed.
}
pMTree->pIter = NULL;
}
@ -1117,7 +1128,7 @@ bool tMergeTreeNext(SMergeTree *pMTree) {
if (pMTree->pIter && pIter) {
int32_t c = pMTree->rbt.cmprFn(&pMTree->pIter->node, &pIter->node);
if (c > 0) {
(void) tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pMTree->pIter);
(void)tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pMTree->pIter);
pMTree->pIter = NULL;
} else {
ASSERT(c);
@ -1132,7 +1143,8 @@ bool tMergeTreeNext(SMergeTree *pMTree) {
}
}
return pMTree->pIter != NULL;
*pHasNext = (pMTree->pIter != NULL);
return code;
}
void tMergeTreeClose(SMergeTree *pMTree) {

92
source/dnode/vnode/src/tsdb/tsdbRead2.c
View File

@ -1383,7 +1383,6 @@ static int32_t copyBlockDataToSDataBlock(STsdbReader* pReader, SRowKey* pLastPro
static FORCE_INLINE STSchema* getTableSchemaImpl(STsdbReader* pReader, uint64_t uid) {
ASSERT(pReader->info.pSchema == NULL);
int32_t code = metaGetTbTSchemaEx(pReader->pTsdb->pVnode->pMeta, pReader->info.suid, uid, -1, &pReader->info.pSchema);
if (code != TSDB_CODE_SUCCESS || pReader->info.pSchema == NULL) {
terrno = code;
@ -1414,7 +1413,9 @@ static int32_t doLoadFileBlockData(STsdbReader* pReader, SDataBlockIter* pBlockI
if (pReader->info.pSchema == NULL) {
pSchema = getTableSchemaImpl(pReader, uid);
if (pSchema == NULL) {
tsdbDebug("%p table uid:%" PRIu64 " has been dropped, no data existed, %s", pReader, uid, pReader->idStr);
code = terrno;
tsdbError("%p table uid:%" PRIu64 " failed to get tableschema, code:%s, %s", pReader, uid, tstrerror(code),
pReader->idStr);
return code;
}
}
@ -1449,7 +1450,7 @@ static int32_t doLoadFileBlockData(STsdbReader* pReader, SDataBlockIter* pBlockI
pReader->cost.blockLoadTime += elapsedTime;
pDumpInfo->allDumped = false;
return TSDB_CODE_SUCCESS;
return code;
}
/**
@ -1759,14 +1760,22 @@ static bool tryCopyDistinctRowFromFileBlock(STsdbReader* pReader, SBlockData* pB
return code;
}
static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, int32_t pkSrcSlot,
SVersionRange* pVerRange) {
static int32_t nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, int32_t pkSrcSlot,
SVersionRange* pVerRange) {
int32_t code = 0;
int32_t order = pSttBlockReader->order;
int32_t step = ASCENDING_TRAVERSE(order) ? 1 : -1;
SRowKey* pNextProc = &pScanInfo->sttKeyInfo.nextProcKey;
while (1) {
bool hasVal = tMergeTreeNext(&pSttBlockReader->mergeTree);
bool hasVal = false;
code = tMergeTreeNext(&pSttBlockReader->mergeTree, &hasVal);
if (code) {
tsdbError("failed to iter the next row in stt-file merge tree, code:%s, %s", tstrerror(code),
pSttBlockReader->mergeTree.idStr);
return code;
}
if (!hasVal) { // the next value will be the accessed key in stt
pScanInfo->sttKeyInfo.status = STT_FILE_NO_DATA;
@ -1779,7 +1788,7 @@ static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockSc
memset(pNextProc->pks[0].pData, 0, pNextProc->pks[0].nData);
}
}
return false;
return code;
}
TSDBROW* pRow = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
@ -1798,13 +1807,15 @@ static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockSc
if (!hasBeenDropped(pScanInfo->delSkyline, &pScanInfo->sttBlockDelIndex, key, ver, order, pVerRange,
pSttBlockReader->numOfPks > 0)) {
pScanInfo->sttKeyInfo.status = STT_FILE_HAS_DATA;
return true;
return code;
}
} else {
pScanInfo->sttKeyInfo.status = STT_FILE_HAS_DATA;
return true;
return code;
}
}
return code;
}
static void doPinSttBlock(SSttBlockReader* pSttBlockReader) { tMergeTreePinSttBlock(&pSttBlockReader->mergeTree); }
@ -1819,9 +1830,14 @@ static bool tryCopyDistinctRowFromSttBlock(TSDBROW* fRow, SSttBlockReader* pSttB
// avoid the fetch next row replace the referenced stt block in buffer
doPinSttBlock(pSttBlockReader);
bool hasVal = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
doUnpinSttBlock(pSttBlockReader);
if (hasVal) {
if (code) {
return code;
}
if (hasDataInSttBlock(pScanInfo)) {
SRowKey* pNext = getCurrentKeyInSttBlock(pSttBlockReader);
if (pkCompEx(pSttKey, pNext) != 0) {
code = doAppendRowFromFileBlock(pReader->resBlockInfo.pResBlock, pReader, fRow->pBlockData, fRow->iRow);
@ -2125,7 +2141,7 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
if (piRow->type == TSDBROW_ROW_FMT) {
piSchema = doGetSchemaForTSRow(TSDBROW_SVERSION(piRow), pReader, pBlockScanInfo->uid);
if (piSchema == NULL) {
return code;
return terrno;
}
}
@ -2380,14 +2396,13 @@ static bool isValidFileBlockRow(SBlockData* pBlockData, int32_t rowIndex, STable
return true;
}
static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, STsdbReader* pReader) {
bool hasData = true;
static void initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, STsdbReader* pReader) {
int32_t order = pReader->info.order;
bool asc = ASCENDING_TRAVERSE(order);
// the stt block reader has been initialized for this table.
if (pSttBlockReader->uid == pScanInfo->uid) {
return hasDataInSttBlock(pScanInfo);
return;
}
if (pSttBlockReader->uid != 0) {
@ -2396,9 +2411,14 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
pSttBlockReader->uid = pScanInfo->uid;
// second time init stt block reader
// second or third time init stt block reader
if (pScanInfo->cleanSttBlocks && (pReader->info.execMode == READER_EXEC_ROWS)) {
return !pScanInfo->sttBlockReturned;
// only allowed to retrieve clean stt blocks for count once
if (pScanInfo->sttBlockReturned) {
pScanInfo->sttKeyInfo.status = STT_FILE_NO_DATA;
tsdbDebug("uid:%" PRIu64 " set no stt-file data after stt-block retrieved, %s", pScanInfo->uid, pReader->idStr);
}
return;
}
STimeWindow w = pSttBlockReader->window;
@ -2435,28 +2455,28 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
SSttDataInfoForTable info = {.pKeyRangeList = taosArrayInit(4, sizeof(SSttKeyRange))};
if (info.pKeyRangeList == NULL) {
pReader->code = terrno;
return false;
return;
}
int32_t code = tMergeTreeOpen2(&pSttBlockReader->mergeTree, &conf, &info);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(info.pKeyRangeList);
pReader->code = code;
return false;
return;
}
code = initMemDataIterator(pScanInfo, pReader);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(info.pKeyRangeList);
pReader->code = code;
return false;
return;
}
code = initDelSkylineIterator(pScanInfo, pReader->info.order, &pReader->cost);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(info.pKeyRangeList);
pReader->code = code;
return false;
return;
}
if (conf.rspRows) {
@ -2484,27 +2504,26 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
SRowKey* p = asc ? &pScanInfo->sttRange.skey : &pScanInfo->sttRange.ekey;
tRowKeyAssign(&pScanInfo->sttKeyInfo.nextProcKey, p);
hasData = (pScanInfo->sttKeyInfo.status == STT_FILE_HAS_DATA);
} else { // not clean stt blocks
INIT_KEYRANGE(&pScanInfo->sttRange); // reset the time window
pScanInfo->sttBlockReturned = false;
hasData = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
}
} else {
pScanInfo->cleanSttBlocks = false;
INIT_KEYRANGE(&pScanInfo->sttRange); // reset the time window
pScanInfo->sttBlockReturned = false;
hasData = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
}
pScanInfo->sttBlockReturned = false;
taosArrayDestroy(info.pKeyRangeList);
int64_t el = taosGetTimestampUs() - st;
pReader->cost.initSttBlockReader += (el / 1000.0);
tsdbDebug("init stt block reader completed, elapsed time:%" PRId64 "us %s", el, pReader->idStr);
return hasData;
if (code != 0) {
pReader->code = code;
}
}
static bool hasDataInSttBlock(STableBlockScanInfo* pInfo) { return pInfo->sttKeyInfo.status == STT_FILE_HAS_DATA; }
@ -2772,7 +2791,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
}
SBlockData* pBlockData = &pReader->status.fileBlockData;
(void) initSttBlockReader(pSttBlockReader, pBlockScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pBlockScanInfo, pReader);
if (pReader->code != 0) {
code = pReader->code;
goto _end;
@ -3190,12 +3209,12 @@ static int32_t doLoadSttBlockSequentially(STsdbReader* pReader) {
continue;
}
bool hasDataInSttFile = initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
if (pReader->code != TSDB_CODE_SUCCESS) {
return pReader->code;
}
if (!hasDataInSttFile) {
if (!hasDataInSttBlock(pScanInfo)) {
bool hasNexTable = moveToNextTable(pUidList, pStatus);
if (!hasNexTable) {
return TSDB_CODE_SUCCESS;
@ -3287,7 +3306,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
}
if (pScanInfo->sttKeyInfo.status == STT_FILE_READER_UNINIT) {
(void) initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
if (pReader->code != 0) {
return pReader->code;
}
@ -3331,7 +3350,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
int64_t st = taosGetTimestampUs();
// let's load data from stt files, make sure clear the cleanStt block flag before load the data from stt files
(void) initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
if (pReader->code != 0) {
return pReader->code;
}
@ -4087,7 +4106,12 @@ int32_t doMergeRowsInSttBlock(SSttBlockReader* pSttBlockReader, STableBlockScanI
SRowKey* pRowKey = &pScanInfo->lastProcKey;
int32_t code = TSDB_CODE_SUCCESS;
while (nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pkSrcSlot, pVerRange)) {
while (1) {
code = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pkSrcSlot, pVerRange);
if (code || (!hasDataInSttBlock(pScanInfo))) {
return code;
}
SRowKey* pNextKey = getCurrentKeyInSttBlock(pSttBlockReader);
int32_t ret = pkCompEx(pRowKey, pNextKey);

4
source/dnode/vnode/src/tsdb/tsdbReadUtil.c
View File

@ -194,7 +194,7 @@ int32_t initRowKey(SRowKey* pKey, int64_t ts, int32_t numOfPks, int32_t type, in
break;
}
default:
ASSERT(0);
return TSDB_CODE_INVALID_PARA;
}
} else {
switch (type) {
@ -223,7 +223,7 @@ int32_t initRowKey(SRowKey* pKey, int64_t ts, int32_t numOfPks, int32_t type, in
pKey->pks[0].val = UINT8_MAX;
break;
default:
ASSERT(0);
return TSDB_CODE_INVALID_PARA;
}
}
} else {

36
source/dnode/vnode/src/tsdb/tsdbReaderWriter.c
View File

@ -77,7 +77,9 @@ static int32_t tsdbOpenFileImpl(STsdbFD *pFD) {
}
}
ASSERT(pFD->szFile % szPage == 0);
if (pFD->szFile % szPage != 0) {
TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit);
}
pFD->szFile = pFD->szFile / szPage;
_exit:
@ -149,7 +151,6 @@ static int32_t tsdbWriteFilePage(STsdbFD *pFD, int32_t encryptAlgorithm, char *e
offset -= chunkoffset;
}
ASSERT(offset >= 0);
int64_t n = taosLSeekFile(pFD->pFD, offset, SEEK_SET);
if (n < 0) {
@ -202,7 +203,6 @@ static int32_t tsdbReadFilePage(STsdbFD *pFD, int64_t pgno, int32_t encryptAlgor
int32_t code = 0;
int32_t lino;
// ASSERT(pgno <= pFD->szFile);
if (!pFD->pFD) {
code = tsdbOpenFileImpl(pFD);
TSDB_CHECK_CODE(code, lino, _exit);
@ -216,7 +216,6 @@ static int32_t tsdbReadFilePage(STsdbFD *pFD, int64_t pgno, int32_t encryptAlgor
offset -= chunkoffset;
}
ASSERT(offset >= 0);
// seek
int64_t n = taosLSeekFile(pFD->pFD, offset, SEEK_SET);
@ -317,8 +316,9 @@ static int32_t tsdbReadFileImp(STsdbFD *pFD, int64_t offset, uint8_t *pBuf, int6
int32_t szPgCont = PAGE_CONTENT_SIZE(pFD->szPage);
int64_t bOffset = fOffset % pFD->szPage;
// ASSERT(pgno && pgno <= pFD->szFile);
ASSERT(bOffset < szPgCont);
if (bOffset >= szPgCont) {
TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit);
}
while (n < size) {
if (pFD->pgno != pgno) {
@ -417,7 +417,9 @@ static int32_t tsdbReadFileS3(STsdbFD *pFD, int64_t offset, uint8_t *pBuf, int64
int64_t pgno = OFFSET_PGNO(fOffset, pFD->szPage);
int64_t bOffset = fOffset % pFD->szPage;
ASSERT(bOffset < szPgCont);
if (bOffset >= szPgCont) {
TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit);
}
// 1, find pgnoStart & pgnoEnd to fetch from s3, if all pgs are local, no need to fetch
// 2, fetch pgnoStart ~ pgnoEnd from s3
@ -690,7 +692,9 @@ int32_t tsdbReadBlockIdx(SDataFReader *pReader, SArray *aBlockIdx) {
TSDB_CHECK_CODE(code = TSDB_CODE_OUT_OF_MEMORY, lino, _exit);
}
}
ASSERT(n == size);
if (n != size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -731,7 +735,9 @@ int32_t tsdbReadSttBlk(SDataFReader *pReader, int32_t iStt, SArray *aSttBlk) {
TSDB_CHECK_CODE(code = TSDB_CODE_OUT_OF_MEMORY, lino, _exit);
}
}
ASSERT(n == size);
if (n != size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -760,7 +766,9 @@ int32_t tsdbReadDataBlk(SDataFReader *pReader, SBlockIdx *pBlockIdx, SMapData *m
int32_t n;
code = tGetMapData(pReader->aBuf[0], mDataBlk, &n);
if (code) goto _exit;
ASSERT(n == size);
if (n != size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -861,7 +869,9 @@ int32_t tsdbReadDelDatav1(SDelFReader *pReader, SDelIdx *pDelIdx, SArray *aDelDa
}
}
ASSERT(n == size);
if (n != size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -901,7 +911,9 @@ int32_t tsdbReadDelIdx(SDelFReader *pReader, SArray *aDelIdx) {
}
}
ASSERT(n == size);
if (n != size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {

2
source/dnode/vnode/src/tsdb/tsdbRetention.c
View File

@ -403,8 +403,6 @@ static int32_t tsdbS3FidLevel(int32_t fid, STsdbKeepCfg *pKeepCfg, int32_t s3Kee
nowSec = nowSec * 1000000l;
} else if (pKeepCfg->precision == TSDB_TIME_PRECISION_NANO) {
nowSec = nowSec * 1000000000l;
} else {
ASSERT(0);
}
nowSec = nowSec - pKeepCfg->keepTimeOffset * tsTickPerHour[pKeepCfg->precision];

23
source/dnode/vnode/src/tsdb/tsdbSnapshot.c
View File

@ -67,9 +67,6 @@ static int32_t tsdbSnapReadFileSetOpenReader(STsdbSnapReader* reader) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(reader->dataReader == NULL);
ASSERT(TARRAY2_SIZE(reader->sttReaderArr) == 0);
// data
SDataFileReaderConfig config = {
.tsdb = reader->tsdb,
@ -125,11 +122,6 @@ static int32_t tsdbSnapReadFileSetOpenIter(STsdbSnapReader* reader) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(reader->dataIterMerger == NULL);
ASSERT(reader->tombIterMerger == NULL);
ASSERT(TARRAY2_SIZE(reader->dataIterArr) == 0);
ASSERT(TARRAY2_SIZE(reader->tombIterArr) == 0);
STsdbIter* iter;
STsdbIterConfig config = {
.filterByVersion = true,
@ -210,8 +202,6 @@ static int32_t tsdbSnapReadRangeBegin(STsdbSnapReader* reader) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(reader->ctx->fsr == NULL);
if (reader->ctx->fsrArrIdx < TARRAY2_SIZE(reader->fsrArr)) {
reader->ctx->fsr = TARRAY2_GET(reader->fsrArr, reader->ctx->fsrArrIdx++);
reader->ctx->isDataDone = false;
@ -335,7 +325,6 @@ static int32_t tsdbSnapReadTimeSeriesData(STsdbSnapReader* reader, uint8_t** dat
}
if (reader->blockData->nRow > 0) {
ASSERT(reader->blockData->suid || reader->blockData->uid);
code = tsdbSnapCmprData(reader, data);
TSDB_CHECK_CODE(code, lino, _exit);
}
@ -616,7 +605,6 @@ static int32_t tsdbSnapWriteTimeSeriesRow(STsdbSnapWriter* writer, SRowInfo* row
}
if (row->suid == INT64_MAX) {
ASSERT(writer->ctx->hasData == false);
goto _exit;
}
@ -634,9 +622,6 @@ static int32_t tsdbSnapWriteFileSetOpenReader(STsdbSnapWriter* writer) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(writer->ctx->dataReader == NULL);
ASSERT(TARRAY2_SIZE(writer->ctx->sttReaderArr) == 0);
if (writer->ctx->fset) {
// open data reader
SDataFileReaderConfig dataFileReaderConfig = {
@ -825,8 +810,6 @@ static int32_t tsdbSnapWriteFileSetBegin(STsdbSnapWriter* writer, int32_t fid) {
int32_t code = 0;
int32_t lino = 0;
ASSERT(writer->ctx->fsetWriteBegin == false);
STFileSet* fset = &(STFileSet){.fid = fid};
writer->ctx->fid = fid;
@ -885,7 +868,6 @@ static int32_t tsdbSnapWriteTombRecord(STsdbSnapWriter* writer, const STombRecor
}
if (record->suid == INT64_MAX) {
ASSERT(writer->ctx->hasTomb == false);
goto _exit;
}
@ -996,7 +978,6 @@ static int32_t tsdbSnapWriteDecmprTombBlock(SSnapDataHdr* hdr, STombBlock* tombB
TAOS_UNUSED(tTombBlockClear(tombBlock));
int64_t size = hdr->size;
ASSERT(size % TOMB_RECORD_ELEM_NUM == 0);
size = size / TOMB_RECORD_ELEM_NUM;
tombBlock->numOfRecords = size / sizeof(int64_t);
@ -1043,8 +1024,6 @@ static int32_t tsdbSnapWriteTombData(STsdbSnapWriter* writer, SSnapDataHdr* hdr)
TSDB_CHECK_CODE(code, lino, _exit);
}
ASSERT(writer->ctx->hasData == false);
for (int32_t i = 0; i < TOMB_BLOCK_SIZE(tombBlock); ++i) {
code = tTombBlockGet(tombBlock, i, &record);
TSDB_CHECK_CODE(code, lino, _exit);
@ -1177,7 +1156,7 @@ int32_t tsdbSnapWrite(STsdbSnapWriter* writer, SSnapDataHdr* hdr) {
code = tsdbSnapWriteTombData(writer, hdr);
TSDB_CHECK_CODE(code, lino, _exit);
} else {
ASSERT(0);
TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit);
}
_exit:

36
source/dnode/vnode/src/tsdb/tsdbSttFileRW.c
View File

@ -61,7 +61,9 @@ int32_t tsdbSttFileReaderOpen(const char *fname, const SSttFileReaderConfig *con
// // open each segment reader
int64_t offset = config->file->size - sizeof(SSttFooter);
ASSERT(offset >= TSDB_FHDR_SIZE);
if (offset < TSDB_FHDR_SIZE) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
int32_t encryptAlgoirthm = config->tsdb->pVnode->config.tsdbCfg.encryptAlgorithm;
char *encryptKey = config->tsdb->pVnode->config.tsdbCfg.encryptKey;
@ -115,7 +117,9 @@ int32_t tsdbSttFileReaderClose(SSttFileReader **reader) {
int32_t tsdbSttFileReadStatisBlk(SSttFileReader *reader, const TStatisBlkArray **statisBlkArray) {
if (!reader->ctx->statisBlkLoaded) {
if (reader->footer->statisBlkPtr->size > 0) {
ASSERT(reader->footer->statisBlkPtr->size % sizeof(SStatisBlk) == 0);
if (reader->footer->statisBlkPtr->size % sizeof(SStatisBlk) != 0) {
return TSDB_CODE_FILE_CORRUPTED;
}
int32_t size = reader->footer->statisBlkPtr->size / sizeof(SStatisBlk);
void *data = taosMemoryMalloc(reader->footer->statisBlkPtr->size);
@ -147,7 +151,9 @@ int32_t tsdbSttFileReadStatisBlk(SSttFileReader *reader, const TStatisBlkArray *
int32_t tsdbSttFileReadTombBlk(SSttFileReader *reader, const TTombBlkArray **tombBlkArray) {
if (!reader->ctx->tombBlkLoaded) {
if (reader->footer->tombBlkPtr->size > 0) {
ASSERT(reader->footer->tombBlkPtr->size % sizeof(STombBlk) == 0);
if (reader->footer->tombBlkPtr->size % sizeof(STombBlk) != 0) {
return TSDB_CODE_FILE_CORRUPTED;
}
int32_t size = reader->footer->tombBlkPtr->size / sizeof(STombBlk);
void *data = taosMemoryMalloc(reader->footer->tombBlkPtr->size);
@ -179,7 +185,9 @@ int32_t tsdbSttFileReadTombBlk(SSttFileReader *reader, const TTombBlkArray **tom
int32_t tsdbSttFileReadSttBlk(SSttFileReader *reader, const TSttBlkArray **sttBlkArray) {
if (!reader->ctx->sttBlkLoaded) {
if (reader->footer->sttBlkPtr->size > 0) {
ASSERT(reader->footer->sttBlkPtr->size % sizeof(SSttBlk) == 0);
if (reader->footer->sttBlkPtr->size % sizeof(SSttBlk) != 0) {
return TSDB_CODE_FILE_CORRUPTED;
}
int32_t size = reader->footer->sttBlkPtr->size / sizeof(SSttBlk);
void *data = taosMemoryMalloc(reader->footer->sttBlkPtr->size);
@ -256,7 +264,9 @@ int32_t tsdbSttFileReadBlockDataByColumn(SSttFileReader *reader, const SSttBlk *
SBufferReader br = BUFFER_READER_INITIALIZER(0, buffer0);
TAOS_CHECK_GOTO(tGetDiskDataHdr(&br, &hdr), &lino, _exit);
ASSERT(hdr.delimiter == TSDB_FILE_DLMT);
if (hdr.delimiter != TSDB_FILE_DLMT) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
// set data container
tBlockDataReset(bData);
@ -266,7 +276,9 @@ int32_t tsdbSttFileReadBlockDataByColumn(SSttFileReader *reader, const SSttBlk *
// key part
TAOS_CHECK_GOTO(tBlockDataDecompressKeyPart(&hdr, &br, bData, assist), &lino, _exit);
ASSERT(br.offset == buffer0->size);
if (br.offset != buffer0->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
bool loadExtra = false;
for (int i = 0; i < ncid; i++) {
@ -376,7 +388,10 @@ int32_t tsdbSttFileReadTombBlock(SSttFileReader *reader, const STombBlk *tombBlk
br.offset += tombBlk->size[i];
}
ASSERT(br.offset == tombBlk->dp->size);
if (br.offset != tombBlk->dp->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(reader->config->tsdb->pVnode), __func__, __FILE__, lino,
@ -444,7 +459,9 @@ int32_t tsdbSttFileReadStatisBlock(SSttFileReader *reader, const SStatisBlk *sta
}
}
ASSERT(br.offset == buffer0->size);
if (br.offset != buffer0->size) {
TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit);
}
_exit:
if (code) {
@ -814,8 +831,6 @@ _exit:
}
static void tsdbSttFWriterDoClose(SSttFileWriter *writer) {
ASSERT(writer->fd == NULL);
for (int32_t i = 0; i < ARRAY_SIZE(writer->local); ++i) {
tBufferDestroy(writer->local + i);
}
@ -854,7 +869,6 @@ static int32_t tsdbSttFWriterCloseCommit(SSttFileWriter *writer, TFileOpArray *o
tsdbCloseFile(&writer->fd);
ASSERT(writer->file->size > 0);
STFileOp op = (STFileOp){
.optype = TSDB_FOP_CREATE,
.fid = writer->config->fid,

107
source/dnode/vnode/src/tsdb/tsdbUtil.c
View File

@ -106,7 +106,6 @@ _exit:
#endif
void tMapDataGetItemByIdx(SMapData *pMapData, int32_t idx, void *pItem, int32_t (*tGetItemFn)(uint8_t *, void *)) {
ASSERT(idx >= 0 && idx < pMapData->nItem);
(void)tGetItemFn(pMapData->pData + pMapData->aOffset[idx], pItem);
}
@ -584,7 +583,8 @@ int32_t tsdbFidLevel(int32_t fid, STsdbKeepCfg *pKeepCfg, int64_t nowSec) {
} else if (pKeepCfg->precision == TSDB_TIME_PRECISION_NANO) {
nowSec = nowSec * 1000000000l;
} else {
ASSERT(0);
tsdbError("invalid time precision:%d", pKeepCfg->precision);
return 0;
}
nowSec = nowSec - pKeepCfg->keepTimeOffset * tsTickPerHour[pKeepCfg->precision];
@ -628,8 +628,6 @@ void tsdbRowGetColVal(TSDBROW *pRow, STSchema *pTSchema, int32_t iCol, SColVal *
*pColVal = COL_VAL_NONE(pTColumn->colId, pTColumn->type);
}
}
} else {
ASSERT(0);
}
}
@ -697,8 +695,6 @@ int32_t tsdbRowIterOpen(STSDBRowIter *pIter, TSDBROW *pRow, STSchema *pTSchema)
if (code) return code;
} else if (pRow->type == TSDBROW_COL_FMT) {
pIter->iColData = 0;
} else {
ASSERT(0);
}
return 0;
@ -730,8 +726,8 @@ SColVal *tsdbRowIterNext(STSDBRowIter *pIter) {
return NULL;
}
} else {
ASSERT(0);
return NULL; // suppress error report by compiler
tsdbError("invalid row type:%d", pIter->pRow->type);
return NULL;
}
}
@ -902,76 +898,6 @@ int32_t tsdbRowMergerGetRow(SRowMerger *pMerger, SRow **ppRow) {
return tRowBuild(pMerger->pArray, pMerger->pTSchema, ppRow);
}
/*
// delete skyline ======================================================
static int32_t tsdbMergeSkyline2(SArray *aSkyline1, SArray *aSkyline2, SArray *aSkyline) {
int32_t code = 0;
int32_t i1 = 0;
int32_t n1 = taosArrayGetSize(aSkyline1);
int32_t i2 = 0;
int32_t n2 = taosArrayGetSize(aSkyline2);
TSDBKEY *pSkyline1;
TSDBKEY *pSkyline2;
TSDBKEY item;
int64_t version1 = 0;
int64_t version2 = 0;
ASSERT(n1 > 0 && n2 > 0);
taosArrayClear(aSkyline);
while (i1 < n1 && i2 < n2) {
pSkyline1 = (TSDBKEY *)taosArrayGet(aSkyline1, i1);
pSkyline2 = (TSDBKEY *)taosArrayGet(aSkyline2, i2);
if (pSkyline1->ts < pSkyline2->ts) {
version1 = pSkyline1->version;
i1++;
} else if (pSkyline1->ts > pSkyline2->ts) {
version2 = pSkyline2->version;
i2++;
} else {
version1 = pSkyline1->version;
version2 = pSkyline2->version;
i1++;
i2++;
}
item.ts = TMIN(pSkyline1->ts, pSkyline2->ts);
item.version = TMAX(version1, version2);
if (taosArrayPush(aSkyline, &item) == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
}
while (i1 < n1) {
pSkyline1 = (TSDBKEY *)taosArrayGet(aSkyline1, i1);
item.ts = pSkyline1->ts;
item.version = pSkyline1->version;
if (taosArrayPush(aSkyline, &item) == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
i1++;
}
while (i2 < n2) {
pSkyline2 = (TSDBKEY *)taosArrayGet(aSkyline2, i2);
item.ts = pSkyline2->ts;
item.version = pSkyline2->version;
if (taosArrayPush(aSkyline, &item) == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
i2++;
}
_exit:
return code;
}
*/
// delete skyline ======================================================
static void tsdbMergeSkyline(SArray *pSkyline1, SArray *pSkyline2, SArray *pSkyline) {
int32_t i1 = 0;
@ -983,8 +909,6 @@ static void tsdbMergeSkyline(SArray *pSkyline1, SArray *pSkyline2, SArray *pSkyl
int64_t version1 = 0;
int64_t version2 = 0;
ASSERT(n1 > 0 && n2 > 0);
taosArrayClear(pSkyline);
TSDBKEY **pItem = TARRAY_GET_ELEM(pSkyline, 0);
@ -1217,7 +1141,9 @@ _exit:
int32_t tBlockDataInit(SBlockData *pBlockData, TABLEID *pId, STSchema *pTSchema, int16_t *aCid, int32_t nCid) {
int32_t code = 0;
ASSERT(pId->suid || pId->uid);
if (!pId->suid && !pId->uid) {
return TSDB_CODE_INVALID_PARA;
}
pBlockData->suid = pId->suid;
pBlockData->uid = pId->uid;
@ -1277,8 +1203,6 @@ void tBlockDataReset(SBlockData *pBlockData) {
}
void tBlockDataClear(SBlockData *pBlockData) {
ASSERT(pBlockData->suid || pBlockData->uid);
pBlockData->nRow = 0;
for (int32_t iColData = 0; iColData < pBlockData->nColData; iColData++) {
tColDataClear(tBlockDataGetColDataByIdx(pBlockData, iColData));
@ -1286,7 +1210,9 @@ void tBlockDataClear(SBlockData *pBlockData) {
}
int32_t tBlockDataAddColData(SBlockData *pBlockData, int16_t cid, int8_t type, int8_t cflag, SColData **ppColData) {
ASSERT(pBlockData->nColData == 0 || pBlockData->aColData[pBlockData->nColData - 1].cid < cid);
if (pBlockData->nColData != 0 && pBlockData->aColData[pBlockData->nColData - 1].cid >= cid) {
return TSDB_CODE_INVALID_PARA;
}
SColData *newColData = taosMemoryRealloc(pBlockData->aColData, sizeof(SColData) * (pBlockData->nColData + 1));
if (newColData == NULL) {
@ -1350,7 +1276,9 @@ int32_t tBlockDataAppendRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS
// uid
if (pBlockData->uid == 0) {
ASSERT(uid);
if (!uid) {
return TSDB_CODE_INVALID_PARA;
}
code = tRealloc((uint8_t **)&pBlockData->aUid, sizeof(int64_t) * (pBlockData->nRow + 1));
if (code) goto _exit;
pBlockData->aUid[pBlockData->nRow] = uid;
@ -1384,7 +1312,9 @@ int32_t tBlockDataUpdateRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS
// version
int64_t lversion = pBlockData->aVersion[pBlockData->nRow - 1];
int64_t rversion = TSDBROW_VERSION(pRow);
ASSERT(lversion != rversion);
if (lversion == rversion) {
return TSDB_CODE_INVALID_PARA;
}
if (rversion > lversion) {
pBlockData->aVersion[pBlockData->nRow - 1] = rversion;
}
@ -1398,7 +1328,8 @@ int32_t tBlockDataUpdateRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS
code = tBlockDataUpsertBlockRow(pBlockData, pRow->pBlockData, pRow->iRow, (rversion > lversion) ? 1 : -1);
if (code) goto _exit;
} else {
ASSERT(0);
code = TSDB_CODE_INVALID_PARA;
goto _exit;
}
_exit:
@ -1745,8 +1676,6 @@ int32_t tBlockDataDecompressColData(const SDiskDataHdr *hdr, const SBlockCol *bl
code = tBlockDataAddColData(blockData, blockCol->cid, blockCol->type, blockCol->cflag, &colData);
TSDB_CHECK_CODE(code, lino, _exit);
// ASSERT(blockCol->flag != HAS_NONE);
SColDataCompressInfo info = {
.cmprAlg = blockCol->alg,
.columnFlag = blockCol->cflag,

6
source/dnode/vnode/src/tsdb/tsdbUtil2.c
View File

@ -171,7 +171,7 @@ static int32_t tStatisBlockUpdate(STbStatisBlock *block, SRowInfo *row) {
TAOS_CHECK_RETURN(tBufferPutAt(&block->counts, (block->numOfRecords - 1) * sizeof(record.count), &record.count,
sizeof(record.count)));
} else {
ASSERT(0);
return TSDB_CODE_INVALID_PARA;
}
return 0;
@ -276,7 +276,9 @@ int32_t tBrinBlockClear(SBrinBlock *brinBlock) {
}
int32_t tBrinBlockPut(SBrinBlock *brinBlock, const SBrinRecord *record) {
ASSERT(record->firstKey.key.numOfPKs == record->lastKey.key.numOfPKs);
if (record->firstKey.key.numOfPKs != record->lastKey.key.numOfPKs) {
return TSDB_CODE_INVALID_PARA;
}
if (brinBlock->numOfRecords == 0) { // the first row
brinBlock->numOfPKs = record->firstKey.key.numOfPKs;

15
source/dnode/vnode/src/vnd/vnodeAsync.c
View File

@ -165,9 +165,7 @@ static int32_t vnodeAsyncTaskDone(SVAsync *async, SVATask *task) {
}
ret = vHashDrop(async->taskTable, task);
if (ret != 0) {
ASSERT(0);
}
TAOS_UNUSED(ret);
async->numTasks--;
if (task->numWait == 0) {
@ -403,7 +401,6 @@ static int32_t vnodeAsyncDestroy(SVAsync **async) {
}
(void)taosThreadJoin((*async)->workers[i].thread, NULL);
ASSERT((*async)->workers[i].state == EVA_WORKER_STATE_STOP);
(*async)->workers[i].state = EVA_WORKER_STATE_UINIT;
}
@ -413,18 +410,11 @@ static int32_t vnodeAsyncDestroy(SVAsync **async) {
channel->prev->next = channel->next;
int32_t ret = vHashDrop((*async)->channelTable, channel);
if (ret) {
ASSERT(0);
}
TAOS_UNUSED(ret);
(*async)->numChannels--;
taosMemoryFree(channel);
}
ASSERT((*async)->numLaunchWorkers == 0);
ASSERT((*async)->numIdleWorkers == 0);
ASSERT((*async)->numChannels == 0);
ASSERT((*async)->numTasks == 0);
(void)taosThreadMutexDestroy(&(*async)->mutex);
(void)taosThreadCondDestroy(&(*async)->hasTask);
@ -438,7 +428,6 @@ static int32_t vnodeAsyncDestroy(SVAsync **async) {
static int32_t vnodeAsyncLaunchWorker(SVAsync *async) {
for (int32_t i = 0; i < async->numWorkers; i++) {
ASSERT(async->workers[i].state != EVA_WORKER_STATE_IDLE);
if (async->workers[i].state == EVA_WORKER_STATE_ACTIVE) {
continue;
} else if (async->workers[i].state == EVA_WORKER_STATE_STOP) {

25
source/dnode/vnode/src/vnd/vnodeBufPool.c
View File

@ -103,17 +103,18 @@ int vnodeCloseBufPool(SVnode *pVnode) {
}
void vnodeBufPoolReset(SVBufPool *pPool) {
ASSERT(pPool->nQuery == 0);
if (pPool->nQuery != 0) {
vError("vgId:%d, buffer pool %p of id %d has %d queries, reset it may cause problem", TD_VID(pPool->pVnode), pPool,
pPool->id, pPool->nQuery);
}
for (SVBufPoolNode *pNode = pPool->pTail; pNode->prev; pNode = pPool->pTail) {
ASSERT(pNode->pnext == &pPool->pTail);
pNode->prev->pnext = &pPool->pTail;
pPool->pTail = pNode->prev;
pPool->size = pPool->size - sizeof(*pNode) - pNode->size;
taosMemoryFree(pNode);
}
ASSERT(pPool->size == pPool->ptr - pPool->node.data);
pPool->size = 0;
pPool->ptr = pPool->node.data;
}
@ -123,7 +124,11 @@ void *vnodeBufPoolMallocAligned(SVBufPool *pPool, int size) {
void *p = NULL;
uint8_t *ptr = NULL;
int paddingLen = 0;
ASSERT(pPool != NULL);
if (pPool == NULL) {
terrno = TSDB_CODE_INVALID_PARA;
return NULL;
}
if (pPool->lock) taosThreadSpinLock(pPool->lock);
@ -162,7 +167,11 @@ void *vnodeBufPoolMallocAligned(SVBufPool *pPool, int size) {
void *vnodeBufPoolMalloc(SVBufPool *pPool, int size) {
SVBufPoolNode *pNode;
void *p = NULL;
ASSERT(pPool != NULL);
if (pPool == NULL) {
terrno = TSDB_CODE_INVALID_PARA;
return NULL;
}
if (pPool->lock) taosThreadSpinLock(pPool->lock);
if (pPool->node.size >= pPool->ptr - pPool->node.data + size) {
@ -207,7 +216,9 @@ void vnodeBufPoolFree(SVBufPool *pPool, void *p) {
void vnodeBufPoolRef(SVBufPool *pPool) {
int32_t nRef = atomic_fetch_add_32(&pPool->nRef, 1);
ASSERT(nRef > 0);
if (nRef <= 0) {
vError("vgId:%d, buffer pool %p of id %d is referenced by %d", TD_VID(pPool->pVnode), pPool, pPool->id, nRef);
}
}
void vnodeBufPoolAddToFreeList(SVBufPool *pPool) {

4
source/dnode/vnode/src/vnd/vnodeCfg.c
View File

@ -375,11 +375,11 @@ int vnodeDecodeConfig(const SJson *pJson, void *pObj) {
}
tjsonGetNumberValue(pJson, "s3ChunkSize", pCfg->s3ChunkSize, code);
if (code < 0) {
if (code < 0 || pCfg->s3ChunkSize < TSDB_MIN_S3_CHUNK_SIZE) {
pCfg->s3ChunkSize = TSDB_DEFAULT_S3_CHUNK_SIZE;
}
tjsonGetNumberValue(pJson, "s3KeepLocal", pCfg->s3KeepLocal, code);
if (code < 0) {
if (code < 0 || pCfg->s3KeepLocal < TSDB_MIN_S3_KEEP_LOCAL) {
pCfg->s3KeepLocal = TSDB_DEFAULT_S3_KEEP_LOCAL;
}
tjsonGetNumberValue(pJson, "s3Compact", pCfg->s3Compact, code);

3
source/dnode/vnode/src/vnd/vnodeCommit.c
View File

@ -302,7 +302,6 @@ static int32_t vnodePrepareCommit(SVnode *pVnode, SCommitInfo *pInfo) {
TSDB_CHECK_CODE(code, lino, _exit);
(void)taosThreadMutexLock(&pVnode->mutex);
ASSERT(pVnode->onCommit == NULL);
pVnode->onCommit = pVnode->inUse;
pVnode->inUse = NULL;
(void)taosThreadMutexUnlock(&pVnode->mutex);
@ -339,7 +338,7 @@ static void vnodeReturnBufPool(SVnode *pVnode) {
pVnode->recycleTail = pPool;
}
} else {
ASSERT(0);
vError("vgId:%d, buffer pool %p of id %d nRef:%d", TD_VID(pVnode), pPool, pPool->id, nRef);
}
(void)taosThreadMutexUnlock(&pVnode->mutex);

1
source/dnode/vnode/src/vnd/vnodeHash.c
View File

@ -77,7 +77,6 @@ int32_t vHashDestroy(SVHashTable** ht) {
}
if (*ht) {
ASSERT((*ht)->numEntries == 0);
taosMemoryFree((*ht)->buckets);
taosMemoryFree(*ht);
(*ht) = NULL;

4
source/dnode/vnode/src/vnd/vnodeOpen.c
View File

@ -558,7 +558,9 @@ void vnodeClose(SVnode *pVnode) {
// start the sync timer after the queue is ready
int32_t vnodeStart(SVnode *pVnode) {
ASSERT(pVnode);
if (pVnode == NULL) {
return TSDB_CODE_INVALID_PARA;
}
return vnodeSyncStart(pVnode);
}

12
source/dnode/vnode/src/vnd/vnodeQuery.c
View File

@ -19,7 +19,6 @@
#define VNODE_GET_LOAD_RESET_VALS(pVar, oVal, vType, tags) \
do { \
int##vType##_t newVal = atomic_sub_fetch_##vType(&(pVar), (oVal)); \
ASSERT(newVal >= 0); \
if (newVal < 0) { \
vWarn("vgId:%d, %s, abnormal val:%" PRIi64 ", old val:%" PRIi64, TD_VID(pVnode), tags, newVal, (oVal)); \
} \
@ -37,7 +36,10 @@ int32_t fillTableColCmpr(SMetaReader *reader, SSchemaExt *pExt, int32_t numOfCol
int8_t tblType = reader->me.type;
if (useCompress(tblType)) {
SColCmprWrapper *p = &(reader->me.colCmpr);
ASSERT(numOfCol == p->nCols);
if (numOfCol != p->nCols) {
vError("fillTableColCmpr table type:%d, col num:%d, col cmpr num:%d mismatch", tblType, numOfCol, p->nCols);
return TSDB_CODE_APP_ERROR;
}
for (int i = 0; i < p->nCols; i++) {
SColCmpr *pCmpr = &p->pColCmpr[i];
pExt[i].colId = pCmpr->id;
@ -104,7 +106,8 @@ int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
} else if (mer1.me.type == TSDB_NORMAL_TABLE) {
schema = mer1.me.ntbEntry.schemaRow;
} else {
ASSERT(0);
vError("vnodeGetTableMeta get invalid table type:%d", mer1.me.type);
return TSDB_CODE_APP_ERROR;
}
metaRsp.numOfTags = schemaTag.nCols;
@ -262,7 +265,8 @@ int32_t vnodeGetTableCfg(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
}
}
} else {
ASSERT(0);
vError("vnodeGetTableCfg get invalid table type:%d", mer1.me.type);
return TSDB_CODE_APP_ERROR;
}
cfgRsp.numOfTags = schemaTag.nCols;

12
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -934,20 +934,23 @@ _exit:
}
static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp) {
int ret = 0;
SVDropTtlTableReq ttlReq = {0};
if (tDeserializeSVDropTtlTableReq(pReq, len, &ttlReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
ret = TSDB_CODE_INVALID_MSG;
goto end;
}
ASSERT(ttlReq.nUids == taosArrayGetSize(ttlReq.pTbUids));
if (ttlReq.nUids != taosArrayGetSize(ttlReq.pTbUids)) {
ret = TSDB_CODE_INVALID_MSG;
goto end;
}
if (ttlReq.nUids != 0) {
vInfo("vgId:%d, drop ttl table req will be processed, time:%d, ntbUids:%d", pVnode->config.vgId,
ttlReq.timestampSec, ttlReq.nUids);
}
int ret = 0;
if (ttlReq.nUids > 0) {
metaDropTables(pVnode->pMeta, ttlReq.pTbUids);
(void)tqUpdateTbUidList(pVnode->pTq, ttlReq.pTbUids, false);
@ -1787,8 +1790,7 @@ static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t ver, void *pReq, in
}
if (info.suid) {
code = metaGetInfo(pVnode->pMeta, info.suid, &info, NULL);
ASSERT(code == 0);
(void)metaGetInfo(pVnode->pMeta, info.suid, &info, NULL);
}
if (pSubmitTbData->sver != info.skmVer) {

40
source/libs/catalog/inc/catalogInt.h
View File

@ -852,34 +852,58 @@ typedef struct SCtgCacheItemInfo {
#define CTG_LOCK(type, _lock) \
do { \
if (CTG_READ == (type)) { \
ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value before read lock"); \
if (atomic_load_32((_lock)) < 0) { \
qError("invalid lock value before read lock"); \
break; \
} \
CTG_LOCK_DEBUG("CTG RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
taosRLockLatch(_lock); \
CTG_LOCK_DEBUG("CTG RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value after read lock"); \
if (atomic_load_32((_lock)) <= 0) { \
qError("invalid lock value after read lock"); \
break; \
} \
} else { \
ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value before write lock"); \
if (atomic_load_32((_lock)) < 0) { \
qError("invalid lock value before write lock"); \
break; \
} \
CTG_LOCK_DEBUG("CTG WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
taosWLockLatch(_lock); \
CTG_LOCK_DEBUG("CTG WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
ASSERTS(atomic_load_32((_lock)) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value after write lock"); \
if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \
qError("invalid lock value after write lock"); \
break; \
} \
} \
} while (0)
#define CTG_UNLOCK(type, _lock) \
do { \
if (CTG_READ == (type)) { \
ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value before read unlock"); \
if (atomic_load_32((_lock)) <= 0) { \
qError("invalid lock value before read unlock"); \
break; \
} \
CTG_LOCK_DEBUG("CTG RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
taosRUnLockLatch(_lock); \
CTG_LOCK_DEBUG("CTG RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after read unlock"); \
if (atomic_load_32((_lock)) < 0) { \
qError("invalid lock value after read unlock"); \
break; \
} \
} else { \
ASSERTS(atomic_load_32((_lock)) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value before write unlock"); \
if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \
qError("invalid lock value before write unlock"); \
break; \
} \
CTG_LOCK_DEBUG("CTG WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
taosWUnLockLatch(_lock); \
CTG_LOCK_DEBUG("CTG WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \
ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after write unlock"); \
if (atomic_load_32((_lock)) < 0) { \
qError("invalid lock value after write unlock"); \
break; \
} \
} \
} while (0)

2
source/libs/catalog/src/catalog.c
View File

@ -773,8 +773,6 @@ int32_t ctgGetTsma(SCatalog* pCtg, SRequestConnInfo* pConn, const SName* pTsmaNa
}
CTG_ERR_JRET(code);
ASSERT(tsmaRsp.pTsmas && tsmaRsp.pTsmas->size == 1);
*pTsma = taosArrayGetP(tsmaRsp.pTsmas, 0);
taosArrayDestroy(tsmaRsp.pTsmas);

52
source/libs/catalog/src/ctgAsync.c
View File

@ -2448,30 +2448,33 @@ int32_t ctgHandleGetTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf*
STableTSMAInfoRsp* pOut = pMsgCtx->out;
pRes->code = 0;
if (pOut->pTsmas->size > 0) {
ASSERT(pOut->pTsmas->size == 1);
pRes->pRes = pOut;
pMsgCtx->out = NULL;
TSWAP(pTask->res, pCtx->pResList);
if (1 != pOut->pTsmas->size) {
ctgError("invalid tsma num:%d", (int32_t)pOut->pTsmas->size);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
STableTSMAInfo* pTsma = taosArrayGetP(pOut->pTsmas, 0);
if (NULL == pTsma) {
ctgError("fail to get the 0th STableTSMAInfo, totalNum:%d", (int32_t)taosArrayGetSize(pOut->pTsmas));
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
int32_t exists = false;
CTG_ERR_JRET(ctgTbMetaExistInCache(pCtg, pTsma->targetDbFName, pTsma->targetTb, &exists));
if (!exists) {
TSWAP(pMsgCtx->lastOut, pMsgCtx->out);
CTG_RET(ctgGetTbMetaFromMnodeImpl(pCtg, pConn, pTsma->targetDbFName, pTsma->targetTb, NULL, tReq));
}
pRes->pRes = pOut;
pMsgCtx->out = NULL;
TSWAP(pTask->res, pCtx->pResList);
STableTSMAInfo* pTsma = taosArrayGetP(pOut->pTsmas, 0);
if (NULL == pTsma) {
ctgError("fail to get the 0th STableTSMAInfo, totalNum:%d", (int32_t)taosArrayGetSize(pOut->pTsmas));
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
int32_t exists = false;
CTG_ERR_JRET(ctgTbMetaExistInCache(pCtg, pTsma->targetDbFName, pTsma->targetTb, &exists));
if (!exists) {
TSWAP(pMsgCtx->lastOut, pMsgCtx->out);
CTG_RET(ctgGetTbMetaFromMnodeImpl(pCtg, pConn, pTsma->targetDbFName, pTsma->targetTb, NULL, tReq));
}
break;
}
default:
ASSERT(0);
ctgError("invalid reqType:%d while getting tsma rsp", reqType);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
_return:
@ -2635,14 +2638,14 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf
break;
}
default:
ASSERT(0);
ctgError("invalid fetchType:%d while getting tb tsma rsp", pFetch->fetchType);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
break;
}
case TDMT_VND_TABLE_META: {
// handle source tb meta
ASSERT(pFetch->fetchType == FETCH_TSMA_SOURCE_TB_META);
STableMetaOutput* pOut = (STableMetaOutput*)pMsgCtx->out;
pFetch->fetchType = FETCH_TB_TSMA;
pFetch->tsmaSourceTbName = *pTbName;
@ -2663,7 +2666,8 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf
break;
}
default:
ASSERT(0);
ctgError("invalid reqType:%d while getting tsma rsp", reqType);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
_return:
@ -3628,7 +3632,8 @@ int32_t ctgLaunchGetTbTSMATask(SCtgTask* pTask) {
break;
}
default:
ASSERT(0);
ctgError("invalid fetchType:%d in getting tb tsma task", pFetch->fetchType);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
break;
}
}
@ -3644,14 +3649,12 @@ int32_t ctgLaunchGetTSMATask(SCtgTask* pTask) {
SCtgJob* pJob = pTask->pJob;
// currently, only support fetching one tsma
ASSERT(pCtx->pNames->size == 1);
STablesReq* pReq = taosArrayGet(pCtx->pNames, 0);
if (NULL == pReq) {
ctgError("fail to get the 0th STablesReq, totalNum:%d", (int32_t)taosArrayGetSize(pCtx->pNames));
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
ASSERT(pReq->pTables->size == 1);
SName* pTsmaName = taosArrayGet(pReq->pTables, 0);
if (NULL == pReq) {
ctgError("fail to get the 0th SName, totalNum:%d", (int32_t)taosArrayGetSize(pReq->pTables));
@ -3686,7 +3689,6 @@ int32_t ctgLaunchGetTSMATask(SCtgTask* pTask) {
}
STableTSMAInfoRsp* pRsp = (STableTSMAInfoRsp*)pRes->pRes;
ASSERT(pRsp->pTsmas->size == 1);
const STSMACache* pTsma = taosArrayGetP(pRsp->pTsmas, 0);
if (NULL == pTsma) {

7
source/libs/catalog/src/ctgRemote.c
View File

@ -42,7 +42,8 @@ int32_t ctgHandleBatchRsp(SCtgJob* pJob, SCtgTaskCallbackParam* cbParam, SDataBu
msgNum = taosArrayGetSize(batchRsp.pRsps);
}
if (ASSERTS(taskNum == msgNum || 0 == msgNum, "taskNum %d mis-match msgNum %d", taskNum, msgNum)) {
if (taskNum != msgNum && 0 != msgNum) {
ctgError("taskNum %d mis-match msgNum %d", taskNum, msgNum);
msgNum = 0;
}
@ -77,7 +78,9 @@ int32_t ctgHandleBatchRsp(SCtgJob* pJob, SCtgTaskCallbackParam* cbParam, SDataBu
if (msgNum > 0) {
pRsp = taosArrayGet(batchRsp.pRsps, i);
if (ASSERTS(pRsp->msgIdx == *msgIdx, "rsp msgIdx %d mis-match msgIdx %d", pRsp->msgIdx, *msgIdx)) {
if (pRsp->msgIdx != *msgIdx) {
ctgError("rsp msgIdx %d mis-match msgIdx %d", pRsp->msgIdx, *msgIdx);
pRsp = &rsp;
pRsp->msgIdx = *msgIdx;
pRsp->reqType = -1;

2
source/libs/catalog/src/ctgUtil.c
View File

@ -2631,7 +2631,7 @@ bool hasOutOfDateTSMACache(SArray* pTsmas) {
for (int32_t i = 0; i < pTsmas->size; ++i) {
STSMACache* pTsmaInfo = taosArrayGetP(pTsmas, i);
if (NULL == pTsmaInfo) {
ASSERT(0);
continue;
}
if (isCtgTSMACacheOutOfDate(pTsmaInfo)) {
return true;

8
source/libs/executor/inc/executil.h
View File

@ -24,10 +24,10 @@
#include "tpagedbuf.h"
#include "tsimplehash.h"
#define T_LONG_JMP(_obj, _c) \
do { \
ASSERT(1); \
longjmp((_obj), (_c)); \
#define T_LONG_JMP(_obj, _c) \
do { \
qError("error happens at %s, line:%d, code:%s", __func__, __LINE__, tstrerror((_c))); \
longjmp((_obj), (_c)); \
} while (0)
#define SET_RES_WINDOW_KEY(_k, _ori, _len, _uid) \

6
source/libs/executor/src/cachescanoperator.c
View File

@ -298,7 +298,11 @@ int32_t doScanCacheNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) {
int32_t resultRows = pBufRes->info.rows;
// the results may be null, if last values are all null
ASSERT(resultRows == 0 || resultRows == taosArrayGetSize(pInfo->pUidList));
if (resultRows != 0 && resultRows != taosArrayGetSize(pInfo->pUidList)) {
pTaskInfo->code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(pTaskInfo->code));
T_LONG_JMP(pTaskInfo->env, pTaskInfo->code);
}
pInfo->indexOfBufferedRes = 0;
}

13
source/libs/executor/src/countwindowoperator.c
View File

@ -66,14 +66,17 @@ static void clearWinStateBuff(SCountWindowResult* pBuff) { pBuff->winRows = 0; }
static SCountWindowResult* getCountWinStateInfo(SCountWindowSupp* pCountSup) {
SCountWindowResult* pBuffInfo = taosArrayGet(pCountSup->pWinStates, pCountSup->stateIndex);
if (!pBuffInfo) {
terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno));
return NULL;
}
int32_t size = taosArrayGetSize(pCountSup->pWinStates);
// coverity scan
ASSERTS(size > 0, "WinStates is empty");
if (size > 0) {
pCountSup->stateIndex = (pCountSup->stateIndex + 1) % size;
int32_t size = taosArrayGetSize(pCountSup->pWinStates);
if (size == 0) {
terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno));
return NULL;
}
pCountSup->stateIndex = (pCountSup->stateIndex + 1) % size;
return pBuffInfo;
}

10
source/libs/executor/src/dataDeleter.c
View File

@ -87,7 +87,10 @@ static int32_t toDataCacheEntry(SDataDeleterHandle* pHandle, const SInputData* p
if (pRes->affectedRows) {
pRes->skey = *(int64_t*)pColSKey->pData;
pRes->ekey = *(int64_t*)pColEKey->pData;
ASSERT(pRes->skey <= pRes->ekey);
if (pRes->skey > pRes->ekey) {
qError("data delter skey:%" PRId64 " is bigger than ekey:%" PRId64, pRes->skey, pRes->ekey);
QRY_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
} else {
pRes->skey = pHandle->pDeleter->deleteTimeRange.skey;
pRes->ekey = pHandle->pDeleter->deleteTimeRange.ekey;
@ -205,7 +208,10 @@ static void getDataLength(SDataSinkHandle* pHandle, int64_t* pLen, int64_t* pRaw
static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) {
SDataDeleterHandle* pDeleter = (SDataDeleterHandle*)pHandle;
if (NULL == pDeleter->nextOutput.pData) {
ASSERT(pDeleter->queryEnd);
if (!pDeleter->queryEnd) {
qError("empty res while query not end in data deleter");
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
pOutput->useconds = pDeleter->useconds;
pOutput->precision = pDeleter->pSchema->precision;
pOutput->bufStatus = DS_BUF_EMPTY;

10
source/libs/executor/src/dataDispatcher.c
View File

@ -242,7 +242,11 @@ static void getDataLength(SDataSinkHandle* pHandle, int64_t* pLen, int64_t* pRow
static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) {
SDataDispatchHandle* pDispatcher = (SDataDispatchHandle*)pHandle;
if (NULL == pDispatcher->nextOutput.pData) {
ASSERT(pDispatcher->queryEnd);
if (!pDispatcher->queryEnd) {
qError("empty res while query not end in data dispatcher");
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
pOutput->useconds = pDispatcher->useconds;
pOutput->precision = pDispatcher->pSchema->precision;
pOutput->bufStatus = DS_BUF_EMPTY;
@ -318,6 +322,7 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pD
dispatcher->sink.fGetCacheSize = getCacheSize;
dispatcher->pManager = pManager;
pManager = NULL;
dispatcher->pSchema = pDataSink->pInputDataBlockDesc;
dispatcher->status = DS_BUF_EMPTY;
dispatcher->queryEnd = false;
@ -336,6 +341,9 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pD
return TSDB_CODE_SUCCESS;
_return:
taosMemoryFree(pManager);
if (dispatcher) {
dsDestroyDataSinker(dispatcher);
}

6
source/libs/executor/src/dataInserter.c
View File

@ -234,7 +234,11 @@ int32_t buildSubmitReqFromBlock(SDataInserterHandle* pInserter, SSubmitReq2** pp
case TSDB_DATA_TYPE_NCHAR:
case TSDB_DATA_TYPE_VARBINARY:
case TSDB_DATA_TYPE_VARCHAR: { // TSDB_DATA_TYPE_BINARY
ASSERT(pColInfoData->info.type == pCol->type);
if (pColInfoData->info.type != pCol->type) {
qError("column:%d type:%d in block dismatch with schema col:%d type:%d", colIdx, pColInfoData->info.type, k, pCol->type);
terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
goto _end;
}
if (colDataIsNull_s(pColInfoData, j)) {
SColVal cv = COL_VAL_NULL(pCol->colId, pCol->type);
if (NULL == taosArrayPush(pVals, &cv)) {

6
source/libs/executor/src/dynqueryctrloperator.c
View File

@ -320,7 +320,7 @@ static int32_t buildMergeJoinOperatorParam(SOperatorParam** ppRes, bool initPara
return code;
}
(*ppRes)->pChildren = taosArrayInit(2, POINTER_BYTES);
if (NULL == *ppRes) {
if (NULL == (*ppRes)->pChildren) {
code = terrno;
freeOperatorParam(pChild0, OP_GET_PARAM);
freeOperatorParam(pChild1, OP_GET_PARAM);
@ -823,12 +823,14 @@ static void postProcessStbJoinTableHash(SOperatorInfo* pOperator) {
pStbJoin->execInfo.leftCacheNum = tSimpleHashGetSize(pStbJoin->ctx.prev.leftCache);
qDebug("more than 1 ref build table num %" PRId64, (int64_t)tSimpleHashGetSize(pStbJoin->ctx.prev.leftCache));
/*
// debug only
iter = 0;
uint32_t* num = NULL;
while (NULL != (num = tSimpleHashIterate(pStbJoin->ctx.prev.leftCache, num, &iter))) {
ASSERT(*num > 1);
A S S E R T(*num > 1);
}
*/
}
static void buildStbJoinTableList(SOperatorInfo* pOperator) {

5
source/libs/executor/src/exchangeoperator.c
View File

@ -225,7 +225,10 @@ static SSDataBlock* doLoadRemoteDataImpl(SOperatorInfo* pOperator) {
} else {
concurrentlyLoadRemoteDataImpl(pOperator, pExchangeInfo, pTaskInfo);
}
if (TSDB_CODE_SUCCESS != pOperator->pTaskInfo->code) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
T_LONG_JMP(pTaskInfo->env, pOperator->pTaskInfo->code);
}
if (taosArrayGetSize(pExchangeInfo->pResultBlockList) == 0) {
return NULL;
} else {

70
source/libs/executor/src/executor.c
View File

@ -31,7 +31,7 @@ int32_t exchangeObjRefPool = -1;
static void cleanupRefPool() {
int32_t ref = atomic_val_compare_exchange_32(&exchangeObjRefPool, exchangeObjRefPool, 0);
(void)taosCloseRef(ref);
taosCloseRef(ref);
}
static void initRefPool() {
@ -158,7 +158,8 @@ static int32_t doSetStreamBlock(SOperatorInfo* pOperator, void* input, size_t nu
SStreamScanInfo* pInfo = pOperator->info;
qDebug("s-task:%s in this batch, %d blocks need to be processed", id, (int32_t)numOfBlocks);
ASSERT(pInfo->validBlockIndex == 0 && taosArrayGetSize(pInfo->pBlockLists) == 0);
QUERY_CHECK_CONDITION((pInfo->validBlockIndex == 0 && taosArrayGetSize(pInfo->pBlockLists) == 0), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
if (type == STREAM_INPUT__MERGED_SUBMIT) {
for (int32_t i = 0; i < numOfBlocks; i++) {
@ -189,7 +190,8 @@ static int32_t doSetStreamBlock(SOperatorInfo* pOperator, void* input, size_t nu
pInfo->blockType = STREAM_INPUT__CHECKPOINT;
} else {
ASSERT(0);
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
return TSDB_CODE_SUCCESS;
@ -543,7 +545,9 @@ int32_t qGetQueryTableSchemaVersion(qTaskInfo_t tinfo, char* dbName, char* table
int32_t* tversion, int32_t idx, bool* tbGet) {
*tbGet = false;
ASSERT(tinfo != NULL && dbName != NULL && tableName != NULL);
if (tinfo == NULL || dbName == NULL || tableName == NULL) {
return TSDB_CODE_INVALID_PARA;
}
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
if (taosArrayGetSize(pTaskInfo->schemaInfos) <= idx) {
@ -663,7 +667,7 @@ int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bo
if (isTaskKilled(pTaskInfo)) {
atomic_store_64(&pTaskInfo->owner, 0);
qDebug("%s already killed, abort", GET_TASKID(pTaskInfo));
return TSDB_CODE_SUCCESS;
return pTaskInfo->code;
}
// error occurs, record the error code and return to client
@ -722,7 +726,8 @@ int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bo
blockIndex += 1;
current += p->info.rows;
ASSERT(p->info.rows > 0 || p->info.type == STREAM_CHECKPOINT);
QUERY_CHECK_CONDITION((p->info.rows > 0 || p->info.type == STREAM_CHECKPOINT), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
void* tmp = taosArrayPush(pResList, &p);
QUERY_CHECK_NULL(tmp, code, lino, _end, terrno);
@ -785,7 +790,7 @@ int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pRes, uint64_t* useconds) {
qDebug("%s already killed, abort", GET_TASKID(pTaskInfo));
taosRUnLockLatch(&pTaskInfo->lock);
return TSDB_CODE_SUCCESS;
return pTaskInfo->code;
}
if (pTaskInfo->owner != 0) {
@ -987,15 +992,17 @@ void qExtractStreamScanner(qTaskInfo_t tinfo, void** scanner) {
*scanner = pOperator->info;
break;
} else {
ASSERT(pOperator->numOfDownstream == 1);
pOperator = pOperator->pDownstream[0];
}
}
}
int32_t qStreamSourceScanParamForHistoryScanStep1(qTaskInfo_t tinfo, SVersionRange* pVerRange, STimeWindow* pWindow) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
ASSERT(pTaskInfo->execModel == OPTR_EXEC_MODEL_STREAM);
QUERY_CHECK_CONDITION((pTaskInfo->execModel == OPTR_EXEC_MODEL_STREAM), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
SStreamTaskInfo* pStreamInfo = &pTaskInfo->streamInfo;
@ -1007,12 +1014,19 @@ int32_t qStreamSourceScanParamForHistoryScanStep1(qTaskInfo_t tinfo, SVersionRan
", window:%" PRId64 " - %" PRId64,
GET_TASKID(pTaskInfo), pStreamInfo->fillHistoryVer.minVer, pStreamInfo->fillHistoryVer.maxVer, pWindow->skey,
pWindow->ekey);
return 0;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
int32_t qStreamSourceScanParamForHistoryScanStep2(qTaskInfo_t tinfo, SVersionRange* pVerRange, STimeWindow* pWindow) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
ASSERT(pTaskInfo->execModel == OPTR_EXEC_MODEL_STREAM);
QUERY_CHECK_CONDITION((pTaskInfo->execModel == OPTR_EXEC_MODEL_STREAM), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
SStreamTaskInfo* pStreamInfo = &pTaskInfo->streamInfo;
@ -1024,14 +1038,26 @@ int32_t qStreamSourceScanParamForHistoryScanStep2(qTaskInfo_t tinfo, SVersionRan
"-%" PRId64,
GET_TASKID(pTaskInfo), pStreamInfo->fillHistoryVer.minVer, pStreamInfo->fillHistoryVer.maxVer, pWindow->skey,
pWindow->ekey);
return 0;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
int32_t qStreamRecoverFinish(qTaskInfo_t tinfo) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
ASSERT(pTaskInfo->execModel == OPTR_EXEC_MODEL_STREAM);
QUERY_CHECK_CONDITION((pTaskInfo->execModel == OPTR_EXEC_MODEL_STREAM), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
pTaskInfo->streamInfo.recoverStep = STREAM_RECOVER_STEP__NONE;
return 0;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
@ -1046,9 +1072,6 @@ int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
SStreamIntervalOperatorInfo* pInfo = pOperator->info;
STimeWindowAggSupp* pSup = &pInfo->twAggSup;
ASSERT(pSup->calTrigger == STREAM_TRIGGER_AT_ONCE || pSup->calTrigger == STREAM_TRIGGER_WINDOW_CLOSE);
ASSERT(pSup->calTriggerSaved == 0 && pSup->deleteMarkSaved == 0);
qInfo("save stream param for interval: %d, %" PRId64, pSup->calTrigger, pSup->deleteMark);
pSup->calTriggerSaved = pSup->calTrigger;
@ -1063,9 +1086,6 @@ int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
SStreamSessionAggOperatorInfo* pInfo = pOperator->info;
STimeWindowAggSupp* pSup = &pInfo->twAggSup;
ASSERT(pSup->calTrigger == STREAM_TRIGGER_AT_ONCE || pSup->calTrigger == STREAM_TRIGGER_WINDOW_CLOSE);
ASSERT(pSup->calTriggerSaved == 0 && pSup->deleteMarkSaved == 0);
qInfo("save stream param for session: %d, %" PRId64, pSup->calTrigger, pSup->deleteMark);
pSup->calTriggerSaved = pSup->calTrigger;
@ -1078,9 +1098,6 @@ int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
SStreamStateAggOperatorInfo* pInfo = pOperator->info;
STimeWindowAggSupp* pSup = &pInfo->twAggSup;
ASSERT(pSup->calTrigger == STREAM_TRIGGER_AT_ONCE || pSup->calTrigger == STREAM_TRIGGER_WINDOW_CLOSE);
ASSERT(pSup->calTriggerSaved == 0 && pSup->deleteMarkSaved == 0);
qInfo("save stream param for state: %d, %" PRId64, pSup->calTrigger, pSup->deleteMark);
pSup->calTriggerSaved = pSup->calTrigger;
@ -1093,9 +1110,6 @@ int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
SStreamEventAggOperatorInfo* pInfo = pOperator->info;
STimeWindowAggSupp* pSup = &pInfo->twAggSup;
ASSERT(pSup->calTrigger == STREAM_TRIGGER_AT_ONCE || pSup->calTrigger == STREAM_TRIGGER_WINDOW_CLOSE);
ASSERT(pSup->calTriggerSaved == 0 && pSup->deleteMarkSaved == 0);
qInfo("save stream param for state: %d, %" PRId64, pSup->calTrigger, pSup->deleteMark);
pSup->calTriggerSaved = pSup->calTrigger;
@ -1108,9 +1122,6 @@ int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
SStreamCountAggOperatorInfo* pInfo = pOperator->info;
STimeWindowAggSupp* pSup = &pInfo->twAggSup;
ASSERT(pSup->calTrigger == STREAM_TRIGGER_AT_ONCE || pSup->calTrigger == STREAM_TRIGGER_WINDOW_CLOSE);
ASSERT(pSup->calTriggerSaved == 0 && pSup->deleteMarkSaved == 0);
qInfo("save stream param for state: %d, %" PRId64, pSup->calTrigger, pSup->deleteMark);
pSup->calTriggerSaved = pSup->calTrigger;
@ -1126,7 +1137,6 @@ int32_t qSetStreamOperatorOptionForScanHistory(qTaskInfo_t tinfo) {
if (pOperator->numOfDownstream != 1 || pOperator->pDownstream[0] == NULL) {
if (pOperator->numOfDownstream > 1) {
qError("unexpected stream, multiple downstream");
ASSERT(0);
return -1;
}
return 0;

23
source/libs/executor/src/executorInt.c
View File

@ -163,7 +163,11 @@ SResultRow* doSetResultOutBufByKey(SDiskbasedBuf* pResultBuf, SResultRowInfo* pR
return NULL;
}
ASSERT(pResult->pageId == p1->pageId && pResult->offset == p1->offset);
if (pResult->pageId != p1->pageId || pResult->offset != p1->offset) {
terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
pTaskInfo->code = terrno;
return NULL;
}
}
} else {
// In case of group by column query, the required SResultRow object must be existInCurrentResusltRowInfo in the
@ -176,7 +180,11 @@ SResultRow* doSetResultOutBufByKey(SDiskbasedBuf* pResultBuf, SResultRowInfo* pR
return NULL;
}
ASSERT(pResult->pageId == p1->pageId && pResult->offset == p1->offset);
if (pResult->pageId != p1->pageId || pResult->offset != p1->offset) {
terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
pTaskInfo->code = terrno;
return NULL;
}
}
}
@ -324,6 +332,7 @@ _end:
static int32_t doSetInputDataBlock(SExprSupp* pExprSup, SSDataBlock* pBlock, int32_t order, int32_t scanFlag,
bool createDummyCol) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SqlFunctionCtx* pCtx = pExprSup->pCtx;
for (int32_t i = 0; i < pExprSup->numOfExprs; ++i) {
@ -363,7 +372,7 @@ static int32_t doSetInputDataBlock(SExprSupp* pExprSup, SSDataBlock* pBlock, int
if (hasPk && (j == pkParamIdx)) {
pInput->pPrimaryKey = pInput->pData[j];
}
ASSERT(pInput->pData[j] != NULL);
QUERY_CHECK_CONDITION((pInput->pData[j] != NULL), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
} else if (pFuncParam->type == FUNC_PARAM_TYPE_VALUE) {
// todo avoid case: top(k, 12), 12 is the value parameter.
// sum(11), 11 is also the value parameter.
@ -374,14 +383,16 @@ static int32_t doSetInputDataBlock(SExprSupp* pExprSup, SSDataBlock* pBlock, int
pInput->blankFill = pBlock->info.blankFill;
code = doCreateConstantValColumnInfo(pInput, pFuncParam, j, pBlock->info.rows);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
QUERY_CHECK_CODE(code, lino, _end);
}
}
}
}
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}

52
source/libs/executor/src/filloperator.c
View File

@ -55,7 +55,6 @@ typedef struct SFillOperatorInfo {
SExprSupp noFillExprSupp;
} SFillOperatorInfo;
static void revisedFillStartKey(SFillOperatorInfo* pInfo, SSDataBlock* pBlock, int32_t order);
static void destroyFillOperatorInfo(void* param);
static void doApplyScalarCalculation(SOperatorInfo* pOperator, SSDataBlock* pBlock, int32_t order, int32_t scanFlag);
static int32_t fillResetPrevForNewGroup(SFillInfo* pFillInfo);
@ -168,56 +167,6 @@ _end:
return code;
}
// todo refactor: decide the start key according to the query time range.
static void revisedFillStartKey(SFillOperatorInfo* pInfo, SSDataBlock* pBlock, int32_t order) {
if (order == TSDB_ORDER_ASC) {
int64_t skey = pBlock->info.window.skey;
if (skey < pInfo->pFillInfo->start) { // the start key may be smaller than the
ASSERT(taosFillNotStarted(pInfo->pFillInfo));
taosFillUpdateStartTimestampInfo(pInfo->pFillInfo, skey);
} else if (pInfo->pFillInfo->start < skey) {
int64_t t = skey;
SInterval* pInterval = &pInfo->pFillInfo->interval;
while (1) {
int64_t prev = taosTimeAdd(t, -pInterval->sliding, pInterval->slidingUnit, pInterval->precision);
if (prev <= pInfo->pFillInfo->start) {
t = prev;
break;
}
t = prev;
}
// todo time window chosen problem: t or prev value?
taosFillUpdateStartTimestampInfo(pInfo->pFillInfo, t);
}
} else {
int64_t ekey = pBlock->info.window.ekey;
if (ekey > pInfo->pFillInfo->start) {
ASSERT(taosFillNotStarted(pInfo->pFillInfo));
taosFillUpdateStartTimestampInfo(pInfo->pFillInfo, ekey);
} else if (ekey < pInfo->pFillInfo->start) {
int64_t t = ekey;
SInterval* pInterval = &pInfo->pFillInfo->interval;
int64_t prev = t;
while (1) {
int64_t next = taosTimeAdd(t, pInterval->sliding, pInterval->slidingUnit, pInterval->precision);
if (next >= pInfo->pFillInfo->start) {
prev = t;
t = next;
break;
}
prev = t;
t = next;
}
// todo time window chosen problem: t or next value?
if (t > pInfo->pFillInfo->start) t = prev;
taosFillUpdateStartTimestampInfo(pInfo->pFillInfo, t);
}
}
}
static SSDataBlock* doFillImpl(SOperatorInfo* pOperator) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
@ -602,7 +551,6 @@ static void reviseFillStartAndEndKey(SFillOperatorInfo* pInfo, int32_t order) {
}
pInfo->win.ekey = ekey;
} else {
assert(order == TSDB_ORDER_DESC);
skey = taosTimeTruncate(pInfo->win.skey, &pInfo->pFillInfo->interval);
next = skey;
while (next < pInfo->win.skey) {

6
source/libs/executor/src/hashjoin.c
View File

@ -60,7 +60,7 @@ int32_t hInnerJoinDo(struct SOperatorInfo* pOperator) {
/*
size_t keySize = 0;
int32_t* pKey = tSimpleHashGetKey(pGroup, &keySize);
ASSERT(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen));
A S S E R T(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen));
int64_t rows = getSingleKeyRowsNum(pGroup->rows);
pJoin->execInfo.expectRows += rows;
qTrace("hash_key:%d, rows:%" PRId64, *pKey, rows);
@ -145,7 +145,7 @@ int32_t hLeftJoinHandleSeqProbeRows(struct SOperatorInfo* pOperator, SHJoinOpera
/*
size_t keySize = 0;
int32_t* pKey = tSimpleHashGetKey(pGroup, &keySize);
ASSERT(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen));
A S S E R T(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen));
int64_t rows = getSingleKeyRowsNum(pGroup->rows);
pJoin->execInfo.expectRows += rows;
qTrace("hash_key:%d, rows:%" PRId64, *pKey, rows);
@ -248,7 +248,7 @@ int32_t hLeftJoinHandleProbeRows(struct SOperatorInfo* pOperator, SHJoinOperator
/*
size_t keySize = 0;
int32_t* pKey = tSimpleHashGetKey(pGroup, &keySize);
ASSERT(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen));
A S S E R T(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen));
int64_t rows = getSingleKeyRowsNum(pGroup->rows);
pJoin->execInfo.expectRows += rows;
qTrace("hash_key:%d, rows:%" PRId64, *pKey, rows);

3
source/libs/executor/src/hashjoinoperator.c
View File

@ -38,8 +38,6 @@ bool hJoinBlkReachThreshold(SHJoinOperatorInfo* pInfo, int64_t blkRows) {
}
int32_t hJoinHandleMidRemains(SHJoinOperatorInfo* pJoin, SHJoinCtx* pCtx) {
ASSERT(0 < pJoin->midBlk->info.rows);
TSWAP(pJoin->midBlk, pJoin->finBlk);
pCtx->midRemains = false;
@ -1153,7 +1151,6 @@ int32_t hJoinInitResBlocks(SHJoinOperatorInfo* pJoin, SHashJoinPhysiNode* pJoinN
if (NULL == pJoin->finBlk) {
QRY_ERR_RET(terrno);
}
ASSERT(pJoinNode->node.pOutputDataBlockDesc->totalRowSize > 0);
int32_t code = blockDataEnsureCapacity(pJoin->finBlk, hJoinGetFinBlkCapacity(pJoin, pJoinNode));
if (TSDB_CODE_SUCCESS != code) {

46
source/libs/executor/src/mergejoin.c
View File

@ -321,7 +321,7 @@ static int32_t mOuterJoinMergeSeqCart(SMJoinMergeCtx* pCtx) {
int32_t buildEndIdx = buildGrp->endIdx;
buildGrp->endIdx = buildGrp->readIdx + rowsLeft - 1;
ASSERT(buildGrp->endIdx >= buildGrp->readIdx);
//A S S E R T(buildGrp->endIdx >= buildGrp->readIdx);
MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->midBlk, true, probeGrp, buildGrp));
buildGrp->readIdx += rowsLeft;
buildGrp->endIdx = buildEndIdx;
@ -1383,9 +1383,9 @@ static int32_t mSemiJoinHashGrpCartFilter(SMJoinMergeCtx* pCtx, SMJoinGrpRows* p
continue;
}
ASSERT(1 == pCtx->midBlk->info.rows);
//A S S E R T(1 == pCtx->midBlk->info.rows);
MJ_ERR_RET(mJoinCopyMergeMidBlk(pCtx, &pCtx->midBlk, &pCtx->finBlk));
ASSERT(false == pCtx->midRemains);
//A S S E R T(false == pCtx->midRemains);
break;
} while (true);
@ -1449,10 +1449,10 @@ static int32_t mSemiJoinHashFullCart(SMJoinMergeCtx* pCtx) {
}
build->pHashCurGrp = *(SArray**)pGrp;
ASSERT(1 == taosArrayGetSize(build->pHashCurGrp));
//A S S E R T(1 == taosArrayGetSize(build->pHashCurGrp));
build->grpRowIdx = 0;
MJ_ERR_RET(mJoinHashGrpCart(pCtx->finBlk, probeGrp, true, probe, build, NULL));
ASSERT(build->grpRowIdx < 0);
//A S S E R T(build->grpRowIdx < 0);
}
pCtx->grpRemains = probeGrp->readIdx <= probeGrp->endIdx;
@ -1497,7 +1497,7 @@ static int32_t mSemiJoinMergeSeqCart(SMJoinMergeCtx* pCtx) {
int32_t buildEndIdx = buildGrp->endIdx;
buildGrp->endIdx = buildGrp->readIdx + rowsLeft - 1;
ASSERT(buildGrp->endIdx >= buildGrp->readIdx);
//A S S E R T(buildGrp->endIdx >= buildGrp->readIdx);
MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->midBlk, true, probeGrp, buildGrp));
buildGrp->readIdx += rowsLeft;
buildGrp->endIdx = buildEndIdx;
@ -1513,9 +1513,9 @@ static int32_t mSemiJoinMergeSeqCart(SMJoinMergeCtx* pCtx) {
continue;
}
} else {
ASSERT(1 == pCtx->midBlk->info.rows);
//A S S E R T(1 == pCtx->midBlk->info.rows);
MJ_ERR_RET(mJoinCopyMergeMidBlk(pCtx, &pCtx->midBlk, &pCtx->finBlk));
ASSERT(false == pCtx->midRemains);
//A S S E R T(false == pCtx->midRemains);
if (build->grpIdx == buildGrpNum) {
continue;
@ -1555,8 +1555,8 @@ static int32_t mSemiJoinMergeFullCart(SMJoinMergeCtx* pCtx) {
int32_t probeRows = GRP_REMAIN_ROWS(probeGrp);
int32_t probeEndIdx = probeGrp->endIdx;
ASSERT(1 == taosArrayGetSize(build->eqGrps));
ASSERT(buildGrp->beginIdx == buildGrp->endIdx);
//A S S E R T(1 == taosArrayGetSize(build->eqGrps));
//A S S E R T(buildGrp->beginIdx == buildGrp->endIdx);
if (probeRows <= rowsLeft) {
MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->finBlk, true, probeGrp, buildGrp));
@ -1826,7 +1826,7 @@ static int32_t mAntiJoinMergeSeqCart(SMJoinMergeCtx* pCtx) {
int32_t buildEndIdx = buildGrp->endIdx;
buildGrp->endIdx = buildGrp->readIdx + rowsLeft - 1;
ASSERT(buildGrp->endIdx >= buildGrp->readIdx);
//A S S E R T(buildGrp->endIdx >= buildGrp->readIdx);
MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->midBlk, true, probeGrp, buildGrp));
buildGrp->readIdx += rowsLeft;
buildGrp->endIdx = buildEndIdx;
@ -2381,7 +2381,7 @@ int32_t mAsofForwardTrimCacheBlk(SMJoinWindowCtx* pCtx) {
if (pGrp->blk == pCtx->cache.outBlk && pCtx->pJoin->build->blkRowIdx > 0) {
MJ_ERR_RET(blockDataTrimFirstRows(pGrp->blk, pCtx->pJoin->build->blkRowIdx));
pCtx->pJoin->build->blkRowIdx = 0;
ASSERT(pCtx->pJoin->build->blk == pGrp->blk);
//A S S E R T(pCtx->pJoin->build->blk == pGrp->blk);
MJOIN_SAVE_TB_BLK(&pCtx->cache, pCtx->pJoin->build);
}
@ -2419,16 +2419,16 @@ int32_t mAsofForwardChkFillGrpCache(SMJoinWindowCtx* pCtx) {
MJ_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
ASSERT(pGrp->blk == pCache->outBlk);
//A S S E R T(pGrp->blk == pCache->outBlk);
//pGrp->endIdx = pGrp->blk->info.rows - pGrp->beginIdx;
}
//ASSERT((pGrp->endIdx - pGrp->beginIdx + 1) == pCtx->cache.rowNum);
//A S S E R T((pGrp->endIdx - pGrp->beginIdx + 1) == pCtx->cache.rowNum);
}
ASSERT(taosArrayGetSize(pCache->grps) == 1);
ASSERT(pGrp->blk->info.rows - pGrp->beginIdx == pCtx->cache.rowNum);
//A S S E R T(taosArrayGetSize(pCache->grps) == 1);
//A S S E R T(pGrp->blk->info.rows - pGrp->beginIdx == pCtx->cache.rowNum);
}
do {
@ -2473,7 +2473,7 @@ int32_t mAsofForwardUpdateBuildGrpEndIdx(SMJoinWindowCtx* pCtx) {
return TSDB_CODE_SUCCESS;
}
ASSERT(pCtx->jLimit > (pGrp->blk->info.rows - pGrp->beginIdx));
//A S S E R T(pCtx->jLimit > (pGrp->blk->info.rows - pGrp->beginIdx));
pGrp->endIdx = pGrp->blk->info.rows - 1;
int64_t remainRows = pCtx->jLimit - (pGrp->endIdx - pGrp->beginIdx + 1);
@ -2550,8 +2550,8 @@ int32_t mAsofForwardSkipAllEqRows(SMJoinWindowCtx* pCtx, int64_t timestamp) {
MJOIN_RESTORE_TB_BLK(cache, pTable);
} while (!MJOIN_BUILD_TB_ROWS_DONE(pTable));
ASSERT(pCtx->cache.rowNum == 0);
ASSERT(taosArrayGetSize(pCtx->cache.grps) == 0);
//A S S E R T(pCtx->cache.rowNum == 0);
//A S S E R T(taosArrayGetSize(pCtx->cache.grps) == 0);
if (pTable->dsFetchDone) {
return TSDB_CODE_SUCCESS;
@ -2648,7 +2648,7 @@ static int32_t mAsofForwardRetrieve(SOperatorInfo* pOperator, SMJoinOperatorInfo
if ((probeGot || MJOIN_DS_NEED_INIT(pOperator, pJoin->build)) && pCtx->cache.rowNum < pCtx->jLimit) {
pJoin->build->newBlk = false;
MJOIN_SAVE_TB_BLK(&pCtx->cache, pCtx->pJoin->build);
ASSERT(taosArrayGetSize(pCtx->cache.grps) <= 1);
//A S S E R T(taosArrayGetSize(pCtx->cache.grps) <= 1);
buildGot = mJoinRetrieveBlk(pJoin, &pJoin->build->blkRowIdx, &pJoin->build->blk, pJoin->build);
}
@ -3375,7 +3375,7 @@ int32_t mWinJoinMoveAscWinEnd(SMJoinWindowCtx* pCtx) {
continue;
}
ASSERT(pGrp->endIdx > startIdx);
//A S S E R T(pGrp->endIdx > startIdx);
pGrp->endIdx--;
break;
@ -3419,7 +3419,7 @@ int32_t mWinJoinMoveDescWinEnd(SMJoinWindowCtx* pCtx) {
continue;
}
ASSERT(pGrp->endIdx > startIdx);
//A S S E R T(pGrp->endIdx > startIdx);
pGrp->endIdx--;
break;
@ -3676,7 +3676,7 @@ int32_t mJoinInitMergeCtx(SMJoinOperatorInfo* pJoin, SSortMergeJoinPhysiNode* pJ
MJ_ERR_RET(terrno);
}
ASSERT(pJoinNode->node.pOutputDataBlockDesc->totalRowSize > 0);
//A S S E R T(pJoinNode->node.pOutputDataBlockDesc->totalRowSize > 0);
MJ_ERR_RET(blockDataEnsureCapacity(pCtx->finBlk, mJoinGetFinBlkCapacity(pJoin, pJoinNode)));

17
source/libs/executor/src/mergejoinoperator.c
View File

@ -490,8 +490,6 @@ int32_t mJoinCopyMergeMidBlk(SMJoinMergeCtx* pCtx, SSDataBlock** ppMid, SSDataBl
int32_t mJoinHandleMidRemains(SMJoinMergeCtx* pCtx) {
ASSERT(0 < pCtx->midBlk->info.rows);
TSWAP(pCtx->midBlk, pCtx->finBlk);
pCtx->midRemains = false;
@ -567,7 +565,6 @@ int32_t mJoinMergeGrpCart(SMJoinOperatorInfo* pJoin, SSDataBlock* pRes, bool app
int32_t currRows = append ? pRes->info.rows : 0;
int32_t firstRows = GRP_REMAIN_ROWS(pFirst);
int32_t secondRows = GRP_REMAIN_ROWS(pSecond);
ASSERT(secondRows > 0);
for (int32_t c = 0; c < probe->finNum; ++c) {
SMJoinColMap* pFirstCol = probe->finCols + c;
@ -581,9 +578,15 @@ int32_t mJoinMergeGrpCart(SMJoinOperatorInfo* pJoin, SSDataBlock* pRes, bool app
if (colDataIsNull_s(pInCol, pFirst->readIdx + r)) {
colDataSetNItemsNull(pOutCol, currRows + r * secondRows, secondRows);
} else {
ASSERT(pRes->info.capacity >= (pRes->info.rows + firstRows * secondRows));
if (pRes->info.capacity < (pRes->info.rows + firstRows * secondRows)) {
qError("capacity:%d not enough, rows:%" PRId64 ", firstRows:%d, secondRows:%d", pRes->info.capacity, pRes->info.rows, firstRows, secondRows);
MJ_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
uint32_t startOffset = (IS_VAR_DATA_TYPE(pOutCol->info.type)) ? pOutCol->varmeta.length : ((currRows + r * secondRows) * pOutCol->info.bytes);
ASSERT((startOffset + 1 * pOutCol->info.bytes) <= pRes->info.capacity * pOutCol->info.bytes);
if ((startOffset + 1 * pOutCol->info.bytes) > pRes->info.capacity * pOutCol->info.bytes) {
qError("col buff not enough, startOffset:%d, bytes:%d, capacity:%d", startOffset, pOutCol->info.bytes, pRes->info.capacity);
MJ_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
MJ_ERR_RET(colDataSetNItems(pOutCol, currRows + r * secondRows, colDataGetData(pInCol, pFirst->readIdx + r), secondRows, true));
}
}
@ -1097,7 +1100,6 @@ SSDataBlock* mJoinGrpRetrieveImpl(SMJoinOperatorInfo* pJoin, SMJoinTableCtx* pTa
}
SMJoinTableCtx* pProbe = pJoin->probe;
ASSERT(pProbe->lastInGid);
while (true) {
if (pTable->remainInBlk) {
@ -1746,6 +1748,9 @@ void destroyGrpArray(void* ppArray) {
}
void destroyMergeJoinTableCtx(SMJoinTableCtx* pTable) {
if (NULL == pTable) {
return;
}
mJoinDestroyCreatedBlks(pTable->createdBlks);
taosArrayDestroy(pTable->createdBlks);
tSimpleHashCleanup(pTable->pGrpHash);

10
source/libs/executor/src/projectoperator.c
View File

@ -681,6 +681,7 @@ int32_t doApplyIndefinitFunction(SOperatorInfo* pOperator, SSDataBlock** pResBlo
}
int32_t initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size) {
int32_t code = TSDB_CODE_SUCCESS;
for (int32_t j = 0; j < size; ++j) {
struct SResultRowEntryInfo* pResInfo = GET_RES_INFO(&pCtx[j]);
if (isRowEntryInitialized(pResInfo) || fmIsPseudoColumnFunc(pCtx[j].functionId) || pCtx[j].functionId == -1 ||
@ -688,7 +689,10 @@ int32_t initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size) {
continue;
}
(void)pCtx[j].fpSet.init(&pCtx[j], pCtx[j].resultInfo);
code = pCtx[j].fpSet.init(&pCtx[j], pCtx[j].resultInfo);
if (code) {
return code;
}
}
return 0;
@ -1033,8 +1037,8 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc
// do nothing
} else if (fmIsIndefiniteRowsFunc(pfCtx->functionId)) {
SResultRowEntryInfo* pResInfo = GET_RES_INFO(pfCtx);
(void) pfCtx->fpSet.init(pfCtx, pResInfo);
code = pfCtx->fpSet.init(pfCtx, pResInfo);
TSDB_CHECK_CODE(code, lino, _exit);
pfCtx->pOutput = taosArrayGet(pResult->pDataBlock, outputSlotId);
if (pfCtx->pOutput == NULL) {
code = terrno;

67
source/libs/executor/src/scanoperator.c
View File

@ -464,15 +464,10 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableSca
SSDataBlock* p = NULL;
code = pAPI->tsdReader.tsdReaderRetrieveDataBlock(pTableScanInfo->dataReader, &p, NULL);
if (p == NULL || code != TSDB_CODE_SUCCESS) {
if (p == NULL || code != TSDB_CODE_SUCCESS || p != pBlock) {
return code;
}
if(p != pBlock) {
qError("[loadDataBlock] p != pBlock");
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
code = doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, pBlock->info.rows);
if (code) {
return code;
@ -1241,14 +1236,11 @@ static SSDataBlock* groupSeqTableScan(SOperatorInfo* pOperator) {
taosRUnLockLatch(&pTaskInfo->lock);
QUERY_CHECK_CODE(code, lino, _end);
ASSERT(pInfo->base.dataReader == NULL);
QUERY_CHECK_CONDITION((pInfo->base.dataReader == NULL), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
code = pAPI->tsdReader.tsdReaderOpen(pInfo->base.readHandle.vnode, &pInfo->base.cond, pList, num, pInfo->pResBlock,
(void**)&pInfo->base.dataReader, GET_TASKID(pTaskInfo), &pInfo->pIgnoreTables);
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
T_LONG_JMP(pTaskInfo->env, code);
}
QUERY_CHECK_CODE(code, lino, _end);
if (pInfo->filesetDelimited) {
pAPI->tsdReader.tsdSetFilesetDelimited(pInfo->base.dataReader);
@ -1590,7 +1582,6 @@ static bool isCountWindow(SStreamScanInfo* pInfo) {
static void setGroupId(SStreamScanInfo* pInfo, SSDataBlock* pBlock, int32_t groupColIndex, int32_t rowIndex) {
SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, groupColIndex);
uint64_t* groupCol = (uint64_t*)pColInfo->pData;
ASSERT(rowIndex < pBlock->info.rows);
pInfo->groupId = groupCol[rowIndex];
}
@ -1659,9 +1650,8 @@ _end:
bool comparePrimaryKey(SColumnInfoData* pCol, int32_t rowId, void* pVal) {
// coverity scan
ASSERTS(pVal != NULL, "pVal should not be NULL");
if (!pVal) {
qError("failed to compare primary key, since primary key is null");
if (!pVal || !pCol) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_INVALID_PARA));
return false;
}
void* pData = colDataGetData(pCol, rowId);
@ -1725,6 +1715,8 @@ static uint64_t getGroupIdByData(SStreamScanInfo* pInfo, uint64_t uid, TSKEY ts,
}
static void prepareRangeScan(SStreamScanInfo* pInfo, SSDataBlock* pBlock, int32_t* pRowIndex, bool* pRes) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (pBlock->info.rows == 0) {
if (pRes) {
(*pRes) = false;
@ -1738,7 +1730,6 @@ static void prepareRangeScan(SStreamScanInfo* pInfo, SSDataBlock* pBlock, int32_
goto _end;
}
ASSERT(taosArrayGetSize(pBlock->pDataBlock) >= 3);
SColumnInfoData* pStartTsCol = taosArrayGet(pBlock->pDataBlock, START_TS_COLUMN_INDEX);
TSKEY* startData = (TSKEY*)pStartTsCol->pData;
SColumnInfoData* pEndTsCol = taosArrayGet(pBlock->pDataBlock, END_TS_COLUMN_INDEX);
@ -1770,27 +1761,25 @@ static void prepareRangeScan(SStreamScanInfo* pInfo, SSDataBlock* pBlock, int32_
win.skey = TMIN(win.skey, startData[*pRowIndex]);
continue;
}
ASSERT(!(win.skey > startData[*pRowIndex] && win.ekey < endData[*pRowIndex]) ||
!(isInTimeWindow(&win, startData[*pRowIndex], 0) || isInTimeWindow(&win, endData[*pRowIndex], 0)));
break;
}
STableScanInfo* pTScanInfo = pInfo->pTableScanOp->info;
// coverity scan
ASSERTS(pInfo->pUpdateInfo != NULL, "Failed to set data version, since pInfo->pUpdateInfo is NULL");
if (pInfo->pUpdateInfo) {
qDebug("prepare range scan start:%" PRId64 ",end:%" PRId64 ",maxVer:%" PRIu64, win.skey, win.ekey,
pInfo->pUpdateInfo->maxDataVersion);
resetTableScanInfo(pInfo->pTableScanOp->info, &win, pInfo->pUpdateInfo->maxDataVersion);
}
QUERY_CHECK_NULL(pInfo->pUpdateInfo, code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
qDebug("prepare range scan start:%" PRId64 ",end:%" PRId64 ",maxVer:%" PRIu64, win.skey, win.ekey,
pInfo->pUpdateInfo->maxDataVersion);
resetTableScanInfo(pInfo->pTableScanOp->info, &win, pInfo->pUpdateInfo->maxDataVersion);
pInfo->pTableScanOp->status = OP_OPENED;
if (pRes) {
(*pRes) = true;
}
_end:
qTrace("%s success", __func__);
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
}
static STimeWindow getSlidingWindow(TSKEY* startTsCol, TSKEY* endTsCol, uint64_t* gpIdCol, SInterval* pInterval,
@ -2198,7 +2187,6 @@ static int32_t generateIntervalScanRange(SStreamScanInfo* pInfo, SSDataBlock* pS
}
uint64_t* srcUidData = (uint64_t*)pSrcUidCol->pData;
ASSERT(pSrcStartTsCol->info.type == TSDB_DATA_TYPE_TIMESTAMP);
TSKEY* srcStartTsCol = (TSKEY*)pSrcStartTsCol->pData;
TSKEY* srcEndTsCol = (TSKEY*)pSrcEndTsCol->pData;
int64_t ver = pSrcBlock->info.version - 1;
@ -2296,7 +2284,6 @@ static int32_t generatePartitionDelResBlock(SStreamScanInfo* pInfo, SSDataBlock*
SColumnInfoData* pSrcGpCol = taosArrayGet(pSrcBlock->pDataBlock, GROUPID_COLUMN_INDEX);
uint64_t* srcGp = (uint64_t*)pSrcGpCol->pData;
ASSERT(pSrcStartTsCol->info.type == TSDB_DATA_TYPE_TIMESTAMP);
TSKEY* srcStartTsCol = (TSKEY*)pSrcStartTsCol->pData;
TSKEY* srcEndTsCol = (TSKEY*)pSrcEndTsCol->pData;
int64_t ver = pSrcBlock->info.version - 1;
@ -2357,7 +2344,6 @@ static int32_t generateDeleteResultBlockImpl(SStreamScanInfo* pInfo, SSDataBlock
pSrcPkCol = taosArrayGet(pSrcBlock->pDataBlock, PRIMARY_KEY_COLUMN_INDEX);
}
ASSERT(pSrcStartTsCol->info.type == TSDB_DATA_TYPE_TIMESTAMP);
TSKEY* srcStartTsCol = (TSKEY*)pSrcStartTsCol->pData;
TSKEY* srcEndTsCol = (TSKEY*)pSrcEndTsCol->pData;
int64_t ver = pSrcBlock->info.version - 1;
@ -2477,7 +2463,6 @@ static int32_t checkUpdateData(SStreamScanInfo* pInfo, bool invertible, SSDataBl
QUERY_CHECK_CODE(code, lino, _end);
}
SColumnInfoData* pColDataInfo = taosArrayGet(pBlock->pDataBlock, pInfo->primaryTsIndex);
ASSERT(pColDataInfo->info.type == TSDB_DATA_TYPE_TIMESTAMP);
TSKEY* tsCol = (TSKEY*)pColDataInfo->pData;
SColumnInfoData* pPkColDataInfo = NULL;
if (hasPrimaryKeyCol(pInfo)) {
@ -2554,7 +2539,7 @@ static int32_t doBlockDataWindowFilter(SSDataBlock* pBlock, int32_t tsIndex, STi
if (pWindow->skey != INT64_MIN) {
qDebug("%s filter for additional history window, skey:%" PRId64, id, pWindow->skey);
ASSERT(pCol->pData != NULL);
QUERY_CHECK_NULL(pCol->pData, code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
for (int32_t i = 0; i < pBlock->info.rows; ++i) {
int64_t* ts = (int64_t*)colDataGetData(pCol, i);
p[i] = (*ts >= pWindow->skey);
@ -2603,9 +2588,11 @@ static int32_t doBlockDataPrimaryKeyFilter(SSDataBlock* pBlock, STqOffsetVal* of
SColumnInfoData* pColPk = taosArrayGet(pBlock->pDataBlock, 1);
qDebug("doBlockDataWindowFilter primary key, ts:%" PRId64 " %" PRId64, offset->ts, offset->primaryKey.val);
ASSERT(pColPk->info.type == offset->primaryKey.type);
QUERY_CHECK_CONDITION((pColPk->info.type == offset->primaryKey.type), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
__compar_fn_t func = getComparFunc(pColPk->info.type, 0);
QUERY_CHECK_NULL(func, code, lino, _end, terrno);
for (int32_t i = 0; i < pBlock->info.rows; ++i) {
int64_t* ts = (int64_t*)colDataGetData(pColTs, i);
void* data = colDataGetData(pColPk, i);
@ -3945,7 +3932,9 @@ void streamScanReloadState(SOperatorInfo* pOperator) {
pInfo->stateStore.windowSBfDelete(pInfo->pUpdateInfo, 1);
code = pInfo->stateStore.windowSBfAdd(pInfo->pUpdateInfo, 1);
QUERY_CHECK_CODE(code, lino, _end);
ASSERT(pInfo->pUpdateInfo->minTS > pUpInfo->minTS);
QUERY_CHECK_CONDITION((pInfo->pUpdateInfo->minTS > pUpInfo->minTS), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
pInfo->pUpdateInfo->maxDataVersion = TMAX(pInfo->pUpdateInfo->maxDataVersion, pUpInfo->maxDataVersion);
SHashObj* curMap = pInfo->pUpdateInfo->pMap;
void* pIte = taosHashIterate(curMap, NULL);
@ -4105,12 +4094,11 @@ int32_t createStreamScanOperatorInfo(SReadHandle* pHandle, STableScanPhysiNode*
}
if (pHandle->initTqReader) {
ASSERT(pHandle->tqReader == NULL);
pInfo->tqReader = pAPI->tqReaderFn.tqReaderOpen(pHandle->vnode);
ASSERT(pInfo->tqReader);
QUERY_CHECK_NULL(pInfo->tqReader, code, lino, _error, terrno);
} else {
ASSERT(pHandle->tqReader);
pInfo->tqReader = pHandle->tqReader;
QUERY_CHECK_NULL(pInfo->tqReader, code, lino, _error, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
pInfo->pUpdateInfo = NULL;
@ -5887,7 +5875,10 @@ void destroyTableMergeScanOperatorInfo(void* param) {
}
int32_t getTableMergeScanExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain, uint32_t* len) {
ASSERT(pOptr != NULL);
if (pOptr == NULL) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_INVALID_PARA));
return TSDB_CODE_INVALID_PARA;
}
// TODO: merge these two info into one struct
STableMergeScanExecInfo* execInfo = taosMemoryCalloc(1, sizeof(STableMergeScanExecInfo));
if (!execInfo) {
@ -6200,7 +6191,7 @@ int32_t fillTableCountScanDataBlock(STableCountScanSupp* pSupp, char* dbName, ch
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (pSupp->dbNameSlotId != -1) {
ASSERT(strlen(dbName));
QUERY_CHECK_CONDITION((strlen(dbName) > 0), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
SColumnInfoData* colInfoData = taosArrayGet(pRes->pDataBlock, pSupp->dbNameSlotId);
QUERY_CHECK_NULL(colInfoData, code, lino, _end, terrno);

7
source/libs/executor/src/sortoperator.c
View File

@ -735,7 +735,12 @@ int32_t doGroupSort(SOperatorInfo* pOperator, SSDataBlock** pResBlock) {
}
// beginSortGroup would fetch all child blocks of pInfo->currGroupId;
ASSERT(pInfo->childOpStatus != CHILD_OP_SAME_GROUP);
if (pInfo->childOpStatus == CHILD_OP_SAME_GROUP) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
pOperator->pTaskInfo->code = code;
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
T_LONG_JMP(pOperator->pTaskInfo->env, code);
}
code = getGroupSortedBlockData(pInfo->pCurrSortHandle, pInfo->binfo.pRes, pOperator->resultInfo.capacity,
pInfo->matchInfo.pList, pInfo, &pBlock);
if (pBlock != NULL && (code == 0)) {

8
source/libs/executor/src/streamcountwindowoperator.c
View File

@ -88,7 +88,7 @@ int32_t setCountOutputBuf(SStreamAggSupporter* pAggSup, TSKEY ts, uint64_t group
winCode = TSDB_CODE_FAILED;
} else if (pBuffInfo->winBuffOp == MOVE_NEXT_WINDOW) {
ASSERT(pBuffInfo->pCur);
QUERY_CHECK_NULL(pBuffInfo->pCur, code, lino, _end, terrno);
pAggSup->stateStore.streamStateCurNext(pAggSup->pState, pBuffInfo->pCur);
winCode = pAggSup->stateStore.streamStateSessionGetKVByCur(pBuffInfo->pCur, &pCurWin->winInfo.sessionWin,
(void**)&pCurWin->winInfo.pStatePos, &size);
@ -345,7 +345,6 @@ static void doStreamCountAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBl
if (slidingRows + winRows > pAggSup->windowSliding) {
buffInfo.winBuffOp = CREATE_NEW_WINDOW;
winRows = pAggSup->windowSliding - slidingRows;
ASSERT(i >= 0);
}
} else {
buffInfo.winBuffOp = MOVE_NEXT_WINDOW;
@ -690,7 +689,10 @@ static int32_t doStreamCountAggNext(SOperatorInfo* pOperator, SSDataBlock** ppRe
QUERY_CHECK_CODE(code, lino, _end);
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_NORMAL && pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {

7
source/libs/executor/src/streameventwindowoperator.c
View File

@ -378,7 +378,6 @@ static void doStreamEventAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBl
rows, i, pAggSup->pResultRows, pSeUpdated, pStDeleted, &rebuild, &winRows);
QUERY_CHECK_CODE(code, lino, _end);
ASSERT(winRows >= 1);
if (rebuild) {
uint64_t uid = 0;
code = appendDataToSpecialBlock(pAggSup->pScanBlock, &curWin.winInfo.sessionWin.win.skey,
@ -660,7 +659,10 @@ static int32_t doStreamEventAggNext(SOperatorInfo* pOperator, SSDataBlock** ppRe
QUERY_CHECK_CODE(code, lino, _end);
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_NORMAL && pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {
@ -779,7 +781,6 @@ void streamEventReloadState(SOperatorInfo* pOperator) {
int32_t num = (size - sizeof(TSKEY)) / sizeof(SSessionKey);
qDebug("===stream=== event window operator reload state. get result count:%d", num);
SSessionKey* pSeKeyBuf = (SSessionKey*)pBuf;
ASSERT(size == num * sizeof(SSessionKey) + sizeof(TSKEY));
TSKEY ts = *(TSKEY*)((char*)pBuf + size - sizeof(TSKEY));
pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, ts);

14
source/libs/executor/src/streamfilloperator.c
View File

@ -331,7 +331,7 @@ void setDeleteFillValueInfo(TSKEY start, TSKEY end, SStreamFillSupporter* pFillS
pFillInfo->pLinearInfo->winIndex = 0;
} break;
default:
ASSERT(0);
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR));
break;
}
}
@ -419,7 +419,6 @@ void setFillValueInfo(SSDataBlock* pBlock, TSKEY ts, int32_t rowId, SStreamFillS
pFillSup->next.pRowVal = pFillSup->cur.pRowVal;
pFillInfo->preRowKey = INT64_MIN;
} else {
ASSERT(hasNextWindow(pFillSup));
setFillKeyInfo(ts, nextWKey, &pFillSup->interval, pFillInfo);
pFillInfo->pos = FILL_POS_START;
}
@ -447,7 +446,6 @@ void setFillValueInfo(SSDataBlock* pBlock, TSKEY ts, int32_t rowId, SStreamFillS
pFillInfo->pResRow = &pFillSup->prev;
pFillInfo->pLinearInfo->hasNext = false;
} else {
ASSERT(hasNextWindow(pFillSup));
setFillKeyInfo(ts, nextWKey, &pFillSup->interval, pFillInfo);
pFillInfo->pos = FILL_POS_START;
pFillInfo->pLinearInfo->nextEnd = INT64_MIN;
@ -458,10 +456,9 @@ void setFillValueInfo(SSDataBlock* pBlock, TSKEY ts, int32_t rowId, SStreamFillS
}
} break;
default:
ASSERT(0);
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR));
break;
}
ASSERT(pFillInfo->pos != FILL_POS_INVALID);
}
static int32_t checkResult(SStreamFillSupporter* pFillSup, TSKEY ts, uint64_t groupId, bool* pRes) {
@ -628,7 +625,9 @@ static void keepResultInDiscBuf(SOperatorInfo* pOperator, uint64_t groupId, SRes
SWinKey key = {.groupId = groupId, .ts = pRow->key};
int32_t code = pAPI->stateStore.streamStateFillPut(pOperator->pTaskInfo->streamInfo.pState, &key, pRow->pRowVal, len);
qDebug("===stream===fill operator save key ts:%" PRId64 " group id:%" PRIu64 " code:%d", key.ts, key.groupId, code);
ASSERT(code == TSDB_CODE_SUCCESS);
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
}
}
static void doStreamFillRange(SStreamFillInfo* pFillInfo, SStreamFillSupporter* pFillSup, SSDataBlock* pRes) {
@ -795,7 +794,6 @@ static int32_t buildDeleteRange(SOperatorInfo* pOp, TSKEY start, TSKEY end, uint
if (winCode != TSDB_CODE_SUCCESS) {
colDataSetNULL(pTableCol, pBlock->info.rows);
} else {
ASSERT(tbname);
char parTbName[VARSTR_HEADER_SIZE + TSDB_TABLE_NAME_LEN];
STR_WITH_MAXSIZE_TO_VARSTR(parTbName, tbname, sizeof(parTbName));
code = colDataSetVal(pTableCol, pBlock->info.rows, (const char*)parTbName, false);
@ -1125,7 +1123,7 @@ static int32_t doStreamFillNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) {
return code;
} break;
default:
ASSERTS(false, "invalid SSDataBlock type");
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
}

60
source/libs/executor/src/streamtimewindowoperator.c
View File

@ -213,7 +213,6 @@ static void removeDeleteResults(SSHashObj* pUpdatedMap, SArray* pDelWins) {
}
bool isOverdue(TSKEY ekey, STimeWindowAggSupp* pTwSup) {
ASSERTS(pTwSup->maxTs == INT64_MIN || pTwSup->maxTs > 0, "maxts should greater than 0");
return pTwSup->maxTs != INT64_MIN && ekey < pTwSup->maxTs - pTwSup->waterMark;
}
@ -415,7 +414,7 @@ static void doBuildDeleteResult(SStreamIntervalOperatorInfo* pInfo, SArray* pWin
code = appendDataToSpecialBlock(pBlock, &pWin->ts, &pWin->ts, &uid, &pWin->groupId, NULL);
QUERY_CHECK_CODE(code, lino, _end);
} else {
ASSERT(tbname);
QUERY_CHECK_CONDITION((tbname), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
char parTbName[VARSTR_HEADER_SIZE + TSDB_TABLE_NAME_LEN];
STR_WITH_MAXSIZE_TO_VARSTR(parTbName, tbname, sizeof(parTbName));
code = appendDataToSpecialBlock(pBlock, &pWin->ts, &pWin->ts, &uid, &pWin->groupId, parTbName);
@ -439,6 +438,11 @@ void destroyFlusedPos(void* pRes) {
}
}
void destroyFlusedppPos(void* ppRes) {
void *pRes = *(void **)ppRes;
destroyFlusedPos(pRes);
}
void clearGroupResInfo(SGroupResInfo* pGroupResInfo) {
if (pGroupResInfo->freeItem) {
int32_t size = taosArrayGetSize(pGroupResInfo->pRows);
@ -916,7 +920,6 @@ void buildDataBlockFromGroupRes(SOperatorInfo* pOperator, void* pState, SSDataBl
}
if (pBlock->info.rows + pRow->numOfRows > pBlock->info.capacity) {
ASSERT(pBlock->info.rows > 0);
break;
}
pGroupResInfo->index += 1;
@ -1366,7 +1369,7 @@ void doStreamIntervalDecodeOpState(void* buf, int32_t len, SOperatorInfo* pOpera
int32_t winCode = TSDB_CODE_SUCCESS;
code = pInfo->stateStore.streamStateAddIfNotExist(pInfo->pState, &key, (void**)&pPos, &resSize, &winCode);
QUERY_CHECK_CODE(code, lino, _end);
ASSERT(winCode == TSDB_CODE_SUCCESS);
QUERY_CHECK_CONDITION((winCode == TSDB_CODE_SUCCESS), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
code = tSimpleHashPut(pInfo->aggSup.pResultRowHashTable, &key, sizeof(SWinKey), &pPos, POINTER_BYTES);
QUERY_CHECK_CODE(code, lino, _end);
@ -1694,7 +1697,10 @@ static int32_t doStreamFinalIntervalAggNext(SOperatorInfo* pOperator, SSDataBloc
} else if (IS_FINAL_INTERVAL_OP(pOperator) && pBlock->info.type == STREAM_MID_RETRIEVE) {
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {
@ -1883,7 +1889,6 @@ int32_t createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiN
.deleteMarkSaved = 0,
.calTriggerSaved = 0,
};
ASSERTS(pInfo->twAggSup.calTrigger != STREAM_TRIGGER_MAX_DELAY, "trigger type should not be max delay");
pInfo->primaryTsIndex = ((SColumnNode*)pIntervalPhyNode->window.pTspk)->slotId;
size_t keyBufSize = sizeof(int64_t) + sizeof(int64_t) + POINTER_BYTES;
initResultSizeInfo(&pOperator->resultInfo, 4096);
@ -1920,6 +1925,7 @@ int32_t createStreamFinalIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiN
code = initAggSup(&pOperator->exprSupp, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str,
pInfo->pState, &pTaskInfo->storageAPI.functionStore);
QUERY_CHECK_CODE(code, lino, _error);
tSimpleHashSetFreeFp(pInfo->aggSup.pResultRowHashTable, destroyFlusedppPos);
code = initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window);
QUERY_CHECK_CODE(code, lino, _error);
@ -2090,7 +2096,6 @@ int32_t initBasicInfoEx(SOptrBasicInfo* pBasicInfo, SExprSupp* pSup, SExprInfo*
pSup->pCtx[i].saveHandle.pBuf = NULL;
}
ASSERT(numOfCols > 0);
return TSDB_CODE_SUCCESS;
}
@ -2391,7 +2396,6 @@ _end:
static int32_t initSessionOutputBuf(SResultWindowInfo* pWinInfo, SResultRow** pResult, SqlFunctionCtx* pCtx,
int32_t numOfOutput, int32_t* rowEntryInfoOffset) {
ASSERT(pWinInfo->sessionWin.win.skey <= pWinInfo->sessionWin.win.ekey);
*pResult = (SResultRow*)pWinInfo->pStatePos->pRowBuff;
// set time window for current result
(*pResult)->win = pWinInfo->sessionWin.win;
@ -3022,7 +3026,6 @@ int32_t buildSessionResultDataBlock(SOperatorInfo* pOperator, void* pState, SSDa
}
if (pBlock->info.rows + pRow->numOfRows > pBlock->info.capacity) {
ASSERT(pBlock->info.rows > 0);
break;
}
@ -3262,7 +3265,7 @@ int32_t doStreamSessionDecodeOpState(void* buf, int32_t len, SOperatorInfo* pOpe
code = pAggSup->stateStore.streamStateSessionAddIfNotExist(
pAggSup->pState, &winfo.sessionWin, pAggSup->gap, (void**)&winfo.pStatePos, &pAggSup->resultRowSize, &winCode);
QUERY_CHECK_CODE(code, lino, _end);
ASSERT(winCode == TSDB_CODE_SUCCESS);
QUERY_CHECK_CONDITION((winCode == TSDB_CODE_SUCCESS), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
buf = decodeSResultWindowInfo(buf, &winfo, pInfo->streamAggSup.resultRowSize);
code =
@ -3276,7 +3279,6 @@ int32_t doStreamSessionDecodeOpState(void* buf, int32_t len, SOperatorInfo* pOpe
// 3.pChildren
int32_t size = 0;
buf = taosDecodeFixedI32(buf, &size);
ASSERT(size <= taosArrayGetSize(pInfo->pChildren));
for (int32_t i = 0; i < size; i++) {
SOperatorInfo* pChOp = taosArrayGetP(pInfo->pChildren, i);
code = doStreamSessionDecodeOpState(buf, 0, pChOp, false, &buf);
@ -3447,7 +3449,10 @@ static int32_t doStreamSessionAggNext(SOperatorInfo* pOperator, SSDataBlock** pp
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_NORMAL && pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {
@ -3621,7 +3626,6 @@ void streamSessionSemiReloadState(SOperatorInfo* pOperator) {
int32_t num = (size - sizeof(TSKEY)) / sizeof(SSessionKey);
SSessionKey* pSeKeyBuf = (SSessionKey*)pBuf;
ASSERT(size == num * sizeof(SSessionKey) + sizeof(TSKEY));
for (int32_t i = 0; i < num; i++) {
SResultWindowInfo winInfo = {0};
code = getSessionWindowInfoByKey(pAggSup, pSeKeyBuf + i, &winInfo);
@ -3667,7 +3671,6 @@ void streamSessionReloadState(SOperatorInfo* pOperator) {
int32_t num = (size - sizeof(TSKEY)) / sizeof(SSessionKey);
SSessionKey* pSeKeyBuf = (SSessionKey*)pBuf;
ASSERT(size == num * sizeof(SSessionKey) + sizeof(TSKEY));
TSKEY ts = *(TSKEY*)((char*)pBuf + size - sizeof(TSKEY));
pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, ts);
@ -3990,7 +3993,10 @@ static int32_t doStreamSessionSemiAggNext(SOperatorInfo* pOperator, SSDataBlock*
doStreamSessionSaveCheckpoint(pOperator);
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_NORMAL && pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {
@ -4205,7 +4211,8 @@ int32_t getStateWindowInfoByKey(SStreamAggSupporter* pAggSup, SSessionKey* pKey,
pCurWin->winInfo.sessionWin.win.ekey = pKey->win.ekey;
code = getSessionWindowInfoByKey(pAggSup, pKey, &pCurWin->winInfo);
QUERY_CHECK_CODE(code, lino, _end);
ASSERT(IS_VALID_SESSION_WIN(pCurWin->winInfo));
QUERY_CHECK_CONDITION((IS_VALID_SESSION_WIN(pCurWin->winInfo)), code, lino, _end,
TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
pCurWin->pStateKey =
(SStateKeys*)((char*)pCurWin->winInfo.pStatePos->pRowBuff + (pAggSup->resultRowSize - pAggSup->stateKeySize));
@ -4575,7 +4582,6 @@ int32_t doStreamStateDecodeOpState(void* buf, int32_t len, SOperatorInfo* pOpera
// 3.pChildren
int32_t size = 0;
buf = taosDecodeFixedI32(buf, &size);
ASSERT(size <= taosArrayGetSize(pInfo->pChildren));
for (int32_t i = 0; i < size; i++) {
SOperatorInfo* pChOp = taosArrayGetP(pInfo->pChildren, i);
code = doStreamStateDecodeOpState(buf, 0, pChOp, false, &buf);
@ -4717,7 +4723,10 @@ static int32_t doStreamStateAggNext(SOperatorInfo* pOperator, SSDataBlock** ppRe
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_NORMAL && pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {
@ -4829,7 +4838,6 @@ void streamStateReloadState(SOperatorInfo* pOperator) {
int32_t num = (size - sizeof(TSKEY)) / sizeof(SSessionKey);
qDebug("===stream=== reload state. get result count:%d", num);
SSessionKey* pSeKeyBuf = (SSessionKey*)pBuf;
ASSERT(size == num * sizeof(SSessionKey) + sizeof(TSKEY));
TSKEY ts = *(TSKEY*)((char*)pBuf + size - sizeof(TSKEY));
pInfo->twAggSup.maxTs = TMAX(pInfo->twAggSup.maxTs, ts);
@ -5135,7 +5143,10 @@ static int32_t doStreamIntervalAggNext(SOperatorInfo* pOperator, SSDataBlock** p
continue;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_NORMAL && pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pBlock->info.type == STREAM_NORMAL && pBlock->info.version != 0) {
@ -5254,8 +5265,6 @@ int32_t createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiNode*
.minTs = INT64_MAX,
.deleteMark = getDeleteMark(&pIntervalPhyNode->window, pIntervalPhyNode->interval)};
ASSERTS(pInfo->twAggSup.calTrigger != STREAM_TRIGGER_MAX_DELAY, "trigger type should not be max delay");
pOperator->pTaskInfo = pTaskInfo;
SStorageAPI* pAPI = &pOperator->pTaskInfo->storageAPI;
@ -5283,6 +5292,7 @@ int32_t createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiNode*
code = initAggSup(pSup, &pInfo->aggSup, pExprInfo, numOfCols, keyBufSize, pTaskInfo->id.str, pInfo->pState,
&pTaskInfo->storageAPI.functionStore);
QUERY_CHECK_CODE(code, lino, _error);
tSimpleHashSetFreeFp(pInfo->aggSup.pResultRowHashTable, destroyFlusedppPos);
if (pIntervalPhyNode->window.pExprs != NULL) {
int32_t numOfScalar = 0;
@ -5635,7 +5645,6 @@ static int32_t doStreamMidIntervalAggNext(SOperatorInfo* pOperator, SSDataBlock*
} else {
pInfo->pDelRes->info.type = STREAM_DELETE_RESULT;
}
ASSERT(taosArrayGetSize(pInfo->pUpdated) == 0);
(*ppRes) = pInfo->pDelRes;
return code;
}
@ -5684,7 +5693,10 @@ static int32_t doStreamMidIntervalAggNext(SOperatorInfo* pOperator, SSDataBlock*
pInfo->clearState = true;
break;
} else {
ASSERTS(pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
if (pBlock->info.type != STREAM_INVALID) {
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (pInfo->scalarSupp.pExprInfo != NULL) {

18
source/libs/executor/src/sysscanoperator.c
View File

@ -343,27 +343,45 @@ int optSysDoCompare(__compar_fn_t func, int8_t comparType, void* a, void* b) {
static int optSysFilterFuncImpl__LowerThan(void* a, void* b, int16_t dtype) {
__compar_fn_t func = getComparFunc(dtype, 0);
if (func == NULL) {
return -1;
}
return optSysDoCompare(func, OP_TYPE_LOWER_THAN, a, b);
}
static int optSysFilterFuncImpl__LowerEqual(void* a, void* b, int16_t dtype) {
__compar_fn_t func = getComparFunc(dtype, 0);
if (func == NULL) {
return -1;
}
return optSysDoCompare(func, OP_TYPE_LOWER_EQUAL, a, b);
}
static int optSysFilterFuncImpl__GreaterThan(void* a, void* b, int16_t dtype) {
__compar_fn_t func = getComparFunc(dtype, 0);
if (func == NULL) {
return -1;
}
return optSysDoCompare(func, OP_TYPE_GREATER_THAN, a, b);
}
static int optSysFilterFuncImpl__GreaterEqual(void* a, void* b, int16_t dtype) {
__compar_fn_t func = getComparFunc(dtype, 0);
if (func == NULL) {
return -1;
}
return optSysDoCompare(func, OP_TYPE_GREATER_EQUAL, a, b);
}
static int optSysFilterFuncImpl__Equal(void* a, void* b, int16_t dtype) {
__compar_fn_t func = getComparFunc(dtype, 0);
if (func == NULL) {
return -1;
}
return optSysDoCompare(func, OP_TYPE_EQUAL, a, b);
}
static int optSysFilterFuncImpl__NoEqual(void* a, void* b, int16_t dtype) {
__compar_fn_t func = getComparFunc(dtype, 0);
if (func == NULL) {
return -1;
}
return optSysDoCompare(func, OP_TYPE_NOT_EQUAL, a, b);
}

3
source/libs/executor/src/timesliceoperator.c
View File

@ -114,7 +114,6 @@ static void doKeepLinearInfo(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlo
pLinearInfo->start.key = *(int64_t*)colDataGetData(pTsCol, rowIndex);
char* p = colDataGetData(pColInfoData, rowIndex);
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
ASSERT(varDataTLen(p) <= pColInfoData->info.bytes);
memcpy(pLinearInfo->start.val, p, varDataTLen(p));
} else {
memcpy(pLinearInfo->start.val, p, pLinearInfo->bytes);
@ -127,7 +126,6 @@ static void doKeepLinearInfo(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlo
char* p = colDataGetData(pColInfoData, rowIndex);
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
ASSERT(varDataTLen(p) <= pColInfoData->info.bytes);
memcpy(pLinearInfo->end.val, p, varDataTLen(p));
} else {
memcpy(pLinearInfo->end.val, p, pLinearInfo->bytes);
@ -143,7 +141,6 @@ static void doKeepLinearInfo(STimeSliceOperatorInfo* pSliceInfo, const SSDataBlo
char* p = colDataGetData(pColInfoData, rowIndex);
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
ASSERT(varDataTLen(p) <= pColInfoData->info.bytes);
memcpy(pLinearInfo->end.val, p, varDataTLen(p));
} else {
memcpy(pLinearInfo->end.val, p, pLinearInfo->bytes);

118
source/libs/executor/src/timewindowoperator.c
View File

@ -136,7 +136,6 @@ FORCE_INLINE int32_t getForwardStepsInBlock(int32_t numOfRows, __block_search_fn
// }
}
ASSERT(forwardRows >= 0);
return forwardRows;
}
@ -211,8 +210,6 @@ int32_t binarySearchForKey(char* pValue, int num, TSKEY key, int order) {
int32_t getNumOfRowsInTimeWindow(SDataBlockInfo* pDataBlockInfo, TSKEY* pPrimaryColumn, int32_t startPos, TSKEY ekey,
__block_search_fn_t searchFn, STableQueryInfo* item, int32_t order) {
ASSERT(startPos >= 0 && startPos < pDataBlockInfo->rows);
int32_t num = -1;
int32_t step = GET_FORWARD_DIRECTION_FACTOR(order);
@ -259,8 +256,6 @@ void doTimeWindowInterpolation(SArray* pPrevValues, SArray* pDataBlock, TSKEY pr
SFunctParam* pParam = &pCtx[k].param[0];
SColumnInfoData* pColInfo = taosArrayGet(pDataBlock, pParam->pCol->slotId);
ASSERT(pColInfo->info.type == pParam->pCol->type && curTs != windowKey);
double v1 = 0, v2 = 0, v = 0;
if (prevRowIndex == -1) {
SGroupKeys* p = taosArrayGet(pPrevValues, index);
@ -356,9 +351,11 @@ static bool setTimeWindowInterpolationStartTs(SIntervalAggOperatorInfo* pInfo, i
return true;
}
static bool setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, SExprSupp* pSup, int32_t endRowIndex,
static int32_t setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, SExprSupp* pSup, int32_t endRowIndex,
int32_t nextRowIndex, SArray* pDataBlock, const TSKEY* tsCols,
TSKEY blockEkey, STimeWindow* win) {
TSKEY blockEkey, STimeWindow* win, bool* pRes) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
int32_t order = pInfo->binfo.inputTsOrder;
TSKEY actualEndKey = tsCols[endRowIndex];
@ -367,21 +364,27 @@ static bool setTimeWindowInterpolationEndTs(SIntervalAggOperatorInfo* pInfo, SEx
// not ended in current data block, do not invoke interpolation
if ((key > blockEkey && (order == TSDB_ORDER_ASC)) || (key < blockEkey && (order == TSDB_ORDER_DESC))) {
setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_END_INTERP);
return false;
(*pRes) = false;
return code;
}
// there is actual end point of current time window, no interpolation needs
if (key == actualEndKey) {
setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_END_INTERP);
return true;
(*pRes) = true;
return code;
}
ASSERT(nextRowIndex >= 0);
if (nextRowIndex < 0) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR));
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
TSKEY nextKey = tsCols[nextRowIndex];
doTimeWindowInterpolation(pInfo->pPrevValues, pDataBlock, actualEndKey, endRowIndex, nextKey, nextRowIndex, key,
RESULT_ROW_END_INTERP, pSup);
return true;
(*pRes) = true;
return code;
}
bool inCalSlidingWindow(SInterval* pInterval, STimeWindow* pWin, TSKEY calStart, TSKEY calEnd, EStreamType blockType) {
@ -437,13 +440,7 @@ int32_t getNextQualifiedWindow(SInterval* pInterval, STimeWindow* pNext, SDataBl
* This time window does not cover any data, try next time window,
* this case may happen when the time window is too small
*/
if (primaryKeys == NULL) {
if (ascQuery) {
ASSERT(pDataBlockInfo->window.skey <= pNext->ekey);
} else {
ASSERT(pDataBlockInfo->window.ekey >= pNext->skey);
}
} else {
if (primaryKeys != NULL) {
if (ascQuery && primaryKeys[startPos] > pNext->ekey) {
TSKEY next = primaryKeys[startPos];
if (pInterval->intervalUnit == 'n' || pInterval->intervalUnit == 'y') {
@ -469,7 +466,6 @@ int32_t getNextQualifiedWindow(SInterval* pInterval, STimeWindow* pNext, SDataBl
}
static bool isResultRowInterpolated(SResultRow* pResult, SResultTsInterpType type) {
ASSERT(pResult != NULL && (type == RESULT_ROW_START_INTERP || type == RESULT_ROW_END_INTERP));
if (type == RESULT_ROW_START_INTERP) {
return pResult->startInterp == true;
} else {
@ -485,15 +481,21 @@ static void setResultRowInterpo(SResultRow* pResult, SResultTsInterpType type) {
}
}
static void doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataBlock* pBlock, SResultRow* pResult,
STimeWindow* win, int32_t startPos, int32_t forwardRows, SExprSupp* pSup) {
static int32_t doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataBlock* pBlock, SResultRow* pResult,
STimeWindow* win, int32_t startPos, int32_t forwardRows, SExprSupp* pSup) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (!pInfo->timeWindowInterpo) {
return;
return code;
}
if (pBlock == NULL) {
code = TSDB_CODE_INVALID_PARA;
return code;
}
ASSERT(pBlock != NULL);
if (pBlock->pDataBlock == NULL) {
return;
return code;
}
SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, pInfo->primaryTsIndex);
@ -528,14 +530,22 @@ static void doWindowBorderInterpolation(SIntervalAggOperatorInfo* pInfo, SSDataB
}
TSKEY endKey = (pInfo->binfo.inputTsOrder == TSDB_ORDER_ASC) ? pBlock->info.window.ekey : pBlock->info.window.skey;
bool interp = setTimeWindowInterpolationEndTs(pInfo, pSup, endRowIndex, nextRowIndex, pBlock->pDataBlock, tsCols,
endKey, win);
bool interp = false;
code = setTimeWindowInterpolationEndTs(pInfo, pSup, endRowIndex, nextRowIndex, pBlock->pDataBlock, tsCols,
endKey, win, &interp);
QUERY_CHECK_CODE(code, lino, _end);
if (interp) {
setResultRowInterpo(pResult, RESULT_ROW_END_INTERP);
}
} else {
setNotInterpoWindowKey(pSup->pCtx, pSup->numOfExprs, RESULT_ROW_END_INTERP);
}
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
static void saveDataBlockLastRow(SArray* pPrevKeys, const SSDataBlock* pBlock, SArray* pCols) {
@ -558,7 +568,6 @@ static void saveDataBlockLastRow(SArray* pPrevKeys, const SSDataBlock* pBlock, S
char* val = colDataGetData(pColInfo, i);
if (IS_VAR_DATA_TYPE(pkey->type)) {
memcpy(pkey->pData, val, varDataTLen(val));
ASSERT(varDataTLen(val) <= pkey->bytes);
} else {
memcpy(pkey->pData, val, pkey->bytes);
}
@ -594,11 +603,17 @@ static void doInterpUnclosedTimeWindow(SOperatorInfo* pOperatorInfo, int32_t num
T_LONG_JMP(pTaskInfo->env, terrno);
}
ASSERT(pr->offset == p1->offset && pr->pageId == p1->pageId);
if (!(pr->offset == p1->offset && pr->pageId == p1->pageId)) {
pTaskInfo->code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
T_LONG_JMP(pTaskInfo->env, terrno);
}
if (pr->closed) {
ASSERT(isResultRowInterpolated(pr, RESULT_ROW_START_INTERP) &&
isResultRowInterpolated(pr, RESULT_ROW_END_INTERP));
if (!(isResultRowInterpolated(pr, RESULT_ROW_START_INTERP) &&
isResultRowInterpolated(pr, RESULT_ROW_END_INTERP)) ) {
pTaskInfo->code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
T_LONG_JMP(pTaskInfo->env, terrno);
}
SListNode* pNode = tdListPopHead(pResultRowInfo->openWindow);
taosMemoryFree(pNode);
continue;
@ -611,7 +626,10 @@ static void doInterpUnclosedTimeWindow(SOperatorInfo* pOperatorInfo, int32_t num
T_LONG_JMP(pTaskInfo->env, TSDB_CODE_OUT_OF_MEMORY);
}
ASSERT(!isResultRowInterpolated(pResult, RESULT_ROW_END_INTERP));
if(isResultRowInterpolated(pResult, RESULT_ROW_END_INTERP)) {
pTaskInfo->code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
T_LONG_JMP(pTaskInfo->env, terrno);
}
SGroupKeys* pTsKey = taosArrayGet(pInfo->pPrevValues, 0);
if (!pTsKey) {
@ -764,11 +782,14 @@ static bool hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul
ret = setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pSup->pCtx,
numOfOutput, pSup->rowEntryInfoOffset, &pInfo->aggSup, pTaskInfo);
if (ret != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, TSDB_CODE_OUT_OF_MEMORY);
T_LONG_JMP(pTaskInfo->env, ret);
}
// window start key interpolation
doWindowBorderInterpolation(pInfo, pBlock, pResult, &win, startPos, forwardRows, pSup);
ret = doWindowBorderInterpolation(pInfo, pBlock, pResult, &win, startPos, forwardRows, pSup);
if (ret != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, ret);
}
}
updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &win, 1);
@ -796,7 +817,10 @@ static bool hashIntervalAgg(SOperatorInfo* pOperatorInfo, SResultRowInfo* pResul
forwardRows = getNumOfRowsInTimeWindow(&pBlock->info, tsCols, startPos, ekey, binarySearchForKey, NULL,
pInfo->binfo.inputTsOrder);
// window start(end) key interpolation
doWindowBorderInterpolation(pInfo, pBlock, pResult, &nextWin, startPos, forwardRows, pSup);
code = doWindowBorderInterpolation(pInfo, pBlock, pResult, &nextWin, startPos, forwardRows, pSup);
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, code);
}
// TODO: add to open window? how to close the open windows after input blocks exhausted?
#if 0
if ((ascScan && ekey <= pBlock->info.window.ekey) ||
@ -869,7 +893,10 @@ int64_t* extractTsCol(SSDataBlock* pBlock, const SIntervalAggOperatorInfo* pInfo
}
tsCols = (int64_t*)pColDataInfo->pData;
ASSERT(tsCols[0] != 0);
if(tsCols[0] == 0) {
pTaskInfo->code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
T_LONG_JMP(pTaskInfo->env, terrno);
}
// no data in primary ts
if (tsCols[0] == 0 && tsCols[pBlock->info.rows - 1] == 0) {
@ -1959,7 +1986,10 @@ static void doMergeAlignedIntervalAgg(SOperatorInfo* pOperator) {
} else {
if (pMiaInfo->groupId != pBlock->info.id.groupId) {
// if there are unclosed time window, close it firstly.
ASSERT(pMiaInfo->curTs != INT64_MIN);
if (pMiaInfo->curTs == INT64_MIN) {
pTaskInfo->code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
T_LONG_JMP(pTaskInfo->env, terrno);
}
finalizeResultRows(pIaInfo->aggSup.pResultBuf, &pResultRowInfo->cur, pSup, pRes, pTaskInfo);
resetResultRow(pMiaInfo->pResultRow, pIaInfo->aggSup.resultRowSize - sizeof(SResultRow));
@ -2216,7 +2246,9 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo*
TSKEY ekey = ascScan ? win.ekey : win.skey;
int32_t forwardRows = getNumOfRowsInTimeWindow(&pBlock->info, tsCols, startPos, ekey, binarySearchForKey, NULL,
iaInfo->binfo.inputTsOrder);
ASSERT(forwardRows > 0);
if(forwardRows <= 0) {
T_LONG_JMP(pTaskInfo->env, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
// prev time window not interpolation yet.
if (iaInfo->timeWindowInterpo) {
@ -2227,11 +2259,14 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo*
ret = setTimeWindowOutputBuf(pResultRowInfo, &win, (scanFlag == MAIN_SCAN), &pResult, tableGroupId, pExprSup->pCtx,
numOfOutput, pExprSup->rowEntryInfoOffset, &iaInfo->aggSup, pTaskInfo);
if (ret != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, TSDB_CODE_OUT_OF_MEMORY);
T_LONG_JMP(pTaskInfo->env, ret);
}
// window start key interpolation
doWindowBorderInterpolation(iaInfo, pBlock, pResult, &win, startPos, forwardRows, pExprSup);
ret = doWindowBorderInterpolation(iaInfo, pBlock, pResult, &win, startPos, forwardRows, pExprSup);
if (ret != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, ret);
}
}
updateTimeWindowInfo(&iaInfo->twAggSup.timeWindowData, &win, 1);
@ -2268,7 +2303,10 @@ static void doMergeIntervalAggImpl(SOperatorInfo* pOperatorInfo, SResultRowInfo*
iaInfo->binfo.inputTsOrder);
// window start(end) key interpolation
doWindowBorderInterpolation(iaInfo, pBlock, pResult, &nextWin, startPos, forwardRows, pExprSup);
code = doWindowBorderInterpolation(iaInfo, pBlock, pResult, &nextWin, startPos, forwardRows, pExprSup);
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, code);
}
updateTimeWindowInfo(&iaInfo->twAggSup.timeWindowData, &nextWin, 1);
applyAggFunctionOnPartialTuples(pTaskInfo, pExprSup->pCtx, &iaInfo->twAggSup.timeWindowData, startPos, forwardRows,

2
source/libs/function/inc/thistogram.h
View File

@ -59,7 +59,7 @@ int32_t tHistogramCreate(int32_t numOfEntries, SHistogramInfo** pHisto);
SHistogramInfo* tHistogramCreateFrom(void* pBuf, int32_t numOfBins);
int32_t tHistogramAdd(SHistogramInfo** pHisto, double val);
int64_t tHistogramSum(SHistogramInfo* pHisto, double v);
int32_t tHistogramSum(SHistogramInfo* pHisto, double v, int64_t *res);
int32_t tHistogramUniform(SHistogramInfo* pHisto, double* ratio, int32_t num, double** pVal);
int32_t tHistogramMerge(SHistogramInfo* pHisto1, SHistogramInfo* pHisto2, int32_t numOfEntries,

2
source/libs/function/inc/tpercentile.h
View File

@ -47,7 +47,7 @@ typedef struct tMemBucketSlot {
} tMemBucketSlot;
struct tMemBucket;
typedef int32_t (*__perc_hash_func_t)(struct tMemBucket *pBucket, const void *value);
typedef int32_t (*__perc_hash_func_t)(struct tMemBucket *pBucket, const void *value, int32_t *index);
typedef struct tMemBucket {
int16_t numOfSlots;

4
source/libs/function/src/builtinsimpl.c
View File

@ -4018,7 +4018,9 @@ int32_t saveTupleData(SqlFunctionCtx* pCtx, int32_t rowIndex, const SSDataBlock*
if (NULL == pColInfo) {
return TSDB_CODE_OUT_OF_RANGE;
}
ASSERT(pColInfo->info.type == TSDB_DATA_TYPE_TIMESTAMP);
if (pColInfo->info.type != TSDB_DATA_TYPE_TIMESTAMP) {
return TSDB_CODE_FUNC_FUNTION_PARA_TYPE;
}
key.groupId = pSrcBlock->info.id.groupId;
key.ts = *(int64_t*)colDataGetData(pColInfo, rowIndex);
}

Some files were not shown because too many files have changed in this diff Show More