Merge branch '3.0' into merge/mainto3.0

docs/zh/08-operation/09-backup.md

@@ -6,45 +6,154 @@ toc_max_heading_level: 4
 
 为了防止数据丢失、误删操作，TDengine 提供全面的数据备份、恢复、容错、异地数据实时同步等功能，以保证数据存储的安全。本节简要说明备份和恢复功能。
 
-## 基于 taosdump 进行数据备份恢复
+# 1. 基于 taosdump 进行数据备份恢复
 
-taosdump 是一个开源工具，用于支持从运行中的 TDengine 集群备份数据并将备份的数据恢复到相同或另一个正在运行的 TDengine 集群中。taosdump 可以将数据库作为逻辑数据单元进行备份，也可以对数据库中指定时间段内的数据记录进行备份。在使用taosdump 时，可以指定数据备份的目录路径。如果不指定目录路径，taosdump 将默认将数据备份到当前目录。
+taosdump 是一个开源工具，用于支持从运行中的 TDengine 集群备份数据并将备份的数据恢复到相同或另一个正在运行的 TDengine
+集群中。taosdump 可以将数据库作为逻辑数据单元进行备份，也可以对数据库中指定时间段内的数据记录进行备份。在使用taosdump
+时，可以指定数据备份的目录路径。如果不指定目录路径，taosdump 将默认将数据备份到当前目录。
 
 以下为 taosdump 执行数据备份的使用示例。
+
 ```shell
 taosdump -h localhost -P 6030 -D dbname -o /file/path
 ```
 
-执行上述命令后，taosdump 会连接 localhost:6030 所在的 TDengine 集群，查询数据库 dbname 中的所有数据，并将数据备份到 /f ile/path 下。
+执行上述命令后，taosdump 会连接 localhost:6030 所在的 TDengine 集群，查询数据库 dbname 中的所有数据，并将数据备份到 /f
+ile/path 下。
 
-在使用 taosdump 时，如果指定的存储路径已经包含数据文件，taosdump 会提示用户并立即退出，以避免数据被覆盖。这意味着同一存储路径只能用于一次备份。如果你看到相关提示，请谨慎操作，以免误操作导致数据丢失。
+在使用 taosdump 时，如果指定的存储路径已经包含数据文件，taosdump
+会提示用户并立即退出，以避免数据被覆盖。这意味着同一存储路径只能用于一次备份。如果你看到相关提示，请谨慎操作，以免误操作导致数据丢失。
+
+要将本地指定文件路径中的数据文件恢复到正在运行的 TDengine 集群中，可以通过指定命令行参数和数据文件所在路径来执行 taosdump
+命令。以下为 taosdump 执行数据恢复的示例代码。
 
-要将本地指定文件路径中的数据文件恢复到正在运行的 TDengine 集群中，可以通过指定命令行参数和数据文件所在路径来执行 taosdump 命令。以下为 taosdump 执行数据恢复的示例代码。
 ```shell
 taosdump -i /file/path -h localhost -P 6030
 ```
 
 执行上述命令后，taosdump 会连接 localhost:6030 所在的 TDengine 集群，并将 /file/path 下的数据文件恢复到 TDengine 集群中。
 
-## 基于 TDengine Enterprise 进行数据备份恢复
+# 2. 基于 TDengine Enterprise 进行数据备份恢复
 
-TDengine Enterprise 提供了一个高效的增量备份功能，具体流程如下。
+## 2.1. 概念
 
-第 1 步，通过浏览器访问 taosExplorer 服务，访问地址通常为 TDengine 集群所在 IP 地址的端口 6060，如 http://localhost:6060。
+基于 TDengine 的数据订阅功能，TDengine Enterprise 实现了数据的增量备份和恢复。用户可以通过 taosExplorer 对 TDengine
+集群进行备份和恢复。
 
-第 2 步，在 taosExplorer 服务页面中的“系统管理 - 备份”页面新增一个数据备份任务，在任务配置信息中填写需要备份的数据库名称和备份存储文件路径，完成创建任务
-后即可启动数据备份。 在数据备份配置页面中可以配置三个参数：
-  - 备份周期：必填项，配置每次执行数据备份的时间间隔，可通过下拉框选择每天、每 7 天、每 30 天执行一次数据备份，配置后，会在对应的备份周期的0:00时启动一次数据备份任务；
-  - 数据库：必填项，配置需要备份的数据库名（数据库的 wal_retention_period 参数需大于0）；
-  - 目录：必填项，配置将数据备份到 taosX 所在运行环境中指定的路径下，如 /root/data_backup；
+TDengine Enterprise 的备份和恢复功能包括以下几个概念：
 
-第 3 步，在数据备份任务完成后，在相同页面的已创建任务列表中找到创建的数据备份任务，直接执行一键恢复，就能够将数据恢复到 TDengine 中。
+1. 备份对象：用户可以对一个数据库，或者一个超级表进行备份。
+2. 备份计划：用户可以为某个备份对象创建一个备份计划。备份计划从指定的时间点开始，周期性的执行一次备份任务，并生成一组备份文件。
+3. 备份点：每执行一次备份任务，生成一组备份文件，它们对应一个时间点，称为**备份点**。第一个备份点称为**初始备份点**。
+4. 备份文件：多个备份点，组成备份计划的备份文件。
+5. 恢复任务：用户可以选择备份计划的某个备份点，创建一个恢复任务。恢复任务会从初始备份点开始，逐个应用备份点，恢复到指定的备份点。
 
-与 taosdump 相比，如果对相同的数据在指定存储路径下进行多次备份操作，由于TDengine Enterprise 不仅备份效率高，而且实行的是增量处理，因此每次备份任务都会很快完成。而由于 taosdump 永远是全量备份，因此 TDengine Enterprise 在数据量较大的场景下可以显著减小系统开销，而且更加方便。
+![backup-zh-00.png](./pic/backup-00-concept.png "数据备份和恢复")
 
-**常见错误排查**
+以上面的图为例：
 
-1. 如果任务启动失败并报以下错误：
+1. 用户创建了一个备份计划，从 2024-08-27 00:00:00 开始，每隔 1 天执行一次备份任务。
+2. 在 2024-08-27 00:00:00 执行了第一次备份任务，生成了一个备份点。
+3. 之后，每隔 1 天执行一次备份任务，生成了多个备份点，组成了备份文件。
+4. 用户可以选择某个备份点，创建一个恢复任务，恢复到指定的备份点。
+5. 恢复任务会从初始备份点开始，逐个应用备份点，恢复到指定的备份点。
+
+## 2.2. 备份计划
+
+### 2.1.1. 创建
+
+1. 通过浏览器访问 taosExplorer 服务，访问地址通常为 TDengine 集群所在 IP 地址的端口 6060，如 http://localhost:6060。
+2. 在 taosExplorer 服务页面中，进入“系统管理 - 备份”页面，点击“创建备份计划”按钮。
+
+![backup-zh-01.png](./pic/backup-01-create.png "创建备份计划")
+
+3. 在弹出的“创建备份计划”表单中，填写备份计划的相关信息。
+
+![backup-zh-02.png](./pic/backup-02-form.png "填写备份计划信息")
+
+需要填写的信息包括：
+
+* 数据库：需要备份的数据库名称。一个备份计划只能备份一个数据库/超级表。
+* 超级表：需要备份的超级表名称。如果不填写，则备份整个数据库。
+* 下次执行时间：首次执行备份任务的日期时间。
+* 备份周期：备份点之间的时间间隔。注意：备份周期必须大于数据库的 WAL_RETENTION_PERIOD 参数值。
+* 错误重试次数：对于可通过重试解决的错误，系统会按照此次数进行重试。
+* 错误重试间隔：每次重试之间的时间间隔。
+* 目录：存储备份文件的目录。
+* 备份文件大小：备份文件的大小限制。当备份文件大小达到此限制时，会自动创建新的备份文件。
+* 文件压缩等级：备份文件的压缩等级。支持：最快速度、最佳压缩比、兼具速度和压缩比。
+
+创建成功后，备份计划会开始按照配置的参数运行。
+
+### 2.1.2. 查看
+
+在“备份计划”下的列表中，可以查看已创建的备份计划。
+
+![backup-zh-03.png](./pic/backup-03-list.png "查看备份计划列表")
+
+点击“操作”中的“查看”按钮，可以查看备份计划的详细信息。
+
+![backup-zh-04.png](./pic/backup-04-view.png "查看备份计划详情")
+
+### 2.1.3. 修改
+
+点击“操作”中的“修改”按钮，可以修改备份计划的配置。
+
+![backup-zh-05.png](./pic/backup-05-edit.png "修改备份计划")
+
+修改备份计划的配置后，当前运行的备份任务会先停止，然后按照新的配置重新运行。
+
+### 2.1.4. 复制
+
+点击“操作”中的“复制”按钮，可以复制备份计划。
+
+![backup-zh-06.png](./pic/backup-06-copy.png "复制备份计划")
+
+除了数据库和超级表被置为空外，其他配置项和被复制的计划相同。用户点击“确认”后，创建一个新的备份计划。
+
+### 2.1.5. 删除
+
+在操作中点击关闭按钮，可以停止当前备份计划。点击“操作”中的“删除”按钮，可以删除备份计划。
+
+![backup-zh-07.png](./pic/backup-07-del.png "删除备份计划")
+
+删除备份计划时，可以选择，是否删除关联的备份文件。
+
+## 2.2. 备份文件
+
+### 2.2.1. 查看
+
+在备份计划列表中，选择要一个备份计划。在“备份文件”列中，点击“查看”按钮。可以查看和备份计划的所有备份点。
+
+![backup-zh-08.png](./pic/backup-08-files.png "查看备份文件")
+
+在备份文件列表中，可以查看备份文件的详细信息。
+
+![backup-zh-09.png](./pic/backup-09-filelist.png "查看备份文件列表")
+
+## 2.3. 恢复任务
+
+### 2.3.1. 创建
+
+在备份文件列表中，点击“操作”中的“恢复”按钮，可以创建一个恢复任务。
+
+![backup-zh-10.png](./pic/backup-10-restore-create.png "创建恢复任务")
+
+在弹出的对话框中，选择使用哪个备份点开始恢复，默认为最早的备份点。点击“确定”后，创建恢复任务，并跳转至“恢复任务”列表。
+
+### 2.3.2. 查看
+
+在“恢复任务”列表中，可以查看已创建的恢复任务。
+
+![backup-zh-11.png](./pic/backup-11-restore-list.png "查看恢复任务列表")
+
+恢复任务可以终止。点击“操作”中的开关，可以终止当前恢复任务。
+
+# 3. 常见错误排查
+
+## 3.1. 端口访问异常
+
+如果任务启动失败并报以下错误：
 
 ```text
 Error: tmq to td task exec error
@@ -52,9 +161,12 @@ Error: tmq to td task exec error
 Caused by:
     [0x000B] Unable to establish connection
 ```
+
 产生原因是与数据源的端口链接异常，需检查数据源 FQDN 是否联通及端口 6030 是否可正常访问。
 
-2. 如果使用 WebSocket 连接，任务启动失败并报以下错误：
+## 3.2. 连接异常
+
+如果使用 WebSocket 连接，任务启动失败并报以下错误：
 
 ```text
 Error: tmq to td task exec error
@@ -73,7 +185,9 @@ Caused by:
 - "HTTP error: *": 可能连接到错误的 taosAdapter 端口或 LSB/Nginx/Proxy 配置错误。
 - "WebSocket protocol error: Handshake not finished": WebSocket 连接错误，通常是因为配置的端口不正确。
 
-3. 如果任务启动失败并报以下错误：
+## 3.3. 任务启动失败
+
+如果任务启动失败并报以下错误：
 
 ```text
 Error: tmq to td task exec error
@@ -88,5 +202,6 @@ Caused by:
 修改数据 WAL 配置：
 
 ```sql
-alter database test wal_retention_period 3600;
+alter
+database test wal_retention_period 3600;
 ```

include/common/tglobal.h

@@ -114,6 +114,7 @@ extern int32_t tsRetentionSpeedLimitMB;
 
 extern const char *tsAlterCompactTaskKeywords;
 extern int32_t     tsNumOfCompactThreads;
+extern int32_t     tsNumOfRetentionThreads;
 
 // sync raft
 extern int32_t tsElectInterval;
@@ -291,6 +292,7 @@ extern bool    tsFilterScalarMode;
 extern int32_t tsMaxStreamBackendCache;
 extern int32_t tsPQSortMemThreshold;
 extern int32_t tsResolveFQDNRetryTime;
+extern bool    tsStreamCoverage;
 
 extern bool tsExperimental;
 // #define NEEDTO_COMPRESSS_MSG(size) (tsCompressMsgSize != -1 && (size) > tsCompressMsgSize)

include/libs/function/tudf.h

@@ -109,8 +109,9 @@ int32_t doCallUdfAggProcess(UdfcFuncHandle handle, SSDataBlock *block, SUdfInter
 int32_t doCallUdfAggFinalize(UdfcFuncHandle handle, SUdfInterBuf *interBuf, SUdfInterBuf *resultData);
 // input: interbuf1, interbuf2
 // output: resultBuf
-int32_t doCallUdfAggMerge(UdfcFuncHandle handle, SUdfInterBuf *interBuf1, SUdfInterBuf *interBuf2,
-                          SUdfInterBuf *resultBuf);
+// udf todo:  aggmerge
+// int32_t doCallUdfAggMerge(UdfcFuncHandle handle, SUdfInterBuf *interBuf1, SUdfInterBuf *interBuf2,
+//                           SUdfInterBuf *resultBuf);
 // input: block
 // output: resultData
 int32_t doCallUdfScalarFunc(UdfcFuncHandle handle, SScalarParam *input, int32_t numOfCols, SScalarParam *output);

source/client/src/clientRawBlockWrite.c

@@ -68,10 +68,19 @@ static bool  tmqAddJsonArrayItem(cJSON *array, cJSON *item){
 }
 
 
-static int32_t  tmqWriteBatchMetaDataImpl(TAOS* taos, void* meta, int32_t metaLen);
-static tb_uid_t processSuid(tb_uid_t suid, char* db) { return suid + MurmurHash3_32(db, strlen(db)); }
+static int32_t  tmqWriteBatchMetaDataImpl(TAOS* taos, void* meta, uint32_t metaLen);
+static tb_uid_t processSuid(tb_uid_t suid, char* db) {
+  if (db == NULL) {
+    return suid;
+  }
+  return suid + MurmurHash3_32(db, strlen(db));
+}
 static void buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id, int8_t t,
                                  SColCmprWrapper* pColCmprRow, cJSON** pJson) {
+  if (schemaRow == NULL || name == NULL || pColCmprRow == NULL || pJson == NULL) {
+    uError("invalid parameter, schemaRow:%p, name:%p, pColCmprRow:%p, pJson:%p", schemaRow, name, pColCmprRow, pJson);
+    return;
+  }
   int32_t code = TSDB_CODE_SUCCESS;
   int8_t  buildDefaultCompress = 0;
   if (pColCmprRow->nCols <= 0) {
@@ -186,6 +195,9 @@ end:
 }
 
 static int32_t setCompressOption(cJSON* json, uint32_t para) {
+  if (json == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   uint8_t encode = COMPRESS_L1_TYPE_U32(para);
   int32_t code = 0;
   if (encode != 0) {
@@ -219,6 +231,10 @@ end:
   return code;
 }
 static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** pJson) {
+  if (alterData == NULL || pJson == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SMAlterStbReq req = {0};
   cJSON*        json = NULL;
   char*         string = NULL;
@@ -362,6 +378,10 @@ end:
 }
 
 static void processCreateStb(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (metaRsp == NULL || pJson == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SVCreateStbReq req = {0};
   SDecoder       coder;
 
@@ -382,6 +402,10 @@ end:
 }
 
 static void processAlterStb(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (metaRsp == NULL || pJson == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SVCreateStbReq req = {0};
   SDecoder       coder = {0};
   uDebug("alter stable data:%p", metaRsp);
@@ -402,6 +426,10 @@ end:
 }
 
 static void buildChildElement(cJSON* json, SVCreateTbReq* pCreateReq) {
+  if (json == NULL || pCreateReq == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   STag*   pTag = (STag*)pCreateReq->ctb.pTag;
   char*   sname = pCreateReq->ctb.stbName;
   char*   name = pCreateReq->name;
@@ -502,6 +530,10 @@ end:
 }
 
 static void buildCreateCTableJson(SVCreateTbReq* pCreateReq, int32_t nReqs, cJSON** pJson) {
+  if (pJson == NULL || pCreateReq == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   int32_t code = 0;
   char*   string = NULL;
   cJSON*  json = cJSON_CreateObject();
@@ -531,6 +563,10 @@ end:
 }
 
 static void processCreateTable(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (pJson == NULL || metaRsp == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDecoder           decoder = {0};
   SVCreateTbBatchReq req = {0};
   SVCreateTbReq*     pCreateReq;
@@ -561,6 +597,10 @@ end:
 }
 
 static void processAutoCreateTable(SMqDataRsp* rsp, char** string) {
+  if (rsp == NULL || string == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDecoder*      decoder = NULL;
   SVCreateTbReq* pCreateReq = NULL;
   int32_t        code = 0;
@@ -611,6 +651,10 @@ end:
 }
 
 static void processAlterTable(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (pJson == NULL || metaRsp == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDecoder     decoder = {0};
   SVAlterTbReq vAlterTbReq = {0};
   char*        string = NULL;
@@ -850,6 +894,10 @@ end:
 }
 
 static void processDropSTable(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (pJson == NULL || metaRsp == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDecoder     decoder = {0};
   SVDropStbReq req = {0};
   cJSON*       json = NULL;
@@ -884,6 +932,10 @@ end:
   *pJson = json;
 }
 static void processDeleteTable(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (pJson == NULL || metaRsp == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDeleteRes req = {0};
   SDecoder   coder = {0};
   cJSON*     json = NULL;
@@ -921,6 +973,10 @@ end:
 }
 
 static void processDropTable(SMqMetaRsp* metaRsp, cJSON** pJson) {
+  if (pJson == NULL || metaRsp == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDecoder         decoder = {0};
   SVDropTbBatchReq req = {0};
   cJSON*           json = NULL;
@@ -958,7 +1014,11 @@ end:
   *pJson = json;
 }
 
-static int32_t taosCreateStb(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t taosCreateStb(TAOS* taos, void* meta, uint32_t metaLen) {
+  if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SVCreateStbReq req = {0};
   SDecoder       coder;
   SMCreateStbReq pReq = {0};
@@ -973,8 +1033,8 @@ static int32_t taosCreateStb(TAOS* taos, void* meta, int32_t metaLen) {
     goto end;
   }
   // decode and process req
-  void*   data = POINTER_SHIFT(meta, sizeof(SMsgHead));
-  int32_t len = metaLen - sizeof(SMsgHead);
+  void*    data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  uint32_t len = metaLen - sizeof(SMsgHead);
   tDecoderInit(&coder, data, len);
   if (tDecodeSVCreateStbReq(&coder, &req) < 0) {
     code = TSDB_CODE_INVALID_PARA;
@@ -1068,7 +1128,11 @@ end:
   return code;
 }
 
-static int32_t taosDropStb(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t taosDropStb(TAOS* taos, void* meta, uint32_t metaLen) {
+  if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SVDropStbReq req = {0};
   SDecoder     coder = {0};
   SMDropStbReq pReq = {0};
@@ -1083,8 +1147,8 @@ static int32_t taosDropStb(TAOS* taos, void* meta, int32_t metaLen) {
     goto end;
   }
   // decode and process req
-  void*   data = POINTER_SHIFT(meta, sizeof(SMsgHead));
-  int32_t len = metaLen - sizeof(SMsgHead);
+  void*    data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  uint32_t len = metaLen - sizeof(SMsgHead);
   tDecoderInit(&coder, data, len);
   if (tDecodeSVDropStbReq(&coder, &req) < 0) {
     code = TSDB_CODE_INVALID_PARA;
@@ -1173,11 +1237,19 @@ typedef struct SVgroupCreateTableBatch {
 } SVgroupCreateTableBatch;
 
 static void destroyCreateTbReqBatch(void* data) {
+  if (data == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SVgroupCreateTableBatch* pTbBatch = (SVgroupCreateTableBatch*)data;
   taosArrayDestroy(pTbBatch->req.pArray);
 }
 
-static int32_t taosCreateTable(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t taosCreateTable(TAOS* taos, void* meta, uint32_t metaLen) {
+  if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SVCreateTbBatchReq req = {0};
   SDecoder           coder = {0};
   int32_t            code = TSDB_CODE_SUCCESS;
@@ -1195,8 +1267,8 @@ static int32_t taosCreateTable(TAOS* taos, void* meta, int32_t metaLen) {
     goto end;
   }
   // decode and process req
-  void*   data = POINTER_SHIFT(meta, sizeof(SMsgHead));
-  int32_t len = metaLen - sizeof(SMsgHead);
+  void*    data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  uint32_t len = metaLen - sizeof(SMsgHead);
   tDecoderInit(&coder, data, len);
   if (tDecodeSVCreateTbBatchReq(&coder, &req) < 0) {
     code = TSDB_CODE_INVALID_PARA;
@@ -1359,11 +1431,19 @@ typedef struct SVgroupDropTableBatch {
 } SVgroupDropTableBatch;
 
 static void destroyDropTbReqBatch(void* data) {
+  if (data == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SVgroupDropTableBatch* pTbBatch = (SVgroupDropTableBatch*)data;
   taosArrayDestroy(pTbBatch->req.pArray);
 }
 
-static int32_t taosDropTable(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t taosDropTable(TAOS* taos, void* meta, uint32_t metaLen) {
+  if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SVDropTbBatchReq req = {0};
   SDecoder         coder = {0};
   int32_t          code = TSDB_CODE_SUCCESS;
@@ -1380,8 +1460,8 @@ static int32_t taosDropTable(TAOS* taos, void* meta, int32_t metaLen) {
     goto end;
   }
   // decode and process req
-  void*   data = POINTER_SHIFT(meta, sizeof(SMsgHead));
-  int32_t len = metaLen - sizeof(SMsgHead);
+  void*    data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  uint32_t len = metaLen - sizeof(SMsgHead);
   tDecoderInit(&coder, data, len);
   if (tDecodeSVDropTbBatchReq(&coder, &req) < 0) {
     code = TSDB_CODE_INVALID_PARA;
@@ -1475,7 +1555,11 @@ end:
   return code;
 }
 
-static int32_t taosDeleteData(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t taosDeleteData(TAOS* taos, void* meta, uint32_t metaLen) {
+  if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SDeleteRes req = {0};
   SDecoder   coder = {0};
   char       sql[256] = {0};
@@ -1484,8 +1568,8 @@ static int32_t taosDeleteData(TAOS* taos, void* meta, int32_t metaLen) {
   uDebug("connId:0x%" PRIx64 " delete data, meta:%p, len:%d", *(int64_t*)taos, meta, metaLen);
 
   // decode and process req
-  void*   data = POINTER_SHIFT(meta, sizeof(SMsgHead));
-  int32_t len = metaLen - sizeof(SMsgHead);
+  void*    data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  uint32_t len = metaLen - sizeof(SMsgHead);
   tDecoderInit(&coder, data, len);
   if (tDecodeDeleteRes(&coder, &req) < 0) {
     code = TSDB_CODE_INVALID_PARA;
@@ -1510,7 +1594,11 @@ end:
   return code;
 }
 
-static int32_t taosAlterTable(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t taosAlterTable(TAOS* taos, void* meta, uint32_t metaLen) {
+  if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SVAlterTbReq   req = {0};
   SDecoder       dcoder = {0};
   int32_t        code = TSDB_CODE_SUCCESS;
@@ -1527,8 +1615,8 @@ static int32_t taosAlterTable(TAOS* taos, void* meta, int32_t metaLen) {
     goto end;
   }
   // decode and process req
-  void*   data = POINTER_SHIFT(meta, sizeof(SMsgHead));
-  int32_t len = metaLen - sizeof(SMsgHead);
+  void*    data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  uint32_t len = metaLen - sizeof(SMsgHead);
   tDecoderInit(&dcoder, data, len);
   if (tDecodeSVAlterTbReq(&dcoder, &req) < 0) {
     code = TSDB_CODE_INVALID_PARA;
@@ -1632,7 +1720,8 @@ int taos_write_raw_block_with_fields(TAOS* taos, int rows, char* pData, const ch
 
 int taos_write_raw_block_with_fields_with_reqid(TAOS* taos, int rows, char* pData, const char* tbname,
                                                 TAOS_FIELD* fields, int numFields, int64_t reqid) {
-  if (!taos || !pData || !tbname) {
+  if (taos == NULL || pData == NULL || tbname == NULL) {
+    uError("invalid parameter in %s", __func__);
     return TSDB_CODE_INVALID_PARA;
   }
   int32_t     code = TSDB_CODE_SUCCESS;
@@ -1693,7 +1782,7 @@ int taos_write_raw_block(TAOS* taos, int rows, char* pData, const char* tbname)
 }
 
 int taos_write_raw_block_with_reqid(TAOS* taos, int rows, char* pData, const char* tbname, int64_t reqid) {
-  if (!taos || !pData || !tbname) {
+  if (taos == NULL || pData == NULL || tbname == NULL) {
     return TSDB_CODE_INVALID_PARA;
   }
   int32_t     code = TSDB_CODE_SUCCESS;
@@ -1749,6 +1838,10 @@ end:
 }
 
 static void* getRawDataFromRes(void* pRetrieve) {
+  if (pRetrieve == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return NULL;
+  }
   void* rawData = NULL;
   // deal with compatibility
   if (*(int64_t*)pRetrieve == 0) {
@@ -1760,6 +1853,10 @@ static void* getRawDataFromRes(void* pRetrieve) {
 }
 
 static int32_t buildCreateTbMap(SMqDataRsp* rsp, SHashObj* pHashObj) {
+  if (rsp == NULL || pHashObj == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   // find schema data info
   int32_t       code = 0;
   SVCreateTbReq pCreateReq = {0};
@@ -1819,11 +1916,19 @@ typedef struct {
 } tbInfo;
 
 static void tmqFreeMeta(void* data) {
+  if (data == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   STableMeta* pTableMeta = *(STableMeta**)data;
   taosMemoryFree(pTableMeta);
 }
 
 static void freeRawCache(void* data) {
+  if (data == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   rawCacheInfo* pRawCache = (rawCacheInfo*)data;
   taosHashCleanup(pRawCache->pMetaHash);
   taosHashCleanup(pRawCache->pNameHash);
@@ -1842,6 +1947,10 @@ static int32_t initRawCacheHash() {
 }
 
 static bool needRefreshMeta(void* rawData, STableMeta* pTableMeta, SSchemaWrapper* pSW) {
+  if (rawData == NULL || pTableMeta == NULL || pSW == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return false;
+  }
   char* p = (char*)rawData;
   // | version | total length | total rows | blankFill | total columns | flag seg| block group id | column schema | each
   // column length |
@@ -1877,6 +1986,10 @@ static bool needRefreshMeta(void* rawData, STableMeta* pTableMeta, SSchemaWrappe
 }
 
 static int32_t getRawCache(SHashObj** pVgHash, SHashObj** pNameHash, SHashObj** pMetaHash, void* key) {
+  if (pVgHash == NULL || pNameHash == NULL || pMetaHash == NULL || key == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   void*   cacheInfo = taosHashGet(writeRawCache, &key, POINTER_BYTES);
   if (cacheInfo == NULL) {
@@ -1905,6 +2018,10 @@ end:
 }
 
 static int32_t buildRawRequest(TAOS* taos, SRequestObj** pRequest, SCatalog** pCatalog, SRequestConnInfo* conn) {
+  if (taos == NULL || pRequest == NULL || pCatalog == NULL || conn == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   RAW_RETURN_CHECK(buildRequest(*(int64_t*)taos, "", 0, NULL, false, pRequest, 0));
   (*pRequest)->syncQuery = true;
@@ -1924,26 +2041,38 @@ end:
 }
 
 typedef int32_t _raw_decode_func_(SDecoder* pDecoder, SMqDataRsp* pRsp);
-static int32_t  decodeRawData(SDecoder* decoder, void* data, int32_t dataLen, _raw_decode_func_ func,
+static int32_t  decodeRawData(SDecoder* decoder, void* data, uint32_t dataLen, _raw_decode_func_ func,
                               SMqRspObj* rspObj) {
-   int8_t dataVersion = *(int8_t*)data;
-   if (dataVersion >= MQ_DATA_RSP_VERSION) {
-     data = POINTER_SHIFT(data, sizeof(int8_t) + sizeof(int32_t));
-     dataLen -= sizeof(int8_t) + sizeof(int32_t);
+  if (decoder == NULL || data == NULL || func == NULL || rspObj == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
+  int8_t dataVersion = *(int8_t*)data;
+  if (dataVersion >= MQ_DATA_RSP_VERSION) {
+    data = POINTER_SHIFT(data, sizeof(int8_t) + sizeof(int32_t));
+    if (dataLen < sizeof(int8_t) + sizeof(int32_t)) {
+      return TSDB_CODE_INVALID_PARA;
+    }
+    dataLen -= sizeof(int8_t) + sizeof(int32_t);
   }
 
-   rspObj->resIter = -1;
-   tDecoderInit(decoder, data, dataLen);
-   int32_t code = func(decoder, &rspObj->dataRsp);
-   if (code != 0) {
-     SET_ERROR_MSG("decode mq taosx data rsp failed");
+  rspObj->resIter = -1;
+  tDecoderInit(decoder, data, dataLen);
+  int32_t code = func(decoder, &rspObj->dataRsp);
+  if (code != 0) {
+    SET_ERROR_MSG("decode mq taosx data rsp failed");
   }
-   return code;
+  return code;
 }
 
 static int32_t processCacheMeta(SHashObj* pVgHash, SHashObj* pNameHash, SHashObj* pMetaHash,
                                 SVCreateTbReq* pCreateReqDst, SCatalog* pCatalog, SRequestConnInfo* conn, SName* pName,
                                 STableMeta** pMeta, SSchemaWrapper* pSW, void* rawData, int32_t retry) {
+  if (pVgHash == NULL || pNameHash == NULL || pMetaHash == NULL || pCatalog == NULL || conn == NULL || pName == NULL ||
+      pMeta == NULL || pSW == NULL || rawData == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t     code = 0;
   STableMeta* pTableMeta = NULL;
   tbInfo*     tmpInfo = (tbInfo*)taosHashGet(pNameHash, pName->tname, strlen(pName->tname));
@@ -2000,7 +2129,11 @@ end:
   return code;
 }
 
-static int32_t tmqWriteRawDataImpl(TAOS* taos, void* data, int32_t dataLen) {
+static int32_t tmqWriteRawDataImpl(TAOS* taos, void* data, uint32_t dataLen) {
+  if (taos == NULL || data == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t   code = TSDB_CODE_SUCCESS;
   SQuery*   pQuery = NULL;
   SMqRspObj rspObj = {0};
@@ -2073,7 +2206,11 @@ end:
   return code;
 }
 
-static int32_t tmqWriteRawMetaDataImpl(TAOS* taos, void* data, int32_t dataLen) {
+static int32_t tmqWriteRawMetaDataImpl(TAOS* taos, void* data, uint32_t dataLen) {
+  if (taos == NULL || data == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t   code = TSDB_CODE_SUCCESS;
   SQuery*   pQuery = NULL;
   SMqRspObj rspObj = {0};
@@ -2162,6 +2299,10 @@ end:
 }
 
 static void processSimpleMeta(SMqMetaRsp* pMetaRsp, cJSON** meta) {
+  if (pMetaRsp == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   if (pMetaRsp->resMsgType == TDMT_VND_CREATE_STB) {
     processCreateStb(pMetaRsp, meta);
   } else if (pMetaRsp->resMsgType == TDMT_VND_ALTER_STB) {
@@ -2182,6 +2323,10 @@ static void processSimpleMeta(SMqMetaRsp* pMetaRsp, cJSON** meta) {
 }
 
 static void processBatchMetaToJson(SMqBatchMetaRsp* pMsgRsp, char** string) {
+  if (pMsgRsp == NULL || string == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return;
+  }
   SDecoder        coder;
   SMqBatchMetaRsp rsp = {0};
   int32_t         code = 0;
@@ -2228,7 +2373,10 @@ end:
 }
 
 char* tmq_get_json_meta(TAOS_RES* res) {
-  if (res == NULL) return NULL;
+  if (res == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return NULL;
+  }
   uDebug("tmq_get_json_meta res:%p", res);
   if (!TD_RES_TMQ_META(res) && !TD_RES_TMQ_METADATA(res) && !TD_RES_TMQ_BATCH_META(res)) {
     return NULL;
@@ -2256,6 +2404,10 @@ char* tmq_get_json_meta(TAOS_RES* res) {
 void tmq_free_json_meta(char* jsonMeta) { taosMemoryFreeClear(jsonMeta); }
 
 static int32_t getOffSetLen(const SMqDataRsp* pRsp) {
+  if (pRsp == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   SEncoder coder = {0};
   tEncoderInit(&coder, NULL, 0);
   if (tEncodeSTqOffsetVal(&coder, &pRsp->reqOffset) < 0) return -1;
@@ -2267,44 +2419,48 @@ static int32_t getOffSetLen(const SMqDataRsp* pRsp) {
 
 typedef int32_t __encode_func__(SEncoder* pEncoder, const SMqDataRsp* pRsp);
 static int32_t  encodeMqDataRsp(__encode_func__* encodeFunc, SMqDataRsp* rspObj, tmq_raw_data* raw) {
-   int32_t  len = 0;
-   int32_t  code = 0;
-   SEncoder encoder = {0};
-   void*    buf = NULL;
-   tEncodeSize(encodeFunc, rspObj, len, code);
-   if (code < 0) {
-     code = TSDB_CODE_INVALID_MSG;
-     goto FAILED;
+  if (raw == NULL || encodeFunc == NULL || rspObj == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
   }
-   len += sizeof(int8_t) + sizeof(int32_t);
-   buf = taosMemoryCalloc(1, len);
-   if (buf == NULL) {
-     code = terrno;
-     goto FAILED;
+  uint32_t len = 0;
+  int32_t  code = 0;
+  SEncoder encoder = {0};
+  void*    buf = NULL;
+  tEncodeSize(encodeFunc, rspObj, len, code);
+  if (code < 0) {
+    code = TSDB_CODE_INVALID_MSG;
+    goto FAILED;
   }
-   tEncoderInit(&encoder, buf, len);
-   if (tEncodeI8(&encoder, MQ_DATA_RSP_VERSION) < 0) {
-     code = TSDB_CODE_INVALID_MSG;
-     goto FAILED;
+  len += sizeof(int8_t) + sizeof(int32_t);
+  buf = taosMemoryCalloc(1, len);
+  if (buf == NULL) {
+    code = terrno;
+    goto FAILED;
   }
-   int32_t offsetLen = getOffSetLen(rspObj);
-   if (offsetLen <= 0) {
-     code = TSDB_CODE_INVALID_MSG;
-     goto FAILED;
+  tEncoderInit(&encoder, buf, len);
+  if (tEncodeI8(&encoder, MQ_DATA_RSP_VERSION) < 0) {
+    code = TSDB_CODE_INVALID_MSG;
+    goto FAILED;
   }
-   if (tEncodeI32(&encoder, offsetLen) < 0) {
-     code = TSDB_CODE_INVALID_MSG;
-     goto FAILED;
+  int32_t offsetLen = getOffSetLen(rspObj);
+  if (offsetLen <= 0) {
+    code = TSDB_CODE_INVALID_MSG;
+    goto FAILED;
   }
-   if (encodeFunc(&encoder, rspObj) < 0) {
-     code = TSDB_CODE_INVALID_MSG;
-     goto FAILED;
+  if (tEncodeI32(&encoder, offsetLen) < 0) {
+    code = TSDB_CODE_INVALID_MSG;
+    goto FAILED;
   }
-   tEncoderClear(&encoder);
+  if (encodeFunc(&encoder, rspObj) < 0) {
+    code = TSDB_CODE_INVALID_MSG;
+    goto FAILED;
+  }
+  tEncoderClear(&encoder);
 
-   raw->raw = buf;
-   raw->raw_len = len;
-   return code;
+  raw->raw = buf;
+  raw->raw_len = len;
+  return code;
 FAILED:
   tEncoderClear(&encoder);
   taosMemoryFree(buf);
@@ -2312,13 +2468,14 @@ FAILED:
 }
 
 int32_t tmq_get_raw(TAOS_RES* res, tmq_raw_data* raw) {
-  if (!raw || !res) {
+  if (raw == NULL || res == NULL) {
+    uError("invalid parameter in %s", __func__);
     return TSDB_CODE_INVALID_PARA;
   }
   SMqRspObj* rspObj = ((SMqRspObj*)res);
   if (TD_RES_TMQ_META(res)) {
     raw->raw = rspObj->metaRsp.metaRsp;
-    raw->raw_len = rspObj->metaRsp.metaRspLen;
+    raw->raw_len = rspObj->metaRsp.metaRspLen >= 0 ? rspObj->metaRsp.metaRspLen : 0;
     raw->raw_type = rspObj->metaRsp.resMsgType;
     uDebug("tmq get raw type meta:%p", raw);
   } else if (TD_RES_TMQ(res)) {
@@ -2378,6 +2535,10 @@ static int32_t writeRawInit() {
 }
 
 static int32_t writeRawImpl(TAOS* taos, void* buf, uint32_t len, uint16_t type) {
+  if (taos == NULL || buf == NULL) {
+    uError("invalid parameter in %s", __func__);
+    return TSDB_CODE_INVALID_PARA;
+  }
   if (writeRawInit() != 0) {
     return TSDB_CODE_INTERNAL_ERROR;
   }
@@ -2415,8 +2576,9 @@ int32_t tmq_write_raw(TAOS* taos, tmq_raw_data raw) {
   return writeRawImpl(taos, raw.raw, raw.raw_len, raw.raw_type);
 }
 
-static int32_t tmqWriteBatchMetaDataImpl(TAOS* taos, void* meta, int32_t metaLen) {
+static int32_t tmqWriteBatchMetaDataImpl(TAOS* taos, void* meta, uint32_t metaLen) {
   if (taos == NULL || meta == NULL) {
+    uError("invalid parameter in %s", __func__);
     return TSDB_CODE_INVALID_PARA;
   }
   SMqBatchMetaRsp rsp = {0};

source/client/src/clientTmq.c

@@ -554,6 +554,9 @@ char** tmq_list_to_c_array(const tmq_list_t* list) {
 }
 
 static int32_t tmqCommitDone(SMqCommitCbParamSet* pParamSet) {
+  if (pParamSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int64_t refId = pParamSet->refId;
   int32_t code = 0;
   tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, refId);
@@ -575,6 +578,9 @@ static int32_t tmqCommitDone(SMqCommitCbParamSet* pParamSet) {
 }
 
 static int32_t commitRspCountDown(SMqCommitCbParamSet* pParamSet, int64_t consumerId, const char* pTopic, int32_t vgId) {
+  if (pParamSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t waitingRspNum = atomic_sub_fetch_32(&pParamSet->waitingRspNum, 1);
   if (waitingRspNum == 0) {
     tqDebugC("consumer:0x%" PRIx64 " topic:%s vgId:%d all commit-rsp received, commit completed", consumerId, pTopic,
@@ -603,6 +609,9 @@ static int32_t tmqCommitCb(void* param, SDataBuf* pBuf, int32_t code) {
 
 static int32_t doSendCommitMsg(tmq_t* tmq, int32_t vgId, SEpSet* epSet, STqOffsetVal* offset, const char* pTopicName,
                                SMqCommitCbParamSet* pParamSet) {
+  if (tmq == NULL || epSet == NULL || offset == NULL || pTopicName == NULL || pParamSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqVgOffset pOffset = {0};
 
   pOffset.consumerId = tmq->consumerId;
@@ -673,6 +682,9 @@ static int32_t doSendCommitMsg(tmq_t* tmq, int32_t vgId, SEpSet* epSet, STqOffse
 }
 
 static int32_t getTopicByName(tmq_t* tmq, const char* pTopicName, SMqClientTopic** topic) {
+  if (tmq == NULL || pTopicName == NULL || topic == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t numOfTopics = taosArrayGetSize(tmq->clientTopics);
   for (int32_t i = 0; i < numOfTopics; ++i) {
     SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i);
@@ -689,6 +701,9 @@ static int32_t getTopicByName(tmq_t* tmq, const char* pTopicName, SMqClientTopic
 
 static int32_t prepareCommitCbParamSet(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* userParam, int32_t rspNum,
                                        SMqCommitCbParamSet** ppParamSet) {
+  if (tmq == NULL || ppParamSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqCommitCbParamSet* pParamSet = taosMemoryCalloc(1, sizeof(SMqCommitCbParamSet));
   if (pParamSet == NULL) {
     return terrno;
@@ -704,6 +719,9 @@ static int32_t prepareCommitCbParamSet(tmq_t* tmq, tmq_commit_cb* pCommitFp, voi
 }
 
 static int32_t getClientVg(tmq_t* tmq, char* pTopicName, int32_t vgId, SMqClientVg** pVg) {
+  if (tmq == NULL || pTopicName == NULL || pVg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqClientTopic* pTopic = NULL;
   int32_t         code = getTopicByName(tmq, pTopicName, &pTopic);
   if (code != 0) {
@@ -724,6 +742,9 @@ static int32_t getClientVg(tmq_t* tmq, char* pTopicName, int32_t vgId, SMqClient
 }
 
 static int32_t innerCommit(tmq_t* tmq, char* pTopicName, STqOffsetVal* offsetVal, SMqClientVg* pVg, SMqCommitCbParamSet* pParamSet){
+  if (tmq == NULL || pTopicName == NULL || offsetVal == NULL || pVg == NULL || pParamSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   if (offsetVal->type <= 0) {
     code = TSDB_CODE_TMQ_INVALID_MSG;
@@ -754,6 +775,9 @@ static int32_t innerCommit(tmq_t* tmq, char* pTopicName, STqOffsetVal* offsetVal
 
 static int32_t asyncCommitOffset(tmq_t* tmq, char* pTopicName, int32_t vgId, STqOffsetVal* offsetVal,
                                  tmq_commit_cb* pCommitFp, void* userParam) {
+  if (tmq == NULL || pTopicName == NULL || offsetVal == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   tqInfoC("consumer:0x%" PRIx64 " do manual commit offset for %s, vgId:%d", tmq->consumerId, pTopicName, vgId);
   SMqCommitCbParamSet* pParamSet = NULL;
   int32_t code = prepareCommitCbParamSet(tmq, pCommitFp, userParam, 0, &pParamSet);
@@ -807,6 +831,9 @@ end:
 }
 
 static int32_t innerCommitAll(tmq_t* tmq, SMqCommitCbParamSet* pParamSet){
+  if (tmq == NULL || pParamSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   taosRLockLatch(&tmq->lock);
   int32_t numOfTopics = taosArrayGetSize(tmq->clientTopics);
@@ -842,6 +869,9 @@ END:
 }
 
 static void asyncCommitAllOffsets(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* userParam) {
+  if (tmq == NULL) {
+    return;
+  }
   int32_t code = 0;
   SMqCommitCbParamSet* pParamSet = NULL;
   // init waitingRspNum as DEFAULT_COMMIT_CNT to prevent concurrency issue
@@ -1071,12 +1101,15 @@ END:
 }
 
 static void defaultCommitCbFn(tmq_t* pTmq, int32_t code, void* param) {
-  if (code != 0) {
+  if (code != 0 && pTmq != NULL) {
     tqErrorC("consumer:0x%" PRIx64 ", failed to commit offset, code:%s", pTmq->consumerId, tstrerror(code));
   }
 }
 
 static void tmqFreeRspWrapper(SMqRspWrapper* rspWrapper) {
+  if (rspWrapper == NULL) {
+    return;
+  }
   if (rspWrapper->tmqRspType == TMQ_MSG_TYPE__EP_RSP) {
     tDeleteSMqAskEpRsp(&rspWrapper->epRsp);
   } else if (rspWrapper->tmqRspType == TMQ_MSG_TYPE__POLL_DATA_RSP) {
@@ -1091,12 +1124,18 @@ static void tmqFreeRspWrapper(SMqRspWrapper* rspWrapper) {
 }
 
 static void freeClientVg(void* param) {
+  if (param == NULL) {
+    return;
+  }
   SMqClientVg* pVg = param;
   tOffsetDestroy(&pVg->offsetInfo.endOffset);
   tOffsetDestroy(&pVg->offsetInfo.beginOffset);
   tOffsetDestroy(&pVg->offsetInfo.committedOffset);
 }
 static void freeClientTopic(void* param) {
+  if (param == NULL) {
+    return;
+  }
   SMqClientTopic* pTopic = param;
   taosMemoryFreeClear(pTopic->schema.pSchema);
   taosArrayDestroyEx(pTopic->vgs, freeClientVg);
@@ -1104,6 +1143,9 @@ static void freeClientTopic(void* param) {
 
 static void initClientTopicFromRsp(SMqClientTopic* pTopic, SMqSubTopicEp* pTopicEp, SHashObj* pVgOffsetHashMap,
                                    tmq_t* tmq) {
+  if (pTopic == NULL || pTopicEp == NULL || pVgOffsetHashMap == NULL || tmq == NULL) {
+    return;
+  }
   pTopic->schema = pTopicEp->schema;
   pTopicEp->schema.nCols = 0;
   pTopicEp->schema.pSchema = NULL;
@@ -1167,6 +1209,9 @@ static void initClientTopicFromRsp(SMqClientTopic* pTopic, SMqSubTopicEp* pTopic
 }
 
 static void buildNewTopicList(tmq_t* tmq, SArray* newTopics, const SMqAskEpRsp* pRsp){
+        if (tmq == NULL || newTopics == NULL || pRsp == NULL) {
+        return;
+        }
   SHashObj* pVgOffsetHashMap = taosHashInit(64, MurmurHash3_32, false, HASH_NO_LOCK);
   if (pVgOffsetHashMap == NULL) {
     tqErrorC("consumer:0x%" PRIx64 " taos hash init null, code:%d", tmq->consumerId, terrno);
@@ -1221,6 +1266,9 @@ static void buildNewTopicList(tmq_t* tmq, SArray* newTopics, const SMqAskEpRsp*
 }
 
 static void doUpdateLocalEp(tmq_t* tmq, int32_t epoch, const SMqAskEpRsp* pRsp) {
+        if (tmq == NULL || pRsp == NULL) {
+        return;
+        }
   int32_t topicNumGet = taosArrayGetSize(pRsp->topics);
   // vnode transform (epoch == tmq->epoch && topicNumGet != 0)
   // ask ep rsp (epoch == tmq->epoch && topicNumGet == 0)
@@ -1337,6 +1385,9 @@ static int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) {
 }
 
 static int32_t askEp(tmq_t* pTmq, void* param, bool sync, bool updateEpSet) {
+  if (pTmq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqAskEpReq req = {0};
   req.consumerId = pTmq->consumerId;
   req.epoch = updateEpSet ? -1 : pTmq->epoch;
@@ -1395,6 +1446,9 @@ static int32_t askEp(tmq_t* pTmq, void* param, bool sync, bool updateEpSet) {
 }
 
 void tmqHandleAllDelayedTask(tmq_t* pTmq) {
+        if (pTmq == NULL) {
+        return;
+        }
   STaosQall* qall = NULL;
   int32_t    code = 0;
 
@@ -1443,6 +1497,7 @@ void tmqHandleAllDelayedTask(tmq_t* pTmq) {
 }
 
 void tmqClearUnhandleMsg(tmq_t* tmq) {
+  if (tmq == NULL) return;
   SMqRspWrapper* rspWrapper = NULL;
   while (taosGetQitem(tmq->qall, (void**)&rspWrapper) != 0) {
     tmqFreeRspWrapper(rspWrapper);
@@ -1507,6 +1562,7 @@ int32_t tmq_subscription(tmq_t* tmq, tmq_list_t** topics) {
 }
 
 void tmqFreeImpl(void* handle) {
+        if (handle == NULL) return;
   tmq_t*  tmq = (tmq_t*)handle;
   int64_t id = tmq->consumerId;
 
@@ -1715,6 +1771,7 @@ _failed:
 }
 
 static int32_t syncAskEp(tmq_t* pTmq) {
+  if (pTmq == NULL) return TSDB_CODE_INVALID_PARA;
   SAskEpInfo* pInfo = taosMemoryMalloc(sizeof(SAskEpInfo));
   if (pInfo == NULL) return terrno;
   if (tsem2_init(&pInfo->sem, 0, 0) != 0) {
@@ -1897,6 +1954,9 @@ void tmq_conf_set_auto_commit_cb(tmq_conf_t* conf, tmq_commit_cb* cb, void* para
 }
 
 static void getVgInfo(tmq_t* tmq, char* topicName, int32_t vgId, SMqClientVg** pVg) {
+  if (tmq == NULL || topicName == NULL || pVg == NULL) {
+    return;
+  }
   int32_t topicNumCur = taosArrayGetSize(tmq->clientTopics);
   for (int i = 0; i < topicNumCur; i++) {
     SMqClientTopic* pTopicCur = taosArrayGet(tmq->clientTopics, i);
@@ -1914,6 +1974,9 @@ static void getVgInfo(tmq_t* tmq, char* topicName, int32_t vgId, SMqClientVg** p
 }
 
 static SMqClientTopic* getTopicInfo(tmq_t* tmq, char* topicName) {
+  if (tmq == NULL || topicName == NULL) {
+    return NULL;
+  }
   int32_t topicNumCur = taosArrayGetSize(tmq->clientTopics);
   for (int i = 0; i < topicNumCur; i++) {
     SMqClientTopic* pTopicCur = taosArrayGet(tmq->clientTopics, i);
@@ -2026,6 +2089,9 @@ EXIT:
 }
 
 void tmqBuildConsumeReqImpl(SMqPollReq* pReq, tmq_t* tmq, int64_t timeout, SMqClientTopic* pTopic, SMqClientVg* pVg) {
+  if (pReq == NULL || tmq == NULL || pTopic == NULL || pVg == NULL) {
+    return;
+  }
   (void)snprintf(pReq->subKey, TSDB_SUBSCRIBE_KEY_LEN, "%s%s%s", tmq->groupId, TMQ_SEPARATOR, pTopic->topicName);
   pReq->withTbName = tmq->withTbName;
   pReq->consumerId = tmq->consumerId;
@@ -2072,7 +2138,7 @@ void changeByteEndian(char* pData) {
 }
 
 static void tmqGetRawDataRowsPrecisionFromRes(void* pRetrieve, void** rawData, int64_t* rows, int32_t* precision) {
-  if (pRetrieve == NULL) {
+  if (pRetrieve == NULL || rawData == NULL || rows == NULL) {
     return;
   }
   if (*(int64_t*)pRetrieve == 0) {
@@ -2092,6 +2158,9 @@ static void tmqGetRawDataRowsPrecisionFromRes(void* pRetrieve, void** rawData, i
 
 static void tmqBuildRspFromWrapperInner(SMqPollRspWrapper* pWrapper, SMqClientVg* pVg, int64_t* numOfRows,
                                         SMqRspObj* pRspObj) {
+  if (pWrapper == NULL || pVg == NULL || numOfRows == NULL || pRspObj == NULL) {
+    return;
+  }
   pRspObj->resIter = -1;
   pRspObj->resInfo.totalRows = 0;
   pRspObj->resInfo.precision = TSDB_TIME_PRECISION_MILLI;
@@ -2130,6 +2199,9 @@ static void tmqBuildRspFromWrapperInner(SMqPollRspWrapper* pWrapper, SMqClientVg
 }
 
 static int32_t doTmqPollImpl(tmq_t* pTmq, SMqClientTopic* pTopic, SMqClientVg* pVg, int64_t timeout) {
+  if (pTmq == NULL || pTopic == NULL || pVg == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   SMqPollReq      req = {0};
   char*           msg = NULL;
   SMqPollCbParam* pParam = NULL;
@@ -2199,6 +2271,9 @@ static int32_t doTmqPollImpl(tmq_t* pTmq, SMqClientTopic* pTopic, SMqClientVg* p
 
 // broadcast the poll request to all related vnodes
 static int32_t tmqPollImpl(tmq_t* tmq, int64_t timeout) {
+  if (tmq == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t code = 0;
 
   taosWLockLatch(&tmq->lock);
@@ -2258,6 +2333,9 @@ end:
 
 static void updateVgInfo(SMqClientVg* pVg, STqOffsetVal* reqOffset, STqOffsetVal* rspOffset, int64_t sver, int64_t ever,
                          int64_t consumerId, bool hasData) {
+  if (pVg == NULL || reqOffset == NULL || rspOffset == NULL) {
+    return;
+  }
   if (!pVg->seekUpdated) {
     tqDebugC("consumer:0x%" PRIx64 " local offset is update, since seekupdate not set", consumerId);
     if (hasData) {
@@ -2283,6 +2361,9 @@ static SMqRspObj* buildRsp(SMqPollRspWrapper* pollRspWrapper){
     SMqBatchMetaRsp batchMetaRsp;
   } MEMSIZE;
 
+  if (pollRspWrapper == NULL) {
+    return NULL;
+  }
   SMqRspObj* pRspObj = taosMemoryCalloc(1, sizeof(SMqRspObj));
   if (pRspObj == NULL) {
     tqErrorC("buildRsp:failed to allocate memory");
@@ -2297,6 +2378,9 @@ static SMqRspObj* buildRsp(SMqPollRspWrapper* pollRspWrapper){
 }
 
 static void processMqRspError(tmq_t* tmq, SMqRspWrapper* pRspWrapper){
+  if (tmq == NULL || pRspWrapper == NULL) {
+    return;
+  }
   SMqPollRspWrapper* pollRspWrapper = &pRspWrapper->pollRsp;
 
   if (pRspWrapper->code == TSDB_CODE_VND_INVALID_VGROUP_ID) {  // for vnode transform
@@ -2322,6 +2406,9 @@ static void processMqRspError(tmq_t* tmq, SMqRspWrapper* pRspWrapper){
   taosWUnLockLatch(&tmq->lock);
 }
 static SMqRspObj* processMqRsp(tmq_t* tmq, SMqRspWrapper* pRspWrapper){
+  if (tmq == NULL || pRspWrapper == NULL) {
+    return NULL;
+  }
   SMqRspObj* pRspObj = NULL;
 
   if (pRspWrapper->tmqRspType == TMQ_MSG_TYPE__EP_RSP) {
@@ -2401,6 +2488,9 @@ static SMqRspObj* processMqRsp(tmq_t* tmq, SMqRspWrapper* pRspWrapper){
 }
 
 static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) {
+  if (tmq == NULL) {
+    return NULL;
+  }
   tqDebugC("consumer:0x%" PRIx64 " start to handle the rsp, total:%d", tmq->consumerId, taosQallItemSize(tmq->qall));
 
   void* returnVal = NULL;
@@ -2478,6 +2568,7 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) {
 }
 
 static void displayConsumeStatistics(tmq_t* pTmq) {
+  if (pTmq == NULL) return;
   taosRLockLatch(&pTmq->lock);
   int32_t numOfTopics = taosArrayGetSize(pTmq->clientTopics);
   tqInfoC("consumer:0x%" PRIx64 " closing poll:%" PRId64 " rows:%" PRId64 " topics:%d, final epoch:%d",
@@ -2680,7 +2771,11 @@ void tmq_commit_async(tmq_t* tmq, const TAOS_RES* pRes, tmq_commit_cb* cb, void*
   }
 }
 
-static void commitCallBackFn(tmq_t* UNUSED_PARAM(tmq), int32_t code, void* param) {
+static void commitCallBackFn(tmq_t* tmq, int32_t code, void* param) {
+  if (param == NULL) {
+    tqErrorC("invalid param in commit cb");
+    return;
+  }
   SSyncCommitInfo* pInfo = (SSyncCommitInfo*)param;
   pInfo->code = code;
   if (tsem2_post(&pInfo->sem) != 0){
@@ -2732,6 +2827,10 @@ int32_t tmq_commit_sync(tmq_t* tmq, const TAOS_RES* pRes) {
 
 // wal range will be ok after calling tmq_get_topic_assignment or poll interface
 static int32_t checkWalRange(SVgOffsetInfo* offset, int64_t value) {
+  if (offset == NULL) {
+    tqErrorC("invalid offset, null");
+    return TSDB_CODE_INVALID_PARA;
+  }
   if (offset->walVerBegin == -1 || offset->walVerEnd == -1) {
     tqErrorC("Assignment or poll interface need to be called first");
     return TSDB_CODE_TMQ_NEED_INITIALIZED;
@@ -2852,6 +2951,9 @@ end:
 
 
 int32_t tmqGetNextResInfo(TAOS_RES* res, bool convertUcs4, SReqResultInfo** pResInfo) {
+        if (res == NULL || pResInfo == NULL) {
+        return TSDB_CODE_INVALID_PARA;
+        }
   SMqRspObj*  pRspObj = (SMqRspObj*)res;
   SMqDataRsp* data = &pRspObj->dataRsp;
 
@@ -2889,7 +2991,7 @@ int32_t tmqGetNextResInfo(TAOS_RES* res, bool convertUcs4, SReqResultInfo** pRes
 }
 
 static int32_t tmqGetWalInfoCb(void* param, SDataBuf* pMsg, int32_t code) {
-  if (param == NULL) {
+  if (param == NULL || pMsg == NULL) {
     return code;
   }
   SMqVgWalInfoParam* pParam = param;
@@ -2962,6 +3064,9 @@ static bool isInSnapshotMode(int8_t type, bool useSnapshot) {
 }
 
 static int32_t tmCommittedCb(void* param, SDataBuf* pMsg, int32_t code) {
+  if (param == NULL) {
+    return code;
+  }
   SMqCommittedParam* pParam = param;
 
   if (code != 0) {
@@ -2992,6 +3097,9 @@ end:
 }
 
 int64_t getCommittedFromServer(tmq_t* tmq, char* tname, int32_t vgId, SEpSet* epSet) {
+  if (tmq == NULL || tname == NULL || epSet == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t     code = 0;
   SMqVgOffset pOffset = {0};
 

source/common/src/tglobal.c

@@ -14,12 +14,12 @@
  */
 
 #define _DEFAULT_SOURCE
+#include "tglobal.h"
 #include "cJSON.h"
 #include "defines.h"
 #include "os.h"
 #include "osString.h"
 #include "tconfig.h"
-#include "tglobal.h"
 #include "tgrant.h"
 #include "tjson.h"
 #include "tlog.h"
@@ -104,6 +104,7 @@ int32_t tsRetentionSpeedLimitMB = 0;    // unlimited
 
 const char *tsAlterCompactTaskKeywords = "max_compact_tasks";
 int32_t     tsNumOfCompactThreads = 2;
+int32_t     tsNumOfRetentionThreads = 1;
 
 // sync raft
 int32_t tsElectInterval = 25 * 1000;
@@ -328,6 +329,7 @@ int64_t tsStreamBufferSize = 128 * 1024 * 1024;
 bool    tsFilterScalarMode = false;
 int     tsResolveFQDNRetryTime = 100;  // seconds
 int     tsStreamAggCnt = 100000;
+bool    tsStreamCoverage = false;
 
 bool tsUpdateCacheBatch = true;
 
@@ -733,6 +735,9 @@ static int32_t taosAddClientCfg(SConfig *pCfg) {
                                 CFG_DYN_CLIENT, CFG_CATEGORY_LOCAL));
   TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "tsmaDataDeleteMark", tsmaDataDeleteMark, 60 * 60 * 1000, INT64_MAX,
                                 CFG_SCOPE_CLIENT, CFG_DYN_CLIENT, CFG_CATEGORY_LOCAL));
+  
+  TAOS_CHECK_RETURN(cfgAddBool(pCfg, "streamCoverage", tsStreamCoverage, CFG_DYN_CLIENT, CFG_DYN_CLIENT, CFG_CATEGORY_LOCAL));
+
   TAOS_RETURN(TSDB_CODE_SUCCESS);
 }
 
@@ -1463,6 +1468,9 @@ static int32_t taosSetClientCfg(SConfig *pCfg) {
   TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "bypassFlag");
   tsBypassFlag = pItem->i32;
 
+  TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "streamCoverage");
+  tsStreamCoverage = pItem->bval;
+
   TAOS_RETURN(TSDB_CODE_SUCCESS);
 }
 
@@ -2735,7 +2743,8 @@ static int32_t taosCfgDynamicOptionsForClient(SConfig *pCfg, const char *name) {
                                          {"maxTsmaCalcDelay", &tsMaxTsmaCalcDelay},
                                          {"tsmaDataDeleteMark", &tsmaDataDeleteMark},
                                          {"numOfRpcSessions", &tsNumOfRpcSessions},
-                                         {"bypassFlag", &tsBypassFlag}};
+                                         {"bypassFlag", &tsBypassFlag},
+                                         {"streamCoverage", &tsStreamCoverage}};
 
     if ((code = taosCfgSetOption(debugOptions, tListLen(debugOptions), pItem, true)) != TSDB_CODE_SUCCESS) {
       code = taosCfgSetOption(options, tListLen(options), pItem, false);

source/dnode/mgmt/mgmt_dnode/src/dmHandle.c

@@ -475,7 +475,7 @@ int32_t dmProcessGrantRsp(SDnodeMgmt *pMgmt, SRpcMsg *pMsg) {
   return 0;
 }
 
-extern void    tsdbAlterMaxCompactTasks();
+extern void    tsdbAlterNumCompactThreads();
 static int32_t dmAlterMaxCompactTask(const char *value) {
   int32_t max_compact_tasks;
   char   *endptr = NULL;
@@ -489,7 +489,7 @@ static int32_t dmAlterMaxCompactTask(const char *value) {
     dInfo("alter max compact tasks from %d to %d", tsNumOfCompactThreads, max_compact_tasks);
     tsNumOfCompactThreads = max_compact_tasks;
 #ifdef TD_ENTERPRISE
-    tsdbAlterMaxCompactTasks();
+    (void)tsdbAlterNumCompactThreads();
 #endif
   }
 

source/dnode/mgmt/mgmt_vnode/src/vmInt.c

@@ -896,7 +896,7 @@ static int32_t vmInit(SMgmtInputOpt *pInput, SMgmtOutputOpt *pOutput) {
   }
   tmsgReportStartup("vnode-sync", "initialized");
 
-  if ((code = vnodeInit(tsNumOfCommitThreads, pInput->stopDnodeFp)) != 0) {
+  if ((code = vnodeInit(pInput->stopDnodeFp)) != 0) {
     dError("failed to init vnode since %s", tstrerror(code));
     goto _OVER;
   }

source/dnode/mnode/impl/src/mndConsumer.c

@@ -52,6 +52,9 @@ int32_t mndInitConsumer(SMnode *pMnode) {
       .deleteFp = (SdbDeleteFp)mndConsumerActionDelete,
   };
 
+  if (pMnode == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   mndSetMsgHandle(pMnode, TDMT_MND_TMQ_SUBSCRIBE, mndProcessSubscribeReq);
   mndSetMsgHandle(pMnode, TDMT_MND_TMQ_HB, mndProcessMqHbReq);
   mndSetMsgHandle(pMnode, TDMT_MND_TMQ_ASK_EP, mndProcessAskEpReq);
@@ -66,6 +69,9 @@ int32_t mndInitConsumer(SMnode *pMnode) {
 void mndCleanupConsumer(SMnode *pMnode) {}
 
 int32_t mndSendConsumerMsg(SMnode *pMnode, int64_t consumerId, uint16_t msgType, SRpcHandleInfo *info) {
+  if (pMnode == NULL || info == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   void   *msg  = rpcMallocCont(sizeof(int64_t));
   MND_TMQ_NULL_CHECK(msg);
@@ -88,6 +94,9 @@ END:
 }
 
 static int32_t validateTopics(STrans* pTrans, SCMSubscribeReq *subscribe, SMnode *pMnode, const char *pUser) {
+  if (pTrans == NULL || subscribe == NULL || pMnode == NULL || pUser == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqTopicObj *pTopic = NULL;
   int32_t      code = 0;
 
@@ -130,6 +139,9 @@ END:
 }
 
 static int32_t mndProcessConsumerClearMsg(SRpcMsg *pMsg) {
+  if (pMsg == NULL || pMsg->pCont == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t              code = 0;
   SMnode              *pMnode = pMsg->info.node;
   SMqConsumerClearMsg *pClearMsg = pMsg->pCont;
@@ -155,6 +167,9 @@ END:
 }
 
 static int32_t checkPrivilege(SMnode *pMnode, SMqConsumerObj *pConsumer, SMqHbRsp *rsp, char *user) {
+  if (pMnode == NULL || pConsumer == NULL || rsp == NULL || user == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   rsp->topicPrivileges = taosArrayInit(taosArrayGetSize(pConsumer->currentTopics), sizeof(STopicPrivilege));
   MND_TMQ_NULL_CHECK(rsp->topicPrivileges);
@@ -181,6 +196,9 @@ END:
 }
 
 static void storeOffsetRows(SMnode *pMnode, SMqHbReq *req, SMqConsumerObj *pConsumer){
+  if (pMnode == NULL || req == NULL || pConsumer == NULL){
+    return;
+  }
   for (int i = 0; i < taosArrayGetSize(req->topics); i++) {
     TopicOffsetRows *data = taosArrayGet(req->topics, i);
     if (data == NULL){
@@ -210,6 +228,9 @@ static void storeOffsetRows(SMnode *pMnode, SMqHbReq *req, SMqConsumerObj *pCons
 }
 
 static int32_t buildMqHbRsp(SRpcMsg *pMsg, SMqHbRsp *rsp){
+  if (pMsg == NULL || rsp == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t tlen = tSerializeSMqHbRsp(NULL, 0, rsp);
   if (tlen <= 0){
     return TSDB_CODE_TMQ_INVALID_MSG;
@@ -229,6 +250,9 @@ static int32_t buildMqHbRsp(SRpcMsg *pMsg, SMqHbRsp *rsp){
 }
 
 static int32_t mndProcessMqHbReq(SRpcMsg *pMsg) {
+  if (pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t         code = 0;
   SMnode         *pMnode = pMsg->info.node;
   SMqHbReq        req = {0};
@@ -256,6 +280,9 @@ END:
 }
 
 static int32_t addEpSetInfo(SMnode *pMnode, SMqConsumerObj *pConsumer, int32_t epoch, SMqAskEpRsp *rsp){
+  if (pMnode == NULL || pConsumer == NULL || rsp == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   taosRLockLatch(&pConsumer->lock);
 
   int32_t numOfTopics = taosArrayGetSize(pConsumer->currentTopics);
@@ -359,6 +386,9 @@ static int32_t addEpSetInfo(SMnode *pMnode, SMqConsumerObj *pConsumer, int32_t e
 }
 
 static int32_t buildAskEpRsp(SRpcMsg *pMsg, SMqAskEpRsp *rsp, int32_t serverEpoch, int64_t consumerId){
+  if (pMsg == NULL || rsp == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   // encode rsp
   int32_t tlen = sizeof(SMqRspHead) + tEncodeSMqAskEpRsp(NULL, rsp);
@@ -388,6 +418,9 @@ static int32_t buildAskEpRsp(SRpcMsg *pMsg, SMqAskEpRsp *rsp, int32_t serverEpoc
 }
 
 static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
+  if (pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMnode     *pMnode = pMsg->info.node;
   SMqAskEpReq req = {0};
   SMqAskEpRsp rsp = {0};
@@ -431,6 +464,9 @@ END:
 }
 
 int32_t mndSetConsumerDropLogs(STrans *pTrans, SMqConsumerObj *pConsumer) {
+  if (pConsumer == NULL || pTrans == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t  code = 0;
   SSdbRaw *pCommitRaw = mndConsumerActionEncode(pConsumer);
   MND_TMQ_NULL_CHECK(pCommitRaw);
@@ -445,6 +481,9 @@ END:
 }
 
 int32_t mndSetConsumerCommitLogs(STrans *pTrans, SMqConsumerObj *pConsumer) {
+  if (pConsumer == NULL || pTrans == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t  code = 0;
   SSdbRaw *pCommitRaw = mndConsumerActionEncode(pConsumer);
   MND_TMQ_NULL_CHECK(pCommitRaw);
@@ -459,6 +498,9 @@ END:
 }
 
 static void freeItem(void *param) {
+  if (param == NULL) {
+    return;
+  }
   void *pItem = *(void **)param;
   if (pItem != NULL) {
     taosMemoryFree(pItem);
@@ -475,6 +517,9 @@ if (taosArrayPush(pConsumerNew->array, &newTopicCopy) == NULL){\
 }
 
 static int32_t getTopicAddDelete(SMqConsumerObj *pExistedConsumer, SMqConsumerObj *pConsumerNew){
+  if (pExistedConsumer == NULL || pConsumerNew == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   pConsumerNew->rebNewTopics = taosArrayInit(0, sizeof(void *));
   MND_TMQ_NULL_CHECK(pConsumerNew->rebNewTopics);
@@ -528,6 +573,9 @@ END:
 }
 
 static int32_t checkAndSortTopic(SMnode *pMnode, SArray *pTopicList){
+  if (pTopicList == NULL || pMnode == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   taosArraySort(pTopicList, taosArrayCompareString);
   taosArrayRemoveDuplicate(pTopicList, taosArrayCompareString, freeItem);
 
@@ -542,6 +590,9 @@ static int32_t checkAndSortTopic(SMnode *pMnode, SArray *pTopicList){
 }
 
 static int32_t buildSubConsumer(SMnode *pMnode, SCMSubscribeReq *subscribe, SMqConsumerObj** ppConsumer){
+  if (pMnode == NULL || subscribe == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int64_t         consumerId = subscribe->consumerId;
   char           *cgroup     = subscribe->cgroup;
   SMqConsumerObj *pConsumerNew     = NULL;
@@ -581,6 +632,9 @@ END:
 }
 
 int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
+  if (pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMnode *pMnode = pMsg->info.node;
   char   *msgStr = pMsg->pCont;
   int32_t code = 0;
@@ -619,6 +673,9 @@ END:
 }
 
 SSdbRaw *mndConsumerActionEncode(SMqConsumerObj *pConsumer) {
+  if (pConsumer == NULL) {
+    return NULL;
+  }
   int32_t code = 0;
   int32_t lino = 0;
   terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -659,6 +716,9 @@ CM_ENCODE_OVER:
 }
 
 SSdbRow *mndConsumerActionDecode(SSdbRaw *pRaw) {
+  if (pRaw == NULL) {
+    return NULL;
+  }
   int32_t         code = 0;
   int32_t         lino = 0;
   SSdbRow        *pRow = NULL;
@@ -717,6 +777,9 @@ CM_DECODE_OVER:
 }
 
 static int32_t mndConsumerActionInsert(SSdb *pSdb, SMqConsumerObj *pConsumer) {
+  if (pConsumer == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   mInfo("consumer:0x%" PRIx64 " sub insert, cgroup:%s status:%d(%s) epoch:%d", pConsumer->consumerId, pConsumer->cgroup,
         pConsumer->status, mndConsumerStatusName(pConsumer->status), pConsumer->epoch);
   pConsumer->subscribeTime = pConsumer->createTime;
@@ -724,6 +787,9 @@ static int32_t mndConsumerActionInsert(SSdb *pSdb, SMqConsumerObj *pConsumer) {
 }
 
 static int32_t mndConsumerActionDelete(SSdb *pSdb, SMqConsumerObj *pConsumer) {
+  if (pConsumer == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   mInfo("consumer:0x%" PRIx64 " perform delete action, status:(%d)%s", pConsumer->consumerId, pConsumer->status,
         mndConsumerStatusName(pConsumer->status));
   tClearSMqConsumerObj(pConsumer);
@@ -744,6 +810,9 @@ static int32_t mndConsumerActionDelete(SSdb *pSdb, SMqConsumerObj *pConsumer) {
 
 // remove from topic list
 static void removeFromTopicList(SArray *topicList, const char *pTopic, int64_t consumerId, char *type) {
+  if (topicList == NULL || pTopic == NULL) {
+    return;
+  }
   int32_t size = taosArrayGetSize(topicList);
   for (int32_t i = 0; i < size; i++) {
     char *p = taosArrayGetP(topicList, i);
@@ -759,6 +828,9 @@ static void removeFromTopicList(SArray *topicList, const char *pTopic, int64_t c
 }
 
 static bool existInCurrentTopicList(const SMqConsumerObj *pConsumer, const char *pTopic) {
+  if (pConsumer == NULL || pTopic == NULL) {
+    return false;
+  }
   bool    existing = false;
   int32_t size = taosArrayGetSize(pConsumer->currentTopics);
   for (int32_t i = 0; i < size; i++) {
@@ -773,6 +845,9 @@ static bool existInCurrentTopicList(const SMqConsumerObj *pConsumer, const char
 }
 
 static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer, SMqConsumerObj *pNewConsumer) {
+  if (pOldConsumer == NULL || pNewConsumer == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   mInfo("consumer:0x%" PRIx64 " perform update action, update type:%d, subscribe-time:%" PRId64 ", createTime:%" PRId64,
         pOldConsumer->consumerId, pNewConsumer->updateType, pOldConsumer->subscribeTime, pOldConsumer->createTime);
 
@@ -857,6 +932,9 @@ static int32_t mndConsumerActionUpdate(SSdb *pSdb, SMqConsumerObj *pOldConsumer,
 }
 
 int32_t mndAcquireConsumer(SMnode *pMnode, int64_t consumerId, SMqConsumerObj** pConsumer) {
+  if (pMnode == NULL || pConsumer == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SSdb           *pSdb = pMnode->pSdb;
   *pConsumer = sdbAcquire(pSdb, SDB_CONSUMER, &consumerId);
   if (*pConsumer == NULL) {
@@ -866,11 +944,17 @@ int32_t mndAcquireConsumer(SMnode *pMnode, int64_t consumerId, SMqConsumerObj**
 }
 
 void mndReleaseConsumer(SMnode *pMnode, SMqConsumerObj *pConsumer) {
+  if (pMnode == NULL || pConsumer == NULL) {
+    return;
+  }
   SSdb *pSdb = pMnode->pSdb;
   sdbRelease(pSdb, pConsumer);
 }
 
 static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rowsCapacity) {
+  if (pReq == NULL || pShow == NULL || pBlock == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMnode         *pMnode = pReq->info.node;
   SSdb           *pSdb = pMnode->pSdb;
   int32_t         numOfRows = 0;
@@ -1021,6 +1105,7 @@ END:
 }
 
 static void mndCancelGetNextConsumer(SMnode *pMnode, void *pIter) {
+  if (pMnode == NULL || pIter == NULL) return;
   SSdb *pSdb = pMnode->pSdb;
   sdbCancelFetchByType(pSdb, pIter, SDB_CONSUMER);
 }

source/dnode/mnode/impl/src/mndSubscribe.c

@@ -43,6 +43,9 @@ static void     mndCancelGetNextSubscribe(SMnode *pMnode, void *pIter);
 static int32_t  mndCheckConsumer(SRpcMsg *pMsg, SHashObj *hash);
 
 static int32_t mndSetSubCommitLogs(STrans *pTrans, SMqSubscribeObj *pSub) {
+  if (pTrans == NULL || pSub == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t  code = 0;
   SSdbRaw *pCommitRaw = mndSubActionEncode(pSub);
   MND_TMQ_NULL_CHECK(pCommitRaw);
@@ -68,6 +71,9 @@ int32_t mndInitSubscribe(SMnode *pMnode) {
       .deleteFp = (SdbDeleteFp)mndSubActionDelete,
   };
 
+  if (pMnode == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   mndSetMsgHandle(pMnode, TDMT_VND_TMQ_SUBSCRIBE_RSP, mndTransProcessRsp);
   mndSetMsgHandle(pMnode, TDMT_VND_TMQ_DELETE_SUB_RSP, mndTransProcessRsp);
   mndSetMsgHandle(pMnode, TDMT_MND_TMQ_TIMER, mndProcessRebalanceReq);
@@ -81,6 +87,9 @@ int32_t mndInitSubscribe(SMnode *pMnode) {
 }
 
 static int32_t mndCreateSubscription(SMnode *pMnode, const SMqTopicObj *pTopic, const char *subKey, SMqSubscribeObj** pSub) {
+  if(pMnode == NULL || pTopic == NULL || subKey == NULL || pSub == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   MND_TMQ_RETURN_CHECK(tNewSubscribeObj(subKey, pSub));
   (*pSub)->dbUid = pTopic->dbUid;
@@ -99,6 +108,9 @@ END:
 
 static int32_t mndBuildSubChangeReq(void **pBuf, int32_t *pLen, SMqSubscribeObj *pSub, const SMqRebOutputVg *pRebVg,
                                     SSubplan *pPlan) {
+  if (pSub == NULL || pRebVg == NULL || pBuf == NULL || pLen == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqRebVgReq req = {0};
   int32_t     code = 0;
   SEncoder encoder = {0};
@@ -146,6 +158,9 @@ END:
 
 static int32_t mndPersistSubChangeVgReq(SMnode *pMnode, STrans *pTrans, SMqSubscribeObj *pSub,
                                         const SMqRebOutputVg *pRebVg, SSubplan *pPlan) {
+  if (pMnode == NULL || pTrans == NULL || pSub == NULL || pRebVg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   void   *buf  = NULL;
 
@@ -180,6 +195,9 @@ END:
 }
 
 static void mndSplitSubscribeKey(const char *key, char *topic, char *cgroup, bool fullName) {
+  if (key == NULL || topic == NULL || cgroup == NULL) {
+    return;
+  }
   int32_t i = 0;
   while (key[i] != TMQ_SEPARATOR_CHAR) {
     i++;
@@ -197,6 +215,9 @@ static void mndSplitSubscribeKey(const char *key, char *topic, char *cgroup, boo
 }
 
 static int32_t mndGetOrCreateRebSub(SHashObj *pHash, const char *key, SMqRebInfo **pReb) {
+  if (pHash == NULL || key == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   SMqRebInfo* pRebInfo = taosHashGet(pHash, key, strlen(key) + 1);
   if (pRebInfo == NULL) {
@@ -222,6 +243,9 @@ END:
 }
 
 static int32_t pushVgDataToHash(SArray *vgs, SHashObj *pHash, int64_t consumerId, char *key) {
+  if (vgs == NULL || pHash == NULL || key == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t         code = 0;
   SMqVgEp       **pVgEp = (SMqVgEp **)taosArrayPop(vgs);
   MND_TMQ_NULL_CHECK(pVgEp);
@@ -233,6 +257,9 @@ END:
 }
 
 static int32_t processRemovedConsumers(SMqRebOutputObj *pOutput, SHashObj *pHash, const SMqRebInputObj *pInput) {
+  if (pHash == NULL || pOutput == NULL || pInput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t numOfRemoved = taosArrayGetSize(pInput->pRebInfo->removedConsumers);
   int32_t actualRemoved = 0;
@@ -266,6 +293,9 @@ END:
 }
 
 static int32_t processNewConsumers(SMqRebOutputObj *pOutput, const SMqRebInputObj *pInput) {
+  if (pOutput == NULL || pInput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t numOfNewConsumers = taosArrayGetSize(pInput->pRebInfo->newConsumers);
 
@@ -285,6 +315,9 @@ END:
 }
 
 static int32_t processUnassignedVgroups(SMqRebOutputObj *pOutput, SHashObj *pHash) {
+  if (pOutput == NULL || pHash == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t numOfVgroups = taosArrayGetSize(pOutput->pSub->unassignedVgs);
   for (int32_t i = 0; i < numOfVgroups; i++) {
@@ -296,6 +329,9 @@ END:
 
 static int32_t processModifiedConsumers(SMqRebOutputObj *pOutput, SHashObj *pHash, int32_t minVgCnt,
                                      int32_t remainderVgCnt) {
+  if (pOutput == NULL || pHash == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t cnt = 0;
   void   *pIter = NULL;
@@ -328,6 +364,9 @@ END:
 }
 
 static int32_t processRemoveAddVgs(SMnode *pMnode, SMqRebOutputObj *pOutput) {
+  if (pMnode == NULL || pOutput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t totalVgNum = 0;
   SVgObj *pVgroup = NULL;
@@ -403,6 +442,9 @@ END:
 }
 
 static int32_t processSubOffsetRows(SMnode *pMnode, const SMqRebInputObj *pInput, SMqRebOutputObj *pOutput) {
+  if (pMnode == NULL || pInput == NULL || pOutput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqSubscribeObj *pSub = NULL;
   int32_t          code = mndAcquireSubscribeByKey(pMnode, pInput->pRebInfo->key, &pSub);  // put all offset rows
   if( code != 0){
@@ -465,6 +507,7 @@ END:
 }
 
 static void printRebalanceLog(SMqRebOutputObj *pOutput) {
+  if (pOutput == NULL) return;
   mInfo("sub:%s mq rebalance calculation completed, re-balanced vg", pOutput->pSub->key);
   for (int32_t i = 0; i < taosArrayGetSize(pOutput->rebVgs); i++) {
     SMqRebOutputVg *pOutputRebVg = taosArrayGet(pOutput->rebVgs, i);
@@ -492,6 +535,9 @@ static void printRebalanceLog(SMqRebOutputObj *pOutput) {
 
 static void calcVgroupsCnt(const SMqRebInputObj *pInput, int32_t totalVgNum, const char *pSubKey, int32_t *minVgCnt,
                            int32_t *remainderVgCnt) {
+  if (pInput == NULL || pSubKey == NULL || minVgCnt == NULL || remainderVgCnt == NULL) {
+    return;
+  }
   int32_t numOfRemoved = taosArrayGetSize(pInput->pRebInfo->removedConsumers);
   int32_t numOfAdded = taosArrayGetSize(pInput->pRebInfo->newConsumers);
   int32_t numOfFinal = pInput->oldConsumerNum + numOfAdded - numOfRemoved;
@@ -509,6 +555,9 @@ static void calcVgroupsCnt(const SMqRebInputObj *pInput, int32_t totalVgNum, con
 }
 
 static int32_t assignVgroups(SMqRebOutputObj *pOutput, SHashObj *pHash, int32_t minVgCnt) {
+  if (pOutput == NULL || pHash == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqRebOutputVg *pRebVg = NULL;
   void           *pAssignIter = NULL;
   void           *pIter = NULL;
@@ -580,6 +629,9 @@ END:
 }
 
 static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqRebOutputObj *pOutput) {
+  if (pMnode == NULL || pInput == NULL || pOutput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t     totalVgNum = processRemoveAddVgs(pMnode, pOutput);
   if (totalVgNum < 0){
     return totalVgNum;
@@ -605,6 +657,9 @@ END:
 }
 
 static int32_t presistConsumerByType(STrans *pTrans, SArray *consumers, int8_t type, char *cgroup, char *topic) {
+  if (pTrans == NULL || consumers == NULL || cgroup == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t         code = 0;
   SMqConsumerObj *pConsumerNew = NULL;
   int32_t         consumerNum = taosArrayGetSize(consumers);
@@ -623,6 +678,9 @@ END:
 }
 
 static int32_t mndPresistConsumer(STrans *pTrans, const SMqRebOutputObj *pOutput, char *cgroup, char *topic) {
+  if (pTrans == NULL || pOutput == NULL || cgroup == NULL || topic == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   MND_TMQ_RETURN_CHECK(presistConsumerByType(pTrans, pOutput->modifyConsumers, CONSUMER_UPDATE_REB, cgroup, NULL));
   MND_TMQ_RETURN_CHECK(presistConsumerByType(pTrans, pOutput->newConsumers, CONSUMER_ADD_REB, cgroup, topic));
@@ -632,6 +690,9 @@ END:
 }
 
 static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOutputObj *pOutput) {
+  if (pMnode == NULL || pMsg == NULL || pOutput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   struct SSubplan *pPlan = NULL;
   int32_t          code = 0;
   STrans          *pTrans = NULL;
@@ -682,6 +743,7 @@ END:
 }
 
 static void freeRebalanceItem(void *param) {
+  if (param == NULL) return;
   SMqRebInfo *pInfo = param;
   taosArrayDestroy(pInfo->newConsumers);
   taosArrayDestroy(pInfo->removedConsumers);
@@ -689,6 +751,9 @@ static void freeRebalanceItem(void *param) {
 
 // type = 0 remove  type = 1 add
 static int32_t buildRebInfo(SHashObj *rebSubHash, SArray *topicList, int8_t type, char *group, int64_t consumerId) {
+  if (rebSubHash == NULL || topicList == NULL || group == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t topicNum = taosArrayGetSize(topicList);
   for (int32_t i = 0; i < topicNum; i++) {
@@ -709,6 +774,9 @@ END:
 }
 
 static void checkForVgroupSplit(SMnode *pMnode, SMqConsumerObj *pConsumer, SHashObj *rebSubHash) {
+  if (pMnode == NULL || pConsumer == NULL || rebSubHash == NULL) {
+    return;
+  }
   int32_t newTopicNum = taosArrayGetSize(pConsumer->currentTopics);
   for (int32_t i = 0; i < newTopicNum; i++) {
     char            *topic = taosArrayGetP(pConsumer->currentTopics, i);
@@ -754,6 +822,9 @@ static void checkForVgroupSplit(SMnode *pMnode, SMqConsumerObj *pConsumer, SHash
 }
 
 static int32_t mndCheckConsumer(SRpcMsg *pMsg, SHashObj *rebSubHash) {
+  if (pMsg == NULL || rebSubHash == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMnode         *pMnode = pMsg->info.node;
   SSdb           *pSdb = pMnode->pSdb;
   SMqConsumerObj *pConsumer = NULL;
@@ -818,6 +889,9 @@ void mndRebCntDec() {
 }
 
 static void clearRebOutput(SMqRebOutputObj *rebOutput) {
+  if (rebOutput == NULL) {
+    return;
+  }
   taosArrayDestroy(rebOutput->newConsumers);
   taosArrayDestroy(rebOutput->modifyConsumers);
   taosArrayDestroy(rebOutput->removedConsumers);
@@ -827,6 +901,9 @@ static void clearRebOutput(SMqRebOutputObj *rebOutput) {
 }
 
 static int32_t initRebOutput(SMqRebOutputObj *rebOutput) {
+  if (rebOutput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   rebOutput->newConsumers = taosArrayInit(0, sizeof(int64_t));
   MND_TMQ_NULL_CHECK(rebOutput->newConsumers);
@@ -845,6 +922,9 @@ END:
 
 // This function only works when there are dirty consumers
 static int32_t checkConsumer(SMnode *pMnode, SMqSubscribeObj *pSub) {
+  if (pMnode == NULL || pSub == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   void   *pIter = NULL;
   while (1) {
@@ -871,6 +951,9 @@ END:
 }
 
 static int32_t buildRebOutput(SMnode *pMnode, SMqRebInputObj *rebInput, SMqRebOutputObj *rebOutput) {
+  if (pMnode == NULL || rebInput == NULL || rebOutput == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   const char      *key = rebInput->pRebInfo->key;
   SMqSubscribeObj *pSub = NULL;
   int32_t          code = mndAcquireSubscribeByKey(pMnode, key, &pSub);
@@ -922,6 +1005,9 @@ END:
 }
 
 static int32_t mndProcessRebalanceReq(SRpcMsg *pMsg) {
+  if (pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int     code = 0;
   void   *pIter = NULL;
   SMnode *pMnode = pMsg->info.node;
@@ -986,6 +1072,9 @@ END:
 }
 
 static int32_t sendDeleteSubToVnode(SMnode *pMnode, SMqSubscribeObj *pSub, STrans *pTrans) {
+  if (pMnode == NULL || pSub == NULL || pTrans == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void   *pIter = NULL;
   SVgObj *pVgObj = NULL;
   int32_t code = 0;
@@ -1024,6 +1113,9 @@ END:
 }
 
 static int32_t mndCheckConsumerByGroup(SMnode *pMnode, STrans *pTrans, char *cgroup, char *topic) {
+  if (pMnode == NULL || pTrans == NULL || cgroup == NULL || topic == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void           *pIter = NULL;
   SMqConsumerObj *pConsumer = NULL;
   int             code = 0;
@@ -1056,6 +1148,9 @@ END:
 }
 
 static int32_t mndProcessDropCgroupReq(SRpcMsg *pMsg) {
+  if (pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMnode         *pMnode = pMsg->info.node;
   SMDropCgroupReq dropReq = {0};
   STrans         *pTrans = NULL;
@@ -1109,6 +1204,9 @@ END:
 void mndCleanupSubscribe(SMnode *pMnode) {}
 
 static SSdbRaw *mndSubActionEncode(SMqSubscribeObj *pSub) {
+  if (pSub == NULL) {
+    return NULL;
+  }
   int32_t code = 0;
   int32_t lino = 0;
   terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -1149,6 +1247,9 @@ SUB_ENCODE_OVER:
 }
 
 static SSdbRow *mndSubActionDecode(SSdbRaw *pRaw) {
+  if (pRaw == NULL) {
+    return NULL;
+  }
   int32_t code = 0;
   int32_t lino = 0;
   terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -1218,17 +1319,18 @@ SUB_DECODE_OVER:
 }
 
 static int32_t mndSubActionInsert(SSdb *pSdb, SMqSubscribeObj *pSub) {
-  mTrace("subscribe:%s, perform insert action", pSub->key);
+  mTrace("subscribe:%s, perform insert action", pSub != NULL ? pSub->key : "null");
   return 0;
 }
 
 static int32_t mndSubActionDelete(SSdb *pSdb, SMqSubscribeObj *pSub) {
-  mTrace("subscribe:%s, perform delete action", pSub->key);
+  mTrace("subscribe:%s, perform delete action", pSub != NULL ? pSub->key : "null");
   tDeleteSubscribeObj(pSub);
   return 0;
 }
 
 static int32_t mndSubActionUpdate(SSdb *pSdb, SMqSubscribeObj *pOldSub, SMqSubscribeObj *pNewSub) {
+  if (pOldSub == NULL || pNewSub == NULL) return -1;
   mTrace("subscribe:%s, perform update action", pOldSub->key);
   taosWLockLatch(&pOldSub->lock);
 
@@ -1249,6 +1351,9 @@ static int32_t mndSubActionUpdate(SSdb *pSdb, SMqSubscribeObj *pOldSub, SMqSubsc
 }
 
 int32_t mndAcquireSubscribeByKey(SMnode *pMnode, const char *key, SMqSubscribeObj** pSub) {
+  if (pMnode == NULL || key == NULL || pSub == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   SSdb            *pSdb = pMnode->pSdb;
   *pSub = sdbAcquire(pSdb, SDB_SUBSCRIBE, key);
   if (*pSub == NULL) {
@@ -1258,6 +1363,7 @@ int32_t mndAcquireSubscribeByKey(SMnode *pMnode, const char *key, SMqSubscribeOb
 }
 
 int32_t mndGetGroupNumByTopic(SMnode *pMnode, const char *topicName) {
+  if (pMnode == NULL || topicName == NULL) return 0;
   int32_t num = 0;
   SSdb   *pSdb = pMnode->pSdb;
 
@@ -1283,11 +1389,13 @@ int32_t mndGetGroupNumByTopic(SMnode *pMnode, const char *topicName) {
 }
 
 void mndReleaseSubscribe(SMnode *pMnode, SMqSubscribeObj *pSub) {
+  if (pMnode == NULL || pSub == NULL) return;
   SSdb *pSdb = pMnode->pSdb;
   sdbRelease(pSdb, pSub);
 }
 
 int32_t mndSetDropSubCommitLogs(SMnode *pMnode, STrans *pTrans, SMqSubscribeObj *pSub) {
+  if (pMnode == NULL || pTrans == NULL || pSub == NULL) return TSDB_CODE_INVALID_PARA;
   int32_t  code = 0;
   SSdbRaw *pCommitRaw = mndSubActionEncode(pSub);
   MND_TMQ_NULL_CHECK(pCommitRaw);
@@ -1302,6 +1410,7 @@ END:
 }
 
 int32_t mndDropSubByTopic(SMnode *pMnode, STrans *pTrans, const char *topicName) {
+  if (pMnode == NULL || pTrans == NULL || topicName == NULL) return TSDB_CODE_INVALID_PARA;
   SSdb            *pSdb = pMnode->pSdb;
   int32_t          code = 0;
   void            *pIter = NULL;
@@ -1337,6 +1446,9 @@ END:
 
 static int32_t buildResult(SSDataBlock *pBlock, int32_t *numOfRows, int64_t consumerId, const char* user, const char* fqdn,
                            const char *topic, const char *cgroup, SArray *vgs, SArray *offsetRows) {
+  if (pBlock == NULL || numOfRows == NULL || topic == NULL || cgroup == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t sz = taosArrayGetSize(vgs);
   for (int32_t j = 0; j < sz; j++) {
@@ -1424,6 +1536,9 @@ END:
 }
 
 int32_t mndRetrieveSubscribe(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rowsCapacity) {
+  if (pReq == NULL || pShow == NULL || pBlock == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMnode          *pMnode = pReq->info.node;
   SSdb            *pSdb = pMnode->pSdb;
   int32_t          numOfRows = 0;
@@ -1485,10 +1600,16 @@ int32_t mndRetrieveSubscribe(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock
   return numOfRows;
 
 END:
+  taosRUnLockLatch(&pSub->lock);
+  sdbRelease(pSdb, pSub);
+
   return code;
 }
 
 void mndCancelGetNextSubscribe(SMnode *pMnode, void *pIter) {
+  if (pMnode == NULL) {
+    return;
+  }
   SSdb *pSdb = pMnode->pSdb;
   sdbCancelFetchByType(pSdb, pIter, SDB_SUBSCRIBE);
 }

source/dnode/mnode/impl/src/mndTopic.c

@@ -55,6 +55,9 @@ int32_t mndInitTopic(SMnode *pMnode) {
       .deleteFp = (SdbDeleteFp)mndTopicActionDelete,
   };
 
+  if (pMnode == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   mndSetMsgHandle(pMnode, TDMT_MND_TMQ_CREATE_TOPIC, mndProcessCreateTopicReq);
   mndSetMsgHandle(pMnode, TDMT_MND_TMQ_DROP_TOPIC, mndProcessDropTopicReq);
   mndSetMsgHandle(pMnode, TDMT_VND_TMQ_ADD_CHECKINFO_RSP, mndTransProcessRsp);
@@ -81,6 +84,9 @@ void mndTopicGetShowName(const char* fullTopic, char* topic) {
 }
 
 SSdbRaw *mndTopicActionEncode(SMqTopicObj *pTopic) {
+  if (pTopic == NULL) {
+    return NULL;
+  }
   int32_t code = 0;
   int32_t lino = 0;
   terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -172,6 +178,7 @@ TOPIC_ENCODE_OVER:
 }
 
 SSdbRow *mndTopicActionDecode(SSdbRaw *pRaw) {
+  if (pRaw == NULL) return NULL;
   int32_t code = 0;
   int32_t lino = 0;
   terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -193,7 +200,7 @@ SSdbRow *mndTopicActionDecode(SSdbRaw *pRaw) {
   pTopic = sdbGetRowObj(pRow);
   if (pTopic == NULL) goto TOPIC_DECODE_OVER;
 
-  int32_t len;
+  int32_t len = 0;
   int32_t dataPos = 0;
   SDB_GET_BINARY(pRaw, dataPos, pTopic->name, TSDB_TOPIC_FNAME_LEN, TOPIC_DECODE_OVER);
   SDB_GET_BINARY(pRaw, dataPos, pTopic->db, TSDB_DB_FNAME_LEN, TOPIC_DECODE_OVER);
@@ -292,11 +299,12 @@ TOPIC_DECODE_OVER:
 }
 
 static int32_t mndTopicActionInsert(SSdb *pSdb, SMqTopicObj *pTopic) {
-  mTrace("topic:%s perform insert action", pTopic->name);
+  mTrace("topic:%s perform insert action", pTopic != NULL ? pTopic->name : "null");
   return 0;
 }
 
 static int32_t mndTopicActionDelete(SSdb *pSdb, SMqTopicObj *pTopic) {
+  if (pTopic == NULL) return 0;
   mTrace("topic:%s perform delete action", pTopic->name);
   taosMemoryFreeClear(pTopic->sql);
   taosMemoryFreeClear(pTopic->ast);
@@ -307,6 +315,7 @@ static int32_t mndTopicActionDelete(SSdb *pSdb, SMqTopicObj *pTopic) {
 }
 
 static int32_t mndTopicActionUpdate(SSdb *pSdb, SMqTopicObj *pOldTopic, SMqTopicObj *pNewTopic) {
+  if (pOldTopic == NULL || pNewTopic == NULL) return 0;
   mTrace("topic:%s perform update action", pOldTopic->name);
   (void)atomic_exchange_64(&pOldTopic->updateTime, pNewTopic->updateTime);
   (void)atomic_exchange_32(&pOldTopic->version, pNewTopic->version);
@@ -315,6 +324,9 @@ static int32_t mndTopicActionUpdate(SSdb *pSdb, SMqTopicObj *pOldTopic, SMqTopic
 }
 
 int32_t mndAcquireTopic(SMnode *pMnode, const char *topicName, SMqTopicObj **pTopic) {
+  if (pMnode == NULL || topicName == NULL || pTopic == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   SSdb        *pSdb = pMnode->pSdb;
   *pTopic = sdbAcquire(pSdb, SDB_TOPIC, topicName);
   if (*pTopic == NULL) {
@@ -324,11 +336,13 @@ int32_t mndAcquireTopic(SMnode *pMnode, const char *topicName, SMqTopicObj **pTo
 }
 
 void mndReleaseTopic(SMnode *pMnode, SMqTopicObj *pTopic) {
+  if (pMnode == NULL) return;
   SSdb *pSdb = pMnode->pSdb;
   sdbRelease(pSdb, pTopic);
 }
 
 static int32_t mndCheckCreateTopicReq(SCMCreateTopicReq *pCreate) {
+  if (pCreate == NULL) return TSDB_CODE_INVALID_PARA;
   if (pCreate->sql == NULL) return TSDB_CODE_MND_INVALID_TOPIC;
 
   if (pCreate->subType == TOPIC_SUB_TYPE__COLUMN) {
@@ -343,6 +357,7 @@ static int32_t mndCheckCreateTopicReq(SCMCreateTopicReq *pCreate) {
 }
 
 static int32_t extractTopicTbInfo(SNode *pAst, SMqTopicObj *pTopic) {
+  if (pAst == NULL || pTopic == NULL) return TSDB_CODE_INVALID_PARA;
   SNodeList *pNodeList = NULL;
   int32_t   code = 0;
   MND_TMQ_RETURN_CHECK(nodesCollectColumns((SSelectStmt *)pAst, SQL_CLAUSE_FROM, NULL, COLLECT_COL_TYPE_ALL, &pNodeList));
@@ -367,6 +382,7 @@ END:
 }
 
 static int32_t sendCheckInfoToVnode(STrans *pTrans, SMnode *pMnode, SMqTopicObj *topicObj){
+  if (pTrans == NULL || pMnode == NULL || topicObj == NULL) return TSDB_CODE_INVALID_PARA;
   STqCheckInfo info = {0};
   (void)memcpy(info.topic, topicObj->name, TSDB_TOPIC_FNAME_LEN);
   info.ntbUid = topicObj->ntbUid;
@@ -388,7 +404,7 @@ static int32_t sendCheckInfoToVnode(STrans *pTrans, SMnode *pMnode, SMqTopicObj
     }
 
     // encoder check alter info
-    int32_t len;
+    int32_t len = 0;
     tEncodeSize(tEncodeSTqCheckInfo, &info, len, code);
     if (code != 0) {
       code = TSDB_CODE_OUT_OF_MEMORY;
@@ -426,6 +442,7 @@ END:
 
 static int32_t mndCreateTopic(SMnode *pMnode, SRpcMsg *pReq, SCMCreateTopicReq *pCreate, SDbObj *pDb,
                               const char *userName) {
+  if (pMnode == NULL || pReq == NULL || pCreate == NULL || pDb == NULL || userName == NULL) return TSDB_CODE_INVALID_PARA;
   mInfo("start to create topic:%s", pCreate->name);
   STrans *pTrans = NULL;
   int32_t code = 0;
@@ -519,6 +536,9 @@ END:
 }
 
 static int32_t mndProcessCreateTopicReq(SRpcMsg *pReq) {
+  if (pReq == NULL || pReq->contLen <= 0) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   SMnode           *pMnode = pReq->info.node;
   int32_t           code = TDB_CODE_SUCCESS;
   SMqTopicObj      *pTopic = NULL;
@@ -596,6 +616,9 @@ END:
 }
 
 static int32_t mndDropTopic(SMnode *pMnode, STrans *pTrans, SRpcMsg *pReq, SMqTopicObj *pTopic) {
+  if (pMnode == NULL || pTrans == NULL || pReq == NULL || pTopic == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t code = 0;
   SSdbRaw *pCommitRaw = NULL;
   MND_TMQ_RETURN_CHECK(mndUserRemoveTopic(pMnode, pTrans, pTopic->name));
@@ -614,6 +637,9 @@ END:
 }
 
 bool checkTopic(SArray *topics, char *topicName){
+  if (topics == NULL || topicName == NULL) {
+    return false;
+  }
   int32_t sz = taosArrayGetSize(topics);
   for (int32_t i = 0; i < sz; i++) {
     char *name = taosArrayGetP(topics, i);
@@ -625,6 +651,9 @@ bool checkTopic(SArray *topics, char *topicName){
 }
 
 static int32_t mndCheckConsumerByTopic(SMnode *pMnode, STrans *pTrans, char *topicName){
+  if (pMnode == NULL || pTrans == NULL || topicName == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t         code = 0;
   SSdb           *pSdb    = pMnode->pSdb;
   void           *pIter = NULL;
@@ -653,6 +682,9 @@ END:
 }
 
 static int32_t mndDropCheckInfoByTopic(SMnode *pMnode, STrans *pTrans, SMqTopicObj *pTopic){
+  if (pMnode == NULL || pTrans == NULL || pTopic == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   // broadcast to all vnode
   void   *pIter = NULL;
   SVgObj *pVgroup = NULL;
@@ -693,6 +725,9 @@ END:
 }
 
 static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) {
+  if (pReq == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   SMnode        *pMnode  = pReq->info.node;
   SMDropTopicReq dropReq = {0};
   int32_t        code = 0;
@@ -756,6 +791,9 @@ END:
 }
 
 int32_t mndGetNumOfTopics(SMnode *pMnode, char *dbName, int32_t *pNumOfTopics) {
+  if (pMnode == NULL || dbName == NULL || pNumOfTopics == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   *pNumOfTopics = 0;
 
   SSdb   *pSdb = pMnode->pSdb;
@@ -786,6 +824,9 @@ int32_t mndGetNumOfTopics(SMnode *pMnode, char *dbName, int32_t *pNumOfTopics) {
 }
 
 static void schemaToJson(SSchema *schema, int32_t nCols, char *schemaJson){
+  if (schema == NULL || schemaJson == NULL) {
+    return;
+  }
   char*   string = NULL;
   int32_t code = 0;
   cJSON* columns = cJSON_CreateArray();
@@ -838,6 +879,9 @@ END:
 }
 
 static int32_t mndRetrieveTopic(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rowsCapacity) {
+  if (pReq == NULL || pShow == NULL || pBlock == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   SMnode      *pMnode = pReq->info.node;
   SSdb        *pSdb = pMnode->pSdb;
   int32_t      numOfRows = 0;
@@ -945,11 +989,15 @@ END:
 }
 
 static void mndCancelGetNextTopic(SMnode *pMnode, void *pIter) {
+  if (pMnode == NULL) return;
   SSdb *pSdb = pMnode->pSdb;
   sdbCancelFetchByType(pSdb, pIter, SDB_TOPIC);
 }
 
 bool mndTopicExistsForDb(SMnode *pMnode, SDbObj *pDb) {
+  if (pMnode == NULL || pDb == NULL) {
+    return false;
+  }
   SSdb        *pSdb = pMnode->pSdb;
   void        *pIter = NULL;
   SMqTopicObj *pTopic = NULL;

source/dnode/vnode/inc/vnode.h

@@ -51,7 +51,7 @@ extern const SVnodeCfg vnodeCfgDefault;
 
 typedef void (*StopDnodeFp)();
 
-int32_t vnodeInit(int32_t nthreads, StopDnodeFp stopDnodeFp);
+int32_t vnodeInit(StopDnodeFp stopDnodeFp);
 void    vnodeCleanup();
 int32_t vnodeCreate(const char *path, SVnodeCfg *pCfg, int32_t diskPrimary, STfs *pTfs);
 bool    vnodeShouldRemoveWal(SVnode *pVnode);

source/dnode/vnode/src/inc/tq.h

@@ -127,11 +127,11 @@ void    tqPushEmptyDataRsp(STqHandle* pHandle, int32_t vgId);
 int32_t tqMetaOpen(STQ* pTq);
 void    tqMetaClose(STQ* pTq);
 int32_t tqMetaSaveHandle(STQ* pTq, const char* key, const STqHandle* pHandle);
-int32_t tqMetaSaveInfo(STQ* pTq, TTB* ttb, const void* key, int32_t kLen, const void* value, int32_t vLen);
-int32_t tqMetaDeleteInfo(STQ* pTq, TTB* ttb, const void* key, int32_t kLen);
+int32_t tqMetaSaveInfo(STQ* pTq, TTB* ttb, const void* key, uint32_t kLen, const void* value, uint32_t vLen);
+int32_t tqMetaDeleteInfo(STQ* pTq, TTB* ttb, const void* key, uint32_t kLen);
 int32_t tqMetaCreateHandle(STQ* pTq, SMqRebVgReq* req, STqHandle* handle);
-int32_t tqMetaDecodeCheckInfo(STqCheckInfo *info, void *pVal, int32_t vLen);
-int32_t tqMetaDecodeOffsetInfo(STqOffset *info, void *pVal, int32_t vLen);
+int32_t tqMetaDecodeCheckInfo(STqCheckInfo *info, void *pVal, uint32_t vLen);
+int32_t tqMetaDecodeOffsetInfo(STqOffset *info, void *pVal, uint32_t vLen);
 int32_t tqMetaSaveOffset(STQ* pTq, STqOffset* pOffset);
 int32_t tqMetaGetHandle(STQ* pTq, const char* key, STqHandle** pHandle);
 int32_t tqMetaGetOffset(STQ* pTq, const char* subkey, STqOffset** pOffset);

source/dnode/vnode/src/inc/tsdb.h

@@ -1083,9 +1083,6 @@ void tsdbRemoveFile(const char *path);
     }                                    \
   } while (0)
 
-int32_t tsdbInit();
-void    tsdbCleanUp();
-
 #ifdef __cplusplus
 }
 #endif

source/dnode/vnode/src/inc/vnd.h

@@ -55,16 +55,32 @@ typedef enum {
   EVA_PRIORITY_LOW,
 } EVAPriority;
 
-int32_t vnodeAsyncOpen(int32_t numOfThreads);
+typedef enum {
+  EVA_TASK_COMMIT = 1,
+  EVA_TASK_MERGE,
+  EVA_TASK_COMPACT,
+  EVA_TASK_RETENTION,
+} EVATaskT;
+
+#define COMMIT_TASK_ASYNC    1
+#define MERGE_TASK_ASYNC     2
+#define COMPACT_TASK_ASYNC   3
+#define RETENTION_TASK_ASYNC 4
+
+int32_t vnodeAsyncOpen();
 void    vnodeAsyncClose();
 int32_t vnodeAChannelInit(int64_t async, SVAChannelID* channelID);
 int32_t vnodeAChannelDestroy(SVAChannelID* channelID, bool waitRunning);
-int32_t vnodeAsync(SVAChannelID* channelID, EVAPriority priority, int32_t (*execute)(void*), void (*complete)(void*),
-                   void* arg, SVATaskID* taskID);
+int32_t vnodeAsync(int64_t async, EVAPriority priority, int32_t (*execute)(void*), void (*complete)(void*), void* arg,
+                   SVATaskID* taskID);
+int32_t vnodeAsyncC(SVAChannelID* channelID, EVAPriority priority, int32_t (*execute)(void*), void (*complete)(void*),
+                    void* arg, SVATaskID* taskID);
 void    vnodeAWait(SVATaskID* taskID);
 int32_t vnodeACancel(SVATaskID* taskID);
 int32_t vnodeAsyncSetWorkers(int64_t async, int32_t numWorkers);
 
+const char* vnodeGetATaskName(EVATaskT task);
+
 // vnodeBufPool.c
 typedef struct SVBufPoolNode SVBufPoolNode;
 struct SVBufPoolNode {

source/dnode/vnode/src/inc/vnodeInt.h

@@ -479,8 +479,7 @@ struct SVnode {
   SVBufPool*    onRecycle;
 
   // commit variables
-  SVAChannelID commitChannel;
-  SVATaskID    commitTask;
+  SVATaskID commitTask;
 
   SMeta*        pMeta;
   SSma*         pSma;

source/dnode/vnode/src/tq/tq.c

@@ -25,11 +25,12 @@
 // 2: wait to be inited or cleanup
 static int32_t tqInitialize(STQ* pTq);
 
-static FORCE_INLINE bool tqIsHandleExec(STqHandle* pHandle) { return TMQ_HANDLE_STATUS_EXEC == pHandle->status; }
-static FORCE_INLINE void tqSetHandleExec(STqHandle* pHandle) { pHandle->status = TMQ_HANDLE_STATUS_EXEC; }
-static FORCE_INLINE void tqSetHandleIdle(STqHandle* pHandle) { pHandle->status = TMQ_HANDLE_STATUS_IDLE; }
+static FORCE_INLINE bool tqIsHandleExec(STqHandle* pHandle) { return pHandle != NULL ? TMQ_HANDLE_STATUS_EXEC == pHandle->status : true; }
+static FORCE_INLINE void tqSetHandleExec(STqHandle* pHandle) { if (pHandle != NULL) pHandle->status = TMQ_HANDLE_STATUS_EXEC; }
+static FORCE_INLINE void tqSetHandleIdle(STqHandle* pHandle) { if (pHandle != NULL) pHandle->status = TMQ_HANDLE_STATUS_IDLE; }
 
 void tqDestroyTqHandle(void* data) {
+  if (data == NULL) return;
   STqHandle* pData = (STqHandle*)data;
   qDestroyTask(pData->execHandle.task);
 
@@ -59,11 +60,17 @@ void tqDestroyTqHandle(void* data) {
 }
 
 static bool tqOffsetEqual(const STqOffset* pLeft, const STqOffset* pRight) {
+  if (pLeft == NULL || pRight == NULL) {
+    return false;
+  }
   return pLeft->val.type == TMQ_OFFSET__LOG && pRight->val.type == TMQ_OFFSET__LOG &&
          pLeft->val.version == pRight->val.version;
 }
 
 int32_t tqOpen(const char* path, SVnode* pVnode) {
+  if (path == NULL || pVnode == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   STQ* pTq = taosMemoryCalloc(1, sizeof(STQ));
   if (pTq == NULL) {
     return terrno;
@@ -104,6 +111,9 @@ int32_t tqOpen(const char* path, SVnode* pVnode) {
 }
 
 int32_t tqInitialize(STQ* pTq) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t vgId = TD_VID(pTq->pVnode);
   int32_t code = streamMetaOpen(pTq->path, pTq, tqBuildStreamTask, tqExpandStreamTask, vgId, -1,
                                 tqStartTaskCompleteCallback, &pTq->pStreamMeta);
@@ -157,6 +167,9 @@ void tqNotifyClose(STQ* pTq) {
 }
 
 void tqPushEmptyDataRsp(STqHandle* pHandle, int32_t vgId) {
+  if (pHandle == NULL) {
+    return;
+  }
   int32_t    code = 0;
   SMqPollReq req = {0};
   code = tDeserializeSMqPollReq(pHandle->msg->pCont, pHandle->msg->contLen, &req);
@@ -186,6 +199,9 @@ void tqPushEmptyDataRsp(STqHandle* pHandle, int32_t vgId) {
 
 int32_t tqSendDataRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* pReq, const SMqDataRsp* pRsp, int32_t type,
                       int32_t vgId) {
+  if (pHandle == NULL || pMsg == NULL || pReq == NULL || pRsp == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int64_t sver = 0, ever = 0;
   walReaderValidVersionRange(pHandle->execHandle.pTqReader->pWalReader, &sver, &ever);
 
@@ -201,6 +217,9 @@ int32_t tqSendDataRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq*
 }
 
 int32_t tqProcessOffsetCommitReq(STQ* pTq, int64_t sversion, char* msg, int32_t msgLen) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqVgOffset vgOffset = {0};
   int32_t     vgId = TD_VID(pTq->pVnode);
 
@@ -243,7 +262,7 @@ int32_t tqProcessOffsetCommitReq(STQ* pTq, int64_t sversion, char* msg, int32_t
   }
 
   if (tqMetaSaveInfo(pTq, pTq->pOffsetStore, pOffset->subKey, strlen(pOffset->subKey), msg,
-                     msgLen - sizeof(vgOffset.consumerId)) < 0) {
+                     msgLen >= sizeof(vgOffset.consumerId) ? msgLen - sizeof(vgOffset.consumerId) : 0) < 0) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
     return -1;
   }
@@ -255,6 +274,9 @@ end:
 }
 
 int32_t tqProcessSeekReq(STQ* pTq, SRpcMsg* pMsg) {
+  if (pTq == NULL || pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqSeekReq req = {0};
   int32_t    vgId = TD_VID(pTq->pVnode);
   SRpcMsg    rsp = {.info = pMsg->info};
@@ -297,6 +319,9 @@ end:
 }
 
 int32_t tqCheckColModifiable(STQ* pTq, int64_t tbUid, int32_t colId) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void* pIter = NULL;
 
   while (1) {
@@ -327,6 +352,9 @@ int32_t tqCheckColModifiable(STQ* pTq, int64_t tbUid, int32_t colId) {
 }
 
 int32_t tqProcessPollPush(STQ* pTq, SRpcMsg* pMsg) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t vgId = TD_VID(pTq->pVnode);
   taosWLockLatch(&pTq->lock);
   if (taosHashGetSize(pTq->pPushMgr) > 0) {
@@ -362,6 +390,9 @@ int32_t tqProcessPollPush(STQ* pTq, SRpcMsg* pMsg) {
 }
 
 int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg) {
+  if (pTq == NULL || pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqPollReq req = {0};
   int        code = tDeserializeSMqPollReq(pMsg->pCont, pMsg->contLen, &req);
   if (code < 0) {
@@ -439,6 +470,9 @@ END:
 }
 
 int32_t tqProcessVgCommittedInfoReq(STQ* pTq, SRpcMsg* pMsg) {
+  if (pTq == NULL || pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void*   data = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
   int32_t len = pMsg->contLen - sizeof(SMsgHead);
 
@@ -485,6 +519,9 @@ int32_t tqProcessVgCommittedInfoReq(STQ* pTq, SRpcMsg* pMsg) {
 }
 
 int32_t tqProcessVgWalInfoReq(STQ* pTq, SRpcMsg* pMsg) {
+  if (pTq == NULL || pMsg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t    code = 0;
   SMqPollReq req = {0};
   if (tDeserializeSMqPollReq(pMsg->pCont, pMsg->contLen, &req) < 0) {
@@ -570,6 +607,9 @@ END:
 }
 
 int32_t tqProcessDeleteSubReq(STQ* pTq, int64_t sversion, char* msg, int32_t msgLen) {
+  if (pTq == NULL || msg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMqVDeleteReq* pReq = (SMqVDeleteReq*)msg;
   int32_t        vgId = TD_VID(pTq->pVnode);
 
@@ -616,8 +656,11 @@ int32_t tqProcessDeleteSubReq(STQ* pTq, int64_t sversion, char* msg, int32_t msg
 }
 
 int32_t tqProcessAddCheckInfoReq(STQ* pTq, int64_t sversion, char* msg, int32_t msgLen) {
+  if (pTq == NULL || msg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   STqCheckInfo info = {0};
-  int32_t      code = tqMetaDecodeCheckInfo(&info, msg, msgLen);
+  int32_t      code = tqMetaDecodeCheckInfo(&info, msg, msgLen >= 0 ? msgLen : 0);
   if (code != 0) {
     return code;
   }
@@ -628,10 +671,13 @@ int32_t tqProcessAddCheckInfoReq(STQ* pTq, int64_t sversion, char* msg, int32_t
     return code;
   }
 
-  return tqMetaSaveInfo(pTq, pTq->pCheckStore, info.topic, strlen(info.topic), msg, msgLen);
+  return tqMetaSaveInfo(pTq, pTq->pCheckStore, info.topic, strlen(info.topic), msg, msgLen >= 0 ? msgLen : 0);
 }
 
 int32_t tqProcessDelCheckInfoReq(STQ* pTq, int64_t sversion, char* msg, int32_t msgLen) {
+  if (pTq == NULL || msg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   if (taosHashRemove(pTq->pCheckInfo, msg, strlen(msg)) < 0) {
     return TSDB_CODE_TSC_INTERNAL_ERROR;
   }
@@ -639,6 +685,9 @@ int32_t tqProcessDelCheckInfoReq(STQ* pTq, int64_t sversion, char* msg, int32_t
 }
 
 int32_t tqProcessSubscribeReq(STQ* pTq, int64_t sversion, char* msg, int32_t msgLen) {
+  if (pTq == NULL || msg == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int         ret = 0;
   SMqRebVgReq req = {0};
   SDecoder    dc = {0};

source/dnode/vnode/src/tq/tqMeta.c

@@ -16,6 +16,9 @@
 #include "tq.h"
 
 int32_t tEncodeSTqHandle(SEncoder* pEncoder, const STqHandle* pHandle) {
+  if (pEncoder == NULL || pHandle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t lino;
 
@@ -54,6 +57,9 @@ _exit:
 }
 
 int32_t tDecodeSTqHandle(SDecoder* pDecoder, STqHandle* pHandle) {
+  if (pDecoder == NULL || pHandle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t lino;
 
@@ -91,7 +97,10 @@ _exit:
   return code;
 }
 
-int32_t tqMetaDecodeCheckInfo(STqCheckInfo* info, void* pVal, int32_t vLen) {
+int32_t tqMetaDecodeCheckInfo(STqCheckInfo* info, void* pVal, uint32_t vLen) {
+  if (info == NULL || pVal == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SDecoder decoder = {0};
   tDecoderInit(&decoder, (uint8_t*)pVal, vLen);
   int32_t code = tDecodeSTqCheckInfo(&decoder, info);
@@ -104,7 +113,10 @@ int32_t tqMetaDecodeCheckInfo(STqCheckInfo* info, void* pVal, int32_t vLen) {
   return code;
 }
 
-int32_t tqMetaDecodeOffsetInfo(STqOffset* info, void* pVal, int32_t vLen) {
+int32_t tqMetaDecodeOffsetInfo(STqOffset* info, void* pVal, uint32_t vLen) {
+  if (info == NULL || pVal == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SDecoder decoder = {0};
   tDecoderInit(&decoder, (uint8_t*)pVal, vLen);
   int32_t code = tDecodeSTqOffset(&decoder, info);
@@ -118,9 +130,12 @@ int32_t tqMetaDecodeOffsetInfo(STqOffset* info, void* pVal, int32_t vLen) {
 }
 
 int32_t tqMetaSaveOffset(STQ* pTq, STqOffset* pOffset) {
+  if (pTq == NULL || pOffset == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void*    buf = NULL;
   int32_t  code = TDB_CODE_SUCCESS;
-  int32_t  vlen;
+  uint32_t  vlen;
   SEncoder encoder = {0};
   tEncodeSize(tEncodeSTqOffset, pOffset, vlen, code);
   if (code < 0) {
@@ -147,7 +162,10 @@ END:
   return code;
 }
 
-int32_t tqMetaSaveInfo(STQ* pTq, TTB* ttb, const void* key, int32_t kLen, const void* value, int32_t vLen) {
+int32_t tqMetaSaveInfo(STQ* pTq, TTB* ttb, const void* key, uint32_t kLen, const void* value, uint32_t vLen) {
+  if (pTq == NULL || ttb == NULL || key == NULL || value == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = TDB_CODE_SUCCESS;
   TXN*    txn = NULL;
 
@@ -164,7 +182,10 @@ END:
   return code;
 }
 
-int32_t tqMetaDeleteInfo(STQ* pTq, TTB* ttb, const void* key, int32_t kLen) {
+int32_t tqMetaDeleteInfo(STQ* pTq, TTB* ttb, const void* key, uint32_t kLen) {
+  if (pTq == NULL || ttb == NULL || key == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = TDB_CODE_SUCCESS;
   TXN*    txn = NULL;
 
@@ -182,6 +203,9 @@ END:
 }
 
 int32_t tqMetaGetOffset(STQ* pTq, const char* subkey, STqOffset** pOffset) {
+  if (pTq == NULL || subkey == NULL || pOffset == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void* data = taosHashGet(pTq->pOffset, subkey, strlen(subkey));
   if (data == NULL) {
     int vLen = 0;
@@ -191,7 +215,7 @@ int32_t tqMetaGetOffset(STQ* pTq, const char* subkey, STqOffset** pOffset) {
     }
 
     STqOffset offset = {0};
-    if (tqMetaDecodeOffsetInfo(&offset, data, vLen) != TDB_CODE_SUCCESS) {
+    if (tqMetaDecodeOffsetInfo(&offset, data, vLen >= 0 ? vLen : 0) != TDB_CODE_SUCCESS) {
       tdbFree(data);
       return TSDB_CODE_OUT_OF_MEMORY;
     }
@@ -214,8 +238,11 @@ int32_t tqMetaGetOffset(STQ* pTq, const char* subkey, STqOffset** pOffset) {
 }
 
 int32_t tqMetaSaveHandle(STQ* pTq, const char* key, const STqHandle* pHandle) {
+  if (pTq == NULL || key == NULL || pHandle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t  code = TDB_CODE_SUCCESS;
-  int32_t  vlen;
+  uint32_t  vlen;
   void*    buf = NULL;
   SEncoder encoder = {0};
   tEncodeSize(tEncodeSTqHandle, pHandle, vlen, code);
@@ -238,7 +265,7 @@ int32_t tqMetaSaveHandle(STQ* pTq, const char* key, const STqHandle* pHandle) {
     goto END;
   }
 
-  TQ_ERR_GO_TO_END(tqMetaSaveInfo(pTq, pTq->pExecStore, key, (int)strlen(key), buf, vlen));
+  TQ_ERR_GO_TO_END(tqMetaSaveInfo(pTq, pTq->pExecStore, key, strlen(key), buf, vlen));
 
 END:
   tEncoderClear(&encoder);
@@ -247,6 +274,9 @@ END:
 }
 
 static int tqMetaInitHandle(STQ* pTq, STqHandle* handle) {
+  if (pTq == NULL || handle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = TDB_CODE_SUCCESS;
 
   SVnode* pVnode = pTq->pVnode;
@@ -318,7 +348,10 @@ END:
   return code;
 }
 
-static int32_t tqMetaRestoreHandle(STQ* pTq, void* pVal, int vLen, STqHandle* handle) {
+static int32_t tqMetaRestoreHandle(STQ* pTq, void* pVal, uint32_t vLen, STqHandle* handle) {
+  if (pTq == NULL || pVal == NULL || handle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t  vgId = TD_VID(pTq->pVnode);
   SDecoder decoder = {0};
   int32_t  code = TDB_CODE_SUCCESS;
@@ -335,6 +368,9 @@ END:
 }
 
 int32_t tqMetaCreateHandle(STQ* pTq, SMqRebVgReq* req, STqHandle* handle) {
+  if (pTq == NULL || req == NULL || handle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t vgId = TD_VID(pTq->pVnode);
 
   (void)memcpy(handle->subKey, req->subKey, TSDB_SUBSCRIBE_KEY_LEN);
@@ -375,6 +411,9 @@ int32_t tqMetaCreateHandle(STQ* pTq, SMqRebVgReq* req, STqHandle* handle) {
 }
 
 static int32_t tqMetaTransformInfo(TDB* pMetaDB, TTB* pOld, TTB* pNew) {
+  if (pMetaDB == NULL || pOld == NULL || pNew == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   TBC*  pCur = NULL;
   void* pKey = NULL;
   int   kLen = 0;
@@ -404,6 +443,9 @@ END:
 }
 
 int32_t tqMetaGetHandle(STQ* pTq, const char* key, STqHandle** pHandle) {
+  if (pTq == NULL || key == NULL || pHandle == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void* data = taosHashGet(pTq->pHandle, key, strlen(key));
   if (data == NULL) {
     int vLen = 0;
@@ -412,7 +454,7 @@ int32_t tqMetaGetHandle(STQ* pTq, const char* key, STqHandle** pHandle) {
       return TSDB_CODE_MND_SUBSCRIBE_NOT_EXIST;
     }
     STqHandle handle = {0};
-    if (tqMetaRestoreHandle(pTq, data, vLen, &handle) != 0) {
+    if (tqMetaRestoreHandle(pTq, data, vLen >= 0 ? vLen : 0, &handle) != 0) {
       tdbFree(data);
       tqDestroyTqHandle(&handle);
       return TSDB_CODE_OUT_OF_MEMORY;
@@ -429,6 +471,9 @@ int32_t tqMetaGetHandle(STQ* pTq, const char* key, STqHandle** pHandle) {
 }
 
 int32_t tqMetaOpenTdb(STQ* pTq) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = TDB_CODE_SUCCESS;
   TQ_ERR_GO_TO_END(tdbOpen(pTq->path, 16 * 1024, 1, &pTq->pMetaDB, 0, 0, NULL));
   TQ_ERR_GO_TO_END(tdbTbOpen("tq.db", -1, -1, NULL, pTq->pMetaDB, &pTq->pExecStore, 0));
@@ -440,6 +485,9 @@ END:
 }
 
 static int32_t replaceTqPath(char** path) {
+  if (path == NULL || *path == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   char*   tpath = NULL;
   int32_t code = tqBuildFName(&tpath, *path, TQ_SUBSCRIBE_NAME);
   if (code != 0) {
@@ -451,6 +499,9 @@ static int32_t replaceTqPath(char** path) {
 }
 
 static int32_t tqMetaRestoreCheckInfo(STQ* pTq) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   TBC*         pCur = NULL;
   void*        pKey = NULL;
   int          kLen = 0;
@@ -463,7 +514,7 @@ static int32_t tqMetaRestoreCheckInfo(STQ* pTq) {
   TQ_ERR_GO_TO_END(tdbTbcMoveToFirst(pCur));
 
   while (tdbTbcNext(pCur, &pKey, &kLen, &pVal, &vLen) == 0) {
-    TQ_ERR_GO_TO_END(tqMetaDecodeCheckInfo(&info, pVal, vLen));
+    TQ_ERR_GO_TO_END(tqMetaDecodeCheckInfo(&info, pVal, vLen >= 0 ? vLen : 0));
     TQ_ERR_GO_TO_END(taosHashPut(pTq->pCheckInfo, info.topic, strlen(info.topic), &info, sizeof(STqCheckInfo)));
   }
   info.colIdList = NULL;
@@ -477,6 +528,9 @@ END:
 }
 
 int32_t tqMetaOpen(STQ* pTq) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   char*   maindb = NULL;
   char*   offsetNew = NULL;
   int32_t code = TDB_CODE_SUCCESS;
@@ -504,6 +558,9 @@ END:
 }
 
 int32_t tqMetaTransform(STQ* pTq) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = TDB_CODE_SUCCESS;
   TDB*    pMetaDB = NULL;
   TTB*    pExecStore = NULL;
@@ -543,6 +600,9 @@ END:
 }
 
 void tqMetaClose(STQ* pTq) {
+  if (pTq == NULL) {
+    return;
+  }
   int32_t ret = 0;
   if (pTq->pExecStore) {
     tdbTbClose(pTq->pExecStore);

source/dnode/vnode/src/tq/tqOffset.c

@@ -17,6 +17,9 @@
 #include "tq.h"
 
 int32_t tqBuildFName(char** data, const char* path, char* name) {
+  if (data == NULL || path == NULL || name == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t len = strlen(path) + strlen(name) + 2;
   char*   fname = taosMemoryCalloc(1, len);
   if(fname == NULL) {
@@ -33,6 +36,9 @@ int32_t tqBuildFName(char** data, const char* path, char* name) {
 }
 
 int32_t tqOffsetRestoreFromFile(STQ* pTq, char* name) {
+  if (pTq == NULL || name == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t   code = TDB_CODE_SUCCESS;
   void*     pMemBuf = NULL;
 
@@ -54,6 +60,10 @@ int32_t tqOffsetRestoreFromFile(STQ* pTq, char* name) {
     }
     total += INT_BYTES;
     size = htonl(size);
+    if (size <= 0) {
+      code = TSDB_CODE_INVALID_MSG;
+      goto END;
+    }
     pMemBuf = taosMemoryCalloc(1, size);
     if (pMemBuf == NULL) {
       code = terrno;

source/dnode/vnode/src/tq/tqPush.c

@@ -17,6 +17,9 @@
 #include "vnd.h"
 
 int32_t tqProcessSubmitReqForSubscribe(STQ* pTq) {
+  if (pTq == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   if (taosHashGetSize(pTq->pPushMgr) <= 0) {
     return 0;
   }
@@ -64,6 +67,9 @@ int32_t tqPushMsg(STQ* pTq, tmsg_t msgType) {
 }
 
 int32_t tqRegisterPushHandle(STQ* pTq, void* handle, SRpcMsg* pMsg) {
+  if (pTq == NULL || handle == NULL || pMsg == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t    vgId = TD_VID(pTq->pVnode);
   STqHandle* pHandle = (STqHandle*)handle;
 
@@ -101,6 +107,9 @@ int32_t tqRegisterPushHandle(STQ* pTq, void* handle, SRpcMsg* pMsg) {
 }
 
 void tqUnregisterPushHandle(STQ* pTq, void *handle) {
+  if (pTq == NULL || handle == NULL) {
+    return;
+  }
   STqHandle *pHandle = (STqHandle*)handle;
   int32_t    vgId = TD_VID(pTq->pVnode);
 

source/dnode/vnode/src/tq/tqRead.c

@@ -17,6 +17,9 @@
 #include "tq.h"
 
 bool isValValidForTable(STqHandle* pHandle, SWalCont* pHead) {
+  if (pHandle == NULL || pHead == NULL) {
+    return false;
+  }
   if (pHandle->execHandle.subType != TOPIC_SUB_TYPE__TABLE) {
     return true;
   }
@@ -198,6 +201,9 @@ end:
 }
 
 int32_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, uint64_t reqId) {
+  if (pTq == NULL || pHandle == NULL || fetchOffset == NULL) {
+    return -1;
+  }
   int32_t code = -1;
   int32_t vgId = TD_VID(pTq->pVnode);
   int64_t id = pHandle->pWalReader->readerId;
@@ -259,9 +265,17 @@ END:
   return code;
 }
 
-bool tqGetTablePrimaryKey(STqReader* pReader) { return pReader->hasPrimaryKey; }
+bool tqGetTablePrimaryKey(STqReader* pReader) {
+  if (pReader == NULL) {
+    return false;
+  }
+  return pReader->hasPrimaryKey;
+}
 
 void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) {
+  if (pReader == NULL) {
+    return;
+  }
   bool            ret = false;
   SSchemaWrapper* schema = metaGetTableSchema(pReader->pVnodeMeta, uid, -1, 1, NULL);
   if (schema && schema->nCols >= 2 && schema->pSchema[1].flags & COL_IS_KEY) {
@@ -272,6 +286,9 @@ void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) {
 }
 
 STqReader* tqReaderOpen(SVnode* pVnode) {
+  if (pVnode == NULL) {
+    return NULL;
+  }
   STqReader* pReader = taosMemoryCalloc(1, sizeof(STqReader));
   if (pReader == NULL) {
     return NULL;
@@ -323,6 +340,9 @@ void tqReaderClose(STqReader* pReader) {
 }
 
 int32_t tqReaderSeek(STqReader* pReader, int64_t ver, const char* id) {
+  if (pReader == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   if (walReaderSeekVer(pReader->pWalReader, ver) < 0) {
     return -1;
   }
@@ -406,6 +426,9 @@ int32_t extractMsgFromWal(SWalReader* pReader, void** pItem, int64_t maxVer, con
 }
 
 bool tqNextBlockInWal(STqReader* pReader, const char* id, int sourceExcluded) {
+  if (pReader == NULL) {
+    return false;
+  }
   SWalReader* pWalReader = pReader->pWalReader;
 
   int64_t st = taosGetTimestampMs();
@@ -462,6 +485,9 @@ bool tqNextBlockInWal(STqReader* pReader, const char* id, int sourceExcluded) {
 }
 
 int32_t tqReaderSetSubmitMsg(STqReader* pReader, void* msgStr, int32_t msgLen, int64_t ver) {
+if (pReader == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   pReader->msg.msgStr = msgStr;
   pReader->msg.msgLen = msgLen;
   pReader->msg.ver = ver;
@@ -481,14 +507,29 @@ int32_t tqReaderSetSubmitMsg(STqReader* pReader, void* msgStr, int32_t msgLen, i
   return 0;
 }
 
-SWalReader* tqGetWalReader(STqReader* pReader) { return pReader->pWalReader; }
+SWalReader* tqGetWalReader(STqReader* pReader) {
+  if (pReader == NULL) {
+    return NULL;
+  }
+  return pReader->pWalReader;
+}
 
-SSDataBlock* tqGetResultBlock(STqReader* pReader) { return pReader->pResBlock; }
+SSDataBlock* tqGetResultBlock(STqReader* pReader) {
+  if (pReader == NULL) {
+    return NULL;
+  }
+  return pReader->pResBlock;
+}
 
-int64_t tqGetResultBlockTime(STqReader* pReader) { return pReader->lastTs; }
+int64_t tqGetResultBlockTime(STqReader* pReader) {
+  if (pReader == NULL) {
+    return 0;
+  }
+  return pReader->lastTs;
+}
 
 bool tqNextBlockImpl(STqReader* pReader, const char* idstr) {
-  if (pReader->msg.msgStr == NULL) {
+  if (pReader == NULL || pReader->msg.msgStr == NULL) {
     return false;
   }
 
@@ -525,7 +566,7 @@ bool tqNextBlockImpl(STqReader* pReader, const char* idstr) {
 }
 
 bool tqNextDataBlockFilterOut(STqReader* pReader, SHashObj* filterOutUids) {
-  if (pReader->msg.msgStr == NULL) return false;
+  if (pReader == NULL || pReader->msg.msgStr == NULL) return false;
 
   int32_t blockSz = taosArrayGetSize(pReader->submit.aSubmitTbData);
   while (pReader->nextBlk < blockSz) {
@@ -548,6 +589,9 @@ bool tqNextDataBlockFilterOut(STqReader* pReader, SHashObj* filterOutUids) {
 }
 
 int32_t tqMaskBlock(SSchemaWrapper* pDst, SSDataBlock* pBlock, const SSchemaWrapper* pSrc, char* mask) {
+  if (pDst == NULL || pBlock == NULL || pSrc == NULL || mask == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
 
   int32_t cnt = 0;
@@ -577,6 +621,9 @@ int32_t tqMaskBlock(SSchemaWrapper* pDst, SSDataBlock* pBlock, const SSchemaWrap
 }
 
 static int32_t buildResSDataBlock(STqReader* pReader, SSchemaWrapper* pSchema, const SArray* pColIdList) {
+  if (pReader == NULL || pSchema == NULL || pColIdList == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SSDataBlock* pBlock = pReader->pResBlock;
   if (blockDataGetNumOfCols(pBlock) > 0) {
       blockDataDestroy(pBlock);
@@ -659,6 +706,9 @@ static int32_t doSetVal(SColumnInfoData* pColumnInfoData, int32_t rowIndex, SCol
 }
 
 int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* id) {
+  if (pReader == NULL || pRes == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   tqTrace("tq reader retrieve data block %p, index:%d", pReader->msg.msgStr, pReader->nextBlk);
   int32_t        code = 0;
   int32_t        line = 0;
@@ -825,6 +875,10 @@ END:
 static int32_t processBuildNew(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas,
                                SSchemaWrapper* pSchemaWrapper, char* assigned, int32_t numOfRows, int32_t curRow,
                                int32_t* lastRow) {
+  if (pReader == NULL || pSubmitTbData == NULL || blocks == NULL || schemas == NULL || pSchemaWrapper == NULL ||
+      assigned == NULL || lastRow == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t         code = 0;
   SSchemaWrapper* pSW = NULL;
   SSDataBlock*    block = NULL;
@@ -860,6 +914,9 @@ END:
   return code;
 }
 static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas) {
+  if (pReader == NULL || pSubmitTbData == NULL || blocks == NULL || schemas == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   int32_t curRow = 0;
   int32_t lastRow = 0;
@@ -919,6 +976,9 @@ END:
 }
 
 int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas) {
+  if (pReader == NULL || pSubmitTbData == NULL || blocks == NULL || schemas == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t   code = 0;
   STSchema* pTSchema = NULL;
 
@@ -976,6 +1036,9 @@ END:
 }
 
 int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas, SSubmitTbData** pSubmitTbDataRet, int64_t *createTime) {
+  if (pReader == NULL || blocks == NULL || schemas == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   tqTrace("tq reader retrieve data block %p, %d", pReader->msg.msgStr, pReader->nextBlk);
   SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk);
   if (pSubmitTbData == NULL) {
@@ -1007,9 +1070,17 @@ int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas
   }
 }
 
-void tqReaderSetColIdList(STqReader* pReader, SArray* pColIdList) { pReader->pColIdList = pColIdList; }
+void tqReaderSetColIdList(STqReader* pReader, SArray* pColIdList) {
+  if (pReader == NULL){
+    return;
+  }
+  pReader->pColIdList = pColIdList;
+}
 
 void tqReaderSetTbUidList(STqReader* pReader, const SArray* tbUidList, const char* id) {
+  if (pReader == NULL || tbUidList == NULL) {
+    return;
+  }
   if (pReader->tbIdHash) {
     taosHashClear(pReader->tbIdHash);
   } else {
@@ -1032,6 +1103,9 @@ void tqReaderSetTbUidList(STqReader* pReader, const SArray* tbUidList, const cha
 }
 
 void tqReaderAddTbUidList(STqReader* pReader, const SArray* pTableUidList) {
+  if (pReader == NULL || pTableUidList == NULL) {
+    return;
+  }
   if (pReader->tbIdHash == NULL) {
     pReader->tbIdHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_ENTRY_LOCK);
     if (pReader->tbIdHash == NULL) {
@@ -1051,12 +1125,23 @@ void tqReaderAddTbUidList(STqReader* pReader, const SArray* pTableUidList) {
 }
 
 bool tqReaderIsQueriedTable(STqReader* pReader, uint64_t uid) {
-  return taosHashGet(pReader->tbIdHash, &uid, sizeof(uint64_t));
+  if (pReader == NULL) {
+    return false;
+  }
+  return taosHashGet(pReader->tbIdHash, &uid, sizeof(uint64_t)) != NULL;
 }
 
-bool tqCurrentBlockConsumed(const STqReader* pReader) { return pReader->msg.msgStr == NULL; }
+bool tqCurrentBlockConsumed(const STqReader* pReader) {
+  if (pReader == NULL) {
+    return false;
+  }
+  return pReader->msg.msgStr == NULL;
+}
 
 void tqReaderRemoveTbUidList(STqReader* pReader, const SArray* tbUidList) {
+  if (pReader == NULL || tbUidList == NULL) {
+    return;
+  }
   for (int32_t i = 0; i < taosArrayGetSize(tbUidList); i++) {
     int64_t* pKey = (int64_t*)taosArrayGet(tbUidList, i);
     if (pKey && taosHashRemove(pReader->tbIdHash, pKey, sizeof(int64_t)) != 0) {
@@ -1066,6 +1151,9 @@ void tqReaderRemoveTbUidList(STqReader* pReader, const SArray* tbUidList) {
 }
 
 int32_t tqUpdateTbUidList(STQ* pTq, const SArray* tbUidList, bool isAdd) {
+  if (pTq == NULL || tbUidList == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   void*   pIter = NULL;
   int32_t vgId = TD_VID(pTq->pVnode);
 

source/dnode/vnode/src/tq/tqScan.c

@@ -16,6 +16,9 @@
 #include "tq.h"
 
 int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp, int32_t numOfCols, int8_t precision) {
+  if (pBlock == NULL || pRsp == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   size_t dataEncodeBufSize = blockGetEncodeSize(pBlock);
   int32_t dataStrLen = sizeof(SRetrieveTableRspForTmq) + dataEncodeBufSize;
   void*   buf = taosMemoryCalloc(1, dataStrLen);
@@ -47,18 +50,10 @@ int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp, int32_t
   return TSDB_CODE_SUCCESS;
 }
 
-static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, SMqDataRsp* pRsp) {
-  SSchemaWrapper* pSW = tCloneSSchemaWrapper(pExec->pTqReader->pSchemaWrapper);
-  if (pSW == NULL) {
-    return terrno;
-  }
-  if (taosArrayPush(pRsp->blockSchema, &pSW) == NULL) {
-    return terrno;
-  }
-  return 0;
-}
-
 static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, int32_t n) {
+  if (pRsp == NULL || pTq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   SMetaReader mr = {0};
   metaReaderDoInit(&mr, pTq->pVnode->pMeta, META_READER_LOCK);
 
@@ -84,6 +79,9 @@ static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, i
 }
 
 int32_t getDataBlock(qTaskInfo_t task, const STqHandle* pHandle, int32_t vgId, SSDataBlock** res) {
+  if (task == NULL || pHandle == NULL || res == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   uint64_t ts = 0;
   qStreamSetOpen(task);
 
@@ -99,6 +97,9 @@ int32_t getDataBlock(qTaskInfo_t task, const STqHandle* pHandle, int32_t vgId, S
 }
 
 int32_t tqScanData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, STqOffsetVal* pOffset, const SMqPollReq* pRequest) {
+  if (pTq == NULL || pHandle == NULL || pRsp == NULL || pOffset == NULL || pRequest == NULL){
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t vgId = TD_VID(pTq->pVnode);
   int32_t code = 0;
   int32_t line = 0;
@@ -189,6 +190,9 @@ END:
 }
 
 int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, SMqBatchMetaRsp* pBatchMetaRsp, STqOffsetVal* pOffset) {
+  if (pTq == NULL || pHandle == NULL || pRsp == NULL || pBatchMetaRsp == NULL || pOffset == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   const STqExecHandle* pExec = &pHandle->execHandle;
   qTaskInfo_t          task = pExec->task;
   int code = qStreamPrepareScan(task, pOffset, pHandle->execHandle.subType);
@@ -280,6 +284,9 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, SMqBat
 }
 
 static int32_t buildCreateTbInfo(SMqDataRsp* pRsp, SVCreateTbReq* pCreateTbReq){
+  if (pRsp == NULL || pCreateTbReq == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t code = 0;
   void*   createReq = NULL;
   if (pRsp->createTableNum == 0) {
@@ -329,6 +336,9 @@ END:
 }
 
 static void tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded){
+  if (pTq == NULL || pHandle == NULL || pRsp == NULL || totalRows == NULL) {
+    return;
+  }
   int32_t code = 0;
   STqExecHandle* pExec = &pHandle->execHandle;
   STqReader* pReader = pExec->pTqReader;
@@ -407,6 +417,9 @@ END:
 
 int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SPackedData submit, SMqDataRsp* pRsp, int32_t* totalRows,
                        int8_t sourceExcluded) {
+  if (pTq == NULL || pHandle == NULL || pRsp == NULL || totalRows == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   STqExecHandle* pExec = &pHandle->execHandle;
   int32_t        code = 0;
   STqReader* pReader = pExec->pTqReader;

source/dnode/vnode/src/tq/tqSnapshot.c

@@ -27,6 +27,9 @@ struct STqSnapReader {
 };
 
 int32_t tqSnapReaderOpen(STQ* pTq, int64_t sver, int64_t ever, int8_t type, STqSnapReader** ppReader) {
+  if (pTq == NULL || ppReader == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t        code = 0;
   STqSnapReader* pReader = NULL;
 
@@ -77,12 +80,18 @@ _err:
 }
 
 void tqSnapReaderClose(STqSnapReader** ppReader) {
+  if (ppReader == NULL || *ppReader == NULL) {
+    return;
+  }
   tdbTbcClose((*ppReader)->pCur);
   taosMemoryFree(*ppReader);
   *ppReader = NULL;
 }
 
 int32_t tqSnapRead(STqSnapReader* pReader, uint8_t** ppData) {
+  if (pReader == NULL || ppData == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t code = 0;
   void*   pKey = NULL;
   void*   pVal = NULL;
@@ -126,6 +135,9 @@ struct STqSnapWriter {
 };
 
 int32_t tqSnapWriterOpen(STQ* pTq, int64_t sver, int64_t ever, STqSnapWriter** ppWriter) {
+  if (pTq == NULL || ppWriter == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t        code = 0;
   STqSnapWriter* pWriter = NULL;
 
@@ -156,6 +168,9 @@ _err:
 }
 
 int32_t tqSnapWriterClose(STqSnapWriter** ppWriter, int8_t rollback) {
+  if (ppWriter == NULL || *ppWriter == NULL) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t        code = 0;
   STqSnapWriter* pWriter = *ppWriter;
   STQ*           pTq = pWriter->pTq;
@@ -180,6 +195,9 @@ _err:
 }
 
 int32_t tqSnapHandleWrite(STqSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  if (pWriter == NULL || pData == NULL || nData < sizeof(SSnapDataHdr)) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t   code = 0;
   STQ*      pTq = pWriter->pTq;
   SDecoder  decoder = {0};
@@ -190,7 +208,7 @@ int32_t tqSnapHandleWrite(STqSnapWriter* pWriter, uint8_t* pData, uint32_t nData
   code = tDecodeSTqHandle(pDecoder, &handle);
   if (code) goto end;
   taosWLockLatch(&pTq->lock);
-  code = tqMetaSaveInfo(pTq, pTq->pExecStore, handle.subKey, (int)strlen(handle.subKey), pData + sizeof(SSnapDataHdr),
+  code = tqMetaSaveInfo(pTq, pTq->pExecStore, handle.subKey, strlen(handle.subKey), pData + sizeof(SSnapDataHdr),
                         nData - sizeof(SSnapDataHdr));
   taosWUnLockLatch(&pTq->lock);
 
@@ -202,6 +220,9 @@ end:
 }
 
 int32_t tqSnapCheckInfoWrite(STqSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  if (pWriter == NULL || pData == NULL || nData < sizeof(SSnapDataHdr)) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t      code = 0;
   STQ*         pTq = pWriter->pTq;
   STqCheckInfo info = {0};
@@ -223,6 +244,9 @@ _err:
 }
 
 int32_t tqSnapOffsetWrite(STqSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  if (pWriter == NULL || pData == NULL || nData < sizeof(SSnapDataHdr)) {
+    return TSDB_CODE_INVALID_MSG;
+  }
   int32_t code = 0;
   STQ*    pTq = pWriter->pTq;
 

source/dnode/vnode/src/tq/tqUtil.c

@@ -21,6 +21,9 @@ static int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, c
                                       const SMqBatchMetaRsp* pRsp, int32_t vgId);
 
 int32_t tqInitDataRsp(SMqDataRsp* pRsp, STqOffsetVal pOffset) {
+  if (pRsp == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   pRsp->blockData = taosArrayInit(0, sizeof(void*));
   pRsp->blockDataLen = taosArrayInit(0, sizeof(int32_t));
 
@@ -41,6 +44,9 @@ void tqUpdateNodeStage(STQ* pTq, bool isLeader) {
 }
 
 static int32_t tqInitTaosxRsp(SMqDataRsp* pRsp, STqOffsetVal pOffset) {
+  if (pRsp == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   tOffsetCopy(&pRsp->reqOffset, &pOffset);
   tOffsetCopy(&pRsp->rspOffset, &pOffset);
 
@@ -81,6 +87,9 @@ static int32_t tqInitTaosxRsp(SMqDataRsp* pRsp, STqOffsetVal pOffset) {
 
 static int32_t extractResetOffsetVal(STqOffsetVal* pOffsetVal, STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest,
                                      SRpcMsg* pMsg, bool* pBlockReturned) {
+  if (pOffsetVal == NULL || pTq == NULL || pHandle == NULL || pRequest == NULL || pMsg == NULL || pBlockReturned == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   uint64_t   consumerId = pRequest->consumerId;
   STqOffset* pOffset = NULL;
   int32_t    code = tqMetaGetOffset(pTq, pRequest->subKey, &pOffset);
@@ -142,6 +151,9 @@ static int32_t extractResetOffsetVal(STqOffsetVal* pOffsetVal, STQ* pTq, STqHand
 
 static int32_t extractDataAndRspForNormalSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest,
                                                    SRpcMsg* pMsg, STqOffsetVal* pOffset) {
+  if (pTq == NULL || pHandle == NULL || pRequest == NULL || pMsg == NULL || pOffset == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   uint64_t consumerId = pRequest->consumerId;
   int32_t  vgId = TD_VID(pTq->pVnode);
   terrno = 0;
@@ -212,6 +224,9 @@ static void tDeleteCommon(void* parm) {}
 
 static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest,
                                                   SRpcMsg* pMsg, STqOffsetVal* offset) {
+  if (pTq == NULL || pHandle == NULL || pRequest == NULL || pMsg == NULL || offset == NULL) {
+    return TSDB_CODE_INVALID_PARA;
+  }
   int32_t         vgId = TD_VID(pTq->pVnode);
   SMqDataRsp      taosxRsp = {0};
   SMqBatchMetaRsp btMetaRsp = {0};
@@ -410,6 +425,9 @@ END:
 }
 
 int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg) {
+  if (pTq == NULL || pHandle == NULL || pRequest == NULL || pMsg == NULL) {
+    return TSDB_CODE_TMQ_INVALID_MSG;
+  }
   int32_t      code = 0;
   STqOffsetVal reqOffset = {0};
   tOffsetCopy(&reqOffset, &pRequest->reqOffset);
@@ -445,6 +463,9 @@ END:
 
 static void initMqRspHead(SMqRspHead* pMsgHead, int32_t type, int32_t epoch, int64_t consumerId, int64_t sver,
                           int64_t ever) {
+  if (pMsgHead == NULL) {
+    return;
+  }
   pMsgHead->consumerId = consumerId;
   pMsgHead->epoch = epoch;
   pMsgHead->mqMsgType = type;
@@ -454,6 +475,9 @@ static void initMqRspHead(SMqRspHead* pMsgHead, int32_t type, int32_t epoch, int
 
 int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* pReq,
                                const SMqBatchMetaRsp* pRsp, int32_t vgId) {
+  if (pHandle == NULL || pMsg == NULL || pReq == NULL || pRsp == NULL) {
+    return TSDB_CODE_TMQ_INVALID_MSG;
+  }
   int32_t len = 0;
   int32_t code = 0;
   tEncodeSize(tEncodeMqBatchMetaRsp, pRsp, len, code);
@@ -491,6 +515,9 @@ int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SM
 
 int32_t tqSendMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* pReq, const SMqMetaRsp* pRsp,
                           int32_t vgId) {
+  if (pHandle == NULL || pMsg == NULL || pReq == NULL || pRsp == NULL) {
+    return TSDB_CODE_TMQ_INVALID_MSG;
+  }
   int32_t len = 0;
   int32_t code = 0;
   tEncodeSize(tEncodeMqMetaRsp, pRsp, len, code);
@@ -529,6 +556,9 @@ int32_t tqSendMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPoll
 
 int32_t tqDoSendDataRsp(const SRpcHandleInfo* pRpcHandleInfo, const SMqDataRsp* pRsp, int32_t epoch, int64_t consumerId,
                         int32_t type, int64_t sver, int64_t ever) {
+  if (pRpcHandleInfo == NULL || pRsp == NULL) {
+    return TSDB_CODE_TMQ_INVALID_MSG;
+  }
   int32_t len = 0;
   int32_t code = 0;
 

source/dnode/vnode/src/tsdb/tsdbCache.c

@@ -2635,23 +2635,15 @@ _exit:
 }
 
 int32_t tsdbCacheDel(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey) {
-  int32_t code = 0, lino = 0;
-  // fetch schema
+  int32_t   code = 0, lino = 0;
   STSchema *pTSchema = NULL;
   int       sver = -1;
+  int       numKeys = 0;
+  SArray   *remainCols = NULL;
 
   TAOS_CHECK_RETURN(metaGetTbTSchemaEx(pTsdb->pVnode->pMeta, suid, uid, sver, &pTSchema));
 
-  // build keys & multi get from rocks
-  int     numCols = pTSchema->numOfCols;
-  int     numKeys = 0;
-  SArray *remainCols = NULL;
-
-  code = tsdbCacheCommit(pTsdb);
-  if (code != TSDB_CODE_SUCCESS) {
-    tsdbTrace("vgId:%d, %s commit failed at line %d since %s", TD_VID(pTsdb->pVnode), __func__, __LINE__,
-              tstrerror(code));
-  }
+  int numCols = pTSchema->numOfCols;
 
   (void)taosThreadMutexLock(&pTsdb->lruMutex);
 

source/dnode/vnode/src/tsdb/tsdbCommit2.c

@@ -574,7 +574,7 @@ static int32_t tsdbCommitInfoBuild(STsdb *tsdb) {
   // begin tasks on file set
   for (int i = 0; i < taosArrayGetSize(tsdb->commitInfo->arr); i++) {
     SFileSetCommitInfo *info = *(SFileSetCommitInfo **)taosArrayGet(tsdb->commitInfo->arr, i);
-    tsdbBeginTaskOnFileSet(tsdb, info->fid, &fset);
+    tsdbBeginTaskOnFileSet(tsdb, info->fid, EVA_TASK_COMMIT, &fset);
     if (fset) {
       code = tsdbTFileSetInitCopy(tsdb, fset, &info->fset);
       if (code) {
@@ -712,7 +712,7 @@ int32_t tsdbCommitCommit(STsdb *tsdb) {
     for (int32_t i = 0; i < taosArrayGetSize(tsdb->commitInfo->arr); i++) {
       SFileSetCommitInfo *info = *(SFileSetCommitInfo **)taosArrayGet(tsdb->commitInfo->arr, i);
       if (info->fset) {
-        tsdbFinishTaskOnFileSet(tsdb, info->fid);
+        tsdbFinishTaskOnFileSet(tsdb, info->fid, EVA_TASK_COMMIT);
       }
     }
 
@@ -743,7 +743,7 @@ int32_t tsdbCommitAbort(STsdb *pTsdb) {
   for (int32_t i = 0; i < taosArrayGetSize(pTsdb->commitInfo->arr); i++) {
     SFileSetCommitInfo *info = *(SFileSetCommitInfo **)taosArrayGet(pTsdb->commitInfo->arr, i);
     if (info->fset) {
-      tsdbFinishTaskOnFileSet(pTsdb, info->fid);
+      tsdbFinishTaskOnFileSet(pTsdb, info->fid, EVA_TASK_COMMIT);
     }
   }
   (void)taosThreadMutexUnlock(&pTsdb->mutex);

source/dnode/vnode/src/tsdb/tsdbFS2.c

@@ -770,8 +770,8 @@ extern void tsdbStopAllCompTask(STsdb *tsdb);
 
 int32_t tsdbDisableAndCancelAllBgTask(STsdb *pTsdb) {
   STFileSystem *fs = pTsdb->pFS;
-  SArray       *channelArray = taosArrayInit(0, sizeof(SVAChannelID));
-  if (channelArray == NULL) {
+  SArray       *asyncTasks = taosArrayInit(0, sizeof(SVATaskID));
+  if (asyncTasks == NULL) {
     return terrno;
   }
 
@@ -783,30 +783,31 @@ int32_t tsdbDisableAndCancelAllBgTask(STsdb *pTsdb) {
   // collect channel
   STFileSet *fset;
   TARRAY2_FOREACH(fs->fSetArr, fset) {
-    if (fset->channelOpened) {
-      if (taosArrayPush(channelArray, &fset->channel) == NULL) {
-        taosArrayDestroy(channelArray);
-        (void)taosThreadMutexUnlock(&pTsdb->mutex);
-        return terrno;
-      }
-      fset->channel = (SVAChannelID){0};
-      fset->mergeScheduled = false;
-      tsdbFSSetBlockCommit(fset, false);
-      fset->channelOpened = false;
+    if (taosArrayPush(asyncTasks, &fset->mergeTask) == NULL       //
+        || taosArrayPush(asyncTasks, &fset->compactTask) == NULL  //
+        || taosArrayPush(asyncTasks, &fset->retentionTask) == NULL) {
+      taosArrayDestroy(asyncTasks);
+      (void)taosThreadMutexUnlock(&pTsdb->mutex);
+      return terrno;
     }
+    fset->mergeScheduled = false;
+    tsdbFSSetBlockCommit(fset, false);
   }
 
   (void)taosThreadMutexUnlock(&pTsdb->mutex);
 
   // destroy all channels
-  for (int32_t i = 0; i < taosArrayGetSize(channelArray); i++) {
-    SVAChannelID *channel = taosArrayGet(channelArray, i);
-    int32_t       code = vnodeAChannelDestroy(channel, true);
-    if (code) {
-      tsdbError("vgId:%d %s failed at line %d since %s", TD_VID(pTsdb->pVnode), __func__, __LINE__, tstrerror(code));
+  for (int32_t k = 0; k < 2; k++) {
+    for (int32_t i = 0; i < taosArrayGetSize(asyncTasks); i++) {
+      SVATaskID *task = taosArrayGet(asyncTasks, i);
+      if (k == 0) {
+        (void)vnodeACancel(task);
+      } else {
+        (void)vnodeAWait(task);
+      }
     }
   }
-  taosArrayDestroy(channelArray);
+  taosArrayDestroy(asyncTasks);
 
 #ifdef TD_ENTERPRISE
   tsdbStopAllCompTask(pTsdb);
@@ -934,9 +935,6 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) {
       // bool    skipMerge = false;
       int32_t numFile = TARRAY2_SIZE(lvl->fobjArr);
       if (numFile >= sttTrigger && (!fset->mergeScheduled)) {
-        code = tsdbTFileSetOpenChannel(fset);
-        TSDB_CHECK_CODE(code, lino, _exit);
-
         SMergeArg *arg = taosMemoryMalloc(sizeof(*arg));
         if (arg == NULL) {
           code = terrno;
@@ -946,7 +944,7 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) {
         arg->tsdb = fs->tsdb;
         arg->fid = fset->fid;
 
-        code = vnodeAsync(&fset->channel, EVA_PRIORITY_HIGH, tsdbMerge, taosAutoMemoryFree, arg, NULL);
+        code = vnodeAsync(MERGE_TASK_ASYNC, EVA_PRIORITY_HIGH, tsdbMerge, taosAutoMemoryFree, arg, &fset->mergeTask);
         TSDB_CHECK_CODE(code, lino, _exit);
         fset->mergeScheduled = true;
       }
@@ -1202,42 +1200,61 @@ _out:
 
 void tsdbFSDestroyRefRangedSnapshot(TFileSetRangeArray **fsrArr) { tsdbTFileSetRangeArrayDestroy(fsrArr); }
 
-void tsdbBeginTaskOnFileSet(STsdb *tsdb, int32_t fid, STFileSet **fset) {
+void tsdbBeginTaskOnFileSet(STsdb *tsdb, int32_t fid, EVATaskT task, STFileSet **fset) {
+  // Here, sttTrigger is protected by tsdb->mutex, so it is safe to read it without lock
   int16_t sttTrigger = tsdb->pVnode->config.sttTrigger;
 
   tsdbFSGetFSet(tsdb->pFS, fid, fset);
-  if (sttTrigger == 1 && (*fset)) {
-    for (;;) {
-      if ((*fset)->taskRunning) {
-        (*fset)->numWaitTask++;
-
-        (void)taosThreadCondWait(&(*fset)->beginTask, &tsdb->mutex);
-
-        tsdbFSGetFSet(tsdb->pFS, fid, fset);
-
-        (*fset)->numWaitTask--;
-      } else {
-        (*fset)->taskRunning = true;
-        break;
-      }
-    }
-    tsdbInfo("vgId:%d begin task on file set:%d", TD_VID(tsdb->pVnode), fid);
+  if (*fset == NULL) {
+    return;
   }
+
+  struct STFileSetCond *cond = NULL;
+  if (sttTrigger == 1 || task == EVA_TASK_COMMIT) {
+    cond = &(*fset)->conds[0];
+  } else {
+    cond = &(*fset)->conds[1];
+  }
+
+  while (1) {
+    if (cond->running) {
+      cond->numWait++;
+      (void)taosThreadCondWait(&cond->cond, &tsdb->mutex);
+      cond->numWait--;
+    } else {
+      cond->running = true;
+      break;
+    }
+  }
+
+  tsdbInfo("vgId:%d begin %s task on file set:%d", TD_VID(tsdb->pVnode), vnodeGetATaskName(task), fid);
+  return;
 }
 
-void tsdbFinishTaskOnFileSet(STsdb *tsdb, int32_t fid) {
+void tsdbFinishTaskOnFileSet(STsdb *tsdb, int32_t fid, EVATaskT task) {
+  // Here, sttTrigger is protected by tsdb->mutex, so it is safe to read it without lock
   int16_t sttTrigger = tsdb->pVnode->config.sttTrigger;
-  if (sttTrigger == 1) {
-    STFileSet *fset = NULL;
-    tsdbFSGetFSet(tsdb->pFS, fid, &fset);
-    if (fset != NULL && fset->taskRunning) {
-      fset->taskRunning = false;
-      if (fset->numWaitTask > 0) {
-        (void)taosThreadCondSignal(&fset->beginTask);
-      }
-      tsdbInfo("vgId:%d finish task on file set:%d", TD_VID(tsdb->pVnode), fid);
-    }
+
+  STFileSet *fset = NULL;
+  tsdbFSGetFSet(tsdb->pFS, fid, &fset);
+  if (fset == NULL) {
+    return;
   }
+
+  struct STFileSetCond *cond = NULL;
+  if (sttTrigger == 1 || task == EVA_TASK_COMMIT) {
+    cond = &fset->conds[0];
+  } else {
+    cond = &fset->conds[1];
+  }
+
+  cond->running = false;
+  if (cond->numWait > 0) {
+    (void)taosThreadCondSignal(&cond->cond);
+  }
+
+  tsdbInfo("vgId:%d finish %s task on file set:%d", TD_VID(tsdb->pVnode), vnodeGetATaskName(task), fid);
+  return;
 }
 
 struct SFileSetReader {

source/dnode/vnode/src/tsdb/tsdbFS2.h

@@ -14,6 +14,7 @@
  */
 
 #include "tsdbFSet2.h"
+#include "vnd.h"
 
 #ifndef _TSDB_FILE_SYSTEM_H
 #define _TSDB_FILE_SYSTEM_H
@@ -61,8 +62,8 @@ int32_t tsdbFSEditAbort(STFileSystem *fs);
 // other
 void tsdbFSGetFSet(STFileSystem *fs, int32_t fid, STFileSet **fset);
 void tsdbFSCheckCommit(STsdb *tsdb, int32_t fid);
-void tsdbBeginTaskOnFileSet(STsdb *tsdb, int32_t fid, STFileSet **fset);
-void tsdbFinishTaskOnFileSet(STsdb *tsdb, int32_t fid);
+void tsdbBeginTaskOnFileSet(STsdb *tsdb, int32_t fid, EVATaskT task, STFileSet **fset);
+void tsdbFinishTaskOnFileSet(STsdb *tsdb, int32_t fid, EVATaskT task);
 // utils
 int32_t save_fs(const TFileSetArray *arr, const char *fname);
 void    current_fname(STsdb *pTsdb, char *fname, EFCurrentT ftype);

source/dnode/vnode/src/tsdb/tsdbFSet2.c

@@ -480,16 +480,18 @@ int32_t tsdbTFileSetInit(int32_t fid, STFileSet **fset) {
   fset[0]->maxVerValid = VERSION_MAX;
   TARRAY2_INIT(fset[0]->lvlArr);
 
-  // background task queue
-  (void)taosThreadCondInit(&(*fset)->beginTask, NULL);
-  (*fset)->taskRunning = false;
-  (*fset)->numWaitTask = 0;
-
   // block commit variables
   (void)taosThreadCondInit(&fset[0]->canCommit, NULL);
   (*fset)->numWaitCommit = 0;
   (*fset)->blockCommit = false;
 
+  for (int32_t i = 0; i < sizeof((*fset)->conds) / sizeof((*fset)->conds[0]); ++i) {
+    struct STFileSetCond *cond = &(*fset)->conds[i];
+    cond->running = false;
+    cond->numWait = 0;
+    (void)taosThreadCondInit(&cond->cond, NULL);
+  }
+
   return 0;
 }
 
@@ -648,8 +650,10 @@ void tsdbTFileSetClear(STFileSet **fset) {
 
     TARRAY2_DESTROY((*fset)->lvlArr, tsdbSttLvlClear);
 
-    (void)taosThreadCondDestroy(&(*fset)->beginTask);
     (void)taosThreadCondDestroy(&(*fset)->canCommit);
+    for (int32_t i = 0; i < sizeof((*fset)->conds) / sizeof((*fset)->conds[0]); ++i) {
+      (void)taosThreadCondDestroy(&(*fset)->conds[i].cond);
+    }
     taosMemoryFreeClear(*fset);
   }
 }
@@ -703,14 +707,3 @@ bool tsdbTFileSetIsEmpty(const STFileSet *fset) {
   }
   return TARRAY2_SIZE(fset->lvlArr) == 0;
 }
-
-int32_t tsdbTFileSetOpenChannel(STFileSet *fset) {
-  int32_t code;
-  if (!fset->channelOpened) {
-    if ((code = vnodeAChannelInit(2, &fset->channel))) {
-      return code;
-    }
-    fset->channelOpened = true;
-  }
-  return 0;
-}

source/dnode/vnode/src/tsdb/tsdbFSet2.h

@@ -68,8 +68,6 @@ bool tsdbTFileSetIsEmpty(const STFileSet *fset);
 // stt
 int32_t tsdbSttLvlInit(int32_t level, SSttLvl **lvl);
 void    tsdbSttLvlClear(SSttLvl **lvl);
-// open channel
-int32_t tsdbTFileSetOpenChannel(STFileSet *fset);
 
 struct STFileOp {
   tsdb_fop_t optype;
@@ -83,26 +81,30 @@ struct SSttLvl {
   TFileObjArray fobjArr[1];
 };
 
+struct STFileSetCond {
+  bool         running;
+  int32_t      numWait;
+  TdThreadCond cond;
+};
+
 struct STFileSet {
   int32_t      fid;
   int64_t      maxVerValid;
   STFileObj   *farr[TSDB_FTYPE_MAX];  // file array
   TSttLvlArray lvlArr[1];             // level array
 
-  // background task
-  bool         channelOpened;
-  SVAChannelID channel;
-  bool         mergeScheduled;
-
-  // sttTrigger = 1
-  TdThreadCond beginTask;
-  bool         taskRunning;
-  int32_t      numWaitTask;
+  bool      mergeScheduled;
+  SVATaskID mergeTask;
+  SVATaskID compactTask;
+  SVATaskID retentionTask;
 
   // block commit variables
   TdThreadCond canCommit;
   int32_t      numWaitCommit;
   bool         blockCommit;
+
+  // conditions
+  struct STFileSetCond conds[2];
 };
 
 struct STFileSetRange {

source/dnode/vnode/src/tsdb/tsdbMerge.c

@@ -462,21 +462,29 @@ _exit:
 
 static int32_t tsdbMergeGetFSet(SMerger *merger) {
   STFileSet *fset;
+  int32_t    code;
+  STsdb     *tsdb = merger->tsdb;
 
   (void)taosThreadMutexLock(&merger->tsdb->mutex);
-  tsdbFSGetFSet(merger->tsdb->pFS, merger->fid, &fset);
-  if (fset == NULL) {
+
+  if (tsdb->bgTaskDisabled) {
     (void)taosThreadMutexUnlock(&merger->tsdb->mutex);
     return 0;
   }
 
-  fset->mergeScheduled = false;
+  tsdbBeginTaskOnFileSet(tsdb, merger->fid, EVA_TASK_MERGE, &fset);
+  if (NULL == fset) {
+    (void)taosThreadMutexUnlock(&merger->tsdb->mutex);
+    return 0;
+  }
 
-  int32_t code = tsdbTFileSetInitCopy(merger->tsdb, fset, &merger->fset);
+  code = tsdbTFileSetInitCopy(merger->tsdb, fset, &merger->fset);
   if (code) {
     (void)taosThreadMutexUnlock(&merger->tsdb->mutex);
     return code;
   }
+
+  fset->mergeScheduled = false;
   (void)taosThreadMutexUnlock(&merger->tsdb->mutex);
   return 0;
 }
@@ -493,10 +501,13 @@ int32_t tsdbMerge(void *arg) {
       .sttTrigger = tsdb->pVnode->config.sttTrigger,
   }};
 
-  if (merger->sttTrigger <= 1) return 0;
+  if (merger->sttTrigger <= 1) {
+    return 0;
+  }
 
   // copy snapshot
-  TAOS_CHECK_GOTO(tsdbMergeGetFSet(merger), &lino, _exit);
+  code = tsdbMergeGetFSet(merger);
+  TSDB_CHECK_CODE(code, lino, _exit);
 
   if (merger->fset == NULL) {
     return 0;
@@ -509,12 +520,19 @@ int32_t tsdbMerge(void *arg) {
   TSDB_CHECK_CODE(code, lino, _exit);
 
 _exit:
+  if (merger->fset) {
+    (void)taosThreadMutexLock(&tsdb->mutex);
+    tsdbFinishTaskOnFileSet(tsdb, mergeArg->fid, EVA_TASK_MERGE);
+    (void)taosThreadMutexUnlock(&tsdb->mutex);
+  }
+
   if (code) {
     tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(tsdb->pVnode), __func__, __FILE__, lino, tstrerror(code));
     tsdbFatal("vgId:%d, failed to merge stt files since %s. code:%d", TD_VID(tsdb->pVnode), terrstr(), code);
     taosMsleep(100);
     exit(EXIT_FAILURE);
   }
+
   tsdbTFileSetClear(&merger->fset);
   taosMemoryFree(arg);
   return code;

source/dnode/vnode/src/tsdb/tsdbOpen.c

@@ -18,22 +18,6 @@
 
 extern int32_t tsdbOpenCompMonitor(STsdb *tsdb);
 extern void    tsdbCloseCompMonitor(STsdb *tsdb);
-extern int32_t tsdbInitCompact();
-extern void    tsdbCleanupCompact();
-
-int32_t tsdbInit() {
-#ifdef TD_ENTERPRISE
-  return tsdbInitCompact();
-#endif
-  return 0;
-}
-
-void tsdbCleanUp() {
-#ifdef TD_ENTERPRISE
-  tsdbCleanupCompact();
-#endif
-  return;
-}
 
 void tsdbSetKeepCfg(STsdb *pTsdb, STsdbCfg *pCfg) {
   STsdbKeepCfg *pKeepCfg = &pTsdb->keepCfg;

source/dnode/vnode/src/tsdb/tsdbRetention.c

@@ -325,11 +325,21 @@ static int32_t tsdbRetention(void *arg) {
 
   // begin task
   (void)taosThreadMutexLock(&pTsdb->mutex);
-  tsdbBeginTaskOnFileSet(pTsdb, rtnArg->fid, &fset);
+
+  // check if background task is disabled
+  if (pTsdb->bgTaskDisabled) {
+    tsdbInfo("vgId:%d, background task is disabled, skip retention", TD_VID(pTsdb->pVnode));
+    (void)taosThreadMutexUnlock(&pTsdb->mutex);
+    return 0;
+  }
+
+  // set flag and copy
+  tsdbBeginTaskOnFileSet(pTsdb, rtnArg->fid, EVA_TASK_RETENTION, &fset);
   if (fset && (code = tsdbTFileSetInitCopy(pTsdb, fset, &rtner.fset))) {
     (void)taosThreadMutexUnlock(&pTsdb->mutex);
     TSDB_CHECK_CODE(code, lino, _exit);
   }
+
   (void)taosThreadMutexUnlock(&pTsdb->mutex);
 
   // do retention
@@ -346,7 +356,7 @@ static int32_t tsdbRetention(void *arg) {
 _exit:
   if (rtner.fset) {
     (void)taosThreadMutexLock(&pTsdb->mutex);
-    tsdbFinishTaskOnFileSet(pTsdb, rtnArg->fid);
+    tsdbFinishTaskOnFileSet(pTsdb, rtnArg->fid, EVA_TASK_RETENTION);
     (void)taosThreadMutexUnlock(&pTsdb->mutex);
   }
 
@@ -364,26 +374,29 @@ static int32_t tsdbAsyncRetentionImpl(STsdb *tsdb, int64_t now, bool s3Migrate)
   int32_t code = 0;
   int32_t lino = 0;
 
+  // check if background task is disabled
+  if (tsdb->bgTaskDisabled) {
+    tsdbInfo("vgId:%d, background task is disabled, skip retention", TD_VID(tsdb->pVnode));
+    return 0;
+  }
+
   STFileSet *fset;
+  TARRAY2_FOREACH(tsdb->pFS->fSetArr, fset) {
+    SRtnArg *arg = taosMemoryMalloc(sizeof(*arg));
+    if (arg == NULL) {
+      TAOS_CHECK_GOTO(terrno, &lino, _exit);
+    }
 
-  if (!tsdb->bgTaskDisabled) {
-    TARRAY2_FOREACH(tsdb->pFS->fSetArr, fset) {
-      TAOS_CHECK_GOTO(tsdbTFileSetOpenChannel(fset), &lino, _exit);
+    arg->tsdb = tsdb;
+    arg->now = now;
+    arg->fid = fset->fid;
+    arg->s3Migrate = s3Migrate;
 
-      SRtnArg *arg = taosMemoryMalloc(sizeof(*arg));
-      if (arg == NULL) {
-        TAOS_CHECK_GOTO(terrno, &lino, _exit);
-      }
-
-      arg->tsdb = tsdb;
-      arg->now = now;
-      arg->fid = fset->fid;
-      arg->s3Migrate = s3Migrate;
-
-      if ((code = vnodeAsync(&fset->channel, EVA_PRIORITY_LOW, tsdbRetention, tsdbRetentionCancel, arg, NULL))) {
-        taosMemoryFree(arg);
-        TSDB_CHECK_CODE(code, lino, _exit);
-      }
+    code = vnodeAsync(RETENTION_TASK_ASYNC, EVA_PRIORITY_LOW, tsdbRetention, tsdbRetentionCancel, arg,
+                      &fset->retentionTask);
+    if (code) {
+      taosMemoryFree(arg);
+      TSDB_CHECK_CODE(code, lino, _exit);
     }
   }
 

source/dnode/vnode/src/vnd/vnodeAsync.c

@@ -118,9 +118,19 @@ struct SVAsync {
   SVHashTable *taskTable;
 };
 
-SVAsync *vnodeAsyncs[3];
+struct {
+  const char *label;
+  SVAsync    *async;
+} GVnodeAsyncs[] = {
+    [0] = {NULL, NULL},
+    [1] = {"vnode-commit", NULL},
+    [2] = {"vnode-merge", NULL},
+    [3] = {"vnode-compact", NULL},
+    [4] = {"vnode-retention", NULL},
+};
+
 #define MIN_ASYNC_ID 1
-#define MAX_ASYNC_ID (sizeof(vnodeAsyncs) / sizeof(vnodeAsyncs[0]) - 1)
+#define MAX_ASYNC_ID (sizeof(GVnodeAsyncs) / sizeof(GVnodeAsyncs[0]) - 1)
 
 static void vnodeAsyncTaskDone(SVAsync *async, SVATask *task) {
   int32_t ret;
@@ -447,36 +457,47 @@ static void vnodeAsyncLaunchWorker(SVAsync *async) {
   }
 }
 
-int32_t vnodeAsyncOpen(int32_t numOfThreads) {
+int32_t vnodeAsyncOpen() {
   int32_t code = 0;
   int32_t lino = 0;
 
-  // vnode-commit
-  code = vnodeAsyncInit(&vnodeAsyncs[1], "vnode-commit");
-  TSDB_CHECK_CODE(code, lino, _exit);
+  int32_t numOfThreads[] = {
+      0,                        //
+      tsNumOfCommitThreads,     // vnode-commit
+      tsNumOfCommitThreads,     // vnode-merge
+      tsNumOfCompactThreads,    // vnode-compact
+      tsNumOfRetentionThreads,  // vnode-retention
+  };
 
-  code = vnodeAsyncSetWorkers(1, numOfThreads);
-  TSDB_CHECK_CODE(code, lino, _exit);
+  for (int32_t i = 1; i < sizeof(GVnodeAsyncs) / sizeof(GVnodeAsyncs[0]); i++) {
+    code = vnodeAsyncInit(&GVnodeAsyncs[i].async, GVnodeAsyncs[i].label);
+    TSDB_CHECK_CODE(code, lino, _exit);
 
-  // vnode-merge
-  code = vnodeAsyncInit(&vnodeAsyncs[2], "vnode-merge");
-  TSDB_CHECK_CODE(code, lino, _exit);
-
-  code = vnodeAsyncSetWorkers(2, numOfThreads);
-  TSDB_CHECK_CODE(code, lino, _exit);
+    code = vnodeAsyncSetWorkers(i, numOfThreads[i]);
+    TSDB_CHECK_CODE(code, lino, _exit);
+  }
 
 _exit:
   return code;
 }
 
 void vnodeAsyncClose() {
-  int32_t ret;
-  ret = vnodeAsyncDestroy(&vnodeAsyncs[1]);
-  ret = vnodeAsyncDestroy(&vnodeAsyncs[2]);
+  for (int32_t i = 1; i < sizeof(GVnodeAsyncs) / sizeof(GVnodeAsyncs[0]); i++) {
+    int32_t ret = vnodeAsyncDestroy(&GVnodeAsyncs[i].async);
+  }
 }
 
-int32_t vnodeAsync(SVAChannelID *channelID, EVAPriority priority, int32_t (*execute)(void *), void (*cancel)(void *),
-                   void *arg, SVATaskID *taskID) {
+int32_t vnodeAsync(int64_t async, EVAPriority priority, int32_t (*execute)(void *), void (*complete)(void *), void *arg,
+                   SVATaskID *taskID) {
+  SVAChannelID channelID = {
+      .async = async,
+      .id = 0,
+  };
+  return vnodeAsyncC(&channelID, priority, execute, complete, arg, taskID);
+}
+
+int32_t vnodeAsyncC(SVAChannelID *channelID, EVAPriority priority, int32_t (*execute)(void *), void (*cancel)(void *),
+                    void *arg, SVATaskID *taskID) {
   if (channelID == NULL || channelID->async < MIN_ASYNC_ID || channelID->async > MAX_ASYNC_ID || execute == NULL ||
       channelID->id < 0) {
     return TSDB_CODE_INVALID_PARA;
@@ -484,7 +505,7 @@ int32_t vnodeAsync(SVAChannelID *channelID, EVAPriority priority, int32_t (*exec
 
   int32_t  ret;
   int64_t  id;
-  SVAsync *async = vnodeAsyncs[channelID->async];
+  SVAsync *async = GVnodeAsyncs[channelID->async].async;
 
   // create task object
   SVATask *task = (SVATask *)taosMemoryCalloc(1, sizeof(SVATask));
@@ -594,7 +615,7 @@ void vnodeAWait(SVATaskID *taskID) {
     return;
   }
 
-  SVAsync *async = vnodeAsyncs[taskID->async];
+  SVAsync *async = GVnodeAsyncs[taskID->async].async;
   SVATask *task = NULL;
   SVATask  task2 = {
        .taskId = taskID->id,
@@ -623,7 +644,7 @@ int32_t vnodeACancel(SVATaskID *taskID) {
   }
 
   int32_t  ret = 0;
-  SVAsync *async = vnodeAsyncs[taskID->async];
+  SVAsync *async = GVnodeAsyncs[taskID->async].async;
   SVATask *task = NULL;
   SVATask  task2 = {
        .taskId = taskID->id,
@@ -660,7 +681,7 @@ int32_t vnodeAsyncSetWorkers(int64_t asyncID, int32_t numWorkers) {
     return TSDB_CODE_INVALID_PARA;
   }
   int32_t  ret;
-  SVAsync *async = vnodeAsyncs[asyncID];
+  SVAsync *async = GVnodeAsyncs[asyncID].async;
   (void)taosThreadMutexLock(&async->mutex);
   async->numWorkers = numWorkers;
   if (async->numIdleWorkers > 0) {
@@ -676,7 +697,7 @@ int32_t vnodeAChannelInit(int64_t asyncID, SVAChannelID *channelID) {
     return TSDB_CODE_INVALID_PARA;
   }
 
-  SVAsync *async = vnodeAsyncs[asyncID];
+  SVAsync *async = GVnodeAsyncs[asyncID].async;
 
   // create channel object
   SVAChannel *channel = (SVAChannel *)taosMemoryMalloc(sizeof(SVAChannel));
@@ -722,7 +743,7 @@ int32_t vnodeAChannelDestroy(SVAChannelID *channelID, bool waitRunning) {
     return TSDB_CODE_INVALID_PARA;
   }
 
-  SVAsync    *async = vnodeAsyncs[channelID->async];
+  SVAsync    *async = GVnodeAsyncs[channelID->async].async;
   SVAChannel *channel = NULL;
   SVAChannel  channel2 = {
        .channelId = channelID->id,
@@ -806,4 +827,19 @@ int32_t vnodeAChannelDestroy(SVAChannelID *channelID, bool waitRunning) {
   channelID->async = 0;
   channelID->id = 0;
   return 0;
+}
+
+const char *vnodeGetATaskName(EVATaskT taskType) {
+  switch (taskType) {
+    case EVA_TASK_COMMIT:
+      return "vnode-commit";
+    case EVA_TASK_MERGE:
+      return "vnode-merge";
+    case EVA_TASK_COMPACT:
+      return "vnode-compact";
+    case EVA_TASK_RETENTION:
+      return "vnode-retention";
+    default:
+      return "unknown";
+  }
 }

source/dnode/vnode/src/vnd/vnodeCommit.c

@@ -389,8 +389,7 @@ int vnodeAsyncCommit(SVnode *pVnode) {
   TSDB_CHECK_CODE(code, lino, _exit);
 
   // schedule the task
-  code =
-      vnodeAsync(&pVnode->commitChannel, EVA_PRIORITY_HIGH, vnodeCommit, vnodeCommitCancel, pInfo, &pVnode->commitTask);
+  code = vnodeAsync(COMMIT_TASK_ASYNC, EVA_PRIORITY_HIGH, vnodeCommit, vnodeCommitCancel, pInfo, &pVnode->commitTask);
   TSDB_CHECK_CODE(code, lino, _exit);
 
 _exit:

source/dnode/vnode/src/vnd/vnodeModule.c

@@ -20,14 +20,13 @@
 
 static volatile int32_t VINIT = 0;
 
-int vnodeInit(int nthreads, StopDnodeFp stopDnodeFp) {
+int vnodeInit(StopDnodeFp stopDnodeFp) {
   if (atomic_val_compare_exchange_32(&VINIT, 0, 1)) {
     return 0;
   }
 
-  TAOS_CHECK_RETURN(vnodeAsyncOpen(nthreads));
+  TAOS_CHECK_RETURN(vnodeAsyncOpen());
   TAOS_CHECK_RETURN(walInit(stopDnodeFp));
-  TAOS_CHECK_RETURN(tsdbInit());
 
   monInitVnode();
 
@@ -36,7 +35,6 @@ int vnodeInit(int nthreads, StopDnodeFp stopDnodeFp) {
 
 void vnodeCleanup() {
   if (atomic_val_compare_exchange_32(&VINIT, 1, 0) == 0) return;
-  tsdbCleanUp();
   vnodeAsyncClose();
   walCleanUp();
   smaCleanUp();

source/dnode/vnode/src/vnd/vnodeOpen.c

@@ -438,11 +438,6 @@ SVnode *vnodeOpen(const char *path, int32_t diskPrimary, STfs *pTfs, SMsgCb msgC
   (void)taosThreadMutexInit(&pVnode->mutex, NULL);
   (void)taosThreadCondInit(&pVnode->poolNotEmpty, NULL);
 
-  if (vnodeAChannelInit(1, &pVnode->commitChannel) != 0) {
-    vError("vgId:%d, failed to init commit channel", TD_VID(pVnode));
-    goto _err;
-  }
-
   int8_t rollback = vnodeShouldRollback(pVnode);
 
   // open buffer pool
@@ -558,10 +553,6 @@ void vnodePostClose(SVnode *pVnode) { vnodeSyncPostClose(pVnode); }
 void vnodeClose(SVnode *pVnode) {
   if (pVnode) {
     vnodeAWait(&pVnode->commitTask);
-    if (vnodeAChannelDestroy(&pVnode->commitChannel, true) != 0) {
-      vError("vgId:%d, failed to destroy commit channel", TD_VID(pVnode));
-    }
-
     vnodeSyncClose(pVnode);
     vnodeQueryClose(pVnode);
     tqClose(pVnode->pTq);

source/dnode/vnode/src/vnd/vnodeSnapshot.c

@@ -597,13 +597,11 @@ extern void    tsdbEnableBgTask(STsdb *pTsdb);
 static int32_t vnodeCancelAndDisableAllBgTask(SVnode *pVnode) {
   TAOS_CHECK_RETURN(tsdbDisableAndCancelAllBgTask(pVnode->pTsdb));
   TAOS_CHECK_RETURN(vnodeSyncCommit(pVnode));
-  TAOS_CHECK_RETURN(vnodeAChannelDestroy(&pVnode->commitChannel, true));
   return 0;
 }
 
 static int32_t vnodeEnableBgTask(SVnode *pVnode) {
   tsdbEnableBgTask(pVnode->pTsdb);
-  TAOS_CHECK_RETURN(vnodeAChannelInit(1, &pVnode->commitChannel));
   return 0;
 }
 

source/libs/catalog/CMakeLists.txt

@@ -11,6 +11,6 @@ target_link_libraries(
     PRIVATE os util transport qcom nodes
 )
 
-# if(${BUILD_TEST})
+#if(${BUILD_TEST})
 # ADD_SUBDIRECTORY(test)
-# endif(${BUILD_TEST})
+#endif(${BUILD_TEST})

source/libs/catalog/test/catalogTests.cpp

@@ -162,7 +162,7 @@ void ctgTestInitLogFile() {
   (void)ctgdEnableDebug("cache", true);
   (void)ctgdEnableDebug("lock", true);
 
-  if (taosInitLog(defaultLogFileNamePrefix, maxLogFileNum) < 0) {
+  if (taosInitLog(defaultLogFileNamePrefix, 1, false) < 0) {
     (void)printf("failed to open log file in directory:%s\n", tsLogDir);
     ASSERT(0);
   }

source/libs/function/src/tudf.c

@@ -668,8 +668,8 @@ int32_t encodeUdfCallRequest(void **buf, const SUdfCallRequest *call) {
     len += tEncodeDataBlock(buf, &call->block);
     len += encodeUdfInterBuf(buf, &call->interBuf);
   } else if (call->callType == TSDB_UDF_CALL_AGG_MERGE) {
-    len += encodeUdfInterBuf(buf, &call->interBuf);
-    len += encodeUdfInterBuf(buf, &call->interBuf2);
+    // len += encodeUdfInterBuf(buf, &call->interBuf);
+    // len += encodeUdfInterBuf(buf, &call->interBuf2);
   } else if (call->callType == TSDB_UDF_CALL_AGG_FIN) {
     len += encodeUdfInterBuf(buf, &call->interBuf);
   }
@@ -690,10 +690,10 @@ void *decodeUdfCallRequest(const void *buf, SUdfCallRequest *call) {
       buf = tDecodeDataBlock(buf, &call->block);
       buf = decodeUdfInterBuf(buf, &call->interBuf);
       break;
-    case TSDB_UDF_CALL_AGG_MERGE:
-      buf = decodeUdfInterBuf(buf, &call->interBuf);
-      buf = decodeUdfInterBuf(buf, &call->interBuf2);
-      break;
+    // case TSDB_UDF_CALL_AGG_MERGE:
+    //   buf = decodeUdfInterBuf(buf, &call->interBuf);
+    //   buf = decodeUdfInterBuf(buf, &call->interBuf2);
+    //   break;
     case TSDB_UDF_CALL_AGG_FIN:
       buf = decodeUdfInterBuf(buf, &call->interBuf);
       break;
@@ -779,9 +779,9 @@ int32_t encodeUdfCallResponse(void **buf, const SUdfCallResponse *callRsp) {
     case TSDB_UDF_CALL_AGG_PROC:
       len += encodeUdfInterBuf(buf, &callRsp->resultBuf);
       break;
-    case TSDB_UDF_CALL_AGG_MERGE:
-      len += encodeUdfInterBuf(buf, &callRsp->resultBuf);
-      break;
+    // case TSDB_UDF_CALL_AGG_MERGE:
+    //   len += encodeUdfInterBuf(buf, &callRsp->resultBuf);
+    //   break;
     case TSDB_UDF_CALL_AGG_FIN:
       len += encodeUdfInterBuf(buf, &callRsp->resultBuf);
       break;
@@ -801,9 +801,9 @@ void *decodeUdfCallResponse(const void *buf, SUdfCallResponse *callRsp) {
     case TSDB_UDF_CALL_AGG_PROC:
       buf = decodeUdfInterBuf(buf, &callRsp->resultBuf);
       break;
-    case TSDB_UDF_CALL_AGG_MERGE:
-      buf = decodeUdfInterBuf(buf, &callRsp->resultBuf);
-      break;
+    // case TSDB_UDF_CALL_AGG_MERGE:
+    //   buf = decodeUdfInterBuf(buf, &callRsp->resultBuf);
+    //   break;
     case TSDB_UDF_CALL_AGG_FIN:
       buf = decodeUdfInterBuf(buf, &callRsp->resultBuf);
       break;
@@ -1129,8 +1129,9 @@ int32_t callUdf(UdfcFuncHandle handle, int8_t callType, SSDataBlock *input, SUdf
                 SSDataBlock *output, SUdfInterBuf *newState);
 int32_t doCallUdfAggInit(UdfcFuncHandle handle, SUdfInterBuf *interBuf);
 int32_t doCallUdfAggProcess(UdfcFuncHandle handle, SSDataBlock *block, SUdfInterBuf *state, SUdfInterBuf *newState);
-int32_t doCallUdfAggMerge(UdfcFuncHandle handle, SUdfInterBuf *interBuf1, SUdfInterBuf *interBuf2,
-                          SUdfInterBuf *resultBuf);
+// udf todo:  aggmerge
+// int32_t doCallUdfAggMerge(UdfcFuncHandle handle, SUdfInterBuf *interBuf1, SUdfInterBuf *interBuf2,
+//                           SUdfInterBuf *resultBuf);
 int32_t doCallUdfAggFinalize(UdfcFuncHandle handle, SUdfInterBuf *interBuf, SUdfInterBuf *resultData);
 int32_t doCallUdfScalarFunc(UdfcFuncHandle handle, SScalarParam *input, int32_t numOfCols, SScalarParam *output);
 int32_t callUdfScalarFunc(char *udfName, SScalarParam *input, int32_t numOfCols, SScalarParam *output);
@@ -2176,11 +2177,11 @@ int32_t callUdf(UdfcFuncHandle handle, int8_t callType, SSDataBlock *input, SUdf
       req->interBuf = *state;
       break;
     }
-    case TSDB_UDF_CALL_AGG_MERGE: {
-      req->interBuf = *state;
-      req->interBuf2 = *state2;
-      break;
-    }
+    // case TSDB_UDF_CALL_AGG_MERGE: {
+    //   req->interBuf = *state;
+    //   req->interBuf2 = *state2;
+    //   break;
+    // }
     case TSDB_UDF_CALL_AGG_FIN: {
       req->interBuf = *state;
       break;
@@ -2205,10 +2206,10 @@ int32_t callUdf(UdfcFuncHandle handle, int8_t callType, SSDataBlock *input, SUdf
         *newState = rsp->resultBuf;
         break;
       }
-      case TSDB_UDF_CALL_AGG_MERGE: {
-        *newState = rsp->resultBuf;
-        break;
-      }
+      // case TSDB_UDF_CALL_AGG_MERGE: {
+      //   *newState = rsp->resultBuf;
+      //   break;
+      // }
       case TSDB_UDF_CALL_AGG_FIN: {
         *newState = rsp->resultBuf;
         break;
@@ -2241,12 +2242,13 @@ int32_t doCallUdfAggProcess(UdfcFuncHandle handle, SSDataBlock *block, SUdfInter
 
 // input: interbuf1, interbuf2
 // output: resultBuf
-int32_t doCallUdfAggMerge(UdfcFuncHandle handle, SUdfInterBuf *interBuf1, SUdfInterBuf *interBuf2,
-                          SUdfInterBuf *resultBuf) {
-  int8_t  callType = TSDB_UDF_CALL_AGG_MERGE;
-  int32_t err = callUdf(handle, callType, NULL, interBuf1, interBuf2, NULL, resultBuf);
-  return err;
-}
+// udf todo:  aggmerge
+// int32_t doCallUdfAggMerge(UdfcFuncHandle handle, SUdfInterBuf *interBuf1, SUdfInterBuf *interBuf2,
+//                           SUdfInterBuf *resultBuf) {
+//   int8_t  callType = TSDB_UDF_CALL_AGG_MERGE;
+//   int32_t err = callUdf(handle, callType, NULL, interBuf1, interBuf2, NULL, resultBuf);
+//   return err;
+// }
 
 // input: interBuf
 // output: resultData

source/libs/function/src/udfd.c

@@ -194,17 +194,17 @@ int32_t udfdCPluginUdfAggProc(SUdfDataBlock *block, SUdfInterBuf *interBuf, SUdf
   }
 }
 
-int32_t udfdCPluginUdfAggMerge(SUdfInterBuf *inputBuf1, SUdfInterBuf *inputBuf2, SUdfInterBuf *outputBuf,
-                               void *udfCtx) {
-  TAOS_UDF_CHECK_PTR_RCODE(inputBuf1, inputBuf2, outputBuf, udfCtx);
-  SUdfCPluginCtx *ctx = udfCtx;
-  if (ctx->aggMergeFunc) {
-    return ctx->aggMergeFunc(inputBuf1, inputBuf2, outputBuf);
-  } else {
-    fnError("udfd c plugin aggregation merge not implemented");
-    return TSDB_CODE_UDF_FUNC_EXEC_FAILURE;
-  }
-}
+// int32_t udfdCPluginUdfAggMerge(SUdfInterBuf *inputBuf1, SUdfInterBuf *inputBuf2, SUdfInterBuf *outputBuf,
+//                                void *udfCtx) {
+//   TAOS_UDF_CHECK_PTR_RCODE(inputBuf1, inputBuf2, outputBuf, udfCtx);
+//   SUdfCPluginCtx *ctx = udfCtx;
+//   if (ctx->aggMergeFunc) {
+//     return ctx->aggMergeFunc(inputBuf1, inputBuf2, outputBuf);
+//   } else {
+//     fnError("udfd c plugin aggregation merge not implemented");
+//     return TSDB_CODE_UDF_FUNC_EXEC_FAILURE;
+//   }
+// }
 
 int32_t udfdCPluginUdfAggFinish(SUdfInterBuf *buf, SUdfInterBuf *resultData, void *udfCtx) {
   TAOS_UDF_CHECK_PTR_RCODE(buf, resultData, udfCtx);
@@ -378,7 +378,7 @@ int32_t udfdInitializeCPlugin(SUdfScriptPlugin *plugin) {
   plugin->udfScalarProcFunc = udfdCPluginUdfScalarProc;
   plugin->udfAggStartFunc = udfdCPluginUdfAggStart;
   plugin->udfAggProcFunc = udfdCPluginUdfAggProc;
-  plugin->udfAggMergeFunc = udfdCPluginUdfAggMerge;
+  // plugin->udfAggMergeFunc = udfdCPluginUdfAggMerge;
   plugin->udfAggFinishFunc = udfdCPluginUdfAggFinish;
 
   SScriptUdfEnvItem items[1] = {{"LD_LIBRARY_PATH", tsUdfdLdLibPath}};
@@ -889,19 +889,19 @@ void udfdProcessCallRequest(SUvUdfWork *uvUdf, SUdfRequest *request) {
 
       break;
     }
-    case TSDB_UDF_CALL_AGG_MERGE: {
-      SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0};
-      if (outBuf.buf != NULL) {
-        code = udf->scriptPlugin->udfAggMergeFunc(&call->interBuf, &call->interBuf2, &outBuf, udf->scriptUdfCtx);
-        freeUdfInterBuf(&call->interBuf);
-        freeUdfInterBuf(&call->interBuf2);
-        subRsp->resultBuf = outBuf;
-      } else {
-        code = terrno;
-      }
-
-      break;
-    }
+    // case TSDB_UDF_CALL_AGG_MERGE: {
+    //   SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0};
+    //   if (outBuf.buf != NULL) {
+    //     code = udf->scriptPlugin->udfAggMergeFunc(&call->interBuf, &call->interBuf2, &outBuf, udf->scriptUdfCtx);
+    //     freeUdfInterBuf(&call->interBuf);
+    //     freeUdfInterBuf(&call->interBuf2);
+    //     subRsp->resultBuf = outBuf;
+    //   } else {
+    //     code = terrno;
+    //   }
+    // 
+    //   break;
+    // }
     case TSDB_UDF_CALL_AGG_FIN: {
       SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0};
       if (outBuf.buf != NULL) {
@@ -959,10 +959,10 @@ _exit:
       freeUdfInterBuf(&subRsp->resultBuf);
       break;
     }
-    case TSDB_UDF_CALL_AGG_MERGE: {
-      freeUdfInterBuf(&subRsp->resultBuf);
-      break;
-    }
+    // case TSDB_UDF_CALL_AGG_MERGE: {
+    //   freeUdfInterBuf(&subRsp->resultBuf);
+    //   break;
+    // }
     case TSDB_UDF_CALL_AGG_FIN: {
       freeUdfInterBuf(&subRsp->resultBuf);
       break;
@@ -1667,7 +1667,6 @@ static int32_t udfdGlobalDataInit() {
 }
 
 static void udfdGlobalDataDeinit() {
-  taosHashCleanup(global.udfsHash);
   uv_mutex_destroy(&global.udfsMutex);
   uv_mutex_destroy(&global.scriptPluginsMutex);
   taosMemoryFreeClear(global.loop);
@@ -1720,8 +1719,11 @@ void udfdDeinitResidentFuncs() {
     SUdf **udfInHash = taosHashGet(global.udfsHash, funcName, strlen(funcName));
     if (udfInHash) {
       SUdf   *udf = *udfInHash;
-      int32_t code = udf->scriptPlugin->udfDestroyFunc(udf->scriptUdfCtx);
-      fnDebug("udfd destroy function returns %d", code);
+      int32_t code = 0;
+      if (udf->scriptPlugin->udfDestroyFunc) {
+        code = udf->scriptPlugin->udfDestroyFunc(udf->scriptUdfCtx);
+        fnDebug("udfd %s destroy function returns %d", funcName, code);
+      }
       if(taosHashRemove(global.udfsHash, funcName, strlen(funcName)) != 0)
       {
         fnError("udfd remove resident function %s failed", funcName);
@@ -1729,6 +1731,7 @@ void udfdDeinitResidentFuncs() {
       taosMemoryFree(udf);
     }
   }
+  taosHashCleanup(global.udfsHash);
   taosArrayDestroy(global.residentFuncs);
   fnInfo("udfd resident functions are deinit");
 }
@@ -1838,15 +1841,15 @@ int main(int argc, char *argv[]) {
   fnInfo("udfd exit normally");
 
   removeListeningPipe();
-  udfdDeinitScriptPlugins();
 
 _exit:
-  if (globalDataInited) {
-    udfdGlobalDataDeinit();
-  }
   if (residentFuncsInited) {
     udfdDeinitResidentFuncs();
   }
+  udfdDeinitScriptPlugins();
+  if (globalDataInited) {
+    udfdGlobalDataDeinit();
+  }
   if (udfSourceDirInited) {
     udfdDestroyUdfSourceDir();
   }

source/libs/parser/src/parTranslater.c

@@ -11411,7 +11411,7 @@ static int32_t checkStreamQuery(STranslateContext* pCxt, SCreateStreamStmt* pStm
 
   if (pSelect->hasInterpFunc) {
     // Temporary code
-    if (pStmt->pOptions->triggerType != STREAM_TRIGGER_FORCE_WINDOW_CLOSE) {
+    if (tsStreamCoverage == false && pStmt->pOptions->triggerType != STREAM_TRIGGER_FORCE_WINDOW_CLOSE) {
       return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_STREAM_QUERY,
                                      "Stream interp function only support force window close");
     }

source/libs/qworker/src/qwUtil.c

@@ -258,7 +258,7 @@ int32_t qwAddTaskCtxImpl(QW_FPARAMS_DEF, bool acquire, SQWTaskCtx **ctx) {
     }
   }
 
-  atomic_add_fetch_64(&gQueryMgmt.stat.taskInitNum, 1);
+  (void)atomic_add_fetch_64(&gQueryMgmt.stat.taskInitNum, 1);
 
   if (acquire && ctx) {
     QW_RET(qwAcquireTaskCtx(QW_FPARAMS(), ctx));
@@ -283,7 +283,7 @@ void qwFreeTaskHandle(SQWTaskCtx *ctx) {
     qDestroyTask(otaskHandle);
     taosDisableMemPoolUsage();
 
-    atomic_add_fetch_64(&gQueryMgmt.stat.taskExecDestroyNum, 1);
+    (void)atomic_add_fetch_64(&gQueryMgmt.stat.taskExecDestroyNum, 1);
 
     qDebug("task handle destroyed");
   }
@@ -297,7 +297,7 @@ void qwFreeSinkHandle(SQWTaskCtx *ctx) {
     dsDestroyDataSinker(osinkHandle);
     QW_SINK_DISABLE_MEMPOOL();
 
-    atomic_add_fetch_64(&gQueryMgmt.stat.taskSinkDestroyNum, 1);
+    (void)atomic_add_fetch_64(&gQueryMgmt.stat.taskSinkDestroyNum, 1);
     
     qDebug("sink handle destroyed");
   }
@@ -409,6 +409,8 @@ int32_t qwDropTaskCtx(QW_FPARAMS_DEF) {
 
   if (ctx->pJobInfo && TSDB_CODE_SUCCESS != ctx->pJobInfo->errCode) {
     QW_UPDATE_RSP_CODE(ctx, ctx->pJobInfo->errCode);
+  } else {
+    QW_UPDATE_RSP_CODE(ctx, TSDB_CODE_TSC_QUERY_CANCELLED);
   }
 
   atomic_store_ptr(&ctx->taskHandle, NULL);
@@ -428,7 +430,7 @@ int32_t qwDropTaskCtx(QW_FPARAMS_DEF) {
 
   QW_TASK_DLOG_E("task ctx dropped");
   
-  atomic_add_fetch_64(&gQueryMgmt.stat.taskDestroyNum, 1);
+  (void)atomic_add_fetch_64(&gQueryMgmt.stat.taskDestroyNum, 1);
 
   return code;
 }

source/libs/qworker/src/qworker.c

@@ -763,6 +763,11 @@ int32_t qwPreprocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) {
 
   QW_LOCK(QW_WRITE, &ctx->lock);
 
+  if (QW_EVENT_PROCESSED(ctx, QW_EVENT_DROP) || QW_EVENT_RECEIVED(ctx, QW_EVENT_DROP)) {
+    QW_TASK_WLOG("task dropping or already dropped, drop event:%d", QW_GET_EVENT(ctx, QW_EVENT_DROP));
+    QW_ERR_JRET(ctx->rspCode);
+  }
+
   ctx->ctrlConnInfo = qwMsg->connInfo;
   ctx->sId = sId;
   ctx->phase = -1;
@@ -1044,12 +1049,16 @@ _return:
 
   if (code || rsp) {
     bool rsped = false;
+
+    ctx = NULL;
+    (void)qwAcquireTaskCtx(QW_FPARAMS(), &ctx);
+
     if (ctx) {
       qwDbgSimulateRedirect(qwMsg, ctx, &rsped);
       qwDbgSimulateDead(QW_FPARAMS(), ctx, &rsped);
     }
 
-    if (!rsped) {
+    if (!rsped && ctx) {
       code = qwBuildAndSendFetchRsp(ctx, qwMsg->msgType + 1, &qwMsg->connInfo, rsp, dataLen, code);
       if (TSDB_CODE_SUCCESS != code) {
         QW_TASK_ELOG("fetch rsp send fail, msgType:%s, handle:%p, code:%x - %s, dataLen:%d",
@@ -1062,6 +1071,8 @@ _return:
       qwFreeFetchRsp(rsp);
       rsp = NULL;
     }
+
+    qwReleaseTaskCtx(mgmt, ctx);    
   } else {
     // qwQuickRspFetchReq(QW_FPARAMS(), ctx, qwMsg, code);
   }

source/libs/stream/inc/streamInt.h

@@ -250,6 +250,9 @@ void chkptFailedByRetrieveReqToSource(SStreamTask* pTask, int64_t checkpointId);
 // inject stream errors
 void chkptFailedByRetrieveReqToSource(SStreamTask* pTask, int64_t checkpointId);
 
+int32_t uploadCheckpointData(SStreamTask* pTask, int64_t checkpointId, int64_t dbRefId, ECHECKPOINT_BACKUP_TYPE type);
+int32_t chkptTriggerRecvMonitorHelper(SStreamTask* pTask, void* param, SArray** ppNotSendList);
+
 #ifdef __cplusplus
 }
 #endif

source/libs/stream/src/streamCheckpoint.c

@@ -21,7 +21,9 @@
 static int32_t downloadCheckpointDataByName(const char* id, const char* fname, const char* dstName);
 static int32_t deleteCheckpointFile(const char* id, const char* name);
 static int32_t streamTaskUploadCheckpoint(const char* id, const char* path, int64_t checkpointId);
+#ifdef BUILD_NO_CALL
 static int32_t deleteCheckpoint(const char* id);
+#endif
 static int32_t downloadCheckpointByNameS3(const char* id, const char* fname, const char* dstName);
 static int32_t continueDispatchCheckpointTriggerBlock(SStreamDataBlock* pBlock, SStreamTask* pTask);
 static int32_t appendCheckpointIntoInputQ(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId,
@@ -998,7 +1000,7 @@ static int32_t doFindNotSendUpstream(SStreamTask* pTask, SArray* pList, SArray**
   return 0;
 }
 
-static int32_t chkptTriggerRecvMonitorHelper(SStreamTask* pTask, void* param, SArray** ppNotSendList) {
+int32_t chkptTriggerRecvMonitorHelper(SStreamTask* pTask, void* param, SArray** ppNotSendList) {
   const char*            id = pTask->id.idStr;
   SArray*                pList = pTask->upstreamInfo.pList;  // send msg to retrieve checkpoint trigger msg
   SActiveCheckpointInfo* pActiveInfo = pTask->chkInfo.pActiveInfo;
@@ -1492,6 +1494,7 @@ int32_t streamTaskDownloadCheckpointData(const char* id, char* path, int64_t che
   return 0;
 }
 
+#ifdef BUILD_NO_CALL
 int32_t deleteCheckpoint(const char* id) {
   if (id == NULL || strlen(id) == 0) {
     stError("deleteCheckpoint parameters invalid");
@@ -1504,6 +1507,7 @@ int32_t deleteCheckpoint(const char* id) {
   }
   return 0;
 }
+#endif
 
 int32_t deleteCheckpointFile(const char* id, const char* name) {
   char object[128] = {0};

source/libs/stream/test/streamCheckPointTest.cpp

@@ -0,0 +1,270 @@
+#include <gtest/gtest.h>
+#include "tstream.h"
+#include "streamInt.h"
+
+#pragma GCC diagnostic push
+#pragma GCC diagnostic ignored "-Wwrite-strings"
+#pragma GCC diagnostic ignored "-Wunused-function"
+#pragma GCC diagnostic ignored "-Wunused-variable"
+#pragma GCC diagnostic ignored "-Wsign-compare"
+#pragma GCC diagnostic ignored "-Wsign-compare"
+#pragma GCC diagnostic ignored "-Wformat"
+#pragma GCC diagnostic ignored "-Wint-to-pointer-cast"
+#pragma GCC diagnostic ignored "-Wpointer-arith"
+
+void initTaskLock(SStreamTask* pTask) {
+  TdThreadMutexAttr attr = {0};
+  int32_t code = taosThreadMutexAttrInit(&attr);
+  ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+  code = taosThreadMutexAttrSetType(&attr, PTHREAD_MUTEX_RECURSIVE);
+  ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+  code = taosThreadMutexInit(&pTask->lock, &attr);
+  ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+  code = taosThreadMutexAttrDestroy(&attr);
+  ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+}
+
+TEST(streamCheckpointTest, StreamTaskProcessCheckpointTriggerRsp) {
+    SStreamTask* pTask = NULL;
+    int64_t uid = 1111111111111111;
+    SArray* array = taosArrayInit(4, POINTER_BYTES);
+    int32_t code = tNewStreamTask(uid, TASK_LEVEL__SINK, NULL, false, 0, 0, array,
+                                       false, 1, &pTask);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    initTaskLock(pTask);
+
+    code = streamTaskCreateActiveChkptInfo(&pTask->chkInfo.pActiveInfo);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    pTask->chkInfo.pActiveInfo->activeId = 123111;
+    pTask->chkInfo.pActiveInfo->transId = 4561111;
+
+    streamTaskSetStatusReady(pTask);
+    code = streamTaskHandleEvent(pTask->status.pSM, TASK_EVENT_GEN_CHECKPOINT);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    SCheckpointTriggerRsp pRsp;
+    memset(&pRsp, 0, sizeof(SCheckpointTriggerRsp));
+    pRsp.rspCode = TSDB_CODE_SUCCESS;
+    pRsp.checkpointId = 123;
+    pRsp.transId = 456;
+    pRsp.upstreamTaskId = 789;
+
+    code = streamTaskProcessCheckpointTriggerRsp(pTask, &pRsp);
+    ASSERT_NE(code, TSDB_CODE_SUCCESS);
+
+    pRsp.rspCode = TSDB_CODE_FAILED;
+    code = streamTaskProcessCheckpointTriggerRsp(pTask, &pRsp);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    tFreeStreamTask(pTask);
+    taosArrayDestroy(array);
+}
+
+TEST(streamCheckpointTest, StreamTaskSetFailedCheckpointId) {
+    SStreamTask* pTask = NULL;
+    int64_t uid = 1111111111111111;
+    SArray* array = taosArrayInit(4, POINTER_BYTES);
+    int32_t code = tNewStreamTask(uid, TASK_LEVEL__SINK, NULL, false, 0, 0, array,
+                                       false, 1, &pTask);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    initTaskLock(pTask);
+
+    code = streamTaskCreateActiveChkptInfo(&pTask->chkInfo.pActiveInfo);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    SActiveCheckpointInfo* pInfo = pTask->chkInfo.pActiveInfo;
+    pInfo->failedId = 0;
+
+    int64_t failedCheckpointId = 123;
+
+    streamTaskSetFailedCheckpointId(pTask, failedCheckpointId);
+    ASSERT_EQ(pInfo->failedId, failedCheckpointId);
+
+    streamTaskSetFailedCheckpointId(pTask, 0);
+    ASSERT_EQ(pInfo->failedId, failedCheckpointId);
+
+    streamTaskSetFailedCheckpointId(pTask, pInfo->failedId - 1);
+    ASSERT_EQ(pInfo->failedId, failedCheckpointId);
+    tFreeStreamTask(pTask);
+    taosArrayDestroy(array);
+}
+
+TEST(UploadCheckpointDataTest, UploadSuccess) {
+    streamMetaInit();
+    SStreamTask* pTask = NULL;
+    int64_t uid = 1111111111111111;
+    SArray* array = taosArrayInit(4, POINTER_BYTES);
+    int32_t code = tNewStreamTask(uid, TASK_LEVEL__SINK, NULL, false, 0, 0, array,
+                                       false, 1, &pTask);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    initTaskLock(pTask);
+
+    code = streamTaskCreateActiveChkptInfo(&pTask->chkInfo.pActiveInfo);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    int64_t checkpointId = 123;
+    int64_t dbRefId = 1;
+    ECHECKPOINT_BACKUP_TYPE type = DATA_UPLOAD_S3;
+
+    STaskDbWrapper* pBackend = NULL;
+    int64_t processVer = -1;
+    const char *path = "/tmp/backend3/stream";
+    code = streamMetaOpen((path), NULL, NULL, NULL, 0, 0, NULL, &pTask->pMeta);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    SStreamState *pState = streamStateOpen((char *)path, pTask, 0, 0);
+    ASSERT(pState != NULL);
+
+    pTask->pBackend = pState->pTdbState->pOwner->pBackend;
+
+    code = taskDbDoCheckpoint(pTask->pBackend, checkpointId, 0);
+    ASSERT(code == 0);
+
+    int32_t result = uploadCheckpointData(pTask, checkpointId, dbRefId, type);
+
+    EXPECT_EQ(result, TSDB_CODE_SUCCESS) << "uploadCheckpointData should return 0 on success";
+    tFreeStreamTask(pTask);
+    taosRemoveDir(path);
+    streamStateClose(pState, true);
+    taosArrayDestroy(array);
+}
+
+TEST(UploadCheckpointDataTest, UploadDisabled) {
+    SStreamTask* pTask = NULL;
+    int64_t uid = 2222222222222;
+    SArray* array = taosArrayInit(4, POINTER_BYTES);
+    int32_t code = tNewStreamTask(uid, TASK_LEVEL__SINK, NULL, false, 0, 0, array,
+                                       false, 1, &pTask);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    initTaskLock(pTask);
+
+    code = streamTaskCreateActiveChkptInfo(&pTask->chkInfo.pActiveInfo);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    int64_t checkpointId = 123;
+    int64_t dbRefId = 1;
+
+    STaskDbWrapper* pBackend = NULL;
+    int64_t processVer = -1;
+    const char *path = "/tmp/backend4/stream";
+    code = streamMetaOpen((path), NULL, NULL, NULL, 0, 0, NULL, &pTask->pMeta);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    SStreamState *pState = streamStateOpen((char *)path, pTask, 0, 0);
+    ASSERT(pState != NULL);
+
+    pTask->pBackend = pState->pTdbState->pOwner->pBackend;
+
+    code = taskDbDoCheckpoint(pTask->pBackend, checkpointId, 0);
+    ASSERT(code == 0);
+
+    ECHECKPOINT_BACKUP_TYPE type = DATA_UPLOAD_DISABLE;
+
+    int32_t result = uploadCheckpointData(pTask, checkpointId, dbRefId, type);
+
+    EXPECT_NE(result, TSDB_CODE_SUCCESS) << "uploadCheckpointData should return 0 when backup type is disabled";
+
+    streamStateClose(pState, true);
+    tFreeStreamTask(pTask);
+    taosArrayDestroy(array);
+}
+
+TEST(StreamTaskAlreadySendTriggerTest, AlreadySendTrigger) {
+    SStreamTask* pTask = NULL;
+    int64_t uid = 2222222222222;
+    SArray* array = taosArrayInit(4, POINTER_BYTES);
+    int32_t code = tNewStreamTask(uid, TASK_LEVEL__SINK, NULL, false, 0, 0, array,
+                                       false, 1, &pTask);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    initTaskLock(pTask);
+
+    code = streamTaskCreateActiveChkptInfo(&pTask->chkInfo.pActiveInfo);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    pTask->chkInfo.pActiveInfo->activeId = 123111;
+    pTask->chkInfo.pActiveInfo->transId = 4561111;
+
+    streamTaskSetStatusReady(pTask);
+    code = streamTaskHandleEvent(pTask->status.pSM, TASK_EVENT_GEN_CHECKPOINT);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    int32_t downstreamNodeId = 1;
+    int64_t sendingCheckpointId = 123;
+    TSKEY ts = taosGetTimestampMs();
+
+    STaskTriggerSendInfo triggerInfo;
+    triggerInfo.sendTs = ts;
+    triggerInfo.recved = false;
+    triggerInfo.nodeId = downstreamNodeId;
+
+    taosArrayPush(pTask->chkInfo.pActiveInfo->pDispatchTriggerList, &triggerInfo);
+
+    pTask->chkInfo.pActiveInfo->dispatchTrigger = true;
+    bool result = streamTaskAlreadySendTrigger(pTask, downstreamNodeId);
+
+    EXPECT_TRUE(result) << "The trigger message should have been sent to the downstream node";
+
+    tFreeStreamTask(pTask);
+    taosArrayDestroy(array);
+}
+
+TEST(ChkptTriggerRecvMonitorHelperTest, chkptTriggerRecvMonitorHelper) {
+    SStreamTask* pTask = NULL;
+    int64_t uid = 2222222222222;
+    SArray* array = taosArrayInit(4, POINTER_BYTES);
+    int32_t code = tNewStreamTask(uid, TASK_LEVEL__SINK, NULL, false, 0, 0, array,
+                                       false, 1, &pTask);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    initTaskLock(pTask);
+
+    const char *path = "/tmp/backend5/stream";
+    code = streamMetaOpen((path), NULL, NULL, NULL, 0, 0, NULL, &pTask->pMeta);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    code = streamTaskCreateActiveChkptInfo(&pTask->chkInfo.pActiveInfo);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    pTask->chkInfo.pActiveInfo->activeId = 123111;
+    pTask->chkInfo.pActiveInfo->chkptTriggerMsgTmr.launchChkptId = pTask->chkInfo.pActiveInfo->activeId;
+    pTask->chkInfo.pActiveInfo->transId = 4561111;
+    pTask->chkInfo.startTs = 11111;
+
+    streamTaskSetStatusReady(pTask);
+    code = streamTaskHandleEvent(pTask->status.pSM, TASK_EVENT_GEN_CHECKPOINT);
+    ASSERT_EQ(code, TSDB_CODE_SUCCESS);
+
+    int32_t downstreamNodeId = 1;
+    int64_t sendingCheckpointId = 123;
+    TSKEY ts = taosGetTimestampMs();
+
+    STaskTriggerSendInfo triggerInfo;
+    triggerInfo.sendTs = ts;
+    triggerInfo.recved = false;
+    triggerInfo.nodeId = downstreamNodeId;
+
+    taosArrayPush(pTask->chkInfo.pActiveInfo->pDispatchTriggerList, &triggerInfo);
+
+    pTask->chkInfo.pActiveInfo->dispatchTrigger = true;
+    SArray* array1 = NULL;
+    code = chkptTriggerRecvMonitorHelper(pTask, NULL, &array1);
+    EXPECT_EQ(code, TSDB_CODE_SUCCESS);
+
+    pTask->pMeta->fatalInfo.code = TSDB_CODE_SUCCESS;
+    streamSetFatalError(pTask->pMeta, code, __func__, __LINE__);
+
+    pTask->pMeta->fatalInfo.code = TSDB_CODE_FAILED;
+    streamSetFatalError(pTask->pMeta, code, __func__, __LINE__);
+    tFreeStreamTask(pTask);
+    taosArrayDestroy(array);
+    taosArrayDestroy(array1);
+}

source/libs/tdb/src/db/tdbBtree.c

@@ -1491,7 +1491,7 @@ static int tdbBtreeDecodePayload(SPage *pPage, const SCell *pCell, int nHeader,
         ofpCell = tdbPageGetCell(ofp, 0);
 
         int lastKeyPage = 0;
-        if (nLeftKey <= maxLocal - sizeof(SPgno)) {
+        if (nLeftKey <= ofp->maxLocal - sizeof(SPgno)) {
           bytes = nLeftKey;
           lastKeyPage = 1;
           lastKeyPageSpace = ofp->maxLocal - sizeof(SPgno) - nLeftKey;

tests/parallel_test/cases.task

@@ -1422,34 +1422,36 @@
 ,,y,script,./test.sh -f tsim/stream/sliding.sim
 ,,y,script,./test.sh -f tsim/stream/state0.sim
 ,,y,script,./test.sh -f tsim/stream/state1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpDelete0.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpDelete1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpDelete2.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpError.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpDelete0.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpDelete1.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpDelete2.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpError.sim
 ,,y,script,./test.sh -f tsim/stream/streamInterpForceWindowClose.sim
 ,,y,script,./test.sh -f tsim/stream/streamInterpForceWindowClose1.sim
 ,,y,script,./test.sh -f tsim/stream/streamInterpFwcError.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpHistory.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpHistory.sim
 #,,y,script,./test.sh -f tsim/stream/streamInterpHistory1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpLarge.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpLinear0.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpNext0.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpOther.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpLarge.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpLinear0.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpNext0.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpOther.sim
 #,,y,script,./test.sh -f tsim/stream/streamInterpOther1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpPartitionBy0.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpPartitionBy0.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpPartitionBy1.sim
 #,,y,script,./test.sh -f tsim/stream/streamInterpPrev0.sim
 #,,y,script,./test.sh -f tsim/stream/streamInterpPrev1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey0.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey2.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey3.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpUpdate.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpUpdate1.sim
-#,,y,script,./test.sh -f tsim/stream/streamInterpValue0.sim
-#,,y,script,./test.sh -f tsim/stream/streamPrimaryKey0.sim
-#,,y,script,./test.sh -f tsim/stream/streamPrimaryKey1.sim
-#,,y,script,./test.sh -f tsim/stream/streamPrimaryKey2.sim
-#,,y,script,./test.sh -f tsim/stream/streamPrimaryKey3.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey0.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey1.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey2.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpPrimaryKey3.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpUpdate.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpUpdate1.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpUpdate2.sim
+,,y,script,./test.sh -f tsim/stream/streamInterpValue0.sim
+,,y,script,./test.sh -f tsim/stream/streamPrimaryKey0.sim
+,,y,script,./test.sh -f tsim/stream/streamPrimaryKey1.sim
+,,y,script,./test.sh -f tsim/stream/streamPrimaryKey2.sim
+,,y,script,./test.sh -f tsim/stream/streamPrimaryKey3.sim
 ,,y,script,./test.sh -f tsim/stream/streamTwaError.sim
 ,,y,script,./test.sh -f tsim/stream/streamTwaFwcFill.sim
 ,,y,script,./test.sh -f tsim/stream/streamTwaFwcFillPrimaryKey.sim

tests/script/api/test.sh

@@ -0,0 +1,18 @@
+#!/bin/bash
+
+make clean
+
+make
+
+pgrep taosd || taosd >> /dev/null 2>&1 &
+
+sleep 10
+
+./dbTableRoute localhost
+./batchprepare localhost
+./stmt-crash localhost
+./insertSameTs localhost
+./passwdTest localhost
+./whiteListTest localhost
+./tmqViewTest
+

tests/script/tsim/stream/streamInterpDelete0.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpDelete1.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpDelete2.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpError.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step2
 
 sql create database test2 vgroups 1;

tests/script/tsim/stream/streamInterpHistory.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpLarge.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpLinear0.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpNext0.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpOther.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 4;

tests/script/tsim/stream/streamInterpPartitionBy0.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step prev
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpPartitionBy1.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step NULL
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpPrimaryKey0.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpPrimaryKey1.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpPrimaryKey2.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpPrimaryKey3.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpUpdate.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpUpdate1.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpUpdate2.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/stream/streamInterpValue0.sim

@@ -4,6 +4,8 @@ system sh/exec.sh -n dnode1 -s start
 sleep 50
 sql connect
 
+sql alter local 'streamCoverage' '1';
+
 print step1
 print =============== create database
 sql create database test vgroups 1;

tests/script/tsim/tagindex/indexOverflow.sim

@@ -76,7 +76,23 @@ while $i < $maxTinyLimit
   $i = $i + 1
 endw
 
+print =============== create database with big tags
+$bigTagDb = db
+$bigTagStb = stb
+$bigTagTb = tb
+sql create database $bigTagDb
+sql use $bigTagDb
+
+sql create table $bigTagStb (ts timestamp, f1 int) tags(t1 nchar(100), t2 nchar(800), t3 nchar(800), t4 nchar(800));
+
+sql insert into $bigTagTb using $bigTagStb tags("abc", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") values(now, 1);
+
+$tb = $bigTagTb . 2
+sql insert into $tb using $bigTagStb tags("abc", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") values(now, 1);
+
+$tb = $bigTagTb . 3
+sql insert into $tb using $bigTagStb tags("abc", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") values(now, 1);
 
 
 
-system sh/exec.sh -n dnode1 -s stop -x SIGINT
+system sh/exec.sh -n dnode1 -s stop -x SIGINT

tests/system-test/0-others/udfTest.py

@@ -4,6 +4,8 @@ import sys
 import time
 import os
 import platform
+import random
+import string
 
 from util.log import *
 from util.sql import *
@@ -12,7 +14,7 @@ from util.dnodes import *
 import subprocess
 
 class TDTestCase:
-
+    updatecfgDict = {'udfdResFuncs': "udf1,udf2"}
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
         tdLog.debug(f"start to excute {__file__}")
@@ -652,10 +654,20 @@ class TDTestCase:
             tdDnodes.start(1)
             time.sleep(2)
 
+    def test_udfd_cmd(self):
+        tdLog.info(" test udfd -V ")
+        os.system("udfd -V")
+        tdLog.info(" test udfd -c ")
+        os.system("udfd -c")
+        
+        letters = string.ascii_letters + string.digits + '\\'
+        path = ''.join(random.choice(letters) for i in range(5000))
 
-    def run(self):  # sourcery skip: extract-duplicate-method, remove-redundant-fstring
+        os.system(f"udfd -c {path}")
 
+    def run(self):  # sourcery skip: extract-duplicate-method, remove-redundant-fstring     
         print(" env is ok for all ")
+        self.test_udfd_cmd()
         self.prepare_udf_so()
         self.prepare_data()
         self.create_udf_function()

tests/system-test/0-others/udf_restart_taosd.py

@@ -11,7 +11,7 @@ from util.dnodes import *
 import subprocess
 
 class TDTestCase:
-
+    updatecfgDict = {'udfdResFuncs': "udf1,udf2"}
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
         tdLog.debug(f"start to excute {__file__}")