Merge branch '3.0' of github.com:taosdata/TDengine into szhou/tms-duration

docs/en/12-taos-sql/02-database.md

@@ -26,7 +26,6 @@ database_option: {
   | PAGESIZE  value
   | PRECISION {'ms' | 'us' | 'ns'}
   | REPLICA value
-  | RETENTIONS ingestion_duration:keep_duration ...
   | WAL_LEVEL {1 | 2}
   | VGROUPS value
   | SINGLE_STABLE {0 | 1}
@@ -61,7 +60,6 @@ database_option: {
 - PAGESIZE: specifies the size (in KB) of each page in the metadata storage engine cache on each vnode. The default value is 4. Enter a value between 1 and 16384.
 - PRECISION: specifies the precision at which a database records timestamps. Enter ms for milliseconds, us for microseconds, or ns for nanoseconds. The default value is ms.
 - REPLICA: specifies the number of replicas that are made of the database. Enter 1 or 3. The default value is 1. The value of the REPLICA parameter cannot exceed the number of dnodes in the cluster.
-- RETENTIONS: specifies the retention period for data aggregated at various intervals. For example, RETENTIONS 15s:7d,1m:21d,15m:50d indicates that data aggregated every 15 seconds is retained for 7 days, data aggregated every 1 minute is retained for 21 days, and data aggregated every 15 minutes is retained for 50 days. You must enter three aggregation intervals and corresponding retention periods.
 - WAL_LEVEL: specifies whether fsync is enabled. The default value is 1.
   - 1: WAL is enabled but fsync is disabled.
   - 2: WAL and fsync are both enabled.

docs/en/12-taos-sql/03-table.md

@@ -32,9 +32,6 @@ table_options:
 
 table_option: {
     COMMENT 'string_value'
-  | WATERMARK duration[,duration]
-  | MAX_DELAY duration[,duration]
-  | ROLLUP(func_name [, func_name] ...)
   | SMA(col_name [, col_name] ...)
   | TTL value
 }
@@ -54,11 +51,8 @@ table_option: {
 
 **Parameter description**
 1. COMMENT: specifies comments for the table. This parameter can be used with supertables, standard tables, and subtables.
-2. WATERMARK: specifies the time after which the window is closed. The default value is 5 seconds. Enter a value between 0 and 15 minutes in milliseconds, seconds, or minutes. You can enter multiple values separated by commas (,). This parameter applies only to supertables and takes effect only when the RETENTIONS parameter has been specified for the database.
+2. SMA: specifies functions on which to enable small materialized aggregates (SMA). SMA is user-defined precomputation of aggregates based on data blocks. Enter one of the following values: max, min, or sum This parameter can be used with supertables and standard tables.
-3. MAX_DELAY: specifies the maximum latency for pushing computation results. The default value is 15 minutes or the value of the INTERVAL parameter, whichever is smaller. Enter a value between 0 and 15 minutes in milliseconds, seconds, or minutes. You can enter multiple values separated by commas (,). Note: Retain the default value if possible. Configuring a small MAX_DELAY may cause results to be frequently pushed, affecting storage and query performance. This parameter applies only to supertables and takes effect only when the RETENTIONS parameter has been specified for the database.
+3. TTL: specifies the time to live (TTL) for the table. If TTL is specified when creatinga table, after the time period for which the table has been existing is over TTL, TDengine will automatically delete the table. Please be noted that the system may not delete the table at the exact moment that the TTL expires but guarantee there is such a system and finally the table will be deleted. The unit of TTL is in days. The default value is 0, i.e. never expire.
-4. ROLLUP: specifies aggregate functions to roll up. Rolling up a function provides downsampled results based on multiple axes. This parameter applies only to supertables and takes effect only when the RETENTIONS parameter has been specified for the database. You can specify only one function to roll up. The rollup takes effect on all columns except TS. Enter one of the following values: avg, sum, min, max, last, or first.
-5. SMA: specifies functions on which to enable small materialized aggregates (SMA). SMA is user-defined precomputation of aggregates based on data blocks. Enter one of the following values: max, min, or sum This parameter can be used with supertables and standard tables.
-6. TTL: specifies the time to live (TTL) for the table. If TTL is specified when creatinga table, after the time period for which the table has been existing is over TTL, TDengine will automatically delete the table. Please be noted that the system may not delete the table at the exact moment that the TTL expires but guarantee there is such a system and finally the table will be deleted. The unit of TTL is in days. The default value is 0, i.e. never expire.
 
 ## Create Subtables
 

docs/zh/12-taos-sql/02-database.md

@@ -26,7 +26,6 @@ database_option: {
   | PAGESIZE  value
   | PRECISION {'ms' | 'us' | 'ns'}
   | REPLICA value
-  | RETENTIONS ingestion_duration:keep_duration ...
   | WAL_LEVEL {1 | 2}
   | VGROUPS value
   | SINGLE_STABLE {0 | 1}
@@ -61,7 +60,6 @@ database_option: {
 - PAGESIZE：一个 VNODE 中元数据存储引擎的页大小，单位为 KB，默认为 4 KB。范围为 1 到 16384，即 1 KB 到 16 MB。
 - PRECISION：数据库的时间戳精度。ms 表示毫秒，us 表示微秒，ns 表示纳秒，默认 ms 毫秒。
 - REPLICA：表示数据库副本数，取值为 1 或 3，默认为 1。在集群中使用，副本数必须小于或等于 DNODE 的数目。
-- RETENTIONS：表示数据的聚合周期和保存时长，如 RETENTIONS 15s:7d,1m:21d,15m:50d 表示数据原始采集周期为 15 秒，原始数据保存 7 天；按 1 分钟聚合的数据保存 21 天；按 15 分钟聚合的数据保存 50 天。目前支持且只支持三级存储周期。
 - WAL_LEVEL：WAL 级别，默认为 1。
   - 1：写 WAL，但不执行 fsync。
   - 2：写 WAL，而且执行 fsync。

docs/zh/12-taos-sql/03-table.md

@@ -30,9 +30,6 @@ table_options:
 
 table_option: {
     COMMENT 'string_value'
-  | WATERMARK duration[,duration]
-  | MAX_DELAY duration[,duration]
-  | ROLLUP(func_name [, func_name] ...)
   | SMA(col_name [, col_name] ...)
   | TTL value
 }
@@ -52,11 +49,8 @@ table_option: {
 **参数说明**
 
 1. COMMENT：表注释。可用于超级表、子表和普通表。
-2. WATERMARK：指定窗口的关闭时间，默认值为 5 秒，最小单位毫秒，范围为 0 到 15 分钟，多个以逗号分隔。只可用于超级表，且只有当数据库使用了 RETENTIONS 参数时，才可以使用此表参数。
+2. SMA：Small Materialized Aggregates，提供基于数据块的自定义预计算功能。预计算类型包括 MAX、MIN 和 SUM。可用于超级表/普通表。
-3. MAX_DELAY：用于控制推送计算结果的最大延迟，默认值为 interval 的值(但不能超过最大值)，最小单位毫秒，范围为 1 毫秒到 15 分钟，多个以逗号分隔。注：不建议 MAX_DELAY 设置太小，否则会过于频繁的推送结果，影响存储和查询性能，如无特殊需求，取默认值即可。只可用于超级表，且只有当数据库使用了 RETENTIONS 参数时，才可以使用此表参数。
+3. TTL：Time to Live，是用户用来指定表的生命周期的参数。如果创建表时指定了这个参数，当该表的存在时间超过 TTL 指定的时间后，TDengine 自动删除该表。这个 TTL 的时间只是一个大概时间，系统不保证到了时间一定会将其删除，而只保证存在这样一个机制且最终一定会删除。TTL 单位是天，默认为 0，表示不限制，到期时间为表创建时间加上 TTL 时间。TTL 与数据库 KEEP 参数没有关联，如果 KEEP 比 TTL 小，在表被删除之前数据也可能已经被删除。
-4. ROLLUP：Rollup 指定的聚合函数，提供基于多层级的降采样聚合结果。只可用于超级表。只有当数据库使用了 RETENTIONS 参数时，才可以使用此表参数。作用于超级表除 TS 列外的其它所有列，但是只能定义一个聚合函数。 聚合函数支持 avg, sum, min, max, last, first。
-5. SMA：Small Materialized Aggregates，提供基于数据块的自定义预计算功能。预计算类型包括 MAX、MIN 和 SUM。可用于超级表/普通表。
-6. TTL：Time to Live，是用户用来指定表的生命周期的参数。如果创建表时指定了这个参数，当该表的存在时间超过 TTL 指定的时间后，TDengine 自动删除该表。这个 TTL 的时间只是一个大概时间，系统不保证到了时间一定会将其删除，而只保证存在这样一个机制且最终一定会删除。TTL 单位是天，默认为 0，表示不限制，到期时间为表创建时间加上 TTL 时间。TTL 与数据库 KEEP 参数没有关联，如果 KEEP 比 TTL 小，在表被删除之前数据也可能已经被删除。
 
 ## 创建子表
 

include/common/tcommon.h

@@ -258,8 +258,6 @@ typedef struct SQueryTableDataCond {
 int32_t tEncodeDataBlock(void** buf, const SSDataBlock* pBlock);
 void*   tDecodeDataBlock(const void* buf, SSDataBlock* pBlock);
 
-int32_t tEncodeDataBlocks(void** buf, const SArray* blocks);
-void*   tDecodeDataBlocks(const void* buf, SArray** blocks);
 void    colDataDestroy(SColumnInfoData* pColData);
 
 //======================================================================================================================
@@ -294,7 +292,7 @@ typedef struct STableBlockDistInfo {
   int32_t  defMaxRows;
   int32_t  firstSeekTimeUs;
   uint32_t numOfInmemRows;
-  uint32_t numOfSmallBlocks;
+  uint32_t numOfSttRows;
   uint32_t numOfVgroups;
   int32_t  blockRowsHisto[20];
 } STableBlockDistInfo;

include/common/tmsg.h

@@ -35,12 +35,14 @@ extern "C" {
 #define TD_MSG_NUMBER_
 #undef TD_MSG_DICT_
 #undef TD_MSG_INFO_
+#undef TD_MSG_RANGE_CODE_
 #undef TD_MSG_SEG_CODE_
 #include "tmsgdef.h"
 
 #undef TD_MSG_NUMBER_
 #undef TD_MSG_DICT_
 #undef TD_MSG_INFO_
+#undef TD_MSG_RANGE_CODE_
 #define TD_MSG_SEG_CODE_
 #include "tmsgdef.h"
 
@@ -48,33 +50,31 @@ extern "C" {
 #undef TD_MSG_DICT_
 #undef TD_MSG_INFO_
 #undef TD_MSG_SEG_CODE_
+#undef TD_MSG_RANGE_CODE_
 #include "tmsgdef.h"
 
 extern char*   tMsgInfo[];
 extern int32_t tMsgDict[];
-
+extern int32_t tMsgRangeDict[];
-#define TMSG_SEG_CODE(TYPE) (((TYPE)&0xff00) >> 8)
-#define TMSG_SEG_SEQ(TYPE)  ((TYPE)&0xff)
-#define TMSG_INFO(TYPE)                                                                                                \
-  ((TYPE) < TDMT_DND_MAX_MSG || (TYPE) < TDMT_MND_MAX_MSG || (TYPE) < TDMT_VND_MAX_MSG || (TYPE) < TDMT_SCH_MAX_MSG || \
-   (TYPE) < TDMT_STREAM_MAX_MSG || (TYPE) < TDMT_MON_MAX_MSG || (TYPE) < TDMT_SYNC_MAX_MSG) ||                         \
-          (TYPE) < TDMT_VND_STREAM_MSG || (TYPE) < TDMT_VND_TMQ_MSG || (TYPE) < TDMT_VND_TMQ_MAX_MSG                   \
-      ? tMsgInfo[tMsgDict[TMSG_SEG_CODE(TYPE)] + TMSG_SEG_SEQ(TYPE)]                                                   \
-      : 0
-
-#define TMSG_INDEX(TYPE) (tMsgDict[TMSG_SEG_CODE(TYPE)] + TMSG_SEG_SEQ(TYPE))
 
 typedef uint16_t tmsg_t;
 
+#define TMSG_SEG_CODE(TYPE) (((TYPE)&0xff00) >> 8)
+#define TMSG_SEG_SEQ(TYPE)  ((TYPE)&0xff)
+#define TMSG_INDEX(TYPE)    (tMsgDict[TMSG_SEG_CODE(TYPE)] + TMSG_SEG_SEQ(TYPE))
+
 static inline bool tmsgIsValid(tmsg_t type) {
-  if (type < TDMT_DND_MAX_MSG || type < TDMT_MND_MAX_MSG || type < TDMT_VND_MAX_MSG || type < TDMT_SCH_MAX_MSG ||
+  // static int8_t sz = sizeof(tMsgRangeDict) / sizeof(tMsgRangeDict[0]);
-      type < TDMT_STREAM_MAX_MSG || type < TDMT_MON_MAX_MSG || type < TDMT_SYNC_MAX_MSG || type < TDMT_VND_STREAM_MSG ||
+  int8_t maxSegIdx = TMSG_SEG_CODE(TDMT_MAX_MSG);
-      type < TDMT_VND_TMQ_MSG || type < TDMT_VND_TMQ_MAX_MSG) {
+  int    segIdx = TMSG_SEG_CODE(type);
-    return true;
+  if (segIdx >= 0 && segIdx < maxSegIdx) {
-  } else {
+    return type < tMsgRangeDict[segIdx];
-    return false;
   }
+  return false;
 }
+
+#define TMSG_INFO(type) (tmsgIsValid(type) ? tMsgInfo[TMSG_INDEX(type)] : "unKnown")
+
 static inline bool vnodeIsMsgBlock(tmsg_t type) {
   return (type == TDMT_VND_CREATE_TABLE) || (type == TDMT_VND_ALTER_TABLE) || (type == TDMT_VND_DROP_TABLE) ||
          (type == TDMT_VND_UPDATE_TAG_VAL) || (type == TDMT_VND_ALTER_CONFIRM) || (type == TDMT_VND_COMMIT) ||
@@ -169,14 +169,14 @@ typedef enum _mgmt_table {
 #define TSDB_FILL_PREV        6
 #define TSDB_FILL_NEXT        7
 
-#define TSDB_ALTER_USER_PASSWD                 0x1
+#define TSDB_ALTER_USER_PASSWD          0x1
-#define TSDB_ALTER_USER_SUPERUSER              0x2
+#define TSDB_ALTER_USER_SUPERUSER       0x2
-#define TSDB_ALTER_USER_ENABLE                 0x3
+#define TSDB_ALTER_USER_ENABLE          0x3
-#define TSDB_ALTER_USER_SYSINFO                0x4
+#define TSDB_ALTER_USER_SYSINFO         0x4
-#define TSDB_ALTER_USER_ADD_PRIVILEGES         0x5
+#define TSDB_ALTER_USER_ADD_PRIVILEGES  0x5
-#define TSDB_ALTER_USER_DEL_PRIVILEGES         0x6
+#define TSDB_ALTER_USER_DEL_PRIVILEGES  0x6
-#define TSDB_ALTER_USER_ADD_WHITE_LIST         0x7
+#define TSDB_ALTER_USER_ADD_WHITE_LIST  0x7
-#define TSDB_ALTER_USER_DROP_WHITE_LIST        0x8
+#define TSDB_ALTER_USER_DROP_WHITE_LIST 0x8
 
 #define TSDB_KILL_MSG_LEN 30
 
@@ -346,7 +346,7 @@ typedef enum ENodeType {
   QUERY_NODE_RESTORE_MNODE_STMT,
   QUERY_NODE_RESTORE_VNODE_STMT,
   QUERY_NODE_PAUSE_STREAM_STMT,
-  QUERY_NODE_RESUME_STREAM_STMT,  
+  QUERY_NODE_RESUME_STREAM_STMT,
   QUERY_NODE_CREATE_VIEW_STMT,
   QUERY_NODE_DROP_VIEW_STMT,
 
@@ -790,7 +790,7 @@ typedef struct {
 
 int32_t tSerializeSMDropStbReq(void* buf, int32_t bufLen, SMDropStbReq* pReq);
 int32_t tDeserializeSMDropStbReq(void* buf, int32_t bufLen, SMDropStbReq* pReq);
-void    tFreeSMDropStbReq(SMDropStbReq *pReq);
+void    tFreeSMDropStbReq(SMDropStbReq* pReq);
 
 typedef struct {
   char    name[TSDB_TABLE_FNAME_LEN];
@@ -871,18 +871,18 @@ int32_t tSerializeSCreateAcctReq(void* buf, int32_t bufLen, SCreateAcctReq* pReq
 int32_t tDeserializeSCreateAcctReq(void* buf, int32_t bufLen, SCreateAcctReq* pReq);
 
 typedef struct {
-  char user[TSDB_USER_LEN];
+  char    user[TSDB_USER_LEN];
   int32_t sqlLen;
   char*   sql;
 } SDropUserReq, SDropAcctReq;
 
 int32_t tSerializeSDropUserReq(void* buf, int32_t bufLen, SDropUserReq* pReq);
 int32_t tDeserializeSDropUserReq(void* buf, int32_t bufLen, SDropUserReq* pReq);
-void    tFreeSDropUserReq(SDropUserReq *pReq);
+void    tFreeSDropUserReq(SDropUserReq* pReq);
 
-typedef struct SIpV4Range{
+typedef struct SIpV4Range {
-    uint32_t ip;
+  uint32_t ip;
-    uint32_t mask;
+  uint32_t mask;
 } SIpV4Range;
 
 typedef struct {
@@ -892,21 +892,21 @@ typedef struct {
 
 SIpWhiteList* cloneIpWhiteList(SIpWhiteList* pIpWhiteList);
 typedef struct {
-  int8_t createType;
+  int8_t      createType;
-  int8_t superUser;  // denote if it is a super user or not
+  int8_t      superUser;  // denote if it is a super user or not
-  int8_t sysInfo;
+  int8_t      sysInfo;
-  int8_t enable;
+  int8_t      enable;
-  char   user[TSDB_USER_LEN];
+  char        user[TSDB_USER_LEN];
-  char   pass[TSDB_USET_PASSWORD_LEN];
+  char        pass[TSDB_USET_PASSWORD_LEN];
   int32_t     numIpRanges;
   SIpV4Range* pIpRanges;
-  int32_t sqlLen;
+  int32_t     sqlLen;
-  char*   sql;
+  char*       sql;
 } SCreateUserReq;
 
 int32_t tSerializeSCreateUserReq(void* buf, int32_t bufLen, SCreateUserReq* pReq);
 int32_t tDeserializeSCreateUserReq(void* buf, int32_t bufLen, SCreateUserReq* pReq);
-void    tFreeSCreateUserReq(SCreateUserReq *pReq);
+void    tFreeSCreateUserReq(SCreateUserReq* pReq);
 
 typedef struct {
   int64_t     ver;
@@ -933,22 +933,22 @@ int32_t tSerializeRetrieveIpWhite(void* buf, int32_t bufLen, SRetrieveIpWhiteReq
 int32_t tDeserializeRetrieveIpWhite(void* buf, int32_t bufLen, SRetrieveIpWhiteReq* pReq);
 
 typedef struct {
-  int8_t  alterType;
+  int8_t      alterType;
-  int8_t  superUser;
+  int8_t      superUser;
-  int8_t  sysInfo;
+  int8_t      sysInfo;
-  int8_t  enable;
+  int8_t      enable;
-  int8_t  isView;
+  int8_t      isView;
-  char    user[TSDB_USER_LEN];
+  char        user[TSDB_USER_LEN];
-  char    pass[TSDB_USET_PASSWORD_LEN];
+  char        pass[TSDB_USET_PASSWORD_LEN];
-  char    objname[TSDB_DB_FNAME_LEN];  // db or topic
+  char        objname[TSDB_DB_FNAME_LEN];  // db or topic
-  char    tabName[TSDB_TABLE_NAME_LEN];
+  char        tabName[TSDB_TABLE_NAME_LEN];
-  char*   tagCond;
+  char*       tagCond;
-  int32_t tagCondLen;
+  int32_t     tagCondLen;
   int32_t     numIpRanges;
   SIpV4Range* pIpRanges;
   int64_t     privileges;
-  int32_t sqlLen;
+  int32_t     sqlLen;
-  char*   sql;
+  char*       sql;
 } SAlterUserReq;
 
 int32_t tSerializeSAlterUserReq(void* buf, int32_t bufLen, SAlterUserReq* pReq);
@@ -978,9 +978,9 @@ typedef struct {
   SHashObj* alterTbs;
   SHashObj* readViews;
   SHashObj* writeViews;
-  SHashObj* alterViews;  
+  SHashObj* alterViews;
   SHashObj* useDbs;
-  int64_t whiteListVer;
+  int64_t   whiteListVer;
 } SGetUserAuthRsp;
 
 int32_t tSerializeSGetUserAuthRsp(void* buf, int32_t bufLen, SGetUserAuthRsp* pRsp);
@@ -995,8 +995,8 @@ int32_t tSerializeSGetUserWhiteListReq(void* buf, int32_t bufLen, SGetUserWhiteL
 int32_t tDeserializeSGetUserWhiteListReq(void* buf, int32_t bufLen, SGetUserWhiteListReq* pReq);
 
 typedef struct {
-  char user[TSDB_USER_LEN];
+  char        user[TSDB_USER_LEN];
-  int32_t numWhiteLists;
+  int32_t     numWhiteLists;
   SIpV4Range* pWhiteLists;
 } SGetUserWhiteListRsp;
 
@@ -1169,8 +1169,8 @@ int32_t tDeserializeSAlterDbReq(void* buf, int32_t bufLen, SAlterDbReq* pReq);
 void    tFreeSAlterDbReq(SAlterDbReq* pReq);
 
 typedef struct {
-  char   db[TSDB_DB_FNAME_LEN];
+  char    db[TSDB_DB_FNAME_LEN];
-  int8_t ignoreNotExists;
+  int8_t  ignoreNotExists;
   int32_t sqlLen;
   char*   sql;
 } SDropDbReq;
@@ -1378,7 +1378,7 @@ typedef struct {
 
 int32_t tSerializeSCompactDbReq(void* buf, int32_t bufLen, SCompactDbReq* pReq);
 int32_t tDeserializeSCompactDbReq(void* buf, int32_t bufLen, SCompactDbReq* pReq);
-void    tFreeSCompactDbReq(SCompactDbReq *pReq);
+void    tFreeSCompactDbReq(SCompactDbReq* pReq);
 
 typedef struct {
   char    name[TSDB_FUNC_NAME_LEN];
@@ -1817,7 +1817,6 @@ int32_t tSerializeSViewHbRsp(void* buf, int32_t bufLen, SViewHbRsp* pRsp);
 int32_t tDeserializeSViewHbRsp(void* buf, int32_t bufLen, SViewHbRsp* pRsp);
 void    tFreeSViewHbRsp(SViewHbRsp* pRsp);
 
-
 typedef struct {
   int32_t numOfTables;
   int32_t numOfVgroup;
@@ -2006,7 +2005,7 @@ typedef struct {
 
 int32_t tSerializeSRestoreDnodeReq(void* buf, int32_t bufLen, SRestoreDnodeReq* pReq);
 int32_t tDeserializeSRestoreDnodeReq(void* buf, int32_t bufLen, SRestoreDnodeReq* pReq);
-void    tFreeSRestoreDnodeReq(SRestoreDnodeReq *pReq);
+void    tFreeSRestoreDnodeReq(SRestoreDnodeReq* pReq);
 
 typedef struct {
   int32_t dnodeId;
@@ -2018,7 +2017,7 @@ typedef struct {
 
 int32_t tSerializeSMCfgDnodeReq(void* buf, int32_t bufLen, SMCfgDnodeReq* pReq);
 int32_t tDeserializeSMCfgDnodeReq(void* buf, int32_t bufLen, SMCfgDnodeReq* pReq);
-void tFreeSMCfgDnodeReq(SMCfgDnodeReq *pReq);
+void    tFreeSMCfgDnodeReq(SMCfgDnodeReq* pReq);
 
 typedef struct {
   char config[TSDB_DNODE_CONFIG_LEN];
@@ -2037,7 +2036,7 @@ typedef struct {
 
 int32_t tSerializeSCreateDropMQSNodeReq(void* buf, int32_t bufLen, SMCreateQnodeReq* pReq);
 int32_t tDeserializeSCreateDropMQSNodeReq(void* buf, int32_t bufLen, SMCreateQnodeReq* pReq);
-void    tFreeSMCreateQnodeReq(SMCreateQnodeReq *pReq);
+void    tFreeSMCreateQnodeReq(SMCreateQnodeReq* pReq);
 void    tFreeSDDropQnodeReq(SDDropQnodeReq* pReq);
 typedef struct {
   int8_t   replica;
@@ -2079,7 +2078,7 @@ typedef struct {
 
 int32_t tSerializeSBalanceVgroupReq(void* buf, int32_t bufLen, SBalanceVgroupReq* pReq);
 int32_t tDeserializeSBalanceVgroupReq(void* buf, int32_t bufLen, SBalanceVgroupReq* pReq);
-void    tFreeSBalanceVgroupReq(SBalanceVgroupReq *pReq);
+void    tFreeSBalanceVgroupReq(SBalanceVgroupReq* pReq);
 
 typedef struct {
   int32_t vgId1;
@@ -2100,7 +2099,7 @@ typedef struct {
 
 int32_t tSerializeSRedistributeVgroupReq(void* buf, int32_t bufLen, SRedistributeVgroupReq* pReq);
 int32_t tDeserializeSRedistributeVgroupReq(void* buf, int32_t bufLen, SRedistributeVgroupReq* pReq);
-void    tFreeSRedistributeVgroupReq(SRedistributeVgroupReq *pReq);
+void    tFreeSRedistributeVgroupReq(SRedistributeVgroupReq* pReq);
 
 typedef struct {
   int32_t useless;
@@ -2111,7 +2110,7 @@ typedef struct {
 
 int32_t tSerializeSBalanceVgroupLeaderReq(void* buf, int32_t bufLen, SBalanceVgroupLeaderReq* pReq);
 int32_t tDeserializeSBalanceVgroupLeaderReq(void* buf, int32_t bufLen, SBalanceVgroupLeaderReq* pReq);
-void    tFreeSBalanceVgroupLeaderReq(SBalanceVgroupLeaderReq *pReq);
+void    tFreeSBalanceVgroupLeaderReq(SBalanceVgroupLeaderReq* pReq);
 
 typedef struct {
   int32_t vgId;
@@ -2503,15 +2502,15 @@ typedef struct {
 } SMVSubscribeRsp;
 
 typedef struct {
-  char   name[TSDB_TOPIC_FNAME_LEN];
+  char    name[TSDB_TOPIC_FNAME_LEN];
-  int8_t igNotExists;
+  int8_t  igNotExists;
   int32_t sqlLen;
   char*   sql;
 } SMDropTopicReq;
 
 int32_t tSerializeSMDropTopicReq(void* buf, int32_t bufLen, SMDropTopicReq* pReq);
 int32_t tDeserializeSMDropTopicReq(void* buf, int32_t bufLen, SMDropTopicReq* pReq);
-void    tFreeSMDropTopicReq(SMDropTopicReq *pReq);
+void    tFreeSMDropTopicReq(SMDropTopicReq* pReq);
 
 typedef struct {
   char   topic[TSDB_TOPIC_FNAME_LEN];
@@ -3082,8 +3081,8 @@ typedef struct {
 } SMqVDeleteRsp;
 
 typedef struct {
-  char   name[TSDB_STREAM_FNAME_LEN];
+  char    name[TSDB_STREAM_FNAME_LEN];
-  int8_t igNotExists;
+  int8_t  igNotExists;
   int32_t sqlLen;
   char*   sql;
 } SMDropStreamReq;
@@ -3920,7 +3919,7 @@ int32_t tDeserializeSCMDropViewReq(void* buf, int32_t bufLen, SCMDropViewReq* pR
 void    tFreeSCMDropViewReq(SCMDropViewReq* pReq);
 
 typedef struct {
-  char   fullname[TSDB_VIEW_FNAME_LEN];
+  char fullname[TSDB_VIEW_FNAME_LEN];
 } SViewMetaReq;
 int32_t tSerializeSViewMetaReq(void* buf, int32_t bufLen, const SViewMetaReq* pReq);
 int32_t tDeserializeSViewMetaReq(void* buf, int32_t bufLen, SViewMetaReq* pReq);
@@ -3942,7 +3941,6 @@ int32_t tSerializeSViewMetaRsp(void* buf, int32_t bufLen, const SViewMetaRsp* pR
 int32_t tDeserializeSViewMetaRsp(void* buf, int32_t bufLen, SViewMetaRsp* pRsp);
 void    tFreeSViewMetaRsp(SViewMetaRsp* pRsp);
 
-
 #pragma pack(pop)
 
 #ifdef __cplusplus

include/common/tmsgdef.h

@@ -24,48 +24,70 @@
 
 #if defined(TD_MSG_INFO_)
 
-#undef TD_NEW_MSG_SEG
+  #undef TD_NEW_MSG_SEG
-#undef TD_DEF_MSG_TYPE
+  #undef TD_DEF_MSG_TYPE
-#define TD_NEW_MSG_SEG(TYPE) "null",
+  #undef TD_CLOSE_MSG_TYPE
-#define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP) MSG, MSG "-rsp",
+  #define TD_NEW_MSG_SEG(TYPE) "null",
+  #define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP) MSG, MSG "-rsp",
+  #define TD_CLOSE_MSG_TYPE(TYPE)
 
-char *tMsgInfo[] = {
+  char *tMsgInfo[] = {
+
+#elif defined(TD_MSG_RANGE_CODE_)
+
+  #undef TD_NEW_MSG_SEG
+  #undef TD_DEF_MSG_TYPE
+  #undef TD_CLOSE_MSG_TYPE
+  #define TD_NEW_MSG_SEG(TYPE) 
+  #define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP)
+  #define TD_CLOSE_MSG_TYPE(TYPE) TYPE,
+  int32_t tMsgRangeDict[] = {
 
 #elif defined(TD_MSG_NUMBER_)
 
-#undef TD_NEW_MSG_SEG
+  #undef TD_NEW_MSG_SEG
-#undef TD_DEF_MSG_TYPE
+  #undef TD_DEF_MSG_TYPE
-#define TD_NEW_MSG_SEG(TYPE) TYPE##_NUM,
+  #undef TD_CLOSE_MSG_TYPE
-#define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP) TYPE##_NUM, TYPE##_RSP_NUM,
+  #define TD_NEW_MSG_SEG(TYPE) TYPE##_NUM,
+  #define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP) TYPE##_NUM, TYPE##_RSP_NUM,
+  #define TD_CLOSE_MSG_TYPE(TYPE)
 
-enum {
+  enum {
 
 #elif defined(TD_MSG_DICT_)
 
-#undef TD_NEW_MSG_SEG
+  #undef TD_NEW_MSG_SEG
-#undef TD_DEF_MSG_TYPE
+  #undef TD_DEF_MSG_TYPE
-#define TD_NEW_MSG_SEG(TYPE) TYPE##_NUM,
+  #undef TD_CLOSE_MSG_TYPE
-#define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP)
+  #define TD_NEW_MSG_SEG(TYPE) TYPE##_NUM,
+  #define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP)
+  #define TD_CLOSE_MSG_TYPE(type)
+
+  int32_t tMsgDict[] = {
 
-int32_t tMsgDict[] = {
 
 #elif defined(TD_MSG_SEG_CODE_)
 
-#undef TD_NEW_MSG_SEG
+  #undef TD_NEW_MSG_SEG
-#undef TD_DEF_MSG_TYPE
+  #undef TD_DEF_MSG_TYPE
-#define TD_NEW_MSG_SEG(TYPE) TYPE##_SEG_CODE,
+  #undef TD_CLOSE_MSG_TYPE
-#define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP)
+  #define TD_NEW_MSG_SEG(TYPE) TYPE##_SEG_CODE,
+  #define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP)
+  #define TD_CLOSE_MSG_TYPE(TYPE)
 
-enum {
+  enum {
 
-#else
 
-#undef TD_NEW_MSG_SEG
+#else 
-#undef TD_DEF_MSG_TYPE
-#define TD_NEW_MSG_SEG(TYPE) TYPE = ((TYPE##_SEG_CODE) << 8),
-#define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP) TYPE, TYPE##_RSP,
 
-enum { // WARN: new msg should be appended to segment tail
+  #undef TD_NEW_MSG_SEG
+  #undef TD_DEF_MSG_TYPE
+  #undef TD_CLOSE_MSG_TYPE
+  #define TD_NEW_MSG_SEG(TYPE) TYPE = ((TYPE##_SEG_CODE) << 8),
+  #define TD_DEF_MSG_TYPE(TYPE, MSG, REQ, RSP) TYPE, TYPE##_RSP,
+  #define TD_CLOSE_MSG_TYPE(TYPE) TYPE,
+
+  enum { // WARN: new msg should be appended to segment tail
 #endif
   TD_NEW_MSG_SEG(TDMT_DND_MSG)    // 0<<8
   TD_DEF_MSG_TYPE(TDMT_DND_CREATE_MNODE, "dnode-create-mnode", NULL, NULL)
@@ -82,10 +104,12 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_DND_NET_TEST, "net-test", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_DND_CONFIG_DNODE, "config-dnode", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_DND_SYSTABLE_RETRIEVE, "dnode-retrieve", NULL, NULL)
-  TD_DEF_MSG_TYPE(TDMT_DND_MAX_MSG, "dnd-max", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_DND_UNUSED_CODE, "dnd-unused", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_DND_ALTER_MNODE_TYPE, "dnode-alter-mnode-type", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_DND_ALTER_VNODE_TYPE, "dnode-alter-vnode-type", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_DND_CHECK_VNODE_LEARNER_CATCHUP, "dnode-check-vnode-learner-catchup", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_DND_MAX_MSG, "dnd-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_DND_MSG) 
 
   TD_NEW_MSG_SEG(TDMT_MND_MSG)  // 1<<8
   TD_DEF_MSG_TYPE(TDMT_MND_CONNECT, "connect", NULL, NULL)
@@ -194,6 +218,7 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_MND_DROP_VIEW, "drop-view", SCMDropViewReq, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_VIEW_META, "view-meta", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_MAX_MSG, "mnd-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_MND_MSG) 
 
   TD_NEW_MSG_SEG(TDMT_VND_MSG)  // 2<<8
   TD_DEF_MSG_TYPE(TDMT_VND_SUBMIT, "submit", SSubmitReq, SSubmitRsp)
@@ -231,7 +256,7 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_VND_EXEC_RSMA, "vnode-exec-rsma", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_DELETE, "delete-data", SVDeleteReq, SVDeleteRsp)
   TD_DEF_MSG_TYPE(TDMT_VND_BATCH_DEL, "batch-delete", SBatchDeleteReq, NULL)
-  TD_DEF_MSG_TYPE(TDMT_VND_ALTER_CONFIG, "alter-config", NULL, NULL)
+TD_DEF_MSG_TYPE(TDMT_VND_ALTER_CONFIG, "alter-config", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_ALTER_REPLICA, "alter-replica", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_ALTER_CONFIRM, "alter-confirm", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_ALTER_HASHRANGE, "alter-hashrange", NULL, NULL)
@@ -243,6 +268,7 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_VND_DROP_INDEX, "vnode-drop-index", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_DISABLE_WRITE, "vnode-disable-write", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_MAX_MSG, "vnd-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_VND_MSG) 
 
   TD_NEW_MSG_SEG(TDMT_SCH_MSG)  // 3<<8
   TD_DEF_MSG_TYPE(TDMT_SCH_QUERY, "query", NULL, NULL)
@@ -257,6 +283,7 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_SCH_LINK_BROKEN, "link-broken", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_SCH_TASK_NOTIFY, "task-notify", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_SCH_MAX_MSG, "sch-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_SCH_MSG) 
 
 
   TD_NEW_MSG_SEG(TDMT_STREAM_MSG)   //4 << 8
@@ -274,9 +301,11 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_STOP, "stream-task-stop", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_HTASK_DROP, "stream-htask-drop", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_MAX_MSG, "stream-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_STREAM_MSG) 
 
   TD_NEW_MSG_SEG(TDMT_MON_MSG)  //5 << 8
   TD_DEF_MSG_TYPE(TDMT_MON_MAX_MSG, "monitor-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_MON_MSG) 
 
   TD_NEW_MSG_SEG(TDMT_SYNC_MSG) //6 << 8
   TD_DEF_MSG_TYPE(TDMT_SYNC_TIMEOUT, "sync-timer", NULL, NULL)
@@ -308,6 +337,8 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_SYNC_PREP_SNAPSHOT_REPLY, "sync-prep-snapshot-reply", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_SYNC_MAX_MSG, "sync-max", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_SYNC_FORCE_FOLLOWER, "sync-force-become-follower", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_SYNC_MSG) 
+
 
   TD_NEW_MSG_SEG(TDMT_VND_STREAM_MSG) //7 << 8
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_SCAN_HISTORY, "vnode-stream-scan-history", NULL, NULL)
@@ -317,6 +348,7 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_TASK_RESET, "vnode-stream-reset", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_TASK_CHECK, "vnode-stream-task-check", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_MAX_MSG, "vnd-stream-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_VND_STREAM_MSG) 
 
   TD_NEW_MSG_SEG(TDMT_VND_TMQ_MSG)  //8 << 8
   TD_DEF_MSG_TYPE(TDMT_VND_TMQ_SUBSCRIBE, "vnode-tmq-subscribe", SMqRebVgReq, SMqRebVgRsp)
@@ -330,9 +362,15 @@ enum { // WARN: new msg should be appended to segment tail
   TD_DEF_MSG_TYPE(TDMT_VND_TMQ_VG_WALINFO, "vnode-tmq-vg-walinfo", SMqPollReq, SMqDataBlkRsp)
   TD_DEF_MSG_TYPE(TDMT_VND_TMQ_VG_COMMITTEDINFO, "vnode-tmq-committedinfo", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_TMQ_MAX_MSG, "vnd-tmq-max", NULL, NULL)
+  TD_CLOSE_MSG_TYPE(TDMT_END_TMQ_MSG) 
+
+  TD_NEW_MSG_SEG(TDMT_MAX_MSG)  // msg end mark
+  
+
+  
 
 
-#if defined(TD_MSG_NUMBER_)
+  #if defined(TD_MSG_NUMBER_)
-  TDMT_MAX
+    TDMT_MAX
-#endif
+  #endif
 };

include/libs/executor/storageapi.h

@@ -366,6 +366,8 @@ typedef struct SStateStore {
   TSKEY        (*updateInfoFillBlockData)(SUpdateInfo* pInfo, SSDataBlock* pBlock, int32_t primaryTsCol);
   bool         (*updateInfoIsUpdated)(SUpdateInfo* pInfo, uint64_t tableId, TSKEY ts);
   bool         (*updateInfoIsTableInserted)(SUpdateInfo* pInfo, int64_t tbUid);
+  bool         (*isIncrementalTimeStamp)(SUpdateInfo *pInfo, uint64_t tableId, TSKEY ts);
+
   void (*updateInfoDestroy)(SUpdateInfo* pInfo);
   void (*windowSBfDelete)(SUpdateInfo *pInfo, uint64_t count);
   void (*windowSBfAdd)(SUpdateInfo *pInfo, uint64_t count);

include/libs/stream/tstream.h

@@ -216,9 +216,6 @@ int32_t         streamQueuePush(SStreamQueue1* pQueue, SStreamQueueItem* pItem);
 SStreamQueueRes streamQueueGetRes(SStreamQueue1* pQueue);
 #endif
 
-int32_t streamInit();
-void    streamCleanUp();
-
 SStreamDataSubmit* streamDataSubmitNew(SPackedData* pData, int32_t type);
 void               streamDataSubmitDestroy(SStreamDataSubmit* pDataSubmit);
 
@@ -853,7 +850,8 @@ void         streamMetaNotifyClose(SStreamMeta* pMeta);
 int32_t      streamTaskSetDb(SStreamMeta* pMeta, void* pTask, char* key);
 void         streamMetaStartHb(SStreamMeta* pMeta);
 bool         streamMetaTaskInTimer(SStreamMeta* pMeta);
-int32_t      streamMetaUpdateTaskDownstreamStatus(SStreamTask* pTask, int64_t startTs, int64_t endTs, bool succ);
+int32_t      streamMetaUpdateTaskDownstreamStatus(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, int64_t startTs,
+                                                  int64_t endTs, bool ready);
 void         streamMetaRLock(SStreamMeta* pMeta);
 void         streamMetaRUnLock(SStreamMeta* pMeta);
 void         streamMetaWLock(SStreamMeta* pMeta);

include/libs/stream/tstreamUpdate.h

@@ -55,6 +55,7 @@ int32_t      updateInfoSerialize(void *buf, int32_t bufLen, const SUpdateInfo *p
 int32_t      updateInfoDeserialize(void *buf, int32_t bufLen, SUpdateInfo *pInfo);
 void         windowSBfDelete(SUpdateInfo *pInfo, uint64_t count);
 void         windowSBfAdd(SUpdateInfo *pInfo, uint64_t count);
+bool         isIncrementalTimeStamp(SUpdateInfo *pInfo, uint64_t tableId, TSKEY ts);
 
 #ifdef __cplusplus
 }

include/util/tdef.h

@@ -305,7 +305,7 @@ typedef enum ELogicConditionType {
 #define TSDB_SYNC_APPLYQ_SIZE_LIMIT    512
 #define TSDB_SYNC_NEGOTIATION_WIN      512
 
-#define TSDB_SYNC_SNAP_BUFFER_SIZE 2048
+#define TSDB_SYNC_SNAP_BUFFER_SIZE 1024
 
 #define TSDB_TBNAME_COLUMN_INDEX     (-1)
 #define TSDB_MULTI_TABLEMETA_MAX_NUM 100000  // maximum batch size allowed to load table meta

source/common/src/tglobal.c

@@ -1282,7 +1282,7 @@ int32_t taosCreateLog(const char *logname, int32_t logFileNum, const char *cfgDi
 static int32_t taosCheckGlobalCfg() {
   uint32_t ipv4 = taosGetIpv4FromFqdn(tsLocalFqdn);
   if (ipv4 == 0xffffffff) {
-    terrno = TAOS_SYSTEM_ERROR(errno);
+    terrno = TSDB_CODE_RPC_FQDN_ERROR;
     uError("failed to get ip from fqdn:%s since %s, dnode can not be initialized", tsLocalFqdn, terrstr());
     return -1;
   }

source/common/src/tmsg.c

@@ -18,42 +18,51 @@
 
 #undef TD_MSG_NUMBER_
 #undef TD_MSG_DICT_
+#undef TD_MSG_RANGE_CODE_
 #define TD_MSG_INFO_
 #undef TD_MSG_SEG_CODE_
 #include "tmsgdef.h"
 
 #undef TD_MSG_NUMBER_
 #undef TD_MSG_INFO_
+#undef TD_MSG_RANGE_CODE_
 #define TD_MSG_DICT_
 #undef TD_MSG_SEG_CODE_
 #include "tmsgdef.h"
 
+#undef TD_MSG_NUMBER_
+#undef TD_MSG_INFO_
+#undef TD_MSG_DICT_
+#undef TD_MSG_SEG_CODE_
+#define TD_MSG_RANGE_CODE_
+#include "tmsgdef.h"
+
 #include "tlog.h"
 
-#define DECODESQL()                                             \
+#define DECODESQL()                                                                 \
-  do {                                                          \
+  do {                                                                              \
-    if(!tDecodeIsEnd(&decoder)){                                 \
+    if (!tDecodeIsEnd(&decoder)) {                                                  \
-      if(tDecodeI32(&decoder, &pReq->sqlLen) < 0) return -1;    \
+      if (tDecodeI32(&decoder, &pReq->sqlLen) < 0) return -1;                       \
-      if(pReq->sqlLen > 0){                                     \
+      if (pReq->sqlLen > 0) {                                                       \
-        if (tDecodeBinaryAlloc(&decoder, (void **)&pReq->sql, NULL) < 0) return -1;  \
+        if (tDecodeBinaryAlloc(&decoder, (void **)&pReq->sql, NULL) < 0) return -1; \
-      }                                                         \
+      }                                                                             \
-    }                                                           \
+    }                                                                               \
   } while (0)
 
-#define ENCODESQL()                                              \
+#define ENCODESQL()                                                        \
-  do {                                                           \
+  do {                                                                     \
-    if (pReq->sqlLen > 0 && pReq->sql != NULL){                  \
+    if (pReq->sqlLen > 0 && pReq->sql != NULL) {                           \
-      if (tEncodeI32(&encoder, pReq->sqlLen) < 0) return -1;     \
+      if (tEncodeI32(&encoder, pReq->sqlLen) < 0) return -1;               \
-      if (tEncodeBinary(&encoder, pReq->sql, pReq->sqlLen) < 0) return -1;       \
+      if (tEncodeBinary(&encoder, pReq->sql, pReq->sqlLen) < 0) return -1; \
-    }                                                            \
+    }                                                                      \
   } while (0)
 
-#define FREESQL()                     \
+#define FREESQL()                \
-  do {                                \
+  do {                           \
-    if(pReq->sql != NULL){            \
+    if (pReq->sql != NULL) {     \
-      taosMemoryFree(pReq->sql);      \
+      taosMemoryFree(pReq->sql); \
-    }                                 \
+    }                            \
-    pReq->sql = NULL;                 \
+    pReq->sql = NULL;            \
   } while (0)
 
 static int32_t tDecodeSVAlterTbReqCommon(SDecoder *pDecoder, SVAlterTbReq *pReq);
@@ -742,9 +751,7 @@ int32_t tDeserializeSMDropStbReq(void *buf, int32_t bufLen, SMDropStbReq *pReq)
   return 0;
 }
 
-void tFreeSMDropStbReq(SMDropStbReq *pReq) {
+void tFreeSMDropStbReq(SMDropStbReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSMAlterStbReq(void *buf, int32_t bufLen, SMAlterStbReq *pReq) {
   SEncoder encoder = {0};
@@ -1489,9 +1496,7 @@ int32_t tDeserializeSDropUserReq(void *buf, int32_t bufLen, SDropUserReq *pReq)
   return 0;
 }
 
-void tFreeSDropUserReq(SDropUserReq *pReq) {
+void tFreeSDropUserReq(SDropUserReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 SIpWhiteList *cloneIpWhiteList(SIpWhiteList *pIpWhiteList) {
   if (pIpWhiteList == NULL) return NULL;
@@ -1822,7 +1827,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   char *tb = taosHashIterate(pRsp->readTbs, NULL);
   while (tb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(tb, &keyLen);
+    void * key = taosHashGetKey(tb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1837,7 +1842,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   tb = taosHashIterate(pRsp->writeTbs, NULL);
   while (tb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(tb, &keyLen);
+    void * key = taosHashGetKey(tb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1852,7 +1857,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   tb = taosHashIterate(pRsp->alterTbs, NULL);
   while (tb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(tb, &keyLen);
+    void * key = taosHashGetKey(tb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1867,7 +1872,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   tb = taosHashIterate(pRsp->readViews, NULL);
   while (tb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(tb, &keyLen);
+    void * key = taosHashGetKey(tb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1882,7 +1887,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   tb = taosHashIterate(pRsp->writeViews, NULL);
   while (tb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(tb, &keyLen);
+    void * key = taosHashGetKey(tb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1897,7 +1902,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   tb = taosHashIterate(pRsp->alterViews, NULL);
   while (tb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(tb, &keyLen);
+    void * key = taosHashGetKey(tb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1912,7 +1917,7 @@ int32_t tSerializeSGetUserAuthRspImpl(SEncoder *pEncoder, SGetUserAuthRsp *pRsp)
   int32_t *useDb = taosHashIterate(pRsp->useDbs, NULL);
   while (useDb != NULL) {
     size_t keyLen = 0;
-    void  *key = taosHashGetKey(useDb, &keyLen);
+    void * key = taosHashGetKey(useDb, &keyLen);
     if (tEncodeI32(pEncoder, keyLen) < 0) return -1;
     if (tEncodeCStr(pEncoder, key) < 0) return -1;
 
@@ -1954,8 +1959,8 @@ int32_t tDeserializeSGetUserAuthRspImpl(SDecoder *pDecoder, SGetUserAuthRsp *pRs
   pRsp->alterViews = taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK);
   pRsp->useDbs = taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK);
   if (pRsp->createdDbs == NULL || pRsp->readDbs == NULL || pRsp->writeDbs == NULL || pRsp->readTbs == NULL ||
-      pRsp->writeTbs == NULL || pRsp->alterTbs == NULL || pRsp->readViews == NULL ||
+      pRsp->writeTbs == NULL || pRsp->alterTbs == NULL || pRsp->readViews == NULL || pRsp->writeViews == NULL ||
-      pRsp->writeViews == NULL || pRsp->alterViews == NULL ||pRsp->useDbs == NULL) {
+      pRsp->alterViews == NULL || pRsp->useDbs == NULL) {
     goto _err;
   }
 
@@ -2219,7 +2224,7 @@ int32_t tDeserializeSGetUserWhiteListReq(void *buf, int32_t bufLen, SGetUserWhit
   return 0;
 }
 
-int32_t tSerializeSGetUserWhiteListRsp(void* buf, int32_t bufLen, SGetUserWhiteListRsp* pRsp) {
+int32_t tSerializeSGetUserWhiteListRsp(void *buf, int32_t bufLen, SGetUserWhiteListRsp *pRsp) {
   SEncoder encoder = {0};
   tEncoderInit(&encoder, buf, bufLen);
 
@@ -2237,7 +2242,7 @@ int32_t tSerializeSGetUserWhiteListRsp(void* buf, int32_t bufLen, SGetUserWhiteL
   return tlen;
 }
 
-int32_t tDeserializeSGetUserWhiteListRsp(void* buf, int32_t bufLen, SGetUserWhiteListRsp* pRsp) {
+int32_t tDeserializeSGetUserWhiteListRsp(void *buf, int32_t bufLen, SGetUserWhiteListRsp *pRsp) {
   SDecoder decoder = {0};
   tDecoderInit(&decoder, buf, bufLen);
 
@@ -2257,9 +2262,7 @@ int32_t tDeserializeSGetUserWhiteListRsp(void* buf, int32_t bufLen, SGetUserWhit
   return 0;
 }
 
-void tFreeSGetUserWhiteListRsp(SGetUserWhiteListRsp* pRsp) {
+void tFreeSGetUserWhiteListRsp(SGetUserWhiteListRsp *pRsp) { taosMemoryFree(pRsp->pWhiteLists); }
-  taosMemoryFree(pRsp->pWhiteLists);
-}
 
 int32_t tSerializeSCreateDropMQSNodeReq(void *buf, int32_t bufLen, SMCreateQnodeReq *pReq) {
   SEncoder encoder = {0};
@@ -2288,13 +2291,9 @@ int32_t tDeserializeSCreateDropMQSNodeReq(void *buf, int32_t bufLen, SMCreateQno
   return 0;
 }
 
-void tFreeSMCreateQnodeReq(SMCreateQnodeReq *pReq){
+void tFreeSMCreateQnodeReq(SMCreateQnodeReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
-void    tFreeSDDropQnodeReq(SDDropQnodeReq* pReq) {
+void tFreeSDDropQnodeReq(SDDropQnodeReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSDropDnodeReq(void *buf, int32_t bufLen, SDropDnodeReq *pReq) {
   SEncoder encoder = {0};
@@ -2336,9 +2335,7 @@ int32_t tDeserializeSDropDnodeReq(void *buf, int32_t bufLen, SDropDnodeReq *pReq
   return 0;
 }
 
-void tFreeSDropDnodeReq(SDropDnodeReq *pReq) {
+void tFreeSDropDnodeReq(SDropDnodeReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSRestoreDnodeReq(void *buf, int32_t bufLen, SRestoreDnodeReq *pReq) {
   SEncoder encoder = {0};
@@ -2369,9 +2366,7 @@ int32_t tDeserializeSRestoreDnodeReq(void *buf, int32_t bufLen, SRestoreDnodeReq
   return 0;
 }
 
-void tFreeSRestoreDnodeReq(SRestoreDnodeReq *pReq) {
+void tFreeSRestoreDnodeReq(SRestoreDnodeReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSMCfgDnodeReq(void *buf, int32_t bufLen, SMCfgDnodeReq *pReq) {
   SEncoder encoder = {0};
@@ -2404,9 +2399,7 @@ int32_t tDeserializeSMCfgDnodeReq(void *buf, int32_t bufLen, SMCfgDnodeReq *pReq
   return 0;
 }
 
-void tFreeSMCfgDnodeReq(SMCfgDnodeReq *pReq) {
+void tFreeSMCfgDnodeReq(SMCfgDnodeReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSDCfgDnodeReq(void *buf, int32_t bufLen, SDCfgDnodeReq *pReq) {
   SEncoder encoder = {0};
@@ -2464,9 +2457,7 @@ int32_t tDeserializeSCreateDnodeReq(void *buf, int32_t bufLen, SCreateDnodeReq *
   return 0;
 }
 
-void tFreeSCreateDnodeReq(SCreateDnodeReq *pReq) {
+void tFreeSCreateDnodeReq(SCreateDnodeReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSCreateFuncReq(void *buf, int32_t bufLen, SCreateFuncReq *pReq) {
   SEncoder encoder = {0};
@@ -3121,9 +3112,7 @@ int32_t tDeserializeSAlterDbReq(void *buf, int32_t bufLen, SAlterDbReq *pReq) {
   return 0;
 }
 
-void tFreeSAlterDbReq(SAlterDbReq *pReq) {
+void tFreeSAlterDbReq(SAlterDbReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSDropDbReq(void *buf, int32_t bufLen, SDropDbReq *pReq) {
   SEncoder encoder = {0};
@@ -3154,9 +3143,7 @@ int32_t tDeserializeSDropDbReq(void *buf, int32_t bufLen, SDropDbReq *pReq) {
   return 0;
 }
 
-void tFreeSDropDbReq(SDropDbReq *pReq) {
+void tFreeSDropDbReq(SDropDbReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSDropDbRsp(void *buf, int32_t bufLen, SDropDbRsp *pRsp) {
   SEncoder encoder = {0};
@@ -3435,9 +3422,7 @@ int32_t tDeserializeSCompactDbReq(void *buf, int32_t bufLen, SCompactDbReq *pReq
   return 0;
 }
 
-void tFreeSCompactDbReq(SCompactDbReq *pReq) {
+void tFreeSCompactDbReq(SCompactDbReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSUseDbRspImp(SEncoder *pEncoder, const SUseDbRsp *pRsp) {
   if (tEncodeCStr(pEncoder, pRsp->db) < 0) return -1;
@@ -4611,9 +4596,7 @@ int32_t tDeserializeSMDropTopicReq(void *buf, int32_t bufLen, SMDropTopicReq *pR
   return 0;
 }
 
-void tFreeSMDropTopicReq(SMDropTopicReq *pReq) {
+void tFreeSMDropTopicReq(SMDropTopicReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSMDropCgroupReq(void *buf, int32_t bufLen, SMDropCgroupReq *pReq) {
   SEncoder encoder = {0};
@@ -5501,9 +5484,7 @@ int32_t tDeserializeSBalanceVgroupReq(void *buf, int32_t bufLen, SBalanceVgroupR
   return 0;
 }
 
-void tFreeSBalanceVgroupReq(SBalanceVgroupReq *pReq) {
+void tFreeSBalanceVgroupReq(SBalanceVgroupReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSBalanceVgroupLeaderReq(void *buf, int32_t bufLen, SBalanceVgroupLeaderReq *pReq) {
   SEncoder encoder = {0};
@@ -5526,7 +5507,7 @@ int32_t tDeserializeSBalanceVgroupLeaderReq(void *buf, int32_t bufLen, SBalanceV
 
   if (tStartDecode(&decoder) < 0) return -1;
   if (tDecodeI32(&decoder, &pReq->useless) < 0) return -1;
-  if(!tDecodeIsEnd(&decoder)){
+  if (!tDecodeIsEnd(&decoder)) {
     if (tDecodeI32(&decoder, &pReq->vgId) < 0) return -1;
   }
 
@@ -5537,9 +5518,7 @@ int32_t tDeserializeSBalanceVgroupLeaderReq(void *buf, int32_t bufLen, SBalanceV
   return 0;
 }
 
-void tFreeSBalanceVgroupLeaderReq(SBalanceVgroupLeaderReq *pReq) {
+void tFreeSBalanceVgroupLeaderReq(SBalanceVgroupLeaderReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSMergeVgroupReq(void *buf, int32_t bufLen, SMergeVgroupReq *pReq) {
   SEncoder encoder = {0};
@@ -5601,9 +5580,7 @@ int32_t tDeserializeSRedistributeVgroupReq(void *buf, int32_t bufLen, SRedistrib
   return 0;
 }
 
-void tFreeSRedistributeVgroupReq(SRedistributeVgroupReq *pReq) {
+void tFreeSRedistributeVgroupReq(SRedistributeVgroupReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSSplitVgroupReq(void *buf, int32_t bufLen, SSplitVgroupReq *pReq) {
   SEncoder encoder = {0};
@@ -7152,9 +7129,7 @@ int32_t tDeserializeSMDropStreamReq(void *buf, int32_t bufLen, SMDropStreamReq *
   return 0;
 }
 
-void tFreeMDropStreamReq(SMDropStreamReq *pReq) {
+void tFreeMDropStreamReq(SMDropStreamReq *pReq) { FREESQL(); }
-  FREESQL();
-}
 
 int32_t tSerializeSMRecoverStreamReq(void *buf, int32_t bufLen, const SMRecoverStreamReq *pReq) {
   SEncoder encoder = {0};
@@ -7297,8 +7272,8 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) {
   } else {
     ASSERT(0);
   }
-  //ENCODESQL
+  // ENCODESQL
-  if(pReq->sqlLen > 0 && pReq->sql != NULL) {
+  if (pReq->sqlLen > 0 && pReq->sql != NULL) {
     if (tEncodeI32(pCoder, pReq->sqlLen) < 0) return -1;
     if (tEncodeBinary(pCoder, pReq->sql, pReq->sqlLen) < 0) return -1;
   }
@@ -7345,11 +7320,11 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) {
     ASSERT(0);
   }
 
-  //DECODESQL
+  // DECODESQL
-  if(!tDecodeIsEnd(pCoder)){
+  if (!tDecodeIsEnd(pCoder)) {
-    if(tDecodeI32(pCoder, &pReq->sqlLen) < 0) return -1;
+    if (tDecodeI32(pCoder, &pReq->sqlLen) < 0) return -1;
-    if(pReq->sqlLen > 0){
+    if (pReq->sqlLen > 0) {
-      if (tDecodeBinaryAlloc(pCoder, (void**)&pReq->sql, NULL) < 0) return -1;
+      if (tDecodeBinaryAlloc(pCoder, (void **)&pReq->sql, NULL) < 0) return -1;
     }
   }
 
@@ -7375,7 +7350,7 @@ void tDestroySVCreateTbReq(SVCreateTbReq *pReq, int32_t flags) {
     }
   }
 
-  if(pReq->sql != NULL){
+  if (pReq->sql != NULL) {
     taosMemoryFree(pReq->sql);
   }
   pReq->sql = NULL;
@@ -8194,7 +8169,7 @@ int32_t tEncodeMqDataRsp(SEncoder *pEncoder, const SMqDataRsp *pRsp) {
 
     for (int32_t i = 0; i < pRsp->blockNum; i++) {
       int32_t bLen = *(int32_t *)taosArrayGet(pRsp->blockDataLen, i);
-      void   *data = taosArrayGetP(pRsp->blockData, i);
+      void *  data = taosArrayGetP(pRsp->blockData, i);
       if (tEncodeBinary(pEncoder, (const uint8_t *)data, bLen) < 0) return -1;
       if (pRsp->withSchema) {
         SSchemaWrapper *pSW = (SSchemaWrapper *)taosArrayGetP(pRsp->blockSchema, i);
@@ -8227,7 +8202,7 @@ int32_t tDecodeMqDataRsp(SDecoder *pDecoder, SMqDataRsp *pRsp) {
     }
 
     for (int32_t i = 0; i < pRsp->blockNum; i++) {
-      void    *data;
+      void *   data;
       uint64_t bLen;
       if (tDecodeBinaryAlloc(pDecoder, &data, &bLen) < 0) return -1;
       taosArrayPush(pRsp->blockData, &data);
@@ -8273,7 +8248,7 @@ int32_t tEncodeSTaosxRsp(SEncoder *pEncoder, const STaosxRsp *pRsp) {
   if (tEncodeI32(pEncoder, pRsp->createTableNum) < 0) return -1;
   if (pRsp->createTableNum) {
     for (int32_t i = 0; i < pRsp->createTableNum; i++) {
-      void   *createTableReq = taosArrayGetP(pRsp->createTableReq, i);
+      void *  createTableReq = taosArrayGetP(pRsp->createTableReq, i);
       int32_t createTableLen = *(int32_t *)taosArrayGet(pRsp->createTableLen, i);
       if (tEncodeBinary(pEncoder, createTableReq, createTableLen) < 0) return -1;
     }
@@ -8289,7 +8264,7 @@ int32_t tDecodeSTaosxRsp(SDecoder *pDecoder, STaosxRsp *pRsp) {
     pRsp->createTableLen = taosArrayInit(pRsp->createTableNum, sizeof(int32_t));
     pRsp->createTableReq = taosArrayInit(pRsp->createTableNum, sizeof(void *));
     for (int32_t i = 0; i < pRsp->createTableNum; i++) {
-      void    *pCreate = NULL;
+      void *   pCreate = NULL;
       uint64_t len;
       if (tDecodeBinaryAlloc(pDecoder, &pCreate, &len) < 0) return -1;
       int32_t l = (int32_t)len;
@@ -8591,7 +8566,7 @@ void tDestroySubmitTbData(SSubmitTbData *pTbData, int32_t flag) {
       taosArrayDestroy(pTbData->aCol);
     } else {
       int32_t nRow = TARRAY_SIZE(pTbData->aRowP);
-      SRow  **rows = (SRow **)TARRAY_DATA(pTbData->aRowP);
+      SRow ** rows = (SRow **)TARRAY_DATA(pTbData->aRowP);
 
       for (int32_t i = 0; i < nRow; ++i) {
         tRowDestroy(rows[i]);
@@ -8855,7 +8830,7 @@ int32_t tDeserializeSCMCreateViewReq(void *buf, int32_t bufLen, SCMCreateViewReq
     }
 
     for (int32_t i = 0; i < pReq->numOfCols; ++i) {
-      SSchema* pSchema = pReq->pSchema + i;
+      SSchema *pSchema = pReq->pSchema + i;
       if (tDecodeSSchema(&decoder, pSchema) < 0) return -1;
     }
   }
@@ -8866,17 +8841,17 @@ int32_t tDeserializeSCMCreateViewReq(void *buf, int32_t bufLen, SCMCreateViewReq
   return 0;
 }
 
-void    tFreeSCMCreateViewReq(SCMCreateViewReq* pReq) {
+void tFreeSCMCreateViewReq(SCMCreateViewReq *pReq) {
   if (NULL == pReq) {
     return;
   }
 
   taosMemoryFreeClear(pReq->querySql);
-  taosMemoryFreeClear(pReq->sql);  
+  taosMemoryFreeClear(pReq->sql);
   taosMemoryFreeClear(pReq->pSchema);
 }
 
-int32_t tSerializeSCMDropViewReq(void* buf, int32_t bufLen, const SCMDropViewReq* pReq) {
+int32_t tSerializeSCMDropViewReq(void *buf, int32_t bufLen, const SCMDropViewReq *pReq) {
   SEncoder encoder = {0};
   tEncoderInit(&encoder, buf, bufLen);
 
@@ -8894,7 +8869,7 @@ int32_t tSerializeSCMDropViewReq(void* buf, int32_t bufLen, const SCMDropViewReq
   return tlen;
 }
 
-int32_t tDeserializeSCMDropViewReq(void* buf, int32_t bufLen, SCMDropViewReq* pReq) {
+int32_t tDeserializeSCMDropViewReq(void *buf, int32_t bufLen, SCMDropViewReq *pReq) {
   SDecoder decoder = {0};
   tDecoderInit(&decoder, buf, bufLen);
 
@@ -8910,15 +8885,15 @@ int32_t tDeserializeSCMDropViewReq(void* buf, int32_t bufLen, SCMDropViewReq* pR
   tDecoderClear(&decoder);
   return 0;
 }
-void    tFreeSCMDropViewReq(SCMDropViewReq* pReq) {
+void tFreeSCMDropViewReq(SCMDropViewReq *pReq) {
   if (NULL == pReq) {
     return;
   }
-  
+
   taosMemoryFree(pReq->sql);
 }
 
-int32_t tSerializeSViewMetaReq(void* buf, int32_t bufLen, const SViewMetaReq* pReq) {
+int32_t tSerializeSViewMetaReq(void *buf, int32_t bufLen, const SViewMetaReq *pReq) {
   SEncoder encoder = {0};
   tEncoderInit(&encoder, buf, bufLen);
 
@@ -8932,7 +8907,7 @@ int32_t tSerializeSViewMetaReq(void* buf, int32_t bufLen, const SViewMetaReq* pR
   return tlen;
 }
 
-int32_t tDeserializeSViewMetaReq(void* buf, int32_t bufLen, SViewMetaReq* pReq) {
+int32_t tDeserializeSViewMetaReq(void *buf, int32_t bufLen, SViewMetaReq *pReq) {
   SDecoder decoder = {0};
   tDecoderInit(&decoder, buf, bufLen);
 
@@ -8964,8 +8939,7 @@ static int32_t tEncodeSViewMetaRsp(SEncoder *pEncoder, const SViewMetaRsp *pRsp)
   return 0;
 }
 
-
+int32_t tSerializeSViewMetaRsp(void *buf, int32_t bufLen, const SViewMetaRsp *pRsp) {
-int32_t tSerializeSViewMetaRsp(void* buf, int32_t bufLen, const SViewMetaRsp* pRsp) {
   SEncoder encoder = {0};
   tEncoderInit(&encoder, buf, bufLen);
 
@@ -8998,7 +8972,7 @@ static int32_t tDecodeSViewMetaRsp(SDecoder *pDecoder, SViewMetaRsp *pRsp) {
     }
 
     for (int32_t i = 0; i < pRsp->numOfCols; ++i) {
-      SSchema* pSchema = pRsp->pSchema + i;
+      SSchema *pSchema = pRsp->pSchema + i;
       if (tDecodeSSchema(pDecoder, pSchema) < 0) return -1;
     }
   }
@@ -9006,7 +8980,7 @@ static int32_t tDecodeSViewMetaRsp(SDecoder *pDecoder, SViewMetaRsp *pRsp) {
   return 0;
 }
 
-int32_t tDeserializeSViewMetaRsp(void* buf, int32_t bufLen, SViewMetaRsp* pRsp) {
+int32_t tDeserializeSViewMetaRsp(void *buf, int32_t bufLen, SViewMetaRsp *pRsp) {
   SDecoder decoder = {0};
   tDecoderInit(&decoder, buf, bufLen);
 
@@ -9019,7 +8993,7 @@ int32_t tDeserializeSViewMetaRsp(void* buf, int32_t bufLen, SViewMetaRsp* pRsp)
   return 0;
 }
 
-void    tFreeSViewMetaRsp(SViewMetaRsp* pRsp) {
+void tFreeSViewMetaRsp(SViewMetaRsp *pRsp) {
   if (NULL == pRsp) {
     return;
   }
@@ -9064,7 +9038,7 @@ int32_t tDeserializeSViewHbRsp(void *buf, int32_t bufLen, SViewHbRsp *pRsp) {
   }
 
   for (int32_t i = 0; i < numOfMeta; ++i) {
-    SViewMetaRsp* metaRsp = taosMemoryCalloc(1, sizeof(SViewMetaRsp));
+    SViewMetaRsp *metaRsp = taosMemoryCalloc(1, sizeof(SViewMetaRsp));
     if (NULL == metaRsp) return -1;
     if (tDecodeSViewMetaRsp(&decoder, metaRsp) < 0) return -1;
     taosArrayPush(pRsp->pViewRsp, &metaRsp);
@@ -9086,7 +9060,3 @@ void tFreeSViewHbRsp(SViewHbRsp *pRsp) {
 
   taosArrayDestroy(pRsp->pViewRsp);
 }
-
-
-
-

source/common/src/ttime.c

@@ -646,7 +646,7 @@ int32_t parseAbsoluteDuration(const char* token, int32_t tokenlen, int64_t* dura
 
   /* get the basic numeric value */
   int64_t timestamp = taosStr2Int64(token, &endPtr, 10);
-  if (timestamp < 0 || errno != 0) {
+  if ((timestamp == 0 && token[0] != '0') || errno != 0) {
     return -1;
   }
 

source/dnode/mgmt/node_mgmt/src/dmEnv.c

@@ -28,6 +28,9 @@
     }                                \
   } while (0)
 
+extern int32_t streamTimerInit();
+extern void    streamTimerCleanUp();
+
 static SDnode globalDnode = {0};
 
 SDnode *dmInstance() { return &globalDnode; }
@@ -166,6 +169,7 @@ int32_t dmInit() {
 #if defined(USE_S3)
   if (s3Begin() != 0) return -1;
 #endif
+  if (streamTimerInit() != 0) return -1;
 
   dInfo("dnode env is initialized");
   return 0;
@@ -194,6 +198,8 @@ void dmCleanup() {
 #if defined(USE_S3)
   s3End();
 #endif
+  streamTimerCleanUp();
+
   dInfo("dnode env is cleaned up");
 
   taosCleanupCfg();

source/dnode/mnode/impl/src/mndSnode.c

@@ -291,12 +291,17 @@ static int32_t mndProcessCreateSnodeReq(SRpcMsg *pReq) {
     goto _OVER;
   }
 
-  pObj = mndAcquireSnode(pMnode, createReq.dnodeId);
+//  pObj = mndAcquireSnode(pMnode, createReq.dnodeId);
-  if (pObj != NULL) {
+//  if (pObj != NULL) {
+//    terrno = TSDB_CODE_MND_SNODE_ALREADY_EXIST;
+//    goto _OVER;
+//  } else if (terrno != TSDB_CODE_MND_SNODE_NOT_EXIST) {
+//    goto _OVER;
+//  }
+
+  if (sdbGetSize(pMnode->pSdb, SDB_SNODE) >= 1){
     terrno = TSDB_CODE_MND_SNODE_ALREADY_EXIST;
     goto _OVER;
-  } else if (terrno != TSDB_CODE_MND_SNODE_NOT_EXIST) {
-    goto _OVER;
   }
 
   pDnode = mndAcquireDnode(pMnode, createReq.dnodeId);
@@ -314,7 +319,7 @@ _OVER:
     return -1;
   }
 
-  mndReleaseSnode(pMnode, pObj);
+//  mndReleaseSnode(pMnode, pObj);
   mndReleaseDnode(pMnode, pDnode);
   tFreeSMCreateQnodeReq(&createReq);
   return code;

source/dnode/mnode/impl/src/mndStream.c

@@ -363,35 +363,6 @@ static int32_t mndCheckCreateStreamReq(SCMCreateStreamReq *pCreate) {
   return 0;
 }
 
-static int32_t mndStreamGetPlanString(const char *ast, int8_t triggerType, int64_t watermark, char **pStr) {
-  if (NULL == ast) {
-    return TSDB_CODE_SUCCESS;
-  }
-
-  SNode * pAst = NULL;
-  int32_t code = nodesStringToNode(ast, &pAst);
-
-  SQueryPlan *pPlan = NULL;
-  if (TSDB_CODE_SUCCESS == code) {
-    SPlanContext cxt = {
-        .pAstRoot = pAst,
-        .topicQuery = false,
-        .streamQuery = true,
-        .triggerType = (triggerType == STREAM_TRIGGER_MAX_DELAY) ? STREAM_TRIGGER_WINDOW_CLOSE : triggerType,
-        .watermark = watermark,
-    };
-    code = qCreateQueryPlan(&cxt, &pPlan, NULL);
-  }
-
-  if (TSDB_CODE_SUCCESS == code) {
-    code = nodesNodeToString((SNode *)pPlan, false, pStr, NULL);
-  }
-  nodesDestroyNode(pAst);
-  nodesDestroyNode((SNode *)pPlan);
-  terrno = code;
-  return code;
-}
-
 static int32_t mndBuildStreamObjFromCreateReq(SMnode *pMnode, SStreamObj *pObj, SCMCreateStreamReq *pCreate) {
   SNode *     pAst = NULL;
   SQueryPlan *pPlan = NULL;
@@ -733,11 +704,20 @@ static int32_t mndPersistTaskDropReq(SMnode *pMnode, STrans *pTrans, SStreamTask
   pReq->streamId = pTask->id.streamId;
 
   STransAction action = {0};
-  SEpSet       epset = {0};
+  SEpSet  epset = {0};
-  if (pTask->info.nodeId == SNODE_HANDLE) {
+  if(pTask->info.nodeId == SNODE_HANDLE){
-    SSnodeObj *pObj = mndAcquireSnode(pMnode, pTask->info.nodeId);
+    SSnodeObj *pObj = NULL;
-    addEpIntoEpSet(&epset, pObj->pDnode->fqdn, pObj->pDnode->port);
+    void  *pIter = NULL;
-  } else {
+    while (1) {
+      pIter = sdbFetch(pMnode->pSdb, SDB_SNODE, pIter, (void **)&pObj);
+      if (pIter == NULL) {
+        break;
+      }
+
+      addEpIntoEpSet(&epset, pObj->pDnode->fqdn, pObj->pDnode->port);
+      sdbRelease(pMnode->pSdb, pObj);
+    }
+  }else{
     SVgObj *pVgObj = mndAcquireVgroup(pMnode, pTask->info.nodeId);
     epset = mndGetVgroupEpset(pMnode, pVgObj);
     mndReleaseVgroup(pMnode, pVgObj);

source/dnode/mnode/impl/src/mndSubscribe.c

@@ -161,6 +161,7 @@ static int32_t mndBuildSubChangeReq(void **pBuf, int32_t *pLen, SMqSubscribeObj
 static int32_t mndPersistSubChangeVgReq(SMnode *pMnode, STrans *pTrans, SMqSubscribeObj *pSub,
                                         const SMqRebOutputVg *pRebVg, SSubplan* pPlan) {
   if (pRebVg->oldConsumerId == pRebVg->newConsumerId) {
+    if(pRebVg->oldConsumerId == -1) return 0;   //drop stream, no consumer, while split vnode,all consumerId is -1
     terrno = TSDB_CODE_MND_INVALID_SUB_OPTION;
     return -1;
   }

source/dnode/snode/src/snodeInitApi.c

@@ -80,6 +80,7 @@ void initStateStoreAPI(SStateStore* pStore) {
   pStore->updateInfoDestroy = updateInfoDestroy;
   pStore->windowSBfDelete = windowSBfDelete;
   pStore->windowSBfAdd = windowSBfAdd;
+  pStore->isIncrementalTimeStamp = isIncrementalTimeStamp;
 
   pStore->updateInfoInitP = updateInfoInitP;
   pStore->updateInfoAddCloseWindowSBF = updateInfoAddCloseWindowSBF;

source/dnode/vnode/inc/vnode.h

@@ -153,26 +153,6 @@ typedef struct STsdbReader STsdbReader;
 #define CACHESCAN_RETRIEVE_LAST_ROW    0x4
 #define CACHESCAN_RETRIEVE_LAST        0x8
 
-int32_t      tsdbReaderOpen(void *pVnode, SQueryTableDataCond *pCond, void *pTableList, int32_t numOfTables,
-                            SSDataBlock *pResBlock, void **ppReader, const char *idstr, bool countOnly,
-                            SHashObj **pIgnoreTables);
-int32_t      tsdbSetTableList(STsdbReader *pReader, const void *pTableList, int32_t num);
-void         tsdbReaderSetId(STsdbReader *pReader, const char *idstr);
-void         tsdbReaderClose(STsdbReader *pReader);
-int32_t      tsdbNextDataBlock(STsdbReader *pReader, bool *hasNext);
-int32_t      tsdbRetrieveDatablockSMA(STsdbReader *pReader, SSDataBlock *pDataBlock, bool *allHave, bool *hasNullSMA);
-void         tsdbReleaseDataBlock(STsdbReader *pReader);
-SSDataBlock *tsdbRetrieveDataBlock(STsdbReader *pTsdbReadHandle, SArray *pColumnIdList);
-int32_t      tsdbReaderReset(STsdbReader *pReader, SQueryTableDataCond *pCond);
-int32_t      tsdbGetFileBlocksDistInfo(STsdbReader *pReader, STableBlockDistInfo *pTableBlockInfo);
-int64_t      tsdbGetNumOfRowsInMemTable(STsdbReader *pHandle);
-void        *tsdbGetIdx(SMeta *pMeta);
-void        *tsdbGetIvtIdx(SMeta *pMeta);
-uint64_t     tsdbGetReaderMaxVersion(STsdbReader *pReader);
-void         tsdbReaderSetCloseFlag(STsdbReader *pReader);
-int64_t      tsdbGetLastTimestamp(SVnode *pVnode, void *pTableList, int32_t numOfTables, const char *pIdStr);
-
-//======================================================================================================================
 int32_t      tsdbReaderOpen2(void *pVnode, SQueryTableDataCond *pCond, void *pTableList, int32_t numOfTables,
                              SSDataBlock *pResBlock, void **ppReader, const char *idstr, bool countOnly,
                              SHashObj **pIgnoreTables);
@@ -193,7 +173,6 @@ void         tsdbReaderSetCloseFlag(STsdbReader *pReader);
 int64_t      tsdbGetLastTimestamp2(SVnode *pVnode, void *pTableList, int32_t numOfTables, const char *pIdStr);
 void         tsdbSetDurationOrder(STsdbReader* pReader);
 void         tsdbReaderSetNotifyCb(STsdbReader* pReader, TsdReaderNotifyCbFn notifyFn, void* param);
-//======================================================================================================================
 
 int32_t tsdbReuseCacherowsReader(void *pReader, void *pTableIdList, int32_t numOfTables);
 int32_t tsdbCacherowsReaderOpen(void *pVnode, int32_t type, void *pTableIdList, int32_t numOfTables, int32_t numOfCols,

source/dnode/vnode/src/inc/tsdb.h

@@ -377,6 +377,7 @@ struct STsdb {
   SVnode              *pVnode;
   STsdbKeepCfg         keepCfg;
   TdThreadMutex        mutex;
+  bool                 bgTaskDisabled;
   SMemTable           *mem;
   SMemTable           *imem;
   STsdbFS              fs;  // old
@@ -885,10 +886,9 @@ void tMergeTreeUnpinSttBlock(SMergeTree *pMTree);
 bool tMergeTreeIgnoreEarlierTs(SMergeTree *pMTree);
 void tMergeTreeClose(SMergeTree *pMTree);
 
-SSttBlockLoadInfo *tCreateOneLastBlockLoadInfo(STSchema *pSchema, int16_t *colList, int32_t numOfCols);
+SSttBlockLoadInfo *tCreateSttBlockLoadInfo(STSchema *pSchema, int16_t *colList, int32_t numOfCols);
-void               resetLastBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo);
 void               getSttBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo, SSttBlockLoadCostInfo *pLoadCost);
-void              *destroyLastBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo);
+void              *destroySttBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo);
 void              *destroySttBlockReader(SArray *pLDataIterArray, SSttBlockLoadCostInfo *pLoadCost);
 
 // tsdbCache ==============================================================================================

source/dnode/vnode/src/inc/vnodeInt.h

@@ -223,8 +223,6 @@ int32_t tsdbDeleteTableData(STsdb* pTsdb, int64_t version, tb_uid_t suid, tb_uid
 int32_t tsdbSetKeepCfg(STsdb* pTsdb, STsdbCfg* pCfg);
 
 // tq
-int     tqInit();
-void    tqCleanUp();
 STQ*    tqOpen(const char* path, SVnode* pVnode);
 void    tqNotifyClose(STQ*);
 void    tqClose(STQ*);

source/dnode/vnode/src/tq/tq.c

@@ -17,12 +17,6 @@
 #include "vnd.h"
 #include "tqCommon.h"
 
-typedef struct {
-  int8_t inited;
-} STqMgmt;
-
-static STqMgmt tqMgmt = {0};
-
 // 0: not init
 // 1: already inited
 // 2: wait to be inited or cleaup
@@ -32,36 +26,6 @@ static FORCE_INLINE bool tqIsHandleExec(STqHandle* pHandle) { return TMQ_HANDLE_
 static FORCE_INLINE void tqSetHandleExec(STqHandle* pHandle) { pHandle->status = TMQ_HANDLE_STATUS_EXEC; }
 static FORCE_INLINE void tqSetHandleIdle(STqHandle* pHandle) { pHandle->status = TMQ_HANDLE_STATUS_IDLE; }
 
-int32_t tqInit() {
-  int8_t old;
-  while (1) {
-    old = atomic_val_compare_exchange_8(&tqMgmt.inited, 0, 2);
-    if (old != 2) break;
-  }
-
-  if (old == 0) {
-    if (streamInit() < 0) {
-      return -1;
-    }
-    atomic_store_8(&tqMgmt.inited, 1);
-  }
-
-  return 0;
-}
-
-void tqCleanUp() {
-  int8_t old;
-  while (1) {
-    old = atomic_val_compare_exchange_8(&tqMgmt.inited, 1, 2);
-    if (old != 2) break;
-  }
-
-  if (old == 1) {
-    streamCleanUp();
-    atomic_store_8(&tqMgmt.inited, 0);
-  }
-}
-
 void tqDestroyTqHandle(void* data) {
   STqHandle* pData = (STqHandle*)data;
   qDestroyTask(pData->execHandle.task);
@@ -337,7 +301,7 @@ int32_t tqProcessPollPush(STQ* pTq, SRpcMsg* pMsg) {
 
     while (pIter) {
       STqHandle* pHandle = *(STqHandle**)pIter;
-      tqInfo("vgId:%d start set submit for pHandle:%p, consumer:0x%" PRIx64, vgId, pHandle, pHandle->consumerId);
+      tqDebug("vgId:%d start set submit for pHandle:%p, consumer:0x%" PRIx64, vgId, pHandle, pHandle->consumerId);
 
       if (ASSERT(pHandle->msg != NULL)) {
         tqError("pHandle->msg should not be null");

source/dnode/vnode/src/tq/tqPush.c

@@ -72,7 +72,7 @@ int32_t tqRegisterPushHandle(STQ* pTq, void* handle, SRpcMsg* pMsg) {
   memcpy(pHandle->msg->pCont, pMsg->pCont, pMsg->contLen);
   pHandle->msg->contLen = pMsg->contLen;
   int32_t ret = taosHashPut(pTq->pPushMgr, pHandle->subKey, strlen(pHandle->subKey), &pHandle, POINTER_BYTES);
-  tqInfo("vgId:%d data is over, ret:%d, consumerId:0x%" PRIx64 ", register to pHandle:%p, pCont:%p, len:%d", vgId, ret,
+  tqDebug("vgId:%d data is over, ret:%d, consumerId:0x%" PRIx64 ", register to pHandle:%p, pCont:%p, len:%d", vgId, ret,
           pHandle->consumerId, pHandle, pHandle->msg->pCont, pHandle->msg->contLen);
   return 0;
 }

source/dnode/vnode/src/tqCommon/tqCommon.c

@@ -475,6 +475,7 @@ int32_t tqStreamTaskProcessCheckRsp(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLe
       rsp.upstreamTaskId, rsp.upstreamNodeId, rsp.reqId, rsp.downstreamTaskId, rsp.downstreamNodeId, rsp.status);
 
   if (!isLeader) {
+    streamMetaUpdateTaskDownstreamStatus(pMeta, rsp.streamId, rsp.upstreamTaskId, 0, taosGetTimestampMs(), false);
     tqError("vgId:%d not leader, task:0x%x not handle the check rsp, downstream:0x%x (vgId:%d)", vgId,
             rsp.upstreamTaskId, rsp.downstreamTaskId, rsp.downstreamNodeId);
     return code;
@@ -482,6 +483,7 @@ int32_t tqStreamTaskProcessCheckRsp(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLe
 
   SStreamTask* pTask = streamMetaAcquireTask(pMeta, rsp.streamId, rsp.upstreamTaskId);
   if (pTask == NULL) {
+    streamMetaUpdateTaskDownstreamStatus(pMeta, rsp.streamId, rsp.upstreamTaskId, 0, taosGetTimestampMs(), false);
     tqError("tq failed to locate the stream task:0x%" PRIx64 "-0x%x (vgId:%d), it may have been destroyed or stopped",
         rsp.streamId, rsp.upstreamTaskId, vgId);
     terrno = TSDB_CODE_STREAM_TASK_NOT_EXIST;
@@ -670,7 +672,8 @@ int32_t startStreamTasks(SStreamMeta* pMeta) {
         streamLaunchFillHistoryTask(pTask);
       }
 
-      streamMetaUpdateTaskDownstreamStatus(pTask, pTask->execInfo.init, pTask->execInfo.start, true);
+      streamMetaUpdateTaskDownstreamStatus(pTask->pMeta, pTask->id.streamId, pTask->id.taskId, pTask->execInfo.init,
+                                           pTask->execInfo.start, true);
       streamMetaReleaseTask(pMeta, pTask);
       continue;
     }

source/dnode/vnode/src/tsdb/tsdbFS2.c

@@ -745,18 +745,28 @@ _exit:
   return code;
 }
 
-int32_t tsdbFSCancelAllBgTask(STFileSystem *fs) {
+static int32_t tsdbFSSetBlockCommit(STFileSet *fset, bool block);
+
+int32_t tsdbDisableAndCancelAllBgTask(STsdb *pTsdb) {
+  STFileSystem *fs = pTsdb->pFS;
   TARRAY2(int64_t) channelArr = {0};
 
-  // collect all open channels
   taosThreadMutexLock(&fs->tsdb->mutex);
+
+  // disable
+  pTsdb->bgTaskDisabled = true;
+
+  // collect channel
   STFileSet *fset;
   TARRAY2_FOREACH(fs->fSetArr, fset) {
     if (VNODE_ASYNC_VALID_CHANNEL_ID(fset->bgTaskChannel)) {
       TARRAY2_APPEND(&channelArr, fset->bgTaskChannel);
       fset->bgTaskChannel = 0;
     }
+    fset->mergeScheduled = false;
+    tsdbFSSetBlockCommit(fset, false);
   }
+
   taosThreadMutexUnlock(&fs->tsdb->mutex);
 
   // destroy all channels
@@ -766,10 +776,17 @@ int32_t tsdbFSCancelAllBgTask(STFileSystem *fs) {
   return 0;
 }
 
+int32_t tsdbEnableBgTask(STsdb *pTsdb) {
+  taosThreadMutexLock(&pTsdb->mutex);
+  pTsdb->bgTaskDisabled = false;
+  taosThreadMutexUnlock(&pTsdb->mutex);
+  return 0;
+}
+
 int32_t tsdbCloseFS(STFileSystem **fs) {
   if (fs[0] == NULL) return 0;
 
-  tsdbFSCancelAllBgTask(*fs);
+  tsdbDisableAndCancelAllBgTask((*fs)->tsdb);
   close_file_system(fs[0]);
   destroy_fs(fs);
   return 0;
@@ -857,7 +874,7 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) {
 
   // schedule merge
   int32_t sttTrigger = fs->tsdb->pVnode->config.sttTrigger;
-  if (sttTrigger > 1) {
+  if (sttTrigger > 1 && !fs->tsdb->bgTaskDisabled) {
     STFileSet *fset;
     TARRAY2_FOREACH_REVERSE(fs->fSetArr, fset) {
       if (TARRAY2_SIZE(fset->lvlArr) == 0) {
@@ -873,7 +890,7 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) {
 
       bool    skipMerge = false;
       int32_t numFile = TARRAY2_SIZE(lvl->fobjArr);
-      if (numFile >= sttTrigger) {
+      if (numFile >= sttTrigger && (!fset->mergeScheduled)) {
         // launch merge
         {
           extern int8_t  tsS3Enabled;
@@ -917,6 +934,7 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) {
           code = vnodeAsyncC(vnodeAsyncHandle[1], fset->bgTaskChannel, EVA_PRIORITY_HIGH, tsdbMerge, taosMemoryFree,
                              arg, NULL);
           TSDB_CHECK_CODE(code, lino, _exit);
+          fset->mergeScheduled = true;
         }
       }
 

source/dnode/vnode/src/tsdb/tsdbFSet2.c

@@ -453,6 +453,7 @@ int32_t tsdbTFileSetInit(int32_t fid, STFileSet **fset) {
 
   // background task queue
   fset[0]->bgTaskChannel = 0;
+  fset[0]->mergeScheduled = false;
 
   // block commit variables
   taosThreadCondInit(&fset[0]->canCommit, NULL);

source/dnode/vnode/src/tsdb/tsdbFSet2.h

@@ -93,6 +93,7 @@ struct STFileSet {
 
   // background task channel
   int64_t bgTaskChannel;
+  bool    mergeScheduled;
 
   // block commit variables
   TdThreadCond canCommit;

source/dnode/vnode/src/tsdb/tsdbMerge.c

@@ -514,6 +514,8 @@ static int32_t tsdbMergeGetFSet(SMerger *merger) {
     return 0;
   }
 
+  fset->mergeScheduled = false;
+
   int32_t code = tsdbTFileSetInitCopy(merger->tsdb, fset, &merger->fset);
   if (code) {
     taosThreadMutexUnlock(&merger->tsdb->mutex);

source/dnode/vnode/src/tsdb/tsdbMergeTree.c

@@ -22,7 +22,7 @@
 static void tLDataIterClose2(SLDataIter *pIter);
 
 // SLDataIter =================================================
-SSttBlockLoadInfo *tCreateOneLastBlockLoadInfo(STSchema *pSchema, int16_t *colList, int32_t numOfCols) {
+SSttBlockLoadInfo *tCreateSttBlockLoadInfo(STSchema *pSchema, int16_t *colList, int32_t numOfCols) {
   SSttBlockLoadInfo *pLoadInfo = taosMemoryCalloc(1, sizeof(SSttBlockLoadInfo));
   if (pLoadInfo == NULL) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -61,7 +61,7 @@ void getSttBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo, SSttBlockLoadCostInfo* pL
   }
 }
 
-void *destroyLastBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo) {
+void *destroySttBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo) {
   if (pLoadInfo == NULL) {
     return NULL;
   }
@@ -78,14 +78,19 @@ void *destroyLastBlockLoadInfo(SSttBlockLoadInfo *pLoadInfo) {
   pInfo->sttBlockIndex = -1;
   pInfo->pin = false;
 
+  if (pLoadInfo->statisBlock != NULL) {
+    tStatisBlockDestroy(pLoadInfo->statisBlock);
+    taosMemoryFreeClear(pLoadInfo->statisBlock);
+  }
+
   taosArrayDestroy(pLoadInfo->aSttBlk);
   taosMemoryFree(pLoadInfo);
   return NULL;
 }
 
-static void destroyLDataIter(SLDataIter *pIter) {
+void destroyLDataIter(SLDataIter *pIter) {
   tLDataIterClose2(pIter);
-  destroyLastBlockLoadInfo(pIter->pBlockLoadInfo);
+  destroySttBlockLoadInfo(pIter->pBlockLoadInfo);
   taosMemoryFree(pIter);
 }
 
@@ -732,25 +737,6 @@ static FORCE_INLINE int32_t tLDataIterDescCmprFn(const SRBTreeNode *p1, const SR
   return -1 * tLDataIterCmprFn(p1, p2);
 }
 
-static void adjustValidLDataIters(SArray *pLDIterList, int32_t numOfFileObj) {
-  int32_t size = taosArrayGetSize(pLDIterList);
-
-  if (size < numOfFileObj) {
-    int32_t inc = numOfFileObj - size;
-    for (int32_t k = 0; k < inc; ++k) {
-      SLDataIter *pIter = taosMemoryCalloc(1, sizeof(SLDataIter));
-      taosArrayPush(pLDIterList, &pIter);
-    }
-  } else if (size > numOfFileObj) {  // remove unused LDataIter
-    int32_t inc = size - numOfFileObj;
-
-    for (int i = 0; i < inc; ++i) {
-      SLDataIter *pIter = taosArrayPop(pLDIterList);
-      destroyLDataIter(pIter);
-    }
-  }
-}
-
 int32_t tMergeTreeOpen2(SMergeTree *pMTree, SMergeTreeConf *pConf) {
   int32_t code = TSDB_CODE_SUCCESS;
 
@@ -773,19 +759,13 @@ int32_t tMergeTreeOpen2(SMergeTree *pMTree, SMergeTreeConf *pConf) {
   }
 
   // add the list/iter placeholder
-  while (taosArrayGetSize(pConf->pSttFileBlockIterArray) < numOfLevels) {
+  adjustLDataIters(pConf->pSttFileBlockIterArray, pConf->pCurrentFileset);
-    SArray *pList = taosArrayInit(4, POINTER_BYTES);
-    taosArrayPush(pConf->pSttFileBlockIterArray, &pList);
-  }
 
   for (int32_t j = 0; j < numOfLevels; ++j) {
     SSttLvl *pSttLevel = ((STFileSet *)pConf->pCurrentFileset)->lvlArr->data[j];
     SArray  *pList = taosArrayGetP(pConf->pSttFileBlockIterArray, j);
 
-    int32_t numOfFileObj = TARRAY2_SIZE(pSttLevel->fobjArr);
+    for (int32_t i = 0; i < TARRAY2_SIZE(pSttLevel->fobjArr); ++i) {  // open all last file
-    adjustValidLDataIters(pList, numOfFileObj);
-
-    for (int32_t i = 0; i < numOfFileObj; ++i) {  // open all last file
       SLDataIter *pIter = taosArrayGetP(pList, i);
 
       SSttFileReader    *pSttFileReader = pIter->pReader;
@@ -805,7 +785,7 @@ int32_t tMergeTreeOpen2(SMergeTree *pMTree, SMergeTreeConf *pConf) {
       }
 
       if (pLoadInfo == NULL) {
-        pLoadInfo = tCreateOneLastBlockLoadInfo(pConf->pSchema, pConf->pCols, pConf->numOfCols);
+        pLoadInfo = tCreateSttBlockLoadInfo(pConf->pSchema, pConf->pCols, pConf->numOfCols);
       }
 
       memset(pIter, 0, sizeof(SLDataIter));

source/dnode/vnode/src/tsdb/tsdbRead2.c

@@ -23,14 +23,14 @@
 #include "tsimplehash.h"
 
 #define ASCENDING_TRAVERSE(o)        (o == TSDB_ORDER_ASC)
-#define getCurrentKeyInLastBlock(_r) ((_r)->currentKey)
+#define getCurrentKeyInSttBlock(_r)  ((_r)->currentKey)
 
 static SFileDataBlockInfo* getCurrentBlockInfo(SDataBlockIter* pBlockIter);
 static int32_t  buildDataBlockFromBufImpl(STableBlockScanInfo* pBlockScanInfo, int64_t endKey, int32_t capacity,
                                           STsdbReader* pReader);
 static TSDBROW* getValidMemRow(SIterInfo* pIter, const SArray* pDelList, STsdbReader* pReader);
 static int32_t  doMergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pScanInfo, STsdbReader* pReader);
-static int32_t  doMergeRowsInLastBlock(SLastBlockReader* pLastBlockReader, STableBlockScanInfo* pScanInfo, int64_t ts,
+static int32_t  doMergeRowsInSttBlock(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, int64_t ts,
                                        SRowMerger* pMerger, SVersionRange* pVerRange, const char* id);
 static int32_t  doMergeRowsInBuf(SIterInfo* pIter, uint64_t uid, int64_t ts, SArray* pDelList, STsdbReader* pReader);
 static int32_t  doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, SRow* pTSRow,
@@ -52,10 +52,10 @@ static int32_t initDelSkylineIterator(STableBlockScanInfo* pBlockScanInfo, int32
 static STsdb* getTsdbByRetentions(SVnode* pVnode, SQueryTableDataCond* pCond, SRetention* retentions, const char* idstr,
                                   int8_t* pLevel);
 static SVersionRange getQueryVerRange(SVnode* pVnode, SQueryTableDataCond* pCond, int8_t level);
-static bool          hasDataInLastBlock(SLastBlockReader* pLastBlockReader);
 static int32_t       doBuildDataBlock(STsdbReader* pReader);
 static TSDBKEY       getCurrentKeyInBuf(STableBlockScanInfo* pScanInfo, STsdbReader* pReader);
 static bool          hasDataInFileBlock(const SBlockData* pBlockData, const SFileBlockDumpInfo* pDumpInfo);
+static bool          hasDataInSttBlock(SSttBlockReader* pSttBlockReader);
 static void          initBlockDumpInfo(STsdbReader* pReader, SDataBlockIter* pBlockIter);
 static int32_t       getInitialDelIndex(const SArray* pDelSkyline, int32_t order);
 static void          resetTableListIndex(SReaderStatus* pStatus);
@@ -140,16 +140,16 @@ static int32_t initFilesetIterator(SFilesetIter* pIter, TFileSetArray* pFileSetA
   pIter->pFilesetList = pFileSetArray;
   pIter->numOfFiles = numOfFileset;
 
-  if (pIter->pLastBlockReader == NULL) {
+  if (pIter->pSttBlockReader == NULL) {
-    pIter->pLastBlockReader = taosMemoryCalloc(1, sizeof(struct SLastBlockReader));
+    pIter->pSttBlockReader = taosMemoryCalloc(1, sizeof(struct SSttBlockReader));
-    if (pIter->pLastBlockReader == NULL) {
+    if (pIter->pSttBlockReader == NULL) {
       int32_t code = TSDB_CODE_OUT_OF_MEMORY;
       tsdbError("failed to prepare the last block iterator, since:%s %s", tstrerror(code), pReader->idStr);
       return code;
     }
   }
 
-  SLastBlockReader* pLReader = pIter->pLastBlockReader;
+  SSttBlockReader* pLReader = pIter->pSttBlockReader;
   pLReader->order = pReader->info.order;
   pLReader->window = pReader->info.window;
   pLReader->verRange = pReader->info.verRange;
@@ -173,8 +173,8 @@ static int32_t filesetIteratorNext(SFilesetIter* pIter, STsdbReader* pReader, bo
 
   SReadCostSummary* pCost = &pReader->cost;
 
-  pIter->pLastBlockReader->uid = 0;
+  pIter->pSttBlockReader->uid = 0;
-  tMergeTreeClose(&pIter->pLastBlockReader->mergeTree);
+  tMergeTreeClose(&pIter->pSttBlockReader->mergeTree);
   pReader->status.pLDataIterArray = destroySttBlockReader(pReader->status.pLDataIterArray, &pCost->sttCost);
   pReader->status.pLDataIterArray = taosArrayInit(4, POINTER_BYTES);
 
@@ -1409,26 +1409,26 @@ static bool tryCopyDistinctRowFromFileBlock(STsdbReader* pReader, SBlockData* pB
   return code;
 }
 
-static bool nextRowFromSttBlocks(SLastBlockReader* pLastBlockReader, STableBlockScanInfo* pScanInfo,
+static bool nextRowFromSttBlocks(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo,
                                  SVersionRange* pVerRange) {
-  int32_t step = ASCENDING_TRAVERSE(pLastBlockReader->order) ? 1 : -1;
+  int32_t step = ASCENDING_TRAVERSE(pSttBlockReader->order) ? 1 : -1;
 
   while (1) {
-    bool hasVal = tMergeTreeNext(&pLastBlockReader->mergeTree);
+    bool hasVal = tMergeTreeNext(&pSttBlockReader->mergeTree);
     if (!hasVal) {  // the next value will be the accessed key in stt
       pScanInfo->sttKeyInfo.status = STT_FILE_NO_DATA;
       pScanInfo->sttKeyInfo.nextProcKey += step;
       return false;
     }
 
-    TSDBROW* pRow = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+    TSDBROW* pRow = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
     int64_t  key = pRow->pBlockData->aTSKEY[pRow->iRow];
     int64_t  ver = pRow->pBlockData->aVersion[pRow->iRow];
 
-    pLastBlockReader->currentKey = key;
+    pSttBlockReader->currentKey = key;
     pScanInfo->sttKeyInfo.nextProcKey = key;
 
-    if (!hasBeenDropped(pScanInfo->delSkyline, &pScanInfo->sttBlockDelIndex, key, ver, pLastBlockReader->order,
+    if (!hasBeenDropped(pScanInfo->delSkyline, &pScanInfo->sttBlockDelIndex, key, ver, pSttBlockReader->order,
                         pVerRange)) {
       pScanInfo->sttKeyInfo.status = STT_FILE_HAS_DATA;
       return true;
@@ -1436,26 +1436,26 @@ static bool nextRowFromSttBlocks(SLastBlockReader* pLastBlockReader, STableBlock
   }
 }
 
-static void doPinSttBlock(SLastBlockReader* pLastBlockReader) {
+static void doPinSttBlock(SSttBlockReader* pSttBlockReader) {
-  tMergeTreePinSttBlock(&pLastBlockReader->mergeTree);
+  tMergeTreePinSttBlock(&pSttBlockReader->mergeTree);
 }
 
-static void doUnpinSttBlock(SLastBlockReader* pLastBlockReader) {
+static void doUnpinSttBlock(SSttBlockReader* pSttBlockReader) {
-  tMergeTreeUnpinSttBlock(&pLastBlockReader->mergeTree);
+  tMergeTreeUnpinSttBlock(&pSttBlockReader->mergeTree);
 }
 
-static bool tryCopyDistinctRowFromSttBlock(TSDBROW* fRow, SLastBlockReader* pLastBlockReader,
+static bool tryCopyDistinctRowFromSttBlock(TSDBROW* fRow, SSttBlockReader* pSttBlockReader,
                                            STableBlockScanInfo* pScanInfo, int64_t ts, STsdbReader* pReader,
                                            bool* copied) {
   int32_t code = TSDB_CODE_SUCCESS;
   *copied = false;
 
   // avoid the fetch next row replace the referenced stt block in buffer
-  doPinSttBlock(pLastBlockReader);
+  doPinSttBlock(pSttBlockReader);
-  bool hasVal = nextRowFromSttBlocks(pLastBlockReader, pScanInfo, &pReader->info.verRange);
+  bool hasVal = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, &pReader->info.verRange);
-  doUnpinSttBlock(pLastBlockReader);
+  doUnpinSttBlock(pSttBlockReader);
   if (hasVal) {
-    int64_t next1 = getCurrentKeyInLastBlock(pLastBlockReader);
+    int64_t next1 = getCurrentKeyInSttBlock(pSttBlockReader);
     if (next1 != ts) {
       code = doAppendRowFromFileBlock(pReader->resBlockInfo.pResBlock, pReader, fRow->pBlockData, fRow->iRow);
       if (code) {
@@ -1512,15 +1512,15 @@ static FORCE_INLINE STSchema* doGetSchemaForTSRow(int32_t sversion, STsdbReader*
 }
 
 static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo* pBlockScanInfo, TSDBROW* pRow,
-                                     SIterInfo* pIter, int64_t key, SLastBlockReader* pLastBlockReader) {
+                                     SIterInfo* pIter, int64_t key, SSttBlockReader* pSttBlockReader) {
   SRowMerger*         pMerger = &pReader->status.merger;
   SRow*               pTSRow = NULL;
   SBlockData*         pBlockData = &pReader->status.fileBlockData;
   SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
 
   int64_t tsLast = INT64_MIN;
-  if (hasDataInLastBlock(pLastBlockReader)) {
+  if (hasDataInSttBlock(pSttBlockReader)) {
-    tsLast = getCurrentKeyInLastBlock(pLastBlockReader);
+    tsLast = getCurrentKeyInSttBlock(pSttBlockReader);
   }
 
   TSDBKEY k = TSDBROW_KEY(pRow);
@@ -1538,7 +1538,7 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
   int64_t minKey = 0;
   if (pReader->info.order == TSDB_ORDER_ASC) {
     minKey = INT64_MAX;  // chosen the minimum value
-    if (minKey > tsLast && hasDataInLastBlock(pLastBlockReader)) {
+    if (minKey > tsLast && hasDataInSttBlock(pSttBlockReader)) {
       minKey = tsLast;
     }
 
@@ -1551,7 +1551,7 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
     }
   } else {
     minKey = INT64_MIN;
-    if (minKey < tsLast && hasDataInLastBlock(pLastBlockReader)) {
+    if (minKey < tsLast && hasDataInSttBlock(pSttBlockReader)) {
       minKey = tsLast;
     }
 
@@ -1576,12 +1576,12 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
     }
 
     if (minKey == tsLast) {
-      TSDBROW* fRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+      TSDBROW* fRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
       int32_t  code = tsdbRowMergerAdd(pMerger, fRow1, NULL);
       if (code != TSDB_CODE_SUCCESS) {
         return code;
       }
-      doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
+      doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
                              pReader->idStr);
     }
 
@@ -1626,12 +1626,12 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
     }
 
     if (minKey == tsLast) {
-      TSDBROW* fRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+      TSDBROW* fRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
       int32_t  code = tsdbRowMergerAdd(pMerger, fRow1, NULL);
       if (code != TSDB_CODE_SUCCESS) {
         return code;
       }
-      doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
+      doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
                              pReader->idStr);
     }
 
@@ -1657,27 +1657,27 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
   return code;
 }
 
-static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader, STsdbReader* pReader,
+static int32_t doMergeFileBlockAndLastBlock(SSttBlockReader* pSttBlockReader, STsdbReader* pReader,
                                             STableBlockScanInfo* pBlockScanInfo, SBlockData* pBlockData,
                                             bool mergeBlockData) {
   SRowMerger*         pMerger = &pReader->status.merger;
   SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
 
-  int64_t  tsLastBlock = getCurrentKeyInLastBlock(pLastBlockReader);
+  int64_t  tsLastBlock = getCurrentKeyInSttBlock(pSttBlockReader);
   bool     copied = false;
   int32_t  code = TSDB_CODE_SUCCESS;
   SRow*    pTSRow = NULL;
-  TSDBROW* pRow = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+  TSDBROW* pRow = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
 
   // create local variable to hold the row value
   TSDBROW fRow = {.iRow = pRow->iRow, .type = TSDBROW_COL_FMT, .pBlockData = pRow->pBlockData};
 
-  tsdbTrace("fRow ptr:%p, %d, uid:%" PRIu64 ", ts:%" PRId64 " %s", pRow->pBlockData, pRow->iRow, pLastBlockReader->uid,
+  tsdbTrace("fRow ptr:%p, %d, uid:%" PRIu64 ", ts:%" PRId64 " %s", pRow->pBlockData, pRow->iRow, pSttBlockReader->uid,
             fRow.pBlockData->aTSKEY[fRow.iRow], pReader->idStr);
 
   // only stt block exists
   if ((!mergeBlockData) || (tsLastBlock != pBlockData->aTSKEY[pDumpInfo->rowIndex])) {
-    code = tryCopyDistinctRowFromSttBlock(&fRow, pLastBlockReader, pBlockScanInfo, tsLastBlock, pReader, &copied);
+    code = tryCopyDistinctRowFromSttBlock(&fRow, pSttBlockReader, pBlockScanInfo, tsLastBlock, pReader, &copied);
     if (code) {
       return code;
     }
@@ -1691,9 +1691,9 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
         return code;
       }
 
-      TSDBROW* pRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+      TSDBROW* pRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
       tsdbRowMergerAdd(pMerger, pRow1, NULL);
-      doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLastBlock, pMerger, &pReader->info.verRange,
+      doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLastBlock, pMerger, &pReader->info.verRange,
                              pReader->idStr);
 
       code = tsdbRowMergerGetRow(pMerger, &pTSRow);
@@ -1716,7 +1716,7 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
       return code;
     }
 
-    doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLastBlock, pMerger, &pReader->info.verRange,
+    doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLastBlock, pMerger, &pReader->info.verRange,
                            pReader->idStr);
 
     // merge with block data if ts == key
@@ -1742,8 +1742,8 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
   return TSDB_CODE_SUCCESS;
 }
 
-static int32_t mergeFileBlockAndLastBlock(STsdbReader* pReader, SLastBlockReader* pLastBlockReader, int64_t key,
+static int32_t mergeFileBlockAndSttBlock(STsdbReader* pReader, SSttBlockReader* pSttBlockReader, int64_t key,
-                                          STableBlockScanInfo* pBlockScanInfo, SBlockData* pBlockData) {
+                                         STableBlockScanInfo* pBlockScanInfo, SBlockData* pBlockData) {
   SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
   SRowMerger*         pMerger = &pReader->status.merger;
 
@@ -1758,24 +1758,24 @@ static int32_t mergeFileBlockAndLastBlock(STsdbReader* pReader, SLastBlockReader
 
   if (hasDataInFileBlock(pBlockData, pDumpInfo)) {
     // no last block available, only data block exists
-    if (!hasDataInLastBlock(pLastBlockReader)) {
+    if (!hasDataInSttBlock(pSttBlockReader)) {
       return mergeRowsInFileBlocks(pBlockData, pBlockScanInfo, key, pReader);
     }
 
     // row in last file block
     TSDBROW fRow = tsdbRowFromBlockData(pBlockData, pDumpInfo->rowIndex);
-    int64_t tsLast = getCurrentKeyInLastBlock(pLastBlockReader);
+    int64_t tsLast = getCurrentKeyInSttBlock(pSttBlockReader);
     if (ASCENDING_TRAVERSE(pReader->info.order)) {
       if (key < tsLast) {
         return mergeRowsInFileBlocks(pBlockData, pBlockScanInfo, key, pReader);
       } else if (key > tsLast) {
-        return doMergeFileBlockAndLastBlock(pLastBlockReader, pReader, pBlockScanInfo, NULL, false);
+        return doMergeFileBlockAndLastBlock(pSttBlockReader, pReader, pBlockScanInfo, NULL, false);
       }
     } else {
       if (key > tsLast) {
         return mergeRowsInFileBlocks(pBlockData, pBlockScanInfo, key, pReader);
       } else if (key < tsLast) {
-        return doMergeFileBlockAndLastBlock(pLastBlockReader, pReader, pBlockScanInfo, NULL, false);
+        return doMergeFileBlockAndLastBlock(pSttBlockReader, pReader, pBlockScanInfo, NULL, false);
       }
     }
     // the following for key == tsLast
@@ -1787,13 +1787,13 @@ static int32_t mergeFileBlockAndLastBlock(STsdbReader* pReader, SLastBlockReader
 
     doMergeRowsInFileBlocks(pBlockData, pBlockScanInfo, pReader);
 
-    TSDBROW* pRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+    TSDBROW* pRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
     code = tsdbRowMergerAdd(pMerger, pRow1, NULL);
     if (code != TSDB_CODE_SUCCESS) {
       return code;
     }
 
-    doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange, pReader->idStr);
+    doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange, pReader->idStr);
 
     code = tsdbRowMergerGetRow(pMerger, &pTSRow);
     if (code != TSDB_CODE_SUCCESS) {
@@ -1807,12 +1807,12 @@ static int32_t mergeFileBlockAndLastBlock(STsdbReader* pReader, SLastBlockReader
     return code;
 
   } else {  // only last block exists
-    return doMergeFileBlockAndLastBlock(pLastBlockReader, pReader, pBlockScanInfo, NULL, false);
+    return doMergeFileBlockAndLastBlock(pSttBlockReader, pReader, pBlockScanInfo, NULL, false);
   }
 }
 
 static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo* pBlockScanInfo, SBlockData* pBlockData,
-                                     SLastBlockReader* pLastBlockReader) {
+                                     SSttBlockReader* pSttBlockReader) {
   SRowMerger*         pMerger = &pReader->status.merger;
   SRow*               pTSRow = NULL;
   int32_t             code = TSDB_CODE_SUCCESS;
@@ -1823,8 +1823,8 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
   TSDBROW* piRow = getValidMemRow(&pBlockScanInfo->iiter, pDelList, pReader);
 
   int64_t tsLast = INT64_MIN;
-  if (hasDataInLastBlock(pLastBlockReader)) {
+  if (hasDataInSttBlock(pSttBlockReader)) {
-    tsLast = getCurrentKeyInLastBlock(pLastBlockReader);
+    tsLast = getCurrentKeyInSttBlock(pSttBlockReader);
   }
 
   int64_t key = hasDataInFileBlock(pBlockData, pDumpInfo) ? pBlockData->aTSKEY[pDumpInfo->rowIndex] : INT64_MIN;
@@ -1872,7 +1872,7 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
       minKey = key;
     }
 
-    if (minKey > tsLast && hasDataInLastBlock(pLastBlockReader)) {
+    if (minKey > tsLast && hasDataInSttBlock(pSttBlockReader)) {
       minKey = tsLast;
     }
   } else {
@@ -1889,7 +1889,7 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
       minKey = key;
     }
 
-    if (minKey < tsLast && hasDataInLastBlock(pLastBlockReader)) {
+    if (minKey < tsLast && hasDataInSttBlock(pSttBlockReader)) {
       minKey = tsLast;
     }
   }
@@ -1908,13 +1908,13 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
     }
 
     if (minKey == tsLast) {
-      TSDBROW* pRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+      TSDBROW* pRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
       code = tsdbRowMergerAdd(pMerger, pRow1, NULL);
       if (code != TSDB_CODE_SUCCESS) {
         return code;
       }
 
-      doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
+      doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
                              pReader->idStr);
     }
 
@@ -1967,13 +1967,13 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
     }
 
     if (minKey == tsLast) {
-      TSDBROW* pRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+      TSDBROW* pRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
       code = tsdbRowMergerAdd(pMerger, pRow1, NULL);
       if (code != TSDB_CODE_SUCCESS) {
         return code;
       }
 
-      doMergeRowsInLastBlock(pLastBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
+      doMergeRowsInSttBlock(pSttBlockReader, pBlockScanInfo, tsLast, pMerger, &pReader->info.verRange,
                              pReader->idStr);
     }
 
@@ -2092,10 +2092,10 @@ static bool isValidFileBlockRow(SBlockData* pBlockData, SFileBlockDumpInfo* pDum
   return true;
 }
 
-static bool initLastBlockReader(SLastBlockReader* pLBlockReader, STableBlockScanInfo* pScanInfo, STsdbReader* pReader) {
+static bool initSttBlockReader(SSttBlockReader* pLBlockReader, STableBlockScanInfo* pScanInfo, STsdbReader* pReader) {
   // the last block reader has been initialized for this table.
   if (pLBlockReader->uid == pScanInfo->uid) {
-    return hasDataInLastBlock(pLBlockReader);
+    return hasDataInSttBlock(pLBlockReader);
   }
 
   if (pLBlockReader->uid != 0) {
@@ -2144,13 +2144,13 @@ static bool initLastBlockReader(SLastBlockReader* pLBlockReader, STableBlockScan
   code = nextRowFromSttBlocks(pLBlockReader, pScanInfo, &pReader->info.verRange);
 
   int64_t el = taosGetTimestampUs() - st;
-  pReader->cost.initLastBlockReader += (el / 1000.0);
+  pReader->cost.initSttBlockReader += (el / 1000.0);
 
   tsdbDebug("init last block reader completed, elapsed time:%" PRId64 "us %s", el, pReader->idStr);
   return code;
 }
 
-static bool hasDataInLastBlock(SLastBlockReader* pLastBlockReader) { return pLastBlockReader->mergeTree.pIter != NULL; }
+static bool hasDataInSttBlock(SSttBlockReader* pSttBlockReader) { return pSttBlockReader->mergeTree.pIter != NULL; }
 
 bool hasDataInFileBlock(const SBlockData* pBlockData, const SFileBlockDumpInfo* pDumpInfo) {
   if ((pBlockData->nRow > 0) && (pBlockData->nRow != pDumpInfo->totalRows)) {
@@ -2206,7 +2206,7 @@ int32_t mergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pBloc
 }
 
 static int32_t buildComposedDataBlockImpl(STsdbReader* pReader, STableBlockScanInfo* pBlockScanInfo,
-                                          SBlockData* pBlockData, SLastBlockReader* pLastBlockReader) {
+                                          SBlockData* pBlockData, SSttBlockReader* pSttBlockReader) {
   SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
 
   TSDBROW *pRow = NULL, *piRow = NULL;
@@ -2223,21 +2223,21 @@ static int32_t buildComposedDataBlockImpl(STsdbReader* pReader, STableBlockScanI
 
   // two levels of mem-table does contain the valid rows
   if (pRow != NULL && piRow != NULL) {
-    return doMergeMultiLevelRows(pReader, pBlockScanInfo, pBlockData, pLastBlockReader);
+    return doMergeMultiLevelRows(pReader, pBlockScanInfo, pBlockData, pSttBlockReader);
   }
 
   // imem + file + last block
   if (pBlockScanInfo->iiter.hasVal) {
-    return doMergeBufAndFileRows(pReader, pBlockScanInfo, piRow, &pBlockScanInfo->iiter, key, pLastBlockReader);
+    return doMergeBufAndFileRows(pReader, pBlockScanInfo, piRow, &pBlockScanInfo->iiter, key, pSttBlockReader);
   }
 
   // mem + file + last block
   if (pBlockScanInfo->iter.hasVal) {
-    return doMergeBufAndFileRows(pReader, pBlockScanInfo, pRow, &pBlockScanInfo->iter, key, pLastBlockReader);
+    return doMergeBufAndFileRows(pReader, pBlockScanInfo, pRow, &pBlockScanInfo->iter, key, pSttBlockReader);
   }
 
   // files data blocks + last block
-  return mergeFileBlockAndLastBlock(pReader, pLastBlockReader, key, pBlockScanInfo, pBlockData);
+  return mergeFileBlockAndSttBlock(pReader, pSttBlockReader, key, pBlockScanInfo, pBlockData);
 }
 
 static int32_t loadNeighborIfOverlap(SFileDataBlockInfo* pBlockInfo, STableBlockScanInfo* pBlockScanInfo,
@@ -2298,7 +2298,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
   SSDataBlock* pResBlock = pReader->resBlockInfo.pResBlock;
 
   SFileDataBlockInfo* pBlockInfo = getCurrentBlockInfo(&pReader->status.blockIter);
-  SLastBlockReader*   pLastBlockReader = pReader->status.fileIter.pLastBlockReader;
+  SSttBlockReader*   pSttBlockReader = pReader->status.fileIter.pSttBlockReader;
 
   bool         asc = ASCENDING_TRAVERSE(pReader->info.order);
   int64_t      st = taosGetTimestampUs();
@@ -2347,7 +2347,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
   }
 
   SBlockData* pBlockData = &pReader->status.fileBlockData;
-  initLastBlockReader(pLastBlockReader, pBlockScanInfo, pReader);
+  initSttBlockReader(pSttBlockReader, pBlockScanInfo, pReader);
 
   while (1) {
     bool hasBlockData = false;
@@ -2381,7 +2381,7 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
       break;
     }
 
-    code = buildComposedDataBlockImpl(pReader, pBlockScanInfo, pBlockData, pLastBlockReader);
+    code = buildComposedDataBlockImpl(pReader, pBlockScanInfo, pBlockData, pSttBlockReader);
     if (code) {
       goto _end;
     }
@@ -2604,9 +2604,9 @@ static bool moveToNextTable(STableUidList* pOrderedCheckInfo, SReaderStatus* pSt
   return (pStatus->pTableIter != NULL);
 }
 
-static int32_t doLoadLastBlockSequentially(STsdbReader* pReader) {
+static int32_t doLoadSttBlockSequentially(STsdbReader* pReader) {
   SReaderStatus*    pStatus = &pReader->status;
-  SLastBlockReader* pLastBlockReader = pStatus->fileIter.pLastBlockReader;
+  SSttBlockReader* pSttBlockReader = pStatus->fileIter.pSttBlockReader;
   STableUidList*    pUidList = &pStatus->uidList;
   int32_t           code = TSDB_CODE_SUCCESS;
 
@@ -2644,7 +2644,7 @@ static int32_t doLoadLastBlockSequentially(STsdbReader* pReader) {
       continue;
     }
 
-    bool hasDataInLastFile = initLastBlockReader(pLastBlockReader, pScanInfo, pReader);
+    bool hasDataInLastFile = initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
     if (!hasDataInLastFile) {
       bool hasNexTable = moveToNextTable(pUidList, pStatus);
       if (!hasNexTable) {
@@ -2656,14 +2656,14 @@ static int32_t doLoadLastBlockSequentially(STsdbReader* pReader) {
 
     int64_t st = taosGetTimestampUs();
     while (1) {
-      bool hasBlockLData = hasDataInLastBlock(pLastBlockReader);
+      bool hasBlockLData = hasDataInSttBlock(pSttBlockReader);
 
       // no data in last block and block, no need to proceed.
       if (hasBlockLData == false) {
         break;
       }
 
-      code = buildComposedDataBlockImpl(pReader, pScanInfo, &pReader->status.fileBlockData, pLastBlockReader);
+      code = buildComposedDataBlockImpl(pReader, pScanInfo, &pReader->status.fileBlockData, pSttBlockReader);
       if (code) {
         return code;
       }
@@ -2710,7 +2710,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
   SDataBlockIter*      pBlockIter = &pStatus->blockIter;
   STableBlockScanInfo* pScanInfo = NULL;
   SFileDataBlockInfo*  pBlockInfo = getCurrentBlockInfo(pBlockIter);
-  SLastBlockReader*    pLastBlockReader = pReader->status.fileIter.pLastBlockReader;
+  SSttBlockReader*    pSttBlockReader = pReader->status.fileIter.pSttBlockReader;
   bool                 asc = ASCENDING_TRAVERSE(pReader->info.order);
 
   if (pReader->pIgnoreTables && taosHashGet(*pReader->pIgnoreTables, &pBlockInfo->uid, sizeof(pBlockInfo->uid))) {
@@ -2728,7 +2728,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
   }
 
   if (pScanInfo->sttKeyInfo.status == STT_FILE_READER_UNINIT) {
-    initLastBlockReader(pLastBlockReader, pScanInfo, pReader);
+    initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
   }
 
   TSDBKEY keyInBuf = getCurrentKeyInBuf(pScanInfo, pReader);
@@ -2773,13 +2773,13 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
       int64_t st = taosGetTimestampUs();
 
       // let's load data from stt files
-      initLastBlockReader(pLastBlockReader, pScanInfo, pReader);
+      initSttBlockReader(pSttBlockReader, pScanInfo, pReader);
 
       // no data in last block, no need to proceed.
-      while (hasDataInLastBlock(pLastBlockReader)) {
+      while (hasDataInSttBlock(pSttBlockReader)) {
         ASSERT(pScanInfo->sttKeyInfo.status == STT_FILE_HAS_DATA);
 
-        code = buildComposedDataBlockImpl(pReader, pScanInfo, &pReader->status.fileBlockData, pLastBlockReader);
+        code = buildComposedDataBlockImpl(pReader, pScanInfo, &pReader->status.fileBlockData, pSttBlockReader);
         if (code != TSDB_CODE_SUCCESS) {
           return code;
         }
@@ -2789,7 +2789,7 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) {
         }
 
         // data in stt now overlaps with current active file data block, need to composed with file data block.
-        int64_t lastKeyInStt = getCurrentKeyInLastBlock(pLastBlockReader);
+        int64_t lastKeyInStt = getCurrentKeyInSttBlock(pSttBlockReader);
         if ((lastKeyInStt >= pBlockInfo->record.firstKey && asc) ||
             (lastKeyInStt <= pBlockInfo->record.lastKey && (!asc))) {
           tsdbDebug("%p lastKeyInStt:%" PRId64 ", overlap with file block, brange:%" PRId64 "-%" PRId64 " %s", pReader,
@@ -2859,11 +2859,11 @@ _end:
 
 static int32_t doSumSttBlockRows(STsdbReader* pReader) {
   int32_t            code = TSDB_CODE_SUCCESS;
-  SLastBlockReader*  pLastBlockReader = pReader->status.fileIter.pLastBlockReader;
+  SSttBlockReader*  pSttBlockReader = pReader->status.fileIter.pSttBlockReader;
   SSttBlockLoadInfo* pBlockLoadInfo = NULL;
 #if 0
   for (int32_t i = 0; i < pReader->pFileReader->pSet->nSttF; ++i) {  // open all last file
-    pBlockLoadInfo = &pLastBlockReader->pInfo[i];
+    pBlockLoadInfo = &pSttBlockReader->pInfo[i];
 
     code = tsdbReadSttBlk(pReader->pFileReader, i, pBlockLoadInfo->aSttBlk);
     if (code) {
@@ -3062,7 +3062,7 @@ typedef enum {
   TSDB_READ_CONTINUE = 0x2,
 } ERetrieveType;
 
-static ERetrieveType doReadDataFromLastFiles(STsdbReader* pReader) {
+static ERetrieveType doReadDataFromSttFiles(STsdbReader* pReader) {
   int32_t         code = TSDB_CODE_SUCCESS;
   SSDataBlock*    pResBlock = pReader->resBlockInfo.pResBlock;
   SDataBlockIter* pBlockIter = &pReader->status.blockIter;
@@ -3072,7 +3072,7 @@ static ERetrieveType doReadDataFromLastFiles(STsdbReader* pReader) {
   while (1) {
     terrno = 0;
 
-    code = doLoadLastBlockSequentially(pReader);
+    code = doLoadSttBlockSequentially(pReader);
     if (code != TSDB_CODE_SUCCESS) {
       terrno = code;
       return TSDB_READ_RETURN;
@@ -3109,7 +3109,7 @@ static int32_t buildBlockFromFiles(STsdbReader* pReader) {
 
   if (pBlockIter->numOfBlocks == 0) {
     // let's try to extract data from stt files.
-    ERetrieveType type = doReadDataFromLastFiles(pReader);
+    ERetrieveType type = doReadDataFromSttFiles(pReader);
     if (type == TSDB_READ_RETURN) {
       return terrno;
     }
@@ -3144,7 +3144,7 @@ static int32_t buildBlockFromFiles(STsdbReader* pReader) {
           resetDataBlockIterator(pBlockIter, pReader->info.order);
           resetTableListIndex(&pReader->status);
 
-          ERetrieveType type = doReadDataFromLastFiles(pReader);
+          ERetrieveType type = doReadDataFromSttFiles(pReader);
           if (type == TSDB_READ_RETURN) {
             return terrno;
           }
@@ -3476,12 +3476,12 @@ int32_t doMergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pSc
   return TSDB_CODE_SUCCESS;
 }
 
-int32_t doMergeRowsInLastBlock(SLastBlockReader* pLastBlockReader, STableBlockScanInfo* pScanInfo, int64_t ts,
+int32_t doMergeRowsInSttBlock(SSttBlockReader* pSttBlockReader, STableBlockScanInfo* pScanInfo, int64_t ts,
-                               SRowMerger* pMerger, SVersionRange* pVerRange, const char* idStr) {
+                              SRowMerger* pMerger, SVersionRange* pVerRange, const char* idStr) {
-  while (nextRowFromSttBlocks(pLastBlockReader, pScanInfo, pVerRange)) {
+  while (nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pVerRange)) {
-    int64_t next1 = getCurrentKeyInLastBlock(pLastBlockReader);
+    int64_t next1 = getCurrentKeyInSttBlock(pSttBlockReader);
     if (next1 == ts) {
-      TSDBROW* pRow1 = tMergeTreeGetRow(&pLastBlockReader->mergeTree);
+      TSDBROW* pRow1 = tMergeTreeGetRow(&pSttBlockReader->mergeTree);
       tsdbRowMergerAdd(pMerger, pRow1, NULL);
     } else {
       tsdbTrace("uid:%" PRIu64 " last del index:%d, del range:%d, lastKeyInStt:%" PRId64 ", %s", pScanInfo->uid,
@@ -4144,8 +4144,8 @@ void tsdbReaderClose2(STsdbReader* pReader) {
 
   SReadCostSummary* pCost = &pReader->cost;
   SFilesetIter* pFilesetIter = &pReader->status.fileIter;
-  if (pFilesetIter->pLastBlockReader != NULL) {
+  if (pFilesetIter->pSttBlockReader != NULL) {
-    SLastBlockReader* pLReader = pFilesetIter->pLastBlockReader;
+    SSttBlockReader* pLReader = pFilesetIter->pSttBlockReader;
     tMergeTreeClose(&pLReader->mergeTree);
     taosMemoryFree(pLReader);
   }
@@ -4168,12 +4168,12 @@ void tsdbReaderClose2(STsdbReader* pReader) {
       "build in-memory-block-time:%.2f ms, sttBlocks:%" PRId64 ", sttBlocks-time:%.2f ms, sttStatisBlock:%" PRId64
       ", stt-statis-Block-time:%.2f ms, composed-blocks:%" PRId64
       ", composed-blocks-time:%.2fms, STableBlockScanInfo size:%.2f Kb, createTime:%.2f ms,createSkylineIterTime:%.2f "
-      "ms, initLastBlockReader:%.2fms, %s",
+      "ms, initSttBlockReader:%.2fms, %s",
       pReader, pCost->headFileLoad, pCost->headFileLoadTime, pCost->smaDataLoad, pCost->smaLoadTime, pCost->numOfBlocks,
       pCost->blockLoadTime, pCost->buildmemBlock, pCost->sttCost.loadBlocks, pCost->sttCost.blockElapsedTime,
       pCost->sttCost.loadStatisBlocks, pCost->sttCost.statisElapsedTime, pCost->composedBlocks,
       pCost->buildComposedBlockTime, numOfTables * sizeof(STableBlockScanInfo) / 1000.0, pCost->createScanInfoList,
-      pCost->createSkylineIterTime, pCost->initLastBlockReader, pReader->idStr);
+      pCost->createSkylineIterTime, pCost->initSttBlockReader, pReader->idStr);
 
   taosMemoryFree(pReader->idStr);
 
@@ -4838,7 +4838,20 @@ int32_t tsdbGetFileBlocksDistInfo2(STsdbReader* pReader, STableBlockDistInfo* pT
       return code;
     }
   }
+
+  SMergeTreeConf conf = {
+      .pReader = pReader,
+      .pSchema = pReader->info.pSchema,
+      .pCols = pReader->suppInfo.colId,
+      .numOfCols = pReader->suppInfo.numOfCols,
+      .suid = pReader->info.suid,
+  };
+
   SReaderStatus* pStatus = &pReader->status;
+  if (pStatus->pCurrentFileset != NULL) {
+    pTableBlockInfo->numOfSttRows += tsdbGetRowsInSttFiles(pStatus->pCurrentFileset, pStatus->pLDataIterArray,
+                                                           pReader->pTsdb, &conf, pReader->idStr);
+  }
 
   STsdbCfg* pc = &pReader->pTsdb->pVnode->config.tsdbCfg;
   pTableBlockInfo->defMinRows = pc->minRows;
@@ -4875,10 +4888,6 @@ int32_t tsdbGetFileBlocksDistInfo2(STsdbReader* pReader, STableBlockDistInfo* pT
         pTableBlockInfo->minRows = numOfRows;
       }
 
-      if (numOfRows < defaultRows) {
-        pTableBlockInfo->numOfSmallBlocks += 1;
-      }
-
       pTableBlockInfo->totalSize += pBlockInfo->record.blockSize;
 
       int32_t bucketIndex = getBucketIndex(pTableBlockInfo->defMinRows, bucketRange, numOfRows, numOfBuckets);
@@ -4891,13 +4900,18 @@ int32_t tsdbGetFileBlocksDistInfo2(STsdbReader* pReader, STableBlockDistInfo* pT
         break;
       }
 
+      // add the data in stt files of new fileset
+      if (pStatus->pCurrentFileset != NULL) {
+        pTableBlockInfo->numOfSttRows += tsdbGetRowsInSttFiles(pStatus->pCurrentFileset, pStatus->pLDataIterArray,
+                                                               pReader->pTsdb, &conf, pReader->idStr);
+      }
+
       pTableBlockInfo->numOfBlocks += pBlockIter->numOfBlocks;
       hasNext = (pBlockIter->numOfBlocks > 0);
     }
-
-    //    tsdbDebug("%p %d blocks found in file for %d table(s), fid:%d, %s", pReader, numOfBlocks, numOfTables,
-    //              pReader->pFileGroup->fid, pReader->idStr);
   }
+
+  // record the data in stt files
   tsdbReleaseReader(pReader);
   return code;
 }
@@ -5132,7 +5146,7 @@ void tsdbUntakeReadSnap2(STsdbReader* pReader, STsdbReadSnap* pSnap, bool proact
 void tsdbReaderSetId2(STsdbReader* pReader, const char* idstr) {
   taosMemoryFreeClear(pReader->idStr);
   pReader->idStr = taosStrdup(idstr);
-  pReader->status.fileIter.pLastBlockReader->mergeTree.idStr = pReader->idStr;
+  pReader->status.fileIter.pSttBlockReader->mergeTree.idStr = pReader->idStr;
 }
 
 void tsdbReaderSetCloseFlag(STsdbReader* pReader) { /*pReader->code = TSDB_CODE_TSC_QUERY_CANCELLED;*/

source/dnode/vnode/src/tsdb/tsdbReadUtil.c

@@ -488,6 +488,8 @@ typedef enum {
   BLK_CHECK_QUIT = 0x2,
 } ETombBlkCheckEnum;
 
+static void    loadNextStatisticsBlock(SSttFileReader* pSttFileReader, const SSttBlockLoadInfo* pBlockLoadInfo,
+                                       const TStatisBlkArray* pStatisBlkArray, int32_t numOfRows, int32_t* i, int32_t* j);
 static int32_t doCheckTombBlock(STombBlock* pBlock, STsdbReader* pReader, int32_t numOfTables, int32_t* j,
                                 ETombBlkCheckEnum* pRet) {
   int32_t     code = 0;
@@ -659,3 +661,177 @@ void loadMemTombData(SArray** ppMemDelData, STbData* pMemTbData, STbData* piMemT
     }
   }
 }
+
+int32_t getNumOfRowsInSttBlock(SSttFileReader *pSttFileReader, SSttBlockLoadInfo *pBlockLoadInfo, uint64_t suid,
+                               const uint64_t* pUidList, int32_t numOfTables) {
+  int32_t num = 0;
+
+  const TStatisBlkArray *pStatisBlkArray = pBlockLoadInfo->pSttStatisBlkArray;
+  if (TARRAY2_SIZE(pStatisBlkArray) <= 0) {
+    return 0;
+  }
+
+  int32_t i = 0;
+  while((i < TARRAY2_SIZE(pStatisBlkArray)) && (pStatisBlkArray->data[i].minTbid.suid < suid)) {
+    ++i;
+  }
+
+  if (i >= TARRAY2_SIZE(pStatisBlkArray)) {
+    return 0;
+  }
+
+  SStatisBlk *p = &pStatisBlkArray->data[i];
+  if (pBlockLoadInfo->statisBlock == NULL) {
+    pBlockLoadInfo->statisBlock = taosMemoryCalloc(1, sizeof(STbStatisBlock));
+    tStatisBlockInit(pBlockLoadInfo->statisBlock);
+  }
+
+  int64_t st = taosGetTimestampMs();
+  tsdbSttFileReadStatisBlock(pSttFileReader, p, pBlockLoadInfo->statisBlock);
+  pBlockLoadInfo->statisBlockIndex = i;
+
+  double el = (taosGetTimestampMs() - st) / 1000.0;
+  pBlockLoadInfo->cost.loadStatisBlocks += 1;
+  pBlockLoadInfo->cost.statisElapsedTime += el;
+
+  STbStatisBlock *pBlock = pBlockLoadInfo->statisBlock;
+
+  int32_t index = 0;
+  while (index < TARRAY2_SIZE(pBlock->suid) && pBlock->suid->data[index] < suid) {
+    ++index;
+  }
+
+  if (index >= TARRAY2_SIZE(pBlock->suid)) {
+    return num;
+  }
+
+  int32_t j = index;
+  int32_t uidIndex = 0;
+  while (i < TARRAY2_SIZE(pStatisBlkArray) && uidIndex <= numOfTables) {
+    p = &pStatisBlkArray->data[i];
+    if (p->minTbid.suid > suid) {
+      return num;
+    }
+
+    uint64_t uid = pUidList[uidIndex];
+
+    if (pBlock->uid->data[j] == uid) {
+      num += pBlock->count->data[j];
+      uidIndex += 1;
+      j += 1;
+      loadNextStatisticsBlock(pSttFileReader, pBlockLoadInfo, pStatisBlkArray, pBlock->suid->size, &i, &j);
+    } else if (pBlock->uid->data[j] < uid) {
+      j += 1;
+      loadNextStatisticsBlock(pSttFileReader, pBlockLoadInfo, pStatisBlkArray, pBlock->suid->size, &i, &j);
+    } else {
+      uidIndex += 1;
+    }
+  }
+
+  return num;
+}
+
+// load next stt statistics block
+static void loadNextStatisticsBlock(SSttFileReader* pSttFileReader, const SSttBlockLoadInfo* pBlockLoadInfo,
+                                    const TStatisBlkArray* pStatisBlkArray, int32_t numOfRows, int32_t* i, int32_t* j) {
+  if ((*j) >= numOfRows) {
+    (*i) += 1;
+    (*j) = 0;
+    if ((*i) < TARRAY2_SIZE(pStatisBlkArray)) {
+      tsdbSttFileReadStatisBlock(pSttFileReader, &pStatisBlkArray->data[(*i)], pBlockLoadInfo->statisBlock);
+    }
+  }
+}
+
+void doAdjustValidDataIters(SArray* pLDIterList, int32_t numOfFileObj) {
+  int32_t size = taosArrayGetSize(pLDIterList);
+
+  if (size < numOfFileObj) {
+    int32_t inc = numOfFileObj - size;
+    for (int32_t k = 0; k < inc; ++k) {
+      SLDataIter *pIter = taosMemoryCalloc(1, sizeof(SLDataIter));
+      taosArrayPush(pLDIterList, &pIter);
+    }
+  } else if (size > numOfFileObj) {  // remove unused LDataIter
+    int32_t inc = size - numOfFileObj;
+
+    for (int i = 0; i < inc; ++i) {
+      SLDataIter *pIter = taosArrayPop(pLDIterList);
+      destroyLDataIter(pIter);
+    }
+  }
+}
+
+int32_t adjustLDataIters(SArray* pSttFileBlockIterArray, STFileSet* pFileSet) {
+  int32_t numOfLevels = pFileSet->lvlArr->size;
+
+  // add the list/iter placeholder
+  while (taosArrayGetSize(pSttFileBlockIterArray) < numOfLevels) {
+    SArray* pList = taosArrayInit(4, POINTER_BYTES);
+    taosArrayPush(pSttFileBlockIterArray, &pList);
+  }
+
+  for(int32_t j = 0; j < numOfLevels; ++j) {
+    SSttLvl* pSttLevel = pFileSet->lvlArr->data[j];
+    SArray* pList = taosArrayGetP(pSttFileBlockIterArray, j);
+    doAdjustValidDataIters(pList, TARRAY2_SIZE(pSttLevel->fobjArr));
+  }
+
+  return TSDB_CODE_SUCCESS;
+}
+
+int32_t tsdbGetRowsInSttFiles(STFileSet* pFileSet, SArray* pSttFileBlockIterArray, STsdb* pTsdb, SMergeTreeConf* pConf,
+                              const char* pstr) {
+  int32_t numOfRows = 0;
+
+  // no data exists, go to end
+  int32_t numOfLevels = pFileSet->lvlArr->size;
+  if (numOfLevels == 0) {
+    return numOfRows;
+  }
+
+  // add the list/iter placeholder
+  adjustLDataIters(pSttFileBlockIterArray, pFileSet);
+
+  for (int32_t j = 0; j < numOfLevels; ++j) {
+    SSttLvl* pSttLevel = pFileSet->lvlArr->data[j];
+    SArray*  pList = taosArrayGetP(pSttFileBlockIterArray, j);
+
+    for (int32_t i = 0; i < taosArrayGetSize(pList); ++i) {  // open all last file
+      SLDataIter* pIter = taosArrayGetP(pList, i);
+
+      // open stt file reader if not opened yet
+      // if failed to open this stt file, ignore the error and try next one
+      if (pIter->pReader == NULL) {
+        SSttFileReaderConfig conf = {.tsdb = pTsdb, .szPage = pTsdb->pVnode->config.tsdbPageSize};
+        conf.file[0] = *pSttLevel->fobjArr->data[i]->f;
+
+        const char* pName = pSttLevel->fobjArr->data[i]->fname;
+        int32_t     code = tsdbSttFileReaderOpen(pName, &conf, &pIter->pReader);
+        if (code != TSDB_CODE_SUCCESS) {
+          tsdbError("open stt file reader error. file:%s, code %s, %s", pName, tstrerror(code), pstr);
+          continue;
+        }
+      }
+
+      if (pIter->pBlockLoadInfo == NULL) {
+        pIter->pBlockLoadInfo = tCreateSttBlockLoadInfo(pConf->pSchema, pConf->pCols, pConf->numOfCols);
+      }
+
+      // load stt blocks statis for all stt-blocks, to decide if the data of queried table exists in current stt file
+      int32_t code = tsdbSttFileReadStatisBlk(pIter->pReader, (const TStatisBlkArray **)&pIter->pBlockLoadInfo->pSttStatisBlkArray);
+      if (code != TSDB_CODE_SUCCESS) {
+        tsdbError("failed to load stt block statistics, code:%s, %s", tstrerror(code), pstr);
+        continue;
+      }
+
+      // extract rows from each stt file one-by-one
+      STsdbReader* pReader = pConf->pReader;
+      int32_t      numOfTables = tSimpleHashGetSize(pReader->status.pTableMap);
+      uint64_t*    pUidList = pReader->status.uidList.tableUidList;
+      numOfRows += getNumOfRowsInSttBlock(pIter->pReader, pIter->pBlockLoadInfo, pConf->suid, pUidList, numOfTables);
+    }
+  }
+
+  return numOfRows;
+}

source/dnode/vnode/src/tsdb/tsdbReadUtil.h

@@ -110,7 +110,7 @@ typedef struct SReadCostSummary {
   double  buildComposedBlockTime;
   double  createScanInfoList;
   double  createSkylineIterTime;
-  double  initLastBlockReader;
+  double  initSttBlockReader;
 } SReadCostSummary;
 
 typedef struct STableUidList {
@@ -146,21 +146,21 @@ typedef struct SBlockLoadSuppInfo {
   bool                smaValid;  // the sma on all queried columns are activated
 } SBlockLoadSuppInfo;
 
-typedef struct SLastBlockReader {
+typedef struct SSttBlockReader {
   STimeWindow        window;
   SVersionRange      verRange;
   int32_t            order;
   uint64_t           uid;
   SMergeTree         mergeTree;
   int64_t            currentKey;
-} SLastBlockReader;
+} SSttBlockReader;
 
 typedef struct SFilesetIter {
   int32_t           numOfFiles;    // number of total files
   int32_t           index;         // current accessed index in the list
   TFileSetArray*    pFilesetList;  // data file set list
   int32_t           order;
-  SLastBlockReader* pLastBlockReader;  // last file block reader
+  SSttBlockReader*  pSttBlockReader;  // last file block reader
 } SFilesetIter;
 
 typedef struct SFileDataBlockInfo {
@@ -272,7 +272,12 @@ bool    blockIteratorNext(SDataBlockIter* pBlockIter, const char* idStr);
 void    loadMemTombData(SArray** ppMemDelData, STbData* pMemTbData, STbData* piMemTbData, int64_t ver);
 int32_t loadDataFileTombDataForAll(STsdbReader* pReader);
 int32_t loadSttTombDataForAll(STsdbReader* pReader, SSttFileReader* pSttFileReader, SSttBlockLoadInfo* pLoadInfo);
-
+int32_t getNumOfRowsInSttBlock(SSttFileReader *pSttFileReader, SSttBlockLoadInfo *pBlockLoadInfo, uint64_t suid,
+                               const uint64_t* pUidList, int32_t numOfTables);
+void    destroyLDataIter(SLDataIter* pIter);
+int32_t adjustLDataIters(SArray* pSttFileBlockIterArray, STFileSet* pFileSet);
+int32_t tsdbGetRowsInSttFiles(STFileSet* pFileSet, SArray* pSttFileBlockIterArray, STsdb* pTsdb, SMergeTreeConf* pConf,
+                              const char* pstr);
 typedef struct {
   SArray* pTombData;
 } STableLoadInfo;

source/dnode/vnode/src/tsdb/tsdbRetention.c

@@ -430,6 +430,11 @@ int32_t tsdbRetention(STsdb *tsdb, int64_t now, int32_t sync) {
 
   taosThreadMutexLock(&tsdb->mutex);
 
+  if (tsdb->bgTaskDisabled) {
+    taosThreadMutexUnlock(&tsdb->mutex);
+    return 0;
+  }
+
   STFileSet *fset;
   TARRAY2_FOREACH(tsdb->pFS->fSetArr, fset) {
     code = tsdbTFileSetOpenChannel(fset);

source/dnode/vnode/src/tsdb/tsdbSnapshot.c

@@ -278,6 +278,15 @@ _exit:
   return code;
 }
 
+static int64_t tBlockDataSize(SBlockData* pBlockData) {
+  int64_t nData = 0;
+  for (int32_t iCol = 0; iCol < pBlockData->nColData; iCol++) {
+    SColData* pColData = tBlockDataGetColDataByIdx(pBlockData, iCol);
+    nData += pColData->nData;
+  }
+  return nData;
+}
+
 static int32_t tsdbSnapReadTimeSeriesData(STsdbSnapReader* reader, uint8_t** data) {
   int32_t   code = 0;
   int32_t   lino = 0;
@@ -320,8 +329,11 @@ static int32_t tsdbSnapReadTimeSeriesData(STsdbSnapReader* reader, uint8_t** dat
     code = tsdbIterMergerNext(reader->dataIterMerger);
     TSDB_CHECK_CODE(code, lino, _exit);
 
-    if (reader->blockData->nRow >= 81920) {
+    if (!(reader->blockData->nRow % 16)) {
-      break;
+      int64_t nData = tBlockDataSize(reader->blockData);
+      if (nData >= 1 * 1024 * 1024) {
+        break;
+      }
     }
   }
 
@@ -1590,7 +1602,4 @@ _out:
   }
 
   return code;
-}
+}
-
-extern int32_t tsdbFSCancelAllBgTask(STFileSystem* fs);
-int32_t        tsdbCancelAllBgTask(STsdb* tsdb) { return tsdbFSCancelAllBgTask(tsdb->pFS); }

source/dnode/vnode/src/vnd/vnodeInitApi.c

@@ -192,6 +192,7 @@ void initStateStoreAPI(SStateStore* pStore) {
   pStore->updateInfoDestroy = updateInfoDestroy;
   pStore->windowSBfDelete = windowSBfDelete;
   pStore->windowSBfAdd = windowSBfAdd;
+  pStore->isIncrementalTimeStamp = isIncrementalTimeStamp;
 
   pStore->updateInfoInitP = updateInfoInitP;
   pStore->updateInfoAddCloseWindowSBF = updateInfoAddCloseWindowSBF;

source/dnode/vnode/src/vnd/vnodeModule.c

@@ -39,12 +39,6 @@ int vnodeInit(int nthreads) {
   if (walInit() < 0) {
     return -1;
   }
-  if (tqInit() < 0) {
-    return -1;
-  }
-  if (s3Init() < 0) {
-    return -1;
-  }
 
   return 0;
 }
@@ -58,7 +52,5 @@ void vnodeCleanup() {
   vnodeAsyncDestroy(&vnodeAsyncHandle[1]);
 
   walCleanUp();
-  tqCleanUp();
   smaCleanUp();
-  s3CleanUp();
 }

source/dnode/vnode/src/vnd/vnodeSnapshot.c

@@ -519,7 +519,21 @@ _out:
   return code;
 }
 
-extern int32_t tsdbCancelAllBgTask(STsdb *tsdb);
+extern int32_t tsdbDisableAndCancelAllBgTask(STsdb *pTsdb);
+extern int32_t tsdbEnableBgTask(STsdb *pTsdb);
+
+static int32_t vnodeCancelAndDisableAllBgTask(SVnode *pVnode) {
+  tsdbDisableAndCancelAllBgTask(pVnode->pTsdb);
+  vnodeSyncCommit(pVnode);
+  vnodeAChannelDestroy(vnodeAsyncHandle[0], pVnode->commitChannel, true);
+  return 0;
+}
+
+static int32_t vnodeEnableBgTask(SVnode *pVnode) {
+  tsdbEnableBgTask(pVnode->pTsdb);
+  vnodeAChannelInit(vnodeAsyncHandle[0], &pVnode->commitChannel);
+  return 0;
+}
 
 int32_t vnodeSnapWriterOpen(SVnode *pVnode, SSnapshotParam *pParam, SVSnapWriter **ppWriter) {
   int32_t       code = 0;
@@ -527,9 +541,8 @@ int32_t vnodeSnapWriterOpen(SVnode *pVnode, SSnapshotParam *pParam, SVSnapWriter
   int64_t       sver = pParam->start;
   int64_t       ever = pParam->end;
 
-  // commit memory data
+  // cancel and disable all bg task
-  vnodeSyncCommit(pVnode);
+  vnodeCancelAndDisableAllBgTask(pVnode);
-  tsdbCancelAllBgTask(pVnode->pTsdb);
 
   // alloc
   pWriter = (SVSnapWriter *)taosMemoryCalloc(1, sizeof(*pWriter));
@@ -659,6 +672,7 @@ _exit:
     vInfo("vgId:%d, vnode snapshot writer closed, rollback:%d", TD_VID(pVnode), rollback);
     taosMemoryFree(pWriter);
   }
+  vnodeEnableBgTask(pVnode);
   return code;
 }
 

source/libs/executor/inc/executorInt.h

@@ -340,18 +340,19 @@ enum {
 };
 
 typedef struct SStreamAggSupporter {
-  int32_t         resultRowSize;  // the result buffer size for each result row, with the meta data size for each row
+  int32_t             resultRowSize;  // the result buffer size for each result row, with the meta data size for each row
-  SSDataBlock*    pScanBlock;
+  SSDataBlock*        pScanBlock;
-  SStreamState*   pState;
+  SStreamState*       pState;
-  int64_t         gap;        // stream session window gap
+  int64_t             gap;        // stream session window gap
-  SqlFunctionCtx* pDummyCtx;  // for combine
+  SqlFunctionCtx*     pDummyCtx;  // for combine
-  SSHashObj*      pResultRows;
+  SSHashObj*          pResultRows;
-  int32_t         stateKeySize;
+  int32_t             stateKeySize;
-  int16_t         stateKeyType;
+  int16_t             stateKeyType;
-  SDiskbasedBuf*  pResultBuf;
+  SDiskbasedBuf*      pResultBuf;
-  SStateStore     stateStore;
+  SStateStore         stateStore;
-  STimeWindow     winRange;
+  STimeWindow         winRange;
-  SStorageAPI*    pSessionAPI;
+  SStorageAPI*        pSessionAPI;
+  struct SUpdateInfo* pUpdateInfo;
 } SStreamAggSupporter;
 
 typedef struct SWindowSupporter {
@@ -504,38 +505,39 @@ typedef struct SOpCheckPointInfo {
 } SOpCheckPointInfo;
 
 typedef struct SStreamIntervalOperatorInfo {
-  SOptrBasicInfo     binfo;           // basic info
+  SOptrBasicInfo      binfo;           // basic info
-  SAggSupporter      aggSup;          // aggregate supporter
+  SAggSupporter       aggSup;          // aggregate supporter
-  SExprSupp          scalarSupp;      // supporter for perform scalar function
+  SExprSupp           scalarSupp;      // supporter for perform scalar function
-  SGroupResInfo      groupResInfo;    // multiple results build supporter
+  SGroupResInfo       groupResInfo;    // multiple results build supporter
-  SInterval          interval;        // interval info
+  SInterval           interval;        // interval info
-  int32_t            primaryTsIndex;  // primary time stamp slot id from result of downstream operator.
+  int32_t             primaryTsIndex;  // primary time stamp slot id from result of downstream operator.
-  STimeWindowAggSupp twAggSup;
+  STimeWindowAggSupp  twAggSup;
-  bool               invertible;
+  bool                invertible;
-  bool               ignoreExpiredData;
+  bool                ignoreExpiredData;
-  bool               ignoreExpiredDataSaved;
+  bool                ignoreExpiredDataSaved;
-  SArray*            pDelWins;  // SWinRes
+  SArray*             pDelWins;  // SWinRes
-  int32_t            delIndex;
+  int32_t             delIndex;
-  SSDataBlock*       pDelRes;
+  SSDataBlock*        pDelRes;
-  SPhysiNode*        pPhyNode;  // create new child
+  SPhysiNode*         pPhyNode;  // create new child
-  SHashObj*          pPullDataMap;
+  SHashObj*           pPullDataMap;
-  SArray*            pPullWins;  // SPullWindowInfo
+  SArray*             pPullWins;  // SPullWindowInfo
-  int32_t            pullIndex;
+  int32_t             pullIndex;
-  SSDataBlock*       pPullDataRes;
+  SSDataBlock*        pPullDataRes;
-  SArray*            pChildren;
+  SArray*             pChildren;
-  int32_t            numOfChild;
+  int32_t             numOfChild;
-  SStreamState*      pState;  // void
+  SStreamState*       pState;  // void
-  SWinKey            delKey;
+  SWinKey             delKey;
-  uint64_t           numOfDatapack;
+  uint64_t            numOfDatapack;
-  SArray*            pUpdated;
+  SArray*             pUpdated;
-  SSHashObj*         pUpdatedMap;
+  SSHashObj*          pUpdatedMap;
-  int64_t            dataVersion;
+  int64_t             dataVersion;
-  SStateStore        stateStore;
+  SStateStore         stateStore;
-  bool               recvGetAll;
+  bool                recvGetAll;
-  SHashObj*          pFinalPullDataMap;
+  SHashObj*           pFinalPullDataMap;
-  SOpCheckPointInfo  checkPointInfo;
+  SOpCheckPointInfo   checkPointInfo;
-  bool               reCkBlock;
+  bool                reCkBlock;
-  SSDataBlock*       pCheckpointRes;
+  SSDataBlock*        pCheckpointRes;
+  struct SUpdateInfo* pUpdateInfo;
 } SStreamIntervalOperatorInfo;
 
 typedef struct SDataGroupInfo {
@@ -832,6 +834,7 @@ void     compactTimeWindow(SExprSupp* pSup, SStreamAggSupporter* pAggSup, STimeW
                            SSHashObj* pStUpdated, SSHashObj* pStDeleted, bool addGap);
 int32_t  releaseOutputBuf(void* pState, SRowBuffPos* pPos, SStateStore* pAPI);
 void     resetWinRange(STimeWindow* winRange);
+bool     checkExpiredData(SStateStore* pAPI, SUpdateInfo* pUpdateInfo, STimeWindowAggSupp* pTwSup, uint64_t tableId, TSKEY ts);
 
 int32_t encodeSSessionKey(void** buf, SSessionKey* key);
 void*   decodeSSessionKey(void* buf, SSessionKey* key);

source/libs/executor/src/scanoperator.c

@@ -1619,6 +1619,15 @@ void appendOneRowToStreamSpecialBlock(SSDataBlock* pBlock, TSKEY* pStartTs, TSKE
   pBlock->info.rows++;
 }
 
+bool checkExpiredData(SStateStore* pAPI, SUpdateInfo* pUpdateInfo, STimeWindowAggSupp* pTwSup, uint64_t tableId, TSKEY ts) {
+  bool isExpired = false;
+  bool isInc = pAPI->isIncrementalTimeStamp(pUpdateInfo, tableId, ts);
+  if (!isInc) {
+    isExpired = isOverdue(ts, pTwSup);
+  }
+  return isExpired;
+}
+
 static void checkUpdateData(SStreamScanInfo* pInfo, bool invertible, SSDataBlock* pBlock, bool out) {
   if (out) {
     blockDataCleanup(pInfo->pUpdateDataRes);

source/libs/executor/src/streameventwindowoperator.c

@@ -297,7 +297,8 @@ static void doStreamEventAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBl
   int32_t rows = pSDataBlock->info.rows;
   blockDataEnsureCapacity(pAggSup->pScanBlock, rows);
   for (int32_t i = 0; i < rows; i += winRows) {
-    if (pInfo->ignoreExpiredData && isOverdue(tsCols[i], &pInfo->twAggSup)) {
+    if (pInfo->ignoreExpiredData && checkExpiredData(&pInfo->streamAggSup.stateStore, pInfo->streamAggSup.pUpdateInfo,
+                                                     &pInfo->twAggSup, pSDataBlock->info.id.uid, tsCols[i])) {
       i++;
       continue;
     }

source/libs/executor/src/streamtimewindowoperator.c

@@ -450,6 +450,7 @@ void initIntervalDownStream(SOperatorInfo* downstream, uint16_t type, SStreamInt
   pScanInfo->interval = pInfo->interval;
   pScanInfo->twAggSup = pInfo->twAggSup;
   pScanInfo->pState = pInfo->pState;
+  pInfo->pUpdateInfo = pScanInfo->pUpdateInfo;
 }
 
 void compactFunctions(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx, int32_t numOfOutput,
@@ -800,7 +801,9 @@ static void doStreamIntervalAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDat
   }
   while (1) {
     bool isClosed = isCloseWindow(&nextWin, &pInfo->twAggSup);
-    if ((pInfo->ignoreExpiredData && isClosed && !IS_FINAL_INTERVAL_OP(pOperator)) ||
+    if ((!IS_FINAL_INTERVAL_OP(pOperator) && pInfo->ignoreExpiredData &&
+         checkExpiredData(&pInfo->stateStore, pInfo->pUpdateInfo, &pInfo->twAggSup, pSDataBlock->info.id.uid,
+                          nextWin.ekey)) ||
         !inSlidingWindow(&pInfo->interval, &nextWin, &pSDataBlock->info)) {
       startPos = getNexWindowPos(&pInfo->interval, &pSDataBlock->info, tsCols, startPos, nextWin.ekey, &nextWin);
       if (startPos < 0) {
@@ -1623,6 +1626,7 @@ void initDownStream(SOperatorInfo* downstream, SStreamAggSupporter* pAggSup, uin
                                                                 pScanInfo->igCheckUpdate);
   }
   pScanInfo->twAggSup = *pTwSup;
+  pAggSup->pUpdateInfo = pScanInfo->pUpdateInfo;
 }
 
 static TSKEY sesionTs(void* pKey) {
@@ -2018,7 +2022,9 @@ static void doStreamSessionAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSData
 
   TSKEY* endTsCols = (int64_t*)pEndTsCol->pData;
   for (int32_t i = 0; i < rows;) {
-    if (pInfo->ignoreExpiredData && isOverdue(endTsCols[i], &pInfo->twAggSup)) {
+    if (!IS_FINAL_SESSION_OP(pOperator) && pInfo->ignoreExpiredData &&
+        checkExpiredData(&pInfo->streamAggSup.stateStore, pInfo->streamAggSup.pUpdateInfo, &pInfo->twAggSup,
+                         pSDataBlock->info.id.uid, endTsCols[i])) {
       i++;
       continue;
     }
@@ -3334,7 +3340,8 @@ static void doStreamStateAggImpl(SOperatorInfo* pOperator, SSDataBlock* pSDataBl
   blockDataEnsureCapacity(pAggSup->pScanBlock, rows);
   SColumnInfoData* pKeyColInfo = taosArrayGet(pSDataBlock->pDataBlock, pInfo->stateCol.slotId);
   for (int32_t i = 0; i < rows; i += winRows) {
-    if (pInfo->ignoreExpiredData && isOverdue(tsCols[i], &pInfo->twAggSup) || colDataIsNull_s(pKeyColInfo, i)) {
+    if (pInfo->ignoreExpiredData && checkExpiredData(&pInfo->streamAggSup.stateStore, pInfo->streamAggSup.pUpdateInfo,
+                                                     &pInfo->twAggSup, pSDataBlock->info.id.uid, tsCols[i]) || colDataIsNull_s(pKeyColInfo, i)) {
       i++;
       continue;
     }

source/libs/executor/src/timewindowoperator.c

@@ -899,6 +899,7 @@ static void doStateWindowAggImpl(SOperatorInfo* pOperator, SStateWindowOperatorI
 
   SWindowRowsSup* pRowSup = &pInfo->winSup;
   pRowSup->numOfRows = 0;
+  pRowSup->startRowIndex = 0;
 
   struct SColumnDataAgg* pAgg = NULL;
   for (int32_t j = 0; j < pBlock->info.rows; ++j) {
@@ -923,9 +924,6 @@ static void doStateWindowAggImpl(SOperatorInfo* pOperator, SStateWindowOperatorI
       doKeepTuple(pRowSup, tsList[j], gid);
     } else if (compareVal(val, &pInfo->stateKey)) {
       doKeepTuple(pRowSup, tsList[j], gid);
-      if (j == 0 && pRowSup->startRowIndex != 0) {
-        pRowSup->startRowIndex = 0;
-      }
     } else {  // a new state window started
       SResultRow* pResult = NULL;
 

source/libs/function/src/builtins.c

@@ -3425,7 +3425,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
   {
     .name = "_wstart",
     .type = FUNCTION_TYPE_WSTART,
-    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_WINDOW_PC_FUNC | FUNC_MGT_KEEP_ORDER_FUNC,
+    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_WINDOW_PC_FUNC | FUNC_MGT_KEEP_ORDER_FUNC | FUNC_MGT_SKIP_SCAN_CHECK_FUNC,
     .translateFunc = translateTimePseudoColumn,
     .getEnvFunc   = getTimePseudoFuncEnv,
     .initFunc     = NULL,
@@ -3435,7 +3435,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
   {
     .name = "_wend",
     .type = FUNCTION_TYPE_WEND,
-    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_WINDOW_PC_FUNC | FUNC_MGT_KEEP_ORDER_FUNC,
+    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_WINDOW_PC_FUNC | FUNC_MGT_KEEP_ORDER_FUNC | FUNC_MGT_SKIP_SCAN_CHECK_FUNC,
     .translateFunc = translateTimePseudoColumn,
     .getEnvFunc   = getTimePseudoFuncEnv,
     .initFunc     = NULL,
@@ -3445,7 +3445,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
   {
     .name = "_wduration",
     .type = FUNCTION_TYPE_WDURATION,
-    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_WINDOW_PC_FUNC | FUNC_MGT_KEEP_ORDER_FUNC,
+    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_WINDOW_PC_FUNC | FUNC_MGT_KEEP_ORDER_FUNC | FUNC_MGT_SKIP_SCAN_CHECK_FUNC,
     .translateFunc = translateWduration,
     .getEnvFunc   = getTimePseudoFuncEnv,
     .initFunc     = NULL,

source/libs/function/src/builtinsimpl.c

@@ -5472,7 +5472,6 @@ int32_t blockDistFunction(SqlFunctionCtx* pCtx) {
   SColumnInfoData*      pInputCol = pInput->pData[0];
 
   SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx);
-
   STableBlockDistInfo* pDistInfo = GET_ROWCELL_INTERBUF(pResInfo);
 
   STableBlockDistInfo p1 = {0};
@@ -5481,6 +5480,7 @@ int32_t blockDistFunction(SqlFunctionCtx* pCtx) {
   pDistInfo->numOfBlocks += p1.numOfBlocks;
   pDistInfo->numOfTables += p1.numOfTables;
   pDistInfo->numOfInmemRows += p1.numOfInmemRows;
+  pDistInfo->numOfSttRows += p1.numOfSttRows;
   pDistInfo->totalSize += p1.totalSize;
   pDistInfo->totalRows += p1.totalRows;
   pDistInfo->numOfFiles += p1.numOfFiles;
@@ -5488,7 +5488,6 @@ int32_t blockDistFunction(SqlFunctionCtx* pCtx) {
   pDistInfo->defMinRows = p1.defMinRows;
   pDistInfo->defMaxRows = p1.defMaxRows;
   pDistInfo->rowSize = p1.rowSize;
-  pDistInfo->numOfSmallBlocks = p1.numOfSmallBlocks;
 
   if (pDistInfo->minRows > p1.minRows) {
     pDistInfo->minRows = p1.minRows;
@@ -5523,7 +5522,7 @@ int32_t tSerializeBlockDistInfo(void* buf, int32_t bufLen, const STableBlockDist
   if (tEncodeI32(&encoder, pInfo->defMaxRows) < 0) return -1;
   if (tEncodeI32(&encoder, pInfo->defMinRows) < 0) return -1;
   if (tEncodeU32(&encoder, pInfo->numOfInmemRows) < 0) return -1;
-  if (tEncodeU32(&encoder, pInfo->numOfSmallBlocks) < 0) return -1;
+  if (tEncodeU32(&encoder, pInfo->numOfSttRows) < 0) return -1;
   if (tEncodeU32(&encoder, pInfo->numOfVgroups) < 0) return -1;
 
   for (int32_t i = 0; i < tListLen(pInfo->blockRowsHisto); ++i) {
@@ -5555,7 +5554,7 @@ int32_t tDeserializeBlockDistInfo(void* buf, int32_t bufLen, STableBlockDistInfo
   if (tDecodeI32(&decoder, &pInfo->defMaxRows) < 0) return -1;
   if (tDecodeI32(&decoder, &pInfo->defMinRows) < 0) return -1;
   if (tDecodeU32(&decoder, &pInfo->numOfInmemRows) < 0) return -1;
-  if (tDecodeU32(&decoder, &pInfo->numOfSmallBlocks) < 0) return -1;
+  if (tDecodeU32(&decoder, &pInfo->numOfSttRows) < 0) return -1;
   if (tDecodeU32(&decoder, &pInfo->numOfVgroups) < 0) return -1;
 
   for (int32_t i = 0; i < tListLen(pInfo->blockRowsHisto); ++i) {
@@ -5589,7 +5588,7 @@ int32_t blockDistFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
   }
 
   int32_t len = sprintf(st + VARSTR_HEADER_SIZE,
-                        "Total_Blocks=[%d] Total_Size=[%.2f KB] Average_size=[%.2f KB] Compression_Ratio=[%.2f %c]",
+                        "Total_Blocks=[%d] Total_Size=[%.2f KiB] Average_size=[%.2f KiB] Compression_Ratio=[%.2f %c]",
                         pData->numOfBlocks, pData->totalSize / 1024.0, averageSize / 1024.0, compRatio, '%');
 
   varDataSetLen(st, len);
@@ -5600,14 +5599,16 @@ int32_t blockDistFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
     avgRows = pData->totalRows / pData->numOfBlocks;
   }
 
-  len = sprintf(st + VARSTR_HEADER_SIZE,
+  len = sprintf(st + VARSTR_HEADER_SIZE, "Block_Rows=[%" PRId64 "] MinRows=[%d] MaxRows=[%d] AvgRows=[%" PRId64 "]",
-                "Total_Rows=[%" PRId64 "] Inmem_Rows=[%d] MinRows=[%d] MaxRows=[%d] Average_Rows=[%" PRId64 "]",
+                pData->totalRows, pData->minRows, pData->maxRows, avgRows);
-                pData->totalRows, pData->numOfInmemRows, pData->minRows, pData->maxRows, avgRows);
-
   varDataSetLen(st, len);
   colDataSetVal(pColInfo, row++, st, false);
 
-  len = sprintf(st + VARSTR_HEADER_SIZE, "Total_Tables=[%d] Total_Files=[%d] Total_Vgroups=[%d]", pData->numOfTables,
+  len = sprintf(st + VARSTR_HEADER_SIZE, "Inmem_Rows=[%d] Stt_Rows=[%d] ", pData->numOfInmemRows, pData->numOfSttRows);
+  varDataSetLen(st, len);
+  colDataSetVal(pColInfo, row++, st, false);
+
+  len = sprintf(st + VARSTR_HEADER_SIZE, "Total_Tables=[%d] Total_Filesets=[%d] Total_Vgroups=[%d]", pData->numOfTables,
                 pData->numOfFiles, pData->numOfVgroups);
 
   varDataSetLen(st, len);

source/libs/parser/src/parInsertSql.c

@@ -268,26 +268,47 @@ static int32_t parseBoundColumns(SInsertParseContext* pCxt, const char** pSql, E
   return code;
 }
 
-static int parseTime(const char** end, SToken* pToken, int16_t timePrec, int64_t* time, SMsgBuf* pMsgBuf) {
+static int parseTimestampOrInterval(const char** end, SToken* pToken, int16_t timePrec, int64_t* ts, int64_t* interval, SMsgBuf* pMsgBuf, bool* isTs) {
-  int32_t     index = 0;
-  int64_t     interval;
-  int64_t     ts = 0;
-  const char* pTokenEnd = *end;
-
   if (pToken->type == TK_NOW) {
-    ts = taosGetTimestamp(timePrec);
+    *isTs = true;
+    *ts = taosGetTimestamp(timePrec);
   } else if (pToken->type == TK_TODAY) {
-    ts = taosGetTimestampToday(timePrec);
+    *isTs = true;
+    *ts = taosGetTimestampToday(timePrec);
   } else if (pToken->type == TK_NK_INTEGER) {
-    if (TSDB_CODE_SUCCESS != toInteger(pToken->z, pToken->n, 10, &ts)) {
+    *isTs = true;
+    if (TSDB_CODE_SUCCESS != toInteger(pToken->z, pToken->n, 10, ts)) {
       return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
     }
+  } else if (pToken->type == TK_NK_VARIABLE) {
+    char unit = 0;
+    *isTs = false;
+    if (parseAbsoluteDuration(pToken->z, pToken->n, interval, &unit, timePrec) != TSDB_CODE_SUCCESS) {
+      return TSDB_CODE_TSC_INVALID_OPERATION;
+    }
   } else {  // parse the RFC-3339/ISO-8601 timestamp format string
-    if (taosParseTime(pToken->z, time, pToken->n, timePrec, tsDaylight) != TSDB_CODE_SUCCESS) {
+    *isTs = true;
+    if (taosParseTime(pToken->z, ts, pToken->n, timePrec, tsDaylight) != TSDB_CODE_SUCCESS) {
       return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
     }
+  }
 
-    return TSDB_CODE_SUCCESS;
+  return TSDB_CODE_SUCCESS;
+}
+
+static int parseTime(const char** end, SToken* pToken, int16_t timePrec, int64_t* time, SMsgBuf* pMsgBuf) {
+  int32_t     index = 0, i = 0;
+  int64_t     interval = 0, tempInterval = 0;
+  int64_t     ts = 0, tempTs = 0;
+  bool firstIsTS = false, secondIsTs = false;
+  const char* pTokenEnd = *end;
+
+  if (TSDB_CODE_SUCCESS != parseTimestampOrInterval(&pTokenEnd, pToken, timePrec, &ts, &interval, pMsgBuf, &firstIsTS)) {
+    return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+  }
+
+  if (firstIsTS) {
+    *time = ts;
   }
 
   for (int k = pToken->n; pToken->z[k] != '\0'; k++) {
@@ -299,45 +320,98 @@ static int parseTime(const char** end, SToken* pToken, int16_t timePrec, int64_t
     }
     if (pToken->z[k] == ',') {
       *end = pTokenEnd;
+      if (!firstIsTS) {
+        return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+      }
       *time = ts;
-      return 0;
+      return TSDB_CODE_SUCCESS;
     }
-
     break;
   }
 
-  /*
+  while (pTokenEnd[i] != '\0') {
-   * time expression:
+    if (pTokenEnd[i] == ' ' || pTokenEnd[i] == '\t') {
-   * e.g., now+12a, now-5h
+      i++;
-   */
+      continue;
+    }
+    else if (pTokenEnd[i] == ',' || pTokenEnd[i] == ')') {
+      *end = pTokenEnd + i;
+      if (!firstIsTS) {
+        return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+      }
+      *time = ts;
+      return TSDB_CODE_SUCCESS;
+    } else {
+      break;
+    }
+  }
+  pTokenEnd = pTokenEnd + i;
+
   index = 0;
   SToken token = tStrGetToken(pTokenEnd, &index, false, NULL);
-  pTokenEnd += index;
 
   if (token.type == TK_NK_MINUS || token.type == TK_NK_PLUS) {
+    pTokenEnd += index;
     index = 0;
     SToken valueToken = tStrGetToken(pTokenEnd, &index, false, NULL);
     pTokenEnd += index;
+    char tmpTokenBuf[TSDB_MAX_BYTES_PER_ROW];
+    if (TK_NK_STRING == valueToken.type) {
+      if (valueToken.n >= TSDB_MAX_BYTES_PER_ROW) {
+        return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", valueToken.z);
+      }
+      int32_t len = trimString(valueToken.z, valueToken.n, tmpTokenBuf, TSDB_MAX_BYTES_PER_ROW);
+      valueToken.z = tmpTokenBuf;
+      valueToken.n = len;
+    }
+
+    if (TSDB_CODE_SUCCESS != parseTimestampOrInterval(&pTokenEnd, &valueToken, timePrec, &tempTs, &tempInterval, pMsgBuf, &secondIsTs)) {
+      return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+    }
 
     if (valueToken.n < 2) {
       return buildSyntaxErrMsg(pMsgBuf, "value expected in timestamp", token.z);
     }
 
-    char unit = 0;
+    if (secondIsTs) {
-    if (parseAbsoluteDuration(valueToken.z, valueToken.n, &interval, &unit, timePrec) != TSDB_CODE_SUCCESS) {
+      // not support operator between tow timestamp, such as today() + now()
-      return TSDB_CODE_TSC_INVALID_OPERATION;
+      if (firstIsTS) {
+        return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+      }
+      ts = tempTs;
+    }else {
+      // not support operator between tow interval, such as 2h + 3s
+      if (!firstIsTS) {
+        return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+      }
+      interval = tempInterval;
     }
-
+    if (token.type == TK_NK_MINUS) {
-    if (token.type == TK_NK_PLUS) {
+      // not support interval - ts,such as 2h - today()
-      ts += interval;
+      if (secondIsTs) {
+        return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+      }
+      *time = ts - interval;
     } else {
-      ts = ts - interval;
+      *time = ts + interval;
     }
 
-    *end = pTokenEnd;
+    for (int k = valueToken.n; valueToken.z[k] != '\0'; k++) {
+      if (valueToken.z[k] == ' ' || valueToken.z[k] == '\t') continue;
+      if (valueToken.z[k] == '(' && valueToken.z[k + 1] == ')') {  // for insert NOW()/TODAY()
+        *end = pTokenEnd = &valueToken.z[k + 2];
+        k++;
+        continue;
+      }
+      if (valueToken.z[k] == ',' || valueToken.z[k] == ')') {
+        *end = pTokenEnd;
+        return TSDB_CODE_SUCCESS;
+      }
+      return buildSyntaxErrMsg(pMsgBuf, "invalid timestamp format", pToken->z);
+    }
   }
 
-  *time = ts;
+  *end = pTokenEnd;
   return TSDB_CODE_SUCCESS;
 }
 
@@ -671,7 +745,7 @@ static int32_t checkAndTrimValue(SToken* pToken, char* tmpTokenBuf, SMsgBuf* pMs
   if ((pToken->type != TK_NOW && pToken->type != TK_TODAY && pToken->type != TK_NK_INTEGER &&
        pToken->type != TK_NK_STRING && pToken->type != TK_NK_FLOAT && pToken->type != TK_NK_BOOL &&
        pToken->type != TK_NULL && pToken->type != TK_NK_HEX && pToken->type != TK_NK_OCT &&
-       pToken->type != TK_NK_BIN) ||
+       pToken->type != TK_NK_BIN && pToken->type != TK_NK_VARIABLE) ||
       (pToken->n == 0) || (pToken->type == TK_NK_RP)) {
     return buildSyntaxErrMsg(pMsgBuf, "invalid data or symbol", pToken->z);
   }
@@ -850,6 +924,9 @@ static int32_t parseTagsClauseImpl(SInsertParseContext* pCxt, SVnodeModifyOpStmt
     SSchema* pTagSchema = &pSchema[pCxt->tags.pColIndex[i]];
     isJson = pTagSchema->type == TSDB_DATA_TYPE_JSON;
     code = checkAndTrimValue(&token, pCxt->tmpTokenBuf, &pCxt->msg);
+    if (TK_NK_VARIABLE == token.type) {
+      code = buildSyntaxErrMsg(&pCxt->msg, "not expected tags values ", token.z);
+    }
     if (TSDB_CODE_SUCCESS == code) {
       code = parseTagValue(pCxt, pStmt, &pStmt->pSql, pTagSchema, &token, pTagName, pTagVals, &pTag);
     }
@@ -1540,6 +1617,10 @@ static int32_t parseValueToken(SInsertParseContext* pCxt, const char** pSql, STo
     if (TSDB_DATA_TYPE_TIMESTAMP == pSchema->type && PRIMARYKEY_TIMESTAMP_COL_ID == pSchema->colId) {
       return buildSyntaxErrMsg(&pCxt->msg, "primary timestamp should not be null", pToken->z);
     }
+
+    if (TK_NK_VARIABLE == pToken->type && pSchema->type != TSDB_DATA_TYPE_TIMESTAMP) {
+      return buildSyntaxErrMsg(&pCxt->msg, "invalid values", pToken->z);
+    }
     pVal->flag = CV_FLAG_NULL;
     return TSDB_CODE_SUCCESS;
   }
@@ -1592,6 +1673,9 @@ typedef union SRowsDataContext{
 static int32_t parseTbnameToken(SInsertParseContext* pCxt, SStbRowsDataContext* pStbRowsCxt, SToken* pToken, bool* pFoundCtbName) {
   *pFoundCtbName = false;
   int32_t code = checkAndTrimValue(pToken, pCxt->tmpTokenBuf, &pCxt->msg);
+  if (TK_NK_VARIABLE == pToken->type) {
+    code = buildInvalidOperationMsg(&pCxt->msg, "not expected tbname");
+  }
   if (code == TSDB_CODE_SUCCESS){
     if (isNullValue(TSDB_DATA_TYPE_BINARY, pToken)) {
       return buildInvalidOperationMsg(&pCxt->msg, "tbname can not be null value");
@@ -1629,6 +1713,10 @@ static int32_t processCtbTagsAfterCtbName(SInsertParseContext* pCxt, SVnodeModif
       SToken*  pTagToken = (SToken*)(tagTokens + i);
       SSchema* pTagSchema = tagSchemas[i];
       code = checkAndTrimValue(pTagToken, pCxt->tmpTokenBuf, &pCxt->msg);
+      if (TK_NK_VARIABLE == pTagToken->type) {
+        code = buildInvalidOperationMsg(&pCxt->msg, "not expected tag");
+      }
+
       if (code == TSDB_CODE_SUCCESS) {
         code = parseTagValue(pCxt, pStmt, NULL, pTagSchema, pTagToken, pStbRowsCxt->aTagNames, pStbRowsCxt->aTagVals,
                              &pStbRowsCxt->pTag);
@@ -1673,6 +1761,9 @@ static int32_t doGetStbRowValues(SInsertParseContext* pCxt, SVnodeModifyOpStmt*
       const SSchema* pSchema = &pSchemas[pCols->pColIndex[i]];
       SColVal* pVal = taosArrayGet(pStbRowsCxt->aColVals, pCols->pColIndex[i]);
       code = parseValueToken(pCxt, ppSql, pToken, (SSchema*)pSchema, getTableInfo(pStbRowsCxt->pStbMeta).precision, pVal);
+      if (TK_NK_VARIABLE == pToken->type) {
+          code = buildInvalidOperationMsg(&pCxt->msg, "not expected row value");
+      }
     } else if (pCols->pColIndex[i] < tbnameIdx) {
       const SSchema* pTagSchema = &pSchemas[pCols->pColIndex[i]];
       if (canParseTagsAfter) {
@@ -1681,6 +1772,9 @@ static int32_t doGetStbRowValues(SInsertParseContext* pCxt, SVnodeModifyOpStmt*
         ++(*pNumOfTagTokens);
       } else {
         code = checkAndTrimValue(pToken, pCxt->tmpTokenBuf, &pCxt->msg);
+        if (TK_NK_VARIABLE == pToken->type) {
+          code = buildInvalidOperationMsg(&pCxt->msg, "not expected row value");
+        }
         if (code == TSDB_CODE_SUCCESS) {
           code = parseTagValue(pCxt, pStmt, ppSql, (SSchema*)pTagSchema, pToken, pTagNames, pTagVals, &pStbRowsCxt->pTag);
         }

source/libs/stream/inc/streamInt.h

@@ -57,11 +57,6 @@ typedef struct {
   SSDataBlock* pBlock;
 } SStreamTrigger;
 
-typedef struct SStreamGlobalEnv {
-  int8_t inited;
-  void*  timer;
-} SStreamGlobalEnv;
-
 typedef struct SStreamContinueExecInfo {
   SEpSet  epset;
   int32_t taskId;
@@ -92,7 +87,7 @@ struct SStreamQueue {
   int8_t      status;
 };
 
-extern SStreamGlobalEnv streamEnv;
+extern void*            streamTimer;
 extern int32_t          streamBackendId;
 extern int32_t          streamBackendCfWrapperId;
 extern int32_t          taskDbWrapperId;

source/libs/stream/src/stream.c

@@ -16,38 +16,18 @@
 #include "streamInt.h"
 #include "ttimer.h"
 
-SStreamGlobalEnv streamEnv;
+void* streamTimer = NULL;
 
-int32_t streamInit() {
+int32_t streamTimerInit() {
-  int8_t old;
+  streamTimer = taosTmrInit(1000, 100, 10000, "STREAM");
-  while (1) {
+  if (streamTimer == NULL) {
-    old = atomic_val_compare_exchange_8(&streamEnv.inited, 0, 2);
+    return -1;
-    if (old != 2) break;
   }
-
-  if (old == 0) {
-    streamEnv.timer = taosTmrInit(1000, 100, 10000, "STREAM");
-    if (streamEnv.timer == NULL) {
-      atomic_store_8(&streamEnv.inited, 0);
-      return -1;
-    }
-    atomic_store_8(&streamEnv.inited, 1);
-  }
-
   return 0;
 }
 
-void streamCleanUp() {
+void streamTimerCleanUp() {
-  int8_t old;
+  taosTmrCleanUp(streamTimer);
-  while (1) {
-    old = atomic_val_compare_exchange_8(&streamEnv.inited, 1, 2);
-    if (old != 2) break;
-  }
-
-  if (old == 1) {
-    taosTmrCleanUp(streamEnv.timer);
-    atomic_store_8(&streamEnv.inited, 0);
-  }
 }
 
 char* createStreamTaskIdStr(int64_t streamId, int32_t taskId) {
@@ -77,7 +57,7 @@ static void streamSchedByTimer(void* param, void* tmrId) {
       if (pTrigger == NULL) {
         stError("s-task:%s failed to prepare retrieve data trigger, code:%s, try again in %dms", id, "out of memory",
                 nextTrigger);
-        taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamEnv.timer, &pTask->schedInfo.pTimer);
+        taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pTimer);
         return;
       }
 
@@ -88,7 +68,7 @@ static void streamSchedByTimer(void* param, void* tmrId) {
 
         stError("s-task:%s failed to prepare retrieve data trigger, code:%s, try again in %dms", id, "out of memory",
                 nextTrigger);
-        taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamEnv.timer, &pTask->schedInfo.pTimer);
+        taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pTimer);
         return;
       }
 
@@ -97,7 +77,7 @@ static void streamSchedByTimer(void* param, void* tmrId) {
 
       int32_t code = streamTaskPutDataIntoInputQ(pTask, (SStreamQueueItem*)pTrigger);
       if (code != TSDB_CODE_SUCCESS) {
-        taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamEnv.timer, &pTask->schedInfo.pTimer);
+        taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pTimer);
         return;
       }
 
@@ -105,7 +85,7 @@ static void streamSchedByTimer(void* param, void* tmrId) {
     }
   }
 
-  taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamEnv.timer, &pTask->schedInfo.pTimer);
+  taosTmrReset(streamSchedByTimer, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pTimer);
 }
 
 int32_t streamSetupScheduleTrigger(SStreamTask* pTask) {
@@ -115,7 +95,7 @@ int32_t streamSetupScheduleTrigger(SStreamTask* pTask) {
 
     stDebug("s-task:%s setup scheduler trigger, delay:%" PRId64 " ms", pTask->id.idStr, pTask->info.triggerParam);
 
-    pTask->schedInfo.pTimer = taosTmrStart(streamSchedByTimer, (int32_t)pTask->info.triggerParam, pTask, streamEnv.timer);
+    pTask->schedInfo.pTimer = taosTmrStart(streamSchedByTimer, (int32_t)pTask->info.triggerParam, pTask, streamTimer);
     pTask->schedInfo.status = TASK_TRIGGER_STATUS__INACTIVE;
   }
 

source/libs/stream/src/streamDispatch.c

@@ -506,9 +506,9 @@ void streamRetryDispatchData(SStreamTask* pTask, int64_t waitDuration) {
          waitDuration, pTask->execInfo.dispatch, pTask->msgInfo.retryCount);
 
   if (pTask->msgInfo.pTimer != NULL) {
-    taosTmrReset(doRetryDispatchData, waitDuration, pTask, streamEnv.timer, &pTask->msgInfo.pTimer);
+    taosTmrReset(doRetryDispatchData, waitDuration, pTask, streamTimer, &pTask->msgInfo.pTimer);
   } else {
-    pTask->msgInfo.pTimer = taosTmrStart(doRetryDispatchData, waitDuration, pTask, streamEnv.timer);
+    pTask->msgInfo.pTimer = taosTmrStart(doRetryDispatchData, waitDuration, pTask, streamTimer);
   }
 }
 

source/libs/stream/src/streamMeta.c

@@ -369,7 +369,7 @@ SStreamMeta* streamMetaOpen(const char* path, void* ahandle, FTaskExpand expandF
   memcpy(pRid, &pMeta->rid, sizeof(pMeta->rid));
   metaRefMgtAdd(pMeta->vgId, pRid);
 
-  pMeta->pHbInfo->hbTmr = taosTmrStart(metaHbToMnode, META_HB_CHECK_INTERVAL, pRid, streamEnv.timer);
+  pMeta->pHbInfo->hbTmr = taosTmrStart(metaHbToMnode, META_HB_CHECK_INTERVAL, pRid, streamTimer);
   pMeta->pHbInfo->tickCounter = 0;
   pMeta->pHbInfo->stopFlag = 0;
   pMeta->qHandle = taosInitScheduler(32, 1, "stream-chkp", NULL);
@@ -1099,7 +1099,7 @@ void metaHbToMnode(void* param, void* tmrId) {
   }
 
   if (!waitForEnoughDuration(pMeta->pHbInfo)) {
-    taosTmrReset(metaHbToMnode, META_HB_CHECK_INTERVAL, param, streamEnv.timer, &pMeta->pHbInfo->hbTmr);
+    taosTmrReset(metaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr);
     taosReleaseRef(streamMetaId, rid);
     return;
   }
@@ -1215,7 +1215,7 @@ void metaHbToMnode(void* param, void* tmrId) {
 
 _end:
   clearHbMsg(&hbMsg, pIdList);
-  taosTmrReset(metaHbToMnode, META_HB_CHECK_INTERVAL, param, streamEnv.timer, &pMeta->pHbInfo->hbTmr);
+  taosTmrReset(metaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr);
   taosReleaseRef(streamMetaId, rid);
 }
 

source/libs/stream/src/streamStart.c

@@ -67,7 +67,8 @@ int32_t streamTaskSetReady(SStreamTask* pTask) {
   stDebug("s-task:%s all %d downstream ready, init completed, elapsed time:%" PRId64 "ms, task status:%s",
           pTask->id.idStr, numOfDowns, el, p);
 
-  streamMetaUpdateTaskDownstreamStatus(pTask, pTask->execInfo.init, pTask->execInfo.start, true);
+  streamMetaUpdateTaskDownstreamStatus(pTask->pMeta, pTask->id.streamId, pTask->id.taskId, pTask->execInfo.init,
+                                       pTask->execInfo.start, true);
   return TSDB_CODE_SUCCESS;
 }
 
@@ -114,7 +115,7 @@ static void doReExecScanhistory(void* param, void* tmrId) {
     // release the task.
     streamMetaReleaseTask(pTask->pMeta, pTask);
   } else {
-    taosTmrReset(doReExecScanhistory, SCANHISTORY_IDLE_TIME_SLICE, pTask, streamEnv.timer,
+    taosTmrReset(doReExecScanhistory, SCANHISTORY_IDLE_TIME_SLICE, pTask, streamTimer,
                  &pTask->schedHistoryInfo.pTimer);
   }
 }
@@ -137,9 +138,9 @@ int32_t streamReExecScanHistoryFuture(SStreamTask* pTask, int32_t idleDuration)
   stDebug("s-task:%s scan-history resumed in %.2fs, ref:%d", pTask->id.idStr, numOfTicks*0.1, ref);
 
   if (pTask->schedHistoryInfo.pTimer == NULL) {
-    pTask->schedHistoryInfo.pTimer = taosTmrStart(doReExecScanhistory, SCANHISTORY_IDLE_TIME_SLICE, pTask, streamEnv.timer);
+    pTask->schedHistoryInfo.pTimer = taosTmrStart(doReExecScanhistory, SCANHISTORY_IDLE_TIME_SLICE, pTask, streamTimer);
   } else {
-    taosTmrReset(doReExecScanhistory, SCANHISTORY_IDLE_TIME_SLICE, pTask, streamEnv.timer, &pTask->schedHistoryInfo.pTimer);
+    taosTmrReset(doReExecScanhistory, SCANHISTORY_IDLE_TIME_SLICE, pTask, streamTimer, &pTask->schedHistoryInfo.pTimer);
   }
 
   return TSDB_CODE_SUCCESS;
@@ -469,14 +470,16 @@ int32_t streamProcessCheckRsp(SStreamTask* pTask, const SStreamTaskCheckRsp* pRs
         addIntoNodeUpdateList(pTask, pRsp->downstreamNodeId);
       }
 
-      streamMetaUpdateTaskDownstreamStatus(pTask, pTask->execInfo.init, taosGetTimestampMs(), false);
+      streamMetaUpdateTaskDownstreamStatus(pTask->pMeta, pTask->id.streamId, pTask->id.taskId, pTask->execInfo.init,
+                                           taosGetTimestampMs(), false);
 
       // automatically set the related fill-history task to be failed.
       if (HAS_RELATED_FILLHISTORY_TASK(pTask)) {
         STaskId* pId = &pTask->hTaskInfo.id;
 
         SStreamTask* pHTask = streamMetaAcquireTask(pTask->pMeta, pId->streamId, pId->taskId);
-        streamMetaUpdateTaskDownstreamStatus(pHTask, pHTask->execInfo.init, taosGetTimestampMs(), false);
+        streamMetaUpdateTaskDownstreamStatus(pHTask->pMeta, pId->streamId, pId->taskId, pHTask->execInfo.init,
+                                             taosGetTimestampMs(), false);
         streamMetaReleaseTask(pTask->pMeta, pHTask);
       }
     } else {  // TASK_DOWNSTREAM_NOT_READY, let's retry in 100ms
@@ -485,7 +488,7 @@ int32_t streamProcessCheckRsp(SStreamTask* pTask, const SStreamTaskCheckRsp* pRs
       int32_t ref = atomic_add_fetch_32(&pTask->status.timerActive, 1);
       stDebug("s-task:%s downstream taskId:0x%x (vgId:%d) not ready, stage:%"PRId64", retry in 100ms, ref:%d ", id,
               pRsp->downstreamTaskId, pRsp->downstreamNodeId, pRsp->oldStage, ref);
-      pInfo->checkTimer = taosTmrStart(recheckDownstreamTasks, CHECK_DOWNSTREAM_INTERVAL, pInfo, streamEnv.timer);
+      pInfo->checkTimer = taosTmrStart(recheckDownstreamTasks, CHECK_DOWNSTREAM_INTERVAL, pInfo, streamTimer);
     }
   }
 
@@ -726,7 +729,7 @@ static void tryLaunchHistoryTask(void* param, void* tmrId) {
 
     pHTaskInfo->tickCount -= 1;
     if (pHTaskInfo->tickCount > 0) {
-      taosTmrReset(tryLaunchHistoryTask, LAUNCH_HTASK_INTERVAL, pInfo, streamEnv.timer, &pHTaskInfo->pTimer);
+      taosTmrReset(tryLaunchHistoryTask, LAUNCH_HTASK_INTERVAL, pInfo, streamTimer, &pHTaskInfo->pTimer);
       streamMetaReleaseTask(pMeta, pTask);
       return;
     }
@@ -754,7 +757,7 @@ static void tryLaunchHistoryTask(void* param, void* tmrId) {
         stDebug("s-task:%s status:%s failed to launch fill-history task:0x%x, retry launch:%dms, retryCount:%d",
                 pTask->id.idStr, p, hTaskId, pHTaskInfo->waitInterval, pHTaskInfo->retryTimes);
 
-        taosTmrReset(tryLaunchHistoryTask, LAUNCH_HTASK_INTERVAL, pInfo, streamEnv.timer, &pHTaskInfo->pTimer);
+        taosTmrReset(tryLaunchHistoryTask, LAUNCH_HTASK_INTERVAL, pInfo, streamTimer, &pHTaskInfo->pTimer);
         streamMetaReleaseTask(pMeta, pTask);
         return;
       }
@@ -815,7 +818,7 @@ int32_t streamLaunchFillHistoryTask(SStreamTask* pTask) {
     streamTaskInitForLaunchHTask(&pTask->hTaskInfo);
     if (pTask->hTaskInfo.pTimer == NULL) {
       int32_t ref = atomic_add_fetch_32(&pTask->status.timerActive, 1);
-      pTask->hTaskInfo.pTimer = taosTmrStart(tryLaunchHistoryTask, WAIT_FOR_MINIMAL_INTERVAL, pInfo, streamEnv.timer);
+      pTask->hTaskInfo.pTimer = taosTmrStart(tryLaunchHistoryTask, WAIT_FOR_MINIMAL_INTERVAL, pInfo, streamTimer);
       if (pTask->hTaskInfo.pTimer == NULL) {
         atomic_sub_fetch_32(&pTask->status.timerActive, 1);
         stError("s-task:%s failed to start timer, related fill-history task not launched, ref:%d", pTask->id.idStr,
@@ -828,7 +831,7 @@ int32_t streamLaunchFillHistoryTask(SStreamTask* pTask) {
     } else {  // timer exists
       ASSERT(pTask->status.timerActive >= 1);
       stDebug("s-task:%s set timer active flag, task timer not null", pTask->id.idStr);
-      taosTmrReset(tryLaunchHistoryTask, WAIT_FOR_MINIMAL_INTERVAL, pInfo, streamEnv.timer, &pTask->hTaskInfo.pTimer);
+      taosTmrReset(tryLaunchHistoryTask, WAIT_FOR_MINIMAL_INTERVAL, pInfo, streamTimer, &pTask->hTaskInfo.pTimer);
     }
 
     return TSDB_CODE_SUCCESS;
@@ -1066,15 +1069,13 @@ static void displayStatusInfo(SStreamMeta* pMeta, SHashObj* pTaskSet, bool succ)
   }
 }
 
-int32_t streamMetaUpdateTaskDownstreamStatus(SStreamTask* pTask, int64_t startTs, int64_t endTs, bool ready) {
+int32_t streamMetaUpdateTaskDownstreamStatus(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, int64_t startTs,
-  SStreamMeta* pMeta = pTask->pMeta;
+                                             int64_t endTs, bool ready) {
+  STaskStartInfo* pStartInfo = &pMeta->startInfo;
+  STaskId         id = {.streamId = streamId, .taskId = taskId};
 
   streamMetaWLock(pMeta);
-
+  SHashObj* pDst = ready ? pStartInfo->pReadyTaskSet : pStartInfo->pFailedTaskSet;
-  STaskId id = streamTaskExtractKey(pTask);
-  STaskStartInfo* pStartInfo = &pMeta->startInfo;
-
-  SHashObj* pDst = ready? pStartInfo->pReadyTaskSet:pStartInfo->pFailedTaskSet;
 
   STaskInitTs initTs = {.start = startTs, .end = endTs, .success = ready};
   taosHashPut(pDst, &id, sizeof(id), &initTs, sizeof(STaskInitTs));
@@ -1086,15 +1087,14 @@ int32_t streamMetaUpdateTaskDownstreamStatus(SStreamTask* pTask, int64_t startTs
     pStartInfo->readyTs = taosGetTimestampMs();
     pStartInfo->elapsedTime = (pStartInfo->startTs != 0) ? pStartInfo->readyTs - pStartInfo->startTs : 0;
 
-    stDebug("vgId:%d all %d task(s) check downstream completed, last completed task:%s level:%d, startTs:%" PRId64
+    stDebug("vgId:%d all %d task(s) check downstream completed, last completed task:0x%x startTs:%" PRId64
             ", readyTs:%" PRId64 " total elapsed time:%.2fs",
-            pMeta->vgId, numOfTotal, pTask->id.idStr, pTask->info.taskLevel, pStartInfo->startTs, pStartInfo->readyTs,
+            pMeta->vgId, numOfTotal, taskId, pStartInfo->startTs, pStartInfo->readyTs,
             pStartInfo->elapsedTime / 1000.0);
 
     // print the initialization elapsed time and info
     displayStatusInfo(pMeta, pStartInfo->pReadyTaskSet, true);
     displayStatusInfo(pMeta, pStartInfo->pFailedTaskSet, false);
-
     streamMetaResetStartInfo(pStartInfo);
   } else {
     stDebug("vgId:%d recv check down results:%d, total:%d", pMeta->vgId, numOfRecv, numOfTotal);

source/libs/stream/src/streamUpdate.c

@@ -129,9 +129,9 @@ SUpdateInfo *updateInfoInit(int64_t interval, int32_t precision, int64_t waterma
     }
     pInfo->numBuckets = DEFAULT_BUCKET_SIZE;
     pInfo->pCloseWinSBF = NULL;
-    _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT);
-    pInfo->pMap = taosHashInit(DEFAULT_MAP_CAPACITY, hashFn, true, HASH_NO_LOCK);
   }
+  _hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_UBIGINT);
+  pInfo->pMap = taosHashInit(DEFAULT_MAP_CAPACITY, hashFn, true, HASH_NO_LOCK);
   pInfo->maxDataVersion = 0;
   return pInfo;
 }
@@ -384,3 +384,14 @@ int32_t updateInfoDeserialize(void *buf, int32_t bufLen, SUpdateInfo *pInfo) {
   tDecoderClear(&decoder);
   return 0;
 }
+
+bool isIncrementalTimeStamp(SUpdateInfo *pInfo, uint64_t tableId, TSKEY ts) {
+  TSKEY *pMapMaxTs = taosHashGet(pInfo->pMap, &tableId, sizeof(uint64_t));
+  bool res = true;
+  if ( pMapMaxTs && ts < *pMapMaxTs ) {
+    res = false;
+  } else {
+    taosHashPut(pInfo->pMap, &tableId, sizeof(uint64_t), &ts, sizeof(TSKEY));
+  }
+  return res;
+}

source/libs/transport/inc/transComm.h

@@ -23,6 +23,7 @@ extern "C" {
 #include "taoserror.h"
 #include "theap.h"
 #include "tmisce.h"
+#include "tmsg.h"
 #include "transLog.h"
 #include "transportInt.h"
 #include "trpc.h"

source/libs/transport/src/transCli.c

@@ -1468,7 +1468,7 @@ static FORCE_INLINE uint32_t cliGetIpFromFqdnCache(SHashObj* cache, char* fqdn)
   if (v == NULL) {
     addr = taosGetIpv4FromFqdn(fqdn);
     if (addr == 0xffffffff) {
-      terrno = TAOS_SYSTEM_ERROR(errno);
+      terrno = TSDB_CODE_RPC_FQDN_ERROR;
       tError("failed to get ip from fqdn:%s since %s", fqdn, terrstr());
       return addr;
     }

source/os/test/osTests.cpp

@@ -32,6 +32,35 @@
 
 #ifdef WINDOWS
 #include <windows.h>
+#else
+
+#include <arpa/inet.h>
+
+TEST(osTest, osFQDNSuccess) {
+  char     fqdn[1024];
+  char     ipString[INET_ADDRSTRLEN];
+  int      code = taosGetFqdn(fqdn);
+  uint32_t ipv4 = taosGetIpv4FromFqdn(fqdn);
+  ASSERT_NE(ipv4, 0xffffffff);
+
+  struct in_addr addr;
+  addr.s_addr = htonl(ipv4);
+  snprintf(ipString, INET_ADDRSTRLEN, "%u.%u.%u.%u", (unsigned int)(addr.s_addr >> 24) & 0xFF,
+           (unsigned int)(addr.s_addr >> 16) & 0xFF, (unsigned int)(addr.s_addr >> 8) & 0xFF,
+           (unsigned int)(addr.s_addr) & 0xFF);
+  printf("fqdn:%s  ip:%s\n", fqdn, ipString);
+}
+
+TEST(osTest, osFQDNFailed) {
+  char     fqdn[1024] = "fqdn_test_not_found";
+  char     ipString[24];
+  uint32_t ipv4 = taosGetIpv4FromFqdn(fqdn);
+  ASSERT_EQ(ipv4, 0xffffffff);
+
+  terrno = TSDB_CODE_RPC_FQDN_ERROR;
+  printf("fqdn:%s transfer to ip failed!\n", fqdn);
+}
+
 #endif  //  WINDOWS
 
 TEST(osTest, osSystem) {

source/util/src/terror.c

@@ -247,7 +247,8 @@ TAOS_DEFINE_ERROR(TSDB_CODE_MND_MNODE_ALREADY_EXIST,      "Mnode already exists"
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_MNODE_NOT_EXIST,          "Mnode not there")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_QNODE_ALREADY_EXIST,      "Qnode already exists")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_QNODE_NOT_EXIST,          "Qnode not there")
-TAOS_DEFINE_ERROR(TSDB_CODE_MND_SNODE_ALREADY_EXIST,      "Snode already exists")
+//TAOS_DEFINE_ERROR(TSDB_CODE_MND_SNODE_ALREADY_EXIST,      "Snode already exists")
+TAOS_DEFINE_ERROR(TSDB_CODE_MND_SNODE_ALREADY_EXIST,      "Snode can only be created 1")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_SNODE_NOT_EXIST,          "Snode not there")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOO_FEW_MNODES,           "The replica of mnode cannot less than 1")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOO_MANY_MNODES,          "The replica of mnode cannot exceed 3")

tests/parallel_test/cases.task

@@ -6,6 +6,7 @@
 ,,y,unit-test,bash test.sh
 
 #system test
+,,y,system-test,./pytest.sh python3 ./test.py -f 8-stream/stream_basic.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 8-stream/scalar_function.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 8-stream/at_once_interval.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 8-stream/at_once_session.py
@@ -303,6 +304,7 @@ e
 ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/ts-4272.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/test_ts4295.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/test_td27388.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_timestamp.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/show.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/show_tag_index.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/information_schema.py
@@ -856,6 +858,7 @@ e
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/projectionDesc.py -Q 4
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/odbc.py
 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/fill_with_group.py
+,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/state_window.py -Q 3
 ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-21561.py -Q 4
 ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-20582.py
 ,,n,system-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/insertMix.py -N 3

tests/script/tsim/snode/basic1.sim

@@ -113,11 +113,7 @@ sql_error drop snode on dnode 2
 
 print =============== create drop snodes
 sql create snode on dnode 1
-sql create snode on dnode 2
+sql_error create snode on dnode 2
-sql show snodes
-if $rows != 2 then 
-  return -1
-endi
 
 print =============== restart
 system sh/exec.sh -n dnode1 -s stop -x SIGINT
@@ -127,7 +123,7 @@ system sh/exec.sh -n dnode2 -s start
 
 sleep 2000
 sql show snodes
-if $rows != 2 then 
+if $rows != 1 then
   return -1
 endi
 

tests/script/tsim/stream/ignoreExpiredData.sim

@@ -107,7 +107,7 @@ sql select * from information_schema.ins_databases
 
 print ======database=$rows
 
-sql use test1
+sql use test1;
 
 sql create stable st(ts timestamp,a int,b int,c int) tags(ta int,tb int,tc int);
 sql create table ts1 using st tags(1,1,1);
@@ -118,9 +118,9 @@ sql insert into ts1 values(1648791211000,1,2,3);
 sleep 1000
 sql insert into ts1 values(1648791222001,2,2,3);
 sleep 1000
-sql insert into ts2 values(1648791211000,1,2,3);
-sleep 1000
 sql insert into ts2 values(1648791222001,2,2,3);
+sleep 1000
+sql insert into ts2 values(1648791211000,1,2,3);
 
 $loop_count = 0
 loop4:
@@ -132,16 +132,26 @@ if $loop_count == 10 then
   return -1
 endi
 
-if $data01 != 2 then
+if $data01 != 1 then
   print =====data01=$data01
   goto loop4
 endi
 
-if $data02 != 2 then
+if $data02 != 1 then
   print =====data02=$data02
   goto loop4
 endi
 
+if $data11 != 2 then
+  print =====data11=$data11
+  goto loop4
+endi
+
+if $data12 != 2 then
+  print =====data12=$data12
+  goto loop4
+endi
+
 $loop_count = 0
 loop5:
 sleep 1000
@@ -162,4 +172,148 @@ if $data02 != 1 then
   goto loop5
 endi
 
+if $data11 != 2 then
+  print =====data11=$data11
+  goto loop4
+endi
+
+if $data12 != 2 then
+  print =====data12=$data12
+  goto loop4
+endi
+
+
+print =============== create database test2
+sql create database test2 vgroups 4
+sql select * from information_schema.ins_databases
+
+print ======database=$rows
+
+sql use test2;
+
+sql create stable st(ts timestamp,a int,b int,c int) tags(ta int,tb int,tc int);
+sql create table ts1 using st tags(1,1,1);
+sql create table ts2 using st tags(2,2,2);
+sql create table ts3 using st tags(3,3,3);
+sql create table ts4 using st tags(4,4,4);
+sql create stream streams_21 trigger at_once IGNORE EXPIRED 1 into streamt_21 as select  _wstart, count(*) c1 from st interval(10s) ;
+sleep 1000
+
+sql insert into ts1 values(1648791211000,1,2,3);
+sql insert into ts1 values(1648791211001,2,2,3);
+sql insert into ts1 values(1648791211002,2,2,3);
+sql insert into ts1 values(1648791211003,2,2,3);
+sql insert into ts1 values(1648791211004,2,2,3);
+
+sleep 1000
+sql insert into ts2 values(1648791201000,1,2,3);
+sql insert into ts2 values(1648791201001,2,2,3);
+sql insert into ts2 values(1648791201002,2,2,3);
+sql insert into ts2 values(1648791201003,2,2,3);
+sql insert into ts2 values(1648791201004,2,2,3);
+
+sleep 1000
+sql insert into ts2 values(1648791101000,1,2,3);
+sql insert into ts2 values(1648791101001,2,2,3);
+sql insert into ts2 values(1648791101002,2,2,3);
+sql insert into ts2 values(1648791101003,2,2,3);
+sql insert into ts2 values(1648791101004,2,2,3);
+
+
+$loop_count = 0
+loop6:
+sleep 1000
+print 1 select * from streamt_21;
+sql select * from streamt_21;
+
+$loop_count = $loop_count + 1
+if $loop_count == 10 then
+  return -1
+endi
+
+if $rows != 2 then
+  print =====rows=$rows
+  goto loop6
+endi
+
+if $data01 != 5 then
+  print =====data01=$data01
+  goto loop6
+endi
+
+if $data11 != 5 then
+  print =====data11=$data11
+  goto loop6
+endi
+
+sleep 1000
+sql insert into ts3 values(1648791241000,1,2,3);
+
+sleep 1000
+sql insert into ts3 values(1648791231001,2,2,3);
+sql insert into ts3 values(1648791231002,2,2,3);
+sql insert into ts3 values(1648791231003,2,2,3);
+sql insert into ts3 values(1648791231004,2,2,3);
+
+$loop_count = 0
+loop7:
+sleep 1000
+print 2 select * from streamt_21;
+sql select * from streamt_21;
+
+$loop_count = $loop_count + 1
+if $loop_count == 10 then
+  return -1
+endi
+
+if $rows != 3 then
+  print =====rows=$rows
+  goto loop7
+endi
+
+if $data21 != 1 then
+  print =====data21=$data21
+  goto loop7
+endi
+
+sleep 1000
+sql insert into ts4 values(1648791231001,2,2,3);
+sql insert into ts4 values(1648791231002,2,2,3);
+sql insert into ts4 values(1648791231003,2,2,3);
+sql insert into ts4 values(1648791231004,2,2,3);
+
+sleep 1000
+sql insert into ts4 values(1648791211001,2,2,3);
+sql insert into ts4 values(1648791211002,2,2,3);
+sql insert into ts4 values(1648791211003,2,2,3);
+sql insert into ts4 values(1648791211004,2,2,3);
+
+$loop_count = 0
+loop8:
+sleep 1000
+print 3 select * from streamt_21;
+sql select * from streamt_21;
+
+$loop_count = $loop_count + 1
+if $loop_count == 10 then
+  return -1
+endi
+
+if $rows != 4 then
+  print =====rows=$rows
+  goto loop8
+endi
+
+if $data21 != 4 then
+  print =====data21=$data21
+  goto loop8
+endi
+
+if $data31 != 1 then
+  print =====data31=$data31
+  goto loop8
+endi
+
+print ============================end
+
 system sh/stop_dnodes.sh

tests/system-test/1-insert/insert_timestamp.py

@@ -0,0 +1,70 @@
+import sys 
+from util.log import *
+from util.cases import *
+from util.sql import *
+from util.dnodes import tdDnodes
+from math import inf
+
+
+class TDTestCase:
+    def init(self, conn, logSql, replicaVer=1):
+        tdLog.debug("start to execute %s" % __file__)
+        tdSql.init(conn.cursor(), True)
+
+    #def prepare_data(self):
+        
+
+    def run(self):
+        tdSql.execute("create database test_insert_timestamp;")
+        tdSql.execute("use test_insert_timestamp;")
+        tdSql.execute("create stable st(ts timestamp, c1 int) tags(id int);")
+        tdSql.execute("create table test_t using st tags(1);")
+
+        tdSql.error("insert into test_t values(now + today(), 1 ); ")
+        tdSql.error("insert into test_t values(now - today(), 1 ); ")
+        tdSql.error("insert into test_t values(today() + now(), 1 ); ")
+        tdSql.error("insert into test_t values(today() - now(), 1 ); ")
+        tdSql.error("insert into test_t values(2h - now(), 1 ); ")
+        tdSql.error("insert into test_t values(2h - today(), 1 ); ")
+        tdSql.error("insert into test_t values(2h - 1h, 1 ); ")
+        tdSql.error("insert into test_t values(2h + 1h, 1 ); ")
+        tdSql.error("insert into test_t values('2023-11-28 00:00:00.000' + '2023-11-28 00:00:00.000', 1 ); ")
+        tdSql.error("insert into test_t values('2023-11-28 00:00:00.000' + 1701111600000, 1 ); ")
+        tdSql.error("insert into test_t values(1701111500000 + 1701111600000, 1 ); ")
+        tdSql.error("insert into test_insert_timestamp.test_t values(1701111600000 + 1h + 1s, 4); ")
+
+        tdSql.execute("insert into test_insert_timestamp.test_t values(1701111600000 + 1h, 4); ")
+        tdSql.execute("insert into test_insert_timestamp.test_t values(2h + 1701111600000, 5); ")
+        tdSql.execute("insert into test_insert_timestamp.test_t values('2023-11-28 00:00:00.000' + 1h, 1); ")
+        tdSql.execute("insert into test_insert_timestamp.test_t values(3h + '2023-11-28 00:00:00.000', 3); ")
+        tdSql.execute("insert into test_insert_timestamp.test_t values(1701111600000 - 1h, 2); ")
+        tdSql.execute("insert into test_insert_timestamp.test_t values(1701122400000, 6); ")
+        tdSql.execute("insert into test_insert_timestamp.test_t values('2023-11-28 07:00:00.000', 7); ")
+
+        tdSql.query(f'select ts, c1 from test_t order by ts;')
+        tdSql.checkRows(7)
+        tdSql.checkEqual(tdSql.queryResult[0][0], datetime.datetime(2023, 11, 28, 1, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[0][1], 1)
+        tdSql.checkEqual(tdSql.queryResult[1][0], datetime.datetime(2023, 11, 28, 2, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[1][1], 2)
+        tdSql.checkEqual(tdSql.queryResult[2][0], datetime.datetime(2023, 11, 28, 3, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[2][1], 3)
+        tdSql.checkEqual(tdSql.queryResult[3][0], datetime.datetime(2023, 11, 28, 4, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[3][1], 4)
+        tdSql.checkEqual(tdSql.queryResult[4][0], datetime.datetime(2023, 11, 28, 5, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[4][1], 5)
+        tdSql.checkEqual(tdSql.queryResult[5][0], datetime.datetime(2023, 11, 28, 6, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[5][1], 6)
+        tdSql.checkEqual(tdSql.queryResult[6][0], datetime.datetime(2023, 11, 28, 7, 0, 0) )
+        tdSql.checkEqual(tdSql.queryResult[6][1], 7)
+         
+        tdSql.execute("drop table if exists test_t ;")
+        tdSql.execute("drop stable if exists st;")
+        tdSql.execute("drop database if exists test_insert_timestamp;")
+        
+    def stop(self):
+        tdSql.close()
+        tdLog.success("%s successfully executed" % __file__)
+
+tdCases.addWindows(__file__, TDTestCase())
+tdCases.addLinux(__file__, TDTestCase())

tests/system-test/2-query/state_window.py

@@ -0,0 +1,203 @@
+import taos
+import sys
+import time
+import socket
+import os
+import threading
+import math
+
+from util.log import *
+from util.sql import *
+from util.cases import *
+from util.dnodes import *
+from util.common import *
+# from tmqCommon import *
+
+class TDTestCase:
+    def __init__(self):
+        self.vgroups    = 4
+        self.ctbNum     = 1
+        self.rowsPerTbl = 10
+        self.duraion = '1h'
+
+    def init(self, conn, logSql, replicaVar=1):
+        self.replicaVar = int(replicaVar)
+        tdLog.debug(f"start to excute {__file__}")
+        tdSql.init(conn.cursor(), False)
+
+    def create_database(self,tsql, dbName,dropFlag=1,vgroups=2,replica=1, duration:str='1d'):
+        if dropFlag == 1:
+            tsql.execute("drop database if exists %s"%(dbName))
+
+        tsql.execute("create database if not exists %s vgroups %d replica %d duration %s"%(dbName, vgroups, replica, duration))
+        tdLog.debug("complete to create database %s"%(dbName))
+        return
+
+    def create_stable(self,tsql, paraDict):
+        colString = tdCom.gen_column_type_str(colname_prefix=paraDict["colPrefix"], column_elm_list=paraDict["colSchema"])
+        tagString = tdCom.gen_tag_type_str(tagname_prefix=paraDict["tagPrefix"], tag_elm_list=paraDict["tagSchema"])
+        sqlString = f"create table if not exists %s.%s (%s) tags (%s)"%(paraDict["dbName"], paraDict["stbName"], colString, tagString)
+        tdLog.debug("%s"%(sqlString))
+        tsql.execute(sqlString)
+        return
+
+    def create_ctable(self,tsql=None, dbName='dbx',stbName='stb',ctbPrefix='ctb',ctbNum=1,ctbStartIdx=0):
+        for i in range(ctbNum):
+            sqlString = "create table %s.%s%d using %s.%s tags(%d, 'tb%d', 'tb%d', %d, %d, %d)" % \
+                    (dbName,ctbPrefix,i+ctbStartIdx,dbName,stbName,(i+ctbStartIdx) % 5,i+ctbStartIdx,i+ctbStartIdx,i+ctbStartIdx,i+ctbStartIdx,i+ctbStartIdx)
+            tsql.execute(sqlString)
+
+        tdLog.debug("complete to create %d child tables by %s.%s" %(ctbNum, dbName, stbName))
+        return
+
+    def insert_data(self,tsql,dbName,ctbPrefix,ctbNum,rowsPerTbl,batchNum,startTs,tsStep):
+        tdLog.debug("start to insert data ............")
+        tsql.execute("use %s" %dbName)
+        pre_insert = "insert into "
+        sql = pre_insert
+
+        for i in range(ctbNum):
+            rowsBatched = 0
+            sql += " %s%d values "%(ctbPrefix,i)
+            for j in range(rowsPerTbl):
+                if (i < ctbNum/2):
+                    sql += "(%d, %d, %d, %d,%d,%d,%d,true,'binary%d', 'nchar%d') "%(startTs + j*tsStep, j%10, 1, j%10, j%10, j%10, j%10, j%10, j%10)
+                else:
+                    sql += "(%d, %d, NULL, %d,NULL,%d,%d,true,'binary%d', 'nchar%d') "%(startTs + j*tsStep, j%10, j%10, j%10, j%10, j%10, j%10)
+                rowsBatched += 1
+                if ((rowsBatched == batchNum) or (j == rowsPerTbl - 1)):
+                    tsql.execute(sql)
+                    rowsBatched = 0
+                    if j < rowsPerTbl - 1:
+                        sql = "insert into %s%d values " %(ctbPrefix,i)
+                    else:
+                        sql = "insert into "
+        if sql != pre_insert:
+            tsql.execute(sql)
+        tdLog.debug("insert data ............ [OK]")
+        return
+
+    def prepareTestEnv(self):
+        tdLog.printNoPrefix("======== prepare test env include database, stable, ctables, and insert data: ")
+        paraDict = {'dbName':     'test',
+                    'dropFlag':   1,
+                    'vgroups':    2,
+                    'stbName':    'meters',
+                    'colPrefix':  'c',
+                    'tagPrefix':  't',
+                    'colSchema':   [{'type': 'INT', 'count':1},{'type': 'BIGINT', 'count':1},{'type': 'FLOAT', 'count':1},{'type': 'DOUBLE', 'count':1},{'type': 'smallint', 'count':1},{'type': 'tinyint', 'count':1},{'type': 'bool', 'count':1},{'type': 'binary', 'len':10, 'count':1},{'type': 'nchar', 'len':10, 'count':1}],
+                    'tagSchema':   [{'type': 'INT', 'count':1},{'type': 'nchar', 'len':20, 'count':1},{'type': 'binary', 'len':20, 'count':1},{'type': 'BIGINT', 'count':1},{'type': 'smallint', 'count':1},{'type': 'DOUBLE', 'count':1}],
+                    'ctbPrefix':  't',
+                    'ctbStartIdx': 0,
+                    'ctbNum':     100,
+                    'rowsPerTbl': 10000,
+                    'batchNum':   3000,
+                    'startTs':    1537146000000,
+                    'tsStep':     600000}
+
+        paraDict['vgroups'] = self.vgroups
+        paraDict['ctbNum'] = self.ctbNum
+        paraDict['rowsPerTbl'] = self.rowsPerTbl
+
+        tdLog.info("create database")
+        self.create_database(tsql=tdSql, dbName=paraDict["dbName"], dropFlag=paraDict["dropFlag"], vgroups=paraDict["vgroups"], replica=self.replicaVar, duration=self.duraion)
+
+        tdLog.info("create stb")
+        self.create_stable(tsql=tdSql, paraDict=paraDict)
+
+        tdLog.info("create child tables")
+        self.create_ctable(tsql=tdSql, dbName=paraDict["dbName"], \
+                stbName=paraDict["stbName"],ctbPrefix=paraDict["ctbPrefix"],\
+                ctbNum=paraDict["ctbNum"],ctbStartIdx=paraDict["ctbStartIdx"])
+        self.insert_data(tsql=tdSql, dbName=paraDict["dbName"],\
+                ctbPrefix=paraDict["ctbPrefix"],ctbNum=paraDict["ctbNum"],\
+                rowsPerTbl=paraDict["rowsPerTbl"],batchNum=paraDict["batchNum"],\
+                startTs=paraDict["startTs"],tsStep=paraDict["tsStep"])
+        return
+
+    def prepare_original_data(self):
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,3,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("flush database test", queryTimes=1)
+        time.sleep(2)
+
+    def test_crash_for_state_window1(self):
+        tdSql.execute("drop database if exists test")
+        self.prepareTestEnv()
+        tdSql.execute("alter local 'queryPolicy' '3'")
+        self.prepare_original_data()
+        tdSql.execute("insert into t0 values(now, 4,4,4,4,4,4,4,4,4)", queryTimes=1)
+        tdSql.execute("select bottom(c1, 1), c2 from t0 state_window(c2) order by ts", queryTimes=1)
+
+    def test_crash_for_state_window2(self):
+        tdSql.execute("drop database if exists test")
+        self.prepareTestEnv()
+        tdSql.execute("alter local 'queryPolicy' '3'")
+        self.prepare_original_data()
+        tdSql.execute("insert into t0 values(now, 4,NULL,4,4,4,4,4,4,4)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 4,4,4,4,4,4,4,4,4)", queryTimes=1)
+        tdSql.execute("select bottom(c1, 1), c2 from t0 state_window(c2) order by ts", queryTimes=1)
+
+    def test_crash_for_state_window3(self):
+        tdSql.execute("drop database if exists test")
+        self.prepareTestEnv()
+        tdSql.execute("alter local 'queryPolicy' '3'")
+        self.prepare_original_data()
+        tdSql.execute("insert into t0 values(now, 4,NULL,4,4,4,4,4,4,4)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 4,5,4,4,4,4,4,4,4)", queryTimes=1)
+        tdSql.execute("select bottom(c1, 1), c2 from t0 state_window(c2) order by ts", queryTimes=1)
+
+    def test_crash_for_state_window4(self):
+        tdSql.execute("drop database if exists test")
+        self.prepareTestEnv()
+        tdSql.execute("alter local 'queryPolicy' '3'")
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,3,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,NULL,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,NULL,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("flush database test", queryTimes=1)
+        time.sleep(2)
+        tdSql.execute("insert into t0 values(now, 3,NULL,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("select bottom(c1, 1), c2 from t0 state_window(c2) order by ts", queryTimes=1)
+
+    def test_crash_for_state_window5(self):
+        tdSql.execute("drop database if exists test")
+        self.prepareTestEnv()
+        tdSql.execute("alter local 'queryPolicy' '3'")
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 2,2,2,2,2,2,2,2,2)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,3,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,NULL,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,NULL,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("flush database test", queryTimes=1)
+        time.sleep(2)
+        tdSql.execute("insert into t0 values(now, 3,NULL,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("insert into t0 values(now, 3,3,3,3,3,3,3,3,3)", queryTimes=1)
+        tdSql.execute("select bottom(c1, 1), c2 from t0 state_window(c2) order by ts", queryTimes=1)
+
+    def run(self):
+        self.test_crash_for_state_window1()
+        self.test_crash_for_state_window2()
+        self.test_crash_for_state_window3()
+        self.test_crash_for_state_window4()
+        self.test_crash_for_state_window5()
+
+    def stop(self):
+        tdSql.close()
+        tdLog.success(f"{__file__} successfully executed")
+
+event = threading.Event()
+
+tdCases.addLinux(__file__, TDTestCase())
+tdCases.addWindows(__file__, TDTestCase())

tests/system-test/7-tmq/tmqVnodeSplit-column.py

@@ -23,7 +23,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -49,7 +49,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -118,7 +118,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,

tests/system-test/7-tmq/tmqVnodeSplit-db.py

@@ -23,7 +23,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -49,7 +49,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -118,7 +118,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -189,7 +189,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/7-tmq/tmqVnodeSplit-stb-select-duplicatedata-false.py

@@ -25,7 +25,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -51,7 +51,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -120,7 +120,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  120,
@@ -189,7 +189,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/7-tmq/tmqVnodeSplit-stb-select-duplicatedata.py

@@ -25,7 +25,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -51,7 +51,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -120,7 +120,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  120,
@@ -189,7 +189,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/7-tmq/tmqVnodeSplit-stb-select.py

@@ -27,7 +27,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -53,7 +53,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -122,7 +122,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  120,
@@ -192,7 +192,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/7-tmq/tmqVnodeSplit-stb.py

@@ -25,7 +25,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -51,7 +51,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -120,7 +120,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -190,7 +190,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/7-tmq/tmqVnodeTransform-db.py

@@ -20,7 +20,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -46,7 +46,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  30,
@@ -138,7 +138,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  10,
@@ -217,7 +217,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  10,

tests/system-test/7-tmq/tmqVnodeTransform-stb.py

@@ -20,7 +20,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -46,7 +46,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -137,7 +137,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb1',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -207,7 +207,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/7-tmq/tmqVnodeTransform.py

@@ -20,7 +20,7 @@ class TDTestCase:
     def __init__(self):
         self.vgroups    = 1
         self.ctbNum     = 10
-        self.rowsPerTbl = 10000
+        self.rowsPerTbl = 1000
 
     def init(self, conn, logSql, replicaVar=1):
         self.replicaVar = int(replicaVar)
@@ -46,7 +46,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -137,7 +137,7 @@ class TDTestCase:
                     'ctbPrefix':  'ctb',
                     'ctbStartIdx': 0,
                     'ctbNum':     10,
-                    'rowsPerTbl': 10000,
+                    'rowsPerTbl': 1000,
                     'batchNum':   10,
                     'startTs':    1640966400000,  # 2022-01-01 00:00:00.000
                     'pollDelay':  60,
@@ -203,7 +203,7 @@ class TDTestCase:
         expectRows = 1
         resultList = tmqCom.selectConsumeResult(expectRows)
 
-        if expectrowcnt / 2 >= resultList[0]:
+        if expectrowcnt / 2 > resultList[0]:
             tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectrowcnt / 2, resultList[0]))
             tdLog.exit("%d tmq consume rows error!"%consumerId)
 

tests/system-test/8-stream/stream_basic.py

@@ -0,0 +1,110 @@
+###################################################################
+#           Copyright (c) 2016 by TAOS Technologies, Inc.
+#                     All rights reserved.
+#
+#  This file is proprietary and confidential to TAOS Technologies.
+#  No part of this file may be reproduced, stored, transmitted,
+#  disclosed or used in any form or by any means other than as
+#  expressly provided by the written permission from Jianhui Tao
+#
+###################################################################
+
+# -*- coding: utf-8 -*-
+
+
+from util.log import *
+from util.cases import *
+from util.sql import *
+from util.common import *
+from util.sqlset import *
+from util.autogen import *
+
+import random
+import time
+import traceback
+import os
+from   os import path
+
+
+class TDTestCase:
+    # init 
+    def init(self, conn, logSql, replicaVar=1):
+        self.replicaVar = int(replicaVar)
+        tdLog.debug("start to execute %s" % __file__)
+        tdSql.init(conn.cursor(), True)
+        
+        # autoGen
+        self.autoGen = AutoGen()
+                        
+    def waitTranslation(self, waitSeconds):
+        # wait end
+        for i in range(waitSeconds):
+            sql ="show transactions;"
+            rows = tdSql.query(sql)
+            if rows == 0:
+               return True
+            tdLog.info(f"i={i} wait for translation finish ...")
+            time.sleep(1)
+
+        return False
+
+    def getPath(self, tool="taosBenchmark"):
+        if (platform.system().lower() == 'windows'):
+            tool = tool + ".exe"
+        selfPath = os.path.dirname(os.path.realpath(__file__))
+
+        if ("community" in selfPath):
+            projPath = selfPath[:selfPath.find("community")]
+        else:
+            projPath = selfPath[:selfPath.find("tests")]
+
+        paths = []
+        for root, dirs, files in os.walk(projPath):
+            if ((tool) in files):
+                rootRealPath = os.path.dirname(os.path.realpath(root))
+                if ("packaging" not in rootRealPath):
+                    paths.append(os.path.join(root, tool))
+                    break
+        if (len(paths) == 0):
+            tdLog.exit("taosBenchmark not found!")
+            return
+        else:
+            tdLog.info("taosBenchmark found in %s" % paths[0])
+            return paths[0]
+
+    def taosBenchmark(self, param):
+        binPath = self.getPath()
+        cmd = f"{binPath} {param}"
+        tdLog.info(cmd)
+        os.system(cmd)
+
+    # run
+    def run(self):
+        # gen data
+        random.seed(int(time.time()))
+        self.taosBenchmark(" -d db -t 2 -v 2 -n 1000000 -y")
+        # create stream
+        tdSql.execute("use db")
+        tdSql.execute("create stream stream1 fill_history 1 into sta as select count(*) as cnt from meters interval(10a);",show=True)
+        sql = "select count(*) from sta"
+        # loop wait max 60s to check count is ok
+        tdLog.info("loop wait result ...")
+        tdSql.checkDataLoop(0, 0, 99999, sql, loopCount=120, waitTime=0.5)
+
+        # check all data is correct
+        sql = "select * from sta where cnt != 20;"
+        tdSql.query(sql)
+        tdSql.checkRows(0)
+
+        # check ts interval is correct
+        sql = "select * from ( select diff(_wstart) as tsdif from sta ) where tsdif != 10;"
+        tdSql.query(sql)
+        tdSql.checkRows(0)
+
+    # stop
+    def stop(self):
+        tdSql.close()
+        tdLog.success("%s successfully executed" % __file__)
+
+
+tdCases.addLinux(__file__, TDTestCase())