Merge branch '3.0' into test/jcy

cmake/taosadapter_CMakeLists.txt.in

@@ -2,7 +2,7 @@
 # taosadapter
 ExternalProject_Add(taosadapter
         GIT_REPOSITORY https://github.com/taosdata/taosadapter.git
-        GIT_TAG a11131c
+        GIT_TAG 8c3d57d
         SOURCE_DIR "${TD_SOURCE_DIR}/tools/taosadapter"
         BINARY_DIR ""
         #BUILD_IN_SOURCE TRUE

cmake/taostools_CMakeLists.txt.in

@@ -2,7 +2,7 @@
 # taos-tools
 ExternalProject_Add(taos-tools
         GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
-        GIT_TAG 16eb34f
+        GIT_TAG 0fb640b
         SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
         BINARY_DIR ""
         #BUILD_IN_SOURCE TRUE

docs/en/07-develop/03-insert-data/02-influxdb-line.mdx

@@ -37,9 +37,9 @@ meters,location=California.LosAngeles,groupid=2 current=13.4,voltage=223,phase=0
 - All the data in `tag_set` will be converted to NCHAR type automatically .
 - Each data in `field_set` must be self-descriptive for its data type. For example 1.2f32 means a value 1.2 of float type. Without the "f" type suffix, it will be treated as type double.
 - Multiple kinds of precision can be used for the `timestamp` field. Time precision can be from nanosecond (ns) to hour (h).
-- You can configure smlChildTableName in taos.cfg to specify table names, for example, `smlChildTableName=tname`. You can insert `st,tname=cpul,t1=4 c1=3 1626006833639000000` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
+- The child table name is created automatically in a rule to guarantee its uniqueness. But you can configure `smlChildTableName` in taos.cfg to specify a tag value as the table names if the tag value is unique globally. For example, if a tag is called `tname` and you set `smlChildTableName=tname` in taos.cfg, when you insert `st,tname=cpu1,t1=4 c1=3 1626006833639000000`, the child table `cpu1` will be created automatically. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
 - It is assumed that the order of field_set in a supertable is consistent, meaning that the first record contains all fields and subsequent records store fields in the same order. If the order is not consistent, set smlDataFormat in taos.cfg to false. Otherwise, data will be written out of order and a database error will occur.(smlDataFormat in taos.cfg default to false after version of 3.0.1.3)
-  :::
+:::
 
 For more details please refer to [InfluxDB Line Protocol](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/) and [TDengine Schemaless](/reference/schemaless/#Schemaless-Line-Protocol)
 

docs/en/07-develop/03-insert-data/03-opentsdb-telnet.mdx

@@ -32,7 +32,7 @@ For example:
 meters.current 1648432611250 11.3 location=California.LosAngeles groupid=3
 ```
 
-- The defult child table name is generated by rules.You can configure smlChildTableName in taos.cfg to specify child table names, for example, `smlChildTableName=tname`. You can insert `meters.current 1648432611250 11.3 tname=cpu1 location=California.LosAngeles groupid=3` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
+- The child table name is created automatically in a rule to guarantee its uniqueness. But you can configure `smlChildTableName` in taos.cfg to specify a tag value as the table names if the tag value is unique globally. For example, if a tag is called `tname` and you set `smlChildTableName=tname` in taos.cfg, when you insert `st,tname=cpu1,t1=4 c1=3 1626006833639000000`, the child table `cpu1` will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
   Please refer to [OpenTSDB Telnet API](http://opentsdb.net/docs/build/html/api_telnet/put.html) for more details.
 
 ## Examples

docs/en/07-develop/03-insert-data/04-opentsdb-json.mdx

@@ -48,8 +48,8 @@ Please refer to [OpenTSDB HTTP API](http://opentsdb.net/docs/build/html/api_http
 
 - In JSON protocol, strings will be converted to NCHAR type and numeric values will be converted to double type.
 - Only data in array format is accepted and so an array must be used even if there is only one row.
-- The defult child table name is generated by rules.You can configure smlChildTableName in taos.cfg to specify child table names, for example, `smlChildTableName=tname`. You can insert `"tags": { "host": "web02","dc": "lga","tname":"cpu1"}` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
+- The child table name is created automatically in a rule to guarantee its uniqueness. But you can configure `smlChildTableName` in taos.cfg to specify a tag value as the table names if the tag value is unique globally. For example, if a tag is called `tname` and you set `smlChildTableName=tname` in taos.cfg, when you insert `st,tname=cpu1,t1=4 c1=3 1626006833639000000`, the child table `cpu1` will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
-  :::
+:::
 
 ## Examples
 

docs/en/07-develop/09-udf.md

@@ -218,9 +218,9 @@ After compiling your function into a DLL, you add it to TDengine. For more infor
 
 ## Sample Code
 
-### Sample scalar function: [bit_and](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/bit_and.c)
+### Sample scalar function: [bit_and](https://github.com/taosdata/TDengine/blob/3.0/tests/script/sh/bit_and.c)
 
-The bit_add function implements bitwise addition for multiple columns. If there is only one column, the column is returned. The bit_add function ignores null values.
+The bit_and function implements bitwise addition for multiple columns. If there is only one column, the column is returned. The bit_and function ignores null values.
 
 <details>
 <summary>bit_and.c</summary>
@@ -231,7 +231,7 @@ The bit_add function implements bitwise addition for multiple columns. If there
 
 </details>
 
-### Sample aggregate function: [l2norm](https://github.com/taosdata/TDengine/blob/develop/tests/script/sh/l2norm.c)
+### Sample aggregate function: [l2norm](https://github.com/taosdata/TDengine/blob/3.0/tests/script/sh/l2norm.c)
 
 The l2norm function finds the second-order norm for all data in the input column. This squares the values, takes a cumulative sum, and finds the square root.
 

docs/en/14-reference/05-taosbenchmark.md

@@ -217,7 +217,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
 The parameters listed in this section apply to all function modes.
 
 - **filetype** : The function to be tested, with optional values `insert`, `query` and `subscribe`. These correspond to the insert, query, and subscribe functions, respectively. Users can specify only one of these in each configuration file.
-**cfgdir**: specify the TDengine cluster configuration file's directory. The default path is /etc/taos.
+**cfgdir**: specify the TDengine client configuration file's directory. The default path is /etc/taos.
 
 - **host**: Specify the FQDN of the TDengine server to connect. The default value is `localhost`.
 

docs/en/14-reference/12-config/index.md

@@ -106,7 +106,7 @@ The parameters described in this document by the effect that they have on the sy
 | Applicable | Server only                                           |
 | Meaning       | The switch for monitoring inside server. The main object of monitoring is to collect information about load on physical nodes, including CPU usage, memory usage, disk usage, and network bandwidth. Monitoring information is sent over HTTP to the taosKeeper service specified by `monitorFqdn` and `monitorProt`.
 | Value Range   | 0: monitoring disabled, 1: monitoring enabled                                                                                                                                                                |
-| Default   | 1                                                                                                                                                                                                  |
+| Default   | 0                                                                                                                                                                                                  |
 
 ### monitorFqdn
 
@@ -735,19 +735,17 @@ To prevent system resource from being exhausted by multiple concurrent streams,
 | 44  | numOfVnodeQueryThreads | No | Yes |
 | 45  | numOfVnodeStreamThreads | No | Yes |
 | 46  | numOfVnodeFetchThreads | No | Yes |
-| 47  | numOfVnodeWriteThreads | No | Yes |
+| 47  | numOfVnodeRsmaThreads | No | Yes | 
-| 48  | numOfVnodeSyncThreads | No | Yes |
+| 48  | numOfQnodeQueryThreads | No | Yes |
-| 49  | numOfVnodeRsmaThreads | No | Yes | 
+| 49  | numOfQnodeFetchThreads | No | Yes |
-| 50  | numOfQnodeQueryThreads | No | Yes |
+| 50  | numOfSnodeSharedThreads | No | Yes |
-| 51  | numOfQnodeFetchThreads | No | Yes |
+| 51  | numOfSnodeUniqueThreads | No | Yes |
-| 52  | numOfSnodeSharedThreads | No | Yes |
+| 52  | rpcQueueMemoryAllowed | No | Yes |
-| 53  | numOfSnodeUniqueThreads | No | Yes |
+| 53  | logDir | Yes | Yes |
-| 54  | rpcQueueMemoryAllowed | No | Yes |
+| 54  | minimalLogDirGB | Yes | Yes |
-| 55  | logDir | Yes | Yes |
+| 55  | numOfLogLines | Yes | Yes |
-| 56  | minimalLogDirGB | Yes | Yes |
+| 56  | asyncLog | Yes | Yes |
-| 57  | numOfLogLines | Yes | Yes |
+| 57  | logKeepDays | Yes | Yes |
-| 58  | asyncLog | Yes | Yes |
-| 59  | logKeepDays | Yes | Yes |
 | 60  | debugFlag | Yes | Yes |
 | 61  | tmrDebugFlag | Yes | Yes |
 | 62  | uDebugFlag | Yes | Yes |

docs/en/28-releases/01-tdengine.md

@@ -6,6 +6,10 @@ description: TDengine release history, Release Notes and download links.
 
 import Release from "/components/ReleaseV3";
 
+## 3.0.1.6
+
+<Release type="tdengine" version="3.0.1.6" />
+
 ## 3.0.1.5
 
 <Release type="tdengine" version="3.0.1.5" />

docs/en/28-releases/02-tools.md

@@ -6,6 +6,10 @@ description: taosTools release history, Release Notes, download links.
 
 import Release from "/components/ReleaseV3";
 
+## 2.2.7
+
+<Release type="tools" version="2.2.7" />
+
 ## 2.2.6
 
 <Release type="tools" version="2.2.6" />

docs/examples/csharp/wsConnect/Program.cs

@@ -9,9 +9,10 @@ namespace Examples
         {
             string DSN = "ws://root:taosdata@127.0.0.1:6041/test";
             IntPtr wsConn = LibTaosWS.WSConnectWithDSN(DSN);
+  
             if (wsConn == IntPtr.Zero)
             {
-                throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
+                throw new Exception("get WS connection failed");
             }
             else
             {

docs/examples/csharp/wsInsert/Program.cs

@@ -13,7 +13,7 @@ namespace Examples
             // Assert if connection is validate
             if (wsConn == IntPtr.Zero)
             {
-                throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
+                throw new Exception("get WS connection failed");
             }
             else
             {

docs/examples/csharp/wsQuery/Program.cs

@@ -13,7 +13,7 @@ namespace Examples
             IntPtr wsConn = LibTaosWS.WSConnectWithDSN(DSN);
             if (wsConn == IntPtr.Zero)
             {
-                throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
+                throw new Exception("get WS connection failed");
             }
             else
             {

docs/examples/csharp/wsStmt/Program.cs

@@ -21,7 +21,7 @@ namespace Examples
             IntPtr wsConn = LibTaosWS.WSConnectWithDSN(DSN);
             if (wsConn == IntPtr.Zero)
             {
-                throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
+                throw new Exception($"get WS connection failed");
             }
             else
             {

docs/zh/07-develop/03-insert-data/02-influxdb-line.mdx

@@ -38,9 +38,8 @@ meters,location=California.LosAngeles,groupid=2 current=13.4,voltage=223,phase=0
 - field_set 中的每个数据项都需要对自身的数据类型进行描述, 比如 1.2f32 代表 FLOAT 类型的数值 1.2, 如果不带类型后缀会被当作 DOUBLE 处理;
 - timestamp 支持多种时间精度。写入数据的时候需要用参数指定时间精度，支持从小时到纳秒的 6 种时间精度。
 - 为了提高写入的效率，默认假设同一个超级表中 field_set 的顺序是一样的（第一条数据包含所有的 field，后面的数据按照这个顺序），如果顺序不一样，需要配置参数 smlDataFormat 为 false，否则，数据写入按照相同顺序写入，库中数据会异常。（3.0.1.3 之后的版本 smlDataFormat 默认为 false） [TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
-- 默认生产的子表名是根据规则生成的唯一 ID 值。为了让用户可以指定生成的表名，可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定。
+- 默认产生的子表名是根据规则生成的唯一 ID 值。用户也可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定某个标签值作为子表名。该标签值应该具有全局唯一性。举例如下：假设有个标签名为tname, 配置 smlChildTableName=tname, 插入数据为 st,tname=cpu1,t1=4 c1=3 1626006833639000000 则创建的子表名为 cpu1。注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。[TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
-  举例如下：配置 smlChildTableName=tname 插入数据为 st,tname=cpu1,t1=4 c1=3 1626006833639000000 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。[TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
+:::
-  :::
 
 要了解更多可参考：[InfluxDB Line 协议官方文档](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/) 和 [TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
 

docs/zh/07-develop/03-insert-data/03-opentsdb-telnet.mdx

@@ -32,8 +32,7 @@ OpenTSDB 行协议同样采用一行字符串来表示一行数据。OpenTSDB
 meters.current 1648432611250 11.3 location=California.LosAngeles groupid=3
 ```
 
-- 默认生产的子表名是根据规则生成的唯一 ID 值。为了让用户可以指定生成的表名，可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定。
+- 默认生产的子表名是根据规则生成的唯一 ID 值。用户也可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定某个标签值作为子表名。该标签值应该具有全局唯一性。举例如下：假设有个标签名为tname, 配置 smlChildTableName=tname, 插入数据为 meters.current 1648432611250 11.3 tname=cpu1 location=California.LosAngeles groupid=3 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
-  举例如下：配置 smlChildTableName=tname 插入数据为 meters.current 1648432611250 11.3 tname=cpu1 location=California.LosAngeles groupid=3 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
   参考 [OpenTSDB Telnet API 文档](http://opentsdb.net/docs/build/html/api_telnet/put.html)。
 
 ## 示例代码

docs/zh/07-develop/03-insert-data/04-opentsdb-json.mdx

@@ -48,9 +48,8 @@ OpenTSDB JSON 格式协议采用一个 JSON 字符串表示一行或多行数据
 
 - 对于 JSON 格式协议，TDengine 并不会自动把所有标签转成 NCHAR 类型， 字符串将将转为 NCHAR 类型， 数值将同样转换为 DOUBLE 类型。
 - TDengine 只接收 JSON **数组格式**的字符串，即使一行数据也需要转换成数组形式。
-- 默认生产的子表名是根据规则生成的唯一 ID 值。为了让用户可以指定生成的表名，可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定。
+- 默认生成的子表名是根据规则生成的唯一 ID 值。用户也可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定某个标签值作为子表名。该标签值应该具有全局唯一性。举例如下：假设有个标签名为tname, 配置 smlChildTableName=tname, 插入数据为 `"tags": { "host": "web02","dc": "lga","tname":"cpu1"}` 则创建的子表名为 cpu1。注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
-  举例如下：配置 smlChildTableName=tname 插入数据为 `"tags": { "host": "web02","dc": "lga","tname":"cpu1"}` 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
+:::
-  :::
 
 ## 示例代码
 

docs/zh/10-deployment/01-deploy.md

@@ -190,3 +190,16 @@ DROP DNODE dnodeId;
 dnodeID 是集群自动分配的，不得人工指定。它在生成时是递增的，不会重复。
 
 :::
+
+## 常见问题
+
+1、建立集群时使用 CREATE DNODE 增加新节点后，新节点始终显示 offline 状态？
+```sql
+  1）首先要检查增加的新节点上的 taosd 服务是否已经正常启动
+  
+  2）如果已经启动，再检查到新节点的网络是否通畅，可以使用 ping fqdn 验证下
+  
+  3）如果前面两步都没有问题，这一步要检查新节点做为独立集群在运行了，可以使用 taos -h fqdn 连接上后，show dnodes; 命令查看.
+    如果显示的列表与你主节点上显示的不一致，说明此节点自己单独成立了一个集群，解决的方法是停止新节点上的服务，然后清空新节点上 
+    taos.cfg 中配置的 dataDir 目录下的所有文件，重新启动新节点服务即可解决。
+```  

docs/zh/14-reference/05-taosbenchmark.md

@@ -217,7 +217,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
 本节所列参数适用于所有功能模式。
 
 - **filetype** : 要测试的功能，可选值为 `insert`, `query` 和 `subscribe`。分别对应插入、查询和订阅功能。每个配置文件中只能指定其中之一。
-- **cfgdir** : TDengine 集群配置文件所在的目录，默认路径是 /etc/taos 。
+- **cfgdir** : TDengine 客户端配置文件所在的目录，默认路径是 /etc/taos 。
 
 - **host** : 指定要连接的 TDengine 服务端的 FQDN，默认值为 localhost。
 

docs/zh/14-reference/12-config/index.md

@@ -106,7 +106,7 @@ taos --dump-config
 | 适用范围 | 仅服务端适用                                                                                                                                                                                       |
 | 含义     | 服务器内部的系统监控开关。监控主要负责收集物理节点的负载状况，包括 CPU、内存、硬盘、网络带宽的监控记录，监控信息将通过 HTTP 协议发送给由 `monitorFqdn` 和 `monitorProt` 指定的 TaosKeeper 监控服务 |
 | 取值范围 | 0：关闭监控服务， 1：激活监控服务。                                                                                                                                                                |
-| 缺省值   | 1                                                                                                                                                                                                  |
+| 缺省值   | 0                                                                                                                                                                                                  |
 
 ### monitorFqdn
 
@@ -711,19 +711,17 @@ charset 的有效值是 UTF-8。
 | 44  | numOfVnodeQueryThreads  | 否              | 是              |                                                   |
 | 45  | numOfVnodeStreamThreads | 否              | 是              |                                                   |
 | 46  | numOfVnodeFetchThreads  | 否              | 是              |                                                   |
-| 47  | numOfVnodeWriteThreads  | 否              | 是              |                                                   |
+| 47  |  numOfVnodeRsmaThreads  | 否              | 是              |                                                   |
-| 48  |  numOfVnodeSyncThreads  | 否              | 是              |                                                   |
+| 48  | numOfQnodeQueryThreads  | 否              | 是              |                                                   |
-| 49  |  numOfVnodeRsmaThreads  | 否              | 是              |                                                   |
+| 49  | numOfQnodeFetchThreads  | 否              | 是              |                                                   |
-| 50  | numOfQnodeQueryThreads  | 否              | 是              |                                                   |
+| 50  | numOfSnodeSharedThreads | 否              | 是              |                                                   |
-| 51  | numOfQnodeFetchThreads  | 否              | 是              |                                                   |
+| 51  | numOfSnodeUniqueThreads | 否              | 是              |                                                   |
-| 52  | numOfSnodeSharedThreads | 否              | 是              |                                                   |
+| 52  |  rpcQueueMemoryAllowed  | 否              | 是              |                                                   |
-| 53  | numOfSnodeUniqueThreads | 否              | 是              |                                                   |
+| 53  |         logDir          | 是              | 是              |                                                   |
-| 54  |  rpcQueueMemoryAllowed  | 否              | 是              |                                                   |
+| 54  |     minimalLogDirGB     | 是              | 是              |                                                   |
-| 55  |         logDir          | 是              | 是              |                                                   |
+| 55  |      numOfLogLines      | 是              | 是              |                                                   |
-| 56  |     minimalLogDirGB     | 是              | 是              |                                                   |
+| 56  |        asyncLog         | 是              | 是              |                                                   |
-| 57  |      numOfLogLines      | 是              | 是              |                                                   |
+| 57  |       logKeepDays       | 是              | 是              |                                                   |
-| 58  |        asyncLog         | 是              | 是              |                                                   |
-| 59  |       logKeepDays       | 是              | 是              |                                                   |
 | 60  |        debugFlag        | 是              | 是              |                                                   |
 | 61  |      tmrDebugFlag       | 是              | 是              |                                                   |
 | 62  |       uDebugFlag        | 是              | 是              |                                                   |

docs/zh/28-releases/01-tdengine.md

@@ -6,6 +6,10 @@ description: TDengine 发布历史、Release Notes 及下载链接
 
 import Release from "/components/ReleaseV3";
 
+## 3.0.1.6
+
+<Release type="tdengine" version="3.0.1.6" />
+
 ## 3.0.1.5
 
 <Release type="tdengine" version="3.0.1.5" />

docs/zh/28-releases/02-tools.md

@@ -6,6 +6,10 @@ description: taosTools 的发布历史、Release Notes 和下载链接
 
 import Release from "/components/ReleaseV3";
 
+## 2.2.7
+
+<Release type="tools" version="2.2.7" />
+
 ## 2.2.6
 
 <Release type="tools" version="2.2.6" />

include/common/tcommon.h

@@ -225,13 +225,13 @@ typedef struct SVarColAttr {
 // pBlockAgg->numOfNull == info.rows, all data are null
 // pBlockAgg->numOfNull == 0, no data are null.
 typedef struct SColumnInfoData {
-  SColumnInfo info;     // column info
-  bool        hasNull;  // if current column data has null value.
   char*         pData;    // the corresponding block data in memory
   union {
     char*       nullbitmap;  // bitmap, one bit for each item in the list
     SVarColAttr varmeta;
   };
+  SColumnInfo info;     // column info
+  bool        hasNull;  // if current column data has null value.
 } SColumnInfoData;
 
 typedef struct SQueryTableDataCond {

include/common/tglobal.h

@@ -55,8 +55,6 @@ extern int32_t tsNumOfMnodeReadThreads;
 extern int32_t tsNumOfVnodeQueryThreads;
 extern int32_t tsNumOfVnodeStreamThreads;
 extern int32_t tsNumOfVnodeFetchThreads;
-extern int32_t tsNumOfVnodeWriteThreads;
-extern int32_t tsNumOfVnodeSyncThreads;
 extern int32_t tsNumOfVnodeRsmaThreads;
 extern int32_t tsNumOfQnodeQueryThreads;
 extern int32_t tsNumOfQnodeFetchThreads;

include/common/tmsg.h

@@ -216,9 +216,14 @@ typedef struct SEp {
   uint16_t port;
 } SEp;
 
+#define SHOW_REWRITE_MASK() (1 << 0)
+
+#define TEST_SHOW_REWRITE_MASK(m) (((m) & SHOW_REWRITE_MASK()) != 0)
+
 typedef struct {
   int32_t contLen;
   int32_t vgId;
+  int32_t msgMask;
 } SMsgHead;
 
 // Submit message for one table

include/common/tmsgdef.h

@@ -241,6 +241,7 @@ enum {
   TD_DEF_MSG_TYPE(TDMT_STREAM_UNUSED1, "stream-unused1", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_RETRIEVE, "stream-retrieve", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_RECOVER_FINISH, "vnode-stream-finish", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_CHECK, "vnode-stream-task-check", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_MAX_MSG, "stream-max", NULL, NULL)
 
   TD_NEW_MSG_SEG(TDMT_MON_MSG)

include/libs/stream/tstream.h

@@ -46,7 +46,7 @@ enum {
   TASK_STATUS__DROPPING,
   TASK_STATUS__FAIL,
   TASK_STATUS__STOP,
-  TASK_STATUS__RECOVER_DOWNSTREAM,
+  TASK_STATUS__WAIT_DOWNSTREAM,
   TASK_STATUS__RECOVER_PREPARE,
   TASK_STATUS__RECOVER1,
   TASK_STATUS__RECOVER2,
@@ -332,7 +332,10 @@ typedef struct SStreamTask {
   SStreamState* pState;
 
   // do not serialize
-  int32_t recoverWaitingChild;
+  int32_t recoverTryingDownstream;
+  int32_t recoverWaitingUpstream;
+  int64_t checkReqId;
+  SArray* checkReqIds;  // shuffle
 
 } SStreamTask;
 
@@ -418,7 +421,10 @@ typedef struct {
 
 typedef struct {
   int64_t streamId;
-  int32_t taskId;
+  int32_t upstreamNodeId;
+  int32_t upstreamTaskId;
+  int32_t downstreamNodeId;
+  int32_t downstreamTaskId;
   int8_t  inputStatus;
 } SStreamDispatchRsp;
 
@@ -440,6 +446,27 @@ typedef struct {
   int32_t rspToTaskId;
 } SStreamRetrieveRsp;
 
+typedef struct {
+  int64_t reqId;
+  int64_t streamId;
+  int32_t upstreamNodeId;
+  int32_t upstreamTaskId;
+  int32_t downstreamNodeId;
+  int32_t downstreamTaskId;
+  int32_t childId;
+} SStreamTaskCheckReq;
+
+typedef struct {
+  int64_t reqId;
+  int64_t streamId;
+  int32_t upstreamNodeId;
+  int32_t upstreamTaskId;
+  int32_t downstreamNodeId;
+  int32_t downstreamTaskId;
+  int32_t childId;
+  int8_t  status;
+} SStreamTaskCheckRsp;
+
 typedef struct {
   SMsgHead msgHead;
   int64_t  streamId;
@@ -455,47 +482,6 @@ typedef struct {
 int32_t tEncodeSStreamRecoverFinishReq(SEncoder* pEncoder, const SStreamRecoverFinishReq* pReq);
 int32_t tDecodeSStreamRecoverFinishReq(SDecoder* pDecoder, SStreamRecoverFinishReq* pReq);
 
-#if 0
-typedef struct {
-  int64_t streamId;
-  int32_t taskId;
-  int32_t upstreamTaskId;
-  int32_t upstreamNodeId;
-} SStreamTaskRecoverReq;
-
-typedef struct {
-  int64_t streamId;
-  int32_t rspTaskId;
-  int32_t reqTaskId;
-  int8_t  inputStatus;
-} SStreamTaskRecoverRsp;
-
-int32_t tEncodeStreamTaskRecoverReq(SEncoder* pEncoder, const SStreamTaskRecoverReq* pReq);
-int32_t tDecodeStreamTaskRecoverReq(SDecoder* pDecoder, SStreamTaskRecoverReq* pReq);
-
-int32_t tEncodeStreamTaskRecoverRsp(SEncoder* pEncoder, const SStreamTaskRecoverRsp* pRsp);
-int32_t tDecodeStreamTaskRecoverRsp(SDecoder* pDecoder, SStreamTaskRecoverRsp* pRsp);
-
-typedef struct {
-  int64_t streamId;
-  int32_t taskId;
-} SMStreamTaskRecoverReq;
-
-typedef struct {
-  int64_t streamId;
-  int32_t taskId;
-} SMStreamTaskRecoverRsp;
-
-int32_t tEncodeSMStreamTaskRecoverReq(SEncoder* pEncoder, const SMStreamTaskRecoverReq* pReq);
-int32_t tDecodeSMStreamTaskRecoverReq(SDecoder* pDecoder, SMStreamTaskRecoverReq* pReq);
-
-int32_t tEncodeSMStreamTaskRecoverRsp(SEncoder* pEncoder, const SMStreamTaskRecoverRsp* pRsp);
-int32_t tDecodeSMStreamTaskRecoverRsp(SDecoder* pDecoder, SMStreamTaskRecoverRsp* pRsp);
-
-int32_t streamProcessRecoverReq(SStreamTask* pTask, SStreamTaskRecoverReq* pReq, SRpcMsg* pMsg);
-int32_t streamProcessRecoverRsp(SStreamTask* pTask, SStreamTaskRecoverRsp* pRsp);
-#endif
-
 typedef struct {
   int64_t streamId;
   int32_t downstreamTaskId;
@@ -509,20 +495,18 @@ typedef struct {
   SArray* checkpointVer;  // SArray<SStreamCheckpointInfo>
 } SStreamRecoverDownstreamRsp;
 
+int32_t tEncodeSStreamTaskCheckReq(SEncoder* pEncoder, const SStreamTaskCheckReq* pReq);
+int32_t tDecodeSStreamTaskCheckReq(SDecoder* pDecoder, SStreamTaskCheckReq* pReq);
+
+int32_t tEncodeSStreamTaskCheckRsp(SEncoder* pEncoder, const SStreamTaskCheckRsp* pRsp);
+int32_t tDecodeSStreamTaskCheckRsp(SDecoder* pDecoder, SStreamTaskCheckRsp* pRsp);
+
 int32_t tEncodeSStreamTaskRecoverReq(SEncoder* pEncoder, const SStreamRecoverDownstreamReq* pReq);
 int32_t tDecodeSStreamTaskRecoverReq(SDecoder* pDecoder, SStreamRecoverDownstreamReq* pReq);
 
 int32_t tEncodeSStreamTaskRecoverRsp(SEncoder* pEncoder, const SStreamRecoverDownstreamRsp* pRsp);
 int32_t tDecodeSStreamTaskRecoverRsp(SDecoder* pDecoder, SStreamRecoverDownstreamRsp* pRsp);
 
-typedef struct {
-  int64_t streamId;
-  int32_t taskId;
-  int32_t waitingRspCnt;
-  int32_t totReq;
-  SArray* info;  // SArray<SArray<SStreamCheckpointInfo>*>
-} SStreamRecoverStatus;
-
 int32_t tDecodeStreamDispatchReq(SDecoder* pDecoder, SStreamDispatchReq* pReq);
 int32_t tDecodeStreamRetrieveReq(SDecoder* pDecoder, SStreamRetrieveReq* pReq);
 void    tDeleteStreamRetrieveReq(SStreamRetrieveReq* pReq);
@@ -533,7 +517,7 @@ int32_t streamSetupTrigger(SStreamTask* pTask);
 
 int32_t streamProcessRunReq(SStreamTask* pTask);
 int32_t streamProcessDispatchReq(SStreamTask* pTask, SStreamDispatchReq* pReq, SRpcMsg* pMsg, bool exec);
-int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp);
+int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, int32_t code);
 
 int32_t streamProcessRetrieveReq(SStreamTask* pTask, SStreamRetrieveReq* pReq, SRpcMsg* pMsg);
 int32_t streamProcessRetrieveRsp(SStreamTask* pTask, SStreamRetrieveRsp* pRsp);
@@ -544,6 +528,10 @@ int32_t streamSchedExec(SStreamTask* pTask);
 int32_t streamScanExec(SStreamTask* pTask, int32_t batchSz);
 
 // recover and fill history
+int32_t streamTaskCheckDownstream(SStreamTask* pTask, int64_t version);
+int32_t streamTaskLaunchRecover(SStreamTask* pTask, int64_t version);
+int32_t streamProcessTaskCheckReq(SStreamTask* pTask, const SStreamTaskCheckReq* pReq);
+int32_t streamProcessTaskCheckRsp(SStreamTask* pTask, const SStreamTaskCheckRsp* pRsp, int64_t version);
 // common
 int32_t streamSetParamForRecover(SStreamTask* pTask);
 int32_t streamRestoreParam(SStreamTask* pTask);

include/libs/sync/sync.h

@@ -219,6 +219,7 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex);
 int32_t syncEndSnapshot(int64_t rid);
 int32_t syncLeaderTransfer(int64_t rid);
 int32_t syncStepDown(int64_t rid, SyncTerm newTerm);
+bool    syncIsReadyForRead(int64_t rid);
 
 SSyncState  syncGetState(int64_t rid);
 void        syncGetRetryEpSet(int64_t rid, SEpSet* pEpSet);

include/os/osDef.h

@@ -244,16 +244,6 @@ void syslog(int unused, const char *format, ...);
 #define TD_CHARSET_LEN  64
 #define TD_TIMEZONE_LEN 96
 
-#ifdef WINDOWS
-#define TD_PATH_MAX 260
-#elif defined(PATH_MAX)
-#define TD_PATH_MAX PATH_MAX
-#elif defined(_XOPEN_PATH_MAX)
-#define TD_PATH_MAX _XOPEN_PATH_MAX
-#else
-#define TD_PATH_MAX _POSIX_PATH_MAX
-#endif
-
 #ifdef __cplusplus
 }
 #endif

include/os/osFile.h

@@ -42,6 +42,16 @@ extern "C" {
 #define PATH_MAX 256
 #endif
 
+#ifdef WINDOWS
+#define TD_PATH_MAX _MAX_PATH
+#elif defined(PATH_MAX)
+#define TD_PATH_MAX PATH_MAX
+#elif defined(_XOPEN_PATH_MAX)
+#define TD_PATH_MAX _XOPEN_PATH_MAX
+#else
+#define TD_PATH_MAX _POSIX_PATH_MAX
+#endif
+
 typedef struct TdFile *TdFilePtr;
 
 #define TD_FILE_CREATE   0x0001

include/util/taoserror.h

@@ -612,9 +612,12 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_TMQ_CONSUMER_MISMATCH          TAOS_DEF_ERROR_CODE(0, 0x4001)
 #define TSDB_CODE_TMQ_CONSUMER_CLOSED            TAOS_DEF_ERROR_CODE(0, 0x4002)
 
+// stream
+#define TSDB_CODE_STREAM_TASK_NOT_EXIST          TAOS_DEF_ERROR_CODE(0, 0x4100)
+
 // TDLite
-#define TSDB_CODE_TDLITE_IVLD_OPEN_FLAGS         TAOS_DEF_ERROR_CODE(0, 0x4100)
+#define TSDB_CODE_TDLITE_IVLD_OPEN_FLAGS         TAOS_DEF_ERROR_CODE(0, 0x5100)
-#define TSDB_CODE_TDLITE_IVLD_OPEN_DIR           TAOS_DEF_ERROR_CODE(0, 0x4101)
+#define TSDB_CODE_TDLITE_IVLD_OPEN_DIR           TAOS_DEF_ERROR_CODE(0, 0x5101)
 
 #ifdef __cplusplus
 }

include/util/tqueue.h

@@ -59,6 +59,47 @@ typedef enum {
 typedef void (*FItem)(SQueueInfo *pInfo, void *pItem);
 typedef void (*FItems)(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfItems);
 
+typedef struct STaosQnode STaosQnode;
+
+typedef struct STaosQnode {
+  STaosQnode *next;
+  STaosQueue *queue;
+  int64_t     timestamp;
+  int32_t     size;
+  int8_t      itype;
+  int8_t      reserved[3];
+  char        item[];
+} STaosQnode;
+
+typedef struct STaosQueue {
+  STaosQnode   *head;
+  STaosQnode   *tail;
+  STaosQueue   *next;     // for queue set
+  STaosQset    *qset;     // for queue set
+  void         *ahandle;  // for queue set
+  FItem         itemFp;
+  FItems        itemsFp;
+  TdThreadMutex mutex;
+  int64_t       memOfItems;
+  int32_t       numOfItems;
+  int64_t       threadId;
+} STaosQueue;
+
+typedef struct STaosQset {
+  STaosQueue   *head;
+  STaosQueue   *current;
+  TdThreadMutex mutex;
+  tsem_t        sem;
+  int32_t       numOfQueues;
+  int32_t       numOfItems;
+} STaosQset;
+
+typedef struct STaosQall {
+  STaosQnode *current;
+  STaosQnode *start;
+  int32_t     numOfItems;
+} STaosQall;
+
 STaosQueue *taosOpenQueue();
 void        taosCloseQueue(STaosQueue *queue);
 void        taosSetQueueFp(STaosQueue *queue, FItem itemFp, FItems itemsFp);

include/util/tworker.h

@@ -26,8 +26,9 @@ typedef struct SQWorkerPool SQWorkerPool;
 typedef struct SWWorkerPool SWWorkerPool;
 
 typedef struct SQWorker {
-  int32_t       id;      // worker ID
+  int32_t       id;      // worker id
-  TdThread      thread;  // thread
+  int64_t       pid;     // thread pid
+  TdThread      thread;  // thread id
   SQWorkerPool *pool;
 } SQWorker;
 
@@ -43,9 +44,10 @@ typedef struct SQWorkerPool {
 
 typedef struct SWWorker {
   int32_t       id;      // worker id
-  TdThread      thread;  // thread
+  int64_t       pid;     // thread pid
+  TdThread      thread;  // thread id
   STaosQall    *qall;
-  STaosQset    *qset;  // queue set
+  STaosQset    *qset;
   SWWorkerPool *pool;
 } SWWorker;
 

packaging/release.bat

@@ -40,7 +40,7 @@ if not exist %work_dir%\debug\ver-%2-x86 (
 )
 cd %work_dir%\debug\ver-%2-x64
 call vcvarsall.bat x64
-cmake ../../ -G "NMake Makefiles JOM" -DCMAKE_MAKE_PROGRAM=jom -DBUILD_TOOLS=true -DBUILD_HTTP=false -DBUILD_TEST=false -DVERNUMBER=%2 -DCPUTYPE=x64
+cmake ../../ -G "NMake Makefiles JOM" -DCMAKE_MAKE_PROGRAM=jom -DBUILD_TOOLS=true -DWEBSOCKET=true -DBUILD_HTTP=false -DBUILD_TEST=false -DVERNUMBER=%2 -DCPUTYPE=x64
 cmake --build .
 rd /s /Q C:\TDengine
 cmake --install .

packaging/testpackage.sh

@@ -67,6 +67,7 @@ fi
 }
 
 
+
 function wgetFile {
 
 file=$1
@@ -75,7 +76,10 @@ if [ ! -f  ${file}  ];then
     echoColor  BD "wget https://www.taosdata.com/assets-download/3.0/${file}"
     wget https://www.taosdata.com/assets-download/3.0/${file}
 else
-    echoColor  YD "${file} already exists "
+    echoColor  YD "${file} already exists and use new file "
+    rm -rf ${file}
+    echoColor  BD "wget https://www.taosdata.com/assets-download/3.0/${file}"
+    wget https://www.taosdata.com/assets-download/3.0/${file}  
 fi
 }
 

packaging/tools/make_install.bat

@@ -34,7 +34,6 @@ if exist %binary_dir%\\test\\cfg\\taosadapter.toml (
         copy %binary_dir%\\test\\cfg\\taosadapter.toml %tagert_dir%\\cfg\\taosadapter.toml > nul
     )
 )
-
 copy %source_dir%\\include\\client\\taos.h %tagert_dir%\\include > nul
 copy %source_dir%\\include\\util\\taoserror.h %tagert_dir%\\include > nul
 copy %source_dir%\\include\\libs\\function\\taosudf.h %tagert_dir%\\include > nul
@@ -52,6 +51,7 @@ if exist %binary_dir%\\build\\lib\\taosws.dll.lib (
 )
 if exist %binary_dir%\\build\\lib\\taosws.dll (
     copy %binary_dir%\\build\\lib\\taosws.dll %tagert_dir%\\driver  > nul
+    copy %source_dir%\\tools\\taosws-rs\\target\\release\\taosws.h %tagert_dir%\\include > nul
 )
 if exist %binary_dir%\\build\\bin\\taosdump.exe (
     copy %binary_dir%\\build\\bin\\taosdump.exe %tagert_dir% > nul
@@ -67,3 +67,4 @@ if exist C:\\TDengine\\driver\\taosws.dll (
     copy /y C:\\TDengine\\driver\\taosws.dll C:\\Windows\\System32 > nul
 )
 sc query "taosd" >nul || sc create "taosd" binPath= "C:\\TDengine\\taosd.exe --win_service" start= DEMAND
+sc query "taosadapter" >nul || sc create "taosadapter" binPath= "C:\\TDengine\\taosadapter.exe" start= DEMAND

packaging/tools/makepkg.sh

@@ -285,10 +285,14 @@ if [[ $dbName == "taos" ]]; then
   fi
 
   # Add web files
-  if [ -d "${web_dir}/admin" ]; then
+  if [ "$verMode" == "cluster" ] || [ "$verMode" == "cloud" ]; then
+    if [ -d "${web_dir}/admin" ] ; then
       mkdir -p ${install_dir}/share/
       cp ${web_dir}/admin ${install_dir}/share/ -r
       cp ${web_dir}/png/taos.png ${install_dir}/share/admin/images/taos.png
+    else
+      echo "directory not found for enterprise release: ${web_dir}/admin"
+    fi
   fi
 fi
 

packaging/tools/tdengine.iss

@@ -15,10 +15,10 @@
 #define MyAppExeName "\*.exe"
 #define MyAppTaosExeName "\taos.bat"
 #define MyAppTaosdemoExeName "\taosBenchmark.exe"
-#define MyAppDLLName "\driver\taos.dll"
+#define MyAppDLLName "\driver\*.dll"
 ;#define MyAppVersion "3.0"
 ;#define MyAppInstallName "TDengine"
-
+;#define MyAppInstallName "TDengine"
 [Setup]
 VersionInfoVersion={#MyAppVersion}
 AppId={{A0F7A93C-79C4-485D-B2B8-F0D03DF42FAB}

source/client/src/clientTmq.c

@@ -1756,7 +1756,7 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) {
 
     rspObj = tmqHandleAllRsp(tmq, timeout, false);
     if (rspObj) {
-      tscDebug("consumer:%" PRId64 ", return rsp", tmq->consumerId);
+      tscDebug("consumer:%" PRId64 ", return rsp %p", tmq->consumerId, rspObj);
       return (TAOS_RES*)rspObj;
     } else if (terrno == TSDB_CODE_TQ_NO_COMMITTED_OFFSET) {
       tscDebug("consumer:%" PRId64 ", return null since no committed offset", tmq->consumerId);

source/client/test/clientTests.cpp

@@ -112,7 +112,7 @@ void createNewTable(TAOS* pConn, int32_t index) {
   }
   taos_free_result(pRes);
 
-  for(int32_t i = 0; i < 10000; i += 20) {
+  for(int32_t i = 0; i < 20; i += 20) {
     char sql[1024] = {0};
     sprintf(sql,
             "insert into tu%d values(now+%da, %d)(now+%da, %d)(now+%da, %d)(now+%da, %d)"
@@ -692,6 +692,7 @@ TEST(testCase, insert_test) {
   taos_free_result(pRes);
   taos_close(pConn);
 }
+#endif
 
 TEST(testCase, projection_query_tables) {
   TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
@@ -703,7 +704,7 @@ TEST(testCase, projection_query_tables) {
   //  }
   //  taos_free_result(pRes);
 
-  TAOS_RES* pRes = taos_query(pConn, "use benchmarkcpu");
+  TAOS_RES* pRes = taos_query(pConn, "use abc2");
   taos_free_result(pRes);
 
   pRes = taos_query(pConn, "create stable st1 (ts timestamp, k int) tags(a int)");
@@ -725,7 +726,7 @@ TEST(testCase, projection_query_tables) {
   }
   taos_free_result(pRes);
 
-  for (int32_t i = 0; i < 2; ++i) {
+  for (int32_t i = 0; i < 200000; ++i) {
     printf("create table :%d\n", i);
     createNewTable(pConn, i);
   }
@@ -750,7 +751,9 @@ TEST(testCase, projection_query_tables) {
   taos_free_result(pRes);
   taos_close(pConn);
 }
-#endif
+
+
+#if 0
 
 TEST(testCase, tsbs_perf_test) {
   TdThread qid[20] = {0};
@@ -761,7 +764,7 @@ TEST(testCase, tsbs_perf_test) {
   getchar();
 }
 
-#if 0
+
 TEST(testCase, projection_query_stables) {
   TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
   ASSERT_NE(pConn, nullptr);

source/common/src/tglobal.c

@@ -50,8 +50,6 @@ int32_t tsNumOfMnodeReadThreads = 1;
 int32_t tsNumOfVnodeQueryThreads = 4;
 int32_t tsNumOfVnodeStreamThreads = 2;
 int32_t tsNumOfVnodeFetchThreads = 4;
-int32_t tsNumOfVnodeWriteThreads = 2;
-int32_t tsNumOfVnodeSyncThreads = 2;
 int32_t tsNumOfVnodeRsmaThreads = 2;
 int32_t tsNumOfQnodeQueryThreads = 4;
 int32_t tsNumOfQnodeFetchThreads = 1;
@@ -374,14 +372,6 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
   tsNumOfVnodeFetchThreads = TMAX(tsNumOfVnodeFetchThreads, 4);
   if (cfgAddInt32(pCfg, "numOfVnodeFetchThreads", tsNumOfVnodeFetchThreads, 4, 1024, 0) != 0) return -1;
 
-  tsNumOfVnodeWriteThreads = tsNumOfCores;
-  tsNumOfVnodeWriteThreads = TMAX(tsNumOfVnodeWriteThreads, 1);
-  if (cfgAddInt32(pCfg, "numOfVnodeWriteThreads", tsNumOfVnodeWriteThreads, 1, 1024, 0) != 0) return -1;
-
-  tsNumOfVnodeSyncThreads = tsNumOfCores * 2;
-  tsNumOfVnodeSyncThreads = TMAX(tsNumOfVnodeSyncThreads, 16);
-  if (cfgAddInt32(pCfg, "numOfVnodeSyncThreads", tsNumOfVnodeSyncThreads, 1, 1024, 0) != 0) return -1;
-
   tsNumOfVnodeRsmaThreads = tsNumOfCores;
   tsNumOfVnodeRsmaThreads = TMAX(tsNumOfVnodeRsmaThreads, 4);
   if (cfgAddInt32(pCfg, "numOfVnodeRsmaThreads", tsNumOfVnodeRsmaThreads, 1, 1024, 0) != 0) return -1;
@@ -506,22 +496,6 @@ static int32_t taosUpdateServerCfg(SConfig *pCfg) {
     pItem->stype = stype;
   }
 
-  pItem = cfgGetItem(tsCfg, "numOfVnodeWriteThreads");
-  if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
-    tsNumOfVnodeWriteThreads = numOfCores;
-    tsNumOfVnodeWriteThreads = TMAX(tsNumOfVnodeWriteThreads, 1);
-    pItem->i32 = tsNumOfVnodeWriteThreads;
-    pItem->stype = stype;
-  }
-
-  pItem = cfgGetItem(tsCfg, "numOfVnodeSyncThreads");
-  if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
-    tsNumOfVnodeSyncThreads = numOfCores * 2;
-    tsNumOfVnodeSyncThreads = TMAX(tsNumOfVnodeSyncThreads, 16);
-    pItem->i32 = tsNumOfVnodeSyncThreads;
-    pItem->stype = stype;
-  }
-
   pItem = cfgGetItem(tsCfg, "numOfVnodeRsmaThreads");
   if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
     tsNumOfVnodeRsmaThreads = numOfCores;
@@ -699,8 +673,6 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
   tsNumOfVnodeQueryThreads = cfgGetItem(pCfg, "numOfVnodeQueryThreads")->i32;
   tsNumOfVnodeStreamThreads = cfgGetItem(pCfg, "numOfVnodeStreamThreads")->i32;
   tsNumOfVnodeFetchThreads = cfgGetItem(pCfg, "numOfVnodeFetchThreads")->i32;
-  tsNumOfVnodeWriteThreads = cfgGetItem(pCfg, "numOfVnodeWriteThreads")->i32;
-  tsNumOfVnodeSyncThreads = cfgGetItem(pCfg, "numOfVnodeSyncThreads")->i32;
   tsNumOfVnodeRsmaThreads = cfgGetItem(pCfg, "numOfVnodeRsmaThreads")->i32;
   tsNumOfQnodeQueryThreads = cfgGetItem(pCfg, "numOfQnodeQueryThreads")->i32;
   //  tsNumOfQnodeFetchThreads = cfgGetItem(pCfg, "numOfQnodeFetchThreads")->i32;
@@ -786,6 +758,9 @@ int32_t taosSetCfg(SConfig *pCfg, char *name) {
     case 'd': {
       if (strcasecmp("dDebugFlag", name) == 0) {
         dDebugFlag = cfgGetItem(pCfg, "dDebugFlag")->i32;
+      } else if (strcasecmp("debugFlag", name) == 0) {
+        int32_t flag = cfgGetItem(pCfg, "debugFlag")->i32;
+        taosSetAllDebugFlag(flag, true);
       }
       break;
     }
@@ -943,10 +918,6 @@ int32_t taosSetCfg(SConfig *pCfg, char *name) {
               } else if (strcasecmp("numOfVnodeFetchThreads", name) == 0) {
                 tsNumOfVnodeFetchThreads = cfgGetItem(pCfg, "numOfVnodeFetchThreads")->i32;
         */
-      } else if (strcasecmp("numOfVnodeWriteThreads", name) == 0) {
-        tsNumOfVnodeWriteThreads = cfgGetItem(pCfg, "numOfVnodeWriteThreads")->i32;
-      } else if (strcasecmp("numOfVnodeSyncThreads", name) == 0) {
-        tsNumOfVnodeSyncThreads = cfgGetItem(pCfg, "numOfVnodeSyncThreads")->i32;
       } else if (strcasecmp("numOfVnodeRsmaThreads", name) == 0) {
         tsNumOfVnodeRsmaThreads = cfgGetItem(pCfg, "numOfVnodeRsmaThreads")->i32;
       } else if (strcasecmp("numOfQnodeQueryThreads", name) == 0) {

source/dnode/mgmt/mgmt_snode/src/smWorker.c

@@ -58,7 +58,11 @@ static void smProcessStreamQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   dTrace("msg:%p, get from snode-stream queue", pMsg);
   int32_t code = sndProcessStreamMsg(pMgmt->pSnode, pMsg);
   if (code < 0) {
-    dGError("snd, msg:%p failed to process stream since %s", pMsg, terrstr(code));
+    if (pMsg) {
+      dGError("snd, msg:%p failed to process stream msg %s since %s", pMsg, TMSG_INFO(pMsg->msgType), terrstr(code));
+    } else {
+      dGError("snd, msg:%p failed to process stream empty msg since %s", pMsg, terrstr(code));
+    }
     smSendRsp(pMsg, terrno);
   }
 
@@ -136,6 +140,9 @@ int32_t smPutMsgToQueue(SSnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
   if (pSnode == NULL) {
     dError("snode: msg:%p failed to put into vnode queue since %s, type:%s qtype:%d", pMsg, terrstr(),
            TMSG_INFO(pMsg->msgType), qtype);
+    taosFreeQitem(pMsg);
+    rpcFreeCont(pRpc->pCont);
+    pRpc->pCont = NULL;
     return -1;
   }
 

source/dnode/mgmt/mgmt_vnode/inc/vmInt.h

@@ -33,10 +33,6 @@ typedef struct SVnodeMgmt {
   SQWorkerPool   queryPool;
   SQWorkerPool   streamPool;
   SWWorkerPool   fetchPool;
-  SWWorkerPool   syncPool;
-  SWWorkerPool   syncCtrlPool;
-  SWWorkerPool   writePool;
-  SWWorkerPool   applyPool;
   SSingleWorker  mgmtWorker;
   SHashObj      *hash;
   TdThreadRwlock lock;
@@ -58,10 +54,10 @@ typedef struct {
   int8_t        dropped;
   char         *path;
   SVnode       *pImpl;
-  STaosQueue *pWriteQ;
+  SMultiWorker  pWriteW;
-  STaosQueue *pSyncQ;
+  SMultiWorker  pSyncW;
-  STaosQueue *pSyncCtrlQ;
+  SMultiWorker  pSyncCtrlW;
-  STaosQueue *pApplyQ;
+  SMultiWorker  pApplyW;
   STaosQueue   *pQueryQ;
   STaosQueue   *pStreamQ;
   STaosQueue   *pFetchQ;

source/dnode/mgmt/mgmt_vnode/src/vmHandle.c

@@ -403,7 +403,6 @@ SArray *vmGetMsgHandles() {
   if (dmSetMgmtHandle(pArray, TDMT_VND_COMMIT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_SCH_QUERY_HEARTBEAT, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
 
-  if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TRIGGER, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DROP, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
@@ -412,6 +411,9 @@ SArray *vmGetMsgHandles() {
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE_RSP, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_RECOVER_FINISH, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_CHECK, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_CHECK_RSP, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TRIGGER, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
 
   if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA, vmPutMsgToMgmtQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;

source/dnode/mgmt/mgmt_vnode/src/vmInt.c

@@ -79,29 +79,49 @@ int32_t vmOpenVnode(SVnodeMgmt *pMgmt, SWrapperCfg *pCfg, SVnode *pImpl) {
 void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
   char path[TSDB_FILENAME_LEN] = {0};
 
-  vnodePreClose(pVnode->pImpl);
-
   taosThreadRwlockWrlock(&pMgmt->lock);
   taosHashRemove(pMgmt->hash, &pVnode->vgId, sizeof(int32_t));
   taosThreadRwlockUnlock(&pMgmt->lock);
   vmReleaseVnode(pMgmt, pVnode);
 
-  dTrace("vgId:%d, wait for vnode ref become 0", pVnode->vgId);
+  dInfo("vgId:%d, pre close", pVnode->vgId);
-  while (pVnode->refCount > 0) taosMsleep(10);
+  vnodePreClose(pVnode->pImpl);
-  dTrace("vgId:%d, wait for vnode queue is empty", pVnode->vgId);
 
-  while (!taosQueueEmpty(pVnode->pWriteQ)) taosMsleep(10);
+  dInfo("vgId:%d, wait for vnode ref become 0", pVnode->vgId);
-  while (!taosQueueEmpty(pVnode->pSyncQ)) taosMsleep(10);
+  while (pVnode->refCount > 0) taosMsleep(10);
-  while (!taosQueueEmpty(pVnode->pApplyQ)) taosMsleep(10);
+
+  dInfo("vgId:%d, wait for vnode write queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pWriteW.queue,
+        pVnode->pWriteW.queue->threadId);
+  tMultiWorkerCleanup(&pVnode->pWriteW);
+
+  dInfo("vgId:%d, wait for vnode sync queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pSyncW.queue,
+        pVnode->pSyncW.queue->threadId);
+  tMultiWorkerCleanup(&pVnode->pSyncW);
+
+  dInfo("vgId:%d, wait for vnode sync ctrl queue:%p is empty, thread:%08" PRId64, pVnode->vgId,
+        pVnode->pSyncCtrlW.queue, pVnode->pSyncCtrlW.queue->threadId);
+  tMultiWorkerCleanup(&pVnode->pSyncCtrlW);
+
+  dInfo("vgId:%d, wait for vnode apply queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pApplyW.queue,
+        pVnode->pApplyW.queue->threadId);
+  tMultiWorkerCleanup(&pVnode->pApplyW);
+
+  dInfo("vgId:%d, wait for vnode query queue:%p is empty", pVnode->vgId, pVnode->pQueryQ);
   while (!taosQueueEmpty(pVnode->pQueryQ)) taosMsleep(10);
+
+  dInfo("vgId:%d, wait for vnode fetch queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pFetchQ,
+        pVnode->pFetchQ->threadId);
   while (!taosQueueEmpty(pVnode->pFetchQ)) taosMsleep(10);
+
+  dInfo("vgId:%d, wait for vnode stream queue:%p is empty", pVnode->vgId, pVnode->pStreamQ);
   while (!taosQueueEmpty(pVnode->pStreamQ)) taosMsleep(10);
-  dTrace("vgId:%d, vnode queue is empty", pVnode->vgId);
+
+  dInfo("vgId:%d, all vnode queues is empty", pVnode->vgId);
 
   vmFreeQueue(pMgmt, pVnode);
   vnodeClose(pVnode->pImpl);
   pVnode->pImpl = NULL;
-  dDebug("vgId:%d, vnode is closed", pVnode->vgId);
+  dInfo("vgId:%d, vnode is closed", pVnode->vgId);
 
   if (pVnode->dropped) {
     dInfo("vgId:%d, vnode is destroyed, dropped:%d", pVnode->vgId, pVnode->dropped);

source/dnode/mgmt/mgmt_vnode/src/vmWorker.c

@@ -86,7 +86,12 @@ static void vmProcessStreamQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo);
   if (code != 0) {
     if (terrno != 0) code = terrno;
-    dGError("vgId:%d, msg:%p failed to process stream since %s", pVnode->vgId, pMsg, terrstr(code));
+    if (pMsg) {
+      dGError("vgId:%d, msg:%p failed to process stream msg %s since %s", pVnode->vgId, pMsg, TMSG_INFO(pMsg->msgType),
+              terrstr(code));
+    } else {
+      dGError("vgId:%d, msg:%p failed to process stream empty msg since %s", pVnode->vgId, pMsg, terrstr(code));
+    }
     vmSendRsp(pMsg, code);
   }
 
@@ -140,6 +145,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
 
   pHead->contLen = ntohl(pHead->contLen);
   pHead->vgId = ntohl(pHead->vgId);
+  pHead->msgMask = ntohl(pHead->msgMask);
 
   SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId);
   if (pVnode == NULL) {
@@ -150,7 +156,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
 
   switch (qtype) {
     case QUERY_QUEUE:
-      if ((pMsg->msgType == TDMT_SCH_QUERY) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
+      if ((pMsg->msgType == TDMT_SCH_QUERY) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS) && !TEST_SHOW_REWRITE_MASK(pHead->msgMask)) {
         terrno = TSDB_CODE_GRANT_EXPIRED;
         code = terrno;
         dDebug("vgId:%d, msg:%p put into vnode-query queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
@@ -183,30 +189,20 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
         dDebug("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
       } else {
         dGTrace("vgId:%d, msg:%p put into vnode-write queue", pVnode->vgId, pMsg);
-        taosWriteQitem(pVnode->pWriteQ, pMsg);
+        taosWriteQitem(pVnode->pWriteW.queue, pMsg);
-#if 0  // tests for batch writes
-        if (pMsg->msgType == TDMT_VND_CREATE_TABLE) {
-          SRpcMsg *pDup = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM);
-          memcpy(pDup, pMsg, sizeof(SRpcMsg));
-          pDup->pCont = rpcMallocCont(pMsg->contLen);
-          memcpy(pDup->pCont, pMsg->pCont, pMsg->contLen);
-          pDup->info.handle = NULL;
-          taosWriteQitem(pVnode->pWriteQ, pDup);
-        }
-#endif
       }
       break;
     case SYNC_QUEUE:
       dGTrace("vgId:%d, msg:%p put into vnode-sync queue", pVnode->vgId, pMsg);
-      taosWriteQitem(pVnode->pSyncQ, pMsg);
+      taosWriteQitem(pVnode->pSyncW.queue, pMsg);
       break;
     case SYNC_CTRL_QUEUE:
       dGTrace("vgId:%d, msg:%p put into vnode-sync-ctrl queue", pVnode->vgId, pMsg);
-      taosWriteQitem(pVnode->pSyncCtrlQ, pMsg);
+      taosWriteQitem(pVnode->pSyncCtrlW.queue, pMsg);
       break;
     case APPLY_QUEUE:
       dGTrace("vgId:%d, msg:%p put into vnode-apply queue", pVnode->vgId, pMsg);
-      taosWriteQitem(pVnode->pApplyQ, pMsg);
+      taosWriteQitem(pVnode->pApplyW.queue, pMsg);
       break;
     default:
       code = -1;
@@ -218,7 +214,9 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
   return code;
 }
 
-int32_t vmPutMsgToSyncCtrlQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsgToQueue(pMgmt, pMsg, SYNC_CTRL_QUEUE); }
+int32_t vmPutMsgToSyncCtrlQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
+  return vmPutMsgToQueue(pMgmt, pMsg, SYNC_CTRL_QUEUE);
+}
 
 int32_t vmPutMsgToSyncQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsgToQueue(pMgmt, pMsg, SYNC_QUEUE); }
 
@@ -269,13 +267,13 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) {
   if (pVnode != NULL) {
     switch (qtype) {
       case WRITE_QUEUE:
-        size = taosQueueItemSize(pVnode->pWriteQ);
+        size = taosQueueItemSize(pVnode->pWriteW.queue);
         break;
       case SYNC_QUEUE:
-        size = taosQueueItemSize(pVnode->pSyncQ);
+        size = taosQueueItemSize(pVnode->pSyncW.queue);
         break;
       case APPLY_QUEUE:
-        size = taosQueueItemSize(pVnode->pApplyQ);
+        size = taosQueueItemSize(pVnode->pApplyW.queue);
         break;
       case QUERY_QUEUE:
         size = taosQueueItemSize(pVnode->pQueryQ);
@@ -299,40 +297,44 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) {
 }
 
 int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
-  pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode->pImpl, (FItems)vnodeProposeWriteMsg);
+  SMultiWorkerCfg wcfg = {.max = 1, .name = "vnode-write", .fp = (FItems)vnodeProposeWriteMsg, .param = pVnode->pImpl};
-  pVnode->pSyncQ = tWWorkerAllocQueue(&pMgmt->syncPool, pVnode, (FItems)vmProcessSyncQueue);
+  SMultiWorkerCfg scfg = {.max = 1, .name = "vnode-sync", .fp = (FItems)vmProcessSyncQueue, .param = pVnode};
-  pVnode->pSyncCtrlQ = tWWorkerAllocQueue(&pMgmt->syncCtrlPool, pVnode, (FItems)vmProcessSyncQueue);
+  SMultiWorkerCfg sccfg = {.max = 1, .name = "vnode-sync-ctrl", .fp = (FItems)vmProcessSyncQueue, .param = pVnode};
-  pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->applyPool, pVnode->pImpl, (FItems)vnodeApplyWriteMsg);
+  SMultiWorkerCfg acfg = {.max = 1, .name = "vnode-apply", .fp = (FItems)vnodeApplyWriteMsg, .param = pVnode->pImpl};
+  (void)tMultiWorkerInit(&pVnode->pWriteW, &wcfg);
+  (void)tMultiWorkerInit(&pVnode->pSyncW, &scfg);
+  (void)tMultiWorkerInit(&pVnode->pSyncCtrlW, &sccfg);
+  (void)tMultiWorkerInit(&pVnode->pApplyW, &acfg);
+
   pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue);
   pVnode->pStreamQ = tQWorkerAllocQueue(&pMgmt->streamPool, pVnode, (FItem)vmProcessStreamQueue);
   pVnode->pFetchQ = tWWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItems)vmProcessFetchQueue);
 
-  if (pVnode->pWriteQ == NULL || pVnode->pSyncQ == NULL || pVnode->pApplyQ == NULL || pVnode->pQueryQ == NULL ||
+  if (pVnode->pWriteW.queue == NULL || pVnode->pSyncW.queue == NULL || pVnode->pSyncCtrlW.queue == NULL ||
-      pVnode->pStreamQ == NULL || pVnode->pFetchQ == NULL) {
+      pVnode->pApplyW.queue == NULL || pVnode->pQueryQ == NULL || pVnode->pStreamQ == NULL || pVnode->pFetchQ == NULL) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
     return -1;
   }
 
-  dDebug("vgId:%d, write-queue:%p is alloced", pVnode->vgId, pVnode->pWriteQ);
+  dInfo("vgId:%d, write-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pWriteW.queue,
-  dDebug("vgId:%d, sync-queue:%p is alloced", pVnode->vgId, pVnode->pSyncQ);
+        pVnode->pWriteW.queue->threadId);
-  dDebug("vgId:%d, apply-queue:%p is alloced", pVnode->vgId, pVnode->pApplyQ);
+  dInfo("vgId:%d, sync-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pSyncW.queue,
-  dDebug("vgId:%d, query-queue:%p is alloced", pVnode->vgId, pVnode->pQueryQ);
+        pVnode->pSyncW.queue->threadId);
-  dDebug("vgId:%d, stream-queue:%p is alloced", pVnode->vgId, pVnode->pStreamQ);
+  dInfo("vgId:%d, sync-ctrl-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pSyncCtrlW.queue,
-  dDebug("vgId:%d, fetch-queue:%p is alloced", pVnode->vgId, pVnode->pFetchQ);
+        pVnode->pSyncCtrlW.queue->threadId);
+  dInfo("vgId:%d, apply-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pApplyW.queue,
+        pVnode->pApplyW.queue->threadId);
+  dInfo("vgId:%d, query-queue:%p is alloced", pVnode->vgId, pVnode->pQueryQ);
+  dInfo("vgId:%d, fetch-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pFetchQ,
+        pVnode->pFetchQ->threadId);
+  dInfo("vgId:%d, stream-queue:%p is alloced", pVnode->vgId, pVnode->pStreamQ);
   return 0;
 }
 
 void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
-  tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pWriteQ);
-  tWWorkerFreeQueue(&pMgmt->applyPool, pVnode->pApplyQ);
-  tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ);
-  tWWorkerFreeQueue(&pMgmt->syncCtrlPool, pVnode->pSyncCtrlQ);
   tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ);
   tQWorkerFreeQueue(&pMgmt->streamPool, pVnode->pStreamQ);
   tWWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ);
-  pVnode->pWriteQ = NULL;
-  pVnode->pSyncQ = NULL;
-  pVnode->pApplyQ = NULL;
   pVnode->pQueryQ = NULL;
   pVnode->pStreamQ = NULL;
   pVnode->pFetchQ = NULL;
@@ -357,26 +359,6 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
   pFPool->max = tsNumOfVnodeFetchThreads;
   if (tWWorkerInit(pFPool) != 0) return -1;
 
-  SWWorkerPool *pWPool = &pMgmt->writePool;
-  pWPool->name = "vnode-write";
-  pWPool->max = tsNumOfVnodeWriteThreads;
-  if (tWWorkerInit(pWPool) != 0) return -1;
-
-  SWWorkerPool *pAPool = &pMgmt->applyPool;
-  pAPool->name = "vnode-apply";
-  pAPool->max = tsNumOfVnodeWriteThreads;
-  if (tWWorkerInit(pAPool) != 0) return -1;
-
-  SWWorkerPool *pSPool = &pMgmt->syncPool;
-  pSPool->name = "vnode-sync";
-  pSPool->max = tsNumOfVnodeSyncThreads;
-  if (tWWorkerInit(pSPool) != 0) return -1;
-
-  SWWorkerPool *pSCPool = &pMgmt->syncCtrlPool;
-  pSCPool->name = "vnode-sync-ctrl";
-  pSCPool->max = tsNumOfVnodeSyncThreads;
-  if (tWWorkerInit(pSCPool) != 0) return -1;
-
   SSingleWorkerCfg mgmtCfg = {
       .min = 1,
       .max = 1,
@@ -391,10 +373,6 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
 }
 
 void vmStopWorker(SVnodeMgmt *pMgmt) {
-  tWWorkerCleanup(&pMgmt->writePool);
-  tWWorkerCleanup(&pMgmt->applyPool);
-  tWWorkerCleanup(&pMgmt->syncPool);
-  tWWorkerCleanup(&pMgmt->syncCtrlPool);
   tQWorkerCleanup(&pMgmt->queryPool);
   tQWorkerCleanup(&pMgmt->streamPool);
   tWWorkerCleanup(&pMgmt->fetchPool);

source/dnode/mgmt/node_mgmt/src/dmTransport.c

@@ -157,7 +157,11 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
 _OVER:
   if (code != 0) {
     if (terrno != 0) code = terrno;
-    dGTrace("msg:%p, failed to process since %s", pMsg, terrstr());
+    if (pMsg) {
+      dGTrace("msg:%p, failed to process %s since %s", pMsg, TMSG_INFO(pMsg->msgType), terrstr());
+    } else {
+      dGTrace("msg:%p, failed to process empty msg since %s", pMsg, terrstr());
+    }
 
     if (IsReq(pRpc)) {
       SRpcMsg rsp = {.code = code, .info = pRpc->info};

source/dnode/mnode/impl/src/mndStb.c

@@ -466,7 +466,7 @@ static void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pSt
 
   contLen += sizeof(SMsgHead);
 
-  SMsgHead *pHead = taosMemoryMalloc(contLen);
+  SMsgHead *pHead = taosMemoryCalloc(1, contLen);
   if (pHead == NULL) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
     goto _err;
@@ -519,6 +519,7 @@ static void *mndBuildVDropStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb,
 
   pHead->contLen = htonl(contLen);
   pHead->vgId = htonl(pVgroup->vgId);
+  pHead->msgMask = htonl(0);
 
   void *pBuf = POINTER_SHIFT(pHead, sizeof(SMsgHead));
 

source/dnode/mnode/impl/src/mndStream.c

@@ -705,7 +705,8 @@ static int32_t mndProcessDropStreamReq(SRpcMsg *pReq) {
     return -1;
   }
 
-  STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_NOTHING, pReq, "drop-stream");
+  STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB_INSIDE, pReq, "drop-stream");
+  mndTransSetDbName(pTrans, pStream->sourceDb, pStream->targetDb);
   if (pTrans == NULL) {
     mError("stream:%s, failed to drop since %s", dropReq.name, terrstr());
     sdbRelease(pMnode->pSdb, pStream);

source/dnode/mnode/impl/src/mndSubscribe.c

@@ -612,14 +612,14 @@ static int32_t mndProcessRebalanceReq(SRpcMsg *pMsg) {
       mndReleaseSubscribe(pMnode, pSub);
     }
 
-    // TODO replace assert with error check
+    if (mndDoRebalance(pMnode, &rebInput, &rebOutput) < 0) {
-    ASSERT(mndDoRebalance(pMnode, &rebInput, &rebOutput) == 0);
+      mError("mq rebalance internal error");
+    }
 
     // if add more consumer to balanced subscribe,
     // possibly no vg is changed
     /*ASSERT(taosArrayGetSize(rebOutput.rebVgs) != 0);*/
 
-    // TODO replace assert with error check
     if (mndPersistRebResult(pMnode, pMsg, &rebOutput) < 0) {
       mError("mq rebalance persist rebalance output error, possibly vnode splitted or dropped");
     }

source/dnode/mnode/impl/src/mndSync.c

@@ -282,6 +282,8 @@ int32_t mndSyncPropose(SMnode *pMnode, SSdbRaw *pRaw, int32_t transId) {
   pMgmt->errCode = 0;
 
   SRpcMsg req = {.msgType = TDMT_MND_APPLY_MSG, .contLen = sdbGetRawTotalSize(pRaw)};
+  if (req.contLen <= 0) return -1;
+  
   req.pCont = rpcMallocCont(req.contLen);
   if (req.pCont == NULL) return -1;
   memcpy(req.pCont, pRaw, req.contLen);

source/dnode/mnode/impl/src/mndTrans.c

@@ -375,7 +375,7 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
       if (taosArrayPush(pTrans->undoActions, &action) == NULL) goto _OVER;
       action.pCont = NULL;
     } else {
-      if (taosArrayPush(pTrans->redoActions, &action) == NULL) goto _OVER;
+      if (taosArrayPush(pTrans->undoActions, &action) == NULL) goto _OVER;
     }
   }
 

source/dnode/mnode/impl/src/mndVgroup.c

@@ -325,10 +325,10 @@ static void *mndBuildAlterVnodeConfigReq(SMnode *pMnode, SDbObj *pDb, SVgObj *pV
 static void *mndBuildAlterVnodeReplicaReq(SMnode *pMnode, SDbObj *pDb, SVgObj *pVgroup, int32_t dnodeId,
                                           int32_t *pContLen) {
   SAlterVnodeReplicaReq alterReq = {
-      alterReq.vgId = pVgroup->vgId,
+      .vgId = pVgroup->vgId,
-      alterReq.strict = pDb->cfg.strict,
+      .strict = pDb->cfg.strict,
-      alterReq.replica = pVgroup->replica,
+      .replica = pVgroup->replica,
-      alterReq.selfIndex = -1,
+      .selfIndex = -1,
   };
 
   for (int32_t v = 0; v < pVgroup->replica; ++v) {

source/dnode/snode/src/snode.c

@@ -231,10 +231,10 @@ int32_t sndProcessTaskRetrieveReq(SSnode *pSnode, SRpcMsg *pMsg) {
 
 int32_t sndProcessTaskDispatchRsp(SSnode *pSnode, SRpcMsg *pMsg) {
   SStreamDispatchRsp *pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
-  int32_t             taskId = pRsp->taskId;
+  int32_t             taskId = ntohl(pRsp->upstreamTaskId);
   SStreamTask        *pTask = streamMetaGetTask(pSnode->pMeta, taskId);
   if (pTask) {
-    streamProcessDispatchRsp(pTask, pRsp);
+    streamProcessDispatchRsp(pTask, pRsp, pMsg->code);
     return 0;
   } else {
     return -1;

source/dnode/vnode/inc/vnode.h

@@ -158,7 +158,7 @@ int32_t tsdbReaderOpen(SVnode *pVnode, SQueryTableDataCond *pCond, void *pTableL
 void     tsdbReaderClose(STsdbReader *pReader);
 bool     tsdbNextDataBlock(STsdbReader *pReader);
 bool     tsdbTableNextDataBlock(STsdbReader *pReader, uint64_t uid);
-void     tsdbRetrieveDataBlockInfo(const STsdbReader* pReader, int32_t* rows, uint64_t* uid, STimeWindow* pWindow);
+void     tsdbRetrieveDataBlockInfo(const STsdbReader *pReader, int32_t *rows, uint64_t *uid, STimeWindow *pWindow);
 int32_t  tsdbRetrieveDatablockSMA(STsdbReader *pReader, SColumnDataAgg ***pBlockStatis, bool *allHave);
 SArray  *tsdbRetrieveDataBlock(STsdbReader *pTsdbReadHandle, SArray *pColumnIdList);
 int32_t  tsdbReaderReset(STsdbReader *pReader, SQueryTableDataCond *pCond);
@@ -240,7 +240,7 @@ bool    tqNextDataBlock(STqReader *pReader);
 bool    tqNextDataBlockFilterOut(STqReader *pReader, SHashObj *filterOutUids);
 int32_t tqRetrieveDataBlock(SSDataBlock *pBlock, STqReader *pReader);
 
-void vnodeEnqueueStreamMsg(SVnode *pVnode, SRpcMsg *pMsg);
+int32_t vnodeEnqueueStreamMsg(SVnode *pVnode, SRpcMsg *pMsg);
 
 // sma
 int32_t smaGetTSmaDays(SVnodeCfg *pCfg, void *pCont, uint32_t contLen, int32_t *days);

source/dnode/vnode/src/inc/vnd.h

@@ -97,6 +97,7 @@ bool    vnodeShouldRollback(SVnode* pVnode);
 // vnodeSync.c
 int32_t vnodeSyncOpen(SVnode* pVnode, char* path);
 void    vnodeSyncStart(SVnode* pVnode);
+void    vnodeSyncPreClose(SVnode* pVnode);
 void    vnodeSyncClose(SVnode* pVnode);
 void    vnodeRedirectRpcMsg(SVnode* pVnode, SRpcMsg* pMsg);
 bool    vnodeIsLeader(SVnode* pVnode);

source/dnode/vnode/src/inc/vnodeInt.h

@@ -183,13 +183,13 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg);
 // tq-stream
 int32_t tqProcessTaskDeployReq(STQ* pTq, int64_t version, char* msg, int32_t msgLen);
 int32_t tqProcessTaskDropReq(STQ* pTq, int64_t version, char* msg, int32_t msgLen);
+int32_t tqProcessStreamTaskCheckReq(STQ* pTq, SRpcMsg* pMsg);
+int32_t tqProcessStreamTaskCheckRsp(STQ* pTq, int64_t version, char* msg, int32_t msgLen);
 int32_t tqProcessSubmitReq(STQ* pTq, SSubmitReq* data, int64_t ver);
 int32_t tqProcessDelReq(STQ* pTq, void* pReq, int32_t len, int64_t ver);
 int32_t tqProcessTaskRunReq(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessTaskDispatchReq(STQ* pTq, SRpcMsg* pMsg, bool exec);
 int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg);
-// int32_t tqProcessTaskRecoverReq(STQ* pTq, SRpcMsg* pMsg);
-// int32_t tqProcessTaskRecoverRsp(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessTaskRetrieveReq(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessTaskRetrieveRsp(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessTaskRecover1Req(STQ* pTq, SRpcMsg* pMsg);

source/dnode/vnode/src/tq/tq.c

@@ -896,6 +896,10 @@ int32_t tqExpandTask(STQ* pTq, SStreamTask* pTask, int64_t ver) {
   pTask->startVer = ver;
 
   // expand executor
+  if (pTask->fillHistory) {
+    pTask->taskStatus = TASK_STATUS__WAIT_DOWNSTREAM;
+  }
+
   if (pTask->taskLevel == TASK_LEVEL__SOURCE) {
     pTask->pState = streamStateOpen(pTq->pStreamMeta->path, pTask, false, -1, -1);
     if (pTask->pState == NULL) {
@@ -911,9 +915,6 @@ int32_t tqExpandTask(STQ* pTq, SStreamTask* pTask, int64_t ver) {
     pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, &handle);
     ASSERT(pTask->exec.executor);
 
-    if (pTask->fillHistory) {
-      pTask->taskStatus = TASK_STATUS__RECOVER_PREPARE;
-    }
   } else if (pTask->taskLevel == TASK_LEVEL__AGG) {
     pTask->pState = streamStateOpen(pTq->pStreamMeta->path, pTask, false, -1, -1);
     if (pTask->pState == NULL) {
@@ -947,11 +948,90 @@ int32_t tqExpandTask(STQ* pTq, SStreamTask* pTask, int64_t ver) {
 
   streamSetupTrigger(pTask);
 
-  tqInfo("expand stream task on vg %d, task id %d, child id %d", TD_VID(pTq->pVnode), pTask->taskId,
+  tqInfo("expand stream task on vg %d, task id %d, child id %d, level %d", TD_VID(pTq->pVnode), pTask->taskId,
-         pTask->selfChildId);
+         pTask->selfChildId, pTask->taskLevel);
   return 0;
 }
 
+int32_t tqProcessStreamTaskCheckReq(STQ* pTq, SRpcMsg* pMsg) {
+  char*               msgStr = pMsg->pCont;
+  char*               msgBody = POINTER_SHIFT(msgStr, sizeof(SMsgHead));
+  int32_t             msgLen = pMsg->contLen - sizeof(SMsgHead);
+  SStreamTaskCheckReq req;
+  SDecoder            decoder;
+  tDecoderInit(&decoder, msgBody, msgLen);
+  tDecodeSStreamTaskCheckReq(&decoder, &req);
+  tDecoderClear(&decoder);
+  int32_t             taskId = req.downstreamTaskId;
+  SStreamTaskCheckRsp rsp = {
+      .reqId = req.reqId,
+      .streamId = req.streamId,
+      .childId = req.childId,
+      .downstreamNodeId = req.downstreamNodeId,
+      .downstreamTaskId = req.downstreamTaskId,
+      .upstreamNodeId = req.upstreamNodeId,
+      .upstreamTaskId = req.upstreamTaskId,
+  };
+  SStreamTask* pTask = streamMetaGetTask(pTq->pStreamMeta, taskId);
+  if (pTask && atomic_load_8(&pTask->taskStatus) == TASK_STATUS__NORMAL) {
+    rsp.status = 1;
+  } else {
+    rsp.status = 0;
+  }
+
+  tqDebug("tq recv task check req(reqId: %" PRId64 ") %d at node %d check req from task %d at node %d, status %d",
+          rsp.reqId, rsp.downstreamTaskId, rsp.downstreamNodeId, rsp.upstreamTaskId, rsp.upstreamNodeId, rsp.status);
+
+  SEncoder encoder;
+  int32_t  code;
+  int32_t  len;
+  tEncodeSize(tEncodeSStreamTaskCheckRsp, &rsp, len, code);
+  if (code < 0) {
+    ASSERT(0);
+  }
+  void* buf = rpcMallocCont(sizeof(SMsgHead) + len);
+  ((SMsgHead*)buf)->vgId = htonl(req.upstreamNodeId);
+
+  void* abuf = POINTER_SHIFT(buf, sizeof(SMsgHead));
+  tEncoderInit(&encoder, (uint8_t*)abuf, len);
+  tEncodeSStreamTaskCheckRsp(&encoder, &rsp);
+  tEncoderClear(&encoder);
+
+  SRpcMsg rspMsg = {
+      .code = 0,
+      .pCont = buf,
+      .contLen = sizeof(SMsgHead) + len,
+      .info = pMsg->info,
+  };
+
+  tmsgSendRsp(&rspMsg);
+  return 0;
+}
+
+int32_t tqProcessStreamTaskCheckRsp(STQ* pTq, int64_t version, char* msg, int32_t msgLen) {
+  int32_t             code;
+  SStreamTaskCheckRsp rsp;
+
+  SDecoder decoder;
+  tDecoderInit(&decoder, (uint8_t*)msg, msgLen);
+  code = tDecodeSStreamTaskCheckRsp(&decoder, &rsp);
+  if (code < 0) {
+    tDecoderClear(&decoder);
+    return -1;
+  }
+  tDecoderClear(&decoder);
+
+  tqDebug("tq recv task check rsp(reqId: %" PRId64 ") %d at node %d check req from task %d at node %d, status %d",
+          rsp.reqId, rsp.downstreamTaskId, rsp.downstreamNodeId, rsp.upstreamTaskId, rsp.upstreamNodeId, rsp.status);
+
+  SStreamTask* pTask = streamMetaGetTask(pTq->pStreamMeta, rsp.upstreamTaskId);
+  if (pTask == NULL) {
+    return -1;
+  }
+
+  return streamProcessTaskCheckRsp(pTask, &rsp, version);
+}
+
 int32_t tqProcessTaskDeployReq(STQ* pTq, int64_t version, char* msg, int32_t msgLen) {
   int32_t code;
 #if 0
@@ -982,37 +1062,7 @@ int32_t tqProcessTaskDeployReq(STQ* pTq, int64_t version, char* msg, int32_t msg
 
   // 3.go through recover steps to fill history
   if (pTask->fillHistory) {
-    if (pTask->taskLevel == TASK_LEVEL__SOURCE) {
+    streamTaskCheckDownstream(pTask, version);
-      streamSetParamForRecover(pTask);
-      streamSourceRecoverPrepareStep1(pTask, version);
-
-      SStreamRecoverStep1Req req;
-      streamBuildSourceRecover1Req(pTask, &req);
-      int32_t len = sizeof(SStreamRecoverStep1Req);
-
-      void* serializedReq = rpcMallocCont(len);
-      if (serializedReq == NULL) {
-        return -1;
-      }
-
-      memcpy(serializedReq, &req, len);
-
-      SRpcMsg rpcMsg = {
-          .contLen = len,
-          .pCont = serializedReq,
-          .msgType = TDMT_VND_STREAM_RECOVER_STEP1,
-      };
-
-      if (tmsgPutToQueue(&pTq->pVnode->msgCb, STREAM_QUEUE, &rpcMsg) < 0) {
-        /*ASSERT(0);*/
-      }
-
-    } else if (pTask->taskLevel == TASK_LEVEL__AGG) {
-      streamSetParamForRecover(pTask);
-      streamAggRecoverPrepare(pTask);
-    } else if (pTask->taskLevel == TASK_LEVEL__SINK) {
-      // do nothing
-    }
   }
 
   return 0;
@@ -1268,7 +1318,7 @@ int32_t tqProcessSubmitReq(STQ* pTq, SSubmitReq* pReq, int64_t ver) {
     if (pIter == NULL) break;
     SStreamTask* pTask = *(SStreamTask**)pIter;
     if (pTask->taskLevel != TASK_LEVEL__SOURCE) continue;
-    if (pTask->taskStatus == TASK_STATUS__RECOVER_PREPARE) {
+    if (pTask->taskStatus == TASK_STATUS__RECOVER_PREPARE || pTask->taskStatus == TASK_STATUS__WAIT_DOWNSTREAM) {
       tqDebug("skip push task %d, task status %d", pTask->taskId, pTask->taskStatus);
       continue;
     }
@@ -1335,10 +1385,11 @@ int32_t tqProcessTaskDispatchReq(STQ* pTq, SRpcMsg* pMsg, bool exec) {
 
 int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) {
   SStreamDispatchRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
-  int32_t             taskId = pRsp->taskId;
+  int32_t             taskId = ntohl(pRsp->upstreamTaskId);
   SStreamTask*        pTask = streamMetaGetTask(pTq->pStreamMeta, taskId);
+  tqDebug("recv dispatch rsp, code: %x", pMsg->code);
   if (pTask) {
-    streamProcessDispatchRsp(pTask, pRsp);
+    streamProcessDispatchRsp(pTask, pRsp, pMsg->code);
     return 0;
   } else {
     return -1;
@@ -1379,9 +1430,9 @@ int32_t tqProcessTaskRetrieveRsp(STQ* pTq, SRpcMsg* pMsg) {
   return 0;
 }
 
-void vnodeEnqueueStreamMsg(SVnode* pVnode, SRpcMsg* pMsg) {
+int32_t vnodeEnqueueStreamMsg(SVnode* pVnode, SRpcMsg* pMsg) {
   STQ*      pTq = pVnode->pTq;
-  char*   msgStr = pMsg->pCont;
+  SMsgHead* msgStr = pMsg->pCont;
   char*     msgBody = POINTER_SHIFT(msgStr, sizeof(SMsgHead));
   int32_t   msgLen = pMsg->contLen - sizeof(SMsgHead);
   int32_t   code = 0;
@@ -1407,16 +1458,45 @@ void vnodeEnqueueStreamMsg(SVnode* pVnode, SRpcMsg* pMsg) {
     streamProcessDispatchReq(pTask, &req, &rsp, false);
     rpcFreeCont(pMsg->pCont);
     taosFreeQitem(pMsg);
-    return;
+    return 0;
   }
 
+  code = TSDB_CODE_STREAM_TASK_NOT_EXIST;
+
 FAIL:
-  if (pMsg->info.handle == NULL) return;
+  if (pMsg->info.handle == NULL) return -1;
+
+  SMsgHead* pRspHead = rpcMallocCont(sizeof(SMsgHead) + sizeof(SStreamDispatchRsp));
+  if (pRspHead == NULL) {
     SRpcMsg rsp = {
-      .code = code,
+        .code = TSDB_CODE_OUT_OF_MEMORY,
         .info = pMsg->info,
     };
+    tqDebug("send dispatch error rsp, code: %x", code);
     tmsgSendRsp(&rsp);
     rpcFreeCont(pMsg->pCont);
     taosFreeQitem(pMsg);
+    return -1;
+  }
+
+  pRspHead->vgId = htonl(req.upstreamNodeId);
+  SStreamDispatchRsp* pRsp = POINTER_SHIFT(pRspHead, sizeof(SMsgHead));
+  pRsp->streamId = htobe64(req.streamId);
+  pRsp->upstreamTaskId = htonl(req.upstreamTaskId);
+  pRsp->upstreamNodeId = htonl(req.upstreamNodeId);
+  pRsp->downstreamNodeId = htonl(pVnode->config.vgId);
+  pRsp->downstreamTaskId = htonl(req.taskId);
+  pRsp->inputStatus = TASK_OUTPUT_STATUS__NORMAL;
+
+  SRpcMsg rsp = {
+      .code = code,
+      .info = pMsg->info,
+      .contLen = sizeof(SMsgHead) + sizeof(SStreamDispatchRsp),
+      .pCont = pRspHead,
+  };
+  tqDebug("send dispatch error rsp, code: %x", code);
+  tmsgSendRsp(&rsp);
+  rpcFreeCont(pMsg->pCont);
+  taosFreeQitem(pMsg);
+  return -1;
 }

source/dnode/vnode/src/tsdb/tsdbCacheRead.c

@@ -338,8 +338,7 @@ _end:
   tsdbDataFReaderClose(&pr->pDataFReader);
 
   tsdbUntakeReadSnap(pr->pVnode->pTsdb, pr->pReadSnap, "cache-l");
-  pr->pDataFReaderLast = NULL;
+  resetLastBlockLoadInfo(pr->pLoadInfo);
-  pr->pDataFReader = NULL;
 
   for (int32_t j = 0; j < pr->numOfCols; ++j) {
     taosMemoryFree(pRes[j]);

source/dnode/vnode/src/tsdb/tsdbMergeTree.c

@@ -279,7 +279,6 @@ int32_t tLDataIterOpen(struct SLDataIter **pIter, SDataFReader *pReader, int32_t
 
   (*pIter)->pBlockLoadInfo = pBlockLoadInfo;
 
-//  size_t size = taosArrayGetSize(pBlockLoadInfo->aSttBlk);
   if (!pBlockLoadInfo->sttBlockLoaded) {
     int64_t st = taosGetTimestampUs();
     pBlockLoadInfo->sttBlockLoaded = true;

source/dnode/vnode/src/tsdb/tsdbRead.c

@@ -185,11 +185,11 @@ static int32_t  doMergeRowsInLastBlock(SLastBlockReader* pLastBlockReader, STabl
                                        SRowMerger* pMerger, SVersionRange* pVerRange);
 static int32_t  doMergeRowsInBuf(SIterInfo* pIter, uint64_t uid, int64_t ts, SArray* pDelList, SRowMerger* pMerger,
                                  STsdbReader* pReader);
-static int32_t  doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, STSRow* pTSRow, uint64_t uid);
+static int32_t  doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, STSRow* pTSRow, STableBlockScanInfo* pInfo);
 static int32_t  doAppendRowFromFileBlock(SSDataBlock* pResBlock, STsdbReader* pReader, SBlockData* pBlockData,
                                          int32_t rowIndex);
 static void     setComposedBlockFlag(STsdbReader* pReader, bool composed);
-static bool hasBeenDropped(const SArray* pDelList, int32_t* index, TSDBKEY* pKey, int32_t order, SVersionRange* pRange);
+static bool hasBeenDropped(const SArray* pDelList, int32_t* index, TSDBKEY* pKey, int32_t order, SVersionRange* pVerRange);
 
 static int32_t doMergeMemTableMultiRows(TSDBROW* pRow, uint64_t uid, SIterInfo* pIter, SArray* pDelList,
                                         STSRow** pTSRow, STsdbReader* pReader, bool* freeTSRow);
@@ -208,7 +208,6 @@ static bool          hasDataInLastBlock(SLastBlockReader* pLastBlockReader);
 static int32_t       doBuildDataBlock(STsdbReader* pReader);
 static TSDBKEY       getCurrentKeyInBuf(STableBlockScanInfo* pScanInfo, STsdbReader* pReader);
 static bool          hasDataInFileBlock(const SBlockData* pBlockData, const SFileBlockDumpInfo* pDumpInfo);
-static bool          hasDataInLastBlock(SLastBlockReader* pLastBlockReader);
 
 static bool outOfTimeWindow(int64_t ts, STimeWindow* pWindow) { return (ts > pWindow->ekey) || (ts < pWindow->skey); }
 
@@ -534,7 +533,7 @@ static SSDataBlock* createResBlock(SQueryTableDataCond* pCond, int32_t capacity)
   }
 
   for (int32_t i = 0; i < pCond->numOfCols; ++i) {
-    SColumnInfoData colInfo = {{0}, 0};
+    SColumnInfoData colInfo = {0, {0}};
     colInfo.info = pCond->colList[i];
     blockDataAppendColInfo(pResBlock, &colInfo);
   }
@@ -1529,8 +1528,8 @@ static bool tryCopyDistinctRowFromFileBlock(STsdbReader* pReader, SBlockData* pB
   // opt version
   // 1. it is not a border point
   // 2. the direct next point is not an duplicated timestamp
-  if ((pDumpInfo->rowIndex < pDumpInfo->totalRows - 1 && pReader->order == TSDB_ORDER_ASC) ||
+  bool asc = (pReader->order == TSDB_ORDER_ASC);
-      (pDumpInfo->rowIndex > 0 && pReader->order == TSDB_ORDER_DESC)) {
+  if ((pDumpInfo->rowIndex < pDumpInfo->totalRows - 1 && asc) || (pDumpInfo->rowIndex > 0 && (!asc))) {
     int32_t step = pReader->order == TSDB_ORDER_ASC ? 1 : -1;
 
     int64_t nextKey = pBlockData->aTSKEY[pDumpInfo->rowIndex + step];
@@ -1749,7 +1748,7 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo*
     return code;
   }
 
-  doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo->uid);
+  doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo);
 
   taosMemoryFree(pTSRow);
   tRowMergerClear(&merge);
@@ -1770,6 +1769,7 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
   // only last block exists
   if ((!mergeBlockData) || (tsLastBlock != pBlockData->aTSKEY[pDumpInfo->rowIndex])) {
     if (tryCopyDistinctRowFromSttBlock(&fRow, pLastBlockReader, pBlockScanInfo, tsLastBlock, pReader)) {
+      pBlockScanInfo->lastKey = tsLastBlock;
       return TSDB_CODE_SUCCESS;
     } else {
       int32_t code = tRowMergerInit(&merge, &fRow, pReader->pSchema);
@@ -1786,7 +1786,7 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
         return code;
       }
 
-      doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo->uid);
+      doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo);
 
       taosMemoryFree(pTSRow);
       tRowMergerClear(&merge);
@@ -1810,7 +1810,7 @@ static int32_t doMergeFileBlockAndLastBlock(SLastBlockReader* pLastBlockReader,
       return code;
     }
 
-    doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo->uid);
+    doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo);
 
     taosMemoryFree(pTSRow);
     tRowMergerClear(&merge);
@@ -1858,7 +1858,7 @@ static int32_t mergeFileBlockAndLastBlock(STsdbReader* pReader, SLastBlockReader
           return code;
         }
 
-        doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo->uid);
+        doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo);
 
         taosMemoryFree(pTSRow);
         tRowMergerClear(&merge);
@@ -2082,7 +2082,7 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo*
     return code;
   }
 
-  doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo->uid);
+  doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo);
 
   taosMemoryFree(pTSRow);
   tRowMergerClear(&merge);
@@ -2233,6 +2233,7 @@ int32_t mergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pBloc
                               STsdbReader* pReader) {
   SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
   if (tryCopyDistinctRowFromFileBlock(pReader, pBlockData, key, pDumpInfo)) {
+    pBlockScanInfo->lastKey = key;
     return TSDB_CODE_SUCCESS;
   } else {
     TSDBROW fRow = tsdbRowFromBlockData(pBlockData, pDumpInfo->rowIndex);
@@ -2251,7 +2252,7 @@ int32_t mergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pBloc
       return code;
     }
 
-    doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo->uid);
+    doAppendRowFromTSRow(pReader->pResBlock, pReader, pTSRow, pBlockScanInfo);
 
     taosMemoryFree(pTSRow);
     tRowMergerClear(&merge);
@@ -2299,29 +2300,32 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
   SFileDataBlockInfo* pBlockInfo = getCurrentBlockInfo(&pReader->status.blockIter);
   SLastBlockReader*   pLastBlockReader = pReader->status.fileIter.pLastBlockReader;
 
+  bool    asc = ASCENDING_TRAVERSE(pReader->order);
   int64_t st = taosGetTimestampUs();
+  int32_t step = asc ? 1 : -1;
 
   STableBlockScanInfo* pBlockScanInfo = NULL;
   if (pBlockInfo != NULL) {
-    pBlockScanInfo = *(STableBlockScanInfo**)taosHashGet(pReader->status.pTableMap, &pBlockInfo->uid, sizeof(pBlockInfo->uid));
+    void* p = taosHashGet(pReader->status.pTableMap, &pBlockInfo->uid, sizeof(pBlockInfo->uid));
-    if (pBlockScanInfo == NULL) {
+    if (p == NULL) {
       code = TSDB_CODE_INVALID_PARA;
       tsdbError("failed to locate the uid:%" PRIu64 " in query table uid list, total tables:%d, %s", pBlockInfo->uid,
                 taosHashGetSize(pReader->status.pTableMap), pReader->idStr);
       goto _end;
     }
 
+    pBlockScanInfo = *(STableBlockScanInfo**) p;
+
     SDataBlk* pBlock = getCurrentBlock(&pReader->status.blockIter);
     TSDBKEY   keyInBuf = getCurrentKeyInBuf(pBlockScanInfo, pReader);
 
     // it is a clean block, load it directly
     if (isCleanFileDataBlock(pReader, pBlockInfo, pBlock, pBlockScanInfo, keyInBuf, pLastBlockReader)) {
-      if (pReader->order == TSDB_ORDER_ASC ||
+      if (asc || ((!asc) && (!hasDataInLastBlock(pLastBlockReader)))) {
-          (pReader->order == TSDB_ORDER_DESC && (!hasDataInLastBlock(pLastBlockReader)))) {
         copyBlockDataToSDataBlock(pReader, pBlockScanInfo);
 
         // record the last key value
-        pBlockScanInfo->lastKey = ASCENDING_TRAVERSE(pReader->order)? pBlock->maxKey.ts:pBlock->minKey.ts;
+        pBlockScanInfo->lastKey = asc? pBlock->maxKey.ts:pBlock->minKey.ts;
         goto _end;
       }
     }
@@ -2331,7 +2335,6 @@ static int32_t buildComposedDataBlock(STsdbReader* pReader) {
 
   SFileBlockDumpInfo* pDumpInfo = &pReader->status.fBlockDumpInfo;
   SBlockData*         pBlockData = &pReader->status.fileBlockData;
-  int32_t             step = ASCENDING_TRAVERSE(pReader->order) ? 1 : -1;
 
   while (1) {
     bool hasBlockData = false;
@@ -3220,7 +3223,6 @@ int32_t doMergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pSc
 
 int32_t doMergeRowsInLastBlock(SLastBlockReader* pLastBlockReader, STableBlockScanInfo* pScanInfo, int64_t ts,
                                SRowMerger* pMerger, SVersionRange* pVerRange) {
-  pScanInfo->lastKey = ts;
   while (nextRowFromLastBlocks(pLastBlockReader, pScanInfo, pVerRange)) {
     int64_t next1 = getCurrentKeyInLastBlock(pLastBlockReader);
     if (next1 == ts) {
@@ -3413,9 +3415,10 @@ int32_t tsdbGetNextRowInMem(STableBlockScanInfo* pBlockScanInfo, STsdbReader* pR
   return TSDB_CODE_SUCCESS;
 }
 
-int32_t doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, STSRow* pTSRow, uint64_t uid) {
+int32_t doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, STSRow* pTSRow, STableBlockScanInfo* pScanInfo) {
   int32_t numOfRows = pBlock->info.rows;
   int32_t numOfCols = (int32_t)taosArrayGetSize(pBlock->pDataBlock);
+  int64_t uid = pScanInfo->uid;
 
   SBlockLoadSuppInfo* pSupInfo = &pReader->suppInfo;
   STSchema*           pSchema = doGetSchemaForTSRow(pTSRow->sver, pReader, uid);
@@ -3454,6 +3457,7 @@ int32_t doAppendRowFromTSRow(SSDataBlock* pBlock, STsdbReader* pReader, STSRow*
   }
 
   pBlock->info.rows += 1;
+  pScanInfo->lastKey = pTSRow->ts;
   return TSDB_CODE_SUCCESS;
 }
 
@@ -3517,7 +3521,8 @@ int32_t buildDataBlockFromBufImpl(STableBlockScanInfo* pBlockScanInfo, int64_t e
       break;
     }
 
-    doAppendRowFromTSRow(pBlock, pReader, pTSRow, pBlockScanInfo->uid);
+    doAppendRowFromTSRow(pBlock, pReader, pTSRow, pBlockScanInfo);
+
     if (freeTSRow) {
       taosMemoryFree(pTSRow);
     }

source/dnode/vnode/src/vnd/vnodeOpen.c

@@ -242,12 +242,7 @@ _err:
   return NULL;
 }
 
-void vnodePreClose(SVnode *pVnode) {
+void vnodePreClose(SVnode *pVnode) { vnodeSyncPreClose(pVnode); }
-  if (pVnode) {
-    syncLeaderTransfer(pVnode->sync);
-    syncPreStop(pVnode->sync);
-  }
-}
 
 void vnodeClose(SVnode *pVnode) {
   if (pVnode) {

source/dnode/vnode/src/vnd/vnodeSvr.c

@@ -229,8 +229,7 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
       break;
     /* TQ */
     case TDMT_VND_TMQ_SUBSCRIBE:
-      if (tqProcessSubscribeReq(pVnode->pTq, version, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)),
+      if (tqProcessSubscribeReq(pVnode->pTq, version, pReq, len) < 0) {
-                                pMsg->contLen - sizeof(SMsgHead)) < 0) {
         goto _err;
       }
       break;
@@ -240,26 +239,22 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
       }
       break;
     case TDMT_VND_TMQ_COMMIT_OFFSET:
-      if (tqProcessOffsetCommitReq(pVnode->pTq, version, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)),
+      if (tqProcessOffsetCommitReq(pVnode->pTq, version, pReq, pMsg->contLen - sizeof(SMsgHead)) < 0) {
-                                   pMsg->contLen - sizeof(SMsgHead)) < 0) {
         goto _err;
       }
       break;
     case TDMT_VND_TMQ_ADD_CHECKINFO:
-      if (tqProcessAddCheckInfoReq(pVnode->pTq, version, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)),
+      if (tqProcessAddCheckInfoReq(pVnode->pTq, version, pReq, len) < 0) {
-                                   pMsg->contLen - sizeof(SMsgHead)) < 0) {
         goto _err;
       }
       break;
     case TDMT_VND_TMQ_DEL_CHECKINFO:
-      if (tqProcessDelCheckInfoReq(pVnode->pTq, version, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)),
+      if (tqProcessDelCheckInfoReq(pVnode->pTq, version, pReq, len) < 0) {
-                                   pMsg->contLen - sizeof(SMsgHead)) < 0) {
         goto _err;
       }
       break;
     case TDMT_STREAM_TASK_DEPLOY: {
-      if (tqProcessTaskDeployReq(pVnode->pTq, version, POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)),
+      if (tqProcessTaskDeployReq(pVnode->pTq, version, pReq, len) < 0) {
-                                 pMsg->contLen - sizeof(SMsgHead)) < 0) {
         goto _err;
       }
     } break;
@@ -273,6 +268,11 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
         goto _err;
       }
     } break;
+    case TDMT_STREAM_TASK_CHECK_RSP: {
+      if (tqProcessStreamTaskCheckRsp(pVnode->pTq, version, pReq, len) < 0) {
+        goto _err;
+      }
+    } break;
     case TDMT_VND_ALTER_CONFIRM:
       vnodeProcessAlterConfirmReq(pVnode, version, pReq, len, pRsp);
       break;
@@ -334,7 +334,8 @@ int32_t vnodePreprocessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg) {
 
 int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg) {
   vTrace("message in vnode query queue is processing");
-  if ((pMsg->msgType == TDMT_SCH_QUERY) && !vnodeIsLeader(pVnode)) {
+  // if ((pMsg->msgType == TDMT_SCH_QUERY) && !vnodeIsLeader(pVnode)) {
+  if ((pMsg->msgType == TDMT_SCH_QUERY) && !syncIsReadyForRead(pVnode->sync)) {
     vnodeRedirectRpcMsg(pVnode, pMsg);
     return 0;
   }
@@ -356,7 +357,8 @@ int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
   vTrace("vgId:%d, msg:%p in fetch queue is processing", pVnode->config.vgId, pMsg);
   if ((pMsg->msgType == TDMT_SCH_FETCH || pMsg->msgType == TDMT_VND_TABLE_META || pMsg->msgType == TDMT_VND_TABLE_CFG ||
        pMsg->msgType == TDMT_VND_BATCH_META) &&
-      !vnodeIsLeader(pVnode)) {
+      !syncIsReadyForRead(pVnode->sync)) {
+    //      !vnodeIsLeader(pVnode)) {
     vnodeRedirectRpcMsg(pVnode, pMsg);
     return 0;
   }
@@ -388,10 +390,12 @@ int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
       return tqProcessPollReq(pVnode->pTq, pMsg);
     case TDMT_STREAM_TASK_RUN:
       return tqProcessTaskRunReq(pVnode->pTq, pMsg);
-#if 0
+#if 1
     case TDMT_STREAM_TASK_DISPATCH:
       return tqProcessTaskDispatchReq(pVnode->pTq, pMsg, true);
 #endif
+    case TDMT_STREAM_TASK_CHECK:
+      return tqProcessStreamTaskCheckReq(pVnode->pTq, pMsg);
     case TDMT_STREAM_TASK_DISPATCH_RSP:
       return tqProcessTaskDispatchRsp(pVnode->pTq, pMsg);
     case TDMT_STREAM_RETRIEVE:

source/dnode/vnode/src/vnd/vnodeSync.c

@@ -342,52 +342,26 @@ static void vnodeSyncRollBackMsg(const SSyncFSM *pFsm, const SRpcMsg *pMsg, cons
          TMSG_INFO(pMsg->msgType));
 }
 
-#define USE_TSDB_SNAPSHOT
-
 static int32_t vnodeSnapshotStartRead(const SSyncFSM *pFsm, void *pParam, void **ppReader) {
-#ifdef USE_TSDB_SNAPSHOT
   SVnode         *pVnode = pFsm->data;
   SSnapshotParam *pSnapshotParam = pParam;
   int32_t code = vnodeSnapReaderOpen(pVnode, pSnapshotParam->start, pSnapshotParam->end, (SVSnapReader **)ppReader);
   return code;
-#else
-  *ppReader = taosMemoryMalloc(32);
-  return 0;
-#endif
 }
 
 static int32_t vnodeSnapshotStopRead(const SSyncFSM *pFsm, void *pReader) {
-#ifdef USE_TSDB_SNAPSHOT
   SVnode *pVnode = pFsm->data;
   int32_t code = vnodeSnapReaderClose(pReader);
   return code;
-#else
-  taosMemoryFree(pReader);
-  return 0;
-#endif
 }
 
 static int32_t vnodeSnapshotDoRead(const SSyncFSM *pFsm, void *pReader, void **ppBuf, int32_t *len) {
-#ifdef USE_TSDB_SNAPSHOT
   SVnode *pVnode = pFsm->data;
   int32_t code = vnodeSnapRead(pReader, (uint8_t **)ppBuf, len);
   return code;
-#else
-  static int32_t times = 0;
-  if (times++ < 5) {
-    *len = 64;
-    *ppBuf = taosMemoryMalloc(*len);
-    snprintf(*ppBuf, *len, "snapshot block %d", times);
-  } else {
-    *len = 0;
-    *ppBuf = NULL;
-  }
-  return 0;
-#endif
 }
 
 static int32_t vnodeSnapshotStartWrite(const SSyncFSM *pFsm, void *pParam, void **ppWriter) {
-#ifdef USE_TSDB_SNAPSHOT
   SVnode         *pVnode = pFsm->data;
   SSnapshotParam *pSnapshotParam = pParam;
 
@@ -404,14 +378,9 @@ static int32_t vnodeSnapshotStartWrite(const SSyncFSM *pFsm, void *pParam, void
 
   int32_t code = vnodeSnapWriterOpen(pVnode, pSnapshotParam->start, pSnapshotParam->end, (SVSnapWriter **)ppWriter);
   return code;
-#else
-  *ppWriter = taosMemoryMalloc(32);
-  return 0;
-#endif
 }
 
 static int32_t vnodeSnapshotStopWrite(const SSyncFSM *pFsm, void *pWriter, bool isApply, SSnapshot *pSnapshot) {
-#ifdef USE_TSDB_SNAPSHOT
   SVnode *pVnode = pFsm->data;
   vInfo("vgId:%d, stop write vnode snapshot, apply:%d, index:%" PRId64 " term:%" PRIu64 " config:%" PRId64,
         pVnode->config.vgId, isApply, pSnapshot->lastApplyIndex, pSnapshot->lastApplyTerm, pSnapshot->lastConfigIndex);
@@ -419,22 +388,14 @@ static int32_t vnodeSnapshotStopWrite(const SSyncFSM *pFsm, void *pWriter, bool
   int32_t code = vnodeSnapWriterClose(pWriter, !isApply, pSnapshot);
   vInfo("vgId:%d, apply vnode snapshot finished, code:0x%x", pVnode->config.vgId, code);
   return code;
-#else
-  taosMemoryFree(pWriter);
-  return 0;
-#endif
 }
 
 static int32_t vnodeSnapshotDoWrite(const SSyncFSM *pFsm, void *pWriter, void *pBuf, int32_t len) {
-#ifdef USE_TSDB_SNAPSHOT
   SVnode *pVnode = pFsm->data;
   vDebug("vgId:%d, continue write vnode snapshot, len:%d", pVnode->config.vgId, len);
   int32_t code = vnodeSnapWrite(pWriter, pBuf, len);
   vDebug("vgId:%d, continue write vnode snapshot finished, len:%d", pVnode->config.vgId, len);
   return code;
-#else
-  return 0;
-#endif
 }
 
 static void vnodeRestoreFinish(const SSyncFSM *pFsm) {
@@ -461,7 +422,6 @@ static void vnodeBecomeFollower(const SSyncFSM *pFsm) {
   SVnode *pVnode = pFsm->data;
   vDebug("vgId:%d, become follower", pVnode->config.vgId);
 
-  // clear old leader resource
   taosThreadMutexLock(&pVnode->lock);
   if (pVnode->blocked) {
     pVnode->blocked = false;
@@ -474,15 +434,6 @@ static void vnodeBecomeFollower(const SSyncFSM *pFsm) {
 static void vnodeBecomeLeader(const SSyncFSM *pFsm) {
   SVnode *pVnode = pFsm->data;
   vDebug("vgId:%d, become leader", pVnode->config.vgId);
-
-#if 0
-  taosThreadMutexLock(&pVnode->lock);
-  if (pVnode->blocked) {
-    pVnode->blocked = false;
-    tsem_post(&pVnode->syncSem);
-  }
-  taosThreadMutexUnlock(&pVnode->lock);
-#endif
 }
 
 static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) {
@@ -543,12 +494,25 @@ int32_t vnodeSyncOpen(SVnode *pVnode, char *path) {
 }
 
 void vnodeSyncStart(SVnode *pVnode) {
-  vDebug("vgId:%d, start sync", pVnode->config.vgId);
+  vInfo("vgId:%d, start sync", pVnode->config.vgId);
   syncStart(pVnode->sync);
 }
 
+void vnodeSyncPreClose(SVnode *pVnode) {
+  vInfo("vgId:%d, pre close sync", pVnode->config.vgId);
+  syncLeaderTransfer(pVnode->sync);
+  syncPreStop(pVnode->sync);
+  taosThreadMutexLock(&pVnode->lock);
+  if (pVnode->blocked) {
+    vInfo("vgId:%d, post block after close sync", pVnode->config.vgId);
+    pVnode->blocked = false;
+    tsem_post(&pVnode->syncSem);
+  }
+  taosThreadMutexUnlock(&pVnode->lock);
+}
+
 void vnodeSyncClose(SVnode *pVnode) {
-  vDebug("vgId:%d, close sync", pVnode->config.vgId);
+  vInfo("vgId:%d, close sync", pVnode->config.vgId);
   syncStop(pVnode->sync);
 }
 

source/libs/executor/src/executil.c

@@ -421,7 +421,7 @@ static SColumnInfoData* getColInfoResult(void* metaHandle, int64_t suid, SArray*
   }
 
   for (int32_t i = 0; i < taosArrayGetSize(ctx.cInfoList); ++i) {
-    SColumnInfoData colInfo = {{0}, 0};
+    SColumnInfoData colInfo = {0, {0}};
     colInfo.info = *(SColumnInfo*)taosArrayGet(ctx.cInfoList, i);
     blockDataAppendColInfo(pResBlock, &colInfo);
   }
@@ -582,7 +582,7 @@ int32_t getColInfoResultForGroupby(void* metaHandle, SNodeList* group, STableLis
   }
 
   for (int32_t i = 0; i < taosArrayGetSize(ctx.cInfoList); ++i) {
-    SColumnInfoData colInfo = {{0}, 0};
+    SColumnInfoData colInfo = {0, {0}};
     colInfo.info = *(SColumnInfo*)taosArrayGet(ctx.cInfoList, i);
     blockDataAppendColInfo(pResBlock, &colInfo);
   }
@@ -1495,10 +1495,6 @@ void relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SArray
   while (i < numOfSrcCols && j < taosArrayGetSize(pColMatchInfo)) {
     SColumnInfoData* p = taosArrayGet(pCols, i);
     SColMatchItem*   pmInfo = taosArrayGet(pColMatchInfo, j);
-    /*    if (!outputEveryColumn && pmInfo->reserved) {
-          j++;
-          continue;
-        }*/
 
     if (p->info.colId == pmInfo->colId) {
       SColumnInfoData* pDst = taosArrayGet(pBlock->pDataBlock, pmInfo->dstSlotId);

source/libs/executor/src/executorimpl.c

@@ -2783,8 +2783,10 @@ int32_t getBufferPgSize(int32_t rowSize, uint32_t* defaultPgsz, uint32_t* defaul
     *defaultPgsz <<= 1u;
   }
 
+  // The default buffer for each operator in query is 10MB.
   // at least four pages need to be in buffer
-  *defaultBufsz = 4096 * 256;
+  // TODO: make this variable to be configurable.
+  *defaultBufsz = 4096 * 2560;
   if ((*defaultBufsz) <= (*defaultPgsz)) {
     (*defaultBufsz) = (*defaultPgsz) * 4;
   }
@@ -2971,7 +2973,7 @@ SOperatorInfo* createAggregateOperatorInfo(SOperatorInfo* downstream, SAggPhysiN
 
   return pOperator;
 
-  _error:
+_error:
   if (pInfo != NULL) {
     destroyAggOperatorInfo(pInfo);
   }
@@ -3187,11 +3189,12 @@ SOperatorInfo* createFillOperatorInfo(SOperatorInfo* downstream, SFillPhysiNode*
   code = appendDownstream(pOperator, &downstream, 1);
   return pOperator;
 
-  _error:
+_error:
   if (pInfo != NULL) {
     destroyFillOperatorInfo(pInfo);
   }
 
+  pTaskInfo->code = code;
   taosMemoryFreeClear(pOperator);
   return NULL;
 }

source/libs/executor/src/groupoperator.c

@@ -750,6 +750,14 @@ static void destroyPartitionOperatorInfo(void* param) {
   taosArrayDestroy(pInfo->pGroupColVals);
   taosMemoryFree(pInfo->keyBuf);
   taosArrayDestroy(pInfo->sortedGroupArray);
+
+  void* pGroupIter = taosHashIterate(pInfo->pGroupSet, NULL);
+  while (pGroupIter != NULL) {
+    SDataGroupInfo* pGroupInfo = pGroupIter;
+    taosArrayDestroy(pGroupInfo->pPageList);
+    pGroupIter = taosHashIterate(pInfo->pGroupSet, pGroupIter);
+  }
+
   taosHashCleanup(pInfo->pGroupSet);
   taosMemoryFree(pInfo->columnOffset);
 

source/libs/executor/src/scanoperator.c

@@ -1888,11 +1888,15 @@ static SSDataBlock* doStreamScan(SOperatorInfo* pOperator) {
     STableScanInfo* pTSInfo = pInfo->pTableScanOp->info;
     memcpy(&pTSInfo->cond, &pTaskInfo->streamInfo.tableCond, sizeof(SQueryTableDataCond));
     if (pTaskInfo->streamInfo.recoverStep == STREAM_RECOVER_STEP__PREPARE1) {
-      pTSInfo->cond.startVersion = -1;
+      pTSInfo->cond.startVersion = 0;
       pTSInfo->cond.endVersion = pTaskInfo->streamInfo.fillHistoryVer1;
+      qDebug("stream recover step 1, from %" PRId64 " to %" PRId64, pTSInfo->cond.startVersion,
+             pTSInfo->cond.endVersion);
     } else {
       pTSInfo->cond.startVersion = pTaskInfo->streamInfo.fillHistoryVer1 + 1;
       pTSInfo->cond.endVersion = pTaskInfo->streamInfo.fillHistoryVer2;
+      qDebug("stream recover step 2, from %" PRId64 " to %" PRId64, pTSInfo->cond.startVersion,
+             pTSInfo->cond.endVersion);
     }
 
     /*resetTableScanInfo(pTSInfo, pWin);*/
@@ -1909,11 +1913,15 @@ static SSDataBlock* doStreamScan(SOperatorInfo* pOperator) {
     if (pBlock != NULL) {
       calBlockTbName(&pInfo->tbnameCalSup, pBlock);
       updateInfoFillBlockData(pInfo->pUpdateInfo, pBlock, pInfo->primaryTsIndex);
+      qDebug("stream recover scan get block, rows %d", pBlock->info.rows);
       return pBlock;
     }
     pTaskInfo->streamInfo.recoverStep = STREAM_RECOVER_STEP__NONE;
     STableScanInfo* pTSInfo = pInfo->pTableScanOp->info;
-    pTSInfo->cond.startVersion = 0;
+    tsdbReaderClose(pTSInfo->dataReader);
+    pTSInfo->dataReader = NULL;
+
+    pTSInfo->cond.startVersion = -1;
     pTSInfo->cond.endVersion = -1;
 
     return NULL;

source/libs/executor/src/sortoperator.c

@@ -579,21 +579,8 @@ int32_t doOpenMultiwayMergeOperator(SOperatorInfo* pOperator) {
   return TSDB_CODE_SUCCESS;
 }
 
-SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity,
+static void doGetSortedBlockData(SMultiwayMergeOperatorInfo* pInfo, SSortHandle* pHandle, int32_t capacity, SSDataBlock* p) {
-                                        SArray* pColMatchInfo, SOperatorInfo* pOperator) {
-  SMultiwayMergeOperatorInfo* pInfo = pOperator->info;
-  SExecTaskInfo*              pTaskInfo = pOperator->pTaskInfo;
 
-  blockDataCleanup(pDataBlock);
-
-  SSDataBlock* p = tsortGetSortedDataBlock(pHandle);
-  if (p == NULL) {
-    return NULL;
-  }
-
-  blockDataEnsureCapacity(p, capacity);
-
-_retry:
   while (1) {
     STupleHandle* pTupleHandle = NULL;
     if (pInfo->groupSort) {
@@ -638,22 +625,48 @@ _retry:
     pInfo->hasGroupId = false;
   }
 
-  if (p->info.rows > 0) {  // todo extract method
+}
-    applyLimitOffset(&pInfo->limitInfo, p, pTaskInfo, pOperator);
+
-    if (p->info.rows == 0) {
+SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity,
-      goto _retry;
+                                        SArray* pColMatchInfo, SOperatorInfo* pOperator) {
+  SMultiwayMergeOperatorInfo* pInfo = pOperator->info;
+
+  SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
+  blockDataCleanup(pDataBlock);
+
+  SSDataBlock* p = tsortGetSortedDataBlock(pHandle);
+  if (p == NULL) {
+    return NULL;
   }
 
+  blockDataEnsureCapacity(p, capacity);
+
+  while (1) {
+    doGetSortedBlockData(pInfo, pHandle, capacity, p);
+    if (p->info.rows == 0) {
+      break;
+    }
+
+    if (p->info.rows > 0) {
+      applyLimitOffset(&pInfo->limitInfo, p, pTaskInfo, pOperator);
+      if (p->info.rows > 0) {
+        break;
+      }
+    }
+  }
+
+  if (p->info.rows > 0) {
     blockDataEnsureCapacity(pDataBlock, p->info.rows);
+
     int32_t numOfCols = taosArrayGetSize(pColMatchInfo);
     for (int32_t i = 0; i < numOfCols; ++i) {
       SColMatchItem* pmInfo = taosArrayGet(pColMatchInfo, i);
-      //      ASSERT(pColMatchInfo-> == COL_MATCH_FROM_SLOT_ID);
 
       SColumnInfoData* pSrc = taosArrayGet(p->pDataBlock, pmInfo->srcSlotId);
       SColumnInfoData* pDst = taosArrayGet(pDataBlock->pDataBlock, pmInfo->dstSlotId);
       colDataAssign(pDst, pSrc, p->info.rows, &pDataBlock->info);
     }
+
     pInfo->limitInfo.numOfOutputRows += p->info.rows;
     pDataBlock->info.rows = p->info.rows;
     pDataBlock->info.groupId = pInfo->groupId;

source/libs/executor/src/timewindowoperator.c

@@ -985,7 +985,8 @@ void doCloseWindow(SResultRowInfo* pResultRowInfo, const SIntervalAggOperatorInf
   // current result is done in computing final results.
   if (pInfo->timeWindowInterpo && isResultRowInterpolated(pResult, RESULT_ROW_END_INTERP)) {
     closeResultRow(pResult);
-    tdListPopHead(pResultRowInfo->openWindow);
+    SListNode *pNode = tdListPopHead(pResultRowInfo->openWindow);
+    taosMemoryFree(pNode);
   }
 }
 

source/libs/function/src/builtinsimpl.c

@@ -5451,6 +5451,8 @@ int32_t modeFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
     colDataAppendNULL(pCol, currentRow);
   }
 
+  taosHashCleanup(pInfo->pHash);
+
   return pResInfo->numOfRes;
 }
 

source/libs/qworker/src/qworker.c

@@ -644,11 +644,13 @@ _return:
   input.msgType = qwMsg->msgType;
   code = qwHandlePostPhaseEvents(QW_FPARAMS(), QW_PHASE_POST_QUERY, &input, NULL);
 
-  if (ctx != NULL && QW_EVENT_RECEIVED(ctx, QW_EVENT_FETCH)) {
+  if (QUERY_RSP_POLICY_QUICK == tsQueryRspPolicy && ctx != NULL && QW_EVENT_RECEIVED(ctx, QW_EVENT_FETCH)) {
     void         *rsp = NULL;
     int32_t       dataLen = 0;
     SOutputData sOutput = {0};
-    QW_ERR_JRET(qwGetQueryResFromSink(QW_FPARAMS(), ctx, &dataLen, &rsp, &sOutput));
+    if (qwGetQueryResFromSink(QW_FPARAMS(), ctx, &dataLen, &rsp, &sOutput)) {
+      return TSDB_CODE_SUCCESS;
+    }
 
     if (rsp) {
       bool qComplete = (DS_BUF_EMPTY == sOutput.bufStatus && sOutput.queryEnd);

source/libs/scheduler/src/schRemote.c

@@ -185,12 +185,12 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t execId, SDa
               code = rsp->code;
             }
           }
-          SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
 
           if (taosArrayGetSize((SArray*)pJob->execRes.res) <= 0) {        
             taosArrayDestroy((SArray*)pJob->execRes.res);
             pJob->execRes.res = NULL;
           }
+          SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
         }
         
         tDecoderClear(&coder);
@@ -1047,6 +1047,7 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr,
 
       SSubQueryMsg *pMsg = msg;
       pMsg->header.vgId = htonl(addr->nodeId);
+      pMsg->header.msgMask = htonl((pTask->plan->showRewrite) ? SHOW_REWRITE_MASK() : 0);
       pMsg->sId = htobe64(schMgmt.sId);
       pMsg->queryId = htobe64(pJob->queryId);
       pMsg->taskId = htobe64(pTask->taskId);

source/libs/stream/inc/streamInc.h

@@ -43,6 +43,8 @@ int32_t streamBroadcastToChildren(SStreamTask* pTask, const SSDataBlock* pBlock)
 
 int32_t tEncodeStreamRetrieveReq(SEncoder* pEncoder, const SStreamRetrieveReq* pReq);
 
+int32_t streamDispatchOneCheckReq(SStreamTask* pTask, const SStreamTaskCheckReq* pReq, int32_t nodeId, SEpSet* pEpSet);
+
 int32_t streamDispatchOneRecoverFinishReq(SStreamTask* pTask, const SStreamRecoverFinishReq* pReq, int32_t vgId,
                                           SEpSet* pEpSet);
 

source/libs/stream/src/stream.c

@@ -135,8 +135,11 @@ int32_t streamTaskEnqueue(SStreamTask* pTask, const SStreamDispatchReq* pReq, SR
   ((SMsgHead*)buf)->vgId = htonl(pReq->upstreamNodeId);
   SStreamDispatchRsp* pCont = POINTER_SHIFT(buf, sizeof(SMsgHead));
   pCont->inputStatus = status;
-  pCont->streamId = pReq->streamId;
+  pCont->streamId = htobe64(pReq->streamId);
-  pCont->taskId = pReq->upstreamTaskId;
+  pCont->upstreamNodeId = htonl(pReq->upstreamNodeId);
+  pCont->upstreamTaskId = htonl(pReq->upstreamTaskId);
+  pCont->downstreamNodeId = htonl(pTask->nodeId);
+  pCont->downstreamTaskId = htonl(pTask->taskId);
   pRsp->pCont = buf;
   pRsp->contLen = sizeof(SMsgHead) + sizeof(SStreamDispatchRsp);
   tmsgSendRsp(pRsp);
@@ -203,10 +206,10 @@ int32_t streamProcessDispatchReq(SStreamTask* pTask, SStreamDispatchReq* pReq, S
   return 0;
 }
 
-int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp) {
+int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, int32_t code) {
   ASSERT(pRsp->inputStatus == TASK_OUTPUT_STATUS__NORMAL || pRsp->inputStatus == TASK_OUTPUT_STATUS__BLOCKED);
 
-  qDebug("task %d receive dispatch rsp", pTask->taskId);
+  qDebug("task %d receive dispatch rsp, code: %x", pTask->taskId, code);
 
   if (pTask->outputType == TASK_OUTPUT__SHUFFLE_DISPATCH) {
     int32_t leftRsp = atomic_sub_fetch_32(&pTask->shuffleDispatcher.waitingRspCnt, 1);

source/libs/stream/src/streamDispatch.c

@@ -210,6 +210,46 @@ static int32_t streamAddBlockToDispatchMsg(const SSDataBlock* pBlock, SStreamDis
   return 0;
 }
 
+int32_t streamDispatchOneCheckReq(SStreamTask* pTask, const SStreamTaskCheckReq* pReq, int32_t nodeId, SEpSet* pEpSet) {
+  void*   buf = NULL;
+  int32_t code = -1;
+  SRpcMsg msg = {0};
+
+  int32_t tlen;
+  tEncodeSize(tEncodeSStreamTaskCheckReq, pReq, tlen, code);
+  if (code < 0) {
+    return -1;
+  }
+
+  buf = rpcMallocCont(sizeof(SMsgHead) + tlen);
+  if (buf == NULL) {
+    return -1;
+  }
+
+  ((SMsgHead*)buf)->vgId = htonl(nodeId);
+  void* abuf = POINTER_SHIFT(buf, sizeof(SMsgHead));
+
+  SEncoder encoder;
+  tEncoderInit(&encoder, abuf, tlen);
+  if ((code = tEncodeSStreamTaskCheckReq(&encoder, pReq)) < 0) {
+    goto FAIL;
+  }
+  tEncoderClear(&encoder);
+
+  msg.contLen = tlen + sizeof(SMsgHead);
+  msg.pCont = buf;
+  msg.msgType = TDMT_STREAM_TASK_CHECK;
+
+  qDebug("dispatch from task %d to task %d node %d: check msg", pTask->taskId, pReq->downstreamTaskId, nodeId);
+
+  tmsgSendReq(pEpSet, &msg);
+
+  return 0;
+FAIL:
+  if (buf) rpcFreeCont(buf);
+  return code;
+}
+
 int32_t streamDispatchOneRecoverFinishReq(SStreamTask* pTask, const SStreamRecoverFinishReq* pReq, int32_t vgId,
                                           SEpSet* pEpSet) {
   void*   buf = NULL;
@@ -243,7 +283,8 @@ int32_t streamDispatchOneRecoverFinishReq(SStreamTask* pTask, const SStreamRecov
 
   tmsgSendReq(pEpSet, &msg);
 
-  code = 0;
+  qDebug("dispatch from task %d to task %d node %d: recover finish msg", pTask->taskId, pReq->taskId, vgId);
+
   return 0;
 FAIL:
   if (buf) rpcFreeCont(buf);
@@ -279,7 +320,7 @@ int32_t streamDispatchOneDataReq(SStreamTask* pTask, const SStreamDispatchReq* p
   msg.pCont = buf;
   msg.msgType = pTask->dispatchMsgType;
 
-  qDebug("dispatch from task %d to task %d node %d", pTask->taskId, pReq->taskId, vgId);
+  qDebug("dispatch from task %d to task %d node %d: data msg", pTask->taskId, pReq->taskId, vgId);
 
   tmsgSendReq(pEpSet, &msg);
 

source/libs/stream/src/streamExec.c

@@ -202,83 +202,83 @@ int32_t streamPipelineExec(SStreamTask* pTask, int32_t batchNum, bool dispatch)
 int32_t streamExecForAll(SStreamTask* pTask) {
   while (1) {
     int32_t batchCnt = 1;
-    void*   data = NULL;
+    void*   input = NULL;
     while (1) {
       SStreamQueueItem* qItem = streamQueueNextItem(pTask->inputQueue);
       if (qItem == NULL) {
         qDebug("stream task exec over, queue empty, task: %d", pTask->taskId);
         break;
       }
-      if (data == NULL) {
+      if (input == NULL) {
-        data = qItem;
+        input = qItem;
         streamQueueProcessSuccess(pTask->inputQueue);
         if (pTask->taskLevel == TASK_LEVEL__SINK) {
           break;
         }
       } else {
         void* newRet;
-        if ((newRet = streamMergeQueueItem(data, qItem)) == NULL) {
+        if ((newRet = streamMergeQueueItem(input, qItem)) == NULL) {
           streamQueueProcessFail(pTask->inputQueue);
           break;
         } else {
           batchCnt++;
-          data = newRet;
+          input = newRet;
           streamQueueProcessSuccess(pTask->inputQueue);
         }
       }
     }
 
     if (pTask->taskStatus == TASK_STATUS__DROPPING) {
-      if (data) streamFreeQitem(data);
+      if (input) streamFreeQitem(input);
       return 0;
     }
 
-    if (data == NULL) {
+    if (input == NULL) {
       break;
     }
 
     if (pTask->taskLevel == TASK_LEVEL__SINK) {
-      ASSERT(((SStreamQueueItem*)data)->type == STREAM_INPUT__DATA_BLOCK);
+      ASSERT(((SStreamQueueItem*)input)->type == STREAM_INPUT__DATA_BLOCK);
-      streamTaskOutput(pTask, data);
+      streamTaskOutput(pTask, input);
       continue;
     }
 
     SArray* pRes = taosArrayInit(0, sizeof(SSDataBlock));
 
     qDebug("stream task %d exec begin, msg batch: %d", pTask->taskId, batchCnt);
-    streamTaskExecImpl(pTask, data, pRes);
+    streamTaskExecImpl(pTask, input, pRes);
     qDebug("stream task %d exec end", pTask->taskId);
 
     if (taosArrayGetSize(pRes) != 0) {
       SStreamDataBlock* qRes = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM);
       if (qRes == NULL) {
         taosArrayDestroyEx(pRes, (FDelete)blockDataFreeRes);
-        streamFreeQitem(data);
+        streamFreeQitem(input);
         return -1;
       }
       qRes->type = STREAM_INPUT__DATA_BLOCK;
       qRes->blocks = pRes;
 
-      if (((SStreamQueueItem*)data)->type == STREAM_INPUT__DATA_SUBMIT) {
+      if (((SStreamQueueItem*)input)->type == STREAM_INPUT__DATA_SUBMIT) {
-        SStreamDataSubmit* pSubmit = (SStreamDataSubmit*)data;
+        SStreamDataSubmit* pSubmit = (SStreamDataSubmit*)input;
         qRes->childId = pTask->selfChildId;
         qRes->sourceVer = pSubmit->ver;
-      } else if (((SStreamQueueItem*)data)->type == STREAM_INPUT__MERGED_SUBMIT) {
+      } else if (((SStreamQueueItem*)input)->type == STREAM_INPUT__MERGED_SUBMIT) {
-        SStreamMergedSubmit* pMerged = (SStreamMergedSubmit*)data;
+        SStreamMergedSubmit* pMerged = (SStreamMergedSubmit*)input;
         qRes->childId = pTask->selfChildId;
         qRes->sourceVer = pMerged->ver;
       }
 
       if (streamTaskOutput(pTask, qRes) < 0) {
         taosArrayDestroyEx(pRes, (FDelete)blockDataFreeRes);
-        streamFreeQitem(data);
+        streamFreeQitem(input);
         taosFreeQitem(qRes);
         return -1;
       }
     } else {
       taosArrayDestroy(pRes);
     }
-    streamFreeQitem(data);
+    streamFreeQitem(input);
   }
   return 0;
 }

source/libs/stream/src/streamRecover.c

@@ -15,6 +15,148 @@
 
 #include "streamInc.h"
 
+int32_t streamTaskLaunchRecover(SStreamTask* pTask, int64_t version) {
+  qDebug("task %d at node %d launch recover", pTask->taskId, pTask->nodeId);
+  if (pTask->taskLevel == TASK_LEVEL__SOURCE) {
+    atomic_store_8(&pTask->taskStatus, TASK_STATUS__RECOVER_PREPARE);
+    streamSetParamForRecover(pTask);
+    streamSourceRecoverPrepareStep1(pTask, version);
+
+    SStreamRecoverStep1Req req;
+    streamBuildSourceRecover1Req(pTask, &req);
+    int32_t len = sizeof(SStreamRecoverStep1Req);
+
+    void* serializedReq = rpcMallocCont(len);
+    if (serializedReq == NULL) {
+      return -1;
+    }
+
+    memcpy(serializedReq, &req, len);
+
+    SRpcMsg rpcMsg = {
+        .contLen = len,
+        .pCont = serializedReq,
+        .msgType = TDMT_VND_STREAM_RECOVER_STEP1,
+    };
+
+    if (tmsgPutToQueue(pTask->pMsgCb, STREAM_QUEUE, &rpcMsg) < 0) {
+      /*ASSERT(0);*/
+    }
+
+  } else if (pTask->taskLevel == TASK_LEVEL__AGG) {
+    atomic_store_8(&pTask->taskStatus, TASK_STATUS__NORMAL);
+    streamSetParamForRecover(pTask);
+    streamAggRecoverPrepare(pTask);
+  } else if (pTask->taskLevel == TASK_LEVEL__SINK) {
+    atomic_store_8(&pTask->taskStatus, TASK_STATUS__NORMAL);
+  }
+  return 0;
+}
+
+// checkstatus
+int32_t streamTaskCheckDownstream(SStreamTask* pTask, int64_t version) {
+  SStreamTaskCheckReq req = {
+      .streamId = pTask->streamId,
+      .upstreamTaskId = pTask->taskId,
+      .upstreamNodeId = pTask->nodeId,
+      .childId = pTask->selfChildId,
+  };
+  // serialize
+  if (pTask->outputType == TASK_OUTPUT__FIXED_DISPATCH) {
+    req.reqId = tGenIdPI64();
+    req.downstreamNodeId = pTask->fixedEpDispatcher.nodeId;
+    req.downstreamTaskId = pTask->fixedEpDispatcher.taskId;
+    pTask->checkReqId = req.reqId;
+
+    qDebug("task %d at node %d check downstream task %d at node %d", pTask->taskId, pTask->nodeId, req.downstreamTaskId,
+           req.downstreamNodeId);
+    streamDispatchOneCheckReq(pTask, &req, pTask->fixedEpDispatcher.nodeId, &pTask->fixedEpDispatcher.epSet);
+  } else if (pTask->outputType == TASK_OUTPUT__SHUFFLE_DISPATCH) {
+    SArray* vgInfo = pTask->shuffleDispatcher.dbInfo.pVgroupInfos;
+    int32_t vgSz = taosArrayGetSize(vgInfo);
+    pTask->recoverTryingDownstream = vgSz;
+    pTask->checkReqIds = taosArrayInit(vgSz, sizeof(int64_t));
+
+    for (int32_t i = 0; i < vgSz; i++) {
+      SVgroupInfo* pVgInfo = taosArrayGet(vgInfo, i);
+      req.reqId = tGenIdPI64();
+      taosArrayPush(pTask->checkReqIds, &req.reqId);
+      req.downstreamNodeId = pVgInfo->vgId;
+      req.downstreamTaskId = pVgInfo->taskId;
+      qDebug("task %d at node %d check downstream task %d at node %d (shuffle)", pTask->taskId, pTask->nodeId,
+             req.downstreamTaskId, req.downstreamNodeId);
+      streamDispatchOneCheckReq(pTask, &req, pVgInfo->vgId, &pVgInfo->epSet);
+    }
+  } else {
+    qDebug("task %d at node %d direct launch recover since no downstream", pTask->taskId, pTask->nodeId);
+    streamTaskLaunchRecover(pTask, version);
+  }
+  return 0;
+}
+
+int32_t streamRecheckOneDownstream(SStreamTask* pTask, const SStreamTaskCheckRsp* pRsp) {
+  SStreamTaskCheckReq req = {
+      .reqId = pRsp->reqId,
+      .streamId = pRsp->streamId,
+      .upstreamTaskId = pRsp->upstreamTaskId,
+      .upstreamNodeId = pRsp->upstreamNodeId,
+      .downstreamTaskId = pRsp->downstreamTaskId,
+      .downstreamNodeId = pRsp->downstreamNodeId,
+      .childId = pRsp->childId,
+  };
+  qDebug("task %d at node %d check downstream task %d at node %d (recheck)", pTask->taskId, pTask->nodeId,
+         req.downstreamTaskId, req.downstreamNodeId);
+  if (pTask->outputType == TASK_OUTPUT__FIXED_DISPATCH) {
+    streamDispatchOneCheckReq(pTask, &req, pRsp->downstreamNodeId, &pTask->fixedEpDispatcher.epSet);
+  } else if (pTask->outputType == TASK_OUTPUT__SHUFFLE_DISPATCH) {
+    SArray* vgInfo = pTask->shuffleDispatcher.dbInfo.pVgroupInfos;
+    int32_t vgSz = taosArrayGetSize(vgInfo);
+    for (int32_t i = 0; i < vgSz; i++) {
+      SVgroupInfo* pVgInfo = taosArrayGet(vgInfo, i);
+      if (pVgInfo->taskId == req.downstreamTaskId) {
+        streamDispatchOneCheckReq(pTask, &req, pRsp->downstreamNodeId, &pVgInfo->epSet);
+      }
+    }
+  }
+  return 0;
+}
+
+int32_t streamProcessTaskCheckReq(SStreamTask* pTask, const SStreamTaskCheckReq* pReq) {
+  return atomic_load_8(&pTask->taskStatus) == TASK_STATUS__NORMAL;
+}
+
+int32_t streamProcessTaskCheckRsp(SStreamTask* pTask, const SStreamTaskCheckRsp* pRsp, int64_t version) {
+  qDebug("task %d at node %d recv check rsp from task %d at node %d: status %d", pRsp->upstreamTaskId,
+         pRsp->upstreamNodeId, pRsp->downstreamTaskId, pRsp->downstreamNodeId, pRsp->status);
+  if (pRsp->status == 1) {
+    if (pTask->outputType == TASK_OUTPUT__SHUFFLE_DISPATCH) {
+      bool found = false;
+      for (int32_t i = 0; i < taosArrayGetSize(pTask->checkReqIds); i++) {
+        int64_t reqId = *(int64_t*)taosArrayGet(pTask->checkReqIds, i);
+        if (reqId == pRsp->reqId) {
+          found = true;
+          break;
+        }
+      }
+      if (!found) return -1;
+      int32_t left = atomic_sub_fetch_32(&pTask->recoverTryingDownstream, 1);
+      ASSERT(left >= 0);
+      if (left == 0) {
+        taosArrayDestroy(pTask->checkReqIds);
+        streamTaskLaunchRecover(pTask, version);
+      }
+    } else if (pTask->outputType == TASK_OUTPUT__FIXED_DISPATCH) {
+      if (pRsp->reqId != pTask->checkReqId) return -1;
+      streamTaskLaunchRecover(pTask, version);
+    } else {
+      ASSERT(0);
+    }
+  } else {
+    streamRecheckOneDownstream(pTask, pRsp);
+  }
+  return 0;
+}
+
 // common
 int32_t streamSetParamForRecover(SStreamTask* pTask) {
   void* exec = pTask->exec.executor;
@@ -86,10 +228,7 @@ int32_t streamDispatchRecoverFinishReq(SStreamTask* pTask) {
 // agg
 int32_t streamAggRecoverPrepare(SStreamTask* pTask) {
   void* exec = pTask->exec.executor;
-  /*if (qStreamSetParamForRecover(exec) < 0) {*/
+  pTask->recoverWaitingUpstream = taosArrayGetSize(pTask->childEpInfo);
-  /*return -1;*/
-  /*}*/
-  pTask->recoverWaitingChild = taosArrayGetSize(pTask->childEpInfo);
   return 0;
 }
 
@@ -107,7 +246,7 @@ int32_t streamAggChildrenRecoverFinish(SStreamTask* pTask) {
 
 int32_t streamProcessRecoverFinishReq(SStreamTask* pTask, int32_t childId) {
   if (pTask->taskLevel == TASK_LEVEL__AGG) {
-    int32_t left = atomic_sub_fetch_32(&pTask->recoverWaitingChild, 1);
+    int32_t left = atomic_sub_fetch_32(&pTask->recoverWaitingUpstream, 1);
     ASSERT(left >= 0);
     if (left == 0) {
       streamAggChildrenRecoverFinish(pTask);
@@ -116,6 +255,60 @@ int32_t streamProcessRecoverFinishReq(SStreamTask* pTask, int32_t childId) {
   return 0;
 }
 
+int32_t tEncodeSStreamTaskCheckReq(SEncoder* pEncoder, const SStreamTaskCheckReq* pReq) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->reqId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->upstreamNodeId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->upstreamTaskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->downstreamNodeId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->downstreamTaskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->childId) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeSStreamTaskCheckReq(SDecoder* pDecoder, SStreamTaskCheckReq* pReq) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->reqId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->upstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->upstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->downstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->downstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->childId) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
+int32_t tEncodeSStreamTaskCheckRsp(SEncoder* pEncoder, const SStreamTaskCheckRsp* pRsp) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->reqId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->streamId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->upstreamNodeId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->upstreamTaskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->downstreamNodeId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->downstreamTaskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->childId) < 0) return -1;
+  if (tEncodeI8(pEncoder, pRsp->status) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeSStreamTaskCheckRsp(SDecoder* pDecoder, SStreamTaskCheckRsp* pRsp) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->reqId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->streamId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->upstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->upstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->downstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->downstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->childId) < 0) return -1;
+  if (tDecodeI8(pDecoder, &pRsp->status) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
 int32_t tEncodeSStreamRecoverFinishReq(SEncoder* pEncoder, const SStreamRecoverFinishReq* pReq) {
   if (tStartEncode(pEncoder) < 0) return -1;
   if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
@@ -132,79 +325,6 @@ int32_t tDecodeSStreamRecoverFinishReq(SDecoder* pDecoder, SStreamRecoverFinishR
   tEndDecode(pDecoder);
   return 0;
 }
-#if 0
-int32_t tEncodeStreamTaskRecoverReq(SEncoder* pEncoder, const SStreamTaskRecoverReq* pReq) {
-  if (tStartEncode(pEncoder) < 0) return -1;
-  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pReq->taskId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pReq->upstreamTaskId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pReq->upstreamNodeId) < 0) return -1;
-  tEndEncode(pEncoder);
-  return pEncoder->pos;
-}
-
-int32_t tDecodeStreamTaskRecoverReq(SDecoder* pDecoder, SStreamTaskRecoverReq* pReq) {
-  if (tStartDecode(pDecoder) < 0) return -1;
-  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->taskId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->upstreamTaskId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->upstreamNodeId) < 0) return -1;
-  tEndDecode(pDecoder);
-  return 0;
-}
-
-int32_t tEncodeStreamTaskRecoverRsp(SEncoder* pEncoder, const SStreamTaskRecoverRsp* pRsp) {
-  if (tStartEncode(pEncoder) < 0) return -1;
-  if (tEncodeI64(pEncoder, pRsp->streamId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pRsp->reqTaskId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pRsp->rspTaskId) < 0) return -1;
-  if (tEncodeI8(pEncoder, pRsp->inputStatus) < 0) return -1;
-  tEndEncode(pEncoder);
-  return pEncoder->pos;
-}
-
-int32_t tDecodeStreamTaskRecoverRsp(SDecoder* pDecoder, SStreamTaskRecoverRsp* pReq) {
-  if (tStartDecode(pDecoder) < 0) return -1;
-  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->reqTaskId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->rspTaskId) < 0) return -1;
-  if (tDecodeI8(pDecoder, &pReq->inputStatus) < 0) return -1;
-  tEndDecode(pDecoder);
-  return 0;
-}
-
-int32_t tEncodeSMStreamTaskRecoverReq(SEncoder* pEncoder, const SMStreamTaskRecoverReq* pReq) {
-  if (tStartEncode(pEncoder) < 0) return -1;
-  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pReq->taskId) < 0) return -1;
-  tEndEncode(pEncoder);
-  return pEncoder->pos;
-}
-
-int32_t tDecodeSMStreamTaskRecoverReq(SDecoder* pDecoder, SMStreamTaskRecoverReq* pReq) {
-  if (tStartDecode(pDecoder) < 0) return -1;
-  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->taskId) < 0) return -1;
-  tEndDecode(pDecoder);
-  return 0;
-}
-
-int32_t tEncodeSMStreamTaskRecoverRsp(SEncoder* pEncoder, const SMStreamTaskRecoverRsp* pRsp) {
-  if (tStartEncode(pEncoder) < 0) return -1;
-  if (tEncodeI64(pEncoder, pRsp->streamId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pRsp->taskId) < 0) return -1;
-  tEndEncode(pEncoder);
-  return pEncoder->pos;
-}
-
-int32_t tDecodeSMStreamTaskRecoverRsp(SDecoder* pDecoder, SMStreamTaskRecoverRsp* pReq) {
-  if (tStartDecode(pDecoder) < 0) return -1;
-  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->taskId) < 0) return -1;
-  tEndDecode(pDecoder);
-  return 0;
-}
-#endif
 
 int32_t tEncodeSStreamCheckpointInfo(SEncoder* pEncoder, const SStreamCheckpointInfo* pCheckpoint) {
   if (tEncodeI32(pEncoder, pCheckpoint->srcNodeId) < 0) return -1;
@@ -248,308 +368,3 @@ int32_t tDecodeSStreamMultiVgCheckpointInfo(SDecoder* pDecoder, SStreamMultiVgCh
   }
   return 0;
 }
-
-#if 0
-int32_t tEncodeSStreamTaskRecoverReq(SEncoder* pEncoder, const SStreamRecoverDownstreamReq* pReq) {
-  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pReq->downstreamTaskId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pReq->taskId) < 0) return -1;
-  return 0;
-}
-
-int32_t tDecodeSStreamTaskRecoverReq(SDecoder* pDecoder, SStreamRecoverDownstreamReq* pReq) {
-  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->downstreamTaskId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pReq->taskId) < 0) return -1;
-  return 0;
-}
-
-int32_t tEncodeSStreamTaskRecoverRsp(SEncoder* pEncoder, const SStreamRecoverDownstreamRsp* pRsp) {
-  if (tEncodeI64(pEncoder, pRsp->streamId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pRsp->downstreamTaskId) < 0) return -1;
-  if (tEncodeI32(pEncoder, pRsp->taskId) < 0) return -1;
-  int32_t sz = taosArrayGetSize(pRsp->checkpointVer);
-  if (tEncodeI32(pEncoder, sz) < 0) return -1;
-  for (int32_t i = 0; i < sz; i++) {
-    SStreamCheckpointInfo* pInfo = taosArrayGet(pRsp->checkpointVer, i);
-    if (tEncodeSStreamCheckpointInfo(pEncoder, pInfo) < 0) return -1;
-  }
-  return 0;
-}
-
-int32_t tDecodeSStreamTaskRecoverRsp(SDecoder* pDecoder, SStreamRecoverDownstreamRsp* pRsp) {
-  if (tDecodeI64(pDecoder, &pRsp->streamId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pRsp->downstreamTaskId) < 0) return -1;
-  if (tDecodeI32(pDecoder, &pRsp->taskId) < 0) return -1;
-  int32_t sz;
-  if (tDecodeI32(pDecoder, &sz) < 0) return -1;
-  pRsp->checkpointVer = taosArrayInit(sz, sizeof(SStreamCheckpointInfo));
-  if (pRsp->checkpointVer == NULL) return -1;
-  for (int32_t i = 0; i < sz; i++) {
-    SStreamCheckpointInfo info;
-    if (tDecodeSStreamCheckpointInfo(pDecoder, &info) < 0) return -1;
-    taosArrayPush(pRsp->checkpointVer, &info);
-  }
-  return 0;
-}
-#endif
-
-int32_t streamSaveStateInfo(SStreamMeta* pMeta, SStreamTask* pTask) {
-#if 0
-  void* buf = NULL;
-
-  ASSERT(pTask->taskLevel == TASK_LEVEL__SINK);
-
-  SStreamMultiVgCheckpointInfo checkpoint;
-  checkpoint.checkpointId = atomic_fetch_add_32(&pTask->nextCheckId, 1);
-  checkpoint.checkTs = taosGetTimestampMs();
-  checkpoint.streamId = pTask->streamId;
-  checkpoint.taskId = pTask->taskId;
-  checkpoint.checkpointVer = pTask->checkpointInfo;
-
-  int32_t len;
-  int32_t code;
-  tEncodeSize(tEncodeSStreamMultiVgCheckpointInfo, &checkpoint, len, code);
-  if (code < 0) {
-    return -1;
-  }
-
-  buf = taosMemoryCalloc(1, len);
-  if (buf == NULL) {
-    return -1;
-  }
-  SEncoder encoder;
-  tEncoderInit(&encoder, buf, len);
-  tEncodeSStreamMultiVgCheckpointInfo(&encoder, &checkpoint);
-  tEncoderClear(&encoder);
-
-  SStreamCheckpointKey key = {
-      .taskId = pTask->taskId,
-      .checkpointId = checkpoint.checkpointId,
-  };
-
-  if (tdbTbUpsert(pMeta->pStateDb, &key, sizeof(SStreamCheckpointKey), buf, len, &pMeta->txn) < 0) {
-    ASSERT(0);
-    goto FAIL;
-  }
-
-  int32_t sz = taosArrayGetSize(pTask->checkpointInfo);
-  for (int32_t i = 0; i < sz; i++) {
-    SStreamCheckpointInfo* pCheck = taosArrayGet(pTask->checkpointInfo, i);
-    pCheck->stateSaveVer = pCheck->stateProcessedVer;
-  }
-
-  taosMemoryFree(buf);
-  return 0;
-FAIL:
-  if (buf) taosMemoryFree(buf);
-  return -1;
-#endif
-  return 0;
-}
-
-int32_t streamLoadStateInfo(SStreamMeta* pMeta, SStreamTask* pTask) {
-#if 0
-  void*   pVal = NULL;
-  int32_t vLen = 0;
-  if (tdbTbGet(pMeta->pStateDb, &pTask->taskId, sizeof(void*), &pVal, &vLen) < 0) {
-    return -1;
-  }
-  SDecoder decoder;
-  tDecoderInit(&decoder, pVal, vLen);
-  SStreamMultiVgCheckpointInfo aggCheckpoint;
-  tDecodeSStreamMultiVgCheckpointInfo(&decoder, &aggCheckpoint);
-  tDecoderClear(&decoder);
-
-  pTask->nextCheckId = aggCheckpoint.checkpointId + 1;
-  pTask->checkpointInfo = aggCheckpoint.checkpointVer;
-#endif
-  return 0;
-}
-
-int32_t streamSaveSinkLevel(SStreamMeta* pMeta, SStreamTask* pTask) {
-  ASSERT(pTask->taskLevel == TASK_LEVEL__SINK);
-  return streamSaveStateInfo(pMeta, pTask);
-}
-
-int32_t streamRecoverSinkLevel(SStreamMeta* pMeta, SStreamTask* pTask) {
-  ASSERT(pTask->taskLevel == TASK_LEVEL__SINK);
-  return streamLoadStateInfo(pMeta, pTask);
-}
-
-int32_t streamSaveAggLevel(SStreamMeta* pMeta, SStreamTask* pTask) {
-  ASSERT(pTask->taskLevel == TASK_LEVEL__AGG);
-  // TODO save and copy state
-
-  // save state info
-  if (streamSaveStateInfo(pMeta, pTask) < 0) {
-    return -1;
-  }
-  return 0;
-}
-
-#if 0
-int32_t streamFetchRecoverStatus(SStreamTask* pTask, const SVgroupInfo* pVgInfo) {
-  int32_t                     taskId = pVgInfo->taskId;
-  int32_t                     nodeId = pVgInfo->vgId;
-  SStreamRecoverDownstreamReq req = {
-      .streamId = pTask->taskId,
-      .downstreamTaskId = taskId,
-      .taskId = pTask->taskId,
-  };
-  int32_t tlen;
-  int32_t code;
-  tEncodeSize(tEncodeSStreamTaskRecoverReq, &req, tlen, code);
-  if (code < 0) {
-    return -1;
-  }
-  void* buf = taosMemoryCalloc(1, sizeof(SMsgHead) + tlen);
-  if (buf == NULL) {
-    return -1;
-  }
-  void*    abuf = POINTER_SHIFT(buf, sizeof(SMsgHead));
-  SEncoder encoder;
-  tEncoderInit(&encoder, abuf, tlen);
-  if (tEncodeSStreamTaskRecoverReq(&encoder, &req) < 0) {
-    tEncoderClear(&encoder);
-    taosMemoryFree(buf);
-    return -1;
-  }
-  tEncoderClear(&encoder);
-
-  ((SMsgHead*)buf)->vgId = htonl(nodeId);
-  SRpcMsg msg = {
-      .pCont = buf, .contLen = sizeof(SMsgHead) + tlen,
-      /*.msgType = */
-  };
-  tmsgSendReq(&pVgInfo->epSet, &msg);
-
-  return 0;
-}
-
-int32_t streamFetchDownstreamStatus(SStreamMeta* pMeta, SStreamTask* pTask) {
-  // set self status to recover_phase1
-  SStreamRecoverStatus* pRecover;
-  atomic_store_8(&pTask->taskStatus, TASK_STATUS__RECOVER_DOWNSTREAM);
-  pRecover = taosHashGet(pMeta->pRecoverStatus, &pTask->taskId, sizeof(int32_t));
-  if (pRecover == NULL) {
-    pRecover = taosMemoryCalloc(1, sizeof(SStreamRecoverStatus));
-    if (pRecover == NULL) {
-      return -1;
-    }
-    pRecover->info = taosArrayInit(0, sizeof(void*));
-    if (pRecover->info == NULL) {
-      taosMemoryFree(pRecover);
-      return -1;
-    }
-    taosHashPut(pMeta->pRecoverStatus, &pTask->taskId, sizeof(int32_t), &pRecover, sizeof(void*));
-  }
-
-  if (pTask->outputType == TASK_OUTPUT__FIXED_DISPATCH) {
-    pRecover->totReq = 1;
-  } else if (pTask->outputType == TASK_OUTPUT__SHUFFLE_DISPATCH) {
-    int32_t numOfDownstream = taosArrayGetSize(pTask->shuffleDispatcher.dbInfo.pVgroupInfos);
-    pRecover->totReq = numOfDownstream;
-    for (int32_t i = 0; i < numOfDownstream; i++) {
-      SVgroupInfo* pVgInfo = taosArrayGet(pTask->shuffleDispatcher.dbInfo.pVgroupInfos, i);
-      streamFetchRecoverStatus(pTask, pVgInfo);
-    }
-  } else {
-    ASSERT(0);
-  }
-  return 0;
-}
-#endif
-
-#if 0
-int32_t streamProcessFetchStatusRsp(SStreamMeta* pMeta, SStreamTask* pTask, SStreamRecoverDownstreamRsp* pRsp) {
-  // if failed, set timer and retry
-  // if successful
-  int32_t               taskId = pTask->taskId;
-  SStreamRecoverStatus* pRecover = taosHashGet(pMeta->pRecoverStatus, &taskId, sizeof(int32_t));
-  if (pRecover == NULL) {
-    return -1;
-  }
-
-  taosArrayPush(pRecover->info, &pRsp->checkpointVer);
-
-  int32_t leftRsp = atomic_sub_fetch_32(&pRecover->waitingRspCnt, 1);
-  ASSERT(leftRsp >= 0);
-
-  if (leftRsp == 0) {
-    ASSERT(taosArrayGetSize(pRecover->info) == pRecover->totReq);
-
-    // srcNodeId -> SStreamCheckpointInfo*
-    SHashObj* pFinalChecks = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), false, HASH_NO_LOCK);
-    if (pFinalChecks == NULL) return -1;
-
-    for (int32_t i = 0; i < pRecover->totReq; i++) {
-      SArray* pChecks = taosArrayGetP(pRecover->info, i);
-      int32_t sz = taosArrayGetSize(pChecks);
-      for (int32_t j = 0; j < sz; j++) {
-        SStreamCheckpointInfo* pOneCheck = taosArrayGet(pChecks, j);
-        SStreamCheckpointInfo* pCheck = taosHashGet(pFinalChecks, &pOneCheck->srcNodeId, sizeof(int32_t));
-        if (pCheck == NULL) {
-          pCheck = taosMemoryCalloc(1, sizeof(SStreamCheckpointInfo));
-          pCheck->srcNodeId = pOneCheck->srcNodeId;
-          pCheck->srcChildId = pOneCheck->srcChildId;
-          pCheck->stateProcessedVer = pOneCheck->stateProcessedVer;
-          taosHashPut(pFinalChecks, &pCheck->srcNodeId, sizeof(int32_t), &pCheck, sizeof(void*));
-        } else {
-          pCheck->stateProcessedVer = TMIN(pCheck->stateProcessedVer, pOneCheck->stateProcessedVer);
-        }
-      }
-    }
-    // load local state
-    //
-    // recover
-    //
-    if (pTask->taskLevel == TASK_LEVEL__SOURCE) {
-      qStreamPrepareRecover(pTask->exec.executor, pTask->startVer, pTask->recoverSnapVer);
-      if (streamPipelineExec(pTask, 10000, true) < 0) {
-        return -1;
-      }
-    }
-    taosHashCleanup(pFinalChecks);
-    taosHashRemove(pMeta->pRecoverStatus, &taskId, sizeof(int32_t));
-    atomic_store_8(&pTask->taskStatus, TASK_STATUS__NORMAL);
-  }
-  return 0;
-}
-
-int32_t streamRecoverAggLevel(SStreamMeta* pMeta, SStreamTask* pTask) {
-  ASSERT(pTask->taskLevel == TASK_LEVEL__AGG);
-  // recover sink level
-  // after all sink level recovered
-  // choose suitable state to recover
-  return 0;
-}
-
-int32_t streamSaveSourceLevel(SStreamMeta* pMeta, SStreamTask* pTask) {
-  ASSERT(pTask->taskLevel == TASK_LEVEL__SOURCE);
-  // TODO: save and copy state
-  return 0;
-}
-
-int32_t streamRecoverSourceLevel(SStreamMeta* pMeta, SStreamTask* pTask) {
-  ASSERT(pTask->taskLevel == TASK_LEVEL__SOURCE);
-  // if totLevel == 3
-  // fetch agg state
-  // recover from local state to agg state, not send msg
-  // recover from agg state to most recent log v1
-  // enable input queue, set status recover_phase2
-  // recover from v1 to queue msg v2, set status normal
-
-  // if totLevel == 2
-  // fetch sink state
-  // recover from local state to sink state v1, send msg
-  // enable input queue, set status recover_phase2
-  // recover from v1 to queue msg v2, set status normal
-  return 0;
-}
-
-int32_t streamRecoverTask(SStreamTask* pTask) {
-  //
-  return 0;
-}
-#endif

source/libs/sync/inc/syncCommit.h

@@ -49,6 +49,7 @@ extern "C" {
 //        IN commitIndex' = [commitIndex EXCEPT ![i] = newCommitIndex]
 //     /\ UNCHANGED <<messages, serverVars, candidateVars, leaderVars, log>>
 //
+void syncOneReplicaAdvance(SSyncNode* pSyncNode);
 void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode);
 bool syncAgreeIndex(SSyncNode* pSyncNode, SRaftId* pRaftId, SyncIndex index);
 bool syncAgree(SSyncNode* pSyncNode, SyncIndex index);

source/libs/sync/src/syncCommit.c

@@ -44,6 +44,56 @@
 //        IN commitIndex' = [commitIndex EXCEPT ![i] = newCommitIndex]
 //     /\ UNCHANGED <<messages, serverVars, candidateVars, leaderVars, log>>
 //
+void syncOneReplicaAdvance(SSyncNode* pSyncNode) {
+  if (pSyncNode == NULL) {
+    sError("pSyncNode is NULL");
+    return;
+  }
+
+  if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
+    syncNodeErrorLog(pSyncNode, "not leader, can not advance commit index");
+    return;
+  }
+
+  if (pSyncNode->replicaNum != 1) {
+    syncNodeErrorLog(pSyncNode, "not one replica, can not advance commit index");
+    return;
+  }
+
+  // advance commit index to snapshot first
+  SSnapshot snapshot;
+  pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot);
+  if (snapshot.lastApplyIndex > 0 && snapshot.lastApplyIndex > pSyncNode->commitIndex) {
+    SyncIndex commitBegin = pSyncNode->commitIndex;
+    SyncIndex commitEnd = snapshot.lastApplyIndex;
+    pSyncNode->commitIndex = snapshot.lastApplyIndex;
+
+    char eventLog[128];
+    snprintf(eventLog, sizeof(eventLog), "commit by snapshot from index:%" PRId64 " to index:%" PRId64, commitBegin,
+             commitEnd);
+    syncNodeEventLog(pSyncNode, eventLog);
+  }
+
+  // advance commit index as large as possible
+  SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode);
+  if (lastIndex > pSyncNode->commitIndex) {
+    do {
+      char eventLog[128];
+      snprintf(eventLog, sizeof(eventLog), "commit by wal from index:%" PRId64 " to index:%" PRId64,
+               pSyncNode->commitIndex + 1, lastIndex);
+      syncNodeEventLog(pSyncNode, eventLog);
+    } while (0);
+
+    pSyncNode->commitIndex = lastIndex;
+  }
+
+  // call back Wal
+  SyncIndex walCommitVer = logStoreWalCommitVer(pSyncNode->pLogStore);
+  if (pSyncNode->commitIndex > walCommitVer) {
+    pSyncNode->pLogStore->syncLogUpdateCommitIndex(pSyncNode->pLogStore, pSyncNode->commitIndex);
+  }
+}
+
 void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode) {
   if (pSyncNode == NULL) {
     sError("pSyncNode is NULL");

source/libs/sync/src/syncMain.c

@@ -437,6 +437,47 @@ int32_t syncStepDown(int64_t rid, SyncTerm newTerm) {
   return 0;
 }
 
+bool syncIsReadyForRead(int64_t rid) {
+  SSyncNode* pSyncNode = syncNodeAcquire(rid);
+  if (pSyncNode == NULL) {
+    terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
+    return false;
+  }
+  ASSERT(rid == pSyncNode->rid);
+
+  if (pSyncNode->state == TAOS_SYNC_STATE_LEADER && pSyncNode->restoreFinish) {
+    syncNodeRelease(pSyncNode);
+    return true;
+  }
+
+  bool ready = false;
+  if (pSyncNode->state == TAOS_SYNC_STATE_LEADER && !pSyncNode->restoreFinish) {
+    if (!pSyncNode->pLogStore->syncLogIsEmpty(pSyncNode->pLogStore)) {
+      SSyncRaftEntry* pEntry = NULL;
+      int32_t         code = pSyncNode->pLogStore->syncLogGetEntry(
+          pSyncNode->pLogStore, pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore), &pEntry);
+      if (code == 0 && pEntry != NULL) {
+        if (pEntry->originalRpcType == TDMT_SYNC_NOOP && pEntry->term == pSyncNode->pRaftStore->currentTerm) {
+          ready = true;
+        }
+
+        syncEntryDestory(pEntry);
+      }
+    }
+  }
+
+  if (!ready) {
+    if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
+      terrno = TSDB_CODE_SYN_NOT_LEADER;
+    } else {
+      terrno = TSDB_CODE_APP_NOT_READY;
+    }
+  }
+
+  syncNodeRelease(pSyncNode);
+  return ready;
+}
+
 int32_t syncNodeLeaderTransfer(SSyncNode* pSyncNode) {
   if (pSyncNode->peersNum == 0) {
     sDebug("only one replica, cannot leader transfer");
@@ -727,7 +768,7 @@ char* sync2SimpleStr(int64_t rid) {
     sTrace("syncSetRpc get pSyncNode is NULL, rid:%" PRId64, rid);
     return NULL;
   }
-  ASSERT(rid == pSyncNode->rid);
+
   char* s = syncNode2SimpleStr(pSyncNode);
   syncNodeRelease(pSyncNode);
 
@@ -737,11 +778,9 @@ char* sync2SimpleStr(int64_t rid) {
 int32_t syncPropose(int64_t rid, SRpcMsg* pMsg, bool isWeak) {
   SSyncNode* pSyncNode = syncNodeAcquire(rid);
   if (pSyncNode == NULL) {
-    syncNodeRelease(pSyncNode);
     terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
     return -1;
   }
-  ASSERT(rid == pSyncNode->rid);
 
   int32_t ret = syncNodePropose(pSyncNode, pMsg, isWeak);
   syncNodeRelease(pSyncNode);
@@ -2969,7 +3008,11 @@ int32_t syncNodeOnClientRequest(SSyncNode* ths, SyncClientRequest* pMsg, SyncInd
 
     // if only myself, maybe commit right now
     if (ths->replicaNum == 1) {
+      if (syncNodeIsMnode(ths)) {
         syncMaybeAdvanceCommitIndex(ths);
+      } else {
+        syncOneReplicaAdvance(ths);
+      }
     }
   }
 
@@ -3063,15 +3106,15 @@ int32_t syncDoLeaderTransfer(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftEntry* p
 
   if (ths->pFsm->FpLeaderTransferCb != NULL) {
     SFsmCbMeta cbMeta = {
-        cbMeta.code = 0,
+        .code = 0,
-        cbMeta.currentTerm = ths->pRaftStore->currentTerm,
+        .currentTerm = ths->pRaftStore->currentTerm,
-        cbMeta.flag = 0,
+        .flag = 0,
-        cbMeta.index = pEntry->index,
+        .index = pEntry->index,
-        cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, pEntry->index),
+        .lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, pEntry->index),
-        cbMeta.isWeak = pEntry->isWeak,
+        .isWeak = pEntry->isWeak,
-        cbMeta.seqNum = pEntry->seqNum,
+        .seqNum = pEntry->seqNum,
-        cbMeta.state = ths->state,
+        .state = ths->state,
-        cbMeta.term = pEntry->term,
+        .term = pEntry->term,
     };
     ths->pFsm->FpLeaderTransferCb(ths->pFsm, pRpcMsg, &cbMeta);
   }

source/libs/sync/src/syncRaftEntry.c

@@ -72,6 +72,7 @@ SSyncRaftEntry* syncEntryBuildNoop(SyncTerm term, SyncIndex index, int32_t vgId)
   SMsgHead head;
   head.vgId = vgId;
   head.contLen = sizeof(SMsgHead);
+  head.msgMask = 0;
   SRpcMsg rpcMsg;
   memset(&rpcMsg, 0, sizeof(SRpcMsg));
   rpcMsg.contLen = head.contLen;

source/libs/sync/src/syncRaftLog.c

@@ -219,7 +219,7 @@ static int32_t raftLogAppendEntry(struct SSyncLogStore* pLogStore, SSyncRaftEntr
              pEntry->index, err, err, errStr, sysErr, sysErrStr);
     syncNodeErrorLog(pData->pSyncNode, logBuf);
 
-    ASSERT(0);
+    // ASSERT(0);
     return -1;
   }
   pEntry->index = index;
@@ -327,7 +327,7 @@ static int32_t raftLogTruncate(struct SSyncLogStore* pLogStore, SyncIndex fromIn
     sError("vgId:%d, wal truncate error, from-index:%" PRId64 ", err:%d %X, msg:%s, syserr:%d, sysmsg:%s",
            pData->pSyncNode->vgId, fromIndex, err, err, errStr, sysErr, sysErrStr);
 
-    ASSERT(0);
+    // ASSERT(0);
   }
 
   // event log
@@ -376,6 +376,7 @@ int32_t raftLogUpdateCommitIndex(SSyncLogStore* pLogStore, SyncIndex index) {
            pData->pSyncNode->vgId, index, err, err, errStr, sysErr, sysErrStr);
 
     ASSERT(0);
+    return -1;
   }
   return 0;
 }

source/libs/sync/src/syncRequestVote.c

@@ -47,6 +47,19 @@ static bool syncNodeOnRequestVoteLogOK(SSyncNode* pSyncNode, SyncRequestVote* pM
   SyncTerm  myLastTerm = syncNodeGetLastTerm(pSyncNode);
   SyncIndex myLastIndex = syncNodeGetLastIndex(pSyncNode);
 
+  if (pMsg->lastLogIndex < pSyncNode->commitIndex) {
+    do {
+      char logBuf[128];
+      snprintf(logBuf, sizeof(logBuf),
+               "logok:0, {my-lterm:%" PRIu64 ", my-lindex:%" PRId64 ", recv-lterm:%" PRIu64 ", recv-lindex:%" PRId64
+               ", recv-term:%" PRIu64 "}",
+               myLastTerm, myLastIndex, pMsg->lastLogTerm, pMsg->lastLogIndex, pMsg->term);
+      syncNodeEventLog(pSyncNode, logBuf);
+    } while (0);
+
+    return false;
+  }
+
   if (myLastTerm == SYNC_TERM_INVALID) {
     do {
       char logBuf[128];

source/libs/sync/src/syncRespMgr.c

@@ -146,15 +146,15 @@ void syncRespCleanByTTL(SSyncRespMgr *pObj, int64_t ttl, bool rsp) {
       cnt++;
 
       SFsmCbMeta cbMeta = {
-          cbMeta.index = SYNC_INDEX_INVALID,
+          .index = SYNC_INDEX_INVALID,
-          cbMeta.lastConfigIndex = SYNC_INDEX_INVALID,
+          .lastConfigIndex = SYNC_INDEX_INVALID,
-          cbMeta.isWeak = false,
+          .isWeak = false,
-          cbMeta.code = TSDB_CODE_SYN_TIMEOUT,
+          .code = TSDB_CODE_SYN_TIMEOUT,
-          cbMeta.state = pSyncNode->state,
+          .state = pSyncNode->state,
-          cbMeta.seqNum = *pSeqNum,
+          .seqNum = *pSeqNum,
-          cbMeta.term = SYNC_TERM_INVALID,
+          .term = SYNC_TERM_INVALID,
-          cbMeta.currentTerm = pSyncNode->pRaftStore->currentTerm,
+          .currentTerm = pSyncNode->pRaftStore->currentTerm,
-          cbMeta.flag = 0,
+          .flag = 0,
       };
 
       pStub->rpcMsg.pCont = NULL;

source/libs/transport/src/transCli.c

@@ -427,7 +427,7 @@ void cliHandleExceptImpl(SCliConn* pConn, int32_t code) {
 
     if (pCtx == NULL || pCtx->pSem == NULL) {
       if (transMsg.info.ahandle == NULL) {
-        if (REQUEST_NO_RESP(&pMsg->msg) || pMsg->type == Release) destroyCmsg(pMsg);
+        if (pMsg == NULL || REQUEST_NO_RESP(&pMsg->msg) || pMsg->type == Release) destroyCmsg(pMsg);
         once = true;
         continue;
       }

source/util/src/tconfig.c

@@ -722,13 +722,13 @@ int32_t cfgLoadFromEnvFile(SConfig *pConfig, const char *envFile) {
   const char *filepath = ".env";
   if (envFile != NULL && strlen(envFile) > 0) {
     if (!taosCheckExistFile(envFile)) {
-      uError("failed to load env file: %s", envFile);
+      uError("failed to load env file:%s", envFile);
       return -1;
     }
     filepath = envFile;
   } else {
     if (!taosCheckExistFile(filepath)) {
-      uInfo("failed to load env file: %s", filepath);
+      uInfo("env file:%s not load", filepath);
       return 0;
     }
   }
@@ -929,7 +929,7 @@ int32_t cfgLoadFromApollUrl(SConfig *pConfig, const char *url) {
   if (strncmp(url, "jsonFile", 8) == 0) {
     char *filepath = p;
     if (!taosCheckExistFile(filepath)) {
-      uError("failed to load json file: %s", filepath);
+      uError("failed to load json file:%s", filepath);
       return -1;
     }
 
@@ -1056,13 +1056,13 @@ int32_t cfgGetApollUrl(const char **envCmd, const char *envFile, char *apolloUrl
   const char *filepath = ".env";
   if (envFile != NULL && strlen(envFile) > 0) {
     if (!taosCheckExistFile(envFile)) {
-      uError("failed to load env file: %s", envFile);
+      uError("failed to load env file:%s", envFile);
       return -1;
     }
     filepath = envFile;
   } else {
     if (!taosCheckExistFile(filepath)) {
-      uInfo("failed to load env file: %s", filepath);
+      uInfo("env file:%s not load", filepath);
       return 0;
     }
   }

source/util/src/terror.c

@@ -609,6 +609,9 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_INVALID_MSG,                "Invalid message")
 TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_MISMATCH,          "Consumer mismatch")
 TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_CONSUMER_CLOSED,            "Consumer closed")
 
+// stream
+TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_TASK_NOT_EXIST,          "Stream task not exist")
+
 // TDLite
 TAOS_DEFINE_ERROR(TSDB_CODE_TDLITE_IVLD_OPEN_FLAGS,         "Invalid TDLite open flags")
 TAOS_DEFINE_ERROR(TSDB_CODE_TDLITE_IVLD_OPEN_DIR,           "Invalid TDLite open directory")

source/util/src/tlog.c

@@ -81,25 +81,25 @@ int64_t tsNumOfDebugLogs = 0;
 int64_t tsNumOfTraceLogs = 0;
 
 // log
-int32_t dDebugFlag = 135;
+int32_t dDebugFlag = 131;
-int32_t vDebugFlag = 135;
+int32_t vDebugFlag = 131;
-int32_t mDebugFlag = 135;
+int32_t mDebugFlag = 131;
 int32_t cDebugFlag = 131;
 int32_t jniDebugFlag = 131;
 int32_t tmrDebugFlag = 131;
 int32_t uDebugFlag = 131;
 int32_t rpcDebugFlag = 131;
 int32_t qDebugFlag = 131;
-int32_t wDebugFlag = 135;
+int32_t wDebugFlag = 131;
-int32_t sDebugFlag = 135;
+int32_t sDebugFlag = 131;
 int32_t tsdbDebugFlag = 131;
 int32_t tdbDebugFlag = 131;
-int32_t tqDebugFlag = 135;
+int32_t tqDebugFlag = 131;
-int32_t fsDebugFlag = 135;
+int32_t fsDebugFlag = 131;
-int32_t metaDebugFlag = 135;
+int32_t metaDebugFlag = 131;
-int32_t udfDebugFlag = 135;
+int32_t udfDebugFlag = 131;
 int32_t smaDebugFlag = 131;
-int32_t idxDebugFlag = 135;
+int32_t idxDebugFlag = 131;
 
 int64_t dbgEmptyW = 0;
 int64_t dbgWN = 0;

source/util/src/tqueue.c

@@ -21,46 +21,6 @@
 int64_t tsRpcQueueMemoryAllowed = 0;
 int64_t tsRpcQueueMemoryUsed = 0;
 
-typedef struct STaosQnode STaosQnode;
-
-typedef struct STaosQnode {
-  STaosQnode *next;
-  STaosQueue *queue;
-  int64_t     timestamp;
-  int32_t     size;
-  int8_t      itype;
-  int8_t      reserved[3];
-  char        item[];
-} STaosQnode;
-
-typedef struct STaosQueue {
-  STaosQnode   *head;
-  STaosQnode   *tail;
-  STaosQueue   *next;     // for queue set
-  STaosQset    *qset;     // for queue set
-  void         *ahandle;  // for queue set
-  FItem         itemFp;
-  FItems        itemsFp;
-  TdThreadMutex mutex;
-  int64_t       memOfItems;
-  int32_t       numOfItems;
-} STaosQueue;
-
-typedef struct STaosQset {
-  STaosQueue   *head;
-  STaosQueue   *current;
-  TdThreadMutex mutex;
-  tsem_t        sem;
-  int32_t       numOfQueues;
-  int32_t       numOfItems;
-} STaosQset;
-
-typedef struct STaosQall {
-  STaosQnode *current;
-  STaosQnode *start;
-  int32_t     numOfItems;
-} STaosQall;
-
 STaosQueue *taosOpenQueue() {
   STaosQueue *queue = taosMemoryCalloc(1, sizeof(STaosQueue));
   if (queue == NULL) {

source/util/src/tref.c

@@ -361,7 +361,7 @@ int32_t taosListRef() {
 
     if (pSet->state == TSDB_REF_STATE_EMPTY) continue;
 
-    uInfo("rsetId:%d state:%d count::%d", i, pSet->state, pSet->count);
+    uInfo("rsetId:%d state:%d count:%d", i, pSet->state, pSet->count);
 
     for (int32_t j = 0; j < pSet->max; ++j) {
       pNode = pSet->nodeList[j];

source/util/src/tworker.c

@@ -36,14 +36,13 @@ int32_t tQWorkerInit(SQWorkerPool *pool) {
     worker->pool = pool;
   }
 
-  uInfo("worker:%s is initialized, min:%d max:%d", pool->name, pool->min, pool->max);
+  uDebug("worker:%s is initialized, min:%d max:%d", pool->name, pool->min, pool->max);
   return 0;
 }
 
 void tQWorkerCleanup(SQWorkerPool *pool) {
   for (int32_t i = 0; i < pool->max; ++i) {
     SQWorker *worker = pool->workers + i;
-    // if (worker == NULL) continue;
     if (taosCheckPthreadValid(worker->thread)) {
       taosQsetThreadResume(pool->qset);
     }
@@ -51,7 +50,6 @@ void tQWorkerCleanup(SQWorkerPool *pool) {
 
   for (int32_t i = 0; i < pool->max; ++i) {
     SQWorker *worker = pool->workers + i;
-    // if (worker == NULL) continue;
     if (taosCheckPthreadValid(worker->thread)) {
       taosThreadJoin(worker->thread, NULL);
       taosThreadClear(&worker->thread);
@@ -73,11 +71,13 @@ static void *tQWorkerThreadFp(SQWorker *worker) {
 
   taosBlockSIGPIPE();
   setThreadName(pool->name);
-  uDebug("worker:%s:%d is running", pool->name, worker->id);
+  worker->pid = taosGetSelfPthreadId();
+  uInfo("worker:%s:%d is running, thread:%08" PRId64, pool->name, worker->id, worker->pid);
 
   while (1) {
     if (taosReadQitemFromQset(pool->qset, (void **)&msg, &qinfo) == 0) {
-      uDebug("worker:%s:%d qset:%p, got no message and exiting", pool->name, worker->id, pool->qset);
+      uInfo("worker:%s:%d qset:%p, got no message and exiting, thread:%08" PRId64, pool->name, worker->id, pool->qset,
+            worker->pid);
       break;
     }
 
@@ -124,7 +124,7 @@ STaosQueue *tQWorkerAllocQueue(SQWorkerPool *pool, void *ahandle, FItem fp) {
   }
 
   taosThreadMutexUnlock(&pool->mutex);
-  uDebug("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
+  uInfo("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
 
   return queue;
 }
@@ -191,12 +191,14 @@ static void *tWWorkerThreadFp(SWWorker *worker) {
 
   taosBlockSIGPIPE();
   setThreadName(pool->name);
-  uDebug("worker:%s:%d is running", pool->name, worker->id);
+  worker->pid = taosGetSelfPthreadId();
+  uInfo("worker:%s:%d is running, thread:%08" PRId64, pool->name, worker->id, worker->pid);
 
   while (1) {
     numOfMsgs = taosReadAllQitemsFromQset(worker->qset, worker->qall, &qinfo);
     if (numOfMsgs == 0) {
-      uDebug("worker:%s:%d qset:%p, got no message and exiting", pool->name, worker->id, worker->qset);
+      uInfo("worker:%s:%d qset:%p, got no message and exiting, thread:%08" PRId64, pool->name, worker->id, worker->qset,
+            worker->pid);
       break;
     }
 
@@ -244,7 +246,6 @@ STaosQueue *tWWorkerAllocQueue(SWWorkerPool *pool, void *ahandle, FItems fp) {
     pool->nextId = (pool->nextId + 1) % pool->max;
   }
 
-  uDebug("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
   code = 0;
 
 _OVER:
@@ -256,6 +257,9 @@ _OVER:
     if (worker->qall != NULL) taosFreeQall(worker->qall);
     return NULL;
   } else {
+    while (worker->pid <= 0) taosMsleep(10);
+    queue->threadId = worker->pid;
+    uInfo("worker:%s, queue:%p is allocated, ahandle:%p thread:%08" PRId64, pool->name, queue, ahandle, queue->threadId);
     return queue;
   }
 }

tests/develop-test/5-taos-tools/taosbenchmark/json/queryInsertdata.json

@@ -0,0 +1,74 @@
+{
+    "filetype": "insert",
+    "cfgdir": "/etc/taos",
+    "host": "127.0.0.1",
+    "port": 6030,
+    "user": "root",
+    "password": "taosdata",
+    "thread_count": 4,
+    "thread_count_create_tbl": 4,
+    "result_file": "./insert_res.txt",
+    "confirm_parameter_prompt": "no",
+    "insert_interval": 0,
+    "interlace_rows": 0,
+    "num_of_records_per_req": 3000,
+    "max_sql_len": 1024000,
+    "databases": [{
+        "dbinfo": {
+            "name": "db",
+            "drop": "yes",
+            "replica": 1,
+            "precision": "ms"
+        },
+        "super_tables": [{
+            "name": "stb0",
+            "child_table_exists":"no",
+            "childtable_count": 10,
+            "childtable_prefix": "stb00_",
+            "auto_create_table": "no",
+            "batch_create_tbl_num": 10,
+            "data_source": "rand",
+            "insert_mode": "taosc",
+            "insert_rows": 100,
+            "childtable_limit": 0,
+            "childtable_offset": 0,
+            "interlace_rows": 0,
+            "insert_interval": 0,
+            "max_sql_len": 1024000,
+            "disorder_ratio": 0,
+            "disorder_range": 1000,
+            "timestamp_step": 1,
+            "start_timestamp": "2020-11-01 00:00:00.000",
+            "sample_format": "csv",
+            "sample_file": "./sample.csv",
+            "tags_file": "",
+            "columns": [{"type": "BINARY", "len": 1, "count":1}, {"type": "BINARY", "len": 3, "count":1}, {"type": "INT"}, {"type": "DOUBLE", "count":1}],
+            "tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
+        },
+        {
+            "name": "stb1",
+            "child_table_exists":"no",
+            "childtable_count": 10,
+            "childtable_prefix": "stb01_",
+            "auto_create_table": "no",
+            "batch_create_tbl_num": 10,
+            "data_source": "rand",
+            "insert_mode": "taosc",
+            "insert_rows": 200,
+            "childtable_limit": 0,
+            "childtable_offset": 0,
+            "interlace_rows": 0 ,
+            "insert_interval": 0,
+            "max_sql_len": 1024000,
+            "disorder_ratio": 0,
+            "disorder_range": 1000,
+            "timestamp_step": 1,
+            "start_timestamp": "2020-11-01 00:00:00.000",
+            "sample_format": "csv",
+            "sample_file": "./sample.csv",
+            "tags_file": "",
+            "columns": [{"type": "INT"}, {"type": "DOUBLE", "count":6}, {"type": "BINARY", "len": 1, "count":3}, {"type": "BINARY", "len": 2, "count":6}],
+            "tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
+        }]
+    }]
+}

tests/develop-test/5-taos-tools/taosbenchmark/json/queryInsertrestdata.json

@@ -0,0 +1,73 @@
+{
+    "filetype": "insert",
+    "cfgdir": "/etc/taos",
+    "host": "127.0.0.1",
+    "port": 6030,
+    "user": "root",
+    "password": "taosdata",
+    "thread_count": 4,
+    "thread_count_create_tbl": 4,
+    "result_file": "./insert_res.txt",
+    "confirm_parameter_prompt": "no",
+    "insert_interval": 0,
+    "interlace_rows": 0,
+    "num_of_records_per_req": 3000,
+    "max_sql_len": 1024000,
+    "databases": [{
+        "dbinfo": {
+            "name": "db",
+            "drop": "yes",
+            "precision": "ms"
+        },
+        "super_tables": [{
+            "name": "stb0",
+            "child_table_exists":"no",
+            "childtable_count": 2,
+            "childtable_prefix": "stb00_",
+            "auto_create_table": "no",
+            "batch_create_tbl_num": 10,
+            "data_source": "rand",
+            "insert_mode": "taosc",
+            "insert_rows": 10,
+            "childtable_limit": 0,
+            "childtable_offset": 0,
+            "interlace_rows": 0,
+            "insert_interval": 0,
+            "max_sql_len": 1024000,
+            "disorder_ratio": 0,
+            "disorder_range": 1000,
+            "timestamp_step": 1,
+            "start_timestamp": "2020-11-01 00:00:00.000",
+            "sample_format": "csv",
+            "sample_file": "./sample.csv",
+            "tags_file": "",
+            "columns": [{"type": "BINARY", "len": 1, "count":1}, {"type": "BINARY", "len": 3, "count":1}, {"type": "INT"}, {"type": "DOUBLE", "count":1}],
+            "tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
+        },
+        {
+            "name": "stb1",
+            "child_table_exists":"no",
+            "childtable_count": 2,
+            "childtable_prefix": "stb01_",
+            "auto_create_table": "no",
+            "batch_create_tbl_num": 10,
+            "data_source": "rand",
+            "insert_mode": "taosc",
+            "insert_rows": 5,
+            "childtable_limit": 0,
+            "childtable_offset": 0,
+            "interlace_rows": 0 ,
+            "insert_interval": 0,
+            "max_sql_len": 1024000,
+            "disorder_ratio": 0,
+            "disorder_range": 1000,
+            "timestamp_step": 1,
+            "start_timestamp": "2020-11-01 00:00:00.000",
+            "sample_format": "csv",
+            "sample_file": "./sample.csv",
+            "tags_file": "",
+            "columns": [{"type": "INT"}, {"type": "DOUBLE", "count":6}, {"type": "BINARY", "len": 1, "count":3}, {"type": "BINARY", "len": 2, "count":6}],
+            "tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
+        }]
+    }]
+}

tests/develop-test/5-taos-tools/taosbenchmark/json/queryQps.json

@@ -0,0 +1,35 @@
+{
+    "filetype": "query",
+    "cfgdir": "/etc/taos",
+    "host": "127.0.0.1",
+    "port": 6030,
+    "user": "root",
+    "password": "taosdata",
+    "confirm_parameter_prompt": "no",
+    "databases": "db",
+    "query_times": 1,
+    "specified_table_query": {
+        "query_interval": 10,
+        "threads": 4,
+        "sqls": [
+            {
+                "sql": "select last_row(*) from stb00_0",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from stb00_9 ",
+                "result": "./query_res1.txt"
+            }]
+    },
+    "super_table_query": {
+        "stblname": "stb1",
+        "query_interval":20,
+        "threads": 4,
+        "sqls": [
+            {
+                "sql": "select last_row(ts) from xxxx",
+                "result": "./query_res2.txt"
+            }
+        ]
+    }
+}

tests/develop-test/5-taos-tools/taosbenchmark/json/queryRestful.json

@@ -0,0 +1,38 @@
+{
+    "filetype": "query",
+    "cfgdir": "/etc/taos",
+    "host": "127.0.0.1",
+    "port": 6030,
+    "user": "root",
+    "password": "taosdata",
+    "confirm_parameter_prompt": "no",
+    "databases": "db",
+    "query_times": 2,
+    "query_mode": "rest",
+    "specified_table_query": {
+      "query_interval": 1,
+      "threads": 3,
+      "sqls": [
+        {
+          "sql": "select last_row(*) from db.stb0 ",
+          "result": "./query_res0.txt"
+        },
+        {
+          "sql": "select count(*) from db.stb00_1",
+          "result": "./query_res1.txt"
+        }
+      ]
+    },
+    "super_table_query": {
+      "stblname": "stb1",
+      "query_interval": 1,
+      "threads": 3,
+      "sqls": [
+        {
+          "sql": "select last_row(ts) from xxxx",
+          "result": "./query_res2.txt"
+        }
+      ]
+    }
+  }
+  

tests/develop-test/5-taos-tools/taosbenchmark/json/querySpeciMutisql100.json

@@ -0,0 +1,429 @@
+{
+  "filetype": "query",
+  "cfgdir": "/etc/taos",
+  "host": "127.0.0.1",
+  "port": 6030,
+  "user": "root",
+  "password": "taosdata",
+  "confirm_parameter_prompt": "no",
+  "databases": "db",
+  "query_times": 2,
+  "specified_table_query": {
+      "query_interval": 1,
+      "threads": 3,
+      "sqls": [
+        {    
+            "sql": "select last_row(*) from stb00_0",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_1",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_2",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_3",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_4",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_5",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_6",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_7",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_8",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_9",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_10 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_11 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_12 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_13 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_14 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_15 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_16 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_17 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_18 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_19 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_20 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_21 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_22 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_23 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_24 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_25 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_26 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_27 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_28 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_29 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_30 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_31 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_32 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_33 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_34 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_35 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_36 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_37 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_38 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_39 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_40 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_41 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_42 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_43 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_44 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_45 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_46 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_47 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_48 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_49 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_50 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_51 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_52 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_53 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_54 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_55 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_56 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_57 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_58 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_59 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_60",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_61",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_62",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_63",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_64",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_65",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_66",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_67",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_68",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_69",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_70 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_71 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_72 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_73 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_74 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_75 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_76 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_77 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_78 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_79 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_80 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_81 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_82 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_83 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_84 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_85 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_86 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_87 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_88 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_89 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_90 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_91 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_92 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_93 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_94 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_95 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_96 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_97 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_98 ",
+            "result": "./query_res0.txt"
+        },
+        {
+            "sql": "select last_row(*) from stb00_99 ",
+            "result": "./query_res0.txt"
+
+        }]
+    },
+    "super_table_query": {
+        "stblname": "stb1",
+        "query_interval": 1,
+        "threads": 3,
+        "sqls": [
+          {
+            "sql": "select last_row(ts) from xxxx",
+            "result": "./query_res2.txt"
+          }
+        ]
+      }
+}
+  

tests/develop-test/5-taos-tools/taosbenchmark/json/querySuperMutisql100.json

@@ -0,0 +1,419 @@
+{
+    "filetype": "query",
+    "cfgdir": "/etc/taos",
+    "host": "127.0.0.1",
+    "port": 6030,
+    "user": "root",
+    "password": "taosdata",
+    "confirm_parameter_prompt": "no",
+    "databases": "db",
+    "query_times": 3,
+    "super_table_query": {
+        "stblname": "stb0",
+        "query_interval": 10,
+        "threads": 9,
+        "sqls": [
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select last_row(*) from xxxx  ",
+                "result": "./query_res0.txt"
+            },
+            {
+                "sql": "select * from xxxx  ",
+                "result": "./query_res0.txt"
+
+            }]
+    }
+}
+

tests/develop-test/5-taos-tools/taosbenchmark/json/queryTaosc.json

@@ -0,0 +1,37 @@
+{
+  "filetype": "query",
+  "cfgdir": "/etc/taos",
+  "host": "127.0.0.1",
+  "port": 6030,
+  "user": "root",
+  "password": "taosdata",
+  "confirm_parameter_prompt": "no",
+  "databases": "db",
+  "query_times": 2,
+  "query_mode": "taosc",
+  "specified_table_query": {
+    "query_interval": 1,
+    "threads": 3,
+    "sqls": [
+      {
+        "sql": "select last_row(*) from stb0 ",
+        "result": "./query_res0.txt"
+      },
+      {
+        "sql": "select count(*) from stb00_1",
+        "result": "./query_res1.txt"
+      }
+    ]
+  },
+  "super_table_query": {
+    "stblname": "stb1",
+    "query_interval": 1,
+    "threads": 3,
+    "sqls": [
+      {
+        "sql": "select last_row(ts) from xxxx",
+        "result": "./query_res2.txt"
+      }
+    ]
+  }
+}

tests/develop-test/5-taos-tools/taosbenchmark/taosdemoTestQueryWithJson.py

@@ -0,0 +1,241 @@
+###################################################################
+#           Copyright (c) 2016 by TAOS Technologies, Inc.
+#                     All rights reserved.
+#
+#  This file is proprietary and confidential to TAOS Technologies.
+#  No part of this file may be reproduced, stored, transmitted,
+#  disclosed or used in any form or by any means other than as
+#  expressly provided by the written permission from Jianhui Tao
+#
+###################################################################
+
+# -*- coding: utf-8 -*-
+
+import sys
+import os
+from util.log import *
+from util.cases import *
+from util.sql import *
+from util.dnodes import *
+import time
+from datetime import datetime
+import ast
+import re
+
+# from assertpy import assert_that
+import subprocess
+
+
+class TDTestCase:
+    def init(self, conn, logSql, replicaVarl=1):
+        tdLog.debug("start to execute %s" % __file__)
+        tdSql.init(conn.cursor(), logSql)
+
+    def getPath(self, tool="taosBenchmark"):
+        selfPath = os.path.dirname(os.path.realpath(__file__))
+
+        if "community" in selfPath:
+            projPath = selfPath[: selfPath.find("community")]
+        elif "src" in selfPath:
+            projPath = selfPath[: selfPath.find("src")]
+        elif "/tools/" in selfPath:
+            projPath = selfPath[: selfPath.find("/tools/")]
+        elif "/tests/" in selfPath:
+            projPath = selfPath[: selfPath.find("/tests/")]
+        else:
+            tdLog.info("cannot found %s in path: %s, use system's" % (tool, selfPath))
+            projPath = "/usr/local/taos/bin/"
+
+        paths = []
+        for root, dirs, files in os.walk(projPath):
+            if (tool) in files:
+                rootRealPath = os.path.dirname(os.path.realpath(root))
+                if "packaging" not in rootRealPath:
+                    paths.append(os.path.join(root, tool))
+                    break
+        if len(paths) == 0:
+            return ""
+        return paths[0]
+
+    # 获取taosc接口查询的结果文件中的内容,返回每行数据,并断言数据的第一列内容。
+    def assertfileDataTaosc(self, filename, expectResult):
+        self.filename = filename
+        self.expectResult = expectResult
+        with open("%s" % filename, "r+") as f1:
+            for line in f1.readlines():
+                queryResultTaosc = line.strip().split()[0]
+                self.assertCheck(filename, queryResultTaosc, expectResult)
+
+    # 获取restful接口查询的结果文件中的关键内容,目前的关键内容找到第一个key就跳出循，所以就只有一个数据。后续再修改多个结果文件。
+    def getfileDataRestful(self, filename):
+        self.filename = filename
+        with open("%s" % filename, "r+") as f1:
+            for line in f1.readlines():
+                contents = line.strip()
+                if contents.find("data") != -1:
+                    pattern = re.compile("{.*}")
+                    contents = pattern.search(contents).group()
+                    contentsDict = ast.literal_eval(contents)  # 字符串转换为字典
+                    queryResultRest = contentsDict["data"][0][0]
+                    break
+                else:
+                    queryResultRest = ""
+        return queryResultRest
+
+    # 获取taosc接口查询次数
+    def queryTimesTaosc(self, filename):
+        self.filename = filename
+        command = "cat %s |wc -l" % filename
+        times = int(subprocess.getstatusoutput(command)[1])
+        return times
+
+    # 获取restful接口查询次数
+    def queryTimesRestful(self, filename):
+        self.filename = filename
+        command = 'cat %s |grep "200 OK" |wc -l' % filename
+        times = int(subprocess.getstatusoutput(command)[1])
+        return times
+
+    # 定义断言结果是否正确。不正确返回错误结果，正确即通过。
+    def assertCheck(self, filename, queryResult, expectResult):
+        self.filename = filename
+        self.queryResult = queryResult
+        self.expectResult = expectResult
+        args0 = (filename, queryResult, expectResult)
+        assert queryResult == expectResult, (
+            "Queryfile:%s ,result is %s != expect: %s" % args0
+        )
+
+    def run(self):
+        binPath = self.getPath()
+        if binPath == "":
+            tdLog.exit("taosBenchmark not found!")
+        else:
+            tdLog.info("taosBenchmark use %s" % binPath)
+
+        # delete useless files
+        os.system("rm -rf ./query_res*")
+        os.system("rm -rf ./all_query*")
+
+        # taosc query: query specified  table  and query  super table
+        os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath)
+        os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryTaosc.json" % binPath)
+        os.system("cat query_res0.txt* > all_query_res0_taosc.txt")
+        os.system("cat query_res1.txt* > all_query_res1_taosc.txt")
+        os.system("cat query_res2.txt* > all_query_res2_taosc.txt")
+
+        # correct Times testcases
+        queryTimes0Taosc = self.queryTimesTaosc("all_query_res0_taosc.txt")
+        self.assertCheck("all_query_res0_taosc.txt", queryTimes0Taosc, 6)
+
+        queryTimes1Taosc = self.queryTimesTaosc("all_query_res1_taosc.txt")
+        self.assertCheck("all_query_res1_taosc.txt", queryTimes1Taosc, 6)
+
+        queryTimes2Taosc = self.queryTimesTaosc("all_query_res2_taosc.txt")
+        self.assertCheck("all_query_res2_taosc.txt", queryTimes2Taosc, 20)
+
+        # correct data testcase
+        self.assertfileDataTaosc("all_query_res0_taosc.txt", "1604160000099")
+        self.assertfileDataTaosc("all_query_res1_taosc.txt", "100")
+        self.assertfileDataTaosc("all_query_res2_taosc.txt", "1604160000199")
+
+        # delete useless files
+        os.system("rm -rf ./query_res*")
+        os.system("rm -rf ./all_query*")
+
+        # use restful api to query
+        os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertrestdata.json" % binPath)
+        os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryRestful.json" % binPath)
+        os.system("cat query_res0.txt*  > all_query_res0_rest.txt")
+        os.system("cat query_res1.txt*  > all_query_res1_rest.txt")
+        os.system("cat query_res2.txt*  > all_query_res2_rest.txt")
+
+        # correct Times testcases
+        queryTimes0Restful = self.queryTimesRestful("all_query_res0_rest.txt")
+        self.assertCheck("all_query_res0_rest.txt", queryTimes0Restful, 6)
+
+        queryTimes1Restful = self.queryTimesRestful("all_query_res1_rest.txt")
+        self.assertCheck("all_query_res1_rest.txt", queryTimes1Restful, 6)
+
+        queryTimes2Restful = self.queryTimesRestful("all_query_res2_rest.txt")
+        self.assertCheck("all_query_res2_rest.txt", queryTimes2Restful, 4)
+
+        # correct data testcase
+        data0 = self.getfileDataRestful("all_query_res0_rest.txt")
+        if data0 != "2020-11-01 00:00:00.009" and data0 != "2020-10-31T16:00:00.009Z":
+            tdLog.exit(
+                "data0 is not 2020-11-01 00:00:00.009 and 2020-10-31T16:00:00.009Z"
+            )
+
+        data1 = self.getfileDataRestful("all_query_res1_rest.txt")
+        self.assertCheck("all_query_res1_rest.txt", data1, 10)
+
+        data2 = self.getfileDataRestful("all_query_res2_rest.txt")
+        if data2 != "2020-11-01 00:00:00.004" and data2 != "2020-10-31T16:00:00.004Z":
+            tdLog.exit(
+                "data2 is not 2020-11-01 00:00:00.004 and 2020-10-31T16:00:00.004Z"
+            )
+
+        # query times less than or equal to 100
+        assert (
+            os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath) == 0
+        )
+        assert (
+            os.system("%s -f ./5-taos-tools/taosbenchmark/json/querySpeciMutisql100.json" % binPath)
+            != 0
+        )
+        assert (
+            os.system("%s -f ./5-taos-tools/taosbenchmark/json/querySuperMutisql100.json" % binPath)
+            == 0
+        )
+
+        # query result print QPS
+        os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath)
+        exceptcode = os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryQps.json" % binPath)
+        assert exceptcode == 0
+
+        # 2021.02.09 need modify taosBenchmakr code
+        # use illegal or out of range parameters query json file
+        os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath)
+        # 2021.02.09 need modify taosBenchmakr code
+        # exceptcode = os.system(
+        #     "%s -f ./taosbenchmark/json/queryTimes0.json" %
+        #     binPath)
+        # assert exceptcode != 0
+
+        # 2021.02.09 need modify taosBenchmakr code
+        # exceptcode0 = os.system(
+        #     "%s -f ./taosbenchmark/json/queryTimesless0.json" %
+        #     binPath)
+        # assert exceptcode0 != 0
+
+        # exceptcode1 = os.system(
+        #     "%s -f ./taosbenchmark/json/queryConcurrent0.json" %
+        #     binPath)
+        # assert exceptcode2 != 0
+
+        # exceptcode3 = os.system(
+        #     "%s -f ./taosbenchmark/json/querrThreadsless0.json" %
+        #     binPath)
+        # assert exceptcode3 != 0
+
+        # exceptcode4 = os.system(
+        #     "%s -f ./taosbenchmark/json/querrThreads0.json" %
+        #     binPath)
+        # assert exceptcode4 != 0
+
+        # delete useless files
+        os.system("rm -rf ./insert_res.txt")
+        os.system("rm -rf 5-taos-tools/taosbenchmark/*.py.sql")
+        os.system("rm -rf ./querySystemInfo*")
+        os.system("rm -rf ./query_res*")
+        os.system("rm -rf ./all_query*")
+        os.system("rm -rf ./test_query_res0.txt")
+
+    def stop(self):
+        tdSql.close()
+        tdLog.success("%s successfully executed" % __file__)
+
+
+tdCases.addWindows(__file__, TDTestCase())
+tdCases.addLinux(__file__, TDTestCase())