Merge remote-tracking branch 'origin/3.0' into enh/TD-30442

docs/en/07-develop/01-connect/index.md

@@ -83,7 +83,7 @@ If `maven` is used to manage the projects, what needs to be done is only adding
 <dependency>
   <groupId>com.taosdata.jdbc</groupId>
   <artifactId>taos-jdbcdriver</artifactId>
-  <version>3.2.4</version>
+  <version>3.3.0</version>
 </dependency>
 ```
 

docs/en/08-client-libraries/04-java.mdx

@@ -36,6 +36,11 @@ REST connection supports all platforms that can run Java.
 
 | taos-jdbcdriver version |                                                                   major changes                                                                    | TDengine version |
 | :---------------------: | :------------------------------------------------------------------------------------------------------------------------------------------------: | :--------------: |
+|          3.3.0         | 1. Optimized data transmission performance under Websocket connection; 2. SSL validation skipping is supported but disabled by default|     3.3.2.0 or later          |
+|          3.2.11         | Fixed the result set closing bug when using a native connection.|        -         |
+|          3.2.10         | 1. Automatic compression/decompression for data transmission, disabled by default; 2.Automatic reconnection for websocket with configurable parameter, disabled by default; 3. A new method for schemaless writing is added in the connection class; 4. Optimized performance for data fetching on native connection; 5. Fixing for some known issues; 6. The list of supported functions can be returned by the API for retrieving metadata|        -         |
+|          3.2.9          |  Fixed websocket prepareStatement closing bug.                                                                                                         |        -         |
+|          3.2.8          |  Improved autocommit, fixed commit offset on websocket connection bug, websocket prepareStatement uses one connection and meta data supports view. |        -         |
 |          3.2.7          |  Support VARBINARY and GEOMETRY types, and add time zone support for native connections. Support websocket auto reconnection                       | 3.2.0.0 or later |
 |          3.2.5          |                                                Subscription add committed() and assignment() method                                                | 3.1.0.3 or later |
 |          3.2.4          |                     Subscription add the enable.auto.commit parameter and the unsubscribe() method in the WebSocket connection                     |        -         |
@@ -179,7 +184,7 @@ Add following dependency in the `pom.xml` file of your Maven project:
 <dependency>
  <groupId>com.taosdata.jdbc</groupId>
  <artifactId>taos-jdbcdriver</artifactId>
- <version>3.2.7</version>
+ <version>3.3.0</version>
 </dependency>
 ```
 
@@ -368,7 +373,16 @@ The configuration parameters in properties are as follows.
 - TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: message transmission timeout in milliseconds, the default value is 60000 ms. It only takes effect when using JDBC REST connection and batchfetch is true.
 - TSDBDriver.PROPERTY_KEY_USE_SSL: connecting Securely Using SSL. true: using SSL connection, false: not using SSL connection. It only takes effect when using JDBC REST connection.
 - TSDBDriver.HTTP_POOL_SIZE: size of REST concurrent requests. The default value is 20.
-  For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](../../reference/config/#configuration-file-on-client-side).
+- TSDBDriver.PROPERTY_KEY_ENABLE_COMPRESSION: Whether to enable compression during transmission. It only takes effect when using REST/Websocket connections. true: enabled, false: disabled. The default is false.
+- TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: Whether to enable automatic reconnection. It only takes effect when using Websocket connections. true: enabled, false: disabled. The default is false.
+> **Note**：Enabling automatic reconnection is only effective for simple SQL statement execution, schemaless writing, and data subscription. It is not effective for parameter binding. Automatic reconnection is only effective for the database specified by parameters when the connection is established, and it is not effective for the `use db` statement to switch databases later.
+
+- TSDBDriver.PROPERTY_KEY_RECONNECT_INTERVAL_MS: The interval for automatic reconnection retries, in milliseconds, with a default value of 2000. It only takes effect when PROPERTY_KEY_ENABLE_AUTO_RECONNECT is true.
+- TSDBDriver.PROPERTY_KEY_RECONNECT_RETRY_COUNT: The number of automatic reconnection retries, with a default value of 3. It only takes effect when PROPERTY_KEY_ENABLE_AUTO_RECONNECT is true.
+- TSDBDriver.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: Whether to disable SSL certification validation. It only takes effect when using Websocket connections. true: enabled, false: disabled. The default is false.
+
+
+For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](../../reference/config/#configuration-file-on-client-side).
 
 ### Priority of configuration parameters
 

docs/en/20-third-party/01-grafana.mdx

@@ -13,18 +13,15 @@ TDengine can be quickly integrated with the open-source data visualization syste
 
 In order for Grafana to add the TDengine data source successfully, the following preparations are required:
 
-1. The TDengine cluster is deployed and functioning properly
+1. Grafana server is installed and running properly. TDengine currently supports Grafana versions 7.5 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: [https://grafana.com/grafana/download](https://grafana.com/grafana/download).
-2. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
+2. The TDengine cluster is deployed and functioning properly
+3. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
 
 Record these values:
 
 - TDengine REST API url: `http://tdengine.local:6041`.
 - TDengine cluster authorization, with user + password.
 
-## Installing Grafana
-
-TDengine currently supports Grafana versions 7.5 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: [https://grafana.com/grafana/download](https://grafana.com/grafana/download).
-
 ## Configuring Grafana
 
 ### Install Grafana Plugin and Configure Data Source
@@ -34,15 +31,16 @@ TDengine currently supports Grafana versions 7.5 and above. Users can go to the
 
 Under Grafana 8, plugin catalog allows you to [browse and manage plugins within Grafana](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog) (but for Grafana 7.x, use **With Script** or **Install & Configure Manually**). Find the page at **Configurations > Plugins**, search **TDengine** and click it to install.
 
-![Search tdengine in grafana plugins](./grafana/grafana-plugin-search-tdengine.png)
+Installation may cost some minutes, you can **Create a TDengine data source** when installation finished.
-
-Installation may cost some minutes, then you can **Create a TDengine data source**:
-
-![Install and configure Grafana data source](./grafana/grafana-install-and-config.png)
-
 Then you can add a TDengine data source by filling up the configuration options.
 
-![TDengine Database Grafana plugin add data source](./grafana/grafana-data-source.png)
+![TDengine Database Grafana plugin add data source](./grafana/add_datasource3.webp)
+
+- Host: IP address of the server where the components of the TDengine cluster provide REST service and the port number of the TDengine REST service (6041), by default use `http://localhost:6041`.
+- User: TDengine user name.
+- Password: TDengine user password.
+
+Click `Save & Test` to test. You should see a success message if the test worked.
 
 You can create dashboards with TDengine now.
 
@@ -77,7 +75,7 @@ sudo -u grafana grafana-cli plugins install tdengine-datasource
 You can also download zip files from [GitHub](https://github.com/taosdata/grafanaplugin/releases/latest) or [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) and install manually. The commands are as follows:
 
 ```bash
-GF_VERSION=3.3.1
+GF_VERSION=3.5.2
 # from GitHub
 wget https://github.com/taosdata/grafanaplugin/releases/download/v$GF_VERSION/tdengine-datasource-$GF_VERSION.zip
 # from Grafana
@@ -96,26 +94,19 @@ If Grafana is running in a Docker environment, the TDengine plugin can be automa
 GF_INSTALL_PLUGINS=tdengine-datasource
 ```
 
-Now users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure.
+Now users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side.
-
-![TDengine Database TDinsight plugin add datasource 1](./grafana/add_datasource1.webp)
-
-Click `Add data source` to enter the Add data source page, and enter TDengine in the query box to add it, as shown in the following figure.
-
-![TDengine Database TDinsight plugin add datasource 2](./grafana/add_datasource2.webp)
 
+Click `Add data source` to enter the Add data source page, and enter TDengine in the query box to add it.
 Enter the datasource configuration page, and follow the default prompts to modify the corresponding configuration.
 
 ![TDengine Database TDinsight plugin add database 3](./grafana/add_datasource3.webp)
 
-- Host: IP address of the server where the components of the TDengine cluster provide REST service (offered by taosd before 2.4 and by taosAdapter since 2.4) and the port number of the TDengine REST service (6041), by default use `http://localhost:6041`.
+- Host: IP address of the server where the components of the TDengine cluster provide REST service and the port number of the TDengine REST service (6041), by default use `http://localhost:6041`.
 - User: TDengine user name.
 - Password: TDengine user password.
 
 Click `Save & Test` to test. You should see a success message if the test worked.
 
-![TDengine Database TDinsight plugin add database 4](./grafana/add_datasource4.webp)
-
 </TabItem>
 <TabItem value="container" label="Container">
 
@@ -156,7 +147,7 @@ You can setup a zero-configuration stack for TDengine + Grafana by [docker-compo
 
     services:
       tdengine:
-        image: tdengine/tdengine:3.0.2.4
+        image: tdengine/tdengine:3.3.0.0
         environment:
           TAOS_FQDN: tdengine
         volumes:
@@ -192,13 +183,13 @@ Go back to the main interface to create a dashboard and click Add Query to enter
 
 ![TDengine Database TDinsight plugin create dashboard 1](./grafana/create_dashboard1.webp)
 
-As shown above, select the `TDengine` data source in the `Query` and enter the corresponding SQL in the query box below for query.
+As shown above, select the `TDengine` data source in the `Query` and enter the corresponding SQL in the query box below for query. We will continue to use power meters as an example. In order to demonstrate the beautiful curves, **virtual data** is used here.
 
-- INPUT SQL: Enter the desired query (the results being two columns and multiple rows), such as `select _wstart, avg(mem_system) from log.dnodes_info where ts >= $from and ts < $to interval($interval)`. In this statement, $from, $to, and $interval are variables that Grafana replaces with the query time range and interval. In addition to the built-in variables, custom template variables are also supported.
+- INPUT SQL: Enter the desired query (the results being two columns and multiple rows), such as `select _wstart as ts, avg(current) as current from power.meters where ts > $from and ts < $to interval($interval) fill(null)`. In this statement, `$from`, `$to`, and `$interval` are variables that Grafana replaces with the query time range and interval. In addition to the built-in variables, custom template variables are also supported.
 - ALIAS BY: This allows you to set the current query alias.
 - GENERATE SQL: Clicking this button will automatically replace the corresponding variables and generate the final executed statement.
-- Group by column name(s): `group by` or `partition by` columns name split by comma. By setting `Group by column name(s)`, it can show multi-dimension data if Sql is `group by` or `partition by`. Such as, it can show data by `dnode_ep` if sql is  `select _wstart as ts, avg(mem_system), dnode_ep from log.dnodes_info where ts>=$from and ts<=$to partition by dnode_ep interval($interval)` and `Group by column name(s)` is `dnode_ep`.
+- Group by column(s): `group by` or `partition by` columns name split by comma. By setting `Group by column(s)`, it can show multi-dimension data if Sql is `group by` or `partition by`. Such as, it can show data by `groupid` if sql is `select _wstart as ts, groupid, avg(current) as current from power.meters where ts > $from and ts < $to partition by groupid interval($interval) fill(null)` and `Group by column(s)` is `groupid`.
-- Format to: format legend for `group by` or `partition by`. Such as it can display series data by `dnode_ep` if sql is `select _wstart as ts, avg(mem_system), dnode_ep from log.dnodes_info where ts>=$from and ts<=$to partition by dnode_ep interval($interval)` and `Group by column name(s)` is `dnode_ep` and `Format to` is `mem_system_{{dnode_ep}}`.
+- Group By Format: format legend for `group by` or `partition by`. For example, in the above Input SQL, set `Group By Format` to `groupid-{{groupid}}`, and display the legend name as the formatted group name.
 
 :::note
 
@@ -206,11 +197,11 @@ Since the REST connection because is stateless. Grafana plugin can use &lt;db_na
 
 :::
 
-Follow the default prompt to query the average system memory usage for the specified interval on the server where the current TDengine deployment is located as follows.
+Query the average current changes of all devices in the `meters` stable as shown in the following figure:
 
 ![TDengine Database TDinsight plugin create dashboard 2](./grafana/create_dashboard2.webp)
 
-The example to query the average system memory usage for the specified interval on each server as follows.
+Query the average current value of all devices in the 'meters' stable and display it in groups according to the `groupid` as shown in the following figure:
 
 ![TDengine Database TDinsight plugin create dashboard 2](./grafana/create_dashboard3.webp)
 
@@ -222,7 +213,7 @@ You can install TDinsight dashboard in data source configuration page (like `htt
 
 ![TDengine Database Grafana plugine import dashboard](./import_dashboard.webp)
 
-A dashboard for TDengine 2.x has been published on Grafana: [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167)).
+A dashboard for TDengine 2.x has been published on Grafana: [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167).
 
 For more dashboards using TDengine data source, [search here in Grafana](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource). Here is a sub list:
 

docs/en/21-tdinternal/01-arch.md

@@ -197,20 +197,20 @@ By default, TDengine saves all data in /var/lib/taos directory, and the data fil
 dataDir format is as follows:
 
 ```
-dataDir data_path [tier_level]
+dataDir data_path [tier_level] [primary] [disable_create_new_file]
 ```
 
-Where data_path is the folder path of mount point and tier_level is the media storage-tier. The higher the media storage-tier, means the older the data file. Multiple hard disks can be mounted at the same storage-tier, and data files on the same storage-tier are distributed on all hard disks within the tier. TDengine supports up to 3 tiers of storage, so tier_level values are 0, 1, and 2. When configuring dataDir, there must be only one mount path without specifying tier_level, which is called special mount disk (path). The mount path defaults to level 0 storage media and contains special file links, which cannot be removed, otherwise it will have a devastating impact on the written data.
+Where `data_path` is the folder path of mount point, and `tier_level` is the media storage-tier. The higher the media storage-tier, means the older the data file. Multiple hard disks can be mounted at the same storage-tier, and data files on the same storage-tier are distributed on all hard disks within the tier. TDengine supports up to 3 tiers of storage, so tier_level values are 0, 1, and 2. When configuring dataDir, there must be only one mount path without specifying tier_level, which is called special mount disk (path). The mount path defaults to level 0 storage media and contains special file links, which cannot be removed, otherwise it will have a devastating impact on the written data. And `primary` means whether the data dir is the primary mount point. Enter 0 for false or 1 for true. The default value is 1. A TDengine cluster can have only one `primary` mount point, which must be on tier 0. And `disable_create_new_file` means whether to prohibit the creation of new file sets on the specified mount point. Enter 0 for false and 1 for true. The default value is 0. Tier 0 storage must have at least one mount point with disable_create_new_file set to 0. Tier 1 and tier 2 storage do not have this restriction. 
 
 Suppose there is a physical node with six mountable hard disks/mnt/disk1,/mnt/disk2, ..., /mnt/disk6, where disk1 and disk2 need to be designated as level 0 storage media, disk3 and disk4 are level 1 storage media, and disk5 and disk6 are level 2 storage media. Disk1 is a special mount disk, you can configure it in/etc/taos/taos.cfg as follows:
 
 ```
-dataDir /mnt/disk1/taos
+dataDir /mnt/disk1/taos 0 1 0
-dataDir /mnt/disk2/taos 0
+dataDir /mnt/disk2/taos 0 0 0
-dataDir /mnt/disk3/taos 1
+dataDir /mnt/disk3/taos 1 0 0
-dataDir /mnt/disk4/taos 1
+dataDir /mnt/disk4/taos 1 0 1
-dataDir /mnt/disk5/taos 2
+dataDir /mnt/disk5/taos 2 0 0
-dataDir /mnt/disk6/taos 2
+dataDir /mnt/disk6/taos 2 0 0
 ```
 
 Mounted disks can also be a non-local network disk, as long as the system can access it.

docs/examples/java/pom.xml

@@ -22,7 +22,7 @@
     <dependency>
       <groupId>com.taosdata.jdbc</groupId>
       <artifactId>taos-jdbcdriver</artifactId>
-      <version>3.2.7-SNAPSHOT</version>
+      <version>3.3.0</version>
     </dependency>
     <!--    ANCHOR_END: dep-->
     <dependency>

docs/zh/07-develop/01-connect/index.md

@@ -82,7 +82,7 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速
 <dependency>
   <groupId>com.taosdata.jdbc</groupId>
   <artifactId>taos-jdbcdriver</artifactId>
-  <version>3.2.4</version>
+  <version>3.3.0</version>
 </dependency>
 ```
 

docs/zh/08-connector/14-java.mdx

@@ -36,7 +36,12 @@ REST 连接支持所有能运行 Java 的平台。
 
 | taos-jdbcdriver 版本 |                                                                        主要变化                                                                        |   TDengine 版本    |
 | :------------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------: | :----------------: |
-|        3.2.7         | 支持VARBINARY和GEOMETRY类型，增加native连接的时区设置支持。增加websocket自动重连功能。                                                                                                          | 3.2.0.0 及更高版本 |
+|        3.3.0        | 1. 优化 Websocket 连接下的数据传输性能；2. 支持跳过 SSL 验证，默认关闭      |        3.3.2.0 及更高版本         |
+|        3.2.11        | 解决了 Native 连接关闭结果集 bug      |        -         |
+|        3.2.10        | 1. REST/WebSocket 连接支持传输中的数据压缩；2. Websocket 自动重连机制，默认关闭；3. Connection 类提供无模式写入的方法；4. 优化了原生连接的数据拉取性能；5. 修复了一些已知问题；6.元数据获取函数可以返回支持的函数列表。      |        -         |
+|        3.2.9         | 解决了 Websocket prepareStatement 关闭 bug                                    |        -         |
+|        3.2.8         | 优化了自动提交, 解决了 websocket 手动提交 bug, 优化 Websocket prepareStatement 使用一个连接, 元数据支持视图                                       |        -         |
+|        3.2.7         | 支持 VARBINARY 和 GEOMETRY 类型，增加 native 连接的时区设置支持。增加 websocket 自动重连功能。                                                                      | 3.2.0.0 及更高版本 |
 |        3.2.5         |                                                      数据订阅增加 committed()、assignment() 方法                                                       | 3.1.0.3 及更高版本 |
 |        3.2.4         |                                   数据订阅在 WebSocket 连接下增加 enable.auto.commit 参数，以及 unsubscribe() 方法。                                   |         -          |
 |        3.2.3         |                                                         修复 ResultSet 在一些情况数据解析失败                                                          |         -          |
@@ -178,7 +183,7 @@ Maven 项目中，在 pom.xml 中添加以下依赖：
 <dependency>
   <groupId>com.taosdata.jdbc</groupId>
   <artifactId>taos-jdbcdriver</artifactId>
-  <version>3.2.7</version>
+  <version>3.3.0</version>
 </dependency>
 ```
 
@@ -370,7 +375,16 @@ properties 中的配置参数如下：
 - TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms， 默认值为 60000。 仅在 REST 连接且 batchfetch 设置为 true 时生效。
 - TSDBDriver.PROPERTY_KEY_USE_SSL: 连接中是否使用 SSL。仅在 REST 连接时生效。
 - TSDBDriver.HTTP_POOL_SIZE: REST 并发请求大小，默认 20。
-  此外对 JDBC 原生连接，通过指定 URL 和 Properties 还可以指定其他参数，比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](/reference/config/#仅客户端适用)。
+- TSDBDriver.PROPERTY_KEY_ENABLE_COMPRESSION: 传输过程是否启用压缩。仅在使用 REST/Websocket 连接时生效。true: 启用，false: 不启用。默认为 false。
+- TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: 是否启用自动重连。仅在使用 Websocket 连接时生效。true: 启用，false: 不启用。默认为 false。
+> **注意**：启用自动重连仅对简单执行 SQL 语句以及 无模式写入、数据订阅有效。对于参数绑定无效。自动重连仅对连接建立时通过参数指定数据库有效，对后面的 `use db` 语句切换数据库无效。
+
+- TSDBDriver.PROPERTY_KEY_RECONNECT_INTERVAL_MS: 自动重连重试间隔，单位毫秒，默认值 2000。仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。
+- TSDBDriver.PROPERTY_KEY_RECONNECT_RETRY_COUNT: 自动重连重试次数，默认值 3，仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。
+- TSDBDriver.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: 关闭 SSL 证书验证 。仅在使用 Websocket 连接时生效。true: 启用，false: 不启用。默认为 false。
+
+
+此外对 JDBC 原生连接，通过指定 URL 和 Properties 还可以指定其他参数，比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](../../reference/config/#仅客户端适用)。
 
 ### 配置参数的优先级
 

docs/zh/17-operation/04-import.md

@@ -52,8 +52,14 @@ taos> DESCRIBE d1001
 
 那么可以用如下命令导入数据：
 
+
 ```sql
-taos> insert into d1001 file '~/data.csv';
+taos> insert into d1001 file '~/data.csv'; # For Linux / Mac
+Query OK, 9 row(s) affected (0.004763s)
+```
+
+```sql
+taos> insert into d1001 file 'E:\\taos\\data.csv'; # For Windows
 Query OK, 9 row(s) affected (0.004763s)
 ```
 

docs/zh/20-third-party/01-grafana.mdx

@@ -7,24 +7,21 @@ description: 使用 Grafana 与 TDengine 的详细说明
 import Tabs from "@theme/Tabs";
 import TabItem from "@theme/TabItem";
 
-TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/) 快速集成搭建数据监测报警系统，整个过程无需任何代码开发，TDengine 中数据表的内容可以在仪表盘(DashBoard)上进行可视化展现。关于 TDengine 插件的使用您可以在[GitHub](https://github.com/taosdata/grafanaplugin/blob/master/README.md)中了解更多。
+TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/) 快速集成搭建数据监测报警系统，整个过程无需任何代码开发，TDengine 中数据表的内容可以在仪表盘(DashBoard)上进行可视化展现。关于 TDengine 插件的使用您可以在 [GitHub](https://github.com/taosdata/grafanaplugin/blob/master/README.md) 中了解更多。
 
 ## 前置条件
 
 要让 Grafana 能正常添加 TDengine 数据源，需要以下几方面的准备工作。
 
-- TDengine 集群已经部署并正常运行
+- Grafana 服务已经部署并正常运行。目前 TDengine 支持 Grafana 7.5 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：\<https://grafana.com/grafana/download>。
-- taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](/reference/taosadapter)
+- TDengine 集群已经部署并正常运行。
+- taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](../../reference/taosadapter)
 
 记录以下信息：
 
 - TDengine 集群 REST API 地址，如：`http://tdengine.local:6041`。
 - TDengine 集群认证信息，可使用用户名及密码。
 
-## 安装 Grafana
-
-目前 TDengine 支持 Grafana 7.5 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：<https://grafana.com/grafana/download>。
-
 ## 配置 Grafana
 
 ### 安装 Grafana Plugin 并配置数据源
@@ -32,22 +29,22 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/
 <Tabs defaultValue="script">
 <TabItem value="gui" label="图形化界面安装">
 
-使用 Grafana 最新版本（8.5+），您可以在 Grafana 中[浏览和管理插件](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog)（对于 7.x 版本，请使用 **安装脚本** 或 **手动安装并配置** 方式）。在 Grafana 管理界面中的 **Configurations > Plugins** 页面直接搜索并按照提示安装 TDengine。
+使用 Grafana 最新版本（8.5+），您可以在 Grafana 中[浏览和管理插件](https://grafana.com/docs/grafana/next/administration/plugin-management/#plugin-catalog)（对于 7.x 版本，请采用 **安装脚本** 或 **手动安装** 方式）。在 Grafana 管理界面中的 **Configurations > Plugins** 页面直接搜索 `TDengine` 并按照提示安装。
 
-![Search tdengine in grafana plugins](grafana-plugin-search-tdengine.png)
+安装完毕后，按照指示 **Create a TDengine data source** 添加数据源。
+输入 TDengine 相关配置，如下图所示：
 
-如图示即安装完毕，按照指示 **Create a TDengine data source** 添加数据源。
+![TDengine Database Grafana plugin add data source](./add_datasource3.webp)
 
-![Install and configure Grafana data source](grafana-install-and-config.png)
+- Host： TDengine 集群中提供 REST 服务的 IP 地址与端口号，默认 \<http://localhost:6041>。
-
+- User：TDengine 用户名。
-输入 TDengine 相关配置，完成数据源配置。
+- Password：TDengine 用户密码。
-
-![TDengine Database Grafana plugin add data source](./grafana-data-source.png)
 
+点击 `Save & Test` 进行测试，成功会提示：`TDengine Data source is working`。  
 配置完毕，现在可以使用 TDengine 创建 Dashboard 了。
 
 </TabItem>
-<TabItem value="script" label="使用安装脚本">
+<TabItem value="script" label="安装脚本">
 
 对于使用 Grafana 7.x 版本或使用 [Grafana Provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/) 配置的用户，可以在 Grafana 服务器上使用安装脚本自动安装插件即添加数据源 Provisioning 配置文件。
 
@@ -77,7 +74,7 @@ sudo -u grafana grafana-cli plugins install tdengine-datasource
 或者从 [GitHub](https://github.com/taosdata/grafanaplugin/releases/tag/latest) 或 [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) 下载 .zip 文件到本地并解压到 Grafana 插件目录。命令行下载示例如下：
 
 ```bash
-GF_VERSION=3.3.1
+GF_VERSION=3.5.1
 # from GitHub
 wget https://github.com/taosdata/grafanaplugin/releases/download/v$GF_VERSION/tdengine-datasource-$GF_VERSION.zip
 # from Grafana
@@ -96,25 +93,17 @@ sudo unzip tdengine-datasource-$GF_VERSION.zip -d /var/lib/grafana/plugins/
 GF_INSTALL_PLUGINS=tdengine-datasource
 ```
 
-之后，用户可以直接通过 <http://localhost:3000> 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示：
+之后，用户可以直接通过 \<http://localhost:3000> 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，
 
-![TDengine Database Grafana plugin add data source](./add_datasource1.webp)
+点击 `Add data source` 可进入新增数据源页面，在查询框中输入 TDengine， 然后点击 `select` 选择添加后会进入数据源配置页面，按照默认提示修改相应配置即可：
-
-点击 `Add data source` 可进入新增数据源页面，在查询框中输入 TDengine 可选择添加，如下图所示：
-
-![TDengine Database Grafana plugin add data source](./add_datasource2.webp)
-
-进入数据源配置页面，按照默认提示修改相应配置即可：
 
 ![TDengine Database Grafana plugin add data source](./add_datasource3.webp)
 
-- Host： TDengine 集群中提供 REST 服务 （在 2.4 之前由 taosd 提供， 从 2.4 开始由 taosAdapter 提供）的组件所在服务器的 IP 地址与 TDengine REST 服务的端口号(6041)，默认 <http://localhost:6041>。
+- Host： TDengine 集群中提供 REST 服务的 IP 地址与端口号，默认 \<http://localhost:6041>。
 - User：TDengine 用户名。
 - Password：TDengine 用户密码。
 
-点击 `Save & Test` 进行测试，成功会有如下提示：
+点击 `Save & Test` 进行测试，成功会提示：`TDengine Data source is working`
-
-![TDengine Database Grafana plugin add data source](./add_datasource4.webp)
 
 </TabItem>
 <TabItem value="container" label="K8s/Docker 容器">
@@ -156,7 +145,7 @@ docker run -d \
 
     services:
       tdengine:
-        image: tdengine/tdengine:3.0.2.4
+        image: tdengine/tdengine:3.3.0.0
         environment:
           TAOS_FQDN: tdengine
         volumes:
@@ -192,13 +181,14 @@ docker run -d \
 
 ![TDengine Database Grafana plugin create dashboard](./create_dashboard1.webp)
 
-如上图所示，在 Query 中选中 `TDengine` 数据源，在下方查询框可输入相应 SQL 进行查询，具体说明如下：
+如上图所示，在 Query 中选中 `TDengine` 数据源，在下方查询框可输入相应 SQL 进行查询。 我们继续用智能电表来举例，为了展示曲线美观，此处**用了虚拟数据**。
+具体说明如下：
 
-- INPUT SQL：输入要查询的语句（该 SQL 语句的结果集应为两列多行），例如：`select _wstart, avg(mem_system) from log.dnodes_info where ts >= $from and ts < $to interval($interval)` ，其中，from、to 和 interval 为 TDengine 插件的内置变量，表示从 Grafana 插件面板获取的查询范围和时间间隔。除了内置变量外，`也支持可以使用自定义模板变量`。
+- INPUT SQL：输入要查询的语句（该 SQL 语句的结果集应为两列多行），例如：`select _wstart as ts, avg(current) as current from power.meters where ts > $from and ts < $to interval($interval) fill(null)` ，其中，from、to 和 interval 为 TDengine 插件的内置变量，表示从 Grafana 插件面板获取的时间查询范围和窗口切分间隔。除了内置变量外，也支持使用自定义模板变量。
 - ALIAS BY：可设置当前查询别名。
 - GENERATE SQL： 点击该按钮会自动替换相应变量，并生成最终执行的语句。
-- Group by column name(s)： **半角**逗号分隔的 `group by` 或 `partition by` 列名。如果是 `group by` or `partition by` 查询语句，设置 `Group by` 列，可以展示多维数据。例如：INPUT SQL 为 `select _wstart as ts, avg(mem_system), dnode_ep from log.dnodes_info where ts>=$from and ts<=$to partition by dnode_ep interval($interval)`，设置 Group by 列名为 `dnode_ep`，可以按 `dnode_ep` 展示数据。
+- Group by column(s)： **半角**逗号分隔的 `group by` 或 `partition by` 列名。如果是 `group by` or `partition by` 查询语句，设置 `Group by` 列，可以展示多维数据。例如：INPUT SQL 为 `select  _wstart as ts, groupid, avg(current) as current  from power.meters where ts > $from and ts < $to partition by groupid interval($interval) fill(null)`，设置 Group by 列名为 `groupid`，可以按 `groupid` 展示数据。
-- Format to： Group by 或 Partition by 场景下多维数据 legend 格式化格式。例如上述 INPUT SQL，将 Format to 设置为 `mem_system_{{dnode_ep}}`，展示的 legend 名字为格式化的列名。
+- Group By Format： Group by 或 Partition by 场景下多维数据 legend 格式化格式。例如上述 INPUT SQL，将 `Group By Format` 设置为 `groupid-{{groupid}}`，展示的 legend 名字为格式化的分组名。
 
 :::note
 
@@ -206,11 +196,11 @@ docker run -d \
 
 :::
 
-按照默认提示查询当前 TDengine 部署所在服务器指定间隔系统内存平均使用量如下：
+查询 `meters` 超级表所有设备电流平均值变化如下图：
 
 ![TDengine Database Grafana plugin create dashboard](./create_dashboard2.webp)
 
-查询每台 TDengine 服务器指定间隔系统内存平均使用量如下：
+查询 `meters` 超级表所有设备电流平均值，并按照 `groupid` 分组展示如下图：
 
 ![TDengine Database Grafana plugin create dashboard](./create_dashboard3.webp)
 

docs/zh/21-tdinternal/01-arch.md

@@ -206,7 +206,7 @@ TDengine 采用数据驱动的方式让缓存中的数据写入硬盘进行持
 TDengine 多级存储配置方式如下（在配置文件/etc/taos/taos.cfg 中）：
 
 ```
-dataDir [path] <level> <primary>
+dataDir [path] <level> <primary> <disable_create_new_file>
 ```
 
 - path: 挂载点的文件夹路径
@@ -216,16 +216,17 @@ dataDir [path] <level> <primary>
   同一存储等级可挂载多个硬盘，同一存储等级上的数据文件分布在该存储等级的所有硬盘上。
   需要说明的是，数据在不同级别的存储介质上的移动，是由系统自动完成的，用户无需干预。
 - primary: 是否为主挂载点，0（否）或 1（是），省略默认为 1。
+- disable_create_new_file: 是否禁止创建新文件组，0（否）或 1（是），省略默认为 0。
 
 在配置中，只允许一个主挂载点的存在（level=0，primary=1），例如采用如下的配置方式：
 
 ```
-dataDir /mnt/data1 0 1
+dataDir /mnt/data1 0 1 0
-dataDir /mnt/data2 0 0
+dataDir /mnt/data2 0 0 0
-dataDir /mnt/data3 1 0
+dataDir /mnt/data3 1 0 0
-dataDir /mnt/data4 1 0
+dataDir /mnt/data4 1 0 1
-dataDir /mnt/data5 2 0
+dataDir /mnt/data5 2 0 0
-dataDir /mnt/data6 2 0
+dataDir /mnt/data6 2 0 0
 ```
 
 :::note
@@ -233,6 +234,7 @@ dataDir /mnt/data6 2 0
 1. 多级存储不允许跨级配置，合法的配置方案有：仅 0 级，仅 0 级+ 1 级，以及 0 级+ 1 级+ 2 级。而不允许只配置 level=0 和 level=2，而不配置 level=1。
 2. 禁止手动移除使用中的挂载盘，挂载盘目前不支持非本地的网络盘。
 3. 多级存储目前不支持删除已经挂载的硬盘的功能。
+4. 0 级存储至少存在一个 disable_create_new_file 为 0 的挂载点，1 级 和 2 级存储没有该限制。
 
 :::
 

examples/JDBC/JDBCDemo/pom.xml

@@ -19,7 +19,7 @@
         <dependency>
             <groupId>com.taosdata.jdbc</groupId>
             <artifactId>taos-jdbcdriver</artifactId>
-            <version>3.2.7</version>
+            <version>3.3.0</version>
         </dependency>
         <dependency>
             <groupId>org.locationtech.jts</groupId>

examples/JDBC/taosdemo/pom.xml

@@ -67,7 +67,7 @@
         <dependency>
             <groupId>com.taosdata.jdbc</groupId>
             <artifactId>taos-jdbcdriver</artifactId>
-            <version>3.2.7</version>
+            <version>3.3.0</version>
             <!--  <scope>system</scope>-->
             <!--  <systemPath>${project.basedir}/src/main/resources/lib/taos-jdbcdriver-2.0.15-dist.jar</systemPath>-->
         </dependency>

include/common/tmsg.h

@@ -3506,12 +3506,6 @@ int32_t tSerializeSMDropStreamReq(void* buf, int32_t bufLen, const SMDropStreamR
 int32_t tDeserializeSMDropStreamReq(void* buf, int32_t bufLen, SMDropStreamReq* pReq);
 void    tFreeMDropStreamReq(SMDropStreamReq* pReq);
 
-typedef struct {
-  int64_t recoverObjUid;
-  int32_t taskId;
-  int32_t hasCheckPoint;
-} SMVStreamGatherInfoReq;
-
 typedef struct SVUpdateCheckpointInfoReq {
   SMsgHead head;
   int64_t  streamId;
@@ -3537,50 +3531,8 @@ typedef struct {
   int64_t suid;
 } SMqRebVgReq;
 
-static FORCE_INLINE int tEncodeSMqRebVgReq(SEncoder* pCoder, const SMqRebVgReq* pReq) {
+int32_t tEncodeSMqRebVgReq(SEncoder* pCoder, const SMqRebVgReq* pReq);
-  if (tStartEncode(pCoder) < 0) return -1;
+int32_t tDecodeSMqRebVgReq(SDecoder* pCoder, SMqRebVgReq* pReq);
-  if (tEncodeI64(pCoder, pReq->leftForVer) < 0) return -1;
-  if (tEncodeI32(pCoder, pReq->vgId) < 0) return -1;
-  if (tEncodeI64(pCoder, pReq->oldConsumerId) < 0) return -1;
-  if (tEncodeI64(pCoder, pReq->newConsumerId) < 0) return -1;
-  if (tEncodeCStr(pCoder, pReq->subKey) < 0) return -1;
-  if (tEncodeI8(pCoder, pReq->subType) < 0) return -1;
-  if (tEncodeI8(pCoder, pReq->withMeta) < 0) return -1;
-
-  if (pReq->subType == TOPIC_SUB_TYPE__COLUMN) {
-    if (tEncodeCStr(pCoder, pReq->qmsg) < 0) return -1;
-  } else if (pReq->subType == TOPIC_SUB_TYPE__TABLE) {
-    if (tEncodeI64(pCoder, pReq->suid) < 0) return -1;
-    if (tEncodeCStr(pCoder, pReq->qmsg) < 0) return -1;
-  }
-  tEndEncode(pCoder);
-  return 0;
-}
-
-static FORCE_INLINE int tDecodeSMqRebVgReq(SDecoder* pCoder, SMqRebVgReq* pReq) {
-  if (tStartDecode(pCoder) < 0) return -1;
-
-  if (tDecodeI64(pCoder, &pReq->leftForVer) < 0) return -1;
-
-  if (tDecodeI32(pCoder, &pReq->vgId) < 0) return -1;
-  if (tDecodeI64(pCoder, &pReq->oldConsumerId) < 0) return -1;
-  if (tDecodeI64(pCoder, &pReq->newConsumerId) < 0) return -1;
-  if (tDecodeCStrTo(pCoder, pReq->subKey) < 0) return -1;
-  if (tDecodeI8(pCoder, &pReq->subType) < 0) return -1;
-  if (tDecodeI8(pCoder, &pReq->withMeta) < 0) return -1;
-
-  if (pReq->subType == TOPIC_SUB_TYPE__COLUMN) {
-    if (tDecodeCStr(pCoder, &pReq->qmsg) < 0) return -1;
-  } else if (pReq->subType == TOPIC_SUB_TYPE__TABLE) {
-    if (tDecodeI64(pCoder, &pReq->suid) < 0) return -1;
-    if (!tDecodeIsEnd(pCoder)) {
-      if (tDecodeCStr(pCoder, &pReq->qmsg) < 0) return -1;
-    }
-  }
-
-  tEndDecode(pCoder);
-  return 0;
-}
 
 typedef struct {
   char    topic[TSDB_TOPIC_FNAME_LEN];

include/common/tmsgdef.h

@@ -250,7 +250,9 @@
   TD_DEF_MSG_TYPE(TDMT_MND_DROP_TB_WITH_TSMA, "drop-tb-with-tsma", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_STREAM_UPDATE_CHKPT_EVT, "stream-update-chkpt-evt", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_STREAM_CHKPT_REPORT, "stream-chkpt-report", NULL, NULL)
-  TD_CLOSE_MSG_SEG(TDMT_MND_MSG)
+  TD_DEF_MSG_TYPE(TDMT_MND_STREAM_CHKPT_CONSEN, "stream-chkpt-consen", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_MND_MAX_MSG, "mnd-max", NULL, NULL)
+  TD_CLOSE_MSG_SEG(TDMT_END_MND_MSG)
 
   TD_NEW_MSG_SEG(TDMT_VND_MSG)  // 2<<8
   TD_DEF_MSG_TYPE(TDMT_VND_SUBMIT, "submit", SSubmitReq, SSubmitRsp)
@@ -331,7 +333,7 @@
   TD_DEF_MSG_TYPE(TDMT_STREAM_RETRIEVE, "stream-retrieve", NULL, NULL)  //1035 1036
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_CHECKPOINT_READY, "stream-checkpoint-ready", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_REPORT_CHECKPOINT, "stream-report-checkpoint", NULL, NULL)
-  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_RESTORE_CHECKPOINT, "stream-restore-checkpoint", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_RESTORE_CHECKPOINT, "stream-restore-checkpoint", NULL, NULL)  //unused
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_PAUSE, "stream-task-pause", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_RESUME, "stream-task-resume", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_STOP, "stream-task-stop", NULL, NULL)

include/dnode/mnode/mnode.h

@@ -109,7 +109,7 @@ int64_t mndGetRoleTimeMs(SMnode *pMnode);
  * @param pMsg The request msg.
  * @return int32_t 0 for success, -1 for failure.
  */
-int32_t mndProcessRpcMsg(SRpcMsg *pMsg);
+int32_t mndProcessRpcMsg(SRpcMsg *pMsg, SQueueInfo* pQueueInfo);
 int32_t mndProcessSyncMsg(SRpcMsg *pMsg);
 int32_t mndPreProcessQueryMsg(SRpcMsg *pMsg);
 void    mndPostProcessQueryMsg(SRpcMsg *pMsg);

include/dnode/qnode/qnode.h

@@ -60,7 +60,7 @@ int32_t qndGetLoad(SQnode *pQnode, SQnodeLoad *pLoad);
  * @param pQnode The qnode object.
  * @param pMsg The request message
  */
-int32_t qndProcessQueryMsg(SQnode *pQnode, int64_t ts, SRpcMsg *pMsg);
+int32_t qndProcessQueryMsg(SQnode *pQnode, SQueueInfo* pInfo, SRpcMsg *pMsg);
 
 #ifdef __cplusplus
 }

include/dnode/vnode/tqCommon.h

@@ -29,6 +29,7 @@ int32_t tqStreamTaskProcessCheckpointReadyMsg(SStreamMeta* pMeta, SRpcMsg* pMsg)
 int32_t tqStreamProcessStreamHbRsp(SStreamMeta* pMeta, SRpcMsg* pMsg);
 int32_t tqStreamProcessReqCheckpointRsp(SStreamMeta* pMeta, SRpcMsg* pMsg);
 int32_t tqStreamProcessChkptReportRsp(SStreamMeta* pMeta, SRpcMsg* pMsg);
+int32_t tqStreamProcessConsensusChkptRsp(SStreamMeta* pMeta, SRpcMsg* pMsg);
 int32_t tqStreamProcessCheckpointReadyRsp(SStreamMeta* pMeta, SRpcMsg* pMsg);
 int32_t tqStreamTaskProcessDeployReq(SStreamMeta* pMeta, SMsgCb* cb, int64_t sversion, char* msg, int32_t msgLen,
                                      bool isLeader, bool restored);

include/libs/executor/executor.h

@@ -57,6 +57,7 @@ typedef struct {
   STimeWindow winRange;
 
   struct SStorageAPI api;
+  void*              pWorkerCb;
 } SReadHandle;
 
 // in queue mode, data streams are seperated by msg

include/libs/parser/parser.h

@@ -101,6 +101,7 @@ typedef struct SParseContext {
 
 int32_t qParseSql(SParseContext* pCxt, SQuery** pQuery);
 bool    qIsInsertValuesSql(const char* pStr, size_t length);
+bool    qParseDbName(const char* pStr, size_t length, char** pDbName);
 
 // for async mode
 int32_t qParseSqlSyntax(SParseContext* pCxt, SQuery** pQuery, struct SCatalogReq* pCatalogReq);

include/libs/qcom/query.h

@@ -297,6 +297,8 @@ int32_t cleanupTaskQueue();
  * @return
  */
 int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code);
+int32_t taosAsyncWait();
+int32_t taosAsyncRecover();
 
 void destroySendMsgInfo(SMsgSendInfo* pMsgBody);
 

include/libs/stream/streamMsg.h

@@ -17,6 +17,7 @@
 #define TDENGINE_STREAMMSG_H
 
 #include "tmsg.h"
+#include "trpc.h"
 
 #ifdef __cplusplus
 extern "C" {
@@ -162,6 +163,7 @@ int32_t tDecodeStreamTaskCheckpointReq(SDecoder* pDecoder, SStreamTaskCheckpoint
 
 typedef struct SStreamHbMsg {
   int32_t vgId;
+  int32_t msgId;
   int32_t numOfTasks;
   SArray* pTaskStatus;   // SArray<STaskStatusEntry>
   SArray* pUpdateNodes;  // SArray<int32_t>, needs update the epsets in stream tasks for those nodes.
@@ -171,6 +173,11 @@ int32_t tEncodeStreamHbMsg(SEncoder* pEncoder, const SStreamHbMsg* pRsp);
 int32_t tDecodeStreamHbMsg(SDecoder* pDecoder, SStreamHbMsg* pRsp);
 void    tCleanupStreamHbMsg(SStreamHbMsg* pMsg);
 
+typedef struct {
+  SMsgHead head;
+  int32_t  msgId;
+} SMStreamHbRspMsg;
+
 typedef struct SRetrieveChkptTriggerReq {
   SMsgHead head;
   int64_t  streamId;
@@ -204,6 +211,28 @@ typedef struct SCheckpointReport {
 int32_t tEncodeStreamTaskChkptReport(SEncoder* pEncoder, const SCheckpointReport* pReq);
 int32_t tDecodeStreamTaskChkptReport(SDecoder* pDecoder, SCheckpointReport* pReq);
 
+typedef struct SRestoreCheckpointInfo {
+  SMsgHead head;
+  int64_t  startTs;
+  int64_t  streamId;
+  int64_t  checkpointId;   // latest checkpoint id
+  int32_t  taskId;
+  int32_t  nodeId;
+} SRestoreCheckpointInfo;
+
+int32_t tEncodeRestoreCheckpointInfo (SEncoder* pEncoder, const SRestoreCheckpointInfo* pReq);
+int32_t tDecodeRestoreCheckpointInfo(SDecoder* pDecoder, SRestoreCheckpointInfo* pReq);
+
+typedef struct SRestoreCheckpointInfoRsp {
+  int64_t streamId;
+  int64_t checkpointId;
+  int64_t startTs;
+  int32_t taskId;
+} SRestoreCheckpointInfoRsp;
+
+int32_t tEncodeRestoreCheckpointInfoRsp(SEncoder* pCoder, const SRestoreCheckpointInfoRsp* pInfo);
+int32_t tDecodeRestoreCheckpointInfoRsp(SDecoder* pCoder, SRestoreCheckpointInfoRsp* pInfo);
+
 typedef struct {
   SMsgHead head;
   int64_t  streamId;
@@ -211,6 +240,12 @@ typedef struct {
   int32_t  reqType;
 } SStreamTaskRunReq;
 
+typedef struct SCheckpointConsensusEntry {
+  SRestoreCheckpointInfo req;
+  SRpcMsg rsp;
+  int64_t ts;
+} SCheckpointConsensusEntry;
+
 #ifdef __cplusplus
 }
 #endif

include/libs/stream/tstream.h

@@ -17,8 +17,9 @@
 #define TDENGINE_TSTREAM_H
 
 #include "os.h"
+#include "streamMsg.h"
 #include "streamState.h"
-#include "streammsg.h"
+#include "streamMsg.h"
 #include "tdatablock.h"
 #include "tdbInt.h"
 #include "tmsg.h"
@@ -265,12 +266,13 @@ typedef struct SStreamTaskId {
 } SStreamTaskId;
 
 typedef struct SCheckpointInfo {
-  int64_t                startTs;
+  int64_t startTs;
-  int64_t                checkpointId;    // latest checkpoint id
+  int64_t checkpointId;    // latest checkpoint id
-  int64_t                checkpointVer;   // latest checkpoint offset in wal
+  int64_t checkpointVer;   // latest checkpoint offset in wal
-  int64_t                checkpointTime;  // latest checkpoint time
+  int64_t checkpointTime;  // latest checkpoint time
-  int64_t                processedVer;
+  int64_t processedVer;
-  int64_t                nextProcessVer;  // current offset in WAL, not serialize it
+  int64_t nextProcessVer;  // current offset in WAL, not serialize it
+
   SActiveCheckpointInfo* pActiveInfo;
   int64_t                msgVer;
 } SCheckpointInfo;
@@ -523,7 +525,6 @@ typedef struct STaskUpdateEntry {
 } STaskUpdateEntry;
 
 typedef int32_t (*__state_trans_user_fn)(SStreamTask*, void* param);
-typedef int32_t (*__stream_task_expand_fn)(struct SStreamTask* pTask);
 
 SStreamTask* tNewStreamTask(int64_t streamId, int8_t taskLevel, SEpSet* pEpset, bool fillHistory, int64_t triggerParam,
                             SArray* pTaskList, bool hasFillhistory, int8_t subtableWithoutMd5);
@@ -613,6 +614,12 @@ typedef struct SStreamTaskState {
   char*       name;
 } SStreamTaskState;
 
+typedef struct SCheckpointConsensusInfo {
+  SArray* pTaskList;
+  int64_t checkpointId;
+  int64_t genTs;
+} SCheckpointConsensusInfo;
+
 int32_t streamSetupScheduleTrigger(SStreamTask* pTask);
 
 // dispatch related
@@ -704,6 +711,7 @@ void    streamTaskSetRemoveBackendFiles(SStreamTask* pTask);
 
 void streamTaskStatusInit(STaskStatusEntry* pEntry, const SStreamTask* pTask);
 void streamTaskStatusCopy(STaskStatusEntry* pDst, const STaskStatusEntry* pSrc);
+STaskStatusEntry streamTaskGetStatusEntry(SStreamTask* pTask);
 
 // source level
 int32_t streamSetParamForStreamScannerStep1(SStreamTask* pTask, SVersionRange* pVerRange, STimeWindow* pWindow);
@@ -746,14 +754,15 @@ void    streamMetaRLock(SStreamMeta* pMeta);
 void    streamMetaRUnLock(SStreamMeta* pMeta);
 void    streamMetaWLock(SStreamMeta* pMeta);
 void    streamMetaWUnLock(SStreamMeta* pMeta);
-void    streamMetaResetStartInfo(STaskStartInfo* pMeta);
+void    streamMetaResetStartInfo(STaskStartInfo* pMeta, int32_t vgId);
 SArray* streamMetaSendMsgBeforeCloseTasks(SStreamMeta* pMeta);
 void    streamMetaUpdateStageRole(SStreamMeta* pMeta, int64_t stage, bool isLeader);
 void    streamMetaLoadAllTasks(SStreamMeta* pMeta);
-int32_t streamMetaStartAllTasks(SStreamMeta* pMeta, __stream_task_expand_fn fn);
+int32_t streamMetaStartAllTasks(SStreamMeta* pMeta);
 int32_t streamMetaStopAllTasks(SStreamMeta* pMeta);
-int32_t streamMetaStartOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, __stream_task_expand_fn fn);
+int32_t streamMetaStartOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId);
 bool    streamMetaAllTasksReady(const SStreamMeta* pMeta);
+int32_t streamTaskSendRestoreChkptMsg(SStreamTask* pTask);
 
 // timer
 tmr_h streamTimerGetInstance();
@@ -790,6 +799,9 @@ int32_t streamProcessRetrieveReq(SStreamTask* pTask, SStreamRetrieveReq* pReq);
 int32_t streamTaskBroadcastRetrieveReq(SStreamTask* pTask, SStreamRetrieveReq* req);
 void    streamTaskSendRetrieveRsp(SStreamRetrieveReq* pReq, SRpcMsg* pRsp);
 
+int32_t streamProcessHeartbeatRsp(SStreamMeta* pMeta, SMStreamHbRspMsg* pRsp);
+
+
 #ifdef __cplusplus
 }
 #endif

include/util/taoserror.h

@@ -387,6 +387,7 @@ int32_t  taosGetErrSize();
 #define TSDB_CODE_MND_LAST_TRANS_NOT_FINISHED   TAOS_DEF_ERROR_CODE(0, 0x03D6) //internal
 #define TSDB_CODE_MND_TRANS_SYNC_TIMEOUT        TAOS_DEF_ERROR_CODE(0, 0x03D7)
 #define TSDB_CODE_MND_TRANS_CTX_SWITCH          TAOS_DEF_ERROR_CODE(0, 0x03D8)
+#define TSDB_CODE_MND_TRANS_CONFLICT_COMPACT    TAOS_DEF_ERROR_CODE(0, 0x03D9)
 #define TSDB_CODE_MND_TRANS_UNKNOW_ERROR        TAOS_DEF_ERROR_CODE(0, 0x03DF)
 
 // mnode-mq

include/util/tlist.h

@@ -218,6 +218,7 @@ typedef struct {
 #define listEleSize(l)  ((l)->eleSize)
 #define isListEmpty(l)  (TD_DLIST_NELES(l) == 0)
 #define listNodeFree(n) taosMemoryFree(n)
+#define listNode(data) (SListNode*)(((char*)(data)) - sizeof(SListNode))
 
 void       tdListInit(SList *list, int32_t eleSize);
 void       tdListEmpty(SList *list);

include/util/tqueue.h

@@ -49,6 +49,7 @@ typedef struct {
   int32_t workerId;
   int32_t threadNum;
   int64_t timestamp;
+  void   *workerCb;
 } SQueueInfo;
 
 typedef enum {

include/util/tworker.h

@@ -16,6 +16,7 @@
 #ifndef _TD_UTIL_WORKER_H_
 #define _TD_UTIL_WORKER_H_
 
+#include "tlist.h"
 #include "tqueue.h"
 #include "tarray.h"
 
@@ -83,18 +84,25 @@ void        tWWorkerCleanup(SWWorkerPool *pool);
 STaosQueue *tWWorkerAllocQueue(SWWorkerPool *pool, void *ahandle, FItems fp);
 void        tWWorkerFreeQueue(SWWorkerPool *pool, STaosQueue *queue);
 
+typedef enum SQWorkerPoolType {
+  QWORKER_POOL = 0,
+  QUERY_AUTO_QWORKER_POOL,
+} SQWorkerPoolType;
+
 typedef struct {
-  const char *name;
+  const char      *name;
-  int32_t     min;
+  int32_t          min;
-  int32_t     max;
+  int32_t          max;
-  FItem       fp;
+  FItem            fp;
-  void       *param;
+  void            *param;
+  SQWorkerPoolType poolType;
 } SSingleWorkerCfg;
 
 typedef struct {
-  const char  *name;
+  const char      *name;
-  STaosQueue  *queue;
+  STaosQueue      *queue;
-  SQWorkerPool pool;
+  SQWorkerPoolType poolType;  // default to QWORKER_POOL
+  void            *pool;
 } SSingleWorker;
 
 typedef struct {
@@ -115,6 +123,59 @@ void    tSingleWorkerCleanup(SSingleWorker *pWorker);
 int32_t tMultiWorkerInit(SMultiWorker *pWorker, const SMultiWorkerCfg *pCfg);
 void    tMultiWorkerCleanup(SMultiWorker *pWorker);
 
+struct SQueryAutoQWorkerPoolCB;
+
+typedef struct SQueryAutoQWorker {
+  int32_t  id;      // worker id
+  int32_t  backupIdx;// the idx when put into backup pool
+  int64_t  pid;     // thread pid
+  TdThread thread;  // thread id
+  void    *pool;
+} SQueryAutoQWorker;
+
+typedef struct SQueryAutoQWorkerPool {
+  int32_t       num;
+  int32_t       max;
+  int32_t       min;
+  int32_t       maxInUse;
+
+  int64_t       activeRunningN; // 4 bytes for activeN, 4 bytes for runningN
+  // activeN are running workers and workers waiting at reading new queue msgs
+  // runningN are workers processing queue msgs, not include blocking/waitingAfterBlock/waitingBeforeProcessMsg workers.
+
+  int32_t       waitingAfterBlockN; // workers that recovered from blocking but waiting for too many running workers
+  TdThreadMutex waitingAfterBlockLock;
+  TdThreadCond  waitingAfterBlockCond;
+
+  int32_t       waitingBeforeProcessMsgN; // workers that get msg from queue, but waiting for too many running workers
+  TdThreadMutex waitingBeforeProcessMsgLock;
+  TdThreadCond  waitingBeforeProcessMsgCond;
+
+  int32_t       backupNum; // workers that are in backup pool, not reading msg from queue
+  TdThreadMutex backupLock;
+  TdThreadCond  backupCond;
+
+  const char                     *name;
+  TdThreadMutex                   poolLock;
+  SList                          *workers;
+  SList                          *backupWorkers;
+  SList                          *exitedWorkers;
+  STaosQset                      *qset;
+  struct SQueryAutoQWorkerPoolCB *pCb;
+  bool                            exit;
+} SQueryAutoQWorkerPool;
+
+int32_t     tQueryAutoQWorkerInit(SQueryAutoQWorkerPool *pPool);
+void        tQueryAutoQWorkerCleanup(SQueryAutoQWorkerPool *pPool);
+STaosQueue *tQueryAutoQWorkerAllocQueue(SQueryAutoQWorkerPool *pPool, void *ahandle, FItem fp);
+void        tQueryAutoQWorkerFreeQueue(SQueryAutoQWorkerPool* pPool, STaosQueue* pQ);
+
+typedef struct SQueryAutoQWorkerPoolCB {
+  void *pPool;
+  int32_t (*beforeBlocking)(void *pPool);
+  int32_t (*afterRecoverFromBlocking)(void *pPool);
+} SQueryAutoQWorkerPoolCB;
+
 #ifdef __cplusplus
 }
 #endif

source/client/inc/clientStmt.h

@@ -219,6 +219,7 @@ const char *stmtErrstr(TAOS_STMT *stmt);
 int         stmtAffectedRows(TAOS_STMT *stmt);
 int         stmtAffectedRowsOnce(TAOS_STMT *stmt);
 int         stmtPrepare(TAOS_STMT *stmt, const char *sql, unsigned long length);
+int         stmtSetDbName(TAOS_STMT* stmt, const char* dbName);
 int         stmtSetTbName(TAOS_STMT *stmt, const char *tbName);
 int         stmtSetTbTags(TAOS_STMT *stmt, TAOS_MULTI_BIND *tags);
 int         stmtGetTagFields(TAOS_STMT *stmt, int *nums, TAOS_FIELD_E **fields);

source/client/src/clientHb.c

@@ -1029,7 +1029,7 @@ int32_t hbQueryHbReqHandle(SClientHbKey *connKey, void *param, SClientHbReq *req
 #endif
     code = hbGetExpiredTSMAInfo(connKey, pCatalog, req);
   } else {
-    req->app.appId = 0;
+    hbGetAppInfo(hbParam->clusterId, req);
   }
 
   ++hbParam->reqCnt;  // success to get catalog info

source/client/src/clientStmt.c

@@ -896,6 +896,12 @@ int stmtPrepare(TAOS_STMT* stmt, const char* sql, unsigned long length) {
   pStmt->sql.sqlLen = length;
   pStmt->sql.stbInterlaceMode = pStmt->stbInterlaceMode;
 
+  char* dbName = NULL;
+  if (qParseDbName(sql, length, &dbName)) {
+    stmtSetDbName(stmt, dbName);
+    taosMemoryFreeClear(dbName);
+  }
+
   return TSDB_CODE_SUCCESS;
 }
 
@@ -924,6 +930,22 @@ int32_t stmtInitStbInterlaceTableInfo(STscStmt* pStmt) {
   return TSDB_CODE_SUCCESS;
 }
 
+int stmtSetDbName(TAOS_STMT* stmt, const char* dbName) {
+  STscStmt *pStmt = (STscStmt *) stmt;
+
+  STMT_DLOG("start to set dbName: %s", dbName);
+
+  STMT_ERR_RET(stmtCreateRequest(pStmt));
+
+  // The SQL statement specifies a database name, overriding the previously specified database
+  taosMemoryFreeClear(pStmt->exec.pRequest->pDb);
+  pStmt->exec.pRequest->pDb = taosStrdup(dbName);
+  if (pStmt->exec.pRequest->pDb == NULL) {
+    return TSDB_CODE_OUT_OF_MEMORY;
+  }
+  return TSDB_CODE_SUCCESS;
+}
+
 int stmtSetTbName(TAOS_STMT* stmt, const char* tbName) {
   STscStmt* pStmt = (STscStmt*)stmt;
 

source/client/test/clientTests.cpp

@@ -828,7 +828,6 @@ TEST(clientCase, projection_query_tables) {
   //    printf("error in create db, reason:%s\n", taos_errstr(pRes));
   //  }
   //  taos_free_result(pRes);
-
   pRes= taos_query(pConn, "use abc1");
   taos_free_result(pRes);
 

source/common/src/rsync.c

@@ -151,9 +151,9 @@ void startRsync() {
   // start rsync service to backup checkpoint
   code = system(cmd);
   if (code != 0) {
-    uError("[rsync] start server failed, code:%d," ERRNO_ERR_FORMAT, code, ERRNO_ERR_DATA);
+    uError("[rsync] cmd:%s start server failed, code:%d," ERRNO_ERR_FORMAT, cmd, code, ERRNO_ERR_DATA);
   } else {
-    uDebug("[rsync] start server successful");
+    uInfo("[rsync] cmd:%s start server successful", cmd);
   }
 }
 

source/common/src/tglobal.c

@@ -669,12 +669,12 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
   if (cfgAddInt32(pCfg, "retentionSpeedLimitMB", tsRetentionSpeedLimitMB, 0, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
 
   if (cfgAddInt32(pCfg, "numOfMnodeReadThreads", tsNumOfMnodeReadThreads, 1, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
-  if (cfgAddInt32(pCfg, "numOfVnodeQueryThreads", tsNumOfVnodeQueryThreads, 4, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
+  if (cfgAddInt32(pCfg, "numOfVnodeQueryThreads", tsNumOfVnodeQueryThreads, 1, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
   if (cfgAddFloat(pCfg, "ratioOfVnodeStreamThreads", tsRatioOfVnodeStreamThreads, 0.01, 4, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
   if (cfgAddInt32(pCfg, "numOfVnodeFetchThreads", tsNumOfVnodeFetchThreads, 4, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
 
   if (cfgAddInt32(pCfg, "numOfVnodeRsmaThreads", tsNumOfVnodeRsmaThreads, 1, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
-  if (cfgAddInt32(pCfg, "numOfQnodeQueryThreads", tsNumOfQnodeQueryThreads, 4, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
+  if (cfgAddInt32(pCfg, "numOfQnodeQueryThreads", tsNumOfQnodeQueryThreads, 1, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0) return -1;
 
   //  tsNumOfQnodeFetchThreads = tsNumOfCores / 2;
   //  tsNumOfQnodeFetchThreads = TMAX(tsNumOfQnodeFetchThreads, 4);

source/common/src/tmsg.c

@@ -9296,6 +9296,52 @@ int32_t tDecodeSTqCheckInfo(SDecoder *pDecoder, STqCheckInfo *pInfo) {
 }
 void tDeleteSTqCheckInfo(STqCheckInfo *pInfo) { taosArrayDestroy(pInfo->colIdList); }
 
+int32_t tEncodeSMqRebVgReq(SEncoder* pCoder, const SMqRebVgReq* pReq) {
+  if (tStartEncode(pCoder) < 0) return -1;
+  if (tEncodeI64(pCoder, pReq->leftForVer) < 0) return -1;
+  if (tEncodeI32(pCoder, pReq->vgId) < 0) return -1;
+  if (tEncodeI64(pCoder, pReq->oldConsumerId) < 0) return -1;
+  if (tEncodeI64(pCoder, pReq->newConsumerId) < 0) return -1;
+  if (tEncodeCStr(pCoder, pReq->subKey) < 0) return -1;
+  if (tEncodeI8(pCoder, pReq->subType) < 0) return -1;
+  if (tEncodeI8(pCoder, pReq->withMeta) < 0) return -1;
+
+  if (pReq->subType == TOPIC_SUB_TYPE__COLUMN) {
+    if (tEncodeCStr(pCoder, pReq->qmsg) < 0) return -1;
+  } else if (pReq->subType == TOPIC_SUB_TYPE__TABLE) {
+    if (tEncodeI64(pCoder, pReq->suid) < 0) return -1;
+    if (tEncodeCStr(pCoder, pReq->qmsg) < 0) return -1;
+  }
+  tEndEncode(pCoder);
+  return 0;
+}
+
+int32_t tDecodeSMqRebVgReq(SDecoder* pCoder, SMqRebVgReq* pReq) {
+  if (tStartDecode(pCoder) < 0) return -1;
+
+  if (tDecodeI64(pCoder, &pReq->leftForVer) < 0) return -1;
+
+  if (tDecodeI32(pCoder, &pReq->vgId) < 0) return -1;
+  if (tDecodeI64(pCoder, &pReq->oldConsumerId) < 0) return -1;
+  if (tDecodeI64(pCoder, &pReq->newConsumerId) < 0) return -1;
+  if (tDecodeCStrTo(pCoder, pReq->subKey) < 0) return -1;
+  if (tDecodeI8(pCoder, &pReq->subType) < 0) return -1;
+  if (tDecodeI8(pCoder, &pReq->withMeta) < 0) return -1;
+
+  if (pReq->subType == TOPIC_SUB_TYPE__COLUMN) {
+    if (tDecodeCStr(pCoder, &pReq->qmsg) < 0) return -1;
+  } else if (pReq->subType == TOPIC_SUB_TYPE__TABLE) {
+    if (tDecodeI64(pCoder, &pReq->suid) < 0) return -1;
+    if (!tDecodeIsEnd(pCoder)) {
+      if (tDecodeCStr(pCoder, &pReq->qmsg) < 0) return -1;
+    }
+  }
+
+  tEndDecode(pCoder);
+  return 0;
+}
+
+
 int32_t tEncodeDeleteRes(SEncoder *pCoder, const SDeleteRes *pRes) {
   int32_t nUid = taosArrayGetSize(pRes->uidList);
 

source/dnode/mgmt/mgmt_mnode/src/mmHandle.c

@@ -242,6 +242,7 @@ SArray *mmGetMsgHandles() {
   if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_RESET_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_HEARTBEAT, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_REPORT, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_CONSEN, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_REQ_CHKPT, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_VND_KILL_COMPACT_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
 

source/dnode/mgmt/mgmt_mnode/src/mmWorker.c

@@ -53,7 +53,7 @@ static void mmProcessRpcMsg(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   const STraceId *trace = &pMsg->info.traceId;
   dGTrace("msg:%p, get from mnode queue, type:%s", pMsg, TMSG_INFO(pMsg->msgType));
 
-  int32_t code = mndProcessRpcMsg(pMsg);
+  int32_t code = mndProcessRpcMsg(pMsg, pInfo);
 
   if (pInfo->timestamp != 0) {
     int64_t cost = taosGetTimestampUs() - pInfo->timestamp;
@@ -203,6 +203,7 @@ int32_t mmStartWorker(SMnodeMgmt *pMgmt) {
       .name = "mnode-query",
       .fp = (FItem)mmProcessRpcMsg,
       .param = pMgmt,
+      .poolType = QUERY_AUTO_QWORKER_POOL,
   };
   if (tSingleWorkerInit(&pMgmt->queryWorker, &qCfg) != 0) {
     dError("failed to start mnode-query worker since %s", terrstr());

source/dnode/mgmt/mgmt_qnode/src/qmWorker.c

@@ -30,7 +30,7 @@ static void qmProcessQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   SQnodeMgmt *pMgmt = pInfo->ahandle;
   dTrace("msg:%p, get from qnode queue", pMsg);
 
-  int32_t code = qndProcessQueryMsg(pMgmt->pQnode, pInfo->timestamp, pMsg);
+  int32_t code = qndProcessQueryMsg(pMgmt->pQnode, pInfo, pMsg);
   if (IsReq(pMsg) && code != TSDB_CODE_ACTION_IN_PROGRESS) {
     if (code != 0 && terrno != 0) code = terrno;
     qmSendRsp(pMsg, code);
@@ -105,6 +105,7 @@ int32_t qmStartWorker(SQnodeMgmt *pMgmt) {
       .name = "qnode-query",
       .fp = (FItem)qmProcessQueue,
       .param = pMgmt,
+      .poolType = QUERY_AUTO_QWORKER_POOL,
   };
 
   if (tSingleWorkerInit(&pMgmt->queryWorker, &queryCfg) != 0) {

source/dnode/mgmt/mgmt_snode/src/smHandle.c

@@ -96,6 +96,7 @@ SArray *smGetMsgHandles() {
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_HEARTBEAT_RSP, smPutNodeMsgToStreamQueue, 1) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_REQ_CHKPT_RSP, smPutNodeMsgToStreamQueue, 1) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_REPORT_RSP, smPutNodeMsgToStreamQueue, 1) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_CONSEN_RSP, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER;
 
   code = 0;
 _OVER:

source/dnode/mgmt/mgmt_vnode/inc/vmInt.h

@@ -26,20 +26,20 @@ extern "C" {
 #endif
 
 typedef struct SVnodeMgmt {
-  SDnodeData      *pData;
+  SDnodeData           *pData;
-  SMsgCb           msgCb;
+  SMsgCb                msgCb;
-  const char      *path;
+  const char           *path;
-  const char      *name;
+  const char           *name;
-  SQWorkerPool     queryPool;
+  SQueryAutoQWorkerPool queryPool;
-  SAutoQWorkerPool streamPool;
+  SAutoQWorkerPool      streamPool;
-  SWWorkerPool     fetchPool;
+  SWWorkerPool          fetchPool;
-  SSingleWorker    mgmtWorker;
+  SSingleWorker         mgmtWorker;
-  SHashObj        *hash;
+  SHashObj             *hash;
-  TdThreadRwlock   lock;
+  TdThreadRwlock        lock;
-  SVnodesStat      state;
+  SVnodesStat           state;
-  STfs            *pTfs;
+  STfs                 *pTfs;
-  TdThread         thread;
+  TdThread              thread;
-  bool             stop;
+  bool                  stop;
 } SVnodeMgmt;
 
 typedef struct {

source/dnode/mgmt/mgmt_vnode/src/vmHandle.c

@@ -972,6 +972,7 @@ SArray *vmGetMsgHandles() {
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_HEARTBEAT_RSP, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_REQ_CHKPT_RSP, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_REPORT_RSP, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_CONSEN_RSP, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
   if (dmSetMgmtHandle(pArray, TDMT_VND_GET_STREAM_PROGRESS, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER;
 
   if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_UPDATE_CHKPT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;

source/dnode/mgmt/mgmt_vnode/src/vmWorker.c

@@ -82,7 +82,7 @@ static void vmProcessQueryQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
   const STraceId *trace = &pMsg->info.traceId;
 
   dGTrace("vgId:%d, msg:%p get from vnode-query queue", pVnode->vgId, pMsg);
-  int32_t code = vnodeProcessQueryMsg(pVnode->pImpl, pMsg);
+  int32_t code = vnodeProcessQueryMsg(pVnode->pImpl, pMsg, pInfo);
   if (code != 0) {
     if (terrno != 0) code = terrno;
     dGError("vgId:%d, msg:%p failed to query since %s", pVnode->vgId, pMsg, tstrerror(code));
@@ -357,7 +357,7 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
   (void)tMultiWorkerInit(&pVnode->pSyncRdW, &sccfg);
   (void)tMultiWorkerInit(&pVnode->pApplyW, &acfg);
 
-  pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue);
+  pVnode->pQueryQ = tQueryAutoQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue);
   pVnode->pStreamQ = tAutoQWorkerAllocQueue(&pMgmt->streamPool, pVnode, (FItem)vmProcessStreamQueue);
   pVnode->pFetchQ = tWWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItems)vmProcessFetchQueue);
 
@@ -383,7 +383,7 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
 }
 
 void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
-  tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ);
+  tQueryAutoQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ);
   tAutoQWorkerFreeQueue(&pMgmt->streamPool, pVnode->pStreamQ);
   tWWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ);
   pVnode->pQueryQ = NULL;
@@ -393,11 +393,11 @@ void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
 }
 
 int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
-  SQWorkerPool *pQPool = &pMgmt->queryPool;
+  SQueryAutoQWorkerPool *pQPool = &pMgmt->queryPool;
   pQPool->name = "vnode-query";
   pQPool->min = tsNumOfVnodeQueryThreads;
   pQPool->max = tsNumOfVnodeQueryThreads;
-  if (tQWorkerInit(pQPool) != 0) return -1;
+  if (tQueryAutoQWorkerInit(pQPool) != 0) return -1;
 
   SAutoQWorkerPool *pStreamPool = &pMgmt->streamPool;
   pStreamPool->name = "vnode-stream";
@@ -419,7 +419,7 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
 }
 
 void vmStopWorker(SVnodeMgmt *pMgmt) {
-  tQWorkerCleanup(&pMgmt->queryPool);
+  tQueryAutoQWorkerCleanup(&pMgmt->queryPool);
   tAutoQWorkerCleanup(&pMgmt->streamPool);
   tWWorkerCleanup(&pMgmt->fetchPool);
   dDebug("vnode workers are closed");

source/dnode/mnode/impl/inc/mndInt.h

@@ -54,6 +54,7 @@ extern "C" {
 #define SYSTABLE_SCH_COL_NAME_LEN   ((TSDB_COL_NAME_LEN - 1) + VARSTR_HEADER_SIZE)
 
 typedef int32_t (*MndMsgFp)(SRpcMsg *pMsg);
+typedef int32_t (*MndMsgFpExt)(SRpcMsg *pMsg, SQueueInfo* pInfo);
 typedef int32_t (*MndInitFp)(SMnode *pMnode);
 typedef void (*MndCleanupFp)(SMnode *pMnode);
 typedef int32_t (*ShowRetrieveFp)(SRpcMsg *pMsg, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows);
@@ -137,11 +138,13 @@ typedef struct SMnode {
   SEncryptMgmt   encryptMgmt;
   SGrantInfo     grant;
   MndMsgFp       msgFp[TDMT_MAX];
+  MndMsgFpExt    msgFpExt[TDMT_MAX];
   SMsgCb         msgCb;
   int64_t        ipWhiteVer;
 } SMnode;
 
 void    mndSetMsgHandle(SMnode *pMnode, tmsg_t msgType, MndMsgFp fp);
+void    mndSetMsgHandleExt(SMnode *pMnode, tmsg_t msgType, MndMsgFpExt fp);
 int64_t mndGenerateUid(const char *name, int32_t len);
 
 void mndSetRestored(SMnode *pMnode, bool restored);

source/dnode/mnode/impl/inc/mndStream.h

@@ -34,6 +34,7 @@ extern "C" {
 #define MND_STREAM_TASK_RESET_NAME   "stream-task-reset"
 #define MND_STREAM_TASK_UPDATE_NAME  "stream-task-update"
 #define MND_STREAM_CHKPT_UPDATE_NAME "stream-chkpt-update"
+#define MND_STREAM_CHKPT_CONSEN_NAME "stream-chkpt-consen"
 
 typedef struct SStreamTransInfo {
   int64_t     startTime;
@@ -61,6 +62,7 @@ typedef struct SStreamExecInfo {
   TdThreadMutex    lock;
   SHashObj        *pTransferStateStreams;
   SHashObj        *pChkptStreams;
+  SHashObj        *pStreamConsensus;
 } SStreamExecInfo;
 
 extern SStreamExecInfo         execInfo;
@@ -81,7 +83,7 @@ typedef struct SOrphanTask {
 
 typedef struct {
   SMsgHead head;
-} SMStreamHbRspMsg, SMStreamReqCheckpointRsp, SMStreamUpdateChkptRsp;
+} SMStreamReqCheckpointRsp, SMStreamUpdateChkptRsp;
 
 typedef struct STaskChkptInfo {
   int32_t nodeId;
@@ -131,6 +133,8 @@ int32_t     mndCreateStreamResetStatusTrans(SMnode *pMnode, SStreamObj *pStream)
 int32_t     mndStreamSetUpdateChkptAction(SMnode *pMnode, STrans *pTrans, SStreamObj *pStream);
 int32_t     mndCreateStreamChkptInfoUpdateTrans(SMnode *pMnode, SStreamObj *pStream, SArray *pChkptInfoList);
 int32_t     mndScanCheckpointReportInfo(SRpcMsg *pReq);
+int32_t     mndSendConsensusCheckpointIdRsp(SArray* pList, int64_t checkpointId);
+
 void        removeTasksInBuf(SArray *pTaskIds, SStreamExecInfo *pExecInfo);
 
 SStreamTaskIter *createStreamTaskIter(SStreamObj *pStream);
@@ -142,6 +146,14 @@ void             mndInitStreamExecInfo(SMnode *pMnode, SStreamExecInfo *pExecInf
 int32_t          removeExpiredNodeEntryAndTaskInBuf(SArray *pNodeSnapshot);
 void             removeStreamTasksInBuf(SStreamObj *pStream, SStreamExecInfo *pExecNode);
 
+SCheckpointConsensusInfo *mndGetConsensusInfo(SHashObj *pHash, int64_t streamId);
+void mndAddConsensusTasks(SCheckpointConsensusInfo *pInfo, const SRestoreCheckpointInfo *pRestoreInfo, SRpcMsg *pMsg);
+int64_t mndGetConsensusCheckpointId(SCheckpointConsensusInfo *pInfo, SStreamObj *pStream);
+bool    mndAllTaskSendCheckpointId(SCheckpointConsensusInfo *pInfo, int32_t numOfTasks, int32_t* pTotal);
+void    mndClearConsensusRspEntry(SCheckpointConsensusInfo *pInfo);
+int32_t doSendConsensusCheckpointRsp(SRestoreCheckpointInfo *pInfo, SRpcMsg *pMsg, int64_t checkpointId);
+int64_t mndClearConsensusCheckpointId(SHashObj* pHash, int64_t streamId);
+
 #ifdef __cplusplus
 }
 #endif

source/dnode/mnode/impl/inc/mndTrans.h

@@ -84,6 +84,7 @@ void    mndTransSetParallel(STrans *pTrans);
 void    mndTransSetChangeless(STrans *pTrans);
 void    mndTransSetOper(STrans *pTrans, EOperType oper);
 int32_t mndTransCheckConflict(SMnode *pMnode, STrans *pTrans);
+int32_t mndTransCheckConflictWithCompact(SMnode *pMnode, STrans *pTrans);
 #ifndef BUILD_NO_CALL
 static int32_t mndTrancCheckConflict(SMnode *pMnode, STrans *pTrans) {
     return mndTransCheckConflict(pMnode, pTrans);

source/dnode/mnode/impl/src/mndMain.c

@@ -844,21 +844,29 @@ _OVER:
   return -1;
 }
 
-int32_t mndProcessRpcMsg(SRpcMsg *pMsg) {
+int32_t mndProcessRpcMsg(SRpcMsg *pMsg, SQueueInfo* pQueueInfo) {
   SMnode         *pMnode = pMsg->info.node;
   const STraceId *trace = &pMsg->info.traceId;
+  int32_t         code = TSDB_CODE_SUCCESS;
 
-  MndMsgFp fp = pMnode->msgFp[TMSG_INDEX(pMsg->msgType)];
+  MndMsgFp    fp = pMnode->msgFp[TMSG_INDEX(pMsg->msgType)];
+  MndMsgFpExt fpExt = NULL;
   if (fp == NULL) {
-    mGError("msg:%p, failed to get msg handle, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
+    fpExt = pMnode->msgFpExt[TMSG_INDEX(pMsg->msgType)];
-    terrno = TSDB_CODE_MSG_NOT_PROCESSED;
+    if (fpExt == NULL) {
-    return -1;
+      mGError("msg:%p, failed to get msg handle, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
+      terrno = TSDB_CODE_MSG_NOT_PROCESSED;
+      return -1;
+    }
   }
 
   if (mndCheckMnodeState(pMsg) != 0) return -1;
 
   mGTrace("msg:%p, start to process in mnode, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
-  int32_t code = (*fp)(pMsg);
+  if (fp)
+    code = (*fp)(pMsg);
+  else
+    code = (*fpExt)(pMsg, pQueueInfo);
   mndReleaseRpc(pMnode);
 
   if (code == TSDB_CODE_ACTION_IN_PROGRESS) {
@@ -883,6 +891,13 @@ void mndSetMsgHandle(SMnode *pMnode, tmsg_t msgType, MndMsgFp fp) {
   }
 }
 
+void mndSetMsgHandleExt(SMnode *pMnode, tmsg_t msgType, MndMsgFpExt fp) {
+  tmsg_t type = TMSG_INDEX(msgType);
+  if (type < TDMT_MAX) {
+    pMnode->msgFpExt[type] = fp;
+  }
+}
+
 // Note: uid 0 is reserved
 int64_t mndGenerateUid(const char *name, int32_t len) {
   int32_t hashval = MurmurHash3_32(name, len);

source/dnode/mnode/impl/src/mndQuery.c

@@ -30,11 +30,11 @@ void mndPostProcessQueryMsg(SRpcMsg *pMsg) {
   qWorkerAbortPreprocessQueryMsg(pMnode->pQuery, pMsg);
 }
 
-int32_t mndProcessQueryMsg(SRpcMsg *pMsg) {
+int32_t mndProcessQueryMsg(SRpcMsg *pMsg, SQueueInfo* pInfo) {
   int32_t     code = -1;
   SMnode     *pMnode = pMsg->info.node;
 
-  SReadHandle handle = {.mnd = pMnode, .pMsgCb = &pMnode->msgCb};
+  SReadHandle handle = {.mnd = pMnode, .pMsgCb = &pMnode->msgCb, .pWorkerCb = pInfo->workerCb};
 
   mTrace("msg:%p, in query queue is processing", pMsg);
   switch (pMsg->msgType) {
@@ -173,14 +173,14 @@ int32_t mndInitQuery(SMnode *pMnode) {
     return -1;
   }
 
-  mndSetMsgHandle(pMnode, TDMT_SCH_QUERY, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_QUERY, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_MERGE_QUERY, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_MERGE_QUERY, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_QUERY_CONTINUE, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_QUERY_CONTINUE, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_FETCH, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_FETCH, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_MERGE_FETCH, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_MERGE_FETCH, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_TASK_NOTIFY, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_TASK_NOTIFY, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_DROP_TASK, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_DROP_TASK, mndProcessQueryMsg);
-  mndSetMsgHandle(pMnode, TDMT_SCH_QUERY_HEARTBEAT, mndProcessQueryMsg);
+  mndSetMsgHandleExt(pMnode, TDMT_SCH_QUERY_HEARTBEAT, mndProcessQueryMsg);
   mndSetMsgHandle(pMnode, TDMT_MND_BATCH_META, mndProcessBatchMetaMsg);
 
   return 0;

source/dnode/mnode/impl/src/mndStream.c

@@ -59,6 +59,7 @@ static int32_t mndProcessNodeCheckReq(SRpcMsg *pMsg);
 static int32_t extractNodeListFromStream(SMnode *pMnode, SArray *pNodeList);
 static int32_t mndProcessStreamReqCheckpoint(SRpcMsg *pReq);
 static int32_t mndProcessCheckpointReport(SRpcMsg *pReq);
+static int32_t mndProcessConsensusCheckpointId(SRpcMsg *pReq);
 
 static SVgroupChangeInfo mndFindChangedNodeInfo(SMnode *pMnode, const SArray *pPrevNodeList, const SArray *pNodeList);
 
@@ -117,6 +118,7 @@ int32_t mndInitStream(SMnode *pMnode) {
   mndSetMsgHandle(pMnode, TDMT_MND_STREAM_BEGIN_CHECKPOINT, mndProcessStreamCheckpoint);
   mndSetMsgHandle(pMnode, TDMT_MND_STREAM_REQ_CHKPT, mndProcessStreamReqCheckpoint);
   mndSetMsgHandle(pMnode, TDMT_MND_STREAM_CHKPT_REPORT, mndProcessCheckpointReport);
+  mndSetMsgHandle(pMnode, TDMT_MND_STREAM_CHKPT_CONSEN, mndProcessConsensusCheckpointId);
   mndSetMsgHandle(pMnode, TDMT_MND_STREAM_UPDATE_CHKPT_EVT, mndScanCheckpointReportInfo);
   mndSetMsgHandle(pMnode, TDMT_STREAM_TASK_REPORT_CHECKPOINT, mndTransProcessRsp);
   mndSetMsgHandle(pMnode, TDMT_MND_STREAM_HEARTBEAT, mndProcessStreamHb);
@@ -150,6 +152,7 @@ void mndCleanupStream(SMnode *pMnode) {
   taosHashCleanup(execInfo.transMgmt.pDBTrans);
   taosHashCleanup(execInfo.pTransferStateStreams);
   taosHashCleanup(execInfo.pChkptStreams);
+  taosHashCleanup(execInfo.pStreamConsensus);
   taosThreadMutexDestroy(&execInfo.lock);
   mDebug("mnd stream exec info cleanup");
 }
@@ -1233,6 +1236,9 @@ static int32_t mndProcessStreamCheckpoint(SRpcMsg *pReq) {
       code = mndProcessStreamCheckpointTrans(pMnode, p, checkpointId, 1, true);
       sdbRelease(pSdb, p);
 
+      // clear the consensus checkpoint info
+      mndClearConsensusCheckpointId(execInfo.pStreamConsensus, p->uid);
+
       if (code != -1) {
         started += 1;
 
@@ -2513,7 +2519,7 @@ int32_t mndProcessStreamReqCheckpoint(SRpcMsg *pReq) {
   return 0;
 }
 
-static void doAddTaskInfo(SArray *pList, SCheckpointReport *pReport) {
+static void doAddReportStreamTask(SArray* pList, const SCheckpointReport* pReport) {
   bool existed = false;
   for (int32_t i = 0; i < taosArrayGetSize(pList); ++i) {
     STaskChkptInfo *p = taosArrayGet(pList, i);
@@ -2584,12 +2590,12 @@ int32_t mndProcessCheckpointReport(SRpcMsg *pReq) {
   SArray **pReqTaskList = (SArray **)taosHashGet(execInfo.pChkptStreams, &req.streamId, sizeof(req.streamId));
   if (pReqTaskList == NULL) {
     SArray *pList = taosArrayInit(4, sizeof(STaskChkptInfo));
-    doAddTaskInfo(pList, &req);
+    doAddReportStreamTask(pList, &req);
     taosHashPut(execInfo.pChkptStreams, &req.streamId, sizeof(req.streamId), &pList, POINTER_BYTES);
 
     pReqTaskList = (SArray **)taosHashGet(execInfo.pChkptStreams, &req.streamId, sizeof(req.streamId));
   } else {
-    doAddTaskInfo(*pReqTaskList, &req);
+    doAddReportStreamTask(*pReqTaskList, &req);
   }
 
   int32_t total = taosArrayGetSize(*pReqTaskList);
@@ -2597,25 +2603,6 @@ int32_t mndProcessCheckpointReport(SRpcMsg *pReq) {
     mInfo("stream:0x%" PRIx64 " %s all %d tasks send checkpoint-report, checkpoint meta-info for checkpointId:%" PRId64
           " will be issued soon",
           req.streamId, pStream->name, total, req.checkpointId);
-
-    //    if (pStream != NULL) {
-    //      bool conflict = mndStreamTransConflictCheck(pMnode, pStream->uid, MND_STREAM_CHKPT_UPDATE_NAME, false);
-    //      if (conflict) {
-    //        mDebug("stream:0x%"PRIx64" active checkpoint trans not finished yet, wait", req.streamId);
-    //      } else {
-    //        int32_t code = mndCreateStreamChkptInfoUpdateTrans(pMnode, pStream, *pReqTaskList);
-    //        if (code == TSDB_CODE_SUCCESS) { // remove this entry
-    //          taosHashRemove(execInfo.pChkptStreams, &req.streamId, sizeof(req.streamId));
-    //
-    //          int32_t numOfStreams = taosHashGetSize(execInfo.pChkptStreams);
-    //          mDebug("stream:0x%" PRIx64 " removed, remain streams:%d in checkpoint procedure", req.streamId,
-    //          numOfStreams);
-    //        } else {
-    //          mDebug("stream:0x%" PRIx64 " not launch chkpt update trans, due to checkpoint not finished yet",
-    //                 req.streamId);
-    //        }
-    //      }
-    //    }
   }
 
   if (pStream != NULL) {
@@ -2637,6 +2624,101 @@ int32_t mndProcessCheckpointReport(SRpcMsg *pReq) {
   return 0;
 }
 
+static int32_t mndProcessConsensusCheckpointId(SRpcMsg *pReq) {
+  SMnode  *pMnode = pReq->info.node;
+  SDecoder decoder = {0};
+
+  SRestoreCheckpointInfo req = {0};
+  tDecoderInit(&decoder, pReq->pCont, pReq->contLen);
+
+  if (tDecodeRestoreCheckpointInfo(&decoder, &req)) {
+    tDecoderClear(&decoder);
+    terrno = TSDB_CODE_INVALID_MSG;
+    mError("invalid task consensus-checkpoint msg received");
+    return -1;
+  }
+  tDecoderClear(&decoder);
+
+  mDebug("receive stream task consensus-checkpoint msg, vgId:%d, s-task:0x%" PRIx64 "-0x%x, checkpointId:%" PRId64,
+         req.nodeId, req.streamId, req.taskId, req.checkpointId);
+
+  // register to the stream task done map, if all tasks has sent this kinds of message, start the checkpoint trans.
+  taosThreadMutexLock(&execInfo.lock);
+
+  SStreamObj *pStream = mndGetStreamObj(pMnode, req.streamId);
+  if (pStream == NULL) {
+    mWarn("failed to find the stream:0x%" PRIx64 ", not handle checkpoint-report, try to acquire in buf", req.streamId);
+
+    // not in meta-store yet, try to acquire the task in exec buffer
+    // the checkpoint req arrives too soon before the completion of the create stream trans.
+    STaskId id = {.streamId = req.streamId, .taskId = req.taskId};
+    void   *p = taosHashGet(execInfo.pTaskMap, &id, sizeof(id));
+    if (p == NULL) {
+      mError("failed to find the stream:0x%" PRIx64 " in buf, not handle the checkpoint-report", req.streamId);
+      terrno = TSDB_CODE_MND_STREAM_NOT_EXIST;
+      taosThreadMutexUnlock(&execInfo.lock);
+      return -1;
+    } else {
+      mDebug("s-task:0x%" PRIx64 "-0x%x in buf not in mnode/meta, create stream trans may not complete yet",
+             req.streamId, req.taskId);
+    }
+  }
+
+  int32_t numOfTasks = (pStream == NULL) ? 0 : mndGetNumOfStreamTasks(pStream);
+
+  SCheckpointConsensusInfo *pInfo = mndGetConsensusInfo(execInfo.pStreamConsensus, req.streamId);
+  int64_t                   ckId = mndGetConsensusCheckpointId(pInfo, pStream);
+  if (ckId != -1) {  // consensus checkpoint id already exist
+    SRpcMsg rsp = {0};
+    rsp.code = 0;
+    rsp.info = pReq->info;
+    rsp.contLen = sizeof(SRestoreCheckpointInfoRsp) + sizeof(SMsgHead);
+    rsp.pCont = rpcMallocCont(rsp.contLen);
+
+    SMsgHead *pHead = rsp.pCont;
+    pHead->vgId = htonl(req.nodeId);
+
+    mDebug("stream:0x%" PRIx64 " consensus checkpointId:%" PRId64 " exists, return directly", req.streamId, ckId);
+    doSendConsensusCheckpointRsp(&req, &rsp, ckId);
+
+    taosThreadMutexUnlock(&execInfo.lock);
+    pReq->info.handle = NULL;  // disable auto rsp
+
+    return TSDB_CODE_SUCCESS;
+  }
+
+  mndAddConsensusTasks(pInfo, &req, pReq);
+
+  int32_t total = 0;
+  if (mndAllTaskSendCheckpointId(pInfo, numOfTasks, &total)) {  // all tasks has send the reqs
+    // start transaction to set the checkpoint id
+    int64_t checkpointId = mndGetConsensusCheckpointId(pInfo, pStream);
+    mInfo("stream:0x%" PRIx64 " %s all %d tasks send latest checkpointId, the consensus-checkpointId is:%" PRId64
+          " will be issued soon",
+          req.streamId, pStream->name, numOfTasks, checkpointId);
+
+    // start the checkpoint consensus trans
+    int32_t code = mndSendConsensusCheckpointIdRsp(pInfo->pTaskList, checkpointId);
+    if (code == TSDB_CODE_SUCCESS) {
+      mndClearConsensusRspEntry(pInfo);
+      mDebug("clear all waiting for rsp entry for stream:0x%" PRIx64, req.streamId);
+    } else {
+      mDebug("stream:0x%" PRIx64 " not start send consensus-checkpointId msg, due to not all task ready", req.streamId);
+    }
+  } else {
+    mDebug("stream:0x%" PRIx64 " %d/%d tasks send consensus-checkpointId info", req.streamId, total, numOfTasks);
+  }
+
+  if (pStream != NULL) {
+    mndReleaseStream(pMnode, pStream);
+  }
+
+  taosThreadMutexUnlock(&execInfo.lock);
+  pReq->info.handle = NULL;  // disable auto rsp
+
+  return 0;
+}
+
 static int32_t mndProcessCreateStreamReqFromMNode(SRpcMsg *pReq) {
   int32_t code = mndProcessCreateStreamReq(pReq);
   if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) {

source/dnode/mnode/impl/src/mndStreamHb.c

@@ -22,9 +22,18 @@ typedef struct SFailedCheckpointInfo {
   int32_t transId;
 } SFailedCheckpointInfo;
 
-static void mndStreamStartUpdateCheckpointInfo(SMnode *pMnode);
+static void    mndStreamStartUpdateCheckpointInfo(SMnode *pMnode);
+static void    updateStageInfo(STaskStatusEntry *pTaskEntry, int64_t stage);
+static void    addIntoCheckpointList(SArray *pList, const SFailedCheckpointInfo *pInfo);
+static int32_t mndResetStatusFromCheckpoint(SMnode *pMnode, int64_t streamId, int32_t transId);
+static int32_t setNodeEpsetExpiredFlag(const SArray *pNodeList);
+static int32_t mndDropOrphanTasks(SMnode *pMnode, SArray *pList);
+static int32_t suspendAllStreams(SMnode *pMnode, SRpcHandleInfo *info);
+static bool    validateHbMsg(const SArray *pNodeList, int32_t vgId);
+static void    cleanupAfterProcessHbMsg(SStreamHbMsg *pReq, SArray *pFailedChkptList, SArray *pOrphanTasks);
+static void    doSendHbMsgRsp(int32_t code, SRpcHandleInfo *pRpcInfo, int32_t vgId, int32_t msgId);
 
-static void updateStageInfo(STaskStatusEntry *pTaskEntry, int64_t stage) {
+void updateStageInfo(STaskStatusEntry *pTaskEntry, int64_t stage) {
   int32_t numOfNodes = taosArrayGetSize(execInfo.pNodeList);
   for (int32_t j = 0; j < numOfNodes; ++j) {
     SNodeEntry *pNodeEntry = taosArrayGet(execInfo.pNodeList, j);
@@ -39,7 +48,7 @@ static void updateStageInfo(STaskStatusEntry *pTaskEntry, int64_t stage) {
   }
 }
 
-static void addIntoCheckpointList(SArray *pList, const SFailedCheckpointInfo *pInfo) {
+void addIntoCheckpointList(SArray *pList, const SFailedCheckpointInfo *pInfo) {
   int32_t num = taosArrayGetSize(pList);
   for (int32_t i = 0; i < num; ++i) {
     SFailedCheckpointInfo *p = taosArrayGet(pList, i);
@@ -86,7 +95,7 @@ int32_t mndCreateStreamResetStatusTrans(SMnode *pMnode, SStreamObj *pStream) {
   return TSDB_CODE_ACTION_IN_PROGRESS;
 }
 
-static int32_t mndResetStatusFromCheckpoint(SMnode *pMnode, int64_t streamId, int32_t transId) {
+int32_t mndResetStatusFromCheckpoint(SMnode *pMnode, int64_t streamId, int32_t transId) {
   int32_t code = TSDB_CODE_SUCCESS;
   mndKillTransImpl(pMnode, transId, "");
 
@@ -110,7 +119,7 @@ static int32_t mndResetStatusFromCheckpoint(SMnode *pMnode, int64_t streamId, in
   return code;
 }
 
-static int32_t setNodeEpsetExpiredFlag(const SArray *pNodeList) {
+int32_t setNodeEpsetExpiredFlag(const SArray *pNodeList) {
   int32_t num = taosArrayGetSize(pNodeList);
   mInfo("set node expired for %d nodes", num);
 
@@ -133,15 +142,14 @@ static int32_t setNodeEpsetExpiredFlag(const SArray *pNodeList) {
     }
 
     if (!setFlag) {
-      mError("failed to set nodeUpdate flag, nodeId:%d not exists in nodelist, update it", *pVgId);
+      mError("failed to set nodeUpdate flag, nodeId:%d not exists in nodelist", *pVgId);
-      ASSERT(0);
       return TSDB_CODE_FAILED;
     }
   }
   return TSDB_CODE_SUCCESS;
 }
 
-static int32_t mndDropOrphanTasks(SMnode *pMnode, SArray *pList) {
+int32_t mndDropOrphanTasks(SMnode *pMnode, SArray *pList) {
   SOrphanTask *pTask = taosArrayGet(pList, 0);
 
   // check if it is conflict with other trans in both sourceDb and targetDb.
@@ -238,7 +246,7 @@ int32_t mndProcessStreamHb(SRpcMsg *pReq) {
   }
   tDecoderClear(&decoder);
 
-  mTrace("receive stream-meta hb from vgId:%d, active numOfTasks:%d", req.vgId, req.numOfTasks);
+  mTrace("receive stream-meta hb from vgId:%d, active numOfTasks:%d, msgId:%d", req.vgId, req.numOfTasks, req.msgId);
 
   pFailedChkpt = taosArrayInit(4, sizeof(SFailedCheckpointInfo));
   pOrphanTasks = taosArrayInit(4, sizeof(SOrphanTask));
@@ -246,6 +254,16 @@ int32_t mndProcessStreamHb(SRpcMsg *pReq) {
   taosThreadMutexLock(&execInfo.lock);
 
   mndInitStreamExecInfo(pMnode, &execInfo);
+  if (!validateHbMsg(execInfo.pNodeList, req.vgId)) {
+    mError("invalid hbMsg from vgId:%d, discarded", req.vgId);
+
+    terrno = TSDB_CODE_INVALID_MSG;
+    doSendHbMsgRsp(terrno, &pReq->info, req.vgId, req.msgId);
+
+    taosThreadMutexUnlock(&execInfo.lock);
+    cleanupAfterProcessHbMsg(&req, pFailedChkpt, pOrphanTasks);
+    return -1;
+  }
 
   int32_t numOfUpdated = taosArrayGetSize(req.pUpdateNodes);
   if (numOfUpdated > 0) {
@@ -335,21 +353,11 @@ int32_t mndProcessStreamHb(SRpcMsg *pReq) {
   }
 
   taosThreadMutexUnlock(&execInfo.lock);
-  tCleanupStreamHbMsg(&req);
 
-  taosArrayDestroy(pFailedChkpt);
+  terrno = TSDB_CODE_SUCCESS;
-  taosArrayDestroy(pOrphanTasks);
+  doSendHbMsgRsp(terrno, &pReq->info, req.vgId, req.msgId);
-
-  {
-    SRpcMsg rsp = {.code = 0, .info = pReq->info, .contLen = sizeof(SMStreamHbRspMsg)};
-    rsp.pCont = rpcMallocCont(rsp.contLen);
-    SMsgHead *pHead = rsp.pCont;
-    pHead->vgId = htonl(req.vgId);
-
-    tmsgSendRsp(&rsp);
-    pReq->info.handle = NULL;  // disable auto rsp
-  }
 
+  cleanupAfterProcessHbMsg(&req, pFailedChkpt, pOrphanTasks);
   return TSDB_CODE_SUCCESS;
 }
 
@@ -361,3 +369,32 @@ void mndStreamStartUpdateCheckpointInfo(SMnode *pMnode) {  // here reuse the doC
     tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
   }
 }
+
+bool validateHbMsg(const SArray *pNodeList, int32_t vgId) {
+  for (int32_t i = 0; i < taosArrayGetSize(pNodeList); ++i) {
+    SNodeEntry *pEntry = taosArrayGet(pNodeList, i);
+    if (pEntry->nodeId == vgId) {
+      return true;
+    }
+  }
+
+  return false;
+}
+
+void cleanupAfterProcessHbMsg(SStreamHbMsg *pReq, SArray *pFailedChkptList, SArray *pOrphanTasks) {
+  tCleanupStreamHbMsg(pReq);
+  taosArrayDestroy(pFailedChkptList);
+  taosArrayDestroy(pOrphanTasks);
+}
+
+void doSendHbMsgRsp(int32_t code, SRpcHandleInfo *pRpcInfo, int32_t vgId, int32_t msgId) {
+  SRpcMsg rsp = {.code = code, .info = *pRpcInfo, .contLen = sizeof(SMStreamHbRspMsg)};
+  rsp.pCont = rpcMallocCont(rsp.contLen);
+
+  SMStreamHbRspMsg *pMsg = rsp.pCont;
+  pMsg->head.vgId = htonl(vgId);
+  pMsg->msgId = msgId;
+
+  tmsgSendRsp(&rsp);
+  pRpcInfo->handle = NULL;  // disable auto rsp
+}

source/dnode/mnode/impl/src/mndStreamUtil.c

@@ -575,10 +575,12 @@ void mndInitExecInfo() {
   execInfo.transMgmt.pDBTrans = taosHashInit(32, fn, true, HASH_NO_LOCK);
   execInfo.pTransferStateStreams = taosHashInit(32, fn, true, HASH_NO_LOCK);
   execInfo.pChkptStreams = taosHashInit(32, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_NO_LOCK);
+  execInfo.pStreamConsensus = taosHashInit(32, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_NO_LOCK);
   execInfo.pNodeList = taosArrayInit(4, sizeof(SNodeEntry));
 
   taosHashSetFreeFp(execInfo.pTransferStateStreams, freeTaskList);
   taosHashSetFreeFp(execInfo.pChkptStreams, freeTaskList);
+  taosHashSetFreeFp(execInfo.pStreamConsensus, freeTaskList);
 }
 
 void removeExpiredNodeInfo(const SArray *pNodeSnapshot) {
@@ -818,3 +820,149 @@ int32_t mndScanCheckpointReportInfo(SRpcMsg *pReq) {
   taosArrayDestroy(pDropped);
   return TSDB_CODE_SUCCESS;
 }
+
+int32_t doSendConsensusCheckpointRsp(SRestoreCheckpointInfo* pInfo, SRpcMsg* pMsg, int64_t checkpointId) {
+  int32_t code = 0;
+  int32_t blen;
+
+  SRestoreCheckpointInfoRsp req = {
+      .streamId = pInfo->streamId, .taskId = pInfo->taskId, .checkpointId = checkpointId, .startTs = pInfo->startTs};
+
+  tEncodeSize(tEncodeRestoreCheckpointInfoRsp, &req, blen, code);
+  if (code < 0) {
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return -1;
+  }
+
+  int32_t tlen = sizeof(SMsgHead) + blen;
+  void    *abuf = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
+  SEncoder encoder;
+  tEncoderInit(&encoder, abuf, tlen);
+  tEncodeRestoreCheckpointInfoRsp(&encoder, &req);
+
+  SMsgHead *pMsgHead = (SMsgHead *)pMsg->pCont;
+  pMsgHead->contLen = htonl(tlen);
+  pMsgHead->vgId = htonl(pInfo->nodeId);
+  tEncoderClear(&encoder);
+
+  tmsgSendRsp(pMsg);
+  return code;
+}
+
+int32_t mndSendConsensusCheckpointIdRsp(SArray* pInfoList, int64_t checkpointId) {
+  for(int32_t i = 0; i < taosArrayGetSize(pInfoList); ++i) {
+    SCheckpointConsensusEntry* pInfo = taosArrayGet(pInfoList, i);
+    doSendConsensusCheckpointRsp(&pInfo->req, &pInfo->rsp, checkpointId);
+  }
+  return 0;
+}
+
+SCheckpointConsensusInfo* mndGetConsensusInfo(SHashObj* pHash, int64_t streamId) {
+  void* pInfo = taosHashGet(pHash, &streamId, sizeof(streamId));
+  if (pInfo != NULL) {
+    return (SCheckpointConsensusInfo*)pInfo;
+  }
+
+  SCheckpointConsensusInfo p = {
+      .genTs = -1, .checkpointId = -1, .pTaskList = taosArrayInit(4, sizeof(SCheckpointConsensusEntry))};
+  taosHashPut(pHash, &streamId, sizeof(streamId), &p, sizeof(p));
+
+  void* pChkptInfo = (SCheckpointConsensusInfo*)taosHashGet(pHash, &streamId, sizeof(streamId));
+  return pChkptInfo;
+}
+
+// no matter existed or not, add the request into info list anyway, since we need to send rsp mannually
+// discard the msg may lead to the lost of connections.
+void mndAddConsensusTasks(SCheckpointConsensusInfo *pInfo, const SRestoreCheckpointInfo *pRestoreInfo, SRpcMsg *pMsg) {
+  SCheckpointConsensusEntry info = {0};
+  memcpy(&info.req, pRestoreInfo, sizeof(info.req));
+
+  info.rsp.code = 0;
+  info.rsp.info = pMsg->info;
+  info.rsp.contLen = sizeof(SRestoreCheckpointInfoRsp) + sizeof(SMsgHead);
+  info.rsp.pCont = rpcMallocCont(info.rsp.contLen);
+
+  SMsgHead *pHead = info.rsp.pCont;
+  pHead->vgId = htonl(pRestoreInfo->nodeId);
+
+  taosArrayPush(pInfo->pTaskList, &info);
+}
+
+static int32_t entryComparFn(const void* p1, const void* p2) {
+  const SCheckpointConsensusEntry* pe1 = p1;
+  const SCheckpointConsensusEntry* pe2 = p2;
+
+  if (pe1->req.taskId == pe2->req.taskId) {
+    return 0;
+  }
+
+  return pe1->req.taskId < pe2->req.taskId? -1:1;
+}
+
+bool mndAllTaskSendCheckpointId(SCheckpointConsensusInfo* pInfo, int32_t numOfTasks, int32_t* pTotal) {
+  int32_t numOfExisted = taosArrayGetSize(pInfo->pTaskList);
+  if (numOfExisted < numOfTasks) {
+    if (pTotal != NULL) {
+      *pTotal = numOfExisted;
+    }
+    return false;
+  }
+
+  taosArraySort(pInfo->pTaskList, entryComparFn);
+
+  int32_t num = 1;
+  int32_t taskId = ((SCheckpointConsensusEntry*)taosArrayGet(pInfo->pTaskList, 0))->req.taskId;
+  for(int32_t i = 1; i < taosArrayGetSize(pInfo->pTaskList); ++i) {
+    SCheckpointConsensusEntry* pe = taosArrayGet(pInfo->pTaskList, i);
+    if (pe->req.taskId != taskId) {
+      num += 1;
+      taskId = pe->req.taskId;
+    }
+  }
+
+  if (pTotal != NULL) {
+    *pTotal = num;
+  }
+
+  ASSERT(num <= numOfTasks);
+  return num == numOfTasks;
+}
+
+int64_t mndGetConsensusCheckpointId(SCheckpointConsensusInfo* pInfo, SStreamObj* pStream) {
+  if (pInfo->genTs > 0) {
+    ASSERT(pInfo->checkpointId > 0);
+    return pInfo->checkpointId;
+  }
+
+  int32_t numOfTasks = mndGetNumOfStreamTasks(pStream);
+  if (!mndAllTaskSendCheckpointId(pInfo, numOfTasks, NULL)) {
+    return -1;
+  }
+
+  int64_t checkpointId = INT64_MAX;
+
+  for (int32_t i = 0; i < taosArrayGetSize(pInfo->pTaskList); ++i) {
+    SCheckpointConsensusEntry *pEntry = taosArrayGet(pInfo->pTaskList, i);
+    if (pEntry->req.checkpointId < checkpointId) {
+      checkpointId = pEntry->req.checkpointId;
+      mTrace("stream:0x%" PRIx64 " %s task:0x%x vgId:%d latest checkpointId:%" PRId64, pStream->uid, pStream->name,
+             pEntry->req.taskId, pEntry->req.nodeId, pEntry->req.checkpointId);
+    }
+  }
+
+  pInfo->checkpointId = checkpointId;
+  pInfo->genTs = taosGetTimestampMs();
+  return checkpointId;
+}
+
+void mndClearConsensusRspEntry(SCheckpointConsensusInfo* pInfo) {
+  pInfo->pTaskList = taosArrayDestroy(pInfo->pTaskList);
+}
+
+int64_t mndClearConsensusCheckpointId(SHashObj* pHash, int64_t streamId) {
+  taosHashRemove(pHash, &streamId, sizeof(streamId));
+  int32_t numOfStreams = taosHashGetSize(pHash);
+  mDebug("drop stream:0x%" PRIx64 " in consensus-checkpointId list after new checkpoint generated, remain:%d", streamId,
+         numOfStreams);
+  return TSDB_CODE_SUCCESS;
+}

source/dnode/mnode/impl/src/mndTrans.c

@@ -900,6 +900,41 @@ int32_t mndTransCheckConflict(SMnode *pMnode, STrans *pTrans) {
   return 0;
 }
 
+int32_t mndTransCheckConflictWithCompact(SMnode *pMnode, STrans *pTrans) {
+  void        *pIter = NULL;
+  bool         conflict = false;
+  SCompactObj *pCompact = NULL;
+
+  while (1) {
+    bool thisConflict = false;
+    pIter = sdbFetch(pMnode->pSdb, SDB_COMPACT, pIter, (void **)&pCompact);
+    if (pIter == NULL) break;
+
+    if (pTrans->conflict == TRN_CONFLICT_GLOBAL || pTrans->conflict == TRN_CONFLICT_DB ||
+        pTrans->conflict == TRN_CONFLICT_DB_INSIDE) {
+      if (strcasecmp(pTrans->dbname, pCompact->dbname) == 0) thisConflict = true;
+    }
+
+    if (thisConflict) {
+      mError("trans:%d, db:%s stb:%s type:%d, can't execute since conflict with compact:%d db:%s", pTrans->id,
+             pTrans->dbname, pTrans->stbname, pTrans->conflict, pCompact->compactId, pCompact->dbname);
+      conflict = true;
+    } else {
+      mInfo("trans:%d, db:%s stb:%s type:%d, not conflict with compact:%d db:%s", pTrans->id, pTrans->dbname,
+            pTrans->stbname, pTrans->conflict, pCompact->compactId, pCompact->dbname);
+    }
+    sdbRelease(pMnode->pSdb, pCompact);
+  }
+
+  if (conflict) {
+    terrno = TSDB_CODE_MND_TRANS_CONFLICT_COMPACT;
+    mError("trans:%d, failed to prepare since %s", pTrans->id, terrstr());
+    return terrno;
+  }
+
+  return 0;
+}
+
 static bool mndTransActionsOfSameType(SArray *pActions) {
   int32_t size = taosArrayGetSize(pActions);
   ETrnAct lastActType = TRANS_ACTION_NULL;

source/dnode/mnode/impl/src/mndVgroup.c

@@ -1952,6 +1952,10 @@ static int32_t mndRedistributeVgroup(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb,
 
   pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_GLOBAL, pReq, "red-vgroup");
   if (pTrans == NULL) goto _OVER;
+
+  mndTransSetDbName(pTrans, pVgroup->dbName, NULL);
+  if (mndTransCheckConflictWithCompact(pMnode, pTrans) != 0) goto _OVER;
+
   mndTransSetSerial(pTrans);
   mInfo("trans:%d, used to redistribute vgroup, vgId:%d", pTrans->id, pVgroup->vgId);
 

source/dnode/qnode/inc/qndInt.h

@@ -29,7 +29,7 @@
 extern "C" {
 #endif
 
-typedef struct SQueueWorker SQHandle;
+typedef struct SQWorker SQHandle;
 
 typedef struct SQnode {
   int32_t   qndId;

source/dnode/qnode/src/qnode.c

@@ -13,6 +13,7 @@
  * along with this program. If not, see <http://www.gnu.org/licenses/>.
  */
 
+#include "tqueue.h"
 #include "executor.h"
 #include "qndInt.h"
 #include "query.h"
@@ -24,6 +25,7 @@ SQnode *qndOpen(const SQnodeOpt *pOption) {
     qError("calloc SQnode failed");
     return NULL;
   }
+  pQnode->qndId = QNODE_HANDLE;
 
   if (qWorkerInit(NODE_TYPE_QNODE, pQnode->qndId, (void **)&pQnode->pQuery, &pOption->msgCb)) {
     taosMemoryFreeClear(pQnode);
@@ -72,9 +74,10 @@ int32_t qndPreprocessQueryMsg(SQnode *pQnode, SRpcMsg *pMsg) {
   return qWorkerPreprocessQueryMsg(pQnode->pQuery, pMsg, false);
 }
 
-int32_t qndProcessQueryMsg(SQnode *pQnode, int64_t ts, SRpcMsg *pMsg) {
+int32_t qndProcessQueryMsg(SQnode *pQnode, SQueueInfo* pInfo, SRpcMsg *pMsg) {
   int32_t     code = -1;
-  SReadHandle handle = {.pMsgCb = &pQnode->msgCb};
+  int64_t ts = pInfo->timestamp;
+  SReadHandle handle = {.pMsgCb = &pQnode->msgCb, .pWorkerCb = pInfo->workerCb};
   qTrace("message in qnode queue is processing");
 
   switch (pMsg->msgType) {

source/dnode/snode/src/snode.c

@@ -25,7 +25,7 @@
 #define sndDebug(...) do {  if (sndDebugFlag & DEBUG_DEBUG) { taosPrintLog("SND ", DEBUG_DEBUG, sndDebugFlag, __VA_ARGS__);}} while (0)
 // clang-format on
 
-int32_t sndExpandTask(SSnode *pSnode, SStreamTask *pTask, int64_t nextProcessVer) {
+int32_t sndBuildStreamTask(SSnode *pSnode, SStreamTask *pTask, int64_t nextProcessVer) {
   ASSERT(pTask->info.taskLevel == TASK_LEVEL__AGG && taosArrayGetSize(pTask->upstreamInfo.pList) != 0);
   int32_t code = streamTaskInit(pTask, pSnode->pMeta, &pSnode->msgCb, nextProcessVer);
   if (code != TSDB_CODE_SUCCESS) {
@@ -71,8 +71,7 @@ SSnode *sndOpen(const char *path, const SSnodeOpt *pOption) {
   startRsync();
 
   pSnode->msgCb = pOption->msgCb;
-  pSnode->pMeta = streamMetaOpen(path, pSnode, (FTaskBuild *)sndExpandTask, tqExpandStreamTask, SNODE_HANDLE,
+  pSnode->pMeta = streamMetaOpen(path, pSnode, (FTaskBuild *)sndBuildStreamTask, tqExpandStreamTask, SNODE_HANDLE, taosGetTimestampMs(), tqStartTaskCompleteCallback);
-                                 taosGetTimestampMs(), tqStartTaskCompleteCallback);
   if (pSnode->pMeta == NULL) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
     goto FAIL;
@@ -157,6 +156,8 @@ int32_t sndProcessWriteMsg(SSnode *pSnode, SRpcMsg *pMsg, SRpcMsg *pRsp) {
       return tqStreamTaskProcessTaskResumeReq(pSnode->pMeta, pMsg->info.conn.applyIndex, pMsg->pCont, false);
     case TDMT_STREAM_TASK_UPDATE_CHKPT:
       return tqStreamTaskProcessUpdateCheckpointReq(pSnode->pMeta, true, pMsg->pCont, pMsg->contLen);
+    case TDMT_MND_STREAM_CHKPT_CONSEN_RSP:
+      return tqStreamProcessConsensusChkptRsp(pSnode->pMeta, pMsg);
     default:
       ASSERT(0);
   }

source/dnode/vnode/inc/vnode.h

@@ -106,7 +106,7 @@ int32_t vnodePreprocessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg);
 
 int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRpcMsg *pRsp);
 int32_t vnodeProcessSyncMsg(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp);
-int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg);
+int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo* pInfo);
 int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo);
 int32_t vnodeProcessStreamMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo);
 void    vnodeProposeWriteMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs);

source/dnode/vnode/src/inc/vnodeInt.h

@@ -262,6 +262,7 @@ int32_t tqProcessTaskResetReq(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessStreamHbRsp(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessStreamReqCheckpointRsp(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessTaskChkptReportRsp(STQ* pTq, SRpcMsg* pMsg);
+int32_t tqProcessTaskConsensusChkptRsp(STQ* pTq, SRpcMsg* pMsg);
 int32_t tqProcessTaskCheckpointReadyRsp(STQ* pTq, SRpcMsg* pMsg);
 
 int32_t tqBuildStreamTask(void* pTq, SStreamTask* pTask, int64_t ver);

source/dnode/vnode/src/tq/tq.c

@@ -139,9 +139,11 @@ void tqClose(STQ* pTq) {
   taosHashCleanup(pTq->pCheckInfo);
   taosMemoryFree(pTq->path);
   tqMetaClose(pTq);
+
+  int32_t vgId = pTq->pStreamMeta->vgId;
   streamMetaClose(pTq->pStreamMeta);
 
-  qDebug("end to close tq");
+  qDebug("vgId:%d end to close tq", vgId);
   taosMemoryFree(pTq);
 }
 
@@ -1030,7 +1032,6 @@ int32_t tqProcessTaskRetrieveReq(STQ* pTq, SRpcMsg* pMsg) {
 }
 
 int32_t tqProcessTaskRetrieveRsp(STQ* pTq, SRpcMsg* pMsg) {
-  //
   return 0;
 }
 
@@ -1274,3 +1275,7 @@ int32_t tqProcessTaskCheckpointReadyRsp(STQ* pTq, SRpcMsg* pMsg) {
 int32_t tqProcessTaskChkptReportRsp(STQ* pTq, SRpcMsg* pMsg) {
   return tqStreamProcessChkptReportRsp(pTq->pStreamMeta, pMsg);
 }
+
+int32_t tqProcessTaskConsensusChkptRsp(STQ* pTq, SRpcMsg* pMsg) {
+  return tqStreamProcessConsensusChkptRsp(pTq->pStreamMeta, pMsg);
+}

source/dnode/vnode/src/tqCommon/tqCommon.c

@@ -121,7 +121,6 @@ int32_t tqStreamTaskStartAsync(SStreamMeta* pMeta, SMsgCb* cb, bool restart) {
 
 int32_t tqStreamStartOneTaskAsync(SStreamMeta* pMeta, SMsgCb* cb, int64_t streamId, int32_t taskId) {
   int32_t vgId = pMeta->vgId;
-
   int32_t numOfTasks = taosArrayGetSize(pMeta->pTaskList);
   if (numOfTasks == 0) {
     tqDebug("vgId:%d no stream tasks existed to run", vgId);
@@ -132,6 +131,26 @@ int32_t tqStreamStartOneTaskAsync(SStreamMeta* pMeta, SMsgCb* cb, int64_t stream
   return streamTaskSchedTask(cb, vgId, streamId, taskId, STREAM_EXEC_T_START_ONE_TASK);
 }
 
+int32_t tqStreamTaskRestoreCheckpoint(SStreamMeta* pMeta, int64_t streamId, int32_t taskId) {
+  int32_t vgId = pMeta->vgId;
+  int32_t numOfTasks = taosArrayGetSize(pMeta->pTaskList);
+  if (numOfTasks == 0) {
+    tqDebug("vgId:%d no stream tasks existed to run", vgId);
+    return 0;
+  }
+
+  tqDebug("vgId:%d restore task:0x%" PRIx64 "-0x%x checkpointId", vgId, streamId, taskId);
+  SStreamTask* pTask = streamMetaAcquireTask(pMeta, streamId, taskId);
+  if (pTask == NULL) {
+    tqError("failed to acquire task:0x%x when trying to restore checkpointId", taskId);
+    return TSDB_CODE_STREAM_TASK_NOT_EXIST;
+  }
+
+  int32_t code = streamTaskSendRestoreChkptMsg(pTask);
+  streamMetaReleaseTask(pMeta, pTask);
+  return code;
+}
+
 int32_t tqStreamTaskProcessUpdateReq(SStreamMeta* pMeta, SMsgCb* cb, SRpcMsg* pMsg, bool restored) {
   int32_t vgId = pMeta->vgId;
   char*   msg = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
@@ -171,7 +190,11 @@ int32_t tqStreamTaskProcessUpdateReq(SStreamMeta* pMeta, SMsgCb* cb, SRpcMsg* pM
   SStreamTask* pTask = *ppTask;
   const char*  idstr = pTask->id.idStr;
 
-  if ((pMeta->updateInfo.transId != req.transId) && (pMeta->updateInfo.transId != -1)) {
+  if (pMeta->updateInfo.transId == -1) { // info needs to be kept till the new trans to update the nodeEp arrived.
+    streamMetaInitUpdateTaskList(pMeta, req.transId);
+  }
+
+  if (pMeta->updateInfo.transId != req.transId) {
     if (req.transId < pMeta->updateInfo.transId) {
       tqError("s-task:%s vgId:%d disorder update nodeEp msg recv, discarded, newest transId:%d, recv:%d", idstr, vgId,
               pMeta->updateInfo.transId, req.transId);
@@ -187,7 +210,8 @@ int32_t tqStreamTaskProcessUpdateReq(SStreamMeta* pMeta, SMsgCb* cb, SRpcMsg* pM
       streamMetaInitUpdateTaskList(pMeta, req.transId);
     }
   } else {
-    tqDebug("s-task:%s vgId:%d recv trans to update nodeEp from mnode, transId:%d", idstr, vgId, req.transId);
+    tqDebug("s-task:%s vgId:%d recv trans to update nodeEp from mnode, transId:%d, recorded update transId:%d", idstr,
+            vgId, req.transId, pMeta->updateInfo.transId);
   }
 
   // duplicate update epset msg received, discard this redundant message
@@ -656,8 +680,11 @@ int32_t tqStreamTaskProcessUpdateCheckpointReq(SStreamMeta* pMeta, bool restored
   if (ppTask != NULL && (*ppTask) != NULL) {
     streamTaskUpdateTaskCheckpointInfo(*ppTask, restored, pReq);
   } else {  // failed to get the task.
-    tqError("vgId:%d failed to locate the s-task:0x%x to update the checkpoint info, it may have been dropped already",
+    int32_t numOfTasks = streamMetaGetNumOfTasks(pMeta);
-            vgId, pReq->taskId);
+    tqError(
+        "vgId:%d failed to locate the s-task:0x%x to update the checkpoint info, numOfTasks:%d, it may have been "
+        "dropped already",
+        vgId, pReq->taskId, numOfTasks);
   }
 
   streamMetaWUnLock(pMeta);
@@ -708,9 +735,9 @@ static int32_t restartStreamTasks(SStreamMeta* pMeta, bool isLeader) {
 
   if (isLeader && !tsDisableStream) {
     streamMetaWUnLock(pMeta);
-    streamMetaStartAllTasks(pMeta, tqExpandStreamTask);
+    streamMetaStartAllTasks(pMeta);
   } else {
-    streamMetaResetStartInfo(&pMeta->startInfo);
+    streamMetaResetStartInfo(&pMeta->startInfo, pMeta->vgId);
     streamMetaWUnLock(pMeta);
     tqInfo("vgId:%d, follower node not start stream tasks or stream is disabled", vgId);
   }
@@ -726,10 +753,10 @@ int32_t tqStreamTaskProcessRunReq(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLead
   int32_t vgId = pMeta->vgId;
 
   if (type == STREAM_EXEC_T_START_ONE_TASK) {
-    streamMetaStartOneTask(pMeta, pReq->streamId, pReq->taskId, tqExpandStreamTask);
+    streamMetaStartOneTask(pMeta, pReq->streamId, pReq->taskId);
     return 0;
   } else if (type == STREAM_EXEC_T_START_ALL_TASKS) {
-    streamMetaStartAllTasks(pMeta, tqExpandStreamTask);
+    streamMetaStartAllTasks(pMeta);
     return 0;
   } else if (type == STREAM_EXEC_T_RESTART_ALL_TASKS) {
     restartStreamTasks(pMeta, isLeader);
@@ -857,7 +884,7 @@ int32_t tqStreamTaskProcessTaskResetReq(SStreamMeta* pMeta, SRpcMsg* pMsg) {
   } else if (pState->state == TASK_STATUS__UNINIT) {
     tqDebug("s-task:%s start task by checking downstream tasks", pTask->id.idStr);
     ASSERT(pTask->status.downstreamReady == 0);
-    tqStreamStartOneTaskAsync(pMeta, pTask->pMsgCb, pTask->id.streamId, pTask->id.taskId);
+    tqStreamTaskRestoreCheckpoint(pMeta, pTask->id.streamId, pTask->id.taskId);
   } else {
     tqDebug("s-task:%s status:%s do nothing after receiving reset-task from mnode", pTask->id.idStr, pState->name);
   }
@@ -937,7 +964,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerReq(SStreamMeta* pMeta, SRpcMsg* pMsg)
 }
 
 int32_t tqStreamTaskProcessRetrieveTriggerRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) {
-  SCheckpointTriggerRsp* pRsp = pMsg->pCont;
+  SCheckpointTriggerRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
 
   SStreamTask* pTask = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->taskId);
   if (pTask == NULL) {
@@ -1004,12 +1031,6 @@ static int32_t tqProcessTaskResumeImpl(void* handle, SStreamTask* pTask, int64_t
   ETaskStatus status = streamTaskGetStatus(pTask)->state;
 
   int32_t level = pTask->info.taskLevel;
-  if (level == TASK_LEVEL__SINK) {
-    ASSERT(status != TASK_STATUS__UNINIT);
-    streamMetaReleaseTask(pMeta, pTask);
-    return 0;
-  }
-
   if (status == TASK_STATUS__READY || status == TASK_STATUS__SCAN_HISTORY || status == TASK_STATUS__CK) {
     // no lock needs to secure the access of the version
     if (igUntreated && level == TASK_LEVEL__SOURCE && !pTask->info.fillHistory) {
@@ -1031,9 +1052,9 @@ static int32_t tqProcessTaskResumeImpl(void* handle, SStreamTask* pTask, int64_t
     } else {
       streamTrySchedExec(pTask);
     }
-  } else {
+  } /*else {
     ASSERT(status != TASK_STATUS__UNINIT);
-  }
+  }*/
 
   streamMetaReleaseTask(pMeta, pTask);
   return 0;
@@ -1043,16 +1064,32 @@ int32_t tqStreamTaskProcessTaskResumeReq(void* handle, int64_t sversion, char* m
   SVResumeStreamTaskReq* pReq = (SVResumeStreamTaskReq*)msg;
 
   SStreamMeta* pMeta = fromVnode ? ((STQ*)handle)->pStreamMeta : handle;
+
   SStreamTask* pTask = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId);
-  int32_t      code = tqProcessTaskResumeImpl(handle, pTask, sversion, pReq->igUntreated, fromVnode);
+  if (pTask == NULL) {
+    tqError("s-task:0x%x failed to acquire task to resume, it may have been dropped or stopped", pReq->taskId);
+    return TSDB_CODE_STREAM_TASK_IVLD_STATUS;
+  }
+
+  taosThreadMutexLock(&pTask->lock);
+  SStreamTaskState* pState = streamTaskGetStatus(pTask);
+  tqDebug("s-task:%s start to resume from paused, current status:%s", pTask->id.idStr, pState->name);
+  taosThreadMutexUnlock(&pTask->lock);
+
+  int32_t code = tqProcessTaskResumeImpl(handle, pTask, sversion, pReq->igUntreated, fromVnode);
   if (code != 0) {
     return code;
   }
 
   STaskId*     pHTaskId = &pTask->hTaskInfo.id;
-  SStreamTask* pHistoryTask = streamMetaAcquireTask(pMeta, pHTaskId->streamId, pHTaskId->taskId);
+  SStreamTask* pHTask = streamMetaAcquireTask(pMeta, pHTaskId->streamId, pHTaskId->taskId);
-  if (pHistoryTask) {
+  if (pHTask) {
-    code = tqProcessTaskResumeImpl(handle, pHistoryTask, sversion, pReq->igUntreated, fromVnode);
+    taosThreadMutexLock(&pHTask->lock);
+    SStreamTaskState* p = streamTaskGetStatus(pHTask);
+    tqDebug("s-task:%s related history task start to resume from paused, current status:%s", pHTask->id.idStr, p->name);
+    taosThreadMutexUnlock(&pHTask->lock);
+
+    code = tqProcessTaskResumeImpl(handle, pHTask, sversion, pReq->igUntreated, fromVnode);
   }
 
   return code;
@@ -1066,7 +1103,9 @@ int32_t doProcessDummyRspMsg(SStreamMeta* UNUSED_PARAM(pMeta), SRpcMsg* pMsg) {
   return TSDB_CODE_SUCCESS;
 }
 
-int32_t tqStreamProcessStreamHbRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) { return doProcessDummyRspMsg(pMeta, pMsg); }
+int32_t tqStreamProcessStreamHbRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) {
+  return streamProcessHeartbeatRsp(pMeta, pMsg->pCont);
+}
 
 int32_t tqStreamProcessReqCheckpointRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) { return doProcessDummyRspMsg(pMeta, pMsg); }
 
@@ -1086,3 +1125,68 @@ int32_t tqStreamProcessCheckpointReadyRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) {
   streamMetaReleaseTask(pMeta, pTask);
   return TSDB_CODE_SUCCESS;
 }
+
+int32_t tqStreamProcessConsensusChkptRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) {
+  int32_t vgId = pMeta->vgId;
+  char*   msg = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
+  int32_t len = pMsg->contLen - sizeof(SMsgHead);
+  SRpcMsg rsp = {.info = pMsg->info, .code = TSDB_CODE_SUCCESS};
+  int64_t now = taosGetTimestampMs();
+
+  SRestoreCheckpointInfoRsp req = {0};
+
+  SDecoder decoder;
+  tDecoderInit(&decoder, (uint8_t*)msg, len);
+
+  rsp.info.handle = NULL;
+  if (tDecodeRestoreCheckpointInfoRsp(&decoder, &req) < 0) {
+    //    rsp.code = TSDB_CODE_MSG_DECODE_ERROR;  // disable it temporarily
+    tqError("vgId:%d failed to decode restore task checkpointId, code:%s", vgId, tstrerror(rsp.code));
+    tDecoderClear(&decoder);
+    return TSDB_CODE_SUCCESS;
+  }
+
+  tDecoderClear(&decoder);
+
+  SStreamTask* pTask = streamMetaAcquireTask(pMeta, req.streamId, req.taskId);
+  if (pTask == NULL) {
+    tqError("vgId:%d process restore checkpointId req, failed to acquire task:0x%x, it may have been dropped already",
+            pMeta->vgId, req.taskId);
+    streamMetaAddFailedTask(pMeta, req.streamId, req.taskId);
+    return TSDB_CODE_SUCCESS;
+  }
+
+  // discard the rsp from before restart
+  if (req.startTs < pTask->execInfo.created) {
+    tqWarn("s-task:%s vgId:%d create time:%" PRId64 " recv expired consensus checkpointId:%" PRId64
+           " from task createTs:%" PRId64 ", discard",
+           pTask->id.idStr, pMeta->vgId, pTask->execInfo.created, req.checkpointId, req.startTs);
+    streamMetaAddFailedTaskSelf(pTask, now);
+    streamMetaReleaseTask(pMeta, pTask);
+    return TSDB_CODE_SUCCESS;
+  }
+
+  tqDebug("s-task:%s vgId:%d checkpointId:%" PRId64 " restore to consensus-checkpointId:%" PRId64 " from mnode",
+          pTask->id.idStr, vgId, pTask->chkInfo.checkpointId, req.checkpointId);
+
+  taosThreadMutexLock(&pTask->lock);
+  ASSERT(pTask->chkInfo.checkpointId >= req.checkpointId);
+
+  if ((pTask->chkInfo.checkpointId != req.checkpointId) && req.checkpointId != 0) {
+    tqDebug("s-task:%s vgId:%d update the checkpoint from %" PRId64 " to %" PRId64, pTask->id.idStr, vgId,
+            pTask->chkInfo.checkpointId, req.checkpointId);
+    pTask->chkInfo.checkpointId = req.checkpointId;
+    tqSetRestoreVersionInfo(pTask);
+  }
+
+  taosThreadMutexUnlock(&pTask->lock);
+
+  if (pMeta->role == NODE_ROLE_LEADER) {
+    /*code = */ tqStreamStartOneTaskAsync(pMeta, pTask->pMsgCb, req.streamId, req.taskId);
+  } else {
+    tqDebug("vgId:%d follower not start task:%s", vgId, pTask->id.idStr);
+  }
+
+  streamMetaReleaseTask(pMeta, pTask);
+  return TSDB_CODE_SUCCESS;
+}

source/dnode/vnode/src/tsdb/tsdbRetention.c

@@ -361,7 +361,7 @@ _exit:
   TARRAY2_DESTROY(&rtner.fopArr, NULL);
   taosMemoryFree(arg);
   if (code) {
-    tsdbError("vgId:%d, %s failed, code:%d, line:%d", TD_VID(((SRtnArg *)arg)->tsdb->pVnode), __func__, code, lino);
+    tsdbError("vgId:%d, %s failed, code:%d, line:%d", TD_VID(pTsdb->pVnode), __func__, code, lino);
   }
   return code;
 }

source/dnode/vnode/src/vnd/vnodeSvr.c

@@ -647,6 +647,11 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t ver, SRpcMsg
         tqProcessTaskResetReq(pVnode->pTq, pMsg);
       }
     } break;
+    case TDMT_MND_STREAM_CHKPT_CONSEN_RSP: {
+      if (pVnode->restored) {
+        tqProcessTaskConsensusChkptRsp(pVnode->pTq, pMsg);
+      }
+    } break;
     case TDMT_VND_ALTER_CONFIRM:
       needCommit = pVnode->config.hashChange;
       if (vnodeProcessAlterConfirmReq(pVnode, ver, pReq, len, pRsp) < 0) {
@@ -733,7 +738,7 @@ int32_t vnodePreprocessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg) {
   return qWorkerPreprocessQueryMsg(pVnode->pQuery, pMsg, TDMT_SCH_QUERY == pMsg->msgType);
 }
 
-int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg) {
+int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo* pInfo) {
   vTrace("message in vnode query queue is processing");
   if ((pMsg->msgType == TDMT_SCH_QUERY || pMsg->msgType == TDMT_VND_TMQ_CONSUME ||
        pMsg->msgType == TDMT_VND_TMQ_CONSUME_PUSH) &&
@@ -747,7 +752,7 @@ int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg) {
     return 0;
   }
 
-  SReadHandle handle = {.vnode = pVnode, .pMsgCb = &pVnode->msgCb};
+  SReadHandle handle = {.vnode = pVnode, .pMsgCb = &pVnode->msgCb, .pWorkerCb = pInfo->workerCb};
   initStorageAPI(&handle.api);
 
   switch (pMsg->msgType) {

source/libs/audit/src/auditMain.c

@@ -22,7 +22,6 @@
 #include "ttime.h"
 #include "tjson.h"
 #include "tglobal.h"
-#include "mnode.h"
 #include "audit.h"
 #include "osMemory.h"
 

source/libs/executor/inc/executorInt.h

@@ -42,6 +42,7 @@ extern "C" {
 // #include "tstream.h"
 // #include "tstreamUpdate.h"
 #include "tlrucache.h"
+#include "tworker.h"
 
 typedef int32_t (*__block_search_fn_t)(char* data, int32_t num, int64_t key, int32_t order);
 

source/libs/executor/inc/querytask.h

@@ -95,6 +95,7 @@ struct SExecTaskInfo {
   int8_t                dynamicTask;
   SOperatorParam*       pOpParam;
   bool                  paramSet;
+  SQueryAutoQWorkerPoolCB* pWorkerCb;
 };
 
 void           buildTaskId(uint64_t taskId, uint64_t queryId, char* dst);

source/libs/executor/src/exchangeoperator.c

@@ -60,6 +60,8 @@ static int32_t handleLimitOffset(SOperatorInfo* pOperator, SLimitInfo* pLimitInf
                                  bool holdDataInBuf);
 static int32_t doExtractResultBlocks(SExchangeInfo* pExchangeInfo, SSourceDataInfo* pDataInfo);
 
+static int32_t exchangeWait(SOperatorInfo* pOperator, SExchangeInfo* pExchangeInfo);
+
 static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeInfo* pExchangeInfo,
                                            SExecTaskInfo* pTaskInfo) {
   int32_t code = 0;
@@ -74,9 +76,9 @@ static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeIn
 
   while (1) {
     qDebug("prepare wait for ready, %p, %s", pExchangeInfo, GET_TASKID(pTaskInfo));
-    tsem_wait(&pExchangeInfo->ready);
+    code = exchangeWait(pOperator, pExchangeInfo);
 
-    if (isTaskKilled(pTaskInfo)) {
+    if (code != TSDB_CODE_SUCCESS || isTaskKilled(pTaskInfo)) {
       T_LONG_JMP(pTaskInfo->env, pTaskInfo->code);
     }
 
@@ -756,8 +758,8 @@ int32_t seqLoadRemoteData(SOperatorInfo* pOperator) {
     pDataInfo->status = EX_SOURCE_DATA_NOT_READY;
 
     doSendFetchDataRequest(pExchangeInfo, pTaskInfo, pExchangeInfo->current);
-    tsem_wait(&pExchangeInfo->ready);
+    code = exchangeWait(pOperator, pExchangeInfo);
-    if (isTaskKilled(pTaskInfo)) {
+    if (code != TSDB_CODE_SUCCESS || isTaskKilled(pTaskInfo)) {
       T_LONG_JMP(pTaskInfo->env, pTaskInfo->code);
     }
 
@@ -971,3 +973,24 @@ int32_t handleLimitOffset(SOperatorInfo* pOperator, SLimitInfo* pLimitInfo, SSDa
     return PROJECT_RETRIEVE_CONTINUE;
   }
 }
+
+static int32_t exchangeWait(SOperatorInfo* pOperator, SExchangeInfo* pExchangeInfo) {
+  SExecTaskInfo* pTask = pOperator->pTaskInfo;
+  int32_t        code = TSDB_CODE_SUCCESS;
+  if (pTask->pWorkerCb) {
+    code = pTask->pWorkerCb->beforeBlocking(pTask->pWorkerCb->pPool);
+    if (code != TSDB_CODE_SUCCESS) {
+      pTask->code = code;
+      return pTask->code;
+    }
+  }
+  tsem_wait(&pExchangeInfo->ready);
+  if (pTask->pWorkerCb) {
+    code = pTask->pWorkerCb->afterRecoverFromBlocking(pTask->pWorkerCb->pPool);
+    if (code != TSDB_CODE_SUCCESS) {
+      pTask->code = code;
+      return pTask->code;
+    }
+  }
+  return TSDB_CODE_SUCCESS;
+}

source/libs/executor/src/querytask.c

@@ -96,6 +96,7 @@ int32_t createExecTaskInfo(SSubplan* pPlan, SExecTaskInfo** pTaskInfo, SReadHand
   TSWAP((*pTaskInfo)->sql, sql);
 
   (*pTaskInfo)->pSubplan = pPlan;
+  (*pTaskInfo)->pWorkerCb = pHandle->pWorkerCb;
   (*pTaskInfo)->pRoot = createOperator(pPlan->pNode, *pTaskInfo, pHandle, pPlan->pTagCond, pPlan->pTagIndexCond,
                                        pPlan->user, pPlan->dbFName);
 

source/libs/executor/src/scanoperator.c

@@ -2265,13 +2265,16 @@ static void processPrimaryKey(SSDataBlock* pBlock, bool hasPrimaryKey, STqOffset
     doBlockDataPrimaryKeyFilter(pBlock, offset);
     SColumnInfoData* pColPk = taosArrayGet(pBlock->pDataBlock, 1);
 
+    if (pBlock->info.rows < 1) {
+      return ;
+    }
     void* tmp = colDataGetData(pColPk, pBlock->info.rows - 1);
     val.type = pColPk->info.type;
-    if(IS_VAR_DATA_TYPE(pColPk->info.type)) {
+    if (IS_VAR_DATA_TYPE(pColPk->info.type)) {
       val.pData = taosMemoryMalloc(varDataLen(tmp));
       val.nData = varDataLen(tmp);
       memcpy(val.pData, varDataVal(tmp), varDataLen(tmp));
-    }else{
+    } else {
       memcpy(&val.val, tmp, pColPk->info.bytes);
     }
   }
@@ -2292,13 +2295,19 @@ static SSDataBlock* doQueueScan(SOperatorInfo* pOperator) {
   }
 
   if (pTaskInfo->streamInfo.currentOffset.type == TMQ_OFFSET__SNAPSHOT_DATA) {
-    SSDataBlock* pResult = doTableScan(pInfo->pTableScanOp);
+    while (1) {
+      SSDataBlock* pResult = doTableScan(pInfo->pTableScanOp);
 
-    if (pResult && pResult->info.rows > 0) {
+      if (pResult && pResult->info.rows > 0) {
-      bool hasPrimaryKey = pAPI->tqReaderFn.tqGetTablePrimaryKey(pInfo->tqReader);
+        bool hasPrimaryKey = pAPI->tqReaderFn.tqGetTablePrimaryKey(pInfo->tqReader);
-      processPrimaryKey(pResult, hasPrimaryKey, &pTaskInfo->streamInfo.currentOffset);
+        processPrimaryKey(pResult, hasPrimaryKey, &pTaskInfo->streamInfo.currentOffset);
-      qDebug("tmqsnap doQueueScan get data uid:%" PRId64 "", pResult->info.id.uid);
+        qDebug("tmqsnap doQueueScan get data uid:%" PRId64 "", pResult->info.id.uid);
-      return pResult;
+        if (pResult->info.rows > 0) {
+          return pResult;
+        }
+      } else {
+        break;
+      }
     }
 
     STableScanInfo* pTSInfo = pInfo->pTableScanOp->info;

source/libs/executor/src/tfill.c

@@ -84,7 +84,9 @@ static void doSetUserSpecifiedValue(SColumnInfoData* pDst, SVariant* pVar, int32
     GET_TYPED_DATA(v, uint64_t, pVar->nType, &pVar->u);
     colDataSetVal(pDst, rowIndex, (char*)&v, isNull);
   } else if (pDst->info.type == TSDB_DATA_TYPE_TIMESTAMP) {
-    colDataSetVal(pDst, rowIndex, (const char*)&currentKey, isNull);
+    int64_t v = 0;
+    GET_TYPED_DATA(v, int64_t, pVar->nType, &pVar->u);
+    colDataSetVal(pDst, rowIndex, (const char*)&v, isNull);
   } else if (pDst->info.type == TSDB_DATA_TYPE_NCHAR || pDst->info.type == TSDB_DATA_TYPE_VARCHAR ||
              pDst->info.type == TSDB_DATA_TYPE_VARBINARY) {
     colDataSetVal(pDst, rowIndex, pVar->pz, isNull);

source/libs/parser/src/parInsertSql.c

@@ -2428,6 +2428,20 @@ static int32_t checkTableClauseFirstToken(SInsertParseContext* pCxt, SVnodeModif
     return buildSyntaxErrMsg(&pCxt->msg, "table_name is expected", pTbName->z);
   }
 
+  // db.? situation，ensure that the only thing following the '.' mark is '?'
+  char *tbNameAfterDbName = strchr(pTbName->z, '.');
+  if ((tbNameAfterDbName != NULL) && (tbNameAfterDbName + 1 - pTbName->z == pTbName->n - 1) &&
+      (*(tbNameAfterDbName + 1) == '?')) {
+    char *tbName = NULL;
+    int32_t code = (*pCxt->pComCxt->pStmtCb->getTbNameFn)(pCxt->pComCxt->pStmtCb->pStmt, &tbName);
+    if (TSDB_CODE_SUCCESS == code) {
+      pTbName->z = tbName;
+      pTbName->n = strlen(tbName);
+    } else {
+      return code;
+    }
+  }
+
   *pHasData = true;
   return TSDB_CODE_SUCCESS;
 }

source/libs/parser/src/parTokenizer.c

@@ -781,8 +781,9 @@ SToken tStrGetToken(const char* str, int32_t* i, bool isPrevOptr, bool* pIgnoreC
   if ('.' == str[*i + t0.n]) {
     len = tGetToken(&str[*i + t0.n + 1], &type);
 
-    // only id and string are valid
+    // only id、string and ? are valid
-    if (((TK_NK_STRING != t0.type) && (TK_NK_ID != t0.type)) || ((TK_NK_STRING != type) && (TK_NK_ID != type))) {
+    if (((TK_NK_STRING != t0.type) && (TK_NK_ID != t0.type)) ||
+        ((TK_NK_STRING != type) && (TK_NK_ID != type) && (TK_NK_QUESTION != type))) {
       t0.type = TK_NK_ILLEGAL;
       t0.n = 0;
 

source/libs/parser/src/parTranslater.c

@@ -12807,8 +12807,7 @@ static int32_t parseOneStbRow(SMsgBuf* pMsgBuf, SParseFileContext* pParFileCtx)
       if (TSDB_CODE_SUCCESS == code) {
         SArray* aTagNames = pParFileCtx->tagNameFilled ? NULL : pParFileCtx->aTagNames;
         code = parseTagValue(pMsgBuf, &pParFileCtx->pSql, precision, (SSchema*)pTagSchema, &token,
-                             pParFileCtx->aTagNames, pParFileCtx->aTagVals, &pParFileCtx->pTag);
+                             aTagNames, pParFileCtx->aTagVals, &pParFileCtx->pTag);
-        pParFileCtx->tagNameFilled = true;
       }
     } else {
       // parse tbname
@@ -12822,9 +12821,12 @@ static int32_t parseOneStbRow(SMsgBuf* pMsgBuf, SParseFileContext* pParFileCtx)
         code = parseTbnameToken(pMsgBuf, pParFileCtx->ctbName.tname, &token, &bFoundTbName);
       }
     }
+
+    if (TSDB_CODE_SUCCESS != code) break;
   }
 
-  if (TSDB_CODE_SUCCESS == code) {  // may fail to handle json
+  if (TSDB_CODE_SUCCESS == code) {
+    pParFileCtx->tagNameFilled = true;
     code = tTagNew(pParFileCtx->aTagVals, 1, false, &pParFileCtx->pTag);
   }
 
@@ -12897,6 +12899,7 @@ static int32_t parseCsvFile(SMsgBuf* pMsgBuf, SParseContext* pParseCxt, SParseFi
       taosMemoryFreeClear(pParseFileCtx->pTag);
     }
 
+    pParseFileCtx->pTag = NULL;
     taosArrayClearEx(pParseFileCtx->aTagVals, clearTagValArrayFp);
   }
 

source/libs/parser/src/parser.c

@@ -48,6 +48,47 @@ bool qIsInsertValuesSql(const char* pStr, size_t length) {
   return false;
 }
 
+bool qParseDbName(const char* pStr, size_t length, char** pDbName) {
+  (void) length;
+  int32_t index = 0;
+  SToken t;
+
+  if (NULL == pStr) {
+    *pDbName = NULL;
+    return false;
+  }
+
+  t = tStrGetToken((char *) pStr, &index, false, NULL);
+  if (TK_INSERT != t.type && TK_IMPORT != t.type) {
+    *pDbName = NULL;
+    return false;
+  }
+
+  t = tStrGetToken((char *) pStr, &index, false, NULL);
+  if (TK_INTO != t.type) {
+    *pDbName = NULL;
+    return false;
+  }
+
+  t = tStrGetToken((char *) pStr, &index, false, NULL);
+  if (t.n == 0 || t.z == NULL) {
+    *pDbName = NULL;
+    return false;
+  }
+  char *dotPos = strnchr(t.z, '.', t.n, true);
+  if (dotPos != NULL) {
+    int dbNameLen = dotPos - t.z;
+    *pDbName = taosMemoryMalloc(dbNameLen + 1);
+    if (*pDbName == NULL) {
+      return false;
+    }
+    strncpy(*pDbName, t.z, dbNameLen);
+    (*pDbName)[dbNameLen] = '\0';
+    return true;
+  }
+  return false;
+}
+
 static int32_t analyseSemantic(SParseContext* pCxt, SQuery* pQuery, SParseMetaCache* pMetaCache) {
   int32_t code = authenticate(pCxt, pQuery, pMetaCache);
 

source/libs/planner/src/planOptimizer.c

@@ -4051,106 +4051,121 @@ static int32_t lastRowScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogic
   SNode*                           pNode = NULL;
   SColumnNode*                     pPKTsCol = NULL;
   SColumnNode*                     pNonPKCol = NULL;
-  SScanLogicNode* pScan = (SScanLogicNode*)nodesListGetNode(pAgg->node.pChildren, 0);
+  SScanLogicNode*                  pScan = (SScanLogicNode*)nodesListGetNode(pAgg->node.pChildren, 0);
   pScan->scanType = SCAN_TYPE_LAST_ROW;
   pScan->igLastNull = pAgg->hasLast ? true : false;
-  SArray* isDuplicateCol = taosArrayInit(pScan->pScanCols->length, sizeof(bool));
+  SArray*    isDuplicateCol = taosArrayInit(pScan->pScanCols->length, sizeof(bool));
   SNodeList* pLastRowCols = NULL;
+  bool       adjLastRowTsColName = false;
+  char       tsColName[TSDB_COL_NAME_LEN] = {0};
 
   FOREACH(pNode, pAgg->pAggFuncs) {
     SFunctionNode* pFunc = (SFunctionNode*)pNode;
     int32_t        funcType = pFunc->funcType;
-    SNode* pParamNode = nodesListGetNode(pFunc->pParameterList, 0);
+    SNode*         pParamNode = NULL;
-    if (FUNCTION_TYPE_LAST_ROW == funcType || FUNCTION_TYPE_LAST == funcType) {
+    if (FUNCTION_TYPE_LAST == funcType) {
-      int32_t len = snprintf(pFunc->functionName, sizeof(pFunc->functionName),
+      nodesListErase(pFunc->pParameterList, nodesListGetCell(pFunc->pParameterList, 1));
-                             FUNCTION_TYPE_LAST_ROW == funcType ? "_cache_last_row" : "_cache_last");
+      nodesWalkExpr(nodesListGetNode(pFunc->pParameterList, 0), lastRowScanOptSetColDataType, &cxt);
-      pFunc->functionName[len] = '\0';
+    }
-      int32_t code = fmGetFuncInfo(pFunc, NULL, 0);
+    FOREACH(pParamNode, pFunc->pParameterList) {
-      if (TSDB_CODE_SUCCESS != code) {
+      if (FUNCTION_TYPE_LAST_ROW == funcType || FUNCTION_TYPE_LAST == funcType) {
-        nodesClearList(cxt.pLastCols);
+        int32_t len = snprintf(pFunc->functionName, sizeof(pFunc->functionName),
-        return code;
+                              FUNCTION_TYPE_LAST_ROW == funcType ? "_cache_last_row" : "_cache_last");
-      }
+        pFunc->functionName[len] = '\0';
-      cxt.funcType = pFunc->funcType;
+        int32_t code = fmGetFuncInfo(pFunc, NULL, 0);
-      cxt.pkBytes = (pFunc->hasPk) ? pFunc->pkBytes : 0;
+        if (TSDB_CODE_SUCCESS != code) {
-      // add duplicate cols which be removed for both last_row, last
+          nodesClearList(cxt.pLastCols);
-      if (pAgg->hasLast && pAgg->hasLastRow) {
+          return code;
-        if (QUERY_NODE_COLUMN == nodeType(pParamNode)) {
+        }
-          SNode* pColNode = NULL;
+        cxt.funcType = pFunc->funcType;
-          int i = 0;
+        cxt.pkBytes = (pFunc->hasPk) ? pFunc->pkBytes : 0;
-          FOREACH(pColNode, pScan->pScanCols) {
+        // add duplicate cols which be removed for both last_row, last
-            bool isDup = false;
+        if (pAgg->hasLast && pAgg->hasLastRow) {
-            bool* isDuplicate = taosArrayGet(isDuplicateCol, i);
+          if (QUERY_NODE_COLUMN == nodeType(pParamNode)) {
-            if (NULL == isDuplicate) {
+            SNode* pColNode = NULL;
-              taosArrayInsert(isDuplicateCol, i, &isDup);
+            int i = 0;
-              isDuplicate = taosArrayGet(isDuplicateCol, i);
+            FOREACH(pColNode, pScan->pScanCols) {
+              bool isDup = false;
+              bool* isDuplicate = taosArrayGet(isDuplicateCol, i);
+              if (NULL == isDuplicate) {
+                taosArrayInsert(isDuplicateCol, i, &isDup);
+                isDuplicate = taosArrayGet(isDuplicateCol, i);
+              }
+              i++;
+              if (nodesEqualNode(pParamNode, pColNode)) {
+                if (*isDuplicate) {
+                  if (0 == strncmp(((SColumnNode*)pColNode)->colName, "#dup_col.", 9)) {
+                    continue;
+                  }
+                  SNode* newColNode = nodesCloneNode(pColNode);
+                  sprintf(((SColumnNode*)newColNode)->colName, "#dup_col.%p", newColNode);
+                  sprintf(((SColumnNode*)pParamNode)->colName, "#dup_col.%p", newColNode);
+                  if (FUNCTION_TYPE_LAST_ROW == funcType &&
+                      ((SColumnNode*)pParamNode)->colId == PRIMARYKEY_TIMESTAMP_COL_ID) {
+                    if (!adjLastRowTsColName) {
+                      adjLastRowTsColName = true;
+                      strncpy(tsColName, ((SColumnNode*)pParamNode)->colName, TSDB_COL_NAME_LEN);
+                    } else {
+                      strncpy(((SColumnNode*)pParamNode)->colName, tsColName, TSDB_COL_NAME_LEN);
+                      nodesDestroyNode(newColNode);
+                      continue;
+                    }
+                  }
+
+                  nodesListAppend(pScan->pScanCols, newColNode);
+                  isDup = true;
+                  taosArrayInsert(isDuplicateCol, pScan->pScanCols->length, &isDup);
+                  nodesListAppend(pScan->node.pTargets, nodesCloneNode(newColNode));
+                  if (funcType != FUNCTION_TYPE_LAST) {
+                    nodesListMakeAppend(&pLastRowCols, nodesCloneNode(newColNode));
+                  }
+
+                  lastRowScanBuildFuncTypes(pScan, (SColumnNode*)newColNode, pFunc->funcType);
+                } else {
+                  isDup = true;
+                  *isDuplicate = isDup;
+                  if (funcType != FUNCTION_TYPE_LAST && !nodeListNodeEqual(cxt.pLastCols, pColNode)) {
+                    nodesListMakeAppend(&pLastRowCols, nodesCloneNode(pColNode));
+                  }
+                  lastRowScanBuildFuncTypes(pScan, (SColumnNode*)pColNode, pFunc->funcType);
+                }
+                continue;
+              }else if (nodeListNodeEqual(pFunc->pParameterList, pColNode)) {
+                if (funcType != FUNCTION_TYPE_LAST && ((SColumnNode*)pColNode)->colId == PRIMARYKEY_TIMESTAMP_COL_ID &&
+                    !nodeListNodeEqual(pLastRowCols, pColNode)) {
+                  nodesListMakeAppend(&pLastRowCols, nodesCloneNode(pColNode));
+
+                  lastRowScanBuildFuncTypes(pScan, (SColumnNode*)pColNode, pFunc->funcType);
+                  isDup = true;
+                  *isDuplicate = isDup;
+                }
+              }
             }
-            i++;
+            FOREACH(pColNode, pScan->pScanPseudoCols) {
-            if (nodesEqualNode(pParamNode, pColNode)) {
+              if (nodesEqualNode(pParamNode, pColNode)) {
-              if (*isDuplicate) {
-                if (0 == strncmp(((SColumnNode*)pColNode)->colName, "#dup_col.", 9)) {
-                  continue;
-                }
-                SNode* newColNode = nodesCloneNode(pColNode);
-                sprintf(((SColumnNode*)newColNode)->colName, "#dup_col.%p", newColNode);
-                sprintf(((SColumnNode*)pParamNode)->colName, "#dup_col.%p", newColNode);
-
-                nodesListAppend(pScan->pScanCols, newColNode);
-                isDup = true;
-                taosArrayInsert(isDuplicateCol, pScan->pScanCols->length, &isDup);
-                nodesListAppend(pScan->node.pTargets, nodesCloneNode(newColNode));
                 if (funcType != FUNCTION_TYPE_LAST) {
-                  nodesListMakeAppend(&pLastRowCols, nodesCloneNode(newColNode));
-                }
-
-                lastRowScanBuildFuncTypes(pScan, (SColumnNode*)newColNode, pFunc->funcType);
-              } else {
-                isDup = true;
-                *isDuplicate = isDup;
-                if (funcType != FUNCTION_TYPE_LAST && !nodeListNodeEqual(cxt.pLastCols, pColNode)) {
                   nodesListMakeAppend(&pLastRowCols, nodesCloneNode(pColNode));
                 }
-                lastRowScanBuildFuncTypes(pScan, (SColumnNode*)pColNode, pFunc->funcType);
-              }
-              continue;
-            }else if (nodeListNodeEqual(pFunc->pParameterList, pColNode)) {
-              if (funcType != FUNCTION_TYPE_LAST && ((SColumnNode*)pColNode)->colId == PRIMARYKEY_TIMESTAMP_COL_ID &&
-                  !nodeListNodeEqual(pLastRowCols, pColNode)) {
-                nodesListMakeAppend(&pLastRowCols, nodesCloneNode(pColNode));
-
-                lastRowScanBuildFuncTypes(pScan, (SColumnNode*)pColNode, pFunc->funcType);
-                isDup = true;
-                *isDuplicate = isDup;
-              }
-            }
-          }
-          FOREACH(pColNode, pScan->pScanPseudoCols) {
-            if (nodesEqualNode(pParamNode, pColNode)) {
-              if (funcType != FUNCTION_TYPE_LAST) {
-                nodesListMakeAppend(&pLastRowCols, nodesCloneNode(pColNode));
               }
             }
           }
         }
-      }
 
-      if (FUNCTION_TYPE_LAST == funcType) {
+        if (pFunc->hasPk) {
-        nodesWalkExpr(nodesListGetNode(pFunc->pParameterList, 0), lastRowScanOptSetColDataType, &cxt);
+          nodesListMakeAppend(&cxt.pOtherCols, nodesListGetNode(pFunc->pParameterList, LIST_LENGTH(pFunc->pParameterList) - 1));
-        nodesListErase(pFunc->pParameterList, nodesListGetCell(pFunc->pParameterList, 1));
+        }
-      }
+      } else {
-      if (pFunc->hasPk) {
+        pNode = nodesListGetNode(pFunc->pParameterList, 0);
-        nodesListMakeAppend(&cxt.pOtherCols, nodesListGetNode(pFunc->pParameterList, LIST_LENGTH(pFunc->pParameterList) - 1));
+        nodesListMakeAppend(&cxt.pOtherCols, pNode);
-      }
-    } else {
-      pNode = nodesListGetNode(pFunc->pParameterList, 0);
-      nodesListMakeAppend(&cxt.pOtherCols, pNode);
 
-      if (FUNCTION_TYPE_SELECT_VALUE == funcType) {
+        if (FUNCTION_TYPE_SELECT_VALUE == funcType) {
-        if (nodeType(pNode) == QUERY_NODE_COLUMN) {
+          if (nodeType(pNode) == QUERY_NODE_COLUMN) {
-          SColumnNode* pCol = (SColumnNode*)pNode;
+            SColumnNode* pCol = (SColumnNode*)pNode;
-          if (pCol->colId == PRIMARYKEY_TIMESTAMP_COL_ID) {
+            if (pCol->colId == PRIMARYKEY_TIMESTAMP_COL_ID) {
-            pPKTsCol = pCol;
+              pPKTsCol = pCol;
-          } else {
+            } else {
-            pNonPKCol = pCol;
+              pNonPKCol = pCol;
+            }
           }
         }
       }

source/libs/qcom/src/queryUtil.c

@@ -19,10 +19,16 @@
 #include "tmsg.h"
 #include "trpc.h"
 #include "tsched.h"
+#include "tworker.h"
 // clang-format off
 #include "cJSON.h"
 #include "queryInt.h"
 
+typedef struct STaskQueue {
+  SQueryAutoQWorkerPool wrokrerPool;
+  STaosQueue* pTaskQueue;
+} STaskQueue;
+
 int32_t getAsofJoinReverseOp(EOperatorType op) {
   switch (op) {
     case OP_TYPE_GREATER_THAN:
@@ -118,12 +124,26 @@ bool tIsValidSchema(struct SSchema* pSchema, int32_t numOfCols, int32_t numOfTag
   return true;
 }
 
-static SSchedQueue pTaskQueue = {0};
+static STaskQueue taskQueue = {0};
+
+static void processTaskQueue(SQueueInfo *pInfo, SSchedMsg *pSchedMsg) {
+  __async_exec_fn_t execFn = (__async_exec_fn_t)pSchedMsg->ahandle;
+  execFn(pSchedMsg->thandle);
+  taosFreeQitem(pSchedMsg);
+}
 
 int32_t initTaskQueue() {
-  int32_t queueSize = tsMaxShellConns * 2;
+  taskQueue.wrokrerPool.name = "taskWorkPool";
-  void *p = taosInitScheduler(queueSize, tsNumOfTaskQueueThreads, "tsc", &pTaskQueue);
+  taskQueue.wrokrerPool.min = tsNumOfTaskQueueThreads;
-  if (NULL == p) {
+  taskQueue.wrokrerPool.max = tsNumOfTaskQueueThreads;
+  int32_t coce = tQueryAutoQWorkerInit(&taskQueue.wrokrerPool);
+  if (TSDB_CODE_SUCCESS != coce) {
+    qError("failed to init task thread pool");
+    return -1;
+  }
+
+  taskQueue.pTaskQueue = tQueryAutoQWorkerAllocQueue(&taskQueue.wrokrerPool, NULL, (FItem)processTaskQueue);
+  if (NULL == taskQueue.pTaskQueue) {
     qError("failed to init task queue");
     return -1;
   }
@@ -133,26 +153,34 @@ int32_t initTaskQueue() {
 }
 
 int32_t cleanupTaskQueue() {
-  taosCleanUpScheduler(&pTaskQueue);
+  tQueryAutoQWorkerCleanup(&taskQueue.wrokrerPool);
   return 0;
 }
 
-static void execHelper(struct SSchedMsg* pSchedMsg) {
+int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code) {
-  __async_exec_fn_t execFn = (__async_exec_fn_t)pSchedMsg->ahandle;
+  SSchedMsg* pSchedMsg = taosAllocateQitem(sizeof(SSchedMsg), DEF_QITEM, 0);
-  int32_t           code = execFn(pSchedMsg->thandle);
+  pSchedMsg->fp = NULL;
-  if (code != 0 && pSchedMsg->msg != NULL) {
+  pSchedMsg->ahandle = execFn;
-    *(int32_t*)pSchedMsg->msg = code;
+  pSchedMsg->thandle = execParam;
-  }
+  pSchedMsg->msg = code;
+
+  return taosWriteQitem(taskQueue.pTaskQueue, pSchedMsg);
 }
 
-int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code) {
+int32_t taosAsyncWait() {
-  SSchedMsg schedMsg = {0};
+  if (!taskQueue.wrokrerPool.pCb) {
-  schedMsg.fp = execHelper;
+    qError("query task thread pool callback function is null");
-  schedMsg.ahandle = execFn;
+    return -1;
-  schedMsg.thandle = execParam;
+  }
-  schedMsg.msg = code;
+  return taskQueue.wrokrerPool.pCb->beforeBlocking(&taskQueue.wrokrerPool);
+}
 
-  return taosScheduleTask(&pTaskQueue, &schedMsg);
+int32_t taosAsyncRecover() {
+  if (!taskQueue.wrokrerPool.pCb) {
+    qError("query task thread pool callback function is null");
+    return -1;
+  }
+  return taskQueue.wrokrerPool.pCb->afterRecoverFromBlocking(&taskQueue.wrokrerPool);
 }
 
 void destroySendMsgInfo(SMsgSendInfo* pMsgBody) {

source/libs/stream/inc/streamInt.h

@@ -65,6 +65,11 @@ struct SActiveCheckpointInfo {
   tmr_h         pSendReadyMsgTmr;
 };
 
+struct SConsensusCheckpoint {
+  int8_t inProcess;
+
+};
+
 typedef struct {
   int8_t       type;
   SSDataBlock* pBlock;
@@ -157,6 +162,7 @@ extern void*   streamTimer;
 extern int32_t streamBackendId;
 extern int32_t streamBackendCfWrapperId;
 extern int32_t taskDbWrapperId;
+extern int32_t streamMetaId;
 
 int32_t streamTimerInit();
 void    streamTimerCleanUp();
@@ -211,7 +217,13 @@ void*         streamQueueNextItem(SStreamQueue* pQueue);
 void          streamFreeQitem(SStreamQueueItem* data);
 int32_t       streamQueueGetItemSize(const SStreamQueue* pQueue);
 
-void streamMetaRemoveDB(void* arg, char* key);
+void         streamMetaRemoveDB(void* arg, char* key);
+void         streamMetaHbToMnode(void* param, void* tmrId);
+SMetaHbInfo* createMetaHbInfo(int64_t* pRid);
+void*        destroyMetaHbInfo(SMetaHbInfo* pInfo);
+void         streamMetaWaitForHbTmrQuit(SStreamMeta* pMeta);
+void         streamMetaGetHbSendInfo(SMetaHbInfo* pInfo, int64_t* pStartTs, int32_t* pSendCount);
+int32_t      streamMetaSendHbHelper(SStreamMeta* pMeta);
 
 ECHECKPOINT_BACKUP_TYPE streamGetCheckpointBackupType();
 

source/libs/stream/src/streamBackendRocksdb.c

@@ -2289,8 +2289,12 @@ void* taskDbAddRef(void* pTaskDb) {
   STaskDbWrapper* pBackend = pTaskDb;
   return taosAcquireRef(taskDbWrapperId, pBackend->refId);
 }
+
 void taskDbRemoveRef(void* pTaskDb) {
-  if (pTaskDb == NULL) return;
+  if (pTaskDb == NULL) {
+    return;
+  }
+
   STaskDbWrapper* pBackend = pTaskDb;
   taosReleaseRef(taskDbWrapperId, pBackend->refId);
 }

source/libs/stream/src/streamCheckpoint.c

@@ -25,15 +25,12 @@ static int32_t deleteCheckpoint(const char* id);
 static int32_t downloadCheckpointByNameS3(const char* id, const char* fname, const char* dstName);
 static int32_t continueDispatchCheckpointTriggerBlock(SStreamDataBlock* pBlock, SStreamTask* pTask);
 static int32_t appendCheckpointIntoInputQ(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId,
-                                          int32_t transId);
+                                          int32_t transId, int32_t srcTaskId);
 static int32_t doSendRetrieveTriggerMsg(SStreamTask* pTask, SArray* pNotSendList);
 static void    checkpointTriggerMonitorFn(void* param, void* tmrId);
 
-static SStreamDataBlock* createChkptTriggerBlock(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId,
-                                                 int32_t transId);
-
 SStreamDataBlock* createChkptTriggerBlock(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId,
-                                          int32_t transId) {
+                                          int32_t transId, int32_t srcTaskId) {
   SStreamDataBlock* pChkpoint = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SSDataBlock));
   if (pChkpoint == NULL) {
     terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -41,6 +38,10 @@ SStreamDataBlock* createChkptTriggerBlock(SStreamTask* pTask, int32_t checkpoint
   }
 
   pChkpoint->type = checkpointType;
+  if (checkpointType == STREAM_INPUT__CHECKPOINT_TRIGGER && (pTask->info.taskLevel != TASK_LEVEL__SOURCE)) {
+    pChkpoint->srcTaskId = srcTaskId;
+    ASSERT(srcTaskId != 0);
+  }
 
   SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));
   if (pBlock == NULL) {
@@ -71,8 +72,9 @@ SStreamDataBlock* createChkptTriggerBlock(SStreamTask* pTask, int32_t checkpoint
   return pChkpoint;
 }
 
-int32_t appendCheckpointIntoInputQ(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId, int32_t transId) {
+int32_t appendCheckpointIntoInputQ(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId, int32_t transId,
-  SStreamDataBlock* pCheckpoint = createChkptTriggerBlock(pTask, checkpointType, checkpointId, transId);
+                                   int32_t srcTaskId) {
+  SStreamDataBlock* pCheckpoint = createChkptTriggerBlock(pTask, checkpointType, checkpointId, transId, srcTaskId);
 
   if (streamTaskPutDataIntoInputQ(pTask, (SStreamQueueItem*)pCheckpoint) < 0) {
     return TSDB_CODE_OUT_OF_MEMORY;
@@ -97,7 +99,7 @@ int32_t streamProcessCheckpointSourceReq(SStreamTask* pTask, SStreamCheckpointSo
 
   // 2. Put the checkpoint block into inputQ, to make sure all blocks with less version have been handled by this task
   // and this is the last item in the inputQ.
-  return appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT_TRIGGER, pReq->checkpointId, pReq->transId);
+  return appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT_TRIGGER, pReq->checkpointId, pReq->transId, -1);
 }
 
 int32_t streamTaskProcessCheckpointTriggerRsp(SStreamTask* pTask, SCheckpointTriggerRsp* pRsp) {
@@ -109,7 +111,8 @@ int32_t streamTaskProcessCheckpointTriggerRsp(SStreamTask* pTask, SCheckpointTri
     return TSDB_CODE_SUCCESS;
   }
 
-  appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT_TRIGGER, pRsp->checkpointId, pRsp->transId);
+  appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT_TRIGGER, pRsp->checkpointId, pRsp->transId,
+                             pRsp->upstreamTaskId);
   return TSDB_CODE_SUCCESS;
 }
 
@@ -130,6 +133,7 @@ int32_t streamTaskSendCheckpointTriggerMsg(SStreamTask* pTask, int32_t dstTaskId
   pRsp->streamId = pTask->id.streamId;
   pRsp->upstreamTaskId = pTask->id.taskId;
   pRsp->taskId = dstTaskId;
+  pRsp->rspCode = code;
 
   if (code == TSDB_CODE_SUCCESS) {
     pRsp->checkpointId = pTask->chkInfo.pActiveInfo->activeId;
@@ -139,9 +143,7 @@ int32_t streamTaskSendCheckpointTriggerMsg(SStreamTask* pTask, int32_t dstTaskId
     pRsp->transId = -1;
   }
 
-  pRsp->rspCode = code;
+  SRpcMsg rspMsg = {.code = 0, .pCont = pBuf, .contLen = size, .info = *pRpcInfo};
-
-  SRpcMsg rspMsg = {.code = 0, .pCont = pRsp, .contLen = size, .info = *pRpcInfo};
   tmsgSendRsp(&rspMsg);
 
   return 0;
@@ -280,7 +282,7 @@ int32_t streamProcessCheckpointTriggerBlock(SStreamTask* pTask, SStreamDataBlock
       stDebug("s-task:%s set childIdx:%d, and add checkpoint-trigger block into outputQ", id, pTask->info.selfChildId);
       continueDispatchCheckpointTriggerBlock(pBlock, pTask);
     } else {  // only one task exists, no need to dispatch downstream info
-      appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT, pActiveInfo->activeId, pActiveInfo->transId);
+      appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT, pActiveInfo->activeId, pActiveInfo->transId, -1);
       streamFreeQitem((SStreamQueueItem*)pBlock);
     }
   } else if (taskLevel == TASK_LEVEL__SINK || taskLevel == TASK_LEVEL__AGG) {
@@ -377,9 +379,8 @@ int32_t streamProcessCheckpointReadyMsg(SStreamTask* pTask, int64_t checkpointId
   taosThreadMutexUnlock(&pInfo->lock);
 
   if (notReady == 0) {
-    stDebug("s-task:%s all downstream task(s) have completed build checkpoint, start to do checkpoint for current task",
+    stDebug("s-task:%s all downstream tasks have completed build checkpoint, do checkpoint for current task", id);
-            id);
+    appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT, checkpointId, transId, -1);
-    appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT, checkpointId, transId);
   }
 
   return 0;
@@ -459,14 +460,13 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV
 
   SStreamTaskState* pStatus = streamTaskGetStatus(pTask);
 
-  //  if (restored && (pStatus->state != TASK_STATUS__CK)) {
+  if (restored && (pStatus->state != TASK_STATUS__CK) && (pMeta->role == NODE_ROLE_LEADER)) {
-  //    stDebug("s-task:0x%x vgId:%d restored:%d status:%s not update checkpoint-info, checkpointId:%" PRId64 "->%"
+    stDebug("s-task:0x%x vgId:%d restored:%d status:%s not update checkpoint-info, checkpointId:%" PRId64 "->%" PRId64
-  //    PRId64
+            " failed",
-  //            " failed",
+            pReq->taskId, vgId, restored, pStatus->name, pInfo->checkpointId, pReq->checkpointId);
-  //            pReq->taskId, vgId, restored, pStatus->name, pInfo->checkpointId, pReq->checkpointId);
+    taosThreadMutexUnlock(&pTask->lock);
-  //    taosThreadMutexUnlock(&pTask->lock);
+    return TSDB_CODE_STREAM_TASK_IVLD_STATUS;
-  //    return TSDB_CODE_STREAM_TASK_IVLD_STATUS;
+  }
-  //  }
 
   if (!restored) {  // during restore procedure, do update checkpoint-info
     stDebug("s-task:%s vgId:%d status:%s update the checkpoint-info during restore, checkpointId:%" PRId64 "->%" PRId64
@@ -719,11 +719,10 @@ int32_t streamTaskBuildCheckpoint(SStreamTask* pTask) {
     }
   } else {  // clear the checkpoint info if failed
     taosThreadMutexLock(&pTask->lock);
-    streamTaskClearCheckInfo(pTask, false);
+    streamTaskSetFailedCheckpointId(pTask);  // set failed checkpoint id before clear the checkpoint info
     taosThreadMutexUnlock(&pTask->lock);
 
     code = streamTaskHandleEvent(pTask->status.pSM, TASK_EVENT_CHECKPOINT_DONE);
-    streamTaskSetFailedCheckpointId(pTask);
     stDebug("s-task:%s clear checkpoint flag since gen checkpoint failed, checkpointId:%" PRId64, id, ckId);
   }
 
@@ -1158,3 +1157,51 @@ int32_t deleteCheckpointFile(const char* id, const char* name) {
   char* tmp = object;
   return s3DeleteObjects((const char**)&tmp, 1);
 }
+
+int32_t streamTaskSendRestoreChkptMsg(SStreamTask* pTask) {
+  int32_t          code;
+  int32_t          tlen = 0;
+  int32_t          vgId = pTask->pMeta->vgId;
+  const char*      id = pTask->id.idStr;
+  SCheckpointInfo* pInfo = &pTask->chkInfo;
+
+  ASSERT(pTask->pBackend == NULL);
+
+  SRestoreCheckpointInfo req = {
+      .streamId = pTask->id.streamId,
+      .taskId = pTask->id.taskId,
+      .nodeId = vgId,
+      .checkpointId = pInfo->checkpointId,
+      .startTs = pTask->execInfo.created,
+  };
+
+  tEncodeSize(tEncodeRestoreCheckpointInfo, &req, tlen, code);
+  if (code < 0) {
+    stError("s-task:%s vgId:%d encode stream task latest-checkpoint-id failed, code:%s", id, vgId, tstrerror(code));
+    return -1;
+  }
+
+  void* buf = rpcMallocCont(tlen);
+  if (buf == NULL) {
+    stError("s-task:%s vgId:%d encode stream task latest-checkpoint-id msg failed, code:%s", id, vgId,
+            tstrerror(TSDB_CODE_OUT_OF_MEMORY));
+    return -1;
+  }
+
+  SEncoder encoder;
+  tEncoderInit(&encoder, buf, tlen);
+  if ((code = tEncodeRestoreCheckpointInfo(&encoder, &req)) < 0) {
+    rpcFreeCont(buf);
+    stError("s-task:%s vgId:%d encode stream task latest-checkpoint-id msg failed, code:%s", id, vgId, tstrerror(code));
+    return -1;
+  }
+  tEncoderClear(&encoder);
+
+  SRpcMsg msg = {0};
+  initRpcMsg(&msg, TDMT_MND_STREAM_CHKPT_CONSEN, buf, tlen);
+  stDebug("s-task:%s vgId:%d send latest checkpointId:%" PRId64 " to mnode to get the consensus checkpointId", id, vgId,
+          pInfo->checkpointId);
+
+  tmsgSendReq(&pTask->info.mnodeEpset, &msg);
+  return 0;
+}

source/libs/stream/src/streamDispatch.c

@@ -40,6 +40,20 @@ void initRpcMsg(SRpcMsg* pMsg, int32_t msgType, void* pCont, int32_t contLen) {
   pMsg->contLen = contLen;
 }
 
+static void initDispatchInfo(SDispatchMsgInfo* pInfo, int32_t msgId) {
+  pInfo->startTs = taosGetTimestampMs();
+  pInfo->rspTs = -1;
+  pInfo->msgId = msgId;
+}
+
+static void clearDispatchInfo(SDispatchMsgInfo* pInfo) {
+  pInfo->startTs = -1;
+  pInfo->msgId = -1;
+  pInfo->rspTs = -1;
+}
+
+static void updateDispatchInfo(SDispatchMsgInfo* pInfo, int64_t recvTs) { pInfo->rspTs = recvTs; }
+
 static int32_t tInitStreamDispatchReq(SStreamDispatchReq* pReq, const SStreamTask* pTask, int32_t vgId,
                                       int32_t numOfBlocks, int64_t dstTaskId, int32_t type) {
   pReq->streamId = pTask->id.streamId;
@@ -225,12 +239,15 @@ void clearBufferedDispatchMsg(SStreamTask* pTask) {
     destroyDispatchMsg(pMsgInfo->pData, streamTaskGetNumOfDownstream(pTask));
   }
 
+  taosThreadMutexLock(&pMsgInfo->lock);
+
   pMsgInfo->checkpointId = -1;
   pMsgInfo->transId = -1;
   pMsgInfo->pData = NULL;
   pMsgInfo->dispatchMsgType = 0;
 
-  taosThreadMutexLock(&pMsgInfo->lock);
+  clearDispatchInfo(pMsgInfo);
+
   taosArrayClear(pTask->msgInfo.pSendInfo);
   taosThreadMutexUnlock(&pMsgInfo->lock);
 }
@@ -416,6 +433,7 @@ static void setResendInfo(SDispatchEntry* pEntry, int64_t now) {
   pEntry->sendTs = now;
   pEntry->rspTs = -1;
   pEntry->retryCount += 1;
+  pEntry->status = TSDB_CODE_SUCCESS;
 }
 
 static void addDispatchEntry(SDispatchMsgInfo* pMsgInfo, int32_t nodeId, int64_t now, bool lock) {
@@ -642,20 +660,6 @@ int32_t streamSearchAndAddBlock(SStreamTask* pTask, SStreamDispatchReq* pReqs, S
   return 0;
 }
 
-static void initDispatchInfo(SDispatchMsgInfo* pInfo, int32_t msgId) {
-  pInfo->startTs = taosGetTimestampMs();
-  pInfo->rspTs = -1;
-  pInfo->msgId = msgId;
-}
-
-static void clearDispatchInfo(SDispatchMsgInfo* pInfo) {
-  pInfo->startTs = -1;
-  pInfo->msgId = -1;
-  pInfo->rspTs = -1;
-}
-
-static void updateDispatchInfo(SDispatchMsgInfo* pInfo, int64_t recvTs) { pInfo->rspTs = recvTs; }
-
 int32_t streamDispatchStreamBlock(SStreamTask* pTask) {
   ASSERT((pTask->outputInfo.type == TASK_OUTPUT__FIXED_DISPATCH ||
           pTask->outputInfo.type == TASK_OUTPUT__SHUFFLE_DISPATCH));
@@ -698,7 +702,10 @@ int32_t streamDispatchStreamBlock(SStreamTask* pTask) {
          type == STREAM_INPUT__TRANS_STATE);
 
   pTask->execInfo.dispatch += 1;
+
+  taosThreadMutexLock(&pTask->msgInfo.lock);
   initDispatchInfo(&pTask->msgInfo, pTask->execInfo.dispatch);
+  taosThreadMutexUnlock(&pTask->msgInfo.lock);
 
   int32_t code = doBuildDispatchMsg(pTask, pBlock);
   if (code == 0) {
@@ -1221,10 +1228,13 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
   const char*       id = pTask->id.idStr;
   int32_t           vgId = pTask->pMeta->vgId;
   SDispatchMsgInfo* pMsgInfo = &pTask->msgInfo;
-  int32_t           msgId = pMsgInfo->msgId;
   int64_t           now = taosGetTimestampMs();
   int32_t           totalRsp = 0;
 
+  taosThreadMutexLock(&pMsgInfo->lock);
+  int32_t           msgId = pMsgInfo->msgId;
+  taosThreadMutexUnlock(&pMsgInfo->lock);
+
   // follower not handle the dispatch rsp
   if ((pTask->pMeta->role == NODE_ROLE_FOLLOWER) || (pTask->status.downstreamReady != 1)) {
     stError("s-task:%s vgId:%d is follower or task just re-launched, not handle the dispatch rsp, discard it", id,

source/libs/stream/src/streamHb.c

@@ -0,0 +1,342 @@
+/*
+ * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
+ *
+ * This program is free software: you can use, redistribute, and/or modify
+ * it under the terms of the GNU Affero General Public License, version 3
+ * or later ("AGPL"), as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "executor.h"
+#include "streamInt.h"
+#include "tmisce.h"
+#include "tref.h"
+#include "tstream.h"
+#include "ttimer.h"
+#include "wal.h"
+
+int32_t streamMetaId = 0;
+
+struct SMetaHbInfo {
+  tmr_h        hbTmr;
+  int32_t      stopFlag;
+  int32_t      tickCounter;
+  int32_t      hbCount;
+  int64_t      hbStart;
+  int64_t      msgSendTs;
+  SStreamHbMsg hbMsg;
+};
+
+static bool waitForEnoughDuration(SMetaHbInfo* pInfo) {
+  if ((++pInfo->tickCounter) >= META_HB_SEND_IDLE_COUNTER) {  // reset the counter
+    pInfo->tickCounter = 0;
+    return true;
+  }
+  return false;
+}
+
+static bool existInHbMsg(SStreamHbMsg* pMsg, SDownstreamTaskEpset* pTaskEpset) {
+  int32_t numOfExisted = taosArrayGetSize(pMsg->pUpdateNodes);
+  for (int k = 0; k < numOfExisted; ++k) {
+    if (pTaskEpset->nodeId == *(int32_t*)taosArrayGet(pMsg->pUpdateNodes, k)) {
+      return true;
+    }
+  }
+  return false;
+}
+
+static void addUpdateNodeIntoHbMsg(SStreamTask* pTask, SStreamHbMsg* pMsg) {
+  SStreamMeta* pMeta = pTask->pMeta;
+
+  taosThreadMutexLock(&pTask->lock);
+
+  int32_t num = taosArrayGetSize(pTask->outputInfo.pNodeEpsetUpdateList);
+  for (int j = 0; j < num; ++j) {
+    SDownstreamTaskEpset* pTaskEpset = taosArrayGet(pTask->outputInfo.pNodeEpsetUpdateList, j);
+
+    bool exist = existInHbMsg(pMsg, pTaskEpset);
+    if (!exist) {
+      taosArrayPush(pMsg->pUpdateNodes, &pTaskEpset->nodeId);
+      stDebug("vgId:%d nodeId:%d added into hbMsg update list, total:%d", pMeta->vgId, pTaskEpset->nodeId,
+              (int32_t)taosArrayGetSize(pMsg->pUpdateNodes));
+    }
+  }
+
+  taosArrayClear(pTask->outputInfo.pNodeEpsetUpdateList);
+  taosThreadMutexUnlock(&pTask->lock);
+}
+
+static int32_t doSendHbMsgInfo(SStreamHbMsg* pMsg, SStreamMeta* pMeta, SEpSet* pEpset) {
+  int32_t code = 0;
+  int32_t tlen = 0;
+
+  tEncodeSize(tEncodeStreamHbMsg, pMsg, tlen, code);
+  if (code < 0) {
+    stError("vgId:%d encode stream hb msg failed, code:%s", pMeta->vgId, tstrerror(code));
+    return TSDB_CODE_FAILED;
+  }
+
+  void* buf = rpcMallocCont(tlen);
+  if (buf == NULL) {
+    stError("vgId:%d encode stream hb msg failed, code:%s", pMeta->vgId, tstrerror(TSDB_CODE_OUT_OF_MEMORY));
+    return TSDB_CODE_FAILED;
+  }
+
+  SEncoder encoder;
+  tEncoderInit(&encoder, buf, tlen);
+  if ((code = tEncodeStreamHbMsg(&encoder, pMsg)) < 0) {
+    rpcFreeCont(buf);
+    stError("vgId:%d encode stream hb msg failed, code:%s", pMeta->vgId, tstrerror(code));
+    return TSDB_CODE_FAILED;
+  }
+  tEncoderClear(&encoder);
+
+  stDebug("vgId:%d send hb to mnode, numOfTasks:%d msgId:%d", pMeta->vgId, pMsg->numOfTasks, pMsg->msgId);
+
+  SRpcMsg msg = {0};
+  initRpcMsg(&msg, TDMT_MND_STREAM_HEARTBEAT, buf, tlen);
+  tmsgSendReq(pEpset, &msg);
+
+  return TSDB_CODE_SUCCESS;
+}
+
+// NOTE: this task should be executed within the SStreamMeta lock region.
+int32_t streamMetaSendHbHelper(SStreamMeta* pMeta) {
+  SEpSet       epset = {0};
+  bool         hasMnodeEpset = false;
+  int32_t      numOfTasks = streamMetaGetNumOfTasks(pMeta);
+  SMetaHbInfo* pInfo = pMeta->pHbInfo;
+
+  // not recv the hb msg rsp yet, send current hb msg again
+  if (pInfo->msgSendTs > 0) {
+    stDebug("vgId:%d hbMsg rsp not recv, send current hbMsg, msgId:%d, total:%d again", pMeta->vgId, pInfo->hbMsg.msgId,
+            pInfo->hbCount);
+
+    for(int32_t i = 0; i < numOfTasks; ++i) {
+      SStreamTaskId* pId = taosArrayGet(pMeta->pTaskList, i);
+      STaskId       id = {.streamId = pId->streamId, .taskId = pId->taskId};
+      SStreamTask** pTask = taosHashGet(pMeta->pTasksMap, &id, sizeof(id));
+      if (pTask == NULL) {
+        continue;
+      }
+
+      if ((*pTask)->info.fillHistory == 1) {
+        continue;
+      }
+
+      epsetAssign(&epset, &(*pTask)->info.mnodeEpset);
+      break;
+    }
+
+    pInfo->msgSendTs = taosGetTimestampMs();
+    doSendHbMsgInfo(&pInfo->hbMsg, pMeta, &epset);
+    return TSDB_CODE_SUCCESS;
+  }
+
+  SStreamHbMsg* pMsg = &pInfo->hbMsg;
+  stDebug("vgId:%d build stream hbMsg, leader:%d msgId:%d", pMeta->vgId, (pMeta->role == NODE_ROLE_LEADER),
+          pMeta->pHbInfo->hbCount);
+
+  pMsg->vgId = pMeta->vgId;
+  pMsg->msgId = pMeta->pHbInfo->hbCount;
+
+  pMsg->pTaskStatus = taosArrayInit(numOfTasks, sizeof(STaskStatusEntry));
+  pMsg->pUpdateNodes = taosArrayInit(numOfTasks, sizeof(int32_t));
+
+  if (pMsg->pTaskStatus == NULL || pMsg->pUpdateNodes == NULL) {
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return terrno;
+  }
+
+  for (int32_t i = 0; i < numOfTasks; ++i) {
+    SStreamTaskId* pId = taosArrayGet(pMeta->pTaskList, i);
+
+    STaskId       id = {.streamId = pId->streamId, .taskId = pId->taskId};
+    SStreamTask** pTask = taosHashGet(pMeta->pTasksMap, &id, sizeof(id));
+    if (pTask == NULL) {
+      continue;
+    }
+
+    // not report the status of fill-history task
+    if ((*pTask)->info.fillHistory == 1) {
+      continue;
+    }
+
+    STaskStatusEntry entry = streamTaskGetStatusEntry(*pTask);
+
+    entry.inputRate = entry.inputQUsed * 100.0 / (2 * STREAM_TASK_QUEUE_CAPACITY_IN_SIZE);
+    if ((*pTask)->info.taskLevel == TASK_LEVEL__SINK) {
+      entry.sinkQuota = (*pTask)->outputInfo.pTokenBucket->quotaRate;
+      entry.sinkDataSize = SIZE_IN_MiB((*pTask)->execInfo.sink.dataSize);
+    }
+
+    SActiveCheckpointInfo* p = (*pTask)->chkInfo.pActiveInfo;
+    if (p->activeId != 0) {
+      entry.checkpointInfo.failed = (p->failedId >= p->activeId) ? 1 : 0;
+      entry.checkpointInfo.activeId = p->activeId;
+      entry.checkpointInfo.activeTransId = p->transId;
+
+      if (entry.checkpointInfo.failed) {
+        stInfo("s-task:%s set kill checkpoint trans in hbMsg, transId:%d, clear the active checkpointInfo",
+               (*pTask)->id.idStr, p->transId);
+
+        taosThreadMutexLock(&(*pTask)->lock);
+        streamTaskClearCheckInfo((*pTask), true);
+        taosThreadMutexUnlock(&(*pTask)->lock);
+      }
+    }
+
+    if ((*pTask)->exec.pWalReader != NULL) {
+      entry.processedVer = walReaderGetCurrentVer((*pTask)->exec.pWalReader) - 1;
+      if (entry.processedVer < 0) {
+        entry.processedVer = (*pTask)->chkInfo.processedVer;
+      }
+
+      walReaderValidVersionRange((*pTask)->exec.pWalReader, &entry.verRange.minVer, &entry.verRange.maxVer);
+    }
+
+    addUpdateNodeIntoHbMsg(*pTask, pMsg);
+    taosArrayPush(pMsg->pTaskStatus, &entry);
+    if (!hasMnodeEpset) {
+      epsetAssign(&epset, &(*pTask)->info.mnodeEpset);
+      hasMnodeEpset = true;
+    }
+  }
+
+  pMsg->numOfTasks = taosArrayGetSize(pMsg->pTaskStatus);
+
+  if (hasMnodeEpset) {
+    pInfo->msgSendTs = taosGetTimestampMs();
+    doSendHbMsgInfo(pMsg, pMeta, &epset);
+  } else {
+    stDebug("vgId:%d no tasks or no mnd epset, not send stream hb to mnode", pMeta->vgId);
+    tCleanupStreamHbMsg(&pInfo->hbMsg);
+    pInfo->msgSendTs = -1;
+  }
+
+  return TSDB_CODE_SUCCESS;
+}
+
+void streamMetaHbToMnode(void* param, void* tmrId) {
+  int64_t rid = *(int64_t*)param;
+
+  SStreamMeta* pMeta = taosAcquireRef(streamMetaId, rid);
+  if (pMeta == NULL) {
+    stError("invalid rid:%" PRId64 " failed to acquired stream-meta", rid);
+    return;
+  }
+
+  // need to stop, stop now
+  if (pMeta->pHbInfo->stopFlag == STREAM_META_WILL_STOP) {  // todo refactor: not need this now, use closeFlag in Meta
+    pMeta->pHbInfo->stopFlag = STREAM_META_OK_TO_STOP;
+    stDebug("vgId:%d jump out of meta timer", pMeta->vgId);
+    taosReleaseRef(streamMetaId, rid);
+    return;
+  }
+
+  // not leader not send msg
+  if (pMeta->role != NODE_ROLE_LEADER) {
+    stInfo("vgId:%d role:%d not leader not send hb to mnode", pMeta->vgId, pMeta->role);
+    taosReleaseRef(streamMetaId, rid);
+    pMeta->pHbInfo->hbStart = 0;
+    return;
+  }
+
+  // set the hb start time
+  if (pMeta->pHbInfo->hbStart == 0) {
+    pMeta->pHbInfo->hbStart = taosGetTimestampMs();
+  }
+
+  if (!waitForEnoughDuration(pMeta->pHbInfo)) {
+    taosTmrReset(streamMetaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr);
+    taosReleaseRef(streamMetaId, rid);
+    return;
+  }
+
+  streamMetaRLock(pMeta);
+  streamMetaSendHbHelper(pMeta);
+  streamMetaRUnLock(pMeta);
+
+  taosTmrReset(streamMetaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr);
+  taosReleaseRef(streamMetaId, rid);
+}
+
+SMetaHbInfo* createMetaHbInfo(int64_t* pRid) {
+  SMetaHbInfo* pInfo = taosMemoryCalloc(1, sizeof(SMetaHbInfo));
+  if (pInfo == NULL) {
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return pInfo;
+  }
+
+  pInfo->hbTmr = taosTmrStart(streamMetaHbToMnode, META_HB_CHECK_INTERVAL, pRid, streamTimer);
+  pInfo->tickCounter = 0;
+  pInfo->stopFlag = 0;
+  pInfo->msgSendTs = -1;
+  pInfo->hbCount = 0;
+  return pInfo;
+}
+
+void* destroyMetaHbInfo(SMetaHbInfo* pInfo) {
+  if (pInfo != NULL) {
+    tCleanupStreamHbMsg(&pInfo->hbMsg);
+
+    if (pInfo->hbTmr != NULL) {
+      taosTmrStop(pInfo->hbTmr);
+      pInfo->hbTmr = NULL;
+    }
+
+    taosMemoryFree(pInfo);
+  }
+
+  return NULL;
+}
+
+void streamMetaWaitForHbTmrQuit(SStreamMeta* pMeta) {
+  // wait for the stream meta hb function stopping
+  if (pMeta->role == NODE_ROLE_LEADER) {
+    pMeta->pHbInfo->stopFlag = STREAM_META_WILL_STOP;
+    while (pMeta->pHbInfo->stopFlag != STREAM_META_OK_TO_STOP) {
+      taosMsleep(100);
+      stDebug("vgId:%d wait for meta to stop timer", pMeta->vgId);
+    }
+  }
+}
+
+void streamMetaGetHbSendInfo(SMetaHbInfo* pInfo, int64_t* pStartTs, int32_t* pSendCount) {
+  *pStartTs = 0;
+  *pSendCount = 0;
+
+  if (pInfo == NULL) {
+    return;
+  }
+
+  *pStartTs = pInfo->hbStart;
+  *pSendCount = pInfo->hbCount;
+}
+
+int32_t streamProcessHeartbeatRsp(SStreamMeta* pMeta, SMStreamHbRspMsg* pRsp) {
+  stDebug("vgId:%d process hbMsg rsp, msgId:%d rsp confirmed", pMeta->vgId, pRsp->msgId);
+  SMetaHbInfo* pInfo = pMeta->pHbInfo;
+
+  streamMetaRLock(pMeta);
+
+  // current waiting rsp recved
+  if (pRsp->msgId == pInfo->hbCount) {
+    tCleanupStreamHbMsg(&pInfo->hbMsg);
+    stDebug("vgId:%d hbMsg msgId:%d sendTs:%" PRId64 " recved confirmed", pMeta->vgId, pRsp->msgId, pInfo->msgSendTs);
+
+    pInfo->hbCount += 1;
+    pInfo->msgSendTs = -1;
+  } else {
+    stWarn("vgId:%d recv expired hb rsp, msgId:%d, discarded", pMeta->vgId, pRsp->msgId);
+  }
+
+  streamMetaRUnLock(pMeta);
+  return TSDB_CODE_SUCCESS;
+}

source/libs/stream/src/streamMeta.c

@@ -27,10 +27,8 @@ static TdThreadOnce streamMetaModuleInit = PTHREAD_ONCE_INIT;
 
 int32_t streamBackendId = 0;
 int32_t streamBackendCfWrapperId = 0;
-int32_t streamMetaId = 0;
 int32_t taskDbWrapperId = 0;
 
-static void    metaHbToMnode(void* param, void* tmrId);
 static int32_t streamMetaBegin(SStreamMeta* pMeta);
 static void    streamMetaCloseImpl(void* arg);
 
@@ -39,14 +37,6 @@ typedef struct {
   SHashObj*     pTable;
 } SMetaRefMgt;
 
-struct SMetaHbInfo {
-  tmr_h   hbTmr;
-  int32_t stopFlag;
-  int32_t tickCounter;
-  int32_t hbCount;
-  int64_t hbStart;
-};
-
 typedef struct STaskInitTs {
   int64_t start;
   int64_t end;
@@ -357,12 +347,6 @@ SStreamMeta* streamMetaOpen(const char* path, void* ahandle, FTaskBuild buildTas
     goto _err;
   }
 
-  pMeta->pHbInfo = taosMemoryCalloc(1, sizeof(SMetaHbInfo));
-  if (pMeta->pHbInfo == NULL) {
-    terrno = TSDB_CODE_OUT_OF_MEMORY;
-    goto _err;
-  }
-
   // task list
   pMeta->pTaskList = taosArrayInit(4, sizeof(SStreamTaskId));
   if (pMeta->pTaskList == NULL) {
@@ -405,9 +389,12 @@ SStreamMeta* streamMetaOpen(const char* path, void* ahandle, FTaskBuild buildTas
   memcpy(pRid, &pMeta->rid, sizeof(pMeta->rid));
   metaRefMgtAdd(pMeta->vgId, pRid);
 
-  pMeta->pHbInfo->hbTmr = taosTmrStart(metaHbToMnode, META_HB_CHECK_INTERVAL, pRid, streamTimer);
+  pMeta->pHbInfo = createMetaHbInfo(pRid);
-  pMeta->pHbInfo->tickCounter = 0;
+  if (pMeta->pHbInfo == NULL) {
-  pMeta->pHbInfo->stopFlag = 0;
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+
   pMeta->qHandle = taosInitScheduler(32, 1, "stream-chkp", NULL);
 
   pMeta->bkdChkptMgt = bkdMgtCreate(tpath);
@@ -491,7 +478,7 @@ void streamMetaClear(SStreamMeta* pMeta) {
 }
 
 void streamMetaClose(SStreamMeta* pMeta) {
-  stDebug("start to close stream meta");
+  stDebug("vgId:%d start to close stream meta", pMeta->vgId);
   if (pMeta == NULL) {
     return;
   }
@@ -507,11 +494,13 @@ void streamMetaClose(SStreamMeta* pMeta) {
 
 void streamMetaCloseImpl(void* arg) {
   SStreamMeta* pMeta = arg;
-  stDebug("start to do-close stream meta");
   if (pMeta == NULL) {
     return;
   }
 
+  int32_t vgId = pMeta->vgId;
+  stDebug("vgId:%d start to do-close stream meta", vgId);
+
   streamMetaWLock(pMeta);
   streamMetaClear(pMeta);
   streamMetaWUnLock(pMeta);
@@ -531,7 +520,8 @@ void streamMetaCloseImpl(void* arg) {
   taosHashCleanup(pMeta->startInfo.pReadyTaskSet);
   taosHashCleanup(pMeta->startInfo.pFailedTaskSet);
 
-  taosMemoryFree(pMeta->pHbInfo);
+  pMeta->pHbInfo = destroyMetaHbInfo(pMeta->pHbInfo);
+
   taosMemoryFree(pMeta->path);
   taosThreadMutexDestroy(&pMeta->backendMutex);
 
@@ -544,7 +534,7 @@ void streamMetaCloseImpl(void* arg) {
   taosThreadRwlockDestroy(&pMeta->lock);
 
   taosMemoryFree(pMeta);
-  stDebug("end to close stream meta");
+  stDebug("vgId:%d end to close stream meta", vgId);
 }
 
 // todo let's check the status for each task
@@ -906,13 +896,16 @@ void streamMetaLoadAllTasks(SStreamMeta* pMeta) {
       continue;
     }
 
+    stDebug("s-task:0x%" PRIx64 "-0x%x vgId:%d loaded from meta file, checkpointId:%" PRId64 " checkpointVer:%" PRId64,
+            pTask->id.streamId, pTask->id.taskId, vgId, pTask->chkInfo.checkpointId, pTask->chkInfo.checkpointVer);
+
     // do duplicate task check.
     STaskId id = {.streamId = pTask->id.streamId, .taskId = pTask->id.taskId};
     void*   p = taosHashGet(pMeta->pTasksMap, &id, sizeof(id));
     if (p == NULL) {
       code = pMeta->buildTaskFn(pMeta->ahandle, pTask, pTask->chkInfo.checkpointVer + 1);
       if (code < 0) {
-        stError("failed to expand s-task:0x%" PRIx64 ", code:%s, continue", id.taskId, tstrerror(terrno));
+        stError("failed to load s-task:0x%"PRIx64", code:%s, continue", id.taskId, tstrerror(terrno));
         tFreeStreamTask(pTask);
         continue;
       }
@@ -967,215 +960,6 @@ void streamMetaLoadAllTasks(SStreamMeta* pMeta) {
   (void)streamMetaCommit(pMeta);
 }
 
-static bool waitForEnoughDuration(SMetaHbInfo* pInfo) {
-  if ((++pInfo->tickCounter) >= META_HB_SEND_IDLE_COUNTER) {  // reset the counter
-    pInfo->tickCounter = 0;
-    return true;
-  }
-  return false;
-}
-
-static bool existInHbMsg(SStreamHbMsg* pMsg, SDownstreamTaskEpset* pTaskEpset) {
-  int32_t numOfExisted = taosArrayGetSize(pMsg->pUpdateNodes);
-  for (int k = 0; k < numOfExisted; ++k) {
-    if (pTaskEpset->nodeId == *(int32_t*)taosArrayGet(pMsg->pUpdateNodes, k)) {
-      return true;
-    }
-  }
-  return false;
-}
-
-static void addUpdateNodeIntoHbMsg(SStreamTask* pTask, SStreamHbMsg* pMsg) {
-  SStreamMeta* pMeta = pTask->pMeta;
-
-  taosThreadMutexLock(&pTask->lock);
-
-  int32_t num = taosArrayGetSize(pTask->outputInfo.pNodeEpsetUpdateList);
-  for (int j = 0; j < num; ++j) {
-    SDownstreamTaskEpset* pTaskEpset = taosArrayGet(pTask->outputInfo.pNodeEpsetUpdateList, j);
-
-    bool exist = existInHbMsg(pMsg, pTaskEpset);
-    if (!exist) {
-      taosArrayPush(pMsg->pUpdateNodes, &pTaskEpset->nodeId);
-      stDebug("vgId:%d nodeId:%d added into hb update list, total:%d", pMeta->vgId, pTaskEpset->nodeId,
-              (int32_t)taosArrayGetSize(pMsg->pUpdateNodes));
-    }
-  }
-
-  taosArrayClear(pTask->outputInfo.pNodeEpsetUpdateList);
-  taosThreadMutexUnlock(&pTask->lock);
-}
-
-static int32_t metaHeartbeatToMnodeImpl(SStreamMeta* pMeta) {
-  SStreamHbMsg hbMsg = {0};
-  SEpSet       epset = {0};
-  bool         hasMnodeEpset = false;
-  int32_t      numOfTasks = streamMetaGetNumOfTasks(pMeta);
-
-  hbMsg.vgId = pMeta->vgId;
-  hbMsg.pTaskStatus = taosArrayInit(numOfTasks, sizeof(STaskStatusEntry));
-  hbMsg.pUpdateNodes = taosArrayInit(numOfTasks, sizeof(int32_t));
-
-  for (int32_t i = 0; i < numOfTasks; ++i) {
-    SStreamTaskId* pId = taosArrayGet(pMeta->pTaskList, i);
-
-    STaskId       id = {.streamId = pId->streamId, .taskId = pId->taskId};
-    SStreamTask** pTask = taosHashGet(pMeta->pTasksMap, &id, sizeof(id));
-    if (pTask == NULL) {
-      continue;
-    }
-
-    // not report the status of fill-history task
-    if ((*pTask)->info.fillHistory == 1) {
-      continue;
-    }
-
-    STaskStatusEntry entry = {
-        .id = id,
-        .status = streamTaskGetStatus(*pTask)->state,
-        .nodeId = hbMsg.vgId,
-        .stage = pMeta->stage,
-
-        .inputQUsed = SIZE_IN_MiB(streamQueueGetItemSize((*pTask)->inputq.queue)),
-        .startTime = (*pTask)->execInfo.readyTs,
-        .checkpointInfo.latestId = (*pTask)->chkInfo.checkpointId,
-        .checkpointInfo.latestVer = (*pTask)->chkInfo.checkpointVer,
-        .checkpointInfo.latestTime = (*pTask)->chkInfo.checkpointTime,
-        .checkpointInfo.latestSize = 0,
-        .checkpointInfo.remoteBackup = 0,
-        .hTaskId = (*pTask)->hTaskInfo.id.taskId,
-        .procsTotal = SIZE_IN_MiB((*pTask)->execInfo.inputDataSize),
-        .outputTotal = SIZE_IN_MiB((*pTask)->execInfo.outputDataSize),
-        .procsThroughput = SIZE_IN_KiB((*pTask)->execInfo.procsThroughput),
-        .outputThroughput = SIZE_IN_KiB((*pTask)->execInfo.outputThroughput),
-        .startCheckpointId = (*pTask)->execInfo.startCheckpointId,
-        .startCheckpointVer = (*pTask)->execInfo.startCheckpointVer,
-    };
-
-    entry.inputRate = entry.inputQUsed * 100.0 / (2 * STREAM_TASK_QUEUE_CAPACITY_IN_SIZE);
-    if ((*pTask)->info.taskLevel == TASK_LEVEL__SINK) {
-      entry.sinkQuota = (*pTask)->outputInfo.pTokenBucket->quotaRate;
-      entry.sinkDataSize = SIZE_IN_MiB((*pTask)->execInfo.sink.dataSize);
-    }
-
-    if ((*pTask)->chkInfo.pActiveInfo->activeId != 0) {
-      entry.checkpointInfo.failed =
-          ((*pTask)->chkInfo.pActiveInfo->failedId >= (*pTask)->chkInfo.pActiveInfo->activeId) ? 1 : 0;
-      entry.checkpointInfo.activeId = (*pTask)->chkInfo.pActiveInfo->activeId;
-      entry.checkpointInfo.activeTransId = (*pTask)->chkInfo.pActiveInfo->transId;
-
-      if (entry.checkpointInfo.failed) {
-        stInfo("s-task:%s set kill checkpoint trans in hb, transId:%d", (*pTask)->id.idStr,
-               (*pTask)->chkInfo.pActiveInfo->transId);
-      }
-    }
-
-    if ((*pTask)->exec.pWalReader != NULL) {
-      entry.processedVer = walReaderGetCurrentVer((*pTask)->exec.pWalReader) - 1;
-      if (entry.processedVer < 0) {
-        entry.processedVer = (*pTask)->chkInfo.processedVer;
-      }
-
-      walReaderValidVersionRange((*pTask)->exec.pWalReader, &entry.verRange.minVer, &entry.verRange.maxVer);
-    }
-
-    addUpdateNodeIntoHbMsg(*pTask, &hbMsg);
-    taosArrayPush(hbMsg.pTaskStatus, &entry);
-    if (!hasMnodeEpset) {
-      epsetAssign(&epset, &(*pTask)->info.mnodeEpset);
-      hasMnodeEpset = true;
-    }
-  }
-
-  hbMsg.numOfTasks = taosArrayGetSize(hbMsg.pTaskStatus);
-
-  if (hasMnodeEpset) {
-    int32_t code = 0;
-    int32_t tlen = 0;
-
-    tEncodeSize(tEncodeStreamHbMsg, &hbMsg, tlen, code);
-    if (code < 0) {
-      stError("vgId:%d encode stream hb msg failed, code:%s", pMeta->vgId, tstrerror(code));
-      goto _end;
-    }
-
-    void* buf = rpcMallocCont(tlen);
-    if (buf == NULL) {
-      stError("vgId:%d encode stream hb msg failed, code:%s", pMeta->vgId, tstrerror(TSDB_CODE_OUT_OF_MEMORY));
-      goto _end;
-    }
-
-    SEncoder encoder;
-    tEncoderInit(&encoder, buf, tlen);
-    if ((code = tEncodeStreamHbMsg(&encoder, &hbMsg)) < 0) {
-      rpcFreeCont(buf);
-      stError("vgId:%d encode stream hb msg failed, code:%s", pMeta->vgId, tstrerror(code));
-      goto _end;
-    }
-    tEncoderClear(&encoder);
-
-    SRpcMsg msg = {0};
-    initRpcMsg(&msg, TDMT_MND_STREAM_HEARTBEAT, buf, tlen);
-
-    pMeta->pHbInfo->hbCount += 1;
-    stDebug("vgId:%d build and send hb to mnode, numOfTasks:%d total:%d", pMeta->vgId, hbMsg.numOfTasks,
-            pMeta->pHbInfo->hbCount);
-
-    tmsgSendReq(&epset, &msg);
-  } else {
-    stDebug("vgId:%d no tasks and no mnd epset, not send stream hb to mnode", pMeta->vgId);
-  }
-
-_end:
-  tCleanupStreamHbMsg(&hbMsg);
-  return TSDB_CODE_SUCCESS;
-}
-
-void metaHbToMnode(void* param, void* tmrId) {
-  int64_t rid = *(int64_t*)param;
-
-  SStreamMeta* pMeta = taosAcquireRef(streamMetaId, rid);
-  if (pMeta == NULL) {
-    stError("invalid rid:%" PRId64 " failed to acquired stream-meta", rid);
-    return;
-  }
-
-  // need to stop, stop now
-  if (pMeta->pHbInfo->stopFlag == STREAM_META_WILL_STOP) {  // todo refactor: not need this now, use closeFlag in Meta
-    pMeta->pHbInfo->stopFlag = STREAM_META_OK_TO_STOP;
-    stDebug("vgId:%d jump out of meta timer", pMeta->vgId);
-    taosReleaseRef(streamMetaId, rid);
-    return;
-  }
-
-  // not leader not send msg
-  if (pMeta->role != NODE_ROLE_LEADER) {
-    stInfo("vgId:%d role:%d not leader not send hb to mnode", pMeta->vgId, pMeta->role);
-    taosReleaseRef(streamMetaId, rid);
-    pMeta->pHbInfo->hbStart = 0;
-    return;
-  }
-
-  // set the hb start time
-  if (pMeta->pHbInfo->hbStart == 0) {
-    pMeta->pHbInfo->hbStart = taosGetTimestampMs();
-  }
-
-  if (!waitForEnoughDuration(pMeta->pHbInfo)) {
-    taosTmrReset(metaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr);
-    taosReleaseRef(streamMetaId, rid);
-    return;
-  }
-
-  stDebug("vgId:%d build stream task hb, leader:%d", pMeta->vgId, (pMeta->role == NODE_ROLE_LEADER));
-  streamMetaRLock(pMeta);
-  metaHeartbeatToMnodeImpl(pMeta);
-  streamMetaRUnLock(pMeta);
-
-  taosTmrReset(metaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr);
-  taosReleaseRef(streamMetaId, rid);
-}
-
 bool streamMetaTaskInTimer(SStreamMeta* pMeta) {
   bool inTimer = false;
   streamMetaRLock(pMeta);
@@ -1201,14 +985,19 @@ bool streamMetaTaskInTimer(SStreamMeta* pMeta) {
 
 void streamMetaNotifyClose(SStreamMeta* pMeta) {
   int32_t vgId = pMeta->vgId;
+  int64_t startTs = 0;
+  int32_t sendCount = 0;
+  streamMetaGetHbSendInfo(pMeta->pHbInfo, &startTs, &sendCount);
 
-  stDebug("vgId:%d notify all stream tasks that the vnode is closing. isLeader:%d startHb:%" PRId64 ", totalHb:%d",
+  stInfo("vgId:%d notify all stream tasks that current vnode is closing. isLeader:%d startHb:%" PRId64 ", totalHb:%d",
-          vgId, (pMeta->role == NODE_ROLE_LEADER), pMeta->pHbInfo->hbStart, pMeta->pHbInfo->hbCount);
+          vgId, (pMeta->role == NODE_ROLE_LEADER), startTs, sendCount);
+
+  // wait for the stream meta hb function stopping
+  streamMetaWaitForHbTmrQuit(pMeta);
 
   streamMetaWLock(pMeta);
 
   pMeta->closeFlag = true;
-
   void* pIter = NULL;
   while (1) {
     pIter = taosHashIterate(pMeta->pTasksMap, pIter);
@@ -1223,15 +1012,6 @@ void streamMetaNotifyClose(SStreamMeta* pMeta) {
 
   streamMetaWUnLock(pMeta);
 
-  // wait for the stream meta hb function stopping
-  if (pMeta->role == NODE_ROLE_LEADER) {
-    pMeta->pHbInfo->stopFlag = STREAM_META_WILL_STOP;
-    while (pMeta->pHbInfo->stopFlag != STREAM_META_OK_TO_STOP) {
-      taosMsleep(100);
-      stDebug("vgId:%d wait for meta to stop timer", pMeta->vgId);
-    }
-  }
-
   stDebug("vgId:%d start to check all tasks for closing", vgId);
   int64_t st = taosGetTimestampMs();
 
@@ -1248,10 +1028,10 @@ void streamMetaStartHb(SStreamMeta* pMeta) {
   int64_t* pRid = taosMemoryMalloc(sizeof(int64_t));
   metaRefMgtAdd(pMeta->vgId, pRid);
   *pRid = pMeta->rid;
-  metaHbToMnode(pRid, NULL);
+  streamMetaHbToMnode(pRid, NULL);
 }
 
-void streamMetaResetStartInfo(STaskStartInfo* pStartInfo) {
+void streamMetaResetStartInfo(STaskStartInfo* pStartInfo, int32_t vgId) {
   taosHashClear(pStartInfo->pReadyTaskSet);
   taosHashClear(pStartInfo->pFailedTaskSet);
   pStartInfo->tasksWillRestart = 0;
@@ -1259,6 +1039,7 @@ void streamMetaResetStartInfo(STaskStartInfo* pStartInfo) {
 
   // reset the sentinel flag value to be 0
   pStartInfo->startAllTasks = 0;
+  stDebug("vgId:%d clear all start-all-task info", vgId);
 }
 
 void streamMetaRLock(SStreamMeta* pMeta) {
@@ -1337,7 +1118,7 @@ SArray* streamMetaSendMsgBeforeCloseTasks(SStreamMeta* pMeta) {
     streamMetaReleaseTask(pMeta, pTask);
   }
 
-  metaHeartbeatToMnodeImpl(pMeta);
+  streamMetaSendHbHelper(pMeta);
   pMeta->sendMsgBeforeClosing = false;
   return pTaskList;
 }
@@ -1387,16 +1168,17 @@ static int32_t prepareBeforeStartTasks(SStreamMeta* pMeta, SArray** pList, int64
   return TSDB_CODE_SUCCESS;
 }
 
-int32_t streamMetaStartAllTasks(SStreamMeta* pMeta, __stream_task_expand_fn expandFn) {
+// restore the checkpoint id by negotiating the latest consensus checkpoint id
+int32_t streamMetaStartAllTasks(SStreamMeta* pMeta) {
   int32_t code = TSDB_CODE_SUCCESS;
   int32_t vgId = pMeta->vgId;
   int64_t now = taosGetTimestampMs();
 
   int32_t numOfTasks = taosArrayGetSize(pMeta->pTaskList);
-  stInfo("vgId:%d start to check all %d stream task(s) downstream status, start ts:%" PRId64, vgId, numOfTasks, now);
+  stInfo("vgId:%d start to consensus checkpointId for all %d task(s), start ts:%"PRId64, vgId, numOfTasks, now);
 
   if (numOfTasks == 0) {
-    stInfo("vgId:%d no tasks to be started", pMeta->vgId);
+    stInfo("vgId:%d no tasks exist, quit from consensus checkpointId", pMeta->vgId);
     return TSDB_CODE_SUCCESS;
   }
 
@@ -1407,11 +1189,11 @@ int32_t streamMetaStartAllTasks(SStreamMeta* pMeta, __stream_task_expand_fn expa
     return TSDB_CODE_SUCCESS;
   }
 
-  // broadcast the check downstream tasks msg
+  // broadcast the check downstream tasks msg only for tasks with related fill-history tasks.
   numOfTasks = taosArrayGetSize(pTaskList);
 
   // prepare the fill-history task before starting all stream tasks, to avoid fill-history tasks are started without
-  // initialization , when the operation of check downstream tasks status is executed far quickly.
+  // initialization, when the operation of check downstream tasks status is executed far quickly.
   for (int32_t i = 0; i < numOfTasks; ++i) {
     SStreamTaskId* pTaskId = taosArrayGet(pTaskList, i);
     SStreamTask*   pTask = streamMetaAcquireTask(pMeta, pTaskId->streamId, pTaskId->taskId);
@@ -1421,19 +1203,18 @@ int32_t streamMetaStartAllTasks(SStreamMeta* pMeta, __stream_task_expand_fn expa
       continue;
     }
 
-    if (pTask->pBackend == NULL) {  // TODO: add test cases for this
+    if ((pTask->pBackend == NULL) && (pTask->info.fillHistory == 1 || HAS_RELATED_FILLHISTORY_TASK(pTask))) {
-      code = expandFn(pTask);
+      code = pMeta->expandTaskFn(pTask);
       if (code != TSDB_CODE_SUCCESS) {
         stError("s-task:0x%x vgId:%d failed to expand stream backend", pTaskId->taskId, vgId);
         streamMetaAddFailedTaskSelf(pTask, pTask->execInfo.readyTs);
       }
-    } else {
-      stDebug("s-task:0x%x vgId:%d fill-history task backend has initialized already", pTaskId->taskId, vgId);
     }
 
     streamMetaReleaseTask(pMeta, pTask);
   }
 
+  // Tasks, with related fill-history task or without any checkpoint yet, can be started directly here.
   for (int32_t i = 0; i < numOfTasks; ++i) {
     SStreamTaskId* pTaskId = taosArrayGet(pTaskList, i);
 
@@ -1466,16 +1247,28 @@ int32_t streamMetaStartAllTasks(SStreamMeta* pMeta, __stream_task_expand_fn expa
       continue;
     }
 
-    int32_t ret = streamTaskHandleEvent(pTask->status.pSM, TASK_EVENT_INIT);
+    if (HAS_RELATED_FILLHISTORY_TASK(pTask)) {
-    if (ret != TSDB_CODE_SUCCESS) {
+      int32_t ret = streamTaskHandleEvent(pTask->status.pSM, TASK_EVENT_INIT);
-      stError("vgId:%d failed to handle event:%d", pMeta->vgId, TASK_EVENT_INIT);
+      if (ret != TSDB_CODE_SUCCESS) {
-      code = ret;
+        stError("vgId:%d failed to handle event:%d", pMeta->vgId, TASK_EVENT_INIT);
-      streamMetaAddFailedTaskSelf(pTask, pInfo->readyTs);
+        code = ret;
+        streamMetaAddFailedTaskSelf(pTask, pInfo->readyTs);
+      }
+
+      streamMetaReleaseTask(pMeta, pTask);
+      continue;
     }
 
+    // negotiate the consensus checkpoint id for current task
+    ASSERT(pTask->pBackend == NULL);
+    code = streamTaskSendRestoreChkptMsg(pTask);
+
+    // this task may has no checkpoint, but others tasks may generate checkpoint already?
     streamMetaReleaseTask(pMeta, pTask);
   }
 
+  // prepare the fill-history task before starting all stream tasks, to avoid fill-history tasks are started without
+  // initialization, when the operation of check downstream tasks status is executed far quickly.
   stInfo("vgId:%d start all task(s) completed", pMeta->vgId);
   taosArrayDestroy(pTaskList);
   return code;
@@ -1536,7 +1329,7 @@ bool streamMetaAllTasksReady(const SStreamMeta* pMeta) {
   return true;
 }
 
-int32_t streamMetaStartOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, __stream_task_expand_fn expandFn) {
+int32_t streamMetaStartOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId) {
   int32_t code = 0;
   int32_t vgId = pMeta->vgId;
   stInfo("vgId:%d start task:0x%x by checking it's downstream status", vgId, taskId);
@@ -1557,7 +1350,7 @@ int32_t streamMetaStartOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t tas
 
   ASSERT(pTask->status.downstreamReady == 0);
   if (pTask->pBackend == NULL) {
-    code = expandFn(pTask);
+    code = pMeta->expandTaskFn(pTask);
     if (code != TSDB_CODE_SUCCESS) {
       streamMetaAddFailedTaskSelf(pTask, pInfo->readyTs);
     }
@@ -1639,7 +1432,7 @@ int32_t streamMetaAddTaskLaunchResult(SStreamMeta* pMeta, int64_t streamId, int3
     // print the initialization elapsed time and info
     displayStatusInfo(pMeta, pStartInfo->pReadyTaskSet, true);
     displayStatusInfo(pMeta, pStartInfo->pFailedTaskSet, false);
-    streamMetaResetStartInfo(pStartInfo);
+    streamMetaResetStartInfo(pStartInfo, pMeta->vgId);
     streamMetaWUnLock(pMeta);
 
     pStartInfo->completeFn(pMeta);

source/libs/stream/src/streamMsg.c

@@ -13,7 +13,7 @@
  * along with this program. If not, see <http://www.gnu.org/licenses/>.
  */
 
-#include "streammsg.h"
+#include "streamMsg.h"
 #include "os.h"
 #include "tstream.h"
 
@@ -363,6 +363,7 @@ int32_t tEncodeStreamHbMsg(SEncoder* pEncoder, const SStreamHbMsg* pReq) {
     if (tEncodeI32(pEncoder, *pVgId) < 0) return -1;
   }
 
+  if (tEncodeI32(pEncoder, pReq->msgId) < 0) return -1;
   tEndEncode(pEncoder);
   return pEncoder->pos;
 }
@@ -422,6 +423,7 @@ int32_t tDecodeStreamHbMsg(SDecoder* pDecoder, SStreamHbMsg* pReq) {
     taosArrayPush(pReq->pUpdateNodes, &vgId);
   }
 
+  if (tDecodeI32(pDecoder, &pReq->msgId) < 0) return -1;
   tEndDecode(pDecoder);
   return 0;
 }
@@ -432,12 +434,16 @@ void tCleanupStreamHbMsg(SStreamHbMsg* pMsg) {
   }
 
   if (pMsg->pUpdateNodes != NULL) {
-    taosArrayDestroy(pMsg->pUpdateNodes);
+    pMsg->pUpdateNodes = taosArrayDestroy(pMsg->pUpdateNodes);
   }
 
   if (pMsg->pTaskStatus != NULL) {
-    taosArrayDestroy(pMsg->pTaskStatus);
+    pMsg->pTaskStatus = taosArrayDestroy(pMsg->pTaskStatus);
   }
+
+  pMsg->msgId = -1;
+  pMsg->vgId = -1;
+  pMsg->numOfTasks = -1;
 }
 
 int32_t tEncodeStreamTask(SEncoder* pEncoder, const SStreamTask* pTask) {
@@ -622,3 +628,45 @@ int32_t tDecodeStreamTaskChkptReport(SDecoder* pDecoder, SCheckpointReport* pReq
   tEndDecode(pDecoder);
   return 0;
 }
+
+int32_t tEncodeRestoreCheckpointInfo (SEncoder* pEncoder, const SRestoreCheckpointInfo* pReq) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->startTs) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->checkpointId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->taskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->nodeId) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeRestoreCheckpointInfo(SDecoder* pDecoder, SRestoreCheckpointInfo* pReq) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->startTs) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->checkpointId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->taskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->nodeId) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
+int32_t tEncodeRestoreCheckpointInfoRsp(SEncoder* pCoder, const SRestoreCheckpointInfoRsp* pInfo) {
+  if (tStartEncode(pCoder) < 0) return -1;
+  if (tEncodeI64(pCoder, pInfo->startTs) < 0) return -1;
+  if (tEncodeI64(pCoder, pInfo->streamId) < 0) return -1;
+  if (tEncodeI32(pCoder, pInfo->taskId) < 0) return -1;
+  if (tEncodeI64(pCoder, pInfo->checkpointId) < 0) return -1;
+  tEndEncode(pCoder);
+  return 0;
+}
+
+int32_t tDecodeRestoreCheckpointInfoRsp(SDecoder* pCoder, SRestoreCheckpointInfoRsp* pInfo) {
+  if (tStartDecode(pCoder) < 0) return -1;
+  if (tDecodeI64(pCoder, &pInfo->startTs) < 0) return -1;
+  if (tDecodeI64(pCoder, &pInfo->streamId) < 0) return -1;
+  if (tDecodeI32(pCoder, &pInfo->taskId) < 0) return -1;
+  if (tDecodeI64(pCoder, &pInfo->checkpointId) < 0) return -1;
+  tEndDecode(pCoder);
+  return 0;
+}

source/libs/stream/src/streamTask.c

@@ -273,6 +273,7 @@ void tFreeStreamTask(SStreamTask* pTask) {
     stDebug("s-task:0x%x start to free task state", taskId);
     streamStateClose(pTask->pState, status1 == TASK_STATUS__DROPPING);
     taskDbRemoveRef(pTask->pBackend);
+    pTask->pBackend = NULL;
   }
 
   if (pTask->pNameMap) {
@@ -830,6 +831,34 @@ void streamTaskStatusCopy(STaskStatusEntry* pDst, const STaskStatusEntry* pSrc)
   pDst->hTaskId = pSrc->hTaskId;
 }
 
+STaskStatusEntry streamTaskGetStatusEntry(SStreamTask* pTask) {
+  SStreamMeta*         pMeta = pTask->pMeta;
+  STaskExecStatisInfo* pExecInfo = &pTask->execInfo;
+
+  STaskStatusEntry entry = {
+      .id = streamTaskGetTaskId(pTask),
+      .status = streamTaskGetStatus(pTask)->state,
+      .nodeId = pMeta->vgId,
+      .stage = pMeta->stage,
+
+      .inputQUsed = SIZE_IN_MiB(streamQueueGetItemSize(pTask->inputq.queue)),
+      .startTime = pExecInfo->readyTs,
+      .checkpointInfo.latestId = pTask->chkInfo.checkpointId,
+      .checkpointInfo.latestVer = pTask->chkInfo.checkpointVer,
+      .checkpointInfo.latestTime = pTask->chkInfo.checkpointTime,
+      .checkpointInfo.latestSize = 0,
+      .checkpointInfo.remoteBackup = 0,
+      .hTaskId = pTask->hTaskInfo.id.taskId,
+      .procsTotal = SIZE_IN_MiB(pExecInfo->inputDataSize),
+      .outputTotal = SIZE_IN_MiB(pExecInfo->outputDataSize),
+      .procsThroughput = SIZE_IN_KiB(pExecInfo->procsThroughput),
+      .outputThroughput = SIZE_IN_KiB(pExecInfo->outputThroughput),
+      .startCheckpointId = pExecInfo->startCheckpointId,
+      .startCheckpointVer = pExecInfo->startCheckpointVer,
+  };
+  return entry;
+}
+
 static int32_t taskPauseCallback(SStreamTask* pTask, void* param) {
   SStreamMeta* pMeta = pTask->pMeta;
 
@@ -1039,10 +1068,10 @@ void streamTaskDestroyActiveChkptInfo(SActiveCheckpointInfo* pInfo) {
 
 void streamTaskClearActiveInfo(SActiveCheckpointInfo* pInfo) {
   pInfo->activeId = 0;  // clear the checkpoint id
-  pInfo->failedId = 0;
   pInfo->transId = 0;
   pInfo->allUpstreamTriggerRecv = 0;
   pInfo->dispatchTrigger = false;
+  pInfo->failedId = 0;
 
   taosArrayClear(pInfo->pDispatchTriggerList);
   taosArrayClear(pInfo->pCheckpointReadyRecvList);

source/libs/stream/src/streamTaskSm.c

@@ -206,15 +206,15 @@ static int32_t removeEventInWaitingList(SStreamTask* pTask, EStreamTaskEvent eve
     SFutureHandleEventInfo* pInfo = taosArrayGet(pSM->pWaitingEventList, i);
     if (pInfo->event == event) {
       taosArrayRemove(pSM->pWaitingEventList, i);
-      stDebug("s-task:%s pause event in waiting list not be handled yet, remove it from waiting list, remaining:%d",
+      stDebug("s-task:%s %s event in waiting list not be handled yet, remove it from waiting list, remaining events:%d",
-              pTask->id.idStr, pInfo->event);
+              pTask->id.idStr, GET_EVT_NAME(pInfo->event), num - 1);
       removed = true;
       break;
     }
   }
 
   if (!removed) {
-    stDebug("s-task:%s failed to remove event:%s in waiting list", pTask->id.idStr, StreamTaskEventList[event].name);
+    stDebug("s-task:%s failed to remove event:%s in waiting list", pTask->id.idStr, GET_EVT_NAME(event));
   }
 
   return TSDB_CODE_SUCCESS;

source/util/src/terror.c

@@ -303,6 +303,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_MND_TRANS_NETWORK_UNAVAILL,   "Unable to establish c
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_LAST_TRANS_NOT_FINISHED,  "Last Transaction not finished")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_TRANS_SYNC_TIMEOUT,       "Sync timeout While execute transaction and will continue in the background")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_TRANS_CTX_SWITCH,         "Transaction context switch")
+TAOS_DEFINE_ERROR(TSDB_CODE_MND_TRANS_CONFLICT_COMPACT,   "Transaction not completed due to conflict with compact")
 TAOS_DEFINE_ERROR(TSDB_CODE_MND_TRANS_UNKNOW_ERROR,       "Unknown transaction error")
 
 // mnode-mq

source/util/src/tworker.c

@@ -433,28 +433,66 @@ void tWWorkerFreeQueue(SWWorkerPool *pool, STaosQueue *queue) {
 }
 
 int32_t tSingleWorkerInit(SSingleWorker *pWorker, const SSingleWorkerCfg *pCfg) {
-  SQWorkerPool *pPool = &pWorker->pool;
+  pWorker->poolType = pCfg->poolType;
-  pPool->name = pCfg->name;
-  pPool->min = pCfg->min;
-  pPool->max = pCfg->max;
-  if (tQWorkerInit(pPool) != 0) return -1;
-
-  pWorker->queue = tQWorkerAllocQueue(pPool, pCfg->param, pCfg->fp);
-  if (pWorker->queue == NULL) return -1;
-
   pWorker->name = pCfg->name;
+
+  switch (pCfg->poolType) {
+    case QWORKER_POOL: {
+      SQWorkerPool *pPool = taosMemoryCalloc(1, sizeof(SQWorkerPool));
+      if (!pPool) {
+        terrno = TSDB_CODE_OUT_OF_MEMORY;
+        return -1;
+      }
+      pPool->name = pCfg->name;
+      pPool->min = pCfg->min;
+      pPool->max = pCfg->max;
+      pWorker->pool = pPool;
+      if (tQWorkerInit(pPool) != 0) return -1;
+
+      pWorker->queue = tQWorkerAllocQueue(pPool, pCfg->param, pCfg->fp);
+      if (pWorker->queue == NULL) return -1;
+    } break;
+    case QUERY_AUTO_QWORKER_POOL: {
+      SQueryAutoQWorkerPool *pPool = taosMemoryCalloc(1, sizeof(SQueryAutoQWorkerPool));
+      if (!pPool) {
+        terrno = TSDB_CODE_OUT_OF_MEMORY;
+        return -1;
+      }
+      pPool->name = pCfg->name;
+      pPool->min = pCfg->min;
+      pPool->max = pCfg->max;
+      pWorker->pool = pPool;
+      if (tQueryAutoQWorkerInit(pPool) != 0) return -1;
+
+      pWorker->queue = tQueryAutoQWorkerAllocQueue(pPool, pCfg->param, pCfg->fp);
+      if (!pWorker->queue) return -1;
+    } break;
+    default:
+      assert(0);
+  }
   return 0;
 }
 
 void tSingleWorkerCleanup(SSingleWorker *pWorker) {
   if (pWorker->queue == NULL) return;
-
   while (!taosQueueEmpty(pWorker->queue)) {
     taosMsleep(10);
   }
 
-  tQWorkerCleanup(&pWorker->pool);
+  switch (pWorker->poolType) {
-  tQWorkerFreeQueue(&pWorker->pool, pWorker->queue);
+    case QWORKER_POOL:
+      tQWorkerCleanup(pWorker->pool);
+      tQWorkerFreeQueue(pWorker->pool, pWorker->queue);
+      taosMemoryFree(pWorker->pool);
+      break;
+    case QUERY_AUTO_QWORKER_POOL:
+      tQueryAutoQWorkerCleanup(pWorker->pool);
+      tQueryAutoQWorkerFreeQueue(pWorker->pool, pWorker->queue);
+      taosMemoryFree(pWorker->pool);
+      break;
+    default:
+      assert(0);
+  }
 }
 
 int32_t tMultiWorkerInit(SMultiWorker *pWorker, const SMultiWorkerCfg *pCfg) {
@@ -480,3 +518,472 @@ void tMultiWorkerCleanup(SMultiWorker *pWorker) {
   tWWorkerCleanup(&pWorker->pool);
   tWWorkerFreeQueue(&pWorker->pool, pWorker->queue);
 }
+
+static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool* pool);
+static int32_t tQueryAutoQWorkerBeforeBlocking(void *p);
+static int32_t tQueryAutoQWorkerRecoverFromBlocking(void *p);
+static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool* pPool);
+static bool    tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool* pPool, SQueryAutoQWorker* pWorker);
+
+#define GET_ACTIVE_N(int64_val) (int32_t)((int64_val) >> 32)
+#define GET_RUNNING_N(int64_val) (int32_t)(int64_val & 0xFFFFFFFF)
+
+static int32_t atomicFetchSubActive(int64_t *ptr, int32_t val) {
+  int64_t acutalSubVal = val;
+  acutalSubVal <<= 32;
+  int64_t newVal64 = atomic_fetch_sub_64(ptr, acutalSubVal);
+  return GET_ACTIVE_N(newVal64);
+}
+
+static int32_t atomicFetchSubRunning(int64_t *ptr, int32_t val) { return GET_RUNNING_N(atomic_fetch_sub_64(ptr, val)); }
+
+static int32_t atomicFetchAddActive(int64_t *ptr, int32_t val) {
+  int64_t actualAddVal = val;
+  actualAddVal <<= 32;
+  int64_t newVal64 = atomic_fetch_add_64(ptr, actualAddVal);
+  return GET_ACTIVE_N(newVal64);
+}
+
+static int32_t atomicFetchAddRunning(int64_t *ptr, int32_t val) { return GET_RUNNING_N(atomic_fetch_add_64(ptr, val)); }
+
+static bool atomicCompareExchangeActive(int64_t* ptr, int32_t* expectedVal, int32_t newVal) {
+  int64_t oldVal64 = *expectedVal, newVal64 = newVal;
+  int32_t running = GET_RUNNING_N(*ptr);
+  oldVal64 <<= 32;
+  newVal64 <<= 32;
+  oldVal64 |= running;
+  newVal64 |= running;
+  int64_t actualNewVal64 = atomic_val_compare_exchange_64(ptr, oldVal64, newVal64);
+  if (actualNewVal64 == oldVal64) {
+    return true;
+  } else {
+    *expectedVal = GET_ACTIVE_N(actualNewVal64);
+    return false;
+  }
+}
+
+static int64_t atomicCompareExchangeRunning(int64_t* ptr, int32_t* expectedVal, int32_t newVal) {
+  int64_t oldVal64 = *expectedVal, newVal64 = newVal;
+  int64_t activeShifted = GET_ACTIVE_N(*ptr);
+  activeShifted <<= 32;
+  oldVal64 |= activeShifted;
+  newVal64 |= activeShifted;
+  int64_t actualNewVal64 = atomic_val_compare_exchange_64(ptr, oldVal64, newVal64);
+  if (actualNewVal64 == oldVal64) {
+    return true;
+  } else {
+    *expectedVal = GET_RUNNING_N(actualNewVal64);
+    return false;
+  }
+}
+
+static int64_t atomicComapreAndExchangeActiveAndRunning(int64_t *ptr, int32_t *expectedActive, int32_t newActive,
+                                                        int32_t *expectedRunning, int32_t newRunning) {
+  int64_t oldVal64 = *expectedActive, newVal64 = newActive;
+  oldVal64 <<= 32;
+  oldVal64 |= *expectedRunning;
+  newVal64 <<= 32;
+  newVal64 |= newRunning;
+  int64_t actualNewVal64 = atomic_val_compare_exchange_64(ptr, oldVal64, newVal64);
+  if (actualNewVal64 == oldVal64) {
+    return true;
+  } else {
+    *expectedActive = GET_ACTIVE_N(actualNewVal64);
+    *expectedRunning = GET_RUNNING_N(actualNewVal64);
+    return false;
+  }
+}
+
+static void *tQueryAutoQWorkerThreadFp(SQueryAutoQWorker *worker) {
+  SQueryAutoQWorkerPool *pool = worker->pool;
+  SQueueInfo             qinfo = {0};
+  void                  *msg = NULL;
+  int32_t                code = 0;
+
+  taosBlockSIGPIPE();
+  setThreadName(pool->name);
+  worker->pid = taosGetSelfPthreadId();
+  uDebug("worker:%s:%d is running, thread:%08" PRId64, pool->name, worker->id, worker->pid);
+
+  while (1) {
+    if (taosReadQitemFromQset(pool->qset, (void **)&msg, &qinfo) == 0) {
+      uInfo("worker:%s:%d qset:%p, got no message and exiting, thread:%08" PRId64, pool->name, worker->id, pool->qset,
+            worker->pid);
+      break;
+    }
+
+    if (qinfo.timestamp != 0) {
+      int64_t cost = taosGetTimestampUs() - qinfo.timestamp;
+      if (cost > QUEUE_THRESHOLD) {
+        uWarn("worker:%s,message has been queued for too long, cost: %" PRId64 "s", pool->name, cost / QUEUE_THRESHOLD);
+      }
+    }
+
+    tQueryAutoQWorkerWaitingCheck(pool);
+
+    if (qinfo.fp != NULL) {
+      qinfo.workerId = worker->id;
+      qinfo.threadNum = pool->num;
+      qinfo.workerCb = pool->pCb;
+      (*((FItem)qinfo.fp))(&qinfo, msg);
+    }
+
+    taosUpdateItemSize(qinfo.queue, 1);
+    if (!tQueryAutoQWorkerTryRecycleWorker(pool, worker)) {
+      uDebug("worker:%s:%d exited", pool->name, worker->id);
+      break;
+    }
+  }
+
+  destroyThreadLocalGeosCtx();
+  DestoryThreadLocalRegComp();
+
+  return NULL;
+}
+
+static bool tQueryAutoQWorkerTrySignalWaitingAfterBlock(void *p) {
+  SQueryAutoQWorkerPool *pPool = p;
+  bool                   ret = false;
+  int32_t waiting = pPool->waitingAfterBlockN;
+  while (waiting > 0) {
+    int32_t waitingNew = atomic_val_compare_exchange_32(&pPool->waitingAfterBlockN, waiting, waiting - 1);
+    if (waitingNew == waiting) {
+      taosThreadMutexLock(&pPool->waitingAfterBlockLock);
+      taosThreadCondSignal(&pPool->waitingAfterBlockCond);
+      taosThreadMutexUnlock(&pPool->waitingAfterBlockLock);
+      ret = true;
+      break;
+    }
+    waiting = waitingNew;
+  }
+  return ret;
+}
+
+static bool tQueryAutoQWorkerTrySignalWaitingBeforeProcess(void* p) {
+  SQueryAutoQWorkerPool *pPool = p;
+  bool                   ret = false;
+  int32_t waiting = pPool->waitingBeforeProcessMsgN;
+  while (waiting > 0) {
+    int32_t waitingNew = atomic_val_compare_exchange_32(&pPool->waitingBeforeProcessMsgN, waiting, waiting - 1);
+    if (waitingNew == waiting) {
+      taosThreadMutexLock(&pPool->waitingBeforeProcessMsgLock);
+      taosThreadCondSignal(&pPool->waitingBeforeProcessMsgCond);
+      taosThreadMutexUnlock(&pPool->waitingBeforeProcessMsgLock);
+      ret = true;
+      break;
+    }
+    waiting = waitingNew;
+  }
+  return ret;
+}
+
+static bool tQueryAutoQWorkerTryDecActive(void* p, int32_t minActive) {
+  SQueryAutoQWorkerPool *pPool = p;
+  bool                   ret = false;
+  int64_t val64 = pPool->activeRunningN;
+  int32_t                active = GET_ACTIVE_N(val64), running = GET_RUNNING_N(val64);
+  while (active > minActive) {
+    if (atomicComapreAndExchangeActiveAndRunning(&pPool->activeRunningN, &active, active - 1, &running, running - 1))
+      return true;
+  }
+  atomicFetchSubRunning(&pPool->activeRunningN, 1);
+  return false;
+}
+
+static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool* pPool) {
+  int32_t running = GET_RUNNING_N(pPool->activeRunningN);
+  while (running < pPool->num) {
+    if (atomicCompareExchangeRunning(&pPool->activeRunningN, &running, running + 1)) {
+      return TSDB_CODE_SUCCESS;
+    }
+  }
+  atomicFetchSubActive(&pPool->activeRunningN, 1);
+  // to wait for process
+  taosThreadMutexLock(&pPool->waitingBeforeProcessMsgLock);
+  atomic_fetch_add_32(&pPool->waitingBeforeProcessMsgN, 1);
+  if (!pPool->exit) taosThreadCondWait(&pPool->waitingBeforeProcessMsgCond, &pPool->waitingBeforeProcessMsgLock);
+  // recovered from waiting
+  taosThreadMutexUnlock(&pPool->waitingBeforeProcessMsgLock);
+  return TSDB_CODE_SUCCESS;
+}
+
+bool tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool* pPool, SQueryAutoQWorker* pWorker) {
+  if (tQueryAutoQWorkerTrySignalWaitingAfterBlock(pPool) || tQueryAutoQWorkerTrySignalWaitingBeforeProcess(pPool) ||
+      tQueryAutoQWorkerTryDecActive(pPool, pPool->num)) {
+    taosThreadMutexLock(&pPool->poolLock);
+    SListNode* pNode = listNode(pWorker);
+    tdListPopNode(pPool->workers, pNode);
+    // reclaim some workers
+    if (pWorker->id >= pPool->maxInUse) {
+      while (listNEles(pPool->exitedWorkers) > pPool->maxInUse - pPool->num) {
+        SListNode* head = tdListPopHead(pPool->exitedWorkers);
+        SQueryAutoQWorker* pWorker = (SQueryAutoQWorker*)head->data;
+        if (pWorker && taosCheckPthreadValid(pWorker->thread)) {
+          taosThreadJoin(pWorker->thread, NULL);
+          taosThreadClear(&pWorker->thread);
+        }
+        taosMemoryFree(head);
+      }
+      tdListAppendNode(pPool->exitedWorkers, pNode);
+      taosThreadMutexUnlock(&pPool->poolLock);
+      return false;
+    }
+
+    // put back to backup pool
+    tdListAppendNode(pPool->backupWorkers, pNode);
+    taosThreadMutexUnlock(&pPool->poolLock);
+
+    // start to wait at backup cond
+    taosThreadMutexLock(&pPool->backupLock);
+    atomic_fetch_add_32(&pPool->backupNum, 1);
+    if (!pPool->exit) taosThreadCondWait(&pPool->backupCond, &pPool->backupLock);
+    taosThreadMutexUnlock(&pPool->backupLock);
+
+    // recovered from backup
+    taosThreadMutexLock(&pPool->poolLock);
+    if (pPool->exit) {
+      taosThreadMutexUnlock(&pPool->poolLock);
+      return false;
+    }
+    tdListPopNode(pPool->backupWorkers, pNode);
+    tdListAppendNode(pPool->workers, pNode);
+    taosThreadMutexUnlock(&pPool->poolLock);
+
+    return true;
+  } else {
+    return true;
+  }
+}
+
+int32_t tQueryAutoQWorkerInit(SQueryAutoQWorkerPool *pool) {
+  pool->qset = taosOpenQset();
+  if (!pool->qset) return terrno;
+  pool->workers = tdListNew(sizeof(SQueryAutoQWorker));
+  if (!pool->workers) return TSDB_CODE_OUT_OF_MEMORY;
+  pool->backupWorkers = tdListNew(sizeof(SQueryAutoQWorker));
+  if (!pool->backupWorkers) return TSDB_CODE_OUT_OF_MEMORY;
+  pool->exitedWorkers = tdListNew(sizeof(SQueryAutoQWorker));
+  if (!pool->exitedWorkers) return TSDB_CODE_OUT_OF_MEMORY;
+  pool->maxInUse = pool->max * 2 + 2;
+
+  (void)taosThreadMutexInit(&pool->poolLock, NULL);
+  (void)taosThreadMutexInit(&pool->backupLock, NULL);
+  (void)taosThreadMutexInit(&pool->waitingAfterBlockLock, NULL);
+  (void)taosThreadMutexInit(&pool->waitingBeforeProcessMsgLock, NULL);
+
+  (void)taosThreadCondInit(&pool->waitingBeforeProcessMsgCond, NULL);
+  (void)taosThreadCondInit(&pool->waitingAfterBlockCond, NULL);
+  (void)taosThreadCondInit(&pool->backupCond, NULL);
+
+  if (!pool->pCb) {
+    pool->pCb = taosMemoryCalloc(1, sizeof(SQueryAutoQWorkerPoolCB));
+    if (!pool->pCb) return TSDB_CODE_OUT_OF_MEMORY;
+    pool->pCb->pPool = pool;
+    pool->pCb->beforeBlocking = tQueryAutoQWorkerBeforeBlocking;
+    pool->pCb->afterRecoverFromBlocking = tQueryAutoQWorkerRecoverFromBlocking;
+  }
+  return TSDB_CODE_SUCCESS;
+}
+
+void tQueryAutoQWorkerCleanup(SQueryAutoQWorkerPool *pPool) {
+  taosThreadMutexLock(&pPool->poolLock);
+  pPool->exit = true;
+  int32_t size = listNEles(pPool->workers);
+  for (int32_t i = 0; i < size; ++i) {
+    taosQsetThreadResume(pPool->qset);
+  }
+  size = listNEles(pPool->backupWorkers);
+  for (int32_t i = 0; i < size; ++i) {
+    taosQsetThreadResume(pPool->qset);
+  }
+  taosThreadMutexUnlock(&pPool->poolLock);
+
+  taosThreadMutexLock(&pPool->backupLock);
+  taosThreadCondBroadcast(&pPool->backupCond);
+  taosThreadMutexUnlock(&pPool->backupLock);
+
+  taosThreadMutexLock(&pPool->waitingAfterBlockLock);
+  taosThreadCondBroadcast(&pPool->waitingAfterBlockCond);
+  taosThreadMutexUnlock(&pPool->waitingAfterBlockLock);
+
+  taosThreadMutexLock(&pPool->waitingBeforeProcessMsgLock);
+  taosThreadCondBroadcast(&pPool->waitingBeforeProcessMsgCond);
+  taosThreadMutexUnlock(&pPool->waitingBeforeProcessMsgLock);
+
+  int32_t idx = 0;
+  SQueryAutoQWorker* worker = NULL;
+  while (true) {
+    taosThreadMutexLock(&pPool->poolLock);
+    if (listNEles(pPool->workers) == 0) {
+      taosThreadMutexUnlock(&pPool->poolLock);
+      break;
+    }
+    SListNode* pNode = tdListPopHead(pPool->workers);
+    worker = (SQueryAutoQWorker*)pNode->data;
+    taosThreadMutexUnlock(&pPool->poolLock);
+    if (worker && taosCheckPthreadValid(worker->thread)) {
+      taosThreadJoin(worker->thread, NULL);
+      taosThreadClear(&worker->thread);
+    }
+    taosMemoryFree(pNode);
+  }
+
+  while (listNEles(pPool->backupWorkers) > 0) {
+    SListNode* pNode = tdListPopHead(pPool->backupWorkers);
+    worker = (SQueryAutoQWorker*)pNode->data;
+    if (worker && taosCheckPthreadValid(worker->thread)) {
+      taosThreadJoin(worker->thread, NULL);
+      taosThreadClear(&worker->thread);
+    }
+    taosMemoryFree(pNode);
+  }
+
+  while (listNEles(pPool->exitedWorkers) > 0) {
+    SListNode* pNode = tdListPopHead(pPool->exitedWorkers);
+    worker = (SQueryAutoQWorker*)pNode->data;
+    if (worker && taosCheckPthreadValid(worker->thread)) {
+      taosThreadJoin(worker->thread, NULL);
+      taosThreadClear(&worker->thread);
+    }
+    taosMemoryFree(pNode);
+  }
+
+  tdListFree(pPool->workers);
+  tdListFree(pPool->backupWorkers);
+  tdListFree(pPool->exitedWorkers);
+  taosMemoryFree(pPool->pCb);
+
+  taosThreadMutexDestroy(&pPool->poolLock);
+  taosThreadMutexDestroy(&pPool->backupLock);
+  taosThreadMutexDestroy(&pPool->waitingAfterBlockLock);
+  taosThreadMutexDestroy(&pPool->waitingBeforeProcessMsgLock);
+
+  taosThreadCondDestroy(&pPool->backupCond);
+  taosThreadCondDestroy(&pPool->waitingAfterBlockCond);
+  taosThreadCondDestroy(&pPool->waitingBeforeProcessMsgCond);
+  taosCloseQset(pPool->qset);
+}
+
+STaosQueue *tQueryAutoQWorkerAllocQueue(SQueryAutoQWorkerPool *pool, void *ahandle, FItem fp) {
+  STaosQueue *queue = taosOpenQueue();
+  if (queue == NULL) return NULL;
+
+  taosThreadMutexLock(&pool->poolLock);
+  taosSetQueueFp(queue, fp, NULL);
+  taosAddIntoQset(pool->qset, queue, ahandle);
+  SQueryAutoQWorker worker = {0};
+  SQueryAutoQWorker* pWorker = NULL;
+
+  // spawn a thread to process queue
+  if (pool->num < pool->max) {
+    do {
+      worker.id = listNEles(pool->workers);
+      worker.backupIdx = -1;
+      worker.pool = pool;
+      SListNode* pNode = tdListAdd(pool->workers, &worker);
+      if (!pNode) {
+        taosCloseQueue(queue);
+        queue = NULL;
+        terrno = TSDB_CODE_OUT_OF_MEMORY;
+        break;
+      }
+      pWorker = (SQueryAutoQWorker*)pNode->data;
+
+      TdThreadAttr thAttr;
+      taosThreadAttrInit(&thAttr);
+      taosThreadAttrSetDetachState(&thAttr, PTHREAD_CREATE_JOINABLE);
+
+      if (taosThreadCreate(&pWorker->thread, &thAttr, (ThreadFp)tQueryAutoQWorkerThreadFp, pWorker) != 0) {
+        taosCloseQueue(queue);
+        terrno = TSDB_CODE_OUT_OF_MEMORY;
+        queue = NULL;
+        break;
+      }
+
+      taosThreadAttrDestroy(&thAttr);
+      pool->num++;
+      atomicFetchAddActive(&pool->activeRunningN, 1);
+      uInfo("worker:%s:%d is launched, total:%d", pool->name, pWorker->id, pool->num);
+    } while (pool->num < pool->min);
+  }
+
+  taosThreadMutexUnlock(&pool->poolLock);
+  uInfo("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
+
+  return queue;
+}
+
+void tQueryAutoQWorkerFreeQueue(SQueryAutoQWorkerPool *pPool, STaosQueue *pQ) {
+  taosCloseQueue(pQ);
+}
+
+static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool* pool) {
+  // try backup pool
+  int32_t backup = pool->backupNum;
+  while (backup > 0) {
+    int32_t backupNew = atomic_val_compare_exchange_32(&pool->backupNum, backup, backup - 1);
+    if (backupNew == backup) {
+      taosThreadCondSignal(&pool->backupCond);
+      return TSDB_CODE_SUCCESS;
+    }
+    backup = backupNew;
+  }
+  // backup pool is empty, create new
+  SQueryAutoQWorker* pWorker = NULL;
+  SQueryAutoQWorker worker = {0};
+  worker.pool = pool;
+  worker.backupIdx = -1;
+  taosThreadMutexLock(&pool->poolLock);
+  worker.id = listNEles(pool->workers);
+  SListNode* pNode = tdListAdd(pool->workers, &worker);
+  if (!pNode) {
+    taosThreadMutexUnlock(&pool->poolLock);
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return terrno;
+  }
+  taosThreadMutexUnlock(&pool->poolLock);
+  pWorker = (SQueryAutoQWorker*)pNode->data;
+
+  TdThreadAttr thAttr;
+  taosThreadAttrInit(&thAttr);
+  taosThreadAttrSetDetachState(&thAttr, PTHREAD_CREATE_JOINABLE);
+
+  if (taosThreadCreate(&pWorker->thread, &thAttr, (ThreadFp)tQueryAutoQWorkerThreadFp, pWorker) != 0) {
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return terrno;
+  }
+  taosThreadAttrDestroy(&thAttr);
+
+  return TSDB_CODE_SUCCESS;
+}
+
+static int32_t tQueryAutoQWorkerBeforeBlocking(void *p) {
+  SQueryAutoQWorkerPool* pPool = p;
+  if (tQueryAutoQWorkerTrySignalWaitingAfterBlock(p) || tQueryAutoQWorkerTrySignalWaitingBeforeProcess(p) ||
+      tQueryAutoQWorkerTryDecActive(p, 1)) {
+  } else {
+    int32_t code = tQueryAutoQWorkerAddWorker(pPool);
+    if (code != TSDB_CODE_SUCCESS) {
+      return code;
+    }
+  }
+
+  return TSDB_CODE_SUCCESS;
+}
+
+static int32_t tQueryAutoQWorkerRecoverFromBlocking(void *p) {
+  SQueryAutoQWorkerPool* pPool = p;
+  int64_t val64 = pPool->activeRunningN;
+  int32_t running = GET_RUNNING_N(val64), active = GET_ACTIVE_N(val64);
+  while (running < pPool->num) {
+    if (atomicComapreAndExchangeActiveAndRunning(&pPool->activeRunningN, &active, active + 1, &running, running + 1)) {
+      return TSDB_CODE_SUCCESS;
+    }
+  }
+  taosThreadMutexLock(&pPool->waitingAfterBlockLock);
+  atomic_fetch_add_32(&pPool->waitingAfterBlockN, 1);
+  if (!pPool->exit) taosThreadCondWait(&pPool->waitingAfterBlockCond, &pPool->waitingAfterBlockLock);
+  taosThreadMutexUnlock(&pPool->waitingAfterBlockLock);
+  if (pPool->exit) return TSDB_CODE_QRY_QWORKER_QUIT;
+  return TSDB_CODE_SUCCESS;
+}

tests/army/query/fill/fill_null.py

@@ -0,0 +1,36 @@
+# -*- coding: utf-8 -*-
+
+import frame.etool
+
+from frame.log import *
+from frame.cases import *
+from frame.sql import *
+from frame.caseBase import *
+from frame import *
+
+
+class TDTestCase(TBase):
+    def init(self, conn, logSql, replicaVar=1):
+        self.replicaVar = int(replicaVar)
+        self.dbname = "ts_5054"
+        tdLog.debug("start to execute %s" % __file__)
+        tdSql.init(conn.cursor(), logSql)
+
+    def run(self):
+        etool.benchMark(command=f"-d {self.dbname} -t 1 -n 1000 -S 10 -y")
+        tdSql.execute(f"use {self.dbname}")
+        tdSql.execute("select database();")
+        tdSql.query(
+            "select _wstart, first(ts), last(ts) from meters where ts >= '2017-07-14 10:40:00.000' and ts < '2017-07-14 10:40:10.000' partition by groupid interval(3s) fill(NULL);"
+        )
+        tdSql.checkRows(4)
+        tdSql.checkData(0, 1, "2017-07-14 10:40:00.000")
+        tdSql.checkData(0, 2, "2017-07-14 10:40:02.990")
+
+    def stop(self):
+        tdSql.close()
+        tdLog.success("%s successfully executed" % __file__)
+
+
+tdCases.addWindows(__file__, TDTestCase())
+tdCases.addLinux(__file__, TDTestCase())

tests/parallel_test/cases.task

@@ -19,6 +19,7 @@
 ,,y,army,./pytest.sh python3 ./test.py -f query/test_compare.py
 ,,y,army,./pytest.sh python3 ./test.py -f insert/test_column_tag_boundary.py
 ,,y,army,./pytest.sh python3 ./test.py -f query/fill/fill_desc.py -N 3 -L 3 -D 2
+,,y,army,./pytest.sh python3 ./test.py -f query/fill/fill_null.py
 ,,y,army,./pytest.sh python3 ./test.py -f cluster/incSnapshot.py -N 3
 ,,y,army,./pytest.sh python3 ./test.py -f query/query_basic.py -N 3
 ,,y,army,./pytest.sh python3 ./test.py -f query/accuracy/test_query_accuracy.py

tests/script/tsim/query/cache_last.sim

@@ -141,7 +141,7 @@ if $data01 != t2 then
   return -1
 endi
 
-sql select last(ts) ts1,tbname, ts from stb group by tbname;
+sql select last(ts) ts1,tbname, ts from stb group by tbname order by 1 desc;
 
 if $data00 != $data02 then
   print $data00
@@ -156,16 +156,16 @@ endi
 print step 3-------------------------------
 
 sql drop database if exists test1;
-sql create database test1  cachemodel 'both';
+sql create database test1  cachemodel 'both' vgroups 4;
 sql use test1;
 sql create table stb (ts timestamp,a int primary key,b int,c int) tags(ta int,tb int,tc int);
 
-sql create table t1 using stb tags(1,1,1);
+sql create table aaat1 using stb tags(1,1,1);
-sql create table t2 using stb tags(2,2,2);
+sql create table bbbt2 using stb tags(2,2,2);
-sql insert into t1 values('2024-06-05 11:00:00',1,2,3);
+sql insert into aaat1 values('2024-06-05 11:00:00',1,2,3);
-sql insert into t1 values('2024-06-05 12:00:00',2,2,3);
+sql insert into aaat1 values('2024-06-05 12:00:00',2,2,3);
-sql insert into t2 values('2024-06-05 13:00:00',3,2,3);
+sql insert into bbbt2 values('2024-06-05 13:00:00',3,2,3);
-sql insert into t2 values('2024-06-05 14:00:00',4,2,3);
+sql insert into bbbt2 values('2024-06-05 14:00:00',4,2,3);
 
 sql select last(ts) ts1,ts from stb;
 
@@ -179,7 +179,7 @@ if $data00 != @24-06-05 14:00:00.000@ then
   return -1
 endi
 
-sql select last(ts) ts1,ts from stb group by tbname;
+sql select last(ts) ts1,ts from stb group by tbname order by 1 desc;
 
 if $data00 != $data01 then
   print $data00
@@ -203,12 +203,12 @@ if $data00 != @24-06-05 14:00:00.000@ then
   return -1
 endi
 
-if $data01 != t2 then
+if $data01 != bbbt2 then
   print $data01
   return -1
 endi
 
-sql select last(ts) ts1,tbname, ts from stb group by tbname;
+sql select last(ts) ts1,tbname, ts from stb group by tbname order by 1 desc;
 
 if $data00 != $data02 then
   print $data00
@@ -220,7 +220,7 @@ if $data00 != @24-06-05 14:00:00.000@ then
   return -1
 endi
 
-if $data01 != t2 then
+if $data01 != bbbt2 then
   print $data01
   return -1
 endi
@@ -239,7 +239,7 @@ if $data01 != @24-06-05 14:00:00.000@ then
   return -1
 endi
 
-sql select last(a) a,ts from stb group by tbname;
+sql select last(a) a,ts from stb group by tbname order by 1 desc;
 
 if $data00 != 4 then
   print $data00
@@ -258,7 +258,7 @@ if $data00 != 4 then
   return -1
 endi
 
-if $data01 != t2 then
+if $data01 != bbbt2 then
   print $data01
   return -1
 endi
@@ -268,14 +268,14 @@ if $data02 != @24-06-05 14:00:00.000@ then
   return -1
 endi
 
-sql select last(a) a,tbname, ts from stb group by tbname;
+sql select last(a) a,tbname, ts from stb group by tbname order by 1 desc;
 
 if $data00 != 4 then
   print $data00
   return -1
 endi
 
-if $data01 != t2 then
+if $data01 != bbbt2 then
   print $data01
   return -1
 endi
@@ -299,7 +299,7 @@ if $data01 != 4 then
   return -1
 endi
 
-sql select last(ts) ts1,a from stb group by tbname;
+sql select last(ts) ts1,a from stb group by tbname order by 1 desc;
 
 if $data00 != @24-06-05 14:00:00.000@ then
   print $data00
@@ -318,7 +318,7 @@ if $data00 != @24-06-05 14:00:00.000@ then
   return -1
 endi
 
-if $data01 != t2 then
+if $data01 != bbbt2 then
   print $data01
   return -1
 endi
@@ -328,14 +328,14 @@ if $data02 != 4 then
   return -1
 endi
 
-sql select last(ts) ts1,tbname, a from stb group by tbname;
+sql select last(ts) ts1,tbname, a from stb group by tbname order by 1 desc;
 
 if $data00 != @24-06-05 14:00:00.000@ then
   print $data00
   return -1
 endi
 
-if $data01 != t2 then
+if $data01 != bbbt2 then
   print $data01
   return -1
 endi
@@ -345,4 +345,25 @@ if $data02 != 4 then
   return -1
 endi
 
+$loop_count = 0
+
+loop0:
+
+sql select last(ts), last_row(ts) from stb;
+
+if $data00 != $data01 then
+  print ====data00=$data00
+  print ====data01=$data01
+  return -1
+endi
+
+$loop_count = $loop_count + 1
+if $loop_count < 10 then
+  sleep 200
+  print ====checktimes=$loop_count
+  goto loop0
+endi
+
+print ------------------------end
+
 system sh/exec.sh -n dnode1 -s stop -x SIGINT

tests/system-test/2-query/last_row.py

@@ -649,8 +649,8 @@ class TDTestCase:
         tdSql.checkData(0,1,None)
 
         tdSql.query(f"select ts , last_row(c1) ,c1  from (select ts , max(c1) c1  ,t1 from {dbname}.stb1 where ts >now -1h and ts <now+1h interval(10s) fill(value ,10, 10, 10))")
-        tdSql.checkData(0,1,10)
+        tdSql.checkData(0,1,9)
-        tdSql.checkData(0,1,10)
+        tdSql.checkData(0,1,9)
 
         tdSql.error(f"select ts , last_row(c1) ,c1  from (select count(c1) c1 from {dbname}.stb1 where ts >now -1h and ts <now+1h interval(10s) fill(value ,10, 10, 10))")
 

tests/system-test/7-tmq/tmq_primary_key.py

@@ -12,6 +12,7 @@ from util.dnodes import *
 from util.common import *
 from taos.tmq import *
 from util.dnodes import *
+from util.cluster import *
 import datetime
 
 sys.path.append("./7-tmq")
@@ -137,6 +138,7 @@ class TDTestCase:
                 print("index:" + str(index))
         finally:
             consumer.close()
+            tdSql.execute(f'drop topic topic_pk_query;')
 
     def primaryKeyTestIntStable(self):
         print("==============Case 2: primary key test int for stable")
@@ -247,6 +249,7 @@ class TDTestCase:
                 print("index:" + str(index))
         finally:
             consumer.close()
+            tdSql.execute(f'drop topic topic_pk_stable;')
 
     def primaryKeyTestInt(self):
         print("==============Case 3: primary key test int for db")
@@ -356,6 +359,7 @@ class TDTestCase:
                 print("index:" + str(index))
         finally:
             consumer.close()
+            tdSql.execute(f'drop topic topic_in;')
 
     def primaryKeyTestString(self):
         print("==============Case 4: primary key test string for db")
@@ -468,12 +472,90 @@ class TDTestCase:
                 print("index:" + str(index))
         finally:
             consumer.close()
+            tdSql.execute(f'drop topic topic_pk_string;')
 
+    def primaryKeyTestTD_30755(self):
+        print("==============Case 5: primary key test td-30755 for query")
+        tdSql.execute(f'create database if not exists db_pk_query_30755 vgroups 1 wal_retention_period 3600;')
+        tdSql.execute(f'use db_pk_query_30755;')
+        tdSql.execute(f'create table if not exists pk (ts timestamp, c1 int primary key, c2 int);')
+        for i in range(0, 100000):
+            tdSql.execute(f'insert into pk values(1669092069068, {i}, 1);')
+        tdSql.execute(f'flush database db_pk_query_30755')
+
+        tdSql.execute(f'create topic topic_pk_query_30755 as select * from pk')
+
+        consumer_dict = {
+            "group.id": "g1",
+            "td.connect.user": "root",
+            "td.connect.pass": "taosdata",
+            "auto.offset.reset": "earliest",
+            "enable.auto.commit": "false",
+            "experimental.snapshot.enable": "true",
+        }
+        consumer = Consumer(consumer_dict)
+
+        try:
+            consumer.subscribe(["topic_pk_query_30755"])
+        except TmqError:
+            tdLog.exit(f"subscribe error")
+
+        firstConsume = 0
+        try:
+            while firstConsume < 50000:
+                res = consumer.poll(1)
+                if not res:
+                    continue
+                val = res.value()
+                if val is None:
+                    continue
+                for block in val:
+                    data = block.fetchall()
+                    firstConsume += len(data)
+                consumer.commit(res)
+        finally:
+            consumer.close()
+
+        tdSql.query(f'show subscriptions;')
+        sub = tdSql.getData(0, 4);
+        print(sub)
+        if not sub.startswith("tsdb"):
+            tdLog.exit(f"show subscriptions error")
+
+        tdDnodes.stop(1)
+        time.sleep(2)
+        tdDnodes.start(1)
+
+        consumer = Consumer(consumer_dict)
+
+        tdSql.execute(f'use db_pk_query_30755;')
+        try:
+            consumer.subscribe(["topic_pk_query_30755"])
+        except TmqError:
+            tdLog.exit(f"subscribe error")
+
+        secondConsume = 0
+        try:
+            while firstConsume + secondConsume < 100000:
+                res = consumer.poll(1)
+                if not res:
+                    continue
+                val = res.value()
+                if val is None:
+                    continue
+                for block in val:
+                    data = block.fetchall()
+                    secondConsume += len(data)
+                consumer.commit(res)
+        finally:
+            consumer.close()
+            tdSql.execute(f'drop topic topic_pk_query_30755;')
     def run(self):
         self.primaryKeyTestIntQuery()
         self.primaryKeyTestIntStable()
         self.primaryKeyTestInt()
         self.primaryKeyTestString()
+        self.primaryKeyTestTD_30755()
 
     def stop(self):
         tdSql.close()