Merge branch '3.0' of https://github.com/taosdata/TDengine into dynamicIdx

docs/en/07-develop/03-insert-data/20-kafka-writting.mdx

@@ -0,0 +1,46 @@
+---
+title: Write from Kafka
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+import PyKafka from "./_py_kafka.mdx";
+
+## About Kafka 
+
+Apache Kafka is an open-source distributed event streaming platform, used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications. For the key concepts of kafka, please refer to [kafka documentation](https://kafka.apache.org/documentation/#gettingStarted).
+
+### kafka topic
+
+Messages in Kafka are organized by topics. A topic may have one or more partitions. We can manage kafka topics through `kafka-topics`.
+
+create a topic named `kafka-events`:
+ 
+```
+bin/kafka-topics.sh --create --topic kafka-events --bootstrap-server localhost:9092
+```
+
+Alter `kafka-events` topic to set partitions to 3:
+
+```
+bin/kafka-topics.sh --alter --topic kafka-events --partitions 3 --bootstrap-server=localhost:9092
+```
+
+Show all topics and partitions in Kafka:
+
+```
+bin/kafka-topics.sh --bootstrap-server=localhost:9092 --describe
+```
+
+## Insert into TDengine
+
+We can write data into TDengine via SQL or Schemaless. For more information, please refer to [Insert Using SQL](/develop/insert-data/sql-writing/) or [High Performance Writing](/develop/insert-data/high-volume/) or [Schemaless Writing](/reference/schemaless/).
+
+## Examples
+
+<Tabs defaultValue="Python" groupId="lang">
+  <TabItem label="Python" value="Python">
+    <PyKafka />
+  </TabItem>
+</Tabs>
+

docs/en/07-develop/03-insert-data/_py_kafka.mdx

@@ -0,0 +1,60 @@
+### python Kafka 客户端
+
+For python kafka client, please refer to [kafka client](https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Python). In this document, we use [kafka-python](http://github.com/dpkp/kafka-python).
+
+### consume from Kafka
+
+The simple way to consume messages from Kafka is to read messages one by one. The demo is as follows:
+
+```
+from kafka import KafkaConsumer
+consumer = KafkaConsumer('my_favorite_topic')
+for msg in consumer:
+     print (msg)
+```
+
+For higher performance, we can consume message from kafka in batch. The demo is as follows:
+
+```
+from kafka import KafkaConsumer
+consumer = KafkaConsumer('my_favorite_topic')
+while True:
+    msgs = consumer.poll(timeout_ms=500, max_records=1000)
+    if msgs:
+        print (msgs)
+```
+
+### multi-threading
+
+For more higher performance we can process data from kafka in multi-thread. We can use python's ThreadPoolExecutor to achieve multithreading. The demo is as follows:
+
+```
+from concurrent.futures import ThreadPoolExecutor, Future
+pool = ThreadPoolExecutor(max_workers=10)
+pool.submit(...)
+```
+
+### multi-process
+
+For more higher performance, sometimes we use multiprocessing. In this case, the number of Kafka Consumers should not be greater than the number of Kafka Topic Partitions. The demo is as follows:
+
+```
+from multiprocessing import Process
+
+ps = []
+for i in range(5):
+    p = Process(target=Consumer().consume())
+    p.start()
+    ps.append(p)
+
+for p in ps:
+    p.join()
+```
+
+In addition to python's built-in multithreading and multiprocessing library, we can also use the third-party library gunicorn.
+
+### Examples
+
+```py
+{{#include docs/examples/python/kafka_example.py}}
+```

docs/en/20-third-party/01-grafana.mdx

@@ -76,7 +76,7 @@ sudo -u grafana grafana-cli plugins install tdengine-datasource
 You can also download zip files from [GitHub](https://github.com/taosdata/grafanaplugin/releases/tag/latest) or [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) and install manually. The commands are as follows:
 
 ```bash
-GF_VERSION=3.2.2
+GF_VERSION=3.2.7
 # from GitHub
 wget https://github.com/taosdata/grafanaplugin/releases/download/v$GF_VERSION/tdengine-datasource-$GF_VERSION.zip
 # from Grafana

docs/en/25-application/01-telegraf.md

@@ -38,15 +38,9 @@ Download the latest TDengine-server from the [Downloads](http://tdengine.com/en/
 
 ## Data Connection Setup
 
-### Download TDengine plug-in to grafana plug-in directory
+### Install Grafana Plugin and Configure Data Source
 
-```bash
+Please refer to [Install Grafana Plugin and Configure Data Source](/third-party/grafana/#install-grafana-plugin-and-configure-data-source)
-1. wget -c https://github.com/taosdata/grafanaplugin/releases/download/v3.1.3/tdengine-datasource-3.1.3.zip
-2. sudo unzip tdengine-datasource-3.1.3.zip -d /var/lib/grafana/plugins/
-3. sudo chown grafana:grafana -R /var/lib/grafana/plugins/tdengine
-4. echo -e "[plugins]\nallow_loading_unsigned_plugins = tdengine-datasource\n" | sudo tee -a /etc/grafana/grafana.ini
-5. sudo systemctl restart grafana-server.service
-```
 
 ### Modify /etc/telegraf/telegraf.conf
 

docs/en/25-application/02-collectd.md

@@ -41,15 +41,9 @@ Download the latest TDengine-server from the [Downloads](http://tdengine.com/en/
 
 ## Data Connection Setup
 
-### Copy the TDengine plugin to the grafana plugin directory
+### Install Grafana Plugin and Configure Data Source
 
-```bash
+Please refer to [Install Grafana Plugin and Configure Data Source](/third-party/grafana/#install-grafana-plugin-and-configure-data-source)
-1. wget -c https://github.com/taosdata/grafanaplugin/releases/download/v3.1.3/tdengine-datasource-3.1.3.zip
-2. sudo unzip tdengine-datasource-3.1.3.zip -d /var/lib/grafana/plugins/
-3. sudo chown grafana:grafana -R /var/lib/grafana/plugins/tdengine
-4. echo -e "[plugins]\nallow_loading_unsigned_plugins = tdengine-datasource\n" | sudo tee -a /etc/grafana/grafana.ini
-5. sudo systemctl restart grafana-server.service
-```
 
 ### Configure collectd
 

docs/examples/python/kafka_example.py

@@ -0,0 +1,192 @@
+#! encoding = utf-8
+import json
+import time
+from json import JSONDecodeError
+from typing import Callable
+import logging
+from concurrent.futures import ThreadPoolExecutor, Future
+
+import taos
+from kafka import KafkaConsumer
+from kafka.consumer.fetcher import ConsumerRecord
+
+
+class Consumer(object):
+    DEFAULT_CONFIGS = {
+        'kafka_brokers': 'localhost:9092',
+        'kafka_topic': 'python_kafka',
+        'kafka_group_id': 'taos',
+        'taos_host': 'localhost',
+        'taos_user': 'root',
+        'taos_password': 'taosdata',
+        'taos_database': 'power',
+        'taos_port': 6030,
+        'timezone': None,
+        'clean_after_testing': False,
+        'bath_consume': True,
+        'batch_size': 1000,
+        'async_model': True,
+        'workers': 10
+    }
+
+    LOCATIONS = ['California.SanFrancisco', 'California.LosAngles', 'California.SanDiego', 'California.SanJose',
+                 'California.PaloAlto', 'California.Campbell', 'California.MountainView', 'California.Sunnyvale',
+                 'California.SantaClara', 'California.Cupertino']
+
+    CREATE_DATABASE_SQL = 'create database if not exists {} keep 365 duration 10 buffer 16 wal_level 1'
+    USE_DATABASE_SQL = 'use {}'
+    DROP_TABLE_SQL = 'drop table if exists meters'
+    DROP_DATABASE_SQL = 'drop database if exists {}'
+    CREATE_STABLE_SQL = 'create stable meters (ts timestamp, current float, voltage int, phase float) ' \
+                        'tags (location binary(64), groupId int)'
+    CREATE_TABLE_SQL = 'create table if not exists {} using meters tags (\'{}\', {})'
+    INSERT_SQL_HEADER = "insert into "
+    INSERT_PART_SQL = 'power.{} values (\'{}\', {}, {}, {})'
+
+    def __init__(self, **configs):
+        self.config: dict = self.DEFAULT_CONFIGS
+        self.config.update(configs)
+        self.consumer = KafkaConsumer(
+            self.config.get('kafka_topic'),  # topic
+            bootstrap_servers=self.config.get('kafka_brokers'),
+            group_id=self.config.get('kafka_group_id'),
+        )
+        self.taos = taos.connect(
+            host=self.config.get('taos_host'),
+            user=self.config.get('taos_user'),
+            password=self.config.get('taos_password'),
+            port=self.config.get('taos_port'),
+            timezone=self.config.get('timezone'),
+        )
+        if self.config.get('async_model'):
+            self.pool = ThreadPoolExecutor(max_workers=self.config.get('workers'))
+            self.tasks: list[Future] = []
+        # tags and table mapping # key: {location}_{groupId} value:
+        self.tag_table_mapping = {}
+        i = 0
+        for location in self.LOCATIONS:
+            for j in range(1, 11):
+                table_name = 'd{}'.format(i)
+                self._cache_table(location=location, group_id=j, table_name=table_name)
+                i += 1
+
+    def init_env(self):
+        # create database and table
+        self.taos.execute(self.DROP_DATABASE_SQL.format(self.config.get('taos_database')))
+        self.taos.execute(self.CREATE_DATABASE_SQL.format(self.config.get('taos_database')))
+        self.taos.execute(self.USE_DATABASE_SQL.format(self.config.get('taos_database')))
+        self.taos.execute(self.DROP_TABLE_SQL)
+        self.taos.execute(self.CREATE_STABLE_SQL)
+        for tags, table_name in self.tag_table_mapping.items():
+            location, group_id = _get_location_and_group(tags)
+            self.taos.execute(self.CREATE_TABLE_SQL.format(table_name, location, group_id))
+
+    def consume(self):
+        logging.warning('## start consumer topic-[%s]', self.config.get('kafka_topic'))
+        try:
+            if self.config.get('bath_consume'):
+                self._run_batch(self._to_taos_batch)
+            else:
+                self._run(self._to_taos)
+        except KeyboardInterrupt:
+            logging.warning("## caught keyboard interrupt, stopping")
+        finally:
+            self.stop()
+
+    def stop(self):
+        # close consumer
+        if self.consumer is not None:
+            self.consumer.commit()
+            self.consumer.close()
+
+        # multi thread
+        if self.config.get('async_model'):
+            for task in self.tasks:
+                while not task.done():
+                    pass
+            if self.pool is not None:
+                self.pool.shutdown()
+
+        # clean data
+        if self.config.get('clean_after_testing'):
+            self.taos.execute(self.DROP_TABLE_SQL)
+            self.taos.execute(self.DROP_DATABASE_SQL.format(self.config.get('taos_database')))
+        # close taos
+        if self.taos is not None:
+            self.taos.close()
+
+    def _run(self, f: Callable[[ConsumerRecord], bool]):
+        for message in self.consumer:
+            if self.config.get('async_model'):
+                self.pool.submit(f(message))
+            else:
+                f(message)
+
+    def _run_batch(self, f: Callable[[list[list[ConsumerRecord]]], None]):
+        while True:
+            messages = self.consumer.poll(timeout_ms=500, max_records=self.config.get('batch_size'))
+            if messages:
+                if self.config.get('async_model'):
+                    self.pool.submit(f, messages.values())
+                else:
+                    f(list(messages.values()))
+            if not messages:
+                time.sleep(0.1)
+
+    def _to_taos(self, message: ConsumerRecord) -> bool:
+        sql = self.INSERT_SQL_HEADER + self._build_sql(message.value)
+        if len(sql) == 0:  # decode error, skip
+            return True
+        logging.info('## insert sql %s', sql)
+        return self.taos.execute(sql=sql) == 1
+
+    def _to_taos_batch(self, messages: list[list[ConsumerRecord]]):
+        sql = self._build_sql_batch(messages=messages)
+        if len(sql) == 0:  # decode error, skip
+            return
+        self.taos.execute(sql=sql)
+
+    def _build_sql(self, msg_value: str) -> str:
+        try:
+            data = json.loads(msg_value)
+        except JSONDecodeError as e:
+            logging.error('## decode message [%s] error ', msg_value, e)
+            return ''
+        location = data.get('location')
+        group_id = data.get('groupId')
+        ts = data.get('ts')
+        current = data.get('current')
+        voltage = data.get('voltage')
+        phase = data.get('phase')
+
+        table_name = self._get_table_name(location=location, group_id=group_id)
+        return self.INSERT_PART_SQL.format(table_name, ts, current, voltage, phase)
+
+    def _build_sql_batch(self, messages: list[list[ConsumerRecord]]) -> str:
+        sql_list = []
+        for partition_messages in messages:
+            for message in partition_messages:
+                sql_list.append(self._build_sql(message.value))
+
+        return self.INSERT_SQL_HEADER + ' '.join(sql_list)
+
+    def _cache_table(self, location: str, group_id: int, table_name: str):
+        self.tag_table_mapping[_tag_table_mapping_key(location=location, group_id=group_id)] = table_name
+
+    def _get_table_name(self, location: str, group_id: int) -> str:
+        return self.tag_table_mapping.get(_tag_table_mapping_key(location=location, group_id=group_id))
+
+
+def _tag_table_mapping_key(location: str, group_id: int):
+    return '{}_{}'.format(location, group_id)
+
+
+def _get_location_and_group(key: str) -> (str, int):
+    fields = key.split('_')
+    return fields[0], fields[1]
+
+
+if __name__ == '__main__':
+    consumer = Consumer(async_model=True)
+    consumer.init_env()
+    consumer.consume()

docs/zh/07-develop/03-insert-data/20-kafka-writting.mdx

@@ -0,0 +1,47 @@
+---
+title: 从 Kafka 写入
+---
+
+import Tabs from "@theme/Tabs";
+import TabItem from "@theme/TabItem";
+import PyKafka from "./_py_kafka.mdx";
+
+## Kafka 介绍
+
+Apache Kafka 是开源的分布式消息分发平台，被广泛应用于高性能数据管道、流式数据分析、数据集成和事件驱动类型的应用程序。Kafka 包含 Producer、Consumer 和 Topic，其中 Producer 是向 Kafka 发送消息的进程，Consumer 是从 Kafka 消费消息的进程。Kafka 相关概念可以参考[官方文档](https://kafka.apache.org/documentation/#gettingStarted)。
+
+
+### kafka topic
+
+Kafka 的消息按 topic 组织，每个 topic 会有一到多个 partition。可以通过 kafka 的 `kafka-topics` 管理 topic。
+
+创建名为 `kafka-events` 的topic:
+
+```
+bin/kafka-topics.sh --create --topic kafka-events --bootstrap-server localhost:9092
+```
+
+修改 `kafka-events` 的 partition 数量为 3:
+
+```
+bin/kafka-topics.sh --alter --topic kafka-events --partitions 3 --bootstrap-server=localhost:9092
+```
+
+展示所有的 topic 和 partition:
+
+```
+bin/kafka-topics.sh --bootstrap-server=localhost:9092 --describe
+```
+
+## 写入 TDengine
+
+TDengine 支持 Sql 方式和 Schemaless 方式的数据写入，Sql 方式数据写入可以参考 [TDengine SQL 写入](/develop/insert-data/sql-writing/) 和 [TDengine 高效写入](/develop/insert-data/high-volume/)。Schemaless 方式数据写入可以参考 [TDengine Schemaless 写入](/reference/schemaless/) 文档。
+
+## 示例代码
+
+<Tabs defaultValue="Python" groupId="lang">
+  <TabItem label="Python" value="Python">
+    <PyKafka />
+  </TabItem>
+</Tabs>
+

docs/zh/07-develop/03-insert-data/_py_kafka.mdx

@@ -0,0 +1,60 @@
+### python Kafka 客户端
+
+Kafka 的 python 客户端可以参考文档 [kafka client](https://cwiki.apache.org/confluence/display/KAFKA/Clients#Clients-Python)。推荐使用 [confluent-kafka-python](https://github.com/confluentinc/confluent-kafka-python) 和 [kafka-python](http://github.com/dpkp/kafka-python)。以下示例以 [kafka-python](http://github.com/dpkp/kafka-python) 为例。
+
+### 从 Kafka 消费数据
+
+Kafka 客户端采用 pull 的方式从 Kafka 消费数据，可以采用单条消费的方式或批量消费的方式读取数据。使用 [kafka-python](http://github.com/dpkp/kafka-python) 客户端单条消费数据的示例如下：
+
+```
+from kafka import KafkaConsumer
+consumer = KafkaConsumer('my_favorite_topic')
+for msg in consumer:
+     print (msg)
+```
+
+单条消费的方式在数据流量大的情况下往往存在性能瓶颈，导致 Kafka 消息积压，更推荐使用批量消费的方式消费数据。使用 [kafka-python](http://github.com/dpkp/kafka-python) 客户端批量消费数据的示例如下：
+
+```
+from kafka import KafkaConsumer
+consumer = KafkaConsumer('my_favorite_topic')
+while True:
+    msgs = consumer.poll(timeout_ms=500, max_records=1000)
+    if msgs:
+        print (msgs)
+```
+
+### Python 多线程
+
+为了提高数据写入效率，通常采用多线程的方式写入数据，可以使用 python 线程池 ThreadPoolExecutor 实现多线程。示例代码如下：
+
+```
+from concurrent.futures import ThreadPoolExecutor, Future
+pool = ThreadPoolExecutor(max_workers=10)
+pool.submit(...)
+```
+
+### Python 多进程
+
+单个python进程不能充分发挥多核 CPU 的性能，有时候我们会选择多进程的方式。在多进程的情况下，需要注意，Kafka Consumer 的数量应该小于等于 Kafka Topic Partition 数量。Python 多进程示例代码如下：
+
+```
+from multiprocessing import Process
+
+ps = []
+for i in range(5):
+    p = Process(target=Consumer().consume())
+    p.start()
+    ps.append(p)
+
+for p in ps:
+    p.join()
+```
+
+除了 Python 内置的多线程和多进程方式，还可以通过第三方库 gunicorn 实现并发。
+
+### 完整示例
+
+```py
+{{#include docs/examples/python/kafka_example.py}}
+```

docs/zh/25-application/01-telegraf.md

@@ -39,15 +39,9 @@ IT 运维监测数据通常都是对时间特性比较敏感的数据，例如
 
 ## 数据链路设置
 
-### 下载 TDengine 插件到 Grafana 插件目录
+### 安装 Grafana Plugin 并配置数据源
 
-```bash
+请参考[安装 Grafana Plugin 并配置数据源](/third-party/grafana/#%E5%AE%89%E8%A3%85-grafana-plugin-%E5%B9%B6%E9%85%8D%E7%BD%AE%E6%95%B0%E6%8D%AE%E6%BA%90)。
-1. wget -c https://github.com/taosdata/grafanaplugin/releases/download/v3.1.3/tdengine-datasource-3.1.3.zip
-2. sudo unzip tdengine-datasource-3.1.3.zip -d /var/lib/grafana/plugins/
-3. sudo chown grafana:grafana -R /var/lib/grafana/plugins/tdengine
-4. echo -e "[plugins]\nallow_loading_unsigned_plugins = tdengine-datasource\n" | sudo tee -a /etc/grafana/grafana.ini
-5. sudo systemctl restart grafana-server.service
-```
 
 ### 修改 /etc/telegraf/telegraf.conf
 

docs/zh/25-application/02-collectd.md

@@ -41,15 +41,9 @@ IT 运维监测数据通常都是对时间特性比较敏感的数据，例如
 
 ## 数据链路设置
 
-### 复制 TDengine 插件到 grafana 插件目录
+### 安装 Grafana Plugin 并配置数据源
 
-```bash
+请参考[安装 Grafana Plugin 并配置数据源](/third-party/grafana/#%E5%AE%89%E8%A3%85-grafana-plugin-%E5%B9%B6%E9%85%8D%E7%BD%AE%E6%95%B0%E6%8D%AE%E6%BA%90)。
-1. wget -c https://github.com/taosdata/grafanaplugin/releases/download/v3.1.3/tdengine-datasource-3.1.3.zip
-2. sudo unzip tdengine-datasource-3.1.3.zip -d /var/lib/grafana/plugins/
-3. sudo chown grafana:grafana -R /var/lib/grafana/plugins/tdengine
-4. echo -e "[plugins]\nallow_loading_unsigned_plugins = tdengine-datasource\n" | sudo tee -a /etc/grafana/grafana.ini
-5. sudo systemctl restart grafana-server.service
-```
 
 ### 配置 collectd
 

include/common/tglobal.h

@@ -137,6 +137,7 @@ extern int64_t tsWalFsyncDataSizeLimit;
 // internal
 extern int32_t tsTransPullupInterval;
 extern int32_t tsMqRebalanceInterval;
+extern int32_t tsStreamCheckpointTickInterval;
 extern int32_t tsTtlUnit;
 extern int32_t tsTtlPushInterval;
 extern int32_t tsGrantHBInterval;

include/common/tmsg.h

@@ -1150,6 +1150,13 @@ typedef struct {
 int32_t tSerializeSMTimerMsg(void* buf, int32_t bufLen, SMTimerReq* pReq);
 int32_t tDeserializeSMTimerMsg(void* buf, int32_t bufLen, SMTimerReq* pReq);
 
+typedef struct {
+  int64_t tick;
+} SMStreamTickReq;
+
+int32_t tSerializeSMStreamTickMsg(void* buf, int32_t bufLen, SMStreamTickReq* pReq);
+int32_t tDeserializeSMStreamTickMsg(void* buf, int32_t bufLen, SMStreamTickReq* pReq);
+
 typedef struct {
   int32_t  id;
   uint16_t port;                 // node sync Port
@@ -1750,6 +1757,8 @@ typedef struct {
   int64_t watermark;
   int32_t numOfTags;
   SArray* pTags;  // array of SField
+  // 3.0.20
+  int64_t checkpointFreq;  // ms
 } SCMCreateStreamReq;
 
 typedef struct {
@@ -1949,6 +1958,12 @@ typedef struct {
   SHashObj* rebSubHash;  // SHashObj<key, SMqRebSubscribe>
 } SMqDoRebalanceMsg;
 
+typedef struct {
+  int64_t streamId;
+  int64_t checkpointId;
+  char    streamName[TSDB_STREAM_FNAME_LEN];
+} SMStreamDoCheckpointMsg;
+
 typedef struct {
   int64_t status;
 } SMVSubscribeRsp;

include/common/tmsgdef.h

@@ -172,6 +172,8 @@ enum {
   TD_DEF_MSG_TYPE(TDMT_MND_SERVER_VERSION, "server-version", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_UPTIME_TIMER, "uptime-timer", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_TMQ_LOST_CONSUMER_CLEAR, "lost-consumer-clear", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_MND_STREAM_CHECKPOINT_TIMER, "stream-checkpoint-tmr", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_MND_STREAM_BEGIN_CHECKPOINT, "stream-begin-checkpoint", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_MND_MAX_MSG, "mnd-max", NULL, NULL)
 
   TD_NEW_MSG_SEG(TDMT_VND_MSG)
@@ -241,8 +243,11 @@ enum {
   TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_DISPATCH, "stream-task-dispatch", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_UNUSED1, "stream-unused1", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_RETRIEVE, "stream-retrieve", NULL, NULL)
-  TD_DEF_MSG_TYPE(TDMT_STREAM_RECOVER_FINISH, "vnode-stream-finish", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_RECOVER_FINISH, "stream-recover-finish", NULL, NULL)
-  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_CHECK, "vnode-stream-task-check", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_CHECK, "stream-task-check", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_CHECKPOINT, "stream-checkpoint", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_REPORT_CHECKPOINT, "stream-report-checkpoint", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_RESTORE_CHECKPOINT, "stream-restore-checkpoint", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_STREAM_MAX_MSG, "stream-max", NULL, NULL)
 
   TD_NEW_MSG_SEG(TDMT_MON_MSG)
@@ -282,6 +287,7 @@ enum {
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_TRIGGER, "vnode-stream-trigger", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_RECOVER_NONBLOCKING_STAGE, "vnode-stream-recover1", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_RECOVER_BLOCKING_STAGE, "vnode-stream-recover2", NULL, NULL)
+  TD_DEF_MSG_TYPE(TDMT_VND_STREAM_CHECK_POINT_SOURCE, "vnode-stream-checkpoint-source", NULL, NULL)
   TD_DEF_MSG_TYPE(TDMT_VND_STREAM_MAX_MSG, "vnd-stream-max", NULL, NULL)
 
   TD_NEW_MSG_SEG(TDMT_VND_TMQ_MSG)

include/libs/executor/executor.h

@@ -152,7 +152,7 @@ void qCleanExecTaskBlockBuf(qTaskInfo_t tinfo);
  * @param tinfo  qhandle
  * @return
  */
-int32_t qAsyncKillTask(qTaskInfo_t tinfo);
+int32_t qAsyncKillTask(qTaskInfo_t tinfo, int32_t rspCode);
 
 /**
  * destroy query info structure

include/libs/function/function.h

@@ -163,6 +163,7 @@ typedef struct tExprNode {
       int32_t               functionId;
       int32_t               num;
       struct SFunctionNode *pFunctNode;
+      int32_t               functionType;
     } _function;
 
     struct {

include/libs/function/functionMgt.h

@@ -130,6 +130,7 @@ typedef enum EFunctionType {
   FUNCTION_TYPE_GROUP_KEY,
   FUNCTION_TYPE_CACHE_LAST_ROW,
   FUNCTION_TYPE_CACHE_LAST,
+  FUNCTION_TYPE_TABLE_COUNT,
 
   // distributed splitting functions
   FUNCTION_TYPE_APERCENTILE_PARTIAL = 4000,

include/libs/nodes/nodes.h

@@ -60,6 +60,12 @@ extern "C" {
   for (SListCell* cell = (NULL != (list) ? (list)->pHead : NULL); \
        (NULL != cell ? (node = &(cell->pNode), true) : (node = NULL, false)); cell = cell->pNext)
 
+#define NODES_DESTORY_NODE(node) \
+  do {                           \
+    nodesDestroyNode((node));    \
+    (node) = NULL;               \
+  } while (0)
+
 #define NODES_DESTORY_LIST(list) \
   do {                           \
     nodesDestroyList((list));    \
@@ -228,6 +234,7 @@ typedef enum ENodeType {
   QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN,
   QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN,
   QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN,
+  QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN,
   QUERY_NODE_PHYSICAL_PLAN_PROJECT,
   QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN,
   QUERY_NODE_PHYSICAL_PLAN_HASH_AGG,

include/libs/nodes/plannodes.h

@@ -62,7 +62,8 @@ typedef enum EScanType {
   SCAN_TYPE_STREAM,
   SCAN_TYPE_TABLE_MERGE,
   SCAN_TYPE_BLOCK_INFO,
-  SCAN_TYPE_LAST_ROW
+  SCAN_TYPE_LAST_ROW,
+  SCAN_TYPE_TABLE_COUNT
 } EScanType;
 
 typedef struct SScanLogicNode {
@@ -323,6 +324,8 @@ typedef struct SLastRowScanPhysiNode {
   bool           ignoreNull;
 } SLastRowScanPhysiNode;
 
+typedef SLastRowScanPhysiNode STableCountScanPhysiNode;
+
 typedef struct SSystemTableScanPhysiNode {
   SScanPhysiNode scan;
   SEpSet         mgmtEpSet;

include/libs/qcom/query.h

@@ -260,24 +260,26 @@ extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t
   (NEED_CLIENT_RM_TBLMETA_ERROR(_code) || NEED_CLIENT_REFRESH_VG_ERROR(_code) || \
    NEED_CLIENT_REFRESH_TBLMETA_ERROR(_code))
 
-#define SYNC_UNKNOWN_LEADER_REDIRECT_ERROR(_code) ((_code) == TSDB_CODE_SYN_NOT_LEADER || (_code) == TSDB_CODE_SYN_INTERNAL_ERROR)
+#define SYNC_UNKNOWN_LEADER_REDIRECT_ERROR(_code) ((_code) == TSDB_CODE_SYN_NOT_LEADER || (_code) == TSDB_CODE_SYN_INTERNAL_ERROR || (_code) == TSDB_CODE_VND_STOPPED)
 #define SYNC_SELF_LEADER_REDIRECT_ERROR(_code) ((_code) == TSDB_CODE_SYN_NOT_LEADER || (_code) == TSDB_CODE_SYN_INTERNAL_ERROR)
 #define SYNC_OTHER_LEADER_REDIRECT_ERROR(_code) (false)  // used later
 
-#define NEED_REDIRECT_ERROR(_code)                                                  \
+#define NEED_REDIRECT_ERROR(_code)                                                      \
-  ((_code) == TSDB_CODE_RPC_REDIRECT || (_code) == TSDB_CODE_RPC_NETWORK_UNAVAIL || \
+  ((_code) == TSDB_CODE_RPC_REDIRECT || (_code) == TSDB_CODE_RPC_NETWORK_UNAVAIL ||     \
-   (_code) == TSDB_CODE_NODE_NOT_DEPLOYED || SYNC_UNKNOWN_LEADER_REDIRECT_ERROR(_code) || \
+   (_code) == TSDB_CODE_MNODE_NOT_FOUND || SYNC_UNKNOWN_LEADER_REDIRECT_ERROR(_code) || \
-   SYNC_SELF_LEADER_REDIRECT_ERROR(_code) || SYNC_OTHER_LEADER_REDIRECT_ERROR(_code) ||   \
+   SYNC_SELF_LEADER_REDIRECT_ERROR(_code) || SYNC_OTHER_LEADER_REDIRECT_ERROR(_code) || \
-   (_code) == TSDB_CODE_APP_NOT_READY || (_code) == TSDB_CODE_RPC_BROKEN_LINK)
+   (_code) == TSDB_CODE_SYN_RESTORING || (_code) == TSDB_CODE_RPC_BROKEN_LINK ||        \
+   (_code) == TSDB_CODE_APP_IS_STARTING || (_code) == TSDB_CODE_APP_IS_STOPPING)
 
 #define NEED_CLIENT_RM_TBLMETA_REQ(_type)                                                                  \
   ((_type) == TDMT_VND_CREATE_TABLE || (_type) == TDMT_MND_CREATE_STB || (_type) == TDMT_VND_DROP_TABLE || \
    (_type) == TDMT_MND_DROP_STB)
 
-#define NEED_SCHEDULER_REDIRECT_ERROR(_code)                                      \
+#define NEED_SCHEDULER_REDIRECT_ERROR(_code)                                              \
-  ((_code) == TSDB_CODE_RPC_REDIRECT || (_code) == TSDB_CODE_NODE_NOT_DEPLOYED || \
+  ((_code) == TSDB_CODE_RPC_REDIRECT || (_code) == TSDB_CODE_MNODE_NOT_FOUND ||           \
    SYNC_UNKNOWN_LEADER_REDIRECT_ERROR(_code) || SYNC_SELF_LEADER_REDIRECT_ERROR(_code) || \
-   SYNC_OTHER_LEADER_REDIRECT_ERROR(_code) || (_code) == TSDB_CODE_APP_NOT_READY)
+   SYNC_OTHER_LEADER_REDIRECT_ERROR(_code) || (_code) == TSDB_CODE_SYN_RESTORING ||       \
+   (_code) == TSDB_CODE_APP_IS_STARTING || (_code) == TSDB_CODE_APP_IS_STOPPING)
 
 #define REQUEST_TOTAL_EXEC_TIMES 2
 

include/libs/stream/tstream.h

@@ -275,31 +275,6 @@ typedef struct {
   SEpSet  epSet;
 } SStreamChildEpInfo;
 
-typedef struct {
-  int32_t srcNodeId;
-  int32_t srcChildId;
-  int64_t stateSaveVer;
-  int64_t stateProcessedVer;
-} SStreamCheckpointInfo;
-
-typedef struct {
-  int64_t streamId;
-  int64_t checkTs;
-  int32_t checkpointId;  // incremental
-  int32_t taskId;
-  SArray* checkpointVer;  // SArray<SStreamCheckpointInfo>
-} SStreamMultiVgCheckpointInfo;
-
-typedef struct {
-  int32_t taskId;
-  int32_t checkpointId;  // incremental
-} SStreamCheckpointKey;
-
-typedef struct {
-  int32_t taskId;
-  SArray* checkpointVer;
-} SStreamRecoveringState;
-
 typedef struct SStreamTask {
   int64_t streamId;
   int32_t taskId;
@@ -364,6 +339,10 @@ typedef struct SStreamTask {
   int64_t checkReqId;
   SArray* checkReqIds;  // shuffle
   int32_t refCnt;
+
+  int64_t checkpointingId;
+  int32_t checkpointAlignCnt;
+
 } SStreamTask;
 
 int32_t tEncodeStreamEpInfo(SEncoder* pEncoder, const SStreamChildEpInfo* pInfo);
@@ -509,6 +488,60 @@ typedef struct {
 int32_t tEncodeSStreamRecoverFinishReq(SEncoder* pEncoder, const SStreamRecoverFinishReq* pReq);
 int32_t tDecodeSStreamRecoverFinishReq(SDecoder* pDecoder, SStreamRecoverFinishReq* pReq);
 
+typedef struct {
+  int64_t streamId;
+  int64_t checkpointId;
+  int32_t taskId;
+  int32_t nodeId;
+  int64_t expireTime;
+} SStreamCheckpointSourceReq;
+
+typedef struct {
+  int64_t streamId;
+  int64_t checkpointId;
+  int32_t taskId;
+  int32_t nodeId;
+  int64_t expireTime;
+} SStreamCheckpointSourceRsp;
+
+int32_t tEncodeSStreamCheckpointSourceReq(SEncoder* pEncoder, const SStreamCheckpointSourceReq* pReq);
+int32_t tDecodeSStreamCheckpointSourceReq(SDecoder* pDecoder, SStreamCheckpointSourceReq* pReq);
+
+int32_t tEncodeSStreamCheckpointSourceRsp(SEncoder* pEncoder, const SStreamCheckpointSourceRsp* pRsp);
+int32_t tDecodeSStreamCheckpointSourceRsp(SDecoder* pDecoder, SStreamCheckpointSourceRsp* pRsp);
+
+typedef struct {
+  SMsgHead msgHead;
+  int64_t  streamId;
+  int64_t  checkpointId;
+  int32_t  downstreamTaskId;
+  int32_t  downstreamNodeId;
+  int32_t  upstreamTaskId;
+  int32_t  upstreamNodeId;
+  int32_t  childId;
+  int64_t  expireTime;
+  int8_t   taskLevel;
+} SStreamCheckpointReq;
+
+typedef struct {
+  SMsgHead msgHead;
+  int64_t  streamId;
+  int64_t  checkpointId;
+  int32_t  downstreamTaskId;
+  int32_t  downstreamNodeId;
+  int32_t  upstreamTaskId;
+  int32_t  upstreamNodeId;
+  int32_t  childId;
+  int64_t  expireTime;
+  int8_t   taskLevel;
+} SStreamCheckpointRsp;
+
+int32_t tEncodeSStreamCheckpointReq(SEncoder* pEncoder, const SStreamCheckpointReq* pReq);
+int32_t tDecodeSStreamCheckpointReq(SDecoder* pDecoder, SStreamCheckpointReq* pReq);
+
+int32_t tEncodeSStreamCheckpointRsp(SEncoder* pEncoder, const SStreamCheckpointRsp* pRsp);
+int32_t tDecodeSStreamCheckpointRsp(SDecoder* pDecoder, SStreamCheckpointRsp* pRsp);
+
 typedef struct {
   int64_t streamId;
   int32_t downstreamTaskId;
@@ -598,18 +631,22 @@ void         streamMetaClose(SStreamMeta* streamMeta);
 
 int32_t      streamMetaAddTask(SStreamMeta* pMeta, int64_t ver, SStreamTask* pTask);
 int32_t      streamMetaAddSerializedTask(SStreamMeta* pMeta, int64_t startVer, char* msg, int32_t msgLen);
-int32_t      streamMetaRemoveTask(SStreamMeta* pMeta, int32_t taskId);
 SStreamTask* streamMetaGetTask(SStreamMeta* pMeta, int32_t taskId);
 
 SStreamTask* streamMetaAcquireTask(SStreamMeta* pMeta, int32_t taskId);
 void         streamMetaReleaseTask(SStreamMeta* pMeta, SStreamTask* pTask);
-void         streamMetaRemoveTask1(SStreamMeta* pMeta, int32_t taskId);
+void         streamMetaRemoveTask(SStreamMeta* pMeta, int32_t taskId);
 
 int32_t streamMetaBegin(SStreamMeta* pMeta);
 int32_t streamMetaCommit(SStreamMeta* pMeta);
 int32_t streamMetaRollBack(SStreamMeta* pMeta);
 int32_t streamLoadTasks(SStreamMeta* pMeta);
 
+// checkpoint
+int32_t streamProcessCheckpointSourceReq(SStreamMeta* pMeta, SStreamTask* pTask, SStreamCheckpointSourceReq* pReq);
+int32_t streamProcessCheckpointReq(SStreamMeta* pMeta, SStreamTask* pTask, SStreamCheckpointReq* pReq);
+int32_t streamProcessCheckpointRsp(SStreamMeta* pMeta, SStreamTask* pTask, SStreamCheckpointRsp* pRsp);
+
 #ifdef __cplusplus
 }
 #endif

include/util/taoserror.h

@@ -40,21 +40,37 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_FAILED                    -1   // unknown or needn't tell detail error
 
 // rpc
+// #define TSDB_CODE_RPC_ACTION_IN_PROGRESS     TAOS_DEF_ERROR_CODE(0, 0x0001) //2.x
+// #define TSDB_CODE_RPC_AUTH_REQUIRED          TAOS_DEF_ERROR_CODE(0, 0x0002) //2.x
 #define TSDB_CODE_RPC_AUTH_FAILURE              TAOS_DEF_ERROR_CODE(0, 0x0003)
 #define TSDB_CODE_RPC_REDIRECT                  TAOS_DEF_ERROR_CODE(0, 0x0004)
+// #define TSDB_CODE_RPC_NOT_READY              TAOS_DEF_ERROR_CODE(0, 0x0005) //2.x
+// #define TSDB_CODE_RPC_ALREADY_PROCESSED      TAOS_DEF_ERROR_CODE(0, 0x0006) //2.x
+// #define TSDB_CODE_RPC_LAST_SESSION_NOT_FINI. TAOS_DEF_ERROR_CODE(0, 0x0007) //2.x
+// #define TSDB_CODE_RPC_MISMATCHED_LINK_ID     TAOS_DEF_ERROR_CODE(0, 0x0008) //2.x
+// #define TSDB_CODE_RPC_TOO_SLOW               TAOS_DEF_ERROR_CODE(0, 0x0009) //2.x
+// #define TSDB_CODE_RPC_MAX_SESSIONS           TAOS_DEF_ERROR_CODE(0, 0x000A) //2.x
 #define TSDB_CODE_RPC_NETWORK_UNAVAIL           TAOS_DEF_ERROR_CODE(0, 0x000B)
+// #define TSDB_CODE_RPC_APP_ERROR              TAOS_DEF_ERROR_CODE(0, 0x000C) //2.x
+// #define TSDB_CODE_RPC_UNEXPECTED_RESPONSE    TAOS_DEF_ERROR_CODE(0, 0x000D) //2.x
+// #define TSDB_CODE_RPC_INVALID_VALUE          TAOS_DEF_ERROR_CODE(0, 0x000E) //2.x
+// #define TSDB_CODE_RPC_INVALID_TRAN_ID        TAOS_DEF_ERROR_CODE(0, 0x000F) //2.x
+// #define TSDB_CODE_RPC_INVALID_SESSION_ID     TAOS_DEF_ERROR_CODE(0, 0x0010) //2.x
+// #define TSDB_CODE_RPC_INVALID_MSG_TYPE       TAOS_DEF_ERROR_CODE(0, 0x0011) //2.x
+// #define TSDB_CODE_RPC_INVALID_RESPONSE_TYPE  TAOS_DEF_ERROR_CODE(0, 0x0012) //2.x
+#define TSDB_CODE_TIME_UNSYNCED                 TAOS_DEF_ERROR_CODE(0, 0x0013)
+// #define TSDB_CODE_APP_NOT_READY              TAOS_DEF_ERROR_CODE(0, 0x0014) //2.x
 #define TSDB_CODE_RPC_FQDN_ERROR                TAOS_DEF_ERROR_CODE(0, 0x0015)
+// #define TSDB_CODE_RPC_INVALID_VERSION        TAOS_DEF_ERROR_CODE(0, 0x0016) //2.x
 #define TSDB_CODE_RPC_PORT_EADDRINUSE           TAOS_DEF_ERROR_CODE(0, 0x0017)
 #define TSDB_CODE_RPC_BROKEN_LINK               TAOS_DEF_ERROR_CODE(0, 0x0018)
 #define TSDB_CODE_RPC_TIMEOUT                   TAOS_DEF_ERROR_CODE(0, 0x0019)
 
 //common & util
-#define TSDB_CODE_TIME_UNSYNCED                 TAOS_DEF_ERROR_CODE(0, 0x0013)
-#define TSDB_CODE_APP_NOT_READY                 TAOS_DEF_ERROR_CODE(0, 0x0014)
-
 #define TSDB_CODE_OPS_NOT_SUPPORT               TAOS_DEF_ERROR_CODE(0, 0x0100)
 #define TSDB_CODE_MEMORY_CORRUPTED              TAOS_DEF_ERROR_CODE(0, 0x0101)
 #define TSDB_CODE_OUT_OF_MEMORY                 TAOS_DEF_ERROR_CODE(0, 0x0102)
+// #define TSDB_CODE_COM_INVALID_CFG_MSG        TAOS_DEF_ERROR_CODE(0, 0x0103) // 2.x
 #define TSDB_CODE_FILE_CORRUPTED                TAOS_DEF_ERROR_CODE(0, 0x0104)
 #define TSDB_CODE_REF_NO_MEMORY                 TAOS_DEF_ERROR_CODE(0, 0x0105)
 #define TSDB_CODE_REF_FULL                      TAOS_DEF_ERROR_CODE(0, 0x0106)
@@ -69,7 +85,7 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_OUT_OF_SHM_MEM                TAOS_DEF_ERROR_CODE(0, 0x0113)
 #define TSDB_CODE_INVALID_SHM_ID                TAOS_DEF_ERROR_CODE(0, 0x0114)
 #define TSDB_CODE_INVALID_MSG                   TAOS_DEF_ERROR_CODE(0, 0x0115)
-#define TSDB_CODE_INVALID_MSG_LEN               TAOS_DEF_ERROR_CODE(0, 0x0116)
+#define TSDB_CODE_INVALID_MSG_LEN               TAOS_DEF_ERROR_CODE(0, 0x0116) //
 #define TSDB_CODE_INVALID_PTR                   TAOS_DEF_ERROR_CODE(0, 0x0117)
 #define TSDB_CODE_INVALID_PARA                  TAOS_DEF_ERROR_CODE(0, 0x0118)
 #define TSDB_CODE_INVALID_CFG                   TAOS_DEF_ERROR_CODE(0, 0x0119)
@@ -81,7 +97,7 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_CHECKSUM_ERROR                TAOS_DEF_ERROR_CODE(0, 0x011F)
 
 #define TSDB_CODE_COMPRESS_ERROR                TAOS_DEF_ERROR_CODE(0, 0x0120)
-#define TSDB_CODE_MSG_NOT_PROCESSED             TAOS_DEF_ERROR_CODE(0, 0x0121)
+#define TSDB_CODE_MSG_NOT_PROCESSED             TAOS_DEF_ERROR_CODE(0, 0x0121) //
 #define TSDB_CODE_CFG_NOT_FOUND                 TAOS_DEF_ERROR_CODE(0, 0x0122)
 #define TSDB_CODE_REPEAT_INIT                   TAOS_DEF_ERROR_CODE(0, 0x0123)
 #define TSDB_CODE_DUP_KEY                       TAOS_DEF_ERROR_CODE(0, 0x0124)
@@ -94,6 +110,9 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_NO_DISKSPACE                  TAOS_DEF_ERROR_CODE(0, 0x012B)
 #define TSDB_CODE_TIMEOUT_ERROR                 TAOS_DEF_ERROR_CODE(0, 0x012C)
 
+#define TSDB_CODE_APP_IS_STARTING               TAOS_DEF_ERROR_CODE(0, 0x0130) //
+#define TSDB_CODE_APP_IS_STOPPING               TAOS_DEF_ERROR_CODE(0, 0x0131) //
+
 //client
 #define TSDB_CODE_TSC_INVALID_OPERATION         TAOS_DEF_ERROR_CODE(0, 0x0200)
 #define TSDB_CODE_TSC_INVALID_QHANDLE           TAOS_DEF_ERROR_CODE(0, 0x0201)
@@ -279,6 +298,7 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_MND_TRANS_CONFLICT            TAOS_DEF_ERROR_CODE(0, 0x03D3)
 #define TSDB_CODE_MND_TRANS_CLOG_IS_NULL        TAOS_DEF_ERROR_CODE(0, 0x03D4)
 #define TSDB_CODE_MND_TRANS_NETWORK_UNAVAILL    TAOS_DEF_ERROR_CODE(0, 0x03D5)
+#define TSDB_CODE_MND_LAST_TRANS_NOT_FINISHED   TAOS_DEF_ERROR_CODE(0, 0x03D6) //internal
 #define TSDB_CODE_MND_TRANS_UNKNOW_ERROR        TAOS_DEF_ERROR_CODE(0, 0x03DF)
 
 // mnode-mq
@@ -311,19 +331,56 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_MND_INVALID_SMA_OPTION        TAOS_DEF_ERROR_CODE(0, 0x0482)
 
 // dnode
-#define TSDB_CODE_NODE_OFFLINE                  TAOS_DEF_ERROR_CODE(0, 0x0408)
+// #define TSDB_CODE_DND_MSG_NOT_PROCESSED      TAOS_DEF_ERROR_CODE(0, 0x0400) // 2.x
-#define TSDB_CODE_NODE_ALREADY_DEPLOYED         TAOS_DEF_ERROR_CODE(0, 0x0409)
+// #define TSDB_CODE_DND_OUT_OF_MEMORY          TAOS_DEF_ERROR_CODE(0, 0x0401) // 2.x
-#define TSDB_CODE_NODE_NOT_DEPLOYED             TAOS_DEF_ERROR_CODE(0, 0x040A)
+// #define TSDB_CODE_DND_NO_WRITE_ACCESS        TAOS_DEF_ERROR_CODE(0, 0x0402) // 2.x
+// #define TSDB_CODE_DND_INVALID_MSG_LEN        TAOS_DEF_ERROR_CODE(0, 0x0403) // 2.x
+// #define TSDB_CODE_DND_ACTION_IN_PROGRESS     TAOS_DEF_ERROR_CODE(0, 0x0404) // 2.x
+// #define TSDB_CODE_DND_TOO_MANY_VNODES        TAOS_DEF_ERROR_CODE(0, 0x0405) // 2.x
+// #define TSDB_CODE_DND_EXITING                TAOS_DEF_ERROR_CODE(0, 0x0406) // 2.x
+// #define TSDB_CODE_DND_VNODE_OPEN_FAILED      TAOS_DEF_ERROR_CODE(0, 0x0407) // 2.x
+#define TSDB_CODE_DNODE_OFFLINE                 TAOS_DEF_ERROR_CODE(0, 0x0408)
+#define TSDB_CODE_MNODE_ALREADY_DEPLOYED        TAOS_DEF_ERROR_CODE(0, 0x0409)
+#define TSDB_CODE_MNODE_NOT_FOUND               TAOS_DEF_ERROR_CODE(0, 0x040A)
+#define TSDB_CODE_MNODE_NOT_DEPLOYED            TAOS_DEF_ERROR_CODE(0, 0x040B)
+#define TSDB_CODE_QNODE_ALREADY_DEPLOYED        TAOS_DEF_ERROR_CODE(0, 0x040C)
+#define TSDB_CODE_QNODE_NOT_FOUND               TAOS_DEF_ERROR_CODE(0, 0x040D)
+#define TSDB_CODE_QNODE_NOT_DEPLOYED            TAOS_DEF_ERROR_CODE(0, 0x040E)
+#define TSDB_CODE_SNODE_ALREADY_DEPLOYED        TAOS_DEF_ERROR_CODE(0, 0x040F)
+#define TSDB_CODE_SNODE_NOT_FOUND               TAOS_DEF_ERROR_CODE(0, 0x0410)
+#define TSDB_CODE_SNODE_NOT_DEPLOYED            TAOS_DEF_ERROR_CODE(0, 0x0411)
 
 // vnode
+// #define TSDB_CODE_VND_ACTION_IN_PROGRESS     TAOS_DEF_ERROR_CODE(0, 0x0500) // 2.x
+// #define TSDB_CODE_VND_MSG_NOT_PROCESSED      TAOS_DEF_ERROR_CODE(0, 0x0501) // 2.x
+// #define TSDB_CODE_VND_ACTION_NEED_REPROCESS. TAOS_DEF_ERROR_CODE(0, 0x0502) // 2.x
 #define TSDB_CODE_VND_INVALID_VGROUP_ID         TAOS_DEF_ERROR_CODE(0, 0x0503)
+// #define TSDB_CODE_VND_INIT_FAILED            TAOS_DEF_ERROR_CODE(0, 0x0504) // 2.x
+// #define TSDB_CODE_VND_NO_DISKSPACE           TAOS_DEF_ERROR_CODE(0, 0x0505) // 2.x
+// #define TSDB_CODE_VND_NO_DISK_PERMISSIONS    TAOS_DEF_ERROR_CODE(0, 0x0506) // 2.x
+// #define TSDB_CODE_VND_NO_SUCH_FILE_OR_DIR    TAOS_DEF_ERROR_CODE(0, 0x0507) // 2.x
+// #define TSDB_CODE_VND_OUT_OF_MEMORY          TAOS_DEF_ERROR_CODE(0, 0x0508) // 2.x
+// #define TSDB_CODE_VND_APP_ERROR              TAOS_DEF_ERROR_CODE(0, 0x0509) // 2.x
+// #define TSDB_CODE_VND_INVALID_VRESION_FILE   TAOS_DEF_ERROR_CODE(0, 0x050A) // 2.x
+// #define TSDB_CODE_VND_IS_FULL                TAOS_DEF_ERROR_CODE(0, 0x050B) // 2.x
+// #define TSDB_CODE_VND_IS_FLOWCTRL            TAOS_DEF_ERROR_CODE(0, 0x050C) // 2.x
+// #define TSDB_CODE_VND_IS_DROPPING            TAOS_DEF_ERROR_CODE(0, 0x050D) // 2.x
+// #define TSDB_CODE_VND_IS_BALANCING           TAOS_DEF_ERROR_CODE(0, 0x050E) // 2.x
+// #define TSDB_CODE_VND_IS_CLOSING             TAOS_DEF_ERROR_CODE(0, 0x0510) // 2.x
+// #define TSDB_CODE_VND_NOT_SYNCED             TAOS_DEF_ERROR_CODE(0, 0x0511) // 2.x
 #define TSDB_CODE_VND_NO_WRITE_AUTH             TAOS_DEF_ERROR_CODE(0, 0x0512)
+// #define TSDB_CODE_VND_IS_SYNCING             TAOS_DEF_ERROR_CODE(0, 0x0513) // 2.x
+// #define TSDB_CODE_VND_INVALID_TSDB_STATE     TAOS_DEF_ERROR_CODE(0, 0x0514) // 2.x
+// #define TSDB_CODE_WAIT_THREAD_TOO_MANY       TAOS_DEF_ERROR_CODE(0, 0x0515) // 2.x
+#define TSDB_CODE_VND_NOT_EXIST                 TAOS_DEF_ERROR_CODE(0, 0x0520) // internal
+#define TSDB_CODE_VND_ALREADY_EXIST             TAOS_DEF_ERROR_CODE(0, 0x0521) // internal
 #define TSDB_CODE_VND_HASH_MISMATCH             TAOS_DEF_ERROR_CODE(0, 0x0522)
 #define TSDB_CODE_VND_INVALID_TABLE_ACTION      TAOS_DEF_ERROR_CODE(0, 0x0524)
 #define TSDB_CODE_VND_COL_ALREADY_EXISTS        TAOS_DEF_ERROR_CODE(0, 0x0525)
 #define TSDB_CODE_VND_COL_NOT_EXISTS            TAOS_DEF_ERROR_CODE(0, 0x0526)
 #define TSDB_CODE_VND_COL_SUBSCRIBED            TAOS_DEF_ERROR_CODE(0, 0x0527)
 #define TSDB_CODE_VND_NO_AVAIL_BUFPOOL          TAOS_DEF_ERROR_CODE(0, 0x0528)
+#define TSDB_CODE_VND_STOPPED                   TAOS_DEF_ERROR_CODE(0, 0x0529)
 
 // tsdb
 #define TSDB_CODE_TDB_INVALID_TABLE_ID          TAOS_DEF_ERROR_CODE(0, 0x0600)

source/common/src/tglobal.c

@@ -167,6 +167,7 @@ int64_t tsWalFsyncDataSizeLimit = (100 * 1024 * 1024L);
 // internal
 int32_t tsTransPullupInterval = 2;
 int32_t tsMqRebalanceInterval = 2;
+int32_t tsStreamCheckpointTickInterval = 1;
 int32_t tsTtlUnit = 86400;
 int32_t tsTtlPushInterval = 86400;
 int32_t tsGrantHBInterval = 60;

source/common/src/tmsg.c

@@ -3748,6 +3748,31 @@ int32_t tDeserializeSMTimerMsg(void *buf, int32_t bufLen, SMTimerReq *pReq) {
   return 0;
 }
 
+int32_t tSerializeSMStreamTickMsg(void *buf, int32_t bufLen, SMStreamTickReq *pReq) {
+  SEncoder encoder = {0};
+  tEncoderInit(&encoder, buf, bufLen);
+
+  if (tStartEncode(&encoder) < 0) return -1;
+  if (tEncodeI64(&encoder, pReq->tick) < 0) return -1;
+  tEndEncode(&encoder);
+
+  int32_t tlen = encoder.pos;
+  tEncoderClear(&encoder);
+  return tlen;
+}
+
+int32_t tDeserializeSMStreamTickMsg(void *buf, int32_t bufLen, SMStreamTickReq *pReq) {
+  SDecoder decoder = {0};
+  tDecoderInit(&decoder, buf, bufLen);
+
+  if (tStartDecode(&decoder) < 0) return -1;
+  if (tDecodeI64(&decoder, &pReq->tick) < 0) return -1;
+  tEndDecode(&decoder);
+
+  tDecoderClear(&decoder);
+  return 0;
+}
+
 int32_t tEncodeSReplica(SEncoder *pEncoder, SReplica *pReplica) {
   if (tEncodeI32(pEncoder, pReplica->id) < 0) return -1;
   if (tEncodeU16(pEncoder, pReplica->port) < 0) return -1;

source/dnode/mgmt/mgmt_dnode/src/dmWorker.c

@@ -144,6 +144,7 @@ static void dmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
       break;
     default:
       terrno = TSDB_CODE_MSG_NOT_PROCESSED;
+      dGError("msg:%p, not processed in mgmt queue", pMsg);
       break;
   }
 

source/dnode/mgmt/mgmt_vnode/src/vmHandle.c

@@ -216,7 +216,7 @@ int32_t vmProcessCreateVnodeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
     dDebug("vgId:%d, already exist", req.vgId);
     tFreeSCreateVnodeReq(&req);
     vmReleaseVnode(pMgmt, pVnode);
-    terrno = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+    terrno = TSDB_CODE_VND_ALREADY_EXIST;
     code = terrno;
     return 0;
   }
@@ -307,7 +307,7 @@ int32_t vmProcessAlterVnodeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
   SVnodeObj *pVnode = vmAcquireVnode(pMgmt, vgId);
   if (pVnode == NULL) {
     dError("vgId:%d, failed to alter replica since %s", vgId, terrstr());
-    terrno = TSDB_CODE_NODE_NOT_DEPLOYED;
+    terrno = TSDB_CODE_VND_NOT_EXIST;
     return -1;
   }
 
@@ -369,7 +369,7 @@ int32_t vmProcessDropVnodeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
   SVnodeObj *pVnode = vmAcquireVnode(pMgmt, vgId);
   if (pVnode == NULL) {
     dDebug("vgId:%d, failed to drop since %s", vgId, terrstr());
-    terrno = TSDB_CODE_NODE_NOT_DEPLOYED;
+    terrno = TSDB_CODE_VND_NOT_EXIST;
     return -1;
   }
 

source/dnode/mgmt/node_mgmt/src/dmEnv.c

@@ -152,7 +152,19 @@ static int32_t dmProcessCreateNodeReq(EDndNodeType ntype, SRpcMsg *pMsg) {
   SMgmtWrapper *pWrapper = dmAcquireWrapper(pDnode, ntype);
   if (pWrapper != NULL) {
     dmReleaseWrapper(pWrapper);
-    terrno = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+    switch (ntype) {
+      case MNODE:
+        terrno = TSDB_CODE_MNODE_ALREADY_DEPLOYED;
+        break;
+      case QNODE:
+        terrno = TSDB_CODE_QNODE_ALREADY_DEPLOYED;
+        break;
+      case SNODE:
+        terrno = TSDB_CODE_SNODE_ALREADY_DEPLOYED;
+        break;
+      default:
+        terrno = TSDB_CODE_APP_ERROR;
+    }
     dError("failed to create node since %s", terrstr());
     return -1;
   }
@@ -191,7 +203,20 @@ static int32_t dmProcessDropNodeReq(EDndNodeType ntype, SRpcMsg *pMsg) {
 
   SMgmtWrapper *pWrapper = dmAcquireWrapper(pDnode, ntype);
   if (pWrapper == NULL) {
-    terrno = TSDB_CODE_NODE_NOT_DEPLOYED;
+    switch (ntype) {
+      case MNODE:
+        terrno = TSDB_CODE_MNODE_NOT_DEPLOYED;
+        break;
+      case QNODE:
+        terrno = TSDB_CODE_QNODE_NOT_DEPLOYED;
+        break;
+      case SNODE:
+        terrno = TSDB_CODE_SNODE_NOT_DEPLOYED;
+        break;
+      default:
+        terrno = TSDB_CODE_APP_ERROR;
+    }
+
     dError("failed to drop node since %s", terrstr());
     return -1;
   }

source/dnode/mgmt/node_mgmt/src/dmMgmt.c

@@ -189,7 +189,6 @@ SMgmtWrapper *dmAcquireWrapper(SDnode *pDnode, EDndNodeType ntype) {
     int32_t refCount = atomic_add_fetch_32(&pWrapper->refCount, 1);
     // dTrace("node:%s, is acquired, ref:%d", pWrapper->name, refCount);
   } else {
-    terrno = TSDB_CODE_NODE_NOT_DEPLOYED;
     pRetWrapper = NULL;
   }
   taosThreadRwlockUnlock(&pWrapper->lock);
@@ -205,7 +204,20 @@ int32_t dmMarkWrapper(SMgmtWrapper *pWrapper) {
     int32_t refCount = atomic_add_fetch_32(&pWrapper->refCount, 1);
     // dTrace("node:%s, is marked, ref:%d", pWrapper->name, refCount);
   } else {
-    terrno = TSDB_CODE_NODE_NOT_DEPLOYED;
+    switch (pWrapper->ntype) {
+      case MNODE:
+        terrno = TSDB_CODE_MNODE_NOT_FOUND;
+        break;
+      case QNODE:
+        terrno = TSDB_CODE_QNODE_NOT_FOUND;
+        break;
+      case SNODE:
+        terrno = TSDB_CODE_SNODE_NOT_FOUND;
+        break;
+      default:
+        terrno = TSDB_CODE_APP_IS_STOPPING;
+        break;
+    }
     code = -1;
   }
   taosThreadRwlockUnlock(&pWrapper->lock);

source/dnode/mgmt/node_mgmt/src/dmTransport.c

@@ -51,13 +51,15 @@ static inline void dmSendRedirectRsp(SRpcMsg *pMsg, const SEpSet *pNewEpSet) {
 }
 
 int32_t dmProcessNodeMsg(SMgmtWrapper *pWrapper, SRpcMsg *pMsg) {
+  const STraceId *trace = &pMsg->info.traceId;
+
   NodeMsgFp msgFp = pWrapper->msgFps[TMSG_INDEX(pMsg->msgType)];
   if (msgFp == NULL) {
     terrno = TSDB_CODE_MSG_NOT_PROCESSED;
+    dGError("msg:%p, not processed since no handler", pMsg);
     return -1;
   }
 
-  const STraceId *trace = &pMsg->info.traceId;
   dGTrace("msg:%p, will be processed by %s", pMsg, pWrapper->name);
   pMsg->info.wrapper = pWrapper;
   return (*msgFp)(pWrapper->pMgmt, pMsg);
@@ -99,18 +101,23 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
       dmProcessServerStartupStatus(pDnode, pRpc);
       return;
     } else {
-      terrno = TSDB_CODE_APP_NOT_READY;
+      if (pDnode->status == DND_STAT_INIT) {
+        terrno = TSDB_CODE_APP_IS_STARTING;
+      } else {
+        terrno = TSDB_CODE_APP_IS_STOPPING;
+      }
       goto _OVER;
     }
   }
 
-  if (IsReq(pRpc) && pRpc->pCont == NULL) {
+  if (pRpc->pCont == NULL && (IsReq(pRpc) || pRpc->contLen != 0)) {
     dGError("msg:%p, type:%s pCont is NULL", pRpc, TMSG_INFO(pRpc->msgType));
     terrno = TSDB_CODE_INVALID_MSG_LEN;
     goto _OVER;
   }
 
   if (pHandle->defaultNtype == NODE_END) {
+    dGError("msg:%p, type:%s not processed since no handle", pRpc, TMSG_INFO(pRpc->msgType));
     terrno = TSDB_CODE_MSG_NOT_PROCESSED;
     goto _OVER;
   }
@@ -165,8 +172,7 @@ _OVER:
 
     if (IsReq(pRpc)) {
       SRpcMsg rsp = {.code = code, .info = pRpc->info};
-      if ((code == TSDB_CODE_NODE_NOT_DEPLOYED || code == TSDB_CODE_APP_NOT_READY) && pRpc->msgType > TDMT_MND_MSG &&
+      if (code == TSDB_CODE_MNODE_NOT_FOUND) {
-          pRpc->msgType < TDMT_VND_MSG) {
         dmBuildMnodeRedirectRsp(pDnode, &rsp);
       }
 
@@ -219,7 +225,11 @@ static inline int32_t dmSendReq(const SEpSet *pEpSet, SRpcMsg *pMsg) {
   if (pDnode->status != DND_STAT_RUNNING && pMsg->msgType < TDMT_SYNC_MSG) {
     rpcFreeCont(pMsg->pCont);
     pMsg->pCont = NULL;
-    terrno = TSDB_CODE_NODE_OFFLINE;
+    if (pDnode->status == DND_STAT_INIT) {
+      terrno = TSDB_CODE_APP_IS_STARTING;
+    } else {
+      terrno = TSDB_CODE_APP_IS_STOPPING;
+    }
     dError("failed to send rpc msg:%s since %s, handle:%p", TMSG_INFO(pMsg->msgType), terrstr(), pMsg->info.handle);
     return -1;
   } else {
@@ -233,8 +243,9 @@ static inline void dmRegisterBrokenLinkArg(SRpcMsg *pMsg) { rpcRegisterBrokenLin
 static inline void dmReleaseHandle(SRpcHandleInfo *pHandle, int8_t type) { rpcReleaseHandle(pHandle, type); }
 
 static bool rpcRfp(int32_t code, tmsg_t msgType) {
-  if (code == TSDB_CODE_RPC_REDIRECT || code == TSDB_CODE_RPC_NETWORK_UNAVAIL || code == TSDB_CODE_NODE_NOT_DEPLOYED ||
+  if (code == TSDB_CODE_RPC_REDIRECT || code == TSDB_CODE_RPC_NETWORK_UNAVAIL || code == TSDB_CODE_MNODE_NOT_FOUND ||
-      code == TSDB_CODE_SYN_NOT_LEADER || code == TSDB_CODE_APP_NOT_READY || code == TSDB_CODE_RPC_BROKEN_LINK) {
+      code == TSDB_CODE_SYN_NOT_LEADER || code == TSDB_CODE_SYN_RESTORING || code == TSDB_CODE_RPC_BROKEN_LINK ||
+      code == TSDB_CODE_VND_STOPPED || code == TSDB_CODE_APP_IS_STARTING || code == TSDB_CODE_APP_IS_STOPPING) {
     if (msgType == TDMT_SCH_QUERY || msgType == TDMT_SCH_MERGE_QUERY || msgType == TDMT_SCH_FETCH ||
         msgType == TDMT_SCH_MERGE_FETCH) {
       return false;

source/dnode/mgmt/test/mnode/dmnode.cpp

@@ -40,7 +40,7 @@ TEST_F(DndTestMnode, 01_Create_Mnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_MNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_MNODE_ALREADY_DEPLOYED);
   }
 
   {
@@ -57,7 +57,7 @@ TEST_F(DndTestMnode, 01_Create_Mnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_MNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_MNODE_ALREADY_DEPLOYED);
   }
 
   {
@@ -77,7 +77,7 @@ TEST_F(DndTestMnode, 01_Create_Mnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_MNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_MNODE_ALREADY_DEPLOYED);
   }
 }
 
@@ -171,7 +171,7 @@ TEST_F(DndTestMnode, 03_Drop_Mnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_DROP_MNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_MNODE_NOT_DEPLOYED);
   }
 
   {
@@ -188,7 +188,7 @@ TEST_F(DndTestMnode, 03_Drop_Mnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_MND_ALTER_MNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_MNODE_NOT_DEPLOYED);
   }
 
   {

source/dnode/mgmt/test/qnode/dqnode.cpp

@@ -62,7 +62,7 @@ TEST_F(DndTestQnode, 01_Create_Qnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_QNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_QNODE_ALREADY_DEPLOYED);
   }
 
   test.Restart();
@@ -77,7 +77,7 @@ TEST_F(DndTestQnode, 01_Create_Qnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_QNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_QNODE_ALREADY_DEPLOYED);
   }
 }
 
@@ -120,7 +120,7 @@ TEST_F(DndTestQnode, 02_Drop_Qnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_DROP_QNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_QNODE_NOT_DEPLOYED);
   }
 
   test.Restart();
@@ -135,7 +135,7 @@ TEST_F(DndTestQnode, 02_Drop_Qnode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_DROP_QNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_QNODE_NOT_DEPLOYED);
   }
 
   {

source/dnode/mgmt/test/snode/dsnode.cpp

@@ -62,7 +62,7 @@ TEST_F(DndTestSnode, 01_Create_Snode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_SNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_SNODE_ALREADY_DEPLOYED);
   }
 
   test.Restart();
@@ -77,7 +77,7 @@ TEST_F(DndTestSnode, 01_Create_Snode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_CREATE_SNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_SNODE_ALREADY_DEPLOYED);
   }
 }
 
@@ -120,7 +120,7 @@ TEST_F(DndTestSnode, 01_Drop_Snode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_DROP_SNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_SNODE_NOT_DEPLOYED);
   }
 
   test.Restart();
@@ -135,7 +135,7 @@ TEST_F(DndTestSnode, 01_Drop_Snode) {
 
     SRpcMsg* pRsp = test.SendReq(TDMT_DND_DROP_SNODE, pReq, contLen);
     ASSERT_NE(pRsp, nullptr);
-    ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_SNODE_NOT_DEPLOYED);
   }
 
   {

source/dnode/mgmt/test/vnode/vnode.cpp

@@ -63,7 +63,7 @@ TEST_F(DndTestVnode, 01_Create_Vnode) {
       ASSERT_EQ(pRsp->code, 0);
       test.Restart();
     } else {
-      ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_ALREADY_DEPLOYED);
+      ASSERT_EQ(pRsp->code, TSDB_CODE_VND_ALREADY_EXIST);
     }
   }
 }
@@ -285,7 +285,7 @@ TEST_F(DndTestVnode, 06_Drop_Vnode) {
       ASSERT_EQ(pRsp->code, 0);
       test.Restart();
     } else {
-      ASSERT_EQ(pRsp->code, TSDB_CODE_NODE_NOT_DEPLOYED);
+      ASSERT_EQ(pRsp->code, TSDB_CODE_VND_NOT_EXIST);
     }
   }
 }

source/dnode/mnode/impl/inc/mndDef.h

@@ -640,10 +640,14 @@ typedef struct {
   SArray*        tasks;  // SArray<SArray<SStreamTask>>
   SSchemaWrapper outputSchema;
   SSchemaWrapper tagSchema;
+
+  // 3.0.20
+  int64_t checkpointFreq;  // ms
+  int64_t currentTick;     // do not serialize
 } SStreamObj;
 
 int32_t tEncodeSStreamObj(SEncoder* pEncoder, const SStreamObj* pObj);
-int32_t tDecodeSStreamObj(SDecoder* pDecoder, SStreamObj* pObj);
+int32_t tDecodeSStreamObj(SDecoder* pDecoder, SStreamObj* pObj, int32_t sver);
 void    tFreeStreamObj(SStreamObj* pObj);
 
 typedef struct {
@@ -653,15 +657,6 @@ typedef struct {
   SArray* childInfo;  // SArray<SStreamChildEpInfo>
 } SStreamCheckpointObj;
 
-#if 0
-typedef struct {
-  int64_t uid;
-  int64_t streamId;
-  int8_t  status;
-  int8_t  stage;
-} SStreamRecoverObj;
-#endif
-
 #ifdef __cplusplus
 }
 #endif

source/dnode/mnode/impl/inc/mndUser.h

@@ -31,6 +31,7 @@ void      mndReleaseUser(SMnode *pMnode, SUserObj *pUser);
 // for trans test
 SSdbRaw  *mndUserActionEncode(SUserObj *pUser);
 SHashObj *mndDupDbHash(SHashObj *pOld);
+SHashObj *mndDupTopicHash(SHashObj *pOld);
 int32_t   mndValidateUserAuthInfo(SMnode *pMnode, SUserAuthVersion *pUsers, int32_t numOfUses, void **ppRsp,
                                   int32_t *pRspLen);
 

source/dnode/mnode/impl/src/mndDef.c

@@ -76,11 +76,14 @@ int32_t tEncodeSStreamObj(SEncoder *pEncoder, const SStreamObj *pObj) {
 
   if (tEncodeSSchemaWrapper(pEncoder, &pObj->outputSchema) < 0) return -1;
 
+  // 3.0.20
+  if (tEncodeI64(pEncoder, pObj->checkpointFreq) < 0) return -1;
+
   tEndEncode(pEncoder);
   return pEncoder->pos;
 }
 
-int32_t tDecodeSStreamObj(SDecoder *pDecoder, SStreamObj *pObj) {
+int32_t tDecodeSStreamObj(SDecoder *pDecoder, SStreamObj *pObj, int32_t sver) {
   if (tStartDecode(pDecoder) < 0) return -1;
   if (tDecodeCStrTo(pDecoder, pObj->name) < 0) return -1;
 
@@ -139,6 +142,10 @@ int32_t tDecodeSStreamObj(SDecoder *pDecoder, SStreamObj *pObj) {
 
   if (tDecodeSSchemaWrapper(pDecoder, &pObj->outputSchema) < 0) return -1;
 
+  // 3.0.20
+  if (sver >= 2) {
+    if (tDecodeI64(pDecoder, &pObj->checkpointFreq) < 0) return -1;
+  }
   tEndDecode(pDecoder);
   return 0;
 }

source/dnode/mnode/impl/src/mndDnode.c

@@ -787,7 +787,7 @@ static int32_t mndProcessDropDnodeReq(SRpcMsg *pReq) {
   int32_t numOfVnodes = mndGetVnodesNum(pMnode, pDnode->id);
   if ((numOfVnodes > 0 || pMObj != NULL || pSObj != NULL || pQObj != NULL) && !dropReq.force) {
     if (!mndIsDnodeOnline(pDnode, taosGetTimestampMs())) {
-      terrno = TSDB_CODE_NODE_OFFLINE;
+      terrno = TSDB_CODE_DNODE_OFFLINE;
       mError("dnode:%d, failed to drop since %s, vnodes:%d mnode:%d qnode:%d snode:%d", pDnode->id, terrstr(),
              numOfVnodes, pMObj != NULL, pQObj != NULL, pSObj != NULL);
       goto _OVER;

source/dnode/mnode/impl/src/mndInfoSchema.c

@@ -71,7 +71,7 @@ static int32_t mndInsInitMeta(SHashObj *hash) {
 int32_t mndBuildInsTableSchema(SMnode *pMnode, const char *dbFName, const char *tbName, bool sysinfo,
                                STableMetaRsp *pRsp) {
   if (NULL == pMnode->infosMeta) {
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_APP_ERROR;
     return -1;
   }
 
@@ -97,7 +97,7 @@ int32_t mndBuildInsTableSchema(SMnode *pMnode, const char *dbFName, const char *
 
 int32_t mndBuildInsTableCfg(SMnode *pMnode, const char *dbFName, const char *tbName, STableCfgRsp *pRsp) {
   if (NULL == pMnode->infosMeta) {
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_APP_ERROR;
     return -1;
   }
 

source/dnode/mnode/impl/src/mndMain.c

@@ -45,7 +45,7 @@ static inline int32_t mndAcquireRpc(SMnode *pMnode) {
   int32_t code = 0;
   taosThreadRwlockRdlock(&pMnode->lock);
   if (pMnode->stopped) {
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_APP_IS_STOPPING;
     code = -1;
   } else if (!mndIsLeader(pMnode)) {
     code = -1;
@@ -85,6 +85,21 @@ static void *mndBuildTimerMsg(int32_t *pContLen) {
   return pReq;
 }
 
+static void *mndBuildCheckpointTickMsg(int32_t *pContLen, int64_t sec) {
+  SMStreamTickReq timerReq = {
+      .tick = sec,
+  };
+
+  int32_t contLen = tSerializeSMStreamTickMsg(NULL, 0, &timerReq);
+  if (contLen <= 0) return NULL;
+  void *pReq = rpcMallocCont(contLen);
+  if (pReq == NULL) return NULL;
+
+  tSerializeSMStreamTickMsg(pReq, contLen, &timerReq);
+  *pContLen = contLen;
+  return pReq;
+}
+
 static void mndPullupTrans(SMnode *pMnode) {
   int32_t contLen = 0;
   void   *pReq = mndBuildTimerMsg(&contLen);
@@ -105,7 +120,24 @@ static void mndCalMqRebalance(SMnode *pMnode) {
   int32_t contLen = 0;
   void   *pReq = mndBuildTimerMsg(&contLen);
   if (pReq != NULL) {
-    SRpcMsg rpcMsg = {.msgType = TDMT_MND_TMQ_TIMER, .pCont = pReq, .contLen = contLen};
+    SRpcMsg rpcMsg = {
+        .msgType = TDMT_MND_TMQ_TIMER,
+        .pCont = pReq,
+        .contLen = contLen,
+    };
+    tmsgPutToQueue(&pMnode->msgCb, READ_QUEUE, &rpcMsg);
+  }
+}
+
+static void mndStreamCheckpointTick(SMnode *pMnode, int64_t sec) {
+  int32_t contLen = 0;
+  void   *pReq = mndBuildCheckpointTickMsg(&contLen, sec);
+  if (pReq != NULL) {
+    SRpcMsg rpcMsg = {
+        .msgType = TDMT_MND_STREAM_CHECKPOINT_TIMER,
+        .pCont = pReq,
+        .contLen = contLen,
+    };
     tmsgPutToQueue(&pMnode->msgCb, READ_QUEUE, &rpcMsg);
   }
 }
@@ -224,6 +256,12 @@ static void *mndThreadFp(void *param) {
       mndCalMqRebalance(pMnode);
     }
 
+#if 0
+    if (sec % tsStreamCheckpointTickInterval == 0) {
+      mndStreamCheckpointTick(pMnode, sec);
+    }
+#endif
+
     if (sec % tsTelemInterval == (TMIN(60, (tsTelemInterval - 1)))) {
       mndPullupTelem(pMnode);
     }
@@ -561,11 +599,45 @@ static int32_t mndCheckMnodeState(SRpcMsg *pMsg) {
       pMsg->msgType == TDMT_SCH_FETCH || pMsg->msgType == TDMT_SCH_MERGE_FETCH || pMsg->msgType == TDMT_SCH_DROP_TASK) {
     return 0;
   }
-  if (mndAcquireRpc(pMsg->info.node) == 0) return 0;
 
-  SMnode    *pMnode = pMsg->info.node;
+  SMnode *pMnode = pMsg->info.node;
+  taosThreadRwlockRdlock(&pMnode->lock);
+  if (pMnode->stopped) {
+    taosThreadRwlockUnlock(&pMnode->lock);
+    terrno = TSDB_CODE_APP_IS_STOPPING;
+    return -1;
+  }
+
+  terrno = 0;
   SSyncState state = syncGetState(pMnode->syncMgmt.sync);
+  if (terrno != 0) {
+    taosThreadRwlockUnlock(&pMnode->lock);
+    return -1;
+  }
 
+  if (state.state != TAOS_SYNC_STATE_LEADER) {
+    taosThreadRwlockUnlock(&pMnode->lock);
+    terrno = TSDB_CODE_SYN_NOT_LEADER;
+    goto _OVER;
+  }
+
+  if (!state.restored || !pMnode->restored) {
+    taosThreadRwlockUnlock(&pMnode->lock);
+    terrno = TSDB_CODE_SYN_RESTORING;
+    goto _OVER;
+  }
+
+#if 1
+  atomic_add_fetch_32(&pMnode->rpcRef, 1);
+#else
+  int32_t ref = atomic_add_fetch_32(&pMnode->rpcRef, 1);
+  mTrace("mnode rpc is acquired, ref:%d", ref);
+#endif
+
+  taosThreadRwlockUnlock(&pMnode->lock);
+  return 0;
+
+_OVER:
   if (pMsg->msgType == TDMT_MND_TMQ_TIMER || pMsg->msgType == TDMT_MND_TELEM_TIMER ||
       pMsg->msgType == TDMT_MND_TRANS_TIMER || pMsg->msgType == TDMT_MND_TTL_TIMER ||
       pMsg->msgType == TDMT_MND_UPTIME_TIMER) {
@@ -578,42 +650,26 @@ static int32_t mndCheckMnodeState(SRpcMsg *pMsg) {
   SEpSet          epSet = {0};
   mndGetMnodeEpSet(pMnode, &epSet);
 
-  mDebug(
+  mGDebug(
       "msg:%p, type:%s failed to process since %s, mnode restored:%d stopped:%d, sync restored:%d "
       "role:%s, redirect numOfEps:%d inUse:%d",
       pMsg, TMSG_INFO(pMsg->msgType), terrstr(), pMnode->restored, pMnode->stopped, state.restored,
       syncStr(state.restored), epSet.numOfEps, epSet.inUse);
 
-  if (epSet.numOfEps > 0) {
+  if (epSet.numOfEps <= 0) return -1;
-    for (int32_t i = 0; i < epSet.numOfEps; ++i) {
-      mDebug("mnode index:%d, ep:%s:%u", i, epSet.eps[i].fqdn, epSet.eps[i].port);
-    }
 
-    int32_t contLen = tSerializeSEpSet(NULL, 0, &epSet);
+  for (int32_t i = 0; i < epSet.numOfEps; ++i) {
-    pMsg->info.rsp = rpcMallocCont(contLen);
+    mDebug("mnode index:%d, ep:%s:%u", i, epSet.eps[i].fqdn, epSet.eps[i].port);
-    if (pMsg->info.rsp != NULL) {
-      tSerializeSEpSet(pMsg->info.rsp, contLen, &epSet);
-      pMsg->info.hasEpSet = 1;
-      pMsg->info.rspLen = contLen;
-      terrno = TSDB_CODE_RPC_REDIRECT;
-    } else {
-      terrno = TSDB_CODE_OUT_OF_MEMORY;
-    }
-  } else {
-    terrno = TSDB_CODE_APP_NOT_READY;
   }
 
-  return -1;
+  int32_t contLen = tSerializeSEpSet(NULL, 0, &epSet);
-}
+  pMsg->info.rsp = rpcMallocCont(contLen);
+  if (pMsg->info.rsp != NULL) {
+    tSerializeSEpSet(pMsg->info.rsp, contLen, &epSet);
+    pMsg->info.hasEpSet = 1;
+    pMsg->info.rspLen = contLen;
+  }
 
-static int32_t mndCheckMsgContent(SRpcMsg *pMsg) {
-  if (!IsReq(pMsg)) return 0;
-  if (pMsg->contLen != 0 && pMsg->pCont != NULL) return 0;
-
-  const STraceId *trace = &pMsg->info.traceId;
-  mGError("msg:%p, failed to check msg, cont:%p contLen:%d, app:%p type:%s", pMsg, pMsg->pCont, pMsg->contLen,
-          pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));
-  terrno = TSDB_CODE_INVALID_MSG_LEN;
   return -1;
 }
 
@@ -628,7 +684,6 @@ int32_t mndProcessRpcMsg(SRpcMsg *pMsg) {
     return -1;
   }
 
-  if (mndCheckMsgContent(pMsg) != 0) return -1;
   if (mndCheckMnodeState(pMsg) != 0) return -1;
 
   mGTrace("msg:%p, start to process in mnode, app:%p type:%s", pMsg, pMsg->info.ahandle, TMSG_INFO(pMsg->msgType));

source/dnode/mnode/impl/src/mndMnode.c

@@ -292,7 +292,7 @@ static int32_t mndBuildCreateMnodeRedoAction(STrans *pTrans, SDCreateMnodeReq *p
       .pCont = pReq,
       .contLen = contLen,
       .msgType = TDMT_DND_CREATE_MNODE,
-      .acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED,
+      .acceptableCode = TSDB_CODE_MNODE_ALREADY_DEPLOYED,
   };
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
@@ -333,7 +333,7 @@ static int32_t mndBuildDropMnodeRedoAction(STrans *pTrans, SDDropMnodeReq *pDrop
       .pCont = pReq,
       .contLen = contLen,
       .msgType = TDMT_DND_DROP_MNODE,
-      .acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED,
+      .acceptableCode = TSDB_CODE_MNODE_NOT_DEPLOYED,
   };
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
@@ -440,7 +440,7 @@ static int32_t mndProcessCreateMnodeReq(SRpcMsg *pReq) {
   }
 
   if (!mndIsDnodeOnline(pDnode, taosGetTimestampMs())) {
-    terrno = TSDB_CODE_NODE_OFFLINE;
+    terrno = TSDB_CODE_DNODE_OFFLINE;
     goto _OVER;
   }
 
@@ -490,7 +490,7 @@ static int32_t mndSetDropMnodeRedoActions(SMnode *pMnode, STrans *pTrans, SDnode
   if (totalMnodes == 2) {
     if (force) {
       mError("cant't force drop dnode, since a mnode on it and replica is 2");
-      terrno = TSDB_CODE_NODE_OFFLINE;
+      terrno = TSDB_CODE_DNODE_OFFLINE;
       return -1;
     }
     mInfo("vgId:1, has %d mnodes, exec redo log first", totalMnodes);
@@ -574,7 +574,7 @@ static int32_t mndProcessDropMnodeReq(SRpcMsg *pReq) {
   }
 
   if (!mndIsDnodeOnline(pObj->pDnode, taosGetTimestampMs())) {
-    terrno = TSDB_CODE_NODE_OFFLINE;
+    terrno = TSDB_CODE_DNODE_OFFLINE;
     goto _OVER;
   }
 

source/dnode/mnode/impl/src/mndPerfSchema.c

@@ -68,7 +68,7 @@ int32_t mndPerfsInitMeta(SHashObj *hash) {
 
 int32_t mndBuildPerfsTableSchema(SMnode *pMnode, const char *dbFName, const char *tbName, STableMetaRsp *pRsp) {
   if (NULL == pMnode->perfsMeta) {
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_APP_ERROR;
     return -1;
   }
 
@@ -94,7 +94,7 @@ int32_t mndBuildPerfsTableSchema(SMnode *pMnode, const char *dbFName, const char
 
 int32_t mndBuildPerfsTableCfg(SMnode *pMnode, const char *dbFName, const char *tbName, STableCfgRsp *pRsp) {
   if (NULL == pMnode->perfsMeta) {
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_APP_ERROR;
     return -1;
   }
 

source/dnode/mnode/impl/src/mndQnode.c

@@ -205,7 +205,7 @@ static int32_t mndSetCreateQnodeRedoActions(STrans *pTrans, SDnodeObj *pDnode, S
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_CREATE_QNODE;
-  action.acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_QNODE_ALREADY_DEPLOYED;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);
@@ -232,7 +232,7 @@ static int32_t mndSetCreateQnodeUndoActions(STrans *pTrans, SDnodeObj *pDnode, S
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_DROP_QNODE;
-  action.acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_QNODE_NOT_DEPLOYED;
 
   if (mndTransAppendUndoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);
@@ -345,7 +345,7 @@ static int32_t mndSetDropQnodeRedoActions(STrans *pTrans, SDnodeObj *pDnode, SQn
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_DROP_QNODE;
-  action.acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_QNODE_NOT_DEPLOYED;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);

source/dnode/mnode/impl/src/mndShow.c

@@ -108,6 +108,8 @@ static int32_t convertToRetrieveType(char *name, int32_t len) {
     type = TSDB_MGMT_TABLE_APPS;
   } else if (strncasecmp(name, TSDB_INS_TABLE_STREAM_TASKS, len) == 0) {
     type = TSDB_MGMT_TABLE_STREAM_TASKS;
+  } else if (strncasecmp(name, TSDB_INS_TABLE_USER_PRIVILEGES, len) == 0) {
+    type = TSDB_MGMT_TABLE_PRIVILEGES;
   } else {
     //    ASSERT(0);
   }

source/dnode/mnode/impl/src/mndSma.c

@@ -463,7 +463,7 @@ static int32_t mndSetCreateSmaVgroupRedoActions(SMnode *pMnode, STrans *pTrans,
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_CREATE_VNODE;
-  action.acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_VND_ALREADY_EXIST;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);
@@ -780,7 +780,7 @@ static int32_t mndSetDropSmaVgroupRedoActions(SMnode *pMnode, STrans *pTrans, SD
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_DROP_VNODE;
-  action.acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_VND_NOT_EXIST;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);

source/dnode/mnode/impl/src/mndSnode.c

@@ -210,7 +210,7 @@ static int32_t mndSetCreateSnodeRedoActions(STrans *pTrans, SDnodeObj *pDnode, S
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_CREATE_SNODE;
-  action.acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_SNODE_ALREADY_DEPLOYED;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);
@@ -237,7 +237,7 @@ static int32_t mndSetCreateSnodeUndoActions(STrans *pTrans, SDnodeObj *pDnode, S
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_DROP_SNODE;
-  action.acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_SNODE_NOT_DEPLOYED;
 
   if (mndTransAppendUndoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);
@@ -352,7 +352,7 @@ static int32_t mndSetDropSnodeRedoActions(STrans *pTrans, SDnodeObj *pDnode, SSn
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_DROP_SNODE;
-  action.acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_SNODE_NOT_DEPLOYED;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);

source/dnode/mnode/impl/src/mndStream.c

@@ -28,7 +28,7 @@
 #include "parser.h"
 #include "tname.h"
 
-#define MND_STREAM_VER_NUMBER   1
+#define MND_STREAM_VER_NUMBER   2
 #define MND_STREAM_RESERVE_SIZE 64
 
 static int32_t mndStreamActionInsert(SSdb *pSdb, SStreamObj *pStream);
@@ -36,6 +36,8 @@ static int32_t mndStreamActionDelete(SSdb *pSdb, SStreamObj *pStream);
 static int32_t mndStreamActionUpdate(SSdb *pSdb, SStreamObj *pStream, SStreamObj *pNewStream);
 static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq);
 static int32_t mndProcessDropStreamReq(SRpcMsg *pReq);
+static int32_t mndProcessStreamCheckpointTmr(SRpcMsg *pReq);
+static int32_t mndProcessStreamDoCheckpoint(SRpcMsg *pReq);
 /*static int32_t mndProcessRecoverStreamReq(SRpcMsg *pReq);*/
 static int32_t mndProcessStreamMetaReq(SRpcMsg *pReq);
 static int32_t mndGetStreamMeta(SRpcMsg *pReq, SShowObj *pShow, STableMetaRsp *pMeta);
@@ -62,6 +64,10 @@ int32_t mndInitStream(SMnode *pMnode) {
   mndSetMsgHandle(pMnode, TDMT_STREAM_TASK_DEPLOY_RSP, mndTransProcessRsp);
   mndSetMsgHandle(pMnode, TDMT_STREAM_TASK_DROP_RSP, mndTransProcessRsp);
 
+  mndSetMsgHandle(pMnode, TDMT_MND_STREAM_CHECKPOINT_TIMER, mndProcessStreamCheckpointTmr);
+  mndSetMsgHandle(pMnode, TDMT_MND_STREAM_BEGIN_CHECKPOINT, mndProcessStreamDoCheckpoint);
+  mndSetMsgHandle(pMnode, TDMT_STREAM_TASK_REPORT_CHECKPOINT, mndTransProcessRsp);
+
   mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_STREAMS, mndRetrieveStream);
   mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_STREAMS, mndCancelGetNextStream);
   mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_STREAM_TASKS, mndRetrieveStreamTask);
@@ -127,7 +133,7 @@ SSdbRow *mndStreamActionDecode(SSdbRaw *pRaw) {
   int8_t sver = 0;
   if (sdbGetRawSoftVer(pRaw, &sver) != 0) goto STREAM_DECODE_OVER;
 
-  if (sver != MND_STREAM_VER_NUMBER) {
+  if (sver != 1 && sver != 2) {
     terrno = TSDB_CODE_SDB_INVALID_DATA_VER;
     goto STREAM_DECODE_OVER;
   }
@@ -147,7 +153,7 @@ SSdbRow *mndStreamActionDecode(SSdbRaw *pRaw) {
 
   SDecoder decoder;
   tDecoderInit(&decoder, buf, tlen + 1);
-  if (tDecodeSStreamObj(&decoder, pStream) < 0) {
+  if (tDecodeSStreamObj(&decoder, pStream, sver) < 0) {
     tDecoderClear(&decoder);
     goto STREAM_DECODE_OVER;
   }
@@ -680,6 +686,164 @@ _OVER:
   tFreeStreamObj(&streamObj);
   return code;
 }
+static int32_t mndProcessStreamCheckpointTmr(SRpcMsg *pReq) {
+  SMnode     *pMnode = pReq->info.node;
+  SSdb       *pSdb = pMnode->pSdb;
+  void       *pIter = NULL;
+  SStreamObj *pStream = NULL;
+
+  // iterate all stream obj
+  while (1) {
+    pIter = sdbFetch(pSdb, SDB_STREAM, pIter, (void **)&pStream);
+    if (pIter == NULL) break;
+    // incr tick
+    int64_t currentTick = atomic_add_fetch_64(&pStream->currentTick, 1);
+    // if >= checkpointFreq, build msg TDMT_MND_STREAM_BEGIN_CHECKPOINT, put into write q
+    if (currentTick >= pStream->checkpointFreq) {
+      atomic_store_64(&pStream->currentTick, 0);
+      SMStreamDoCheckpointMsg *pMsg = rpcMallocCont(sizeof(SMStreamDoCheckpointMsg));
+
+      pMsg->streamId = pStream->uid;
+      pMsg->checkpointId = tGenIdPI64();
+      memcpy(pMsg->streamName, pStream->name, TSDB_STREAM_FNAME_LEN);
+
+      SRpcMsg rpcMsg = {
+          .msgType = TDMT_MND_STREAM_BEGIN_CHECKPOINT,
+          .pCont = pMsg,
+          .contLen = sizeof(SMStreamDoCheckpointMsg),
+      };
+
+      tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
+    }
+  }
+
+  return 0;
+}
+
+static int32_t mndBuildStreamCheckpointSourceReq(void **pBuf, int32_t *pLen, const SStreamTask *pTask,
+                                                 SMStreamDoCheckpointMsg *pMsg) {
+  SStreamCheckpointSourceReq req = {0};
+  req.checkpointId = pMsg->checkpointId;
+  req.nodeId = pTask->nodeId;
+  req.expireTime = -1;
+  req.streamId = pTask->streamId;
+  req.taskId = pTask->taskId;
+
+  int32_t code;
+  int32_t blen;
+
+  tEncodeSize(tEncodeSStreamCheckpointSourceReq, &req, blen, code);
+  if (code < 0) {
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return -1;
+  }
+
+  int32_t tlen = sizeof(SMsgHead) + blen;
+
+  void *buf = taosMemoryMalloc(tlen);
+  if (buf == NULL) {
+    terrno = TSDB_CODE_OUT_OF_MEMORY;
+    return -1;
+  }
+
+  void    *abuf = POINTER_SHIFT(buf, sizeof(SMsgHead));
+  SEncoder encoder;
+  tEncoderInit(&encoder, abuf, tlen);
+  tEncodeSStreamCheckpointSourceReq(&encoder, &req);
+
+  SMsgHead *pMsgHead = (SMsgHead *)buf;
+  pMsgHead->contLen = htonl(tlen);
+  pMsgHead->vgId = htonl(pTask->nodeId);
+
+  tEncoderClear(&encoder);
+
+  *pBuf = buf;
+  *pLen = tlen;
+
+  return 0;
+}
+
+static int32_t mndProcessStreamDoCheckpoint(SRpcMsg *pReq) {
+  SMnode *pMnode = pReq->info.node;
+  SSdb   *pSdb = pMnode->pSdb;
+
+  SMStreamDoCheckpointMsg *pMsg = (SMStreamDoCheckpointMsg *)pReq->pCont;
+
+  SStreamObj *pStream = mndAcquireStream(pMnode, pMsg->streamName);
+
+  if (pStream == NULL || pStream->uid != pMsg->streamId) {
+    mError("start checkpointing failed since stream %s not found", pMsg->streamName);
+    return -1;
+  }
+
+  // build new transaction:
+  STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB_INSIDE, pReq, "stream-checkpoint");
+  if (pTrans == NULL) return -1;
+  mndTransSetDbName(pTrans, pStream->sourceDb, pStream->targetDb);
+  taosRLockLatch(&pStream->lock);
+  // 1. redo action: broadcast checkpoint source msg for all source vg
+  int32_t totLevel = taosArrayGetSize(pStream->tasks);
+  for (int32_t i = 0; i < totLevel; i++) {
+    SArray      *pLevel = taosArrayGetP(pStream->tasks, i);
+    SStreamTask *pTask = taosArrayGetP(pLevel, 0);
+    if (pTask->taskLevel == TASK_LEVEL__SOURCE) {
+      int32_t sz = taosArrayGetSize(pLevel);
+      for (int32_t j = 0; j < sz; j++) {
+        SStreamTask *pTask = taosArrayGetP(pLevel, j);
+        ASSERT(pTask->nodeId > 0);
+        SVgObj *pVgObj = mndAcquireVgroup(pMnode, pTask->nodeId);
+        if (pVgObj == NULL) {
+          ASSERT(0);
+          taosRUnLockLatch(&pStream->lock);
+          mndReleaseStream(pMnode, pStream);
+          mndTransDrop(pTrans);
+          return -1;
+        }
+
+        void   *buf;
+        int32_t tlen;
+        if (mndBuildStreamCheckpointSourceReq(&buf, &tlen, pTask, pMsg) < 0) {
+          taosRUnLockLatch(&pStream->lock);
+          mndReleaseStream(pMnode, pStream);
+          mndTransDrop(pTrans);
+          return -1;
+        }
+
+        STransAction action = {0};
+        action.epSet = mndGetVgroupEpset(pMnode, pVgObj);
+        action.pCont = buf;
+        action.contLen = tlen;
+        action.msgType = TDMT_VND_STREAM_CHECK_POINT_SOURCE;
+
+        mndReleaseVgroup(pMnode, pVgObj);
+
+        if (mndTransAppendRedoAction(pTrans, &action) != 0) {
+          taosMemoryFree(buf);
+          taosRUnLockLatch(&pStream->lock);
+          mndReleaseStream(pMnode, pStream);
+          mndTransDrop(pTrans);
+          return -1;
+        }
+      }
+    }
+  }
+  // 2. reset tick
+  atomic_store_64(&pStream->currentTick, 0);
+  // 3. commit log: stream checkpoint info
+  taosRUnLockLatch(&pStream->lock);
+
+  if (mndTransPrepare(pMnode, pTrans) != 0) {
+    mError("failed to prepare trans rebalance since %s", terrstr());
+    mndTransDrop(pTrans);
+    mndReleaseStream(pMnode, pStream);
+    return -1;
+  }
+
+  mndReleaseStream(pMnode, pStream);
+  mndTransDrop(pTrans);
+
+  return 0;
+}
 
 static int32_t mndProcessDropStreamReq(SRpcMsg *pReq) {
   SMnode     *pMnode = pReq->info.node;
@@ -748,71 +912,6 @@ static int32_t mndProcessDropStreamReq(SRpcMsg *pReq) {
   return TSDB_CODE_ACTION_IN_PROGRESS;
 }
 
-#if 0
-static int32_t mndProcessRecoverStreamReq(SRpcMsg *pReq) {
-  SMnode     *pMnode = pReq->info.node;
-  SStreamObj *pStream = NULL;
-  /*SDbObj     *pDb = NULL;*/
-  /*SUserObj   *pUser = NULL;*/
-
-  SMRecoverStreamReq recoverReq = {0};
-  if (tDeserializeSMRecoverStreamReq(pReq->pCont, pReq->contLen, &recoverReq) < 0) {
-    ASSERT(0);
-    terrno = TSDB_CODE_INVALID_MSG;
-    return -1;
-  }
-
-  pStream = mndAcquireStream(pMnode, recoverReq.name);
-
-  if (pStream == NULL) {
-    if (recoverReq.igNotExists) {
-      mInfo("stream:%s, not exist, ignore not exist is set", recoverReq.name);
-      sdbRelease(pMnode->pSdb, pStream);
-      return 0;
-    } else {
-      terrno = TSDB_CODE_MND_STREAM_NOT_EXIST;
-      return -1;
-    }
-  }
-
-  if (mndCheckDbPrivilegeByName(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pStream->targetDb) != 0) {
-    return -1;
-  }
-
-  STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_NOTHING, pReq);
-  if (pTrans == NULL) {
-    mError("stream:%s, failed to recover since %s", recoverReq.name, terrstr());
-    sdbRelease(pMnode->pSdb, pStream);
-    return -1;
-  }
-  mInfo("trans:%d, used to drop stream:%s", pTrans->id, recoverReq.name);
-
-  // broadcast to recover all tasks
-  if (mndRecoverStreamTasks(pMnode, pTrans, pStream) < 0) {
-    mError("stream:%s, failed to recover task since %s", recoverReq.name, terrstr());
-    sdbRelease(pMnode->pSdb, pStream);
-    return -1;
-  }
-
-  // update stream status
-  if (mndSetStreamRecover(pMnode, pTrans, pStream) < 0) {
-    sdbRelease(pMnode->pSdb, pStream);
-    return -1;
-  }
-
-  if (mndTransPrepare(pMnode, pTrans) != 0) {
-    mError("trans:%d, failed to prepare recover stream trans since %s", pTrans->id, terrstr());
-    sdbRelease(pMnode->pSdb, pStream);
-    mndTransDrop(pTrans);
-    return -1;
-  }
-
-  sdbRelease(pMnode->pSdb, pStream);
-
-  return TSDB_CODE_ACTION_IN_PROGRESS;
-}
-#endif
-
 int32_t mndDropStreamByDb(SMnode *pMnode, STrans *pTrans, SDbObj *pDb) {
   SSdb *pSdb = pMnode->pSdb;
   void *pIter = NULL;
@@ -847,13 +946,6 @@ int32_t mndDropStreamByDb(SMnode *pMnode, STrans *pTrans, SDbObj *pDb) {
       }
     }
 
-#if 0
-    if (mndSetDropOffsetStreamLogs(pMnode, pTrans, pStream) < 0) {
-      sdbRelease(pSdb, pStream);
-      goto END;
-    }
-#endif
-
     sdbRelease(pSdb, pStream);
   }
 

source/dnode/mnode/impl/src/mndSubscribe.c

@@ -440,9 +440,9 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
 }
 
 static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOutputObj *pOutput) {
-  STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_DB_INSIDE, pMsg, "persist-reb");
+  STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_DB_INSIDE, pMsg, "tmq-reb");
-  mndTransSetDbName(pTrans, pOutput->pSub->dbName, NULL);
   if (pTrans == NULL) return -1;
+  mndTransSetDbName(pTrans, pOutput->pSub->dbName, NULL);
 
   // make txn:
   // 1. redo action: action to all vg
@@ -523,28 +523,6 @@ static int32_t mndPersistRebResult(SMnode *pMnode, SRpcMsg *pMsg, const SMqRebOu
     tDeleteSMqConsumerObj(pConsumerNew);
     taosMemoryFree(pConsumerNew);
   }
-#if 0
-  if (consumerNum) {
-    char topic[TSDB_TOPIC_FNAME_LEN];
-    char cgroup[TSDB_CGROUP_LEN];
-    mndSplitSubscribeKey(pOutput->pSub->key, topic, cgroup, true);
-    SMqTopicObj *pTopic = mndAcquireTopic(pMnode, topic);
-    if (pTopic) {
-      // TODO make topic complete
-      SMqTopicObj topicObj = {0};
-      memcpy(&topicObj, pTopic, sizeof(SMqTopicObj));
-      topicObj.refConsumerCnt = pTopic->refConsumerCnt - consumerNum;
-      // TODO is that correct?
-      pTopic->refConsumerCnt = topicObj.refConsumerCnt;
-      mInfo("subscribe topic %s unref %d consumer cgroup %s, refcnt %d", pTopic->name, consumerNum, cgroup,
-            topicObj.refConsumerCnt);
-      if (mndSetTopicCommitLogs(pMnode, pTrans, &topicObj) != 0) {
-        ASSERT(0);
-        goto REB_FAIL;
-      }
-    }
-  }
-#endif
 
   // 4. TODO commit log: modification log
 

source/dnode/mnode/impl/src/mndSync.c

@@ -318,7 +318,7 @@ int32_t mndSyncPropose(SMnode *pMnode, SSdbRaw *pRaw, int32_t transId) {
   if (pMgmt->transId != 0) {
     mError("trans:%d, can't be proposed since trans:%d already waiting for confirm", transId, pMgmt->transId);
     taosWUnLockLatch(&pMgmt->lock);
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_MND_LAST_TRANS_NOT_FINISHED;
     return -1;
   }
 
@@ -339,13 +339,11 @@ int32_t mndSyncPropose(SMnode *pMnode, SSdbRaw *pRaw, int32_t transId) {
     sdbSetApplyInfo(pMnode->pSdb, req.info.conn.applyIndex, req.info.conn.applyTerm, SYNC_INDEX_INVALID);
     code = 0;
   } else {
-    mInfo("trans:%d, failed to proposed since %s", transId, terrstr());
+    mError("trans:%d, failed to proposed since %s", transId, terrstr());
     taosWLockLatch(&pMgmt->lock);
     pMgmt->transId = 0;
     taosWUnLockLatch(&pMgmt->lock);
-    if (terrno == TSDB_CODE_SYN_NOT_LEADER) {
+    if (terrno == 0) {
-      terrno = TSDB_CODE_APP_NOT_READY;
-    } else {
       terrno = TSDB_CODE_APP_ERROR;
     }
   }
@@ -381,20 +379,23 @@ void mndSyncStop(SMnode *pMnode) {
 }
 
 bool mndIsLeader(SMnode *pMnode) {
+  terrno = 0;
   SSyncState state = syncGetState(pMnode->syncMgmt.sync);
 
-  if (state.state != TAOS_SYNC_STATE_LEADER || !state.restored) {
+  if (terrno != 0) {
-    if (state.state != TAOS_SYNC_STATE_LEADER) {
+    mDebug("vgId:1, mnode is stopping");
-      terrno = TSDB_CODE_SYN_NOT_LEADER;
-    } else {
-      terrno = TSDB_CODE_APP_NOT_READY;
-    }
-    mDebug("vgId:1, mnode not ready, state:%s, restore:%d", syncStr(state.state), state.restored);
     return false;
   }
 
-  if (!mndGetRestored(pMnode)) {
+  if (state.state != TAOS_SYNC_STATE_LEADER) {
-    terrno = TSDB_CODE_APP_NOT_READY;
+    terrno = TSDB_CODE_SYN_NOT_LEADER;
+    mDebug("vgId:1, mnode not leader, state:%s", syncStr(state.state));
+    return false;
+  }
+
+  if (!state.restored || !pMnode->restored) {
+    terrno = TSDB_CODE_SYN_RESTORING;
+    mDebug("vgId:1, mnode not restored:%d:%d", state.restored, pMnode->restored);
     return false;
   }
 

source/dnode/mnode/impl/src/mndTrans.c

@@ -918,10 +918,13 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) {
       sendRsp = true;
     }
   } else {
-    if (pTrans->stage == TRN_STAGE_REDO_ACTION && ((code == TSDB_CODE_APP_NOT_READY && pTrans->failedTimes > 60) ||
+    if (pTrans->stage == TRN_STAGE_REDO_ACTION) {
-                                                   (code != TSDB_CODE_APP_NOT_READY && pTrans->failedTimes > 6))) {
+      if (code == TSDB_CODE_SYN_NOT_LEADER || code == TSDB_CODE_SYN_RESTORING || code == TSDB_CODE_APP_IS_STARTING) {
+        if (pTrans->failedTimes > 60) sendRsp = true;
+      } else {
+        if (pTrans->failedTimes > 6) sendRsp = true;
+      }
       if (code == 0) code = TSDB_CODE_MND_TRANS_UNKNOW_ERROR;
-      sendRsp = true;
     }
   }
 

source/dnode/mnode/impl/src/mndUser.c

@@ -161,7 +161,7 @@ SSdbRaw *mndUserActionEncode(SUserObj *pUser) {
   char *topic = taosHashIterate(pUser->topics, NULL);
   while (topic != NULL) {
     SDB_SET_BINARY(pRaw, dataPos, topic, TSDB_TOPIC_FNAME_LEN, _OVER);
-    db = taosHashIterate(pUser->topics, topic);
+    topic = taosHashIterate(pUser->topics, topic);
   }
 
   SDB_SET_RESERVE(pRaw, dataPos, USER_RESERVE_SIZE, _OVER)
@@ -446,7 +446,7 @@ static int32_t mndAlterUser(SMnode *pMnode, SUserObj *pOld, SUserObj *pNew, SRpc
   return 0;
 }
 
-SHashObj *mndDupDbHash(SHashObj *pOld) {
+SHashObj *mndDupObjHash(SHashObj *pOld, int32_t dataLen) {
   SHashObj *pNew =
       taosHashInit(taosHashGetSize(pOld), taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK);
   if (pNew == NULL) {
@@ -457,7 +457,7 @@ SHashObj *mndDupDbHash(SHashObj *pOld) {
   char *db = taosHashIterate(pOld, NULL);
   while (db != NULL) {
     int32_t len = strlen(db) + 1;
-    if (taosHashPut(pNew, db, len, db, TSDB_DB_FNAME_LEN) != 0) {
+    if (taosHashPut(pNew, db, len, db, dataLen) != 0) {
       taosHashCancelIterate(pOld, db);
       taosHashCleanup(pNew);
       terrno = TSDB_CODE_OUT_OF_MEMORY;
@@ -469,6 +469,10 @@ SHashObj *mndDupDbHash(SHashObj *pOld) {
   return pNew;
 }
 
+SHashObj *mndDupDbHash(SHashObj *pOld) { return mndDupObjHash(pOld, TSDB_DB_FNAME_LEN); }
+
+SHashObj *mndDupTopicHash(SHashObj *pOld) { return mndDupObjHash(pOld, TSDB_TOPIC_FNAME_LEN); }
+
 static int32_t mndProcessAlterUserReq(SRpcMsg *pReq) {
   SMnode       *pMnode = pReq->info.node;
   SSdb         *pSdb = pMnode->pSdb;
@@ -519,7 +523,7 @@ static int32_t mndProcessAlterUserReq(SRpcMsg *pReq) {
   taosRLockLatch(&pUser->lock);
   newUser.readDbs = mndDupDbHash(pUser->readDbs);
   newUser.writeDbs = mndDupDbHash(pUser->writeDbs);
-  newUser.topics = mndDupDbHash(pUser->topics);
+  newUser.topics = mndDupTopicHash(pUser->topics);
   taosRUnLockLatch(&pUser->lock);
 
   if (newUser.readDbs == NULL || newUser.writeDbs == NULL || newUser.topics == NULL) {
@@ -832,6 +836,26 @@ static int32_t mndRetrievePrivileges(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock
     int32_t numOfTopics = taosHashGetSize(pUser->topics);
     if (numOfRows + numOfReadDbs + numOfWriteDbs + numOfTopics >= rows) break;
 
+    if (pUser->superUser) {
+      cols = 0;
+      SColumnInfoData *pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
+      char             userName[TSDB_USER_LEN + VARSTR_HEADER_SIZE] = {0};
+      STR_WITH_MAXSIZE_TO_VARSTR(userName, pUser->user, pShow->pMeta->pSchemas[cols].bytes);
+      colDataAppend(pColInfo, numOfRows, (const char *)userName, false);
+
+      char privilege[20] = {0};
+      STR_WITH_MAXSIZE_TO_VARSTR(privilege, "all", pShow->pMeta->pSchemas[cols].bytes);
+      pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
+      colDataAppend(pColInfo, numOfRows, (const char *)privilege, false);
+
+      char objName[20] = {0};
+      STR_WITH_MAXSIZE_TO_VARSTR(objName, "all", pShow->pMeta->pSchemas[cols].bytes);
+      pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
+      colDataAppend(pColInfo, numOfRows, (const char *)objName, false);
+
+      numOfRows++;
+    }
+
     char *db = taosHashIterate(pUser->readDbs, NULL);
     while (db != NULL) {
       cols = 0;
@@ -902,7 +926,7 @@ static int32_t mndRetrievePrivileges(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock
       colDataAppend(pColInfo, numOfRows, (const char *)topicName, false);
 
       numOfRows++;
-      db = taosHashIterate(pUser->topics, topic);
+      topic = taosHashIterate(pUser->topics, topic);
     }
 
     sdbRelease(pSdb, pUser);

source/dnode/mnode/impl/src/mndVgroup.c

@@ -998,7 +998,7 @@ int32_t mndAddCreateVnodeAction(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVg
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_CREATE_VNODE;
-  action.acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_VND_ALREADY_EXIST;
 
   if (mndTransAppendRedoAction(pTrans, &action) != 0) {
     taosMemoryFree(pReq);
@@ -1098,7 +1098,7 @@ int32_t mndAddDropVnodeAction(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgOb
   action.pCont = pReq;
   action.contLen = contLen;
   action.msgType = TDMT_DND_DROP_VNODE;
-  action.acceptableCode = TSDB_CODE_NODE_NOT_DEPLOYED;
+  action.acceptableCode = TSDB_CODE_VND_NOT_EXIST;
 
   if (isRedo) {
     if (mndTransAppendRedoAction(pTrans, &action) != 0) {

source/dnode/mnode/impl/test/trans/trans2.cpp

@@ -173,7 +173,7 @@ class MndTestTrans2 : public ::testing::Test {
       action.pCont = pReq;
       action.contLen = contLen;
       action.msgType = TDMT_DND_CREATE_MNODE;
-      action.acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+      action.acceptableCode = TSDB_CODE_MNODE_ALREADY_DEPLOYED;
       mndTransAppendRedoAction(pTrans, &action);
     }
 
@@ -190,7 +190,7 @@ class MndTestTrans2 : public ::testing::Test {
       action.pCont = pReq;
       action.contLen = contLen;
       action.msgType = TDMT_DND_CREATE_MNODE;
-      action.acceptableCode = TSDB_CODE_NODE_ALREADY_DEPLOYED;
+      action.acceptableCode = TSDB_CODE_MNODE_ALREADY_DEPLOYED;
       mndTransAppendUndoAction(pTrans, &action);
     }
 

source/dnode/snode/src/snode.c

@@ -168,7 +168,7 @@ int32_t sndProcessTaskDeployReq(SSnode *pSnode, char *msg, int32_t msgLen) {
 
 int32_t sndProcessTaskDropReq(SSnode *pSnode, char *msg, int32_t msgLen) {
   SVDropStreamTaskReq *pReq = (SVDropStreamTaskReq *)msg;
-  streamMetaRemoveTask1(pSnode->pMeta, pReq->taskId);
+  streamMetaRemoveTask(pSnode->pMeta, pReq->taskId);
   return 0;
 }
 

source/dnode/vnode/inc/vnode.h

@@ -106,14 +106,24 @@ int32_t     metaGetTableTagsByUids(SMeta *pMeta, int64_t suid, SArray *uidList,
 int32_t     metaReadNext(SMetaReader *pReader);
 const void *metaGetTableTagVal(void *tag, int16_t type, STagVal *tagVal);
 int         metaGetTableNameByUid(void *meta, uint64_t uid, char *tbName);
-int         metaGetTableUidByName(void *meta, char *tbName, uint64_t *uid);
+
-int         metaGetTableTypeByName(void *meta, char *tbName, ETableType *tbType);
+int      metaGetTableSzNameByUid(void *meta, uint64_t uid, char *tbName);
-bool        metaIsTableExist(SMeta *pMeta, tb_uid_t uid);
+int      metaGetTableUidByName(void *meta, char *tbName, uint64_t *uid);
-int32_t     metaGetCachedTableUidList(SMeta *pMeta, tb_uid_t suid, const uint8_t *key, int32_t keyLen, SArray *pList,
+int      metaGetTableTypeByName(void *meta, char *tbName, ETableType *tbType);
-                                      bool *acquired);
+bool     metaIsTableExist(SMeta *pMeta, tb_uid_t uid);
-int32_t     metaUidFilterCachePut(SMeta *pMeta, uint64_t suid, const void *pKey, int32_t keyLen, void *pPayload,
+int32_t  metaGetCachedTableUidList(SMeta *pMeta, tb_uid_t suid, const uint8_t *key, int32_t keyLen, SArray *pList,
-                                  int32_t payloadLen, double selectivityRatio);
+                                   bool *acquired);
-int32_t     metaUidCacheClear(SMeta *pMeta, uint64_t suid);
+int32_t  metaUidFilterCachePut(SMeta *pMeta, uint64_t suid, const void *pKey, int32_t keyLen, void *pPayload,
+                               int32_t payloadLen, double selectivityRatio);
+int32_t  metaUidCacheClear(SMeta *pMeta, uint64_t suid);
+tb_uid_t metaGetTableEntryUidByName(SMeta *pMeta, const char *name);
+int64_t  metaGetTbNum(SMeta *pMeta);
+int64_t  metaGetNtbNum(SMeta *pMeta);
+typedef struct {
+  int64_t uid;
+  int64_t ctbNum;
+} SMetaStbStats;
+int32_t metaGetStbStats(SMeta *pMeta, int64_t uid, SMetaStbStats *pInfo);
 
 typedef struct SMetaFltParam {
   tb_uid_t suid;

source/dnode/vnode/src/inc/vnodeInt.h

@@ -116,8 +116,6 @@ int32_t         metaGetTbTSchemaEx(SMeta* pMeta, tb_uid_t suid, tb_uid_t uid, in
 int             metaGetTableEntryByName(SMetaReader* pReader, const char* name);
 int             metaAlterCache(SMeta* pMeta, int32_t nPage);
 
-tb_uid_t      metaGetTableEntryUidByName(SMeta* pMeta, const char* name);
-int64_t       metaGetTbNum(SMeta* pMeta);
 int64_t       metaGetTimeSeriesNum(SMeta* pMeta);
 SMCtbCursor*  metaOpenCtbCursor(SMeta* pMeta, tb_uid_t uid, int lock);
 void          metaCloseCtbCursor(SMCtbCursor* pCtbCur, int lock);
@@ -144,12 +142,6 @@ typedef struct SMetaInfo {
 } SMetaInfo;
 int32_t metaGetInfo(SMeta* pMeta, int64_t uid, SMetaInfo* pInfo, SMetaReader* pReader);
 
-typedef struct {
-  int64_t uid;
-  int64_t ctbNum;
-} SMetaStbStats;
-int32_t metaGetStbStats(SMeta* pMeta, int64_t uid, SMetaStbStats* pInfo);
-
 // tsdb
 int     tsdbOpen(SVnode* pVnode, STsdb** ppTsdb, const char* dir, STsdbKeepCfg* pKeepCfg, int8_t rollback);
 int     tsdbClose(STsdb** pTsdb);

source/dnode/vnode/src/meta/metaQuery.c

@@ -223,6 +223,23 @@ int metaGetTableNameByUid(void *meta, uint64_t uid, char *tbName) {
 
   return 0;
 }
+
+int metaGetTableSzNameByUid(void *meta, uint64_t uid, char *tbName) {
+  int         code = 0;
+  SMetaReader mr = {0};
+  metaReaderInit(&mr, (SMeta *)meta, 0);
+  code = metaGetTableEntryByUid(&mr, uid);
+  if (code < 0) {
+    metaReaderClear(&mr);
+    return -1;
+  }
+  strncpy(tbName, mr.me.name, TSDB_TABLE_NAME_LEN);
+  metaReaderClear(&mr);
+
+  return 0;
+}
+
+
 int metaGetTableUidByName(void *meta, char *tbName, uint64_t *uid) {
   int         code = 0;
   SMetaReader mr = {0};
@@ -739,6 +756,10 @@ int64_t metaGetTimeSeriesNum(SMeta *pMeta) {
   return pMeta->pVnode->config.vndStats.numOfTimeSeries + pMeta->pVnode->config.vndStats.numOfNTimeSeries;
 }
 
+int64_t metaGetNtbNum(SMeta *pMeta) {
+  return pMeta->pVnode->config.vndStats.numOfNTables;
+}
+
 typedef struct {
   SMeta   *pMeta;
   TBC     *pCur;

source/dnode/vnode/src/tq/tq.c

@@ -1425,7 +1425,7 @@ int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) {
 
 int32_t tqProcessTaskDropReq(STQ* pTq, int64_t version, char* msg, int32_t msgLen) {
   SVDropStreamTaskReq* pReq = (SVDropStreamTaskReq*)msg;
-  streamMetaRemoveTask1(pTq->pStreamMeta, pReq->taskId);
+  streamMetaRemoveTask(pTq->pStreamMeta, pReq->taskId);
   return 0;
 }
 

source/dnode/vnode/src/vnd/vnodeSync.c

@@ -153,8 +153,8 @@ void vnodeProposeWriteMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs)
 
     if (!pVnode->restored) {
       vGError("vgId:%d, msg:%p failed to process since restore not finished", vgId, pMsg);
-      terrno = TSDB_CODE_APP_NOT_READY;
+      terrno = TSDB_CODE_SYN_RESTORING;
-      vnodeHandleProposeError(pVnode, pMsg, TSDB_CODE_APP_NOT_READY);
+      vnodeHandleProposeError(pVnode, pMsg, TSDB_CODE_SYN_RESTORING);
       rpcFreeCont(pMsg->pCont);
       taosFreeQitem(pMsg);
       continue;
@@ -544,21 +544,23 @@ bool vnodeIsRoleLeader(SVnode *pVnode) {
 }
 
 bool vnodeIsLeader(SVnode *pVnode) {
+  terrno = 0;
   SSyncState state = syncGetState(pVnode->sync);
 
-  if (state.state != TAOS_SYNC_STATE_LEADER || !state.restored) {
+  if (terrno != 0) {
-    if (state.state != TAOS_SYNC_STATE_LEADER) {
+    vInfo("vgId:%d, vnode is stopping", pVnode->config.vgId);
-      terrno = TSDB_CODE_SYN_NOT_LEADER;
-    } else {
-      terrno = TSDB_CODE_APP_NOT_READY;
-    }
-    vInfo("vgId:%d, vnode not ready, state:%s, restore:%d", pVnode->config.vgId, syncStr(state.state), state.restored);
     return false;
   }
 
-  if (!pVnode->restored) {
+  if (state.state != TAOS_SYNC_STATE_LEADER) {
-    vInfo("vgId:%d, vnode not restored", pVnode->config.vgId);
+    terrno = TSDB_CODE_SYN_NOT_LEADER;
-    terrno = TSDB_CODE_APP_NOT_READY;
+    vInfo("vgId:%d, vnode not leader, state:%s", pVnode->config.vgId, syncStr(state.state));
+    return false;
+  }
+
+  if (!state.restored || !pVnode->restored) {
+    terrno = TSDB_CODE_SYN_RESTORING;
+    vInfo("vgId:%d, vnode not restored:%d:%d", pVnode->config.vgId, state.restored, pVnode->restored);
     return false;
   }
 

source/libs/command/inc/commandInt.h

@@ -34,6 +34,7 @@ extern "C" {
 #define EXPLAIN_SYSTBL_SCAN_FORMAT "System Table Scan on %s"
 #define EXPLAIN_DISTBLK_SCAN_FORMAT "Block Dist Scan on %s"
 #define EXPLAIN_LASTROW_SCAN_FORMAT "Last Row Scan on %s"
+#define EXPLAIN_TABLE_COUNT_SCAN_FORMAT "Table Count Row Scan on %s"
 #define EXPLAIN_PROJECTION_FORMAT "Projection"
 #define EXPLAIN_JOIN_FORMAT "%s"
 #define EXPLAIN_AGG_FORMAT "Aggragate"

source/libs/command/src/explain.c

@@ -19,6 +19,7 @@
 #include "query.h"
 #include "tcommon.h"
 #include "tdatablock.h"
+#include "systable.h"
 
 int32_t qExplainGenerateResNode(SPhysiNode *pNode, SExplainGroup *group, SExplainResNode **pRes);
 int32_t qExplainAppendGroupResRows(void *pCtx, int32_t groupId, int32_t level, bool singleChannel);
@@ -212,6 +213,11 @@ int32_t qExplainGenerateResChildren(SPhysiNode *pNode, SExplainGroup *group, SNo
       pPhysiChildren = lastRowPhysiNode->scan.node.pChildren;
       break;
     }
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN: {
+      STableCountScanPhysiNode *tableCountPhysiNode = (STableCountScanPhysiNode *)pNode;
+      pPhysiChildren = tableCountPhysiNode->scan.node.pChildren;
+      break;
+    }
     case QUERY_NODE_PHYSICAL_PLAN_GROUP_SORT: {
       SGroupSortPhysiNode *groupSortPhysiNode = (SGroupSortPhysiNode *)pNode;
       pPhysiChildren = groupSortPhysiNode->node.pChildren;
@@ -1355,6 +1361,48 @@ int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx *ctx, i
       }
       break;
     }
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN: {
+      STableCountScanPhysiNode *pLastRowNode = (STableCountScanPhysiNode *)pNode;
+      EXPLAIN_ROW_NEW(level, EXPLAIN_TABLE_COUNT_SCAN_FORMAT,
+                      ('\0' != pLastRowNode->scan.tableName.tname[0] ? pLastRowNode->scan.tableName.tname : TSDB_INS_TABLE_TABLES));
+      EXPLAIN_ROW_APPEND(EXPLAIN_LEFT_PARENTHESIS_FORMAT);
+      if (pResNode->pExecInfo) {
+        QRY_ERR_RET(qExplainBufAppendExecInfo(pResNode->pExecInfo, tbuf, &tlen));
+        EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
+      }
+      EXPLAIN_ROW_APPEND(EXPLAIN_COLUMNS_FORMAT, LIST_LENGTH(pLastRowNode->scan.pScanCols));
+      EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
+      if (pLastRowNode->scan.pScanPseudoCols) {
+        EXPLAIN_ROW_APPEND(EXPLAIN_PSEUDO_COLUMNS_FORMAT, pLastRowNode->scan.pScanPseudoCols->length);
+        EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
+      }
+      EXPLAIN_ROW_APPEND(EXPLAIN_WIDTH_FORMAT, pLastRowNode->scan.node.pOutputDataBlockDesc->totalRowSize);
+      EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
+      EXPLAIN_ROW_APPEND(EXPLAIN_RIGHT_PARENTHESIS_FORMAT);
+      EXPLAIN_ROW_END();
+      QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level));
+
+      if (verbose) {
+        EXPLAIN_ROW_NEW(level + 1, EXPLAIN_OUTPUT_FORMAT);
+        EXPLAIN_ROW_APPEND(EXPLAIN_COLUMNS_FORMAT,
+                           nodesGetOutputNumFromSlotList(pLastRowNode->scan.node.pOutputDataBlockDesc->pSlots));
+        EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
+        EXPLAIN_ROW_APPEND(EXPLAIN_WIDTH_FORMAT, pLastRowNode->scan.node.pOutputDataBlockDesc->outputRowSize);
+        EXPLAIN_ROW_APPEND_LIMIT(pLastRowNode->scan.node.pLimit);
+        EXPLAIN_ROW_APPEND_SLIMIT(pLastRowNode->scan.node.pSlimit);
+        EXPLAIN_ROW_END();
+        QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1));
+
+        if (pLastRowNode->scan.node.pConditions) {
+          EXPLAIN_ROW_NEW(level + 1, EXPLAIN_FILTER_FORMAT);
+          QRY_ERR_RET(nodesNodeToSQL(pLastRowNode->scan.node.pConditions, tbuf + VARSTR_HEADER_SIZE,
+                                     TSDB_EXPLAIN_RESULT_ROW_SIZE, &tlen));
+          EXPLAIN_ROW_END();
+          QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1));
+        }        
+      }
+      break;
+    }
     case QUERY_NODE_PHYSICAL_PLAN_GROUP_SORT: {
       SGroupSortPhysiNode *pSortNode = (SGroupSortPhysiNode *)pNode;
       EXPLAIN_ROW_NEW(level, EXPLAIN_GROUP_SORT_FORMAT);

source/libs/executor/inc/executorimpl.h

@@ -482,6 +482,34 @@ typedef struct {
   SSnapContext* sContext;
 } SStreamRawScanInfo;
 
+typedef struct STableCountScanSupp {
+  int16_t dbNameSlotId;
+  int16_t stbNameSlotId;
+  int16_t tbCountSlotId;
+
+  bool groupByDbName;
+  bool groupByStbName;
+  char    dbName[TSDB_DB_NAME_LEN];
+  char    stbName[TSDB_TABLE_NAME_LEN];
+
+} STableCountScanSupp;
+
+typedef struct STableCountScanOperatorInfo {
+  SReadHandle  readHandle;
+  SSDataBlock* pRes;
+
+  SName tableName;
+
+  SNodeList* groupTags;
+  SNodeList* scanCols;
+  SNodeList* pseudoCols;
+
+  STableCountScanSupp supp;
+
+  int32_t   currGrpIdx;
+  SArray*   stbUidList; // when group by db_name and stable_name
+} STableCountScanOperatorInfo;
+
 typedef struct SOptrBasicInfo {
   SResultRowInfo resultRowInfo;
   SSDataBlock*   pRes;
@@ -781,7 +809,7 @@ void setInputDataBlock(SExprSupp* pExprSupp, SSDataBlock* pBlock, int32_t order,
 int32_t checkForQueryBuf(size_t numOfTables);
 
 bool    isTaskKilled(SExecTaskInfo* pTaskInfo);
-void    setTaskKilled(SExecTaskInfo* pTaskInfo);
+void    setTaskKilled(SExecTaskInfo* pTaskInfo, int32_t rspCode);
 void    doDestroyTask(SExecTaskInfo* pTaskInfo);
 void    setTaskStatus(SExecTaskInfo* pTaskInfo, int8_t status);
 

source/libs/executor/src/exchangeoperator.c

@@ -70,7 +70,7 @@ static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeIn
     tsem_wait(&pExchangeInfo->ready);
 
     if (isTaskKilled(pTaskInfo)) {
-      longjmp(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
+      longjmp(pTaskInfo->env, pTaskInfo->code);
     }
 
     for (int32_t i = 0; i < totalSources; ++i) {
@@ -573,7 +573,7 @@ int32_t prepareConcurrentlyLoad(SOperatorInfo* pOperator) {
 
   tsem_wait(&pExchangeInfo->ready);
   if (isTaskKilled(pTaskInfo)) {
-    longjmp(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
+    longjmp(pTaskInfo->env, pTaskInfo->code);
   }
 
   tsem_post(&pExchangeInfo->ready);
@@ -621,7 +621,7 @@ int32_t seqLoadRemoteData(SOperatorInfo* pOperator) {
     doSendFetchDataRequest(pExchangeInfo, pTaskInfo, pExchangeInfo->current);
     tsem_wait(&pExchangeInfo->ready);
     if (isTaskKilled(pTaskInfo)) {
-      longjmp(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
+      longjmp(pTaskInfo->env, pTaskInfo->code);
     }
 
     SDownstreamSourceNode* pSource = taosArrayGet(pExchangeInfo->pSources, pExchangeInfo->current);

source/libs/executor/src/executil.c

@@ -1348,6 +1348,7 @@ void createExprFromOneNode(SExprInfo* pExp, SNode* pNode, int16_t slotId) {
 
     pExprNode->_function.functionId = pFuncNode->funcId;
     pExprNode->_function.pFunctNode = pFuncNode;
+    pExprNode->_function.functionType = pFuncNode->funcType;
 
     tstrncpy(pExprNode->_function.functionName, pFuncNode->functionName, tListLen(pExprNode->_function.functionName));
 

source/libs/executor/src/executor.c

@@ -688,7 +688,7 @@ void qStopTaskOperators(SExecTaskInfo* pTaskInfo) {
   taosWUnLockLatch(&pTaskInfo->stopInfo.lock);
 }
 
-int32_t qAsyncKillTask(qTaskInfo_t qinfo) {
+int32_t qAsyncKillTask(qTaskInfo_t qinfo, int32_t rspCode) {
   SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)qinfo;
 
   if (pTaskInfo == NULL) {
@@ -697,7 +697,7 @@ int32_t qAsyncKillTask(qTaskInfo_t qinfo) {
 
   qDebug("%s execTask async killed", GET_TASKID(pTaskInfo));
 
-  setTaskKilled(pTaskInfo);
+  setTaskKilled(pTaskInfo, rspCode);
 
   qStopTaskOperators(pTaskInfo);
 

source/libs/executor/src/executorimpl.c

@@ -610,21 +610,10 @@ void setBlockSMAInfo(SqlFunctionCtx* pCtx, SExprInfo* pExprInfo, SSDataBlock* pB
 }
 
 bool isTaskKilled(SExecTaskInfo* pTaskInfo) {
-  // query has been executed more than tsShellActivityTimer, and the retrieve has not arrived
+  return (0 != pTaskInfo->code) ? true : false;
-  // abort current query execution.
-  if (pTaskInfo->owner != 0 &&
-      ((taosGetTimestampSec() - pTaskInfo->cost.start / 1000) > 10 * getMaximumIdleDurationSec())
-      /*(!needBuildResAfterQueryComplete(pTaskInfo))*/) {
-    assert(pTaskInfo->cost.start != 0);
-    //    qDebug("QInfo:%" PRIu64 " retrieve not arrive beyond %d ms, abort current query execution, start:%" PRId64
-    //           ", current:%d", pQInfo->qId, 1, pQInfo->startExecTs, taosGetTimestampSec());
-    //    return true;
-  }
-
-  return false;
 }
 
-void setTaskKilled(SExecTaskInfo* pTaskInfo) { pTaskInfo->code = TSDB_CODE_TSC_QUERY_CANCELLED; }
+void setTaskKilled(SExecTaskInfo* pTaskInfo, int32_t rspCode) { pTaskInfo->code = rspCode; }
 
 /////////////////////////////////////////////////////////////////////////////////////////////
 STimeWindow getAlignQueryTimeWindow(SInterval* pInterval, int32_t precision, int64_t key) {
@@ -1381,7 +1370,8 @@ int32_t getTableScanInfo(SOperatorInfo* pOperator, int32_t* order, int32_t* scan
   int32_t type = pOperator->operatorType;
   if (type == QUERY_NODE_PHYSICAL_PLAN_EXCHANGE || type == QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN ||
       type == QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN || type == QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN ||
-      type == QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN || type == QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN) {
+      type == QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN || type == QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN ||
+      type == QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN) {
     *order = TSDB_ORDER_ASC;
     *scanFlag = MAIN_SCAN;
     return TSDB_CODE_SUCCESS;
@@ -1883,6 +1873,8 @@ static SExecTaskInfo* createExecTaskInfo(uint64_t queryId, uint64_t taskId, EOPT
 
 SSchemaWrapper* extractQueriedColumnSchema(SScanPhysiNode* pScanNode);
 
+SOperatorInfo* createTableCountScanOperatorInfo(SReadHandle* handle, STableCountScanPhysiNode* pNode,
+                                                SExecTaskInfo* pTaskInfo);
 int32_t extractTableSchemaInfo(SReadHandle* pHandle, SScanPhysiNode* pScanNode, SExecTaskInfo* pTaskInfo) {
   SMetaReader mr = {0};
   metaReaderInit(&mr, pHandle->meta, 0);
@@ -2073,6 +2065,9 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo
     } else if (QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN == type) {
       SSystemTableScanPhysiNode* pSysScanPhyNode = (SSystemTableScanPhysiNode*)pPhyNode;
       pOperator = createSysTableScanOperatorInfo(pHandle, pSysScanPhyNode, pUser, pTaskInfo);
+    } else if (QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN == type) {
+      STableCountScanPhysiNode* pTblCountScanNode = (STableCountScanPhysiNode*)pPhyNode;
+      pOperator = createTableCountScanOperatorInfo(pHandle, pTblCountScanNode, pTaskInfo);
     } else if (QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN == type) {
       STagScanPhysiNode* pScanPhyNode = (STagScanPhysiNode*)pPhyNode;
 

source/libs/executor/src/scanoperator.c

@@ -30,7 +30,6 @@
 #include "tcompare.h"
 #include "thash.h"
 #include "ttypes.h"
-#include "vnode.h"
 
 #define SET_REVERSE_SCAN_FLAG(_info) ((_info)->scanFlag = REVERSE_SCAN)
 #define SWITCH_ORDER(n)              (((n) = ((n) == TSDB_ORDER_ASC) ? TSDB_ORDER_DESC : TSDB_ORDER_ASC))
@@ -629,7 +628,7 @@ static SSDataBlock* doTableScanImpl(SOperatorInfo* pOperator) {
 
   while (tsdbNextDataBlock(pTableScanInfo->base.dataReader)) {
     if (isTaskKilled(pTaskInfo)) {
-      T_LONG_JMP(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
+      T_LONG_JMP(pTaskInfo->env, pTaskInfo->code);
     }
 
     // process this data block based on the probabilities
@@ -2032,7 +2031,7 @@ static SSDataBlock* doRawScan(SOperatorInfo* pOperator) {
 
     if (pInfo->dataReader && tsdbNextDataBlock(pInfo->dataReader)) {
       if (isTaskKilled(pTaskInfo)) {
-        longjmp(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
+        longjmp(pTaskInfo->env, pTaskInfo->code);
       }
 
       int32_t rows = 0;
@@ -2529,7 +2528,7 @@ static SSDataBlock* getTableDataBlockImpl(void* param) {
   STsdbReader* reader = pInfo->base.dataReader;
   while (tsdbNextDataBlock(reader)) {
     if (isTaskKilled(pTaskInfo)) {
-      T_LONG_JMP(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
+      T_LONG_JMP(pTaskInfo->env, pTaskInfo->code);
     }
 
     // process this data block based on the probabilities
@@ -2913,3 +2912,305 @@ _error:
   taosMemoryFree(pOperator);
   return NULL;
 }
+
+// ====================================================================================================================
+// TableCountScanOperator
+static SSDataBlock* doTableCountScan(SOperatorInfo* pOperator);
+static void         destoryTableCountScanOperator(void* param);
+static const char*  GROUP_TAG_DB_NAME = "db_name";
+static const char*  GROUP_TAG_STABLE_NAME = "stable_name";
+
+int32_t tblCountScanGetGroupTagsSlotId(const SNodeList* scanCols, STableCountScanSupp* supp) {
+  if (scanCols != NULL) {
+    SNode* pNode = NULL;
+    FOREACH(pNode, scanCols) {
+      if (nodeType(pNode) != QUERY_NODE_TARGET) {
+        return TSDB_CODE_QRY_SYS_ERROR;
+      }
+      STargetNode* targetNode = (STargetNode*)pNode;
+      if (nodeType(targetNode->pExpr) != QUERY_NODE_COLUMN) {
+        return TSDB_CODE_QRY_SYS_ERROR;
+      }
+      SColumnNode* colNode = (SColumnNode*)(targetNode->pExpr);
+      if (strcmp(colNode->colName, GROUP_TAG_DB_NAME) == 0) {
+        supp->dbNameSlotId = targetNode->slotId;
+      } else if (strcmp(colNode->colName, GROUP_TAG_STABLE_NAME) == 0) {
+        supp->stbNameSlotId = targetNode->slotId;
+      }
+    }
+  }
+  return TSDB_CODE_SUCCESS;
+}
+
+int32_t tblCountScanGetCountSlotId(const SNodeList* pseudoCols, STableCountScanSupp* supp) {
+  if (pseudoCols != NULL) {
+    SNode* pNode = NULL;
+    FOREACH(pNode, pseudoCols) {
+      if (nodeType(pNode) != QUERY_NODE_TARGET) {
+        return TSDB_CODE_QRY_SYS_ERROR;
+      }
+      STargetNode* targetNode = (STargetNode*)pNode;
+      if (nodeType(targetNode->pExpr) != QUERY_NODE_FUNCTION) {
+        return TSDB_CODE_QRY_SYS_ERROR;
+      }
+      SFunctionNode* funcNode = (SFunctionNode*)(targetNode->pExpr);
+      if (funcNode->funcType == FUNCTION_TYPE_TABLE_COUNT) {
+        supp->tbCountSlotId = targetNode->slotId;
+      }
+    }
+  }
+  return TSDB_CODE_SUCCESS;
+}
+
+int32_t tblCountScanGetInputs(SNodeList* groupTags, SName* tableName, STableCountScanSupp* supp) {
+  if (groupTags != NULL) {
+    SNode* pNode = NULL;
+    FOREACH(pNode, groupTags) {
+      if (nodeType(pNode) != QUERY_NODE_COLUMN) {
+        return TSDB_CODE_QRY_SYS_ERROR;
+      }
+      SColumnNode* colNode = (SColumnNode*)pNode;
+      if (strcmp(colNode->colName, GROUP_TAG_DB_NAME) == 0) {
+        supp->groupByDbName = true;
+      }
+      if (strcmp(colNode->colName, GROUP_TAG_STABLE_NAME) == 0) {
+        supp->groupByStbName = true;
+      }
+    }
+  } else {
+    strncpy(supp->dbName, tNameGetDbNameP(tableName), TSDB_DB_NAME_LEN);
+    strncpy(supp->stbName, tNameGetTableName(tableName), TSDB_TABLE_NAME_LEN);
+  }
+  return TSDB_CODE_SUCCESS;
+}
+
+int32_t getTableCountScanSupp(SNodeList* groupTags, SName* tableName, SNodeList* scanCols, SNodeList* pseudoCols,
+                              STableCountScanSupp* supp, SExecTaskInfo* taskInfo) {
+  int32_t code = 0;
+  code = tblCountScanGetInputs(groupTags, tableName, supp);
+  if (code != TSDB_CODE_SUCCESS) {
+    qError("%s get table count scan supp. get inputs error", GET_TASKID(taskInfo));
+    return code;
+  }
+  supp->dbNameSlotId = -1;
+  supp->stbNameSlotId = -1;
+  supp->tbCountSlotId = -1;
+
+  code = tblCountScanGetGroupTagsSlotId(scanCols, supp);
+  if (code != TSDB_CODE_SUCCESS) {
+    qError("%s get table count scan supp. get group tags slot id error", GET_TASKID(taskInfo));
+    return code;
+  }
+  code = tblCountScanGetCountSlotId(pseudoCols, supp);
+  if (code != TSDB_CODE_SUCCESS) {
+    qError("%s get table count scan supp. get count error", GET_TASKID(taskInfo));
+    return code;
+  }
+  return code;
+}
+
+SOperatorInfo* createTableCountScanOperatorInfo(SReadHandle* readHandle, STableCountScanPhysiNode* pTblCountScanNode,
+                                                SExecTaskInfo* pTaskInfo) {
+  int32_t code = TSDB_CODE_SUCCESS;
+
+  SScanPhysiNode*              pScanNode = &pTblCountScanNode->scan;
+  STableCountScanOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(STableCountScanOperatorInfo));
+  SOperatorInfo*               pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
+
+  if (!pInfo || !pOperator) {
+    goto _error;
+  }
+
+  pInfo->readHandle = *readHandle;
+
+  SDataBlockDescNode* pDescNode = pScanNode->node.pOutputDataBlockDesc;
+  initResultSizeInfo(&pOperator->resultInfo, 1);
+  pInfo->pRes = createDataBlockFromDescNode(pDescNode);
+  blockDataEnsureCapacity(pInfo->pRes, pOperator->resultInfo.capacity);
+
+  getTableCountScanSupp(pTblCountScanNode->pGroupTags, &pTblCountScanNode->scan.tableName,
+                        pTblCountScanNode->scan.pScanCols, pTblCountScanNode->scan.pScanPseudoCols, &pInfo->supp,
+                        pTaskInfo);
+
+  setOperatorInfo(pOperator, "TableCountScanOperator", QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN, false, OP_NOT_OPENED,
+                  pInfo, pTaskInfo);
+  pOperator->fpSet =
+      createOperatorFpSet(operatorDummyOpenFn, doTableCountScan, NULL, destoryTableCountScanOperator, NULL);
+  return pOperator;
+
+_error:
+  if (pInfo != NULL) {
+    destoryTableCountScanOperator(pInfo);
+  }
+  taosMemoryFreeClear(pOperator);
+  pTaskInfo->code = code;
+  return NULL;
+}
+
+void fillTableCountScanDataBlock(STableCountScanSupp* pSupp, char* dbName, char* stbName, int64_t count,
+                                 SSDataBlock* pRes) {
+  if (pSupp->dbNameSlotId != -1) {
+    ASSERT(strlen(dbName));
+    SColumnInfoData* colInfoData = taosArrayGet(pRes->pDataBlock, pSupp->dbNameSlotId);
+    char             varDbName[TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
+    strncpy(varDataVal(varDbName), dbName, strlen(dbName));
+    varDataSetLen(varDbName, strlen(dbName));
+    colDataAppend(colInfoData, 0, varDbName, false);
+  }
+
+  if (pSupp->stbNameSlotId != -1) {
+    SColumnInfoData* colInfoData = taosArrayGet(pRes->pDataBlock, pSupp->stbNameSlotId);
+    if (strlen(stbName) != 0) {
+      char             varStbName[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
+      strncpy(varDataVal(varStbName), stbName, strlen(stbName));
+      varDataSetLen(varStbName, strlen(stbName));
+      colDataAppend(colInfoData, 0, varStbName, false);
+    } else {
+      colDataAppendNULL(colInfoData, 0);
+    }
+  }
+
+  if (pSupp->tbCountSlotId != -1) {
+    SColumnInfoData* colInfoData = taosArrayGet(pRes->pDataBlock, pSupp->tbCountSlotId);
+    colDataAppend(colInfoData, 0, (char*)&count, false);
+  }
+  pRes->info.rows = 1;
+}
+
+static SSDataBlock* buildSysDbTableCount(SOperatorInfo* pOperator, STableCountScanOperatorInfo* pInfo) {
+  STableCountScanSupp* pSupp = &pInfo->supp;
+  SSDataBlock*         pRes = pInfo->pRes;
+
+  size_t infodbTableNum;
+  getInfosDbMeta(NULL, &infodbTableNum);
+  size_t perfdbTableNum;
+  getPerfDbMeta(NULL, &perfdbTableNum);
+
+  if (pSupp->groupByDbName) {
+    if (pInfo->currGrpIdx == 0) {
+      uint64_t groupId = calcGroupId(TSDB_INFORMATION_SCHEMA_DB, strlen(TSDB_INFORMATION_SCHEMA_DB));
+      pRes->info.id.groupId = groupId;
+      fillTableCountScanDataBlock(pSupp, TSDB_INFORMATION_SCHEMA_DB, "", infodbTableNum, pRes);
+    } else if (pInfo->currGrpIdx == 1) {
+      uint64_t groupId = calcGroupId(TSDB_PERFORMANCE_SCHEMA_DB, strlen(TSDB_PERFORMANCE_SCHEMA_DB));
+      pRes->info.id.groupId = groupId;
+      fillTableCountScanDataBlock(pSupp, TSDB_PERFORMANCE_SCHEMA_DB, "", perfdbTableNum, pRes);
+    } else {
+      setOperatorCompleted(pOperator);
+      return NULL;
+    }
+    pInfo->currGrpIdx++;
+    return (pRes->info.rows > 0) ? pRes : NULL;
+  } else {
+    if (strcmp(pSupp->dbName, TSDB_INFORMATION_SCHEMA_DB) == 0) {
+      fillTableCountScanDataBlock(pSupp, TSDB_INFORMATION_SCHEMA_DB, "", infodbTableNum, pRes);
+    } else if (strcmp(pSupp->dbName, TSDB_PERFORMANCE_SCHEMA_DB) == 0) {
+      fillTableCountScanDataBlock(pSupp, TSDB_PERFORMANCE_SCHEMA_DB, "", perfdbTableNum, pRes);
+    } else if (strlen(pSupp->dbName) == 0) {
+      fillTableCountScanDataBlock(pSupp, "", "", infodbTableNum + perfdbTableNum, pRes);
+    }
+    setOperatorCompleted(pOperator);
+    return (pRes->info.rows > 0) ? pRes : NULL;
+  }
+}
+
+static SSDataBlock* doTableCountScan(SOperatorInfo* pOperator) {
+  SExecTaskInfo*               pTaskInfo = pOperator->pTaskInfo;
+  STableCountScanOperatorInfo* pInfo = pOperator->info;
+  STableCountScanSupp*         pSupp = &pInfo->supp;
+  SSDataBlock*                 pRes = pInfo->pRes;
+  blockDataCleanup(pRes);
+
+  if (pOperator->status == OP_EXEC_DONE) {
+    return NULL;
+  }
+  if (pInfo->readHandle.mnd != NULL) {
+    return buildSysDbTableCount(pOperator, pInfo);
+  }
+
+  const char* db = NULL;
+  int32_t     vgId = 0;
+  char        dbName[TSDB_DB_NAME_LEN] = {0};
+
+  {
+    // get dbname
+    vnodeGetInfo(pInfo->readHandle.vnode, &db, &vgId);
+    SName            sn = {0};
+    tNameFromString(&sn, db, T_NAME_ACCT | T_NAME_DB);
+    tNameGetDbName(&sn, dbName);
+  }
+  if (pSupp->groupByDbName) {
+    if (pSupp->groupByStbName) {
+      if (pInfo->stbUidList == NULL) {
+        pInfo->stbUidList = taosArrayInit(16, sizeof(tb_uid_t));
+        if (vnodeGetStbIdList(pInfo->readHandle.vnode, 0, pInfo->stbUidList) < 0) {
+          qError("vgId:%d, failed to get stb id list error: %s", vgId, terrstr());
+        }
+      }
+      if (pInfo->currGrpIdx < taosArrayGetSize(pInfo->stbUidList)) {
+        tb_uid_t stbUid = *(tb_uid_t*)taosArrayGet(pInfo->stbUidList, pInfo->currGrpIdx);
+
+        char stbName[TSDB_TABLE_NAME_LEN] = {0};
+        metaGetTableSzNameByUid(pInfo->readHandle.meta, stbUid, stbName);
+
+        char fullStbName[TSDB_TABLE_FNAME_LEN] = {0};
+        snprintf(fullStbName, TSDB_TABLE_FNAME_LEN, "%s.%s", dbName, stbName);
+        uint64_t groupId = calcGroupId(fullStbName, strlen(fullStbName));
+        pRes->info.id.groupId = groupId;
+
+        SMetaStbStats stats = {0};
+        metaGetStbStats(pInfo->readHandle.meta, stbUid, &stats);
+        int64_t ctbNum = stats.ctbNum;
+
+        fillTableCountScanDataBlock(pSupp, dbName, stbName, ctbNum, pRes);
+        
+        pInfo->currGrpIdx++;
+      } else if (pInfo->currGrpIdx == taosArrayGetSize(pInfo->stbUidList)) {
+        char fullStbName[TSDB_TABLE_FNAME_LEN] = {0};
+        snprintf(fullStbName, TSDB_TABLE_FNAME_LEN, "%s.%s", dbName, "");
+        uint64_t groupId = calcGroupId(fullStbName, strlen(fullStbName));
+        pRes->info.id.groupId = groupId;
+        int64_t ntbNum = metaGetNtbNum(pInfo->readHandle.meta);
+        fillTableCountScanDataBlock(pSupp, dbName, "", ntbNum, pRes);
+
+        pInfo->currGrpIdx++;
+      } else {
+        setOperatorCompleted(pOperator);
+        return NULL;
+      }
+    } else {
+      uint64_t groupId = calcGroupId(dbName, strlen(dbName));
+      pRes->info.id.groupId = groupId;
+      int64_t dbTableCount = metaGetTbNum(pInfo->readHandle.meta);
+      fillTableCountScanDataBlock(pSupp, dbName, "", dbTableCount, pRes);
+      setOperatorCompleted(pOperator);
+    }
+  } else {
+    if (strlen(pSupp->dbName) != 0) {
+      if (strlen(pSupp->stbName) != 0) {
+        tb_uid_t      uid = metaGetTableEntryUidByName(pInfo->readHandle.meta, pSupp->stbName);
+        SMetaStbStats stats = {0};
+        metaGetStbStats(pInfo->readHandle.meta, uid, &stats);
+        int64_t ctbNum = stats.ctbNum;
+        fillTableCountScanDataBlock(pSupp, dbName, pSupp->stbName, ctbNum, pRes);
+      } else {
+        int64_t tbNumVnode = metaGetTbNum(pInfo->readHandle.meta);
+        fillTableCountScanDataBlock(pSupp, dbName, "", tbNumVnode, pRes);
+      }
+    } else {
+      int64_t tbNumVnode = metaGetTbNum(pInfo->readHandle.meta);
+      fillTableCountScanDataBlock(pSupp, dbName, "", tbNumVnode, pRes);
+    }
+    setOperatorCompleted(pOperator);
+  }
+  return pRes->info.rows > 0 ? pRes : NULL;
+}
+
+static void destoryTableCountScanOperator(void* param) {
+  STableCountScanOperatorInfo* pTableCountScanInfo = param;
+  blockDataDestroy(pTableCountScanInfo->pRes);
+
+  nodesDestroyList(pTableCountScanInfo->groupTags);
+  taosArrayDestroy(pTableCountScanInfo->stbUidList);
+  taosMemoryFreeClear(param);
+}

source/libs/function/src/builtins.c

@@ -2080,6 +2080,11 @@ static int32_t translateTagsPseudoColumn(SFunctionNode* pFunc, char* pErrBuf, in
   return TSDB_CODE_SUCCESS;
 }
 
+static int32_t translateTableCountPseudoColumn(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
+  pFunc->node.resType = (SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes, .type = TSDB_DATA_TYPE_BIGINT};
+  return TSDB_CODE_SUCCESS;
+}
+
 // clang-format off
 const SBuiltinFuncDefinition funcMgtBuiltins[] = {
   {
@@ -3241,6 +3246,16 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
     .sprocessFunc = NULL,
     .finalizeFunc = NULL
   },
+  {
+    .name = "_table_count",
+    .type = FUNCTION_TYPE_TABLE_COUNT,
+    .classification = FUNC_MGT_PSEUDO_COLUMN_FUNC | FUNC_MGT_SCAN_PC_FUNC,
+    .translateFunc = translateTableCountPseudoColumn,
+    .getEnvFunc   = NULL,
+    .initFunc     = NULL,
+    .sprocessFunc = NULL,
+    .finalizeFunc = NULL
+  },
 };
 // clang-format on
 

source/libs/function/src/udfd.c

@@ -598,8 +598,8 @@ int32_t udfdLoadUdf(char *udfName, SUdf *udf) {
   return 0;
 }
 static bool udfdRpcRfp(int32_t code, tmsg_t msgType) {
-  if (code == TSDB_CODE_RPC_REDIRECT || code == TSDB_CODE_RPC_NETWORK_UNAVAIL || code == TSDB_CODE_NODE_NOT_DEPLOYED ||
+  if (code == TSDB_CODE_RPC_REDIRECT || code == TSDB_CODE_RPC_NETWORK_UNAVAIL || code == TSDB_CODE_RPC_BROKEN_LINK ||
-      code == TSDB_CODE_SYN_NOT_LEADER || code == TSDB_CODE_APP_NOT_READY || code == TSDB_CODE_RPC_BROKEN_LINK) {
+      code == TSDB_CODE_SYN_NOT_LEADER || code == TSDB_CODE_SYN_RESTORING || code == TSDB_CODE_MNODE_NOT_FOUND|| code == TSDB_CODE_APP_IS_STARTING || code == TSDB_CODE_APP_IS_STOPPING) {
     if (msgType == TDMT_SCH_QUERY || msgType == TDMT_SCH_MERGE_QUERY || msgType == TDMT_SCH_FETCH || msgType == TDMT_SCH_MERGE_FETCH) {
       return false;
     } 

source/libs/nodes/src/nodesCodeFuncs.c

@@ -221,6 +221,8 @@ const char* nodesNodeName(ENodeType type) {
       return "PhysiTableScan";
     case QUERY_NODE_PHYSICAL_PLAN_TABLE_SEQ_SCAN:
       return "PhysiTableSeqScan";
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN:
+      return "PhysiTableMergeScan";
     case QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN:
       return "PhysiSreamScan";
     case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN:
@@ -229,8 +231,8 @@ const char* nodesNodeName(ENodeType type) {
       return "PhysiBlockDistScan";
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN:
       return "PhysiLastRowScan";
-    case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN:
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
-      return "PhysiTableMergeScan";
+      return "PhysiTableCountScan";
     case QUERY_NODE_PHYSICAL_PLAN_PROJECT:
       return "PhysiProject";
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN:
@@ -4646,6 +4648,7 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
     case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN:
       return physiScanNodeToJson(pObj, pJson);
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN:
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
       return physiLastRowScanNodeToJson(pObj, pJson);
     case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN:
     case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN:
@@ -4800,6 +4803,7 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
       return jsonToLogicPlan(pJson, pObj);
     case QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN:
     case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN:
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
       return jsonToPhysiScanNode(pJson, pObj);
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN:
       return jsonToPhysiLastRowScanNode(pJson, pObj);

source/libs/nodes/src/nodesMsgFuncs.c

@@ -3640,6 +3640,7 @@ static int32_t specificNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
       code = physiScanNodeToMsg(pObj, pEncoder);
       break;
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN:
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
       code = physiLastRowScanNodeToMsg(pObj, pEncoder);
       break;
     case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN:
@@ -3778,6 +3779,7 @@ static int32_t msgToSpecificNode(STlvDecoder* pDecoder, void* pObj) {
       code = msgToPhysiScanNode(pDecoder, pObj);
       break;
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN:
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
       code = msgToPhysiLastRowScanNode(pDecoder, pObj);
       break;
     case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN:

source/libs/nodes/src/nodesUtilFuncs.c

@@ -494,6 +494,8 @@ SNode* nodesMakeNode(ENodeType type) {
       return makeNode(type, sizeof(SBlockDistScanPhysiNode));
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN:
       return makeNode(type, sizeof(SLastRowScanPhysiNode));
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
+      return makeNode(type, sizeof(STableCountScanPhysiNode));
     case QUERY_NODE_PHYSICAL_PLAN_PROJECT:
       return makeNode(type, sizeof(SProjectPhysiNode));
     case QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN:
@@ -1120,6 +1122,7 @@ void nodesDestroyNode(SNode* pNode) {
     case QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN:
     case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN:
     case QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN:
+    case QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN:
       destroyScanPhysiNode((SScanPhysiNode*)pNode);
       break;
     case QUERY_NODE_PHYSICAL_PLAN_LAST_ROW_SCAN: {

source/libs/parser/inc/parUtil.h

@@ -86,7 +86,7 @@ STableComInfo getTableInfo(const STableMeta* pTableMeta);
 STableMeta*   tableMetaDup(const STableMeta* pTableMeta);
 
 int32_t trimString(const char* src, int32_t len, char* dst, int32_t dlen);
-int32_t getInsTagsTableTargetName(int32_t acctId, SNode* pWhere, SName* pName);
+int32_t getVnodeSysTableTargetName(int32_t acctId, SNode* pWhere, SName* pName);
 
 int32_t buildCatalogReq(const SParseMetaCache* pMetaCache, SCatalogReq* pCatalogReq);
 int32_t putMetaDataToCache(const SCatalogReq* pCatalogReq, const SMetaData* pMetaData, SParseMetaCache* pMetaCache);

source/libs/parser/src/parAstParser.c

@@ -140,7 +140,7 @@ static int32_t collectMetaKeyFromInsTagsImpl(SCollectMetaKeyCxt* pCxt, SName* pN
 static int32_t collectMetaKeyFromInsTags(SCollectMetaKeyCxt* pCxt) {
   SSelectStmt* pSelect = (SSelectStmt*)pCxt->pStmt;
   SName        name = {0};
-  int32_t      code = getInsTagsTableTargetName(pCxt->pParseCxt->acctId, pSelect->pWhere, &name);
+  int32_t      code = getVnodeSysTableTargetName(pCxt->pParseCxt->acctId, pSelect->pWhere, &name);
   if (TSDB_CODE_SUCCESS == code) {
     code = collectMetaKeyFromInsTagsImpl(pCxt, &name);
   }
@@ -165,7 +165,8 @@ static int32_t collectMetaKeyFromRealTableImpl(SCollectMetaKeyCxt* pCxt, const c
   if (TSDB_CODE_SUCCESS == code && (0 == strcmp(pTable, TSDB_INS_TABLE_DNODE_VARIABLES))) {
     code = reserveDnodeRequiredInCache(pCxt->pMetaCache);
   }
-  if (TSDB_CODE_SUCCESS == code && (0 == strcmp(pTable, TSDB_INS_TABLE_TAGS)) &&
+  if (TSDB_CODE_SUCCESS == code &&
+      (0 == strcmp(pTable, TSDB_INS_TABLE_TAGS) || 0 == strcmp(pTable, TSDB_INS_TABLE_TABLES)) &&
       QUERY_NODE_SELECT_STMT == nodeType(pCxt->pStmt)) {
     code = collectMetaKeyFromInsTags(pCxt);
   }

source/libs/parser/src/parTranslater.c

@@ -47,6 +47,7 @@ typedef struct STranslateContext {
   SParseMetaCache* pMetaCache;
   bool             createStream;
   bool             stableQuery;
+  bool             showRewrite;
 } STranslateContext;
 
 typedef struct SFullDatabaseName {
@@ -750,8 +751,8 @@ static bool isPrimaryKeyImpl(SNode* pExpr) {
     return (PRIMARYKEY_TIMESTAMP_COL_ID == ((SColumnNode*)pExpr)->colId);
   } else if (QUERY_NODE_FUNCTION == nodeType(pExpr)) {
     SFunctionNode* pFunc = (SFunctionNode*)pExpr;
-    if (FUNCTION_TYPE_SELECT_VALUE == pFunc->funcType || FUNCTION_TYPE_FIRST == pFunc->funcType ||
+    if (FUNCTION_TYPE_SELECT_VALUE == pFunc->funcType || FUNCTION_TYPE_GROUP_KEY == pFunc->funcType ||
-        FUNCTION_TYPE_LAST == pFunc->funcType) {
+        FUNCTION_TYPE_FIRST == pFunc->funcType || FUNCTION_TYPE_LAST == pFunc->funcType) {
       return isPrimaryKeyImpl(nodesListGetNode(pFunc->pParameterList, 0));
     } else if (FUNCTION_TYPE_WSTART == pFunc->funcType || FUNCTION_TYPE_WEND == pFunc->funcType) {
       return true;
@@ -2209,22 +2210,28 @@ static int32_t dnodeToVgroupsInfo(SArray* pDnodes, SVgroupsInfo** pVgsInfo) {
 }
 
 static bool sysTableFromVnode(const char* pTable) {
-  return (0 == strcmp(pTable, TSDB_INS_TABLE_TABLES)) ||
+  return ((0 == strcmp(pTable, TSDB_INS_TABLE_TABLES)) || (0 == strcmp(pTable, TSDB_INS_TABLE_TAGS)));
-         (0 == strcmp(pTable, TSDB_INS_TABLE_TABLE_DISTRIBUTED) || (0 == strcmp(pTable, TSDB_INS_TABLE_TAGS)));
 }
 
 static bool sysTableFromDnode(const char* pTable) { return 0 == strcmp(pTable, TSDB_INS_TABLE_DNODE_VARIABLES); }
 
-static int32_t getTagsTableVgroupListImpl(STranslateContext* pCxt, SName* pTargetName, SName* pName,
+static int32_t getVnodeSysTableVgroupListImpl(STranslateContext* pCxt, SName* pTargetName, SName* pName,
-                                          SArray** pVgroupList) {
+                                              SArray** pVgroupList) {
   if (0 == pTargetName->type) {
     return getDBVgInfoImpl(pCxt, pName, pVgroupList);
   }
 
+  if (0 == strcmp(pTargetName->dbname, TSDB_INFORMATION_SCHEMA_DB) ||
+      0 == strcmp(pTargetName->dbname, TSDB_PERFORMANCE_SCHEMA_DB)) {
+    pTargetName->type = 0;
+    return TSDB_CODE_SUCCESS;
+  }
+
   if (TSDB_DB_NAME_T == pTargetName->type) {
     int32_t code = getDBVgInfoImpl(pCxt, pTargetName, pVgroupList);
-    if (TSDB_CODE_MND_DB_NOT_EXIST == code || TSDB_CODE_MND_DB_IN_CREATING == code ||
+    if (!pCxt->showRewrite && (TSDB_CODE_MND_DB_NOT_EXIST == code || TSDB_CODE_MND_DB_IN_CREATING == code ||
-        TSDB_CODE_MND_DB_IN_DROPPING == code) {
+                               TSDB_CODE_MND_DB_IN_DROPPING == code)) {
+      // system table query should not report errors
       code = TSDB_CODE_SUCCESS;
     }
     return code;
@@ -2241,50 +2248,44 @@ static int32_t getTagsTableVgroupListImpl(STranslateContext* pCxt, SName* pTarge
     }
   } else if (TSDB_CODE_MND_DB_NOT_EXIST == code || TSDB_CODE_MND_DB_IN_CREATING == code ||
              TSDB_CODE_MND_DB_IN_DROPPING == code) {
+    // system table query should not report errors
     code = TSDB_CODE_SUCCESS;
   }
   return code;
 }
 
-static int32_t getTagsTableVgroupList(STranslateContext* pCxt, SName* pName, SArray** pVgroupList) {
+static int32_t getVnodeSysTableVgroupList(STranslateContext* pCxt, SName* pName, SArray** pVgs, bool* pHasUserDbCond) {
   if (!isSelectStmt(pCxt->pCurrStmt)) {
     return TSDB_CODE_SUCCESS;
   }
   SSelectStmt* pSelect = (SSelectStmt*)pCxt->pCurrStmt;
   SName        targetName = {0};
-  int32_t      code = getInsTagsTableTargetName(pCxt->pParseCxt->acctId, pSelect->pWhere, &targetName);
+  int32_t      code = getVnodeSysTableTargetName(pCxt->pParseCxt->acctId, pSelect->pWhere, &targetName);
   if (TSDB_CODE_SUCCESS == code) {
-    code = getTagsTableVgroupListImpl(pCxt, &targetName, pName, pVgroupList);
+    code = getVnodeSysTableVgroupListImpl(pCxt, &targetName, pName, pVgs);
   }
+  *pHasUserDbCond = (0 != targetName.type && taosArrayGetSize(*pVgs) > 0);
   return code;
 }
 
 static int32_t setVnodeSysTableVgroupList(STranslateContext* pCxt, SName* pName, SRealTableNode* pRealTable) {
-  int32_t code = TSDB_CODE_SUCCESS;
+  bool    hasUserDbCond = false;
-  SArray* vgroupList = NULL;
+  SArray* pVgs = NULL;
-  if (0 == strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TAGS)) {
+  int32_t code = getVnodeSysTableVgroupList(pCxt, pName, &pVgs, &hasUserDbCond);
-    code = getTagsTableVgroupList(pCxt, pName, &vgroupList);
-  } else if ('\0' != pRealTable->qualDbName[0]) {
-    if (0 != strcmp(pRealTable->qualDbName, TSDB_INFORMATION_SCHEMA_DB)) {
-      code = getDBVgInfo(pCxt, pRealTable->qualDbName, &vgroupList);
-    }
-  } else {
-    code = getDBVgInfoImpl(pCxt, pName, &vgroupList);
-  }
 
   if (TSDB_CODE_SUCCESS == code && 0 == strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TAGS) &&
-      isSelectStmt(pCxt->pCurrStmt) && 0 == taosArrayGetSize(vgroupList)) {
+      isSelectStmt(pCxt->pCurrStmt) && 0 == taosArrayGetSize(pVgs)) {
     ((SSelectStmt*)pCxt->pCurrStmt)->isEmptyResult = true;
   }
 
-  if (TSDB_CODE_SUCCESS == code && 0 == strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TABLES)) {
+  if (TSDB_CODE_SUCCESS == code && 0 == strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TABLES) && !hasUserDbCond) {
-    code = addMnodeToVgroupList(&pCxt->pParseCxt->mgmtEpSet, &vgroupList);
+    code = addMnodeToVgroupList(&pCxt->pParseCxt->mgmtEpSet, &pVgs);
   }
 
   if (TSDB_CODE_SUCCESS == code) {
-    code = toVgroupsInfo(vgroupList, &pRealTable->pVgroupList);
+    code = toVgroupsInfo(pVgs, &pRealTable->pVgroupList);
   }
-  taosArrayDestroy(vgroupList);
+  taosArrayDestroy(pVgs);
 
   return code;
 }
@@ -2309,30 +2310,39 @@ static int32_t setSysTableVgroupList(STranslateContext* pCxt, SName* pName, SRea
   }
 }
 
+static int32_t setSuperTableVgroupList(STranslateContext* pCxt, SName* pName, SRealTableNode* pRealTable) {
+  SArray* vgroupList = NULL;
+  int32_t code = getDBVgInfoImpl(pCxt, pName, &vgroupList);
+  if (TSDB_CODE_SUCCESS == code) {
+    code = toVgroupsInfo(vgroupList, &pRealTable->pVgroupList);
+  }
+  taosArrayDestroy(vgroupList);
+  return code;
+}
+
+static int32_t setNormalTableVgroupList(STranslateContext* pCxt, SName* pName, SRealTableNode* pRealTable) {
+  pRealTable->pVgroupList = taosMemoryCalloc(1, sizeof(SVgroupsInfo) + sizeof(SVgroupInfo));
+  if (NULL == pRealTable->pVgroupList) {
+    return TSDB_CODE_OUT_OF_MEMORY;
+  }
+  pRealTable->pVgroupList->numOfVgroups = 1;
+  return getTableHashVgroupImpl(pCxt, pName, pRealTable->pVgroupList->vgroups);
+}
+
 static int32_t setTableVgroupList(STranslateContext* pCxt, SName* pName, SRealTableNode* pRealTable) {
   if (pCxt->pParseCxt->topicQuery) {
     return TSDB_CODE_SUCCESS;
   }
 
-  int32_t code = TSDB_CODE_SUCCESS;
   if (TSDB_SUPER_TABLE == pRealTable->pMeta->tableType) {
-    SArray* vgroupList = NULL;
+    return setSuperTableVgroupList(pCxt, pName, pRealTable);
-    code = getDBVgInfoImpl(pCxt, pName, &vgroupList);
-    if (TSDB_CODE_SUCCESS == code) {
-      code = toVgroupsInfo(vgroupList, &pRealTable->pVgroupList);
-    }
-    taosArrayDestroy(vgroupList);
-  } else if (TSDB_SYSTEM_TABLE == pRealTable->pMeta->tableType) {
-    code = setSysTableVgroupList(pCxt, pName, pRealTable);
-  } else {
-    pRealTable->pVgroupList = taosMemoryCalloc(1, sizeof(SVgroupsInfo) + sizeof(SVgroupInfo));
-    if (NULL == pRealTable->pVgroupList) {
-      return TSDB_CODE_OUT_OF_MEMORY;
-    }
-    pRealTable->pVgroupList->numOfVgroups = 1;
-    code = getTableHashVgroupImpl(pCxt, pName, pRealTable->pVgroupList->vgroups);
   }
-  return code;
+
+  if (TSDB_SYSTEM_TABLE == pRealTable->pMeta->tableType) {
+    return setSysTableVgroupList(pCxt, pName, pRealTable);
+  }
+
+  return setNormalTableVgroupList(pCxt, pName, pRealTable);
 }
 
 static uint8_t getStmtPrecision(SNode* pStmt) {
@@ -2366,7 +2376,6 @@ static bool isSingleTable(SRealTableNode* pRealTable) {
   int8_t tableType = pRealTable->pMeta->tableType;
   if (TSDB_SYSTEM_TABLE == tableType) {
     return 0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TABLES) &&
-           0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TABLE_DISTRIBUTED) &&
            0 != strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_TAGS);
   }
   return (TSDB_CHILD_TABLE == tableType || TSDB_NORMAL_TABLE == tableType);
@@ -6296,6 +6305,7 @@ static int32_t rewriteShow(STranslateContext* pCxt, SQuery* pQuery) {
     code = createShowCondition((SShowStmt*)pQuery->pRoot, pStmt);
   }
   if (TSDB_CODE_SUCCESS == code) {
+    pCxt->showRewrite = true;
     pQuery->showRewrite = true;
     nodesDestroyNode(pQuery->pRoot);
     pQuery->pRoot = (SNode*)pStmt;
@@ -6349,6 +6359,7 @@ static int32_t rewriteShowStableTags(STranslateContext* pCxt, SQuery* pQuery) {
     code = createShowTableTagsProjections(&pSelect->pProjectionList, &pShow->pTags);
   }
   if (TSDB_CODE_SUCCESS == code) {
+    pCxt->showRewrite = true;
     pQuery->showRewrite = true;
     pSelect->tagScan = true;
     nodesDestroyNode(pQuery->pRoot);
@@ -6379,6 +6390,7 @@ static int32_t rewriteShowDnodeVariables(STranslateContext* pCxt, SQuery* pQuery
     }
   }
   if (TSDB_CODE_SUCCESS == code) {
+    pCxt->showRewrite = true;
     pQuery->showRewrite = true;
     nodesDestroyNode(pQuery->pRoot);
     pQuery->pRoot = (SNode*)pSelect;
@@ -6398,6 +6410,7 @@ static int32_t rewriteShowVnodes(STranslateContext* pCxt, SQuery* pQuery) {
     }
   }
   if (TSDB_CODE_SUCCESS == code) {
+    pCxt->showRewrite = true;
     pQuery->showRewrite = true;
     nodesDestroyNode(pQuery->pRoot);
     pQuery->pRoot = (SNode*)pStmt;
@@ -6439,6 +6452,7 @@ static int32_t rewriteShowTableDist(STranslateContext* pCxt, SQuery* pQuery) {
     code = nodesListMakeStrictAppend(&pStmt->pProjectionList, createBlockDistFunc());
   }
   if (TSDB_CODE_SUCCESS == code) {
+    pCxt->showRewrite = true;
     pQuery->showRewrite = true;
     nodesDestroyNode(pQuery->pRoot);
     pQuery->pRoot = (SNode*)pStmt;

source/libs/parser/src/parUtil.c

@@ -474,7 +474,7 @@ static int32_t getInsTagsTableTargetNameFromCond(int32_t acctId, SLogicCondition
   return TSDB_CODE_SUCCESS;
 }
 
-int32_t getInsTagsTableTargetName(int32_t acctId, SNode* pWhere, SName* pName) {
+int32_t getVnodeSysTableTargetName(int32_t acctId, SNode* pWhere, SName* pName) {
   if (NULL == pWhere) {
     return TSDB_CODE_SUCCESS;
   }

source/libs/parser/test/mockCatalog.cpp

@@ -65,9 +65,10 @@ void generateInformationSchema(MockCatalogService* mcs) {
       .addColumn("db_name", TSDB_DATA_TYPE_BINARY, TSDB_DB_NAME_LEN)
       .addColumn("stable_name", TSDB_DATA_TYPE_BINARY, TSDB_TABLE_NAME_LEN)
       .done();
-  mcs->createTableBuilder(TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_TABLES, TSDB_SYSTEM_TABLE, 2)
+  mcs->createTableBuilder(TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_TABLES, TSDB_SYSTEM_TABLE, 3)
-      .addColumn("db_name", TSDB_DATA_TYPE_BINARY, TSDB_DB_NAME_LEN)
       .addColumn("table_name", TSDB_DATA_TYPE_BINARY, TSDB_TABLE_NAME_LEN)
+      .addColumn("db_name", TSDB_DATA_TYPE_BINARY, TSDB_DB_NAME_LEN)
+      .addColumn("stable_name", TSDB_DATA_TYPE_BINARY, TSDB_TABLE_NAME_LEN)
       .done();
   mcs->createTableBuilder(TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_TABLE_DISTRIBUTED, TSDB_SYSTEM_TABLE, 2)
       .addColumn("db_name", TSDB_DATA_TYPE_BINARY, TSDB_DB_NAME_LEN)
@@ -140,7 +141,7 @@ void generatePerformanceSchema(MockCatalogService* mcs) {
 void generateTestTables(MockCatalogService* mcs, const std::string& db) {
   mcs->createTableBuilder(db, "t1", TSDB_NORMAL_TABLE, 6)
       .setPrecision(TSDB_TIME_PRECISION_MILLI)
-      .setVgid(1)
+      .setVgid(2)
       .addColumn("ts", TSDB_DATA_TYPE_TIMESTAMP)
       .addColumn("c1", TSDB_DATA_TYPE_INT)
       .addColumn("c2", TSDB_DATA_TYPE_BINARY, 20)
@@ -182,9 +183,9 @@ void generateTestStables(MockCatalogService* mcs, const std::string& db) {
                                  .addTag("tag2", TSDB_DATA_TYPE_BINARY, 20)
                                  .addTag("tag3", TSDB_DATA_TYPE_TIMESTAMP);
     builder.done();
-    mcs->createSubTable(db, "st1", "st1s1", 1);
+    mcs->createSubTable(db, "st1", "st1s1", 2);
-    mcs->createSubTable(db, "st1", "st1s2", 2);
+    mcs->createSubTable(db, "st1", "st1s2", 3);
-    mcs->createSubTable(db, "st1", "st1s3", 1);
+    mcs->createSubTable(db, "st1", "st1s3", 2);
   }
   {
     ITableBuilder& builder = mcs->createTableBuilder(db, "st2", TSDB_SUPER_TABLE, 3, 1)
@@ -194,8 +195,8 @@ void generateTestStables(MockCatalogService* mcs, const std::string& db) {
                                  .addColumn("c2", TSDB_DATA_TYPE_BINARY, 20)
                                  .addTag("jtag", TSDB_DATA_TYPE_JSON);
     builder.done();
-    mcs->createSubTable(db, "st2", "st2s1", 1);
+    mcs->createSubTable(db, "st2", "st2s1", 2);
-    mcs->createSubTable(db, "st2", "st2s2", 2);
+    mcs->createSubTable(db, "st2", "st2s2", 3);
   }
 }
 

source/libs/parser/test/mockCatalogService.cpp

@@ -20,15 +20,19 @@
 #include <map>
 #include <set>
 
+#include "systable.h"
+#include "tdatablock.h"
+#include "tmisce.h"
 #include "tname.h"
 #include "ttypes.h"
-#include "tmisce.h"
+
+using std::string;
 
 std::unique_ptr<MockCatalogService> g_mockCatalogService;
 
 class TableBuilder : public ITableBuilder {
  public:
-  virtual TableBuilder& addColumn(const std::string& name, int8_t type, int32_t bytes) {
+  virtual TableBuilder& addColumn(const string& name, int8_t type, int32_t bytes) {
     assert(colId_ <= schema()->tableInfo.numOfTags + schema()->tableInfo.numOfColumns);
     SSchema* col = schema()->schema + (colId_ - 1);
     col->type = type;
@@ -142,27 +146,16 @@ class MockCatalogServiceImpl {
   }
 
   int32_t catalogGetDBVgList(const char* pDbFName, SArray** pVgList) const {
-    std::string                 dbFName(pDbFName);
+    string dbName(string(pDbFName).substr(string(pDbFName).find_last_of('.') + 1));
-    DbMetaCache::const_iterator it = meta_.find(dbFName.substr(std::string(pDbFName).find_last_of('.') + 1));
+    if (0 == dbName.compare(TSDB_INFORMATION_SCHEMA_DB) || 0 == dbName.compare(TSDB_PERFORMANCE_SCHEMA_DB)) {
-    if (meta_.end() == it) {
+      return catalogGetAllDBVgList(pVgList);
-      return TSDB_CODE_FAILED;
     }
-    std::set<int32_t> vgSet;
+    return catalogGetDBVgListImpl(dbName, pVgList);
-    *pVgList = taosArrayInit(it->second.size(), sizeof(SVgroupInfo));
-    for (const auto& vgs : it->second) {
-      for (const auto& vg : vgs.second->vgs) {
-        if (0 == vgSet.count(vg.vgId)) {
-          taosArrayPush(*pVgList, &vg);
-          vgSet.insert(vg.vgId);
-        }
-      }
-    }
-    return TSDB_CODE_SUCCESS;
   }
 
   int32_t catalogGetDBCfg(const char* pDbFName, SDbCfgInfo* pDbCfg) const {
-    std::string                dbFName(pDbFName);
+    string                     dbFName(pDbFName);
-    DbCfgCache::const_iterator it = dbCfg_.find(dbFName.substr(std::string(pDbFName).find_last_of('.') + 1));
+    DbCfgCache::const_iterator it = dbCfg_.find(dbFName.substr(string(pDbFName).find_last_of('.') + 1));
     if (dbCfg_.end() == it) {
       return TSDB_CODE_FAILED;
     }
@@ -171,7 +164,7 @@ class MockCatalogServiceImpl {
     return TSDB_CODE_SUCCESS;
   }
 
-  int32_t catalogGetUdfInfo(const std::string& funcName, SFuncInfo* pInfo) const {
+  int32_t catalogGetUdfInfo(const string& funcName, SFuncInfo* pInfo) const {
     auto it = udf_.find(funcName);
     if (udf_.end() == it) {
       return TSDB_CODE_FAILED;
@@ -236,15 +229,15 @@ class MockCatalogServiceImpl {
     return code;
   }
 
-  TableBuilder& createTableBuilder(const std::string& db, const std::string& tbname, int8_t tableType,
+  TableBuilder& createTableBuilder(const string& db, const string& tbname, int8_t tableType, int32_t numOfColumns,
-                                   int32_t numOfColumns, int32_t numOfTags) {
+                                   int32_t numOfTags) {
     builder_ = TableBuilder::createTableBuilder(tableType, numOfColumns, numOfTags);
     meta_[db][tbname] = builder_->table();
     meta_[db][tbname]->schema->uid = getNextId();
     return *(builder_.get());
   }
 
-  void createSubTable(const std::string& db, const std::string& stbname, const std::string& tbname, int16_t vgid) {
+  void createSubTable(const string& db, const string& stbname, const string& tbname, int16_t vgid) {
     std::unique_ptr<STableMeta> table;
     if (TSDB_CODE_SUCCESS != copyTableSchemaMeta(db, stbname, &table)) {
       throw std::runtime_error("copyTableSchemaMeta failed");
@@ -274,13 +267,13 @@ class MockCatalogServiceImpl {
 // string field length
 #define SFL 20
 // string field header
-#define SH(h) CA(SFL, std::string(h))
+#define SH(h) CA(SFL, string(h))
 // string field
 #define SF(n) CA(SFL, n)
 // integer field length
 #define IFL 10
 // integer field header
-#define IH(i) CA(IFL, std::string(i))
+#define IH(i) CA(IFL, string(i))
 // integer field
 #define IF(i) CA(IFL, std::to_string(i))
 // split line
@@ -308,7 +301,7 @@ class MockCatalogServiceImpl {
         int16_t numOfFields = numOfColumns + schema->tableInfo.numOfTags;
         for (int16_t i = 0; i < numOfFields; ++i) {
           const SSchema* col = schema->schema + i;
-          std::cout << SF(std::string(col->name)) << SH(ftToString(i, numOfColumns)) << SH(dtToString(col->type))
+          std::cout << SF(string(col->name)) << SH(ftToString(i, numOfColumns)) << SH(dtToString(col->type))
                     << IF(col->bytes) << std::endl;
         }
         std::cout << std::endl;
@@ -316,7 +309,7 @@ class MockCatalogServiceImpl {
     }
   }
 
-  void createFunction(const std::string& func, int8_t funcType, int8_t outputType, int32_t outputLen, int32_t bufSize) {
+  void createFunction(const string& func, int8_t funcType, int8_t outputType, int32_t outputLen, int32_t bufSize) {
     std::shared_ptr<SFuncInfo> info(new SFuncInfo);
     strcpy(info->name, func.c_str());
     info->funcType = funcType;
@@ -342,19 +335,19 @@ class MockCatalogServiceImpl {
     info.expr = strdup(pReq->expr);
     auto it = index_.find(pReq->stb);
     if (index_.end() == it) {
-      index_.insert(std::make_pair(std::string(pReq->stb), std::vector<STableIndexInfo>{info}));
+      index_.insert(std::make_pair(string(pReq->stb), std::vector<STableIndexInfo>{info}));
     } else {
       it->second.push_back(info);
     }
   }
 
-  void createDnode(int32_t dnodeId, const std::string& host, int16_t port) {
+  void createDnode(int32_t dnodeId, const string& host, int16_t port) {
     SEpSet epSet = {0};
     addEpIntoEpSet(&epSet, host.c_str(), port);
     dnode_.insert(std::make_pair(dnodeId, epSet));
   }
 
-  void createDatabase(const std::string& db, bool rollup, int8_t cacheLast) {
+  void createDatabase(const string& db, bool rollup, int8_t cacheLast) {
     SDbCfgInfo cfg = {0};
     if (rollup) {
       cfg.pRetensions = taosArrayInit(TARRAY_MIN_SIZE, sizeof(SRetention));
@@ -364,12 +357,12 @@ class MockCatalogServiceImpl {
   }
 
  private:
-  typedef std::map<std::string, std::shared_ptr<MockTableMeta>> TableMetaCache;
+  typedef std::map<string, std::shared_ptr<MockTableMeta>> TableMetaCache;
-  typedef std::map<std::string, TableMetaCache>                 DbMetaCache;
+  typedef std::map<string, TableMetaCache>                 DbMetaCache;
-  typedef std::map<std::string, std::shared_ptr<SFuncInfo>>     UdfMetaCache;
+  typedef std::map<string, std::shared_ptr<SFuncInfo>>     UdfMetaCache;
-  typedef std::map<std::string, std::vector<STableIndexInfo>>   IndexMetaCache;
+  typedef std::map<string, std::vector<STableIndexInfo>>   IndexMetaCache;
-  typedef std::map<int32_t, SEpSet>                             DnodeCache;
+  typedef std::map<int32_t, SEpSet>                        DnodeCache;
-  typedef std::map<std::string, SDbCfgInfo>                     DbCfgCache;
+  typedef std::map<string, SDbCfgInfo>                     DbCfgCache;
 
   uint64_t getNextId() { return id_++; }
 
@@ -386,15 +379,15 @@ class MockCatalogServiceImpl {
     return pDst;
   }
 
-  std::string toDbname(const std::string& dbFullName) const {
+  string toDbname(const string& dbFullName) const {
-    std::string::size_type n = dbFullName.find(".");
+    string::size_type n = dbFullName.find(".");
-    if (n == std::string::npos) {
+    if (n == string::npos) {
       return dbFullName;
     }
     return dbFullName.substr(n + 1);
   }
 
-  std::string ttToString(int8_t tableType) const {
+  string ttToString(int8_t tableType) const {
     switch (tableType) {
       case TSDB_SUPER_TABLE:
         return "super table";
@@ -407,7 +400,7 @@ class MockCatalogServiceImpl {
     }
   }
 
-  std::string pToString(uint8_t precision) const {
+  string pToString(uint8_t precision) const {
     switch (precision) {
       case TSDB_TIME_PRECISION_MILLI:
         return "millisecond";
@@ -420,19 +413,18 @@ class MockCatalogServiceImpl {
     }
   }
 
-  std::string dtToString(int8_t type) const { return tDataTypes[type].name; }
+  string dtToString(int8_t type) const { return tDataTypes[type].name; }
 
-  std::string ftToString(int16_t colid, int16_t numOfColumns) const {
+  string ftToString(int16_t colid, int16_t numOfColumns) const {
     return (0 == colid ? "column" : (colid < numOfColumns ? "column" : "tag"));
   }
 
-  STableMeta* getTableSchemaMeta(const std::string& db, const std::string& tbname) const {
+  STableMeta* getTableSchemaMeta(const string& db, const string& tbname) const {
     std::shared_ptr<MockTableMeta> table = getTableMeta(db, tbname);
     return table ? table->schema : nullptr;
   }
 
-  int32_t copyTableSchemaMeta(const std::string& db, const std::string& tbname,
+  int32_t copyTableSchemaMeta(const string& db, const string& tbname, std::unique_ptr<STableMeta>* dst) const {
-                              std::unique_ptr<STableMeta>* dst) const {
     STableMeta* src = getTableSchemaMeta(db, tbname);
     if (nullptr == src) {
       return TSDB_CODE_TSC_INVALID_TABLE_NAME;
@@ -446,7 +438,7 @@ class MockCatalogServiceImpl {
     return TSDB_CODE_SUCCESS;
   }
 
-  int32_t copyTableVgroup(const std::string& db, const std::string& tbname, SVgroupInfo* vg) const {
+  int32_t copyTableVgroup(const string& db, const string& tbname, SVgroupInfo* vg) const {
     std::shared_ptr<MockTableMeta> table = getTableMeta(db, tbname);
     if (table->vgs.empty()) {
       return TSDB_CODE_SUCCESS;
@@ -455,7 +447,7 @@ class MockCatalogServiceImpl {
     return TSDB_CODE_SUCCESS;
   }
 
-  int32_t copyTableVgroup(const std::string& db, const std::string& tbname, SArray** vgList) const {
+  int32_t copyTableVgroup(const string& db, const string& tbname, SArray** vgList) const {
     std::shared_ptr<MockTableMeta> table = getTableMeta(db, tbname);
     if (table->vgs.empty()) {
       return TSDB_CODE_SUCCESS;
@@ -467,7 +459,7 @@ class MockCatalogServiceImpl {
     return TSDB_CODE_SUCCESS;
   }
 
-  std::shared_ptr<MockTableMeta> getTableMeta(const std::string& db, const std::string& tbname) const {
+  std::shared_ptr<MockTableMeta> getTableMeta(const string& db, const string& tbname) const {
     DbMetaCache::const_iterator it = meta_.find(db);
     if (meta_.end() == it) {
       return std::shared_ptr<MockTableMeta>();
@@ -527,6 +519,40 @@ class MockCatalogServiceImpl {
     return code;
   }
 
+  int32_t catalogGetDBVgListImpl(const string& dbName, SArray** pVgList) const {
+    DbMetaCache::const_iterator it = meta_.find(dbName);
+    if (meta_.end() == it) {
+      return TSDB_CODE_FAILED;
+    }
+    std::set<int32_t> vgSet;
+    *pVgList = taosArrayInit(it->second.size(), sizeof(SVgroupInfo));
+    for (const auto& vgs : it->second) {
+      for (const auto& vg : vgs.second->vgs) {
+        if (0 == vgSet.count(vg.vgId)) {
+          taosArrayPush(*pVgList, &vg);
+          vgSet.insert(vg.vgId);
+        }
+      }
+    }
+    return TSDB_CODE_SUCCESS;
+  }
+
+  int32_t catalogGetAllDBVgList(SArray** pVgList) const {
+    std::set<int32_t> vgSet;
+    *pVgList = taosArrayInit(TARRAY_MIN_SIZE, sizeof(SVgroupInfo));
+    for (const auto& db : meta_) {
+      for (const auto& vgs : db.second) {
+        for (const auto& vg : vgs.second->vgs) {
+          if (0 == vgSet.count(vg.vgId)) {
+            taosArrayPush(*pVgList, &vg);
+            vgSet.insert(vg.vgId);
+          }
+        }
+      }
+    }
+    return TSDB_CODE_SUCCESS;
+  }
+
   int32_t getAllDbCfg(SArray* pDbCfgReq, SArray** pDbCfgData) const {
     int32_t code = TSDB_CODE_SUCCESS;
     if (NULL != pDbCfgReq) {
@@ -634,30 +660,29 @@ MockCatalogService::MockCatalogService() : impl_(new MockCatalogServiceImpl()) {
 
 MockCatalogService::~MockCatalogService() {}
 
-ITableBuilder& MockCatalogService::createTableBuilder(const std::string& db, const std::string& tbname,
+ITableBuilder& MockCatalogService::createTableBuilder(const string& db, const string& tbname, int8_t tableType,
-                                                      int8_t tableType, int32_t numOfColumns, int32_t numOfTags) {
+                                                      int32_t numOfColumns, int32_t numOfTags) {
   return impl_->createTableBuilder(db, tbname, tableType, numOfColumns, numOfTags);
 }
 
-void MockCatalogService::createSubTable(const std::string& db, const std::string& stbname, const std::string& tbname,
+void MockCatalogService::createSubTable(const string& db, const string& stbname, const string& tbname, int16_t vgid) {
-                                        int16_t vgid) {
   impl_->createSubTable(db, stbname, tbname, vgid);
 }
 
 void MockCatalogService::showTables() const { impl_->showTables(); }
 
-void MockCatalogService::createFunction(const std::string& func, int8_t funcType, int8_t outputType, int32_t outputLen,
+void MockCatalogService::createFunction(const string& func, int8_t funcType, int8_t outputType, int32_t outputLen,
                                         int32_t bufSize) {
   impl_->createFunction(func, funcType, outputType, outputLen, bufSize);
 }
 
 void MockCatalogService::createSmaIndex(const SMCreateSmaReq* pReq) { impl_->createSmaIndex(pReq); }
 
-void MockCatalogService::createDnode(int32_t dnodeId, const std::string& host, int16_t port) {
+void MockCatalogService::createDnode(int32_t dnodeId, const string& host, int16_t port) {
   impl_->createDnode(dnodeId, host, port);
 }
 
-void MockCatalogService::createDatabase(const std::string& db, bool rollup, int8_t cacheLast) {
+void MockCatalogService::createDatabase(const string& db, bool rollup, int8_t cacheLast) {
   impl_->createDatabase(db, rollup, cacheLast);
 }
 
@@ -683,7 +708,7 @@ int32_t MockCatalogService::catalogGetDBCfg(const char* pDbFName, SDbCfgInfo* pD
   return impl_->catalogGetDBCfg(pDbFName, pDbCfg);
 }
 
-int32_t MockCatalogService::catalogGetUdfInfo(const std::string& funcName, SFuncInfo* pInfo) const {
+int32_t MockCatalogService::catalogGetUdfInfo(const string& funcName, SFuncInfo* pInfo) const {
   return impl_->catalogGetUdfInfo(funcName, pInfo);
 }
 

source/libs/planner/src/planLogicCreater.c

@@ -250,11 +250,12 @@ static EScanType getScanType(SLogicPlanContext* pCxt, SNodeList* pScanPseudoCols
   }
 
   if (NULL == pScanCols) {
-    return NULL == pScanPseudoCols
+    if (NULL == pScanPseudoCols) {
-               ? SCAN_TYPE_TABLE
+      return SCAN_TYPE_TABLE;
-               : ((FUNCTION_TYPE_BLOCK_DIST_INFO == ((SFunctionNode*)nodesListGetNode(pScanPseudoCols, 0))->funcType)
+    }
-                      ? SCAN_TYPE_BLOCK_INFO
+    return FUNCTION_TYPE_BLOCK_DIST_INFO == ((SFunctionNode*)nodesListGetNode(pScanPseudoCols, 0))->funcType
-                      : SCAN_TYPE_TABLE);
+               ? SCAN_TYPE_BLOCK_INFO
+               : SCAN_TYPE_TABLE;
   }
 
   return SCAN_TYPE_TABLE;
@@ -333,6 +334,8 @@ static int32_t makeScanLogicNode(SLogicPlanContext* pCxt, SRealTableNode* pRealT
   return TSDB_CODE_SUCCESS;
 }
 
+static bool needScanDefaultCol(EScanType scanType) { return SCAN_TYPE_TABLE_COUNT != scanType; }
+
 static int32_t createScanLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect, SRealTableNode* pRealTable,
                                    SLogicNode** pLogicNode) {
   SScanLogicNode* pScan = NULL;
@@ -367,7 +370,7 @@ static int32_t createScanLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect
     pScan->hasNormalCols = true;
   }
 
-  if (TSDB_CODE_SUCCESS == code) {
+  if (TSDB_CODE_SUCCESS == code && needScanDefaultCol(pScan->scanType)) {
     code = addDefaultScanCol(pRealTable->pMeta, &pScan->pScanCols);
   }
 

source/libs/planner/src/planOptimizer.c

@@ -16,6 +16,7 @@
 #include "filter.h"
 #include "functionMgt.h"
 #include "planInt.h"
+#include "systable.h"
 #include "tglobal.h"
 #include "ttime.h"
 
@@ -64,6 +65,7 @@ typedef enum ECondAction {
 } ECondAction;
 
 typedef bool (*FMayBeOptimized)(SLogicNode* pNode);
+typedef bool (*FShouldBeOptimized)(SLogicNode* pNode, void* pInfo);
 
 static SLogicNode* optFindPossibleNode(SLogicNode* pNode, FMayBeOptimized func) {
   if (func(pNode)) {
@@ -79,6 +81,19 @@ static SLogicNode* optFindPossibleNode(SLogicNode* pNode, FMayBeOptimized func)
   return NULL;
 }
 
+static bool optFindEligibleNode(SLogicNode* pNode, FShouldBeOptimized func, void* pInfo) {
+  if (func(pNode, pInfo)) {
+    return true;
+  }
+  SNode* pChild;
+  FOREACH(pChild, pNode->pChildren) {
+    if (optFindEligibleNode((SLogicNode*)pChild, func, pInfo)) {
+      return true;
+    }
+  }
+  return false;
+}
+
 static void optResetParent(SLogicNode* pNode) {
   SNode* pChild = NULL;
   FOREACH(pChild, pNode->pChildren) { ((SLogicNode*)pChild)->pParent = pNode; }
@@ -2454,6 +2469,243 @@ static int32_t pushDownLimitOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLog
   return TSDB_CODE_SUCCESS;
 }
 
+typedef struct STbCntScanOptInfo {
+  SAggLogicNode*  pAgg;
+  SScanLogicNode* pScan;
+  SName           table;
+} STbCntScanOptInfo;
+
+static bool tbCntScanOptIsEligibleGroupKeys(SNodeList* pGroupKeys) {
+  if (NULL == pGroupKeys) {
+    return true;
+  }
+
+  SNode* pGroupKey = NULL;
+  FOREACH(pGroupKey, pGroupKeys) {
+    SNode* pKey = nodesListGetNode(((SGroupingSetNode*)pGroupKey)->pParameterList, 0);
+    if (QUERY_NODE_COLUMN != nodeType(pKey)) {
+      return false;
+    }
+    SColumnNode* pCol = (SColumnNode*)pKey;
+    if (0 != strcmp(pCol->colName, "db_name") && 0 != strcmp(pCol->colName, "stable_name")) {
+      return false;
+    }
+  }
+
+  return true;
+}
+
+static bool tbCntScanOptNotNullableExpr(SNode* pNode) {
+  if (QUERY_NODE_COLUMN != nodeType(pNode)) {
+    return false;
+  }
+  const char* pColName = ((SColumnNode*)pNode)->colName;
+  return 0 == strcmp(pColName, "*") || 0 == strcmp(pColName, "db_name") || 0 == strcmp(pColName, "stable_name") ||
+         0 == strcmp(pColName, "table_name");
+}
+
+static bool tbCntScanOptIsEligibleAggFuncs(SNodeList* pAggFuncs) {
+  SNode* pNode = NULL;
+  FOREACH(pNode, pAggFuncs) {
+    SFunctionNode* pFunc = (SFunctionNode*)nodesListGetNode(pAggFuncs, 0);
+    if (FUNCTION_TYPE_COUNT != pFunc->funcType ||
+        !tbCntScanOptNotNullableExpr(nodesListGetNode(pFunc->pParameterList, 0))) {
+      return false;
+    }
+  }
+  return true;
+}
+
+static bool tbCntScanOptIsEligibleAgg(SAggLogicNode* pAgg) {
+  return tbCntScanOptIsEligibleGroupKeys(pAgg->pGroupKeys) && tbCntScanOptIsEligibleAggFuncs(pAgg->pAggFuncs);
+}
+
+static bool tbCntScanOptGetColValFromCond(SOperatorNode* pOper, SColumnNode** pCol, SValueNode** pVal) {
+  if (OP_TYPE_EQUAL != pOper->opType) {
+    return false;
+  }
+
+  *pCol = NULL;
+  *pVal = NULL;
+  if (QUERY_NODE_COLUMN == nodeType(pOper->pLeft)) {
+    *pCol = (SColumnNode*)pOper->pLeft;
+  } else if (QUERY_NODE_VALUE == nodeType(pOper->pLeft)) {
+    *pVal = (SValueNode*)pOper->pLeft;
+  }
+  if (QUERY_NODE_COLUMN == nodeType(pOper->pRight)) {
+    *pCol = (SColumnNode*)pOper->pRight;
+  } else if (QUERY_NODE_VALUE == nodeType(pOper->pRight)) {
+    *pVal = (SValueNode*)pOper->pRight;
+  }
+
+  return NULL != *pCol && NULL != *pVal;
+}
+
+static bool tbCntScanOptIsEligibleLogicCond(STbCntScanOptInfo* pInfo, SLogicConditionNode* pCond) {
+  if (LOGIC_COND_TYPE_AND != pCond->condType) {
+    return false;
+  }
+
+  bool         hasDbCond = false;
+  bool         hasStbCond = false;
+  SColumnNode* pCol = NULL;
+  SValueNode*  pVal = NULL;
+  SNode*       pNode = NULL;
+  FOREACH(pNode, pCond->pParameterList) {
+    if (QUERY_NODE_OPERATOR != nodeType(pNode) || !tbCntScanOptGetColValFromCond((SOperatorNode*)pNode, &pCol, &pVal)) {
+      return false;
+    }
+    if (!hasDbCond && 0 == strcmp(pCol->colName, "db_name")) {
+      hasDbCond = true;
+      strcpy(pInfo->table.dbname, pVal->literal);
+    } else if (!hasStbCond && 0 == strcmp(pCol->colName, "stable_name")) {
+      hasStbCond = true;
+      strcpy(pInfo->table.tname, pVal->literal);
+    } else {
+      return false;
+    }
+  }
+  return hasDbCond;
+}
+
+static bool tbCntScanOptIsEligibleOpCond(SOperatorNode* pCond) {
+  SColumnNode* pCol = NULL;
+  SValueNode*  pVal = NULL;
+  if (!tbCntScanOptGetColValFromCond(pCond, &pCol, &pVal)) {
+    return false;
+  }
+  return 0 == strcmp(pCol->colName, "db_name");
+}
+
+static bool tbCntScanOptIsEligibleConds(STbCntScanOptInfo* pInfo, SNode* pConditions) {
+  if (NULL == pConditions) {
+    return true;
+  }
+
+  if (QUERY_NODE_LOGIC_CONDITION == nodeType(pConditions)) {
+    return tbCntScanOptIsEligibleLogicCond(pInfo, (SLogicConditionNode*)pConditions);
+  }
+
+  if (QUERY_NODE_OPERATOR == nodeType(pConditions)) {
+    return tbCntScanOptIsEligibleOpCond((SOperatorNode*)pConditions);
+  }
+
+  return false;
+}
+
+static bool tbCntScanOptIsEligibleScan(STbCntScanOptInfo* pInfo) {
+  if (0 != strcmp(pInfo->pScan->tableName.dbname, TSDB_INFORMATION_SCHEMA_DB) ||
+      0 != strcmp(pInfo->pScan->tableName.tname, TSDB_INS_TABLE_TABLES) || NULL != pInfo->pScan->pGroupTags) {
+    return false;
+  }
+  if (1 == pInfo->pScan->pVgroupList->numOfVgroups && MNODE_HANDLE == pInfo->pScan->pVgroupList->vgroups[0].vgId) {
+    return false;
+  }
+  return tbCntScanOptIsEligibleConds(pInfo, pInfo->pScan->node.pConditions);
+}
+
+static bool tbCntScanOptShouldBeOptimized(SLogicNode* pNode, STbCntScanOptInfo* pInfo) {
+  if (QUERY_NODE_LOGIC_PLAN_AGG != nodeType(pNode) || 1 != LIST_LENGTH(pNode->pChildren) ||
+      QUERY_NODE_LOGIC_PLAN_SCAN != nodeType(nodesListGetNode(pNode->pChildren, 0))) {
+    return false;
+  }
+
+  pInfo->pAgg = (SAggLogicNode*)pNode;
+  pInfo->pScan = (SScanLogicNode*)nodesListGetNode(pNode->pChildren, 0);
+  return tbCntScanOptIsEligibleAgg(pInfo->pAgg) && tbCntScanOptIsEligibleScan(pInfo);
+}
+
+static SNode* tbCntScanOptCreateTableCountFunc() {
+  SFunctionNode* pFunc = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION);
+  if (NULL == pFunc) {
+    return NULL;
+  }
+  strcpy(pFunc->functionName, "_table_count");
+  strcpy(pFunc->node.aliasName, "_table_count");
+  if (TSDB_CODE_SUCCESS != fmGetFuncInfo(pFunc, NULL, 0)) {
+    nodesDestroyNode((SNode*)pFunc);
+    return NULL;
+  }
+  return (SNode*)pFunc;
+}
+
+static int32_t tbCntScanOptRewriteScan(STbCntScanOptInfo* pInfo) {
+  pInfo->pScan->scanType = SCAN_TYPE_TABLE_COUNT;
+  strcpy(pInfo->pScan->tableName.dbname, pInfo->table.dbname);
+  strcpy(pInfo->pScan->tableName.tname, pInfo->table.tname);
+  NODES_DESTORY_LIST(pInfo->pScan->node.pTargets);
+  NODES_DESTORY_LIST(pInfo->pScan->pScanCols);
+  NODES_DESTORY_NODE(pInfo->pScan->node.pConditions);
+  NODES_DESTORY_LIST(pInfo->pScan->pScanPseudoCols);
+  int32_t code = nodesListMakeStrictAppend(&pInfo->pScan->pScanPseudoCols, tbCntScanOptCreateTableCountFunc());
+  if (TSDB_CODE_SUCCESS == code) {
+    code = createColumnByRewriteExpr(nodesListGetNode(pInfo->pScan->pScanPseudoCols, 0), &pInfo->pScan->node.pTargets);
+  }
+  SNode* pGroupKey = NULL;
+  FOREACH(pGroupKey, pInfo->pAgg->pGroupKeys) {
+    SNode* pGroupCol = nodesListGetNode(((SGroupingSetNode*)pGroupKey)->pParameterList, 0);
+    code = nodesListMakeStrictAppend(&pInfo->pScan->pGroupTags, nodesCloneNode(pGroupCol));
+    if (TSDB_CODE_SUCCESS == code) {
+      code = nodesListMakeStrictAppend(&pInfo->pScan->pScanCols, nodesCloneNode(pGroupCol));
+    }
+    if (TSDB_CODE_SUCCESS == code) {
+      code = nodesListMakeStrictAppend(&pInfo->pScan->node.pTargets, nodesCloneNode(pGroupCol));
+    }
+    if (TSDB_CODE_SUCCESS != code) {
+      break;
+    }
+  }
+  return code;
+}
+
+static int32_t tbCntScanOptCreateSumFunc(SFunctionNode* pCntFunc, SNode* pParam, SNode** pOutput) {
+  SFunctionNode* pFunc = (SFunctionNode*)nodesMakeNode(QUERY_NODE_FUNCTION);
+  if (NULL == pFunc) {
+    return TSDB_CODE_OUT_OF_MEMORY;
+  }
+  strcpy(pFunc->functionName, "sum");
+  strcpy(pFunc->node.aliasName, pCntFunc->node.aliasName);
+  int32_t code = createColumnByRewriteExpr(pParam, &pFunc->pParameterList);
+  if (TSDB_CODE_SUCCESS == code) {
+    code = fmGetFuncInfo(pFunc, NULL, 0);
+  }
+  if (TSDB_CODE_SUCCESS == code) {
+    *pOutput = (SNode*)pFunc;
+  } else {
+    nodesDestroyNode((SNode*)pFunc);
+  }
+  return code;
+}
+
+static int32_t tbCntScanOptRewriteAgg(SAggLogicNode* pAgg) {
+  SScanLogicNode* pScan = (SScanLogicNode*)nodesListGetNode(pAgg->node.pChildren, 0);
+  SNode*          pSum = NULL;
+  int32_t         code = tbCntScanOptCreateSumFunc((SFunctionNode*)nodesListGetNode(pAgg->pAggFuncs, 0),
+                                                   nodesListGetNode(pScan->pScanPseudoCols, 0), &pSum);
+  if (TSDB_CODE_SUCCESS == code) {
+    NODES_DESTORY_LIST(pAgg->pAggFuncs);
+    code = nodesListMakeStrictAppend(&pAgg->pAggFuncs, pSum);
+  }
+  if (TSDB_CODE_SUCCESS == code) {
+    code = partTagsRewriteGroupTagsToFuncs(pScan->pGroupTags, 0, pAgg);
+  }
+  NODES_DESTORY_LIST(pAgg->pGroupKeys);
+  return code;
+}
+
+static int32_t tableCountScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogicSubplan) {
+  STbCntScanOptInfo info = {0};
+  if (!optFindEligibleNode(pLogicSubplan->pNode, (FShouldBeOptimized)tbCntScanOptShouldBeOptimized, &info)) {
+    return TSDB_CODE_SUCCESS;
+  }
+
+  int32_t code = tbCntScanOptRewriteScan(&info);
+  if (TSDB_CODE_SUCCESS == code) {
+    code = tbCntScanOptRewriteAgg(info.pAgg);
+  }
+  return code;
+}
+
 // clang-format off
 static const SOptimizeRule optimizeRuleSet[] = {
   {.pName = "ScanPath",                   .optimizeFunc = scanPathOptimize},
@@ -2468,7 +2720,8 @@ static const SOptimizeRule optimizeRuleSet[] = {
   {.pName = "RewriteUnique",              .optimizeFunc = rewriteUniqueOptimize},
   {.pName = "LastRowScan",                .optimizeFunc = lastRowScanOptimize},
   {.pName = "TagScan",                    .optimizeFunc = tagScanOptimize},
-  {.pName = "PushDownLimit",              .optimizeFunc = pushDownLimitOptimize}
+  {.pName = "PushDownLimit",              .optimizeFunc = pushDownLimitOptimize},
+  {.pName = "TableCountScan",             .optimizeFunc = tableCountScanOptimize},
 };
 // clang-format on
 

source/libs/planner/src/planPhysiCreater.c

@@ -489,6 +489,8 @@ static ENodeType getScanOperatorType(EScanType scanType) {
       return QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN;
     case SCAN_TYPE_BLOCK_INFO:
       return QUERY_NODE_PHYSICAL_PLAN_BLOCK_DIST_SCAN;
+    case SCAN_TYPE_TABLE_COUNT:
+      return QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN;
     default:
       break;
   }
@@ -524,6 +526,24 @@ static int32_t createLastRowScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSu
   pScan->ignoreNull = pScanLogicNode->igLastNull;
   vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode);
 
+  return createScanPhysiNodeFinalize(pCxt, pSubplan, pScanLogicNode, (SScanPhysiNode*)pScan, pPhyNode);
+}
+
+static int32_t createTableCountScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubplan,
+                                             SScanLogicNode* pScanLogicNode, SPhysiNode** pPhyNode) {
+  STableCountScanPhysiNode* pScan = (STableCountScanPhysiNode*)makePhysiNode(pCxt, (SLogicNode*)pScanLogicNode,
+                                                                             QUERY_NODE_PHYSICAL_PLAN_TABLE_COUNT_SCAN);
+  if (NULL == pScan) {
+    return TSDB_CODE_OUT_OF_MEMORY;
+  }
+
+  pScan->pGroupTags = nodesCloneList(pScanLogicNode->pGroupTags);
+  if (NULL != pScanLogicNode->pGroupTags && NULL == pScan->pGroupTags) {
+    nodesDestroyNode((SNode*)pScan);
+    return TSDB_CODE_OUT_OF_MEMORY;
+  }
+
+  pScan->groupSort = pScanLogicNode->groupSort;
   vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode);
 
   return createScanPhysiNodeFinalize(pCxt, pSubplan, pScanLogicNode, (SScanPhysiNode*)pScan, pPhyNode);
@@ -589,7 +609,6 @@ static int32_t createSystemTableScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan*
   pScan->accountId = pCxt->pPlanCxt->acctId;
   pScan->sysInfo = pCxt->pPlanCxt->sysInfo;
   if (0 == strcmp(pScanLogicNode->tableName.tname, TSDB_INS_TABLE_TABLES) ||
-      0 == strcmp(pScanLogicNode->tableName.tname, TSDB_INS_TABLE_TABLE_DISTRIBUTED) ||
       0 == strcmp(pScanLogicNode->tableName.tname, TSDB_INS_TABLE_TAGS)) {
     vgroupInfoToNodeAddr(pScanLogicNode->pVgroupList->vgroups, &pSubplan->execNode);
   } else {
@@ -624,6 +643,8 @@ static int32_t createScanPhysiNode(SPhysiPlanContext* pCxt, SSubplan* pSubplan,
     case SCAN_TYPE_TAG:
     case SCAN_TYPE_BLOCK_INFO:
       return createSimpleScanPhysiNode(pCxt, pSubplan, pScanLogicNode, pPhyNode);
+    case SCAN_TYPE_TABLE_COUNT:
+      return createTableCountScanPhysiNode(pCxt, pSubplan, pScanLogicNode, pPhyNode);
     case SCAN_TYPE_LAST_ROW:
       return createLastRowScanPhysiNode(pCxt, pSubplan, pScanLogicNode, pPhyNode);
     case SCAN_TYPE_TABLE:

source/libs/planner/src/planSpliter.c

@@ -292,6 +292,20 @@ static bool stbSplNeedSplitJoin(bool streamQuery, SJoinLogicNode* pJoin) {
   return true;
 }
 
+static bool stbSplIsTableCountQuery(SLogicNode* pNode) {
+  if (1 != LIST_LENGTH(pNode->pChildren)) {
+    return false;
+  }
+  SNode* pChild = nodesListGetNode(pNode->pChildren, 0);
+  if (QUERY_NODE_LOGIC_PLAN_PARTITION == nodeType(pChild)) {
+    if (1 != LIST_LENGTH(((SLogicNode*)pChild)->pChildren)) {
+      return false;
+    }
+    pChild = nodesListGetNode(((SLogicNode*)pChild)->pChildren, 0);
+  }
+  return QUERY_NODE_LOGIC_PLAN_SCAN == nodeType(pChild) && SCAN_TYPE_TABLE_COUNT == ((SScanLogicNode*)pChild)->scanType;
+}
+
 static SNodeList* stbSplGetPartKeys(SLogicNode* pNode) {
   if (QUERY_NODE_LOGIC_PLAN_SCAN == nodeType(pNode)) {
     return ((SScanLogicNode*)pNode)->pGroupTags;
@@ -340,7 +354,7 @@ static bool stbSplNeedSplit(bool streamQuery, SLogicNode* pNode) {
     case QUERY_NODE_LOGIC_PLAN_AGG:
       return (!stbSplHasGatherExecFunc(((SAggLogicNode*)pNode)->pAggFuncs) ||
               stbSplIsPartTableAgg((SAggLogicNode*)pNode)) &&
-             stbSplHasMultiTbScan(streamQuery, pNode);
+             stbSplHasMultiTbScan(streamQuery, pNode) && !stbSplIsTableCountQuery(pNode);
     case QUERY_NODE_LOGIC_PLAN_WINDOW:
       return stbSplNeedSplitWindow(streamQuery, pNode);
     case QUERY_NODE_LOGIC_PLAN_SORT:

source/libs/planner/test/planSysTbTest.cpp

@@ -20,13 +20,6 @@ using namespace std;
 
 class PlanSysTableTest : public PlannerTestBase {};
 
-TEST_F(PlanSysTableTest, show) {
-  useDb("root", "test");
-
-  run("show tables");
-  run("show stables");
-}
-
 TEST_F(PlanSysTableTest, informationSchema) {
   useDb("root", "information_schema");
 
@@ -38,3 +31,17 @@ TEST_F(PlanSysTableTest, withAgg) {
 
   run("SELECT COUNT(1) FROM ins_users");
 }
+
+TEST_F(PlanSysTableTest, tableCount) {
+  useDb("root", "information_schema");
+
+  run("SELECT COUNT(*) FROM ins_tables");
+
+  run("SELECT COUNT(*) FROM ins_tables WHERE db_name = 'test'");
+
+  run("SELECT COUNT(*) FROM ins_tables WHERE db_name = 'test' AND stable_name = 'st1'");
+
+  run("SELECT db_name, COUNT(*) FROM ins_tables GROUP BY db_name");
+
+  run("SELECT db_name, stable_name, COUNT(*) FROM ins_tables GROUP BY db_name, stable_name");
+}

source/libs/qworker/inc/qwInt.h

@@ -363,7 +363,7 @@ int32_t qwAcquireTaskCtx(QW_FPARAMS_DEF, SQWTaskCtx **ctx);
 int32_t qwGetTaskCtx(QW_FPARAMS_DEF, SQWTaskCtx **ctx);
 int32_t qwAddAcquireTaskCtx(QW_FPARAMS_DEF, SQWTaskCtx **ctx);
 void    qwReleaseTaskCtx(SQWorker *mgmt, void *ctx);
-int32_t qwKillTaskHandle(SQWTaskCtx *ctx);
+int32_t qwKillTaskHandle(SQWTaskCtx *ctx, int32_t rspCode);
 int32_t qwUpdateTaskStatus(QW_FPARAMS_DEF, int8_t status);
 int32_t qwDropTask(QW_FPARAMS_DEF);
 void    qwSaveTbVersionInfo(qTaskInfo_t pTaskInfo, SQWTaskCtx *ctx);

source/libs/qworker/src/qwMsg.c

@@ -336,7 +336,7 @@ int32_t qwRegisterHbBrokenLinkArg(SQWorker *mgmt, uint64_t sId, SRpcHandleInfo *
   }
   if (tSerializeSSchedulerHbReq(msg, msgSize, &req) < 0) {
     QW_SCH_ELOG("tSerializeSSchedulerHbReq hbReq failed, size:%d", msgSize);
-    taosMemoryFree(msg);
+    rpcFreeCont(msg);
     QW_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
   }
 

source/libs/qworker/src/qwUtil.c

@@ -279,14 +279,14 @@ void qwFreeTaskHandle(qTaskInfo_t *taskHandle) {
   }
 }
 
-int32_t qwKillTaskHandle(SQWTaskCtx *ctx) {
+int32_t qwKillTaskHandle(SQWTaskCtx *ctx, int32_t rspCode) {
   int32_t code = 0;
   
   // Note: free/kill may in RC
   qTaskInfo_t taskHandle = atomic_load_ptr(&ctx->taskHandle);
   if (taskHandle && atomic_val_compare_exchange_ptr(&ctx->taskHandle, taskHandle, NULL)) {
     qDebug("start to kill task");
-    code = qAsyncKillTask(taskHandle);
+    code = qAsyncKillTask(taskHandle, rspCode);
     atomic_store_ptr(&ctx->taskHandle, taskHandle);
   }
 

source/libs/qworker/src/qworker.c

@@ -411,7 +411,7 @@ int32_t qwHandlePrePhaseEvents(QW_FPARAMS_DEF, int8_t phase, SQWPhaseInput *inpu
         // qwBuildAndSendDropRsp(&ctx->ctrlConnInfo, code);
         // QW_TASK_DLOG("drop rsp send, handle:%p, code:%x - %s", ctx->ctrlConnInfo.handle, code, tstrerror(code));
 
-        QW_ERR_JRET(TSDB_CODE_QRY_TASK_DROPPED);
+        QW_ERR_JRET(ctx->rspCode);
       }
 
       QW_ERR_JRET(qwUpdateTaskStatus(QW_FPARAMS(), JOB_TASK_STATUS_EXEC));
@@ -420,7 +420,7 @@ int32_t qwHandlePrePhaseEvents(QW_FPARAMS_DEF, int8_t phase, SQWPhaseInput *inpu
     case QW_PHASE_PRE_FETCH: {
       if (QW_EVENT_PROCESSED(ctx, QW_EVENT_DROP) || QW_EVENT_RECEIVED(ctx, QW_EVENT_DROP)) {
         QW_TASK_WLOG("task dropping or already dropped, phase:%s", qwPhaseStr(phase));
-        QW_ERR_JRET(TSDB_CODE_QRY_TASK_DROPPED);
+        QW_ERR_JRET(ctx->rspCode);
       }
 
       if (QW_EVENT_RECEIVED(ctx, QW_EVENT_FETCH)) {
@@ -442,7 +442,7 @@ int32_t qwHandlePrePhaseEvents(QW_FPARAMS_DEF, int8_t phase, SQWPhaseInput *inpu
     case QW_PHASE_PRE_CQUERY: {
       if (QW_EVENT_PROCESSED(ctx, QW_EVENT_DROP)) {
         QW_TASK_WLOG("task already dropped, phase:%s", qwPhaseStr(phase));
-        QW_ERR_JRET(TSDB_CODE_QRY_TASK_DROPPED);
+        QW_ERR_JRET(ctx->rspCode);
       }
 
       if (ctx->rspCode) {
@@ -456,7 +456,7 @@ int32_t qwHandlePrePhaseEvents(QW_FPARAMS_DEF, int8_t phase, SQWPhaseInput *inpu
         // qwBuildAndSendDropRsp(&ctx->ctrlConnInfo, code);
         // QW_TASK_DLOG("drop rsp send, handle:%p, code:%x - %s", ctx->ctrlConnInfo.handle, code, tstrerror(code));
 
-        QW_ERR_JRET(TSDB_CODE_QRY_TASK_DROPPED);
+        QW_ERR_JRET(ctx->rspCode);
       }
 
       break;
@@ -502,7 +502,7 @@ int32_t qwHandlePostPhaseEvents(QW_FPARAMS_DEF, int8_t phase, SQWPhaseInput *inp
 
   if (QW_EVENT_PROCESSED(ctx, QW_EVENT_DROP)) {
     QW_TASK_WLOG("task already dropped, phase:%s", qwPhaseStr(phase));
-    QW_ERR_JRET(TSDB_CODE_QRY_TASK_DROPPED);
+    QW_ERR_JRET(ctx->rspCode);
   }
 
   if (QW_EVENT_RECEIVED(ctx, QW_EVENT_DROP)) {
@@ -515,7 +515,7 @@ int32_t qwHandlePostPhaseEvents(QW_FPARAMS_DEF, int8_t phase, SQWPhaseInput *inp
     // QW_TASK_DLOG("drop rsp send, handle:%p, code:%x - %s", ctx->ctrlConnInfo.handle, code, tstrerror(code));
 
     QW_ERR_JRET(qwDropTask(QW_FPARAMS()));
-    QW_ERR_JRET(TSDB_CODE_QRY_TASK_DROPPED);
+    QW_ERR_JRET(ctx->rspCode);
   }
 
   if (ctx->rspCode) {
@@ -861,7 +861,7 @@ int32_t qwProcessDrop(QW_FPARAMS_DEF, SQWMsg *qwMsg) {
   }
 
   if (QW_QUERY_RUNNING(ctx)) {
-    QW_ERR_JRET(qwKillTaskHandle(ctx));
+    QW_ERR_JRET(qwKillTaskHandle(ctx, TSDB_CODE_TSC_QUERY_CANCELLED));
     qwUpdateTaskStatus(QW_FPARAMS(), JOB_TASK_STATUS_DROP);
   } else {
     QW_ERR_JRET(qwDropTask(QW_FPARAMS()));
@@ -869,6 +869,7 @@ int32_t qwProcessDrop(QW_FPARAMS_DEF, SQWMsg *qwMsg) {
   }
 
   if (!dropped) {
+    QW_UPDATE_RSP_CODE(ctx, TSDB_CODE_TSC_QUERY_CANCELLED);
     QW_SET_EVENT_RECEIVED(ctx, QW_EVENT_DROP);
   }
 
@@ -1195,8 +1196,9 @@ void qWorkerStopAllTasks(void *qWorkerMgmt) {
     }
     
     if (QW_QUERY_RUNNING(ctx)) {
-      qwKillTaskHandle(ctx);
+      qwKillTaskHandle(ctx, TSDB_CODE_VND_STOPPED);
     } else if (!QW_EVENT_PROCESSED(ctx, QW_EVENT_DROP)) {
+      QW_UPDATE_RSP_CODE(ctx, TSDB_CODE_VND_STOPPED);
       QW_SET_EVENT_RECEIVED(ctx, QW_EVENT_DROP);
     }
 

source/libs/qworker/test/qworkerTests.cpp

@@ -302,7 +302,7 @@ int32_t qwtExecTask(qTaskInfo_t tinfo, SSDataBlock **pRes, uint64_t *useconds) {
   return 0;
 }
 
-int32_t qwtKillTask(qTaskInfo_t qinfo) { return 0; }
+int32_t qwtKillTask(qTaskInfo_t qinfo, int32_t rspCode) { return 0; }
 
 void qwtDestroyTask(qTaskInfo_t qHandle) {}
 

source/libs/scalar/inc/filterInt.h

@@ -101,6 +101,11 @@ typedef int32_t (*filter_desc_compare_func)(const void *, const void *);
 typedef bool (*filter_exec_func)(void *, int32_t, SColumnInfoData *, SColumnDataAgg *, int16_t, int32_t *);
 typedef int32_t (*filer_get_col_from_name)(void *, int32_t, char *, void **);
 
+typedef struct SFilterDataInfo {
+  int32_t idx;
+  void*   addr;
+} SFilterDataInfo;
+
 typedef struct SFilterRangeCompare {
   int64_t       s;
   int64_t       e;
@@ -294,9 +299,9 @@ struct SFilterInfo {
 #define CHK_OR_OPTR(ctx)  ((ctx)->isnull == true && (ctx)->notnull == true)
 #define CHK_AND_OPTR(ctx) ((ctx)->isnull == true && (((ctx)->notnull == true) || ((ctx)->isrange == true)))
 
-#define FILTER_GET_FLAG(st, f) (st & f)
+#define FILTER_GET_FLAG(st, f) ((st) & (f))
-#define FILTER_SET_FLAG(st, f) st |= (f)
+#define FILTER_SET_FLAG(st, f) (st) |= (f)
-#define FILTER_CLR_FLAG(st, f) st &= (~f)
+#define FILTER_CLR_FLAG(st, f) (st) &= (~f)
 
 #define SIMPLE_COPY_VALUES(dst, src) *((int64_t *)dst) = *((int64_t *)src)
 #define FLT_PACKAGE_UNIT_HASH_KEY(v, op1, op2, lidx, ridx, ridx2) \

source/libs/scalar/src/filter.c

@@ -963,16 +963,17 @@ int32_t filterGetFiledByDesc(SFilterFields *fields, int32_t type, void *v) {
   return -1;
 }
 
-int32_t filterGetFiledByData(SFilterInfo *info, int32_t type, void *v, int32_t dataLen) {
+int32_t filterGetFiledByData(SFilterInfo *info, int32_t type, void *v, int32_t dataLen, bool *sameBuf) {
   if (type == FLD_TYPE_VALUE) {
     if (info->pctx.valHash == false) {
       qError("value hash is empty");
       return -1;
     }
 
-    void *hv = taosHashGet(info->pctx.valHash, v, dataLen);
+    SFilterDataInfo *dInfo = taosHashGet(info->pctx.valHash, v, dataLen);
-    if (hv) {
+    if (dInfo) {
-      return *(int32_t *)hv;
+      *sameBuf = (dInfo->addr == v);
+      return dInfo->idx;
     }
   }
 
@@ -982,9 +983,10 @@ int32_t filterGetFiledByData(SFilterInfo *info, int32_t type, void *v, int32_t d
 // In the params, we should use void *data instead of void **data, there is no need to use taosMemoryFreeClear(*data) to
 // set *data = 0 Besides, fields data value is a pointer, so dataLen should be POINTER_BYTES for better.
 int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type, SFilterFieldId *fid, int32_t dataLen,
-                       bool freeIfExists) {
+                       bool freeIfExists, int16_t *srcFlag) {
   int32_t   idx = -1;
   uint32_t *num;
+  bool sameBuf = false;
 
   num = &info->fields[type].num;
 
@@ -992,7 +994,7 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
     if (type == FLD_TYPE_COLUMN) {
       idx = filterGetFiledByDesc(&info->fields[type], type, desc);
     } else if (data && (*data) && dataLen > 0 && FILTER_GET_FLAG(info->options, FLT_OPTION_NEED_UNIQE)) {
-      idx = filterGetFiledByData(info, type, *data, dataLen);
+      idx = filterGetFiledByData(info, type, *data, dataLen, &sameBuf);
     }
   }
 
@@ -1020,11 +1022,17 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
                                           taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, false);
       }
 
-      taosHashPut(info->pctx.valHash, *data, dataLen, &idx, sizeof(idx));
+      SFilterDataInfo dInfo = {idx, *data};
+      taosHashPut(info->pctx.valHash, *data, dataLen, &dInfo, sizeof(dInfo));
+      if (srcFlag) {
+        FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE);
+      }
     }
-  } else {
+  } else if (type != FLD_TYPE_COLUMN && data) {
-    if (data && freeIfExists) {
+    if (freeIfExists) {
       taosMemoryFreeClear(*data);
+    } else if (sameBuf) {
+      FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE);
     }
   }
 
@@ -1035,7 +1043,7 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
 }
 
 static FORCE_INLINE int32_t filterAddColFieldFromField(SFilterInfo *info, SFilterField *field, SFilterFieldId *fid) {
-  filterAddField(info, field->desc, &field->data, FILTER_GET_TYPE(field->flag), fid, 0, false);
+  filterAddField(info, field->desc, &field->data, FILTER_GET_TYPE(field->flag), fid, 0, false, NULL);
 
   FILTER_SET_FLAG(field->flag, FLD_DATA_NO_FREE);
 
@@ -1064,7 +1072,7 @@ int32_t filterAddFieldFromNode(SFilterInfo *info, SNode *node, SFilterFieldId *f
     v = node;
   }
 
-  filterAddField(info, v, NULL, type, fid, 0, true);
+  filterAddField(info, v, NULL, type, fid, 0, true, NULL);
 
   return TSDB_CODE_SUCCESS;
 }
@@ -1195,7 +1203,7 @@ int32_t fltAddGroupUnitFromNode(SFilterInfo *info, SNode *tree, SArray *group) {
 
         len = tDataTypes[type].bytes;
 
-        filterAddField(info, NULL, (void **)&out.columnData->pData, FLD_TYPE_VALUE, &right, len, true);
+        filterAddField(info, NULL, (void **)&out.columnData->pData, FLD_TYPE_VALUE, &right, len, true, NULL);
         out.columnData->pData = NULL;
       } else {
         void *data = taosMemoryCalloc(1, tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes);  // reserved space for simple_copy
@@ -1203,7 +1211,7 @@ int32_t fltAddGroupUnitFromNode(SFilterInfo *info, SNode *tree, SArray *group) {
           FLT_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
         }
         memcpy(data, nodesGetValueFromNode(valueNode), tDataTypes[type].bytes);
-        filterAddField(info, NULL, (void **)&data, FLD_TYPE_VALUE, &right, len, true);
+        filterAddField(info, NULL, (void **)&data, FLD_TYPE_VALUE, &right, len, true, NULL);
       }
       filterAddUnit(info, OP_TYPE_EQUAL, &left, &right, &uidx);
 
@@ -1234,28 +1242,26 @@ int32_t filterAddUnitFromUnit(SFilterInfo *dst, SFilterInfo *src, SFilterUnit *u
   uint8_t        type = FILTER_UNIT_DATA_TYPE(u);
   uint16_t       flag = 0;
 
-  filterAddField(dst, FILTER_UNIT_COL_DESC(src, u), NULL, FLD_TYPE_COLUMN, &left, 0, false);
+  filterAddField(dst, FILTER_UNIT_COL_DESC(src, u), NULL, FLD_TYPE_COLUMN, &left, 0, false, NULL);
   SFilterField *t = FILTER_UNIT_LEFT_FIELD(src, u);
 
   if (u->right.type == FLD_TYPE_VALUE) {
     void *data = FILTER_UNIT_VAL_DATA(src, u);
+    SFilterField *rField = FILTER_UNIT_RIGHT_FIELD(src, u);
+
     if (IS_VAR_DATA_TYPE(type)) {
       if (FILTER_UNIT_OPTR(u) == OP_TYPE_IN) {
         filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, POINTER_BYTES,
-                       false);  // POINTER_BYTES should be sizeof(SHashObj), but POINTER_BYTES is also right.
+                       false, &rField->flag);  // POINTER_BYTES should be sizeof(SHashObj), but POINTER_BYTES is also right.
 
         t = FILTER_GET_FIELD(dst, right);
         FILTER_SET_FLAG(t->flag, FLD_DATA_IS_HASH);
       } else {
-        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, varDataTLen(data), false);
+        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, varDataTLen(data), false, &rField->flag);
       }
     } else {
-      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, false);
+      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, false, &rField->flag);
     }
-
-    flag = FLD_DATA_NO_FREE;
-    t = FILTER_UNIT_RIGHT_FIELD(src, u);
-    FILTER_SET_FLAG(t->flag, flag);
   } else {
     pright = NULL;
   }
@@ -1313,17 +1319,17 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
       if (func(&ra->s, &ra->e) == 0) {
         void *data = taosMemoryMalloc(sizeof(int64_t));
         SIMPLE_COPY_VALUES(data, &ra->s);
-        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
         filterAddUnit(dst, OP_TYPE_EQUAL, &left, &right, &uidx);
         filterAddUnitToGroup(g, uidx);
         return TSDB_CODE_SUCCESS;
       } else {
         void *data = taosMemoryMalloc(sizeof(int64_t));
         SIMPLE_COPY_VALUES(data, &ra->s);
-        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
         void *data2 = taosMemoryMalloc(sizeof(int64_t));
         SIMPLE_COPY_VALUES(data2, &ra->e);
-        filterAddField(dst, NULL, &data2, FLD_TYPE_VALUE, &right2, tDataTypes[type].bytes, true);
+        filterAddField(dst, NULL, &data2, FLD_TYPE_VALUE, &right2, tDataTypes[type].bytes, true, NULL);
 
         filterAddUnitImpl(
             dst, FILTER_GET_FLAG(ra->sflag, RANGE_FLG_EXCLUDE) ? OP_TYPE_GREATER_THAN : OP_TYPE_GREATER_EQUAL, &left,
@@ -1337,7 +1343,7 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
     if (!FILTER_GET_FLAG(ra->sflag, RANGE_FLG_NULL)) {
       void *data = taosMemoryMalloc(sizeof(int64_t));
       SIMPLE_COPY_VALUES(data, &ra->s);
-      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
       filterAddUnit(dst, FILTER_GET_FLAG(ra->sflag, RANGE_FLG_EXCLUDE) ? OP_TYPE_GREATER_THAN : OP_TYPE_GREATER_EQUAL,
                     &left, &right, &uidx);
       filterAddUnitToGroup(g, uidx);
@@ -1346,7 +1352,7 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
     if (!FILTER_GET_FLAG(ra->eflag, RANGE_FLG_NULL)) {
       void *data = taosMemoryMalloc(sizeof(int64_t));
       SIMPLE_COPY_VALUES(data, &ra->e);
-      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
       filterAddUnit(dst, FILTER_GET_FLAG(ra->eflag, RANGE_FLG_EXCLUDE) ? OP_TYPE_LOWER_THAN : OP_TYPE_LOWER_EQUAL,
                     &left, &right, &uidx);
       filterAddUnitToGroup(g, uidx);
@@ -1393,16 +1399,16 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
       if (func(&r->ra.s, &r->ra.e) == 0) {
         void *data = taosMemoryMalloc(sizeof(int64_t));
         SIMPLE_COPY_VALUES(data, &r->ra.s);
-        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
         filterAddUnit(dst, OP_TYPE_EQUAL, &left, &right, &uidx);
         filterAddUnitToGroup(g, uidx);
       } else {
         void *data = taosMemoryMalloc(sizeof(int64_t));
         SIMPLE_COPY_VALUES(data, &r->ra.s);
-        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+        filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
         void *data2 = taosMemoryMalloc(sizeof(int64_t));
         SIMPLE_COPY_VALUES(data2, &r->ra.e);
-        filterAddField(dst, NULL, &data2, FLD_TYPE_VALUE, &right2, tDataTypes[type].bytes, true);
+        filterAddField(dst, NULL, &data2, FLD_TYPE_VALUE, &right2, tDataTypes[type].bytes, true, NULL);
 
         filterAddUnitImpl(
             dst, FILTER_GET_FLAG(r->ra.sflag, RANGE_FLG_EXCLUDE) ? OP_TYPE_GREATER_THAN : OP_TYPE_GREATER_EQUAL, &left,
@@ -1421,7 +1427,7 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
     if (!FILTER_GET_FLAG(r->ra.sflag, RANGE_FLG_NULL)) {
       void *data = taosMemoryMalloc(sizeof(int64_t));
       SIMPLE_COPY_VALUES(data, &r->ra.s);
-      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
       filterAddUnit(dst, FILTER_GET_FLAG(r->ra.sflag, RANGE_FLG_EXCLUDE) ? OP_TYPE_GREATER_THAN : OP_TYPE_GREATER_EQUAL,
                     &left, &right, &uidx);
       filterAddUnitToGroup(g, uidx);
@@ -1430,7 +1436,7 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan
     if (!FILTER_GET_FLAG(r->ra.eflag, RANGE_FLG_NULL)) {
       void *data = taosMemoryMalloc(sizeof(int64_t));
       SIMPLE_COPY_VALUES(data, &r->ra.e);
-      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true);
+      filterAddField(dst, NULL, &data, FLD_TYPE_VALUE, &right, tDataTypes[type].bytes, true, NULL);
       filterAddUnit(dst, FILTER_GET_FLAG(r->ra.eflag, RANGE_FLG_EXCLUDE) ? OP_TYPE_LOWER_THAN : OP_TYPE_LOWER_EQUAL,
                     &left, &right, &uidx);
       filterAddUnitToGroup(g, uidx);

source/libs/scheduler/src/schTask.c

@@ -391,6 +391,8 @@ int32_t schChkUpdateRedirectCtx(SSchJob *pJob, SSchTask *pTask, SEpSet *pEpSet)
     int64_t leftTime = tsMaxRetryWaitTime - lastTime;
     pTask->delayExecMs = leftTime < pCtx->periodMs ? leftTime : pCtx->periodMs;
 
+    pCtx->roundTimes = 0;
+
     goto _return;
   }
 
@@ -407,7 +409,7 @@ _return:
 int32_t schDoTaskRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf *pData, int32_t rspCode) {
   int32_t code = 0;
 
-  SCH_TASK_DLOG("task will be redirected now, status:%s", SCH_GET_TASK_STATUS_STR(pTask));
+  SCH_TASK_DLOG("task will be redirected now, status:%s, code:%s", SCH_GET_TASK_STATUS_STR(pTask), tstrerror(rspCode));
 
   if (NULL == pData) {
     pTask->retryTimes = 0;
@@ -430,15 +432,15 @@ int32_t schDoTaskRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf *pData, int32
   if (SCH_IS_DATA_BIND_TASK(pTask)) {
     if (pData && pData->pEpSet) {
       SCH_ERR_JRET(schUpdateTaskCandidateAddr(pJob, pTask, pData->pEpSet));
-    } else if (SYNC_UNKNOWN_LEADER_REDIRECT_ERROR(rspCode)) {
+    } else if (SYNC_SELF_LEADER_REDIRECT_ERROR(rspCode)) {
+      SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx);
+      SEp            *pEp = &addr->epSet.eps[addr->epSet.inUse];
+      SCH_TASK_DLOG("task retry node %d current ep, idx:%d/%d,%s:%d, code:%s", addr->nodeId, addr->epSet.inUse,
+                    addr->epSet.numOfEps, pEp->fqdn, pEp->port, tstrerror(rspCode));
+    } else {
       SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx);
       SCH_SWITCH_EPSET(addr);
       SCH_TASK_DLOG("switch task target node %d epset to %d/%d", addr->nodeId, addr->epSet.inUse, addr->epSet.numOfEps);
-    } else {
-      SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx);
-      SEp            *pEp = &addr->epSet.eps[addr->epSet.inUse];
-      SCH_TASK_DLOG("task retry node %d current ep, idx:%d/%d,%s:%d", addr->nodeId, addr->epSet.inUse,
-                    addr->epSet.numOfEps, pEp->fqdn, pEp->port);
     }
 
     if (SCH_TASK_NEED_FLOW_CTRL(pJob, pTask)) {

source/libs/stream/inc/streamInc.h

@@ -17,7 +17,6 @@
 #define _STREAM_INC_H_
 
 #include "executor.h"
-#include "tref.h"
 #include "tstream.h"
 
 #ifdef __cplusplus
@@ -25,9 +24,8 @@ extern "C" {
 #endif
 
 typedef struct {
-  int8_t  inited;
+  int8_t inited;
-  int32_t refPool;
+  void*  timer;
-  void*   timer;
 } SStreamGlobalEnv;
 
 static SStreamGlobalEnv streamEnv;

source/libs/stream/src/streamCheckpoint.c

@@ -0,0 +1,194 @@
+/*
+ * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
+ *
+ * This program is free software: you can use, redistribute, and/or modify
+ * it under the terms of the GNU Affero General Public License, version 3
+ * or later ("AGPL"), as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but WITHOUT
+ * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
+ * FITNESS FOR A PARTICULAR PURPOSE.
+ *
+ * You should have received a copy of the GNU Affero General Public License
+ * along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include "streamInc.h"
+
+int32_t tEncodeSStreamCheckpointSourceReq(SEncoder* pEncoder, const SStreamCheckpointSourceReq* pReq) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->checkpointId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->taskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->nodeId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->expireTime) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeSStreamCheckpointSourceReq(SDecoder* pDecoder, SStreamCheckpointSourceReq* pReq) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->checkpointId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->taskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->nodeId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->expireTime) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
+int32_t tEncodeSStreamCheckpointSourceRsp(SEncoder* pEncoder, const SStreamCheckpointSourceRsp* pRsp) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->streamId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->checkpointId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->taskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->nodeId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->expireTime) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeSStreamCheckpointSourceRsp(SDecoder* pDecoder, SStreamCheckpointSourceRsp* pRsp) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->streamId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->checkpointId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->taskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->nodeId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->expireTime) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
+int32_t tEncodeSStreamCheckpointReq(SEncoder* pEncoder, const SStreamCheckpointReq* pReq) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->checkpointId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->downstreamTaskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->downstreamNodeId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->upstreamTaskId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->upstreamNodeId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pReq->childId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pReq->expireTime) < 0) return -1;
+  if (tEncodeI8(pEncoder, pReq->taskLevel) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeSStreamCheckpointReq(SDecoder* pDecoder, SStreamCheckpointReq* pReq) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->checkpointId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->downstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->downstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->upstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->upstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pReq->childId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pReq->expireTime) < 0) return -1;
+  if (tDecodeI8(pDecoder, &pReq->taskLevel) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
+int32_t tEncodeSStreamCheckpointRsp(SEncoder* pEncoder, const SStreamCheckpointRsp* pRsp) {
+  if (tStartEncode(pEncoder) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->streamId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->checkpointId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->downstreamTaskId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->downstreamNodeId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->upstreamTaskId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->upstreamNodeId) < 0) return -1;
+  if (tEncodeI32(pEncoder, pRsp->childId) < 0) return -1;
+  if (tEncodeI64(pEncoder, pRsp->expireTime) < 0) return -1;
+  if (tEncodeI8(pEncoder, pRsp->taskLevel) < 0) return -1;
+  tEndEncode(pEncoder);
+  return pEncoder->pos;
+}
+
+int32_t tDecodeSStreamCheckpointRsp(SDecoder* pDecoder, SStreamCheckpointRsp* pRsp) {
+  if (tStartDecode(pDecoder) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->streamId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->checkpointId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->downstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->downstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->upstreamTaskId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->upstreamNodeId) < 0) return -1;
+  if (tDecodeI32(pDecoder, &pRsp->childId) < 0) return -1;
+  if (tDecodeI64(pDecoder, &pRsp->expireTime) < 0) return -1;
+  if (tDecodeI8(pDecoder, &pRsp->taskLevel) < 0) return -1;
+  tEndDecode(pDecoder);
+  return 0;
+}
+
+static int32_t streamAlignCheckpoint(SStreamTask* pTask, int64_t checkpointId, int32_t childId) {
+  if (pTask->checkpointingId == 0) {
+    pTask->checkpointingId = checkpointId;
+    pTask->checkpointAlignCnt = taosArrayGetSize(pTask->childEpInfo);
+  }
+
+  ASSERT(pTask->checkpointingId == checkpointId);
+
+  return atomic_sub_fetch_32(&pTask->checkpointAlignCnt, 1);
+}
+
+static int32_t streamDoCheckpoint(SStreamMeta* pMeta, SStreamTask* pTask, int64_t checkpointId) {
+  // commit tdb state
+  streamStateCommit(pTask->pState);
+  // commit non-tdb state
+  // copy and save new state
+  // report to mnode
+  // send checkpoint req to downstream
+  return 0;
+}
+
+static int32_t streamDoSourceCheckpoint(SStreamMeta* pMeta, SStreamTask* pTask, int64_t checkpointId) {
+  // ref wal
+  // set status checkpointing
+  // do checkpoint
+  return 0;
+}
+int32_t streamProcessCheckpointSourceReq(SStreamMeta* pMeta, SStreamTask* pTask, SStreamCheckpointSourceReq* pReq) {
+  int32_t code;
+  int64_t checkpointId = pReq->checkpointId;
+
+  code = streamDoSourceCheckpoint(pMeta, pTask, checkpointId);
+  if (code < 0) {
+    // rsp error
+    return -1;
+  }
+
+  return 0;
+}
+
+int32_t streamProcessCheckpointReq(SStreamMeta* pMeta, SStreamTask* pTask, SStreamCheckpointReq* pReq) {
+  int32_t code;
+  int64_t checkpointId = pReq->checkpointId;
+  int32_t childId = pReq->childId;
+
+  if (taosArrayGetSize(pTask->childEpInfo) > 0) {
+    code = streamAlignCheckpoint(pTask, checkpointId, childId);
+    if (code > 0) {
+      return 0;
+    }
+    if (code < 0) {
+      ASSERT(0);
+      return -1;
+    }
+  }
+
+  code = streamDoCheckpoint(pMeta, pTask, checkpointId);
+  if (code < 0) {
+    // rsp error
+    return -1;
+  }
+
+  // send rsp to all children
+
+  return 0;
+}
+
+int32_t streamProcessCheckpointRsp(SStreamMeta* pMeta, SStreamTask* pTask, SStreamCheckpointRsp* pRsp) {
+  // recover step2, scan from wal
+  // unref wal
+  // set status normal
+  return 0;
+}

source/libs/stream/src/streamMeta.c

@@ -202,7 +202,7 @@ void streamMetaReleaseTask(SStreamMeta* pMeta, SStreamTask* pTask) {
   }
 }
 
-void streamMetaRemoveTask1(SStreamMeta* pMeta, int32_t taskId) {
+void streamMetaRemoveTask(SStreamMeta* pMeta, int32_t taskId) {
   SStreamTask** ppTask = (SStreamTask**)taosHashGet(pMeta->pTasks, &taskId, sizeof(int32_t));
   if (ppTask) {
     SStreamTask* pTask = *ppTask;
@@ -219,35 +219,6 @@ void streamMetaRemoveTask1(SStreamMeta* pMeta, int32_t taskId) {
   }
 }
 
-int32_t streamMetaRemoveTask(SStreamMeta* pMeta, int32_t taskId) {
-  SStreamTask** ppTask = (SStreamTask**)taosHashGet(pMeta->pTasks, &taskId, sizeof(int32_t));
-  if (ppTask) {
-    SStreamTask* pTask = *ppTask;
-    taosHashRemove(pMeta->pTasks, &taskId, sizeof(int32_t));
-    atomic_store_8(&pTask->taskStatus, TASK_STATUS__DROPPING);
-
-    if (tdbTbDelete(pMeta->pTaskDb, &taskId, sizeof(int32_t), &pMeta->txn) < 0) {
-      /*return -1;*/
-    }
-
-    if (pTask->triggerParam != 0) {
-      taosTmrStop(pTask->timer);
-    }
-
-    while (1) {
-      int8_t schedStatus =
-          atomic_val_compare_exchange_8(&pTask->schedStatus, TASK_SCHED_STATUS__INACTIVE, TASK_SCHED_STATUS__DROPPING);
-      if (schedStatus != TASK_SCHED_STATUS__ACTIVE) {
-        tFreeSStreamTask(pTask);
-        break;
-      }
-      taosMsleep(10);
-    }
-  }
-
-  return 0;
-}
-
 int32_t streamMetaBegin(SStreamMeta* pMeta) {
   if (tdbTxnOpen(&pMeta->txn, 0, tdbDefaultMalloc, tdbDefaultFree, NULL, TDB_TXN_WRITE | TDB_TXN_READ_UNCOMMITTED) <
       0) {