Merge branch '3.0' of github.com:taosdata/TDengine into test/chr/TD-14699

2022-07-10 14:21:31 +08:00 · 2022-07-10 14:21:31 +08:00 · dd6b510059
parent 4d4ec8e717 644963b745
commit dd6b510059
279 changed files with 9408 additions and 8180 deletions
--- a/.gitmodules
+++ b/.gitmodules
@ -19,3 +19,6 @@
 [submodule "tools/taosadapter"]
 	path = tools/taosadapter
 	url = https://github.com/taosdata/taosadapter.git
+[submodule "tools/taosws-rs"]
+	path = tools/taosws-rs
+	url = https://github.com/taosdata/taosws-rs.git
--- a/README-CN.md
+++ b/README-CN.md
@ -1,38 +1,59 @@
+<p>
+<p align="center">
+  <a href="https://tdengine.com" target="_blank">
+  <img
+    src="docs/assets/tdengine.svg"
+    alt="TDengine"
+    width="500"
+  />
+  </a>
+</p>
+<p>
+
 [![Build Status](https://travis-ci.org/taosdata/TDengine.svg?branch=master)](https://travis-ci.org/taosdata/TDengine)
 [![Build status](https://ci.appveyor.com/api/projects/status/kf3pwh2or5afsgl9/branch/master?svg=true)](https://ci.appveyor.com/project/sangshuduo/tdengine-2n8ge/branch/master)
 [![Coverage Status](https://coveralls.io/repos/github/taosdata/TDengine/badge.svg?branch=develop)](https://coveralls.io/github/taosdata/TDengine?branch=develop)
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4201/badge)](https://bestpractices.coreinfrastructure.org/projects/4201)
 [![tdengine](https://snapcraft.io//tdengine/badge.svg)](https://snapcraft.io/tdengine)

-[![TDengine](TDenginelogo.png)](https://www.taosdata.com)
-
-简体中文 | [English](./README.md) 
+简体中文 | [English](README.md) | 很多职位正在热招中，请看[这里](https://www.taosdata.com/cn/careers/)

 # TDengine 简介

-TDengine是涛思数据专为物联网、车联网、工业互联网、IT运维等设计和优化的大数据平台。除核心的快10倍以上的时序数据库功能外，还提供缓存、数据订阅、流式计算等功能，最大程度减少研发和运维的复杂度，且核心代码，包括集群功能全部开源（开源协议，AGPL v3.0）。
+TDengine 是一款高性能、分布式、支持 SQL 的时序数据库（Time-Series Database）。而且除时序数据库功能外，它还提供缓存、数据订阅、流式计算等功能，最大程度减少研发和运维的复杂度，且核心代码，包括集群功能全部开源（开源协议，AGPL v3.0）。与其他时序数据数据库相比，TDengine 有以下特点：

- 10 倍以上性能提升。定义了创新的数据存储结构，单核每秒就能处理至少2万次请求，插入数百万个数据点，读出一千万以上数据点，比现有通用数据库快了十倍以上。
- 硬件或云服务成本降至1/5。由于超强性能，计算资源不到通用大数据方案的1/5；通过列式存储和先进的压缩算法，存储空间不到通用数据库的1/10。
- 全栈时序数据处理引擎。将数据库、消息队列、缓存、流式计算等功能融合一起，应用无需再集成Kafka/Redis/HBase/Spark等软件，大幅降低应用开发和维护成本。
- 强大的分析功能。无论是十年前还是一秒钟前的数据，指定时间范围即可查询。数据可在时间轴上或多个设备上进行聚合。即席查询可通过Shell/Python/R/Matlab随时进行。
- 与第三方工具无缝连接。不用一行代码，即可与Telegraf, Grafana, EMQ X, Prometheus, Matlab, R集成。后续还将支持MQTT, OPC, Hadoop，Spark等, BI工具也将无缝连接。
- 零运维成本、零学习成本。安装、集群一秒搞定，无需分库分表，实时备份。标准SQL，支持JDBC,RESTful，支持Python/Java/C/C++/Go/Node.JS, 与MySQL相似，零学习成本。
+- **高性能**：通过创新的存储引擎设计，无论是数据写入还是查询，TDengine 的性能比通用数据库快 10 倍以上，也远超其他时序数据库，而且存储空间也大为节省。
+
+- **分布式**：通过原生分布式的设计，TDengine 提供了水平扩展的能力，只需要增加节点就能获得更强的数据处理能力，同时通过多副本机制保证了系统的高可用。
+
+- **支持 SQL**：TDengine 采用 SQL 作为数据查询语言，减少学习和迁移成本，同时提供 SQL 扩展来处理时序数据特有的分析，而且支持方便灵活的 schemaless 数据写入。
+
+- **All in One**：将数据库、消息队列、缓存、流式计算等功能融合一起，应用无需再集成 Kafka/Redis/HBase/Spark 等软件，大幅降低应用开发和维护成本。
+
+- **零管理**：安装、集群几秒搞定，无任何依赖，不用分库分表，系统运行状态监测能与 Grafana 或其他运维工具无缝集成。
+
+- **零学习成本**：采用 SQL 查询语言，支持 Python、Java、C/C++、Go、Rust、Node.js 等多种编程语言，与 MySQL 相似，零学习成本。
+
+- **无缝集成**：不用一行代码，即可与 Telegraf、Grafana、EMQX、Prometheus、StatsD、collectd、Matlab、R 等第三方工具无缝集成。
+
+- **互动 Console**: 通过命令行 console，不用编程，执行 SQL 语句就能做即席查询、各种数据库的操作、管理以及集群的维护.
+
+TDengine 可以广泛应用于物联网、工业互联网、车联网、IT 运维、能源、金融等领域，让大量设备、数据采集器每天产生的高达 TB 甚至 PB 级的数据能得到高效实时的处理，对业务的运行状态进行实时的监测、预警，从大数据中挖掘出商业价值。

 # 文档

-TDengine是一个高效的存储、查询、分析时序大数据的平台，专为物联网、车联网、工业互联网、运维监测等优化而设计。您可以像使用关系型数据库MySQL一样来使用它，但建议您在使用前仔细阅读一遍下面的文档，特别是 [数据模型](https://www.taosdata.com/cn/documentation/architecture) 与 [数据建模](https://www.taosdata.com/cn/documentation/model)。除本文档之外，欢迎 [下载产品白皮书](https://www.taosdata.com/downloads/TDengine%20White%20Paper.pdf)。
+TDengine 采用传统的关系数据库模型，您可以像使用关系型数据库 MySQL 一样来使用它。但由于引入了超级表，一个采集点一张表的概念，建议您在使用前仔细阅读一遍下面的文档，特别是 [数据模型](https://www.taosdata.com/cn/documentation/architecture) 与 [数据建模](https://www.taosdata.com/cn/documentation/model)。除本文档之外，欢迎 [下载产品白皮书](https://www.taosdata.com/downloads/TDengine%20White%20Paper.pdf)。

 # 构建

-TDengine目前2.0版服务器仅能在Linux系统上安装和运行，后续会支持Windows、macOS等系统。客户端可以在Windows或Linux上安装和运行。任何OS的应用也可以选择RESTful接口连接服务器taosd。CPU支持X64/ARM64/MIPS64/Alpha64，后续会支持ARM32、RISC-V等CPU架构。用户可根据需求选择通过[源码](https://www.taosdata.com/cn/getting-started/#通过源码安装)或者[安装包](https://www.taosdata.com/cn/getting-started/#通过安装包安装)来安装。本快速指南仅适用于通过源码安装。
+TDengine 目前 2.0 版服务器仅能在 Linux 系统上安装和运行，后续会支持 Windows、macOS 等系统。客户端可以在 Windows 或 Linux 上安装和运行。任何 OS 的应用也可以选择 RESTful 接口连接服务器 taosd。CPU 支持 X64/ARM64/MIPS64/Alpha64，后续会支持 ARM32、RISC-V 等 CPU 架构。用户可根据需求选择通过[源码](https://www.taosdata.com/cn/getting-started/#通过源码安装)或者[安装包](https://www.taosdata.com/cn/getting-started/#通过安装包安装)来安装。本快速指南仅适用于通过源码安装。

 ## 安装工具

 ### Ubuntu 16.04 及以上版本 & Debian：

 ```bash
-sudo apt-get install -y gcc cmake build-essential git
+sudo apt-get install -y gcc cmake build-essential git libssl-dev
 ```

 ### Ubuntu 14.04：
@ -56,10 +77,22 @@ sudo apt-get install -y openjdk-8-jdk
 sudo apt-get install -y  maven
 ```

+#### 为 taos-tools 安装编译需要的软件
+
+taosTools 是用于 TDengine 的辅助工具软件集合。目前它包含 taosBenchmark（曾命名为 taosdemo）和 taosdump 两个软件。
+
+默认 TDengine 编译不包含 taosTools。您可以在编译 TDengine 时使用`cmake .. -DBUILD_TOOLS=true` 来同时编译 taosTools。
+
+为了在 Ubuntu/Debian 系统上编译 [taos-tools](https://github.com/taosdata/taos-tools) 需要安装如下软件：
+
+```bash
+sudo apt install build-essential libjansson-dev libsnappy-dev liblzma-dev libz-dev pkg-config
+```
+
 ### CentOS 7：

 ```bash
-sudo yum install -y gcc gcc-c++ make cmake git
+sudo yum install -y gcc gcc-c++ make cmake git openssl-devel
 ```

 安装 OpenJDK 8：
@ -74,10 +107,10 @@ sudo yum install -y java-1.8.0-openjdk
 sudo yum install -y maven
 ```

-### CentOS 8 & Fedora:
+### CentOS 8 & Fedora

 ```bash
-sudo dnf install -y gcc gcc-c++ make cmake epel-release git
+sudo dnf install -y gcc gcc-c++ make cmake epel-release git openssl-devel
 ```

 安装 OpenJDK 8：
@ -92,6 +125,33 @@ sudo dnf install -y java-1.8.0-openjdk
 sudo dnf install -y maven
 ```

+#### 在 CentOS 上构建 taosTools 安装依赖软件
+
+为了在 CentOS 上构建 [taosTools](https://github.com/taosdata/taos-tools) 需要安装如下依赖软件
+
+```bash
+sudo yum install zlib-devel xz-devel snappy-devel jansson jansson-devel pkgconfig libatomic libstdc++-static openssl-devel
+```
+
+注意：由于 snappy 缺乏 pkg-config 支持
+（参考 [链接](https://github.com/google/snappy/pull/86)），会导致
+cmake 提示无法发现 libsnappy，实际上工作正常。
+
+### 设置 golang 开发环境
+
+TDengine 包含数个使用 Go 语言开发的组件，请参考 golang.org 官方文档设置 go 开发环境。
+
+请使用 1.14 及以上版本。对于中国用户，我们建议使用代理来加速软件包下载。
+
+```
+go env -w GO111MODULE=on
+go env -w GOPROXY=https://goproxy.cn,direct
+```
+
+### 设置 rust 开发环境
+
+TDengine 包含数个使用 Rust 语言开发的组件. 请参考 rust-lang.org 官方文档设置 rust 开发环境。
+
 ## 获取源码

 首先，你需要从 GitHub 克隆源码：
@ -107,22 +167,41 @@ Go 连接器和 Grafana 插件在其他独立仓库，如果安装它们的话
 git submodule update --init --recursive
 ```

+如果使用 https 协议下载比较慢，可以通过修改 ~/.gitconfig 文件添加以下两行设置使用 ssh 协议下载。需要首先上传 ssh 密钥到 GitHub，详细方法请参考 GitHub 官方文档。
+
+```
+[url "git@github.com:"]
+    insteadOf = https://github.com/
+```
+
 ## 构建 TDengine

 ### Linux 系统

+可以运行代码仓库中的 `build.sh` 脚本编译出 TDengine 和 taosTools（包含 taosBenchmark 和 taosdump）。
+
 ```bash
-mkdir debug && cd debug
-cmake .. && cmake --build .
+./build.sh
 ```

-您可以选择使用 Jemalloc 作为内存分配器，替代默认的 glibc：
+这个脚本等价于执行如下命令：
+
+```bash
+git submodule update --init --recursive
+mkdir debug
+cd debug
+cmake .. -DBUILD_TOOLS=true
+make
+```
+
+您也可以选择使用 jemalloc 作为内存分配器，替代默认的 glibc：
+
 ```bash
 apt install autoconf
 cmake .. -DJEMALLOC_ENABLED=true
 ```

-在X86-64、X86、arm64、arm32 和 mips64 平台上，TDengine 生成脚本可以自动检测机器架构。也可以手动配置 CPUTYPE 参数来指定 CPU 类型，如 aarch64 或 aarch32 等。
+在 X86-64、X86、arm64、arm32 和 mips64 平台上，TDengine 生成脚本可以自动检测机器架构。也可以手动配置 CPUTYPE 参数来指定 CPU 类型，如 aarch64 或 aarch32 等。

 aarch64：

@ -157,7 +236,7 @@ nmake

 如果你使用的是 Visual Studio 2019 或 2017 版本：

-打开cmd.exe，执行 vcvarsall.bat 时，为 64 位操作系统指定“x64”，为 32 位操作系统指定“x86”。
+打开 cmd.exe，执行 vcvarsall.bat 时，为 64 位操作系统指定“x64”，为 32 位操作系统指定“x86”。

 ```bash
 mkdir debug && cd debug
@ -174,9 +253,7 @@ cmake .. -G "NMake Makefiles"
 nmake
 ```

-如果你使用的是 Visual Studio 2022 版本， 脚本 `vcvarsall.bat` 的默认安装路径是 `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat`。 
-
-### Mac OS X 系统
+### macOS 系统

 安装 Xcode 命令行工具和 cmake. 在 Catalina 和 Big Sur 操作系统上，需要安装 XCode 11.4+ 版本。

@ -187,13 +264,17 @@ cmake .. && cmake --build .

 # 安装

-生成完成后，安装 TDengine（下文给出的指令以 Linux 为例，如果是在 Windows 下，那么对应的指令会是 `nmake install`）：
+## Linux 系统
+
+生成完成后，安装 TDengine：

 ```bash
 sudo make install
 ```

 用户可以在[文件目录结构](https://www.taosdata.com/cn/documentation/administrator#directories)中了解更多在操作系统中生成的目录或文件。
+从 2.0 版本开始, 从源代码安装也会为 TDengine 配置服务管理。
+用户也可以选择[从安装包中安装](https://www.taosdata.com/en/getting-started/#Install-from-Package)。

 安装成功后，在终端中启动 TDengine 服务：

@ -209,6 +290,40 @@ taos

 如果 TDengine Shell 连接服务成功，将会打印出欢迎消息和版本信息。如果失败，则会打印出错误消息。

+## Windows 系统
+
+生成完成后，安装 TDengine：
+
+```cmd
+nmake install
+```
+
+## macOS 系统
+
+生成完成后，安装 TDengine：
+
+```bash
+sudo make install
+```
+
+安装成功后，如果想以服务形式启动，先配置 `.plist` 文件，在终端中执行：
+
+```bash
+sudo cp ../packaging/macOS/com.taosdata.tdengine.plist /Library/LaunchDaemons
+```
+
+在终端中启动 TDengine 服务：
+
+```bash
+sudo launchctl load /Library/LaunchDaemons/com.taosdata.tdengine.plist
+```
+
+在终端中停止 TDengine 服务：
+
+```bash
+sudo launchctl unload /Library/LaunchDaemons/com.taosdata.tdengine.plist
+```
+
 ## 快速运行

 如果不希望以服务方式运行 TDengine，也可以在终端中直接运行它。也即在生成完成后，执行以下命令（在 Windows 下，生成的可执行文件会带有 .exe 后缀，例如会名为 taosd.exe ）：
@ -227,15 +342,15 @@ taos

 # 体验 TDengine

-在TDengine终端中，用户可以通过SQL命令来创建/删除数据库、表等，并进行插入查询操作。
+在 TDengine 终端中，用户可以通过 SQL 命令来创建/删除数据库、表等，并进行插入查询操作。

-```bash
-create database demo;
-use demo;
-create table t (ts timestamp, speed int);
-insert into t values ('2019-07-15 00:00:00', 10);
-insert into t values ('2019-07-15 01:00:00', 20);
-select * from t;
+```sql
+CREATE DATABASE demo;
+USE demo;
+CREATE TABLE t (ts TIMESTAMP, speed INT);
+INSERT INTO t VALUES('2019-07-15 00:00:00', 10);
+INSERT INTO t VALUES('2019-07-15 01:00:00', 20);
+SELECT * FROM t;
          ts          |   speed   |
 ===================================
 19-07-15 00:00:00.000|         10|
@ -247,33 +362,35 @@ Query OK, 2 row(s) in set (0.001700s)

 ## 官方连接器

-TDengine 提供了丰富的应用程序开发接口，其中包括C/C++、Java、Python、Go、Node.js、C# 、RESTful 等，便于用户快速开发应用：
+TDengine 提供了丰富的应用程序开发接口，其中包括 C/C++、Java、Python、Go、Node.js、C# 、RESTful 等，便于用户快速开发应用：

- Java
+- [Java](https://www.taosdata.com/cn/documentation/connector/java)

- C/C++
+- [C/C++](https://www.taosdata.com/cn/documentation/connector#c-cpp)

- Python
+- [Python](https://www.taosdata.com/cn/documentation/connector#python)

- Go
+- [Go](https://www.taosdata.com/cn/documentation/connector#go)

- RESTful API
+- [RESTful API](https://www.taosdata.com/cn/documentation/connector#restful)

- Node.js
+- [Node.js](https://www.taosdata.com/cn/documentation/connector#nodejs)
+
+- [Rust](https://www.taosdata.com/cn/documentation/connector/rust)

 ## 第三方连接器

 TDengine 社区生态中也有一些非常友好的第三方连接器，可以通过以下链接访问它们的源码。

- [Rust Connector](https://github.com/taosdata/TDengine/tree/master/tests/examples/rust)
+- [Rust Bindings](https://github.com/songtianyi/tdengine-rust-bindings/tree/master/examples)
 - [.Net Core Connector](https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos)
- [Lua Connector](https://github.com/taosdata/TDengine/tree/develop/tests/examples/lua)
+- [Lua Connector](https://github.com/taosdata/TDengine/tree/develop/examples/lua)

 # 运行和添加测试例

 TDengine 的测试框架和所有测试例全部开源。

-点击 [这里](tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md)，了解如何运行测试例和添加新的测试例。
+点击 [这里](https://github.com/taosdata/TDengine/blob/develop/tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md)，了解如何运行测试例和添加新的测试例。

 # 成为社区贡献者

@ -281,8 +398,8 @@ TDengine 的测试框架和所有测试例全部开源。

 # 加入技术交流群

-TDengine 官方社群「物联网大数据群」对外开放，欢迎您加入讨论。搜索微信号 "tdengine"，加小T为好友，即可入群。
+TDengine 官方社群「物联网大数据群」对外开放，欢迎您加入讨论。搜索微信号 "tdengine"，加小 T 为好友，即可入群。

-# [谁在使用TDengine](https://github.com/taosdata/TDengine/issues/2432)
+# [谁在使用 TDengine](https://github.com/taosdata/TDengine/issues/2432)

-欢迎所有 TDengine 用户及贡献者在 [这里](https://github.com/taosdata/TDengine/issues/2432) 分享您在当前工作中开发/使用 TDengine 的故事。 
+欢迎所有 TDengine 用户及贡献者在 [这里](https://github.com/taosdata/TDengine/issues/2432) 分享您在当前工作中开发/使用 TDengine 的故事。
--- a/README.md
+++ b/README.md
@ -1,116 +1,218 @@
+<p>
+<p align="center">
+  <a href="https://tdengine.com" target="_blank">
+  <img
+    src="docs/assets/tdengine.svg"
+    alt="TDengine"
+    width="500"
+  />
+  </a>
+</p>
+<p>
+
 [![Build Status](https://cloud.drone.io/api/badges/taosdata/TDengine/status.svg?ref=refs/heads/master)](https://cloud.drone.io/taosdata/TDengine)
 [![Build status](https://ci.appveyor.com/api/projects/status/kf3pwh2or5afsgl9/branch/master?svg=true)](https://ci.appveyor.com/project/sangshuduo/tdengine-2n8ge/branch/master)
 [![Coverage Status](https://coveralls.io/repos/github/taosdata/TDengine/badge.svg?branch=develop)](https://coveralls.io/github/taosdata/TDengine?branch=develop)
 [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4201/badge)](https://bestpractices.coreinfrastructure.org/projects/4201)
 [![tdengine](https://snapcraft.io//tdengine/badge.svg)](https://snapcraft.io/tdengine)

-[![TDengine](TDenginelogo.png)](https://www.taosdata.com)
-
-English | [简体中文](./README-CN.md) 
+English | [简体中文](README-CN.md) | We are hiring, check [here](https://tdengine.com/careers)

 # What is TDengine？

-TDengine is an open-sourced big data platform under [GNU AGPL v3.0](http://www.gnu.org/licenses/agpl-3.0.html), designed and optimized for the Internet of Things (IoT), Connected Cars, Industrial IoT, and IT Infrastructure and Application Monitoring. Besides the 10x faster time-series database, it provides caching, stream computing, message queuing and other functionalities to reduce the complexity and cost of development and operation.
+TDengine is a high-performance, scalable time-series database with SQL support. Its code including cluster feature is open source under [GNU AGPL v3.0](http://www.gnu.org/licenses/agpl-3.0.html). Besides the database, it provides caching, stream processing, data subscription and other functionalities to reduce the complexity and cost of development and operation. TDengine differentiates itself from other TSDBs with the following advantages.

- **10x Faster on Insert/Query Speeds**: Through the innovative design on storage, on a single-core machine, over 20K requests can be processed, millions of data points can be ingested, and over 10 million data points can be retrieved in a second. It is 10 times faster than other databases.
+- **High Performance**: TDengine outperforms other time series databases in data ingestion and querying while significantly reducing storage cost and compute costs, with an innovatively designed and purpose-built storage engine.

- **1/5 Hardware/Cloud Service Costs**: Compared with typical big data solutions, less than 1/5 of computing resources are required. Via column-based storage and tuned compression algorithms for different data types, less than 1/10 of storage space is needed.
+- **Scalable**: TDengine provides out-of-box scalability and high-availability through its native distributed design. Nodes can be added through simple configuration to achieve greater data processing power. In addition, this feature is open source.

- **Full Stack for Time-Series Data**: By integrating a database with message queuing, caching, and stream computing features together, it is no longer necessary to integrate Kafka/Redis/HBase/Spark or other software. It makes the system architecture much simpler and more robust.
+- **SQL Support**: TDengine uses SQL as the query language, thereby reducing learning and migration costs, while adding SQL extensions to handle time-series data better, and supporting convenient and flexible schemaless data ingestion.

- **Powerful Data Analysis**: Whether it is 10 years or one minute ago, data can be queried just by specifying the time range. Data can be aggregated over time, multiple time streams or both. Ad Hoc queries or analyses can be executed via TDengine shell, Python, R or Matlab.
+- **All in One**: TDengine has built-in caching, stream processing and data subscription functions, it is no longer necessary to integrate Kafka/Redis/HBase/Spark or other software in some scenarios. It makes the system architecture much simpler and easy to maintain.

- **Seamless Integration with Other Tools**: Telegraf, Grafana, Matlab, R, and other tools can be integrated with TDengine without a line of code. MQTT, OPC, Hadoop, Spark, and many others will be integrated soon.
+- **Seamless Integration**: Without a single line of code, TDengine provide seamless integration with third-party tools such as Telegraf, Grafana, EMQX, Prometheus, StatsD, collectd, etc. More will be integrated.

- **Zero Management, No Learning Curve**: It takes only seconds to download, install, and run it successfully; there are no other dependencies. Automatic partitioning on tables or DBs. Standard SQL is used, with C/C++, Python, JDBC, Go and RESTful connectors.
+- **Zero Management**: Installation and cluster setup can be done in seconds. Data partitioning and sharding are executed automatically. TDengine’s running status can be monitored via Grafana or other DevOps tools.
+
+- **Zero Learning Cost**: With SQL as the query language, support for ubiquitous tools like Python, Java, C/C++, Go, Rust, Node.js connectors, there is zero learning cost.
+
+- **Interactive Console**: TDengine provides convenient console access to the database to run ad hoc queries, maintain the database, or manage the cluster without any programming.
+
+TDengine can be widely applied to Internet of Things (IoT), Connected Vehicles, Industrial IoT, DevOps, energy, finance and many other scenarios.

 # Documentation
+
 For user manual, system design and architecture, engineering blogs, refer to [TDengine Documentation](https://www.taosdata.com/en/documentation/)(中文版请点击[这里](https://www.taosdata.com/cn/documentation20/))
- for details. The documentation from our website can also be downloaded locally from *documentation/tdenginedocs-en* or *documentation/tdenginedocs-cn*.
+for details. The documentation from our website can also be downloaded locally from _documentation/tdenginedocs-en_ or _documentation/tdenginedocs-cn_.

 # Building
-At the moment, TDengine only supports building and running on Linux systems. You can choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) or from the source code. This quick guide is for installation from the source only.

-To build TDengine, use [CMake](https://cmake.org/) 2.8.12.x or higher versions in the project directory. 
+At the moment, TDengine server only supports running on Linux systems. You can choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) or build it from the source code. This quick guide is for installation from the source only.

-## Install tools
+To build TDengine, use [CMake](https://cmake.org/) 3.0.2 or higher versions in the project directory.
+
+## Install build dependencies
+
+### Ubuntu 16.04 and above or Debian

-### Ubuntu 16.04 and above & Debian:
 ```bash
-sudo apt-get install -y gcc cmake build-essential git
+sudo apt-get install -y gcc cmake build-essential git libssl-dev
 ```

-### Ubuntu 14.04:
+### Ubuntu 14.04
+
 ```bash
 sudo apt-get install -y gcc cmake3 build-essential git binutils-2.26
 export PATH=/usr/lib/binutils-2.26/bin:$PATH
 ```

-To compile and package the JDBC driver source code, you should have a Java jdk-8 or higher and Apache Maven 2.7 or higher installed. 
+To compile and package the JDBC driver source code, you should have a Java jdk-8 or higher and Apache Maven 2.7 or higher installed.
+
 To install openjdk-8:
+
 ```bash
 sudo apt-get install -y openjdk-8-jdk
 ```

 To install Apache Maven:
+
 ```bash
-sudo apt-get install -y  maven
+sudo apt-get install -y maven
 ```

-### Centos 7:
+#### Install build dependencies for taosTools
+
+We provide a few useful tools such as taosBenchmark (was named taosdemo) and taosdump. They were part of TDengine. From TDengine 2.4.0.0, taosBenchmark and taosdump were not released together with TDengine.
+By default, TDengine compiling does not include taosTools. You can use 'cmake .. -DBUILD_TOOLS=true' to make them be compiled with TDengine.
+
+To build the [taosTools](https://github.com/taosdata/taos-tools) on Ubuntu/Debian, the following packages need to be installed.
+
 ```bash
-sudo yum install -y gcc gcc-c++ make cmake git
+sudo apt install build-essential libjansson-dev libsnappy-dev liblzma-dev libz-dev pkg-config
+```
+
+### CentOS 7
+
+```bash
+sudo yum install epel-release
+sudo yum update
+sudo yum install -y gcc gcc-c++ make cmake3 git openssl-devel
+sudo ln -sf /usr/bin/cmake3 /usr/bin/cmake
 ```

 To install openjdk-8:
+
 ```bash
 sudo yum install -y java-1.8.0-openjdk
 ```

 To install Apache Maven:
+
 ```bash
 sudo yum install -y maven
 ```

-### Centos 8 & Fedora:
+### CentOS 8 & Fedora
+
 ```bash
-sudo dnf install -y gcc gcc-c++ make cmake epel-release git
+sudo dnf install -y gcc gcc-c++ make cmake epel-release git openssl-devel
 ```

 To install openjdk-8:
+
 ```bash
 sudo dnf install -y java-1.8.0-openjdk
 ```

 To install Apache Maven:
+
 ```bash
 sudo dnf install -y maven
 ```

+#### Install build dependencies for taosTools on CentOS
+
+To build the [taosTools](https://github.com/taosdata/taos-tools) on CentOS, the following packages need to be installed.
+
+```bash
+sudo yum install zlib-devel xz-devel snappy-devel jansson jansson-devel pkgconfig libatomic libstdc++-static openssl-devel
+```
+
+Note: Since snappy lacks pkg-config support (refer to [link](https://github.com/google/snappy/pull/86)), it lead a cmake prompt libsnappy not found. But snappy will works well.
+
+### Setup golang environment
+
+TDengine includes few components developed by Go language. Please refer to golang.org official documentation for golang environment setup.
+
+Please use version 1.14+. For the user in China, we recommend using a proxy to accelerate package downloading.
+
+```
+go env -w GO111MODULE=on
+go env -w GOPROXY=https://goproxy.cn,direct
+```
+
+### Setup rust environment
+
+TDengine includees few compoments developed by Rust language. Please refer to rust-lang.org official documentation for rust environment setup.
+
 ## Get the source codes

 First of all, you may clone the source codes from github:
+
 ```bash
 git clone https://github.com/taosdata/TDengine.git
 cd TDengine
 ```

-The connectors for go & grafana have been moved to separated repositories,
+The connectors for go & Grafana and some tools have been moved to separated repositories,
 so you should run this command in the TDengine directory to install them:
+
 ```bash
 git submodule update --init --recursive
 ```

+You can modify the file ~/.gitconfig to use ssh protocol instead of https for better download speed. You need to upload ssh public key to GitHub first. Please refer to GitHub official documentation for detail.
+
+```
+[url "git@github.com:"]
+    insteadOf = https://github.com/
+```
+
 ## Build TDengine

 ### On Linux platform

+You can run the bash script `build.sh` to build both TDengine and taosTools including taosBenchmark and taosdump as below:
+
 ```bash
-mkdir debug && cd debug
-cmake .. && cmake --build .
+./build.sh
+```
+
+It equals to execute following commands:
+
+```bash
+git submodule update --init --recursive
+mkdir debug
+cd debug
+cmake .. -DBUILD_TOOLS=true
+make
+```
+
+Note TDengine 2.3.x.0 and later use a component named 'taosAdapter' to play http daemon role by default instead of the http daemon embedded in the early version of TDengine. The taosAdapter is programmed by go language. If you pull TDengine source code to the latest from an existing codebase, please execute 'git submodule update --init --recursive' to pull taosAdapter source code. Please install go language version 1.14 or above for compiling taosAdapter. If you meet difficulties regarding 'go mod', especially you are from China, you can use a proxy to solve the problem.
+
+```
+go env -w GO111MODULE=on
+go env -w GOPROXY=https://goproxy.cn,direct
+```
+
+The embedded http daemon still be built from TDengine source code by default. Or you can use the following command to choose to build taosAdapter.
+
+```
+cmake .. -DBUILD_HTTP=false
 ```

 You can use Jemalloc as memory allocator instead of glibc:
+
 ```
 apt install autoconf
 cmake .. -DJEMALLOC_ENABLED=true
@ -120,24 +222,28 @@ TDengine build script can detect the host machine's architecture on X86-64, X86,
 You can also specify CPUTYPE option like aarch64 or aarch32 too if the detection result is not correct:

 aarch64:
+
 ```bash
 cmake .. -DCPUTYPE=aarch64 && cmake --build .
 ```

 aarch32:
+
 ```bash
 cmake .. -DCPUTYPE=aarch32 && cmake --build .
 ```

 mips64:
+
 ```bash
 cmake .. -DCPUTYPE=mips64 && cmake --build .
 ```

 ### On Windows platform

-If you use Visual Studio 2013, please open a command window by executing "cmd.exe".
+If you use the Visual Studio 2013, please open a command window by executing "cmd.exe".
 Please specify "amd64" for 64 bits Windows or specify "x86" is for 32 bits Windows when you execute vcvarsall.bat.
+
 ```cmd
 mkdir debug && cd debug
 "C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\vcvarsall.bat" < amd64 | x86 >
@ -145,7 +251,7 @@ cmake .. -G "NMake Makefiles"
 nmake
 ```

-If you use Visual Studio 2019 or 2017:
+If you use the Visual Studio 2019 or 2017:

 please open a command window by executing "cmd.exe".
 Please specify "x64" for 64 bits Windows or specify "x86" is for 32 bits Windows when you execute vcvarsall.bat.
@ -158,15 +264,14 @@ nmake
 ```

 Or, you can simply open a command window by clicking Windows Start -> "Visual Studio < 2019 | 2017 >" folder -> "x64 Native Tools Command Prompt for VS < 2019 | 2017 >" or "x86 Native Tools Command Prompt for VS < 2019 | 2017 >" depends what architecture your Windows is, then execute commands as follows:
+
 ```cmd
 mkdir debug && cd debug
 cmake .. -G "NMake Makefiles"
 nmake
 ```

-If you use Visual Studio 2022, the only change is the default path of `vcvarsall.bat`, which is `C:\Program Files\Microsoft Visual Studio\2022\Community\VC\Auxiliary\Build\vcvarsall.bat`.
-
-### On Mac OS X platform
+### On macOS platform

 Please install XCode command line tools and cmake. Verified with XCode 11.4+ on Catalina and Big Sur.

@ -177,7 +282,10 @@ cmake .. && cmake --build .

 # Installing

-After building successfully, TDengine can be installed by: (On Windows platform, the following command should be `nmake install`)
+## On Linux platform
+
+After building successfully, TDengine can be installed by
+
 ```bash
 sudo make install
 ```
@ -186,68 +294,129 @@ Users can find more information about directories installed on the system in the
 Users can also choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) for it.

 To start the service after installation, in a terminal, use:
+
 ```bash
 sudo systemctl start taosd
 ```

 Then users can use the [TDengine shell](https://www.taosdata.com/en/getting-started/#TDengine-Shell) to connect the TDengine server. In a terminal, use:
+
 ```bash
 taos
 ```

 If TDengine shell connects the server successfully, welcome messages and version info are printed. Otherwise, an error message is shown.

+### Install TDengine by apt-get
+
+If you use Debian or Ubuntu system, you can use 'apt-get' command to install TDengine from official repository. Please use following commands to setup:
+
+```
+wget -qO - http://repos.taosdata.com/tdengine.key | sudo apt-key add -
+echo "deb [arch=amd64] http://repos.taosdata.com/tdengine-stable stable main" | sudo tee /etc/apt/sources.list.d/tdengine-stable.list
+[Optional] echo "deb [arch=amd64] http://repos.taosdata.com/tdengine-beta beta main" | sudo tee /etc/apt/sources.list.d/tdengine-beta.list
+sudo apt-get update
+apt-cache policy tdengine
+sudo apt-get install tdengine
+```
+
+## On Windows platform
+
+After building successfully, TDengine can be installed by:
+
+```cmd
+nmake install
+```
+
+## On macOS platform
+
+After building successfully, TDengine can be installed by:
+
+```bash
+sudo make install
+```
+
+To start the service after installation, config `.plist` file first, in a terminal, use:
+
+```bash
+sudo cp ../packaging/macOS/com.taosdata.tdengine.plist /Library/LaunchDaemons
+```
+
+To start the service, in a terminal, use:
+
+```bash
+sudo launchctl load /Library/LaunchDaemons/com.taosdata.tdengine.plist
+```
+
+To stop the service, in a terminal, use:
+
+```bash
+sudo launchctl unload /Library/LaunchDaemons/com.taosdata.tdengine.plist
+```
+
 ## Quick Run

 If you don't want to run TDengine as a service, you can run it in current shell. For example, to quickly start a TDengine server after building, run the command below in terminal: (We take Linux as an example, command on Windows will be `taosd.exe`)
+
 ```bash
 ./build/bin/taosd -c test/cfg
 ```

 In another terminal, use the TDengine shell to connect the server:
+
 ```bash
 ./build/bin/taos -c test/cfg
 ```

-option "-c test/cfg" specifies the system configuration file directory. 
+option "-c test/cfg" specifies the system configuration file directory.

 # Try TDengine
+
 It is easy to run SQL commands from TDengine shell which is the same as other SQL databases.
+
 ```sql
-create database db;
-use db;
-create table t (ts timestamp, a int);
-insert into t values ('2019-07-15 00:00:00', 1);
-insert into t values ('2019-07-15 01:00:00', 2);
-select * from t;
-drop database db;
+CREATE DATABASE demo;
+USE demo;
+CREATE TABLE t (ts TIMESTAMP, speed INT);
+INSERT INTO t VALUES('2019-07-15 00:00:00', 10);
+INSERT INTO t VALUES('2019-07-15 01:00:00', 20);
+SELECT * FROM t;
+          ts          |   speed   |
+===================================
+ 19-07-15 00:00:00.000|         10|
+ 19-07-15 01:00:00.000|         20|
+Query OK, 2 row(s) in set (0.001700s)
 ```

 # Developing with TDengine
-### Official Connectors
+
+## Official Connectors

 TDengine provides abundant developing tools for users to develop on TDengine. Follow the links below to find your desired connectors and relevant documentation.

- [Java](https://www.taosdata.com/en/documentation/connector/#Java-Connector)
- [C/C++](https://www.taosdata.com/en/documentation/connector/#C/C++-Connector)
- [Python](https://www.taosdata.com/en/documentation/connector/#Python-Connector)
- [Go](https://www.taosdata.com/en/documentation/connector/#Go-Connector)
- [RESTful API](https://www.taosdata.com/en/documentation/connector/#RESTful-Connector)
- [Node.js](https://www.taosdata.com/en/documentation/connector/#Node.js-Connector)
+- [Java](https://www.taosdata.com/en/documentation/connector/java)
+- [C/C++](https://www.taosdata.com/en/documentation/connector#c-cpp)
+- [Python](https://www.taosdata.com/en/documentation/connector#python)
+- [Go](https://www.taosdata.com/en/documentation/connector#go)
+- [RESTful API](https://www.taosdata.com/en/documentation/connector#restful)
+- [Node.js](https://www.taosdata.com/en/documentation/connector#nodejs)
+- [Rust](https://www.taosdata.com/en/documentation/connector/rust)

-### Third Party Connectors
+## Third Party Connectors

 The TDengine community has also kindly built some of their own connectors! Follow the links below to find the source code for them.

- [Rust Connector](https://github.com/taosdata/TDengine/tree/master/tests/examples/rust)
+- [Rust Bindings](https://github.com/songtianyi/tdengine-rust-bindings/tree/master/examples)
 - [.Net Core Connector](https://github.com/maikebing/Maikebing.EntityFrameworkCore.Taos)
 - [Lua Connector](https://github.com/taosdata/TDengine/tree/develop/tests/examples/lua)

-# How to run the test cases and how to add a new test case?
-  TDengine's test framework and all test cases are fully open source.
-  Please refer to [this document](tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md) for how to run test and develop new test case.
+# How to run the test cases and how to add a new test case
+
+TDengine's test framework and all test cases are fully open source.
+Please refer to [this document](https://github.com/taosdata/TDengine/blob/develop/tests/How-To-Run-Test-And-How-To-Add-New-Test-Case.md) for how to run test and develop new test case.

 # TDengine Roadmap
+
 - Support event-driven stream computing
 - Support user defined functions
 - Support MQTT connection
--- a/cmake/cmake.define
+++ b/cmake/cmake.define
@ -18,6 +18,14 @@ if (NOT DEFINED TD_GRANT)
  SET(TD_GRANT FALSE)
 endif()

+IF ("${WEBSOCKET}" MATCHES "true")
+    SET(TD_WEBSOCKET TRUE)
+    MESSAGE("Enable websocket")
+    ADD_DEFINITIONS(-DWEBSOCKET)
+ELSE ()
+    SET(TD_WEBSOCKET FALSE)
+ENDIF ()
+
 IF ("${BUILD_HTTP}" STREQUAL "")
  IF (TD_LINUX)
    IF (TD_ARM_32)
@ -97,10 +105,10 @@ ELSE ()
        SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${GCC_COVERAGE_COMPILE_FLAGS} ${GCC_COVERAGE_LINK_FLAGS}")
    ENDIF ()

-    IF (${SANITIZER} MATCHES "true")
+    IF (${BUILD_SANITIZER})
        SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS}     -Werror -Werror=return-type -fPIC -gdwarf-2 -fsanitize=address -fsanitize=undefined -fsanitize-recover=all -fsanitize=float-divide-by-zero -fsanitize=float-cast-overflow -fno-sanitize=shift-base -fno-sanitize=alignment -g3 -Wformat=0")
        SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Werror -Wno-literal-suffix -Werror=return-type -fPIC -gdwarf-2 -fsanitize=address -fsanitize=undefined -fsanitize-recover=all -fsanitize=float-divide-by-zero -fsanitize=float-cast-overflow -fno-sanitize=shift-base -fno-sanitize=alignment -g3 -Wformat=0")
-	MESSAGE(STATUS "Will compile with Address Sanitizer!")
+	      MESSAGE(STATUS "Will compile with Address Sanitizer!")
    ELSE ()
        SET(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Werror -Werror=return-type -fPIC -gdwarf-2 -g3 -Wformat=0")
        SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Werror -Wno-literal-suffix -Werror=return-type -fPIC -gdwarf-2 -g3 -Wformat=0")
--- a/cmake/cmake.options
+++ b/cmake/cmake.options
@ -77,6 +77,12 @@ ELSE ()
    ENDIF ()
 ENDIF ()

+option(
+    BUILD_SANITIZER
+    "If build addr2line"
+    OFF
+    )
+
 option(
    BUILD_ADDR2LINE
    "If build addr2line"
--- a/docs/en/25-application/03-immigrate.md
+++ b/docs/en/25-application/03-immigrate.md
@ -379,11 +379,11 @@ We still use the hypothetical environment from Chapter 4. There are three measur

 ### Storage resource estimation

-Assuming that the number of sensor devices that generate data and need to be stored is `n`, the frequency of data generation is `t` per second, and the length of each record is `L` bytes, the scale of data generated per day is `n * t * L` bytes. Assuming the compression ratio is `C`, the daily data size is `(n * t * L)/C` bytes. The storage resources are estimated to accommodate the data scale for 1.5 years. In the production environment, the compression ratio C of TDengine is generally between 5 and 7.
+Assuming that the number of sensor devices that generate data and need to be stored is `n`, the frequency of data generation is `t` per second, and the length of each record is `L` bytes, the scale of data generated per day is `86400 * n * t * L` bytes. Assuming the compression ratio is `C`, the daily data size is `(86400 * n * t * L)/C` bytes. The storage resources are estimated to accommodate the data scale for 1.5 years. In the production environment, the compression ratio C of TDengine is generally between 5 and 7.
 With additional 20% redundancy, you can calculate the required storage resources:

 ```matlab
-(n * t * L) * (365 * 1.5) * (1+20%)/C
+(86400 * n * t * L) * (365 * 1.5) * (1+20%)/C
 ````
 Substituting in the above formula, the raw data generated every year is 11.8TB without considering the label information. Note that tag information is associated with each timeline in TDengine, not every record. The amount of data to be recorded is somewhat reduced relative to the generated data, and label data can be ignored as a whole. Assuming a compression ratio of 5, the size of the retained data ends up being 2.56 TB.

--- a/docs/zh/25-application/03-immigrate.md
+++ b/docs/zh/25-application/03-immigrate.md
@ -367,10 +367,10 @@ WHERE ts>=1510560000 AND ts<=1515000009

 ### 存储资源估算

-假设产生数据并需要存储的传感器设备数量为 `n`，数据生成的频率为`t`条/秒，每条记录的长度为 `L` bytes，则每天产生的数据规模为 `n×t×L` bytes。假设压缩比为 C，则每日产生数据规模为 `(n×t×L)/C` bytes。存储资源预估为能够容纳 1.5 年的数据规模，生产环境下 TDengine 的压缩比 C 一般在 5 ~ 7 之间，同时为最后结果增加 20% 的冗余，可计算得到需要存储资源：
+假设产生数据并需要存储的传感器设备数量为 `n`，数据生成的频率为`t`条/秒，每条记录的长度为 `L` bytes，则每天产生的数据规模为 `86400×n×t×L` bytes。假设压缩比为 C，则每日产生数据规模为 `(86400×n×t×L)/C` bytes。存储资源预估为能够容纳 1.5 年的数据规模，生产环境下 TDengine 的压缩比 C 一般在 5 ~ 7 之间，同时为最后结果增加 20% 的冗余，可计算得到需要存储资源：

 ```matlab
-(n×t×L)×(365×1.5)×(1+20%)/C
+(86400×n×t×L)×(365×1.5)×(1+20%)/C
 ```

 结合以上的计算公式，将参数带入计算公式，在不考虑标签信息的情况下，每年产生的原始数据规模是 11.8TB。需要注意的是，由于标签信息在 TDengine 中关联到每个时间线，并不是每条记录。所以需要记录的数据量规模相对于产生的数据有一定的降低，而这部分标签数据整体上可以忽略不记。假设压缩比为 5，则保留的数据规模最终为 2.56 TB。
--- a/examples/c/tmq.c
+++ b/examples/c/tmq.c
@ -28,15 +28,23 @@ static void msg_process(TAOS_RES* msg) {
  printf("db: %s\n", tmq_get_db_name(msg));
  printf("vg: %d\n", tmq_get_vgroup_id(msg));
  if (tmq_get_res_type(msg) == TMQ_RES_TABLE_META) {
-    void*   meta;
-    int32_t metaLen;
-    tmq_get_raw_meta(msg, &meta, &metaLen);
+    tmq_raw_data *raw = tmq_get_raw_meta(msg);
+    if(raw){
+      TAOS* pConn = taos_connect("192.168.1.86", "root", "taosdata", "abc1", 0);
+      if (pConn == NULL) {
+        return;
+      }
+      int32_t ret = taos_write_raw_meta(pConn, raw);
+      printf("write raw data: %s\n", tmq_err2str(ret));
+      free(raw);
+      taos_close(pConn);
+    }
    char* result = tmq_get_json_meta(msg);
    if(result){
      printf("meta result: %s\n", result);
      free(result);
    }
-    printf("meta, len is %d\n", metaLen);
+    printf("meta:%p\n", raw);
    return;
  }
  while (1) {
--- a/include/client/taos.h
+++ b/include/client/taos.h
@ -260,15 +260,16 @@ enum tmq_res_t {
 };

 typedef enum tmq_res_t tmq_res_t;
+typedef struct tmq_raw_data tmq_raw_data;

-DLL_EXPORT tmq_res_t   tmq_get_res_type(TAOS_RES *res);
-DLL_EXPORT int32_t     tmq_get_raw_meta(TAOS_RES *res, void **raw_meta, int32_t *raw_meta_len);
-DLL_EXPORT int32_t     taos_write_raw_meta(TAOS *res, void *raw_meta, int32_t raw_meta_len);
-DLL_EXPORT char       *tmq_get_json_meta(TAOS_RES *res);   // Returning null means error. Returned result need to be freed.
-DLL_EXPORT const char *tmq_get_topic_name(TAOS_RES *res);
-DLL_EXPORT const char *tmq_get_db_name(TAOS_RES *res);
-DLL_EXPORT int32_t     tmq_get_vgroup_id(TAOS_RES *res);
-DLL_EXPORT const char *tmq_get_table_name(TAOS_RES *res);
+DLL_EXPORT tmq_res_t     tmq_get_res_type(TAOS_RES *res);
+DLL_EXPORT tmq_raw_data *tmq_get_raw_meta(TAOS_RES *res);
+DLL_EXPORT int32_t       taos_write_raw_meta(TAOS *taos, tmq_raw_data *raw_meta);
+DLL_EXPORT char         *tmq_get_json_meta(TAOS_RES *res);   // Returning null means error. Returned result need to be freed.
+DLL_EXPORT const char   *tmq_get_topic_name(TAOS_RES *res);
+DLL_EXPORT const char   *tmq_get_db_name(TAOS_RES *res);
+DLL_EXPORT int32_t       tmq_get_vgroup_id(TAOS_RES *res);
+DLL_EXPORT const char   *tmq_get_table_name(TAOS_RES *res);

 /* ------------------------------ TMQ END -------------------------------- */

--- a/include/common/tcommon.h
+++ b/include/common/tcommon.h
@ -55,7 +55,8 @@ enum {
 enum {
  STREAM_INPUT__DATA_SUBMIT = 1,
  STREAM_INPUT__DATA_BLOCK,
-  STREAM_INPUT__DATA_SCAN,
+  STREAM_INPUT__TABLE_SCAN,
+  STREAM_INPUT__TQ_SCAN,
  STREAM_INPUT__DATA_RETRIEVE,
  STREAM_INPUT__TRIGGER,
  STREAM_INPUT__CHECKPOINT,
@ -88,8 +89,6 @@ typedef struct {
 #pragma pack(push, 1)
 typedef struct SColumnDataAgg {
  int16_t colId;
-  int16_t minIndex;
-  int16_t maxIndex;
  int16_t numOfNull;
  int64_t sum;
  int64_t max;
@ -124,7 +123,8 @@ enum {
 };

 typedef struct {
-  int8_t fetchType;
+  int8_t       fetchType;
+  STqOffsetVal offset;
  union {
    SSDataBlock data;
    void*       meta;
--- a/include/common/tdatablock.h
+++ b/include/common/tdatablock.h
@ -231,7 +231,7 @@ SSDataBlock* createDataBlock();
 int32_t      blockDataAppendColInfo(SSDataBlock* pBlock, SColumnInfoData* pColInfoData);

 SColumnInfoData  createColumnInfoData(int16_t type, int32_t bytes, int16_t colId);
-SColumnInfoData* bdGetColumnInfoData(SSDataBlock* pBlock, int32_t index);
+SColumnInfoData* bdGetColumnInfoData(const SSDataBlock* pBlock, int32_t index);

 void blockEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols, int8_t needCompress);
 const char* blockDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRows, const char* pData);
--- a/include/common/tmsg.h
+++ b/include/common/tmsg.h
@ -57,8 +57,8 @@ extern int32_t tMsgDict[];
 #define TMSG_SEG_SEQ(TYPE)  ((TYPE)&0xff)
 #define TMSG_INFO(TYPE)                                                                                             \
  ((TYPE) >= 0 &&                                                                                                   \
-   ((TYPE) < TDMT_DND_MAX_MSG | (TYPE) < TDMT_MND_MAX_MSG | (TYPE) < TDMT_VND_MAX_MSG | (TYPE) < TDMT_SCH_MAX_MSG | \
-    (TYPE) < TDMT_STREAM_MAX_MSG | (TYPE) < TDMT_MON_MAX_MSG | (TYPE) < TDMT_SYNC_MAX_MSG))                         \
+   ((TYPE) < TDMT_DND_MAX_MSG || (TYPE) < TDMT_MND_MAX_MSG || (TYPE) < TDMT_VND_MAX_MSG || (TYPE) < TDMT_SCH_MAX_MSG || \
+    (TYPE) < TDMT_STREAM_MAX_MSG || (TYPE) < TDMT_MON_MAX_MSG || (TYPE) < TDMT_SYNC_MAX_MSG))                         \
      ? tMsgInfo[tMsgDict[TMSG_SEG_CODE(TYPE)] + TMSG_SEG_SEQ(TYPE)]                                                \
      : 0
 #define TMSG_INDEX(TYPE) (tMsgDict[TMSG_SEG_CODE(TYPE)] + TMSG_SEG_SEQ(TYPE))
@ -169,6 +169,9 @@ typedef enum _mgmt_table {
 #define TD_CHILD_TABLE  TSDB_CHILD_TABLE
 #define TD_NORMAL_TABLE TSDB_NORMAL_TABLE

+#define TD_REQ_FROM_APP  0
+#define TD_REQ_FROM_TAOX 1
+
 typedef struct {
  int32_t vgId;
  char*   dbFName;
@ -432,25 +435,30 @@ static FORCE_INLINE int32_t tDecodeSSchemaWrapperEx(SDecoder* pDecoder, SSchemaW
 STSchema* tdGetSTSChemaFromSSChema(SSchema** pSchema, int32_t nCols);

 typedef struct {
-  char    name[TSDB_TABLE_FNAME_LEN];
-  int8_t  igExists;
-  int64_t delay1;
-  int64_t delay2;
-  int64_t watermark1;
-  int64_t watermark2;
-  int32_t ttl;
-  int32_t numOfColumns;
-  int32_t numOfTags;
-  int32_t numOfFuncs;
-  int32_t commentLen;
-  int32_t ast1Len;
-  int32_t ast2Len;
-  SArray* pColumns;  // array of SField
-  SArray* pTags;     // array of SField
-  SArray* pFuncs;
-  char*   pComment;
-  char*   pAst1;
-  char*   pAst2;
+  char     name[TSDB_TABLE_FNAME_LEN];
+  int8_t   igExists;
+  int8_t   source;  // 1-taosX or 0-taosClient
+  int8_t   reserved[6];
+  tb_uid_t suid;
+  int64_t  delay1;
+  int64_t  delay2;
+  int64_t  watermark1;
+  int64_t  watermark2;
+  int32_t  ttl;
+  int32_t  colVer;
+  int32_t  tagVer;
+  int32_t  numOfColumns;
+  int32_t  numOfTags;
+  int32_t  numOfFuncs;
+  int32_t  commentLen;
+  int32_t  ast1Len;
+  int32_t  ast2Len;
+  SArray*  pColumns;  // array of SField
+  SArray*  pTags;     // array of SField
+  SArray*  pFuncs;
+  char*    pComment;
+  char*    pAst1;
+  char*    pAst2;
 } SMCreateStbReq;

 int32_t tSerializeSMCreateStbReq(void* buf, int32_t bufLen, SMCreateStbReq* pReq);
@ -458,8 +466,11 @@ int32_t tDeserializeSMCreateStbReq(void* buf, int32_t bufLen, SMCreateStbReq* pR
 void    tFreeSMCreateStbReq(SMCreateStbReq* pReq);

 typedef struct {
-  char   name[TSDB_TABLE_FNAME_LEN];
-  int8_t igNotExists;
+  char     name[TSDB_TABLE_FNAME_LEN];
+  int8_t   igNotExists;
+  int8_t   source;  // 1-taosX or 0-taosClient
+  int8_t   reserved[6];
+  tb_uid_t suid;
 } SMDropStbReq;

 int32_t tSerializeSMDropStbReq(void* buf, int32_t bufLen, SMDropStbReq* pReq);
@ -468,8 +479,6 @@ int32_t tDeserializeSMDropStbReq(void* buf, int32_t bufLen, SMDropStbReq* pReq);
 typedef struct {
  char    name[TSDB_TABLE_FNAME_LEN];
  int8_t  alterType;
-  int32_t tagVer;
-  int32_t colVer;
  int32_t numOfFields;
  SArray* pFields;
  int32_t ttl;
@ -665,6 +674,7 @@ typedef struct {
  char    tbFName[TSDB_TABLE_FNAME_LEN];
  int32_t sversion;
  int32_t tversion;
+  int64_t affectedRows;
 } SQueryTableRsp;

 int32_t tSerializeSQueryTableRsp(void* buf, int32_t bufLen, SQueryTableRsp* pRsp);
@ -1510,6 +1520,7 @@ typedef struct SSubQueryMsg {
  int32_t  execId;
  int8_t   taskType;
  int8_t   explain;
+  int8_t   needFetch;
  uint32_t sqlLen;  // the query sql,
  uint32_t phyLen;
  char     msg[];
--- a/include/common/tname.h
+++ b/include/common/tname.h
@ -37,6 +37,8 @@ typedef struct SName {
  char    tname[TSDB_TABLE_NAME_LEN];
 } SName;

+SName* toName(int32_t acctId, const char* pDbName, const char* pTableName, SName* pName);
+
 int32_t tNameExtractFullName(const SName* name, char* dst);

 int32_t tNameLen(const SName* name);
--- a/include/common/ttokendef.h
+++ b/include/common/ttokendef.h
@ -79,195 +79,196 @@
 #define TK_EXISTS         61
 #define TK_BUFFER         62
 #define TK_CACHELAST      63
-#define TK_COMP           64
-#define TK_DURATION       65
-#define TK_NK_VARIABLE    66
-#define TK_FSYNC          67
-#define TK_MAXROWS        68
-#define TK_MINROWS        69
-#define TK_KEEP           70
-#define TK_PAGES          71
-#define TK_PAGESIZE       72
-#define TK_PRECISION      73
-#define TK_REPLICA        74
-#define TK_STRICT         75
-#define TK_WAL            76
-#define TK_VGROUPS        77
-#define TK_SINGLE_STABLE  78
-#define TK_RETENTIONS     79
-#define TK_SCHEMALESS     80
-#define TK_NK_COLON       81
-#define TK_TABLE          82
-#define TK_NK_LP          83
-#define TK_NK_RP          84
-#define TK_STABLE         85
-#define TK_ADD            86
-#define TK_COLUMN         87
-#define TK_MODIFY         88
-#define TK_RENAME         89
-#define TK_TAG            90
-#define TK_SET            91
-#define TK_NK_EQ          92
-#define TK_USING          93
-#define TK_TAGS           94
-#define TK_COMMENT        95
-#define TK_BOOL           96
-#define TK_TINYINT        97
-#define TK_SMALLINT       98
-#define TK_INT            99
-#define TK_INTEGER        100
-#define TK_BIGINT         101
-#define TK_FLOAT          102
-#define TK_DOUBLE         103
-#define TK_BINARY         104
-#define TK_TIMESTAMP      105
-#define TK_NCHAR          106
-#define TK_UNSIGNED       107
-#define TK_JSON           108
-#define TK_VARCHAR        109
-#define TK_MEDIUMBLOB     110
-#define TK_BLOB           111
-#define TK_VARBINARY      112
-#define TK_DECIMAL        113
-#define TK_MAX_DELAY      114
-#define TK_WATERMARK      115
-#define TK_ROLLUP         116
-#define TK_TTL            117
-#define TK_SMA            118
-#define TK_FIRST          119
-#define TK_LAST           120
-#define TK_SHOW           121
-#define TK_DATABASES      122
-#define TK_TABLES         123
-#define TK_STABLES        124
-#define TK_MNODES         125
-#define TK_MODULES        126
-#define TK_QNODES         127
-#define TK_FUNCTIONS      128
-#define TK_INDEXES        129
-#define TK_ACCOUNTS       130
-#define TK_APPS           131
-#define TK_CONNECTIONS    132
-#define TK_LICENCE        133
-#define TK_GRANTS         134
-#define TK_QUERIES        135
-#define TK_SCORES         136
-#define TK_TOPICS         137
-#define TK_VARIABLES      138
-#define TK_BNODES         139
-#define TK_SNODES         140
-#define TK_CLUSTER        141
-#define TK_TRANSACTIONS   142
-#define TK_DISTRIBUTED    143
-#define TK_CONSUMERS      144
-#define TK_SUBSCRIPTIONS  145
-#define TK_LIKE           146
-#define TK_INDEX          147
-#define TK_FUNCTION       148
-#define TK_INTERVAL       149
-#define TK_TOPIC          150
-#define TK_AS             151
-#define TK_WITH           152
-#define TK_META           153
-#define TK_CONSUMER       154
-#define TK_GROUP          155
-#define TK_DESC           156
-#define TK_DESCRIBE       157
-#define TK_RESET          158
-#define TK_QUERY          159
-#define TK_CACHE          160
-#define TK_EXPLAIN        161
-#define TK_ANALYZE        162
-#define TK_VERBOSE        163
-#define TK_NK_BOOL        164
-#define TK_RATIO          165
-#define TK_NK_FLOAT       166
-#define TK_COMPACT        167
-#define TK_VNODES         168
-#define TK_IN             169
-#define TK_OUTPUTTYPE     170
-#define TK_AGGREGATE      171
-#define TK_BUFSIZE        172
-#define TK_STREAM         173
-#define TK_INTO           174
-#define TK_TRIGGER        175
-#define TK_AT_ONCE        176
-#define TK_WINDOW_CLOSE   177
-#define TK_IGNORE         178
-#define TK_EXPIRED        179
-#define TK_KILL           180
-#define TK_CONNECTION     181
-#define TK_TRANSACTION    182
-#define TK_BALANCE        183
-#define TK_VGROUP         184
-#define TK_MERGE          185
-#define TK_REDISTRIBUTE   186
-#define TK_SPLIT          187
-#define TK_SYNCDB         188
-#define TK_DELETE         189
-#define TK_INSERT         190
-#define TK_NULL           191
-#define TK_NK_QUESTION    192
-#define TK_NK_ARROW       193
-#define TK_ROWTS          194
-#define TK_TBNAME         195
-#define TK_QSTARTTS       196
-#define TK_QENDTS         197
-#define TK_WSTARTTS       198
-#define TK_WENDTS         199
-#define TK_WDURATION      200
-#define TK_CAST           201
-#define TK_NOW            202
-#define TK_TODAY          203
-#define TK_TIMEZONE       204
-#define TK_CLIENT_VERSION 205
-#define TK_SERVER_VERSION 206
-#define TK_SERVER_STATUS  207
-#define TK_CURRENT_USER   208
-#define TK_COUNT          209
-#define TK_LAST_ROW       210
-#define TK_BETWEEN        211
-#define TK_IS             212
-#define TK_NK_LT          213
-#define TK_NK_GT          214
-#define TK_NK_LE          215
-#define TK_NK_GE          216
-#define TK_NK_NE          217
-#define TK_MATCH          218
-#define TK_NMATCH         219
-#define TK_CONTAINS       220
-#define TK_JOIN           221
-#define TK_INNER          222
-#define TK_SELECT         223
-#define TK_DISTINCT       224
-#define TK_WHERE          225
-#define TK_PARTITION      226
-#define TK_BY             227
-#define TK_SESSION        228
-#define TK_STATE_WINDOW   229
-#define TK_SLIDING        230
-#define TK_FILL           231
-#define TK_VALUE          232
-#define TK_NONE           233
-#define TK_PREV           234
-#define TK_LINEAR         235
-#define TK_NEXT           236
-#define TK_HAVING         237
-#define TK_RANGE          238
-#define TK_EVERY          239
-#define TK_ORDER          240
-#define TK_SLIMIT         241
-#define TK_SOFFSET        242
-#define TK_LIMIT          243
-#define TK_OFFSET         244
-#define TK_ASC            245
-#define TK_NULLS          246
-#define TK_ID             247
-#define TK_NK_BITNOT      248
-#define TK_VALUES         249
-#define TK_IMPORT         250
-#define TK_NK_SEMI        251
-#define TK_FILE           252
+#define TK_CACHELASTSIZE  64
+#define TK_COMP           65
+#define TK_DURATION       66
+#define TK_NK_VARIABLE    67
+#define TK_FSYNC          68
+#define TK_MAXROWS        69
+#define TK_MINROWS        70
+#define TK_KEEP           71
+#define TK_PAGES          72
+#define TK_PAGESIZE       73
+#define TK_PRECISION      74
+#define TK_REPLICA        75
+#define TK_STRICT         76
+#define TK_WAL            77
+#define TK_VGROUPS        78
+#define TK_SINGLE_STABLE  79
+#define TK_RETENTIONS     80
+#define TK_SCHEMALESS     81
+#define TK_NK_COLON       82
+#define TK_TABLE          83
+#define TK_NK_LP          84
+#define TK_NK_RP          85
+#define TK_STABLE         86
+#define TK_ADD            87
+#define TK_COLUMN         88
+#define TK_MODIFY         89
+#define TK_RENAME         90
+#define TK_TAG            91
+#define TK_SET            92
+#define TK_NK_EQ          93
+#define TK_USING          94
+#define TK_TAGS           95
+#define TK_COMMENT        96
+#define TK_BOOL           97
+#define TK_TINYINT        98
+#define TK_SMALLINT       99
+#define TK_INT            100
+#define TK_INTEGER        101
+#define TK_BIGINT         102
+#define TK_FLOAT          103
+#define TK_DOUBLE         104
+#define TK_BINARY         105
+#define TK_TIMESTAMP      106
+#define TK_NCHAR          107
+#define TK_UNSIGNED       108
+#define TK_JSON           109
+#define TK_VARCHAR        110
+#define TK_MEDIUMBLOB     111
+#define TK_BLOB           112
+#define TK_VARBINARY      113
+#define TK_DECIMAL        114
+#define TK_MAX_DELAY      115
+#define TK_WATERMARK      116
+#define TK_ROLLUP         117
+#define TK_TTL            118
+#define TK_SMA            119
+#define TK_FIRST          120
+#define TK_LAST           121
+#define TK_SHOW           122
+#define TK_DATABASES      123
+#define TK_TABLES         124
+#define TK_STABLES        125
+#define TK_MNODES         126
+#define TK_MODULES        127
+#define TK_QNODES         128
+#define TK_FUNCTIONS      129
+#define TK_INDEXES        130
+#define TK_ACCOUNTS       131
+#define TK_APPS           132
+#define TK_CONNECTIONS    133
+#define TK_LICENCE        134
+#define TK_GRANTS         135
+#define TK_QUERIES        136
+#define TK_SCORES         137
+#define TK_TOPICS         138
+#define TK_VARIABLES      139
+#define TK_BNODES         140
+#define TK_SNODES         141
+#define TK_CLUSTER        142
+#define TK_TRANSACTIONS   143
+#define TK_DISTRIBUTED    144
+#define TK_CONSUMERS      145
+#define TK_SUBSCRIPTIONS  146
+#define TK_LIKE           147
+#define TK_INDEX          148
+#define TK_FUNCTION       149
+#define TK_INTERVAL       150
+#define TK_TOPIC          151
+#define TK_AS             152
+#define TK_WITH           153
+#define TK_META           154
+#define TK_CONSUMER       155
+#define TK_GROUP          156
+#define TK_DESC           157
+#define TK_DESCRIBE       158
+#define TK_RESET          159
+#define TK_QUERY          160
+#define TK_CACHE          161
+#define TK_EXPLAIN        162
+#define TK_ANALYZE        163
+#define TK_VERBOSE        164
+#define TK_NK_BOOL        165
+#define TK_RATIO          166
+#define TK_NK_FLOAT       167
+#define TK_COMPACT        168
+#define TK_VNODES         169
+#define TK_IN             170
+#define TK_OUTPUTTYPE     171
+#define TK_AGGREGATE      172
+#define TK_BUFSIZE        173
+#define TK_STREAM         174
+#define TK_INTO           175
+#define TK_TRIGGER        176
+#define TK_AT_ONCE        177
+#define TK_WINDOW_CLOSE   178
+#define TK_IGNORE         179
+#define TK_EXPIRED        180
+#define TK_KILL           181
+#define TK_CONNECTION     182
+#define TK_TRANSACTION    183
+#define TK_BALANCE        184
+#define TK_VGROUP         185
+#define TK_MERGE          186
+#define TK_REDISTRIBUTE   187
+#define TK_SPLIT          188
+#define TK_SYNCDB         189
+#define TK_DELETE         190
+#define TK_INSERT         191
+#define TK_NULL           192
+#define TK_NK_QUESTION    193
+#define TK_NK_ARROW       194
+#define TK_ROWTS          195
+#define TK_TBNAME         196
+#define TK_QSTARTTS       197
+#define TK_QENDTS         198
+#define TK_WSTARTTS       199
+#define TK_WENDTS         200
+#define TK_WDURATION      201
+#define TK_CAST           202
+#define TK_NOW            203
+#define TK_TODAY          204
+#define TK_TIMEZONE       205
+#define TK_CLIENT_VERSION 206
+#define TK_SERVER_VERSION 207
+#define TK_SERVER_STATUS  208
+#define TK_CURRENT_USER   209
+#define TK_COUNT          210
+#define TK_LAST_ROW       211
+#define TK_BETWEEN        212
+#define TK_IS             213
+#define TK_NK_LT          214
+#define TK_NK_GT          215
+#define TK_NK_LE          216
+#define TK_NK_GE          217
+#define TK_NK_NE          218
+#define TK_MATCH          219
+#define TK_NMATCH         220
+#define TK_CONTAINS       221
+#define TK_JOIN           222
+#define TK_INNER          223
+#define TK_SELECT         224
+#define TK_DISTINCT       225
+#define TK_WHERE          226
+#define TK_PARTITION      227
+#define TK_BY             228
+#define TK_SESSION        229
+#define TK_STATE_WINDOW   230
+#define TK_SLIDING        231
+#define TK_FILL           232
+#define TK_VALUE          233
+#define TK_NONE           234
+#define TK_PREV           235
+#define TK_LINEAR         236
+#define TK_NEXT           237
+#define TK_HAVING         238
+#define TK_RANGE          239
+#define TK_EVERY          240
+#define TK_ORDER          241
+#define TK_SLIMIT         242
+#define TK_SOFFSET        243
+#define TK_LIMIT          244
+#define TK_OFFSET         245
+#define TK_ASC            246
+#define TK_NULLS          247
+#define TK_ID             248
+#define TK_NK_BITNOT      249
+#define TK_VALUES         250
+#define TK_IMPORT         251
+#define TK_NK_SEMI        252
+#define TK_FILE           253

 #define TK_NK_SPACE   300
 #define TK_NK_COMMENT 301
--- a/include/libs/executor/dataSinkMgt.h
+++ b/include/libs/executor/dataSinkMgt.h
@ -45,6 +45,10 @@ typedef struct SDeleterParam {
  SArray*  pUidList;
 } SDeleterParam;

+typedef struct SInserterParam {
+  SReadHandle* readHandle;
+} SInserterParam;
+
 typedef struct SDataSinkStat {
  uint64_t cachedSize;
 } SDataSinkStat;
@ -96,7 +100,7 @@ void dsEndPut(DataSinkHandle handle, uint64_t useconds);
 * @param handle
 * @param pLen data length
 */
-void dsGetDataLength(DataSinkHandle handle, int32_t* pLen, bool* pQueryEnd);
+void dsGetDataLength(DataSinkHandle handle, int64_t* pLen, bool* pQueryEnd);

 /**
 * Get data, the caller needs to allocate data memory.
--- a/include/libs/executor/executor.h
+++ b/include/libs/executor/executor.h
@ -157,7 +157,7 @@ int64_t qGetQueriedTableUid(qTaskInfo_t tinfo);
 */
 int32_t qGetQualifiedTableIdList(void* pTableList, const char* tagCond, int32_t tagCondLen, SArray* pTableIdList);

-void qProcessFetchRsp(void* parent, struct SRpcMsg* pMsg, struct SEpSet* pEpSet);
+void qProcessRspMsg(void* parent, struct SRpcMsg* pMsg, struct SEpSet* pEpSet);

 int32_t qGetExplainExecInfo(qTaskInfo_t tinfo, int32_t* resNum, SExplainExecInfo** pRes);

@ -174,7 +174,13 @@ int32_t qDeserializeTaskStatus(qTaskInfo_t tinfo, const char* pInput, int32_t le
 */
 int32_t qGetStreamScanStatus(qTaskInfo_t tinfo, uint64_t* uid, int64_t* ts);

-int32_t qStreamPrepareScan(qTaskInfo_t tinfo, uint64_t uid, int64_t ts);
+int32_t qStreamPrepareTsdbScan(qTaskInfo_t tinfo, uint64_t uid, int64_t ts);
+
+int32_t qStreamPrepareScan(qTaskInfo_t tinfo, const STqOffsetVal* pOffset);
+
+int32_t qStreamExtractOffset(qTaskInfo_t tinfo, STqOffsetVal* pOffset);
+
+void* qStreamExtractMetaMsg(qTaskInfo_t tinfo);

 void*   qExtractReaderFromStreamScanner(void* scanner);
 int32_t qExtractStreamScanner(qTaskInfo_t tinfo, void** scanner);
--- a/include/libs/function/function.h
+++ b/include/libs/function/function.h
@ -172,7 +172,13 @@ typedef struct tExprNode {

 void tExprTreeDestroy(tExprNode *pNode, void (*fp)(void *));

+typedef enum {
+  SHOULD_FREE_COLDATA    = 0x1,   // the newly created column data needs to be destroyed.
+  DELEGATED_MGMT_COLDATA = 0x2,   // input column data should not be released.
+} ECOLDATA_MGMT_TYPE_E;
+
 struct SScalarParam {
+  ECOLDATA_MGMT_TYPE_E type;
  SColumnInfoData *columnData;
  SHashObj        *pHashFilter;
  int32_t          hashValueType;
--- a/include/libs/nodes/cmdnodes.h
+++ b/include/libs/nodes/cmdnodes.h
@ -51,7 +51,8 @@ extern "C" {
 typedef struct SDatabaseOptions {
  ENodeType   type;
  int32_t     buffer;
-  int8_t      cachelast;
+  int8_t      cacheLast;
+  int32_t     cacheLastSize;
  int8_t      compressionLevel;
  int32_t     daysPerFile;
  SValueNode* pDaysPerFile;
--- a/include/libs/parser/parser.h
+++ b/include/libs/parser/parser.h
@ -96,6 +96,9 @@ int32_t smlBindData(void* handle, SArray* tags, SArray* colsSchema, SArray* cols
                    char* tableName, char* msgBuf, int16_t msgBufLen);
 int32_t smlBuildOutput(void* handle, SHashObj* pVgHash);

+int32_t rewriteToVnodeModifyOpStmt(SQuery* pQuery, SArray* pBufArray);
+SArray* serializeVgroupsCreateTableBatch(SHashObj* pVgroupHashmap);
+SArray* serializeVgroupsDropTableBatch(SHashObj* pVgroupHashmap);
 #ifdef __cplusplus
 }
 #endif
--- a/include/libs/qworker/qworker.h
+++ b/include/libs/qworker/qworker.h
@ -67,7 +67,7 @@ int32_t qWorkerProcessCQueryMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, in

 int32_t qWorkerProcessFetchMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts);

-int32_t qWorkerProcessFetchRsp(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts);
+int32_t qWorkerProcessRspMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts);

 int32_t qWorkerProcessCancelMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts);

--- a/include/libs/sync/sync.h
+++ b/include/libs/sync/sync.h
@ -163,9 +163,6 @@ typedef struct SSyncLogStore {
  // return commit index of log
  SyncIndex (*getCommitIndex)(struct SSyncLogStore* pLogStore);

-  // refactor, log[0 .. n] ==> log[m .. n]
-  // int32_t (*syncLogSetBeginIndex)(struct SSyncLogStore* pLogStore, SyncIndex beginIndex);
-
  SyncIndex (*syncLogBeginIndex)(struct SSyncLogStore* pLogStore);
  SyncIndex (*syncLogEndIndex)(struct SSyncLogStore* pLogStore);
  bool (*syncLogIsEmpty)(struct SSyncLogStore* pLogStore);
--- a/include/libs/transport/trpc.h
+++ b/include/libs/transport/trpc.h
@ -27,7 +27,7 @@ extern "C" {

 #define TAOS_CONN_SERVER 0
 #define TAOS_CONN_CLIENT 1
-#define IsReq(pMsg) (pMsg->msgType & 1U)
+#define IsReq(pMsg)      (pMsg->msgType & 1U)

 extern int32_t tsRpcHeadSize;

--- a/include/libs/wal/wal.h
+++ b/include/libs/wal/wal.h
@ -24,42 +24,14 @@
 extern "C" {
 #endif

-#define wFatal(...)                                              \
-  {                                                              \
-    if (wDebugFlag & DEBUG_FATAL) {                              \
-      taosPrintLog("WAL FATAL ", DEBUG_FATAL, 255, __VA_ARGS__); \
-    }                                                            \
-  }
-#define wError(...)                                              \
-  {                                                              \
-    if (wDebugFlag & DEBUG_ERROR) {                              \
-      taosPrintLog("WAL ERROR ", DEBUG_ERROR, 255, __VA_ARGS__); \
-    }                                                            \
-  }
-#define wWarn(...)                                             \
-  {                                                            \
-    if (wDebugFlag & DEBUG_WARN) {                             \
-      taosPrintLog("WAL WARN ", DEBUG_WARN, 255, __VA_ARGS__); \
-    }                                                          \
-  }
-#define wInfo(...)                                        \
-  {                                                       \
-    if (wDebugFlag & DEBUG_INFO) {                        \
-      taosPrintLog("WAL ", DEBUG_INFO, 255, __VA_ARGS__); \
-    }                                                     \
-  }
-#define wDebug(...)                                               \
-  {                                                               \
-    if (wDebugFlag & DEBUG_DEBUG) {                               \
-      taosPrintLog("WAL ", DEBUG_DEBUG, wDebugFlag, __VA_ARGS__); \
-    }                                                             \
-  }
-#define wTrace(...)                                               \
-  {                                                               \
-    if (wDebugFlag & DEBUG_TRACE) {                               \
-      taosPrintLog("WAL ", DEBUG_TRACE, wDebugFlag, __VA_ARGS__); \
-    }                                                             \
-  }
+// clang-format off
+#define wFatal(...) { if (wDebugFlag & DEBUG_FATAL) { taosPrintLog("WAL FATAL ", DEBUG_FATAL, 255,        __VA_ARGS__); }}
+#define wError(...) { if (wDebugFlag & DEBUG_ERROR) { taosPrintLog("WAL ERROR ", DEBUG_ERROR, 255,        __VA_ARGS__); }}
+#define wWarn(...)  { if (wDebugFlag & DEBUG_WARN)  { taosPrintLog("WAL WARN ",  DEBUG_WARN, 255,         __VA_ARGS__); }}
+#define wInfo(...)  { if (wDebugFlag & DEBUG_INFO)  { taosPrintLog("WAL ",       DEBUG_INFO, 255,         __VA_ARGS__); }}
+#define wDebug(...) { if (wDebugFlag & DEBUG_DEBUG) { taosPrintLog("WAL ",       DEBUG_DEBUG, wDebugFlag, __VA_ARGS__); }}
+#define wTrace(...) { if (wDebugFlag & DEBUG_TRACE) { taosPrintLog("WAL ",       DEBUG_TRACE, wDebugFlag, __VA_ARGS__); }}
+// clang-format on

 #define WAL_PROTO_VER    0
 #define WAL_NOSUFFIX_LEN 20
@ -161,6 +133,7 @@ typedef struct {
  int64_t        curFileFirstVer;
  int64_t        curVersion;
  int64_t        capacity;
+  int8_t         curInvalid;
  TdThreadMutex  mutex;
  SWalFilterCond cond;
  SWalCkHead    *pHead;
@ -194,6 +167,7 @@ int32_t walRestoreFromSnapshot(SWal *, int64_t ver);
 SWalReader *walOpenReader(SWal *, SWalFilterCond *pCond);
 void        walCloseReader(SWalReader *pRead);
 int32_t     walReadVer(SWalReader *pRead, int64_t ver);
+int32_t     walReadSeekVer(SWalReader *pRead, int64_t ver);
 int32_t     walNextValidMsg(SWalReader *pRead);

 // only for tq usage
--- a/include/util/taoserror.h
+++ b/include/util/taoserror.h
@ -247,9 +247,11 @@ int32_t* taosGetErrno();
 #define TSDB_CODE_MND_COLUMN_NOT_EXIST          TAOS_DEF_ERROR_CODE(0, 0x03AE)
 #define TSDB_CODE_MND_FIELD_CONFLICT_WITH_TOPIC TAOS_DEF_ERROR_CODE(0, 0x03AF)
 #define TSDB_CODE_MND_SINGLE_STB_MODE_DB        TAOS_DEF_ERROR_CODE(0, 0x03B0)
+#define TSDB_CODE_MND_INVALID_SCHEMA_VER        TAOS_DEF_ERROR_CODE(0, 0x03B1)
+#define TSDB_CODE_MND_STABLE_UID_NOT_MATCH      TAOS_DEF_ERROR_CODE(0, 0x03B2)

 // mnode-infoSchema
-#define TSDB_CODE_MND_INVALID_SYS_TABLENAME         TAOS_DEF_ERROR_CODE(0, 0x03BA)
+#define TSDB_CODE_MND_INVALID_SYS_TABLENAME     TAOS_DEF_ERROR_CODE(0, 0x03BA)

 // mnode-func
 #define TSDB_CODE_MND_FUNC_ALREADY_EXIST        TAOS_DEF_ERROR_CODE(0, 0x03C0)
--- a/include/util/tencode.h
+++ b/include/util/tencode.h
@ -477,22 +477,6 @@ static FORCE_INLINE void* tDecoderMalloc(SDecoder* pCoder, int32_t size) {
    return n;                          \
  } while (0)

-#define tGetV(p, v)                              \
-  do {                                           \
-    int32_t n = 0;                               \
-    if (v) *v = 0;                               \
-    for (;;) {                                   \
-      if (p[n] <= 0x7f) {                        \
-        if (v) (*v) |= (p[n] << (7 * n));        \
-        n++;                                     \
-        break;                                   \
-      }                                          \
-      if (v) (*v) |= ((p[n] & 0x7f) << (7 * n)); \
-      n++;                                       \
-    }                                            \
-    return n;                                    \
-  } while (0)
-
 // PUT
 static FORCE_INLINE int32_t tPutU8(uint8_t* p, uint8_t v) {
  if (p) ((uint8_t*)p)[0] = v;
@ -607,7 +591,22 @@ static FORCE_INLINE int32_t tGetI64(uint8_t* p, int64_t* v) {
  return sizeof(int64_t);
 }

-static FORCE_INLINE int32_t tGetU16v(uint8_t* p, uint16_t* v) { tGetV(p, v); }
+static FORCE_INLINE int32_t tGetU16v(uint8_t* p, uint16_t* v) {
+  int32_t n = 0;
+
+  if (v) *v = 0;
+  for (;;) {
+    if (p[n] <= 0x7f) {
+      if (v) (*v) |= (((uint16_t)p[n]) << (7 * n));
+      n++;
+      break;
+    }
+    if (v) (*v) |= (((uint16_t)(p[n] & 0x7f)) << (7 * n));
+    n++;
+  }
+
+  return n;
+}

 static FORCE_INLINE int32_t tGetI16v(uint8_t* p, int16_t* v) {
  int32_t  n;
@ -619,7 +618,22 @@ static FORCE_INLINE int32_t tGetI16v(uint8_t* p, int16_t* v) {
  return n;
 }

-static FORCE_INLINE int32_t tGetU32v(uint8_t* p, uint32_t* v) { tGetV(p, v); }
+static FORCE_INLINE int32_t tGetU32v(uint8_t* p, uint32_t* v) {
+  int32_t n = 0;
+
+  if (v) *v = 0;
+  for (;;) {
+    if (p[n] <= 0x7f) {
+      if (v) (*v) |= (((uint32_t)p[n]) << (7 * n));
+      n++;
+      break;
+    }
+    if (v) (*v) |= (((uint32_t)(p[n] & 0x7f)) << (7 * n));
+    n++;
+  }
+
+  return n;
+}

 static FORCE_INLINE int32_t tGetI32v(uint8_t* p, int32_t* v) {
  int32_t  n;
@ -631,7 +645,22 @@ static FORCE_INLINE int32_t tGetI32v(uint8_t* p, int32_t* v) {
  return n;
 }

-static FORCE_INLINE int32_t tGetU64v(uint8_t* p, uint64_t* v) { tGetV(p, v); }
+static FORCE_INLINE int32_t tGetU64v(uint8_t* p, uint64_t* v) {
+  int32_t n = 0;
+
+  if (v) *v = 0;
+  for (;;) {
+    if (p[n] <= 0x7f) {
+      if (v) (*v) |= (((uint64_t)p[n]) << (7 * n));
+      n++;
+      break;
+    }
+    if (v) (*v) |= (((uint64_t)(p[n] & 0x7f)) << (7 * n));
+    n++;
+  }
+
+  return n;
+}

 static FORCE_INLINE int32_t tGetI64v(uint8_t* p, int64_t* v) {
  int32_t  n;
--- a/include/util/tlog.h
+++ b/include/util/tlog.h
@ -60,7 +60,7 @@ extern int32_t tsdbDebugFlag;
 extern int32_t tqDebugFlag;
 extern int32_t fsDebugFlag;
 extern int32_t metaDebugFlag;
-extern int32_t fnDebugFlag;
+extern int32_t udfDebugFlag;
 extern int32_t smaDebugFlag;
 extern int32_t idxDebugFlag;

@ -94,7 +94,7 @@ void taosPrintLongString(const char *flags, ELogLevel level, int32_t dflag, cons
 #define pError(...) { taosPrintLog("APP ERROR ", DEBUG_ERROR, 255, __VA_ARGS__); }
 #define pPrint(...) { taosPrintLog("APP ", DEBUG_INFO, 255, __VA_ARGS__); }
 // clang-format on
-
+#define BUF_PAGE_DEBUG
 #ifdef __cplusplus
 }
 #endif
--- a/packaging/deb/DEBIAN/preinst
+++ b/packaging/deb/DEBIAN/preinst
@ -37,4 +37,5 @@ if [ -f "${install_main_dir}/taosadapter.service" ]; then
 fi

 # there can not libtaos.so*, otherwise ln -s  error
-${csudo}rm -f ${install_main_dir}/driver/libtaos*   || :
+${csudo}rm -f ${install_main_dir}/driver/libtaos.*   || :
+${csudo}rm -f ${install_main_dir}/driver/libtaosws.*   || :
--- a/packaging/deb/DEBIAN/prerm
+++ b/packaging/deb/DEBIAN/prerm
@ -29,8 +29,12 @@ else
    ${csudo}rm -f ${bin_link_dir}/taosdemo   || :
    ${csudo}rm -f ${cfg_link_dir}/*          || :
    ${csudo}rm -f ${inc_link_dir}/taos.h     || :
+    ${csudo}rm -f ${inc_link_dir}/taosdef.h     || :
+    ${csudo}rm -f ${inc_link_dir}/taoserror.h     || :
    ${csudo}rm -f ${inc_link_dir}/taosudf.h     || :
+    ${csudo}rm -f ${inc_link_dir}/taosws.h     || :
    ${csudo}rm -f ${lib_link_dir}/libtaos.*  || :
+    ${csudo}rm -f ${lib_link_dir}/libtaosws.*  || :

    ${csudo}rm -f ${log_link_dir}            || :
    ${csudo}rm -f ${data_link_dir}           || :
--- a/packaging/deb/makedeb.sh
+++ b/packaging/deb/makedeb.sh
@ -30,6 +30,7 @@ mkdir -p ${pkg_dir}
 cd ${pkg_dir}

 libfile="libtaos.so.${tdengine_ver}"
+wslibfile="libtaosws.so"

 # create install dir
 install_home_path="/usr/local/taos"
@ -67,10 +68,12 @@ fi

 cp ${compile_dir}/build/bin/taos                    ${pkg_dir}${install_home_path}/bin
 cp ${compile_dir}/build/lib/${libfile}              ${pkg_dir}${install_home_path}/driver
+cp ${compile_dir}/build/lib/${wslibfile}            ${pkg_dir}${install_home_path}/driver ||:
 cp ${compile_dir}/../include/client/taos.h          ${pkg_dir}${install_home_path}/include
 cp ${compile_dir}/../include/common/taosdef.h       ${pkg_dir}${install_home_path}/include
 cp ${compile_dir}/../include/util/taoserror.h       ${pkg_dir}${install_home_path}/include
 cp ${compile_dir}/../include/libs/function/taosudf.h       ${pkg_dir}${install_home_path}/include
+cp ${compile_dir}/../src/inc/taosws.h               ${pkg_dir}${install_home_path}/include ||:
 cp -r ${top_dir}/examples/*                         ${pkg_dir}${install_home_path}/examples
 #cp -r ${top_dir}/src/connector/python               ${pkg_dir}${install_home_path}/connector
 #cp -r ${top_dir}/src/connector/go                   ${pkg_dir}${install_home_path}/connector
--- a/packaging/rpm/tdengine.spec
+++ b/packaging/rpm/tdengine.spec
@ -42,6 +42,7 @@ echo version: %{_version}
 echo buildroot: %{buildroot}

 libfile="libtaos.so.%{_version}"
+wslibfile="libtaosws.so"

 # create install path, and cp file
 mkdir -p %{buildroot}%{homepath}/bin
@ -74,10 +75,12 @@ if [ -f %{_compiledir}/build/bin/taosadapter ]; then
    cp %{_compiledir}/build/bin/taosadapter                    %{buildroot}%{homepath}/bin ||:
 fi
 cp %{_compiledir}/build/lib/${libfile}              %{buildroot}%{homepath}/driver
+cp %{_compiledir}/build/lib/${wslibfile}            %{buildroot}%{homepath}/driver ||:
 cp %{_compiledir}/../include/client/taos.h          %{buildroot}%{homepath}/include
 cp %{_compiledir}/../include/common/taosdef.h       %{buildroot}%{homepath}/include
 cp %{_compiledir}/../include/util/taoserror.h       %{buildroot}%{homepath}/include
 cp %{_compiledir}/../include/libs/function/taosudf.h       %{buildroot}%{homepath}/include
+cp %{_compiledir}/../src/inc/taosws.h               %{buildroot}%{homepath}/include ||:
 #cp -r %{_compiledir}/../src/connector/python        %{buildroot}%{homepath}/connector
 #cp -r %{_compiledir}/../src/connector/go            %{buildroot}%{homepath}/connector
 #cp -r %{_compiledir}/../src/connector/nodejs        %{buildroot}%{homepath}/connector
--- a/packaging/tools/install.sh
+++ b/packaging/tools/install.sh
@ -30,6 +30,7 @@ configDir="/etc/taos"
 installDir="/usr/local/taos"
 adapterName="taosadapter"
 benchmarkName="taosBenchmark"
+tmqName="tmq_sim"
 dumpName="taosdump"
 demoName="taosdemo"

@ -205,6 +206,7 @@ function install_bin() {
  [ -x ${install_main_dir}/bin/${adapterName} ] && ${csudo}ln -s ${install_main_dir}/bin/${adapterName} ${bin_link_dir}/${adapterName} || :
  [ -x ${install_main_dir}/bin/${benchmarkName} ] && ${csudo}ln -s ${install_main_dir}/bin/${benchmarkName} ${bin_link_dir}/${demoName} || :
  [ -x ${install_main_dir}/bin/${benchmarkName} ] && ${csudo}ln -s ${install_main_dir}/bin/${benchmarkName} ${bin_link_dir}/${benchmarkName} || :
+  [ -x ${install_main_dir}/bin/${tmqName} ] && ${csudo}ln -s ${install_main_dir}/bin/${tmqName} ${bin_link_dir}/${tmqName} || :
  [ -x ${install_main_dir}/bin/${dumpName} ] && ${csudo}ln -s ${install_main_dir}/bin/${dumpName} ${bin_link_dir}/${dumpName} || :
  [ -x ${install_main_dir}/bin/TDinsight.sh ] && ${csudo}ln -s ${install_main_dir}/bin/TDinsight.sh ${bin_link_dir}/TDinsight.sh || :
  [ -x ${install_main_dir}/bin/remove.sh ] && ${csudo}ln -s ${install_main_dir}/bin/remove.sh ${bin_link_dir}/${uninstallScript} || :
@ -227,9 +229,13 @@ function install_lib() {
  ${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
  ${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so

+  ${csudo}ln -s ${lib_link_dir}/libtaosws.so ${lib_link_dir}/libtaosws.so || :
+
  if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.so ]]; then
    ${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
    ${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
+
+    ${csudo}ln -s ${lib64_link_dir}/libtaosws.so ${lib64_link_dir}/libtaosws.so || :
  fi

  ${csudo}ldconfig
@ -313,11 +319,16 @@ function install_jemalloc() {

 function install_header() {
  ${csudo}rm -f ${inc_link_dir}/taos.h ${inc_link_dir}/taosdef.h ${inc_link_dir}/taoserror.h ${inc_link_dir}/taosudf.h || :
+
+  ${csudo}rm -f ${inc_link_dir}/taosws.h || :
+
  ${csudo}cp -f ${script_dir}/inc/* ${install_main_dir}/include && ${csudo}chmod 644 ${install_main_dir}/include/*
  ${csudo}ln -s ${install_main_dir}/include/taos.h ${inc_link_dir}/taos.h
  ${csudo}ln -s ${install_main_dir}/include/taosdef.h ${inc_link_dir}/taosdef.h
  ${csudo}ln -s ${install_main_dir}/include/taoserror.h ${inc_link_dir}/taoserror.h
  ${csudo}ln -s ${install_main_dir}/include/taosudf.h ${inc_link_dir}/taosudf.h  
+
+  ${csudo}ln -s ${install_main_dir}/include/taosws.h ${inc_link_dir}/taosws.h || :
 }

 function add_newHostname_to_hosts() {
--- a/packaging/tools/make_install.sh
+++ b/packaging/tools/make_install.sh
@ -294,21 +294,29 @@ function install_avro() {
 function install_lib() {
  # Remove links
  ${csudo}rm -f ${lib_link_dir}/libtaos.* || :
+  ${csudo}rm -f ${lib_link_dir}/libtaosws.* || :
  if [ "$osType" != "Darwin" ]; then
    ${csudo}rm -f ${lib64_link_dir}/libtaos.* || :
+    ${csudo}rm -f ${lib64_link_dir}/libtaosws.* || :
  fi

  if [ "$osType" != "Darwin" ]; then
    ${csudo}cp ${binary_dir}/build/lib/libtaos.so.${verNumber} \
      ${install_main_dir}/driver &&
-      ${csudo}chmod 777 ${install_main_dir}/driver/*
+      ${csudo}chmod 777 ${install_main_dir}/driver/libtaos.so.${verNumber}
+
+    ${csudo}cp ${binary_dir}/build/lib/libtaosws.so \
+      ${install_main_dir}/driver &&
+      ${csudo}chmod 777 ${install_main_dir}/driver/libtaosws.so

    ${csudo}ln -sf ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
    ${csudo}ln -sf ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
+    ${csudo}ln -sf ${install_main_dir}/driver/libtaosws.so ${lib_link_dir}/libtaosws.so || :

    if [ -d "${lib64_link_dir}" ]; then
      ${csudo}ln -sf ${install_main_dir}/driver/libtaos.* ${lib64_link_dir}/libtaos.so.1
      ${csudo}ln -sf ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so
+      ${csudo}ln -sf ${lib64_link_dir}/libtaosws.so ${lib64_link_dir}/libtaosws.so || :
    fi
  else
    ${csudo}cp -Rf ${binary_dir}/build/lib/libtaos.${verNumber}.dylib \
@ -337,8 +345,8 @@ function install_lib() {
  fi

  install_jemalloc
-  install_avro lib
-  install_avro lib64
+  #install_avro lib
+  #install_avro lib64

  if [ "$osType" != "Darwin" ]; then
    ${csudo}ldconfig
@ -350,11 +358,19 @@ function install_header() {
  if [ "$osType" != "Darwin" ]; then
    ${csudo}rm -f ${inc_link_dir}/taos.h ${inc_link_dir}/taosdef.h ${inc_link_dir}/taoserror.h ${inc_link_dir}/taosudf.h || :
    ${csudo}cp -f ${source_dir}/include/client/taos.h ${source_dir}/include/common/taosdef.h ${source_dir}/include/util/taoserror.h ${source_dir}/include/libs/function/taosudf.h \
+    ${csudo}rm -f ${inc_link_dir}/taosws.h || :
+
+    ${csudo}cp -f ${source_dir}/src/inc/taos.h ${source_dir}/src/inc/taosdef.h ${source_dir}/src/inc/taoserror.h \
      ${install_main_dir}/include && ${csudo}chmod 644 ${install_main_dir}/include/*
+
+    ${csudo}cp -f ${binary_dir}/build/include/taosws.h ${install_main_dir}/include && ${csudo}chmod 644 ${install_main_dir}/include/taosws.h
+
    ${csudo}ln -s ${install_main_dir}/include/taos.h ${inc_link_dir}/taos.h
    ${csudo}ln -s ${install_main_dir}/include/taosdef.h ${inc_link_dir}/taosdef.h
    ${csudo}ln -s ${install_main_dir}/include/taoserror.h ${inc_link_dir}/taoserror.h
    ${csudo}ln -s ${install_main_dir}/include/taosudf.h ${inc_link_dir}/taosudf.h
+
+    ${csudo}ln -s ${install_main_dir}/include/taosws.h ${inc_link_dir}/taosws.h || :
  else
    ${csudo}cp -f ${source_dir}/include/client/taos.h ${source_dir}/include/common/taosdef.h ${source_dir}/include/util/taoserror.h ${source_dir}/include/libs/function/taosudf.h \
      ${install_main_dir}/include ||
--- a/packaging/tools/makepkg.sh
+++ b/packaging/tools/makepkg.sh
@ -60,7 +60,7 @@ if [ "$pagMode" == "lite" ]; then
  strip ${build_dir}/bin/${serverName}
  strip ${build_dir}/bin/${clientName}
  # lite version doesn't include taosadapter,  which will lead to no restful interface
-  bin_files="${build_dir}/bin/${serverName} ${build_dir}/bin/${clientName} ${script_dir}/remove.sh ${script_dir}/startPre.sh ${build_dir}/bin/taosBenchmark"
+  bin_files="${build_dir}/bin/${serverName} ${build_dir}/bin/${clientName} ${script_dir}/remove.sh ${script_dir}/startPre.sh ${build_dir}/bin/taosBenchmark  ${build_dir}/bin/tmq_sim"
  taostools_bin_files=""
 else

@ -78,6 +78,7 @@ else

  taostools_bin_files=" ${build_dir}/bin/taosdump \
      ${build_dir}/bin/taosBenchmark \
+      ${build_dir}/bin/tmq_sim \
      ${build_dir}/bin/TDinsight.sh \
      $tdinsight_caches"

@ -92,8 +93,11 @@ else
 fi

 lib_files="${build_dir}/lib/libtaos.so.${version}"
+wslib_files="${build_dir}/lib/libtaosws.so."
 header_files="${code_dir}/include/client/taos.h ${code_dir}/include/common/taosdef.h ${code_dir}/include/util/taoserror.h ${code_dir}/include/libs/function/taosudf.h"

+wsheader_files="${code_dir}/inc/taosws.h"
+
 if [ "$dbName" != "taos" ]; then
  cfg_dir="${top_dir}/../enterprise/packaging/cfg"
 else
@ -109,6 +113,9 @@ init_file_rpm=${script_dir}/../rpm/taosd
 # make directories.
 mkdir -p ${install_dir}
 mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
+
+${wsheader_files} ${install_dir}/inc || :
+
 mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/${configFile} ${install_dir}/cfg/${configFile}

 if [ -f "${compile_dir}/test/cfg/taosadapter.toml" ]; then
@ -283,6 +290,7 @@ fi

 # Copy driver
 mkdir -p ${install_dir}/driver && cp ${lib_files} ${install_dir}/driver && echo "${versionComp}" >${install_dir}/driver/vercomp.txt
+cp ${wslib_files} ${install_dir}/driver || :

 # Copy connector
 if [ "$verMode" == "cluster" ]; then
--- a/packaging/tools/remove.sh
+++ b/packaging/tools/remove.sh
@ -102,7 +102,10 @@ function clean_local_bin() {
 function clean_lib() {
  # Remove link
  ${csudo}rm -f ${lib_link_dir}/libtaos.* || :
+  ${csudo}rm -f ${lib_link_dir}/libtaosws.* || :
+
  ${csudo}rm -f ${lib64_link_dir}/libtaos.* || :
+  ${csudo}rm -f ${lib64_link_dir}/libtaosws.* || :
  #${csudo}rm -rf ${v15_java_app_dir}           || :
 }

@ -111,6 +114,8 @@ function clean_header() {
  ${csudo}rm -f ${inc_link_dir}/taos.h || :
  ${csudo}rm -f ${inc_link_dir}/taosdef.h || :
  ${csudo}rm -f ${inc_link_dir}/taoserror.h || :
+
+  ${csudo}rm -f ${inc_link_dir}/taosws.h || :
 }

 function clean_config() {
--- a/source/client/src/TMQConnector.c
+++ b/source/client/src/TMQConnector.c
@ -324,9 +324,7 @@ JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_tmq_TMQConnector_fetchRawBlockImp(
  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetNumOfRowsFp, (jint)numOfRows);
  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetNumOfColsFp, (jint)numOfFields);

-  char   *chars = (char *)data;
-  int32_t len = chars[0] + (chars[1] << 8) + (chars[2] << 16) + (chars[3] << 24);
-  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetByteArrayFp, len, jniFromNCharToByteArray(env, (char *)data, len));
-
+  int32_t len = *(int32_t *)data;
+  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetByteArrayFp, jniFromNCharToByteArray(env, (char *)data, len));
  return JNI_SUCCESS;
 }
--- a/source/client/src/TSDBJNIConnector.c
+++ b/source/client/src/TSDBJNIConnector.c
@ -592,8 +592,7 @@ JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_fetchBlockImp(JNI
  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetNumOfRowsFp, (jint)numOfRows);
  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetNumOfColsFp, (jint)numOfFields);

-  char   *chars = (char *)data;
-  int32_t len = chars[0] + (chars[1] << 8) + (chars[2] << 16) + (chars[3] << 24);
+  int32_t len = *(int32_t *)data;
  (*env)->CallVoidMethod(env, rowobj, g_blockdataSetByteArrayFp, jniFromNCharToByteArray(env, (char *)data, len));

  return JNI_SUCCESS;
--- a/source/client/src/clientImpl.c
+++ b/source/client/src/clientImpl.c
@ -325,11 +325,13 @@ int32_t updateQnodeList(SAppInstInfo* pInfo, SArray* pNodeList) {
  if (pInfo->pQnodeList) {
    taosArrayDestroy(pInfo->pQnodeList);
    pInfo->pQnodeList = NULL;
+    tscDebug("QnodeList cleared in cluster 0x%" PRIx64, pInfo->clusterId);
  }

  if (pNodeList) {
    pInfo->pQnodeList = taosArrayDup(pNodeList);
    taosArraySort(pInfo->pQnodeList, compareQueryNodeLoad);
+    tscDebug("QnodeList updated in cluster 0x%" PRIx64 ", num:%d", pInfo->clusterId, taosArrayGetSize(pInfo->pQnodeList));
  }
  taosThreadMutexUnlock(&pInfo->qnodeMutex);

--- a/source/client/src/clientSml.c
+++ b/source/client/src/clientSml.c
@ -1570,41 +1570,40 @@ static int32_t smlParseTSFromJSONObj(SSmlHandle *info, cJSON *root, int64_t *tsV
    return TSDB_CODE_TSC_INVALID_TIME_STAMP;
  }

+  *tsVal = timeDouble;
  size_t typeLen = strlen(type->valuestring);
  if (typeLen == 1 && (type->valuestring[0] == 's' || type->valuestring[0] == 'S')) {
    // seconds
-    timeDouble = timeDouble * 1e9;
+    *tsVal = *tsVal * NANOSECOND_PER_SEC;
+    timeDouble = timeDouble * NANOSECOND_PER_SEC;
    if (smlDoubleToInt64OverFlow(timeDouble)) {
      smlBuildInvalidDataMsg(&info->msgBuf, "timestamp is too large", NULL);
      return TSDB_CODE_TSC_INVALID_TIME_STAMP;
    }
-    *tsVal = timeDouble;
  } else if (typeLen == 2 && (type->valuestring[1] == 's' || type->valuestring[1] == 'S')) {
    switch (type->valuestring[0]) {
      case 'm':
      case 'M':
        // milliseconds
-        timeDouble = timeDouble * 1e6;
+        *tsVal = *tsVal * NANOSECOND_PER_MSEC;
+        timeDouble = timeDouble * NANOSECOND_PER_MSEC;
        if (smlDoubleToInt64OverFlow(timeDouble)) {
          smlBuildInvalidDataMsg(&info->msgBuf, "timestamp is too large", NULL);
          return TSDB_CODE_TSC_INVALID_TIME_STAMP;
        }
-        *tsVal = timeDouble;
        break;
      case 'u':
      case 'U':
        // microseconds
-        timeDouble = timeDouble * 1e3;
+        *tsVal = *tsVal * NANOSECOND_PER_USEC;
+        timeDouble = timeDouble * NANOSECOND_PER_USEC;
        if (smlDoubleToInt64OverFlow(timeDouble)) {
          smlBuildInvalidDataMsg(&info->msgBuf, "timestamp is too large", NULL);
          return TSDB_CODE_TSC_INVALID_TIME_STAMP;
        }
-        *tsVal = timeDouble;
        break;
      case 'n':
      case 'N':
-        // nanoseconds
-        *tsVal = timeDouble;
        break;
      default:
        return TSDB_CODE_TSC_INVALID_JSON;
@ -1641,21 +1640,23 @@ static int32_t smlParseTSFromJSON(SSmlHandle *info, cJSON *root, SArray *cols) {
    if (timeDouble < 0) {
      return TSDB_CODE_TSC_INVALID_TIME_STAMP;
    }
+
    uint8_t tsLen = smlGetTimestampLen((int64_t)timeDouble);
+    tsVal = (int64_t)timeDouble;
    if (tsLen == TSDB_TIME_PRECISION_SEC_DIGITS) {
-      timeDouble = timeDouble * 1e9;
+      tsVal = tsVal * NANOSECOND_PER_SEC;
+      timeDouble = timeDouble * NANOSECOND_PER_SEC;
      if (smlDoubleToInt64OverFlow(timeDouble)) {
        smlBuildInvalidDataMsg(&info->msgBuf, "timestamp is too large", NULL);
        return TSDB_CODE_TSC_INVALID_TIME_STAMP;
      }
-      tsVal = timeDouble;
    } else if (tsLen == TSDB_TIME_PRECISION_MILLI_DIGITS) {
-      timeDouble = timeDouble * 1e6;
+      tsVal = tsVal * NANOSECOND_PER_MSEC;
+      timeDouble = timeDouble * NANOSECOND_PER_MSEC;
      if (smlDoubleToInt64OverFlow(timeDouble)) {
        smlBuildInvalidDataMsg(&info->msgBuf, "timestamp is too large", NULL);
        return TSDB_CODE_TSC_INVALID_TIME_STAMP;
      }
-      tsVal = timeDouble;
    } else if (timeDouble == 0) {
      tsVal = taosGetTimestampNs();
    } else {
--- a/source/client/src/tmq.c
+++ b/source/client/src/tmq.c
@ -106,6 +106,12 @@ struct tmq_t {
  tsem_t rspSem;
 };

+struct tmq_raw_data{
+  void    *raw_meta;
+  int32_t raw_meta_len;
+  int16_t raw_meta_type;
+};
+
 enum {
  TMQ_VG_STATUS__IDLE = 0,
  TMQ_VG_STATUS__WAIT,
@ -396,7 +402,7 @@ int32_t tmqCommitCb2(void* param, SDataBuf* pBuf, int32_t code) {
  }
 #endif

-  /*tscDebug("receive offset commit cb of %s on vg %d, offset is %ld", pParam->pOffset->subKey, pParam->->vgId,
+  /*tscDebug("receive offset commit cb of %s on vgId:%d, offset is %" PRId64, pParam->pOffset->subKey, pParam->->vgId,
   * pOffset->version);*/

  // count down waiting rsp
@ -471,8 +477,8 @@ static int32_t tmqSendCommitReq(tmq_t* tmq, SMqClientVg* pVg, SMqClientTopic* pT
      .handle = NULL,
  };

-  tscDebug("consumer %ld commit offset of %s on vg %d, offset is %ld", tmq->consumerId, pOffset->subKey, pVg->vgId,
-           pOffset->val.version);
+  tscDebug("consumer:%" PRId64 ", commit offset of %s on vgId:%d, offset is %" PRId64, tmq->consumerId, pOffset->subKey,
+           pVg->vgId, pOffset->val.version);

  // TODO: put into cb
  pVg->committedOffsetNew = pVg->currentOffsetNew;
@ -585,13 +591,14 @@ int32_t tmqCommitInner2(tmq_t* tmq, const TAOS_RES* msg, int8_t automatic, int8_
  for (int32_t i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) {
    SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i);

-    tscDebug("consumer %ld begin commit for topic %s, vgNum %d", tmq->consumerId, pTopic->topicName,
+    tscDebug("consumer:%" PRId64 ", begin commit for topic %s, vgNum %d", tmq->consumerId, pTopic->topicName,
             (int32_t)taosArrayGetSize(pTopic->vgs));

    for (int32_t j = 0; j < taosArrayGetSize(pTopic->vgs); j++) {
      SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j);

-      tscDebug("consumer %ld begin commit for topic %s, vgId %d", tmq->consumerId, pTopic->topicName, pVg->vgId);
+      tscDebug("consumer:%" PRId64 ", begin commit for topic %s, vgId:%d", tmq->consumerId, pTopic->topicName,
+               pVg->vgId);

      if (pVg->currentOffsetNew.type > 0 && !tOffsetEqual(&pVg->currentOffsetNew, &pVg->committedOffsetNew)) {
        if (tmqSendCommitReq(tmq, pVg, pTopic, pParamSet) < 0) {
@ -1082,13 +1089,13 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {
  int32_t         epoch = pParam->epoch;
  taosMemoryFree(pParam);
  if (code != 0) {
-    tscWarn("msg discard from vg %d, epoch %d, code:%x", vgId, epoch, code);
+    tscWarn("msg discard from vgId:%d, epoch %d, code:%x", vgId, epoch, code);
    if (pMsg->pData) taosMemoryFree(pMsg->pData);
    if (code == TSDB_CODE_TQ_NO_COMMITTED_OFFSET) {
      SMqPollRspWrapper* pRspWrapper = taosAllocateQitem(sizeof(SMqPollRspWrapper), DEF_QITEM);
      if (pRspWrapper == NULL) {
        taosMemoryFree(pMsg->pData);
-        tscWarn("msg discard from vg %d, epoch %d since out of memory", vgId, epoch);
+        tscWarn("msg discard from vgId:%d, epoch %d since out of memory", vgId, epoch);
        goto CREATE_MSG_FAIL;
      }
      pRspWrapper->tmqRspType = TMQ_MSG_TYPE__END_RSP;
@ -1104,7 +1111,7 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {
  int32_t tmqEpoch = atomic_load_32(&tmq->epoch);
  if (msgEpoch < tmqEpoch) {
    // do not write into queue since updating epoch reset
-    tscWarn("msg discard from vg %d since from earlier epoch, rsp epoch %d, current epoch %d", vgId, msgEpoch,
+    tscWarn("msg discard from vgId:%d since from earlier epoch, rsp epoch %d, current epoch %d", vgId, msgEpoch,
            tmqEpoch);
    tsem_post(&tmq->rspSem);
    taosMemoryFree(pMsg->pData);
@ -1112,7 +1119,7 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {
  }

  if (msgEpoch != tmqEpoch) {
-    tscWarn("mismatch rsp from vg %d, epoch %d, current epoch %d", vgId, msgEpoch, tmqEpoch);
+    tscWarn("mismatch rsp from vgId:%d, epoch %d, current epoch %d", vgId, msgEpoch, tmqEpoch);
  }

  // handle meta rsp
@ -1121,7 +1128,7 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {
  SMqPollRspWrapper* pRspWrapper = taosAllocateQitem(sizeof(SMqPollRspWrapper), DEF_QITEM);
  if (pRspWrapper == NULL) {
    taosMemoryFree(pMsg->pData);
-    tscWarn("msg discard from vg %d, epoch %d since out of memory", vgId, epoch);
+    tscWarn("msg discard from vgId:%d, epoch %d since out of memory", vgId, epoch);
    goto CREATE_MSG_FAIL;
  }

@ -1143,8 +1150,9 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {

  taosMemoryFree(pMsg->pData);

-  tscDebug("consumer %ld recv poll: vg %d, req offset %ld, rsp offset %ld, type %d", tmq->consumerId, pVg->vgId,
-           pRspWrapper->dataRsp.reqOffset.version, pRspWrapper->dataRsp.rspOffset.version, rspType);
+  tscDebug("consumer:%" PRId64 ", recv poll: vgId:%d, req offset %" PRId64 ", rsp offset %" PRId64 " type %d",
+           tmq->consumerId, pVg->vgId, pRspWrapper->dataRsp.reqOffset.version, pRspWrapper->dataRsp.rspOffset.version,
+           rspType);

  taosWriteQitem(tmq->mqueue, pRspWrapper);
  tsem_post(&tmq->rspSem);
@ -1163,7 +1171,7 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {

  int32_t topicNumGet = taosArrayGetSize(pRsp->topics);
  char    vgKey[TSDB_TOPIC_FNAME_LEN + 22];
-  tscDebug("consumer %ld update ep epoch %d to epoch %d, topic num: %d", tmq->consumerId, tmq->epoch, epoch,
+  tscDebug("consumer:%" PRId64 ", update ep epoch %d to epoch %d, topic num:%d", tmq->consumerId, tmq->epoch, epoch,
           topicNumGet);

  SArray* newTopics = taosArrayInit(topicNumGet, sizeof(SMqClientTopic));
@ -1182,14 +1190,14 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
    SMqClientTopic* pTopicCur = taosArrayGet(tmq->clientTopics, i);
    if (pTopicCur->vgs) {
      int32_t vgNumCur = taosArrayGetSize(pTopicCur->vgs);
-      tscDebug("consumer %ld new vg num: %d", tmq->consumerId, vgNumCur);
+      tscDebug("consumer:%" PRId64 ", new vg num: %d", tmq->consumerId, vgNumCur);
      for (int32_t j = 0; j < vgNumCur; j++) {
        SMqClientVg* pVgCur = taosArrayGet(pTopicCur->vgs, j);
        sprintf(vgKey, "%s:%d", pTopicCur->topicName, pVgCur->vgId);
        char buf[50];
        tFormatOffset(buf, 50, &pVgCur->currentOffsetNew);
-        tscDebug("consumer %ld epoch %d vg %d vgKey is %s, offset is %s", tmq->consumerId, epoch, pVgCur->vgId, vgKey,
-                 buf);
+        tscDebug("consumer:%" PRId64 ", epoch %d vgId:%d vgKey is %s, offset is %s", tmq->consumerId, epoch, pVgCur->vgId,
+                 vgKey, buf);
        taosHashPut(pHash, vgKey, strlen(vgKey), &pVgCur->currentOffsetNew, sizeof(STqOffsetVal));
      }
    }
@ -1202,7 +1210,7 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
    topic.topicName = strdup(pTopicEp->topic);
    tstrncpy(topic.db, pTopicEp->db, TSDB_DB_FNAME_LEN);

-    tscDebug("consumer %ld update topic: %s", tmq->consumerId, topic.topicName);
+    tscDebug("consumer:%" PRId64 ", update topic: %s", tmq->consumerId, topic.topicName);

    int32_t vgNumGet = taosArrayGetSize(pTopicEp->vgs);
    topic.vgs = taosArrayInit(vgNumGet, sizeof(SMqClientVg));
@ -1215,7 +1223,8 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
        offsetNew = *pOffset;
      }

-      /*tscDebug("consumer %ld(epoch %d) offset of vg %d updated to %ld, vgKey is %s", tmq->consumerId, epoch,*/
+      /*tscDebug("consumer:%" PRId64 ", (epoch %d) offset of vgId:%d updated to %" PRId64 ", vgKey is %s",
+       * tmq->consumerId, epoch,*/
      /*pVgEp->vgId, offset, vgKey);*/
      SMqClientVg clientVg = {
          .pollCnt = 0,
@ -1249,7 +1258,7 @@ bool tmqUpdateEp(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
  bool    set = false;
  int32_t topicNumGet = taosArrayGetSize(pRsp->topics);
  char    vgKey[TSDB_TOPIC_FNAME_LEN + 22];
-  tscDebug("consumer %ld update ep epoch %d to epoch %d, topic num: %d", tmq->consumerId, tmq->epoch, epoch,
+  tscDebug("consumer:%" PRId64 ", update ep epoch %d to epoch %d, topic num: %d", tmq->consumerId, tmq->epoch, epoch,
           topicNumGet);
  SArray* newTopics = taosArrayInit(topicNumGet, sizeof(SMqClientTopic));
  if (newTopics == NULL) {
@ -1270,19 +1279,19 @@ bool tmqUpdateEp(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
    topic.topicName = strdup(pTopicEp->topic);
    tstrncpy(topic.db, pTopicEp->db, TSDB_DB_FNAME_LEN);

-    tscDebug("consumer %ld update topic: %s", tmq->consumerId, topic.topicName);
+    tscDebug("consumer:%" PRId64 ", update topic: %s", tmq->consumerId, topic.topicName);
    int32_t topicNumCur = taosArrayGetSize(tmq->clientTopics);
    for (int32_t j = 0; j < topicNumCur; j++) {
      // find old topic
      SMqClientTopic* pTopicCur = taosArrayGet(tmq->clientTopics, j);
      if (pTopicCur->vgs && strcmp(pTopicCur->topicName, pTopicEp->topic) == 0) {
        int32_t vgNumCur = taosArrayGetSize(pTopicCur->vgs);
-        tscDebug("consumer %ld new vg num: %d", tmq->consumerId, vgNumCur);
+        tscDebug("consumer:%" PRId64 ", new vg num: %d", tmq->consumerId, vgNumCur);
        if (vgNumCur == 0) break;
        for (int32_t k = 0; k < vgNumCur; k++) {
          SMqClientVg* pVgCur = taosArrayGet(pTopicCur->vgs, k);
          sprintf(vgKey, "%s:%d", topic.topicName, pVgCur->vgId);
-          tscDebug("consumer %ld epoch %d vg %d build %s", tmq->consumerId, epoch, pVgCur->vgId, vgKey);
+          tscDebug("consumer:%" PRId64 ", epoch %d vgId:%d build %s", tmq->consumerId, epoch, pVgCur->vgId, vgKey);
          taosHashPut(pHash, vgKey, strlen(vgKey), &pVgCur->currentOffset, sizeof(int64_t));
        }
        break;
@ -1296,13 +1305,13 @@ bool tmqUpdateEp(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
      sprintf(vgKey, "%s:%d", topic.topicName, pVgEp->vgId);
      int64_t* pOffset = taosHashGet(pHash, vgKey, strlen(vgKey));
      int64_t  offset = pVgEp->offset;
-      tscDebug("consumer %ld(epoch %d) original offset of vg %d is %ld", tmq->consumerId, epoch, pVgEp->vgId, offset);
+      tscDebug("consumer:%" PRId64 ", (epoch %d) original offset of vgId:%d is %" PRId64, tmq->consumerId, epoch, pVgEp->vgId, offset);
      if (pOffset != NULL) {
        offset = *pOffset;
-        tscDebug("consumer %ld(epoch %d) receive offset of vg %d, full key is %s", tmq->consumerId, epoch, pVgEp->vgId,
+        tscDebug("consumer:%" PRId64 ", (epoch %d) receive offset of vgId:%d, full key is %s", tmq->consumerId, epoch, pVgEp->vgId,
                 vgKey);
      }
-      tscDebug("consumer %ld(epoch %d) offset of vg %d updated to %ld", tmq->consumerId, epoch, pVgEp->vgId, offset);
+      tscDebug("consumer:%" PRId64 ", (epoch %d) offset of vgId:%d updated to %" PRId64, tmq->consumerId, epoch, pVgEp->vgId, offset);
      SMqClientVg clientVg = {
          .pollCnt = 0,
          .currentOffset = offset,
@ -1336,7 +1345,7 @@ int32_t tmqAskEpCb(void* param, SDataBuf* pMsg, int32_t code) {
  int8_t           async = pParam->async;
  pParam->code = code;
  if (code != 0) {
-    tscError("consumer %ld get topic endpoint error, not ready, wait:%d", tmq->consumerId, pParam->async);
+    tscError("consumer:%" PRId64 ", get topic endpoint error, not ready, wait:%d", tmq->consumerId, pParam->async);
    goto END;
  }

@ -1345,7 +1354,7 @@ int32_t tmqAskEpCb(void* param, SDataBuf* pMsg, int32_t code) {
  // Epoch will only increase when received newer epoch ep msg
  SMqRspHead* head = pMsg->pData;
  int32_t     epoch = atomic_load_32(&tmq->epoch);
-  tscDebug("consumer %ld recv ep, msg epoch %d, current epoch %d", tmq->consumerId, head->epoch, epoch);
+  tscDebug("consumer:%" PRId64 ", recv ep, msg epoch %d, current epoch %d", tmq->consumerId, head->epoch, epoch);
  if (head->epoch <= epoch) {
    goto END;
  }
@ -1353,8 +1362,8 @@ int32_t tmqAskEpCb(void* param, SDataBuf* pMsg, int32_t code) {
  if (!async) {
    SMqAskEpRsp rsp;
    tDecodeSMqAskEpRsp(POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), &rsp);
-    /*printf("rsp epoch %ld sz %ld\n", rsp.epoch, rsp.topics->size);*/
-    /*printf("tmq epoch %ld sz %ld\n", tmq->epoch, tmq->clientTopics->size);*/
+    /*printf("rsp epoch %" PRId64 " sz %" PRId64 "\n", rsp.epoch, rsp.topics->size);*/
+    /*printf("tmq epoch %" PRId64 " sz %" PRId64 "\n", tmq->epoch, tmq->clientTopics->size);*/
    tmqUpdateEp2(tmq, head->epoch, &rsp);
    tDeleteSMqAskEpRsp(&rsp);
  } else {
@ -1389,7 +1398,7 @@ int32_t tmqAskEp(tmq_t* tmq, bool async) {
  int8_t  epStatus = atomic_val_compare_exchange_8(&tmq->epStatus, 0, 1);
  if (epStatus == 1) {
    int32_t epSkipCnt = atomic_add_fetch_32(&tmq->epSkipCnt, 1);
-    tscTrace("consumer %ld skip ask ep cnt %d", tmq->consumerId, epSkipCnt);
+    tscTrace("consumer:%" PRId64 ", skip ask ep cnt %d", tmq->consumerId, epSkipCnt);
    if (epSkipCnt < 5000) return 0;
  }
  atomic_store_32(&tmq->epSkipCnt, 0);
@ -1439,7 +1448,7 @@ int32_t tmqAskEp(tmq_t* tmq, bool async) {

  SEpSet epSet = getEpSet_s(&tmq->pTscObj->pAppInfo->mgmtEp);

-  tscDebug("consumer %ld ask ep", tmq->consumerId);
+  tscDebug("consumer:%" PRId64 ", ask ep", tmq->consumerId);

  int64_t transporterId = 0;
  asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo);
@ -1555,14 +1564,14 @@ int32_t tmqPollImpl(tmq_t* tmq, int64_t timeout) {
      int32_t      vgStatus = atomic_val_compare_exchange_32(&pVg->vgStatus, TMQ_VG_STATUS__IDLE, TMQ_VG_STATUS__WAIT);
      if (vgStatus != TMQ_VG_STATUS__IDLE) {
        int32_t vgSkipCnt = atomic_add_fetch_32(&pVg->vgSkipCnt, 1);
-        tscTrace("consumer %ld epoch %d skip vg %d skip cnt %d", tmq->consumerId, tmq->epoch, pVg->vgId, vgSkipCnt);
+        tscTrace("consumer:%" PRId64 ", epoch %d skip vgId:%d skip cnt %d", tmq->consumerId, tmq->epoch, pVg->vgId, vgSkipCnt);
        continue;
        /*if (vgSkipCnt < 10000) continue;*/
 #if 0
        if (skipCnt < 30000) {
          continue;
        } else {
-        tscDebug("consumer %ld skip vg %d skip too much reset", tmq->consumerId, pVg->vgId);
+        tscDebug("consumer:%" PRId64 ",skip vgId:%d skip too much reset", tmq->consumerId, pVg->vgId);
        }
 #endif
      }
@ -1611,9 +1620,9 @@ int32_t tmqPollImpl(tmq_t* tmq, int64_t timeout) {

      char offsetFormatBuf[80];
      tFormatOffset(offsetFormatBuf, 80, &pVg->currentOffsetNew);
-      tscDebug("consumer %ld send poll to %s : vg %d, epoch %d, req offset %s, reqId %lu", tmq->consumerId,
+      tscDebug("consumer:%" PRId64 ", send poll to %s vgId:%d, epoch %d, req offset:%s, reqId:%" PRIu64, tmq->consumerId,
               pTopic->topicName, pVg->vgId, tmq->epoch, offsetFormatBuf, pReq->reqId);
-      /*printf("send vg %d %ld\n", pVg->vgId, pVg->currentOffset);*/
+      /*printf("send vgId:%d %" PRId64 "\n", pVg->vgId, pVg->currentOffset);*/
      asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &pVg->epSet, &transporterId, sendInfo);
      pVg->pollCnt++;
      tmq->pollCnt++;
@ -1660,7 +1669,7 @@ void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout, bool pollIfReset) {
      int32_t consumerEpoch = atomic_load_32(&tmq->epoch);
      if (pollRspWrapper->dataRsp.head.epoch == consumerEpoch) {
        SMqClientVg* pVg = pollRspWrapper->vgHandle;
-        /*printf("vg %d offset %ld up to %ld\n", pVg->vgId, pVg->currentOffset, rspMsg->msg.rspOffset);*/
+        /*printf("vgId:%d offset %" PRId64 " up to %" PRId64 "\n", pVg->vgId, pVg->currentOffset, rspMsg->msg.rspOffset);*/
        pVg->currentOffsetNew = pollRspWrapper->dataRsp.rspOffset;
        atomic_store_32(&pVg->vgStatus, TMQ_VG_STATUS__IDLE);
        if (pollRspWrapper->dataRsp.blockNum == 0) {
@ -1682,7 +1691,7 @@ void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout, bool pollIfReset) {
      int32_t            consumerEpoch = atomic_load_32(&tmq->epoch);
      if (pollRspWrapper->metaRsp.head.epoch == consumerEpoch) {
        SMqClientVg* pVg = pollRspWrapper->vgHandle;
-        /*printf("vg %d offset %ld up to %ld\n", pVg->vgId, pVg->currentOffset, rspMsg->msg.rspOffset);*/
+        /*printf("vgId:%d offset %" PRId64 " up to %" PRId64 "\n", pVg->vgId, pVg->currentOffset, rspMsg->msg.rspOffset);*/
        pVg->currentOffsetNew.version = pollRspWrapper->metaRsp.rspOffset;
        pVg->currentOffsetNew.type = TMQ_OFFSET__LOG;
        atomic_store_32(&pVg->vgStatus, TMQ_VG_STATUS__IDLE);
@ -1701,7 +1710,7 @@ void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout, bool pollIfReset) {
      tmqHandleNoPollRsp(tmq, rspWrapper, &reset);
      taosFreeQitem(rspWrapper);
      if (pollIfReset && reset) {
-        tscDebug("consumer %ld reset and repoll", tmq->consumerId);
+        tscDebug("consumer:%" PRId64 ", reset and repoll", tmq->consumerId);
        tmqPollImpl(tmq, timeout);
      }
    }
@ -1741,7 +1750,7 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) {
      int64_t endTime = taosGetTimestampMs();
      int64_t leftTime = endTime - startTime;
      if (leftTime > timeout) {
-        tscDebug("consumer %ld (epoch %d) timeout, no rsp", tmq->consumerId, tmq->epoch);
+        tscDebug("consumer:%" PRId64 ", (epoch %d) timeout, no rsp", tmq->consumerId, tmq->epoch);
        return NULL;
      }
      tsem_timewait(&tmq->rspSem, leftTime * 1000);
@ -1839,14 +1848,16 @@ const char* tmq_get_table_name(TAOS_RES* res) {
  return NULL;
 }

-int32_t tmq_get_raw_meta(TAOS_RES* res, void** raw_meta, int32_t* raw_meta_len) {
+tmq_raw_data *tmq_get_raw_meta(TAOS_RES* res) {
  if (TD_RES_TMQ_META(res)) {
+    tmq_raw_data *raw = taosMemoryCalloc(1, sizeof(tmq_raw_data));
    SMqMetaRspObj* pMetaRspObj = (SMqMetaRspObj*)res;
-    *raw_meta = pMetaRspObj->metaRsp.metaRsp;
-    *raw_meta_len = pMetaRspObj->metaRsp.metaRspLen;
-    return 0;
+    raw->raw_meta = pMetaRspObj->metaRsp.metaRsp;
+    raw->raw_meta_len = pMetaRspObj->metaRsp.metaRspLen;
+    raw->raw_meta_type = pMetaRspObj->metaRsp.resMsgType;
+    return raw;
  }
-  return -1;
+  return NULL;
 }

 static char *buildCreateTableJson(SSchemaWrapper *schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id, int8_t t){
@ -2249,6 +2260,515 @@ char *tmq_get_json_meta(TAOS_RES *res){
  return NULL;
 }

+static int32_t taosCreateStb(TAOS *taos, void *meta, int32_t metaLen){
+  SVCreateStbReq req = {0};
+  SDecoder       coder;
+  SMCreateStbReq pReq = {0};
+  int32_t code = TSDB_CODE_SUCCESS;
+  SRequestObj* pRequest = NULL;
+
+  STscObj *pTscObj = acquireTscObj(*(int64_t *)taos);
+  if (NULL == pTscObj) {
+    code = TSDB_CODE_TSC_DISCONNECTED;
+    goto end;
+  }
+
+  code = buildRequest(pTscObj, "", 0, &pRequest);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  if(!pRequest->pDb){
+    code = TSDB_CODE_PAR_DB_NOT_SPECIFIED;
+    goto end;
+  }
+  // decode and process req
+  void* data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  int32_t len = metaLen - sizeof(SMsgHead);
+  tDecoderInit(&coder, data, len);
+  if (tDecodeSVCreateStbReq(&coder, &req) < 0) {
+    code = TSDB_CODE_INVALID_PARA;
+    goto end;
+  }
+  // build create stable
+  pReq.pColumns = taosArrayInit(req.schemaRow.nCols, sizeof(SField));
+  for(int32_t i = 0; i < req.schemaRow.nCols; i++){
+    SSchema* pSchema = req.schemaRow.pSchema + i;
+    SField   field   = {.type = pSchema->type, .bytes = pSchema->bytes};
+    strcpy(field.name, pSchema->name);
+    taosArrayPush(pReq.pColumns, &field);
+  }
+  pReq.pTags = taosArrayInit(req.schemaTag.nCols, sizeof(SField));
+  for(int32_t i = 0; i < req.schemaTag.nCols; i++){
+    SSchema* pSchema = req.schemaTag.pSchema + i;
+    SField   field   = {.type = pSchema->type, .bytes = pSchema->bytes};
+    strcpy(field.name, pSchema->name);
+    taosArrayPush(pReq.pTags, &field);
+  }
+  pReq.colVer = req.schemaRow.version;
+  pReq.tagVer = req.schemaTag.version;
+  pReq.numOfColumns = req.schemaRow.nCols;
+  pReq.numOfTags = req.schemaTag.nCols;
+  pReq.commentLen = -1;
+  pReq.suid = req.suid;
+  pReq.source = 1;
+
+  SName tableName;
+  tNameExtractFullName(toName(pTscObj->acctId, pRequest->pDb, req.name, &tableName), pReq.name);
+
+  SCmdMsgInfo pCmdMsg = {0};
+  pCmdMsg.epSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp);
+  pCmdMsg.msgType = TDMT_MND_CREATE_STB;
+  pCmdMsg.msgLen = tSerializeSMCreateStbReq(NULL, 0, &pReq);
+  pCmdMsg.pMsg = taosMemoryMalloc(pCmdMsg.msgLen);
+  if (NULL == pCmdMsg.pMsg) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+  tSerializeSMCreateStbReq(pCmdMsg.pMsg, pCmdMsg.msgLen, &pReq);
+
+  SQuery      pQuery = {0};
+  pQuery.execMode = QUERY_EXEC_MODE_RPC;
+  pQuery.pCmdMsg = &pCmdMsg;
+  pQuery.msgType = pQuery.pCmdMsg->msgType;
+  pQuery.stableQuery = true;
+
+  launchQueryImpl(pRequest, &pQuery, true, NULL);
+  code = pRequest->code;
+  taosMemoryFree(pCmdMsg.pMsg);
+
+  end:
+  destroyRequest(pRequest);
+  tFreeSMCreateStbReq(&pReq);
+  tDecoderClear(&coder);
+  return code;
+}
+
+static int32_t taosDropStb(TAOS *taos, void *meta, int32_t metaLen){
+  SVDropStbReq req = {0};
+  SDecoder     coder;
+  SMDropStbReq pReq = {0};
+  int32_t code = TSDB_CODE_SUCCESS;
+  SRequestObj* pRequest = NULL;
+
+  STscObj *pTscObj = acquireTscObj(*(int64_t *)taos);
+  if (NULL == pTscObj) {
+    code = TSDB_CODE_TSC_DISCONNECTED;
+    goto end;
+  }
+
+  code = buildRequest(pTscObj, "", 0, &pRequest);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  if(!pRequest->pDb){
+    code = TSDB_CODE_PAR_DB_NOT_SPECIFIED;
+    goto end;
+  }
+  // decode and process req
+  void* data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  int32_t len = metaLen - sizeof(SMsgHead);
+  tDecoderInit(&coder, data, len);
+  if (tDecodeSVDropStbReq(&coder, &req) < 0) {
+    code = TSDB_CODE_INVALID_PARA;
+    goto end;
+  }
+
+  // build drop stable
+  pReq.igNotExists = true;
+  pReq.source = 1;
+  pReq.suid = req.suid;
+  SName tableName;
+  tNameExtractFullName(toName(pTscObj->acctId, pRequest->pDb, req.name, &tableName), pReq.name);
+
+  SCmdMsgInfo pCmdMsg = {0};
+  pCmdMsg.epSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp);
+  pCmdMsg.msgType = TDMT_MND_DROP_STB;
+  pCmdMsg.msgLen = tSerializeSMDropStbReq(NULL, 0, &pReq);
+  pCmdMsg.pMsg = taosMemoryMalloc(pCmdMsg.msgLen);
+  if (NULL == pCmdMsg.pMsg) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+  tSerializeSMDropStbReq(pCmdMsg.pMsg, pCmdMsg.msgLen, &pReq);
+
+  SQuery      pQuery = {0};
+  pQuery.execMode = QUERY_EXEC_MODE_RPC;
+  pQuery.pCmdMsg = &pCmdMsg;
+  pQuery.msgType = pQuery.pCmdMsg->msgType;
+  pQuery.stableQuery = true;
+
+  launchQueryImpl(pRequest, &pQuery, true, NULL);
+  code = pRequest->code;
+  taosMemoryFree(pCmdMsg.pMsg);
+
+  end:
+  destroyRequest(pRequest);
+  tDecoderClear(&coder);
+  return code;
+}
+
+typedef struct SVgroupCreateTableBatch {
+  SVCreateTbBatchReq req;
+  SVgroupInfo        info;
+  char               dbName[TSDB_DB_NAME_LEN];
+} SVgroupCreateTableBatch;
+
+static void destroyCreateTbReqBatch(void* data) {
+  SVgroupCreateTableBatch* pTbBatch = (SVgroupCreateTableBatch*) data;
+  taosArrayDestroy(pTbBatch->req.pArray);
+}
+
+static int32_t taosCreateTable(TAOS *taos, void *meta, int32_t metaLen){
+  SVCreateTbBatchReq  req             = {0};
+  SDecoder            coder           = {0};
+  int32_t             code            = TSDB_CODE_SUCCESS;
+  SRequestObj        *pRequest        = NULL;
+  SQuery             *pQuery          = NULL;
+  SHashObj           *pVgroupHashmap  = NULL;
+  STscObj            *pTscObj         = acquireTscObj(*(int64_t *)taos);
+
+  if (NULL == pTscObj) {
+    code = TSDB_CODE_TSC_DISCONNECTED;
+    goto end;
+  }
+
+  code = buildRequest(pTscObj, "", 0, &pRequest);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  if(!pRequest->pDb){
+    code = TSDB_CODE_PAR_DB_NOT_SPECIFIED;
+    goto end;
+  }
+  // decode and process req
+  void* data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  int32_t len = metaLen - sizeof(SMsgHead);
+  tDecoderInit(&coder, data, len);
+  if (tDecodeSVCreateTbBatchReq(&coder, &req) < 0) {
+    code = TSDB_CODE_INVALID_PARA;
+    goto end;
+  }
+
+  SVCreateTbReq     *pCreateReq = NULL;
+  SCatalog* pCatalog = NULL;
+  code = catalogGetHandle(pTscObj->pAppInfo->clusterId, &pCatalog);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  pVgroupHashmap = taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), false, HASH_NO_LOCK);
+  if (NULL == pVgroupHashmap) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+  taosHashSetFreeFp(pVgroupHashmap, destroyCreateTbReqBatch);
+
+  SRequestConnInfo conn = {.pTrans = pTscObj->pAppInfo->pTransporter,
+      .requestId = pRequest->requestId,
+      .requestObjRefId = pRequest->self,
+      .mgmtEps = getEpSet_s(&pTscObj->pAppInfo->mgmtEp)};
+  // loop to create table
+  for (int32_t iReq = 0; iReq < req.nReqs; iReq++) {
+    pCreateReq = req.pReqs + iReq;
+
+    SVgroupInfo pInfo = {0};
+    SName pName;
+    toName(pTscObj->acctId, pRequest->pDb, pCreateReq->name, &pName);
+    code = catalogGetTableHashVgroup(pCatalog, &conn, &pName, &pInfo);
+    if (code != TSDB_CODE_SUCCESS) {
+      goto end;
+    }
+
+    SVgroupCreateTableBatch* pTableBatch = taosHashGet(pVgroupHashmap, &pInfo.vgId, sizeof(pInfo.vgId));
+    if (pTableBatch == NULL) {
+      SVgroupCreateTableBatch tBatch = {0};
+      tBatch.info = pInfo;
+      strcpy(tBatch.dbName, pRequest->pDb);
+
+      tBatch.req.pArray = taosArrayInit(4, sizeof(struct SVCreateTbReq));
+      taosArrayPush(tBatch.req.pArray, pCreateReq);
+
+      taosHashPut(pVgroupHashmap, &pInfo.vgId, sizeof(pInfo.vgId), &tBatch, sizeof(tBatch));
+    } else {  // add to the correct vgroup
+      taosArrayPush(pTableBatch->req.pArray, pCreateReq);
+    }
+  }
+
+  SArray* pBufArray = serializeVgroupsCreateTableBatch(pVgroupHashmap);
+  if (NULL == pBufArray) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+
+  pQuery = (SQuery*)nodesMakeNode(QUERY_NODE_QUERY);
+  pQuery->execMode = QUERY_EXEC_MODE_SCHEDULE;
+  pQuery->msgType = TDMT_VND_CREATE_TABLE;
+  pQuery->stableQuery = false;
+  pQuery->pRoot   = nodesMakeNode(QUERY_NODE_CREATE_TABLE_STMT);
+
+  code = rewriteToVnodeModifyOpStmt(pQuery, pBufArray);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  launchQueryImpl(pRequest, pQuery, false, NULL);
+  pQuery  = NULL;          // no need to free in the end
+  code    = pRequest->code;
+
+  end:
+  taosHashCleanup(pVgroupHashmap);
+  destroyRequest(pRequest);
+  tDecoderClear(&coder);
+  qDestroyQuery(pQuery);
+  return code;
+}
+
+typedef struct SVgroupDropTableBatch {
+  SVDropTbBatchReq req;
+  SVgroupInfo      info;
+  char             dbName[TSDB_DB_NAME_LEN];
+} SVgroupDropTableBatch;
+
+static void destroyDropTbReqBatch(void* data) {
+  SVgroupDropTableBatch* pTbBatch = (SVgroupDropTableBatch*)data;
+  taosArrayDestroy(pTbBatch->req.pArray);
+}
+
+static int32_t taosDropTable(TAOS *taos, void *meta, int32_t metaLen){
+  SVDropTbBatchReq    req             = {0};
+  SDecoder            coder           = {0};
+  int32_t             code            = TSDB_CODE_SUCCESS;
+  SRequestObj        *pRequest        = NULL;
+  SQuery             *pQuery          = NULL;
+  SHashObj           *pVgroupHashmap  = NULL;
+  STscObj            *pTscObj         = acquireTscObj(*(int64_t *)taos);
+
+  if (NULL == pTscObj) {
+    code = TSDB_CODE_TSC_DISCONNECTED;
+    goto end;
+  }
+
+  code = buildRequest(pTscObj, "", 0, &pRequest);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  if(!pRequest->pDb){
+    code = TSDB_CODE_PAR_DB_NOT_SPECIFIED;
+    goto end;
+  }
+  // decode and process req
+  void* data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  int32_t len = metaLen - sizeof(SMsgHead);
+  tDecoderInit(&coder, data, len);
+  if (tDecodeSVDropTbBatchReq(&coder, &req) < 0) {
+    code = TSDB_CODE_INVALID_PARA;
+    goto end;
+  }
+
+  SVDropTbReq     *pDropReq = NULL;
+  SCatalog        *pCatalog = NULL;
+  code = catalogGetHandle(pTscObj->pAppInfo->clusterId, &pCatalog);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  pVgroupHashmap = taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), false, HASH_NO_LOCK);
+  if (NULL == pVgroupHashmap) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+  taosHashSetFreeFp(pVgroupHashmap, destroyDropTbReqBatch);
+
+  SRequestConnInfo conn = {.pTrans = pTscObj->pAppInfo->pTransporter,
+      .requestId = pRequest->requestId,
+      .requestObjRefId = pRequest->self,
+      .mgmtEps = getEpSet_s(&pTscObj->pAppInfo->mgmtEp)};
+  // loop to create table
+  for (int32_t iReq = 0; iReq < req.nReqs; iReq++) {
+    pDropReq = req.pReqs + iReq;
+
+    SVgroupInfo pInfo = {0};
+    SName pName;
+    toName(pTscObj->acctId, pRequest->pDb, pDropReq->name, &pName);
+    code = catalogGetTableHashVgroup(pCatalog, &conn, &pName, &pInfo);
+    if (code != TSDB_CODE_SUCCESS) {
+      goto end;
+    }
+
+    SVgroupDropTableBatch* pTableBatch = taosHashGet(pVgroupHashmap, &pInfo.vgId, sizeof(pInfo.vgId));
+    if (pTableBatch == NULL) {
+      SVgroupDropTableBatch tBatch = {0};
+      tBatch.info = pInfo;
+      tBatch.req.pArray = taosArrayInit(TARRAY_MIN_SIZE, sizeof(SVDropTbReq));
+      taosArrayPush(tBatch.req.pArray, pDropReq);
+
+      taosHashPut(pVgroupHashmap, &pInfo.vgId, sizeof(pInfo.vgId), &tBatch, sizeof(tBatch));
+    } else {  // add to the correct vgroup
+      taosArrayPush(pTableBatch->req.pArray, pDropReq);
+    }
+  }
+
+  SArray* pBufArray = serializeVgroupsDropTableBatch(pVgroupHashmap);
+  if (NULL == pBufArray) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+
+  pQuery = (SQuery*)nodesMakeNode(QUERY_NODE_QUERY);
+  pQuery->execMode = QUERY_EXEC_MODE_SCHEDULE;
+  pQuery->msgType = TDMT_VND_DROP_TABLE;
+  pQuery->stableQuery = false;
+  pQuery->pRoot   = nodesMakeNode(QUERY_NODE_DROP_TABLE_STMT);
+
+  code = rewriteToVnodeModifyOpStmt(pQuery, pBufArray);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  launchQueryImpl(pRequest, pQuery, false, NULL);
+  pQuery  = NULL;          // no need to free in the end
+  code    = pRequest->code;
+
+  end:
+  taosHashCleanup(pVgroupHashmap);
+  destroyRequest(pRequest);
+  tDecoderClear(&coder);
+  qDestroyQuery(pQuery);
+  return code;
+}
+
+static int32_t taosAlterTable(TAOS *taos, void *meta, int32_t metaLen){
+  SVAlterTbReq        req             = {0};
+  SDecoder            coder           = {0};
+  int32_t             code            = TSDB_CODE_SUCCESS;
+  SRequestObj        *pRequest        = NULL;
+  SQuery             *pQuery          = NULL;
+  SArray             *pArray          = NULL;
+  SVgDataBlocks      *pVgData         = NULL;
+  STscObj            *pTscObj         = acquireTscObj(*(int64_t *)taos);
+
+  if (NULL == pTscObj) {
+    code = TSDB_CODE_TSC_DISCONNECTED;
+    goto end;
+  }
+
+  code = buildRequest(pTscObj, "", 0, &pRequest);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  if(!pRequest->pDb){
+    code = TSDB_CODE_PAR_DB_NOT_SPECIFIED;
+    goto end;
+  }
+  // decode and process req
+  void* data = POINTER_SHIFT(meta, sizeof(SMsgHead));
+  int32_t len = metaLen - sizeof(SMsgHead);
+  tDecoderInit(&coder, data, len);
+  if (tDecodeSVAlterTbReq(&coder, &req) < 0) {
+    code = TSDB_CODE_INVALID_PARA;
+    goto end;
+  }
+
+  // do not deal TSDB_ALTER_TABLE_UPDATE_OPTIONS
+  if(req.action == TSDB_ALTER_TABLE_UPDATE_OPTIONS){
+    goto end;
+  }
+
+  SCatalog        *pCatalog = NULL;
+  code = catalogGetHandle(pTscObj->pAppInfo->clusterId, &pCatalog);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  SRequestConnInfo conn = {.pTrans = pTscObj->pAppInfo->pTransporter,
+      .requestId = pRequest->requestId,
+      .requestObjRefId = pRequest->self,
+      .mgmtEps = getEpSet_s(&pTscObj->pAppInfo->mgmtEp)};
+
+  SVgroupInfo pInfo = {0};
+  SName pName = {0};
+  toName(pTscObj->acctId, pRequest->pDb, req.tbName, &pName);
+  code = catalogGetTableHashVgroup(pCatalog, &conn, &pName, &pInfo);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  pArray = taosArrayInit(1, sizeof(void*));
+  if (NULL == pArray) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+
+  pVgData = taosMemoryCalloc(1, sizeof(SVgDataBlocks));
+  if (NULL == pVgData) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+  pVgData->vg = pInfo;
+  pVgData->pData = taosMemoryMalloc(metaLen);
+  if (NULL == pVgData->pData) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto end;
+  }
+  memcpy(pVgData->pData, meta, metaLen);
+  ((SMsgHead*)pVgData->pData)->vgId = htonl(pInfo.vgId);
+  pVgData->size = metaLen;
+  pVgData->numOfTables = 1;
+  taosArrayPush(pArray, &pVgData);
+
+  pQuery = (SQuery*)nodesMakeNode(QUERY_NODE_QUERY);
+  pQuery->execMode = QUERY_EXEC_MODE_SCHEDULE;
+  pQuery->msgType = TDMT_VND_ALTER_TABLE;
+  pQuery->stableQuery = false;
+  pQuery->pRoot   = nodesMakeNode(QUERY_NODE_ALTER_TABLE_STMT);
+
+  code = rewriteToVnodeModifyOpStmt(pQuery, pArray);
+  if (code != TSDB_CODE_SUCCESS) {
+    goto end;
+  }
+
+  launchQueryImpl(pRequest, pQuery, false, NULL);
+  pQuery  = NULL;          // no need to free in the end
+  pVgData = NULL;
+  pArray  = NULL;
+  code    = pRequest->code;
+
+end:
+  taosArrayDestroy(pArray);
+  if(pVgData) taosMemoryFreeClear(pVgData->pData);
+  taosMemoryFreeClear(pVgData);
+  destroyRequest(pRequest);
+  tDecoderClear(&coder);
+  qDestroyQuery(pQuery);
+  return code;
+}
+
+int32_t taos_write_raw_meta(TAOS *taos, tmq_raw_data *raw_meta){
+  if (!taos || !raw_meta) {
+    return TSDB_CODE_INVALID_PARA;
+  }
+
+  if(raw_meta->raw_meta_type == TDMT_VND_CREATE_STB) {
+    return taosCreateStb(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
+  }else if(raw_meta->raw_meta_type == TDMT_VND_ALTER_STB){
+    return taosCreateStb(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
+  }else if(raw_meta->raw_meta_type == TDMT_VND_DROP_STB){
+    return taosDropStb(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
+  }else if(raw_meta->raw_meta_type == TDMT_VND_CREATE_TABLE){
+    return taosCreateTable(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
+  }else if(raw_meta->raw_meta_type == TDMT_VND_ALTER_TABLE){
+    return taosAlterTable(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
+  }else if(raw_meta->raw_meta_type == TDMT_VND_DROP_TABLE){
+    return taosDropTable(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
+  }
+  return TSDB_CODE_INVALID_PARA;
+}
+
 void tmq_commit_async(tmq_t* tmq, const TAOS_RES* msg, tmq_commit_cb* cb, void* param) {
  tmqCommitInner2(tmq, msg, 0, 1, cb, param);
 }
--- a/source/client/test/clientTests.cpp
+++ b/source/client/test/clientTests.cpp
@ -750,7 +750,6 @@ TEST(testCase, projection_query_stables) {
  taos_close(pConn);
 }

-
 TEST(testCase, agg_query_tables) {
  TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
  ASSERT_NE(pConn, nullptr);
@ -763,7 +762,7 @@ TEST(testCase, agg_query_tables) {
  }
  taos_free_result(pRes);

-  pRes = taos_query(pConn, "show table distributed st1");
+  pRes = taos_query(pConn, "show table distributed tup");
  if (taos_errno(pRes) != 0) {
    printf("failed to select from table, reason:%s\n", taos_errstr(pRes));
    taos_free_result(pRes);
@ -822,13 +821,29 @@ TEST(testCase, async_api_test) {
 }
 #endif

+
 TEST(testCase, update_test) {
  TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
  ASSERT_NE(pConn, nullptr);

-  taos_query(pConn, "use abc1");
+  TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1");
+  if (taos_errno(pRes) != TSDB_CODE_SUCCESS) {
+    printf("failed to create database, code:%s", taos_errstr(pRes));
+    taos_free_result(pRes);
+    return;
+  }

-  TAOS_RES* pRes = taos_query(pConn, "create table tup (ts timestamp, k int);");
+  taos_free_result(pRes);
+
+  pRes = taos_query(pConn, "use abc1");
+  if (taos_errno(pRes) != TSDB_CODE_SUCCESS) {
+    printf("failed to use db, code:%s", taos_errstr(pRes));
+    taos_free_result(pRes);
+    return;
+  }
+  taos_free_result(pRes);
+
+  pRes = taos_query(pConn, "create table tup (ts timestamp, k int);");
  if (taos_errno(pRes) != 0) {
    printf("failed to create table, reason:%s", taos_errstr(pRes));
  }
@ -836,11 +851,10 @@ TEST(testCase, update_test) {
  taos_free_result(pRes);

  char s[256]  = {0};
-  for(int32_t i = 0; i < 7000; ++i) {
-    sprintf(s, "insert into tup values('2020-1-1 1:1:1', %d)", i);
+  for(int32_t i = 0; i < 17000; ++i) {
+    sprintf(s, "insert into tup values(now+%da, %d)", i, i);
    pRes = taos_query(pConn, s);
    taos_free_result(pRes);
  }
 }
-
 #pragma GCC diagnostic pop
--- a/source/client/test/smlTest.cpp
+++ b/source/client/test/smlTest.cpp
@ -1284,4 +1284,210 @@ TEST(testCase, sml_dup_time_Test) {
  ASSERT_EQ(taos_errno(pRes), 0);
  taos_free_result(pRes);
 }
-*/
+
+
+TEST(testCase, sml_16960_Test) {
+TAOS *taos = taos_connect("localhost", "root", "taosdata", NULL, 0);
+ASSERT_NE(taos, nullptr);
+
+TAOS_RES* pRes = taos_query(taos, "create database if not exists d16368 schemaless 1");
+taos_free_result(pRes);
+
+pRes = taos_query(taos, "use d16368");
+taos_free_result(pRes);
+
+const char *sql[] = {
+    "[\n"
+    "{\n"
+    "\"timestamp\":\n"
+    "\n"
+    "{ \"value\": 1349020800000, \"type\": \"ms\" }\n"
+    ",\n"
+    "\"value\":\n"
+    "\n"
+    "{ \"value\": 830525384, \"type\": \"int\" }\n"
+    ",\n"
+    "\"tags\": {\n"
+    "\"id\": \"stb00_0\",\n"
+    "\"t0\":\n"
+    "\n"
+    "{ \"value\": 83972721, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t1\":\n"
+    "\n"
+    "{ \"value\": 539147525, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t2\":\n"
+    "\n"
+    "{ \"value\": 618258572, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t3\":\n"
+    "\n"
+    "{ \"value\": -10536201, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t4\":\n"
+    "\n"
+    "{ \"value\": 349227409, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t5\":\n"
+    "\n"
+    "{ \"value\": 249347042, \"type\": \"int\" }\n"
+    "},\n"
+    "\"metric\": \"stb0\"\n"
+    "},\n"
+    "{\n"
+    "\"timestamp\":\n"
+    "\n"
+    "{ \"value\": 1349020800001, \"type\": \"ms\" }\n"
+    ",\n"
+    "\"value\":\n"
+    "\n"
+    "{ \"value\": -588348364, \"type\": \"int\" }\n"
+    ",\n"
+    "\"tags\": {\n"
+    "\"id\": \"stb00_0\",\n"
+    "\"t0\":\n"
+    "\n"
+    "{ \"value\": 83972721, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t1\":\n"
+    "\n"
+    "{ \"value\": 539147525, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t2\":\n"
+    "\n"
+    "{ \"value\": 618258572, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t3\":\n"
+    "\n"
+    "{ \"value\": -10536201, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t4\":\n"
+    "\n"
+    "{ \"value\": 349227409, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t5\":\n"
+    "\n"
+    "{ \"value\": 249347042, \"type\": \"int\" }\n"
+    "},\n"
+    "\"metric\": \"stb0\"\n"
+    "},\n"
+    "{\n"
+    "\"timestamp\":\n"
+    "\n"
+    "{ \"value\": 1349020800002, \"type\": \"ms\" }\n"
+    ",\n"
+    "\"value\":\n"
+    "\n"
+    "{ \"value\": -370310823, \"type\": \"int\" }\n"
+    ",\n"
+    "\"tags\": {\n"
+    "\"id\": \"stb00_0\",\n"
+    "\"t0\":\n"
+    "\n"
+    "{ \"value\": 83972721, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t1\":\n"
+    "\n"
+    "{ \"value\": 539147525, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t2\":\n"
+    "\n"
+    "{ \"value\": 618258572, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t3\":\n"
+    "\n"
+    "{ \"value\": -10536201, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t4\":\n"
+    "\n"
+    "{ \"value\": 349227409, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t5\":\n"
+    "\n"
+    "{ \"value\": 249347042, \"type\": \"int\" }\n"
+    "},\n"
+    "\"metric\": \"stb0\"\n"
+    "},\n"
+    "{\n"
+    "\"timestamp\":\n"
+    "\n"
+    "{ \"value\": 1349020800003, \"type\": \"ms\" }\n"
+    ",\n"
+    "\"value\":\n"
+    "\n"
+    "{ \"value\": -811250191, \"type\": \"int\" }\n"
+    ",\n"
+    "\"tags\": {\n"
+    "\"id\": \"stb00_0\",\n"
+    "\"t0\":\n"
+    "\n"
+    "{ \"value\": 83972721, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t1\":\n"
+    "\n"
+    "{ \"value\": 539147525, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t2\":\n"
+    "\n"
+    "{ \"value\": 618258572, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t3\":\n"
+    "\n"
+    "{ \"value\": -10536201, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t4\":\n"
+    "\n"
+    "{ \"value\": 349227409, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t5\":\n"
+    "\n"
+    "{ \"value\": 249347042, \"type\": \"int\" }\n"
+    "},\n"
+    "\"metric\": \"stb0\"\n"
+    "},\n"
+    "{\n"
+    "\"timestamp\":\n"
+    "\n"
+    "{ \"value\": 1349020800004, \"type\": \"ms\" }\n"
+    ",\n"
+    "\"value\":\n"
+    "\n"
+    "{ \"value\": -330340558, \"type\": \"int\" }\n"
+    ",\n"
+    "\"tags\": {\n"
+    "\"id\": \"stb00_0\",\n"
+    "\"t0\":\n"
+    "\n"
+    "{ \"value\": 83972721, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t1\":\n"
+    "\n"
+    "{ \"value\": 539147525, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t2\":\n"
+    "\n"
+    "{ \"value\": 618258572, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t3\":\n"
+    "\n"
+    "{ \"value\": -10536201, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t4\":\n"
+    "\n"
+    "{ \"value\": 349227409, \"type\": \"int\" }\n"
+    ",\n"
+    "\"t5\":\n"
+    "\n"
+    "{ \"value\": 249347042, \"type\": \"int\" }\n"
+    "},\n"
+    "\"metric\": \"stb0\"\n"
+    "}\n"
+    "]"
+};
+
+pRes = taos_schemaless_insert(taos, (char**)sql, sizeof(sql)/sizeof(sql[0]), TSDB_SML_JSON_PROTOCOL, TSDB_SML_TIMESTAMP_MILLI_SECONDS);
+ASSERT_EQ(taos_errno(pRes), 0);
+taos_free_result(pRes);
+}
+*/
--- a/source/common/src/tdata.c
+++ b/source/common/src/tdata.c
@ -1,14 +0,0 @@
-/*
- * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
- *
- * This program is free software: you can use, redistribute, and/or modify
- * it under the terms of the GNU Affero General Public License, version 3
- * or later ("AGPL"), as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
--- a/source/common/src/tdatablock.c
+++ b/source/common/src/tdatablock.c
@ -228,7 +228,7 @@ int32_t colDataMergeCol(SColumnInfoData* pColumnInfoData, uint32_t numOfRow1, ui
  uint32_t finalNumOfRows = numOfRow1 + numOfRow2;
  if (IS_VAR_DATA_TYPE(pColumnInfoData->info.type)) {
    // Handle the bitmap
-    if (finalNumOfRows > *capacity) {
+    if (finalNumOfRows > *capacity || numOfRow1 == 0) {
      char* p = taosMemoryRealloc(pColumnInfoData->varmeta.offset, sizeof(int32_t) * (numOfRow1 + numOfRow2));
      if (p == NULL) {
        return TSDB_CODE_OUT_OF_MEMORY;
@ -262,7 +262,7 @@ int32_t colDataMergeCol(SColumnInfoData* pColumnInfoData, uint32_t numOfRow1, ui
    memcpy(pColumnInfoData->pData + oldLen, pSource->pData, len);
    pColumnInfoData->varmeta.length = len + oldLen;
  } else {
-    if (finalNumOfRows > *capacity) {
+    if (finalNumOfRows > *capacity || numOfRow1 == 0) {
      ASSERT(finalNumOfRows * pColumnInfoData->info.bytes);
      char* tmp = taosMemoryRealloc(pColumnInfoData->pData, finalNumOfRows * pColumnInfoData->info.bytes);
      if (tmp == NULL) {
@ -1356,7 +1356,7 @@ SColumnInfoData createColumnInfoData(int16_t type, int32_t bytes, int16_t colId)
  return col;
 }

-SColumnInfoData* bdGetColumnInfoData(SSDataBlock* pBlock, int32_t index) {
+SColumnInfoData* bdGetColumnInfoData(const SSDataBlock* pBlock, int32_t index) {
  ASSERT(pBlock != NULL);
  if (index >= taosArrayGetSize(pBlock->pDataBlock)) {
    return NULL;
@ -1735,9 +1735,9 @@ char* dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf)
  int32_t colNum = taosArrayGetSize(pDataBlock->pDataBlock);
  int32_t rows = pDataBlock->info.rows;
  int32_t len = 0;
-  len += snprintf(dumpBuf + len, size - len, "\n%s |block type %d |child id %d|group id %lu|\n", flag,
+  len += snprintf(dumpBuf + len, size - len, "\n%s |block type %d |child id %d|group id:%" PRIu64 "|\n", flag,
                  (int32_t)pDataBlock->info.type, pDataBlock->info.childId, pDataBlock->info.groupId);
-  if (len >= size -1) return dumpBuf;
+  if (len >= size - 1) return dumpBuf;

  for (int32_t j = 0; j < rows; j++) {
    len += snprintf(dumpBuf + len, size - len, "%s |", flag);
@ -2119,3 +2119,4 @@ const char* blockDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRow
  ASSERT(pStart - pData == dataLen);
  return pStart;
 }
+
--- a/source/common/src/tglobal.c
+++ b/source/common/src/tglobal.c
@ -314,7 +314,7 @@ static int32_t taosAddServerLogCfg(SConfig *pCfg) {
  if (cfgAddInt32(pCfg, "tsdbDebugFlag", tsdbDebugFlag, 0, 255, 0) != 0) return -1;
  if (cfgAddInt32(pCfg, "tqDebugFlag", tqDebugFlag, 0, 255, 0) != 0) return -1;
  if (cfgAddInt32(pCfg, "fsDebugFlag", fsDebugFlag, 0, 255, 0) != 0) return -1;
-  if (cfgAddInt32(pCfg, "fnDebugFlag", fnDebugFlag, 0, 255, 0) != 0) return -1;
+  if (cfgAddInt32(pCfg, "udfDebugFlag", udfDebugFlag, 0, 255, 0) != 0) return -1;
  if (cfgAddInt32(pCfg, "smaDebugFlag", smaDebugFlag, 0, 255, 0) != 0) return -1;
  if (cfgAddInt32(pCfg, "idxDebugFlag", idxDebugFlag, 0, 255, 0) != 0) return -1;
  return 0;
@ -504,7 +504,7 @@ static void taosSetServerLogCfg(SConfig *pCfg) {
  tsdbDebugFlag = cfgGetItem(pCfg, "tsdbDebugFlag")->i32;
  tqDebugFlag = cfgGetItem(pCfg, "tqDebugFlag")->i32;
  fsDebugFlag = cfgGetItem(pCfg, "fsDebugFlag")->i32;
-  fnDebugFlag = cfgGetItem(pCfg, "fnDebugFlag")->i32;
+  udfDebugFlag = cfgGetItem(pCfg, "udfDebugFlag")->i32;
  smaDebugFlag = cfgGetItem(pCfg, "smaDebugFlag")->i32;
  idxDebugFlag = cfgGetItem(pCfg, "idxDebugFlag")->i32;
 }
@ -715,8 +715,6 @@ int32_t taosSetCfg(SConfig *pCfg, char* name) {
        cfgSetItem(pCfg, "firstEp", tsFirst, pFirstEpItem->stype);
      } else if (strcasecmp("fsDebugFlag", name) == 0) {
        fsDebugFlag = cfgGetItem(pCfg, "fsDebugFlag")->i32;
-      } else if (strcasecmp("fnDebugFlag", name) == 0) {
-        fnDebugFlag = cfgGetItem(pCfg, "fnDebugFlag")->i32;
      }
      break;
    }
@ -817,6 +815,8 @@ int32_t taosSetCfg(SConfig *pCfg, char* name) {
        case 'u': {
          if (strcasecmp("multiProcess", name) == 0) {
            tsMultiProcess = cfgGetItem(pCfg, "multiProcess")->bval;
+          } else if (strcasecmp("udfDebugFlag", name) == 0) {
+            udfDebugFlag = cfgGetItem(pCfg, "udfDebugFlag")->i32;
          }
          break;
        }
--- a/source/common/src/tmsg.c
+++ b/source/common/src/tmsg.c
@ -381,9 +381,11 @@ static int32_t tDeserializeSClientHbRsp(SDecoder *pDecoder, SClientHbRsp *pRsp)
    if (pQnodeNum > 0) {
      pRsp->query->pQnodeList = taosArrayInit(pQnodeNum, sizeof(SQueryNodeLoad));
      if (NULL == pRsp->query->pQnodeList) return -1;
-      SQueryNodeLoad load = {0};
-      if (tDecodeSQueryNodeLoad(pDecoder, &load) < 0) return -1;
-      taosArrayPush(pRsp->query->pQnodeList, &load);
+      for (int32_t i = 0; i < pQnodeNum; ++i) {
+        SQueryNodeLoad load = {0};
+        if (tDecodeSQueryNodeLoad(pDecoder, &load) < 0) return -1;
+        taosArrayPush(pRsp->query->pQnodeList, &load);
+      }
    }
  }

@ -496,11 +498,18 @@ int32_t tSerializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pReq
  if (tStartEncode(&encoder) < 0) return -1;
  if (tEncodeCStr(&encoder, pReq->name) < 0) return -1;
  if (tEncodeI8(&encoder, pReq->igExists) < 0) return -1;
+  if (tEncodeI8(&encoder, pReq->source) < 0) return -1;
+  for (int32_t i = 0; i < sizeof(pReq->reserved) / sizeof(int8_t); ++i) {
+    if (tEncodeI8(&encoder, pReq->reserved[i]) < 0) return -1;
+  }
+  if (tEncodeI64(&encoder, pReq->suid) < 0) return -1;
  if (tEncodeI64(&encoder, pReq->delay1) < 0) return -1;
  if (tEncodeI64(&encoder, pReq->delay2) < 0) return -1;
  if (tEncodeI64(&encoder, pReq->watermark1) < 0) return -1;
  if (tEncodeI64(&encoder, pReq->watermark2) < 0) return -1;
  if (tEncodeI32(&encoder, pReq->ttl) < 0) return -1;
+  if (tEncodeI32(&encoder, pReq->colVer) < 0) return -1;
+  if (tEncodeI32(&encoder, pReq->tagVer) < 0) return -1;
  if (tEncodeI32(&encoder, pReq->numOfColumns) < 0) return -1;
  if (tEncodeI32(&encoder, pReq->numOfTags) < 0) return -1;
  if (tEncodeI32(&encoder, pReq->numOfFuncs) < 0) return -1;
@ -553,11 +562,18 @@ int32_t tDeserializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pR
  if (tStartDecode(&decoder) < 0) return -1;
  if (tDecodeCStrTo(&decoder, pReq->name) < 0) return -1;
  if (tDecodeI8(&decoder, &pReq->igExists) < 0) return -1;
+  if (tDecodeI8(&decoder, &pReq->source) < 0) return -1;
+  for (int32_t i = 0; i < sizeof(pReq->reserved) / sizeof(int8_t); ++i) {
+    if (tDecodeI8(&decoder, &pReq->reserved[i]) < 0) return -1;
+  }
+  if (tDecodeI64(&decoder, &pReq->suid) < 0) return -1;
  if (tDecodeI64(&decoder, &pReq->delay1) < 0) return -1;
  if (tDecodeI64(&decoder, &pReq->delay2) < 0) return -1;
  if (tDecodeI64(&decoder, &pReq->watermark1) < 0) return -1;
  if (tDecodeI64(&decoder, &pReq->watermark2) < 0) return -1;
  if (tDecodeI32(&decoder, &pReq->ttl) < 0) return -1;
+  if (tDecodeI32(&decoder, &pReq->colVer) < 0) return -1;
+  if (tDecodeI32(&decoder, &pReq->tagVer) < 0) return -1;
  if (tDecodeI32(&decoder, &pReq->numOfColumns) < 0) return -1;
  if (tDecodeI32(&decoder, &pReq->numOfTags) < 0) return -1;
  if (tDecodeI32(&decoder, &pReq->numOfFuncs) < 0) return -1;
@ -645,6 +661,11 @@ int32_t tSerializeSMDropStbReq(void *buf, int32_t bufLen, SMDropStbReq *pReq) {
  if (tStartEncode(&encoder) < 0) return -1;
  if (tEncodeCStr(&encoder, pReq->name) < 0) return -1;
  if (tEncodeI8(&encoder, pReq->igNotExists) < 0) return -1;
+  if (tEncodeI8(&encoder, pReq->source) < 0) return -1;
+  for (int32_t i = 0; i < sizeof(pReq->reserved) / sizeof(int8_t); ++i) {
+    if (tEncodeI8(&encoder, pReq->reserved[i]) < 0) return -1;
+  }
+  if (tEncodeI64(&encoder, pReq->suid) < 0) return -1;
  tEndEncode(&encoder);

  int32_t tlen = encoder.pos;
@ -659,6 +680,12 @@ int32_t tDeserializeSMDropStbReq(void *buf, int32_t bufLen, SMDropStbReq *pReq)
  if (tStartDecode(&decoder) < 0) return -1;
  if (tDecodeCStrTo(&decoder, pReq->name) < 0) return -1;
  if (tDecodeI8(&decoder, &pReq->igNotExists) < 0) return -1;
+  if (tDecodeI8(&decoder, &pReq->source) < 0) return -1;
+  for (int32_t i = 0; i < sizeof(pReq->reserved) / sizeof(int8_t); ++i) {
+    if (tDecodeI8(&decoder, &pReq->reserved[i]) < 0) return -1;
+  }
+  if (tDecodeI64(&decoder, &pReq->suid) < 0) return -1;
+
  tEndDecode(&decoder);

  tDecoderClear(&decoder);
@ -672,8 +699,6 @@ int32_t tSerializeSMAlterStbReq(void *buf, int32_t bufLen, SMAlterStbReq *pReq)
  if (tStartEncode(&encoder) < 0) return -1;
  if (tEncodeCStr(&encoder, pReq->name) < 0) return -1;
  if (tEncodeI8(&encoder, pReq->alterType) < 0) return -1;
-  if (tEncodeI32(&encoder, pReq->tagVer) < 0) return -1;
-  if (tEncodeI32(&encoder, pReq->colVer) < 0) return -1;
  if (tEncodeI32(&encoder, pReq->numOfFields) < 0) return -1;
  for (int32_t i = 0; i < pReq->numOfFields; ++i) {
    SField *pField = taosArrayGet(pReq->pFields, i);
@ -700,8 +725,6 @@ int32_t tDeserializeSMAlterStbReq(void *buf, int32_t bufLen, SMAlterStbReq *pReq
  if (tStartDecode(&decoder) < 0) return -1;
  if (tDecodeCStrTo(&decoder, pReq->name) < 0) return -1;
  if (tDecodeI8(&decoder, &pReq->alterType) < 0) return -1;
-  if (tDecodeI32(&decoder, &pReq->tagVer) < 0) return -1;
-  if (tDecodeI32(&decoder, &pReq->colVer) < 0) return -1;
  if (tDecodeI32(&decoder, &pReq->numOfFields) < 0) return -1;
  pReq->pFields = taosArrayInit(pReq->numOfFields, sizeof(SField));
  if (pReq->pFields == NULL) {
@ -5406,11 +5429,11 @@ int32_t tFormatOffset(char *buf, int32_t maxLen, const STqOffsetVal *pVal) {
  } else if (pVal->type == TMQ_OFFSET__RESET_LATEST) {
    snprintf(buf, maxLen, "offset(reset to latest)");
  } else if (pVal->type == TMQ_OFFSET__LOG) {
-    snprintf(buf, maxLen, "offset(log) ver:%ld", pVal->version);
+    snprintf(buf, maxLen, "offset(log) ver:%" PRId64, pVal->version);
  } else if (pVal->type == TMQ_OFFSET__SNAPSHOT_DATA) {
-    snprintf(buf, maxLen, "offset(ss data) uid:%ld, ts:%ld", pVal->uid, pVal->ts);
+    snprintf(buf, maxLen, "offset(ss data) uid:%" PRId64 ", ts:%" PRId64, pVal->uid, pVal->ts);
  } else if (pVal->type == TMQ_OFFSET__SNAPSHOT_META) {
-    snprintf(buf, maxLen, "offset(ss meta) uid:%ld, ts:%ld", pVal->uid, pVal->ts);
+    snprintf(buf, maxLen, "offset(ss meta) uid:%" PRId64 ", ts:%" PRId64, pVal->uid, pVal->ts);
  } else {
    ASSERT(0);
  }
--- a/source/common/src/tmsgtype.c
+++ b/source/common/src/tmsgtype.c
@ -1,18 +0,0 @@
-/*
- * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
- *
- * This program is free software: you can use, redistribute, and/or modify
- * it under the terms of the GNU Affero General Public License, version 3
- * or later ("AGPL"), as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-#define _DEFAULT_SOURCE
-#define TSDB_SQL_C
-#include "tmsgtype.h"
--- a/source/common/src/tname.c
+++ b/source/common/src/tname.c
@ -115,6 +115,14 @@ int64_t taosGetIntervalStartTimestamp(int64_t startTime, int64_t slidingTime, in

 #endif

+SName* toName(int32_t acctId, const char* pDbName, const char* pTableName, SName* pName) {
+  pName->type = TSDB_TABLE_NAME_T;
+  pName->acctId = acctId;
+  strcpy(pName->dbname, pDbName);
+  strcpy(pName->tname, pTableName);
+  return pName;
+}
+
 int32_t tNameExtractFullName(const SName* name, char* dst) {
  assert(name != NULL && dst != NULL);

--- a/source/dnode/mgmt/exe/dmMain.c
+++ b/source/dnode/mgmt/exe/dmMain.c
@ -216,7 +216,7 @@ int main(int argc, char const *argv[]) {
    return -1;
  }

-  dInfo("start to open dnode");
+  dInfo("start to init service");
  dmSetSignalHandle();
  int32_t code = dmRun();
  dInfo("shutting down the service");
--- a/source/dnode/mgmt/mgmt_dnode/src/dmHandle.c
+++ b/source/dnode/mgmt/mgmt_dnode/src/dmHandle.c
@ -21,7 +21,7 @@ extern SConfig *tsCfg;

 static void dmUpdateDnodeCfg(SDnodeMgmt *pMgmt, SDnodeCfg *pCfg) {
  if (pMgmt->pData->dnodeId == 0 || pMgmt->pData->clusterId == 0) {
-    dInfo("set dnodeId:%d clusterId:%" PRId64, pCfg->dnodeId, pCfg->clusterId);
+    dInfo("set local info, dnodeId:%d clusterId:%" PRId64, pCfg->dnodeId, pCfg->clusterId);
    taosThreadRwlockWrlock(&pMgmt->pData->lock);
    pMgmt->pData->dnodeId = pCfg->dnodeId;
    pMgmt->pData->clusterId = pCfg->clusterId;
--- a/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c
+++ b/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c
@ -167,7 +167,7 @@ int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
  if (pMsg == NULL) return -1;
  memcpy(pMsg, pRpc, sizeof(SRpcMsg));

-  dTrace("msg:%p, is created and will put int %s queue", pMsg, pWorker->name);
+  dTrace("msg:%p, is created and will put into %s queue, type:%s", pMsg, pWorker->name, TMSG_INFO(pRpc->msgType));
  return mmPutMsgToWorker(pMgmt, pWorker, pMsg);
 }

--- a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
+++ b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
@ -384,6 +384,8 @@ SArray *vmGetMsgHandles() {
  if (dmSetMgmtHandle(pArray, TDMT_SYNC_APPEND_ENTRIES, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;
  if (dmSetMgmtHandle(pArray, TDMT_SYNC_APPEND_ENTRIES_BATCH, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;
  if (dmSetMgmtHandle(pArray, TDMT_SYNC_APPEND_ENTRIES_REPLY, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_SYNC_SNAPSHOT_SEND, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;
+  if (dmSetMgmtHandle(pArray, TDMT_SYNC_SNAPSHOT_RSP, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;
  if (dmSetMgmtHandle(pArray, TDMT_SYNC_SET_VNODE_STANDBY, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;

  code = 0;
--- a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c
+++ b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c
@ -173,7 +173,7 @@ static int32_t vmOpenVnodes(SVnodeMgmt *pMgmt) {
    pThread->pCfgs[pThread->vnodeNum++] = pCfgs[v];
  }

-  dInfo("start %d threads to open %d vnodes", threadNum, numOfVnodes);
+  dInfo("open %d vnodes with %d threads", numOfVnodes, threadNum);

  for (int32_t t = 0; t < threadNum; ++t) {
    SVnodeThread *pThread = &threads[t];
@ -204,7 +204,7 @@ static int32_t vmOpenVnodes(SVnodeMgmt *pMgmt) {
    dError("there are total vnodes:%d, opened:%d", pMgmt->state.totalVnodes, pMgmt->state.openVnodes);
    return -1;
  } else {
-    dInfo("total vnodes:%d open successfully", pMgmt->state.totalVnodes);
+    dInfo("successfully opened %d vnodes", pMgmt->state.totalVnodes);
    return 0;
  }
 }
--- a/source/dnode/mgmt/node_mgmt/src/dmMgmt.c
+++ b/source/dnode/mgmt/node_mgmt/src/dmMgmt.c
@ -128,7 +128,7 @@ static void dmClearVars(SDnode *pDnode) {
 }

 int32_t dmInitDnode(SDnode *pDnode, EDndNodeType rtype) {
-  dInfo("start to create dnode");
+  dDebug("start to create dnode");
  int32_t code = -1;
  char    path[PATH_MAX + 100] = {0};

--- a/source/dnode/mgmt/node_mgmt/src/dmNodes.c
+++ b/source/dnode/mgmt/node_mgmt/src/dmNodes.c
@ -277,7 +277,7 @@ int32_t dmRunDnode(SDnode *pDnode) {

  while (1) {
    if (pDnode->stop) {
-      dInfo("dnode is about to stop");
+      dInfo("TDengine is about to stop");
      dmSetStatus(pDnode, DND_STAT_STOPPED);
      dmStopNodes(pDnode);
      dmCloseNodes(pDnode);
--- a/source/dnode/mgmt/node_mgmt/src/dmTransport.c
+++ b/source/dnode/mgmt/node_mgmt/src/dmTransport.c
@ -89,7 +89,8 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
    case TDMT_DND_SYSTABLE_RETRIEVE_RSP:
    case TDMT_SCH_FETCH_RSP:
    case TDMT_SCH_MERGE_FETCH_RSP:
-      qWorkerProcessFetchRsp(NULL, NULL, pRpc, 0);
+    case TDMT_VND_SUBMIT_RSP:
+      qWorkerProcessRspMsg(NULL, NULL, pRpc, 0);
      return;
    case TDMT_MND_STATUS_RSP:
      if (pEpSet != NULL) {
--- a/source/dnode/mnode/impl/inc/mndInt.h
+++ b/source/dnode/mnode/impl/inc/mndInt.h
@ -34,8 +34,6 @@ extern "C" {
 #endif

 // clang-format off
-
-
 #define mFatal(...) { if (mDebugFlag & DEBUG_FATAL) { taosPrintLog("MND FATAL ", DEBUG_FATAL, 255,        __VA_ARGS__); }}
 #define mError(...) { if (mDebugFlag & DEBUG_ERROR) { taosPrintLog("MND ERROR ", DEBUG_ERROR, 255,        __VA_ARGS__); }}
 #define mWarn(...)  { if (mDebugFlag & DEBUG_WARN)  { taosPrintLog("MND WARN ",  DEBUG_WARN,  255,        __VA_ARGS__); }}
@ -49,7 +47,6 @@ extern "C" {
 #define mGInfo(param, ...)  { char buf[40] = {0}; TRACE_TO_STR(trace, buf); mInfo (param ", gtid:%s", __VA_ARGS__, buf);}
 #define mGDebug(param, ...) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); mDebug(param ", gtid:%s", __VA_ARGS__, buf);}
 #define mGTrace(param, ...) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); mTrace(param ", gtid:%s", __VA_ARGS__, buf);}
-
 // clang-format on

 #define SYSTABLE_SCH_TABLE_NAME_LEN ((TSDB_TABLE_NAME_LEN - 1) + VARSTR_HEADER_SIZE)
--- a/source/dnode/mnode/impl/src/mndConsumer.c
+++ b/source/dnode/mnode/impl/src/mndConsumer.c
@ -92,7 +92,7 @@ static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) {
  SMqConsumerObj     *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId);
  ASSERT(pConsumer);

-  mInfo("receive consumer lost msg, consumer id %ld, status %s", pLostMsg->consumerId,
+  mInfo("receive consumer lost msg, consumer id %" PRId64 ", status %s", pLostMsg->consumerId,
        mndConsumerStatusName(pConsumer->status));

  if (pConsumer->status != MQ_CONSUMER_STATUS__READY) {
@ -124,7 +124,7 @@ static int32_t mndProcessConsumerRecoverMsg(SRpcMsg *pMsg) {
  SMqConsumerObj        *pConsumer = mndAcquireConsumer(pMnode, pRecoverMsg->consumerId);
  ASSERT(pConsumer);

-  mInfo("receive consumer recover msg, consumer id %ld, status %s", pRecoverMsg->consumerId,
+  mInfo("receive consumer recover msg, consumer id %" PRId64 ", status %s", pRecoverMsg->consumerId,
        mndConsumerStatusName(pConsumer->status));

  if (pConsumer->status != MQ_CONSUMER_STATUS__READY) {
@ -296,7 +296,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
  // 2. check epoch, only send ep info when epoches do not match
  if (epoch != serverEpoch) {
    taosRLockLatch(&pConsumer->lock);
-    mInfo("process ask ep, consumer %ld(epoch %d), server epoch %d", consumerId, epoch, serverEpoch);
+    mInfo("process ask ep, consumer:%" PRId64 "(epoch %d), server epoch %d", consumerId, epoch, serverEpoch);
    int32_t numOfTopics = taosArrayGetSize(pConsumer->currentTopics);

    rsp.topics = taosArrayInit(numOfTopics, sizeof(SMqSubTopicEp));
@ -441,7 +441,7 @@ static int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
    SMqTopicObj topicObj = {0};
    memcpy(&topicObj, pTopic, sizeof(SMqTopicObj));
    topicObj.refConsumerCnt = pTopic->refConsumerCnt + 1;
-    mInfo("subscribe topic %s by consumer %ld cgroup %s, refcnt %d", pTopic->name, consumerId, cgroup,
+    mInfo("subscribe topic %s by consumer:%" PRId64 ",cgroup %s, refcnt %d", pTopic->name, consumerId, cgroup,
          topicObj.refConsumerCnt);
    if (mndSetTopicCommitLogs(pMnode, pTrans, &topicObj) != 0) goto SUBSCRIBE_OVER;
 #endif
@ -451,7 +451,7 @@ static int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {

  pConsumerOld = mndAcquireConsumer(pMnode, consumerId);
  if (pConsumerOld == NULL) {
-    mInfo("receive subscribe request from new consumer: %ld", consumerId);
+    mInfo("receive subscribe request from new consumer:%" PRId64, consumerId);

    pConsumerNew = tNewSMqConsumerObj(consumerId, cgroup);
    tstrncpy(pConsumerNew->clientId, subscribe.clientId, 256);
@ -472,7 +472,7 @@ static int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {

    int32_t status = atomic_load_32(&pConsumerOld->status);

-    mInfo("receive subscribe request from old consumer: %ld, current status: %s", consumerId,
+    mInfo("receive subscribe request from old consumer:%" PRId64 ", current status: %s", consumerId,
          mndConsumerStatusName(status));

    if (status != MQ_CONSUMER_STATUS__READY) {
--- a/source/dnode/mnode/impl/src/mndDnode.c
+++ b/source/dnode/mnode/impl/src/mndDnode.c
@ -406,7 +406,7 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
    }

    if (statusReq.dnodeId == 0) {
-      mInfo("dnode:%d, %s first access, set clusterId %" PRId64, pDnode->id, pDnode->ep, pMnode->clusterId);
+      mInfo("dnode:%d, %s first access, clusterId:%" PRId64, pDnode->id, pDnode->ep, pMnode->clusterId);
    } else {
      if (statusReq.clusterId != pMnode->clusterId) {
        if (pDnode != NULL) {
--- a/source/dnode/mnode/impl/src/mndOffset.c
+++ b/source/dnode/mnode/impl/src/mndOffset.c
@ -185,7 +185,7 @@ static int32_t mndProcessCommitOffsetReq(SRpcMsg *pMsg) {

  for (int32_t i = 0; i < commitOffsetReq.num; i++) {
    SMqOffset *pOffset = &commitOffsetReq.offsets[i];
-    mInfo("commit offset %ld to vg %d of consumer group %s on topic %s", pOffset->offset, pOffset->vgId,
+    mInfo("commit offset %" PRId64 " to vgId:%d of consumer group %s on topic %s", pOffset->offset, pOffset->vgId,
          pOffset->cgroup, pOffset->topicName);
    if (mndMakePartitionKey(key, pOffset->cgroup, pOffset->topicName, pOffset->vgId) < 0) {
      mError("submit offset to topic %s failed", pOffset->topicName);
--- a/source/dnode/mnode/impl/src/mndStb.c
+++ b/source/dnode/mnode/impl/src/mndStb.c
@ -705,10 +705,10 @@ int32_t mndBuildStbFromReq(SMnode *pMnode, SStbObj *pDst, SMCreateStbReq *pCreat
  memcpy(pDst->db, pDb->name, TSDB_DB_FNAME_LEN);
  pDst->createdTime = taosGetTimestampMs();
  pDst->updateTime = pDst->createdTime;
-  pDst->uid = mndGenerateUid(pCreate->name, TSDB_TABLE_FNAME_LEN);
+  pDst->uid = (pCreate->source == 1) ? pCreate->suid : mndGenerateUid(pCreate->name, TSDB_TABLE_FNAME_LEN);
  pDst->dbUid = pDb->uid;
-  pDst->tagVer = 1;
-  pDst->colVer = 1;
+  pDst->tagVer = (pCreate->source != TD_REQ_FROM_APP) ? pCreate->tagVer : 1;
+  pDst->colVer = (pCreate->source != TD_REQ_FROM_APP) ? pCreate->colVer : 1;
  pDst->smaVer = 1;
  pDst->nextColId = 1;
  pDst->maxdelay[0] = pCreate->delay1;
@ -869,9 +869,38 @@ static int32_t mndProcessCreateStbReq(SRpcMsg *pReq) {
  pStb = mndAcquireStb(pMnode, createReq.name);
  if (pStb != NULL) {
    if (createReq.igExists) {
-      mDebug("stb:%s, already exist, ignore exist is set", createReq.name);
-      code = 0;
-      goto _OVER;
+      if (createReq.source == TD_REQ_FROM_APP) {
+        mDebug("stb:%s, already exist, ignore exist is set", createReq.name);
+        code = 0;
+        goto _OVER;
+      } else if (pStb->uid != createReq.suid) {
+        mError("stb:%s, already exist while create, input suid:%" PRId64 " not match with exist suid:%" PRId64,
+               createReq.name, createReq.suid, pStb->uid);
+        terrno = TSDB_CODE_MND_STABLE_UID_NOT_MATCH;
+        goto _OVER;
+      } else if (createReq.tagVer > 0 || createReq.colVer > 0) {
+        int32_t tagDelta = pStb->tagVer - createReq.tagVer;
+        int32_t colDelta = pStb->colVer - createReq.colVer;
+        int32_t verDelta = tagDelta + verDelta;
+        mInfo("stb:%s, already exist while create, input tagVer:%d colVer:%d, exist tagVer:%d colVer:%d",
+              createReq.name, createReq.tagVer, createReq.colVer, pStb->tagVer, pStb->colVer);
+        if (tagDelta <= 0 && colDelta <= 0) {
+          mInfo("stb:%s, schema version is not incremented and nothing needs to be done", createReq.name);
+          code = 0;
+          goto _OVER;
+        } else if ((tagDelta == 1 || colDelta == 1) && (verDelta == 1)) {
+          mInfo("stb:%s, schema version is only increased by 1 number, do alter operation", createReq.name);
+        } else {
+          mError("stb:%s, schema version increase more than 1 number, error is returned", createReq.name);
+          terrno = TSDB_CODE_MND_INVALID_SCHEMA_VER;
+          goto _OVER;
+        }
+      } else {
+        mError("stb:%s, already exist while create, input tagVer:%d colVer:%d is invalid", createReq.name,
+               createReq.tagVer, createReq.colVer, pStb->tagVer, pStb->colVer);
+        terrno = TSDB_CODE_MND_INVALID_SCHEMA_VER;
+        goto _OVER;
+      }
    } else {
      terrno = TSDB_CODE_MND_STB_ALREADY_EXIST;
      goto _OVER;
@ -1614,14 +1643,6 @@ static int32_t mndProcessAlterStbReq(SRpcMsg *pReq) {
    goto _OVER;
  }

-  if ((alterReq.tagVer > 0 && alterReq.colVer > 0) &&
-      (alterReq.tagVer <= pStb->tagVer || alterReq.colVer <= pStb->colVer)) {
-    mDebug("stb:%s, already exist, tagVer:%d colVer:%d smaller than in mnode, tagVer:%d colVer:%d, alter success",
-           alterReq.name, alterReq.tagVer, alterReq.colVer, pStb->tagVer, pStb->colVer);
-    code = 0;
-    goto _OVER;
-  }
-
  if (mndCheckDbPrivilege(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) {
    goto _OVER;
  }
@ -1752,6 +1773,11 @@ static int32_t mndProcessDropStbReq(SRpcMsg *pReq) {
    }
  }

+  if (dropReq.source != TD_REQ_FROM_APP && pStb->uid != dropReq.suid) {
+    terrno = TSDB_CODE_MND_STB_NOT_EXIST;
+    goto _OVER;
+  }
+
  pDb = mndAcquireDbByStb(pMnode, dropReq.name);
  if (pDb == NULL) {
    terrno = TSDB_CODE_MND_DB_NOT_SELECTED;
--- a/source/dnode/mnode/impl/src/mndSubscribe.c
+++ b/source/dnode/mnode/impl/src/mndSubscribe.c
@ -235,7 +235,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
            .pVgEp = pVgEp,
        };
        taosHashPut(pHash, &pVgEp->vgId, sizeof(int32_t), &outputVg, sizeof(SMqRebOutputVg));
-        mInfo("mq rebalance: remove vg %d from consumer %ld", pVgEp->vgId, consumerId);
+        mInfo("mq rebalance: remove vgId:%d from consumer:%" PRId64, pVgEp->vgId, consumerId);
      }
      taosHashRemove(pOutput->pSub->consumerHash, &consumerId, sizeof(int64_t));
      // put into removed
@ -255,7 +255,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
          .pVgEp = pVgEp,
      };
      taosHashPut(pHash, &pVgEp->vgId, sizeof(int32_t), &rebOutput, sizeof(SMqRebOutputVg));
-      mInfo("mq rebalance: remove vg %d from unassigned", pVgEp->vgId);
+      mInfo("mq rebalance: remove vgId:%d from unassigned", pVgEp->vgId);
    }
  }

@ -298,7 +298,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
                .pVgEp = pVgEp,
            };
            taosHashPut(pHash, &pVgEp->vgId, sizeof(int32_t), &outputVg, sizeof(SMqRebOutputVg));
-            mInfo("mq rebalance: remove vg %d from consumer %ld (first scan)", pVgEp->vgId, pConsumerEp->consumerId);
+            mInfo("mq rebalance: remove vgId:%d from consumer:%" PRId64 ",(first scan)", pVgEp->vgId, pConsumerEp->consumerId);
          }
          imbCnt++;
        }
@ -312,7 +312,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
              .pVgEp = pVgEp,
          };
          taosHashPut(pHash, &pVgEp->vgId, sizeof(int32_t), &outputVg, sizeof(SMqRebOutputVg));
-          mInfo("mq rebalance: remove vg %d from consumer %ld (first scan)", pVgEp->vgId, pConsumerEp->consumerId);
+          mInfo("mq rebalance: remove vgId:%d from consumer:%" PRId64 ",(first scan)", pVgEp->vgId, pConsumerEp->consumerId);
        }
      }
    }
@ -329,7 +329,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
      newConsumerEp.vgs = taosArrayInit(0, sizeof(void *));
      taosHashPut(pOutput->pSub->consumerHash, &consumerId, sizeof(int64_t), &newConsumerEp, sizeof(SMqConsumerEp));
      taosArrayPush(pOutput->newConsumers, &consumerId);
-      mInfo("mq rebalance: add new consumer %ld", consumerId);
+      mInfo("mq rebalance: add new consumer:%" PRId64, consumerId);
    }
  }

@ -354,7 +354,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
      taosArrayPush(pConsumerEp->vgs, &pRebVg->pVgEp);
      pRebVg->newConsumerId = pConsumerEp->consumerId;
      taosArrayPush(pOutput->rebVgs, pRebVg);
-      mInfo("mq rebalance: add vg %d to consumer %ld (second scan)", pRebVg->pVgEp->vgId, pConsumerEp->consumerId);
+      mInfo("mq rebalance: add vgId:%d to consumer:%" PRId64 ",(second scan)", pRebVg->pVgEp->vgId, pConsumerEp->consumerId);
    }
  }

@ -372,7 +372,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
      taosArrayPush(pConsumerEp->vgs, &pRebVg->pVgEp);
      pRebVg->newConsumerId = pConsumerEp->consumerId;
      taosArrayPush(pOutput->rebVgs, pRebVg);
-      mInfo("mq rebalance: add vg %d to consumer %ld (second scan)", pRebVg->pVgEp->vgId, pConsumerEp->consumerId);
+      mInfo("mq rebalance: add vgId:%d to consumer:%" PRId64 ",(second scan)", pRebVg->pVgEp->vgId, pConsumerEp->consumerId);
    }
  } else {
    // if all consumer is removed, put all vg into unassigned
@ -385,7 +385,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
      ASSERT(pRebOutput->newConsumerId == -1);
      taosArrayPush(pOutput->pSub->unassignedVgs, &pRebOutput->pVgEp);
      taosArrayPush(pOutput->rebVgs, pRebOutput);
-      mInfo("mq rebalance: unassign vg %d (second scan)", pRebOutput->pVgEp->vgId);
+      mInfo("mq rebalance: unassign vgId:%d (second scan)", pRebOutput->pVgEp->vgId);
    }
  }

@ -393,7 +393,7 @@ static int32_t mndDoRebalance(SMnode *pMnode, const SMqRebInputObj *pInput, SMqR
  mInfo("rebalance calculation completed, rebalanced vg:");
  for (int32_t i = 0; i < taosArrayGetSize(pOutput->rebVgs); i++) {
    SMqRebOutputVg *pOutputRebVg = taosArrayGet(pOutput->rebVgs, i);
-    mInfo("vgId:%d, moved from consumer %" PRId64 " to consumer %" PRId64, pOutputRebVg->pVgEp->vgId,
+    mInfo("vgId:%d, moved from consumer:%" PRId64 ", to consumer:%" PRId64, pOutputRebVg->pVgEp->vgId,
          pOutputRebVg->oldConsumerId, pOutputRebVg->newConsumerId);
  }

@ -546,7 +546,11 @@ static int32_t mndProcessRebalanceReq(SRpcMsg *pMsg) {
      char cgroup[TSDB_CGROUP_LEN];
      mndSplitSubscribeKey(pRebInfo->key, topic, cgroup, true);
      SMqTopicObj *pTopic = mndAcquireTopic(pMnode, topic);
-      ASSERT(pTopic);
+      /*ASSERT(pTopic);*/
+      if (pTopic == NULL) {
+        mError("rebalance %s failed since topic %s was dropped, abort", pRebInfo->key, topic);
+        continue;
+      }
      taosRLockLatch(&pTopic->lock);

      rebOutput.pSub = mndCreateSub(pMnode, pTopic, pRebInfo->key);
--- a/source/dnode/mnode/impl/src/mndSync.c
+++ b/source/dnode/mnode/impl/src/mndSync.c
@ -188,7 +188,7 @@ int32_t mndInitSync(SMnode *pMnode) {
    SNodeInfo *pNode = &pCfg->nodeInfo[0];
    tstrncpy(pNode->nodeFqdn, pMgmt->replica.fqdn, sizeof(pNode->nodeFqdn));
    pNode->nodePort = pMgmt->replica.port;
-    mInfo("fqdn:%s port:%u", pNode->nodeFqdn, pNode->nodePort);
+    mInfo("mnode ep:%s:%u", pNode->nodeFqdn, pNode->nodePort);
  }

  tsem_init(&pMgmt->syncSem, 0, 0);
--- a/source/dnode/mnode/impl/src/mndTopic.c
+++ b/source/dnode/mnode/impl/src/mndTopic.c
@ -558,7 +558,7 @@ static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) {
        mndReleaseConsumer(pMnode, pConsumer);
        mndReleaseTopic(pMnode, pTopic);
        terrno = TSDB_CODE_MND_TOPIC_SUBSCRIBED;
-        mError("topic:%s, failed to drop since subscribed by consumer %ld in consumer group %s", dropReq.name,
+        mError("topic:%s, failed to drop since subscribed by consumer:%" PRId64 ", in consumer group %s", dropReq.name,
               pConsumer->consumerId, pConsumer->cgroup);
        return -1;
      }
--- a/source/dnode/mnode/impl/test/stb/stb.cpp
+++ b/source/dnode/mnode/impl/test/stb/stb.cpp
@ -277,8 +277,6 @@ void* MndTestStb::BuildAlterStbUpdateColumnBytesReq(const char* stbname, const c
  req.numOfFields = 1;
  req.pFields = taosArrayInit(1, sizeof(SField));
  req.alterType = TSDB_ALTER_TABLE_UPDATE_COLUMN_BYTES;
-  req.tagVer = verInBlock;
-  req.colVer = verInBlock;

  SField field = {0};
  field.bytes = bytes;
@ -818,7 +816,7 @@ TEST_F(MndTestStb, 08_Alter_Stb_AlterTagBytes) {
  {
    void*    pReq = BuildAlterStbUpdateColumnBytesReq(stbname, "col_not_exist", 20, &contLen, 1);
    SRpcMsg* pRsp = test.SendReq(TDMT_MND_ALTER_STB, pReq, contLen);
-    ASSERT_EQ(pRsp->code, 0);
+    ASSERT_EQ(pRsp->code, TSDB_CODE_MND_COLUMN_NOT_EXIST);

    test.SendShowReq(TSDB_MGMT_TABLE_STB, "user_stables", dbname);
    EXPECT_EQ(test.GetShowRows(), 1);
--- a/source/dnode/qnode/src/qnode.c
+++ b/source/dnode/qnode/src/qnode.c
@ -89,9 +89,6 @@ int32_t qndProcessQueryMsg(SQnode *pQnode, int64_t ts, SRpcMsg *pMsg) {
    case TDMT_SCH_MERGE_FETCH:
      code = qWorkerProcessFetchMsg(pQnode, pQnode->pQuery, pMsg, ts);
      break;
-    case TDMT_SCH_FETCH_RSP:
-      code = qWorkerProcessFetchRsp(pQnode, pQnode->pQuery, pMsg, ts);
-      break;
    case TDMT_SCH_CANCEL_TASK:
      code = qWorkerProcessCancelMsg(pQnode, pQnode->pQuery, pMsg, ts);
      break;
--- a/source/dnode/snode/src/snode.c
+++ b/source/dnode/snode/src/snode.c
@ -111,7 +111,7 @@ static int32_t sndProcessTaskDeployReq(SSnode *pNode, SRpcMsg *pMsg) {

  streamSetupTrigger(pTask);

-  qInfo("deploy stream: stream id %ld task id %d child id %d on snode", pTask->streamId, pTask->taskId,
+  qInfo("deploy stream: stream id %" PRId64 " task id %d child id %d on snode", pTask->streamId, pTask->taskId,
        pTask->selfChildId);

  taosHashPut(pMeta->pHash, &pTask->taskId, sizeof(int32_t), &pTask, sizeof(void *));
--- a/source/dnode/vnode/CMakeLists.txt
+++ b/source/dnode/vnode/CMakeLists.txt
@ -3,18 +3,17 @@ add_library(vnode STATIC "")
 target_sources(
    vnode
    PRIVATE
+
    # vnode
    "src/vnd/vnodeOpen.c"
    "src/vnd/vnodeBufPool.c"
    "src/vnd/vnodeCfg.c"
    "src/vnd/vnodeCommit.c"
    "src/vnd/vnodeQuery.c"
-    "src/vnd/vnodeStateMgr.c"
    "src/vnd/vnodeModule.c"
    "src/vnd/vnodeSvr.c"
    "src/vnd/vnodeSync.c"
    "src/vnd/vnodeSnapshot.c"
-    "src/vnd/vnodeUtil.c"

    # meta
    "src/meta/metaOpen.c"
@ -27,12 +26,10 @@ target_sources(
    "src/meta/metaSnapshot.c"

    # sma
-    "src/sma/sma.c"
    "src/sma/smaEnv.c"
    "src/sma/smaUtil.c"
    "src/sma/smaOpen.c"
    "src/sma/smaCommit.c"
-    "src/sma/smaSnapshot.c"
    "src/sma/smaRollup.c"
    "src/sma/smaTimeRange.c"

@ -43,13 +40,13 @@ target_sources(
    "src/tsdb/tsdbOpen.c"
    "src/tsdb/tsdbMemTable.c"
    "src/tsdb/tsdbRead.c"
-    "src/tsdb/tsdbReadImpl.c"
    "src/tsdb/tsdbCache.c"
    "src/tsdb/tsdbWrite.c"
    "src/tsdb/tsdbReaderWriter.c"
    "src/tsdb/tsdbUtil.c"
    "src/tsdb/tsdbSnapshot.c"
    "src/tsdb/tsdbCacheRead.c"
+    "src/tsdb/tsdbRetention.c"

    # tq
    "src/tq/tq.c"
@ -66,7 +63,6 @@ target_include_directories(
    PUBLIC "inc"
    PRIVATE "src/inc"
    PUBLIC "${TD_SOURCE_DIR}/include/libs/scalar"
-    
 )
 target_link_libraries(
    vnode
@ -80,18 +76,19 @@ target_link_libraries(
    PUBLIC executor
    PUBLIC scheduler
    PUBLIC tdb
-    #PUBLIC bdb
-    #PUBLIC scalar
+
+    # PUBLIC bdb
+    # PUBLIC scalar
    PUBLIC transport
    PUBLIC stream
    PUBLIC index
 )
 target_compile_definitions(vnode PUBLIC -DMETA_REFACT)
-if (${BUILD_WITH_INVERTEDINDEX})
-  add_definitions(-DUSE_INVERTED_INDEX) 
+
+if(${BUILD_WITH_INVERTEDINDEX})
+    add_definitions(-DUSE_INVERTED_INDEX)
 endif(${BUILD_WITH_INVERTEDINDEX})
+
 if(${BUILD_TEST})
    add_subdirectory(test)
 endif(${BUILD_TEST})
-
-
--- a/source/dnode/vnode/inc/vnode.h
+++ b/source/dnode/vnode/inc/vnode.h
@ -38,10 +38,11 @@ extern "C" {
 #endif

 // vnode
-typedef struct SVnode           SVnode;
-typedef struct STsdbCfg         STsdbCfg;  // todo: remove
-typedef struct SVnodeCfg        SVnodeCfg;
-typedef struct SVSnapshotReader SVSnapshotReader;
+typedef struct SVnode       SVnode;
+typedef struct STsdbCfg     STsdbCfg;  // todo: remove
+typedef struct SVnodeCfg    SVnodeCfg;
+typedef struct SVSnapReader SVSnapReader;
+typedef struct SVSnapWriter SVSnapWriter;

 extern const SVnodeCfg vnodeCfgDefault;

@ -57,10 +58,6 @@ void    vnodeStop(SVnode *pVnode);
 int64_t vnodeGetSyncHandle(SVnode *pVnode);
 void    vnodeGetSnapshot(SVnode *pVnode, SSnapshot *pSnapshot);
 void    vnodeGetInfo(SVnode *pVnode, const char **dbname, int32_t *vgId);
-int32_t vnodeSnapshotReaderOpen(SVnode *pVnode, SVSnapshotReader **ppReader, int64_t sver, int64_t ever);
-int32_t vnodeSnapshotReaderClose(SVSnapshotReader *pReader);
-int32_t vnodeSnapshotRead(SVSnapshotReader *pReader, const void **ppData, uint32_t *nData);
-
 int32_t vnodeProcessCreateTSma(SVnode *pVnode, void *pCont, uint32_t contLen);
 int32_t vnodeGetAllTableList(SVnode *pVnode, uint64_t uid, SArray *list);
 int32_t vnodeGetCtbIdList(SVnode *pVnode, int64_t suid, SArray *list);
@ -131,7 +128,7 @@ int32_t tsdbReaderOpen(SVnode *pVnode, SQueryTableDataCond *pCond, SArray *pTabl
 void    tsdbReaderClose(STsdbReader *pReader);
 bool    tsdbNextDataBlock(STsdbReader *pReader);
 void    tsdbRetrieveDataBlockInfo(STsdbReader *pReader, SDataBlockInfo *pDataBlockInfo);
-int32_t tsdbRetrieveDataBlockStatisInfo(STsdbReader *pReader, SColumnDataAgg ***pBlockStatis, bool *allHave);
+int32_t tsdbRetrieveDatablockSMA(STsdbReader *pReader, SColumnDataAgg ***pBlockStatis, bool *allHave);
 SArray *tsdbRetrieveDataBlock(STsdbReader *pTsdbReadHandle, SArray *pColumnIdList);
 int32_t tsdbReaderReset(STsdbReader *pReader, SQueryTableDataCond *pCond, int32_t tWinIdx);
 int32_t tsdbGetFileBlocksDistInfo(STsdbReader *pReader, STableBlockDistInfo *pTableBlockInfo);
@ -143,6 +140,7 @@ int32_t tsdbLastRowReaderOpen(void *pVnode, int32_t type, SArray *pTableIdList,
                              void **pReader);
 int32_t tsdbRetrieveLastRow(void *pReader, SSDataBlock *pResBlock, const int32_t *slotIds);
 int32_t tsdbLastrowReaderClose(void *pReader);
+int32_t tsdbGetTableSchema(SVnode* pVnode, int64_t uid, STSchema** pSchema, int64_t* suid);

 // tq

@ -173,6 +171,9 @@ int32_t tqReaderSetTbUidList(STqReader *pReader, const SArray *tbUidList);
 int32_t tqReaderAddTbUidList(STqReader *pReader, const SArray *tbUidList);
 int32_t tqReaderRemoveTbUidList(STqReader *pReader, const SArray *tbUidList);

+int32_t tqSeekVer(STqReader *pReader, int64_t ver);
+int32_t tqNextBlock(STqReader *pReader, SFetchRet *ret);
+
 int32_t tqReaderSetDataMsg(STqReader *pReader, SSubmitReq *pMsg, int64_t ver);
 bool    tqNextDataBlock(STqReader *pReader);
 bool    tqNextDataBlockFilterOut(STqReader *pReader, SHashObj *filterOutUids);
@ -181,7 +182,14 @@ int32_t tqRetrieveDataBlock(SSDataBlock *pBlock, STqReader *pReader);
 // sma
 int32_t smaGetTSmaDays(SVnodeCfg *pCfg, void *pCont, uint32_t contLen, int32_t *days);

-// need to reposition
+// SVSnapReader
+int32_t vnodeSnapReaderOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapReader **ppReader);
+int32_t vnodeSnapReaderClose(SVSnapReader *pReader);
+int32_t vnodeSnapRead(SVSnapReader *pReader, uint8_t **ppData, uint32_t *nData);
+// SVSnapWriter
+int32_t vnodeSnapWriterOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapWriter **ppWriter);
+int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback);
+int32_t vnodeSnapWrite(SVSnapWriter *pWriter, uint8_t *pData, uint32_t nData);

 // structs
 struct STsdbCfg {
--- a/source/dnode/vnode/src/inc/meta.h
+++ b/source/dnode/vnode/src/inc/meta.h
@ -57,6 +57,9 @@ int  metaRemoveTableFromIdx(SMeta* pMeta, tb_uid_t uid);
 // metaCommit ==================
 static FORCE_INLINE tb_uid_t metaGenerateUid(SMeta* pMeta) { return tGenIdPI64(); }

+// metaTable ==================
+int metaHandleEntry(SMeta* pMeta, const SMetaEntry* pME);
+
 struct SMeta {
  TdThreadRwlock lock;

--- a/source/dnode/vnode/src/inc/sma.h
+++ b/source/dnode/vnode/src/inc/sma.h
@ -47,7 +47,9 @@ struct SSmaEnv {
 };

 typedef struct {
-  int32_t smaRef;
+  int8_t  inited;
+  int32_t rsetId;
+  void   *tmrHandle;  // shared by all fetch tasks
 } SSmaMgmt;

 #define SMA_ENV_LOCK(env) ((env)->lock)
@ -64,7 +66,6 @@ struct SRSmaStat {
  SSma     *pSma;
  int64_t   submitVer;
  int64_t   refId;         // shared by fetch tasks
-  void     *tmrHandle;     // shared by fetch tasks
  int8_t    triggerStat;   // shared by fetch tasks
  int8_t    runningStat;   // for persistence task 
  SHashObj *rsmaInfoHash;  // key: stbUid, value: SRSmaInfo;
@ -81,7 +82,6 @@ struct SSmaStat {
 #define SMA_TSMA_STAT(s)     (&(s)->tsmaStat)
 #define SMA_RSMA_STAT(s)     (&(s)->rsmaStat)
 #define RSMA_INFO_HASH(r)    ((r)->rsmaInfoHash)
-#define RSMA_TMR_HANDLE(r)   ((r)->tmrHandle)
 #define RSMA_TRIGGER_STAT(r) (&(r)->triggerStat)
 #define RSMA_RUNNING_STAT(r) (&(r)->runningStat)
 #define RSMA_REF_ID(r)       ((r)->refId)
@ -95,6 +95,7 @@ enum {
  TASK_TRIGGER_STAT_CANCELLED = 4,
  TASK_TRIGGER_STAT_FINISHED = 5,
 };
+
 void  tdDestroySmaEnv(SSmaEnv *pSmaEnv);
 void *tdFreeSmaEnv(SSmaEnv *pSmaEnv);

@ -104,6 +105,10 @@ int32_t tdInsertRSmaData(SSma *pSma, char *msg);

 int32_t tdRefSmaStat(SSma *pSma, SSmaStat *pStat);
 int32_t tdUnRefSmaStat(SSma *pSma, SSmaStat *pStat);
+
+void   *tdAcquireSmaRef(int32_t rsetId, int64_t refId, const char *tags, int32_t ln);
+int32_t tdReleaseSmaRef(int32_t rsetId, int64_t refId, const char *tags, int32_t ln);
+
 int32_t tdCheckAndInitSmaEnv(SSma *pSma, int8_t smaType);

 int32_t tdLockSma(SSma *pSma);
@ -183,7 +188,7 @@ static FORCE_INLINE void tdSmaStatSetDropped(STSmaStat *pTStat) {

 static int32_t tdDestroySmaState(SSmaStat *pSmaStat, int8_t smaType);
 void          *tdFreeSmaState(SSmaStat *pSmaStat, int8_t smaType);
-void          *tdFreeRSmaInfo(SRSmaInfo *pInfo);
+void          *tdFreeRSmaInfo(SSma *pSma, SRSmaInfo *pInfo);
 int32_t        tdRSmaPersistExecImpl(SRSmaStat *pRSmaStat);

 int32_t tdProcessRSmaCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, const char *tbName);
--- a/source/dnode/vnode/src/inc/tq.h
+++ b/source/dnode/vnode/src/inc/tq.h
@ -129,6 +129,7 @@ typedef struct {
 static STqMgmt tqMgmt = {0};

 // tqRead
+int64_t tqScanLog(STQ* pTq, const STqExecHandle* pExec, SMqDataRsp* pRsp, STqOffsetVal* offset);
 int64_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, SWalCkHead** pHeadWithCkSum);

 // tqExec
--- a/source/dnode/vnode/src/inc/tsdb.h
+++ b/source/dnode/vnode/src/inc/tsdb.h
@ -64,6 +64,7 @@ typedef struct SRowIter       SRowIter;
 typedef struct STsdbFS        STsdbFS;
 typedef struct SRowMerger     SRowMerger;
 typedef struct STsdbFSState   STsdbFSState;
+typedef struct STsdbSnapHdr   STsdbSnapHdr;

 #define TSDB_MAX_SUBBLOCKS 8
 #define TSDB_FHDR_SIZE     512
@ -245,6 +246,8 @@ int32_t tsdbCacheRelease(SLRUCache *pCache, LRUHandle *h);

 int32_t tsdbCacheDelete(SLRUCache *pCache, tb_uid_t uid, TSKEY eKey);

+int32_t tsdbCacheLastArray2Row(SArray *pLastArray, STSRow **ppRow, STSchema *pSchema);
+
 // structs =======================
 typedef struct {
  int   minFid;
--- a/source/dnode/vnode/src/inc/vnodeInt.h
+++ b/source/dnode/vnode/src/inc/vnodeInt.h
@ -49,17 +49,20 @@
 extern "C" {
 #endif

-typedef struct SVnodeInfo          SVnodeInfo;
-typedef struct SMeta               SMeta;
-typedef struct SSma                SSma;
-typedef struct STsdb               STsdb;
-typedef struct STQ                 STQ;
-typedef struct SVState             SVState;
-typedef struct SVBufPool           SVBufPool;
-typedef struct SQWorker            SQHandle;
-typedef struct STsdbKeepCfg        STsdbKeepCfg;
-typedef struct SMetaSnapshotReader SMetaSnapshotReader;
-typedef struct STsdbSnapshotReader STsdbSnapshotReader;
+typedef struct SVnodeInfo      SVnodeInfo;
+typedef struct SMeta           SMeta;
+typedef struct SSma            SSma;
+typedef struct STsdb           STsdb;
+typedef struct STQ             STQ;
+typedef struct SVState         SVState;
+typedef struct SVBufPool       SVBufPool;
+typedef struct SQWorker        SQHandle;
+typedef struct STsdbKeepCfg    STsdbKeepCfg;
+typedef struct SMetaSnapReader SMetaSnapReader;
+typedef struct SMetaSnapWriter SMetaSnapWriter;
+typedef struct STsdbSnapReader STsdbSnapReader;
+typedef struct STsdbSnapWriter STsdbSnapWriter;
+typedef struct SSnapDataHdr    SSnapDataHdr;

 #define VNODE_META_DIR  "meta"
 #define VNODE_TSDB_DIR  "tsdb"
@ -72,10 +75,8 @@ typedef struct STsdbSnapshotReader STsdbSnapshotReader;
 #define VNODE_RSMA2_DIR "rsma2"

 // vnd.h
-void*   vnodeBufPoolMalloc(SVBufPool* pPool, int size);
-void    vnodeBufPoolFree(SVBufPool* pPool, void* p);
-int32_t vnodeRealloc(void** pp, int32_t size);
-void    vnodeFree(void* p);
+void* vnodeBufPoolMalloc(SVBufPool* pPool, int size);
+void  vnodeBufPoolFree(SVBufPool* pPool, void* p);

 // meta
 typedef struct SMCtbCursor SMCtbCursor;
@ -109,9 +110,6 @@ STSma*          metaGetSmaInfoByIndex(SMeta* pMeta, int64_t indexUid);
 STSmaWrapper*   metaGetSmaInfoByTable(SMeta* pMeta, tb_uid_t uid, bool deepCopy);
 SArray*         metaGetSmaIdsByTable(SMeta* pMeta, tb_uid_t uid);
 SArray*         metaGetSmaTbUids(SMeta* pMeta);
-int32_t         metaSnapshotReaderOpen(SMeta* pMeta, SMetaSnapshotReader** ppReader, int64_t sver, int64_t ever);
-int32_t         metaSnapshotReaderClose(SMetaSnapshotReader* pReader);
-int32_t         metaSnapshotRead(SMetaSnapshotReader* pReader, void** ppData, uint32_t* nData);
 void*           metaGetIdx(SMeta* pMeta);
 void*           metaGetIvtIdx(SMeta* pMeta);
 int             metaTtlSmaller(SMeta* pMeta, uint64_t time, SArray* uidList);
@ -131,9 +129,6 @@ int32_t     tsdbInsertTableData(STsdb* pTsdb, int64_t version, SSubmitMsgIter* p
 int32_t     tsdbDeleteTableData(STsdb* pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey);
 STsdbReader tsdbQueryCacheLastT(STsdb* tsdb, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId,
                                void* pMemRef);
-int32_t     tsdbSnapshotReaderOpen(STsdb* pTsdb, STsdbSnapshotReader** ppReader, int64_t sver, int64_t ever);
-int32_t     tsdbSnapshotReaderClose(STsdbSnapshotReader* pReader);
-int32_t     tsdbSnapshotRead(STsdbSnapshotReader* pReader, void** ppData, uint32_t* nData);

 // tq
 int     tqInit();
@ -163,6 +158,8 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pSchema, bool
                            const char* stbFullName, int32_t vgId);

 // sma
+int32_t smaInit();
+void    smaCleanUp();
 int32_t smaOpen(SVnode* pVnode);
 int32_t smaClose(SSma* pSma);
 int32_t smaBegin(SSma* pSma);
@ -181,6 +178,23 @@ int32_t tdUpdateTbUidList(SSma* pSma, STbUidStore* pUidStore);
 void    tdUidStoreDestory(STbUidStore* pStore);
 void*   tdUidStoreFree(STbUidStore* pStore);

+// SMetaSnapReader ========================================
+int32_t metaSnapReaderOpen(SMeta* pMeta, int64_t sver, int64_t ever, SMetaSnapReader** ppReader);
+int32_t metaSnapReaderClose(SMetaSnapReader** ppReader);
+int32_t metaSnapRead(SMetaSnapReader* pReader, uint8_t** ppData);
+// SMetaSnapWriter ========================================
+int32_t metaSnapWriterOpen(SMeta* pMeta, int64_t sver, int64_t ever, SMetaSnapWriter** ppWriter);
+int32_t metaSnapWrite(SMetaSnapWriter* pWriter, uint8_t* pData, uint32_t nData);
+int32_t metaSnapWriterClose(SMetaSnapWriter** ppWriter, int8_t rollback);
+// STsdbSnapReader ========================================
+int32_t tsdbSnapReaderOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapReader** ppReader);
+int32_t tsdbSnapReaderClose(STsdbSnapReader** ppReader);
+int32_t tsdbSnapRead(STsdbSnapReader* pReader, uint8_t** ppData);
+// STsdbSnapWriter ========================================
+int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWriter** ppWriter);
+int32_t tsdbSnapWrite(STsdbSnapWriter* pWriter, uint8_t* pData, uint32_t nData);
+int32_t tsdbSnapWriterClose(STsdbSnapWriter** ppWriter, int8_t rollback);
+
 typedef struct {
  int8_t  streamType;  // sma or other
  int8_t  dstType;
@ -200,7 +214,9 @@ typedef struct {
 struct SVState {
  int64_t committed;
  int64_t applied;
+  int64_t applyTerm;
  int64_t commitID;
+  int64_t commitTerm;
 };

 struct SVnodeInfo {
@ -289,6 +305,12 @@ struct SSma {
 // sma
 void smaHandleRes(void* pVnode, int64_t smaId, const SArray* data);

+struct SSnapDataHdr {
+  int8_t  type;
+  int64_t size;
+  uint8_t data[];
+};
+
 #ifdef __cplusplus
 }
 #endif
--- a/source/dnode/vnode/src/meta/metaSnapshot.c
+++ b/source/dnode/vnode/src/meta/metaSnapshot.c
@ -15,53 +15,57 @@

 #include "meta.h"

-struct SMetaSnapshotReader {
+// SMetaSnapReader ========================================
+struct SMetaSnapReader {
  SMeta*  pMeta;
-  TBC*    pTbc;
  int64_t sver;
  int64_t ever;
+  TBC*    pTbc;
 };

-int32_t metaSnapshotReaderOpen(SMeta* pMeta, SMetaSnapshotReader** ppReader, int64_t sver, int64_t ever) {
-  int32_t              code = 0;
-  int32_t              c = 0;
-  SMetaSnapshotReader* pMetaReader = NULL;
+int32_t metaSnapReaderOpen(SMeta* pMeta, int64_t sver, int64_t ever, SMetaSnapReader** ppReader) {
+  int32_t          code = 0;
+  int32_t          c = 0;
+  SMetaSnapReader* pMetaSnapReader = NULL;

-  pMetaReader = (SMetaSnapshotReader*)taosMemoryCalloc(1, sizeof(*pMetaReader));
-  if (pMetaReader == NULL) {
+  // alloc
+  pMetaSnapReader = (SMetaSnapReader*)taosMemoryCalloc(1, sizeof(*pMetaSnapReader));
+  if (pMetaSnapReader == NULL) {
    code = TSDB_CODE_OUT_OF_MEMORY;
    goto _err;
  }
-  pMetaReader->pMeta = pMeta;
-  pMetaReader->sver = sver;
-  pMetaReader->ever = ever;
-  code = tdbTbcOpen(pMeta->pTbDb, &pMetaReader->pTbc, NULL);
+  pMetaSnapReader->pMeta = pMeta;
+  pMetaSnapReader->sver = sver;
+  pMetaSnapReader->ever = ever;
+
+  // impl
+  code = tdbTbcOpen(pMeta->pTbDb, &pMetaSnapReader->pTbc, NULL);
  if (code) {
    goto _err;
  }

-  code = tdbTbcMoveTo(pMetaReader->pTbc, &(STbDbKey){.version = sver, .uid = INT64_MIN}, sizeof(STbDbKey), &c);
+  code = tdbTbcMoveTo(pMetaSnapReader->pTbc, &(STbDbKey){.version = sver, .uid = INT64_MIN}, sizeof(STbDbKey), &c);
  if (code) {
    goto _err;
  }

-  *ppReader = pMetaReader;
+  *ppReader = pMetaSnapReader;
  return code;

 _err:
+  metaError("vgId:%d meta snap reader open failed since %s", TD_VID(pMeta->pVnode), tstrerror(code));
  *ppReader = NULL;
  return code;
 }

-int32_t metaSnapshotReaderClose(SMetaSnapshotReader* pReader) {
-  if (pReader) {
-    tdbTbcClose(pReader->pTbc);
-    taosMemoryFree(pReader);
-  }
+int32_t metaSnapReaderClose(SMetaSnapReader** ppReader) {
+  tdbTbcClose((*ppReader)->pTbc);
+  taosMemoryFree(*ppReader);
+  *ppReader = NULL;
  return 0;
 }

-int32_t metaSnapshotRead(SMetaSnapshotReader* pReader, void** ppData, uint32_t* nDatap) {
+int32_t metaSnapRead(SMetaSnapReader* pReader, uint8_t** ppData) {
  const void* pKey = NULL;
  const void* pData = NULL;
  int32_t     nKey = 0;
@ -71,23 +75,110 @@ int32_t metaSnapshotRead(SMetaSnapshotReader* pReader, void** ppData, uint32_t*
  for (;;) {
    code = tdbTbcGet(pReader->pTbc, &pKey, &nKey, &pData, &nData);
    if (code || ((STbDbKey*)pData)->version > pReader->ever) {
-      return TSDB_CODE_VND_READ_END;
+      code = TSDB_CODE_VND_READ_END;
+      goto _exit;
    }

    if (((STbDbKey*)pData)->version < pReader->sver) {
+      tdbTbcMoveToNext(pReader->pTbc);
      continue;
    }

+    tdbTbcMoveToNext(pReader->pTbc);
    break;
  }

  // copy the data
-  if (vnodeRealloc(ppData, nData) < 0) {
+  if (tRealloc(ppData, sizeof(SSnapDataHdr) + nData) < 0) {
    code = TSDB_CODE_OUT_OF_MEMORY;
    return code;
  }
+  ((SSnapDataHdr*)(*ppData))->type = 0;  // TODO: use macro
+  ((SSnapDataHdr*)(*ppData))->size = nData;
+  memcpy(((SSnapDataHdr*)(*ppData))->data, pData, nData);

-  memcpy(*ppData, pData, nData);
-  *nDatap = nData;
+_exit:
+  return code;
+}
+
+// SMetaSnapWriter ========================================
+struct SMetaSnapWriter {
+  SMeta*  pMeta;
+  int64_t sver;
+  int64_t ever;
+};
+
+static int32_t metaSnapRollback(SMetaSnapWriter* pWriter) {
+  int32_t code = 0;
+  // TODO
+  return code;
+}
+
+static int32_t metaSnapCommit(SMetaSnapWriter* pWriter) {
+  int32_t code = 0;
+  // TODO
+  return code;
+}
+
+int32_t metaSnapWriterOpen(SMeta* pMeta, int64_t sver, int64_t ever, SMetaSnapWriter** ppWriter) {
+  int32_t          code = 0;
+  SMetaSnapWriter* pWriter;
+
+  // alloc
+  pWriter = (SMetaSnapWriter*)taosMemoryCalloc(1, sizeof(*pWriter));
+  if (pWriter == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+  pWriter->pMeta = pMeta;
+  pWriter->sver = sver;
+  pWriter->ever = ever;
+
+  *ppWriter = pWriter;
+  return code;
+
+_err:
+  metaError("vgId:%d meta snapshot writer open failed since %s", TD_VID(pMeta->pVnode), tstrerror(code));
+  *ppWriter = NULL;
+  return code;
+}
+
+int32_t metaSnapWriterClose(SMetaSnapWriter** ppWriter, int8_t rollback) {
+  int32_t          code = 0;
+  SMetaSnapWriter* pWriter = *ppWriter;
+
+  if (rollback) {
+    code = metaSnapRollback(pWriter);
+    if (code) goto _err;
+  } else {
+    code = metaSnapCommit(pWriter);
+    if (code) goto _err;
+  }
+  taosMemoryFree(pWriter);
+  *ppWriter = NULL;
+
+  return code;
+
+_err:
+  metaError("vgId:%d meta snapshot writer close failed since %s", TD_VID(pWriter->pMeta->pVnode), tstrerror(code));
+  return code;
+}
+
+int32_t metaSnapWrite(SMetaSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  int32_t    code = 0;
+  SMeta*     pMeta = pWriter->pMeta;
+  SMetaEntry metaEntry = {0};
+  SDecoder*  pDecoder = &(SDecoder){0};
+
+  tDecoderInit(pDecoder, pData, nData);
+  metaDecodeEntry(pDecoder, &metaEntry);
+
+  code = metaHandleEntry(pMeta, &metaEntry);
+  if (code) goto _err;
+
+  return code;
+
+_err:
+  metaError("vgId:%d meta snapshot write failed since %s", TD_VID(pMeta->pVnode), tstrerror(code));
  return code;
 }
--- a/source/dnode/vnode/src/meta/metaTable.c
+++ b/source/dnode/vnode/src/meta/metaTable.c
@ -17,7 +17,6 @@

 static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSchema *pSchema);
 static int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSchema *pSchema);
-static int metaHandleEntry(SMeta *pMeta, const SMetaEntry *pME);
 static int metaSaveToTbDb(SMeta *pMeta, const SMetaEntry *pME);
 static int metaUpdateUidIdx(SMeta *pMeta, const SMetaEntry *pME);
 static int metaUpdateNameIdx(SMeta *pMeta, const SMetaEntry *pME);
@ -51,7 +50,7 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const
  if (pMeta->pTagIvtIdx == NULL || pCtbEntry == NULL) {
    return -1;
  }
-  void *      data = pCtbEntry->ctbEntry.pTags;
+  void       *data = pCtbEntry->ctbEntry.pTags;
  const char *tagName = pSchema->name;

  tb_uid_t    suid = pCtbEntry->ctbEntry.suid;
@ -70,7 +69,7 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const
    STagVal *pTagVal = (STagVal *)taosArrayGet(pTagVals, i);
    char     type = pTagVal->type;

-    char *  key = pTagVal->pKey;
+    char   *key = pTagVal->pKey;
    int32_t nKey = strlen(key);

    SIndexTerm *term = NULL;
@ -78,7 +77,7 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const
      term = indexTermCreate(suid, ADD_VALUE, TSDB_DATA_TYPE_VARCHAR, key, nKey, NULL, 0);
    } else if (type == TSDB_DATA_TYPE_NCHAR) {
      if (pTagVal->nData > 0) {
-        char *  val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE);
+        char   *val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE);
        int32_t len = taosUcs4ToMbs((TdUcs4 *)pTagVal->pData, pTagVal->nData, val + VARSTR_HEADER_SIZE);
        memcpy(val, (uint16_t *)&len, VARSTR_HEADER_SIZE);
        type = TSDB_DATA_TYPE_VARCHAR;
@ -109,7 +108,7 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche
  if (pMeta->pTagIvtIdx == NULL || pCtbEntry == NULL) {
    return -1;
  }
-  void *      data = pCtbEntry->ctbEntry.pTags;
+  void       *data = pCtbEntry->ctbEntry.pTags;
  const char *tagName = pSchema->name;

  tb_uid_t    suid = pCtbEntry->ctbEntry.suid;
@ -128,7 +127,7 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche
    STagVal *pTagVal = (STagVal *)taosArrayGet(pTagVals, i);
    char     type = pTagVal->type;

-    char *  key = pTagVal->pKey;
+    char   *key = pTagVal->pKey;
    int32_t nKey = strlen(key);

    SIndexTerm *term = NULL;
@ -136,7 +135,7 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche
      term = indexTermCreate(suid, DEL_VALUE, TSDB_DATA_TYPE_VARCHAR, key, nKey, NULL, 0);
    } else if (type == TSDB_DATA_TYPE_NCHAR) {
      if (pTagVal->nData > 0) {
-        char *  val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE);
+        char   *val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE);
        int32_t len = taosUcs4ToMbs((TdUcs4 *)pTagVal->pData, pTagVal->nData, val + VARSTR_HEADER_SIZE);
        memcpy(val, (uint16_t *)&len, VARSTR_HEADER_SIZE);
        type = TSDB_DATA_TYPE_VARCHAR;
@ -169,9 +168,9 @@ int metaCreateSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
  int         vLen = 0;
  const void *pKey = NULL;
  const void *pVal = NULL;
-  void *      pBuf = NULL;
+  void       *pBuf = NULL;
  int32_t     szBuf = 0;
-  void *      p = NULL;
+  void       *p = NULL;
  SMetaReader mr = {0};

  // validate req
@ -229,7 +228,7 @@ int metaDropSTable(SMeta *pMeta, int64_t verison, SVDropStbReq *pReq) {
  }

  // drop all child tables
-  TBC *   pCtbIdxc = NULL;
+  TBC    *pCtbIdxc = NULL;
  SArray *pArray = taosArrayInit(8, sizeof(tb_uid_t));

  tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, &pMeta->txn);
@ -285,8 +284,8 @@ _exit:
 int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
  SMetaEntry  oStbEntry = {0};
  SMetaEntry  nStbEntry = {0};
-  TBC *       pUidIdxc = NULL;
-  TBC *       pTbDbc = NULL;
+  TBC        *pUidIdxc = NULL;
+  TBC        *pTbDbc = NULL;
  const void *pData;
  int         nData;
  int64_t     oversion;
@ -409,7 +408,7 @@ _err:
 }

 int metaDropTable(SMeta *pMeta, int64_t version, SVDropTbReq *pReq, SArray *tbUids) {
-  void *   pData = NULL;
+  void    *pData = NULL;
  int      nData = 0;
  int      rc = 0;
  tb_uid_t uid;
@ -477,7 +476,7 @@ static int metaDeleteTtlIdx(SMeta *pMeta, const SMetaEntry *pME) {
 }

 static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type) {
-  void *     pData = NULL;
+  void      *pData = NULL;
  int        nData = 0;
  int        rc = 0;
  SMetaEntry e = {0};
@ -538,14 +537,14 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type) {
 }

 static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq, STableMetaRsp *pMetaRsp) {
-  void *          pVal = NULL;
+  void           *pVal = NULL;
  int             nVal = 0;
-  const void *    pData = NULL;
+  const void     *pData = NULL;
  int             nData = 0;
  int             ret = 0;
  tb_uid_t        uid;
  int64_t         oversion;
-  SSchema *       pColumn = NULL;
+  SSchema        *pColumn = NULL;
  SMetaEntry      entry = {0};
  SSchemaWrapper *pSchema;
  int             c;
@ -699,7 +698,7 @@ _err:
 static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq) {
  SMetaEntry  ctbEntry = {0};
  SMetaEntry  stbEntry = {0};
-  void *      pVal = NULL;
+  void       *pVal = NULL;
  int         nVal = 0;
  int         ret;
  int         c;
@ -730,7 +729,7 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA
  oversion = *(int64_t *)pData;

  // search table.db
-  TBC *    pTbDbc = NULL;
+  TBC     *pTbDbc = NULL;
  SDecoder dc1 = {0};
  SDecoder dc2 = {0};

@ -754,7 +753,7 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA
  metaDecodeEntry(&dc2, &stbEntry);

  SSchemaWrapper *pTagSchema = &stbEntry.stbEntry.schemaTag;
-  SSchema *       pColumn = NULL;
+  SSchema        *pColumn = NULL;
  int32_t         iCol = 0;
  for (;;) {
    pColumn = NULL;
@ -784,8 +783,8 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA
    memcpy((void *)ctbEntry.ctbEntry.pTags, pAlterTbReq->pTagVal, pAlterTbReq->nTagVal);
  } else {
    const STag *pOldTag = (const STag *)ctbEntry.ctbEntry.pTags;
-    STag *      pNewTag = NULL;
-    SArray *    pTagArray = taosArrayInit(pTagSchema->nCols, sizeof(STagVal));
+    STag       *pNewTag = NULL;
+    SArray     *pTagArray = taosArrayInit(pTagSchema->nCols, sizeof(STagVal));
    if (!pTagArray) {
      terrno = TSDB_CODE_OUT_OF_MEMORY;
      goto _err;
@ -844,7 +843,7 @@ _err:
 }

 static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq) {
-  void *      pVal = NULL;
+  void       *pVal = NULL;
  int         nVal = 0;
  const void *pData = NULL;
  int         nData = 0;
@ -948,8 +947,8 @@ int metaAlterTable(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMeta

 static int metaSaveToTbDb(SMeta *pMeta, const SMetaEntry *pME) {
  STbDbKey tbDbKey;
-  void *   pKey = NULL;
-  void *   pVal = NULL;
+  void    *pKey = NULL;
+  void    *pVal = NULL;
  int      kLen = 0;
  int      vLen = 0;
  SEncoder coder = {0};
@ -1055,14 +1054,14 @@ static void metaDestroyTagIdxKey(STagIdxKey *pTagIdxKey) {
 }

 static int metaUpdateTagIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry) {
-  void *         pData = NULL;
+  void          *pData = NULL;
  int            nData = 0;
  STbDbKey       tbDbKey = {0};
  SMetaEntry     stbEntry = {0};
-  STagIdxKey *   pTagIdxKey = NULL;
+  STagIdxKey    *pTagIdxKey = NULL;
  int32_t        nTagIdxKey;
  const SSchema *pTagColumn;       // = &stbEntry.stbEntry.schema.pSchema[0];
-  const void *   pTagData = NULL;  //
+  const void    *pTagData = NULL;  //
  int32_t        nTagData = 0;
  SDecoder       dc = {0};

@ -1109,7 +1108,7 @@ static int metaUpdateTagIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry) {

 static int metaSaveToSkmDb(SMeta *pMeta, const SMetaEntry *pME) {
  SEncoder              coder = {0};
-  void *                pVal = NULL;
+  void                 *pVal = NULL;
  int                   vLen = 0;
  int                   rcode = 0;
  SSkmDbKey             skmDbKey = {0};
@ -1151,7 +1150,7 @@ _exit:
  return rcode;
 }

-static int metaHandleEntry(SMeta *pMeta, const SMetaEntry *pME) {
+int metaHandleEntry(SMeta *pMeta, const SMetaEntry *pME) {
  metaWLock(pMeta);

  // save to table.db
--- a/source/dnode/vnode/src/sma/sma.c
+++ b/source/dnode/vnode/src/sma/sma.c
@ -1,257 +0,0 @@
-/*
- * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
- *
- * This program is free software: you can use, redistribute, and/or modify
- * it under the terms of the GNU Affero General Public License, version 3
- * or later ("AGPL"), as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-#include "sma.h"
-
-// functions for external invocation
-
-// TODO: Who is responsible for resource allocate and release?
-int32_t tdProcessTSmaInsert(SSma* pSma, int64_t indexUid, const char* msg) {
-  int32_t code = TSDB_CODE_SUCCESS;
-
-  if ((code = tdProcessTSmaInsertImpl(pSma, indexUid, msg)) < 0) {
-    smaWarn("vgId:%d, insert tsma data failed since %s", SMA_VID(pSma), tstrerror(terrno));
-  }
-  // TODO: destroy SSDataBlocks(msg)
-  return code;
-}
-
-int32_t tdProcessTSmaCreate(SSma* pSma, int64_t version, const char* msg) {
-  int32_t code = TSDB_CODE_SUCCESS;
-
-  if ((code = tdProcessTSmaCreateImpl(pSma, version, msg)) < 0) {
-    smaWarn("vgId:%d, create tsma failed since %s", SMA_VID(pSma), tstrerror(terrno));
-  }
-  // TODO: destroy SSDataBlocks(msg)
-  return code;
-}
-
-int32_t smaGetTSmaDays(SVnodeCfg* pCfg, void* pCont, uint32_t contLen, int32_t* days) {
-  int32_t code = TSDB_CODE_SUCCESS;
-  if ((code = tdProcessTSmaGetDaysImpl(pCfg, pCont, contLen, days)) < 0) {
-    smaWarn("vgId:%d, get tsma days failed since %s", pCfg->vgId, tstrerror(terrno));
-  }
-  smaDebug("vgId:%d, get tsma days %d", pCfg->vgId, *days);
-  return code;
-}
-
-
-// functions for internal invocation
-
-#if 0
-
-/**
- * @brief TODO: Assume that the final generated result it less than 3M
- *
- * @param pReq
- * @param pDataBlocks
- * @param vgId
- * @param suid  // TODO: check with Liao whether suid response is reasonable
- *
- * TODO: colId should be set
- */
-int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks, STSchema* pTSchema, int32_t vgId,
-                                    tb_uid_t suid, const char* stbName, bool isCreateCtb) {
-  int32_t sz = taosArrayGetSize(pDataBlocks);
-  int32_t bufSize = sizeof(SSubmitReq);
-  for (int32_t i = 0; i < sz; ++i) {
-    SDataBlockInfo* pBlkInfo = &((SSDataBlock*)taosArrayGet(pDataBlocks, i))->info;
-    bufSize += pBlkInfo->rows * (TD_ROW_HEAD_LEN + pBlkInfo->rowSize + BitmapLen(pBlkInfo->numOfCols));
-    bufSize += sizeof(SSubmitBlk);
-  }
-
-  *pReq = taosMemoryCalloc(1, bufSize);
-  if (!(*pReq)) {
-    terrno = TSDB_CODE_OUT_OF_MEMORY;
-    return TSDB_CODE_FAILED;
-  }
-  void* pDataBuf = *pReq;
-
-  SArray*     pTagArray = NULL;
-  int32_t     msgLen = sizeof(SSubmitReq);
-  int32_t     numOfBlks = 0;
-  int32_t     schemaLen = 0;
-  SRowBuilder rb = {0};
-  tdSRowInit(&rb, pTSchema->version);
-
-  for (int32_t i = 0; i < sz; ++i) {
-    SSDataBlock*    pDataBlock = taosArrayGet(pDataBlocks, i);
-    SDataBlockInfo* pDataBlkInfo = &pDataBlock->info;
-    int32_t         colNum = pDataBlkInfo->numOfCols;
-    int32_t         rows = pDataBlkInfo->rows;
-    int32_t         rowSize = pDataBlkInfo->rowSize;
-    int64_t         groupId = pDataBlkInfo->groupId;
-
-    if (rb.nCols != colNum) {
-      tdSRowSetTpInfo(&rb, colNum, pTSchema->flen);
-    }
-
-   if(isCreateCtb) {
-     SMetaReader  mr = {0};
-     const char*  ctbName = buildCtbNameByGroupId(stbName, pDataBlock->info.groupId);
-     if (metaGetTableEntryByName(&mr, ctbName) != 0) {
-       smaDebug("vgId:%d, no tsma ctb %s exists", vgId, ctbName);
-     }
-     SVCreateTbReq ctbReq = {0};
-     ctbReq.name = ctbName;
-     ctbReq.type = TSDB_CHILD_TABLE;
-     ctbReq.ctb.suid = suid;
-
-     STagVal tagVal = {.cid = colNum + PRIMARYKEY_TIMESTAMP_COL_ID,
-                       .type = TSDB_DATA_TYPE_BIGINT,
-                       .i64 = groupId};
-     STag*   pTag = NULL;
-     if(!pTagArray) {
-       pTagArray = taosArrayInit(1, sizeof(STagVal));
-       if (!pTagArray) goto _err;
-     }
-     taosArrayClear(pTagArray);
-     taosArrayPush(pTagArray, &tagVal);
-     tTagNew(pTagArray, 1, false, &pTag);
-     if (pTag == NULL) {
-       tdDestroySVCreateTbReq(&ctbReq);
-       goto _err;
-      }
-      ctbReq.ctb.pTag = (uint8_t*)pTag;
-
-      int32_t code;
-      tEncodeSize(tEncodeSVCreateTbReq, &ctbReq, schemaLen, code);
-
-      tdDestroySVCreateTbReq(&ctbReq);
-      if (code < 0) {
-        goto _err;
-      }
-   }
-
-
-
-    SSubmitBlk* pSubmitBlk = POINTER_SHIFT(pDataBuf, msgLen);
-    pSubmitBlk->suid = suid;
-    pSubmitBlk->uid = groupId;
-    pSubmitBlk->numOfRows = rows;
-
-    msgLen += sizeof(SSubmitBlk);
-    int32_t dataLen = 0;
-    for (int32_t j = 0; j < rows; ++j) {                     // iterate by row
-      tdSRowResetBuf(&rb, POINTER_SHIFT(pDataBuf, msgLen));  // set row buf
-      bool    isStartKey = false;
-      int32_t offset = 0;
-      for (int32_t k = 0; k < colNum; ++k) {  // iterate by column
-        SColumnInfoData* pColInfoData = taosArrayGet(pDataBlock->pDataBlock, k);
-        STColumn*        pCol = &pTSchema->columns[k];
-        void*            var = POINTER_SHIFT(pColInfoData->pData, j * pColInfoData->info.bytes);
-        switch (pColInfoData->info.type) {
-          case TSDB_DATA_TYPE_TIMESTAMP:
-            if (!isStartKey) {
-              isStartKey = true;
-              tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID, TSDB_DATA_TYPE_TIMESTAMP, TD_VTYPE_NORM, var, true,
-                                  offset, k);
-
-            } else {
-              tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_TIMESTAMP, TD_VTYPE_NORM, var,
-                                  true, offset, k);
-            }
-            break;
-          case TSDB_DATA_TYPE_NCHAR: {
-            tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_NCHAR, TD_VTYPE_NORM, var, true,
-                                offset, k);
-            break;
-          }
-          case TSDB_DATA_TYPE_VARCHAR: {  // TSDB_DATA_TYPE_BINARY
-            tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_VARCHAR, TD_VTYPE_NORM, var, true,
-                                offset, k);
-            break;
-          }
-          case TSDB_DATA_TYPE_VARBINARY:
-          case TSDB_DATA_TYPE_DECIMAL:
-          case TSDB_DATA_TYPE_BLOB:
-          case TSDB_DATA_TYPE_JSON:
-          case TSDB_DATA_TYPE_MEDIUMBLOB:
-            uError("the column type %" PRIi16 " is defined but not implemented yet", pColInfoData->info.type);
-            TASSERT(0);
-            break;
-          default:
-            if (pColInfoData->info.type < TSDB_DATA_TYPE_MAX && pColInfoData->info.type > TSDB_DATA_TYPE_NULL) {
-              if (pCol->type == pColInfoData->info.type) {
-                tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, pCol->type, TD_VTYPE_NORM, var, true, offset,
-                                    k);
-              } else {
-                char tv[8] = {0};
-                if (pColInfoData->info.type == TSDB_DATA_TYPE_FLOAT) {
-                  float v = 0;
-                  GET_TYPED_DATA(v, float, pColInfoData->info.type, var);
-                  SET_TYPED_DATA(&tv, pCol->type, v);
-                } else if (pColInfoData->info.type == TSDB_DATA_TYPE_DOUBLE) {
-                  double v = 0;
-                  GET_TYPED_DATA(v, double, pColInfoData->info.type, var);
-                  SET_TYPED_DATA(&tv, pCol->type, v);
-                } else if (IS_SIGNED_NUMERIC_TYPE(pColInfoData->info.type)) {
-                  int64_t v = 0;
-                  GET_TYPED_DATA(v, int64_t, pColInfoData->info.type, var);
-                  SET_TYPED_DATA(&tv, pCol->type, v);
-                } else {
-                  uint64_t v = 0;
-                  GET_TYPED_DATA(v, uint64_t, pColInfoData->info.type, var);
-                  SET_TYPED_DATA(&tv, pCol->type, v);
-                }
-                tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, pCol->type, TD_VTYPE_NORM, tv, true, offset,
-                                    k);
-              }
-            } else {
-              uError("the column type %" PRIi16 " is undefined\n", pColInfoData->info.type);
-              TASSERT(0);
-            }
-            break;
-        }
-        offset += TYPE_BYTES[pCol->type];  // sum/avg would convert to int64_t/uint64_t/double during aggregation
-      }
-      dataLen += TD_ROW_LEN(rb.pBuf);
-#ifdef TD_DEBUG_PRINT_ROW
-      tdSRowPrint(rb.pBuf, pTSchema, __func__);
-#endif
-    }
-
-    ++numOfBlks;
-
-    pSubmitBlk->dataLen = dataLen;
-    msgLen += pSubmitBlk->dataLen;
-  }
-
-  (*pReq)->length = msgLen;
-
-  (*pReq)->header.vgId = htonl(vgId);
-  (*pReq)->header.contLen = htonl(msgLen);
-  (*pReq)->length = (*pReq)->header.contLen;
-  (*pReq)->numOfBlocks = htonl(numOfBlks);
-  SSubmitBlk* blk = (SSubmitBlk*)((*pReq) + 1);
-  while (numOfBlks--) {
-    int32_t dataLen = blk->dataLen;
-    blk->uid = htobe64(blk->uid);
-    blk->suid = htobe64(blk->suid);
-    blk->padding = htonl(blk->padding);
-    blk->sversion = htonl(blk->sversion);
-    blk->dataLen = htonl(blk->dataLen);
-    blk->schemaLen = htonl(blk->schemaLen);
-    blk->numOfRows = htons(blk->numOfRows);
-    blk = (SSubmitBlk*)(blk->data + dataLen);
-  }
-  return TSDB_CODE_SUCCESS;
-_err:
-  taosMemoryFreeClear(*pReq);
-  taosArrayDestroy(pTagArray);
-
-  return TSDB_CODE_FAILED;
-}
-#endif
--- a/source/dnode/vnode/src/sma/smaCommit.c
+++ b/source/dnode/vnode/src/sma/smaCommit.c
@ -121,7 +121,7 @@ static int32_t tdProcessRSmaPreCommitImpl(SSma *pSma) {
  // step 3: perform persist task for qTaskInfo
  tdRSmaPersistExecImpl(pRSmaStat);

-  smaDebug("vgId:%d, rsma pre commit succeess", SMA_VID(pSma));
+  smaDebug("vgId:%d, rsma pre commit success", SMA_VID(pSma));

  return TSDB_CODE_SUCCESS;
 }
@ -173,6 +173,7 @@ static int32_t tdProcessRSmaPostCommitImpl(SSma *pSma) {
  }

  if ((pDir = taosOpenDir(dir)) == NULL) {
+    regfree(&regex);
    terrno = TAOS_SYSTEM_ERROR(errno);
    smaWarn("vgId:%d, rsma post commit, open dir %s failed since %s", TD_VID(pVnode), dir, terrstr());
    return TSDB_CODE_FAILED;
--- a/source/dnode/vnode/src/sma/smaEnv.c
+++ b/source/dnode/vnode/src/sma/smaEnv.c
@ -18,7 +18,7 @@
 typedef struct SSmaStat SSmaStat;

 #define RSMA_TASK_INFO_HASH_SLOT 8
-#define SMA_MGMT_REF_NUM         1024
+#define SMA_MGMT_REF_NUM         10240

 extern SSmaMgmt smaMgmt;

@ -30,7 +30,73 @@ static int32_t  tdInitSmaEnv(SSma *pSma, int8_t smaType, const char *path, SSmaE
 static void    *tdFreeTSmaStat(STSmaStat *pStat);
 static void     tdDestroyRSmaStat(void *pRSmaStat);

+/**
+ * @brief rsma init
+ *
+ * @return int32_t
+ */
 // implementation
+int32_t smaInit() {
+  int8_t  old;
+  int32_t nLoops = 0;
+  while (1) {
+    old = atomic_val_compare_exchange_8(&smaMgmt.inited, 0, 2);
+    if (old != 2) break;
+    if (++nLoops > 1000) {
+      sched_yield();
+      nLoops = 0;
+    }
+  }
+
+  if (old == 0) {
+    // init tref rset
+    smaMgmt.rsetId = taosOpenRef(SMA_MGMT_REF_NUM, tdDestroyRSmaStat);
+
+    if (smaMgmt.rsetId < 0) {
+      atomic_store_8(&smaMgmt.inited, 0);
+      smaError("failed to init sma rset since %s", terrstr());
+      return TSDB_CODE_FAILED;
+    }
+
+    // init fetch timer handle
+    smaMgmt.tmrHandle = taosTmrInit(10000, 100, 10000, "RSMA");
+    if (!smaMgmt.tmrHandle) {
+      taosCloseRef(smaMgmt.rsetId);
+      atomic_store_8(&smaMgmt.inited, 0);
+      smaError("failed to init sma tmr hanle since %s", terrstr());
+      return TSDB_CODE_FAILED;
+    }
+
+    atomic_store_8(&smaMgmt.inited, 1);
+    smaInfo("sma mgmt env is initialized, rsetId:%d, tmrHandle:%p", smaMgmt.rsetId, smaMgmt.tmrHandle);
+  }
+
+  return TSDB_CODE_SUCCESS;
+}
+
+/**
+ * @brief rsma cleanup
+ *
+ */
+void smaCleanUp() {
+  int8_t  old;
+  int32_t nLoops = 0;
+  while (1) {
+    old = atomic_val_compare_exchange_8(&smaMgmt.inited, 1, 2);
+    if (old != 2) break;
+    if (++nLoops > 1000) {
+      sched_yield();
+      nLoops = 0;
+    }
+  }
+
+  if (old == 1) {
+    taosCloseRef(smaMgmt.rsetId);
+    taosTmrCleanUp(smaMgmt.tmrHandle);
+    smaInfo("sma mgmt env is cleaned up, rsetId:%d, tmrHandle:%p", smaMgmt.rsetId, smaMgmt.tmrHandle);
+    atomic_store_8(&smaMgmt.inited, 0);
+  }
+}

 static SSmaEnv *tdNewSmaEnv(const SSma *pSma, int8_t smaType, const char *path) {
  SSmaEnv *pEnv = NULL;
@ -135,34 +201,24 @@ static int32_t tdInitSmaStat(SSmaStat **pSmaStat, int8_t smaType, const SSma *pS
      atomic_store_8(RSMA_TRIGGER_STAT(pRSmaStat), TASK_TRIGGER_STAT_INIT);

      // init smaMgmt
-      smaMgmt.smaRef = taosOpenRef(SMA_MGMT_REF_NUM, tdDestroyRSmaStat);
-      if (smaMgmt.smaRef < 0) {
-        smaError("init smaRef failed, num:%d", SMA_MGMT_REF_NUM);
-        terrno = TSDB_CODE_OUT_OF_MEMORY;
-        return TSDB_CODE_FAILED;
-      }
+      smaInit();

-      int64_t refId = taosAddRef(smaMgmt.smaRef, pRSmaStat);
+      int64_t refId = taosAddRef(smaMgmt.rsetId, pRSmaStat);
      if (refId < 0) {
-        smaError("taosAddRef smaRef failed, since:%s", tstrerror(terrno));
+        smaError("vgId:%d, taosAddRef refId:%" PRIi64 " to rsetId rsetId:%d max:%d failed since:%s", SMA_VID(pSma),
+                 refId, smaMgmt.rsetId, SMA_MGMT_REF_NUM, tstrerror(terrno));
        return TSDB_CODE_FAILED;
+      } else {
+        smaDebug("vgId:%d, taosAddRef refId:%" PRIi64 " to rsetId rsetId:%d max:%d succeed", SMA_VID(pSma), refId,
+                 smaMgmt.rsetId, SMA_MGMT_REF_NUM);
      }
      pRSmaStat->refId = refId;

-      // init timer
-      RSMA_TMR_HANDLE(pRSmaStat) = taosTmrInit(10000, 100, 10000, "RSMA");
-      if (!RSMA_TMR_HANDLE(pRSmaStat)) {
-        taosMemoryFreeClear(*pSmaStat);
-        return TSDB_CODE_FAILED;
-      }

      // init hash
      RSMA_INFO_HASH(pRSmaStat) = taosHashInit(
          RSMA_TASK_INFO_HASH_SLOT, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_ENTRY_LOCK);
      if (!RSMA_INFO_HASH(pRSmaStat)) {
-        if (RSMA_TMR_HANDLE(pRSmaStat)) {
-          taosTmrCleanUp(RSMA_TMR_HANDLE(pRSmaStat));
-        }
        taosMemoryFreeClear(*pSmaStat);
        return TSDB_CODE_FAILED;
      }
@ -223,7 +279,7 @@ static void tdDestroyRSmaStat(void *pRSmaStat) {
      void *infoHash = taosHashIterate(RSMA_INFO_HASH(pStat), NULL);
      while (infoHash) {
        SRSmaInfo *pSmaInfo = *(SRSmaInfo **)infoHash;
-        tdFreeRSmaInfo(pSmaInfo);
+        tdFreeRSmaInfo(pSma, pSmaInfo);
        infoHash = taosHashIterate(RSMA_INFO_HASH(pStat), infoHash);
      }
    }
@ -244,11 +300,6 @@ static void tdDestroyRSmaStat(void *pRSmaStat) {
        nLoops = 0;
      }
    }
-
-    // step 6: cleanup the timer handle
-    if (RSMA_TMR_HANDLE(pStat)) {
-      taosTmrCleanUp(RSMA_TMR_HANDLE(pStat));
-    }
  }
 }

@ -275,8 +326,13 @@ int32_t tdDestroySmaState(SSmaStat *pSmaStat, int8_t smaType) {
      tdDestroyTSmaStat(SMA_TSMA_STAT(pSmaStat));
    } else if (smaType == TSDB_SMA_TYPE_ROLLUP) {
      SRSmaStat *pRSmaStat = SMA_RSMA_STAT(pSmaStat);
-      if (taosRemoveRef(smaMgmt.smaRef, RSMA_REF_ID(pRSmaStat)) < 0) {
-        smaError("remove refId from rsmaRef:0x%" PRIx64 " failed since %s", RSMA_REF_ID(pRSmaStat), terrstr());
+      if (taosRemoveRef(smaMgmt.rsetId, RSMA_REF_ID(pRSmaStat)) < 0) {
+        smaError("vgId:%d, remove refId:%" PRIi64 " from rsmaRef:%" PRIi32 " failed since %s", SMA_VID(pRSmaStat->pSma),
+                 RSMA_REF_ID(pRSmaStat), smaMgmt.rsetId, terrstr());
+        ASSERT(0);
+      } else {
+        smaDebug("vgId:%d, remove refId:%" PRIi64 " from rsmaRef:%" PRIi32 " succeed", SMA_VID(pRSmaStat->pSma),
+                 RSMA_REF_ID(pRSmaStat), smaMgmt.rsetId);
      }
    } else {
      ASSERT(0);
@ -323,7 +379,7 @@ int32_t tdCheckAndInitSmaEnv(SSma *pSma, int8_t smaType) {
      }
      break;
    default:
-      TASSERT(0);
+      smaError("vgId:%d undefined smaType:%", SMA_VID(pSma), smaType);
      return TSDB_CODE_FAILED;
  }

--- a/source/dnode/vnode/src/sma/smaRollup.c
+++ b/source/dnode/vnode/src/sma/smaRollup.c
@ -19,23 +19,27 @@
 #define RSMA_QTASKINFO_HEAD_LEN (sizeof(int32_t) + sizeof(int8_t) + sizeof(int64_t))  // len + type + suid

 SSmaMgmt smaMgmt = {
-    .smaRef = -1,
+    .inited = 0,
+    .rsetId = -1,
 };

 #define TD_QTASKINFO_FNAME_PREFIX "qtaskinfo.ver"
 typedef struct SRSmaQTaskInfoItem SRSmaQTaskInfoItem;
 typedef struct SRSmaQTaskInfoIter SRSmaQTaskInfoIter;

-static int32_t tdUidStorePut(STbUidStore *pStore, tb_uid_t suid, tb_uid_t *uid);
-static int32_t tdUpdateTbUidListImpl(SSma *pSma, tb_uid_t *suid, SArray *tbUids);
-static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaInfo *pRSmaInfo, SReadHandle *handle,
-                                       int8_t idx);
-static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, SRSmaInfoItem *rsmaItem,
-                                 tb_uid_t suid, int8_t level);
-static void    tdRSmaFetchTrigger(void *param, void *tmrId);
-static void    tdRSmaPersistTrigger(void *param, void *tmrId);
-static void   *tdRSmaPersistExec(void *param);
-static void    tdRSmaQTaskInfoGetFName(int32_t vid, int64_t version, char *outputName);
+static int32_t    tdUidStorePut(STbUidStore *pStore, tb_uid_t suid, tb_uid_t *uid);
+static int32_t    tdUpdateTbUidListImpl(SSma *pSma, tb_uid_t *suid, SArray *tbUids);
+static int32_t    tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaStat *pStat, SRSmaInfo *pRSmaInfo,
+                                          SReadHandle *handle, int8_t idx);
+static int32_t    tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, SRSmaInfoItem *rsmaItem,
+                                    STSchema *pTSchema, tb_uid_t suid, int8_t level);
+static SRSmaInfo *tdGetRSmaInfoBySuid(SSma *pSma, int64_t suid);
+static int32_t    tdRSmaFetchAndSubmitResult(SRSmaInfoItem *pItem, STSchema *pTSchema, int64_t suid, SRSmaStat *pStat,
+                                             int8_t blkType);
+static void       tdRSmaFetchTrigger(void *param, void *tmrId);
+static void       tdRSmaPersistTrigger(void *param, void *tmrId);
+static void      *tdRSmaPersistExec(void *param);
+static void       tdRSmaQTaskInfoGetFName(int32_t vid, int64_t version, char *outputName);

 static int32_t tdRSmaQTaskInfoIterInit(SRSmaQTaskInfoIter *pIter, STFile *pTFile);
 static int32_t tdRSmaQTaskInfoIterNextBlock(SRSmaQTaskInfoIter *pIter, bool *isFinish);
@ -47,25 +51,26 @@ static int32_t tdRSmaRestoreQTaskInfoReload(SSma *pSma, int64_t *committed);
 static int32_t tdRSmaRestoreTSDataReload(SSma *pSma, int64_t committed);

 struct SRSmaInfoItem {
-  SRSmaInfo *pRsmaInfo;
-  int64_t    refId;
-  void      *taskInfo;  // qTaskInfo_t
-  tmr_h      tmrId;
-  int8_t     level;
-  int8_t     tmrInitFlag;
-  int8_t     triggerStat;
-  int32_t    maxDelay;
+  void   *taskInfo;  // qTaskInfo_t
+  int64_t refId;
+  tmr_h   tmrId;
+  int32_t maxDelay;
+  int8_t  level;
+  int8_t  triggerStat;
 };

 struct SRSmaInfo {
  STSchema     *pTSchema;
-  SRSmaStat    *pStat;
  int64_t       suid;
  SRSmaInfoItem items[TSDB_RETENTION_L2];
 };

-#define RSMA_INFO_SMA(r)  ((r)->pStat->pSma)
-#define RSMA_INFO_STAT(r) ((r)->pStat)
+static SRSmaInfo *tdGetRSmaInfoByItem(SRSmaInfoItem *pItem) {
+  // adapt accordingly if definition of SRSmaInfo update
+  int32_t rsmaInfoHeadLen = sizeof(int64_t) + sizeof(STSchema *);
+  ASSERT(pItem->level == 1 || pItem->level == 2);
+  return (SRSmaInfo *)POINTER_SHIFT(pItem, -sizeof(SRSmaInfoItem) * (pItem->level - 1) - rsmaInfoHeadLen);
+}

 struct SRSmaQTaskInfoItem {
  int32_t len;
@ -107,9 +112,8 @@ static FORCE_INLINE void tdFreeTaskHandle(qTaskInfo_t *taskHandle, int32_t vgId,
  }
 }

-void *tdFreeRSmaInfo(SRSmaInfo *pInfo) {
+void *tdFreeRSmaInfo(SSma *pSma, SRSmaInfo *pInfo) {
  if (pInfo) {
-    SSma *pSma = RSMA_INFO_SMA(pInfo);
    for (int32_t i = 0; i < TSDB_RETENTION_L2; ++i) {
      SRSmaInfoItem *pItem = &pInfo->items[i];
      if (pItem->taskInfo) {
@ -142,8 +146,6 @@ static FORCE_INLINE int32_t tdUidStoreInit(STbUidStore **pStore) {
 }

 static int32_t tdUpdateTbUidListImpl(SSma *pSma, tb_uid_t *suid, SArray *tbUids) {
-  SSmaEnv   *pEnv = SMA_RSMA_ENV(pSma);
-  SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv);
  SRSmaInfo *pRSmaInfo = NULL;

  if (!suid || !tbUids) {
@ -152,8 +154,9 @@ static int32_t tdUpdateTbUidListImpl(SSma *pSma, tb_uid_t *suid, SArray *tbUids)
    return TSDB_CODE_FAILED;
  }

-  pRSmaInfo = taosHashGet(RSMA_INFO_HASH(pStat), suid, sizeof(tb_uid_t));
-  if (!pRSmaInfo || !(pRSmaInfo = *(SRSmaInfo **)pRSmaInfo)) {
+  pRSmaInfo = tdGetRSmaInfoBySuid(pSma, *suid);
+
+  if (!pRSmaInfo) {
    smaError("vgId:%d, failed to get rsma info for uid:%" PRIi64, SMA_VID(pSma), *suid);
    terrno = TSDB_CODE_RSMA_INVALID_STAT;
    return TSDB_CODE_FAILED;
@ -251,15 +254,14 @@ int32_t tdFetchTbUidList(SSma *pSma, STbUidStore **ppStore, tb_uid_t suid, tb_ui
  return TSDB_CODE_SUCCESS;
 }

-static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaInfo *pRSmaInfo, SReadHandle *pReadHandle,
-                                       int8_t idx) {
+static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaStat *pStat, SRSmaInfo *pRSmaInfo,
+                                       SReadHandle *pReadHandle, int8_t idx) {
  SRetention *pRetention = SMA_RETENTION(pSma);
  STsdbCfg   *pTsdbCfg = SMA_TSDB_CFG(pSma);

  if (param->qmsg[idx]) {
    SRSmaInfoItem *pItem = &(pRSmaInfo->items[idx]);
-    pItem->refId = RSMA_REF_ID(pRSmaInfo->pStat);
-    pItem->pRsmaInfo = pRSmaInfo;
+    pItem->refId = RSMA_REF_ID(pStat);
    pItem->taskInfo = qCreateStreamExecTaskInfo(param->qmsg[idx], pReadHandle);
    if (!pItem->taskInfo) {
      terrno = TSDB_CODE_RSMA_QTASKINFO_CREATE;
@ -277,8 +279,9 @@ static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaInfo
      pItem->maxDelay = TSDB_MAX_ROLLUP_MAX_DELAY;
    }
    pItem->level = (idx == 0 ? TSDB_RETENTION_L1 : TSDB_RETENTION_L2);
-    smaInfo("vgId:%d table:%" PRIi64 " level:%" PRIi8 " maxdelay:%" PRIi64 " watermark:%" PRIi64 ", finally maxdelay:%"PRIi32, SMA_VID(pSma),
-            pRSmaInfo->suid, idx + 1, param->maxdelay[idx], param->watermark[idx], pItem->maxDelay);
+    smaInfo("vgId:%d table:%" PRIi64 " level:%" PRIi8 " maxdelay:%" PRIi64 " watermark:%" PRIi64
+            ", finally maxdelay:%" PRIi32,
+            SMA_VID(pSma), pRSmaInfo->suid, idx + 1, param->maxdelay[idx], param->watermark[idx], pItem->maxDelay);
  }
  return TSDB_CODE_SUCCESS;
 _err:
@ -346,14 +349,13 @@ int32_t tdProcessRSmaCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, con
    goto _err;
  }
  pRSmaInfo->pTSchema = pTSchema;
-  pRSmaInfo->pStat = pStat;
  pRSmaInfo->suid = suid;

-  if (tdSetRSmaInfoItemParams(pSma, param, pRSmaInfo, &handle, 0) < 0) {
+  if (tdSetRSmaInfoItemParams(pSma, param, pStat, pRSmaInfo, &handle, 0) < 0) {
    goto _err;
  }

-  if (tdSetRSmaInfoItemParams(pSma, param, pRSmaInfo, &handle, 1) < 0) {
+  if (tdSetRSmaInfoItemParams(pSma, param, pStat, pRSmaInfo, &handle, 1) < 0) {
    goto _err;
  }

@ -365,7 +367,7 @@ int32_t tdProcessRSmaCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, con

  return TSDB_CODE_SUCCESS;
 _err:
-  tdFreeRSmaInfo(pRSmaInfo);
+  tdFreeRSmaInfo(pSma, pRSmaInfo);
  taosMemoryFree(pReader);
  return TSDB_CODE_FAILED;
 }
@ -536,10 +538,10 @@ int64_t tdRSmaGetMaxSubmitVer(SSma *pSma, int8_t level) {
  return atomic_load_64(&pRSmaStat->submitVer);
 }

-static int32_t tdFetchAndSubmitRSmaResult(SRSmaInfoItem *pItem, int8_t blkType) {
-  SArray    *pResult = NULL;
-  SRSmaInfo *pRSmaInfo = pItem->pRsmaInfo;
-  SSma      *pSma = RSMA_INFO_SMA(pRSmaInfo);
+static int32_t tdRSmaFetchAndSubmitResult(SRSmaInfoItem *pItem, STSchema *pTSchema, int64_t suid, SRSmaStat *pStat,
+                                          int8_t blkType) {
+  SArray *pResult = NULL;
+  SSma   *pSma = pStat->pSma;

  while (1) {
    SSDataBlock *output = NULL;
@ -571,12 +573,16 @@ static int32_t tdFetchAndSubmitRSmaResult(SRSmaInfoItem *pItem, int8_t blkType)
    STsdb      *sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb1 : pSma->pRSmaTsdb2);
    SSubmitReq *pReq = NULL;
    // TODO: the schema update should be handled
-    if (buildSubmitReqFromDataBlock(&pReq, pResult, pRSmaInfo->pTSchema, SMA_VID(pSma), pRSmaInfo->suid) < 0) {
+    if (buildSubmitReqFromDataBlock(&pReq, pResult, pTSchema, SMA_VID(pSma), suid) < 0) {
+      smaError("vgId:%d, build submit req for rsma table %" PRIi64 "l evel %" PRIi8 " failed since %s", SMA_VID(pSma),
+               suid, pItem->level, terrstr());
      goto _err;
    }

-    if (pReq && tdProcessSubmitReq(sinkTsdb, atomic_add_fetch_64(&pRSmaInfo->pStat->submitVer, 1), pReq) < 0) {
+    if (pReq && tdProcessSubmitReq(sinkTsdb, atomic_add_fetch_64(&pStat->submitVer, 1), pReq) < 0) {
      taosMemoryFreeClear(pReq);
+      smaError("vgId:%d, process submit req for rsma table %" PRIi64 " level %" PRIi8 " failed since %s", SMA_VID(pSma),
+               suid, pItem->level, terrstr());
      goto _err;
    }

@ -594,81 +600,16 @@ _err:
  return TSDB_CODE_FAILED;
 }

-/**
- * @brief trigger to get rsma result
- *
- * @param param
- * @param tmrId
- */
-static void tdRSmaFetchTrigger(void *param, void *tmrId) {
-  SRSmaInfoItem *pItem = param;
-  SSma          *pSma = NULL;
-  SRSmaStat     *pStat = (SRSmaStat *)taosAcquireRef(smaMgmt.smaRef, pItem->refId);
-  if (!pStat) {
-    smaDebug("rsma fetch task not start since already destroyed");
-    return;
-  }
-
-  pSma = RSMA_INFO_SMA(pItem->pRsmaInfo);
-
-  // if rsma trigger stat in paused, cancelled or finished, not start fetch task
-  int8_t rsmaTriggerStat = atomic_load_8(RSMA_TRIGGER_STAT(pStat));
-  switch (rsmaTriggerStat) {
-    case TASK_TRIGGER_STAT_PAUSED:
-    case TASK_TRIGGER_STAT_CANCELLED:
-    case TASK_TRIGGER_STAT_FINISHED: {
-      taosReleaseRef(smaMgmt.smaRef, pItem->refId);
-      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is cancelled",
-               SMA_VID(pSma), pItem->level, pItem->pRsmaInfo->suid);
-      return;
-    }
-    default:
-      break;
-  }
-
-  int8_t fetchTriggerStat =
-      atomic_val_compare_exchange_8(&pItem->triggerStat, TASK_TRIGGER_STAT_ACTIVE, TASK_TRIGGER_STAT_INACTIVE);
-  switch (fetchTriggerStat) {
-    case TASK_TRIGGER_STAT_ACTIVE: {
-      smaDebug("vgId:%d, fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is active", SMA_VID(pSma),
-               pItem->level, pItem->pRsmaInfo->suid);
-
-      tdRefSmaStat(pSma, (SSmaStat *)pStat);
-
-      SSDataBlock dataBlock = {.info.type = STREAM_GET_ALL};
-      qSetStreamInput(pItem->taskInfo, &dataBlock, STREAM_INPUT__DATA_BLOCK, false);
-      tdFetchAndSubmitRSmaResult(pItem, STREAM_INPUT__DATA_BLOCK);
-
-      tdUnRefSmaStat(pSma, (SSmaStat *)pStat);
-    } break;
-    case TASK_TRIGGER_STAT_PAUSED: {
-      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is paused",
-               SMA_VID(pSma), pItem->level, pItem->pRsmaInfo->suid);
-    } break;
-    case TASK_TRIGGER_STAT_INACTIVE: {
-      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is inactive",
-               SMA_VID(pSma), pItem->level, pItem->pRsmaInfo->suid);
-    } break;
-    case TASK_TRIGGER_STAT_INIT: {
-      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is init", SMA_VID(pSma),
-               pItem->level, pItem->pRsmaInfo->suid);
-    } break;
-    default: {
-      smaWarn("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is unknown",
-              SMA_VID(pSma), pItem->level, pItem->pRsmaInfo->suid);
-    } break;
-  }
-
-_end:
-  taosReleaseRef(smaMgmt.smaRef, pItem->refId);
-}
-
-static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, SRSmaInfoItem *pItem, tb_uid_t suid,
-                                 int8_t level) {
+static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType, SRSmaInfoItem *pItem,
+                                 STSchema *pTSchema, tb_uid_t suid, int8_t level) {
  if (!pItem || !pItem->taskInfo) {
    smaDebug("vgId:%d, no qTaskInfo to execute rsma %" PRIi8 " task for suid:%" PRIu64, SMA_VID(pSma), level, suid);
    return TSDB_CODE_SUCCESS;
  }
+  if (!pTSchema) {
+    smaWarn("vgId:%d, no schema to execute rsma %" PRIi8 " task for suid:%" PRIu64, SMA_VID(pSma), level, suid);
+    return TSDB_CODE_FAILED;
+  }

  smaDebug("vgId:%d, execute rsma %" PRIi8 " task for qTaskInfo:%p suid:%" PRIu64, SMA_VID(pSma), level,
           pItem->taskInfo, suid);
@ -678,14 +619,14 @@ static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType
    return TSDB_CODE_FAILED;
  }

-  tdFetchAndSubmitRSmaResult(pItem, STREAM_INPUT__DATA_SUBMIT);
-  atomic_store_8(&pItem->triggerStat, TASK_TRIGGER_STAT_ACTIVE);
-
  SSmaEnv   *pEnv = SMA_RSMA_ENV(pSma);
  SRSmaStat *pStat = SMA_RSMA_STAT(pEnv->pStat);

-  if (pStat->tmrHandle) {
-    taosTmrReset(tdRSmaFetchTrigger, pItem->maxDelay, pItem, pStat->tmrHandle, &pItem->tmrId);
+  tdRSmaFetchAndSubmitResult(pItem, pTSchema, suid, pStat, STREAM_INPUT__DATA_SUBMIT);
+  atomic_store_8(&pItem->triggerStat, TASK_TRIGGER_STAT_ACTIVE);
+
+  if (smaMgmt.tmrHandle) {
+    taosTmrReset(tdRSmaFetchTrigger, pItem->maxDelay, pItem, smaMgmt.tmrHandle, &pItem->tmrId);
  } else {
    ASSERT(0);
  }
@ -693,19 +634,29 @@ static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t inputType
  return TSDB_CODE_SUCCESS;
 }

-static int32_t tdExecuteRSma(SSma *pSma, const void *pMsg, int32_t inputType, tb_uid_t suid) {
-  SSmaEnv *pEnv = SMA_RSMA_ENV(pSma);
+static SRSmaInfo *tdGetRSmaInfoBySuid(SSma *pSma, int64_t suid) {
+  SSmaEnv   *pEnv = SMA_RSMA_ENV(pSma);
+  SRSmaStat *pStat = NULL;
  if (!pEnv) {
    // only applicable when rsma env exists
-    return TSDB_CODE_SUCCESS;
+    return NULL;
  }

-  SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv);
-  SRSmaInfo *pRSmaInfo = NULL;
-
-  pRSmaInfo = taosHashGet(RSMA_INFO_HASH(pStat), &suid, sizeof(tb_uid_t));
+  pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv);
+  if (!pStat || !RSMA_INFO_HASH(pStat)) {
+    return NULL;
+  }

+  SRSmaInfo *pRSmaInfo = taosHashGet(RSMA_INFO_HASH(pStat), &suid, sizeof(tb_uid_t));
  if (!pRSmaInfo || !(pRSmaInfo = *(SRSmaInfo **)pRSmaInfo)) {
+    return NULL;
+  }
+  return pRSmaInfo;
+}
+
+static int32_t tdExecuteRSma(SSma *pSma, const void *pMsg, int32_t inputType, tb_uid_t suid) {
+  SRSmaInfo *pRSmaInfo = tdGetRSmaInfoBySuid(pSma, suid);
+  if (!pRSmaInfo) {
    smaDebug("vgId:%d, return as no rsma info for suid:%" PRIu64, SMA_VID(pSma), suid);
    return TSDB_CODE_SUCCESS;
  }
@ -716,8 +667,8 @@ static int32_t tdExecuteRSma(SSma *pSma, const void *pMsg, int32_t inputType, tb
  }

  if (inputType == STREAM_INPUT__DATA_SUBMIT) {
-    tdExecuteRSmaImpl(pSma, pMsg, inputType, &pRSmaInfo->items[0], suid, TSDB_RETENTION_L1);
-    tdExecuteRSmaImpl(pSma, pMsg, inputType, &pRSmaInfo->items[1], suid, TSDB_RETENTION_L2);
+    tdExecuteRSmaImpl(pSma, pMsg, inputType, &pRSmaInfo->items[0], pRSmaInfo->pTSchema, suid, TSDB_RETENTION_L1);
+    tdExecuteRSmaImpl(pSma, pMsg, inputType, &pRSmaInfo->items[1], pRSmaInfo->pTSchema, suid, TSDB_RETENTION_L2);
  }

  return TSDB_CODE_SUCCESS;
@ -930,13 +881,11 @@ _err:
 }

 static int32_t tdRSmaQTaskInfoItemRestore(SSma *pSma, const SRSmaQTaskInfoItem *pItem) {
-  SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT((SSmaEnv *)pSma->pRSmaEnv);
  SRSmaInfo *pRSmaInfo = NULL;
  void      *qTaskInfo = NULL;

-  pRSmaInfo = taosHashGet(RSMA_INFO_HASH(pStat), &pItem->suid, sizeof(pItem->suid));
-
-  if (!pRSmaInfo || !(pRSmaInfo = *(SRSmaInfo **)pRSmaInfo)) {
+  pRSmaInfo = tdGetRSmaInfoBySuid(pSma, pItem->suid);
+  if (!pRSmaInfo) {
    smaDebug("vgId:%d, no restore as no rsma info for table:%" PRIu64, SMA_VID(pSma), pItem->suid);
    return TSDB_CODE_SUCCESS;
  }
@ -1253,7 +1202,8 @@ _end:
  }

  atomic_store_8(RSMA_RUNNING_STAT(pRSmaStat), 0);
-  taosReleaseRef(smaMgmt.smaRef, pRSmaStat->refId);
+  smaDebug("vgId:%d, release rsetId rsetId:%" PRIi64 " refId:%d", SMA_VID(pSma), smaMgmt.rsetId, pRSmaStat->refId);
+  tdReleaseSmaRef(smaMgmt.rsetId, pRSmaStat->refId, __func__, __LINE__);
  taosThreadExit(NULL);
  return NULL;
 }
@ -1278,7 +1228,9 @@ static void tdRSmaPersistTask(SRSmaStat *pRSmaStat) {
              atomic_load_8(RSMA_TRIGGER_STAT(pRSmaStat)));
    }
    atomic_store_8(RSMA_RUNNING_STAT(pRSmaStat), 0);
-    taosReleaseRef(smaMgmt.smaRef, pRSmaStat->refId);
+    smaDebug("vgId:%d, release rsetId rsetId:%" PRIi64 " refId:%d)", SMA_VID(pRSmaStat->pSma), smaMgmt.rsetId,
+             pRSmaStat->refId);
+    tdReleaseSmaRef(smaMgmt.rsetId, pRSmaStat->refId, __func__, __LINE__);
  }

  taosThreadAttrDestroy(&thAttr);
@ -1292,8 +1244,8 @@ static void tdRSmaPersistTask(SRSmaStat *pRSmaStat) {
 */
 static void tdRSmaPersistTrigger(void *param, void *tmrId) {
  SRSmaStat *rsmaStat = param;
-  SRSmaStat *pRSmaStat = (SRSmaStat *)taosAcquireRef(smaMgmt.smaRef, rsmaStat->refId);
-
+  SRSmaStat *pRSmaStat = (SRSmaStat *)taosAcquireRef(smaMgmt.rsetId, rsmaStat->refId);
+  ASSERT(0);
  if (!pRSmaStat) {
    smaDebug("rsma persistence task not start since already destroyed");
    return;
@ -1336,5 +1288,81 @@ static void tdRSmaPersistTrigger(void *param, void *tmrId) {
      smaWarn("rsma persistence not start since unknown stat %" PRIi8, tmrStat);
    } break;
  }
-  taosReleaseRef(smaMgmt.smaRef, rsmaStat->refId);
+  taosReleaseRef(smaMgmt.rsetId, rsmaStat->refId);
+}
+
+/**
+ * @brief trigger to get rsma result
+ *
+ * @param param
+ * @param tmrId
+ */
+static void tdRSmaFetchTrigger(void *param, void *tmrId) {
+  SRSmaInfoItem *pItem = param;
+  SSma          *pSma = NULL;
+  SRSmaStat     *pStat = (SRSmaStat *)tdAcquireSmaRef(smaMgmt.rsetId, pItem->refId, __func__, __LINE__);
+
+  if (!pStat) {
+    smaDebug("rsma fetch task not start since already destroyed, rsetId rsetId:%" PRIi64 " refId:%d)", smaMgmt.rsetId,
+             pItem->refId);
+    return;
+  }
+
+  pSma = pStat->pSma;
+
+  // if rsma trigger stat in paused, cancelled or finished, not start fetch task
+  int8_t rsmaTriggerStat = atomic_load_8(RSMA_TRIGGER_STAT(pStat));
+  switch (rsmaTriggerStat) {
+    case TASK_TRIGGER_STAT_PAUSED:
+    case TASK_TRIGGER_STAT_CANCELLED:
+    case TASK_TRIGGER_STAT_FINISHED: {
+      tdReleaseSmaRef(smaMgmt.rsetId, pItem->refId, __func__, __LINE__);
+      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data since stat is %" PRIi8 ", rsetId rsetId:%" PRIi64
+               " refId:%d",
+               SMA_VID(pSma), pItem->level, rsmaTriggerStat, smaMgmt.rsetId, pItem->refId);
+      return;
+    }
+    default:
+      break;
+  }
+
+  SRSmaInfo *pRSmaInfo = tdGetRSmaInfoByItem(pItem);
+
+  ASSERT(pRSmaInfo->suid > 0);
+
+  int8_t fetchTriggerStat =
+      atomic_val_compare_exchange_8(&pItem->triggerStat, TASK_TRIGGER_STAT_ACTIVE, TASK_TRIGGER_STAT_INACTIVE);
+  switch (fetchTriggerStat) {
+    case TASK_TRIGGER_STAT_ACTIVE: {
+      smaDebug("vgId:%d, fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is active", SMA_VID(pSma),
+               pItem->level, pRSmaInfo->suid);
+
+      tdRefSmaStat(pSma, (SSmaStat *)pStat);
+
+      SSDataBlock dataBlock = {.info.type = STREAM_GET_ALL};
+      qSetStreamInput(pItem->taskInfo, &dataBlock, STREAM_INPUT__DATA_BLOCK, false);
+      tdRSmaFetchAndSubmitResult(pItem, pRSmaInfo->pTSchema, pRSmaInfo->suid, pStat, STREAM_INPUT__DATA_BLOCK);
+
+      tdUnRefSmaStat(pSma, (SSmaStat *)pStat);
+    } break;
+    case TASK_TRIGGER_STAT_PAUSED: {
+      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is paused",
+               SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
+    } break;
+    case TASK_TRIGGER_STAT_INACTIVE: {
+      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is inactive",
+               SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
+    } break;
+    case TASK_TRIGGER_STAT_INIT: {
+      smaDebug("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is init", SMA_VID(pSma),
+               pItem->level, pRSmaInfo->suid);
+    } break;
+    default: {
+      smaWarn("vgId:%d, not fetch rsma level %" PRIi8 " data for table:%" PRIi64 " since stat is unknown",
+              SMA_VID(pSma), pItem->level, pRSmaInfo->suid);
+    } break;
+  }
+
+_end:
+  tdReleaseSmaRef(smaMgmt.rsetId, pItem->refId, __func__, __LINE__);
 }
--- a/source/dnode/vnode/src/sma/smaSnapshot.c
+++ b/source/dnode/vnode/src/sma/smaSnapshot.c
@ -1,16 +0,0 @@
-/*
- * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
- *
- * This program is free software: you can use, redistribute, and/or modify
- * it under the terms of the GNU Affero General Public License, version 3
- * or later ("AGPL"), as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
-
-#include "sma.h"
--- a/source/dnode/vnode/src/sma/smaTimeRange.c
+++ b/source/dnode/vnode/src/sma/smaTimeRange.c
@ -20,6 +20,36 @@
 #define SMA_STORAGE_MINUTES_DAY  1440
 #define SMA_STORAGE_SPLIT_FACTOR 14400  // least records in tsma file

+// TODO: Who is responsible for resource allocate and release?
+int32_t tdProcessTSmaInsert(SSma *pSma, int64_t indexUid, const char *msg) {
+  int32_t code = TSDB_CODE_SUCCESS;
+
+  if ((code = tdProcessTSmaInsertImpl(pSma, indexUid, msg)) < 0) {
+    smaWarn("vgId:%d, insert tsma data failed since %s", SMA_VID(pSma), tstrerror(terrno));
+  }
+  // TODO: destroy SSDataBlocks(msg)
+  return code;
+}
+
+int32_t tdProcessTSmaCreate(SSma *pSma, int64_t version, const char *msg) {
+  int32_t code = TSDB_CODE_SUCCESS;
+
+  if ((code = tdProcessTSmaCreateImpl(pSma, version, msg)) < 0) {
+    smaWarn("vgId:%d, create tsma failed since %s", SMA_VID(pSma), tstrerror(terrno));
+  }
+  // TODO: destroy SSDataBlocks(msg)
+  return code;
+}
+
+int32_t smaGetTSmaDays(SVnodeCfg *pCfg, void *pCont, uint32_t contLen, int32_t *days) {
+  int32_t code = TSDB_CODE_SUCCESS;
+  if ((code = tdProcessTSmaGetDaysImpl(pCfg, pCont, contLen, days)) < 0) {
+    smaWarn("vgId:%d, get tsma days failed since %s", pCfg->vgId, tstrerror(terrno));
+  }
+  smaDebug("vgId:%d, get tsma days %d", pCfg->vgId, *days);
+  return code;
+}
+
 /**
 * @brief Judge the tsma file split days
 *
--- a/source/dnode/vnode/src/sma/smaUtil.c
+++ b/source/dnode/vnode/src/sma/smaUtil.c
@ -294,4 +294,23 @@ int32_t tdRemoveTFile(STFile *pTFile) {
 }

 // smaXXXUtil ================
+void *tdAcquireSmaRef(int32_t rsetId, int64_t refId, const char *tags, int32_t ln) {
+  void *pResult = taosAcquireRef(rsetId, refId);
+  if (!pResult) {
+    smaWarn("%s:%d taosAcquireRef for rsetId:%" PRIi64 " refId:%d failed since %s", tags, ln, rsetId, refId, terrstr());
+  } else {
+    smaDebug("%s:%d taosAcquireRef for rsetId:%" PRIi64 " refId:%d success", tags, ln, rsetId, refId);
+  }
+  return pResult;
+}
+
+int32_t tdReleaseSmaRef(int32_t rsetId, int64_t refId, const char *tags, int32_t ln) {
+  if (taosReleaseRef(rsetId, refId) < 0) {
+    smaWarn("%s:%d taosReleaseRef for rsetId:%" PRIi64 " refId:%d failed since %s", tags, ln, rsetId, refId, terrstr());
+    return TSDB_CODE_FAILED;
+  }
+  smaDebug("%s:%d taosReleaseRef for rsetId:%" PRIi64 " refId:%d success", tags, ln, rsetId, refId);
+
+  return TSDB_CODE_SUCCESS;
+}
 // ...
--- a/source/dnode/vnode/src/tq/tq.c
+++ b/source/dnode/vnode/src/tq/tq.c
@ -112,7 +112,7 @@ int32_t tqSendMetaPollRsp(STQ* pTq, const SRpcMsg* pMsg, const SMqPollReq* pReq,
  };
  tmsgSendRsp(&resp);

-  tqDebug("vg %d from consumer %ld (epoch %d) send rsp, res msg type %d, reqOffset: %ld, rspOffset: %ld",
+  tqDebug("vgId:%d from consumer:%" PRId64 ", (epoch %d) send rsp, res msg type %d, reqOffset:%" PRId64 ", rspOffset:%" PRId64,
          TD_VID(pTq->pVnode), pReq->consumerId, pReq->epoch, pRsp->resMsgType, pRsp->reqOffset, pRsp->rspOffset);

  return 0;
@ -162,7 +162,7 @@ int32_t tqSendDataRsp(STQ* pTq, const SRpcMsg* pMsg, const SMqPollReq* pReq, con
  char buf2[80];
  tFormatOffset(buf1, 80, &pRsp->reqOffset);
  tFormatOffset(buf2, 80, &pRsp->rspOffset);
-  tqDebug("vg %d from consumer %ld (epoch %d) send rsp, block num: %d, reqOffset: %s, rspOffset: %s",
+  tqDebug("vgId:%d from consumer:%" PRId64 ", (epoch %d) send rsp, block num: %d, reqOffset:%s, rspOffset:%s",
          TD_VID(pTq->pVnode), pReq->consumerId, pReq->epoch, pRsp->blockNum, buf1, buf2);

  return 0;
@ -179,10 +179,10 @@ int32_t tqProcessOffsetCommitReq(STQ* pTq, char* msg, int32_t msgLen) {
  tDecoderClear(&decoder);

  if (offset.val.type == TMQ_OFFSET__SNAPSHOT_DATA) {
-    tqDebug("receive offset commit msg to %s on vg %d, offset(type:snapshot) uid: %ld, ts: %ld", offset.subKey,
+    tqDebug("receive offset commit msg to %s on vgId:%d, offset(type:snapshot) uid:%" PRId64 ", ts:%" PRId64, offset.subKey,
            TD_VID(pTq->pVnode), offset.val.uid, offset.val.ts);
  } else if (offset.val.type == TMQ_OFFSET__LOG) {
-    tqDebug("receive offset commit msg to %s on vg %d, offset(type:log) version: %ld", offset.subKey,
+    tqDebug("receive offset commit msg to %s on vgId:%d, offset(type:log) version:%" PRId64, offset.subKey,
            TD_VID(pTq->pVnode), offset.val.version);
  } else {
    ASSERT(0);
@ -244,22 +244,18 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
  STqOffsetVal fetchOffsetNew;

  // 1.find handle
-  char buf[80];
-  tFormatOffset(buf, 80, &reqOffset);
-  tqDebug("tmq poll: consumer %ld (epoch %d) recv poll req in vg %d, req offset %s", consumerId, pReq->epoch,
-          TD_VID(pTq->pVnode), buf);
-
  STqHandle* pHandle = taosHashGet(pTq->handles, pReq->subKey, strlen(pReq->subKey));
  /*ASSERT(pHandle);*/
  if (pHandle == NULL) {
-    tqError("tmq poll: no consumer handle for consumer %ld in vg %d, subkey %s", consumerId, TD_VID(pTq->pVnode),
-            pReq->subKey);
+    tqError("tmq poll: no consumer handle for consumer:%" PRId64 ", in vgId:%d, subkey %s", consumerId,
+            TD_VID(pTq->pVnode), pReq->subKey);
    return -1;
  }

  // check rebalance
  if (pHandle->consumerId != consumerId) {
-    tqError("tmq poll: consumer handle mismatch for consumer %ld in vg %d, subkey %s, handle consumer id %ld",
+    tqError("tmq poll: consumer handle mismatch for consumer:%" PRId64
+            ", in vgId:%d, subkey %s, handle consumer id %" PRId64,
            consumerId, TD_VID(pTq->pVnode), pReq->subKey, pHandle->consumerId);
    return -1;
  }
@ -270,6 +266,14 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
    consumerEpoch = atomic_val_compare_exchange_32(&pHandle->epoch, consumerEpoch, reqEpoch);
  }

+  char buf[80];
+  tFormatOffset(buf, 80, &reqOffset);
+  tqDebug("tmq poll: consumer %ld (epoch %d), subkey %s, recv poll req in vg %d, req offset %s", consumerId,
+          pReq->epoch, pHandle->subKey, TD_VID(pTq->pVnode), buf);
+
+  SMqDataRsp dataRsp = {0};
+  tqInitDataRsp(&dataRsp, pReq, pHandle->execHandle.subType);
+
  // 2.reset offset if needed
  if (reqOffset.type > 0) {
    fetchOffsetNew = reqOffset;
@ -279,7 +283,7 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
      fetchOffsetNew = pOffset->val;
      char formatBuf[80];
      tFormatOffset(formatBuf, 80, &fetchOffsetNew);
-      tqDebug("tmq poll: consumer %ld, offset reset to %s", consumerId, formatBuf);
+      tqDebug("tmq poll: consumer %" PRId64 ", subkey %s, offset reset to %s", consumerId, pHandle->subKey, formatBuf);
    } else {
      if (reqOffset.type == TMQ_OFFSET__RESET_EARLIEAST) {
        if (pReq->useSnapshot && pHandle->execHandle.subType == TOPIC_SUB_TYPE__COLUMN) {
@ -293,25 +297,48 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
          tqOffsetResetToLog(&fetchOffsetNew, walGetFirstVer(pTq->pVnode->pWal));
        }
      } else if (reqOffset.type == TMQ_OFFSET__RESET_LATEST) {
-        tqOffsetResetToLog(&fetchOffsetNew, walGetLastVer(pTq->pVnode->pWal));
+        tqOffsetResetToLog(&dataRsp.rspOffset, walGetLastVer(pTq->pVnode->pWal));
+        tqDebug("tmq poll: consumer %ld, subkey %s, offset reset to %ld", consumerId, pHandle->subKey,
+                dataRsp.rspOffset.version);
+        if (tqSendDataRsp(pTq, pMsg, pReq, &dataRsp) < 0) {
+          code = -1;
+        }
+        goto OVER;
      } else if (reqOffset.type == TMQ_OFFSET__RESET_NONE) {
-        tqError("tmq poll: no offset committed for consumer %ld in vg %d, subkey %s, reset none failed", consumerId,
-                TD_VID(pTq->pVnode), pReq->subKey);
+        tqError("tmq poll: subkey %s, no offset committed for consumer %" PRId64
+                " in vg %d, subkey %s, reset none failed",
+                pHandle->subKey, consumerId, TD_VID(pTq->pVnode), pReq->subKey);
        terrno = TSDB_CODE_TQ_NO_COMMITTED_OFFSET;
-        return -1;
+        code = -1;
+        goto OVER;
      }
    }
  }

  // 3.query
-  SMqDataRsp dataRsp = {0};
-  tqInitDataRsp(&dataRsp, pReq, pHandle->execHandle.subType);
+  if (pHandle->execHandle.subType == TOPIC_SUB_TYPE__COLUMN && fetchOffsetNew.type == TMQ_OFFSET__LOG) {
+    fetchOffsetNew.version++;
+    if (tqScanLog(pTq, &pHandle->execHandle, &dataRsp, &fetchOffsetNew) < 0) {
+      ASSERT(0);
+      code = -1;
+      goto OVER;
+    }
+    if (dataRsp.blockNum == 0) {
+      // TODO add to async task pool
+      /*dataRsp.rspOffset.version--;*/
+    }
+    if (tqSendDataRsp(pTq, pMsg, pReq, &dataRsp) < 0) {
+      code = -1;
+    }
+    goto OVER;
+  }

-  if (fetchOffsetNew.type == TMQ_OFFSET__LOG) {
+  if (pHandle->execHandle.subType != TOPIC_SUB_TYPE__COLUMN && fetchOffsetNew.type == TMQ_OFFSET__LOG) {
    int64_t     fetchVer = fetchOffsetNew.version + 1;
    SWalCkHead* pCkHead = taosMemoryMalloc(sizeof(SWalCkHead) + 2048);
    if (pCkHead == NULL) {
-      return -1;
+      code = -1;
+      goto OVER;
    }

    walSetReaderCapacity(pHandle->pWalReader, 2048);
@ -319,8 +346,9 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
    while (1) {
      consumerEpoch = atomic_load_32(&pHandle->epoch);
      if (consumerEpoch > reqEpoch) {
-        tqWarn("tmq poll: consumer %ld (epoch %d) vg %d offset %ld, found new consumer epoch %d, discard req epoch %d",
-               consumerId, pReq->epoch, TD_VID(pTq->pVnode), fetchVer, consumerEpoch, reqEpoch);
+        tqWarn("tmq poll: consumer %ld (epoch %d), subkey %s, vg %d offset %" PRId64
+               ", found new consumer epoch %d, discard req epoch %d",
+               consumerId, pReq->epoch, pHandle->subKey, TD_VID(pTq->pVnode), fetchVer, consumerEpoch, reqEpoch);
        break;
      }

@ -337,8 +365,8 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {

      SWalCont* pHead = &pCkHead->head;

-      tqDebug("tmq poll: consumer %ld (epoch %d) iter log, vg %d offset %ld msgType %d", consumerId, pReq->epoch,
-              TD_VID(pTq->pVnode), fetchVer, pHead->msgType);
+      tqDebug("tmq poll: consumer:%" PRId64 ", (epoch %d) iter log, vgId:%d offset %" PRId64 " msgType %d", consumerId,
+              pReq->epoch, TD_VID(pTq->pVnode), fetchVer, pHead->msgType);

      if (pHead->msgType == TDMT_VND_SUBMIT) {
        SSubmitReq* pCont = (SSubmitReq*)&pHead->body;
@ -363,7 +391,7 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
      } else {
        ASSERT(pHandle->fetchMeta);
        ASSERT(IS_META_MSG(pHead->msgType));
-        tqInfo("fetch meta msg, ver: %ld, type: %d", pHead->version, pHead->msgType);
+        tqInfo("fetch meta msg, ver:%" PRId64 ", type:%d", pHead->version, pHead->msgType);
        SMqMetaRsp metaRsp = {0};
        /*metaRsp.reqOffset = pReq->reqOffset.version;*/
        /*metaRsp.rspOffset = fetchVer;*/
@ -384,7 +412,7 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {

    taosMemoryFree(pCkHead);
  } else if (fetchOffsetNew.type == TMQ_OFFSET__SNAPSHOT_DATA) {
-    tqInfo("retrieve using snapshot actual offset: uid %ld ts %ld", fetchOffsetNew.uid, fetchOffsetNew.ts);
+    tqInfo("retrieve using snapshot actual offset: uid %" PRId64 " ts %" PRId64, fetchOffsetNew.uid, fetchOffsetNew.ts);
    if (tqScanSnapshot(pTq, &pHandle->execHandle, &dataRsp, fetchOffsetNew, workerId) < 0) {
      ASSERT(0);
    }
@ -482,10 +510,10 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) {
      pHandle->execHandle.execTb.suid = req.suid;
      SArray* tbUidList = taosArrayInit(0, sizeof(int64_t));
      vnodeGetCtbIdList(pTq->pVnode, req.suid, tbUidList);
-      tqDebug("vg %d, tq try get suid: %ld", pTq->pVnode->config.vgId, req.suid);
+      tqDebug("vgId:%d, tq try get suid:%" PRId64, pTq->pVnode->config.vgId, req.suid);
      for (int32_t i = 0; i < taosArrayGetSize(tbUidList); i++) {
        int64_t tbUid = *(int64_t*)taosArrayGet(tbUidList, i);
-        tqDebug("vg %d, idx %d, uid: %ld", TD_VID(pTq->pVnode), i, tbUid);
+        tqDebug("vgId:%d, idx %d, uid:%" PRId64, TD_VID(pTq->pVnode), i, tbUid);
      }
      for (int32_t i = 0; i < 5; i++) {
        tqReaderSetTbUidList(pHandle->execHandle.pExecReader[i], tbUidList);
@ -572,7 +600,7 @@ int32_t tqProcessTaskDeployReq(STQ* pTq, char* msg, int32_t msgLen) {

  streamSetupTrigger(pTask);

-  tqInfo("deploy stream task id %d child id %d on vg %d", pTask->taskId, pTask->selfChildId, TD_VID(pTq->pVnode));
+  tqInfo("deploy stream task id %d child id %d on vgId:%d", pTask->taskId, pTask->selfChildId, TD_VID(pTq->pVnode));

  taosHashPut(pTq->pStreamTasks, &pTask->taskId, sizeof(int32_t), &pTask, sizeof(void*));

--- a/source/dnode/vnode/src/tq/tqExec.c
+++ b/source/dnode/vnode/src/tq/tqExec.c
@ -46,7 +46,7 @@ static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, int32_t workerI
  return 0;
 }

-static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, int32_t workerId) {
+static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp) {
  SMetaReader mr = {0};
  metaReaderInit(&mr, pTq->pVnode->pMeta, 0);
  if (metaGetTableEntryByUid(&mr, uid) < 0) {
@ -59,15 +59,58 @@ static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, i
  return 0;
 }

+int64_t tqScanLog(STQ* pTq, const STqExecHandle* pExec, SMqDataRsp* pRsp, STqOffsetVal* pOffset) {
+  qTaskInfo_t task = pExec->execCol.task[0];
+
+  if (qStreamPrepareScan(task, pOffset) < 0) {
+    pRsp->rspOffset = *pOffset;
+    pRsp->rspOffset.version--;
+    return 0;
+  }
+
+  while (1) {
+    SSDataBlock* pDataBlock = NULL;
+    uint64_t     ts = 0;
+    if (qExecTask(task, &pDataBlock, &ts) < 0) {
+      ASSERT(0);
+    }
+
+    if (pDataBlock != NULL) {
+      tqAddBlockDataToRsp(pDataBlock, pRsp);
+      if (pRsp->withTbName) {
+        int64_t uid = pExec->pExecReader[0]->msgIter.uid;
+        tqAddTbNameToRsp(pTq, uid, pRsp);
+      }
+      pRsp->blockNum++;
+      continue;
+    }
+
+    void* meta = qStreamExtractMetaMsg(task);
+    if (meta != NULL) {
+      // tq add meta to rsp
+    }
+
+    if (qStreamExtractOffset(task, &pRsp->rspOffset) < 0) {
+      ASSERT(0);
+    }
+
+    if (pRsp->rspOffset.type == TMQ_OFFSET__LOG) {
+      ASSERT(pRsp->rspOffset.version + 1 >= pRsp->reqOffset.version);
+    }
+
+    ASSERT(pRsp->rspOffset.type != 0);
+
+    break;
+  }
+
+  return 0;
+}
+
 int32_t tqScanSnapshot(STQ* pTq, const STqExecHandle* pExec, SMqDataRsp* pRsp, STqOffsetVal offset, int32_t workerId) {
  ASSERT(pExec->subType == TOPIC_SUB_TYPE__COLUMN);
  qTaskInfo_t task = pExec->execCol.task[workerId];

-  /*if (qStreamScanSnapshot(task) < 0) {*/
-  /*ASSERT(0);*/
-  /*}*/
-
-  if (qStreamPrepareScan(task, offset.uid, offset.ts) < 0) {
+  if (qStreamPrepareTsdbScan(task, offset.uid, offset.ts) < 0) {
    ASSERT(0);
  }

@ -93,7 +136,7 @@ int32_t tqScanSnapshot(STQ* pTq, const STqExecHandle* pExec, SMqDataRsp* pRsp, S
      if (qGetStreamScanStatus(task, &uid, &ts) < 0) {
        ASSERT(0);
      }
-      tqAddTbNameToRsp(pTq, uid, pRsp, workerId);
+      tqAddTbNameToRsp(pTq, uid, pRsp);
 #endif
    }
    pRsp->blockNum++;
@ -129,7 +172,7 @@ int32_t tqLogScanExec(STQ* pTq, STqExecHandle* pExec, SSubmitReq* pReq, SMqDataR
      tqAddBlockDataToRsp(pDataBlock, pRsp);
      if (pRsp->withTbName) {
        int64_t uid = pExec->pExecReader[workerId]->msgIter.uid;
-        tqAddTbNameToRsp(pTq, uid, pRsp, workerId);
+        tqAddTbNameToRsp(pTq, uid, pRsp);
      }
      pRsp->blockNum++;
    }
@ -146,7 +189,7 @@ int32_t tqLogScanExec(STQ* pTq, STqExecHandle* pExec, SSubmitReq* pReq, SMqDataR
      tqAddBlockDataToRsp(&block, pRsp);
      if (pRsp->withTbName) {
        int64_t uid = pExec->pExecReader[workerId]->msgIter.uid;
-        tqAddTbNameToRsp(pTq, uid, pRsp, workerId);
+        tqAddTbNameToRsp(pTq, uid, pRsp);
      }
      tqAddBlockSchemaToRsp(pExec, workerId, pRsp);
      pRsp->blockNum++;
@ -164,7 +207,7 @@ int32_t tqLogScanExec(STQ* pTq, STqExecHandle* pExec, SSubmitReq* pReq, SMqDataR
      tqAddBlockDataToRsp(&block, pRsp);
      if (pRsp->withTbName) {
        int64_t uid = pExec->pExecReader[workerId]->msgIter.uid;
-        tqAddTbNameToRsp(pTq, uid, pRsp, workerId);
+        tqAddTbNameToRsp(pTq, uid, pRsp);
      }
      tqAddBlockSchemaToRsp(pExec, workerId, pRsp);
      pRsp->blockNum++;
--- a/source/dnode/vnode/src/tq/tqOffset.c
+++ b/source/dnode/vnode/src/tq/tqOffset.c
@ -139,7 +139,7 @@ int32_t tqOffsetSnapshot(STqOffsetStore* pStore) {
    int64_t writeLen;
    if ((writeLen = taosWriteFile(pFile, buf, totLen)) != totLen) {
      ASSERT(0);
-      tqError("write offset incomplete, len %d, write len %ld", bodyLen, writeLen);
+      tqError("write offset incomplete, len %d, write len %" PRId64, bodyLen, writeLen);
      taosHashCancelIterate(pStore->pHash, pIter);
      return -1;
    }
--- a/source/dnode/vnode/src/tq/tqPush.c
+++ b/source/dnode/vnode/src/tq/tqPush.c
@ -223,7 +223,7 @@ int32_t tqPushMsgNew(STQ* pTq, void* msg, int32_t msgLen, tmsg_t msgType, int64_
    memset(&pHandle->pushHandle.rpcInfo, 0, sizeof(SRpcHandleInfo));
    taosWUnLockLatch(&pHandle->pushHandle.lock);

-    tqDebug("vg %d offset %ld from consumer %ld (epoch %d) send rsp, block num: %d, reqOffset: %ld, rspOffset: %ld",
+    tqDebug("vgId:%d offset %" PRId64 " from consumer:%" PRId64 ", (epoch %d) send rsp, block num: %d, reqOffset:%" PRId64 ", rspOffset:%" PRId64,
            TD_VID(pTq->pVnode), fetchOffset, pHandle->pushHandle.consumerId, pHandle->pushHandle.epoch, rsp.blockNum,
            rsp.reqOffset, rsp.rspOffset);

--- a/source/dnode/vnode/src/tq/tqRead.c
+++ b/source/dnode/vnode/src/tq/tqRead.c
@ -15,11 +15,6 @@

 #include "tq.h"

-int64_t tqScanLog(STQ* pTq, const STqExecHandle* pExec, SMqDataRsp* pRsp, STqOffsetVal offset) {
-  /*if ()*/
-  return 0;
-}
-
 int64_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, SWalCkHead** ppCkHead) {
  int32_t code = 0;
  taosThreadMutexLock(&pHandle->pWalReader->mutex);
@ -27,8 +22,8 @@ int64_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, SWalCkHea

  while (1) {
    if (walFetchHead(pHandle->pWalReader, offset, *ppCkHead) < 0) {
-      tqDebug("tmq poll: consumer %ld (epoch %d) vg %d offset %ld, no more log to return", pHandle->consumerId,
-              pHandle->epoch, TD_VID(pTq->pVnode), offset);
+      tqDebug("tmq poll: consumer:%" PRId64 ", (epoch %d) vgId:%d offset %" PRId64 ", no more log to return",
+              pHandle->consumerId, pHandle->epoch, TD_VID(pTq->pVnode), offset);
      *fetchOffset = offset - 1;
      code = -1;
      goto END;
@ -84,8 +79,10 @@ STqReader* tqOpenReader(SVnode* pVnode) {
    return NULL;
  }

-  // TODO open
-  /*pReader->pWalReader = walOpenReader(pVnode->pWal, NULL);*/
+  pReader->pWalReader = walOpenReader(pVnode->pWal, NULL);
+  if (pReader->pWalReader == NULL) {
+    return NULL;
+  }

  pReader->pVnodeMeta = pVnode->pMeta;
  pReader->pMsg = NULL;
@ -106,13 +103,27 @@ void tqCloseReader(STqReader* pReader) {
  taosMemoryFree(pReader);
 }

+int32_t tqSeekVer(STqReader* pReader, int64_t ver) {
+  if (walReadSeekVer(pReader->pWalReader, ver) < 0) {
+    ASSERT(pReader->pWalReader->curInvalid);
+    ASSERT(pReader->pWalReader->curVersion == ver);
+    return -1;
+  }
+  ASSERT(pReader->pWalReader->curVersion == ver);
+  return 0;
+}
+
 int32_t tqNextBlock(STqReader* pReader, SFetchRet* ret) {
  bool fromProcessedMsg = pReader->pMsg != NULL;

  while (1) {
    if (!fromProcessedMsg) {
      if (walNextValidMsg(pReader->pWalReader) < 0) {
+        pReader->ver = pReader->pWalReader->curVersion - pReader->pWalReader->curInvalid;
+        ret->offset.type = TMQ_OFFSET__LOG;
+        ret->offset.version = pReader->ver;
        ret->fetchType = FETCH_TYPE__NONE;
+        ASSERT(ret->offset.version >= 0);
        return -1;
      }
      void* body = pReader->pWalReader->pHead->head.body;
@ -127,22 +138,21 @@ int32_t tqNextBlock(STqReader* pReader, SFetchRet* ret) {
    }

    while (tqNextDataBlock(pReader)) {
+      // TODO mem free
      memset(&ret->data, 0, sizeof(SSDataBlock));
      int32_t code = tqRetrieveDataBlock(&ret->data, pReader);
      if (code != 0 || ret->data.info.rows == 0) {
-        if (fromProcessedMsg) {
-          ret->fetchType = FETCH_TYPE__NONE;
-          return 0;
-        } else {
-          break;
-        }
+        ASSERT(0);
+        continue;
      }
-
      ret->fetchType = FETCH_TYPE__DATA;
      return 0;
    }

    if (fromProcessedMsg) {
+      ret->offset.type = TMQ_OFFSET__LOG;
+      ret->offset.version = pReader->ver;
+      ASSERT(pReader->ver >= 0);
      ret->fetchType = FETCH_TYPE__NONE;
      return 0;
    }
@ -179,9 +189,9 @@ bool tqNextDataBlock(STqReader* pReader) {
      return true;
    }
    void* ret = taosHashGet(pReader->tbIdHash, &pReader->msgIter.uid, sizeof(int64_t));
-    /*tqDebug("search uid %ld", pHandle->msgIter.uid);*/
+    /*tqDebug("search uid %" PRId64, pHandle->msgIter.uid);*/
    if (ret != NULL) {
-      /*tqDebug("find   uid %ld", pHandle->msgIter.uid);*/
+      /*tqDebug("find   uid %" PRId64, pHandle->msgIter.uid);*/
      return true;
    }
  }
@ -212,7 +222,7 @@ int32_t tqRetrieveDataBlock(SSDataBlock* pBlock, STqReader* pReader) {
    if (pReader->pSchema) taosMemoryFree(pReader->pSchema);
    pReader->pSchema = metaGetTbTSchema(pReader->pVnodeMeta, pReader->msgIter.uid, sversion);
    if (pReader->pSchema == NULL) {
-      tqWarn("cannot found tsschema for table: uid: %ld (suid: %ld), version %d, possibly dropped table",
+      tqWarn("cannot found tsschema for table: uid:%" PRId64 " (suid:%" PRId64 "), version %d, possibly dropped table",
             pReader->msgIter.uid, pReader->msgIter.suid, pReader->cachedSchemaVer);
      /*ASSERT(0);*/
      terrno = TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND;
@ -222,7 +232,7 @@ int32_t tqRetrieveDataBlock(SSDataBlock* pBlock, STqReader* pReader) {
    if (pReader->pSchemaWrapper) tDeleteSSchemaWrapper(pReader->pSchemaWrapper);
    pReader->pSchemaWrapper = metaGetTableSchema(pReader->pVnodeMeta, pReader->msgIter.uid, sversion, true);
    if (pReader->pSchemaWrapper == NULL) {
-      tqWarn("cannot found schema wrapper for table: suid: %ld, version %d, possibly dropped table",
+      tqWarn("cannot found schema wrapper for table: suid:%" PRId64 ", version %d, possibly dropped table",
             pReader->msgIter.uid, pReader->cachedSchemaVer);
      /*ASSERT(0);*/
      terrno = TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND;
--- a/source/dnode/vnode/src/tsdb/tsdbCache.c
+++ b/source/dnode/vnode/src/tsdb/tsdbCache.c
@ -59,6 +59,8 @@ static void getTableCacheKey(tb_uid_t uid, int cacheType, char *key, int *len) {

 static void deleteTableCacheLastrow(const void *key, size_t keyLen, void *value) { taosMemoryFree(value); }

+static void deleteTableCacheLast(const void *key, size_t keyLen, void *value) { taosArrayDestroy(value); }
+
 static int32_t tsdbCacheDeleteLastrow(SLRUCache *pCache, tb_uid_t uid, TSKEY eKey) {
  int32_t code = 0;
  char    key[32] = {0};
@ -153,7 +155,7 @@ int32_t tsdbCacheInsertLastrow(SLRUCache *pCache, STsdb *pTsdb, tb_uid_t uid, ST
        /* tsdbCacheInsertLastrow(pCache, uid, row, dup); */
      }
    }
-  } else {
+  } /*else {
    if (dup) {
      cacheRow = tdRowDup(row);
    } else {
@ -166,7 +168,7 @@ int32_t tsdbCacheInsertLastrow(SLRUCache *pCache, STsdb *pTsdb, tb_uid_t uid, ST
    if (status != TAOS_LRU_STATUS_OK) {
      code = -1;
    }
-  }
+    }*/

  return code;
 }
@ -761,7 +763,6 @@ static int32_t mergeLastRow(tb_uid_t uid, STsdb *pTsdb, bool *dup, STSRow **ppRo
    for (int i = 0; i < nMax; ++i) {
      TSDBKEY maxKey = TSDBROW_KEY(max[i]);

-      // bool deleted = false;
      bool deleted = tsdbKeyDeleted(&maxKey, pSkyline, &iSkyline);
      if (!deleted) {
        // iMerge[nMerge] = i;
@ -818,12 +819,22 @@ _err:
  return code;
 }

-static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
-  int32_t code = 0;
+typedef struct {
+  TSKEY   ts;
+  SColVal colVal;
+} SLastCol;
+
+// static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
+static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray) {
+  int32_t  code = 0;
+  SArray  *pSkyline = NULL;
+  STSRow  *pRow = NULL;
+  STSRow **ppRow = &pRow;

  STSchema *pTSchema = metaGetTbTSchema(pTsdb->pVnode->pMeta, uid, -1);
  int16_t   nCol = pTSchema->numOfCols;
-  SArray   *pColArray = taosArrayInit(nCol, sizeof(SColVal));
+  //  SArray   *pColArray = taosArrayInit(nCol, sizeof(SColVal));
+  SArray *pColArray = taosArrayInit(nCol, sizeof(SLastCol));

  tb_uid_t suid = getTableSuidByUid(uid, pTsdb);

@ -837,9 +848,9 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
    tsdbGetTbDataFromMemTable(pTsdb->imem, suid, uid, &pIMem);
  }

-  *ppRow = NULL;
+  *ppLastArray = NULL;

-  SArray *pSkyline = taosArrayInit(32, sizeof(TSDBKEY));
+  pSkyline = taosArrayInit(32, sizeof(TSDBKEY));

  SDelIdx delIdx;

@ -943,7 +954,6 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
    for (int i = 0; i < nMax; ++i) {
      TSDBKEY maxKey = TSDBROW_KEY(max[i]);

-      // bool deleted = false;
      bool deleted = tsdbKeyDeleted(&maxKey, pSkyline, &iSkyline);
      if (!deleted) {
        iMerge[nMerge] = iMax[i];
@ -970,8 +980,9 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
        tRowMergerClear(&merger);
      }
    } else {
-      *ppRow = NULL;
-      return code;
+      /* *ppRow = NULL; */
+      /* return code; */
+      continue;
    }

    if (iCol == 0) {
@ -980,7 +991,8 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {

      *pColVal = COL_VAL_VALUE(pTColumn->colId, pTColumn->type, (SValue){.ts = maxKey});

-      if (taosArrayPush(pColArray, pColVal) == NULL) {
+      // if (taosArrayPush(pColArray, pColVal) == NULL) {
+      if (taosArrayPush(pColArray, &(SLastCol){.ts = maxKey, .colVal = *pColVal}) == NULL) {
        code = TSDB_CODE_OUT_OF_MEMORY;
        goto _err;
      }
@ -991,7 +1003,8 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
      for (int16_t i = iCol; i < nCol; ++i) {
        // tsdbRowGetColVal(*ppRow, pTSchema, i, pColVal);
        tTSRowGetVal(*ppRow, pTSchema, i, pColVal);
-        if (taosArrayPush(pColArray, pColVal) == NULL) {
+        // if (taosArrayPush(pColArray, pColVal) == NULL) {
+        if (taosArrayPush(pColArray, &(SLastCol){.ts = maxKey, .colVal = *pColVal}) == NULL) {
          code = TSDB_CODE_OUT_OF_MEMORY;
          goto _err;
        }
@ -1012,11 +1025,11 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
          --nilColCount;
        }
      }
-      /*
+
      if (*ppRow) {
        taosMemoryFreeClear(*ppRow);
      }
-      */
+
      continue;
    }

@ -1024,12 +1037,16 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
    for (int16_t i = iCol; i < nCol; ++i) {
      SColVal colVal = {0};
      tTSRowGetVal(*ppRow, pTSchema, i, &colVal);
+      TSKEY rowTs = (*ppRow)->ts;

-      SColVal *tColVal = (SColVal *)taosArrayGet(pColArray, i);
+      // SColVal *tColVal = (SColVal *)taosArrayGet(pColArray, i);
+      SLastCol *tTsVal = (SLastCol *)taosArrayGet(pColArray, i);
+      SColVal  *tColVal = &tTsVal->colVal;

      if (!colVal.isNone && !colVal.isNull) {
        if (tColVal->isNull || tColVal->isNone) {
-          taosArraySet(pColArray, i, &colVal);
+          // taosArraySet(pColArray, i, &colVal);
+          taosArraySet(pColArray, i, &(SLastCol){.ts = rowTs, .colVal = colVal});
          --nilColCount;
        }
      } else {
@ -1054,16 +1071,45 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, STSRow **ppRow) {
  } while (nilColCount > 0);

  // if () new ts row from pColArray if non empty
-  if (taosArrayGetSize(pColArray) == nCol) {
-    code = tdSTSRowNew(pColArray, pTSchema, ppRow);
-    if (code) goto _err;
+  /* if (taosArrayGetSize(pColArray) == nCol) { */
+  /*   code = tdSTSRowNew(pColArray, pTSchema, ppRow); */
+  /*   if (code) goto _err; */
+  /* } */
+  /* taosArrayDestroy(pColArray); */
+  if (taosArrayGetSize(pColArray) <= 0) {
+    *ppLastArray = NULL;
+    taosArrayDestroy(pColArray);
+  } else {
+    *ppLastArray = pColArray;
+  }
+  if (*ppRow) {
+    taosMemoryFreeClear(*ppRow);
+  }
+
+  for (int i = 0; i < 3; ++i) {
+    if (input[i].nextRowClearFn) {
+      input[i].nextRowClearFn(input[i].iter);
+    }
+  }
+  if (pSkyline) {
+    taosArrayDestroy(pSkyline);
  }
-  taosArrayDestroy(pColArray);
  taosMemoryFreeClear(pTSchema);

  return code;
 _err:
  taosArrayDestroy(pColArray);
+  if (*ppRow) {
+    taosMemoryFreeClear(*ppRow);
+  }
+  for (int i = 0; i < 3; ++i) {
+    if (input[i].nextRowClearFn) {
+      input[i].nextRowClearFn(input[i].iter);
+    }
+  }
+  if (pSkyline) {
+    taosArrayDestroy(pSkyline);
+  }
  taosMemoryFreeClear(pTSchema);
  tsdbError("vgId:%d merge last_row failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
  return code;
@ -1081,7 +1127,7 @@ int32_t tsdbCacheGetLastrowH(SLRUCache *pCache, tb_uid_t uid, STsdb *pTsdb, LRUH
    //*ppRow = (STSRow *)taosLRUCacheValue(pCache, h);
  } else {
    STSRow *pRow = NULL;
-    bool    dup = false;
+    bool    dup = false;  // which is always false for now
    code = mergeLastRow(uid, pTsdb, &dup, &pRow);
    // if table's empty or error, return code of -1
    if (code < 0 || pRow == NULL) {
@ -1093,7 +1139,14 @@ int32_t tsdbCacheGetLastrowH(SLRUCache *pCache, tb_uid_t uid, STsdb *pTsdb, LRUH
      return 0;
    }

-    tsdbCacheInsertLastrow(pCache, pTsdb, uid, pRow, dup);
+    _taos_lru_deleter_t deleter = deleteTableCacheLastrow;
+    LRUStatus           status =
+        taosLRUCacheInsert(pCache, key, keyLen, pRow, TD_ROW_LEN(pRow), deleter, NULL, TAOS_LRU_PRIORITY_LOW);
+    if (status != TAOS_LRU_STATUS_OK) {
+      code = -1;
+    }
+
+    //    tsdbCacheInsertLastrow(pCache, pTsdb, uid, pRow, dup);
    h = taosLRUCacheLookup(pCache, key, keyLen);
    //*ppRow = (STSRow *)taosLRUCacheValue(pCache, h);
  }
@ -1103,6 +1156,30 @@ int32_t tsdbCacheGetLastrowH(SLRUCache *pCache, tb_uid_t uid, STsdb *pTsdb, LRUH
  return code;
 }

+int32_t tsdbCacheLastArray2Row(SArray *pLastArray, STSRow **ppRow, STSchema *pTSchema) {
+  int32_t code = 0;
+  int16_t nCol = taosArrayGetSize(pLastArray);
+  SArray *pColArray = taosArrayInit(nCol, sizeof(SColVal));
+
+  for (int16_t iCol = 0; iCol < nCol; ++iCol) {
+    SLastCol *tTsVal = (SLastCol *)taosArrayGet(pLastArray, iCol);
+    SColVal  *tColVal = &tTsVal->colVal;
+    taosArrayPush(pColArray, tColVal);
+  }
+
+  code = tdSTSRowNew(pColArray, pTSchema, ppRow);
+  if (code) goto _err;
+
+  taosArrayDestroy(pColArray);
+
+  return code;
+
+_err:
+  taosArrayDestroy(pColArray);
+
+  return code;
+}
+
 int32_t tsdbCacheGetLastH(SLRUCache *pCache, tb_uid_t uid, STsdb *pTsdb, LRUHandle **handle) {
  int32_t code = 0;
  char    key[32] = {0};
@ -1115,21 +1192,24 @@ int32_t tsdbCacheGetLastH(SLRUCache *pCache, tb_uid_t uid, STsdb *pTsdb, LRUHand
    //*ppRow = (STSRow *)taosLRUCacheValue(pCache, h);

  } else {
-    STSRow *pRow = NULL;
-    code = mergeLast(uid, pTsdb, &pRow);
+    // STSRow *pRow = NULL;
+    // code = mergeLast(uid, pTsdb, &pRow);
+    SArray *pLastArray = NULL;
+    code = mergeLast(uid, pTsdb, &pLastArray);
    // if table's empty or error, return code of -1
-    if (code < 0 || pRow == NULL) {
+    // if (code < 0 || pRow == NULL) {
+    if (code < 0 || pLastArray == NULL) {
      *handle = NULL;
      return 0;
    }

-    _taos_lru_deleter_t deleter = deleteTableCacheLastrow;
+    _taos_lru_deleter_t deleter = deleteTableCacheLast;
    LRUStatus           status =
-        taosLRUCacheInsert(pCache, key, keyLen, pRow, TD_ROW_LEN(pRow), deleter, NULL, TAOS_LRU_PRIORITY_LOW);
+        taosLRUCacheInsert(pCache, key, keyLen, pLastArray, pLastArray->capacity, deleter, NULL, TAOS_LRU_PRIORITY_LOW);
    if (status != TAOS_LRU_STATUS_OK) {
      code = -1;
    }
-    /* tsdbCacheInsertLast(pCache, uid, pRow); */
+
    h = taosLRUCacheLookup(pCache, key, keyLen);
    //*ppRow = (STSRow *)taosLRUCacheValue(pCache, h);
  }
@ -1162,9 +1242,23 @@ int32_t tsdbCacheDelete(SLRUCache *pCache, tb_uid_t uid, TSKEY eKey) {
  getTableCacheKey(uid, 1, key, &keyLen);
  h = taosLRUCacheLookup(pCache, key, keyLen);
  if (h) {
-    // clear last cache anyway, no matter where eKey ends.
-    taosLRUCacheRelease(pCache, h, true);
+    SArray *pLast = (SArray *)taosLRUCacheValue(pCache, h);
+    bool    invalidate = false;
+    int16_t nCol = taosArrayGetSize(pLast);

+    for (int16_t iCol = 0; iCol < nCol; ++iCol) {
+      SLastCol *tTsVal = (SLastCol *)taosArrayGet(pLast, iCol);
+      if (eKey >= tTsVal->ts) {
+        invalidate = true;
+        break;
+      }
+    }
+
+    if (invalidate) {
+      taosLRUCacheRelease(pCache, h, true);
+    } else {
+      taosLRUCacheRelease(pCache, h, false);
+    }
    // void taosLRUCacheErase(SLRUCache * cache, const void *key, size_t keyLen);
  }

--- a/source/dnode/vnode/src/tsdb/tsdbCacheRead.c
+++ b/source/dnode/vnode/src/tsdb/tsdbCacheRead.c
@ -128,6 +128,8 @@ int32_t tsdbRetrieveLastRow(void* pReader, SSDataBlock* pResBlock, const int32_t
      }

      pRow = (STSRow*)taosLRUCacheValue(lruCache, h);
+      // SArray* pLast = (SArray*)taosLRUCacheValue(lruCache, h);
+      // tsdbCacheLastArray2Row(pLast, &pRow, pr->pSchema);
      if (pRow->ts > lastKey) {
        // Set result row into the same rowIndex repeatly, so we need to check if the internal result row has already
        // appended or not.
@ -140,6 +142,7 @@ int32_t tsdbRetrieveLastRow(void* pReader, SSDataBlock* pResBlock, const int32_t
        lastKey = pRow->ts;
      }

+      // taosMemoryFree(pRow);
      tsdbCacheRelease(lruCache, h);
    }
  } else if (pr->type == LASTROW_RETRIEVE_TYPE_ALL) {
@ -158,8 +161,12 @@ int32_t tsdbRetrieveLastRow(void* pReader, SSDataBlock* pResBlock, const int32_t
      }

      pRow = (STSRow*)taosLRUCacheValue(lruCache, h);
+      // SArray* pLast = (SArray*)taosLRUCacheValue(lruCache, h);
+      // tsdbCacheLastArray2Row(pLast, &pRow, pr->pSchema);
+
      saveOneRow(pRow, pResBlock, pr, slotIds);

+      // taosMemoryFree(pRow);
      tsdbCacheRelease(lruCache, h);

      pr->tableIndex += 1;
--- a/source/dnode/vnode/src/tsdb/tsdbCommit.c
+++ b/source/dnode/vnode/src/tsdb/tsdbCommit.c
@ -602,7 +602,7 @@ static int32_t tsdbGetOvlpNRow(STbDataIter *pIter, SBlock *pBlock) {

  iter.pRow = NULL;
  while (true) {
-    pRow = tsdbTbDataIterGet(pIter);
+    pRow = tsdbTbDataIterGet(&iter);

    if (pRow == NULL) break;
    key = TSDBROW_KEY(pRow);
@ -610,7 +610,7 @@ static int32_t tsdbGetOvlpNRow(STbDataIter *pIter, SBlock *pBlock) {
    c = tBlockCmprFn(&(SBlock){.maxKey = key, .minKey = key}, pBlock);
    if (c == 0) {
      nRow++;
-      tsdbTbDataIterNext(pIter);
+      tsdbTbDataIterNext(&iter);
    } else if (c > 0) {
      break;
    } else {
@ -635,7 +635,7 @@ static int32_t tsdbMergeAsSubBlock(SCommitter *pCommitter, STbDataIter *pIter, S
  code = tsdbCommitterUpdateRowSchema(pCommitter, pBlockIdx->suid, pBlockIdx->uid, TSDBROW_SVERSION(pRow));
  if (code) goto _err;
  while (true) {
-    if (pRow) break;
+    if (pRow == NULL) break;
    code = tBlockDataAppendRow(pBlockData, pRow, pCommitter->skmRow.pTSchema);
    if (code) goto _err;

--- a/source/dnode/vnode/src/tsdb/tsdbRead.c
+++ b/source/dnode/vnode/src/tsdb/tsdbRead.c
--- a/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c
+++ b/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c
@ -424,10 +424,12 @@ int32_t tsdbReadDelIdx(SDelFReader *pReader, SArray *aDelIdx, uint8_t **ppBuf) {

  ASSERT(n == size - sizeof(TSCKSUM));

+  tFree(pBuf);
  return code;

 _err:
  tsdbError("vgId:%d read del idx failed since %s", TD_VID(pReader->pTsdb->pVnode), tstrerror(code));
+  tFree(pBuf);
  return code;
 }

@ -969,6 +971,8 @@ int32_t tsdbReadColData(SDataFReader *pReader, SBlockIdx *pBlockIdx, SBlock *pBl
    SBlockData *pBlockData1 = &(SBlockData){0};
    SBlockData *pBlockData2 = &(SBlockData){0};

+    tBlockDataInit(pBlockData1);
+    tBlockDataInit(pBlockData2);
    for (int32_t iSubBlock = 1; iSubBlock < pBlock->nSubBlock; iSubBlock++) {
      code = tsdbReadSubColData(pReader, pBlockIdx, pBlock, iSubBlock, aColId, nCol, pBlockData1, ppBuf1, ppBuf2);
      if (code) goto _err;
@ -1106,6 +1110,8 @@ int32_t tsdbReadBlockData(SDataFReader *pReader, SBlockIdx *pBlockIdx, SBlock *p
    SBlockData *pBlockData1 = &(SBlockData){0};
    SBlockData *pBlockData2 = &(SBlockData){0};

+    tBlockDataInit(pBlockData1);
+    tBlockDataInit(pBlockData2);
    for (iSubBlock = 1; iSubBlock < pBlock->nSubBlock; iSubBlock++) {
      code = tsdbReadSubBlockData(pReader, pBlockIdx, pBlock, iSubBlock, pBlockData1, ppBuf1, ppBuf2);
      if (code) {
@ -1152,7 +1158,7 @@ _err:
 int32_t tsdbReadBlockSma(SDataFReader *pReader, SBlock *pBlock, SArray *aColumnDataAgg, uint8_t **ppBuf) {
  int32_t   code = 0;
  TdFilePtr pFD = pReader->pSmaFD;
-  int64_t   offset = pBlock->aSubBlock[0].offset;
+  int64_t   offset = pBlock->aSubBlock[0].sOffset;
  int64_t   size = pBlock->aSubBlock[0].nSma * sizeof(SColumnDataAgg) + sizeof(TSCKSUM);
  uint8_t  *pBuf = NULL;
  int64_t   n;
@ -1175,10 +1181,13 @@ int32_t tsdbReadBlockSma(SDataFReader *pReader, SBlock *pBlock, SArray *aColumnD
  if (n < 0) {
    code = TAOS_SYSTEM_ERROR(errno);
    goto _err;
+  } else if (n < size) {
+    code = TSDB_CODE_FILE_CORRUPTED;
+    goto _err;
  }

  // check
-  if (!taosCheckChecksumWhole(NULL, size)) {
+  if (!taosCheckChecksumWhole(*ppBuf, size)) {
    code = TSDB_CODE_FILE_CORRUPTED;
    goto _err;
  }
--- a/source/dnode/vnode/src/tsdb/tsdbRetention.c
+++ b/source/dnode/vnode/src/tsdb/tsdbRetention.c
@ -14,3 +14,29 @@
 */

 #include "tsdb.h"
+
+int32_t tsdbDoRetention(STsdb *pTsdb, int64_t now) {
+  int32_t code = 0;
+
+  // begin
+  code = tsdbFSBegin(pTsdb->fs);
+  if (code) goto _err;
+
+  // do retention
+  for (int32_t iSet = 0; iSet < taosArrayGetSize(pTsdb->fs->nState->aDFileSet); iSet++) {
+    SDFileSet *pDFileSet = (SDFileSet *)taosArrayGet(pTsdb->fs->nState->aDFileSet, iSet);
+
+    // TODO
+  }
+
+  // commit
+  code = tsdbFSCommit(pTsdb->fs);
+  if (code) goto _err;
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb do retention failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  return code;
+}
--- a/source/dnode/vnode/src/tsdb/tsdbSnapshot.c
+++ b/source/dnode/vnode/src/tsdb/tsdbSnapshot.c
@ -15,22 +15,686 @@

 #include "tsdb.h"

-struct STsdbSnapshotReader {
-  STsdb* pTsdb;
-  // TODO
+// STsdbSnapReader ========================================
+struct STsdbSnapReader {
+  STsdb*  pTsdb;
+  int64_t sver;
+  int64_t ever;
+  // for data file
+  int8_t        dataDone;
+  int32_t       fid;
+  SDataFReader* pDataFReader;
+  SArray*       aBlockIdx;  // SArray<SBlockIdx>
+  int32_t       iBlockIdx;
+  SBlockIdx*    pBlockIdx;
+  SMapData      mBlock;  // SMapData<SBlock>
+  int32_t       iBlock;
+  SBlockData    blkData;
+  // for del file
+  int8_t       delDone;
+  SDelFReader* pDelFReader;
+  int32_t      iDelIdx;
+  SArray*      aDelIdx;   // SArray<SDelIdx>
+  SArray*      aDelData;  // SArray<SDelData>
 };

-int32_t tsdbSnapshotReaderOpen(STsdb* pTsdb, STsdbSnapshotReader** ppReader, int64_t sver, int64_t ever) {
-  // TODO
-  return 0;
+static int32_t tsdbSnapReadData(STsdbSnapReader* pReader, uint8_t** ppData) {
+  int32_t code = 0;
+
+  while (true) {
+    if (pReader->pDataFReader == NULL) {
+      SDFileSet* pSet = NULL;
+
+      // search the next data file set to read (todo)
+      if (0 /* TODO */) {
+        code = TSDB_CODE_VND_READ_END;
+        goto _exit;
+      }
+
+      // open
+      code = tsdbDataFReaderOpen(&pReader->pDataFReader, pReader->pTsdb, pSet);
+      if (code) goto _err;
+
+      // SBlockIdx
+      code = tsdbReadBlockIdx(pReader->pDataFReader, pReader->aBlockIdx, NULL);
+      if (code) goto _err;
+
+      pReader->iBlockIdx = 0;
+      pReader->pBlockIdx = NULL;
+    }
+
+    while (true) {
+      if (pReader->pBlockIdx == NULL) {
+        if (pReader->iBlockIdx >= taosArrayGetSize(pReader->aBlockIdx)) {
+          tsdbDataFReaderClose(&pReader->pDataFReader);
+          break;
+        }
+
+        pReader->pBlockIdx = (SBlockIdx*)taosArrayGet(pReader->aBlockIdx, pReader->iBlockIdx);
+        pReader->iBlockIdx++;
+
+        // SBlock
+        code = tsdbReadBlock(pReader->pDataFReader, pReader->pBlockIdx, &pReader->mBlock, NULL);
+        if (code) goto _err;
+
+        pReader->iBlock = 0;
+      }
+
+      while (true) {
+        SBlock  block;
+        SBlock* pBlock = &block;
+
+        if (pReader->iBlock >= pReader->mBlock.nItem) {
+          pReader->pBlockIdx = NULL;
+          break;
+        }
+
+        tMapDataGetItemByIdx(&pReader->mBlock, pReader->iBlock, pBlock, tGetBlock);
+        pReader->iBlock++;
+
+        if ((pBlock->minVersion >= pReader->sver && pBlock->minVersion <= pReader->ever) ||
+            (pBlock->maxVersion >= pReader->sver && pBlock->maxVersion <= pReader->ever)) {
+          // overlap (todo)
+
+          code = tsdbReadBlockData(pReader->pDataFReader, pReader->pBlockIdx, pBlock, &pReader->blkData, NULL, NULL);
+          if (code) goto _err;
+
+          goto _exit;
+        }
+      }
+    }
+  }
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d snap read data failed since %s", TD_VID(pReader->pTsdb->pVnode), tstrerror(code));
+  return code;
 }

-int32_t tsdbSnapshotReaderClose(STsdbSnapshotReader* pReader) {
-  // TODO
-  return 0;
+static int32_t tsdbSnapReadDel(STsdbSnapReader* pReader, uint8_t** ppData) {
+  int32_t   code = 0;
+  STsdb*    pTsdb = pReader->pTsdb;
+  SDelFile* pDelFile = pTsdb->fs->cState->pDelFile;
+
+  if (pReader->pDelFReader == NULL) {
+    if (pDelFile == NULL) {
+      code = TSDB_CODE_VND_READ_END;
+      goto _exit;
+    }
+
+    // open
+    code = tsdbDelFReaderOpen(&pReader->pDelFReader, pDelFile, pTsdb, NULL);
+    if (code) goto _err;
+
+    // read index
+    code = tsdbReadDelIdx(pReader->pDelFReader, pReader->aDelIdx, NULL);
+    if (code) goto _err;
+
+    pReader->iDelIdx = 0;
+  }
+
+  while (pReader->iDelIdx < taosArrayGetSize(pReader->aDelIdx)) {
+    SDelIdx* pDelIdx = (SDelIdx*)taosArrayGet(pReader->aDelIdx, pReader->iDelIdx);
+    int32_t  size = 0;
+
+    pReader->iDelIdx++;
+
+    code = tsdbReadDelData(pReader->pDelFReader, pDelIdx, pReader->aDelData, NULL);
+    if (code) goto _err;
+
+    for (int32_t iDelData = 0; iDelData < taosArrayGetSize(pReader->aDelData); iDelData++) {
+      SDelData* pDelData = (SDelData*)taosArrayGet(pReader->aDelData, iDelData);
+
+      if (pDelData->version >= pReader->sver && pDelData->version <= pReader->ever) {
+        size += tPutDelData(NULL, pDelData);
+      }
+    }
+
+    if (size > 0) {
+      int64_t n = 0;
+
+      size = size + sizeof(SSnapDataHdr) + sizeof(TABLEID);
+      code = tRealloc(ppData, size);
+      if (code) goto _err;
+
+      // SSnapDataHdr
+      SSnapDataHdr* pSnapDataHdr = (SSnapDataHdr*)(*ppData + n);
+      pSnapDataHdr->type = 1;
+      pSnapDataHdr->size = size;  // TODO: size here may incorrect
+      n += sizeof(SSnapDataHdr);
+
+      // TABLEID
+      TABLEID* pId = (TABLEID*)(*ppData + n);
+      pId->suid = pDelIdx->suid;
+      pId->uid = pDelIdx->uid;
+      n += sizeof(*pId);
+
+      // DATA
+      for (int32_t iDelData = 0; iDelData < taosArrayGetSize(pReader->aDelData); iDelData++) {
+        SDelData* pDelData = (SDelData*)taosArrayGet(pReader->aDelData, iDelData);
+
+        if (pDelData->version >= pReader->sver && pDelData->version <= pReader->ever) {
+          n += tPutDelData(*ppData + n, pDelData);
+        }
+      }
+
+      goto _exit;
+    }
+  }
+
+  code = TSDB_CODE_VND_READ_END;
+  tsdbDelFReaderClose(&pReader->pDelFReader);
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d snap read del failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  return code;
 }

-int32_t tsdbSnapshotRead(STsdbSnapshotReader* pReader, void** ppData, uint32_t* nData) {
-  // TODO
-  return 0;
+int32_t tsdbSnapReaderOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapReader** ppReader) {
+  int32_t          code = 0;
+  STsdbSnapReader* pReader = NULL;
+
+  // alloc
+  pReader = (STsdbSnapReader*)taosMemoryCalloc(1, sizeof(*pReader));
+  if (pReader == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+  pReader->pTsdb = pTsdb;
+  pReader->sver = sver;
+  pReader->ever = ever;
+
+  pReader->aBlockIdx = taosArrayInit(0, sizeof(SBlockIdx));
+  if (pReader->aBlockIdx == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+
+  pReader->mBlock = tMapDataInit();
+
+  code = tBlockDataInit(&pReader->blkData);
+  if (code) goto _err;
+
+  pReader->aDelIdx = taosArrayInit(0, sizeof(SDelIdx));
+  if (pReader->aDelIdx == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+
+  pReader->aDelData = taosArrayInit(0, sizeof(SDelData));
+  if (pReader->aDelData == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+
+  *ppReader = pReader;
+  return code;
+
+_err:
+  tsdbError("vgId:%d snapshot reader open failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  *ppReader = NULL;
+  return code;
+}
+
+int32_t tsdbSnapReaderClose(STsdbSnapReader** ppReader) {
+  int32_t          code = 0;
+  STsdbSnapReader* pReader = *ppReader;
+
+  taosArrayDestroy(pReader->aDelData);
+  taosArrayDestroy(pReader->aDelIdx);
+  if (pReader->pDelFReader) {
+    tsdbDelFReaderClose(&pReader->pDelFReader);
+  }
+  tBlockDataClear(&pReader->blkData);
+  tMapDataClear(&pReader->mBlock);
+  taosArrayDestroy(pReader->aBlockIdx);
+  if (pReader->pDataFReader) {
+    tsdbDataFReaderClose(&pReader->pDataFReader);
+  }
+  taosMemoryFree(pReader);
+  *ppReader = NULL;
+
+  return code;
+}
+
+int32_t tsdbSnapRead(STsdbSnapReader* pReader, uint8_t** ppData) {
+  int32_t code = 0;
+
+  // read data file
+  if (!pReader->dataDone) {
+    code = tsdbSnapReadData(pReader, ppData);
+    if (code) {
+      if (code == TSDB_CODE_VND_READ_END) {
+        pReader->dataDone = 1;
+      } else {
+        goto _err;
+      }
+    } else {
+      goto _exit;
+    }
+  }
+
+  // read del file
+  if (!pReader->delDone) {
+    code = tsdbSnapReadDel(pReader, ppData);
+    if (code) {
+      if (code == TSDB_CODE_VND_READ_END) {
+        pReader->delDone = 1;
+      } else {
+        goto _err;
+      }
+    } else {
+      goto _exit;
+    }
+  }
+
+  code = TSDB_CODE_VND_READ_END;
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d snapshot read failed since %s", TD_VID(pReader->pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+// STsdbSnapWriter ========================================
+struct STsdbSnapWriter {
+  STsdb*  pTsdb;
+  int64_t sver;
+  int64_t ever;
+
+  // config
+  int32_t minutes;
+  int8_t  precision;
+
+  // for data file
+  int32_t       fid;
+  SDataFReader* pDataFReader;
+  SArray*       aBlockIdx;
+  int32_t       iBlockIdx;
+  SBlockIdx*    pBlockIdx;
+  SMapData      mBlock;
+  int32_t       iBlock;
+  SBlockData    blockData;
+  int32_t       iRow;
+
+  SDataFWriter* pDataFWriter;
+  SArray*       aBlockIdxN;
+  SBlockIdx     blockIdx;
+  SMapData      mBlockN;
+  SBlock        block;
+  SBlockData    nBlockData;
+
+  // for del file
+  SDelFReader* pDelFReader;
+  SDelFWriter* pDelFWriter;
+  int32_t      iDelIdx;
+  SArray*      aDelIdx;
+  SArray*      aDelData;
+  SArray*      aDelIdxN;
+};
+
+static int32_t tsdbSnapRollback(STsdbSnapWriter* pWriter) {
+  int32_t code = 0;
+  // TODO
+  return code;
+}
+
+static int32_t tsdbSnapCommit(STsdbSnapWriter* pWriter) {
+  int32_t code = 0;
+  // TODO
+  return code;
+}
+
+static int32_t tsdbSnapWriteDataEnd(STsdbSnapWriter* pWriter) {
+  int32_t code = 0;
+  STsdb*  pTsdb = pWriter->pTsdb;
+
+  if (pWriter->pDataFWriter == NULL) goto _exit;
+
+  // TODO
+
+  code = tsdbDataFWriterClose(&pWriter->pDataFWriter, 0);
+  if (code) goto _err;
+
+  if (pWriter->pDataFReader) {
+    code = tsdbDataFReaderClose(&pWriter->pDataFReader);
+    if (code) goto _err;
+  }
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshot writer data end failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+static int32_t tsdbSnapWriteAppendData(STsdbSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  int32_t     code = 0;
+  int32_t     iRow = 0;           // todo
+  int32_t     nRow = 0;           // todo
+  SBlockData* pBlockData = NULL;  // todo
+
+  while (iRow < nRow) {
+    code = tBlockDataAppendRow(&pWriter->nBlockData, &tsdbRowFromBlockData(pBlockData, iRow), NULL);
+    if (code) goto _err;
+  }
+
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshot write append data failed since %s", TD_VID(pWriter->pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+static int32_t tsdbSnapWriteData(STsdbSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  int32_t code = 0;
+  STsdb*  pTsdb = pWriter->pTsdb;
+  int64_t suid = 0;  // todo
+  int64_t uid = 0;   // todo
+  int64_t skey;      // todo
+  int64_t ekey;      // todo
+
+  int32_t fid = tsdbKeyFid(skey, pWriter->minutes, pWriter->precision);
+  ASSERT(fid == tsdbKeyFid(ekey, pWriter->minutes, pWriter->precision));
+
+  // begin
+  if (pWriter->pDataFWriter == NULL || pWriter->fid != fid) {
+    code = tsdbSnapWriteDataEnd(pWriter);
+    if (code) goto _err;
+
+    pWriter->fid = fid;
+    SDFileSet* pSet = tsdbFSStateGetDFileSet(pTsdb->fs->nState, fid);
+    // reader
+    if (pSet) {
+      // open
+      code = tsdbDataFReaderOpen(&pWriter->pDataFReader, pTsdb, pSet);
+      if (code) goto _err;
+
+      // SBlockIdx
+      code = tsdbReadBlockIdx(pWriter->pDataFReader, pWriter->aBlockIdx, NULL);
+      if (code) goto _err;
+    } else {
+      taosArrayClear(pWriter->aBlockIdx);
+    }
+    pWriter->iBlockIdx = 0;
+
+    // writer
+    SDFileSet wSet = {0};
+    if (pSet == NULL) {
+      wSet = (SDFileSet){0};  // todo
+    } else {
+      wSet = (SDFileSet){0};  // todo
+    }
+
+    code = tsdbDataFWriterOpen(&pWriter->pDataFWriter, pTsdb, &wSet);
+    if (code) goto _err;
+
+    taosArrayClear(pWriter->aBlockIdxN);
+  }
+
+  // process
+  TABLEID id = {0};    // TODO
+  TSKEY   minKey = 0;  // TODO
+  TSKEY   maxKey = 0;  // TODO
+
+  while (true) {
+    if (pWriter->pBlockIdx) {
+      int32_t c = tTABLEIDCmprFn(&id, pWriter->pBlockIdx);
+
+      if (c == 0) {
+      } else if (c < 0) {
+        // keep merge
+      } else {
+        // code = tsdbSnapWriteTableDataEnd(pWriter);
+        if (code) goto _err;
+
+        pWriter->iBlockIdx++;
+        if (pWriter->iBlockIdx < taosArrayGetSize(pWriter->aBlockIdx)) {
+          pWriter->pBlockIdx = (SBlockIdx*)taosArrayGet(pWriter->aBlockIdx, pWriter->iBlockIdx);
+        } else {
+          pWriter->pBlockIdx = NULL;
+        }
+
+        if (pWriter->pBlockIdx) {
+          code = tsdbReadBlock(pWriter->pDataFReader, pWriter->pBlockIdx, &pWriter->mBlock, NULL);
+          if (code) goto _err;
+        }
+      }
+    } else {
+      int32_t c = tTABLEIDCmprFn(&id, &pWriter->blockIdx);
+
+      if (c == 0) {
+        // merge commit the block data
+      } else if (c > 0) {
+        // code = tsdbSnapWriteTableDataEnd(pWriter);
+        if (code) goto _err;
+      } else {
+        ASSERT(0);
+      }
+    }
+  }
+
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshot write data failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+static int32_t tsdbSnapWriteDel(STsdbSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  int32_t code = 0;
+  STsdb*  pTsdb = pWriter->pTsdb;
+
+  if (pWriter->pDelFWriter == NULL) {
+    SDelFile* pDelFile = tsdbFSStateGetDelFile(pTsdb->fs->nState);
+
+    // reader
+    if (pDelFile) {
+      code = tsdbDelFReaderOpen(&pWriter->pDelFReader, pDelFile, pTsdb, NULL);
+      if (code) goto _err;
+
+      code = tsdbReadDelIdx(pWriter->pDelFReader, pWriter->aDelIdx, NULL);
+      if (code) goto _err;
+    }
+
+    // writer
+    SDelFile delFile = {.commitID = pTsdb->pVnode->state.commitID, .offset = 0, .size = 0};
+    code = tsdbDelFWriterOpen(&pWriter->pDelFWriter, &delFile, pTsdb);
+    if (code) goto _err;
+  }
+
+  // process the del data
+  TABLEID id = {0};  // todo
+
+  while (true) {
+    SDelIdx* pDelIdx = NULL;
+    int64_t  n = 0;
+    SDelData delData;
+    SDelIdx  delIdx;
+    int8_t   toBreak = 0;
+
+    if (pWriter->iDelIdx < taosArrayGetSize(pWriter->aDelIdx)) {
+      pDelIdx = taosArrayGet(pWriter->aDelIdx, pWriter->iDelIdx);
+    }
+
+    if (pDelIdx) {
+      int32_t c = tTABLEIDCmprFn(&id, pDelIdx);
+      if (c < 0) {
+        goto _new_del;
+      } else {
+        code = tsdbReadDelData(pWriter->pDelFReader, pDelIdx, pWriter->aDelData, NULL);
+        if (code) goto _err;
+
+        pWriter->iDelIdx++;
+        if (c == 0) {
+          toBreak = 1;
+          delIdx = (SDelIdx){.suid = id.suid, .uid = id.uid};
+          goto _merge_del;
+        } else {
+          delIdx = (SDelIdx){.suid = pDelIdx->suid, .uid = pDelIdx->uid};
+          goto _write_del;
+        }
+      }
+    }
+
+  _new_del:
+    toBreak = 1;
+    delIdx = (SDelIdx){.suid = id.suid, .uid = id.uid};
+    taosArrayClear(pWriter->aDelData);
+
+  _merge_del:
+    while (n < nData) {
+      n += tGetDelData(pData + n, &delData);
+      if (taosArrayPush(pWriter->aDelData, &delData) == NULL) {
+        code = TSDB_CODE_OUT_OF_MEMORY;
+        goto _err;
+      }
+    }
+
+  _write_del:
+    code = tsdbWriteDelData(pWriter->pDelFWriter, pWriter->aDelData, NULL, &delIdx);
+    if (code) goto _err;
+
+    if (taosArrayPush(pWriter->aDelIdxN, &delIdx) == NULL) {
+      code = TSDB_CODE_OUT_OF_MEMORY;
+      goto _err;
+    }
+
+    if (toBreak) break;
+  }
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshot write del failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+static int32_t tsdbSnapWriteDelEnd(STsdbSnapWriter* pWriter) {
+  int32_t code = 0;
+  STsdb*  pTsdb = pWriter->pTsdb;
+
+  if (pWriter->pDelFWriter == NULL) goto _exit;
+  for (; pWriter->iDelIdx < taosArrayGetSize(pWriter->aDelIdx); pWriter->iDelIdx++) {
+    SDelIdx* pDelIdx = (SDelIdx*)taosArrayGet(pWriter->aDelIdx, pWriter->iDelIdx);
+
+    code = tsdbReadDelData(pWriter->pDelFReader, pDelIdx, pWriter->aDelData, NULL);
+    if (code) goto _err;
+
+    SDelIdx delIdx = (SDelIdx){.suid = pDelIdx->suid, .uid = pDelIdx->uid};
+    code = tsdbWriteDelData(pWriter->pDelFWriter, pWriter->aDelData, NULL, &delIdx);
+    if (code) goto _err;
+
+    if (taosArrayPush(pWriter->aDelIdx, &delIdx) == NULL) {
+      code = TSDB_CODE_OUT_OF_MEMORY;
+      goto _err;
+    }
+  }
+
+  code = tsdbUpdateDelFileHdr(pWriter->pDelFWriter);
+  if (code) goto _err;
+
+  code = tsdbFSStateUpsertDelFile(pTsdb->fs->nState, &pWriter->pDelFWriter->fDel);
+  if (code) goto _err;
+
+  code = tsdbDelFWriterClose(&pWriter->pDelFWriter, 1);
+  if (code) goto _err;
+
+  if (pWriter->pDelFReader) {
+    code = tsdbDelFReaderClose(&pWriter->pDelFReader);
+    if (code) goto _err;
+  }
+
+_exit:
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshow write del end failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWriter** ppWriter) {
+  int32_t          code = 0;
+  STsdbSnapWriter* pWriter = NULL;
+
+  // alloc
+  pWriter = (STsdbSnapWriter*)taosMemoryCalloc(1, sizeof(*pWriter));
+  if (pWriter == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+  pWriter->pTsdb = pTsdb;
+  pWriter->sver = sver;
+  pWriter->ever = ever;
+
+  *ppWriter = pWriter;
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshot writer open failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
+  *ppWriter = NULL;
+  return code;
+}
+
+int32_t tsdbSnapWriterClose(STsdbSnapWriter** ppWriter, int8_t rollback) {
+  int32_t          code = 0;
+  STsdbSnapWriter* pWriter = *ppWriter;
+
+  if (rollback) {
+    code = tsdbSnapRollback(pWriter);
+    if (code) goto _err;
+  } else {
+    code = tsdbSnapWriteDataEnd(pWriter);
+    if (code) goto _err;
+
+    code = tsdbSnapWriteDelEnd(pWriter);
+    if (code) goto _err;
+
+    code = tsdbSnapCommit(pWriter);
+    if (code) goto _err;
+  }
+
+  taosMemoryFree(pWriter);
+  *ppWriter = NULL;
+
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshot writer close failed since %s", TD_VID(pWriter->pTsdb->pVnode), tstrerror(code));
+  return code;
+}
+
+int32_t tsdbSnapWrite(STsdbSnapWriter* pWriter, uint8_t* pData, uint32_t nData) {
+  int32_t code = 0;
+  int8_t  type = pData[0];
+
+  // ts data
+  if (type == 0) {
+    code = tsdbSnapWriteData(pWriter, pData + 1, nData - 1);
+    if (code) goto _err;
+  } else {
+    code = tsdbSnapWriteDataEnd(pWriter);
+    if (code) goto _err;
+  }
+
+  // del data
+  if (type == 1) {
+    code = tsdbSnapWriteDel(pWriter, pData + 1, nData - 1);
+    if (code) goto _err;
+  }
+
+  return code;
+
+_err:
+  tsdbError("vgId:%d tsdb snapshow write failed since %s", TD_VID(pWriter->pTsdb->pVnode), tstrerror(code));
+  return code;
 }
--- a/source/dnode/vnode/src/tsdb/tsdbUtil.c
+++ b/source/dnode/vnode/src/tsdb/tsdbUtil.c
@ -87,8 +87,10 @@ int32_t tPutMapData(uint8_t *p, SMapData *pMapData) {

  n += tPutI32v(p ? p + n : p, pMapData->nItem);
  if (pMapData->nItem) {
+    int32_t lOffset = 0;
    for (int32_t iItem = 0; iItem < pMapData->nItem; iItem++) {
-      n += tPutI32v(p ? p + n : p, pMapData->aOffset[iItem]);
+      n += tPutI32v(p ? p + n : p, pMapData->aOffset[iItem] - lOffset);
+      lOffset = pMapData->aOffset[iItem];
    }

    n += tPutI32v(p ? p + n : p, pMapData->nData);
@ -111,8 +113,11 @@ int32_t tGetMapData(uint8_t *p, SMapData *pMapData) {
  if (pMapData->nItem) {
    if (tRealloc((uint8_t **)&pMapData->aOffset, sizeof(int32_t) * pMapData->nItem)) return -1;

+    int32_t lOffset = 0;
    for (int32_t iItem = 0; iItem < pMapData->nItem; iItem++) {
      n += tGetI32v(p + n, &pMapData->aOffset[iItem]);
+      pMapData->aOffset[iItem] += lOffset;
+      lOffset = pMapData->aOffset[iItem];
    }

    n += tGetI32v(p + n, &pMapData->nData);
@ -821,16 +826,20 @@ int32_t tColDataCopy(SColData *pColDataSrc, SColData *pColDataDest) {
  int32_t code = 0;
  int32_t size;

+  ASSERT(pColDataSrc->nVal > 0);
+
  pColDataDest->cid = pColDataSrc->cid;
  pColDataDest->type = pColDataSrc->type;
  pColDataDest->smaOn = pColDataSrc->smaOn;
  pColDataDest->nVal = pColDataSrc->nVal;
  pColDataDest->flag = pColDataSrc->flag;

-  size = BIT2_SIZE(pColDataSrc->nVal);
-  code = tRealloc(&pColDataDest->pBitMap, size);
-  if (code) goto _exit;
-  memcpy(pColDataDest->pBitMap, pColDataSrc->pBitMap, size);
+  if (pColDataSrc->flag != HAS_NONE && pColDataSrc->flag != HAS_NULL && pColDataSrc->flag != HAS_VALUE) {
+    size = BIT2_SIZE(pColDataSrc->nVal);
+    code = tRealloc(&pColDataDest->pBitMap, size);
+    if (code) goto _exit;
+    memcpy(pColDataDest->pBitMap, pColDataSrc->pBitMap, size);
+  }

  if (IS_VAR_DATA_TYPE(pColDataDest->type)) {
    size = sizeof(int32_t) * pColDataSrc->nVal;
@ -1012,7 +1021,7 @@ int32_t tBlockDataAppendRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS
  SColData *pColData;
  SColVal  *pColVal;

-  ASSERT(nColData > 0);
+  if (nColData == 0) goto _exit;

  tRowIterInit(pIter, pRow, pTSchema);
  pColData = tBlockDataGetColDataByIdx(pBlockData, iColData);
@ -1042,6 +1051,7 @@ int32_t tBlockDataAppendRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS
    }
  }

+_exit:
  pBlockData->nRow++;
  return code;

@ -1230,10 +1240,26 @@ void tsdbCalcColDataSMA(SColData *pColData, SColumnDataAgg *pColAgg) {
          break;
        case TSDB_DATA_TYPE_SMALLINT:
          break;
-        case TSDB_DATA_TYPE_INT:
+        case TSDB_DATA_TYPE_INT: {
+          pColAgg->sum += colVal.value.i32;
+          if (pColAgg->min > colVal.value.i32) {
+            pColAgg->min = colVal.value.i32;
+          }
+          if (pColAgg->max < colVal.value.i32) {
+            pColAgg->max = colVal.value.i32;
+          }
          break;
-        case TSDB_DATA_TYPE_BIGINT:
+        }
+        case TSDB_DATA_TYPE_BIGINT: {
+          pColAgg->sum += colVal.value.i64;
+          if (pColAgg->min > colVal.value.i64) {
+            pColAgg->min = colVal.value.i64;
+          }
+          if (pColAgg->max < colVal.value.i64) {
+            pColAgg->max = colVal.value.i64;
+          }
          break;
+        }
        case TSDB_DATA_TYPE_FLOAT:
          break;
        case TSDB_DATA_TYPE_DOUBLE:
--- a/source/dnode/vnode/src/vnd/vnodeCommit.c
+++ b/source/dnode/vnode/src/vnd/vnodeCommit.c
@ -223,6 +223,7 @@ int vnodeCommit(SVnode *pVnode) {
  // save info
  info.config = pVnode->config;
  info.state.committed = pVnode->state.applied;
+  info.state.commitTerm = pVnode->state.applyTerm;
  info.state.commitID = pVnode->state.commitID;
  snprintf(dir, TSDB_FILENAME_LEN, "%s%s%s", tfsGetPrimaryPath(pVnode->pTfs), TD_DIRSEP, pVnode->path);
  if (vnodeSaveInfo(dir, &info) < 0) {
@ -270,7 +271,7 @@ int vnodeCommit(SVnode *pVnode) {
    ASSERT(0);
    return -1;
  }
-  
+
  pVnode->state.committed = info.state.committed;

  // postCommit
@ -316,6 +317,7 @@ static int vnodeEncodeState(const void *pObj, SJson *pJson) {

  if (tjsonAddIntegerToObject(pJson, "commit version", pState->committed) < 0) return -1;
  if (tjsonAddIntegerToObject(pJson, "commit ID", pState->commitID) < 0) return -1;
+  if (tjsonAddIntegerToObject(pJson, "commit term", pState->commitTerm) < 0) return -1;

  return 0;
 }
@ -328,6 +330,8 @@ static int vnodeDecodeState(const SJson *pJson, void *pObj) {
  if (code < 0) return -1;
  tjsonGetNumberValue(pJson, "commit ID", pState->commitID, code);
  if (code < 0) return -1;
+  tjsonGetNumberValue(pJson, "commit term", pState->commitTerm, code);
+  if (code < 0) return -1;

  return 0;
 }
--- a/source/dnode/vnode/src/vnd/vnodeModule.c
+++ b/source/dnode/vnode/src/vnd/vnodeModule.c
@ -100,6 +100,7 @@ void vnodeCleanup() {

  walCleanUp();
  tqCleanUp();
+  smaCleanUp();
 }

 int vnodeScheduleTask(int (*execute)(void*), void* arg) {
--- a/source/dnode/vnode/src/vnd/vnodeOpen.c
+++ b/source/dnode/vnode/src/vnd/vnodeOpen.c
@ -79,8 +79,10 @@ SVnode *vnodeOpen(const char *path, STfs *pTfs, SMsgCb msgCb) {
  strcpy(pVnode->path, path);
  pVnode->config = info.config;
  pVnode->state.committed = info.state.committed;
+  pVnode->state.commitTerm = info.state.commitTerm;
  pVnode->state.applied = info.state.committed;
  pVnode->state.commitID = info.state.commitID;
+  pVnode->state.commitTerm = info.state.commitTerm;
  pVnode->pTfs = pTfs;
  pVnode->msgCb = msgCb;
  pVnode->blockCount = 0;
@ -194,4 +196,9 @@ void vnodeStop(SVnode *pVnode) {}

 int64_t vnodeGetSyncHandle(SVnode *pVnode) { return pVnode->sync; }

-void vnodeGetSnapshot(SVnode *pVnode, SSnapshot *pSnapshot) { pSnapshot->lastApplyIndex = pVnode->state.committed; }
+void vnodeGetSnapshot(SVnode *pVnode, SSnapshot *pSnapshot) {
+  pSnapshot->data = NULL;
+  pSnapshot->lastApplyIndex = pVnode->state.committed;
+  pSnapshot->lastApplyTerm = pVnode->state.commitTerm;
+  pSnapshot->lastConfigIndex = -1;
+}
--- a/source/dnode/vnode/src/vnd/vnodeSnapshot.c
+++ b/source/dnode/vnode/src/vnd/vnodeSnapshot.c
@ -13,24 +13,27 @@
 * along with this program. If not, see <http://www.gnu.org/licenses/>.
 */

-#include "vnodeInt.h"
+#include "vnd.h"

-struct SVSnapshotReader {
-  SVnode              *pVnode;
-  int64_t              sver;
-  int64_t              ever;
-  int8_t               isMetaEnd;
-  int8_t               isTsdbEnd;
-  SMetaSnapshotReader *pMetaReader;
-  STsdbSnapshotReader *pTsdbReader;
-  void                *pData;
-  int32_t              nData;
+// SVSnapReader ========================================================
+struct SVSnapReader {
+  SVnode *pVnode;
+  int64_t sver;
+  int64_t ever;
+  // meta
+  int8_t           metaDone;
+  SMetaSnapReader *pMetaReader;
+  // tsdb
+  int8_t           tsdbDone;
+  STsdbSnapReader *pTsdbReader;
+  uint8_t         *pData;
 };

-int32_t vnodeSnapshotReaderOpen(SVnode *pVnode, SVSnapshotReader **ppReader, int64_t sver, int64_t ever) {
-  SVSnapshotReader *pReader = NULL;
+int32_t vnodeSnapReaderOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapReader **ppReader) {
+  int32_t       code = 0;
+  SVSnapReader *pReader = NULL;

-  pReader = (SVSnapshotReader *)taosMemoryCalloc(1, sizeof(*pReader));
+  pReader = (SVSnapReader *)taosMemoryCalloc(1, sizeof(*pReader));
  if (pReader == NULL) {
    terrno = TSDB_CODE_OUT_OF_MEMORY;
    goto _err;
@ -38,72 +41,169 @@ int32_t vnodeSnapshotReaderOpen(SVnode *pVnode, SVSnapshotReader **ppReader, int
  pReader->pVnode = pVnode;
  pReader->sver = sver;
  pReader->ever = ever;
-  pReader->isMetaEnd = 0;
-  pReader->isTsdbEnd = 0;

-  if (metaSnapshotReaderOpen(pVnode->pMeta, &pReader->pMetaReader, sver, ever) < 0) {
-    taosMemoryFree(pReader);
-    goto _err;
-  }
+  code = metaSnapReaderOpen(pVnode->pMeta, sver, ever, &pReader->pMetaReader);
+  if (code) goto _err;

-  if (tsdbSnapshotReaderOpen(pVnode->pTsdb, &pReader->pTsdbReader, sver, ever) < 0) {
-    metaSnapshotReaderClose(pReader->pMetaReader);
-    taosMemoryFree(pReader);
-    goto _err;
-  }
+  code = tsdbSnapReaderOpen(pVnode->pTsdb, sver, ever, &pReader->pTsdbReader);
+  if (code) goto _err;

-_exit:
  *ppReader = pReader;
-  return 0;
+  return code;

 _err:
+  vError("vgId:%d vnode snapshot reader open failed since %s", TD_VID(pVnode), tstrerror(code));
  *ppReader = NULL;
-  return -1;
+  return code;
 }

-int32_t vnodeSnapshotReaderClose(SVSnapshotReader *pReader) {
-  if (pReader) {
-    vnodeFree(pReader->pData);
-    tsdbSnapshotReaderClose(pReader->pTsdbReader);
-    metaSnapshotReaderClose(pReader->pMetaReader);
-    taosMemoryFree(pReader);
-  }
-  return 0;
-}
-
-int32_t vnodeSnapshotRead(SVSnapshotReader *pReader, const void **ppData, uint32_t *nData) {
+int32_t vnodeSnapReaderClose(SVSnapReader *pReader) {
  int32_t code = 0;

-  if (!pReader->isMetaEnd) {
-    code = metaSnapshotRead(pReader->pMetaReader, &pReader->pData, &pReader->nData);
+  tFree(pReader->pData);
+  if (pReader->pTsdbReader) tsdbSnapReaderClose(&pReader->pTsdbReader);
+  if (pReader->pMetaReader) metaSnapReaderClose(&pReader->pMetaReader);
+  taosMemoryFree(pReader);
+
+  return code;
+}
+
+int32_t vnodeSnapRead(SVSnapReader *pReader, uint8_t **ppData, uint32_t *nData) {
+  int32_t code = 0;
+
+  if (!pReader->metaDone) {
+    code = metaSnapRead(pReader->pMetaReader, &pReader->pData);
    if (code) {
      if (code == TSDB_CODE_VND_READ_END) {
-        pReader->isMetaEnd = 1;
+        pReader->metaDone = 1;
      } else {
-        return code;
+        goto _err;
      }
    } else {
      *ppData = pReader->pData;
-      *nData = pReader->nData;
-      return code;
+      *nData = sizeof(SSnapDataHdr) + ((SSnapDataHdr *)pReader->pData)->size;
+      goto _exit;
    }
  }

-  if (!pReader->isTsdbEnd) {
-    code = tsdbSnapshotRead(pReader->pTsdbReader, &pReader->pData, &pReader->nData);
+  if (!pReader->tsdbDone) {
+    code = tsdbSnapRead(pReader->pTsdbReader, &pReader->pData);
    if (code) {
      if (code == TSDB_CODE_VND_READ_END) {
-        pReader->isTsdbEnd = 1;
+        pReader->tsdbDone = 1;
      } else {
-        return code;
+        goto _err;
      }
    } else {
      *ppData = pReader->pData;
-      *nData = pReader->nData;
-      return code;
+      *nData = sizeof(SSnapDataHdr) + ((SSnapDataHdr *)pReader->pData)->size;
+      goto _exit;
    }
  }

  code = TSDB_CODE_VND_READ_END;
+
+_exit:
+  return code;
+
+_err:
+  vError("vgId:% snapshot read failed since %s", TD_VID(pReader->pVnode), tstrerror(code));
+  return code;
+}
+
+// SVSnapWriter ========================================================
+struct SVSnapWriter {
+  SVnode *pVnode;
+  int64_t sver;
+  int64_t ever;
+  // meta
+  SMetaSnapWriter *pMetaSnapWriter;
+  // tsdb
+  STsdbSnapWriter *pTsdbSnapWriter;
+};
+
+static int32_t vnodeSnapRollback(SVSnapWriter *pWriter) {
+  int32_t code = 0;
+  // TODO
+  return code;
+}
+
+static int32_t vnodeSnapCommit(SVSnapWriter *pWriter) {
+  int32_t code = 0;
+  // TODO
+  return code;
+}
+
+int32_t vnodeSnapWriterOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapWriter **ppWriter) {
+  int32_t       code = 0;
+  SVSnapWriter *pWriter = NULL;
+
+  // alloc
+  pWriter = (SVSnapWriter *)taosMemoryCalloc(1, sizeof(*pWriter));
+  if (pWriter == NULL) {
+    code = TSDB_CODE_OUT_OF_MEMORY;
+    goto _err;
+  }
+  pWriter->pVnode = pVnode;
+  pWriter->sver = sver;
+  pWriter->ever = ever;
+
+  return code;
+
+_err:
+  vError("vgId:%d vnode snapshot writer open failed since %s", TD_VID(pVnode), tstrerror(code));
+  return code;
+}
+
+int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback) {
+  int32_t code = 0;
+
+  if (rollback) {
+    code = vnodeSnapRollback(pWriter);
+    if (code) goto _err;
+  } else {
+    code = vnodeSnapCommit(pWriter);
+    if (code) goto _err;
+  }
+
+  taosMemoryFree(pWriter);
+  return code;
+
+_err:
+  vError("vgId:%d vnode snapshow writer close failed since %s", TD_VID(pWriter->pVnode), tstrerror(code));
+  return code;
+}
+
+int32_t vnodeSnapWrite(SVSnapWriter *pWriter, uint8_t *pData, uint32_t nData) {
+  int32_t       code = 0;
+  SSnapDataHdr *pSnapDataHdr = (SSnapDataHdr *)pData;
+  SVnode       *pVnode = pWriter->pVnode;
+
+  ASSERT(pSnapDataHdr->size + sizeof(SSnapDataHdr) == nData);
+
+  if (pSnapDataHdr->type == 0) {
+    // meta
+    if (pWriter->pMetaSnapWriter == NULL) {
+      code = metaSnapWriterOpen(pVnode->pMeta, pWriter->sver, pWriter->ever, &pWriter->pMetaSnapWriter);
+      if (code) goto _err;
+    }
+
+    code = metaSnapWrite(pWriter->pMetaSnapWriter, pData + sizeof(SSnapDataHdr), nData - sizeof(SSnapDataHdr));
+    if (code) goto _err;
+  } else {
+    // tsdb
+    if (pWriter->pTsdbSnapWriter == NULL) {
+      code = tsdbSnapWriterOpen(pVnode->pTsdb, pWriter->sver, pWriter->ever, &pWriter->pTsdbSnapWriter);
+      if (code) goto _err;
+    }
+
+    code = tsdbSnapWrite(pWriter->pTsdbSnapWriter, pData + sizeof(SSnapDataHdr), nData - sizeof(SSnapDataHdr));
+    if (code) goto _err;
+  }
+
+  return code;
+
+_err:
+  vError("vgId:%d vnode snapshot write failed since %s", TD_VID(pVnode), tstrerror(code));
  return code;
 }
--- a/source/dnode/vnode/src/vnd/vnodeStateMgr.c
+++ b/source/dnode/vnode/src/vnd/vnodeStateMgr.c
@ -1,14 +0,0 @@
-/*
- * Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
- *
- * This program is free software: you can use, redistribute, and/or modify
- * it under the terms of the GNU Affero General Public License, version 3
- * or later ("AGPL"), as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
- * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
- * FITNESS FOR A PARTICULAR PURPOSE.
- *
- * You should have received a copy of the GNU Affero General Public License
- * along with this program. If not, see <http://www.gnu.org/licenses/>.
- */
--- a/Show More
+++ b/Show More