Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/tsdb_snapshot

2022-07-15 06:55:17 +00:00 · 2022-07-15 06:55:17 +00:00 · 74b31e8c81
parent ec2a069185 a83d41f635
commit 74b31e8c81
84 changed files with 1223 additions and 2179 deletions
--- a/docs/en/07-develop/08-cache.md
+++ b/docs/en/07-develop/08-cache.md
@ -1,19 +1,52 @@
 ---
 sidebar_label: Cache
 title: Cache
-description: "The latest row of each table is kept in cache to provide high performance query of latest state."
+description: "Caching System inside TDengine"
 ---

+To achieve the purpose of high performance data writing and querying, TDengine employs a lot of caching technologies in both server side and client side. 
+
+## Write Cache
+
 The cache management policy in TDengine is First-In-First-Out (FIFO). FIFO is also known as insert driven cache management policy and it is different from read driven cache management, which is more commonly known as Least-Recently-Used (LRU). FIFO simply stores the latest data in cache and flushes the oldest data in cache to disk, when the cache usage reaches a threshold. In IoT use cases, it is the current state i.e. the latest or most recent data that is important. The cache policy in TDengine, like much of the design and architecture of TDengine, is based on the nature of IoT data.

-Caching the latest data provides the capability of retrieving data in milliseconds. With this capability, TDengine can be configured properly to be used as a caching system without deploying another separate caching system. This simplifies the system architecture and minimizes operational costs. The cache is emptied after TDengine is restarted. TDengine does not reload data from disk into cache, like a key-value caching system.
-
-The memory space used by the TDengine cache is fixed in size and configurable. It should be allocated based on application requirements and system resources. An independent memory pool is allocated for and managed by each vnode (virtual node) in TDengine. There is no sharing of memory pools between vnodes. All the tables belonging to a vnode share all the cache memory of the vnode.
-
-The memory pool is divided into blocks and data is stored in row format in memory and each block follows FIFO policy. The size of each block is determined by configuration parameter `cache` and the number of blocks for each vnode is determined by the parameter `blocks`. For each vnode, the total cache size is `cache * blocks`.  A cache block needs to ensure that each table can store at least dozens of records, to be efficient.
-
-`last_row` function can be used to retrieve the last row of a table or a STable to quickly show the current state of devices on monitoring screen. For example the below SQL statement retrieves the latest voltage of all meters in San Francisco, California.
+The memory space used by each vnode as write cache is determined when creating a database. Parameter `vgroups` and `buffer` can be used to specify the number of vnode and the size of write cache for each vnode when creating the database. Then, the total size of write cache for this database is `vgroups * buffer`.

 ```sql
-select last_row(voltage) from meters where location='California.SanFrancisco';
+create database db0 vgroups 100 buffer 16MB
 ```
+
+The above statement creates a database of 100 vnodes while each vnode has a write cache of 16MB.
+
+Even though in theory it's always better to have a larger cache, the extra effect would be very minor once the size of cache grows beyond a threshold. So normally it's enough to use the default value of `buffer` parameter.
+
+## Read Cache
+
+When creating a database, it's also possible to specify whether to cache the latest data of each sub table, using parameter `cachelast`. There are 3 cases:
+- 0: No cache for latest data
+- 1: The last row of each table is cached, `last_row` function can benefit significantly from it
+- 2: The latest non-NULL value of each column for each table is cached, `last` function can benefit very much when there is no `where`, `group by`, `order by` or `interval` clause
+- 3: Bot hthe last row and the latest non-NULL value of each column for each table are cached, identical to the behavior of both 1 and 2 are set together
+
+
+## Meta Cache
+
+To process data writing and querying efficiently, each vnode caches the metadata that's already retrieved. Parameters `pages` and `pagesize` are used to specify the size of metadata cache for each vnode.
+
+```sql
+create database db0 pages 128 pagesize 16kb
+```
+
+The above statement will create a database db0 each of whose vnode is allocated a meta cache of `128 * 16 KB = 2 MB` .
+
+## File System Cache
+
+TDengine utilizes WAL to provide basic reliability. The essential of WAL is to append data in a disk file, so the file system cache also plays an important role in the writing performance. Parameter `wal` can be used to specify the policy of writing WAL, there are 2 cases:
+- 1: Write data to WAL without calling fsync, the data is actually written to the file system cache without flushing immediately, in this way you can get better write performance
+- 2: Write data to WAL and invoke fsync, the data is immediately flushed to disk, in this way you can get higher reliability
+
+## Client Cache
+
+To improve the overall efficiency of processing data, besides the above caches, the core library `libtaos.so` (also referred to as `taosc`) which all client programs depend on also has its own cache. `taosc` caches the metadata of the databases, super tables, tables that the invoking client has accessed, plus other critical metadata such as the cluster topology. 
+
+When multiple client programs are accessing a TDengine cluster, if one of the clients modifies some metadata, the cache may become invalid in other clients. If this case happens, the client programs need to "reset query cache" to invalidate the whole cache so that `taosc` is enforced to repull the metadata it needs to rebuild the cache.
--- a/docs/zh/07-develop/08-cache.md
+++ b/docs/zh/07-develop/08-cache.md
@ -1,21 +1,49 @@
 ---
 sidebar_label: 缓存
 title: 缓存
-description: "提供写驱动的缓存管理机制，将每个表最近写入的一条记录持续保存在缓存中，可以提供高性能的最近状态查询。"
+description: "TDengine 内部的缓存设计"
 ---

+为了实现高效的写入和查询，TDengine 充分利用了各种缓存技术，本节将对 TDengine 中对缓存的使用做详细的说明。
+
+## 写缓存
+
 TDengine 采用时间驱动缓存管理策略（First-In-First-Out，FIFO），又称为写驱动的缓存管理机制。这种策略有别于读驱动的数据缓存模式（Least-Recent-Used，LRU），直接将最近写入的数据保存在系统的缓存中。当缓存达到临界值的时候，将最早的数据批量写入磁盘。一般意义上来说，对于物联网数据的使用，用户最为关心最近产生的数据，即当前状态。TDengine 充分利用了这一特性，将最近到达的（当前状态）数据保存在缓存中。

-TDengine 通过查询函数向用户提供毫秒级的数据获取能力。直接将最近到达的数据保存在缓存中，可以更加快速地响应用户针对最近一条或一批数据的查询分析，整体上提供更快的数据库查询响应能力。从这个意义上来说，可通过设置合适的配置参数将 TDengine 作为数据缓存来使用，而不需要再部署额外的缓存系统，可有效地简化系统架构，降低运维的成本。需要注意的是，TDengine 重启以后系统的缓存将被清空，之前缓存的数据均会被批量写入磁盘，缓存的数据将不会像专门的 key-value 缓存系统再将之前缓存的数据重新加载到缓存中。
-
-TDengine 分配固定大小的内存空间作为缓存空间，缓存空间可根据应用的需求和硬件资源配置。通过适当的设置缓存空间，TDengine 可以提供极高性能的写入和查询的支持。TDengine 中每个虚拟节点（virtual node）创建时分配独立的缓存池。每个虚拟节点管理自己的缓存池，不同虚拟节点间不共享缓存池。每个虚拟节点内部所属的全部表共享该虚拟节点的缓存池。
-
-TDengine 将内存池按块划分进行管理，数据在内存块里是以行（row）的形式存储。一个 vnode 的内存池是在 vnode 创建时按块分配好，而且每个内存块按照先进先出的原则进行管理。在创建内存池时，块的大小由系统配置参数 cache 决定；每个 vnode 中内存块的数目则由配置参数 blocks 决定。因此对于一个 vnode，总的内存大小为：`cache * blocks`。一个 cache block 需要保证每张表能存储至少几十条以上记录，才会有效率。
-
-你可以通过函数 last_row() 快速获取一张表或一张超级表的最后一条记录，这样很便于在大屏显示各设备的实时状态或采集值。例如：
+每个 vnode 的写入缓存大小在创建数据库时决定，创建数据库时的两个关键参数 vgroups 和 buffer 分别决定了该数据库中的数据由多少个 vgroup 处理，以及向其中的每个 vnode 分配多少写入缓存。

 ```sql
-select last_row(voltage) from meters where location='California.SanFrancisco';
+create database db0 vgroups 100 buffer 16MB
 ```

-该 SQL 语句将获取所有位于加利福尼亚州旧金山市的电表最后记录的电压值。
+理论上缓存越大越好，但超过一定阈值后再增加缓存对写入性能提升并无帮助，一般情况下使用默认值即可。
+
+## 读缓存
+
+在创建数据库时可以选择是否缓存该数据库中每个子表的最新数据。由参数 cachelast 设置，分为三种情况：
+- 0: 不缓存
+- 1: 缓存子表最近一行数据，这将显著改善 last_row 函数的性能
+- 2: 缓存子表每一列最近的非 NULL 值，这将显著改善无特殊影响（比如 WHERE, ORDER BY, GROUP BY, INTERVAL）时的 last 函数的性能
+- 3: 同时缓存行和列，即等同于上述 cachelast 值为 1 或 2 时的行为同时生效
+
+## 元数据缓存
+
+为了更高效地处理查询和写入，每个 vnode 都会缓存自己曾经获取到的元数据。元数据缓存由创建数据库时的两个参数 pages 和 pagesize 决定。
+
+```sql
+create database db0 pages 128 pagesize 16kb
+```
+
+上述语句会为数据库 db0 的每个 vnode 创建 128 个 page，每个 page 16kb 的元数据缓存。
+
+## 文件系统缓存
+
+TDengine 利用 WAL 技术来提供基本的数据可靠性。写入 WAL 本质上是以顺序追加的方式写入磁盘文件。此时文件系统缓存在写入性能中也会扮演关键角色。在创建数据库时可以利用 wal 参数来选择性能优先或者可靠性优先。
+- 1: 写 WAL 但不执行 fsync ，新写入 WAL 的数据保存在文件系统缓存中但并未写入磁盘，这种方式性能优先
+- 2: 写 WAL 且执行 fsync，新写入 WAL 的数据被立即同步到磁盘上，可靠性更高
+
+## 客户端缓存
+
+为了进一步提升整个系统的处理效率，除了以上提到的服务端缓存技术之外，在 TDengine 的所有客户端都要调用的核心库 libtaos.so （也称为 taosc ）中也充分利用了缓存技术。在 taosc 中会缓存所访问过的各个数据库、超级表以及子表的元数据，集群的拓扑结构等关键元数据。
+
+当有多个客户端同时访问 TDengine 集群，且其中一个客户端对某些元数据进行了修改的情况下，有可能会出现其它客户端所缓存的元数据不同步或失效的情况，此时需要在客户端执行 "reset query cache" 以让整个缓存失效从而强制重新拉取最新的元数据重新建立缓存。
--- a/include/libs/index/index.h
+++ b/include/libs/index/index.h
@ -28,7 +28,6 @@ extern "C" {

 typedef struct SIndex               SIndex;
 typedef struct SIndexTerm           SIndexTerm;
-typedef struct SIndexOpts           SIndexOpts;
 typedef struct SIndexMultiTermQuery SIndexMultiTermQuery;
 typedef struct SArray               SIndexMultiTerm;

@ -62,6 +61,9 @@ typedef enum {
  QUERY_MAX
 } EIndexQueryType;

+typedef struct SIndexOpts {
+  int32_t cacheSize;  // MB
+} SIndexOpts;
 /*
 * create multi query
 * @param oper (input, relation between querys)
@ -173,7 +175,7 @@ void             indexMultiTermDestroy(SIndexMultiTerm* terms);
 * @param:
 * @param:
 */
-SIndexOpts* indexOptsCreate();
+SIndexOpts* indexOptsCreate(int32_t cacheSize);
 void        indexOptsDestroy(SIndexOpts* opts);

 /*
--- a/include/libs/scalar/scalar.h
+++ b/include/libs/scalar/scalar.h
@ -101,6 +101,7 @@ int32_t countScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam
 int32_t sumScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
 int32_t minScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
 int32_t maxScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
+int32_t avgScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
 int32_t stddevScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);

 #ifdef __cplusplus
--- a/include/libs/scheduler/scheduler.h
+++ b/include/libs/scheduler/scheduler.h
@ -25,11 +25,6 @@ extern "C" {

 extern tsem_t schdRspSem;

-typedef struct SSchedulerCfg {
-  uint32_t maxJobNum;
-  int32_t  maxNodeTableNum;
-} SSchedulerCfg;
-
 typedef struct SQueryProfileSummary {
  int64_t startTs;      // Object created and added into the message queue
  int64_t endTs;        // the timestamp when the task is completed
@ -84,7 +79,7 @@ typedef struct SSchedulerReq {
 } SSchedulerReq;


-int32_t schedulerInit(SSchedulerCfg *cfg);
+int32_t schedulerInit(void);

 int32_t schedulerExecJob(SSchedulerReq *pReq, int64_t *pJob);

@ -96,6 +91,8 @@ int32_t schedulerGetTasksStatus(int64_t job, SArray *pSub);

 void schedulerStopQueryHb(void *pTrans);

+int32_t schedulerUpdatePolicy(int32_t policy);
+int32_t schedulerEnableReSchedule(bool enableResche);

 /**
 * Cancel query job
--- a/source/client/inc/clientInt.h
+++ b/source/client/inc/clientInt.h
@ -314,6 +314,11 @@ int taos_options_imp(TSDB_OPTION option, const char* str);

 void* openTransporter(const char* user, const char* auth, int32_t numOfThreads);

+typedef struct AsyncArg {
+  SRpcMsg msg;
+  SEpSet* pEpset;
+} AsyncArg;
+
 bool persistConnForSpecificMsg(void* parenct, tmsg_t msgType);
 void processMsgFromServer(void* parent, SRpcMsg* pMsg, SEpSet* pEpSet);

--- a/source/client/src/clientEnv.c
+++ b/source/client/src/clientEnv.c
@ -363,8 +363,7 @@ void taos_init_imp(void) {
  SCatalogCfg cfg = {.maxDBCacheNum = 100, .maxTblCacheNum = 100};
  catalogInit(&cfg);

-  SSchedulerCfg scfg = {.maxJobNum = 100};
-  schedulerInit(&scfg);
+  schedulerInit();
  tscDebug("starting to initialize TAOS driver");

  taosSetCoreDump(true);
--- a/source/client/src/clientImpl.c
+++ b/source/client/src/clientImpl.c
@ -834,6 +834,7 @@ void schedulerExecCb(SExecResult* pResult, void* param, int32_t code) {
    tscDebug("0x%" PRIx64 " client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%" PRIx64,
             pRequest->self, code, tstrerror(code), pRequest->retry, pRequest->requestId);
    pRequest->prevCode = code;
+    schedulerFreeJob(&pRequest->body.queryJob, 0);
    doAsyncQuery(pRequest, true);
    return;
  }
@ -1266,13 +1267,8 @@ void updateTargetEpSet(SMsgSendInfo* pSendInfo, STscObj* pTscObj, SRpcMsg* pMsg,
  }
 }

-typedef struct SchedArg {
-  SRpcMsg msg;
-  SEpSet* pEpset;
-} SchedArg;
-
 int32_t doProcessMsgFromServer(void* param) {
-  SchedArg* arg = (SchedArg*)param;
+  AsyncArg* arg = (AsyncArg*)param;
  SRpcMsg*  pMsg = &arg->msg;
  SEpSet*   pEpSet = arg->pEpset;

@ -1335,7 +1331,7 @@ void processMsgFromServer(void* parent, SRpcMsg* pMsg, SEpSet* pEpSet) {
    memcpy((void*)tEpSet, (void*)pEpSet, sizeof(SEpSet));
  }

-  SchedArg* arg = taosMemoryCalloc(1, sizeof(SchedArg));
+  AsyncArg* arg = taosMemoryCalloc(1, sizeof(AsyncArg));
  arg->msg = *pMsg;
  arg->pEpset = tEpSet;

--- a/source/client/src/clientMain.c
+++ b/source/client/src/clientMain.c
@ -131,6 +131,7 @@ void taos_close(TAOS *taos) {

  STscObj *pObj = acquireTscObj(*(int64_t *)taos);
  if (NULL == pObj) {
+    taosMemoryFree(taos);
    return;
  }

--- a/source/dnode/vnode/src/meta/metaOpen.c
+++ b/source/dnode/vnode/src/meta/metaOpen.c
@ -99,12 +99,12 @@ int metaOpen(SVnode *pVnode, SMeta **ppMeta) {
    goto _err;
  }

-  // open pTagIdx
-  // TODO(yihaoDeng), refactor later
  char indexFullPath[128] = {0};
  sprintf(indexFullPath, "%s/%s", pMeta->path, "invert");
  taosMkDir(indexFullPath);
-  ret = indexOpen(indexOptsCreate(), indexFullPath, (SIndex **)&pMeta->pTagIvtIdx);
+
+  SIndexOpts opts = {.cacheSize = 8 * 1024 * 1024};
+  ret = indexOpen(&opts, indexFullPath, (SIndex **)&pMeta->pTagIvtIdx);
  if (ret < 0) {
    metaError("vgId:%d, failed to open meta tag index since %s", TD_VID(pVnode), tstrerror(terrno));
    goto _err;
--- a/source/libs/command/CMakeLists.txt
+++ b/source/libs/command/CMakeLists.txt
@ -8,7 +8,7 @@ target_include_directories(

 target_link_libraries(
        command
-        PRIVATE os util nodes catalog function transport qcom
+        PRIVATE os util nodes catalog function transport qcom scheduler
 )

 if(${BUILD_TEST})
--- a/source/libs/command/inc/commandInt.h
+++ b/source/libs/command/inc/commandInt.h
@ -77,6 +77,10 @@ extern "C" {
 #define EXPLAIN_MODE_FORMAT "mode=%s"
 #define EXPLAIN_STRING_TYPE_FORMAT "%s"

+#define COMMAND_RESET_LOG "resetLog"
+#define COMMAND_SCHEDULE_POLICY "schedulePolicy"
+#define COMMAND_ENABLE_RESCHEDULE "enableReSchedule"
+
 typedef struct SExplainGroup {
  int32_t   nodeNum;
  int32_t   physiPlanExecNum;
--- a/source/libs/command/src/command.c
+++ b/source/libs/command/src/command.c
@ -17,6 +17,8 @@
 #include "catalog.h"
 #include "tdatablock.h"
 #include "tglobal.h"
+#include "commandInt.h"
+#include "scheduler.h"

 extern SConfig* tsCfg;

@ -479,7 +481,42 @@ static int32_t execShowCreateSTable(SShowCreateTableStmt* pStmt, SRetrieveTableR
  return execShowCreateTable(pStmt, pRsp);
 }

+static int32_t execAlterCmd(char* cmd, char* value, bool* processed) {
+  int32_t code = 0;
+  
+  if (0 == strcasecmp(cmd, COMMAND_RESET_LOG)) {
+    taosResetLog();
+    cfgDumpCfg(tsCfg, 0, false);
+  } else if (0 == strcasecmp(cmd, COMMAND_SCHEDULE_POLICY)) {
+    code = schedulerUpdatePolicy(atoi(value));
+  } else if (0 == strcasecmp(cmd, COMMAND_ENABLE_RESCHEDULE)) {
+    code = schedulerEnableReSchedule(atoi(value));
+  } else {
+    goto _return;
+  }
+
+  *processed = true;
+
+_return:
+
+  if (code) {
+    terrno = code;
+  }
+  
+  return code;  
+}
+
 static int32_t execAlterLocal(SAlterLocalStmt* pStmt) {
+  bool processed = false;
+  
+  if (execAlterCmd(pStmt->config, pStmt->value, &processed)) {
+    return terrno;
+  }
+
+  if (processed) {
+    goto _return;
+  }
+  
  if (cfgSetItem(tsCfg, pStmt->config, pStmt->value, CFG_STYPE_ALTER_CMD)) {
    return terrno;
  }
@ -488,6 +525,8 @@ static int32_t execAlterLocal(SAlterLocalStmt* pStmt) {
    return terrno;
  }

+_return:
+
  return TSDB_CODE_SUCCESS;
 }

--- a/source/libs/function/src/builtins.c
+++ b/source/libs/function/src/builtins.c
@ -1994,6 +1994,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
    .getEnvFunc   = getAvgFuncEnv,
    .initFunc     = avgFunctionSetup,
    .processFunc  = avgFunction,
+    .sprocessFunc = avgScalarFunction,
    .finalizeFunc = avgFinalize,
    .invertFunc   = avgInvertFunction,
    .combineFunc  = avgCombine,
--- a/source/libs/index/inc/indexFstFile.h
+++ b/source/libs/index/inc/indexFstFile.h
@ -27,7 +27,7 @@ extern "C" {
 #define DefaultMem 1024 * 1024

 static char tmpFile[] = "./index";
-typedef enum WriterType { TMemory, TFile } WriterType;
+typedef enum WriterType { TMEMORY, TFILE } WriterType;

 typedef struct IFileCtx {
  int (*write)(struct IFileCtx* ctx, uint8_t* buf, int len);
@ -35,6 +35,8 @@ typedef struct IFileCtx {
  int (*flush)(struct IFileCtx* ctx);
  int (*readFrom)(struct IFileCtx* ctx, uint8_t* buf, int len, int32_t offset);
  int (*size)(struct IFileCtx* ctx);
+
+  SLRUCache* lru;
  WriterType type;
  union {
    struct {
--- a/source/libs/index/inc/indexInt.h
+++ b/source/libs/index/inc/indexInt.h
@ -24,12 +24,9 @@
 #include "tchecksum.h"
 #include "thash.h"
 #include "tlog.h"
+#include "tlrucache.h"
 #include "tutil.h"

-#ifdef USE_LUCENE
-#include <lucene++/Lucene_c.h>
-#endif
-
 #ifdef __cplusplus
 extern "C" {
 #endif
@ -63,7 +60,7 @@ struct SIndex {

  int64_t    suid;      // current super table id, -1 is normal table
  int32_t    cVersion;  // current version allocated to cache
-
+  SLRUCache* lru;
  char*      path;

  int8_t        status;
@ -71,18 +68,7 @@ struct SIndex {
  TdThreadMutex mtx;
  tsem_t        sem;
  bool          quit;
-};
-
-struct SIndexOpts {
-#ifdef USE_LUCENE
-  void* opts;
-#endif
-
-#ifdef USE_INVERTED_INDEX
-  int32_t cacheSize;  // MB
-  // add cache module later
-#endif
-  int32_t cacheOpt;  // MB
+  SIndexOpts    opts;
 };

 struct SIndexMultiTermQuery {
--- a/source/libs/index/inc/indexTfile.h
+++ b/source/libs/index/inc/indexTfile.h
@ -71,6 +71,7 @@ typedef struct TFileReader {
  IFileCtx*   ctx;
  TFileHeader header;
  bool        remove;
+  void*       lru;
 } TFileReader;

 typedef struct IndexTFile {
@ -95,14 +96,14 @@ typedef struct TFileReaderOpt {
 } TFileReaderOpt;

 // tfile cache, manage tindex reader
-TFileCache*  tfileCacheCreate(const char* path);
+TFileCache*  tfileCacheCreate(SIndex* idx, const char* path);
 void         tfileCacheDestroy(TFileCache* tcache);
 TFileReader* tfileCacheGet(TFileCache* tcache, ICacheKey* key);
 void         tfileCachePut(TFileCache* tcache, ICacheKey* key, TFileReader* reader);

 TFileReader* tfileGetReaderByCol(IndexTFile* tf, uint64_t suid, char* colName);

-TFileReader* tfileReaderOpen(char* path, uint64_t suid, int64_t version, const char* colName);
+TFileReader* tfileReaderOpen(SIndex* idx, uint64_t suid, int64_t version, const char* colName);
 TFileReader* tfileReaderCreate(IFileCtx* ctx);
 void         tfileReaderDestroy(TFileReader* reader);
 int          tfileReaderSearch(TFileReader* reader, SIndexTermQuery* query, SIdxTRslt* tr);
@ -117,7 +118,7 @@ int          tfileWriterPut(TFileWriter* tw, void* data, bool order);
 int          tfileWriterFinish(TFileWriter* tw);

 //
-IndexTFile* idxTFileCreate(const char* path);
+IndexTFile* idxTFileCreate(SIndex* idx, const char* path);
 void        idxTFileDestroy(IndexTFile* tfile);
 int         idxTFilePut(void* tfile, SIndexTerm* term, uint64_t uid);
 int         idxTFileSearch(void* tfile, SIndexTermQuery* query, SIdxTRslt* tr);
--- a/source/libs/index/src/index.c
+++ b/source/libs/index/src/index.c
@ -103,44 +103,59 @@ static void indexWait(void* idx) {
 int indexOpen(SIndexOpts* opts, const char* path, SIndex** index) {
  int ret = TSDB_CODE_SUCCESS;
  taosThreadOnce(&isInit, indexInit);
-  SIndex* sIdx = taosMemoryCalloc(1, sizeof(SIndex));
-  if (sIdx == NULL) {
+  SIndex* idx = taosMemoryCalloc(1, sizeof(SIndex));
+  if (idx == NULL) {
    return TSDB_CODE_OUT_OF_MEMORY;
  }

-  sIdx->tindex = idxTFileCreate(path);
-  if (sIdx->tindex == NULL) {
+  idx->lru = taosLRUCacheInit(opts->cacheSize, -1, .5);
+  if (idx->lru == NULL) {
+    ret = TSDB_CODE_OUT_OF_MEMORY;
+    goto END;
+  }
+  taosLRUCacheSetStrictCapacity(idx->lru, false);
+
+  idx->tindex = idxTFileCreate(idx, path);
+  if (idx->tindex == NULL) {
    ret = TSDB_CODE_OUT_OF_MEMORY;
    goto END;
  }

-  sIdx->colObj = taosHashInit(8, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK);
-  sIdx->cVersion = 1;
-  sIdx->path = tstrdup(path);
-  taosThreadMutexInit(&sIdx->mtx, NULL);
-  tsem_init(&sIdx->sem, 0, 0);
+  idx->colObj = taosHashInit(8, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK);
+  idx->cVersion = 1;
+  idx->path = tstrdup(path);
+  taosThreadMutexInit(&idx->mtx, NULL);
+  tsem_init(&idx->sem, 0, 0);

-  sIdx->refId = idxAddRef(sIdx);
-  idxAcquireRef(sIdx->refId);
+  idx->refId = idxAddRef(idx);
+  idx->opts = *opts;
+  idxAcquireRef(idx->refId);

-  *index = sIdx;
+  *index = idx;
  return ret;

 END:
-  if (sIdx != NULL) {
-    indexClose(sIdx);
+  if (idx != NULL) {
+    indexClose(idx);
  }
  *index = NULL;
  return ret;
 }

 void indexDestroy(void* handle) {
-  SIndex* sIdx = handle;
-  taosThreadMutexDestroy(&sIdx->mtx);
-  tsem_destroy(&sIdx->sem);
-  idxTFileDestroy(sIdx->tindex);
-  taosMemoryFree(sIdx->path);
-  taosMemoryFree(sIdx);
+  SIndex* idx = handle;
+  taosThreadMutexDestroy(&idx->mtx);
+  tsem_destroy(&idx->sem);
+  idxTFileDestroy(idx->tindex);
+  taosMemoryFree(idx->path);
+
+  SLRUCache* lru = idx->lru;
+  if (lru != NULL) {
+    taosLRUCacheEraseUnrefEntries(lru);
+    taosLRUCacheCleanup(lru);
+  }
+  idx->lru = NULL;
+  taosMemoryFree(idx);
  return;
 }
 void indexClose(SIndex* sIdx) {
@ -159,6 +174,7 @@ void indexClose(SIndex* sIdx) {
    taosHashCleanup(sIdx->colObj);
    sIdx->colObj = NULL;
  }
+
  idxReleaseRef(sIdx->refId);
  idxRemoveRef(sIdx->refId);
 }
@ -234,8 +250,12 @@ int indexSearch(SIndex* index, SIndexMultiTermQuery* multiQuerys, SArray* result
 int indexDelete(SIndex* index, SIndexMultiTermQuery* query) { return 1; }
 // int indexRebuild(SIndex* index, SIndexOpts* opts) { return 0; }

-SIndexOpts* indexOptsCreate() { return NULL; }
-void        indexOptsDestroy(SIndexOpts* opts) { return; }
+SIndexOpts* indexOptsCreate(int32_t cacheSize) {
+  SIndexOpts* opts = taosMemoryCalloc(1, sizeof(SIndexOpts));
+  opts->cacheSize = cacheSize;
+  return opts;
+}
+void indexOptsDestroy(SIndexOpts* opts) { return taosMemoryFree(opts); }
 /*
 * @param: oper
 *
@ -641,7 +661,7 @@ static int idxGenTFile(SIndex* sIdx, IndexCache* cache, SArray* batch) {
  }
  tfileWriterClose(tw);

-  TFileReader* reader = tfileReaderOpen(sIdx->path, cache->suid, version, cache->colName);
+  TFileReader* reader = tfileReaderOpen(sIdx, cache->suid, version, cache->colName);
  if (reader == NULL) {
    return -1;
  }
--- a/source/libs/index/src/indexCache.c
+++ b/source/libs/index/src/indexCache.c
@ -462,8 +462,8 @@ Iterate* idxCacheIteratorCreate(IndexCache* cache) {
  if (cache->imm == NULL) {
    return NULL;
  }
-  Iterate* iiter = taosMemoryCalloc(1, sizeof(Iterate));
-  if (iiter == NULL) {
+  Iterate* iter = taosMemoryCalloc(1, sizeof(Iterate));
+  if (iter == NULL) {
    return NULL;
  }
  taosThreadMutexLock(&cache->mtx);
@ -471,15 +471,15 @@ Iterate* idxCacheIteratorCreate(IndexCache* cache) {
  idxMemRef(cache->imm);

  MemTable* tbl = cache->imm;
-  iiter->val.val = taosArrayInit(1, sizeof(uint64_t));
-  iiter->val.colVal = NULL;
-  iiter->iter = tbl != NULL ? tSkipListCreateIter(tbl->mem) : NULL;
-  iiter->next = idxCacheIteratorNext;
-  iiter->getValue = idxCacheIteratorGetValue;
+  iter->val.val = taosArrayInit(1, sizeof(uint64_t));
+  iter->val.colVal = NULL;
+  iter->iter = tbl != NULL ? tSkipListCreateIter(tbl->mem) : NULL;
+  iter->next = idxCacheIteratorNext;
+  iter->getValue = idxCacheIteratorGetValue;

  taosThreadMutexUnlock(&cache->mtx);

-  return iiter;
+  return iter;
 }
 void idxCacheIteratorDestroy(Iterate* iter) {
  if (iter == NULL) {
@ -564,13 +564,13 @@ int idxCachePut(void* cache, SIndexTerm* term, uint64_t uid) {
  idxMemUnRef(tbl);

  taosThreadMutexUnlock(&pCache->mtx);
-
  idxCacheUnRef(pCache);
  return 0;
  // encode end
 }
 void idxCacheForceToMerge(void* cache) {
  IndexCache* pCache = cache;
+
  idxCacheRef(pCache);
  taosThreadMutexLock(&pCache->mtx);

--- a/source/libs/index/src/indexFst.c
+++ b/source/libs/index/src/indexFst.c
@ -772,6 +772,7 @@ void fstBuilderDestroy(FstBuilder* b) {
  if (b == NULL) {
    return;
  }
+  fstBuilderFinish(b);

  idxFileDestroy(b->wrt);
  fstUnFinishedNodesDestroy(b->unfinished);
@ -1074,8 +1075,8 @@ FStmStBuilder* fstSearchWithState(Fst* fst, FAutoCtx* ctx) {
 }

 FstNode* fstGetRoot(Fst* fst) {
-  CompiledAddr rAddr = fstGetRootAddr(fst);
-  return fstGetNode(fst, rAddr);
+  CompiledAddr addr = fstGetRootAddr(fst);
+  return fstGetNode(fst, addr);
 }

 FstNode* fstGetNode(Fst* fst, CompiledAddr addr) {
--- a/source/libs/index/src/indexFstFile.c
+++ b/source/libs/index/src/indexFstFile.c
@ -4,8 +4,7 @@
 * This program is free software: you can use, redistribute, and/or modify
 * it under the terms of the GNU Affero General Public License, version 3
 * or later ("AGPL"), as published by the Free Software Foundation.
- *
- * This program is distributed in the hope that it will be useful, but WITHOUT
+ * * This program is distributed in the hope that it will be useful, but WITHOUT
 * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
 * FITNESS FOR A PARTICULAR PURPOSE.
 *
@ -14,13 +13,32 @@
 */

 #include "indexFstFile.h"
+#include "indexComm.h"
 #include "indexFstUtil.h"
 #include "indexInt.h"
+#include "indexUtil.h"
 #include "os.h"
 #include "tutil.h"

+static int32_t kBlockSize = 4096;
+
+typedef struct {
+  int32_t blockId;
+  int32_t nread;
+  char    buf[0];
+} SDataBlock;
+
+static void deleteDataBlockFromLRU(const void* key, size_t keyLen, void* value) { taosMemoryFree(value); }
+
+static void idxGenLRUKey(char* buf, const char* path, int32_t blockId) {
+  char* p = buf;
+  SERIALIZE_STR_VAR_TO_BUF(p, path, strlen(path));
+  SERIALIZE_VAR_TO_BUF(p, '_', char);
+  idxInt2str(blockId, p, 0);
+  return;
+}
 static int idxFileCtxDoWrite(IFileCtx* ctx, uint8_t* buf, int len) {
-  if (ctx->type == TFile) {
+  if (ctx->type == TFILE) {
    assert(len == taosWriteFile(ctx->file.pFile, buf, len));
  } else {
    memcpy(ctx->mem.buf + ctx->offset, buf, len);
@ -30,7 +48,7 @@ static int idxFileCtxDoWrite(IFileCtx* ctx, uint8_t* buf, int len) {
 }
 static int idxFileCtxDoRead(IFileCtx* ctx, uint8_t* buf, int len) {
  int nRead = 0;
-  if (ctx->type == TFile) {
+  if (ctx->type == TFILE) {
 #ifdef USE_MMAP
    nRead = len < ctx->file.size ? len : ctx->file.size;
    memcpy(buf, ctx->file.ptr, nRead);
@ -45,24 +63,54 @@ static int idxFileCtxDoRead(IFileCtx* ctx, uint8_t* buf, int len) {
  return nRead;
 }
 static int idxFileCtxDoReadFrom(IFileCtx* ctx, uint8_t* buf, int len, int32_t offset) {
-  int nRead = 0;
-  if (ctx->type == TFile) {
-    // tfLseek(ctx->file.pFile, offset, 0);
-#ifdef USE_MMAP
-    int32_t last = ctx->file.size - offset;
-    nRead = last >= len ? len : last;
-    memcpy(buf, ctx->file.ptr + offset, nRead);
-#else
-    nRead = taosPReadFile(ctx->file.pFile, buf, len, offset);
-#endif
+  int32_t total = 0, nread = 0;
+  int32_t blkId = offset / kBlockSize;
+  int32_t blkOffset = offset % kBlockSize;
+  int32_t blkLeft = kBlockSize - blkOffset;
+
+  do {
+    char key[128] = {0};
+    idxGenLRUKey(key, ctx->file.buf, blkId);
+    LRUHandle* h = taosLRUCacheLookup(ctx->lru, key, strlen(key));
+
+    if (h) {
+      SDataBlock* blk = taosLRUCacheValue(ctx->lru, h);
+      nread = TMIN(blkLeft, len);
+      memcpy(buf + total, blk->buf + blkOffset, nread);
+      taosLRUCacheRelease(ctx->lru, h, false);
    } else {
-    // refactor later
-    assert(0);
+      int32_t cacheMemSize = sizeof(SDataBlock) + kBlockSize;
+
+      SDataBlock* blk = taosMemoryCalloc(1, cacheMemSize);
+      blk->blockId = blkId;
+      blk->nread = taosPReadFile(ctx->file.pFile, blk->buf, kBlockSize, blkId * kBlockSize);
+      assert(blk->nread <= kBlockSize);
+      nread = TMIN(blkLeft, len);
+
+      if (blk->nread < kBlockSize && blk->nread < len) {
+        break;
      }
-  return nRead;
+      memcpy(buf + total, blk->buf + blkOffset, nread);
+
+      LRUStatus s = taosLRUCacheInsert(ctx->lru, key, strlen(key), blk, cacheMemSize, deleteDataBlockFromLRU, NULL,
+                                       TAOS_LRU_PRIORITY_LOW);
+      if (s != TAOS_LRU_STATUS_OK) {
+        return -1;
+      }
+    }
+    total += nread;
+    len -= nread;
+    offset += nread;
+
+    blkId = offset / kBlockSize;
+    blkOffset = offset % kBlockSize;
+    blkLeft = kBlockSize - blkOffset;
+
+  } while (len > 0);
+  return total;
 }
 static int idxFileCtxGetSize(IFileCtx* ctx) {
-  if (ctx->type == TFile) {
+  if (ctx->type == TFILE) {
    int64_t file_size = 0;
    taosStatFile(ctx->file.buf, &file_size, NULL);
    return (int)file_size;
@ -70,7 +118,7 @@ static int idxFileCtxGetSize(IFileCtx* ctx) {
  return 0;
 }
 static int idxFileCtxDoFlush(IFileCtx* ctx) {
-  if (ctx->type == TFile) {
+  if (ctx->type == TFILE) {
    taosFsyncFile(ctx->file.pFile);
  } else {
    // do nothing
@ -85,7 +133,7 @@ IFileCtx* idxFileCtxCreate(WriterType type, const char* path, bool readOnly, int
  }

  ctx->type = type;
-  if (ctx->type == TFile) {
+  if (ctx->type == TFILE) {
    // ugly code, refactor later
    ctx->file.readOnly = readOnly;
    memcpy(ctx->file.buf, path, strlen(path));
@ -93,8 +141,6 @@ IFileCtx* idxFileCtxCreate(WriterType type, const char* path, bool readOnly, int
      ctx->file.pFile = taosOpenFile(path, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND);
      taosFtruncateFile(ctx->file.pFile, 0);
      taosStatFile(path, &ctx->file.size, NULL);
-      // ctx->file.size = (int)size;
-
    } else {
      ctx->file.pFile = taosOpenFile(path, TD_FILE_READ);

@ -109,10 +155,11 @@ IFileCtx* idxFileCtxCreate(WriterType type, const char* path, bool readOnly, int
      indexError("failed to open file, error %d", errno);
      goto END;
    }
-  } else if (ctx->type == TMemory) {
+  } else if (ctx->type == TMEMORY) {
    ctx->mem.buf = taosMemoryCalloc(1, sizeof(char) * capacity);
    ctx->mem.cap = capacity;
  }
+
  ctx->write = idxFileCtxDoWrite;
  ctx->read = idxFileCtxDoRead;
  ctx->flush = idxFileCtxDoFlush;
@ -124,14 +171,14 @@ IFileCtx* idxFileCtxCreate(WriterType type, const char* path, bool readOnly, int

  return ctx;
 END:
-  if (ctx->type == TMemory) {
+  if (ctx->type == TMEMORY) {
    taosMemoryFree(ctx->mem.buf);
  }
  taosMemoryFree(ctx);
  return NULL;
 }
 void idxFileCtxDestroy(IFileCtx* ctx, bool remove) {
-  if (ctx->type == TMemory) {
+  if (ctx->type == TMEMORY) {
    taosMemoryFree(ctx->mem.buf);
  } else {
    ctx->flush(ctx);
@ -183,6 +230,7 @@ int idxFileWrite(IdxFstFile* write, uint8_t* buf, uint32_t len) {
  write->summer = taosCalcChecksum(write->summer, buf, len);
  return len;
 }
+
 int idxFileRead(IdxFstFile* write, uint8_t* buf, uint32_t len) {
  if (write == NULL) {
    return 0;
--- a/source/libs/index/src/indexTfile.c
+++ b/source/libs/index/src/indexTfile.c
@ -90,7 +90,7 @@ static int32_t (*tfSearch[][QUERY_MAX])(void* reader, SIndexTerm* tem, SIdxTRslt
    {tfSearchEqual_JSON, tfSearchPrefix_JSON, tfSearchSuffix_JSON, tfSearchRegex_JSON, tfSearchLessThan_JSON,
     tfSearchLessEqual_JSON, tfSearchGreaterThan_JSON, tfSearchGreaterEqual_JSON, tfSearchRange_JSON}};

-TFileCache* tfileCacheCreate(const char* path) {
+TFileCache* tfileCacheCreate(SIndex* idx, const char* path) {
  TFileCache* tcache = taosMemoryCalloc(1, sizeof(TFileCache));
  if (tcache == NULL) {
    return NULL;
@ -103,17 +103,20 @@ TFileCache* tfileCacheCreate(const char* path) {
  for (size_t i = 0; i < taosArrayGetSize(files); i++) {
    char* file = taosArrayGetP(files, i);

-    IFileCtx* wc = idxFileCtxCreate(TFile, file, true, 1024 * 1024 * 64);
-    if (wc == NULL) {
+    IFileCtx* ctx = idxFileCtxCreate(TFILE, file, true, 1024 * 1024 * 64);
+    if (ctx == NULL) {
      indexError("failed to open index:%s", file);
      goto End;
    }
+    ctx->lru = idx->lru;

-    TFileReader* reader = tfileReaderCreate(wc);
+    TFileReader* reader = tfileReaderCreate(ctx);
    if (reader == NULL) {
      indexInfo("skip invalid file: %s", file);
      continue;
    }
+    reader->lru = idx->lru;
+
    TFileHeader* header = &reader->header;
    ICacheKey    key = {.suid = header->suid, .colName = header->colName, .nColName = (int32_t)strlen(header->colName)};

@ -162,7 +165,6 @@ TFileReader* tfileCacheGet(TFileCache* tcache, ICacheKey* key) {
 void tfileCachePut(TFileCache* tcache, ICacheKey* key, TFileReader* reader) {
  char          buf[128] = {0};
  int32_t       sz = idxSerialCacheKey(key, buf);
-  // remove last version index reader
  TFileReader** p = taosHashGet(tcache->tableCache, buf, sz);
  if (p != NULL && *p != NULL) {
    TFileReader* oldRdr = *p;
@ -493,7 +495,7 @@ TFileWriter* tfileWriterOpen(char* path, uint64_t suid, int64_t version, const c
  char fullname[256] = {0};
  tfileGenFileFullName(fullname, path, suid, colName, version);
  // indexInfo("open write file name %s", fullname);
-  IFileCtx* wcx = idxFileCtxCreate(TFile, fullname, false, 1024 * 1024 * 64);
+  IFileCtx* wcx = idxFileCtxCreate(TFILE, fullname, false, 1024 * 1024 * 64);
  if (wcx == NULL) {
    return NULL;
  }
@ -506,16 +508,17 @@ TFileWriter* tfileWriterOpen(char* path, uint64_t suid, int64_t version, const c

  return tfileWriterCreate(wcx, &tfh);
 }
-TFileReader* tfileReaderOpen(char* path, uint64_t suid, int64_t version, const char* colName) {
+TFileReader* tfileReaderOpen(SIndex* idx, uint64_t suid, int64_t version, const char* colName) {
  char fullname[256] = {0};
-  tfileGenFileFullName(fullname, path, suid, colName, version);
+  tfileGenFileFullName(fullname, idx->path, suid, colName, version);

-  IFileCtx* wc = idxFileCtxCreate(TFile, fullname, true, 1024 * 1024 * 1024);
+  IFileCtx* wc = idxFileCtxCreate(TFILE, fullname, true, 1024 * 1024 * 1024);
  if (wc == NULL) {
    terrno = TAOS_SYSTEM_ERROR(errno);
    indexError("failed to open readonly file: %s, reason: %s", fullname, terrstr());
    return NULL;
  }
+  wc->lru = idx->lru;
  indexTrace("open read file name:%s, file size: %" PRId64 "", wc->file.buf, wc->file.size);

  TFileReader* reader = tfileReaderCreate(wc);
@ -598,17 +601,11 @@ int tfileWriterPut(TFileWriter* tw, void* data, bool order) {
      indexError("failed to write data: %s, offset: %d len: %d", v->colVal, v->offset,
                 (int)taosArrayGetSize(v->tableId));
    } else {
-      // indexInfo("success to write data: %s, offset: %d len: %d", v->colVal, v->offset,
-      //          (int)taosArrayGetSize(v->tableId));
-
-      // indexInfo("tfile write data size: %d", tw->ctx->size(tw->ctx));
+      indexInfo("success to write data: %s, offset: %d len: %d", v->colVal, v->offset,
+                (int)taosArrayGetSize(v->tableId));
    }
  }
-
-  fstBuilderFinish(tw->fb);
  fstBuilderDestroy(tw->fb);
-  tw->fb = NULL;
-
  tfileWriteFooter(tw);
  return 0;
 }
@ -627,8 +624,8 @@ void tfileWriterDestroy(TFileWriter* tw) {
  taosMemoryFree(tw);
 }

-IndexTFile* idxTFileCreate(const char* path) {
-  TFileCache* cache = tfileCacheCreate(path);
+IndexTFile* idxTFileCreate(SIndex* idx, const char* path) {
+  TFileCache* cache = tfileCacheCreate(idx, path);
  if (cache == NULL) {
    return NULL;
  }
@ -859,18 +856,6 @@ static int tfileWriteData(TFileWriter* write, TFileValue* tval) {
    return 0;
  }
  return -1;
-
-  // if (colType == TSDB_DATA_TYPE_BINARY || colType == TSDB_DATA_TYPE_NCHAR) {
-  //  FstSlice key = fstSliceCreate((uint8_t*)(tval->colVal), (size_t)strlen(tval->colVal));
-  //  if (fstBuilderInsert(write->fb, key, tval->offset)) {
-  //    fstSliceDestroy(&key);
-  //    return 0;
-  //  }
-  //  fstSliceDestroy(&key);
-  //  return -1;
-  //} else {
-  //  // handle other type later
-  //}
 }
 static int tfileWriteFooter(TFileWriter* write) {
  char  buf[sizeof(FILE_MAGIC_NUMBER) + 1] = {0};
@ -887,6 +872,7 @@ static int tfileReaderLoadHeader(TFileReader* reader) {
  char buf[TFILE_HEADER_SIZE] = {0};

  int64_t nread = reader->ctx->readFrom(reader->ctx, buf, sizeof(buf), 0);
+
  if (nread == -1) {
    indexError("actual Read: %d, to read: %d, errno: %d, filename: %s", (int)(nread), (int)sizeof(buf), errno,
               reader->ctx->file.buf);
@ -914,7 +900,7 @@ static int tfileReaderLoadFst(TFileReader* reader) {
  int64_t cost = taosGetTimestampUs() - ts;
  indexInfo("nread = %d, and fst offset=%d, fst size: %d, filename: %s, file size: %" PRId64 ", time cost: %" PRId64
            "us",
-            nread, reader->header.fstOffset, fstSize, ctx->file.buf, ctx->file.size, cost);
+            nread, reader->header.fstOffset, fstSize, ctx->file.buf, size, cost);
  // we assuse fst size less than FST_MAX_SIZE
  assert(nread > 0 && nread <= fstSize);

--- a/source/libs/index/test/fstTest.cc
+++ b/source/libs/index/test/fstTest.cc
@ -19,7 +19,7 @@ class FstWriter {
 public:
  FstWriter() {
    taosRemoveFile(fileName.c_str());
-    _wc = idxFileCtxCreate(TFile, fileName.c_str(), false, 64 * 1024 * 1024);
+    _wc = idxFileCtxCreate(TFILE, fileName.c_str(), false, 64 * 1024 * 1024);
    _b = fstBuilderCreate(_wc, 0);
  }
  bool Put(const std::string& key, uint64_t val) {
@ -34,7 +34,7 @@ class FstWriter {
    return ok;
  }
  ~FstWriter() {
-    fstBuilderFinish(_b);
+    // fstBuilderFinish(_b);
    fstBuilderDestroy(_b);

    idxFileCtxDestroy(_wc, false);
@ -48,7 +48,7 @@ class FstWriter {
 class FstReadMemory {
 public:
  FstReadMemory(int32_t size, const std::string& fileName = TD_TMP_DIR_PATH "tindex.tindex") {
-    _wc = idxFileCtxCreate(TFile, fileName.c_str(), true, 64 * 1024);
+    _wc = idxFileCtxCreate(TFILE, fileName.c_str(), true, 64 * 1024);
    _w = idxFileCreate(_wc);
    _size = size;
    memset((void*)&_s, 0, sizeof(_s));
@ -598,7 +598,9 @@ void fst_get(Fst* fst) {
 void validateTFile(char* arg) {
  std::thread threads[NUM_OF_THREAD];
  // std::vector<std::thread> threads;
-  TFileReader* reader = tfileReaderOpen(arg, 0, 20000000, "tag1");
+  SIndex* index = (SIndex*)taosMemoryCalloc(1, sizeof(SIndex));
+  index->path = strdup(arg);
+  TFileReader* reader = tfileReaderOpen(index, 0, 20000000, "tag1");

  for (int i = 0; i < NUM_OF_THREAD; i++) {
    threads[i] = std::thread(fst_get, reader->fst);
@ -617,7 +619,7 @@ void iterTFileReader(char* path, char* uid, char* colName, char* ver) {
  uint64_t suid = atoi(uid);
  int      version = atoi(ver);

-  TFileReader* reader = tfileReaderOpen(path, suid, version, colName);
+  TFileReader* reader = tfileReaderOpen(NULL, suid, version, colName);

  Iterate* iter = tfileIteratorCreate(reader);
  bool     tn = iter ? iter->next(iter) : false;
--- a/source/libs/index/test/fstUT.cc
+++ b/source/libs/index/test/fstUT.cc
@ -39,7 +39,7 @@ static void EnvCleanup() {}
 class FstWriter {
 public:
  FstWriter() {
-    _wc = idxFileCtxCreate(TFile, tindex, false, 64 * 1024 * 1024);
+    _wc = idxFileCtxCreate(TFILE, tindex, false, 64 * 1024 * 1024);
    _b = fstBuilderCreate(_wc, 0);
  }
  bool Put(const std::string& key, uint64_t val) {
@ -54,7 +54,6 @@ class FstWriter {
    return ok;
  }
  ~FstWriter() {
-    fstBuilderFinish(_b);
    fstBuilderDestroy(_b);

    idxFileCtxDestroy(_wc, false);
@ -68,7 +67,7 @@ class FstWriter {
 class FstReadMemory {
 public:
  FstReadMemory(size_t size) {
-    _wc = idxFileCtxCreate(TFile, tindex, true, 64 * 1024);
+    _wc = idxFileCtxCreate(TFILE, tindex, true, 64 * 1024);
    _w = idxFileCreate(_wc);
    _size = size;
    memset((void*)&_s, 0, sizeof(_s));
--- a/source/libs/index/test/indexTests.cc
+++ b/source/libs/index/test/indexTests.cc
@ -50,7 +50,7 @@ class DebugInfo {
 class FstWriter {
 public:
  FstWriter() {
-    _wc = idxFileCtxCreate(TFile, TD_TMP_DIR_PATH "tindex", false, 64 * 1024 * 1024);
+    _wc = idxFileCtxCreate(TFILE, TD_TMP_DIR_PATH "tindex", false, 64 * 1024 * 1024);
    _b = fstBuilderCreate(NULL, 0);
  }
  bool Put(const std::string& key, uint64_t val) {
@ -60,7 +60,7 @@ class FstWriter {
    return ok;
  }
  ~FstWriter() {
-    fstBuilderFinish(_b);
+    // fstBuilderFinish(_b);
    fstBuilderDestroy(_b);

    idxFileCtxDestroy(_wc, false);
@ -74,7 +74,7 @@ class FstWriter {
 class FstReadMemory {
 public:
  FstReadMemory(size_t size) {
-    _wc = idxFileCtxCreate(TFile, TD_TMP_DIR_PATH "tindex", true, 64 * 1024);
+    _wc = idxFileCtxCreate(TFILE, TD_TMP_DIR_PATH "tindex", true, 64 * 1024);
    _w = idxFileCreate(_wc);
    _size = size;
    memset((void*)&_s, 0, sizeof(_s));
@ -292,14 +292,12 @@ class IndexEnv : public ::testing::Test {
  virtual void SetUp() {
    initLog();
    taosRemoveDir(path);
-    opts = indexOptsCreate();
-    int ret = indexOpen(opts, path, &index);
+    SIndexOpts opts;
+    opts.cacheSize = 1024 * 1024 * 4;
+    int ret = indexOpen(&opts, path, &index);
    assert(ret == 0);
  }
-  virtual void TearDown() {
-    indexClose(index);
-    indexOptsDestroy(opts);
-  }
+  virtual void TearDown() { indexClose(index); }

  const char* path = TD_TMP_DIR_PATH "tindex";
  SIndexOpts* opts;
@ -391,13 +389,15 @@ class TFileObj {

    fileName_ = path;

-    IFileCtx* ctx = idxFileCtxCreate(TFile, path.c_str(), false, 64 * 1024 * 1024);
+    IFileCtx* ctx = idxFileCtxCreate(TFILE, path.c_str(), false, 64 * 1024 * 1024);
+    ctx->lru = taosLRUCacheInit(1024 * 1024 * 4, -1, .5);

    writer_ = tfileWriterCreate(ctx, &header);
    return writer_ != NULL ? true : false;
  }
  bool InitReader() {
-    IFileCtx* ctx = idxFileCtxCreate(TFile, fileName_.c_str(), true, 64 * 1024 * 1024);
+    IFileCtx* ctx = idxFileCtxCreate(TFILE, fileName_.c_str(), true, 64 * 1024 * 1024);
+    ctx->lru = taosLRUCacheInit(1024 * 1024 * 4, -1, .5);
    reader_ = tfileReaderCreate(ctx);
    return reader_ != NULL ? true : false;
  }
@ -698,6 +698,9 @@ class IndexObj {
      taosMkDir(dir.c_str());
    }
    taosMkDir(dir.c_str());
+    SIndexOpts opts;
+    opts.cacheSize = 1024 * 1024 * 4;
+
    int ret = indexOpen(&opts, dir.c_str(), &idx);
    if (ret != 0) {
      // opt
@ -845,7 +848,7 @@ class IndexObj {
  }

 private:
-  SIndexOpts opts;
+  SIndexOpts* opts;
  SIndex*     idx;
  int         numOfWrite;
  int         numOfRead;
--- a/source/libs/index/test/jsonUT.cc
+++ b/source/libs/index/test/jsonUT.cc
@ -54,13 +54,12 @@ class JsonEnv : public ::testing::Test {
    printf("set up\n");

    initLog();
-    opts = indexOptsCreate();
+    opts = indexOptsCreate(1024 * 1024 * 4);
    int ret = indexJsonOpen(opts, dir.c_str(), &index);
    assert(ret == 0);
  }
  virtual void TearDown() {
    indexJsonClose(index);
-    indexOptsDestroy(opts);
    printf("destory\n");
    taosMsleep(1000);
  }
--- a/source/libs/nodes/src/nodesUtilFuncs.c
+++ b/source/libs/nodes/src/nodesUtilFuncs.c
@ -388,6 +388,11 @@ static void destroyDataSinkNode(SDataSinkNode* pNode) { nodesDestroyNode((SNode*

 static void destroyExprNode(SExprNode* pExpr) { taosArrayDestroy(pExpr->pAssociation); }

+static void nodesDestroyNodePointer(void* node) {
+  SNode* pNode = *(SNode**)node;
+  nodesDestroyNode(pNode);
+}
+
 void nodesDestroyNode(SNode* pNode) {
  if (NULL == pNode) {
    return;
@ -718,6 +723,7 @@ void nodesDestroyNode(SNode* pNode) {
      }
      taosArrayDestroy(pQuery->pDbList);
      taosArrayDestroy(pQuery->pTableList);
+      taosArrayDestroyEx(pQuery->pPlaceholderValues, nodesDestroyNodePointer);
      break;
    }
    case QUERY_NODE_LOGIC_PLAN_SCAN: {
--- a/source/libs/parser/src/parAstParser.c
+++ b/source/libs/parser/src/parAstParser.c
@ -118,36 +118,33 @@ static bool needGetTableIndex(SNode* pStmt) {
  return false;
 }

-static int32_t collectMetaKeyFromRealTableImpl(SCollectMetaKeyCxt* pCxt, SRealTableNode* pRealTable,
+static int32_t collectMetaKeyFromRealTableImpl(SCollectMetaKeyCxt* pCxt, const char* pDb, const char* pTable,
                                               AUTH_TYPE authType) {
-  int32_t code = reserveTableMetaInCache(pCxt->pParseCxt->acctId, pRealTable->table.dbName, pRealTable->table.tableName,
-                                         pCxt->pMetaCache);
+  int32_t code = reserveTableMetaInCache(pCxt->pParseCxt->acctId, pDb, pTable, pCxt->pMetaCache);
  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveTableVgroupInCache(pCxt->pParseCxt->acctId, pRealTable->table.dbName, pRealTable->table.tableName,
-                                     pCxt->pMetaCache);
+    code = reserveTableVgroupInCache(pCxt->pParseCxt->acctId, pDb, pTable, pCxt->pMetaCache);
  }
  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveUserAuthInCache(pCxt->pParseCxt->acctId, pCxt->pParseCxt->pUser, pRealTable->table.dbName, authType,
-                                  pCxt->pMetaCache);
+    code = reserveUserAuthInCache(pCxt->pParseCxt->acctId, pCxt->pParseCxt->pUser, pDb, authType, pCxt->pMetaCache);
  }
  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveDbVgInfoInCache(pCxt->pParseCxt->acctId, pRealTable->table.dbName, pCxt->pMetaCache);
+    code = reserveDbVgInfoInCache(pCxt->pParseCxt->acctId, pDb, pCxt->pMetaCache);
  }
  if (TSDB_CODE_SUCCESS == code && needGetTableIndex(pCxt->pStmt)) {
-    code = reserveTableIndexInCache(pCxt->pParseCxt->acctId, pRealTable->table.dbName, pRealTable->table.tableName,
-                                    pCxt->pMetaCache);
+    code = reserveTableIndexInCache(pCxt->pParseCxt->acctId, pDb, pTable, pCxt->pMetaCache);
  }
-  if (TSDB_CODE_SUCCESS == code && (0 == strcmp(pRealTable->table.tableName, TSDB_INS_TABLE_DNODE_VARIABLES))) {
+  if (TSDB_CODE_SUCCESS == code && (0 == strcmp(pTable, TSDB_INS_TABLE_DNODE_VARIABLES))) {
    code = reserveDnodeRequiredInCache(pCxt->pMetaCache);
  }
  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveDbCfgInCache(pCxt->pParseCxt->acctId, pRealTable->table.dbName, pCxt->pMetaCache);
+    code = reserveDbCfgInCache(pCxt->pParseCxt->acctId, pDb, pCxt->pMetaCache);
  }
  return code;
 }

 static EDealRes collectMetaKeyFromRealTable(SCollectMetaKeyFromExprCxt* pCxt, SRealTableNode* pRealTable) {
-  pCxt->errCode = collectMetaKeyFromRealTableImpl(pCxt->pComCxt, pRealTable, AUTH_TYPE_READ);
+  pCxt->errCode = collectMetaKeyFromRealTableImpl(pCxt->pComCxt, pRealTable->table.dbName, pRealTable->table.tableName,
+                                                  AUTH_TYPE_READ);
  return TSDB_CODE_SUCCESS == pCxt->errCode ? DEAL_RES_CONTINUE : DEAL_RES_ERROR;
 }

@ -454,11 +451,13 @@ static int32_t collectMetaKeyFromShowTransactions(SCollectMetaKeyCxt* pCxt, SSho
 }

 static int32_t collectMetaKeyFromDelete(SCollectMetaKeyCxt* pCxt, SDeleteStmt* pStmt) {
-  return collectMetaKeyFromRealTableImpl(pCxt, (SRealTableNode*)pStmt->pFromTable, AUTH_TYPE_WRITE);
+  STableNode* pTable = (STableNode*)pStmt->pFromTable;
+  return collectMetaKeyFromRealTableImpl(pCxt, pTable->dbName, pTable->tableName, AUTH_TYPE_WRITE);
 }

 static int32_t collectMetaKeyFromInsert(SCollectMetaKeyCxt* pCxt, SInsertStmt* pStmt) {
-  int32_t code = collectMetaKeyFromRealTableImpl(pCxt, (SRealTableNode*)pStmt->pTable, AUTH_TYPE_WRITE);
+  STableNode* pTable = (STableNode*)pStmt->pTable;
+  int32_t     code = collectMetaKeyFromRealTableImpl(pCxt, pTable->dbName, pTable->tableName, AUTH_TYPE_WRITE);
  if (TSDB_CODE_SUCCESS == code) {
    code = collectMetaKeyFromQuery(pCxt, pStmt->pQuery);
  }
@ -471,14 +470,7 @@ static int32_t collectMetaKeyFromShowBlockDist(SCollectMetaKeyCxt* pCxt, SShowTa
  strcpy(name.tname, pStmt->tableName);
  int32_t code = catalogRemoveTableMeta(pCxt->pParseCxt->pCatalog, &name);
  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveTableMetaInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, pCxt->pMetaCache);
-  }
-
-  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveTableVgroupInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, pCxt->pMetaCache);
-  }
-  if (TSDB_CODE_SUCCESS == code) {
-    code = reserveDbVgInfoInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pCxt->pMetaCache);
+    code = collectMetaKeyFromRealTableImpl(pCxt, pStmt->dbName, pStmt->tableName, AUTH_TYPE_READ);
  }
  return code;
 }
--- a/source/libs/parser/src/parTranslater.c
+++ b/source/libs/parser/src/parTranslater.c
@ -1177,6 +1177,29 @@ static int32_t translateRepeatScanFunc(STranslateContext* pCxt, SFunctionNode* p
                                 "%s is only supported in single table query", pFunc->functionName);
 }

+static bool isStar(SNode* pNode) {
+  return (QUERY_NODE_COLUMN == nodeType(pNode)) && ('\0' == ((SColumnNode*)pNode)->tableAlias[0]) &&
+         (0 == strcmp(((SColumnNode*)pNode)->colName, "*"));
+}
+
+static bool isTableStar(SNode* pNode) {
+  return (QUERY_NODE_COLUMN == nodeType(pNode)) && ('\0' != ((SColumnNode*)pNode)->tableAlias[0]) &&
+         (0 == strcmp(((SColumnNode*)pNode)->colName, "*"));
+}
+
+static int32_t translateMultiResFunc(STranslateContext* pCxt, SFunctionNode* pFunc) {
+  if (!fmIsMultiResFunc(pFunc->funcId)) {
+    return TSDB_CODE_SUCCESS;
+  }
+  if (SQL_CLAUSE_SELECT != pCxt->currClause ) {
+    SNode* pPara = nodesListGetNode(pFunc->pParameterList, 0);
+    if (isStar(pPara) || isTableStar(pPara)) {
+      return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_NOT_ALLOWED_FUNC,
+                                     "%s(*) is only supported in SELECTed list", pFunc->functionName);
+    }
+  }
+  return TSDB_CODE_SUCCESS;
+}
 static void setFuncClassification(SNode* pCurrStmt, SFunctionNode* pFunc) {
  if (NULL != pCurrStmt && QUERY_NODE_SELECT_STMT == nodeType(pCurrStmt)) {
    SSelectStmt* pSelect = (SSelectStmt*)pCurrStmt;
@ -1311,6 +1334,9 @@ static int32_t translateNoramlFunction(STranslateContext* pCxt, SFunctionNode* p
  if (TSDB_CODE_SUCCESS == code) {
    code = translateRepeatScanFunc(pCxt, pFunc);
  }
+  if (TSDB_CODE_SUCCESS == code) {
+    code = translateMultiResFunc(pCxt, pFunc);
+  }
  if (TSDB_CODE_SUCCESS == code) {
    setFuncClassification(pCxt->pCurrStmt, pFunc);
  }
@ -1908,16 +1934,6 @@ static int32_t createTableAllCols(STranslateContext* pCxt, SColumnNode* pCol, bo
  return code;
 }

-static bool isStar(SNode* pNode) {
-  return (QUERY_NODE_COLUMN == nodeType(pNode)) && ('\0' == ((SColumnNode*)pNode)->tableAlias[0]) &&
-         (0 == strcmp(((SColumnNode*)pNode)->colName, "*"));
-}
-
-static bool isTableStar(SNode* pNode) {
-  return (QUERY_NODE_COLUMN == nodeType(pNode)) && ('\0' != ((SColumnNode*)pNode)->tableAlias[0]) &&
-         (0 == strcmp(((SColumnNode*)pNode)->colName, "*"));
-}
-
 static int32_t createMultiResFuncsParas(STranslateContext* pCxt, SNodeList* pSrcParas, SNodeList** pOutput) {
  int32_t code = TSDB_CODE_SUCCESS;

--- a/source/libs/parser/test/mockCatalog.cpp
+++ b/source/libs/parser/test/mockCatalog.cpp
@ -159,8 +159,8 @@ void generatePerformanceSchema(MockCatalogService* mcs) {
 *          c4         |       column       |       DOUBLE       |    8     |
 *          c5         |       column       |       DOUBLE       |    8     |
 */
-void generateTestTables(MockCatalogService* mcs) {
-  ITableBuilder& builder = mcs->createTableBuilder("test", "t1", TSDB_NORMAL_TABLE, 6)
+void generateTestTables(MockCatalogService* mcs, const std::string& db) {
+  ITableBuilder& builder = mcs->createTableBuilder(db, "t1", TSDB_NORMAL_TABLE, 6)
                               .setPrecision(TSDB_TIME_PRECISION_MILLI)
                               .setVgid(1)
                               .addColumn("ts", TSDB_DATA_TYPE_TIMESTAMP)
@ -193,9 +193,9 @@ void generateTestTables(MockCatalogService* mcs) {
 *         jtag        |        tag         |        json        |    --    |
 * Child Table: st2s1, st2s2
 */
-void generateTestStables(MockCatalogService* mcs) {
+void generateTestStables(MockCatalogService* mcs, const std::string& db) {
  {
-    ITableBuilder& builder = mcs->createTableBuilder("test", "st1", TSDB_SUPER_TABLE, 3, 3)
+    ITableBuilder& builder = mcs->createTableBuilder(db, "st1", TSDB_SUPER_TABLE, 3, 3)
                                 .setPrecision(TSDB_TIME_PRECISION_MILLI)
                                 .addColumn("ts", TSDB_DATA_TYPE_TIMESTAMP)
                                 .addColumn("c1", TSDB_DATA_TYPE_INT)
@ -204,20 +204,20 @@ void generateTestStables(MockCatalogService* mcs) {
                                 .addTag("tag2", TSDB_DATA_TYPE_BINARY, 20)
                                 .addTag("tag3", TSDB_DATA_TYPE_TIMESTAMP);
    builder.done();
-    mcs->createSubTable("test", "st1", "st1s1", 1);
-    mcs->createSubTable("test", "st1", "st1s2", 2);
-    mcs->createSubTable("test", "st1", "st1s3", 1);
+    mcs->createSubTable(db, "st1", "st1s1", 1);
+    mcs->createSubTable(db, "st1", "st1s2", 2);
+    mcs->createSubTable(db, "st1", "st1s3", 1);
  }
  {
-    ITableBuilder& builder = mcs->createTableBuilder("test", "st2", TSDB_SUPER_TABLE, 3, 1)
+    ITableBuilder& builder = mcs->createTableBuilder(db, "st2", TSDB_SUPER_TABLE, 3, 1)
                                 .setPrecision(TSDB_TIME_PRECISION_MILLI)
                                 .addColumn("ts", TSDB_DATA_TYPE_TIMESTAMP)
                                 .addColumn("c1", TSDB_DATA_TYPE_INT)
                                 .addColumn("c2", TSDB_DATA_TYPE_BINARY, 20)
                                 .addTag("jtag", TSDB_DATA_TYPE_JSON);
    builder.done();
-    mcs->createSubTable("test", "st2", "st2s1", 1);
-    mcs->createSubTable("test", "st2", "st2s2", 2);
+    mcs->createSubTable(db, "st2", "st2s1", 1);
+    mcs->createSubTable(db, "st2", "st2s2", 2);
  }
 }

@ -237,6 +237,11 @@ void generateDatabases(MockCatalogService* mcs) {
  mcs->createDatabase(TSDB_INFORMATION_SCHEMA_DB);
  mcs->createDatabase(TSDB_PERFORMANCE_SCHEMA_DB);
  mcs->createDatabase("test");
+  generateTestTables(g_mockCatalogService.get(), "test");
+  generateTestStables(g_mockCatalogService.get(), "test");
+  mcs->createDatabase("cache_db", false, 1);
+  generateTestTables(g_mockCatalogService.get(), "cache_db");
+  generateTestStables(g_mockCatalogService.get(), "cache_db");
  mcs->createDatabase("rollup_db", true);
 }

@ -369,11 +374,8 @@ void generateMetaData() {
  generateDatabases(g_mockCatalogService.get());
  generateInformationSchema(g_mockCatalogService.get());
  generatePerformanceSchema(g_mockCatalogService.get());
-  generateTestTables(g_mockCatalogService.get());
-  generateTestStables(g_mockCatalogService.get());
  generateFunctions(g_mockCatalogService.get());
  generateDnodes(g_mockCatalogService.get());
-  g_mockCatalogService->showTables();
 }

 void destroyMetaDataEnv() { g_mockCatalogService.reset(); }
--- a/source/libs/parser/test/mockCatalogService.cpp
+++ b/source/libs/parser/test/mockCatalogService.cpp
@ -334,11 +334,12 @@ class MockCatalogServiceImpl {
    dnode_.insert(std::make_pair(dnodeId, epSet));
  }

-  void createDatabase(const std::string& db, bool rollup) {
+  void createDatabase(const std::string& db, bool rollup, int8_t cacheLast) {
    SDbCfgInfo cfg = {0};
    if (rollup) {
      cfg.pRetensions = taosArrayInit(TARRAY_MIN_SIZE, sizeof(SRetention));
    }
+    cfg.cacheLast = cacheLast;
    dbCfg_.insert(std::make_pair(db, cfg));
  }

@ -627,7 +628,9 @@ void MockCatalogService::createDnode(int32_t dnodeId, const std::string& host, i
  impl_->createDnode(dnodeId, host, port);
 }

-void MockCatalogService::createDatabase(const std::string& db, bool rollup) { impl_->createDatabase(db, rollup); }
+void MockCatalogService::createDatabase(const std::string& db, bool rollup, int8_t cacheLast) {
+  impl_->createDatabase(db, rollup, cacheLast);
+}

 int32_t MockCatalogService::catalogGetTableMeta(const SName* pTableName, STableMeta** pTableMeta) const {
  return impl_->catalogGetTableMeta(pTableName, pTableMeta);
--- a/source/libs/parser/test/mockCatalogService.h
+++ b/source/libs/parser/test/mockCatalogService.h
@ -63,7 +63,7 @@ class MockCatalogService {
  void createFunction(const std::string& func, int8_t funcType, int8_t outputType, int32_t outputLen, int32_t bufSize);
  void createSmaIndex(const SMCreateSmaReq* pReq);
  void createDnode(int32_t dnodeId, const std::string& host, int16_t port);
-  void createDatabase(const std::string& db, bool rollup = false);
+  void createDatabase(const std::string& db, bool rollup = false, int8_t cacheLast = 0);

  int32_t catalogGetTableMeta(const SName* pTableName, STableMeta** pTableMeta) const;
  int32_t catalogGetTableHashVgroup(const SName* pTableName, SVgroupInfo* vgInfo) const;
--- a/source/libs/parser/test/parShowToUse.cpp
+++ b/source/libs/parser/test/parShowToUse.cpp
@ -179,6 +179,12 @@ TEST_F(ParserShowToUseTest, showTables) {
  run("SHOW test.tables like 'c%'");
 }

+TEST_F(ParserShowToUseTest, showTableDistributed) {
+  useDb("root", "test");
+
+  run("SHOW TABLE DISTRIBUTED st1");
+}
+
 // todo SHOW topics

 TEST_F(ParserShowToUseTest, showUsers) {
--- a/source/libs/planner/src/planOptimizer.c
+++ b/source/libs/planner/src/planOptimizer.c
@ -2011,6 +2011,7 @@ static int32_t lastRowScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogic
  SNode* pNode = NULL;
  FOREACH(pNode, pAgg->pAggFuncs) {
    SFunctionNode* pFunc = (SFunctionNode*)pNode;
+    if (FUNCTION_TYPE_LAST_ROW == pFunc->funcType) {
      int32_t len = snprintf(pFunc->functionName, sizeof(pFunc->functionName), "_cache_last_row");
      pFunc->functionName[len] = '\0';
      int32_t code = fmGetFuncInfo(pFunc, NULL, 0);
@ -2018,6 +2019,7 @@ static int32_t lastRowScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogic
        return code;
      }
    }
+  }
  pAgg->hasLastRow = false;

  ((SScanLogicNode*)nodesListGetNode(pAgg->node.pChildren, 0))->scanType = SCAN_TYPE_LAST_ROW;
--- a/source/libs/planner/test/planBasicTest.cpp
+++ b/source/libs/planner/test/planBasicTest.cpp
@ -98,6 +98,24 @@ TEST_F(PlanBasicTest, interpFunc) {
 }

 TEST_F(PlanBasicTest, lastRowFunc) {
+  useDb("root", "cache_db");
+
+  run("SELECT LAST_ROW(c1) FROM t1");
+
+  run("SELECT LAST_ROW(*) FROM t1");
+
+  run("SELECT LAST_ROW(c1, c2) FROM t1");
+
+  run("SELECT LAST_ROW(c1), c2 FROM t1");
+
+  run("SELECT LAST_ROW(c1) FROM st1");
+
+  run("SELECT LAST_ROW(c1) FROM st1 PARTITION BY TBNAME");
+
+  run("SELECT LAST_ROW(c1), SUM(c3) FROM t1");
+}
+
+TEST_F(PlanBasicTest, lastRowFuncWithoutCache) {
  useDb("root", "test");

  run("SELECT LAST_ROW(c1) FROM t1");
--- a/source/libs/qworker/inc/qwInt.h
+++ b/source/libs/qworker/inc/qwInt.h
@ -378,6 +378,7 @@ void qwDbgDumpMgmtInfo(SQWorker *mgmt);
 int32_t qwDbgValidateStatus(QW_FPARAMS_DEF, int8_t oriStatus, int8_t newStatus, bool *ignore);
 int32_t qwDbgBuildAndSendRedirectRsp(int32_t rspType, SRpcHandleInfo *pConn, int32_t code, SEpSet *pEpSet);
 int32_t qwAddTaskCtx(QW_FPARAMS_DEF);
+int32_t qwDbgResponseRedirect(SQWMsg *qwMsg, SQWTaskCtx *ctx);


 #ifdef __cplusplus
--- a/source/libs/qworker/inc/qwMsg.h
+++ b/source/libs/qworker/inc/qwMsg.h
@ -24,7 +24,7 @@ extern "C" {
 #include "dataSinkMgt.h"

 int32_t qwAbortPrerocessQuery(QW_FPARAMS_DEF);
-int32_t qwPrerocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg);
+int32_t qwPreprocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg);
 int32_t qwProcessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg, const char* sql);
 int32_t qwProcessCQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg);
 int32_t qwProcessReady(QW_FPARAMS_DEF, SQWMsg *qwMsg);
--- a/source/libs/qworker/src/qwDbg.c
+++ b/source/libs/qworker/src/qwDbg.c
@ -9,7 +9,7 @@
 #include "tmsg.h"
 #include "tname.h"

-SQWDebug     gQWDebug = {.statusEnable = true, .dumpEnable = false, .tmp = true};
+SQWDebug     gQWDebug = {.statusEnable = true, .dumpEnable = false, .tmp = false};

 int32_t qwDbgValidateStatus(QW_FPARAMS_DEF, int8_t oriStatus, int8_t newStatus, bool *ignore) {
  if (!gQWDebug.statusEnable) {
@ -147,9 +147,9 @@ int32_t qwDbgBuildAndSendRedirectRsp(int32_t rspType, SRpcHandleInfo *pConn, int
  return TSDB_CODE_SUCCESS;
 }

-int32_t qwDbgResponseREdirect(SQWMsg *qwMsg, SQWTaskCtx *ctx) {
+int32_t qwDbgResponseRedirect(SQWMsg *qwMsg, SQWTaskCtx *ctx) {
  if (gQWDebug.tmp) {
-    if (TDMT_SCH_QUERY == qwMsg->msgType) {
+    if (TDMT_SCH_QUERY == qwMsg->msgType && (0 == taosRand() % 3)) {
      SEpSet epSet = {0};
      epSet.inUse = 1;
      epSet.numOfEps = 3;
@ -160,15 +160,14 @@ int32_t qwDbgResponseREdirect(SQWMsg *qwMsg, SQWTaskCtx *ctx) {
      strcpy(epSet.eps[2].fqdn, "localhost");
      epSet.eps[2].port = 7300;

+      ctx->phase = QW_PHASE_POST_QUERY;      
      qwDbgBuildAndSendRedirectRsp(qwMsg->msgType + 1, &qwMsg->connInfo, TSDB_CODE_RPC_REDIRECT, &epSet);
-      gQWDebug.tmp = false;
      return TSDB_CODE_SUCCESS;
    }
    
-    if (TDMT_SCH_MERGE_QUERY == qwMsg->msgType) {
+    if (TDMT_SCH_MERGE_QUERY == qwMsg->msgType && (0 == taosRand() % 3)) {
      ctx->phase = QW_PHASE_POST_QUERY;
      qwDbgBuildAndSendRedirectRsp(qwMsg->msgType + 1, &qwMsg->connInfo, TSDB_CODE_RPC_REDIRECT, NULL);
-      gQWDebug.tmp = false;
      return TSDB_CODE_SUCCESS;
    }
  }
--- a/source/libs/qworker/src/qwMsg.c
+++ b/source/libs/qworker/src/qwMsg.c
@ -315,10 +315,10 @@ int32_t qWorkerPreprocessQueryMsg(void *qWorkerMgmt, SRpcMsg *pMsg) {
  int64_t  rId = msg->refId;
  int32_t  eId = msg->execId;

-  SQWMsg qwMsg = {.msg = msg->msg + msg->sqlLen, .msgLen = msg->phyLen, .connInfo = pMsg->info};
+  SQWMsg qwMsg = {.msgType = pMsg->msgType, .msg = msg->msg + msg->sqlLen, .msgLen = msg->phyLen, .connInfo = pMsg->info};

  QW_SCH_TASK_DLOG("prerocessQuery start, handle:%p", pMsg->info.handle);
-  QW_ERR_RET(qwPrerocessQuery(QW_FPARAMS(), &qwMsg));
+  QW_ERR_RET(qwPreprocessQuery(QW_FPARAMS(), &qwMsg));
  QW_SCH_TASK_DLOG("prerocessQuery end, handle:%p", pMsg->info.handle);

  return TSDB_CODE_SUCCESS;
--- a/source/libs/qworker/src/qworker.c
+++ b/source/libs/qworker/src/qworker.c
@ -469,7 +469,7 @@ int32_t qwAbortPrerocessQuery(QW_FPARAMS_DEF) {
 }


-int32_t qwPrerocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) {
+int32_t qwPreprocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) {
  int32_t        code = 0;
  bool           queryRsped = false;
  SSubplan      *plan = NULL;
@ -488,6 +488,8 @@ int32_t qwPrerocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) {

  QW_ERR_JRET(qwAddTaskStatus(QW_FPARAMS(), JOB_TASK_STATUS_INIT));

+  qwDbgResponseRedirect(qwMsg, ctx);
+
 _return:

  if (ctx) {
--- a/source/libs/scalar/src/sclfunc.c
+++ b/source/libs/scalar/src/sclfunc.c
@ -1919,6 +1919,113 @@ int32_t maxScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *
  return doMinMaxScalarFunction(pInput, inputNum, pOutput, false);
 }

+int32_t avgScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput) {
+  SColumnInfoData *pInputData  = pInput->columnData;
+  SColumnInfoData *pOutputData = pOutput->columnData;
+
+  int32_t type = GET_PARAM_TYPE(pInput);
+  int64_t count = 0;
+  bool hasNull = false;
+
+  for (int32_t i = 0; i < pInput->numOfRows; ++i) {
+    if (colDataIsNull_s(pInputData, i)) {
+      hasNull = true;
+      break;
+    }
+
+    switch(type) {
+      case TSDB_DATA_TYPE_TINYINT: {
+        int8_t *in   = (int8_t *)pInputData->pData;
+        int64_t *out = (int64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_SMALLINT: {
+        int16_t *in  = (int16_t *)pInputData->pData;
+        int64_t *out = (int64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_INT: {
+        int32_t *in  = (int32_t *)pInputData->pData;
+        int64_t *out = (int64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_BIGINT: {
+        int64_t *in  = (int64_t *)pInputData->pData;
+        int64_t *out = (int64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_UTINYINT: {
+        uint8_t *in   = (uint8_t *)pInputData->pData;
+        uint64_t *out = (uint64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_USMALLINT: {
+        uint16_t *in  = (uint16_t *)pInputData->pData;
+        uint64_t *out = (uint64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_UINT: {
+        uint32_t *in  = (uint32_t *)pInputData->pData;
+        uint64_t *out = (uint64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_UBIGINT: {
+        uint64_t *in  = (uint64_t *)pInputData->pData;
+        uint64_t *out = (uint64_t *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_FLOAT: {
+        float *in  = (float *)pInputData->pData;
+        float *out = (float *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+      case TSDB_DATA_TYPE_DOUBLE: {
+        double *in  = (double *)pInputData->pData;
+        double *out = (double *)pOutputData->pData;
+        *out += in[i];
+        count++;
+        break;
+      }
+    }
+  }
+
+  if (hasNull) {
+    colDataAppendNULL(pOutputData, 0);
+  } else {
+    if (IS_SIGNED_NUMERIC_TYPE(type)) {
+      int64_t *out   = (int64_t *)pOutputData->pData;
+      *(double *)out = *out / (double)count;
+    } else if (IS_UNSIGNED_NUMERIC_TYPE(type)) {
+      uint64_t *out  = (uint64_t *)pOutputData->pData;
+      *(double *)out = *out / (double)count;
+    } else if (IS_FLOAT_TYPE(type)) {
+      double *out    = (double *)pOutputData->pData;
+      *(double *)out = *out / (double)count;
+    }
+  }
+
+  pOutput->numOfRows = 1;
+  return TSDB_CODE_SUCCESS;
+}
+
 int32_t stddevScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput) {
  SColumnInfoData *pInputData  = pInput->columnData;
  SColumnInfoData *pOutputData = pOutput->columnData;
@ -2031,3 +2138,4 @@ int32_t stddevScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarPara
  pOutput->numOfRows = 1;
  return TSDB_CODE_SUCCESS;
 }
+
--- a/source/libs/scheduler/inc/schInt.h
+++ b/source/libs/scheduler/inc/schInt.h
@ -28,15 +28,6 @@ extern "C" {
 #include "trpc.h"
 #include "command.h"

-#define SCHEDULE_DEFAULT_MAX_JOB_NUM 1000
-#define SCHEDULE_DEFAULT_MAX_TASK_NUM 1000
-#define SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM 200  // unit is TSDB_TABLE_NUM_UNIT
-
-#define SCH_DEFAULT_TASK_TIMEOUT_USEC 10000000
-#define SCH_MAX_TASK_TIMEOUT_USEC 60000000
-
-#define SCH_MAX_CANDIDATE_EP_NUM TSDB_MAX_REPLICA
-
 enum {
  SCH_READ = 1,
  SCH_WRITE,
@ -54,6 +45,24 @@ typedef enum {
  SCH_OP_GET_STATUS,
 } SCH_OP_TYPE;

+typedef enum {
+  SCH_LOAD_SEQ = 1,
+  SCH_RANDOM,
+  SCH_ALL,
+} SCH_POLICY;
+
+#define SCHEDULE_DEFAULT_MAX_JOB_NUM 1000
+#define SCHEDULE_DEFAULT_MAX_TASK_NUM 1000
+#define SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM 200  // unit is TSDB_TABLE_NUM_UNIT
+#define SCHEDULE_DEFAULT_POLICY SCH_LOAD_SEQ
+
+#define SCH_DEFAULT_TASK_TIMEOUT_USEC 10000000
+#define SCH_MAX_TASK_TIMEOUT_USEC 60000000
+#define SCH_MAX_CANDIDATE_EP_NUM TSDB_MAX_REPLICA
+
+
+
+
 typedef struct SSchDebug {
  bool     lockEnable;
  bool     apiEnable;
@ -126,6 +135,13 @@ typedef struct SSchStatusFps {
  schStatusEventFp eventFp;
 } SSchStatusFps;

+typedef struct SSchedulerCfg {
+  uint32_t   maxJobNum;
+  int32_t    maxNodeTableNum;
+  SCH_POLICY schPolicy;
+  bool       enableReSchedule;
+} SSchedulerCfg;
+
 typedef struct SSchedulerMgmt {
  uint64_t        taskId; // sequential taksId
  uint64_t        sId;    // schedulerId
@ -184,7 +200,7 @@ typedef struct SSchLevel {

 typedef struct SSchTaskProfile {
  int64_t  startTs;
-  int64_t* execTime;
+  SArray*  execTime;
  int64_t  waitTime;
  int64_t  endTs;
 } SSchTaskProfile;
@ -192,8 +208,10 @@ typedef struct SSchTaskProfile {
 typedef struct SSchTask {
  uint64_t             taskId;          // task id
  SRWLatch             lock;            // task reentrant lock
-  int32_t              maxExecTimes;   // task may exec times
-  int32_t              execId;        // task current execute try index
+  int32_t              maxExecTimes;    // task max exec times
+  int32_t              maxRetryTimes;   // task max retry times
+  int32_t              retryTimes;      // task retry times
+  int32_t              execId;          // task current execute index
  SSchLevel           *level;           // level
  SRWLatch             planLock;        // task update plan lock
  SSubplan            *plan;            // subplan
@ -201,7 +219,7 @@ typedef struct SSchTask {
  int32_t              msgLen;          // msg length
  int8_t               status;          // task status
  int32_t              lastMsgType;     // last sent msg type
-  int64_t              timeoutUsec;    // taks timeout useconds before reschedule
+  int64_t              timeoutUsec;     // task timeout useconds before reschedule
  SQueryNodeAddr       succeedAddr;     // task executed success node address
  int8_t               candidateIdx;    // current try condidation index
  SArray              *candidateAddrs;  // condidate node addresses, element is SQueryNodeAddr
@ -265,7 +283,7 @@ typedef struct SSchJob {

 extern SSchedulerMgmt schMgmt;

-#define SCH_TASK_TIMEOUT(_task) ((taosGetTimestampUs() - (_task)->profile.execTime[(_task)->execId % (_task)->maxExecTimes]) > (_task)->timeoutUsec)
+#define SCH_TASK_TIMEOUT(_task) ((taosGetTimestampUs() - *(int64_t*)taosArrayGet((_task)->profile.execTime, (_task)->execId)) > (_task)->timeoutUsec)

 #define SCH_TASK_READY_FOR_LAUNCH(readyNum, task) ((readyNum) >= taosArrayGetSize((task)->children))

@ -299,7 +317,6 @@ extern SSchedulerMgmt schMgmt;
 #define SCH_TASK_NEED_FLOW_CTRL(_job, _task) (SCH_IS_DATA_BIND_QRY_TASK(_task) && SCH_JOB_NEED_FLOW_CTRL(_job) && SCH_IS_LEVEL_UNFINISHED((_task)->level))
 #define SCH_FETCH_TYPE(_pSrcTask) (SCH_IS_DATA_BIND_QRY_TASK(_pSrcTask) ? TDMT_SCH_FETCH : TDMT_SCH_MERGE_FETCH)
 #define SCH_TASK_NEED_FETCH(_task) ((_task)->plan->subplanType != SUBPLAN_TYPE_MODIFY)
-#define SCH_TASK_MAX_EXEC_TIMES(_levelIdx, _levelNum) (SCH_MAX_CANDIDATE_EP_NUM * ((_levelNum) - (_levelIdx)))

 #define SCH_SET_JOB_TYPE(_job, type) do { if ((type) != SUBPLAN_TYPE_MODIFY) { (_job)->attr.queryJob = true; } } while (0)
 #define SCH_IS_QUERY_JOB(_job) ((_job)->attr.queryJob) 
@ -321,8 +338,7 @@ extern SSchedulerMgmt schMgmt;
 #define SCH_LOG_TASK_START_TS(_task)                          \
  do {                                                        \
    int64_t us = taosGetTimestampUs();                        \
-    int32_t idx = (_task)->execId % (_task)->maxExecTimes; \
-    (_task)->profile.execTime[idx] = us;                    \
+    taosArrayPush((_task)->profile.execTime, &us);           \
    if (0 == (_task)->execId) {                              \
      (_task)->profile.startTs = us;                          \
    }                                                         \
@ -331,8 +347,7 @@ extern SSchedulerMgmt schMgmt;
 #define SCH_LOG_TASK_WAIT_TS(_task)                        \
  do {                                                    \
    int64_t us = taosGetTimestampUs();                    \
-    int32_t idx = (_task)->execId % (_task)->maxExecTimes; \
-    (_task)->profile.waitTime += us - (_task)->profile.execTime[idx];    \
+    (_task)->profile.waitTime += us - *(int64_t*)taosArrayGet((_task)->profile.execTime, (_task)->execId);    \
  } while (0)  


@ -340,7 +355,8 @@ extern SSchedulerMgmt schMgmt;
  do {                                                    \
    int64_t us = taosGetTimestampUs();                    \
    int32_t idx = (_task)->execId % (_task)->maxExecTimes; \
-    (_task)->profile.execTime[idx] = us - (_task)->profile.execTime[idx];    \
+    int64_t *startts = taosArrayGet((_task)->profile.execTime, (_task)->execId); \
+    *startts = us - *startts;                        \
    (_task)->profile.endTs = us;                          \
  } while (0)  

@ -471,9 +487,11 @@ void    schFreeTask(SSchJob *pJob, SSchTask *pTask);
 void    schDropTaskInHashList(SSchJob *pJob, SHashObj *list);
 int32_t schLaunchLevelTasks(SSchJob *pJob, SSchLevel *level);
 int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTask);
-int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel, int32_t levelNum);
+int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel);
 int32_t schSwitchTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask);
 void    schDirectPostJobRes(SSchedulerReq* pReq, int32_t errCode);
+int32_t schHandleJobFailure(SSchJob *pJob, int32_t errCode);
+int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode);
 bool    schChkCurrentOp(SSchJob *pJob, int32_t op, bool sync);

 extern SSchDebug gSCHDebug;
--- a/source/libs/scheduler/src/schJob.c
+++ b/source/libs/scheduler/src/schJob.c
@ -343,7 +343,7 @@ int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob) {
        SCH_ERR_JRET(TSDB_CODE_QRY_OUT_OF_MEMORY);
      }

-      SCH_ERR_JRET(schInitTask(pJob, pTask, plan, pLevel, levelNum));
+      SCH_ERR_JRET(schInitTask(pJob, pTask, plan, pLevel));

      SCH_ERR_JRET(schAppendJobDataSrc(pJob, pTask));

@ -476,7 +476,7 @@ _return:
  SCH_UNLOCK(SCH_WRITE, &pJob->opStatus.lock);
 }

-int32_t schProcessOnJobFailureImpl(SSchJob *pJob, int32_t status, int32_t errCode) {
+int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode) {
  schUpdateJobErrCode(pJob, errCode);
  
  int32_t code = atomic_load_32(&pJob->errCode);
@ -489,21 +489,29 @@ int32_t schProcessOnJobFailureImpl(SSchJob *pJob, int32_t status, int32_t errCod
  SCH_RET(TSDB_CODE_SCH_IGNORE_ERROR);
 }

-// Note: no more task error processing, handled in function internal
-int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode) {
+int32_t schHandleJobFailure(SSchJob *pJob, int32_t errCode) {
  if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) {
    return TSDB_CODE_SCH_IGNORE_ERROR;
  }

-  schProcessOnJobFailureImpl(pJob, JOB_TASK_STATUS_FAIL, errCode);
+  schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, &errCode);
  return TSDB_CODE_SCH_IGNORE_ERROR;
 }

-// Note: no more error processing, handled in function internal
 int32_t schProcessOnJobDropped(SSchJob *pJob, int32_t errCode) {
-  SCH_RET(schProcessOnJobFailureImpl(pJob, JOB_TASK_STATUS_DROP, errCode));
+  SCH_RET(schProcessOnJobFailure(pJob, errCode));
 }

+int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode) {
+  if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) {
+    return TSDB_CODE_SCH_IGNORE_ERROR;
+  }
+
+  schSwitchJobStatus(pJob, JOB_TASK_STATUS_DROP, &errCode);
+  return TSDB_CODE_SCH_IGNORE_ERROR;
+}
+
+
 int32_t schProcessOnJobPartialSuccess(SSchJob *pJob) {
  schPostJobRes(pJob, SCH_OP_EXEC);

@ -828,7 +836,7 @@ void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq* pReq, int
  }

  if (errCode) {
-    schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, (void*)&errCode);
+    schHandleJobFailure(pJob, errCode);
  }

  SCH_JOB_DLOG("job end %s operation with code %s", schGetOpStr(type), tstrerror(errCode));
@ -907,7 +915,7 @@ void schProcessOnCbEnd(SSchJob *pJob, SSchTask *pTask, int32_t errCode) {
  }

  if (errCode) {
-    schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, (void*)&errCode);
+    schHandleJobFailure(pJob, errCode);
  }
  
  if (pJob) {
--- a/source/libs/scheduler/src/schTask.c
+++ b/source/libs/scheduler/src/schTask.c
@ -42,32 +42,47 @@ void schFreeTask(SSchJob *pJob, SSchTask *pTask) {
    taosHashCleanup(pTask->execNodes);
  }

-  taosMemoryFree(pTask->profile.execTime);
+  taosArrayDestroy(pTask->profile.execTime);
 }

-int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel, int32_t levelNum) {
+void schInitTaskRetryTimes(SSchJob *pJob, SSchTask *pTask, SSchLevel *pLevel) {
+  if (SCH_IS_DATA_BIND_TASK(pTask) || (!SCH_IS_QUERY_JOB(pJob)) || (SCH_ALL != schMgmt.cfg.schPolicy)) {
+    pTask->maxRetryTimes = SCH_MAX_CANDIDATE_EP_NUM;
+  } else {
+    int32_t nodeNum = taosArrayGetSize(pJob->nodeList);
+    pTask->maxRetryTimes = TMAX(nodeNum, SCH_MAX_CANDIDATE_EP_NUM);
+  }
+  
+  pTask->maxExecTimes = pTask->maxRetryTimes * (pLevel->level + 1);
+}
+
+int32_t schInitTask(SSchJob *pJob, SSchTask *pTask, SSubplan *pPlan, SSchLevel *pLevel) {
  int32_t code = 0;

  pTask->plan = pPlan;
  pTask->level = pLevel;
  pTask->execId = -1;
-  pTask->maxExecTimes = SCH_TASK_MAX_EXEC_TIMES(pLevel->level, levelNum);
  pTask->timeoutUsec = SCH_DEFAULT_TASK_TIMEOUT_USEC;
  pTask->taskId = schGenTaskId();
  pTask->execNodes =
      taosHashInit(SCH_MAX_CANDIDATE_EP_NUM, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), true, HASH_NO_LOCK);
-  pTask->profile.execTime = taosMemoryCalloc(pTask->maxExecTimes, sizeof(int64_t));
+
+  schInitTaskRetryTimes(pJob, pTask, pLevel);
+
+  pTask->profile.execTime = taosArrayInit(pTask->maxExecTimes, sizeof(int64_t));
  if (NULL == pTask->execNodes || NULL == pTask->profile.execTime) {
    SCH_ERR_JRET(TSDB_CODE_QRY_OUT_OF_MEMORY);
  }

  SCH_SET_TASK_STATUS(pTask, JOB_TASK_STATUS_INIT);

+  SCH_TASK_DLOG("task initialized, max times %d:%d", pTask->maxRetryTimes, pTask->maxExecTimes);
+
  return TSDB_CODE_SUCCESS;

 _return:

-  taosMemoryFreeClear(pTask->profile.execTime);
+  taosArrayDestroy(pTask->profile.execTime);
  taosHashCleanup(pTask->execNodes);

  SCH_RET(code);
@ -105,7 +120,7 @@ int32_t schDropTaskExecNode(SSchJob *pJob, SSchTask *pTask, void *handle, int32_
  }

  if (taosHashRemove(pTask->execNodes, &execId, sizeof(execId))) {
-    SCH_TASK_ELOG("fail to remove execId %d from execNodeList", execId);
+    SCH_TASK_DLOG("execId %d already not in execNodeList", execId);
  } else {
    SCH_TASK_DLOG("execId %d removed from execNodeList", execId);
  }
@ -235,7 +250,7 @@ int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask) {
      }

      if (pTask->level->taskFailed > 0) {
-        SCH_RET(schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, NULL));
+        SCH_RET(schHandleJobFailure(pJob, pJob->errCode));
      } else {
        SCH_RET(schSwitchJobStatus(pJob, JOB_TASK_STATUS_PART_SUCC, NULL));
      }
@ -285,6 +300,10 @@ int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask) {
 }

 int32_t schRescheduleTask(SSchJob *pJob, SSchTask *pTask) {
+  if (!schMgmt.cfg.enableReSchedule) {
+    return TSDB_CODE_SUCCESS;
+  }
+  
  if (SCH_IS_DATA_BIND_TASK(pTask)) {
    return TSDB_CODE_SUCCESS;
  }
@ -304,13 +323,17 @@ int32_t schRescheduleTask(SSchJob *pJob, SSchTask *pTask) {
 int32_t schDoTaskRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf *pData, int32_t rspCode) {
  int32_t code = 0;

-  if ((pTask->execId + 1) >= pTask->maxExecTimes) {
-    SCH_TASK_DLOG("task no more retry since reach max try times, execId:%d", pTask->execId);
-    schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, (void *)&rspCode);
-    return TSDB_CODE_SUCCESS;
+  SCH_TASK_DLOG("task will be redirected now, status:%s", SCH_GET_TASK_STATUS_STR(pTask));
+
+  if (NULL == pData) {
+    pTask->retryTimes = 0;
  }

-  SCH_TASK_DLOG("task will be redirected now, status:%s", SCH_GET_TASK_STATUS_STR(pTask));
+  if (((pTask->execId + 1) >= pTask->maxExecTimes) || ((pTask->retryTimes + 1) > pTask->maxRetryTimes)) {
+    SCH_TASK_DLOG("task no more retry since reach max times %d:%d, execId %d", pTask->maxRetryTimes, pTask->maxExecTimes, pTask->execId);
+    schHandleJobFailure(pJob, rspCode);
+    return TSDB_CODE_SUCCESS;
+  }

  schDropTaskOnExecNode(pJob, pTask);
  taosHashClear(pTask->execNodes);
@ -493,9 +516,15 @@ int32_t schTaskCheckSetRetry(SSchJob *pJob, SSchTask *pTask, int32_t errCode, bo
    }
  }

+  if ((pTask->retryTimes + 1) > pTask->maxRetryTimes) {
+    *needRetry = false;
+    SCH_TASK_DLOG("task no more retry since reach max retry times, retryTimes:%d/%d", pTask->retryTimes, pTask->maxRetryTimes);
+    return TSDB_CODE_SUCCESS;
+  }
+
  if ((pTask->execId + 1) >= pTask->maxExecTimes) {
    *needRetry = false;
-    SCH_TASK_DLOG("task no more retry since reach max try times, execId:%d", pTask->execId);
+    SCH_TASK_DLOG("task no more retry since reach max exec times, execId:%d/%d", pTask->execId, pTask->maxExecTimes);
    return TSDB_CODE_SUCCESS;
  }

@ -649,10 +678,31 @@ int32_t schUpdateTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask, SEpSet *pEpSe

 int32_t schSwitchTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask) {
  int32_t candidateNum = taosArrayGetSize(pTask->candidateAddrs);
+  if (candidateNum <= 1) {
+    goto _return;
+  }
+  
+  switch (schMgmt.cfg.schPolicy) {
+    case SCH_LOAD_SEQ:
+    case SCH_ALL: 
+    default:
      if (++pTask->candidateIdx >= candidateNum) {
        pTask->candidateIdx = 0;
      }
-  SCH_TASK_DLOG("switch task candiateIdx to %d", pTask->candidateIdx);
+      break;
+    case SCH_RANDOM: {
+      int32_t lastIdx = pTask->candidateIdx;
+      while (lastIdx == pTask->candidateIdx) {
+        pTask->candidateIdx = taosRand() % candidateNum;
+      }
+      break;
+    }
+  }
+
+_return:
+
+  SCH_TASK_DLOG("switch task candiateIdx to %d/%d", pTask->candidateIdx, candidateNum);
+  
  return TSDB_CODE_SUCCESS;
 }

@ -739,8 +789,9 @@ int32_t schLaunchTaskImpl(SSchJob *pJob, SSchTask *pTask) {

  atomic_add_fetch_32(&pTask->level->taskLaunchedNum, 1);
  pTask->execId++;
+  pTask->retryTimes++;

-  SCH_TASK_DLOG("start to launch task's %dth exec", pTask->execId);
+  SCH_TASK_DLOG("start to launch task, execId %d, retry %d", pTask->execId, pTask->retryTimes);

  SCH_LOG_TASK_START_TS(pTask);

--- a/source/libs/scheduler/src/scheduler.c
+++ b/source/libs/scheduler/src/scheduler.c
@ -22,25 +22,18 @@ SSchedulerMgmt schMgmt = {
    .jobRef = -1,
 };

-int32_t schedulerInit(SSchedulerCfg *cfg) {
+int32_t schedulerInit() {
  if (schMgmt.jobRef >= 0) {
    qError("scheduler already initialized");
    return TSDB_CODE_QRY_INVALID_INPUT;
  }

-  if (cfg) {
-    schMgmt.cfg = *cfg;
+  schMgmt.cfg.maxJobNum = SCHEDULE_DEFAULT_MAX_JOB_NUM;
+  schMgmt.cfg.maxNodeTableNum = SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM;
+  schMgmt.cfg.schPolicy = SCHEDULE_DEFAULT_POLICY;
+  schMgmt.cfg.enableReSchedule = true;

-    if (schMgmt.cfg.maxJobNum == 0) {
-      schMgmt.cfg.maxJobNum = SCHEDULE_DEFAULT_MAX_JOB_NUM;
-    }
-    if (schMgmt.cfg.maxNodeTableNum <= 0) {
-      schMgmt.cfg.maxNodeTableNum = SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM;
-    }
-  } else {
-    schMgmt.cfg.maxJobNum = SCHEDULE_DEFAULT_MAX_JOB_NUM;
-    schMgmt.cfg.maxNodeTableNum = SCHEDULE_DEFAULT_MAX_NODE_TABLE_NUM;
-  }
+  qDebug("schedule policy init to %d", schMgmt.cfg.schPolicy);
  
  schMgmt.jobRef = taosOpenRef(schMgmt.cfg.maxJobNum, schFreeJobImpl);
  if (schMgmt.jobRef < 0) {
@ -130,6 +123,26 @@ void schedulerStopQueryHb(void *pTrans) {
  schCleanClusterHb(pTrans);
 }

+int32_t schedulerUpdatePolicy(int32_t policy) {
+  switch (policy) {
+    case SCH_LOAD_SEQ:
+    case SCH_RANDOM:
+    case SCH_ALL:
+      schMgmt.cfg.schPolicy = policy;
+      qDebug("schedule policy updated to %d", schMgmt.cfg.schPolicy);
+      break;
+    default:
+      return TSDB_CODE_TSC_INVALID_INPUT;
+  }
+
+  return TSDB_CODE_SUCCESS;
+}
+
+int32_t schedulerEnableReSchedule(bool enableResche) {
+  schMgmt.cfg.enableReSchedule = enableResche;
+  return TSDB_CODE_SUCCESS;
+}
+
 void schedulerFreeJob(int64_t* jobId, int32_t errCode) {
  if (0 == *jobId) {
    return;
@ -141,7 +154,7 @@ void schedulerFreeJob(int64_t* jobId, int32_t errCode) {
    return;
  }

-  schSwitchJobStatus(pJob, JOB_TASK_STATUS_DROP, (void*)&errCode);
+  schHandleJobDrop(pJob, errCode);
  
  schReleaseJob(*jobId);
  *jobId = 0;
--- a/source/libs/scheduler/test/schedulerTests.cpp
+++ b/source/libs/scheduler/test/schedulerTests.cpp
@ -477,7 +477,7 @@ void* schtRunJobThread(void *aa) {
  schtInitLogFile();

  
-  int32_t code = schedulerInit(NULL);
+  int32_t code = schedulerInit();
  assert(code == 0);


@ -649,7 +649,7 @@ TEST(queryTest, normalCase) {
  qnodeAddr.port = 6031;
  taosArrayPush(qnodeList, &qnodeAddr);
  
-  int32_t code = schedulerInit(NULL);
+  int32_t code = schedulerInit();
  ASSERT_EQ(code, 0);

  schtBuildQueryDag(&dag);
@ -756,7 +756,7 @@ TEST(queryTest, readyFirstCase) {
  qnodeAddr.port = 6031;
  taosArrayPush(qnodeList, &qnodeAddr);
  
-  int32_t code = schedulerInit(NULL);
+  int32_t code = schedulerInit();
  ASSERT_EQ(code, 0);

  schtBuildQueryDag(&dag);
@ -866,7 +866,7 @@ TEST(queryTest, flowCtrlCase) {
  qnodeAddr.port = 6031;
  taosArrayPush(qnodeList, &qnodeAddr);
  
-  int32_t code = schedulerInit(NULL);
+  int32_t code = schedulerInit();
  ASSERT_EQ(code, 0);

  schtBuildQueryFlowCtrlDag(&dag);
@ -975,7 +975,7 @@ TEST(insertTest, normalCase) {
  qnodeAddr.port = 6031;
  taosArrayPush(qnodeList, &qnodeAddr);
  
-  int32_t code = schedulerInit(NULL);
+  int32_t code = schedulerInit();
  ASSERT_EQ(code, 0);

  schtBuildInsertDag(&dag);
--- a/source/libs/sync/src/syncMain.c
+++ b/source/libs/sync/src/syncMain.c
@ -1853,8 +1853,8 @@ void syncNodeDoConfigChange(SSyncNode* pSyncNode, SSyncCfg* pNewConfig, SyncInde
      syncNodeBecomeLeader(pSyncNode, tmpbuf);

      // Raft 3.6.2 Committing entries from previous terms
-      syncNodeReplicate(pSyncNode);
      syncNodeAppendNoop(pSyncNode);
+      syncNodeReplicate(pSyncNode);
      syncMaybeAdvanceCommitIndex(pSyncNode);

    } else {
@ -2029,8 +2029,8 @@ void syncNodeCandidate2Leader(SSyncNode* pSyncNode) {
  syncNodeLog2("==state change syncNodeCandidate2Leader==", pSyncNode);

  // Raft 3.6.2 Committing entries from previous terms
-  syncNodeReplicate(pSyncNode);
  syncNodeAppendNoop(pSyncNode);
+  syncNodeReplicate(pSyncNode);
  syncMaybeAdvanceCommitIndex(pSyncNode);
 }

--- a/source/libs/transport/src/transCli.c
+++ b/source/libs/transport/src/transCli.c
@ -771,6 +771,7 @@ static void cliHandleRelease(SCliMsg* pMsg, SCliThrd* pThrd) {
  SExHandle* exh = transAcquireExHandle(transGetRefMgt(), refId);
  if (exh == NULL) {
    tDebug("%" PRId64 " already release", refId);
+    destroyCmsg(pMsg);
    return;
  }

--- a/source/util/src/terror.c
+++ b/source/util/src/terror.c
@ -135,7 +135,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TSC_STMT_API_ERROR,           "Stmt API usage error"
 TAOS_DEFINE_ERROR(TSDB_CODE_TSC_STMT_TBNAME_ERROR,        "Stmt table name not set")
 TAOS_DEFINE_ERROR(TSDB_CODE_TSC_STMT_CLAUSE_ERROR,        "not supported stmt clause")
 TAOS_DEFINE_ERROR(TSDB_CODE_TSC_QUERY_KILLED,             "Query killed")
-TAOS_DEFINE_ERROR(TSDB_CODE_TSC_NO_EXEC_NODE,             "No available execution node")
+TAOS_DEFINE_ERROR(TSDB_CODE_TSC_NO_EXEC_NODE,             "No available execution node in current query policy configuration")
 TAOS_DEFINE_ERROR(TSDB_CODE_TSC_NOT_STABLE_ERROR,         "Table is not a super table")

 // mnode-common
--- a/tests/script/api/batchprepare.c
+++ b/tests/script/api/batchprepare.c
@ -2685,6 +2685,8 @@ int main(int argc, char *argv[])

  runAll(taos);

+  taos_close(taos);
+
  return 0;
 }

--- a/tests/script/jenkins/basic.txt
+++ b/tests/script/jenkins/basic.txt
@ -82,12 +82,12 @@
 ./test.sh -f tsim/insert/update0.sim

 # ---- parser
-# ./test.sh -f tsim/parser/alter.sim
-# ./test.sh -f tsim/parser/alter1.sim
-## ./test.sh -f tsim/parser/alter__for_community_version.sim
-## ./test.sh -f tsim/parser/alter_column.sim
-# ./test.sh -f tsim/parser/alter_stable.sim
-# ./test.sh -f tsim/parser/auto_create_tb.sim
+./test.sh -f tsim/parser/alter.sim
+# nojira ./test.sh -f tsim/parser/alter1.sim
+./test.sh -f tsim/parser/alter__for_community_version.sim
+./test.sh -f tsim/parser/alter_column.sim
+./test.sh -f tsim/parser/alter_stable.sim
+# jira ./test.sh -f tsim/parser/auto_create_tb.sim
 # ./test.sh -f tsim/parser/auto_create_tb_drop_tb.sim
 # ./test.sh -f tsim/parser/between_and.sim
 # ./test.sh -f tsim/parser/binary_escapeCharacter.sim
@ -205,7 +205,7 @@
 ./test.sh -f tsim/qnode/basic1.sim

 # ---- snode
-# ./test.sh -f tsim/snode/basic1.sim
+# unsupport ./test.sh -f tsim/snode/basic1.sim

 # ---- bnode
 ./test.sh -f tsim/bnode/basic1.sim
@ -235,9 +235,9 @@
 ./test.sh -f tsim/table/createmulti.sim
 ./test.sh -f tsim/table/date.sim
 ./test.sh -f tsim/table/db.table.sim
-# ./test.sh -f tsim/table/delete_reuse1.sim
-# ./test.sh -f tsim/table/delete_reuse2.sim
-# ./test.sh -f tsim/table/delete_writing.sim
+./test.sh -f tsim/table/delete_reuse1.sim
+./test.sh -f tsim/table/delete_reuse2.sim
+./test.sh -f tsim/table/delete_writing.sim
 ./test.sh -f tsim/table/describe.sim
 ./test.sh -f tsim/table/double.sim
 ./test.sh -f tsim/table/float.sim
@ -314,12 +314,12 @@
 ./test.sh -f tsim/db/basic3.sim -m
 ./test.sh -f tsim/db/error1.sim -m
 ./test.sh -f tsim/insert/backquote.sim -m
-# ./test.sh -f tsim/parser/fourArithmetic-basic.sim -m
+# nojira ./test.sh -f tsim/parser/fourArithmetic-basic.sim -m
 ./test.sh -f tsim/query/interval-offset.sim -m
 ./test.sh -f tsim/tmq/basic3.sim -m
 ./test.sh -f tsim/stable/vnode3.sim -m
 ./test.sh -f tsim/qnode/basic1.sim -m
-#./test.sh -f tsim/mnode/basic1.sim -m
+# nojira ./test.sh -f tsim/mnode/basic1.sim -m

 # --- sma
 ./test.sh -f tsim/sma/drop_sma.sim
@ -333,13 +333,13 @@
 ./test.sh -f tsim/valgrind/checkError3.sim

 # --- vnode
-# ./test.sh -f tsim/vnode/replica3_basic.sim
-# ./test.sh -f tsim/vnode/replica3_repeat.sim
-# ./test.sh -f tsim/vnode/replica3_vgroup.sim
-# ./test.sh -f tsim/vnode/replica3_many.sim
-# ./test.sh -f tsim/vnode/replica3_import.sim
-# ./test.sh -f tsim/vnode/stable_balance_replica1.sim
-# ./test.sh -f tsim/vnode/stable_dnode2_stop.sim
+# unsupport ./test.sh -f tsim/vnode/replica3_basic.sim
+# unsupport ./test.sh -f tsim/vnode/replica3_repeat.sim
+# unsupport ./test.sh -f tsim/vnode/replica3_vgroup.sim
+# unsupport ./test.sh -f tsim/vnode/replica3_many.sim
+# unsupport ./test.sh -f tsim/vnode/replica3_import.sim
+# unsupport ./test.sh -f tsim/vnode/stable_balance_replica1.sim
+# unsupport ./test.sh -f tsim/vnode/stable_dnode2_stop.sim
 ./test.sh -f tsim/vnode/stable_dnode2.sim
 ./test.sh -f tsim/vnode/stable_dnode3.sim
 ./test.sh -f tsim/vnode/stable_replica3_dnode6.sim
@ -350,7 +350,6 @@
 ./test.sh -f tsim/sync/3Replica5VgElect.sim
 ./test.sh -f tsim/sync/oneReplica1VgElect.sim
 ./test.sh -f tsim/sync/oneReplica5VgElect.sim
-# ./test.sh -f tsim/sync/3Replica5VgElect3mnode.sim

 # --- catalog
 ./test.sh -f tsim/catalog/alterInCurrent.sim
@ -382,7 +381,7 @@

 # ---- compute
 ./test.sh -f tsim/compute/avg.sim
-# jira ./test.sh -f tsim/compute/block_dist.sim
+./test.sh -f tsim/compute/block_dist.sim
 ./test.sh -f tsim/compute/bottom.sim
 ./test.sh -f tsim/compute/count.sim
 ./test.sh -f tsim/compute/diff.sim
@ -433,12 +432,6 @@
 # ---- wal
 ./test.sh -f tsim/wal/kill.sim

-# ---- issue
-#./test.sh -f tsim/issue/TD-2677.sim
-#./test.sh -f tsim/issue/TD-2680.sim
-#./test.sh -f tsim/issue/TD-2713.sim
-#./test.sh -f tsim/issue/TD-3300.sim
-
 # ---- tag
 ./test.sh -f tsim/tag/3.sim
 ./test.sh -f tsim/tag/4.sim
--- a/tests/script/tsim/compute/block_dist.sim
+++ b/tests/script/tsim/compute/block_dist.sim
@ -1,5 +1,6 @@
 system sh/stop_dnodes.sh
 system sh/deploy.sh -n dnode1 -i 1
+system sh/cfg.sh -n dnode1 -c debugflag -v 131
 system sh/exec.sh -n dnode1 -s start
 sql connect

@ -80,11 +81,11 @@ $nt = $ntPrefix . $i

 #sql select _block_dist() from $nt
 print show table distributed $nt
-sql show table distributed $nt
+sql_error show table distributed $nt

-if $rows == 0 then
-  return -1
-endi
+#if $rows == 0 then
+#  return -1
+#endi

 print ============== TD-5998
 sql_error select _block_dist() from (select * from $nt)
--- a/tests/script/tsim/issue/TD-2677.sim
+++ b/tests/script/tsim/issue/TD-2677.sim
@ -1,111 +0,0 @@
-system sh/stop_dnodes.sh
-
-system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 3
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 3
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 3
-
-system sh/cfg.sh -n dnode1 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode2 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode3 -c mnodeEqualVnodeNum -v 4
-
-print ============== deploy
-
-system sh/exec.sh -n dnode1 -s start 
-sql connect
-
-sql create dnode $hostname2
-sql create dnode $hostname3
-system sh/exec.sh -n dnode2 -s start 
-system sh/exec.sh -n dnode3 -s start 
-
-print  =============== step1
-$x = 0
-step1: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step1
-endi
-if $data4_2 != ready then
-  goto step1
-endi
-if $data4_3 != ready then
-  goto step1
-endi
-
-sql show mnodes
-$mnode1Role = $data2_1
-print mnode1Role $mnode1Role
-$mnode2Role = $data2_2
-print mnode2Role $mnode2Role
-$mnode3Role = $data2_3
-print mnode3Role $mnode3Role
-
-if $mnode1Role != master then
-  goto step1
-endi
-if $mnode2Role != slave then
-  goto step1
-endi
-if $mnode3Role != slave then
-  goto step1
-endi
-
-$x = 1
-show2:
-
-print  =============== step1
-sql create database d1 replica 2 quorum 2
-sql create table d1.t1 (ts timestamp, i int)
-sql_error create table d1.t1 (ts timestamp, i int)
-sql insert into d1.t1 values(now, 1)
-sql select * from d1.t1;
-if $rows != 1 then
-  return -1
-endi
-
-print  =============== step2
-sql create database d2 replica 3 quorum 2
-sql create table d2.t1 (ts timestamp, i int)
-sql_error create table d2.t1 (ts timestamp, i int)
-sql insert into d2.t1 values(now, 1)
-sql select * from d2.t1;
-if $rows != 1 then
-  return -1
-endi
-
-print  =============== step3
-sql       create database d4 replica 1 quorum 1
-sql_error create database d5 replica 1 quorum 2
-sql_error create database d6 replica 1 quorum 3
-sql_error create database d7 replica 1 quorum 4
-sql_error create database d8 replica 1 quorum 0
-sql       create database d9 replica 2 quorum 1
-sql       create database d10 replica 2 quorum 2
-sql_error create database d11 replica 2 quorum 3
-sql_error create database d12 replica 2 quorum 4
-sql_error create database d12 replica 2 quorum 0
-sql       create database d13 replica 3 quorum 1
-sql       create database d14 replica 3 quorum 2
-sql_error create database d15 replica 3 quorum 3
-sql_error create database d16 replica 3 quorum 4
-sql_error create database d17 replica 3 quorum 0
-
-
-system sh/exec.sh -n dnode1 -s stop  -x SIGINT
-system sh/exec.sh -n dnode2 -s stop  -x SIGINT
-system sh/exec.sh -n dnode3 -s stop  -x SIGINT
-system sh/exec.sh -n dnode4 -s stop  -x SIGINT
--- a/tests/script/tsim/issue/TD-2680.sim
+++ b/tests/script/tsim/issue/TD-2680.sim
@ -1,202 +0,0 @@
-system sh/stop_dnodes.sh
-system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-system sh/deploy.sh -n dnode4 -i 4
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode4 -c numOfMnodes -v 1
-
-system sh/cfg.sh -n dnode1 -c walLevel -v 2
-system sh/cfg.sh -n dnode2 -c walLevel -v 2
-system sh/cfg.sh -n dnode3 -c walLevel -v 2
-system sh/cfg.sh -n dnode4 -c walLevel -v 2
-
-system sh/cfg.sh -n dnode1 -c balanceInterval -v 10
-system sh/cfg.sh -n dnode2 -c balanceInterval -v 10
-system sh/cfg.sh -n dnode3 -c balanceInterval -v 10
-system sh/cfg.sh -n dnode4 -c balanceInterval -v 10
-
-system sh/cfg.sh -n dnode1 -c role -v 1
-system sh/cfg.sh -n dnode2 -c role -v 2
-system sh/cfg.sh -n dnode3 -c role -v 2
-system sh/cfg.sh -n dnode4 -c role -v 2
-
-system sh/cfg.sh -n dnode1 -c arbitrator -v $arbitrator
-system sh/cfg.sh -n dnode2 -c arbitrator -v $arbitrator
-system sh/cfg.sh -n dnode3 -c arbitrator -v $arbitrator
-system sh/cfg.sh -n dnode4 -c arbitrator -v $arbitrator
-
-print ============== step0
-system sh/exec_tarbitrator.sh -s start
-
-print ============== step1
-system sh/exec.sh -n dnode1 -s start
-sql connect
-sql create dnode $hostname2
-sql create dnode $hostname3
-system sh/exec.sh -n dnode2 -s start
-system sh/exec.sh -n dnode3 -s start
-
-$x = 0
-step1: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step1
-endi
-if $data4_2 != ready then
-  goto step1
-endi
-if $data4_3 != ready then
-  goto step1
-endi
-
-sql show mnodes
-print mnode1 $data2_1
-print mnode1 $data2_2
-print mnode1 $data2_3
-if $data2_1 != master then
-  goto step1
-endi
-
-print ============== step2
-sql show dnodes
-if $rows != 4 then
-  return -1
-endi
-
-print $data00 $data01 $data02 $data03 $data04 $data05 $data06 $data07
-print $data10 $data11 $data12 $data13 $data14 $data15 $data16 $data17
-print $data20 $data21 $data22 $data23 $data24 $data25 $data26 $data27
-print $data30 $data31 $data32 $data33 $data34 $data35 $data36 $data37
-
-if $data30 != 0 then
-  return -1
-endi
-
-if $data32 != 0 then
-  return -1
-endi
-
-if $data33 != 0 then
-  return -1
-endi
-
-if $data34 != ready then
-  return -1
-endi
-
-if $data35 != arb then
-  return -1
-endi
-
-if $data37 != - then
-  return -1
-endi
-
-print ============== step4
-system sh/exec_tarbitrator.sh -s stop
-
-$x = 0
-step4: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 20 then
-		return -1
-	endi
-
-sql show dnodes
-if $rows != 4 then
-  return -1
-endi
-
-print $data00 $data01 $data02 $data03 $data04 $data05 $data06 $data07
-print $data10 $data11 $data12 $data13 $data14 $data15 $data16 $data17
-print $data20 $data21 $data22 $data23 $data24 $data25 $data26 $data27
-print $data30 $data31 $data32 $data33 $data34 $data35 $data36 $data37
-
-if $data30 != 0 then
-  return -1
-endi
-
-if $data32 != 0 then
-  return -1
-endi
-
-if $data33 != 0 then
-  return -1
-endi
-
-if $data34 != offline then
-  goto step4
-endi
-
-if $data35 != arb then
-  return -1
-endi
-
-if $data37 != - then
-  return -1
-endi
-
-print ============== step5
-system sh/exec_tarbitrator.sh -s start
-
-$x = 0
-step5: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 20 then
-		return -1
-	endi
-
-sql show dnodes
-if $rows != 4 then
-  return -1
-endi
-
-print $data00 $data01 $data02 $data03 $data04 $data05 $data06 $data07
-print $data10 $data11 $data12 $data13 $data14 $data15 $data16 $data17
-print $data20 $data21 $data22 $data23 $data24 $data25 $data26 $data27
-print $data30 $data31 $data32 $data33 $data34 $data35 $data36 $data37
-
-if $data30 != 0 then
-  return -1
-endi
-
-if $data32 != 0 then
-  return -1
-endi
-
-if $data33 != 0 then
-  return -1
-endi
-
-if $data34 != ready then
-  goto step5
-endi
-
-if $data35 != arb then
-  return -1
-endi
-
-if $data37 != - then
-  return -1
-endi
-
-system sh/exec.sh -n dnode1 -s stop  -x SIGINT
-system sh/exec.sh -n dnode2 -s stop  -x SIGINT
-system sh/exec.sh -n dnode3 -s stop  -x SIGINT
-system sh/exec.sh -n dnode4 -s stop  -x SIGINT
--- a/tests/script/tsim/issue/TD-2713.sim
+++ b/tests/script/tsim/issue/TD-2713.sim
@ -1,145 +0,0 @@
-system sh/stop_dnodes.sh
-
-system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-system sh/deploy.sh -n dnode4 -i 4
-
-system sh/cfg.sh -n dnode1 -c wallevel -v 2
-system sh/cfg.sh -n dnode2 -c wallevel -v 2
-system sh/cfg.sh -n dnode3 -c wallevel -v 2
-system sh/cfg.sh -n dnode4 -c wallevel -v 2
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 3
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 3
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 3
-system sh/cfg.sh -n dnode4 -c numOfMnodes -v 3
-
-system sh/cfg.sh -n dnode1 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode2 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode3 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode4 -c mnodeEqualVnodeNum -v 4
-
-system sh/cfg.sh -n dnode1 -c slaveQuery -v 1
-system sh/cfg.sh -n dnode2 -c slaveQuery -v 1
-system sh/cfg.sh -n dnode3 -c slaveQuery -v 1
-system sh/cfg.sh -n dnode4 -c slaveQuery -v 1
-
-print ========= step1 
-system sh/exec.sh -n dnode1 -s start
-sql connect
-sql create dnode $hostname2
-sql create dnode $hostname3
-system sh/exec.sh -n dnode2 -s start
-system sh/exec.sh -n dnode3 -s start
-
-$x = 0
-step1: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step1
-endi
-if $data4_2 != ready then
-  goto step1
-endi
-if $data4_3 != ready then
-  goto step1
-endi
-
-sql show mnodes
-print mnode1 $data2_1
-print mnode1 $data2_2
-print mnode1 $data2_3
-if $data2_1 != master then
-  goto step1
-endi
-if $data2_2 != slave then
-  goto step1
-endi
-if $data2_3 != slave then
-  goto step1
-endi
-
-print ========= step2
-sql create database d1 replica 3
-sql create table d1.t1 (ts timestamp, i int)
-sql insert into d1.t1 values(now, 1)
-
-$x = 0
-step2: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show d1.vgroups
-print online vgroups: $data03
-if $data03 != 3 then
-  goto step2
-endi
-sleep 1000
-
-print ========= step3
-$i = 0
-while $i < 100
-	$i = $i + 1
-	sql select * from d1.t1
-	print d1.t1 rows: $rows
-	if $rows != 1 then
-		return -1
-	endi	
-endw 
-
-print ========= step4
-system sh/exec.sh -n dnode1 -s stop  -x SIGINT
-system sh/exec.sh -n dnode2 -s stop  -x SIGINT
-system sh/exec.sh -n dnode3 -s stop  -x SIGINT
-
-system rm -rf ../../../sim/dnode3/data/vnode/vnode2/tsdb/data/*
-system rm -rf ../../../sim/dnode3/data/vnode/vnode2/version.json
-
-system sh/exec.sh -n dnode1 -s start  -x SIGINT
-system sh/exec.sh -n dnode2 -s start  -x SIGINT
-system sh/exec.sh -n dnode3 -s start  -x SIGINT
-
-$x = 0
-step4: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 30 then
-		return -1
-	endi
-
-sql show d1.vgroups
-print online vgroups: $data03
-if $data03 != 3 then
-  goto step4
-endi
-sleep 1000
-
-print ========= step5
-$i = 0
-while $i < 100
-	$i = $i + 1
-	sql select * from d1.t1
-	if $rows != 1 then
-		return -1
-	endi
-	print d1.t1 rows: $rows
-endw 
-
-system sh/exec.sh -n dnode1 -s stop  -x SIGINT
-system sh/exec.sh -n dnode2 -s stop  -x SIGINT
-system sh/exec.sh -n dnode3 -s stop  -x SIGINT
-system sh/exec.sh -n dnode4 -s stop  -x SIGINT
--- a/tests/script/tsim/issue/TD-3300.sim
+++ b/tests/script/tsim/issue/TD-3300.sim
@ -1,556 +0,0 @@
-system sh/stop_dnodes.sh
-system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-system sh/deploy.sh -n dnode4 -i 4
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode4 -c numOfMnodes -v 1
-
-system sh/cfg.sh -n dnode1 -c role -v 1
-system sh/cfg.sh -n dnode2 -c role -v 2
-system sh/cfg.sh -n dnode3 -c role -v 2
-system sh/cfg.sh -n dnode4 -c role -v 2
-
-system sh/cfg.sh -n dnode1 -c arbitrator -v $arbitrator
-system sh/cfg.sh -n dnode2 -c arbitrator -v $arbitrator
-system sh/cfg.sh -n dnode3 -c arbitrator -v $arbitrator
-system sh/cfg.sh -n dnode4 -c arbitrator -v $arbitrator
-
-print ============== step0: start tarbitrator
-system sh/exec_tarbitrator.sh -s start
-
-print ============== step1: start dnode1, only deploy mnode
-system sh/exec.sh -n dnode1 -s start
-sql connect
-
-print ============== step2: start dnode2/dnode3
-system sh/exec.sh -n dnode2 -s start 
-system sh/exec.sh -n dnode3 -s start
-sql create dnode $hostname2
-sql create dnode $hostname3
-
-$x = 0
-step2: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step2
-endi
-if $data4_2 != ready then
-  goto step2
-endi
-if $data4_3 != ready then
-  goto step2
-endi
-
-sleep 1000
-
-print ============== step3
-sql create database db replica 2
-sql use db
-
-sql create table stb (ts timestamp, c1 int, c2 int) tags(t1 int)
-sql create table t1 using stb tags(1)
-sql insert into t1 values(1577980800000, 1, 5)
-sql insert into t1 values(1577980800001, 2, 4)
-sql insert into t1 values(1577980800002, 3, 3)
-sql insert into t1 values(1577980800003, 4, 2)
-sql insert into t1 values(1577980800004, 5, 1)
-
-sql show db.vgroups
-if $data04 != 3 then
-	return -1
-endi
-if $data06 != 2 then
-	return -1
-endi
-if $data05 != master then
-	return -1
-endi
-if $data07 != slave then
-	return -1
-endi
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-
-system sh/exec.sh -n dnode2 -s stop -x SIGKILL
-system sh/exec.sh -n dnode3 -s stop -x SIGKILL
-
-print ============== step4
-system sh/exec.sh -n dnode2 -s start
-system sh/exec.sh -n dnode3 -s start
-
-$x = 0
-step4: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step4
-endi
-if $data4_2 != ready then
-  goto step4
-endi
-if $data4_3 != ready then
-  goto step4
-endi
-
-sql show db.vgroups
-if $data04 != 3 then
-	goto step4
-endi
-if $data06 != 2 then
-	goto step4
-endi
-if $data05 != master then
-	goto step4
-endi
-if $data07 != slave then
-	goto step4
-endi
-
-sql create table t2 using stb tags(1)
-sql insert into t2 values(1577980800000, 1, 5)
-sql insert into t2 values(1577980800001, 2, 4)
-sql insert into t2 values(1577980800002, 3, 3)
-sql insert into t2 values(1577980800003, 4, 2)
-sql insert into t2 values(1577980800004, 5, 1)
-
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-
-print ============== step5
-system sh/exec.sh -n dnode3 -s stop  -x SIGKILL
-
-$x = 0
-step5: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step5
-endi
-if $data4_2 != ready then
-  goto step5
-endi
-if $data4_3 != offline then
-  goto step5
-endi
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-
-sql show db.vgroups
-if $data04 != 3 then
-	goto step5
-endi
-if $data06 != 2 then
-	goto step5
-endi
-if $data05 != offline then
-	goto step5
-endi
-if $data07 != master then
-	goto step5
-endi
-
-print ============== step6
-sql create table t3 using stb tags(1)
-sql insert into t3 values(1577980800000, 1, 5)
-sql insert into t3 values(1577980800001, 2, 4)
-sql insert into t3 values(1577980800002, 3, 3)
-sql insert into t3 values(1577980800003, 4, 2)
-sql insert into t3 values(1577980800004, 5, 1)
-sql insert into t3 values(1577980800010, 11, 5)
-sql insert into t3 values(1577980800011, 12, 4)
-sql insert into t3 values(1577980800012, 13, 3)
-sql insert into t3 values(1577980800013, 14, 2)
-sql insert into t3 values(1577980800014, 15, 1)
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-
-system sh/exec.sh -n dnode3 -s start
-
-$x = 0
-step6: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step6
-endi
-if $data4_2 != ready then
-  goto step6
-endi
-if $data4_3 != ready then
-  goto step6
-endi
-
-sql show db.vgroups
-if $data04 != 3 then
-	goto step6
-endi
-if $data06 != 2 then
-	goto step6
-endi
-if $data05 != slave then
-	goto step6
-endi
-if $data07 != master then
-	goto step6
-endi
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-
-print ============== step7
-sql create table t4 using stb tags(1)
-sql insert into t4 values(1577980800000, 1, 5)
-sql insert into t4 values(1577980800001, 2, 4)
-sql insert into t4 values(1577980800002, 3, 3)
-sql insert into t4 values(1577980800003, 4, 2)
-sql insert into t4 values(1577980800004, 5, 1)
-sql insert into t4 values(1577980800010, 11, 5)
-sql insert into t4 values(1577980800011, 12, 4)
-sql insert into t4 values(1577980800012, 13, 3)
-sql insert into t4 values(1577980800013, 14, 2)
-sql insert into t4 values(1577980800014, 15, 1)
-sql insert into t4 values(1577980800020, 21, 5)
-sql insert into t4 values(1577980800021, 22, 4)
-sql insert into t4 values(1577980800022, 23, 3)
-sql insert into t4 values(1577980800023, 24, 2)
-sql insert into t4 values(1577980800024, 25, 1)
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-sql select * from t4
-if $rows != 15 then
-	return -1
-endi
-
-system sh/exec.sh -n dnode2 -s stop -x SIGKILL
-$x = 0
-step7: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step7
-endi
-if $data4_2 != offline then
-  goto step7
-endi
-if $data4_3 !=  ready then
-  goto step7
-endi
-
-sql show db.vgroups
-if $data04 != 3 then
-	goto step7
-endi
-if $data06 != 2 then
-	goto step7
-endi
-if $data05 != master then
-	goto step7
-endi
-if $data07 != offline then
-	goto step7
-endi
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-sql select * from t4
-if $rows != 15 then
-	return -1
-endi
-
-print ============== step8
-sql create table t5 using stb tags(1)
-sql insert into t5 values(1577980800000, 1, 5)
-sql insert into t5 values(1577980800001, 2, 4)
-sql insert into t5 values(1577980800002, 3, 3)
-sql insert into t5 values(1577980800003, 4, 2)
-sql insert into t5 values(1577980800004, 5, 1)
-sql insert into t5 values(1577980800010, 11, 5)
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-sql select * from t4
-if $rows != 15 then
-	return -1
-endi
-sql select * from t5
-if $rows != 6 then
-	return -1
-endi
-
-system sh/exec.sh -n dnode2 -s start
-$x = 0
-step8: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step8
-endi
-if $data4_2 != ready then
-  goto step8
-endi
-if $data4_3 != ready then
-  goto step8
-endi
-
-sql show db.vgroups
-if $data04 != 3 then
-	goto step8
-endi
-if $data06 != 2 then
-	goto step8
-endi
-if $data05 != master then
-	goto step8
-endi
-if $data07 != slave then
-	goto step8
-endi
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-sql select * from t4
-if $rows != 15 then
-	return -1
-endi
-sql select * from t5
-if $rows != 6 then
-	return -1
-endi
-
-print ============== step9
-sql create table t6 using stb tags(1)
-sql insert into t6 values(1577980800000, 1, 5)
-sql insert into t6 values(1577980800001, 2, 4)
-sql insert into t6 values(1577980800002, 3, 3)
-sql insert into t6 values(1577980800003, 4, 2)
-sql insert into t6 values(1577980800004, 5, 1)
-sql insert into t6 values(1577980800010, 11, 5)
-sql insert into t6 values(1577980800011, 12, 4)
-
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then
-	return -1
-endi
-sql select * from t4
-if $rows != 15 then
-	return -1
-endi
-sql select * from t5
-if $rows != 6 then
-	return -1
-endi
-sql select * from t6
-if $rows != 7 then
-	return -1
-endi
-
-system sh/exec.sh -n dnode3 -s stop -x SIGKILL
-$x = 0
-step9: 
-	$x = $x + 1
-	sleep 1000
-	if $x == 10 then
-		return -1
-	endi
-
-sql show dnodes
-print dnode1 $data4_1
-print dnode2 $data4_2
-print dnode3 $data4_3
-
-if $data4_1 != ready then
-  goto step9
-endi
-if $data4_2 != ready then
-  goto step9
-endi
-if $data4_3 != offline then
-  goto step9
-endi
-
-print ============== 2
-sql show db.vgroups
-
-if $data04 != 3 then
-	goto step7
-endi
-if $data06 != 2 then
-	goto step7
-endi
-if $data05 != offline then
-	goto step7
-endi
-if $data07 != master then
-	goto step7
-endi
-
-print ============== 3
-sql select * from t1
-if $rows != 5 then
-	return -1
-endi
-sql select * from t2
-if $rows != 5 then
-	return -1
-endi
-sql select * from t3
-if $rows != 10 then	
-	return -1
-endi
-sql select * from t4
-if $rows != 15 then
-	return -1
-endi
-sql select * from t5
-if $rows != 6 then
-	return -1
-endi
-sql select * from t6
-if $rows != 7 then
-	return -1
-endi
-
-system sh/exec.sh -n dnode1 -s stop
-system sh/exec.sh -n dnode2 -s stop
-system sh/exec.sh -n dnode3 -s stop
--- a/tests/script/tsim/parser/alter.sim
+++ b/tests/script/tsim/parser/alter.sim
@ -1,9 +1,6 @@
 system sh/stop_dnodes.sh
-
 system sh/deploy.sh -n dnode1 -i 1
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
 system sh/exec.sh -n dnode1 -s start
-sleep 100
 sql connect

 $dbPrefix = m_alt_db
@ -40,62 +37,61 @@ sql_error alter database $db keep 20,20,20,20
 sql_error alter database $db keep 365001,365001,365001
 sql alter database $db keep 21
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 21,21,21 then 
+if $data27 != 30240m,30240m,30240m then 
 return -1
 endi
 sql alter database $db keep 11,12
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 11,12,12 then 
+if $data27 != 15840m,17280m,17280m then 
 return -1
 endi
 sql alter database $db keep 20,20,20
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 20,20,20 then 
+if $data27 != 28800m,28800m,28800m then 
 return -1
 endi
 sql alter database $db keep 10,10,10
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 10,10,10 then 
+if $data27 != 14400m,14400m,14400m then 
 return -1
 endi
 sql alter database $db keep 10,10,11
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 10,10,11 then 
+if $data27 != 14400m,14400m,15840m then 
 return -1
 endi
 sql alter database $db keep 11,12,13
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 11,12,13 then 
+if $data27 != 15840m,17280m,18720m then 
 return -1
 endi
 sql alter database $db keep 365000,365000,365000
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 365000,365000,365000 then 
+if $data27 != 525600000m,525600000m,525600000m then 
 return -1
 endi

-
 ##### alter table test, simeplest case
 sql create table tb (ts timestamp, c1 int, c2 int, c3 int)
 sql insert into tb values (now, 1, 1, 1)
@ -187,7 +183,6 @@ endi
 sql drop table tb
 sql drop table mt

-sleep 100
 ### ALTER TABLE WHILE STREAMING [TBASE271]
 #sql create table tb1 (ts timestamp, c1 int, c2 nchar(5), c3 int)
 #sql create table strm as select count(*), avg(c1), first(c2), sum(c3) from tb1 interval(2s)
@ -195,9 +190,9 @@ sleep 100
 #if $rows != 0 then
 #  return -1
 #endi
-##sleep 12000
+
 #sql insert into tb1 values (now, 1, 'taos', 1)
-#sleep 20000
+
 #sql select * from strm
 #print rows = $rows
 #if $rows != 1 then
@ -207,9 +202,9 @@ sleep 100
 #  return -1
 #endi
 #sql alter table tb1 drop column c3
-#sleep 500
+
 #sql insert into tb1 values (now, 2, 'taos')
-#sleep 30000
+
 #sql select * from strm
 #if $rows != 2 then
 #   return -1
@ -218,9 +213,9 @@ sleep 100
 #  return -1
 #endi
 #sql alter table tb1 add column c3 int
-#sleep 500
+
 #sql insert into tb1 values (now, 3, 'taos', 3);
-#sleep 100
+
 #sql select * from strm
 #if $rows != 3 then
 #   return -1
@ -259,7 +254,7 @@ sql create database $db
 sql use $db
 sql create table mt (ts timestamp, c1 int, c2 nchar(7), c3 int) tags (t1 int)
 sql create table tb using mt tags(1)
-sleep 100
+
 sql insert into tb values ('2018-11-01 16:30:00.000', 1, 'insert', 1)
 sql alter table mt drop column c3

--- a/tests/script/tsim/parser/alter1.sim
+++ b/tests/script/tsim/parser/alter1.sim
@ -1,11 +1,7 @@
 system sh/stop_dnodes.sh
-
 system sh/deploy.sh -n dnode1 -i 1
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
 system sh/exec.sh -n dnode1 -s start
-sleep 100
 sql connect
-sql reset query cache

 $dbPrefix = alt1_db

@ -87,9 +83,8 @@ if $data13 != NULL then
  return -1
 endi

-sleep 100
 print ================== insert values into table
-sql insert into car1 values (now, 1, 1,1 ) (now +1s, 2,2,2,) car2 values (now, 1,3,3)
+sql insert into car1 values (now, 1, 1,1 ) (now +1s, 2,2,2) car2 values (now, 1,3,3)

 sql select c1+speed from stb where c1 > 0
 if $rows != 3 then
--- a/tests/script/tsim/parser/alter__for_community_version.sim
+++ b/tests/script/tsim/parser/alter__for_community_version.sim
@ -1,9 +1,6 @@
 system sh/stop_dnodes.sh
-
 system sh/deploy.sh -n dnode1 -i 1
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
 system sh/exec.sh -n dnode1 -s start
-sleep 100
 sql connect

 $dbPrefix = m_alt_db
@ -23,10 +20,10 @@ sql drop database if exists $db
 sql create database $db duration 10 keep 20
 sql use $db
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 20 then 
+if $data27 != 28800m,28800m,28800m then 
 return -1
 endi

@ -47,44 +44,44 @@ sql_error alter database $db keep 20,19,18
 sql_error alter database $db keep 20,20,20,20
 sql_error alter database $db keep 365001,365001,365001
 sql_error alter database $db keep 365001
-sql alter database $db keep 20
+sql_error alter database $db keep 20
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 20 then 
+if $data27 != 28800m,28800m,28800m then 
 return -1
 endi
 sql alter database $db keep 10
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 10 then 
+if $data27 != 14400m,14400m,14400m then 
 return -1
 endi
 sql alter database $db keep 11
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 11 then 
+if $data27 != 15840m,15840m,15840m then 
 return -1
 endi
 sql alter database $db keep 13
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 13 then 
+if $data27 != 18720m,18720m,18720m then 
 return -1
 endi
 sql alter database $db keep 365000
 sql show databases
-if $rows != 1 then
+if $rows != 3 then
  return -1
 endi
-if $data07 != 365000 then 
+if $data27 != 525600000m,525600000m,525600000m then 
 return -1
 endi

@ -180,7 +177,6 @@ endi
 sql drop table tb
 sql drop table mt

-sleep 100
 ### ALTER TABLE WHILE STREAMING [TBASE271]
 #sql create table tb1 (ts timestamp, c1 int, c2 nchar(5), c3 int)
 #sql create table strm as select count(*), avg(c1), first(c2), sum(c3) from tb1 interval(2s)
@ -188,9 +184,7 @@ sleep 100
 #if $rows != 0 then
 #  return -1
 #endi
-##sleep 12000
 #sql insert into tb1 values (now, 1, 'taos', 1)
-#sleep 20000
 #sql select * from strm
 #print rows = $rows
 #if $rows != 1 then
@ -200,9 +194,7 @@ sleep 100
 #  return -1
 #endi
 #sql alter table tb1 drop column c3
-#sleep 500
 #sql insert into tb1 values (now, 2, 'taos')
-#sleep 30000
 #sql select * from strm
 #if $rows != 2 then
 #   return -1
@ -211,9 +203,7 @@ sleep 100
 #  return -1
 #endi
 #sql alter table tb1 add column c3 int
-#sleep 500
 #sql insert into tb1 values (now, 3, 'taos', 3);
-#sleep 100
 #sql select * from strm
 #if $rows != 3 then
 #   return -1
@ -252,7 +242,6 @@ sql create database $db
 sql use $db
 sql create table mt (ts timestamp, c1 int, c2 nchar(7), c3 int) tags (t1 int)
 sql create table tb using mt tags(1)
-sleep 100
 sql insert into tb values ('2018-11-01 16:30:00.000', 1, 'insert', 1)
 sql alter table mt drop column c3

--- a/tests/script/tsim/parser/alter_column.sim
+++ b/tests/script/tsim/parser/alter_column.sim
@ -1,9 +1,6 @@
 system sh/stop_dnodes.sh
-
 system sh/deploy.sh -n dnode1 -i 1
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
 system sh/exec.sh -n dnode1 -s start
-sleep 100
 sql connect

 $dbPrefix = m_alt_db
@ -26,51 +23,20 @@ sql use $db
 sql create table tb (ts timestamp, c1 int, c2 binary(10), c3 nchar(10))
 sql insert into tb values (now, 1, "1", "1")
 sql alter table tb modify column c2 binary(20);
-if $rows != 0 then
-  return -1
-endi
 sql alter table tb modify column c3 nchar(20);
-if $rows != 0 then
-  return -1
-endi


 sql create stable stb (ts timestamp, c1 int, c2 binary(10), c3 nchar(10)) tags(id1 int, id2 binary(10), id3 nchar(10))
 sql create table tb1 using stb tags(1, "a", "b")
 sql insert into tb1 values (now, 1, "1", "1")
 sql alter stable stb modify column c2 binary(20);
-if $rows != 0 then
-  return -1
-endi
 sql alter table stb modify column c2 binary(30);
-if $rows != 0 then
-  return -1
-endi
 sql alter stable stb modify column c3 nchar(20);
-if $rows != 0 then
-  return -1
-endi
 sql alter table stb modify column c3 nchar(30);
-if $rows != 0 then
-  return -1
-endi
-
 sql alter table stb modify tag id2 binary(11);
-if $rows != 0 then
-  return -1
-endi
-sql alter stable stb modify tag id2 binary(11);
-if $rows != 0 then
-  return -1
-endi
+sql_error alter stable stb modify tag id2 binary(11);
 sql alter table stb modify tag id3 nchar(11);
-if $rows != 0 then
-  return -1
-endi
-sql alter stable stb modify tag id3 nchar(11);
-if $rows != 0 then
-  return -1
-endi
+sql_error alter stable stb modify tag id3 nchar(11);

 ##### ILLEGAL OPERATIONS

@ -82,14 +48,14 @@ sql_error alter table tb modify column c2 binary(10);
 sql_error alter table tb modify column c2 binary(9);
 sql_error alter table tb modify column c2 binary(-9);
 sql_error alter table tb modify column c2 binary(0);
-sql_error alter table tb modify column c2 binary(17000);
+sql alter table tb modify column c2 binary(17000);
 sql_error alter table tb modify column c2 nchar(30);
 sql_error alter table tb modify column c3 double;
 sql_error alter table tb modify column c3 nchar(10);
 sql_error alter table tb modify column c3 nchar(0);
 sql_error alter table tb modify column c3 nchar(-1);
 sql_error alter table tb modify column c3 binary(80);
-sql_error alter table tb modify column c3 nchar(17000);
+sql alter table tb modify column c3 nchar(17000);
 sql_error alter table tb modify column c3 nchar(100), c2 binary(30);
 sql_error alter table tb modify column c1 nchar(100), c2 binary(30);
 sql_error alter stable tb modify column c2 binary(30);
--- a/tests/script/tsim/parser/alter_stable.sim
+++ b/tests/script/tsim/parser/alter_stable.sim
@ -1,9 +1,6 @@
 system sh/stop_dnodes.sh
-
 system sh/deploy.sh -n dnode1 -i 1
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
 system sh/exec.sh -n dnode1 -s start
-sleep 100
 sql connect

 print ========== alter_stable.sim
@ -13,19 +10,19 @@ sql drop database if exists $db
 sql create database $db
 sql use $db

-##### alter stable test : change tag name
-# case-1  change tag name: new name inclue old name
+##### alter stable test : rename tag name
+# case-1  rename tag name: new name inclue old name
 sql create table mt1 (ts timestamp, c1 int) tags (a int)
-sql alter table mt1 change tag a abcd
-sql alter table mt1 change tag abcd a
-sql_error alter table mt1 change tag a 1
+sql alter table mt1 rename tag a abcd
+sql alter table mt1 rename tag abcd a
+sql_error alter table mt1 rename tag a 1

 sql_error create table mtx1 (ts timestamp, c1 int) tags (123 int)

 sql_error create table mt2 (ts timestamp, c1 int) tags (abc012345678901234567890123456789012345678901234567890123456789def int)
 sql create table mt3 (ts timestamp, c1 int) tags (abc012345678901234567890123456789012345678901234567890123456789 int)
-sql_error alter table mt3 change tag abc012345678901234567890123456789012345678901234567890123456789 abcdefg012345678901234567890123456789012345678901234567890123456789
-sql alter table mt3 change tag abc012345678901234567890123456789012345678901234567890123456789 abcdefg0123456789012345678901234567890123456789
+sql_error alter table mt3 rename tag abc012345678901234567890123456789012345678901234567890123456789 abcdefg012345678901234567890123456789012345678901234567890123456789
+sql alter table mt3 rename tag abc012345678901234567890123456789012345678901234567890123456789 abcdefg0123456789012345678901234567890123456789

 # case-2 set tag value
 sql create table mt4 (ts timestamp, c1 int) tags (name binary(16), len int)
@ -37,7 +34,7 @@ sql alter table tb1 set tag len = 379

 # case TD-5594
 sql create stable st5520(ts timestamp, f int) tags(t0 bool, t1 nchar(4093), t2 nchar(1))
-sql_error alter stable st5520 modify tag t2 nchar(2);
+sql alter stable st5520 modify tag t2 nchar(2);
 # test end   
 sql drop database $db

--- a/tests/script/tsim/parser/auto_create_tb.sim
+++ b/tests/script/tsim/parser/auto_create_tb.sim
@ -1,11 +1,8 @@
 system sh/stop_dnodes.sh
 system sh/deploy.sh -n dnode1 -i 1
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
-system sh/cfg.sh -n dnode1 -c maxtablesPerVnode -v 2
 system sh/exec.sh -n dnode1 -s start
-
-sleep 100
 sql connect
+
 print ======================== dnode1 start

 $dbPrefix = ac_db
@ -153,36 +150,37 @@ print $rows $data00 $data10 $data20
 if $rows != 3 then
  return -1
 endi
-if $data00 != tb1 then
+if $data(tb1)[0] != tb1 then
  return -1
 endi
-if $data10 != tb2 then
+if $data(tb2)[0] != tb2 then
  return -1
 endi
-if $data20 != tb3 then
+if $data(tb3)[0] != tb3 then
  return -1
 endi

-sql select ts,c1,c2,c3,c4,c5,c7,c8,c9 from $stb
+sql select c1,c1,c2,c3,c4,c5,c7,c8,c9 from $stb
+print ===> $data00 $data01 $data02 $data03 $data04 $data05 $data06 $data07 $data08 $data09
+print ===> $data10 $data11 $data12 $data13 $data14 $data15 $data16 $data17 $data18 $data19
+print ===> $data20 $data21 $data22 $data23 $data24 $data25 $data26 $data27 $data28 $data29
+
 if $rows != 3 then
  return -1
 endi
-#if $data00 != @18-09-17 09:00:00.000@ then
-#  return -1
-#endi
-if $data01 != 1 then
+if $data(1)[1] != 1 then
  return -1
 endi
-if $data08 != 涛思数据1 then
+if $data(1)[8] != 涛思数据1 then
  return -1
 endi
-if $data14 != 2.000000000 then
+if $data(2)[4] != 2.000000000 then
  return -1
 endi
-if $data18 != 涛思数据2 then
+if $data(2)[8] != 涛思数据2 then
  return -1
 endi
-if $data28 != 涛思数据3 then
+if $data(3)[8] != 涛思数据3 then
  return -1
 endi

@ -208,12 +206,7 @@ endi

 print ================== restart server to commit data into disk
 system sh/exec.sh -n dnode1 -s stop -x SIGINT
-sleep 500
 system sh/exec.sh -n dnode1 -s start
-print ================== server restart completed
-sql connect
-sleep 100
-sql use $db

 #### auto create multiple tables
 sql insert into tb1 using $stb tags(1) values ( $ts0 , 1, 1, 1, 1, 'bin1', 1, 1, 1, '涛思数据1') tb2 using $stb tags(2) values ( $ts0 , 2, 2, 2, 2, 'bin2', 2, 2, 2, '涛思数据2') tb3 using $stb tags(3) values ( $ts0 , 3, 3, 3, 3, 'bin3', 3, 3, 3, '涛思数据3')
@ -221,36 +214,37 @@ sql show tables
 if $rows != 3 then
  return -1
 endi
-if $data00 != tb1 then
+if $data(tb1)[0] != tb1 then
  return -1
 endi
-if $data10 != tb2 then
+if $data(tb2)[0] != tb2 then
  return -1
 endi
-if $data20 != tb3 then
+if $data(tb3)[0] != tb3 then
  return -1
 endi

-sql select ts,c1,c2,c3,c4,c5,c7,c8,c9 from $stb
+sql select c1,c1,c2,c3,c4,c5,c7,c8,c9 from $stb
+print ===> $data00 $data01 $data02 $data03 $data04 $data05 $data06 $data07 $data08 $data09
+print ===> $data10 $data11 $data12 $data13 $data14 $data15 $data16 $data17 $data18 $data19
+print ===> $data20 $data21 $data22 $data23 $data24 $data25 $data26 $data27 $data28 $data29
+
 if $rows != 3 then
  return -1
 endi
-#if $data00 != @18-09-17 09:00:00.000@ then
-#  return -1
-#endi
-if $data01 != 1 then
+if $data(1)[1] != 1 then
  return -1
 endi
-if $data08 != 涛思数据1 then
+if $data(1)[8] != 涛思数据1 then
  return -1
 endi
-if $data14 != 2.000000000 then
+if $data(2)[4] != 2.000000000 then
  return -1
 endi
-if $data18 != 涛思数据2 then
+if $data(2)[8] != 涛思数据2 then
  return -1
 endi
-if $data28 != 涛思数据3 then
+if $data(3)[8] != 涛思数据3 then
  return -1
 endi

--- a/tests/script/tsim/qnode/basic1.sim
+++ b/tests/script/tsim/qnode/basic1.sim
@ -6,32 +6,32 @@ system sh/exec.sh -n dnode2 -s start
 sql connect

 print =============== show dnodes
-sql show dnodes;
-if $rows != 1 then 
-  return -1
-endi
-
-if $data00 != 1 then 
-  return -1
-endi
-
-sql show mnodes;
-if $rows != 1 then 
-  return -1
-endi
-
-if $data00 != 1 then 
-  return -1
-endi
-
-if $data02 != leader then 
-  return -1
-endi
-
-print =============== create dnodes
 sql create dnode $hostname port 7200
-sleep 2000

+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
+		return -1
+	endi
+sql show dnodes
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 2 then
+  return -1
+endi
+if $data(1)[4] != ready then
+  goto step1
+endi
+if $data(2)[4] != ready then
+  goto step1
+endi
+
+print =============== show dnodes
 sql show dnodes;
 if $rows != 2 then 
  return -1
@ -125,7 +125,6 @@ system sh/exec.sh -n dnode2 -s stop -x SIGINT
 system sh/exec.sh -n dnode1 -s start
 system sh/exec.sh -n dnode2 -s start

-sleep 2000
 sql show qnodes
 if $rows != 2 then 
  return -1
--- a/tests/script/tsim/snode/basic1.sim
+++ b/tests/script/tsim/snode/basic1.sim
@ -6,32 +6,32 @@ system sh/exec.sh -n dnode2 -s start
 sql connect

 print =============== show dnodes
-sql show dnodes;
-if $rows != 1 then 
-  return -1
-endi
-
-if $data00 != 1 then 
-  return -1
-endi
-
-sql show mnodes;
-if $rows != 1 then 
-  return -1
-endi
-
-if $data00 != 1 then 
-  return -1
-endi
-
-if $data02 != leader then 
-  return -1
-endi
-
-print =============== create dnodes
 sql create dnode $hostname port 7200
-sleep 2000

+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
+		return -1
+	endi
+sql show dnodes
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 2 then
+  return -1
+endi
+if $data(1)[4] != ready then
+  goto step1
+endi
+if $data(2)[4] != ready then
+  goto step1
+endi
+
+print =============== show dnodes
 sql show dnodes;
 if $rows != 2 then 
  return -1
--- a/tests/script/tsim/sync/3Replica1VgElect.sim
+++ b/tests/script/tsim/sync/3Replica1VgElect.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/sync/3Replica5VgElect.sim
+++ b/tests/script/tsim/sync/3Replica5VgElect.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/sync/3Replica5VgElect3mnode.sim
+++ b/tests/script/tsim/sync/3Replica5VgElect3mnode.sim
@ -3,66 +3,44 @@ system sh/deploy.sh -n dnode1 -i 1
 system sh/deploy.sh -n dnode2 -i 2
 system sh/deploy.sh -n dnode3 -i 3
 system sh/deploy.sh -n dnode4 -i 4
-
 system sh/cfg.sh -n dnode1 -c supportVnodes -v 0
-
 system sh/exec.sh -n dnode1 -s start
 system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-### create clusters using four dnodes;
-
-
-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> 1-dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/sync/3Replica5VgElect3mnodedrop.sim
+++ b/tests/script/tsim/sync/3Replica5VgElect3mnodedrop.sim
@ -11,57 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-print ===> create clusters using four dnodes;
-
-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> 1-dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/sync/electTest.sim
+++ b/tests/script/tsim/sync/electTest.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/sync/oneReplica1VgElect.sim
+++ b/tests/script/tsim/sync/oneReplica1VgElect.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $vgroups = 1
--- a/tests/script/tsim/sync/oneReplica1VgElectWithInsert.sim
+++ b/tests/script/tsim/sync/oneReplica1VgElectWithInsert.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $vgroups = 1
--- a/tests/script/tsim/sync/oneReplica5VgElect.sim
+++ b/tests/script/tsim/sync/oneReplica5VgElect.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 1
--- a/tests/script/tsim/sync/threeReplica1VgElect.sim
+++ b/tests/script/tsim/sync/threeReplica1VgElect.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
-#sql connect
+sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $vgroups = 1
--- a/tests/script/tsim/sync/threeReplica1VgElectWihtInsert.sim
+++ b/tests/script/tsim/sync/threeReplica1VgElectWihtInsert.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $vgroups = 1
--- a/tests/script/tsim/sync/vnodesnapshot-test.sim
+++ b/tests/script/tsim/sync/vnodesnapshot-test.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/sync/vnodesnapshot.sim
+++ b/tests/script/tsim/sync/vnodesnapshot.sim
@ -11,55 +11,38 @@ system sh/exec.sh -n dnode2 -s start
 system sh/exec.sh -n dnode3 -s start
 system sh/exec.sh -n dnode4 -s start

-$loop_cnt = 0
-check_dnode_ready:
-	$loop_cnt = $loop_cnt + 1
-	sleep 200
-	if $loop_cnt == 10 then
-	  print ====> dnode not ready!
-		return -1
-	endi
-sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][0] != 1 then
-  return -1
-endi
-if $data[0][4] != ready then
-  goto check_dnode_ready
-endi
-
 sql connect
 sql create dnode $hostname port 7200
 sql create dnode $hostname port 7300
 sql create dnode $hostname port 7400

-$loop_cnt = 0
-check_dnode_ready_1:
-$loop_cnt = $loop_cnt + 1
-sleep 200
-if $loop_cnt == 10 then
-  print ====> dnodes not ready!
+$x = 0
+step1:
+	$x = $x + 1
+	sleep 1000
+	if $x == 10 then
+	  print ====> dnode not ready!
 		return -1
 	endi
 sql show dnodes
-print ===> $rows $data[0][0] $data[0][1] $data[0][2] $data[0][3] $data[0][4] $data[0][5] $data[0][6]
-print ===> $rows $data[1][0] $data[1][1] $data[1][2] $data[1][3] $data[1][4] $data[1][5] $data[1][6]
-print ===> $rows $data[2][0] $data[2][1] $data[2][2] $data[2][3] $data[2][4] $data[2][5] $data[2][6]
-print ===> $rows $data[3][0] $data[3][1] $data[3][2] $data[3][3] $data[3][4] $data[3][5] $data[3][6]
-if $data[0][4] != ready then
-  goto check_dnode_ready_1
+print ===> $data00 $data01 $data02 $data03 $data04 $data05
+print ===> $data10 $data11 $data12 $data13 $data14 $data15
+print ===> $data20 $data21 $data22 $data23 $data24 $data25
+print ===> $data30 $data31 $data32 $data33 $data34 $data35
+if $rows != 4 then
+  return -1
 endi
-if $data[1][4] != ready then
-  goto check_dnode_ready_1
+if $data(1)[4] != ready then
+  goto step1
 endi
-if $data[2][4] != ready then
-  goto check_dnode_ready_1
+if $data(2)[4] != ready then
+  goto step1
 endi
-if $data[3][4] != ready then
-  goto check_dnode_ready_1
+if $data(3)[4] != ready then
+  goto step1
+endi
+if $data(4)[4] != ready then
+  goto step1
 endi

 $replica = 3
--- a/tests/script/tsim/table/back_insert.sim
+++ b/tests/script/tsim/table/back_insert.sim
@ -3,5 +3,6 @@ $x = 1
 begin:
  sql insert into db.tb values(now, $x ) -x begin  
  #print ===> insert successed $x 
+  sleep 100
  $x = $x + 1   
 goto begin
--- a/tests/script/tsim/table/delete_reuse1.sim
+++ b/tests/script/tsim/table/delete_reuse1.sim
@ -1,28 +1,6 @@
 system sh/stop_dnodes.sh
 system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-system sh/deploy.sh -n dnode4 -i 4
-
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
-system sh/cfg.sh -n dnode2 -c walLevel -v 1
-system sh/cfg.sh -n dnode3 -c walLevel -v 1
-system sh/cfg.sh -n dnode4 -c walLevel -v 1
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode4 -c numOfMnodes -v 1
-
-system sh/cfg.sh -n dnode1 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode2 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode3 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode4 -c mnodeEqualVnodeNum -v 4
-
-print ========= start dnodes
 system sh/exec.sh -n dnode1 -s start
-
-sleep 2000
 sql connect

 print ======== step1 
@ -37,7 +15,6 @@ endi

 print ======== step2
 sql drop table d1.t1
-sleep 1000
 sql insert into d1.t1 values(now, 2) -x step2
  return -1
 step2:
@ -58,7 +35,6 @@ $x = 0
 while $x < 20

  sql drop table d1.t1
-  sleep 1000
  sql insert into d1.t1 values(now, -1) -x step4
    return -1
  step4:
--- a/tests/script/tsim/table/delete_reuse2.sim
+++ b/tests/script/tsim/table/delete_reuse2.sim
@ -1,28 +1,6 @@
 system sh/stop_dnodes.sh
 system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-system sh/deploy.sh -n dnode4 -i 4
-
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
-system sh/cfg.sh -n dnode2 -c walLevel -v 1
-system sh/cfg.sh -n dnode3 -c walLevel -v 1
-system sh/cfg.sh -n dnode4 -c walLevel -v 1
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode4 -c numOfMnodes -v 1
-
-system sh/cfg.sh -n dnode1 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode2 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode3 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode4 -c mnodeEqualVnodeNum -v 4
-
-print ========= start dnodes
 system sh/exec.sh -n dnode1 -s start
-
-sleep 2000
 sql connect

 print ======== step1 
@ -37,7 +15,6 @@ endi

 print ======== step2
 sql drop table db1.t1
-sleep 1000
 sql_error insert into db1.t1 values(now, 2)

 print ========= step3
@ -59,7 +36,6 @@ while $x < 20
  $tb = tb . $x
  sql drop table $tb
  
-  sleep 1000
  sql_error insert into $tb values(now, -1)
  
  step4:
--- a/tests/script/tsim/table/delete_writing.sim
+++ b/tests/script/tsim/table/delete_writing.sim
@ -1,23 +1,7 @@
 system sh/stop_dnodes.sh
 system sh/deploy.sh -n dnode1 -i 1
-system sh/deploy.sh -n dnode2 -i 2
-system sh/deploy.sh -n dnode3 -i 3
-system sh/deploy.sh -n dnode4 -i 4
-
-system sh/cfg.sh -n dnode1 -c walLevel -v 1
-system sh/cfg.sh -n dnode2 -c walLevel -v 1
-system sh/cfg.sh -n dnode3 -c walLevel -v 1
-system sh/cfg.sh -n dnode4 -c walLevel -v 1
-
-system sh/cfg.sh -n dnode1 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode2 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode3 -c numOfMnodes -v 1
-system sh/cfg.sh -n dnode4 -c numOfMnodes -v 1
-
-system sh/cfg.sh -n dnode1 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode2 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode3 -c mnodeEqualVnodeNum -v 4
-system sh/cfg.sh -n dnode4 -c mnodeEqualVnodeNum -v 4
+system sh/exec.sh -n dnode1 -s start
+sql connect

 print ========= start dnodes
 system sh/exec.sh -n dnode1 -s start
@ -28,12 +12,12 @@ sql create table db.tb (ts timestamp, i int)
 sql insert into db.tb values(now, 1)

 print ======== start back
-run_back general/table/back_insert.sim
+run_back tsim/table/back_insert.sim
 sleep 1000

 print ======== step1 
 $x = 1 	
-while $x < 15
+while $x < 10

    print drop table times $x
 	sql drop table db.tb -x step1
--- a/tests/system-test/0-others/sysinfo.py
+++ b/tests/system-test/0-others/sysinfo.py
@ -24,6 +24,7 @@ class TDTestCase:
        tdLog.debug("start to execute %s" % __file__)
        tdSql.init(conn.cursor())
        self.dbname = 'db'
+        self.delaytime = 10
    def get_database_info(self):
        tdSql.query('select database()')
        tdSql.checkData(0,0,None)
@ -43,14 +44,15 @@ class TDTestCase:
    def get_server_status(self):
        tdSql.query('select server_status()')
        tdSql.checkData(0,0,1)
-        tdDnodes.stoptaosd(1)
+        #!for bug
+        # tdDnodes.stoptaosd(1)
+        # sleep(self.delaytime)
+        # tdSql.error('select server_status()')
        
-        tdSql.query('select server_status()')
-        print(tdSql.queryResult)
    def run(self):
        self.get_database_info()
        self.check_version()
-        # self.get_server_status()
+        self.get_server_status()
    def stop(self):
        tdSql.close()
        tdLog.success("%s successfully executed" % __file__)
--- a/tests/system-test/2-query/tsbsQuery.py
+++ b/tests/system-test/2-query/tsbsQuery.py
@ -85,7 +85,7 @@ class TDTestCase:
        # tdSql.checkRows(10)

        # test partition interval Pseudo time-column
-        tdSql.query("SELECT count(ms1)/144  FROM (SELECT _wstartts as ts1,model, fleet,avg(status) AS ms1 FROM diagnostics WHERE ts >= '2016-01-01T00:00:00Z' AND ts < '2016-01-05T00:00:01Z'  partition by model, fleet interval(10m)) WHERE ts1 >= '2016-01-01T00:00:00Z' AND ts1 < '2016-01-05T00:00:01Z' AND ms1<1;")
+        tdSql.query("SELECT count(ms1)/144  FROM (SELECT _wstart as ts1,model, fleet,avg(status) AS ms1 FROM diagnostics WHERE ts >= '2016-01-01T00:00:00Z' AND ts < '2016-01-05T00:00:01Z'  partition by model, fleet interval(10m)) WHERE ts1 >= '2016-01-01T00:00:00Z' AND ts1 < '2016-01-05T00:00:01Z' AND ms1<1;")


        # test
--- a/tools/CMakeLists.txt
+++ b/tools/CMakeLists.txt
@ -114,7 +114,7 @@ ELSE ()
                COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -ldflags "-s -w -X github.com/taosdata/taosadapter/version.Version=${taos_version} -X github.com/taosdata/taosadapter/version.CommitID=${taosadapter_commit_sha1}"
                COMMAND CGO_CFLAGS=-I${CMAKE_CURRENT_SOURCE_DIR}/../include/client CGO_LDFLAGS=-L${CMAKE_BINARY_DIR}/build/lib  go build -a -o taosadapter-debug -ldflags "-X github.com/taosdata/taosadapter/version.Version=${taos_version} -X github.com/taosdata/taosadapter/version.CommitID=${taosadapter_commit_sha1}"
                INSTALL_COMMAND
-                COMMAND curl -sL https://github.com/upx/upx/releases/download/v3.96/upx-3.96-${PLATFORM_ARCH_STR}_linux.tar.xz -o upx.tar.xz && tar -xvJf upx.tar.xz -C ${CMAKE_BINARY_DIR} --strip-components 1 > /dev/null && ${CMAKE_BINARY_DIR}/upx taosadapter || :
+                COMMAND wget -c https://github.com/upx/upx/releases/download/v3.96/upx-3.96-${PLATFORM_ARCH_STR}_linux.tar.xz -O ${CMAKE_CURRENT_SOURCE_DIR}/upx.tar.xz && tar -xvJf ${CMAKE_CURRENT_SOURCE_DIR}/upx.tar.xz -C ${CMAKE_CURRENT_SOURCE_DIR} --strip-components 1 > /dev/null && ${CMAKE_CURRENT_SOURCE_DIR}/upx taosadapter || :
                COMMAND cmake -E copy taosadapter ${CMAKE_BINARY_DIR}/build/bin
                COMMAND cmake -E make_directory ${CMAKE_BINARY_DIR}/test/cfg/
                COMMAND cmake -E copy ./example/config/taosadapter.toml ${CMAKE_BINARY_DIR}/test/cfg/