xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

other: merge 3.0

This commit is contained in:
Haojun Liao 2022-11-07 09:28:22 +08:00
parent 341112923b f22029b886
commit 270404318c
110 changed files with 2236 additions and 916 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
cmake/taostools_CMakeLists.txt.in
View File

@ -2,7 +2,7 @@
# taos-tools
ExternalProject_Add(taos-tools
GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
GIT_TAG 16eb34f
GIT_TAG 0fb640b
SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
BINARY_DIR ""
#BUILD_IN_SOURCE TRUE

4
docs/en/07-develop/03-insert-data/02-influxdb-line.mdx
View File

@ -37,9 +37,9 @@ meters,location=California.LosAngeles,groupid=2 current=13.4,voltage=223,phase=0
- All the data in `tag_set` will be converted to NCHAR type automatically .
- Each data in `field_set` must be self-descriptive for its data type. For example 1.2f32 means a value 1.2 of float type. Without the "f" type suffix, it will be treated as type double.
- Multiple kinds of precision can be used for the `timestamp` field. Time precision can be from nanosecond (ns) to hour (h).
- You can configure smlChildTableName in taos.cfg to specify table names, for example, `smlChildTableName=tname`. You can insert `st,tname=cpul,t1=4 c1=3 1626006833639000000` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
- The child table name is created automatically in a rule to guarantee its uniqueness. But you can configure `smlChildTableName` in taos.cfg to specify a tag value as the table names if the tag value is unique globally. For example, if a tag is called `tname` and you set `smlChildTableName=tname` in taos.cfg, when you insert `st,tname=cpu1,t1=4 c1=3 1626006833639000000`, the child table `cpu1` will be created automatically. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
- It is assumed that the order of field_set in a supertable is consistent, meaning that the first record contains all fields and subsequent records store fields in the same order. If the order is not consistent, set smlDataFormat in taos.cfg to false. Otherwise, data will be written out of order and a database error will occur.(smlDataFormat in taos.cfg default to false after version of 3.0.1.3)
:::
:::
For more details please refer to [InfluxDB Line Protocol](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/) and [TDengine Schemaless](/reference/schemaless/#Schemaless-Line-Protocol)

2
docs/en/07-develop/03-insert-data/03-opentsdb-telnet.mdx
View File

@ -32,7 +32,7 @@ For example:
meters.current 1648432611250 11.3 location=California.LosAngeles groupid=3
```
- The defult child table name is generated by rules.You can configure smlChildTableName in taos.cfg to specify child table names, for example, `smlChildTableName=tname`. You can insert `meters.current 1648432611250 11.3 tname=cpu1 location=California.LosAngeles groupid=3` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
- The child table name is created automatically in a rule to guarantee its uniqueness. But you can configure `smlChildTableName` in taos.cfg to specify a tag value as the table names if the tag value is unique globally. For example, if a tag is called `tname` and you set `smlChildTableName=tname` in taos.cfg, when you insert `st,tname=cpu1,t1=4 c1=3 1626006833639000000`, the child table `cpu1` will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
Please refer to [OpenTSDB Telnet API](http://opentsdb.net/docs/build/html/api_telnet/put.html) for more details.
## Examples

4
docs/en/07-develop/03-insert-data/04-opentsdb-json.mdx
View File

@ -48,8 +48,8 @@ Please refer to [OpenTSDB HTTP API](http://opentsdb.net/docs/build/html/api_http
- In JSON protocol, strings will be converted to NCHAR type and numeric values will be converted to double type.
- Only data in array format is accepted and so an array must be used even if there is only one row.
- The defult child table name is generated by rules.You can configure smlChildTableName in taos.cfg to specify child table names, for example, `smlChildTableName=tname`. You can insert `"tags": { "host": "web02","dc": "lga","tname":"cpu1"}` and the cpu1 table will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
:::
- The child table name is created automatically in a rule to guarantee its uniqueness. But you can configure `smlChildTableName` in taos.cfg to specify a tag value as the table names if the tag value is unique globally. For example, if a tag is called `tname` and you set `smlChildTableName=tname` in taos.cfg, when you insert `st,tname=cpu1,t1=4 c1=3 1626006833639000000`, the child table `cpu1` will be automatically created. Note that if multiple rows have the same tname but different tag_set values, the tag_set of the first row is used to create the table and the others are ignored.
:::
## Examples

2
docs/en/14-reference/05-taosbenchmark.md
View File

@ -217,7 +217,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
The parameters listed in this section apply to all function modes.
- **filetype** : The function to be tested, with optional values `insert`, `query` and `subscribe`. These correspond to the insert, query, and subscribe functions, respectively. Users can specify only one of these in each configuration file.
**cfgdir**: specify the TDengine cluster configuration file's directory. The default path is /etc/taos.
**cfgdir**: specify the TDengine client configuration file's directory. The default path is /etc/taos.
- **host**: Specify the FQDN of the TDengine server to connect. The default value is `localhost`.

24
docs/en/14-reference/12-config/index.md
View File

@ -735,19 +735,17 @@ To prevent system resource from being exhausted by multiple concurrent streams,
| 44 | numOfVnodeQueryThreads | No | Yes |
| 45 | numOfVnodeStreamThreads | No | Yes |
| 46 | numOfVnodeFetchThreads | No | Yes |
| 47 | numOfVnodeWriteThreads | No | Yes |
| 48 | numOfVnodeSyncThreads | No | Yes |
| 49 | numOfVnodeRsmaThreads | No | Yes |
| 50 | numOfQnodeQueryThreads | No | Yes |
| 51 | numOfQnodeFetchThreads | No | Yes |
| 52 | numOfSnodeSharedThreads | No | Yes |
| 53 | numOfSnodeUniqueThreads | No | Yes |
| 54 | rpcQueueMemoryAllowed | No | Yes |
| 55 | logDir | Yes | Yes |
| 56 | minimalLogDirGB | Yes | Yes |
| 57 | numOfLogLines | Yes | Yes |
| 58 | asyncLog | Yes | Yes |
| 59 | logKeepDays | Yes | Yes |
| 47 | numOfVnodeRsmaThreads | No | Yes |
| 48 | numOfQnodeQueryThreads | No | Yes |
| 49 | numOfQnodeFetchThreads | No | Yes |
| 50 | numOfSnodeSharedThreads | No | Yes |
| 51 | numOfSnodeUniqueThreads | No | Yes |
| 52 | rpcQueueMemoryAllowed | No | Yes |
| 53 | logDir | Yes | Yes |
| 54 | minimalLogDirGB | Yes | Yes |
| 55 | numOfLogLines | Yes | Yes |
| 56 | asyncLog | Yes | Yes |
| 57 | logKeepDays | Yes | Yes |
| 60 | debugFlag | Yes | Yes |
| 61 | tmrDebugFlag | Yes | Yes |
| 62 | uDebugFlag | Yes | Yes |

3
docs/examples/csharp/wsConnect/Program.cs
View File

@ -9,9 +9,10 @@ namespace Examples
{
string DSN = "ws://root:taosdata@127.0.0.1:6041/test";
IntPtr wsConn = LibTaosWS.WSConnectWithDSN(DSN);
if (wsConn == IntPtr.Zero)
{
throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
throw new Exception("get WS connection failed");
}
else
{

2
docs/examples/csharp/wsInsert/Program.cs
View File

@ -13,7 +13,7 @@ namespace Examples
// Assert if connection is validate
if (wsConn == IntPtr.Zero)
{
throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
throw new Exception("get WS connection failed");
}
else
{

2
docs/examples/csharp/wsQuery/Program.cs
View File

@ -13,7 +13,7 @@ namespace Examples
IntPtr wsConn = LibTaosWS.WSConnectWithDSN(DSN);
if (wsConn == IntPtr.Zero)
{
throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
throw new Exception("get WS connection failed");
}
else
{

2
docs/examples/csharp/wsStmt/Program.cs
View File

@ -21,7 +21,7 @@ namespace Examples
IntPtr wsConn = LibTaosWS.WSConnectWithDSN(DSN);
if (wsConn == IntPtr.Zero)
{
throw new Exception($"get WS connection failed,reason:{LibTaosWS.WSErrorStr(IntPtr.Zero)} code:{LibTaosWS.WSErrorNo(IntPtr.Zero)}");
throw new Exception($"get WS connection failed");
}
else
{

5
docs/zh/07-develop/03-insert-data/02-influxdb-line.mdx
View File

@ -38,9 +38,8 @@ meters,location=California.LosAngeles,groupid=2 current=13.4,voltage=223,phase=0
- field_set 中的每个数据项都需要对自身的数据类型进行描述, 比如 1.2f32 代表 FLOAT 类型的数值 1.2, 如果不带类型后缀会被当作 DOUBLE 处理;
- timestamp 支持多种时间精度。写入数据的时候需要用参数指定时间精度，支持从小时到纳秒的 6 种时间精度。
- 为了提高写入的效率，默认假设同一个超级表中 field_set 的顺序是一样的（第一条数据包含所有的 field，后面的数据按照这个顺序），如果顺序不一样，需要配置参数 smlDataFormat 为 false，否则，数据写入按照相同顺序写入，库中数据会异常。（3.0.1.3 之后的版本 smlDataFormat 默认为 false） [TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
- 默认生产的子表名是根据规则生成的唯一 ID 值。为了让用户可以指定生成的表名，可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定。
举例如下：配置 smlChildTableName=tname 插入数据为 st,tname=cpu1,t1=4 c1=3 1626006833639000000 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。[TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
:::
- 默认产生的子表名是根据规则生成的唯一 ID 值。用户也可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定某个标签值作为子表名。该标签值应该具有全局唯一性。举例如下：假设有个标签名为tname, 配置 smlChildTableName=tname, 插入数据为 st,tname=cpu1,t1=4 c1=3 1626006833639000000 则创建的子表名为 cpu1。注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。[TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)
:::
要了解更多可参考：[InfluxDB Line 协议官方文档](https://docs.influxdata.com/influxdb/v2.0/reference/syntax/line-protocol/) 和 [TDengine 无模式写入参考指南](/reference/schemaless/#无模式写入行协议)

3
docs/zh/07-develop/03-insert-data/03-opentsdb-telnet.mdx
View File

@ -32,8 +32,7 @@ OpenTSDB 行协议同样采用一行字符串来表示一行数据。OpenTSDB
meters.current 1648432611250 11.3 location=California.LosAngeles groupid=3
```
- 默认生产的子表名是根据规则生成的唯一 ID 值。为了让用户可以指定生成的表名，可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定。
举例如下：配置 smlChildTableName=tname 插入数据为 meters.current 1648432611250 11.3 tname=cpu1 location=California.LosAngeles groupid=3 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
- 默认生产的子表名是根据规则生成的唯一 ID 值。用户也可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定某个标签值作为子表名。该标签值应该具有全局唯一性。举例如下：假设有个标签名为tname, 配置 smlChildTableName=tname, 插入数据为 meters.current 1648432611250 11.3 tname=cpu1 location=California.LosAngeles groupid=3 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
参考 [OpenTSDB Telnet API 文档](http://opentsdb.net/docs/build/html/api_telnet/put.html)。
## 示例代码

5
docs/zh/07-develop/03-insert-data/04-opentsdb-json.mdx
View File

@ -48,9 +48,8 @@ OpenTSDB JSON 格式协议采用一个 JSON 字符串表示一行或多行数据
- 对于 JSON 格式协议，TDengine 并不会自动把所有标签转成 NCHAR 类型， 字符串将将转为 NCHAR 类型， 数值将同样转换为 DOUBLE 类型。
- TDengine 只接收 JSON **数组格式**的字符串，即使一行数据也需要转换成数组形式。
- 默认生产的子表名是根据规则生成的唯一 ID 值。为了让用户可以指定生成的表名，可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定。
举例如下：配置 smlChildTableName=tname 插入数据为 `"tags": { "host": "web02","dc": "lga","tname":"cpu1"}` 则创建的表名为 cpu1，注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
:::
- 默认生成的子表名是根据规则生成的唯一 ID 值。用户也可以通过在 taos.cfg 里配置 smlChildTableName 参数来指定某个标签值作为子表名。该标签值应该具有全局唯一性。举例如下：假设有个标签名为tname, 配置 smlChildTableName=tname, 插入数据为 `"tags": { "host": "web02","dc": "lga","tname":"cpu1"}` 则创建的子表名为 cpu1。注意如果多行数据 tname 相同，但是后面的 tag_set 不同，则使用第一行自动建表时指定的 tag_set，其他的行会忽略）。
:::
## 示例代码

2
docs/zh/14-reference/05-taosbenchmark.md
View File

@ -217,7 +217,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
本节所列参数适用于所有功能模式。
- **filetype** : 要测试的功能，可选值为 `insert`, `query` 和 `subscribe`。分别对应插入、查询和订阅功能。每个配置文件中只能指定其中之一。
- **cfgdir** : TDengine 集群配置文件所在的目录，默认路径是 /etc/taos 。
- **cfgdir** : TDengine 客户端配置文件所在的目录，默认路径是 /etc/taos 。
- **host** : 指定要连接的 TDengine 服务端的 FQDN，默认值为 localhost。

24
docs/zh/14-reference/12-config/index.md
View File

@ -711,19 +711,17 @@ charset 的有效值是 UTF-8。
| 44 | numOfVnodeQueryThreads | 否 | 是 | |
| 45 | numOfVnodeStreamThreads | 否 | 是 | |
| 46 | numOfVnodeFetchThreads | 否 | 是 | |
| 47 | numOfVnodeWriteThreads | 否 | 是 | |
| 48 | numOfVnodeSyncThreads | 否 | 是 | |
| 49 | numOfVnodeRsmaThreads | 否 | 是 | |
| 50 | numOfQnodeQueryThreads | 否 | 是 | |
| 51 | numOfQnodeFetchThreads | 否 | 是 | |
| 52 | numOfSnodeSharedThreads | 否 | 是 | |
| 53 | numOfSnodeUniqueThreads | 否 | 是 | |
| 54 | rpcQueueMemoryAllowed | 否 | 是 | |
| 55 | logDir | 是 | 是 | |
| 56 | minimalLogDirGB | 是 | 是 | |
| 57 | numOfLogLines | 是 | 是 | |
| 58 | asyncLog | 是 | 是 | |
| 59 | logKeepDays | 是 | 是 | |
| 47 | numOfVnodeRsmaThreads | 否 | 是 | |
| 48 | numOfQnodeQueryThreads | 否 | 是 | |
| 49 | numOfQnodeFetchThreads | 否 | 是 | |
| 50 | numOfSnodeSharedThreads | 否 | 是 | |
| 51 | numOfSnodeUniqueThreads | 否 | 是 | |
| 52 | rpcQueueMemoryAllowed | 否 | 是 | |
| 53 | logDir | 是 | 是 | |
| 54 | minimalLogDirGB | 是 | 是 | |
| 55 | numOfLogLines | 是 | 是 | |
| 56 | asyncLog | 是 | 是 | |
| 57 | logKeepDays | 是 | 是 | |
| 60 | debugFlag | 是 | 是 | |
| 61 | tmrDebugFlag | 是 | 是 | |
| 62 | uDebugFlag | 是 | 是 | |

4
examples/go/taosdemo.go
View File

@ -26,7 +26,7 @@ import (
"sync"
"time"
_ "github.com/taosdata/driver-go/taosSql"
_ "github.com/taosdata/driver-go/v3/taosSql"
)
const (
@ -173,7 +173,7 @@ func createDatabase(dbName string, supTblName string) {
time.Sleep(time.Second)
// create database
sqlStr = "create database " + dbName + " keep " + strconv.Itoa(configPara.keep) + " days " + strconv.Itoa(configPara.days)
sqlStr = "create database " + dbName
_, err = db.Exec(sqlStr)
checkErr(err, sqlStr)

5
include/common/tdatablock.h
View File

@ -137,6 +137,9 @@ static FORCE_INLINE void colDataAppendNNULL(SColumnInfoData* pColumnInfoData, ui
for (int32_t i = start; i < start + nRows; ++i) {
colDataSetNull_f(pColumnInfoData->nullbitmap, i);
}
int32_t bytes = pColumnInfoData->info.bytes;
memset(pColumnInfoData->pData + start * bytes, 0, nRows * bytes);
}
pColumnInfoData->hasNull = true;
@ -215,7 +218,7 @@ size_t blockDataGetSerialMetaSize(uint32_t numOfCols);
int32_t blockDataSort(SSDataBlock* pDataBlock, SArray* pOrderInfo);
int32_t blockDataSort_rv(SSDataBlock* pDataBlock, SArray* pOrderInfo, bool nullFirst);
int32_t colInfoDataEnsureCapacity(SColumnInfoData* pColumn, uint32_t numOfRows);
int32_t colInfoDataEnsureCapacity(SColumnInfoData* pColumn, uint32_t numOfRows, bool clearPayload);
int32_t blockDataEnsureCapacity(SSDataBlock* pDataBlock, uint32_t numOfRows);
void colInfoDataCleanup(SColumnInfoData* pColumn, uint32_t numOfRows);

7
include/common/tdataformat.h
View File

@ -241,13 +241,6 @@ struct STag {
memcpy(varDataVal(x), (str), __len); \
} while (0);
#define STR_TO_NET_VARSTR(x, str) \
do { \
VarDataLenT __len = (VarDataLenT)strlen(str); \
*(VarDataLenT *)(x) = htons(__len); \
memcpy(varDataVal(x), (str), __len); \
} while (0);
#define STR_WITH_MAXSIZE_TO_VARSTR(x, str, _maxs) \
do { \
char *_e = stpncpy(varDataVal(x), (str), (_maxs)-VARSTR_HEADER_SIZE); \

2
include/common/tglobal.h
View File

@ -55,8 +55,6 @@ extern int32_t tsNumOfMnodeReadThreads;
extern int32_t tsNumOfVnodeQueryThreads;
extern int32_t tsNumOfVnodeStreamThreads;
extern int32_t tsNumOfVnodeFetchThreads;
extern int32_t tsNumOfVnodeWriteThreads;
extern int32_t tsNumOfVnodeSyncThreads;
extern int32_t tsNumOfVnodeRsmaThreads;
extern int32_t tsNumOfQnodeQueryThreads;
extern int32_t tsNumOfQnodeFetchThreads;

5
include/common/tmsg.h
View File

@ -216,9 +216,14 @@ typedef struct SEp {
uint16_t port;
} SEp;
#define SHOW_REWRITE_MASK() (1 << 0)
#define TEST_SHOW_REWRITE_MASK(m) (((m) & SHOW_REWRITE_MASK()) != 0)
typedef struct {
int32_t contLen;
int32_t vgId;
int32_t msgMask;
} SMsgHead;
// Submit message for one table

2
include/libs/function/function.h
View File

@ -182,7 +182,7 @@ struct SScalarParam {
};
void cleanupResultRowEntry(struct SResultRowEntryInfo *pCell);
int32_t getNumOfResult(SqlFunctionCtx *pCtx, int32_t num, SSDataBlock *pResBlock);
//int32_t getNumOfResult(SqlFunctionCtx *pCtx, int32_t num, SSDataBlock *pResBlock);
bool isRowEntryCompleted(struct SResultRowEntryInfo *pEntry);
bool isRowEntryInitialized(struct SResultRowEntryInfo *pEntry);

2
include/libs/function/functionMgt.h
View File

@ -224,7 +224,7 @@ int32_t fmGetDistMethod(const SFunctionNode* pFunc, SFunctionNode** pPartialFunc
typedef enum EFuncDataRequired {
FUNC_DATA_REQUIRED_DATA_LOAD = 1,
FUNC_DATA_REQUIRED_STATIS_LOAD,
FUNC_DATA_REQUIRED_SMA_LOAD,
FUNC_DATA_REQUIRED_NOT_LOAD,
FUNC_DATA_REQUIRED_FILTEROUT,
} EFuncDataRequired;

10
include/os/osDef.h
View File

@ -244,16 +244,6 @@ void syslog(int unused, const char *format, ...);
#define TD_CHARSET_LEN 64
#define TD_TIMEZONE_LEN 96
#ifdef WINDOWS
#define TD_PATH_MAX 260
#elif defined(PATH_MAX)
#define TD_PATH_MAX PATH_MAX
#elif defined(_XOPEN_PATH_MAX)
#define TD_PATH_MAX _XOPEN_PATH_MAX
#else
#define TD_PATH_MAX _POSIX_PATH_MAX
#endif
#ifdef __cplusplus
}
#endif

10
include/os/osFile.h
View File

@ -42,6 +42,16 @@ extern "C" {
#define PATH_MAX 256
#endif
#ifdef WINDOWS
#define TD_PATH_MAX _MAX_PATH
#elif defined(PATH_MAX)
#define TD_PATH_MAX PATH_MAX
#elif defined(_XOPEN_PATH_MAX)
#define TD_PATH_MAX _XOPEN_PATH_MAX
#else
#define TD_PATH_MAX _POSIX_PATH_MAX
#endif
typedef struct TdFile *TdFilePtr;
#define TD_FILE_CREATE 0x0001

41
include/util/tqueue.h
View File

@ -59,6 +59,47 @@ typedef enum {
typedef void (*FItem)(SQueueInfo *pInfo, void *pItem);
typedef void (*FItems)(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfItems);
typedef struct STaosQnode STaosQnode;
typedef struct STaosQnode {
STaosQnode *next;
STaosQueue *queue;
int64_t timestamp;
int32_t size;
int8_t itype;
int8_t reserved[3];
char item[];
} STaosQnode;
typedef struct STaosQueue {
STaosQnode *head;
STaosQnode *tail;
STaosQueue *next; // for queue set
STaosQset *qset; // for queue set
void *ahandle; // for queue set
FItem itemFp;
FItems itemsFp;
TdThreadMutex mutex;
int64_t memOfItems;
int32_t numOfItems;
int64_t threadId;
} STaosQueue;
typedef struct STaosQset {
STaosQueue *head;
STaosQueue *current;
TdThreadMutex mutex;
tsem_t sem;
int32_t numOfQueues;
int32_t numOfItems;
} STaosQset;
typedef struct STaosQall {
STaosQnode *current;
STaosQnode *start;
int32_t numOfItems;
} STaosQall;
STaosQueue *taosOpenQueue();
void taosCloseQueue(STaosQueue *queue);
void taosSetQueueFp(STaosQueue *queue, FItem itemFp, FItems itemsFp);

10
include/util/tworker.h
View File

@ -26,8 +26,9 @@ typedef struct SQWorkerPool SQWorkerPool;
typedef struct SWWorkerPool SWWorkerPool;
typedef struct SQWorker {
int32_t id; // worker ID
TdThread thread; // thread
int32_t id; // worker id
int64_t pid; // thread pid
TdThread thread; // thread id
SQWorkerPool *pool;
} SQWorker;
@ -43,9 +44,10 @@ typedef struct SQWorkerPool {
typedef struct SWWorker {
int32_t id; // worker id
TdThread thread; // thread
int64_t pid; // thread pid
TdThread thread; // thread id
STaosQall *qall;
STaosQset *qset; // queue set
STaosQset *qset;
SWWorkerPool *pool;
} SWWorker;

6
packaging/testpackage.sh
View File

@ -67,6 +67,7 @@ fi
}
function wgetFile {
file=$1
@ -75,7 +76,10 @@ if [ ! -f ${file} ];then
echoColor BD "wget https://www.taosdata.com/assets-download/3.0/${file}"
wget https://www.taosdata.com/assets-download/3.0/${file}
else
echoColor YD "${file} already exists "
echoColor YD "${file} already exists and use new file "
rm -rf ${file}
echoColor BD "wget https://www.taosdata.com/assets-download/3.0/${file}"
wget https://www.taosdata.com/assets-download/3.0/${file}
fi
}

2
packaging/tools/make_install.bat
View File

@ -34,7 +34,6 @@ if exist %binary_dir%\\test\\cfg\\taosadapter.toml (
copy %binary_dir%\\test\\cfg\\taosadapter.toml %tagert_dir%\\cfg\\taosadapter.toml > nul
)
)
copy %source_dir%\\include\\client\\taos.h %tagert_dir%\\include > nul
copy %source_dir%\\include\\util\\taoserror.h %tagert_dir%\\include > nul
copy %source_dir%\\include\\libs\\function\\taosudf.h %tagert_dir%\\include > nul
@ -52,6 +51,7 @@ if exist %binary_dir%\\build\\lib\\taosws.dll.lib (
)
if exist %binary_dir%\\build\\lib\\taosws.dll (
copy %binary_dir%\\build\\lib\\taosws.dll %tagert_dir%\\driver > nul
copy %source_dir%\\tools\\taosws-rs\\target\\release\\taosws.h %tagert_dir%\\include > nul
)
if exist %binary_dir%\\build\\bin\\taosdump.exe (
copy %binary_dir%\\build\\bin\\taosdump.exe %tagert_dir% > nul

4
packaging/tools/tdengine.iss
View File

@ -15,10 +15,10 @@
#define MyAppExeName "\*.exe"
#define MyAppTaosExeName "\taos.bat"
#define MyAppTaosdemoExeName "\taosBenchmark.exe"
#define MyAppDLLName "\driver\taos.dll"
#define MyAppDLLName "\driver\*.dll"
;#define MyAppVersion "3.0"
;#define MyAppInstallName "TDengine"
;#define MyAppInstallName "TDengine"
[Setup]
VersionInfoVersion={#MyAppVersion}
AppId={{A0F7A93C-79C4-485D-B2B8-F0D03DF42FAB}

46
source/common/src/tdatablock.c
View File

@ -246,7 +246,7 @@ int32_t colDataMergeCol(SColumnInfoData* pColumnInfoData, int32_t numOfRow1, int
uint32_t finalNumOfRows = numOfRow1 + numOfRow2;
if (IS_VAR_DATA_TYPE(pColumnInfoData->info.type)) {
// Handle the bitmap
if (finalNumOfRows > *capacity || (numOfRow1 == 0 && pColumnInfoData->info.bytes != 0)) {
if (finalNumOfRows > (*capacity)) {
char* p = taosMemoryRealloc(pColumnInfoData->varmeta.offset, sizeof(int32_t) * (numOfRow1 + numOfRow2));
if (p == NULL) {
return TSDB_CODE_OUT_OF_MEMORY;
@ -280,16 +280,14 @@ int32_t colDataMergeCol(SColumnInfoData* pColumnInfoData, int32_t numOfRow1, int
memcpy(pColumnInfoData->pData + oldLen, pSource->pData, len);
pColumnInfoData->varmeta.length = len + oldLen;
} else {
if (finalNumOfRows > *capacity || (numOfRow1 == 0 && pColumnInfoData->info.bytes != 0)) {
if (finalNumOfRows > (*capacity)) {
// all data may be null, when the pColumnInfoData->info.type == 0, bytes == 0;
// ASSERT(finalNumOfRows * pColumnInfoData->info.bytes);
char* tmp = taosMemoryRealloc(pColumnInfoData->pData, finalNumOfRows * pColumnInfoData->info.bytes);
if (tmp == NULL) {
return TSDB_CODE_OUT_OF_MEMORY;
}
pColumnInfoData->pData = tmp;
if (BitmapLen(numOfRow1) < BitmapLen(finalNumOfRows)) {
char* btmp = taosMemoryRealloc(pColumnInfoData->nullbitmap, BitmapLen(finalNumOfRows));
uint32_t extend = BitmapLen(finalNumOfRows) - BitmapLen(numOfRow1);
@ -823,7 +821,7 @@ static int32_t blockDataAssign(SColumnInfoData* pCols, const SSDataBlock* pDataB
}
static SColumnInfoData* createHelpColInfoData(const SSDataBlock* pDataBlock) {
int32_t rows = pDataBlock->info.rows;
int32_t rows = pDataBlock->info.capacity;
size_t numOfCols = taosArrayGetSize(pDataBlock->pDataBlock);
SColumnInfoData* pCols = taosMemoryCalloc(numOfCols, sizeof(SColumnInfoData));
@ -1126,26 +1124,28 @@ int32_t blockDataSort_rv(SSDataBlock* pDataBlock, SArray* pOrderInfo, bool nullF
}
void blockDataCleanup(SSDataBlock* pDataBlock) {
pDataBlock->info.rows = 0;
pDataBlock->info.groupId = 0;
SDataBlockInfo* pInfo = &pDataBlock->info;
pDataBlock->info.window.ekey = 0;
pDataBlock->info.window.skey = 0;
pInfo->rows = 0;
pInfo->groupId = 0;
pInfo->window.ekey = 0;
pInfo->window.skey = 0;
if (pDataBlock->info.capacity == 0) {
if (pInfo->capacity == 0) {
return;
}
size_t numOfCols = taosArrayGetSize(pDataBlock->pDataBlock);
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* p = taosArrayGet(pDataBlock->pDataBlock, i);
colInfoDataCleanup(p, pDataBlock->info.capacity);
colInfoDataCleanup(p, pInfo->capacity);
}
}
static int32_t doEnsureCapacity(SColumnInfoData* pColumn, const SDataBlockInfo* pBlockInfo, uint32_t numOfRows) {
ASSERT(numOfRows > 0 && pBlockInfo->capacity >= pBlockInfo->rows);
if (numOfRows < pBlockInfo->capacity) {
// todo temporarily disable it
static int32_t doEnsureCapacity(SColumnInfoData* pColumn, const SDataBlockInfo* pBlockInfo, uint32_t numOfRows, bool clearPayload) {
ASSERT(numOfRows > 0 /*&& pBlockInfo->capacity >= pBlockInfo->rows*/);
if (numOfRows <= pBlockInfo->capacity) {
return TSDB_CODE_SUCCESS;
}
@ -1182,8 +1182,10 @@ static int32_t doEnsureCapacity(SColumnInfoData* pColumn, const SDataBlockInfo*
return TSDB_CODE_OUT_OF_MEMORY;
}
memset(tmp + pColumn->info.bytes * existedRows, 0, pColumn->info.bytes * (numOfRows - existedRows));
pColumn->pData = tmp;
if (clearPayload) {
memset(tmp + pColumn->info.bytes * existedRows, 0, pColumn->info.bytes * (numOfRows - existedRows));
}
}
return TSDB_CODE_SUCCESS;
@ -1191,6 +1193,7 @@ static int32_t doEnsureCapacity(SColumnInfoData* pColumn, const SDataBlockInfo*
void colInfoDataCleanup(SColumnInfoData* pColumn, uint32_t numOfRows) {
pColumn->hasNull = false;
if (IS_VAR_DATA_TYPE(pColumn->info.type)) {
pColumn->varmeta.length = 0;
if (pColumn->varmeta.offset != NULL) {
@ -1203,30 +1206,27 @@ void colInfoDataCleanup(SColumnInfoData* pColumn, uint32_t numOfRows) {
}
}
int32_t colInfoDataEnsureCapacity(SColumnInfoData* pColumn, uint32_t numOfRows) {
int32_t colInfoDataEnsureCapacity(SColumnInfoData* pColumn, uint32_t numOfRows, bool clearPayload) {
SDataBlockInfo info = {0};
return doEnsureCapacity(pColumn, &info, numOfRows);
return doEnsureCapacity(pColumn, &info, numOfRows, clearPayload);
}
int32_t blockDataEnsureCapacity(SSDataBlock* pDataBlock, uint32_t numOfRows) {
int32_t code = 0;
if (numOfRows == 0) {
if (numOfRows == 0 || numOfRows <= pDataBlock->info.capacity) {
return TSDB_CODE_SUCCESS;
}
if (pDataBlock->info.capacity < numOfRows) {
pDataBlock->info.capacity = numOfRows;
}
size_t numOfCols = taosArrayGetSize(pDataBlock->pDataBlock);
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData* p = taosArrayGet(pDataBlock->pDataBlock, i);
code = doEnsureCapacity(p, &pDataBlock->info, numOfRows);
code = doEnsureCapacity(p, &pDataBlock->info, numOfRows, true);
if (code) {
return code;
}
}
pDataBlock->info.capacity = numOfRows;
return TSDB_CODE_SUCCESS;
}

35
source/common/src/tglobal.c
View File

@ -50,8 +50,6 @@ int32_t tsNumOfMnodeReadThreads = 1;
int32_t tsNumOfVnodeQueryThreads = 4;
int32_t tsNumOfVnodeStreamThreads = 2;
int32_t tsNumOfVnodeFetchThreads = 4;
int32_t tsNumOfVnodeWriteThreads = 2;
int32_t tsNumOfVnodeSyncThreads = 2;
int32_t tsNumOfVnodeRsmaThreads = 2;
int32_t tsNumOfQnodeQueryThreads = 4;
int32_t tsNumOfQnodeFetchThreads = 1;
@ -374,14 +372,6 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
tsNumOfVnodeFetchThreads = TMAX(tsNumOfVnodeFetchThreads, 4);
if (cfgAddInt32(pCfg, "numOfVnodeFetchThreads", tsNumOfVnodeFetchThreads, 4, 1024, 0) != 0) return -1;
tsNumOfVnodeWriteThreads = tsNumOfCores;
tsNumOfVnodeWriteThreads = TMAX(tsNumOfVnodeWriteThreads, 1);
if (cfgAddInt32(pCfg, "numOfVnodeWriteThreads", tsNumOfVnodeWriteThreads, 1, 1024, 0) != 0) return -1;
tsNumOfVnodeSyncThreads = tsNumOfCores * 2;
tsNumOfVnodeSyncThreads = TMAX(tsNumOfVnodeSyncThreads, 16);
if (cfgAddInt32(pCfg, "numOfVnodeSyncThreads", tsNumOfVnodeSyncThreads, 1, 1024, 0) != 0) return -1;
tsNumOfVnodeRsmaThreads = tsNumOfCores;
tsNumOfVnodeRsmaThreads = TMAX(tsNumOfVnodeRsmaThreads, 4);
if (cfgAddInt32(pCfg, "numOfVnodeRsmaThreads", tsNumOfVnodeRsmaThreads, 1, 1024, 0) != 0) return -1;
@ -506,22 +496,6 @@ static int32_t taosUpdateServerCfg(SConfig *pCfg) {
pItem->stype = stype;
}
pItem = cfgGetItem(tsCfg, "numOfVnodeWriteThreads");
if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
tsNumOfVnodeWriteThreads = numOfCores;
tsNumOfVnodeWriteThreads = TMAX(tsNumOfVnodeWriteThreads, 1);
pItem->i32 = tsNumOfVnodeWriteThreads;
pItem->stype = stype;
}
pItem = cfgGetItem(tsCfg, "numOfVnodeSyncThreads");
if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
tsNumOfVnodeSyncThreads = numOfCores * 2;
tsNumOfVnodeSyncThreads = TMAX(tsNumOfVnodeSyncThreads, 16);
pItem->i32 = tsNumOfVnodeSyncThreads;
pItem->stype = stype;
}
pItem = cfgGetItem(tsCfg, "numOfVnodeRsmaThreads");
if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
tsNumOfVnodeRsmaThreads = numOfCores;
@ -699,8 +673,6 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
tsNumOfVnodeQueryThreads = cfgGetItem(pCfg, "numOfVnodeQueryThreads")->i32;
tsNumOfVnodeStreamThreads = cfgGetItem(pCfg, "numOfVnodeStreamThreads")->i32;
tsNumOfVnodeFetchThreads = cfgGetItem(pCfg, "numOfVnodeFetchThreads")->i32;
tsNumOfVnodeWriteThreads = cfgGetItem(pCfg, "numOfVnodeWriteThreads")->i32;
tsNumOfVnodeSyncThreads = cfgGetItem(pCfg, "numOfVnodeSyncThreads")->i32;
tsNumOfVnodeRsmaThreads = cfgGetItem(pCfg, "numOfVnodeRsmaThreads")->i32;
tsNumOfQnodeQueryThreads = cfgGetItem(pCfg, "numOfQnodeQueryThreads")->i32;
// tsNumOfQnodeFetchThreads = cfgGetItem(pCfg, "numOfQnodeFetchThreads")->i32;
@ -786,6 +758,9 @@ int32_t taosSetCfg(SConfig *pCfg, char *name) {
case 'd': {
if (strcasecmp("dDebugFlag", name) == 0) {
dDebugFlag = cfgGetItem(pCfg, "dDebugFlag")->i32;
} else if (strcasecmp("debugFlag", name) == 0) {
int32_t flag = cfgGetItem(pCfg, "debugFlag")->i32;
taosSetAllDebugFlag(flag, true);
}
break;
}
@ -943,10 +918,6 @@ int32_t taosSetCfg(SConfig *pCfg, char *name) {
} else if (strcasecmp("numOfVnodeFetchThreads", name) == 0) {
tsNumOfVnodeFetchThreads = cfgGetItem(pCfg, "numOfVnodeFetchThreads")->i32;
*/
} else if (strcasecmp("numOfVnodeWriteThreads", name) == 0) {
tsNumOfVnodeWriteThreads = cfgGetItem(pCfg, "numOfVnodeWriteThreads")->i32;
} else if (strcasecmp("numOfVnodeSyncThreads", name) == 0) {
tsNumOfVnodeSyncThreads = cfgGetItem(pCfg, "numOfVnodeSyncThreads")->i32;
} else if (strcasecmp("numOfVnodeRsmaThreads", name) == 0) {
tsNumOfVnodeRsmaThreads = cfgGetItem(pCfg, "numOfVnodeRsmaThreads")->i32;
} else if (strcasecmp("numOfQnodeQueryThreads", name) == 0) {

4
source/common/src/trow.c
View File

@ -1065,8 +1065,8 @@ void tdSTSRowIterInit(STSRowIter *pIter, STSchema *pSchema) {
void tTSRowGetVal(STSRow *pRow, STSchema *pTSchema, int16_t iCol, SColVal *pColVal) {
STColumn *pTColumn = &pTSchema->columns[iCol];
SCellVal cv;
SValue value;
SCellVal cv = {0};
SValue value = {0};
ASSERT((pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID) || (iCol > 0));

3
source/dnode/mgmt/mgmt_snode/src/smWorker.c
View File

@ -140,6 +140,9 @@ int32_t smPutMsgToQueue(SSnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
if (pSnode == NULL) {
dError("snode: msg:%p failed to put into vnode queue since %s, type:%s qtype:%d", pMsg, terrstr(),
TMSG_INFO(pMsg->msgType), qtype);
taosFreeQitem(pMsg);
rpcFreeCont(pRpc->pCont);
pRpc->pCont = NULL;
return -1;
}

30
source/dnode/mgmt/mgmt_vnode/inc/vmInt.h
View File

@ -33,10 +33,6 @@ typedef struct SVnodeMgmt {
SQWorkerPool queryPool;
SQWorkerPool streamPool;
SWWorkerPool fetchPool;
SWWorkerPool syncPool;
SWWorkerPool syncCtrlPool;
SWWorkerPool writePool;
SWWorkerPool applyPool;
SSingleWorker mgmtWorker;
SHashObj *hash;
TdThreadRwlock lock;
@ -52,19 +48,19 @@ typedef struct {
} SWrapperCfg;
typedef struct {
int32_t vgId;
int32_t vgVersion;
int32_t refCount;
int8_t dropped;
char *path;
SVnode *pImpl;
STaosQueue *pWriteQ;
STaosQueue *pSyncQ;
STaosQueue *pSyncCtrlQ;
STaosQueue *pApplyQ;
STaosQueue *pQueryQ;
STaosQueue *pStreamQ;
STaosQueue *pFetchQ;
int32_t vgId;
int32_t vgVersion;
int32_t refCount;
int8_t dropped;
char *path;
SVnode *pImpl;
SMultiWorker pWriteW;
SMultiWorker pSyncW;
SMultiWorker pSyncCtrlW;
SMultiWorker pApplyW;
STaosQueue *pQueryQ;
STaosQueue *pStreamQ;
STaosQueue *pFetchQ;
} SVnodeObj;
typedef struct {

40
source/dnode/mgmt/mgmt_vnode/src/vmInt.c
View File

@ -79,29 +79,49 @@ int32_t vmOpenVnode(SVnodeMgmt *pMgmt, SWrapperCfg *pCfg, SVnode *pImpl) {
void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
char path[TSDB_FILENAME_LEN] = {0};
vnodePreClose(pVnode->pImpl);
taosThreadRwlockWrlock(&pMgmt->lock);
taosHashRemove(pMgmt->hash, &pVnode->vgId, sizeof(int32_t));
taosThreadRwlockUnlock(&pMgmt->lock);
vmReleaseVnode(pMgmt, pVnode);
dTrace("vgId:%d, wait for vnode ref become 0", pVnode->vgId);
while (pVnode->refCount > 0) taosMsleep(10);
dTrace("vgId:%d, wait for vnode queue is empty", pVnode->vgId);
dInfo("vgId:%d, pre close", pVnode->vgId);
vnodePreClose(pVnode->pImpl);
while (!taosQueueEmpty(pVnode->pWriteQ)) taosMsleep(10);
while (!taosQueueEmpty(pVnode->pSyncQ)) taosMsleep(10);
while (!taosQueueEmpty(pVnode->pApplyQ)) taosMsleep(10);
dInfo("vgId:%d, wait for vnode ref become 0", pVnode->vgId);
while (pVnode->refCount > 0) taosMsleep(10);
dInfo("vgId:%d, wait for vnode write queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pWriteW.queue,
pVnode->pWriteW.queue->threadId);
tMultiWorkerCleanup(&pVnode->pWriteW);
dInfo("vgId:%d, wait for vnode sync queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pSyncW.queue,
pVnode->pSyncW.queue->threadId);
tMultiWorkerCleanup(&pVnode->pSyncW);
dInfo("vgId:%d, wait for vnode sync ctrl queue:%p is empty, thread:%08" PRId64, pVnode->vgId,
pVnode->pSyncCtrlW.queue, pVnode->pSyncCtrlW.queue->threadId);
tMultiWorkerCleanup(&pVnode->pSyncCtrlW);
dInfo("vgId:%d, wait for vnode apply queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pApplyW.queue,
pVnode->pApplyW.queue->threadId);
tMultiWorkerCleanup(&pVnode->pApplyW);
dInfo("vgId:%d, wait for vnode query queue:%p is empty", pVnode->vgId, pVnode->pQueryQ);
while (!taosQueueEmpty(pVnode->pQueryQ)) taosMsleep(10);
dInfo("vgId:%d, wait for vnode fetch queue:%p is empty, thread:%08" PRId64, pVnode->vgId, pVnode->pFetchQ,
pVnode->pFetchQ->threadId);
while (!taosQueueEmpty(pVnode->pFetchQ)) taosMsleep(10);
dInfo("vgId:%d, wait for vnode stream queue:%p is empty", pVnode->vgId, pVnode->pStreamQ);
while (!taosQueueEmpty(pVnode->pStreamQ)) taosMsleep(10);
dTrace("vgId:%d, vnode queue is empty", pVnode->vgId);
dInfo("vgId:%d, all vnode queues is empty", pVnode->vgId);
vmFreeQueue(pMgmt, pVnode);
vnodeClose(pVnode->pImpl);
pVnode->pImpl = NULL;
dDebug("vgId:%d, vnode is closed", pVnode->vgId);
dInfo("vgId:%d, vnode is closed", pVnode->vgId);
if (pVnode->dropped) {
dInfo("vgId:%d, vnode is destroyed, dropped:%d", pVnode->vgId, pVnode->dropped);

93
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
View File

@ -145,6 +145,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
pHead->contLen = ntohl(pHead->contLen);
pHead->vgId = ntohl(pHead->vgId);
pHead->msgMask = ntohl(pHead->msgMask);
SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId);
if (pVnode == NULL) {
@ -155,7 +156,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
switch (qtype) {
case QUERY_QUEUE:
if ((pMsg->msgType == TDMT_SCH_QUERY) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
if ((pMsg->msgType == TDMT_SCH_QUERY) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS) && !TEST_SHOW_REWRITE_MASK(pHead->msgMask)) {
terrno = TSDB_CODE_GRANT_EXPIRED;
code = terrno;
dDebug("vgId:%d, msg:%p put into vnode-query queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
@ -188,30 +189,20 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
dDebug("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr(code));
} else {
dGTrace("vgId:%d, msg:%p put into vnode-write queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pWriteQ, pMsg);
#if 0 // tests for batch writes
if (pMsg->msgType == TDMT_VND_CREATE_TABLE) {
SRpcMsg *pDup = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM);
memcpy(pDup, pMsg, sizeof(SRpcMsg));
pDup->pCont = rpcMallocCont(pMsg->contLen);
memcpy(pDup->pCont, pMsg->pCont, pMsg->contLen);
pDup->info.handle = NULL;
taosWriteQitem(pVnode->pWriteQ, pDup);
}
#endif
taosWriteQitem(pVnode->pWriteW.queue, pMsg);
}
break;
case SYNC_QUEUE:
dGTrace("vgId:%d, msg:%p put into vnode-sync queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pSyncQ, pMsg);
taosWriteQitem(pVnode->pSyncW.queue, pMsg);
break;
case SYNC_CTRL_QUEUE:
dGTrace("vgId:%d, msg:%p put into vnode-sync-ctrl queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pSyncCtrlQ, pMsg);
taosWriteQitem(pVnode->pSyncCtrlW.queue, pMsg);
break;
case APPLY_QUEUE:
dGTrace("vgId:%d, msg:%p put into vnode-apply queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pApplyQ, pMsg);
taosWriteQitem(pVnode->pApplyW.queue, pMsg);
break;
default:
code = -1;
@ -276,13 +267,13 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) {
if (pVnode != NULL) {
switch (qtype) {
case WRITE_QUEUE:
size = taosQueueItemSize(pVnode->pWriteQ);
size = taosQueueItemSize(pVnode->pWriteW.queue);
break;
case SYNC_QUEUE:
size = taosQueueItemSize(pVnode->pSyncQ);
size = taosQueueItemSize(pVnode->pSyncW.queue);
break;
case APPLY_QUEUE:
size = taosQueueItemSize(pVnode->pApplyQ);
size = taosQueueItemSize(pVnode->pApplyW.queue);
break;
case QUERY_QUEUE:
size = taosQueueItemSize(pVnode->pQueryQ);
@ -306,40 +297,44 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) {
}
int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode->pImpl, (FItems)vnodeProposeWriteMsg);
pVnode->pSyncQ = tWWorkerAllocQueue(&pMgmt->syncPool, pVnode, (FItems)vmProcessSyncQueue);
pVnode->pSyncCtrlQ = tWWorkerAllocQueue(&pMgmt->syncCtrlPool, pVnode, (FItems)vmProcessSyncQueue);
pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->applyPool, pVnode->pImpl, (FItems)vnodeApplyWriteMsg);
SMultiWorkerCfg wcfg = {.max = 1, .name = "vnode-write", .fp = (FItems)vnodeProposeWriteMsg, .param = pVnode->pImpl};
SMultiWorkerCfg scfg = {.max = 1, .name = "vnode-sync", .fp = (FItems)vmProcessSyncQueue, .param = pVnode};
SMultiWorkerCfg sccfg = {.max = 1, .name = "vnode-sync-ctrl", .fp = (FItems)vmProcessSyncQueue, .param = pVnode};
SMultiWorkerCfg acfg = {.max = 1, .name = "vnode-apply", .fp = (FItems)vnodeApplyWriteMsg, .param = pVnode->pImpl};
(void)tMultiWorkerInit(&pVnode->pWriteW, &wcfg);
(void)tMultiWorkerInit(&pVnode->pSyncW, &scfg);
(void)tMultiWorkerInit(&pVnode->pSyncCtrlW, &sccfg);
(void)tMultiWorkerInit(&pVnode->pApplyW, &acfg);
pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue);
pVnode->pStreamQ = tQWorkerAllocQueue(&pMgmt->streamPool, pVnode, (FItem)vmProcessStreamQueue);
pVnode->pFetchQ = tWWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItems)vmProcessFetchQueue);
if (pVnode->pWriteQ == NULL || pVnode->pSyncQ == NULL || pVnode->pApplyQ == NULL || pVnode->pQueryQ == NULL ||
pVnode->pStreamQ == NULL || pVnode->pFetchQ == NULL) {
if (pVnode->pWriteW.queue == NULL || pVnode->pSyncW.queue == NULL || pVnode->pSyncCtrlW.queue == NULL ||
pVnode->pApplyW.queue == NULL || pVnode->pQueryQ == NULL || pVnode->pStreamQ == NULL || pVnode->pFetchQ == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
return -1;
}
dDebug("vgId:%d, write-queue:%p is alloced", pVnode->vgId, pVnode->pWriteQ);
dDebug("vgId:%d, sync-queue:%p is alloced", pVnode->vgId, pVnode->pSyncQ);
dDebug("vgId:%d, apply-queue:%p is alloced", pVnode->vgId, pVnode->pApplyQ);
dDebug("vgId:%d, query-queue:%p is alloced", pVnode->vgId, pVnode->pQueryQ);
dDebug("vgId:%d, stream-queue:%p is alloced", pVnode->vgId, pVnode->pStreamQ);
dDebug("vgId:%d, fetch-queue:%p is alloced", pVnode->vgId, pVnode->pFetchQ);
dInfo("vgId:%d, write-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pWriteW.queue,
pVnode->pWriteW.queue->threadId);
dInfo("vgId:%d, sync-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pSyncW.queue,
pVnode->pSyncW.queue->threadId);
dInfo("vgId:%d, sync-ctrl-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pSyncCtrlW.queue,
pVnode->pSyncCtrlW.queue->threadId);
dInfo("vgId:%d, apply-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pApplyW.queue,
pVnode->pApplyW.queue->threadId);
dInfo("vgId:%d, query-queue:%p is alloced", pVnode->vgId, pVnode->pQueryQ);
dInfo("vgId:%d, fetch-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pFetchQ,
pVnode->pFetchQ->threadId);
dInfo("vgId:%d, stream-queue:%p is alloced", pVnode->vgId, pVnode->pStreamQ);
return 0;
}
void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pWriteQ);
tWWorkerFreeQueue(&pMgmt->applyPool, pVnode->pApplyQ);
tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ);
tWWorkerFreeQueue(&pMgmt->syncCtrlPool, pVnode->pSyncCtrlQ);
tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ);
tQWorkerFreeQueue(&pMgmt->streamPool, pVnode->pStreamQ);
tWWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ);
pVnode->pWriteQ = NULL;
pVnode->pSyncQ = NULL;
pVnode->pApplyQ = NULL;
pVnode->pQueryQ = NULL;
pVnode->pStreamQ = NULL;
pVnode->pFetchQ = NULL;
@ -364,26 +359,6 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
pFPool->max = tsNumOfVnodeFetchThreads;
if (tWWorkerInit(pFPool) != 0) return -1;
SWWorkerPool *pWPool = &pMgmt->writePool;
pWPool->name = "vnode-write";
pWPool->max = tsNumOfVnodeWriteThreads;
if (tWWorkerInit(pWPool) != 0) return -1;
SWWorkerPool *pAPool = &pMgmt->applyPool;
pAPool->name = "vnode-apply";
pAPool->max = tsNumOfVnodeWriteThreads;
if (tWWorkerInit(pAPool) != 0) return -1;
SWWorkerPool *pSPool = &pMgmt->syncPool;
pSPool->name = "vnode-sync";
pSPool->max = tsNumOfVnodeSyncThreads;
if (tWWorkerInit(pSPool) != 0) return -1;
SWWorkerPool *pSCPool = &pMgmt->syncCtrlPool;
pSCPool->name = "vnode-sync-ctrl";
pSCPool->max = tsNumOfVnodeSyncThreads;
if (tWWorkerInit(pSCPool) != 0) return -1;
SSingleWorkerCfg mgmtCfg = {
.min = 1,
.max = 1,
@ -398,10 +373,6 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
}
void vmStopWorker(SVnodeMgmt *pMgmt) {
tWWorkerCleanup(&pMgmt->writePool);
tWWorkerCleanup(&pMgmt->applyPool);
tWWorkerCleanup(&pMgmt->syncPool);
tWWorkerCleanup(&pMgmt->syncCtrlPool);
tQWorkerCleanup(&pMgmt->queryPool);
tQWorkerCleanup(&pMgmt->streamPool);
tWWorkerCleanup(&pMgmt->fetchPool);

15
source/dnode/mnode/impl/src/mndShow.c
View File

@ -240,25 +240,22 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) {
return -1;
}
int32_t numOfCols = pShow->pMeta->numOfColumns;
SSDataBlock *pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));
pBlock->pDataBlock = taosArrayInit(numOfCols, sizeof(SColumnInfoData));
int32_t numOfCols = pShow->pMeta->numOfColumns;
SSDataBlock *pBlock = createDataBlock();
for (int32_t i = 0; i < numOfCols; ++i) {
SColumnInfoData idata = {0};
SSchema *p = &pShow->pMeta->pSchemas[i];
SSchema *p = &pShow->pMeta->pSchemas[i];
idata.info.bytes = p->bytes;
idata.info.type = p->type;
idata.info.colId = p->colId;
taosArrayPush(pBlock->pDataBlock, &idata);
if (IS_VAR_DATA_TYPE(p->type)) {
pBlock->info.hasVarCol = true;
}
blockDataAppendColInfo(pBlock, &idata);
}
blockDataEnsureCapacity(pBlock, rowsToRead);
if (mndCheckRetrieveFinished(pShow)) {
mDebug("show:0x%" PRIx64 ", read finished, numOfRows:%d", pShow->id, pShow->numOfRows);
rowsRead = 0;

3
source/dnode/mnode/impl/src/mndStb.c
View File

@ -466,7 +466,7 @@ static void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pSt
contLen += sizeof(SMsgHead);
SMsgHead *pHead = taosMemoryMalloc(contLen);
SMsgHead *pHead = taosMemoryCalloc(1, contLen);
if (pHead == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _err;
@ -519,6 +519,7 @@ static void *mndBuildVDropStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb,
pHead->contLen = htonl(contLen);
pHead->vgId = htonl(pVgroup->vgId);
pHead->msgMask = htonl(0);
void *pBuf = POINTER_SHIFT(pHead, sizeof(SMsgHead));

2
source/dnode/mnode/impl/src/mndSync.c
View File

@ -282,6 +282,8 @@ int32_t mndSyncPropose(SMnode *pMnode, SSdbRaw *pRaw, int32_t transId) {
pMgmt->errCode = 0;
SRpcMsg req = {.msgType = TDMT_MND_APPLY_MSG, .contLen = sdbGetRawTotalSize(pRaw)};
if (req.contLen <= 0) return -1;
req.pCont = rpcMallocCont(req.contLen);
if (req.pCont == NULL) return -1;
memcpy(req.pCont, pRaw, req.contLen);

2
source/dnode/mnode/impl/src/mndTrans.c
View File

@ -375,7 +375,7 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
if (taosArrayPush(pTrans->undoActions, &action) == NULL) goto _OVER;
action.pCont = NULL;
} else {
if (taosArrayPush(pTrans->redoActions, &action) == NULL) goto _OVER;
if (taosArrayPush(pTrans->undoActions, &action) == NULL) goto _OVER;
}
}

8
source/dnode/mnode/impl/src/mndVgroup.c
View File

@ -325,10 +325,10 @@ static void *mndBuildAlterVnodeConfigReq(SMnode *pMnode, SDbObj *pDb, SVgObj *pV
static void *mndBuildAlterVnodeReplicaReq(SMnode *pMnode, SDbObj *pDb, SVgObj *pVgroup, int32_t dnodeId,
int32_t *pContLen) {
SAlterVnodeReplicaReq alterReq = {
alterReq.vgId = pVgroup->vgId,
alterReq.strict = pDb->cfg.strict,
alterReq.replica = pVgroup->replica,
alterReq.selfIndex = -1,
.vgId = pVgroup->vgId,
.strict = pDb->cfg.strict,
.replica = pVgroup->replica,
.selfIndex = -1,
};
for (int32_t v = 0; v < pVgroup->replica; ++v) {

7
source/dnode/vnode/src/inc/tsdb.h
View File

@ -644,6 +644,13 @@ typedef struct SSttBlockLoadInfo {
int16_t *colIds;
int32_t numOfCols;
bool sttBlockLoaded;
// keep the last access position, this position may be used to reduce the binary times for
// starting last block data for a new table
struct {
int32_t blockIndex;
int32_t rowIndex;
} prevEndPos;
} SSttBlockLoadInfo;
typedef struct SMergeTree {

1
source/dnode/vnode/src/inc/vnd.h
View File

@ -97,6 +97,7 @@ bool vnodeShouldRollback(SVnode* pVnode);
// vnodeSync.c
int32_t vnodeSyncOpen(SVnode* pVnode, char* path);
void vnodeSyncStart(SVnode* pVnode);
void vnodeSyncPreClose(SVnode* pVnode);
void vnodeSyncClose(SVnode* pVnode);
void vnodeRedirectRpcMsg(SVnode* pVnode, SRpcMsg* pMsg);
bool vnodeIsLeader(SVnode* pVnode);

1
source/dnode/vnode/src/tsdb/tsdbRead.c
View File

@ -544,7 +544,6 @@ static SSDataBlock* createResBlock(SQueryTableDataCond* pCond, int32_t capacity)
taosMemoryFree(pResBlock);
return NULL;
}
return pResBlock;
}

7
source/dnode/vnode/src/vnd/vnodeOpen.c
View File

@ -242,12 +242,7 @@ _err:
return NULL;
}
void vnodePreClose(SVnode *pVnode) {
if (pVnode) {
syncLeaderTransfer(pVnode->sync);
syncPreStop(pVnode->sync);
}
}
void vnodePreClose(SVnode *pVnode) { vnodeSyncPreClose(pVnode); }
void vnodeClose(SVnode *pVnode) {
if (pVnode) {

66
source/dnode/vnode/src/vnd/vnodeSync.c
View File

@ -342,52 +342,26 @@ static void vnodeSyncRollBackMsg(const SSyncFSM *pFsm, const SRpcMsg *pMsg, cons
TMSG_INFO(pMsg->msgType));
}
#define USE_TSDB_SNAPSHOT
static int32_t vnodeSnapshotStartRead(const SSyncFSM *pFsm, void *pParam, void **ppReader) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
SSnapshotParam *pSnapshotParam = pParam;
int32_t code = vnodeSnapReaderOpen(pVnode, pSnapshotParam->start, pSnapshotParam->end, (SVSnapReader **)ppReader);
return code;
#else
*ppReader = taosMemoryMalloc(32);
return 0;
#endif
}
static int32_t vnodeSnapshotStopRead(const SSyncFSM *pFsm, void *pReader) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
int32_t code = vnodeSnapReaderClose(pReader);
return code;
#else
taosMemoryFree(pReader);
return 0;
#endif
}
static int32_t vnodeSnapshotDoRead(const SSyncFSM *pFsm, void *pReader, void **ppBuf, int32_t *len) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
int32_t code = vnodeSnapRead(pReader, (uint8_t **)ppBuf, len);
return code;
#else
static int32_t times = 0;
if (times++ < 5) {
*len = 64;
*ppBuf = taosMemoryMalloc(*len);
snprintf(*ppBuf, *len, "snapshot block %d", times);
} else {
*len = 0;
*ppBuf = NULL;
}
return 0;
#endif
}
static int32_t vnodeSnapshotStartWrite(const SSyncFSM *pFsm, void *pParam, void **ppWriter) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
SSnapshotParam *pSnapshotParam = pParam;
@ -404,14 +378,9 @@ static int32_t vnodeSnapshotStartWrite(const SSyncFSM *pFsm, void *pParam, void
int32_t code = vnodeSnapWriterOpen(pVnode, pSnapshotParam->start, pSnapshotParam->end, (SVSnapWriter **)ppWriter);
return code;
#else
*ppWriter = taosMemoryMalloc(32);
return 0;
#endif
}
static int32_t vnodeSnapshotStopWrite(const SSyncFSM *pFsm, void *pWriter, bool isApply, SSnapshot *pSnapshot) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
vInfo("vgId:%d, stop write vnode snapshot, apply:%d, index:%" PRId64 " term:%" PRIu64 " config:%" PRId64,
pVnode->config.vgId, isApply, pSnapshot->lastApplyIndex, pSnapshot->lastApplyTerm, pSnapshot->lastConfigIndex);
@ -419,22 +388,14 @@ static int32_t vnodeSnapshotStopWrite(const SSyncFSM *pFsm, void *pWriter, bool
int32_t code = vnodeSnapWriterClose(pWriter, !isApply, pSnapshot);
vInfo("vgId:%d, apply vnode snapshot finished, code:0x%x", pVnode->config.vgId, code);
return code;
#else
taosMemoryFree(pWriter);
return 0;
#endif
}
static int32_t vnodeSnapshotDoWrite(const SSyncFSM *pFsm, void *pWriter, void *pBuf, int32_t len) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
vDebug("vgId:%d, continue write vnode snapshot, len:%d", pVnode->config.vgId, len);
int32_t code = vnodeSnapWrite(pWriter, pBuf, len);
vDebug("vgId:%d, continue write vnode snapshot finished, len:%d", pVnode->config.vgId, len);
return code;
#else
return 0;
#endif
}
static void vnodeRestoreFinish(const SSyncFSM *pFsm) {
@ -461,7 +422,6 @@ static void vnodeBecomeFollower(const SSyncFSM *pFsm) {
SVnode *pVnode = pFsm->data;
vDebug("vgId:%d, become follower", pVnode->config.vgId);
// clear old leader resource
taosThreadMutexLock(&pVnode->lock);
if (pVnode->blocked) {
pVnode->blocked = false;
@ -474,15 +434,6 @@ static void vnodeBecomeFollower(const SSyncFSM *pFsm) {
static void vnodeBecomeLeader(const SSyncFSM *pFsm) {
SVnode *pVnode = pFsm->data;
vDebug("vgId:%d, become leader", pVnode->config.vgId);
#if 0
taosThreadMutexLock(&pVnode->lock);
if (pVnode->blocked) {
pVnode->blocked = false;
tsem_post(&pVnode->syncSem);
}
taosThreadMutexUnlock(&pVnode->lock);
#endif
}
static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) {
@ -543,12 +494,25 @@ int32_t vnodeSyncOpen(SVnode *pVnode, char *path) {
}
void vnodeSyncStart(SVnode *pVnode) {
vDebug("vgId:%d, start sync", pVnode->config.vgId);
vInfo("vgId:%d, start sync", pVnode->config.vgId);
syncStart(pVnode->sync);
}
void vnodeSyncPreClose(SVnode *pVnode) {
vInfo("vgId:%d, pre close sync", pVnode->config.vgId);
syncLeaderTransfer(pVnode->sync);
syncPreStop(pVnode->sync);
taosThreadMutexLock(&pVnode->lock);
if (pVnode->blocked) {
vInfo("vgId:%d, post block after close sync", pVnode->config.vgId);
pVnode->blocked = false;
tsem_post(&pVnode->syncSem);
}
taosThreadMutexUnlock(&pVnode->lock);
}
void vnodeSyncClose(SVnode *pVnode) {
vDebug("vgId:%d, close sync", pVnode->config.vgId);
vInfo("vgId:%d, close sync", pVnode->config.vgId);
syncStop(pVnode->sync);
}

15
source/libs/executor/inc/executorimpl.h
View File

@ -533,10 +533,10 @@ typedef struct SSysTableScanInfo {
} SSysTableScanInfo;
typedef struct SBlockDistInfo {
SSDataBlock* pResBlock;
STsdbReader* pHandle;
SReadHandle readHandle;
uint64_t uid; // table uid
SSDataBlock* pResBlock;
STsdbReader* pHandle;
SReadHandle readHandle;
uint64_t uid; // table uid
} SBlockDistInfo;
// todo remove this
@ -547,7 +547,6 @@ typedef struct SOptrBasicInfo {
} SOptrBasicInfo;
typedef struct SIntervalAggOperatorInfo {
// SOptrBasicInfo should be first, SAggSupporter should be second for stream encode
SOptrBasicInfo binfo; // basic info
SAggSupporter aggSup; // aggregate supporter
SExprSupp scalarSupp; // supporter for perform scalar function
@ -819,10 +818,6 @@ typedef struct SSortOperatorInfo {
SLimitInfo limitInfo;
} SSortOperatorInfo;
typedef struct STagFilterOperatorInfo {
SOptrBasicInfo binfo;
} STagFilterOperatorInfo;
typedef struct SJoinOperatorInfo {
SSDataBlock* pRes;
int32_t joinType;
@ -887,7 +882,7 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, SExprInfo* pPseudoExpr, int
void cleanupAggSup(SAggSupporter* pAggSup);
void appendOneRowToDataBlock(SSDataBlock* pBlock, STupleHandle* pTupleHandle);
void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData* pColInfoData, int32_t functionId);
void setTbNameColData(const SSDataBlock* pBlock, SColumnInfoData* pColInfoData, int32_t functionId, const char* name);
int32_t doPrepareScan(SOperatorInfo* pOperator, uint64_t uid, int64_t ts);
int32_t doGetScanStatus(SOperatorInfo* pOperator, uint64_t* uid, int64_t* ts);

14
source/libs/executor/src/cachescanoperator.c
View File

@ -170,14 +170,12 @@ SSDataBlock* doScanCache(SOperatorInfo* pOperator) {
pRes->info.uid = *(tb_uid_t*)taosArrayGet(pInfo->pUidList, pInfo->indexOfBufferedRes);
pRes->info.rows = 1;
if (pInfo->pseudoExprSup.numOfExprs > 0) {
SExprSupp* pSup = &pInfo->pseudoExprSup;
int32_t code = addTagPseudoColumnData(&pInfo->readHandle, pSup->pExprInfo, pSup->numOfExprs, pRes, pRes->info.rows,
GET_TASKID(pTaskInfo));
if (code != TSDB_CODE_SUCCESS) {
pTaskInfo->code = code;
return NULL;
}
SExprSupp* pSup = &pInfo->pseudoExprSup;
int32_t code = addTagPseudoColumnData(&pInfo->readHandle, pSup->pExprInfo, pSup->numOfExprs, pRes,
pRes->info.rows, GET_TASKID(pTaskInfo));
if (code != TSDB_CODE_SUCCESS) {
pTaskInfo->code = code;
return NULL;
}
pRes->info.groupId = getTableGroupId(pTableList, pRes->info.uid);

9
source/libs/executor/src/executil.c
View File

@ -379,7 +379,7 @@ static int32_t createResultData(SDataType* pType, int32_t numOfRows, SScalarPara
pColumnData->info.scale = pType->scale;
pColumnData->info.precision = pType->precision;
int32_t code = colInfoDataEnsureCapacity(pColumnData, numOfRows);
int32_t code = colInfoDataEnsureCapacity(pColumnData, numOfRows, true);
if (code != TSDB_CODE_SUCCESS) {
terrno = code;
taosMemoryFree(pColumnData);
@ -1824,8 +1824,6 @@ static int32_t orderbyGroupIdComparFn(const void* p1, const void* p2) {
}
static int32_t sortTableGroup(STableListInfo* pTableListInfo) {
int32_t code = TSDB_CODE_SUCCESS;
taosArraySort(pTableListInfo->pTableList, orderbyGroupIdComparFn);
int32_t size = taosArrayGetSize(pTableListInfo->pTableList);
@ -1847,6 +1845,11 @@ static int32_t sortTableGroup(STableListInfo* pTableListInfo) {
pTableListInfo->numOfOuputGroups = taosArrayGetSize(pList);
pTableListInfo->groupOffset = taosMemoryMalloc(sizeof(int32_t) * pTableListInfo->numOfOuputGroups);
if (pTableListInfo->groupOffset == NULL) {
taosArrayDestroy(pList);
return TSDB_CODE_OUT_OF_MEMORY;
}
memcpy(pTableListInfo->groupOffset, taosArrayGet(pList, 0), sizeof(int32_t) * pTableListInfo->numOfOuputGroups);
taosArrayDestroy(pList);
return TDB_CODE_SUCCESS;

6
source/libs/executor/src/executorimpl.c
View File

@ -87,7 +87,7 @@ static void releaseQueryBuf(size_t numOfTables);
static void destroyFillOperatorInfo(void* param);
static void destroyProjectOperatorInfo(void* param);
static void destroyOrderOperatorInfo(void* param);
static void destroySortOperatorInfo(void* param);
static void destroyAggOperatorInfo(void* param);
static void destroyIntervalOperatorInfo(void* param);
@ -322,7 +322,7 @@ void initExecTimeWindowInfo(SColumnInfoData* pColData, STimeWindow* pQueryWindow
pColData->info.type = TSDB_DATA_TYPE_TIMESTAMP;
pColData->info.bytes = sizeof(int64_t);
colInfoDataEnsureCapacity(pColData, 5);
colInfoDataEnsureCapacity(pColData, 5, false);
colDataAppendInt64(pColData, 0, &pQueryWindow->skey);
colDataAppendInt64(pColData, 1, &pQueryWindow->ekey);
@ -439,7 +439,7 @@ static int32_t doCreateConstantValColumnInfo(SInputColumnInfoData* pInput, SFunc
pColInfo = pInput->pData[paramIndex];
}
colInfoDataEnsureCapacity(pColInfo, numOfRows);
colInfoDataEnsureCapacity(pColInfo, numOfRows, false);
int8_t type = pFuncParam->param.nType;
if (type == TSDB_DATA_TYPE_BIGINT || type == TSDB_DATA_TYPE_UBIGINT) {

15
source/libs/executor/src/groupoperator.c
View File

@ -337,8 +337,6 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SGroupbyOperatorInfo* pInfo = pOperator->info;
SSDataBlock* pRes = pInfo->binfo.pRes;
if (pOperator->status == OP_RES_TO_RETURN) {
return buildGroupResultDataBlock(pOperator);
}
@ -391,7 +389,6 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) {
}
}
#endif
blockDataEnsureCapacity(pRes, pOperator->resultInfo.capacity);
initGroupedResultInfo(&pInfo->groupResInfo, pInfo->aggSup.pResultRowHashTable, 0);
pOperator->cost.openCost = (taosGetTimestampUs() - st) / 1000.0;
@ -421,6 +418,8 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SAggPhysiNode*
}
initResultSizeInfo(&pOperator->resultInfo, 4096);
blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity);
code = initGroupOptrInfo(&pInfo->pGroupColVals, &pInfo->groupKeyLen, &pInfo->keyBuf, pInfo->pGroupCols);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
@ -1086,14 +1085,16 @@ SOperatorInfo* createStreamPartitionOperatorInfo(SOperatorInfo* downstream, SStr
}
pInfo->partitionSup.needCalc = true;
SSDataBlock* pResBlock = createResDataBlock(pPartNode->part.node.pOutputDataBlockDesc);
if (!pResBlock) {
pInfo->binfo.pRes = createResDataBlock(pPartNode->part.node.pOutputDataBlockDesc);
if (pInfo->binfo.pRes == NULL) {
goto _error;
}
blockDataEnsureCapacity(pResBlock, 4096);
pInfo->binfo.pRes = pResBlock;
blockDataEnsureCapacity(pInfo->binfo.pRes, 4096);
pInfo->parIte = NULL;
pInfo->pInputDataBlock = NULL;
_hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY);
pInfo->pPartitions = taosHashInit(1024, hashFn, false, HASH_NO_LOCK);
pInfo->tsColIndex = 0;

1
source/libs/executor/src/projectoperator.c
View File

@ -384,6 +384,7 @@ SOperatorInfo* createIndefinitOutputOperatorInfo(SOperatorInfo* downstream, SPhy
initBasicInfo(&pInfo->binfo, pResBlock);
initResultSizeInfo(&pOperator->resultInfo, numOfRows);
blockDataEnsureCapacity(pResBlock, numOfRows);
int32_t code = initAggInfo(pSup, &pInfo->aggSup, pExprInfo, numOfExpr, keyBufSize, pTaskInfo->id.str);
if (code != TSDB_CODE_SUCCESS) {

282
source/libs/executor/src/scanoperator.c
View File

@ -58,6 +58,18 @@ typedef struct {
void* pVnode;
} SSTabFltArg;
typedef struct STableMergeScanExecInfo {
SFileBlockLoadRecorder blockRecorder;
SSortExecInfo sortExecInfo;
} STableMergeScanExecInfo;
typedef struct STableMergeScanSortSourceParam {
SOperatorInfo* pOperator;
int32_t readerIdx;
uint64_t uid;
SSDataBlock* inputBlock;
} STableMergeScanSortSourceParam;
static int32_t sysChkFilter__Comm(SNode* pNode);
static int32_t sysChkFilter__DBName(SNode* pNode);
static int32_t sysChkFilter__VgroupId(SNode* pNode);
@ -69,15 +81,15 @@ static int32_t sysChkFilter__STableName(SNode* pNode);
static int32_t sysChkFilter__Uid(SNode* pNode);
static int32_t sysChkFilter__Type(SNode* pNode);
static int32_t sysFilte__DbName(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__VgroupId(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__TableName(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__CreateTime(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__Ncolumn(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__Ttl(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__STableName(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__Uid(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__Type(void* pMeta, SNode* pNode, SArray* result);
static int32_t sysFilte__DbName(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__VgroupId(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__TableName(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__CreateTime(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__Ncolumn(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__Ttl(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__STableName(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__Uid(void* arg, SNode* pNode, SArray* result);
static int32_t sysFilte__Type(void* arg, SNode* pNode, SArray* result);
const SSTabFltFuncDef filterDict[] = {
{.name = "table_name", .chkFunc = sysChkFilter__TableName, .fltFunc = sysFilte__TableName},
@ -95,7 +107,7 @@ const SSTabFltFuncDef filterDict[] = {
static int32_t optSysTabFilte(void* arg, SNode* cond, SArray* result);
static int32_t optSysTabFilteImpl(void* arg, SNode* cond, SArray* result);
static int32_t optSysCheckOper(SNode* pOpear);
static int32_t optSysMergeRslt(SArray* multiRslt, SArray* reslt);
static int32_t optSysMergeRslt(SArray* mRslt, SArray* rslt);
static bool processBlockWithProbability(const SSampleExecInfo* pInfo);
@ -125,29 +137,6 @@ static void switchCtxOrder(SqlFunctionCtx* pCtx, int32_t numOfOutput) {
}
}
static void setupQueryRangeForReverseScan(STableScanInfo* pTableScanInfo) {
#if 0
int32_t numOfGroups = (int32_t)(GET_NUM_OF_TABLEGROUP(pRuntimeEnv));
for(int32_t i = 0; i < numOfGroups; ++i) {
SArray *group = GET_TABLEGROUP(pRuntimeEnv, i);
SArray *tableKeyGroup = taosArrayGetP(pQueryAttr->tableGroupInfo.pGroupList, i);
size_t t = taosArrayGetSize(group);
for (int32_t j = 0; j < t; ++j) {
STableQueryInfo *pCheckInfo = taosArrayGetP(group, j);
updateTableQueryInfoForReverseScan(pCheckInfo);
// update the last key in tableKeyInfo list, the tableKeyInfo is used to build the tsdbQueryHandle and decide
// the start check timestamp of tsdbQueryHandle
// STableKeyInfo *pTableKeyInfo = taosArrayGet(tableKeyGroup, j);
// pTableKeyInfo->lastKey = pCheckInfo->lastKey;
//
// assert(pCheckInfo->pTable == pTableKeyInfo->pTable);
}
}
#endif
}
static void getNextTimeWindow(SInterval* pInterval, STimeWindow* tw, int32_t order) {
int32_t factor = GET_FORWARD_DIRECTION_FACTOR(order);
if (pInterval->intervalUnit != 'n' && pInterval->intervalUnit != 'y') {
@ -293,7 +282,7 @@ static int32_t doDynamicPruneDataBlock(SOperatorInfo* pOperator, SDataBlockInfo*
return TSDB_CODE_SUCCESS;
}
static FORCE_INLINE bool doFilterByBlockSMA(SFilterInfo* pFilterInfo, SColumnDataAgg** pColsAgg, int32_t numOfCols,
static bool doFilterByBlockSMA(SFilterInfo* pFilterInfo, SColumnDataAgg** pColsAgg, int32_t numOfCols,
int32_t numOfRows) {
if (pColsAgg == NULL || pFilterInfo == NULL) {
return true;
@ -356,13 +345,13 @@ static void doSetTagColumnData(STableScanInfo* pTableScanInfo, SSDataBlock* pBlo
// todo handle the slimit info
void applyLimitOffset(SLimitInfo* pLimitInfo, SSDataBlock* pBlock, SExecTaskInfo* pTaskInfo, SOperatorInfo* pOperator) {
SLimit* pLimit = &pLimitInfo->limit;
const char* id = GET_TASKID(pTaskInfo);
if (pLimit->offset > 0 && pLimitInfo->remainOffset > 0) {
if (pLimitInfo->remainOffset >= pBlock->info.rows) {
pLimitInfo->remainOffset -= pBlock->info.rows;
pBlock->info.rows = 0;
qDebug("current block ignore due to offset, current:%" PRId64 ", %s", pLimitInfo->remainOffset,
GET_TASKID(pTaskInfo));
qDebug("current block ignore due to offset, current:%" PRId64 ", %s", pLimitInfo->remainOffset, id);
} else {
blockDataTrimFirstNRows(pBlock, pLimitInfo->remainOffset);
pLimitInfo->remainOffset = 0;
@ -375,22 +364,11 @@ void applyLimitOffset(SLimitInfo* pLimitInfo, SSDataBlock* pBlock, SExecTaskInfo
int32_t keep = pBlock->info.rows - overflowRows;
blockDataKeepFirstNRows(pBlock, keep);
qDebug("output limit %" PRId64 " has reached, %s", pLimit->limit, GET_TASKID(pTaskInfo));
qDebug("output limit %" PRId64 " has reached, %s", pLimit->limit, id);
pOperator->status = OP_EXEC_DONE;
}
}
static void ensureBlockCapacity(SSDataBlock* pBlock, int32_t capacity) {
// keep the value of rows temporarily
int32_t rows = pBlock->info.rows;
pBlock->info.rows = 0;
blockDataEnsureCapacity(pBlock, capacity);
// restore the rows number
pBlock->info.rows = rows;
}
static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanInfo* pTableScanInfo, SSDataBlock* pBlock,
uint32_t* status) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
@ -421,24 +399,16 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanInfo* pTableSca
} else if (*status == FUNC_DATA_REQUIRED_NOT_LOAD) {
qDebug("%s data block skipped, brange:%" PRId64 "-%" PRId64 ", rows:%d", GET_TASKID(pTaskInfo),
pBlockInfo->window.skey, pBlockInfo->window.ekey, pBlockInfo->rows);
if (pTableScanInfo->pseudoSup.numOfExprs > 0) {
ensureBlockCapacity(pBlock, pBlock->info.rows);
}
doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, 1);
pCost->skipBlocks += 1;
return TSDB_CODE_SUCCESS;
} else if (*status == FUNC_DATA_REQUIRED_STATIS_LOAD) {
} else if (*status == FUNC_DATA_REQUIRED_SMA_LOAD) {
pCost->loadBlockStatis += 1;
loadSMA = true; // mark the operation of load sma;
bool success = doLoadBlockSMA(pTableScanInfo, pBlock, pTaskInfo);
if (success) { // failed to load the block sma data, data block statistics does not exist, load data block instead
qDebug("%s data block SMA loaded, brange:%" PRId64 "-%" PRId64 ", rows:%d", GET_TASKID(pTaskInfo),
pBlockInfo->window.skey, pBlockInfo->window.ekey, pBlockInfo->rows);
if (pTableScanInfo->pseudoSup.numOfExprs > 0) {
ensureBlockCapacity(pBlock, pBlock->info.rows);
}
doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, 1);
return TSDB_CODE_SUCCESS;
} else {
@ -488,7 +458,6 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanInfo* pTableSca
return terrno;
}
ensureBlockCapacity(pBlock, pBlock->info.rows);
relocateColumnData(pBlock, pTableScanInfo->matchInfo.pList, pCols, true);
doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, pBlock->info.rows);
@ -532,7 +501,7 @@ static void prepareForDescendingScan(STableScanInfo* pTableScanInfo, SqlFunction
int32_t addTagPseudoColumnData(SReadHandle* pHandle, SExprInfo* pPseudoExpr, int32_t numOfPseudoExpr,
SSDataBlock* pBlock, int32_t rows, const char* idStr) {
// currently only the tbname pseudo column
if (numOfPseudoExpr == 0) {
if (numOfPseudoExpr <= 0) {
return TSDB_CODE_SUCCESS;
}
@ -562,7 +531,7 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, SExprInfo* pPseudoExpr, int
// this is to handle the tbname
if (fmIsScanPseudoColumnFunc(functionId)) {
setTbNameColData(pHandle->meta, pBlock, pColInfoData, functionId);
setTbNameColData(pBlock, pColInfoData, functionId, mr.me.name);
} else { // these are tags
STagVal tagVal = {0};
tagVal.cid = pExpr->base.pParam[0].pCol->colId;
@ -598,16 +567,20 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, SExprInfo* pPseudoExpr, int
return TSDB_CODE_SUCCESS;
}
void setTbNameColData(void* pMeta, const SSDataBlock* pBlock, SColumnInfoData* pColInfoData, int32_t functionId) {
void setTbNameColData(const SSDataBlock* pBlock, SColumnInfoData* pColInfoData, int32_t functionId, const char* name) {
struct SScalarFuncExecFuncs fpSet = {0};
fmGetScalarFuncExecFuncs(functionId, &fpSet);
SColumnInfoData infoData = createColumnInfoData(TSDB_DATA_TYPE_BIGINT, sizeof(uint64_t), 1);
colInfoDataEnsureCapacity(&infoData, 1);
size_t len = TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE;
char buf[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(buf, name)
colDataAppendInt64(&infoData, 0, (int64_t*)&pBlock->info.uid);
SScalarParam srcParam = {.numOfRows = pBlock->info.rows, .param = pMeta, .columnData = &infoData};
SColumnInfoData infoData = createColumnInfoData(TSDB_DATA_TYPE_VARCHAR, len, 1);
colInfoDataEnsureCapacity(&infoData, 1, false);
colDataAppend(&infoData, 0, buf, false);
SScalarParam srcParam = {.numOfRows = pBlock->info.rows, .columnData = &infoData};
SScalarParam param = {.columnData = pColInfoData};
if (fpSet.process != NULL) {
@ -638,9 +611,13 @@ static SSDataBlock* doTableScanImpl(SOperatorInfo* pOperator) {
}
blockDataCleanup(pBlock);
SDataBlockInfo* pBInfo = &pBlock->info;
tsdbRetrieveDataBlockInfo(pTableScanInfo->dataReader, &pBInfo->rows, &pBInfo->uid, &pBInfo->window);
int32_t rows = 0;
tsdbRetrieveDataBlockInfo(pTableScanInfo->dataReader, &rows, &pBInfo->uid, &pBInfo->window);
blockDataEnsureCapacity(pBlock, rows); // todo remove it latter
pBInfo->rows = rows;
ASSERT(pBInfo->uid != 0);
pBlock->info.groupId = getTableGroupId(pTaskInfo->pTableInfoList, pBlock->info.uid);
@ -675,7 +652,7 @@ static SSDataBlock* doTableScanImpl(SOperatorInfo* pOperator) {
return NULL;
}
static SSDataBlock* doTableScanGroup(SOperatorInfo* pOperator) {
static SSDataBlock* doGroupedTableScan(SOperatorInfo* pOperator) {
STableScanInfo* pTableScanInfo = pOperator->info;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
@ -688,7 +665,6 @@ static SSDataBlock* doTableScanGroup(SOperatorInfo* pOperator) {
while (pTableScanInfo->scanTimes < pTableScanInfo->scanInfo.numOfAsc) {
SSDataBlock* p = doTableScanImpl(pOperator);
if (p != NULL) {
ASSERT(p->info.uid != 0);
return p;
}
@ -742,7 +718,7 @@ static SSDataBlock* doTableScan(SOperatorInfo* pOperator) {
int32_t numOfTables = tableListGetSize(pTaskInfo->pTableInfoList);
while (1) {
SSDataBlock* result = doTableScanGroup(pOperator);
SSDataBlock* result = doGroupedTableScan(pOperator);
if (result) {
return result;
}
@ -780,7 +756,7 @@ static SSDataBlock* doTableScan(SOperatorInfo* pOperator) {
}
}
SSDataBlock* result = doTableScanGroup(pOperator);
SSDataBlock* result = doGroupedTableScan(pOperator);
if (result != NULL) {
ASSERT(result->info.uid != 0);
return result;
@ -804,7 +780,7 @@ static SSDataBlock* doTableScan(SOperatorInfo* pOperator) {
tsdbReaderReset(pInfo->dataReader, &pInfo->cond);
pInfo->scanTimes = 0;
result = doTableScanGroup(pOperator);
result = doGroupedTableScan(pOperator);
if (result != NULL) {
return result;
}
@ -847,24 +823,25 @@ SOperatorInfo* createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode,
goto _error;
}
SDataBlockDescNode* pDescNode = pTableScanNode->scan.node.pOutputDataBlockDesc;
SScanPhysiNode* pScanNode = &pTableScanNode->scan;
SDataBlockDescNode* pDescNode = pScanNode->node.pOutputDataBlockDesc;
int32_t numOfCols = 0;
int32_t code = extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID,
int32_t code = extractColMatchInfo(pScanNode->pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID,
&pInfo->matchInfo);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
initLimitInfo(pTableScanNode->scan.node.pLimit, pTableScanNode->scan.node.pSlimit, &pInfo->limitInfo);
initLimitInfo(pScanNode->node.pLimit, pScanNode->node.pSlimit, &pInfo->limitInfo);
code = initQueryTableDataCond(&pInfo->cond, pTableScanNode);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
if (pTableScanNode->scan.pScanPseudoCols != NULL) {
if (pScanNode->pScanPseudoCols != NULL) {
SExprSupp* pSup = &pInfo->pseudoSup;
pSup->pExprInfo = createExprInfo(pTableScanNode->scan.pScanPseudoCols, NULL, &pSup->numOfExprs);
pSup->pExprInfo = createExprInfo(pScanNode->pScanPseudoCols, NULL, &pSup->numOfExprs);
pSup->pCtx = createSqlFunctionCtx(pSup->pExprInfo, pSup->numOfExprs, &pSup->rowEntryInfoOffset);
}
@ -875,7 +852,11 @@ SOperatorInfo* createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode,
pInfo->sample.seed = taosGetTimestampSec();
pInfo->dataBlockLoadFlag = pTableScanNode->dataRequired;
initResultSizeInfo(&pOperator->resultInfo, 4096);
pInfo->pResBlock = createResDataBlock(pDescNode);
blockDataEnsureCapacity(pInfo->pResBlock, pOperator->resultInfo.capacity);
code = filterInitFromNode((SNode*)pTableScanNode->scan.node.pConditions, &pOperator->exprSupp.pFilterInfo, 0);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
@ -980,7 +961,7 @@ static SSDataBlock* doBlockInfoScan(SOperatorInfo* pOperator) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
STableBlockDistInfo blockDistInfo = {.minRows = INT_MAX, .maxRows = INT_MIN};
int32_t code = doGetTableRowSize(pBlockScanInfo->readHandle.meta, pBlockScanInfo->uid, &blockDistInfo.rowSize,
int32_t code = doGetTableRowSize(pBlockScanInfo->readHandle.meta, pBlockScanInfo->uid, (int32_t*)&blockDistInfo.rowSize,
GET_TASKID(pTaskInfo));
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, code);
@ -999,12 +980,10 @@ static SSDataBlock* doBlockInfoScan(SOperatorInfo* pOperator) {
tSerializeBlockDistInfo(varDataVal(p), len, &blockDistInfo);
varDataSetLen(p, len);
blockDataEnsureCapacity(pBlock, 1);
colDataAppend(pColInfo, 0, p, false);
taosMemoryFree(p);
pBlock->info.rows = 1;
pOperator->status = OP_EXEC_DONE;
return pBlock;
}
@ -1070,7 +1049,9 @@ SOperatorInfo* createDataBlockInfoScanOperator(SReadHandle* readHandle, SBlockDi
pInfo->readHandle = *readHandle;
pInfo->uid = pBlockScanNode->suid;
pInfo->pResBlock = createResDataBlock(pBlockScanNode->node.pOutputDataBlockDesc);
blockDataEnsureCapacity(pInfo->pResBlock, 1);
int32_t numOfCols = 0;
SExprInfo* pExprInfo = createExprInfo(pBlockScanNode->pScanPseudoCols, NULL, &numOfCols);
@ -2422,7 +2403,7 @@ SOperatorInfo* createStreamScanOperatorInfo(SReadHandle* pHandle, STableScanPhys
if (pHandle->initTableReader) {
pTSInfo->scanMode = TABLE_SCAN__TABLE_ORDER;
pTSInfo->dataReader = NULL;
int32_t code = tsdbReaderOpen(pHandle->vnode, &pTSInfo->cond, pList, num, &pTSInfo->dataReader, NULL);
code = tsdbReaderOpen(pHandle->vnode, &pTSInfo->cond, pList, num, &pTSInfo->dataReader, NULL);
if (code != 0) {
terrno = code;
destroyTableScanOperatorInfo(pTableScanOp);
@ -4094,7 +4075,7 @@ SOperatorInfo* createSysTableScanOperatorInfo(void* readHandle, SSystemTableScan
goto _error;
}
SScanPhysiNode* pScanNode = &pScanPhyNode->scan;
SScanPhysiNode* pScanNode = &pScanPhyNode->scan;
SDataBlockDescNode* pDescNode = pScanNode->node.pOutputDataBlockDesc;
int32_t num = 0;
@ -4145,7 +4126,6 @@ _error:
if (pInfo != NULL) {
destroySysScanOperator(pInfo);
}
taosMemoryFreeClear(pOperator);
pTaskInfo->code = code;
return NULL;
@ -4249,16 +4229,15 @@ SOperatorInfo* createTagScanOperatorInfo(SReadHandle* pReadHandle, STagScanPhysi
SDataBlockDescNode* pDescNode = pPhyNode->node.pOutputDataBlockDesc;
int32_t num = 0;
int32_t numOfExprs = 0;
SExprInfo* pExprInfo = createExprInfo(pPhyNode->pScanPseudoCols, NULL, &numOfExprs);
int32_t code =
extractColMatchInfo(pPhyNode->pScanPseudoCols, pDescNode, &num, COL_MATCH_FROM_COL_ID, &pInfo->matchInfo);
int32_t code = initExprSupp(&pOperator->exprSupp, pExprInfo, numOfExprs);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
code = initExprSupp(&pOperator->exprSupp, pExprInfo, numOfExprs);
int32_t num = 0;
code = extractColMatchInfo(pPhyNode->pScanPseudoCols, pDescNode, &num, COL_MATCH_FROM_COL_ID, &pInfo->matchInfo);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
@ -4290,21 +4269,9 @@ _error:
return NULL;
}
int32_t createMultipleDataReaders(SQueryTableDataCond* pQueryCond, SReadHandle* pHandle, STableListInfo* pTableListInfo,
int32_t tableStartIdx, int32_t tableEndIdx, SArray* arrayReader, const char* idstr) {
for (int32_t i = tableStartIdx; i <= tableEndIdx; ++i) {
STableKeyInfo* pList = tableListGetInfo(pTableListInfo, i);
STsdbReader* pReader = NULL;
tsdbReaderOpen(pHandle->vnode, pQueryCond, pList, 1, &pReader, idstr);
taosArrayPush(arrayReader, &pReader);
}
return TSDB_CODE_SUCCESS;
}
// todo refactor
static int32_t loadDataBlockFromOneTable(SOperatorInfo* pOperator, STableMergeScanInfo* pTableScanInfo,
int32_t readerIdx, SSDataBlock* pBlock, uint32_t* status) {
SSDataBlock* pBlock, uint32_t* status) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
STableMergeScanInfo* pInfo = pOperator->info;
@ -4339,12 +4306,11 @@ static int32_t loadDataBlockFromOneTable(SOperatorInfo* pOperator, STableMergeSc
}
return TSDB_CODE_SUCCESS;
} else if (*status == FUNC_DATA_REQUIRED_STATIS_LOAD) {
} else if (*status == FUNC_DATA_REQUIRED_SMA_LOAD) {
pCost->loadBlockStatis += 1;
bool allColumnsHaveAgg = true;
SColumnDataAgg** pColAgg = NULL;
// STsdbReader* reader = pTableScanInfo->pReader; // taosArrayGetP(pTableScanInfo->dataReaders, readerIdx);
if (allColumnsHaveAgg == true) {
int32_t numOfCols = taosArrayGetSize(pBlock->pDataBlock);
@ -4393,13 +4359,12 @@ static int32_t loadDataBlockFromOneTable(SOperatorInfo* pOperator, STableMergeSc
relocateColumnData(pBlock, pTableScanInfo->matchInfo.pList, pCols, true);
// currently only the tbname pseudo column
if (pTableScanInfo->pseudoSup.numOfExprs > 0) {
int32_t code =
addTagPseudoColumnData(&pTableScanInfo->readHandle, pTableScanInfo->pseudoSup.pExprInfo,
pTableScanInfo->pseudoSup.numOfExprs, pBlock, pBlock->info.rows, GET_TASKID(pTaskInfo));
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, code);
}
SExprSupp* pSup = &pTableScanInfo->pseudoSup;
int32_t code = addTagPseudoColumnData(&pTableScanInfo->readHandle, pSup->pExprInfo, pSup->numOfExprs, pBlock,
pBlock->info.rows, GET_TASKID(pTaskInfo));
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, code);
}
if (pOperator->exprSupp.pFilterInfo!= NULL) {
@ -4421,13 +4386,6 @@ static int32_t loadDataBlockFromOneTable(SOperatorInfo* pOperator, STableMergeSc
return TSDB_CODE_SUCCESS;
}
typedef struct STableMergeScanSortSourceParam {
SOperatorInfo* pOperator;
int32_t readerIdx;
uint64_t uid;
SSDataBlock* inputBlock;
} STableMergeScanSortSourceParam;
static SSDataBlock* getTableDataBlockImpl(void* param) {
STableMergeScanSortSourceParam* source = param;
SOperatorInfo* pOperator = source->pOperator;
@ -4438,7 +4396,6 @@ static SSDataBlock* getTableDataBlockImpl(void* param) {
STableMergeScanInfo* pTableScanInfo = pOperator->info;
SQueryTableDataCond* pQueryCond = taosArrayGet(pTableScanInfo->queryConds, readIdx);
blockDataCleanup(pBlock);
int64_t st = taosGetTimestampUs();
@ -4448,13 +4405,13 @@ static SSDataBlock* getTableDataBlockImpl(void* param) {
int32_t code = tsdbReaderOpen(pHandle->vnode, pQueryCond, p, 1, &pInfo->pReader, GET_TASKID(pTaskInfo));
if (code != 0) {
T_LONG_JMP(pOperator->pTaskInfo->env, code);
T_LONG_JMP(pTaskInfo->env, code);
}
STsdbReader* reader = pInfo->pReader;
while (tsdbNextDataBlock(reader)) {
if (isTaskKilled(pOperator->pTaskInfo)) {
T_LONG_JMP(pOperator->pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
if (isTaskKilled(pTaskInfo)) {
T_LONG_JMP(pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
}
// process this data block based on the probabilities
@ -4477,9 +4434,9 @@ static SSDataBlock* getTableDataBlockImpl(void* param) {
}
uint32_t status = 0;
int32_t code = loadDataBlockFromOneTable(pOperator, pTableScanInfo, readIdx, pBlock, &status);
code = loadDataBlockFromOneTable(pOperator, pTableScanInfo, pBlock, &status);
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pOperator->pTaskInfo->env, code);
T_LONG_JMP(pTaskInfo->env, code);
}
// current block is filter out according to filter condition, continue load the next block
@ -4487,71 +4444,21 @@ static SSDataBlock* getTableDataBlockImpl(void* param) {
continue;
}
pBlock->info.groupId = getTableGroupId(pOperator->pTaskInfo->pTableInfoList, pBlock->info.uid);
pBlock->info.groupId = getTableGroupId(pTaskInfo->pTableInfoList, pBlock->info.uid);
pOperator->resultInfo.totalRows += pBlock->info.rows; // pTableScanInfo->readRecorder.totalRows;
pOperator->resultInfo.totalRows += pBlock->info.rows;
pTableScanInfo->readRecorder.elapsedTime += (taosGetTimestampUs() - st) / 1000.0;
tsdbReaderClose(pInfo->pReader);
pInfo->pReader = NULL;
return pBlock;
}
tsdbReaderClose(pInfo->pReader);
pInfo->pReader = NULL;
return NULL;
}
static SSDataBlock* getTableDataBlock(void* param) {
STableMergeScanSortSourceParam* source = param;
SOperatorInfo* pOperator = source->pOperator;
int32_t readerIdx = source->readerIdx;
SSDataBlock* pBlock = source->inputBlock;
STableMergeScanInfo* pTableScanInfo = pOperator->info;
int64_t st = taosGetTimestampUs();
blockDataCleanup(pBlock);
STsdbReader* reader = taosArrayGetP(pTableScanInfo->dataReaders, readerIdx);
while (tsdbNextDataBlock(reader)) {
if (isTaskKilled(pOperator->pTaskInfo)) {
T_LONG_JMP(pOperator->pTaskInfo->env, TSDB_CODE_TSC_QUERY_CANCELLED);
}
// process this data block based on the probabilities
bool processThisBlock = processBlockWithProbability(&pTableScanInfo->sample);
if (!processThisBlock) {
continue;
}
blockDataCleanup(pBlock);
int32_t rows = 0;
tsdbRetrieveDataBlockInfo(reader, &rows, &pBlock->info.uid, &pBlock->info.window);
blockDataEnsureCapacity(pBlock, rows);
pBlock->info.rows = rows;
uint32_t status = 0;
int32_t code = loadDataBlockFromOneTable(pOperator, pTableScanInfo, readerIdx, pBlock, &status);
// int32_t code = loadDataBlockOnDemand(pOperator->pRuntimeEnv, pTableScanInfo, pBlock, &status);
if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pOperator->pTaskInfo->env, code);
}
// current block is filter out according to filter condition, continue load the next block
if (status == FUNC_DATA_REQUIRED_FILTEROUT || pBlock->info.rows == 0) {
continue;
}
pBlock->info.groupId = getTableGroupId(pOperator->pTaskInfo->pTableInfoList, pBlock->info.uid);
pOperator->resultInfo.totalRows = pTableScanInfo->readRecorder.totalRows;
pTableScanInfo->readRecorder.elapsedTime += (taosGetTimestampUs() - st) / 1000.0;
return pBlock;
}
return NULL;
}
SArray* generateSortByTsInfo(SArray* colMatchInfo, int32_t order) {
int32_t tsTargetSlotId = 0;
for (int32_t i = 0; i < taosArrayGetSize(colMatchInfo); ++i) {
@ -4680,7 +4587,6 @@ SSDataBlock* getSortedTableMergeScanBlockData(SSortHandle* pHandle, SSDataBlock*
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
blockDataCleanup(pResBlock);
blockDataEnsureCapacity(pResBlock, capacity);
while (1) {
STupleHandle* pTupleHandle = tsortNextTuple(pHandle);
@ -4786,11 +4692,6 @@ void destroyTableMergeScanOperatorInfo(void* param) {
taosMemoryFreeClear(param);
}
typedef struct STableMergeScanExecInfo {
SFileBlockLoadRecorder blockRecorder;
SSortExecInfo sortExecInfo;
} STableMergeScanExecInfo;
int32_t getTableMergeScanExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain, uint32_t* len) {
ASSERT(pOptr != NULL);
// TODO: merge these two info into one struct
@ -4805,18 +4706,6 @@ int32_t getTableMergeScanExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExpla
return TSDB_CODE_SUCCESS;
}
int32_t compareTableKeyInfoByGid(const void* p1, const void* p2) {
const STableKeyInfo* info1 = p1;
const STableKeyInfo* info2 = p2;
if (info1->groupId < info2->groupId) {
return -1;
} else if (info1->groupId > info2->groupId) {
return 1;
} else {
return 0;
}
}
SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanNode, STableListInfo* pTableListInfo,
SReadHandle* readHandle, SExecTaskInfo* pTaskInfo) {
STableMergeScanInfo* pInfo = taosMemoryCalloc(1, sizeof(STableMergeScanInfo));
@ -4830,6 +4719,9 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN
int32_t numOfCols = 0;
int32_t code = extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID,
&pInfo->matchInfo);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
code = initQueryTableDataCond(&pInfo->cond, pTableScanNode);
if (code != TSDB_CODE_SUCCESS) {
@ -4860,7 +4752,10 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN
pInfo->tableListInfo = pTableListInfo;
pInfo->scanFlag = MAIN_SCAN;
initResultSizeInfo(&pOperator->resultInfo, 1024);
pInfo->pResBlock = createResDataBlock(pDescNode);
blockDataEnsureCapacity(pInfo->pResBlock, pOperator->resultInfo.capacity);
pInfo->sortSourceParams = taosArrayInit(64, sizeof(STableMergeScanSortSourceParam));
pInfo->pSortInfo = generateSortByTsInfo(pInfo->matchInfo.pList, pInfo->cond.order);
@ -4877,7 +4772,6 @@ SOperatorInfo* createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanN
pOperator->info = pInfo;
pOperator->exprSupp.numOfExprs = numOfCols;
pOperator->pTaskInfo = pTaskInfo;
initResultSizeInfo(&pOperator->resultInfo, 1024);
pOperator->fpSet = createOperatorFpSet(operatorDummyOpenFn, doTableMergeScan, NULL, NULL,
destroyTableMergeScanOperatorInfo, getTableMergeScanExplainExecInfo);

28
source/libs/executor/src/sortoperator.c
View File

@ -21,7 +21,7 @@ static SSDataBlock* doSort(SOperatorInfo* pOperator);
static int32_t doOpenSortOperator(SOperatorInfo* pOperator);
static int32_t getExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain, uint32_t* len);
static void destroyOrderOperatorInfo(void* param);
static void destroySortOperatorInfo(void* param);
// todo add limit/offset impl
SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortNode, SExecTaskInfo* pTaskInfo) {
@ -67,7 +67,7 @@ SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode*
// TODO dynamic set the available sort buffer
pOperator->fpSet =
createOperatorFpSet(doOpenSortOperator, doSort, NULL, NULL, destroyOrderOperatorInfo, getExplainExecInfo);
createOperatorFpSet(doOpenSortOperator, doSort, NULL, NULL, destroySortOperatorInfo, getExplainExecInfo);
code = appendDownstream(pOperator, &downstream, 1);
if (code != TSDB_CODE_SUCCESS) {
@ -252,7 +252,7 @@ SSDataBlock* doSort(SOperatorInfo* pOperator) {
return blockDataGetNumOfRows(pBlock) > 0 ? pBlock : NULL;
}
void destroyOrderOperatorInfo(void* param) {
void destroySortOperatorInfo(void* param) {
SSortOperatorInfo* pInfo = (SSortOperatorInfo*)param;
pInfo->binfo.pRes = blockDataDestroy(pInfo->binfo.pRes);
@ -630,10 +630,12 @@ static void doGetSortedBlockData(SMultiwayMergeOperatorInfo* pInfo, SSortHandle*
}
SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, int32_t capacity,
SArray* pColMatchInfo, SOperatorInfo* pOperator) {
SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, SArray* pColMatchInfo,
SOperatorInfo* pOperator) {
SMultiwayMergeOperatorInfo* pInfo = pOperator->info;
int32_t capacity = pOperator->resultInfo.capacity;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
blockDataCleanup(pDataBlock);
@ -643,7 +645,6 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData
}
blockDataEnsureCapacity(p, capacity);
while (1) {
doGetSortedBlockData(pInfo, pHandle, capacity, p);
if (p->info.rows == 0) {
@ -659,8 +660,6 @@ SSDataBlock* getMultiwaySortedBlockData(SSortHandle* pHandle, SSDataBlock* pData
}
if (p->info.rows > 0) {
blockDataEnsureCapacity(pDataBlock, p->info.rows);
int32_t numOfCols = taosArrayGetSize(pColMatchInfo);
for (int32_t i = 0; i < numOfCols; ++i) {
SColMatchItem* pmInfo = taosArrayGet(pColMatchInfo, i);
@ -695,13 +694,13 @@ SSDataBlock* doMultiwayMerge(SOperatorInfo* pOperator) {
T_LONG_JMP(pTaskInfo->env, code);
}
SSDataBlock* pBlock = getMultiwaySortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes,
pOperator->resultInfo.capacity, pInfo->matchInfo.pList, pOperator);
SSDataBlock* pBlock = getMultiwaySortedBlockData(pInfo->pSortHandle, pInfo->binfo.pRes, pInfo->matchInfo.pList, pOperator);
if (pBlock != NULL) {
pOperator->resultInfo.totalRows += pBlock->info.rows;
} else {
doSetOperatorCompleted(pOperator);
}
return pBlock;
}
@ -745,12 +744,11 @@ SOperatorInfo* createMultiwayMergeOperatorInfo(SOperatorInfo** downStreams, size
initLimitInfo(pMergePhyNode->node.pLimit, pMergePhyNode->node.pSlimit, &pInfo->limitInfo);
pInfo->binfo.pRes = createResDataBlock(pDescNode);
int32_t rowSize = pInfo->binfo.pRes->info.rowSize;
ASSERT(rowSize < 100 * 1024 * 1024);
SArray* pSortInfo = createSortInfo(pMergePhyNode->pMergeKeys);
int32_t numOfOutputCols = 0;
code = extractColMatchInfo(pMergePhyNode->pTargets, pDescNode, &numOfOutputCols, COL_MATCH_FROM_SLOT_ID,
&pInfo->matchInfo);
if (code != TSDB_CODE_SUCCESS) {
@ -759,10 +757,12 @@ SOperatorInfo* createMultiwayMergeOperatorInfo(SOperatorInfo** downStreams, size
SPhysiNode* pChildNode = (SPhysiNode*)nodesListGetNode(pPhyNode->pChildren, 0);
SSDataBlock* pInputBlock = createResDataBlock(pChildNode->pOutputDataBlockDesc);
initResultSizeInfo(&pOperator->resultInfo, 1024);
blockDataEnsureCapacity(pInfo->binfo.pRes, pOperator->resultInfo.capacity);
pInfo->groupSort = pMergePhyNode->groupSort;
pInfo->pSortInfo = pSortInfo;
pInfo->pSortInfo = createSortInfo(pMergePhyNode->pMergeKeys);
pInfo->pInputBlock = pInputBlock;
pInfo->bufPageSize = getProperSortPageSize(rowSize);
pInfo->sortBufSize = pInfo->bufPageSize * (numStreams + 1); // one additional is reserved for merged result.
@ -784,11 +784,11 @@ SOperatorInfo* createMultiwayMergeOperatorInfo(SOperatorInfo** downStreams, size
return pOperator;
_error:
pTaskInfo->code = TSDB_CODE_OUT_OF_MEMORY;
if (pInfo != NULL) {
destroyMultiwayMergeOperatorInfo(pInfo);
}
pTaskInfo->code = code;
taosMemoryFree(pOperator);
return NULL;
}

3
source/libs/executor/src/timewindowoperator.c
View File

@ -986,7 +986,8 @@ void doCloseWindow(SResultRowInfo* pResultRowInfo, const SIntervalAggOperatorInf
// current result is done in computing final results.
if (pInfo->timeWindowInterpo && isResultRowInterpolated(pResult, RESULT_ROW_END_INTERP)) {
closeResultRow(pResult);
tdListPopHead(pResultRowInfo->openWindow);
SListNode *pNode = tdListPopHead(pResultRowInfo->openWindow);
taosMemoryFree(pNode);
}
}

6
source/libs/function/src/builtinsimpl.c
View File

@ -489,7 +489,7 @@ EFuncDataRequired countDataRequired(SFunctionNode* pFunc, STimeWindow* pTimeWind
if (QUERY_NODE_COLUMN == nodeType(pParam) && PRIMARYKEY_TIMESTAMP_COL_ID == ((SColumnNode*)pParam)->colId) {
return FUNC_DATA_REQUIRED_NOT_LOAD;
}
return FUNC_DATA_REQUIRED_STATIS_LOAD;
return FUNC_DATA_REQUIRED_SMA_LOAD;
}
bool getCountFuncEnv(SFunctionNode* UNUSED_PARAM(pFunc), SFuncExecEnv* pEnv) {
@ -1103,7 +1103,7 @@ int32_t avgPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
}
EFuncDataRequired statisDataRequired(SFunctionNode* pFunc, STimeWindow* pTimeWindow) {
return FUNC_DATA_REQUIRED_STATIS_LOAD;
return FUNC_DATA_REQUIRED_SMA_LOAD;
}
bool minmaxFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo) {
@ -5451,6 +5451,8 @@ int32_t modeFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
colDataAppendNULL(pCol, currentRow);
}
taosHashCleanup(pInfo->pHash);
return pResInfo->numOfRes;
}

3
source/libs/index/test/index_executor_tests.cpp
View File

@ -86,8 +86,7 @@ void sifAppendReservedSlot(SArray *pBlockList, int16_t *dataBlockId, int16_t *sl
SColumnInfoData idata = {0};
idata.info = *colInfo;
colInfoDataEnsureCapacity(&idata, rows);
colInfoDataEnsureCapacity(&idata, rows, true);
taosArrayPush(res->pDataBlock, &idata);
*dataBlockId = taosArrayGetSize(pBlockList) - 1;

2
source/libs/planner/src/planOptimizer.c
View File

@ -279,7 +279,7 @@ static EFuncDataRequired scanPathOptPromoteDataRequired(EFuncDataRequired l, EFu
switch (l) {
case FUNC_DATA_REQUIRED_DATA_LOAD:
return l;
case FUNC_DATA_REQUIRED_STATIS_LOAD:
case FUNC_DATA_REQUIRED_SMA_LOAD:
return FUNC_DATA_REQUIRED_DATA_LOAD == r ? r : l;
case FUNC_DATA_REQUIRED_NOT_LOAD:
return FUNC_DATA_REQUIRED_FILTEROUT == r ? l : r;

1
source/libs/scalar/CMakeLists.txt
View File

@ -14,7 +14,6 @@ target_link_libraries(scalar
PRIVATE nodes
PRIVATE function
PRIVATE qcom
PRIVATE vnode
)
if(${BUILD_TEST})

8
source/libs/scalar/src/scalar.c
View File

@ -49,7 +49,7 @@ int32_t sclCreateColumnInfoData(SDataType *pType, int32_t numOfRows, SScalarPara
pColumnData->info.scale = pType->scale;
pColumnData->info.precision = pType->precision;
int32_t code = colInfoDataEnsureCapacity(pColumnData, numOfRows);
int32_t code = colInfoDataEnsureCapacity(pColumnData, numOfRows, true);
if (code != TSDB_CODE_SUCCESS) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
taosMemoryFree(pColumnData);
@ -70,7 +70,7 @@ int32_t sclConvertValueToSclParam(SValueNode* pValueNode, SScalarParam* out, int
colDataAppend(in.columnData, 0, nodesGetValueFromNode(pValueNode), false);
colInfoDataEnsureCapacity(out->columnData, 1);
colInfoDataEnsureCapacity(out->columnData, 1, true);
code = vectorConvertSingleColImpl(&in, out, overflow, -1, -1);
sclFreeParam(&in);
@ -88,7 +88,7 @@ int32_t sclExtendResRows(SScalarParam *pDst, SScalarParam *pSrc, SArray *pBlockL
pLeft->numOfRows = pb->info.rows;
if (pDst->numOfRows < pb->info.rows) {
colInfoDataEnsureCapacity(pDst->columnData, pb->info.rows);
colInfoDataEnsureCapacity(pDst->columnData, pb->info.rows, true);
}
_bin_scalar_fn_t OperatorFn = getBinScalarOperatorFn(OP_TYPE_ASSIGN);
@ -1604,7 +1604,7 @@ int32_t scalarCalculate(SNode *pNode, SArray *pBlockList, SScalarParam *pDst) {
if (1 == res->numOfRows) {
SCL_ERR_JRET(sclExtendResRows(pDst, res, pBlockList));
} else {
colInfoDataEnsureCapacity(pDst->columnData, res->numOfRows);
colInfoDataEnsureCapacity(pDst->columnData, res->numOfRows, true);
colDataAssign(pDst->columnData, res->columnData, res->numOfRows, NULL);
pDst->numOfRows = res->numOfRows;
pDst->numOfQualified = res->numOfQualified;

8
source/libs/scalar/src/sclfunc.c
View File

@ -6,7 +6,6 @@
#include "tdatablock.h"
#include "tjson.h"
#include "ttime.h"
#include "vnode.h"
typedef float (*_float_fn)(float);
typedef double (*_double_fn)(double);
@ -1718,12 +1717,9 @@ int32_t winEndTsFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *p
int32_t qTbnameFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput) {
ASSERT(inputNum == 1);
char* p = colDataGetVarData(pInput->columnData, 0);
uint64_t uid = *(uint64_t *)colDataGetData(pInput->columnData, 0);
char str[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
metaGetTableNameByUid(pInput->param, uid, str);
colDataAppendNItems(pOutput->columnData, pOutput->numOfRows, str, pInput->numOfRows);
colDataAppendNItems(pOutput->columnData, pOutput->numOfRows, p, pInput->numOfRows);
pOutput->numOfRows += pInput->numOfRows;
return TSDB_CODE_SUCCESS;
}

12
source/libs/scalar/test/scalar/scalarTests.cpp
View File

@ -87,7 +87,7 @@ void scltAppendReservedSlot(SArray *pBlockList, int16_t *dataBlockId, int16_t *s
SColumnInfoData idata = {0};
idata.info = *colInfo;
colInfoDataEnsureCapacity(&idata, rows);
colInfoDataEnsureCapacity(&idata, rows, true);
blockDataAppendColInfo(res, &idata);
@ -104,7 +104,7 @@ void scltAppendReservedSlot(SArray *pBlockList, int16_t *dataBlockId, int16_t *s
SSDataBlock *res = *(SSDataBlock **)taosArrayGetLast(pBlockList);
SColumnInfoData idata = {0};
idata.info = *colInfo;
colInfoDataEnsureCapacity(&idata, rows);
colInfoDataEnsureCapacity(&idata, rows, true);
blockDataAppendColInfo(res, &idata);
*dataBlockId = taosArrayGetSize(pBlockList) - 1;
@ -146,12 +146,12 @@ void scltMakeColumnNode(SNode **pNode, SSDataBlock **block, int32_t dataType, in
SSDataBlock *res = createDataBlock();
for (int32_t i = 0; i < 2; ++i) {
SColumnInfoData idata = createColumnInfoData(TSDB_DATA_TYPE_INT, 10, i + 1);
colInfoDataEnsureCapacity(&idata, rowNum);
colInfoDataEnsureCapacity(&idata, rowNum, true);
blockDataAppendColInfo(res, &idata);
}
SColumnInfoData idata = createColumnInfoData(dataType, dataBytes, 3);
colInfoDataEnsureCapacity(&idata, rowNum);
colInfoDataEnsureCapacity(&idata, rowNum, true);
blockDataAppendColInfo(res, &idata);
res->info.capacity = rowNum;
@ -175,7 +175,7 @@ void scltMakeColumnNode(SNode **pNode, SSDataBlock **block, int32_t dataType, in
int32_t idx = taosArrayGetSize(res->pDataBlock);
SColumnInfoData idata = createColumnInfoData(dataType, dataBytes, 1 + idx);
colInfoDataEnsureCapacity(&idata, rowNum);
colInfoDataEnsureCapacity(&idata, rowNum, true);
res->info.capacity = rowNum;
blockDataAppendColInfo(res, &idata);
@ -2022,7 +2022,7 @@ void scltMakeDataBlock(SScalarParam **pInput, int32_t type, void *pVal, int32_t
input->numOfRows = num;
input->columnData->info = createColumnInfo(0, type, bytes);
colInfoDataEnsureCapacity(input->columnData, num);
colInfoDataEnsureCapacity(input->columnData, num, true);
if (setVal) {
for (int32_t i = 0; i < num; ++i) {

3
source/libs/scheduler/src/schRemote.c
View File

@ -185,12 +185,12 @@ int32_t schHandleResponseMsg(SSchJob *pJob, SSchTask *pTask, int32_t execId, SDa
code = rsp->code;
}
}
SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
if (taosArrayGetSize((SArray*)pJob->execRes.res) <= 0) {
taosArrayDestroy((SArray*)pJob->execRes.res);
pJob->execRes.res = NULL;
}
SCH_UNLOCK(SCH_WRITE, &pJob->resLock);
}
tDecoderClear(&coder);
@ -1047,6 +1047,7 @@ int32_t schBuildAndSendMsg(SSchJob *pJob, SSchTask *pTask, SQueryNodeAddr *addr,
SSubQueryMsg *pMsg = msg;
pMsg->header.vgId = htonl(addr->nodeId);
pMsg->header.msgMask = htonl((pTask->plan->showRewrite) ? SHOW_REWRITE_MASK() : 0);
pMsg->sId = htobe64(schMgmt.sId);
pMsg->queryId = htobe64(pJob->queryId);
pMsg->taskId = htobe64(pTask->taskId);

1
source/libs/sync/inc/syncCommit.h
View File

@ -49,6 +49,7 @@ extern "C" {
// IN commitIndex' = [commitIndex EXCEPT ![i] = newCommitIndex]
// /\ UNCHANGED <<messages, serverVars, candidateVars, leaderVars, log>>
//
void syncOneReplicaAdvance(SSyncNode* pSyncNode);
void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode);
bool syncAgreeIndex(SSyncNode* pSyncNode, SRaftId* pRaftId, SyncIndex index);
bool syncAgree(SSyncNode* pSyncNode, SyncIndex index);

50
source/libs/sync/src/syncCommit.c
View File

@ -44,6 +44,56 @@
// IN commitIndex' = [commitIndex EXCEPT ![i] = newCommitIndex]
// /\ UNCHANGED <<messages, serverVars, candidateVars, leaderVars, log>>
//
void syncOneReplicaAdvance(SSyncNode* pSyncNode) {
if (pSyncNode == NULL) {
sError("pSyncNode is NULL");
return;
}
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
syncNodeErrorLog(pSyncNode, "not leader, can not advance commit index");
return;
}
if (pSyncNode->replicaNum != 1) {
syncNodeErrorLog(pSyncNode, "not one replica, can not advance commit index");
return;
}
// advance commit index to snapshot first
SSnapshot snapshot;
pSyncNode->pFsm->FpGetSnapshotInfo(pSyncNode->pFsm, &snapshot);
if (snapshot.lastApplyIndex > 0 && snapshot.lastApplyIndex > pSyncNode->commitIndex) {
SyncIndex commitBegin = pSyncNode->commitIndex;
SyncIndex commitEnd = snapshot.lastApplyIndex;
pSyncNode->commitIndex = snapshot.lastApplyIndex;
char eventLog[128];
snprintf(eventLog, sizeof(eventLog), "commit by snapshot from index:%" PRId64 " to index:%" PRId64, commitBegin,
commitEnd);
syncNodeEventLog(pSyncNode, eventLog);
}
// advance commit index as large as possible
SyncIndex lastIndex = syncNodeGetLastIndex(pSyncNode);
if (lastIndex > pSyncNode->commitIndex) {
do {
char eventLog[128];
snprintf(eventLog, sizeof(eventLog), "commit by wal from index:%" PRId64 " to index:%" PRId64,
pSyncNode->commitIndex + 1, lastIndex);
syncNodeEventLog(pSyncNode, eventLog);
} while (0);
pSyncNode->commitIndex = lastIndex;
}
// call back Wal
SyncIndex walCommitVer = logStoreWalCommitVer(pSyncNode->pLogStore);
if (pSyncNode->commitIndex > walCommitVer) {
pSyncNode->pLogStore->syncLogUpdateCommitIndex(pSyncNode->pLogStore, pSyncNode->commitIndex);
}
}
void syncMaybeAdvanceCommitIndex(SSyncNode* pSyncNode) {
if (pSyncNode == NULL) {
sError("pSyncNode is NULL");

28
source/libs/sync/src/syncMain.c
View File

@ -768,7 +768,7 @@ char* sync2SimpleStr(int64_t rid) {
sTrace("syncSetRpc get pSyncNode is NULL, rid:%" PRId64, rid);
return NULL;
}
ASSERT(rid == pSyncNode->rid);
char* s = syncNode2SimpleStr(pSyncNode);
syncNodeRelease(pSyncNode);
@ -778,11 +778,9 @@ char* sync2SimpleStr(int64_t rid) {
int32_t syncPropose(int64_t rid, SRpcMsg* pMsg, bool isWeak) {
SSyncNode* pSyncNode = syncNodeAcquire(rid);
if (pSyncNode == NULL) {
syncNodeRelease(pSyncNode);
terrno = TSDB_CODE_SYN_INTERNAL_ERROR;
return -1;
}
ASSERT(rid == pSyncNode->rid);
int32_t ret = syncNodePropose(pSyncNode, pMsg, isWeak);
syncNodeRelease(pSyncNode);
@ -3010,7 +3008,11 @@ int32_t syncNodeOnClientRequest(SSyncNode* ths, SyncClientRequest* pMsg, SyncInd
// if only myself, maybe commit right now
if (ths->replicaNum == 1) {
syncMaybeAdvanceCommitIndex(ths);
if (syncNodeIsMnode(ths)) {
syncMaybeAdvanceCommitIndex(ths);
} else {
syncOneReplicaAdvance(ths);
}
}
}
@ -3104,15 +3106,15 @@ int32_t syncDoLeaderTransfer(SSyncNode* ths, SRpcMsg* pRpcMsg, SSyncRaftEntry* p
if (ths->pFsm->FpLeaderTransferCb != NULL) {
SFsmCbMeta cbMeta = {
cbMeta.code = 0,
cbMeta.currentTerm = ths->pRaftStore->currentTerm,
cbMeta.flag = 0,
cbMeta.index = pEntry->index,
cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, pEntry->index),
cbMeta.isWeak = pEntry->isWeak,
cbMeta.seqNum = pEntry->seqNum,
cbMeta.state = ths->state,
cbMeta.term = pEntry->term,
.code = 0,
.currentTerm = ths->pRaftStore->currentTerm,
.flag = 0,
.index = pEntry->index,
.lastConfigIndex = syncNodeGetSnapshotConfigIndex(ths, pEntry->index),
.isWeak = pEntry->isWeak,
.seqNum = pEntry->seqNum,
.state = ths->state,
.term = pEntry->term,
};
ths->pFsm->FpLeaderTransferCb(ths->pFsm, pRpcMsg, &cbMeta);
}

1
source/libs/sync/src/syncRaftEntry.c
View File

@ -72,6 +72,7 @@ SSyncRaftEntry* syncEntryBuildNoop(SyncTerm term, SyncIndex index, int32_t vgId)
SMsgHead head;
head.vgId = vgId;
head.contLen = sizeof(SMsgHead);
head.msgMask = 0;
SRpcMsg rpcMsg;
memset(&rpcMsg, 0, sizeof(SRpcMsg));
rpcMsg.contLen = head.contLen;

18
source/libs/sync/src/syncRespMgr.c
View File

@ -146,15 +146,15 @@ void syncRespCleanByTTL(SSyncRespMgr *pObj, int64_t ttl, bool rsp) {
cnt++;
SFsmCbMeta cbMeta = {
cbMeta.index = SYNC_INDEX_INVALID,
cbMeta.lastConfigIndex = SYNC_INDEX_INVALID,
cbMeta.isWeak = false,
cbMeta.code = TSDB_CODE_SYN_TIMEOUT,
cbMeta.state = pSyncNode->state,
cbMeta.seqNum = *pSeqNum,
cbMeta.term = SYNC_TERM_INVALID,
cbMeta.currentTerm = pSyncNode->pRaftStore->currentTerm,
cbMeta.flag = 0,
.index = SYNC_INDEX_INVALID,
.lastConfigIndex = SYNC_INDEX_INVALID,
.isWeak = false,
.code = TSDB_CODE_SYN_TIMEOUT,
.state = pSyncNode->state,
.seqNum = *pSeqNum,
.term = SYNC_TERM_INVALID,
.currentTerm = pSyncNode->pRaftStore->currentTerm,
.flag = 0,
};
pStub->rpcMsg.pCont = NULL;

10
source/util/src/tconfig.c
View File

@ -722,13 +722,13 @@ int32_t cfgLoadFromEnvFile(SConfig *pConfig, const char *envFile) {
const char *filepath = ".env";
if (envFile != NULL && strlen(envFile) > 0) {
if (!taosCheckExistFile(envFile)) {
uError("failed to load env file: %s", envFile);
uError("failed to load env file:%s", envFile);
return -1;
}
filepath = envFile;
} else {
if (!taosCheckExistFile(filepath)) {
uInfo("failed to load env file: %s", filepath);
uInfo("env file:%s not load", filepath);
return 0;
}
}
@ -929,7 +929,7 @@ int32_t cfgLoadFromApollUrl(SConfig *pConfig, const char *url) {
if (strncmp(url, "jsonFile", 8) == 0) {
char *filepath = p;
if (!taosCheckExistFile(filepath)) {
uError("failed to load json file: %s", filepath);
uError("failed to load json file:%s", filepath);
return -1;
}
@ -1056,13 +1056,13 @@ int32_t cfgGetApollUrl(const char **envCmd, const char *envFile, char *apolloUrl
const char *filepath = ".env";
if (envFile != NULL && strlen(envFile) > 0) {
if (!taosCheckExistFile(envFile)) {
uError("failed to load env file: %s", envFile);
uError("failed to load env file:%s", envFile);
return -1;
}
filepath = envFile;
} else {
if (!taosCheckExistFile(filepath)) {
uInfo("failed to load env file: %s", filepath);
uInfo("env file:%s not load", filepath);
return 0;
}
}

20
source/util/src/tlog.c
View File

@ -81,25 +81,25 @@ int64_t tsNumOfDebugLogs = 0;
int64_t tsNumOfTraceLogs = 0;
// log
int32_t dDebugFlag = 135;
int32_t vDebugFlag = 135;
int32_t mDebugFlag = 135;
int32_t dDebugFlag = 131;
int32_t vDebugFlag = 131;
int32_t mDebugFlag = 131;
int32_t cDebugFlag = 131;
int32_t jniDebugFlag = 131;
int32_t tmrDebugFlag = 131;
int32_t uDebugFlag = 131;
int32_t rpcDebugFlag = 131;
int32_t qDebugFlag = 131;
int32_t wDebugFlag = 135;
int32_t sDebugFlag = 135;
int32_t wDebugFlag = 131;
int32_t sDebugFlag = 131;
int32_t tsdbDebugFlag = 131;
int32_t tdbDebugFlag = 131;
int32_t tqDebugFlag = 135;
int32_t fsDebugFlag = 135;
int32_t metaDebugFlag = 135;
int32_t udfDebugFlag = 135;
int32_t tqDebugFlag = 131;
int32_t fsDebugFlag = 131;
int32_t metaDebugFlag = 131;
int32_t udfDebugFlag = 131;
int32_t smaDebugFlag = 131;
int32_t idxDebugFlag = 135;
int32_t idxDebugFlag = 131;
int64_t dbgEmptyW = 0;
int64_t dbgWN = 0;

40
source/util/src/tqueue.c
View File

@ -21,46 +21,6 @@
int64_t tsRpcQueueMemoryAllowed = 0;
int64_t tsRpcQueueMemoryUsed = 0;
typedef struct STaosQnode STaosQnode;
typedef struct STaosQnode {
STaosQnode *next;
STaosQueue *queue;
int64_t timestamp;
int32_t size;
int8_t itype;
int8_t reserved[3];
char item[];
} STaosQnode;
typedef struct STaosQueue {
STaosQnode *head;
STaosQnode *tail;
STaosQueue *next; // for queue set
STaosQset *qset; // for queue set
void *ahandle; // for queue set
FItem itemFp;
FItems itemsFp;
TdThreadMutex mutex;
int64_t memOfItems;
int32_t numOfItems;
} STaosQueue;
typedef struct STaosQset {
STaosQueue *head;
STaosQueue *current;
TdThreadMutex mutex;
tsem_t sem;
int32_t numOfQueues;
int32_t numOfItems;
} STaosQset;
typedef struct STaosQall {
STaosQnode *current;
STaosQnode *start;
int32_t numOfItems;
} STaosQall;
STaosQueue *taosOpenQueue() {
STaosQueue *queue = taosMemoryCalloc(1, sizeof(STaosQueue));
if (queue == NULL) {

2
source/util/src/tref.c
View File

@ -361,7 +361,7 @@ int32_t taosListRef() {
if (pSet->state == TSDB_REF_STATE_EMPTY) continue;
uInfo("rsetId:%d state:%d count::%d", i, pSet->state, pSet->count);
uInfo("rsetId:%d state:%d count:%d", i, pSet->state, pSet->count);
for (int32_t j = 0; j < pSet->max; ++j) {
pNode = pSet->nodeList[j];

22
source/util/src/tworker.c
View File

@ -36,14 +36,13 @@ int32_t tQWorkerInit(SQWorkerPool *pool) {
worker->pool = pool;
}
uInfo("worker:%s is initialized, min:%d max:%d", pool->name, pool->min, pool->max);
uDebug("worker:%s is initialized, min:%d max:%d", pool->name, pool->min, pool->max);
return 0;
}
void tQWorkerCleanup(SQWorkerPool *pool) {
for (int32_t i = 0; i < pool->max; ++i) {
SQWorker *worker = pool->workers + i;
// if (worker == NULL) continue;
if (taosCheckPthreadValid(worker->thread)) {
taosQsetThreadResume(pool->qset);
}
@ -51,7 +50,6 @@ void tQWorkerCleanup(SQWorkerPool *pool) {
for (int32_t i = 0; i < pool->max; ++i) {
SQWorker *worker = pool->workers + i;
// if (worker == NULL) continue;
if (taosCheckPthreadValid(worker->thread)) {
taosThreadJoin(worker->thread, NULL);
taosThreadClear(&worker->thread);
@ -73,11 +71,13 @@ static void *tQWorkerThreadFp(SQWorker *worker) {
taosBlockSIGPIPE();
setThreadName(pool->name);
uDebug("worker:%s:%d is running", pool->name, worker->id);
worker->pid = taosGetSelfPthreadId();
uInfo("worker:%s:%d is running, thread:%08" PRId64, pool->name, worker->id, worker->pid);
while (1) {
if (taosReadQitemFromQset(pool->qset, (void **)&msg, &qinfo) == 0) {
uDebug("worker:%s:%d qset:%p, got no message and exiting", pool->name, worker->id, pool->qset);
uInfo("worker:%s:%d qset:%p, got no message and exiting, thread:%08" PRId64, pool->name, worker->id, pool->qset,
worker->pid);
break;
}
@ -124,7 +124,7 @@ STaosQueue *tQWorkerAllocQueue(SQWorkerPool *pool, void *ahandle, FItem fp) {
}
taosThreadMutexUnlock(&pool->mutex);
uDebug("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
uInfo("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
return queue;
}
@ -191,12 +191,14 @@ static void *tWWorkerThreadFp(SWWorker *worker) {
taosBlockSIGPIPE();
setThreadName(pool->name);
uDebug("worker:%s:%d is running", pool->name, worker->id);
worker->pid = taosGetSelfPthreadId();
uInfo("worker:%s:%d is running, thread:%08" PRId64, pool->name, worker->id, worker->pid);
while (1) {
numOfMsgs = taosReadAllQitemsFromQset(worker->qset, worker->qall, &qinfo);
if (numOfMsgs == 0) {
uDebug("worker:%s:%d qset:%p, got no message and exiting", pool->name, worker->id, worker->qset);
uInfo("worker:%s:%d qset:%p, got no message and exiting, thread:%08" PRId64, pool->name, worker->id, worker->qset,
worker->pid);
break;
}
@ -244,7 +246,6 @@ STaosQueue *tWWorkerAllocQueue(SWWorkerPool *pool, void *ahandle, FItems fp) {
pool->nextId = (pool->nextId + 1) % pool->max;
}
uDebug("worker:%s, queue:%p is allocated, ahandle:%p", pool->name, queue, ahandle);
code = 0;
_OVER:
@ -256,6 +257,9 @@ _OVER:
if (worker->qall != NULL) taosFreeQall(worker->qall);
return NULL;
} else {
while (worker->pid <= 0) taosMsleep(10);
queue->threadId = worker->pid;
uInfo("worker:%s, queue:%p is allocated, ahandle:%p thread:%08" PRId64, pool->name, queue, ahandle, queue->threadId);
return queue;
}
}

74
tests/develop-test/5-taos-tools/taosbenchmark/json/queryInsertdata.json Normal file
View File

@ -0,0 +1,74 @@
{
"filetype": "insert",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"thread_count": 4,
"thread_count_create_tbl": 4,
"result_file": "./insert_res.txt",
"confirm_parameter_prompt": "no",
"insert_interval": 0,
"interlace_rows": 0,
"num_of_records_per_req": 3000,
"max_sql_len": 1024000,
"databases": [{
"dbinfo": {
"name": "db",
"drop": "yes",
"replica": 1,
"precision": "ms"
},
"super_tables": [{
"name": "stb0",
"child_table_exists":"no",
"childtable_count": 10,
"childtable_prefix": "stb00_",
"auto_create_table": "no",
"batch_create_tbl_num": 10,
"data_source": "rand",
"insert_mode": "taosc",
"insert_rows": 100,
"childtable_limit": 0,
"childtable_offset": 0,
"interlace_rows": 0,
"insert_interval": 0,
"max_sql_len": 1024000,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 1,
"start_timestamp": "2020-11-01 00:00:00.000",
"sample_format": "csv",
"sample_file": "./sample.csv",
"tags_file": "",
"columns": [{"type": "BINARY", "len": 1, "count":1}, {"type": "BINARY", "len": 3, "count":1}, {"type": "INT"}, {"type": "DOUBLE", "count":1}],
"tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
},
{
"name": "stb1",
"child_table_exists":"no",
"childtable_count": 10,
"childtable_prefix": "stb01_",
"auto_create_table": "no",
"batch_create_tbl_num": 10,
"data_source": "rand",
"insert_mode": "taosc",
"insert_rows": 200,
"childtable_limit": 0,
"childtable_offset": 0,
"interlace_rows": 0 ,
"insert_interval": 0,
"max_sql_len": 1024000,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 1,
"start_timestamp": "2020-11-01 00:00:00.000",
"sample_format": "csv",
"sample_file": "./sample.csv",
"tags_file": "",
"columns": [{"type": "INT"}, {"type": "DOUBLE", "count":6}, {"type": "BINARY", "len": 1, "count":3}, {"type": "BINARY", "len": 2, "count":6}],
"tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
}]
}]
}

73
tests/develop-test/5-taos-tools/taosbenchmark/json/queryInsertrestdata.json Normal file
View File

@ -0,0 +1,73 @@
{
"filetype": "insert",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"thread_count": 4,
"thread_count_create_tbl": 4,
"result_file": "./insert_res.txt",
"confirm_parameter_prompt": "no",
"insert_interval": 0,
"interlace_rows": 0,
"num_of_records_per_req": 3000,
"max_sql_len": 1024000,
"databases": [{
"dbinfo": {
"name": "db",
"drop": "yes",
"precision": "ms"
},
"super_tables": [{
"name": "stb0",
"child_table_exists":"no",
"childtable_count": 2,
"childtable_prefix": "stb00_",
"auto_create_table": "no",
"batch_create_tbl_num": 10,
"data_source": "rand",
"insert_mode": "taosc",
"insert_rows": 10,
"childtable_limit": 0,
"childtable_offset": 0,
"interlace_rows": 0,
"insert_interval": 0,
"max_sql_len": 1024000,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 1,
"start_timestamp": "2020-11-01 00:00:00.000",
"sample_format": "csv",
"sample_file": "./sample.csv",
"tags_file": "",
"columns": [{"type": "BINARY", "len": 1, "count":1}, {"type": "BINARY", "len": 3, "count":1}, {"type": "INT"}, {"type": "DOUBLE", "count":1}],
"tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
},
{
"name": "stb1",
"child_table_exists":"no",
"childtable_count": 2,
"childtable_prefix": "stb01_",
"auto_create_table": "no",
"batch_create_tbl_num": 10,
"data_source": "rand",
"insert_mode": "taosc",
"insert_rows": 5,
"childtable_limit": 0,
"childtable_offset": 0,
"interlace_rows": 0 ,
"insert_interval": 0,
"max_sql_len": 1024000,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 1,
"start_timestamp": "2020-11-01 00:00:00.000",
"sample_format": "csv",
"sample_file": "./sample.csv",
"tags_file": "",
"columns": [{"type": "INT"}, {"type": "DOUBLE", "count":6}, {"type": "BINARY", "len": 1, "count":3}, {"type": "BINARY", "len": 2, "count":6}],
"tags": [{"type": "TINYINT", "count":2}, {"type": "BINARY", "len": 16, "count":5}]
}]
}]
}

35
tests/develop-test/5-taos-tools/taosbenchmark/json/queryQps.json Normal file
View File

@ -0,0 +1,35 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 1,
"specified_table_query": {
"query_interval": 10,
"threads": 4,
"sqls": [
{
"sql": "select last_row(*) from stb00_0",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_9 ",
"result": "./query_res1.txt"
}]
},
"super_table_query": {
"stblname": "stb1",
"query_interval":20,
"threads": 4,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}

38
tests/develop-test/5-taos-tools/taosbenchmark/json/queryRestful.json Normal file
View File

@ -0,0 +1,38 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 2,
"query_mode": "rest",
"specified_table_query": {
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(*) from db.stb0 ",
"result": "./query_res0.txt"
},
{
"sql": "select count(*) from db.stb00_1",
"result": "./query_res1.txt"
}
]
},
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}

429
tests/develop-test/5-taos-tools/taosbenchmark/json/querySpeciMutisql100.json Normal file
View File

@ -0,0 +1,429 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 2,
"specified_table_query": {
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(*) from stb00_0",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_1",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_2",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_3",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_4",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_5",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_6",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_7",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_8",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_9",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_10 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_11 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_12 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_13 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_14 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_15 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_16 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_17 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_18 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_19 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_20 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_21 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_22 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_23 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_24 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_25 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_26 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_27 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_28 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_29 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_30 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_31 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_32 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_33 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_34 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_35 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_36 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_37 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_38 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_39 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_40 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_41 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_42 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_43 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_44 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_45 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_46 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_47 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_48 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_49 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_50 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_51 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_52 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_53 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_54 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_55 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_56 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_57 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_58 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_59 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_60",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_61",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_62",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_63",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_64",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_65",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_66",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_67",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_68",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_69",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_70 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_71 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_72 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_73 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_74 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_75 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_76 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_77 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_78 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_79 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_80 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_81 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_82 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_83 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_84 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_85 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_86 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_87 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_88 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_89 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_90 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_91 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_92 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_93 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_94 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_95 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_96 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_97 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_98 ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from stb00_99 ",
"result": "./query_res0.txt"
}]
},
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}

419
tests/develop-test/5-taos-tools/taosbenchmark/json/querySuperMutisql100.json Normal file
View File

@ -0,0 +1,419 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 3,
"super_table_query": {
"stblname": "stb0",
"query_interval": 10,
"threads": 9,
"sqls": [
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select last_row(*) from xxxx ",
"result": "./query_res0.txt"
},
{
"sql": "select * from xxxx ",
"result": "./query_res0.txt"
}]
}
}

37
tests/develop-test/5-taos-tools/taosbenchmark/json/queryTaosc.json Normal file
View File

@ -0,0 +1,37 @@
{
"filetype": "query",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"confirm_parameter_prompt": "no",
"databases": "db",
"query_times": 2,
"query_mode": "taosc",
"specified_table_query": {
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(*) from stb0 ",
"result": "./query_res0.txt"
},
{
"sql": "select count(*) from stb00_1",
"result": "./query_res1.txt"
}
]
},
"super_table_query": {
"stblname": "stb1",
"query_interval": 1,
"threads": 3,
"sqls": [
{
"sql": "select last_row(ts) from xxxx",
"result": "./query_res2.txt"
}
]
}
}

241
tests/develop-test/5-taos-tools/taosbenchmark/taosdemoTestQueryWithJson.py Normal file
View File

@ -0,0 +1,241 @@
###################################################################
# Copyright (c) 2016 by TAOS Technologies, Inc.
# All rights reserved.
#
# This file is proprietary and confidential to TAOS Technologies.
# No part of this file may be reproduced, stored, transmitted,
# disclosed or used in any form or by any means other than as
# expressly provided by the written permission from Jianhui Tao
#
###################################################################
# -*- coding: utf-8 -*-
import sys
import os
from util.log import *
from util.cases import *
from util.sql import *
from util.dnodes import *
import time
from datetime import datetime
import ast
import re
# from assertpy import assert_that
import subprocess
class TDTestCase:
def init(self, conn, logSql, replicaVarl=1):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor(), logSql)
def getPath(self, tool="taosBenchmark"):
selfPath = os.path.dirname(os.path.realpath(__file__))
if "community" in selfPath:
projPath = selfPath[: selfPath.find("community")]
elif "src" in selfPath:
projPath = selfPath[: selfPath.find("src")]
elif "/tools/" in selfPath:
projPath = selfPath[: selfPath.find("/tools/")]
elif "/tests/" in selfPath:
projPath = selfPath[: selfPath.find("/tests/")]
else:
tdLog.info("cannot found %s in path: %s, use system's" % (tool, selfPath))
projPath = "/usr/local/taos/bin/"
paths = []
for root, dirs, files in os.walk(projPath):
if (tool) in files:
rootRealPath = os.path.dirname(os.path.realpath(root))
if "packaging" not in rootRealPath:
paths.append(os.path.join(root, tool))
break
if len(paths) == 0:
return ""
return paths[0]
# 获取taosc接口查询的结果文件中的内容,返回每行数据,并断言数据的第一列内容。
def assertfileDataTaosc(self, filename, expectResult):
self.filename = filename
self.expectResult = expectResult
with open("%s" % filename, "r+") as f1:
for line in f1.readlines():
queryResultTaosc = line.strip().split()[0]
self.assertCheck(filename, queryResultTaosc, expectResult)
# 获取restful接口查询的结果文件中的关键内容,目前的关键内容找到第一个key就跳出循，所以就只有一个数据。后续再修改多个结果文件。
def getfileDataRestful(self, filename):
self.filename = filename
with open("%s" % filename, "r+") as f1:
for line in f1.readlines():
contents = line.strip()
if contents.find("data") != -1:
pattern = re.compile("{.*}")
contents = pattern.search(contents).group()
contentsDict = ast.literal_eval(contents) # 字符串转换为字典
queryResultRest = contentsDict["data"][0][0]
break
else:
queryResultRest = ""
return queryResultRest
# 获取taosc接口查询次数
def queryTimesTaosc(self, filename):
self.filename = filename
command = "cat %s |wc -l" % filename
times = int(subprocess.getstatusoutput(command)[1])
return times
# 获取restful接口查询次数
def queryTimesRestful(self, filename):
self.filename = filename
command = 'cat %s |grep "200 OK" |wc -l' % filename
times = int(subprocess.getstatusoutput(command)[1])
return times
# 定义断言结果是否正确。不正确返回错误结果，正确即通过。
def assertCheck(self, filename, queryResult, expectResult):
self.filename = filename
self.queryResult = queryResult
self.expectResult = expectResult
args0 = (filename, queryResult, expectResult)
assert queryResult == expectResult, (
"Queryfile:%s ,result is %s != expect: %s" % args0
)
def run(self):
binPath = self.getPath()
if binPath == "":
tdLog.exit("taosBenchmark not found!")
else:
tdLog.info("taosBenchmark use %s" % binPath)
# delete useless files
os.system("rm -rf ./query_res*")
os.system("rm -rf ./all_query*")
# taosc query: query specified table and query super table
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath)
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryTaosc.json" % binPath)
os.system("cat query_res0.txt* > all_query_res0_taosc.txt")
os.system("cat query_res1.txt* > all_query_res1_taosc.txt")
os.system("cat query_res2.txt* > all_query_res2_taosc.txt")
# correct Times testcases
queryTimes0Taosc = self.queryTimesTaosc("all_query_res0_taosc.txt")
self.assertCheck("all_query_res0_taosc.txt", queryTimes0Taosc, 6)
queryTimes1Taosc = self.queryTimesTaosc("all_query_res1_taosc.txt")
self.assertCheck("all_query_res1_taosc.txt", queryTimes1Taosc, 6)
queryTimes2Taosc = self.queryTimesTaosc("all_query_res2_taosc.txt")
self.assertCheck("all_query_res2_taosc.txt", queryTimes2Taosc, 20)
# correct data testcase
self.assertfileDataTaosc("all_query_res0_taosc.txt", "1604160000099")
self.assertfileDataTaosc("all_query_res1_taosc.txt", "100")
self.assertfileDataTaosc("all_query_res2_taosc.txt", "1604160000199")
# delete useless files
os.system("rm -rf ./query_res*")
os.system("rm -rf ./all_query*")
# use restful api to query
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertrestdata.json" % binPath)
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryRestful.json" % binPath)
os.system("cat query_res0.txt* > all_query_res0_rest.txt")
os.system("cat query_res1.txt* > all_query_res1_rest.txt")
os.system("cat query_res2.txt* > all_query_res2_rest.txt")
# correct Times testcases
queryTimes0Restful = self.queryTimesRestful("all_query_res0_rest.txt")
self.assertCheck("all_query_res0_rest.txt", queryTimes0Restful, 6)
queryTimes1Restful = self.queryTimesRestful("all_query_res1_rest.txt")
self.assertCheck("all_query_res1_rest.txt", queryTimes1Restful, 6)
queryTimes2Restful = self.queryTimesRestful("all_query_res2_rest.txt")
self.assertCheck("all_query_res2_rest.txt", queryTimes2Restful, 4)
# correct data testcase
data0 = self.getfileDataRestful("all_query_res0_rest.txt")
if data0 != "2020-11-01 00:00:00.009" and data0 != "2020-10-31T16:00:00.009Z":
tdLog.exit(
"data0 is not 2020-11-01 00:00:00.009 and 2020-10-31T16:00:00.009Z"
)
data1 = self.getfileDataRestful("all_query_res1_rest.txt")
self.assertCheck("all_query_res1_rest.txt", data1, 10)
data2 = self.getfileDataRestful("all_query_res2_rest.txt")
if data2 != "2020-11-01 00:00:00.004" and data2 != "2020-10-31T16:00:00.004Z":
tdLog.exit(
"data2 is not 2020-11-01 00:00:00.004 and 2020-10-31T16:00:00.004Z"
)
# query times less than or equal to 100
assert (
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath) == 0
)
assert (
os.system("%s -f ./5-taos-tools/taosbenchmark/json/querySpeciMutisql100.json" % binPath)
!= 0
)
assert (
os.system("%s -f ./5-taos-tools/taosbenchmark/json/querySuperMutisql100.json" % binPath)
== 0
)
# query result print QPS
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath)
exceptcode = os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryQps.json" % binPath)
assert exceptcode == 0
# 2021.02.09 need modify taosBenchmakr code
# use illegal or out of range parameters query json file
os.system("%s -f ./5-taos-tools/taosbenchmark/json/queryInsertdata.json" % binPath)
# 2021.02.09 need modify taosBenchmakr code
# exceptcode = os.system(
# "%s -f ./taosbenchmark/json/queryTimes0.json" %
# binPath)
# assert exceptcode != 0
# 2021.02.09 need modify taosBenchmakr code
# exceptcode0 = os.system(
# "%s -f ./taosbenchmark/json/queryTimesless0.json" %
# binPath)
# assert exceptcode0 != 0
# exceptcode1 = os.system(
# "%s -f ./taosbenchmark/json/queryConcurrent0.json" %
# binPath)
# assert exceptcode2 != 0
# exceptcode3 = os.system(
# "%s -f ./taosbenchmark/json/querrThreadsless0.json" %
# binPath)
# assert exceptcode3 != 0
# exceptcode4 = os.system(
# "%s -f ./taosbenchmark/json/querrThreads0.json" %
# binPath)
# assert exceptcode4 != 0
# delete useless files
os.system("rm -rf ./insert_res.txt")
os.system("rm -rf 5-taos-tools/taosbenchmark/*.py.sql")
os.system("rm -rf ./querySystemInfo*")
os.system("rm -rf ./query_res*")
os.system("rm -rf ./all_query*")
os.system("rm -rf ./test_query_res0.txt")
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
tdCases.addWindows(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())

1
tests/develop-test/fulltest.sh
View File

@ -18,3 +18,4 @@ python3 ./test.py -f 5-taos-tools/taosbenchmark/sml_json_alltypes.py
#python3 ./test.py -f 5-taos-tools/taosbenchmark/sml_telnet_alltypes.py
#python3 ./test.py -f 5-taos-tools/taosbenchmark/taosadapter_json.py
#python3 ./test.py -f 5-taos-tools/taosbenchmark/telnet_tcp.py
python3 ./test.py -f 5-taos-tools/taosbenchmark/taosdemoTestQueryWithJson.py -R

275
tests/develop-test/test.py
View File

@ -24,6 +24,7 @@ import socket
import threading
import toml
sys.path.append("../pytest")
from util.log import *
from util.dnodes import *
@ -34,14 +35,16 @@ from util.taosadapter import *
import taos
import taosrest
def checkRunTimeError():
import win32gui
timeCount = 0
while 1:
time.sleep(1)
timeCount = timeCount + 1
print("checkRunTimeError",timeCount)
if (timeCount>600):
print("checkRunTimeError", timeCount)
if timeCount > 600:
print("stop the test.")
os.system("TASKKILL /F /IM taosd.exe")
os.system("TASKKILL /F /IM taos.exe")
@ -53,6 +56,7 @@ def checkRunTimeError():
if hwnd:
os.system("TASKKILL /F /IM taosd.exe")
if __name__ == "__main__":
fileName = "all"
@ -73,102 +77,124 @@ if __name__ == "__main__":
createDnodeNums = 1
restful = False
replicaVar = 1
opts, args = getopt.gnu_getopt(sys.argv[1:], 'f:p:m:l:scghrd:k:e:N:M:Q:C:RD:n:', [
'file=', 'path=', 'master', 'logSql', 'stop', 'cluster', 'valgrind', 'help', 'restart', 'updateCfgDict', 'killv', 'execCmd','dnodeNums','mnodeNums','queryPolicy','createDnodeNums','restful','adaptercfgupdate','replicaVar'])
opts, args = getopt.gnu_getopt(
sys.argv[1:],
"f:p:m:l:scghrd:k:e:N:M:Q:C:RD:n:",
[
"file=",
"path=",
"master",
"logSql",
"stop",
"cluster",
"valgrind",
"help",
"restart",
"updateCfgDict",
"killv",
"execCmd",
"dnodeNums",
"mnodeNums",
"queryPolicy",
"createDnodeNums",
"restful",
"adaptercfgupdate",
"replicaVar",
],
)
for key, value in opts:
if key in ['-h', '--help']:
tdLog.printNoPrefix(
'A collection of test cases written using Python')
tdLog.printNoPrefix('-f Name of test case file written by Python')
tdLog.printNoPrefix('-p Deploy Path for Simulator')
tdLog.printNoPrefix('-m Master Ip for Simulator')
tdLog.printNoPrefix('-l <True:False> logSql Flag')
tdLog.printNoPrefix('-s stop All dnodes')
tdLog.printNoPrefix('-c Test Cluster Flag')
tdLog.printNoPrefix('-g valgrind Test Flag')
tdLog.printNoPrefix('-r taosd restart test')
tdLog.printNoPrefix('-d update cfg dict, base64 json str')
tdLog.printNoPrefix('-k not kill valgrind processer')
tdLog.printNoPrefix('-e eval str to run')
tdLog.printNoPrefix('-N start dnodes numbers in clusters')
tdLog.printNoPrefix('-M create mnode numbers in clusters')
tdLog.printNoPrefix('-Q set queryPolicy in one dnode')
tdLog.printNoPrefix('-C create Dnode Numbers in one cluster')
tdLog.printNoPrefix('-R restful realization form')
tdLog.printNoPrefix('-D taosadapter update cfg dict ')
tdLog.printNoPrefix('-n the number of replicas')
if key in ["-h", "--help"]:
tdLog.printNoPrefix("A collection of test cases written using Python")
tdLog.printNoPrefix("-f Name of test case file written by Python")
tdLog.printNoPrefix("-p Deploy Path for Simulator")
tdLog.printNoPrefix("-m Master Ip for Simulator")
tdLog.printNoPrefix("-l <True:False> logSql Flag")
tdLog.printNoPrefix("-s stop All dnodes")
tdLog.printNoPrefix("-c Test Cluster Flag")
tdLog.printNoPrefix("-g valgrind Test Flag")
tdLog.printNoPrefix("-r taosd restart test")
tdLog.printNoPrefix("-d update cfg dict, base64 json str")
tdLog.printNoPrefix("-k not kill valgrind processer")
tdLog.printNoPrefix("-e eval str to run")
tdLog.printNoPrefix("-N start dnodes numbers in clusters")
tdLog.printNoPrefix("-M create mnode numbers in clusters")
tdLog.printNoPrefix("-Q set queryPolicy in one dnode")
tdLog.printNoPrefix("-C create Dnode Numbers in one cluster")
tdLog.printNoPrefix("-R restful realization form")
tdLog.printNoPrefix("-D taosadapter update cfg dict ")
tdLog.printNoPrefix("-n the number of replicas")
sys.exit(0)
if key in ['-r', '--restart']:
if key in ["-r", "--restart"]:
restart = True
if key in ['-f', '--file']:
if key in ["-f", "--file"]:
fileName = value
if key in ['-p', '--path']:
if key in ["-p", "--path"]:
deployPath = value
if key in ['-m', '--master']:
if key in ["-m", "--master"]:
masterIp = value
if key in ['-l', '--logSql']:
if (value.upper() == "TRUE"):
if key in ["-l", "--logSql"]:
if value.upper() == "TRUE":
logSql = True
elif (value.upper() == "FALSE"):
elif value.upper() == "FALSE":
logSql = False
else:
tdLog.printNoPrefix("logSql value %s is invalid" % logSql)
sys.exit(0)
if key in ['-c', '--cluster']:
if key in ["-c", "--cluster"]:
testCluster = True
if key in ['-g', '--valgrind']:
if key in ["-g", "--valgrind"]:
valgrind = 1
if key in ['-s', '--stop']:
if key in ["-s", "--stop"]:
stop = 1
if key in ['-d', '--updateCfgDict']:
if key in ["-d", "--updateCfgDict"]:
try:
updateCfgDict = eval(base64.b64decode(value.encode()).decode())
except:
print('updateCfgDict convert fail.')
print("updateCfgDict convert fail.")
sys.exit(0)
if key in ['-k', '--killValgrind']:
if key in ["-k", "--killValgrind"]:
killValgrind = 0
if key in ['-e', '--execCmd']:
if key in ["-e", "--execCmd"]:
try:
execCmd = base64.b64decode(value.encode()).decode()
except:
print('execCmd run fail.')
print("execCmd run fail.")
sys.exit(0)
if key in ['-N', '--dnodeNums']:
if key in ["-N", "--dnodeNums"]:
dnodeNums = value
if key in ['-M', '--mnodeNums']:
if key in ["-M", "--mnodeNums"]:
mnodeNums = value
if key in ['-Q', '--queryPolicy']:
if key in ["-Q", "--queryPolicy"]:
queryPolicy = value
if key in ['-C', '--createDnodeNums']:
if key in ["-C", "--createDnodeNums"]:
createDnodeNums = value
if key in ['-R', '--restful']:
if key in ["-R", "--restful"]:
restful = True
if key in ['-D', '--adaptercfgupdate']:
if key in ["-D", "--adaptercfgupdate"]:
try:
adaptercfgupdate = eval(base64.b64decode(value.encode()).decode())
except:
print('adapter cfg update convert fail.')
print("adapter cfg update convert fail.")
sys.exit(0)
if key in ['-n', '--replicaVar']:
if key in ["-n", "--replicaVar"]:
replicaVar = value
if not execCmd == "":
@ -180,18 +206,21 @@ if __name__ == "__main__":
exec(execCmd)
quit()
if (stop != 0):
if (valgrind == 0):
if stop != 0:
if valgrind == 0:
toBeKilled = "taosd"
else:
toBeKilled = "valgrind.bin"
killCmd = "ps -ef|grep -w %s| grep -v grep | awk '{print $2}' | xargs kill -TERM > /dev/null 2>&1" % toBeKilled
killCmd = (
"ps -ef|grep -w %s| grep -v grep | awk '{print $2}' | xargs kill -TERM > /dev/null 2>&1"
% toBeKilled
)
psCmd = "ps -ef|grep -w %s| grep -v grep | awk '{print $2}'" % toBeKilled
processID = subprocess.check_output(psCmd, shell=True)
while(processID):
while processID:
os.system(killCmd)
time.sleep(1)
processID = subprocess.check_output(psCmd, shell=True)
@ -218,7 +247,7 @@ if __name__ == "__main__":
# psCmd = f"pgrep {toBeKilled}"
processID = subprocess.check_output(psCmd, shell=True)
while(processID):
while processID:
os.system(killCmd)
time.sleep(1)
processID = subprocess.check_output(psCmd, shell=True)
@ -233,9 +262,9 @@ if __name__ == "__main__":
fuserCmd = f"fuser -k -n tcp {port}"
os.system(fuserCmd)
tdLog.info('stop taosadapter')
tdLog.info("stop taosadapter")
tdLog.info('stop All dnodes')
tdLog.info("stop All dnodes")
if masterIp == "":
host = socket.gethostname()
@ -247,33 +276,40 @@ if __name__ == "__main__":
host = masterIp
tdLog.info("Procedures for tdengine deployed in %s" % (host))
if platform.system().lower() == 'windows':
if platform.system().lower() == "windows":
fileName = fileName.replace("/", os.sep)
if (masterIp == "" and not fileName == "0-others\\udf_create.py"):
threading.Thread(target=checkRunTimeError,daemon=True).start()
if masterIp == "" and not fileName == "0-others\\udf_create.py":
threading.Thread(target=checkRunTimeError, daemon=True).start()
tdLog.info("Procedures for testing self-deployment")
tdDnodes.init(deployPath, masterIp)
tdDnodes.setTestCluster(testCluster)
tdDnodes.setValgrind(valgrind)
tdDnodes.stopAll()
key_word = 'tdCases.addWindows'
key_word = "tdCases.addWindows"
is_test_framework = 0
try:
if key_word in open(fileName, encoding='UTF-8').read():
if key_word in open(fileName, encoding="UTF-8").read():
is_test_framework = 1
except Exception as r:
print(r)
updateCfgDictStr = ''
updateCfgDictStr = ""
# adapter_cfg_dict_str = ''
if is_test_framework:
moduleName = fileName.replace(".py", "").replace(os.sep, ".")
uModule = importlib.import_module(moduleName)
try:
ucase = uModule.TDTestCase()
if ((json.dumps(updateCfgDict) == '{}') and hasattr(ucase, 'updatecfgDict')):
if (json.dumps(updateCfgDict) == "{}") and hasattr(
ucase, "updatecfgDict"
):
updateCfgDict = ucase.updatecfgDict
updateCfgDictStr = "-d %s"%base64.b64encode(json.dumps(updateCfgDict).encode()).decode()
if ((json.dumps(adapter_cfg_dict) == '{}') and hasattr(ucase, 'taosadapter_cfg_dict')):
updateCfgDictStr = (
"-d %s"
% base64.b64encode(json.dumps(updateCfgDict).encode()).decode()
)
if (json.dumps(adapter_cfg_dict) == "{}") and hasattr(
ucase, "taosadapter_cfg_dict"
):
adapter_cfg_dict = ucase.taosadapter_cfg_dict
# adapter_cfg_dict_str = f"-D {base64.b64encode(toml.dumps(adapter_cfg_dict).encode()).decode()}"
except Exception as r:
@ -284,8 +320,8 @@ if __name__ == "__main__":
tAdapter.init(deployPath, masterIp)
tAdapter.stop(force_kill=True)
if dnodeNums == 1 :
tdDnodes.deploy(1,updateCfgDict)
if dnodeNums == 1:
tdDnodes.deploy(1, updateCfgDict)
tdDnodes.start(1)
tdCases.logSql(logSql)
if restful:
@ -293,11 +329,11 @@ if __name__ == "__main__":
tAdapter.start()
if queryPolicy != 1:
queryPolicy=int(queryPolicy)
queryPolicy = int(queryPolicy)
if restful:
conn = taosrest.connect(url=f"http://{host}:6041")
else:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
cursor = conn.cursor()
cursor.execute("create qnode on dnode 1")
@ -305,22 +341,29 @@ if __name__ == "__main__":
cursor.execute("show local variables")
res = cursor.fetchall()
for i in range(cursor.rowcount):
if res[i][0] == "queryPolicy" :
if res[i][0] == "queryPolicy":
if int(res[i][1]) == int(queryPolicy):
tdLog.success(f'alter queryPolicy to {queryPolicy} successfully')
tdLog.success(
f"alter queryPolicy to {queryPolicy} successfully"
)
else:
tdLog.debug(res)
tdLog.exit(f"alter queryPolicy to {queryPolicy} failed")
else :
tdLog.debug("create an cluster with %s nodes and make %s dnode as independent mnode"%(dnodeNums,mnodeNums))
dnodeslist = cluster.configure_cluster(dnodeNums=dnodeNums,mnodeNums=mnodeNums)
else:
tdLog.debug(
"create an cluster with %s nodes and make %s dnode as independent mnode"
% (dnodeNums, mnodeNums)
)
dnodeslist = cluster.configure_cluster(
dnodeNums=dnodeNums, mnodeNums=mnodeNums
)
tdDnodes = ClusterDnodes(dnodeslist)
tdDnodes.init(deployPath, masterIp)
tdDnodes.setTestCluster(testCluster)
tdDnodes.setValgrind(valgrind)
tdDnodes.stopAll()
for dnode in tdDnodes.dnodes:
tdDnodes.deploy(dnode.index,{})
tdDnodes.deploy(dnode.index, {})
for dnode in tdDnodes.dnodes:
tdDnodes.starttaosd(dnode.index)
tdCases.logSql(logSql)
@ -330,26 +373,26 @@ if __name__ == "__main__":
tAdapter.start()
if not restful:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
else:
conn = taosrest.connect(url=f"http://{host}:6041")
# tdLog.info(tdDnodes.getSimCfgPath(),host)
if createDnodeNums == 1:
createDnodeNums=dnodeNums
createDnodeNums = dnodeNums
else:
createDnodeNums=createDnodeNums
cluster.create_dnode(conn,createDnodeNums)
createDnodeNums = createDnodeNums
cluster.create_dnode(conn, createDnodeNums)
try:
if cluster.check_dnode(conn) :
if cluster.check_dnode(conn):
print("check dnode ready")
except Exception as r:
print(r)
if queryPolicy != 1:
queryPolicy=int(queryPolicy)
queryPolicy = int(queryPolicy)
if restful:
conn = taosrest.connect(url=f"http://{host}:6041")
else:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
cursor = conn.cursor()
cursor.execute("create qnode on dnode 1")
@ -357,18 +400,20 @@ if __name__ == "__main__":
cursor.execute("show local variables")
res = cursor.fetchall()
for i in range(cursor.rowcount):
if res[i][0] == "queryPolicy" :
if res[i][0] == "queryPolicy":
if int(res[i][1]) == int(queryPolicy):
tdLog.success(f'alter queryPolicy to {queryPolicy} successfully')
tdLog.success(
f"alter queryPolicy to {queryPolicy} successfully"
)
else:
tdLog.debug(res)
tdLog.exit(f"alter queryPolicy to {queryPolicy} failed")
if ucase is not None and hasattr(ucase, 'noConn') and ucase.noConn == True:
if ucase is not None and hasattr(ucase, "noConn") and ucase.noConn == True:
conn = None
else:
if not restful:
conn = taos.connect(host="%s"%(host), config=tdDnodes.sim.getCfgDir())
conn = taos.connect(host="%s" % (host), config=tdDnodes.sim.getCfgDir())
else:
conn = taosrest.connect(url=f"http://{host}:6041")
if is_test_framework:
@ -382,7 +427,7 @@ if __name__ == "__main__":
tdDnodes.setValgrind(valgrind)
tdDnodes.stopAll()
is_test_framework = 0
key_word = 'tdCases.addLinux'
key_word = "tdCases.addLinux"
try:
if key_word in open(fileName).read():
is_test_framework = 1
@ -393,9 +438,9 @@ if __name__ == "__main__":
uModule = importlib.import_module(moduleName)
try:
ucase = uModule.TDTestCase()
if (json.dumps(updateCfgDict) == '{}'):
if json.dumps(updateCfgDict) == "{}":
updateCfgDict = ucase.updatecfgDict
if (json.dumps(adapter_cfg_dict) == '{}'):
if json.dumps(adapter_cfg_dict) == "{}":
adapter_cfg_dict = ucase.taosadapter_cfg_dict
except:
pass
@ -404,8 +449,8 @@ if __name__ == "__main__":
tAdapter.init(deployPath, masterIp)
tAdapter.stop(force_kill=True)
if dnodeNums == 1 :
tdDnodes.deploy(1,updateCfgDict)
if dnodeNums == 1:
tdDnodes.deploy(1, updateCfgDict)
tdDnodes.start(1)
tdCases.logSql(logSql)
@ -414,9 +459,9 @@ if __name__ == "__main__":
tAdapter.start()
if queryPolicy != 1:
queryPolicy=int(queryPolicy)
queryPolicy = int(queryPolicy)
if not restful:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
else:
conn = taosrest.connect(url=f"http://{host}:6041")
# tdSql.init(conn.cursor())
@ -437,23 +482,30 @@ if __name__ == "__main__":
cursor.execute("show local variables")
res = cursor.fetchall()
for i in range(cursor.rowcount):
if res[i][0] == "queryPolicy" :
if res[i][0] == "queryPolicy":
if int(res[i][1]) == int(queryPolicy):
tdLog.success(f'alter queryPolicy to {queryPolicy} successfully')
tdLog.success(
f"alter queryPolicy to {queryPolicy} successfully"
)
else:
tdLog.debug(res)
tdLog.exit(f"alter queryPolicy to {queryPolicy} failed")
else :
tdLog.debug("create an cluster with %s nodes and make %s dnode as independent mnode"%(dnodeNums,mnodeNums))
dnodeslist = cluster.configure_cluster(dnodeNums=dnodeNums,mnodeNums=mnodeNums)
else:
tdLog.debug(
"create an cluster with %s nodes and make %s dnode as independent mnode"
% (dnodeNums, mnodeNums)
)
dnodeslist = cluster.configure_cluster(
dnodeNums=dnodeNums, mnodeNums=mnodeNums
)
tdDnodes = ClusterDnodes(dnodeslist)
tdDnodes.init(deployPath, masterIp)
tdDnodes.setTestCluster(testCluster)
tdDnodes.setValgrind(valgrind)
tdDnodes.stopAll()
for dnode in tdDnodes.dnodes:
tdDnodes.deploy(dnode.index,{})
tdDnodes.deploy(dnode.index, {})
for dnode in tdDnodes.dnodes:
tdDnodes.starttaosd(dnode.index)
tdCases.logSql(logSql)
@ -463,27 +515,27 @@ if __name__ == "__main__":
tAdapter.start()
if not restful:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
else:
conn = taosrest.connect(url=f"http://{host}:6041")
print(tdDnodes.getSimCfgPath(),host)
print(tdDnodes.getSimCfgPath(), host)
if createDnodeNums == 1:
createDnodeNums=dnodeNums
createDnodeNums = dnodeNums
else:
createDnodeNums=createDnodeNums
cluster.create_dnode(conn,createDnodeNums)
createDnodeNums = createDnodeNums
cluster.create_dnode(conn, createDnodeNums)
try:
if cluster.check_dnode(conn) :
if cluster.check_dnode(conn):
print("check dnode ready")
except Exception as r:
print(r)
if queryPolicy != 1:
queryPolicy=int(queryPolicy)
queryPolicy = int(queryPolicy)
if restful:
conn = taosrest.connect(url=f"http://{host}:6041")
else:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
cursor = conn.cursor()
cursor.execute("create qnode on dnode 1")
@ -491,14 +543,15 @@ if __name__ == "__main__":
cursor.execute("show local variables")
res = cursor.fetchall()
for i in range(cursor.rowcount):
if res[i][0] == "queryPolicy" :
if res[i][0] == "queryPolicy":
if int(res[i][1]) == int(queryPolicy):
tdLog.success(f'alter queryPolicy to {queryPolicy} successfully')
tdLog.success(
f"alter queryPolicy to {queryPolicy} successfully"
)
else:
tdLog.debug(res)
tdLog.exit(f"alter queryPolicy to {queryPolicy} failed")
if testCluster:
tdLog.info("Procedures for testing cluster")
if fileName == "all":
@ -508,7 +561,7 @@ if __name__ == "__main__":
else:
tdLog.info("Procedures for testing self-deployment")
if not restful:
conn = taos.connect(host,config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
else:
conn = taosrest.connect(url=f"http://{host}:6041")
@ -527,7 +580,7 @@ if __name__ == "__main__":
tdDnodes.start(1)
time.sleep(1)
if not restful:
conn = taos.connect( host, config=tdDnodes.getSimCfgPath())
conn = taos.connect(host, config=tdDnodes.getSimCfgPath())
else:
conn = taosrest.connect(url=f"http://{host}:6041")
tdLog.info("Procedures for tdengine deployed in %s" % (host))

10
tests/docs-examples-test/csharp.sh
View File

@ -27,11 +27,11 @@ dotnet run --project optsTelnet/optstelnet.csproj
taos -s "drop database if exists test"
dotnet run --project optsJSON/optsJSON.csproj
taos -s "create database if exists test"
dotnet run --project wsConnect/wsConnect.csproj
dotnet run --project wsInsert/wsInsert.csproj
dotnet run --project wsStmt/wsStmt.csproj
dotnet run --project wsQuery/wsQuery.csproj
taos -s "create database if not exists test"
# dotnet run --project wsConnect/wsConnect.csproj
# dotnet run --project wsInsert/wsInsert.csproj
# dotnet run --project wsStmt/wsStmt.csproj
# dotnet run --project wsQuery/wsQuery.csproj
taos -s "drop database if exists test"
taos -s "drop database if exists power"

11
tests/pytest/util/cluster.py
View File

@ -36,7 +36,7 @@ class ConfigureyCluster:
self.portStep = 100
self.mnodeNums = 0
def configure_cluster(self ,dnodeNums=5,mnodeNums=0,startPort=6030,portStep=100,hostname="%s"%hostname):
def configure_cluster(self ,dnodeNums=5,mnodeNums=0,independentMnode=True,startPort=6030,portStep=100,hostname="%s"%hostname):
self.startPort=int(startPort)
self.portStep=int(portStep)
self.hostname=hostname
@ -52,7 +52,7 @@ class ConfigureyCluster:
dnode.addExtraCfg("secondEp", f"{hostname}:{startPort_sec}")
# configure dnoe of independent mnodes
if num <= self.mnodeNums and self.mnodeNums != 0 :
if num <= self.mnodeNums and self.mnodeNums != 0 and independentMnode == True :
dnode.addExtraCfg("supportVnodes", 0)
# print(dnode)
self.dnodes.append(dnode)
@ -67,6 +67,13 @@ class ConfigureyCluster:
tdSql.execute(" create dnode '%s';"%dnode_id)
def create_mnode(self,conn,mnodeNums):
tdSql.init(conn.cursor())
mnodeNums=int(mnodeNums)
for i in range(2,mnodeNums+1):
tdSql.execute(" create mnode on dnode %d;"%i)
def check_dnode(self,conn):
tdSql.init(conn.cursor())

2
tests/pytest/util/dnodes.py
View File

@ -418,7 +418,7 @@ class TDDnode:
if i > 50:
break
with open(logFile) as f:
timeout = time.time() + 60 * 2
timeout = time.time() + 10 * 2
while True:
line = f.readline().encode('utf-8')
if bkey in line:

2
tests/script/jenkins/basic.txt
View File

@ -231,7 +231,7 @@
# ---- stream
./test.sh -f tsim/stream/basic0.sim -g
./test.sh -f tsim/stream/basic1.sim
# TD-20201 ./test.sh -f tsim/stream/basic1.sim
./test.sh -f tsim/stream/basic2.sim
./test.sh -f tsim/stream/drop_stream.sim
./test.sh -f tsim/stream/fillHistoryBasic1.sim

2
tests/script/sh/deploy.sh
View File

@ -139,7 +139,7 @@ echo "fsDebugFlag 143" >> $TAOS_CFG
echo "idxDebugFlag 143" >> $TAOS_CFG
echo "udfDebugFlag 143" >> $TAOS_CFG
echo "smaDebugFlag 143" >> $TAOS_CFG
echo "idxDebugFlag 143" >> $TAOS_CFG
echo "metaDebugFlag 143" >> $TAOS_CFG
echo "numOfLogLines 20000000" >> $TAOS_CFG
echo "asyncLog 0" >> $TAOS_CFG
echo "locale en_US.UTF-8" >> $TAOS_CFG

76
tests/system-test/0-others/compa4096.json Normal file
View File

@ -0,0 +1,76 @@
{
"filetype": "insert",
"cfgdir": "/etc/taos",
"host": "localhost",
"port": 6030,
"rest_port": 6041,
"user": "root",
"password": "taosdata",
"thread_count": 100,
"create_table_thread_count": 24,
"result_file": "taosBenchmark_result.log",
"confirm_parameter_prompt": "no",
"insert_interval": 0,
"num_of_records_per_req": 1000000,
"max_sql_len": 1024000,
"databases": [
{
"dbinfo": {
"name": "db4096",
"drop": "yes",
"replica": 1,
"duration": 10,
"precision": "ms",
"keep": 3650,
"comp": 2,
"vgroups": 2,
"buffer": 1000
},
"super_tables": [
{
"name": "stb0",
"child_table_exists": "no",
"childtable_count": 5,
"childtable_prefix": "ctb0",
"escape_character": "no",
"auto_create_table": "no",
"batch_create_tbl_num": 500,
"data_source": "rand",
"insert_mode": "taosc",
"rollup": null,
"interlace_rows": 0,
"line_protocol": null,
"tcp_transfer": "no",
"insert_rows": 10000,
"childtable_limit": 0,
"childtable_offset": 0,
"rows_per_tbl": 0,
"max_sql_len": 1048576,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 1000,
"start_timestamp": "2022-10-22 17:20:36",
"sample_format": "csv",
"sample_file": "./sample.csv",
"tags_file": "",
"columns": [
{
"type": "INT",
"count": 4094
}
],
"tags": [
{
"type": "TINYINT",
"count": 1
}
]
}
]
}
],
"prepare_rand": 10000,
"chinese": "no",
"streams": false,
"test_log": "/root/testlog/"
}

12
tests/system-test/0-others/compatibility.py
View File

@ -93,6 +93,7 @@ class TDTestCase:
oldServerVersion=tdsqlF.queryResult[0][0]
tdLog.info(f"Base server version is {oldServerVersion}")
tdsqlF.query(f"SELECT CLIENT_VERSION();")
# the oldClientVersion can't be updated in the same python process,so the version is new compiled verison
oldClientVersion=tdsqlF.queryResult[0][0]
tdLog.info(f"Base client version is {oldClientVersion}")
@ -105,7 +106,16 @@ class TDTestCase:
# tdsqlF.query(f"select count(*) from {stb}")
# tdsqlF.checkData(0,0,tableNumbers*recordNumbers1)
os.system("pkill taosd")
sleep(1)
sleep(2)
print(f"start taosd: nohup taosd -c {cPath} & ")
os.system(f" nohup taosd -c {cPath} & " )
sleep(10)
tdLog.info(" LD_LIBRARY_PATH=/usr/lib taosBenchmark -f 0-others/compa4096.json -y ")
os.system("LD_LIBRARY_PATH=/usr/lib taosBenchmark -f 0-others/compa4096.json -y")
os.system("pkill -9 taosd")
tdLog.printNoPrefix("==========step2:update new version ")
self.buildTaosd(bPath)

2
tests/system-test/2-query/cast.py
View File

@ -15,7 +15,7 @@ class TDTestCase:
def init(self, conn, logSql, replicaVar=1):
self.replicaVar = int(replicaVar)
tdLog.debug(f"start to excute {__file__}")
tdSql.init(conn.cursor())
tdSql.init(conn.cursor(), True)
self.dbname = "db"
def __cast_to_bigint(self, col_name, tbname):

12
tests/system-test/6-cluster/5dnode3mnodeAdd1Ddnoe.py
View File

@ -126,14 +126,12 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

11
tests/system-test/6-cluster/5dnode3mnodeRecreateMnode.py
View File

@ -126,14 +126,11 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

11
tests/system-test/6-cluster/5dnode3mnodeRestartDnodeInsertData.py
View File

@ -126,14 +126,11 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

11
tests/system-test/6-cluster/5dnode3mnodeRestartDnodeInsertDataAsync.py
View File

@ -126,14 +126,11 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

12
tests/system-test/6-cluster/5dnode3mnodeSep1VnodeStopDnodeCreateDb.py
View File

@ -120,14 +120,10 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")
@ -199,7 +195,7 @@ class TDTestCase:
def run(self):
# print(self.master_dnode.cfgDict)
self.fiveDnodeThreeMnode(dnodeNumbers=6,mnodeNums=3,restartNumbers=10,stopRole='dnode')
self.fiveDnodeThreeMnode(dnodeNumbers=6,mnodeNums=3,restartNumbers=4,stopRole='dnode')
def stop(self):
tdSql.close()

10
tests/system-test/6-cluster/5dnode3mnodeSep1VnodeStopDnodeCreateStb.py
View File

@ -94,14 +94,10 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

10
tests/system-test/6-cluster/5dnode3mnodeSep1VnodeStopDnodeInsertData.py
View File

@ -124,14 +124,10 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

12
tests/system-test/6-cluster/5dnode3mnodeSep1VnodeStopMnodeCreateDb.py
View File

@ -95,16 +95,10 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdLog.info("create mnode on dnode 2")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdLog.info("create mnode on dnode 3")
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

10
tests/system-test/6-cluster/5dnode3mnodeSep1VnodeStopMnodeCreateDbRep3.py
View File

@ -94,14 +94,10 @@ class TDTestCase:
tdSql.checkData(0,1,'%s:6030'%self.host)
tdSql.checkData(4,1,'%s:6430'%self.host)
clusterComCheck.checkDnodes(dnodeNumbers)
clusterComCheck.checkMnodeStatus(1)
# fisr add three mnodes;
tdLog.info("fisr add three mnodes and check mnode status")
tdSql.execute("create mnode on dnode 2")
clusterComCheck.checkMnodeStatus(2)
tdSql.execute("create mnode on dnode 3")
clusterComCheck.checkMnodeStatus(3)
#check mnode status
tdLog.info("check mnode status")
clusterComCheck.checkMnodeStatus(mnodeNums)
# add some error operations and
tdLog.info("Confirm the status of the dnode again")

Some files were not shown because too many files have changed in this diff Show More