xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch '2.0' of github.com:taosdata/TDengine into szhou/python-udf

This commit is contained in:
shenglian zhou 2023-02-21 14:46:15 +08:00
parent b4d8a327ea d3f257ee5e
commit f801f7303a
104 changed files with 5092 additions and 1411 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -130,3 +130,4 @@ tools/COPYING
tools/BUGS
tools/taos-tools
tools/taosws-rs
tags

12
README-CN.md
View File

@ -104,6 +104,16 @@ sudo yum install -y zlib-devel zlib-static xz-devel snappy-devel jansson jansson
sudo yum config-manager --set-enabled Powertools
```
#### CentOS + devtoolset
除上述编译依赖包，需要执行以下命令：
```
sudo yum install centos-release-scl
sudo yum install devtoolset-9 devtoolset-9-libatomic-devel
scl enable devtoolset-9 -- bash
```
### macOS
```
@ -276,7 +286,7 @@ sudo make install
安装成功后，可以在应用程序中双击 TDengine 图标启动服务，或者在终端中启动 TDengine 服务：
```bash
launchctl start com.tdengine.taosd
sudo launchctl start com.tdengine.taosd
```
用户可以使用 TDengine CLI 来连接 TDengine 服务，在终端中，输入：

12
README.md
View File

@ -111,6 +111,16 @@ If the PowerTools installation fails, you can try to use:
sudo yum config-manager --set-enabled powertools
```
#### For CentOS + devtoolset
Besides above dependencies, please run following commands:
```
sudo yum install centos-release-scl
sudo yum install devtoolset-9 devtoolset-9-libatomic-devel
scl enable devtoolset-9 -- bash
```
### macOS
```
@ -286,7 +296,7 @@ Installing from source code will also configure service management for TDengine.
To start the service after installation, double-click the /applications/TDengine to start the program, or in a terminal, use:
```bash
launchctl start com.tdengine.taosd
sudo launchctl start com.tdengine.taosd
```
Then users can use the TDengine CLI to connect the TDengine server. In a terminal, use:

2
cmake/taostools_CMakeLists.txt.in
View File

@ -2,7 +2,7 @@
# taos-tools
ExternalProject_Add(taos-tools
GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
GIT_TAG 22627d7
GIT_TAG 634399d
SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
BINARY_DIR ""
#BUILD_IN_SOURCE TRUE

12
docs/en/07-develop/03-insert-data/01-sql-writing.mdx
View File

@ -30,25 +30,31 @@ Application programs can execute `INSERT` statement through connectors to insert
The below SQL statement is used to insert one row into table "d1001".
```sql
INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31);
INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31);
```
`ts1` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detial, refer to [TDengine SQL insert timestamp section](/taos-sql/insert).
### Insert Multiple Rows
Multiple rows can be inserted in a single SQL statement. The example below inserts 2 rows into table "d1001".
```sql
INSERT INTO d1001 VALUES (1538548684000, 10.2, 220, 0.23) (1538548696650, 10.3, 218, 0.25);
INSERT INTO d1001 VALUES (ts2, 10.2, 220, 0.23) (ts2, 10.3, 218, 0.25);
```
`ts1` and `ts2` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detial, refer to [TDengine SQL insert timestamp section](/taos-sql/insert).
### Insert into Multiple Tables
Data can be inserted into multiple tables in the same SQL statement. The example below inserts 2 rows into table "d1001" and 1 row into table "d1002".
```sql
INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31) (1538548695000, 12.6, 218, 0.33) d1002 VALUES (1538548696800, 12.3, 221, 0.31);
INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31) (ts2, 12.6, 218, 0.33) d1002 VALUES (ts3, 12.3, 221, 0.31);
```
`ts1`, `ts2` and `ts3` is Unix timestamp, the timestamps which is larger than the difference between current time and KEEP in config is only allowed. For further detial, refer to [TDengine SQL insert timestamp section](/taos-sql/insert).
For more details about `INSERT` please refer to [INSERT](/taos-sql/insert).
:::info

16
docs/en/12-taos-sql/02-database.md
View File

@ -126,6 +126,22 @@ alter_database_option: {
}
```
### ALTER CACHESIZE
The command of changing database configuration parameters is easy to use, but it's hard to determine whether a parameter is proper or not. In this section we will describe how to determine whether cachesize is big enough.
1. How to check cachesize?
You can use `select * from information_schema.ins_databases;` to get the value of cachesize.
2. How to check cacheload?
You can use `show <db_name>.vgroups;` to check the value of cacheload.
3. Determine whether cachesize is big engough
If the value of `cacheload` is very close to the value of `cachesize`, then it's very probably that `cachesize` is too small. If the value of `cacheload` is much smaller than the value of `cachesize`, then `cachesize` is big enough. You can use this simple principle to determine. Depending on how much memory is available in your system, you can choose to double `cachesize` or incrase it by even 5 or more times.
:::note
Other parameters cannot be modified after the database has been created.

4
docs/en/12-taos-sql/10-function.md
View File

@ -877,8 +877,8 @@ INTERP(expr)
- The number of rows in the result set of `INTERP` is determined by the parameter `EVERY(time_unit)`. Starting from timestamp1, one interpolation is performed for every time interval specified `time_unit` parameter. The parameter `time_unit` must be an integer, with no quotes, with a time unit of: a(millisecond)), s(second), m(minute), h(hour), d(day), or w(week). For example, `EVERY(500a)` will interpolate every 500 milliseconds.
- Interpolation is performed based on `FILL` parameter. For more information about FILL clause, see [FILL Clause](../distinguished/#fill-clause).
- `INTERP` can only be used to interpolate in single timeline. So it must be used with `partition by tbname` when it's used on a STable.
- Pseudocolumn `_irowts` can be used along with `INTERP` to return the timestamps associated with interpolation points(support after version 3.0.1.4).
- Pseudocolumn `_isfilled` can be used along with `INTERP` to indicate whether the results are original records or data points generated by interpolation algorithm(support after version 3.0.2.3).
- Pseudocolumn `_irowts` can be used along with `INTERP` to return the timestamps associated with interpolation points(support after version 3.0.2.0).
- Pseudocolumn `_isfilled` can be used along with `INTERP` to indicate whether the results are original records or data points generated by interpolation algorithm(support after version 3.0.3.0).
### LAST

2
docs/en/12-taos-sql/26-udf.md
View File

@ -41,7 +41,7 @@ CREATE AGGREGATE FUNCTION function_name AS library_path OUTPUTTYPE output_type [
```sql
CREATE AGGREGATE FUNCTION l2norm AS "/home/taos/udf_example/libl2norm.so" OUTPUTTYPE DOUBLE bufsize 8;
```
For more information about user-defined functions, see [User-Defined Functions](../../develop/udf).
For more information about user-defined functions, see [User-Defined Functions](/develop/udf).
## Manage UDF

80
docs/en/14-reference/03-connector/08-node.mdx
View File

@ -32,7 +32,9 @@ Please refer to [version support list](/reference/connector#version-support)
## Supported features
### Native connectors
<Tabs defaultValue="native">
<TabItem value="native" label="Native connector">
1. Connection Management
2. General Query
@ -41,12 +43,16 @@ Please refer to [version support list](/reference/connector#version-support)
5. Subscription
6. Schemaless
### REST Connector
</TabItem>
<TabItem value="rest" label="REST connector">
1. Connection Management
2. General Query
3. Continuous Query
</TabItem>
</Tabs>
## Installation Steps
### Pre-installation preparation
@ -115,6 +121,9 @@ npm install @tdengine/rest
### Verify
<Tabs defaultValue="native">
<TabItem value="native" label="Native connector">
After installing the TDengine client, use the `nodejsChecker.js` program to verify that the current environment supports Node.js access to TDengine.
Verification in details:
@ -131,6 +140,28 @@ node nodejsChecker.js host=localhost
- After executing the above steps, the command-line will output the result of `nodejsChecker.js` connecting to the TDengine instance and performing a simple insert and query.
</TabItem>
<TabItem value="rest" label="REST connector">
After installing the TDengine client, use the `restChecker.js` program to verify that the current environment supports Node.js access to TDengine.
Verification in details:
- Create an installation test folder such as `~/tdengine-test`. Download the [restChecker.js source code](https://github.com/taosdata/TDengine/tree/3.0/docs/examples/node/restexample/restChecker.js) to your local.
- Execute the following command from the command-line.
```bash
npm init -y
npm install @tdengine/rest
node restChecker.js
```
- After executing the above steps, the command-line will output the result of `restChecker.js` connecting to the TDengine instance and performing a simple insert and query.
</TabItem>
</Tabs>
## Establishing a connection
Please choose to use one of the connectors.
@ -182,24 +213,69 @@ let cursor = conn.cursor();
#### SQL Write
<Tabs defaultValue="native">
<TabItem value="native" label="native connection">
<NodeInsert />
</TabItem>
<TabItem value="rest" label="REST connection">
```js
{{#include docs/examples/node/restexample/insert_example.js}}
```
</TabItem>
</Tabs>
#### InfluxDB line protocol write
<Tabs defaultValue="native">
<TabItem value="native" label="native connection">
<NodeInfluxLine />
</TabItem>
</Tabs>
#### OpenTSDB Telnet line protocol write
<Tabs defaultValue="native">
<TabItem value="native" label="native connection">
<NodeOpenTSDBTelnet />
</TabItem>
</Tabs>
#### OpenTSDB JSON line protocol write
<Tabs defaultValue="native">
<TabItem value="native" label="native connection">
<NodeOpenTSDBJson />
</TabItem>
</Tabs>
### Querying data
<Tabs defaultValue="native">
<TabItem value="native" label="native connection">
<NodeQuery />
</TabItem>
<TabItem value="rest" label="REST connection">
```js
{{#include docs/examples/node/restexample/query_example.js}}
```
</TabItem>
</Tabs>
## More sample programs

5
docs/en/14-reference/12-config/index.md
View File

@ -106,7 +106,7 @@ The parameters described in this document by the effect that they have on the sy
| Applicable | Server only |
| Meaning | The switch for monitoring inside server. The main object of monitoring is to collect information about load on physical nodes, including CPU usage, memory usage, disk usage, and network bandwidth. Monitoring information is sent over HTTP to the taosKeeper service specified by `monitorFqdn` and `monitorProt`.
| Value Range | 0: monitoring disabled, 1: monitoring enabled |
| Default | 1 |
| Default | 0 |
### monitorFqdn
@ -179,9 +179,10 @@ The parameters described in this document by the effect that they have on the sy
| Attribute | Description |
| -------- | -------------------------------- |
| Applicable | Server only |
| Meaning | count()/hyperloglog() return value or not if the result data is NULL |
| Meaning | count()/hyperloglog() return value or not if the input data is empty or NULL |
| Vlue Range | 0：Return empty line，1：Return 0 |
| Default | 1 |
| Notes | When this parameter is setting to 1, for queries containing GROUP BY, PARTITION BY and INTERVAL clause, and input data in certain groups or windows is empty or NULL, the corresponding groups or windows have no return values |
### maxNumOfDistinctRes

9
docs/en/14-reference/14-taosKeeper.md
View File

@ -24,7 +24,14 @@ You can compile taosKeeper separately and install it. Please refer to the [taosK
taosKeeper needs to be executed on the terminal of the operating system, it supports three configuration methods: [Command-line arguments](#command-line-arguments-in-detail), [environment variable](#environment-variable-in-detail) and [configuration file](#configuration-file-parameters-in-detail). The precedence of those is Command-line, environment variable and configuration file.
**Make sure that the TDengine cluster is running correctly before running taosKeeper. ** Ensure that the monitoring service in TDengine has been started. For more information, see [TDengine Monitoring Configuration](../config/#monitoring).
**Make sure that the TDengine cluster is running correctly before running taosKeeper.** Ensure that the monitoring service in TDengine has been started. At least the values of `monitor` and `monitorFqdn` need to be set in `taos.cfg`.
```shell
monitor 1
monitorFqdn localhost # taoskeeper's FQDN
```
For more information, see [TDengine Monitoring Configuration](../config/#monitoring).
### Command-Line Parameters

19
docs/examples/node/restexample/insert_example.js Normal file
View File

@ -0,0 +1,19 @@
const { options, connect } = require("@tdengine/rest");
async function sqlInsert() {
options.path = "/rest/sql";
options.host = "localhost";
options.port = 6041;
let conn = connect(options);
let cursor = conn.cursor();
try {
let res = await cursor.query('CREATE DATABASE power');
res = await cursor.query('CREATE STABLE power.meters (ts timestamp, current float, voltage int, phase float) TAGS (location binary(64), groupId int)');
res = await cursor.query('INSERT INTO power.d1001 USING power.meters TAGS ("California.SanFrancisco", 2) VALUES (NOW, 10.2, 219, 0.32)');
console.log("res.getResult()", res.getResult());
} catch (err) {
console.log(err);
}
}
sqlInsert();

16
docs/examples/node/restexample/query_example.js Normal file
View File

@ -0,0 +1,16 @@
const { options, connect } = require("@tdengine/rest");
async function query() {
options.path = "/rest/sql";
options.host = "localhost";
options.port = 6041;
let conn = connect(options);
let cursor = conn.cursor();
try {
let res = await cursor.query('select * from power.meters');
console.log("res.getResult()", res.getResult());
} catch (err) {
console.log(err);
}
}
query();

78
docs/examples/node/restexample/restChecker.js Normal file
View File

@ -0,0 +1,78 @@
const { options, connect } = require("@tdengine/rest");
options.path = '/rest/sql/'
options.host = 'localhost';
options.port = 6041;
options.user = "root";
options.passwd = "taosdata";
//optional
// options.url = "http://127.0.0.1:6041";
const db = 'rest_ts_db';
const table = 'rest'
const createDB = `create database if not exists ${db} keep 3650`;
const dropDB = `drop database if exists ${db}`;
const createTB = `create table if not exists ${db}.${table}(ts timestamp,i8 tinyint,i16 smallint,i32 int,i64 bigint,bnr binary(40),nchr nchar(40))`;
const addColumn = `alter table ${db}.${table} add column new_column nchar(40) `;
const dropColumn = `alter table ${db}.${table} drop column new_column`;
const insertSql = `insert into ${db}.${table} values('2022-03-30 18:30:51.567',1,2,3,4,'binary1','nchar1')` +
`('2022-03-30 18:30:51.568',5,6,7,8,'binary2','nchar2')` +
`('2022-03-30 18:30:51.569',9,0,1,2,'binary3','nchar3')`;
const querySql = `select * from ${db}.${table}`;
const errorSql = 'show database';
let conn = connect(options);
let cursor = conn.cursor();
async function execute(sql, pure = true) {
let result = await cursor.query(sql, pure);
// print query result as taos shell
// Get Result object, return Result object.
console.log("result.getResult()",result.getResult());
// Get Meta data, return Meta[]|undefined(when execute failed this is undefined).
console.log("result.getMeta()",result.getMeta());
// Get data,return Array<Array<any>>|undefined(when execute failed this is undefined).
console.log("result.getData()",result.getData());
// Get affect rows,return number|undefined(when execute failed this is undefined).
console.log("result.getAffectRows()",result.getAffectRows());
// Get command,return SQL send to server(need to `query(sql,false)`,set 'pure=false',default true).
console.log("result.getCommand()",result.getCommand());
// Get error code ,return number|undefined(when execute failed this is undefined).
console.log("result.getErrCode()",result.getErrCode());
// Get error string,return string|undefined(when execute failed this is undefined).
console.log("result.getErrStr()",result.getErrStr());
}
(async () => {
// start execute time
let start = new Date().getTime();
await execute(createDB);
console.log("-----------------------------------")
await execute(createTB);
console.log("-----------------------------------")
await execute(addColumn);
console.log("----------------------------------")
await execute(dropColumn);
console.log("-----------------------------------")
await execute(insertSql);
console.log("-----------------------------------")
await execute(querySql);
console.log("-----------------------------------")
await execute(errorSql);
console.log("-----------------------------------")
await execute(dropDB);
// finish time
let end = new Date().getTime();
console.log("total spend time:%d ms",end - start);
})()

2
docs/zh/05-get-started/03-package.md
View File

@ -10,7 +10,7 @@ import PkgListV3 from "/components/PkgListV3";
您可以[用 Docker 立即体验](../../get-started/docker/) TDengine。如果您希望对 TDengine 贡献代码或对内部实现感兴趣，请参考我们的 [TDengine GitHub 主页](https://github.com/taosdata/TDengine) 下载源码构建和安装.
TDengine 完整的软件包包括服务端（taosd）、应用驱动（taosc）、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、命令行程序（CLI，taos）和一些工具软件。目前 taosdump、TDinsight 仅在 Linux 系统上安装和运行，后续将支持 Windows、macOS 等系统。TDengine 除了提供多种语言的连接器之外，还通过 [taosAdapter](../../reference/taosadapter/) 提供 [RESTful 接口](../../connector/rest-api/)。
TDengine 完整的软件包包括服务端（taosd）、应用驱动（taosc）、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、命令行程序（CLI，taos）和一些工具软件。目前 TDinsight 仅在 Linux 系统上安装和运行，后续将支持 Windows、macOS 等系统。TDengine 除了提供多种语言的连接器之外，还通过 [taosAdapter](../../reference/taosadapter/) 提供 [RESTful 接口](../../connector/rest-api/)。
为方便使用，标准的服务端安装包包含了 taosd、taosAdapter、taosc、taos、taosdump、taosBenchmark、TDinsight 安装脚本和示例代码；如果您只需要用到服务端程序和客户端连接的 C/C++ 语言支持，也可以仅下载 Lite 版本的安装包。

12
docs/zh/07-develop/03-insert-data/01-sql-writing.mdx
View File

@ -30,25 +30,31 @@ import PhpStmt from "./_php_stmt.mdx";
下面这条 INSERT 就将一条记录写入到表 d1001 中：
```sql
INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31);
INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31);
```
这里的`ts1`为Unix时间戳(Unix timestamp)，允许插入的最老记录的时间戳，是相对于当前服务器时间，减去配置的 KEEP 值。时间戳详情规则参考 [TDengine SQL数据写入 关于时间戳一节](/taos-sql/insert)
### 一次写入多条
TDengine 支持一次写入多条记录，比如下面这条命令就将两条记录写入到表 d1001 中：
```sql
INSERT INTO d1001 VALUES (1538548684000, 10.2, 220, 0.23) (1538548696650, 10.3, 218, 0.25);
INSERT INTO d1001 VALUES (ts1, 10.2, 220, 0.23) (ts2, 10.3, 218, 0.25);
```
这里的`ts1`和`ts2`为Unix时间戳(Unix timestamp)，允许插入的最老记录的时间戳，是相对于当前服务器时间，减去配置的 KEEP 值。时间戳详情规则参考 [TDengine SQL数据写入 关于时间戳一节](/taos-sql/insert)
### 一次写入多表
TDengine 也支持一次向多个表写入数据，比如下面这条命令就向 d1001 写入两条记录，向 d1002 写入一条记录：
```sql
INSERT INTO d1001 VALUES (1538548685000, 10.3, 219, 0.31) (1538548695000, 12.6, 218, 0.33) d1002 VALUES (1538548696800, 12.3, 221, 0.31);
INSERT INTO d1001 VALUES (ts1, 10.3, 219, 0.31) (ts2, 12.6, 218, 0.33) d1002 VALUES (ts3, 12.3, 221, 0.31);
```
这里的`ts1`、`ts2`和`ts3`为Unix时间戳(Unix timestamp)，允许插入的最老记录的时间戳，是相对于当前服务器时间，减去配置的 KEEP 值。时间戳详情规则参考 [TDengine SQL数据写入 关于时间戳一节](/taos-sql/insert)
详细的 SQL INSERT 语法规则参考 [TDengine SQL 的数据写入](/taos-sql/insert)。
:::info

1
docs/zh/08-connector/10-cpp.mdx
View File

@ -330,6 +330,7 @@ typedef struct taosField {
:::note
2.0 及以上版本 TDengine 推荐数据库应用的每个线程都建立一个独立的连接，或基于线程建立连接池。而不推荐在应用中将该连接 (TAOS\*) 结构体传递到不同的线程共享使用。基于 TAOS 结构体发出的查询、写入等操作具有多线程安全性，但 “USE statement” 等状态量有可能在线程之间相互干扰。此外，C 语言的连接器可以按照需求动态建立面向数据库的新连接（该过程对用户不可见），同时建议只有在程序最后退出的时候才调用 `taos_close()` 关闭连接。
另一个需要注意的是，在上述同步 API 执行过程中，不能调用类似 pthread_cancel 之类的 API 来强制结束线程，因为涉及一些模块的同步操作，如果强制结束线程有可能造成包括但不限于死锁等异常状况。
:::

86
docs/zh/08-connector/35-node.mdx
View File

@ -31,7 +31,8 @@ REST 连接器支持所有能运行 Node.js 的平台。
## 支持的功能特性
### 原生连接器
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接器">
1. 连接管理
2. 普通查询
@ -40,12 +41,17 @@ REST 连接器支持所有能运行 Node.js 的平台。
5. 订阅功能
6. Schemaless
### REST 连接器
</TabItem>
<TabItem value="rest" label="REST 连接器">
1. 连接管理
2. 普通查询
3. 连续查询
</TabItem>
</Tabs>
## 安装步骤
### 安装前准备
@ -114,6 +120,9 @@ npm install @tdengine/rest
### 安装验证
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接器">
在安装好 TDengine 客户端后，使用 nodejsChecker.js 程序能够验证当前环境是否支持 Node.js 方式访问 TDengine。
验证方法：
@ -130,11 +139,35 @@ node nodejsChecker.js host=localhost
- 执行以上步骤后，在命令行会输出 nodejsChecker.js 连接 TDengine 实例，并执行简单插入和查询的结果。
</TabItem>
<TabItem value="rest" label="REST 连接器">
在安装好 TDengine 客户端后，使用 nodejsChecker.js 程序能够验证当前环境是否支持 Node.js 方式访问 TDengine。
验证方法：
- 新建安装验证目录，例如：`~/tdengine-test`，下载 GitHub 上 [restChecker.js 源代码](https://github.com/taosdata/TDengine/tree/3.0/docs/examples/node/restexample/restChecker.js)到本地。
- 在命令行中执行以下命令。
```bash
npm init -y
npm install @tdengine/rest
node restChecker.js
```
- 执行以上步骤后，在命令行会输出 restChecker.js 连接 TDengine 实例，并执行简单插入和查询的结果。
</TabItem>
</Tabs>
## 建立连接
请选择使用一种连接器。
<Tabs defaultValue="rest">
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接">
安装并引用 `@tdengine/client` 包。
@ -181,24 +214,71 @@ let cursor = conn.cursor();
#### SQL 写入
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接">
<NodeInsert />
</TabItem>
<TabItem value="rest" label="REST 连接">
```js
{{#include docs/examples/node/restexample/insert_example.js}}
```
</TabItem>
</Tabs>
#### InfluxDB 行协议写入
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接">
<NodeInfluxLine />
</TabItem>
</Tabs>
#### OpenTSDB Telnet 行协议写入
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接">
<NodeOpenTSDBTelnet />
</TabItem>
</Tabs>
#### OpenTSDB JSON 行协议写入
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接">
<NodeOpenTSDBJson />
</TabItem>
</Tabs>
### 查询数据
<Tabs defaultValue="native">
<TabItem value="native" label="原生连接">
<NodeQuery />
</TabItem>
<TabItem value="rest" label="REST 连接">
```js
{{#include docs/examples/node/restexample/query_example.js}}
```
</TabItem>
</Tabs>
## 更多示例程序

16
docs/zh/12-taos-sql/02-database.md
View File

@ -126,6 +126,22 @@ alter_database_option: {
}
```
### 修改 CACHESIZE
修改数据库参数的命令使用简单，难的是如何确定是否需要修改以及如何修改。本小节描述如何判断数据库的 cachesize 是否够用。
1. 如何查看 cachesize?
通过 select * from information_schema.ins_databases; 可以查看这些 cachesize 的具体值。
2. 如何查看 cacheload?
通过 show <db_name>.vgroups; 可以查看 cacheload
3. 判断 cachesize 是否够用
如果 cacheload 非常接近 cachesize，则 cachesize 可能过小。 如果 cacheload 明显小于 cachesize 则 cachesize 是够用的。可以根据这个原则判断是否需要修改 cachesize 。具体修改值可以根据系统可用内存情况来决定是加倍或者是提高几倍。
:::note
其它参数在 3.0.0.0 中暂不支持修改

4
docs/zh/12-taos-sql/10-function.md
View File

@ -879,8 +879,8 @@ INTERP(expr)
- INTERP 根据 EVERY(time_unit) 字段来确定输出时间范围内的结果条数，即从 timestamp1 开始每隔固定长度的时间（time_unit 值）进行插值，time_unit 可取值时间单位：1a(毫秒)，1s(秒)，1m(分)，1h(小时)，1d(天)，1w(周)。例如 EVERY(500a) 将对于指定数据每500毫秒间隔进行一次插值.
- INTERP 根据 FILL 字段来决定在每个符合输出条件的时刻如何进行插值。关于 FILL 子句如何使用请参考 [FILL 子句](../distinguished/#fill-子句)
- INTERP 只能在一个时间序列内进行插值，因此当作用于超级表时必须跟 partition by tbname 一起使用。
- INTERP 可以与伪列 _irowts 一起使用，返回插值点所对应的时间戳(3.0.1.4版本以后支持)。
- INTERP 可以与伪列 _isfilled 一起使用，显示返回结果是否为原始记录或插值算法产生的数据(3.0.2.3版本以后支持)。
- INTERP 可以与伪列 _irowts 一起使用，返回插值点所对应的时间戳(3.0.2.0版本以后支持)。
- INTERP 可以与伪列 _isfilled 一起使用，显示返回结果是否为原始记录或插值算法产生的数据(3.0.3.0版本以后支持)。
### LAST

7
docs/zh/14-reference/12-config/index.md
View File

@ -106,7 +106,7 @@ taos --dump-config
| 适用范围 | 仅服务端适用 |
| 含义 | 服务器内部的系统监控开关。监控主要负责收集物理节点的负载状况，包括 CPU、内存、硬盘、网络带宽的监控记录，监控信息将通过 HTTP 协议发送给由 `monitorFqdn` 和 `monitorProt` 指定的 TaosKeeper 监控服务 |
| 取值范围 | 0：关闭监控服务， 1：激活监控服务。 |
| 缺省值 | 1 |
| 缺省值 | 0 |
### monitorFqdn
@ -192,14 +192,15 @@ taos --dump-config
| 取值范围 | 0 表示包含函数名，1 表示不包含函数名。 |
| 缺省值 | 0 |
### countAlwaysReturnValue
### countAlwaysReturnValue
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 仅服务端适用 |
| 含义 | count/hyperloglog函数在数据为空或者NULL的情况下是否返回值 |
| 含义 | count/hyperloglog函数在输入数据为空或者NULL的情况下是否返回值 |
| 取值范围 | 0：返回空行，1：返回 0 |
| 缺省值 | 1 |
| 补充说明 | 该参数设置为 1 时，如果查询中含有 GROUP BY，PARTITION BY 以及 INTERVAL 子句且相应的组或窗口内数据为空或者NULL， 对应的组或窗口将不返回查询结果 |
## 区域相关

10
docs/zh/14-reference/14-taosKeeper.md
View File

@ -24,7 +24,15 @@ taosKeeper 安装方式：
taosKeeper 需要在操作系统终端执行，该工具支持三种配置方式：[命令行参数](#命令行参数启动)、[环境变量](#环境变量启动) 和 [配置文件](#配置文件启动)。优先级为：命令行参数、环境变量、配置文件参数。
**在运行 taosKeeper 之前要确保 TDengine 集群与 taosAdapter 已经在正确运行。** 并且 TDengine 已经开启监控服务，具体请参考：[TDengine 监控配置](../config/#监控相关)。
**在运行 taosKeeper 之前要确保 TDengine 集群与 taosAdapter 已经在正确运行。** 并且 TDengine 已经开启监控服务，TDengine 配置文件 `taos.cfg` 中至少需要配置 `monitor` 和 `monitorFqdn`。
```shell
monitor 1
monitorFqdn localhost # taoskeeper 服务的 FQDN
```
TDengine 监控配置相关，具体请参考：[TDengine 监控配置](../config/#监控相关)。
### 命令行参数启动

3
include/common/tdataformat.h
View File

@ -29,6 +29,7 @@ extern "C" {
typedef struct SBuffer SBuffer;
typedef struct SSchema SSchema;
typedef struct SSchema2 SSchema2;
typedef struct STColumn STColumn;
typedef struct STSchema STSchema;
typedef struct SValue SValue;
@ -205,7 +206,7 @@ struct SColData {
int32_t numOfNull; // # of null
int32_t numOfValue; // # of vale
int32_t nVal;
uint8_t flag;
int8_t flag;
uint8_t *pBitMap;
int32_t *aOffset;
int32_t nData;

56
include/common/tmsg.h
View File

@ -144,6 +144,8 @@ typedef enum _mgmt_table {
#define TSDB_ALTER_TABLE_UPDATE_TAG_BYTES 8
#define TSDB_ALTER_TABLE_UPDATE_OPTIONS 9
#define TSDB_ALTER_TABLE_UPDATE_COLUMN_NAME 10
#define TSDB_ALTER_TABLE_ADD_TAG_INDEX 11
#define TSDB_ALTER_TABLE_DROP_TAG_INDEX 12
#define TSDB_FILL_NONE 0
#define TSDB_FILL_NULL 1
@ -294,6 +296,15 @@ struct SSchema {
char name[TSDB_COL_NAME_LEN];
};
struct SSchema2 {
int8_t type;
int8_t flags;
col_id_t colId;
int32_t bytes;
char name[TSDB_COL_NAME_LEN];
char alias[TSDB_COL_NAME_LEN];
};
typedef struct {
char tbName[TSDB_TABLE_NAME_LEN];
char stbName[TSDB_TABLE_NAME_LEN];
@ -347,8 +358,19 @@ void tFreeSSubmitRsp(SSubmitRsp* pRsp);
#define COL_CLR_SET(FLG) ((FLG) &= (~(COL_SET_VAL | COL_SET_NULL)))
#define IS_BSMA_ON(s) (((s)->flags & 0x01) == COL_SMA_ON)
#define IS_IDX_ON(s) (((s)->flags & 0x02) == COL_IDX_ON)
#define IS_SET_NULL(s) (((s)->flags & COL_SET_NULL) == COL_SET_NULL)
#define SSCHMEA_SET_IDX_ON(s) \
do { \
(s)->flags |= COL_IDX_ON; \
} while (0)
#define SSCHMEA_SET_IDX_OFF(s) \
do { \
(s)->flags &= (~COL_IDX_ON); \
} while (0)
#define SSCHMEA_TYPE(s) ((s)->type)
#define SSCHMEA_FLAGS(s) ((s)->flags)
#define SSCHMEA_COLID(s) ((s)->colId)
@ -1240,6 +1262,17 @@ typedef struct {
int32_t tSerializeSDropVnodeReq(void* buf, int32_t bufLen, SDropVnodeReq* pReq);
int32_t tDeserializeSDropVnodeReq(void* buf, int32_t bufLen, SDropVnodeReq* pReq);
typedef struct {
char colName[TSDB_COL_NAME_LEN];
char stb[TSDB_TABLE_FNAME_LEN];
int64_t stbUid;
int64_t dbUid;
int64_t reserved[8];
} SDropIndexReq;
int32_t tSerializeSDropIdxReq(void* buf, int32_t bufLen, SDropIndexReq* pReq);
int32_t tDeserializeSDropIdxReq(void* buf, int32_t bufLen, SDropIndexReq* pReq);
typedef struct {
int64_t dbUid;
char db[TSDB_DB_FNAME_LEN];
@ -2812,6 +2845,22 @@ typedef struct {
int32_t tSerializeSMDropSmaReq(void* buf, int32_t bufLen, SMDropSmaReq* pReq);
int32_t tDeserializeSMDropSmaReq(void* buf, int32_t bufLen, SMDropSmaReq* pReq);
typedef struct {
char dbFName[TSDB_DB_FNAME_LEN];
char stbName[TSDB_TABLE_NAME_LEN];
char colName[TSDB_COL_NAME_LEN];
char idxName[TSDB_COL_NAME_LEN];
int8_t idxType;
} SCreateTagIndexReq;
int32_t tSerializeSCreateTagIdxReq(void* buf, int32_t bufLen, SCreateTagIndexReq* pReq);
int32_t tDeserializeSCreateTagIdxReq(void* buf, int32_t bufLen, SCreateTagIndexReq* pReq);
typedef SMDropSmaReq SDropTagIndexReq;
int32_t tSerializeSDropTagIdxReq(void* buf, int32_t bufLen, SDropTagIndexReq* pReq);
int32_t tDeserializeSDropTagIdxReq(void* buf, int32_t bufLen, SDropTagIndexReq* pReq);
typedef struct {
int8_t version; // for compatibility(default 0)
int8_t intervalUnit; // MACRO: TIME_UNIT_XXX
@ -3307,6 +3356,12 @@ typedef struct {
SArray* aSubmitTbData; // SArray<SSubmitTbData>
} SSubmitReq2;
typedef struct {
SMsgHead header;
int64_t version;
char data[]; // SSubmitReq2
} SSubmitReq2Msg;
int32_t tEncodeSSubmitReq2(SEncoder* pCoder, const SSubmitReq2* pReq);
int32_t tDecodeSSubmitReq2(SDecoder* pCoder, SSubmitReq2* pReq);
void tDestroySSubmitTbData(SSubmitTbData* pTbData, int32_t flag);
@ -3323,6 +3378,7 @@ void tDestroySSubmitRsp2(SSubmitRsp2* pRsp, int32_t flag);
#define TSDB_MSG_FLG_ENCODE 0x1
#define TSDB_MSG_FLG_DECODE 0x2
#define TSDB_MSG_FLG_CMPT 0x3
typedef struct {
union {

3
include/common/tmsgdef.h
View File

@ -220,6 +220,9 @@ enum {
TD_DEF_MSG_TYPE(TDMT_VND_DROP_TTL_TABLE, "vnode-drop-ttl-stb", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_TRIM, "vnode-trim", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_COMMIT, "vnode-commit", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_CREATE_INDEX, "vnode-create-index", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_DROP_INDEX, "vnode-drop-index", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_DISABLE_WRITE, "vnode-disable-write", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_MAX_MSG, "vnd-max", NULL, NULL)

2
include/os/osMemory.h
View File

@ -33,6 +33,8 @@ extern "C" {
#endif // if !defined(WINDOWS)
int32_t taosMemoryDbgInit();
int32_t taosMemoryDbgInitRestore();
void *taosMemoryMalloc(int64_t size);
void *taosMemoryCalloc(int64_t num, int64_t size);
void *taosMemoryRealloc(void *ptr, int64_t size);

9
include/util/taoserror.h
View File

@ -253,6 +253,7 @@ int32_t* taosGetErrno();
#define TSDB_CODE_MND_INVALID_STB_OPTION TAOS_DEF_ERROR_CODE(0, 0x036E)
#define TSDB_CODE_MND_INVALID_ROW_BYTES TAOS_DEF_ERROR_CODE(0, 0x036F)
// mnode-func
#define TSDB_CODE_MND_INVALID_FUNC_NAME TAOS_DEF_ERROR_CODE(0, 0x0370)
// #define TSDB_CODE_MND_INVALID_FUNC_LEN TAOS_DEF_ERROR_CODE(0, 0x0371) // 2.x
@ -265,6 +266,8 @@ int32_t* taosGetErrno();
#define TSDB_CODE_MND_INVALID_FUNC_COMMENT TAOS_DEF_ERROR_CODE(0, 0x0378)
#define TSDB_CODE_MND_INVALID_FUNC_RETRIEVE TAOS_DEF_ERROR_CODE(0, 0x0379)
// mnode-db
#define TSDB_CODE_MND_DB_NOT_SELECTED TAOS_DEF_ERROR_CODE(0, 0x0380)
#define TSDB_CODE_MND_DB_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0381) //
@ -359,12 +362,18 @@ int32_t* taosGetErrno();
#define TSDB_CODE_MND_STREAM_TASK_DROPPED TAOS_DEF_ERROR_CODE(0, 0x03F4)
#define TSDB_CODE_MND_MULTI_REPLICA_SOURCE_DB TAOS_DEF_ERROR_CODE(0, 0x03F5)
#define TSDB_CODE_MND_TOO_MANY_STREAMS TAOS_DEF_ERROR_CODE(0, 0x03F6)
#define TSDB_CODE_MND_INVALID_TARGET_TABLE TAOS_DEF_ERROR_CODE(0, 0x03F7)
// mnode-sma
#define TSDB_CODE_MND_SMA_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0480)
#define TSDB_CODE_MND_SMA_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x0481)
#define TSDB_CODE_MND_INVALID_SMA_OPTION TAOS_DEF_ERROR_CODE(0, 0x0482)
// mnode-tag-indxe
#define TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0483)
#define TSDB_CODE_MND_TAG_INDEX_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x0484)
// dnode
// #define TSDB_CODE_DND_MSG_NOT_PROCESSED TAOS_DEF_ERROR_CODE(0, 0x0400) // 2.x
// #define TSDB_CODE_DND_OUT_OF_MEMORY TAOS_DEF_ERROR_CODE(0, 0x0401) // 2.x

1
include/util/tdef.h
View File

@ -104,6 +104,7 @@ extern const int32_t TYPE_BYTES[16];
#define TSDB_INDEX_TYPE_SMA "SMA"
#define TSDB_INDEX_TYPE_FULLTEXT "FULLTEXT"
#define TSDB_INDEX_TYPE_NORMAL "NORMAL"
#define TSDB_INS_USER_STABLES_DBNAME_COLID 2

7
packaging/tools/mac_before_install.txt
View File

@ -1,6 +1,9 @@
TDengine is a high-efficient, scalable, high-available distributed time-series database, which makes a lot of optimizations on inserting and querying data, which is far more efficient than normal regular databases. So TDengine can meet the high requirements of IOT and other areas on storing and querying a large amount of data.
TDengine is an open-source, cloud-native time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. With its built-in caching, stream processing, and data subscription capabilities, TDengine offers a simplified solution for time-series data processing.
To configure TDengine : edit /etc/taos/taos.cfg
To start service : launchctl start com.tdengine.taosd
To start Taos Adapter : launchctl start com.tdengine.taosadapter
To access TDengine : use taos in shell
To access TDengine : use taos in shell
If you're experiencing problems installing TDengine, check the file /var/log/taos/tdengine_install.log to help troubleshoot the installation.

9
packaging/tools/mac_before_install_client.txt Normal file
View File

@ -0,0 +1,9 @@
TDengine is an open-source, cloud-native time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. With its built-in caching, stream processing, and data subscription capabilities, TDengine offers a simplified solution for time-series data processing.
Once it's installed, please take the steps below:
1: open a terminal/shell in Mac
2: if connecting to Cloud Service, follow the instructions on your cloud service account and configure the environment variable
3: if connecting to another TDengine Service, you can also view help information via "taos --help"
4: execute command taos
If you're experiencing problems installing TDengine, check the file /var/log/taos/tdengine_install.log to help troubleshoot the installation.

8
packaging/tools/make_install.sh
View File

@ -531,13 +531,13 @@ function install_taosadapter_service() {
}
function install_service_on_launchctl() {
${csudouser}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosd.plist > /dev/null 2>&1 || :
${csudo}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosd.plist > /dev/null 2>&1 || :
${csudo}cp ${script_dir}/com.taosdata.taosd.plist /Library/LaunchDaemons/com.taosdata.taosd.plist
${csudouser}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosd.plist > /dev/null 2>&1 || :
${csudo}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosd.plist > /dev/null 2>&1 || :
${csudouser}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist > /dev/null 2>&1 || :
${csudo}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist > /dev/null 2>&1 || :
${csudo}cp ${script_dir}/com.taosdata.taosadapter.plist /Library/LaunchDaemons/com.taosdata.taosadapter.plist
${csudouser}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist > /dev/null 2>&1 || :
${csudo}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist > /dev/null 2>&1 || :
}
function install_service() {

10
packaging/tools/post.sh
View File

@ -48,7 +48,7 @@ fi
data_link_dir="${install_main_dir}/data"
log_link_dir="${install_main_dir}/log"
install_log_path="${log_dir}/taos_install.log"
install_log_path="${log_dir}/tdengine_install.log"
# static directory
cfg_dir="${install_main_dir}/cfg"
@ -573,14 +573,14 @@ function install_service_on_systemd() {
function install_service_on_launchctl() {
if [ -f ${install_main_dir}/service/com.taosdata.taosd.plist ]; then
${csudouser}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosd.plist > /dev/null 2>&1 || :
${csudo}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosd.plist > /dev/null 2>&1 || :
${csudo}cp ${install_main_dir}/service/com.taosdata.taosd.plist /Library/LaunchDaemons/com.taosdata.taosd.plist || :
${csudouser}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosd.plist || :
${csudo}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosd.plist || :
fi
if [ -f ${install_main_dir}/service/com.taosdata.taosadapter.plist ]; then
${csudouser}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist > /dev/null 2>&1 || :
${csudo}launchctl unload -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist > /dev/null 2>&1 || :
${csudo}cp ${install_main_dir}/service/com.taosdata.taosadapter.plist /Library/LaunchDaemons/com.taosdata.taosadapter.plist || :
${csudouser}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist || :
${csudo}launchctl load -w /Library/LaunchDaemons/com.taosdata.taosadapter.plist || :
fi
}

6
packaging/tools/tdengine.iss
View File

@ -58,9 +58,9 @@ Source: {#MyAppSourceDir}{#MyAppDriverName}; DestDir: "{app}\driver"; Flags: igN
Source: {#MyAppSourceDir}{#MyAppIncludeName}; DestDir: "{app}\include"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}{#MyAppExeName}; DestDir: "{app}"; Excludes: {#MyAppExcludeSource} ; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}{#MyAppTaosdemoExeName}; DestDir: "{app}"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}\taos.exe; DestDir: "{app}"; DestName: "{#CusPrompt}.EXE"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}\taosBenchmark.exe; DestDir: "{app}"; DestName: "{#CusPrompt}Benchmark.EXE"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}\taosdump.exe; DestDir: "{app}"; DestName: "{#CusPrompt}dump.EXE"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}\taos.exe; DestDir: "{app}"; DestName: "{#CusPrompt}.exe"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}\taosBenchmark.exe; DestDir: "{app}"; DestName: "{#CusPrompt}Benchmark.exe"; Flags: igNoreversion recursesubdirs createallsubdirs
Source: {#MyAppSourceDir}\taosdump.exe; DestDir: "{app}"; DestName: "{#CusPrompt}dump.exe"; Flags: igNoreversion recursesubdirs createallsubdirs
[run]

4
source/client/inc/clientInt.h
View File

@ -151,6 +151,10 @@ typedef struct STscObj {
SHashObj* pRequests;
} STscObj;
typedef struct STscDbg {
bool memEnable;
} STscDbg;
typedef struct SResultColumn {
union {
char* nullbitmap; // bitmap, one bit for each item in the list

13
source/client/src/clientEnv.c
View File

@ -33,6 +33,7 @@
#define TSC_VAR_NOT_RELEASE 1
#define TSC_VAR_RELEASED 0
STscDbg tscDbg = {0};
SAppInfo appInfo;
int64_t lastClusterId = 0;
int32_t clientReqRefPool = -1;
@ -515,6 +516,18 @@ void tscWriteCrashInfo(int signum, void *sigInfo, void *context) {
}
void taos_init_imp(void) {
#if defined(LINUX)
if (tscDbg.memEnable) {
int32_t code = taosMemoryDbgInit();
if (code) {
printf("failed to init memory dbg, error:%s\n", tstrerror(code));
} else {
tsAsyncLog = false;
printf("memory dbg enabled\n");
}
}
#endif
// In the APIs of other program language, taos_cleanup is not available yet.
// So, to make sure taos_cleanup will be invoked to clean up the allocated resource to suppress the valgrind warning.
atexit(taos_cleanup);

4
source/client/src/clientImpl.c
View File

@ -2346,6 +2346,8 @@ TAOS_RES* taosQueryImpl(TAOS* taos, const char* sql, bool validateOnly) {
return NULL;
}
tscDebug("taos_query start with sql:%s", sql);
SSyncQueryParam* param = taosMemoryCalloc(1, sizeof(SSyncQueryParam));
tsem_init(&param->sem, 0, 0);
@ -2360,6 +2362,8 @@ TAOS_RES* taosQueryImpl(TAOS* taos, const char* sql, bool validateOnly) {
taosMemoryFree(param);
}
tscDebug("taos_query end with sql:%s", sql);
return pRequest;
}

2
source/client/src/clientMain.c
View File

@ -884,7 +884,9 @@ void continueInsertFromCsv(SSqlCallbackWrapper *pWrapper, SRequestObj *pRequest)
void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param) {
int64_t connId = *(int64_t *)taos;
tscDebug("taos_query_a start with sql:%s", sql);
taosAsyncQueryImpl(connId, sql, fp, param, false);
tscDebug("taos_query_a end with sql:%s", sql);
}
void taos_query_a_with_reqid(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param, int64_t reqid) {

5
source/client/src/clientRawBlockWrite.c
View File

@ -25,7 +25,9 @@
#include "tref.h"
#include "ttimer.h"
static tb_uid_t processSuid(tb_uid_t suid, char* db) { return suid + MurmurHash3_32(db, strlen(db)); }
static tb_uid_t processSuid(tb_uid_t suid, char* db){
return suid + MurmurHash3_32(db, strlen(db));
}
static char* buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id,
int8_t t) {
@ -1642,6 +1644,7 @@ static int32_t tmqWriteRawMetaDataImpl(TAOS* taos, void* data, int32_t dataLen)
}
if (strcmp(tbName, pCreateReq.name) == 0) {
cloneSVreateTbReq(&pCreateReq, &pCreateReqDst);
pCreateReqDst->ctb.suid = processSuid(pCreateReqDst->ctb.suid, pRequest->pDb);
tDecoderClear(&decoderTmp);
break;
}

2
source/client/src/clientSml.c
View File

@ -1273,7 +1273,7 @@ static int32_t smlInsertData(SSmlHandle *info) {
}
static void smlPrintStatisticInfo(SSmlHandle *info) {
uError(
uDebug(
"SML:0x%" PRIx64
" smlInsertLines result, code:%d,lineNum:%d,stable num:%d,ctable num:%d,create stable num:%d,alter stable tag num:%d,alter stable col num:%d \
parse cost:%" PRId64 ",schema cost:%" PRId64 ",bind cost:%" PRId64 ",rpc cost:%" PRId64 ",total cost:%" PRId64

2
source/client/src/clientTmq.c
View File

@ -418,7 +418,7 @@ int32_t tmqCommitDone(SMqCommitCbParamSet* pParamSet) {
static void tmqCommitRspCountDown(SMqCommitCbParamSet* pParamSet) {
int32_t waitingRspNum = atomic_sub_fetch_32(&pParamSet->waitingRspNum, 1);
if(ASSERT(waitingRspNum >= 0)){
if (ASSERT(waitingRspNum >= 0)) {
tscError("tmqCommitRspCountDown error:%d", waitingRspNum);
return;
}

2
source/common/src/systable.c
View File

@ -120,6 +120,8 @@ static const SSysDbTableSchema userIdxSchema[] = {
{.name = "table_name", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = false},
{.name = "vgroup_id", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = true},
{.name = "create_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP, .sysInfo = false},
{.name = "column_name", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = false},
{.name = "index_type", .bytes = SYSTABLE_SCH_TABLE_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = false},
};
static const SSysDbTableSchema userStbsSchema[] = {

105
source/common/src/tmsg.c
View File

@ -918,6 +918,65 @@ int32_t tDeserializeSMDropSmaReq(void *buf, int32_t bufLen, SMDropSmaReq *pReq)
tDecoderClear(&decoder);
return 0;
}
int32_t tSerializeSCreateTagIdxReq(void *buf, int32_t bufLen, SCreateTagIndexReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->dbFName) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->stbName) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->colName) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->idxName) < 0) return -1;
if (tEncodeI8(&encoder, pReq->idxType) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSCreateTagIdxReq(void *buf, int32_t bufLen, SCreateTagIndexReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->dbFName) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->stbName) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->colName) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->idxName) < 0) return -1;
if (tDecodeI8(&decoder, &pReq->idxType) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
int32_t tSerializeSDropTagIdxReq(void *buf, int32_t bufLen, SDropTagIndexReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
tEndEncode(&encoder);
if (tEncodeCStr(&encoder, pReq->name) < 0) return -1;
if (tEncodeI8(&encoder, pReq->igNotExists) < 0) return -1;
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSDropTagIdxReq(void *buf, int32_t bufLen, SDropTagIndexReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->name) < 0) return -1;
if (tDecodeI8(&decoder, &pReq->igNotExists) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
int32_t tSerializeSMCreateFullTextReq(void *buf, int32_t bufLen, SMCreateFullTextReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
@ -3984,6 +4043,44 @@ int32_t tDeserializeSDropVnodeReq(void *buf, int32_t bufLen, SDropVnodeReq *pReq
tDecoderClear(&decoder);
return 0;
}
int32_t tSerializeSDropIdxReq(void *buf, int32_t bufLen, SDropIndexReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->colName) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->stb) < 0) return -1;
if (tEncodeI64(&encoder, pReq->stbUid) < 0) return -1;
if (tEncodeI64(&encoder, pReq->dbUid) < 0) return -1;
for (int32_t i = 0; i < 8; ++i) {
if (tEncodeI64(&encoder, pReq->reserved[i]) < 0) return -1;
}
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
// TODO
return 0;
}
int32_t tDeserializeSDropIdxReq(void *buf, int32_t bufLen, SDropIndexReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->colName) < 0) return -1;
if (tDecodeCStrTo(&decoder, pReq->stb) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->stbUid) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->dbUid) < 0) return -1;
for (int32_t i = 0; i < 8; ++i) {
if (tDecodeI64(&decoder, &pReq->reserved[i]) < 0) return -1;
}
tEndDecode(&decoder);
tDecoderClear(&decoder);
// TODO
return 0;
}
int32_t tSerializeSCompactVnodeReq(void *buf, int32_t bufLen, SCompactVnodeReq *pReq) {
SEncoder encoder = {0};
@ -6995,9 +7092,13 @@ void tDestroySSubmitTbData(SSubmitTbData *pTbData, int32_t flag) {
return;
}
if (flag == TSDB_MSG_FLG_ENCODE) {
if (flag == TSDB_MSG_FLG_ENCODE || flag == TSDB_MSG_FLG_CMPT) {
if (pTbData->pCreateTbReq) {
tdDestroySVCreateTbReq(pTbData->pCreateTbReq);
if (flag == TSDB_MSG_FLG_ENCODE) {
tdDestroySVCreateTbReq(pTbData->pCreateTbReq);
} else {
tDestroySVCreateTbReq(pTbData->pCreateTbReq, TSDB_MSG_FLG_DECODE);
}
taosMemoryFree(pTbData->pCreateTbReq);
}

19
source/dnode/mgmt/exe/dmMain.c
View File

@ -29,6 +29,7 @@
#define DM_MACHINE_CODE "Get machine code."
#define DM_VERSION "Print program version."
#define DM_EMAIL "<support@taosdata.com>"
#define DM_MEM_DBG "Enable memory debug"
// clang-format on
static struct {
#ifdef WINDOWS
@ -37,6 +38,7 @@ static struct {
bool dumpConfig;
bool dumpSdb;
bool generateGrant;
bool memDbg;
bool printAuth;
bool printVersion;
bool printHelp;
@ -166,8 +168,10 @@ static int32_t dmParseArgs(int32_t argc, char const *argv[]) {
} else if (strcmp(argv[i], "-e") == 0) {
global.envCmd[cmdEnvIndex] = argv[++i];
cmdEnvIndex++;
} else if (strcmp(argv[i], "-dm") == 0) {
global.memDbg = true;
} else if (strcmp(argv[i], "-h") == 0 || strcmp(argv[i], "--help") == 0 || strcmp(argv[i], "--usage") == 0 ||
strcmp(argv[i], "-?")) {
strcmp(argv[i], "-?") == 0) {
global.printHelp = true;
} else {
}
@ -212,6 +216,7 @@ static void dmPrintHelp() {
printf("%s%s%s%s\n", indent, "-e,", indent, DM_ENV_CMD);
printf("%s%s%s%s\n", indent, "-E,", indent, DM_ENV_FILE);
printf("%s%s%s%s\n", indent, "-k,", indent, DM_MACHINE_CODE);
printf("%s%s%s%s\n", indent, "-dm,", indent, DM_MEM_DBG);
printf("%s%s%s%s\n", indent, "-V,", indent, DM_VERSION);
printf("\n\nReport bugs to %s.\n", DM_EMAIL);
@ -272,6 +277,18 @@ int mainWindows(int argc, char **argv) {
return 0;
}
#if defined(LINUX)
if (global.memDbg) {
int32_t code = taosMemoryDbgInit();
if (code) {
printf("failed to init memory dbg, error:%s\n", tstrerror(code));
return code;
}
tsAsyncLog = false;
printf("memory dbg enabled\n");
}
#endif
if (dmInitLog() != 0) {
printf("failed to start since init log error\n");
taosCleanupArgs();

4
source/dnode/mgmt/mgmt_mnode/src/mmHandle.c
View File

@ -157,6 +157,8 @@ SArray *mmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_MND_AUTH, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_SHOW_VARIABLES, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_SERVER_VERSION, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_CREATE_INDEX, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_DROP_INDEX, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_QUERY, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_MERGE_QUERY, mmPutMsgToQueryQueue, 1) == NULL) goto _OVER;
@ -182,6 +184,8 @@ SArray *mmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIRM_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_HASHRANGE_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_COMPACT_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_INDEX_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_INDEX_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DISABLE_WRITE_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SYNC_TIMEOUT, mmPutMsgToSyncQueue, 1) == NULL) goto _OVER;

2
source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
View File

@ -522,6 +522,8 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_VND_BATCH_DEL, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_COMMIT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SCH_QUERY_HEARTBEAT, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_INDEX, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_INDEX, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DROP, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;

12
source/dnode/mnode/impl/inc/mndDef.h
View File

@ -389,6 +389,18 @@ typedef struct {
SSchemaWrapper schemaTag; // for dstVgroup
} SSmaObj;
typedef struct {
char name[TSDB_TABLE_FNAME_LEN];
char stb[TSDB_TABLE_FNAME_LEN];
char db[TSDB_DB_FNAME_LEN];
char dstTbName[TSDB_TABLE_FNAME_LEN];
char colName[TSDB_COL_NAME_LEN];
int64_t createdTime;
int64_t uid;
int64_t stbUid;
int64_t dbUid;
} SIdxObj;
typedef struct {
char name[TSDB_TABLE_FNAME_LEN];
char db[TSDB_DB_FNAME_LEN];

34
source/dnode/mnode/impl/inc/mndIndex.h Normal file
View File

@ -0,0 +1,34 @@
#ifndef _TD_MND_IDX_H_
#define _TD_MND_IDX_H_
#include "mndInt.h"
#ifdef __cplusplus
extern "C" {
#endif
int32_t mndInitIdx(SMnode *pMnode);
void mndCleanupIdx(SMnode *pMnode);
SIdxObj *mndAcquireIdx(SMnode *pMnode, char *Name);
void mndReleaseIdx(SMnode *pMnode, SIdxObj *pSma);
int32_t mndDropIdxsByStb(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb);
int32_t mndDropIdxsByDb(SMnode *pMnode, STrans *pTrans, SDbObj *pDb);
int32_t mndGetIdxsByTagName(SMnode *pMnode, SStbObj *pStb, char *tagName, SIdxObj *pIdx);
int32_t mndGetTableIdx(SMnode *pMnode, char *tbFName, STableIndexRsp *rsp, bool *exist);
int32_t mndRetrieveTagIdx(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows);
int32_t mndProcessDropTagIdxReq(SRpcMsg *pReq);
int32_t mndSetCreateIdxRedoLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx);
int32_t mndSetCreateIdxCommitLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx);
int32_t mndSetDropIdxRedoLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx);
int32_t mndSetDropIdxCommitLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx);
int32_t mndSetAlterIdxRedoLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx);
int32_t mndSetAlterIdxCommitLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx);
#ifdef __cplusplus
}
#endif
#endif /*_TD_MND_IDX_H_*/

43
source/dnode/mnode/impl/inc/mndIndexComm.h Normal file
View File

@ -0,0 +1,43 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef _TD_MND_IDX_COMM_H_
#define _TD_MND_IDX_COMM_H_
#include "mndInt.h"
#ifdef __cplusplus
extern "C" {
#endif
typedef struct SSIdx {
int type; // sma or idx
void *pIdx;
} SSIdx;
// retrieve sma index and tag index
typedef struct {
void *pSmaIter;
void *pIdxIter;
} SSmaAndTagIter;
int32_t mndAcquireGlobalIdx(SMnode *pMnode, char *name, int type, SSIdx *idx);
#ifdef __cplusplus
}
#endif
#endif /*_TD_MND_IDX_COMM_H_*/

5
source/dnode/mnode/impl/inc/mndStb.h
View File

@ -42,6 +42,11 @@ void mndExtractTbNameFromStbFullName(const char *stbFullName, char *dst, int32_t
const char *mndGetStbStr(const char *src);
int32_t mndAllocStbSchemas(const SStbObj *pOld, SStbObj *pNew);
int32_t mndCheckColAndTagModifiable(SMnode *pMnode, const char *stbFullName, int64_t suid, col_id_t colId);
void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb, int32_t *pContLen, void *alterOriData,
int32_t alterOriDataLen);
#ifdef __cplusplus
}
#endif

2
source/dnode/mnode/impl/src/mndDb.c
View File

@ -17,6 +17,7 @@
#include "mndDb.h"
#include "mndCluster.h"
#include "mndDnode.h"
#include "mndIndex.h"
#include "mndPrivilege.h"
#include "mndShow.h"
#include "mndSma.h"
@ -1053,6 +1054,7 @@ static int32_t mndDropDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb) {
/*if (mndDropTopicByDB(pMnode, pTrans, pDb) != 0) goto _OVER;*/
if (mndDropStreamByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
if (mndDropSmasByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
if (mndDropIdxsByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
if (mndSetDropDbRedoActions(pMnode, pTrans, pDb) != 0) goto _OVER;
if (mndUserRemoveDb(pMnode, pTrans, pDb->name) != 0) goto _OVER;

877
source/dnode/mnode/impl/src/mndIndex.c Normal file
View File

@ -0,0 +1,877 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#define _DEFAULT_SOURCE
#include "mndIndex.h"
#include "mndDb.h"
#include "mndDnode.h"
#include "mndIndexComm.h"
#include "mndInfoSchema.h"
#include "mndMnode.h"
#include "mndPrivilege.h"
#include "mndScheduler.h"
#include "mndShow.h"
#include "mndStb.h"
#include "mndStream.h"
#include "mndTrans.h"
#include "mndUser.h"
#include "mndVgroup.h"
#include "parser.h"
#include "tname.h"
#define TSDB_IDX_VER_NUMBER 1
#define TSDB_IDX_RESERVE_SIZE 64
static SSdbRaw *mndIdxActionEncode(SIdxObj *pSma);
static SSdbRow *mndIdxActionDecode(SSdbRaw *pRaw);
static int32_t mndIdxActionInsert(SSdb *pSdb, SIdxObj *pIdx);
static int32_t mndIdxActionDelete(SSdb *pSdb, SIdxObj *pIdx);
static int32_t mndIdxActionUpdate(SSdb *pSdb, SIdxObj *pOld, SIdxObj *pNew);
static int32_t mndProcessCreateIdxReq(SRpcMsg *pReq);
// static int32_t mndProcessDropIdxReq(SRpcMsg *pReq);
static int32_t mndProcessGetIdxReq(SRpcMsg *pReq);
static int32_t mndProcessGetTbIdxReq(SRpcMsg *pReq);
// static int32_t mndRetrieveIdx(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows);
// static void mndCancelGetNextIdx(SMnode *pMnode, void *pIter);
static void mndDestroyIdxObj(SIdxObj *pIdxObj);
static int32_t mndAddIndex(SMnode *pMnode, SRpcMsg *pReq, SCreateTagIndexReq *req, SDbObj *pDb, SStbObj *pStb);
int32_t mndInitIdx(SMnode *pMnode) {
SSdbTable table = {
.sdbType = SDB_IDX,
.keyType = SDB_KEY_BINARY,
.encodeFp = (SdbEncodeFp)mndIdxActionEncode,
.decodeFp = (SdbDecodeFp)mndIdxActionDecode,
.insertFp = (SdbInsertFp)mndIdxActionInsert,
.updateFp = (SdbUpdateFp)mndIdxActionUpdate,
.deleteFp = (SdbDeleteFp)mndIdxActionDelete,
};
mndSetMsgHandle(pMnode, TDMT_MND_CREATE_INDEX, mndProcessCreateIdxReq);
// mndSetMsgHandle(pMnode, TDMT_MND_DROP_INDEX, mndProcessDropIdxReq);
mndSetMsgHandle(pMnode, TDMT_VND_CREATE_INDEX_RSP, mndTransProcessRsp);
mndSetMsgHandle(pMnode, TDMT_VND_DROP_INDEX_RSP, mndTransProcessRsp);
// mndSetMsgHandle(pMnode, TDMT_MND_CREATE_SMA, mndProcessCreateIdxReq);
// mndSetMsgHandle(pMnode, TDMT_MND_DROP_SMA, mndProcessDropIdxReq);
// mndSetMsgHandle(pMnode, TDMT_VND_CREATE_SMA_RSP, mndTransProcessRsp);
// mndSetMsgHandle(pMnode, TDMT_VND_DROP_SMA_RSP, mndTransProcessRsp);
// mndSetMsgHandle(pMnode, TDMT_MND_GET_INDEX, mndProcessGetIdxReq);
// mndSetMsgHandle(pMnode, TDMT_MND_GET_TABLE_INDEX, mndProcessGetTbIdxReq);
// type same with sma
// mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndRetrieveIdx);
// mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndCancelGetNextIdx);
return sdbSetTable(pMnode->pSdb, table);
}
static int32_t mndFindSuperTableTagId(const SStbObj *pStb, const char *tagName) {
for (int32_t tag = 0; tag < pStb->numOfTags; tag++) {
if (strcasecmp(pStb->pTags[tag].name, tagName) == 0) {
return tag;
}
}
return -1;
}
int mndSetCreateIdxRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb, SIdxObj *pIdx) {
SSdb *pSdb = pMnode->pSdb;
SVgObj *pVgroup = NULL;
void *pIter = NULL;
int32_t contLen;
while (1) {
pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void **)&pVgroup);
if (pIter == NULL) break;
if (!mndVgroupInDb(pVgroup, pDb->uid)) {
sdbRelease(pSdb, pVgroup);
continue;
}
void *pReq = mndBuildVCreateStbReq(pMnode, pVgroup, pStb, &contLen, NULL, 0);
if (pReq == NULL) {
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
return -1;
}
STransAction action = {0};
action.epSet = mndGetVgroupEpset(pMnode, pVgroup);
action.pCont = pReq;
action.contLen = contLen;
action.msgType = TDMT_VND_CREATE_INDEX;
if (mndTransAppendRedoAction(pTrans, &action) != 0) {
taosMemoryFree(pReq);
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
return -1;
}
sdbRelease(pSdb, pVgroup);
}
return 0;
}
static void *mndBuildDropIdxReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStbObj, SIdxObj *pIdx, int32_t *contLen) {
int32_t len = 0;
int32_t ret = 0;
SDropIndexReq req = {0};
memcpy(req.colName, pIdx->colName, sizeof(pIdx->colName));
memcpy(req.stb, pIdx->stb, sizeof(pIdx->stb));
req.dbUid = pIdx->dbUid;
req.stbUid = pIdx->stbUid;
mInfo("idx: %s start to build drop index req", pIdx->name);
len = tSerializeSDropIdxReq(NULL, 0, &req);
if (ret < 0) {
goto _err;
}
len += sizeof(SMsgHead);
SMsgHead *pHead = taosMemoryCalloc(1, len);
if (pHead == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _err;
}
pHead->contLen = htonl(len);
pHead->vgId = htonl(pVgroup->vgId);
void *pBuf = POINTER_SHIFT(pHead, sizeof(SMsgHead));
tSerializeSDropIdxReq(pBuf, len - sizeof(SMsgHead), &req);
*contLen = len;
return pHead;
_err:
return NULL;
}
int mndSetDropIdxRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb, SIdxObj *pIdx) {
SSdb *pSdb = pMnode->pSdb;
SVgObj *pVgroup = NULL;
void *pIter = NULL;
int32_t contLen;
while (1) {
pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void **)&pVgroup);
if (pIter == NULL) break;
if (!mndVgroupInDb(pVgroup, pDb->uid)) {
sdbRelease(pSdb, pVgroup);
continue;
}
int32_t len;
void *pReq = mndBuildDropIdxReq(pMnode, pVgroup, pStb, pIdx, &len);
if (pReq == NULL) {
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
return -1;
}
STransAction action = {0};
action.epSet = mndGetVgroupEpset(pMnode, pVgroup);
action.pCont = pReq;
action.contLen = len;
action.msgType = TDMT_VND_DROP_INDEX;
if (mndTransAppendRedoAction(pTrans, &action) != 0) {
taosMemoryFree(pReq);
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
return -1;
}
sdbRelease(pSdb, pVgroup);
}
return 0;
}
void mndCleanupIdx(SMnode *pMnode) {
// do nothing
return;
}
static SSdbRaw *mndIdxActionEncode(SIdxObj *pIdx) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
// int32_t size =
// sizeof(SSmaObj) + pSma->exprLen + pSma->tagsFilterLen + pSma->sqlLen + pSma->astLen + TSDB_IDX_RESERVE_SIZE;
int32_t size = sizeof(SIdxObj) + TSDB_IDX_RESERVE_SIZE;
SSdbRaw *pRaw = sdbAllocRaw(SDB_IDX, TSDB_IDX_VER_NUMBER, size);
if (pRaw == NULL) goto _OVER;
int32_t dataPos = 0;
SDB_SET_BINARY(pRaw, dataPos, pIdx->name, TSDB_TABLE_FNAME_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pIdx->stb, TSDB_TABLE_FNAME_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pIdx->db, TSDB_DB_FNAME_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pIdx->dstTbName, TSDB_DB_FNAME_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pIdx->colName, TSDB_COL_NAME_LEN, _OVER)
SDB_SET_INT64(pRaw, dataPos, pIdx->createdTime, _OVER)
SDB_SET_INT64(pRaw, dataPos, pIdx->uid, _OVER)
SDB_SET_INT64(pRaw, dataPos, pIdx->stbUid, _OVER)
SDB_SET_INT64(pRaw, dataPos, pIdx->dbUid, _OVER)
SDB_SET_RESERVE(pRaw, dataPos, TSDB_IDX_RESERVE_SIZE, _OVER)
SDB_SET_DATALEN(pRaw, dataPos, _OVER)
terrno = 0;
_OVER:
if (terrno != 0) {
mError("idx:%s, failed to encode to raw:%p since %s", pIdx->name, pRaw, terrstr());
sdbFreeRaw(pRaw);
return NULL;
}
mTrace("idx:%s, encode to raw:%p, row:%p", pIdx->name, pRaw, pIdx);
return pRaw;
}
static SSdbRow *mndIdxActionDecode(SSdbRaw *pRaw) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
SSdbRow *pRow = NULL;
SIdxObj *pIdx = NULL;
int8_t sver = 0;
if (sdbGetRawSoftVer(pRaw, &sver) != 0) goto _OVER;
if (sver != TSDB_IDX_VER_NUMBER) {
terrno = TSDB_CODE_SDB_INVALID_DATA_VER;
goto _OVER;
}
pRow = sdbAllocRow(sizeof(SIdxObj));
if (pRow == NULL) goto _OVER;
pIdx = sdbGetRowObj(pRow);
if (pIdx == NULL) goto _OVER;
int32_t dataPos = 0;
SDB_GET_BINARY(pRaw, dataPos, pIdx->name, TSDB_TABLE_FNAME_LEN, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pIdx->stb, TSDB_TABLE_FNAME_LEN, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pIdx->db, TSDB_DB_FNAME_LEN, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pIdx->dstTbName, TSDB_DB_FNAME_LEN, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pIdx->colName, TSDB_COL_NAME_LEN, _OVER)
SDB_GET_INT64(pRaw, dataPos, &pIdx->createdTime, _OVER)
SDB_GET_INT64(pRaw, dataPos, &pIdx->uid, _OVER)
SDB_GET_INT64(pRaw, dataPos, &pIdx->stbUid, _OVER)
SDB_GET_INT64(pRaw, dataPos, &pIdx->dbUid, _OVER)
SDB_GET_RESERVE(pRaw, dataPos, TSDB_IDX_RESERVE_SIZE, _OVER)
terrno = 0;
_OVER:
if (terrno != 0) {
taosMemoryFree(pRow);
return NULL;
}
mTrace("idx:%s, decode from raw:%p, row:%p", pIdx->name, pRaw, pIdx);
return pRow;
}
static int32_t mndIdxActionInsert(SSdb *pSdb, SIdxObj *pIdx) {
mTrace("idx:%s, perform insert action, row:%p", pIdx->name, pIdx);
return 0;
}
static int32_t mndIdxActionDelete(SSdb *pSdb, SIdxObj *pIdx) {
mTrace("idx:%s, perform delete action, row:%p", pIdx->name, pIdx);
return 0;
}
static int32_t mndIdxActionUpdate(SSdb *pSdb, SIdxObj *pOld, SIdxObj *pNew) {
// lock no not
if (strncmp(pOld->colName, pNew->colName, TSDB_COL_NAME_LEN) != 0) {
memcpy(pOld->colName, pNew->colName, sizeof(pNew->colName));
}
mTrace("idx:%s, perform update action, old row:%p new row:%p", pOld->name, pOld, pNew);
return 0;
}
SIdxObj *mndAcquireIdx(SMnode *pMnode, char *idxName) {
SSdb *pSdb = pMnode->pSdb;
SIdxObj *pIdx = sdbAcquire(pSdb, SDB_IDX, idxName);
if (pIdx == NULL && terrno == TSDB_CODE_SDB_OBJ_NOT_THERE) {
terrno = TSDB_CODE_MND_TAG_INDEX_NOT_EXIST;
}
return pIdx;
}
void mndReleaseIdx(SMnode *pMnode, SIdxObj *pIdx) {
SSdb *pSdb = pMnode->pSdb;
sdbRelease(pSdb, pIdx);
}
SDbObj *mndAcquireDbByIdx(SMnode *pMnode, const char *idxName) {
SName name = {0};
tNameFromString(&name, idxName, T_NAME_ACCT | T_NAME_DB | T_NAME_TABLE);
char db[TSDB_TABLE_FNAME_LEN] = {0};
tNameGetFullDbName(&name, db);
return mndAcquireDb(pMnode, db);
}
int32_t mndSetCreateIdxRedoLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx) {
SSdbRaw *pRedoRaw = mndIdxActionEncode(pIdx);
if (pRedoRaw == NULL) return -1;
if (mndTransAppendRedolog(pTrans, pRedoRaw) != 0) return -1;
if (sdbSetRawStatus(pRedoRaw, SDB_STATUS_CREATING) != 0) return -1;
return 0;
}
int32_t mndSetCreateIdxCommitLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx) {
SSdbRaw *pCommitRaw = mndIdxActionEncode(pIdx);
if (pCommitRaw == NULL) return -1;
if (mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) return -1;
if (sdbSetRawStatus(pCommitRaw, SDB_STATUS_READY) != 0) return -1;
return 0;
}
int32_t mndSetAlterIdxRedoLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx) {
SSdbRaw *pRedoRaw = mndIdxActionEncode(pIdx);
if (pRedoRaw == NULL) return -1;
if (mndTransAppendRedolog(pTrans, pRedoRaw) != 0) {
sdbFreeRaw(pRedoRaw);
return -1;
}
if (sdbSetRawStatus(pRedoRaw, SDB_STATUS_READY) != 0) return -1;
return 0;
}
int32_t mndSetAlterIdxCommitLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx) {
SSdbRaw *pCommitRaw = mndIdxActionEncode(pIdx);
if (pCommitRaw == NULL) return -1;
if (mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) {
sdbFreeRaw(pCommitRaw);
return -1;
}
if (sdbSetRawStatus(pCommitRaw, SDB_STATUS_READY) != 0) return -1;
return 0;
}
static int32_t mndSetCreateIdxVgroupRedoLogs(SMnode *pMnode, STrans *pTrans, SVgObj *pVgroup) {
SSdbRaw *pVgRaw = mndVgroupActionEncode(pVgroup);
if (pVgRaw == NULL) return -1;
if (mndTransAppendRedolog(pTrans, pVgRaw) != 0) return -1;
if (sdbSetRawStatus(pVgRaw, SDB_STATUS_CREATING) != 0) return -1;
return 0;
}
static int32_t mndSetCreateIdxVgroupCommitLogs(SMnode *pMnode, STrans *pTrans, SVgObj *pVgroup) {
SSdbRaw *pVgRaw = mndVgroupActionEncode(pVgroup);
if (pVgRaw == NULL) return -1;
if (mndTransAppendCommitlog(pTrans, pVgRaw) != 0) return -1;
if (sdbSetRawStatus(pVgRaw, SDB_STATUS_READY) != 0) return -1;
return 0;
}
// static int32_t mndSetUpdateIdxStbCommitLogs(SMnode *pMnode, STrans *pTrans, SStbObj *pStb) {
// SStbObj stbObj = {0};
// taosRLockLatch(&pStb->lock);
// memcpy(&stbObj, pStb, sizeof(SStbObj));
// taosRUnLockLatch(&pStb->lock);
// stbObj.numOfColumns = 0;
// stbObj.pColumns = NULL;
// stbObj.numOfTags = 0;
// stbObj.pTags = NULL;
// stbObj.numOfFuncs = 0;
// stbObj.pFuncs = NULL;
// stbObj.updateTime = taosGetTimestampMs();
// stbObj.lock = 0;
// stbObj.tagVer++;
// SSdbRaw *pCommitRaw = mndStbActionEncode(&stbObj);
// if (pCommitRaw == NULL) return -1;
// if (mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) return -1;
// if (sdbSetRawStatus(pCommitRaw, SDB_STATUS_READY) != 0) return -1;
//
// return 0;
//}
static void mndDestroyIdxObj(SIdxObj *pIdxObj) {
if (pIdxObj) {
// do nothing
}
}
static int32_t mndProcessCreateIdxReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
int32_t code = -1;
SStbObj *pStb = NULL;
SIdxObj *pIdx = NULL;
SDbObj *pDb = NULL;
SCreateTagIndexReq createReq = {0};
if (tDeserializeSCreateTagIdxReq(pReq->pCont, pReq->contLen, &createReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
goto _OVER;
}
mInfo("idx:%s start to create", createReq.idxName);
// if (mndCheckCreateIdxReq(&createReq) != 0) {
// goto _OVER;
// }
pDb = mndAcquireDbByStb(pMnode, createReq.stbName);
if (pDb == NULL) {
terrno = TSDB_CODE_MND_INVALID_DB;
goto _OVER;
}
pStb = mndAcquireStb(pMnode, createReq.stbName);
if (pStb == NULL) {
mError("idx:%s, failed to create since stb:%s not exist", createReq.idxName, createReq.stbName);
goto _OVER;
}
SSIdx idx = {0};
if (mndAcquireGlobalIdx(pMnode, createReq.idxName, SDB_IDX, &idx) == 0) {
pIdx = idx.pIdx;
} else {
goto _OVER;
}
if (pIdx != NULL) {
terrno = TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST;
goto _OVER;
}
if (mndCheckDbPrivilege(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) {
goto _OVER;
}
code = mndAddIndex(pMnode, pReq, &createReq, pDb, pStb);
if (terrno == TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST || terrno == TSDB_CODE_MND_TAG_NOT_EXIST) {
return terrno;
} else {
if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS;
}
_OVER:
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) {
mError("stb:%s, failed to create since %s", createReq.idxName, terrstr());
}
mndReleaseStb(pMnode, pStb);
mndReleaseIdx(pMnode, pIdx);
mndReleaseDb(pMnode, pDb);
return code;
}
int32_t mndSetDropIdxRedoLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx) {
SSdbRaw *pRedoRaw = mndIdxActionEncode(pIdx);
if (pRedoRaw == NULL) return -1;
if (mndTransAppendRedolog(pTrans, pRedoRaw) != 0) return -1;
if (sdbSetRawStatus(pRedoRaw, SDB_STATUS_DROPPING) != 0) return -1;
return 0;
}
int32_t mndSetDropIdxCommitLogs(SMnode *pMnode, STrans *pTrans, SIdxObj *pIdx) {
SSdbRaw *pCommitRaw = mndIdxActionEncode(pIdx);
if (pCommitRaw == NULL) return -1;
if (mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) return -1;
if (sdbSetRawStatus(pCommitRaw, SDB_STATUS_DROPPED) != 0) return -1;
return 0;
}
static int32_t mndProcessGetTbIdxReq(SRpcMsg *pReq) {
//
return 0;
}
int32_t mndRetrieveTagIdx(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) {
SMnode *pMnode = pReq->info.node;
SSdb *pSdb = pMnode->pSdb;
int32_t numOfRows = 0;
SIdxObj *pIdx = NULL;
int32_t cols = 0;
SDbObj *pDb = NULL;
if (strlen(pShow->db) > 0) {
pDb = mndAcquireDb(pMnode, pShow->db);
if (pDb == NULL) return 0;
}
SSmaAndTagIter *pIter = pShow->pIter;
int invalid = -1;
while (numOfRows < rows) {
pIter->pIdxIter = sdbFetch(pSdb, SDB_IDX, pIter->pIdxIter, (void **)&pIdx);
if (pIter->pIdxIter == NULL) break;
if (NULL != pDb && pIdx->dbUid != pDb->uid) {
sdbRelease(pSdb, pIdx);
continue;
}
cols = 0;
SName idxName = {0};
tNameFromString(&idxName, pIdx->name, T_NAME_ACCT | T_NAME_DB | T_NAME_TABLE);
char n1[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(n1, (char *)tNameGetTableName(&idxName));
char n2[TSDB_DB_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(n2, (char *)mndGetDbStr(pIdx->db));
SName stbName = {0};
tNameFromString(&stbName, pIdx->stb, T_NAME_ACCT | T_NAME_DB | T_NAME_TABLE);
char n3[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(n3, (char *)tNameGetTableName(&stbName));
SColumnInfoData *pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)n1, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)n2, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)n3, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)&invalid, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)&pIdx->createdTime, false);
char col[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(col, (char *)pIdx->colName);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)col, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
char tag[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(tag, (char *)"tag_index");
colDataAppend(pColInfo, numOfRows, (const char *)tag, false);
numOfRows++;
sdbRelease(pSdb, pIdx);
}
mndReleaseDb(pMnode, pDb);
pShow->numOfRows += numOfRows;
return numOfRows;
}
// static void mndCancelGetNextIdx(SMnode *pMnode, void *pIter) {
// SSdb *pSdb = pMnode->pSdb;
//
// sdbCancelFetch(pSdb, pIter);
//}
static int32_t mndCheckIndexReq(SCreateTagIndexReq *pReq) {
// impl
return TSDB_CODE_SUCCESS;
}
static int32_t mndSetUpdateIdxStbCommitLogs(SMnode *pMnode, STrans *pTrans, SStbObj *pOld, SStbObj *pNew, char *tagName,
int on) {
taosRLockLatch(&pOld->lock);
memcpy(pNew, pOld, sizeof(SStbObj));
taosRUnLockLatch(&pOld->lock);
pNew->pTags = NULL;
pNew->updateTime = taosGetTimestampMs();
pNew->lock = 0;
int32_t tag = mndFindSuperTableTagId(pOld, tagName);
if (tag < 0) {
terrno = TSDB_CODE_MND_TAG_NOT_EXIST;
return -1;
}
col_id_t colId = pOld->pTags[tag].colId;
if (mndCheckColAndTagModifiable(pMnode, pOld->name, pOld->uid, colId) != 0) {
return -1;
}
if (mndAllocStbSchemas(pOld, pNew) != 0) {
return -1;
}
SSchema *pTag = pNew->pTags + tag;
if (on == 1) {
if (IS_IDX_ON(pTag)) {
terrno = TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST;
return -1;
} else {
SSCHMEA_SET_IDX_ON(pTag);
}
} else {
if (!IS_IDX_ON(pTag)) {
terrno = TSDB_CODE_MND_SMA_NOT_EXIST;
} else {
SSCHMEA_SET_IDX_OFF(pTag);
pTag->flags = 0;
}
}
pNew->tagVer++;
SSdbRaw *pCommitRaw = mndStbActionEncode(pNew);
if (pCommitRaw == NULL) return -1;
if (mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) return -1;
if (sdbSetRawStatus(pCommitRaw, SDB_STATUS_READY) != 0) return -1;
return 0;
}
int32_t mndAddIndexImpl(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *pStb, SIdxObj *pIdx) {
// impl later
int32_t code = 0;
SStbObj newStb = {0};
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB_INSIDE, pReq, "create-stb-index");
if (pTrans == NULL) goto _OVER;
// mInfo("trans:%d, used to add index to stb:%s", pTrans->id, pStb->name);
mndTransSetDbName(pTrans, pDb->name, pStb->name);
if (mndTrancCheckConflict(pMnode, pTrans) != 0) goto _OVER;
mndTransSetSerial(pTrans);
if (mndSetCreateIdxRedoLogs(pMnode, pTrans, pIdx) != 0) goto _OVER;
if (mndSetCreateIdxCommitLogs(pMnode, pTrans, pIdx) != 0) goto _OVER;
if (mndSetUpdateIdxStbCommitLogs(pMnode, pTrans, pStb, &newStb, pIdx->colName, 1) != 0) goto _OVER;
if (mndSetCreateIdxRedoActions(pMnode, pTrans, pDb, &newStb, pIdx) != 0) goto _OVER;
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
code = 0;
_OVER:
// mndDestoryIdxObj(pIdx);
if (newStb.pTags != NULL) {
taosMemoryFree(newStb.pTags);
taosMemoryFree(newStb.pColumns);
}
mndTransDrop(pTrans);
return code;
}
static int32_t mndAddIndex(SMnode *pMnode, SRpcMsg *pReq, SCreateTagIndexReq *req, SDbObj *pDb, SStbObj *pStb) {
SIdxObj idxObj = {0};
memcpy(idxObj.name, req->idxName, TSDB_TABLE_FNAME_LEN);
memcpy(idxObj.stb, pStb->name, TSDB_TABLE_FNAME_LEN);
memcpy(idxObj.db, pDb->name, TSDB_DB_FNAME_LEN);
memcpy(idxObj.colName, req->colName, TSDB_COL_NAME_LEN);
idxObj.createdTime = taosGetTimestampMs();
idxObj.uid = mndGenerateUid(req->idxName, strlen(req->idxName));
idxObj.stbUid = pStb->uid;
idxObj.dbUid = pStb->dbUid;
int32_t code = -1;
// SField *pField0 = NULL;
// SStbObj stbObj = {0};
// SStbObj *pNew = &stbObj;
// taosRLockLatch(&pOld->lock);
// memcpy(&stbObj, pOld, sizeof(SStbObj));
// taosRUnLockLatch(&pOld->lock);
// stbObj.pColumns = NULL;
// stbObj.pTags = NULL;
// stbObj.updateTime = taosGetTimestampMs();
// stbObj.lock = 0;
int32_t tag = mndFindSuperTableTagId(pStb, req->colName);
if (tag < 0) {
terrno = TSDB_CODE_MND_TAG_NOT_EXIST;
return -1;
} else if (tag == 0) {
terrno = TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST;
return -1;
}
col_id_t colId = pStb->pTags[tag].colId;
if (mndCheckColAndTagModifiable(pMnode, pStb->name, pStb->uid, colId) != 0) {
return -1;
}
SSchema *pTag = pStb->pTags + tag;
if (IS_IDX_ON(pTag)) {
terrno = TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST;
return -1;
}
code = mndAddIndexImpl(pMnode, pReq, pDb, pStb, &idxObj);
return code;
}
static int32_t mndDropIdx(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SIdxObj *pIdx) {
int32_t code = -1;
SStbObj *pStb = NULL;
STrans *pTrans = NULL;
SStbObj newObj = {0};
pStb = mndAcquireStb(pMnode, pIdx->stb);
if (pStb == NULL) goto _OVER;
pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB, pReq, "drop-index");
if (pTrans == NULL) goto _OVER;
mInfo("trans:%d, used to drop idx:%s", pTrans->id, pIdx->name);
mndTransSetDbName(pTrans, pDb->name, NULL);
if (mndTrancCheckConflict(pMnode, pTrans) != 0) goto _OVER;
mndTransSetSerial(pTrans);
if (mndSetDropIdxRedoLogs(pMnode, pTrans, pIdx) != 0) goto _OVER;
if (mndSetDropIdxCommitLogs(pMnode, pTrans, pIdx) != 0) goto _OVER;
if (mndSetUpdateIdxStbCommitLogs(pMnode, pTrans, pStb, &newObj, pIdx->colName, 0) != 0) goto _OVER;
if (mndSetDropIdxRedoActions(pMnode, pTrans, pDb, &newObj, pIdx) != 0) goto _OVER;
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
code = 0;
_OVER:
taosMemoryFree(newObj.pTags);
taosMemoryFree(newObj.pColumns);
mndTransDrop(pTrans);
mndReleaseStb(pMnode, pStb);
return code;
}
int32_t mndProcessDropTagIdxReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
int32_t code = -1;
SDbObj *pDb = NULL;
SIdxObj *pIdx = NULL;
SDropTagIndexReq req = {0};
if (tDeserializeSDropTagIdxReq(pReq->pCont, pReq->contLen, &req) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
goto _OVER;
}
mInfo("idx:%s, start to drop", req.name);
SSIdx idx = {0};
if (mndAcquireGlobalIdx(pMnode, req.name, SDB_IDX, &idx) == 0) {
pIdx = idx.pIdx;
} else {
goto _OVER;
}
if (pIdx == NULL) {
if (req.igNotExists) {
mInfo("idx:%s, not exist, ignore not exist is set", req.name);
code = 0;
goto _OVER;
} else {
terrno = TSDB_CODE_MND_TAG_INDEX_NOT_EXIST;
goto _OVER;
}
}
pDb = mndAcquireDbByIdx(pMnode, req.name);
if (pDb == NULL) {
terrno = TSDB_CODE_MND_DB_NOT_SELECTED;
goto _OVER;
}
if (mndCheckDbPrivilege(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) {
goto _OVER;
}
code = mndDropIdx(pMnode, pReq, pDb, pIdx);
if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS;
_OVER:
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) {
mError("idx:%s, failed to drop since %s", req.name, terrstr());
}
mndReleaseIdx(pMnode, pIdx);
mndReleaseDb(pMnode, pDb);
return code;
}
static int32_t mndProcessGetIdxReq(SRpcMsg *pReq) {
// do nothing
return 0;
}
int32_t mndDropIdxsByStb(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb) {
SSdb *pSdb = pMnode->pSdb;
void *pIter = NULL;
while (1) {
SIdxObj *pIdx = NULL;
pIter = sdbFetch(pSdb, SDB_IDX, pIter, (void **)&pIdx);
if (pIter == NULL) break;
if (pIdx->stbUid == pStb->uid) {
if (mndSetDropIdxCommitLogs(pMnode, pTrans, pIdx) != 0) {
sdbRelease(pSdb, pIdx);
sdbCancelFetch(pSdb, pIdx);
return -1;
}
}
sdbRelease(pSdb, pIdx);
}
return 0;
}
int32_t mndGetIdxsByTagName(SMnode *pMnode, SStbObj *pStb, char *tagName, SIdxObj *idx) {
SSdb *pSdb = pMnode->pSdb;
void *pIter = NULL;
while (1) {
SIdxObj *pIdx = NULL;
pIter = sdbFetch(pSdb, SDB_IDX, pIter, (void **)&pIdx);
if (pIter == NULL) break;
if (pIdx->stbUid == pStb->uid && strcasecmp(pIdx->colName, tagName) == 0) {
memcpy((char *)idx, (char *)pIdx, sizeof(SIdxObj));
sdbRelease(pSdb, pIdx);
return 0;
}
sdbRelease(pSdb, pIdx);
}
return -1;
}
int32_t mndDropIdxsByDb(SMnode *pMnode, STrans *pTrans, SDbObj *pDb) {
SSdb *pSdb = pMnode->pSdb;
void *pIter = NULL;
while (1) {
SIdxObj *pIdx = NULL;
pIter = sdbFetch(pSdb, SDB_IDX, pIter, (void **)&pIdx);
if (pIter == NULL) break;
if (pIdx->dbUid == pDb->uid) {
if (mndSetDropIdxCommitLogs(pMnode, pTrans, pIdx) != 0) {
sdbRelease(pSdb, pIdx);
sdbCancelFetch(pSdb, pIdx);
return -1;
}
}
sdbRelease(pSdb, pIdx);
}
return 0;
}

57
source/dnode/mnode/impl/src/mndIndexCom.c Normal file
View File

@ -0,0 +1,57 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "mndIndex.h"
#include "mndIndexComm.h"
#include "mndSma.h"
static void *mndGetIdx(SMnode *pMnode, char *name, int type) {
SSdb *pSdb = pMnode->pSdb;
void *pIdx = sdbAcquire(pSdb, type, name);
if (pIdx == NULL && terrno == TSDB_CODE_SDB_OBJ_NOT_THERE) {
terrno = 0;
}
return pIdx;
}
int mndAcquireGlobalIdx(SMnode *pMnode, char *name, int type, SSIdx *idx) {
SSmaObj *pSma = mndGetIdx(pMnode, name, SDB_SMA);
SIdxObj *pIdx = mndGetIdx(pMnode, name, SDB_IDX);
terrno = 0;
if (pSma == NULL && pIdx == NULL) return 0;
if (pSma != NULL) {
if (type == SDB_SMA) {
idx->type = SDB_SMA;
idx->pIdx = pSma;
} else {
mndReleaseSma(pMnode, pSma);
terrno = TSDB_CODE_MND_SMA_ALREADY_EXIST;
return -1;
}
} else {
if (type == SDB_IDX) {
idx->type = SDB_IDX;
idx->pIdx = pIdx;
} else {
mndReleaseIdx(pMnode, pIdx);
terrno = TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST;
return -1;
}
}
return 0;
}

2
source/dnode/mnode/impl/src/mndMain.c
View File

@ -21,6 +21,7 @@
#include "mndDnode.h"
#include "mndFunc.h"
#include "mndGrant.h"
#include "mndIndex.h"
#include "mndInfoSchema.h"
#include "mndMnode.h"
#include "mndPerfSchema.h"
@ -425,6 +426,7 @@ static int32_t mndInitSteps(SMnode *pMnode) {
if (mndAllocStep(pMnode, "mnode-vgroup", mndInitVgroup, mndCleanupVgroup) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-stb", mndInitStb, mndCleanupStb) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-sma", mndInitSma, mndCleanupSma) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-idx", mndInitIdx, mndCleanupIdx) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-infos", mndInitInfos, mndCleanupInfos) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-perfs", mndInitPerfs, mndCleanupPerfs) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-db", mndInitDb, mndCleanupDb) != 0) return -1;

84
source/dnode/mnode/impl/src/mndSma.c
View File

@ -17,6 +17,8 @@
#include "mndSma.h"
#include "mndDb.h"
#include "mndDnode.h"
#include "mndIndex.h"
#include "mndIndexComm.h"
#include "mndInfoSchema.h"
#include "mndMnode.h"
#include "mndPrivilege.h"
@ -43,9 +45,13 @@ static int32_t mndProcessDropSmaReq(SRpcMsg *pReq);
static int32_t mndProcessGetSmaReq(SRpcMsg *pReq);
static int32_t mndProcessGetTbSmaReq(SRpcMsg *pReq);
static int32_t mndRetrieveSma(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows);
static void mndCancelGetNextSma(SMnode *pMnode, void *pIter);
static void mndDestroySmaObj(SSmaObj *pSmaObj);
// sma and tag index comm func
static int32_t mndProcessDropIdxReq(SRpcMsg *pReq);
static int32_t mndRetrieveIdx(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows);
static void mndCancelRetrieveIdx(SMnode *pMnode, void *pIter);
int32_t mndInitSma(SMnode *pMnode) {
SSdbTable table = {
.sdbType = SDB_SMA,
@ -58,14 +64,14 @@ int32_t mndInitSma(SMnode *pMnode) {
};
mndSetMsgHandle(pMnode, TDMT_MND_CREATE_SMA, mndProcessCreateSmaReq);
mndSetMsgHandle(pMnode, TDMT_MND_DROP_SMA, mndProcessDropSmaReq);
mndSetMsgHandle(pMnode, TDMT_MND_DROP_SMA, mndProcessDropIdxReq);
mndSetMsgHandle(pMnode, TDMT_VND_CREATE_SMA_RSP, mndTransProcessRsp);
mndSetMsgHandle(pMnode, TDMT_VND_DROP_SMA_RSP, mndTransProcessRsp);
mndSetMsgHandle(pMnode, TDMT_MND_GET_INDEX, mndProcessGetSmaReq);
mndSetMsgHandle(pMnode, TDMT_MND_GET_TABLE_INDEX, mndProcessGetTbSmaReq);
mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndRetrieveSma);
mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndCancelGetNextSma);
mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndRetrieveIdx);
mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndCancelRetrieveIdx);
return sdbSetTable(pMnode->pSdb, table);
}
@ -712,8 +718,13 @@ static int32_t mndProcessCreateSmaReq(SRpcMsg *pReq) {
terrno = TSDB_CODE_MND_STREAM_ALREADY_EXIST;
goto _OVER;
}
SSIdx idx = {0};
if (mndAcquireGlobalIdx(pMnode, createReq.name, SDB_SMA, &idx) == 0) {
pSma = idx.pIdx;
} else {
goto _OVER;
}
pSma = mndAcquireSma(pMnode, createReq.name);
if (pSma != NULL) {
if (createReq.igExists) {
mInfo("sma:%s, already exist in sma:%s, ignore exist is set", createReq.name, pSma->name);
@ -959,7 +970,12 @@ static int32_t mndProcessDropSmaReq(SRpcMsg *pReq) {
mInfo("sma:%s, start to drop", dropReq.name);
pSma = mndAcquireSma(pMnode, dropReq.name);
SSIdx idx = {0};
if (mndAcquireGlobalIdx(pMnode, dropReq.name, SDB_SMA, &idx) == 0) {
pSma = idx.pIdx;
} else {
goto _OVER;
}
if (pSma == NULL) {
if (dropReq.igNotExists) {
mInfo("sma:%s, not exist, ignore not exist is set", dropReq.name);
@ -998,7 +1014,14 @@ static int32_t mndGetSma(SMnode *pMnode, SUserIndexReq *indexReq, SUserIndexRsp
int32_t code = -1;
SSmaObj *pSma = NULL;
pSma = mndAcquireSma(pMnode, indexReq->indexFName);
SSIdx idx = {0};
if (0 == mndAcquireGlobalIdx(pMnode, indexReq->indexFName, SDB_SMA, &idx)) {
pSma = idx.pIdx;
} else {
*exist = false;
return 0;
}
if (pSma == NULL) {
*exist = false;
return 0;
@ -1207,10 +1230,10 @@ static int32_t mndRetrieveSma(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBloc
pDb = mndAcquireDb(pMnode, pShow->db);
if (pDb == NULL) return 0;
}
SSmaAndTagIter *pIter = pShow->pIter;
while (numOfRows < rows) {
pShow->pIter = sdbFetch(pSdb, SDB_SMA, pShow->pIter, (void **)&pSma);
if (pShow->pIter == NULL) break;
pIter->pSmaIter = sdbFetch(pSdb, SDB_SMA, pIter->pSmaIter, (void **)&pSma);
if (pIter->pSmaIter == NULL) break;
if (NULL != pDb && pSma->dbUid != pDb->uid) {
sdbRelease(pSdb, pSma);
@ -1247,6 +1270,18 @@ static int32_t mndRetrieveSma(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBloc
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)&pSma->createdTime, false);
char col[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(col, (char *)"");
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)col, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
char tag[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(tag, (char *)"sma_index");
colDataAppend(pColInfo, numOfRows, (const char *)tag, false);
numOfRows++;
sdbRelease(pSdb, pSma);
}
@ -1256,7 +1291,30 @@ static int32_t mndRetrieveSma(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBloc
return numOfRows;
}
static void mndCancelGetNextSma(SMnode *pMnode, void *pIter) {
SSdb *pSdb = pMnode->pSdb;
sdbCancelFetch(pSdb, pIter);
// sma and tag index comm func
static int32_t mndProcessDropIdxReq(SRpcMsg *pReq) {
int ret = mndProcessDropSmaReq(pReq);
if (terrno == TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST) {
terrno = 0;
ret = mndProcessDropTagIdxReq(pReq);
}
return ret;
}
static int32_t mndRetrieveIdx(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) {
if (pShow->pIter == NULL) {
pShow->pIter = taosMemoryCalloc(1, sizeof(SSmaAndTagIter));
}
int32_t read = mndRetrieveSma(pReq, pShow, pBlock, rows);
if (read < rows) read += mndRetrieveTagIdx(pReq, pShow, pBlock, rows - read);
return read;
}
static void mndCancelRetrieveIdx(SMnode *pMnode, void *pIter) {
SSmaAndTagIter *p = pIter;
if (p != NULL) {
SSdb *pSdb = pMnode->pSdb;
sdbCancelFetch(pSdb, p->pSmaIter);
sdbCancelFetch(pSdb, p->pIdxIter);
}
taosMemoryFree(p);
}

380
source/dnode/mnode/impl/src/mndStb.c
View File

@ -17,6 +17,7 @@
#include "mndStb.h"
#include "mndDb.h"
#include "mndDnode.h"
#include "mndIndex.h"
#include "mndInfoSchema.h"
#include "mndMnode.h"
#include "mndPerfSchema.h"
@ -49,7 +50,11 @@ static void mndCancelGetNextStb(SMnode *pMnode, void *pIter);
static int32_t mndProcessTableCfgReq(SRpcMsg *pReq);
static int32_t mndAlterStbImp(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *pStb, bool needRsp,
void *alterOriData, int32_t alterOriDataLen);
static int32_t mndCheckColAndTagModifiable(SMnode *pMnode, const char *stbname, int64_t suid, col_id_t colId);
static int32_t mndAlterStbAndUpdateTagIdxImp(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *pStb, bool needRsp,
void *alterOriData, int32_t alterOriDataLen, const SMAlterStbReq *pAlter);
static int32_t mndProcessCreateIndexReq(SRpcMsg *pReq);
static int32_t mndProcessDropIndexReq(SRpcMsg *pReq);
int32_t mndInitStb(SMnode *pMnode) {
SSdbTable table = {
@ -72,7 +77,12 @@ int32_t mndInitStb(SMnode *pMnode) {
mndSetMsgHandle(pMnode, TDMT_MND_TABLE_META, mndProcessTableMetaReq);
mndSetMsgHandle(pMnode, TDMT_MND_TTL_TIMER, mndProcessTtlTimer);
mndSetMsgHandle(pMnode, TDMT_MND_TABLE_CFG, mndProcessTableCfgReq);
// mndSetMsgHandle(pMnode, TDMT_MND_SYSTABLE_RETRIEVE, mndProcessRetrieveStbReq);
// mndSetMsgHandle(pMnode, TDMT_MND_SYSTABLE_RETRIEVE, mndProcessRetrieveStbReq);
// mndSetMsgHandle(pMnode, TDMT_MND_CREATE_INDEX, mndProcessCreateIndexReq);
// mndSetMsgHandle(pMnode, TDMT_MND_DROP_INDEX, mndProcessDropIndexReq);
// mndSetMsgHandle(pMnode, TDMT_VND_CREATE_INDEX_RSP, mndTransProcessRsp);
// mndSetMsgHandle(pMnode, TDMT_VND_DROP_INDEX_RSP, mndTransProcessRsp);
mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_STB, mndRetrieveStb);
mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_STB, mndCancelGetNextStb);
@ -426,8 +436,8 @@ static FORCE_INLINE int32_t schemaExColIdCompare(const void *colId, const void *
return 0;
}
static void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb, int32_t *pContLen,
void *alterOriData, int32_t alterOriDataLen) {
void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb, int32_t *pContLen, void *alterOriData,
int32_t alterOriDataLen) {
SEncoder encoder = {0};
int32_t contLen;
SName name = {0};
@ -802,6 +812,9 @@ int32_t mndBuildStbFromReq(SMnode *pMnode, SStbObj *pDst, SMCreateStbReq *pCreat
SSchema *pSchema = &pDst->pTags[i];
pSchema->type = pField->type;
pSchema->bytes = pField->bytes;
if (i == 0) {
SSCHMEA_SET_IDX_ON(pSchema);
}
memcpy(pSchema->name, pField->name, TSDB_COL_NAME_LEN);
pSchema->colId = pDst->nextColId;
pDst->nextColId++;
@ -1086,7 +1099,7 @@ static int32_t mndCheckAlterStbReq(SMAlterStbReq *pAlter) {
return 0;
}
static int32_t mndAllocStbSchemas(const SStbObj *pOld, SStbObj *pNew) {
int32_t mndAllocStbSchemas(const SStbObj *pOld, SStbObj *pNew) {
pNew->pTags = taosMemoryCalloc(pNew->numOfTags, sizeof(SSchema));
pNew->pColumns = taosMemoryCalloc(pNew->numOfColumns, sizeof(SSchema));
if (pNew->pTags == NULL || pNew->pColumns == NULL) {
@ -1348,6 +1361,10 @@ static int32_t mndDropSuperTableTag(SMnode *pMnode, const SStbObj *pOld, SStbObj
pNew->numOfTags--;
pNew->tagVer++;
// if (mndDropIndexByTag(pMnode, pOld, tagName) != 0) {
// return -1;
// }
mInfo("stb:%s, start to drop tag %s", pNew->name, tagName);
return 0;
}
@ -1609,6 +1626,43 @@ static int32_t mndSetAlterStbRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj
return 0;
}
static int32_t mndSetAlterStbRedoActions2(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SStbObj *pStb,
void *alterOriData, int32_t alterOriDataLen) {
SSdb *pSdb = pMnode->pSdb;
SVgObj *pVgroup = NULL;
void *pIter = NULL;
int32_t contLen;
while (1) {
pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void **)&pVgroup);
if (pIter == NULL) break;
if (!mndVgroupInDb(pVgroup, pDb->uid)) {
sdbRelease(pSdb, pVgroup);
continue;
}
void *pReq = mndBuildVCreateStbReq(pMnode, pVgroup, pStb, &contLen, alterOriData, alterOriDataLen);
if (pReq == NULL) {
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
return -1;
}
STransAction action = {0};
action.epSet = mndGetVgroupEpset(pMnode, pVgroup);
action.pCont = pReq;
action.contLen = contLen;
action.msgType = TDMT_VND_CREATE_INDEX;
if (mndTransAppendRedoAction(pTrans, &action) != 0) {
taosMemoryFree(pReq);
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
return -1;
}
sdbRelease(pSdb, pVgroup);
}
return 0;
}
static int32_t mndBuildStbSchemaImp(SDbObj *pDb, SStbObj *pStb, const char *tbName, STableMetaRsp *pRsp) {
taosRLockLatch(&pStb->lock);
@ -1647,6 +1701,7 @@ static int32_t mndBuildStbSchemaImp(SDbObj *pDb, SStbObj *pStb, const char *tbNa
SSchema *pSrcSchema = &pStb->pTags[i];
memcpy(pSchema->name, pSrcSchema->name, TSDB_COL_NAME_LEN);
pSchema->type = pSrcSchema->type;
pSchema->flags = pSrcSchema->flags;
pSchema->colId = pSrcSchema->colId;
pSchema->bytes = pSrcSchema->bytes;
}
@ -1888,6 +1943,74 @@ _OVER:
return code;
}
static int32_t mndAlterStbAndUpdateTagIdxImp(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *pStb, bool needRsp,
void *alterOriData, int32_t alterOriDataLen, const SMAlterStbReq *pAlter) {
int32_t code = -1;
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB_INSIDE, pReq, "alter-stb");
if (pTrans == NULL) goto _OVER;
mInfo("trans:%d, used to alter stb:%s", pTrans->id, pStb->name);
mndTransSetDbName(pTrans, pDb->name, pStb->name);
if (mndTrancCheckConflict(pMnode, pTrans) != 0) goto _OVER;
if (needRsp) {
void *pCont = NULL;
int32_t contLen = 0;
if (mndBuildSMAlterStbRsp(pDb, pStb, &pCont, &contLen) != 0) goto _OVER;
mndTransSetRpcRsp(pTrans, pCont, contLen);
}
if (pAlter->alterType == TSDB_ALTER_TABLE_DROP_TAG) {
SIdxObj idxObj = {0};
SField *pField0 = taosArrayGet(pAlter->pFields, 0);
bool exist = false;
if (mndGetIdxsByTagName(pMnode, pStb, pField0->name, &idxObj) == 0) {
exist = true;
}
if (mndSetAlterStbRedoLogs(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndSetAlterStbCommitLogs(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (exist == true) {
if (mndSetDropIdxRedoLogs(pMnode, pTrans, &idxObj) != 0) goto _OVER;
if (mndSetDropIdxCommitLogs(pMnode, pTrans, &idxObj) != 0) goto _OVER;
}
if (mndSetAlterStbRedoActions(pMnode, pTrans, pDb, pStb, alterOriData, alterOriDataLen) != 0) goto _OVER;
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
} else if (pAlter->alterType == TSDB_ALTER_TABLE_UPDATE_TAG_NAME) {
SIdxObj idxObj = {0};
SField *pField0 = taosArrayGet(pAlter->pFields, 0);
SField *pField1 = taosArrayGet(pAlter->pFields, 1);
const char *oTagName = pField0->name;
const char *nTagName = pField1->name;
bool exist = false;
if (mndGetIdxsByTagName(pMnode, pStb, pField0->name, &idxObj) == 0) {
exist = true;
}
if (mndSetAlterStbRedoLogs(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndSetAlterStbCommitLogs(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (exist == true) {
memcpy(idxObj.colName, nTagName, strlen(nTagName));
idxObj.colName[strlen(nTagName)] = 0;
if (mndSetAlterIdxRedoLogs(pMnode, pTrans, &idxObj) != 0) goto _OVER;
if (mndSetAlterIdxCommitLogs(pMnode, pTrans, &idxObj) != 0) goto _OVER;
}
if (mndSetAlterStbRedoActions(pMnode, pTrans, pDb, pStb, alterOriData, alterOriDataLen) != 0) goto _OVER;
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
}
code = 0;
_OVER:
mndTransDrop(pTrans);
return code;
}
static int32_t mndAlterStb(SMnode *pMnode, SRpcMsg *pReq, const SMAlterStbReq *pAlter, SDbObj *pDb, SStbObj *pOld) {
bool needRsp = true;
int32_t code = -1;
@ -1901,7 +2024,7 @@ static int32_t mndAlterStb(SMnode *pMnode, SRpcMsg *pReq, const SMAlterStbReq *p
stbObj.pTags = NULL;
stbObj.updateTime = taosGetTimestampMs();
stbObj.lock = 0;
bool updateTagIndex = false;
switch (pAlter->alterType) {
case TSDB_ALTER_TABLE_ADD_TAG:
code = mndAddSuperTableTag(pOld, &stbObj, pAlter->pFields, pAlter->numOfFields);
@ -1909,9 +2032,11 @@ static int32_t mndAlterStb(SMnode *pMnode, SRpcMsg *pReq, const SMAlterStbReq *p
case TSDB_ALTER_TABLE_DROP_TAG:
pField0 = taosArrayGet(pAlter->pFields, 0);
code = mndDropSuperTableTag(pMnode, pOld, &stbObj, pField0->name);
updateTagIndex = true;
break;
case TSDB_ALTER_TABLE_UPDATE_TAG_NAME:
code = mndAlterStbTagName(pMnode, pOld, &stbObj, pAlter->pFields);
updateTagIndex = true;
break;
case TSDB_ALTER_TABLE_UPDATE_TAG_BYTES:
pField0 = taosArrayGet(pAlter->pFields, 0);
@ -1939,7 +2064,11 @@ static int32_t mndAlterStb(SMnode *pMnode, SRpcMsg *pReq, const SMAlterStbReq *p
}
if (code != 0) goto _OVER;
code = mndAlterStbImp(pMnode, pReq, pDb, &stbObj, needRsp, pReq->pCont, pReq->contLen);
if (updateTagIndex == false) {
code = mndAlterStbImp(pMnode, pReq, pDb, &stbObj, needRsp, pReq->pCont, pReq->contLen);
} else {
code = mndAlterStbAndUpdateTagIdxImp(pMnode, pReq, pDb, &stbObj, needRsp, pReq->pCont, pReq->contLen, pAlter);
}
_OVER:
taosMemoryFreeClear(stbObj.pTags);
@ -2072,9 +2201,9 @@ static int32_t mndDropStb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *p
if (mndSetDropStbRedoLogs(pMnode, pTrans, pStb) != 0) goto _OVER;
if (mndSetDropStbCommitLogs(pMnode, pTrans, pStb) != 0) goto _OVER;
if (mndSetDropStbRedoActions(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndDropIdxsByStb(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndDropSmasByStb(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
code = 0;
_OVER:
@ -2182,9 +2311,7 @@ static int32_t mndCheckDropStbForStream(SMnode *pMnode, const char *stbFullName,
return 0;
}
static int32_t mndProcessDropTtltbReq(SRpcMsg *pRsp) {
return 0;
}
static int32_t mndProcessDropTtltbReq(SRpcMsg *pRsp) { return 0; }
static int32_t mndProcessDropStbReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
@ -2499,30 +2626,30 @@ void mndExtractTbNameFromStbFullName(const char *stbFullName, char *dst, int32_t
}
}
//static int32_t mndProcessRetrieveStbReq(SRpcMsg *pReq) {
// SMnode *pMnode = pReq->info.node;
// SShowMgmt *pMgmt = &pMnode->showMgmt;
// SShowObj *pShow = NULL;
// int32_t rowsToRead = SHOW_STEP_SIZE;
// int32_t rowsRead = 0;
// static int32_t mndProcessRetrieveStbReq(SRpcMsg *pReq) {
// SMnode *pMnode = pReq->info.node;
// SShowMgmt *pMgmt = &pMnode->showMgmt;
// SShowObj *pShow = NULL;
// int32_t rowsToRead = SHOW_STEP_SIZE;
// int32_t rowsRead = 0;
//
// SRetrieveTableReq retrieveReq = {0};
// if (tDeserializeSRetrieveTableReq(pReq->pCont, pReq->contLen, &retrieveReq) != 0) {
// terrno = TSDB_CODE_INVALID_MSG;
// return -1;
// }
// SRetrieveTableReq retrieveReq = {0};
// if (tDeserializeSRetrieveTableReq(pReq->pCont, pReq->contLen, &retrieveReq) != 0) {
// terrno = TSDB_CODE_INVALID_MSG;
// return -1;
// }
//
// SMnode *pMnode = pReq->info.node;
// SSdb *pSdb = pMnode->pSdb;
// int32_t numOfRows = 0;
// SDbObj *pDb = NULL;
// ESdbStatus objStatus = 0;
// SMnode *pMnode = pReq->info.node;
// SSdb *pSdb = pMnode->pSdb;
// int32_t numOfRows = 0;
// SDbObj *pDb = NULL;
// ESdbStatus objStatus = 0;
//
// SUserObj *pUser = mndAcquireUser(pMnode, pReq->info.conn.user);
// if (pUser == NULL) return 0;
// bool sysinfo = pUser->sysInfo;
// SUserObj *pUser = mndAcquireUser(pMnode, pReq->info.conn.user);
// if (pUser == NULL) return 0;
// bool sysinfo = pUser->sysInfo;
//
// // Append the information_schema database into the result.
// // Append the information_schema database into the result.
//// if (!pShow->sysDbRsp) {
//// SDbObj infoschemaDb = {0};
//// setInformationSchemaDbCfg(pMnode, &infoschemaDb);
@ -2775,7 +2902,6 @@ void mndExtractTbNameFromStbFullName(const char *stbFullName, char *dst, int32_t
// return TSDB_CODE_SUCCESS;
//}
static int32_t mndRetrieveStb(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) {
SMnode *pMnode = pReq->info.node;
SSdb *pSdb = pMnode->pSdb;
@ -2886,8 +3012,8 @@ static int32_t mndRetrieveStb(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBloc
return numOfRows;
}
static int32_t buildDbColsInfoBlock(const SSDataBlock* p, const SSysTableMeta* pSysDbTableMeta, size_t size,
const char* dbName, const char* tbName) {
static int32_t buildDbColsInfoBlock(const SSDataBlock *p, const SSysTableMeta *pSysDbTableMeta, size_t size,
const char *dbName, const char *tbName) {
char tName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
char dName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
@ -2897,19 +3023,19 @@ static int32_t buildDbColsInfoBlock(const SSDataBlock* p, const SSysTableMeta* p
STR_TO_VARSTR(typeName, "SYSTEM_TABLE");
for (int32_t i = 0; i < size; ++i) {
const SSysTableMeta* pm = &pSysDbTableMeta[i];
// if (pm->sysInfo) {
// continue;
// }
if(tbName[0] && strncmp(tbName, pm->name, TSDB_TABLE_NAME_LEN) != 0){
const SSysTableMeta *pm = &pSysDbTableMeta[i];
// if (pm->sysInfo) {
// continue;
// }
if (tbName[0] && strncmp(tbName, pm->name, TSDB_TABLE_NAME_LEN) != 0) {
continue;
}
STR_TO_VARSTR(tName, pm->name);
for(int32_t j = 0; j < pm->colNum; j++){
for (int32_t j = 0; j < pm->colNum; j++) {
// table name
SColumnInfoData* pColInfoData = taosArrayGet(p->pDataBlock, 0);
SColumnInfoData *pColInfoData = taosArrayGet(p->pDataBlock, 0);
colDataAppend(pColInfoData, numOfRows, tName, false);
// database name
@ -2931,18 +3057,17 @@ static int32_t buildDbColsInfoBlock(const SSDataBlock* p, const SSysTableMeta* p
char colTypeStr[VARSTR_HEADER_SIZE + 32];
int colTypeLen = sprintf(varDataVal(colTypeStr), "%s", tDataTypes[colType].name);
if (colType == TSDB_DATA_TYPE_VARCHAR) {
colTypeLen += sprintf(varDataVal(colTypeStr) + colTypeLen, "(%d)",
(int32_t)(pm->schema[j].bytes - VARSTR_HEADER_SIZE));
colTypeLen +=
sprintf(varDataVal(colTypeStr) + colTypeLen, "(%d)", (int32_t)(pm->schema[j].bytes - VARSTR_HEADER_SIZE));
} else if (colType == TSDB_DATA_TYPE_NCHAR) {
colTypeLen += sprintf(
varDataVal(colTypeStr) + colTypeLen, "(%d)",
(int32_t)((pm->schema[j].bytes - VARSTR_HEADER_SIZE) / TSDB_NCHAR_SIZE));
colTypeLen += sprintf(varDataVal(colTypeStr) + colTypeLen, "(%d)",
(int32_t)((pm->schema[j].bytes - VARSTR_HEADER_SIZE) / TSDB_NCHAR_SIZE));
}
varDataSetLen(colTypeStr, colTypeLen);
colDataAppend(pColInfoData, numOfRows, (char*)colTypeStr, false);
colDataAppend(pColInfoData, numOfRows, (char *)colTypeStr, false);
pColInfoData = taosArrayGet(p->pDataBlock, 5);
colDataAppend(pColInfoData, numOfRows, (const char*)&pm->schema[j].bytes, false);
colDataAppend(pColInfoData, numOfRows, (const char *)&pm->schema[j].bytes, false);
for (int32_t k = 6; k <= 8; ++k) {
pColInfoData = taosArrayGet(p->pDataBlock, k);
colDataAppendNULL(pColInfoData, numOfRows);
@ -2955,11 +3080,12 @@ static int32_t buildDbColsInfoBlock(const SSDataBlock* p, const SSysTableMeta* p
return numOfRows;
}
static int32_t buildSysDbColsInfo(SSDataBlock* p, char* db, char* tb) {
static int32_t buildSysDbColsInfo(SSDataBlock *p, char *db, char *tb) {
size_t size = 0;
const SSysTableMeta* pSysDbTableMeta = NULL;
const SSysTableMeta *pSysDbTableMeta = NULL;
if(db[0] && strncmp(db, TSDB_INFORMATION_SCHEMA_DB, TSDB_DB_FNAME_LEN) != 0 && strncmp(db, TSDB_PERFORMANCE_SCHEMA_DB, TSDB_DB_FNAME_LEN) != 0){
if (db[0] && strncmp(db, TSDB_INFORMATION_SCHEMA_DB, TSDB_DB_FNAME_LEN) != 0 &&
strncmp(db, TSDB_PERFORMANCE_SCHEMA_DB, TSDB_DB_FNAME_LEN) != 0) {
return p->info.rows;
}
@ -2985,7 +3111,7 @@ static int32_t mndRetrieveStbCol(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
if (pDb == NULL) return terrno;
}
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(typeName, "SUPER_TABLE");
while (numOfRows < rows) {
pShow->pIter = sdbFetch(pSdb, SDB_STB, pShow->pIter, (void **)&pStb);
@ -2999,7 +3125,7 @@ static int32_t mndRetrieveStbCol(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
SName name = {0};
char stbName[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
mndExtractTbNameFromStbFullName(pStb->name, &stbName[VARSTR_HEADER_SIZE], TSDB_TABLE_NAME_LEN);
if(pShow->filterTb[0] && strncmp(pShow->filterTb, &stbName[VARSTR_HEADER_SIZE], TSDB_TABLE_NAME_LEN) != 0){
if (pShow->filterTb[0] && strncmp(pShow->filterTb, &stbName[VARSTR_HEADER_SIZE], TSDB_TABLE_NAME_LEN) != 0) {
sdbRelease(pSdb, pStb);
continue;
}
@ -3012,8 +3138,8 @@ static int32_t mndRetrieveStbCol(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
tNameGetDbName(&name, varDataVal(db));
varDataSetLen(db, strlen(varDataVal(db)));
for(int i = 0; i < pStb->numOfColumns; i++){
int32_t cols = 0;
for (int i = 0; i < pStb->numOfColumns; i++) {
int32_t cols = 0;
SColumnInfoData *pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)stbName, false);
@ -3038,16 +3164,15 @@ static int32_t mndRetrieveStbCol(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
colTypeLen += sprintf(varDataVal(colTypeStr) + colTypeLen, "(%d)",
(int32_t)(pStb->pColumns[i].bytes - VARSTR_HEADER_SIZE));
} else if (colType == TSDB_DATA_TYPE_NCHAR) {
colTypeLen += sprintf(
varDataVal(colTypeStr) + colTypeLen, "(%d)",
(int32_t)((pStb->pColumns[i].bytes - VARSTR_HEADER_SIZE) / TSDB_NCHAR_SIZE));
colTypeLen += sprintf(varDataVal(colTypeStr) + colTypeLen, "(%d)",
(int32_t)((pStb->pColumns[i].bytes - VARSTR_HEADER_SIZE) / TSDB_NCHAR_SIZE));
}
varDataSetLen(colTypeStr, colTypeLen);
colDataAppend(pColInfo, numOfRows, (char*)colTypeStr, false);
colDataAppend(pColInfo, numOfRows, (char *)colTypeStr, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char*)&pStb->pColumns[i].bytes, false);
while(cols < pShow->numOfColumns) {
colDataAppend(pColInfo, numOfRows, (const char *)&pStb->pColumns[i].bytes, false);
while (cols < pShow->numOfColumns) {
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppendNULL(pColInfo, numOfRows);
}
@ -3082,3 +3207,136 @@ const char *mndGetStbStr(const char *src) {
if (posStb == NULL) return posDb;
return posStb;
}
static int32_t mndCheckIndexReq(SCreateTagIndexReq *pReq) {
// impl
return TSDB_CODE_SUCCESS;
}
/*int32_t mndAddIndexImpl(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SStbObj *pStb, bool needRsp, void *sql,
int32_t len) {
// impl later
int32_t code = 0;
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB_INSIDE, pReq, "create-stb-index");
if (pTrans == NULL) goto _OVER;
mInfo("trans:%d, used to add index to stb:%s", pTrans->id, pStb->name);
mndTransSetDbName(pTrans, pDb->name, pStb->name);
if (mndTrancCheckConflict(pMnode, pTrans) != 0) goto _OVER;
if (mndSetAlterStbRedoLogs(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndSetAlterStbCommitLogs(pMnode, pTrans, pDb, pStb) != 0) goto _OVER;
if (mndSetAlterStbRedoActions2(pMnode, pTrans, pDb, pStb, sql, len) != 0) goto _OVER;
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
return code;
_OVER:
mndTransDrop(pTrans);
return code;
}
static int32_t mndAddIndex(SMnode *pMnode, SRpcMsg *pReq, SCreateTagIndexReq *tagIdxReq, SDbObj *pDb, SStbObj *pOld) {
bool needRsp = true;
int32_t code = -1;
SField *pField0 = NULL;
SStbObj stbObj = {0};
SStbObj *pNew = &stbObj;
taosRLockLatch(&pOld->lock);
memcpy(&stbObj, pOld, sizeof(SStbObj));
taosRUnLockLatch(&pOld->lock);
stbObj.pColumns = NULL;
stbObj.pTags = NULL;
stbObj.updateTime = taosGetTimestampMs();
stbObj.lock = 0;
int32_t tag = mndFindSuperTableTagIndex(pOld, tagIdxReq->colName);
if (tag < 0) {
terrno = TSDB_CODE_MND_TAG_NOT_EXIST;
return -1;
}
col_id_t colId = pOld->pTags[tag].colId;
if (mndCheckColAndTagModifiable(pMnode, pOld->name, pOld->uid, colId) != 0) {
return -1;
}
if (mndAllocStbSchemas(pOld, pNew) != 0) {
return -1;
}
SSchema *pTag = pNew->pTags + tag;
if (IS_IDX_ON(pTag)) {
terrno = TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST;
return -1;
} else {
pTag->flags |= COL_IDX_ON;
}
pNew->tagVer++;
code = mndAddIndexImpl(pMnode, pReq, pDb, pNew, needRsp, pReq->pCont, pReq->contLen);
return code;
}
static int32_t mndProcessCreateIndexReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
int32_t code = -1;
SDbObj *pDb = NULL;
SStbObj *pStb = NULL;
SCreateTagIndexReq tagIdxReq = {0};
if (tDeserializeSCreateTagIdxReq(pReq->pCont, pReq->contLen, &tagIdxReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
goto _OVER;
}
mInfo("stb:%s, start to alter", tagIdxReq.stbName);
if (mndCheckIndexReq(&tagIdxReq) != TSDB_CODE_SUCCESS) {
goto _OVER;
}
pDb = mndAcquireDbByStb(pMnode, tagIdxReq.dbFName);
if (pDb == NULL) {
terrno = TSDB_CODE_MND_INVALID_DB;
goto _OVER;
}
pStb = mndAcquireStb(pMnode, tagIdxReq.stbName);
if (pStb == NULL) {
terrno = TSDB_CODE_MND_STB_NOT_EXIST;
goto _OVER;
}
if (mndCheckDbPrivilege(pMnode, pReq->info.conn.user, MND_OPER_WRITE_DB, pDb) != 0) {
goto _OVER;
}
code = mndAddIndex(pMnode, pReq, &tagIdxReq, pDb, pStb);
if (terrno == TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST || terrno == TSDB_CODE_MND_TAG_NOT_EXIST) {
return terrno;
} else {
if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS;
}
_OVER:
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) {
mError("stb:%s, failed to create index since %s", tagIdxReq.stbName, terrstr());
}
mndReleaseStb(pMnode, pStb);
mndReleaseDb(pMnode, pDb);
return code;
}
static int32_t mndProcessDropIndexReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
int32_t code = -1;
SDbObj *pDb = NULL;
SStbObj *pStb = NULL;
SDropTagIndexReq dropReq = {0};
if (tDeserializeSDropTagIdxReq(pReq->pCont, pReq->contLen, &dropReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
goto _OVER;
}
//
return TSDB_CODE_SUCCESS;
_OVER:
return code;
}*/

12
source/dnode/mnode/impl/src/mndStream.c
View File

@ -31,7 +31,7 @@
#define MND_STREAM_VER_NUMBER 2
#define MND_STREAM_RESERVE_SIZE 64
#define MND_STREAM_MAX_NUM 10
#define MND_STREAM_MAX_NUM 60
static int32_t mndStreamActionInsert(SSdb *pSdb, SStreamObj *pStream);
static int32_t mndStreamActionDelete(SSdb *pSdb, SStreamObj *pStream);
@ -680,7 +680,7 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
pIter = sdbFetch(pMnode->pSdb, SDB_STREAM, pIter, (void **)&pStream);
if (pIter == NULL) {
if (numOfStream > MND_STREAM_MAX_NUM) {
mError("too many streams, no more than 10 for each database");
mError("too many streams, no more than %d for each database", MND_STREAM_MAX_NUM);
terrno = TSDB_CODE_MND_TOO_MANY_STREAMS;
goto _OVER;
}
@ -692,10 +692,16 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
}
sdbRelease(pMnode->pSdb, pStream);
if (numOfStream > MND_STREAM_MAX_NUM) {
mError("too many streams, no more than 10 for each database");
mError("too many streams, no more than %d for each database", MND_STREAM_MAX_NUM);
terrno = TSDB_CODE_MND_TOO_MANY_STREAMS;
goto _OVER;
}
if (pStream->targetStbUid == streamObj.targetStbUid) {
mError("Cannot write the same stable as other stream:%s", pStream->name);
terrno = TSDB_CODE_MND_INVALID_TARGET_TABLE;
goto _OVER;
}
}
}

6
source/dnode/mnode/impl/src/mndTrans.c
View File

@ -1008,6 +1008,12 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) {
if (0 == mndBuildSMCreateStbRsp(pMnode, pTrans->dbname, pTrans->stbname, &pCont, &contLen) != 0) {
mndTransSetRpcRsp(pTrans, pCont, contLen);
}
} else if (pTrans->originRpcType == TDMT_MND_CREATE_INDEX) {
void *pCont = NULL;
int32_t contLen = 0;
if (0 == mndBuildSMCreateStbRsp(pMnode, pTrans->dbname, pTrans->stbname, &pCont, &contLen) != 0) {
mndTransSetRpcRsp(pTrans, pCont, contLen);
}
}
if (pTrans->rpcRspLen != 0) {

3
source/dnode/mnode/sdb/inc/sdb.h
View File

@ -147,7 +147,8 @@ typedef enum {
SDB_STB = 18,
SDB_DB = 19,
SDB_FUNC = 20,
SDB_MAX = 21
SDB_IDX = 21,
SDB_MAX = 22
} ESdbType;
typedef struct SSdbRaw {

2
source/dnode/mnode/sdb/src/sdbHash.c
View File

@ -60,6 +60,8 @@ const char *sdbTableName(ESdbType type) {
return "db";
case SDB_FUNC:
return "func";
case SDB_IDX:
return "idx";
default:
return "undefine";
}

7
source/dnode/vnode/inc/vnode.h
View File

@ -59,7 +59,7 @@ void vnodePostClose(SVnode *pVnode);
void vnodeSyncCheckTimeout(SVnode *pVnode);
void vnodeClose(SVnode *pVnode);
int32_t vnodeSyncCommit(SVnode *pVnode);
int32_t vnodeBegin(SVnode* pVnode);
int32_t vnodeBegin(SVnode *pVnode);
int32_t vnodeStart(SVnode *pVnode);
void vnodeStop(SVnode *pVnode);
@ -137,6 +137,7 @@ typedef struct SMetaFltParam {
int16_t type;
void *val;
bool reverse;
bool equal;
int (*filterFunc)(void *a, void *b, int16_t type);
} SMetaFltParam;
@ -270,8 +271,8 @@ int32_t tqReaderSetSubmitReq2(STqReader *pReader, void *msgStr, int32_t msgLen,
// int32_t tqReaderSetDataMsg(STqReader *pReader, const SSubmitReq *pMsg, int64_t ver);
bool tqNextDataBlock2(STqReader *pReader);
bool tqNextDataBlockFilterOut2(STqReader *pReader, SHashObj *filterOutUids);
int32_t tqRetrieveDataBlock2(SSDataBlock *pBlock, STqReader *pReader, SSubmitTbData** pSubmitTbDataRet);
int32_t tqRetrieveTaosxBlock2(STqReader *pReader, SArray *blocks, SArray *schemas, SSubmitTbData** pSubmitTbDataRet);
int32_t tqRetrieveDataBlock2(SSDataBlock *pBlock, STqReader *pReader, SSubmitTbData **pSubmitTbDataRet);
int32_t tqRetrieveTaosxBlock2(STqReader *pReader, SArray *blocks, SArray *schemas, SSubmitTbData **pSubmitTbDataRet);
// int32_t tqRetrieveDataBlock(SSDataBlock *pBlock, STqReader *pReader);
// int32_t tqRetrieveTaosxBlock(STqReader *pReader, SArray *blocks, SArray *schemas);

3
source/dnode/vnode/src/inc/vnodeInt.h
View File

@ -142,6 +142,9 @@ int32_t metaGetTbTSchemaEx(SMeta* pMeta, tb_uid_t suid, tb_uid_t uid, in
int metaGetTableEntryByName(SMetaReader* pReader, const char* name);
int metaAlterCache(SMeta* pMeta, int32_t nPage);
int metaAddIndexToSTable(SMeta* pMeta, int64_t version, SVCreateStbReq* pReq);
int metaDropIndexFromSTable(SMeta* pMeta, int64_t version, SDropIndexReq* pReq);
int64_t metaGetTimeSeriesNum(SMeta* pMeta);
SMCtbCursor* metaOpenCtbCursor(SMeta* pMeta, tb_uid_t uid, int lock);
void metaCloseCtbCursor(SMCtbCursor* pCtbCur, int lock);

89
source/dnode/vnode/src/meta/metaQuery.c
View File

@ -1108,26 +1108,30 @@ int32_t metaFilterCreateTime(SMeta *pMeta, SMetaFltParam *param, SArray *pUids)
}
int32_t valid = 0;
while (1) {
int32_t count = 0;
static const int8_t TRY_ERROR_LIMIT = 1;
do {
void *entryKey = NULL;
int32_t nEntryKey = -1;
valid = tdbTbcGet(pCursor->pCur, (const void **)&entryKey, &nEntryKey, NULL, NULL);
if (valid < 0) break;
SCtimeIdxKey *p = entryKey;
if (count > TRY_ERROR_LIMIT) break;
int32_t cmp = (*param->filterFunc)((void *)&p->ctime, (void *)&pCtimeKey->ctime, param->type);
if (cmp == 0) taosArrayPush(pUids, &p->uid);
if (param->reverse == false) {
if (cmp == -1) break;
} else if (param->reverse) {
if (cmp == 1) break;
if (cmp == 0)
taosArrayPush(pUids, &p->uid);
else {
if (param->equal == true) {
if (count > TRY_ERROR_LIMIT) break;
count++;
}
}
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) break;
}
} while (1);
END:
if (pCursor->pMeta) metaULock(pCursor->pMeta);
@ -1162,29 +1166,34 @@ int32_t metaFilterTableName(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
if (tdbTbcMoveTo(pCursor->pCur, pName, strlen(pName) + 1, &cmp) < 0) {
goto END;
}
bool first = true;
int32_t valid = 0;
while (1) {
int32_t count = 0;
int32_t TRY_ERROR_LIMIT = 1;
do {
void *pEntryKey = NULL, *pEntryVal = NULL;
int32_t nEntryKey = -1, nEntryVal = 0;
valid = tdbTbcGet(pCursor->pCur, (const void **)pEntryKey, &nEntryKey, (const void **)&pEntryVal, &nEntryVal);
if (valid < 0) break;
if (count > TRY_ERROR_LIMIT) break;
char *pTableKey = (char *)pEntryKey;
cmp = (*param->filterFunc)(pTableKey, pName, pCursor->type);
if (cmp == 0) {
tb_uid_t tuid = *(tb_uid_t *)pEntryVal;
taosArrayPush(pUids, &tuid);
} else if (cmp == 1) {
// next
} else {
break;
if (param->equal == true) {
if (count > TRY_ERROR_LIMIT) break;
count++;
}
}
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) {
break;
}
}
} while (1);
END:
if (pCursor->pMeta) metaULock(pCursor->pMeta);
@ -1240,7 +1249,7 @@ int32_t metaFilterTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
pCursor->type = param->type;
metaRLock(pMeta);
ret = tdbTbcOpen(pMeta->pCtimeIdx, &pCursor->pCur, NULL);
ret = tdbTbcOpen(pMeta->pTagIdx, &pCursor->pCur, NULL);
if (ret < 0) {
goto END;
}
@ -1284,32 +1293,40 @@ int32_t metaFilterTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
goto END;
}
bool first = true;
int count = 0;
int32_t valid = 0;
while (1) {
bool found = false;
static const int8_t TRY_ERROR_LIMIT = 1;
/// src: [[suid, cid1, type1]....[suid, cid2, type2]....[suid, cid3, type3]...]
/// target: [suid, cid2, type2]
do {
void *entryKey = NULL, *entryVal = NULL;
int32_t nEntryKey, nEntryVal;
valid = tdbTbcGet(pCursor->pCur, (const void **)&entryKey, &nEntryKey, (const void **)&entryVal, &nEntryVal);
if (valid < 0) {
tdbFree(entryVal);
break;
}
if (count > TRY_ERROR_LIMIT) {
break;
}
STagIdxKey *p = entryKey;
if (p == NULL) break;
if (p->type != pCursor->type) {
if (first) {
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) break;
continue;
} else {
if (p->type != pCursor->type || p->suid != pCursor->suid || p->cid != pCursor->cid) {
if (found == true) break;
count++;
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) {
break;
} else {
continue;
}
}
if (p->suid != pKey->suid) {
break;
}
first = false;
int32_t cmp = (*param->filterFunc)(p->data, pKey->data, pKey->type);
if (cmp == 0) {
// match
@ -1320,17 +1337,18 @@ int32_t metaFilterTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
tuid = *(tb_uid_t *)(p->data + tDataTypes[pCursor->type].bytes);
}
taosArrayPush(pUids, &tuid);
} else if (cmp == 1) {
// not match but should continue to iter
found = true;
} else {
// not match and no more result
break;
if (param->equal == true) {
if (count > TRY_ERROR_LIMIT) break;
count++;
}
}
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) {
break;
}
}
} while (1);
END:
if (pCursor->pMeta) metaULock(pCursor->pMeta);
@ -1358,7 +1376,7 @@ static int32_t metaGetTableTagByUid(SMeta *pMeta, int64_t suid, int64_t uid, voi
return ret;
}
int32_t metaGetTableTagsByUids(SMeta *pMeta, int64_t suid, SArray *uidList, SHashObj *tags) {
const int32_t LIMIT = 128;
const int32_t LIMIT = 4096;
int32_t isLock = false;
int32_t sz = uidList ? taosArrayGetSize(uidList) : 0;
@ -1401,6 +1419,7 @@ int32_t metaGetTableTags(SMeta *pMeta, uint64_t suid, SArray *uidList, SHashObj
taosHashPut(uHash, uid, sizeof(int64_t), &i, sizeof(i));
}
}
while (1) {
tb_uid_t id = metaCtbCursorNext(pCur);
if (id == 0) {

643
source/dnode/vnode/src/meta/metaTable.c
View File

@ -394,6 +394,301 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
tdbTbcClose(pUidIdxc);
return 0;
}
int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
SMetaEntry oStbEntry = {0};
SMetaEntry nStbEntry = {0};
STbDbKey tbDbKey = {0};
TBC *pUidIdxc = NULL;
TBC *pTbDbc = NULL;
void *pData = NULL;
int nData = 0;
int64_t oversion;
SDecoder dc = {0};
int32_t ret;
int32_t c = -2;
tb_uid_t suid = pReq->suid;
// get super table
if (tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pData, &nData) != 0) {
ret = -1;
goto _err;
}
tbDbKey.uid = suid;
tbDbKey.version = ((SUidIdxVal *)pData)[0].version;
tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData);
tDecoderInit(&dc, pData, nData);
ret = metaDecodeEntry(&dc, &oStbEntry);
if (ret < 0) {
goto _err;
}
if (oStbEntry.stbEntry.schemaTag.pSchema == NULL || oStbEntry.stbEntry.schemaTag.pSchema == NULL) {
goto _err;
}
if (oStbEntry.stbEntry.schemaTag.version == pReq->schemaTag.version) {
goto _err;
}
if (oStbEntry.stbEntry.schemaTag.nCols != pReq->schemaTag.nCols) {
goto _err;
}
int diffIdx = -1;
for (int i = 0; i < pReq->schemaTag.nCols; i++) {
SSchema *pNew = pReq->schemaTag.pSchema + i;
SSchema *pOld = oStbEntry.stbEntry.schemaTag.pSchema + i;
if (pNew->type != pOld->type || pNew->colId != pOld->colId || pNew->bytes != pOld->bytes ||
strncmp(pOld->name, pNew->name, sizeof(pNew->name))) {
goto _err;
}
if (IS_IDX_ON(pNew) && !IS_IDX_ON(pOld)) {
if (diffIdx != -1) goto _err;
diffIdx = i;
}
}
if (diffIdx == -1 && diffIdx == 0) {
goto _err;
}
// Get target schema info
SSchemaWrapper *pTagSchema = &pReq->schemaTag;
if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) {
terrno = TSDB_CODE_VND_COL_ALREADY_EXISTS;
goto _err;
}
SSchema *pCol = pTagSchema->pSchema + diffIdx;
/*
* iterator all pTdDbc by uid and version
*/
TBC *pCtbIdxc = NULL;
tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL);
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) {
tdbTbcClose(pCtbIdxc);
goto _err;
}
for (;;) {
void *pKey = NULL, *pVal = NULL;
int nKey = 0, nVal = 0;
rc = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal);
if (rc < 0) {
tdbFree(pKey);
tdbFree(pVal);
tdbTbcClose(pCtbIdxc);
pCtbIdxc = NULL;
break;
}
if (((SCtbIdxKey *)pKey)->suid != suid) {
tdbFree(pKey);
tdbFree(pVal);
continue;
}
STagIdxKey *pTagIdxKey = NULL;
int32_t nTagIdxKey;
const void *pTagData = NULL;
int32_t nTagData = 0;
SCtbIdxKey *table = (SCtbIdxKey *)pKey;
STagVal tagVal = {.cid = pCol->colId};
tTagGet((const STag *)pVal, &tagVal);
if (IS_VAR_DATA_TYPE(pCol->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pCol->type].bytes;
}
rc = metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, table->uid, &pTagIdxKey, &nTagIdxKey);
tdbFree(pKey);
tdbFree(pVal);
if (rc < 0) {
metaDestroyTagIdxKey(pTagIdxKey);
tdbTbcClose(pCtbIdxc);
goto _err;
}
metaWLock(pMeta);
tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn);
metaULock(pMeta);
metaDestroyTagIdxKey(pTagIdxKey);
}
nStbEntry.version = version;
nStbEntry.type = TSDB_SUPER_TABLE;
nStbEntry.uid = pReq->suid;
nStbEntry.name = pReq->name;
nStbEntry.stbEntry.schemaRow = pReq->schemaRow;
nStbEntry.stbEntry.schemaTag = pReq->schemaTag;
metaWLock(pMeta);
// update table.db
metaSaveToTbDb(pMeta, &nStbEntry);
// update uid index
metaUpdateUidIdx(pMeta, &nStbEntry);
metaULock(pMeta);
if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf);
tDecoderClear(&dc);
tdbFree(pData);
tdbTbcClose(pCtbIdxc);
return TSDB_CODE_SUCCESS;
_err:
if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf);
tDecoderClear(&dc);
tdbFree(pData);
return TSDB_CODE_VND_COL_ALREADY_EXISTS;
}
int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) {
SMetaEntry oStbEntry = {0};
SMetaEntry nStbEntry = {0};
STbDbKey tbDbKey = {0};
TBC *pUidIdxc = NULL;
TBC *pTbDbc = NULL;
int ret = 0;
int c = -2;
void *pData = NULL;
int nData = 0;
int64_t oversion;
SDecoder dc = {0};
tb_uid_t suid = pReq->stbUid;
if (tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pData, &nData) != 0) {
ret = -1;
goto _err;
}
tbDbKey.uid = suid;
tbDbKey.version = ((SUidIdxVal *)pData)[0].version;
tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData);
tDecoderInit(&dc, pData, nData);
ret = metaDecodeEntry(&dc, &oStbEntry);
if (ret < 0) {
goto _err;
}
SSchema *pCol = NULL;
int32_t colId = -1;
for (int i = 0; i < oStbEntry.stbEntry.schemaTag.nCols; i++) {
SSchema *schema = oStbEntry.stbEntry.schemaTag.pSchema + i;
if (0 == strncmp(schema->name, pReq->colName, sizeof(pReq->colName))) {
if (i != 0 || IS_IDX_ON(schema)) {
pCol = schema;
}
break;
}
}
if (pCol == NULL) {
goto _err;
}
/*
* iterator all pTdDbc by uid and version
*/
TBC *pCtbIdxc = NULL;
tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL);
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) {
tdbTbcClose(pCtbIdxc);
goto _err;
}
for (;;) {
void *pKey = NULL, *pVal = NULL;
int nKey = 0, nVal = 0;
rc = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal);
if (rc < 0) {
tdbFree(pKey);
tdbFree(pVal);
tdbTbcClose(pCtbIdxc);
pCtbIdxc = NULL;
break;
}
if (((SCtbIdxKey *)pKey)->suid != suid) {
tdbFree(pKey);
tdbFree(pVal);
continue;
}
STagIdxKey *pTagIdxKey = NULL;
int32_t nTagIdxKey;
const void *pTagData = NULL;
int32_t nTagData = 0;
SCtbIdxKey *table = (SCtbIdxKey *)pKey;
STagVal tagVal = {.cid = pCol->colId};
tTagGet((const STag *)pVal, &tagVal);
if (IS_VAR_DATA_TYPE(pCol->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pCol->type].bytes;
}
rc = metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, table->uid, &pTagIdxKey, &nTagIdxKey);
tdbFree(pKey);
tdbFree(pVal);
if (rc < 0) {
metaDestroyTagIdxKey(pTagIdxKey);
tdbTbcClose(pCtbIdxc);
goto _err;
}
metaWLock(pMeta);
tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn);
metaULock(pMeta);
metaDestroyTagIdxKey(pTagIdxKey);
}
// clear idx flag
SSCHMEA_SET_IDX_OFF(pCol);
nStbEntry.version = version;
nStbEntry.type = TSDB_SUPER_TABLE;
nStbEntry.uid = oStbEntry.uid;
nStbEntry.name = oStbEntry.name;
SSchemaWrapper *row = tCloneSSchemaWrapper(&oStbEntry.stbEntry.schemaRow);
SSchemaWrapper *tag = tCloneSSchemaWrapper(&oStbEntry.stbEntry.schemaTag);
nStbEntry.stbEntry.schemaRow = *row;
nStbEntry.stbEntry.schemaTag = *tag;
nStbEntry.stbEntry.rsmaParam = oStbEntry.stbEntry.rsmaParam;
metaWLock(pMeta);
// update table.db
metaSaveToTbDb(pMeta, &nStbEntry);
// update uid index
metaUpdateUidIdx(pMeta, &nStbEntry);
metaULock(pMeta);
tDeleteSSchemaWrapper(tag);
tDeleteSSchemaWrapper(row);
if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf);
tDecoderClear(&dc);
tdbFree(pData);
tdbTbcClose(pCtbIdxc);
return TSDB_CODE_SUCCESS;
_err:
if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf);
tDecoderClear(&dc);
tdbFree(pData);
return -1;
}
int metaCreateTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq, STableMetaRsp **pMetaRsp) {
SMetaEntry me = {0};
@ -649,31 +944,38 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type) {
tDecoderInit(&tdc, tData, tLen);
metaDecodeEntry(&tdc, &stbEntry);
const SSchema *pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[0];
if (pTagColumn->type == TSDB_DATA_TYPE_JSON) {
SSchema *pTagColumn = NULL;
SSchemaWrapper *pTagSchema = &stbEntry.stbEntry.schemaTag;
if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) {
pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[0];
metaDelJsonVarFromIdx(pMeta, &e, pTagColumn);
} else {
STagIdxKey *pTagIdxKey = NULL;
int32_t nTagIdxKey;
for (int i = 0; i < pTagSchema->nCols; i++) {
pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[i];
if (!IS_IDX_ON(pTagColumn)) continue;
STagIdxKey *pTagIdxKey = NULL;
int32_t nTagIdxKey;
const void *pTagData = NULL;
int32_t nTagData = 0;
const void *pTagData = NULL;
int32_t nTagData = 0;
STagVal tagVal = {.cid = pTagColumn->colId};
tTagGet((const STag *)e.ctbEntry.pTags, &tagVal);
if (IS_VAR_DATA_TYPE(pTagColumn->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pTagColumn->type].bytes;
STagVal tagVal = {.cid = pTagColumn->colId};
tTagGet((const STag *)e.ctbEntry.pTags, &tagVal);
if (IS_VAR_DATA_TYPE(pTagColumn->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pTagColumn->type].bytes;
}
if (metaCreateTagIdxKey(e.ctbEntry.suid, pTagColumn->colId, pTagData, nTagData, pTagColumn->type, uid,
&pTagIdxKey, &nTagIdxKey) == 0) {
tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn);
}
metaDestroyTagIdxKey(pTagIdxKey);
}
if (metaCreateTagIdxKey(e.ctbEntry.suid, pTagColumn->colId, pTagData, nTagData, pTagColumn->type, uid,
&pTagIdxKey, &nTagIdxKey) == 0) {
tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn);
}
metaDestroyTagIdxKey(pTagIdxKey);
}
tDecoderClear(&tdc);
}
@ -931,12 +1233,11 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl
entry.version = version;
metaDeleteNcolIdx(pMeta, &oldEntry);
metaUpdateNcolIdx(pMeta, &entry);
// do actual write
metaWLock(pMeta);
metaDeleteNcolIdx(pMeta, &oldEntry);
metaUpdateNcolIdx(pMeta, &entry);
// save to table db
metaSaveToTbDb(pMeta, &entry);
@ -1117,9 +1418,7 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA
// save to uid.idx
metaUpdateUidIdx(pMeta, &ctbEntry);
if (iCol == 0) {
metaUpdateTagIdx(pMeta, &ctbEntry);
}
metaUpdateTagIdx(pMeta, &ctbEntry);
if (NULL == ctbEntry.ctbEntry.pTags) {
metaError("meta/table: null tags, update tag val failed.");
@ -1254,6 +1553,243 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p
return 0;
}
static int metaAddTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq) {
SMetaEntry stbEntry = {0};
void *pVal = NULL;
int nVal = 0;
int ret;
int c;
tb_uid_t uid, suid;
int64_t oversion;
const void *pData = NULL;
int nData = 0;
SDecoder dc = {0};
if (pAlterTbReq->tagName == NULL) {
terrno = TSDB_CODE_INVALID_MSG;
return -1;
}
// search name index
ret = tdbTbGet(pMeta->pNameIdx, pAlterTbReq->tbName, strlen(pAlterTbReq->tbName) + 1, &pVal, &nVal);
if (ret < 0) {
terrno = TSDB_CODE_TDB_TABLE_NOT_EXIST;
return -1;
}
uid = *(tb_uid_t *)pVal;
tdbFree(pVal);
pVal = NULL;
if (tdbTbGet(pMeta->pUidIdx, &uid, sizeof(tb_uid_t), &pVal, &nVal) == -1) {
ret = -1;
goto _err;
}
suid = ((SUidIdxVal *)pVal)[0].suid;
STbDbKey tbDbKey = {0};
tbDbKey.uid = suid;
tbDbKey.version = ((SUidIdxVal *)pVal)[0].version;
tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pVal, &nVal);
tDecoderInit(&dc, pVal, nVal);
ret = metaDecodeEntry(&dc, &stbEntry);
if (ret < 0) {
goto _err;
}
// Get target schema info
SSchemaWrapper *pTagSchema = &stbEntry.stbEntry.schemaTag;
if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) {
terrno = TSDB_CODE_VND_COL_ALREADY_EXISTS;
goto _err;
}
SSchema *pCol = NULL;
int32_t iCol = 0;
for (;;) {
pCol = NULL;
if (iCol >= pTagSchema->nCols) break;
pCol = &pTagSchema->pSchema[iCol];
if (strcmp(pCol->name, pAlterTbReq->tagName) == 0) break;
iCol++;
}
if (iCol == 0) {
terrno = TSDB_CODE_VND_COL_ALREADY_EXISTS;
goto _err;
}
if (pCol == NULL) {
terrno = TSDB_CODE_VND_COL_NOT_EXISTS;
goto _err;
}
/*
* iterator all pTdDbc by uid and version
*/
TBC *pCtbIdxc = NULL;
tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL);
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) {
tdbTbcClose(pCtbIdxc);
goto _err;
}
for (;;) {
void *pKey, *pVal;
int nKey, nVal;
rc = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal);
if (rc < 0) break;
if (((SCtbIdxKey *)pKey)->suid != uid) {
tdbFree(pKey);
tdbFree(pVal);
continue;
}
STagIdxKey *pTagIdxKey = NULL;
int32_t nTagIdxKey;
const void *pTagData = NULL;
int32_t nTagData = 0;
STagVal tagVal = {.cid = pCol->colId};
tTagGet((const STag *)pVal, &tagVal);
if (IS_VAR_DATA_TYPE(pCol->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pCol->type].bytes;
}
if (metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, uid, &pTagIdxKey, &nTagIdxKey) < 0) {
metaDestroyTagIdxKey(pTagIdxKey);
goto _err;
}
tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn);
metaDestroyTagIdxKey(pTagIdxKey);
}
return 0;
_err:
// tDecoderClear(&dc1);
// tDecoderClear(&dc2);
// if (ctbEntry.pBuf) taosMemoryFree(ctbEntry.pBuf);
// if (stbEntry.pBuf) tdbFree(stbEntry.pBuf);
// tdbTbcClose(pTbDbc);
// tdbTbcClose(pUidIdxc);
return -1;
}
typedef struct SMetaPair {
void *key;
int nkey;
} SMetaPair;
static int metaDropTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq) {
SMetaEntry stbEntry = {0};
void *pVal = NULL;
int nVal = 0;
int ret;
int c;
tb_uid_t suid;
int64_t oversion;
const void *pData = NULL;
int nData = 0;
SDecoder dc = {0};
if (pAlterTbReq->tagName == NULL) {
terrno = TSDB_CODE_INVALID_MSG;
return -1;
}
// search name index
ret = tdbTbGet(pMeta->pNameIdx, pAlterTbReq->tbName, strlen(pAlterTbReq->tbName) + 1, &pVal, &nVal);
if (ret < 0) {
terrno = TSDB_CODE_TDB_TABLE_NOT_EXIST;
return -1;
}
suid = *(tb_uid_t *)pVal;
tdbFree(pVal);
pVal = NULL;
if (tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pVal, &nVal) == -1) {
ret = -1;
goto _err;
}
STbDbKey tbDbKey = {0};
tbDbKey.uid = suid;
tbDbKey.version = ((SUidIdxVal *)pVal)[0].version;
tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pVal, &nVal);
tDecoderInit(&dc, pVal, nVal);
ret = metaDecodeEntry(&dc, &stbEntry);
if (ret < 0) {
goto _err;
}
// Get targe schema info
SSchemaWrapper *pTagSchema = &stbEntry.stbEntry.schemaTag;
if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) {
terrno = TSDB_CODE_VND_COL_ALREADY_EXISTS;
goto _err;
}
SSchema *pCol = NULL;
int32_t iCol = 0;
for (;;) {
pCol = NULL;
if (iCol >= pTagSchema->nCols) break;
pCol = &pTagSchema->pSchema[iCol];
if (strcmp(pCol->name, pAlterTbReq->tagName) == 0) break;
iCol++;
}
if (iCol == 0) {
// cannot drop 1th tag index
terrno = -1;
goto _err;
}
if (pCol == NULL) {
terrno = TSDB_CODE_VND_COL_NOT_EXISTS;
goto _err;
}
if (IS_IDX_ON(pCol)) {
terrno = TSDB_CODE_VND_COL_ALREADY_EXISTS;
goto _err;
}
SArray *tagIdxList = taosArrayInit(512, sizeof(SMetaPair));
TBC *pTagIdxc = NULL;
tdbTbcOpen(pMeta->pTagIdx, &pTagIdxc, NULL);
int rc =
tdbTbcMoveTo(pTagIdxc, &(STagIdxKey){.suid = suid, .cid = INT32_MIN, .type = pCol->type}, sizeof(STagIdxKey), &c);
for (;;) {
void *pKey, *pVal;
int nKey, nVal;
rc = tdbTbcNext(pTagIdxc, &pKey, &nKey, &pVal, &nVal);
STagIdxKey *pIdxKey = (STagIdxKey *)pKey;
if (pIdxKey->suid != suid || pIdxKey->cid != pCol->colId) {
tdbFree(pKey);
tdbFree(pVal);
continue;
}
SMetaPair pair = {.key = pKey, nKey = nKey};
taosArrayPush(tagIdxList, &pair);
}
tdbTbcClose(pTagIdxc);
metaWLock(pMeta);
for (int i = 0; i < taosArrayGetSize(tagIdxList); i++) {
SMetaPair *pair = taosArrayGet(tagIdxList, i);
tdbTbDelete(pMeta->pTagIdx, pair->key, pair->nkey, pMeta->txn);
}
metaULock(pMeta);
taosArrayDestroy(tagIdxList);
// set pCol->flags; INDEX_ON
return 0;
_err:
return -1;
}
int metaAlterTable(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMetaRsp *pMetaRsp) {
switch (pReq->action) {
case TSDB_ALTER_TABLE_ADD_COLUMN:
@ -1265,6 +1801,10 @@ int metaAlterTable(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMeta
return metaUpdateTableTagVal(pMeta, version, pReq);
case TSDB_ALTER_TABLE_UPDATE_OPTIONS:
return metaUpdateTableOptions(pMeta, version, pReq);
case TSDB_ALTER_TABLE_ADD_TAG_INDEX:
return metaAddTagIndex(pMeta, version, pReq);
case TSDB_ALTER_TABLE_DROP_TAG_INDEX:
return metaDropTagIndex(pMeta, version, pReq);
default:
terrno = TSDB_CODE_VND_INVALID_TABLE_ACTION;
return -1;
@ -1429,36 +1969,43 @@ static int metaUpdateTagIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry) {
goto end;
}
pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[0];
SSchemaWrapper *pTagSchema = &stbEntry.stbEntry.schemaTag;
if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) {
pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[0];
STagVal tagVal = {.cid = pTagColumn->colId};
STagVal tagVal = {.cid = pTagColumn->colId};
if (pTagColumn->type != TSDB_DATA_TYPE_JSON) {
tTagGet((const STag *)pCtbEntry->ctbEntry.pTags, &tagVal);
if (IS_VAR_DATA_TYPE(pTagColumn->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pTagColumn->type].bytes;
}
} else {
// pTagData = pCtbEntry->ctbEntry.pTags;
// nTagData = ((const STag *)pCtbEntry->ctbEntry.pTags)->len;
pTagData = pCtbEntry->ctbEntry.pTags;
nTagData = ((const STag *)pCtbEntry->ctbEntry.pTags)->len;
ret = metaSaveJsonVarToIdx(pMeta, pCtbEntry, pTagColumn);
goto end;
}
if (pTagData != NULL) {
if (metaCreateTagIdxKey(pCtbEntry->ctbEntry.suid, pTagColumn->colId, pTagData, nTagData, pTagColumn->type,
pCtbEntry->uid, &pTagIdxKey, &nTagIdxKey) < 0) {
ret = -1;
goto end;
} else {
for (int i = 0; i < pTagSchema->nCols; i++) {
pTagColumn = &pTagSchema->pSchema[i];
if (i != 0 && !IS_IDX_ON(pTagColumn)) continue;
STagVal tagVal = {.cid = pTagColumn->colId};
tTagGet((const STag *)pCtbEntry->ctbEntry.pTags, &tagVal);
if (IS_VAR_DATA_TYPE(pTagColumn->type)) {
pTagData = tagVal.pData;
nTagData = (int32_t)tagVal.nData;
} else {
pTagData = &(tagVal.i64);
nTagData = tDataTypes[pTagColumn->type].bytes;
}
if (pTagData != NULL) {
if (metaCreateTagIdxKey(pCtbEntry->ctbEntry.suid, pTagColumn->colId, pTagData, nTagData, pTagColumn->type,
pCtbEntry->uid, &pTagIdxKey, &nTagIdxKey) < 0) {
ret = -1;
goto end;
}
tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn);
}
metaDestroyTagIdxKey(pTagIdxKey);
}
tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn);
}
end:
metaDestroyTagIdxKey(pTagIdxKey);
// metaDestroyTagIdxKey(pTagIdxKey);
tDecoderClear(&dc);
tdbFree(pData);
return ret;

10
source/dnode/vnode/src/sma/smaTimeRange.c
View File

@ -257,7 +257,6 @@ int32_t smaBlockToSubmit(SVnode *pVnode, const SArray *pBlocks, const STSchema *
int32_t rows = pDataBlock->info.rows;
SSubmitTbData tbData = {0};
if (!(tbData.aRowP = taosArrayInit(rows, sizeof(SRow *)))) {
goto _end;
@ -313,14 +312,15 @@ int32_t smaBlockToSubmit(SVnode *pVnode, const SArray *pBlocks, const STSchema *
tEncodeSize(tEncodeSSubmitReq2, pReq, len, terrno);
if (TSDB_CODE_SUCCESS == terrno) {
SEncoder encoder;
len += sizeof(SMsgHead);
len += sizeof(SSubmitReq2Msg);
pBuf = rpcMallocCont(len);
if (NULL == pBuf) {
goto _end;
}
((SMsgHead *)pBuf)->vgId = TD_VID(pVnode);
((SMsgHead *)pBuf)->contLen = htonl(len);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SMsgHead)), len - sizeof(SMsgHead));
((SSubmitReq2Msg *)pBuf)->header.vgId = TD_VID(pVnode);
((SSubmitReq2Msg *)pBuf)->header.contLen = htonl(len);
((SSubmitReq2Msg *)pBuf)->version = htobe64(1);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SSubmitReq2Msg)), len - sizeof(SSubmitReq2Msg));
if (tEncodeSSubmitReq2(&encoder, pReq) < 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
/*vError("failed to encode submit req since %s", terrstr());*/

8
source/dnode/vnode/src/tq/tq.c
View File

@ -683,13 +683,13 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg) {
if (pHead->msgType == TDMT_VND_SUBMIT) {
SPackedData submit = {
.msgStr = POINTER_SHIFT(pHead->body, sizeof(SMsgHead)),
.msgLen = pHead->bodyLen - sizeof(SMsgHead),
.msgStr = POINTER_SHIFT(pHead->body, sizeof(SSubmitReq2Msg)),
.msgLen = pHead->bodyLen - sizeof(SSubmitReq2Msg),
.ver = pHead->version,
};
if (tqTaosxScanLog(pTq, pHandle, submit, &taosxRsp) < 0) {
tqError("tmq poll: tqTaosxScanLog error %" PRId64 ", in vgId:%d, subkey %s", consumerId,
TD_VID(pTq->pVnode), req.subKey);
tqError("tmq poll: tqTaosxScanLog error %" PRId64 ", in vgId:%d, subkey %s", consumerId, TD_VID(pTq->pVnode),
req.subKey);
return -1;
}
if (taosxRsp.blockNum > 0 /* threshold */) {

4
source/dnode/vnode/src/tq/tqPush.c
View File

@ -207,8 +207,8 @@ int32_t tqPushMsgNew(STQ* pTq, void* msg, int32_t msgLen, tmsg_t msgType, int64_
#endif
int tqPushMsg(STQ* pTq, void* msg, int32_t msgLen, tmsg_t msgType, int64_t ver) {
void* pReq = POINTER_SHIFT(msg, sizeof(SMsgHead));
int32_t len = msgLen - sizeof(SMsgHead);
void* pReq = POINTER_SHIFT(msg, sizeof(SSubmitReq2Msg));
int32_t len = msgLen - sizeof(SSubmitReq2Msg);
tqDebug("vgId:%d, tq push msg ver %" PRId64 ", type: %s, p head %p, p body %p, len %d", pTq->pVnode->config.vgId, ver,
TMSG_INFO(msgType), msg, pReq, len);

38
source/dnode/vnode/src/tq/tqRead.c
View File

@ -311,8 +311,8 @@ int32_t tqNextBlock(STqReader* pReader, SFetchRet* ret) {
tqDebug("return offset %" PRId64 ", no more valid", ret->offset.version);
return -1;
}
void* body = POINTER_SHIFT(pReader->pWalReader->pHead->head.body, sizeof(SMsgHead));
int32_t bodyLen = pReader->pWalReader->pHead->head.bodyLen - sizeof(SMsgHead);
void* body = POINTER_SHIFT(pReader->pWalReader->pHead->head.body, sizeof(SSubmitReq2Msg));
int32_t bodyLen = pReader->pWalReader->pHead->head.bodyLen - sizeof(SSubmitReq2Msg);
int64_t ver = pReader->pWalReader->pHead->head.version;
#if 0
if (pReader->pWalReader->pHead->head.msgType != TDMT_VND_SUBMIT) {
@ -560,7 +560,7 @@ int32_t tqRetrieveDataBlock2(SSDataBlock* pBlock, STqReader* pReader, SSubmitTbD
SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk);
pReader->nextBlk++;
if(pSubmitTbDataRet) *pSubmitTbDataRet = pSubmitTbData;
if (pSubmitTbDataRet) *pSubmitTbDataRet = pSubmitTbData;
int32_t sversion = pSubmitTbData->sver;
int64_t suid = pSubmitTbData->suid;
int64_t uid = pSubmitTbData->uid;
@ -1012,7 +1012,7 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk);
pReader->nextBlk++;
if(pSubmitTbDataRet) *pSubmitTbDataRet = pSubmitTbData;
if (pSubmitTbDataRet) *pSubmitTbDataRet = pSubmitTbData;
int32_t sversion = pSubmitTbData->sver;
int64_t suid = pSubmitTbData->suid;
int64_t uid = pSubmitTbData->uid;
@ -1022,7 +1022,7 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
pReader->pSchema = metaGetTbTSchema(pReader->pVnodeMeta, uid, sversion, 1);
if (pReader->pSchema == NULL) {
tqWarn("vgId:%d, cannot found tsschema for table: uid:%" PRId64 " (suid:%" PRId64
"), version %d, possibly dropped table",
"), version %d, possibly dropped table",
pReader->pWalReader->pWal->cfg.vgId, uid, suid, sversion);
pReader->cachedSchemaSuid = 0;
terrno = TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND;
@ -1041,7 +1041,7 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
STSchema* pTschema = pReader->pSchema;
SSchemaWrapper* pSchemaWrapper = pReader->pSchemaWrapper;
int32_t numOfRows = 0;
int32_t numOfRows = 0;
if (pSubmitTbData->flags & SUBMIT_REQ_COLUMN_DATA_FORMAT) {
SArray* pCols = pSubmitTbData->aCol;
@ -1054,7 +1054,7 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
int32_t curRow = 0;
int32_t lastRow = 0;
char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols);
char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols);
if (assigned == NULL) return -1;
// convert and scan one block
@ -1064,9 +1064,9 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
for (int32_t i = 0; i < numOfRows; i++) {
bool buildNew = false;
for (int32_t j = 0; j < numOfCols; j++){
for (int32_t j = 0; j < numOfCols; j++) {
SColData* pCol = taosArrayGet(pCols, j);
SColVal colVal;
SColVal colVal;
tColDataGetValue(pCol, i, &colVal);
if (curRow == 0) {
assigned[j] = !COL_VAL_IS_NONE(&colVal);
@ -1087,9 +1087,9 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
lastRow = curRow;
}
SSDataBlock block = {0};
SSDataBlock block = {0};
SSchemaWrapper* pSW = taosMemoryCalloc(1, sizeof(SSchemaWrapper));
if(pSW == NULL){
if (pSW == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto FAIL;
}
@ -1158,10 +1158,10 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
} else {
SArray* pRows = pSubmitTbData->aRowP;
for (int32_t i = 0; i < numOfRows; i++) {
SRow* pRow = taosArrayGetP(pRows, i);
bool buildNew = false;
SRow* pRow = taosArrayGetP(pRows, i);
bool buildNew = false;
for (int32_t j = 0; j < pTschema->numOfCols; j++){
for (int32_t j = 0; j < pTschema->numOfCols; j++) {
SColVal colVal;
tRowGet(pRow, pTschema, j, &colVal);
if (curRow == 0) {
@ -1183,9 +1183,9 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
lastRow = curRow;
}
SSDataBlock block = {0};
SSDataBlock block = {0};
SSchemaWrapper* pSW = taosMemoryCalloc(1, sizeof(SSchemaWrapper));
if(pSW == NULL){
if (pSW == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto FAIL;
}
@ -1220,7 +1220,7 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
int32_t colActual = blockDataGetNumOfCols(pBlock);
while (targetIdx < colActual) {
SColumnInfoData* pColData = taosArrayGet(pBlock->pDataBlock, targetIdx);
SColVal colVal;
SColVal colVal;
tRowGet(pRow, pTschema, sourceIdx, &colVal);
if (colVal.cid < pColData->info.colId) {
@ -1256,7 +1256,7 @@ int32_t tqRetrieveTaosxBlock2(STqReader* pReader, SArray* blocks, SArray* schema
taosMemoryFree(assigned);
return 0;
FAIL:
FAIL:
taosMemoryFree(assigned);
return -1;
}
@ -1358,7 +1358,7 @@ int32_t tqUpdateTbUidList(STQ* pTq, const SArray* tbUidList, bool isAdd) {
}
taosArrayDestroy(qa);
} else {
// TODO handle delete table from stb
tqReaderRemoveTbUidList(pExec->execHandle.pExecReader, tbUidList);
}
}
}

32
source/dnode/vnode/src/tq/tqSink.c
View File

@ -71,7 +71,6 @@ int32_t tqBuildDeleteReq(SVnode* pVnode, const char* stbFullName, const SSDataBl
return 0;
}
void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* data) {
const SArray* pBlocks = (const SArray*)data;
SVnode* pVnode = (SVnode*)vnode;
@ -324,7 +323,7 @@ void tqSinkToTablePipeline(SStreamTask* pTask, void* vnode, int64_t ver, void* d
}
static int32_t encodeCreateChildTableForRPC(SVCreateTbBatchReq* pReqs, int32_t vgId, void** pBuf, int32_t* contLen) {
int32_t ret = 0;
int32_t ret = 0;
tEncodeSize(tEncodeSVCreateTbBatchReq, pReqs, *contLen, ret);
if (ret < 0) {
@ -340,7 +339,7 @@ static int32_t encodeCreateChildTableForRPC(SVCreateTbBatchReq* pReqs, int32_t v
((SMsgHead*)(*pBuf))->vgId = vgId;
((SMsgHead*)(*pBuf))->contLen = htonl(*contLen);
SEncoder coder = {0};
tEncoderInit(&coder, POINTER_SHIFT(*pBuf, sizeof(SMsgHead)), (*contLen) - sizeof(SMsgHead) );
tEncoderInit(&coder, POINTER_SHIFT(*pBuf, sizeof(SMsgHead)), (*contLen) - sizeof(SMsgHead));
if (tEncodeSVCreateTbBatchReq(&coder, pReqs) < 0) {
rpcFreeCont(*pBuf);
*pBuf = NULL;
@ -440,7 +439,7 @@ void tqSinkToTablePipeline2(SStreamTask* pTask, void* vnode, int64_t ver, void*
goto _end;
}
for (int32_t rowId = 0; rowId < rows; rowId++) {
SVCreateTbReq createTbReq = {0};
SVCreateTbReq createTbReq = {0};
SVCreateTbReq* pCreateTbReq = &createTbReq;
if (!pCreateTbReq) {
goto _end;
@ -482,9 +481,9 @@ void tqSinkToTablePipeline2(SStreamTask* pTask, void* vnode, int64_t ver, void*
}
for (int32_t tagId = UD_TAG_COLUMN_INDEX, step = 1; tagId < size; tagId++, step++) {
SColumnInfoData* pTagData = taosArrayGet(pDataBlock->pDataBlock, tagId);
STagVal tagVal = {
.cid = pTSchema->numOfCols + step,
.type = pTagData->info.type,
STagVal tagVal = {
.cid = pTSchema->numOfCols + step,
.type = pTagData->info.type,
};
void* pData = colDataGetData(pTagData, rowId);
if (colDataIsNull_s(pTagData, rowId)) {
@ -498,7 +497,7 @@ void tqSinkToTablePipeline2(SStreamTask* pTask, void* vnode, int64_t ver, void*
taosArrayPush(tagArray, &tagVal);
}
}
pCreateTbReq->ctb.tagNum = taosArrayGetSize(tagArray);
pCreateTbReq->ctb.tagNum = size;
STag* pTag = NULL;
tTagNew(tagArray, 1, false, &pTag);
@ -514,7 +513,7 @@ void tqSinkToTablePipeline2(SStreamTask* pTask, void* vnode, int64_t ver, void*
SColumnInfoData* pTbColInfo = taosArrayGet(pDataBlock->pDataBlock, UD_TABLE_NAME_COLUMN_INDEX);
if (colDataIsNull_s(pTbColInfo, rowId)) {
SColumnInfoData* pGpIdColInfo = taosArrayGet(pDataBlock->pDataBlock, UD_GROUPID_COLUMN_INDEX);
void* pGpIdData = colDataGetData(pGpIdColInfo, rowId);
void* pGpIdData = colDataGetData(pGpIdColInfo, rowId);
pCreateTbReq->name = buildCtbNameByGroupId(stbFullName, *(uint64_t*)pGpIdData);
} else {
void* pTbData = colDataGetData(pTbColInfo, rowId);
@ -639,16 +638,16 @@ void tqSinkToTablePipeline2(SStreamTask* pTask, void* vnode, int64_t ver, void*
taosArrayClear(pVals);
int32_t dataIndex = 0;
for (int32_t k = 0; k < pTSchema->numOfCols; k++) {
const STColumn* pCol = &pTSchema->columns[k];
const STColumn* pCol = &pTSchema->columns[k];
if (k == 0) {
SColumnInfoData* pColData = taosArrayGet(pDataBlock->pDataBlock, dataIndex);
void* colData = colDataGetData(pColData, j);
void* colData = colDataGetData(pColData, j);
tqDebug("tq sink pipe2, row %d, col %d ts %" PRId64, j, k, *(int64_t*)colData);
}
if (IS_SET_NULL(pCol)) {
SColVal cv = COL_VAL_NULL(pCol->colId, pCol->type);
taosArrayPush(pVals, &cv);
} else{
} else {
SColumnInfoData* pColData = taosArrayGet(pDataBlock->pDataBlock, dataIndex);
if (colDataIsNull_s(pColData, j)) {
SColVal cv = COL_VAL_NULL(pCol->colId, pCol->type);
@ -692,14 +691,15 @@ void tqSinkToTablePipeline2(SStreamTask* pTask, void* vnode, int64_t ver, void*
int32_t code;
tEncodeSize(tEncodeSSubmitReq2, &submitReq, len, code);
SEncoder encoder;
len += sizeof(SMsgHead);
len += sizeof(SSubmitReq2Msg);
pBuf = rpcMallocCont(len);
if (NULL == pBuf) {
goto _end;
}
((SMsgHead*)pBuf)->vgId = TD_VID(pVnode);
((SMsgHead*)pBuf)->contLen = htonl(len);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SMsgHead)), len - sizeof(SMsgHead));
((SSubmitReq2Msg*)pBuf)->header.vgId = TD_VID(pVnode);
((SSubmitReq2Msg*)pBuf)->header.contLen = htonl(len);
((SSubmitReq2Msg*)pBuf)->version = htobe64(1);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SSubmitReq2Msg)), len - sizeof(SSubmitReq2Msg));
if (tEncodeSSubmitReq2(&encoder, &submitReq) < 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
tqError("failed to encode submit req since %s", terrstr());

92
source/dnode/vnode/src/tsdb/tsdbWrite.c
View File

@ -22,7 +22,7 @@
* us: 3600*1000000*8765*1000 // 1970 + 1000 years
* ns: 3600*1000000000*8765*292 // 1970 + 292 years
*/
static int64_t tsMaxKeyByPrecision[] = {31556995200000L, 31556995200000000L, 9214646400000000000L};
int64_t tsMaxKeyByPrecision[] = {31556995200000L, 31556995200000000L, 9214646400000000000L};
// static int tsdbScanAndConvertSubmitMsg(STsdb *pTsdb, SSubmitReq *pMsg);
@ -60,23 +60,6 @@ int tsdbInsertData(STsdb *pTsdb, int64_t version, SSubmitReq2 *pMsg, SSubmitRsp2
return 0;
}
#if 0
static FORCE_INLINE int tsdbCheckRowRange(STsdb *pTsdb, STable *pTable, STSRow *row, TSKEY minKey, TSKEY maxKey,
TSKEY now) {
TSKEY rowKey = TD_ROW_KEY(row);
if (rowKey < minKey || rowKey > maxKey) {
tsdbError("vgId:%d, table %s tid %d uid %" PRIu64 " timestamp is out of range! now %" PRId64 " minKey %" PRId64
" maxKey %" PRId64 " row key %" PRId64,
REPO_ID(pTsdb), TABLE_CHAR_NAME(pTable), TABLE_TID(pTable), TABLE_UID(pTable), now, minKey, maxKey,
rowKey);
terrno = TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE;
return -1;
}
return 0;
}
#endif
static FORCE_INLINE int tsdbCheckRowRange(STsdb *pTsdb, tb_uid_t uid, TSKEY rowKey, TSKEY minKey, TSKEY maxKey,
TSKEY now) {
if (rowKey < minKey || rowKey > maxKey) {
@ -89,79 +72,6 @@ static FORCE_INLINE int tsdbCheckRowRange(STsdb *pTsdb, tb_uid_t uid, TSKEY rowK
return 0;
}
#if 0
int tsdbScanAndConvertSubmitMsg(STsdb *pTsdb, SSubmitReq *pMsg) {
ASSERT(pMsg != NULL);
// STsdbMeta * pMeta = pTsdb->tsdbMeta;
SSubmitMsgIter msgIter = {0};
SSubmitBlk *pBlock = NULL;
SSubmitBlkIter blkIter = {0};
STSRow *row = NULL;
STsdbKeepCfg *pCfg = &pTsdb->keepCfg;
TSKEY now = taosGetTimestamp(pCfg->precision);
TSKEY minKey = now - tsTickPerMin[pCfg->precision] * pCfg->keep2;
TSKEY maxKey = tsMaxKeyByPrecision[pCfg->precision];
terrno = TSDB_CODE_SUCCESS;
// pMsg->length = htonl(pMsg->length);
// pMsg->numOfBlocks = htonl(pMsg->numOfBlocks);
if (tInitSubmitMsgIter(pMsg, &msgIter) < 0) return -1;
while (true) {
if (tGetSubmitMsgNext(&msgIter, &pBlock) < 0) return -1;
if (pBlock == NULL) break;
// pBlock->uid = htobe64(pBlock->uid);
// pBlock->suid = htobe64(pBlock->suid);
// pBlock->sversion = htonl(pBlock->sversion);
// pBlock->dataLen = htonl(pBlock->dataLen);
// pBlock->schemaLen = htonl(pBlock->schemaLen);
// pBlock->numOfRows = htonl(pBlock->numOfRows);
#if 0
if (pBlock->tid <= 0 || pBlock->tid >= pMeta->maxTables) {
tsdbError("vgId:%d, failed to get table to insert data, uid %" PRIu64 " tid %d", REPO_ID(pTsdb), pBlock->uid,
pBlock->tid);
terrno = TSDB_CODE_TDB_INVALID_TABLE_ID;
return -1;
}
STable *pTable = pMeta->tables[pBlock->tid];
if (pTable == NULL || TABLE_UID(pTable) != pBlock->uid) {
tsdbError("vgId:%d, failed to get table to insert data, uid %" PRIu64 " tid %d", REPO_ID(pTsdb), pBlock->uid,
pBlock->tid);
terrno = TSDB_CODE_TDB_INVALID_TABLE_ID;
return -1;
}
if (TABLE_TYPE(pTable) == TSDB_SUPER_TABLE) {
tsdbError("vgId:%d, invalid action trying to insert a super table %s", REPO_ID(pTsdb), TABLE_CHAR_NAME(pTable));
terrno = TSDB_CODE_TDB_INVALID_ACTION;
return -1;
}
// Check schema version and update schema if needed
if (tsdbCheckTableSchema(pTsdb, pBlock, pTable) < 0) {
if (terrno == TSDB_CODE_TDB_TABLE_RECONFIGURE) {
continue;
} else {
return -1;
}
}
#endif
tInitSubmitBlkIter(&msgIter, pBlock, &blkIter);
while ((row = tGetSubmitBlkNext(&blkIter)) != NULL) {
if (tsdbCheckRowRange(pTsdb, msgIter.uid, row, minKey, maxKey, now) < 0) {
return -1;
}
}
}
if (terrno != TSDB_CODE_SUCCESS) return -1;
return 0;
}
#endif
int tsdbScanAndConvertSubmitMsg(STsdb *pTsdb, SSubmitReq2 *pMsg) {
int32_t code = 0;
STsdbKeepCfg *pCfg = &pTsdb->keepCfg;

27
source/dnode/vnode/src/vnd/vnodeQuery.c
View File

@ -96,6 +96,7 @@ int vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
metaRsp.numOfColumns = schema.nCols;
metaRsp.precision = pVnode->config.tsdbCfg.precision;
metaRsp.sversion = schema.version;
metaRsp.tversion = schemaTag.version;
metaRsp.pSchemas = (SSchema *)taosMemoryMalloc(sizeof(SSchema) * (metaRsp.numOfColumns + metaRsp.numOfTags));
memcpy(metaRsp.pSchemas, schema.pSchema, sizeof(SSchema) * schema.nCols);
@ -264,26 +265,25 @@ _exit:
return TSDB_CODE_SUCCESS;
}
static FORCE_INLINE void vnodeFreeSBatchRspMsg(void* p) {
static FORCE_INLINE void vnodeFreeSBatchRspMsg(void *p) {
if (NULL == p) {
return;
}
SBatchRspMsg* pRsp = (SBatchRspMsg*)p;
SBatchRspMsg *pRsp = (SBatchRspMsg *)p;
rpcFreeCont(pRsp->msg);
}
int32_t vnodeGetBatchMeta(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
int32_t rspSize = 0;
SBatchReq batchReq = {0};
SBatchMsg *req = NULL;
int32_t code = 0;
int32_t rspSize = 0;
SBatchReq batchReq = {0};
SBatchMsg *req = NULL;
SBatchRspMsg rsp = {0};
SBatchRsp batchRsp = {0};
SRpcMsg reqMsg = *pMsg;
SRpcMsg rspMsg = {0};
void *pRsp = NULL;
SBatchRsp batchRsp = {0};
SRpcMsg reqMsg = *pMsg;
SRpcMsg rspMsg = {0};
void *pRsp = NULL;
if (tDeserializeSBatchReq(pMsg->pCont, pMsg->contLen, &batchReq)) {
code = TSDB_CODE_OUT_OF_MEMORY;
@ -291,7 +291,7 @@ int32_t vnodeGetBatchMeta(SVnode *pVnode, SRpcMsg *pMsg) {
goto _exit;
}
int32_t msgNum = taosArrayGetSize(batchReq.pMsgs);
int32_t msgNum = taosArrayGetSize(batchReq.pMsgs);
if (msgNum >= MAX_META_MSG_IN_BATCH) {
code = TSDB_CODE_INVALID_MSG;
qError("too many msgs %d in vnode batch meta req", msgNum);
@ -405,7 +405,8 @@ void vnodeResetLoad(SVnode *pVnode, SVnodeLoad *pLoad) {
VNODE_GET_LOAD_RESET_VALS(pVnode->statis.nInsert, pLoad->numOfInsertReqs, 64, "nInsert");
VNODE_GET_LOAD_RESET_VALS(pVnode->statis.nInsertSuccess, pLoad->numOfInsertSuccessReqs, 64, "nInsertSuccess");
VNODE_GET_LOAD_RESET_VALS(pVnode->statis.nBatchInsert, pLoad->numOfBatchInsertReqs, 64, "nBatchInsert");
VNODE_GET_LOAD_RESET_VALS(pVnode->statis.nBatchInsertSuccess, pLoad->numOfBatchInsertSuccessReqs, 64, "nBatchInsertSuccess");
VNODE_GET_LOAD_RESET_VALS(pVnode->statis.nBatchInsertSuccess, pLoad->numOfBatchInsertSuccessReqs, 64,
"nBatchInsertSuccess");
}
void vnodeGetInfo(SVnode *pVnode, const char **dbname, int32_t *vgId) {

802
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -13,7 +13,11 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "tencode.h"
#include "tmsg.h"
#include "vnd.h"
#include "vnode.h"
#include "vnodeInt.h"
static int32_t vnodeProcessCreateStbReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessAlterStbReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
@ -29,160 +33,258 @@ static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t version, void *p
static int32_t vnodeProcessTrimReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessBatchDeleteReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessCreateIndexReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessDropIndexReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessCompactVnodeReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp);
int32_t vnodePreProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
static int32_t vnodePreprocessCreateTableReq(SVnode *pVnode, SDecoder *pCoder, int64_t ctime, int64_t *pUid) {
int32_t code = 0;
int32_t lino = 0;
if (tStartDecode(pCoder) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
// flags
if (tDecodeI32v(pCoder, NULL) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
// name
char *name = NULL;
if (tDecodeCStr(pCoder, &name) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
// uid
int64_t uid = metaGetTableEntryUidByName(pVnode->pMeta, name);
if (uid == 0) {
uid = tGenIdPI64();
}
*(int64_t *)(pCoder->data + pCoder->pos) = uid;
// ctime
*(int64_t *)(pCoder->data + pCoder->pos + 8) = ctime;
tEndDecode(pCoder);
_exit:
if (code) {
vError("vgId:%d %s failed at line %d since %s", TD_VID(pVnode), __func__, lino, tstrerror(code));
} else {
vTrace("vgId:%d %s done, table:%s uid generated:%" PRId64, TD_VID(pVnode), __func__, name, uid);
if (pUid) *pUid = uid;
}
return code;
}
static int32_t vnodePreProcessCreateTableMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
int32_t lino = 0;
int64_t ctime = taosGetTimestampMs();
SDecoder dc = {0};
int32_t nReqs;
tDecoderInit(&dc, (uint8_t *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead));
if (tStartDecode(&dc) < 0) {
code = TSDB_CODE_INVALID_MSG;
return code;
}
if (tDecodeI32v(&dc, &nReqs) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
for (int32_t iReq = 0; iReq < nReqs; iReq++) {
code = vnodePreprocessCreateTableReq(pVnode, &dc, ctime, NULL);
TSDB_CHECK_CODE(code, lino, _exit);
}
tEndDecode(&dc);
_exit:
tDecoderClear(&dc);
return code;
}
extern int64_t tsMaxKeyByPrecision[];
static int32_t vnodePreProcessSubmitTbData(SVnode *pVnode, SDecoder *pCoder, int64_t ctime) {
int32_t code = 0;
int32_t lino = 0;
if (tStartDecode(pCoder) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
SSubmitTbData submitTbData;
if (tDecodeI32v(pCoder, &submitTbData.flags) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
int64_t uid;
if (submitTbData.flags & SUBMIT_REQ_AUTO_CREATE_TABLE) {
code = vnodePreprocessCreateTableReq(pVnode, pCoder, ctime, &uid);
TSDB_CHECK_CODE(code, lino, _exit);
}
// submit data
if (tDecodeI64(pCoder, &submitTbData.suid) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
if (submitTbData.flags & SUBMIT_REQ_AUTO_CREATE_TABLE) {
*(int64_t *)(pCoder->data + pCoder->pos) = uid;
pCoder->pos += sizeof(int64_t);
} else {
tDecodeI64(pCoder, &submitTbData.uid);
}
if (tDecodeI32v(pCoder, &submitTbData.sver) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
// scan and check
TSKEY now = ctime;
if (pVnode->config.tsdbCfg.precision == TSDB_TIME_PRECISION_MICRO) {
now *= 1000;
} else if (pVnode->config.tsdbCfg.precision == TSDB_TIME_PRECISION_NANO) {
now *= 1000000;
}
TSKEY minKey = now - tsTickPerMin[pVnode->config.tsdbCfg.precision] * pVnode->config.tsdbCfg.keep2;
TSKEY maxKey = tsMaxKeyByPrecision[pVnode->config.tsdbCfg.precision];
if (submitTbData.flags & SUBMIT_REQ_COLUMN_DATA_FORMAT) {
uint64_t nColData;
if (tDecodeU64v(pCoder, &nColData) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _exit;
}
SColData colData = {0};
pCoder->pos += tGetColData(pCoder->data + pCoder->pos, &colData);
for (int32_t iRow = 0; iRow < colData.nVal; iRow++) {
if (((TSKEY *)colData.pData)[iRow] < minKey || ((TSKEY *)colData.pData)[iRow] > maxKey) {
code = TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE;
goto _exit;
}
}
} else {
uint64_t nRow;
if (tDecodeU64v(pCoder, &nRow) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _exit;
}
for (int32_t iRow = 0; iRow < nRow; ++iRow) {
SRow *pRow = (SRow *)(pCoder->data + pCoder->pos);
pCoder->pos += pRow->len;
if (pRow->ts < minKey || pRow->ts > maxKey) {
code = TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE;
goto _exit;
}
}
}
tEndDecode(pCoder);
_exit:
return code;
}
static int32_t vnodePreProcessSubmitMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
int32_t lino = 0;
SDecoder *pCoder = &(SDecoder){0};
tDecoderInit(pCoder, (uint8_t *)pMsg->pCont + sizeof(SSubmitReq2Msg), pMsg->contLen - sizeof(SSubmitReq2Msg));
if (tStartDecode(pCoder) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
uint64_t nSubmitTbData;
if (tDecodeU64v(pCoder, &nSubmitTbData) < 0) {
code = TSDB_CODE_INVALID_MSG;
TSDB_CHECK_CODE(code, lino, _exit);
}
int64_t ctime = taosGetTimestampMs();
for (int32_t i = 0; i < nSubmitTbData; i++) {
code = vnodePreProcessSubmitTbData(pVnode, pCoder, ctime);
TSDB_CHECK_CODE(code, lino, _exit);
}
tEndDecode(pCoder);
_exit:
tDecoderClear(pCoder);
return code;
}
static int32_t vnodePreProcessDeleteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
int32_t size;
int32_t ret;
uint8_t *pCont;
SEncoder *pCoder = &(SEncoder){0};
SDeleteRes res = {0};
SReadHandle handle = {.meta = pVnode->pMeta, .config = &pVnode->config, .vnode = pVnode, .pMsgCb = &pVnode->msgCb};
code = qWorkerProcessDeleteMsg(&handle, pVnode->pQuery, pMsg, &res);
if (code) goto _exit;
// malloc and encode
tEncodeSize(tEncodeDeleteRes, &res, size, ret);
pCont = rpcMallocCont(size + sizeof(SMsgHead));
((SMsgHead *)pCont)->contLen = size + sizeof(SMsgHead);
((SMsgHead *)pCont)->vgId = TD_VID(pVnode);
tEncoderInit(pCoder, pCont + sizeof(SMsgHead), size);
tEncodeDeleteRes(pCoder, &res);
tEncoderClear(pCoder);
rpcFreeCont(pMsg->pCont);
pMsg->pCont = pCont;
pMsg->contLen = size + sizeof(SMsgHead);
taosArrayDestroy(res.uidList);
_exit:
return code;
}
int32_t vnodePreProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
switch (pMsg->msgType) {
case TDMT_VND_CREATE_TABLE: {
int64_t ctime = taosGetTimestampMs();
int32_t nReqs;
tDecoderInit(&dc, (uint8_t *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead));
if (tStartDecode(&dc) < 0) {
code = TSDB_CODE_INVALID_MSG;
return code;
}
if (tDecodeI32v(&dc, &nReqs) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
for (int32_t iReq = 0; iReq < nReqs; iReq++) {
tb_uid_t uid = tGenIdPI64();
char *name = NULL;
if (tStartDecode(&dc) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
if (tDecodeI32v(&dc, NULL) < 0) {
code = TSDB_CODE_INVALID_MSG;
return code;
}
if (tDecodeCStr(&dc, &name) < 0) {
code = TSDB_CODE_INVALID_MSG;
return code;
}
*(int64_t *)(dc.data + dc.pos) = uid;
*(int64_t *)(dc.data + dc.pos + 8) = ctime;
vTrace("vgId:%d, table:%s uid:%" PRId64 " is generated", pVnode->config.vgId, name, uid);
tEndDecode(&dc);
}
tEndDecode(&dc);
tDecoderClear(&dc);
code = vnodePreProcessCreateTableMsg(pVnode, pMsg);
} break;
case TDMT_VND_SUBMIT: {
int64_t ctime = taosGetTimestampMs();
tDecoderInit(&dc, (uint8_t *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead));
tStartDecode(&dc);
uint64_t nSubmitTbData;
if (tDecodeU64v(&dc, &nSubmitTbData) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
for (int32_t i = 0; i < nSubmitTbData; i++) {
if (tStartDecode(&dc) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
int32_t flags;
if (tDecodeI32v(&dc, &flags) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
if (flags & SUBMIT_REQ_AUTO_CREATE_TABLE) {
// SVCreateTbReq
if (tStartDecode(&dc) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
if (tDecodeI32v(&dc, NULL) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
char *name = NULL;
if (tDecodeCStr(&dc, &name) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
int64_t uid = metaGetTableEntryUidByName(pVnode->pMeta, name);
if (uid == 0) {
uid = tGenIdPI64();
}
*(int64_t *)(dc.data + dc.pos) = uid;
*(int64_t *)(dc.data + dc.pos + 8) = ctime;
tEndDecode(&dc);
// SSubmitTbData
int64_t suid;
if (tDecodeI64(&dc, &suid) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _err;
}
*(int64_t *)(dc.data + dc.pos) = uid;
}
tEndDecode(&dc);
}
tEndDecode(&dc);
tDecoderClear(&dc);
code = vnodePreProcessSubmitMsg(pVnode, pMsg);
} break;
case TDMT_VND_DELETE: {
int32_t size;
int32_t ret;
uint8_t *pCont;
SEncoder *pCoder = &(SEncoder){0};
SDeleteRes res = {0};
SReadHandle handle = {
.meta = pVnode->pMeta, .config = &pVnode->config, .vnode = pVnode, .pMsgCb = &pVnode->msgCb};
code = qWorkerProcessDeleteMsg(&handle, pVnode->pQuery, pMsg, &res);
if (code) {
goto _err;
}
// malloc and encode
tEncodeSize(tEncodeDeleteRes, &res, size, ret);
pCont = rpcMallocCont(size + sizeof(SMsgHead));
((SMsgHead *)pCont)->contLen = size + sizeof(SMsgHead);
((SMsgHead *)pCont)->vgId = TD_VID(pVnode);
tEncoderInit(pCoder, pCont + sizeof(SMsgHead), size);
tEncodeDeleteRes(pCoder, &res);
tEncoderClear(pCoder);
rpcFreeCont(pMsg->pCont);
pMsg->pCont = pCont;
pMsg->contLen = size + sizeof(SMsgHead);
taosArrayDestroy(res.uidList);
code = vnodePreProcessDeleteMsg(pVnode, pMsg);
} break;
default:
break;
}
return code;
_err:
vError("vgId%d, preprocess request failed since %s", TD_VID(pVnode), tstrerror(code));
_exit:
if (code) {
vError("vgId%d failed to preprocess write request since %s, msg type:%d", TD_VID(pVnode), tstrerror(code),
pMsg->msgType);
}
return code;
}
@ -256,7 +358,7 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
break;
/* TSDB */
case TDMT_VND_SUBMIT:
if (vnodeProcessSubmitReq(pVnode, version, pReq, len, pRsp) < 0) goto _err;
if (vnodeProcessSubmitReq(pVnode, version, pMsg->pCont, pMsg->contLen, pRsp) < 0) goto _err;
break;
case TDMT_VND_DELETE:
if (vnodeProcessDeleteReq(pVnode, version, pReq, len, pRsp) < 0) goto _err;
@ -319,6 +421,12 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t version, SRp
case TDMT_VND_COMMIT:
needCommit = true;
break;
case TDMT_VND_CREATE_INDEX:
vnodeProcessCreateIndexReq(pVnode, version, pReq, len, pRsp);
break;
case TDMT_VND_DROP_INDEX:
vnodeProcessDropIndexReq(pVnode, version, pReq, len, pRsp);
break;
case TDMT_VND_COMPACT:
vnodeProcessCompactVnodeReq(pVnode, version, pReq, len, pRsp);
goto _exit;
@ -511,6 +619,13 @@ static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t version, void *p
tqUpdateTbUidList(pVnode->pTq, tbUids, false);
}
// process
ret = tsdbDoRetention(pVnode->pTsdb, ttlReq.timestamp);
if (ret) goto end;
ret = smaDoRetention(pVnode->pSma, ttlReq.timestamp);
if (ret) goto end;
end:
taosArrayDestroy(tbUids);
return ret;
@ -874,8 +989,188 @@ static int32_t vnodeDebugPrintSingleSubmitMsg(SMeta *pMeta, SSubmitBlk *pBlock,
return TSDB_CODE_SUCCESS;
}
typedef struct SSubmitReqConvertCxt {
SSubmitMsgIter msgIter;
SSubmitBlk *pBlock;
SSubmitBlkIter blkIter;
STSRow *pRow;
STSRowIter rowIter;
SSubmitTbData *pTbData;
STSchema *pTbSchema;
SArray *pColValues;
} SSubmitReqConvertCxt;
static int32_t vnodeResetTableCxt(SMeta *pMeta, SSubmitReqConvertCxt *pCxt) {
taosMemoryFreeClear(pCxt->pTbSchema);
pCxt->pTbSchema = metaGetTbTSchema(pMeta, pCxt->msgIter.suid > 0 ? pCxt->msgIter.suid : pCxt->msgIter.uid,
pCxt->msgIter.sversion, 1);
if (NULL == pCxt->pTbSchema) {
return TSDB_CODE_INVALID_MSG;
}
tdSTSRowIterInit(&pCxt->rowIter, pCxt->pTbSchema);
tDestroySSubmitTbData(pCxt->pTbData, TSDB_MSG_FLG_ENCODE);
if (NULL == pCxt->pTbData) {
pCxt->pTbData = taosMemoryCalloc(1, sizeof(SSubmitTbData));
if (NULL == pCxt->pTbData) {
return TSDB_CODE_OUT_OF_MEMORY;
}
}
pCxt->pTbData->flags = 0;
pCxt->pTbData->suid = pCxt->msgIter.suid;
pCxt->pTbData->uid = pCxt->msgIter.uid;
pCxt->pTbData->sver = pCxt->msgIter.sversion;
pCxt->pTbData->pCreateTbReq = NULL;
pCxt->pTbData->aRowP = taosArrayInit(128, POINTER_BYTES);
if (NULL == pCxt->pTbData->aRowP) {
return TSDB_CODE_OUT_OF_MEMORY;
}
taosArrayDestroy(pCxt->pColValues);
pCxt->pColValues = taosArrayInit(pCxt->pTbSchema->numOfCols, sizeof(SColVal));
if (NULL == pCxt->pColValues) {
return TSDB_CODE_OUT_OF_MEMORY;
}
for (int32_t i = 0; i < pCxt->pTbSchema->numOfCols; ++i) {
SColVal val = COL_VAL_NONE(pCxt->pTbSchema->columns[i].colId, pCxt->pTbSchema->columns[i].type);
taosArrayPush(pCxt->pColValues, &val);
}
return TSDB_CODE_SUCCESS;
}
static void vnodeDestroySubmitReqConvertCxt(SSubmitReqConvertCxt *pCxt) {
taosMemoryFreeClear(pCxt->pTbSchema);
tDestroySSubmitTbData(pCxt->pTbData, TSDB_MSG_FLG_ENCODE);
taosMemoryFreeClear(pCxt->pTbData);
taosArrayDestroy(pCxt->pColValues);
}
static int32_t vnodeCellValConvertToColVal(STColumn *pCol, SCellVal *pCellVal, SColVal *pColVal) {
if (tdValTypeIsNone(pCellVal->valType)) {
pColVal->flag = CV_FLAG_NONE;
return TSDB_CODE_SUCCESS;
}
if (tdValTypeIsNull(pCellVal->valType)) {
pColVal->flag = CV_FLAG_NULL;
return TSDB_CODE_SUCCESS;
}
if (IS_VAR_DATA_TYPE(pCol->type)) {
pColVal->value.nData = varDataLen(pCellVal->val);
pColVal->value.pData = varDataVal(pCellVal->val);
} else if (TSDB_DATA_TYPE_FLOAT == pCol->type) {
float f = GET_FLOAT_VAL(pCellVal->val);
memcpy(&pColVal->value.val, &f, sizeof(f));
} else if (TSDB_DATA_TYPE_DOUBLE == pCol->type) {
pColVal->value.val = *(int64_t *)pCellVal->val;
} else {
GET_TYPED_DATA(pColVal->value.val, int64_t, pCol->type, pCellVal->val);
}
pColVal->flag = CV_FLAG_VALUE;
return TSDB_CODE_SUCCESS;
}
static int32_t vnodeTSRowConvertToColValArray(SSubmitReqConvertCxt *pCxt) {
int32_t code = TSDB_CODE_SUCCESS;
tdSTSRowIterReset(&pCxt->rowIter, pCxt->pRow);
for (int32_t i = 0; TSDB_CODE_SUCCESS == code && i < pCxt->pTbSchema->numOfCols; ++i) {
STColumn *pCol = pCxt->pTbSchema->columns + i;
SCellVal cellVal = {0};
if (!tdSTSRowIterFetch(&pCxt->rowIter, pCol->colId, pCol->type, &cellVal)) {
break;
}
code = vnodeCellValConvertToColVal(pCol, &cellVal, (SColVal *)taosArrayGet(pCxt->pColValues, i));
}
return code;
}
static int32_t vnodeDecodeCreateTbReq(SSubmitReqConvertCxt *pCxt) {
if (pCxt->msgIter.schemaLen <= 0) {
return TSDB_CODE_SUCCESS;
}
pCxt->pTbData->pCreateTbReq = taosMemoryCalloc(1, sizeof(SVCreateTbReq));
if (NULL == pCxt->pTbData->pCreateTbReq) {
return TSDB_CODE_OUT_OF_MEMORY;
}
SDecoder decoder = {0};
tDecoderInit(&decoder, pCxt->pBlock->data, pCxt->msgIter.schemaLen);
int32_t code = tDecodeSVCreateTbReq(&decoder, pCxt->pTbData->pCreateTbReq);
tDecoderClear(&decoder);
return code;
}
static int32_t vnodeSubmitReqConvertToSubmitReq2(SVnode *pVnode, SSubmitReq *pReq, SSubmitReq2 *pReq2) {
pReq2->aSubmitTbData = taosArrayInit(128, sizeof(SSubmitTbData));
if (NULL == pReq2->aSubmitTbData) {
return TSDB_CODE_OUT_OF_MEMORY;
}
SSubmitReqConvertCxt cxt = {0};
int32_t code = tInitSubmitMsgIter(pReq, &cxt.msgIter);
while (TSDB_CODE_SUCCESS == code) {
code = tGetSubmitMsgNext(&cxt.msgIter, &cxt.pBlock);
if (TSDB_CODE_SUCCESS == code) {
if (NULL == cxt.pBlock) {
break;
}
code = vnodeResetTableCxt(pVnode->pMeta, &cxt);
}
if (TSDB_CODE_SUCCESS == code) {
code = tInitSubmitBlkIter(&cxt.msgIter, cxt.pBlock, &cxt.blkIter);
}
if (TSDB_CODE_SUCCESS == code) {
code = vnodeDecodeCreateTbReq(&cxt);
}
while (TSDB_CODE_SUCCESS == code && (cxt.pRow = tGetSubmitBlkNext(&cxt.blkIter)) != NULL) {
code = vnodeTSRowConvertToColValArray(&cxt);
if (TSDB_CODE_SUCCESS == code) {
SRow **pNewRow = taosArrayReserve(cxt.pTbData->aRowP, 1);
code = tRowBuild(cxt.pColValues, cxt.pTbSchema, pNewRow);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = (NULL == taosArrayPush(pReq2->aSubmitTbData, cxt.pTbData) ? TSDB_CODE_OUT_OF_MEMORY : TSDB_CODE_SUCCESS);
}
if (TSDB_CODE_SUCCESS == code) {
taosMemoryFreeClear(cxt.pTbData);
}
}
vnodeDestroySubmitReqConvertCxt(&cxt);
return code;
}
static int32_t vnodeRebuildSubmitReqMsg(SSubmitReq2 *pSubmitReq, void **ppMsg) {
int32_t code = TSDB_CODE_SUCCESS;
char *pMsg = NULL;
uint32_t msglen = 0;
tEncodeSize(tEncodeSSubmitReq2, pSubmitReq, msglen, code);
if (TSDB_CODE_SUCCESS == code) {
pMsg = taosMemoryMalloc(msglen);
if (NULL == pMsg) {
code = TSDB_CODE_OUT_OF_MEMORY;
}
}
if (TSDB_CODE_SUCCESS == code) {
SEncoder encoder;
tEncoderInit(&encoder, pMsg, msglen);
code = tEncodeSSubmitReq2(&encoder, pSubmitReq);
tEncoderClear(&encoder);
}
if (TSDB_CODE_SUCCESS == code) {
*ppMsg = pMsg;
}
return code;
}
static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) {
#if 1
int32_t code = 0;
terrno = 0;
@ -887,19 +1182,30 @@ static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t version, void *pReq
pRsp->code = TSDB_CODE_SUCCESS;
// decode
SDecoder dc = {0};
tDecoderInit(&dc, pReq, len);
if (tDecodeSSubmitReq2(&dc, pSubmitReq) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _exit;
}
tDecoderClear(&dc);
// check
code = tsdbScanAndConvertSubmitMsg(pVnode->pTsdb, pSubmitReq);
if (code) {
goto _exit;
void *pAllocMsg = NULL;
SSubmitReq2Msg *pMsg = (SSubmitReq2Msg *)pReq;
if (0 == pMsg->version) {
code = vnodeSubmitReqConvertToSubmitReq2(pVnode, (SSubmitReq *)pMsg, pSubmitReq);
if (TSDB_CODE_SUCCESS == code) {
code = vnodeRebuildSubmitReqMsg(pSubmitReq, &pReq);
}
if (TSDB_CODE_SUCCESS == code) {
pAllocMsg = pReq;
}
if (TSDB_CODE_SUCCESS != code) {
goto _exit;
}
} else {
// decode
pReq = POINTER_SHIFT(pReq, sizeof(SSubmitReq2Msg));
len -= sizeof(SSubmitReq2Msg);
SDecoder dc = {0};
tDecoderInit(&dc, pReq, len);
if (tDecodeSSubmitReq2(&dc, pSubmitReq) < 0) {
code = TSDB_CODE_INVALID_MSG;
goto _exit;
}
tDecoderClear(&dc);
}
for (int32_t i = 0; i < TARRAY_SIZE(pSubmitReq->aSubmitTbData); ++i) {
@ -1040,151 +1346,14 @@ _exit:
// clear
taosArrayDestroy(newTbUids);
tDestroySSubmitReq2(pSubmitReq, TSDB_MSG_FLG_DECODE);
tDestroySSubmitReq2(pSubmitReq, 0 == pMsg->version ? TSDB_MSG_FLG_CMPT : TSDB_MSG_FLG_DECODE);
tDestroySSubmitRsp2(pSubmitRsp, TSDB_MSG_FLG_ENCODE);
if (code) terrno = code;
taosMemoryFree(pAllocMsg);
return code;
#else
SSubmitReq *pSubmitReq = (SSubmitReq *)pReq;
SSubmitRsp submitRsp = {0};
int32_t nRows = 0;
int32_t tsize, ret;
SEncoder encoder = {0};
SArray *newTbUids = NULL;
SVStatis statis = {0};
bool tbCreated = false;
terrno = TSDB_CODE_SUCCESS;
pRsp->code = 0;
pSubmitReq->version = version;
statis.nBatchInsert = 1;
if (tsdbScanAndConvertSubmitMsg(pVnode->pTsdb, pSubmitReq) < 0) {
pRsp->code = terrno;
goto _exit;
}
submitRsp.pArray = taosArrayInit(msgIter.numOfBlocks, sizeof(SSubmitBlkRsp));
newTbUids = taosArrayInit(msgIter.numOfBlocks, sizeof(int64_t));
if (!submitRsp.pArray || !newTbUids) {
pRsp->code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
for (;;) {
tGetSubmitMsgNext(&msgIter, &pBlock);
if (pBlock == NULL) break;
SSubmitBlkRsp submitBlkRsp = {0};
tbCreated = false;
// create table for auto create table mode
if (msgIter.schemaLen > 0) {
// tDecoderInit(&decoder, pBlock->data, msgIter.schemaLen);
// if (tDecodeSVCreateTbReq(&decoder, &createTbReq) < 0) {
// pRsp->code = TSDB_CODE_INVALID_MSG;
// tDecoderClear(&decoder);
// taosArrayDestroy(createTbReq.ctb.tagName);
// goto _exit;
// }
// if ((terrno = grantCheck(TSDB_GRANT_TIMESERIES)) < 0) {
// pRsp->code = terrno;
// tDecoderClear(&decoder);
// taosArrayDestroy(createTbReq.ctb.tagName);
// goto _exit;
// }
// if ((terrno = grantCheck(TSDB_GRANT_TABLE)) < 0) {
// pRsp->code = terrno;
// tDecoderClear(&decoder);
// taosArrayDestroy(createTbReq.ctb.tagName);
// goto _exit;
// }
if (metaCreateTable(pVnode->pMeta, version, &createTbReq, &submitBlkRsp.pMeta) < 0) {
// if (terrno != TSDB_CODE_TDB_TABLE_ALREADY_EXIST) {
// submitBlkRsp.code = terrno;
// pRsp->code = terrno;
// tDecoderClear(&decoder);
// taosArrayDestroy(createTbReq.ctb.tagName);
// goto _exit;
// }
} else {
if (NULL != submitBlkRsp.pMeta) {
vnodeUpdateMetaRsp(pVnode, submitBlkRsp.pMeta);
}
// taosArrayPush(newTbUids, &createTbReq.uid);
submitBlkRsp.uid = createTbReq.uid;
submitBlkRsp.tblFName = taosMemoryMalloc(strlen(pVnode->config.dbname) + strlen(createTbReq.name) + 2);
sprintf(submitBlkRsp.tblFName, "%s.%s", pVnode->config.dbname, createTbReq.name);
tbCreated = true;
}
// msgIter.uid = createTbReq.uid;
// if (createTbReq.type == TSDB_CHILD_TABLE) {
// msgIter.suid = createTbReq.ctb.suid;
// } else {
// msgIter.suid = 0;
// }
// tDecoderClear(&decoder);
// taosArrayDestroy(createTbReq.ctb.tagName);
}
if (tsdbInsertTableData(pVnode->pTsdb, version, &msgIter, pBlock, &submitBlkRsp) < 0) {
submitBlkRsp.code = terrno;
}
submitRsp.numOfRows += submitBlkRsp.numOfRows;
submitRsp.affectedRows += submitBlkRsp.affectedRows;
if (tbCreated || submitBlkRsp.code) {
taosArrayPush(submitRsp.pArray, &submitBlkRsp);
}
}
// if (taosArrayGetSize(newTbUids) > 0) {
// vDebug("vgId:%d, add %d table into query table list in handling submit", TD_VID(pVnode),
// (int32_t)taosArrayGetSize(newTbUids));
// }
// tqUpdateTbUidList(pVnode->pTq, newTbUids, true);
_exit:
taosArrayDestroy(newTbUids);
// tEncodeSize(tEncodeSSubmitRsp, &submitRsp, tsize, ret);
// pRsp->pCont = rpcMallocCont(tsize);
// pRsp->contLen = tsize;
// tEncoderInit(&encoder, pRsp->pCont, tsize);
// tEncodeSSubmitRsp(&encoder, &submitRsp);
// tEncoderClear(&encoder);
taosArrayDestroyEx(submitRsp.pArray, tFreeSSubmitBlkRsp);
// TODO: the partial success scenario and the error case
// => If partial success, extract the success submitted rows and reconstruct a new submit msg, and push to level
// 1/level 2.
// TODO: refactor
if ((terrno == TSDB_CODE_SUCCESS) && (pRsp->code == TSDB_CODE_SUCCESS)) {
statis.nBatchInsertSuccess = 1;
tdProcessRSmaSubmit(pVnode->pSma, pReq, STREAM_INPUT__DATA_SUBMIT);
}
// N.B. not strict as the following procedure is not atomic
atomic_add_fetch_64(&pVnode->statis.nInsert, submitRsp.numOfRows);
atomic_add_fetch_64(&pVnode->statis.nInsertSuccess, submitRsp.affectedRows);
atomic_add_fetch_64(&pVnode->statis.nBatchInsert, statis.nBatchInsert);
atomic_add_fetch_64(&pVnode->statis.nBatchInsertSuccess, statis.nBatchInsertSuccess);
vDebug("vgId:%d, submit success, index:%" PRId64, pVnode->config.vgId, version);
return 0;
#endif
return 0;
}
static int32_t vnodeProcessCreateTSmaReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) {
@ -1408,6 +1577,49 @@ static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t version, void *pReq
_err:
return code;
}
static int32_t vnodeProcessCreateIndexReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) {
SVCreateStbReq req = {0};
SDecoder dc = {0};
pRsp->msgType = TDMT_VND_CREATE_INDEX_RSP;
pRsp->code = TSDB_CODE_SUCCESS;
pRsp->pCont = NULL;
pRsp->contLen = 0;
tDecoderInit(&dc, pReq, len);
// decode req
if (tDecodeSVCreateStbReq(&dc, &req) < 0) {
terrno = TSDB_CODE_INVALID_MSG;
tDecoderClear(&dc);
return -1;
}
if (metaAddIndexToSTable(pVnode->pMeta, version, &req) < 0) {
pRsp->code = terrno;
goto _err;
}
tDecoderClear(&dc);
return 0;
_err:
tDecoderClear(&dc);
return -1;
}
static int32_t vnodeProcessDropIndexReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) {
SDropIndexReq req = {0};
pRsp->msgType = TDMT_VND_CREATE_INDEX_RSP;
pRsp->code = TSDB_CODE_SUCCESS;
pRsp->pCont = NULL;
pRsp->contLen = 0;
if (tDeserializeSDropIdxReq(pReq, len, &req)) {
terrno = TSDB_CODE_INVALID_MSG;
return -1;
}
if (metaDropIndexFromSTable(pVnode->pMeta, version, &req) < 0) {
pRsp->code = terrno;
return -1;
}
return TSDB_CODE_SUCCESS;
}
static int32_t vnodeProcessCompactVnodeReq(SVnode *pVnode, int64_t version, void *pReq, int32_t len, SRpcMsg *pRsp) {
SCompactVnodeReq req = {0};
@ -1422,4 +1634,4 @@ static int32_t vnodeProcessCompactVnodeReq(SVnode *pVnode, int64_t version, void
vnodeBegin(pVnode);
return 0;
}
}

2
source/libs/command/inc/commandInt.h
View File

@ -101,6 +101,8 @@ extern "C" {
#define COMMAND_SCHEDULE_POLICY "schedulePolicy"
#define COMMAND_ENABLE_RESCHEDULE "enableReSchedule"
#define COMMAND_CATALOG_DEBUG "catalogDebug"
#define COMMAND_ENABLE_MEM_DEBUG "enableMemDebug"
#define COMMAND_DISABLE_MEM_DEBUG "disableMemDebug"
typedef struct SExplainGroup {
int32_t nodeNum;

15
source/libs/command/src/command.c
View File

@ -687,6 +687,21 @@ static int32_t execAlterCmd(char* cmd, char* value, bool* processed) {
code = schedulerEnableReSchedule(atoi(value));
} else if (0 == strcasecmp(cmd, COMMAND_CATALOG_DEBUG)) {
code = ctgdHandleDbgCommand(value);
} else if (0 == strcasecmp(cmd, COMMAND_ENABLE_MEM_DEBUG)) {
code = taosMemoryDbgInit();
if (code) {
qError("failed to init memory dbg, error:%s", tstrerror(code));
return code;
}
tsAsyncLog = false;
qInfo("memory dbg enabled");
} else if (0 == strcasecmp(cmd, COMMAND_DISABLE_MEM_DEBUG)) {
code = taosMemoryDbgInitRestore();
if (code) {
qError("failed to restore from memory dbg, error:%s", tstrerror(code));
return code;
}
qInfo("memory dbg disabled");
} else {
goto _return;
}

2
source/libs/executor/inc/executorimpl.h
View File

@ -873,7 +873,7 @@ void getNextIntervalWindow(SInterval* pInterval, STimeWindow* tw, int32_t ord
int32_t qAppendTaskStopInfo(SExecTaskInfo* pTaskInfo, SExchangeOpStopInfo* pInfo);
int32_t getForwardStepsInBlock(int32_t numOfRows, __block_search_fn_t searchFn, TSKEY ekey, int32_t pos, int32_t order,
int64_t* pData);
void appendCreateTableRow(SStreamState* pState, SExprSupp* pTableSup, SExprSupp* pTagSup, int64_t groupId,
void appendCreateTableRow(SStreamState* pState, SExprSupp* pTableSup, SExprSupp* pTagSup, uint64_t groupId,
SSDataBlock* pSrcBlock, int32_t rowId, SSDataBlock* pDestBlock);
SSDataBlock* buildCreateTableBlock(SExprSupp* tbName, SExprSupp* tag);

9
source/libs/executor/src/dataInserter.c
View File

@ -129,14 +129,15 @@ static int32_t submitReqToMsg(int32_t vgId, SSubmitReq2* pReq, void** pData, int
tEncodeSize(tEncodeSSubmitReq2, pReq, len, code);
if (TSDB_CODE_SUCCESS == code) {
SEncoder encoder;
len += sizeof(SMsgHead);
len += sizeof(SSubmitReq2Msg);
pBuf = taosMemoryMalloc(len);
if (NULL == pBuf) {
return TSDB_CODE_OUT_OF_MEMORY;
}
((SMsgHead*)pBuf)->vgId = htonl(vgId);
((SMsgHead*)pBuf)->contLen = htonl(len);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SMsgHead)), len - sizeof(SMsgHead));
((SSubmitReq2Msg*)pBuf)->header.vgId = htonl(vgId);
((SSubmitReq2Msg*)pBuf)->header.contLen = htonl(len);
((SSubmitReq2Msg*)pBuf)->version = htobe64(1);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SSubmitReq2Msg)), len - sizeof(SSubmitReq2Msg));
code = tEncodeSSubmitReq2(&encoder, pReq);
tEncoderClear(&encoder);
}

61
source/libs/executor/src/eventwindowoperator.c
View File

@ -26,23 +26,19 @@ typedef struct SEventWindowOperatorInfo {
SOptrBasicInfo binfo;
SAggSupporter aggSup;
SExprSupp scalarSup;
SGroupResInfo groupResInfo;
SWindowRowsSup winSup;
bool hasKey;
SStateKeys stateKey;
int32_t tsSlotId; // primary timestamp column slot id
STimeWindowAggSupp twAggSup;
SFilterInfo* pStartCondInfo;
SFilterInfo* pEndCondInfo;
bool inWindow;
SResultRow* pRow;
uint64_t groupId; // current group id, used to identify the data block from different groups
SFilterInfo* pStartCondInfo;
SFilterInfo* pEndCondInfo;
bool inWindow;
SResultRow* pRow;
} SEventWindowOperatorInfo;
static SSDataBlock* eventWindowAggregate(SOperatorInfo* pOperator);
static void destroyEWindowOperatorInfo(void* param);
static void eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInfo, SSDataBlock* pBlock);
static SSDataBlock* doEventWindowAgg(SOperatorInfo* pOperator);
static int32_t eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInfo, SSDataBlock* pBlock);
// todo : move to util
static void doKeepNewWindowStartInfo(SWindowRowsSup* pRowSup, const int64_t* tsList, int32_t rowIndex,
@ -176,7 +172,6 @@ void destroyEWindowOperatorInfo(void* param) {
colDataDestroy(&pInfo->twAggSup.timeWindowData);
cleanupAggSup(&pInfo->aggSup);
cleanupGroupResInfo(&pInfo->groupResInfo);
taosMemoryFreeClear(param);
}
@ -254,36 +249,45 @@ static void doEventWindowAggImpl(SEventWindowOperatorInfo* pInfo, SExprSupp* pSu
pBlock->info.rows, numOfOutput);
}
void eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInfo, SSDataBlock* pBlock) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SExprSupp* pSup = &pOperator->exprSupp;
SSDataBlock* pRes = pInfo->binfo.pRes;
int64_t gid = pBlock->info.id.groupId;
int32_t eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInfo, SSDataBlock* pBlock) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SExprSupp* pSup = &pOperator->exprSupp;
SSDataBlock* pRes = pInfo->binfo.pRes;
int64_t gid = pBlock->info.id.groupId;
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, pInfo->tsSlotId);
TSKEY* tsList = (TSKEY*)pColInfoData->pData;
SWindowRowsSup* pRowSup = &pInfo->winSup;
SColumnInfoData *ps = NULL, *pe = NULL;
int32_t rowIndex = 0;
SWindowRowsSup* pRowSup = &pInfo->winSup;
pRowSup->numOfRows = 0;
if (pInfo->groupId == 0) {
pInfo->groupId = gid;
} else if (pInfo->groupId != gid) {
// this is a new group, reset the info
pInfo->inWindow = false;
}
SFilterColumnParam param1 = {.numOfCols = taosArrayGetSize(pBlock->pDataBlock), .pDataBlock = pBlock->pDataBlock};
int32_t code = filterSetDataFromSlotId(pInfo->pStartCondInfo, &param1);
int32_t code = filterSetDataFromSlotId(pInfo->pStartCondInfo, &param1);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
int32_t status1 = 0;
bool keep1 = filterExecute(pInfo->pStartCondInfo, pBlock, &ps, NULL, param1.numOfCols, &status1);
filterExecute(pInfo->pStartCondInfo, pBlock, &ps, NULL, param1.numOfCols, &status1);
SFilterColumnParam param2 = {.numOfCols = taosArrayGetSize(pBlock->pDataBlock), .pDataBlock = pBlock->pDataBlock};
code = filterSetDataFromSlotId(pInfo->pEndCondInfo, &param2);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
int32_t status2 = 0;
bool keep2 = filterExecute(pInfo->pEndCondInfo, pBlock, &pe, NULL, param2.numOfCols, &status2);
filterExecute(pInfo->pEndCondInfo, pBlock, &pe, NULL, param2.numOfCols, &status2);
int32_t rowIndex = 0;
int32_t startIndex = pInfo->inWindow ? 0 : -1;
while (rowIndex < pBlock->info.rows) {
if (pInfo->inWindow) { // let's find the first end value
for (rowIndex = startIndex; rowIndex < pBlock->info.rows; ++rowIndex) {
@ -294,7 +298,6 @@ void eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInf
if (rowIndex < pBlock->info.rows) {
doEventWindowAggImpl(pInfo, pSup, startIndex, rowIndex, pBlock, tsList, pTaskInfo);
doUpdateNumOfRows(pSup->pCtx, pInfo->pRow, pSup->numOfExprs, pSup->rowEntryInfoOffset);
// check buffer size
@ -324,9 +327,9 @@ void eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInf
}
if (pInfo->inWindow) {
continue;
continue; // try to find the end position
} else {
break;
break; // no valid start position, quit
}
}
}
@ -335,4 +338,6 @@ void eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInf
taosMemoryFree(ps);
colDataDestroy(pe);
taosMemoryFree(pe);
return TSDB_CODE_SUCCESS;
}

46
source/libs/executor/src/executil.c
View File

@ -43,7 +43,14 @@ typedef struct tagFilterAssist {
SArray* cInfoList;
} tagFilterAssist;
static int32_t removeInvalidTable(SArray* uids, SHashObj* tags);
typedef enum {
FILTER_NO_LOGIC = 1,
FILTER_AND,
FILTER_OTHER,
} FilterCondType;
static FilterCondType checkTagCond(SNode* cond);
static int32_t removeInvalidTable(SArray* uids, SHashObj* tags);
static int32_t optimizeTbnameInCond(void* metaHandle, int64_t suid, SArray* list, SNode* pTagCond, SHashObj* tags);
static int32_t optimizeTbnameInCondImpl(void* metaHandle, int64_t suid, SArray* list, SNode* pTagCond);
static int32_t getTableList(void* metaHandle, void* pVnode, SScanPhysiNode* pScanNode, SNode* pTagCond,
@ -392,7 +399,8 @@ static int32_t createResultData(SDataType* pType, int32_t numOfRows, SScalarPara
return TSDB_CODE_SUCCESS;
}
static SColumnInfoData* getColInfoResult(void* metaHandle, int64_t suid, SArray* uidList, SNode* pTagCond) {
static SColumnInfoData* getColInfoResult(void* metaHandle, int64_t suid, SArray* uidList, SNode* pTagCond,
SIdxFltStatus status) {
int32_t code = TSDB_CODE_SUCCESS;
SArray* pBlockList = NULL;
SSDataBlock* pResBlock = NULL;
@ -430,14 +438,22 @@ static SColumnInfoData* getColInfoResult(void* metaHandle, int64_t suid, SArray*
// int64_t stt = taosGetTimestampUs();
tags = taosHashInit(32, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK);
FilterCondType condType = checkTagCond(pTagCond);
int32_t filter = optimizeTbnameInCond(metaHandle, suid, uidList, pTagCond, tags);
if (filter == -1) {
code = metaGetTableTags(metaHandle, suid, uidList, tags);
if ((condType == FILTER_NO_LOGIC || condType == FILTER_AND) && status != SFLT_NOT_INDEX) {
code = metaGetTableTagsByUids(metaHandle, suid, uidList, tags);
} else {
code = metaGetTableTags(metaHandle, suid, uidList, tags);
}
if (code != TSDB_CODE_SUCCESS) {
qError("failed to get table tags from meta, reason:%s, suid:%" PRIu64, tstrerror(code), suid);
terrno = code;
goto end;
}
} else {
qDebug("succ to get table tags from meta by tbname in cond, suid:%" PRIu64, suid);
}
if (suid != 0) {
removeInvalidTable(uidList, tags);
@ -842,6 +858,15 @@ static int tableUidCompare(const void* a, const void* b) {
return u1 < u2 ? -1 : 1;
}
static FilterCondType checkTagCond(SNode* cond) {
if (nodeType(cond) == QUERY_NODE_OPERATOR) {
return FILTER_NO_LOGIC;
}
if (nodeType(cond) != QUERY_NODE_LOGIC_CONDITION || ((SLogicConditionNode*)cond)->condType != LOGIC_COND_TYPE_AND) {
return FILTER_AND;
}
return FILTER_OTHER;
}
static int32_t optimizeTbnameInCond(void* metaHandle, int64_t suid, SArray* list, SNode* cond, SHashObj* tags) {
int32_t ret = -1;
if (nodeType(cond) == QUERY_NODE_OPERATOR) {
@ -954,7 +979,7 @@ static int32_t optimizeTbnameInCondImpl(void* metaHandle, int64_t suid, SArray*
return -1;
}
} else {
// qWarn("failed to get tableIds from by table name: %s, reason: %s", name, tstrerror(terrno));
// qWarn("failed to get tableIds from by table name: %s, reason: %s", name, tstrerror(terrno));
terrno = 0;
}
}
@ -983,13 +1008,14 @@ static void genTagFilterDigest(const SNode* pTagCond, T_MD5_CTX* pContext) {
taosMemoryFree(payload);
}
static int32_t doFilterByTagCond(STableListInfo* pListInfo, SArray* res, SNode* pTagCond, void* metaHandle) {
static int32_t doFilterByTagCond(STableListInfo* pListInfo, SArray* res, SNode* pTagCond, void* metaHandle,
SIdxFltStatus status) {
if (pTagCond == NULL) {
return TSDB_CODE_SUCCESS;
}
terrno = TDB_CODE_SUCCESS;
SColumnInfoData* pColInfoData = getColInfoResult(metaHandle, pListInfo->suid, res, pTagCond);
SColumnInfoData* pColInfoData = getColInfoResult(metaHandle, pListInfo->suid, res, pTagCond, status);
if (terrno != TDB_CODE_SUCCESS) {
colDataDestroy(pColInfoData);
taosMemoryFreeClear(pColInfoData);
@ -1034,12 +1060,13 @@ int32_t getTableList(void* metaHandle, void* pVnode, SScanPhysiNode* pScanNode,
pListInfo->suid = pScanNode->suid;
SArray* res = taosArrayInit(8, sizeof(uint64_t));
SIdxFltStatus status = SFLT_NOT_INDEX;
if (pScanNode->tableType != TSDB_SUPER_TABLE) {
if (metaIsTableExist(metaHandle, tableUid)) {
taosArrayPush(res, &tableUid);
}
code = doFilterByTagCond(pListInfo, res, pTagCond, metaHandle);
code = doFilterByTagCond(pListInfo, res, pTagCond, metaHandle, status);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
@ -1064,16 +1091,17 @@ int32_t getTableList(void* metaHandle, void* pVnode, SScanPhysiNode* pScanNode,
SIndexMetaArg metaArg = {
.metaEx = metaHandle, .idx = tsdbGetIdx(metaHandle), .ivtIdx = tsdbGetIvtIdx(metaHandle), .suid = tableUid};
SIdxFltStatus status = SFLT_NOT_INDEX;
code = doFilterTag(pTagIndexCond, &metaArg, res, &status);
if (code != 0 || status == SFLT_NOT_INDEX) {
qError("failed to get tableIds from index, reason:%s, suid:%" PRIu64, tstrerror(code), tableUid);
code = TDB_CODE_SUCCESS;
} else {
qInfo("succ to get filter result, table num: %d", (int)taosArrayGetSize(res));
}
}
}
code = doFilterByTagCond(pListInfo, res, pTagCond, metaHandle);
code = doFilterByTagCond(pListInfo, res, pTagCond, metaHandle, status);
if (code != TSDB_CODE_SUCCESS) {
return code;
}

6
source/libs/executor/src/filloperator.c
View File

@ -1259,9 +1259,11 @@ static SSDataBlock* doStreamFill(SOperatorInfo* pOperator) {
memcpy(pInfo->pSrcBlock->info.parTbName, pBlock->info.parTbName, TSDB_TABLE_NAME_LEN);
pInfo->srcRowIndex = 0;
} break;
case STREAM_CREATE_CHILD_TABLE: {
return pBlock;
} break;
default:
ASSERT(0);
break;
ASSERTS(pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");
}
}

36
source/libs/executor/src/groupoperator.c
View File

@ -427,9 +427,11 @@ static SSDataBlock* hashGroupbyAggregate(SOperatorInfo* pOperator) {
}
SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SAggPhysiNode* pAggNode, SExecTaskInfo* pTaskInfo) {
int32_t code = TSDB_CODE_SUCCESS;
SGroupbyOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SGroupbyOperatorInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
if (pInfo == NULL || pOperator == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _error;
}
@ -443,7 +445,7 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SAggPhysiNode*
}
pInfo->pGroupCols = extractColumnInfo(pAggNode->pGroupKeys);
int32_t code = initExprSupp(&pInfo->scalarSup, pScalarExprInfo, numOfScalarExpr);
code = initExprSupp(&pInfo->scalarSup, pScalarExprInfo, numOfScalarExpr);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
@ -482,7 +484,7 @@ SOperatorInfo* createGroupOperatorInfo(SOperatorInfo* downstream, SAggPhysiNode*
return pOperator;
_error:
pTaskInfo->code = TSDB_CODE_OUT_OF_MEMORY;
pTaskInfo->code = code;
if (pInfo != NULL) {
destroyGroupOperatorInfo(pInfo);
}
@ -983,20 +985,23 @@ static SSDataBlock* buildStreamPartitionResult(SOperatorInfo* pOperator) {
return pDest;
}
void appendCreateTableRow(SStreamState* pState, SExprSupp* pTableSup, SExprSupp* pTagSup, int64_t groupId,
void appendCreateTableRow(SStreamState* pState, SExprSupp* pTableSup, SExprSupp* pTagSup, uint64_t groupId,
SSDataBlock* pSrcBlock, int32_t rowId, SSDataBlock* pDestBlock) {
void* pValue = NULL;
if (streamStateGetParName(pState, groupId, &pValue) != 0) {
SSDataBlock* pTmpBlock = blockCopyOneRow(pSrcBlock, rowId);
memset(pTmpBlock->info.parTbName, 0, TSDB_TABLE_NAME_LEN);
pTmpBlock->info.id.groupId = groupId;
if (pTableSup->numOfExprs > 0) {
projectApplyFunctions(pTableSup->pExprInfo, pDestBlock, pTmpBlock, pTableSup->pCtx, pTableSup->numOfExprs, NULL);
SColumnInfoData* pTbCol = taosArrayGet(pDestBlock->pDataBlock, UD_TABLE_NAME_COLUMN_INDEX);
void* pData = colDataGetVarData(pTbCol, pDestBlock->info.rows - 1);
char* tbName = pSrcBlock->info.parTbName;
void* pData = colDataGetVarData(pTbCol, pDestBlock->info.rows - 1);
char* tbName = pSrcBlock->info.parTbName;
memset(tbName, 0, TSDB_TABLE_NAME_LEN);
int32_t len = TMIN(varDataLen(pData), TSDB_TABLE_NAME_LEN - 1);
memcpy(tbName, varDataVal(pData), len);
streamStatePutParName(pState, groupId, tbName);
memcpy(pTmpBlock->info.parTbName, tbName, len);
pDestBlock->info.rows--;
} else {
void* pTbNameCol = taosArrayGet(pDestBlock->pDataBlock, UD_TABLE_NAME_COLUMN_INDEX);
@ -1011,7 +1016,6 @@ void appendCreateTableRow(SStreamState* pState, SExprSupp* pTableSup, SExprSupp*
void* pGpIdCol = taosArrayGet(pDestBlock->pDataBlock, UD_GROUPID_COLUMN_INDEX);
colDataAppend(pGpIdCol, pDestBlock->info.rows, (const char*)&groupId, false);
pDestBlock->info.rows++;
blockDataDestroy(pTmpBlock);
}
@ -1020,18 +1024,19 @@ void appendCreateTableRow(SStreamState* pState, SExprSupp* pTableSup, SExprSupp*
static SSDataBlock* buildStreamCreateTableResult(SOperatorInfo* pOperator) {
SStreamPartitionOperatorInfo* pInfo = pOperator->info;
if ( (pInfo->tbnameCalSup.numOfExprs == 0 && pInfo->tagCalSup.numOfExprs == 0) || taosHashGetSize(pInfo->pPartitions) == 0) {
if ((pInfo->tbnameCalSup.numOfExprs == 0 && pInfo->tagCalSup.numOfExprs == 0) ||
taosHashGetSize(pInfo->pPartitions) == 0) {
return NULL;
}
blockDataCleanup(pInfo->pCreateTbRes);
blockDataEnsureCapacity(pInfo->pCreateTbRes, taosHashGetSize(pInfo->pPartitions));
SSDataBlock* pSrc = pInfo->pInputDataBlock;
while (pInfo->pTbNameIte != NULL) {
if (pInfo->pTbNameIte != NULL) {
SPartitionDataInfo* pParInfo = (SPartitionDataInfo*)pInfo->pTbNameIte;
int32_t rowId = *(int32_t*) taosArrayGet(pParInfo->rowIds, 0);
int32_t rowId = *(int32_t*)taosArrayGet(pParInfo->rowIds, 0);
appendCreateTableRow(pOperator->pTaskInfo->streamInfo.pState, &pInfo->tbnameCalSup, &pInfo->tagCalSup,
pParInfo->groupId, pSrc, rowId, pInfo->pCreateTbRes);
pParInfo->groupId, pSrc, rowId, pInfo->pCreateTbRes);
pInfo->pTbNameIte = taosHashIterate(pInfo->pPartitions, pInfo->pTbNameIte);
}
return pInfo->pCreateTbRes->info.rows > 0 ? pInfo->pCreateTbRes : NULL;
@ -1200,12 +1205,14 @@ SSDataBlock* buildCreateTableBlock(SExprSupp* tbName, SExprSupp* tag) {
SOperatorInfo* createStreamPartitionOperatorInfo(SOperatorInfo* downstream, SStreamPartitionPhysiNode* pPartNode,
SExecTaskInfo* pTaskInfo) {
int32_t code = TSDB_CODE_SUCCESS;
SStreamPartitionOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SStreamPartitionOperatorInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
if (pInfo == NULL || pOperator == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _error;
}
int32_t code = TSDB_CODE_SUCCESS;
pInfo->partitionSup.pGroupCols = extractPartitionColInfo(pPartNode->part.pPartitionKeys);
if (pPartNode->part.pExprs != NULL) {
@ -1237,11 +1244,11 @@ SOperatorInfo* createStreamPartitionOperatorInfo(SOperatorInfo* downstream, SStr
int32_t numOfTags;
SExprInfo* pTagExpr = createExpr(pPartNode->pTags, &numOfTags);
if (pTagExpr == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
code = TSDB_CODE_OUT_OF_MEMORY;
goto _error;
}
if (initExprSupp(&pInfo->tagCalSup, pTagExpr, numOfTags) != 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
code = TSDB_CODE_OUT_OF_MEMORY;
goto _error;
}
}
@ -1262,6 +1269,7 @@ SOperatorInfo* createStreamPartitionOperatorInfo(SOperatorInfo* downstream, SStr
pInfo->binfo.pRes = createDataBlockFromDescNode(pPartNode->part.node.pOutputDataBlockDesc);
if (pInfo->binfo.pRes == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _error;
}
@ -1291,7 +1299,7 @@ SOperatorInfo* createStreamPartitionOperatorInfo(SOperatorInfo* downstream, SStr
return pOperator;
_error:
pTaskInfo->code = TSDB_CODE_OUT_OF_MEMORY;
pTaskInfo->code = code;
destroyStreamPartitionOperatorInfo(pInfo);
taosMemoryFreeClear(pOperator);
return NULL;

15
source/libs/executor/src/scanoperator.c
View File

@ -2304,13 +2304,14 @@ SOperatorInfo* createStreamScanOperatorInfo(SReadHandle* pHandle, STableScanPhys
if (pHandle->initTableReader) {
pTSInfo->scanMode = TABLE_SCAN__TABLE_ORDER;
pTSInfo->base.dataReader = NULL;
code = tsdbReaderOpen(pHandle->vnode, &pTSInfo->base.cond, pList, num, pTSInfo->pResBlock,
&pTSInfo->base.dataReader, NULL);
if (code != 0) {
terrno = code;
destroyTableScanOperatorInfo(pTableScanOp);
goto _error;
}
pTaskInfo->streamInfo.lastStatus.uid = -1;
// code = tsdbReaderOpen(pHandle->vnode, &pTSInfo->base.cond, pList, num, pTSInfo->pResBlock,
// &pTSInfo->base.dataReader, NULL);
// if (code != 0) {
// terrno = code;
// destroyTableScanOperatorInfo(pTableScanOp);
// goto _error;
// }
}
if (pHandle->initTqReader) {

46
source/libs/executor/src/sysscanoperator.c
View File

@ -134,7 +134,7 @@ static SSDataBlock* buildInfoSchemaTableMetaBlock(char* tableName);
static void destroySysScanOperator(void* param);
static int32_t loadSysTableCallback(void* param, SDataBuf* pMsg, int32_t code);
static SSDataBlock* doFilterResult(SSDataBlock* pDataBlock, SFilterInfo* pFilterInfo);
static __optSysFilter optSysGetFilterFunc(int32_t ctype, bool* reverse);
static __optSysFilter optSysGetFilterFunc(int32_t ctype, bool* reverse, bool* equal);
static int32_t sysTableUserTagsFillOneTableTags(const SSysTableScanInfo* pInfo, SMetaReader* smrSuperTable,
SMetaReader* smrChildTable, const char* dbname, const char* tableName,
@ -164,7 +164,8 @@ int32_t sysFilte__DbName(void* arg, SNode* pNode, SArray* result) {
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
int ret = func(dbname, pVal->datum.p, TSDB_DATA_TYPE_VARCHAR);
@ -182,9 +183,9 @@ int32_t sysFilte__VgroupId(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool reverse = false;
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
int ret = func(&vgId, &pVal->datum.i, TSDB_DATA_TYPE_BIGINT);
@ -198,9 +199,9 @@ int32_t sysFilte__TableName(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool reverse = false, equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
SMetaFltParam param = {.suid = 0,
@ -208,6 +209,7 @@ int32_t sysFilte__TableName(void* arg, SNode* pNode, SArray* result) {
.type = TSDB_DATA_TYPE_VARCHAR,
.val = pVal->datum.p,
.reverse = reverse,
.equal = equal,
.filterFunc = func};
return -1;
}
@ -217,9 +219,9 @@ int32_t sysFilte__CreateTime(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool reverse = false, equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
SMetaFltParam param = {.suid = 0,
@ -227,6 +229,7 @@ int32_t sysFilte__CreateTime(void* arg, SNode* pNode, SArray* result) {
.type = TSDB_DATA_TYPE_BIGINT,
.val = &pVal->datum.i,
.reverse = reverse,
.equal = equal,
.filterFunc = func};
int32_t ret = metaFilterCreateTime(pMeta, &param, result);
@ -239,8 +242,8 @@ int32_t sysFilte__Ncolumn(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
return -1;
}
@ -251,8 +254,8 @@ int32_t sysFilte__Ttl(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
return -1;
}
@ -263,8 +266,8 @@ int32_t sysFilte__STableName(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
return -1;
}
@ -275,8 +278,8 @@ int32_t sysFilte__Uid(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
return -1;
}
@ -287,8 +290,8 @@ int32_t sysFilte__Type(void* arg, SNode* pNode, SArray* result) {
SOperatorNode* pOper = (SOperatorNode*)pNode;
SValueNode* pVal = (SValueNode*)pOper->pRight;
bool reverse = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse);
bool equal = false;
__optSysFilter func = optSysGetFilterFunc(pOper->opType, &reverse, &equal);
if (func == NULL) return -1;
return -1;
}
@ -359,10 +362,13 @@ void extractTbnameSlotId(SSysTableScanInfo* pInfo, const SScanPhy
static void sysTableScanFillTbName(SOperatorInfo* pOperator, const SSysTableScanInfo* pInfo, const char* name,
SSDataBlock* pBlock);
__optSysFilter optSysGetFilterFunc(int32_t ctype, bool* reverse) {
__optSysFilter optSysGetFilterFunc(int32_t ctype, bool* reverse, bool* equal) {
if (ctype == OP_TYPE_LOWER_EQUAL || ctype == OP_TYPE_LOWER_THAN) {
*reverse = true;
}
if (ctype == OP_TYPE_EQUAL) {
*equal = true;
}
if (ctype == OP_TYPE_LOWER_THAN)
return optSysFilterFuncImpl__LowerThan;
else if (ctype == OP_TYPE_LOWER_EQUAL)

1
source/libs/executor/src/timewindowoperator.c
View File

@ -4778,6 +4778,7 @@ static SSDataBlock* doStreamIntervalAgg(SOperatorInfo* pOperator) {
getAllIntervalWindow(pInfo->aggSup.pResultRowHashTable, pInfo->pUpdatedMap);
continue;
} else if (pBlock->info.type == STREAM_CREATE_CHILD_TABLE) {
printDataBlock(pBlock, "single interval");
return pBlock;
} else {
ASSERTS(pBlock->info.type == STREAM_NORMAL || pBlock->info.type == STREAM_INVALID, "invalid SSDataBlock type");

4
source/libs/index/src/indexCache.c
View File

@ -22,8 +22,8 @@
#define MAX_INDEX_KEY_LEN 256 // test only, change later
#define MEM_TERM_LIMIT 10 * 10000
#define MEM_THRESHOLD 8 * 512 * 1024 // 8M
#define MEM_SIGNAL_QUIT MEM_THRESHOLD * 20
#define MEM_THRESHOLD 128 * 1024 * 1024 // 8M
#define MEM_SIGNAL_QUIT MEM_THRESHOLD * 5
#define MEM_ESTIMATE_RADIO 1.5
static void idxMemRef(MemTable* tbl);

9
source/libs/index/src/indexFilter.c
View File

@ -383,12 +383,13 @@ static FORCE_INLINE int sifEqual(void *a, void *b, int16_t dtype) {
//__compar_fn_t func = idxGetCompar(dtype);
return (int)tDoCompare(func, QUERY_TERM, a, b);
}
static FORCE_INLINE FilterFunc sifGetFilterFunc(EIndexQueryType type, bool *reverse) {
static FORCE_INLINE FilterFunc sifGetFilterFunc(EIndexQueryType type, bool *reverse, bool *equal) {
if (type == QUERY_LESS_EQUAL || type == QUERY_LESS_THAN) {
*reverse = true;
} else {
*reverse = false;
}
if (type == QUERY_LESS_EQUAL)
return sifLessEqual;
else if (type == QUERY_LESS_THAN)
@ -398,6 +399,7 @@ static FORCE_INLINE FilterFunc sifGetFilterFunc(EIndexQueryType type, bool *reve
else if (type == QUERY_GREATER_THAN)
return sifGreaterThan;
else if (type == QUERY_TERM) {
*equal = true;
return sifEqual;
}
return NULL;
@ -474,14 +476,15 @@ static int32_t sifDoIndex(SIFParam *left, SIFParam *right, int8_t operType, SIFP
ret = indexJsonSearch(arg->ivtIdx, mtm, output->result);
indexMultiTermQueryDestroy(mtm);
} else {
bool reverse;
FilterFunc filterFunc = sifGetFilterFunc(qtype, &reverse);
bool reverse = false, equal = false;
FilterFunc filterFunc = sifGetFilterFunc(qtype, &reverse, &equal);
SMetaFltParam param = {.suid = arg->suid,
.cid = left->colId,
.type = left->colValType,
.val = right->condValue,
.reverse = reverse,
.equal = equal,
.filterFunc = filterFunc};
char buf[128] = {0};

16
source/libs/parser/src/parInsertUtil.c
View File

@ -290,7 +290,7 @@ int32_t insGetTableDataCxt(SHashObj* pHash, void* id, int32_t idLen, STableMeta*
}
int32_t code = createTableDataCxt(pTableMeta, pCreateTbReq, pTableCxt, colMode);
if (TSDB_CODE_SUCCESS == code) {
void* pData = *pTableCxt; // deal scan coverity
void* pData = *pTableCxt; // deal scan coverity
code = taosHashPut(pHash, id, idLen, &pData, POINTER_BYTES);
}
return code;
@ -501,14 +501,15 @@ static int32_t buildSubmitReq(int32_t vgId, SSubmitReq2* pReq, void** pData, uin
tEncodeSize(tEncodeSSubmitReq2, pReq, len, code);
if (TSDB_CODE_SUCCESS == code) {
SEncoder encoder;
len += sizeof(SMsgHead);
len += sizeof(SSubmitReq2Msg);
pBuf = taosMemoryMalloc(len);
if (NULL == pBuf) {
return TSDB_CODE_OUT_OF_MEMORY;
}
((SMsgHead*)pBuf)->vgId = htonl(vgId);
((SMsgHead*)pBuf)->contLen = htonl(len);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SMsgHead)), len - sizeof(SMsgHead));
((SSubmitReq2Msg*)pBuf)->header.vgId = htonl(vgId);
((SSubmitReq2Msg*)pBuf)->header.contLen = htonl(len);
((SSubmitReq2Msg*)pBuf)->version = htobe64(1);
tEncoderInit(&encoder, POINTER_SHIFT(pBuf, sizeof(SSubmitReq2Msg)), len - sizeof(SSubmitReq2Msg));
code = tEncodeSSubmitReq2(&encoder, pReq);
tEncoderClear(&encoder);
}
@ -679,13 +680,12 @@ int rawBlockBindData(SQuery* query, STableMeta* pTableMeta, void* data, SVCreate
pStart += BitmapLen(numOfRows);
}
char* pData = pStart;
// uError("rawBlockBindData col bytes:%d, type:%d, size:%d, htonl size:%d", pColSchema->bytes, pColSchema->type, colLength[c], htonl(colLength[c]));
tColDataAddValueByDataBlock(pCol, pColSchema->type, pColSchema->bytes, numOfRows, offset, pData);
fields += sizeof(int8_t) + sizeof(int32_t);
if(needChangeLength) {
if (needChangeLength) {
pStart += htonl(colLength[c]);
}else{
} else {
pStart += colLength[c];
}
}

62
source/libs/parser/src/parTranslater.c
View File

@ -784,7 +784,7 @@ static void setColumnInfoBySchema(const SRealTableNode* pTable, const SSchema* p
pCol->tableType = pTable->pMeta->tableType;
pCol->colId = pColSchema->colId;
pCol->colType = (tagFlag >= 0 ? COLUMN_TYPE_TAG : COLUMN_TYPE_COLUMN);
pCol->hasIndex = (0 == tagFlag);
pCol->hasIndex = ((0 == tagFlag) || (pColSchema != NULL && IS_IDX_ON(pColSchema)));
pCol->node.resType.type = pColSchema->type;
pCol->node.resType.bytes = pColSchema->bytes;
if (TSDB_DATA_TYPE_TIMESTAMP == pCol->node.resType.type) {
@ -2984,11 +2984,14 @@ static int32_t checkFill(STranslateContext* pCxt, SFillNode* pFill, SValueNode*
return TSDB_CODE_SUCCESS;
}
if (!pCxt->createStream && (TSWINDOW_IS_EQUAL(pFill->timeRange, TSWINDOW_INITIALIZER) ||
TSWINDOW_IS_EQUAL(pFill->timeRange, TSWINDOW_DESC_INITIALIZER))) {
if (!pCxt->createStream && TSWINDOW_IS_EQUAL(pFill->timeRange, TSWINDOW_INITIALIZER)) {
return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_FILL_TIME_RANGE);
}
if (TSWINDOW_IS_EQUAL(pFill->timeRange, TSWINDOW_DESC_INITIALIZER)) {
return TSDB_CODE_SUCCESS;
}
// interp FILL clause
if (NULL == pInterval) {
return TSDB_CODE_SUCCESS;
@ -5021,6 +5024,18 @@ static const SSchema* getColSchema(const STableMeta* pTableMeta, const char* pCo
return NULL;
}
static const SSchema* getNormalColSchema(const STableMeta* pTableMeta, const char* pColName) {
int32_t numOfCols = getNumOfColumns(pTableMeta);
SSchema* pColsSchema = getTableColumnSchema(pTableMeta);
for (int32_t i = 0; i < numOfCols; ++i) {
const SSchema* pSchema = pColsSchema + i;
if (0 == strcmp(pColName, pSchema->name)) {
return pSchema;
}
}
return NULL;
}
static SSchema* getTagSchema(const STableMeta* pTableMeta, const char* pTagName) {
int32_t numOfTags = getNumOfTags(pTableMeta);
SSchema* pTagsSchema = getTableTagSchema(pTableMeta);
@ -5364,6 +5379,28 @@ static int32_t buildCreateFullTextReq(STranslateContext* pCxt, SCreateIndexStmt*
return TSDB_CODE_SUCCESS;
}
static int32_t buildCreateTagIndexReq(STranslateContext* pCxt, SCreateIndexStmt* pStmt, SCreateTagIndexReq* pReq) {
SName name;
tNameExtractFullName(toName(pCxt->pParseCxt->acctId, pStmt->indexDbName, pStmt->indexName, &name), pReq->idxName);
memset(&name, 0, sizeof(SName));
tNameExtractFullName(toName(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, &name), pReq->stbName);
memset(&name, 0, sizeof(SName));
tNameSetDbName(&name, pCxt->pParseCxt->acctId, pStmt->dbName, strlen(pStmt->dbName));
tNameGetFullDbName(&name, pReq->dbFName);
SNode* pNode = NULL;
ASSERT(LIST_LENGTH(pStmt->pCols) == 1);
FOREACH(pNode, pStmt->pCols) {
SColumnNode* p = (SColumnNode*)pNode;
memcpy(pReq->colName, p->colName, sizeof(p->colName));
}
// impl later
return TSDB_CODE_SUCCESS;
}
static int32_t translateCreateFullTextIndex(STranslateContext* pCxt, SCreateIndexStmt* pStmt) {
SMCreateFullTextReq createFTReq = {0};
int32_t code = buildCreateFullTextReq(pCxt, pStmt, &createFTReq);
@ -5374,9 +5411,20 @@ static int32_t translateCreateFullTextIndex(STranslateContext* pCxt, SCreateInde
return code;
}
static int32_t translateCreateNormalIndex(STranslateContext* pCxt, SCreateIndexStmt* pStmt) {
SCreateTagIndexReq createTagIdxReq = {0};
int32_t code = buildCreateTagIndexReq(pCxt, pStmt, &createTagIdxReq);
if (TSDB_CODE_SUCCESS == code) {
code = buildCmdMsg(pCxt, TDMT_MND_CREATE_INDEX, (FSerializeFunc)tSerializeSCreateTagIdxReq, &createTagIdxReq);
}
return code;
}
static int32_t translateCreateIndex(STranslateContext* pCxt, SCreateIndexStmt* pStmt) {
if (INDEX_TYPE_FULLTEXT == pStmt->indexType) {
return translateCreateFullTextIndex(pCxt, pStmt);
} else if (INDEX_TYPE_NORMAL == pStmt->indexType) {
return translateCreateNormalIndex(pCxt, pStmt);
}
return translateCreateSmaIndex(pCxt, pStmt);
}
@ -5906,7 +5954,7 @@ static int32_t adjustOrderOfProjections(STranslateContext* pCxt, SNodeList* pCol
SNode* pCol = NULL;
SNode* pProj = NULL;
FORBOTH(pCol, pCols, pProj, *pProjections) {
const SSchema* pSchema = getColSchema(pMeta, ((SColumnNode*)pCol)->colName);
const SSchema* pSchema = getNormalColSchema(pMeta, ((SColumnNode*)pCol)->colName);
if (NULL == pSchema) {
code = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_COLUMN, ((SColumnNode*)pCol)->colName);
}
@ -6114,11 +6162,15 @@ static int32_t translateStreamTargetTable(STranslateContext* pCxt, SCreateStream
pReq->createStb = STREAM_CREATE_STABLE_TRUE;
pReq->targetStbUid = 0;
return TSDB_CODE_SUCCESS;
} else {
} else if (TSDB_CODE_SUCCESS == code) {
if (isTagDef(pStmt->pTags)) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_STREAM_QUERY, "Table already exist: %s",
pStmt->targetTabName);
}
if (TSDB_SUPER_TABLE != (*pMeta)->tableType) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_STREAM_QUERY,
"Stream can only be written to super table");
}
pReq->createStb = STREAM_CREATE_STABLE_FALSE;
pReq->targetStbUid = (*pMeta)->suid;
}

26
source/os/src/osMemory.c
View File

@ -228,6 +228,32 @@ void taosPrintBackTrace() {
void taosPrintBackTrace() { return; }
#endif
int32_t taosMemoryDbgInit() {
#if defined(LINUX) && !defined(_ALPINE)
int ret = mallopt(M_MMAP_THRESHOLD, 0);
if (0 == ret) {
return TAOS_SYSTEM_ERROR(errno);
}
return 0;
#else
return TSDB_CODE_FAILED;
#endif
}
int32_t taosMemoryDbgInitRestore() {
#if defined(LINUX) && !defined(_ALPINE)
int ret = mallopt(M_MMAP_THRESHOLD, 128 * 1024);
if (0 == ret) {
return TAOS_SYSTEM_ERROR(errno);
}
return 0;
#else
return TSDB_CODE_FAILED;
#endif
}
void *taosMemoryMalloc(int64_t size) {
#ifdef USE_TD_MEMORY
void *tmp = malloc(size + sizeof(TdMemoryInfo));

5
source/util/src/terror.c
View File

@ -208,6 +208,10 @@ TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_FUNC_BUFSIZE, "Invalid func bufSize"
TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_FUNC_COMMENT, "Invalid func comment")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_FUNC_RETRIEVE, "Invalid func retrieve msg")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_TAG_INDEX_ALREADY_EXIST, "Tag index already exists")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_TAG_INDEX_NOT_EXIST, "Tag index not exists")
// mnode-db
TAOS_DEFINE_ERROR(TSDB_CODE_MND_DB_NOT_SELECTED, "Database not specified or available")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_DB_ALREADY_EXIST, "Database already exists")
@ -291,6 +295,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_STREAM_OPTION, "Invalid stream option
TAOS_DEFINE_ERROR(TSDB_CODE_MND_STREAM_MUST_BE_DELETED, "Stream must be dropped first")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_MULTI_REPLICA_SOURCE_DB, "Stream temporarily does not support source db having replica > 1")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOO_MANY_STREAMS, "Too many streams")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_INVALID_TARGET_TABLE, "Cannot write the same stable as other stream")
// mnode-sma
TAOS_DEFINE_ERROR(TSDB_CODE_MND_SMA_ALREADY_EXIST, "SMA already exists")

2
tests/parallel_test/cases.task
View File

@ -744,7 +744,7 @@
,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/stbTagFilter-1ctb.py
,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/dataFromTsdbNWal.py
,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/dataFromTsdbNWal-multiCtb.py
#,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_taosx.py
,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_taosx.py
,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/stbTagFilter-multiCtb.py
,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-19201.py
,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-21561.py

2
tests/requirements.txt
View File

@ -7,4 +7,4 @@ toml
distro
requests
pexpect
faker
faker

15
tests/script/tsim/stream/checkStreamSTable.sim
View File

@ -261,6 +261,21 @@ if $data04 != NULL then
goto loop2
endi
print ===== drop ...
sql drop stream if exists streams0;
sql drop stream if exists streams1;
sql drop stream if exists streams2;
sql drop stream if exists streams3;
sql drop database if exists test;
sql drop database if exists test1;
sql drop database if exists test2;
sql drop database if exists test3;
sql drop database if exists result;
sql drop database if exists result1;
sql drop database if exists result2;
sql drop database if exists result3;
print ===== step6
sql create database result4 vgroups 1;

4
tests/script/tsim/stream/fillIntervalDelete0.sim
View File

@ -163,7 +163,7 @@ endi
if $data01 != 4 then
print =====data01=$data01
return -1
goto loop3
endi
@ -239,7 +239,7 @@ endi
if $data01 != 6 then
print =====data01=$data01
return -1
goto loop5
endi

8
tests/script/tsim/stream/fillIntervalDelete1.sim
View File

@ -325,19 +325,19 @@ if $rows != 12 then
endi
if $data21 != 3 then
print ====streamt10=data21=$data21
return -1
goto loop6
endi
if $data31 != 4 then
print ====streamt10=data31=$data31
return -1
goto loop6
endi
if $data71 != 8 then
print ====streamt10=data71=$data71
return -1
goto loop6
endi
if $data91 != 10 then
print ====streamt10=data91=$data91
return -1
goto loop6
endi

227
tests/script/tsim/stream/fillIntervalLinear.sim
View File

@ -9,6 +9,8 @@ system sh/exec.sh -n dnode1 -s start
sleep 200
sql connect
print step 1 start
sql drop stream if exists streams1;
sql drop database if exists test1;
sql create database test1 vgroups 1;
@ -38,161 +40,165 @@ endi
if $data01 != 2.000000000 then
print =====data01=$data01
return -1
goto loop1
endi
if $data02 != 1.000000000 then
print =====data02=$data02
return -1
goto loop1
endi
if $data03 != ccc then
print =====data03=$data03
return -1
goto loop1
endi
if $data04 != 1 then
print =====data04=$data04
return -1
goto loop1
endi
if $data11 != 4.000000000 then
print =====data11=$data11
return -1
goto loop1
endi
if $data12 != 2.000000000 then
print =====data12=$data12
return -1
goto loop1
endi
if $data13 != NULL then
print =====data13=$data13
return -1
goto loop1
endi
if $data21 != 6.000000000 then
print =====data21=$data21
return -1
goto loop1
endi
if $data22 != 3.000000000 then
print =====data22=$data22
return -1
goto loop1
endi
if $data23 != NULL then
print =====data23=$data23
return -1
goto loop1
endi
if $data31 != 8.000000000 then
print =====data31=$data31
return -1
goto loop1
endi
if $data32 != 4.000000000 then
print =====data32=$data32
return -1
goto loop1
endi
if $data33 != aaa then
print =====data33=$data33
return -1
goto loop1
endi
if $data41 != 8.666666667 then
print =====data41=$data41
return -1
goto loop1
endi
if $data42 != 4.333333333 then
print =====data42=$data42
return -1
goto loop1
endi
if $data43 != NULL then
print =====data43=$data43
return -1
goto loop1
endi
if $data51 != 9.333333333 then
print =====data01=$data01
return -1
goto loop1
endi
if $data52 != 4.666666667 then
print =====data52=$data52
return -1
goto loop1
endi
if $data53 != NULL then
print =====data53=$data53
return -1
goto loop1
endi
if $data61 != 10.000000000 then
print =====data61=$data61
return -1
goto loop1
endi
if $data62 != 5.000000000 then
print =====data62=$data62
return -1
goto loop1
endi
if $data71 != 8.000000000 then
print =====data71=$data71
return -1
goto loop1
endi
if $data72 != 4.000000000 then
print =====data72=$data72
return -1
goto loop1
endi
if $data81 != 6.000000000 then
print =====data81=$data81
return -1
goto loop1
endi
if $data82 != 3.000000000 then
print =====data82=$data82
return -1
goto loop1
endi
if $data91 != 4.000000000 then
print =====data91=$data91
return -1
goto loop1
endi
if $data92 != 2.000000000 then
print =====data92=$data92
return -1
goto loop1
endi
if $data[10][1] != 4.666666667 then
print =====data[10][1]=$data[10][1]
return -1
goto loop1
endi
if $data[10][2] != 2.333333333 then
print =====data[10][2]=$data[10][2]
return -1
goto loop1
endi
if $data[11][1] != 5.333333333 then
print =====data[11][1]=$data[11][1]
return -1
goto loop1
endi
if $data[11][2] != 2.666666667 then
print =====data[11][2]=$data[11][2]
return -1
goto loop1
endi
if $data[12][1] != 6.000000000 then
print =====data[12][1]=$data[12][1]
return -1
goto loop1
endi
if $data[12][2] != 3.000000000 then
print =====data[12][2]=$data[12][2]
return -1
goto loop1
endi
print step 1 end
print step 2 start
sql drop stream if exists streams2;
sql drop database if exists test2;
@ -224,161 +230,163 @@ endi
if $data01 != 2.000000000 then
print =====data01=$data01
return -1
goto loop2
endi
if $data02 != 1.000000000 then
print =====data02=$data02
return -1
goto loop2
endi
if $data03 != ccc then
print =====data03=$data03
return -1
goto loop2
endi
if $data04 != 1 then
print =====data04=$data04
return -1
goto loop2
endi
if $data11 != 4.000000000 then
print =====data11=$data11
return -1
goto loop2
endi
if $data12 != 2.000000000 then
print =====data12=$data12
return -1
goto loop2
endi
if $data13 != NULL then
print =====data13=$data13
return -1
goto loop2
endi
if $data21 != 6.000000000 then
print =====data21=$data21
return -1
goto loop2
endi
if $data22 != 3.000000000 then
print =====data22=$data22
return -1
goto loop2
endi
if $data23 != NULL then
print =====data23=$data23
return -1
goto loop2
endi
if $data31 != 8.000000000 then
print =====data31=$data31
return -1
goto loop2
endi
if $data32 != 4.000000000 then
print =====data32=$data32
return -1
goto loop2
endi
if $data33 != aaa then
print =====data33=$data33
return -1
goto loop2
endi
if $data41 != 8.666666667 then
print =====data41=$data41
return -1
goto loop2
endi
if $data42 != 4.333333333 then
print =====data42=$data42
return -1
goto loop2
endi
if $data43 != NULL then
print =====data43=$data43
return -1
goto loop2
endi
if $data51 != 9.333333333 then
print =====data01=$data01
return -1
goto loop2
endi
if $data52 != 4.666666667 then
print =====data52=$data52
return -1
goto loop2
endi
if $data53 != NULL then
print =====data53=$data53
return -1
goto loop2
endi
if $data61 != 10.000000000 then
print =====data61=$data61
return -1
goto loop2
endi
if $data62 != 5.000000000 then
print =====data62=$data62
return -1
goto loop2
endi
if $data71 != 8.000000000 then
print =====data71=$data71
return -1
goto loop2
endi
if $data72 != 4.000000000 then
print =====data72=$data72
return -1
goto loop2
endi
if $data81 != 6.000000000 then
print =====data81=$data81
return -1
goto loop2
endi
if $data82 != 3.000000000 then
print =====data82=$data82
return -1
goto loop2
endi
if $data91 != 4.000000000 then
print =====data91=$data91
return -1
goto loop2
endi
if $data92 != 2.000000000 then
print =====data92=$data92
return -1
goto loop2
endi
if $data[10][1] != 4.666666667 then
print =====data[10][1]=$data[10][1]
return -1
goto loop2
endi
if $data[10][2] != 2.333333333 then
print =====data[10][2]=$data[10][2]
return -1
goto loop2
endi
if $data[11][1] != 5.333333333 then
print =====data[11][1]=$data[11][1]
return -1
goto loop2
endi
if $data[11][2] != 2.666666667 then
print =====data[11][2]=$data[11][2]
return -1
goto loop2
endi
if $data[12][1] != 6.000000000 then
print =====data[12][1]=$data[12][1]
return -1
goto loop2
endi
if $data[12][2] != 3.000000000 then
print =====data[12][2]=$data[12][2]
return -1
goto loop2
endi
print step 2 end
print step 3 start
sql drop stream if exists streams3;
sql drop database if exists test3;
@ -412,98 +420,98 @@ endi
if $data01 != 3 then
print =====data01=$data01
return -1
goto loop3
endi
if $data02 != 6.000000000 then
print =====data02=$data02
return -1
goto loop3
endi
if $data03 != ccc then
print =====data03=$data03
return -1
goto loop3
endi
if $data11 != 3 then
print =====data11=$data11
return -1
goto loop3
endi
if $data12 != 7.000000000 then
print =====data12=$data12
return -1
goto loop3
endi
if $data13 != NULL then
print =====data13=$data13
return -1
goto loop3
endi
if $data21 != 4 then
print =====data21=$data21
return -1
goto loop3
endi
if $data22 != 8.000000000 then
print =====data22=$data22
return -1
goto loop3
endi
if $data23 != ddd then
print =====data23=$data23
return -1
goto loop3
endi
if $data31 != 2 then
print =====data31=$data31
return -1
goto loop3
endi
if $data32 != 5.000000000 then
print =====data32=$data32
return -1
goto loop3
endi
if $data33 != NULL then
print =====data33=$data33
return -1
goto loop3
endi
if $data41 != 1 then
print =====data41=$data41
return -1
goto loop3
endi
if $data42 != 2.000000000 then
print =====data42=$data42
return -1
goto loop3
endi
if $data43 != aaa then
print =====data43=$data43
return -1
goto loop3
endi
if $data51 != 1 then
print =====data51=$data51
return -1
goto loop3
endi
if $data52 != 3.000000000 then
print =====data52=$data52
return -1
goto loop3
endi
if $data53 != NULL then
print =====data53=$data53
return -1
goto loop3
endi
if $data61 != 2 then
print =====data61=$data61
return -1
goto loop3
endi
if $data62 != 4.000000000 then
print =====data62=$data62
return -1
goto loop3
endi
if $data63 != bbb then
print =====data63=$data63
return -1
goto loop3
endi
@ -534,42 +542,42 @@ endi
if $data01 != 6 then
print =====data01=$data01
return -1
goto loop4
endi
if $data02 != 12.000000000 then
print =====data02=$data02
return -1
goto loop4
endi
if $data03 != fff then
print =====data03=$data03
return -1
goto loop4
endi
if $data11 != 6 then
print =====data11=$data11
return -1
goto loop4
endi
if $data12 != 13.000000000 then
print =====data12=$data12
return -1
goto loop4
endi
if $data13 != NULL then
print =====data13=$data13
return -1
goto loop4
endi
if $data21 != 7 then
print =====data21=$data21
return -1
goto loop4
endi
if $data22 != 14.000000000 then
print =====data22=$data22
return -1
goto loop4
endi
if $data23 != ggg then
print =====data23=$data23
return -1
goto loop4
endi
@ -626,34 +634,11 @@ if $data[12][3] != hhh then
return -1
endi
if $data[13][1] != 8 then
print =====data[13][1]=$data[13][1]
return -1
endi
if $data[13][2] != 17.000000000 then
print =====data[13][2]=$data[13][2]
return -1
endi
if $data[13][3] != NULL then
print =====data[13][3]=$data[13][3]
return -1
endi
if $data[14][1] != 9 then
print =====data[14][1]=$data[14][1]
return -1
endi
if $data[14][2] != 18.000000000 then
print =====data[14][2]=$data[14][2]
return -1
endi
if $data[14][3] != iii then
print =====data[14][3]=$data[14][3]
return -1
endi
print step 3 end

10
tests/script/tsim/stream/fillIntervalPartitionBy.sim
View File

@ -149,7 +149,7 @@ endi
if $data04 == 0 then
print ====streamt1=data04=$data04
return -1
goto loop4
endi
sql select group_id,count(*) from streamt1 group by group_id;
@ -168,7 +168,7 @@ endi
if $data04 == 0 then
print ====streamt2=data04=$data04
return -1
goto loop4
endi
sql select group_id,count(*) from streamt2 group by group_id;
@ -187,7 +187,7 @@ endi
if $data04 == 0 then
print ====streamt3=data04=$data04
return -1
goto loop4
endi
sql select group_id,count(*) from streamt3 group by group_id;
@ -207,7 +207,7 @@ endi
if $data04 == 0 then
print ====streamt4=data04=$data04
return -1
goto loop4
endi
sql select group_id,count(*) from streamt4 group by group_id;
@ -226,7 +226,7 @@ endi
if $data04 == 0 then
print ====streamt5=data04=$data04
return -1
goto loop4
endi
sql select group_id,count(*) from streamt5 group by group_id;

526
tests/script/tsim/stream/fillIntervalPrevNext.sim
View File

@ -228,31 +228,31 @@ endi
if $data72 != 6.000000000 then
print =====data72=$data72
return -1
goto loop1
endi
if $data73 != 3.000000000 then
print =====data73=$data73
return -1
goto loop1
endi
if $data82 != 6.000000000 then
print =====data82=$data82
return -1
goto loop1
endi
if $data83 != 3.000000000 then
print =====data83=$data83
return -1
goto loop1
endi
if $data92 != 6.000000000 then
print =====data92=$data92
return -1
goto loop1
endi
if $data93 != 3.000000000 then
print =====data93=$data93
return -1
goto loop1
endi
@ -483,535 +483,33 @@ endi
if $data72 != 6.000000000 then
print =====data72=$data72
return -1
goto loop6
endi
if $data73 != 3.000000000 then
print =====data73=$data73
return -1
goto loop6
endi
if $data82 != 6.000000000 then
print =====data82=$data82
return -1
goto loop6
endi
if $data83 != 3.000000000 then
print =====data83=$data83
return -1
goto loop6
endi
if $data92 != 6.000000000 then
print =====data92=$data92
return -1
goto loop6
endi
if $data93 != 3.000000000 then
print =====data93=$data93
return -1
goto loop6
endi
sql drop stream if exists streams7;
sql drop stream if exists streams8;
sql drop database if exists test7;
sql create database test7 vgroups 1;
sql use test7;
sql create table t1(ts timestamp, a int, b int , c int, d double, s varchar(20));
sql create stream streams7 trigger at_once into streamt7 as select _wstart as ts, max(a), b+c, s from t1 where ts >= 1648791150000 and ts < 1648791261000 interval(1s) fill(prev);
sql create stream streams8 trigger at_once into streamt8 as select _wstart as ts, max(a), 1, b+1 from t1 where ts >= 1648791150000 and ts < 1648791261000 interval(1s) fill(next);
sql insert into t1 values(1648791215000,1,1,1,1.0,'aaa');
sql insert into t1 values(1648791217000,2,2,2,2.0,'bbb');
sql insert into t1 values(1648791211000,3,3,3,3.0,'ccc');
sql insert into t1 values(1648791213000,4,4,4,4.0,'ddd');
$loop_count = 0
loop7:
sleep 300
sql select * from streamt7 order by ts;
$loop_count = $loop_count + 1
if $loop_count == 10 then
return -1
endi
if $rows != 7 then
print =====rows=$rows
goto loop7
endi
if $data01 != 3 then
print =====data01=$data01
return -1
endi
if $data02 != 6.000000000 then
print =====data02=$data02
return -1
endi
if $data03 != ccc then
print =====data03=$data03
return -1
endi
if $data11 != 3 then
print =====data11=$data11
return -1
endi
if $data12 != 6.000000000 then
print =====data12=$data12
return -1
endi
if $data13 != ccc then
print =====data13=$data13
return -1
endi
if $data21 != 4 then
print =====data21=$data21
return -1
endi
if $data22 != 8.000000000 then
print =====data22=$data22
return -1
endi
if $data23 != ddd then
print =====data23=$data23
return -1
endi
if $data31 != 4 then
print =====data31=$data31
return -1
endi
if $data32 != 8.000000000 then
print =====data32=$data32
return -1
endi
if $data33 != ddd then
print =====data33=$data33
return -1
endi
if $data41 != 1 then
print =====data41=$data41
return -1
endi
if $data42 != 2.000000000 then
print =====data42=$data42
return -1
endi
if $data43 != aaa then
print =====data43=$data43
return -1
endi
if $data51 != 1 then
print =====data51=$data51
return -1
endi
if $data52 != 2.000000000 then
print =====data52=$data52
return -1
endi
if $data53 != aaa then
print =====data53=$data53
return -1
endi
if $data61 != 2 then
print =====data61=$data61
return -1
endi
if $data62 != 4.000000000 then
print =====data62=$data62
return -1
endi
if $data63 != bbb then
print =====data63=$data63
return -1
endi
#--------------
sleep 200
sql select * from streamt8 order by ts;
if $rows != 7 then
print =====rows=$rows
return -1
endi
if $data01 != 3 then
print =====data01=$data01
return -1
endi
if $data02 != 1 then
print =====data02=$data02
return -1
endi
if $data03 != 4.000000000 then
print =====data03=$data03
return -1
endi
if $data11 != 4 then
print =====data11=$data11
return -1
endi
if $data12 != 1 then
print =====data12=$data12
return -1
endi
if $data13 != 5.000000000 then
print =====data13=$data13
return -1
endi
if $data21 != 4 then
print =====data21=$data21
return -1
endi
if $data22 != 1 then
print =====data22=$data22
return -1
endi
if $data23 != 5.000000000 then
print =====data23=$data23
return -1
endi
if $data31 != 1 then
print =====data31=$data31
return -1
endi
if $data32 != 1 then
print =====data32=$data32
return -1
endi
if $data33 != 2.000000000 then
print =====data33=$data33
return -1
endi
if $data41 != 1 then
print =====data41=$data41
return -1
endi
if $data42 != 1 then
print =====data42=$data42
return -1
endi
if $data43 != 2.000000000 then
print =====data43=$data43
return -1
endi
if $data51 != 2 then
print =====data51=$data51
return -1
endi
if $data52 != 1 then
print =====data52=$data52
return -1
endi
if $data53 != 3.000000000 then
print =====data53=$data53
return -1
endi
if $data61 != 2 then
print =====data61=$data61
return -1
endi
if $data62 != 1 then
print =====data62=$data62
return -1
endi
if $data63 != 3.000000000 then
print =====data63=$data63
return -1
endi
sql insert into t1 values(1648791212000,5,5,5,5.0,'eee');
sql insert into t1 values(1648791207000,6,6,6,6.0,'fff') (1648791209000,7,7,7,7.0,'ggg') (1648791219000,8,8,8,8.0,'hhh') (1648791221000,9,9,9,9.0,'iii');
$loop_count = 0
loop8:
sleep 200
sql select * from streamt7 order by ts;
$loop_count = $loop_count + 1
if $loop_count == 10 then
return -1
endi
if $rows != 15 then
print =====rows=$rows
goto loop8
endi
if $data01 != 6 then
print =====data01=$data01
return -1
endi
if $data02 != 12.000000000 then
print =====data02=$data02
return -1
endi
if $data03 != fff then
print =====data03=$data03
return -1
endi
if $data11 != 6 then
print =====data11=$data11
return -1
endi
if $data12 != 12.000000000 then
print =====data12=$data12
return -1
endi
if $data13 != fff then
print =====data13=$data13
return -1
endi
if $data21 != 7 then
print =====data21=$data21
return -1
endi
if $data22 != 14.000000000 then
print =====data22=$data22
return -1
endi
if $data23 != ggg then
print =====data23=$data23
return -1
endi
if $data31 != 7 then
print =====data31=$data31
return -1
endi
if $data32 != 14.000000000 then
print =====data32=$data32
return -1
endi
if $data33 != ggg then
print =====data33=$data33
return -1
endi
if $data51 != 5 then
print =====data51=$data51
return -1
endi
if $data52 != 10.000000000 then
print =====data52=$data52
return -1
endi
if $data53 != eee then
print =====data53=$data53
return -1
endi
if $data[11][1] != 2 then
print =====data[11][1]=$data[11][1]
return -1
endi
if $data[11][2] != 4.000000000 then
print =====data[11][2]=$data[11][2]
return -1
endi
if $data[11][3] != bbb then
print =====data[11][3]=$data[11][3]
return -1
endi
if $data[12][1] != 8 then
print =====data[12][1]=$data[12][1]
return -1
endi
if $data[12][2] != 16.000000000 then
print =====data[12][2]=$data[12][2]
return -1
endi
if $data[12][3] != hhh then
print =====data[12][3]=$data[12][3]
return -1
endi
if $data[13][1] != 8 then
print =====data[13][1]=$data[13][1]
return -1
endi
if $data[13][2] != 16.000000000 then
print =====data[13][2]=$data[13][2]
return -1
endi
if $data[13][3] != hhh then
print =====data[13][3]=$data[13][3]
return -1
endi
if $data[14][1] != 9 then
print =====data[14][1]=$data[14][1]
return -1
endi
if $data[14][2] != 18.000000000 then
print =====data[14][2]=$data[14][2]
return -1
endi
if $data[14][3] != iii then
print =====data[14][3]=$data[14][3]
return -1
endi
print fill next-----------------890
sql use test7;
sql select * from streamt8 order by ts;
if $rows != 15 then
print =====rows=$rows
goto loop8
endi
if $data01 != 6 then
print =====data01=$data01
return -1
endi
if $data02 != 1 then
print =====data02=$data02
return -1
endi
if $data03 != 7.000000000 then
print =====data03=$data03
return -1
endi
if $data11 != 7 then
print =====data11=$data11
return -1
endi
if $data13 != 8.000000000 then
print =====data13=$data13
return -1
endi
if $data21 != 7 then
print =====data21=$data21
return -1
endi
if $data23 != 8.000000000 then
print =====data23=$data23
return -1
endi
if $data31 != 3 then
print =====data31=$data31
return -1
endi
if $data33 != 4.000000000 then
print =====data33=$data33
return -1
endi
if $data51 != 5 then
print =====data51=$data51
return -1
endi
if $data53 != 6.000000000 then
print =====data53=$data53
return -1
endi
if $data[11][1] != 8 then
print =====data[11][1]=$data[11][1]
return -1
endi
if $data[11][2] != 1 then
print =====data[11][2]=$data[11][2]
return -1
endi
if $data[11][3] != 9.000000000 then
print =====data[11][3]=$data[11][3]
return -1
endi
if $data[12][1] != 8 then
print =====data[12][1]=$data[12][1]
return -1
endi
if $data[12][3] != 9.000000000 then
print =====data[12][3]=$data[12][3]
return -1
endi
if $data[13][1] != 9 then
print =====data[13][1]=$data[13][1]
return -1
endi
if $data[13][3] != 10.000000000 then
print =====data[13][3]=$data[13][3]
return -1
endi
if $data[14][1] != 9 then
print =====data[14][1]=$data[14][1]
return -1
endi
if $data[14][3] != 10.000000000 then
print =====data[14][3]=$data[14][3]
return -1
endi
#==system sh/exec.sh -n dnode1 -s stop -x SIGINT

559
tests/script/tsim/stream/fillIntervalPrevNext1.sim Normal file
View File

@ -0,0 +1,559 @@
$loop_all = 0
looptest:
system sh/stop_dnodes.sh
system sh/deploy.sh -n dnode1 -i 1
system sh/exec.sh -n dnode1 -s start
#==system sh/exec.sh -n dnode1 -s start -v
sleep 200
sql connect
sql drop stream if exists streams7;
sql drop stream if exists streams8;
sql drop database if exists test7;
sql create database test7 vgroups 1;
sql use test7;
sql create table t1(ts timestamp, a int, b int , c int, d double, s varchar(20));
sql create stream streams7 trigger at_once into streamt7 as select _wstart as ts, max(a), b+c, s from t1 where ts >= 1648791150000 and ts < 1648791261000 interval(1s) fill(prev);
sql create stream streams8 trigger at_once into streamt8 as select _wstart as ts, max(a), 1, b+1 from t1 where ts >= 1648791150000 and ts < 1648791261000 interval(1s) fill(next);
sql insert into t1 values(1648791215000,1,1,1,1.0,'aaa');
sql insert into t1 values(1648791217000,2,2,2,2.0,'bbb');
sql insert into t1 values(1648791211000,3,3,3,3.0,'ccc');
sql insert into t1 values(1648791213000,4,4,4,4.0,'ddd');
$loop_count = 0
loop7:
sleep 300
sql select * from streamt7 order by ts;
$loop_count = $loop_count + 1
if $loop_count == 10 then
return -1
endi
if $rows != 7 then
print =====rows=$rows
goto loop7
endi
if $data01 != 3 then
print =====data01=$data01
goto loop7
endi
if $data02 != 6.000000000 then
print =====data02=$data02
goto loop7
endi
if $data03 != ccc then
print =====data03=$data03
goto loop7
endi
if $data11 != 3 then
print =====data11=$data11
goto loop7
endi
if $data12 != 6.000000000 then
print =====data12=$data12
goto loop7
endi
if $data13 != ccc then
print =====data13=$data13
goto loop7
endi
if $data21 != 4 then
print =====data21=$data21
goto loop7
endi
if $data22 != 8.000000000 then
print =====data22=$data22
goto loop7
endi
if $data23 != ddd then
print =====data23=$data23
goto loop7
endi
if $data31 != 4 then
print =====data31=$data31
goto loop7
endi
if $data32 != 8.000000000 then
print =====data32=$data32
goto loop7
endi
if $data33 != ddd then
print =====data33=$data33
goto loop7
endi
if $data41 != 1 then
print =====data41=$data41
goto loop7
endi
if $data42 != 2.000000000 then
print =====data42=$data42
goto loop7
endi
if $data43 != aaa then
print =====data43=$data43
goto loop7
endi
if $data51 != 1 then
print =====data51=$data51
goto loop7
endi
if $data52 != 2.000000000 then
print =====data52=$data52
goto loop7
endi
if $data53 != aaa then
print =====data53=$data53
goto loop7
endi
if $data61 != 2 then
print =====data61=$data61
goto loop7
endi
if $data62 != 4.000000000 then
print =====data62=$data62
goto loop7
endi
if $data63 != bbb then
print =====data63=$data63
goto loop7
endi
#--------------
loop71:
$loop_count = 0
sleep 300
sql select * from streamt8 order by ts;
$loop_count = $loop_count + 1
if $loop_count == 10 then
return -1
endi
if $rows != 7 then
print =====rows=$rows
goto loop71
endi
if $data01 != 3 then
print =====data01=$data01
goto loop71
endi
if $data02 != 1 then
print =====data02=$data02
goto loop71
endi
if $data03 != 4.000000000 then
print =====data03=$data03
goto loop71
endi
if $data11 != 4 then
print =====data11=$data11
goto loop71
endi
if $data12 != 1 then
print =====data12=$data12
goto loop71
endi
if $data13 != 5.000000000 then
print =====data13=$data13
goto loop71
endi
if $data21 != 4 then
print =====data21=$data21
goto loop71
endi
if $data22 != 1 then
print =====data22=$data22
goto loop71
endi
if $data23 != 5.000000000 then
print =====data23=$data23
goto loop71
endi
if $data31 != 1 then
print =====data31=$data31
goto loop71
endi
if $data32 != 1 then
print =====data32=$data32
goto loop71
endi
if $data33 != 2.000000000 then
print =====data33=$data33
goto loop71
endi
if $data41 != 1 then
print =====data41=$data41
goto loop71
endi
if $data42 != 1 then
print =====data42=$data42
goto loop71
endi
if $data43 != 2.000000000 then
print =====data43=$data43
goto loop71
endi
if $data51 != 2 then
print =====data51=$data51
goto loop71
endi
if $data52 != 1 then
print =====data52=$data52
goto loop71
endi
if $data53 != 3.000000000 then
print =====data53=$data53
goto loop71
endi
if $data61 != 2 then
print =====data61=$data61
goto loop71
endi
if $data62 != 1 then
print =====data62=$data62
goto loop71
endi
if $data63 != 3.000000000 then
print =====data63=$data63
goto loop71
endi
sql insert into t1 values(1648791212000,5,5,5,5.0,'eee');
sql insert into t1 values(1648791207000,6,6,6,6.0,'fff') (1648791209000,7,7,7,7.0,'ggg') (1648791219000,8,8,8,8.0,'hhh') (1648791221000,9,9,9,9.0,'iii');
$loop_count = 0
loop8:
sleep 200
sql select * from streamt7 order by ts;
$loop_count = $loop_count + 1
if $loop_count == 10 then
return -1
endi
if $rows != 15 then
print =====rows=$rows
goto loop8
endi
if $data01 != 6 then
print =====data01=$data01
goto loop8
endi
if $data02 != 12.000000000 then
print =====data02=$data02
goto loop8
endi
if $data03 != fff then
print =====data03=$data03
goto loop8
endi
if $data11 != 6 then
print =====data11=$data11
goto loop8
endi
if $data12 != 12.000000000 then
print =====data12=$data12
goto loop8
endi
if $data13 != fff then
print =====data13=$data13
goto loop8
endi
if $data21 != 7 then
print =====data21=$data21
goto loop8
endi
if $data22 != 14.000000000 then
print =====data22=$data22
goto loop8
endi
if $data23 != ggg then
print =====data23=$data23
goto loop8
endi
if $data31 != 7 then
print =====data31=$data31
goto loop8
endi
if $data32 != 14.000000000 then
print =====data32=$data32
goto loop8
endi
if $data33 != ggg then
print =====data33=$data33
goto loop8
endi
if $data51 != 5 then
print =====data51=$data51
goto loop8
endi
if $data52 != 10.000000000 then
print =====data52=$data52
goto loop8
endi
if $data53 != eee then
print =====data53=$data53
goto loop8
endi
if $data[11][1] != 2 then
print =====data[11][1]=$data[11][1]
goto loop8
endi
if $data[11][2] != 4.000000000 then
print =====data[11][2]=$data[11][2]
goto loop8
endi
if $data[11][3] != bbb then
print =====data[11][3]=$data[11][3]
goto loop8
endi
if $data[12][1] != 8 then
print =====data[12][1]=$data[12][1]
goto loop8
endi
if $data[12][2] != 16.000000000 then
print =====data[12][2]=$data[12][2]
goto loop8
endi
if $data[12][3] != hhh then
print =====data[12][3]=$data[12][3]
goto loop8
endi
if $data[13][1] != 8 then
print =====data[13][1]=$data[13][1]
goto loop8
endi
if $data[13][2] != 16.000000000 then
print =====data[13][2]=$data[13][2]
goto loop8
endi
if $data[13][3] != hhh then
print =====data[13][3]=$data[13][3]
goto loop8
endi
if $data[14][1] != 9 then
print =====data[14][1]=$data[14][1]
goto loop8
endi
if $data[14][2] != 18.000000000 then
print =====data[14][2]=$data[14][2]
goto loop8
endi
if $data[14][3] != iii then
print =====data[14][3]=$data[14][3]
goto loop8
endi
print fill next-----------------890
sql use test7;
sql select * from streamt8 order by ts;
if $rows != 15 then
print =====rows=$rows
goto loop8
endi
if $data01 != 6 then
print =====data01=$data01
goto loop8
endi
if $data02 != 1 then
print =====data02=$data02
goto loop8
endi
if $data03 != 7.000000000 then
print =====data03=$data03
goto loop8
endi
if $data11 != 7 then
print =====data11=$data11
goto loop8
endi
if $data13 != 8.000000000 then
print =====data13=$data13
goto loop8
endi
if $data21 != 7 then
print =====data21=$data21
goto loop8
endi
if $data23 != 8.000000000 then
print =====data23=$data23
goto loop8
endi
if $data31 != 3 then
print =====data31=$data31
goto loop8
endi
if $data33 != 4.000000000 then
print =====data33=$data33
goto loop8
endi
if $data51 != 5 then
print =====data51=$data51
goto loop8
endi
if $data53 != 6.000000000 then
print =====data53=$data53
goto loop8
endi
if $data[11][1] != 8 then
print =====data[11][1]=$data[11][1]
goto loop8
endi
if $data[11][2] != 1 then
print =====data[11][2]=$data[11][2]
goto loop8
endi
if $data[11][3] != 9.000000000 then
print =====data[11][3]=$data[11][3]
goto loop8
endi
if $data[12][1] != 8 then
print =====data[12][1]=$data[12][1]
goto loop8
endi
if $data[12][3] != 9.000000000 then
print =====data[12][3]=$data[12][3]
goto loop8
endi
if $data[13][1] != 9 then
print =====data[13][1]=$data[13][1]
goto loop8
endi
if $data[13][3] != 10.000000000 then
print =====data[13][3]=$data[13][3]
goto loop8
endi
if $data[14][1] != 9 then
print =====data[14][1]=$data[14][1]
goto loop8
endi
if $data[14][3] != 10.000000000 then
print =====data[14][3]=$data[14][3]
goto loop8
endi
#==system sh/exec.sh -n dnode1 -s stop -x SIGINT
#==print =============== check
#==$null=
#==system_content sh/checkValgrind.sh -n dnode1
#==print cmd return result ----> [ $system_content ]
#==if $system_content > 0 then
#== return -1
#==endi
#==if $system_content == $null then
#== return -1
#==endi
#==return 1
sql drop stream if exists streams0;
sql drop stream if exists streams1;
sql drop stream if exists streams2;
sql drop stream if exists streams3;
sql drop stream if exists streams4;
sql drop stream if exists streams5;
sql drop stream if exists streams6;
sql drop stream if exists streams7;
sql drop stream if exists streams8;
sql use test7;
sql select * from t1;
print $data00
$loop_all = $loop_all + 1
print ============loop_all=$loop_all
system sh/stop_dnodes.sh
#goto looptest

4
tests/script/tsim/stream/ignoreExpiredData.sim
View File

@ -116,7 +116,7 @@ sql create stream stream_t1 trigger at_once IGNORE EXPIRED 1 into streamtST1 as
sql create stream stream_t2 trigger at_once IGNORE EXPIRED 1 into streamtST2 as select _wstart, count(*) c1, count(a) c2 , sum(a) c3 , max(b) c5, min(c) c6 from st session(ts, 10s) ;
sql insert into ts1 values(1648791211000,1,2,3);
sql insert into ts1 values(1648791222001,2,2,3);
sleep 300
sleep 200
sql insert into ts2 values(1648791211000,1,2,3);
sql insert into ts2 values(1648791222001,2,2,3);
@ -160,4 +160,4 @@ if $data02 != 1 then
goto loop5
endi
system sh/stop_dnodes.sh
system sh/stop_dnodes.sh

224
tests/script/tsim/tagindex/add_index.sim Normal file
View File

@ -0,0 +1,224 @@
system sh/stop_dnodes.sh
system sh/deploy.sh -n dnode1 -i 1
system sh/exec.sh -n dnode1 -s start
sql connect
print ======== step0
$dbPrefix = ta_3_db
$tbPrefix = ta_3_tb
$mtPrefix = ta_3_mt
$tbNum = 100
$rowNum = 20
$totalNum = 200
print =============== create database
sql create database $dbPrefix
sql use $dbPrefix
print =============== create super table and register tag index
sql create table if not exists $mtPrefix (ts timestamp, c1 int) tags (t1 int, t2 int, t3 int, t4 int, t5 int)
sql show stables
if $rows != 1 then
return -1
endi
print =============== create child table
$i = 0
while $i < $tbNum
$tb = $tbPrefix . $i
sql create table $tb using $mtPrefix tags( $i , $i , $i , $i , $i );
$i = $i + 1
endw
sql show tables
if $rows != $tbNum then
return -1
endi
print =============== insert data into each table
$i = 0
while $i < $tbNum
$tb = $tbPrefix . $i
sql insert into $tb values(now, $i );
$i = $i + 1
endw
sql create index ti2 on $mtPrefix (t2)
print ==== test name conflict
#
sql_error create index ti3 on $mtPrefix(t2)
sql_error create index ti2 on $mtPrefix(t2)
sql_error create index ti2 on $mtPrefix(t1)
sql_error create index ti2 on $mtPrefix(t3)
sql_error create index ti2 on $mtPrefix(txx)
print ===== test operator equal
$i = 0
while $i < $tbNum
sql select * from $mtPrefix where t2= $i ;
if $rows != 1 then
return -1
endi
$i = $i + 1
endw
print ===== test operator great equal
# great equal than
$i = 0
while $i < $tbNum
sql select * from $mtPrefix where t2 >= $i ;
$tmp = $tbNum - $i
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== test operator great
$i = 0
while $i < $tbNum
sql select * from $mtPrefix where t2 > $i ;
$tmp = $tbNum - $i
$tmp = $tmp - 1
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== test operator lower
$i = 0
while $i < $tbNum
sql select * from $mtPrefix where t2 <= $i ;
$tmp = $i + 1
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== test operator lower than
# lower equal than
$i = 0
while $i < $tbNum
sql select * from $mtPrefix where t2 < $i ;
$tmp = $i
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== add table after index created
$interval = $tbNum + $tbNum
$i = $interval
$limit = $interval + $tbNum
while $i < $limit
$tb = $tbPrefix . $i
sql insert into $tb using $mtPrefix tags( $i , $i , $i , $i , $i ) values( now, $i )
$i = $i + 1
endw
print ===== add table after index created (opeator great equal than)
# great equal than
$i = $interval
while $i < $limit
sql select * from $mtPrefix where t2 >= $i ;
$tmp = $limit - $i
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== add table after index created (opeator great than)
# great than
$i = $interval
while $i < $limit
sql select * from $mtPrefix where t2 > $i ;
$tmp = $limit - $i
$tmp = $tmp - 1
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== drop table after index created
# drop table
$dropTbNum = $tbNum / 2
$i = 0
while $i < $tbNum
$tb = $tbPrefix . $i
sql drop table $tb
$i = $i + 1
endw
print ===== drop table after index created(opeator lower than)
# lower equal than
$i = $interval
while $i < $limit
sql select * from $mtPrefix where t2 < $i ;
$tmp = $i - $interval
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print ===== drop table after index created(opeator lower equal than)
# lower equal than
$i = $interval
while $i < $limit
sql select * from $mtPrefix where t2 <= $i ;
$tmp = $i - $interval
$tmp = $tmp + 1
if $rows != $tmp then
return -1
endi
$i = $i + 1
endw
print === show index
sql select * from information_schema.ins_indexes
if $rows != 1 then
return -1
endi
print === drop index ti2
sql drop index ti2
print === drop not exist index
sql_error drop index t2
sql_error drop index t3
sql_error create index ti0 on $mtPrefix (t1)
sql_error create index t2i on ta_3_tb17 (t2)
system sh/exec.sh -n dnode1 -s stop -x SIGINT

0
tests/script/tsim/tagindex/drop_index.sim Normal file
View File

Some files were not shown because too many files have changed in this diff Show More