xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch '3.0' into feat/agg_client_api

This commit is contained in:
Ganlin Zhao 2022-07-18 13:42:58 +08:00
parent e37bdbd5c9 05e5beb8f3
commit 7b5a96fa80
71 changed files with 1472 additions and 837 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
Jenkinsfile2
View File

@ -333,6 +333,11 @@ pipeline {
}
stages {
stage('run test') {
when {
allOf {
not { expression { env.CHANGE_BRANCH =~ /docs\// }}
}
}
parallel {
stage('windows test') {
agent{label " windows10_01 || windows10_02 || windows10_03 || windows10_04 "}

34
docs/en/12-taos-sql/14-limit.md
View File

@ -1,5 +1,5 @@
---
title: Limits & Restrictions
title: Naming & Restrictions
---
## Naming Rules
@ -16,41 +16,21 @@ The legal character set is `[a-zA-Z0-9!?$%^&*()_–+={[}]:;@~#|<,>.?/]`.
## General Limits
- Maximum length of database name is 32 bytes.
- Maximum length of database name is 32 bytes, and it can't include "." or special characters.
- Maximum length of table name is 192 bytes, excluding the database name prefix and the separator.
- Maximum length of each data row is 48K bytes since version 2.1.7.0 , before which the limit was 16K bytes. Please note that the upper limit includes the extra 2 bytes consumed by each column of BINARY/NCHAR type.
- Maximum length of each data row is 48K bytes. Please note that the upper limit includes the extra 2 bytes consumed by each column of BINARY/NCHAR type.
- Maximum length of column name is 64.
- Maximum number of columns is 4096. There must be at least 2 columns, and the first column must be timestamp.
- Maximum length of tag name is 64.
- Maximum number of tags is 128. There must be at least 1 tag. The total length of tag values should not exceed 16K bytes.
- Maximum length of singe SQL statement is 1048576, i.e. 1 MB. It can be configured in the parameter `maxSQLLength` in the client side, the applicable range is [65480, 1048576].
- At most 4096 columns (or 1024 prior to 2.1.7.0) can be returned by `SELECT`. Functions in the query statement constitute columns. An error is returned if the limit is exceeded.
- At most 4096 columns can be returned by `SELECT`. Functions in the query statement constitute columns. An error is returned if the limit is exceeded.
- Maximum numbers of databases, STables, tables are dependent only on the system resources.
- Maximum of database name is 32 bytes, and it can't include "." or special characters.
- Maximum number of replicas for a database is 3.
- Maximum length of user name is 23 bytes.
- Maximum length of password is 15 bytes.
- Maximum number of rows depends only on the storage space.
- Maximum number of tables depends only on the number of nodes.
- Maximum number of databases depends only on the number of nodes.
- Maximum number of vnodes for a single database is 64.
## Restrictions of `GROUP BY`
`GROUP BY` can be performed on tags and `TBNAME`. It can be performed on data columns too, with the only restriction being it can only be performed on one data column and the number of unique values in that column is lower than 100,000. Please note that `GROUP BY` cannot be performed on float or double types.
## Restrictions of `IS NOT NULL`
`IS NOT NULL` can be used on any data type of columns. The non-empty string evaluation expression, i.e. `< > ""` can only be used on non-numeric data types.
## Restrictions of `ORDER BY`
- Only one `order by` is allowed for normal table and subtable.
- At most two `order by` are allowed for STable, and the second one must be `ts`.
- `order by tag` must be used with `group by tag` on same tag. This rule is also applicable to `tbname`.
- `order by column` must be used with `group by column` or `top/bottom` on same column. This rule is applicable to table and STable.
- `order by ts` is applicable to table and STable.
- If `order by ts` is used with `group by`, the result set is sorted using `ts` in each group.
- Maximum number of vnodes for a single database is 1024.
## Restrictions of Table/Column Names
@ -71,7 +51,3 @@ For example:
The characters inside escape characters must be printable characters.
:::
### Applicable Versions
Escape character "\`" is available from version 2.3.0.1.

131
docs/zh/12-taos-sql/12-interval.md
View File

@ -1,13 +1,80 @@
---
sidebar_label: 按窗口切分聚合
title: 按窗口切分聚合
sidebar_label: 时序数据特色查询
title: 时序数据特色查询
---
TDengine是专为时序数据而研发的大数据平台，存储和计算都针对时序数据的特定进行了量身定制，在支持标准SQL的基础之上，还提供了一系列贴合时序业务场景的特色查询语法，极大的方便时序场景的应用开发。
TDengine 支持按时间段窗口切分方式进行聚合结果查询，比如温度传感器每秒采集一次数据，但需查询每隔 10 分钟的温度平均值。这种场景下可以使用窗口子句来获得需要的查询结果。
窗口子句用于针对查询的数据集合进行按照窗口切分成为查询子集并进行聚合，窗口包含时间窗口（time window）、状态窗口（status window）、会话窗口（session window）三种窗口。其中时间窗口又可划分为滑动时间窗口和翻转时间窗口。
TDengine提供的特色查询包括标签切分查询和窗口切分查询。
## 时间窗口
## 标签切分查询
超级表查询中，当需要针对标签进行数据切分然后在切分出的数据空间内再进行一系列的计算时使用标签切分子句，标签切分的语句如下：
```sql
PARTITION BY tag_list
```
其中 `tag_list` 是标签列的列表，还可以包括tbname伪列。
TDengine按如下方式处理标签切分子句：
标签切分子句位于 `WHERE` 子句之后，且不能和 `JOIN` 子句一起使用。
标签切分子句将超级表数据按指定的标签组合进行切分，然后对每个切分的分片进行指定的计算。计算由之后的子句定义（窗口子句、`GROUP BY` 子句或`SELECT` 子句）。
标签切分子句可以和窗口切分子句（或 `GROUP BY` 子句）一起使用，此时后面的子句作用在每个切分的分片上。例如，下面的示例将数据按标签 `location` 进行分组，并对每个组按10分钟进行降采样，取其最大值。
```sql
select max(current) from meters partition by location interval(10m)
```
## 窗口切分查询
TDengine 支持按时间段窗口切分方式进行聚合结果查询，比如温度传感器每秒采集一次数据，但需查询每隔 10 分钟的温度平均值。这种场景下可以使用窗口子句来获得需要的查询结果。窗口子句用于针对查询的数据集合按照窗口切分成为查询子集并进行聚合，窗口包含时间窗口（time window）、状态窗口（status window）、会话窗口（session window）三种窗口。其中时间窗口又可划分为滑动时间窗口和翻转时间窗口。窗口切分查询语法如下：
```sql
SELECT function_list FROM tb_name
[WHERE where_condition]
[SESSION(ts_col, tol_val)]
[STATE_WINDOW(col)]
[INTERVAL(interval [, offset]) [SLIDING sliding]]
[FILL({NONE | VALUE | PREV | NULL | LINEAR | NEXT})]
```
在上述语法中的具体限制如下
### 窗口切分查询中使用函数的限制
- 在聚合查询中，function_list 位置允许使用聚合和选择函数，并要求每个函数仅输出单个结果（例如：COUNT、AVG、SUM、STDDEV、LEASTSQUARES、PERCENTILE、MIN、MAX、FIRST、LAST），而不能使用具有多行输出结果的函数（例如：DIFF 以及四则运算）。
- 此外 LAST_ROW 查询也不能与窗口聚合同时出现。
- 标量函数（如：CEIL/FLOOR 等）也不能使用在窗口聚合查询中。
### 窗口子句的规则
- 窗口子句位于标签切分子句之后，GROUP BY子句之前，且不可以和GROUP BY子句一起使用。
- 窗口子句将数据按窗口进行切分，对每个窗口进行SELECT列表中的表达式的计算，SELECT列表中的表达式只能包含：
- 常量。
- 聚集函数。
- 包含上面表达式的表达式。
- 窗口子句不可以和GROUP BY子句一起使用。
- WHERE 语句可以指定查询的起止时间和其他过滤条件。
### FILL 子句
FILL 语句指定某一窗口区间数据缺失的情况下的填充模式。填充模式包括以下几种：
1. 不进行填充：NONE（默认填充模式）。
2. VALUE 填充：固定值填充，此时需要指定填充的数值。例如：FILL(VALUE, 1.23)。这里需要注意，最终填充的值受由相应列的类型决定，如FILL(VALUE, 1.23)，相应列为INT类型，则填充值为1。
3. PREV 填充：使用前一个非 NULL 值填充数据。例如：FILL(PREV)。
4. NULL 填充：使用 NULL 填充数据。例如：FILL(NULL)。
5. LINEAR 填充：根据前后距离最近的非 NULL 值做线性插值填充。例如：FILL(LINEAR)。
6. NEXT 填充：使用下一个非 NULL 值填充数据。例如：FILL(NEXT)。
:::info
1. 使用 FILL 语句的时候可能生成大量的填充输出，务必指定查询的时间区间。针对每次查询，系统可返回不超过 1 千万条具有插值的结果。
2. 在时间维度聚合中，返回的结果中时间序列严格单调递增。
3. 如果查询对象是超级表，则聚合函数会作用于该超级表下满足值过滤条件的所有表的数据。如果查询中没有使用 GROUP BY 语句，则返回的结果按照时间序列严格单调递增；如果查询中使用了 GROUP BY 语句分组，则返回结果中每个 GROUP 内不按照时间序列严格单调递增。
:::
### 时间窗口
时间窗口又可分为滑动时间窗口和翻转时间窗口。
INTERVAL 子句用于产生相等时间周期的窗口，SLIDING 用以指定窗口向前滑动的时间。每次执行的查询是一个时间窗口，时间窗口随着时间流动向前滑动。在定义连续查询的时候需要指定时间窗口（time window ）大小和每次前向增量时间（forward sliding times）。如图，[t0s, t0e] ，[t1s , t1e]， [t2s, t2e] 是分别是执行三次连续查询的时间窗口范围，窗口的前向滑动的时间范围 sliding time 标识 。查询过滤、聚合等操作按照每个时间窗口为独立的单位执行。当 SLIDING 与 INTERVAL 相等的时候，滑动窗口即为翻转窗口。
@ -25,11 +92,12 @@ SLIDING 的向前滑动的时间不能超过一个窗口的时间范围。以下
SELECT COUNT(*) FROM temp_tb_1 INTERVAL(1m) SLIDING(2m);
```
当 SLIDING 与 INTERVAL 取值相等的时候，滑动窗口即为翻转窗口。
_ 聚合时间段的窗口宽度由关键词 INTERVAL 指定，最短时间间隔 10 毫秒（10a）；并且支持偏移 offset（偏移必须小于间隔），也即时间窗口划分与“UTC 时刻 0”相比的偏移量。SLIDING 语句用于指定聚合时间段的前向增量，也即每次窗口向前滑动的时长。
_ 从 2.1.5.0 版本开始，INTERVAL 语句允许的最短时间间隔调整为 1 微秒（1u），当然如果所查询的 DATABASE 的时间精度设置为毫秒级，那么允许的最短时间间隔为 1 毫秒（1a）。 \* **注意**：用到 INTERVAL 语句时，除非极特殊的情况，都要求把客户端和服务端的 taos.cfg 配置文件中的 timezone 参数配置为相同的取值，以避免时间处理函数频繁进行跨时区转换而导致的严重性能影响。
使用时间窗口需要注意：
- 聚合时间段的窗口宽度由关键词 INTERVAL 指定，最短时间间隔 10 毫秒（10a）；并且支持偏移 offset（偏移必须小于间隔），也即时间窗口划分与“UTC 时刻 0”相比的偏移量。SLIDING 语句用于指定聚合时间段的前向增量，也即每次窗口向前滑动的时长。
- 使用 INTERVAL 语句时，除非极特殊的情况，都要求把客户端和服务端的 taos.cfg 配置文件中的 timezone 参数配置为相同的取值，以避免时间处理函数频繁进行跨时区转换而导致的严重性能影响。
- 返回的结果中时间序列严格单调递增。
## 状态窗口
### 状态窗口
使用整数（布尔值）或字符串来标识产生记录时候设备的状态量。产生的记录如果具有相同的状态量数值则归属于同一个状态窗口，数值改变后该窗口关闭。如下图所示，根据状态量确定的状态窗口分别是[2019-04-28 14:22:07，2019-04-28 14:22:10]和[2019-04-28 14:22:11，2019-04-28 14:22:12]两个。（状态窗口暂不支持对超级表使用）
@ -41,7 +109,7 @@ _ 从 2.1.5.0 版本开始，INTERVAL 语句允许的最短时间间隔调整为
SELECT COUNT(*), FIRST(ts), status FROM temp_tb_1 STATE_WINDOW(status);
```
## 会话窗口
### 会话窗口
会话窗口根据记录的时间戳主键的值来确定是否属于同一个会话。如下图所示，如果设置时间戳的连续的间隔小于等于 12 秒，则以下 6 条记录构成 2 个会话窗口，分别是：[2019-04-28 14:22:10，2019-04-28 14:22:30]和[2019-04-28 14:23:10，2019-04-28 14:23:30]。因为 2019-04-28 14:22:30 与 2019-04-28 14:23:10 之间的时间间隔是 40 秒，超过了连续时间间隔（12 秒）。
@ -54,48 +122,7 @@ SELECT COUNT(*), FIRST(ts), status FROM temp_tb_1 STATE_WINDOW(status);
SELECT COUNT(*), FIRST(ts) FROM temp_tb_1 SESSION(ts, tol_val);
```
这种类型的查询语法如下：
```
SELECT function_list FROM tb_name
[WHERE where_condition]
[SESSION(ts_col, tol_val)]
[STATE_WINDOW(col)]
[INTERVAL(interval [, offset]) [SLIDING sliding]]
[FILL({NONE | VALUE | PREV | NULL | LINEAR | NEXT})]
SELECT function_list FROM stb_name
[WHERE where_condition]
[INTERVAL(interval [, offset]) [SLIDING sliding]]
[FILL({NONE | VALUE | PREV | NULL | LINEAR | NEXT})]
[GROUP BY tags]
```
- 在聚合查询中，function_list 位置允许使用聚合和选择函数，并要求每个函数仅输出单个结果（例如：COUNT、AVG、SUM、STDDEV、LEASTSQUARES、PERCENTILE、MIN、MAX、FIRST、LAST），而不能使用具有多行输出结果的函数（例如：DIFF 以及四则运算）。
- 此外 LAST_ROW 查询也不能与窗口聚合同时出现。
- 标量函数（如：CEIL/FLOOR 等）也不能使用在窗口聚合查询中。
-
- WHERE 语句可以指定查询的起止时间和其他过滤条件。
- FILL 语句指定某一窗口区间数据缺失的情况下的填充模式。填充模式包括以下几种：
1. 不进行填充：NONE（默认填充模式）。
2. VALUE 填充：固定值填充，此时需要指定填充的数值。例如：FILL(VALUE, 1.23)。
3. PREV 填充：使用前一个非 NULL 值填充数据。例如：FILL(PREV)。
4. NULL 填充：使用 NULL 填充数据。例如：FILL(NULL)。
5. LINEAR 填充：根据前后距离最近的非 NULL 值做线性插值填充。例如：FILL(LINEAR)。
6. NEXT 填充：使用下一个非 NULL 值填充数据。例如：FILL(NEXT)。
:::info
1. 使用 FILL 语句的时候可能生成大量的填充输出，务必指定查询的时间区间。针对每次查询，系统可返回不超过 1 千万条具有插值的结果。
2. 在时间维度聚合中，返回的结果中时间序列严格单调递增。
3. 如果查询对象是超级表，则聚合函数会作用于该超级表下满足值过滤条件的所有表的数据。如果查询中没有使用 GROUP BY 语句，则返回的结果按照时间序列严格单调递增；如果查询中使用了 GROUP BY 语句分组，则返回结果中每个 GROUP 内不按照时间序列严格单调递增。
:::
时间聚合也常被用于连续查询场景，可以参考文档 [连续查询(Continuous Query)](/develop/continuous-query)。
## 示例
### 示例
智能电表的建表语句如下：

57
docs/zh/12-taos-sql/14-limit.md
View File

@ -1,37 +1,43 @@
---
sidebar_label: 边界限制
title: 边界限制
sidebar_label: 命名与边界限制
title: 命名与边界限制
---
## 名称命名规则
1. 合法字符：英文字符、数字和下划线
2. 允许英文字符或下划线开头，不允许以数字开头
3. 不区分大小写
4. 转义后表（列）名规则：
为了兼容支持更多形式的表（列）名，TDengine 引入新的转义符 "`"。可用让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查。
转义后的表（列）名同样受到长度限制要求，且长度计算的时候不计算转义符。使用转义字符以后，不再对转义字符中的内容进行大小写统一。
例如：\`aBc\` 和 \`abc\` 是不同的表（列）名，但是 abc 和 aBc 是相同的表（列）名。
需要注意的是转义字符中的内容必须是可打印字符。
## 密码合法字符集
`[a-zA-Z0-9!?$%^&*()_–+={[}]:;@~#|<,>.?/]`
去掉了 `` ‘“`\ `` (单双引号、撇号、反斜杠、空格)
## 一般限制
- 数据库名最大长度为 32。
- 表名最大长度为 192，不包括数据库名前缀和分隔符
- 每行数据最大长度 48KB （注意：数据行内每个 BINARY/NCHAR 类型的列还会额外占用 2 个字节的存储位置）。
- 列名最大长度为 64，最多允许 4096 列，最少需要 2 列，第一列必须是时间戳。注：从 2.1.7.0 版本（不含)以前最多允许 4096 列
- 标签名最大长度为 64，最多允许 128 个，至少要有 1 个标签，一个表中标签值的总长度不超过 16KB 。
- 列名最大长度为 64
- 最多允许 4096 列，最少需要 2 列，第一列必须是时间戳。
- 标签名最大长度为 64
- 最多允许 128 个，至少要有 1 个标签，一个表中标签值的总长度不超过 16KB 。
- SQL 语句最大长度 1048576 个字符，也可通过客户端配置参数 maxSQLLength 修改，取值范围 65480 ~ 1048576。
- SELECT 语句的查询结果，最多允许返回 4096 列（语句中的函数调用可能也会占用一些列空间），超限时需要显式指定较少的返回数据列，以避免语句执行报错。注： 2.1.7.0 版本（不含）之前为最多允许 1024 列
- SELECT 语句的查询结果，最多允许返回 4096 列（语句中的函数调用可能也会占用一些列空间），超限时需要显式指定较少的返回数据列，以避免语句执行报错。
- 库的数目，超级表的数目、表的数目，系统不做限制，仅受系统资源限制。
## GROUP BY 的限制
TAOS SQL 支持对标签、TBNAME 进行 GROUP BY 操作，也支持普通列进行 GROUP BY，前提是：仅限一列且该列的唯一值小于 10 万个。注意：group by 不支持 float,double 类型。
## IS NOT NULL 的限制
IS NOT NULL 与不为空的表达式适用范围。
IS NOT NULL 支持所有类型的列。不为空的表达式为 <\>""，仅对非数值类型的列适用。
## ORDER BY 的限制
- 非超级表只能有一个 order by.
- 超级表最多两个 order by， 并且第二个必须为 ts.
- order by tag，必须和 group by tag 一起，并且是同一个 tag。 tbname 和 tag 一样逻辑。 只适用于超级表
- order by 普通列，必须和 group by 一起或者和 top/bottom 一起，并且是同一个普通列。 适用于超级表和普通表。如果同时存在 group by 和 top/bottom 一起，order by 优先必须和 group by 同一列。
- order by ts. 适用于超级表和普通表。
- order by ts 同时含有 group by 时 针对 group 内部用 ts 排序
- 数据库的副本数只能设置为 1 或 3
- 用户名的最大长度是 23 个字节
- 用户密码的最大长度是 15 个字节
- 总数据行数取决于可用资源
- 单个数据库的虚拟结点数上限为 1024
## 表(列)名合法性说明
@ -49,6 +55,3 @@ IS NOT NULL 支持所有类型的列。不为空的表达式为 <\>""，仅对
转义字符中的内容必须是可打印字符。
:::
### 支持版本
支持转义符的功能从 2.3.0.1 版本开始。

42
docs/zh/12-taos-sql/20-keywords.md
View File

@ -1,46 +1,8 @@
---
sidebar_label: 参数限制与保留关键字
title: TDengine 参数限制与保留关键字
sidebar_label: 保留关键字
title: TDengine 保留关键字
---
## 名称命名规则
1. 合法字符：英文字符、数字和下划线
2. 允许英文字符或下划线开头，不允许以数字开头
3. 不区分大小写
4. 转义后表（列）名规则：
为了兼容支持更多形式的表（列）名，TDengine 引入新的转义符 "`"。可用让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查。
转义后的表（列）名同样受到长度限制要求，且长度计算的时候不计算转义符。使用转义字符以后，不再对转义字符中的内容进行大小写统一。
例如：\`aBc\` 和 \`abc\` 是不同的表（列）名，但是 abc 和 aBc 是相同的表（列）名。
需要注意的是转义字符中的内容必须是可打印字符。
支持转义符的功能从 2.3.0.1 版本开始。
## 密码合法字符集
`[a-zA-Z0-9!?$%^&*()_–+={[}]:;@~#|<,>.?/]`
去掉了 `` ‘“`\ `` (单双引号、撇号、反斜杠、空格)
- 数据库名：不能包含“.”以及特殊字符，不能超过 32 个字符
- 表名：不能包含“.”以及特殊字符，与所属数据库名一起，不能超过 192 个字节 ，每行数据最大长度 48KB
- 表的列名：不能包含特殊字符，不能超过 64 个字节
- 数据库名、表名、列名，都不能以数字开头，合法的可用字符集是“英文字符、数字和下划线”
- 表的列数：不能超过 1024 列，最少需要 2 列，第一列必须是时间戳（从 2.1.7.0 版本开始，改为最多支持 4096 列）
- 记录的最大长度：包括时间戳 8 字节，不能超过 48KB（每个 BINARY/NCHAR 类型的列还会额外占用 2 个 字节 的存储位置）
- 单条 SQL 语句默认最大字符串长度：1048576 字节，但可通过系统配置参数 maxSQLLength 修改，取值范围 65480 ~ 1048576 字节
- 数据库副本数：不能超过 3
- 用户名：不能超过 23 个 字节
- 用户密码：不能超过 15 个 字节
- 标签(Tags)数量：不能超过 128 个，可以 0 个
- 标签的总长度：不能超过 16KB
- 记录条数：仅受存储空间限制
- 表的个数：仅受节点个数限制
- 库的个数：仅受节点个数限制
- 单个库上虚拟节点个数：不能超过 64 个
- 库的数目，超级表的数目、表的数目，系统不做限制，仅受系统资源限制
- SELECT 语句的查询结果，最多允许返回 1024 列（语句中的函数调用可能也会占用一些列空间），超限时需要显式指定较少的返回数据列，以避免语句执行报错。（从 2.1.7.0 版本开始，改为最多允许 4096 列）
## 保留关键字
目前 TDengine 有将近 200 个内部保留关键字，这些关键字无论大小写均不可以用作库名、表名、STable 名、数据列名及标签列名等。这些关键字列表如下：

30
examples/c/tmq.c
View File

@ -15,10 +15,10 @@
#include <assert.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>
#include "taos.h"
#include <stdlib.h>
static int running = 1;
static void msg_process(TAOS_RES* msg) {
@ -28,8 +28,8 @@ static void msg_process(TAOS_RES* msg) {
printf("db: %s\n", tmq_get_db_name(msg));
printf("vg: %d\n", tmq_get_vgroup_id(msg));
if (tmq_get_res_type(msg) == TMQ_RES_TABLE_META) {
tmq_raw_data *raw = tmq_get_raw_meta(msg);
if(raw){
tmq_raw_data* raw = tmq_get_raw_meta(msg);
if (raw) {
TAOS* pConn = taos_connect("192.168.1.86", "root", "taosdata", NULL, 0);
if (pConn == NULL) {
return;
@ -55,7 +55,7 @@ static void msg_process(TAOS_RES* msg) {
}
tmq_free_raw_meta(raw);
char* result = tmq_get_json_meta(msg);
if(result){
if (result) {
printf("meta result: %s\n", result);
}
tmq_free_json_meta(result);
@ -96,7 +96,9 @@ int32_t init_env() {
}
taos_free_result(pRes);
pRes = taos_query(pConn, "create stable if not exists st1 (ts timestamp, c1 int, c2 float, c3 binary(16)) tags(t1 int, t3 nchar(8), t4 bool)");
pRes = taos_query(pConn,
"create stable if not exists st1 (ts timestamp, c1 int, c2 float, c3 binary(16)) tags(t1 int, t3 "
"nchar(8), t4 bool)");
if (taos_errno(pRes) != 0) {
printf("failed to create super table st1, reason:%s\n", taos_errstr(pRes));
return -1;
@ -152,6 +154,7 @@ int32_t init_env() {
}
taos_free_result(pRes);
#if 0
pRes = taos_query(pConn, "alter table st1 add column c4 bigint");
if (taos_errno(pRes) != 0) {
printf("failed to alter super table st1, reason:%s\n", taos_errstr(pRes));
@ -264,7 +267,9 @@ int32_t init_env() {
}
taos_free_result(pRes);
pRes = taos_query(pConn, "create stable if not exists st1 (ts timestamp, c1 int, c2 float, c3 binary(16)) tags(t1 int, t3 nchar(8), t4 bool)");
pRes = taos_query(pConn,
"create stable if not exists st1 (ts timestamp, c1 int, c2 float, c3 binary(16)) tags(t1 int, t3 "
"nchar(8), t4 bool)");
if (taos_errno(pRes) != 0) {
printf("failed to create super table st1, reason:%s\n", taos_errstr(pRes));
return -1;
@ -277,6 +282,7 @@ int32_t init_env() {
return -1;
}
taos_free_result(pRes);
#endif
return 0;
}
@ -296,8 +302,15 @@ int32_t create_topic() {
}
taos_free_result(pRes);
pRes = taos_query(pConn, "create topic topic_ctb_column with meta as database abc1");
/*pRes = taos_query(pConn, "create topic topic_ctb_column as select ts, c1, c2, c3 from st1");*/
/*pRes = taos_query(pConn, "create topic topic_ctb_column with meta as database abc1");*/
pRes = taos_query(pConn, "create topic topic_ctb_column as select ts, c1, c2, c3 from st1");
if (taos_errno(pRes) != 0) {
printf("failed to create topic topic_ctb_column, reason:%s\n", taos_errstr(pRes));
return -1;
}
taos_free_result(pRes);
pRes = taos_query(pConn, "create topic topic2 as select ts, c1, c2, c3 from st1");
if (taos_errno(pRes) != 0) {
printf("failed to create topic topic_ctb_column, reason:%s\n", taos_errstr(pRes));
return -1;
@ -353,6 +366,7 @@ tmq_t* build_consumer() {
tmq_conf_t* conf = tmq_conf_new();
tmq_conf_set(conf, "group.id", "tg2");
tmq_conf_set(conf, "client.id", "my app 1");
tmq_conf_set(conf, "td.connect.user", "root");
tmq_conf_set(conf, "td.connect.pass", "taosdata");
tmq_conf_set(conf, "msg.with.table.name", "true");

1
include/common/tmsg.h
View File

@ -2924,7 +2924,6 @@ typedef struct {
SMqRspHead head;
STqOffsetVal reqOffset;
STqOffsetVal rspOffset;
int32_t skipLogNum;
int32_t blockNum;
int8_t withTbName;
int8_t withSchema;

7
include/libs/function/function.h
View File

@ -172,13 +172,8 @@ typedef struct tExprNode {
void tExprTreeDestroy(tExprNode *pNode, void (*fp)(void *));
typedef enum {
SHOULD_FREE_COLDATA = 0x1, // the newly created column data needs to be destroyed.
DELEGATED_MGMT_COLDATA = 0x2, // input column data should not be released.
} ECOLDATA_MGMT_TYPE_E;
struct SScalarParam {
ECOLDATA_MGMT_TYPE_E type;
bool colAlloced;
SColumnInfoData *columnData;
SHashObj *pHashFilter;
int32_t hashValueType;

4
include/libs/stream/tstream.h
View File

@ -83,6 +83,7 @@ typedef struct {
int32_t srcVgId;
int32_t childId;
int64_t sourceVer;
int64_t reqId;
SArray* blocks; // SArray<SSDataBlock*>
} SStreamDataBlock;
@ -324,6 +325,8 @@ static FORCE_INLINE int32_t streamTaskInput(SStreamTask* pTask, SStreamQueueItem
if (pItem->type == STREAM_INPUT__DATA_SUBMIT) {
SStreamDataSubmit* pSubmitClone = streamSubmitRefClone((SStreamDataSubmit*)pItem);
if (pSubmitClone == NULL) {
qDebug("task %d %p submit enqueue failed since out of memory", pTask->taskId, pTask);
terrno = TSDB_CODE_OUT_OF_MEMORY;
atomic_store_8(&pTask->inputStatus, TASK_INPUT_STATUS__FAILED);
return -1;
}
@ -412,6 +415,7 @@ typedef struct {
typedef struct {
int64_t streamId;
int64_t reqId;
int32_t srcTaskId;
int32_t srcNodeId;
int32_t dstTaskId;

2
include/libs/sync/sync.h
View File

@ -134,7 +134,7 @@ typedef struct SSyncFSM {
int32_t (*FpSnapshotDoRead)(struct SSyncFSM* pFsm, void* pReader, void** ppBuf, int32_t* len);
int32_t (*FpSnapshotStartWrite)(struct SSyncFSM* pFsm, void* pWriterParam, void** ppWriter);
int32_t (*FpSnapshotStopWrite)(struct SSyncFSM* pFsm, void* pWriter, bool isApply);
int32_t (*FpSnapshotStopWrite)(struct SSyncFSM* pFsm, void* pWriter, bool isApply, SSnapshot* pSnapshot);
int32_t (*FpSnapshotDoWrite)(struct SSyncFSM* pFsm, void* pWriter, void* pBuf, int32_t len);
} SSyncFSM;

10
source/client/src/clientImpl.c
View File

@ -1609,11 +1609,13 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
if (pResultInfo->convertJson == NULL) return TSDB_CODE_OUT_OF_MEMORY;
char* p1 = pResultInfo->convertJson;
int32_t totalLen = 0;
int32_t len = sizeof(int32_t) + sizeof(uint64_t) + numOfCols * (sizeof(int16_t) + sizeof(int32_t));
memcpy(p1, p, len);
p += len;
p1 += len;
totalLen += len;
len = sizeof(int32_t) * numOfCols;
int32_t* colLength = (int32_t*)p;
@ -1621,6 +1623,7 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
memcpy(p1, p, len);
p += len;
p1 += len;
totalLen += len;
char* pStart = p;
char* pStart1 = p1;
@ -1636,6 +1639,7 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
memcpy(pStart1, pStart, len);
pStart += len;
pStart1 += len;
totalLen += len;
len = 0;
for (int32_t j = 0; j < numOfRows; ++j) {
@ -1680,24 +1684,30 @@ static int32_t doConvertJson(SReqResultInfo* pResultInfo, int32_t numOfCols, int
len += varDataTLen(dst);
}
colLen1 = len;
totalLen += colLen1;
colLength1[i] = htonl(len);
} else if (IS_VAR_DATA_TYPE(pResultInfo->fields[i].type)) {
len = numOfRows * sizeof(int32_t);
memcpy(pStart1, pStart, len);
pStart += len;
pStart1 += len;
totalLen += len;
totalLen += colLen;
memcpy(pStart1, pStart, colLen);
} else {
len = BitmapLen(pResultInfo->numOfRows);
memcpy(pStart1, pStart, len);
pStart += len;
pStart1 += len;
totalLen += len;
totalLen += colLen;
memcpy(pStart1, pStart, colLen);
}
pStart += colLen;
pStart1 += colLen1;
}
*(int32_t*)(pResultInfo->convertJson) = totalLen;
pResultInfo->pData = pResultInfo->convertJson;
return TSDB_CODE_SUCCESS;
}

6
source/client/src/clientSml.c
View File

@ -609,7 +609,7 @@ static bool smlParseNumber(SSmlKv *kvVal, SSmlMsgBuf *msg) {
}
kvVal->type = TSDB_DATA_TYPE_BIGINT;
kvVal->i = (int64_t)result;
} else if ((left == 3 && strncasecmp(endptr, "u64", left) == 0)) {
} else if ((left == 1 && *endptr == 'u') || (left == 3 && strncasecmp(endptr, "u64", left) == 0)) {
if (result >= (double)UINT64_MAX || result < 0) {
errno = 0;
uint64_t tmp = taosStr2UInt64(pVal, &endptr, 10);
@ -1047,6 +1047,10 @@ static int32_t smlParseTelnetTags(const char *data, SArray *cols, char *childTab
continue;
}
if(valueLen > (TSDB_MAX_NCHAR_LEN - VARSTR_HEADER_SIZE) / TSDB_NCHAR_SIZE){
return TSDB_CODE_PAR_INVALID_VAR_COLUMN_LEN;
}
// add kv to SSmlKv
SSmlKv *kv = (SSmlKv *)taosMemoryCalloc(sizeof(SSmlKv), 1);
if (!kv) return TSDB_CODE_OUT_OF_MEMORY;

12
source/client/src/tmq.c
View File

@ -1052,6 +1052,7 @@ int32_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) {
int32_t code = -1;
req.consumerId = tmq->consumerId;
tstrncpy(req.clientId, tmq->clientId, 256);
tstrncpy(req.cgroup, tmq->groupId, TSDB_CGROUP_LEN);
req.topicNames = taosArrayInit(sz, sizeof(void*));
if (req.topicNames == NULL) goto FAIL;
@ -1146,14 +1147,6 @@ void tmq_conf_set_auto_commit_cb(tmq_conf_t* conf, tmq_commit_cb* cb, void* para
conf->commitCbUserParam = param;
}
#if 0
int32_t tmqGetSkipLogNum(tmq_message_t* tmq_message) {
if (tmq_message == NULL) return 0;
SMqPollRsp* pRsp = &tmq_message->msg;
return pRsp->skipLogNum;
}
#endif
int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {
SMqPollCbParam* pParam = (SMqPollCbParam*)param;
SMqClientVg* pVg = pParam->pVg;
@ -1296,9 +1289,6 @@ bool tmqUpdateEp2(tmq_t* tmq, int32_t epoch, SMqAskEpRsp* pRsp) {
offsetNew = *pOffset;
}
/*tscDebug("consumer:%" PRId64 ", (epoch %d) offset of vgId:%d updated to %" PRId64 ", vgKey is %s",
* tmq->consumerId, epoch,*/
/*pVgEp->vgId, offset, vgKey);*/
SMqClientVg clientVg = {
.pollCnt = 0,
.currentOffsetNew = offsetNew,

26
source/client/test/smlTest.cpp
View File

@ -499,6 +499,32 @@ TEST(testCase, smlGetTimestampLen_Test) {
ASSERT_EQ(len, 3);
}
TEST(testCase, smlParseNumber_Test) {
SSmlKv kv = {0};
char buf[64] = {0};
SSmlMsgBuf msg = {0};
msg.buf = buf;
msg.len = 64;
kv.value = "3.2e-900";
kv.length = 8;
bool res = smlParseNumber(&kv, &msg);
printf("res:%d,v:%f, %f\n", res,kv.d, HUGE_VAL);
}
//#include <stdlib.h>
//TEST(testCase, number_Test) {
// char *str[] = {
//// "-000 0999",
// "- abc",
// };
// for(int i = 0; i < sizeof(str)/sizeof(str[0]); i++){
// errno = 0;
// char *end = NULL;
// long result = strtol(str[i], &end, 10);
// printf("errno:%d,len:%d,result:%ld\n", errno, end - str[i], result);
// }
//
//}
/*
TEST(testCase, smlProcess_influx_Test) {
TAOS *taos = taos_connect("localhost", "root", "taosdata", NULL, 0);

4
source/common/src/tdatablock.c
View File

@ -1479,10 +1479,12 @@ static int32_t colDataMoveVarData(SColumnInfoData* pColInfoData, size_t start, s
}
beigin++;
}
if (dataOffset > 0) {
memmove(pColInfoData->pData, pColInfoData->pData + dataOffset, dataLen);
memmove(pColInfoData->varmeta.offset, &pColInfoData->varmeta.offset[start], (end - start) * sizeof(int32_t));
}
memmove(pColInfoData->varmeta.offset, &pColInfoData->varmeta.offset[start], (end - start) * sizeof(int32_t));
return dataLen;
}

6
source/common/src/tmsg.c
View File

@ -4919,8 +4919,8 @@ int tDecodeSVCreateStbReq(SDecoder *pCoder, SVCreateStbReq *pReq) {
if (tDecodeCStr(pCoder, &pReq->name) < 0) return -1;
if (tDecodeI64(pCoder, &pReq->suid) < 0) return -1;
if (tDecodeI8(pCoder, &pReq->rollup) < 0) return -1;
if (tDecodeSSchemaWrapper(pCoder, &pReq->schemaRow) < 0) return -1;
if (tDecodeSSchemaWrapper(pCoder, &pReq->schemaTag) < 0) return -1;
if (tDecodeSSchemaWrapperEx(pCoder, &pReq->schemaRow) < 0) return -1;
if (tDecodeSSchemaWrapperEx(pCoder, &pReq->schemaTag) < 0) return -1;
if (pReq->rollup) {
if (tDecodeSRSmaParam(pCoder, &pReq->rsmaParam) < 0) return -1;
}
@ -5648,7 +5648,6 @@ int32_t tDecodeDeleteRes(SDecoder *pCoder, SDeleteRes *pRes) {
int32_t tEncodeSMqDataRsp(SEncoder *pEncoder, const SMqDataRsp *pRsp) {
if (tEncodeSTqOffsetVal(pEncoder, &pRsp->reqOffset) < 0) return -1;
if (tEncodeSTqOffsetVal(pEncoder, &pRsp->rspOffset) < 0) return -1;
if (tEncodeI32(pEncoder, pRsp->skipLogNum) < 0) return -1;
if (tEncodeI32(pEncoder, pRsp->blockNum) < 0) return -1;
if (pRsp->blockNum != 0) {
if (tEncodeI8(pEncoder, pRsp->withTbName) < 0) return -1;
@ -5674,7 +5673,6 @@ int32_t tEncodeSMqDataRsp(SEncoder *pEncoder, const SMqDataRsp *pRsp) {
int32_t tDecodeSMqDataRsp(SDecoder *pDecoder, SMqDataRsp *pRsp) {
if (tDecodeSTqOffsetVal(pDecoder, &pRsp->reqOffset) < 0) return -1;
if (tDecodeSTqOffsetVal(pDecoder, &pRsp->rspOffset) < 0) return -1;
if (tDecodeI32(pDecoder, &pRsp->skipLogNum) < 0) return -1;
if (tDecodeI32(pDecoder, &pRsp->blockNum) < 0) return -1;
if (pRsp->blockNum != 0) {
pRsp->blockData = taosArrayInit(pRsp->blockNum, sizeof(void *));

23
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
View File

@ -153,15 +153,9 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
switch (qtype) {
case QUERY_QUEUE:
if ((pMsg->msgType == TDMT_SCH_QUERY) && (grantCheck(TSDB_GRANT_TIME) != TSDB_CODE_SUCCESS)) {
terrno = TSDB_CODE_GRANT_EXPIRED;
code = terrno;
dDebug("vgId:%d, msg:%p put into vnode-query queue failed since %s", pVnode->vgId, pMsg, terrstr());
} else {
vnodePreprocessQueryMsg(pVnode->pImpl, pMsg);
dGTrace("vgId:%d, msg:%p put into vnode-query queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pQueryQ, pMsg);
}
vnodePreprocessQueryMsg(pVnode->pImpl, pMsg);
dGTrace("vgId:%d, msg:%p put into vnode-query queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pQueryQ, pMsg);
break;
case STREAM_QUEUE:
dGTrace("vgId:%d, msg:%p put into vnode-stream queue", pVnode->vgId, pMsg);
@ -172,14 +166,9 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
taosWriteQitem(pVnode->pFetchQ, pMsg);
break;
case WRITE_QUEUE:
if ((pMsg->msgType == TDMT_VND_SUBMIT) && (grantCheck(TSDB_GRANT_STORAGE) != TSDB_CODE_SUCCESS)) {
terrno = TSDB_CODE_VND_NO_WRITE_AUTH;
code = terrno;
dDebug("vgId:%d, msg:%p put into vnode-write queue failed since %s", pVnode->vgId, pMsg, terrstr());
} else {
dGTrace("vgId:%d, msg:%p put into vnode-write queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pWriteQ, pMsg);
}
dGTrace("vgId:%d, msg:%p put into vnode-write queue", pVnode->vgId, pMsg);
taosWriteQitem(pVnode->pWriteQ, pMsg);
break;
case SYNC_QUEUE:
dGTrace("vgId:%d, msg:%p put into vnode-sync queue", pVnode->vgId, pMsg);

9
source/dnode/mnode/impl/src/mndConsumer.c
View File

@ -92,7 +92,9 @@ static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) {
SMnode *pMnode = pMsg->info.node;
SMqConsumerLostMsg *pLostMsg = pMsg->pCont;
SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId);
ASSERT(pConsumer);
if (pConsumer == NULL) {
return 0;
}
mInfo("receive consumer lost msg, consumer id %" PRId64 ", status %s", pLostMsg->consumerId,
mndConsumerStatusName(pConsumer->status));
@ -450,6 +452,7 @@ static int32_t mndProcessSubscribeReq(SRpcMsg *pMsg) {
int32_t code = -1;
SArray *newSub = subscribe.topicNames;
taosArraySortString(newSub, taosArrayCompareString);
taosArrayRemoveDuplicate(newSub, taosArrayCompareString, taosMemoryFree);
int32_t newTopicNum = taosArrayGetSize(newSub);
// check topic existance
@ -907,8 +910,8 @@ static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *
colDataAppend(pColInfo, numOfRows, (const char *)cgroup, false);
// client id
char clientId[TSDB_CGROUP_LEN + VARSTR_HEADER_SIZE] = {0};
tstrncpy(varDataVal(clientId), pConsumer->clientId, TSDB_CGROUP_LEN);
char clientId[256 + VARSTR_HEADER_SIZE] = {0};
tstrncpy(varDataVal(clientId), pConsumer->clientId, 256);
varDataSetLen(clientId, strlen(varDataVal(clientId)));
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataAppend(pColInfo, numOfRows, (const char *)clientId, false);

10
source/dnode/mnode/impl/src/mndDb.c
View File

@ -509,11 +509,11 @@ static int32_t mndProcessCreateDbReq(SRpcMsg *pReq) {
SUserObj *pUser = NULL;
SCreateDbReq createReq = {0};
code = grantCheck(TSDB_GRANT_DB);
if (code != 0) {
terrno = code;
goto _OVER;
}
// code = grantCheck(TSDB_GRANT_DB);
// if (code != 0) {
// terrno = code;
// goto _OVER;
// }
if (tDeserializeSCreateDbReq(pReq->pCont, pReq->contLen, &createReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;

2
source/dnode/mnode/impl/src/mndDef.c
View File

@ -199,6 +199,7 @@ int32_t tEncodeSMqConsumerObj(void **buf, const SMqConsumerObj *pConsumer) {
int32_t tlen = 0;
int32_t sz;
tlen += taosEncodeFixedI64(buf, pConsumer->consumerId);
tlen += taosEncodeString(buf, pConsumer->clientId);
tlen += taosEncodeString(buf, pConsumer->cgroup);
tlen += taosEncodeFixedI8(buf, pConsumer->updateType);
tlen += taosEncodeFixedI32(buf, pConsumer->epoch);
@ -264,6 +265,7 @@ int32_t tEncodeSMqConsumerObj(void **buf, const SMqConsumerObj *pConsumer) {
void *tDecodeSMqConsumerObj(const void *buf, SMqConsumerObj *pConsumer) {
int32_t sz;
buf = taosDecodeFixedI64(buf, &pConsumer->consumerId);
buf = taosDecodeStringTo(buf, pConsumer->clientId);
buf = taosDecodeStringTo(buf, pConsumer->cgroup);
buf = taosDecodeFixedI8(buf, &pConsumer->updateType);
buf = taosDecodeFixedI32(buf, &pConsumer->epoch);

10
source/dnode/mnode/impl/src/mndDnode.c
View File

@ -621,11 +621,11 @@ static int32_t mndProcessCreateDnodeReq(SRpcMsg *pReq) {
SDnodeObj *pDnode = NULL;
SCreateDnodeReq createReq = {0};
code = grantCheck(TSDB_GRANT_DNODE);
if (code != TSDB_CODE_SUCCESS) {
terrno = code;
goto _OVER;
}
// code = grantCheck(TSDB_GRANT_DNODE);
// if (code != TSDB_CODE_SUCCESS) {
// terrno = code;
// goto _OVER;
// }
if (tDeserializeSCreateDnodeReq(pReq->pCont, pReq->contLen, &createReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;

2
source/dnode/mnode/impl/src/mndSync.c
View File

@ -143,7 +143,7 @@ int32_t mndSnapshotStartWrite(struct SSyncFSM *pFsm, void *pParam, void **ppWrit
return sdbStartWrite(pMnode->pSdb, (SSdbIter **)ppWriter);
}
int32_t mndSnapshotStopWrite(struct SSyncFSM *pFsm, void *pWriter, bool isApply) {
int32_t mndSnapshotStopWrite(struct SSyncFSM *pFsm, void *pWriter, bool isApply, SSnapshot *pSnapshot) {
mInfo("stop to apply snapshot to sdb, apply:%d", isApply);
SMnode *pMnode = pFsm->data;
return sdbStopWrite(pMnode->pSdb, pWriter, isApply);

10
source/dnode/mnode/impl/src/mndUser.c
View File

@ -363,11 +363,11 @@ static int32_t mndProcessCreateUserReq(SRpcMsg *pReq) {
goto _OVER;
}
code = grantCheck(TSDB_GRANT_USER);
if (code != TSDB_CODE_SUCCESS) {
terrno = code;
goto _OVER;
}
// code = grantCheck(TSDB_GRANT_USER);
// if (code != TSDB_CODE_SUCCESS) {
// terrno = code;
// goto _OVER;
// }
code = mndCreateUser(pMnode, pOperUser->acct, &createReq, pReq);
if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS;

12
source/dnode/vnode/inc/vnode.h
View File

@ -116,11 +116,11 @@ int32_t metaTbCursorNext(SMTbCursor *pTbCur);
// typedef struct STsdb STsdb;
typedef struct STsdbReader STsdbReader;
#define BLOCK_LOAD_OFFSET_ORDER 1
#define BLOCK_LOAD_OFFSET_ORDER 1
#define BLOCK_LOAD_TABLESEQ_ORDER 2
#define BLOCK_LOAD_EXTERN_ORDER 3
#define BLOCK_LOAD_EXTERN_ORDER 3
#define LASTROW_RETRIEVE_TYPE_ALL 0x1
#define LASTROW_RETRIEVE_TYPE_ALL 0x1
#define LASTROW_RETRIEVE_TYPE_SINGLE 0x2
int32_t tsdbSetTableId(STsdbReader *pReader, int64_t uid);
@ -191,7 +191,7 @@ int32_t vnodeSnapReaderClose(SVSnapReader *pReader);
int32_t vnodeSnapRead(SVSnapReader *pReader, uint8_t **ppData, uint32_t *nData);
// SVSnapWriter
int32_t vnodeSnapWriterOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapWriter **ppWriter);
int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback);
int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback, SSnapshot *pSnapshot);
int32_t vnodeSnapWrite(SVSnapWriter *pWriter, uint8_t *pData, uint32_t nData);
// structs
@ -237,8 +237,8 @@ typedef struct {
uint64_t groupId;
} STableKeyInfo;
#define TABLE_ROLLUP_ON ((int8_t)0x1)
#define TABLE_IS_ROLLUP(FLG) (((FLG) & (TABLE_ROLLUP_ON)) != 0)
#define TABLE_ROLLUP_ON ((int8_t)0x1)
#define TABLE_IS_ROLLUP(FLG) (((FLG) & (TABLE_ROLLUP_ON)) != 0)
#define TABLE_SET_ROLLUP(FLG) ((FLG) |= TABLE_ROLLUP_ON)
struct SMetaEntry {
int64_t version;

1
source/dnode/vnode/src/inc/tq.h
View File

@ -52,7 +52,6 @@ typedef struct {
int64_t reqOffset;
int64_t processedVer;
int32_t epoch;
int32_t skipLogNum;
// rpc info
int64_t reqId;
SRpcHandleInfo rpcInfo;

2
source/dnode/vnode/src/meta/metaQuery.c
View File

@ -466,7 +466,7 @@ _err:
// N.B. Called by statusReq per second
int64_t metaGetTbNum(SMeta *pMeta) {
// TODO
return 100;
return 0;
}
// N.B. Called by statusReq per second

6
source/dnode/vnode/src/tq/tq.c
View File

@ -635,6 +635,8 @@ int32_t tqProcessStreamTrigger(STQ* pTq, SSubmitReq* pReq) {
pSubmit = streamDataSubmitNew(pReq);
if (pSubmit == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
qError("failed to create data submit for stream since out of memory");
failed = true;
}
@ -644,12 +646,16 @@ int32_t tqProcessStreamTrigger(STQ* pTq, SSubmitReq* pReq) {
SStreamTask* pTask = *(SStreamTask**)pIter;
if (!pTask->isDataScan) continue;
qDebug("data submit enqueue stream task: %d", pTask->taskId);
if (!failed) {
if (streamTaskInput(pTask, (SStreamQueueItem*)pSubmit) < 0) {
qError("stream task input failed, task id %d", pTask->taskId);
continue;
}
if (streamLaunchByWrite(pTask, TD_VID(pTq->pVnode)) < 0) {
qError("stream task launch failed, task id %d", pTask->taskId);
continue;
}
} else {

1
source/dnode/vnode/src/tq/tqExec.c
View File

@ -217,7 +217,6 @@ int32_t tqLogScanExec(STQ* pTq, STqExecHandle* pExec, SSubmitReq* pReq, SMqDataR
}
if (pRsp->blockNum == 0) {
pRsp->skipLogNum++;
return -1;
}

2
source/dnode/vnode/src/tq/tqPush.c
View File

@ -242,6 +242,8 @@ int tqPushMsg(STQ* pTq, void* msg, int32_t msgLen, tmsg_t msgType, int64_t ver)
void* data = taosMemoryMalloc(msgLen);
if (data == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
qError("failed to copy data for stream since out of memory");
return -1;
}
memcpy(data, msg, msgLen);

1
source/dnode/vnode/src/tsdb/tsdbRead.c
View File

@ -3186,6 +3186,7 @@ int32_t tsdbGetTableSchema(SVnode* pVnode, int64_t uid, STSchema** pSchema, int6
*suid = 0;
if (mr.me.type == TSDB_CHILD_TABLE) {
tDecoderClear(&mr.coder);
*suid = mr.me.ctbEntry.suid;
code = metaGetTableEntryByUid(&mr, *suid);
if (code != TSDB_CODE_SUCCESS) {

28
source/dnode/vnode/src/vnd/vnodeSnapshot.c
View File

@ -173,8 +173,9 @@ _err:
return code;
}
int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback) {
int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback, SSnapshot *pSnapshot) {
int32_t code = 0;
SVnode *pVnode = pWriter->pVnode;
if (pWriter->pMetaSnapWriter) {
code = metaSnapWriterClose(&pWriter->pMetaSnapWriter, rollback);
@ -186,8 +187,31 @@ int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback) {
if (code) goto _err;
}
if (!rollback) {
SVnodeInfo info = {0};
char dir[TSDB_FILENAME_LEN];
pVnode->state.committed = pWriter->ever;
pVnode->state.applied = pWriter->ever;
pVnode->state.applyTerm = pSnapshot->lastApplyTerm;
pVnode->state.commitTerm = pSnapshot->lastApplyTerm;
info.config = pVnode->config;
info.state.committed = pVnode->state.applied;
info.state.commitTerm = pVnode->state.applyTerm;
info.state.commitID = pVnode->state.commitID;
snprintf(dir, TSDB_FILENAME_LEN, "%s%s%s", tfsGetPrimaryPath(pVnode->pTfs), TD_DIRSEP, pVnode->path);
code = vnodeSaveInfo(dir, &info);
if (code) goto _err;
code = vnodeCommitInfo(dir, &info);
if (code) goto _err;
} else {
ASSERT(0);
}
_exit:
vInfo("vgId:%d vnode snapshot writer closed, rollback:%d", TD_VID(pWriter->pVnode), rollback);
vInfo("vgId:%d vnode snapshot writer closed, rollback:%d", TD_VID(pVnode), rollback);
taosMemoryFree(pWriter);
return code;

4
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -422,8 +422,8 @@ static int32_t vnodeProcessCreateStbReq(SVnode *pVnode, int64_t version, void *p
goto _err;
}
taosMemoryFree(req.schemaRow.pSchema);
taosMemoryFree(req.schemaTag.pSchema);
// taosMemoryFree(req.schemaRow.pSchema);
// taosMemoryFree(req.schemaTag.pSchema);
tDecoderClear(&coder);
return 0;

4
source/dnode/vnode/src/vnd/vnodeSync.c
View File

@ -515,10 +515,10 @@ static int32_t vnodeSnapshotStartWrite(struct SSyncFSM *pFsm, void *pParam, void
#endif
}
static int32_t vnodeSnapshotStopWrite(struct SSyncFSM *pFsm, void *pWriter, bool isApply) {
static int32_t vnodeSnapshotStopWrite(struct SSyncFSM *pFsm, void *pWriter, bool isApply, SSnapshot *pSnapshot) {
#ifdef USE_TSDB_SNAPSHOT
SVnode *pVnode = pFsm->data;
int32_t code = vnodeSnapWriterClose(pWriter, !isApply);
int32_t code = vnodeSnapWriterClose(pWriter, !isApply, pSnapshot);
return code;
#else
taosMemoryFree(pWriter);

26
source/libs/executor/inc/executorimpl.h
View File

@ -247,6 +247,16 @@ typedef struct SLoadRemoteDataInfo {
uint64_t totalElapsed; // total elapsed time
} SLoadRemoteDataInfo;
typedef struct SLimitInfo {
SLimit limit;
SLimit slimit;
uint64_t currentGroupId;
int64_t remainGroupOffset;
int64_t numOfOutputGroups;
int64_t remainOffset;
int64_t numOfOutputRows;
} SLimitInfo;
typedef struct SExchangeInfo {
SArray* pSources;
SArray* pSourceDataInfo;
@ -257,6 +267,7 @@ typedef struct SExchangeInfo {
int32_t current;
SLoadRemoteDataInfo loadInfo;
uint64_t self;
SLimitInfo limitInfo;
} SExchangeInfo;
typedef struct SColMatchInfo {
@ -542,15 +553,7 @@ typedef struct SProjectOperatorInfo {
SNode* pFilterNode; // filter info, which is push down by optimizer
SSDataBlock* existDataBlock;
SArray* pPseudoColInfo;
SLimit limit;
SLimit slimit;
uint64_t groupId;
int64_t curSOffset;
int64_t curGroupOutput;
int64_t curOffset;
int64_t curOutput;
SLimitInfo limitInfo;
} SProjectOperatorInfo;
typedef struct SIndefOperatorInfo {
@ -791,6 +794,9 @@ int32_t initAggInfo(SExprSupp *pSup, SAggSupporter* pAggSup, SExprInfo* pExprInf
const char* pkey);
void initResultSizeInfo(SOperatorInfo* pOperator, int32_t numOfRows);
void doBuildResultDatablock(SOperatorInfo* pOperator, SOptrBasicInfo* pbInfo, SGroupResInfo* pGroupResInfo, SDiskbasedBuf* pBuf);
int32_t handleLimitOffset(SOperatorInfo *pOperator, SLimitInfo* pLimitInfo, SSDataBlock* pBlock, bool holdDataInBuf);
bool hasLimitOffsetInfo(SLimitInfo* pLimitInfo);
void initLimitInfo(const SNode* pLimit, const SNode* pSLimit, SLimitInfo* pLimitInfo);
void doApplyFunctions(SExecTaskInfo* taskInfo, SqlFunctionCtx* pCtx, STimeWindow* pWin, SColumnInfoData* pTimeWindowData, int32_t offset,
int32_t forwardStep, TSKEY* tsCol, int32_t numOfTotal, int32_t numOfOutput, int32_t order);
@ -837,7 +843,7 @@ SOperatorInfo* createAggregateOperatorInfo(SOperatorInfo* downstream, SExprInfo*
SOperatorInfo* createIndefinitOutputOperatorInfo(SOperatorInfo* downstream, SPhysiNode *pNode, SExecTaskInfo* pTaskInfo);
SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SProjectPhysiNode* pProjPhyNode, SExecTaskInfo* pTaskInfo);
SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortPhyNode, SExecTaskInfo* pTaskInfo);
SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortNode, SExecTaskInfo* pTaskInfo);
SOperatorInfo* createMultiwayMergeOperatorInfo(SOperatorInfo** dowStreams, size_t numStreams, SMergePhysiNode* pMergePhysiNode, SExecTaskInfo* pTaskInfo);
SOperatorInfo* createSortedMergeOperatorInfo(SOperatorInfo** downstream, int32_t numOfDownstream, SExprInfo* pExprInfo, int32_t num, SArray* pSortInfo, SArray* pGroupInfo, SExecTaskInfo* pTaskInfo);
SOperatorInfo* createLastrowScanOperator(SLastRowScanPhysiNode* pTableScanNode, SReadHandle* readHandle, SExecTaskInfo* pTaskInfo);

3
source/libs/executor/src/dataInserter.c
View File

@ -90,6 +90,7 @@ _return:
tsem_post(&pInserter->ready);
taosMemoryFree(pMsg->pData);
taosMemoryFree(param);
return TSDB_CODE_SUCCESS;
@ -283,6 +284,8 @@ static int32_t destroyDataSinker(SDataSinkHandle* pHandle) {
atomic_sub_fetch_64(&gDataSinkStat.cachedSize, pInserter->cachedSize);
taosArrayDestroy(pInserter->pDataBlocks);
taosMemoryFree(pInserter->pSchema);
taosMemoryFree(pInserter->pParam);
taosHashCleanup(pInserter->pCols);
taosThreadMutexDestroy(&pInserter->mutex);
return TSDB_CODE_SUCCESS;
}

19
source/libs/executor/src/executil.c
View File

@ -899,3 +899,22 @@ STimeWindow getActiveTimeWindow(SDiskbasedBuf* pBuf, SResultRowInfo* pResultRowI
return w;
}
bool hasLimitOffsetInfo(SLimitInfo* pLimitInfo) {
return (pLimitInfo->limit.limit != -1 || pLimitInfo->limit.offset != -1 || pLimitInfo->slimit.limit != -1 ||
pLimitInfo->slimit.offset != -1);
}
static int64_t getLimit(const SNode* pLimit) { return NULL == pLimit ? -1 : ((SLimitNode*)pLimit)->limit; }
static int64_t getOffset(const SNode* pLimit) { return NULL == pLimit ? -1 : ((SLimitNode*)pLimit)->offset; }
void initLimitInfo(const SNode* pLimit, const SNode* pSLimit, SLimitInfo* pLimitInfo) {
SLimit limit = {.limit = getLimit(pLimit), .offset = getOffset(pLimit)};
SLimit slimit = {.limit = getLimit(pSLimit), .offset = getOffset(pSLimit)};
pLimitInfo->limit = limit;
pLimitInfo->slimit= slimit;
pLimitInfo->remainOffset = limit.offset;
pLimitInfo->remainGroupOffset = slimit.offset;
}

133
source/libs/executor/src/executorimpl.c
View File

@ -43,6 +43,11 @@
#define GET_FORWARD_DIRECTION_FACTOR(ord) (((ord) == TSDB_ORDER_ASC) ? QUERY_ASC_FORWARD_STEP : QUERY_DESC_FORWARD_STEP)
enum {
PROJECT_RETRIEVE_CONTINUE = 0x1,
PROJECT_RETRIEVE_DONE = 0x2,
};
#if 0
static UNUSED_FUNC void *u_malloc (size_t __size) {
uint32_t v = taosRand();
@ -624,6 +629,7 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc
int32_t startOffset = createNewColModel ? 0 : pResult->info.rows;
ASSERT(pResult->info.capacity > 0);
colDataMergeCol(pResColData, startOffset, &pResult->info.capacity, &idata, dest.numOfRows);
colDataDestroy(&idata);
numOfRows = dest.numOfRows;
taosArrayDestroy(pBlockList);
@ -679,6 +685,7 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc
int32_t startOffset = createNewColModel ? 0 : pResult->info.rows;
ASSERT(pResult->info.capacity > 0);
colDataMergeCol(pResColData, startOffset, &pResult->info.capacity, &idata, dest.numOfRows);
colDataDestroy(&idata);
numOfRows = dest.numOfRows;
taosArrayDestroy(pBlockList);
@ -2341,7 +2348,7 @@ static int32_t prepareLoadRemoteData(SOperatorInfo* pOperator) {
return TSDB_CODE_SUCCESS;
}
static SSDataBlock* doLoadRemoteData(SOperatorInfo* pOperator) {
static SSDataBlock* doLoadRemoteDataImpl(SOperatorInfo* pOperator) {
SExchangeInfo* pExchangeInfo = pOperator->info;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
@ -2367,6 +2374,44 @@ static SSDataBlock* doLoadRemoteData(SOperatorInfo* pOperator) {
}
}
static SSDataBlock* doLoadRemoteData(SOperatorInfo* pOperator) {
SExchangeInfo* pExchangeInfo = pOperator->info;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
if (pOperator->status == OP_EXEC_DONE) {
return NULL;
}
while(1) {
SSDataBlock* pBlock = doLoadRemoteDataImpl(pOperator);
if (pBlock == NULL) {
return NULL;
}
ASSERT(pBlock == pExchangeInfo->pResult);
SLimitInfo* pLimitInfo = &pExchangeInfo->limitInfo;
if (hasLimitOffsetInfo(pLimitInfo)) {
int32_t status = handleLimitOffset(pOperator, pLimitInfo, pExchangeInfo->pResult, false);
if (status == PROJECT_RETRIEVE_CONTINUE) {
continue;
} else if (status == PROJECT_RETRIEVE_DONE) {
size_t rows = pExchangeInfo->pResult->info.rows;
pExchangeInfo->limitInfo.numOfOutputRows += rows;
if (rows == 0) {
doSetOperatorCompleted(pOperator);
return NULL;
} else {
return pExchangeInfo->pResult;
}
}
} else {
return pExchangeInfo->pResult;
}
}
}
static int32_t initDataSource(int32_t numOfSources, SExchangeInfo* pInfo, const char* id) {
pInfo->pSourceDataInfo = taosArrayInit(numOfSources, sizeof(SSourceDataInfo));
if (pInfo->pSourceDataInfo == NULL) {
@ -2406,6 +2451,7 @@ static int32_t initExchangeOperator(SExchangePhysiNode* pExNode, SExchangeInfo*
taosArrayPush(pInfo->pSources, pNode);
}
initLimitInfo(pExNode->node.pLimit, pExNode->node.pSlimit, &pInfo->limitInfo);
pInfo->self = taosAddRef(exchangeObjRefPool, pInfo);
return initDataSource(numOfSources, pInfo, id);
@ -3149,68 +3195,60 @@ int32_t aggDecodeResultRow(SOperatorInfo* pOperator, char* result) {
return TDB_CODE_SUCCESS;
}
enum {
PROJECT_RETRIEVE_CONTINUE = 0x1,
PROJECT_RETRIEVE_DONE = 0x2,
};
static int32_t handleLimitOffset(SOperatorInfo* pOperator, SSDataBlock* pBlock) {
SProjectOperatorInfo* pProjectInfo = pOperator->info;
SOptrBasicInfo* pInfo = &pProjectInfo->binfo;
SSDataBlock* pRes = pInfo->pRes;
if (pProjectInfo->curSOffset > 0) {
if (pProjectInfo->groupId == 0) { // it is the first group
pProjectInfo->groupId = pBlock->info.groupId;
blockDataCleanup(pInfo->pRes);
int32_t handleLimitOffset(SOperatorInfo* pOperator, SLimitInfo* pLimitInfo, SSDataBlock* pBlock, bool holdDataInBuf) {
if (pLimitInfo->remainGroupOffset > 0) {
if (pLimitInfo->currentGroupId == 0) { // it is the first group
pLimitInfo->currentGroupId = pBlock->info.groupId;
blockDataCleanup(pBlock);
return PROJECT_RETRIEVE_CONTINUE;
} else if (pProjectInfo->groupId != pBlock->info.groupId) {
pProjectInfo->curSOffset -= 1;
} else if (pLimitInfo->currentGroupId != pBlock->info.groupId) {
// now it is the data from a new group
pLimitInfo->remainGroupOffset -= 1;
// ignore data block in current group
if (pProjectInfo->curSOffset > 0) {
blockDataCleanup(pInfo->pRes);
if (pLimitInfo->remainGroupOffset > 0) {
blockDataCleanup(pBlock);
return PROJECT_RETRIEVE_CONTINUE;
}
}
// set current group id of the project operator
pProjectInfo->groupId = pBlock->info.groupId;
pLimitInfo->currentGroupId = pBlock->info.groupId;
}
if (pProjectInfo->groupId != 0 && pProjectInfo->groupId != pBlock->info.groupId) {
pProjectInfo->curGroupOutput += 1;
if ((pProjectInfo->slimit.limit > 0) && (pProjectInfo->slimit.limit <= pProjectInfo->curGroupOutput)) {
if (pLimitInfo->currentGroupId != 0 && pLimitInfo->currentGroupId != pBlock->info.groupId) {
pLimitInfo->numOfOutputGroups += 1;
if ((pLimitInfo->slimit.limit > 0) && (pLimitInfo->slimit.limit <= pLimitInfo->numOfOutputGroups)) {
pOperator->status = OP_EXEC_DONE;
blockDataCleanup(pRes);
blockDataCleanup(pBlock);
return PROJECT_RETRIEVE_DONE;
}
// reset the value for a new group data
pProjectInfo->curOffset = 0;
pProjectInfo->curOutput = 0;
pLimitInfo->numOfOutputRows = 0;
pLimitInfo->remainOffset = pLimitInfo->limit.offset;
}
// here we reach the start position, according to the limit/offset requirements.
// set current group id
pProjectInfo->groupId = pBlock->info.groupId;
pLimitInfo->currentGroupId = pBlock->info.groupId;
if (pProjectInfo->curOffset >= pRes->info.rows) {
pProjectInfo->curOffset -= pRes->info.rows;
blockDataCleanup(pRes);
if (pLimitInfo->remainOffset >= pBlock->info.rows) {
pLimitInfo->remainOffset -= pBlock->info.rows;
blockDataCleanup(pBlock);
return PROJECT_RETRIEVE_CONTINUE;
} else if (pProjectInfo->curOffset < pRes->info.rows && pProjectInfo->curOffset > 0) {
blockDataTrimFirstNRows(pRes, pProjectInfo->curOffset);
pProjectInfo->curOffset = 0;
} else if (pLimitInfo->remainOffset < pBlock->info.rows && pLimitInfo->remainOffset > 0) {
blockDataTrimFirstNRows(pBlock, pLimitInfo->remainOffset);
pLimitInfo->remainOffset = 0;
}
// check for the limitation in each group
if (pProjectInfo->limit.limit >= 0 && pProjectInfo->curOutput + pRes->info.rows >= pProjectInfo->limit.limit) {
int32_t keepRows = (int32_t)(pProjectInfo->limit.limit - pProjectInfo->curOutput);
blockDataKeepFirstNRows(pRes, keepRows);
if (pProjectInfo->slimit.limit > 0 && pProjectInfo->slimit.limit <= pProjectInfo->curGroupOutput) {
if (pLimitInfo->limit.limit >= 0 && pLimitInfo->numOfOutputRows + pBlock->info.rows >= pLimitInfo->limit.limit) {
int32_t keepRows = (int32_t)(pLimitInfo->limit.limit - pLimitInfo->numOfOutputRows);
blockDataKeepFirstNRows(pBlock, keepRows);
if (pLimitInfo->slimit.limit > 0 && pLimitInfo->slimit.limit <= pLimitInfo->numOfOutputGroups) {
pOperator->status = OP_EXEC_DONE;
}
@ -3220,8 +3258,8 @@ static int32_t handleLimitOffset(SOperatorInfo* pOperator, SSDataBlock* pBlock)
// todo optimize performance
// If there are slimit/soffset value exists, multi-round result can not be packed into one group, since the
// they may not belong to the same group the limit/offset value is not valid in this case.
if (pRes->info.rows >= pOperator->resultInfo.threshold || pProjectInfo->slimit.offset != -1 ||
pProjectInfo->slimit.limit != -1) {
if ((!holdDataInBuf) || (pBlock->info.rows >= pOperator->resultInfo.threshold) || pLimitInfo->slimit.offset != -1 ||
pLimitInfo->slimit.limit != -1) {
return PROJECT_RETRIEVE_DONE;
} else { // not full enough, continue to accumulate the output data in the buffer.
return PROJECT_RETRIEVE_CONTINUE;
@ -3307,7 +3345,7 @@ static SSDataBlock* doProjectOperation(SOperatorInfo* pOperator) {
longjmp(pTaskInfo->env, code);
}
int32_t status = handleLimitOffset(pOperator, pBlock);
int32_t status = handleLimitOffset(pOperator, &pProjectInfo->limitInfo, pInfo->pRes, true);
// filter shall be applied after apply functions and limit/offset on the result
doFilter(pProjectInfo->pFilterNode, pInfo->pRes);
@ -3319,9 +3357,9 @@ static SSDataBlock* doProjectOperation(SOperatorInfo* pOperator) {
}
}
pProjectInfo->curOutput += pInfo->pRes->info.rows;
size_t rows = pInfo->pRes->info.rows;
pProjectInfo->limitInfo.numOfOutputRows += rows;
pOperator->resultInfo.totalRows += rows;
if (pOperator->cost.openCost == 0) {
@ -3765,10 +3803,6 @@ static SArray* setRowTsColumnOutputInfo(SqlFunctionCtx* pCtx, int32_t numOfCols)
return pList;
}
static int64_t getLimit(SNode* pLimit) { return NULL == pLimit ? -1 : ((SLimitNode*)pLimit)->limit; }
static int64_t getOffset(SNode* pLimit) { return NULL == pLimit ? -1 : ((SLimitNode*)pLimit)->offset; }
SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SProjectPhysiNode* pProjPhyNode,
SExecTaskInfo* pTaskInfo) {
SProjectOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SProjectOperatorInfo));
@ -3781,13 +3815,8 @@ SOperatorInfo* createProjectOperatorInfo(SOperatorInfo* downstream, SProjectPhys
SExprInfo* pExprInfo = createExprInfo(pProjPhyNode->pProjections, NULL, &numOfCols);
SSDataBlock* pResBlock = createResDataBlock(pProjPhyNode->node.pOutputDataBlockDesc);
SLimit limit = {.limit = getLimit(pProjPhyNode->node.pLimit), .offset = getOffset(pProjPhyNode->node.pLimit)};
SLimit slimit = {.limit = getLimit(pProjPhyNode->node.pSlimit), .offset = getOffset(pProjPhyNode->node.pSlimit)};
initLimitInfo(pProjPhyNode->node.pLimit, pProjPhyNode->node.pSlimit, &pInfo->limitInfo);
pInfo->limit = limit;
pInfo->slimit = slimit;
pInfo->curOffset = limit.offset;
pInfo->curSOffset = slimit.offset;
pInfo->binfo.pRes = pResBlock;
pInfo->pFilterNode = pProjPhyNode->node.pConditions;

22
source/libs/executor/src/groupoperator.c
View File

@ -440,19 +440,19 @@ static void doHashPartition(SOperatorInfo* pOperator, SSDataBlock* pBlock) {
recordNewGroupKeys(pInfo->pGroupCols, pInfo->pGroupColVals, pBlock, j);
int32_t len = buildGroupKeys(pInfo->keyBuf, pInfo->pGroupColVals);
SDataGroupInfo* pGInfo = NULL;
void *pPage = getCurrentDataGroupInfo(pInfo, &pGInfo, len);
SDataGroupInfo* pGroupInfo = NULL;
void *pPage = getCurrentDataGroupInfo(pInfo, &pGroupInfo, len);
pGInfo->numOfRows += 1;
if (pGInfo->groupId == 0) {
pGInfo->groupId = calcGroupId(pInfo->keyBuf, len);
pGroupInfo->numOfRows += 1;
// group id
if (pGroupInfo->groupId == 0) {
pGroupInfo->groupId = calcGroupId(pInfo->keyBuf, len);
}
// number of rows
int32_t* rows = (int32_t*) pPage;
// group id
size_t numOfCols = pOperator->exprSupp.numOfExprs;
for(int32_t i = 0; i < numOfCols; ++i) {
SExprInfo* pExpr = &pOperator->exprSupp.pExprInfo[i];
@ -603,7 +603,13 @@ static void clearPartitionOperator(SPartitionOperatorInfo* pInfo) {
static int compareDataGroupInfo(const void* group1, const void* group2) {
const SDataGroupInfo* pGroupInfo1 = group1;
const SDataGroupInfo* pGroupInfo2 = group2;
return pGroupInfo1->groupId - pGroupInfo2->groupId;
if (pGroupInfo1->groupId == pGroupInfo2->groupId) {
ASSERT(0);
return 0;
}
return (pGroupInfo1->groupId < pGroupInfo2->groupId)? -1:1;
}
static SSDataBlock* buildPartitionResult(SOperatorInfo* pOperator) {

110
source/libs/executor/src/scanoperator.c
View File

@ -1802,83 +1802,6 @@ static SSDataBlock* buildInfoSchemaTableMetaBlock(char* tableName) {
return pBlock;
}
// TODO: check more datatype, json? and return detailed error when len is not enough
static int32_t convertTagDataToTagVarchar(int8_t tagType, char* tagVal, uint32_t tagLen, char* varData,
int32_t bufSize) {
int outputLen = -1;
switch (tagType) {
case TSDB_DATA_TYPE_TINYINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%d", *((int8_t*)tagVal));
break;
case TSDB_DATA_TYPE_UTINYINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%u", *((uint8_t*)tagVal));
break;
case TSDB_DATA_TYPE_SMALLINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%d", *((int16_t*)tagVal));
break;
case TSDB_DATA_TYPE_USMALLINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%u", *((uint16_t*)tagVal));
break;
case TSDB_DATA_TYPE_INT:
outputLen = snprintf(varDataVal(varData), bufSize, "%d", *((int32_t*)tagVal));
break;
case TSDB_DATA_TYPE_UINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%u", *((uint32_t*)tagVal));
break;
case TSDB_DATA_TYPE_BIGINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%" PRId64, *((int64_t*)tagVal));
break;
case TSDB_DATA_TYPE_UBIGINT:
outputLen = snprintf(varDataVal(varData), bufSize, "%" PRIu64, *((uint64_t*)tagVal));
break;
case TSDB_DATA_TYPE_FLOAT: {
float fv = 0;
fv = GET_FLOAT_VAL(tagVal);
outputLen = snprintf(varDataVal(varData), bufSize, "%f", fv);
break;
}
case TSDB_DATA_TYPE_DOUBLE: {
double dv = 0;
dv = GET_DOUBLE_VAL(tagVal);
outputLen = snprintf(varDataVal(varData), bufSize, "%lf", dv);
break;
}
case TSDB_DATA_TYPE_BINARY:
case TSDB_DATA_TYPE_NCHAR:
case TSDB_DATA_TYPE_JSON: {
memcpy(varDataVal(varData), tagVal, tagLen);
outputLen = tagLen;
break;
}
case TSDB_DATA_TYPE_TIMESTAMP:
outputLen = snprintf(varDataVal(varData), bufSize, "%" PRId64, *((int64_t*)tagVal));
break;
case TSDB_DATA_TYPE_BOOL:
outputLen = snprintf(varDataVal(varData), bufSize, "%d", *((int8_t*)tagVal));
break;
default:
return TSDB_CODE_FAILED;
}
if (outputLen < 0 || outputLen == bufSize && !IS_VAR_DATA_TYPE(tagType) || outputLen > bufSize) {
return TSDB_CODE_FAILED;
}
varDataSetLen(varData, outputLen);
return TSDB_CODE_SUCCESS;
}
static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SSysTableScanInfo* pInfo = pOperator->info;
@ -1962,10 +1885,9 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
tagVal.cid = smr.me.stbEntry.schemaTag.pSchema[i].colId;
char* tagData = NULL;
uint32_t tagLen = 0;
if (tagType == TSDB_DATA_TYPE_JSON) {
// TODO: json type?+varheader+data
tagData = varDataVal(pInfo->pCur->mr.me.ctbEntry.pTags + 1);
tagLen = varDataLen(pInfo->pCur->mr.me.ctbEntry.pTags + 1);
tagData = (char*)pInfo->pCur->mr.me.ctbEntry.pTags;
} else {
bool exist = tTagGet((STag*)pInfo->pCur->mr.me.ctbEntry.pTags, &tagVal);
if (exist) {
@ -1979,27 +1901,27 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
}
}
int32_t bufSize = IS_VAR_DATA_TYPE(tagType) ? (tagLen + VARSTR_HEADER_SIZE)
: (3 + DBL_MANT_DIG - DBL_MIN_EXP + VARSTR_HEADER_SIZE);
char* tagVarChar = NULL;
char* tagVarChar = NULL;
if (tagData != NULL) {
tagVarChar = taosMemoryMalloc(bufSize);
code = convertTagDataToTagVarchar(tagType, tagData, tagLen, tagVarChar, bufSize);
if (code != TSDB_CODE_SUCCESS) {
qError("failed to get super table meta, uid:0x%" PRIx64 ", code:%s, %s", suid, tstrerror(terrno),
GET_TASKID(pTaskInfo));
taosMemoryFree(tagVarChar);
metaReaderClear(&smr);
metaCloseTbCursor(pInfo->pCur);
pInfo->pCur = NULL;
longjmp(pTaskInfo->env, terrno);
if (tagType == TSDB_DATA_TYPE_JSON) {
char* tagJson = parseTagDatatoJson(tagData);
tagVarChar = taosMemoryMalloc(strlen(tagJson) + VARSTR_HEADER_SIZE);
memcpy(varDataVal(tagVarChar), tagJson, strlen(tagJson));
varDataSetLen(tagVarChar, strlen(tagJson));
taosMemoryFree(tagJson);
} else {
int32_t bufSize = IS_VAR_DATA_TYPE(tagType) ? (tagLen + VARSTR_HEADER_SIZE)
: (3 + DBL_MANT_DIG - DBL_MIN_EXP + VARSTR_HEADER_SIZE);
tagVarChar = taosMemoryMalloc(bufSize);
int32_t len = -1;
dataConverToStr(varDataVal(tagVarChar), tagType, tagData, tagLen, &len);
varDataSetLen(tagVarChar, len);
}
}
pColInfoData = taosArrayGet(p->pDataBlock, 5);
colDataAppend(pColInfoData, numOfRows, tagVarChar,
(tagData == NULL) || (tagType == TSDB_DATA_TYPE_JSON && tTagIsJsonNull(tagData)));
taosMemoryFree(tagVarChar);
++numOfRows;
}
metaReaderClear(&smr);

14
source/libs/executor/src/sortoperator.c
View File

@ -22,31 +22,31 @@ static int32_t getExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain
static void destroyOrderOperatorInfo(void* param, int32_t numOfOutput);
SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortPhyNode,
SExecTaskInfo* pTaskInfo) {
// todo add limit/offset impl
SOperatorInfo* createSortOperatorInfo(SOperatorInfo* downstream, SSortPhysiNode* pSortNode, SExecTaskInfo* pTaskInfo) {
SSortOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SSortOperatorInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
if (pInfo == NULL || pOperator == NULL /* || rowSize > 100 * 1024 * 1024*/) {
goto _error;
}
SDataBlockDescNode* pDescNode = pSortPhyNode->node.pOutputDataBlockDesc;
SDataBlockDescNode* pDescNode = pSortNode->node.pOutputDataBlockDesc;
int32_t numOfCols = 0;
SSDataBlock* pResBlock = createResDataBlock(pDescNode);
SExprInfo* pExprInfo = createExprInfo(pSortPhyNode->pExprs, NULL, &numOfCols);
SExprInfo* pExprInfo = createExprInfo(pSortNode->pExprs, NULL, &numOfCols);
int32_t numOfOutputCols = 0;
SArray* pColMatchColInfo =
extractColMatchInfo(pSortPhyNode->pTargets, pDescNode, &numOfOutputCols, COL_MATCH_FROM_SLOT_ID);
extractColMatchInfo(pSortNode->pTargets, pDescNode, &numOfOutputCols, COL_MATCH_FROM_SLOT_ID);
pOperator->exprSupp.pCtx = createSqlFunctionCtx(pExprInfo, numOfCols, &pOperator->exprSupp.rowEntryInfoOffset);
pInfo->binfo.pRes = pResBlock;
initResultSizeInfo(pOperator, 1024);
pInfo->pSortInfo = createSortInfo(pSortPhyNode->pSortKeys);
pInfo->pCondition = pSortPhyNode->node.pConditions;
pInfo->pSortInfo = createSortInfo(pSortNode->pSortKeys);
pInfo->pCondition = pSortNode->node.pConditions;
pInfo->pColMatchInfo = pColMatchColInfo;
pOperator->name = "SortOperator";
pOperator->operatorType = QUERY_NODE_PHYSICAL_PLAN_SORT;

2
source/libs/executor/src/tsort.c
View File

@ -91,7 +91,7 @@ SSortHandle* tsortCreateSortHandle(SArray* pSortInfo, int32_t type, int32_t page
tsortSetComparFp(pSortHandle, msortComparFn);
if (idstr != NULL) {
pSortHandle->idStr = strdup(idstr);
pSortHandle->idStr = strdup(idstr);
}
return pSortHandle;

1
source/libs/function/src/tudf.c
View File

@ -855,6 +855,7 @@ int32_t convertDataBlockToScalarParm(SSDataBlock *input, SScalarParam *output) {
memcpy(output->columnData,
taosArrayGet(input->pDataBlock, 0),
sizeof(SColumnInfoData));
output->colAlloced = true;
return 0;
}

11
source/libs/index/src/indexFstDfa.c
View File

@ -19,8 +19,8 @@
const static uint32_t STATE_LIMIT = 1000;
static int dfaInstsEqual(const void *a, const void *b, size_t size) {
SArray *ar = (SArray *)a;
SArray *br = (SArray *)b;
SArray *ar = *(SArray **)a;
SArray *br = *(SArray **)b;
size_t al = ar != NULL ? taosArrayGetSize(ar) : 0;
size_t bl = br != NULL ? taosArrayGetSize(br) : 0;
if (al != bl) {
@ -71,8 +71,8 @@ FstDfa *dfaBuilderBuild(FstDfaBuilder *builder) {
dfaAdd(builder->dfa, cur, 0);
SArray *states = taosArrayInit(0, sizeof(uint32_t));
uint32_t result;
SArray *states = taosArrayInit(0, sizeof(uint32_t));
if (dfaBuilderCacheState(builder, cur, &result)) {
taosArrayPush(states, &result);
}
@ -146,10 +146,9 @@ bool dfaBuilderCacheState(FstDfaBuilder *builder, FstSparseSet *set, uint32_t *r
*result = *v;
taosArrayDestroy(tinsts);
} else {
DfaState st;
st.insts = tinsts;
st.isMatch = isMatch;
DfaState st = {.insts = tinsts, .isMatch = isMatch};
taosArrayPush(builder->dfa->states, &st);
int32_t sz = taosArrayGetSize(builder->dfa->states) - 1;
taosHashPut(builder->cache, &tinsts, sizeof(POINTER_BYTES), &sz, sizeof(sz));
*result = sz;

3
source/libs/index/src/indexFstFile.c
View File

@ -85,11 +85,12 @@ static int idxFileCtxDoReadFrom(IFileCtx* ctx, uint8_t* buf, int len, int32_t of
blk->blockId = blkId;
blk->nread = taosPReadFile(ctx->file.pFile, blk->buf, kBlockSize, blkId * kBlockSize);
assert(blk->nread <= kBlockSize);
nread = TMIN(blkLeft, len);
if (blk->nread < kBlockSize && blk->nread < len) {
break;
}
nread = TMIN(blkLeft, len);
memcpy(buf + total, blk->buf + blkOffset, nread);
LRUStatus s = taosLRUCacheInsert(ctx->lru, key, strlen(key), blk, cacheMemSize, deleteDataBlockFromLRU, NULL,

2
source/libs/index/src/indexFstSparse.c
View File

@ -78,8 +78,8 @@ bool sparSetContains(FstSparseSet *ss, int32_t ip) {
if (ip >= ss->cap || ip < 0) {
return false;
}
int32_t i = ss->sparse[ip];
int32_t i = ss->sparse[ip];
if (i >= 0 && i < ss->cap && i < ss->size && ss->dense[i] == ip) {
return true;
} else {

1
source/libs/nodes/src/nodesUtilFuncs.c
View File

@ -952,6 +952,7 @@ void nodesDestroyNode(SNode* pNode) {
case QUERY_NODE_PHYSICAL_PLAN_QUERY_INSERT: {
SQueryInserterNode* pSink = (SQueryInserterNode*)pNode;
destroyDataSinkNode((SDataSinkNode*)pSink);
nodesDestroyList(pSink->pCols);
break;
}
case QUERY_NODE_PHYSICAL_PLAN_DELETE: {

46
source/libs/scalar/src/scalar.c
View File

@ -55,7 +55,7 @@ int32_t sclCreateColumnInfoData(SDataType* pType, int32_t numOfRows, SScalarPara
}
pParam->columnData = pColumnData;
pParam->type = SHOULD_FREE_COLDATA;
pParam->colAlloced = true;
return TSDB_CODE_SUCCESS;
}
@ -166,6 +166,10 @@ void sclFreeRes(SHashObj *res) {
}
void sclFreeParam(SScalarParam *param) {
if (!param->colAlloced) {
return;
}
if (param->columnData != NULL) {
colDataDestroy(param->columnData);
taosMemoryFreeClear(param->columnData);
@ -173,6 +177,7 @@ void sclFreeParam(SScalarParam *param) {
if (param->pHashFilter != NULL) {
taosHashCleanup(param->pHashFilter);
param->pHashFilter = NULL;
}
}
@ -191,6 +196,19 @@ int32_t sclCopyValueNodeValue(SValueNode *pNode, void **res) {
return TSDB_CODE_SUCCESS;
}
void sclFreeParamList(SScalarParam *param, int32_t paramNum) {
if (NULL == param) {
return;
}
for (int32_t i = 0; i < paramNum; ++i) {
SScalarParam* p = param + i;
sclFreeParam(p);
}
taosMemoryFree(param);
}
int32_t sclInitParam(SNode* node, SScalarParam *param, SScalarCtx *ctx, int32_t *rowNum) {
switch (nodeType(node)) {
case QUERY_NODE_LEFT_VALUE: {
@ -225,11 +243,14 @@ int32_t sclInitParam(SNode* node, SScalarParam *param, SScalarCtx *ctx, int32_t
SCL_ERR_RET(scalarGenerateSetFromList((void **)&param->pHashFilter, node, type));
param->hashValueType = type;
param->colAlloced = true;
if (taosHashPut(ctx->pRes, &node, POINTER_BYTES, param, sizeof(*param))) {
taosHashCleanup(param->pHashFilter);
param->pHashFilter = NULL;
sclError("taosHashPut nodeList failed, size:%d", (int32_t)sizeof(*param));
return TSDB_CODE_QRY_OUT_OF_MEMORY;
}
param->colAlloced = false;
break;
}
case QUERY_NODE_COLUMN: {
@ -274,6 +295,7 @@ int32_t sclInitParam(SNode* node, SScalarParam *param, SScalarCtx *ctx, int32_t
SCL_ERR_RET(TSDB_CODE_QRY_APP_ERROR);
}
*param = *res;
param->colAlloced = false;
break;
}
default:
@ -455,11 +477,7 @@ int32_t sclExecFunction(SFunctionNode *node, SScalarCtx *ctx, SScalarParam *outp
_return:
for (int32_t i = 0; i < paramNum; ++i) {
// sclFreeParamNoData(params + i);
}
taosMemoryFreeClear(params);
sclFreeParamList(params, paramNum);
SCL_RET(code);
}
@ -533,11 +551,7 @@ int32_t sclExecLogic(SLogicConditionNode *node, SScalarCtx *ctx, SScalarParam *o
_return:
for (int32_t i = 0; i < paramNum; ++i) {
// sclFreeParamNoData(params + i);
}
taosMemoryFreeClear(params);
sclFreeParamList(params, paramNum);
SCL_RET(code);
}
@ -573,14 +587,8 @@ int32_t sclExecOperator(SOperatorNode *node, SScalarCtx *ctx, SScalarParam *outp
code = terrno;
_return:
for (int32_t i = 0; i < paramNum; ++i) {
if (params[i].type == SHOULD_FREE_COLDATA) {
colDataDestroy(params[i].columnData);
taosMemoryFreeClear(params[i].columnData);
}
}
taosMemoryFreeClear(params);
sclFreeParamList(params, paramNum);
SCL_RET(code);
}
@ -871,7 +879,6 @@ EDealRes sclWalkFunction(SNode* pNode, SScalarCtx *ctx) {
return DEAL_RES_ERROR;
}
output.type = DELEGATED_MGMT_COLDATA;
if (taosHashPut(ctx->pRes, &pNode, POINTER_BYTES, &output, sizeof(output))) {
ctx->code = TSDB_CODE_QRY_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
@ -906,7 +913,6 @@ EDealRes sclWalkOperator(SNode* pNode, SScalarCtx *ctx) {
return DEAL_RES_ERROR;
}
output.type = DELEGATED_MGMT_COLDATA;
if (taosHashPut(ctx->pRes, &pNode, POINTER_BYTES, &output, sizeof(output))) {
ctx->code = TSDB_CODE_QRY_OUT_OF_MEMORY;
return DEAL_RES_ERROR;

2
source/libs/scalar/test/filter/filterTests.cpp
View File

@ -207,7 +207,7 @@ void flttMakeListNode(SNode **pNode, SNodeList *list, int32_t resType) {
void initScalarParam(SScalarParam* pParam) {
memset(pParam, 0, sizeof(SScalarParam));
pParam->type = SHOULD_FREE_COLDATA;
pParam->colAlloced = true;
}
}

3
source/libs/stream/src/stream.c
View File

@ -143,6 +143,9 @@ int32_t streamTaskEnqueueRetrieve(SStreamTask* pTask, SStreamRetrieveReq* pReq,
// enqueue
if (pData != NULL) {
qDebug("task %d(child %d) recv retrieve req from task %d, reqId %ld", pTask->taskId, pTask->selfChildId,
pReq->srcTaskId, pReq->reqId);
pData->type = STREAM_INPUT__DATA_RETRIEVE;
pData->srcVgId = 0;
// decode

2
source/libs/stream/src/streamData.c
View File

@ -57,7 +57,9 @@ int32_t streamRetrieveReqToData(const SStreamRetrieveReq* pReq, SStreamDataBlock
pDataBlock->info.type = pRetrieve->streamBlockType;
pData->reqId = pReq->reqId;
pData->blocks = pArray;
return 0;
}

6
source/libs/stream/src/streamDispatch.c
View File

@ -65,6 +65,7 @@ int32_t tDecodeStreamDispatchReq(SDecoder* pDecoder, SStreamDispatchReq* pReq) {
int32_t tEncodeStreamRetrieveReq(SEncoder* pEncoder, const SStreamRetrieveReq* pReq) {
if (tStartEncode(pEncoder) < 0) return -1;
if (tEncodeI64(pEncoder, pReq->streamId) < 0) return -1;
if (tEncodeI64(pEncoder, pReq->reqId) < 0) return -1;
if (tEncodeI32(pEncoder, pReq->dstNodeId) < 0) return -1;
if (tEncodeI32(pEncoder, pReq->dstTaskId) < 0) return -1;
if (tEncodeI32(pEncoder, pReq->srcNodeId) < 0) return -1;
@ -77,6 +78,7 @@ int32_t tEncodeStreamRetrieveReq(SEncoder* pEncoder, const SStreamRetrieveReq* p
int32_t tDecodeStreamRetrieveReq(SDecoder* pDecoder, SStreamRetrieveReq* pReq) {
if (tStartDecode(pDecoder) < 0) return -1;
if (tDecodeI64(pDecoder, &pReq->streamId) < 0) return -1;
if (tDecodeI64(pDecoder, &pReq->reqId) < 0) return -1;
if (tDecodeI32(pDecoder, &pReq->dstNodeId) < 0) return -1;
if (tDecodeI32(pDecoder, &pReq->dstTaskId) < 0) return -1;
if (tDecodeI32(pDecoder, &pReq->srcNodeId) < 0) return -1;
@ -121,6 +123,7 @@ int32_t streamBroadcastToChildren(SStreamTask* pTask, const SSDataBlock* pBlock)
int32_t sz = taosArrayGetSize(pTask->childEpInfo);
ASSERT(sz > 0);
for (int32_t i = 0; i < sz; i++) {
req.reqId = tGenIdPI64();
SStreamChildEpInfo* pEpInfo = taosArrayGetP(pTask->childEpInfo, i);
req.dstNodeId = pEpInfo->nodeId;
req.dstTaskId = pEpInfo->taskId;
@ -154,6 +157,9 @@ int32_t streamBroadcastToChildren(SStreamTask* pTask, const SSDataBlock* pBlock)
ASSERT(0);
return -1;
}
qDebug("task %d(child %d) send retrieve req to task %d at node %d, reqId %ld", pTask->taskId, pTask->selfChildId,
pEpInfo->taskId, pEpInfo->nodeId, req.reqId);
}
return 0;
FAIL:

3
source/libs/stream/src/streamExec.c
View File

@ -54,6 +54,9 @@ static int32_t streamTaskExecImpl(SStreamTask* pTask, void* data, SArray* pRes)
block.info.type = STREAM_PULL_OVER;
block.info.childId = pTask->selfChildId;
taosArrayPush(pRes, &block);
qDebug("task %d(child %d) processed retrieve, reqId %ld", pTask->taskId, pTask->selfChildId,
pRetrieveBlock->reqId);
}
break;
}

25
source/libs/sync/src/syncSnapshot.c
View File

@ -374,14 +374,14 @@ cJSON *snapshotSender2Json(SSyncSnapshotSender *pSender) {
char *snapshotSender2Str(SSyncSnapshotSender *pSender) {
cJSON *pJson = snapshotSender2Json(pSender);
char * serialized = cJSON_Print(pJson);
char *serialized = cJSON_Print(pJson);
cJSON_Delete(pJson);
return serialized;
}
char *snapshotSender2SimpleStr(SSyncSnapshotSender *pSender, char *event) {
int32_t len = 256;
char * s = taosMemoryMalloc(len);
char *s = taosMemoryMalloc(len);
SRaftId destId = pSender->pSyncNode->replicasId[pSender->replicaIndex];
char host[64];
@ -434,8 +434,8 @@ void snapshotReceiverDestroy(SSyncSnapshotReceiver *pReceiver) {
if (pReceiver != NULL) {
// close writer
if (pReceiver->pWriter != NULL) {
int32_t ret =
pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, false);
int32_t ret = pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter,
false, &(pReceiver->snapshot));
ASSERT(ret == 0);
pReceiver->pWriter = NULL;
}
@ -483,8 +483,8 @@ static void snapshotReceiverDoStart(SSyncSnapshotReceiver *pReceiver, SyncSnapsh
static void snapshotReceiverForceStop(SSyncSnapshotReceiver *pReceiver) {
// force close, abandon incomplete data
if (pReceiver->pWriter != NULL) {
int32_t ret =
pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, false);
int32_t ret = pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, false,
&(pReceiver->snapshot));
ASSERT(ret == 0);
pReceiver->pWriter = NULL;
}
@ -524,8 +524,8 @@ int32_t snapshotReceiverStart(SSyncSnapshotReceiver *pReceiver, SyncSnapshotSend
// FpSnapshotStopWrite should not be called, assert writer == NULL
int32_t snapshotReceiverStop(SSyncSnapshotReceiver *pReceiver) {
if (pReceiver->pWriter != NULL) {
int32_t ret =
pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, false);
int32_t ret = pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, false,
&(pReceiver->snapshot));
ASSERT(ret == 0);
pReceiver->pWriter = NULL;
}
@ -574,7 +574,8 @@ static int32_t snapshotReceiverFinish(SSyncSnapshotReceiver *pReceiver, SyncSnap
}
// stop writer, apply data
code = pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, true);
code = pReceiver->pSyncNode->pFsm->FpSnapshotStopWrite(pReceiver->pSyncNode->pFsm, pReceiver->pWriter, true,
&(pReceiver->snapshot));
if (code != 0) {
syncNodeErrorLog(pReceiver->pSyncNode, "snapshot stop writer true error");
ASSERT(0);
@ -646,7 +647,7 @@ cJSON *snapshotReceiver2Json(SSyncSnapshotReceiver *pReceiver) {
cJSON_AddStringToObject(pFromId, "addr", u64buf);
{
uint64_t u64 = pReceiver->fromId.addr;
cJSON * pTmp = pFromId;
cJSON *pTmp = pFromId;
char host[128] = {0};
uint16_t port;
syncUtilU642Addr(u64, host, sizeof(host), &port);
@ -679,14 +680,14 @@ cJSON *snapshotReceiver2Json(SSyncSnapshotReceiver *pReceiver) {
char *snapshotReceiver2Str(SSyncSnapshotReceiver *pReceiver) {
cJSON *pJson = snapshotReceiver2Json(pReceiver);
char * serialized = cJSON_Print(pJson);
char *serialized = cJSON_Print(pJson);
cJSON_Delete(pJson);
return serialized;
}
char *snapshotReceiver2SimpleStr(SSyncSnapshotReceiver *pReceiver, char *event) {
int32_t len = 256;
char * s = taosMemoryMalloc(len);
char *s = taosMemoryMalloc(len);
SRaftId fromId = pReceiver->fromId;
char host[128];

2
source/libs/sync/test/syncConfigChangeSnapshotTest.cpp
View File

@ -125,7 +125,7 @@ int32_t SnapshotStartWrite(struct SSyncFSM* pFsm, void* pParam, void** ppWriter)
return 0;
}
int32_t SnapshotStopWrite(struct SSyncFSM* pFsm, void* pWriter, bool isApply) {
int32_t SnapshotStopWrite(struct SSyncFSM* pFsm, void* pWriter, bool isApply, SSnapshot *pSnapshot) {
char logBuf[256] = {0};
snprintf(logBuf, sizeof(logBuf), "==callback== ==SnapshotStopWrite== pFsm:%p, pWriter:%p, isApply:%d", pFsm, pWriter,
isApply);

2
source/libs/sync/test/syncSnapshotReceiverTest.cpp
View File

@ -30,7 +30,7 @@ int32_t SnapshotStopRead(struct SSyncFSM* pFsm, void* pReader) { return 0; }
int32_t SnapshotDoRead(struct SSyncFSM* pFsm, void* pReader, void** ppBuf, int32_t* len) { return 0; }
int32_t SnapshotStartWrite(struct SSyncFSM* pFsm, void* pParam, void** ppWriter) { return 0; }
int32_t SnapshotStopWrite(struct SSyncFSM* pFsm, void* pWriter, bool isApply) { return 0; }
int32_t SnapshotStopWrite(struct SSyncFSM* pFsm, void* pWriter, bool isApply, SSnapshot *pSnapshot) { return 0; }
int32_t SnapshotDoWrite(struct SSyncFSM* pFsm, void* pWriter, void* pBuf, int32_t len) { return 0; }
SSyncSnapshotReceiver* createReceiver() {

2
source/libs/sync/test/syncTestTool.cpp
View File

@ -126,7 +126,7 @@ int32_t SnapshotStartWrite(struct SSyncFSM* pFsm, void* pParam, void** ppWriter)
return 0;
}
int32_t SnapshotStopWrite(struct SSyncFSM* pFsm, void* pWriter, bool isApply) {
int32_t SnapshotStopWrite(struct SSyncFSM* pFsm, void* pWriter, bool isApply, SSnapshot *pSnapshot) {
if (isApply) {
gSnapshotLastApplyIndex = gFinishLastApplyIndex;
gSnapshotLastApplyTerm = gFinishLastApplyTerm;

1
tests/script/jenkins/basic.txt
View File

@ -311,6 +311,7 @@
./test.sh -f tsim/valgrind/checkError2.sim
./test.sh -f tsim/valgrind/checkError3.sim
./test.sh -f tsim/valgrind/checkError4.sim
./test.sh -f tsim/valgrind/checkError5.sim
# --- vnode
# unsupport ./test.sh -f tsim/vnode/replica3_basic.sim

2
tests/script/tsim/stable/alter_count.sim
View File

@ -249,9 +249,7 @@ endi
print ============== step18
system sh/exec.sh -n dnode1 -s stop -x SIGINT
sleep 3000
system sh/exec.sh -n dnode1 -s start
sleep 3000
#sql select count(g) from tb
#if $data00 != 12 then

146
tests/script/tsim/stable/disk.sim
View File

@ -77,12 +77,11 @@ if $data00 != $rowNum then
endi
print =============== step3
# TODO : where condition
# sql select count(tbcol) from $tb where ts <= 1519833840000
# print ===> $data00
# if $data00 != 5 then
# return -1
# endi
sql select count(tbcol) from $tb where ts <= 1519833840000
print ===> $data00
if $data00 != 5 then
return -1
endi
print =============== step4
sql select count(tbcol) as b from $tb
@ -105,89 +104,82 @@ if $data00 != $rowNum then
endi
print =============== step6
# TODO
# sql select count(tbcol) as b from $tb where ts <= 1519833840000 interval(1m)
# print ===> $data00
# if $data00 != 1 then
# return -1
# endi
# if $rows != 5 then
# return -1
# endi
sql select count(tbcol) as b from $tb where ts <= 1519833840000 interval(1m)
print ===> $data00
if $data00 != 1 then
return -1
endi
if $rows != 5 then
return -1
endi
print =============== step7
# TODO
# sql select count(*) from $mt
# print select count(*) from $mt ===> $data00
# if $data00 != $totalNum then
# return -1
# endi
sql select count(*) from $mt
print select count(*) from $mt ===> $data00
if $data00 != $totalNum then
return -1
endi
# TODO
# print ==========> block opt will cause this crash, table scan need to fix this during plan gen ===============>
#sql select count(tbcol) from $mt
#print ===> $data00
#if $data00 != $totalNum then
# return -1
#endi
print ==========> block opt will cause this crash, table scan need to fix this during plan gen ===============>
sql select count(tbcol) from $mt
print ===> $data00
if $data00 != $totalNum then
return -1
endi
print =============== step8
# TODO
# sql select count(tbcol) as c from $mt where ts <= 1519833840000
# print ===> $data00
# if $data00 != 50 then
# return -1
# endi
#
# sql select count(tbcol) as c from $mt where tgcol < 5
# print ===> $data00
# if $data00 != 100 then
# return -1
# endi
#
# sql select count(tbcol) as c from $mt where tgcol < 5 and ts <= 1519833840000
# print ===> $data00
# if $data00 != 25 then
# return -1
# endi
sql select count(tbcol) as c from $mt where ts <= 1519833840000
print ===> $data00
if $data00 != 50 then
return -1
endi
sql select count(tbcol) as c from $mt where tgcol < 5
print ===> $data00
if $data00 != 100 then
return -1
endi
sql select count(tbcol) as c from $mt where tgcol < 5 and ts <= 1519833840000
print ===> $data00
if $data00 != 25 then
return -1
endi
print =============== step9
# TODO : count from stable
# sql select count(tbcol) as b from $mt interval(1m)
# print select count(tbcol) as b from $mt interval(1m) ===> $data01
# if $data01 != 10 then
# return -1
# endi
sql select count(tbcol) as b from $mt interval(1m)
print select count(tbcol) as b from $mt interval(1m) ===> $data01
if $data00 != 10 then
return -1
endi
# sql select count(tbcol) as b from $mt interval(1d)
# print ===> $data02
# if $data01 != 200 then
# return -1
# endi
sql select count(tbcol) as b from $mt interval(1d)
print ===> $data02
if $data00 != 200 then
return -1
endi
print =============== step10
# TODO
# print select count(tbcol) as b from $mt group by tgcol
# sql select count(tbcol) as b from $mt group by tgcol
# print ===> $data00
# if $data00 != $rowNum then
# return -1
# endi
print select count(tbcol) as b from $mt group by tgcol
sql select count(tbcol) as b from $mt group by tgcol
print ===> $data00
if $data00 != $rowNum then
return -1
endi
if $rows != $tbNum then
return -1
endi
# if $rows != $tbNum then
# return -1
# endi
#
print =============== step11
# TODO : where condition
# sql select count(tbcol) as b from $mt where ts <= 1519833840000 interval(1m) group by tgcol
# print ===> $data01
# if $data01 != 1 then
# return -1
# endi
# if $rows != 50 then
# return -1
# endi
sql select count(tbcol) as b from $mt where ts <= 1519833840000 partition by tgcol interval(1m)
print ===> $data01
if $data00 != 1 then
return -1
endi
if $rows != 50 then
return -1
endi
print =============== clear
sql drop database $db

142
tests/script/tsim/stable/dnode3.sim
View File

@ -86,12 +86,11 @@ if $data00 != $rowNum then
endi
print =============== step3
# TODO : where condition
# sql select count(tbcol) from $tb where ts <= 1519833840000
# print ===> $data00
# if $data00 != 5 then
# return -1
# endi
sql select count(tbcol) from $tb where ts <= 1519833840000
print ===> $data00
if $data00 != 5 then
return -1
endi
print =============== step4
sql select count(tbcol) as b from $tb
@ -114,82 +113,81 @@ if $data00 != $rowNum then
endi
print =============== step6
# sql select count(tbcol) as b from $tb where ts <= 1519833840000 interval(1m)
# print ===> $data00
# if $data00 != 1 then
# return -1
# endi
# if $rows != 5 then
# return -1
# endi
sql select count(tbcol) as b from $tb where ts <= 1519833840000 interval(1m)
print ===> $data00
if $data00 != 1 then
return -1
endi
if $rows != 5 then
return -1
endi
print =============== step7
# print select count(*) from $mt
# sql select count(*) from $mt
# print ===> $data00
# if $data00 != $totalNum then
# return -1
# endi
#
# sql select count(tbcol) from $mt
# print ===> $data00
# if $data00 != $totalNum then
# return -1
# endi
print select count(*) from $mt
sql select count(*) from $mt
print ===> $data00
if $data00 != $totalNum then
return -1
endi
sql select count(tbcol) from $mt
print ===> $data00
if $data00 != $totalNum then
return -1
endi
print =============== step8
# sql select count(tbcol) as c from $mt where ts <= 1519833840000
# print ===> $data00
# if $data00 != 50 then
# return -1
# endi
#
# sql select count(tbcol) as c from $mt where tgcol < 5
# print ===> $data00
# if $data00 != 100 then
# return -1
# endi
#
# sql select count(tbcol) as c from $mt where tgcol < 5 and ts <= 1519833840000
# print ===> $data00
# if $data00 != 25 then
# return -1
# endi
sql select count(tbcol) as c from $mt where ts <= 1519833840000
print ===> $data00
if $data00 != 50 then
return -1
endi
sql select count(tbcol) as c from $mt where tgcol < 5
print ===> $data00
if $data00 != 100 then
return -1
endi
sql select count(tbcol) as c from $mt where tgcol < 5 and ts <= 1519833840000
print ===> $data00
if $data00 != 25 then
return -1
endi
print =============== step9
# TODO : group by in stable
# sql select count(tbcol) as b from $mt interval(1m)
# print ===> $data00
# if $data00 != 10 then
# return -1
# endi
#
# sql select count(tbcol) as b from $mt interval(1d)
# print ===> $data00
# if $data00 != 200 then
# return -1
# endi
sql select count(tbcol) as b from $mt interval(1m)
print ===> $data00
if $data00 != 10 then
return -1
endi
sql select count(tbcol) as b from $mt interval(1d)
print ===> $data00
if $data00 != 200 then
return -1
endi
print =============== step10
# sql select count(tbcol) as b from $mt group by tgcol
# print ===> $data00
# if $data00 != $rowNum then
# return -1
# endi
#
# if $rows != $tbNum then
# return -1
# endi
sql select count(tbcol) as b from $mt group by tgcol
print ===> $data00
if $data00 != $rowNum then
return -1
endi
if $rows != $tbNum then
return -1
endi
print =============== step11
# sql select count(tbcol) as b from $mt where ts <= 1519833840000 interval(1m) group by tgcol
# print ===> $data00
# if $data00 != 1 then
# return -1
# endi
# if $rows != 50 then
# return -1
# endi
sql select count(tbcol) as b from $mt where ts <= 1519833840000 partition by tgcol interval(1m)
print ===> $data00
if $data00 != 1 then
return -1
endi
if $rows != 50 then
return -1
endi
print =============== clear
sql drop database $db

67
tests/script/tsim/stable/metrics.sim
View File

@ -24,17 +24,15 @@ if $rows != 1 then
return -1
endi
#TODO OPEN THIS WHEN STABLE DELETE WORKS
#print =============== step2
#sql drop table $mt
#sql show stables
#if $rows != 0 then
# return -1
#endi
print =============== step2
sql drop table $mt
sql show stables
if $rows != 0 then
return -1
endi
#print =============== step3
#sql create table $mt (ts timestamp, speed int) TAGS(sp int)
#TODO OPEN THIS WHEN STABLE DELETE WORKS
print =============== step3
sql create table $mt (ts timestamp, speed int) TAGS(sp int)
sql show stables
if $rows != 1 then
@ -46,11 +44,10 @@ endi
if $data04 != 1 then
return -1
endi
# TODO : select * from stable
# sql select * from $mt
# if $rows != 0 then
# return -1
# endi
sql select * from $mt
if $rows != 0 then
return -1
endi
print =============== step4
$i = 0
@ -67,9 +64,9 @@ sql show tables
if $rows != 3 then
return -1
endi
# if $data03 != $mt then
# return -1
# endi
if $data04 != $mt then
return -1
endi
sql show stables
if $rows != 1 then
@ -78,9 +75,9 @@ endi
if $data00 != $mt then
return -1
endi
# if $data04 != 3 then
# return -1
# endi
if $data03 != 2 then
return -1
endi
print =============== step5
$i = 0
@ -95,28 +92,28 @@ sql insert into $tb values (now + 1m , 1 )
print =============== step6
# sql select * from $mt
# print select * from $mt ==> $rows $data00
# if $rows != 3 then
# return -1
# endi
sql select * from $mt
print select * from $mt ==> $rows $data00
if $rows != 3 then
return -1
endi
print =============== step7
# sql select * from $mt where sp = 1
# print select * from $mt where sp = 1 ==> $rows $data00
# if $rows != 1 then
# return -1
# endi
sql select * from $mt where sp = 1
print select * from $mt where sp = 1 ==> $rows $data00
if $rows != 1 then
return -1
endi
print =============== step8
sql drop table $mt
print =============== step9
#sql show tables
#if $rows != 0 then
# return -1
#endi
sql show tables
if $rows != 0 then
return -1
endi
sql show stables
if $rows != 0 then

48
tests/script/tsim/stable/refcount.sim
View File

@ -11,7 +11,7 @@ sql create table d1.t2 (ts timestamp, i int);
sql create table d1.t3 (ts timestamp, i int);
sql insert into d1.t1 values(now, 1);
sql insert into d1.t2 values(now, 1);
# sql drop table d1.t1;
sql drop table d1.t1;
sql drop database d1;
sql show databases;
@ -27,14 +27,14 @@ sql create table d2.t2 (ts timestamp, i int);
sql create table d2.t3 (ts timestamp, i int);
sql insert into d2.t1 values(now, 1);
sql insert into d2.t2 values(now, 1);
# sql drop table d2.t1;
# sql drop table d2.t2;
# sql drop table d2.t3;
#
# sql show d2.tables;
# if $rows != 0 then
# return -1
# endi
sql drop table d2.t1;
sql drop table d2.t2;
sql drop table d2.t3;
sql show d2.tables;
if $rows != 0 then
return -1
endi
sql show d2.vgroups;
if $rows != 2 then
@ -56,14 +56,14 @@ sql create table d3.t1 using d3.st tags(1);
sql create table d3.t2 using d3.st tags(1);
sql create table d3.t3 using d3.st tags(1);
sql insert into d3.t1 values(now, 1);
# sql drop table d3.t1;
# sql drop table d3.t2;
# sql drop table d3.t3;
#
# sql show d3.tables;
# if $rows != 0 then
# return -1
# endi
sql drop table d3.t1;
sql drop table d3.t2;
sql drop table d3.t3;
sql show d3.tables;
if $rows != 0 then
return -1
endi
sql show d3.vgroups;
if $rows != 2 then
@ -85,13 +85,13 @@ sql create table d4.t1 using d4.st tags(1);
sql create table d4.t2 using d4.st tags(1);
sql create table d4.t3 using d4.st tags(1);
sql insert into d4.t1 values(now, 1);
# sql drop table d4.t1;
sql drop table d4.t1;
sql drop table d4.st;
#
# sql show d4.tables;
# if $rows != 0 then
# return -1
# endi
sql show d4.tables;
if $rows != 0 then
return -1
endi
sql show d4.stables;
if $rows != 0 then
@ -113,7 +113,7 @@ sql create table d5.t1 using d5.st tags(1);
sql create table d5.t2 using d5.st tags(1);
sql create table d5.t3 using d5.st tags(1);
sql insert into d5.t1 values(now, 1);
# sql drop table d5.t1;
sql drop table d5.t1;
sql drop database d5;

1
tests/script/tsim/stable/tag_filter.sim
View File

@ -31,7 +31,6 @@ if $rows != 1 then
return -1
endi
sql select * from db.stb where t1 < 1
if $rows != 0 then
return -=1

94
tests/script/tsim/stable/values.sim
View File

@ -12,22 +12,22 @@ sql create table vdb0.vtb01 using vdb0.mt tags( 0 )
sql create database vdb1
sql create table vdb1.mt (ts timestamp, tbcol int) TAGS(tgcol int)
# sql_error create table vdb1.vtb10 using vdb0.mt tags( 1 )
# sql_error create table vdb1.vtb11 using vdb0.mt tags( 1 )
sql_error create table vdb1.vtb10 using vdb0.mt tags( 1 )
sql_error create table vdb1.vtb11 using vdb0.mt tags( 1 )
sql create table vdb1.vtb10 using vdb1.mt tags( 1 )
sql create table vdb1.vtb11 using vdb1.mt tags( 1 )
sql create database vdb2
sql create table vdb2.mt (ts timestamp, tbcol int) TAGS(tgcol int)
# sql_error create table vdb2.vtb20 using vdb0.mt tags( 2 )
# sql_error create table vdb2.vtb21 using vdb0.mt tags( 2 )
sql_error create table vdb2.vtb20 using vdb0.mt tags( 2 )
sql_error create table vdb2.vtb21 using vdb0.mt tags( 2 )
sql create table vdb2.vtb20 using vdb2.mt tags( 2 )
sql create table vdb2.vtb21 using vdb2.mt tags( 2 )
sql create database vdb3
sql create table vdb3.mt (ts timestamp, tbcol int) TAGS(tgcol int)
# sql_error create table vdb3.vtb20 using vdb0.mt tags( 2 )
# sql_error create table vdb3.vtb21 using vdb0.mt tags( 2 )
sql_error create table vdb3.vtb20 using vdb0.mt tags( 2 )
sql_error create table vdb3.vtb21 using vdb0.mt tags( 2 )
sql create table vdb3.vtb30 using vdb3.mt tags( 3 )
sql create table vdb3.vtb31 using vdb3.mt tags( 3 )
@ -40,7 +40,7 @@ sql insert into vdb2.vtb20 values (1519833600000 , 12) (1519833600001, 22) (1519
sql insert into vdb2.vtb21 values (1519833600000 , 12) (1519833600001, 22) (1519833600002, 32)
sql insert into vdb3.vtb30 values (1519833600000 , 13) (1519833600001, 23) (1519833600002, 33)
sql insert into vdb3.vtb31 values (1519833600000 , 13) (1519833600001, 23) (1519833600002, 33)
# sql select * from vdb0.mt
sql select * from vdb0.mt
sql select ts from vdb0.mt
if $rows != 6 then
@ -56,8 +56,7 @@ sql insert into vdb2.vtb20 values (1519833600003 , 42) (1519833600005, 52) (1519
sql insert into vdb2.vtb21 values (1519833600003 , 42) (1519833600005, 52) (1519833600004, 62)
sql insert into vdb3.vtb30 values (1519833600003 , 43) (1519833600005, 53) (1519833600004, 63)
sql insert into vdb3.vtb31 values (1519833600003 , 43) (1519833600005, 53) (1519833600004, 63)
# TODO : select * from stable
# sql select * from vdb0.mt
sql select * from vdb0.mt
sql select ts from vdb0.mt
if $rows != 12 then
@ -65,50 +64,49 @@ if $rows != 12 then
endi
print =============== step4
# TODO : insert into diffrent table
# sql insert into vdb0.vtb00 values(1519833600006, 60) (1519833600007, 70) vdb0.vtb01 values(1519833600006, 60) (1519833600007, 70)
# sql insert into vdb1.vtb10 values(1519833600006, 61) (1519833600007, 71) vdb1.vtb11 values(1519833600006, 61) (1519833600007, 71)
# sql insert into vdb2.vtb20 values(1519833600006, 62) (1519833600007, 72) vdb2.vtb21 values(1519833600006, 62) (1519833600007, 72)
# sql insert into vdb3.vtb30 values(1519833600006, 63) (1519833600007, 73) vdb3.vtb31 values(1519833600006, 63) (1519833600007, 73)
# # sql select * from vdb0.mt
# sql select ts from vdb0.mt
#
# if $rows != 16 then
# return -1
# endi
sql insert into vdb0.vtb00 values(1519833600006, 60) (1519833600007, 70) vdb0.vtb01 values(1519833600006, 60) (1519833600007, 70)
sql insert into vdb1.vtb10 values(1519833600006, 61) (1519833600007, 71) vdb1.vtb11 values(1519833600006, 61) (1519833600007, 71)
sql insert into vdb2.vtb20 values(1519833600006, 62) (1519833600007, 72) vdb2.vtb21 values(1519833600006, 62) (1519833600007, 72)
sql insert into vdb3.vtb30 values(1519833600006, 63) (1519833600007, 73) vdb3.vtb31 values(1519833600006, 63) (1519833600007, 73)
sql select * from vdb0.mt
sql select ts from vdb0.mt
if $rows != 16 then
return -1
endi
print =============== step5
# sql insert into vdb0.vtb00 values(1519833600008, 80) (1519833600007, 70) vdb0.vtb01 values(1519833600006, 80) (1519833600007, 70)
# sql insert into vdb1.vtb10 values(1519833600008, 81) (1519833600007, 71) vdb1.vtb11 values(1519833600006, 81) (1519833600007, 71)
# sql insert into vdb2.vtb20 values(1519833600008, 82) (1519833600007, 72) vdb2.vtb21 values(1519833600006, 82) (1519833600007, 72)
# sql insert into vdb3.vtb30 values(1519833600008, 83) (1519833600007, 73) vdb3.vtb31 values(1519833600006, 83) (1519833600007, 73)
# # sql select * from vdb0.mt
# sql select ts from vdb0.mt
#
# if $rows != 17 then
# return -1
# endi
sql insert into vdb0.vtb00 values(1519833600008, 80) (1519833600007, 70) vdb0.vtb01 values(1519833600006, 80) (1519833600007, 70)
sql insert into vdb1.vtb10 values(1519833600008, 81) (1519833600007, 71) vdb1.vtb11 values(1519833600006, 81) (1519833600007, 71)
sql insert into vdb2.vtb20 values(1519833600008, 82) (1519833600007, 72) vdb2.vtb21 values(1519833600006, 82) (1519833600007, 72)
sql insert into vdb3.vtb30 values(1519833600008, 83) (1519833600007, 73) vdb3.vtb31 values(1519833600006, 83) (1519833600007, 73)
sql select * from vdb0.mt
sql select ts from vdb0.mt
if $rows != 17 then
return -1
endi
print =============== step6
# sql insert into vdb0.vtb00 values(1519833600009, 90) (1519833600010, 100) vdb1.vtb10 values(1519833600009, 90) (1519833600010, 100) vdb2.vtb20 values(1519833600009, 90) (1519833600010, 100) vdb3.vtb30 values(1519833600009, 90) (1519833600010, 100)
# sql insert into vdb0.vtb01 values(1519833600009, 90) (1519833600010, 100) vdb1.vtb11 values(1519833600009, 90) (1519833600010, 100) vdb2.vtb21 values(1519833600009, 90) (1519833600010, 100) vdb3.vtb31 values(1519833600009, 90) (1519833600010, 100)
#
# # sql select * from vdb0.mt
# sql select ts from vdb0.mt
#
# if $rows != 21 then
# return -1
# endi
sql insert into vdb0.vtb00 values(1519833600009, 90) (1519833600010, 100) vdb1.vtb10 values(1519833600009, 90) (1519833600010, 100) vdb2.vtb20 values(1519833600009, 90) (1519833600010, 100) vdb3.vtb30 values(1519833600009, 90) (1519833600010, 100)
sql insert into vdb0.vtb01 values(1519833600009, 90) (1519833600010, 100) vdb1.vtb11 values(1519833600009, 90) (1519833600010, 100) vdb2.vtb21 values(1519833600009, 90) (1519833600010, 100) vdb3.vtb31 values(1519833600009, 90) (1519833600010, 100)
sql select * from vdb0.mt
sql select ts from vdb0.mt
if $rows != 21 then
return -1
endi
print =============== step7
# sql insert into vdb0.vtb00 values(1519833600012, 120) (1519833600011, 110) vdb1.vtb10 values(1519833600012, 120) (1519833600011, 110) vdb2.vtb20 values(1519833600012, 120) (1519833600011, 110) vdb3.vtb30 values(1519833600012, 120) (1519833600011, 110)
# sql insert into vdb0.vtb01 values(1519833600012, 120) (1519833600011, 110) vdb1.vtb11 values(1519833600012, 120) (1519833600011, 110) vdb2.vtb21 values(1519833600012, 120) (1519833600011, 110) vdb3.vtb31 values(1519833600012, 120) (1519833600011, 110)
#
# # sql select * from vdb0.mt
# sql select ts from vdb0.mt
#
# if $rows != 25 then
# return -1
# endi
sql insert into vdb0.vtb00 values(1519833600012, 120) (1519833600011, 110) vdb1.vtb10 values(1519833600012, 120) (1519833600011, 110) vdb2.vtb20 values(1519833600012, 120) (1519833600011, 110) vdb3.vtb30 values(1519833600012, 120) (1519833600011, 110)
sql insert into vdb0.vtb01 values(1519833600012, 120) (1519833600011, 110) vdb1.vtb11 values(1519833600012, 120) (1519833600011, 110) vdb2.vtb21 values(1519833600012, 120) (1519833600011, 110) vdb3.vtb31 values(1519833600012, 120) (1519833600011, 110)
sql select * from vdb0.mt
sql select ts from vdb0.mt
if $rows != 25 then
return -1
endi
system sh/exec.sh -n dnode1 -s stop -x SIGINT

136
tests/script/tsim/stable/vnode3.sim
View File

@ -60,12 +60,11 @@ if $data00 != $rowNum then
endi
print =============== step3
# TODO : where condition
# sql select count(tbcol) from $tb where ts <= 1519833840000
# print ===> $data00
# if $data00 != 5 then
# return -1
# endi
sql select count(tbcol) from $tb where ts <= 1519833840000
print ===> $data00
if $data00 != 5 then
return -1
endi
print =============== step4
sql select count(tbcol) as b from $tb
@ -88,81 +87,80 @@ if $data00 != $rowNum then
endi
print =============== step6
# sql select count(tbcol) as b from $tb where ts <= 1519833840000 interval(1m)
# print ===> $data00
# if $data00 != 1 then
# return -1
# endi
# if $rows != 5 then
# return -1
#endi
sql select count(tbcol) as b from $tb where ts <= 1519833840000 interval(1m)
print ===> $data00
if $data00 != 1 then
return -1
endi
if $rows != 5 then
return -1
endi
print =============== step7
# TODO : count(*) err
# sql select count(*) from $mt
# print ===> $data00
# if $data00 != $totalNum then
# return -1
# endi
#
# sql select count(tbcol) from $mt
# print ===> $data00
# if $data00 != $totalNum then
# return -1
# endi
sql select count(*) from $mt
print ===> $data00
if $data00 != $totalNum then
return -1
endi
sql select count(tbcol) from $mt
print ===> $data00
if $data00 != $totalNum then
return -1
endi
print =============== step8
# sql select count(tbcol) as c from $mt where ts <= 1519833840000
# print ===> $data00
# if $data00 != 50 then
# return -1
# endi
sql select count(tbcol) as c from $mt where ts <= 1519833840000
print ===> $data00
if $data00 != 50 then
return -1
endi
# sql select count(tbcol) as c from $mt where tgcol < 5
# print ===> $data00
# if $data00 != 100 then
# return -1
# endi
sql select count(tbcol) as c from $mt where tgcol < 5
print ===> $data00
if $data00 != 100 then
return -1
endi
# sql select count(tbcol) as c from $mt where tgcol < 5 and ts <= 1519833840000
# print ===> $data00
# if $data00 != 25 then
# return -1
# endi
sql select count(tbcol) as c from $mt where tgcol < 5 and ts <= 1519833840000
print ===> $data00
if $data00 != 25 then
return -1
endi
print =============== step9
# sql select count(tbcol) as b from $mt interval(1m)
# print ===> $data00
# if $data00 != 10 then
# return -1
# endi
#
# sql select count(tbcol) as b from $mt interval(1d)
# print ===> $data00
# if $data00 != 200 then
# return -1
# endi
sql select count(tbcol) as b from $mt interval(1m)
print ===> $data00
if $data00 != 10 then
return -1
endi
sql select count(tbcol) as b from $mt interval(1d)
print ===> $data00
if $data00 != 200 then
return -1
endi
print =============== step10
# sql select count(tbcol) as b from $mt group by tgcol
# print ===> $data00
# if $data00 != $rowNum then
# return -1
# endi
#
# if $rows != $tbNum then
# return -1
# endi
sql select count(tbcol) as b from $mt group by tgcol
print ===> $data00
if $data00 != $rowNum then
return -1
endi
if $rows != $tbNum then
return -1
endi
print =============== step11
# sql select count(tbcol) as b from $mt where ts <= 1519833840000 interval(1m) group by tgcol
# print ===> $data01
# if $data01 != 1 then
# return -1
# endi
# if $rows != 50 then
# return -1
# endi
sql select count(tbcol) as b from $mt where ts <= 1519833840000 partition by tgcol interval(1m)
print ===> $data00
if $data00 != 1 then
return -1
endi
if $rows != 50 then
return -1
endi
print =============== clear
sql drop database $db

129
tests/script/tsim/valgrind/basic1.sim
View File

@ -1,50 +1,99 @@
system sh/stop_dnodes.sh
system sh/deploy.sh -n dnode1 -i 1
system sh/cfg.sh -n dnode1 -c debugflag -v 131
system sh/exec.sh -n dnode1 -s start -v
sql connect
print ======== step1
sql drop database if exists db1;
sql create database db1 vgroups 3;
sql use db1;
sql create stable st1 (ts timestamp, f1 int, f2 binary(200)) tags(t1 int);
sql create table tb1 using st1 tags(1);
sql insert into tb1 values ('2022-07-07 10:01:01', 11, "aaa");
sql insert into tb1 values ('2022-07-07 11:01:02', 12, "bbb");
sql create table tb2 using st1 tags(2);
sql insert into tb2 values ('2022-07-07 10:02:01', 21, "aaa");
sql insert into tb2 values ('2022-07-07 11:02:02', 22, "bbb");
sql create table tb3 using st1 tags(3);
sql insert into tb3 values ('2022-07-07 10:03:01', 31, "aaa");
sql insert into tb3 values ('2022-07-07 11:03:02', 32, "bbb");
sql create table tb4 using st1 tags(4);
print =============== step1: create drop show dnodes
$x = 0
step1:
$x = $x + 1
sleep 1000
if $x == 10 then
print ---> dnode not ready!
return -1
endi
sql show dnodes
print ---> $data00 $data01 $data02 $data03 $data04 $data05
if $rows != 1 then
return -1
endi
if $data(1)[4] != ready then
goto step1
endi
sql insert into tb4 select * from tb1;
print =============== step2: create db
sql create database db
sql use db
sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 float, t3 binary(16)) comment "abd"
sql create table db.c1 using db.stb tags(101, 102, "103")
goto _OVER
print =============== step3: alter stb
sql_error alter table db.stb add column ts int
sql alter table db.stb add column c3 int
sql alter table db.stb add column c4 bigint
sql alter table db.stb add column c5 binary(12)
sql alter table db.stb drop column c1
sql alter table db.stb drop column c4
sql alter table db.stb MODIFY column c2 binary(32)
sql alter table db.stb add tag t4 bigint
sql alter table db.stb add tag c1 int
sql alter table db.stb add tag t5 binary(12)
sql alter table db.stb drop tag c1
sql alter table db.stb drop tag t5
sql alter table db.stb MODIFY tag t3 binary(32)
sql alter table db.stb rename tag t1 tx
sql alter table db.stb comment 'abcde' ;
sql drop table db.stb
sql select * from tb4;
if $rows != 2 then
return -1
endi
sql insert into tb4 select ts,f1,f2 from st1;
sql select * from tb4;
if $rows != 6 then
return -1
endi
sql create table tba (ts timestamp, f1 binary(10), f2 bigint, f3 double);
sql_error insert into tba select * from tb1;
sql insert into tba (ts,f2,f1) select * from tb1;
sql select * from tba;
if $rows != 2 then
return -1
endi
sql create table tbb (ts timestamp, f1 binary(10), f2 bigint, f3 double);
sql insert into tbb (f2,f1,ts) select f1+1,f2,ts+3 from tb2;
sql select * from tbb;
if $rows != 2 then
return -1
endi
print =============== step4: alter tb
sql create table tb (ts timestamp, a int)
sql insert into tb values(now-28d, -28)
sql select count(a) from tb
sql alter table tb add column b smallint
sql insert into tb values(now-25d, -25, 0)
sql select count(b) from tb
sql alter table tb add column c tinyint
sql insert into tb values(now-22d, -22, 3, 0)
sql select count(c) from tb
sql alter table tb add column d int
sql insert into tb values(now-19d, -19, 6, 0, 0)
sql select count(d) from tb
sql alter table tb add column e bigint
sql alter table tb add column f float
sql alter table tb add column g double
sql alter table tb add column h binary(10)
sql select count(a), count(b), count(c), count(d), count(e), count(f), count(g), count(h) from tb
sql select * from tb order by ts desc
print =============== step5: alter stb and insert data
sql create table stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 float, t3 binary(16)) comment "abd"
sql show db.stables
sql describe stb
sql_error alter table stb add column ts int
sql create table db.ctb using db.stb tags(101, 102, "103")
sql insert into db.ctb values(now, 1, "2")
sql show db.tables
sql select * from db.stb
sql select * from tb
sql alter table stb add column c3 int
sql describe stb
sql select * from db.stb
sql select * from tb
sql insert into db.ctb values(now+1s, 1, 2, 3)
sql select * from db.stb
sql alter table db.stb add column c4 bigint
sql select * from db.stb
sql insert into db.ctb values(now+2s, 1, 2, 3, 4)
sql alter table db.stb drop column c1
sql reset query cache
sql select * from tb
sql insert into db.ctb values(now+3s, 2, 3, 4)
sql select * from db.stb
_OVER:
system sh/exec.sh -n dnode1 -s stop -x SIGINT
@ -53,7 +102,7 @@ $null=
system_content sh/checkValgrind.sh -n dnode1
print cmd return result ----> [ $system_content ]
if $system_content > 1 then
if $system_content > 0 then
return -1
endi

20
tests/script/tsim/valgrind/checkError2.sim
View File

@ -78,18 +78,18 @@ sql select max(c1), max(c2), max(c3) from ct1
sql select sum(c1), sum(c2), sum(c3) from ct1
print =============== step9: insert select
#sql create table ct4 using stb tags(4000);
#sql insert into ct4 select * from ct1;
#sql select * from ct4;
#sql insert into ct4 select ts,c1,c2,c3 from stb;
sql create table ct4 using stb tags(4000);
sql insert into ct4 select * from ct1;
sql select * from ct4;
sql insert into ct4 select ts,c1,c2,c3 from stb;
#sql create table tb1 (ts timestamp, c1 int, c2 float, c3 double);
#sql insert into tb1 (ts, c1, c2, c3) select * from ct1;
#sql select * from tb1;
sql create table tb1 (ts timestamp, c1 int, c2 float, c3 double);
sql insert into tb1 (ts, c1, c2, c3) select * from ct1;
sql select * from tb1;
#sql create table tb2 (ts timestamp, f1 binary(10), c1 int, c2 double);
#sql insert into tb2 (c2, c1, ts) select c2+1, c1, ts+3 from ct2;
#sql select * from tb2;
sql create table tb2 (ts timestamp, f1 binary(10), c1 int, c2 double);
sql insert into tb2 (c2, c1, ts) select c2+1, c1, ts+3 from ct2;
sql select * from tb2;
_OVER:
system sh/exec.sh -n dnode1 -s stop -x SIGINT

39
tests/system-test/2-query/json_tag_large_tables.py
View File

@ -43,13 +43,48 @@ class TDTestCase:
tdSql.execute('create database db vgroups 1')
tdSql.execute('use db')
print("============== STEP 1 ===== prepare data & validate json string")
i = 0
# add 100000 table
tdSql.execute("create table if not exists jsons1(ts timestamp, dataInt int, dataBool bool, dataStr nchar(50), dataStrBin binary(150)) tags(jtag json)")
while i <= 10 0000:
sql = """insert into jsons1_{%d} using jsons1 tags('{"tag1":{%d}}') values(1591060618000, 1, false, 'json1', '你是') (1591060608000, 23, true, '等等', 'json')"""%(i, i)
tdSql.execute(sql)
i = i + 1
// do query
i = 0
while i <= 10 0000:
sql = """select count(*) from jsons1 where jtag->'tag1' = %d"""%(i)
tdSql.query(sql)
if 1 != tdSql.getRows():
print("err: %s"%(sql))
while i <= 10000000
sql = """insert into jsons1_{%d} using jsons1 tags('{"tag1":{%d}}') values(1591060618000, 1, false, 'json1', '你是') (1591060608000, 23, true, '等等', 'json')"""%(i, i)
tdSql.execute(sql)
i = i + 1
i = 0
# drop super table
tdSql.execute("create table if not exists jsons1(ts timestamp, dataInt int, dataBool bool, dataStr nchar(50), dataStrBin binary(150)) tags(jtag json)")
while i <= 100000:
sql = """insert into jsons1_{%d} using jsons1 tags('{"tag1":{%d}}') values(1591060618000, 1, false, 'json1', '你是') (1591060608000, 23, true, '等等', 'json')"""%(i, i)
tdSql.execute(sql)
i = i + 1
tdSql.execute('drop stable jsons1')
# drop database
i = 0
tdSql.execute("create table if not exists jsons1(ts timestamp, dataInt int, dataBool bool, dataStr nchar(50), dataStrBin binary(150)) tags(jtag json)")
while i <= 100000:
f = "insert into jsons1_{} using jsons1 tags('{\"tag1\":\"fff\",\"tag2\":{}, \"tag3\":true}') values(1591060618000, 1, false, 'json1', '你是') (1591060608000, 23, true, '等等', 'json')".format
sql = f(i, i)
sql = """insert into jsons1_{%d} using jsons1 tags('{"tag1":{%d}}') values(1591060618000, 1, false, 'json1', '你是') (1591060608000, 23, true, '等等', 'json')"""%(i, i)
tdSql.execute(sql)
i = i + 1
tdSql.execute('drop database db')
# test duplicate key using the first one. elimate empty key
#tdSql.execute("CREATE TABLE if not exists jsons1_8 using jsons1 tags('{\"tag1\":null, \"tag1\":true, \"tag1\":45, \"1tag$\":2, \" \":90, \"\":32}')") tdSql.query("select jtag from jsons1_8") tdSql.checkRows(0);

4
tests/test/c/tmqDemo.c
View File

@ -388,13 +388,11 @@ void perf_loop(tmq_t* tmq, tmq_list_t* topics, int32_t totalMsgs, int64_t walLog
}
/*taosSsleep(3);*/
int32_t batchCnt = 0;
int32_t skipLogNum = 0;
int64_t startTime = taosGetTimestampUs();
while (running) {
TAOS_RES* tmqmessage = tmq_consumer_poll(tmq, 3000);
if (tmqmessage) {
batchCnt++;
/*skipLogNum += tmqGetSkipLogNum(tmqmessage);*/
if (0 != g_stConfInfo.showMsgFlag) {
/*msg_process(tmqmessage);*/
}
@ -412,7 +410,7 @@ void perf_loop(tmq_t* tmq, tmq_list_t* topics, int32_t totalMsgs, int64_t walLog
}
if (0 == g_stConfInfo.simCase) {
printf("consume result: msgs: %d, skip log cnt: %d, time used:%.3f second\n", batchCnt, skipLogNum, consumeTime);
printf("consume result: msgs: %d, time used:%.3f second\n", batchCnt, consumeTime);
} else {
printf("{consume success: %d}", totalMsgs);
}

523
tests/test/c/tmqSim.c
View File

@ -20,6 +20,7 @@
#include <sys/stat.h>
#include <sys/types.h>
#include <time.h>
#include <math.h>
#include "taos.h"
#include "taosdef.h"
@ -35,6 +36,8 @@
#define MAX_ROW_STR_LEN (16 * 1024)
#define MAX_CONSUMER_THREAD_CNT (16)
#define MAX_VGROUP_CNT (32)
#define SEND_TIME_UNIT 10 // ms
#define MAX_SQL_LEN 1048576
typedef enum {
NOTIFY_CMD_START_CONSUM,
@ -42,6 +45,12 @@ typedef enum {
NOTIFY_CMD_ID_BUTT,
} NOTIFY_CMD_ID;
typedef enum enumQUERY_TYPE {
NO_INSERT_TYPE,
INSERT_TYPE,
QUERY_TYPE_BUT
} QUERY_TYPE;
typedef struct {
TdThread thread;
int32_t consumerId;
@ -58,6 +67,7 @@ typedef struct {
int64_t consumeMsgCnt;
int64_t consumeRowCnt;
int64_t consumeLen;
int32_t checkresult;
char topicString[1024];
@ -79,12 +89,17 @@ typedef struct {
TAOS* taos;
// below parameters is used by omb test
int32_t producerRate; // unit: msgs/s
int64_t totalProduceMsgs;
int64_t totalMsgsLen;
} SThreadInfo;
typedef struct {
// input from argvs
char cdbName[32];
char dbName[32];
char dbName[64];
int32_t showMsgFlag;
int32_t showRowFlag;
int32_t saveRowFlag;
@ -93,11 +108,22 @@ typedef struct {
int32_t useSnapshot;
int64_t nowTime;
SThreadInfo stThreads[MAX_CONSUMER_THREAD_CNT];
SThreadInfo stProdThreads[MAX_CONSUMER_THREAD_CNT];
// below parameters is used by omb test
char topic[64];
int32_t producers;
int32_t producerRate;
int32_t runDurationMinutes;
int32_t batchSize;
int32_t payloadLen;
} SConfInfo;
static SConfInfo g_stConfInfo;
TdFilePtr g_fp = NULL;
static int running = 1;
char* g_payload = NULL;
// char* g_pRowValue = NULL;
// TdFilePtr g_fp = NULL;
@ -117,7 +143,29 @@ static void printHelp() {
printf("%s%s\n", indent, "-s");
printf("%s%s%s%d\n", indent, indent, "saveRowFlag, default is ", g_stConfInfo.saveRowFlag);
printf("%s%s\n", indent, "-y");
printf("%s%s%s%d\n", indent, indent, "consume delay, default is s", g_stConfInfo.consumeDelay);
printf("%s%s%s%ds\n", indent, indent, "consume delay, default is ", g_stConfInfo.consumeDelay);
printf("%s%s\n", indent, "-e");
printf("%s%s%s%d\n", indent, indent, "snapshot, default is ", g_stConfInfo.useSnapshot);
printf("%s%s\n", indent, "-t");
printf("%s%s%s\n", indent, indent, "topic name, default is null");
printf("%s%s\n", indent, "-x");
printf("%s%s%s\n", indent, indent, "consume thread number, default is 1");
printf("%s%s\n", indent, "-l");
printf("%s%s%s\n", indent, indent, "run duration unit is minutes, default is ", g_stConfInfo.runDurationMinutes);
printf("%s%s\n", indent, "-p");
printf("%s%s%s\n", indent, indent, "producer thread number, default is 0");
printf("%s%s\n", indent, "-b");
printf("%s%s%s\n", indent, indent, "batch size, default is 1");
printf("%s%s\n", indent, "-i");
printf("%s%s%s\n", indent, indent, "produce rate unit is msgs /s, default is 100000");
printf("%s%s\n", indent, "-n");
printf("%s%s%s\n", indent, indent, "payload len unit is byte, default is 1000");
exit(EXIT_SUCCESS);
}
@ -144,7 +192,11 @@ void initLogFile() {
pid_t process_id = getpid();
sprintf(filename, "%s/../log/tmqlog-%d-%s.txt", configDir, process_id, getCurrentTimeString(tmpString));
if (0 != strlen(g_stConfInfo.topic)) {
sprintf(filename, "/tmp/tmqlog-%d-%s.txt", process_id, getCurrentTimeString(tmpString));
} else {
sprintf(filename, "%s/../log/tmqlog-%d-%s.txt", configDir, process_id, getCurrentTimeString(tmpString));
}
#ifdef WINDOWS
for (int i = 2; i < sizeof(filename); i++) {
if (filename[i] == ':') filename[i] = '-';
@ -199,6 +251,9 @@ void parseArgument(int32_t argc, char* argv[]) {
g_stConfInfo.showRowFlag = 0;
g_stConfInfo.saveRowFlag = 0;
g_stConfInfo.consumeDelay = 5;
g_stConfInfo.numOfThread = 1;
g_stConfInfo.batchSize = 1;
g_stConfInfo.producers = 0;
g_stConfInfo.nowTime = taosGetTimestampMs();
@ -222,12 +277,38 @@ void parseArgument(int32_t argc, char* argv[]) {
g_stConfInfo.consumeDelay = atol(argv[++i]);
} else if (strcmp(argv[i], "-e") == 0) {
g_stConfInfo.useSnapshot = atol(argv[++i]);
} else if (strcmp(argv[i], "-t") == 0) {
char tmpBuf[56];
strcpy(tmpBuf, argv[++i]);
sprintf(g_stConfInfo.topic, "`%s`", tmpBuf);
} else if (strcmp(argv[i], "-x") == 0) {
g_stConfInfo.numOfThread = atol(argv[++i]);
} else if (strcmp(argv[i], "-l") == 0) {
g_stConfInfo.runDurationMinutes = atol(argv[++i]);
} else if (strcmp(argv[i], "-p") == 0) {
g_stConfInfo.producers = atol(argv[++i]);
} else if (strcmp(argv[i], "-b") == 0) {
g_stConfInfo.batchSize = atol(argv[++i]);
} else if (strcmp(argv[i], "-i") == 0) {
g_stConfInfo.producerRate = atol(argv[++i]);
} else if (strcmp(argv[i], "-n") == 0) {
g_stConfInfo.payloadLen = atol(argv[++i]);
} else {
pError("%s unknow para: %s %s", GREEN, argv[++i], NC);
exit(-1);
}
}
g_payload = taosMemoryCalloc(g_stConfInfo.payloadLen + 1, 1);
if (NULL == g_payload) {
pPrint("%s failed to malloc for payload %s", GREEN, NC);
exit(-1);
}
for (int32_t i = 0; i < g_stConfInfo.payloadLen; i++) {
strcpy(&g_payload[i], "a");
}
initLogFile();
taosFprintfFile(g_fp, "====parseArgument() success\n");
@ -240,6 +321,11 @@ void parseArgument(int32_t argc, char* argv[]) {
pPrint("%s showMsgFlag:%d %s", GREEN, g_stConfInfo.showMsgFlag, NC);
pPrint("%s showRowFlag:%d %s", GREEN, g_stConfInfo.showRowFlag, NC);
pPrint("%s saveRowFlag:%d %s", GREEN, g_stConfInfo.saveRowFlag, NC);
pPrint("%s snapshot:%d %s", GREEN, g_stConfInfo.useSnapshot, NC);
pPrint("%s omb topic:%s %s", GREEN, g_stConfInfo.topic, NC);
pPrint("%s numOfThread:%d %s", GREEN, g_stConfInfo.numOfThread, NC);
#endif
}
@ -909,8 +995,439 @@ int32_t getConsumeInfo() {
return 0;
}
static int32_t omb_data_msg_process(TAOS_RES* msg, SThreadInfo* pInfo, int32_t msgIndex, int64_t* lenOfRows) {
char buf[16*1024];
int32_t totalRows = 0;
int32_t totalLen = 0;
// printf("topic: %s\n", tmq_get_topic_name(msg));
//int32_t vgroupId = tmq_get_vgroup_id(msg);
//const char* dbName = tmq_get_db_name(msg);
//taosFprintfFile(g_fp, "consumerId: %d, msg index:%" PRId64 "\n", pInfo->consumerId, msgIndex);
//taosFprintfFile(g_fp, "dbName: %s, topic: %s, vgroupId: %d\n", dbName != NULL ? dbName : "invalid table",
// tmq_get_topic_name(msg), vgroupId);
while (1) {
TAOS_ROW row = taos_fetch_row(msg);
if (row == NULL) break;
TAOS_FIELD* fields = taos_fetch_fields(msg);
int32_t numOfFields = taos_field_count(msg);
//int32_t* length = taos_fetch_lengths(msg);
//int32_t precision = taos_result_precision(msg);
//const char* tbName = tmq_get_table_name(msg);
taos_print_row(buf, row, fields, numOfFields);
totalLen += strlen(buf);
totalRows++;
}
*lenOfRows = totalLen;
return totalRows;
}
void omb_loop_consume(SThreadInfo* pInfo) {
int32_t code;
int32_t once_flag = 0;
int64_t totalMsgs = 0;
int64_t totalRows = 0;
char tmpString[128];
taosFprintfFile(g_fp, "%s consumer id %d start to loop pull msg\n", getCurrentTimeString(tmpString),
pInfo->consumerId);
printf("%s consumer id %d start to loop pull msg\n", getCurrentTimeString(tmpString),
pInfo->consumerId);
pInfo->ts = taosGetTimestampMs();
int64_t lastTotalMsgs = 0;
uint64_t lastPrintTime = taosGetTimestampMs();
uint64_t startTs = taosGetTimestampMs();
int64_t totalLenOfMsg = 0;
int64_t lastTotalLenOfMsg = 0;
int32_t consumeDelay = g_stConfInfo.consumeDelay == -1 ? -1 : (g_stConfInfo.consumeDelay * 1000);
while (running) {
TAOS_RES* tmqMsg = tmq_consumer_poll(pInfo->tmq, consumeDelay);
if (tmqMsg) {
int64_t lenOfMsg = 0;
totalRows += omb_data_msg_process(tmqMsg, pInfo, totalMsgs, &lenOfMsg);
totalLenOfMsg += lenOfMsg;
taos_free_result(tmqMsg);
totalMsgs++;
int64_t currentPrintTime = taosGetTimestampMs();
if (currentPrintTime - lastPrintTime > 10 * 1000) {
int64_t currentLenOfMsg = totalLenOfMsg - lastTotalLenOfMsg;
int64_t deltaTime = currentPrintTime - lastPrintTime;
printf("consumer id %d has currently cons total rows: %" PRId64 ", msgs: %" PRId64 ", rate: %.3f msgs/s, %.1f MB/s\n",
pInfo->consumerId, totalRows, totalMsgs,
(totalMsgs - lastTotalMsgs) * 1000.0 / deltaTime,
currentLenOfMsg*1000.0/(1024*1024)/deltaTime);
taosFprintfFile(
g_fp, "consumer id %d has currently poll total msgs: %" PRId64 ", period cons rate: %.3f msgs/s, %.1f MB/s\n",
pInfo->consumerId, totalMsgs, (totalMsgs - lastTotalMsgs) * 1000.0 / deltaTime, currentLenOfMsg*1000.0/deltaTime);
lastPrintTime = currentPrintTime;
lastTotalMsgs = totalMsgs;
lastTotalLenOfMsg = totalLenOfMsg;
}
} else {
char tmpString[128];
taosFprintfFile(g_fp, "%s no poll more msg when time over, break consume\n", getCurrentTimeString(tmpString));
printf("%s no poll more msg when time over, break consume\n", getCurrentTimeString(tmpString));
int64_t currentPrintTime = taosGetTimestampMs();
int64_t currentLenOfMsg = totalLenOfMsg - lastTotalLenOfMsg;
int64_t deltaTime = currentPrintTime - lastPrintTime;
printf("consumer id %d has currently cons total rows: %" PRId64 ", msgs: %" PRId64 ", rate: %.3f msgs/s, %.1f MB/s\n",
pInfo->consumerId, totalRows, totalMsgs,
(totalMsgs - lastTotalMsgs) * 1000.0 / deltaTime,
currentLenOfMsg*1000.0/(1024*1024)/deltaTime);
break;
}
}
pInfo->consumeMsgCnt = totalMsgs;
pInfo->consumeRowCnt = totalRows;
pInfo->consumeLen = totalLenOfMsg;
}
void* ombConsumeThreadFunc(void* param) {
SThreadInfo* pInfo = (SThreadInfo*)param;
//################### set key ########################
tmq_conf_t* conf = tmq_conf_new();
// tmq_conf_set(conf, "td.connect.ip", "localhost");
// tmq_conf_set(conf, "td.connect.port", "6030");
tmq_conf_set(conf, "td.connect.user", "root");
tmq_conf_set(conf, "td.connect.pass", "taosdata");
// tmq_conf_set(conf, "td.connect.db", g_stConfInfo.dbName);
tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, pInfo);
tmq_conf_set(conf, "group.id", "ombCgrp");
// tmq_conf_set(conf, "msg.with.table.name", "true");
// tmq_conf_set(conf, "client.id", "c-001");
// tmq_conf_set(conf, "enable.auto.commit", "true");
tmq_conf_set(conf, "enable.auto.commit", "false");
// tmq_conf_set(conf, "auto.commit.interval.ms", "1000");
// tmq_conf_set(conf, "auto.offset.reset", "none");
// tmq_conf_set(conf, "auto.offset.reset", "earliest");
tmq_conf_set(conf, "auto.offset.reset", "earliest");
//
if (g_stConfInfo.useSnapshot) {
tmq_conf_set(conf, "experimental.snapshot.enable", "true");
}
pInfo->tmq = tmq_consumer_new(conf, NULL, 0);
tmq_conf_destroy(conf);
//################### set topic ##########################
pInfo->topicList = tmq_list_new();
tmq_list_append(pInfo->topicList, g_stConfInfo.topic);
if ((NULL == pInfo->tmq) || (NULL == pInfo->topicList)) {
taosFprintfFile(g_fp, "create consumer fail! tmq is null or topicList is null\n");
assert(0);
return NULL;
}
int32_t err = tmq_subscribe(pInfo->tmq, pInfo->topicList);
if (err != 0) {
pError("tmq_subscribe() fail, reason: %s\n", tmq_err2str(err));
taosFprintfFile(g_fp, "tmq_subscribe() fail! reason: %s\n", tmq_err2str(err));
assert(0);
return NULL;
}
tmq_list_destroy(pInfo->topicList);
pInfo->topicList = NULL;
omb_loop_consume(pInfo);
err = tmq_unsubscribe(pInfo->tmq);
if (err != 0) {
pError("tmq_unsubscribe() fail, reason: %s\n", tmq_err2str(err));
taosFprintfFile(g_fp, "tmq_unsubscribe()! reason: %s\n", tmq_err2str(err));
}
err = tmq_consumer_close(pInfo->tmq);
if (err != 0) {
pError("tmq_consumer_close() fail, reason: %s\n", tmq_err2str(err));
taosFprintfFile(g_fp, "tmq_consumer_close()! reason: %s\n", tmq_err2str(err));
}
pInfo->tmq = NULL;
return NULL;
}
static int queryDbExec(TAOS *taos, char *command, QUERY_TYPE type) {
TAOS_RES *res = taos_query(taos, command);
int32_t code = taos_errno(res);
if (code != 0) {
pPrint("%s Failed to execute <%s>, reason: %s %s", GREEN, command, taos_errstr(res), NC);
taos_free_result(res);
return -1;
}
if (INSERT_TYPE == type) {
int affectedRows = taos_affected_rows(res);
taos_free_result(res);
return affectedRows;
}
taos_free_result(res);
return 0;
}
void* ombProduceThreadFunc(void* param) {
SThreadInfo* pInfo = (SThreadInfo*)param;
pInfo->taos = taos_connect(NULL, "root", "taosdata", NULL, 0);
if (pInfo->taos == NULL) {
printf("taos_connect() fail\n");
return NULL;
}
int64_t affectedRowsTotal = 0;
int64_t sendMsgs = 0;
uint32_t totalSendLoopTimes = g_stConfInfo.runDurationMinutes * 60 * 1000 / SEND_TIME_UNIT; // send some msgs per 10ms
uint32_t batchPerTblTimes = pInfo->producerRate / 100 / g_stConfInfo.batchSize;
uint32_t remainder = (pInfo->producerRate / 100) % g_stConfInfo.batchSize;
if (remainder) {
batchPerTblTimes += 1;
}
char* sqlBuf = taosMemoryMalloc(MAX_SQL_LEN);
if (NULL == sqlBuf) {
printf("malloc fail for sqlBuf\n");
return NULL;
}
printf("Produce Info: totalSendLoopTimes: %d, batchPerTblTimes: %d, producerRate: %d\n", totalSendLoopTimes, batchPerTblTimes, pInfo->producerRate);
char ctbName[64] = {0};
sprintf(ctbName, "%s.ctb%d", g_stConfInfo.dbName, pInfo->consumerId);
int64_t lastPrintTime = taosGetTimestampUs();
int64_t totalMsgLen = 0;
//int64_t timeStamp = taosGetTimestampUs();
while (totalSendLoopTimes) {
int64_t startTs = taosGetTimestampUs();
for (int i = 0; i < batchPerTblTimes; ++i) {
uint32_t msgsOfSql = g_stConfInfo.batchSize;
if ((i == batchPerTblTimes - 1) && (0 != remainder)) {
msgsOfSql = remainder;
}
int len = 0;
len += snprintf(sqlBuf+len, MAX_SQL_LEN - len, "insert into %s values ", ctbName);
for (int j = 0; j < msgsOfSql; j++) {
int64_t timeStamp = taosGetTimestampNs();
len += snprintf(sqlBuf+len, MAX_SQL_LEN - len, "(%" PRId64 ", \"%s\")", timeStamp, g_payload);
sendMsgs++;
pInfo->totalProduceMsgs++;
}
totalMsgLen += len;
pInfo->totalMsgsLen += len;
int64_t affectedRows = queryDbExec(pInfo->taos, sqlBuf, INSERT_TYPE);
if (affectedRows < 0) {
return NULL;
}
affectedRowsTotal += affectedRows;
//printf("Produce Info: affectedRows: %" PRId64 "\n", affectedRows);
}
totalSendLoopTimes -= 1;
// calc spent time
int64_t currentTs = taosGetTimestampUs();
int64_t delta = currentTs - startTs;
if (delta < SEND_TIME_UNIT * 1000) {
int64_t sleepLen = (int32_t)(SEND_TIME_UNIT * 1000 - delta);
//printf("sleep %" PRId64 " us, use time: %" PRId64 " us\n", sleepLen, delta);
taosUsleep((int32_t)sleepLen);
}
currentTs = taosGetTimestampUs();
delta = currentTs - lastPrintTime;
if (delta > 10 * 1000 * 1000) {
printf("producer[%d] info: %" PRId64 " msgs, %" PRId64 " Byte, %" PRId64 " us, totalSendLoopTimes: %d\n",
pInfo->consumerId, sendMsgs, totalMsgLen, delta, totalSendLoopTimes);
printf("producer[%d] rate: %1.f msgs/s, %1.f KB/s\n",
pInfo->consumerId,
sendMsgs * 1000.0 * 1000 / delta,
(totalMsgLen / 1024.0) / (delta / (1000*1000)));
lastPrintTime = currentTs;
sendMsgs = 0;
totalMsgLen = 0;
}
}
printf("affectedRowsTotal: %"PRId64"\n", affectedRowsTotal);
return NULL;
}
void printProduceInfo(int64_t start) {
int64_t totalMsgs = 0;
int64_t totalLenOfMsgs = 0;
for (int i = 0; i < g_stConfInfo.producers; i++) {
totalMsgs += g_stConfInfo.stProdThreads[i].totalProduceMsgs;
totalLenOfMsgs += g_stConfInfo.stProdThreads[i].totalMsgsLen;
}
int64_t end = taosGetTimestampUs();
int64_t t = end - start;
if (0 == t) t = 1;
double tInMs = (double)t / 1000000.0;
printf("Spent %.3f seconds to prod %" PRIu64 " msgs, %" PRIu64 " Byte\n\n", tInMs, totalMsgs, totalLenOfMsgs);
printf("Spent %.3f seconds to prod %" PRIu64 " msgs with %d producer(s), throughput: %.3f msgs/s, %.1f MB/s\n\n",
tInMs, totalMsgs, g_stConfInfo.producers,
(double)totalMsgs / tInMs,
(double)totalLenOfMsgs/(1024.0*1024)/tInMs);
return;
}
void startOmbConsume() {
TdThreadAttr thattr;
taosThreadAttrInit(&thattr);
taosThreadAttrSetDetachState(&thattr, PTHREAD_CREATE_JOINABLE);
if (0 != g_stConfInfo.producers) {
TAOS* taos = taos_connect(NULL, "root", "taosdata", NULL, 0);
if (taos == NULL) {
taosFprintfFile(g_fp, "taos_connect() fail, can not notify and save consume result to main scripte\n");
ASSERT(0);
return ;
}
char stbName[16] = "stb";
char ctbPrefix[16] = "ctb";
char sql[256] = {0};
sprintf(sql, "drop database if exists %s", g_stConfInfo.dbName);
printf("SQL: %s\n", sql);
queryDbExec(taos, sql, NO_INSERT_TYPE);
sprintf(sql, "create database if not exists %s precision 'ns' vgroups %d", g_stConfInfo.dbName, g_stConfInfo.producers);
printf("SQL: %s\n", sql);
queryDbExec(taos, sql, NO_INSERT_TYPE);
sprintf(sql, "create stable %s.%s (ts timestamp, payload binary(%d)) tags (t bigint) ", g_stConfInfo.dbName, stbName, g_stConfInfo.payloadLen);
printf("SQL: %s\n", sql);
queryDbExec(taos, sql, NO_INSERT_TYPE);
for (int i = 0; i < g_stConfInfo.producers; i++) {
sprintf(sql, "create table %s.%s%d using %s.stb tags(%d) ", g_stConfInfo.dbName, ctbPrefix, i, g_stConfInfo.dbName, i);
printf("SQL: %s\n", sql);
queryDbExec(taos, sql, NO_INSERT_TYPE);
}
// create topic
sprintf(sql, "create topic %s as stable %s.%s", g_stConfInfo.topic, g_stConfInfo.dbName, stbName);
printf("SQL: %s\n", sql);
queryDbExec(taos, sql, NO_INSERT_TYPE);
int32_t producerRate = ceil(g_stConfInfo.producerRate / g_stConfInfo.producers);
printf("==== create %d produce thread ====\n", g_stConfInfo.producers);
for (int32_t i = 0; i < g_stConfInfo.producers; ++i) {
g_stConfInfo.stProdThreads[i].consumerId = i;
g_stConfInfo.stProdThreads[i].producerRate = producerRate;
taosThreadCreate(&(g_stConfInfo.stProdThreads[i].thread), &thattr, ombProduceThreadFunc,
(void*)(&(g_stConfInfo.stProdThreads[i])));
}
if (0 == g_stConfInfo.numOfThread) {
int64_t start = taosGetTimestampUs();
for (int32_t i = 0; i < g_stConfInfo.producers; i++) {
taosThreadJoin(g_stConfInfo.stProdThreads[i].thread, NULL);
taosThreadClear(&g_stConfInfo.stProdThreads[i].thread);
}
printProduceInfo(start);
taosFprintfFile(g_fp, "==== close tmqlog ====\n");
taosCloseFile(&g_fp);
return;
}
}
// pthread_create one thread to consume
taosFprintfFile(g_fp, "==== create %d consume thread ====\n", g_stConfInfo.numOfThread);
for (int32_t i = 0; i < g_stConfInfo.numOfThread; ++i) {
g_stConfInfo.stThreads[i].consumerId = i;
taosThreadCreate(&(g_stConfInfo.stThreads[i].thread), &thattr, ombConsumeThreadFunc,
(void*)(&(g_stConfInfo.stThreads[i])));
}
int64_t start = taosGetTimestampUs();
for (int32_t i = 0; i < g_stConfInfo.numOfThread; i++) {
taosThreadJoin(g_stConfInfo.stThreads[i].thread, NULL);
taosThreadClear(&g_stConfInfo.stThreads[i].thread);
}
int64_t end = taosGetTimestampUs();
int64_t totalRows = 0;
int64_t totalMsgs = 0;
int64_t totalLenOfMsgs = 0;
for (int32_t i = 0; i < g_stConfInfo.numOfThread; i++) {
totalMsgs += g_stConfInfo.stThreads[i].consumeMsgCnt;
totalLenOfMsgs += g_stConfInfo.stThreads[i].consumeLen;
totalRows += g_stConfInfo.stThreads[i].consumeRowCnt;
}
int64_t t = end - start;
if (0 == t) t = 1;
double tInMs = (double)t / 1000000.0;
taosFprintfFile(g_fp,
"Spent %.3f seconds to poll msgs: %" PRIu64 " with %d thread(s), throughput: %.3f msgs/s, %.1f MB/s\n\n",
tInMs, totalMsgs, g_stConfInfo.numOfThread,
(double)(totalMsgs / tInMs),
(double)totalLenOfMsgs/(1024*1024)/tInMs);
printf("Spent %.3f seconds to cons rows: %" PRIu64 " msgs: %" PRIu64 " with %d thread(s), throughput: %.3f msgs/s, %.1f MB/s\n\n",
tInMs, totalRows, totalMsgs, g_stConfInfo.numOfThread,
(double)(totalMsgs / tInMs),
(double)totalLenOfMsgs/(1024*1024)/tInMs);
taosFprintfFile(g_fp, "==== close tmqlog ====\n");
taosCloseFile(&g_fp);
return;
}
int main(int32_t argc, char* argv[]) {
parseArgument(argc, argv);
if (0 != strlen(g_stConfInfo.topic)) {
startOmbConsume();
return 0;
}
getConsumeInfo();
saveConfigToLogFile();

7
tools/shell/src/shellEngine.c
View File

@ -748,6 +748,13 @@ int32_t shellDumpResult(TAOS_RES *tres, char *fname, int32_t *error_no, bool ver
void shellReadHistory() {
SShellHistory *pHistory = &shell.history;
int64_t file_size;
if (taosStatFile(pHistory->file, &file_size, NULL) != 0) {
return;
} else if (file_size > SHELL_MAX_COMMAND_SIZE) {
taosRemoveFile(pHistory->file);
return;
}
TdFilePtr pFile = taosOpenFile(pHistory->file, TD_FILE_READ | TD_FILE_STREAM);
if (pFile == NULL) return;