Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837

This commit is contained in:
54liuyao 2024-08-14 19:21:59 +08:00
commit 141de6e19b
114 changed files with 10356 additions and 5941 deletions

View File

@ -25,7 +25,7 @@ create_definition:
col_name column_definition col_name column_definition
column_definition: column_definition:
type_name [comment 'string_value'] [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type'] type_name [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type']
table_options: table_options:
table_option ... table_option ...

View File

@ -13,7 +13,7 @@ create_definition:
col_name column_definition col_name column_definition
column_definition: column_definition:
type_name [comment 'string_value'] [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type'] type_name [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type']
table_options: table_options:
table_option ... table_option ...

View File

@ -0,0 +1,149 @@
---
toc_max_heading_level: 4
title: "View"
sidebar_label: "View"
---
## Introduction
Starting from TDengine 3.2.1.0, TDengine Enterprise Edition provides view function, which is convenient for users to simplify operation and improve sharing ability among users.
A view is essentially a query statement stored in a database. The view (non-materialized view) itself does not contain data, and only dynamically executes the query statement specified by the view when reading data from the view. We specify a name when creating a view, and then we can query it like using a regular table. The use of views should follow the following rules:
- Views can be defined and used nested, and are bound to the specified or current database when created.
- Within the same database, duplicate view names are not allowed, and it is recommended not to have duplicate view names and table names (not mandatory). When the view and table names have the same name, operations such as writing, querying, authorizing, and revoking permissions will prioritize using the same-named table.
## Grammar
### Create (update) view
```sql
CREATE [ OR REPLACE ] VIEW [db_name.]view_name AS query
```
Description:
- When creating a view, you can specify the database name ( db_name ) to which the view is bound. If not explicitly specified, it defaults to the database bound to the current connection.
- It is recommended to specify the database name in the query statement, support cross-database views, and default to the database bound to the view when not specified (it may not be the database specified by the current connection);
### View View
1. View all views under a database
```sql
SHOW [db_name.]VIEWS;
```
2. View the creation statement of the view
```sql
SHOW CREATE VIEW [db_name.]view_name;
```
3. View view column information
```sql
DESCRIBE [db_name.]view_name;
```
4. View all view information
```sql
SELECT ... FROM information_schema.ins_views;
```
### Delete a view
```sql
DROP VIEW [IF EXISTS] [db_name.]view_name;
```
## Permissions
### Description
View permissions are divided into three types: READ, WRITE, and ALTER. Query operations require READ permissions, write operations require WRITE permissions, and delete and modify operations on the view itself require ALTER permissions.
### Rules
- The creator of the view and the root user have all permissions by default.
- Authorization and revocation of permissions for other users can be performed through the GRANT and REVOKE statements, which can only be performed by the root user.
- View permissions need to be authorized and revoked separately. Authorization and revocation through db. * do not include view permissions.
- Views can be defined and used nested, and the verification of view permissions is also performed by recursion.
- In order to facilitate the sharing and use of views, the concept of view effective user (i.e. the user who creates the view) is introduced. Authorized users can use the read and write permissions of the view effective user's library, table, and nested view. Note: After the view is REPLACE, the effective user will also be updated.
The detailed rules for controlling relevant permissions are summarized as follows:
| Serial number | Operation | Permission requirements |
| --- | --- | --- |
| 1 | CREATE OR REPLACE VIEW (Create a new view) | The user has WRITE permission on the database to which the view belongs And Users have query permissions for the target library, table, and view of the view. If the object in the query is a view, it must meet rule 8 in the current table. |
| 2 | CREATE OR REPLACE VIEW (Overwrite old view) | The user has WRITE permission on the database to which the view belongs, and ALTER permission on the old view And Users have query permissions for the target library, table, and view of the view. If the object in the query is a view, it must meet rule 8 in the current table. |
| 3 | DROP VIEW | The user has ALTER permission on the view |
| 4 | SHOW VIEWS | No |
| 5 | SHOW CREATE VIEW | No |
| 6 | DESCRIBE VIEW | No |
| 7 | System table query | No |
| 8 | SELECT FROM VIEW | The operating user has READ permissions for the view And Operating users or view effective users have READ permissions on the target library, table, and view of the view |
| 9 | INSERT INTO VIEW | The operation user has WRITE permission on the view And Operating users or view effective users have WRITE permissions on the target library, table, and view of the view |
| 10 | GRANT/REVOKE | Only the root user has permission |
### Grammar
#### Authorization
```sql
GRANT privileges ON [db_name.]view_name TO user_name
privileges: {
ALL,
| priv_type [, priv_type] ...
}
priv_type: {
READ
| WRITE
| ALTER
}
```
#### Recover permissions
```sql
REVOKE privileges ON [db_name.]view_name FROM user_name
privileges: {
ALL,
| priv_type [, priv_type] ...
}
priv_type: {
READ
| WRITE
| ALTER
}
```
## Usage scenarios
| SQL query | SQL write | STMT query | STMT write | Subscribe | Stream |
| --- | --- | --- | --- | --- | --- |
| Support | Not supported yet | Not supported yet | Not supported yet | Support | Not supported yet |
## Example
- Create a view
```sql
CREATE VIEW view1 AS SELECT _wstart, count(*) FROM table1 INTERVAL(1d);
CREATE VIEW view2 AS SELECT ts, col2 FROM table1;
CREATE VIEW view3 AS SELECT * from view1;
```
- Query data
```sql
SELECT * from view1;
```
- Delete data
```sql
DROP VIEW view1;
```

View File

@ -91,7 +91,7 @@ toc_max_heading_level: 4
5. 对于小数据量场景,私有化部署太重:在物联网、车联网场景中,因为涉及到生产经营数据的安全,很多还是采取私有化部署。而每个私有化部署,处理的数据量有很大的区别,从几百台联网设备到数千万台设备不等。对于数据量小的场景,通用的大数据解决方案就显得过于臃肿,投入产出不成正比。因此有的平台提供商往往有两套方案,一套针对大数据场景,使用通用的大数据平台,一套针对小数据规模场景,就使用 MySQL 或其他数据库来搞定一切,但是随着历史数据的累积,或接入设备量的增长,关系型数据库性能不足、运维复杂、扩展性差等缺点都会逐渐暴露出来,终究不是长久之计。 5. 对于小数据量场景,私有化部署太重:在物联网、车联网场景中,因为涉及到生产经营数据的安全,很多还是采取私有化部署。而每个私有化部署,处理的数据量有很大的区别,从几百台联网设备到数千万台设备不等。对于数据量小的场景,通用的大数据解决方案就显得过于臃肿,投入产出不成正比。因此有的平台提供商往往有两套方案,一套针对大数据场景,使用通用的大数据平台,一套针对小数据规模场景,就使用 MySQL 或其他数据库来搞定一切,但是随着历史数据的累积,或接入设备量的增长,关系型数据库性能不足、运维复杂、扩展性差等缺点都会逐渐暴露出来,终究不是长久之计。
由于存在这些根本性的缺陷,导致高速增长的时序大数据市场一直没有一个简单好用而又高效的工具。于是,近些年一批专注时序数据处理的企业杀入了这个赛道,比如美国的 InfluxData其产品 InfluxDB 在 IT 运维监测方面有相当的市场占有率。开源社区也十分活跃,比如基于 HBase 开发的 OpenTSDB中国国内阿里、百度、华为都有基于 OpenTSDB 的产品,涛思数据不依赖任何第三方,推出了自主研发而且开源的 TDengine。 由于存在这些根本性的缺陷,导致高速增长的时序大数据市场一直没有一个简单好用而又高效的工具。于是,近些年一批专注时序数据处理的企业杀入了这个赛道,比如美国的 InfluxData其产品 InfluxDB 在 IT 运维监测方面有相当的市场占有率。开源社区也十分活跃,比如基于 HBase 开发的 OpenTSDB中国国内阿里、百度、华为都有基于 OpenTSDB 的产品,涛思数据不依赖任何第三方,推出了自主研发而且开源的 TDengine。
由于数据量巨大且应用方式特殊,对时序数据的处理具有相当大的技术挑战,因此要使用专业的大数据平台。对实时时序数据的科学合理地高效处理能够帮助企业实时监控生产与经营过程,对历史时序数据的分析有助于对资源的使用和生产配置做出科学的决策。 由于数据量巨大且应用方式特殊,对时序数据的处理具有相当大的技术挑战,因此要使用专业的大数据平台。对实时时序数据的科学合理地高效处理能够帮助企业实时监控生产与经营过程,对历史时序数据的分析有助于对资源的使用和生产配置做出科学的决策。

View File

@ -34,9 +34,9 @@ TDengine 经过特别优化,以适应时间序列数据的独特需求,引
6. 可视化/BITDengine 本身不提供可视化或 BI 的功能。但通过其 RESTful API 标准的 JDBC、ODBC 接口TDengine 能够 Grafana、Google Data Studio、Power BI、Tableau 以及国产 BI 工具无缝集成。 6. 可视化/BITDengine 本身不提供可视化或 BI 的功能。但通过其 RESTful API 标准的 JDBC、ODBC 接口TDengine 能够 Grafana、Google Data Studio、Power BI、Tableau 以及国产 BI 工具无缝集成。
7. 集群功能TDengine 支持集群部署,能够随着业务数据量的增长,通过增加节点线性提升系统处理能力,实现水平扩展。同时,通过多副本技术提供高可用性,并支持 Kubernetes 部署。 7. 集群功能TDengine 支持集群部署,能够随着业务数据量的增长,通过增加节点线性提升系统处理能力,实现水平扩展。同时,通过多副本技术提供高可用性,并支持 Kubernetes 部署。同时还提供了多种运维工具,方便系统管理员更好地管理和维护集群的健壮运行。
8. 数据迁移TDengine 提供了多种便捷的数据导入导出功能包括脚本文件导入导出、数据文件导入导出、taosdump 工具导入导出等。企业版还支持边云协同、数据同步等场景,兼容多种数据源,如 AVEVA PI System 等。 8. 数据迁移TDengine 提供了多种便捷的数据导入导出功能包括脚本文件导入导出、数据文件导入导出、taosdump 工具导入导出等。
9. 编程连接器TDengine 提供不同语言的连接器,包括 C/C++、Java、Go、Node.js、Rust、Python、C#、R、PHP 等。这些连接器大多都支持原生连接和 WebSocket 两种连接方式。TDengine 也提供 REST 接口,任何语言的应用程序可以直接通过 HTTP 请求访问数据库。 9. 编程连接器TDengine 提供不同语言的连接器,包括 C/C++、Java、Go、Node.js、Rust、Python、C#、R、PHP 等。这些连接器大多都支持原生连接和 WebSocket 两种连接方式。TDengine 也提供 REST 接口,任何语言的应用程序可以直接通过 HTTP 请求访问数据库。
@ -44,6 +44,8 @@ TDengine 经过特别优化,以适应时间序列数据的独特需求,引
11. 常用工具TDengine 还提供了交互式命令行程序CLI便于管理集群、检查系统状态、做即时查询。压力测试工具 taosBenchmark用于测试 TDengine 的性能。TDengine 还提供了图形化管理界面,简化了操作和管理过程。 11. 常用工具TDengine 还提供了交互式命令行程序CLI便于管理集群、检查系统状态、做即时查询。压力测试工具 taosBenchmark用于测试 TDengine 的性能。TDengine 还提供了图形化管理界面,简化了操作和管理过程。
12. 零代码数据接入TDengine 企业版提供了丰富的数据接入功能依托强大的数据接入平台无需一行代码只需要做简单的配置即可实现多种数据源的数据接入目前已经支持的数据源包括OPC UA, OPC DA, Pi, MQTT, Kafka, InfluxDB, OpenTSDB, MySql, SQL Server, Oracle, Wonderware Historian, MongoDB。
## TDengine 与典型时序数据库的区别 ## TDengine 与典型时序数据库的区别
由于充分利用了时序数据特点,并采用独特创新的“一个数据采集点一张表” “超级表”的属于模型与其他时序数据库相比TDengine 拥有以下特点: 由于充分利用了时序数据特点,并采用独特创新的“一个数据采集点一张表” “超级表”的属于模型与其他时序数据库相比TDengine 拥有以下特点:

View File

@ -492,7 +492,7 @@ Rust 连接器创建消费者的参数为 DSN 可以设置的参数列表请
3. 对于每个分配,使用 `tmq_offset_seek` 函数将消费者的偏移量设置到最早的偏移量。 3. 对于每个分配,使用 `tmq_offset_seek` 函数将消费者的偏移量设置到最早的偏移量。
4. 如果设置偏移量失败,则打印错误信息。 4. 如果设置偏移量失败,则打印错误信息。
5. 释放分配信息数组以释放资源。 5. 释放分配信息数组以释放资源。
6. 调用 `basic_consume_loop` 函数开始新的消费循环,处理消息。 6. 调用 `basic_consume_loop` 函数开始新的消费循环,处理消息。
</TabItem> </TabItem>
<TabItem label="REST API" value="rest"> <TabItem label="REST API" value="rest">

View File

@ -46,7 +46,7 @@ import TabItem from "@theme/TabItem";
从服务端配置的角度,要根据系统中磁盘的数量,磁盘的 I/O 能力,以及处理器能力在创建数据库时设置适当的 vgroups 数量以充分发挥系统性能。如果 vgroups 过少,则系统性能无法发挥;如果 vgroups 过多,会造成无谓的资源竞争。常规推荐 vgroups 数量为 CPU 核数的 2 倍,但仍然要结合具体的系统资源配置进行调优。 从服务端配置的角度,要根据系统中磁盘的数量,磁盘的 I/O 能力,以及处理器能力在创建数据库时设置适当的 vgroups 数量以充分发挥系统性能。如果 vgroups 过少,则系统性能无法发挥;如果 vgroups 过多,会造成无谓的资源竞争。常规推荐 vgroups 数量为 CPU 核数的 2 倍,但仍然要结合具体的系统资源配置进行调优。
更多调优参数,请参考 [数据库管理](../../../taos-sql/database) 和 [服务端配置](../../../reference/config)。 更多调优参数,请参考 [数据库管理](../../reference/taos-sql/database) 和 [服务端配置](../../reference/components/taosd)。
## 高效写入示例 {#sample-code} ## 高效写入示例 {#sample-code}

View File

@ -26,7 +26,7 @@ create_definition:
col_name column_definition col_name column_definition
column_definition: column_definition:
type_name [comment 'string_value'] [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type'] type_name [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type']
table_options: table_options:
table_option ... table_option ...

View File

@ -13,7 +13,7 @@ create_definition:
col_name column_definition col_name column_definition
column_definition: column_definition:
type_name [comment 'string_value'] [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type'] type_name [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type']
table_options: table_options:
table_option ... table_option ...

View File

@ -0,0 +1,139 @@
---
toc_max_heading_level: 4
title: "视图"
sidebar_label: "视图"
---
## 简介
从 TDengine 3.2.1.0 开始TDengine 企业版提供视图功能,便于用户简化操作,提升用户间的分享能力。
视图View本质上是一个存储在数据库中的查询语句。视图非物化视图本身不包含数据只有在从视图读取数据时才动态执行视图所指定的查询语句。我们在创建视图时指定一个名称然后可以像使用普通表一样对其进行查询等操作。视图的使用需遵循以下规则
- 视图可以嵌套定义和使用,视图与创建时指定的或当前数据库绑定使用。
- 在同一个数据库内,视图名称不允许重名,视图名跟表名也推荐不重名(不强制)。当出现视图与表名重名时,写入、查询、授权、回收权限等操作优先使用同名表。
## 语法
### 创建(更新)视图
```sql
CREATE [ OR REPLACE ] VIEW [db_name.]view_name AS query
```
说明:
- 创建视图时可以指定视图绑定的数据库名db_name未明确指定时默认为当前连接绑定的数据库
- 查询语句query中推荐指定数据库名支持跨库视图未指定时默认为与视图绑定的数据库(有可能非当前连接指定的数据库)
### 查看视图
1. 查看某个数据库下的所有视图
```sql
SHOW [db_name.]VIEWS;
```
2. 查看视图的创建语句
```sql
SHOW CREATE VIEW [db_name.]view_name;
```
3. 查看视图列信息
```sql
DESCRIBE [db_name.]view_name;
```
4. 查看所有视图信息
```sql
SELECT ... FROM information_schema.ins_views;
```
### 删除视图
```sql
DROP VIEW [IF EXISTS] [db_name.]view_name;
```
## 权限
### 说明
视图的权限分为 READ、WRITE、ALTER 三种,查询操作需要具备 READ 权限,写入操作需要具备 WRITE 权限,对视图本身的删改操作需要具备 ALTER 权限。
### 规则
- 视图的创建者和 root 用户默认具备所有权限。
- 对其他用户进行授权与回收权限可以通过 GRANT 和 REVOKE 语句进行,该操作只能由 root 用户进行。
- 视图权限需单独授权与回收通过db.*进行的授权与回收不含视图权限。
- 视图可以嵌套定义与使用,同理对视图权限的校验也是递归进行的。
- 为了方便视图的分享与使用,引入视图有效用户(即视图的创建用户)的概念,被授权用户可以使用视图有效用户的库、表及嵌套视图的读写权限。注:视图被 REPLACE 后有效用户也会被更新。
具体相关权限控制细则如下表所示:
| 序号 | 操作 | 权限要求 |
| ---- | --------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | CREATE VIEW <br/>(创建新用户) | 用户对视图所属数据库有 WRITE 权限<br/><br/> 用户对视图的目标库、表、视图有查询权限若查询中的对象是视图需满足当前表中第8条规则 |
| 2 | CREATE OR REPLACE VIEW <br/>(覆盖旧视图) | 用户对视图所属数据库有 WRITE 权限 且 对旧有视图有 ALTER 权限 <br/><br/> 用户对视图的目标库、表、视图有查询权限若查询中的对象是视图需满足当前表中第8条规则 |
| 3 | DROP VIEW | 用户对视图有 ALTER 权限 |
| 4 | SHOW VIEWS | 无 |
| 5 | SHOW CREATE VIEW | 无 |
| 6 | DESCRIBE VIEW | 无 |
| 7 | 系统表查询 | 无 |
| 8 | SELECT FROM VIEW | 操作用户对视图有 READ 权限 且 操作用户或视图有效用户对视图的目标库、表、视图有 READ 权限 |
| 9 | INSERT INTO VIEW | 操作用户对视图有 WRITE 权限 且 操作用户或视图有效用户对视图的目标库、表、视图有 WRITE 权限 |
| 10 | GRANT/REVOKE | 只有 root 用户有权限 |
### 语法
#### 授权
```sql
GRANT privileges ON [db_name.]view_name TO user_name
privileges: {
ALL,
| priv_type [, priv_type] ...
}
priv_type: {
READ
| WRITE
| ALTER
}
```
#### 回收权限
```sql
REVOKE privileges ON [db_name.]view_name FROM user_name
privileges: {
ALL,
| priv_type [, priv_type] ...
}
priv_type: {
READ
| WRITE
| ALTER
}
```
## 使用场景
| SQL 查询 | SQL 写入 | STMT 查询 | STMT 写入 | 订阅 | 流计算 |
| -------- | -------- | --------- | --------- | ---- | -------- |
| 支持 | 暂不支持 | 暂不支持 | 暂不支持 | 支持 | 暂不支持 |
## 举例
- 创建视图
```sql
CREATE VIEW view1 AS SELECT _wstart, count(*) FROM table1 INTERVAL(1d);
CREATE VIEW view2 AS SELECT ts, col2 FROM table1;
CREATE VIEW view3 AS SELECT * from view1;
```
- 查询数据
```sql
SELECT * from view1;
```
- 删除视图
```sql
DROP VIEW view1;
```

View File

@ -783,7 +783,7 @@ TDengine 的异步 API 均采用非阻塞调用模式。应用程序可以用多
- **返回值**:非 `NULL`:成功,返回一个 TAOS * 类型的指针,指向与 TDengine 数据库的连接句柄。`NULL`:失败,非法的输入参数。 - **返回值**:非 `NULL`:成功,返回一个 TAOS * 类型的指针,指向与 TDengine 数据库的连接句柄。`NULL`:失败,非法的输入参数。
- `const char *tmq_get_table_name(TAOS_RES *res)` - `const char *tmq_get_table_name(TAOS_RES *res)`
- **接口说明**:从 TMQ 消费者获取的消息结果中获取所属的表名。 - **接口说明**:从 TMQ 消费者获取的消息结果中获取所属的表名。
- res[入参] 指向一个有效的 TAOS_RES 结构体指针,该结构体包含了从 TMQ 消费者轮询得到的消息。 - res[入参] 指向一个有效的 TAOS_RES 结构体指针,该结构体包含了从 TMQ 消费者轮询得到的消息。
- **返回值**:非 `NULL`:成功,返回一个 const char * 类型的指针,指向表名字符串。`NULL`:失败,非法的输入参数。 - **返回值**:非 `NULL`:成功,返回一个 const char * 类型的指针,指向表名字符串。`NULL`:失败,非法的输入参数。

View File

@ -307,7 +307,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。
- **参数说明** - **参数说明**
- `topic`: 订阅的主题。 - `topic`: 订阅的主题。
- `vg_id`: vgroupid。 - `vg_id`: vgroupid。
- `offset`:需要设置的偏移量。 - `offset`:需要设置的偏移量。
- **异常**:操作失败抛出 ConsumerException 异常。 - **异常**:操作失败抛出 ConsumerException 异常。
- `fn committed(&mut self, topic: &str, vg_id: i32) -> PyResult<i64>` - `fn committed(&mut self, topic: &str, vg_id: i32) -> PyResult<i64>`
- **接口说明**获取订阅主题的vgroupid分区最后提交的偏移量。 - **接口说明**获取订阅主题的vgroupid分区最后提交的偏移量。
@ -489,7 +489,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。
- `def seek(self, partition)` - `def seek(self, partition)`
- **接口说明**:将给定分区的偏移量设置到指定的位置。 - **接口说明**:将给定分区的偏移量设置到指定的位置。
- **参数说明** - **参数说明**
- `partition`: 需要设置的偏移量。 - `partition`: 需要设置的偏移量。
- `topic`: 订阅的主题 - `topic`: 订阅的主题
- `partition`: 分区 - `partition`: 分区
- `offset`: 偏移量 - `offset`: 偏移量
@ -497,7 +497,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。
- `def committed(self, partitions)` - `def committed(self, partitions)`
- **接口说明**:获取订阅主题的分区最后提交的偏移量。 - **接口说明**:获取订阅主题的分区最后提交的偏移量。
- **参数说明** - **参数说明**
- `partition`: 需要设置的偏移量。 - `partition`: 需要设置的偏移量。
- `topic`: 订阅的主题 - `topic`: 订阅的主题
- `partition`: 分区 - `partition`: 分区
- **返回值**`partition`,分区最后提交的偏移量。 - **返回值**`partition`,分区最后提交的偏移量。
@ -505,7 +505,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。
- `def position(self, partitions)` - `def position(self, partitions)`
- **接口说明**:获取给定分区当前的偏移量。 - **接口说明**:获取给定分区当前的偏移量。
- **参数说明** - **参数说明**
- `partition`: 需要设置的偏移量。 - `partition`: 需要设置的偏移量。
- `topic`: 订阅的主题 - `topic`: 订阅的主题
- `partition`: 分区 - `partition`: 分区
- **返回值**`partition`,分区最后提交的偏移量。 - **返回值**`partition`,分区最后提交的偏移量。

View File

@ -8,7 +8,7 @@ import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem"; import TabItem from "@theme/TabItem";
import RequestId from "./_request_id.mdx"; import RequestId from "./_request_id.mdx";
`@tdengine/websocket` 是 TDengine 的官方 Node.js 语言连接器。Node.js 开发人员可以通过它开发存取 TDengine 数据库的应用软件。 `@tdengine/websocket` 是 TDengine 的官方 Node.js 语言连接器。Node.js 开发人员可以通过它开发存取 TDengine 数据库的应用软件。
Node.js 连接器源码托管在 [GitHub](https://github.com/taosdata/taos-connector-node/tree/main)。 Node.js 连接器源码托管在 [GitHub](https://github.com/taosdata/taos-connector-node/tree/main)。
@ -194,7 +194,7 @@ WSConfig 中的配置如下:
### 数据集 ### 数据集
- `getMeta():Array<TDengineMeta> | null` - `getMeta():Array<TDengineMeta> | null`
- **接口说明**:获取查询结果的列的数量、类型和长度。 - **接口说明**:获取查询结果的列的数量、类型和长度。
- **返回值**TDengineMeta 数据对象数组。 - **返回值**TDengineMeta 数据对象数组。
```js ```js
export interface TDengineMeta { export interface TDengineMeta {

View File

@ -14,7 +14,7 @@ description: "TDengine 服务端、客户端和连接器支持的平台列表"
| M1 | | | | | | | | ● | | M1 | | | | | | | | ● |
1) ● 表示经过官方测试验证, ○ 表示非官方测试验证E 表示仅企业版支持。 1) ● 表示经过官方测试验证, ○ 表示非官方测试验证E 表示仅企业版支持。
2) 社区版仅支持主流操作系统的较新版本,包括 Ubuntu 18+/CentOS 7+/RetHat/Debian/CoreOS/FreeBSD/OpenSUSE/SUSE Linux/Fedora/macOS 等。如果有其他操作系统及版本的需求,请联系企业版支持。 2) 社区版仅支持主流操作系统的较新版本,包括 Ubuntu 18+/CentOS 7+/RedHat/Debian/CoreOS/FreeBSD/OpenSUSE/SUSE Linux/Fedora/macOS 等。如果有其他操作系统及版本的需求,请联系企业版支持。
## TDengine 客户端和连接器支持的平台列表 ## TDengine 客户端和连接器支持的平台列表

View File

@ -1,5 +1,5 @@
--- ---
sidebar_label: TDengine 错误码 sidebar_label: 错误码
title: TDengine 错误码 title: TDengine 错误码
description: TDengine 服务端的错误码列表和详细说明 description: TDengine 服务端的错误码列表和详细说明
--- ---

View File

@ -292,7 +292,7 @@ int32_t buildCtbNameByGroupIdImpl(const char* stbName, uint64_t groupId, char* p
int32_t trimDataBlock(SSDataBlock* pBlock, int32_t totalRows, const bool* pBoolList); int32_t trimDataBlock(SSDataBlock* pBlock, int32_t totalRows, const bool* pBoolList);
void copyPkVal(SDataBlockInfo* pDst, const SDataBlockInfo* pSrc); int32_t copyPkVal(SDataBlockInfo* pDst, const SDataBlockInfo* pSrc);
#ifdef __cplusplus #ifdef __cplusplus
} }

View File

@ -86,7 +86,7 @@ extern int32_t tsNumOfQnodeQueryThreads;
extern int32_t tsNumOfQnodeFetchThreads; extern int32_t tsNumOfQnodeFetchThreads;
extern int32_t tsNumOfSnodeStreamThreads; extern int32_t tsNumOfSnodeStreamThreads;
extern int32_t tsNumOfSnodeWriteThreads; extern int32_t tsNumOfSnodeWriteThreads;
extern int64_t tsRpcQueueMemoryAllowed; extern int64_t tsQueueMemoryAllowed;
extern int32_t tsRetentionSpeedLimitMB; extern int32_t tsRetentionSpeedLimitMB;
// sync raft // sync raft

View File

@ -45,6 +45,7 @@ typedef enum EFunctionType {
FUNCTION_TYPE_TWA, FUNCTION_TYPE_TWA,
FUNCTION_TYPE_HISTOGRAM, FUNCTION_TYPE_HISTOGRAM,
FUNCTION_TYPE_HYPERLOGLOG, FUNCTION_TYPE_HYPERLOGLOG,
FUNCTION_TYPE_STDVAR,
// nonstandard SQL function // nonstandard SQL function
FUNCTION_TYPE_BOTTOM = 500, FUNCTION_TYPE_BOTTOM = 500,
@ -77,6 +78,15 @@ typedef enum EFunctionType {
FUNCTION_TYPE_ASIN, FUNCTION_TYPE_ASIN,
FUNCTION_TYPE_ACOS, FUNCTION_TYPE_ACOS,
FUNCTION_TYPE_ATAN, FUNCTION_TYPE_ATAN,
FUNCTION_TYPE_PI,
FUNCTION_TYPE_EXP,
FUNCTION_TYPE_LN,
FUNCTION_TYPE_MOD,
FUNCTION_TYPE_RAND,
FUNCTION_TYPE_SIGN,
FUNCTION_TYPE_DEGREES,
FUNCTION_TYPE_RADIANS,
FUNCTION_TYPE_TRUNCATE,
// string function // string function
FUNCTION_TYPE_LENGTH = 1500, FUNCTION_TYPE_LENGTH = 1500,
@ -89,6 +99,13 @@ typedef enum EFunctionType {
FUNCTION_TYPE_RTRIM, FUNCTION_TYPE_RTRIM,
FUNCTION_TYPE_SUBSTR, FUNCTION_TYPE_SUBSTR,
FUNCTION_TYPE_MD5, FUNCTION_TYPE_MD5,
FUNCTION_TYPE_CHAR,
FUNCTION_TYPE_ASCII,
FUNCTION_TYPE_POSITION,
FUNCTION_TYPE_TRIM,
FUNCTION_TYPE_REPLACE,
FUNCTION_TYPE_REPEAT,
FUNCTION_TYPE_SUBSTR_IDX,
// conversion function // conversion function
FUNCTION_TYPE_CAST = 2000, FUNCTION_TYPE_CAST = 2000,
@ -104,6 +121,10 @@ typedef enum EFunctionType {
FUNCTION_TYPE_TIMETRUNCATE, FUNCTION_TYPE_TIMETRUNCATE,
FUNCTION_TYPE_TIMEZONE, FUNCTION_TYPE_TIMEZONE,
FUNCTION_TYPE_TODAY, FUNCTION_TYPE_TODAY,
FUNCTION_TYPE_WEEK,
FUNCTION_TYPE_WEEKDAY,
FUNCTION_TYPE_WEEKOFYEAR,
FUNCTION_TYPE_DAYOFWEEK,
// system function // system function
FUNCTION_TYPE_DATABASE = 3000, FUNCTION_TYPE_DATABASE = 3000,
@ -162,8 +183,9 @@ typedef enum EFunctionType {
FUNCTION_TYPE_LAST_MERGE, FUNCTION_TYPE_LAST_MERGE,
FUNCTION_TYPE_AVG_PARTIAL, FUNCTION_TYPE_AVG_PARTIAL,
FUNCTION_TYPE_AVG_MERGE, FUNCTION_TYPE_AVG_MERGE,
FUNCTION_TYPE_STDDEV_PARTIAL, FUNCTION_TYPE_STD_PARTIAL,
FUNCTION_TYPE_STDDEV_MERGE, FUNCTION_TYPE_STDDEV_MERGE,
FUNCTION_TYPE_STDVAR_MERGE,
FUNCTION_TYPE_IRATE_PARTIAL, FUNCTION_TYPE_IRATE_PARTIAL,
FUNCTION_TYPE_IRATE_MERGE, FUNCTION_TYPE_IRATE_MERGE,
FUNCTION_TYPE_AVG_STATE, FUNCTION_TYPE_AVG_STATE,
@ -174,8 +196,8 @@ typedef enum EFunctionType {
FUNCTION_TYPE_LAST_STATE_MERGE, FUNCTION_TYPE_LAST_STATE_MERGE,
FUNCTION_TYPE_SPREAD_STATE, FUNCTION_TYPE_SPREAD_STATE,
FUNCTION_TYPE_SPREAD_STATE_MERGE, FUNCTION_TYPE_SPREAD_STATE_MERGE,
FUNCTION_TYPE_STDDEV_STATE, FUNCTION_TYPE_STD_STATE,
FUNCTION_TYPE_STDDEV_STATE_MERGE, FUNCTION_TYPE_STD_STATE_MERGE,
FUNCTION_TYPE_HYPERLOGLOG_STATE, FUNCTION_TYPE_HYPERLOGLOG_STATE,
FUNCTION_TYPE_HYPERLOGLOG_STATE_MERGE, FUNCTION_TYPE_HYPERLOGLOG_STATE_MERGE,

View File

@ -170,6 +170,12 @@ typedef struct SNodeListNode {
SNodeList* pNodeList; SNodeList* pNodeList;
} SNodeListNode; } SNodeListNode;
typedef enum ETrimType {
TRIM_TYPE_LEADING = 1,
TRIM_TYPE_TRAILING,
TRIM_TYPE_BOTH,
} ETrimType;
typedef struct SFunctionNode { typedef struct SFunctionNode {
SExprNode node; // QUERY_NODE_FUNCTION SExprNode node; // QUERY_NODE_FUNCTION
char functionName[TSDB_FUNC_NAME_LEN]; char functionName[TSDB_FUNC_NAME_LEN];
@ -181,6 +187,7 @@ typedef struct SFunctionNode {
int32_t pkBytes; int32_t pkBytes;
bool hasOriginalFunc; bool hasOriginalFunc;
int32_t originalFuncId; int32_t originalFuncId;
ETrimType trimType;
} SFunctionNode; } SFunctionNode;
typedef struct STableNode { typedef struct STableNode {

View File

@ -61,6 +61,15 @@ int32_t atanFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutp
int32_t ceilFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t ceilFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t floorFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t floorFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t roundFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t roundFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t truncFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t piFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t expFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t lnFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t modFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t signFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t degreesFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t radiansFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
/* String functions */ /* String functions */
int32_t lengthFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t lengthFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
@ -73,6 +82,13 @@ int32_t ltrimFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOut
int32_t rtrimFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t rtrimFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t substrFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t substrFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t md5Function(SScalarParam* pInput, int32_t inputNum, SScalarParam* pOutput); int32_t md5Function(SScalarParam* pInput, int32_t inputNum, SScalarParam* pOutput);
int32_t charFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t asciiFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t positionFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t trimFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t replaceFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t repeatFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t substrIdxFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
/* Conversion functions */ /* Conversion functions */
int32_t castFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t castFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
@ -89,6 +105,10 @@ int32_t timeDiffFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *p
int32_t nowFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t nowFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t todayFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t todayFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t timezoneFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t timezoneFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t weekdayFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t dayofweekFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t weekFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t weekofyearFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
bool getTimePseudoFuncEnv(struct SFunctionNode *pFunc, SFuncExecEnv *pEnv); bool getTimePseudoFuncEnv(struct SFunctionNode *pFunc, SFuncExecEnv *pEnv);
@ -106,7 +126,7 @@ int32_t sumScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *
int32_t minScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t minScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t maxScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t maxScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t avgScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t avgScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t stddevScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t stdScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t leastSQRScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t leastSQRScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t percentileScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t percentileScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);
int32_t apercentileScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput); int32_t apercentileScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput);

View File

@ -124,7 +124,8 @@ extern "C" {
#include "taoserror.h" #include "taoserror.h"
#include "tlog.h" #include "tlog.h"
extern int32_t tsRandErrChance; extern int32_t tsRandErrChance;
extern int64_t tsRandErrDivisor;
extern threadlocal bool tsEnableRandErr; extern threadlocal bool tsEnableRandErr;
#ifdef __cplusplus #ifdef __cplusplus

View File

@ -877,6 +877,7 @@ int32_t taosGetErrSize();
#define TSDB_CODE_FUNC_TIME_UNIT_TOO_SMALL TAOS_DEF_ERROR_CODE(0, 0x280B) #define TSDB_CODE_FUNC_TIME_UNIT_TOO_SMALL TAOS_DEF_ERROR_CODE(0, 0x280B)
#define TSDB_CODE_FUNC_INVALID_VALUE_RANGE TAOS_DEF_ERROR_CODE(0, 0x280C) #define TSDB_CODE_FUNC_INVALID_VALUE_RANGE TAOS_DEF_ERROR_CODE(0, 0x280C)
#define TSDB_CODE_FUNC_SETUP_ERROR TAOS_DEF_ERROR_CODE(0, 0x280D) #define TSDB_CODE_FUNC_SETUP_ERROR TAOS_DEF_ERROR_CODE(0, 0x280D)
#define TSDB_CODE_FUNC_INVALID_RES_LENGTH TAOS_DEF_ERROR_CODE(0, 0x280E)
//udf //udf

View File

@ -29,7 +29,7 @@ static FORCE_INLINE TSCKSUM taosCalcChecksum(TSCKSUM csi, const uint8_t *stream,
} }
static FORCE_INLINE int32_t taosCalcChecksumAppend(TSCKSUM csi, uint8_t *stream, uint32_t ssize) { static FORCE_INLINE int32_t taosCalcChecksumAppend(TSCKSUM csi, uint8_t *stream, uint32_t ssize) {
if (ssize < sizeof(TSCKSUM)) return -1; if (ssize < sizeof(TSCKSUM)) return TSDB_CODE_INVALID_PARA;
*((TSCKSUM *)(stream + ssize - sizeof(TSCKSUM))) = (*crc32c)(csi, stream, (size_t)(ssize - sizeof(TSCKSUM))); *((TSCKSUM *)(stream + ssize - sizeof(TSCKSUM))) = (*crc32c)(csi, stream, (size_t)(ssize - sizeof(TSCKSUM)));

View File

@ -294,6 +294,8 @@ typedef enum ELogicConditionType {
#define TSDB_SHOW_SUBQUERY_LEN 1000 #define TSDB_SHOW_SUBQUERY_LEN 1000
#define TSDB_LOG_VAR_LEN 32 #define TSDB_LOG_VAR_LEN 32
#define TSDB_MAX_EP_NUM 10
#define TSDB_ARB_GROUP_MEMBER_NUM 2 #define TSDB_ARB_GROUP_MEMBER_NUM 2
#define TSDB_ARB_TOKEN_SIZE 32 #define TSDB_ARB_TOKEN_SIZE 32

View File

@ -746,7 +746,13 @@ static int32_t smlParseJSONStringExt(SSmlHandle *info, cJSON *root, SSmlLineInfo
int64_t ts = smlParseTSFromJSON(info, tsJson); int64_t ts = smlParseTSFromJSON(info, tsJson);
if (unlikely(ts < 0)) { if (unlikely(ts < 0)) {
char* tmp = cJSON_PrintUnformatted(tsJson); char* tmp = cJSON_PrintUnformatted(tsJson);
uError("OTD:0x%" PRIx64 " Unable to parse timestamp from JSON payload %s %s %" PRId64, info->id, info->msgBuf.buf,tmp, ts); if (tmp == NULL) {
uError("cJSON_PrintUnformatted failed since %s", tstrerror(TSDB_CODE_OUT_OF_MEMORY));
uError("OTD:0x%" PRIx64 " Unable to parse timestamp from JSON payload %s %" PRId64, info->id, info->msgBuf.buf, ts);
} else {
uError("OTD:0x%" PRIx64 " Unable to parse timestamp from JSON payload %s %s %" PRId64, info->id, info->msgBuf.buf,tmp, ts);
taosMemoryFree(tmp);
}
return TSDB_CODE_INVALID_TIMESTAMP; return TSDB_CODE_INVALID_TIMESTAMP;
} }
SSmlKv kvTs = {0}; SSmlKv kvTs = {0};

View File

@ -35,6 +35,13 @@ TARGET_INCLUDE_DIRECTORIES(
PRIVATE "${TD_SOURCE_DIR}/source/client/inc" PRIVATE "${TD_SOURCE_DIR}/source/client/inc"
) )
IF(${TD_LINUX})
add_test(
NAME clientTest
COMMAND clientTest
)
ENDIF ()
TARGET_INCLUDE_DIRECTORIES( TARGET_INCLUDE_DIRECTORIES(
tmqTest tmqTest
PUBLIC "${TD_SOURCE_DIR}/include/client/" PUBLIC "${TD_SOURCE_DIR}/include/client/"

View File

@ -16,6 +16,7 @@
#include <gtest/gtest.h> #include <gtest/gtest.h>
#include <iostream> #include <iostream>
#include "clientInt.h" #include "clientInt.h"
#include "osSemaphore.h"
#include "taoserror.h" #include "taoserror.h"
#include "tglobal.h" #include "tglobal.h"
#include "thash.h" #include "thash.h"
@ -126,6 +127,9 @@ void queryCallback(void* param, void* res, int32_t code) {
} }
(void)printf("start to fetch data\n"); (void)printf("start to fetch data\n");
taos_fetch_raw_block_a(res, fetchCallback, param); taos_fetch_raw_block_a(res, fetchCallback, param);
taos_free_result(res);
tsem_t *sem = (tsem_t *)param;
tsem_post(sem);
} }
void createNewTable(TAOS* pConn, int32_t index, int32_t numOfRows, int64_t startTs, const char* pVarchar) { void createNewTable(TAOS* pConn, int32_t index, int32_t numOfRows, int64_t startTs, const char* pVarchar) {
@ -189,7 +193,7 @@ void* queryThread(void* arg) {
int64_t el = 0; int64_t el = 0;
for (int32_t i = 0; i < 5000000; ++i) { for (int32_t i = 0; i < 5; ++i) {
int64_t st = taosGetTimestampUs(); int64_t st = taosGetTimestampUs();
TAOS_RES* pRes = taos_query(pConn, TAOS_RES* pRes = taos_query(pConn,
"SELECT _wstart as ts,max(usage_user) FROM benchmarkcpu.host_49 WHERE ts >= " "SELECT _wstart as ts,max(usage_user) FROM benchmarkcpu.host_49 WHERE ts >= "
@ -320,6 +324,13 @@ TEST(clientCase, connect_Test) {
if (pConn == NULL) { if (pConn == NULL) {
(void)printf("failed to connect to server, reason:%s\n", taos_errstr(NULL)); (void)printf("failed to connect to server, reason:%s\n", taos_errstr(NULL));
} }
TAOS_RES* pRes = taos_query(pConn, "drop database abc1");
if (taos_errno(pRes) != 0) {
(void)printf("error in drop db, reason:%s\n", taos_errstr(pRes));
}
taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
} }
@ -412,7 +423,7 @@ TEST(clientCase, show_db_Test) {
int32_t code = taos_print_row(str, pRow, pFields, numOfFields); int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
(void)printf("%s\n", str); (void)printf("%s\n", str);
} }
taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
} }
@ -437,6 +448,7 @@ TEST(clientCase, create_db_Test) {
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
(void)printf("error in create db, reason:%s\n", taos_errstr(pRes)); (void)printf("error in create db, reason:%s\n", taos_errstr(pRes));
} }
taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
} }
@ -473,6 +485,7 @@ TEST(clientCase, drop_dnode_Test) {
int32_t numOfFields = taos_num_fields(pRes); int32_t numOfFields = taos_num_fields(pRes);
ASSERT_EQ(numOfFields, 0); ASSERT_EQ(numOfFields, 0);
taos_free_result(pRes);
pRes = taos_query(pConn, "drop dnode 4"); pRes = taos_query(pConn, "drop dnode 4");
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
@ -498,31 +511,10 @@ TEST(clientCase, use_db_test) {
int32_t numOfFields = taos_num_fields(pRes); int32_t numOfFields = taos_num_fields(pRes);
ASSERT_EQ(numOfFields, 0); ASSERT_EQ(numOfFields, 0);
taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
} }
// TEST(clientCase, drop_db_test) {
// TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
// assert(pConn != NULL);
//
// showDB(pConn);
//
// TAOS_RES* pRes = taos_query(pConn, "drop database abc1");
// if (taos_errno(pRes) != 0) {
// (void)printf("failed to drop db, reason:%s\n", taos_errstr(pRes));
// }
// taos_free_result(pRes);
//
// showDB(pConn);
//
// pRes = taos_query(pConn, "create database abc1");
// if (taos_errno(pRes) != 0) {
// (void)printf("create to drop db, reason:%s\n", taos_errstr(pRes));
// }
// taos_free_result(pRes);
// taos_close(pConn);
//}
TEST(clientCase, create_stable_Test) { TEST(clientCase, create_stable_Test) {
TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
assert(pConn != NULL); assert(pConn != NULL);
@ -534,6 +526,7 @@ TEST(clientCase, create_stable_Test) {
taos_free_result(pRes); taos_free_result(pRes);
pRes = taos_query(pConn, "use abc1"); pRes = taos_query(pConn, "use abc1");
taos_free_result(pRes);
pRes = taos_query(pConn, "create table if not exists abc1.st1(ts timestamp, k int) tags(a int)"); pRes = taos_query(pConn, "create table if not exists abc1.st1(ts timestamp, k int) tags(a int)");
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
@ -547,18 +540,6 @@ TEST(clientCase, create_stable_Test) {
ASSERT_EQ(numOfFields, 0); ASSERT_EQ(numOfFields, 0);
taos_free_result(pRes); taos_free_result(pRes);
// pRes = taos_query(pConn, "create stable if not exists abc1.`123_$^)` (ts timestamp, `abc` int) tags(a int)");
// if (taos_errno(pRes) != 0) {
// (void)printf("failed to create super table 123_$^), reason:%s\n", taos_errstr(pRes));
// }
//
// pRes = taos_query(pConn, "use abc1");
// taos_free_result(pRes);
// pRes = taos_query(pConn, "drop stable `123_$^)`");
// if (taos_errno(pRes) != 0) {
// (void)printf("failed to drop super table 123_$^), reason:%s\n", taos_errstr(pRes));
// }
taos_close(pConn); taos_close(pConn);
} }
@ -743,7 +724,8 @@ TEST(clientCase, show_table_Test) {
} }
taos_free_result(pRes); taos_free_result(pRes);
(void)taos_query(pConn, "use abc1"); pRes = taos_query(pConn, "use abc1");
taos_free_result(pRes);
pRes = taos_query(pConn, "show tables"); pRes = taos_query(pConn, "show tables");
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
@ -760,38 +742,15 @@ TEST(clientCase, show_table_Test) {
while ((pRow = taos_fetch_row(pRes)) != NULL) { while ((pRow = taos_fetch_row(pRes)) != NULL) {
int32_t code = taos_print_row(str, pRow, pFields, numOfFields); int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
(void)printf("%d: %s\n", ++count, str); if(code > 0) {
(void)printf("%d: %s\n", ++count, str);
}
} }
taos_free_result(pRes); taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
} }
// TEST(clientCase, drop_stable_Test) {
// TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
// assert(pConn != nullptr);
//
// TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1");
// if (taos_errno(pRes) != 0) {
// (void)printf("error in creating db, reason:%s\n", taos_errstr(pRes));
// }
// taos_free_result(pRes);
//
// pRes = taos_query(pConn, "use abc1");
// if (taos_errno(pRes) != 0) {
// (void)printf("error in using db, reason:%s\n", taos_errstr(pRes));
// }
// taos_free_result(pRes);
//
// pRes = taos_query(pConn, "drop stable st1");
// if (taos_errno(pRes) != 0) {
// (void)printf("failed to drop stable, reason:%s\n", taos_errstr(pRes));
// }
//
// taos_free_result(pRes);
// taos_close(pConn);
// }
TEST(clientCase, generated_request_id_test) { TEST(clientCase, generated_request_id_test) {
SHashObj* phash = taosHashInit(10000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_ENTRY_LOCK); SHashObj* phash = taosHashInit(10000, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_ENTRY_LOCK);
@ -832,11 +791,6 @@ TEST(clientCase, projection_query_tables) {
TAOS_RES* pRes = NULL; TAOS_RES* pRes = NULL;
// TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 1");
// if (taos_errno(pRes) != 0) {
// (void)printf("error in create db, reason:%s\n", taos_errstr(pRes));
// }
// taos_free_result(pRes);
pRes= taos_query(pConn, "use abc1"); pRes= taos_query(pConn, "use abc1");
taos_free_result(pRes); taos_free_result(pRes);
@ -846,6 +800,12 @@ TEST(clientCase, projection_query_tables) {
} }
taos_free_result(pRes); taos_free_result(pRes);
pRes = taos_query(pConn, "create table st2 (ts timestamp, v1 int) tags(t1 int)");
if (taos_errno(pRes) != 0) {
(void)printf("failed to create table tu, reason:%s\n", taos_errstr(pRes));
}
taos_free_result(pRes);
int64_t start = 1685959190000; int64_t start = 1685959190000;
const char* pstr = const char* pstr =
"abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefgh" "abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyzabcdefgh"
@ -861,7 +821,7 @@ TEST(clientCase, projection_query_tables) {
TAOS_RES* px = taos_query(pConn, str); TAOS_RES* px = taos_query(pConn, str);
if (taos_errno(px) != 0) { if (taos_errno(px) != 0) {
(void)printf("failed to create table tu, reason:%s\n", taos_errstr(pRes)); (void)printf("failed to create table tu, reason:%s\n", taos_errstr(px));
} }
taos_free_result(px); taos_free_result(px);
} }
@ -869,26 +829,26 @@ TEST(clientCase, projection_query_tables) {
for(int32_t j = 0; j < 1; ++j) { for(int32_t j = 0; j < 1; ++j) {
start += 20; start += 20;
for (int32_t i = 0; i < 1; ++i) { for (int32_t i = 0; i < 1; ++i) {
createNewTable(pConn, i, 100000, 0, pstr); createNewTable(pConn, i, 10000, 0, pstr);
} }
} }
pRes = taos_query(pConn, "select * from abc1.st2"); pRes = taos_query(pConn, "select * from abc1.st2");
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
(void)printf("failed to select from table, reason:%s\n", taos_errstr(pRes)); (void)printf("failed to select from table, reason:%s\n", taos_errstr(pRes));
taos_free_result(pRes); taos_free_result(pRes);
ASSERT_TRUE(false); ASSERT_TRUE(false);
} }
TAOS_ROW pRow = NULL; TAOS_ROW pRow = NULL;
TAOS_FIELD* pFields = taos_fetch_fields(pRes); TAOS_FIELD* pFields = taos_fetch_fields(pRes);
int32_t numOfFields = taos_num_fields(pRes); int32_t numOfFields = taos_num_fields(pRes);
char str[512] = {0}; char str[512] = {0};
while ((pRow = taos_fetch_row(pRes)) != NULL) { while ((pRow = taos_fetch_row(pRes)) != NULL) {
// int32_t code = taos_print_row(str, pRow, pFields, numOfFields); // int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
// (void)printf("%s\n", str); // (void)printf("%s\n", str);
} }
taos_free_result(pRes); taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
@ -900,7 +860,9 @@ TEST(clientCase, tsbs_perf_test) {
for (int32_t i = 0; i < numOfThreads; ++i) { for (int32_t i = 0; i < numOfThreads; ++i) {
(void)taosThreadCreate(&qid[i], NULL, queryThread, NULL); (void)taosThreadCreate(&qid[i], NULL, queryThread, NULL);
} }
getchar(); for (int32_t i = 0; i < numOfThreads; ++i) {
(void)taosThreadJoin(qid[i], NULL);
}
} }
TEST(clientCase, projection_query_stables) { TEST(clientCase, projection_query_stables) {
@ -908,14 +870,6 @@ TEST(clientCase, projection_query_stables) {
ASSERT_NE(pConn, nullptr); ASSERT_NE(pConn, nullptr);
TAOS_RES* pRes = taos_query(pConn, "explain select * from dbvg.st where tbname='ct1'"); TAOS_RES* pRes = taos_query(pConn, "explain select * from dbvg.st where tbname='ct1'");
// taos_free_result(pRes);
// pRes = taos_query(pConn, "select * from st2");
// if (taos_errno(pRes) != 0) {
// (void)printf("failed to select from table, reason:%s\n", taos_errstr(pRes));
// taos_free_result(pRes);
// ASSERT_TRUE(false);
// }
TAOS_ROW pRow = NULL; TAOS_ROW pRow = NULL;
TAOS_FIELD* pFields = taos_fetch_fields(pRes); TAOS_FIELD* pFields = taos_fetch_fields(pRes);
@ -939,12 +893,6 @@ TEST(clientCase, projection_query_stables) {
} }
//(void)printf("%d\n", i); //(void)printf("%d\n", i);
} }
// while ((pRow = taos_fetch_row(pRes)) != NULL) {
// int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
// if (i++ % 100000 == 0) {
// (void)printf("%d\n", i);
// }
// }
taos_free_result(pRes); taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
@ -971,55 +919,41 @@ TEST(clientCase, agg_query_tables) {
pRes = taos_query(pConn, s); pRes = taos_query(pConn, s);
int32_t ret = taos_errno(pRes); int32_t ret = taos_errno(pRes);
taos_free_result(pRes);
if (ret == 0) { if (ret != 0) {
break; (void)printf("failed to insert into table, reason:%s\n", taos_errstr(pRes));
} }
taos_free_result(pRes);
break;
} }
// while (1) {
// sprintf(s, "insert into t2 values(%ld, %d)", st + i, i);
// pRes = taos_query(pConn, s);
// int32_t ret = taos_errno(pRes);
//
// taos_free_result(pRes);
// if (ret == 0) {
// break;
// }
// }
} }
// pRes = taos_query(pConn, "show table distributed tup");
// if (taos_errno(pRes) != 0) {
// (void)printf("failed to select from table, reason:%s\n", taos_errstr(pRes));
// taos_free_result(pRes);
// ASSERT_TRUE(false);
// }
printResult(pRes);
taos_free_result(pRes);
taos_close(pConn); taos_close(pConn);
} }
/* // --- copy the following script in the shell to setup the environment ---
--- copy the following script in the shell to setup the environment --- //
// create database test;
// use test;
// create table m1(ts timestamp, k int) tags(a int);
// create table tm0 using m1 tags(1);
// create table tm1 using m1 tags(2);
// insert into tm0 values('2021-1-1 1:1:1.120', 1) ('2021-1-1 1:1:2.9', 2) tm1 values('2021-1-1 1:1:1.120', 11) ('2021-1-1
// 1:1:2.99', 22);
create database test;
use test;
create table m1(ts timestamp, k int) tags(a int);
create table tm0 using m1 tags(1);
create table tm1 using m1 tags(2);
insert into tm0 values('2021-1-1 1:1:1.120', 1) ('2021-1-1 1:1:2.9', 2) tm1 values('2021-1-1 1:1:1.120', 11) ('2021-1-1
1:1:2.99', 22);
*/
TEST(clientCase, async_api_test) { TEST(clientCase, async_api_test) {
TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
ASSERT_NE(pConn, nullptr); ASSERT_NE(pConn, nullptr);
(void)taos_query(pConn, "use abc1"); TAOS_RES* pRes = taos_query(pConn, "use abc1");
if (taos_errno(pRes) != 0) {
(void)printf("failed to use db, reason:%s\n", taos_errstr(pRes));
taos_free_result(pRes);
taos_close(pConn);
return;
}
taos_free_result(pRes);
TAOS_RES* pRes = taos_query(pConn, "insert into tu(ts) values('2022-02-27 12:12:61')"); pRes = taos_query(pConn, "insert into tu(ts) values('2022-02-27 12:12:61')");
if (taos_errno(pRes) != 0) { if (taos_errno(pRes) != 0) {
(void)printf("failed, reason:%s\n", taos_errstr(pRes)); (void)printf("failed, reason:%s\n", taos_errstr(pRes));
} }
@ -1041,9 +975,12 @@ TEST(clientCase, async_api_test) {
(void)printf("%s\n", str); (void)printf("%s\n", str);
(void)memset(str, 0, sizeof(str)); (void)memset(str, 0, sizeof(str));
} }
taos_free_result(pRes);
taos_query_a(pConn, "select count(*) from tu", queryCallback, pConn); tsem_t sem;
(void)getchar(); (void)tsem_init(&sem, 0, 0);
taos_query_a(pConn, "select count(*) from tu", queryCallback, &sem);
tsem_wait(&sem);
tsem_destroy(&sem);
taos_close(pConn); taos_close(pConn);
} }
@ -1055,6 +992,7 @@ TEST(clientCase, update_test) {
if (taos_errno(pRes) != TSDB_CODE_SUCCESS) { if (taos_errno(pRes) != TSDB_CODE_SUCCESS) {
(void)printf("failed to create database, code:%s", taos_errstr(pRes)); (void)printf("failed to create database, code:%s", taos_errstr(pRes));
taos_free_result(pRes); taos_free_result(pRes);
taos_close(pConn);
return; return;
} }
@ -1064,6 +1002,7 @@ TEST(clientCase, update_test) {
if (taos_errno(pRes) != TSDB_CODE_SUCCESS) { if (taos_errno(pRes) != TSDB_CODE_SUCCESS) {
(void)printf("failed to use db, code:%s", taos_errstr(pRes)); (void)printf("failed to use db, code:%s", taos_errstr(pRes));
taos_free_result(pRes); taos_free_result(pRes);
taos_close(pConn);
return; return;
} }
taos_free_result(pRes); taos_free_result(pRes);
@ -1076,11 +1015,12 @@ TEST(clientCase, update_test) {
taos_free_result(pRes); taos_free_result(pRes);
char s[256] = {0}; char s[256] = {0};
for (int32_t i = 0; i < 17000; ++i) { for (int32_t i = 0; i < 10; ++i) {
(void)sprintf(s, "insert into tup values(now+%da, %d)", i, i); (void)sprintf(s, "insert into tup values(now+%da, %d)", i, i);
pRes = taos_query(pConn, s); pRes = taos_query(pConn, s);
taos_free_result(pRes); taos_free_result(pRes);
} }
taos_close(pConn);
} }
TEST(clientCase, sub_db_test) { TEST(clientCase, sub_db_test) {
@ -1158,10 +1098,13 @@ TEST(clientCase, sub_db_test) {
(void)printf("precision: %d, row content: %s\n", precision, buf); (void)printf("precision: %d, row content: %s\n", precision, buf);
} }
taos_free_result(pRes); taos_free_result(pRes);
} else {
break;
} }
} }
(void)fprintf(stderr, "%d msg consumed, include %d rows\n", msgCnt, totalRows); (void)fprintf(stderr, "%d msg consumed, include %d rows\n", msgCnt, totalRows);
taos_close(pConn);
} }
TEST(clientCase, tmq_commit) { TEST(clientCase, tmq_commit) {
@ -1550,38 +1493,125 @@ TEST(clientCase, sub_tb_mt_test) {
} }
} }
//static void concatStrings(SArray *list, char* buf, int size){ TEST(clientCase, timezone_Test) {
// int len = 0; {
// for(int i = 0; i < taosArrayGetSize(list); i++){ // taos_options( TSDB_OPTION_TIMEZONE, "UTC-8");
// char* db = (char*)taosArrayGet(list, i); int code = taos_options(TSDB_OPTION_TIMEZONE, "UTC-8");
// char* dot = strchr(db, '.'); ASSERT_TRUE(code == 0);
// if (dot != NULL) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
// db = dot + 1; ASSERT_NE(pConn, nullptr);
// }
// if (i != 0){ TAOS_RES* pRes = taos_query(pConn, "drop database if exists db1");
// strcat(buf, ","); ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
// len += 1; taos_free_result(pRes);
// }
// int ret = snprintf(buf + len, size - len, "%s", db); pRes = taos_query(pConn, "create database db1");
// if (ret < 0) { ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
// (void)printf("snprintf failed, buf:%s, ret:%d", buf, ret); taos_free_result(pRes);
// break;
// } pRes = taos_query(pConn, "create table db1.t1 (ts timestamp, v int)");
// len += ret; ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
// if (len >= size){ taos_free_result(pRes);
// (void)printf("dbList is truncated, buf:%s, len:%d", buf, len);
// break; char sql[256] = {0};
// } (void)sprintf(sql, "insert into db1.t1 values('2023-09-16 17:00:00', 1)");
// } pRes = taos_query(pConn, sql);
//} ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
// taos_free_result(pRes);
//TEST(clientCase, concat_string_test) {
// SArray* list = taosArrayInit(10, TSDB_DB_FNAME_LEN); pRes = taos_query(pConn, "select * from db1.t1 where ts == '2023-09-16 17:00:00'");
// taosArrayPush(list, "1.db1"); ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
// taosArrayPush(list, "2.db2");
// TAOS_ROW pRow = NULL;
// char buf[32] = {0}; TAOS_FIELD* pFields = taos_fetch_fields(pRes);
// concatStrings(list, buf, sizeof(buf) - 1); int32_t numOfFields = taos_num_fields(pRes);
//}
char str[512] = {0};
int rows = 0;
while ((pRow = taos_fetch_row(pRes)) != NULL) {
rows++;
}
ASSERT_TRUE(rows == 1);
taos_free_result(pRes);
taos_close(pConn);
}
{
// taos_options( TSDB_OPTION_TIMEZONE, "UTC+8");
int code = taos_options(TSDB_OPTION_TIMEZONE, "UTC+8");
ASSERT_TRUE(code == 0);
TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
ASSERT_NE(pConn, nullptr);
TAOS_RES* pRes = taos_query(pConn, "select * from db1.t1 where ts == '2023-09-16 01:00:00'");
ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
TAOS_ROW pRow = NULL;
TAOS_FIELD* pFields = taos_fetch_fields(pRes);
int32_t numOfFields = taos_num_fields(pRes);
int rows = 0;
char str[512] = {0};
while ((pRow = taos_fetch_row(pRes)) != NULL) {
rows++;
}
ASSERT_TRUE(rows == 1);
taos_free_result(pRes);
char sql[256] = {0};
(void)sprintf(sql, "insert into db1.t1 values('2023-09-16 17:00:01', 1)");
pRes = taos_query(pConn, sql);
ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
taos_free_result(pRes);
taos_close(pConn);
}
{
// taos_options( TSDB_OPTION_TIMEZONE, "UTC+0");
int code = taos_options(TSDB_OPTION_TIMEZONE, "UTC+0");
ASSERT_TRUE(code == 0);
TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0);
ASSERT_NE(pConn, nullptr);
TAOS_RES* pRes = taos_query(pConn, "select * from db1.t1 where ts == '2023-09-16 09:00:00'");
ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
TAOS_ROW pRow = NULL;
TAOS_FIELD* pFields = taos_fetch_fields(pRes);
int32_t numOfFields = taos_num_fields(pRes);
int rows = 0;
char str[512] = {0};
while ((pRow = taos_fetch_row(pRes)) != NULL) {
rows++;
}
ASSERT_TRUE(rows == 1);
taos_free_result(pRes);
{
TAOS_RES* pRes = taos_query(pConn, "select * from db1.t1 where ts == '2023-09-17 01:00:01'");
ASSERT_EQ(taos_errno(pRes), TSDB_CODE_SUCCESS);
TAOS_ROW pRow = NULL;
TAOS_FIELD* pFields = taos_fetch_fields(pRes);
int32_t numOfFields = taos_num_fields(pRes);
int rows = 0;
char str[512] = {0};
while ((pRow = taos_fetch_row(pRes)) != NULL) {
rows++;
}
ASSERT_TRUE(rows == 1);
taos_free_result(pRes);
}
taos_close(pConn);
}
}
#pragma GCC diagnostic pop #pragma GCC diagnostic pop

View File

@ -5,14 +5,25 @@
#include "tdef.h" #include "tdef.h"
#include "tutil.h" #include "tutil.h"
extern char tsS3Endpoint[]; extern int8_t tsS3EpNum;
extern char tsS3AccessKeyId[]; extern char tsS3Endpoint[][TSDB_FQDN_LEN];
extern char tsS3AccessKeySecret[]; extern char tsS3AccessKeyId[][TSDB_FQDN_LEN];
extern char tsS3BucketName[]; extern char tsS3AccessKeySecret[][TSDB_FQDN_LEN];
extern char tsS3AppId[]; extern char tsS3BucketName[][TSDB_FQDN_LEN];
extern char tsS3Hostname[]; extern char tsS3AppId[][TSDB_FQDN_LEN];
extern char tsS3Hostname[][TSDB_FQDN_LEN];
extern int8_t tsS3Https; extern int8_t tsS3Https;
static int32_t s3ListBucketByEp(char const *bucketname, int8_t epIndex);
static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *object_name, int64_t offset, int64_t size,
int8_t epIndex);
static int32_t s3DeleteObjectsByEp(const char *object_name[], int nobject, int8_t epIndex);
static SArray *getListByPrefixByEp(const char *prefix, int8_t epIndex);
static int32_t s3GetObjectBlockByEp(const char *object_name, int64_t offset, int64_t size, bool check,
uint8_t **ppBlock, int8_t epIndex);
static int32_t s3GetObjectToFileByEp(const char *object_name, const char *fileName, int8_t epIndex);
static long s3SizeByEp(const char *object_name, int8_t epIndex);
#if defined(USE_S3) #if defined(USE_S3)
#include "libs3.h" #include "libs3.h"
@ -32,7 +43,7 @@ extern int8_t tsS3Oss;
int32_t s3Begin() { int32_t s3Begin() {
S3Status status; S3Status status;
const char *hostname = tsS3Hostname; const char *hostname = tsS3Hostname[0];
const char *env_hn = getenv("S3_HOSTNAME"); const char *env_hn = getenv("S3_HOSTNAME");
if (env_hn) { if (env_hn) {
@ -52,7 +63,7 @@ int32_t s3Begin() {
TAOS_RETURN(TSDB_CODE_SUCCESS); TAOS_RETURN(TSDB_CODE_SUCCESS);
} }
void s3End() { S3_deinitialize(); } void s3End() { (void)S3_deinitialize(); }
int32_t s3Init() { TAOS_RETURN(TSDB_CODE_SUCCESS); /*s3Begin();*/ } int32_t s3Init() { TAOS_RETURN(TSDB_CODE_SUCCESS); /*s3Begin();*/ }
@ -63,6 +74,7 @@ static int32_t s3ListBucket(char const *bucketname);
int32_t s3CheckCfg() { int32_t s3CheckCfg() {
int32_t code = 0, lino = 0; int32_t code = 0, lino = 0;
int8_t i = 0;
if (!tsS3Enabled) { if (!tsS3Enabled) {
(void)fprintf(stderr, "s3 not configured.\n"); (void)fprintf(stderr, "s3 not configured.\n");
@ -75,83 +87,91 @@ int32_t s3CheckCfg() {
TAOS_CHECK_GOTO(code, &lino, _exit); TAOS_CHECK_GOTO(code, &lino, _exit);
} }
// test put for (; i < tsS3EpNum; i++) {
char testdata[17] = "0123456789abcdef"; (void)fprintf(stdout, "test s3 ep: %d/%d.\n", i + 1, tsS3EpNum);
const char *objectname[] = {"s3test.txt"};
char path[PATH_MAX] = {0};
int ds_len = strlen(TD_DIRSEP);
int tmp_len = strlen(tsTempDir);
(void)snprintf(path, PATH_MAX, "%s", tsTempDir); // test put
if (strncmp(tsTempDir + tmp_len - ds_len, TD_DIRSEP, ds_len) != 0) { char testdata[17] = "0123456789abcdef";
(void)snprintf(path + tmp_len, PATH_MAX - tmp_len, "%s", TD_DIRSEP); const char *objectname[] = {"s3test.txt"};
(void)snprintf(path + tmp_len + ds_len, PATH_MAX - tmp_len - ds_len, "%s", objectname[0]); char path[PATH_MAX] = {0};
} else { int ds_len = strlen(TD_DIRSEP);
(void)snprintf(path + tmp_len, PATH_MAX - tmp_len, "%s", objectname[0]); int tmp_len = strlen(tsTempDir);
}
TdFilePtr fp = taosOpenFile(path, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_READ | TD_FILE_TRUNC); (void)snprintf(path, PATH_MAX, "%s", tsTempDir);
if (!fp) { if (strncmp(tsTempDir + tmp_len - ds_len, TD_DIRSEP, ds_len) != 0) {
(void)fprintf(stderr, "failed to open test file: %s.\n", path); (void)snprintf(path + tmp_len, PATH_MAX - tmp_len, "%s", TD_DIRSEP);
// uError("ERROR: %s Failed to open %s", __func__, path); (void)snprintf(path + tmp_len + ds_len, PATH_MAX - tmp_len - ds_len, "%s", objectname[0]);
TAOS_CHECK_GOTO(TAOS_SYSTEM_ERROR(errno), &lino, _exit); } else {
} (void)snprintf(path + tmp_len, PATH_MAX - tmp_len, "%s", objectname[0]);
if (taosWriteFile(fp, testdata, strlen(testdata)) < 0) { }
(void)fprintf(stderr, "failed to write test file: %s.\n", path);
TAOS_CHECK_GOTO(TAOS_SYSTEM_ERROR(errno), &lino, _exit);
}
if (taosFsyncFile(fp) < 0) {
(void)fprintf(stderr, "failed to fsync test file: %s.\n", path);
TAOS_CHECK_GOTO(TAOS_SYSTEM_ERROR(errno), &lino, _exit);
}
(void)taosCloseFile(&fp);
(void)fprintf(stderr, "\nstart to put object: %s, file: %s content: %s\n", objectname[0], path, testdata); TdFilePtr fp = taosOpenFile(path, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_READ | TD_FILE_TRUNC);
code = s3PutObjectFromFileOffset(path, objectname[0], 0, 16); if (!fp) {
if (code != 0) { (void)fprintf(stderr, "failed to open test file: %s.\n", path);
(void)fprintf(stderr, "put object %s : failed.\n", objectname[0]); // uError("ERROR: %s Failed to open %s", __func__, path);
TAOS_CHECK_GOTO(code, &lino, _exit); TAOS_CHECK_GOTO(TAOS_SYSTEM_ERROR(errno), &lino, _exit);
} }
(void)fprintf(stderr, "put object %s: success.\n\n", objectname[0]); if (taosWriteFile(fp, testdata, strlen(testdata)) < 0) {
(void)fprintf(stderr, "failed to write test file: %s.\n", path);
TAOS_CHECK_GOTO(TAOS_SYSTEM_ERROR(errno), &lino, _exit);
}
if (taosFsyncFile(fp) < 0) {
(void)fprintf(stderr, "failed to fsync test file: %s.\n", path);
TAOS_CHECK_GOTO(TAOS_SYSTEM_ERROR(errno), &lino, _exit);
}
(void)taosCloseFile(&fp);
// list buckets (void)fprintf(stderr, "\nstart to put object: %s, file: %s content: %s\n", objectname[0], path, testdata);
(void)fprintf(stderr, "start to list bucket %s by prefix s3.\n", tsS3BucketName); code = s3PutObjectFromFileOffsetByEp(path, objectname[0], 0, 16, i);
code = s3ListBucket(tsS3BucketName); if (code != 0) {
if (code != 0) { (void)fprintf(stderr, "put object %s : failed.\n", objectname[0]);
(void)fprintf(stderr, "listing bucket %s : failed.\n", tsS3BucketName); TAOS_CHECK_GOTO(code, &lino, _exit);
TAOS_CHECK_GOTO(code, &lino, _exit); }
} (void)fprintf(stderr, "put object %s: success.\n\n", objectname[0]);
(void)fprintf(stderr, "listing bucket %s: success.\n\n", tsS3BucketName);
// test range get // list buckets
uint8_t *pBlock = NULL; (void)fprintf(stderr, "start to list bucket %s by prefix s3.\n", tsS3BucketName[i]);
int c_offset = 10; code = s3ListBucketByEp(tsS3BucketName[i], i);
int c_len = 6; if (code != 0) {
(void)fprintf(stderr, "listing bucket %s : failed.\n", tsS3BucketName[i]);
TAOS_CHECK_GOTO(code, &lino, _exit);
}
(void)fprintf(stderr, "listing bucket %s: success.\n\n", tsS3BucketName[i]);
(void)fprintf(stderr, "start to range get object %s offset: %d len: %d.\n", objectname[0], c_offset, c_len); // test range get
code = s3GetObjectBlock(objectname[0], c_offset, c_len, true, &pBlock); uint8_t *pBlock = NULL;
if (code != 0) { int c_offset = 10;
(void)fprintf(stderr, "get object %s : failed.\n", objectname[0]); int c_len = 6;
TAOS_CHECK_GOTO(code, &lino, _exit);
}
char buf[7] = {0};
(void)memcpy(buf, pBlock, c_len);
taosMemoryFree(pBlock);
(void)fprintf(stderr, "object content: %s\n", buf);
(void)fprintf(stderr, "get object %s: success.\n\n", objectname[0]);
// delete test object (void)fprintf(stderr, "start to range get object %s offset: %d len: %d.\n", objectname[0], c_offset, c_len);
(void)fprintf(stderr, "start to delete object: %s.\n", objectname[0]); code = s3GetObjectBlockByEp(objectname[0], c_offset, c_len, true, &pBlock, i);
code = s3DeleteObjects(objectname, 1); if (code != 0) {
if (code != 0) { (void)fprintf(stderr, "get object %s : failed.\n", objectname[0]);
(void)fprintf(stderr, "delete object %s : failed.\n", objectname[0]); TAOS_CHECK_GOTO(code, &lino, _exit);
TAOS_CHECK_GOTO(code, &lino, _exit); }
char buf[7] = {0};
(void)memcpy(buf, pBlock, c_len);
taosMemoryFree(pBlock);
(void)fprintf(stderr, "object content: %s\n", buf);
(void)fprintf(stderr, "get object %s: success.\n\n", objectname[0]);
// delete test object
(void)fprintf(stderr, "start to delete object: %s.\n", objectname[0]);
code = s3DeleteObjectsByEp(objectname, 1, i);
if (code != 0) {
(void)fprintf(stderr, "delete object %s : failed.\n", objectname[0]);
TAOS_CHECK_GOTO(code, &lino, _exit);
}
(void)fprintf(stderr, "delete object %s: success.\n\n", objectname[0]);
} }
(void)fprintf(stderr, "delete object %s: success.\n\n", objectname[0]);
s3End(); s3End();
_exit: _exit:
if (TSDB_CODE_SUCCESS != code) {
(void)fprintf(stderr, "s3 check failed, code: %d, line: %d, index: %d.\n", code, lino, i);
}
TAOS_RETURN(code); TAOS_RETURN(code);
} }
@ -241,6 +261,27 @@ static void responseCompleteCallback(S3Status status, const S3ErrorDetails *erro
static SArray *getListByPrefix(const char *prefix); static SArray *getListByPrefix(const char *prefix);
static void s3FreeObjectKey(void *pItem); static void s3FreeObjectKey(void *pItem);
static int32_t s3ListBucketByEp(char const *bucketname, int8_t epIndex) {
int32_t code = 0;
SArray *objectArray = getListByPrefixByEp("s3", epIndex);
if (objectArray == NULL) {
TAOS_RETURN(TSDB_CODE_FAILED);
}
const char **object_name = TARRAY_DATA(objectArray);
int size = TARRAY_SIZE(objectArray);
(void)fprintf(stderr, "objects:\n");
for (int i = 0; i < size; ++i) {
(void)fprintf(stderr, "%s\n", object_name[i]);
}
taosArrayDestroyEx(objectArray, s3FreeObjectKey);
TAOS_RETURN(code);
}
static int32_t s3ListBucket(char const *bucketname) { static int32_t s3ListBucket(char const *bucketname) {
int32_t code = 0; int32_t code = 0;
@ -906,7 +947,7 @@ _exit:
TAOS_RETURN(code); TAOS_RETURN(code);
} }
int32_t s3PutObjectFromFile2(const char *file, const char *object_name, int8_t withcp) { int32_t s3PutObjectFromFile2ByEp(const char *file, const char *object_name, int8_t withcp, int8_t epIndex) {
int32_t code = 0; int32_t code = 0;
int32_t lmtime = 0; int32_t lmtime = 0;
const char *filename = 0; const char *filename = 0;
@ -933,8 +974,14 @@ int32_t s3PutObjectFromFile2(const char *file, const char *object_name, int8_t w
data.totalContentLength = data.totalOriginalContentLength = data.contentLength = data.originalContentLength = data.totalContentLength = data.totalOriginalContentLength = data.contentLength = data.originalContentLength =
contentLength; contentLength;
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3PutProperties putProperties = {contentType, md5, S3PutProperties putProperties = {contentType, md5,
cacheControl, contentDispositionFilename, cacheControl, contentDispositionFilename,
@ -961,7 +1008,23 @@ int32_t s3PutObjectFromFile2(const char *file, const char *object_name, int8_t w
TAOS_RETURN(code); TAOS_RETURN(code);
} }
int32_t s3PutObjectFromFileOffset(const char *file, const char *object_name, int64_t offset, int64_t size) { int32_t s3PutObjectFromFile2(const char *file, const char *object_name, int8_t withcp) {
int32_t code = TSDB_CODE_SUCCESS;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
code = s3PutObjectFromFile2ByEp(file, object_name, withcp, epIndex);
if (code == TSDB_CODE_SUCCESS) {
break;
}
}
return code;
}
static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *object_name, int64_t offset, int64_t size,
int8_t epIndex) {
int32_t code = 0; int32_t code = 0;
int32_t lmtime = 0; int32_t lmtime = 0;
const char *filename = 0; const char *filename = 0;
@ -994,8 +1057,14 @@ int32_t s3PutObjectFromFileOffset(const char *file, const char *object_name, int
data.totalContentLength = data.totalOriginalContentLength = data.contentLength = data.originalContentLength = data.totalContentLength = data.totalOriginalContentLength = data.contentLength = data.originalContentLength =
contentLength; contentLength;
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3PutProperties putProperties = {contentType, md5, S3PutProperties putProperties = {contentType, md5,
cacheControl, contentDispositionFilename, cacheControl, contentDispositionFilename,
@ -1018,6 +1087,21 @@ int32_t s3PutObjectFromFileOffset(const char *file, const char *object_name, int
TAOS_RETURN(code); TAOS_RETURN(code);
} }
int32_t s3PutObjectFromFileOffset(const char *file, const char *object_name, int64_t offset, int64_t size) {
int32_t code = TSDB_CODE_SUCCESS;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
code = s3PutObjectFromFileOffsetByEp(file, object_name, offset, size, epIndex);
if (code == TSDB_CODE_SUCCESS) {
break;
}
}
return code;
}
typedef struct list_bucket_callback_data { typedef struct list_bucket_callback_data {
char err_msg[512]; char err_msg[512];
S3Status status; S3Status status;
@ -1068,9 +1152,15 @@ static void s3FreeObjectKey(void *pItem) {
taosMemoryFree(key); taosMemoryFree(key);
} }
static SArray *getListByPrefix(const char *prefix) { static SArray *getListByPrefixByEp(const char *prefix, int8_t epIndex) {
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3ListBucketHandler listBucketHandler = {{&responsePropertiesCallbackNull, &responseCompleteCallback}, S3ListBucketHandler listBucketHandler = {{&responsePropertiesCallbackNull, &responseCompleteCallback},
&listBucketCallback}; &listBucketCallback};
@ -1114,11 +1204,31 @@ static SArray *getListByPrefix(const char *prefix) {
return NULL; return NULL;
} }
int32_t s3DeleteObjects(const char *object_name[], int nobject) { static SArray *getListByPrefix(const char *prefix) {
SArray *objectArray = NULL;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
objectArray = getListByPrefixByEp(prefix, epIndex);
if (objectArray) {
break;
}
}
return objectArray;
}
static int32_t s3DeleteObjectsByEp(const char *object_name[], int nobject, int8_t epIndex) {
int32_t code = 0; int32_t code = 0;
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3ResponseHandler responseHandler = {0, &responseCompleteCallback}; S3ResponseHandler responseHandler = {0, &responseCompleteCallback};
for (int i = 0; i < nobject; ++i) { for (int i = 0; i < nobject; ++i) {
@ -1136,6 +1246,21 @@ int32_t s3DeleteObjects(const char *object_name[], int nobject) {
TAOS_RETURN(code); TAOS_RETURN(code);
} }
int32_t s3DeleteObjects(const char *object_name[], int nobject) {
int32_t code = 0;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
code = s3DeleteObjectsByEp(object_name, nobject, epIndex);
if (code == TSDB_CODE_SUCCESS) {
break;
}
}
return code;
}
void s3DeleteObjectsByPrefix(const char *prefix) { void s3DeleteObjectsByPrefix(const char *prefix) {
SArray *objectArray = getListByPrefix(prefix); SArray *objectArray = getListByPrefix(prefix);
if (objectArray == NULL) return; if (objectArray == NULL) return;
@ -1166,13 +1291,20 @@ static S3Status getObjectDataCallback(int bufferSize, const char *buffer, void *
} }
} }
int32_t s3GetObjectBlock(const char *object_name, int64_t offset, int64_t size, bool check, uint8_t **ppBlock) { static int32_t s3GetObjectBlockByEp(const char *object_name, int64_t offset, int64_t size, bool check,
uint8_t **ppBlock, int8_t epIndex) {
int status = 0; int status = 0;
int64_t ifModifiedSince = -1, ifNotModifiedSince = -1; int64_t ifModifiedSince = -1, ifNotModifiedSince = -1;
const char *ifMatch = 0, *ifNotMatch = 0; const char *ifMatch = 0, *ifNotMatch = 0;
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3GetConditions getConditions = {ifModifiedSince, ifNotModifiedSince, ifMatch, ifNotMatch}; S3GetConditions getConditions = {ifModifiedSince, ifNotModifiedSince, ifMatch, ifNotMatch};
S3GetObjectHandler getObjectHandler = {{&responsePropertiesCallback, &responseCompleteCallback}, S3GetObjectHandler getObjectHandler = {{&responsePropertiesCallback, &responseCompleteCallback},
&getObjectDataCallback}; &getObjectDataCallback};
@ -1213,18 +1345,39 @@ _retry:
TAOS_RETURN(TSDB_CODE_SUCCESS); TAOS_RETURN(TSDB_CODE_SUCCESS);
} }
int32_t s3GetObjectBlock(const char *object_name, int64_t offset, int64_t size, bool check, uint8_t **ppBlock) {
int32_t code = 0;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
code = s3GetObjectBlockByEp(object_name, offset, size, check, ppBlock, epIndex);
if (code == TSDB_CODE_SUCCESS) {
break;
}
}
return code;
}
static S3Status getObjectCallback(int bufferSize, const char *buffer, void *callbackData) { static S3Status getObjectCallback(int bufferSize, const char *buffer, void *callbackData) {
TS3GetData *cbd = (TS3GetData *)callbackData; TS3GetData *cbd = (TS3GetData *)callbackData;
size_t wrote = taosWriteFile(cbd->file, buffer, bufferSize); size_t wrote = taosWriteFile(cbd->file, buffer, bufferSize);
return ((wrote < (size_t)bufferSize) ? S3StatusAbortedByCallback : S3StatusOK); return ((wrote < (size_t)bufferSize) ? S3StatusAbortedByCallback : S3StatusOK);
} }
int32_t s3GetObjectToFile(const char *object_name, const char *fileName) { static int32_t s3GetObjectToFileByEp(const char *object_name, const char *fileName, int8_t epIndex) {
int64_t ifModifiedSince = -1, ifNotModifiedSince = -1; int64_t ifModifiedSince = -1, ifNotModifiedSince = -1;
const char *ifMatch = 0, *ifNotMatch = 0; const char *ifMatch = 0, *ifNotMatch = 0;
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3GetConditions getConditions = {ifModifiedSince, ifNotModifiedSince, ifMatch, ifNotMatch}; S3GetConditions getConditions = {ifModifiedSince, ifNotModifiedSince, ifMatch, ifNotMatch};
S3GetObjectHandler getObjectHandler = {{&responsePropertiesCallbackNull, &responseCompleteCallback}, S3GetObjectHandler getObjectHandler = {{&responsePropertiesCallbackNull, &responseCompleteCallback},
&getObjectCallback}; &getObjectCallback};
@ -1252,6 +1405,21 @@ int32_t s3GetObjectToFile(const char *object_name, const char *fileName) {
TAOS_RETURN(TSDB_CODE_SUCCESS); TAOS_RETURN(TSDB_CODE_SUCCESS);
} }
int32_t s3GetObjectToFile(const char *object_name, const char *fileName) {
int32_t code = 0;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
code = s3GetObjectToFileByEp(object_name, fileName, epIndex);
if (code == TSDB_CODE_SUCCESS) {
break;
}
}
return code;
}
int32_t s3GetObjectsByPrefix(const char *prefix, const char *path) { int32_t s3GetObjectsByPrefix(const char *prefix, const char *path) {
SArray *objectArray = getListByPrefix(prefix); SArray *objectArray = getListByPrefix(prefix);
if (objectArray == NULL) TAOS_RETURN(TSDB_CODE_FAILED); if (objectArray == NULL) TAOS_RETURN(TSDB_CODE_FAILED);
@ -1275,12 +1443,18 @@ int32_t s3GetObjectsByPrefix(const char *prefix, const char *path) {
return 0; return 0;
} }
long s3Size(const char *object_name) { static long s3SizeByEp(const char *object_name, int8_t epIndex) {
long size = 0; long size = 0;
int status = 0; int status = 0;
S3BucketContext bucketContext = {0, tsS3BucketName, protocolG, uriStyleG, tsS3AccessKeyId, tsS3AccessKeySecret, S3BucketContext bucketContext = {0,
0, awsRegionG}; tsS3BucketName[epIndex],
protocolG,
uriStyleG,
tsS3AccessKeyId[epIndex],
tsS3AccessKeySecret[epIndex],
0,
awsRegionG};
S3ResponseHandler responseHandler = {&responsePropertiesCallback, &responseCompleteCallback}; S3ResponseHandler responseHandler = {&responsePropertiesCallback, &responseCompleteCallback};
@ -1300,6 +1474,21 @@ long s3Size(const char *object_name) {
return size; return size;
} }
long s3Size(const char *object_name) {
long size = 0;
int8_t startIndex = taosRand() % tsS3EpNum;
for (int8_t i = 0; i < tsS3EpNum; ++i) {
int8_t epIndex = (startIndex + i) % tsS3EpNum;
size = s3SizeByEp(object_name, epIndex);
if (size > 0) {
break;
}
}
return size;
}
void s3EvictCache(const char *path, long object_size) {} void s3EvictCache(const char *path, long object_size) {}
#elif defined(USE_COS) #elif defined(USE_COS)

View File

@ -1737,7 +1737,11 @@ int32_t copyDataBlock(SSDataBlock* pDst, const SSDataBlock* pSrc) {
uint32_t cap = pDst->info.capacity; uint32_t cap = pDst->info.capacity;
pDst->info = pSrc->info; pDst->info = pSrc->info;
copyPkVal(&pDst->info, &pSrc->info); code = copyPkVal(&pDst->info, &pSrc->info);
if (code != TSDB_CODE_SUCCESS) {
uError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
return code;
}
pDst->info.capacity = cap; pDst->info.capacity = cap;
return code; return code;
@ -1902,9 +1906,11 @@ int32_t blockCopyOneRow(const SSDataBlock* pDataBlock, int32_t rowIdx, SSDataBlo
return code; return code;
} }
void copyPkVal(SDataBlockInfo* pDst, const SDataBlockInfo* pSrc) { int32_t copyPkVal(SDataBlockInfo* pDst, const SDataBlockInfo* pSrc) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (!IS_VAR_DATA_TYPE(pSrc->pks[0].type)) { if (!IS_VAR_DATA_TYPE(pSrc->pks[0].type)) {
return; return code;
} }
// prepare the pk buffer if needed // prepare the pk buffer if needed
@ -1912,14 +1918,24 @@ void copyPkVal(SDataBlockInfo* pDst, const SDataBlockInfo* pSrc) {
p->type = pDst->pks[0].type; p->type = pDst->pks[0].type;
p->pData = taosMemoryCalloc(1, pDst->pks[0].nData); p->pData = taosMemoryCalloc(1, pDst->pks[0].nData);
QUERY_CHECK_NULL(p->pData, code, lino, _end, terrno);
p->nData = pDst->pks[0].nData; p->nData = pDst->pks[0].nData;
memcpy(p->pData, pDst->pks[0].pData, p->nData); memcpy(p->pData, pDst->pks[0].pData, p->nData);
p = &pDst->pks[1]; p = &pDst->pks[1];
p->type = pDst->pks[1].type; p->type = pDst->pks[1].type;
p->pData = taosMemoryCalloc(1, pDst->pks[1].nData); p->pData = taosMemoryCalloc(1, pDst->pks[1].nData);
QUERY_CHECK_NULL(p->pData, code, lino, _end, terrno);
p->nData = pDst->pks[1].nData; p->nData = pDst->pks[1].nData;
memcpy(p->pData, pDst->pks[1].pData, p->nData); memcpy(p->pData, pDst->pks[1].pData, p->nData);
_end:
if (code != TSDB_CODE_SUCCESS) {
uError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
} }
int32_t createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData, SSDataBlock** pResBlock) { int32_t createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData, SSDataBlock** pResBlock) {
@ -1958,7 +1974,11 @@ int32_t createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData, SSDataB
} }
} }
copyPkVal(&pDstBlock->info, &pDataBlock->info); code = copyPkVal(&pDstBlock->info, &pDataBlock->info);
if (code != TSDB_CODE_SUCCESS) {
uError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
return code;
}
if (copyData) { if (copyData) {
code = blockDataEnsureCapacity(pDstBlock, pDataBlock->info.rows); code = blockDataEnsureCapacity(pDstBlock, pDataBlock->info.rows);

View File

@ -294,19 +294,20 @@ bool tsFilterScalarMode = false;
int tsResolveFQDNRetryTime = 100; // seconds int tsResolveFQDNRetryTime = 100; // seconds
int tsStreamAggCnt = 100000; int tsStreamAggCnt = 100000;
char tsS3Endpoint[TSDB_FQDN_LEN] = "<endpoint>"; int8_t tsS3EpNum = 0;
char tsS3AccessKey[TSDB_FQDN_LEN] = "<accesskey>"; char tsS3Endpoint[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<endpoint>"};
char tsS3AccessKeyId[TSDB_FQDN_LEN] = "<accesskeyid>"; char tsS3AccessKey[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<accesskey>"};
char tsS3AccessKeySecret[TSDB_FQDN_LEN] = "<accesskeysecrect>"; char tsS3AccessKeyId[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<accesskeyid>"};
char tsS3BucketName[TSDB_FQDN_LEN] = "<bucketname>"; char tsS3AccessKeySecret[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<accesskeysecrect>"};
char tsS3AppId[TSDB_FQDN_LEN] = "<appid>"; char tsS3BucketName[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<bucketname>"};
char tsS3AppId[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<appid>"};
int8_t tsS3Enabled = false; int8_t tsS3Enabled = false;
int8_t tsS3EnabledCfg = false; int8_t tsS3EnabledCfg = false;
int8_t tsS3Oss = false; int8_t tsS3Oss = false;
int8_t tsS3StreamEnabled = false; int8_t tsS3StreamEnabled = false;
int8_t tsS3Https = true; int8_t tsS3Https = true;
char tsS3Hostname[TSDB_FQDN_LEN] = "<hostname>"; char tsS3Hostname[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN] = {"<hostname>"};
int32_t tsS3BlockSize = -1; // number of tsdb pages (4096) int32_t tsS3BlockSize = -1; // number of tsdb pages (4096)
int32_t tsS3BlockCacheSize = 16; // number of blocks int32_t tsS3BlockCacheSize = 16; // number of blocks
@ -349,49 +350,89 @@ int32_t taosSetTfsCfg(SConfig *pCfg) {
int32_t taosSetTfsCfg(SConfig *pCfg); int32_t taosSetTfsCfg(SConfig *pCfg);
#endif #endif
int32_t taosSetS3Cfg(SConfig *pCfg) { static int32_t taosSplitS3Cfg(SConfig *pCfg, const char *name, char gVarible[TSDB_MAX_EP_NUM][TSDB_FQDN_LEN],
SConfigItem *pItem = NULL; int8_t *pNum) {
int32_t code = TSDB_CODE_SUCCESS;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3Accesskey"); SConfigItem *pItem = NULL;
tstrncpy(tsS3AccessKey, pItem->str, TSDB_FQDN_LEN); int32_t num = 0;
if (tsS3AccessKey[0] == '<') { TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, name);
char *strDup = NULL;
if ((strDup = taosStrdup(pItem->str))== NULL){
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
char **slices = strsplit(strDup, ",", &num);
if (num > TSDB_MAX_EP_NUM) {
code = TSDB_CODE_INVALID_CFG;
goto _exit;
}
for (int i = 0; i < num; ++i) {
tstrncpy(gVarible[i], slices[i], TSDB_FQDN_LEN);
}
*pNum = num;
_exit:
taosMemoryFreeClear(slices);
taosMemoryFreeClear(strDup);
TAOS_RETURN(code);
}
int32_t taosSetS3Cfg(SConfig *pCfg) {
int8_t num = 0;
TAOS_CHECK_RETURN(taosSplitS3Cfg(pCfg, "s3Accesskey", tsS3AccessKey, &num));
if (num == 0) TAOS_RETURN(TSDB_CODE_SUCCESS);
tsS3EpNum = num;
if (tsS3AccessKey[0][0] == '<') {
TAOS_RETURN(TSDB_CODE_SUCCESS); TAOS_RETURN(TSDB_CODE_SUCCESS);
} }
char *colon = strchr(tsS3AccessKey, ':'); for (int i = 0; i < tsS3EpNum; ++i) {
if (!colon) { char *colon = strchr(tsS3AccessKey[i], ':');
uError("invalid access key:%s", tsS3AccessKey); if (!colon) {
TAOS_RETURN(TSDB_CODE_INVALID_CFG); uError("invalid access key:%s", tsS3AccessKey[i]);
} TAOS_RETURN(TSDB_CODE_INVALID_CFG);
*colon = '\0'; }
tstrncpy(tsS3AccessKeyId, tsS3AccessKey, TSDB_FQDN_LEN); *colon = '\0';
tstrncpy(tsS3AccessKeySecret, colon + 1, TSDB_FQDN_LEN); tstrncpy(tsS3AccessKeyId[i], tsS3AccessKey[i], TSDB_FQDN_LEN);
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3Endpoint"); tstrncpy(tsS3AccessKeySecret[i], colon + 1, TSDB_FQDN_LEN);
tstrncpy(tsS3Endpoint, pItem->str, TSDB_FQDN_LEN);
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3BucketName");
tstrncpy(tsS3BucketName, pItem->str, TSDB_FQDN_LEN);
char *proto = strstr(tsS3Endpoint, "https://");
if (!proto) {
tsS3Https = false;
tstrncpy(tsS3Hostname, tsS3Endpoint + 7, TSDB_FQDN_LEN);
} else {
tstrncpy(tsS3Hostname, tsS3Endpoint + 8, TSDB_FQDN_LEN);
} }
char *cos = strstr(tsS3Endpoint, "cos."); TAOS_CHECK_RETURN(taosSplitS3Cfg(pCfg, "s3Endpoint", tsS3Endpoint, &num));
if (cos) { if (num != tsS3EpNum) TAOS_RETURN(TSDB_CODE_INVALID_CFG);
char *appid = strrchr(tsS3BucketName, '-');
if (!appid) { TAOS_CHECK_RETURN(taosSplitS3Cfg(pCfg, "s3BucketName", tsS3BucketName, &num));
uError("failed to locate appid in bucket:%s", tsS3BucketName); if (num != tsS3EpNum) TAOS_RETURN(TSDB_CODE_INVALID_CFG);
TAOS_RETURN(TSDB_CODE_INVALID_CFG);
for (int i = 0; i < tsS3EpNum; ++i) {
char *proto = strstr(tsS3Endpoint[i], "https://");
if (!proto) {
tstrncpy(tsS3Hostname[i], tsS3Endpoint[i] + 7, TSDB_FQDN_LEN);
} else { } else {
tstrncpy(tsS3AppId, appid + 1, TSDB_FQDN_LEN); tstrncpy(tsS3Hostname[i], tsS3Endpoint[i] + 8, TSDB_FQDN_LEN);
}
char *cos = strstr(tsS3Endpoint[i], "cos.");
if (cos) {
char *appid = strrchr(tsS3BucketName[i], '-');
if (!appid) {
uError("failed to locate appid in bucket:%s", tsS3BucketName[i]);
TAOS_RETURN(TSDB_CODE_INVALID_CFG);
} else {
tstrncpy(tsS3AppId[i], appid + 1, TSDB_FQDN_LEN);
}
} }
} }
char *oss = strstr(tsS3Endpoint, "aliyuncs.");
if (oss) { tsS3Https = (strstr(tsS3Endpoint[0], "https://") != NULL);
tsS3Oss = true; tsS3Oss = (strstr(tsS3Endpoint[0], "aliyuncs.") != NULL);
}
if (tsS3BucketName[0] != '<') { if (tsS3BucketName[0][0] != '<') {
#if defined(USE_COS) || defined(USE_S3) #if defined(USE_COS) || defined(USE_S3)
#ifdef TD_ENTERPRISE #ifdef TD_ENTERPRISE
/*if (tsDiskCfgNum > 1) */ tsS3Enabled = true; /*if (tsDiskCfgNum > 1) */ tsS3Enabled = true;
@ -650,8 +691,8 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
tsNumOfSnodeWriteThreads = tsNumOfCores / 4; tsNumOfSnodeWriteThreads = tsNumOfCores / 4;
tsNumOfSnodeWriteThreads = TRANGE(tsNumOfSnodeWriteThreads, 2, 4); tsNumOfSnodeWriteThreads = TRANGE(tsNumOfSnodeWriteThreads, 2, 4);
tsRpcQueueMemoryAllowed = tsTotalMemoryKB * 1024 * 0.1; tsQueueMemoryAllowed = tsTotalMemoryKB * 1024 * 0.1;
tsRpcQueueMemoryAllowed = TRANGE(tsRpcQueueMemoryAllowed, TSDB_MAX_MSG_SIZE * 10LL, TSDB_MAX_MSG_SIZE * 10000LL); tsQueueMemoryAllowed = TRANGE(tsQueueMemoryAllowed, TSDB_MAX_MSG_SIZE * 10LL, TSDB_MAX_MSG_SIZE * 10000LL);
// clang-format off // clang-format off
TAOS_CHECK_RETURN(cfgAddDir(pCfg, "dataDir", tsDataDir, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddDir(pCfg, "dataDir", tsDataDir, CFG_SCOPE_SERVER, CFG_DYN_NONE));
@ -681,7 +722,7 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "numOfSnodeSharedThreads", tsNumOfSnodeStreamThreads, 2, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "numOfSnodeSharedThreads", tsNumOfSnodeStreamThreads, 2, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "numOfSnodeUniqueThreads", tsNumOfSnodeWriteThreads, 2, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "numOfSnodeUniqueThreads", tsNumOfSnodeWriteThreads, 2, 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt64(pCfg, "rpcQueueMemoryAllowed", tsRpcQueueMemoryAllowed, TSDB_MAX_MSG_SIZE * 10L, INT64_MAX, CFG_SCOPE_BOTH, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt64(pCfg, "rpcQueueMemoryAllowed", tsQueueMemoryAllowed, TSDB_MAX_MSG_SIZE * 10L, INT64_MAX, CFG_SCOPE_BOTH, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "syncElectInterval", tsElectInterval, 10, 1000 * 60 * 24 * 2, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "syncElectInterval", tsElectInterval, 10, 1000 * 60 * 24 * 2, CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "syncHeartbeatInterval", tsHeartbeatInterval, 10, 1000 * 60 * 24 * 2, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "syncHeartbeatInterval", tsHeartbeatInterval, 10, 1000 * 60 * 24 * 2, CFG_SCOPE_SERVER, CFG_DYN_NONE));
@ -734,6 +775,7 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "compactPullupInterval", tsCompactPullupInterval, 1, 10000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "compactPullupInterval", tsCompactPullupInterval, 1, 10000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "mqRebalanceInterval", tsMqRebalanceInterval, 1, 10000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "mqRebalanceInterval", tsMqRebalanceInterval, 1, 10000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "randErrorChance", tsRandErrChance, 0, 10000, CFG_SCOPE_BOTH, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "randErrorChance", tsRandErrChance, 0, 10000, CFG_SCOPE_BOTH, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt64(pCfg, "randErrorDivisor", tsRandErrDivisor, 1, INT64_MAX, CFG_SCOPE_BOTH, CFG_DYN_SERVER));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "ttlUnit", tsTtlUnit, 1, 86400 * 365, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "ttlUnit", tsTtlUnit, 1, 86400 * 365, CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "ttlPushInterval", tsTtlPushIntervalSec, 1, 100000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "ttlPushInterval", tsTtlPushIntervalSec, 1, 100000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER));
@ -775,9 +817,9 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "pqSortMemThreshold", tsPQSortMemThreshold, 1, 10240, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "pqSortMemThreshold", tsPQSortMemThreshold, 1, 10240, CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "resolveFQDNRetryTime", tsResolveFQDNRetryTime, 1, 10240, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "resolveFQDNRetryTime", tsResolveFQDNRetryTime, 1, 10240, CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddString(pCfg, "s3Accesskey", tsS3AccessKey, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddString(pCfg, "s3Accesskey", tsS3AccessKey[0], CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddString(pCfg, "s3Endpoint", tsS3Endpoint, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddString(pCfg, "s3Endpoint", tsS3Endpoint[0], CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddString(pCfg, "s3BucketName", tsS3BucketName, CFG_SCOPE_SERVER, CFG_DYN_NONE)); TAOS_CHECK_RETURN(cfgAddString(pCfg, "s3BucketName", tsS3BucketName[0], CFG_SCOPE_SERVER, CFG_DYN_NONE));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "s3PageCacheSize", tsS3PageCacheSize, 4, 1024 * 1024 * 1024, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "s3PageCacheSize", tsS3PageCacheSize, 4, 1024 * 1024 * 1024, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "s3UploadDelaySec", tsS3UploadDelaySec, 1, 60 * 60 * 24 * 30, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER)); TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "s3UploadDelaySec", tsS3UploadDelaySec, 1, 60 * 60 * 24 * 30, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER));
@ -916,9 +958,9 @@ static int32_t taosUpdateServerCfg(SConfig *pCfg) {
pItem = cfgGetItem(pCfg, "rpcQueueMemoryAllowed"); pItem = cfgGetItem(pCfg, "rpcQueueMemoryAllowed");
if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) { if (pItem != NULL && pItem->stype == CFG_STYPE_DEFAULT) {
tsRpcQueueMemoryAllowed = totalMemoryKB * 1024 * 0.1; tsQueueMemoryAllowed = totalMemoryKB * 1024 * 0.1;
tsRpcQueueMemoryAllowed = TRANGE(tsRpcQueueMemoryAllowed, TSDB_MAX_MSG_SIZE * 10LL, TSDB_MAX_MSG_SIZE * 10000LL); tsQueueMemoryAllowed = TRANGE(tsQueueMemoryAllowed, TSDB_MAX_MSG_SIZE * 10LL, TSDB_MAX_MSG_SIZE * 10000LL);
pItem->i64 = tsRpcQueueMemoryAllowed; pItem->i64 = tsQueueMemoryAllowed;
pItem->stype = stype; pItem->stype = stype;
} }
@ -1315,7 +1357,7 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
tsNumOfSnodeWriteThreads = pItem->i32; tsNumOfSnodeWriteThreads = pItem->i32;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "rpcQueueMemoryAllowed"); TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "rpcQueueMemoryAllowed");
tsRpcQueueMemoryAllowed = pItem->i64; tsQueueMemoryAllowed = pItem->i64;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "simdEnable"); TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "simdEnable");
tsSIMDEnable = (bool)pItem->bval; tsSIMDEnable = (bool)pItem->bval;
@ -1427,6 +1469,9 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "randErrorChance"); TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "randErrorChance");
tsRandErrChance = pItem->i32; tsRandErrChance = pItem->i32;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "randErrorDivisor");
tsRandErrDivisor = pItem->i64;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "ttlUnit"); TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "ttlUnit");
tsTtlUnit = pItem->i32; tsTtlUnit = pItem->i32;
@ -1881,6 +1926,7 @@ static int32_t taosCfgDynamicOptionsForServer(SConfig *pCfg, const char *name) {
{"mndSdbWriteDelta", &tsMndSdbWriteDelta}, {"mndSdbWriteDelta", &tsMndSdbWriteDelta},
{"minDiskFreeSize", &tsMinDiskFreeSize}, {"minDiskFreeSize", &tsMinDiskFreeSize},
{"randErrorDivisor", &tsRandErrDivisor},
{"cacheLazyLoadThreshold", &tsCacheLazyLoadThreshold}, {"cacheLazyLoadThreshold", &tsCacheLazyLoadThreshold},
{"checkpointInterval", &tsStreamCheckpointInterval}, {"checkpointInterval", &tsStreamCheckpointInterval},

View File

@ -169,7 +169,7 @@ void tNameAssign(SName* dst, const SName* src) { memcpy(dst, src, sizeof(SName))
int32_t tNameSetDbName(SName* dst, int32_t acct, const char* dbName, size_t nameLen) { int32_t tNameSetDbName(SName* dst, int32_t acct, const char* dbName, size_t nameLen) {
// too long account id or too long db name // too long account id or too long db name
if (nameLen <= 0 || nameLen >= tListLen(dst->dbname)) { if (nameLen <= 0 || nameLen >= tListLen(dst->dbname)) {
return -1; return TSDB_CODE_INVALID_PARA;
} }
dst->type = TSDB_DB_NAME_T; dst->type = TSDB_DB_NAME_T;

View File

@ -124,7 +124,7 @@ void dmSendStatusReq(SDnodeMgmt *pMgmt) {
req.numOfSupportVnodes = tsNumOfSupportVnodes; req.numOfSupportVnodes = tsNumOfSupportVnodes;
req.numOfDiskCfg = tsDiskCfgNum; req.numOfDiskCfg = tsDiskCfgNum;
req.memTotal = tsTotalMemoryKB * 1024; req.memTotal = tsTotalMemoryKB * 1024;
req.memAvail = req.memTotal - tsRpcQueueMemoryAllowed - 16 * 1024 * 1024; req.memAvail = req.memTotal - tsQueueMemoryAllowed - 16 * 1024 * 1024;
tstrncpy(req.dnodeEp, tsLocalEp, TSDB_EP_LEN); tstrncpy(req.dnodeEp, tsLocalEp, TSDB_EP_LEN);
tstrncpy(req.machineId, pMgmt->pData->machineId, TSDB_MACHINE_ID_LEN + 1); tstrncpy(req.machineId, pMgmt->pData->machineId, TSDB_MACHINE_ID_LEN + 1);

View File

@ -218,7 +218,7 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
pRpc->info.wrapper = pWrapper; pRpc->info.wrapper = pWrapper;
EQItype itype = IsReq(pRpc) ? RPC_QITEM : DEF_QITEM; // rsp msg is not restricted by tsRpcQueueMemoryUsed EQItype itype = IsReq(pRpc) ? RPC_QITEM : DEF_QITEM; // rsp msg is not restricted by tsQueueMemoryUsed
code = taosAllocateQitem(sizeof(SRpcMsg), itype, pRpc->contLen, (void **)&pMsg); code = taosAllocateQitem(sizeof(SRpcMsg), itype, pRpc->contLen, (void **)&pMsg);
if (code) goto _OVER; if (code) goto _OVER;

View File

@ -36,7 +36,7 @@ void Testbase::InitLog(const char* path) {
tstrncpy(tsLogDir, path, PATH_MAX); tstrncpy(tsLogDir, path, PATH_MAX);
taosGetSystemInfo(); taosGetSystemInfo();
tsRpcQueueMemoryAllowed = tsTotalMemoryKB * 0.1; tsQueueMemoryAllowed = tsTotalMemoryKB * 0.1;
if (taosInitLog("taosdlog", 1, false) != 0) { if (taosInitLog("taosdlog", 1, false) != 0) {
printf("failed to init log file\n"); printf("failed to init log file\n");
} }

View File

@ -152,6 +152,7 @@ bool streamTaskIterNextTask(SStreamTaskIter *pIter);
int32_t streamTaskIterGetCurrent(SStreamTaskIter *pIter, SStreamTask **pTask); int32_t streamTaskIterGetCurrent(SStreamTaskIter *pIter, SStreamTask **pTask);
int32_t mndInitExecInfo(); int32_t mndInitExecInfo();
void mndInitStreamExecInfo(SMnode *pMnode, SStreamExecInfo *pExecInfo); void mndInitStreamExecInfo(SMnode *pMnode, SStreamExecInfo *pExecInfo);
void mndInitStreamExecInfoForLeader(SMnode *pMnode);
int32_t removeExpiredNodeEntryAndTaskInBuf(SArray *pNodeSnapshot); int32_t removeExpiredNodeEntryAndTaskInBuf(SArray *pNodeSnapshot);
void removeStreamTasksInBuf(SStreamObj *pStream, SStreamExecInfo *pExecNode); void removeStreamTasksInBuf(SStreamObj *pStream, SStreamExecInfo *pExecNode);

View File

@ -96,7 +96,7 @@ static void mndPullupTrans(SMnode *pMnode) {
void *pReq = mndBuildTimerMsg(&contLen); void *pReq = mndBuildTimerMsg(&contLen);
if (pReq != NULL) { if (pReq != NULL) {
SRpcMsg rpcMsg = {.msgType = TDMT_MND_TRANS_TIMER, .pCont = pReq, .contLen = contLen}; SRpcMsg rpcMsg = {.msgType = TDMT_MND_TRANS_TIMER, .pCont = pReq, .contLen = contLen};
//TODO check return value // TODO check return value
(void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg); (void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
} }
} }
@ -107,7 +107,7 @@ static void mndPullupCompacts(SMnode *pMnode) {
void *pReq = mndBuildTimerMsg(&contLen); void *pReq = mndBuildTimerMsg(&contLen);
if (pReq != NULL) { if (pReq != NULL) {
SRpcMsg rpcMsg = {.msgType = TDMT_MND_COMPACT_TIMER, .pCont = pReq, .contLen = contLen}; SRpcMsg rpcMsg = {.msgType = TDMT_MND_COMPACT_TIMER, .pCont = pReq, .contLen = contLen};
//TODO check return value // TODO check return value
(void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg); (void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
} }
} }
@ -117,7 +117,7 @@ static void mndPullupTtl(SMnode *pMnode) {
int32_t contLen = 0; int32_t contLen = 0;
void *pReq = mndBuildTimerMsg(&contLen); void *pReq = mndBuildTimerMsg(&contLen);
SRpcMsg rpcMsg = {.msgType = TDMT_MND_TTL_TIMER, .pCont = pReq, .contLen = contLen}; SRpcMsg rpcMsg = {.msgType = TDMT_MND_TTL_TIMER, .pCont = pReq, .contLen = contLen};
//TODO check return value // TODO check return value
(void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg); (void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
} }
@ -125,7 +125,7 @@ static void mndPullupTrimDb(SMnode *pMnode) {
mTrace("pullup s3migrate"); mTrace("pullup s3migrate");
int32_t contLen = 0; int32_t contLen = 0;
void *pReq = mndBuildTimerMsg(&contLen); void *pReq = mndBuildTimerMsg(&contLen);
SRpcMsg rpcMsg = {.msgType = TDMT_MND_S3MIGRATE_DB_TIMER, .pCont = pReq, .contLen = contLen}; SRpcMsg rpcMsg = {.msgType = TDMT_MND_TRIM_DB_TIMER, .pCont = pReq, .contLen = contLen};
// TODO check return value // TODO check return value
(void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg); (void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
} }
@ -134,8 +134,8 @@ static void mndPullupS3MigrateDb(SMnode *pMnode) {
mTrace("pullup trim"); mTrace("pullup trim");
int32_t contLen = 0; int32_t contLen = 0;
void *pReq = mndBuildTimerMsg(&contLen); void *pReq = mndBuildTimerMsg(&contLen);
SRpcMsg rpcMsg = {.msgType = TDMT_MND_TRIM_DB_TIMER, .pCont = pReq, .contLen = contLen};
// TODO check return value // TODO check return value
SRpcMsg rpcMsg = {.msgType = TDMT_MND_S3MIGRATE_DB_TIMER, .pCont = pReq, .contLen = contLen};
(void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg); (void)tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
} }
@ -477,7 +477,7 @@ static int32_t mndCreateDir(SMnode *pMnode, const char *path) {
static int32_t mndInitWal(SMnode *pMnode) { static int32_t mndInitWal(SMnode *pMnode) {
int32_t code = 0; int32_t code = 0;
char path[PATH_MAX + 20] = {0}; char path[PATH_MAX + 20] = {0};
(void)snprintf(path, sizeof(path), "%s%swal", pMnode->path, TD_DIRSEP); (void)snprintf(path, sizeof(path), "%s%swal", pMnode->path, TD_DIRSEP);
SWalCfg cfg = {.vgId = 1, SWalCfg cfg = {.vgId = 1,
.fsyncPeriod = 0, .fsyncPeriod = 0,
@ -490,13 +490,12 @@ static int32_t mndInitWal(SMnode *pMnode) {
.encryptKey = {0}}; .encryptKey = {0}};
#if defined(TD_ENTERPRISE) #if defined(TD_ENTERPRISE)
if(tsiEncryptAlgorithm == DND_CA_SM4 && (tsiEncryptScope & DND_CS_MNODE_WAL) == DND_CS_MNODE_WAL){ if (tsiEncryptAlgorithm == DND_CA_SM4 && (tsiEncryptScope & DND_CS_MNODE_WAL) == DND_CS_MNODE_WAL) {
cfg.encryptAlgorithm = (tsiEncryptScope & DND_CS_MNODE_WAL)? tsiEncryptAlgorithm : 0; cfg.encryptAlgorithm = (tsiEncryptScope & DND_CS_MNODE_WAL) ? tsiEncryptAlgorithm : 0;
if(tsEncryptKey[0] == '\0'){ if (tsEncryptKey[0] == '\0') {
code = TSDB_CODE_DNODE_INVALID_ENCRYPTKEY; code = TSDB_CODE_DNODE_INVALID_ENCRYPTKEY;
TAOS_RETURN(code); TAOS_RETURN(code);
} } else {
else{
(void)strncpy(cfg.encryptKey, tsEncryptKey, ENCRYPT_KEY_LEN); (void)strncpy(cfg.encryptKey, tsEncryptKey, ENCRYPT_KEY_LEN);
} }
} }
@ -911,7 +910,7 @@ int32_t mndProcessRpcMsg(SRpcMsg *pMsg, SQueueInfo *pQueueInfo) {
} else if (code == 0) { } else if (code == 0) {
mGTrace("msg:%p, successfully processed", pMsg); mGTrace("msg:%p, successfully processed", pMsg);
} else { } else {
//TODO removve this wrong set code // TODO removve this wrong set code
if (code == -1) { if (code == -1) {
code = terrno; code = terrno;
} }

View File

@ -2564,6 +2564,10 @@ int32_t mndProcessCheckpointReport(SRpcMsg *pReq) {
} }
tDecoderClear(&decoder); tDecoderClear(&decoder);
streamMutexLock(&execInfo.lock);
mndInitStreamExecInfo(pMnode, &execInfo);
streamMutexUnlock(&execInfo.lock);
mDebug("receive stream task checkpoint-report msg, vgId:%d, s-task:0x%x, checkpointId:%" PRId64 mDebug("receive stream task checkpoint-report msg, vgId:%d, s-task:0x%x, checkpointId:%" PRId64
" checkpointVer:%" PRId64 " transId:%d", " checkpointVer:%" PRId64 " transId:%d",
req.nodeId, req.taskId, req.checkpointId, req.checkpointVer, req.transId); req.nodeId, req.taskId, req.checkpointId, req.checkpointVer, req.transId);
@ -2831,6 +2835,12 @@ void mndInitStreamExecInfo(SMnode *pMnode, SStreamExecInfo *pExecInfo) {
pExecInfo->initTaskList = true; pExecInfo->initTaskList = true;
} }
void mndInitStreamExecInfoForLeader(SMnode* pMnode) {
execInfo.initTaskList = false;
mInfo("init stream execInfo for leader");
mndInitStreamExecInfo(pMnode, &execInfo);
}
void addAllStreamTasksIntoBuf(SMnode *pMnode, SStreamExecInfo *pExecInfo) { void addAllStreamTasksIntoBuf(SMnode *pMnode, SStreamExecInfo *pExecInfo) {
SSdb *pSdb = pMnode->pSdb; SSdb *pSdb = pMnode->pSdb;
SStreamObj *pStream = NULL; SStreamObj *pStream = NULL;

View File

@ -18,6 +18,7 @@
#include "mndCluster.h" #include "mndCluster.h"
#include "mndTrans.h" #include "mndTrans.h"
#include "mndUser.h" #include "mndUser.h"
#include "mndStream.h"
static int32_t mndSyncEqCtrlMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) { static int32_t mndSyncEqCtrlMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) {
if (pMsg == NULL || pMsg->pCont == NULL) { if (pMsg == NULL || pMsg->pCont == NULL) {
@ -381,6 +382,7 @@ static void mndBecomeLearner(const SSyncFSM *pFsm) {
static void mndBecomeLeader(const SSyncFSM *pFsm) { static void mndBecomeLeader(const SSyncFSM *pFsm) {
mInfo("vgId:1, become leader"); mInfo("vgId:1, become leader");
SMnode *pMnode = pFsm->data; SMnode *pMnode = pFsm->data;
mndInitStreamExecInfoForLeader(pMnode);
} }
static bool mndApplyQueueEmpty(const SSyncFSM *pFsm) { static bool mndApplyQueueEmpty(const SSyncFSM *pFsm) {

File diff suppressed because one or more lines are too long

View File

@ -343,7 +343,6 @@ typedef struct {
rocksdb_readoptions_t *readoptions; rocksdb_readoptions_t *readoptions;
rocksdb_writebatch_t *writebatch; rocksdb_writebatch_t *writebatch;
rocksdb_writebatch_t *rwritebatch; rocksdb_writebatch_t *rwritebatch;
TdThreadMutex rMutex;
STSchema *pTSchema; STSchema *pTSchema;
} SRocksCache; } SRocksCache;
@ -915,12 +914,20 @@ typedef enum {
READER_EXEC_ROWS = 0x2, READER_EXEC_ROWS = 0x2,
} EExecMode; } EExecMode;
#define LAST_COL_VERSION (0x1) #define LAST_COL_VERSION_1 (0x1) // add primary key, version
#define LAST_COL_VERSION_2 (0x2) // add cache status
#define LAST_COL_VERSION LAST_COL_VERSION_2
typedef enum {
TSDB_LAST_CACHE_VALID = 0, // last_cache has valid data
TSDB_LAST_CACHE_NO_CACHE, // last_cache has no data, but tsdb may have data
} ELastCacheStatus;
typedef struct { typedef struct {
SRowKey rowKey; SRowKey rowKey;
int8_t dirty; int8_t dirty;
SColVal colVal; SColVal colVal;
ELastCacheStatus cacheStatus;
} SLastCol; } SLastCol;
typedef struct { typedef struct {

View File

@ -271,7 +271,7 @@ int32_t metaCacheUpsert(SMeta* pMeta, SMetaInfo* pInfo) {
if (*ppEntry) { // update if (*ppEntry) { // update
if (pInfo->suid != (*ppEntry)->info.suid) { if (pInfo->suid != (*ppEntry)->info.suid) {
metaError("meta/cache: suid should be same as the one in cache."); metaError("meta/cache: suid should be same as the one in cache.");
return TSDB_CODE_FAILED; return TSDB_CODE_INVALID_PARA;
} }
if (pInfo->version > (*ppEntry)->info.version) { if (pInfo->version > (*ppEntry)->info.version) {
(*ppEntry)->info.version = pInfo->version; (*ppEntry)->info.version = pInfo->version;
@ -543,7 +543,7 @@ int32_t metaGetCachedTableUidList(void* pVnode, tb_uid_t suid, const uint8_t* pK
STagFilterResEntry** pEntry = taosHashGet(pTableMap, &suid, sizeof(uint64_t)); STagFilterResEntry** pEntry = taosHashGet(pTableMap, &suid, sizeof(uint64_t));
if (NULL == pEntry) { if (NULL == pEntry) {
metaError("meta/cache: pEntry should not be NULL."); metaError("meta/cache: pEntry should not be NULL.");
return TSDB_CODE_FAILED; return TSDB_CODE_NOT_FOUND;
} }
*acquireRes = 1; *acquireRes = 1;
@ -750,7 +750,7 @@ int32_t metaGetCachedTbGroup(void* pVnode, tb_uid_t suid, const uint8_t* pKey, i
STagFilterResEntry** pEntry = taosHashGet(pTableMap, &suid, sizeof(uint64_t)); STagFilterResEntry** pEntry = taosHashGet(pTableMap, &suid, sizeof(uint64_t));
if (NULL == pEntry) { if (NULL == pEntry) {
metaDebug("suid %" PRIu64 " not in tb group cache", suid); metaDebug("suid %" PRIu64 " not in tb group cache", suid);
return TSDB_CODE_FAILED; return TSDB_CODE_NOT_FOUND;
} }
*pList = taosArrayDup(taosLRUCacheValue(pCache, pHandle), NULL); *pList = taosArrayDup(taosLRUCacheValue(pCache, pHandle), NULL);

View File

@ -1222,7 +1222,7 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
SMetaFltParam *param = arg; SMetaFltParam *param = arg;
SMetaEntry oStbEntry = {0}; SMetaEntry oStbEntry = {0};
int32_t ret = -1; int32_t code = 0;
char *buf = NULL; char *buf = NULL;
void *pData = NULL; void *pData = NULL;
int nData = 0; int nData = 0;
@ -1244,34 +1244,34 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
metaRLock(pMeta); metaRLock(pMeta);
if (tdbTbGet(pMeta->pUidIdx, &param->suid, sizeof(tb_uid_t), &pData, &nData) != 0) { TAOS_CHECK_GOTO(tdbTbGet(pMeta->pUidIdx, &param->suid, sizeof(tb_uid_t), &pData, &nData), NULL, END);
goto END;
}
tbDbKey.uid = param->suid; tbDbKey.uid = param->suid;
tbDbKey.version = ((SUidIdxVal *)pData)[0].version; tbDbKey.version = ((SUidIdxVal *)pData)[0].version;
(void)tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData);
TAOS_CHECK_GOTO(tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData), NULL, END);
tDecoderInit(&dc, pData, nData); tDecoderInit(&dc, pData, nData);
ret = metaDecodeEntry(&dc, &oStbEntry);
TAOS_CHECK_GOTO(metaDecodeEntry(&dc, &oStbEntry), NULL, END);
if (oStbEntry.stbEntry.schemaTag.pSchema == NULL || oStbEntry.stbEntry.schemaTag.pSchema == NULL) { if (oStbEntry.stbEntry.schemaTag.pSchema == NULL || oStbEntry.stbEntry.schemaTag.pSchema == NULL) {
ret = -1; TAOS_CHECK_GOTO(TSDB_CODE_INVALID_PARA, NULL, END);
goto END;
} }
ret = -1;
code = TSDB_CODE_INVALID_PARA;
for (int i = 0; i < oStbEntry.stbEntry.schemaTag.nCols; i++) { for (int i = 0; i < oStbEntry.stbEntry.schemaTag.nCols; i++) {
SSchema *schema = oStbEntry.stbEntry.schemaTag.pSchema + i; SSchema *schema = oStbEntry.stbEntry.schemaTag.pSchema + i;
if (schema->colId == param->cid && param->type == schema->type && (IS_IDX_ON(schema))) { if (schema->colId == param->cid && param->type == schema->type && (IS_IDX_ON(schema))) {
ret = 0; code = 0;
} else {
TAOS_CHECK_GOTO(code, NULL, END);
} }
} }
if (ret != 0) {
goto END;
}
ret = tdbTbcOpen(pMeta->pTagIdx, &pCursor->pCur, NULL); code = tdbTbcOpen(pMeta->pTagIdx, &pCursor->pCur, NULL);
if (ret != 0) { if (code != 0) {
goto END; TAOS_CHECK_GOTO(terrno, NULL, END);
} }
int32_t maxSize = 0; int32_t maxSize = 0;
@ -1280,7 +1280,6 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
if (param->val == NULL) { if (param->val == NULL) {
metaError("vgId:%d, failed to filter NULL data", TD_VID(pMeta->pVnode)); metaError("vgId:%d, failed to filter NULL data", TD_VID(pMeta->pVnode));
ret = -1;
goto END; goto END;
} else { } else {
if (IS_VAR_DATA_TYPE(param->type)) { if (IS_VAR_DATA_TYPE(param->type)) {
@ -1290,9 +1289,12 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
if (param->type == TSDB_DATA_TYPE_NCHAR) { if (param->type == TSDB_DATA_TYPE_NCHAR) {
maxSize = 4 * nTagData + 1; maxSize = 4 * nTagData + 1;
buf = taosMemoryCalloc(1, maxSize); buf = taosMemoryCalloc(1, maxSize);
if (buf == NULL) {
TAOS_CHECK_GOTO(terrno, NULL, END);
}
if (false == taosMbsToUcs4(tagData, nTagData, (TdUcs4 *)buf, maxSize, &maxSize)) { if (false == taosMbsToUcs4(tagData, nTagData, (TdUcs4 *)buf, maxSize, &maxSize)) {
ret = -1; TAOS_CHECK_GOTO(terrno, NULL, END);
goto END;
} }
tagData = buf; tagData = buf;
@ -1303,18 +1305,12 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
nTagData = tDataTypes[param->type].bytes; nTagData = tDataTypes[param->type].bytes;
} }
} }
ret = metaCreateTagIdxKey(pCursor->suid, pCursor->cid, tagData, nTagData, pCursor->type,
param->reverse ? INT64_MAX : INT64_MIN, &pKey, &nKey);
if (ret != 0) { TAOS_CHECK_GOTO(metaCreateTagIdxKey(pCursor->suid, pCursor->cid, tagData, nTagData, pCursor->type,
goto END; param->reverse ? INT64_MAX : INT64_MIN, &pKey, &nKey), NULL, END);
}
int cmp = 0; int cmp = 0;
ret = tdbTbcMoveTo(pCursor->pCur, pKey, nKey, &cmp); TAOS_CHECK_GOTO(tdbTbcMoveTo(pCursor->pCur, pKey, nKey, &cmp), 0, END);
if (ret != 0) {
goto END;
}
int count = 0; int count = 0;
int32_t valid = 0; int32_t valid = 0;
@ -1331,7 +1327,7 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
valid = tdbTbcGet(pCursor->pCur, (const void **)&entryKey, &nEntryKey, (const void **)&entryVal, &nEntryVal); valid = tdbTbcGet(pCursor->pCur, (const void **)&entryKey, &nEntryKey, (const void **)&entryVal, &nEntryVal);
if (valid < 0) { if (valid < 0) {
break; code = valid;
} }
if (count > TRY_ERROR_LIMIT) { if (count > TRY_ERROR_LIMIT) {
break; break;
@ -1347,6 +1343,7 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
count++; count++;
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur); valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) { if (valid < 0) {
code = valid;
break; break;
} else { } else {
continue; continue;
@ -1363,8 +1360,7 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
tuid = *(tb_uid_t *)(p->data + tDataTypes[pCursor->type].bytes); tuid = *(tb_uid_t *)(p->data + tDataTypes[pCursor->type].bytes);
} }
if (taosArrayPush(pUids, &tuid) == NULL) { if (taosArrayPush(pUids, &tuid) == NULL) {
ret = terrno; TAOS_CHECK_GOTO(terrno, NULL, END);
break;
} }
found = true; found = true;
} else { } else {
@ -1375,6 +1371,7 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) {
} }
valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur); valid = param->reverse ? tdbTbcMoveToPrev(pCursor->pCur) : tdbTbcMoveToNext(pCursor->pCur);
if (valid < 0) { if (valid < 0) {
code = valid;
break; break;
} }
} while (1); } while (1);
@ -1391,7 +1388,7 @@ END:
taosMemoryFree(pCursor); taosMemoryFree(pCursor);
return ret; return code;
} }
static int32_t metaGetTableTagByUid(SMeta *pMeta, int64_t suid, int64_t uid, void **tag, int32_t *len, bool lock) { static int32_t metaGetTableTagByUid(SMeta *pMeta, int64_t suid, int64_t uid, void **tag, int32_t *len, bool lock) {

View File

@ -557,8 +557,7 @@ _exit:
int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
SMetaEntry oStbEntry = {0}; SMetaEntry oStbEntry = {0};
SMetaEntry nStbEntry = {0}; SMetaEntry nStbEntry = {0};
STbDbKey tbDbKey = {0};
STbDbKey tbDbKey = {0};
TBC *pUidIdxc = NULL; TBC *pUidIdxc = NULL;
TBC *pTbDbc = NULL; TBC *pTbDbc = NULL;
@ -569,32 +568,37 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
int32_t ret; int32_t ret;
int32_t c = -2; int32_t c = -2;
tb_uid_t suid = pReq->suid; tb_uid_t suid = pReq->suid;
int32_t code = 0;
// get super table // get super table
if (tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pData, &nData) != 0) { if ((code = tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pData, &nData)) != 0) {
ret = -1;
goto _err; goto _err;
} }
tbDbKey.uid = suid; tbDbKey.uid = suid;
tbDbKey.version = ((SUidIdxVal *)pData)[0].version; tbDbKey.version = ((SUidIdxVal *)pData)[0].version;
(void)tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData); if ((code = tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData)) != 0) {
goto _err;
}
tDecoderInit(&dc, pData, nData); tDecoderInit(&dc, pData, nData);
ret = metaDecodeEntry(&dc, &oStbEntry); code = metaDecodeEntry(&dc, &oStbEntry);
if (ret < 0) { if (code < 0) {
goto _err; goto _err;
} }
if (oStbEntry.stbEntry.schemaTag.pSchema == NULL || oStbEntry.stbEntry.schemaTag.pSchema == NULL) { if (oStbEntry.stbEntry.schemaTag.pSchema == NULL || oStbEntry.stbEntry.schemaTag.pSchema == NULL) {
code = TSDB_CODE_INVALID_PARA;
goto _err; goto _err;
} }
if (oStbEntry.stbEntry.schemaTag.version == pReq->schemaTag.version) { if (oStbEntry.stbEntry.schemaTag.version == pReq->schemaTag.version) {
code = TSDB_CODE_INVALID_PARA;
goto _err; goto _err;
} }
if (oStbEntry.stbEntry.schemaTag.nCols != pReq->schemaTag.nCols) { if (oStbEntry.stbEntry.schemaTag.nCols != pReq->schemaTag.nCols) {
code = TSDB_CODE_INVALID_PARA;
goto _err; goto _err;
} }
@ -604,6 +608,7 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
SSchema *pOld = oStbEntry.stbEntry.schemaTag.pSchema + i; SSchema *pOld = oStbEntry.stbEntry.schemaTag.pSchema + i;
if (pNew->type != pOld->type || pNew->colId != pOld->colId || pNew->bytes != pOld->bytes || if (pNew->type != pOld->type || pNew->colId != pOld->colId || pNew->bytes != pOld->bytes ||
strncmp(pOld->name, pNew->name, sizeof(pNew->name))) { strncmp(pOld->name, pNew->name, sizeof(pNew->name))) {
code = TSDB_CODE_INVALID_PARA;
goto _err; goto _err;
} }
if (IS_IDX_ON(pNew) && !IS_IDX_ON(pOld)) { if (IS_IDX_ON(pNew) && !IS_IDX_ON(pOld)) {
@ -614,13 +619,14 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
} }
if (diffIdx == -1) { if (diffIdx == -1) {
code = TSDB_CODE_INVALID_PARA;
goto _err; goto _err;
} }
// Get target schema info // Get target schema info
SSchemaWrapper *pTagSchema = &pReq->schemaTag; SSchemaWrapper *pTagSchema = &pReq->schemaTag;
if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) { if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) {
terrno = TSDB_CODE_VND_COL_ALREADY_EXISTS; code = TSDB_CODE_INVALID_PARA;
goto _err; goto _err;
} }
SSchema *pCol = pTagSchema->pSchema + diffIdx; SSchema *pCol = pTagSchema->pSchema + diffIdx;
@ -629,17 +635,21 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
* iterator all pTdDbc by uid and version * iterator all pTdDbc by uid and version
*/ */
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL)); code = tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL);
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (code != 0) {
if (rc < 0) { goto _err;
}
code = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (code < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
goto _err; goto _err;
} }
for (;;) { for (;;) {
void *pKey = NULL, *pVal = NULL; void *pKey = NULL, *pVal = NULL;
int nKey = 0, nVal = 0; int nKey = 0, nVal = 0;
rc = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal); code = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal);
if (rc < 0) { if (code < 0) {
tdbFree(pKey); tdbFree(pKey);
tdbFree(pVal); tdbFree(pVal);
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
@ -672,10 +682,10 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
nTagData = tDataTypes[pCol->type].bytes; nTagData = tDataTypes[pCol->type].bytes;
} }
} }
rc = metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, table->uid, &pTagIdxKey, &nTagIdxKey); code = metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, table->uid, &pTagIdxKey, &nTagIdxKey);
tdbFree(pKey); tdbFree(pKey);
tdbFree(pVal); tdbFree(pVal);
if (rc < 0) { if (code < 0) {
metaDestroyTagIdxKey(pTagIdxKey); metaDestroyTagIdxKey(pTagIdxKey);
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
goto _err; goto _err;
@ -684,6 +694,7 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
metaWLock(pMeta); metaWLock(pMeta);
(void)tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn); (void)tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn);
metaULock(pMeta); metaULock(pMeta);
metaDestroyTagIdxKey(pTagIdxKey); metaDestroyTagIdxKey(pTagIdxKey);
pTagIdxKey = NULL; pTagIdxKey = NULL;
} }
@ -714,9 +725,10 @@ _err:
tDecoderClear(&dc); tDecoderClear(&dc);
tdbFree(pData); tdbFree(pData);
return TSDB_CODE_VND_COL_ALREADY_EXISTS; return code;
} }
int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) { int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) {
int32_t code = 0;
SMetaEntry oStbEntry = {0}; SMetaEntry oStbEntry = {0};
SMetaEntry nStbEntry = {0}; SMetaEntry nStbEntry = {0};
@ -732,17 +744,19 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq)
tb_uid_t suid = pReq->stbUid; tb_uid_t suid = pReq->stbUid;
if (tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pData, &nData) != 0) { if ((code = tdbTbGet(pMeta->pUidIdx, &suid, sizeof(tb_uid_t), &pData, &nData)) != 0) {
ret = -1;
goto _err; goto _err;
} }
tbDbKey.uid = suid; tbDbKey.uid = suid;
tbDbKey.version = ((SUidIdxVal *)pData)[0].version; tbDbKey.version = ((SUidIdxVal *)pData)[0].version;
(void)tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData); if ((code = tdbTbGet(pMeta->pTbDb, &tbDbKey, sizeof(tbDbKey), &pData, &nData)) != 0) {
goto _err;
}
tDecoderInit(&dc, pData, nData); tDecoderInit(&dc, pData, nData);
ret = metaDecodeEntry(&dc, &oStbEntry); code = metaDecodeEntry(&dc, &oStbEntry);
if (ret < 0) { if (code != 0) {
goto _err; goto _err;
} }
@ -759,6 +773,7 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq)
} }
if (pCol == NULL) { if (pCol == NULL) {
code = TSDB_CODE_VND_COL_NOT_EXISTS;
goto _err; goto _err;
} }
@ -766,17 +781,22 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq)
* iterator all pTdDbc by uid and version * iterator all pTdDbc by uid and version
*/ */
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL)); code = tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL);
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (code != 0) {
if (rc < 0) { goto _err;
}
code = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (code < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
goto _err; goto _err;
} }
for (;;) { for (;;) {
void *pKey = NULL, *pVal = NULL; void *pKey = NULL, *pVal = NULL;
int nKey = 0, nVal = 0; int nKey = 0, nVal = 0;
rc = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal);
if (rc < 0) { code = tdbTbcNext(pCtbIdxc, &pKey, &nKey, &pVal, &nVal);
if (code < 0) {
tdbFree(pKey); tdbFree(pKey);
tdbFree(pVal); tdbFree(pVal);
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
@ -809,10 +829,11 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq)
nTagData = tDataTypes[pCol->type].bytes; nTagData = tDataTypes[pCol->type].bytes;
} }
} }
rc = metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, table->uid, &pTagIdxKey, &nTagIdxKey);
code = metaCreateTagIdxKey(suid, pCol->colId, pTagData, nTagData, pCol->type, table->uid, &pTagIdxKey, &nTagIdxKey);
tdbFree(pKey); tdbFree(pKey);
tdbFree(pVal); tdbFree(pVal);
if (rc < 0) { if (code < 0) {
metaDestroyTagIdxKey(pTagIdxKey); metaDestroyTagIdxKey(pTagIdxKey);
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
goto _err; goto _err;
@ -836,6 +857,16 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq)
SSchemaWrapper *row = tCloneSSchemaWrapper(&oStbEntry.stbEntry.schemaRow); SSchemaWrapper *row = tCloneSSchemaWrapper(&oStbEntry.stbEntry.schemaRow);
SSchemaWrapper *tag = tCloneSSchemaWrapper(&oStbEntry.stbEntry.schemaTag); SSchemaWrapper *tag = tCloneSSchemaWrapper(&oStbEntry.stbEntry.schemaTag);
SColCmprWrapper *cmpr = tCloneSColCmprWrapper(&oStbEntry.colCmpr); SColCmprWrapper *cmpr = tCloneSColCmprWrapper(&oStbEntry.colCmpr);
if (row == NULL || tag == NULL || cmpr == NULL) {
tDeleteSchemaWrapper(row);
tDeleteSchemaWrapper(tag);
tDeleteSColCmprWrapper(cmpr);
code = TSDB_CODE_OUT_OF_MEMORY;
(void)tdbTbcClose(pCtbIdxc);
goto _err;
}
nStbEntry.stbEntry.schemaRow = *row; nStbEntry.stbEntry.schemaRow = *row;
nStbEntry.stbEntry.schemaTag = *tag; nStbEntry.stbEntry.schemaTag = *tag;
@ -866,7 +897,7 @@ _err:
tDecoderClear(&dc); tDecoderClear(&dc);
tdbFree(pData); tdbFree(pData);
return -1; return code;
} }
int metaCreateTable(SMeta *pMeta, int64_t ver, SVCreateTbReq *pReq, STableMetaRsp **pMetaRsp) { int metaCreateTable(SMeta *pMeta, int64_t ver, SVCreateTbReq *pReq, STableMetaRsp **pMetaRsp) {

View File

@ -1008,7 +1008,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerReq(SStreamMeta* pMeta, SRpcMsg* pMsg)
tqDebug("s-task:%s re-send checkpoint-trigger to:0x%x, checkpointId:%" PRId64 ", transId:%d", pTask->id.idStr, tqDebug("s-task:%s re-send checkpoint-trigger to:0x%x, checkpointId:%" PRId64 ", transId:%d", pTask->id.idStr,
(int32_t)pReq->downstreamTaskId, checkpointId, transId); (int32_t)pReq->downstreamTaskId, checkpointId, transId);
code = streamTaskSendCheckpointTriggerMsg(pTask, pReq->downstreamTaskId, pReq->downstreamNodeId, &pMsg->info, code = streamTaskSendCheckpointTriggerMsg(pTask, pReq->downstreamTaskId, pReq->downstreamNodeId, &pMsg->info,
TSDB_CODE_SUCCESS); TSDB_CODE_SUCCESS);
} else { // not send checkpoint-trigger yet, wait } else { // not send checkpoint-trigger yet, wait
int32_t recv = 0, total = 0; int32_t recv = 0, total = 0;
streamTaskGetTriggerRecvStatus(pTask, &recv, &total); streamTaskGetTriggerRecvStatus(pTask, &recv, &total);
@ -1050,8 +1050,10 @@ int32_t tqStreamTaskProcessRetrieveTriggerRsp(SStreamMeta* pMeta, SRpcMsg* pMsg)
return code; return code;
} }
tqDebug("s-task:%s recv re-send checkpoint-trigger msg from upstream:0x%x, checkpointId:%" PRId64 ", transId:%d", tqDebug(
pTask->id.idStr, pRsp->upstreamTaskId, pRsp->checkpointId, pRsp->transId); "s-task:%s recv re-send checkpoint-trigger msg from through retrieve/rsp channel, upstream:0x%x, "
"checkpointId:%" PRId64 ", transId:%d",
pTask->id.idStr, pRsp->upstreamTaskId, pRsp->checkpointId, pRsp->transId);
code = streamTaskProcessCheckpointTriggerRsp(pTask, pRsp); code = streamTaskProcessCheckpointTriggerRsp(pTask, pRsp);
streamMetaReleaseTask(pMeta, pTask); streamMetaReleaseTask(pMeta, pTask);

File diff suppressed because it is too large Load Diff

View File

@ -5187,6 +5187,7 @@ int32_t tsdbNextDataBlock2(STsdbReader* pReader, bool* hasNext) {
resetAllDataBlockScanInfo(pReader->status.pTableMap, pReader->innerReader[0]->info.window.ekey, step); resetAllDataBlockScanInfo(pReader->status.pTableMap, pReader->innerReader[0]->info.window.ekey, step);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
(void) tsdbReleaseReader(pReader);
return code; return code;
} }
@ -5213,6 +5214,7 @@ int32_t tsdbNextDataBlock2(STsdbReader* pReader, bool* hasNext) {
code = doOpenReaderImpl(pReader->innerReader[1]); code = doOpenReaderImpl(pReader->innerReader[1]);
resetAllDataBlockScanInfo(pReader->innerReader[1]->status.pTableMap, pReader->info.window.ekey, step); resetAllDataBlockScanInfo(pReader->innerReader[1]->status.pTableMap, pReader->info.window.ekey, step);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
(void) tsdbReleaseReader(pReader);
return code; return code;
} }
@ -5493,6 +5495,7 @@ int32_t tsdbReaderReset2(STsdbReader* pReader, SQueryTableDataCond* pCond) {
code = initFilesetIterator(&pStatus->fileIter, pReader->pReadSnap->pfSetArray, pReader); code = initFilesetIterator(&pStatus->fileIter, pReader->pReadSnap->pfSetArray, pReader);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
(void) tsdbReleaseReader(pReader);
return code; return code;
} }
@ -5826,9 +5829,12 @@ int32_t tsdbTakeReadSnap2(STsdbReader* pReader, _query_reseek_func_t reseek, STs
pSnap->pIMem = pTsdb->imem; pSnap->pIMem = pTsdb->imem;
pSnap->pINode = taosMemoryMalloc(sizeof(*pSnap->pINode)); pSnap->pINode = taosMemoryMalloc(sizeof(*pSnap->pINode));
if (pSnap->pINode == NULL) { if (pSnap->pINode == NULL) {
tsdbUnrefMemTable(pTsdb->mem, pSnap->pNode, true); // unref the previous refed mem
code = terrno; code = terrno;
if (pTsdb->mem && pSnap->pNode) {
(void) tsdbUnrefMemTable(pTsdb->mem, pSnap->pNode, true); // unref the previous refed mem
}
(void) taosThreadMutexUnlock(&pTsdb->mutex); (void) taosThreadMutexUnlock(&pTsdb->mutex);
goto _exit; goto _exit;
} }
@ -5843,11 +5849,11 @@ int32_t tsdbTakeReadSnap2(STsdbReader* pReader, _query_reseek_func_t reseek, STs
code = tsdbFSCreateRefSnapshotWithoutLock(pTsdb->pFS, &pSnap->pfSetArray); code = tsdbFSCreateRefSnapshotWithoutLock(pTsdb->pFS, &pSnap->pfSetArray);
if (code) { if (code) {
if (pSnap->pNode) { if (pSnap->pNode) {
tsdbUnrefMemTable(pTsdb->mem, pSnap->pNode, true); // unref the previous refed mem (void) tsdbUnrefMemTable(pTsdb->mem, pSnap->pNode, true); // unref the previous refed mem
} }
if (pSnap->pINode) { if (pSnap->pINode) {
tsdbUnrefMemTable(pTsdb->imem, pSnap->pINode, true); (void) tsdbUnrefMemTable(pTsdb->imem, pSnap->pINode, true);
} }
(void) taosThreadMutexUnlock(&pTsdb->mutex); (void) taosThreadMutexUnlock(&pTsdb->mutex);

View File

@ -139,7 +139,7 @@ int32_t getPosInBlockInfoBuf(SBlockInfoBuf* pBuf, int32_t index, STableBlockScan
int32_t bucketIndex = index / pBuf->numPerBucket; int32_t bucketIndex = index / pBuf->numPerBucket;
char** pBucket = taosArrayGet(pBuf->pData, bucketIndex); char** pBucket = taosArrayGet(pBuf->pData, bucketIndex);
if (pBucket == NULL) { if (pBucket == NULL) {
return TSDB_CODE_FAILED; return TSDB_CODE_NOT_FOUND;
} }
*pInfo = (STableBlockScanInfo*)((*pBucket) + (index % pBuf->numPerBucket) * sizeof(STableBlockScanInfo)); *pInfo = (STableBlockScanInfo*)((*pBucket) + (index % pBuf->numPerBucket) * sizeof(STableBlockScanInfo));

View File

@ -301,8 +301,7 @@ int vnodeDecodeConfig(const SJson *pJson, void *pObj) {
#if defined(TD_ENTERPRISE) #if defined(TD_ENTERPRISE)
if (pCfg->tdbEncryptAlgorithm == DND_CA_SM4) { if (pCfg->tdbEncryptAlgorithm == DND_CA_SM4) {
if (tsEncryptKey[0] == 0) { if (tsEncryptKey[0] == 0) {
terrno = TSDB_CODE_DNODE_INVALID_ENCRYPTKEY; return terrno = TSDB_CODE_DNODE_INVALID_ENCRYPTKEY;
return -1;
} else { } else {
strncpy(pCfg->tdbEncryptKey, tsEncryptKey, ENCRYPT_KEY_LEN); strncpy(pCfg->tdbEncryptKey, tsEncryptKey, ENCRYPT_KEY_LEN);
} }

View File

@ -413,9 +413,9 @@ static int vnodeCommitImpl(SCommitInfo *pInfo) {
pInfo->info.state.commitID, pInfo->info.state.committed, pInfo->info.state.commitTerm); pInfo->info.state.commitID, pInfo->info.state.committed, pInfo->info.state.commitTerm);
// persist wal before starting // persist wal before starting
if (walPersist(pVnode->pWal) < 0) { if ((code = walPersist(pVnode->pWal)) < 0) {
vError("vgId:%d, failed to persist wal since %s", TD_VID(pVnode), terrstr()); vError("vgId:%d, failed to persist wal since %s", TD_VID(pVnode), tstrerror(code));
return -1; return code;
} }
(void)vnodeGetPrimaryDir(pVnode->path, pVnode->diskPrimary, pVnode->pTfs, dir, TSDB_FILENAME_LEN); (void)vnodeGetPrimaryDir(pVnode->path, pVnode->diskPrimary, pVnode->pTfs, dir, TSDB_FILENAME_LEN);
@ -556,7 +556,6 @@ int vnodeDecodeInfo(uint8_t *pData, SVnodeInfo *pInfo) {
pJson = tjsonParse(pData); pJson = tjsonParse(pData);
if (pJson == NULL) { if (pJson == NULL) {
TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_DATA_FMT, lino, _exit); TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_DATA_FMT, lino, _exit);
return -1;
} }
code = tjsonToObject(pJson, "config", vnodeDecodeConfig, (void *)&pInfo->config); code = tjsonToObject(pJson, "config", vnodeDecodeConfig, (void *)&pInfo->config);

View File

@ -39,13 +39,14 @@ static int32_t vnodeMkDir(STfs *pTfs, const char *path) {
} }
int32_t vnodeCreate(const char *path, SVnodeCfg *pCfg, int32_t diskPrimary, STfs *pTfs) { int32_t vnodeCreate(const char *path, SVnodeCfg *pCfg, int32_t diskPrimary, STfs *pTfs) {
int32_t code = 0;
SVnodeInfo info = {0}; SVnodeInfo info = {0};
char dir[TSDB_FILENAME_LEN] = {0}; char dir[TSDB_FILENAME_LEN] = {0};
// check config // check config
if (vnodeCheckCfg(pCfg) < 0) { if ((code = vnodeCheckCfg(pCfg)) < 0) {
vError("vgId:%d, failed to create vnode since:%s", pCfg->vgId, tstrerror(terrno)); vError("vgId:%d, failed to create vnode since:%s", pCfg->vgId, tstrerror(code));
return -1; return code;
} }
// create vnode env // create vnode env
@ -72,9 +73,9 @@ int32_t vnodeCreate(const char *path, SVnodeCfg *pCfg, int32_t diskPrimary, STfs
} }
vInfo("vgId:%d, save config while create", info.config.vgId); vInfo("vgId:%d, save config while create", info.config.vgId);
if (vnodeSaveInfo(dir, &info) < 0 || vnodeCommitInfo(dir) < 0) { if ((code = vnodeSaveInfo(dir, &info)) < 0 || (code = vnodeCommitInfo(dir)) < 0) {
vError("vgId:%d, failed to save vnode config since %s", pCfg ? pCfg->vgId : 0, tstrerror(terrno)); vError("vgId:%d, failed to save vnode config since %s", pCfg ? pCfg->vgId : 0, tstrerror(code));
return -1; return code;
} }
vInfo("vgId:%d, vnode is created", info.config.vgId); vInfo("vgId:%d, vnode is created", info.config.vgId);
@ -93,7 +94,7 @@ int32_t vnodeAlterReplica(const char *path, SAlterVnodeReplicaReq *pReq, int32_t
ret = vnodeLoadInfo(dir, &info); ret = vnodeLoadInfo(dir, &info);
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to read vnode config from %s since %s", pReq->vgId, path, tstrerror(terrno)); vError("vgId:%d, failed to read vnode config from %s since %s", pReq->vgId, path, tstrerror(terrno));
return -1; return ret;
} }
SSyncCfg *pCfg = &info.config.syncCfg; SSyncCfg *pCfg = &info.config.syncCfg;
@ -144,13 +145,13 @@ int32_t vnodeAlterReplica(const char *path, SAlterVnodeReplicaReq *pReq, int32_t
ret = vnodeSaveInfo(dir, &info); ret = vnodeSaveInfo(dir, &info);
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to save vnode config since %s", pReq->vgId, tstrerror(terrno)); vError("vgId:%d, failed to save vnode config since %s", pReq->vgId, tstrerror(terrno));
return -1; return ret;
} }
ret = vnodeCommitInfo(dir); ret = vnodeCommitInfo(dir);
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to commit vnode config since %s", pReq->vgId, tstrerror(terrno)); vError("vgId:%d, failed to commit vnode config since %s", pReq->vgId, tstrerror(terrno));
return -1; return ret;
} }
vInfo("vgId:%d, vnode config is saved", info.config.vgId); vInfo("vgId:%d, vnode config is saved", info.config.vgId);
@ -226,7 +227,7 @@ int32_t vnodeAlterHashRange(const char *srcPath, const char *dstPath, SAlterVnod
ret = vnodeLoadInfo(dir, &info); ret = vnodeLoadInfo(dir, &info);
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to read vnode config from %s since %s", pReq->srcVgId, srcPath, tstrerror(terrno)); vError("vgId:%d, failed to read vnode config from %s since %s", pReq->srcVgId, srcPath, tstrerror(terrno));
return -1; return ret;
} }
vInfo("vgId:%d, alter hashrange from [%u, %u] to [%u, %u]", pReq->srcVgId, info.config.hashBegin, info.config.hashEnd, vInfo("vgId:%d, alter hashrange from [%u, %u] to [%u, %u]", pReq->srcVgId, info.config.hashBegin, info.config.hashEnd,
@ -256,13 +257,13 @@ int32_t vnodeAlterHashRange(const char *srcPath, const char *dstPath, SAlterVnod
ret = vnodeSaveInfo(dir, &info); ret = vnodeSaveInfo(dir, &info);
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to save vnode config since %s", pReq->dstVgId, tstrerror(terrno)); vError("vgId:%d, failed to save vnode config since %s", pReq->dstVgId, tstrerror(terrno));
return -1; return ret;
} }
ret = vnodeCommitInfo(dir); ret = vnodeCommitInfo(dir);
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to commit vnode config since %s", pReq->dstVgId, tstrerror(terrno)); vError("vgId:%d, failed to commit vnode config since %s", pReq->dstVgId, tstrerror(terrno));
return -1; return ret;
} }
vInfo("vgId:%d, rename %s to %s", pReq->dstVgId, srcPath, dstPath); vInfo("vgId:%d, rename %s to %s", pReq->dstVgId, srcPath, dstPath);
@ -270,7 +271,7 @@ int32_t vnodeAlterHashRange(const char *srcPath, const char *dstPath, SAlterVnod
if (ret < 0) { if (ret < 0) {
vError("vgId:%d, failed to rename vnode from %s to %s since %s", pReq->dstVgId, srcPath, dstPath, vError("vgId:%d, failed to rename vnode from %s to %s since %s", pReq->dstVgId, srcPath, dstPath,
tstrerror(terrno)); tstrerror(terrno));
return -1; return ret;
} }
vInfo("vgId:%d, vnode hashrange is altered", info.config.vgId); vInfo("vgId:%d, vnode hashrange is altered", info.config.vgId);
@ -293,9 +294,9 @@ int32_t vnodeRestoreVgroupId(const char *srcPath, const char *dstPath, int32_t s
} }
(void)vnodeGetPrimaryDir(srcPath, diskPrimary, pTfs, dir, TSDB_FILENAME_LEN); (void)vnodeGetPrimaryDir(srcPath, diskPrimary, pTfs, dir, TSDB_FILENAME_LEN);
if (vnodeLoadInfo(dir, &info) < 0) { if ((code = vnodeLoadInfo(dir, &info)) < 0) {
vError("vgId:%d, failed to read vnode config from %s since %s", srcVgId, srcPath, tstrerror(terrno)); vError("vgId:%d, failed to read vnode config from %s since %s", srcVgId, srcPath, tstrerror(terrno));
return -1; return code;
} }
if (info.config.vgId == srcVgId) { if (info.config.vgId == srcVgId) {

View File

@ -545,7 +545,7 @@ int32_t vnodeGetAllTableList(SVnode *pVnode, uint64_t uid, SArray *list) {
SMCtbCursor *pCur = metaOpenCtbCursor(pVnode, uid, 1); SMCtbCursor *pCur = metaOpenCtbCursor(pVnode, uid, 1);
if (NULL == pCur) { if (NULL == pCur) {
qError("vnode get all table list failed"); qError("vnode get all table list failed");
return TSDB_CODE_FAILED; return terrno;
} }
while (1) { while (1) {
@ -576,7 +576,7 @@ int32_t vnodeGetCtbIdList(void *pVnode, int64_t suid, SArray *list) {
SMCtbCursor *pCur = metaOpenCtbCursor(pVnodeObj, suid, 1); SMCtbCursor *pCur = metaOpenCtbCursor(pVnodeObj, suid, 1);
if (NULL == pCur) { if (NULL == pCur) {
qError("vnode get all table list failed"); qError("vnode get all table list failed");
return TSDB_CODE_FAILED; return terrno;
} }
while (1) { while (1) {
@ -627,7 +627,7 @@ int32_t vnodeGetStbIdListByFilter(SVnode *pVnode, int64_t suid, SArray *list, bo
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
SMStbCursor *pCur = metaOpenStbCursor(pVnode->pMeta, suid); SMStbCursor *pCur = metaOpenStbCursor(pVnode->pMeta, suid);
if (!pCur) { if (!pCur) {
return TSDB_CODE_FAILED; return terrno;
} }
while (1) { while (1) {
@ -655,7 +655,7 @@ _exit:
int32_t vnodeGetCtbNum(SVnode *pVnode, int64_t suid, int64_t *num) { int32_t vnodeGetCtbNum(SVnode *pVnode, int64_t suid, int64_t *num) {
SMCtbCursor *pCur = metaOpenCtbCursor(pVnode, suid, 0); SMCtbCursor *pCur = metaOpenCtbCursor(pVnode, suid, 0);
if (!pCur) { if (!pCur) {
return TSDB_CODE_FAILED; return terrno;
} }
*num = 0; *num = 0;
@ -757,8 +757,7 @@ int32_t vnodeGetTimeSeriesNum(SVnode *pVnode, int64_t *num) {
SArray *suidList = NULL; SArray *suidList = NULL;
if (!(suidList = taosArrayInit(1, sizeof(tb_uid_t)))) { if (!(suidList = taosArrayInit(1, sizeof(tb_uid_t)))) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return terrno = TSDB_CODE_OUT_OF_MEMORY;
return TSDB_CODE_FAILED;
} }
int32_t tbFilterSize = 0; int32_t tbFilterSize = 0;
@ -774,7 +773,7 @@ int32_t vnodeGetTimeSeriesNum(SVnode *pVnode, int64_t *num) {
(tbFilterSize && vnodeGetStbIdListByFilter(pVnode, 0, suidList, vnodeTimeSeriesFilter, pVnode) < 0)) { (tbFilterSize && vnodeGetStbIdListByFilter(pVnode, 0, suidList, vnodeTimeSeriesFilter, pVnode) < 0)) {
qError("vgId:%d, failed to get stb id list error: %s", TD_VID(pVnode), terrstr()); qError("vgId:%d, failed to get stb id list error: %s", TD_VID(pVnode), terrstr());
taosArrayDestroy(suidList); taosArrayDestroy(suidList);
return TSDB_CODE_FAILED; return terrno;
} }
*num = 0; *num = 0;
@ -799,7 +798,7 @@ _exit:
int32_t vnodeGetAllCtbNum(SVnode *pVnode, int64_t *num) { int32_t vnodeGetAllCtbNum(SVnode *pVnode, int64_t *num) {
SMStbCursor *pCur = metaOpenStbCursor(pVnode->pMeta, 0); SMStbCursor *pCur = metaOpenStbCursor(pVnode->pMeta, 0);
if (!pCur) { if (!pCur) {
return TSDB_CODE_FAILED; return terrno;
} }
*num = 0; *num = 0;

View File

@ -1980,7 +1980,7 @@ _err:
tDecoderClear(&coder); tDecoderClear(&coder);
vError("vgId:%d, failed to create tsma %s:%" PRIi64 " version %" PRIi64 "for table %" PRIi64 " since %s", vError("vgId:%d, failed to create tsma %s:%" PRIi64 " version %" PRIi64 "for table %" PRIi64 " since %s",
TD_VID(pVnode), req.indexName, req.indexUid, ver, req.tableUid, terrstr()); TD_VID(pVnode), req.indexName, req.indexUid, ver, req.tableUid, terrstr());
return -1; return terrno;
} }
/** /**

View File

@ -381,7 +381,7 @@ int32_t vnodeProcessSyncMsg(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp) {
static int32_t vnodeSyncEqCtrlMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) { static int32_t vnodeSyncEqCtrlMsg(const SMsgCb *msgcb, SRpcMsg *pMsg) {
if (pMsg == NULL || pMsg->pCont == NULL) { if (pMsg == NULL || pMsg->pCont == NULL) {
return -1; return TSDB_CODE_INVALID_PARA;
} }
if (msgcb == NULL || msgcb->putToQueueFp == NULL) { if (msgcb == NULL || msgcb->putToQueueFp == NULL) {

View File

@ -332,17 +332,12 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pD
goto _return; goto _return;
} }
if (NULL == dispatcher->pDataBlocks) {
taosMemoryFree(dispatcher);
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _return;
}
*pHandle = dispatcher; *pHandle = dispatcher;
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
_return: _return:
if (dispatcher) {
taosMemoryFree(pManager); dsDestroyDataSinker(dispatcher);
}
return terrno; return terrno;
} }

View File

@ -485,6 +485,9 @@ void freeSourceDataInfo(void* p) {
} }
void doDestroyExchangeOperatorInfo(void* param) { void doDestroyExchangeOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SExchangeInfo* pExInfo = (SExchangeInfo*)param; SExchangeInfo* pExInfo = (SExchangeInfo*)param;
if (pExInfo->pFetchRpcHandles) { if (pExInfo->pFetchRpcHandles) {
for (int32_t i = 0; i < pExInfo->pFetchRpcHandles->size; ++i) { for (int32_t i = 0; i < pExInfo->pFetchRpcHandles->size; ++i) {

View File

@ -316,7 +316,6 @@ qTaskInfo_t qCreateQueueExecTaskInfo(void* msg, SReadHandle* pReaderHandle, int3
qTaskInfo_t pTaskInfo = NULL; qTaskInfo_t pTaskInfo = NULL;
code = qCreateExecTask(pReaderHandle, vgId, 0, pPlan, &pTaskInfo, NULL, 0, NULL, OPTR_EXEC_MODEL_QUEUE); code = qCreateExecTask(pReaderHandle, vgId, 0, pPlan, &pTaskInfo, NULL, 0, NULL, OPTR_EXEC_MODEL_QUEUE);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
nodesDestroyNode((SNode*)pPlan);
qDestroyTask(pTaskInfo); qDestroyTask(pTaskInfo);
terrno = code; terrno = code;
return NULL; return NULL;
@ -352,7 +351,6 @@ qTaskInfo_t qCreateStreamExecTaskInfo(void* msg, SReadHandle* readers, int32_t v
qTaskInfo_t pTaskInfo = NULL; qTaskInfo_t pTaskInfo = NULL;
code = qCreateExecTask(readers, vgId, taskId, pPlan, &pTaskInfo, NULL, 0, NULL, OPTR_EXEC_MODEL_STREAM); code = qCreateExecTask(readers, vgId, taskId, pPlan, &pTaskInfo, NULL, 0, NULL, OPTR_EXEC_MODEL_STREAM);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
nodesDestroyNode((SNode*)pPlan);
qDestroyTask(pTaskInfo); qDestroyTask(pTaskInfo);
terrno = code; terrno = code;
return NULL; return NULL;
@ -360,7 +358,6 @@ qTaskInfo_t qCreateStreamExecTaskInfo(void* msg, SReadHandle* readers, int32_t v
code = qStreamInfoResetTimewindowFilter(pTaskInfo); code = qStreamInfoResetTimewindowFilter(pTaskInfo);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
nodesDestroyNode((SNode*)pPlan);
qDestroyTask(pTaskInfo); qDestroyTask(pTaskInfo);
terrno = code; terrno = code;
return NULL; return NULL;

View File

@ -65,6 +65,7 @@ int32_t doCreateTask(uint64_t queryId, uint64_t taskId, int32_t vgId, EOPTR_EXEC
p->id.taskId = taskId; p->id.taskId = taskId;
p->id.str = taosMemoryMalloc(64); p->id.str = taosMemoryMalloc(64);
if (p->id.str == NULL) { if (p->id.str == NULL) {
doDestroyTask(p);
return terrno; return terrno;
} }
@ -100,6 +101,7 @@ int32_t createExecTaskInfo(SSubplan* pPlan, SExecTaskInfo** pTaskInfo, SReadHand
int32_t vgId, char* sql, EOPTR_EXEC_MODEL model) { int32_t vgId, char* sql, EOPTR_EXEC_MODEL model) {
int32_t code = doCreateTask(pPlan->id.queryId, taskId, vgId, model, &pHandle->api, pTaskInfo); int32_t code = doCreateTask(pPlan->id.queryId, taskId, vgId, model, &pHandle->api, pTaskInfo);
if (*pTaskInfo == NULL || code != 0) { if (*pTaskInfo == NULL || code != 0) {
nodesDestroyNode((SNode*)pPlan);
taosMemoryFree(sql); taosMemoryFree(sql);
return code; return code;
} }

View File

@ -255,21 +255,28 @@ static int32_t doFilterByBlockSMA(SFilterInfo* pFilterInfo, SColumnDataAgg* pCol
return filterRangeExecute(pFilterInfo, pColsAgg, numOfCols, numOfRows, keep); return filterRangeExecute(pFilterInfo, pColsAgg, numOfCols, numOfRows, keep);
} }
static bool doLoadBlockSMA(STableScanBase* pTableScanInfo, SSDataBlock* pBlock, SExecTaskInfo* pTaskInfo) { static int32_t doLoadBlockSMA(STableScanBase* pTableScanInfo, SSDataBlock* pBlock, SExecTaskInfo* pTaskInfo,
bool* pLoad) {
SStorageAPI* pAPI = &pTaskInfo->storageAPI; SStorageAPI* pAPI = &pTaskInfo->storageAPI;
bool allColumnsHaveAgg = true;
bool hasNullSMA = false;
if (pLoad != NULL) {
*pLoad = false;
}
bool allColumnsHaveAgg = true;
bool hasNullSMA = false;
int32_t code = pAPI->tsdReader.tsdReaderRetrieveBlockSMAInfo(pTableScanInfo->dataReader, pBlock, &allColumnsHaveAgg, int32_t code = pAPI->tsdReader.tsdReaderRetrieveBlockSMAInfo(pTableScanInfo->dataReader, pBlock, &allColumnsHaveAgg,
&hasNullSMA); &hasNullSMA);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
T_LONG_JMP(pTaskInfo->env, code); return code;
} }
if (!allColumnsHaveAgg || hasNullSMA) { if (!allColumnsHaveAgg || hasNullSMA) {
return false; *pLoad = false;
} else {
*pLoad = true;
} }
return true;
return code;
} }
static void doSetTagColumnData(STableScanBase* pTableScanInfo, SSDataBlock* pBlock, SExecTaskInfo* pTaskInfo, static void doSetTagColumnData(STableScanBase* pTableScanInfo, SSDataBlock* pBlock, SExecTaskInfo* pTaskInfo,
@ -330,14 +337,14 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableSca
int32_t lino = 0; int32_t lino = 0;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SStorageAPI* pAPI = &pTaskInfo->storageAPI; SStorageAPI* pAPI = &pTaskInfo->storageAPI;
bool loadSMA = false;
SFileBlockLoadRecorder* pCost = &pTableScanInfo->readRecorder; SFileBlockLoadRecorder* pCost = &pTableScanInfo->readRecorder;
pCost->totalBlocks += 1; pCost->totalBlocks += 1;
pCost->totalRows += pBlock->info.rows; pCost->totalRows += pBlock->info.rows;
bool loadSMA = false;
*status = pTableScanInfo->dataBlockLoadFlag; *status = pTableScanInfo->dataBlockLoadFlag;
if (pOperator->exprSupp.pFilterInfo != NULL) { if (pOperator->exprSupp.pFilterInfo != NULL) {
(*status) = FUNC_DATA_REQUIRED_DATA_LOAD; (*status) = FUNC_DATA_REQUIRED_DATA_LOAD;
} else { } else {
@ -373,7 +380,14 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableSca
} else if (*status == FUNC_DATA_REQUIRED_SMA_LOAD) { } else if (*status == FUNC_DATA_REQUIRED_SMA_LOAD) {
pCost->loadBlockStatis += 1; pCost->loadBlockStatis += 1;
loadSMA = true; // mark the operation of load sma; loadSMA = true; // mark the operation of load sma;
bool success = doLoadBlockSMA(pTableScanInfo, pBlock, pTaskInfo); bool success = true;
code = doLoadBlockSMA(pTableScanInfo, pBlock, pTaskInfo, &success);
if (code) {
pAPI->tsdReader.tsdReaderReleaseDataBlock(pTableScanInfo->dataReader);
qError("%s failed to retrieve sma info", GET_TASKID(pTaskInfo));
QUERY_CHECK_CODE(code, lino, _end);
}
if (success) { // failed to load the block sma data, data block statistics does not exist, load data block instead if (success) { // failed to load the block sma data, data block statistics does not exist, load data block instead
qDebug("%s data block SMA loaded, brange:%" PRId64 "-%" PRId64 ", rows:%" PRId64, GET_TASKID(pTaskInfo), qDebug("%s data block SMA loaded, brange:%" PRId64 "-%" PRId64 ", rows:%" PRId64, GET_TASKID(pTaskInfo),
pBlockInfo->window.skey, pBlockInfo->window.ekey, pBlockInfo->rows); pBlockInfo->window.skey, pBlockInfo->window.ekey, pBlockInfo->rows);
@ -387,13 +401,21 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableSca
} }
if(*status != FUNC_DATA_REQUIRED_DATA_LOAD) { if(*status != FUNC_DATA_REQUIRED_DATA_LOAD) {
qError("[loadDataBlock] invalid status:%d", *status); pAPI->tsdReader.tsdReaderReleaseDataBlock(pTableScanInfo->dataReader);
qError("%s loadDataBlock invalid status:%d", GET_TASKID(pTaskInfo), *status);
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
} }
// try to filter data block according to sma info // try to filter data block according to sma info
if (pOperator->exprSupp.pFilterInfo != NULL && (!loadSMA)) { if (pOperator->exprSupp.pFilterInfo != NULL && (!loadSMA)) {
bool success = doLoadBlockSMA(pTableScanInfo, pBlock, pTaskInfo); bool success = true;
code = doLoadBlockSMA(pTableScanInfo, pBlock, pTaskInfo, &success);
if (code) {
pAPI->tsdReader.tsdReaderReleaseDataBlock(pTableScanInfo->dataReader);
qError("%s failed to retrieve sma info", GET_TASKID(pTaskInfo));
QUERY_CHECK_CODE(code, lino, _end);
}
if (success) { if (success) {
size_t size = taosArrayGetSize(pBlock->pDataBlock); size_t size = taosArrayGetSize(pBlock->pDataBlock);
bool keep = false; bool keep = false;
@ -509,7 +531,7 @@ static void freeTableCachedVal(void* param) {
taosMemoryFree(pVal); taosMemoryFree(pVal);
} }
static STableCachedVal* createTableCacheVal(const SMetaReader* pMetaReader) { static int32_t createTableCacheVal(const SMetaReader* pMetaReader, STableCachedVal** ppResVal) {
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0; int32_t lino = 0;
STableCachedVal* pVal = taosMemoryMalloc(sizeof(STableCachedVal)); STableCachedVal* pVal = taosMemoryMalloc(sizeof(STableCachedVal));
@ -526,12 +548,14 @@ static STableCachedVal* createTableCacheVal(const SMetaReader* pMetaReader) {
memcpy(pVal->pTags, pTag, pTag->len); memcpy(pVal->pTags, pTag, pTag->len);
} }
(*ppResVal) = pVal;
_end: _end:
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
return NULL; freeTableCachedVal(pVal);
} }
return pVal; return code;
} }
// const void *key, size_t keyLen, void *value // const void *key, size_t keyLen, void *value
@ -551,6 +575,11 @@ static void doSetNullValue(SSDataBlock* pBlock, const SExprInfo* pExpr, int32_t
} }
} }
static void freeTableCachedValObj(STableCachedVal* pVal) {
taosMemoryFree((void*)pVal->pName);
taosMemoryFree(pVal->pTags);
}
int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int32_t numOfExpr, SSDataBlock* pBlock, int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int32_t numOfExpr, SSDataBlock* pBlock,
int32_t rows, SExecTaskInfo* pTask, STableMetaCacheInfo* pCache) { int32_t rows, SExecTaskInfo* pTask, STableMetaCacheInfo* pCache) {
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
@ -560,6 +589,7 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int
STableCachedVal val = {0}; STableCachedVal val = {0};
SMetaReader mr = {0}; SMetaReader mr = {0};
const char* idStr = pTask->id.str; const char* idStr = pTask->id.str;
int32_t insertRet = TAOS_LRU_STATUS_OK;
// currently only the tbname pseudo column // currently only the tbname pseudo column
if (numOfExpr <= 0) { if (numOfExpr <= 0) {
@ -581,18 +611,18 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int
code = pHandle->api.metaReaderFn.getEntryGetUidCache(&mr, pBlock->info.id.uid); code = pHandle->api.metaReaderFn.getEntryGetUidCache(&mr, pBlock->info.id.uid);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
// when encounter the TSDB_CODE_PAR_TABLE_NOT_EXIST error, we proceed. // when encounter the TSDB_CODE_PAR_TABLE_NOT_EXIST error, we proceed.
if (terrno == TSDB_CODE_PAR_TABLE_NOT_EXIST) { if (code == TSDB_CODE_PAR_TABLE_NOT_EXIST) {
qWarn("failed to get table meta, table may have been dropped, uid:0x%" PRIx64 ", code:%s, %s", qWarn("failed to get table meta, table may have been dropped, uid:0x%" PRIx64 ", code:%s, %s",
pBlock->info.id.uid, tstrerror(terrno), idStr); pBlock->info.id.uid, tstrerror(code), idStr);
// append null value before return to caller, since the caller will ignore this error code and proceed // append null value before return to caller, since the caller will ignore this error code and proceed
doSetNullValue(pBlock, pExpr, numOfExpr); doSetNullValue(pBlock, pExpr, numOfExpr);
} else { } else {
qError("failed to get table meta, uid:0x%" PRIx64 ", code:%s, %s", pBlock->info.id.uid, tstrerror(terrno), qError("failed to get table meta, uid:0x%" PRIx64 ", code:%s, %s", pBlock->info.id.uid, tstrerror(code),
idStr); idStr);
} }
pHandle->api.metaReaderFn.clearReader(&mr); pHandle->api.metaReaderFn.clearReader(&mr);
return terrno; return code;
} }
pHandle->api.metaReaderFn.readerReleaseLock(&mr); pHandle->api.metaReaderFn.readerReleaseLock(&mr);
@ -609,34 +639,33 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int
pHandle->api.metaReaderFn.initReader(&mr, pHandle->vnode, META_READER_LOCK, &pHandle->api.metaFn); pHandle->api.metaReaderFn.initReader(&mr, pHandle->vnode, META_READER_LOCK, &pHandle->api.metaFn);
code = pHandle->api.metaReaderFn.getEntryGetUidCache(&mr, pBlock->info.id.uid); code = pHandle->api.metaReaderFn.getEntryGetUidCache(&mr, pBlock->info.id.uid);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
if (terrno == TSDB_CODE_PAR_TABLE_NOT_EXIST) { if (code == TSDB_CODE_PAR_TABLE_NOT_EXIST) {
qWarn("failed to get table meta, table may have been dropped, uid:0x%" PRIx64 ", code:%s, %s", qWarn("failed to get table meta, table may have been dropped, uid:0x%" PRIx64 ", code:%s, %s",
pBlock->info.id.uid, tstrerror(terrno), idStr); pBlock->info.id.uid, tstrerror(code), idStr);
// append null value before return to caller, since the caller will ignore this error code and proceed // append null value before return to caller, since the caller will ignore this error code and proceed
doSetNullValue(pBlock, pExpr, numOfExpr); doSetNullValue(pBlock, pExpr, numOfExpr);
} else { } else {
qError("failed to get table meta, uid:0x%" PRIx64 ", code:%s, %s", pBlock->info.id.uid, tstrerror(terrno), qError("failed to get table meta, uid:0x%" PRIx64 ", code:%s, %s", pBlock->info.id.uid, tstrerror(code),
idStr); idStr);
} }
pHandle->api.metaReaderFn.clearReader(&mr); pHandle->api.metaReaderFn.clearReader(&mr);
return terrno; return code;
} }
pHandle->api.metaReaderFn.readerReleaseLock(&mr); pHandle->api.metaReaderFn.readerReleaseLock(&mr);
STableCachedVal* pVal = createTableCacheVal(&mr); STableCachedVal* pVal = NULL;
if(!pVal) { code = createTableCacheVal(&mr, &pVal);
return terrno; QUERY_CHECK_CODE(code, lino, _end);
}
val = *pVal; val = *pVal;
freeReader = true; freeReader = true;
int32_t ret = taosLRUCacheInsert(pCache->pTableMetaEntryCache, &pBlock->info.id.uid, sizeof(uint64_t), pVal, insertRet = taosLRUCacheInsert(pCache->pTableMetaEntryCache, &pBlock->info.id.uid, sizeof(uint64_t), pVal,
sizeof(STableCachedVal), freeCachedMetaItem, NULL, TAOS_LRU_PRIORITY_LOW, NULL); sizeof(STableCachedVal), freeCachedMetaItem, NULL, TAOS_LRU_PRIORITY_LOW, NULL);
if (ret != TAOS_LRU_STATUS_OK) { if (insertRet != TAOS_LRU_STATUS_OK) {
qError("failed to put meta into lru cache, code:%d, %s", ret, idStr); qError("failed to put meta into lru cache, code:%d, %s", insertRet, idStr);
freeTableCachedVal(pVal); taosMemoryFreeClear(pVal);
} }
} else { } else {
pCache->cacheHit += 1; pCache->cacheHit += 1;
@ -709,11 +738,14 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int
// restore the rows // restore the rows
pBlock->info.rows = backupRows; pBlock->info.rows = backupRows;
_end:
if (insertRet != TAOS_LRU_STATUS_OK) {
freeTableCachedValObj(&val);
}
if (freeReader) { if (freeReader) {
pHandle->api.metaReaderFn.clearReader(&mr); pHandle->api.metaReaderFn.clearReader(&mr);
} }
_end:
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
} }
@ -3954,6 +3986,9 @@ _end:
} }
static void destroyStreamScanOperatorInfo(void* param) { static void destroyStreamScanOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SStreamScanInfo* pStreamScan = (SStreamScanInfo*)param; SStreamScanInfo* pStreamScan = (SStreamScanInfo*)param;
if (pStreamScan->pTableScanOp && pStreamScan->pTableScanOp->info) { if (pStreamScan->pTableScanOp && pStreamScan->pTableScanOp->info) {

View File

@ -43,6 +43,9 @@ typedef struct SEventWindowInfo {
} SEventWindowInfo; } SEventWindowInfo;
void destroyStreamEventOperatorInfo(void* param) { void destroyStreamEventOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SStreamEventAggOperatorInfo* pInfo = (SStreamEventAggOperatorInfo*)param; SStreamEventAggOperatorInfo* pInfo = (SStreamEventAggOperatorInfo*)param;
cleanupBasicInfo(&pInfo->binfo); cleanupBasicInfo(&pInfo->binfo);
destroyStreamAggSupporter(&pInfo->streamAggSup); destroyStreamAggSupporter(&pInfo->streamAggSup);

View File

@ -459,6 +459,9 @@ void clearGroupResInfo(SGroupResInfo* pGroupResInfo) {
} }
void destroyStreamFinalIntervalOperatorInfo(void* param) { void destroyStreamFinalIntervalOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SStreamIntervalOperatorInfo* pInfo = (SStreamIntervalOperatorInfo*)param; SStreamIntervalOperatorInfo* pInfo = (SStreamIntervalOperatorInfo*)param;
cleanupBasicInfo(&pInfo->binfo); cleanupBasicInfo(&pInfo->binfo);
cleanupAggSup(&pInfo->aggSup); cleanupAggSup(&pInfo->aggSup);
@ -2043,6 +2046,9 @@ void destroyStreamAggSupporter(SStreamAggSupporter* pSup) {
} }
void destroyStreamSessionAggOperatorInfo(void* param) { void destroyStreamSessionAggOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SStreamSessionAggOperatorInfo* pInfo = (SStreamSessionAggOperatorInfo*)param; SStreamSessionAggOperatorInfo* pInfo = (SStreamSessionAggOperatorInfo*)param;
cleanupBasicInfo(&pInfo->binfo); cleanupBasicInfo(&pInfo->binfo);
destroyStreamAggSupporter(&pInfo->streamAggSup); destroyStreamAggSupporter(&pInfo->streamAggSup);
@ -4125,6 +4131,9 @@ _error:
} }
void destroyStreamStateOperatorInfo(void* param) { void destroyStreamStateOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SStreamStateAggOperatorInfo* pInfo = (SStreamStateAggOperatorInfo*)param; SStreamStateAggOperatorInfo* pInfo = (SStreamStateAggOperatorInfo*)param;
cleanupBasicInfo(&pInfo->binfo); cleanupBasicInfo(&pInfo->binfo);
destroyStreamAggSupporter(&pInfo->streamAggSup); destroyStreamAggSupporter(&pInfo->streamAggSup);

View File

@ -552,6 +552,8 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
blockDataCleanup(pInfo->pRes); blockDataCleanup(pInfo->pRes);
pDataBlock = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_COLS); pDataBlock = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_COLS);
QUERY_CHECK_NULL(pDataBlock, code, lino, _end, terrno);
code = blockDataEnsureCapacity(pDataBlock, pOperator->resultInfo.capacity); code = blockDataEnsureCapacity(pDataBlock, pOperator->resultInfo.capacity);
QUERY_CHECK_CODE(code, lino, _end); QUERY_CHECK_CODE(code, lino, _end);
@ -708,6 +710,8 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
int32_t numOfRows = 0; int32_t numOfRows = 0;
dataBlock = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_TAGS); dataBlock = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_TAGS);
QUERY_CHECK_NULL(dataBlock, code, lino, _end, terrno);
code = blockDataEnsureCapacity(dataBlock, pOperator->resultInfo.capacity); code = blockDataEnsureCapacity(dataBlock, pOperator->resultInfo.capacity);
QUERY_CHECK_CODE(code, lino, _end); QUERY_CHECK_CODE(code, lino, _end);
@ -1354,6 +1358,8 @@ static SSDataBlock* sysTableBuildUserTablesByUids(SOperatorInfo* pOperator) {
varDataSetLen(dbname, strlen(varDataVal(dbname))); varDataSetLen(dbname, strlen(varDataVal(dbname)));
p = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_TABLES); p = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_TABLES);
QUERY_CHECK_NULL(p, code, lino, _end, terrno);
code = blockDataEnsureCapacity(p, pOperator->resultInfo.capacity); code = blockDataEnsureCapacity(p, pOperator->resultInfo.capacity);
QUERY_CHECK_CODE(code, lino, _end); QUERY_CHECK_CODE(code, lino, _end);
@ -2075,6 +2081,7 @@ static SSDataBlock* sysTableScanFromMNode(SOperatorInfo* pOperator, SSysTableSca
int32_t tempRes = tSerializeSRetrieveTableReq(buf1, contLen, &pInfo->req); int32_t tempRes = tSerializeSRetrieveTableReq(buf1, contLen, &pInfo->req);
if (tempRes < 0) { if (tempRes < 0) {
code = terrno; code = terrno;
taosMemoryFree(buf1);
return NULL; return NULL;
} }
@ -2083,6 +2090,7 @@ static SSDataBlock* sysTableScanFromMNode(SOperatorInfo* pOperator, SSysTableSca
if (NULL == pMsgSendInfo) { if (NULL == pMsgSendInfo) {
qError("%s prepare message %d failed", GET_TASKID(pTaskInfo), (int32_t)sizeof(SMsgSendInfo)); qError("%s prepare message %d failed", GET_TASKID(pTaskInfo), (int32_t)sizeof(SMsgSendInfo));
pTaskInfo->code = TSDB_CODE_OUT_OF_MEMORY; pTaskInfo->code = TSDB_CODE_OUT_OF_MEMORY;
taosMemoryFree(buf1);
return NULL; return NULL;
} }

View File

@ -1208,6 +1208,9 @@ static void freeItem(void* param) {
} }
void destroyIntervalOperatorInfo(void* param) { void destroyIntervalOperatorInfo(void* param) {
if (param == NULL) {
return;
}
SIntervalAggOperatorInfo* pInfo = (SIntervalAggOperatorInfo*)param; SIntervalAggOperatorInfo* pInfo = (SIntervalAggOperatorInfo*)param;
cleanupBasicInfo(&pInfo->binfo); cleanupBasicInfo(&pInfo->binfo);
cleanupAggSup(&pInfo->aggSup); cleanupAggSup(&pInfo->aggSup);

View File

@ -1917,14 +1917,21 @@ static int32_t getBufIncForNewRow(SSortHandle* pHandle, int32_t dstRowIndex, SSD
} }
static int32_t initMergeSup(SBlkMergeSupport* pSup, SArray* pBlockList, int32_t tsOrder, int32_t tsSlotId, SBlockOrderInfo* pPkOrderInfo) { static int32_t initMergeSup(SBlkMergeSupport* pSup, SArray* pBlockList, int32_t tsOrder, int32_t tsSlotId, SBlockOrderInfo* pPkOrderInfo) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
memset(pSup, 0, sizeof(SBlkMergeSupport)); memset(pSup, 0, sizeof(SBlkMergeSupport));
int32_t numOfBlocks = taosArrayGetSize(pBlockList); int32_t numOfBlocks = taosArrayGetSize(pBlockList);
pSup->aRowIdx = taosMemoryCalloc(numOfBlocks, sizeof(int32_t)); pSup->aRowIdx = taosMemoryCalloc(numOfBlocks, sizeof(int32_t));
QUERY_CHECK_NULL(pSup->aRowIdx, code, lino, _end, terrno);
pSup->aTs = taosMemoryCalloc(numOfBlocks, sizeof(int64_t*)); pSup->aTs = taosMemoryCalloc(numOfBlocks, sizeof(int64_t*));
QUERY_CHECK_NULL(pSup->aTs, code, lino, _end, terrno);
pSup->tsOrder = tsOrder; pSup->tsOrder = tsOrder;
pSup->aBlks = taosMemoryCalloc(numOfBlocks, sizeof(SSDataBlock*)); pSup->aBlks = taosMemoryCalloc(numOfBlocks, sizeof(SSDataBlock*));
QUERY_CHECK_NULL(pSup->aBlks, code, lino, _end, terrno);
for (int32_t i = 0; i < numOfBlocks; ++i) { for (int32_t i = 0; i < numOfBlocks; ++i) {
SSDataBlock* pBlock = taosArrayGetP(pBlockList, i); SSDataBlock* pBlock = taosArrayGetP(pBlockList, i);
@ -1935,7 +1942,12 @@ static int32_t initMergeSup(SBlkMergeSupport* pSup, SArray* pBlockList, int32_t
} }
pSup->pPkOrder = pPkOrderInfo; pSup->pPkOrder = pPkOrderInfo;
return TSDB_CODE_SUCCESS;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
} }
static void cleanupMergeSup(SBlkMergeSupport* pSup) { static void cleanupMergeSup(SBlkMergeSupport* pSup) {

View File

@ -96,19 +96,20 @@ int32_t avgInvertFunction(SqlFunctionCtx* pCtx);
int32_t avgCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx); int32_t avgCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx);
int32_t getAvgInfoSize(); int32_t getAvgInfoSize();
bool getStddevFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getStdFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
int32_t stddevFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo); int32_t stdFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo);
int32_t stddevFunction(SqlFunctionCtx* pCtx); int32_t stdFunction(SqlFunctionCtx* pCtx);
int32_t stddevFunctionMerge(SqlFunctionCtx* pCtx); int32_t stdFunctionMerge(SqlFunctionCtx* pCtx);
int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t stddevPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock); int32_t stdvarFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
int32_t stdPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock);
#ifdef BUILD_NO_CALL #ifdef BUILD_NO_CALL
int32_t stddevInvertFunction(SqlFunctionCtx* pCtx); int32_t stdInvertFunction(SqlFunctionCtx* pCtx);
#endif #endif
int32_t stddevCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx); int32_t stdCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx);
int32_t getStddevInfoSize(); int32_t getStdInfoSize();
bool getLeastSQRFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv); bool getLeastSQRFuncEnv(struct SFunctionNode* pFunc, SFuncExecEnv* pEnv);
int32_t leastSQRFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo); int32_t leastSQRFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo);

View File

@ -247,7 +247,7 @@ static int32_t addTimezoneParam(SNodeList* pList) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t addDbPrecisonParam(SNodeList** pList, uint8_t precision) { static int32_t addUint8Param(SNodeList** pList, uint8_t param) {
SValueNode* pVal = NULL; SValueNode* pVal = NULL;
int32_t code = nodesMakeNode(QUERY_NODE_VALUE, (SNode**)&pVal); int32_t code = nodesMakeNode(QUERY_NODE_VALUE, (SNode**)&pVal);
if (pVal == NULL) { if (pVal == NULL) {
@ -259,9 +259,9 @@ static int32_t addDbPrecisonParam(SNodeList** pList, uint8_t precision) {
pVal->notReserved = true; pVal->notReserved = true;
pVal->node.resType.type = TSDB_DATA_TYPE_TINYINT; pVal->node.resType.type = TSDB_DATA_TYPE_TINYINT;
pVal->node.resType.bytes = tDataTypes[TSDB_DATA_TYPE_TINYINT].bytes; pVal->node.resType.bytes = tDataTypes[TSDB_DATA_TYPE_TINYINT].bytes;
pVal->node.resType.precision = precision; pVal->node.resType.precision = param;
pVal->datum.i = (int64_t)precision; pVal->datum.i = (int64_t)param;
pVal->typeData = (int64_t)precision; pVal->typeData = (int64_t)param;
code = nodesListMakeAppend(pList, (SNode*)pVal); code = nodesListMakeAppend(pList, (SNode*)pVal);
if (TSDB_CODE_SUCCESS != code) { if (TSDB_CODE_SUCCESS != code) {
@ -493,7 +493,7 @@ static int32_t translateAvgStateMerge(SFunctionNode* pFunc, char* pErrBuf, int32
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateStddevPartial(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateStdPartial(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) { if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
} }
@ -503,11 +503,11 @@ static int32_t translateStddevPartial(SFunctionNode* pFunc, char* pErrBuf, int32
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} }
pFunc->node.resType = (SDataType){.bytes = getStddevInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY}; pFunc->node.resType = (SDataType){.bytes = getStdInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY};
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateStddevMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateStdMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) { if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
} }
@ -522,7 +522,7 @@ static int32_t translateStddevMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateStddevState(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateStdState(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) { if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
} }
@ -532,11 +532,11 @@ static int32_t translateStddevState(SFunctionNode* pFunc, char* pErrBuf, int32_t
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} }
pFunc->node.resType = (SDataType){.bytes = getStddevInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY}; pFunc->node.resType = (SDataType){.bytes = getStdInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY};
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateStddevStateMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateStdStateMerge(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) { if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
} }
@ -546,7 +546,7 @@ static int32_t translateStddevStateMerge(SFunctionNode* pFunc, char* pErrBuf, in
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} }
pFunc->node.resType = (SDataType){.bytes = getStddevInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY}; pFunc->node.resType = (SDataType){.bytes = getStdInfoSize() + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_BINARY};
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
@ -563,7 +563,7 @@ static int32_t translateNowToday(SFunctionNode* pFunc, char* pErrBuf, int32_t le
// add database precision as param // add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision; uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -573,6 +573,53 @@ static int32_t translateNowToday(SFunctionNode* pFunc, char* pErrBuf, int32_t le
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translatePi(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
pFunc->node.resType =
(SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_DOUBLE].bytes, .type = TSDB_DATA_TYPE_DOUBLE};
return TSDB_CODE_SUCCESS;
}
static int32_t translateRound(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (2 != LIST_LENGTH(pFunc->pParameterList) && 1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t paraType = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (!IS_NUMERIC_TYPE(paraType) && !IS_NULL_TYPE(paraType)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} else if (IS_NULL_TYPE(paraType)) {
paraType = TSDB_DATA_TYPE_BIGINT;
}
if (2 == LIST_LENGTH(pFunc->pParameterList)) {
uint8_t paraType2 = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->type;
if (!IS_NUMERIC_TYPE(paraType2) && !IS_NULL_TYPE(paraType2)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
}
pFunc->node.resType = (SDataType){.bytes = tDataTypes[paraType].bytes, .type = paraType};
return TSDB_CODE_SUCCESS;
}
static int32_t translateTrunc(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (2 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t paraType = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (!IS_NUMERIC_TYPE(paraType) && !IS_NULL_TYPE(paraType)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t paraType2 = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->type;
if (!IS_NUMERIC_TYPE(paraType2) && !IS_NULL_TYPE(paraType2)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = tDataTypes[paraType].bytes, .type = paraType};
return TSDB_CODE_SUCCESS;
}
static int32_t translateTimePseudoColumn(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateTimePseudoColumn(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
// pseudo column do not need to check parameters // pseudo column do not need to check parameters
@ -1745,7 +1792,7 @@ static int32_t translateIrate(SFunctionNode* pFunc, char* pErrBuf, int32_t len)
// add database precision as param // add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision; uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -1776,7 +1823,7 @@ static int32_t translateIrateImpl(SFunctionNode* pFunc, char* pErrBuf, int32_t l
// add database precision as param // add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision; uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -2032,7 +2079,22 @@ static int32_t translateLength(SFunctionNode* pFunc, char* pErrBuf, int32_t len)
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
} }
if (!IS_STR_DATA_TYPE(getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type)) { uint8_t paraType = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (!IS_STR_DATA_TYPE(paraType) && !IS_NULL_TYPE(paraType)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes, .type = TSDB_DATA_TYPE_BIGINT};
return TSDB_CODE_SUCCESS;
}
static int32_t translateCharLength(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t paraType = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (paraType == TSDB_DATA_TYPE_VARBINARY || (!IS_STR_DATA_TYPE(paraType) && !IS_NULL_TYPE(paraType))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} }
@ -2124,31 +2186,182 @@ static int32_t translateSubstr(SFunctionNode* pFunc, char* pErrBuf, int32_t len)
uint8_t para0Type = pPara0->resType.type; uint8_t para0Type = pPara0->resType.type;
uint8_t para1Type = pPara1->resType.type; uint8_t para1Type = pPara1->resType.type;
if (TSDB_DATA_TYPE_VARBINARY == para0Type || !IS_STR_DATA_TYPE(para0Type) || !IS_INTEGER_TYPE(para1Type)) { if (TSDB_DATA_TYPE_VARBINARY == para0Type ||
(!IS_STR_DATA_TYPE(para0Type) && !IS_NULL_TYPE(para0Type)) ||
(!IS_INTEGER_TYPE(para1Type) && !IS_NULL_TYPE(para1Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} }
if (((SValueNode*)pPara1)->datum.i == 0) {
return invaildFuncParaValueErrMsg(pErrBuf, len, pFunc->functionName);
}
if (3 == numOfParams) { if (3 == numOfParams) {
SExprNode* pPara2 = (SExprNode*)nodesListGetNode(pFunc->pParameterList, 2); SExprNode* pPara2 = (SExprNode*)nodesListGetNode(pFunc->pParameterList, 2);
uint8_t para2Type = pPara2->resType.type; uint8_t para2Type = pPara2->resType.type;
if (!IS_INTEGER_TYPE(para2Type)) { if (!IS_INTEGER_TYPE(para2Type) && !IS_NULL_TYPE(para2Type)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
} }
int64_t v = ((SValueNode*)pPara2)->datum.i; int64_t v = ((SValueNode*)pPara2)->datum.i;
if (v < 0) {
return invaildFuncParaValueErrMsg(pErrBuf, len, pFunc->functionName);
}
} }
pFunc->node.resType = (SDataType){.bytes = pPara0->resType.bytes, .type = pPara0->resType.type}; pFunc->node.resType = (SDataType){.bytes = pPara0->resType.bytes, .type = pPara0->resType.type};
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateSubstrIdx(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (3 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
SExprNode* pPara0 = (SExprNode*)nodesListGetNode(pFunc->pParameterList, 0);
SExprNode* pPara1 = (SExprNode*)nodesListGetNode(pFunc->pParameterList, 1);
SExprNode* pPara2 = (SExprNode*)nodesListGetNode(pFunc->pParameterList, 2);
uint8_t para0Type = pPara0->resType.type;
uint8_t para1Type = pPara1->resType.type;
uint8_t para2Type = pPara2->resType.type;
if (TSDB_DATA_TYPE_VARBINARY == para0Type || (!IS_STR_DATA_TYPE(para0Type) && !IS_NULL_TYPE(para0Type)) ||
TSDB_DATA_TYPE_VARBINARY == para1Type || (!IS_STR_DATA_TYPE(para1Type) && !IS_NULL_TYPE(para1Type)) ||
(!IS_INTEGER_TYPE(para2Type) && !IS_NULL_TYPE(para2Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = pPara0->resType.bytes, .type = pPara0->resType.type};
return TSDB_CODE_SUCCESS;
}
static int32_t translateChar(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
SNode *node;
FOREACH(node, pFunc->pParameterList) {
uint8_t paraType = getSDataTypeFromNode(node)->type;
if (paraType == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(paraType) && !IS_NUMERIC_TYPE(paraType) && !IS_NULL_TYPE(paraType))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
}
int32_t numOfParams = LIST_LENGTH(pFunc->pParameterList);
pFunc->node.resType = (SDataType){.bytes = 4 * numOfParams + 2, .type = TSDB_DATA_TYPE_VARCHAR};
return TSDB_CODE_SUCCESS;
}
static int32_t translateAscii(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t paraType = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (paraType == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(paraType) && !IS_NULL_TYPE(paraType))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_UTINYINT].bytes, .type = TSDB_DATA_TYPE_UTINYINT};
return TSDB_CODE_SUCCESS;
}
static int32_t translatePosition(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (2 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para0Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (para0Type == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(para0Type) && !IS_NULL_TYPE(para0Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para1Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->type;
if (para1Type == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(para1Type) && !IS_NULL_TYPE(para1Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
pFunc->node.resType = (SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes, .type = TSDB_DATA_TYPE_BIGINT};
return TSDB_CODE_SUCCESS;
}
static int32_t translateTrim(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (2 != LIST_LENGTH(pFunc->pParameterList) && 1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para0Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (para0Type == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(para0Type) && !IS_NULL_TYPE(para0Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
int32_t resLen = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->bytes;
uint8_t type = para0Type;
if (2 == LIST_LENGTH(pFunc->pParameterList)) {
uint8_t para1Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->type;
if (para1Type == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(para1Type) && !IS_NULL_TYPE(para1Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
resLen = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->bytes;
type = para1Type;
}
if (type == TSDB_DATA_TYPE_NCHAR) {
resLen *= TSDB_NCHAR_SIZE;
}
uint8_t trimType = pFunc->trimType;
int32_t code = addUint8Param(&pFunc->pParameterList, trimType);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
pFunc->node.resType = (SDataType){.bytes = resLen, .type = type};
return TSDB_CODE_SUCCESS;
}
static int32_t translateReplace(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (3 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
for (int32_t i = 0; i < 3; ++i) {
uint8_t paraType = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, i))->type;
if (paraType == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(paraType) && !IS_NULL_TYPE(paraType))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
}
uint8_t type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
int32_t orgLen = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->bytes;
int32_t fromLen = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->bytes;
int32_t toLen = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 2))->bytes;
int32_t resLen = orgLen + orgLen / fromLen * (toLen - fromLen);
pFunc->node.resType = (SDataType){.bytes = resLen, .type = type};
return TSDB_CODE_SUCCESS;
}
static int32_t translateRepeat(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (2 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para0Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if (para0Type == TSDB_DATA_TYPE_VARBINARY ||
(!IS_STR_DATA_TYPE(para0Type) && !IS_NULL_TYPE(para0Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para1Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->type;
if (!IS_INTEGER_TYPE(para1Type) && !IS_NULL_TYPE(para1Type)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
int32_t orgLen = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->bytes;
int32_t count = TMAX((int32_t)((SValueNode*)nodesListGetNode(pFunc->pParameterList, 1))->datum.i, 1);
int32_t resLen = orgLen * count;
pFunc->node.resType = (SDataType){.bytes = resLen, .type = type};
return TSDB_CODE_SUCCESS;
}
static int32_t translateCast(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateCast(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
// The number of parameters has been limited by the syntax definition // The number of parameters has been limited by the syntax definition
@ -2180,7 +2393,7 @@ static int32_t translateCast(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
// add database precision as param // add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision; uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -2251,7 +2464,7 @@ static int32_t translateToUnixtimestamp(SFunctionNode* pFunc, char* pErrBuf, int
// add database precision as param // add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision; uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -2325,7 +2538,7 @@ static int32_t translateTimeTruncate(SFunctionNode* pFunc, char* pErrBuf, int32_
// add database precision as param // add database precision as param
code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -2375,7 +2588,7 @@ static int32_t translateTimeDiff(SFunctionNode* pFunc, char* pErrBuf, int32_t le
} }
} }
int32_t code = addDbPrecisonParam(&pFunc->pParameterList, dbPrec); int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
return code; return code;
} }
@ -2384,6 +2597,91 @@ static int32_t translateTimeDiff(SFunctionNode* pFunc, char* pErrBuf, int32_t le
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t translateWeekday(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para1Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if ((!IS_STR_DATA_TYPE(para1Type) && !IS_INTEGER_TYPE(para1Type) &&
!IS_TIMESTAMP_TYPE(para1Type) && !IS_NULL_TYPE(para1Type))) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
// add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
pFunc->node.resType =
(SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes, .type = TSDB_DATA_TYPE_BIGINT};
return TSDB_CODE_SUCCESS;
}
static int32_t translateWeek(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList) && 2 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para1Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if ((!IS_STR_DATA_TYPE(para1Type) && !IS_INTEGER_TYPE(para1Type) &&
!IS_TIMESTAMP_TYPE(para1Type)) && !IS_NULL_TYPE(para1Type)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
if (2 == LIST_LENGTH(pFunc->pParameterList)) {
uint8_t para2Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 1))->type;
if (!IS_INTEGER_TYPE(para2Type) && !IS_NULL_TYPE(para2Type)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
if (IS_INTEGER_TYPE(para2Type)) {
SValueNode* pValue = (SValueNode*)nodesListGetNode(pFunc->pParameterList, 1);
if (pValue->datum.i < 0 || pValue->datum.i > 7) {
return invaildFuncParaValueErrMsg(pErrBuf, len, pFunc->functionName);
}
}
}
// add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
pFunc->node.resType =
(SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes, .type = TSDB_DATA_TYPE_BIGINT};
return TSDB_CODE_SUCCESS;
}
static int32_t translateWeekofyear(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
}
uint8_t para1Type = getSDataTypeFromNode(nodesListGetNode(pFunc->pParameterList, 0))->type;
if ((!IS_STR_DATA_TYPE(para1Type) && !IS_INTEGER_TYPE(para1Type) &&
!IS_TIMESTAMP_TYPE(para1Type)) && !IS_NULL_TYPE(para1Type)) {
return invaildFuncParaTypeErrMsg(pErrBuf, len, pFunc->functionName);
}
// add database precision as param
uint8_t dbPrec = pFunc->node.resType.precision;
int32_t code = addUint8Param(&pFunc->pParameterList, dbPrec);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
pFunc->node.resType =
(SDataType){.bytes = tDataTypes[TSDB_DATA_TYPE_BIGINT].bytes, .type = TSDB_DATA_TYPE_BIGINT};
return TSDB_CODE_SUCCESS;
}
static int32_t translateToJson(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { static int32_t translateToJson(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
if (1 != LIST_LENGTH(pFunc->pParameterList)) { if (1 != LIST_LENGTH(pFunc->pParameterList)) {
return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName); return invaildFuncParaNumErrMsg(pErrBuf, len, pFunc->functionName);
@ -2623,47 +2921,47 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.type = FUNCTION_TYPE_STDDEV, .type = FUNCTION_TYPE_STDDEV,
.classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC, .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC,
.translateFunc = translateInNumOutDou, .translateFunc = translateInNumOutDou,
.getEnvFunc = getStddevFuncEnv, .getEnvFunc = getStdFuncEnv,
.initFunc = stddevFunctionSetup, .initFunc = stdFunctionSetup,
.processFunc = stddevFunction, .processFunc = stdFunction,
.sprocessFunc = stddevScalarFunction, .sprocessFunc = stdScalarFunction,
.finalizeFunc = stddevFinalize, .finalizeFunc = stddevFinalize,
#ifdef BUILD_NO_CALL #ifdef BUILD_NO_CALL
.invertFunc = stddevInvertFunction, .invertFunc = stdInvertFunction,
#endif #endif
.combineFunc = stddevCombine, .combineFunc = stdCombine,
.pPartialFunc = "_stddev_partial", .pPartialFunc = "_std_partial",
.pStateFunc = "_stddev_state", .pStateFunc = "_std_state",
.pMergeFunc = "_stddev_merge" .pMergeFunc = "_stddev_merge"
}, },
{ {
.name = "_stddev_partial", .name = "_std_partial",
.type = FUNCTION_TYPE_STDDEV_PARTIAL, .type = FUNCTION_TYPE_STD_PARTIAL,
.classification = FUNC_MGT_AGG_FUNC, .classification = FUNC_MGT_AGG_FUNC,
.translateFunc = translateStddevPartial, .translateFunc = translateStdPartial,
.getEnvFunc = getStddevFuncEnv, .getEnvFunc = getStdFuncEnv,
.initFunc = stddevFunctionSetup, .initFunc = stdFunctionSetup,
.processFunc = stddevFunction, .processFunc = stdFunction,
.finalizeFunc = stddevPartialFinalize, .finalizeFunc = stdPartialFinalize,
#ifdef BUILD_NO_CALL #ifdef BUILD_NO_CALL
.invertFunc = stddevInvertFunction, .invertFunc = stdInvertFunction,
#endif #endif
.combineFunc = stddevCombine, .combineFunc = stdCombine,
}, },
{ {
.name = "_stddev_merge", .name = "_stddev_merge",
.type = FUNCTION_TYPE_STDDEV_MERGE, .type = FUNCTION_TYPE_STDDEV_MERGE,
.classification = FUNC_MGT_AGG_FUNC, .classification = FUNC_MGT_AGG_FUNC,
.translateFunc = translateStddevMerge, .translateFunc = translateStdMerge,
.getEnvFunc = getStddevFuncEnv, .getEnvFunc = getStdFuncEnv,
.initFunc = stddevFunctionSetup, .initFunc = stdFunctionSetup,
.processFunc = stddevFunctionMerge, .processFunc = stdFunctionMerge,
.finalizeFunc = stddevFinalize, .finalizeFunc = stddevFinalize,
#ifdef BUILD_NO_CALL #ifdef BUILD_NO_CALL
.invertFunc = stddevInvertFunction, .invertFunc = stdInvertFunction,
#endif #endif
.combineFunc = stddevCombine, .combineFunc = stdCombine,
.pPartialFunc = "_stddev_state_merge", .pPartialFunc = "_std_state_merge",
.pMergeFunc = "_stddev_merge", .pMergeFunc = "_stddev_merge",
}, },
{ {
@ -3409,7 +3707,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.name = "round", .name = "round",
.type = FUNCTION_TYPE_ROUND, .type = FUNCTION_TYPE_ROUND,
.classification = FUNC_MGT_SCALAR_FUNC, .classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateInOutNum, .translateFunc = translateRound,
.getEnvFunc = NULL, .getEnvFunc = NULL,
.initFunc = NULL, .initFunc = NULL,
.sprocessFunc = roundFunction, .sprocessFunc = roundFunction,
@ -3489,7 +3787,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.name = "char_length", .name = "char_length",
.type = FUNCTION_TYPE_CHAR_LENGTH, .type = FUNCTION_TYPE_CHAR_LENGTH,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC, .classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateLength, .translateFunc = translateCharLength,
.getEnvFunc = NULL, .getEnvFunc = NULL,
.initFunc = NULL, .initFunc = NULL,
.sprocessFunc = charLengthFunction, .sprocessFunc = charLengthFunction,
@ -3998,26 +4296,26 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.finalizeFunc = NULL .finalizeFunc = NULL
}, },
{ {
.name = "_stddev_state", .name = "_std_state",
.type = FUNCTION_TYPE_STDDEV_STATE, .type = FUNCTION_TYPE_STD_STATE,
.classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC, .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC,
.translateFunc = translateStddevState, .translateFunc = translateStdState,
.getEnvFunc = getStddevFuncEnv, .getEnvFunc = getStdFuncEnv,
.initFunc = stddevFunctionSetup, .initFunc = stdFunctionSetup,
.processFunc = stddevFunction, .processFunc = stdFunction,
.finalizeFunc = stddevPartialFinalize, .finalizeFunc = stdPartialFinalize,
.pPartialFunc = "_stddev_partial", .pPartialFunc = "_std_partial",
.pMergeFunc = "_stddev_state_merge", .pMergeFunc = "_std_state_merge",
}, },
{ {
.name = "_stddev_state_merge", .name = "_std_state_merge",
.type = FUNCTION_TYPE_STDDEV_STATE_MERGE, .type = FUNCTION_TYPE_STD_STATE_MERGE,
.classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC, .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC,
.translateFunc = translateStddevStateMerge, .translateFunc = translateStdStateMerge,
.getEnvFunc = getStddevFuncEnv, .getEnvFunc = getStdFuncEnv,
.initFunc = stddevFunctionSetup, .initFunc = stdFunctionSetup,
.processFunc = stddevFunctionMerge, .processFunc = stdFunctionMerge,
.finalizeFunc = stddevPartialFinalize, .finalizeFunc = stdPartialFinalize,
}, },
{ {
.name = "_avg_state", .name = "_avg_state",
@ -4152,6 +4450,268 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.processFunc = groupConstValueFunction, .processFunc = groupConstValueFunction,
.finalizeFunc = groupConstValueFinalize, .finalizeFunc = groupConstValueFinalize,
}, },
{
.name = "stddev_pop",
.type = FUNCTION_TYPE_STDDEV,
.classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC,
.translateFunc = translateInNumOutDou,
.getEnvFunc = getStdFuncEnv,
.initFunc = stdFunctionSetup,
.processFunc = stdFunction,
.sprocessFunc = stdScalarFunction,
.finalizeFunc = stddevFinalize,
#ifdef BUILD_NO_CALL
.invertFunc = stdInvertFunction,
#endif
.combineFunc = stdCombine,
.pPartialFunc = "_std_partial",
.pStateFunc = "_std_state",
.pMergeFunc = "_stddev_merge"
},
{
.name = "var_pop",
.type = FUNCTION_TYPE_STDVAR,
.classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_TSMA_FUNC,
.translateFunc = translateInNumOutDou,
.getEnvFunc = getStdFuncEnv,
.initFunc = stdFunctionSetup,
.processFunc = stdFunction,
.sprocessFunc = stdScalarFunction,
.finalizeFunc = stdvarFinalize,
#ifdef BUILD_NO_CALL
.invertFunc = stdInvertFunction,
#endif
.combineFunc = stdCombine,
.pPartialFunc = "_std_partial",
.pStateFunc = "_std_state",
.pMergeFunc = "_stdvar_merge"
},
{
.name = "_stdvar_merge",
.type = FUNCTION_TYPE_STDVAR_MERGE,
.classification = FUNC_MGT_AGG_FUNC,
.translateFunc = translateStdMerge,
.getEnvFunc = getStdFuncEnv,
.initFunc = stdFunctionSetup,
.processFunc = stdFunctionMerge,
.finalizeFunc = stdvarFinalize,
#ifdef BUILD_NO_CALL
.invertFunc = stdInvertFunction,
#endif
.combineFunc = stdCombine,
.pPartialFunc = "_std_state_merge",
.pMergeFunc = "_stdvar_merge",
},
{
.name = "pi",
.type = FUNCTION_TYPE_PI,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translatePi,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = piFunction,
.finalizeFunc = NULL
},
{
.name = "exp",
.type = FUNCTION_TYPE_EXP,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateInNumOutDou,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = expFunction,
.finalizeFunc = NULL
},
{
.name = "ln",
.type = FUNCTION_TYPE_LN,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateInNumOutDou,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = lnFunction,
.finalizeFunc = NULL
},
{
.name = "mod",
.type = FUNCTION_TYPE_MOD,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateIn2NumOutDou,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = modFunction,
.finalizeFunc = NULL
},
{
.name = "sign",
.type = FUNCTION_TYPE_SIGN,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateInOutNum,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = signFunction,
.finalizeFunc = NULL
},
{
.name = "degrees",
.type = FUNCTION_TYPE_DEGREES,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateInNumOutDou,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = degreesFunction,
.finalizeFunc = NULL
},
{
.name = "radians",
.type = FUNCTION_TYPE_RADIANS,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateInNumOutDou,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = radiansFunction,
.finalizeFunc = NULL
},
{
.name = "truncate",
.type = FUNCTION_TYPE_TRUNCATE,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateTrunc,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = truncFunction,
.finalizeFunc = NULL
},
{
.name = "trunc",
.type = FUNCTION_TYPE_TRUNCATE,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateTrunc,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = truncFunction,
.finalizeFunc = NULL
},
{
.name = "substring",
.type = FUNCTION_TYPE_SUBSTR,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateSubstr,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = substrFunction,
.finalizeFunc = NULL
},
{
.name = "substring_index",
.type = FUNCTION_TYPE_SUBSTR_IDX,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateSubstrIdx,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = substrIdxFunction,
.finalizeFunc = NULL
},
{
.name = "char",
.type = FUNCTION_TYPE_CHAR,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateChar,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = charFunction,
.finalizeFunc = NULL
},
{
.name = "ascii",
.type = FUNCTION_TYPE_ASCII,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateAscii,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = asciiFunction,
.finalizeFunc = NULL
},
{
.name = "position",
.type = FUNCTION_TYPE_POSITION,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translatePosition,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = positionFunction,
.finalizeFunc = NULL
},
{
.name = "trim",
.type = FUNCTION_TYPE_TRIM,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateTrim,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = trimFunction,
.finalizeFunc = NULL
},
{
.name = "replace",
.type = FUNCTION_TYPE_REPLACE,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateReplace,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = replaceFunction,
.finalizeFunc = NULL
},
{
.name = "repeat",
.type = FUNCTION_TYPE_REPEAT,
.classification = FUNC_MGT_SCALAR_FUNC | FUNC_MGT_STRING_FUNC,
.translateFunc = translateRepeat,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = repeatFunction,
.finalizeFunc = NULL
},
{
.name = "weekday",
.type = FUNCTION_TYPE_WEEKDAY,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateWeekday,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = weekdayFunction,
.finalizeFunc = NULL
},
{
.name = "dayofweek",
.type = FUNCTION_TYPE_DAYOFWEEK,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateWeekday,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = dayofweekFunction,
.finalizeFunc = NULL
},
{
.name = "week",
.type = FUNCTION_TYPE_WEEK,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateWeek,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = weekFunction,
.finalizeFunc = NULL
},
{
.name = "weekofyear",
.type = FUNCTION_TYPE_WEEKOFYEAR,
.classification = FUNC_MGT_SCALAR_FUNC,
.translateFunc = translateWeekofyear,
.getEnvFunc = NULL,
.initFunc = NULL,
.sprocessFunc = weekofyearFunction,
.finalizeFunc = NULL
},
}; };
// clang-format on // clang-format on

View File

@ -63,7 +63,7 @@ typedef struct STopBotRes {
STopBotResItem* pItems; STopBotResItem* pItems;
} STopBotRes; } STopBotRes;
typedef struct SStddevRes { typedef struct SStdRes {
double result; double result;
int64_t count; int64_t count;
union { union {
@ -77,7 +77,7 @@ typedef struct SStddevRes {
uint64_t usum; uint64_t usum;
}; };
int16_t type; int16_t type;
} SStddevRes; } SStdRes;
typedef struct SLeastSQRInfo { typedef struct SLeastSQRInfo {
double matrix[2][3]; double matrix[2][3];
@ -1300,14 +1300,14 @@ int32_t maxCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) {
return minMaxCombine(pDestCtx, pSourceCtx, 0); return minMaxCombine(pDestCtx, pSourceCtx, 0);
} }
int32_t getStddevInfoSize() { return (int32_t)sizeof(SStddevRes); } int32_t getStdInfoSize() { return (int32_t)sizeof(SStdRes); }
bool getStddevFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) { bool getStdFuncEnv(SFunctionNode* pFunc, SFuncExecEnv* pEnv) {
pEnv->calcMemSize = sizeof(SStddevRes); pEnv->calcMemSize = sizeof(SStdRes);
return true; return true;
} }
int32_t stddevFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo) { int32_t stdFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultInfo) {
if (pResultInfo->initialized) { if (pResultInfo->initialized) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
@ -1315,20 +1315,20 @@ int32_t stddevFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResultIn
return TSDB_CODE_FUNC_SETUP_ERROR; return TSDB_CODE_FUNC_SETUP_ERROR;
} }
SStddevRes* pRes = GET_ROWCELL_INTERBUF(pResultInfo); SStdRes* pRes = GET_ROWCELL_INTERBUF(pResultInfo);
(void)memset(pRes, 0, sizeof(SStddevRes)); (void)memset(pRes, 0, sizeof(SStdRes));
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t stddevFunction(SqlFunctionCtx* pCtx) { int32_t stdFunction(SqlFunctionCtx* pCtx) {
int32_t numOfElem = 0; int32_t numOfElem = 0;
// Only the pre-computing information loaded and actual data does not loaded // Only the pre-computing information loaded and actual data does not loaded
SInputColumnInfoData* pInput = &pCtx->input; SInputColumnInfoData* pInput = &pCtx->input;
int32_t type = pInput->pData[0]->info.type; int32_t type = pInput->pData[0]->info.type;
SStddevRes* pStddevRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); SStdRes* pStdRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
pStddevRes->type = type; pStdRes->type = type;
// computing based on the true data block // computing based on the true data block
SColumnInfoData* pCol = pInput->pData[0]; SColumnInfoData* pCol = pInput->pData[0];
@ -1350,9 +1350,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->isum += plist[i]; pStdRes->isum += plist[i];
pStddevRes->quadraticISum += plist[i] * plist[i]; pStdRes->quadraticISum += plist[i] * plist[i];
} }
break; break;
@ -1366,9 +1366,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->isum += plist[i]; pStdRes->isum += plist[i];
pStddevRes->quadraticISum += plist[i] * plist[i]; pStdRes->quadraticISum += plist[i] * plist[i];
} }
break; break;
} }
@ -1381,9 +1381,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->isum += plist[i]; pStdRes->isum += plist[i];
pStddevRes->quadraticISum += plist[i] * plist[i]; pStdRes->quadraticISum += plist[i] * plist[i];
} }
break; break;
@ -1397,9 +1397,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->isum += plist[i]; pStdRes->isum += plist[i];
pStddevRes->quadraticISum += plist[i] * plist[i]; pStdRes->quadraticISum += plist[i] * plist[i];
} }
break; break;
} }
@ -1412,9 +1412,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->usum += plist[i]; pStdRes->usum += plist[i];
pStddevRes->quadraticUSum += plist[i] * plist[i]; pStdRes->quadraticUSum += plist[i] * plist[i];
} }
break; break;
@ -1428,9 +1428,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->usum += plist[i]; pStdRes->usum += plist[i];
pStddevRes->quadraticUSum += plist[i] * plist[i]; pStdRes->quadraticUSum += plist[i] * plist[i];
} }
break; break;
} }
@ -1443,9 +1443,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->usum += plist[i]; pStdRes->usum += plist[i];
pStddevRes->quadraticUSum += plist[i] * plist[i]; pStdRes->quadraticUSum += plist[i] * plist[i];
} }
break; break;
@ -1459,9 +1459,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->usum += plist[i]; pStdRes->usum += plist[i];
pStddevRes->quadraticUSum += plist[i] * plist[i]; pStdRes->quadraticUSum += plist[i] * plist[i];
} }
break; break;
} }
@ -1474,9 +1474,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->dsum += plist[i]; pStdRes->dsum += plist[i];
pStddevRes->quadraticDSum += plist[i] * plist[i]; pStdRes->quadraticDSum += plist[i] * plist[i];
} }
break; break;
} }
@ -1489,9 +1489,9 @@ int32_t stddevFunction(SqlFunctionCtx* pCtx) {
} }
numOfElem += 1; numOfElem += 1;
pStddevRes->count += 1; pStdRes->count += 1;
pStddevRes->dsum += plist[i]; pStdRes->dsum += plist[i];
pStddevRes->quadraticDSum += plist[i] * plist[i]; pStdRes->quadraticDSum += plist[i] * plist[i];
} }
break; break;
} }
@ -1506,7 +1506,7 @@ _stddev_over:
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static void stddevTransferInfo(SStddevRes* pInput, SStddevRes* pOutput) { static void stdTransferInfo(SStdRes* pInput, SStdRes* pOutput) {
if (IS_NULL_TYPE(pInput->type)) { if (IS_NULL_TYPE(pInput->type)) {
return; return;
} }
@ -1525,7 +1525,7 @@ static void stddevTransferInfo(SStddevRes* pInput, SStddevRes* pOutput) {
pOutput->count += pInput->count; pOutput->count += pInput->count;
} }
int32_t stddevFunctionMerge(SqlFunctionCtx* pCtx) { int32_t stdFunctionMerge(SqlFunctionCtx* pCtx) {
SInputColumnInfoData* pInput = &pCtx->input; SInputColumnInfoData* pInput = &pCtx->input;
SColumnInfoData* pCol = pInput->pData[0]; SColumnInfoData* pCol = pInput->pData[0];
@ -1538,13 +1538,13 @@ int32_t stddevFunctionMerge(SqlFunctionCtx* pCtx) {
return TSDB_CODE_FUNC_FUNTION_PARA_TYPE; return TSDB_CODE_FUNC_FUNTION_PARA_TYPE;
} }
SStddevRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); SStdRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
for (int32_t i = pInput->startRowIndex; i < pInput->startRowIndex + pInput->numOfRows; ++i) { for (int32_t i = pInput->startRowIndex; i < pInput->startRowIndex + pInput->numOfRows; ++i) {
if (colDataIsNull_s(pCol, i)) continue; if (colDataIsNull_s(pCol, i)) continue;
char* data = colDataGetData(pCol, i); char* data = colDataGetData(pCol, i);
SStddevRes* pInputInfo = (SStddevRes*)varDataVal(data); SStdRes* pInputInfo = (SStdRes*)varDataVal(data);
stddevTransferInfo(pInputInfo, pInfo); stdTransferInfo(pInputInfo, pInfo);
} }
SET_VAL(GET_RES_INFO(pCtx), 1, 1); SET_VAL(GET_RES_INFO(pCtx), 1, 1);
@ -1552,14 +1552,14 @@ int32_t stddevFunctionMerge(SqlFunctionCtx* pCtx) {
} }
#ifdef BUILD_NO_CALL #ifdef BUILD_NO_CALL
int32_t stddevInvertFunction(SqlFunctionCtx* pCtx) { int32_t stdInvertFunction(SqlFunctionCtx* pCtx) {
int32_t numOfElem = 0; int32_t numOfElem = 0;
// Only the pre-computing information loaded and actual data does not loaded // Only the pre-computing information loaded and actual data does not loaded
SInputColumnInfoData* pInput = &pCtx->input; SInputColumnInfoData* pInput = &pCtx->input;
int32_t type = pInput->pData[0]->info.type; int32_t type = pInput->pData[0]->info.type;
SStddevRes* pStddevRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); SStdRes* pStdRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
// computing based on the true data block // computing based on the true data block
SColumnInfoData* pCol = pInput->pData[0]; SColumnInfoData* pCol = pInput->pData[0];
@ -1569,43 +1569,43 @@ int32_t stddevInvertFunction(SqlFunctionCtx* pCtx) {
switch (type) { switch (type) {
case TSDB_DATA_TYPE_TINYINT: { case TSDB_DATA_TYPE_TINYINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, int8_t); LIST_STDDEV_SUB_N(pStdRes->isum, int8_t);
break; break;
} }
case TSDB_DATA_TYPE_SMALLINT: { case TSDB_DATA_TYPE_SMALLINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, int16_t); LIST_STDDEV_SUB_N(pStdRes->isum, int16_t);
break; break;
} }
case TSDB_DATA_TYPE_INT: { case TSDB_DATA_TYPE_INT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, int32_t); LIST_STDDEV_SUB_N(pStdRes->isum, int32_t);
break; break;
} }
case TSDB_DATA_TYPE_BIGINT: { case TSDB_DATA_TYPE_BIGINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, int64_t); LIST_STDDEV_SUB_N(pStdRes->isum, int64_t);
break; break;
} }
case TSDB_DATA_TYPE_UTINYINT: { case TSDB_DATA_TYPE_UTINYINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, uint8_t); LIST_STDDEV_SUB_N(pStdRes->isum, uint8_t);
break; break;
} }
case TSDB_DATA_TYPE_USMALLINT: { case TSDB_DATA_TYPE_USMALLINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, uint16_t); LIST_STDDEV_SUB_N(pStdRes->isum, uint16_t);
break; break;
} }
case TSDB_DATA_TYPE_UINT: { case TSDB_DATA_TYPE_UINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, uint32_t); LIST_STDDEV_SUB_N(pStdRes->isum, uint32_t);
break; break;
} }
case TSDB_DATA_TYPE_UBIGINT: { case TSDB_DATA_TYPE_UBIGINT: {
LIST_STDDEV_SUB_N(pStddevRes->isum, uint64_t); LIST_STDDEV_SUB_N(pStdRes->isum, uint64_t);
break; break;
} }
case TSDB_DATA_TYPE_FLOAT: { case TSDB_DATA_TYPE_FLOAT: {
LIST_STDDEV_SUB_N(pStddevRes->dsum, float); LIST_STDDEV_SUB_N(pStdRes->dsum, float);
break; break;
} }
case TSDB_DATA_TYPE_DOUBLE: { case TSDB_DATA_TYPE_DOUBLE: {
LIST_STDDEV_SUB_N(pStddevRes->dsum, double); LIST_STDDEV_SUB_N(pStdRes->dsum, double);
break; break;
} }
default: default:
@ -1620,7 +1620,7 @@ int32_t stddevInvertFunction(SqlFunctionCtx* pCtx) {
int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
SInputColumnInfoData* pInput = &pCtx->input; SInputColumnInfoData* pInput = &pCtx->input;
SStddevRes* pStddevRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); SStdRes* pStddevRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
int32_t type = pStddevRes->type; int32_t type = pStddevRes->type;
double avg; double avg;
@ -1648,10 +1648,40 @@ int32_t stddevFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
return functionFinalize(pCtx, pBlock); return functionFinalize(pCtx, pBlock);
} }
int32_t stddevPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { int32_t stdvarFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
SInputColumnInfoData* pInput = &pCtx->input;
SStdRes* pStdvarRes = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
int32_t type = pStdvarRes->type;
double avg;
if (pStdvarRes->count == 0) {
GET_RES_INFO(pCtx)->numOfRes = 0;
return functionFinalize(pCtx, pBlock);
}
if (IS_SIGNED_NUMERIC_TYPE(type)) {
avg = pStdvarRes->isum / ((double)pStdvarRes->count);
pStdvarRes->result = fabs(pStdvarRes->quadraticISum / ((double)pStdvarRes->count) - avg * avg);
} else if (IS_UNSIGNED_NUMERIC_TYPE(type)) {
avg = pStdvarRes->usum / ((double)pStdvarRes->count);
pStdvarRes->result = fabs(pStdvarRes->quadraticUSum / ((double)pStdvarRes->count) - avg * avg);
} else {
avg = pStdvarRes->dsum / ((double)pStdvarRes->count);
pStdvarRes->result = fabs(pStdvarRes->quadraticDSum / ((double)pStdvarRes->count) - avg * avg);
}
// check for overflow
if (isinf(pStdvarRes->result) || isnan(pStdvarRes->result)) {
GET_RES_INFO(pCtx)->numOfRes = 0;
}
return functionFinalize(pCtx, pBlock);
}
int32_t stdPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx); SResultRowEntryInfo* pResInfo = GET_RES_INFO(pCtx);
SStddevRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx)); SStdRes* pInfo = GET_ROWCELL_INTERBUF(GET_RES_INFO(pCtx));
int32_t resultBytes = getStddevInfoSize(); int32_t resultBytes = getStdInfoSize();
char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char)); char* res = taosMemoryCalloc(resultBytes + VARSTR_HEADER_SIZE, sizeof(char));
if (NULL == res) { if (NULL == res) {
@ -1673,15 +1703,15 @@ int32_t stddevPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
return code; return code;
} }
int32_t stddevCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) { int32_t stdCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) {
SResultRowEntryInfo* pDResInfo = GET_RES_INFO(pDestCtx); SResultRowEntryInfo* pDResInfo = GET_RES_INFO(pDestCtx);
SStddevRes* pDBuf = GET_ROWCELL_INTERBUF(pDResInfo); SStdRes* pDBuf = GET_ROWCELL_INTERBUF(pDResInfo);
SResultRowEntryInfo* pSResInfo = GET_RES_INFO(pSourceCtx); SResultRowEntryInfo* pSResInfo = GET_RES_INFO(pSourceCtx);
SStddevRes* pSBuf = GET_ROWCELL_INTERBUF(pSResInfo); SStdRes* pSBuf = GET_ROWCELL_INTERBUF(pSResInfo);
int16_t type = pDBuf->type == TSDB_DATA_TYPE_NULL ? pSBuf->type : pDBuf->type; int16_t type = pDBuf->type == TSDB_DATA_TYPE_NULL ? pSBuf->type : pDBuf->type;
stddevTransferInfo(pSBuf, pDBuf); stdTransferInfo(pSBuf, pDBuf);
pDResInfo->numOfRes = TMAX(pDResInfo->numOfRes, pSResInfo->numOfRes); pDResInfo->numOfRes = TMAX(pDResInfo->numOfRes, pSResInfo->numOfRes);
pDResInfo->isNullRes &= pSResInfo->isNullRes; pDResInfo->isNullRes &= pSResInfo->isNullRes;
@ -4346,7 +4376,7 @@ int32_t spreadPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) {
_exit: _exit:
taosMemoryFree(res); taosMemoryFree(res);
return TSDB_CODE_SUCCESS; return code;
} }
int32_t spreadCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) { int32_t spreadCombine(SqlFunctionCtx* pDestCtx, SqlFunctionCtx* pSourceCtx) {

View File

@ -882,11 +882,10 @@ static int32_t sifExecLogic(SLogicConditionNode *node, SIFCtx *ctx, SIFParam *ou
if (ctx->noExec == false) { if (ctx->noExec == false) {
for (int32_t m = 0; m < node->pParameterList->length; m++) { for (int32_t m = 0; m < node->pParameterList->length; m++) {
if (node->condType == LOGIC_COND_TYPE_AND) { if (node->condType == LOGIC_COND_TYPE_AND) {
(void)taosArrayAddAll(output->result, params[m].result); if (taosArrayAddAll(output->result, params[m].result) == NULL) return terrno;
} else if (node->condType == LOGIC_COND_TYPE_OR) { } else if (node->condType == LOGIC_COND_TYPE_OR) {
(void)taosArrayAddAll(output->result, params[m].result); if (taosArrayAddAll(output->result, params[m].result) == NULL) return terrno;
} else if (node->condType == LOGIC_COND_TYPE_NOT) { } else if (node->condType == LOGIC_COND_TYPE_NOT) {
// taosArrayAddAll(output->result, params[m].result);
} }
taosArraySort(output->result, uidCompare); taosArraySort(output->result, uidCompare);
taosArrayRemoveDuplicate(output->result, uidCompare, NULL); taosArrayRemoveDuplicate(output->result, uidCompare, NULL);
@ -894,8 +893,6 @@ static int32_t sifExecLogic(SLogicConditionNode *node, SIFCtx *ctx, SIFParam *ou
} else { } else {
for (int32_t m = 0; m < node->pParameterList->length; m++) { for (int32_t m = 0; m < node->pParameterList->length; m++) {
output->status = sifMergeCond(node->condType, output->status, params[m].status); output->status = sifMergeCond(node->condType, output->status, params[m].status);
// taosArrayDestroy(params[m].result);
// params[m].result = NULL;
} }
} }
_return: _return:
@ -904,10 +901,13 @@ _return:
} }
static EDealRes sifWalkFunction(SNode *pNode, void *context) { static EDealRes sifWalkFunction(SNode *pNode, void *context) {
SIFCtx *ctx = context;
SFunctionNode *node = (SFunctionNode *)pNode; SFunctionNode *node = (SFunctionNode *)pNode;
SIFParam output = {.result = taosArrayInit(8, sizeof(uint64_t)), .status = SFLT_COARSE_INDEX}; SIFParam output = {.result = taosArrayInit(8, sizeof(uint64_t)), .status = SFLT_COARSE_INDEX};
if (output.result == NULL) {
SIFCtx *ctx = context; ctx->code = TSDB_CODE_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
}
ctx->code = sifExecFunction(node, ctx, &output); ctx->code = sifExecFunction(node, ctx, &output);
if (ctx->code != TSDB_CODE_SUCCESS) { if (ctx->code != TSDB_CODE_SUCCESS) {
sifFreeParam(&output); sifFreeParam(&output);
@ -921,11 +921,15 @@ static EDealRes sifWalkFunction(SNode *pNode, void *context) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
static EDealRes sifWalkLogic(SNode *pNode, void *context) { static EDealRes sifWalkLogic(SNode *pNode, void *context) {
SIFCtx *ctx = context;
SLogicConditionNode *node = (SLogicConditionNode *)pNode; SLogicConditionNode *node = (SLogicConditionNode *)pNode;
SIFParam output = {.result = taosArrayInit(8, sizeof(uint64_t)), .status = SFLT_COARSE_INDEX}; SIFParam output = {.result = taosArrayInit(8, sizeof(uint64_t)), .status = SFLT_COARSE_INDEX};
if (output.result == NULL) {
ctx->code = TSDB_CODE_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
}
SIFCtx *ctx = context;
ctx->code = sifExecLogic(node, ctx, &output); ctx->code = sifExecLogic(node, ctx, &output);
if (ctx->code) { if (ctx->code) {
sifFreeParam(&output); sifFreeParam(&output);
@ -939,10 +943,14 @@ static EDealRes sifWalkLogic(SNode *pNode, void *context) {
return DEAL_RES_CONTINUE; return DEAL_RES_CONTINUE;
} }
static EDealRes sifWalkOper(SNode *pNode, void *context) { static EDealRes sifWalkOper(SNode *pNode, void *context) {
SIFCtx *ctx = context;
SOperatorNode *node = (SOperatorNode *)pNode; SOperatorNode *node = (SOperatorNode *)pNode;
SIFParam output = {.result = taosArrayInit(8, sizeof(uint64_t)), .status = SFLT_COARSE_INDEX}; SIFParam output = {.result = taosArrayInit(8, sizeof(uint64_t)), .status = SFLT_COARSE_INDEX};
if (output.result == NULL) {
ctx->code = TSDB_CODE_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
}
SIFCtx *ctx = context;
ctx->code = sifExecOper(node, ctx, &output); ctx->code = sifExecOper(node, ctx, &output);
if (ctx->code) { if (ctx->code) {
sifFreeParam(&output); sifFreeParam(&output);
@ -1018,7 +1026,9 @@ static int32_t sifCalculate(SNode *pNode, SIFParam *pDst) {
SIF_ERR_RET(TSDB_CODE_APP_ERROR); SIF_ERR_RET(TSDB_CODE_APP_ERROR);
} }
if (res->result != NULL) { if (res->result != NULL) {
(void)taosArrayAddAll(pDst->result, res->result); if (taosArrayAddAll(pDst->result, res->result) == NULL) {
SIF_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
}
} }
pDst->status = res->status; pDst->status = res->status;
@ -1077,7 +1087,11 @@ int32_t doFilterTag(SNode *pFilterNode, SIndexMetaArg *metaArg, SArray *result,
SFilterInfo *filter = NULL; SFilterInfo *filter = NULL;
SArray *output = taosArrayInit(8, sizeof(uint64_t)); SArray *output = taosArrayInit(8, sizeof(uint64_t));
if (output == NULL) {
return TSDB_CODE_OUT_OF_MEMORY;
}
SIFParam param = {.arg = *metaArg, .result = output, .status = SFLT_NOT_INDEX, .api = *pAPI}; SIFParam param = {.arg = *metaArg, .result = output, .status = SFLT_NOT_INDEX, .api = *pAPI};
int32_t code = sifCalculate((SNode *)pFilterNode, &param); int32_t code = sifCalculate((SNode *)pFilterNode, &param);
if (code != 0) { if (code != 0) {
@ -1090,7 +1104,12 @@ int32_t doFilterTag(SNode *pFilterNode, SIndexMetaArg *metaArg, SArray *result,
*status = st; *status = st;
} }
(void)taosArrayAddAll(result, param.result);
if (taosArrayAddAll(result, param.result) == NULL) {
sifFreeParam(&param);
return TSDB_CODE_OUT_OF_MEMORY;
}
sifFreeParam(&param); sifFreeParam(&param);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }

View File

@ -4370,6 +4370,7 @@ static const char* jkFunctionHasPk = "HasPk";
static const char* jkFunctionPkBytes = "PkBytes"; static const char* jkFunctionPkBytes = "PkBytes";
static const char* jkFunctionIsMergeFunc = "IsMergeFunc"; static const char* jkFunctionIsMergeFunc = "IsMergeFunc";
static const char* jkFunctionMergeFuncOf = "MergeFuncOf"; static const char* jkFunctionMergeFuncOf = "MergeFuncOf";
static const char* jkFunctionTrimType = "TrimType";
static int32_t functionNodeToJson(const void* pObj, SJson* pJson) { static int32_t functionNodeToJson(const void* pObj, SJson* pJson) {
const SFunctionNode* pNode = (const SFunctionNode*)pObj; const SFunctionNode* pNode = (const SFunctionNode*)pObj;
@ -4402,7 +4403,9 @@ static int32_t functionNodeToJson(const void* pObj, SJson* pJson) {
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkFunctionMergeFuncOf, pNode->originalFuncId); code = tjsonAddIntegerToObject(pJson, jkFunctionMergeFuncOf, pNode->originalFuncId);
} }
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkFunctionTrimType, pNode->trimType);
}
return code; return code;
} }
@ -4437,6 +4440,9 @@ static int32_t jsonToFunctionNode(const SJson* pJson, void* pObj) {
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetIntValue(pJson, jkFunctionMergeFuncOf, &pNode->originalFuncId); code = tjsonGetIntValue(pJson, jkFunctionMergeFuncOf, &pNode->originalFuncId);
} }
if (TSDB_CODE_SUCCESS == code) {
tjsonGetNumberValue(pJson, jkFunctionTrimType, pNode->trimType, code);
}
return code; return code;
} }

View File

@ -1112,6 +1112,7 @@ enum {
FUNCTION_NODE_PK_BYTES, FUNCTION_NODE_PK_BYTES,
FUNCTION_CODE_IS_MERGE_FUNC, FUNCTION_CODE_IS_MERGE_FUNC,
FUNCTION_CODE_MERGE_FUNC_OF, FUNCTION_CODE_MERGE_FUNC_OF,
FUNCTION_CODE_TRIM_TYPE,
}; };
static int32_t functionNodeToMsg(const void* pObj, STlvEncoder* pEncoder) { static int32_t functionNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
@ -1145,6 +1146,9 @@ static int32_t functionNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
if (TSDB_CODE_SUCCESS == code) { if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeI32(pEncoder, FUNCTION_CODE_MERGE_FUNC_OF, pNode->originalFuncId); code = tlvEncodeI32(pEncoder, FUNCTION_CODE_MERGE_FUNC_OF, pNode->originalFuncId);
} }
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeEnum(pEncoder, FUNCTION_CODE_TRIM_TYPE, pNode->trimType);
}
return code; return code;
} }
@ -1186,6 +1190,9 @@ static int32_t msgToFunctionNode(STlvDecoder* pDecoder, void* pObj) {
case FUNCTION_CODE_MERGE_FUNC_OF: case FUNCTION_CODE_MERGE_FUNC_OF:
code = tlvDecodeI32(pTlv, &pNode->originalFuncId); code = tlvDecodeI32(pTlv, &pNode->originalFuncId);
break; break;
case FUNCTION_CODE_TRIM_TYPE:
code = tlvDecodeEnum(pTlv, &pNode->trimType, sizeof(pNode->trimType));
break;
default: default:
break; break;
} }

View File

@ -136,6 +136,11 @@ SNode* createBetweenAnd(SAstCreateContext* pCxt, SNode* pExpr, SNode* pLeft,
SNode* createNotBetweenAnd(SAstCreateContext* pCxt, SNode* pExpr, SNode* pLeft, SNode* pRight); SNode* createNotBetweenAnd(SAstCreateContext* pCxt, SNode* pExpr, SNode* pLeft, SNode* pRight);
SNode* createFunctionNode(SAstCreateContext* pCxt, const SToken* pFuncName, SNodeList* pParameterList); SNode* createFunctionNode(SAstCreateContext* pCxt, const SToken* pFuncName, SNodeList* pParameterList);
SNode* createCastFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SDataType dt); SNode* createCastFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SDataType dt);
SNode* createPositionFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2);
SNode* createTrimFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, ETrimType type);
SNode* createTrimFunctionNodeExt(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2, ETrimType type);
SNode* createSubstrFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2);
SNode* createSubstrFunctionNodeExt(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2, SNode* pExpr3);
SNode* createNodeListNode(SAstCreateContext* pCxt, SNodeList* pList); SNode* createNodeListNode(SAstCreateContext* pCxt, SNodeList* pList);
SNode* createNodeListNodeEx(SAstCreateContext* pCxt, SNode* p1, SNode* p2); SNode* createNodeListNodeEx(SAstCreateContext* pCxt, SNode* p1, SNode* p2);
SNode* createRealTableNode(SAstCreateContext* pCxt, SToken* pDbName, SToken* pTableName, SToken* pTableAlias); SNode* createRealTableNode(SAstCreateContext* pCxt, SToken* pDbName, SToken* pTableName, SToken* pTableAlias);

View File

@ -1184,19 +1184,45 @@ pseudo_column(A) ::= IROWTS(B).
pseudo_column(A) ::= ISFILLED(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); } pseudo_column(A) ::= ISFILLED(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); }
pseudo_column(A) ::= QTAGS(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); } pseudo_column(A) ::= QTAGS(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); }
function_expression(A) ::= function_name(B) NK_LP expression_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); } function_expression(A) ::= function_name(B) NK_LP expression_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); }
function_expression(A) ::= star_func(B) NK_LP star_func_para_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); } function_expression(A) ::= star_func(B) NK_LP star_func_para_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); }
function_expression(A) ::= function_expression(A) ::=
CAST(B) NK_LP expr_or_subquery(C) AS type_name(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, C), D)); } CAST(B) NK_LP expr_or_subquery(C) AS type_name(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, C), D)); }
function_expression(A) ::= function_expression(A) ::=
CAST(B) NK_LP expr_or_subquery(C) AS type_name_default_len(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, C), D)); } CAST(B) NK_LP expr_or_subquery(C) AS type_name_default_len(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, C), D)); }
function_expression(A) ::=
function_expression(A) ::= literal_func(B). { A = B; } POSITION(B) NK_LP expr_or_subquery(C) IN expr_or_subquery(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createPositionFunctionNode(pCxt, releaseRawExprNode(pCxt, C), releaseRawExprNode(pCxt, D))); }
function_expression(A) ::=
TRIM(B) NK_LP expr_or_subquery(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createTrimFunctionNode(pCxt, releaseRawExprNode(pCxt, C), TRIM_TYPE_BOTH)); }
function_expression(A) ::=
TRIM(B) NK_LP trim_specification_type(C) FROM expr_or_subquery(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createTrimFunctionNode(pCxt, releaseRawExprNode(pCxt, D), C)); }
function_expression(A) ::=
TRIM(B) NK_LP trim_specification_type(C) expr_or_subquery(D) FROM expr_or_subquery(E) NK_RP(F). { A = createRawExprNodeExt(pCxt, &B, &F, createTrimFunctionNodeExt(pCxt, releaseRawExprNode(pCxt, D), releaseRawExprNode(pCxt, E), C)); }
function_expression(A) ::=
substr_func(B) NK_LP expression_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); }
function_expression(A) ::=
substr_func(B) NK_LP expr_or_subquery(C) FROM expr_or_subquery(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createSubstrFunctionNode(pCxt, releaseRawExprNode(pCxt, C), releaseRawExprNode(pCxt, D))); }
function_expression(A) ::=
substr_func(B) NK_LP expr_or_subquery(C) FROM expr_or_subquery(D) FOR expr_or_subquery(E) NK_RP(F). { A = createRawExprNodeExt(pCxt, &B, &F, createSubstrFunctionNodeExt(pCxt, releaseRawExprNode(pCxt, C), releaseRawExprNode(pCxt, D), releaseRawExprNode(pCxt, E))); }
function_expression(A) ::= REPLACE(B) NK_LP expression_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); }
function_expression(A) ::= literal_func(B). { A = B; }
literal_func(A) ::= noarg_func(B) NK_LP NK_RP(C). { A = createRawExprNodeExt(pCxt, &B, &C, createFunctionNode(pCxt, &B, NULL)); } literal_func(A) ::= noarg_func(B) NK_LP NK_RP(C). { A = createRawExprNodeExt(pCxt, &B, &C, createFunctionNode(pCxt, &B, NULL)); }
literal_func(A) ::= NOW(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); } literal_func(A) ::= NOW(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); }
literal_func(A) ::= TODAY(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); } literal_func(A) ::= TODAY(B). { A = createRawExprNode(pCxt, &B, createFunctionNode(pCxt, &B, NULL)); }
%type substr_func { SToken }
%destructor substr_func { }
substr_func(A) ::= SUBSTR(B). { A = B; }
substr_func(A) ::= SUBSTRING(B). { A = B; }
%type trim_specification_type ETrimType
%destructor trim_specification_type { }
trim_specification_type(A) ::= . { A = TRIM_TYPE_BOTH; }
trim_specification_type(A) ::= BOTH. { A = TRIM_TYPE_BOTH; }
trim_specification_type(A) ::= TRAILING. { A = TRIM_TYPE_TRAILING; }
trim_specification_type(A) ::= LEADING. { A = TRIM_TYPE_LEADING; }
%type noarg_func { SToken } %type noarg_func { SToken }
%destructor noarg_func { } %destructor noarg_func { }
noarg_func(A) ::= NOW(B). { A = B; } noarg_func(A) ::= NOW(B). { A = B; }
@ -1208,6 +1234,7 @@ noarg_func(A) ::= SERVER_VERSION(B).
noarg_func(A) ::= SERVER_STATUS(B). { A = B; } noarg_func(A) ::= SERVER_STATUS(B). { A = B; }
noarg_func(A) ::= CURRENT_USER(B). { A = B; } noarg_func(A) ::= CURRENT_USER(B). { A = B; }
noarg_func(A) ::= USER(B). { A = B; } noarg_func(A) ::= USER(B). { A = B; }
noarg_func(A) ::= PI(B). { A = B; }
%type star_func { SToken } %type star_func { SToken }
%destructor star_func { } %destructor star_func { }

View File

@ -997,6 +997,73 @@ SNode* createCastFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SDataType d
return (SNode*)func; return (SNode*)func;
} }
SNode* createPositionFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2) {
CHECK_PARSER_STATUS(pCxt);
SFunctionNode* func = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "position");
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)func;
}
SNode* createTrimFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, ETrimType type) {
CHECK_PARSER_STATUS(pCxt);
SFunctionNode* func = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "trim");
func->trimType = type;
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)func;
}
SNode* createTrimFunctionNodeExt(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2, ETrimType type) {
CHECK_PARSER_STATUS(pCxt);
SFunctionNode* func = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "trim");
func->trimType = type;
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)func;
}
SNode* createSubstrFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2) {
CHECK_PARSER_STATUS(pCxt);
SFunctionNode* func = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "substr");
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)func;
}
SNode* createSubstrFunctionNodeExt(SAstCreateContext* pCxt, SNode* pExpr, SNode* pExpr2, SNode* pExpr3) {
CHECK_PARSER_STATUS(pCxt);
SFunctionNode* func = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "substr");
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr3);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)func;
}
SNode* createNodeListNode(SAstCreateContext* pCxt, SNodeList* pList) { SNode* createNodeListNode(SAstCreateContext* pCxt, SNodeList* pList) {
CHECK_PARSER_STATUS(pCxt); CHECK_PARSER_STATUS(pCxt);
SNodeListNode* list = NULL; SNodeListNode* list = NULL;

View File

@ -52,6 +52,7 @@ static SKeyword keywordTable[] = {
{"BNODE", TK_BNODE}, {"BNODE", TK_BNODE},
{"BNODES", TK_BNODES}, {"BNODES", TK_BNODES},
{"BOOL", TK_BOOL}, {"BOOL", TK_BOOL},
{"BOTH", TK_BOTH},
{"BUFFER", TK_BUFFER}, {"BUFFER", TK_BUFFER},
{"BUFSIZE", TK_BUFSIZE}, {"BUFSIZE", TK_BUFSIZE},
{"BY", TK_BY}, {"BY", TK_BY},
@ -111,6 +112,7 @@ static SKeyword keywordTable[] = {
{"FLOAT", TK_FLOAT}, {"FLOAT", TK_FLOAT},
{"FLUSH", TK_FLUSH}, {"FLUSH", TK_FLUSH},
{"FROM", TK_FROM}, {"FROM", TK_FROM},
{"FOR", TK_FOR},
{"FORCE", TK_FORCE}, {"FORCE", TK_FORCE},
{"FULL", TK_FULL}, {"FULL", TK_FULL},
{"FUNCTION", TK_FUNCTION}, {"FUNCTION", TK_FUNCTION},
@ -148,6 +150,7 @@ static SKeyword keywordTable[] = {
{"LAST", TK_LAST}, {"LAST", TK_LAST},
{"LAST_ROW", TK_LAST_ROW}, {"LAST_ROW", TK_LAST_ROW},
{"LEADER", TK_LEADER}, {"LEADER", TK_LEADER},
{"LEADING", TK_LEADING},
{"LEFT", TK_LEFT}, {"LEFT", TK_LEFT},
{"LICENCES", TK_LICENCES}, {"LICENCES", TK_LICENCES},
{"LIKE", TK_LIKE}, {"LIKE", TK_LIKE},
@ -191,6 +194,7 @@ static SKeyword keywordTable[] = {
{"PARTITION_FIRST", TK_PARTITION_FIRST}, {"PARTITION_FIRST", TK_PARTITION_FIRST},
{"PASS", TK_PASS}, {"PASS", TK_PASS},
{"PORT", TK_PORT}, {"PORT", TK_PORT},
{"POSITION", TK_POSITION},
{"PPS", TK_PPS}, {"PPS", TK_PPS},
{"PRIMARY", TK_PRIMARY}, {"PRIMARY", TK_PRIMARY},
{"PRECISION", TK_PRECISION}, {"PRECISION", TK_PRECISION},
@ -201,6 +205,7 @@ static SKeyword keywordTable[] = {
{"QTIME", TK_QTIME}, {"QTIME", TK_QTIME},
{"QUERIES", TK_QUERIES}, {"QUERIES", TK_QUERIES},
{"QUERY", TK_QUERY}, {"QUERY", TK_QUERY},
{"PI", TK_PI},
{"RANGE", TK_RANGE}, {"RANGE", TK_RANGE},
{"RATIO", TK_RATIO}, {"RATIO", TK_RATIO},
{"PAUSE", TK_PAUSE}, {"PAUSE", TK_PAUSE},
@ -250,6 +255,8 @@ static SKeyword keywordTable[] = {
{"STT_TRIGGER", TK_STT_TRIGGER}, {"STT_TRIGGER", TK_STT_TRIGGER},
{"SUBSCRIBE", TK_SUBSCRIBE}, {"SUBSCRIBE", TK_SUBSCRIBE},
{"SUBSCRIPTIONS", TK_SUBSCRIPTIONS}, {"SUBSCRIPTIONS", TK_SUBSCRIPTIONS},
{"SUBSTR", TK_SUBSTR},
{"SUBSTRING", TK_SUBSTRING},
{"SUBTABLE", TK_SUBTABLE}, {"SUBTABLE", TK_SUBTABLE},
{"SYSINFO", TK_SYSINFO}, {"SYSINFO", TK_SYSINFO},
{"SYSTEM", TK_SYSTEM}, {"SYSTEM", TK_SYSTEM},
@ -268,6 +275,7 @@ static SKeyword keywordTable[] = {
{"TODAY", TK_TODAY}, {"TODAY", TK_TODAY},
{"TOPIC", TK_TOPIC}, {"TOPIC", TK_TOPIC},
{"TOPICS", TK_TOPICS}, {"TOPICS", TK_TOPICS},
{"TRAILING", TK_TRAILING},
{"TRANSACTION", TK_TRANSACTION}, {"TRANSACTION", TK_TRANSACTION},
{"TRANSACTIONS", TK_TRANSACTIONS}, {"TRANSACTIONS", TK_TRANSACTIONS},
{"TRIGGER", TK_TRIGGER}, {"TRIGGER", TK_TRIGGER},

View File

@ -13251,6 +13251,10 @@ static int32_t serializeVgroupCreateTableBatch(SVgroupCreateTableBatch* pTbBatch
if (TSDB_CODE_SUCCESS != ret) { if (TSDB_CODE_SUCCESS != ret) {
return ret; return ret;
} }
if (tlen >= TSDB_MAX_MSG_SIZE - sizeof(SMsgHead)) {
return TSDB_CODE_INVALID_MSG_LEN;
}
tlen += sizeof(SMsgHead); tlen += sizeof(SMsgHead);
void* buf = taosMemoryMalloc(tlen); void* buf = taosMemoryMalloc(tlen);
if (NULL == buf) { if (NULL == buf) {
@ -14088,6 +14092,7 @@ int32_t serializeVgroupsCreateTableBatch(SHashObj* pVgroupHashmap, SArray** pOut
code = serializeVgroupCreateTableBatch(pTbBatch, pBufArray); code = serializeVgroupCreateTableBatch(pTbBatch, pBufArray);
if (TSDB_CODE_SUCCESS != code) { if (TSDB_CODE_SUCCESS != code) {
qError("failed to serialize create table batch msg, since:%s", tstrerror(code));
taosHashCancelIterate(pVgroupHashmap, pTbBatch); taosHashCancelIterate(pVgroupHashmap, pTbBatch);
break; break;
} }
@ -14165,6 +14170,9 @@ static int32_t rewriteCreateTableFromFile(STranslateContext* pCxt, SQuery* pQuer
code = serializeVgroupsCreateTableBatch(pModifyStmt->pVgroupsHashObj, &pBufArray); code = serializeVgroupsCreateTableBatch(pModifyStmt->pVgroupsHashObj, &pBufArray);
taosHashClear(pModifyStmt->pVgroupsHashObj); taosHashClear(pModifyStmt->pVgroupsHashObj);
if (TSDB_CODE_SUCCESS != code) { if (TSDB_CODE_SUCCESS != code) {
if (TSDB_CODE_INVALID_MSG_LEN == code) {
qError("maxInsertBatchRows may need to be reduced, current:%d", tsMaxInsertBatchRows);
}
taosHashCleanup(pModifyStmt->pVgroupsHashObj); taosHashCleanup(pModifyStmt->pVgroupsHashObj);
return code; return code;
} }

File diff suppressed because it is too large Load Diff

View File

@ -778,11 +778,13 @@ int32_t qwProcessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg, char *sql) {
sql = NULL; sql = NULL;
if (code) { if (code) {
QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code)); QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code));
qDestroyTask(pTaskInfo);
QW_ERR_JRET(code); QW_ERR_JRET(code);
} }
if (NULL == sinkHandle || NULL == pTaskInfo) { if (NULL == sinkHandle || NULL == pTaskInfo) {
QW_TASK_ELOG("create task result error, taskHandle:%p, sinkHandle:%p", pTaskInfo, sinkHandle); QW_TASK_ELOG("create task result error, taskHandle:%p, sinkHandle:%p", pTaskInfo, sinkHandle);
qDestroyTask(pTaskInfo);
QW_ERR_JRET(TSDB_CODE_APP_ERROR); QW_ERR_JRET(TSDB_CODE_APP_ERROR);
} }
@ -1277,11 +1279,13 @@ int32_t qwProcessDelete(QW_FPARAMS_DEF, SQWMsg *qwMsg, SDeleteRes *pRes) {
if (code) { if (code) {
QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code)); QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code));
qDestroyTask(pTaskInfo);
QW_ERR_JRET(code); QW_ERR_JRET(code);
} }
if (NULL == sinkHandle || NULL == pTaskInfo) { if (NULL == sinkHandle || NULL == pTaskInfo) {
QW_TASK_ELOG("create task result error, taskHandle:%p, sinkHandle:%p", pTaskInfo, sinkHandle); QW_TASK_ELOG("create task result error, taskHandle:%p, sinkHandle:%p", pTaskInfo, sinkHandle);
qDestroyTask(pTaskInfo);
QW_ERR_JRET(TSDB_CODE_APP_ERROR); QW_ERR_JRET(TSDB_CODE_APP_ERROR);
} }

View File

@ -54,6 +54,52 @@ typedef struct SScalarCtx {
#define SCL_DOWNGRADE_DATETYPE(_type) \ #define SCL_DOWNGRADE_DATETYPE(_type) \
((_type) == TSDB_DATA_TYPE_BIGINT || TSDB_DATA_TYPE_DOUBLE == (_type) || (_type) == TSDB_DATA_TYPE_UBIGINT) ((_type) == TSDB_DATA_TYPE_BIGINT || TSDB_DATA_TYPE_DOUBLE == (_type) || (_type) == TSDB_DATA_TYPE_UBIGINT)
/** Flags for calculateWeekNum() function. */
#define WEEK_FLAG_MONDAY_FIRST 1 /* If set this, monday is first day of week, else, sunday is first day.*/
#define WEEK_FLAG_FROM_ONE 2 /* If set this, week start from 1, else, week start from 0. */
#define WEEK_FLAG_INCLUDE_FIRST_DAY 4 /* If set this, week contains 'first day of week' is week one,
* else, weeks are numbered according to ISO 8601:1988. */
#ifndef M_PI
#define M_PI 3.14159265358979323846264338327950288 /* pi */
#endif
#ifndef M_1_PI
#define M_1_PI 0.318309886183790671537767526745028724 /* 1/pi */
#endif
#define INT1TOCHAR(T, A) \
{ \
const unsigned long def_temp = (unsigned long)(A); \
((unsigned char *)(T))[0] = (unsigned char)(def_temp); \
T += 1; \
}
#define INT2TOCHAR(T, A) \
{ \
const unsigned long def_temp = (unsigned long)(A); \
((unsigned char *)(T))[1] = (unsigned char)(def_temp); \
((unsigned char *)(T))[0] = (unsigned char)(def_temp >> 8); \
T += 2; \
}
#define INT3TOCHAR(T, A) \
{ \
const unsigned long def_temp = (unsigned long)(A); \
((unsigned char *)(T))[2] = (unsigned char)(def_temp); \
((unsigned char *)(T))[1] = (unsigned char)(def_temp >> 8); \
((unsigned char *)(T))[0] = (unsigned char)(def_temp >> 16); \
T += 3; \
}
#define INT4TOCHAR(T, A) \
{ \
const unsigned long def_temp = (unsigned long)(A); \
((unsigned char *)(T))[3] = (unsigned char)(def_temp); \
((unsigned char *)(T))[2] = (unsigned char)(def_temp >> 8); \
((unsigned char *)(T))[1] = (unsigned char)(def_temp >> 16); \
((unsigned char *)(T))[0] = (unsigned char)(def_temp >> 24); \
T += 4; \
}
#define sclFatal(...) qFatal(__VA_ARGS__) #define sclFatal(...) qFatal(__VA_ARGS__)
#define sclError(...) qError(__VA_ARGS__) #define sclError(...) qError(__VA_ARGS__)
#define sclWarn(...) qWarn(__VA_ARGS__) #define sclWarn(...) qWarn(__VA_ARGS__)

View File

@ -1066,7 +1066,9 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
info->fields[type].fields = info->fields[type].fields =
taosMemoryRealloc(info->fields[type].fields, info->fields[type].size * sizeof(SFilterField)); taosMemoryRealloc(info->fields[type].fields, info->fields[type].size * sizeof(SFilterField));
if (info->fields[type].fields == NULL) { if (info->fields[type].fields == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; *num = 0;
fltError("taosMemoryRealloc failed, size:%d", (int32_t)(info->fields[type].size * sizeof(SFilterField)));
FLT_ERR_RET(terrno);
} }
} }
@ -1086,14 +1088,20 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, false); taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, false);
if (NULL == info->pctx.valHash) { if (NULL == info->pctx.valHash) {
fltError("taosHashInit failed, size:%d", FILTER_DEFAULT_GROUP_SIZE * FILTER_DEFAULT_VALUE_SIZE); fltError("taosHashInit failed, size:%d", FILTER_DEFAULT_GROUP_SIZE * FILTER_DEFAULT_VALUE_SIZE);
FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); if (srcFlag) {
FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE);
}
FLT_ERR_RET(terrno);
} }
} }
SFilterDataInfo dInfo = {idx, *data}; SFilterDataInfo dInfo = {idx, *data};
if (taosHashPut(info->pctx.valHash, *data, dataLen, &dInfo, sizeof(dInfo))) { if (taosHashPut(info->pctx.valHash, *data, dataLen, &dInfo, sizeof(dInfo))) {
fltError("taosHashPut to set failed"); fltError("taosHashPut to set failed");
FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); if (srcFlag) {
FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE);
}
FLT_ERR_RET(terrno);
} }
if (srcFlag) { if (srcFlag) {
FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE); FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE);
@ -1102,7 +1110,7 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
} else if (type != FLD_TYPE_COLUMN && data) { } else if (type != FLD_TYPE_COLUMN && data) {
if (freeIfExists) { if (freeIfExists) {
taosMemoryFreeClear(*data); taosMemoryFreeClear(*data);
} else if (sameBuf) { } else if (sameBuf && srcFlag) {
FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE); FILTER_SET_FLAG(*srcFlag, FLD_DATA_NO_FREE);
} }
} }
@ -1114,11 +1122,11 @@ int32_t filterAddField(SFilterInfo *info, void *desc, void **data, int32_t type,
} }
static FORCE_INLINE int32_t filterAddColFieldFromField(SFilterInfo *info, SFilterField *field, SFilterFieldId *fid) { static FORCE_INLINE int32_t filterAddColFieldFromField(SFilterInfo *info, SFilterField *field, SFilterFieldId *fid) {
FLT_ERR_RET(filterAddField(info, field->desc, &field->data, FILTER_GET_TYPE(field->flag), fid, 0, false, NULL)); int32_t code = filterAddField(info, field->desc, &field->data, FILTER_GET_TYPE(field->flag), fid, 0, false, NULL);
FILTER_SET_FLAG(field->flag, FLD_DATA_NO_FREE); FILTER_SET_FLAG(field->flag, FLD_DATA_NO_FREE);
return TSDB_CODE_SUCCESS; return code;
} }
int32_t filterAddFieldFromNode(SFilterInfo *info, SNode *node, SFilterFieldId *fid) { int32_t filterAddFieldFromNode(SFilterInfo *info, SNode *node, SFilterFieldId *fid) {
@ -2769,6 +2777,7 @@ int32_t filterConvertGroupFromArray(SFilterInfo *info, SArray *group) {
if (info->groupNum > 0) { if (info->groupNum > 0) {
info->groups = taosMemoryCalloc(info->groupNum, sizeof(*info->groups)); info->groups = taosMemoryCalloc(info->groupNum, sizeof(*info->groups));
if (info->groups == NULL) { if (info->groups == NULL) {
info->groupNum = 0;
FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
} }
@ -2780,6 +2789,7 @@ int32_t filterConvertGroupFromArray(SFilterInfo *info, SArray *group) {
} }
pg->unitFlags = taosMemoryCalloc(pg->unitNum, sizeof(*pg->unitFlags)); pg->unitFlags = taosMemoryCalloc(pg->unitNum, sizeof(*pg->unitFlags));
if (pg->unitFlags == NULL) { if (pg->unitFlags == NULL) {
pg->unitNum = 0;
FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); FLT_ERR_RET(TSDB_CODE_OUT_OF_MEMORY);
} }
info->groups[i] = *pg; info->groups[i] = *pg;
@ -2798,6 +2808,8 @@ int32_t filterRewrite(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t gResNum
FILTER_SET_FLAG(oinfo.status, FI_STATUS_CLONED); FILTER_SET_FLAG(oinfo.status, FI_STATUS_CLONED);
(void)memset(info, 0, sizeof(*info));
SFilterGroupCtx *res = NULL; SFilterGroupCtx *res = NULL;
SFilterColInfo *colInfo = NULL; SFilterColInfo *colInfo = NULL;
int32_t optr = 0; int32_t optr = 0;
@ -2808,8 +2820,6 @@ int32_t filterRewrite(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t gResNum
FLT_ERR_JRET(terrno); FLT_ERR_JRET(terrno);
} }
(void)memset(info, 0, sizeof(*info));
info->colRangeNum = oinfo.colRangeNum; info->colRangeNum = oinfo.colRangeNum;
info->colRange = oinfo.colRange; info->colRange = oinfo.colRange;
oinfo.colRangeNum = 0; oinfo.colRangeNum = 0;
@ -2855,10 +2865,9 @@ int32_t filterRewrite(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t gResNum
} }
} }
} }
FLT_ERR_JRET(filterConvertGroupFromArray(info, group));
_return: _return:
FLT_ERR_RET(filterConvertGroupFromArray(info, group));
taosArrayDestroy(group); taosArrayDestroy(group);
filterFreeInfo(&oinfo); filterFreeInfo(&oinfo);

File diff suppressed because it is too large Load Diff

View File

@ -128,6 +128,28 @@ int32_t streamTaskProcessCheckpointTriggerRsp(SStreamTask* pTask, SCheckpointTri
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
streamMutexLock(&pTask->lock);
SStreamTaskState status = streamTaskGetStatus(pTask);
if (status.state != TASK_STATUS__CK) {
stError("s-task:%s status:%s not in checkpoint status, discard the checkpoint-trigger msg", pTask->id.idStr, status.name);
streamMutexUnlock(&pTask->lock);
return TSDB_CODE_STREAM_TASK_IVLD_STATUS;
}
streamMutexUnlock(&pTask->lock);
SActiveCheckpointInfo* pInfo = pTask->chkInfo.pActiveInfo;
streamMutexLock(&pInfo->lock);
if (pInfo->activeId != pRsp->checkpointId || pInfo->transId != pRsp->transId) {
stError("s-task:%s status:%s not in checkpoint status, discard the checkpoint-trigger msg", pTask->id.idStr, status.name);
streamMutexUnlock(&pInfo->lock);
return TSDB_CODE_STREAM_TASK_IVLD_STATUS;
}
streamMutexUnlock(&pInfo->lock);
// NOTE: here we do not do the duplicated checkpoint-trigger msg check, since it will be done by following functions.
(void)appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT_TRIGGER, pRsp->checkpointId, pRsp->transId, (void)appendCheckpointIntoInputQ(pTask, STREAM_INPUT__CHECKPOINT_TRIGGER, pRsp->checkpointId, pRsp->transId,
pRsp->upstreamTaskId); pRsp->upstreamTaskId);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -136,10 +158,8 @@ int32_t streamTaskProcessCheckpointTriggerRsp(SStreamTask* pTask, SCheckpointTri
int32_t streamTaskSendCheckpointTriggerMsg(SStreamTask* pTask, int32_t dstTaskId, int32_t downstreamNodeId, int32_t streamTaskSendCheckpointTriggerMsg(SStreamTask* pTask, int32_t dstTaskId, int32_t downstreamNodeId,
SRpcHandleInfo* pRpcInfo, int32_t code) { SRpcHandleInfo* pRpcInfo, int32_t code) {
int32_t size = sizeof(SMsgHead) + sizeof(SCheckpointTriggerRsp); int32_t size = sizeof(SMsgHead) + sizeof(SCheckpointTriggerRsp);
void* pBuf = rpcMallocCont(size);
void* pBuf = rpcMallocCont(size);
if (pBuf == NULL) { if (pBuf == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
return terrno; return terrno;
} }
@ -1016,7 +1036,6 @@ bool streamTaskAlreadySendTrigger(SStreamTask* pTask, int32_t downstreamNodeId)
const char* id = pTask->id.idStr; const char* id = pTask->id.idStr;
SActiveCheckpointInfo* pInfo = pTask->chkInfo.pActiveInfo; SActiveCheckpointInfo* pInfo = pTask->chkInfo.pActiveInfo;
SStreamTaskState pStatus = streamTaskGetStatus(pTask); SStreamTaskState pStatus = streamTaskGetStatus(pTask);
bool alreadySend = false;
if (pStatus.state != TASK_STATUS__CK) { if (pStatus.state != TASK_STATUS__CK) {
return false; return false;
@ -1126,7 +1145,11 @@ int32_t streamTaskGetNumOfConfirmed(SActiveCheckpointInfo* pInfo) {
void streamTaskSetTriggerDispatchConfirmed(SStreamTask* pTask, int32_t vgId) { void streamTaskSetTriggerDispatchConfirmed(SStreamTask* pTask, int32_t vgId) {
SActiveCheckpointInfo* pInfo = pTask->chkInfo.pActiveInfo; SActiveCheckpointInfo* pInfo = pTask->chkInfo.pActiveInfo;
int64_t now = taosGetTimestampMs();
int32_t taskId = 0; int32_t taskId = 0;
int32_t total = streamTaskGetNumOfDownstream(pTask);
bool alreadyRecv = false;
streamMutexLock(&pInfo->lock); streamMutexLock(&pInfo->lock);
for (int32_t i = 0; i < taosArrayGetSize(pInfo->pDispatchTriggerList); ++i) { for (int32_t i = 0; i < taosArrayGetSize(pInfo->pDispatchTriggerList); ++i) {
@ -1136,11 +1159,16 @@ void streamTaskSetTriggerDispatchConfirmed(SStreamTask* pTask, int32_t vgId) {
} }
if (p->nodeId == vgId) { if (p->nodeId == vgId) {
ASSERT(p->recved == false); if (p->recved) {
stWarn("s-task:%s already recv checkpoint-trigger msg rsp from vgId:%d down:0x%x %.2fs ago, req send:%" PRId64
p->recved = true; " discard",
p->recvTs = taosGetTimestampMs(); pTask->id.idStr, vgId, p->taskId, (now - p->recvTs) / 1000.0, p->sendTs);
taskId = p->taskId; alreadyRecv = true;
} else {
p->recved = true;
p->recvTs = taosGetTimestampMs();
taskId = p->taskId;
}
break; break;
} }
} }
@ -1148,12 +1176,13 @@ void streamTaskSetTriggerDispatchConfirmed(SStreamTask* pTask, int32_t vgId) {
int32_t numOfConfirmed = streamTaskGetNumOfConfirmed(pInfo); int32_t numOfConfirmed = streamTaskGetNumOfConfirmed(pInfo);
streamMutexUnlock(&pInfo->lock); streamMutexUnlock(&pInfo->lock);
int32_t total = streamTaskGetNumOfDownstream(pTask);
if (taskId == 0) { if (taskId == 0) {
stError("s-task:%s recv invalid trigger-dispatch confirm, vgId:%d", pTask->id.idStr, vgId); stError("s-task:%s recv invalid trigger-dispatch confirm, vgId:%d", pTask->id.idStr, vgId);
} else { } else {
stDebug("s-task:%s set downstream:0x%x(vgId:%d) checkpoint-trigger dispatch confirmed, total confirmed:%d/%d", if (!alreadyRecv) {
pTask->id.idStr, taskId, vgId, numOfConfirmed, total); stDebug("s-task:%s set downstream:0x%x(vgId:%d) checkpoint-trigger dispatch confirmed, total confirmed:%d/%d",
pTask->id.idStr, taskId, vgId, numOfConfirmed, total);
}
} }
} }

View File

@ -1388,7 +1388,8 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
// we only set the dispatch msg info for current checkpoint trans // we only set the dispatch msg info for current checkpoint trans
streamMutexLock(&pTask->lock); streamMutexLock(&pTask->lock);
triggerDispatchRsp = (streamTaskGetStatus(pTask).state == TASK_STATUS__CK) && triggerDispatchRsp = (streamTaskGetStatus(pTask).state == TASK_STATUS__CK) &&
(pTask->chkInfo.pActiveInfo->activeId == pMsgInfo->checkpointId); (pTask->chkInfo.pActiveInfo->activeId == pMsgInfo->checkpointId) &&
(pTask->chkInfo.pActiveInfo->transId != pMsgInfo->transId);
streamMutexUnlock(&pTask->lock); streamMutexUnlock(&pTask->lock);
streamMutexLock(&pMsgInfo->lock); streamMutexLock(&pMsgInfo->lock);
@ -1449,7 +1450,6 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
if (delayDispatch) { if (delayDispatch) {
// we only set the dispatch msg info for current checkpoint trans // we only set the dispatch msg info for current checkpoint trans
if (triggerDispatchRsp) { if (triggerDispatchRsp) {
ASSERT(pTask->chkInfo.pActiveInfo->transId == pMsgInfo->transId);
stDebug("s-task:%s checkpoint-trigger msg to 0x%x rsp for checkpointId:%" PRId64 " transId:%d confirmed", stDebug("s-task:%s checkpoint-trigger msg to 0x%x rsp for checkpointId:%" PRId64 " transId:%d confirmed",
pTask->id.idStr, pRsp->downstreamTaskId, pMsgInfo->checkpointId, pMsgInfo->transId); pTask->id.idStr, pRsp->downstreamTaskId, pMsgInfo->checkpointId, pMsgInfo->transId);

View File

@ -1589,11 +1589,33 @@ static void displayStatusInfo(SStreamMeta* pMeta, SHashObj* pTaskSet, bool succ)
} }
} }
// check all existed tasks are received rsp
static bool allCheckDownstreamRsp(SStreamMeta* pMeta, STaskStartInfo* pStartInfo, int32_t numOfTotal) {
for (int32_t i = 0; i < numOfTotal; ++i) {
SStreamTaskId* pTaskId = taosArrayGet(pMeta->pTaskList, i);
if (pTaskId == NULL) {
continue;
}
STaskId idx = {.streamId = pTaskId->streamId, .taskId = pTaskId->taskId};
void* px = taosHashGet(pStartInfo->pReadyTaskSet, &idx, sizeof(idx));
if (px == NULL) {
px = taosHashGet(pStartInfo->pFailedTaskSet, &idx, sizeof(idx));
if (px == NULL) {
return false;
}
}
}
return true;
}
int32_t streamMetaAddTaskLaunchResult(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, int64_t startTs, int32_t streamMetaAddTaskLaunchResult(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, int64_t startTs,
int64_t endTs, bool ready) { int64_t endTs, bool ready) {
STaskStartInfo* pStartInfo = &pMeta->startInfo; STaskStartInfo* pStartInfo = &pMeta->startInfo;
STaskId id = {.streamId = streamId, .taskId = taskId}; STaskId id = {.streamId = streamId, .taskId = taskId};
int32_t vgId = pMeta->vgId; int32_t vgId = pMeta->vgId;
bool allRsp = true;
streamMetaWLock(pMeta); streamMetaWLock(pMeta);
SStreamTask** p = taosHashGet(pMeta->pTasksMap, &id, sizeof(id)); SStreamTask** p = taosHashGet(pMeta->pTasksMap, &id, sizeof(id));
@ -1618,9 +1640,8 @@ int32_t streamMetaAddTaskLaunchResult(SStreamMeta* pMeta, int64_t streamId, int3
return 0; return 0;
} }
SHashObj* pDst = ready ? pStartInfo->pReadyTaskSet : pStartInfo->pFailedTaskSet;
STaskInitTs initTs = {.start = startTs, .end = endTs, .success = ready}; STaskInitTs initTs = {.start = startTs, .end = endTs, .success = ready};
SHashObj* pDst = ready ? pStartInfo->pReadyTaskSet : pStartInfo->pFailedTaskSet;
int32_t code = taosHashPut(pDst, &id, sizeof(id), &initTs, sizeof(STaskInitTs)); int32_t code = taosHashPut(pDst, &id, sizeof(id), &initTs, sizeof(STaskInitTs));
if (code) { if (code) {
if (code == TSDB_CODE_DUP_KEY) { if (code == TSDB_CODE_DUP_KEY) {
@ -1637,7 +1658,8 @@ int32_t streamMetaAddTaskLaunchResult(SStreamMeta* pMeta, int64_t streamId, int3
int32_t numOfTotal = streamMetaGetNumOfTasks(pMeta); int32_t numOfTotal = streamMetaGetNumOfTasks(pMeta);
int32_t numOfRecv = taosHashGetSize(pStartInfo->pReadyTaskSet) + taosHashGetSize(pStartInfo->pFailedTaskSet); int32_t numOfRecv = taosHashGetSize(pStartInfo->pReadyTaskSet) + taosHashGetSize(pStartInfo->pFailedTaskSet);
if (numOfRecv == numOfTotal) { allRsp = allCheckDownstreamRsp(pMeta, pStartInfo, numOfTotal);
if (allRsp) {
pStartInfo->readyTs = taosGetTimestampMs(); pStartInfo->readyTs = taosGetTimestampMs();
pStartInfo->elapsedTime = (pStartInfo->startTs != 0) ? pStartInfo->readyTs - pStartInfo->startTs : 0; pStartInfo->elapsedTime = (pStartInfo->startTs != 0) ? pStartInfo->readyTs - pStartInfo->startTs : 0;

View File

@ -221,12 +221,12 @@ int tdbTbcOpen(TTB *pTb, TBC **ppTbc, TXN *pTxn) {
*ppTbc = NULL; *ppTbc = NULL;
pTbc = (TBC *)tdbOsMalloc(sizeof(*pTbc)); pTbc = (TBC *)tdbOsMalloc(sizeof(*pTbc));
if (pTbc == NULL) { if (pTbc == NULL) {
return -1; return TSDB_CODE_OUT_OF_MEMORY;
} }
if (tdbBtcOpen(&pTbc->btc, pTb->pBt, pTxn)) { if ((ret = tdbBtcOpen(&pTbc->btc, pTb->pBt, pTxn)) != 0) {
taosMemoryFree(pTbc); taosMemoryFree(pTbc);
return -1; return ret;
} }
*ppTbc = pTbc; *ppTbc = pTbc;

View File

@ -190,6 +190,7 @@ static void cliDestroyConnMsgs(SCliConn* conn, bool destroy);
static void doFreeTimeoutMsg(void* param); static void doFreeTimeoutMsg(void* param);
static int32_t cliPreCheckSessionLimitForMsg(SCliThrd* pThrd, char* addr, SCliMsg** pMsg); static int32_t cliPreCheckSessionLimitForMsg(SCliThrd* pThrd, char* addr, SCliMsg** pMsg);
static void cliDestroyBatch(SCliBatch* pBatch);
// cli util func // cli util func
static FORCE_INLINE bool cliIsEpsetUpdated(int32_t code, STransConnCtx* pCtx); static FORCE_INLINE bool cliIsEpsetUpdated(int32_t code, STransConnCtx* pCtx);
static FORCE_INLINE void cliMayCvtFqdnToIp(SEpSet* pEpSet, SCvtAddr* pCvtAddr); static FORCE_INLINE void cliMayCvtFqdnToIp(SEpSet* pEpSet, SCvtAddr* pCvtAddr);
@ -662,7 +663,7 @@ static SCliConn* getConnFromPool(SCliThrd* pThrd, char* key, bool* exceed) {
if (QUEUE_IS_EMPTY(&plist->conns)) { if (QUEUE_IS_EMPTY(&plist->conns)) {
if (plist->list->numOfConn >= pTranInst->connLimitNum) { if (plist->list->numOfConn >= pTranInst->connLimitNum) {
*exceed = true; *exceed = true;
return NULL;; return NULL;
} }
plist->list->numOfConn++; plist->list->numOfConn++;
return NULL; return NULL;
@ -708,7 +709,8 @@ static SCliConn* getConnFromPool2(SCliThrd* pThrd, char* key, SCliMsg** pMsg) {
SMsgList* list = plist->list; SMsgList* list = plist->list;
if ((list)->numOfConn >= pTransInst->connLimitNum) { if ((list)->numOfConn >= pTransInst->connLimitNum) {
STraceId* trace = &(*pMsg)->msg.info.traceId; STraceId* trace = &(*pMsg)->msg.info.traceId;
if (pTransInst->notWaitAvaliableConn || (pTransInst->noDelayFp != NULL && pTransInst->noDelayFp((*pMsg)->msg.msgType))) { if (pTransInst->notWaitAvaliableConn ||
(pTransInst->noDelayFp != NULL && pTransInst->noDelayFp((*pMsg)->msg.msgType))) {
tDebug("%s msg %s not to send, reason: %s", pTransInst->label, TMSG_INFO((*pMsg)->msg.msgType), tDebug("%s msg %s not to send, reason: %s", pTransInst->label, TMSG_INFO((*pMsg)->msg.msgType),
tstrerror(TSDB_CODE_RPC_NETWORK_BUSY)); tstrerror(TSDB_CODE_RPC_NETWORK_BUSY));
doNotifyApp(*pMsg, pThrd, TSDB_CODE_RPC_NETWORK_BUSY); doNotifyApp(*pMsg, pThrd, TSDB_CODE_RPC_NETWORK_BUSY);
@ -903,11 +905,10 @@ static int32_t specifyConnRef(SCliConn* conn, bool update, int64_t handle) {
exh->handle = conn; exh->handle = conn;
exh->pThrd = conn->hostThrd; exh->pThrd = conn->hostThrd;
taosWUnLockLatch(&exh->latch); taosWUnLockLatch(&exh->latch);
conn->refId = exh->refId;
taosWUnLockLatch(&exh->latch);
tDebug("conn %p specified by %"PRId64"", conn, handle); conn->refId = exh->refId;
tDebug("conn %p specified by %" PRId64 "", conn, handle);
(void)transReleaseExHandle(transGetRefMgt(), handle); (void)transReleaseExHandle(transGetRefMgt(), handle);
return 0; return 0;
@ -919,7 +920,6 @@ static void cliAllocRecvBufferCb(uv_handle_t* handle, size_t suggested_size, uv_
int32_t code = transAllocBuffer(pBuf, buf); int32_t code = transAllocBuffer(pBuf, buf);
if (code < 0) { if (code < 0) {
tError("conn %p failed to alloc buffer, since %s", conn, tstrerror(code)); tError("conn %p failed to alloc buffer, since %s", conn, tstrerror(code));
// cliDestroyConn(conn, true);
} }
} }
static void cliRecvCb(uv_stream_t* handle, ssize_t nread, const uv_buf_t* buf) { static void cliRecvCb(uv_stream_t* handle, ssize_t nread, const uv_buf_t* buf) {
@ -1081,9 +1081,8 @@ static void cliDestroy(uv_handle_t* handle) {
(void)atomic_sub_fetch_32(&pThrd->connCount, 1); (void)atomic_sub_fetch_32(&pThrd->connCount, 1);
if (conn->refId > 0) { if (conn->refId > 0) {
(void)transReleaseExHandle(transGetRefMgt(), conn->refId); (void)transReleaseExHandle(transGetRefMgt(), conn->refId);
(void)transRemoveExHandle(transGetRefMgt(), conn->refId); (void)transRemoveExHandle(transGetRefMgt(), conn->refId);
} }
taosMemoryFree(conn->dstAddr); taosMemoryFree(conn->dstAddr);
taosMemoryFree(conn->stream); taosMemoryFree(conn->stream);
@ -1149,6 +1148,7 @@ static void cliSendCb(uv_write_t* req, int status) {
(void)uv_read_start((uv_stream_t*)pConn->stream, cliAllocRecvBufferCb, cliRecvCb); (void)uv_read_start((uv_stream_t*)pConn->stream, cliAllocRecvBufferCb, cliRecvCb);
} }
void cliSendBatch(SCliConn* pConn) { void cliSendBatch(SCliConn* pConn) {
int32_t code = 0;
SCliThrd* pThrd = pConn->hostThrd; SCliThrd* pThrd = pConn->hostThrd;
STrans* pTransInst = pThrd->pTransInst; STrans* pTransInst = pThrd->pTransInst;
@ -1158,8 +1158,13 @@ void cliSendBatch(SCliConn* pConn) {
pBatch->pList->connCnt += 1; pBatch->pList->connCnt += 1;
uv_buf_t* wb = taosMemoryCalloc(wLen, sizeof(uv_buf_t)); uv_buf_t* wb = taosMemoryCalloc(wLen, sizeof(uv_buf_t));
int i = 0; if (wb == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
tError("%s conn %p failed to send batch msg since:%s", CONN_GET_INST_LABEL(pConn), pConn, tstrerror(code));
goto _exception;
}
int i = 0;
queue* h = NULL; queue* h = NULL;
QUEUE_FOREACH(h, &pBatch->wq) { QUEUE_FOREACH(h, &pBatch->wq) {
SCliMsg* pCliMsg = QUEUE_DATA(h, SCliMsg, q); SCliMsg* pCliMsg = QUEUE_DATA(h, SCliMsg, q);
@ -1169,6 +1174,11 @@ void cliSendBatch(SCliConn* pConn) {
STransMsg* pMsg = (STransMsg*)(&pCliMsg->msg); STransMsg* pMsg = (STransMsg*)(&pCliMsg->msg);
if (pMsg->pCont == 0) { if (pMsg->pCont == 0) {
pMsg->pCont = (void*)rpcMallocCont(0); pMsg->pCont = (void*)rpcMallocCont(0);
if (pMsg->pCont == NULL) {
code = TSDB_CODE_OUT_OF_BUFFER;
tError("%s conn %p failed to send batch msg since:%s", CONN_GET_INST_LABEL(pConn), pConn, tstrerror(code));
goto _exception;
}
pMsg->contLen = 0; pMsg->contLen = 0;
} }
@ -1202,11 +1212,29 @@ void cliSendBatch(SCliConn* pConn) {
} }
uv_write_t* req = taosMemoryCalloc(1, sizeof(uv_write_t)); uv_write_t* req = taosMemoryCalloc(1, sizeof(uv_write_t));
if (req == NULL) {
code = TSDB_CODE_OUT_OF_MEMORY;
tError("%s conn %p failed to send batch msg since:%s", CONN_GET_INST_LABEL(pConn), pConn, tstrerror(code));
goto _exception;
}
req->data = pConn; req->data = pConn;
tDebug("%s conn %p start to send batch msg, batch size:%d, msgLen:%d", CONN_GET_INST_LABEL(pConn), pConn, tDebug("%s conn %p start to send batch msg, batch size:%d, msgLen:%d", CONN_GET_INST_LABEL(pConn), pConn,
pBatch->wLen, pBatch->batchSize); pBatch->wLen, pBatch->batchSize);
(void)uv_write(req, (uv_stream_t*)pConn->stream, wb, wLen, cliSendBatchCb);
code = uv_write(req, (uv_stream_t*)pConn->stream, wb, wLen, cliSendBatchCb);
if (code != 0) {
tDebug("%s conn %p failed to to send batch msg since %s", CONN_GET_INST_LABEL(pConn), pConn, uv_err_name(code));
goto _exception;
}
taosMemoryFree(wb); taosMemoryFree(wb);
return;
_exception:
cliDestroyBatch(pBatch);
taosMemoryFree(wb);
pConn->pBatch = NULL;
return;
} }
void cliSend(SCliConn* pConn) { void cliSend(SCliConn* pConn) {
SCliThrd* pThrd = pConn->hostThrd; SCliThrd* pThrd = pConn->hostThrd;
@ -1627,7 +1655,7 @@ static void cliHandleFreeById(SCliMsg* pMsg, SCliThrd* pThrd) {
return; return;
} }
_exception: _exception:
tDebug("already free conn %p by id %" PRId64"", conn, refId); tDebug("already free conn %p by id %" PRId64 "", conn, refId);
(void)transReleaseExHandle(transGetRefMgt(), refId); (void)transReleaseExHandle(transGetRefMgt(), refId);
(void)transReleaseExHandle(transGetRefMgt(), refId); (void)transReleaseExHandle(transGetRefMgt(), refId);
@ -1921,8 +1949,70 @@ SCliBatch* cliGetHeadFromList(SCliBatchList* pList) {
return batch; return batch;
} }
static int32_t createBatchList(SCliBatchList** ppBatchList, char* key, char* ip, uint32_t port) {
SCliBatchList* pBatchList = taosMemoryCalloc(1, sizeof(SCliBatchList));
if (pBatchList == NULL) {
tError("failed to create batch list, reason:%s", tstrerror(TSDB_CODE_OUT_OF_MEMORY));
return TSDB_CODE_OUT_OF_MEMORY;
}
QUEUE_INIT(&pBatchList->wq);
pBatchList->port = port;
pBatchList->connMax = 1;
pBatchList->connCnt = 0;
pBatchList->batchLenLimit = 0;
pBatchList->len += 1;
pBatchList->ip = taosStrdup(ip);
pBatchList->dst = taosStrdup(key);
if (pBatchList->ip == NULL || pBatchList->dst == NULL) {
taosMemoryFree(pBatchList->ip);
taosMemoryFree(pBatchList->dst);
taosMemoryFree(pBatchList);
tError("failed to create batch list, reason:%s", tstrerror(TSDB_CODE_OUT_OF_MEMORY));
return TSDB_CODE_OUT_OF_MEMORY;
}
*ppBatchList = pBatchList;
return 0;
}
static void destroyBatchList(SCliBatchList* pList) {
if (pList == NULL) {
return;
}
while (!QUEUE_IS_EMPTY(&pList->wq)) {
queue* h = QUEUE_HEAD(&pList->wq);
QUEUE_REMOVE(h);
SCliBatch* pBatch = QUEUE_DATA(h, SCliBatch, listq);
cliDestroyBatch(pBatch);
}
taosMemoryFree(pList->ip);
taosMemoryFree(pList->dst);
taosMemoryFree(pList);
}
static int32_t createBatch(SCliBatch** ppBatch, SCliBatchList* pList, SCliMsg* pMsg) {
SCliBatch* pBatch = taosMemoryCalloc(1, sizeof(SCliBatch));
if (pBatch == NULL) {
tError("failed to create batch, reason:%s", tstrerror(TSDB_CODE_OUT_OF_MEMORY));
return TSDB_CODE_OUT_OF_MEMORY;
}
QUEUE_INIT(&pBatch->wq);
QUEUE_INIT(&pBatch->listq);
QUEUE_PUSH(&pBatch->wq, &pMsg->q);
pBatch->wLen += 1;
pBatch->batchSize = pMsg->msg.contLen;
pBatch->pList = pList;
QUEUE_PUSH(&pList->wq, &pBatch->listq);
pList->len += 1;
*ppBatch = pBatch;
return 0;
}
static void cliBatchDealReq(queue* wq, SCliThrd* pThrd) { static void cliBatchDealReq(queue* wq, SCliThrd* pThrd) {
STrans* pInst = pThrd->pTransInst; STrans* pInst = pThrd->pTransInst;
int32_t code = 0;
int count = 0; int count = 0;
while (!QUEUE_IS_EMPTY(wq)) { while (!QUEUE_IS_EMPTY(wq)) {
@ -1946,64 +2036,48 @@ static void cliBatchDealReq(queue* wq, SCliThrd* pThrd) {
size_t klen = strlen(key); size_t klen = strlen(key);
SCliBatchList** ppBatchList = taosHashGet(pThrd->batchCache, key, klen); SCliBatchList** ppBatchList = taosHashGet(pThrd->batchCache, key, klen);
if (ppBatchList == NULL || *ppBatchList == NULL) { if (ppBatchList == NULL || *ppBatchList == NULL) {
SCliBatchList* pBatchList = taosMemoryCalloc(1, sizeof(SCliBatchList)); SCliBatchList* pBatchList = NULL;
QUEUE_INIT(&pBatchList->wq); code = createBatchList(&pBatchList, key, ip, port);
pBatchList->connMax = pInst->connLimitNum; if (code != 0) {
pBatchList->connCnt = 0; destroyCmsg(pMsg);
continue;
}
pBatchList->batchLenLimit = pInst->batchSize; pBatchList->batchLenLimit = pInst->batchSize;
pBatchList->len += 1;
pBatchList->ip = taosStrdup(ip);
pBatchList->dst = taosStrdup(key);
pBatchList->port = port;
SCliBatch* pBatch = taosMemoryCalloc(1, sizeof(SCliBatch));
QUEUE_INIT(&pBatch->wq);
QUEUE_INIT(&pBatch->listq);
QUEUE_PUSH(&pBatch->wq, h);
pBatch->wLen += 1;
pBatch->batchSize += pMsg->msg.contLen;
pBatch->pList = pBatchList;
QUEUE_PUSH(&pBatchList->wq, &pBatch->listq);
(void)taosHashPut(pThrd->batchCache, key, klen, &pBatchList, sizeof(void*));
} else {
if (QUEUE_IS_EMPTY(&(*ppBatchList)->wq)) {
SCliBatch* pBatch = taosMemoryCalloc(1, sizeof(SCliBatch));
QUEUE_INIT(&pBatch->wq);
QUEUE_INIT(&pBatch->listq);
QUEUE_PUSH(&pBatch->wq, h);
pBatch->wLen += 1;
pBatch->batchSize = pMsg->msg.contLen;
pBatch->pList = *ppBatchList;
QUEUE_PUSH(&((*ppBatchList)->wq), &pBatch->listq);
(*ppBatchList)->len += 1;
SCliBatch* pBatch = NULL;
code = createBatch(&pBatch, pBatchList, pMsg);
if (code != 0) {
destroyBatchList(pBatchList);
destroyCmsg(pMsg);
continue; continue;
} }
queue* hdr = QUEUE_TAIL(&((*ppBatchList)->wq)); code = taosHashPut(pThrd->batchCache, key, klen, &pBatchList, sizeof(void*));
SCliBatch* pBatch = QUEUE_DATA(hdr, SCliBatch, listq); if (code != 0) {
if ((pBatch->batchSize + pMsg->msg.contLen) < (*ppBatchList)->batchLenLimit) { destroyBatchList(pBatchList);
QUEUE_PUSH(&pBatch->wq, h); }
pBatch->batchSize += pMsg->msg.contLen; } else {
pBatch->wLen += 1; if (QUEUE_IS_EMPTY(&(*ppBatchList)->wq)) {
SCliBatch* pBatch = NULL;
code = createBatch(&pBatch, *ppBatchList, pMsg);
if (code != 0) {
destroyCmsg(pMsg);
cliDestroyBatch(pBatch);
}
} else { } else {
SCliBatch* pBatch = taosMemoryCalloc(1, sizeof(SCliBatch)); queue* hdr = QUEUE_TAIL(&((*ppBatchList)->wq));
QUEUE_INIT(&pBatch->wq); SCliBatch* pBatch = QUEUE_DATA(hdr, SCliBatch, listq);
QUEUE_INIT(&pBatch->listq); if ((pBatch->batchSize + pMsg->msg.contLen) < (*ppBatchList)->batchLenLimit) {
QUEUE_PUSH(&pBatch->wq, h);
QUEUE_PUSH(&pBatch->wq, h); pBatch->batchSize += pMsg->msg.contLen;
pBatch->wLen += 1; pBatch->wLen += 1;
pBatch->batchSize += pMsg->msg.contLen; } else {
pBatch->pList = *ppBatchList; SCliBatch* tBatch = NULL;
code = createBatch(&tBatch, *ppBatchList, pMsg);
QUEUE_PUSH(&((*ppBatchList)->wq), &pBatch->listq); if (code != 0) {
(*ppBatchList)->len += 1; destroyCmsg(pMsg);
}
}
} }
} }
continue; continue;
@ -3277,7 +3351,7 @@ int32_t transFreeConnById(void* shandle, int64_t transpointId) {
return TSDB_CODE_RPC_MODULE_QUIT; return TSDB_CODE_RPC_MODULE_QUIT;
} }
if (transpointId == 0) { if (transpointId == 0) {
tDebug("not free by refId:%"PRId64"", transpointId); tDebug("not free by refId:%" PRId64 "", transpointId);
TAOS_CHECK_GOTO(0, NULL, _exception); TAOS_CHECK_GOTO(0, NULL, _exception);
} }

View File

@ -32,7 +32,7 @@ int32_t transCompressMsg(char* msg, int32_t len) {
char* buf = taosMemoryMalloc(len + compHdr + 8); // 8 extra bytes char* buf = taosMemoryMalloc(len + compHdr + 8); // 8 extra bytes
if (buf == NULL) { if (buf == NULL) {
tError("failed to allocate memory for rpc msg compression, contLen:%d", len); tWarn("failed to allocate memory for rpc msg compression, contLen:%d", len);
ret = len; ret = len;
return ret; return ret;
} }
@ -206,6 +206,8 @@ int32_t transAllocBuffer(SConnBuffer* connBuf, uv_buf_t* uvBuf) {
p->cap = p->left + p->len; p->cap = p->left + p->len;
p->buf = taosMemoryRealloc(p->buf, p->cap); p->buf = taosMemoryRealloc(p->buf, p->cap);
if (p->buf == NULL) { if (p->buf == NULL) {
uvBuf->base = NULL;
uvBuf->len = 0;
return TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_OUT_OF_MEMORY;
} }
uvBuf->base = p->buf + p->len; uvBuf->base = p->buf + p->len;
@ -440,11 +442,11 @@ void transReqQueueClear(queue* q) {
int32_t transQueueInit(STransQueue* queue, void (*freeFunc)(const void* arg)) { int32_t transQueueInit(STransQueue* queue, void (*freeFunc)(const void* arg)) {
queue->q = taosArrayInit(2, sizeof(void*)); queue->q = taosArrayInit(2, sizeof(void*));
queue->freeFunc = (void (*)(const void*))freeFunc;
if (queue->q == NULL) { if (queue->q == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return TSDB_CODE_OUT_OF_MEMORY;
} }
queue->freeFunc = (void (*)(const void*))freeFunc;
return 0; return 0;
} }
bool transQueuePush(STransQueue* queue, void* arg) { bool transQueuePush(STransQueue* queue, void* arg) {

View File

@ -205,7 +205,6 @@ void uvAllocRecvBufferCb(uv_handle_t* handle, size_t suggested_size, uv_buf_t* b
int32_t code = transAllocBuffer(pBuf, buf); int32_t code = transAllocBuffer(pBuf, buf);
if (code < 0) { if (code < 0) {
tError("conn %p failed to alloc buffer, since %s", conn, tstrerror(code)); tError("conn %p failed to alloc buffer, since %s", conn, tstrerror(code));
// destroyConn(conn, true);
} }
} }
@ -542,6 +541,9 @@ void uvOnRecvCb(uv_stream_t* cli, ssize_t nread, const uv_buf_t* buf) {
void uvAllocConnBufferCb(uv_handle_t* handle, size_t suggested_size, uv_buf_t* buf) { void uvAllocConnBufferCb(uv_handle_t* handle, size_t suggested_size, uv_buf_t* buf) {
buf->len = 2; buf->len = 2;
buf->base = taosMemoryCalloc(1, sizeof(char) * buf->len); buf->base = taosMemoryCalloc(1, sizeof(char) * buf->len);
if (buf->base == NULL) {
tError("failed to alloc conn read buffer since %s", tstrerror(TSDB_CODE_OUT_OF_MEMORY));
}
} }
void uvOnTimeoutCb(uv_timer_t* handle) { void uvOnTimeoutCb(uv_timer_t* handle) {

View File

@ -818,7 +818,10 @@ int32_t walMetaDeserialize(SWal* pWal, const char* bytes) {
int sz = cJSON_GetArraySize(pFiles); int sz = cJSON_GetArraySize(pFiles);
// deserialize // deserialize
SArray* pArray = pWal->fileInfoSet; SArray* pArray = pWal->fileInfoSet;
(void)taosArrayEnsureCap(pArray, sz); if (taosArrayEnsureCap(pArray, sz)) {
cJSON_Delete(pRoot);
return terrno;
}
for (int i = 0; i < sz; i++) { for (int i = 0; i < sz; i++) {
pInfoJson = cJSON_GetArrayItem(pFiles, i); pInfoJson = cJSON_GetArrayItem(pFiles, i);
@ -841,7 +844,10 @@ int32_t walMetaDeserialize(SWal* pWal, const char* bytes) {
pField = cJSON_GetObjectItem(pInfoJson, "fileSize"); pField = cJSON_GetObjectItem(pInfoJson, "fileSize");
if (!pField) goto _err; if (!pField) goto _err;
info.fileSize = atoll(cJSON_GetStringValue(pField)); info.fileSize = atoll(cJSON_GetStringValue(pField));
(void)taosArrayPush(pArray, &info); if (!taosArrayPush(pArray, &info)) {
cJSON_Delete(pRoot);
return terrno;
}
} }
pWal->fileInfoSet = pArray; pWal->fileInfoSet = pArray;
pWal->writeCur = sz - 1; pWal->writeCur = sz - 1;
@ -860,8 +866,8 @@ static int walFindCurMetaVer(SWal* pWal) {
TdDirPtr pDir = taosOpenDir(pWal->path); TdDirPtr pDir = taosOpenDir(pWal->path);
if (pDir == NULL) { if (pDir == NULL) {
wError("vgId:%d, path:%s, failed to open since %s", pWal->cfg.vgId, pWal->path, strerror(errno)); wError("vgId:%d, path:%s, failed to open since %s", pWal->cfg.vgId, pWal->path, tstrerror(terrno));
return -1; return terrno;
} }
TdDirEntryPtr pDirEntry; TdDirEntryPtr pDirEntry;

View File

@ -64,6 +64,20 @@ typedef struct TdFile {
#define FILE_WITH_LOCK 1 #define FILE_WITH_LOCK 1
#ifdef BUILD_WITH_RAND_ERR
#define STUB_RAND_IO_ERR(ret) \
if (tsEnableRandErr) { \
uint32_t r = taosRand() % tsRandErrDivisor; \
if ((r + 1) <= tsRandErrChance) { \
errno = EIO; \
terrno = TAOS_SYSTEM_ERROR(errno); \
return (ret); \
} \
}
#else
#define STUB_RAND_IO_ERR(ret)
#endif
void taosGetTmpfilePath(const char *inputTmpDir, const char *fileNamePrefix, char *dstPath) { void taosGetTmpfilePath(const char *inputTmpDir, const char *fileNamePrefix, char *dstPath) {
#ifdef WINDOWS #ifdef WINDOWS
@ -145,7 +159,7 @@ int64_t taosCopyFile(const char *from, const char *to) {
code = terrno; code = terrno;
goto _err; goto _err;
} }
if (bytes == 0) break; if (bytes == 0) break;
size += bytes; size += bytes;
@ -166,7 +180,7 @@ int64_t taosCopyFile(const char *from, const char *to) {
terrno = code; terrno = code;
return -1; return -1;
} }
return size; return size;
_err: _err:
@ -197,12 +211,12 @@ TdFilePtr taosCreateFile(const char *path, int32_t tdFileOptions) {
} }
} }
} }
return fp; return fp;
} }
int32_t taosRemoveFile(const char *path) { int32_t taosRemoveFile(const char *path) {
int32_t code = remove(path); int32_t code = remove(path);
if (-1 == code) { if (-1 == code) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
return terrno; return terrno;
@ -300,7 +314,7 @@ int32_t taosDevInoFile(TdFilePtr pFile, int64_t *stDev, int64_t *stIno) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
struct stat fileStat; struct stat fileStat;
int32_t code = fstat(pFile->fd, &fileStat); int32_t code = fstat(pFile->fd, &fileStat);
if (-1 == code) { if (-1 == code) {
@ -335,7 +349,7 @@ FILE *taosOpenFileForStream(const char *path, int32_t tdFileOptions) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return NULL; return NULL;
} }
FILE* f = fopen(path, mode); FILE *f = fopen(path, mode);
if (NULL == f) { if (NULL == f) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
} }
@ -694,6 +708,7 @@ int taosOpenFileNotStream(const char *path, int32_t tdFileOptions) {
} }
int64_t taosReadFile(TdFilePtr pFile, void *buf, int64_t count) { int64_t taosReadFile(TdFilePtr pFile, void *buf, int64_t count) {
STUB_RAND_IO_ERR(terrno)
#if FILE_WITH_LOCK #if FILE_WITH_LOCK
(void)taosThreadRwlockRdlock(&(pFile->rwlock)); (void)taosThreadRwlockRdlock(&(pFile->rwlock));
#endif #endif
@ -705,7 +720,7 @@ int64_t taosReadFile(TdFilePtr pFile, void *buf, int64_t count) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
int64_t leftbytes = count; int64_t leftbytes = count;
int64_t readbytes; int64_t readbytes;
char *tbuf = (char *)buf; char *tbuf = (char *)buf;
@ -747,6 +762,7 @@ int64_t taosReadFile(TdFilePtr pFile, void *buf, int64_t count) {
} }
int64_t taosWriteFile(TdFilePtr pFile, const void *buf, int64_t count) { int64_t taosWriteFile(TdFilePtr pFile, const void *buf, int64_t count) {
STUB_RAND_IO_ERR(terrno)
if (pFile == NULL) { if (pFile == NULL) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return 0; return 0;
@ -792,6 +808,7 @@ int64_t taosWriteFile(TdFilePtr pFile, const void *buf, int64_t count) {
} }
int64_t taosPWriteFile(TdFilePtr pFile, const void *buf, int64_t count, int64_t offset) { int64_t taosPWriteFile(TdFilePtr pFile, const void *buf, int64_t count, int64_t offset) {
STUB_RAND_IO_ERR(terrno)
if (pFile == NULL) { if (pFile == NULL) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return 0; return 0;
@ -850,7 +867,7 @@ int64_t taosLSeekFile(TdFilePtr pFile, int64_t offset, int32_t whence) {
int32_t code = 0; int32_t code = 0;
ASSERT(pFile->fd >= 0); // Please check if you have closed the file. ASSERT(pFile->fd >= 0); // Please check if you have closed the file.
#ifdef WINDOWS #ifdef WINDOWS
int64_t ret = _lseeki64(pFile->fd, offset, whence); int64_t ret = _lseeki64(pFile->fd, offset, whence);
#else #else
@ -888,7 +905,7 @@ int32_t taosFStatFile(TdFilePtr pFile, int64_t *size, int32_t *mtime) {
int32_t code = _fstat64(pFile->fd, &fileStat); int32_t code = _fstat64(pFile->fd, &fileStat);
#else #else
struct stat fileStat; struct stat fileStat;
int32_t code = fstat(pFile->fd, &fileStat); int32_t code = fstat(pFile->fd, &fileStat);
#endif #endif
if (-1 == code) { if (-1 == code) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
@ -944,7 +961,7 @@ int32_t taosUnLockFile(TdFilePtr pFile) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
#ifdef WINDOWS #ifdef WINDOWS
BOOL fSuccess = FALSE; BOOL fSuccess = FALSE;
OVERLAPPED overlapped = {0}; OVERLAPPED overlapped = {0};
@ -970,7 +987,7 @@ int32_t taosFtruncateFile(TdFilePtr pFile, int64_t l_size) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
#ifdef WINDOWS #ifdef WINDOWS
HANDLE h = (HANDLE)_get_osfhandle(pFile->fd); HANDLE h = (HANDLE)_get_osfhandle(pFile->fd);
@ -1030,9 +1047,9 @@ int64_t taosFSendFile(TdFilePtr pFileOut, TdFilePtr pFileIn, int64_t *offset, in
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
if (pFileIn->fd < 0 || pFileOut->fd < 0) { if (pFileIn->fd < 0 || pFileOut->fd < 0) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
#ifdef WINDOWS #ifdef WINDOWS
@ -1131,6 +1148,7 @@ bool lastErrorIsFileNotExist() { return terrno == TAOS_SYSTEM_ERROR(ENOENT); }
#endif // WINDOWS #endif // WINDOWS
TdFilePtr taosOpenFile(const char *path, int32_t tdFileOptions) { TdFilePtr taosOpenFile(const char *path, int32_t tdFileOptions) {
STUB_RAND_IO_ERR(NULL)
FILE *fp = NULL; FILE *fp = NULL;
#ifdef WINDOWS #ifdef WINDOWS
HANDLE hFile = NULL; HANDLE hFile = NULL;
@ -1228,16 +1246,16 @@ int32_t taosCloseFile(TdFilePtr *ppFile) {
#endif #endif
taosMemoryFree(*ppFile); taosMemoryFree(*ppFile);
*ppFile = NULL; *ppFile = NULL;
return code; return code;
} }
int64_t taosPReadFile(TdFilePtr pFile, void *buf, int64_t count, int64_t offset) { int64_t taosPReadFile(TdFilePtr pFile, void *buf, int64_t count, int64_t offset) {
STUB_RAND_IO_ERR(terrno)
if (pFile == NULL) { if (pFile == NULL) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
int32_t code = 0; int32_t code = 0;
#ifdef WINDOWS #ifdef WINDOWS
@ -1249,11 +1267,11 @@ int64_t taosPReadFile(TdFilePtr pFile, void *buf, int64_t count, int64_t offset)
#if FILE_WITH_LOCK #if FILE_WITH_LOCK
(void)taosThreadRwlockUnlock(&(pFile->rwlock)); (void)taosThreadRwlockUnlock(&(pFile->rwlock));
#endif #endif
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
DWORD ret = 0; DWORD ret = 0;
OVERLAPPED ol = {0}; OVERLAPPED ol = {0};
ol.OffsetHigh = (uint32_t)((offset & 0xFFFFFFFF00000000LL) >> 0x20); ol.OffsetHigh = (uint32_t)((offset & 0xFFFFFFFF00000000LL) >> 0x20);
@ -1290,7 +1308,7 @@ int64_t taosPReadFile(TdFilePtr pFile, void *buf, int64_t count, int64_t offset)
terrno = code; terrno = code;
return code; return code;
} }
return ret; return ret;
} }
@ -1308,10 +1326,10 @@ int32_t taosFsyncFile(TdFilePtr pFile) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
return terrno; return terrno;
} }
return 0; return 0;
} }
#ifdef WINDOWS #ifdef WINDOWS
if (pFile->hFile != NULL) { if (pFile->hFile != NULL) {
if (pFile->tdFileOptions & TD_FILE_WRITE_THROUGH) { if (pFile->tdFileOptions & TD_FILE_WRITE_THROUGH) {
@ -1361,7 +1379,7 @@ int32_t taosUmaskFile(int32_t maskVal) {
int64_t taosGetLineFile(TdFilePtr pFile, char **__restrict ptrBuf) { int64_t taosGetLineFile(TdFilePtr pFile, char **__restrict ptrBuf) {
int64_t ret = -1; int64_t ret = -1;
int32_t code = 0; int32_t code = 0;
#if FILE_WITH_LOCK #if FILE_WITH_LOCK
(void)taosThreadRwlockRdlock(&(pFile->rwlock)); (void)taosThreadRwlockRdlock(&(pFile->rwlock));
#endif #endif
@ -1377,7 +1395,7 @@ int64_t taosGetLineFile(TdFilePtr pFile, char **__restrict ptrBuf) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
goto END; goto END;
} }
#ifdef WINDOWS #ifdef WINDOWS
size_t bufferSize = 512; size_t bufferSize = 512;
*ptrBuf = taosMemoryMalloc(bufferSize); *ptrBuf = taosMemoryMalloc(bufferSize);
@ -1437,12 +1455,12 @@ int64_t taosGetsFile(TdFilePtr pFile, int32_t maxSize, char *__restrict buf) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
if (pFile->fp == NULL) { if (pFile->fp == NULL) {
terrno = TSDB_CODE_INVALID_PARA; terrno = TSDB_CODE_INVALID_PARA;
return terrno; return terrno;
} }
if (fgets(buf, maxSize, pFile->fp) == NULL) { if (fgets(buf, maxSize, pFile->fp) == NULL) {
if (feof(pFile->fp)) { if (feof(pFile->fp)) {
return 0; return 0;
@ -1451,7 +1469,7 @@ int64_t taosGetsFile(TdFilePtr pFile, int32_t maxSize, char *__restrict buf) {
return terrno; return terrno;
} }
} }
return strlen(buf); return strlen(buf);
} }
@ -1493,17 +1511,17 @@ bool taosCheckAccessFile(const char *pathname, int32_t tdFileAccessOptions) {
bool taosCheckExistFile(const char *pathname) { return taosCheckAccessFile(pathname, TD_FILE_ACCESS_EXIST_OK); }; bool taosCheckExistFile(const char *pathname) { return taosCheckAccessFile(pathname, TD_FILE_ACCESS_EXIST_OK); };
int32_t taosCompressFile(char *srcFileName, char *destFileName) { int32_t taosCompressFile(char *srcFileName, char *destFileName) {
int32_t compressSize = 163840; int32_t compressSize = 163840;
int32_t ret = 0; int32_t ret = 0;
int32_t len = 0; int32_t len = 0;
gzFile dstFp = NULL; gzFile dstFp = NULL;
TdFilePtr pSrcFile = NULL; TdFilePtr pSrcFile = NULL;
char *data = taosMemoryMalloc(compressSize); char *data = taosMemoryMalloc(compressSize);
if (NULL == data) { if (NULL == data) {
return terrno; return terrno;
} }
pSrcFile = taosOpenFile(srcFileName, TD_FILE_READ | TD_FILE_STREAM); pSrcFile = taosOpenFile(srcFileName, TD_FILE_READ | TD_FILE_STREAM);
if (pSrcFile == NULL) { if (pSrcFile == NULL) {
ret = terrno; ret = terrno;
@ -1582,12 +1600,13 @@ int32_t taosLinkFile(char *src, char *dst) {
return 0; return 0;
} }
FILE *taosOpenCFile(const char *filename, const char *mode) { FILE *taosOpenCFile(const char *filename, const char *mode) {
FILE* f = fopen(filename, mode); STUB_RAND_IO_ERR(NULL)
FILE *f = fopen(filename, mode);
if (NULL == f) { if (NULL == f) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
} }
return f; return f;
} }
int taosSeekCFile(FILE *file, int64_t offset, int whence) { int taosSeekCFile(FILE *file, int64_t offset, int whence) {
@ -1603,10 +1622,12 @@ int taosSeekCFile(FILE *file, int64_t offset, int whence) {
} }
size_t taosReadFromCFile(void *buffer, size_t size, size_t count, FILE *stream) { size_t taosReadFromCFile(void *buffer, size_t size, size_t count, FILE *stream) {
STUB_RAND_IO_ERR(terrno)
return fread(buffer, size, count, stream); return fread(buffer, size, count, stream);
} }
size_t taosWriteToCFile(const void *ptr, size_t size, size_t nitems, FILE *stream) { size_t taosWriteToCFile(const void *ptr, size_t size, size_t nitems, FILE *stream) {
STUB_RAND_IO_ERR(terrno)
return fwrite(ptr, size, nitems, stream); return fwrite(ptr, size, nitems, stream);
} }
@ -1620,6 +1641,6 @@ int taosSetAutoDelFile(char *path) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
return terrno; return terrno;
} }
return 0; return 0;
#endif #endif
} }

View File

@ -21,10 +21,10 @@
#endif #endif
#include "os.h" #include "os.h"
int32_t tsRandErrChance = 1; int32_t tsRandErrChance = 1;
int64_t tsRandErrDivisor = 10001;
threadlocal bool tsEnableRandErr = 0; threadlocal bool tsEnableRandErr = 0;
#if defined(USE_TD_MEMORY) || defined(USE_ADDR2LINE) #if defined(USE_TD_MEMORY) || defined(USE_ADDR2LINE)
#define TD_MEMORY_SYMBOL ('T' << 24 | 'A' << 16 | 'O' << 8 | 'S') #define TD_MEMORY_SYMBOL ('T' << 24 | 'A' << 16 | 'O' << 8 | 'S')
@ -273,7 +273,7 @@ void *taosMemoryMalloc(int64_t size) {
#ifdef BUILD_WITH_RAND_ERR #ifdef BUILD_WITH_RAND_ERR
if (tsEnableRandErr) { if (tsEnableRandErr) {
uint32_t r = taosRand() % 10001; uint32_t r = taosRand() % tsRandErrDivisor;
if ((r + 1) <= tsRandErrChance) { if ((r + 1) <= tsRandErrChance) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL; return NULL;
@ -303,7 +303,7 @@ void *taosMemoryCalloc(int64_t num, int64_t size) {
#else #else
#ifdef BUILD_WITH_RAND_ERR #ifdef BUILD_WITH_RAND_ERR
if (tsEnableRandErr) { if (tsEnableRandErr) {
uint32_t r = taosRand() % 10001; uint32_t r = taosRand() % tsRandErrDivisor;
if ((r + 1) <= tsRandErrChance) { if ((r + 1) <= tsRandErrChance) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL; return NULL;
@ -343,7 +343,7 @@ void *taosMemoryRealloc(void *ptr, int64_t size) {
#else #else
#ifdef BUILD_WITH_RAND_ERR #ifdef BUILD_WITH_RAND_ERR
if (tsEnableRandErr) { if (tsEnableRandErr) {
uint32_t r = taosRand() % 10001; uint32_t r = taosRand() % tsRandErrDivisor;
if ((r + 1) <= tsRandErrChance) { if ((r + 1) <= tsRandErrChance) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL; return NULL;
@ -378,7 +378,7 @@ char *taosStrdup(const char *ptr) {
#else #else
#ifdef BUILD_WITH_RAND_ERR #ifdef BUILD_WITH_RAND_ERR
if (tsEnableRandErr) { if (tsEnableRandErr) {
uint32_t r = taosRand() % 10001; uint32_t r = taosRand() % tsRandErrDivisor;
if ((r + 1) <= tsRandErrChance) { if ((r + 1) <= tsRandErrChance) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL; return NULL;
@ -444,7 +444,7 @@ void *taosMemoryMallocAlign(uint32_t alignment, int64_t size) {
#if defined(LINUX) #if defined(LINUX)
#ifdef BUILD_WITH_RAND_ERR #ifdef BUILD_WITH_RAND_ERR
if (tsEnableRandErr) { if (tsEnableRandErr) {
uint32_t r = taosRand() % 10001; uint32_t r = taosRand() % tsRandErrDivisor;
if ((r + 1) <= tsRandErrChance) { if ((r + 1) <= tsRandErrChance) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL; return NULL;

View File

@ -798,7 +798,7 @@ int32_t taosSetSystemTimezone(const char *inTimezoneStr, char *outTimezoneStr, i
memcpy(&winStr[3], pp, ppp - pp); memcpy(&winStr[3], pp, ppp - pp);
indexStr = ppp - pp + 3; indexStr = ppp - pp + 3;
} }
sprintf(&winStr[indexStr], "%c%c%c:%c%c:00", (p[0] == '+' ? '-' : '+'), p[1], p[2], p[3], p[4]); sprintf(&winStr[indexStr], "%c%c%c:%c%c:00", (p[0] == '+' ? '+' : '-'), p[1], p[2], p[3], p[4]);
*tsTimezone = -taosStr2Int32(p, NULL, 10); *tsTimezone = -taosStr2Int32(p, NULL, 10);
} else { } else {
*tsTimezone = 0; *tsTimezone = 0;

View File

@ -131,16 +131,16 @@ void tBloomFilterDestroy(SBloomFilter* pBF) {
} }
int32_t tBloomFilterEncode(const SBloomFilter* pBF, SEncoder* pEncoder) { int32_t tBloomFilterEncode(const SBloomFilter* pBF, SEncoder* pEncoder) {
if (tEncodeU32(pEncoder, pBF->hashFunctions) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU32(pEncoder, pBF->hashFunctions));
if (tEncodeU64(pEncoder, pBF->expectedEntries) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU64(pEncoder, pBF->expectedEntries));
if (tEncodeU64(pEncoder, pBF->numUnits) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU64(pEncoder, pBF->numUnits));
if (tEncodeU64(pEncoder, pBF->numBits) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU64(pEncoder, pBF->numBits));
if (tEncodeU64(pEncoder, pBF->size) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU64(pEncoder, pBF->size));
for (uint64_t i = 0; i < pBF->numUnits; i++) { for (uint64_t i = 0; i < pBF->numUnits; i++) {
uint64_t* pUnits = (uint64_t*)pBF->buffer; uint64_t* pUnits = (uint64_t*)pBF->buffer;
if (tEncodeU64(pEncoder, pUnits[i]) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU64(pEncoder, pUnits[i]));
} }
if (tEncodeDouble(pEncoder, pBF->errorRate) < 0) return -1; TAOS_CHECK_RETURN(tEncodeDouble(pEncoder, pBF->errorRate));
return 0; return 0;
} }

View File

@ -92,7 +92,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_CFG_NOT_FOUND, "Config not found")
TAOS_DEFINE_ERROR(TSDB_CODE_REPEAT_INIT, "Repeat initialization") TAOS_DEFINE_ERROR(TSDB_CODE_REPEAT_INIT, "Repeat initialization")
TAOS_DEFINE_ERROR(TSDB_CODE_DUP_KEY, "Cannot add duplicate keys to hash") TAOS_DEFINE_ERROR(TSDB_CODE_DUP_KEY, "Cannot add duplicate keys to hash")
TAOS_DEFINE_ERROR(TSDB_CODE_NEED_RETRY, "Retry needed") TAOS_DEFINE_ERROR(TSDB_CODE_NEED_RETRY, "Retry needed")
TAOS_DEFINE_ERROR(TSDB_CODE_OUT_OF_RPC_MEMORY_QUEUE, "Out of memory in rpc queue") TAOS_DEFINE_ERROR(TSDB_CODE_OUT_OF_RPC_MEMORY_QUEUE, "Out of memory in queue")
TAOS_DEFINE_ERROR(TSDB_CODE_INVALID_TIMESTAMP, "Invalid timestamp format") TAOS_DEFINE_ERROR(TSDB_CODE_INVALID_TIMESTAMP, "Invalid timestamp format")
TAOS_DEFINE_ERROR(TSDB_CODE_MSG_DECODE_ERROR, "Msg decode error") TAOS_DEFINE_ERROR(TSDB_CODE_MSG_DECODE_ERROR, "Msg decode error")
TAOS_DEFINE_ERROR(TSDB_CODE_MSG_ENCODE_ERROR, "Msg encode error") TAOS_DEFINE_ERROR(TSDB_CODE_MSG_ENCODE_ERROR, "Msg encode error")
@ -723,6 +723,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_TIME_UNIT_INVALID, "Invalid function tim
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_TIME_UNIT_TOO_SMALL, "Function time unit cannot be smaller than db precision") TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_TIME_UNIT_TOO_SMALL, "Function time unit cannot be smaller than db precision")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_INVALID_VALUE_RANGE, "Function got invalid value range") TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_INVALID_VALUE_RANGE, "Function got invalid value range")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_SETUP_ERROR, "Function set up failed") TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_SETUP_ERROR, "Function set up failed")
TAOS_DEFINE_ERROR(TSDB_CODE_FUNC_INVALID_RES_LENGTH, "Function result exceed max length")
//udf //udf
TAOS_DEFINE_ERROR(TSDB_CODE_UDF_STOPPING, "udf is stopping") TAOS_DEFINE_ERROR(TSDB_CODE_UDF_STOPPING, "udf is stopping")

View File

@ -19,8 +19,8 @@
#include "tlog.h" #include "tlog.h"
#include "tutil.h" #include "tutil.h"
int64_t tsRpcQueueMemoryAllowed = 0; int64_t tsQueueMemoryAllowed = 0;
int64_t tsRpcQueueMemoryUsed = 0; int64_t tsQueueMemoryUsed = 0;
struct STaosQueue { struct STaosQueue {
STaosQnode *head; STaosQnode *head;
@ -148,10 +148,20 @@ int64_t taosQueueMemorySize(STaosQueue *queue) {
} }
int32_t taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize, void **item) { int32_t taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize, void **item) {
*item = NULL; int64_t alloced = atomic_add_fetch_64(&tsQueueMemoryUsed, size + dataSize);
if (alloced > tsQueueMemoryAllowed) {
if (itype == RPC_QITEM) {
uError("failed to alloc qitem, size:%" PRId64 " alloc:%" PRId64 " allowed:%" PRId64, size + dataSize, alloced,
tsQueueMemoryAllowed);
(void)atomic_sub_fetch_64(&tsQueueMemoryUsed, size + dataSize);
return (terrno = TSDB_CODE_OUT_OF_RPC_MEMORY_QUEUE);
}
}
*item = NULL;
STaosQnode *pNode = taosMemoryCalloc(1, sizeof(STaosQnode) + size); STaosQnode *pNode = taosMemoryCalloc(1, sizeof(STaosQnode) + size);
if (pNode == NULL) { if (pNode == NULL) {
(void)atomic_sub_fetch_64(&tsQueueMemoryUsed, size + dataSize);
return terrno = TSDB_CODE_OUT_OF_MEMORY; return terrno = TSDB_CODE_OUT_OF_MEMORY;
} }
@ -159,21 +169,7 @@ int32_t taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize, void **
pNode->size = size; pNode->size = size;
pNode->itype = itype; pNode->itype = itype;
pNode->timestamp = taosGetTimestampUs(); pNode->timestamp = taosGetTimestampUs();
uTrace("item:%p, node:%p is allocated, alloc:%" PRId64, pNode->item, pNode, alloced);
if (itype == RPC_QITEM) {
int64_t alloced = atomic_add_fetch_64(&tsRpcQueueMemoryUsed, size + dataSize);
if (alloced > tsRpcQueueMemoryAllowed) {
uError("failed to alloc qitem, size:%" PRId64 " alloc:%" PRId64 " allowed:%" PRId64, size + dataSize, alloced,
tsRpcQueueMemoryAllowed);
(void)atomic_sub_fetch_64(&tsRpcQueueMemoryUsed, size + dataSize);
taosMemoryFree(pNode);
return (terrno = TSDB_CODE_OUT_OF_RPC_MEMORY_QUEUE);
}
uTrace("item:%p, node:%p is allocated, alloc:%" PRId64, pNode->item, pNode, alloced);
} else {
uTrace("item:%p, node:%p is allocated", pNode->item, pNode);
}
*item = pNode->item; *item = pNode->item;
return 0; return 0;
} }
@ -182,12 +178,8 @@ void taosFreeQitem(void *pItem) {
if (pItem == NULL) return; if (pItem == NULL) return;
STaosQnode *pNode = (STaosQnode *)((char *)pItem - sizeof(STaosQnode)); STaosQnode *pNode = (STaosQnode *)((char *)pItem - sizeof(STaosQnode));
if (pNode->itype == RPC_QITEM) { int64_t alloced = atomic_sub_fetch_64(&tsQueueMemoryUsed, pNode->size + pNode->dataSize);
int64_t alloced = atomic_sub_fetch_64(&tsRpcQueueMemoryUsed, pNode->size + pNode->dataSize); uTrace("item:%p, node:%p is freed, alloc:%" PRId64, pItem, pNode, alloced);
uTrace("item:%p, node:%p is freed, alloc:%" PRId64, pItem, pNode, alloced);
} else {
uTrace("item:%p, node:%p is freed", pItem, pNode);
}
taosMemoryFree(pNode); taosMemoryFree(pNode);
} }
@ -380,7 +372,7 @@ void taosQsetThreadResume(STaosQset *qset) {
} }
int32_t taosAddIntoQset(STaosQset *qset, STaosQueue *queue, void *ahandle) { int32_t taosAddIntoQset(STaosQset *qset, STaosQueue *queue, void *ahandle) {
if (queue->qset) return -1; if (queue->qset) return TSDB_CODE_INVALID_PARA;
(void)taosThreadMutexLock(&qset->mutex); (void)taosThreadMutexLock(&qset->mutex);

View File

@ -188,19 +188,19 @@ void tScalableBfDestroy(SScalableBf* pSBf) {
int32_t tScalableBfEncode(const SScalableBf* pSBf, SEncoder* pEncoder) { int32_t tScalableBfEncode(const SScalableBf* pSBf, SEncoder* pEncoder) {
if (!pSBf) { if (!pSBf) {
if (tEncodeI32(pEncoder, 0) < 0) return -1; TAOS_CHECK_RETURN(tEncodeI32(pEncoder, 0));
return 0; return 0;
} }
int32_t size = taosArrayGetSize(pSBf->bfArray); int32_t size = taosArrayGetSize(pSBf->bfArray);
if (tEncodeI32(pEncoder, size) < 0) return -1; TAOS_CHECK_RETURN(tEncodeI32(pEncoder, size));
for (int32_t i = 0; i < size; i++) { for (int32_t i = 0; i < size; i++) {
SBloomFilter* pBF = taosArrayGetP(pSBf->bfArray, i); SBloomFilter* pBF = taosArrayGetP(pSBf->bfArray, i);
if (tBloomFilterEncode(pBF, pEncoder) < 0) return -1; TAOS_CHECK_RETURN(tBloomFilterEncode(pBF, pEncoder));
} }
if (tEncodeU32(pEncoder, pSBf->growth) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU32(pEncoder, pSBf->growth));
if (tEncodeU64(pEncoder, pSBf->numBits) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU64(pEncoder, pSBf->numBits));
if (tEncodeU32(pEncoder, pSBf->maxBloomFilters) < 0) return -1; TAOS_CHECK_RETURN(tEncodeU32(pEncoder, pSBf->maxBloomFilters));
if (tEncodeI8(pEncoder, pSBf->status) < 0) return -1; TAOS_CHECK_RETURN(tEncodeI8(pEncoder, pSBf->status));
return 0; return 0;
} }

View File

@ -189,12 +189,12 @@ int taosScheduleTask(void *queueScheduler, SSchedMsg *pMsg) {
if (pSched == NULL) { if (pSched == NULL) {
uError("sched is not ready, msg:%p is dropped", pMsg); uError("sched is not ready, msg:%p is dropped", pMsg);
return -1; return TSDB_CODE_INVALID_PARA;
} }
if (atomic_load_8(&pSched->stop)) { if (atomic_load_8(&pSched->stop)) {
uError("sched is already stopped, msg:%p is dropped", pMsg); uError("sched is already stopped, msg:%p is dropped", pMsg);
return -1; return TSDB_CODE_INVALID_PARA;
} }
if ((ret = tsem_wait(&pSched->emptySem)) != 0) { if ((ret = tsem_wait(&pSched->emptySem)) != 0) {

View File

@ -209,7 +209,7 @@ static int32_t tSimpleHashTableResize(SSHashObj *pHashObj) {
int32_t tSimpleHashPut(SSHashObj *pHashObj, const void *key, size_t keyLen, const void *data, size_t dataLen) { int32_t tSimpleHashPut(SSHashObj *pHashObj, const void *key, size_t keyLen, const void *data, size_t dataLen) {
if (!pHashObj || !key) { if (!pHashObj || !key) {
return TSDB_CODE_FAILED; return TSDB_CODE_INVALID_PARA;
} }
uint32_t hashVal = (*pHashObj->hashFp)(key, (uint32_t)keyLen); uint32_t hashVal = (*pHashObj->hashFp)(key, (uint32_t)keyLen);

View File

@ -536,7 +536,7 @@ class TDCom:
cur = self.newcur(host=host,port=port,user=user,password=password) cur = self.newcur(host=host,port=port,user=user,password=password)
newTdSql.init(cur, False) newTdSql.init(cur, False)
return newTdSql return newTdSql
################################################################################################################ ################################################################################################################
# port from the common.py of new test frame # port from the common.py of new test frame
################################################################################################################ ################################################################################################################

View File

@ -0,0 +1,101 @@
from frame.log import *
from frame.cases import *
from frame.sql import *
from frame.caseBase import *
from frame import *
class TDTestCase(TBase):
def init(self, conn, logSql, replicaVar=1):
self.replicaVar = int(replicaVar)
tdLog.debug(f"start to excute {__file__}")
tdSql.init(conn.cursor(), logSql)
def prepare_data(self):
tdSql.execute("create database db;")
tdSql.execute("use db;")
# data for fill(prev)
tdSql.execute("create stable st_pre (ts timestamp, c1 int) tags(t1 int);")
tdSql.execute("create table ct1 using st_pre tags(1);")
start_ts = 1705783972000
sql = "insert into ct1 values "
for i in range(100):
sql += f"({start_ts + i * 1000}, {str(i+1)})"
sql += ";"
tdSql.execute(sql)
# data for fill(next)
tdSql.execute("create stable st_next (ts timestamp, c1 int) tags(t1 int);")
tdSql.execute("create table ct2 using st_next tags(1);")
start_ts = 1705783972000
sql = "insert into ct1 values "
for i in range(100):
sql += f"({start_ts + i * 1000}, NULL)"
sql += ";"
tdSql.execute(sql)
# data for fill(linear)
tdSql.execute("create stable st_linear (ts timestamp, c1 int) tags(t1 int);")
tdSql.execute("create table ct3 using st_linear tags(1);")
start_ts = 1705783972000
sql = "insert into ct1 values "
for i in range(100):
if i % 2 == 0:
sql += f"({start_ts + i * 1000}, {str(i+1)})"
else:
sql += f"({start_ts + i * 1000}, NULL)"
sql += ";"
tdSql.execute(sql)
tdLog.info("prepare data done")
def test_fill_pre_compare_asc_desc(self):
tdSql.execute("use db;")
for func in ["avg(c1)", "count(c1)", "first(c1)", "last(c1)", "max(c1)", "min(c1)", "sum(c1)"]:
tdSql.query(f"select _wstart, {func} from st_pre where ts between '2024-01-21 04:52:52.000' and '2024-01-21 04:54:31.000' interval(5s) fill(prev) order by _wstart asc;")
res1 = tdSql.res
tdSql.query(f"select _wstart, {func} from st_pre where ts between '2024-01-21 04:52:52.000' and '2024-01-21 04:54:31.000' interval(5s) fill(prev) order by _wstart desc;")
res2 = tdSql.res
assert len(res1) == len(res2)
for i in range(len(res1)):
assert res1[i] in res2
tdLog.info(f"fill(prev) {func} compare asc and desc done")
tdLog.info("Finish the test case 'test_fill_pre_compare_asc_desc'")
def test_fill_next_compare_asc_desc(self):
tdSql.execute("use db;")
for func in ["avg(c1)", "count(c1)", "first(c1)", "last(c1)", "max(c1)", "min(c1)", "sum(c1)"]:
tdSql.query(f"select _wstart, {func} from st_next where ts between '2024-01-21 04:52:52.000' and '2024-01-21 04:54:31.000' interval(5s) fill(next) order by _wstart asc;")
res1 = tdSql.res
tdSql.query(f"select _wstart, {func} from st_next where ts between '2024-01-21 04:52:52.000' and '2024-01-21 04:54:31.000' interval(5s) fill(next) order by _wstart desc;")
res2 = tdSql.res
assert len(res1) == len(res2)
for i in range(len(res1)):
assert res1[i] in res2
tdLog.info(f"fill(next) {func} compare asc and desc done")
tdLog.info("Finish the test case 'test_fill_next_compare_asc_desc'")
def test_fill_linear_compare_asc_desc(self):
tdSql.execute("use db;")
for func in ["avg(c1)", "count(c1)", "first(c1)", "last(c1)", "max(c1)", "min(c1)", "sum(c1)"]:
tdSql.query(f"select _wstart, {func} from st_linear where ts between '2024-01-21 04:52:52.000' and '2024-01-21 04:54:31.000' interval(5s) fill(linear) order by _wstart asc;")
res1 = tdSql.res
tdSql.query(f"select _wstart, {func} from st_linear where ts between '2024-01-21 04:52:52.000' and '2024-01-21 04:54:31.000' interval(5s) fill(linear) order by _wstart desc;")
res2 = tdSql.res
assert len(res1) == len(res2)
for i in range(len(res1)):
assert res1[i] in res2
tdLog.info(f"fill(linear) {func} compare asc and desc done")
tdLog.info("Finish the test case 'test_fill_linear_compare_asc_desc'")
def run(self):
self.prepare_data()
self.test_fill_pre_compare_asc_desc()
self.test_fill_next_compare_asc_desc()
self.test_fill_linear_compare_asc_desc()
def stop(self):
tdSql.close()
tdLog.success(f"{__file__} successfully executed")
tdCases.addLinux(__file__, TDTestCase())
tdCases.addWindows(__file__, TDTestCase())

View File

@ -0,0 +1,44 @@
import taos
import frame
import frame.etool
from frame.log import *
from frame.cases import *
from frame.sql import *
from frame.caseBase import *
from frame import *
class TDTestCase(TBase):
"""Verify the concat function
"""
def init(self, conn, logSql, replicaVar=1):
self.replicaVar = int(replicaVar)
tdSql.init(conn.cursor())
self.dbname = 'db'
self.tbname = f'{self.dbname}.tbconcat'
def checkConcat(self):
tdSql.execute(f'drop database if exists {self.dbname}')
tdSql.execute(f'create database {self.dbname}')
tdSql.execute(f'use {self.dbname}')
tdSql.execute(f'create table if not exists {self.tbname}(ts timestamp, name varchar(20), name2 nchar(20))')
tdSql.execute(f'insert into {self.tbname} values(now(),"abcdefg","你好")')
tdSql.execute(f'insert into {self.tbname} values(now(),"abcdefgh","我好")')
tdSql.execute(f'insert into {self.tbname} values(now(),"abcdefg", "")')
tdSql.execute(f'select concat("你好",name2) from {self.tbname}')
tdSql.execute(f'select concat(name2,"你好") from {self.tbname}')
tdSql.execute(f'select concat(name2,"") from {self.tbname}')
tdSql.execute(f'select concat("", name2) from {self.tbname}')
def run(self):
tdLog.debug(f"start to excute {__file__}")
# check concat function
self.checkConcat()
tdLog.success(f"{__file__} successfully executed")
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
tdCases.addLinux(__file__, TDTestCase())
tdCases.addWindows(__file__, TDTestCase())

View File

@ -262,9 +262,9 @@ class TDTestCase(TBase):
sql1 = f"select substr(bin,1) from {self.db}.d0 order by ts desc limit 100" sql1 = f"select substr(bin,1) from {self.db}.d0 order by ts desc limit 100"
sql2 = f"select bin from {self.db}.d0 order by ts desc limit 100" sql2 = f"select bin from {self.db}.d0 order by ts desc limit 100"
self.checkSameResult(sql1, sql2) self.checkSameResult(sql1, sql2)
#substr error input pos is zero
sql = f"select substr(bin,0,3) from {self.db}.d0 order by ts desc limit 100" sql = f"select substr(bin,0,3) from {self.db}.d0 order by ts desc limit 100"
tdSql.error(sql) tdSql.query(sql)
tdSql.checkData(0, 0, "")
# cast # cast
nch = 99 nch = 99

Some files were not shown because too many files have changed in this diff Show More