WANG MINGMING
GitHub
commit
ebf31f6f06
|
|
@ -212,7 +212,7 @@ curl -L -o php-tdengine.tar.gz https://github.com/Yurunsoft/php-tdengine/archive
|
|||
&& tar -xzf php-tdengine.tar.gz -C php-tdengine --strip-components=1
|
||||
```
|
||||
|
||||
> 版本 `v1.0.0` 可替换为任意更新的版本，可在 Release 中查看最新版本。
|
||||
> 版本 `v1.0.2` 只是示例，可替换为任意更新的版本，可在 [TDengine PHP Connector 发布历史](https://github.com/Yurunsoft/php-tdengine/releases) 中查看可用版本。
|
||||
|
||||
**非 Swoole 环境：**
|
||||
|
||||
|
|
|
|||
|
|
@ -93,8 +93,8 @@ Maven 项目中，在 pom.xml 中添加以下依赖：
|
|||
可以通过下载 TDengine 的源码，自己编译最新版本的 Java connector
|
||||
|
||||
```shell
|
||||
git clone https://github.com/taosdata/TDengine.git
|
||||
cd TDengine/src/connector/jdbc
|
||||
git clone https://github.com/taosdata/taos-connector-jdbc.git
|
||||
cd taos-connector-jdbc
|
||||
mvn clean install -Dmaven.test.skip=true
|
||||
```
|
||||
|
||||
|
|
@ -199,6 +199,7 @@ url 中的配置参数如下：
|
|||
- user：登录 TDengine 用户名，默认值 'root'。
|
||||
- password：用户登录密码，默认值 'taosdata'。
|
||||
- batchfetch: true：在执行查询时批量拉取结果集；false：逐行拉取结果集。默认值为：false。逐行拉取结果集使用 HTTP 方式进行数据传输。从 taos-jdbcdriver-2.0.38 和 TDengine 2.4.0.12 版本开始，JDBC REST 连接增加批量拉取数据功能。taos-jdbcdriver 与 TDengine 之间通过 WebSocket 连接进行数据传输。相较于 HTTP，WebSocket 可以使 JDBC REST 连接支持大数据量查询，并提升查询性能。
|
||||
- charset: 当开启批量拉取数据时，指定解析字符串数据的字符集。
|
||||
- batchErrorIgnore：true：在执行 Statement 的 executeBatch 时，如果中间有一条 SQL 执行失败，继续执行下面的 SQL 了。false：不再执行失败 SQL 后的任何语句。默认值为：false。
|
||||
|
||||
**注意**：部分配置项（比如：locale、timezone）在 REST 连接中不生效。
|
||||
|
|
@ -260,7 +261,7 @@ properties 中的配置参数如下：
|
|||
- TSDBDriver.PROPERTY_KEY_BATCH_LOAD: true：在执行查询时批量拉取结果集；false：逐行拉取结果集。默认值为：false。
|
||||
- TSDBDriver.PROPERTY_KEY_BATCH_ERROR_IGNORE：true：在执行 Statement 的 executeBatch 时，如果中间有一条 SQL 执行失败，继续执行下面的 sq 了。false：不再执行失败 SQL 后的任何语句。默认值为：false。
|
||||
- TSDBDriver.PROPERTY_KEY_CONFIG_DIR：仅在使用 JDBC 原生连接时生效。客户端配置文件目录路径，Linux OS 上默认值 `/etc/taos`，Windows OS 上默认值 `C:/TDengine/cfg`。
|
||||
- TSDBDriver.PROPERTY_KEY_CHARSET：仅在使用 JDBC 原生连接时生效。 客户端使用的字符集，默认值为系统字符集。
|
||||
- TSDBDriver.PROPERTY_KEY_CHARSET：客户端使用的字符集，默认值为系统字符集。
|
||||
- TSDBDriver.PROPERTY_KEY_LOCALE：仅在使用 JDBC 原生连接时生效。 客户端语言环境，默认值系统当前 locale。
|
||||
- TSDBDriver.PROPERTY_KEY_TIME_ZONE：仅在使用 JDBC 原生连接时生效。 客户端使用的时区，默认值为系统当前时区。
|
||||
- 此外对 JDBC 原生连接，通过指定 URL 和 Properties 还可以指定其他参数，比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](/reference/config/#仅客户端适用)。
|
||||
|
|
@ -348,7 +349,7 @@ JDBC 连接器可能报错的错误码包括 3 种：JDBC driver 本身的报错
|
|||
|
||||
具体的错误码请参考：
|
||||
|
||||
- [TDengine Java Connector](https://github.com/taosdata/TDengine/blob/develop/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBErrorNumbers.java)
|
||||
- [TDengine Java Connector](https://github.com/taosdata/taos-connector-jdbc/blob/main/src/main/java/com/taosdata/jdbc/TSDBErrorNumbers.java)
|
||||
- [TDengine_ERROR_CODE](https://github.com/taosdata/TDengine/blob/develop/src/inc/taoserror.h)
|
||||
|
||||
### 通过参数绑定写入数据
|
||||
|
|
|
|||
|
|
@ -0,0 +1,150 @@
|
|||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: PHP
|
||||
title: PHP Connector
|
||||
---
|
||||
|
||||
`php-tdengine` 是由社区贡献的 PHP 连接器扩展，还特别支持了 Swoole 协程化。
|
||||
|
||||
PHP 连接器依赖 TDengine 客户端驱动。
|
||||
|
||||
项目地址：<https://github.com/Yurunsoft/php-tdengine>
|
||||
|
||||
TDengine 服务端或客户端安装后，`taos.h` 位于：
|
||||
|
||||
- Linux：`/usr/local/taos/include`
|
||||
- Windows：`C:\TDengine\include`
|
||||
|
||||
TDengine 客户端驱动的动态库位于：
|
||||
|
||||
- Linux: `/usr/local/taos/driver/libtaos.so`
|
||||
- Windows: `C:\TDengine\taos.dll`
|
||||
|
||||
## 支持的平台
|
||||
|
||||
* Windows、Linux、MacOS
|
||||
|
||||
* PHP >= 7.4
|
||||
|
||||
* TDengine >= 2.0
|
||||
|
||||
* Swoole >= 4.8 (可选)
|
||||
|
||||
## 支持的版本
|
||||
|
||||
TDengine 客户端驱动的版本号与 TDengine 服务端的版本号是一一对应的强对应关系，建议使用与 TDengine 服务端完全相同的客户端驱动。虽然低版本的客户端驱动在前三段版本号一致（即仅第四段版本号不同）的情况下也能够与高版本的服务端相兼容，但这并非推荐用法。强烈不建议使用高版本的客户端驱动访问低版本的服务端。
|
||||
|
||||
## 安装步骤
|
||||
|
||||
### 安装 TDengine 客户端驱动
|
||||
|
||||
TDengine 客户端驱动的安装请参考 [安装指南](/reference/connector#安装步骤)
|
||||
|
||||
### 编译安装 php-tdengine
|
||||
|
||||
**下载代码并解压：**
|
||||
|
||||
```shell
|
||||
curl -L -o php-tdengine.tar.gz https://github.com/Yurunsoft/php-tdengine/archive/refs/tags/v1.0.2.tar.gz \
|
||||
&& mkdir php-tdengine \
|
||||
&& tar -xzf php-tdengine.tar.gz -C php-tdengine --strip-components=1
|
||||
```
|
||||
|
||||
> 版本 `v1.0.2` 可替换为任意更新的版本，可在 [TDengine PHP Connector 发布历史](https://github.com/Yurunsoft/php-tdengine/releases)。
|
||||
|
||||
**非 Swoole 环境：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure && make -j && make install
|
||||
```
|
||||
|
||||
**手动指定 tdengine 目录：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/2.4.0.0 && make -j && make install
|
||||
```
|
||||
|
||||
> `--with-tdengine-dir=` 后跟上 tdengine 目录。
|
||||
> 适用于默认找不到的情况，或者 MacOS 系统用户。
|
||||
|
||||
**Swoole 环境：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --enable-swoole && make -j && make install
|
||||
```
|
||||
|
||||
**启用扩展：**
|
||||
|
||||
方法一：在 `php.ini` 中加入 `extension=tdengine`
|
||||
|
||||
方法二：运行带参数 `php -dextension=tdengine test.php`
|
||||
|
||||
## 示例程序
|
||||
|
||||
本节展示了使用客户端驱动访问 TDengine 集群的常见访问方式的示例代码。
|
||||
|
||||
> 所有错误都会抛出异常: `TDengine\Exception\TDengineException`
|
||||
|
||||
### 建立连接
|
||||
|
||||
<details>
|
||||
<summary>建立连接</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/connect.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### 插入数据
|
||||
|
||||
<details>
|
||||
<summary>插入数据</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/insert.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### 同步查询
|
||||
|
||||
<details>
|
||||
<summary>同步查询</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/query.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### 参数绑定
|
||||
|
||||
<details>
|
||||
<summary>参数绑定</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/insert_stmt.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## 常量
|
||||
|
||||
| 常量 | 说明 |
|
||||
| ------------ | ------------
|
||||
| `TDengine\TSDB_DATA_TYPE_NULL` | null |
|
||||
| `TDengine\TSDB_DATA_TYPE_BOOL` | bool |
|
||||
| `TDengine\TSDB_DATA_TYPE_TINYINT` | tinyint |
|
||||
| `TDengine\TSDB_DATA_TYPE_SMALLINT` | smallint |
|
||||
| `TDengine\TSDB_DATA_TYPE_INT` | int |
|
||||
| `TDengine\TSDB_DATA_TYPE_BIGINT` | bigint |
|
||||
| `TDengine\TSDB_DATA_TYPE_FLOAT` | float |
|
||||
| `TDengine\TSDB_DATA_TYPE_DOUBLE` | double |
|
||||
| `TDengine\TSDB_DATA_TYPE_BINARY` | binary |
|
||||
| `TDengine\TSDB_DATA_TYPE_TIMESTAMP` | timestamp |
|
||||
| `TDengine\TSDB_DATA_TYPE_NCHAR` | nchar |
|
||||
| `TDengine\TSDB_DATA_TYPE_UTINYINT` | utinyint |
|
||||
| `TDengine\TSDB_DATA_TYPE_USMALLINT` | usmallint |
|
||||
| `TDengine\TSDB_DATA_TYPE_UINT` | uint |
|
||||
| `TDengine\TSDB_DATA_TYPE_UBIGINT` | ubigint |
|
||||
|
|
@ -199,10 +199,9 @@ curl -u root:taosdata http://<FQDN>:<PORT>/rest/sql -d "select server_version()"
|
|||
|
||||
`connect()` 函数的所有参数都是可选的关键字参数。下面是连接参数的具体说明：
|
||||
|
||||
- `host`： 要连接的主机。默认是 localhost。
|
||||
- `url`： taosAdapter REST 服务的 URL。默认是 <http://localhost:6041>。
|
||||
- `user`： TDenigne 用户名。默认是 root。
|
||||
- `password`： TDeingine 用户密码。默认是 taosdata。
|
||||
- `port`: taosAdapter REST 服务监听端口。默认是 6041.
|
||||
- `timeout`: HTTP 请求超时时间。单位为秒。默认为 `socket._GLOBAL_DEFAULT_TIMEOUT`。 一般无需配置。
|
||||
|
||||
</TabItem>
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ taosBenchmark 有两种安装方式:
|
|||
|
||||
### 配置和运行方式
|
||||
|
||||
taosBenchmark 支持两种配置方式：[命令行参数](#命令行参数详解) 和 [JSON 配置文件](#配置文件参数详解)。这两种方式是互斥的，在使用配置文件时只能使用一个命令行参数 `-f <json file>` 指定配置文件。在使用命令行参数运行 taosBenchmark 并控制其行为时则不能使用 `-f` 参数而要用其它参数来进行配置。除此之外，taosBenchmark 还提供了一种特殊的运行方式，即无参数运行。
|
||||
taosBenchmark 需要在操作系统的终端执行,该工具支持两种配置方式：[命令行参数](#命令行参数详解) 和 [JSON 配置文件](#配置文件参数详解)。这两种方式是互斥的，在使用配置文件时只能使用一个命令行参数 `-f <json file>` 指定配置文件。在使用命令行参数运行 taosBenchmark 并控制其行为时则不能使用 `-f` 参数而要用其它参数来进行配置。除此之外，taosBenchmark 还提供了一种特殊的运行方式，即无参数运行。
|
||||
|
||||
taosBenchmark 支持对 TDengine 做完备的性能测试，其所支持的 TDengine 功能分为三大类：写入、查询和订阅。这三种功能之间是互斥的，每次运行 taosBenchmark 只能选择其中之一。值得注意的是，所要测试的功能类型在使用命令行配置方式时是不可配置的，命令行配置方式只能测试写入性能。若要测试 TDengine 的查询和订阅性能，必须使用配置文件的方式，通过配置文件中的参数 `filetype` 指定所要测试的功能类型。
|
||||
|
||||
|
|
|
|||
|
|
@ -134,7 +134,7 @@ taos --dump-config
|
|||
| 适用范围 | 仅服务端适用 |
|
||||
| 含义 | 服务器内部的系统监控开关。监控主要负责收集物理节点的负载状况，包括 CPU、内存、硬盘、网络带宽、HTTP 请求量的监控记录，记录信息存储在`LOG`库中。 |
|
||||
| 取值范围 | 0：关闭监控服务， 1：激活监控服务。 |
|
||||
| 缺省值 | 0 |
|
||||
| 缺省值 | 1 |
|
||||
|
||||
### monitorInterval
|
||||
|
||||
|
|
|
|||
|
|
@ -12,12 +12,38 @@ TDengine 能够与开源数据可视化系统 [Grafana](https://www.grafana.com/
|
|||
- TDengine 集群已经部署并正常运行
|
||||
- taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](/reference/taosadapter)
|
||||
|
||||
记录以下信息：
|
||||
|
||||
- TDengine 集群 REST API 地址，如：`http://tdengine.local:6041`。
|
||||
- TDengine 集群认证信息，可使用用户名及密码。
|
||||
|
||||
## 安装 Grafana
|
||||
|
||||
目前 TDengine 支持 Grafana 7.0 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：<https://grafana.com/grafana/download>。
|
||||
目前 TDengine 支持 Grafana 7.5 以上的版本。用户可以根据当前的操作系统，到 Grafana 官网下载安装包，并执行安装。下载地址如下：<https://grafana.com/grafana/download>。
|
||||
|
||||
## 配置 Grafana
|
||||
|
||||
### 安装方式一：安装脚本
|
||||
|
||||
将集群信息设置为环境变量（也可以使用 [`.env`(dotenv) 文件](https://hexdocs.pm/dotenvy/dotenv-file-format.html) ）：
|
||||
|
||||
```sh
|
||||
export TDENGINE_API=http://tdengine.local:6041
|
||||
# user + password
|
||||
export TDENGINE_USER=user
|
||||
export TDENGINE_PASSWORD=password
|
||||
```
|
||||
|
||||
运行安装脚本：
|
||||
|
||||
```sh
|
||||
bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)"
|
||||
```
|
||||
|
||||
该脚本将自动安装 Grafana 插件并配置数据源。安装完毕后，需要重启 Grafana 服务后生效。
|
||||
|
||||
### 安装方式二：手动安装 TDengine 数据源插件
|
||||
|
||||
使用 [`grafana-cli` 命令行工具](https://grafana.com/docs/grafana/latest/administration/cli/) 进行插件[安装](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation)。
|
||||
|
||||
```bash
|
||||
|
|
@ -48,11 +74,7 @@ sudo unzip tdengine-datasource-$GF_VERSION.zip -d /var/lib/grafana/plugins/
|
|||
GF_INSTALL_PLUGINS=tdengine-datasource
|
||||
```
|
||||
|
||||
## 使用 Grafana
|
||||
|
||||
### 配置数据源
|
||||
|
||||
用户可以直接通过 <http://localhost:3000> 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示：
|
||||
之后，用户可以直接通过 <http://localhost:3000> 的网址，登录 Grafana 服务器（用户名/密码：admin/admin），通过左侧 `Configuration -> Data Sources` 可以添加数据源，如下图所示：
|
||||
|
||||
|
||||
|
||||
|
|
@ -92,4 +114,11 @@ GF_INSTALL_PLUGINS=tdengine-datasource
|
|||
|
||||
### 导入 Dashboard
|
||||
|
||||
在 2.3.3.0 及以上版本，您可以导入 TDinsight Dashboard (Grafana Dashboard ID： [15167](https://grafana.com/grafana/dashboards/15167)) 作为 TDengine 集群的监控可视化工具。安装和使用说明请见 [TDinsight 用户手册](/reference/tdinsight/)。
|
||||
在数据源配置页面，您可以为该数据源导入 TDinsight 面板，作为 TDengine 集群的监控可视化工具。该 Dashboard 已发布在 Grafana：[Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167)) 。其他安装方式和相关使用说明请见 [TDinsight 用户手册](/reference/tdinsight/)。
|
||||
|
||||
使用 TDengine 作为数据源的其他面板，可以[在此搜索](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource)。以下是一份不完全列表：
|
||||
|
||||
- [15146](https://grafana.com/grafana/dashboards/15146): 监控多个 TDengine 集群
|
||||
- [15155](https://grafana.com/grafana/dashboards/15155): TDengine 告警示例
|
||||
- [15167](https://grafana.com/grafana/dashboards/15167): TDinsight
|
||||
- [16388](https://grafana.com/grafana/dashboards/16388): Telegraf 采集节点信息的数据展示
|
||||
|
|
|
|||
|
|
@ -8,31 +8,24 @@ MQTT 是流行的物联网数据传输协议，[EMQX](https://github.com/emqx/em
|
|||
## 前置条件
|
||||
|
||||
要让 EMQX 能正常添加 TDengine 数据源，需要以下几方面的准备工作。
|
||||
|
||||
- TDengine 集群已经部署并正常运行
|
||||
- taosAdapter 已经安装并正常运行。具体细节请参考 [taosAdapter 的使用手册](/reference/taosadapter)
|
||||
- 如果使用后文介绍的模拟写入程序，需要安装合适版本的 Node.js，推荐安装 v12。
|
||||
- 如果使用后文介绍的模拟写入程序，需要安装合适版本的 Node.js，推荐安装 v12
|
||||
|
||||
## 安装并启动 EMQX
|
||||
|
||||
用户可以根据当前的操作系统，到 EMQX 官网下载安装包，并执行安装。下载地址如下：<https://www.emqx.io/zh/downloads>。安装后使用 `sudo emqx start` 或 `sudo systemctl start emqx` 启动 EMQX 服务。
|
||||
|
||||
## 在 TDengine 中为接收 MQTT 数据创建相应数据库和表结构
|
||||
|
||||
### 以 Docker 安装 TDengine 为例
|
||||
## 创建数据库和表
|
||||
|
||||
```bash
|
||||
docker exec -it tdengine bash
|
||||
taos
|
||||
```
|
||||
|
||||
### 创建数据库和表
|
||||
在 TDengine 中为接收 MQTT 数据创建相应数据库和表结构。进入 TDengine CLI 复制并执行以下 SQL 语句：
|
||||
|
||||
```sql
|
||||
create database test;
|
||||
use test;
|
||||
create table:
|
||||
|
||||
CREATE TABLE sensor_data (ts timestamp, temperature float, humidity float, volume float, PM10 float, pm25 float, SO2 float, NO2 float, CO float, sensor_id NCHAR(255), area TINYINT, coll_time timestamp);
|
||||
CREATE DATABASE test;
|
||||
USE test;
|
||||
CREATE TABLE sensor_data (ts TIMESTAMP, temperature FLOAT, humidity FLOAT, volume FLOAT, pm10 FLOAT, pm25 FLOAT, so2 FLOAT, no2 FLOAT, co FLOAT, sensor_id NCHAR(255), area TINYINT, coll_time TIMESTAMP);
|
||||
```
|
||||
|
||||
注：表结构以博客[数据传输、存储、展现，EMQX + TDengine 搭建 MQTT 物联网数据可视化平台](https://www.taosdata.com/blog/2020/08/04/1722.html)为例。后续操作均以此博客场景为例进行，请你根据实际应用场景进行修改。
|
||||
|
|
@ -43,7 +36,7 @@ MQTT 是流行的物联网数据传输协议，[EMQX](https://github.com/emqx/em
|
|||
|
||||
### 登录 EMQX Dashboard
|
||||
|
||||
使用浏览器打开网址 http://IP:18083 并登录 EMQX Dashboard。初次安装用户名为 `admin` 密码为：`public`
|
||||
使用浏览器打开网址 http://IP:18083 并登录 EMQX Dashboard。初次安装用户名为 `admin` 密码为：`public`。
|
||||
|
||||
|
||||
|
||||
|
|
@ -55,6 +48,17 @@ MQTT 是流行的物联网数据传输协议，[EMQX](https://github.com/emqx/em
|
|||
|
||||
### 编辑 SQL 字段
|
||||
|
||||
复制以下内容输入到 SQL 编辑框：
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
payload
|
||||
FROM
|
||||
"sensor/data"
|
||||
```
|
||||
|
||||
其中 `payload` 代表整个消息体， `sensor/data` 为本规则选取的消息主题。
|
||||
|
||||
|
||||
|
||||
### 新增“动作（action handler）”
|
||||
|
|
@ -65,101 +69,54 @@ MQTT 是流行的物联网数据传输协议，[EMQX](https://github.com/emqx/em
|
|||
|
||||
|
||||
|
||||
选择“发送数据到 Web 服务“并点击“新建资源”按钮：
|
||||
选择“发送数据到 Web 服务”并点击“新建资源”按钮：
|
||||
|
||||
### 编辑“资源（Resource）”
|
||||
|
||||
选择“发送数据到 Web 服务“并填写 请求 URL 为 运行 taosAdapter 的服务器地址和端口（默认为 6041）。其他属性请保持默认值。
|
||||
选择“WebHook”并填写“请求 URL”为 taosAdapter 提供 REST 服务的地址，如果是本地启动的 taosadapter， 那么默认地址为：
|
||||
|
||||
```
|
||||
http://127.0.0.1:6041/rest/sql
|
||||
```
|
||||
|
||||
其他属性请保持默认值。
|
||||
|
||||
|
||||
|
||||
### 编辑“动作（action）”
|
||||
|
||||
编辑资源配置，增加 Authorization 认证的键/值配对项，相关文档请参考[ TDengine REST API 文档](https://docs.taosdata.com/reference/rest-api/)。在消息体中输入规则引擎替换模板。
|
||||
编辑资源配置，增加 Authorization 认证的键/值配对项。默认用户名和密码对应的 Authorization 值为：
|
||||
```
|
||||
Basic cm9vdDp0YW9zZGF0YQ==
|
||||
```
|
||||
相关文档请参考[ TDengine REST API 文档](/reference/rest-api/)。
|
||||
|
||||
在消息体中输入规则引擎替换模板:
|
||||
|
||||
```sql
|
||||
INSERT INTO test.sensor_data VALUES(
|
||||
now,
|
||||
${payload.temperature},
|
||||
${payload.humidity},
|
||||
${payload.volume},
|
||||
${payload.PM10},
|
||||
${payload.pm25},
|
||||
${payload.SO2},
|
||||
${payload.NO2},
|
||||
${payload.CO},
|
||||
'${payload.id}',
|
||||
${payload.area},
|
||||
${payload.ts}
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
最后点击左下方的 “Create” 按钮，保存规则。
|
||||
## 编写模拟测试程序
|
||||
|
||||
```javascript
|
||||
// mock.js
|
||||
const mqtt = require('mqtt')
|
||||
const Mock = require('mockjs')
|
||||
const EMQX_SERVER = 'mqtt://localhost:1883'
|
||||
const CLIENT_NUM = 10
|
||||
const STEP = 5000 // 模拟采集时间间隔 ms
|
||||
const AWAIT = 5000 // 每次发送完后休眠时间，防止消息速率过快 ms
|
||||
const CLIENT_POOL = []
|
||||
startMock()
|
||||
function sleep(timer = 100) {
|
||||
return new Promise(resolve => {
|
||||
setTimeout(resolve, timer)
|
||||
})
|
||||
}
|
||||
async function startMock() {
|
||||
const now = Date.now()
|
||||
for (let i = 0; i < CLIENT_NUM; i++) {
|
||||
const client = await createClient(`mock_client_${i}`)
|
||||
CLIENT_POOL.push(client)
|
||||
}
|
||||
// last 24h every 5s
|
||||
const last = 24 * 3600 * 1000
|
||||
for (let ts = now - last; ts <= now; ts += STEP) {
|
||||
for (const client of CLIENT_POOL) {
|
||||
const mockData = generateMockData()
|
||||
const data = {
|
||||
...mockData,
|
||||
id: client.clientId,
|
||||
area: 0,
|
||||
ts,
|
||||
}
|
||||
client.publish('sensor/data', JSON.stringify(data))
|
||||
}
|
||||
const dateStr = new Date(ts).toLocaleTimeString()
|
||||
console.log(`${dateStr} send success.`)
|
||||
await sleep(AWAIT)
|
||||
}
|
||||
console.log(`Done, use ${(Date.now() - now) / 1000}s`)
|
||||
}
|
||||
/**
|
||||
* Init a virtual mqtt client
|
||||
* @param {string} clientId ClientID
|
||||
*/
|
||||
function createClient(clientId) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const client = mqtt.connect(EMQX_SERVER, {
|
||||
clientId,
|
||||
})
|
||||
client.on('connect', () => {
|
||||
console.log(`client ${clientId} connected`)
|
||||
resolve(client)
|
||||
})
|
||||
client.on('reconnect', () => {
|
||||
console.log('reconnect')
|
||||
})
|
||||
client.on('error', (e) => {
|
||||
console.error(e)
|
||||
reject(e)
|
||||
})
|
||||
})
|
||||
}
|
||||
/**
|
||||
* Generate mock data
|
||||
*/
|
||||
function generateMockData() {
|
||||
return {
|
||||
"temperature": parseFloat(Mock.Random.float(22, 100).toFixed(2)),
|
||||
"humidity": parseFloat(Mock.Random.float(12, 86).toFixed(2)),
|
||||
"volume": parseFloat(Mock.Random.float(20, 200).toFixed(2)),
|
||||
"PM10": parseFloat(Mock.Random.float(0, 300).toFixed(2)),
|
||||
"pm25": parseFloat(Mock.Random.float(0, 300).toFixed(2)),
|
||||
"SO2": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"NO2": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"CO": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"area": Mock.Random.integer(0, 20),
|
||||
"ts": 1596157444170,
|
||||
}
|
||||
}
|
||||
{{#include docs-examples/other/mock.js}}
|
||||
```
|
||||
|
||||
注意：代码中 CLIENT_NUM 在开始测试中可以先设置一个较小的值，避免硬件性能不能完全处理较大并发客户端数量。
|
||||
|
|
@ -189,4 +146,3 @@ node mock.js
|
|||
|
||||
TDengine 详细使用方法请参考 [TDengine 官方文档](https://docs.taosdata.com/)。
|
||||
EMQX 详细使用方法请参考 [EMQX 官方文档](https://www.emqx.io/docs/zh/v4.4/rule/rule-engine.html)。
|
||||
|
||||
|
|
|
|||
|
|
@ -74,7 +74,7 @@ title: 常见问题及反馈
|
|||
检查服务器侧 TCP 端口连接是否工作：`nc -l {port}`
|
||||
检查客户端侧 TCP 端口连接是否工作：`nc {hostIP} {port}`
|
||||
|
||||
- Windows 系统请使用 PowerShell 命令 Net-TestConnection -ComputerName {fqdn} -Port {port} 检测服务段端口是否访问
|
||||
- Windows 系统请使用 PowerShell 命令 Test-NetConnection -ComputerName {fqdn} -Port {port} 检测服务段端口是否访问
|
||||
|
||||
10. 也可以使用 taos 程序内嵌的网络连通检测功能，来验证服务器和客户端之间指定的端口连接是否通畅（包括 TCP 和 UDP）：[TDengine 内嵌网络检测工具使用指南](https://www.taosdata.com/blog/2020/09/08/1816.html)。
|
||||
|
||||
|
|
|
|||
|
|
@ -19,7 +19,7 @@ import InstallOnLinux from "../../14-reference/03-connector/\_windows_install.md
|
|||
import VerifyLinux from "../../14-reference/03-connector/\_verify_linux.mdx";
|
||||
import VerifyWindows from "../../14-reference/03-connector/\_verify_windows.mdx";
|
||||
|
||||
Any application programs running on any kind of platform can access TDengine through the REST API provided by TDengine. For details, please refer to [REST API](/reference/rest-api/). Additionally, application programs can use the connectors of multiple programming languages including C/C++, Java, Python, Go, Node.js, C#, and Rust to access TDengine. This chapter describes how to establish a connection to TDengine and briefly introduces how to install and use connectors. For details about the connectors, please refer to [Connectors](/reference/connector/)
|
||||
Any application programs running on any kind of platform can access TDengine through the REST API provided by TDengine. For details, please refer to [REST API](/reference/rest-api/). Additionally, application programs can use the connectors of multiple programming languages including C/C++, Java, Python, Go, Node.js, C#, Rust to access TDengine. This chapter describes how to establish a connection to TDengine and briefly introduces how to install and use connectors. TDengine community also provides connectors in LUA and PHP languages. For details about the connectors, please refer to [Connectors](/reference/connector/).
|
||||
|
||||
## Establish Connection
|
||||
|
||||
|
|
@ -200,6 +200,46 @@ install.packages("RJDBC")
|
|||
If the client driver (taosc) is already installed, then the C connector is already available.
|
||||
<br/>
|
||||
|
||||
</TabItem>
|
||||
<TabItem label="PHP" value="php">
|
||||
|
||||
**Download Source Code Package and Unzip：**
|
||||
|
||||
```shell
|
||||
curl -L -o php-tdengine.tar.gz https://github.com/Yurunsoft/php-tdengine/archive/refs/tags/v1.0.2.tar.gz \
|
||||
&& mkdir php-tdengine \
|
||||
&& tar -xzf php-tdengine.tar.gz -C php-tdengine --strip-components=1
|
||||
```
|
||||
|
||||
> Version number `v1.0.2` is only for example, it can be replaced to any newer version, please check available version from [TDengine PHP Connector Releases](https://github.com/Yurunsoft/php-tdengine/releases).
|
||||
|
||||
**Non-Swoole Environment：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure && make -j && make install
|
||||
```
|
||||
|
||||
**Specify TDengine Location：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/2.4.0.0 && make -j && make install
|
||||
```
|
||||
|
||||
> `--with-tdengine-dir=` is followed by the TDengine installation location.
|
||||
> This way is useful in case TDengine location can't be found automatically or macOS.
|
||||
|
||||
**Swoole Environment：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --enable-swoole && make -j && make install
|
||||
```
|
||||
|
||||
**Enable The Extension:**
|
||||
|
||||
Option One: Add `extension=tdengine` in `php.ini`
|
||||
|
||||
Option Two: Specify the extension on CLI `php -d extension=tdengine test.php`
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
|
|
|||
|
|
@ -26,7 +26,7 @@ Please refer to [list of supported platforms](/reference/connector#supported-pla
|
|||
|
||||
## Supported versions
|
||||
|
||||
The version number of the TDengine client driver and the version number of the TDengine server should be the same. A lower version of the client driver is compatible with a higher version of the server, if the first three version numbers are the same (i.e., only the fourth version number is different). For e.g. if the client version is x.y.z.1 and the server version is x.y.z.2 the client and server are compatible. But in general we do not recommend using a lower client version with a newer server version. It is also strongly discouraged to use a higher version of the client driver to access a lower version of the TDengine server.
|
||||
The version number of the TDengine client driver and the version number of the TDengine server should be same. A lower version of the client driver is compatible with a higher version of the server, if the first three version numbers are the same (i.e., only the fourth version number is different). For e.g. if the client version is x.y.z.1 and the server version is x.y.z.2 the client and server are compatible. But in general we do not recommend using a lower client version with a newer server version. It is also strongly discouraged to use a higher version of the client driver to access a lower version of the TDengine server.
|
||||
|
||||
## Installation steps
|
||||
|
||||
|
|
|
|||
|
|
@ -42,18 +42,18 @@ Please refer to [Version Support List](/reference/connector#version-support).
|
|||
TDengine currently supports timestamp, number, character, Boolean type, and the corresponding type conversion with Java is as follows:
|
||||
|
||||
| TDengine DataType | JDBCType (driver version < 2.0.24) | JDBCType (driver version > = 2.0.24) |
|
||||
| ----------------- | --------------------------------- | ---------------------------------- |
|
||||
| TIMESTAMP | java.lang.Long | java.sql.Timestamp |
|
||||
| INT | java.lang.Integer | java.lang.Integer |
|
||||
| BIGINT | java.lang.Long | java.lang.Long |
|
||||
| FLOAT | java.lang.Float | java.lang.Float |
|
||||
| DOUBLE | java.lang.Double | java.lang.Double |
|
||||
| SMALLINT | java.lang.Short | java.lang.Short |
|
||||
| TINYINT | java.lang.Byte | java.lang.Byte |
|
||||
| BOOL | java.lang.Boolean | java.lang.Boolean |
|
||||
| BINARY | java.lang.String | byte array |
|
||||
| NCHAR | java.lang.String | java.lang.String |
|
||||
| JSON | - | java.lang.String |
|
||||
| ----------------- | ---------------------------------- | ------------------------------------ |
|
||||
| TIMESTAMP | java.lang.Long | java.sql.Timestamp |
|
||||
| INT | java.lang.Integer | java.lang.Integer |
|
||||
| BIGINT | java.lang.Long | java.lang.Long |
|
||||
| FLOAT | java.lang.Float | java.lang.Float |
|
||||
| DOUBLE | java.lang.Double | java.lang.Double |
|
||||
| SMALLINT | java.lang.Short | java.lang.Short |
|
||||
| TINYINT | java.lang.Byte | java.lang.Byte |
|
||||
| BOOL | java.lang.Boolean | java.lang.Boolean |
|
||||
| BINARY | java.lang.String | byte array |
|
||||
| NCHAR | java.lang.String | java.lang.String |
|
||||
| JSON | - | java.lang.String |
|
||||
|
||||
**Note**: Only TAG supports JSON types
|
||||
|
||||
|
|
@ -91,8 +91,8 @@ Add following dependency in the `pom.xml` file of your Maven project:
|
|||
You can build Java connector from source code after cloning the TDengine project:
|
||||
|
||||
```shell
|
||||
git clone https://github.com/taosdata/TDengine.git
|
||||
cd TDengine/src/connector/jdbc
|
||||
git clone https://github.com/taosdata/taos-connector-jdbc.git
|
||||
cd taos-connector-jdbc
|
||||
mvn clean install -Dmaven.test.skip=true
|
||||
```
|
||||
|
||||
|
|
@ -197,6 +197,7 @@ The configuration parameters in the URL are as follows.
|
|||
- user: Login TDengine user name, default value 'root'.
|
||||
- password: user login password, default value 'taosdata'.
|
||||
- batchfetch: true: pull the result set in batch when executing the query; false: pull the result set row by row. The default value is false. batchfetch uses HTTP for data transfer. The JDBC REST connection supports bulk data pulling function in taos-jdbcdriver-2.0.38 and TDengine 2.4.0.12 and later versions. taos-jdbcdriver and TDengine transfer data via WebSocket connection. Compared with HTTP, WebSocket enables JDBC REST connection to support large data volume querying and improve query performance.
|
||||
- charset: specify the charset to parse the string, this parameter is valid only when set batchfetch to true.
|
||||
- batchErrorIgnore: true: when executing executeBatch of Statement, if one SQL execution fails in the middle, continue to execute the following SQL. false: no longer execute any statement after the failed SQL. The default value is: false.
|
||||
|
||||
**Note**: Some configuration items (e.g., locale, timezone) do not work in the REST connection.
|
||||
|
|
@ -258,10 +259,10 @@ The configuration parameters in properties are as follows.
|
|||
- TSDBDriver.PROPERTY_KEY_BATCH_LOAD: true: pull the result set in batch when executing query; false: pull the result set row by row. The default value is: false.
|
||||
- TSDBDriver.PROPERTY_KEY_BATCH_ERROR_IGNORE: true: when executing executeBatch of Statement, if there is a SQL execution failure in the middle, continue to execute the following sq. false: no longer execute any statement after the failed SQL. The default value is: false.
|
||||
- TSDBDriver.PROPERTY_KEY_CONFIG_DIR: Only works when using JDBC native connection. Client configuration file directory path, default value `/etc/taos` on Linux OS, default value `C:/TDengine/cfg` on Windows OS.
|
||||
- TSDBDriver.PROPERTY_KEY_CHARSET: takes effect only when using JDBC native connection. In the character set used by the client, the default value is the system character set.
|
||||
- TSDBDriver.PROPERTY_KEY_CHARSET: In the character set used by the client, the default value is the system character set.
|
||||
- TSDBDriver.PROPERTY_KEY_LOCALE: this only takes effect when using JDBC native connection. Client language environment, the default value is system current locale.
|
||||
- TSDBDriver.PROPERTY_KEY_TIME_ZONE: only takes effect when using JDBC native connection. In the time zone used by the client, the default value is the system's current time zone.
|
||||
For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](/reference/config/#Client-Only).
|
||||
For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](/reference/config/#Client-Only).
|
||||
|
||||
### Priority of configuration parameters
|
||||
|
||||
|
|
@ -350,7 +351,7 @@ There are three types of error codes that the JDBC connector can report:
|
|||
|
||||
For specific error codes, please refer to.
|
||||
|
||||
- [TDengine Java Connector](https://github.com/taosdata/TDengine/blob/develop/src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBErrorNumbers.java)
|
||||
- [TDengine Java Connector](https://github.com/taosdata/taos-connector-jdbc/blob/main/src/main/java/com/taosdata/jdbc/TSDBErrorNumbers.java)
|
||||
- [TDengine_ERROR_CODE](https://github.com/taosdata/TDengine/blob/develop/src/inc/taoserror.h)
|
||||
|
||||
### Writing data via parameter binding
|
||||
|
|
@ -808,11 +809,11 @@ Please refer to: [JDBC example](https://github.com/taosdata/TDengine/tree/develo
|
|||
|
||||
## Recent update logs
|
||||
|
||||
| taos-jdbcdriver version | major changes |
|
||||
| :------------------: | :----------------------------: |
|
||||
| 2.0.38 | JDBC REST connections add bulk pull function |
|
||||
| 2.0.37 | Added support for json tags |
|
||||
| 2.0.36 | Add support for schemaless writing |
|
||||
| taos-jdbcdriver version | major changes |
|
||||
| :---------------------: | :------------------------------------------: |
|
||||
| 2.0.38 | JDBC REST connections add bulk pull function |
|
||||
| 2.0.37 | Added support for json tags |
|
||||
| 2.0.36 | Add support for schemaless writing |
|
||||
|
||||
## Frequently Asked Questions
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,150 @@
|
|||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: PHP
|
||||
title: PHP Connector
|
||||
---
|
||||
|
||||
`php-tdengine` is the TDengine PHP connector provided by TDengine community. In particular, it supports Swoole coroutine.
|
||||
|
||||
PHP Connector relies on TDengine client driver.
|
||||
|
||||
Project Repository：<https://github.com/Yurunsoft/php-tdengine>
|
||||
|
||||
After TDengine client or server is installed, `taos.h` is located at:
|
||||
|
||||
- Linux：`/usr/local/taos/include`
|
||||
- Windows：`C:\TDengine\include`
|
||||
|
||||
TDengine client driver is located at:
|
||||
|
||||
- Linux: `/usr/local/taos/driver/libtaos.so`
|
||||
- Windows: `C:\TDengine\taos.dll`
|
||||
|
||||
## Supported Platforms
|
||||
|
||||
- Windows、Linux、MacOS
|
||||
|
||||
- PHP >= 7.4
|
||||
|
||||
- TDengine >= 2.0
|
||||
|
||||
- Swoole >= 4.8 (Optional)
|
||||
|
||||
## Supported Versions
|
||||
|
||||
Because the version of TDengine client driver is tightly associated with that of TDengine server, it's strongly suggested to use the client driver of same version as TDengine server, even though the client driver can work with TDengine server if the first 3 sections of the versions are same.
|
||||
|
||||
## Installation
|
||||
|
||||
### Install TDengine Client Driver
|
||||
|
||||
Regarding how to install TDengine client driver please refer to [Install Client Driver](/reference/connector#installation-steps)
|
||||
|
||||
### Install php-tdengine
|
||||
|
||||
**Download Source Code Package and Unzip：**
|
||||
|
||||
```shell
|
||||
curl -L -o php-tdengine.tar.gz https://github.com/Yurunsoft/php-tdengine/archive/refs/tags/v1.0.2.tar.gz \
|
||||
&& mkdir php-tdengine \
|
||||
&& tar -xzf php-tdengine.tar.gz -C php-tdengine --strip-components=1
|
||||
```
|
||||
|
||||
> Version number `v1.0.2` is only for example, it can be replaced to any newer version, please find available versions in [TDengine PHP Connector Releases](https://github.com/Yurunsoft/php-tdengine/releases).
|
||||
|
||||
**Non-Swoole Environment：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure && make -j && make install
|
||||
```
|
||||
|
||||
**Specify TDengine location：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/2.4.0.0 && make -j && make install
|
||||
```
|
||||
|
||||
> `--with-tdengine-dir=` is followed by TDengine location.
|
||||
> It's useful in case TDengine installatio location can't be found automatically or MacOS.
|
||||
|
||||
**Swoole Environment：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --enable-swoole && make -j && make install
|
||||
```
|
||||
|
||||
**Enable Extension:**
|
||||
|
||||
Option One: Add `extension=tdengine` in `php.ini`.
|
||||
|
||||
Option Two: Use CLI `php -dextension=tdengine test.php`.
|
||||
|
||||
## Sample Programs
|
||||
|
||||
In this section a few sample programs which use TDengine PHP connector to access TDengine cluster are demonstrated.
|
||||
|
||||
> Any error would throw exception: `TDengine\Exception\TDengineException`
|
||||
|
||||
### Establish Conection
|
||||
|
||||
<details>
|
||||
<summary>Establish Connection</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/connect.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Insert Data
|
||||
|
||||
<details>
|
||||
<summary>Insert Data</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/insert.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Synchronous Query
|
||||
|
||||
<details>
|
||||
<summary>Synchronous Query</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/query.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
### Parameter Binding
|
||||
|
||||
<details>
|
||||
<summary>Parameter Binding</summary>
|
||||
|
||||
```c
|
||||
{{#include docs-examples/php/insert_stmt.php}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
## Constants
|
||||
|
||||
| Constant | Description |
|
||||
| ----------------------------------- | ----------- |
|
||||
| `TDengine\TSDB_DATA_TYPE_NULL` | null |
|
||||
| `TDengine\TSDB_DATA_TYPE_BOOL` | bool |
|
||||
| `TDengine\TSDB_DATA_TYPE_TINYINT` | tinyint |
|
||||
| `TDengine\TSDB_DATA_TYPE_SMALLINT` | smallint |
|
||||
| `TDengine\TSDB_DATA_TYPE_INT` | int |
|
||||
| `TDengine\TSDB_DATA_TYPE_BIGINT` | bigint |
|
||||
| `TDengine\TSDB_DATA_TYPE_FLOAT` | float |
|
||||
| `TDengine\TSDB_DATA_TYPE_DOUBLE` | double |
|
||||
| `TDengine\TSDB_DATA_TYPE_BINARY` | binary |
|
||||
| `TDengine\TSDB_DATA_TYPE_TIMESTAMP` | timestamp |
|
||||
| `TDengine\TSDB_DATA_TYPE_NCHAR` | nchar |
|
||||
| `TDengine\TSDB_DATA_TYPE_UTINYINT` | utinyint |
|
||||
| `TDengine\TSDB_DATA_TYPE_USMALLINT` | usmallint |
|
||||
| `TDengine\TSDB_DATA_TYPE_UINT` | uint |
|
||||
| `TDengine\TSDB_DATA_TYPE_UBIGINT` | ubigint |
|
||||
|
|
@ -199,10 +199,9 @@ The `connect()` function returns a `taos.TaosConnection` instance. In client-sid
|
|||
|
||||
All arguments to the `connect()` function are optional keyword arguments. The following are the connection parameters specified.
|
||||
|
||||
- `host`: The host to connect to. The default is localhost.
|
||||
- `url`： The URL of taosAdapter REST service. The default is <http://localhost:6041>.
|
||||
- `user`: TDengine user name. The default is `root`.
|
||||
- `password`: TDengine user password. The default is `taosdata`.
|
||||
- `port`: The port on which the taosAdapter REST service listens. Default is 6041.
|
||||
- `timeout`: HTTP request timeout in seconds. The default is `socket._GLOBAL_DEFAULT_TIMEOUT`. Usually, no configuration is needed.
|
||||
|
||||
</TabItem>
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ There are two ways to install taosBenchmark:
|
|||
|
||||
### Configuration and running methods
|
||||
|
||||
taosBenchmark supports two configuration methods: [Command-line arguments](#Command-line arguments in detailed) and [JSON configuration file](#Configuration file arguments in detailed). These two methods are mutually exclusive. Users can use `-f <json file>` to specify a configuration file. When running taosBenchmark with command-line arguments to control its behavior, users should use other parameters for configuration, but not the `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters.
|
||||
TaosBenchmark needs to be executed on the terminal of the operating system, it supports two configuration methods: [Command-line arguments](#Command-line arguments in detailed) and [JSON configuration file](#Configuration file arguments in detailed). These two methods are mutually exclusive. Users can use `-f <json file>` to specify a configuration file. When running taosBenchmark with command-line arguments to control its behavior, users should use other parameters for configuration, but not the `-f` parameter. In addition, taosBenchmark offers a special way of running without parameters.
|
||||
|
||||
taosBenchmark supports complete performance testing of TDengine. taosBenchmark supports the TDengine functions in three categories: write, query, and subscribe. These three functions are mutually exclusive, and users can select only one of them each time taosBenchmark runs. It is important to note that the type of functionality to be tested is not configurable when using the command-line configuration method, which can only test writing performance. To test the query and subscription performance of the TDengine, you must use the configuration file method and specify the function type to test via the parameter `filetype` in the configuration file.
|
||||
|
||||
|
|
|
|||
|
|
@ -134,7 +134,7 @@ TDengine uses 13 continuous ports, both TCP and UDP, starting with the port spec
|
|||
| Applicable | Server Only |
|
||||
| Meaning | The switch for monitoring inside server. The workload of the hosts, including CPU, memory, disk, network, TTP requests, are collected and stored in a system builtin database `LOG` |
|
||||
| Value Range | 0: monitoring disabled, 1: monitoring enabled |
|
||||
| Default Value | 0 |
|
||||
| Default Value | 1 |
|
||||
|
||||
### monitorInterval
|
||||
|
||||
|
|
|
|||
|
|
@ -14,12 +14,40 @@ In order for Grafana to add the TDengine data source successfully, the following
|
|||
1. The TDengine cluster is deployed and functioning properly
|
||||
2. taosAdapter is installed and running properly. Please refer to the taosAdapter manual for details.
|
||||
|
||||
Record these values:
|
||||
|
||||
- TDengine REST API url: `http://tdengine.local:6041`.
|
||||
- TDengine cluster authorization, with user + password.
|
||||
|
||||
## Installing Grafana
|
||||
|
||||
TDengine currently supports Grafana versions 7.0 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: <https://grafana.com/grafana/download>.
|
||||
TDengine currently supports Grafana versions 7.5 and above. Users can go to the Grafana official website to download the installation package and execute the installation according to the current operating system. The download address is as follows: <https://grafana.com/grafana/download>.
|
||||
|
||||
## Configuring Grafana
|
||||
|
||||
### Option 1: Install with `install.sh`
|
||||
|
||||
Set the url and authorization environment variables by `export` or a [`.env`(dotenv) file](https://hexdocs.pm/dotenvy/dotenv-file-format.html):
|
||||
|
||||
```sh
|
||||
export TDENGINE_API=http://tdengine.local:6041
|
||||
# user + password
|
||||
export TDENGINE_USER=user
|
||||
export TDENGINE_PASSWORD=password
|
||||
```
|
||||
|
||||
Run `install.sh`:
|
||||
|
||||
```sh
|
||||
bash -c "$(curl -fsSL https://raw.githubusercontent.com/taosdata/grafanaplugin/master/install.sh)"
|
||||
```
|
||||
|
||||
With this script, TDengine data source plugin and the Grafana data source will be installed and created automatically with Grafana provisioning configurations.
|
||||
|
||||
And then, restart Grafana service and open Grafana in web-browser, usually <http://localhost:3000>.
|
||||
|
||||
### Option 2: Install Plugin Manually
|
||||
|
||||
Follow the installation steps in [Grafana](https://grafana.com/grafana/plugins/tdengine-datasource/?tab=installation) with the [``grafana-cli`` command-line tool](https://grafana.com/docs/grafana/latest/administration/cli/) for plugin installation.
|
||||
|
||||
```bash
|
||||
|
|
@ -50,11 +78,7 @@ If Grafana is running in a Docker environment, the TDengine plugin can be automa
|
|||
GF_INSTALL_PLUGINS=tdengine-datasource
|
||||
```
|
||||
|
||||
## Using Grafana
|
||||
|
||||
### Configuring Data Sources
|
||||
|
||||
Users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure.
|
||||
Now users can log in to the Grafana server (username/password: admin/admin) directly through the URL `http://localhost:3000` and add a datasource through `Configuration -> Data Sources` on the left side, as shown in the following figure.
|
||||
|
||||
|
||||
|
||||
|
|
@ -94,4 +118,11 @@ Follow the default prompt to query the average system memory usage for the speci
|
|||
|
||||
### Importing the Dashboard
|
||||
|
||||
In version 2.3.3.0 and above, you can import the TDinsight Dashboard (Grafana Dashboard ID: [15168](https://grafana.com/grafana/dashboards/15167)) as a monitoring visualization tool for TDengine clusters. You can find installation and usage instructions in the TDinsight User Manual (/reference/tdinsight/).
|
||||
You can install TDinsight dashboard in data source configuration page (like `http://localhost:3000/datasources/edit/1/dashboards`) as a monitoring visualization tool for TDengine cluster. The dashboard is published in Grafana as [Dashboard 15167 - TDinsight](https://grafana.com/grafana/dashboards/15167). Check the [TDinsight User Manual](/reference/tdinsight/) for the details.
|
||||
|
||||
For more dashboards using TDengine data source, [search here in Grafana](https://grafana.com/grafana/dashboards/?dataSource=tdengine-datasource). Here is a sub list:
|
||||
|
||||
- [15146](https://grafana.com/grafana/dashboards/15146): Monitor multiple TDengine clusters.
|
||||
- [15155](https://grafana.com/grafana/dashboards/15155): TDengine alert demo.
|
||||
- [15167](https://grafana.com/grafana/dashboards/15167): TDinsight.
|
||||
- [16388](https://grafana.com/grafana/dashboards/16388): Telegraf node metrics dashboard using TDengine data source.
|
||||
|
|
|
|||
|
|
@ -16,22 +16,15 @@ The following preparations are required for EMQX to add TDengine data sources co
|
|||
|
||||
Depending on the current operating system, users can download the installation package from the [EMQX official website](https://www.emqx.io/downloads) and execute the installation. After installation, use `sudo emqx start` or `sudo systemctl start emqx` to start the EMQX service.
|
||||
|
||||
## Create the appropriate database and table schema in TDengine for receiving MQTT data
|
||||
|
||||
### Take the Docker installation of TDengine as an example
|
||||
## Create Database and Table
|
||||
|
||||
```bash
|
||||
docker exec -it tdengine bash
|
||||
taos
|
||||
```
|
||||
|
||||
### Create Database and Table
|
||||
In this step we create the appropriate database and table schema in TDengine for receiving MQTT data. Open TDengine CLI and execute SQL bellow:
|
||||
|
||||
```sql
|
||||
CREATE DATABASE test;
|
||||
USE test;
|
||||
|
||||
CREATE TABLE sensor_data (ts timestamp, temperature float, humidity float, volume float, PM10 float, pm25 float, SO2 float, NO2 float, CO float, sensor_id NCHAR(255), area TINYINT, coll_time timestamp);
|
||||
CREATE DATABASE test;
|
||||
USE test;
|
||||
CREATE TABLE sensor_data (ts TIMESTAMP, temperature FLOAT, humidity FLOAT, volume FLOAT, pm10 FLOAT, pm25 FLOAT, so2 FLOAT, no2 FLOAT, co FLOAT, sensor_id NCHAR(255), area TINYINT, coll_time TIMESTAMP);
|
||||
```
|
||||
|
||||
Note: The table schema is based on the blog [(In Chinese) Data Transfer, Storage, Presentation, EMQX + TDengine Build MQTT IoT Data Visualization Platform](https://www.taosdata.com/blog/2020/08/04/1722.html) as an example. Subsequent operations are carried out with this blog scenario too. Please modify it according to your actual application scenario.
|
||||
|
|
@ -54,6 +47,15 @@ Select "Rule" in the "Rule Engine" on the left and click the "Create" button: !
|
|||
|
||||
### Edit SQL fields
|
||||
|
||||
Copy SQL bellow and paste it to the SQL edit area：
|
||||
|
||||
```sql
|
||||
SELECT
|
||||
payload
|
||||
FROM
|
||||
"sensor/data"
|
||||
```
|
||||
|
||||
|
||||
|
||||
### Add "action handler"
|
||||
|
|
@ -68,97 +70,45 @@ Select "Data to Web Service" and click the "New Resource" button.
|
|||
|
||||
### Edit "Resource"
|
||||
|
||||
Select "Data to Web Service" and fill in the request URL as the address and port of the server running taosAdapter (default is 6041). Leave the other properties at their default values.
|
||||
Select "WebHook" and fill in the request URL as the address and port of the server running taosAdapter (default is 6041). Leave the other properties at their default values.
|
||||
|
||||
|
||||
|
||||
### Edit "action"
|
||||
|
||||
Edit the resource configuration to add the key/value pairing for Authorization. Please refer to the [ TDengine REST API documentation ](https://docs.taosdata.com/reference/rest-api/) for the authorization in details. Enter the rule engine replacement template in the message body.
|
||||
Edit the resource configuration to add the key/value pairing for Authorization. If you use the default TDengine username and password then the value of key Authorization is：
|
||||
```
|
||||
Basic cm9vdDp0YW9zZGF0YQ==
|
||||
```
|
||||
|
||||
Please refer to the [ TDengine REST API documentation ](/reference/rest-api/) for the authorization in details.
|
||||
|
||||
Enter the rule engine replacement template in the message body:
|
||||
|
||||
```sql
|
||||
INSERT INTO test.sensor_data VALUES(
|
||||
now,
|
||||
${payload.temperature},
|
||||
${payload.humidity},
|
||||
${payload.volume},
|
||||
${payload.PM10},
|
||||
${payload.pm25},
|
||||
${payload.SO2},
|
||||
${payload.NO2},
|
||||
${payload.CO},
|
||||
'${payload.id}',
|
||||
${payload.area},
|
||||
${payload.ts}
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
|
||||
Finally, click the "Create" button at bottom left corner saving the rule.
|
||||
## Compose program to mock data
|
||||
|
||||
```javascript
|
||||
// mock.js
|
||||
const mqtt = require('mqtt')
|
||||
const Mock = require('mockjs')
|
||||
const EMQX_SERVER = 'mqtt://localhost:1883'
|
||||
const CLIENT_NUM = 10
|
||||
const STEP = 5000 // Data interval in ms
|
||||
const AWAIT = 5000 // Sleep time after data be written once to avoid data writing too fast
|
||||
const CLIENT_POOL = []
|
||||
startMock()
|
||||
function sleep(timer = 100) {
|
||||
return new Promise(resolve => {
|
||||
setTimeout(resolve, timer)
|
||||
})
|
||||
}
|
||||
async function startMock() {
|
||||
const now = Date.now()
|
||||
for (let i = 0; i < CLIENT_NUM; i++) {
|
||||
const client = await createClient(`mock_client_${i}`)
|
||||
CLIENT_POOL.push(client)
|
||||
}
|
||||
// last 24h every 5s
|
||||
const last = 24 * 3600 * 1000
|
||||
for (let ts = now - last; ts <= now; ts += STEP) {
|
||||
for (const client of CLIENT_POOL) {
|
||||
const mockData = generateMockData()
|
||||
const data = {
|
||||
...mockData,
|
||||
id: client.clientId,
|
||||
area: 0,
|
||||
ts,
|
||||
}
|
||||
client.publish('sensor/data', JSON.stringify(data))
|
||||
}
|
||||
const dateStr = new Date(ts).toLocaleTimeString()
|
||||
console.log(`${dateStr} send success.`)
|
||||
await sleep(AWAIT)
|
||||
}
|
||||
console.log(`Done, use ${(Date.now() - now) / 1000}s`)
|
||||
}
|
||||
/**
|
||||
* Init a virtual mqtt client
|
||||
* @param {string} clientId ClientID
|
||||
*/
|
||||
function createClient(clientId) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const client = mqtt.connect(EMQX_SERVER, {
|
||||
clientId,
|
||||
})
|
||||
client.on('connect', () => {
|
||||
console.log(`client ${clientId} connected`)
|
||||
resolve(client)
|
||||
})
|
||||
client.on('reconnect', () => {
|
||||
console.log('reconnect')
|
||||
})
|
||||
client.on('error', (e) => {
|
||||
console.error(e)
|
||||
reject(e)
|
||||
})
|
||||
})
|
||||
}
|
||||
/**
|
||||
* Generate mock data
|
||||
*/
|
||||
function generateMockData() {
|
||||
return {
|
||||
"temperature": parseFloat(Mock.Random.float(22, 100).toFixed(2)),
|
||||
"humidity": parseFloat(Mock.Random.float(12, 86).toFixed(2)),
|
||||
"volume": parseFloat(Mock.Random.float(20, 200).toFixed(2)),
|
||||
"PM10": parseFloat(Mock.Random.float(0, 300).toFixed(2)),
|
||||
"pm25": parseFloat(Mock.Random.float(0, 300).toFixed(2)),
|
||||
"SO2": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"NO2": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"CO": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"area": Mock.Random.integer(0, 20),
|
||||
"ts": 1596157444170,
|
||||
}
|
||||
}
|
||||
{{#include docs-examples/other/mock.js}}
|
||||
```
|
||||
|
||||
Note: `CLIENT_NUM` in the code can be set to a smaller value at the beginning of the test to avoid hardware performance be not capable to handle a more significant number of concurrent clients.
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ When the client is unable to connect to the server, you can try the following wa
|
|||
Check whether a TCP port on server side is open: `nc -l {port}`
|
||||
Check whether a TCP port on client side is open: `nc {hostIP} {port}`
|
||||
|
||||
- On Windows system `Net-TestConnection -ComputerName {fqdn} -Port {port}` on PowerShell can be used to check whether the port on server side is open for access.
|
||||
- On Windows system `Test-NetConnection -ComputerName {fqdn} -Port {port}` on PowerShell can be used to check whether the port on server side is open for access.
|
||||
|
||||
7. TDengine CLI `taos` can also be used to check network, please refer to [TDengine CLI](/reference/taos-shell).
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,78 @@
|
|||
// mock.js
|
||||
const mqtt = require('mqtt')
|
||||
const Mock = require('mockjs')
|
||||
const EMQX_SERVER = 'mqtt://localhost:1883'
|
||||
const CLIENT_NUM = 10
|
||||
const STEP = 5000 // Data interval in ms
|
||||
const AWAIT = 5000 // Sleep time after data be written once to avoid data writing too fast
|
||||
const CLIENT_POOL = []
|
||||
startMock()
|
||||
function sleep(timer = 100) {
|
||||
return new Promise(resolve => {
|
||||
setTimeout(resolve, timer)
|
||||
})
|
||||
}
|
||||
async function startMock() {
|
||||
const now = Date.now()
|
||||
for (let i = 0; i < CLIENT_NUM; i++) {
|
||||
const client = await createClient(`mock_client_${i}`)
|
||||
CLIENT_POOL.push(client)
|
||||
}
|
||||
// last 24h every 5s
|
||||
const last = 24 * 3600 * 1000
|
||||
for (let ts = now - last; ts <= now; ts += STEP) {
|
||||
for (const client of CLIENT_POOL) {
|
||||
const mockData = generateMockData()
|
||||
const data = {
|
||||
...mockData,
|
||||
id: client.clientId,
|
||||
area: 0,
|
||||
ts,
|
||||
}
|
||||
client.publish('sensor/data', JSON.stringify(data))
|
||||
}
|
||||
const dateStr = new Date(ts).toLocaleTimeString()
|
||||
console.log(`${dateStr} send success.`)
|
||||
await sleep(AWAIT)
|
||||
}
|
||||
console.log(`Done, use ${(Date.now() - now) / 1000}s`)
|
||||
}
|
||||
/**
|
||||
* Init a virtual mqtt client
|
||||
* @param {string} clientId ClientID
|
||||
*/
|
||||
function createClient(clientId) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const client = mqtt.connect(EMQX_SERVER, {
|
||||
clientId,
|
||||
})
|
||||
client.on('connect', () => {
|
||||
console.log(`client ${clientId} connected`)
|
||||
resolve(client)
|
||||
})
|
||||
client.on('reconnect', () => {
|
||||
console.log('reconnect')
|
||||
})
|
||||
client.on('error', (e) => {
|
||||
console.error(e)
|
||||
reject(e)
|
||||
})
|
||||
})
|
||||
}
|
||||
/**
|
||||
* Generate mock data
|
||||
*/
|
||||
function generateMockData() {
|
||||
return {
|
||||
"temperature": parseFloat(Mock.Random.float(22, 100).toFixed(2)),
|
||||
"humidity": parseFloat(Mock.Random.float(12, 86).toFixed(2)),
|
||||
"volume": parseFloat(Mock.Random.float(20, 200).toFixed(2)),
|
||||
"PM10": parseFloat(Mock.Random.float(0, 300).toFixed(2)),
|
||||
"pm25": parseFloat(Mock.Random.float(0, 300).toFixed(2)),
|
||||
"SO2": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"NO2": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"CO": parseFloat(Mock.Random.float(0, 50).toFixed(2)),
|
||||
"area": Mock.Random.integer(0, 20),
|
||||
"ts": 1596157444170,
|
||||
}
|
||||
}
|
||||
|
|
@ -22,7 +22,7 @@ try {
|
|||
|
||||
// set table name and tags
|
||||
$stmt->setTableNameTags('d1001', [
|
||||
// 支持格式同参数绑定
|
||||
// same format as parameter binding
|
||||
[TDengine\TSDB_DATA_TYPE_BINARY, 'California.SanFrancisco'],
|
||||
[TDengine\TSDB_DATA_TYPE_INT, 2],
|
||||
]);
|
||||
|
|
|
|||
|
|
@ -1,10 +1,9 @@
|
|||
# ANCHOR: connect
|
||||
from taosrest import connect, TaosRestConnection, TaosRestCursor
|
||||
|
||||
conn: TaosRestConnection = connect(host="localhost",
|
||||
conn: TaosRestConnection = connect(url="http://localhost:6041",
|
||||
user="root",
|
||||
password="taosdata",
|
||||
port=6041,
|
||||
timeout=30)
|
||||
|
||||
# ANCHOR_END: connect
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
from taosrest import RestClient
|
||||
|
||||
client = RestClient("localhost", 6041, "root", "taosdata")
|
||||
client = RestClient("http://localhost:6041", user="root", password="taosdata")
|
||||
res: dict = client.sql("SELECT ts, current FROM power.meters LIMIT 1")
|
||||
print(res)
|
||||
|
||||
|
|
|
|||
|
|
@ -25,7 +25,7 @@ int32_t init_env() {
|
|||
return -1;
|
||||
}
|
||||
|
||||
TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 1");
|
||||
TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 2");
|
||||
if (taos_errno(pRes) != 0) {
|
||||
printf("error in create db, reason:%s\n", taos_errstr(pRes));
|
||||
return -1;
|
||||
|
|
@ -82,7 +82,7 @@ int32_t create_stream() {
|
|||
/*const char* sql = "select sum(k) from tu1 interval(10m)";*/
|
||||
/*pRes = tmq_create_stream(pConn, "stream1", "out1", sql);*/
|
||||
pRes = taos_query(
|
||||
pConn, "create stream stream1 trigger at_once into outstb as select _wstartts, sum(k) from tu1 interval(10m)");
|
||||
pConn, "create stream stream1 trigger at_once into outstb as select _wstartts, sum(k) from st1 interval(10m)");
|
||||
if (taos_errno(pRes) != 0) {
|
||||
printf("failed to create stream stream1, reason:%s\n", taos_errstr(pRes));
|
||||
return -1;
|
||||
|
|
|
|||
|
|
@ -103,10 +103,10 @@ typedef void (*__taos_async_fn_t)(void *param, TAOS_RES *, int code);
|
|||
|
||||
typedef struct TAOS_MULTI_BIND {
|
||||
int buffer_type;
|
||||
void * buffer;
|
||||
void *buffer;
|
||||
uintptr_t buffer_length;
|
||||
int32_t * length;
|
||||
char * is_null;
|
||||
int32_t *length;
|
||||
char *is_null;
|
||||
int num;
|
||||
} TAOS_MULTI_BIND;
|
||||
|
||||
|
|
@ -130,7 +130,7 @@ DLL_EXPORT void taos_cleanup(void);
|
|||
DLL_EXPORT int taos_options(TSDB_OPTION option, const void *arg, ...);
|
||||
DLL_EXPORT setConfRet taos_set_config(const char *config);
|
||||
DLL_EXPORT int taos_init(void);
|
||||
DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port);
|
||||
DLL_EXPORT TAOS *taos_connect(const char *ip, const char *user, const char *pass, const char *db, uint16_t port);
|
||||
DLL_EXPORT TAOS *taos_connect_l(const char *ip, int ipLen, const char *user, int userLen, const char *pass, int passLen,
|
||||
const char *db, int dbLen, uint16_t port);
|
||||
DLL_EXPORT TAOS *taos_connect_auth(const char *ip, const char *user, const char *auth, const char *db, uint16_t port);
|
||||
|
|
@ -147,17 +147,17 @@ DLL_EXPORT int taos_stmt_set_sub_tbname(TAOS_STMT *stmt, const char *name
|
|||
DLL_EXPORT int taos_stmt_get_tag_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
|
||||
DLL_EXPORT int taos_stmt_get_col_fields(TAOS_STMT *stmt, int *fieldNum, TAOS_FIELD_E **fields);
|
||||
|
||||
DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert);
|
||||
DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums);
|
||||
DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes);
|
||||
DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
|
||||
DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
|
||||
DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx);
|
||||
DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt);
|
||||
DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt);
|
||||
DLL_EXPORT int taos_stmt_is_insert(TAOS_STMT *stmt, int *insert);
|
||||
DLL_EXPORT int taos_stmt_num_params(TAOS_STMT *stmt, int *nums);
|
||||
DLL_EXPORT int taos_stmt_get_param(TAOS_STMT *stmt, int idx, int *type, int *bytes);
|
||||
DLL_EXPORT int taos_stmt_bind_param(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
|
||||
DLL_EXPORT int taos_stmt_bind_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind);
|
||||
DLL_EXPORT int taos_stmt_bind_single_param_batch(TAOS_STMT *stmt, TAOS_MULTI_BIND *bind, int colIdx);
|
||||
DLL_EXPORT int taos_stmt_add_batch(TAOS_STMT *stmt);
|
||||
DLL_EXPORT int taos_stmt_execute(TAOS_STMT *stmt);
|
||||
DLL_EXPORT TAOS_RES *taos_stmt_use_result(TAOS_STMT *stmt);
|
||||
DLL_EXPORT int taos_stmt_close(TAOS_STMT *stmt);
|
||||
DLL_EXPORT char * taos_stmt_errstr(TAOS_STMT *stmt);
|
||||
DLL_EXPORT char *taos_stmt_errstr(TAOS_STMT *stmt);
|
||||
DLL_EXPORT int taos_stmt_affected_rows(TAOS_STMT *stmt);
|
||||
DLL_EXPORT int taos_stmt_affected_rows_once(TAOS_STMT *stmt);
|
||||
|
||||
|
|
@ -179,11 +179,11 @@ DLL_EXPORT bool taos_is_update_query(TAOS_RES *res);
|
|||
DLL_EXPORT int taos_fetch_block(TAOS_RES *res, TAOS_ROW *rows);
|
||||
DLL_EXPORT int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows);
|
||||
DLL_EXPORT int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData);
|
||||
DLL_EXPORT int * taos_get_column_data_offset(TAOS_RES *res, int columnIndex);
|
||||
DLL_EXPORT int *taos_get_column_data_offset(TAOS_RES *res, int columnIndex);
|
||||
DLL_EXPORT int taos_validate_sql(TAOS *taos, const char *sql);
|
||||
DLL_EXPORT void taos_reset_current_db(TAOS *taos);
|
||||
|
||||
DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res);
|
||||
DLL_EXPORT int *taos_fetch_lengths(TAOS_RES *res);
|
||||
DLL_EXPORT TAOS_ROW *taos_result_block(TAOS_RES *res);
|
||||
|
||||
DLL_EXPORT const char *taos_get_server_info(TAOS *taos);
|
||||
|
|
@ -194,6 +194,7 @@ DLL_EXPORT int taos_errno(TAOS_RES *tres);
|
|||
|
||||
DLL_EXPORT void taos_query_a(TAOS *taos, const char *sql, __taos_async_fn_t fp, void *param);
|
||||
DLL_EXPORT void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param);
|
||||
DLL_EXPORT void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param);
|
||||
|
||||
// Shuduo: temporary enable for app build
|
||||
#if 1
|
||||
|
|
@ -204,17 +205,17 @@ DLL_EXPORT TAOS_RES *taos_consume(TAOS_SUB *tsub);
|
|||
DLL_EXPORT void taos_unsubscribe(TAOS_SUB *tsub, int keepProgress);
|
||||
#endif
|
||||
|
||||
DLL_EXPORT int taos_load_table_info(TAOS *taos, const char *tableNameList);
|
||||
DLL_EXPORT int taos_load_table_info(TAOS *taos, const char *tableNameList);
|
||||
DLL_EXPORT TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLines, int protocol, int precision);
|
||||
|
||||
/* --------------------------TMQ INTERFACE------------------------------- */
|
||||
|
||||
enum tmq_resp_err_t {
|
||||
enum {
|
||||
TMQ_RESP_ERR__FAIL = -1,
|
||||
TMQ_RESP_ERR__SUCCESS = 0,
|
||||
};
|
||||
|
||||
typedef enum tmq_resp_err_t tmq_resp_err_t;
|
||||
typedef int32_t tmq_resp_err_t;
|
||||
|
||||
typedef struct tmq_t tmq_t;
|
||||
typedef struct tmq_topic_vgroup_t tmq_topic_vgroup_t;
|
||||
|
|
@ -229,7 +230,7 @@ DLL_EXPORT tmq_list_t *tmq_list_new();
|
|||
DLL_EXPORT int32_t tmq_list_append(tmq_list_t *, const char *);
|
||||
DLL_EXPORT void tmq_list_destroy(tmq_list_t *);
|
||||
DLL_EXPORT int32_t tmq_list_get_size(const tmq_list_t *);
|
||||
DLL_EXPORT char ** tmq_list_to_c_array(const tmq_list_t *);
|
||||
DLL_EXPORT char **tmq_list_to_c_array(const tmq_list_t *);
|
||||
|
||||
DLL_EXPORT tmq_t *tmq_consumer_new(tmq_conf_t *conf, char *errstr, int32_t errstrLen);
|
||||
|
||||
|
|
@ -240,7 +241,7 @@ DLL_EXPORT const char *tmq_err2str(tmq_resp_err_t);
|
|||
DLL_EXPORT tmq_resp_err_t tmq_subscribe(tmq_t *tmq, const tmq_list_t *topic_list);
|
||||
DLL_EXPORT tmq_resp_err_t tmq_unsubscribe(tmq_t *tmq);
|
||||
DLL_EXPORT tmq_resp_err_t tmq_subscription(tmq_t *tmq, tmq_list_t **topics);
|
||||
DLL_EXPORT TAOS_RES * tmq_consumer_poll(tmq_t *tmq, int64_t timeout);
|
||||
DLL_EXPORT TAOS_RES *tmq_consumer_poll(tmq_t *tmq, int64_t timeout);
|
||||
DLL_EXPORT tmq_resp_err_t tmq_consumer_close(tmq_t *tmq);
|
||||
DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets);
|
||||
DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const tmq_topic_vgroup_list_t *offsets, tmq_commit_cb *cb, void *param);
|
||||
|
|
@ -260,7 +261,7 @@ enum tmq_conf_res_t {
|
|||
|
||||
typedef enum tmq_conf_res_t tmq_conf_res_t;
|
||||
|
||||
DLL_EXPORT tmq_conf_t * tmq_conf_new();
|
||||
DLL_EXPORT tmq_conf_t *tmq_conf_new();
|
||||
DLL_EXPORT tmq_conf_res_t tmq_conf_set(tmq_conf_t *conf, const char *key, const char *value);
|
||||
DLL_EXPORT void tmq_conf_destroy(tmq_conf_t *conf);
|
||||
DLL_EXPORT void tmq_conf_set_auto_commit_cb(tmq_conf_t *conf, tmq_commit_cb *cb, void *param);
|
||||
|
|
|
|||
|
|
@ -108,6 +108,7 @@ typedef struct SColumnInfoData {
|
|||
|
||||
typedef struct SQueryTableDataCond {
|
||||
// STimeWindow twindow;
|
||||
uint64_t suid;
|
||||
int32_t order; // desc|asc order to iterate the data block
|
||||
int32_t numOfCols;
|
||||
SColumnInfo* colList;
|
||||
|
|
|
|||
|
|
@ -71,7 +71,7 @@ SEpSet getEpSet_s(SCorEpSet* pEpSet);
|
|||
#define colDataGetData(p1_, r_) \
|
||||
((IS_VAR_DATA_TYPE((p1_)->info.type)) ? colDataGetVarData(p1_, r_) : colDataGetNumData(p1_, r_))
|
||||
|
||||
#define IS_JSON_NULL(type,data) ((type) == TSDB_DATA_TYPE_JSON && *(data) == TSDB_DATA_TYPE_NULL)
|
||||
#define IS_JSON_NULL(type, data) ((type) == TSDB_DATA_TYPE_JSON && *(data) == TSDB_DATA_TYPE_NULL)
|
||||
|
||||
static FORCE_INLINE bool colDataIsNull_s(const SColumnInfoData* pColumnInfoData, uint32_t row) {
|
||||
if (!pColumnInfoData->hasNull) {
|
||||
|
|
@ -180,7 +180,7 @@ static FORCE_INLINE void colDataAppendDouble(SColumnInfoData* pColumnInfoData, u
|
|||
*(double*)p = *(double*)v;
|
||||
}
|
||||
|
||||
int32_t getJsonValueLen(const char *data);
|
||||
int32_t getJsonValueLen(const char* data);
|
||||
|
||||
int32_t colDataAppend(SColumnInfoData* pColumnInfoData, uint32_t currentRow, const char* pData, bool isNull);
|
||||
int32_t colDataMergeCol(SColumnInfoData* pColumnInfoData, uint32_t numOfRow1, int32_t* capacity,
|
||||
|
|
@ -206,7 +206,7 @@ SSDataBlock* blockDataExtractBlock(SSDataBlock* pBlock, int32_t startIndex, int3
|
|||
size_t blockDataGetSize(const SSDataBlock* pBlock);
|
||||
size_t blockDataGetRowSize(SSDataBlock* pBlock);
|
||||
double blockDataGetSerialRowSize(const SSDataBlock* pBlock);
|
||||
size_t blockDataGetSerialMetaSize(const SSDataBlock* pBlock);
|
||||
size_t blockDataGetSerialMetaSize(uint32_t numOfCols);
|
||||
|
||||
int32_t blockDataSort(SSDataBlock* pDataBlock, SArray* pOrderInfo);
|
||||
int32_t blockDataSort_rv(SSDataBlock* pDataBlock, SArray* pOrderInfo, bool nullFirst);
|
||||
|
|
@ -223,7 +223,8 @@ int32_t blockDataTrimFirstNRows(SSDataBlock* pBlock, size_t n);
|
|||
|
||||
SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData);
|
||||
|
||||
void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols, int8_t needCompress);
|
||||
void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols,
|
||||
int8_t needCompress);
|
||||
const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRows, const char* pData);
|
||||
|
||||
void blockDebugShowData(const SArray* dataBlocks, const char* flag);
|
||||
|
|
@ -231,11 +232,13 @@ void blockDebugShowData(const SArray* dataBlocks, const char* flag);
|
|||
int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks, STSchema* pTSchema, int32_t vgId,
|
||||
tb_uid_t suid);
|
||||
|
||||
char* buildCtbNameByGroupId(const char* stbName, uint64_t groupId);
|
||||
|
||||
SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pSchema, bool createTb, int64_t suid,
|
||||
const char* stbFullName, int32_t vgId);
|
||||
|
||||
static FORCE_INLINE int32_t blockGetEncodeSize(const SSDataBlock* pBlock) {
|
||||
return blockDataGetSerialMetaSize(pBlock) + blockDataGetSize(pBlock);
|
||||
return blockDataGetSerialMetaSize(pBlock->info.numOfCols) + blockDataGetSize(pBlock);
|
||||
}
|
||||
|
||||
static FORCE_INLINE int32_t blockCompressColData(SColumnInfoData* pColRes, int32_t numOfRows, char* data,
|
||||
|
|
|
|||
|
|
@ -933,6 +933,7 @@ typedef struct {
|
|||
int64_t numOfProcessedFetch;
|
||||
int64_t numOfProcessedDrop;
|
||||
int64_t numOfProcessedHb;
|
||||
int64_t numOfProcessedDelete;
|
||||
int64_t cacheDataSize;
|
||||
int64_t numOfQueryInQueue;
|
||||
int64_t numOfFetchInQueue;
|
||||
|
|
@ -1314,6 +1315,31 @@ typedef struct {
|
|||
int32_t tSerializeSKillTransReq(void* buf, int32_t bufLen, SKillTransReq* pReq);
|
||||
int32_t tDeserializeSKillTransReq(void* buf, int32_t bufLen, SKillTransReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
int32_t useless; // useless
|
||||
} SBalanceVgroupReq;
|
||||
|
||||
int32_t tSerializeSBalanceVgroupReq(void* buf, int32_t bufLen, SBalanceVgroupReq* pReq);
|
||||
int32_t tDeserializeSBalanceVgroupReq(void* buf, int32_t bufLen, SBalanceVgroupReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
int32_t vgId1;
|
||||
int32_t vgId2;
|
||||
} SMergeVgroupReq;
|
||||
|
||||
int32_t tSerializeSMergeVgroupReq(void* buf, int32_t bufLen, SMergeVgroupReq* pReq);
|
||||
int32_t tDeserializeSMergeVgroupReq(void* buf, int32_t bufLen, SMergeVgroupReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
int32_t vgId;
|
||||
int32_t dnodeId1;
|
||||
int32_t dnodeId2;
|
||||
int32_t dnodeId3;
|
||||
} SRedistributeVgroupReq;
|
||||
|
||||
int32_t tSerializeSRedistributeVgroupReq(void* buf, int32_t bufLen, SRedistributeVgroupReq* pReq);
|
||||
int32_t tDeserializeSRedistributeVgroupReq(void* buf, int32_t bufLen, SRedistributeVgroupReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
char user[TSDB_USER_LEN];
|
||||
char spi;
|
||||
|
|
@ -2459,6 +2485,33 @@ typedef struct {
|
|||
int32_t tSerializeSUserIndexRsp(void* buf, int32_t bufLen, const SUserIndexRsp* pRsp);
|
||||
int32_t tDeserializeSUserIndexRsp(void* buf, int32_t bufLen, SUserIndexRsp* pRsp);
|
||||
|
||||
typedef struct {
|
||||
char tbFName[TSDB_TABLE_FNAME_LEN];
|
||||
} STableIndexReq;
|
||||
|
||||
int32_t tSerializeSTableIndexReq(void* buf, int32_t bufLen, STableIndexReq* pReq);
|
||||
int32_t tDeserializeSTableIndexReq(void* buf, int32_t bufLen, STableIndexReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
int8_t intervalUnit;
|
||||
int8_t slidingUnit;
|
||||
int64_t interval;
|
||||
int64_t offset;
|
||||
int64_t sliding;
|
||||
int64_t dstTbUid;
|
||||
int32_t dstVgId; // for stream
|
||||
SEpSet epSet;
|
||||
char* expr;
|
||||
} STableIndexInfo;
|
||||
|
||||
typedef struct {
|
||||
SArray* pIndex;
|
||||
} STableIndexRsp;
|
||||
|
||||
int32_t tSerializeSTableIndexRsp(void* buf, int32_t bufLen, const STableIndexRsp* pRsp);
|
||||
int32_t tDeserializeSTableIndexRsp(void* buf, int32_t bufLen, STableIndexRsp* pRsp);
|
||||
|
||||
|
||||
typedef struct {
|
||||
int8_t mqMsgType;
|
||||
int32_t code;
|
||||
|
|
@ -2689,20 +2742,20 @@ int32_t tEncodeSVSubmitReq(SEncoder* pCoder, const SVSubmitReq* pReq);
|
|||
int32_t tDecodeSVSubmitReq(SDecoder* pCoder, SVSubmitReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
int64_t delUid;
|
||||
int64_t tbUid; // super/child/normal table
|
||||
int8_t type; // table type
|
||||
int16_t nWnds;
|
||||
char* tbFullName;
|
||||
char* subPlan;
|
||||
STimeWindow wnds[];
|
||||
SMsgHead header;
|
||||
uint64_t sId;
|
||||
uint64_t queryId;
|
||||
uint64_t taskId;
|
||||
uint32_t sqlLen;
|
||||
uint32_t phyLen;
|
||||
char* sql;
|
||||
char* msg;
|
||||
} SVDeleteReq;
|
||||
|
||||
int32_t tEncodeSVDeleteReq(SEncoder* pCoder, const SVDeleteReq* pReq);
|
||||
int32_t tDecodeSVDeleteReq(SDecoder* pCoder, SVDeleteReq* pReq);
|
||||
int32_t tSerializeSVDeleteReq(void *buf, int32_t bufLen, SVDeleteReq *pReq);
|
||||
int32_t tDeserializeSVDeleteReq(void *buf, int32_t bufLen, SVDeleteReq *pReq);
|
||||
|
||||
typedef struct {
|
||||
int32_t code;
|
||||
int64_t affectedRows;
|
||||
} SVDeleteRsp;
|
||||
|
||||
|
|
|
|||
|
|
@ -130,6 +130,7 @@ enum {
|
|||
TD_DEF_MSG_TYPE(TDMT_MND_CREATE_INDEX, "create-index", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_DROP_INDEX, "drop-index", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_GET_INDEX, "get-index", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_GET_TABLE_INDEX, "get-table-index", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_CREATE_TOPIC, "create-topic", SMCreateTopicReq, SMCreateTopicRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_ALTER_TOPIC, "alter-topic", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_DROP_TOPIC, "drop-topic", NULL, NULL)
|
||||
|
|
@ -153,6 +154,9 @@ enum {
|
|||
TD_DEF_MSG_TYPE(TDMT_MND_GRANT, "grant", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_AUTH, "auth", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_APPLY_MSG, "mnode-apply", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_BALANCE_VGROUP, "balance-vgroup", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_MERGE_VGROUP, "merge-vgroup", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_MND_REDISTRIBUTE_VGROUP, "redistribute-vgroup", NULL, NULL)
|
||||
|
||||
TD_NEW_MSG_SEG(TDMT_VND_MSG)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_SUBMIT, "submit", SSubmitReq, SSubmitRsp)
|
||||
|
|
@ -169,10 +173,6 @@ enum {
|
|||
TD_DEF_MSG_TYPE(TDMT_VND_CREATE_STB, "vnode-create-stb", SVCreateStbReq, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_ALTER_STB, "vnode-alter-stb", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_DROP_STB, "vnode-drop-stb", SVDropStbReq, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_MQ_CONSUME, "vnode-mq-consume", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_MQ_QUERY, "vnode-mq-query", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_MQ_CONNECT, "vnode-mq-connect", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_MQ_DISCONNECT, "vnode-mq-disconnect", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_MQ_VG_CHANGE, "vnode-mq-vg-change", SMqRebVgReq, SMqRebVgRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_MQ_VG_DELETE, "vnode-mq-vg-delete", SMqVDeleteReq, SMqVDeleteRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_CANCEL_TASK, "vnode-cancel-task", NULL, NULL)
|
||||
|
|
@ -183,11 +183,8 @@ enum {
|
|||
TD_DEF_MSG_TYPE(TDMT_VND_EXPLAIN, "vnode-explain", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_SUBSCRIBE, "vnode-subscribe", SMVSubscribeReq, SMVSubscribeRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_CONSUME, "vnode-consume", SMqPollReq, SMqDataBlkRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TASK_DEPLOY, "vnode-task-deploy", SStreamTaskDeployReq, SStreamTaskDeployRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_STREAM_TRIGGER, "vnode-stream-trigger", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TASK_RUN, "vnode-stream-task-run", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TASK_DISPATCH, "vnode-stream-task-dispatch", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TASK_RECOVER, "vnode-stream-task-recover", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_STREAM_DISPATCH_WRITE, "vnode-stream-task-dispatch-write", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_CREATE_SMA, "vnode-create-sma", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_CANCEL_SMA, "vnode-cancel-sma", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_DROP_SMA, "vnode-drop-sma", NULL, NULL)
|
||||
|
|
@ -201,11 +198,12 @@ enum {
|
|||
|
||||
TD_NEW_MSG_SEG(TDMT_QND_MSG)
|
||||
|
||||
TD_NEW_MSG_SEG(TDMT_SND_MSG)
|
||||
TD_DEF_MSG_TYPE(TDMT_SND_TASK_DEPLOY, "snode-task-deploy", SStreamTaskDeployReq, SStreamTaskDeployRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_SND_TASK_RUN, "snode-stream-task-run", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_SND_TASK_DISPATCH, "snode-stream-task-dispatch", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_SND_TASK_RECOVER, "snode-stream-task-recover", NULL, NULL)
|
||||
//shared by snode and vnode
|
||||
TD_NEW_MSG_SEG(TDMT_STREAM_MSG)
|
||||
TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_DEPLOY, "stream-task-deploy", SStreamTaskDeployReq, SStreamTaskDeployRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_RUN, "stream-task-run", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_DISPATCH, "stream-task-dispatch", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_STREAM_TASK_RECOVER, "stream-task-recover", NULL, NULL)
|
||||
|
||||
TD_NEW_MSG_SEG(TDMT_SCH_MSG)
|
||||
TD_DEF_MSG_TYPE(TDMT_SCH_LINK_BROKEN, "link-broken", NULL, NULL)
|
||||
|
|
|
|||
|
|
@ -188,10 +188,10 @@
|
|||
#define TK_KILL 170
|
||||
#define TK_CONNECTION 171
|
||||
#define TK_TRANSACTION 172
|
||||
#define TK_MERGE 173
|
||||
#define TK_BALANCE 173
|
||||
#define TK_VGROUP 174
|
||||
#define TK_REDISTRIBUTE 175
|
||||
#define TK_SPLIT 176
|
||||
#define TK_MERGE 175
|
||||
#define TK_REDISTRIBUTE 176
|
||||
#define TK_SYNCDB 177
|
||||
#define TK_DELETE 178
|
||||
#define TK_NULL 179
|
||||
|
|
|
|||
|
|
@ -71,16 +71,21 @@ typedef struct SCatalogReq {
|
|||
bool forceUpdate;
|
||||
} SCatalogReq;
|
||||
|
||||
typedef struct SMetaRes {
|
||||
int32_t code;
|
||||
void* pRes;
|
||||
} SMetaRes;
|
||||
|
||||
typedef struct SMetaData {
|
||||
SArray *pDbVgroup; // SArray<SArray<SVgroupInfo>*>
|
||||
SArray *pDbCfg; // SArray<SDbCfgInfo>
|
||||
SArray *pDbInfo; // SArray<SDbInfo>
|
||||
SArray *pTableMeta; // SArray<STableMeta*>
|
||||
SArray *pTableHash; // SArray<SVgroupInfo>
|
||||
SArray *pUdfList; // SArray<SFuncInfo>
|
||||
SArray *pIndex; // SArray<SIndexInfo>
|
||||
SArray *pUser; // SArray<bool>
|
||||
SArray *pQnodeList; // SArray<SQueryNodeAddr>
|
||||
SArray *pDbVgroup; // pRes = SArray<SVgroupInfo>*
|
||||
SArray *pDbCfg; // pRes = SDbCfgInfo*
|
||||
SArray *pDbInfo; // pRes = SDbInfo*
|
||||
SArray *pTableMeta; // pRes = STableMeta*
|
||||
SArray *pTableHash; // pRes = SVgroupInfo*
|
||||
SArray *pUdfList; // pRes = SFuncInfo*
|
||||
SArray *pIndex; // pRes = SIndexInfo*
|
||||
SArray *pUser; // pRes = bool*
|
||||
SArray *pQnodeList; // pRes = SQueryNodeAddr*
|
||||
} SMetaData;
|
||||
|
||||
typedef struct SCatalogCfg {
|
||||
|
|
@ -272,6 +277,8 @@ int32_t catalogGetDBCfg(SCatalog* pCtg, void *pRpc, const SEpSet* pMgmtEps, cons
|
|||
|
||||
int32_t catalogGetIndexMeta(SCatalog* pCtg, void *pRpc, const SEpSet* pMgmtEps, const char* indexName, SIndexInfo* pInfo);
|
||||
|
||||
int32_t catalogGetTableIndex(SCatalog* pCtg, void *pTrans, const SEpSet* pMgmtEps, const char* tbFName, SArray** pRes);
|
||||
|
||||
int32_t catalogGetUdfInfo(SCatalog* pCtg, void *pRpc, const SEpSet* pMgmtEps, const char* funcName, SFuncInfo* pInfo);
|
||||
|
||||
int32_t catalogChkAuth(SCatalog* pCtg, void *pRpc, const SEpSet* pMgmtEps, const char* user, const char* dbFName, AUTH_TYPE type, bool *pass);
|
||||
|
|
@ -280,7 +287,6 @@ int32_t catalogUpdateUserAuthInfo(SCatalog* pCtg, SGetUserAuthRsp* pAuth);
|
|||
|
||||
int32_t catalogUpdateVgEpSet(SCatalog* pCtg, const char* dbFName, int32_t vgId, SEpSet *epSet);
|
||||
|
||||
|
||||
int32_t ctgdLaunchAsyncCall(SCatalog* pCtg, void *pTrans, const SEpSet* pMgmtEps, uint64_t reqId, bool forceUpdate);
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -32,6 +32,18 @@ extern "C" {
|
|||
struct SDataSink;
|
||||
struct SSDataBlock;
|
||||
|
||||
typedef struct SDeleterRes {
|
||||
uint64_t uid;
|
||||
SArray* uidList;
|
||||
int64_t skey;
|
||||
int64_t ekey;
|
||||
int64_t affectedRows;
|
||||
} SDeleterRes;
|
||||
|
||||
typedef struct SDeleterParam {
|
||||
SArray* pUidList;
|
||||
} SDeleterParam;
|
||||
|
||||
typedef struct SDataSinkStat {
|
||||
uint64_t cachedSize;
|
||||
} SDataSinkStat;
|
||||
|
|
@ -64,7 +76,7 @@ typedef struct SOutputData {
|
|||
* @param pHandle output
|
||||
* @return error code
|
||||
*/
|
||||
int32_t dsCreateDataSinker(const SDataSinkNode* pDataSink, DataSinkHandle* pHandle);
|
||||
int32_t dsCreateDataSinker(const SDataSinkNode* pDataSink, DataSinkHandle* pHandle, void* pParam);
|
||||
|
||||
int32_t dsDataSinkGetCacheSize(SDataSinkStat *pStat);
|
||||
|
||||
|
|
|
|||
|
|
@ -197,6 +197,7 @@ typedef struct SAggFunctionInfo {
|
|||
struct SScalarParam {
|
||||
SColumnInfoData *columnData;
|
||||
SHashObj *pHashFilter;
|
||||
int32_t hashValueType;
|
||||
void *param; // other parameter, such as meta handle from vnode, to extract table name/tag value
|
||||
int32_t numOfRows;
|
||||
};
|
||||
|
|
|
|||
|
|
@ -122,6 +122,14 @@ typedef enum EFunctionType {
|
|||
// internal function
|
||||
FUNCTION_TYPE_SELECT_VALUE,
|
||||
|
||||
// distributed splitting functions
|
||||
FUNCTION_TYPE_APERCENTILE_PARTIAL,
|
||||
FUNCTION_TYPE_APERCENTILE_MERGE,
|
||||
FUNCTION_TYPE_SPREAD_PARTIAL,
|
||||
FUNCTION_TYPE_SPREAD_MERGE,
|
||||
FUNCTION_TYPE_HISTOGRAM_PARTIAL,
|
||||
FUNCTION_TYPE_HISTOGRAM_MERGE,
|
||||
|
||||
// user defined funcion
|
||||
FUNCTION_TYPE_UDF = 10000
|
||||
} EFunctionType;
|
||||
|
|
|
|||
|
|
@ -136,14 +136,14 @@ int indexRebuild(SIndex* index, SIndexOpts* opt);
|
|||
* @param index (output, index json object)
|
||||
* @return error code
|
||||
*/
|
||||
int tIndexJsonOpen(SIndexJsonOpts* opts, const char* path, SIndexJson** index);
|
||||
int indexJsonOpen(SIndexJsonOpts* opts, const char* path, SIndexJson** index);
|
||||
/*
|
||||
* close index
|
||||
* @param index (input, index to be closed)
|
||||
* @return void
|
||||
*/
|
||||
|
||||
void tIndexJsonClose(SIndexJson* index);
|
||||
void indexJsonClose(SIndexJson* index);
|
||||
|
||||
/*
|
||||
* insert terms into index
|
||||
|
|
@ -152,7 +152,7 @@ void tIndexJsonClose(SIndexJson* index);
|
|||
* @param uid (input, uid of terms)
|
||||
* @return error code
|
||||
*/
|
||||
int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid);
|
||||
int indexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid);
|
||||
/*
|
||||
* search index
|
||||
* @param index (input, index object)
|
||||
|
|
@ -161,7 +161,7 @@ int tIndexJsonPut(SIndexJson* index, SIndexJsonMultiTerm* terms, uint64_t uid);
|
|||
* @return error code
|
||||
*/
|
||||
|
||||
int tIndexJsonSearch(SIndexJson* index, SIndexJsonMultiTermQuery* query, SArray* result);
|
||||
int indexJsonSearch(SIndexJson* index, SIndexJsonMultiTermQuery* query, SArray* result);
|
||||
/*
|
||||
* @param
|
||||
* @param
|
||||
|
|
|
|||
|
|
@ -28,6 +28,14 @@ extern "C" {
|
|||
#define DESCRIBE_RESULT_TYPE_LEN (20 + VARSTR_HEADER_SIZE)
|
||||
#define DESCRIBE_RESULT_NOTE_LEN (8 + VARSTR_HEADER_SIZE)
|
||||
|
||||
#define PRIVILEGE_TYPE_MASK(n) (1 << n)
|
||||
|
||||
#define PRIVILEGE_TYPE_ALL PRIVILEGE_TYPE_MASK(0)
|
||||
#define PRIVILEGE_TYPE_READ PRIVILEGE_TYPE_MASK(1)
|
||||
#define PRIVILEGE_TYPE_WRITE PRIVILEGE_TYPE_MASK(2)
|
||||
|
||||
#define PRIVILEGE_TYPE_TEST_MASK(val, mask) (((val) & (mask)) != 0)
|
||||
|
||||
typedef struct SDatabaseOptions {
|
||||
ENodeType type;
|
||||
int32_t buffer;
|
||||
|
|
@ -316,14 +324,6 @@ typedef struct SDropFunctionStmt {
|
|||
bool ignoreNotExists;
|
||||
} SDropFunctionStmt;
|
||||
|
||||
#define PRIVILEGE_TYPE_MASK(n) (1 << n)
|
||||
|
||||
#define PRIVILEGE_TYPE_ALL PRIVILEGE_TYPE_MASK(0)
|
||||
#define PRIVILEGE_TYPE_READ PRIVILEGE_TYPE_MASK(1)
|
||||
#define PRIVILEGE_TYPE_WRITE PRIVILEGE_TYPE_MASK(2)
|
||||
|
||||
#define PRIVILEGE_TYPE_TEST_MASK(val, mask) (((val) & (mask)) != 0)
|
||||
|
||||
typedef struct SGrantStmt {
|
||||
ENodeType type;
|
||||
char userName[TSDB_USER_LEN];
|
||||
|
|
@ -333,6 +333,30 @@ typedef struct SGrantStmt {
|
|||
|
||||
typedef SGrantStmt SRevokeStmt;
|
||||
|
||||
typedef struct SBalanceVgroupStmt {
|
||||
ENodeType type;
|
||||
} SBalanceVgroupStmt;
|
||||
|
||||
typedef struct SMergeVgroupStmt {
|
||||
ENodeType type;
|
||||
int32_t vgId1;
|
||||
int32_t vgId2;
|
||||
} SMergeVgroupStmt;
|
||||
|
||||
typedef struct SRedistributeVgroupStmt {
|
||||
ENodeType type;
|
||||
int32_t vgId;
|
||||
int32_t dnodeId1;
|
||||
int32_t dnodeId2;
|
||||
int32_t dnodeId3;
|
||||
SNodeList* pDnodes;
|
||||
} SRedistributeVgroupStmt;
|
||||
|
||||
typedef struct SSplitVgroupStmt {
|
||||
ENodeType type;
|
||||
int32_t vgId;
|
||||
} SSplitVgroupStmt;
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -140,6 +140,7 @@ typedef enum ENodeType {
|
|||
QUERY_NODE_DROP_FUNCTION_STMT,
|
||||
QUERY_NODE_CREATE_STREAM_STMT,
|
||||
QUERY_NODE_DROP_STREAM_STMT,
|
||||
QUERY_NODE_BALANCE_VGROUP_STMT,
|
||||
QUERY_NODE_MERGE_VGROUP_STMT,
|
||||
QUERY_NODE_REDISTRIBUTE_VGROUP_STMT,
|
||||
QUERY_NODE_SPLIT_VGROUP_STMT,
|
||||
|
|
@ -206,8 +207,8 @@ typedef enum ENodeType {
|
|||
QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN,
|
||||
QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN,
|
||||
QUERY_NODE_PHYSICAL_PLAN_PROJECT,
|
||||
QUERY_NODE_PHYSICAL_PLAN_JOIN,
|
||||
QUERY_NODE_PHYSICAL_PLAN_AGG,
|
||||
QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN,
|
||||
QUERY_NODE_PHYSICAL_PLAN_HASH_AGG,
|
||||
QUERY_NODE_PHYSICAL_PLAN_EXCHANGE,
|
||||
QUERY_NODE_PHYSICAL_PLAN_MERGE,
|
||||
QUERY_NODE_PHYSICAL_PLAN_SORT,
|
||||
|
|
@ -217,11 +218,11 @@ typedef enum ENodeType {
|
|||
QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_SEMI_INTERVAL,
|
||||
QUERY_NODE_PHYSICAL_PLAN_FILL,
|
||||
QUERY_NODE_PHYSICAL_PLAN_SESSION_WINDOW,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION_WINDOW,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION_WINDOW,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STATE_WINDOW,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE_WINDOW,
|
||||
QUERY_NODE_PHYSICAL_PLAN_MERGE_SESSION,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_SESSION,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_SESSION,
|
||||
QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE,
|
||||
QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE,
|
||||
QUERY_NODE_PHYSICAL_PLAN_PARTITION,
|
||||
QUERY_NODE_PHYSICAL_PLAN_INDEF_ROWS_FUNC,
|
||||
QUERY_NODE_PHYSICAL_PLAN_DISPATCH,
|
||||
|
|
|
|||
|
|
@ -210,8 +210,8 @@ extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t
|
|||
#define NEED_CLIENT_RM_TBLMETA_ERROR(_code) \
|
||||
((_code) == TSDB_CODE_PAR_TABLE_NOT_EXIST || (_code) == TSDB_CODE_VND_TB_NOT_EXIST || \
|
||||
(_code) == TSDB_CODE_PAR_INVALID_COLUMNS_NUM || (_code) == TSDB_CODE_PAR_INVALID_COLUMN || \
|
||||
(_code) == TSDB_CODE_PAR_TAGS_NOT_MATCHED || (_code == TSDB_CODE_PAR_VALUE_TOO_LONG) || \
|
||||
(_code == TSDB_CODE_PAR_INVALID_DROP_COL))
|
||||
(_code) == TSDB_CODE_PAR_TAGS_NOT_MATCHED || (_code) == TSDB_CODE_PAR_VALUE_TOO_LONG || \
|
||||
(_code) == TSDB_CODE_PAR_INVALID_DROP_COL || ((_code) == TSDB_CODE_TDB_INVALID_TABLE_ID))
|
||||
#define NEED_CLIENT_REFRESH_VG_ERROR(_code) \
|
||||
((_code) == TSDB_CODE_VND_HASH_MISMATCH || (_code) == TSDB_CODE_VND_INVALID_VGROUP_ID)
|
||||
#define NEED_CLIENT_REFRESH_TBLMETA_ERROR(_code) ((_code) == TSDB_CODE_TDB_TABLE_RECREATED)
|
||||
|
|
|
|||
|
|
@ -31,7 +31,12 @@ enum {
|
|||
NODE_TYPE_MNODE,
|
||||
};
|
||||
|
||||
|
||||
typedef struct SDeleteRes {
|
||||
uint64_t uid;
|
||||
SArray* uidList;
|
||||
int64_t skey;
|
||||
int64_t ekey;
|
||||
} SDeleteRes;
|
||||
|
||||
typedef struct SQWorkerCfg {
|
||||
uint32_t maxSchedulerNum;
|
||||
|
|
@ -47,6 +52,7 @@ typedef struct {
|
|||
uint64_t fetchProcessed;
|
||||
uint64_t dropProcessed;
|
||||
uint64_t hbProcessed;
|
||||
uint64_t deleteProcessed;
|
||||
|
||||
uint64_t numOfQueryInQueue;
|
||||
uint64_t numOfFetchInQueue;
|
||||
|
|
@ -74,6 +80,8 @@ int32_t qWorkerProcessDropMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int6
|
|||
|
||||
int32_t qWorkerProcessHbMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, int64_t ts);
|
||||
|
||||
int32_t qWorkerProcessDeleteMsg(void *node, void *qWorkerMgmt, SRpcMsg *pMsg, SRpcMsg *pRsp, SDeleteRes *pRes);
|
||||
|
||||
void qWorkerDestroy(void **qWorkerMgmt);
|
||||
|
||||
int32_t qWorkerGetStat(SReadHandle *handle, void *qWorkerMgmt, SQWorkerStat *pStat);
|
||||
|
|
|
|||
|
|
@ -131,40 +131,13 @@ static FORCE_INLINE void* streamQueueNextItem(SStreamQueue* queue) {
|
|||
|
||||
SStreamDataSubmit* streamDataSubmitNew(SSubmitReq* pReq);
|
||||
|
||||
static FORCE_INLINE void streamDataSubmitRefDec(SStreamDataSubmit* pDataSubmit) {
|
||||
int32_t ref = atomic_sub_fetch_32(pDataSubmit->dataRef, 1);
|
||||
ASSERT(ref >= 0);
|
||||
if (ref == 0) {
|
||||
taosMemoryFree(pDataSubmit->data);
|
||||
taosMemoryFree(pDataSubmit->dataRef);
|
||||
}
|
||||
}
|
||||
void streamDataSubmitRefDec(SStreamDataSubmit* pDataSubmit);
|
||||
|
||||
SStreamDataSubmit* streamSubmitRefClone(SStreamDataSubmit* pSubmit);
|
||||
|
||||
#if 0
|
||||
int32_t streamDataBlockEncode(void** buf, const SStreamDataBlock* pOutput);
|
||||
void* streamDataBlockDecode(const void* buf, SStreamDataBlock* pInput);
|
||||
|
||||
static FORCE_INLINE int32_t streamEnqueue1(SStreamQueue* queue, SStreamQueueItem* pItem) {
|
||||
int8_t inputStatus = atomic_load_8(&queue->enqueueStatus);
|
||||
if (inputStatus == TASK_INPUT_STATUS__NORMAL) {
|
||||
if (pItem->type == STREAM_INPUT__DATA_SUBMIT) {
|
||||
SStreamDataSubmit* pSubmitClone = streamSubmitRefClone((SStreamDataSubmit*)pItem);
|
||||
if (pSubmitClone == NULL) {
|
||||
atomic_store_8(&queue->enqueueStatus, TASK_INPUT_STATUS__FAILED);
|
||||
return -1;
|
||||
}
|
||||
taosWriteQitem(queue->queue, pSubmitClone);
|
||||
} else if (pItem->type == STREAM_INPUT__DATA_BLOCK) {
|
||||
taosWriteQitem(queue->queue, pItem);
|
||||
} else if (pItem->type == STREAM_INPUT__CHECKPOINT) {
|
||||
taosWriteQitem(queue->queue, pItem);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
|
|
@ -209,6 +182,7 @@ typedef void FSmaSink(void* vnode, int64_t smaId, const SArray* data);
|
|||
typedef struct {
|
||||
int64_t smaId;
|
||||
// following are not applicable to encoder and decoder
|
||||
void* vnode;
|
||||
FSmaSink* smaSink;
|
||||
} STaskSinkSma;
|
||||
|
||||
|
|
@ -285,18 +259,12 @@ struct SStreamTask {
|
|||
|
||||
int8_t inputStatus;
|
||||
int8_t outputStatus;
|
||||
#if 0
|
||||
STaosQueue* inputQ;
|
||||
STaosQall* inputQAll;
|
||||
STaosQueue* outputQ;
|
||||
STaosQall* outputQAll;
|
||||
#endif
|
||||
|
||||
SStreamQueue* inputQueue;
|
||||
SStreamQueue* outputQueue;
|
||||
|
||||
// application storage
|
||||
void* ahandle;
|
||||
// void* ahandle;
|
||||
};
|
||||
|
||||
SStreamTask* tNewSStreamTask(int64_t streamId, int32_t childId);
|
||||
|
|
@ -337,7 +305,16 @@ static FORCE_INLINE void streamTaskInputFail(SStreamTask* pTask) {
|
|||
}
|
||||
|
||||
static FORCE_INLINE int32_t streamTaskOutput(SStreamTask* pTask, SStreamDataBlock* pBlock) {
|
||||
taosWriteQitem(pTask->outputQueue->queue, pBlock);
|
||||
if (pTask->sinkType == TASK_SINK__TABLE) {
|
||||
ASSERT(pTask->dispatchType == TASK_DISPATCH__NONE);
|
||||
pTask->tbSink.tbSinkFunc(pTask, pTask->tbSink.vnode, 0, pBlock->blocks);
|
||||
} else if (pTask->sinkType == TASK_SINK__SMA) {
|
||||
ASSERT(pTask->dispatchType == TASK_DISPATCH__NONE);
|
||||
pTask->smaSink.smaSink(pTask->smaSink.vnode, pTask->smaSink.smaId, pBlock->blocks);
|
||||
} else {
|
||||
ASSERT(pTask->dispatchType != TASK_DISPATCH__NONE);
|
||||
taosWriteQitem(pTask->outputQueue->queue, pBlock);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -345,45 +322,24 @@ typedef struct {
|
|||
int32_t reserved;
|
||||
} SStreamTaskDeployRsp;
|
||||
|
||||
typedef struct {
|
||||
// SMsgHead head;
|
||||
int64_t streamId;
|
||||
int32_t taskId;
|
||||
SArray* data; // SArray<SSDataBlock>
|
||||
} SStreamTaskExecReq;
|
||||
|
||||
typedef struct {
|
||||
// SMsgHead head;
|
||||
SStreamTask* task;
|
||||
} SStreamTaskDeployReq;
|
||||
|
||||
int32_t tEncodeSStreamTaskExecReq(void** buf, const SStreamTaskExecReq* pReq);
|
||||
void* tDecodeSStreamTaskExecReq(const void* buf, SStreamTaskExecReq* pReq);
|
||||
void tFreeSStreamTaskExecReq(SStreamTaskExecReq* pReq);
|
||||
|
||||
typedef struct {
|
||||
int32_t reserved;
|
||||
} SStreamTaskExecRsp;
|
||||
|
||||
typedef struct {
|
||||
SMsgHead head;
|
||||
int64_t streamId;
|
||||
int32_t taskId;
|
||||
} SStreamTaskRunReq;
|
||||
|
||||
typedef struct {
|
||||
// SMsgHead head;
|
||||
int64_t streamId;
|
||||
int64_t version;
|
||||
SArray* res; // SArray<SSDataBlock>
|
||||
} SStreamSinkReq;
|
||||
|
||||
typedef struct {
|
||||
int64_t streamId;
|
||||
int32_t taskId;
|
||||
int32_t sourceTaskId;
|
||||
int32_t sourceVg;
|
||||
int32_t sourceChildId;
|
||||
int32_t upstreamNodeId;
|
||||
#if 0
|
||||
int64_t sourceVer;
|
||||
#endif
|
||||
|
|
@ -411,11 +367,9 @@ typedef struct {
|
|||
int8_t inputStatus;
|
||||
} SStreamTaskRecoverRsp;
|
||||
|
||||
int32_t streamTriggerByWrite(SStreamTask* pTask, int32_t vgId, SMsgCb* pMsgCb);
|
||||
int32_t tDecodeStreamDispatchReq(SDecoder* pDecoder, SStreamDispatchReq* pReq);
|
||||
|
||||
int32_t streamEnqueueDataSubmit(SStreamTask* pTask, SStreamDataSubmit* input);
|
||||
int32_t streamEnqueueDataBlk(SStreamTask* pTask, SStreamDataBlock* input);
|
||||
int32_t streamDequeueOutput(SStreamTask* pTask, void** output);
|
||||
int32_t streamTriggerByWrite(SStreamTask* pTask, int32_t vgId, SMsgCb* pMsgCb);
|
||||
|
||||
int32_t streamTaskRun(SStreamTask* pTask);
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,7 @@ SSyncNode* syncNodeAcquire(int64_t rid);
|
|||
void syncNodeRelease(SSyncNode* pNode);
|
||||
|
||||
int32_t syncGetRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg);
|
||||
int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcMsg* msg);
|
||||
int32_t syncGetAndDelRespRpc(int64_t rid, uint64_t index, SRpcHandleInfo* pInfo);
|
||||
void syncSetMsgCb(int64_t rid, const SMsgCb* msgcb);
|
||||
char* sync2SimpleStr(int64_t rid);
|
||||
|
||||
|
|
|
|||
|
|
@ -33,7 +33,10 @@ extern int32_t tsRpcHeadSize;
|
|||
typedef struct {
|
||||
uint32_t clientIp;
|
||||
uint16_t clientPort;
|
||||
char user[TSDB_USER_LEN];
|
||||
union {
|
||||
char user[TSDB_USER_LEN];
|
||||
int64_t applyIndex;
|
||||
};
|
||||
} SRpcConnInfo;
|
||||
|
||||
typedef struct SRpcHandleInfo {
|
||||
|
|
@ -43,6 +46,7 @@ typedef struct SRpcHandleInfo {
|
|||
int32_t noResp; // has response or not(default 0, 0: resp, 1: no resp);
|
||||
int32_t persistHandle; // persist handle or not
|
||||
|
||||
SRpcConnInfo connInfo;
|
||||
// app info
|
||||
void *ahandle; // app handle set by client
|
||||
void *wrapper; // wrapper handle
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ void taosqsort(void *src, int64_t numOfElem, int64_t size, const void *param, __
|
|||
* @param flags
|
||||
* @return
|
||||
*/
|
||||
void *taosbsearch(const void *key, const void *base, int64_t nmemb, int64_t size, __compar_fn_t fn, int32_t flags);
|
||||
void *taosbsearch(const void *key, const void *base, int32_t nmemb, int32_t size, __compar_fn_t compar, int32_t flags);
|
||||
|
||||
/**
|
||||
* adjust heap
|
||||
|
|
@ -82,7 +82,7 @@ void *taosbsearch(const void *key, const void *base, int64_t nmemb, int64_t size
|
|||
* @return
|
||||
*/
|
||||
void taosheapadjust(void *base, int32_t size, int32_t start, int32_t end, const void *parcompar,
|
||||
__ext_compar_fn_t compar, char* buf, bool maxroot);
|
||||
__ext_compar_fn_t compar, char *buf, bool maxroot);
|
||||
|
||||
/**
|
||||
* sort heap to make sure it is a max/min root heap
|
||||
|
|
@ -97,8 +97,7 @@ void taosheapadjust(void *base, int32_t size, int32_t start, int32_t end, const
|
|||
* @param maxroot: if heap is max root heap
|
||||
* @return
|
||||
*/
|
||||
void taosheapsort(void *base, int32_t size, int32_t len, const void *parcompar, __ext_compar_fn_t compar,
|
||||
bool maxroot);
|
||||
void taosheapsort(void *base, int32_t size, int32_t len, const void *parcompar, __ext_compar_fn_t compar, bool maxroot);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
|
|
|
|||
|
|
@ -185,7 +185,7 @@ int32_t* taosGetErrno();
|
|||
#define TSDB_CODE_MND_BNODE_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x0357)
|
||||
#define TSDB_CODE_MND_TOO_FEW_MNODES TAOS_DEF_ERROR_CODE(0, 0x0358)
|
||||
#define TSDB_CODE_MND_TOO_MANY_MNODES TAOS_DEF_ERROR_CODE(0, 0x0359)
|
||||
#define TSDB_CODE_MND_CANT_DROP_MASTER TAOS_DEF_ERROR_CODE(0, 0x035A)
|
||||
#define TSDB_CODE_MND_CANT_DROP_LEADER TAOS_DEF_ERROR_CODE(0, 0x035A)
|
||||
|
||||
// mnode-acct
|
||||
#define TSDB_CODE_MND_ACCT_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0360)
|
||||
|
|
@ -339,9 +339,9 @@ int32_t* taosGetErrno();
|
|||
#define TSDB_CODE_TDB_TAG_VER_OUT_OF_DATE TAOS_DEF_ERROR_CODE(0, 0x060D)
|
||||
#define TSDB_CODE_TDB_TIMESTAMP_OUT_OF_RANGE TAOS_DEF_ERROR_CODE(0, 0x060E)
|
||||
#define TSDB_CODE_TDB_SUBMIT_MSG_MSSED_UP TAOS_DEF_ERROR_CODE(0, 0x060F)
|
||||
#define TSDB_CODE_TDB_INVALID_ACTION TAOS_DEF_ERROR_CODE(0, 0x0600)
|
||||
#define TSDB_CODE_TDB_INVALID_CREATE_TB_MSG TAOS_DEF_ERROR_CODE(0, 0x0601)
|
||||
#define TSDB_CODE_TDB_NO_TABLE_DATA_IN_MEM TAOS_DEF_ERROR_CODE(0, 0x0602)
|
||||
#define TSDB_CODE_TDB_INVALID_ACTION TAOS_DEF_ERROR_CODE(0, 0x0610)
|
||||
#define TSDB_CODE_TDB_INVALID_CREATE_TB_MSG TAOS_DEF_ERROR_CODE(0, 0x0611)
|
||||
#define TSDB_CODE_TDB_NO_TABLE_DATA_IN_MEM TAOS_DEF_ERROR_CODE(0, 0x0612)
|
||||
#define TSDB_CODE_TDB_FILE_ALREADY_EXISTS TAOS_DEF_ERROR_CODE(0, 0x0613)
|
||||
#define TSDB_CODE_TDB_TABLE_RECONFIGURE TAOS_DEF_ERROR_CODE(0, 0x0614)
|
||||
#define TSDB_CODE_TDB_IVD_CREATE_TABLE_INFO TAOS_DEF_ERROR_CODE(0, 0x0615)
|
||||
|
|
@ -653,6 +653,7 @@ int32_t* taosGetErrno();
|
|||
#define TSDB_CODE_PAR_VALUE_TOO_LONG TAOS_DEF_ERROR_CODE(0, 0x2653)
|
||||
#define TSDB_CODE_PAR_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x2654)
|
||||
#define TSDB_CODE_PAR_INVALID_DELETE_WHERE TAOS_DEF_ERROR_CODE(0, 0x2655)
|
||||
#define TSDB_CODE_PAR_INVALID_REDISTRIBUTE_VG TAOS_DEF_ERROR_CODE(0, 0x2656)
|
||||
|
||||
//planner
|
||||
#define TSDB_CODE_PLAN_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x2700)
|
||||
|
|
|
|||
|
|
@ -32,7 +32,7 @@ extern "C" {
|
|||
#define TD_VER_MAX UINT64_MAX // TODO: use the real max version from query handle
|
||||
|
||||
// Bytes for each type.
|
||||
extern const int32_t TYPE_BYTES[15];
|
||||
extern const int32_t TYPE_BYTES[16];
|
||||
|
||||
// TODO: replace and remove code below
|
||||
#define CHAR_BYTES sizeof(char)
|
||||
|
|
@ -356,8 +356,8 @@ typedef enum ELogicConditionType {
|
|||
|
||||
#define TSDB_DEFAULT_EXPLAIN_VERBOSE false
|
||||
|
||||
#define TSDB_EXPLAIN_RESULT_ROW_SIZE 1024
|
||||
#define TSDB_EXPLAIN_RESULT_COLUMN_NAME "QUERY PLAN"
|
||||
#define TSDB_EXPLAIN_RESULT_ROW_SIZE 512
|
||||
#define TSDB_EXPLAIN_RESULT_COLUMN_NAME "QUERY_PLAN"
|
||||
|
||||
#define TSDB_MAX_FIELD_LEN 16384
|
||||
#define TSDB_MAX_BINARY_LEN (TSDB_MAX_FIELD_LEN - TSDB_KEYSIZE) // keep 16384
|
||||
|
|
|
|||
|
|
@ -188,6 +188,13 @@ SDiskbasedBufStatis getDBufStatis(const SDiskbasedBuf* pBuf);
|
|||
*/
|
||||
void dBufPrintStatis(const SDiskbasedBuf* pBuf);
|
||||
|
||||
/**
|
||||
* Set all of page buffer are not need
|
||||
* @param pBuf
|
||||
* @return
|
||||
*/
|
||||
void clearDiskbasedBuf(SDiskbasedBuf* pBuf);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -45,7 +45,7 @@ extern "C" {
|
|||
|
||||
#define ERROR_MSG_BUF_DEFAULT_SIZE 512
|
||||
#define HEARTBEAT_INTERVAL 1500 // ms
|
||||
#define SYNC_ON_TOP_OF_ASYNC 0
|
||||
#define SYNC_ON_TOP_OF_ASYNC 1
|
||||
|
||||
enum {
|
||||
RES_TYPE__QUERY = 1,
|
||||
|
|
@ -144,7 +144,7 @@ typedef struct STscObj {
|
|||
int32_t numOfReqs; // number of sqlObj bound to this connection
|
||||
SAppInstInfo* pAppInfo;
|
||||
SHashObj* pRequests;
|
||||
int8_t schemalessType;
|
||||
int8_t schemalessType; // todo remove it, this attribute should be move to request
|
||||
} STscObj;
|
||||
|
||||
typedef struct SResultColumn {
|
||||
|
|
@ -171,6 +171,7 @@ typedef struct SReqResultInfo {
|
|||
uint32_t current;
|
||||
bool completed;
|
||||
int32_t precision;
|
||||
bool convertUcs4;
|
||||
int32_t payloadLen;
|
||||
} SReqResultInfo;
|
||||
|
||||
|
|
@ -222,7 +223,7 @@ typedef struct SSyncQueryParam {
|
|||
SRequestObj* pRequest;
|
||||
} SSyncQueryParam;
|
||||
|
||||
void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4);
|
||||
void* doAsyncFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4);
|
||||
void* doFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4);
|
||||
|
||||
void doSetOneRowPtr(SReqResultInfo* pResultInfo);
|
||||
|
|
|
|||
|
|
@ -191,6 +191,8 @@ void *createRequest(STscObj *pObj, int32_t type) {
|
|||
pRequest->requestId = generateRequestId();
|
||||
pRequest->metric.start = taosGetTimestampUs();
|
||||
|
||||
pRequest->body.resInfo.convertUcs4 = true; // convert ucs4 by default
|
||||
|
||||
pRequest->type = type;
|
||||
pRequest->pTscObj = pObj;
|
||||
pRequest->msgBuf = taosMemoryCalloc(1, ERROR_MSG_BUF_DEFAULT_SIZE);
|
||||
|
|
|
|||
|
|
@ -457,7 +457,7 @@ int32_t scheduleQuery(SRequestObj* pRequest, SQueryPlan* pDag, SArray* pNodeList
|
|||
return pRequest->code;
|
||||
}
|
||||
|
||||
if (TDMT_VND_SUBMIT == pRequest->type || TDMT_VND_CREATE_TABLE == pRequest->type) {
|
||||
if (TDMT_VND_SUBMIT == pRequest->type || TDMT_VND_DELETE == pRequest->type || TDMT_VND_CREATE_TABLE == pRequest->type) {
|
||||
pRequest->body.resInfo.numOfRows = res.numOfRows;
|
||||
|
||||
if (pRequest->body.queryJob != 0) {
|
||||
|
|
@ -580,7 +580,8 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) {
|
|||
|
||||
STscObj* pTscObj = pRequest->pTscObj;
|
||||
if (code != TSDB_CODE_SUCCESS && NEED_CLIENT_HANDLE_ERROR(code)) {
|
||||
tscDebug("0x%"PRIx64" client retry to handle the error, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId);
|
||||
tscDebug("0x%"PRIx64" client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code),
|
||||
pRequest->retry, pRequest->requestId);
|
||||
pRequest->prevCode = code;
|
||||
doAsyncQuery(pRequest, true);
|
||||
return;
|
||||
|
|
@ -592,6 +593,7 @@ void schedulerExecCb(SQueryResult* pResult, void* param, int32_t code) {
|
|||
pRequest->code = code;
|
||||
}
|
||||
|
||||
tscDebug("schedulerExecCb request type %s", TMSG_INFO(pRequest->type));
|
||||
if (NEED_CLIENT_RM_TBLMETA_REQ(pRequest->type)) {
|
||||
removeMeta(pTscObj, pRequest->tableList);
|
||||
}
|
||||
|
|
@ -695,6 +697,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) {
|
|||
if (TSDB_CODE_SUCCESS == code) {
|
||||
schedulerAsyncExecJob(pAppInfo->pTransporter, pNodeList, pRequest->body.pDag, &pRequest->body.queryJob,
|
||||
pRequest->sqlstr, pRequest->metric.start, schedulerExecCb, pRequest);
|
||||
} else {
|
||||
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
|
||||
}
|
||||
|
||||
//todo not to be released here
|
||||
|
|
@ -702,7 +706,8 @@ void launchAsyncQuery(SRequestObj* pRequest, SQuery* pQuery) {
|
|||
break;
|
||||
}
|
||||
case QUERY_EXEC_MODE_EMPTY_RESULT:
|
||||
pRequest->type = TSDB_SQL_RETRIEVE_EMPTY_RESULT;
|
||||
pRequest->type = TSDB_SQL_RETRIEVE_EMPTY_RESULT;
|
||||
pRequest->body.queryFp(pRequest->body.param, pRequest, 0);
|
||||
break;
|
||||
default:
|
||||
break;
|
||||
|
|
@ -1109,7 +1114,7 @@ static void syncFetchFn(void* param, TAOS_RES* res, int32_t numOfRows) {
|
|||
tsem_post(&pParam->sem);
|
||||
}
|
||||
|
||||
void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) {
|
||||
void* doAsyncFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) {
|
||||
assert(pRequest != NULL);
|
||||
|
||||
SReqResultInfo* pResultInfo = &pRequest->body.resInfo;
|
||||
|
|
@ -1121,6 +1126,10 @@ void* doAsyncFetchRow(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUc
|
|||
}
|
||||
|
||||
SSyncQueryParam* pParam = pRequest->body.param;
|
||||
|
||||
// convert ucs4 to native multi-bytes string
|
||||
pResultInfo->convertUcs4 = convertUcs4;
|
||||
|
||||
taos_fetch_rows_a(pRequest, syncFetchFn, pParam);
|
||||
tsem_wait(&pParam->sem);
|
||||
}
|
||||
|
|
@ -1280,8 +1289,7 @@ static int32_t doConvertUCS4(SReqResultInfo* pResultInfo, int32_t numOfRows, int
|
|||
int32_t length = taosUcs4ToMbs((TdUcs4*)varDataVal(jsonInnerData), varDataLen(jsonInnerData),
|
||||
varDataVal(dst) + CHAR_BYTES);
|
||||
if (length <= 0) {
|
||||
tscError("charset:%s to %s. val:%s convert failed.", DEFAULT_UNICODE_ENCODEC, tsCharset,
|
||||
varDataVal(jsonInnerData));
|
||||
tscError("charset:%s to %s. convert failed.", DEFAULT_UNICODE_ENCODEC, tsCharset);
|
||||
length = 0;
|
||||
}
|
||||
varDataSetLen(dst, length + CHAR_BYTES * 2);
|
||||
|
|
@ -1340,6 +1348,17 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32
|
|||
uint64_t groupId = *(uint64_t*)p;
|
||||
p += sizeof(uint64_t);
|
||||
|
||||
// check fields
|
||||
for(int32_t i = 0; i < numOfCols; ++i) {
|
||||
int16_t type = *(int16_t*) p;
|
||||
p += sizeof(int16_t);
|
||||
|
||||
int32_t bytes = *(int32_t*) p;
|
||||
p += sizeof(int32_t);
|
||||
|
||||
// ASSERT(type == pFields[i].type && bytes == pFields[i].bytes);
|
||||
}
|
||||
|
||||
int32_t* colLength = (int32_t*)p;
|
||||
p += sizeof(int32_t) * numOfCols;
|
||||
|
||||
|
|
|
|||
|
|
@ -219,14 +219,13 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) {
|
|||
|
||||
if (TD_RES_QUERY(res)) {
|
||||
SRequestObj *pRequest = (SRequestObj *)res;
|
||||
#if SYNC_ON_TOP_OF_ASYNC
|
||||
return doAsyncFetchRows(pRequest, true, true);
|
||||
#else
|
||||
if (pRequest->type == TSDB_SQL_RETRIEVE_EMPTY_RESULT || pRequest->type == TSDB_SQL_INSERT ||
|
||||
pRequest->code != TSDB_CODE_SUCCESS || taos_num_fields(res) == 0) {
|
||||
return NULL;
|
||||
}
|
||||
|
||||
#if SYNC_ON_TOP_OF_ASYNC
|
||||
return doAsyncFetchRow(pRequest, true, true);
|
||||
#else
|
||||
return doFetchRows(pRequest, true, true);
|
||||
#endif
|
||||
|
||||
|
|
@ -489,6 +488,7 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) {
|
|||
if (res == NULL) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
if (TD_RES_QUERY(res)) {
|
||||
SRequestObj *pRequest = (SRequestObj *)res;
|
||||
|
||||
|
|
@ -500,7 +500,11 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) {
|
|||
return 0;
|
||||
}
|
||||
|
||||
doFetchRows(pRequest, false, true);
|
||||
#if SYNC_ON_TOP_OF_ASYNC
|
||||
doAsyncFetchRows(pRequest, false, true);
|
||||
#else
|
||||
doFetchRows(pRequest, true, true);
|
||||
#endif
|
||||
|
||||
// TODO refactor
|
||||
SReqResultInfo *pResultInfo = &pRequest->body.resInfo;
|
||||
|
|
@ -548,7 +552,11 @@ int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData) {
|
|||
return 0;
|
||||
}
|
||||
|
||||
#if SYNC_ON_TOP_OF_ASYNC
|
||||
doAsyncFetchRows(pRequest, false, false);
|
||||
#else
|
||||
doFetchRows(pRequest, false, false);
|
||||
#endif
|
||||
|
||||
SReqResultInfo *pResultInfo = &pRequest->body.resInfo;
|
||||
|
||||
|
|
@ -625,8 +633,10 @@ void retrieveMetaCallback(SMetaData* pResultMeta, void* param, int32_t code) {
|
|||
taosMemoryFree(pWrapper);
|
||||
launchAsyncQuery(pRequest, pQuery);
|
||||
} else {
|
||||
tscDebug("error happens, code:%d", code);
|
||||
if (NEED_CLIENT_HANDLE_ERROR(code)) {
|
||||
tscDebug("0x%"PRIx64" client retry to handle the error, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId);
|
||||
tscDebug("0x%"PRIx64" client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code),
|
||||
pRequest->retry, pRequest->requestId);
|
||||
pRequest->prevCode = code;
|
||||
doAsyncQuery(pRequest, true);
|
||||
return;
|
||||
|
|
@ -691,6 +701,7 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) {
|
|||
.pTransporter = pTscObj->pAppInfo->pTransporter,
|
||||
.pStmtCb = NULL,
|
||||
.pUser = pTscObj->user,
|
||||
.schemalessType = pTscObj->schemalessType,
|
||||
.isSuperUser = (0 == strcmp(pTscObj->user, TSDB_DEFAULT_USER)),
|
||||
.async = true,};
|
||||
return TSDB_CODE_SUCCESS;
|
||||
|
|
@ -699,13 +710,14 @@ int32_t createParseContext(const SRequestObj *pRequest, SParseContext** pCxt) {
|
|||
void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) {
|
||||
SParseContext* pCxt = NULL;
|
||||
STscObj *pTscObj = pRequest->pTscObj;
|
||||
int32_t code = 0;
|
||||
|
||||
if (pRequest->retry++ > REQUEST_TOTAL_EXEC_TIMES) {
|
||||
pRequest->code = pRequest->prevCode;
|
||||
code = pRequest->prevCode;
|
||||
goto _error;
|
||||
}
|
||||
|
||||
int32_t code = createParseContext(pRequest, &pCxt);
|
||||
code = createParseContext(pRequest, &pCxt);
|
||||
if (code != TSDB_CODE_SUCCESS) {
|
||||
goto _error;
|
||||
}
|
||||
|
|
@ -742,7 +754,7 @@ void doAsyncQuery(SRequestObj* pRequest, bool updateMetaForce) {
|
|||
}
|
||||
|
||||
_error:
|
||||
tscError("0x%"PRIx64" error happens, code:%s, reqId:0x%"PRIx64, pRequest->self, tstrerror(code), pRequest->requestId);
|
||||
tscError("0x%"PRIx64" error happens, code:%d - %s, reqId:0x%"PRIx64, pRequest->self, code, tstrerror(code), pRequest->requestId);
|
||||
terrno = code;
|
||||
pRequest->code = code;
|
||||
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
|
||||
|
|
@ -763,11 +775,11 @@ static void fetchCallback(void* pResult, void* param, int32_t code) {
|
|||
}
|
||||
|
||||
if (pRequest->code != TSDB_CODE_SUCCESS) {
|
||||
pRequest->code = code;
|
||||
pRequest->body.fetchFp(pRequest->body.param, pRequest, 0);
|
||||
return;
|
||||
}
|
||||
|
||||
pRequest->code = setQueryResultFromRsp(&pRequest->body.resInfo, (SRetrieveTableRsp*)pResultInfo->pData, true, false);
|
||||
pRequest->code = setQueryResultFromRsp(pResultInfo, (SRetrieveTableRsp*)pResultInfo->pData, pResultInfo->convertUcs4, false);
|
||||
if (pRequest->code != TSDB_CODE_SUCCESS) {
|
||||
pResultInfo->numOfRows = 0;
|
||||
pRequest->code = code;
|
||||
|
|
@ -807,6 +819,13 @@ void taos_fetch_rows_a(TAOS_RES *res, __taos_async_fn_t fp, void *param) {
|
|||
schedulerAsyncFetchRows(pRequest->body.queryJob, fetchCallback, pRequest);
|
||||
}
|
||||
|
||||
void taos_fetch_raw_block_a(TAOS_RES* res, __taos_async_fn_t fp, void* param) {
|
||||
ASSERT(res != NULL && fp != NULL);
|
||||
SRequestObj *pRequest = res;
|
||||
pRequest->body.resInfo.convertUcs4 = false;
|
||||
taos_fetch_rows_a(res, fp, param);
|
||||
}
|
||||
|
||||
TAOS_SUB *taos_subscribe(TAOS *taos, int restart, const char *topic, const char *sql, TAOS_SUBSCRIBE_CALLBACK fp,
|
||||
void *param, int interval) {
|
||||
// TODO
|
||||
|
|
|
|||
|
|
@ -195,7 +195,7 @@ typedef struct {
|
|||
|
||||
tmq_conf_t* tmq_conf_new() {
|
||||
tmq_conf_t* conf = taosMemoryCalloc(1, sizeof(tmq_conf_t));
|
||||
conf->withTbName = -1;
|
||||
conf->withTbName = false;
|
||||
conf->autoCommit = true;
|
||||
conf->autoCommitInterval = 5000;
|
||||
conf->resetOffset = TMQ_CONF__RESET_OFFSET__EARLIEAST;
|
||||
|
|
@ -256,13 +256,10 @@ tmq_conf_res_t tmq_conf_set(tmq_conf_t* conf, const char* key, const char* value
|
|||
|
||||
if (strcmp(key, "msg.with.table.name") == 0) {
|
||||
if (strcmp(value, "true") == 0) {
|
||||
conf->withTbName = 1;
|
||||
conf->withTbName = true;
|
||||
return TMQ_CONF_OK;
|
||||
} else if (strcmp(value, "false") == 0) {
|
||||
conf->withTbName = 0;
|
||||
return TMQ_CONF_OK;
|
||||
} else if (strcmp(value, "none") == 0) {
|
||||
conf->withTbName = -1;
|
||||
conf->withTbName = false;
|
||||
return TMQ_CONF_OK;
|
||||
} else {
|
||||
return TMQ_CONF_INVALID;
|
||||
|
|
@ -326,7 +323,7 @@ static int32_t tmqMakeTopicVgKey(char* dst, const char* topicName, int32_t vg) {
|
|||
|
||||
int32_t tmqCommitCb(void* param, const SDataBuf* pMsg, int32_t code) {
|
||||
SMqCommitCbParam* pParam = (SMqCommitCbParam*)param;
|
||||
pParam->rspErr = code == 0 ? TMQ_RESP_ERR__SUCCESS : TMQ_RESP_ERR__FAIL;
|
||||
pParam->rspErr = code;
|
||||
if (pParam->async) {
|
||||
if (pParam->automatic && pParam->tmq->commitCb) {
|
||||
pParam->tmq->commitCb(pParam->tmq, pParam->rspErr, (tmq_topic_vgroup_list_t*)pParam->offsets,
|
||||
|
|
@ -435,12 +432,13 @@ int32_t tmqCommitInner(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int8_
|
|||
code = pParam->rspErr;
|
||||
tsem_destroy(&pParam->rspSem);
|
||||
taosMemoryFree(pParam);
|
||||
} else {
|
||||
code = 0;
|
||||
}
|
||||
|
||||
// avoid double free if msg is sent
|
||||
buf = NULL;
|
||||
|
||||
code = 0;
|
||||
END:
|
||||
if (buf) taosMemoryFree(buf);
|
||||
/*if (pParam) taosMemoryFree(pParam);*/
|
||||
|
|
@ -448,9 +446,9 @@ END:
|
|||
|
||||
if (code != 0 && async) {
|
||||
if (automatic) {
|
||||
tmq->commitCb(tmq, TMQ_RESP_ERR__FAIL, (tmq_topic_vgroup_list_t*)pOffsets, tmq->commitCbUserParam);
|
||||
tmq->commitCb(tmq, code, (tmq_topic_vgroup_list_t*)pOffsets, tmq->commitCbUserParam);
|
||||
} else {
|
||||
userCb(tmq, TMQ_RESP_ERR__FAIL, (tmq_topic_vgroup_list_t*)pOffsets, userParam);
|
||||
userCb(tmq, code, (tmq_topic_vgroup_list_t*)pOffsets, userParam);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -496,7 +494,6 @@ int32_t tmqHandleAllDelayedTask(tmq_t* tmq) {
|
|||
tmqAskEp(tmq, true);
|
||||
taosTmrReset(tmqAssignDelayedHbTask, 1000, tmq, tmqMgmt.timer, &tmq->hbTimer);
|
||||
} else if (*pTaskType == TMQ_DELAYED_TASK__COMMIT) {
|
||||
/*tmq_commit(tmq, NULL, true);*/
|
||||
tmqCommitInner(tmq, NULL, 1, 1, tmq->commitCb, tmq->commitCbUserParam);
|
||||
taosTmrReset(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, tmq, tmqMgmt.timer, &tmq->commitTimer);
|
||||
} else if (*pTaskType == TMQ_DELAYED_TASK__REPORT) {
|
||||
|
|
@ -669,94 +666,6 @@ FAIL:
|
|||
|
||||
tmq_resp_err_t tmq_commit(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, int32_t async) {
|
||||
return tmqCommitInner(tmq, offsets, 0, async, tmq->commitCb, tmq->commitCbUserParam);
|
||||
#if 0
|
||||
// TODO: add read write lock
|
||||
SRequestObj* pRequest = NULL;
|
||||
tmq_resp_err_t resp = TMQ_RESP_ERR__SUCCESS;
|
||||
// build msg
|
||||
// send to mnode
|
||||
SMqCMCommitOffsetReq req;
|
||||
SArray* pOffsets = NULL;
|
||||
|
||||
if (offsets == NULL) {
|
||||
pOffsets = taosArrayInit(0, sizeof(SMqOffset));
|
||||
for (int i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) {
|
||||
SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i);
|
||||
for (int j = 0; j < taosArrayGetSize(pTopic->vgs); j++) {
|
||||
SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j);
|
||||
SMqOffset offset;
|
||||
strcpy(offset.topicName, pTopic->topicName);
|
||||
strcpy(offset.cgroup, tmq->groupId);
|
||||
offset.vgId = pVg->vgId;
|
||||
offset.offset = pVg->currentOffset;
|
||||
taosArrayPush(pOffsets, &offset);
|
||||
}
|
||||
}
|
||||
req.num = pOffsets->size;
|
||||
req.offsets = pOffsets->pData;
|
||||
} else {
|
||||
req.num = taosArrayGetSize(&offsets->container);
|
||||
req.offsets = (SMqOffset*)offsets->container.pData;
|
||||
}
|
||||
|
||||
SEncoder encoder;
|
||||
|
||||
tEncoderInit(&encoder, NULL, 0);
|
||||
tEncodeSMqCMCommitOffsetReq(&encoder, &req);
|
||||
int32_t tlen = encoder.pos;
|
||||
void* buf = taosMemoryMalloc(tlen);
|
||||
if (buf == NULL) {
|
||||
tEncoderClear(&encoder);
|
||||
return -1;
|
||||
}
|
||||
tEncoderClear(&encoder);
|
||||
|
||||
tEncoderInit(&encoder, buf, tlen);
|
||||
tEncodeSMqCMCommitOffsetReq(&encoder, &req);
|
||||
tEncoderClear(&encoder);
|
||||
|
||||
pRequest = createRequest(tmq->pTscObj, NULL, NULL, TDMT_MND_MQ_COMMIT_OFFSET);
|
||||
if (pRequest == NULL) {
|
||||
tscError("failed to malloc request");
|
||||
}
|
||||
|
||||
SMqCommitCbParam* pParam = taosMemoryCalloc(1, sizeof(SMqCommitCbParam));
|
||||
if (pParam == NULL) {
|
||||
return -1;
|
||||
}
|
||||
pParam->tmq = tmq;
|
||||
tsem_init(&pParam->rspSem, 0, 0);
|
||||
pParam->async = async;
|
||||
pParam->offsets = pOffsets;
|
||||
|
||||
pRequest->body.requestMsg = (SDataBuf){
|
||||
.pData = buf,
|
||||
.len = tlen,
|
||||
.handle = NULL,
|
||||
};
|
||||
|
||||
SMsgSendInfo* sendInfo = buildMsgInfoImpl(pRequest);
|
||||
sendInfo->requestObjRefId = 0;
|
||||
sendInfo->param = pParam;
|
||||
sendInfo->fp = tmqCommitCb;
|
||||
SEpSet epSet = getEpSet_s(&tmq->pTscObj->pAppInfo->mgmtEp);
|
||||
|
||||
int64_t transporterId = 0;
|
||||
asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo);
|
||||
|
||||
if (!async) {
|
||||
tsem_wait(&pParam->rspSem);
|
||||
resp = pParam->rspErr;
|
||||
tsem_destroy(&pParam->rspSem);
|
||||
taosMemoryFree(pParam);
|
||||
|
||||
if (pOffsets) {
|
||||
taosArrayDestroy(pOffsets);
|
||||
}
|
||||
}
|
||||
|
||||
return resp;
|
||||
#endif
|
||||
}
|
||||
|
||||
tmq_resp_err_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) {
|
||||
|
|
@ -861,93 +770,6 @@ void tmq_conf_set_auto_commit_cb(tmq_conf_t* conf, tmq_commit_cb* cb, void* para
|
|||
conf->commitCbUserParam = param;
|
||||
}
|
||||
|
||||
#if 0
|
||||
TAOS_RES* tmq_create_stream(TAOS* taos, const char* streamName, const char* tbName, const char* sql) {
|
||||
STscObj* pTscObj = (STscObj*)taos;
|
||||
SRequestObj* pRequest = NULL;
|
||||
SQuery* pQueryNode = NULL;
|
||||
char* astStr = NULL;
|
||||
int32_t sqlLen;
|
||||
|
||||
terrno = TSDB_CODE_SUCCESS;
|
||||
if (taos == NULL || streamName == NULL || sql == NULL) {
|
||||
tscError("invalid parameters for creating stream, connObj:%p, stream name:%s, sql:%s", taos, streamName, sql);
|
||||
terrno = TSDB_CODE_TSC_INVALID_INPUT;
|
||||
goto _return;
|
||||
}
|
||||
sqlLen = strlen(sql);
|
||||
|
||||
if (strlen(tbName) >= TSDB_TABLE_NAME_LEN) {
|
||||
tscError("output tb name too long, max length:%d", TSDB_TABLE_NAME_LEN - 1);
|
||||
terrno = TSDB_CODE_TSC_INVALID_INPUT;
|
||||
goto _return;
|
||||
}
|
||||
|
||||
if (sqlLen > TSDB_MAX_ALLOWED_SQL_LEN) {
|
||||
tscError("sql string exceeds max length:%d", TSDB_MAX_ALLOWED_SQL_LEN);
|
||||
terrno = TSDB_CODE_TSC_EXCEED_SQL_LIMIT;
|
||||
goto _return;
|
||||
}
|
||||
|
||||
tscDebug("start to create stream: %s", streamName);
|
||||
|
||||
int32_t code = 0;
|
||||
CHECK_CODE_GOTO(buildRequest(pTscObj, sql, sqlLen, &pRequest), _return);
|
||||
CHECK_CODE_GOTO(parseSql(pRequest, false, &pQueryNode, NULL), _return);
|
||||
CHECK_CODE_GOTO(nodesNodeToString(pQueryNode->pRoot, false, &astStr, NULL), _return);
|
||||
|
||||
/*printf("%s\n", pStr);*/
|
||||
|
||||
SName name = {.acctId = pTscObj->acctId, .type = TSDB_TABLE_NAME_T};
|
||||
strcpy(name.dbname, pRequest->pDb);
|
||||
strcpy(name.tname, streamName);
|
||||
|
||||
SCMCreateStreamReq req = {
|
||||
.igExists = 1,
|
||||
.ast = astStr,
|
||||
.sql = (char*)sql,
|
||||
};
|
||||
tNameExtractFullName(&name, req.name);
|
||||
strcpy(req.targetStbFullName, tbName);
|
||||
|
||||
int tlen = tSerializeSCMCreateStreamReq(NULL, 0, &req);
|
||||
void* buf = taosMemoryMalloc(tlen);
|
||||
if (buf == NULL) {
|
||||
goto _return;
|
||||
}
|
||||
|
||||
tSerializeSCMCreateStreamReq(buf, tlen, &req);
|
||||
|
||||
pRequest->body.requestMsg = (SDataBuf){
|
||||
.pData = buf,
|
||||
.len = tlen,
|
||||
.handle = NULL,
|
||||
};
|
||||
pRequest->type = TDMT_MND_CREATE_STREAM;
|
||||
|
||||
SMsgSendInfo* sendInfo = buildMsgInfoImpl(pRequest);
|
||||
SEpSet epSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp);
|
||||
|
||||
int64_t transporterId = 0;
|
||||
asyncSendMsgToServer(pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo);
|
||||
|
||||
tsem_wait(&pRequest->body.rspSem);
|
||||
|
||||
_return:
|
||||
taosMemoryFreeClear(astStr);
|
||||
qDestroyQuery(pQueryNode);
|
||||
/*if (sendInfo != NULL) {*/
|
||||
/*destroySendMsgInfo(sendInfo);*/
|
||||
/*}*/
|
||||
|
||||
if (pRequest != NULL && terrno != TSDB_CODE_SUCCESS) {
|
||||
pRequest->code = terrno;
|
||||
}
|
||||
|
||||
return pRequest;
|
||||
}
|
||||
#endif
|
||||
|
||||
#if 0
|
||||
int32_t tmqGetSkipLogNum(tmq_message_t* tmq_message) {
|
||||
if (tmq_message == NULL) return 0;
|
||||
|
|
@ -1477,16 +1299,16 @@ TAOS_RES* tmq_consumer_poll(tmq_t* tmq, int64_t timeout) {
|
|||
tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) {
|
||||
if (tmq->status == TMQ_CONSUMER_STATUS__READY) {
|
||||
tmq_resp_err_t rsp = tmq_commit_sync(tmq, NULL);
|
||||
if (rsp == TMQ_RESP_ERR__FAIL) {
|
||||
return TMQ_RESP_ERR__FAIL;
|
||||
if (rsp != TMQ_RESP_ERR__SUCCESS) {
|
||||
return rsp;
|
||||
}
|
||||
|
||||
tmq_list_t* lst = tmq_list_new();
|
||||
rsp = tmq_subscribe(tmq, lst);
|
||||
tmq_list_destroy(lst);
|
||||
|
||||
if (rsp == TMQ_RESP_ERR__FAIL) {
|
||||
return TMQ_RESP_ERR__FAIL;
|
||||
if (rsp != TMQ_RESP_ERR__SUCCESS) {
|
||||
return rsp;
|
||||
}
|
||||
}
|
||||
// TODO: free resources
|
||||
|
|
@ -1496,8 +1318,11 @@ tmq_resp_err_t tmq_consumer_close(tmq_t* tmq) {
|
|||
const char* tmq_err2str(tmq_resp_err_t err) {
|
||||
if (err == TMQ_RESP_ERR__SUCCESS) {
|
||||
return "success";
|
||||
} else if (err == TMQ_RESP_ERR__FAIL) {
|
||||
return "fail";
|
||||
} else {
|
||||
return tstrerror(err);
|
||||
}
|
||||
return "fail";
|
||||
}
|
||||
|
||||
const char* tmq_get_topic_name(TAOS_RES* res) {
|
||||
|
|
@ -1539,10 +1364,11 @@ const char* tmq_get_table_name(TAOS_RES* res) {
|
|||
}
|
||||
return NULL;
|
||||
}
|
||||
DLL_EXPORT void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) {
|
||||
|
||||
void tmq_commit_async(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets, tmq_commit_cb* cb, void* param) {
|
||||
tmqCommitInner(tmq, offsets, 0, 1, cb, param);
|
||||
}
|
||||
|
||||
DLL_EXPORT tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) {
|
||||
tmq_resp_err_t tmq_commit_sync(tmq_t* tmq, const tmq_topic_vgroup_list_t* offsets) {
|
||||
return tmqCommitInner(tmq, offsets, 0, 0, NULL, NULL);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -780,31 +780,28 @@ TEST(testCase, async_api_test) {
|
|||
|
||||
taos_query(pConn, "use test");
|
||||
|
||||
TAOS_RES* pRes = taos_query(pConn, "select * from t1");
|
||||
|
||||
taos_query(pConn, "alter table t1 add column b int");
|
||||
pRes = taos_query(pConn, "insert into t1 values(now, 1, 2)");
|
||||
TAOS_RES* pRes = taos_query(pConn, "desc abc1.tu");
|
||||
if (taos_errno(pRes) != 0) {
|
||||
printf("failed, reason:%s\n", taos_errstr(pRes));
|
||||
}
|
||||
|
||||
// int32_t n = 0;
|
||||
// TAOS_ROW pRow = NULL;
|
||||
// TAOS_FIELD* pFields = taos_fetch_fields(pRes);
|
||||
// int32_t numOfFields = taos_num_fields(pRes);
|
||||
//
|
||||
// char str[512] = {0};
|
||||
// while ((pRow = taos_fetch_row(pRes)) != NULL) {
|
||||
// int32_t* length = taos_fetch_lengths(pRes);
|
||||
// for(int32_t i = 0; i < numOfFields; ++i) {
|
||||
// printf("(%d):%d " , i, length[i]);
|
||||
// }
|
||||
// printf("\n");
|
||||
//
|
||||
// int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
|
||||
// printf("%s\n", str);
|
||||
// memset(str, 0, sizeof(str));
|
||||
// }
|
||||
int32_t n = 0;
|
||||
TAOS_ROW pRow = NULL;
|
||||
TAOS_FIELD* pFields = taos_fetch_fields(pRes);
|
||||
int32_t numOfFields = taos_num_fields(pRes);
|
||||
|
||||
char str[512] = {0};
|
||||
while ((pRow = taos_fetch_row(pRes)) != NULL) {
|
||||
int32_t* length = taos_fetch_lengths(pRes);
|
||||
for(int32_t i = 0; i < numOfFields; ++i) {
|
||||
printf("(%d):%d " , i, length[i]);
|
||||
}
|
||||
printf("\n");
|
||||
|
||||
int32_t code = taos_print_row(str, pRow, pFields, numOfFields);
|
||||
printf("%s\n", str);
|
||||
memset(str, 0, sizeof(str));
|
||||
}
|
||||
|
||||
taos_query_a(pConn, "alter table test.m1 comment 'abcde' ", queryCallback, pConn);
|
||||
getchar();
|
||||
|
|
|
|||
|
|
@ -18,6 +18,7 @@
|
|||
#include "tcompare.h"
|
||||
#include "tglobal.h"
|
||||
#include "tlog.h"
|
||||
#include "tname.h"
|
||||
|
||||
int32_t taosGetFqdnPortFromEp(const char* ep, SEp* pEp) {
|
||||
pEp->port = 0;
|
||||
|
|
@ -99,7 +100,7 @@ void colDataTrim(SColumnInfoData* pColumnInfoData) {
|
|||
// TODO
|
||||
}
|
||||
|
||||
int32_t getJsonValueLen(const char *data) {
|
||||
int32_t getJsonValueLen(const char* data) {
|
||||
int32_t dataLen = 0;
|
||||
if (*data == TSDB_DATA_TYPE_NULL) {
|
||||
dataLen = CHAR_BYTES;
|
||||
|
|
@ -109,7 +110,7 @@ int32_t getJsonValueLen(const char *data) {
|
|||
dataLen = DOUBLE_BYTES + CHAR_BYTES;
|
||||
} else if (*data == TSDB_DATA_TYPE_BOOL) {
|
||||
dataLen = CHAR_BYTES + CHAR_BYTES;
|
||||
} else if (*data & TD_TAG_JSON) { // json string
|
||||
} else if (*data & TD_TAG_JSON) { // json string
|
||||
dataLen = ((STag*)(data))->len;
|
||||
} else {
|
||||
ASSERT(0);
|
||||
|
|
@ -137,7 +138,7 @@ int32_t colDataAppend(SColumnInfoData* pColumnInfoData, uint32_t currentRow, con
|
|||
int32_t dataLen = 0;
|
||||
if (type == TSDB_DATA_TYPE_JSON) {
|
||||
dataLen = getJsonValueLen(pData);
|
||||
}else {
|
||||
} else {
|
||||
dataLen = varDataTLen(pData);
|
||||
}
|
||||
|
||||
|
|
@ -681,9 +682,10 @@ size_t blockDataGetRowSize(SSDataBlock* pBlock) {
|
|||
* @param pBlock
|
||||
* @return
|
||||
*/
|
||||
size_t blockDataGetSerialMetaSize(const SSDataBlock* pBlock) {
|
||||
// | total rows/total length | block group id | each column length |
|
||||
return sizeof(int32_t) + sizeof(uint64_t) + pBlock->info.numOfCols * sizeof(int32_t);
|
||||
size_t blockDataGetSerialMetaSize(uint32_t numOfCols) {
|
||||
// | total rows/total length | block group id | column schema | each column length |
|
||||
return sizeof(int32_t) + sizeof(uint64_t) + numOfCols * (sizeof(int16_t) + sizeof(int32_t)) +
|
||||
numOfCols * sizeof(int32_t);
|
||||
}
|
||||
|
||||
double blockDataGetSerialRowSize(const SSDataBlock* pBlock) {
|
||||
|
|
@ -1215,7 +1217,8 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) {
|
|||
|
||||
pBlock->info.numOfCols = numOfCols;
|
||||
pBlock->info.hasVarCol = pDataBlock->info.hasVarCol;
|
||||
pBlock->info.rowSize = pDataBlock->info.rows;
|
||||
pBlock->info.rowSize = pDataBlock->info.rowSize;
|
||||
pBlock->info.groupId = pDataBlock->info.groupId;
|
||||
|
||||
for (int32_t i = 0; i < numOfCols; ++i) {
|
||||
SColumnInfoData colInfo = {0};
|
||||
|
|
@ -1245,7 +1248,7 @@ SSDataBlock* createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData) {
|
|||
}
|
||||
|
||||
size_t blockDataGetCapacityInRow(const SSDataBlock* pBlock, size_t pageSize) {
|
||||
int32_t payloadSize = pageSize - blockDataGetSerialMetaSize(pBlock);
|
||||
int32_t payloadSize = pageSize - blockDataGetSerialMetaSize(pBlock->info.numOfCols);
|
||||
|
||||
int32_t rowSize = pBlock->info.rowSize;
|
||||
|
||||
|
|
@ -1270,12 +1273,12 @@ size_t blockDataGetCapacityInRow(const SSDataBlock* pBlock, size_t pageSize) {
|
|||
|
||||
void colDataDestroy(SColumnInfoData* pColData) {
|
||||
if (IS_VAR_DATA_TYPE(pColData->info.type)) {
|
||||
taosMemoryFree(pColData->varmeta.offset);
|
||||
taosMemoryFreeClear(pColData->varmeta.offset);
|
||||
} else {
|
||||
taosMemoryFree(pColData->nullbitmap);
|
||||
taosMemoryFreeClear(pColData->nullbitmap);
|
||||
}
|
||||
|
||||
taosMemoryFree(pColData->pData);
|
||||
taosMemoryFreeClear(pColData->pData);
|
||||
}
|
||||
|
||||
static void doShiftBitmap(char* nullBitmap, size_t n, size_t total) {
|
||||
|
|
@ -1285,7 +1288,7 @@ static void doShiftBitmap(char* nullBitmap, size_t n, size_t total) {
|
|||
if (n % 8 == 0) {
|
||||
memmove(nullBitmap, nullBitmap + n / 8, newLen);
|
||||
} else {
|
||||
int32_t tail = n % 8;
|
||||
int32_t tail = n % 8;
|
||||
int32_t i = 0;
|
||||
uint8_t* p = (uint8_t*)nullBitmap;
|
||||
|
||||
|
|
@ -1303,7 +1306,7 @@ static void doShiftBitmap(char* nullBitmap, size_t n, size_t total) {
|
|||
}
|
||||
} else if (n > 8) {
|
||||
int32_t gap = len - newLen;
|
||||
while(i < newLen) {
|
||||
while (i < newLen) {
|
||||
uint8_t v = p[i + gap];
|
||||
p[i] = (v << tail);
|
||||
|
||||
|
|
@ -1318,7 +1321,6 @@ static void doShiftBitmap(char* nullBitmap, size_t n, size_t total) {
|
|||
}
|
||||
}
|
||||
|
||||
|
||||
static void colDataTrimFirstNRows(SColumnInfoData* pColInfoData, size_t n, size_t total) {
|
||||
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
|
||||
memmove(pColInfoData->varmeta.offset, &pColInfoData->varmeta.offset[n], (total - n) * sizeof(int32_t));
|
||||
|
|
@ -1502,7 +1504,7 @@ void blockDebugShowData(const SArray* dataBlocks, const char* flag) {
|
|||
for (int32_t k = 0; k < colNum; k++) {
|
||||
SColumnInfoData* pColInfoData = taosArrayGet(pDataBlock->pDataBlock, k);
|
||||
void* var = POINTER_SHIFT(pColInfoData->pData, j * pColInfoData->info.bytes);
|
||||
if (pColInfoData->hasNull) {
|
||||
if (colDataIsNull(pColInfoData, rows, j, NULL)) {
|
||||
printf(" %15s |", "NULL");
|
||||
continue;
|
||||
}
|
||||
|
|
@ -1546,7 +1548,8 @@ void blockDebugShowData(const SArray* dataBlocks, const char* flag) {
|
|||
*
|
||||
* TODO: colId should be set
|
||||
*/
|
||||
int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks, STSchema* pTSchema, int32_t vgId, tb_uid_t suid) {
|
||||
int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks, STSchema* pTSchema, int32_t vgId,
|
||||
tb_uid_t suid) {
|
||||
int32_t sz = taosArrayGetSize(pDataBlocks);
|
||||
int32_t bufSize = sizeof(SSubmitReq);
|
||||
for (int32_t i = 0; i < sz; ++i) {
|
||||
|
|
@ -1587,12 +1590,12 @@ int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks
|
|||
int32_t dataLen = 0;
|
||||
for (int32_t j = 0; j < rows; ++j) { // iterate by row
|
||||
tdSRowResetBuf(&rb, POINTER_SHIFT(pDataBuf, msgLen)); // set row buf
|
||||
bool isStartKey = false;
|
||||
bool isStartKey = false;
|
||||
int32_t offset = 0;
|
||||
for (int32_t k = 0; k < colNum; ++k) { // iterate by column
|
||||
SColumnInfoData* pColInfoData = taosArrayGet(pDataBlock->pDataBlock, k);
|
||||
STColumn* pCol = &pTSchema->columns[k];
|
||||
void* var = POINTER_SHIFT(pColInfoData->pData, j * pColInfoData->info.bytes);
|
||||
STColumn* pCol = &pTSchema->columns[k];
|
||||
void* var = POINTER_SHIFT(pColInfoData->pData, j * pColInfoData->info.bytes);
|
||||
switch (pColInfoData->info.type) {
|
||||
case TSDB_DATA_TYPE_TIMESTAMP:
|
||||
if (!isStartKey) {
|
||||
|
|
@ -1601,15 +1604,18 @@ int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks
|
|||
offset, k);
|
||||
|
||||
} else {
|
||||
tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_TIMESTAMP, TD_VTYPE_NORM, var, true, offset, k);
|
||||
tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_TIMESTAMP, TD_VTYPE_NORM, var,
|
||||
true, offset, k);
|
||||
}
|
||||
break;
|
||||
case TSDB_DATA_TYPE_NCHAR: {
|
||||
tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_NCHAR, TD_VTYPE_NORM, var, true, offset, k);
|
||||
tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_NCHAR, TD_VTYPE_NORM, var, true,
|
||||
offset, k);
|
||||
break;
|
||||
}
|
||||
case TSDB_DATA_TYPE_VARCHAR: { // TSDB_DATA_TYPE_BINARY
|
||||
tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_VARCHAR, TD_VTYPE_NORM, var, true, offset, k);
|
||||
tdAppendColValToRow(&rb, PRIMARYKEY_TIMESTAMP_COL_ID + k, TSDB_DATA_TYPE_VARCHAR, TD_VTYPE_NORM, var, true,
|
||||
offset, k);
|
||||
break;
|
||||
}
|
||||
case TSDB_DATA_TYPE_VARBINARY:
|
||||
|
|
@ -1647,7 +1653,7 @@ int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks
|
|||
}
|
||||
break;
|
||||
}
|
||||
offset += TYPE_BYTES[pCol->type]; // sum/avg would convert to int64_t/uint64_t/double during aggregation
|
||||
offset += TYPE_BYTES[pCol->type]; // sum/avg would convert to int64_t/uint64_t/double during aggregation
|
||||
}
|
||||
dataLen += TD_ROW_LEN(rb.pBuf);
|
||||
#ifdef TD_DEBUG_PRINT_ROW
|
||||
|
|
@ -1683,11 +1689,38 @@ int32_t buildSubmitReqFromDataBlock(SSubmitReq** pReq, const SArray* pDataBlocks
|
|||
return TSDB_CODE_SUCCESS;
|
||||
}
|
||||
|
||||
char* buildCtbNameByGroupId(const char* stbName, uint64_t groupId) {
|
||||
SArray* tags = taosArrayInit(0, sizeof(void*));
|
||||
SSmlKv* pTag = taosMemoryCalloc(1, sizeof(SSmlKv));
|
||||
pTag->key = "group_id";
|
||||
pTag->keyLen = strlen(pTag->key);
|
||||
pTag->type = TSDB_DATA_TYPE_UBIGINT;
|
||||
pTag->u = groupId;
|
||||
taosArrayPush(tags, &pTag);
|
||||
|
||||
void* cname = taosMemoryCalloc(1, TSDB_TABLE_NAME_LEN + 1);
|
||||
|
||||
RandTableName rname = {
|
||||
.tags = tags,
|
||||
.sTableName = stbName,
|
||||
.sTableNameLen = strlen(stbName),
|
||||
.childTableName = cname,
|
||||
};
|
||||
|
||||
buildChildTableName(&rname);
|
||||
|
||||
taosMemoryFree(pTag);
|
||||
taosArrayDestroy(tags);
|
||||
|
||||
ASSERT(rname.childTableName && rname.childTableName[0]);
|
||||
return rname.childTableName;
|
||||
}
|
||||
|
||||
SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, bool createTb, int64_t suid,
|
||||
const char* stbFullName, int32_t vgId) {
|
||||
SSubmitReq* ret = NULL;
|
||||
SArray* tagArray = taosArrayInit(1, sizeof(STagVal));
|
||||
if(!tagArray) {
|
||||
if (!tagArray) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
return NULL;
|
||||
}
|
||||
|
|
@ -1705,15 +1738,12 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, boo
|
|||
|
||||
if (createTb) {
|
||||
SVCreateTbReq createTbReq = {0};
|
||||
char* cname = taosMemoryCalloc(1, TSDB_TABLE_FNAME_LEN);
|
||||
snprintf(cname, TSDB_TABLE_FNAME_LEN, "%s:%ld", stbFullName, pDataBlock->info.groupId);
|
||||
char* cname = buildCtbNameByGroupId(stbFullName, pDataBlock->info.groupId);
|
||||
createTbReq.name = cname;
|
||||
createTbReq.flags = 0;
|
||||
createTbReq.type = TSDB_CHILD_TABLE;
|
||||
createTbReq.ctb.suid = suid;
|
||||
|
||||
|
||||
|
||||
STagVal tagVal = {.cid = 1,
|
||||
.type = TSDB_DATA_TYPE_UBIGINT,
|
||||
.pData = (uint8_t*)&pDataBlock->info.groupId,
|
||||
|
|
@ -1851,39 +1881,50 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, boo
|
|||
}
|
||||
|
||||
ret->length = htonl(ret->length);
|
||||
taosArrayDestroy(tagArray);
|
||||
taosArrayDestroy(tagArray);
|
||||
return ret;
|
||||
}
|
||||
|
||||
void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols, int8_t needCompress) {
|
||||
void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen, int32_t numOfCols,
|
||||
int8_t needCompress) {
|
||||
// todo extract method
|
||||
int32_t* actualLen = (int32_t*)data;
|
||||
data += sizeof(int32_t);
|
||||
|
||||
uint64_t* groupId = (uint64_t*)data;
|
||||
data += sizeof(uint64_t);
|
||||
|
||||
for (int32_t i = 0; i < numOfCols; ++i) {
|
||||
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
|
||||
*((int16_t*)data) = pColInfoData->info.type;
|
||||
data += sizeof(int16_t);
|
||||
|
||||
*((int32_t*)data) = pColInfoData->info.bytes;
|
||||
data += sizeof(int32_t);
|
||||
}
|
||||
|
||||
int32_t* colSizes = (int32_t*)data;
|
||||
data += numOfCols * sizeof(int32_t);
|
||||
|
||||
*dataLen = (numOfCols * sizeof(int32_t) + sizeof(uint64_t) + sizeof(int32_t));
|
||||
*dataLen = blockDataGetSerialMetaSize(numOfCols);
|
||||
|
||||
int32_t numOfRows = pBlock->info.rows;
|
||||
for (int32_t col = 0; col < numOfCols; ++col) {
|
||||
SColumnInfoData* pColRes = (SColumnInfoData*)taosArrayGet(pBlock->pDataBlock, col);
|
||||
|
||||
// copy the null bitmap
|
||||
size_t metaSize = 0;
|
||||
if (IS_VAR_DATA_TYPE(pColRes->info.type)) {
|
||||
size_t metaSize = numOfRows * sizeof(int32_t);
|
||||
metaSize = numOfRows * sizeof(int32_t);
|
||||
memcpy(data, pColRes->varmeta.offset, metaSize);
|
||||
data += metaSize;
|
||||
(*dataLen) += metaSize;
|
||||
} else {
|
||||
int32_t len = BitmapLen(numOfRows);
|
||||
memcpy(data, pColRes->nullbitmap, len);
|
||||
data += len;
|
||||
(*dataLen) += len;
|
||||
metaSize = BitmapLen(numOfRows);
|
||||
memcpy(data, pColRes->nullbitmap, metaSize);
|
||||
}
|
||||
|
||||
data += metaSize;
|
||||
(*dataLen) += metaSize;
|
||||
|
||||
if (needCompress) {
|
||||
colSizes[col] = blockCompressColData(pColRes, numOfRows, data, needCompress);
|
||||
data += colSizes[col];
|
||||
|
|
@ -1904,6 +1945,8 @@ void blockCompressEncode(const SSDataBlock* pBlock, char* data, int32_t* dataLen
|
|||
|
||||
const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t numOfRows, const char* pData) {
|
||||
blockDataEnsureCapacity(pBlock, numOfRows);
|
||||
pBlock->info.rows = numOfRows;
|
||||
|
||||
const char* pStart = pData;
|
||||
|
||||
int32_t dataLen = *(int32_t*)pStart;
|
||||
|
|
@ -1912,6 +1955,29 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t
|
|||
pBlock->info.groupId = *(uint64_t*)pStart;
|
||||
pStart += sizeof(uint64_t);
|
||||
|
||||
if (pBlock->pDataBlock == NULL) {
|
||||
pBlock->pDataBlock = taosArrayInit(numOfCols, sizeof(SColumnInfoData));
|
||||
taosArraySetSize(pBlock->pDataBlock, numOfCols);
|
||||
}
|
||||
|
||||
pBlock->info.numOfCols = taosArrayGetSize(pBlock->pDataBlock);
|
||||
ASSERT(pBlock->pDataBlock->size >= numOfCols);
|
||||
|
||||
for (int32_t i = 0; i < numOfCols; ++i) {
|
||||
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i);
|
||||
pColInfoData->info.type = *(int16_t*)pStart;
|
||||
pStart += sizeof(int16_t);
|
||||
|
||||
pColInfoData->info.bytes = *(int32_t*)pStart;
|
||||
pStart += sizeof(int32_t);
|
||||
|
||||
if (IS_VAR_DATA_TYPE(pColInfoData->info.type)) {
|
||||
pBlock->info.hasVarCol = true;
|
||||
}
|
||||
}
|
||||
|
||||
blockDataEnsureCapacity(pBlock, numOfRows);
|
||||
|
||||
int32_t* colLen = (int32_t*)pStart;
|
||||
pStart += sizeof(int32_t) * numOfCols;
|
||||
|
||||
|
|
@ -1932,11 +1998,17 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t
|
|||
pColInfoData->pData = taosMemoryMalloc(colLen[i]);
|
||||
}
|
||||
} else {
|
||||
if (pColInfoData->nullbitmap == NULL) {
|
||||
pColInfoData->nullbitmap = taosMemoryCalloc(1, BitmapLen(numOfRows));
|
||||
}
|
||||
memcpy(pColInfoData->nullbitmap, pStart, BitmapLen(numOfRows));
|
||||
pStart += BitmapLen(numOfRows);
|
||||
}
|
||||
|
||||
if (colLen[i] > 0) {
|
||||
if (pColInfoData->pData == NULL) {
|
||||
pColInfoData->pData = taosMemoryCalloc(1, colLen[i]);
|
||||
}
|
||||
memcpy(pColInfoData->pData, pStart, colLen[i]);
|
||||
}
|
||||
|
||||
|
|
@ -1949,4 +2021,4 @@ const char* blockCompressDecode(SSDataBlock* pBlock, int32_t numOfCols, int32_t
|
|||
|
||||
ASSERT(pStart - pData == dataLen);
|
||||
return pStart;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -208,7 +208,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
if (iColumn == 0) {
|
||||
ASSERT(pColVal->cid == pTColumn->colId);
|
||||
ASSERT(pTColumn->type == TSDB_DATA_TYPE_TIMESTAMP);
|
||||
ASSERT(pTColumn->colId == 0);
|
||||
ASSERT(pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID);
|
||||
|
||||
iColVal++;
|
||||
} else {
|
||||
|
|
@ -244,7 +244,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
}
|
||||
}
|
||||
|
||||
ASSERT(flags);
|
||||
// ASSERT(flags); // only 1 column(ts)
|
||||
|
||||
// decide
|
||||
uint32_t nData = 0;
|
||||
|
|
@ -268,7 +268,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
nDataT = BIT2_SIZE(pTSchema->numOfCols - 1) + pTSchema->flen + ntv;
|
||||
break;
|
||||
default:
|
||||
ASSERT(0);
|
||||
break; // only ts column
|
||||
// ASSERT(0);
|
||||
}
|
||||
|
||||
uint8_t tflags = 0;
|
||||
|
|
@ -283,7 +284,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
tflags |= TSROW_KV_BIG;
|
||||
}
|
||||
|
||||
if (nDataT < nDataK) {
|
||||
if (nDataT <= nDataK) {
|
||||
nData = nDataT;
|
||||
} else {
|
||||
nData = nDataK;
|
||||
|
|
@ -352,7 +353,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
ntv = 0;
|
||||
iColVal = 1;
|
||||
|
||||
if (flags & 0xf0 == 0) {
|
||||
if ((flags & 0xf0) == 0) {
|
||||
switch (flags & 0xf) {
|
||||
case TSROW_HAS_VAL:
|
||||
pf = (*ppRow)->pData;
|
||||
|
|
@ -373,7 +374,8 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
ptv = pf + pTSchema->flen;
|
||||
break;
|
||||
default:
|
||||
ASSERT(0);
|
||||
// ASSERT(0);
|
||||
break;
|
||||
}
|
||||
} else {
|
||||
pTSKVRow = (STSKVRow *)(*ppRow)->pData;
|
||||
|
|
@ -417,13 +419,13 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
}
|
||||
|
||||
_set_none:
|
||||
if (flags & 0xf0 == 0) {
|
||||
if ((flags & 0xf0) == 0) {
|
||||
setBitMap(pb, 0, iColumn - 1, flags);
|
||||
}
|
||||
continue;
|
||||
|
||||
_set_null:
|
||||
if (flags & 0xf0 == 0) {
|
||||
if ((flags & 0xf0) == 0) {
|
||||
setBitMap(pb, 1, iColumn - 1, flags);
|
||||
} else {
|
||||
SET_IDX(pidx, pTSKVRow->nCols, nkv, flags);
|
||||
|
|
@ -433,7 +435,7 @@ int32_t tTSRowNew(STSRowBuilder *pBuilder, SArray *pArray, STSchema *pTSchema, S
|
|||
continue;
|
||||
|
||||
_set_value:
|
||||
if (flags & 0xf0 == 0) {
|
||||
if ((flags & 0xf0) == 0) {
|
||||
setBitMap(pb, 2, iColumn - 1, flags);
|
||||
|
||||
if (IS_VAR_DATA_TYPE(pTColumn->type)) {
|
||||
|
|
@ -489,13 +491,13 @@ void tTSRowFree(STSRow2 *pRow) {
|
|||
}
|
||||
|
||||
void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal) {
|
||||
uint8_t isTuple = (pRow->flags & 0xf0 == 0) ? 1 : 0;
|
||||
uint8_t isTuple = ((pRow->flags & 0xf0) == 0) ? 1 : 0;
|
||||
STColumn *pTColumn = &pTSchema->columns[iCol];
|
||||
uint8_t flags = pRow->flags & (uint8_t)0xf;
|
||||
SValue value;
|
||||
|
||||
ASSERT(iCol < pTSchema->numOfCols);
|
||||
ASSERT(flags);
|
||||
// ASSERT(flags); // only 1 ts column
|
||||
ASSERT(pRow->sver == pTSchema->version);
|
||||
|
||||
if (iCol == 0) {
|
||||
|
|
@ -505,7 +507,7 @@ void tTSRowGet(STSRow2 *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal
|
|||
|
||||
if (flags == TSROW_HAS_NONE) {
|
||||
goto _return_none;
|
||||
} else if (flags == TSROW_HAS_NONE) {
|
||||
} else if (flags == TSROW_HAS_NULL) {
|
||||
goto _return_null;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -332,7 +332,7 @@ static int32_t taosAddClientCfg(SConfig *pCfg) {
|
|||
if (cfgAddBool(pCfg, "keepColumnName", tsKeepOriginalColumnName, 1) != 0) return -1;
|
||||
if (cfgAddInt32(pCfg, "queryPolicy", tsQueryPolicy, 1, 3, 1) != 0) return -1;
|
||||
if (cfgAddString(pCfg, "smlChildTableName", "", 1) != 0) return -1;
|
||||
if (cfgAddString(pCfg, "smlTagNullName", tsSmlTagName, 1) != 0) return -1;
|
||||
if (cfgAddString(pCfg, "smlTagName", tsSmlTagName, 1) != 0) return -1;
|
||||
if (cfgAddBool(pCfg, "smlDataFormat", tsSmlDataFormat, 1) != 0) return -1;
|
||||
|
||||
tsNumOfTaskQueueThreads = tsNumOfCores / 4;
|
||||
|
|
@ -532,7 +532,7 @@ static int32_t taosSetClientCfg(SConfig *pCfg) {
|
|||
}
|
||||
|
||||
tstrncpy(tsSmlChildTableName, cfgGetItem(pCfg, "smlChildTableName")->str, TSDB_TABLE_NAME_LEN);
|
||||
tstrncpy(tsSmlTagName, cfgGetItem(pCfg, "smlTagNullName")->str, TSDB_COL_NAME_LEN);
|
||||
tstrncpy(tsSmlTagName, cfgGetItem(pCfg, "smlTagName")->str, TSDB_COL_NAME_LEN);
|
||||
tsSmlDataFormat = cfgGetItem(pCfg, "smlDataFormat")->bval;
|
||||
|
||||
tsShellActivityTimer = cfgGetItem(pCfg, "shellActivityTimer")->i32;
|
||||
|
|
|
|||
|
|
@ -932,6 +932,7 @@ int32_t tSerializeSStatusReq(void *buf, int32_t bufLen, SStatusReq *pReq) {
|
|||
if (tEncodeI64(&encoder, pReq->qload.numOfProcessedFetch) < 0) return -1;
|
||||
if (tEncodeI64(&encoder, pReq->qload.numOfProcessedDrop) < 0) return -1;
|
||||
if (tEncodeI64(&encoder, pReq->qload.numOfProcessedHb) < 0) return -1;
|
||||
if (tEncodeI64(&encoder, pReq->qload.numOfProcessedDelete) < 0) return -1;
|
||||
if (tEncodeI64(&encoder, pReq->qload.cacheDataSize) < 0) return -1;
|
||||
if (tEncodeI64(&encoder, pReq->qload.numOfQueryInQueue) < 0) return -1;
|
||||
if (tEncodeI64(&encoder, pReq->qload.numOfFetchInQueue) < 0) return -1;
|
||||
|
|
@ -1001,6 +1002,7 @@ int32_t tDeserializeSStatusReq(void *buf, int32_t bufLen, SStatusReq *pReq) {
|
|||
if (tDecodeI64(&decoder, &pReq->qload.numOfProcessedFetch) < 0) return -1;
|
||||
if (tDecodeI64(&decoder, &pReq->qload.numOfProcessedDrop) < 0) return -1;
|
||||
if (tDecodeI64(&decoder, &pReq->qload.numOfProcessedHb) < 0) return -1;
|
||||
if (tDecodeI64(&decoder, &pReq->qload.numOfProcessedDelete) < 0) return -1;
|
||||
if (tDecodeI64(&decoder, &pReq->qload.cacheDataSize) < 0) return -1;
|
||||
if (tDecodeI64(&decoder, &pReq->qload.numOfQueryInQueue) < 0) return -1;
|
||||
if (tDecodeI64(&decoder, &pReq->qload.numOfFetchInQueue) < 0) return -1;
|
||||
|
|
@ -2392,6 +2394,104 @@ int32_t tDeserializeSUserIndexRsp(void *buf, int32_t bufLen, SUserIndexRsp *pRsp
|
|||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSTableIndexReq(void *buf, int32_t bufLen, STableIndexReq *pReq) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
||||
if (tStartEncode(&encoder) < 0) return -1;
|
||||
if (tEncodeCStr(&encoder, pReq->tbFName) < 0) return -1;
|
||||
tEndEncode(&encoder);
|
||||
|
||||
int32_t tlen = encoder.pos;
|
||||
tEncoderClear(&encoder);
|
||||
return tlen;
|
||||
}
|
||||
|
||||
int32_t tDeserializeSTableIndexReq(void *buf, int32_t bufLen, STableIndexReq *pReq) {
|
||||
SDecoder decoder = {0};
|
||||
tDecoderInit(&decoder, buf, bufLen);
|
||||
|
||||
if (tStartDecode(&decoder) < 0) return -1;
|
||||
if (tDecodeCStrTo(&decoder, pReq->tbFName) < 0) return -1;
|
||||
tEndDecode(&decoder);
|
||||
|
||||
tDecoderClear(&decoder);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSTableIndexInfo(SEncoder *pEncoder, STableIndexInfo* pInfo) {
|
||||
if (tEncodeI8(pEncoder, pInfo->intervalUnit) < 0) return -1;
|
||||
if (tEncodeI8(pEncoder, pInfo->slidingUnit) < 0) return -1;
|
||||
if (tEncodeI64(pEncoder, pInfo->interval) < 0) return -1;
|
||||
if (tEncodeI64(pEncoder, pInfo->offset) < 0) return -1;
|
||||
if (tEncodeI64(pEncoder, pInfo->sliding) < 0) return -1;
|
||||
if (tEncodeI64(pEncoder, pInfo->dstTbUid) < 0) return -1;
|
||||
if (tEncodeI32(pEncoder, pInfo->dstVgId) < 0) return -1;
|
||||
if (tEncodeSEpSet(pEncoder, &pInfo->epSet) < 0) return -1;
|
||||
if (tEncodeCStr(pEncoder, pInfo->expr) < 0) return -1;
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSTableIndexRsp(void *buf, int32_t bufLen, const STableIndexRsp *pRsp) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
||||
if (tStartEncode(&encoder) < 0) return -1;
|
||||
int32_t num = taosArrayGetSize(pRsp->pIndex);
|
||||
if (tEncodeI32(&encoder, num) < 0) return -1;
|
||||
if (num > 0) {
|
||||
for (int32_t i = 0; i < num; ++i) {
|
||||
STableIndexInfo* pInfo = (STableIndexInfo*)taosArrayGet(pRsp->pIndex, i);
|
||||
if (tSerializeSTableIndexInfo(&encoder, pInfo) < 0) return -1;
|
||||
}
|
||||
}
|
||||
tEndEncode(&encoder);
|
||||
|
||||
int32_t tlen = encoder.pos;
|
||||
tEncoderClear(&encoder);
|
||||
return tlen;
|
||||
}
|
||||
|
||||
int32_t tDeserializeSTableIndexInfo(SDecoder *pDecoder, STableIndexInfo *pInfo) {
|
||||
if (tDecodeI8(pDecoder, &pInfo->intervalUnit) < 0) return -1;
|
||||
if (tDecodeI8(pDecoder, &pInfo->slidingUnit) < 0) return -1;
|
||||
if (tDecodeI64(pDecoder, &pInfo->interval) < 0) return -1;
|
||||
if (tDecodeI64(pDecoder, &pInfo->offset) < 0) return -1;
|
||||
if (tDecodeI64(pDecoder, &pInfo->sliding) < 0) return -1;
|
||||
if (tDecodeI64(pDecoder, &pInfo->dstTbUid) < 0) return -1;
|
||||
if (tDecodeI32(pDecoder, &pInfo->dstVgId) < 0) return -1;
|
||||
if (tDecodeSEpSet(pDecoder, &pInfo->epSet) < 0) return -1;
|
||||
if (tDecodeCStrAlloc(pDecoder, &pInfo->expr) < 0) return -1;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tDeserializeSTableIndexRsp(void *buf, int32_t bufLen, STableIndexRsp *pRsp) {
|
||||
SDecoder decoder = {0};
|
||||
tDecoderInit(&decoder, buf, bufLen);
|
||||
|
||||
if (tStartDecode(&decoder) < 0) return -1;
|
||||
int32_t num = 0;
|
||||
if (tDecodeI32(&decoder, &num) < 0) return -1;
|
||||
if (num > 0) {
|
||||
pRsp->pIndex = taosArrayInit(num, sizeof(STableIndexInfo));
|
||||
if (NULL == pRsp->pIndex) return -1;
|
||||
STableIndexInfo info;
|
||||
for (int32_t i = 0; i < num; ++i) {
|
||||
if (tDeserializeSTableIndexInfo(&decoder, &info) < 0) return -1;
|
||||
if (NULL == taosArrayPush(pRsp->pIndex, &info)) {
|
||||
taosMemoryFree(info.expr);
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
}
|
||||
tEndDecode(&decoder);
|
||||
|
||||
tDecoderClear(&decoder);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
||||
int32_t tSerializeSShowReq(void *buf, int32_t bufLen, SShowReq *pReq) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
|
@ -3256,6 +3356,89 @@ int32_t tDeserializeSKillTransReq(void *buf, int32_t bufLen, SKillTransReq *pReq
|
|||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSBalanceVgroupReq(void *buf, int32_t bufLen, SBalanceVgroupReq *pReq) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
||||
if (tStartEncode(&encoder) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->useless) < 0) return -1;
|
||||
tEndEncode(&encoder);
|
||||
|
||||
int32_t tlen = encoder.pos;
|
||||
tEncoderClear(&encoder);
|
||||
return tlen;
|
||||
}
|
||||
|
||||
int32_t tDeserializeSBalanceVgroupReq(void *buf, int32_t bufLen, SBalanceVgroupReq *pReq) {
|
||||
SDecoder decoder = {0};
|
||||
tDecoderInit(&decoder, buf, bufLen);
|
||||
|
||||
if (tStartDecode(&decoder) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->useless) < 0) return -1;
|
||||
tEndDecode(&decoder);
|
||||
|
||||
tDecoderClear(&decoder);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSMergeVgroupReq(void *buf, int32_t bufLen, SMergeVgroupReq *pReq) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
||||
if (tStartEncode(&encoder) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->vgId1) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->vgId2) < 0) return -1;
|
||||
tEndEncode(&encoder);
|
||||
|
||||
int32_t tlen = encoder.pos;
|
||||
tEncoderClear(&encoder);
|
||||
return tlen;
|
||||
}
|
||||
|
||||
int32_t tDeserializeSMergeVgroupReq(void *buf, int32_t bufLen, SMergeVgroupReq *pReq) {
|
||||
SDecoder decoder = {0};
|
||||
tDecoderInit(&decoder, buf, bufLen);
|
||||
|
||||
if (tStartDecode(&decoder) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->vgId1) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->vgId2) < 0) return -1;
|
||||
tEndDecode(&decoder);
|
||||
|
||||
tDecoderClear(&decoder);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSRedistributeVgroupReq(void *buf, int32_t bufLen, SRedistributeVgroupReq *pReq) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
||||
if (tStartEncode(&encoder) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->vgId) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->dnodeId1) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->dnodeId2) < 0) return -1;
|
||||
if (tEncodeI32(&encoder, pReq->dnodeId3) < 0) return -1;
|
||||
tEndEncode(&encoder);
|
||||
|
||||
int32_t tlen = encoder.pos;
|
||||
tEncoderClear(&encoder);
|
||||
return tlen;
|
||||
}
|
||||
|
||||
int32_t tDeserializeSRedistributeVgroupReq(void *buf, int32_t bufLen, SRedistributeVgroupReq *pReq) {
|
||||
SDecoder decoder = {0};
|
||||
tDecoderInit(&decoder, buf, bufLen);
|
||||
|
||||
if (tStartDecode(&decoder) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->vgId) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->dnodeId1) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->dnodeId2) < 0) return -1;
|
||||
if (tDecodeI32(&decoder, &pReq->dnodeId3) < 0) return -1;
|
||||
tEndDecode(&decoder);
|
||||
|
||||
tDecoderClear(&decoder);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tSerializeSDCreateMnodeReq(void *buf, int32_t bufLen, SDCreateMnodeReq *pReq) {
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
|
@ -3823,46 +4006,70 @@ int32_t tDecodeSVGetTsmaExpWndsRsp(SDecoder *pCoder, SVGetTsmaExpWndsRsp *pReq)
|
|||
return 0;
|
||||
}
|
||||
|
||||
int32_t tEncodeSVDeleteReq(SEncoder *pCoder, const SVDeleteReq *pReq) {
|
||||
if (tStartEncode(pCoder) < 0) return -1;
|
||||
|
||||
if (tEncodeI64(pCoder, pReq->delUid) < 0) return -1;
|
||||
if (tEncodeI64(pCoder, pReq->tbUid) < 0) return -1;
|
||||
if (tEncodeI8(pCoder, pReq->type) < 0) return -1;
|
||||
if (tEncodeI16v(pCoder, pReq->nWnds) < 0) return -1;
|
||||
if (tEncodeCStr(pCoder, pReq->tbFullName) < 0) return -1;
|
||||
if (tEncodeCStr(pCoder, pReq->subPlan) < 0) return -1;
|
||||
for (int16_t i = 0; i < pReq->nWnds; ++i) {
|
||||
if (tEncodeI64(pCoder, pReq->wnds[i].skey) < 0) return -1;
|
||||
if (tEncodeI64(pCoder, pReq->wnds[i].ekey) < 0) return -1;
|
||||
int32_t tSerializeSVDeleteReq(void *buf, int32_t bufLen, SVDeleteReq *pReq) {
|
||||
int32_t headLen = sizeof(SMsgHead);
|
||||
if (buf != NULL) {
|
||||
buf = (char *)buf + headLen;
|
||||
bufLen -= headLen;
|
||||
}
|
||||
|
||||
tEndEncode(pCoder);
|
||||
return 0;
|
||||
SEncoder encoder = {0};
|
||||
tEncoderInit(&encoder, buf, bufLen);
|
||||
|
||||
if (tStartEncode(&encoder) < 0) return -1;
|
||||
if (tEncodeU64(&encoder, pReq->sId) < 0) return -1;
|
||||
if (tEncodeU64(&encoder, pReq->queryId) < 0) return -1;
|
||||
if (tEncodeU64(&encoder, pReq->taskId) < 0) return -1;
|
||||
if (tEncodeU32(&encoder, pReq->sqlLen) < 0) return -1;
|
||||
if (tEncodeU32(&encoder, pReq->phyLen) < 0) return -1;
|
||||
if (tEncodeCStr(&encoder, pReq->sql) < 0) return -1;
|
||||
if (tEncodeCStr(&encoder, pReq->msg) < 0) return -1;
|
||||
tEndEncode(&encoder);
|
||||
|
||||
int32_t tlen = encoder.pos;
|
||||
tEncoderClear(&encoder);
|
||||
|
||||
if (buf != NULL) {
|
||||
SMsgHead *pHead = (SMsgHead *)((char *)buf - headLen);
|
||||
pHead->vgId = htonl(pReq->header.vgId);
|
||||
pHead->contLen = htonl(tlen + headLen);
|
||||
}
|
||||
|
||||
return tlen + headLen;
|
||||
}
|
||||
|
||||
int32_t tDecodeSVDeleteReq(SDecoder *pCoder, SVDeleteReq *pReq) {
|
||||
if (tStartDecode(pCoder) < 0) return -1;
|
||||
int32_t tDeserializeSVDeleteReq(void *buf, int32_t bufLen, SVDeleteReq *pReq) {
|
||||
int32_t headLen = sizeof(SMsgHead);
|
||||
|
||||
if (tDecodeI64(pCoder, &pReq->delUid) < 0) return -1;
|
||||
if (tDecodeI64(pCoder, &pReq->tbUid) < 0) return -1;
|
||||
if (tDecodeI8(pCoder, &pReq->type) < 0) return -1;
|
||||
if (tDecodeI16v(pCoder, &pReq->nWnds) < 0) return -1;
|
||||
if (tDecodeCStr(pCoder, &pReq->tbFullName) < 0) return -1;
|
||||
if (tDecodeCStr(pCoder, &pReq->subPlan) < 0) return -1;
|
||||
for (int16_t i = 0; i < pReq->nWnds; ++i) {
|
||||
if (tDecodeI64(pCoder, &pReq->wnds[i].skey) < 0) return -1;
|
||||
if (tDecodeI64(pCoder, &pReq->wnds[i].ekey) < 0) return -1;
|
||||
}
|
||||
SMsgHead *pHead = buf;
|
||||
pHead->vgId = pReq->header.vgId;
|
||||
pHead->contLen = pReq->header.contLen;
|
||||
|
||||
tEndDecode(pCoder);
|
||||
SDecoder decoder = {0};
|
||||
tDecoderInit(&decoder, (char *)buf + headLen, bufLen - headLen);
|
||||
|
||||
if (tStartDecode(&decoder) < 0) return -1;
|
||||
if (tDecodeU64(&decoder, &pReq->sId) < 0) return -1;
|
||||
if (tDecodeU64(&decoder, &pReq->queryId) < 0) return -1;
|
||||
if (tDecodeU64(&decoder, &pReq->taskId) < 0) return -1;
|
||||
if (tDecodeU32(&decoder, &pReq->sqlLen) < 0) return -1;
|
||||
if (tDecodeU32(&decoder, &pReq->phyLen) < 0) return -1;
|
||||
pReq->sql = taosMemoryCalloc(1, pReq->sqlLen + 1);
|
||||
if (NULL == pReq->sql) return -1;
|
||||
pReq->msg = taosMemoryCalloc(1, pReq->phyLen + 1);
|
||||
if (NULL == pReq->msg) return -1;
|
||||
if (tDecodeCStrTo(&decoder, pReq->sql) < 0) return -1;
|
||||
if (tDecodeCStrTo(&decoder, pReq->msg) < 0) return -1;
|
||||
|
||||
tEndDecode(&decoder);
|
||||
|
||||
tDecoderClear(&decoder);
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t tEncodeSVDeleteRsp(SEncoder *pCoder, const SVDeleteRsp *pReq) {
|
||||
if (tStartEncode(pCoder) < 0) return -1;
|
||||
|
||||
if (tEncodeI32(pCoder, pReq->code) < 0) return -1;
|
||||
if (tEncodeI64(pCoder, pReq->affectedRows) < 0) return -1;
|
||||
|
||||
tEndEncode(pCoder);
|
||||
|
|
@ -3872,7 +4079,6 @@ int32_t tEncodeSVDeleteRsp(SEncoder *pCoder, const SVDeleteRsp *pReq) {
|
|||
int32_t tDecodeSVDeleteRsp(SDecoder *pCoder, SVDeleteRsp *pReq) {
|
||||
if (tStartDecode(pCoder) < 0) return -1;
|
||||
|
||||
if (tDecodeI32(pCoder, &pReq->code) < 0) return -1;
|
||||
if (tDecodeI64(pCoder, &pReq->affectedRows) < 0) return -1;
|
||||
|
||||
tEndDecode(pCoder);
|
||||
|
|
|
|||
|
|
@ -18,7 +18,7 @@
|
|||
#include "tcompression.h"
|
||||
#include "trow.h"
|
||||
|
||||
const int32_t TYPE_BYTES[15] = {
|
||||
const int32_t TYPE_BYTES[16] = {
|
||||
-1, // TSDB_DATA_TYPE_NULL
|
||||
CHAR_BYTES, // TSDB_DATA_TYPE_BOOL
|
||||
CHAR_BYTES, // TSDB_DATA_TYPE_TINYINT
|
||||
|
|
@ -34,6 +34,7 @@ const int32_t TYPE_BYTES[15] = {
|
|||
SHORT_BYTES, // TSDB_DATA_TYPE_USMALLINT
|
||||
INT_BYTES, // TSDB_DATA_TYPE_UINT
|
||||
sizeof(uint64_t), // TSDB_DATA_TYPE_UBIGINT
|
||||
TSDB_MAX_JSON_TAG_LEN, // TSDB_DATA_TYPE_JSON
|
||||
};
|
||||
|
||||
#define DO_STATICS(__sum, __min, __max, __minIndex, __maxIndex, _list, _index) \
|
||||
|
|
|
|||
|
|
@ -5,7 +5,12 @@ MESSAGE(STATUS "build parser unit test")
|
|||
SET(CMAKE_CXX_STANDARD 11)
|
||||
AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)
|
||||
|
||||
ADD_EXECUTABLE(commonTest ${SOURCE_LIST})
|
||||
ADD_EXECUTABLE(commonTest "")
|
||||
TARGET_SOURCES(
|
||||
commonTest
|
||||
PRIVATE
|
||||
"commonTests.cpp"
|
||||
)
|
||||
TARGET_LINK_LIBRARIES(
|
||||
commonTest
|
||||
PUBLIC os util common gtest
|
||||
|
|
@ -24,7 +29,12 @@ target_sources(
|
|||
PRIVATE
|
||||
"dataformatTest.cpp"
|
||||
)
|
||||
target_link_libraries(dataformatTest gtest gtest_main util)
|
||||
target_link_libraries(dataformatTest gtest gtest_main util common)
|
||||
target_include_directories(
|
||||
dataformatTest
|
||||
PUBLIC "${TD_SOURCE_DIR}/include/common"
|
||||
PUBLIC "${TD_SOURCE_DIR}/include/util"
|
||||
)
|
||||
|
||||
# tmsg test
|
||||
# add_executable(tmsgTest "")
|
||||
|
|
|
|||
|
|
@ -1 +1,481 @@
|
|||
#include "gtest/gtest.h"
|
||||
/*
|
||||
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
|
||||
*
|
||||
* This program is free software: you can use, redistribute, and/or modify
|
||||
* it under the terms of the GNU Affero General Public License, version 3
|
||||
* or later ("AGPL"), as published by the Free Software Foundation.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful, but WITHOUT
|
||||
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
||||
* FITNESS FOR A PARTICULAR PURPOSE.
|
||||
*
|
||||
* You should have received a copy of the GNU Affero General Public License
|
||||
* along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
#include <gtest/gtest.h>
|
||||
|
||||
#include <taoserror.h>
|
||||
#include <tdataformat.h>
|
||||
#include <tglobal.h>
|
||||
#include <tmsg.h>
|
||||
#include <iostream>
|
||||
|
||||
#pragma GCC diagnostic push
|
||||
#pragma GCC diagnostic ignored "-Wwrite-strings"
|
||||
#pragma GCC diagnostic ignored "-Wunused-function"
|
||||
#pragma GCC diagnostic ignored "-Wunused-variable"
|
||||
#pragma GCC diagnostic ignored "-Wsign-compare"
|
||||
|
||||
#define NONE_CSTR "no"
|
||||
#define NULL_CSTR "nu"
|
||||
#define NONE_LEN 2
|
||||
#define NULL_LEN 2
|
||||
const static int16_t MAX_COLS = 14;
|
||||
|
||||
int main(int argc, char **argv) {
|
||||
testing::InitGoogleTest(&argc, argv);
|
||||
return RUN_ALL_TESTS();
|
||||
}
|
||||
|
||||
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, c9 bool
|
||||
STSchema *genSTSchema(int16_t nCols) {
|
||||
EXPECT_LE(nCols, MAX_COLS);
|
||||
SSchema *pSchema = (SSchema *)taosMemoryCalloc(nCols, sizeof(SSchema));
|
||||
EXPECT_NE(pSchema, nullptr);
|
||||
|
||||
for (int16_t i = 0; i < nCols; ++i) {
|
||||
pSchema[i].colId = PRIMARYKEY_TIMESTAMP_COL_ID + i;
|
||||
char colName[TSDB_COL_NAME_LEN] = {0};
|
||||
snprintf(colName, TSDB_COL_NAME_LEN, "c%" PRIi16, i);
|
||||
strncpy(pSchema[i].name, colName, TSDB_COL_NAME_LEN);
|
||||
|
||||
switch (i) {
|
||||
case 0: {
|
||||
pSchema[0].type = TSDB_DATA_TYPE_TIMESTAMP;
|
||||
pSchema[0].bytes = TYPE_BYTES[pSchema[0].type];
|
||||
} break;
|
||||
case 1: {
|
||||
pSchema[1].type = TSDB_DATA_TYPE_INT;
|
||||
pSchema[1].bytes = TYPE_BYTES[pSchema[1].type];
|
||||
;
|
||||
} break;
|
||||
case 2: {
|
||||
pSchema[2].type = TSDB_DATA_TYPE_BIGINT;
|
||||
pSchema[2].bytes = TYPE_BYTES[pSchema[2].type];
|
||||
} break;
|
||||
case 3: {
|
||||
pSchema[3].type = TSDB_DATA_TYPE_FLOAT;
|
||||
pSchema[3].bytes = TYPE_BYTES[pSchema[3].type];
|
||||
} break;
|
||||
case 4: {
|
||||
pSchema[4].type = TSDB_DATA_TYPE_DOUBLE;
|
||||
pSchema[4].bytes = TYPE_BYTES[pSchema[4].type];
|
||||
} break;
|
||||
case 5: {
|
||||
pSchema[5].type = TSDB_DATA_TYPE_BINARY;
|
||||
pSchema[5].bytes = 12;
|
||||
} break;
|
||||
case 6: {
|
||||
pSchema[6].type = TSDB_DATA_TYPE_NCHAR;
|
||||
pSchema[6].bytes = 42;
|
||||
} break;
|
||||
case 7: {
|
||||
pSchema[7].type = TSDB_DATA_TYPE_TINYINT;
|
||||
pSchema[7].bytes = TYPE_BYTES[pSchema[7].type];
|
||||
} break;
|
||||
case 8: {
|
||||
pSchema[8].type = TSDB_DATA_TYPE_SMALLINT;
|
||||
pSchema[8].bytes = TYPE_BYTES[pSchema[8].type];
|
||||
} break;
|
||||
case 9: {
|
||||
pSchema[9].type = TSDB_DATA_TYPE_BOOL;
|
||||
pSchema[9].bytes = TYPE_BYTES[pSchema[9].type];
|
||||
} break;
|
||||
case 10: {
|
||||
pSchema[10].type = TSDB_DATA_TYPE_UTINYINT;
|
||||
pSchema[10].bytes = TYPE_BYTES[pSchema[10].type];
|
||||
} break;
|
||||
case 11: {
|
||||
pSchema[11].type = TSDB_DATA_TYPE_USMALLINT;
|
||||
pSchema[11].bytes = TYPE_BYTES[pSchema[11].type];
|
||||
} break;
|
||||
case 12: {
|
||||
pSchema[12].type = TSDB_DATA_TYPE_UINT;
|
||||
pSchema[12].bytes = TYPE_BYTES[pSchema[12].type];
|
||||
} break;
|
||||
case 13: {
|
||||
pSchema[13].type = TSDB_DATA_TYPE_UBIGINT;
|
||||
pSchema[13].bytes = TYPE_BYTES[pSchema[13].type];
|
||||
} break;
|
||||
|
||||
default:
|
||||
ASSERT(0);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
STSchema *pResult = NULL;
|
||||
pResult = tdGetSTSChemaFromSSChema(&pSchema, nCols);
|
||||
|
||||
taosMemoryFree(pSchema);
|
||||
return pResult;
|
||||
}
|
||||
|
||||
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint, c9 bool
|
||||
static int32_t genTestData(const char **data, int16_t nCols, SArray **pArray) {
|
||||
if (!(*pArray)) {
|
||||
*pArray = taosArrayInit(nCols, sizeof(SColVal));
|
||||
if (!(*pArray)) return -1;
|
||||
}
|
||||
|
||||
for (int16_t i = 0; i < nCols; ++i) {
|
||||
SColVal colVal = {0};
|
||||
colVal.cid = PRIMARYKEY_TIMESTAMP_COL_ID + i;
|
||||
if (strncasecmp(data[i], NONE_CSTR, NONE_LEN) == 0) {
|
||||
colVal.isNone = 1;
|
||||
taosArrayPush(*pArray, &colVal);
|
||||
continue;
|
||||
} else if (strncasecmp(data[i], NULL_CSTR, NULL_LEN) == 0) {
|
||||
colVal.isNull = 1;
|
||||
taosArrayPush(*pArray, &colVal);
|
||||
continue;
|
||||
}
|
||||
|
||||
switch (i) {
|
||||
case 0:
|
||||
sscanf(data[i], "%" PRIi64, &colVal.value.ts);
|
||||
break;
|
||||
case 1: {
|
||||
sscanf(data[i], "%" PRIi32, &colVal.value.i32);
|
||||
} break;
|
||||
case 2:
|
||||
sscanf(data[i], "%" PRIi64, &colVal.value.i64);
|
||||
break;
|
||||
case 3:
|
||||
sscanf(data[i], "%f", &colVal.value.f);
|
||||
break;
|
||||
case 4:
|
||||
sscanf(data[i], "%lf", &colVal.value.d);
|
||||
break;
|
||||
case 5: {
|
||||
int16_t dataLen = strlen(data[i]) + 1;
|
||||
colVal.value.nData = dataLen < 10 ? dataLen : 10;
|
||||
colVal.value.pData = (uint8_t *)data[i];
|
||||
} break;
|
||||
case 6: {
|
||||
int16_t dataLen = strlen(data[i]) + 1;
|
||||
colVal.value.nData = dataLen < 40 ? dataLen : 40;
|
||||
colVal.value.pData = (uint8_t *)data[i]; // just for test, not real nchar
|
||||
} break;
|
||||
case 7:
|
||||
case 9: {
|
||||
int32_t d8;
|
||||
sscanf(data[i], "%" PRId32, &d8);
|
||||
colVal.value.i8 = (int8_t)d8;
|
||||
} break;
|
||||
case 8: {
|
||||
int32_t d16;
|
||||
sscanf(data[i], "%" PRId32, &d16);
|
||||
colVal.value.i16 = (int16_t)d16;
|
||||
} break;
|
||||
case 10: {
|
||||
uint32_t u8;
|
||||
sscanf(data[i], "%" PRId32, &u8);
|
||||
colVal.value.u8 = (uint8_t)u8;
|
||||
} break;
|
||||
case 11: {
|
||||
uint32_t u16;
|
||||
sscanf(data[i], "%" PRId32, &u16);
|
||||
colVal.value.u16 = (uint16_t)u16;
|
||||
} break;
|
||||
case 12: {
|
||||
sscanf(data[i], "%" PRIu32, &colVal.value.u32);
|
||||
} break;
|
||||
case 13: {
|
||||
sscanf(data[i], "%" PRIu64, &colVal.value.u64);
|
||||
} break;
|
||||
default:
|
||||
ASSERT(0);
|
||||
}
|
||||
taosArrayPush(*pArray, &colVal);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
int32_t debugPrintSColVal(SColVal *cv, int8_t type) {
|
||||
if (cv->isNone) {
|
||||
printf("None ");
|
||||
return 0;
|
||||
}
|
||||
if (cv->isNull) {
|
||||
printf("Null ");
|
||||
return 0;
|
||||
}
|
||||
switch (type) {
|
||||
case TSDB_DATA_TYPE_BOOL:
|
||||
printf("%s ", cv->value.i8 == 0 ? "false" : "true");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_TINYINT:
|
||||
printf("%" PRIi8 " ", cv->value.i8);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_SMALLINT:
|
||||
printf("%" PRIi16 " ", cv->value.i16);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_INT:
|
||||
printf("%" PRIi32 " ", cv->value.i32);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_BIGINT:
|
||||
printf("%" PRIi64 " ", cv->value.i64);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_FLOAT:
|
||||
printf("%f ", cv->value.f);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_DOUBLE:
|
||||
printf("%lf ", cv->value.d);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_VARCHAR: {
|
||||
char tv[15] = {0};
|
||||
snprintf(tv, 15, "%s", cv->value.pData);
|
||||
printf("%s ", tv);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_TIMESTAMP:
|
||||
printf("%" PRIi64 " ", cv->value.i64);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_NCHAR: {
|
||||
char tv[15] = {0};
|
||||
snprintf(tv, 15, "%s", cv->value.pData);
|
||||
printf("%s ", tv);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_UTINYINT:
|
||||
printf("%" PRIu8 " ", cv->value.u8);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_USMALLINT:
|
||||
printf("%" PRIu16 " ", cv->value.u16);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_UINT:
|
||||
printf("%" PRIu32 " ", cv->value.u32);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_UBIGINT:
|
||||
printf("%" PRIu64 " ", cv->value.u64);
|
||||
break;
|
||||
case TSDB_DATA_TYPE_JSON:
|
||||
printf("JSON ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_VARBINARY:
|
||||
printf("VARBIN ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_DECIMAL:
|
||||
printf("DECIMAL ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_BLOB:
|
||||
printf("BLOB ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_MEDIUMBLOB:
|
||||
printf("MedBLOB ");
|
||||
break;
|
||||
// case TSDB_DATA_TYPE_BINARY:
|
||||
// printf("BINARY ");
|
||||
// break;
|
||||
case TSDB_DATA_TYPE_MAX:
|
||||
printf("UNDEF ");
|
||||
break;
|
||||
default:
|
||||
printf("UNDEF ");
|
||||
break;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
void debugPrintTSRow(STSRow2 *row, STSchema *pTSchema, const char *tags, int32_t ln) {
|
||||
printf("%s:%d %s:v%d:%d ", tags, ln, (row->flags & 0xf0) ? "KV" : "TP", row->sver, row->nData);
|
||||
for (int16_t i = 0; i < schemaNCols(pTSchema); ++i) {
|
||||
SColVal cv = {0};
|
||||
tTSRowGet(row, pTSchema, i, &cv);
|
||||
debugPrintSColVal(&cv, pTSchema->columns[i].type);
|
||||
}
|
||||
printf("\n");
|
||||
fflush(stdout);
|
||||
}
|
||||
|
||||
static int32_t checkSColVal(const char *rawVal, SColVal *cv, int8_t type) {
|
||||
ASSERT(rawVal);
|
||||
|
||||
if (cv->isNone) {
|
||||
EXPECT_STRCASEEQ(rawVal, NONE_CSTR);
|
||||
return 0;
|
||||
}
|
||||
if (cv->isNull) {
|
||||
EXPECT_STRCASEEQ(rawVal, NULL_CSTR);
|
||||
return 0;
|
||||
}
|
||||
|
||||
SValue rawSVal = {0};
|
||||
switch (type) {
|
||||
case TSDB_DATA_TYPE_BOOL:
|
||||
case TSDB_DATA_TYPE_TINYINT: {
|
||||
int32_t d8;
|
||||
sscanf(rawVal, "%" PRId32, &d8);
|
||||
EXPECT_EQ(cv->value.i8, (int8_t)d8);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_SMALLINT: {
|
||||
int32_t d16;
|
||||
sscanf(rawVal, "%" PRId32, &d16);
|
||||
EXPECT_EQ(cv->value.i16, (int16_t)d16);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_INT: {
|
||||
sscanf(rawVal, "%" PRId32, &rawSVal.i32);
|
||||
EXPECT_EQ(cv->value.i32, rawSVal.i32);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_BIGINT: {
|
||||
sscanf(rawVal, "%" PRIi64, &rawSVal.i64);
|
||||
EXPECT_EQ(cv->value.i64, rawSVal.i64);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_FLOAT: {
|
||||
sscanf(rawVal, "%f", &rawSVal.f);
|
||||
EXPECT_FLOAT_EQ(cv->value.f, rawSVal.f);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_DOUBLE: {
|
||||
sscanf(rawVal, "%lf", &rawSVal.d);
|
||||
EXPECT_DOUBLE_EQ(cv->value.d, rawSVal.d);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_VARCHAR: {
|
||||
EXPECT_STRCASEEQ(rawVal, (const char *)cv->value.pData);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_TIMESTAMP: {
|
||||
sscanf(rawVal, "%" PRIi64, &rawSVal.ts);
|
||||
EXPECT_DOUBLE_EQ(cv->value.ts, rawSVal.ts);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_NCHAR: {
|
||||
EXPECT_STRCASEEQ(rawVal, (const char *)cv->value.pData); // informal nchar comparsion
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_UTINYINT: {
|
||||
uint32_t u8;
|
||||
sscanf(rawVal, "%" PRIu32, &u8);
|
||||
EXPECT_EQ(cv->value.u8, (uint8_t)u8);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_USMALLINT: {
|
||||
uint32_t u16;
|
||||
sscanf(rawVal, "%" PRIu32, &u16);
|
||||
EXPECT_EQ(cv->value.u16, (uint16_t)u16);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_UINT: {
|
||||
sscanf(rawVal, "%" PRIu32, &rawSVal.u32);
|
||||
EXPECT_EQ(cv->value.u32, rawSVal.u32);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_UBIGINT: {
|
||||
sscanf(rawVal, "%" PRIu64, &rawSVal.u64);
|
||||
EXPECT_EQ(cv->value.u64, rawSVal.u64);
|
||||
} break;
|
||||
case TSDB_DATA_TYPE_JSON:
|
||||
printf("JSON ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_VARBINARY:
|
||||
printf("VARBIN ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_DECIMAL:
|
||||
printf("DECIMAL ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_BLOB:
|
||||
printf("BLOB ");
|
||||
break;
|
||||
case TSDB_DATA_TYPE_MEDIUMBLOB:
|
||||
printf("MedBLOB ");
|
||||
break;
|
||||
// case TSDB_DATA_TYPE_BINARY:
|
||||
// printf("BINARY ");
|
||||
// break;
|
||||
case TSDB_DATA_TYPE_MAX:
|
||||
printf("UNDEF ");
|
||||
break;
|
||||
default:
|
||||
printf("UNDEF ");
|
||||
break;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
static void checkTSRow(const char **data, STSRow2 *row, STSchema *pTSchema) {
|
||||
for (int16_t i = 0; i < schemaNCols(pTSchema); ++i) {
|
||||
SColVal cv = {0};
|
||||
tTSRowGet(row, pTSchema, i, &cv);
|
||||
checkSColVal(data[i], &cv, pTSchema->columns[i].type);
|
||||
}
|
||||
}
|
||||
|
||||
TEST(testCase, AllNormTest) {
|
||||
int16_t nCols = 1;
|
||||
STSRow2 *row = nullptr;
|
||||
SArray *pArray = taosArrayInit(nCols, sizeof(SColVal));
|
||||
EXPECT_NE(pArray, nullptr);
|
||||
|
||||
STSchema *pTSchema = genSTSchema(nCols);
|
||||
EXPECT_NE(pTSchema, nullptr);
|
||||
|
||||
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint,
|
||||
// c9 bool
|
||||
char *data[10] = {"1653694220000", "10", "20", "10.1", "10.1", "binary10", "nchar10", "10", "10", "1"};
|
||||
|
||||
genTestData((const char **)&data, nCols, &pArray);
|
||||
|
||||
tTSRowNew(NULL, pArray, pTSchema, &row);
|
||||
|
||||
debugPrintTSRow(row, pTSchema, __func__, __LINE__);
|
||||
checkTSRow((const char **)&data, row, pTSchema);
|
||||
|
||||
taosArrayDestroy(pArray);
|
||||
taosMemoryFree(pTSchema);
|
||||
}
|
||||
|
||||
#if 1
|
||||
TEST(testCase, NoneTest) {
|
||||
const static int nCols = 14;
|
||||
const static int nRows = 20;
|
||||
STSRow2 *row = nullptr;
|
||||
SArray *pArray = taosArrayInit(nCols, sizeof(SColVal));
|
||||
EXPECT_NE(pArray, nullptr);
|
||||
|
||||
STSchema *pTSchema = genSTSchema(nCols);
|
||||
EXPECT_NE(pTSchema, nullptr);
|
||||
|
||||
// ts timestamp, c1 int, c2 bigint, c3 float, c4 double, c5 binary(10), c6 nchar(10), c7 tinyint, c8 smallint,
|
||||
// c9 bool c10 tinyint unsigned, c11 smallint unsigned, c12 int unsigned, c13 bigint unsigned
|
||||
const char *data[nRows][nCols] = {
|
||||
{"1653694220000", "no", "20", "10.1", "10.1", "binary10", "no", "10", "10", "nu", "10", "20", "30", "40"},
|
||||
{"1653694220001", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no", "no"},
|
||||
{"1653694220002", "no", "no", "no", "no", "no", "nu", "no", "no", "no", "no", "no", "no", "nu"},
|
||||
{"1653694220003", "nu", "no", "no", "no", "no", "nu", "no", "no", "no", "no", "no", "no", "no"},
|
||||
{"1653694220004", "no", "20", "no", "no", "no", "nchar10", "no", "no", "no", "no", "no", "no", "no"},
|
||||
{"1653694220005", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu"},
|
||||
{"1653694220006", "no", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu"},
|
||||
{"1653694220007", "no", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "no"},
|
||||
{"1653694220008", "no", "nu", "nu", "nu", "binary10", "nu", "nu", "nu", "nu", "nu", "nu", "nu", "no"},
|
||||
{"1653694220009", "no", "nu", "nu", "nu", "binary10", "nu", "nu", "10", "no", "nu", "nu", "nu", "100"},
|
||||
{"1653694220010", "-1", "-1", "-1", "-1", "binary10", "nu", "-1", "0", "0", "0", "0", "0", "0"},
|
||||
{"1653694220011", "-2147483648", "nu", "nu", "nu", "biy10", "nu", "nu", "32767", "no", "nu", "nu", "nu", "100"},
|
||||
{"1653694220012", "2147483647", "nu", "nu", "nu", "ary10", "nu", "nu", "-32768", "no", "nu", "nu", "nu", "100"},
|
||||
{"1653694220013", "no", "-9223372036854775818", "nu", "nu", "b1", "nu", "nu", "10", "no", "nu", "nu", "nu", "nu"},
|
||||
{"1653694220014", "no", "nu", "nu", "nu", "b0", "nu", "nu", "10", "no", "nu", "nu", "nu", "9223372036854775808"},
|
||||
{"1653694220015", "no", "nu", "nu", "nu", "binary30", "char4", "nu", "10", "no", "nu", "nu", "nu",
|
||||
"18446744073709551615"},
|
||||
{"1653694220016", "2147483647", "nu", "nu", "nu", "bin50", "nu", "nu", "10", "no", "nu", "nu", "nu", "100"},
|
||||
{"1653694220017", "2147483646", "0", "0", "0", "binary10", "0", "0", "0", "0", "255", "0", "0", "0"},
|
||||
{"1653694220018", "no", "-9223372036854775808", "nu", "nu", "binary10", "nu", "nu", "10", "no", "nu", "nu",
|
||||
"4294967295", "100"},
|
||||
{"1653694220019", "no", "9223372036854775807", "nu", "nu", "bin10", "nu", "nu", "10", "no", "254", "nu", "nu",
|
||||
"no"}};
|
||||
|
||||
|
||||
for (int r = 0; r < nRows; ++r) {
|
||||
genTestData((const char **)&data[r], nCols, &pArray);
|
||||
tTSRowNew(NULL, pArray, pTSchema, &row);
|
||||
debugPrintTSRow(row, pTSchema, __func__, __LINE__); // debug print
|
||||
checkTSRow((const char **)&data[r], row, pTSchema); // check
|
||||
taosMemoryFreeClear(row);
|
||||
taosArrayClear(pArray);
|
||||
}
|
||||
|
||||
taosArrayDestroy(pArray);
|
||||
taosMemoryFree(pTSchema);
|
||||
}
|
||||
#endif
|
||||
|
|
@ -181,6 +181,7 @@ SArray *mmGetMsgHandles() {
|
|||
if (dmSetMgmtHandle(pArray, TDMT_MND_DROP_SMA, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MND_CREATE_STREAM, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MND_GET_INDEX, mmPutNodeMsgToReadQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MND_GET_TABLE_INDEX, mmPutNodeMsgToReadQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MND_CREATE_TOPIC, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MND_ALTER_TOPIC, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MND_DROP_TOPIC, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
|
|
@ -210,7 +211,7 @@ SArray *mmGetMsgHandles() {
|
|||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_VG_CHANGE_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_VG_DELETE_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TASK, mmPutNodeMsgToQueryQueue, 1) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TASK_DEPLOY_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIRM_RSP, mmPutNodeMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
|
|
|
|||
|
|
@ -94,9 +94,10 @@ SArray *smGetMsgHandles() {
|
|||
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MON_SM_INFO, smPutNodeMsgToMonitorQueue, 0) == NULL) goto _OVER;
|
||||
|
||||
// Requests handled by SNODE
|
||||
if (dmSetMgmtHandle(pArray, TDMT_SND_TASK_DEPLOY, smPutNodeMsgToMgmtQueue, 0) == NULL) goto _OVER;
|
||||
/*if (dmSetMgmtHandle(pArray, TDMT_SND_TASK_EXEC, smPutNodeMsgToExecQueue, 0) == NULL) goto _OVER;*/
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, smPutNodeMsgToMgmtQueue, 1) == NULL) goto _OVER;
|
||||
|
||||
code = 0;
|
||||
_OVER:
|
||||
|
|
|
|||
|
|
@ -34,6 +34,7 @@ typedef struct SVnodeMgmt {
|
|||
SQWorkerPool fetchPool;
|
||||
SWWorkerPool syncPool;
|
||||
SWWorkerPool writePool;
|
||||
SWWorkerPool applyPool;
|
||||
SWWorkerPool mergePool;
|
||||
SSingleWorker mgmtWorker;
|
||||
SSingleWorker monitorWorker;
|
||||
|
|
@ -52,10 +53,9 @@ typedef struct {
|
|||
|
||||
typedef struct {
|
||||
int32_t vgId;
|
||||
int32_t refCount;
|
||||
int32_t vgVersion;
|
||||
int32_t refCount;
|
||||
int8_t dropped;
|
||||
int8_t accessState;
|
||||
char *path;
|
||||
SVnode *pImpl;
|
||||
STaosQueue *pWriteQ;
|
||||
|
|
|
|||
|
|
@ -210,8 +210,7 @@ int32_t vmProcessCreateVnodeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
|
|||
return -1;
|
||||
}
|
||||
|
||||
dDebug("vgId:%d, start to create vnode, tsma:%d standby:%d", createReq.vgId, createReq.isTsma,
|
||||
createReq.standby);
|
||||
dDebug("vgId:%d, start to create vnode, tsma:%d standby:%d", createReq.vgId, createReq.isTsma, createReq.standby);
|
||||
vmGenerateVnodeCfg(&createReq, &vnodeCfg);
|
||||
|
||||
if (vmTsmaAdjustDays(&vnodeCfg, &createReq) < 0) {
|
||||
|
|
@ -324,7 +323,6 @@ SArray *vmGetMsgHandles() {
|
|||
if (dmSetMgmtHandle(pArray, TDMT_MON_VM_INFO, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_MON_VM_LOAD, vmPutMsgToMonitorQueue, 0) == NULL) goto _OVER;
|
||||
|
||||
// Requests handled by VNODE
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_SUBMIT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_CONTINUE, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
|
||||
|
|
@ -333,11 +331,6 @@ SArray *vmGetMsgHandles() {
|
|||
if (dmSetMgmtHandle(pArray, TDMT_VND_UPDATE_TAG_VAL, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TABLE_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TABLES_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_CONSUME, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_QUERY, vmPutMsgToQueryQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_CONNECT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_DISCONNECT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
// if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_SET_CUR, vmPutMsgToWriteQueue, 0)== NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_CANCEL_TASK, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TASK, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_STB, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
|
|
@ -352,12 +345,16 @@ SArray *vmGetMsgHandles() {
|
|||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_VG_CHANGE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_MQ_VG_DELETE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_CONSUME, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_DELETE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_QUERY_HEARTBEAT, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TRIGGER, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TASK_RUN, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TASK_DISPATCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_TASK_RECOVER, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DEPLOY, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH_RSP, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RECOVER_RSP, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
|
||||
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
|
||||
|
|
|
|||
|
|
@ -49,10 +49,9 @@ int32_t vmOpenVnode(SVnodeMgmt *pMgmt, SWrapperCfg *pCfg, SVnode *pImpl) {
|
|||
}
|
||||
|
||||
pVnode->vgId = pCfg->vgId;
|
||||
pVnode->refCount = 0;
|
||||
pVnode->vgVersion = pCfg->vgVersion;
|
||||
pVnode->refCount = 0;
|
||||
pVnode->dropped = 0;
|
||||
pVnode->accessState = TSDB_VN_ALL_ACCCESS;
|
||||
pVnode->path = tstrdup(pCfg->path);
|
||||
pVnode->pImpl = pImpl;
|
||||
|
||||
|
|
@ -96,7 +95,7 @@ void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
|
|||
dDebug("vgId:%d, vnode is closed", pVnode->vgId);
|
||||
|
||||
if (pVnode->dropped) {
|
||||
dDebug("vgId:%d, vnode is destroyed for dropped:%d", pVnode->vgId, pVnode->dropped);
|
||||
dInfo("vgId:%d, vnode is destroyed, dropped:%d", pVnode->vgId, pVnode->dropped);
|
||||
snprintf(path, TSDB_FILENAME_LEN, "vnode%svnode%d", TD_DIRSEP, pVnode->vgId);
|
||||
vnodeDestroy(path, pMgmt->pTfs);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -16,10 +16,8 @@
|
|||
#define _DEFAULT_SOURCE
|
||||
#include "vmInt.h"
|
||||
|
||||
#include "sync.h"
|
||||
#include "syncTools.h"
|
||||
|
||||
static inline void vmSendRsp(SRpcMsg *pMsg, int32_t code) {
|
||||
if (pMsg->info.handle == NULL) return;
|
||||
SRpcMsg rsp = {
|
||||
.code = code,
|
||||
.pCont = pMsg->info.rsp,
|
||||
|
|
@ -53,9 +51,9 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
|
|||
}
|
||||
|
||||
if (IsReq(pMsg)) {
|
||||
if (code != 0 && terrno != 0) {
|
||||
dError("msg:%p failed to process since %s", pMsg, terrstr());
|
||||
code = terrno;
|
||||
if (code != 0) {
|
||||
if (terrno != 0) code = terrno;
|
||||
dError("msg:%p, failed to process since %s", pMsg, terrstr());
|
||||
}
|
||||
vmSendRsp(pMsg, code);
|
||||
}
|
||||
|
|
@ -97,110 +95,6 @@ static void vmProcessFetchQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
|
|||
taosFreeQitem(pMsg);
|
||||
}
|
||||
|
||||
static void vmProcessWriteQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
|
||||
int32_t code = 0;
|
||||
SRpcMsg *pMsg = NULL;
|
||||
SVnodeObj *pVnode = pInfo->ahandle;
|
||||
int64_t sync = vnodeGetSyncHandle(pVnode->pImpl);
|
||||
SArray *pArray = taosArrayInit(numOfMsgs, sizeof(SRpcMsg **));
|
||||
|
||||
for (int32_t m = 0; m < numOfMsgs; m++) {
|
||||
if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
|
||||
dTrace("vgId:%d, msg:%p get from vnode-write queue", pVnode->vgId, pMsg);
|
||||
|
||||
if (taosArrayPush(pArray, &pMsg) == NULL) {
|
||||
dError("vgId:%d, failed to push msg:%p to vnode-write array", pVnode->vgId, pMsg);
|
||||
vmSendRsp(pMsg, TSDB_CODE_OUT_OF_MEMORY);
|
||||
}
|
||||
}
|
||||
|
||||
for (int32_t m = 0; m < taosArrayGetSize(pArray); m++) {
|
||||
pMsg = *(SRpcMsg **)taosArrayGet(pArray, m);
|
||||
code = vnodePreprocessReq(pVnode->pImpl, pMsg);
|
||||
|
||||
if (code == TSDB_CODE_ACTION_IN_PROGRESS) {
|
||||
dTrace("vgId:%d, msg:%p in progress and no rsp", pVnode->vgId, pMsg);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (pMsg->msgType != TDMT_VND_ALTER_REPLICA) {
|
||||
code = syncPropose(sync, pMsg, false);
|
||||
}
|
||||
|
||||
if (code == TAOS_SYNC_PROPOSE_SUCCESS) {
|
||||
dTrace("vgId:%d, msg:%p is proposed and no rsp", pVnode->vgId, pMsg);
|
||||
continue;
|
||||
} else if (code == TAOS_SYNC_PROPOSE_NOT_LEADER) {
|
||||
SEpSet newEpSet = {0};
|
||||
syncGetEpSet(sync, &newEpSet);
|
||||
SEp *pEp = &newEpSet.eps[newEpSet.inUse];
|
||||
if (pEp->port == tsServerPort && strcmp(pEp->fqdn, tsLocalFqdn) == 0) {
|
||||
newEpSet.inUse = (newEpSet.inUse + 1) % newEpSet.numOfEps;
|
||||
}
|
||||
|
||||
dTrace("vgId:%d, msg:%p is redirect since not leader, numOfEps:%d inUse:%d", pVnode->vgId, pMsg,
|
||||
newEpSet.numOfEps, newEpSet.inUse);
|
||||
for (int32_t i = 0; i < newEpSet.numOfEps; ++i) {
|
||||
dTrace("vgId:%d, msg:%p ep:%s:%u", pVnode->vgId, pMsg, newEpSet.eps[i].fqdn, newEpSet.eps[i].port);
|
||||
}
|
||||
|
||||
SRpcMsg rsp = {.code = TSDB_CODE_RPC_REDIRECT, .info = pMsg->info};
|
||||
tmsgSendRedirectRsp(&rsp, &newEpSet);
|
||||
} else {
|
||||
dError("vgId:%d, msg:%p failed to propose write since %s, code:0x%x", pVnode->vgId, pMsg, tstrerror(code), code);
|
||||
vmSendRsp(pMsg, code);
|
||||
}
|
||||
}
|
||||
|
||||
for (int32_t i = 0; i < numOfMsgs; i++) {
|
||||
pMsg = *(SRpcMsg **)taosArrayGet(pArray, i);
|
||||
dTrace("vgId:%d, msg:%p is freed", pVnode->vgId, pMsg);
|
||||
rpcFreeCont(pMsg->pCont);
|
||||
taosFreeQitem(pMsg);
|
||||
}
|
||||
|
||||
taosArrayDestroy(pArray);
|
||||
}
|
||||
|
||||
static void vmProcessApplyQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
|
||||
SVnodeObj *pVnode = pInfo->ahandle;
|
||||
SRpcMsg *pMsg = NULL;
|
||||
|
||||
for (int32_t i = 0; i < numOfMsgs; ++i) {
|
||||
if (taosGetQitem(qall, (void **)&pMsg) == 0) continue;
|
||||
dTrace("vgId:%d, msg:%p get from vnode-apply queue", pVnode->vgId, pMsg);
|
||||
|
||||
// init response rpc msg
|
||||
SRpcMsg rsp = {0};
|
||||
|
||||
// get original rpc msg
|
||||
assert(pMsg->msgType == TDMT_SYNC_APPLY_MSG);
|
||||
SyncApplyMsg *pSyncApplyMsg = syncApplyMsgFromRpcMsg2(pMsg);
|
||||
syncApplyMsgLog2("==vmProcessApplyQueue==", pSyncApplyMsg);
|
||||
SRpcMsg originalRpcMsg;
|
||||
syncApplyMsg2OriginalRpcMsg(pSyncApplyMsg, &originalRpcMsg);
|
||||
|
||||
// apply data into tsdb
|
||||
if (vnodeProcessWriteReq(pVnode->pImpl, &originalRpcMsg, pSyncApplyMsg->fsmMeta.index, &rsp) < 0) {
|
||||
rsp.code = terrno;
|
||||
dError("vgId:%d, msg:%p failed to apply since %s", pVnode->vgId, pMsg, terrstr());
|
||||
}
|
||||
|
||||
syncApplyMsgDestroy(pSyncApplyMsg);
|
||||
rpcFreeCont(originalRpcMsg.pCont);
|
||||
|
||||
// if leader, send response
|
||||
if (pMsg->info.handle != NULL) {
|
||||
rsp.info = pMsg->info;
|
||||
tmsgSendRsp(&rsp);
|
||||
}
|
||||
|
||||
dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, rsp.code);
|
||||
rpcFreeCont(pMsg->pCont);
|
||||
taosFreeQitem(pMsg);
|
||||
}
|
||||
}
|
||||
|
||||
static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) {
|
||||
SVnodeObj *pVnode = pInfo->ahandle;
|
||||
SRpcMsg *pMsg = NULL;
|
||||
|
|
@ -211,11 +105,9 @@ static void vmProcessSyncQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOf
|
|||
|
||||
int32_t code = vnodeProcessSyncReq(pVnode->pImpl, pMsg, NULL);
|
||||
if (code != 0) {
|
||||
if (terrno != 0) code = terrno;
|
||||
dError("vgId:%d, msg:%p failed to sync since %s", pVnode->vgId, pMsg, terrstr());
|
||||
if (pMsg->info.handle != NULL) {
|
||||
if (terrno != 0) code = terrno;
|
||||
vmSendRsp(pMsg, code);
|
||||
}
|
||||
vmSendRsp(pMsg, code);
|
||||
}
|
||||
|
||||
dTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, code);
|
||||
|
|
@ -234,8 +126,8 @@ static void vmProcessMergeQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numO
|
|||
|
||||
int32_t code = vnodeProcessFetchMsg(pVnode->pImpl, pMsg, pInfo);
|
||||
if (code != 0) {
|
||||
dError("vgId:%d, msg:%p failed to merge since %s", pVnode->vgId, pMsg, terrstr());
|
||||
if (terrno != 0) code = terrno;
|
||||
dError("vgId:%d, msg:%p failed to merge since %s", pVnode->vgId, pMsg, terrstr());
|
||||
vmSendRsp(pMsg, code);
|
||||
}
|
||||
|
||||
|
|
@ -254,7 +146,7 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp
|
|||
|
||||
SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pHead->vgId);
|
||||
if (pVnode == NULL) {
|
||||
dError("vgId:%d, failed to put msg:%p into vnode queue since %s, type:%s", pHead->vgId, pMsg, terrstr(),
|
||||
dError("vgId:%d, msg:%p failed to put into vnode queue since %s, type:%s", pHead->vgId, pMsg, terrstr(),
|
||||
TMSG_INFO(pMsg->msgType));
|
||||
return terrno != 0 ? terrno : -1;
|
||||
}
|
||||
|
|
@ -322,7 +214,7 @@ int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
|
|||
if (pMsg == NULL) return -1;
|
||||
|
||||
SMsgHead *pHead = pRpc->pCont;
|
||||
dTrace("vgId:%d, msg:%p is created, type:%s", pHead->vgId, pMsg, TMSG_INFO(pMsg->msgType));
|
||||
dTrace("vgId:%d, msg:%p is created, type:%s", pHead->vgId, pMsg, TMSG_INFO(pRpc->msgType));
|
||||
|
||||
pHead->contLen = htonl(pHead->contLen);
|
||||
pHead->vgId = htonl(pHead->vgId);
|
||||
|
|
@ -362,9 +254,9 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) {
|
|||
}
|
||||
|
||||
int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
|
||||
pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode, (FItems)vmProcessWriteQueue);
|
||||
pVnode->pWriteQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode->pImpl, (FItems)vnodeProposeMsg);
|
||||
pVnode->pSyncQ = tWWorkerAllocQueue(&pMgmt->syncPool, pVnode, (FItems)vmProcessSyncQueue);
|
||||
pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->writePool, pVnode, (FItems)vmProcessApplyQueue);
|
||||
pVnode->pApplyQ = tWWorkerAllocQueue(&pMgmt->applyPool, pVnode->pImpl, (FItems)vnodeApplyMsg);
|
||||
pVnode->pQueryQ = tQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue);
|
||||
pVnode->pFetchQ = tQWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItem)vmProcessFetchQueue);
|
||||
pVnode->pMergeQ = tWWorkerAllocQueue(&pMgmt->mergePool, pVnode, (FItems)vmProcessMergeQueue);
|
||||
|
|
@ -381,8 +273,8 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
|
|||
|
||||
void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) {
|
||||
tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pWriteQ);
|
||||
tWWorkerFreeQueue(&pMgmt->applyPool, pVnode->pApplyQ);
|
||||
tWWorkerFreeQueue(&pMgmt->syncPool, pVnode->pSyncQ);
|
||||
tWWorkerFreeQueue(&pMgmt->writePool, pVnode->pApplyQ);
|
||||
tQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ);
|
||||
tQWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ);
|
||||
tWWorkerFreeQueue(&pMgmt->mergePool, pVnode->pMergeQ);
|
||||
|
|
@ -413,6 +305,11 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) {
|
|||
pWPool->max = tsNumOfVnodeWriteThreads;
|
||||
if (tWWorkerInit(pWPool) != 0) return -1;
|
||||
|
||||
SWWorkerPool *pAPool = &pMgmt->applyPool;
|
||||
pAPool->name = "vnode-apply";
|
||||
pAPool->max = tsNumOfVnodeWriteThreads;
|
||||
if (tWWorkerInit(pAPool) != 0) return -1;
|
||||
|
||||
SWWorkerPool *pSPool = &pMgmt->syncPool;
|
||||
pSPool->name = "vnode-sync";
|
||||
pSPool->max = tsNumOfVnodeSyncThreads;
|
||||
|
|
@ -449,6 +346,7 @@ void vmStopWorker(SVnodeMgmt *pMgmt) {
|
|||
tSingleWorkerCleanup(&pMgmt->monitorWorker);
|
||||
tSingleWorkerCleanup(&pMgmt->mgmtWorker);
|
||||
tWWorkerCleanup(&pMgmt->writePool);
|
||||
tWWorkerCleanup(&pMgmt->applyPool);
|
||||
tWWorkerCleanup(&pMgmt->syncPool);
|
||||
tQWorkerCleanup(&pMgmt->queryPool);
|
||||
tQWorkerCleanup(&pMgmt->fetchPool);
|
||||
|
|
|
|||
|
|
@ -22,17 +22,17 @@ static void dmSendRsp(SRpcMsg *pMsg);
|
|||
static void dmBuildMnodeRedirectRsp(SDnode *pDnode, SRpcMsg *pMsg);
|
||||
|
||||
static inline int32_t dmBuildNodeMsg(SRpcMsg *pMsg, SRpcMsg *pRpc) {
|
||||
SRpcConnInfo connInfo = {0};
|
||||
if (IsReq(pRpc) && rpcGetConnInfo(pRpc->info.handle, &connInfo) != 0) {
|
||||
terrno = TSDB_CODE_MND_NO_USER_FROM_CONN;
|
||||
dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle);
|
||||
return -1;
|
||||
}
|
||||
SRpcConnInfo *pConnInfo = &(pRpc->info.connInfo);
|
||||
// if (IsReq(pRpc)) {
|
||||
// terrno = TSDB_CODE_MND_NO_USER_FROM_CONN;
|
||||
// dError("failed to build msg since %s, app:%p handle:%p", terrstr(), pRpc->info.ahandle, pRpc->info.handle);
|
||||
// return -1;
|
||||
//}
|
||||
|
||||
memcpy(pMsg, pRpc, sizeof(SRpcMsg));
|
||||
memcpy(pMsg->conn.user, connInfo.user, TSDB_USER_LEN);
|
||||
pMsg->conn.clientIp = connInfo.clientIp;
|
||||
pMsg->conn.clientPort = connInfo.clientPort;
|
||||
memcpy(pMsg->conn.user, pConnInfo->user, TSDB_USER_LEN);
|
||||
pMsg->conn.clientIp = pConnInfo->clientIp;
|
||||
pMsg->conn.clientPort = pConnInfo->clientPort;
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -49,9 +49,9 @@ int32_t dmProcessNodeMsg(SMgmtWrapper *pWrapper, SRpcMsg *pMsg) {
|
|||
}
|
||||
|
||||
static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
|
||||
SDnodeTrans *pTrans = &pDnode->trans;
|
||||
SDnodeTrans * pTrans = &pDnode->trans;
|
||||
int32_t code = -1;
|
||||
SRpcMsg *pMsg = NULL;
|
||||
SRpcMsg * pMsg = NULL;
|
||||
SMgmtWrapper *pWrapper = NULL;
|
||||
SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pRpc->msgType)];
|
||||
|
||||
|
|
@ -95,6 +95,8 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
|
|||
int32_t vgId = ntohl(pHead->vgId);
|
||||
if (vgId == QNODE_HANDLE) {
|
||||
pWrapper = &pDnode->wrappers[QNODE];
|
||||
} else if (vgId == SNODE_HANDLE) {
|
||||
pWrapper = &pDnode->wrappers[SNODE];
|
||||
} else if (vgId == MNODE_HANDLE) {
|
||||
pWrapper = &pDnode->wrappers[MNODE];
|
||||
} else {
|
||||
|
|
@ -131,7 +133,8 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
|
|||
|
||||
_OVER:
|
||||
if (code != 0) {
|
||||
dTrace("msg:%p, failed to process since %s, type:%s", pMsg, terrstr(), TMSG_INFO(pRpc->msgType));
|
||||
dTrace("failed to process msg:%p since %s, handle:%p", pMsg, terrstr(), pRpc->info.handle);
|
||||
|
||||
if (terrno != 0) code = terrno;
|
||||
|
||||
if (IsReq(pRpc)) {
|
||||
|
|
@ -149,8 +152,10 @@ _OVER:
|
|||
}
|
||||
}
|
||||
|
||||
dTrace("msg:%p, is freed", pMsg);
|
||||
taosFreeQitem(pMsg);
|
||||
if (pMsg != NULL) {
|
||||
dTrace("msg:%p, is freed", pMsg);
|
||||
taosFreeQitem(pMsg);
|
||||
}
|
||||
rpcFreeCont(pRpc->pCont);
|
||||
}
|
||||
|
||||
|
|
@ -162,11 +167,11 @@ int32_t dmInitMsgHandle(SDnode *pDnode) {
|
|||
|
||||
for (EDndNodeType ntype = DNODE; ntype < NODE_END; ++ntype) {
|
||||
SMgmtWrapper *pWrapper = &pDnode->wrappers[ntype];
|
||||
SArray *pArray = (*pWrapper->func.getHandlesFp)();
|
||||
SArray * pArray = (*pWrapper->func.getHandlesFp)();
|
||||
if (pArray == NULL) return -1;
|
||||
|
||||
for (int32_t i = 0; i < taosArrayGetSize(pArray); ++i) {
|
||||
SMgmtHandle *pMgmt = taosArrayGet(pArray, i);
|
||||
SMgmtHandle * pMgmt = taosArrayGet(pArray, i);
|
||||
SDnodeHandle *pHandle = &pTrans->msgHandles[TMSG_INDEX(pMgmt->msgType)];
|
||||
if (pMgmt->needCheckVgId) {
|
||||
pHandle->needCheckVgId = pMgmt->needCheckVgId;
|
||||
|
|
|
|||
|
|
@ -318,9 +318,9 @@ void dmGetMnodeEpSet(SDnodeData *pData, SEpSet *pEpSet) {
|
|||
|
||||
void dmGetMnodeEpSetForRedirect(SDnodeData *pData, SRpcMsg *pMsg, SEpSet *pEpSet) {
|
||||
dmGetMnodeEpSet(pData, pEpSet);
|
||||
dDebug("msg:%p, is redirected, num:%d use:%d", pMsg, pEpSet->numOfEps, pEpSet->inUse);
|
||||
dTrace("msg is redirected, handle:%p num:%d use:%d", pMsg->info.handle, pEpSet->numOfEps, pEpSet->inUse);
|
||||
for (int32_t i = 0; i < pEpSet->numOfEps; ++i) {
|
||||
dDebug("mnode index:%d %s:%u", i, pEpSet->eps[i].fqdn, pEpSet->eps[i].port);
|
||||
dTrace("mnode index:%d %s:%u", i, pEpSet->eps[i].fqdn, pEpSet->eps[i].port);
|
||||
if (strcmp(pEpSet->eps[i].fqdn, tsLocalFqdn) == 0 && pEpSet->eps[i].port == tsServerPort) {
|
||||
pEpSet->inUse = (i + 1) % pEpSet->numOfEps;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -309,6 +309,7 @@ typedef struct {
|
|||
int8_t slidingUnit;
|
||||
int8_t timezone;
|
||||
int32_t dstVgId; // for stream
|
||||
int64_t dstTbUid;
|
||||
int64_t interval;
|
||||
int64_t offset;
|
||||
int64_t sliding;
|
||||
|
|
|
|||
|
|
@ -36,19 +36,19 @@ typedef enum {
|
|||
} ETrnAct;
|
||||
|
||||
typedef struct {
|
||||
int32_t id;
|
||||
int32_t errCode;
|
||||
int32_t acceptableCode;
|
||||
int8_t stage;
|
||||
ETrnAct actionType;
|
||||
int8_t rawWritten;
|
||||
int8_t msgSent;
|
||||
int8_t msgReceived;
|
||||
tmsg_t msgType;
|
||||
SEpSet epSet;
|
||||
int32_t contLen;
|
||||
void *pCont;
|
||||
SSdbRaw *pRaw;
|
||||
int32_t id;
|
||||
int32_t errCode;
|
||||
int32_t acceptableCode;
|
||||
ETrnStage stage;
|
||||
ETrnAct actionType;
|
||||
int8_t rawWritten;
|
||||
int8_t msgSent;
|
||||
int8_t msgReceived;
|
||||
tmsg_t msgType;
|
||||
SEpSet epSet;
|
||||
int32_t contLen;
|
||||
void *pCont;
|
||||
SSdbRaw *pRaw;
|
||||
} STransAction;
|
||||
|
||||
typedef void (*TransCbFp)(SMnode *pMnode, void *param, int32_t paramLen);
|
||||
|
|
|
|||
|
|
@ -92,6 +92,15 @@ static int32_t mndProcessConsumerLostMsg(SRpcMsg *pMsg) {
|
|||
SMqConsumerObj *pConsumer = mndAcquireConsumer(pMnode, pLostMsg->consumerId);
|
||||
ASSERT(pConsumer);
|
||||
|
||||
|
||||
mInfo("receive consumer lost msg, consumer id %ld, status %s", pLostMsg->consumerId,
|
||||
mndConsumerStatusName(pConsumer->status));
|
||||
|
||||
if (pConsumer->status != MQ_CONSUMER_STATUS__READY) {
|
||||
mndReleaseConsumer(pMnode, pConsumer);
|
||||
return -1;
|
||||
}
|
||||
|
||||
SMqConsumerObj *pConsumerNew = tNewSMqConsumerObj(pConsumer->consumerId, pConsumer->cgroup);
|
||||
pConsumerNew->updateType = CONSUMER_UPDATE__LOST;
|
||||
|
||||
|
|
|
|||
|
|
@ -87,6 +87,10 @@ int32_t mndInitDnode(SMnode *pMnode) {
|
|||
void mndCleanupDnode(SMnode *pMnode) {}
|
||||
|
||||
static int32_t mndCreateDefaultDnode(SMnode *pMnode) {
|
||||
int32_t code = -1;
|
||||
SSdbRaw *pRaw = NULL;
|
||||
STrans *pTrans = NULL;
|
||||
|
||||
SDnodeObj dnodeObj = {0};
|
||||
dnodeObj.id = 1;
|
||||
dnodeObj.createdTime = taosGetTimestampMs();
|
||||
|
|
@ -95,54 +99,42 @@ static int32_t mndCreateDefaultDnode(SMnode *pMnode) {
|
|||
memcpy(&dnodeObj.fqdn, pMnode->replicas[0].fqdn, TSDB_FQDN_LEN);
|
||||
snprintf(dnodeObj.ep, TSDB_EP_LEN, "%s:%u", dnodeObj.fqdn, dnodeObj.port);
|
||||
|
||||
SSdbRaw *pRaw = mndDnodeActionEncode(&dnodeObj);
|
||||
if (pRaw == NULL) return -1;
|
||||
if (sdbSetRawStatus(pRaw, SDB_STATUS_READY) != 0) return -1;
|
||||
pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_GLOBAL, NULL);
|
||||
if (pTrans == NULL) goto _OVER;
|
||||
mDebug("trans:%d, used to create dnode:%s on first deploy", pTrans->id, dnodeObj.ep);
|
||||
|
||||
mDebug("dnode:%d, will be created when deploying, raw:%p", dnodeObj.id, pRaw);
|
||||
|
||||
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_GLOBAL, NULL);
|
||||
if (pTrans == NULL) {
|
||||
mError("dnode:%s, failed to create since %s", dnodeObj.ep, terrstr());
|
||||
return -1;
|
||||
}
|
||||
mDebug("trans:%d, used to create dnode:%s", pTrans->id, dnodeObj.ep);
|
||||
|
||||
if (mndTransAppendCommitlog(pTrans, pRaw) != 0) {
|
||||
mError("trans:%d, failed to append commit log since %s", pTrans->id, terrstr());
|
||||
mndTransDrop(pTrans);
|
||||
return -1;
|
||||
}
|
||||
pRaw = mndDnodeActionEncode(&dnodeObj);
|
||||
if (pRaw == NULL || mndTransAppendCommitlog(pTrans, pRaw) != 0) goto _OVER;
|
||||
sdbSetRawStatus(pRaw, SDB_STATUS_READY);
|
||||
pRaw = NULL;
|
||||
|
||||
if (mndTransPrepare(pMnode, pTrans) != 0) {
|
||||
mError("trans:%d, failed to prepare since %s", pTrans->id, terrstr());
|
||||
mndTransDrop(pTrans);
|
||||
return -1;
|
||||
}
|
||||
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
|
||||
code = 0;
|
||||
|
||||
_OVER:
|
||||
mndTransDrop(pTrans);
|
||||
return 0;
|
||||
sdbFreeRaw(pRaw);
|
||||
return code;
|
||||
}
|
||||
|
||||
static SSdbRaw *mndDnodeActionEncode(SDnodeObj *pDnode) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
|
||||
SSdbRaw *pRaw = sdbAllocRaw(SDB_DNODE, TSDB_DNODE_VER_NUMBER, sizeof(SDnodeObj) + TSDB_DNODE_RESERVE_SIZE);
|
||||
if (pRaw == NULL) goto DNODE_ENCODE_OVER;
|
||||
if (pRaw == NULL) goto _OVER;
|
||||
|
||||
int32_t dataPos = 0;
|
||||
SDB_SET_INT32(pRaw, dataPos, pDnode->id, DNODE_ENCODE_OVER)
|
||||
SDB_SET_INT64(pRaw, dataPos, pDnode->createdTime, DNODE_ENCODE_OVER)
|
||||
SDB_SET_INT64(pRaw, dataPos, pDnode->updateTime, DNODE_ENCODE_OVER)
|
||||
SDB_SET_INT16(pRaw, dataPos, pDnode->port, DNODE_ENCODE_OVER)
|
||||
SDB_SET_BINARY(pRaw, dataPos, pDnode->fqdn, TSDB_FQDN_LEN, DNODE_ENCODE_OVER)
|
||||
SDB_SET_RESERVE(pRaw, dataPos, TSDB_DNODE_RESERVE_SIZE, DNODE_ENCODE_OVER)
|
||||
SDB_SET_DATALEN(pRaw, dataPos, DNODE_ENCODE_OVER);
|
||||
SDB_SET_INT32(pRaw, dataPos, pDnode->id, _OVER)
|
||||
SDB_SET_INT64(pRaw, dataPos, pDnode->createdTime, _OVER)
|
||||
SDB_SET_INT64(pRaw, dataPos, pDnode->updateTime, _OVER)
|
||||
SDB_SET_INT16(pRaw, dataPos, pDnode->port, _OVER)
|
||||
SDB_SET_BINARY(pRaw, dataPos, pDnode->fqdn, TSDB_FQDN_LEN, _OVER)
|
||||
SDB_SET_RESERVE(pRaw, dataPos, TSDB_DNODE_RESERVE_SIZE, _OVER)
|
||||
SDB_SET_DATALEN(pRaw, dataPos, _OVER);
|
||||
|
||||
terrno = 0;
|
||||
|
||||
DNODE_ENCODE_OVER:
|
||||
_OVER:
|
||||
if (terrno != 0) {
|
||||
mError("dnode:%d, failed to encode to raw:%p since %s", pDnode->id, pRaw, terrstr());
|
||||
sdbFreeRaw(pRaw);
|
||||
|
|
@ -154,33 +146,32 @@ DNODE_ENCODE_OVER:
|
|||
}
|
||||
|
||||
static SSdbRow *mndDnodeActionDecode(SSdbRaw *pRaw) {
|
||||
SSdbRow *pRow = NULL;
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
|
||||
int8_t sver = 0;
|
||||
if (sdbGetRawSoftVer(pRaw, &sver) != 0) goto DNODE_DECODE_OVER;
|
||||
|
||||
if (sdbGetRawSoftVer(pRaw, &sver) != 0) goto _OVER;
|
||||
if (sver != TSDB_DNODE_VER_NUMBER) {
|
||||
terrno = TSDB_CODE_SDB_INVALID_DATA_VER;
|
||||
goto DNODE_DECODE_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
SSdbRow *pRow = sdbAllocRow(sizeof(SDnodeObj));
|
||||
if (pRow == NULL) goto DNODE_DECODE_OVER;
|
||||
|
||||
pRow = sdbAllocRow(sizeof(SDnodeObj));
|
||||
if (pRow == NULL) goto _OVER;
|
||||
SDnodeObj *pDnode = sdbGetRowObj(pRow);
|
||||
if (pDnode == NULL) goto DNODE_DECODE_OVER;
|
||||
if (pDnode == NULL) goto _OVER;
|
||||
|
||||
int32_t dataPos = 0;
|
||||
SDB_GET_INT32(pRaw, dataPos, &pDnode->id, DNODE_DECODE_OVER)
|
||||
SDB_GET_INT64(pRaw, dataPos, &pDnode->createdTime, DNODE_DECODE_OVER)
|
||||
SDB_GET_INT64(pRaw, dataPos, &pDnode->updateTime, DNODE_DECODE_OVER)
|
||||
SDB_GET_INT16(pRaw, dataPos, &pDnode->port, DNODE_DECODE_OVER)
|
||||
SDB_GET_BINARY(pRaw, dataPos, pDnode->fqdn, TSDB_FQDN_LEN, DNODE_DECODE_OVER)
|
||||
SDB_GET_RESERVE(pRaw, dataPos, TSDB_DNODE_RESERVE_SIZE, DNODE_DECODE_OVER)
|
||||
SDB_GET_INT32(pRaw, dataPos, &pDnode->id, _OVER)
|
||||
SDB_GET_INT64(pRaw, dataPos, &pDnode->createdTime, _OVER)
|
||||
SDB_GET_INT64(pRaw, dataPos, &pDnode->updateTime, _OVER)
|
||||
SDB_GET_INT16(pRaw, dataPos, &pDnode->port, _OVER)
|
||||
SDB_GET_BINARY(pRaw, dataPos, pDnode->fqdn, TSDB_FQDN_LEN, _OVER)
|
||||
SDB_GET_RESERVE(pRaw, dataPos, TSDB_DNODE_RESERVE_SIZE, _OVER)
|
||||
|
||||
terrno = 0;
|
||||
|
||||
DNODE_DECODE_OVER:
|
||||
_OVER:
|
||||
if (terrno != 0) {
|
||||
mError("dnode:%d, failed to decode from raw:%p since %s", pDnode->id, pRaw, terrstr());
|
||||
taosMemoryFreeClear(pRow);
|
||||
|
|
@ -246,6 +237,7 @@ static SDnodeObj *mndAcquireDnodeByEp(SMnode *pMnode, char *pEpStr) {
|
|||
sdbRelease(pSdb, pDnode);
|
||||
}
|
||||
|
||||
terrno = TSDB_CODE_MND_DNODE_NOT_EXIST;
|
||||
return NULL;
|
||||
}
|
||||
|
||||
|
|
@ -290,25 +282,26 @@ static void mndGetDnodeData(SMnode *pMnode, SArray *pDnodeEps) {
|
|||
}
|
||||
}
|
||||
|
||||
static int32_t mndCheckClusterCfgPara(SMnode *pMnode, const SClusterCfg *pCfg) {
|
||||
static int32_t mndCheckClusterCfgPara(SMnode *pMnode, SDnodeObj *pDnode, const SClusterCfg *pCfg) {
|
||||
if (pCfg->statusInterval != tsStatusInterval) {
|
||||
mError("statusInterval [%d - %d] cfg inconsistent", pCfg->statusInterval, tsStatusInterval);
|
||||
mError("dnode:%d, statusInterval:%d inconsistent with cluster:%d", pDnode->id, pCfg->statusInterval,
|
||||
tsStatusInterval);
|
||||
return DND_REASON_STATUS_INTERVAL_NOT_MATCH;
|
||||
}
|
||||
|
||||
if ((0 != strcasecmp(pCfg->timezone, tsTimezoneStr)) && (pMnode->checkTime != pCfg->checkTime)) {
|
||||
mError("timezone [%s - %s] [%" PRId64 " - %" PRId64 "] cfg inconsistent", pCfg->timezone, tsTimezoneStr,
|
||||
pCfg->checkTime, pMnode->checkTime);
|
||||
mError("dnode:%d, timezone:%s checkTime:%" PRId64 " inconsistent with cluster %s %" PRId64, pDnode->id,
|
||||
pCfg->timezone, pCfg->checkTime, tsTimezoneStr, pMnode->checkTime);
|
||||
return DND_REASON_TIME_ZONE_NOT_MATCH;
|
||||
}
|
||||
|
||||
if (0 != strcasecmp(pCfg->locale, tsLocale)) {
|
||||
mError("locale [%s - %s] cfg inconsistent", pCfg->locale, tsLocale);
|
||||
mError("dnode:%d, locale:%s inconsistent with cluster:%s", pDnode->id, pCfg->locale, tsLocale);
|
||||
return DND_REASON_LOCALE_NOT_MATCH;
|
||||
}
|
||||
|
||||
if (0 != strcasecmp(pCfg->charset, tsCharset)) {
|
||||
mError("charset [%s - %s] cfg inconsistent.", pCfg->charset, tsCharset);
|
||||
mError("dnode:%d, charset:%s inconsistent with cluster:%s", pDnode->id, pCfg->charset, tsCharset);
|
||||
return DND_REASON_CHARSET_NOT_MATCH;
|
||||
}
|
||||
|
||||
|
|
@ -323,15 +316,14 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
|
||||
if (tDeserializeSStatusReq(pReq->pCont, pReq->contLen, &statusReq) != 0) {
|
||||
terrno = TSDB_CODE_INVALID_MSG;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
if (statusReq.dnodeId == 0) {
|
||||
pDnode = mndAcquireDnodeByEp(pMnode, statusReq.dnodeEp);
|
||||
if (pDnode == NULL) {
|
||||
mDebug("dnode:%s, not created yet", statusReq.dnodeEp);
|
||||
terrno = TSDB_CODE_MND_DNODE_NOT_EXIST;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
} else {
|
||||
pDnode = mndAcquireDnode(pMnode, statusReq.dnodeId);
|
||||
|
|
@ -341,13 +333,11 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
pDnode->offlineReason = DND_REASON_DNODE_ID_NOT_MATCH;
|
||||
}
|
||||
mError("dnode:%d, %s not exist", statusReq.dnodeId, statusReq.dnodeEp);
|
||||
terrno = TSDB_CODE_MND_DNODE_NOT_EXIST;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
}
|
||||
|
||||
int32_t numOfVloads = (int32_t)taosArrayGetSize(statusReq.pVloads);
|
||||
for (int32_t v = 0; v < numOfVloads; ++v) {
|
||||
for (int32_t v = 0; v < taosArrayGetSize(statusReq.pVloads); ++v) {
|
||||
SVnodeLoad *pVload = taosArrayGet(statusReq.pVloads, v);
|
||||
|
||||
SVgObj *pVgroup = mndAcquireVgroup(pMnode, pVload->vgId);
|
||||
|
|
@ -366,6 +356,7 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
roleChanged = true;
|
||||
}
|
||||
pVgroup->vnodeGid[vg].role = pVload->syncState;
|
||||
break;
|
||||
}
|
||||
}
|
||||
if (roleChanged) {
|
||||
|
|
@ -405,11 +396,11 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
}
|
||||
mError("dnode:%d, status msg version:%d not match cluster:%d", statusReq.dnodeId, statusReq.sver, tsVersion);
|
||||
terrno = TSDB_CODE_VERSION_NOT_COMPATIBLE;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
if (statusReq.dnodeId == 0) {
|
||||
mDebug("dnode:%d, %s first access, set clusterId %" PRId64, pDnode->id, pDnode->ep, pMnode->clusterId);
|
||||
mInfo("dnode:%d, %s first access, set clusterId %" PRId64, pDnode->id, pDnode->ep, pMnode->clusterId);
|
||||
} else {
|
||||
if (statusReq.clusterId != pMnode->clusterId) {
|
||||
if (pDnode != NULL) {
|
||||
|
|
@ -418,7 +409,7 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
mError("dnode:%d, clusterId %" PRId64 " not match exist %" PRId64, pDnode->id, statusReq.clusterId,
|
||||
pMnode->clusterId);
|
||||
terrno = TSDB_CODE_MND_INVALID_CLUSTER_ID;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
} else {
|
||||
pDnode->accessTimes++;
|
||||
mTrace("dnode:%d, status received, access times %d", pDnode->id, pDnode->accessTimes);
|
||||
|
|
@ -426,18 +417,17 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
}
|
||||
|
||||
// Verify whether the cluster parameters are consistent when status change from offline to ready
|
||||
int32_t ret = mndCheckClusterCfgPara(pMnode, &statusReq.clusterCfg);
|
||||
if (0 != ret) {
|
||||
pDnode->offlineReason = ret;
|
||||
mError("dnode:%d, cluster cfg inconsistent since:%s", pDnode->id, offlineReason[ret]);
|
||||
pDnode->offlineReason = mndCheckClusterCfgPara(pMnode, pDnode, &statusReq.clusterCfg);
|
||||
if (pDnode->offlineReason != 0) {
|
||||
mError("dnode:%d, cluster cfg inconsistent since:%s", pDnode->id, offlineReason[pDnode->offlineReason]);
|
||||
terrno = TSDB_CODE_MND_INVALID_CLUSTER_CFG;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
if (!online) {
|
||||
mInfo("dnode:%d, from offline to online", pDnode->id);
|
||||
} else {
|
||||
mDebug("dnode:%d, send dnode epset, online:%d ver:% " PRId64 ":%" PRId64 " reboot:%d", pDnode->id, online,
|
||||
mDebug("dnode:%d, send dnode epset, online:%d dnode_ver:%" PRId64 ":%" PRId64 " reboot:%d", pDnode->id, online,
|
||||
statusReq.dnodeVer, dnodeVer, reboot);
|
||||
}
|
||||
|
||||
|
|
@ -452,7 +442,7 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
statusRsp.pDnodeEps = taosArrayInit(mndGetDnodeSize(pMnode), sizeof(SDnodeEp));
|
||||
if (statusRsp.pDnodeEps == NULL) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto PROCESS_STATUS_MSG_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
mndGetDnodeData(pMnode, statusRsp.pDnodeEps);
|
||||
|
|
@ -469,13 +459,17 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
|
|||
pDnode->lastAccessTime = curMs;
|
||||
code = 0;
|
||||
|
||||
PROCESS_STATUS_MSG_OVER:
|
||||
_OVER:
|
||||
mndReleaseDnode(pMnode, pDnode);
|
||||
taosArrayDestroy(statusReq.pVloads);
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t mndCreateDnode(SMnode *pMnode, SRpcMsg *pReq, SCreateDnodeReq *pCreate) {
|
||||
int32_t code = -1;
|
||||
SSdbRaw *pRaw = NULL;
|
||||
STrans *pTrans = NULL;
|
||||
|
||||
SDnodeObj dnodeObj = {0};
|
||||
dnodeObj.id = sdbGetMaxId(pMnode->pSdb, SDB_DNODE);
|
||||
dnodeObj.createdTime = taosGetTimestampMs();
|
||||
|
|
@ -484,29 +478,22 @@ static int32_t mndCreateDnode(SMnode *pMnode, SRpcMsg *pReq, SCreateDnodeReq *pC
|
|||
memcpy(dnodeObj.fqdn, pCreate->fqdn, TSDB_FQDN_LEN);
|
||||
snprintf(dnodeObj.ep, TSDB_EP_LEN, "%s:%u", dnodeObj.fqdn, dnodeObj.port);
|
||||
|
||||
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_GLOBAL, pReq);
|
||||
if (pTrans == NULL) {
|
||||
mError("dnode:%s, failed to create since %s", dnodeObj.ep, terrstr());
|
||||
return -1;
|
||||
}
|
||||
pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_GLOBAL, pReq);
|
||||
if (pTrans == NULL) goto _OVER;
|
||||
mDebug("trans:%d, used to create dnode:%s", pTrans->id, dnodeObj.ep);
|
||||
|
||||
SSdbRaw *pCommitRaw = mndDnodeActionEncode(&dnodeObj);
|
||||
if (pCommitRaw == NULL || mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) {
|
||||
mError("trans:%d, failed to append commit log since %s", pTrans->id, terrstr());
|
||||
mndTransDrop(pTrans);
|
||||
return -1;
|
||||
}
|
||||
sdbSetRawStatus(pCommitRaw, SDB_STATUS_READY);
|
||||
pRaw = mndDnodeActionEncode(&dnodeObj);
|
||||
if (pRaw == NULL || mndTransAppendCommitlog(pTrans, pRaw) != 0) goto _OVER;
|
||||
sdbSetRawStatus(pRaw, SDB_STATUS_READY);
|
||||
pRaw = NULL;
|
||||
|
||||
if (mndTransPrepare(pMnode, pTrans) != 0) {
|
||||
mError("trans:%d, failed to prepare since %s", pTrans->id, terrstr());
|
||||
mndTransDrop(pTrans);
|
||||
return -1;
|
||||
}
|
||||
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
|
||||
code = 0;
|
||||
|
||||
_OVER:
|
||||
mndTransDrop(pTrans);
|
||||
return 0;
|
||||
sdbFreeRaw(pRaw);
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t mndProcessCreateDnodeReq(SRpcMsg *pReq) {
|
||||
|
|
@ -518,38 +505,37 @@ static int32_t mndProcessCreateDnodeReq(SRpcMsg *pReq) {
|
|||
|
||||
if (tDeserializeSCreateDnodeReq(pReq->pCont, pReq->contLen, &createReq) != 0) {
|
||||
terrno = TSDB_CODE_INVALID_MSG;
|
||||
goto CREATE_DNODE_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
mDebug("dnode:%s:%d, start to create", createReq.fqdn, createReq.port);
|
||||
|
||||
if (createReq.fqdn[0] == 0 || createReq.port <= 0 || createReq.port > UINT16_MAX) {
|
||||
terrno = TSDB_CODE_MND_INVALID_DNODE_EP;
|
||||
goto CREATE_DNODE_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
char ep[TSDB_EP_LEN];
|
||||
snprintf(ep, TSDB_EP_LEN, "%s:%d", createReq.fqdn, createReq.port);
|
||||
pDnode = mndAcquireDnodeByEp(pMnode, ep);
|
||||
if (pDnode != NULL) {
|
||||
terrno = TSDB_CODE_MND_DNODE_ALREADY_EXIST;
|
||||
goto CREATE_DNODE_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
pUser = mndAcquireUser(pMnode, pReq->conn.user);
|
||||
if (pUser == NULL) {
|
||||
terrno = TSDB_CODE_MND_NO_USER_FROM_CONN;
|
||||
goto CREATE_DNODE_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
if (mndCheckNodeAuth(pUser) != 0) {
|
||||
goto CREATE_DNODE_OVER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
code = mndCreateDnode(pMnode, pReq, &createReq);
|
||||
if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS;
|
||||
|
||||
CREATE_DNODE_OVER:
|
||||
_OVER:
|
||||
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) {
|
||||
mError("dnode:%s:%d, failed to create since %s", createReq.fqdn, createReq.port, terrstr());
|
||||
}
|
||||
|
|
@ -559,7 +545,7 @@ CREATE_DNODE_OVER:
|
|||
return code;
|
||||
}
|
||||
|
||||
static int32_t mndDropDnode(SMnode *pMnode, SRpcMsg *pReq, SDnodeObj *pDnode, SMnodeObj *pMObj) {
|
||||
static int32_t mndDropDnode(SMnode *pMnode, SRpcMsg *pReq, SDnodeObj *pDnode, SMnodeObj *pMObj, int32_t numOfVnodes) {
|
||||
int32_t code = -1;
|
||||
SSdbRaw *pRaw = NULL;
|
||||
STrans *pTrans = NULL;
|
||||
|
|
@ -579,8 +565,12 @@ static int32_t mndDropDnode(SMnode *pMnode, SRpcMsg *pReq, SDnodeObj *pDnode, SM
|
|||
sdbSetRawStatus(pRaw, SDB_STATUS_DROPPED);
|
||||
pRaw = NULL;
|
||||
|
||||
if (mndSetDropMnodeInfoToTrans(pMnode, pTrans, pMObj) != 0) goto _OVER;
|
||||
if (mndSetMoveVgroupsInfoToTrans(pMnode, pTrans, pDnode->id) != 0) goto _OVER;
|
||||
if (pMObj != NULL) {
|
||||
if (mndSetDropMnodeInfoToTrans(pMnode, pTrans, pMObj) != 0) goto _OVER;
|
||||
}
|
||||
if (numOfVnodes > 0) {
|
||||
if (mndSetMoveVgroupsInfoToTrans(pMnode, pTrans, pDnode->id) != 0) goto _OVER;
|
||||
}
|
||||
if (mndTransPrepare(pMnode, pTrans) != 0) goto _OVER;
|
||||
|
||||
code = 0;
|
||||
|
|
@ -617,11 +607,6 @@ static int32_t mndProcessDropDnodeReq(SRpcMsg *pReq) {
|
|||
goto _OVER;
|
||||
}
|
||||
|
||||
if (!mndIsDnodeOnline(pDnode, taosGetTimestampMs())) {
|
||||
terrno = TSDB_CODE_NODE_OFFLINE;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
pMObj = mndAcquireMnode(pMnode, dropReq.dnodeId);
|
||||
if (pMObj != NULL) {
|
||||
if (sdbGetSize(pMnode->pSdb, SDB_MNODE) <= 1) {
|
||||
|
|
@ -629,7 +614,17 @@ static int32_t mndProcessDropDnodeReq(SRpcMsg *pReq) {
|
|||
goto _OVER;
|
||||
}
|
||||
if (pMnode->selfDnodeId == dropReq.dnodeId) {
|
||||
terrno = TSDB_CODE_MND_CANT_DROP_MASTER;
|
||||
terrno = TSDB_CODE_MND_CANT_DROP_LEADER;
|
||||
goto _OVER;
|
||||
}
|
||||
}
|
||||
|
||||
int32_t numOfVnodes = mndGetVnodesNum(pMnode, pDnode->id);
|
||||
if (numOfVnodes > 0 || pMObj != NULL) {
|
||||
if (!mndIsDnodeOnline(pDnode, taosGetTimestampMs())) {
|
||||
terrno = TSDB_CODE_NODE_OFFLINE;
|
||||
mError("dnode:%d, failed to drop since %s, has_mnode:%d numOfVnodes:%d", pDnode->id, terrstr(), pMObj != NULL,
|
||||
numOfVnodes);
|
||||
goto _OVER;
|
||||
}
|
||||
}
|
||||
|
|
@ -644,7 +639,7 @@ static int32_t mndProcessDropDnodeReq(SRpcMsg *pReq) {
|
|||
goto _OVER;
|
||||
}
|
||||
|
||||
code = mndDropDnode(pMnode, pReq, pDnode, pMObj);
|
||||
code = mndDropDnode(pMnode, pReq, pDnode, pMObj, numOfVnodes);
|
||||
if (code == 0) code = TSDB_CODE_ACTION_IN_PROGRESS;
|
||||
|
||||
_OVER:
|
||||
|
|
@ -669,7 +664,6 @@ static int32_t mndProcessConfigDnodeReq(SRpcMsg *pReq) {
|
|||
|
||||
SDnodeObj *pDnode = mndAcquireDnode(pMnode, cfgReq.dnodeId);
|
||||
if (pDnode == NULL) {
|
||||
terrno = TSDB_CODE_MND_DNODE_NOT_EXIST;
|
||||
mError("dnode:%d, failed to config since %s ", cfgReq.dnodeId, terrstr());
|
||||
return -1;
|
||||
}
|
||||
|
|
@ -679,17 +673,18 @@ static int32_t mndProcessConfigDnodeReq(SRpcMsg *pReq) {
|
|||
|
||||
int32_t bufLen = tSerializeSMCfgDnodeReq(NULL, 0, &cfgReq);
|
||||
void *pBuf = rpcMallocCont(bufLen);
|
||||
|
||||
if (pBuf == NULL) return -1;
|
||||
tSerializeSMCfgDnodeReq(pBuf, bufLen, &cfgReq);
|
||||
|
||||
mDebug("dnode:%d, send config req to dnode, app:%p", cfgReq.dnodeId, pReq->info.ahandle);
|
||||
SRpcMsg rpcMsg = {.msgType = TDMT_DND_CONFIG_DNODE, .pCont = pBuf, .contLen = bufLen, .info = pReq->info};
|
||||
|
||||
mDebug("dnode:%d, send config req to dnode, app:%p", cfgReq.dnodeId, rpcMsg.info.ahandle);
|
||||
return tmsgSendReq(&epSet, &rpcMsg);
|
||||
}
|
||||
|
||||
static int32_t mndProcessConfigDnodeRsp(SRpcMsg *pRsp) {
|
||||
mDebug("config rsp from dnode, app:%p", pRsp->info.ahandle);
|
||||
return TSDB_CODE_SUCCESS;
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int32_t mndRetrieveConfigs(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) {
|
||||
|
|
@ -698,7 +693,7 @@ static int32_t mndRetrieveConfigs(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p
|
|||
int32_t numOfRows = 0;
|
||||
char *cfgOpts[TSDB_CONFIG_NUMBER] = {0};
|
||||
char cfgVals[TSDB_CONFIG_NUMBER][TSDB_CONIIG_VALUE_LEN + 1] = {0};
|
||||
char *pWrite;
|
||||
char *pWrite = NULL;
|
||||
int32_t cols = 0;
|
||||
|
||||
cfgOpts[totalRows] = "statusInterval";
|
||||
|
|
@ -724,7 +719,6 @@ static int32_t mndRetrieveConfigs(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p
|
|||
cols = 0;
|
||||
|
||||
STR_WITH_MAXSIZE_TO_VARSTR(buf, cfgOpts[i], TSDB_CONFIG_OPTION_LEN);
|
||||
|
||||
SColumnInfoData *pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
|
||||
colDataAppend(pColInfo, numOfRows, (const char *)buf, false);
|
||||
|
||||
|
|
@ -774,7 +768,6 @@ static int32_t mndRetrieveDnodes(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
|
|||
|
||||
char b1[9] = {0};
|
||||
STR_TO_VARSTR(b1, online ? "ready" : "offline");
|
||||
|
||||
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
|
||||
colDataAppend(pColInfo, numOfRows, b1, false);
|
||||
|
||||
|
|
@ -792,7 +785,6 @@ static int32_t mndRetrieveDnodes(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pB
|
|||
}
|
||||
|
||||
pShow->numOfRows += numOfRows;
|
||||
|
||||
return numOfRows;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -588,7 +588,7 @@ static int32_t mndProcessDropMnodeReq(SRpcMsg *pReq) {
|
|||
}
|
||||
|
||||
if (pMnode->selfDnodeId == dropReq.dnodeId) {
|
||||
terrno = TSDB_CODE_MND_CANT_DROP_MASTER;
|
||||
terrno = TSDB_CODE_MND_CANT_DROP_LEADER;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -132,7 +132,7 @@ int32_t mndAssignTaskToVg(SMnode* pMnode, STrans* pTrans, SStreamTask* pTask, SS
|
|||
terrno = TSDB_CODE_QRY_INVALID_INPUT;
|
||||
return -1;
|
||||
}
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &plan->execNode.epSet, TDMT_VND_TASK_DEPLOY, pVgroup->vgId);
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &plan->execNode.epSet, TDMT_STREAM_TASK_DEPLOY, pVgroup->vgId);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -156,7 +156,7 @@ int32_t mndAssignTaskToSnode(SMnode* pMnode, STrans* pTrans, SStreamTask* pTask,
|
|||
terrno = TSDB_CODE_QRY_INVALID_INPUT;
|
||||
return -1;
|
||||
}
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &plan->execNode.epSet, TDMT_SND_TASK_DEPLOY, 0);
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &plan->execNode.epSet, TDMT_STREAM_TASK_DEPLOY, 0);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -222,7 +222,7 @@ int32_t mndAddShuffledSinkToStream(SMnode* pMnode, STrans* pTrans, SStreamObj* p
|
|||
// dispatch
|
||||
pTask->dispatchType = TASK_DISPATCH__NONE;
|
||||
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &pTask->epSet, TDMT_VND_TASK_DEPLOY, pVgroup->vgId);
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &pTask->epSet, TDMT_STREAM_TASK_DEPLOY, pVgroup->vgId);
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -267,8 +267,7 @@ int32_t mndAddFixedSinkToStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStr
|
|||
// dispatch
|
||||
pTask->dispatchType = TASK_DISPATCH__NONE;
|
||||
|
||||
/*mndPersistTaskDeployReq(pTrans, pTask, &pTask->epSet, TDMT_VND_TASK_DEPLOY, pVgroup->vgId);*/
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &pTask->epSet, TDMT_VND_TASK_DEPLOY, pStream->fixedSinkVg.vgId);
|
||||
mndPersistTaskDeployReq(pTrans, pTask, &pTask->epSet, TDMT_STREAM_TASK_DEPLOY, pStream->fixedSinkVg.vgId);
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -361,7 +360,7 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) {
|
|||
ASSERT(taosArrayGetSize(pArray) == 1);
|
||||
SStreamTask* lastLevelTask = taosArrayGetP(pArray, 0);
|
||||
/*pTask->dispatchMsgType = TDMT_VND_TASK_MERGE_EXEC;*/
|
||||
pTask->dispatchMsgType = TDMT_VND_TASK_DISPATCH;
|
||||
pTask->dispatchMsgType = TDMT_STREAM_TASK_DISPATCH;
|
||||
pTask->dispatchType = TASK_DISPATCH__FIXED;
|
||||
|
||||
pTask->fixedEpDispatcher.taskId = lastLevelTask->taskId;
|
||||
|
|
@ -407,7 +406,7 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) {
|
|||
pTask->dispatchType = TASK_DISPATCH__SHUFFLE;
|
||||
|
||||
/*pTask->dispatchMsgType = TDMT_VND_TASK_WRITE_EXEC;*/
|
||||
pTask->dispatchMsgType = TDMT_VND_TASK_DISPATCH;
|
||||
pTask->dispatchMsgType = TDMT_STREAM_TASK_DISPATCH;
|
||||
SDbObj* pDb = mndAcquireDb(pMnode, pStream->targetDb);
|
||||
ASSERT(pDb);
|
||||
if (mndExtractDbInfo(pMnode, pDb, &pTask->shuffleDispatcher.dbInfo, NULL) < 0) {
|
||||
|
|
@ -438,7 +437,7 @@ int32_t mndScheduleStream(SMnode* pMnode, STrans* pTrans, SStreamObj* pStream) {
|
|||
} else {
|
||||
pTask->dispatchType = TASK_DISPATCH__FIXED;
|
||||
/*pTask->dispatchMsgType = TDMT_VND_TASK_WRITE_EXEC;*/
|
||||
pTask->dispatchMsgType = TDMT_VND_TASK_DISPATCH;
|
||||
pTask->dispatchMsgType = TDMT_STREAM_TASK_DISPATCH;
|
||||
SArray* pArray = taosArrayGetP(pStream->tasks, 0);
|
||||
// one sink only
|
||||
ASSERT(taosArrayGetSize(pArray) == 1);
|
||||
|
|
|
|||
|
|
@ -266,7 +266,7 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) {
|
|||
}
|
||||
|
||||
size = sizeof(SRetrieveMetaTableRsp) + sizeof(int32_t) + sizeof(SSysTableSchema) * pShow->pMeta->numOfColumns +
|
||||
blockDataGetSize(pBlock) + blockDataGetSerialMetaSize(pBlock);
|
||||
blockDataGetSize(pBlock) + blockDataGetSerialMetaSize(pBlock->info.numOfCols);
|
||||
|
||||
SRetrieveMetaTableRsp *pRsp = rpcMallocCont(size);
|
||||
if (pRsp == NULL) {
|
||||
|
|
|
|||
|
|
@ -40,6 +40,7 @@ static int32_t mndSmaGetVgEpSet(SMnode *pMnode, SDbObj *pDb, SVgEpSet **ppVgEpS
|
|||
static int32_t mndProcessMCreateSmaReq(SRpcMsg *pReq);
|
||||
static int32_t mndProcessMDropSmaReq(SRpcMsg *pReq);
|
||||
static int32_t mndProcessGetSmaReq(SRpcMsg *pReq);
|
||||
static int32_t mndProcessGetTbSmaReq(SRpcMsg *pReq);
|
||||
static int32_t mndRetrieveSma(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows);
|
||||
static void mndCancelGetNextSma(SMnode *pMnode, void *pIter);
|
||||
|
||||
|
|
@ -59,6 +60,7 @@ int32_t mndInitSma(SMnode *pMnode) {
|
|||
mndSetMsgHandle(pMnode, TDMT_VND_CREATE_SMA_RSP, mndTransProcessRsp);
|
||||
mndSetMsgHandle(pMnode, TDMT_VND_DROP_SMA_RSP, mndTransProcessRsp);
|
||||
mndSetMsgHandle(pMnode, TDMT_MND_GET_INDEX, mndProcessGetSmaReq);
|
||||
mndSetMsgHandle(pMnode, TDMT_MND_GET_TABLE_INDEX, mndProcessGetTbSmaReq);
|
||||
|
||||
mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndRetrieveSma);
|
||||
mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_INDEX, mndCancelGetNextSma);
|
||||
|
|
@ -870,6 +872,64 @@ static int32_t mndGetSma(SMnode *pMnode, SUserIndexReq *indexReq, SUserIndexRsp
|
|||
return code;
|
||||
}
|
||||
|
||||
static int32_t mndGetTableSma(SMnode *pMnode, STableIndexReq *indexReq, STableIndexRsp *rsp, bool *exist) {
|
||||
int32_t code = 0;
|
||||
SSmaObj *pSma = NULL;
|
||||
SSdb *pSdb = pMnode->pSdb;
|
||||
void *pIter = NULL;
|
||||
STableIndexInfo info;
|
||||
|
||||
while (1) {
|
||||
pIter = sdbFetch(pSdb, SDB_SMA, pIter, (void **)&pSma);
|
||||
if (pIter == NULL) break;
|
||||
|
||||
if (pSma->stb[0] != indexReq->tbFName[0] || strcmp(pSma->stb, indexReq->tbFName)) {
|
||||
continue;
|
||||
}
|
||||
|
||||
info.intervalUnit = pSma->intervalUnit;
|
||||
info.slidingUnit = pSma->slidingUnit;
|
||||
info.interval = pSma->interval;
|
||||
info.offset = pSma->offset;
|
||||
info.sliding = pSma->sliding;
|
||||
info.dstTbUid = pSma->dstTbUid;
|
||||
info.dstVgId = pSma->dstVgId;
|
||||
|
||||
SVgObj* pVg = mndAcquireVgroup(pMnode, pSma->dstVgId);
|
||||
if (pVg == NULL) {
|
||||
code = -1;
|
||||
sdbRelease(pSdb, pSma);
|
||||
return code;
|
||||
}
|
||||
info.epSet = mndGetVgroupEpset(pMnode, pVg);
|
||||
|
||||
info.expr = taosMemoryMalloc(pSma->exprLen + 1);
|
||||
if (info.expr == NULL) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
code = -1;
|
||||
sdbRelease(pSdb, pSma);
|
||||
return code;
|
||||
}
|
||||
|
||||
memcpy(info.expr, pSma->expr, pSma->exprLen);
|
||||
info.expr[pSma->exprLen] = 0;
|
||||
|
||||
if (NULL == taosArrayPush(rsp->pIndex, &info)) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
code = -1;
|
||||
taosMemoryFree(info.expr);
|
||||
sdbRelease(pSdb, pSma);
|
||||
return code;
|
||||
}
|
||||
|
||||
*exist = true;
|
||||
|
||||
sdbRelease(pSdb, pSma);
|
||||
}
|
||||
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t mndProcessGetSmaReq(SRpcMsg *pReq) {
|
||||
SUserIndexReq indexReq = {0};
|
||||
SMnode *pMnode = pReq->info.node;
|
||||
|
|
@ -916,6 +976,59 @@ _OVER:
|
|||
return code;
|
||||
}
|
||||
|
||||
static int32_t mndProcessGetTbSmaReq(SRpcMsg *pReq) {
|
||||
STableIndexReq indexReq = {0};
|
||||
SMnode *pMnode = pReq->info.node;
|
||||
int32_t code = -1;
|
||||
STableIndexRsp rsp = {0};
|
||||
bool exist = false;
|
||||
|
||||
if (tDeserializeSTableIndexReq(pReq->pCont, pReq->contLen, &indexReq) != 0) {
|
||||
terrno = TSDB_CODE_INVALID_MSG;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
rsp.pIndex = taosArrayInit(10, sizeof(STableIndexInfo));
|
||||
if (NULL == rsp.pIndex) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
code = -1;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
code = mndGetTableSma(pMnode, &indexReq, &rsp, &exist);
|
||||
if (code) {
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
if (!exist) {
|
||||
code = -1;
|
||||
terrno = TSDB_CODE_MND_DB_INDEX_NOT_EXIST;
|
||||
} else {
|
||||
int32_t contLen = tSerializeSTableIndexRsp(NULL, 0, &rsp);
|
||||
void *pRsp = rpcMallocCont(contLen);
|
||||
if (pRsp == NULL) {
|
||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||
code = -1;
|
||||
goto _OVER;
|
||||
}
|
||||
|
||||
tSerializeSTableIndexRsp(pRsp, contLen, &rsp);
|
||||
|
||||
pReq->info.rsp = pRsp;
|
||||
pReq->info.rspLen = contLen;
|
||||
|
||||
code = 0;
|
||||
}
|
||||
|
||||
_OVER:
|
||||
if (code != 0) {
|
||||
mError("failed to get table index %s since %s", indexReq.tbFName, terrstr());
|
||||
}
|
||||
|
||||
return code;
|
||||
}
|
||||
|
||||
|
||||
static int32_t mndRetrieveSma(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) {
|
||||
SMnode *pMnode = pReq->info.node;
|
||||
SSdb *pSdb = pMnode->pSdb;
|
||||
|
|
|
|||
|
|
@ -54,8 +54,8 @@ int32_t mndInitStream(SMnode *pMnode) {
|
|||
};
|
||||
|
||||
mndSetMsgHandle(pMnode, TDMT_MND_CREATE_STREAM, mndProcessCreateStreamReq);
|
||||
mndSetMsgHandle(pMnode, TDMT_VND_TASK_DEPLOY_RSP, mndTransProcessRsp);
|
||||
mndSetMsgHandle(pMnode, TDMT_SND_TASK_DEPLOY_RSP, mndTransProcessRsp);
|
||||
mndSetMsgHandle(pMnode, TDMT_STREAM_TASK_DEPLOY_RSP, mndTransProcessRsp);
|
||||
/*mndSetMsgHandle(pMnode, TDMT_SND_TASK_DEPLOY_RSP, mndTransProcessRsp);*/
|
||||
/*mndSetMsgHandle(pMnode, TDMT_MND_DROP_STREAM, mndProcessDropStreamReq);*/
|
||||
/*mndSetMsgHandle(pMnode, TDMT_MND_DROP_STREAM_RSP, mndProcessDropStreamInRsp);*/
|
||||
|
||||
|
|
|
|||
|
|
@ -193,7 +193,7 @@ int32_t mndInitSync(SMnode *pMnode) {
|
|||
void mndCleanupSync(SMnode *pMnode) {
|
||||
SSyncMgmt *pMgmt = &pMnode->syncMgmt;
|
||||
syncStop(pMgmt->sync);
|
||||
mDebug("sync:%" PRId64 " is stopped", pMgmt->sync);
|
||||
mDebug("mnode sync is stopped, id:%" PRId64, pMgmt->sync);
|
||||
|
||||
tsem_destroy(&pMgmt->syncSem);
|
||||
if (pMgmt->pWal != NULL) {
|
||||
|
|
|
|||
|
|
@ -117,10 +117,10 @@ static SSdbRaw *mndTransActionEncode(STrans *pTrans) {
|
|||
|
||||
int32_t dataPos = 0;
|
||||
SDB_SET_INT32(pRaw, dataPos, pTrans->id, _OVER)
|
||||
SDB_SET_INT16(pRaw, dataPos, pTrans->stage, _OVER)
|
||||
SDB_SET_INT16(pRaw, dataPos, pTrans->policy, _OVER)
|
||||
SDB_SET_INT16(pRaw, dataPos, pTrans->conflict, _OVER)
|
||||
SDB_SET_INT16(pRaw, dataPos, pTrans->exec, _OVER)
|
||||
SDB_SET_INT8(pRaw, dataPos, pTrans->stage, _OVER)
|
||||
SDB_SET_INT8(pRaw, dataPos, pTrans->policy, _OVER)
|
||||
SDB_SET_INT8(pRaw, dataPos, pTrans->conflict, _OVER)
|
||||
SDB_SET_INT8(pRaw, dataPos, pTrans->exec, _OVER)
|
||||
SDB_SET_INT64(pRaw, dataPos, pTrans->createdTime, _OVER)
|
||||
SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname, TSDB_DB_FNAME_LEN, _OVER)
|
||||
SDB_SET_INT32(pRaw, dataPos, pTrans->redoActionPos, _OVER)
|
||||
|
|
@ -256,15 +256,15 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
|
|||
|
||||
SDB_GET_INT32(pRaw, dataPos, &pTrans->id, _OVER)
|
||||
|
||||
int16_t stage = 0;
|
||||
int16_t policy = 0;
|
||||
int16_t conflict = 0;
|
||||
int16_t exec = 0;
|
||||
int8_t actionType = 0;
|
||||
SDB_GET_INT16(pRaw, dataPos, &stage, _OVER)
|
||||
SDB_GET_INT16(pRaw, dataPos, &policy, _OVER)
|
||||
SDB_GET_INT16(pRaw, dataPos, &conflict, _OVER)
|
||||
SDB_GET_INT16(pRaw, dataPos, &exec, _OVER)
|
||||
int8_t stage = 0;
|
||||
int8_t policy = 0;
|
||||
int8_t conflict = 0;
|
||||
int8_t exec = 0;
|
||||
int8_t actionType = 0;
|
||||
SDB_GET_INT8(pRaw, dataPos, &stage, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &policy, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &conflict, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &exec, _OVER)
|
||||
pTrans->stage = stage;
|
||||
pTrans->policy = policy;
|
||||
pTrans->conflict = conflict;
|
||||
|
|
@ -290,7 +290,8 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
|
|||
SDB_GET_INT32(pRaw, dataPos, &action.acceptableCode, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &actionType, _OVER)
|
||||
action.actionType = actionType;
|
||||
SDB_GET_INT8(pRaw, dataPos, &action.stage, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &stage, _OVER)
|
||||
action.stage = stage;
|
||||
if (action.actionType == TRANS_ACTION_RAW) {
|
||||
SDB_GET_INT8(pRaw, dataPos, &action.rawWritten, _OVER)
|
||||
SDB_GET_INT32(pRaw, dataPos, &dataLen, _OVER)
|
||||
|
|
@ -322,7 +323,8 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
|
|||
SDB_GET_INT32(pRaw, dataPos, &action.acceptableCode, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &actionType, _OVER)
|
||||
action.actionType = actionType;
|
||||
SDB_GET_INT8(pRaw, dataPos, &action.stage, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &stage, _OVER)
|
||||
action.stage = stage;
|
||||
if (action.actionType == TRANS_ACTION_RAW) {
|
||||
SDB_GET_INT8(pRaw, dataPos, &action.rawWritten, _OVER)
|
||||
SDB_GET_INT32(pRaw, dataPos, &dataLen, _OVER)
|
||||
|
|
@ -354,7 +356,8 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
|
|||
SDB_GET_INT32(pRaw, dataPos, &action.acceptableCode, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &actionType, _OVER)
|
||||
action.actionType = actionType;
|
||||
SDB_GET_INT8(pRaw, dataPos, &action.stage, _OVER)
|
||||
SDB_GET_INT8(pRaw, dataPos, &stage, _OVER)
|
||||
action.stage = stage;
|
||||
if (action.actionType) {
|
||||
SDB_GET_INT8(pRaw, dataPos, &action.rawWritten, _OVER)
|
||||
SDB_GET_INT32(pRaw, dataPos, &dataLen, _OVER)
|
||||
|
|
@ -878,7 +881,6 @@ static void mndTransResetActions(SMnode *pMnode, STrans *pTrans, SArray *pArray)
|
|||
mDebug("trans:%d, %s:%d execute status is reset", pTrans->id, mndTransStr(pAction->stage), action);
|
||||
}
|
||||
pAction->errCode = 0;
|
||||
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -890,11 +892,12 @@ static int32_t mndTransWriteSingleLog(SMnode *pMnode, STrans *pTrans, STransActi
|
|||
pAction->rawWritten = true;
|
||||
pAction->errCode = 0;
|
||||
code = 0;
|
||||
mDebug("trans:%d, %s:%d write to sdb", pTrans->id, mndTransStr(pAction->stage), pAction->id);
|
||||
mDebug("trans:%d, %s:%d write to sdb, type:%s status:%s", pTrans->id, mndTransStr(pAction->stage), pAction->id,
|
||||
sdbTableName(pAction->pRaw->type), sdbStatusName(pAction->pRaw->status));
|
||||
} else {
|
||||
pAction->errCode = (terrno != 0) ? terrno : code;
|
||||
mError("trans:%d, %s:%d failed to write sdb since %s", pTrans->id, mndTransStr(pAction->stage), pAction->id,
|
||||
terrstr());
|
||||
mError("trans:%d, %s:%d failed to write sdb since %s, type:%s status:%s", pTrans->id, mndTransStr(pAction->stage),
|
||||
pAction->id, terrstr(), sdbTableName(pAction->pRaw->type), sdbStatusName(pAction->pRaw->status));
|
||||
}
|
||||
|
||||
return code;
|
||||
|
|
@ -916,18 +919,26 @@ static int32_t mndTransSendSingleMsg(SMnode *pMnode, STrans *pTrans, STransActio
|
|||
}
|
||||
memcpy(rpcMsg.pCont, pAction->pCont, pAction->contLen);
|
||||
|
||||
char detail[1024] = {0};
|
||||
int32_t len = snprintf(detail, sizeof(detail), "msgType:%s numOfEps:%d inUse:%d", TMSG_INFO(pAction->msgType),
|
||||
pAction->epSet.numOfEps, pAction->epSet.inUse);
|
||||
for (int32_t i = 0; i < pTrans->lastErrorEpset.numOfEps; ++i) {
|
||||
len += snprintf(detail + len, sizeof(detail) - len, " ep:%d-%s:%u", i, pAction->epSet.eps[i].fqdn,
|
||||
pAction->epSet.eps[i].port);
|
||||
}
|
||||
|
||||
int32_t code = tmsgSendReq(&pAction->epSet, &rpcMsg);
|
||||
if (code == 0) {
|
||||
pAction->msgSent = 1;
|
||||
pAction->msgReceived = 0;
|
||||
pAction->errCode = 0;
|
||||
mDebug("trans:%d, %s:%d is sent to %s:%u", pTrans->id, mndTransStr(pAction->stage), pAction->id,
|
||||
pAction->epSet.eps[pAction->epSet.inUse].fqdn, pAction->epSet.eps[pAction->epSet.inUse].port);
|
||||
mDebug("trans:%d, %s:%d is sent, %s", pTrans->id, mndTransStr(pAction->stage), pAction->id, detail);
|
||||
} else {
|
||||
pAction->msgSent = 0;
|
||||
pAction->msgReceived = 0;
|
||||
pAction->errCode = (terrno != 0) ? terrno : code;
|
||||
mError("trans:%d, %s:%d not send since %s", pTrans->id, mndTransStr(pAction->stage), pAction->id, terrstr());
|
||||
mError("trans:%d, %s:%d not send since %s, %s", pTrans->id, mndTransStr(pAction->stage), pAction->id, terrstr(),
|
||||
detail);
|
||||
}
|
||||
|
||||
return code;
|
||||
|
|
@ -1424,9 +1435,9 @@ static int32_t mndRetrieveTrans(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBl
|
|||
if (epset.numOfEps > 0) {
|
||||
len += snprintf(detail + len, sizeof(detail) - len, "msgType:%s numOfEps:%d inUse:%d ",
|
||||
TMSG_INFO(pTrans->lastErrorMsgType), epset.numOfEps, epset.inUse);
|
||||
}
|
||||
for (int32_t i = 0; i < pTrans->lastErrorEpset.numOfEps; ++i) {
|
||||
len += snprintf(detail + len, sizeof(detail) - len, "ep:%d-%s:%u ", i, epset.eps[i].fqdn, epset.eps[i].port);
|
||||
for (int32_t i = 0; i < pTrans->lastErrorEpset.numOfEps; ++i) {
|
||||
len += snprintf(detail + len, sizeof(detail) - len, "ep:%d-%s:%u ", i, epset.eps[i].fqdn, epset.eps[i].port);
|
||||
}
|
||||
}
|
||||
}
|
||||
STR_WITH_MAXSIZE_TO_VARSTR(lastError, detail, pShow->pMeta->pSchemas[cols].bytes);
|
||||
|
|
|
|||
|
|
@ -392,6 +392,7 @@ int32_t sdbStopWrite(SSdb *pSdb, SSdbIter *pIter, bool isApply);
|
|||
int32_t sdbDoWrite(SSdb *pSdb, SSdbIter *pIter, void *pBuf, int32_t len);
|
||||
|
||||
const char *sdbTableName(ESdbType type);
|
||||
const char *sdbStatusName(ESdbStatus status);
|
||||
void sdbPrintOper(SSdb *pSdb, SSdbRow *pRow, const char *oper);
|
||||
int32_t sdbGetIdFromRaw(SSdb *pSdb, SSdbRaw *pRaw);
|
||||
|
||||
|
|
|
|||
|
|
@ -65,7 +65,7 @@ const char *sdbTableName(ESdbType type) {
|
|||
}
|
||||
}
|
||||
|
||||
static const char *sdbStatusName(ESdbStatus status) {
|
||||
const char *sdbStatusName(ESdbStatus status) {
|
||||
switch (status) {
|
||||
case SDB_STATUS_CREATING:
|
||||
return "creating";
|
||||
|
|
|
|||
|
|
@ -59,6 +59,7 @@ int32_t qndGetLoad(SQnode *pQnode, SQnodeLoad *pLoad) {
|
|||
pLoad->numOfProcessedFetch = stat.fetchProcessed;
|
||||
pLoad->numOfProcessedDrop = stat.dropProcessed;
|
||||
pLoad->numOfProcessedHb = stat.hbProcessed;
|
||||
pLoad->numOfProcessedDelete = stat.deleteProcessed;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -90,7 +90,7 @@ void sndProcessUMsg(SSnode *pSnode, SRpcMsg *pMsg) {
|
|||
// stream deploy
|
||||
// stream stop/resume
|
||||
// operator exec
|
||||
if (pMsg->msgType == TDMT_SND_TASK_DEPLOY) {
|
||||
if (pMsg->msgType == TDMT_STREAM_TASK_DEPLOY) {
|
||||
void *msg = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
|
||||
SStreamTask *pTask = taosMemoryMalloc(sizeof(SStreamTask));
|
||||
if (pTask == NULL) {
|
||||
|
|
|
|||
|
|
@ -36,12 +36,10 @@ target_sources(
|
|||
|
||||
# tsdb
|
||||
"src/tsdb/tsdbCommit.c"
|
||||
"src/tsdb/tsdbCommit2.c"
|
||||
"src/tsdb/tsdbFile.c"
|
||||
"src/tsdb/tsdbFS.c"
|
||||
"src/tsdb/tsdbOpen.c"
|
||||
"src/tsdb/tsdbMemTable.c"
|
||||
"src/tsdb/tsdbMemTable2.c"
|
||||
"src/tsdb/tsdbRead.c"
|
||||
"src/tsdb/tsdbReadImpl.c"
|
||||
"src/tsdb/tsdbWrite.c"
|
||||
|
|
|
|||
|
|
@ -71,6 +71,9 @@ int32_t vnodeSnapshotReaderClose(SVSnapshotReader *pReader);
|
|||
int32_t vnodeSnapshotRead(SVSnapshotReader *pReader, const void **ppData, uint32_t *nData);
|
||||
int32_t vnodeProcessCreateTSma(SVnode *pVnode, void *pCont, uint32_t contLen);
|
||||
|
||||
void vnodeProposeMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs);
|
||||
void vnodeApplyMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs);
|
||||
|
||||
// meta
|
||||
typedef struct SMeta SMeta; // todo: remove
|
||||
typedef struct SMetaReader SMetaReader;
|
||||
|
|
|
|||
|
|
@ -39,24 +39,24 @@ typedef struct SDelOp SDelOp;
|
|||
|
||||
static int tsdbKeyCmprFn(const void *p1, const void *p2);
|
||||
|
||||
// tsdbMemTable2.c ==============================================================================================
|
||||
typedef struct SMemTable SMemTable;
|
||||
typedef struct SMemData SMemData;
|
||||
typedef struct SMemDataIter SMemDataIter;
|
||||
// tsdbMemTable ==============================================================================================
|
||||
typedef struct STbData STbData;
|
||||
typedef struct SMemTable SMemTable;
|
||||
typedef struct STbDataIter STbDataIter;
|
||||
typedef struct SMergeInfo SMergeInfo;
|
||||
typedef struct STable STable;
|
||||
|
||||
int32_t tsdbMemTableCreate2(STsdb *pTsdb, SMemTable **ppMemTable);
|
||||
void tsdbMemTableDestroy2(SMemTable *pMemTable);
|
||||
int32_t tsdbInsertTableData2(STsdb *pTsdb, int64_t version, SVSubmitBlk *pSubmitBlk);
|
||||
int32_t tsdbDeleteTableData2(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey);
|
||||
// SMemTable
|
||||
int32_t tsdbMemTableCreate(STsdb *pTsdb, SMemTable **ppMemTable);
|
||||
void tsdbMemTableDestroy(SMemTable *pMemTable);
|
||||
void tsdbGetTbDataFromMemTable(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, STbData **ppTbData);
|
||||
|
||||
/* SMemDataIter */
|
||||
void tsdbMemDataIterOpen(SMemData *pMemData, TSDBKEY *pKey, int8_t backward, SMemDataIter *pIter);
|
||||
bool tsdbMemDataIterNext(SMemDataIter *pIter);
|
||||
void tsdbMemDataIterGet(SMemDataIter *pIter, TSDBROW **ppRow);
|
||||
|
||||
// tsdbCommit2.c ==============================================================================================
|
||||
int32_t tsdbBegin2(STsdb *pTsdb);
|
||||
int32_t tsdbCommit2(STsdb *pTsdb);
|
||||
// STbDataIter
|
||||
int32_t tsdbTbDataIterCreate(STbData *pTbData, TSDBKEY *pFrom, int8_t backward, STbDataIter **ppIter);
|
||||
void *tsdbTbDataIterDestroy(STbDataIter *pIter);
|
||||
void tsdbTbDataIterOpen(STbData *pTbData, TSDBKEY *pFrom, int8_t backward, STbDataIter *pIter);
|
||||
bool tsdbTbDataIterNext(STbDataIter *pIter);
|
||||
bool tsdbTbDataIterGet(STbDataIter *pIter, TSDBROW *pRow);
|
||||
|
||||
// tsdbFile.c ==============================================================================================
|
||||
typedef int32_t TSDB_FILE_T;
|
||||
|
|
@ -124,17 +124,6 @@ int tsdbRLockFS(STsdbFS *pFs);
|
|||
int tsdbWLockFS(STsdbFS *pFs);
|
||||
int tsdbUnLockFS(STsdbFS *pFs);
|
||||
|
||||
// tsdbMemTable ================
|
||||
typedef struct STbData STbData;
|
||||
typedef struct STsdbMemTable STsdbMemTable;
|
||||
typedef struct SMergeInfo SMergeInfo;
|
||||
typedef struct STable STable;
|
||||
|
||||
int tsdbMemTableCreate(STsdb *pTsdb, STsdbMemTable **ppMemTable);
|
||||
void tsdbMemTableDestroy(STsdbMemTable *pMemTable);
|
||||
int tsdbLoadDataFromCache(STsdb *pTsdb, STable *pTable, SSkipListIterator *pIter, TSKEY maxKey, int maxRowsToRead,
|
||||
SDataCols *pCols, TKEY *filterKeys, int nFilterKeys, bool keepDup, SMergeInfo *pMergeInfo);
|
||||
|
||||
// structs
|
||||
typedef struct {
|
||||
int minFid;
|
||||
|
|
@ -145,30 +134,27 @@ typedef struct {
|
|||
|
||||
#define TSDB_DATA_DIR_LEN 6 // adapt accordingly
|
||||
struct STsdb {
|
||||
char *path;
|
||||
SVnode *pVnode;
|
||||
TdThreadMutex mutex;
|
||||
char dir[TSDB_DATA_DIR_LEN];
|
||||
bool repoLocked;
|
||||
STsdbKeepCfg keepCfg;
|
||||
STsdbMemTable *mem;
|
||||
STsdbMemTable *imem;
|
||||
SRtn rtn;
|
||||
STsdbFS *fs;
|
||||
char *path;
|
||||
SVnode *pVnode;
|
||||
TdThreadMutex mutex;
|
||||
char dir[TSDB_DATA_DIR_LEN];
|
||||
bool repoLocked;
|
||||
STsdbKeepCfg keepCfg;
|
||||
SMemTable *mem;
|
||||
SMemTable *imem;
|
||||
SRtn rtn;
|
||||
STsdbFS *fs;
|
||||
};
|
||||
|
||||
#if 1 // ======================================
|
||||
|
||||
struct STable {
|
||||
uint64_t tid;
|
||||
uint64_t suid;
|
||||
uint64_t uid;
|
||||
STSchema *pSchema; // latest schema
|
||||
STSchema *pCacheSchema; // cached cache
|
||||
};
|
||||
|
||||
#define TABLE_TID(t) (t)->tid
|
||||
#define TABLE_UID(t) (t)->uid
|
||||
|
||||
// int tsdbPrepareCommit(STsdb *pTsdb);
|
||||
typedef enum {
|
||||
TSDB_FILE_HEAD = 0, // .head
|
||||
|
|
@ -206,27 +192,44 @@ struct SDFileSet {
|
|||
SDFile files[TSDB_FILE_MAX];
|
||||
};
|
||||
|
||||
struct STbData {
|
||||
tb_uid_t uid;
|
||||
TSKEY keyMin;
|
||||
TSKEY keyMax;
|
||||
int64_t minVer;
|
||||
int64_t maxVer;
|
||||
int64_t nrows;
|
||||
SSkipList *pData;
|
||||
struct TSDBKEY {
|
||||
int64_t version;
|
||||
TSKEY ts;
|
||||
};
|
||||
|
||||
struct STsdbMemTable {
|
||||
SVBufPool *pPool;
|
||||
T_REF_DECLARE()
|
||||
SRWLatch latch;
|
||||
TSKEY keyMin;
|
||||
TSKEY keyMax;
|
||||
int64_t minVer;
|
||||
int64_t maxVer;
|
||||
int64_t nRow;
|
||||
SSkipList *pSlIdx; // SSkiplist<STbData>
|
||||
SHashObj *pHashIdx;
|
||||
typedef struct SMemSkipListNode SMemSkipListNode;
|
||||
struct SMemSkipListNode {
|
||||
int8_t level;
|
||||
SMemSkipListNode *forwards[0];
|
||||
};
|
||||
typedef struct SMemSkipList {
|
||||
uint32_t seed;
|
||||
int64_t size;
|
||||
int8_t maxLevel;
|
||||
int8_t level;
|
||||
SMemSkipListNode *pHead;
|
||||
SMemSkipListNode *pTail;
|
||||
} SMemSkipList;
|
||||
|
||||
struct STbData {
|
||||
tb_uid_t suid;
|
||||
tb_uid_t uid;
|
||||
TSDBKEY minKey;
|
||||
TSDBKEY maxKey;
|
||||
SDelOp *pHead;
|
||||
SDelOp *pTail;
|
||||
SMemSkipList sl;
|
||||
};
|
||||
|
||||
struct SMemTable {
|
||||
SRWLatch latch;
|
||||
STsdb *pTsdb;
|
||||
int32_t nRef;
|
||||
TSDBKEY minKey;
|
||||
TSDBKEY maxKey;
|
||||
int64_t nRow;
|
||||
int64_t nDelOp;
|
||||
SArray *aTbData; // SArray<STbData>
|
||||
};
|
||||
|
||||
struct STsdbFSMeta {
|
||||
|
|
@ -237,9 +240,11 @@ struct STsdbFSMeta {
|
|||
|
||||
// ==================
|
||||
typedef struct {
|
||||
STsdbFSMeta meta; // FS meta
|
||||
SArray *df; // data file array
|
||||
SArray *sf; // sma data file array v2f1900.index_name_1
|
||||
STsdbFSMeta meta; // FS meta
|
||||
SDFile cacheFile; // cache file
|
||||
SDFile tombstone; // tomestome file
|
||||
SArray *df; // data file array
|
||||
SArray *sf; // sma data file array v2f1900.index_name_1
|
||||
} SFSStatus;
|
||||
|
||||
struct STsdbFS {
|
||||
|
|
@ -292,16 +297,24 @@ static void *taosTZfree(void *ptr);
|
|||
static size_t taosTSizeof(void *ptr);
|
||||
static void taosTMemset(void *ptr, int c);
|
||||
|
||||
static FORCE_INLINE STSRow *tsdbNextIterRow(SSkipListIterator *pIter) {
|
||||
struct TSDBROW {
|
||||
int64_t version;
|
||||
STSRow *pTSRow;
|
||||
};
|
||||
|
||||
static FORCE_INLINE STSRow *tsdbNextIterRow(STbDataIter *pIter) {
|
||||
TSDBROW row;
|
||||
|
||||
if (pIter == NULL) return NULL;
|
||||
|
||||
SSkipListNode *node = tSkipListIterGet(pIter);
|
||||
if (node == NULL) return NULL;
|
||||
if (tsdbTbDataIterGet(pIter, &row)) {
|
||||
return row.pTSRow;
|
||||
}
|
||||
|
||||
return (STSRow *)SL_GET_NODE_DATA(node);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
static FORCE_INLINE TSKEY tsdbNextIterKey(SSkipListIterator *pIter) {
|
||||
static FORCE_INLINE TSKEY tsdbNextIterKey(STbDataIter *pIter) {
|
||||
STSRow *row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL) return TSDB_DATA_TIMESTAMP_NULL;
|
||||
|
||||
|
|
@ -311,11 +324,6 @@ static FORCE_INLINE TSKEY tsdbNextIterKey(SSkipListIterator *pIter) {
|
|||
// tsdbReadImpl
|
||||
typedef struct SReadH SReadH;
|
||||
|
||||
struct TSDBKEY {
|
||||
int64_t version;
|
||||
TSKEY ts;
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
uint64_t suid;
|
||||
uint64_t uid;
|
||||
|
|
@ -354,7 +362,7 @@ typedef struct {
|
|||
|
||||
typedef struct {
|
||||
int32_t delimiter; // For recovery usage
|
||||
int32_t tid;
|
||||
uint64_t suid;
|
||||
uint64_t uid;
|
||||
SBlock blocks[];
|
||||
} SBlockInfo;
|
||||
|
|
@ -650,11 +658,6 @@ struct SFSIter {
|
|||
#define TSDB_FS_ITER_FORWARD TSDB_ORDER_ASC
|
||||
#define TSDB_FS_ITER_BACKWARD TSDB_ORDER_DESC
|
||||
|
||||
struct TSDBROW {
|
||||
int64_t version;
|
||||
STSRow2 tsRow;
|
||||
};
|
||||
|
||||
struct TABLEID {
|
||||
tb_uid_t suid;
|
||||
tb_uid_t uid;
|
||||
|
|
@ -675,17 +678,6 @@ typedef struct {
|
|||
TSKEY eKey;
|
||||
} SDelInfo;
|
||||
|
||||
struct SMemTable {
|
||||
STsdb *pTsdb;
|
||||
int32_t nRef;
|
||||
TSDBKEY minKey;
|
||||
TSDBKEY maxKey;
|
||||
int64_t nRows;
|
||||
int64_t nDelOp;
|
||||
SArray *aSkmInfo;
|
||||
SArray *aMemData;
|
||||
};
|
||||
|
||||
static FORCE_INLINE int tsdbKeyCmprFn(const void *p1, const void *p2) {
|
||||
TSDBKEY *pKey1 = (TSDBKEY *)p1;
|
||||
TSDBKEY *pKey2 = (TSDBKEY *)p2;
|
||||
|
|
@ -705,32 +697,10 @@ static FORCE_INLINE int tsdbKeyCmprFn(const void *p1, const void *p2) {
|
|||
return 0;
|
||||
}
|
||||
|
||||
typedef struct SMemSkipListNode SMemSkipListNode;
|
||||
typedef struct SMemSkipList {
|
||||
uint32_t seed;
|
||||
int32_t size;
|
||||
int8_t maxLevel;
|
||||
int8_t level;
|
||||
SMemSkipListNode *pHead;
|
||||
SMemSkipListNode *pTail;
|
||||
} SMemSkipList;
|
||||
|
||||
struct SMemData {
|
||||
tb_uid_t suid;
|
||||
tb_uid_t uid;
|
||||
TSDBKEY minKey;
|
||||
TSDBKEY maxKey;
|
||||
SDelOp *delOpHead;
|
||||
SDelOp *delOpTail;
|
||||
SMemSkipList sl;
|
||||
};
|
||||
|
||||
struct SMemDataIter {
|
||||
SMemData *pMemData;
|
||||
struct STbDataIter {
|
||||
STbData *pTbData;
|
||||
int8_t backward;
|
||||
TSDBROW *pRow;
|
||||
SMemSkipListNode *pNode; // current node
|
||||
TSDBROW row;
|
||||
SMemSkipListNode *pNode;
|
||||
};
|
||||
|
||||
#endif
|
||||
|
|
|
|||
|
|
@ -84,7 +84,6 @@ int32_t vnodeAsyncCommit(SVnode* pVnode);
|
|||
int32_t vnodeSyncOpen(SVnode* pVnode, char* path);
|
||||
void vnodeSyncStart(SVnode* pVnode);
|
||||
void vnodeSyncClose(SVnode* pVnode);
|
||||
int32_t vnodeSyncAlter(SVnode* pVnode, SRpcMsg* pMsg);
|
||||
|
||||
#ifdef __cplusplus
|
||||
}
|
||||
|
|
|
|||
|
|
@ -112,11 +112,13 @@ int32_t metaDropTSma(SMeta* pMeta, int64_t indexUid);
|
|||
// tsdb
|
||||
int tsdbOpen(SVnode* pVnode, STsdb** ppTsdb, const char* dir, STsdbKeepCfg* pKeepCfg);
|
||||
int tsdbClose(STsdb** pTsdb);
|
||||
int tsdbBegin(STsdb* pTsdb);
|
||||
int32_t tsdbBegin(STsdb* pTsdb);
|
||||
int32_t tsdbCommit(STsdb* pTsdb);
|
||||
int tsdbScanAndConvertSubmitMsg(STsdb* pTsdb, SSubmitReq* pMsg);
|
||||
int tsdbInsertData(STsdb* pTsdb, int64_t version, SSubmitReq* pMsg, SSubmitRsp* pRsp);
|
||||
int tsdbInsertTableData(STsdb* pTsdb, SSubmitMsgIter* pMsgIter, SSubmitBlk* pBlock, SSubmitBlkRsp* pRsp);
|
||||
int32_t tsdbInsertTableData(STsdb* pTsdb, int64_t version, SSubmitMsgIter* pMsgIter, SSubmitBlk* pBlock,
|
||||
SSubmitBlkRsp* pRsp);
|
||||
int32_t tsdbDeleteTableData(STsdb* pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey);
|
||||
tsdbReaderT* tsdbQueryTables(SVnode* pVnode, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId,
|
||||
uint64_t taskId);
|
||||
tsdbReaderT tsdbQueryCacheLastT(STsdb* tsdb, SQueryTableDataCond* pCond, STableListInfo* tableList, uint64_t qId,
|
||||
|
|
@ -159,18 +161,6 @@ int32_t tdUpdateTbUidList(SSma* pSma, STbUidStore* pUidStore);
|
|||
void tdUidStoreDestory(STbUidStore* pStore);
|
||||
void* tdUidStoreFree(STbUidStore* pStore);
|
||||
|
||||
#if 0
|
||||
int32_t tsdbUpdateSmaWindow(STsdb* pTsdb, SSubmitReq* pMsg, int64_t version);
|
||||
int32_t tsdbCreateTSma(STsdb* pTsdb, char* pMsg);
|
||||
int32_t tsdbInsertTSmaData(STsdb* pTsdb, int64_t indexUid, const char* msg);
|
||||
int32_t tsdbRegisterRSma(STsdb* pTsdb, SMeta* pMeta, SVCreateStbReq* pReq, SMsgCb* pMsgCb);
|
||||
int32_t tsdbFetchTbUidList(STsdb* pTsdb, STbUidStore** ppStore, tb_uid_t suid, tb_uid_t uid);
|
||||
int32_t tsdbUpdateTbUidList(STsdb* pTsdb, STbUidStore* pUidStore);
|
||||
void tsdbUidStoreDestory(STbUidStore* pStore);
|
||||
void* tsdbUidStoreFree(STbUidStore* pStore);
|
||||
int32_t tsdbTriggerRSma(STsdb* pTsdb, void* pMsg, int32_t inputType);
|
||||
#endif
|
||||
|
||||
typedef struct {
|
||||
int8_t streamType; // sma or other
|
||||
int8_t dstType;
|
||||
|
|
@ -230,8 +220,10 @@ struct SVnode {
|
|||
SWal* pWal;
|
||||
STQ* pTq;
|
||||
SSink* pSink;
|
||||
int64_t sync;
|
||||
tsem_t canCommit;
|
||||
int64_t sync;
|
||||
int32_t syncCount;
|
||||
sem_t syncSem;
|
||||
SQHandle* pQuery;
|
||||
};
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ void metaReaderClear(SMetaReader *pReader) {
|
|||
}
|
||||
|
||||
int metaGetTableEntryByVersion(SMetaReader *pReader, int64_t version, tb_uid_t uid) {
|
||||
SMeta * pMeta = pReader->pMeta;
|
||||
SMeta *pMeta = pReader->pMeta;
|
||||
STbDbKey tbDbKey = {.version = version, .uid = uid};
|
||||
|
||||
// query table.db
|
||||
|
|
@ -54,7 +54,7 @@ _err:
|
|||
}
|
||||
|
||||
int metaGetTableEntryByUid(SMetaReader *pReader, tb_uid_t uid) {
|
||||
SMeta * pMeta = pReader->pMeta;
|
||||
SMeta *pMeta = pReader->pMeta;
|
||||
int64_t version;
|
||||
|
||||
// query uid.idx
|
||||
|
|
@ -68,7 +68,7 @@ int metaGetTableEntryByUid(SMetaReader *pReader, tb_uid_t uid) {
|
|||
}
|
||||
|
||||
int metaGetTableEntryByName(SMetaReader *pReader, const char *name) {
|
||||
SMeta * pMeta = pReader->pMeta;
|
||||
SMeta *pMeta = pReader->pMeta;
|
||||
tb_uid_t uid;
|
||||
|
||||
// query name.idx
|
||||
|
|
@ -82,7 +82,7 @@ int metaGetTableEntryByName(SMetaReader *pReader, const char *name) {
|
|||
}
|
||||
|
||||
tb_uid_t metaGetTableEntryUidByName(SMeta *pMeta, const char *name) {
|
||||
void * pData = NULL;
|
||||
void *pData = NULL;
|
||||
int nData = 0;
|
||||
tb_uid_t uid = 0;
|
||||
|
||||
|
|
@ -134,7 +134,7 @@ void metaCloseTbCursor(SMTbCursor *pTbCur) {
|
|||
|
||||
int metaTbCursorNext(SMTbCursor *pTbCur) {
|
||||
int ret;
|
||||
void * pBuf;
|
||||
void *pBuf;
|
||||
STbCfg tbCfg;
|
||||
|
||||
for (;;) {
|
||||
|
|
@ -155,7 +155,7 @@ int metaTbCursorNext(SMTbCursor *pTbCur) {
|
|||
}
|
||||
|
||||
SSchemaWrapper *metaGetTableSchema(SMeta *pMeta, tb_uid_t uid, int32_t sver, bool isinline) {
|
||||
void * pData = NULL;
|
||||
void *pData = NULL;
|
||||
int nData = 0;
|
||||
int64_t version;
|
||||
SSchemaWrapper schema = {0};
|
||||
|
|
@ -163,37 +163,47 @@ SSchemaWrapper *metaGetTableSchema(SMeta *pMeta, tb_uid_t uid, int32_t sver, boo
|
|||
SDecoder dc = {0};
|
||||
|
||||
metaRLock(pMeta);
|
||||
if (sver < 0) {
|
||||
if (tdbTbGet(pMeta->pUidIdx, &uid, sizeof(uid), &pData, &nData) < 0) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
version = *(int64_t *)pData;
|
||||
|
||||
tdbTbGet(pMeta->pTbDb, &(STbDbKey){.uid = uid, .version = version}, sizeof(STbDbKey), &pData, &nData);
|
||||
|
||||
SMetaEntry me = {0};
|
||||
tDecoderInit(&dc, pData, nData);
|
||||
metaDecodeEntry(&dc, &me);
|
||||
if (me.type == TSDB_SUPER_TABLE) {
|
||||
pSchema = tCloneSSchemaWrapper(&me.stbEntry.schemaRow);
|
||||
} else if (me.type == TSDB_NORMAL_TABLE) {
|
||||
pSchema = tCloneSSchemaWrapper(&me.ntbEntry.schemaRow);
|
||||
} else {
|
||||
ASSERT(0);
|
||||
}
|
||||
tDecoderClear(&dc);
|
||||
} else {
|
||||
if (tdbTbGet(pMeta->pSkmDb, &(SSkmDbKey){.uid = uid, .sver = sver}, sizeof(SSkmDbKey), &pData, &nData) < 0) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
tDecoderInit(&dc, pData, nData);
|
||||
tDecodeSSchemaWrapper(&dc, &schema);
|
||||
pSchema = tCloneSSchemaWrapper(&schema);
|
||||
tDecoderClear(&dc);
|
||||
_query:
|
||||
if (tdbTbGet(pMeta->pUidIdx, &uid, sizeof(uid), &pData, &nData) < 0) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
version = *(int64_t *)pData;
|
||||
|
||||
tdbTbGet(pMeta->pTbDb, &(STbDbKey){.uid = uid, .version = version}, sizeof(STbDbKey), &pData, &nData);
|
||||
SMetaEntry me = {0};
|
||||
tDecoderInit(&dc, pData, nData);
|
||||
metaDecodeEntry(&dc, &me);
|
||||
if (me.type == TSDB_SUPER_TABLE) {
|
||||
if (sver == -1 || sver == me.stbEntry.schemaRow.version) {
|
||||
pSchema = tCloneSSchemaWrapper(&me.stbEntry.schemaRow);
|
||||
tDecoderClear(&dc);
|
||||
goto _exit;
|
||||
}
|
||||
} else if (me.type == TSDB_CHILD_TABLE) {
|
||||
uid = me.ctbEntry.suid;
|
||||
tDecoderClear(&dc);
|
||||
goto _query;
|
||||
} else {
|
||||
if (sver == -1 || sver == me.ntbEntry.schemaRow.version) {
|
||||
pSchema = tCloneSSchemaWrapper(&me.ntbEntry.schemaRow);
|
||||
tDecoderClear(&dc);
|
||||
goto _exit;
|
||||
}
|
||||
}
|
||||
tDecoderClear(&dc);
|
||||
|
||||
// query from skm db
|
||||
if (tdbTbGet(pMeta->pSkmDb, &(SSkmDbKey){.uid = uid, .sver = sver}, sizeof(SSkmDbKey), &pData, &nData) < 0) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
tDecoderInit(&dc, pData, nData);
|
||||
tDecodeSSchemaWrapper(&dc, &schema);
|
||||
pSchema = tCloneSSchemaWrapper(&schema);
|
||||
tDecoderClear(&dc);
|
||||
|
||||
_exit:
|
||||
metaULock(pMeta);
|
||||
tdbFree(pData);
|
||||
return pSchema;
|
||||
|
|
@ -205,11 +215,11 @@ _err:
|
|||
}
|
||||
|
||||
struct SMCtbCursor {
|
||||
SMeta * pMeta;
|
||||
TBC * pCur;
|
||||
SMeta *pMeta;
|
||||
TBC *pCur;
|
||||
tb_uid_t suid;
|
||||
void * pKey;
|
||||
void * pVal;
|
||||
void *pKey;
|
||||
void *pVal;
|
||||
int kLen;
|
||||
int vLen;
|
||||
};
|
||||
|
|
@ -279,25 +289,13 @@ tb_uid_t metaCtbCursorNext(SMCtbCursor *pCtbCur) {
|
|||
}
|
||||
|
||||
STSchema *metaGetTbTSchema(SMeta *pMeta, tb_uid_t uid, int32_t sver) {
|
||||
tb_uid_t quid;
|
||||
SMetaReader mr = {0};
|
||||
STSchema * pTSchema = NULL;
|
||||
// SMetaReader mr = {0};
|
||||
STSchema *pTSchema = NULL;
|
||||
SSchemaWrapper *pSW = NULL;
|
||||
STSchemaBuilder sb = {0};
|
||||
SSchema * pSchema;
|
||||
SSchema *pSchema;
|
||||
|
||||
metaReaderInit(&mr, pMeta, 0);
|
||||
metaGetTableEntryByUid(&mr, uid);
|
||||
|
||||
if (mr.me.type == TSDB_CHILD_TABLE) {
|
||||
quid = mr.me.ctbEntry.suid;
|
||||
} else {
|
||||
quid = uid;
|
||||
}
|
||||
|
||||
metaReaderClear(&mr);
|
||||
|
||||
pSW = metaGetTableSchema(pMeta, quid, sver, 0);
|
||||
pSW = metaGetTableSchema(pMeta, uid, sver, 0);
|
||||
if (!pSW) return NULL;
|
||||
|
||||
tdInitTSchemaBuilder(&sb, pSW->version);
|
||||
|
|
@ -321,11 +319,11 @@ int metaGetTbNum(SMeta *pMeta) {
|
|||
}
|
||||
|
||||
typedef struct {
|
||||
SMeta * pMeta;
|
||||
TBC * pCur;
|
||||
SMeta *pMeta;
|
||||
TBC *pCur;
|
||||
tb_uid_t uid;
|
||||
void * pKey;
|
||||
void * pVal;
|
||||
void *pKey;
|
||||
void *pVal;
|
||||
int kLen;
|
||||
int vLen;
|
||||
} SMSmaCursor;
|
||||
|
|
@ -397,7 +395,7 @@ tb_uid_t metaSmaCursorNext(SMSmaCursor *pSmaCur) {
|
|||
|
||||
STSmaWrapper *metaGetSmaInfoByTable(SMeta *pMeta, tb_uid_t uid, bool deepCopy) {
|
||||
STSmaWrapper *pSW = NULL;
|
||||
SArray * pSmaIds = NULL;
|
||||
SArray *pSmaIds = NULL;
|
||||
|
||||
if (!(pSmaIds = metaGetSmaIdsByTable(pMeta, uid))) {
|
||||
return NULL;
|
||||
|
|
@ -421,7 +419,7 @@ STSmaWrapper *metaGetSmaInfoByTable(SMeta *pMeta, tb_uid_t uid, bool deepCopy) {
|
|||
metaReaderInit(&mr, pMeta, 0);
|
||||
int64_t smaId;
|
||||
int smaIdx = 0;
|
||||
STSma * pTSma = NULL;
|
||||
STSma *pTSma = NULL;
|
||||
for (int i = 0; i < pSW->number; ++i) {
|
||||
smaId = *(tb_uid_t *)taosArrayGet(pSmaIds, i);
|
||||
if (metaGetTableEntryByUid(&mr, smaId) < 0) {
|
||||
|
|
@ -469,7 +467,7 @@ _err:
|
|||
}
|
||||
|
||||
STSma *metaGetSmaInfoByIndex(SMeta *pMeta, int64_t indexUid) {
|
||||
STSma * pTSma = NULL;
|
||||
STSma *pTSma = NULL;
|
||||
SMetaReader mr = {0};
|
||||
metaReaderInit(&mr, pMeta, 0);
|
||||
if (metaGetTableEntryByUid(&mr, indexUid) < 0) {
|
||||
|
|
@ -491,7 +489,7 @@ STSma *metaGetSmaInfoByIndex(SMeta *pMeta, int64_t indexUid) {
|
|||
}
|
||||
|
||||
SArray *metaGetSmaIdsByTable(SMeta *pMeta, tb_uid_t uid) {
|
||||
SArray * pUids = NULL;
|
||||
SArray *pUids = NULL;
|
||||
SSmaIdxKey *pSmaIdxKey = NULL;
|
||||
|
||||
SMSmaCursor *pCur = metaOpenSmaCursor(pMeta, uid);
|
||||
|
|
@ -529,7 +527,7 @@ SArray *metaGetSmaIdsByTable(SMeta *pMeta, tb_uid_t uid) {
|
|||
}
|
||||
|
||||
SArray *metaGetSmaTbUids(SMeta *pMeta) {
|
||||
SArray * pUids = NULL;
|
||||
SArray *pUids = NULL;
|
||||
SSmaIdxKey *pSmaIdxKey = NULL;
|
||||
tb_uid_t lastUid = 0;
|
||||
|
||||
|
|
@ -591,13 +589,13 @@ const void *metaGetTableTagVal(SMetaEntry *pEntry, int16_t type, STagVal *val) {
|
|||
}
|
||||
|
||||
typedef struct {
|
||||
SMeta * pMeta;
|
||||
TBC * pCur;
|
||||
SMeta *pMeta;
|
||||
TBC *pCur;
|
||||
tb_uid_t suid;
|
||||
int16_t cid;
|
||||
int16_t type;
|
||||
void * pKey;
|
||||
void * pVal;
|
||||
void *pKey;
|
||||
void *pVal;
|
||||
int32_t kLen;
|
||||
int32_t vLen;
|
||||
} SIdxCursor;
|
||||
|
|
@ -621,7 +619,7 @@ int32_t metaFilteTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
|
|||
int32_t nKey = 0;
|
||||
|
||||
int32_t nTagData = 0;
|
||||
void * tagData = NULL;
|
||||
void *tagData = NULL;
|
||||
|
||||
if (IS_VAR_DATA_TYPE(param->type)) {
|
||||
tagData = varDataVal(param->val);
|
||||
|
|
@ -640,7 +638,7 @@ int32_t metaFilteTableIds(SMeta *pMeta, SMetaFltParam *param, SArray *pUids) {
|
|||
goto END;
|
||||
}
|
||||
|
||||
void * entryKey = NULL, *entryVal = NULL;
|
||||
void *entryKey = NULL, *entryVal = NULL;
|
||||
int32_t nEntryKey, nEntryVal;
|
||||
while (1) {
|
||||
valid = tdbTbcGet(pCursor->pCur, (const void **)&entryKey, &nEntryKey, (const void **)&entryVal, &nEntryVal);
|
||||
|
|
|
|||
|
|
@ -61,14 +61,14 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const
|
|||
if (tTagToValArray((const STag *)data, &pTagVals) != 0) {
|
||||
return -1;
|
||||
}
|
||||
char key[512] = {0};
|
||||
|
||||
SIndexMultiTerm *terms = indexMultiTermCreate();
|
||||
int16_t nCols = taosArrayGetSize(pTagVals);
|
||||
for (int i = 0; i < nCols; i++) {
|
||||
STagVal *pTagVal = (STagVal *)taosArrayGet(pTagVals, i);
|
||||
char type = pTagVal->type;
|
||||
sprintf(key, "%s_%s", tagName, pTagVal->pKey);
|
||||
|
||||
char * key = pTagVal->pKey;
|
||||
int32_t nKey = strlen(key);
|
||||
|
||||
SIndexTerm *term = NULL;
|
||||
|
|
@ -93,14 +93,13 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const
|
|||
} else if (type == TSDB_DATA_TYPE_BOOL) {
|
||||
int val = *(int *)(&pTagVal->i64);
|
||||
int len = 0;
|
||||
term = indexTermCreate(suid, ADD_VALUE, type, key, nKey, (const char *)&val, len);
|
||||
term = indexTermCreate(suid, ADD_VALUE, TSDB_DATA_TYPE_INT, key, nKey, (const char *)&val, len);
|
||||
}
|
||||
if (term != NULL) {
|
||||
indexMultiTermAdd(terms, term);
|
||||
}
|
||||
memset(key, 0, sizeof(key));
|
||||
}
|
||||
tIndexJsonPut(pMeta->pTagIvtIdx, terms, tuid);
|
||||
indexJsonPut(pMeta->pTagIvtIdx, terms, tuid);
|
||||
indexMultiTermDestroy(terms);
|
||||
#endif
|
||||
return 0;
|
||||
|
|
|
|||
|
|
@ -47,15 +47,16 @@ int32_t tdUpdateExpireWindow(SSma* pSma, const SSubmitReq* pMsg, int64_t version
|
|||
int32_t tdGetTSmaData(SSma* pSma, char* pData, int64_t indexUid, TSKEY querySKey, int32_t nMaxResult) {
|
||||
int32_t code = TSDB_CODE_SUCCESS;
|
||||
if ((code = tdGetTSmaDataImpl(pSma, pData, indexUid, querySKey, nMaxResult)) < 0) {
|
||||
smaWarn("vgId:%d, get tSma data failed since %s", SMA_VID(pSma), tstrerror(terrno));
|
||||
smaWarn("vgId:%d, get tsma data failed since %s", SMA_VID(pSma), tstrerror(terrno));
|
||||
}
|
||||
return code;
|
||||
}
|
||||
|
||||
int32_t smaGetTSmaDays(SVnodeCfg* pCfg, void* pCont, uint32_t contLen, int32_t *days) {
|
||||
int32_t smaGetTSmaDays(SVnodeCfg* pCfg, void* pCont, uint32_t contLen, int32_t* days) {
|
||||
int32_t code = TSDB_CODE_SUCCESS;
|
||||
if ((code = tdGetTSmaDaysImpl(pCfg, pCont, contLen, days)) < 0) {
|
||||
smaWarn("vgId:%d get tSma days failed since %s", pCfg->vgId, tstrerror(terrno));
|
||||
smaWarn("vgId:%d, get tsma days failed since %s", pCfg->vgId, tstrerror(terrno));
|
||||
}
|
||||
smaDebug("vgId:%d, get tsma days %d", pCfg->vgId, *days);
|
||||
return code;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -97,12 +97,16 @@ int32_t tdGetTSmaDaysImpl(SVnodeCfg *pCfg, void *pCont, uint32_t contLen, int32_
|
|||
goto _err;
|
||||
}
|
||||
STsdbCfg *pTsdbCfg = &pCfg->tsdbCfg;
|
||||
int64_t mInterval = convertTimeFromPrecisionToUnit(tsma.interval, pTsdbCfg->precision, TIME_UNIT_MINUTE);
|
||||
int64_t records = pTsdbCfg->days / mInterval;
|
||||
|
||||
int64_t sInterval = convertTimeFromPrecisionToUnit(tsma.interval, pTsdbCfg->precision, TIME_UNIT_SECOND);
|
||||
if (sInterval <= 0) {
|
||||
*days = pTsdbCfg->days;
|
||||
return 0;
|
||||
}
|
||||
int64_t records = pTsdbCfg->days * 60 / sInterval;
|
||||
if (records >= SMA_STORAGE_SPLIT_FACTOR) {
|
||||
*days = pTsdbCfg->days;
|
||||
} else {
|
||||
int64_t mInterval = convertTimeFromPrecisionToUnit(tsma.interval, pTsdbCfg->precision, TIME_UNIT_MINUTE);
|
||||
int64_t daysPerFile = mInterval * SMA_STORAGE_MINUTES_DAY * 2;
|
||||
|
||||
if (daysPerFile > SMA_STORAGE_MINUTES_MAX) {
|
||||
|
|
@ -111,7 +115,7 @@ int32_t tdGetTSmaDaysImpl(SVnodeCfg *pCfg, void *pCont, uint32_t contLen, int32_
|
|||
*days = (int32_t)daysPerFile;
|
||||
}
|
||||
|
||||
if(*days < pTsdbCfg->days) {
|
||||
if (*days < pTsdbCfg->days) {
|
||||
*days = pTsdbCfg->days;
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -216,8 +216,8 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg, int32_t workerId) {
|
|||
}
|
||||
|
||||
// TODO wrap in destroy func
|
||||
taosArrayDestroy(rsp.blockData);
|
||||
taosArrayDestroy(rsp.blockDataLen);
|
||||
taosArrayDestroyP(rsp.blockData, (FDelete)taosMemoryFree);
|
||||
|
||||
if (rsp.withSchema) {
|
||||
taosArrayDestroyP(rsp.blockSchema, (FDelete)tDeleteSSchemaWrapper);
|
||||
|
|
@ -349,8 +349,9 @@ int32_t tqProcessTaskDeploy(STQ* pTq, char* msg, int32_t msgLen) {
|
|||
}
|
||||
|
||||
// sink
|
||||
pTask->ahandle = pTq->pVnode;
|
||||
/*pTask->ahandle = pTq->pVnode;*/
|
||||
if (pTask->sinkType == TASK_SINK__SMA) {
|
||||
pTask->smaSink.vnode = pTq->pVnode;
|
||||
pTask->smaSink.smaSink = smaHandleRes;
|
||||
} else if (pTask->sinkType == TASK_SINK__TABLE) {
|
||||
pTask->tbSink.vnode = pTq->pVnode;
|
||||
|
|
@ -421,10 +422,20 @@ int32_t tqProcessTaskRunReq(STQ* pTq, SRpcMsg* pMsg) {
|
|||
}
|
||||
|
||||
int32_t tqProcessTaskDispatchReq(STQ* pTq, SRpcMsg* pMsg) {
|
||||
SStreamDispatchReq* pReq = pMsg->pCont;
|
||||
int32_t taskId = pReq->taskId;
|
||||
SStreamTask* pTask = taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t));
|
||||
streamProcessDispatchReq(pTask, &pTq->pVnode->msgCb, pReq, pMsg);
|
||||
char* msgStr = pMsg->pCont;
|
||||
char* msgBody = POINTER_SHIFT(msgStr, sizeof(SMsgHead));
|
||||
int32_t msgLen = pMsg->contLen - sizeof(SMsgHead);
|
||||
SStreamDispatchReq req;
|
||||
SDecoder decoder;
|
||||
tDecoderInit(&decoder, msgBody, msgLen);
|
||||
tDecodeStreamDispatchReq(&decoder, &req);
|
||||
int32_t taskId = req.taskId;
|
||||
SStreamTask* pTask = taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t));
|
||||
SRpcMsg rsp = {
|
||||
.info = pMsg->info,
|
||||
.code = 0,
|
||||
};
|
||||
streamProcessDispatchReq(pTask, &pTq->pVnode->msgCb, &req, &rsp);
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -437,7 +448,7 @@ int32_t tqProcessTaskRecoverReq(STQ* pTq, SRpcMsg* pMsg) {
|
|||
}
|
||||
|
||||
int32_t tqProcessTaskDispatchRsp(STQ* pTq, SRpcMsg* pMsg) {
|
||||
SStreamDispatchRsp* pRsp = pMsg->pCont;
|
||||
SStreamDispatchRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
|
||||
int32_t taskId = pRsp->taskId;
|
||||
SStreamTask* pTask = taosHashGet(pTq->pStreamTasks, &taskId, sizeof(int32_t));
|
||||
streamProcessDispatchRsp(pTask, &pTq->pVnode->msgCb, pRsp);
|
||||
|
|
|
|||
|
|
@ -39,6 +39,9 @@ static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataBlkRsp* pRs
|
|||
|
||||
static int32_t tqAddBlockSchemaToRsp(const STqExecHandle* pExec, int32_t workerId, SMqDataBlkRsp* pRsp) {
|
||||
SSchemaWrapper* pSW = tCloneSSchemaWrapper(pExec->pExecReader[workerId]->pSchemaWrapper);
|
||||
if (pSW == NULL) {
|
||||
return -1;
|
||||
}
|
||||
taosArrayPush(pRsp->blockSchema, &pSW);
|
||||
return 0;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -141,10 +141,10 @@ int32_t tqRetrieveDataBlock(SArray** ppCols, STqReadHandle* pHandle, uint64_t* p
|
|||
}
|
||||
|
||||
// this interface use suid instead of uid
|
||||
pHandle->pSchemaWrapper = metaGetTableSchema(pHandle->pVnodeMeta, pHandle->msgIter.suid, sversion, true);
|
||||
pHandle->pSchemaWrapper = metaGetTableSchema(pHandle->pVnodeMeta, pHandle->msgIter.uid, sversion, true);
|
||||
if (pHandle->pSchemaWrapper == NULL) {
|
||||
tqWarn("cannot found schema wrapper for table: suid: %ld, version %d, possibly dropped table",
|
||||
pHandle->msgIter.suid, pHandle->cachedSchemaVer);
|
||||
pHandle->msgIter.uid, pHandle->cachedSchemaVer);
|
||||
/*ASSERT(0);*/
|
||||
terrno = TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND;
|
||||
return -1;
|
||||
|
|
|
|||
|
|
@ -18,8 +18,8 @@
|
|||
#define TSDB_MAX_SUBBLOCKS 8
|
||||
|
||||
typedef struct {
|
||||
STable *pTable;
|
||||
SSkipListIterator *pIter;
|
||||
STable *pTable;
|
||||
STbDataIter *pIter;
|
||||
} SCommitIter;
|
||||
|
||||
typedef struct {
|
||||
|
|
@ -28,6 +28,8 @@ typedef struct {
|
|||
int niters; // memory iterators
|
||||
SCommitIter *iters;
|
||||
bool isRFileSet; // read and commit FSET
|
||||
int32_t fid;
|
||||
SDFileSet *pSet;
|
||||
SReadH readh;
|
||||
SDFileSet wSet;
|
||||
bool isDFileSame;
|
||||
|
|
@ -58,26 +60,29 @@ typedef struct {
|
|||
#define TSDB_COMMIT_DEFAULT_ROWS(ch) TSDB_DEFAULT_BLOCK_ROWS(TSDB_COMMIT_REPO(ch)->pVnode->config.tsdbCfg.maxRows)
|
||||
#define TSDB_COMMIT_TXN_VERSION(ch) FS_TXN_VERSION(REPO_FS(TSDB_COMMIT_REPO(ch)))
|
||||
|
||||
static void tsdbStartCommit(STsdb *pRepo);
|
||||
static void tsdbEndCommit(STsdb *pTsdb, int eno);
|
||||
static int tsdbInitCommitH(SCommitH *pCommith, STsdb *pRepo);
|
||||
static void tsdbSeekCommitIter(SCommitH *pCommith, TSKEY key);
|
||||
static int tsdbNextCommitFid(SCommitH *pCommith);
|
||||
static void tsdbDestroyCommitH(SCommitH *pCommith);
|
||||
static int tsdbCreateCommitIters(SCommitH *pCommith);
|
||||
static void tsdbDestroyCommitIters(SCommitH *pCommith);
|
||||
static int tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid);
|
||||
static void tsdbResetCommitFile(SCommitH *pCommith);
|
||||
static int tsdbSetAndOpenCommitFile(SCommitH *pCommith, SDFileSet *pSet, int fid);
|
||||
static int tsdbCommitToTable(SCommitH *pCommith, int tid);
|
||||
static bool tsdbCommitIsSameFile(SCommitH *pCommith, int bidx);
|
||||
static int tsdbMoveBlkIdx(SCommitH *pCommith, SBlockIdx *pIdx);
|
||||
static int tsdbSetCommitTable(SCommitH *pCommith, STable *pTable);
|
||||
static int tsdbComparKeyBlock(const void *arg1, const void *arg2);
|
||||
static int tsdbWriteBlockInfo(SCommitH *pCommih);
|
||||
static int tsdbCommitMemData(SCommitH *pCommith, SCommitIter *pIter, TSKEY keyLimit, bool toData);
|
||||
static int tsdbMergeMemData(SCommitH *pCommith, SCommitIter *pIter, int bidx);
|
||||
static int tsdbMoveBlock(SCommitH *pCommith, int bidx);
|
||||
static int32_t tsdbCommitData(SCommitH *pCommith);
|
||||
static int32_t tsdbCommitDel(SCommitH *pCommith);
|
||||
static int32_t tsdbCommitCache(SCommitH *pCommith);
|
||||
static int32_t tsdbStartCommit(STsdb *pTsdb, SCommitH *pCHandle);
|
||||
static int32_t tsdbEndCommit(SCommitH *pCHandle, int eno);
|
||||
|
||||
static int tsdbInitCommitH(SCommitH *pCommith, STsdb *pRepo);
|
||||
static void tsdbSeekCommitIter(SCommitH *pCommith, TSKEY key);
|
||||
static int tsdbNextCommitFid(SCommitH *pCommith);
|
||||
static void tsdbDestroyCommitH(SCommitH *pCommith);
|
||||
static int32_t tsdbCreateCommitIters(SCommitH *pCommith);
|
||||
static void tsdbDestroyCommitIters(SCommitH *pCommith);
|
||||
static int tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid);
|
||||
static int tsdbSetAndOpenCommitFile(SCommitH *pCommith, SDFileSet *pSet, int fid);
|
||||
static int tsdbCommitToTable(SCommitH *pCommith, int tid);
|
||||
static bool tsdbCommitIsSameFile(SCommitH *pCommith, int bidx);
|
||||
static int tsdbMoveBlkIdx(SCommitH *pCommith, SBlockIdx *pIdx);
|
||||
static int tsdbSetCommitTable(SCommitH *pCommith, STable *pTable);
|
||||
static int tsdbComparKeyBlock(const void *arg1, const void *arg2);
|
||||
static int tsdbWriteBlockInfo(SCommitH *pCommih);
|
||||
static int tsdbCommitMemData(SCommitH *pCommith, SCommitIter *pIter, TSKEY keyLimit, bool toData);
|
||||
static int tsdbMergeMemData(SCommitH *pCommith, SCommitIter *pIter, int bidx);
|
||||
static int tsdbMoveBlock(SCommitH *pCommith, int bidx);
|
||||
static int tsdbCommitAddBlock(SCommitH *pCommith, const SBlock *pSupBlock, const SBlock *pSubBlocks, int nSubBlocks);
|
||||
static int tsdbMergeBlockData(SCommitH *pCommith, SCommitIter *pIter, SDataCols *pDataCols, TSKEY keyLimit,
|
||||
bool isLastOneBlock);
|
||||
|
|
@ -88,11 +93,14 @@ static void tsdbLoadAndMergeFromCache(STsdb *pTsdb, SDataCols *pDataCols, int *i
|
|||
SDataCols *pTarget, TSKEY maxKey, int maxRows, int8_t update);
|
||||
static int tsdbWriteBlockIdx(SDFile *pHeadf, SArray *pIdxA, void **ppBuf);
|
||||
static int tsdbApplyRtnOnFSet(STsdb *pRepo, SDFileSet *pSet, SRtn *pRtn);
|
||||
static int tsdbLoadDataFromCache(STsdb *pTsdb, STable *pTable, STbDataIter *pIter, TSKEY maxKey, int maxRowsToRead,
|
||||
SDataCols *pCols, TKEY *filterKeys, int nFilterKeys, bool keepDup,
|
||||
SMergeInfo *pMergeInfo);
|
||||
|
||||
int tsdbBegin(STsdb *pTsdb) {
|
||||
int32_t tsdbBegin(STsdb *pTsdb) {
|
||||
if (!pTsdb) return 0;
|
||||
|
||||
STsdbMemTable *pMem;
|
||||
SMemTable *pMem;
|
||||
|
||||
if (tsdbMemTableCreate(pTsdb, &pTsdb->mem) < 0) {
|
||||
return -1;
|
||||
|
|
@ -112,15 +120,50 @@ int32_t tsdbCommit(STsdb *pTsdb) {
|
|||
pTsdb->mem = NULL;
|
||||
|
||||
// start commit
|
||||
tsdbStartCommit(pTsdb);
|
||||
if (tsdbInitCommitH(&commith, pTsdb) < 0) {
|
||||
return -1;
|
||||
code = tsdbStartCommit(pTsdb, &commith);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// commit impl
|
||||
code = tsdbCommitData(&commith);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
code = tsdbCommitDel(&commith);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
code = tsdbCommitCache(&commith);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// end commit
|
||||
code = tsdbEndCommit(&commith, 0);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
tsdbError("vgId:%d failed to commit since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitData(SCommitH *pCommith) {
|
||||
int32_t fid;
|
||||
SDFileSet *pSet = NULL;
|
||||
int32_t code = 0;
|
||||
STsdb *pTsdb = TSDB_COMMIT_REPO(pCommith);
|
||||
|
||||
// Skip expired memory data and expired FSET
|
||||
tsdbSeekCommitIter(&commith, commith.rtn.minKey);
|
||||
while ((pSet = tsdbFSIterNext(&(commith.fsIter)))) {
|
||||
if (pSet->fid < commith.rtn.minFid) {
|
||||
tsdbSeekCommitIter(pCommith, pCommith->rtn.minKey);
|
||||
while ((pSet = tsdbFSIterNext(&(pCommith->fsIter)))) {
|
||||
if (pSet->fid < pCommith->rtn.minFid) {
|
||||
tsdbInfo("vgId:%d, FSET %d on level %d disk id %d expires, remove it", REPO_ID(pTsdb), pSet->fid,
|
||||
TSDB_FSET_LEVEL(pSet), TSDB_FSET_ID(pSet));
|
||||
} else {
|
||||
|
|
@ -129,7 +172,7 @@ int32_t tsdbCommit(STsdb *pTsdb) {
|
|||
}
|
||||
|
||||
// commit
|
||||
fid = tsdbNextCommitFid(&(commith));
|
||||
fid = tsdbNextCommitFid(pCommith);
|
||||
while (true) {
|
||||
// Loop over both on disk and memory
|
||||
if (pSet == NULL && fid == TSDB_IVLD_FID) break;
|
||||
|
|
@ -137,12 +180,12 @@ int32_t tsdbCommit(STsdb *pTsdb) {
|
|||
if (pSet && (fid == TSDB_IVLD_FID || pSet->fid < fid)) {
|
||||
// Only has existing FSET but no memory data to commit in this
|
||||
// existing FSET, only check if file in correct retention
|
||||
if (tsdbApplyRtnOnFSet(pTsdb, pSet, &(commith.rtn)) < 0) {
|
||||
tsdbDestroyCommitH(&commith);
|
||||
if (tsdbApplyRtnOnFSet(TSDB_COMMIT_REPO(pCommith), pSet, &(pCommith->rtn)) < 0) {
|
||||
tsdbDestroyCommitH(pCommith);
|
||||
return -1;
|
||||
}
|
||||
|
||||
pSet = tsdbFSIterNext(&(commith.fsIter));
|
||||
pSet = tsdbFSIterNext(&(pCommith->fsIter));
|
||||
} else {
|
||||
// Has memory data to commit
|
||||
SDFileSet *pCSet;
|
||||
|
|
@ -156,22 +199,30 @@ int32_t tsdbCommit(STsdb *pTsdb) {
|
|||
// Commit to an existing FSET
|
||||
pCSet = pSet;
|
||||
cfid = pSet->fid;
|
||||
pSet = tsdbFSIterNext(&(commith.fsIter));
|
||||
pSet = tsdbFSIterNext(&(pCommith->fsIter));
|
||||
}
|
||||
|
||||
if (tsdbCommitToFile(&commith, pCSet, cfid) < 0) {
|
||||
tsdbDestroyCommitH(&commith);
|
||||
if (tsdbCommitToFile(pCommith, pCSet, cfid) < 0) {
|
||||
tsdbDestroyCommitH(pCommith);
|
||||
return -1;
|
||||
}
|
||||
|
||||
fid = tsdbNextCommitFid(&commith);
|
||||
fid = tsdbNextCommitFid(pCommith);
|
||||
}
|
||||
}
|
||||
|
||||
// end commit
|
||||
tsdbDestroyCommitH(&commith);
|
||||
tsdbEndCommit(pTsdb, TSDB_CODE_SUCCESS);
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitDel(SCommitH *pCommith) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitCache(SCommitH *pCommith) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
|
|
@ -216,16 +267,6 @@ static int tsdbApplyRtnOnFSet(STsdb *pRepo, SDFileSet *pSet, SRtn *pRtn) {
|
|||
return 0;
|
||||
}
|
||||
|
||||
// int tsdbPrepareCommit(STsdb *pTsdb) {
|
||||
// if (pTsdb->mem == NULL) return 0;
|
||||
|
||||
// ASSERT(pTsdb->imem == NULL);
|
||||
|
||||
// pTsdb->imem = pTsdb->mem;
|
||||
// pTsdb->mem = NULL;
|
||||
// return 0;
|
||||
// }
|
||||
|
||||
void tsdbGetRtnSnap(STsdb *pRepo, SRtn *pRtn) {
|
||||
STsdbKeepCfg *pCfg = REPO_KEEP_CFG(pRepo);
|
||||
TSKEY minKey, midKey, maxKey, now;
|
||||
|
|
@ -243,19 +284,32 @@ void tsdbGetRtnSnap(STsdb *pRepo, SRtn *pRtn) {
|
|||
pRtn->minFid, pRtn->midFid, pRtn->maxFid);
|
||||
}
|
||||
|
||||
static void tsdbStartCommit(STsdb *pRepo) {
|
||||
STsdbMemTable *pMem = pRepo->imem;
|
||||
static int32_t tsdbStartCommit(STsdb *pTsdb, SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
|
||||
tsdbInfo("vgId:%d, start to commit", REPO_ID(pRepo));
|
||||
tsdbInfo("vgId:%d, start to commit", REPO_ID(pTsdb));
|
||||
|
||||
tsdbStartFSTxn(pRepo, 0, 0);
|
||||
if (tsdbInitCommitH(pCHandle, pTsdb) < 0) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
tsdbStartFSTxn(pTsdb, 0, 0);
|
||||
|
||||
return code;
|
||||
}
|
||||
|
||||
static void tsdbEndCommit(STsdb *pTsdb, int eno) {
|
||||
static int32_t tsdbEndCommit(SCommitH *pCHandle, int eno) {
|
||||
int32_t code = 0;
|
||||
STsdb *pTsdb = TSDB_COMMIT_REPO(pCHandle);
|
||||
|
||||
tsdbDestroyCommitH(pCHandle);
|
||||
tsdbEndFSTxn(pTsdb);
|
||||
tsdbMemTableDestroy(pTsdb->imem);
|
||||
pTsdb->imem = NULL;
|
||||
|
||||
tsdbInfo("vgId:%d, commit over, %s", REPO_ID(pTsdb), (eno == TSDB_CODE_SUCCESS) ? "succeed" : "failed");
|
||||
|
||||
return code;
|
||||
}
|
||||
|
||||
static int tsdbInitCommitH(SCommitH *pCommith, STsdb *pRepo) {
|
||||
|
|
@ -354,34 +408,73 @@ static void tsdbDestroyCommitH(SCommitH *pCommith) {
|
|||
tsdbCloseDFileSet(TSDB_COMMIT_WRITE_FSET(pCommith));
|
||||
}
|
||||
|
||||
static int tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCommith);
|
||||
static int32_t tsdbCommitToFileStart(SCommitH *pCHandle, SDFileSet *pSet, int32_t fid) {
|
||||
int32_t code = 0;
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCHandle);
|
||||
STsdbKeepCfg *pCfg = REPO_KEEP_CFG(pRepo);
|
||||
|
||||
ASSERT(pSet == NULL || pSet->fid == fid);
|
||||
|
||||
tsdbResetCommitFile(pCommith);
|
||||
tsdbGetFidKeyRange(pCfg->days, pCfg->precision, fid, &(pCommith->minKey), &(pCommith->maxKey));
|
||||
pCHandle->fid = fid;
|
||||
pCHandle->pSet = pSet;
|
||||
pCHandle->isRFileSet = false;
|
||||
pCHandle->isDFileSame = false;
|
||||
pCHandle->isLFileSame = false;
|
||||
taosArrayClear(pCHandle->aBlkIdx);
|
||||
|
||||
// Set and open files
|
||||
if (tsdbSetAndOpenCommitFile(pCommith, pSet, fid) < 0) {
|
||||
tsdbGetFidKeyRange(pCfg->days, pCfg->precision, fid, &(pCHandle->minKey), &(pCHandle->maxKey));
|
||||
|
||||
code = tsdbSetAndOpenCommitFile(pCHandle, pSet, fid);
|
||||
|
||||
return code;
|
||||
}
|
||||
static int32_t tsdbCommitToFileImpl(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
static int32_t tsdbCommitToFileEnd(SCommitH *pCommith) {
|
||||
int32_t code = 0;
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCommith);
|
||||
|
||||
if (tsdbWriteBlockIdx(TSDB_COMMIT_HEAD_FILE(pCommith), pCommith->aBlkIdx, (void **)(&(TSDB_COMMIT_BUF(pCommith)))) <
|
||||
0) {
|
||||
tsdbError("vgId:%d, failed to write SBlockIdx part to FSET %d since %s", REPO_ID(pRepo), pCommith->fid,
|
||||
tstrerror(terrno));
|
||||
tsdbCloseCommitFile(pCommith, true);
|
||||
// revert the file change
|
||||
tsdbApplyDFileSetChange(TSDB_COMMIT_WRITE_FSET(pCommith), pCommith->pSet);
|
||||
return -1;
|
||||
}
|
||||
#if 0
|
||||
// Loop to commit each table data
|
||||
for (int tid = 0; tid < pCommith->niters; tid++) {
|
||||
SCommitIter *pIter = pCommith->iters + tid;
|
||||
|
||||
if (pIter->pTable == NULL) continue;
|
||||
|
||||
if (tsdbCommitToTable(pCommith, tid) < 0) {
|
||||
tsdbCloseCommitFile(pCommith, true);
|
||||
// revert the file change
|
||||
tsdbApplyDFileSetChange(TSDB_COMMIT_WRITE_FSET(pCommith), pSet);
|
||||
return -1;
|
||||
}
|
||||
if (tsdbUpdateDFileSetHeader(&(pCommith->wSet)) < 0) {
|
||||
tsdbError("vgId:%d, failed to update FSET %d header since %s", REPO_ID(pRepo), pCommith->fid, tstrerror(terrno));
|
||||
tsdbCloseCommitFile(pCommith, true);
|
||||
// revert the file change
|
||||
tsdbApplyDFileSetChange(TSDB_COMMIT_WRITE_FSET(pCommith), pCommith->pSet);
|
||||
return -1;
|
||||
}
|
||||
#endif
|
||||
|
||||
// Close commit file
|
||||
tsdbCloseCommitFile(pCommith, false);
|
||||
|
||||
if (tsdbUpdateDFileSet(REPO_FS(pRepo), &(pCommith->wSet)) < 0) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
return code;
|
||||
}
|
||||
static int32_t tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
|
||||
int32_t code = 0;
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCommith);
|
||||
STsdbKeepCfg *pCfg = REPO_KEEP_CFG(pRepo);
|
||||
|
||||
// commit to file start
|
||||
code = tsdbCommitToFileStart(pCommith, pSet, fid);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// Loop to commit each table data in mem and file
|
||||
int mIter = 0, fIter = 0;
|
||||
int nBlkIdx = taosArrayGetSize(pCommith->readh.aBlkIdx);
|
||||
|
|
@ -400,7 +493,7 @@ static int tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
|
|||
break;
|
||||
}
|
||||
|
||||
if (pIter && pIter->pTable && (!pIdx || (pIter->pTable->uid <= pIdx->uid))) {
|
||||
if (pIter && pIter->pTable && (!pIdx || (pIter->pTable->suid <= pIdx->suid || pIter->pTable->uid <= pIdx->uid))) {
|
||||
if (tsdbCommitToTable(pCommith, mIter) < 0) {
|
||||
tsdbCloseCommitFile(pCommith, true);
|
||||
// revert the file change
|
||||
|
|
@ -426,84 +519,60 @@ static int tsdbCommitToFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
|
|||
}
|
||||
}
|
||||
|
||||
if (tsdbWriteBlockIdx(TSDB_COMMIT_HEAD_FILE(pCommith), pCommith->aBlkIdx, (void **)(&(TSDB_COMMIT_BUF(pCommith)))) <
|
||||
0) {
|
||||
tsdbError("vgId:%d, failed to write SBlockIdx part to FSET %d since %s", REPO_ID(pRepo), fid, tstrerror(terrno));
|
||||
tsdbCloseCommitFile(pCommith, true);
|
||||
// revert the file change
|
||||
tsdbApplyDFileSetChange(TSDB_COMMIT_WRITE_FSET(pCommith), pSet);
|
||||
return -1;
|
||||
// commit to file end
|
||||
code = tsdbCommitToFileEnd(pCommith);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
if (tsdbUpdateDFileSetHeader(&(pCommith->wSet)) < 0) {
|
||||
tsdbError("vgId:%d, failed to update FSET %d header since %s", REPO_ID(pRepo), fid, tstrerror(terrno));
|
||||
tsdbCloseCommitFile(pCommith, true);
|
||||
// revert the file change
|
||||
tsdbApplyDFileSetChange(TSDB_COMMIT_WRITE_FSET(pCommith), pSet);
|
||||
return -1;
|
||||
}
|
||||
return code;
|
||||
|
||||
// Close commit file
|
||||
tsdbCloseCommitFile(pCommith, false);
|
||||
|
||||
if (tsdbUpdateDFileSet(REPO_FS(pRepo), &(pCommith->wSet)) < 0) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int tsdbCreateCommitIters(SCommitH *pCommith) {
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCommith);
|
||||
STsdbMemTable *pMem = pRepo->imem;
|
||||
SSkipListIterator *pSlIter;
|
||||
SCommitIter *pCommitIter;
|
||||
SSkipListNode *pNode;
|
||||
STbData *pTbData;
|
||||
STSchema *pTSchema = NULL;
|
||||
static int32_t tsdbCreateCommitIters(SCommitH *pCommith) {
|
||||
int32_t code = 0;
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCommith);
|
||||
SMemTable *pMem = pRepo->imem;
|
||||
STbData *pTbData;
|
||||
SCommitIter *pCommitIter;
|
||||
STSchema *pTSchema = NULL;
|
||||
|
||||
pCommith->niters = SL_SIZE(pMem->pSlIdx);
|
||||
pCommith->niters = taosArrayGetSize(pMem->aTbData);
|
||||
pCommith->iters = (SCommitIter *)taosMemoryCalloc(pCommith->niters, sizeof(SCommitIter));
|
||||
if (pCommith->iters == NULL) {
|
||||
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
|
||||
return -1;
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// Loop to create iters for each skiplist
|
||||
pSlIter = tSkipListCreateIter(pMem->pSlIdx);
|
||||
if (pSlIter == NULL) {
|
||||
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
|
||||
return -1;
|
||||
}
|
||||
for (int i = 0; i < pCommith->niters; i++) {
|
||||
tSkipListIterNext(pSlIter);
|
||||
pNode = tSkipListIterGet(pSlIter);
|
||||
pTbData = (STbData *)pNode->pData;
|
||||
for (int32_t iIter = 0; iIter < pCommith->niters; iIter++) {
|
||||
pTbData = (STbData *)taosArrayGetP(pMem->aTbData, iIter);
|
||||
pCommitIter = &pCommith->iters[iIter];
|
||||
|
||||
pCommitIter = pCommith->iters + i;
|
||||
pTSchema = metaGetTbTSchema(REPO_META(pRepo), pTbData->uid, -1);
|
||||
|
||||
if (pTSchema) {
|
||||
pCommitIter->pIter = tSkipListCreateIter(pTbData->pData);
|
||||
tSkipListIterNext(pCommitIter->pIter);
|
||||
tsdbTbDataIterCreate(pTbData, NULL, 0, &pCommitIter->pIter);
|
||||
|
||||
pCommitIter->pTable = (STable *)taosMemoryMalloc(sizeof(STable));
|
||||
pCommitIter->pTable->uid = pTbData->uid;
|
||||
pCommitIter->pTable->tid = pTbData->uid;
|
||||
pCommitIter->pTable->suid = pTbData->suid;
|
||||
pCommitIter->pTable->pSchema = pTSchema;
|
||||
pCommitIter->pTable->pCacheSchema = NULL;
|
||||
}
|
||||
}
|
||||
tSkipListDestroyIter(pSlIter);
|
||||
|
||||
return 0;
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static void tsdbDestroyCommitIters(SCommitH *pCommith) {
|
||||
if (pCommith->iters == NULL) return;
|
||||
|
||||
for (int i = 1; i < pCommith->niters; i++) {
|
||||
tSkipListDestroyIter(pCommith->iters[i].pIter);
|
||||
tsdbTbDataIterDestroy(pCommith->iters[i].pIter);
|
||||
if (pCommith->iters[i].pTable) {
|
||||
tdFreeSchema(pCommith->iters[i].pTable->pSchema);
|
||||
tdFreeSchema(pCommith->iters[i].pTable->pCacheSchema);
|
||||
|
|
@ -516,13 +585,6 @@ static void tsdbDestroyCommitIters(SCommitH *pCommith) {
|
|||
pCommith->niters = 0;
|
||||
}
|
||||
|
||||
static void tsdbResetCommitFile(SCommitH *pCommith) {
|
||||
pCommith->isRFileSet = false;
|
||||
pCommith->isDFileSame = false;
|
||||
pCommith->isLFileSame = false;
|
||||
taosArrayClear(pCommith->aBlkIdx);
|
||||
}
|
||||
|
||||
static int tsdbSetAndOpenCommitFile(SCommitH *pCommith, SDFileSet *pSet, int fid) {
|
||||
SDiskID did;
|
||||
STsdb *pRepo = TSDB_COMMIT_REPO(pCommith);
|
||||
|
|
@ -743,8 +805,8 @@ static int tsdbWriteBlockInfoImpl(SDFile *pHeadf, STable *pTable, SArray *pSupA,
|
|||
pBlkInfo = *ppBuf;
|
||||
|
||||
pBlkInfo->delimiter = TSDB_FILE_DELIMITER;
|
||||
pBlkInfo->tid = TABLE_TID(pTable);
|
||||
pBlkInfo->uid = TABLE_UID(pTable);
|
||||
pBlkInfo->suid = pTable->suid;
|
||||
pBlkInfo->uid = pTable->uid;
|
||||
|
||||
memcpy((void *)(pBlkInfo->blocks), taosArrayGet(pSupA, 0), nSupBlocks * sizeof(SBlock));
|
||||
if (nSubBlocks > 0) {
|
||||
|
|
@ -770,7 +832,8 @@ static int tsdbWriteBlockInfoImpl(SDFile *pHeadf, STable *pTable, SArray *pSupA,
|
|||
// Set pIdx
|
||||
pBlock = taosArrayGetLast(pSupA);
|
||||
|
||||
pIdx->uid = TABLE_UID(pTable);
|
||||
pIdx->suid = pTable->suid;
|
||||
pIdx->uid = pTable->uid;
|
||||
pIdx->hasLast = pBlock->last ? 1 : 0;
|
||||
pIdx->maxKey = pBlock->maxKey;
|
||||
pIdx->numOfBlocks = (uint32_t)nSupBlocks;
|
||||
|
|
@ -925,7 +988,7 @@ static int tsdbMoveBlkIdx(SCommitH *pCommith, SBlockIdx *pIdx) {
|
|||
return -1;
|
||||
}
|
||||
|
||||
STable table = {.tid = pIdx->uid, .uid = pIdx->uid, .pSchema = NULL};
|
||||
STable table = {.suid = pIdx->suid, .uid = pIdx->uid, .pSchema = NULL};
|
||||
pCommith->pTable = &table;
|
||||
|
||||
while (bidx < nBlocks) {
|
||||
|
|
@ -1186,7 +1249,7 @@ static int tsdbWriteBlockImpl(STsdb *pRepo, STable *pTable, SDFile *pDFile, SDFi
|
|||
}
|
||||
|
||||
pBlockData->delimiter = TSDB_FILE_DELIMITER;
|
||||
pBlockData->uid = TABLE_UID(pTable);
|
||||
pBlockData->uid = pTable->uid;
|
||||
pBlockData->numOfCols = nColsNotAllNull;
|
||||
|
||||
taosCalcChecksumAppend(0, (uint8_t *)pBlockData, tsize);
|
||||
|
|
@ -1226,7 +1289,7 @@ static int tsdbWriteBlockImpl(STsdb *pRepo, STable *pTable, SDFile *pDFile, SDFi
|
|||
|
||||
tsdbDebug("vgId:%d, uid:%" PRId64 " a block of data is written to file %s, offset %" PRId64
|
||||
" numOfRows %d len %d numOfCols %" PRId16 " keyFirst %" PRId64 " keyLast %" PRId64,
|
||||
REPO_ID(pRepo), TABLE_UID(pTable), TSDB_FILE_FULL_NAME(pDFile), offset, rowsToWrite, pBlock->len,
|
||||
REPO_ID(pRepo), pTable->uid, TSDB_FILE_FULL_NAME(pDFile), offset, rowsToWrite, pBlock->len,
|
||||
pBlock->numOfCols, pBlock->minKey.ts, pBlock->maxKey.ts);
|
||||
|
||||
return 0;
|
||||
|
|
@ -1313,7 +1376,7 @@ static int tsdbMergeMemData(SCommitH *pCommith, SCommitIter *pIter, int bidx) {
|
|||
keyLimit = pBlock[1].minKey.ts - 1;
|
||||
}
|
||||
|
||||
SSkipListIterator titer = *(pIter->pIter);
|
||||
STbDataIter titer = *(pIter->pIter);
|
||||
if (tsdbLoadBlockDataCols(&(pCommith->readh), pBlock, NULL, &colId, 1, false) < 0) return -1;
|
||||
|
||||
tsdbLoadDataFromCache(TSDB_COMMIT_REPO(pCommith), pIter->pTable, &titer, keyLimit, INT32_MAX, NULL,
|
||||
|
|
@ -1522,7 +1585,7 @@ static void tsdbLoadAndMergeFromCache(STsdb *pTsdb, SDataCols *pDataCols, int *i
|
|||
lastKey = key2;
|
||||
}
|
||||
|
||||
tSkipListIterNext(pCommitIter->pIter);
|
||||
tsdbTbDataIterNext(pCommitIter->pIter);
|
||||
} else {
|
||||
if (lastKey != key1) {
|
||||
if (lastKey != TSKEY_INITIAL_VAL) {
|
||||
|
|
@ -1554,7 +1617,7 @@ static void tsdbLoadAndMergeFromCache(STsdb *pTsdb, SDataCols *pDataCols, int *i
|
|||
tdAppendSTSRowToDataCol(row, pSchema, pTarget, true);
|
||||
}
|
||||
++(*iter);
|
||||
tSkipListIterNext(pCommitIter->pIter);
|
||||
tsdbTbDataIterNext(pCommitIter->pIter);
|
||||
}
|
||||
|
||||
if (pTarget->numOfRows >= (maxRows - 1)) break;
|
||||
|
|
@ -1598,4 +1661,171 @@ static bool tsdbCanAddSubBlock(SCommitH *pCommith, SBlock *pBlock, SMergeInfo *p
|
|||
}
|
||||
|
||||
return false;
|
||||
}
|
||||
|
||||
static int tsdbAppendTableRowToCols(STsdb *pTsdb, STable *pTable, SDataCols *pCols, STSchema **ppSchema, STSRow *row,
|
||||
bool merge) {
|
||||
if (pCols) {
|
||||
if (*ppSchema == NULL || schemaVersion(*ppSchema) != TD_ROW_SVER(row)) {
|
||||
*ppSchema = tsdbGetTableSchemaImpl(pTsdb, pTable, false, false, TD_ROW_SVER(row));
|
||||
if (*ppSchema == NULL) {
|
||||
ASSERT(false);
|
||||
return -1;
|
||||
}
|
||||
}
|
||||
|
||||
tdAppendSTSRowToDataCol(row, *ppSchema, pCols, merge);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int tsdbLoadDataFromCache(STsdb *pTsdb, STable *pTable, STbDataIter *pIter, TSKEY maxKey, int maxRowsToRead,
|
||||
SDataCols *pCols, TKEY *filterKeys, int nFilterKeys, bool keepDup,
|
||||
SMergeInfo *pMergeInfo) {
|
||||
ASSERT(maxRowsToRead > 0 && nFilterKeys >= 0);
|
||||
if (pIter == NULL) return 0;
|
||||
STSchema *pSchema = NULL;
|
||||
TSKEY rowKey = 0;
|
||||
TSKEY fKey = 0;
|
||||
// only fetch lastKey from mem data as file data not used in this function actually
|
||||
TSKEY lastKey = TSKEY_INITIAL_VAL;
|
||||
bool isRowDel = false;
|
||||
int filterIter = 0;
|
||||
STSRow *row = NULL;
|
||||
SMergeInfo mInfo;
|
||||
|
||||
// TODO: support Multi-Version(the rows with the same TS keys in memory can't be merged if its version refered by
|
||||
// query handle)
|
||||
|
||||
if (pMergeInfo == NULL) pMergeInfo = &mInfo;
|
||||
|
||||
memset(pMergeInfo, 0, sizeof(*pMergeInfo));
|
||||
pMergeInfo->keyFirst = INT64_MAX;
|
||||
pMergeInfo->keyLast = INT64_MIN;
|
||||
if (pCols) tdResetDataCols(pCols);
|
||||
|
||||
row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL || TD_ROW_KEY(row) > maxKey) {
|
||||
rowKey = INT64_MAX;
|
||||
isRowDel = false;
|
||||
} else {
|
||||
rowKey = TD_ROW_KEY(row);
|
||||
isRowDel = TD_ROW_IS_DELETED(row);
|
||||
}
|
||||
|
||||
if (filterIter >= nFilterKeys) {
|
||||
fKey = INT64_MAX;
|
||||
} else {
|
||||
fKey = tdGetKey(filterKeys[filterIter]);
|
||||
}
|
||||
// 1. fkey - no dup since merged up to maxVersion of each query handle by tsdbLoadBlockDataCols
|
||||
// 2. rowKey - would dup since Multi-Version supported
|
||||
while (true) {
|
||||
if (fKey == INT64_MAX && rowKey == INT64_MAX) break;
|
||||
|
||||
if (fKey < rowKey) {
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, fKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, fKey);
|
||||
|
||||
filterIter++;
|
||||
if (filterIter >= nFilterKeys) {
|
||||
fKey = INT64_MAX;
|
||||
} else {
|
||||
fKey = tdGetKey(filterKeys[filterIter]);
|
||||
}
|
||||
#if 1
|
||||
} else if (fKey > rowKey) {
|
||||
if (isRowDel) {
|
||||
// TODO: support delete function
|
||||
pMergeInfo->rowsDeleteFailed++;
|
||||
} else {
|
||||
if (pMergeInfo->rowsInserted - pMergeInfo->rowsDeleteSucceed >= maxRowsToRead) break;
|
||||
if (pCols && pMergeInfo->nOperations >= pCols->maxPoints) break;
|
||||
|
||||
if (lastKey != rowKey) {
|
||||
pMergeInfo->rowsInserted++;
|
||||
pMergeInfo->nOperations++;
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, rowKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, rowKey);
|
||||
if (pCols) {
|
||||
if (lastKey != TSKEY_INITIAL_VAL) {
|
||||
++pCols->numOfRows;
|
||||
}
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, false);
|
||||
}
|
||||
lastKey = rowKey;
|
||||
} else {
|
||||
if (keepDup) {
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, true);
|
||||
} else {
|
||||
// discard
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tsdbTbDataIterNext(pIter);
|
||||
row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL || TD_ROW_KEY(row) > maxKey) {
|
||||
rowKey = INT64_MAX;
|
||||
isRowDel = false;
|
||||
} else {
|
||||
rowKey = TD_ROW_KEY(row);
|
||||
isRowDel = TD_ROW_IS_DELETED(row);
|
||||
}
|
||||
} else { // fkey == rowKey
|
||||
if (isRowDel) { // TODO: support delete function(How to stands for delete in file? rowVersion = -1?)
|
||||
ASSERT(!keepDup);
|
||||
if (pCols && pMergeInfo->nOperations >= pCols->maxPoints) break;
|
||||
pMergeInfo->rowsDeleteSucceed++;
|
||||
pMergeInfo->nOperations++;
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, false);
|
||||
} else {
|
||||
if (keepDup) {
|
||||
if (pCols && pMergeInfo->nOperations >= pCols->maxPoints) break;
|
||||
if (lastKey != rowKey) {
|
||||
pMergeInfo->rowsUpdated++;
|
||||
pMergeInfo->nOperations++;
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, rowKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, rowKey);
|
||||
if (pCols) {
|
||||
if (lastKey != TSKEY_INITIAL_VAL) {
|
||||
++pCols->numOfRows;
|
||||
}
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, false);
|
||||
}
|
||||
lastKey = rowKey;
|
||||
} else {
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, true);
|
||||
}
|
||||
} else {
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, fKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, fKey);
|
||||
}
|
||||
}
|
||||
|
||||
tsdbTbDataIterNext(pIter);
|
||||
row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL || TD_ROW_KEY(row) > maxKey) {
|
||||
rowKey = INT64_MAX;
|
||||
isRowDel = false;
|
||||
} else {
|
||||
rowKey = TD_ROW_KEY(row);
|
||||
isRowDel = TD_ROW_IS_DELETED(row);
|
||||
}
|
||||
|
||||
filterIter++;
|
||||
if (filterIter >= nFilterKeys) {
|
||||
fKey = INT64_MAX;
|
||||
} else {
|
||||
fKey = tdGetKey(filterKeys[filterIter]);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
if (pCols && (lastKey != TSKEY_INITIAL_VAL)) {
|
||||
++pCols->numOfRows;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
|
@ -1,436 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
|
||||
*
|
||||
* This program is free software: you can use, redistribute, and/or modify
|
||||
* it under the terms of the GNU Affero General Public License, version 3
|
||||
* or later ("AGPL"), as published by the Free Software Foundation.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful, but WITHOUT
|
||||
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
||||
* FITNESS FOR A PARTICULAR PURPOSE.
|
||||
*
|
||||
* You should have received a copy of the GNU Affero General Public License
|
||||
* along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
#include "tsdb.h"
|
||||
|
||||
typedef struct {
|
||||
SMemTable *pMemTable;
|
||||
int32_t minutes;
|
||||
int8_t precision;
|
||||
TSKEY nCommitKey;
|
||||
int32_t fid;
|
||||
TSKEY minKey;
|
||||
TSKEY maxKey;
|
||||
SReadH readh;
|
||||
SDFileSet wSet;
|
||||
SArray *aBlkIdx;
|
||||
SArray *aSupBlk;
|
||||
SArray *aSubBlk;
|
||||
SArray *aDelInfo;
|
||||
} SCommitH;
|
||||
|
||||
static int32_t tsdbCommitStart(SCommitH *pCHandle, STsdb *pTsdb);
|
||||
static int32_t tsdbCommitEnd(SCommitH *pCHandle);
|
||||
static int32_t tsdbCommitImpl(SCommitH *pCHandle);
|
||||
|
||||
int32_t tsdbBegin2(STsdb *pTsdb) {
|
||||
int32_t code = 0;
|
||||
|
||||
ASSERT(pTsdb->mem == NULL);
|
||||
code = tsdbMemTableCreate2(pTsdb, (SMemTable **)&pTsdb->mem);
|
||||
if (code) {
|
||||
tsdbError("vgId:%d failed to begin TSDB since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
|
||||
goto _exit;
|
||||
}
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
}
|
||||
|
||||
int32_t tsdbCommit2(STsdb *pTsdb) {
|
||||
int32_t code = 0;
|
||||
SCommitH ch = {0};
|
||||
|
||||
// start to commit
|
||||
code = tsdbCommitStart(&ch, pTsdb);
|
||||
if (code) {
|
||||
goto _exit;
|
||||
}
|
||||
|
||||
// commit
|
||||
code = tsdbCommitImpl(&ch);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// end commit
|
||||
code = tsdbCommitEnd(&ch);
|
||||
if (code) {
|
||||
goto _exit;
|
||||
}
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
|
||||
_err:
|
||||
tsdbError("vgId:%d failed to commit since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitStart(SCommitH *pCHandle, STsdb *pTsdb) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = (SMemTable *)pTsdb->mem;
|
||||
|
||||
tsdbInfo("vgId:%d start to commit", TD_VID(pTsdb->pVnode));
|
||||
|
||||
// switch to commit
|
||||
ASSERT(pTsdb->imem == NULL && pTsdb->mem);
|
||||
pTsdb->imem = pTsdb->mem;
|
||||
pTsdb->mem = NULL;
|
||||
|
||||
// open handle
|
||||
pCHandle->pMemTable = pMemTable;
|
||||
pCHandle->minutes = pTsdb->keepCfg.days;
|
||||
pCHandle->precision = pTsdb->keepCfg.precision;
|
||||
pCHandle->nCommitKey = pMemTable->minKey.ts;
|
||||
|
||||
code = tsdbInitReadH(&pCHandle->readh, pTsdb);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
pCHandle->aBlkIdx = taosArrayInit(0, sizeof(SBlockIdx));
|
||||
if (pCHandle->aBlkIdx == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pCHandle->aSupBlk = taosArrayInit(0, sizeof(SBlock));
|
||||
if (pCHandle->aSupBlk == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pCHandle->aSubBlk = taosArrayInit(0, sizeof(SBlock));
|
||||
if (pCHandle->aSubBlk == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pCHandle->aDelInfo = taosArrayInit(0, sizeof(SDelInfo));
|
||||
if (pCHandle->aDelInfo == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// start FS transaction
|
||||
tsdbStartFSTxn(pTsdb, 0, 0);
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitEnd(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
STsdb *pTsdb = pCHandle->pMemTable->pTsdb;
|
||||
SMemTable *pMemTable = (SMemTable *)pTsdb->imem;
|
||||
|
||||
// end transaction
|
||||
code = tsdbEndFSTxn(pTsdb);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// close handle
|
||||
taosArrayClear(pCHandle->aDelInfo);
|
||||
taosArrayClear(pCHandle->aSubBlk);
|
||||
taosArrayClear(pCHandle->aSupBlk);
|
||||
taosArrayClear(pCHandle->aBlkIdx);
|
||||
tsdbDestroyReadH(&pCHandle->readh);
|
||||
|
||||
// destroy memtable (todo: unref it)
|
||||
pTsdb->imem = NULL;
|
||||
tsdbMemTableDestroy2(pMemTable);
|
||||
|
||||
tsdbInfo("vgId:%d commit over", TD_VID(pTsdb->pVnode));
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitTableStart(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitTableEnd(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitTable(SCommitH *pCHandle, SMemData *pMemData, SBlockIdx *pBlockIdx) {
|
||||
int32_t code = 0;
|
||||
SMemDataIter iter = {0};
|
||||
|
||||
// commit table start
|
||||
code = tsdbCommitTableStart(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// commit table impl
|
||||
if (pMemData && pBlockIdx) {
|
||||
// TODO
|
||||
} else if (pMemData) {
|
||||
// TODO
|
||||
} else {
|
||||
// TODO
|
||||
}
|
||||
|
||||
// commit table end
|
||||
code = tsdbCommitTableEnd(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbTableIdCmprFn(const void *p1, const void *p2) {
|
||||
TABLEID *pId1 = (TABLEID *)p1;
|
||||
TABLEID *pId2 = (TABLEID *)p2;
|
||||
|
||||
if (pId1->suid < pId2->suid) {
|
||||
return -1;
|
||||
} else if (pId1->suid > pId2->suid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (pId1->uid < pId2->uid) {
|
||||
return -1;
|
||||
} else if (pId1->uid > pId2->uid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int32_t tsdbWriteBlockIdx(SDFile *pFile, SArray *pArray, uint8_t **ppBuf) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitFileStart(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
STsdb *pTsdb = pCHandle->pMemTable->pTsdb;
|
||||
SDFileSet *pSet = NULL;
|
||||
|
||||
taosArrayClear(pCHandle->aBlkIdx);
|
||||
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitFileEnd(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitFile(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
SMemData *pMemData;
|
||||
SBlockIdx *pBlockIdx;
|
||||
int32_t iMemData;
|
||||
int32_t nMemData;
|
||||
int32_t iBlockIdx;
|
||||
int32_t nBlockIdx;
|
||||
|
||||
// commit file start
|
||||
code = tsdbCommitFileStart(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// commit file impl
|
||||
iMemData = 0;
|
||||
nMemData = taosArrayGetSize(pCHandle->pMemTable->aMemData);
|
||||
iBlockIdx = 0;
|
||||
nBlockIdx = 0; // todo
|
||||
|
||||
for (;;) {
|
||||
if (iMemData >= nMemData && iBlockIdx >= nBlockIdx) break;
|
||||
|
||||
pMemData = NULL;
|
||||
pBlockIdx = NULL;
|
||||
if (iMemData < nMemData) {
|
||||
pMemData = (SMemData *)taosArrayGetP(pCHandle->pMemTable->aMemData, iMemData);
|
||||
}
|
||||
if (iBlockIdx < nBlockIdx) {
|
||||
// pBlockIdx = ;
|
||||
}
|
||||
|
||||
if (pMemData && pBlockIdx) {
|
||||
int32_t c = tsdbTableIdCmprFn(pMemData, pBlockIdx);
|
||||
if (c < 0) {
|
||||
iMemData++;
|
||||
pBlockIdx = NULL;
|
||||
} else if (c == 0) {
|
||||
iMemData++;
|
||||
iBlockIdx++;
|
||||
} else {
|
||||
iBlockIdx++;
|
||||
pMemData = NULL;
|
||||
}
|
||||
} else {
|
||||
if (pMemData) {
|
||||
iMemData++;
|
||||
} else {
|
||||
iBlockIdx++;
|
||||
}
|
||||
}
|
||||
|
||||
code = tsdbCommitTable(pCHandle, pMemData, pBlockIdx);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
}
|
||||
|
||||
// commit file end
|
||||
code = tsdbCommitFileEnd(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitData(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
int32_t fid;
|
||||
|
||||
if (pCHandle->pMemTable->nRows == 0) goto _exit;
|
||||
|
||||
// loop to commit to each file
|
||||
for (;;) {
|
||||
if (pCHandle->nCommitKey == TSKEY_MAX) break;
|
||||
|
||||
pCHandle->fid = TSDB_KEY_FID(pCHandle->nCommitKey, pCHandle->minutes, pCHandle->precision);
|
||||
tsdbGetFidKeyRange(pCHandle->minutes, pCHandle->precision, pCHandle->fid, &pCHandle->minKey, &pCHandle->maxKey);
|
||||
code = tsdbCommitFile(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
}
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t delInfoCmprFn(const void *p1, const void *p2) {
|
||||
SDelInfo *pDelInfo1 = (SDelInfo *)p1;
|
||||
SDelInfo *pDelInfo2 = (SDelInfo *)p2;
|
||||
|
||||
if (pDelInfo1->suid < pDelInfo2->suid) {
|
||||
return -1;
|
||||
} else if (pDelInfo1->suid > pDelInfo2->suid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (pDelInfo1->uid < pDelInfo2->uid) {
|
||||
return -1;
|
||||
} else if (pDelInfo1->uid > pDelInfo2->uid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (pDelInfo1->version < pDelInfo2->version) {
|
||||
return -1;
|
||||
} else if (pDelInfo1->version > pDelInfo2->version) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
static int32_t tsdbCommitDelete(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
SDelInfo delInfo;
|
||||
SMemData *pMemData;
|
||||
|
||||
if (pCHandle->pMemTable->nDelOp == 0) goto _exit;
|
||||
|
||||
// load del array (todo)
|
||||
|
||||
// loop to append SDelInfo
|
||||
for (int32_t iMemData = 0; iMemData < taosArrayGetSize(pCHandle->pMemTable->aMemData); iMemData++) {
|
||||
pMemData = (SMemData *)taosArrayGetP(pCHandle->pMemTable->aMemData, iMemData);
|
||||
|
||||
for (SDelOp *pDelOp = pMemData->delOpHead; pDelOp; pDelOp = pDelOp->pNext) {
|
||||
delInfo = (SDelInfo){.suid = pMemData->suid,
|
||||
.uid = pMemData->uid,
|
||||
.version = pDelOp->version,
|
||||
.sKey = pDelOp->sKey,
|
||||
.eKey = pDelOp->eKey};
|
||||
if (taosArrayPush(pCHandle->aDelInfo, &delInfo) == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
taosArraySort(pCHandle->aDelInfo, delInfoCmprFn);
|
||||
|
||||
// write to new file
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitCache(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
// TODO
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbCommitImpl(SCommitH *pCHandle) {
|
||||
int32_t code = 0;
|
||||
|
||||
// commit data
|
||||
code = tsdbCommitData(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// commit delete
|
||||
code = tsdbCommitDelete(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// commit cache if need (todo)
|
||||
if (0) {
|
||||
code = tsdbCommitCache(pCHandle);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
}
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
|
@ -25,8 +25,7 @@ static const char *TSDB_FNAME_SUFFIX[] = {
|
|||
"meta", // TSDB_FILE_META
|
||||
};
|
||||
|
||||
static void tsdbGetFilename(int vid, int fid, uint32_t ver, TSDB_FILE_T ftype, const char *dname, char *fname);
|
||||
// static int tsdbRollBackMFile(SMFile *pMFile);
|
||||
static void tsdbGetFilename(int vid, int fid, uint32_t ver, TSDB_FILE_T ftype, const char *dname, char *fname);
|
||||
static int tsdbEncodeDFInfo(void **buf, SDFInfo *pInfo);
|
||||
static void *tsdbDecodeDFInfo(void *buf, SDFInfo *pInfo);
|
||||
static int tsdbRollBackDFile(SDFile *pDFile);
|
||||
|
|
|
|||
|
|
@ -15,248 +15,102 @@
|
|||
|
||||
#include "tsdb.h"
|
||||
|
||||
static STbData *tsdbNewTbData(tb_uid_t uid);
|
||||
static void tsdbFreeTbData(STbData *pTbData);
|
||||
static char *tsdbGetTsTupleKey(const void *data);
|
||||
static int tsdbTbDataComp(const void *arg1, const void *arg2);
|
||||
static char *tsdbTbDataGetUid(const void *arg);
|
||||
static int tsdbAppendTableRowToCols(STsdb *pTsdb, STable *pTable, SDataCols *pCols, STSchema **ppSchema, STSRow *row,
|
||||
bool merge);
|
||||
#define SL_MAX_LEVEL 5
|
||||
|
||||
int tsdbMemTableCreate(STsdb *pTsdb, STsdbMemTable **ppMemTable) {
|
||||
STsdbMemTable *pMemTable;
|
||||
SVnode *pVnode;
|
||||
#define SL_NODE_SIZE(l) (sizeof(SMemSkipListNode) + sizeof(SMemSkipListNode *) * (l)*2)
|
||||
#define SL_NODE_FORWARD(n, l) ((n)->forwards[l])
|
||||
#define SL_NODE_BACKWARD(n, l) ((n)->forwards[(n)->level + (l)])
|
||||
#define SL_NODE_DATA(n) (&SL_NODE_BACKWARD(n, (n)->level))
|
||||
|
||||
*ppMemTable = NULL;
|
||||
pVnode = pTsdb->pVnode;
|
||||
#define SL_MOVE_BACKWARD 0x1
|
||||
#define SL_MOVE_FROM_POS 0x2
|
||||
|
||||
// alloc handle
|
||||
pMemTable = (STsdbMemTable *)taosMemoryCalloc(1, sizeof(*pMemTable));
|
||||
static int32_t tPutTSDBRow(uint8_t *p, TSDBROW *pRow);
|
||||
static int32_t tGetTSDBRow(uint8_t *p, TSDBROW *pRow);
|
||||
static void tbDataMovePosTo(STbData *pTbData, SMemSkipListNode **pos, TSDBKEY *pKey, int32_t flags);
|
||||
static int32_t tsdbGetOrCreateTbData(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, STbData **ppTbData);
|
||||
static int32_t tsdbInsertTableDataImpl(SMemTable *pMemTable, STbData *pTbData, int64_t version,
|
||||
SSubmitMsgIter *pMsgIter, SSubmitBlk *pBlock, SSubmitBlkRsp *pRsp);
|
||||
|
||||
int32_t tsdbMemTableCreate(STsdb *pTsdb, SMemTable **ppMemTable) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = NULL;
|
||||
|
||||
pMemTable = (SMemTable *)taosMemoryCalloc(1, sizeof(*pMemTable));
|
||||
if (pMemTable == NULL) {
|
||||
return -1;
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
pMemTable->pPool = pTsdb->pVnode->inUse;
|
||||
T_REF_INIT_VAL(pMemTable, 1);
|
||||
taosInitRWLatch(&pMemTable->latch);
|
||||
pMemTable->keyMin = TSKEY_MAX;
|
||||
pMemTable->keyMax = TSKEY_MIN;
|
||||
pMemTable->pTsdb = pTsdb;
|
||||
pMemTable->nRef = 1;
|
||||
pMemTable->minKey = (TSDBKEY){.ts = TSKEY_MAX, .version = INT64_MAX};
|
||||
pMemTable->maxKey = (TSDBKEY){.ts = TSKEY_MIN, .version = -1};
|
||||
pMemTable->nRow = 0;
|
||||
pMemTable->pSlIdx = tSkipListCreate(pVnode->config.tsdbCfg.slLevel, TSDB_DATA_TYPE_BIGINT, sizeof(tb_uid_t),
|
||||
tsdbTbDataComp, SL_DISCARD_DUP_KEY, tsdbTbDataGetUid);
|
||||
if (pMemTable->pSlIdx == NULL) {
|
||||
pMemTable->nDelOp = 0;
|
||||
pMemTable->aTbData = taosArrayInit(128, sizeof(STbData *));
|
||||
if (pMemTable->aTbData == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
taosMemoryFree(pMemTable);
|
||||
return -1;
|
||||
}
|
||||
|
||||
pMemTable->pHashIdx = taosHashInit(1024, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK);
|
||||
if (pMemTable->pHashIdx == NULL) {
|
||||
tSkipListDestroy(pMemTable->pSlIdx);
|
||||
taosMemoryFree(pMemTable);
|
||||
return -1;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
*ppMemTable = pMemTable;
|
||||
return 0;
|
||||
return code;
|
||||
|
||||
_err:
|
||||
*ppMemTable = NULL;
|
||||
return code;
|
||||
}
|
||||
|
||||
void tsdbMemTableDestroy(STsdbMemTable *pMemTable) {
|
||||
void tsdbMemTableDestroy(SMemTable *pMemTable) {
|
||||
if (pMemTable) {
|
||||
taosHashCleanup(pMemTable->pHashIdx);
|
||||
SSkipListIterator *pIter = tSkipListCreateIter(pMemTable->pSlIdx);
|
||||
SSkipListNode *pNode = NULL;
|
||||
STbData *pTbData = NULL;
|
||||
for (;;) {
|
||||
if (!tSkipListIterNext(pIter)) break;
|
||||
pNode = tSkipListIterGet(pIter);
|
||||
pTbData = (STbData *)pNode->pData;
|
||||
tsdbFreeTbData(pTbData);
|
||||
}
|
||||
tSkipListDestroyIter(pIter);
|
||||
tSkipListDestroy(pMemTable->pSlIdx);
|
||||
taosArrayDestroy(pMemTable->aTbData);
|
||||
taosMemoryFree(pMemTable);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* This is an important function to load data or try to load data from memory skiplist iterator.
|
||||
*
|
||||
* This function load memory data until:
|
||||
* 1. iterator ends
|
||||
* 2. data key exceeds maxKey
|
||||
* 3. rowsIncreased = rowsInserted - rowsDeleteSucceed >= maxRowsToRead
|
||||
* 4. operations in pCols not exceeds its max capacity if pCols is given
|
||||
*
|
||||
* The function tries to procceed AS MUCH AS POSSIBLE.
|
||||
*/
|
||||
int tsdbLoadDataFromCache(STsdb *pTsdb, STable *pTable, SSkipListIterator *pIter, TSKEY maxKey, int maxRowsToRead,
|
||||
SDataCols *pCols, TKEY *filterKeys, int nFilterKeys, bool keepDup, SMergeInfo *pMergeInfo) {
|
||||
ASSERT(maxRowsToRead > 0 && nFilterKeys >= 0);
|
||||
if (pIter == NULL) return 0;
|
||||
STSchema *pSchema = NULL;
|
||||
TSKEY rowKey = 0;
|
||||
TSKEY fKey = 0;
|
||||
// only fetch lastKey from mem data as file data not used in this function actually
|
||||
TSKEY lastKey = TSKEY_INITIAL_VAL;
|
||||
bool isRowDel = false;
|
||||
int filterIter = 0;
|
||||
STSRow *row = NULL;
|
||||
SMergeInfo mInfo;
|
||||
static int32_t tbDataPCmprFn(const void *p1, const void *p2) {
|
||||
STbData *pTbData1 = *(STbData **)p1;
|
||||
STbData *pTbData2 = *(STbData **)p2;
|
||||
|
||||
// TODO: support Multi-Version(the rows with the same TS keys in memory can't be merged if its version refered by
|
||||
// query handle)
|
||||
|
||||
if (pMergeInfo == NULL) pMergeInfo = &mInfo;
|
||||
|
||||
memset(pMergeInfo, 0, sizeof(*pMergeInfo));
|
||||
pMergeInfo->keyFirst = INT64_MAX;
|
||||
pMergeInfo->keyLast = INT64_MIN;
|
||||
if (pCols) tdResetDataCols(pCols);
|
||||
|
||||
row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL || TD_ROW_KEY(row) > maxKey) {
|
||||
rowKey = INT64_MAX;
|
||||
isRowDel = false;
|
||||
} else {
|
||||
rowKey = TD_ROW_KEY(row);
|
||||
isRowDel = TD_ROW_IS_DELETED(row);
|
||||
if (pTbData1->suid < pTbData2->suid) {
|
||||
return -1;
|
||||
} else if (pTbData1->suid > pTbData2->suid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (filterIter >= nFilterKeys) {
|
||||
fKey = INT64_MAX;
|
||||
} else {
|
||||
fKey = tdGetKey(filterKeys[filterIter]);
|
||||
}
|
||||
// 1. fkey - no dup since merged up to maxVersion of each query handle by tsdbLoadBlockDataCols
|
||||
// 2. rowKey - would dup since Multi-Version supported
|
||||
while (true) {
|
||||
if (fKey == INT64_MAX && rowKey == INT64_MAX) break;
|
||||
|
||||
if (fKey < rowKey) {
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, fKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, fKey);
|
||||
|
||||
filterIter++;
|
||||
if (filterIter >= nFilterKeys) {
|
||||
fKey = INT64_MAX;
|
||||
} else {
|
||||
fKey = tdGetKey(filterKeys[filterIter]);
|
||||
}
|
||||
#if 1
|
||||
} else if (fKey > rowKey) {
|
||||
if (isRowDel) {
|
||||
// TODO: support delete function
|
||||
pMergeInfo->rowsDeleteFailed++;
|
||||
} else {
|
||||
if (pMergeInfo->rowsInserted - pMergeInfo->rowsDeleteSucceed >= maxRowsToRead) break;
|
||||
if (pCols && pMergeInfo->nOperations >= pCols->maxPoints) break;
|
||||
|
||||
if (lastKey != rowKey) {
|
||||
pMergeInfo->rowsInserted++;
|
||||
pMergeInfo->nOperations++;
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, rowKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, rowKey);
|
||||
if (pCols) {
|
||||
if (lastKey != TSKEY_INITIAL_VAL) {
|
||||
++pCols->numOfRows;
|
||||
}
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, false);
|
||||
}
|
||||
lastKey = rowKey;
|
||||
} else {
|
||||
if (keepDup) {
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, true);
|
||||
} else {
|
||||
// discard
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
tSkipListIterNext(pIter);
|
||||
row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL || TD_ROW_KEY(row) > maxKey) {
|
||||
rowKey = INT64_MAX;
|
||||
isRowDel = false;
|
||||
} else {
|
||||
rowKey = TD_ROW_KEY(row);
|
||||
isRowDel = TD_ROW_IS_DELETED(row);
|
||||
}
|
||||
} else { // fkey == rowKey
|
||||
if (isRowDel) { // TODO: support delete function(How to stands for delete in file? rowVersion = -1?)
|
||||
ASSERT(!keepDup);
|
||||
if (pCols && pMergeInfo->nOperations >= pCols->maxPoints) break;
|
||||
pMergeInfo->rowsDeleteSucceed++;
|
||||
pMergeInfo->nOperations++;
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, false);
|
||||
} else {
|
||||
if (keepDup) {
|
||||
if (pCols && pMergeInfo->nOperations >= pCols->maxPoints) break;
|
||||
if (lastKey != rowKey) {
|
||||
pMergeInfo->rowsUpdated++;
|
||||
pMergeInfo->nOperations++;
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, rowKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, rowKey);
|
||||
if (pCols) {
|
||||
if (lastKey != TSKEY_INITIAL_VAL) {
|
||||
++pCols->numOfRows;
|
||||
}
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, false);
|
||||
}
|
||||
lastKey = rowKey;
|
||||
} else {
|
||||
tsdbAppendTableRowToCols(pTsdb, pTable, pCols, &pSchema, row, true);
|
||||
}
|
||||
} else {
|
||||
pMergeInfo->keyFirst = TMIN(pMergeInfo->keyFirst, fKey);
|
||||
pMergeInfo->keyLast = TMAX(pMergeInfo->keyLast, fKey);
|
||||
}
|
||||
}
|
||||
|
||||
tSkipListIterNext(pIter);
|
||||
row = tsdbNextIterRow(pIter);
|
||||
if (row == NULL || TD_ROW_KEY(row) > maxKey) {
|
||||
rowKey = INT64_MAX;
|
||||
isRowDel = false;
|
||||
} else {
|
||||
rowKey = TD_ROW_KEY(row);
|
||||
isRowDel = TD_ROW_IS_DELETED(row);
|
||||
}
|
||||
|
||||
filterIter++;
|
||||
if (filterIter >= nFilterKeys) {
|
||||
fKey = INT64_MAX;
|
||||
} else {
|
||||
fKey = tdGetKey(filterKeys[filterIter]);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
if (pCols && (lastKey != TSKEY_INITIAL_VAL)) {
|
||||
++pCols->numOfRows;
|
||||
if (pTbData1->uid < pTbData2->uid) {
|
||||
return -1;
|
||||
} else if (pTbData1->uid > pTbData2->uid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
void tsdbGetTbDataFromMemTable(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, STbData **ppTbData) {
|
||||
STbData *pTbData = &(STbData){.suid = suid, .uid = uid};
|
||||
void *p = taosArraySearch(pMemTable->aTbData, &pTbData, tbDataPCmprFn, TD_EQ);
|
||||
*ppTbData = p ? *(STbData **)p : NULL;
|
||||
}
|
||||
|
||||
int tsdbInsertTableData(STsdb *pTsdb, SSubmitMsgIter *pMsgIter, SSubmitBlk *pBlock, SSubmitBlkRsp *pRsp) {
|
||||
SSubmitBlkIter blkIter = {0};
|
||||
STsdbMemTable *pMemTable = pTsdb->mem;
|
||||
void *tptr;
|
||||
STbData *pTbData;
|
||||
STSRow *row;
|
||||
TSKEY keyMin;
|
||||
TSKEY keyMax;
|
||||
SSubmitBlk *pBlkCopy;
|
||||
int64_t sverNew;
|
||||
int32_t tsdbInsertTableData(STsdb *pTsdb, int64_t version, SSubmitMsgIter *pMsgIter, SSubmitBlk *pBlock,
|
||||
SSubmitBlkRsp *pRsp) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = pTsdb->mem;
|
||||
STbData *pTbData = NULL;
|
||||
tb_uid_t suid = pMsgIter->suid;
|
||||
tb_uid_t uid = pMsgIter->uid;
|
||||
int32_t sverNew;
|
||||
|
||||
// check if table exists
|
||||
// check if table exists (todo: refact)
|
||||
SMetaReader mr = {0};
|
||||
SMetaEntry me = {0};
|
||||
metaReaderInit(&mr, pTsdb->pVnode->pMeta, 0);
|
||||
if (metaGetTableEntryByUid(&mr, pMsgIter->uid) < 0) {
|
||||
metaReaderClear(&mr);
|
||||
terrno = TSDB_CODE_PAR_TABLE_NOT_EXIST;
|
||||
return -1;
|
||||
code = TSDB_CODE_PAR_TABLE_NOT_EXIST;
|
||||
goto _err;
|
||||
}
|
||||
if (pRsp->tblFName) strcat(pRsp->tblFName, mr.me.name);
|
||||
|
||||
|
|
@ -267,122 +121,453 @@ int tsdbInsertTableData(STsdb *pTsdb, SSubmitMsgIter *pMsgIter, SSubmitBlk *pBlo
|
|||
sverNew = mr.me.stbEntry.schemaRow.version;
|
||||
}
|
||||
metaReaderClear(&mr);
|
||||
|
||||
// create container is nedd
|
||||
tptr = taosHashGet(pMemTable->pHashIdx, &(pMsgIter->uid), sizeof(pMsgIter->uid));
|
||||
if (tptr == NULL) {
|
||||
pTbData = tsdbNewTbData(pMsgIter->uid);
|
||||
if (pTbData == NULL) {
|
||||
return -1;
|
||||
}
|
||||
|
||||
// Put into hash
|
||||
taosHashPut(pMemTable->pHashIdx, &(pMsgIter->uid), sizeof(pMsgIter->uid), &(pTbData), sizeof(pTbData));
|
||||
|
||||
// Put into skiplist
|
||||
tSkipListPut(pMemTable->pSlIdx, pTbData);
|
||||
} else {
|
||||
pTbData = *(STbData **)tptr;
|
||||
}
|
||||
|
||||
// copy data to buffer pool
|
||||
int32_t tlen = pMsgIter->dataLen + pMsgIter->schemaLen + sizeof(*pBlock);
|
||||
pBlkCopy = (SSubmitBlk *)vnodeBufPoolMalloc(pTsdb->mem->pPool, tlen);
|
||||
memcpy(pBlkCopy, pBlock, tlen);
|
||||
|
||||
tInitSubmitBlkIter(pMsgIter, pBlkCopy, &blkIter);
|
||||
if (blkIter.row == NULL) return 0;
|
||||
keyMin = TD_ROW_KEY(blkIter.row);
|
||||
|
||||
tSkipListPutBatchByIter(pTbData->pData, &blkIter, (iter_next_fn_t)tGetSubmitBlkNext);
|
||||
|
||||
#ifdef TD_DEBUG_PRINT_ROW
|
||||
printf("!!! %s:%d vgId:%d dir:%s table:%" PRIi64 " has %d rows in skiplist\n\n", __func__, __LINE__,
|
||||
TD_VID(pTsdb->pVnode), pTsdb->dir, pTbData->uid, SL_SIZE(pTbData->pData));
|
||||
#endif
|
||||
|
||||
// Set statistics
|
||||
keyMax = TD_ROW_KEY(blkIter.row);
|
||||
|
||||
pTbData->nrows += pMsgIter->numOfRows;
|
||||
if (pTbData->keyMin > keyMin) pTbData->keyMin = keyMin;
|
||||
if (pTbData->keyMax < keyMax) pTbData->keyMax = keyMax;
|
||||
|
||||
pMemTable->nRow += pMsgIter->numOfRows;
|
||||
if (pMemTable->keyMin > keyMin) pMemTable->keyMin = keyMin;
|
||||
if (pMemTable->keyMax < keyMax) pMemTable->keyMax = keyMax;
|
||||
|
||||
pRsp->numOfRows = pMsgIter->numOfRows;
|
||||
pRsp->affectedRows = pMsgIter->numOfRows;
|
||||
pRsp->sver = sverNew;
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static STbData *tsdbNewTbData(tb_uid_t uid) {
|
||||
STbData *pTbData = (STbData *)taosMemoryCalloc(1, sizeof(*pTbData));
|
||||
if (pTbData == NULL) {
|
||||
return NULL;
|
||||
// create/get STbData to op
|
||||
code = tsdbGetOrCreateTbData(pMemTable, suid, uid, &pTbData);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
pTbData->uid = uid;
|
||||
pTbData->keyMin = TSKEY_MAX;
|
||||
pTbData->keyMax = TSKEY_MIN;
|
||||
pTbData->nrows = 0;
|
||||
#if 0
|
||||
pTbData->pData = tSkipListCreate(5, TSDB_DATA_TYPE_TIMESTAMP, sizeof(int64_t), tkeyComparFn, SL_DISCARD_DUP_KEY,
|
||||
tsdbGetTsTupleKey);
|
||||
#endif
|
||||
pTbData->pData =
|
||||
tSkipListCreate(5, TSDB_DATA_TYPE_TIMESTAMP, sizeof(int64_t), tkeyComparFn, SL_ALLOW_DUP_KEY, tsdbGetTsTupleKey);
|
||||
if (pTbData->pData == NULL) {
|
||||
taosMemoryFree(pTbData);
|
||||
return NULL;
|
||||
// do insert impl
|
||||
code = tsdbInsertTableDataImpl(pMemTable, pTbData, version, pMsgIter, pBlock, pRsp);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
return pTbData;
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static void tsdbFreeTbData(STbData *pTbData) {
|
||||
if (pTbData) {
|
||||
tSkipListDestroy(pTbData->pData);
|
||||
taosMemoryFree(pTbData);
|
||||
int32_t tsdbDeleteTableData(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = pTsdb->mem;
|
||||
STbData *pTbData = NULL;
|
||||
SVBufPool *pPool = pTsdb->pVnode->inUse;
|
||||
|
||||
// check if table exists (todo)
|
||||
|
||||
code = tsdbGetOrCreateTbData(pMemTable, suid, uid, &pTbData);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
}
|
||||
|
||||
static char *tsdbGetTsTupleKey(const void *data) { return (char *)TD_ROW_KEY_ADDR((STSRow *)data); }
|
||||
|
||||
static int tsdbTbDataComp(const void *arg1, const void *arg2) {
|
||||
STbData *pTbData1 = (STbData *)arg1;
|
||||
STbData *pTbData2 = (STbData *)arg2;
|
||||
|
||||
if (pTbData1->uid > pTbData2->uid) {
|
||||
return 1;
|
||||
} else if (pTbData1->uid == pTbData2->uid) {
|
||||
return 0;
|
||||
// do delete
|
||||
SDelOp *pDelOp = (SDelOp *)vnodeBufPoolMalloc(pPool, sizeof(*pDelOp));
|
||||
if (pDelOp == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pDelOp->version = version;
|
||||
pDelOp->sKey = sKey;
|
||||
pDelOp->eKey = eKey;
|
||||
pDelOp->pNext = NULL;
|
||||
if (pTbData->pHead == NULL) {
|
||||
ASSERT(pTbData->pTail == NULL);
|
||||
pTbData->pHead = pTbData->pTail = pDelOp;
|
||||
} else {
|
||||
return -1;
|
||||
pTbData->pTail->pNext = pDelOp;
|
||||
pTbData->pTail = pDelOp;
|
||||
}
|
||||
|
||||
// update the state of pMemTable and other (todo)
|
||||
|
||||
pMemTable->nDelOp++;
|
||||
|
||||
tsdbError("vgId:%d delete data from table suid:%" PRId64 " uid:%" PRId64 " skey:%" PRId64 " eKey:%" PRId64
|
||||
" since %s",
|
||||
TD_VID(pTsdb->pVnode), suid, uid, sKey, eKey, tstrerror(code));
|
||||
return code;
|
||||
|
||||
_err:
|
||||
tsdbError("vgId:%d failed to delete data from table suid:%" PRId64 " uid:%" PRId64 " skey:%" PRId64 " eKey:%" PRId64
|
||||
" since %s",
|
||||
TD_VID(pTsdb->pVnode), suid, uid, sKey, eKey, tstrerror(code));
|
||||
return code;
|
||||
}
|
||||
|
||||
int32_t tsdbTbDataIterCreate(STbData *pTbData, TSDBKEY *pFrom, int8_t backward, STbDataIter **ppIter) {
|
||||
int32_t code = 0;
|
||||
|
||||
(*ppIter) = (STbDataIter *)taosMemoryCalloc(1, sizeof(STbDataIter));
|
||||
if ((*ppIter) == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _exit;
|
||||
}
|
||||
|
||||
tsdbTbDataIterOpen(pTbData, pFrom, backward, *ppIter);
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
}
|
||||
|
||||
void *tsdbTbDataIterDestroy(STbDataIter *pIter) {
|
||||
if (pIter) {
|
||||
taosMemoryFree(pIter);
|
||||
}
|
||||
|
||||
return NULL;
|
||||
}
|
||||
|
||||
void tsdbTbDataIterOpen(STbData *pTbData, TSDBKEY *pFrom, int8_t backward, STbDataIter *pIter) {
|
||||
SMemSkipListNode *pos[SL_MAX_LEVEL];
|
||||
|
||||
pIter->pTbData = pTbData;
|
||||
pIter->backward = backward;
|
||||
if (pFrom == NULL) {
|
||||
// create from head or tail
|
||||
if (backward) {
|
||||
pIter->pNode = SL_NODE_BACKWARD(pTbData->sl.pTail, 0);
|
||||
} else {
|
||||
pIter->pNode = SL_NODE_FORWARD(pTbData->sl.pHead, 0);
|
||||
}
|
||||
} else {
|
||||
// create from a key
|
||||
if (backward) {
|
||||
tbDataMovePosTo(pTbData, pos, pFrom, SL_MOVE_BACKWARD);
|
||||
pIter->pNode = SL_NODE_BACKWARD(pos[0], 0);
|
||||
} else {
|
||||
tbDataMovePosTo(pTbData, pos, pFrom, 0);
|
||||
pIter->pNode = SL_NODE_FORWARD(pos[0], 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static char *tsdbTbDataGetUid(const void *arg) {
|
||||
STbData *pTbData = (STbData *)arg;
|
||||
return (char *)(&(pTbData->uid));
|
||||
}
|
||||
bool tsdbTbDataIterNext(STbDataIter *pIter) {
|
||||
SMemSkipListNode *pHead = pIter->pTbData->sl.pHead;
|
||||
SMemSkipListNode *pTail = pIter->pTbData->sl.pTail;
|
||||
|
||||
static int tsdbAppendTableRowToCols(STsdb *pTsdb, STable *pTable, SDataCols *pCols, STSchema **ppSchema, STSRow *row,
|
||||
bool merge) {
|
||||
if (pCols) {
|
||||
if (*ppSchema == NULL || schemaVersion(*ppSchema) != TD_ROW_SVER(row)) {
|
||||
*ppSchema = tsdbGetTableSchemaImpl(pTsdb, pTable, false, false, TD_ROW_SVER(row));
|
||||
if (*ppSchema == NULL) {
|
||||
ASSERT(false);
|
||||
return -1;
|
||||
}
|
||||
if (pIter->backward) {
|
||||
ASSERT(pIter->pNode != pTail);
|
||||
|
||||
if (pIter->pNode == pHead) {
|
||||
return false;
|
||||
}
|
||||
|
||||
tdAppendSTSRowToDataCol(row, *ppSchema, pCols, merge);
|
||||
pIter->pNode = SL_NODE_BACKWARD(pIter->pNode, 0);
|
||||
if (pIter->pNode == pHead) {
|
||||
return false;
|
||||
}
|
||||
} else {
|
||||
ASSERT(pIter->pNode != pHead);
|
||||
|
||||
if (pIter->pNode == pTail) {
|
||||
return false;
|
||||
}
|
||||
|
||||
pIter->pNode = SL_NODE_FORWARD(pIter->pNode, 0);
|
||||
if (pIter->pNode == pTail) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return 0;
|
||||
return true;
|
||||
}
|
||||
|
||||
bool tsdbTbDataIterGet(STbDataIter *pIter, TSDBROW *pRow) {
|
||||
SMemSkipListNode *pHead = pIter->pTbData->sl.pHead;
|
||||
SMemSkipListNode *pTail = pIter->pTbData->sl.pTail;
|
||||
TSDBROW row = {0};
|
||||
|
||||
if (pRow == NULL) {
|
||||
pRow = &row;
|
||||
}
|
||||
|
||||
if (pIter->backward) {
|
||||
ASSERT(pIter->pNode != pTail);
|
||||
|
||||
if (pIter->pNode == pHead) {
|
||||
return false;
|
||||
}
|
||||
} else {
|
||||
ASSERT(pIter->pNode != pHead);
|
||||
|
||||
if (pIter->pNode == pTail) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
tGetTSDBRow((uint8_t *)SL_NODE_DATA(pIter->pNode), pRow);
|
||||
return true;
|
||||
}
|
||||
|
||||
static int32_t tsdbGetOrCreateTbData(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, STbData **ppTbData) {
|
||||
int32_t code = 0;
|
||||
int32_t idx = 0;
|
||||
STbData *pTbData = NULL;
|
||||
STbData *pTbDataT = &(STbData){.suid = suid, .uid = uid};
|
||||
|
||||
// get
|
||||
idx = taosArraySearchIdx(pMemTable->aTbData, &pTbDataT, tbDataPCmprFn, TD_GE);
|
||||
if (idx >= 0) {
|
||||
pTbData = (STbData *)taosArrayGetP(pMemTable->aTbData, idx);
|
||||
if (tbDataPCmprFn(&pTbDataT, &pTbData) == 0) goto _exit;
|
||||
}
|
||||
|
||||
// create
|
||||
SVBufPool *pPool = pMemTable->pTsdb->pVnode->inUse;
|
||||
int8_t maxLevel = pMemTable->pTsdb->pVnode->config.tsdbCfg.slLevel;
|
||||
|
||||
pTbData = vnodeBufPoolMalloc(pPool, sizeof(*pTbData) + SL_NODE_SIZE(maxLevel) * 2);
|
||||
if (pTbData == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pTbData->suid = suid;
|
||||
pTbData->uid = uid;
|
||||
pTbData->minKey = (TSDBKEY){.ts = TSKEY_MAX, .version = INT64_MAX};
|
||||
pTbData->maxKey = (TSDBKEY){.ts = TSKEY_MIN, .version = -1};
|
||||
pTbData->pHead = NULL;
|
||||
pTbData->pTail = NULL;
|
||||
pTbData->sl.seed = taosRand();
|
||||
pTbData->sl.size = 0;
|
||||
pTbData->sl.maxLevel = maxLevel;
|
||||
pTbData->sl.level = 0;
|
||||
pTbData->sl.pHead = (SMemSkipListNode *)&pTbData[1];
|
||||
pTbData->sl.pTail = (SMemSkipListNode *)POINTER_SHIFT(pTbData->sl.pHead, SL_NODE_SIZE(maxLevel));
|
||||
pTbData->sl.pHead->level = maxLevel;
|
||||
pTbData->sl.pTail->level = maxLevel;
|
||||
for (int8_t iLevel = 0; iLevel < maxLevel; iLevel++) {
|
||||
SL_NODE_FORWARD(pTbData->sl.pHead, iLevel) = pTbData->sl.pTail;
|
||||
SL_NODE_BACKWARD(pTbData->sl.pTail, iLevel) = pTbData->sl.pHead;
|
||||
|
||||
SL_NODE_BACKWARD(pTbData->sl.pHead, iLevel) = NULL;
|
||||
SL_NODE_FORWARD(pTbData->sl.pTail, iLevel) = NULL;
|
||||
}
|
||||
|
||||
void *p;
|
||||
if (idx < 0) {
|
||||
p = taosArrayPush(pMemTable->aTbData, &pTbData);
|
||||
} else {
|
||||
p = taosArrayInsert(pMemTable->aTbData, idx, &pTbData);
|
||||
}
|
||||
if (p == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
_exit:
|
||||
*ppTbData = pTbData;
|
||||
return code;
|
||||
|
||||
_err:
|
||||
*ppTbData = NULL;
|
||||
return code;
|
||||
}
|
||||
|
||||
static void tbDataMovePosTo(STbData *pTbData, SMemSkipListNode **pos, TSDBKEY *pKey, int32_t flags) {
|
||||
SMemSkipListNode *px;
|
||||
SMemSkipListNode *pn;
|
||||
TSDBKEY *pTKey;
|
||||
int c;
|
||||
int backward = flags & SL_MOVE_BACKWARD;
|
||||
int fromPos = flags & SL_MOVE_FROM_POS;
|
||||
|
||||
if (backward) {
|
||||
px = pTbData->sl.pTail;
|
||||
|
||||
for (int8_t iLevel = pTbData->sl.maxLevel - 1; iLevel >= pTbData->sl.level; iLevel--) {
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
|
||||
if (pTbData->sl.level) {
|
||||
if (fromPos) px = pos[pTbData->sl.level - 1];
|
||||
|
||||
for (int8_t iLevel = pTbData->sl.level - 1; iLevel >= 0; iLevel--) {
|
||||
pn = SL_NODE_BACKWARD(px, iLevel);
|
||||
while (pn != pTbData->sl.pHead) {
|
||||
pTKey = (TSDBKEY *)SL_NODE_DATA(pn);
|
||||
|
||||
c = tsdbKeyCmprFn(pTKey, pKey);
|
||||
if (c <= 0) {
|
||||
break;
|
||||
} else {
|
||||
px = pn;
|
||||
pn = SL_NODE_BACKWARD(px, iLevel);
|
||||
}
|
||||
}
|
||||
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
px = pTbData->sl.pHead;
|
||||
|
||||
for (int8_t iLevel = pTbData->sl.maxLevel - 1; iLevel >= pTbData->sl.level; iLevel--) {
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
|
||||
if (pTbData->sl.level) {
|
||||
if (fromPos) px = pos[pTbData->sl.level - 1];
|
||||
|
||||
for (int8_t iLevel = pTbData->sl.level - 1; iLevel >= 0; iLevel--) {
|
||||
pn = SL_NODE_FORWARD(px, iLevel);
|
||||
while (pn != pTbData->sl.pTail) {
|
||||
pTKey = (TSDBKEY *)SL_NODE_DATA(pn);
|
||||
|
||||
c = tsdbKeyCmprFn(pTKey, pKey);
|
||||
if (c >= 0) {
|
||||
break;
|
||||
} else {
|
||||
px = pn;
|
||||
pn = SL_NODE_FORWARD(px, iLevel);
|
||||
}
|
||||
}
|
||||
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static FORCE_INLINE int8_t tsdbMemSkipListRandLevel(SMemSkipList *pSl) {
|
||||
int8_t level = 1;
|
||||
int8_t tlevel = TMIN(pSl->maxLevel, pSl->level + 1);
|
||||
const uint32_t factor = 4;
|
||||
|
||||
while ((taosRandR(&pSl->seed) % factor) == 0 && level < tlevel) {
|
||||
level++;
|
||||
}
|
||||
|
||||
return level;
|
||||
}
|
||||
static int32_t tbDataDoPut(SMemTable *pMemTable, STbData *pTbData, SMemSkipListNode **pos, TSDBROW *pRow,
|
||||
int8_t forward) {
|
||||
int32_t code = 0;
|
||||
int8_t level;
|
||||
SMemSkipListNode *pNode;
|
||||
SVBufPool *pPool = pMemTable->pTsdb->pVnode->inUse;
|
||||
|
||||
// node
|
||||
level = tsdbMemSkipListRandLevel(&pTbData->sl);
|
||||
pNode = (SMemSkipListNode *)vnodeBufPoolMalloc(pPool, SL_NODE_SIZE(level) + tPutTSDBRow(NULL, pRow));
|
||||
if (pNode == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _exit;
|
||||
}
|
||||
pNode->level = level;
|
||||
for (int8_t iLevel = 0; iLevel < level; iLevel++) {
|
||||
SL_NODE_FORWARD(pNode, iLevel) = NULL;
|
||||
SL_NODE_BACKWARD(pNode, iLevel) = NULL;
|
||||
}
|
||||
|
||||
tPutTSDBRow((uint8_t *)SL_NODE_DATA(pNode), pRow);
|
||||
|
||||
// put
|
||||
for (int8_t iLevel = 0; iLevel < pNode->level; iLevel++) {
|
||||
SMemSkipListNode *px = pos[iLevel];
|
||||
|
||||
if (forward) {
|
||||
SMemSkipListNode *pNext = SL_NODE_FORWARD(px, iLevel);
|
||||
|
||||
SL_NODE_FORWARD(pNode, iLevel) = pNext;
|
||||
SL_NODE_BACKWARD(pNode, iLevel) = px;
|
||||
|
||||
SL_NODE_BACKWARD(pNext, iLevel) = pNode;
|
||||
SL_NODE_FORWARD(px, iLevel) = pNode;
|
||||
} else {
|
||||
SMemSkipListNode *pPrev = SL_NODE_BACKWARD(px, iLevel);
|
||||
|
||||
SL_NODE_FORWARD(pNode, iLevel) = px;
|
||||
SL_NODE_BACKWARD(pNode, iLevel) = pPrev;
|
||||
|
||||
SL_NODE_FORWARD(pPrev, iLevel) = pNode;
|
||||
SL_NODE_BACKWARD(px, iLevel) = pNode;
|
||||
}
|
||||
}
|
||||
|
||||
pTbData->sl.size++;
|
||||
if (pTbData->sl.level < pNode->level) {
|
||||
pTbData->sl.level = pNode->level;
|
||||
}
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbInsertTableDataImpl(SMemTable *pMemTable, STbData *pTbData, int64_t version,
|
||||
SSubmitMsgIter *pMsgIter, SSubmitBlk *pBlock, SSubmitBlkRsp *pRsp) {
|
||||
int32_t code = 0;
|
||||
SSubmitBlkIter blkIter = {0};
|
||||
TSDBKEY key = {.version = version};
|
||||
SMemSkipListNode *pos[SL_MAX_LEVEL];
|
||||
TSDBROW row = {.version = version, .pTSRow = NULL};
|
||||
int32_t nRow = 0;
|
||||
|
||||
tInitSubmitBlkIter(pMsgIter, pBlock, &blkIter);
|
||||
|
||||
// backward put first data
|
||||
row.pTSRow = tGetSubmitBlkNext(&blkIter);
|
||||
key.ts = row.pTSRow->ts;
|
||||
nRow++;
|
||||
tbDataMovePosTo(pTbData, pos, &key, SL_MOVE_BACKWARD);
|
||||
code = tbDataDoPut(pMemTable, pTbData, pos, &row, 0);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
if (tsdbKeyCmprFn(&key, &pTbData->minKey) < 0) {
|
||||
pTbData->minKey = key;
|
||||
}
|
||||
|
||||
if (tsdbKeyCmprFn(&key, &pMemTable->minKey) < 0) {
|
||||
pMemTable->minKey = key;
|
||||
}
|
||||
|
||||
// forward put rest data
|
||||
row.pTSRow = tGetSubmitBlkNext(&blkIter);
|
||||
if (row.pTSRow) {
|
||||
for (int8_t iLevel = 0; iLevel < pTbData->sl.maxLevel; iLevel++) {
|
||||
pos[iLevel] = SL_NODE_BACKWARD(pos[iLevel], iLevel);
|
||||
}
|
||||
do {
|
||||
key.ts = row.pTSRow->ts;
|
||||
nRow++;
|
||||
tbDataMovePosTo(pTbData, pos, &key, SL_MOVE_FROM_POS);
|
||||
code = tbDataDoPut(pMemTable, pTbData, pos, &row, 1);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
row.pTSRow = tGetSubmitBlkNext(&blkIter);
|
||||
} while (row.pTSRow);
|
||||
}
|
||||
|
||||
if (tsdbKeyCmprFn(&key, &pTbData->maxKey) > 0) {
|
||||
pTbData->maxKey = key;
|
||||
}
|
||||
|
||||
if (tsdbKeyCmprFn(&key, &pMemTable->maxKey) > 0) {
|
||||
pMemTable->maxKey = key;
|
||||
}
|
||||
pMemTable->nRef++;
|
||||
pRsp->numOfRows = nRow;
|
||||
pRsp->affectedRows = nRow;
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tPutTSDBRow(uint8_t *p, TSDBROW *pRow) {
|
||||
int32_t n = 0;
|
||||
|
||||
n += tPutI64(p, pRow->version);
|
||||
if (p) memcpy(p + n, pRow->pTSRow, pRow->pTSRow->len);
|
||||
n += pRow->pTSRow->len;
|
||||
|
||||
return n;
|
||||
}
|
||||
|
||||
static int32_t tGetTSDBRow(uint8_t *p, TSDBROW *pRow) {
|
||||
int32_t n = 0;
|
||||
|
||||
n += tGetI64(p, &pRow->version);
|
||||
pRow->pTSRow = (STSRow *)(p + n);
|
||||
n += pRow->pTSRow->len;
|
||||
|
||||
return n;
|
||||
}
|
||||
|
|
@ -1,535 +0,0 @@
|
|||
/*
|
||||
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
|
||||
*
|
||||
* This program is free software: you can use, redistribute, and/or modify
|
||||
* it under the terms of the GNU Affero General Public License, version 3
|
||||
* or later ("AGPL"), as published by the Free Software Foundation.
|
||||
*
|
||||
* This program is distributed in the hope that it will be useful, but WITHOUT
|
||||
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
|
||||
* FITNESS FOR A PARTICULAR PURPOSE.
|
||||
*
|
||||
* You should have received a copy of the GNU Affero General Public License
|
||||
* along with this program. If not, see <http://www.gnu.org/licenses/>.
|
||||
*/
|
||||
|
||||
#include "tsdb.h"
|
||||
|
||||
struct SMemSkipListNode {
|
||||
int8_t level;
|
||||
SMemSkipListNode *forwards[0];
|
||||
};
|
||||
|
||||
typedef struct {
|
||||
tb_uid_t uid;
|
||||
STSchema *pTSchema;
|
||||
} SSkmInfo;
|
||||
|
||||
#define SL_MAX_LEVEL 5
|
||||
|
||||
#define SL_NODE_SIZE(l) (sizeof(SMemSkipListNode) + sizeof(SMemSkipListNode *) * (l)*2)
|
||||
#define SL_NODE_FORWARD(n, l) ((n)->forwards[l])
|
||||
#define SL_NODE_BACKWARD(n, l) ((n)->forwards[(n)->level + (l)])
|
||||
#define SL_NODE_DATA(n) (&SL_NODE_BACKWARD(n, (n)->level))
|
||||
|
||||
#define SL_MOVE_BACKWARD 0x1
|
||||
#define SL_MOVE_FROM_POS 0x2
|
||||
|
||||
static int32_t tsdbGetOrCreateMemData(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, SMemData **ppMemData);
|
||||
static int memDataPCmprFn(const void *p1, const void *p2);
|
||||
static int32_t tPutTSDBRow(uint8_t *p, TSDBROW *pRow);
|
||||
static int32_t tGetTSDBRow(uint8_t *p, TSDBROW *pRow);
|
||||
static int8_t tsdbMemSkipListRandLevel(SMemSkipList *pSl);
|
||||
static int32_t tsdbInsertTableDataImpl(SMemTable *pMemTable, SMemData *pMemData, int64_t version,
|
||||
SVSubmitBlk *pSubmitBlk);
|
||||
static void memDataMovePosTo(SMemData *pMemData, SMemSkipListNode **pos, TSDBKEY *pKey, int32_t flags);
|
||||
|
||||
// SMemTable ==============================================
|
||||
int32_t tsdbMemTableCreate2(STsdb *pTsdb, SMemTable **ppMemTable) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = NULL;
|
||||
|
||||
pMemTable = (SMemTable *)taosMemoryCalloc(1, sizeof(*pMemTable));
|
||||
if (pMemTable == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pMemTable->pTsdb = pTsdb;
|
||||
pMemTable->nRef = 1;
|
||||
pMemTable->minKey = (TSDBKEY){.version = INT64_MAX, .ts = TSKEY_MAX};
|
||||
pMemTable->maxKey = (TSDBKEY){.version = -1, .ts = TSKEY_MIN};
|
||||
pMemTable->nRows = 0;
|
||||
pMemTable->nDelOp = 0;
|
||||
pMemTable->aMemData = taosArrayInit(512, sizeof(SMemData *));
|
||||
if (pMemTable->aMemData == NULL) {
|
||||
taosMemoryFree(pMemTable);
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
*ppMemTable = pMemTable;
|
||||
return code;
|
||||
|
||||
_err:
|
||||
*ppMemTable = NULL;
|
||||
return code;
|
||||
}
|
||||
|
||||
void tsdbMemTableDestroy2(SMemTable *pMemTable) {
|
||||
taosArrayDestroyEx(pMemTable->aMemData, NULL /*TODO*/);
|
||||
taosMemoryFree(pMemTable);
|
||||
}
|
||||
|
||||
int32_t tsdbInsertTableData2(STsdb *pTsdb, int64_t version, SVSubmitBlk *pSubmitBlk) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = (SMemTable *)pTsdb->mem; // TODO
|
||||
SMemData *pMemData;
|
||||
TSDBROW row = {.version = version};
|
||||
|
||||
ASSERT(pMemTable);
|
||||
ASSERT(pSubmitBlk->nData > 0);
|
||||
|
||||
{
|
||||
// check if table exists (todo)
|
||||
}
|
||||
|
||||
code = tsdbGetOrCreateMemData(pMemTable, pSubmitBlk->suid, pSubmitBlk->uid, &pMemData);
|
||||
if (code) {
|
||||
tsdbError("vgId:%d, failed to create/get table data since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// do insert
|
||||
code = tsdbInsertTableDataImpl(pMemTable, pMemData, version, pSubmitBlk);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
return code;
|
||||
|
||||
_err:
|
||||
return code;
|
||||
}
|
||||
|
||||
int32_t tsdbDeleteTableData2(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey) {
|
||||
int32_t code = 0;
|
||||
SMemTable *pMemTable = (SMemTable *)pTsdb->mem; // TODO
|
||||
SMemData *pMemData;
|
||||
SVBufPool *pPool = pTsdb->pVnode->inUse;
|
||||
|
||||
ASSERT(pMemTable);
|
||||
|
||||
{
|
||||
// check if table exists (todo)
|
||||
}
|
||||
|
||||
code = tsdbGetOrCreateMemData(pMemTable, suid, uid, &pMemData);
|
||||
if (code) {
|
||||
goto _err;
|
||||
}
|
||||
|
||||
// do delete
|
||||
SDelOp *pDelOp = (SDelOp *)vnodeBufPoolMalloc(pPool, sizeof(*pDelOp));
|
||||
if (pDelOp == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pDelOp->version = version;
|
||||
pDelOp->sKey = sKey;
|
||||
pDelOp->eKey = eKey;
|
||||
pDelOp->pNext = NULL;
|
||||
if (pMemData->delOpHead == NULL) {
|
||||
ASSERT(pMemData->delOpTail == NULL);
|
||||
pMemData->delOpHead = pMemData->delOpTail = pDelOp;
|
||||
} else {
|
||||
pMemData->delOpTail->pNext = pDelOp;
|
||||
pMemData->delOpTail = pDelOp;
|
||||
}
|
||||
|
||||
{
|
||||
// update the state of pMemTable, pMemData, last and lastrow (todo)
|
||||
}
|
||||
|
||||
pMemTable->nDelOp++;
|
||||
|
||||
tsdbDebug("vgId:%d, delete data from table suid:%" PRId64 " uid:%" PRId64 " sKey:%" PRId64 " eKey:%" PRId64
|
||||
" since %s",
|
||||
TD_VID(pTsdb->pVnode), suid, uid, sKey, eKey, tstrerror(code));
|
||||
return code;
|
||||
|
||||
_err:
|
||||
tsdbError("vgId:%d, failed to delete data from table suid:%" PRId64 " uid:%" PRId64 " sKey:%" PRId64 " eKey:%" PRId64
|
||||
" since %s",
|
||||
TD_VID(pTsdb->pVnode), suid, uid, sKey, eKey, tstrerror(code));
|
||||
return code;
|
||||
}
|
||||
|
||||
void tsdbMemDataIterOpen(SMemData *pMemData, TSDBKEY *pKey, int8_t backward, SMemDataIter *pIter) {
|
||||
SMemSkipListNode *pos[SL_MAX_LEVEL];
|
||||
|
||||
pIter->pMemData = pMemData;
|
||||
pIter->backward = backward;
|
||||
pIter->pRow = NULL;
|
||||
if (pKey == NULL) {
|
||||
// create from head or tail
|
||||
if (backward) {
|
||||
pIter->pNode = SL_NODE_BACKWARD(pMemData->sl.pTail, 0);
|
||||
} else {
|
||||
pIter->pNode = SL_NODE_FORWARD(pMemData->sl.pHead, 0);
|
||||
}
|
||||
} else {
|
||||
// create from a key
|
||||
if (backward) {
|
||||
memDataMovePosTo(pMemData, pos, pKey, SL_MOVE_BACKWARD);
|
||||
pIter->pNode = SL_NODE_BACKWARD(pos[0], 0);
|
||||
} else {
|
||||
memDataMovePosTo(pMemData, pos, pKey, 0);
|
||||
pIter->pNode = SL_NODE_FORWARD(pos[0], 0);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
bool tsdbMemDataIterNext(SMemDataIter *pIter) {
|
||||
SMemSkipListNode *pHead = pIter->pMemData->sl.pHead;
|
||||
SMemSkipListNode *pTail = pIter->pMemData->sl.pTail;
|
||||
|
||||
pIter->pRow = NULL;
|
||||
if (pIter->backward) {
|
||||
ASSERT(pIter->pNode != pTail);
|
||||
|
||||
if (pIter->pNode == pHead) {
|
||||
return false;
|
||||
}
|
||||
|
||||
pIter->pNode = SL_NODE_BACKWARD(pIter->pNode, 0);
|
||||
if (pIter->pNode == pHead) {
|
||||
return false;
|
||||
}
|
||||
} else {
|
||||
ASSERT(pIter->pNode != pHead);
|
||||
|
||||
if (pIter->pNode == pTail) {
|
||||
return false;
|
||||
}
|
||||
|
||||
pIter->pNode = SL_NODE_FORWARD(pIter->pNode, 0);
|
||||
if (pIter->pNode == pTail) {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
|
||||
void tsdbMemDataIterGet(SMemDataIter *pIter, TSDBROW **ppRow) {
|
||||
if (pIter->pRow) {
|
||||
*ppRow = pIter->pRow;
|
||||
} else {
|
||||
SMemSkipListNode *pHead = pIter->pMemData->sl.pHead;
|
||||
SMemSkipListNode *pTail = pIter->pMemData->sl.pTail;
|
||||
|
||||
if (pIter->backward) {
|
||||
ASSERT(pIter->pNode != pTail);
|
||||
|
||||
if (pIter->pNode == pHead) {
|
||||
*ppRow = NULL;
|
||||
} else {
|
||||
tGetTSDBRow((uint8_t *)SL_NODE_DATA(pIter->pNode), &pIter->row);
|
||||
*ppRow = &pIter->row;
|
||||
}
|
||||
} else {
|
||||
ASSERT(pIter->pNode != pHead);
|
||||
|
||||
if (pIter->pNode == pTail) {
|
||||
*ppRow = NULL;
|
||||
} else {
|
||||
tGetTSDBRow((uint8_t *)SL_NODE_DATA(pIter->pNode), &pIter->row);
|
||||
*ppRow = &pIter->row;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static int32_t tsdbGetOrCreateMemData(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, SMemData **ppMemData) {
|
||||
int32_t code = 0;
|
||||
int32_t idx = 0;
|
||||
SMemData *pMemDataT = &(SMemData){.suid = suid, .uid = uid};
|
||||
SMemData *pMemData = NULL;
|
||||
SVBufPool *pPool = pMemTable->pTsdb->pVnode->inUse;
|
||||
int8_t maxLevel = pMemTable->pTsdb->pVnode->config.tsdbCfg.slLevel;
|
||||
|
||||
// get
|
||||
idx = taosArraySearchIdx(pMemTable->aMemData, &pMemDataT, memDataPCmprFn, TD_GE);
|
||||
if (idx >= 0) {
|
||||
pMemData = (SMemData *)taosArrayGet(pMemTable->aMemData, idx);
|
||||
if (memDataPCmprFn(&pMemDataT, &pMemData) == 0) goto _exit;
|
||||
}
|
||||
|
||||
// create
|
||||
pMemData = vnodeBufPoolMalloc(pPool, sizeof(*pMemData) + SL_NODE_SIZE(maxLevel) * 2);
|
||||
if (pMemData == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
pMemData->suid = suid;
|
||||
pMemData->uid = uid;
|
||||
pMemData->minKey = (TSDBKEY){.version = INT64_MAX, .ts = TSKEY_MAX};
|
||||
pMemData->maxKey = (TSDBKEY){.version = -1, .ts = TSKEY_MIN};
|
||||
pMemData->delOpHead = pMemData->delOpTail = NULL;
|
||||
pMemData->sl.seed = taosRand();
|
||||
pMemData->sl.size = 0;
|
||||
pMemData->sl.maxLevel = maxLevel;
|
||||
pMemData->sl.level = 0;
|
||||
pMemData->sl.pHead = (SMemSkipListNode *)&pMemData[1];
|
||||
pMemData->sl.pTail = (SMemSkipListNode *)POINTER_SHIFT(pMemData->sl.pHead, SL_NODE_SIZE(maxLevel));
|
||||
pMemData->sl.pHead->level = maxLevel;
|
||||
pMemData->sl.pTail->level = maxLevel;
|
||||
|
||||
for (int8_t iLevel = 0; iLevel < pMemData->sl.maxLevel; iLevel++) {
|
||||
SL_NODE_FORWARD(pMemData->sl.pHead, iLevel) = pMemData->sl.pTail;
|
||||
SL_NODE_BACKWARD(pMemData->sl.pHead, iLevel) = NULL;
|
||||
SL_NODE_BACKWARD(pMemData->sl.pTail, iLevel) = pMemData->sl.pHead;
|
||||
SL_NODE_FORWARD(pMemData->sl.pTail, iLevel) = NULL;
|
||||
}
|
||||
|
||||
if (idx < 0) idx = 0;
|
||||
if (taosArrayInsert(pMemTable->aMemData, idx, &pMemData) == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _err;
|
||||
}
|
||||
|
||||
_exit:
|
||||
*ppMemData = pMemData;
|
||||
return code;
|
||||
|
||||
_err:
|
||||
*ppMemData = NULL;
|
||||
return code;
|
||||
}
|
||||
|
||||
static int memDataPCmprFn(const void *p1, const void *p2) {
|
||||
SMemData *pMemData1 = *(SMemData **)p1;
|
||||
SMemData *pMemData2 = *(SMemData **)p2;
|
||||
|
||||
if (pMemData1->suid < pMemData2->suid) {
|
||||
return -1;
|
||||
} else if (pMemData1->suid > pMemData2->suid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
if (pMemData1->uid < pMemData2->uid) {
|
||||
return -1;
|
||||
} else if (pMemData1->uid > pMemData2->uid) {
|
||||
return 1;
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
static int32_t tPutTSDBRow(uint8_t *p, TSDBROW *pRow) {
|
||||
int32_t n = 0;
|
||||
|
||||
n += tPutI64(p ? p + n : p, pRow->version);
|
||||
n += tPutTSRow(p ? p + n : p, &pRow->tsRow);
|
||||
|
||||
return n;
|
||||
}
|
||||
|
||||
static int32_t tGetTSDBRow(uint8_t *p, TSDBROW *pRow) {
|
||||
int32_t n = 0;
|
||||
|
||||
n += tGetI64(p + n, &pRow->version);
|
||||
n += tGetTSRow(p + n, &pRow->tsRow);
|
||||
|
||||
return n;
|
||||
}
|
||||
|
||||
static FORCE_INLINE int8_t tsdbMemSkipListRandLevel(SMemSkipList *pSl) {
|
||||
int8_t level = 1;
|
||||
int8_t tlevel = TMIN(pSl->maxLevel, pSl->level + 1);
|
||||
const uint32_t factor = 4;
|
||||
|
||||
while ((taosRandR(&pSl->seed) % factor) == 0 && level < tlevel) {
|
||||
level++;
|
||||
}
|
||||
|
||||
return level;
|
||||
}
|
||||
|
||||
static void memDataMovePosTo(SMemData *pMemData, SMemSkipListNode **pos, TSDBKEY *pKey, int32_t flags) {
|
||||
SMemSkipListNode *px;
|
||||
SMemSkipListNode *pn;
|
||||
TSDBKEY *pTKey;
|
||||
int c;
|
||||
int backward = flags & SL_MOVE_BACKWARD;
|
||||
int fromPos = flags & SL_MOVE_FROM_POS;
|
||||
|
||||
if (backward) {
|
||||
px = pMemData->sl.pTail;
|
||||
|
||||
for (int8_t iLevel = pMemData->sl.maxLevel - 1; iLevel >= pMemData->sl.level; iLevel--) {
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
|
||||
if (pMemData->sl.level) {
|
||||
if (fromPos) px = pos[pMemData->sl.level - 1];
|
||||
|
||||
for (int8_t iLevel = pMemData->sl.level - 1; iLevel >= 0; iLevel--) {
|
||||
pn = SL_NODE_BACKWARD(px, iLevel);
|
||||
while (pn != pMemData->sl.pHead) {
|
||||
pTKey = (TSDBKEY *)SL_NODE_DATA(pn);
|
||||
|
||||
c = tsdbKeyCmprFn(pTKey, pKey);
|
||||
if (c <= 0) {
|
||||
break;
|
||||
} else {
|
||||
px = pn;
|
||||
pn = SL_NODE_BACKWARD(px, iLevel);
|
||||
}
|
||||
}
|
||||
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
px = pMemData->sl.pHead;
|
||||
|
||||
for (int8_t iLevel = pMemData->sl.maxLevel - 1; iLevel >= pMemData->sl.level; iLevel--) {
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
|
||||
if (pMemData->sl.level) {
|
||||
if (fromPos) px = pos[pMemData->sl.level - 1];
|
||||
|
||||
for (int8_t iLevel = pMemData->sl.level - 1; iLevel >= 0; iLevel--) {
|
||||
pn = SL_NODE_FORWARD(px, iLevel);
|
||||
while (pn != pMemData->sl.pHead) {
|
||||
pTKey = (TSDBKEY *)SL_NODE_DATA(pn);
|
||||
|
||||
c = tsdbKeyCmprFn(pTKey, pKey);
|
||||
if (c >= 0) {
|
||||
break;
|
||||
} else {
|
||||
px = pn;
|
||||
pn = SL_NODE_FORWARD(px, iLevel);
|
||||
}
|
||||
}
|
||||
|
||||
pos[iLevel] = px;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
static int32_t memDataDoPut(SMemTable *pMemTable, SMemData *pMemData, SMemSkipListNode **pos, TSDBROW *pRow,
|
||||
int8_t forward) {
|
||||
int32_t code = 0;
|
||||
int8_t level;
|
||||
SMemSkipListNode *pNode;
|
||||
SVBufPool *pPool = pMemTable->pTsdb->pVnode->inUse;
|
||||
|
||||
// node
|
||||
level = tsdbMemSkipListRandLevel(&pMemData->sl);
|
||||
pNode = (SMemSkipListNode *)vnodeBufPoolMalloc(pPool, SL_NODE_SIZE(level) + tPutTSDBRow(NULL, pRow));
|
||||
if (pNode == NULL) {
|
||||
code = TSDB_CODE_OUT_OF_MEMORY;
|
||||
goto _exit;
|
||||
}
|
||||
pNode->level = level;
|
||||
for (int8_t iLevel = 0; iLevel < level; iLevel++) {
|
||||
SL_NODE_FORWARD(pNode, iLevel) = NULL;
|
||||
SL_NODE_BACKWARD(pNode, iLevel) = NULL;
|
||||
}
|
||||
|
||||
tPutTSDBRow((uint8_t *)SL_NODE_DATA(pNode), pRow);
|
||||
|
||||
// put
|
||||
for (int8_t iLevel = 0; iLevel < pNode->level; iLevel++) {
|
||||
SMemSkipListNode *px = pos[iLevel];
|
||||
|
||||
if (forward) {
|
||||
SMemSkipListNode *pNext = SL_NODE_FORWARD(px, iLevel);
|
||||
|
||||
SL_NODE_FORWARD(pNode, iLevel) = pNext;
|
||||
SL_NODE_BACKWARD(pNode, iLevel) = px;
|
||||
|
||||
SL_NODE_BACKWARD(pNext, iLevel) = pNode;
|
||||
SL_NODE_FORWARD(px, iLevel) = pNode;
|
||||
} else {
|
||||
SMemSkipListNode *pPrev = SL_NODE_BACKWARD(px, iLevel);
|
||||
|
||||
SL_NODE_FORWARD(pNode, iLevel) = px;
|
||||
SL_NODE_BACKWARD(pNode, iLevel) = pPrev;
|
||||
|
||||
SL_NODE_FORWARD(pPrev, iLevel) = pNode;
|
||||
SL_NODE_BACKWARD(px, iLevel) = pNode;
|
||||
}
|
||||
}
|
||||
|
||||
pMemData->sl.size++;
|
||||
if (pMemData->sl.level < pNode->level) {
|
||||
pMemData->sl.level = pNode->level;
|
||||
}
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
}
|
||||
|
||||
static int32_t tsdbInsertTableDataImpl(SMemTable *pMemTable, SMemData *pMemData, int64_t version,
|
||||
SVSubmitBlk *pSubmitBlk) {
|
||||
int32_t code = 0;
|
||||
int32_t n = 0;
|
||||
uint8_t *p = pSubmitBlk->pData;
|
||||
int32_t nRow = 0;
|
||||
TSDBROW row = {.version = version};
|
||||
|
||||
SMemSkipListNode *pos[SL_MAX_LEVEL];
|
||||
|
||||
ASSERT(pSubmitBlk->nData);
|
||||
|
||||
// backward put first data
|
||||
n += tGetTSRow(p + n, &row.tsRow);
|
||||
ASSERT(n <= pSubmitBlk->nData);
|
||||
|
||||
memDataMovePosTo(pMemData, pos, &(TSDBKEY){.version = version, .ts = row.tsRow.ts}, SL_MOVE_BACKWARD);
|
||||
code = memDataDoPut(pMemTable, pMemData, pos, &row, 0);
|
||||
if (code) {
|
||||
goto _exit;
|
||||
}
|
||||
nRow++;
|
||||
|
||||
if (tsdbKeyCmprFn((TSDBKEY *)&row, &pMemData->minKey) < 0) {
|
||||
pMemData->minKey = *(TSDBKEY *)&row;
|
||||
}
|
||||
if (tsdbKeyCmprFn((TSDBKEY *)&row, &pMemTable->minKey) < 0) {
|
||||
pMemTable->minKey = *(TSDBKEY *)&row;
|
||||
}
|
||||
|
||||
// forward put rest
|
||||
for (int8_t iLevel = 0; iLevel < pMemData->sl.maxLevel; iLevel++) {
|
||||
pos[iLevel] = SL_NODE_BACKWARD(pos[iLevel], iLevel);
|
||||
}
|
||||
while (n < pSubmitBlk->nData) {
|
||||
n += tGetTSRow(p + n, &row.tsRow);
|
||||
ASSERT(n <= pSubmitBlk->nData);
|
||||
|
||||
memDataMovePosTo(pMemData, pos, &(TSDBKEY){.version = version, .ts = row.tsRow.ts}, SL_MOVE_FROM_POS);
|
||||
code = memDataDoPut(pMemTable, pMemData, pos, &row, 1);
|
||||
if (code) {
|
||||
goto _exit;
|
||||
}
|
||||
|
||||
nRow++;
|
||||
}
|
||||
|
||||
if (tsdbKeyCmprFn((TSDBKEY *)&row, &pMemData->maxKey) > 0) {
|
||||
pMemData->maxKey = *(TSDBKEY *)&row;
|
||||
}
|
||||
if (tsdbKeyCmprFn((TSDBKEY *)&row, &pMemTable->maxKey) > 0) {
|
||||
pMemTable->maxKey = *(TSDBKEY *)&row;
|
||||
}
|
||||
pMemTable->nRows += nRow;
|
||||
|
||||
_exit:
|
||||
return code;
|
||||
}
|
||||
|
|
@ -67,15 +67,16 @@ enum {
|
|||
};
|
||||
|
||||
typedef struct STableCheckInfo {
|
||||
uint64_t tableId;
|
||||
TSKEY lastKey;
|
||||
SBlockInfo* pCompInfo;
|
||||
int32_t compSize;
|
||||
int32_t numOfBlocks : 29; // number of qualified data blocks not the original blocks
|
||||
uint8_t chosen : 2; // indicate which iterator should move forward
|
||||
bool initBuf : 1; // whether to initialize the in-memory skip list iterator or not
|
||||
SSkipListIterator* iter; // mem buffer skip list iterator
|
||||
SSkipListIterator* iiter; // imem buffer skip list iterator
|
||||
uint64_t suid;
|
||||
uint64_t tableId;
|
||||
TSKEY lastKey;
|
||||
SBlockInfo* pCompInfo;
|
||||
int32_t compSize;
|
||||
int32_t numOfBlocks : 29; // number of qualified data blocks not the original blocks
|
||||
uint8_t chosen : 2; // indicate which iterator should move forward
|
||||
bool initBuf : 1; // whether to initialize the in-memory skip list iterator or not
|
||||
STbDataIter* iter; // mem buffer skip list iterator
|
||||
STbDataIter* iiter; // imem buffer skip list iterator
|
||||
} STableCheckInfo;
|
||||
|
||||
typedef struct STableBlockInfo {
|
||||
|
|
@ -107,6 +108,7 @@ typedef struct SBlockLoadSuppInfo {
|
|||
|
||||
typedef struct STsdbReadHandle {
|
||||
STsdb* pTsdb;
|
||||
uint64_t suid;
|
||||
SQueryFilePos cur; // current position
|
||||
int16_t order;
|
||||
STimeWindow window; // the primary query time window that applies to all queries
|
||||
|
|
@ -200,8 +202,8 @@ static SArray* getDefaultLoadColumns(STsdbReadHandle* pTsdbReadHandle, bool load
|
|||
int64_t tsdbGetNumOfRowsInMemTable(tsdbReaderT* pHandle) {
|
||||
STsdbReadHandle* pTsdbReadHandle = (STsdbReadHandle*)pHandle;
|
||||
|
||||
int64_t rows = 0;
|
||||
STsdbMemTable* pMemTable = NULL; // pTsdbReadHandle->pMemTable;
|
||||
int64_t rows = 0;
|
||||
SMemTable* pMemTable = NULL; // pTsdbReadHandle->pMemTable;
|
||||
if (pMemTable == NULL) {
|
||||
return rows;
|
||||
}
|
||||
|
|
@ -237,6 +239,7 @@ static SArray* createCheckInfoFromTableGroup(STsdbReadHandle* pTsdbReadHandle, S
|
|||
STableKeyInfo* pKeyInfo = (STableKeyInfo*)taosArrayGet(pTableList->pTableList, j);
|
||||
|
||||
STableCheckInfo info = {.lastKey = pKeyInfo->lastKey, .tableId = pKeyInfo->uid};
|
||||
info.suid = pTsdbReadHandle->suid;
|
||||
if (ASCENDING_TRAVERSE(pTsdbReadHandle->order)) {
|
||||
if (info.lastKey == INT64_MIN || info.lastKey < pTsdbReadHandle->window.skey) {
|
||||
info.lastKey = pTsdbReadHandle->window.skey;
|
||||
|
|
@ -265,8 +268,8 @@ static void resetCheckInfo(STsdbReadHandle* pTsdbReadHandle) {
|
|||
for (int32_t i = 0; i < numOfTables; ++i) {
|
||||
STableCheckInfo* pCheckInfo = (STableCheckInfo*)taosArrayGet(pTsdbReadHandle->pTableCheckInfo, i);
|
||||
pCheckInfo->lastKey = pTsdbReadHandle->window.skey;
|
||||
pCheckInfo->iter = tSkipListDestroyIter(pCheckInfo->iter);
|
||||
pCheckInfo->iiter = tSkipListDestroyIter(pCheckInfo->iiter);
|
||||
pCheckInfo->iter = tsdbTbDataIterDestroy(pCheckInfo->iter);
|
||||
pCheckInfo->iiter = tsdbTbDataIterDestroy(pCheckInfo->iiter);
|
||||
pCheckInfo->initBuf = false;
|
||||
|
||||
if (ASCENDING_TRAVERSE(pTsdbReadHandle->order)) {
|
||||
|
|
@ -387,6 +390,7 @@ static STsdbReadHandle* tsdbQueryTablesImpl(SVnode* pVnode, SQueryTableDataCond*
|
|||
pReadHandle->locateStart = false;
|
||||
pReadHandle->loadType = pCond->type;
|
||||
|
||||
pReadHandle->suid = pCond->suid;
|
||||
pReadHandle->outputCapacity = 4096; //((STsdb*)tsdb)->config.maxRowsPerFileBlock;
|
||||
pReadHandle->loadExternalRow = pCond->loadExternalRows;
|
||||
pReadHandle->currentLoadExternalRows = pCond->loadExternalRows;
|
||||
|
|
@ -658,7 +662,7 @@ tsdbReaderT tsdbQueryLastRow(SVnode* pVnode, SQueryTableDataCond* pCond, STableL
|
|||
}
|
||||
|
||||
#if 0
|
||||
tsdbReaderT tsdbQueryCacheLastT(STsdb *tsdb, SQueryTableDataCond *pCond, STableGroupInfo *groupList, uint64_t qId, STsdbMemTable* pMemRef) {
|
||||
tsdbReaderT tsdbQueryCacheLastT(STsdb *tsdb, SQueryTableDataCond *pCond, STableGroupInfo *groupList, uint64_t qId, SMemTable* pMemRef) {
|
||||
STsdbReadHandle *pTsdbReadHandle = (STsdbReadHandle*) tsdbQueryTablesT(tsdb, pCond, groupList, qId, pMemRef);
|
||||
if (pTsdbReadHandle == NULL) {
|
||||
return NULL;
|
||||
|
|
@ -752,23 +756,22 @@ static bool initTableMemIterator(STsdbReadHandle* pHandle, STableCheckInfo* pChe
|
|||
pCheckInfo->initBuf = true;
|
||||
int32_t order = pHandle->order;
|
||||
|
||||
STbData** pMem = NULL;
|
||||
STbData** pIMem = NULL;
|
||||
STbData* pMem = NULL;
|
||||
STbData* pIMem = NULL;
|
||||
int8_t backward = (pHandle->order == TSDB_ORDER_DESC) ? 1 : 0;
|
||||
|
||||
TSKEY tLastKey = keyToTkey(pCheckInfo->lastKey);
|
||||
if (pHandle->pTsdb->mem != NULL) {
|
||||
pMem = taosHashGet(pHandle->pTsdb->mem->pHashIdx, &pCheckInfo->tableId, sizeof(pCheckInfo->tableId));
|
||||
tsdbGetTbDataFromMemTable(pHandle->pTsdb->mem, pCheckInfo->suid, pCheckInfo->tableId, &pMem);
|
||||
if (pMem != NULL) {
|
||||
pCheckInfo->iter =
|
||||
tSkipListCreateIterFromVal((*pMem)->pData, (const char*)&tLastKey, TSDB_DATA_TYPE_TIMESTAMP, order);
|
||||
tsdbTbDataIterCreate(pMem, &(TSDBKEY){.version = 0, .ts = tLastKey}, backward, &pCheckInfo->iter);
|
||||
}
|
||||
}
|
||||
|
||||
if (pHandle->pTsdb->imem != NULL) {
|
||||
pIMem = taosHashGet(pHandle->pTsdb->imem->pHashIdx, &pCheckInfo->tableId, sizeof(pCheckInfo->tableId));
|
||||
tsdbGetTbDataFromMemTable(pHandle->pTsdb->mem, pCheckInfo->suid, pCheckInfo->tableId, &pIMem);
|
||||
if (pIMem != NULL) {
|
||||
pCheckInfo->iiter =
|
||||
tSkipListCreateIterFromVal((*pIMem)->pData, (const char*)&tLastKey, TSDB_DATA_TYPE_TIMESTAMP, order);
|
||||
tsdbTbDataIterCreate(pIMem, &(TSDBKEY){.version = 0, .ts = tLastKey}, backward, &pCheckInfo->iiter);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -777,22 +780,23 @@ static bool initTableMemIterator(STsdbReadHandle* pHandle, STableCheckInfo* pChe
|
|||
return false;
|
||||
}
|
||||
|
||||
bool memEmpty = (pCheckInfo->iter == NULL) || (pCheckInfo->iter != NULL && !tSkipListIterNext(pCheckInfo->iter));
|
||||
bool imemEmpty = (pCheckInfo->iiter == NULL) || (pCheckInfo->iiter != NULL && !tSkipListIterNext(pCheckInfo->iiter));
|
||||
bool memEmpty =
|
||||
(pCheckInfo->iter == NULL) || (pCheckInfo->iter != NULL && !tsdbTbDataIterGet(pCheckInfo->iter, NULL));
|
||||
bool imemEmpty =
|
||||
(pCheckInfo->iiter == NULL) || (pCheckInfo->iiter != NULL && !tsdbTbDataIterGet(pCheckInfo->iiter, NULL));
|
||||
if (memEmpty && imemEmpty) { // buffer is empty
|
||||
return false;
|
||||
}
|
||||
|
||||
if (!memEmpty) {
|
||||
SSkipListNode* node = tSkipListIterGet(pCheckInfo->iter);
|
||||
assert(node != NULL);
|
||||
TSDBROW row;
|
||||
|
||||
STSRow* row = (STSRow*)SL_GET_NODE_DATA(node);
|
||||
TSKEY key = TD_ROW_KEY(row); // first timestamp in buffer
|
||||
tsdbTbDataIterGet(pCheckInfo->iter, &row);
|
||||
TSKEY key = row.pTSRow->ts; // first timestamp in buffer
|
||||
tsdbDebug("%p uid:%" PRId64 ", check data in mem from skey:%" PRId64 ", order:%d, ts range in buf:%" PRId64
|
||||
"-%" PRId64 ", lastKey:%" PRId64 ", numOfRows:%" PRId64 ", %s",
|
||||
pHandle, pCheckInfo->tableId, key, order, (*pMem)->keyMin, (*pMem)->keyMax, pCheckInfo->lastKey,
|
||||
(*pMem)->nrows, pHandle->idStr);
|
||||
pHandle, pCheckInfo->tableId, key, order, pMem->minKey.ts, pMem->maxKey.ts, pCheckInfo->lastKey,
|
||||
pMem->sl.size, pHandle->idStr);
|
||||
|
||||
if (ASCENDING_TRAVERSE(order)) {
|
||||
assert(pCheckInfo->lastKey <= key);
|
||||
|
|
@ -805,15 +809,14 @@ static bool initTableMemIterator(STsdbReadHandle* pHandle, STableCheckInfo* pChe
|
|||
}
|
||||
|
||||
if (!imemEmpty) {
|
||||
SSkipListNode* node = tSkipListIterGet(pCheckInfo->iiter);
|
||||
assert(node != NULL);
|
||||
TSDBROW row;
|
||||
|
||||
STSRow* row = (STSRow*)SL_GET_NODE_DATA(node);
|
||||
TSKEY key = TD_ROW_KEY(row); // first timestamp in buffer
|
||||
tsdbTbDataIterGet(pCheckInfo->iter, &row);
|
||||
TSKEY key = row.pTSRow->ts; // first timestamp in buffer
|
||||
tsdbDebug("%p uid:%" PRId64 ", check data in imem from skey:%" PRId64 ", order:%d, ts range in buf:%" PRId64
|
||||
"-%" PRId64 ", lastKey:%" PRId64 ", numOfRows:%" PRId64 ", %s",
|
||||
pHandle, pCheckInfo->tableId, key, order, (*pIMem)->keyMin, (*pIMem)->keyMax, pCheckInfo->lastKey,
|
||||
(*pIMem)->nrows, pHandle->idStr);
|
||||
pHandle, pCheckInfo->tableId, key, order, pIMem->minKey.ts, pIMem->maxKey.ts, pCheckInfo->lastKey,
|
||||
pIMem->sl.size, pHandle->idStr);
|
||||
|
||||
if (ASCENDING_TRAVERSE(order)) {
|
||||
assert(pCheckInfo->lastKey <= key);
|
||||
|
|
@ -828,31 +831,23 @@ static bool initTableMemIterator(STsdbReadHandle* pHandle, STableCheckInfo* pChe
|
|||
}
|
||||
|
||||
static void destroyTableMemIterator(STableCheckInfo* pCheckInfo) {
|
||||
tSkipListDestroyIter(pCheckInfo->iter);
|
||||
tSkipListDestroyIter(pCheckInfo->iiter);
|
||||
tsdbTbDataIterDestroy(pCheckInfo->iter);
|
||||
tsdbTbDataIterDestroy(pCheckInfo->iiter);
|
||||
}
|
||||
|
||||
static TSKEY extractFirstTraverseKey(STableCheckInfo* pCheckInfo, int32_t order, int32_t update, TDRowVerT maxVer) {
|
||||
TSDBROW row = {0};
|
||||
STSRow *rmem = NULL, *rimem = NULL;
|
||||
|
||||
if (pCheckInfo->iter) {
|
||||
SSkipListNode* node = tSkipListIterGet(pCheckInfo->iter);
|
||||
if (node != NULL) {
|
||||
rmem = (STSRow*)SL_GET_NODE_DATA(node);
|
||||
// TODO: filter max version
|
||||
// if (TD_ROW_VER(rmem) > maxVer) {
|
||||
// rmem = NULL;
|
||||
// }
|
||||
if (tsdbTbDataIterGet(pCheckInfo->iter, &row)) {
|
||||
rmem = row.pTSRow;
|
||||
}
|
||||
}
|
||||
|
||||
if (pCheckInfo->iiter) {
|
||||
SSkipListNode* node = tSkipListIterGet(pCheckInfo->iiter);
|
||||
if (node != NULL) {
|
||||
rimem = (STSRow*)SL_GET_NODE_DATA(node);
|
||||
// TODO: filter max version
|
||||
// if (TD_ROW_VER(rimem) > maxVer) {
|
||||
// rimem = NULL;
|
||||
// }
|
||||
if (tsdbTbDataIterGet(pCheckInfo->iiter, &row)) {
|
||||
rimem = row.pTSRow;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -889,7 +884,7 @@ static TSKEY extractFirstTraverseKey(STableCheckInfo* pCheckInfo, int32_t order,
|
|||
pCheckInfo->chosen = CHECKINFO_CHOSEN_BOTH;
|
||||
} else {
|
||||
pCheckInfo->chosen = CHECKINFO_CHOSEN_IMEM;
|
||||
tSkipListIterNext(pCheckInfo->iter);
|
||||
tsdbTbDataIterNext(pCheckInfo->iter);
|
||||
}
|
||||
return r1;
|
||||
} else if (r1 < r2 && ASCENDING_TRAVERSE(order)) {
|
||||
|
|
@ -903,28 +898,17 @@ static TSKEY extractFirstTraverseKey(STableCheckInfo* pCheckInfo, int32_t order,
|
|||
|
||||
static STSRow* getSRowInTableMem(STableCheckInfo* pCheckInfo, int32_t order, int32_t update, STSRow** extraRow,
|
||||
TDRowVerT maxVer) {
|
||||
TSDBROW row;
|
||||
STSRow *rmem = NULL, *rimem = NULL;
|
||||
if (pCheckInfo->iter) {
|
||||
SSkipListNode* node = tSkipListIterGet(pCheckInfo->iter);
|
||||
if (node != NULL) {
|
||||
rmem = (STSRow*)SL_GET_NODE_DATA(node);
|
||||
#if 0 // TODO: skiplist refactor
|
||||
if (TD_ROW_VER(rmem) > maxVer) {
|
||||
rmem = NULL;
|
||||
}
|
||||
#endif
|
||||
if (tsdbTbDataIterGet(pCheckInfo->iter, &row)) {
|
||||
rmem = row.pTSRow;
|
||||
}
|
||||
}
|
||||
|
||||
if (pCheckInfo->iiter) {
|
||||
SSkipListNode* node = tSkipListIterGet(pCheckInfo->iiter);
|
||||
if (node != NULL) {
|
||||
rimem = (STSRow*)SL_GET_NODE_DATA(node);
|
||||
#if 0 // TODO: skiplist refactor
|
||||
if (TD_ROW_VER(rimem) > maxVer) {
|
||||
rimem = NULL;
|
||||
}
|
||||
#endif
|
||||
if (tsdbTbDataIterGet(pCheckInfo->iiter, &row)) {
|
||||
rimem = row.pTSRow;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -966,7 +950,7 @@ static STSRow* getSRowInTableMem(STableCheckInfo* pCheckInfo, int32_t order, int
|
|||
*extraRow = rimem;
|
||||
return rmem;
|
||||
} else {
|
||||
tSkipListIterNext(pCheckInfo->iter);
|
||||
tsdbTbDataIterNext(pCheckInfo->iter);
|
||||
pCheckInfo->chosen = CHECKINFO_CHOSEN_IMEM;
|
||||
return rimem;
|
||||
}
|
||||
|
|
@ -995,7 +979,7 @@ static bool moveToNextRowInMem(STableCheckInfo* pCheckInfo) {
|
|||
bool hasNext = false;
|
||||
if (pCheckInfo->chosen == CHECKINFO_CHOSEN_MEM) {
|
||||
if (pCheckInfo->iter != NULL) {
|
||||
hasNext = tSkipListIterNext(pCheckInfo->iter);
|
||||
hasNext = tsdbTbDataIterNext(pCheckInfo->iter);
|
||||
}
|
||||
|
||||
if (hasNext) {
|
||||
|
|
@ -1003,11 +987,11 @@ static bool moveToNextRowInMem(STableCheckInfo* pCheckInfo) {
|
|||
}
|
||||
|
||||
if (pCheckInfo->iiter != NULL) {
|
||||
return tSkipListIterGet(pCheckInfo->iiter) != NULL;
|
||||
return tsdbTbDataIterGet(pCheckInfo->iiter, NULL);
|
||||
}
|
||||
} else if (pCheckInfo->chosen == CHECKINFO_CHOSEN_IMEM) {
|
||||
if (pCheckInfo->iiter != NULL) {
|
||||
hasNext = tSkipListIterNext(pCheckInfo->iiter);
|
||||
hasNext = tsdbTbDataIterNext(pCheckInfo->iiter);
|
||||
}
|
||||
|
||||
if (hasNext) {
|
||||
|
|
@ -1015,14 +999,14 @@ static bool moveToNextRowInMem(STableCheckInfo* pCheckInfo) {
|
|||
}
|
||||
|
||||
if (pCheckInfo->iter != NULL) {
|
||||
return tSkipListIterGet(pCheckInfo->iter) != NULL;
|
||||
return tsdbTbDataIterGet(pCheckInfo->iter, NULL);
|
||||
}
|
||||
} else {
|
||||
if (pCheckInfo->iter != NULL) {
|
||||
hasNext = tSkipListIterNext(pCheckInfo->iter);
|
||||
hasNext = tsdbTbDataIterNext(pCheckInfo->iter);
|
||||
}
|
||||
if (pCheckInfo->iiter != NULL) {
|
||||
hasNext = tSkipListIterNext(pCheckInfo->iiter) || hasNext;
|
||||
hasNext = tsdbTbDataIterNext(pCheckInfo->iiter) || hasNext;
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1126,7 +1110,7 @@ static int32_t loadBlockInfo(STsdbReadHandle* pTsdbReadHandle, int32_t index, in
|
|||
STableCheckInfo* pCheckInfo = taosArrayGet(pTsdbReadHandle->pTableCheckInfo, index);
|
||||
pCheckInfo->numOfBlocks = 0;
|
||||
|
||||
STable table = {.uid = pCheckInfo->tableId, .tid = pCheckInfo->tableId};
|
||||
STable table = {.uid = pCheckInfo->tableId, .suid = pCheckInfo->suid};
|
||||
table.pSchema = pTsdbReadHandle->pSchema;
|
||||
|
||||
if (tsdbSetReadTable(&pTsdbReadHandle->rhelper, &table) != TSDB_CODE_SUCCESS) {
|
||||
|
|
@ -2924,7 +2908,7 @@ static bool loadBlockOfActiveTable(STsdbReadHandle* pTsdbReadHandle) {
|
|||
// current result is empty
|
||||
if (pTsdbReadHandle->currentLoadExternalRows && pTsdbReadHandle->window.skey == pTsdbReadHandle->window.ekey &&
|
||||
pTsdbReadHandle->cur.rows == 0) {
|
||||
// STsdbMemTable* pMemRef = pTsdbReadHandle->pMemTable;
|
||||
// SMemTable* pMemRef = pTsdbReadHandle->pMemTable;
|
||||
|
||||
// doGetExternalRow(pTsdbReadHandle, TSDB_PREV_ROW, pMemRef);
|
||||
// doGetExternalRow(pTsdbReadHandle, TSDB_NEXT_ROW, pMemRef);
|
||||
|
|
@ -3222,7 +3206,7 @@ bool tsdbNextDataBlock(tsdbReaderT pHandle) {
|
|||
}
|
||||
}
|
||||
|
||||
// static int32_t doGetExternalRow(STsdbReadHandle* pTsdbReadHandle, int16_t type, STsdbMemTable* pMemRef) {
|
||||
// static int32_t doGetExternalRow(STsdbReadHandle* pTsdbReadHandle, int16_t type, SMemTable* pMemRef) {
|
||||
// STsdbReadHandle* pSecQueryHandle = NULL;
|
||||
//
|
||||
// if (type == TSDB_PREV_ROW && pTsdbReadHandle->prev) {
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue