xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat:[TS-3718]merge from 3.0

This commit is contained in:
wangmm0220 2024-08-08 17:35:24 +08:00
parent bce6f40abb 0d1d462e08
commit e7b990f3dd
60 changed files with 1658 additions and 1026 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

BIN
deps/win/x64/dm_static/dmodule.lib vendored
View File

Binary file not shown.

4
docs/zh/06-advanced/05-data-in/index.md
View File

@ -1,6 +1,6 @@
--- ---
sidebar_label: 数据写入 sidebar_label: 数据接入
title: 零代码数据源接入 title: 零代码第三方数据接入
toc_max_heading_level: 4 toc_max_heading_level: 4
--- ---

391
docs/zh/07-operation/05-monitor.md
View File

@ -8,57 +8,13 @@ toc_max_heading_level: 4
至于如何获取和使用这些监控数据，用户可以使用第三方的监测工具比如 Zabbix 来获取这些保存的系统监测数据，进而将 TDengine 的运行状况无缝集成到现有的 IT 监控系统中。也可以使用 TDengine 提供的 TDinsight 插件，使用该插件用户可以通过 Grafana 平台直观地展示和管理这些监控信息，如下图所示。这为用户提供了灵活的监控选项，以满足不同场景下的运维需求。 至于如何获取和使用这些监控数据，用户可以使用第三方的监测工具比如 Zabbix 来获取这些保存的系统监测数据，进而将 TDengine 的运行状况无缝集成到现有的 IT 监控系统中。也可以使用 TDengine 提供的 TDinsight 插件，使用该插件用户可以通过 Grafana 平台直观地展示和管理这些监控信息，如下图所示。这为用户提供了灵活的监控选项，以满足不同场景下的运维需求。
~[通过监控组件管理监控信息](./grafana.png) ![通过监控组件管理监控信息](./grafana.png)
## 配置 taosKeeper ## 配置 taosKeeper
因为 TDengine 的监控数据都通过 taosKeeper 上报并存储，所以本节先介绍 taosKeeper 的配置。 因为 TDengine 的监控数据都通过 taosKeeper 上报并存储，所以本节先介绍 taosKeeper 的配置。
taosKeeper 的配置文件默认位于 `/etc/taos/taoskeeper.toml`。 下面为一个示例配置文件，更多详细信息见参考手册。其中最为关键的一个配置项是 `database`，它决定了收集到的监控数据存储在目标系统的哪个数据库中。 taosKeeper 的配置文件默认位于 `/etc/taos/taoskeeper.toml`。 详细配置见 [参考手册](../../reference/components/taoskeeper/#配置文件)。其中最为关键的一个配置项是 `database`，它决定了收集到的监控数据存储在目标系统的哪个数据库中。
```toml
# gin 框架是否启用 debug
debug = false
# 服务监听端口, 默认为 6043
port = 6043
# 日志级别，包含 panic、error、info、debug、trace等
loglevel = "info"
# 程序中使用协程池的大小
gopoolsize = 50000
# 查询 TDengine 监控数据轮询间隔
RotationInterval = "15s"
[tdengine]
host = "127.0.0.1"
port = 6041
username = "root"
password = "taosdata"
# 需要被监控的 taosAdapter
[taosAdapter]
address = ["127.0.0.1:6041"]
[metrics]
# 监控指标前缀
prefix = "taos"
# 集群数据的标识符
cluster = "production"
# 存放监控数据的数据库
database = "log"
# 指定需要监控的普通表
tables = []
# database options for db storing metrics data
[metrics.databaseoptions]
cachemodel = "none"
```
## 监控 taosd ## 监控 taosd
@ -66,9 +22,9 @@ cachemodel = "none"
为了简化用户在 TDengine 监控方面的配置工作，TDengine 提供了一个名为 TDinsight 的 Grafana 插件。该插件与 taosKeeper 协同工作，能够实时监控 TDengine 的各项性能指标。 为了简化用户在 TDengine 监控方面的配置工作，TDengine 提供了一个名为 TDinsight 的 Grafana 插件。该插件与 taosKeeper 协同工作，能够实时监控 TDengine 的各项性能指标。
通过集成 Grafana 和 TDengine 数据源插件，TDinsight 能够读取 taosKeeper 收集并存储的监控数据。这使得用户可以在 Grafana 平台上直观地查看 TDengine 集群的状态、节点信息、读写请求以及资源使用情况等关键指标，实现数据的可视化展示。 通过集成 Grafana 和 TDengine 数据源插件，TDinsight 能够读取 taosKeeper 收集的监控数据。这使得用户可以在 Grafana 平台上直观地查看 TDengine 集群的状态、节点信息、读写请求以及资源使用情况等关键指标，实现数据的可视化展示。
此外，TDinsight 还具备针对 vnode、dnode 和 mnode 节点的异常状态告警功能，为开发者提供实时的集群运行状态监控，确保 TDengine 集群的稳定性和可靠性。以下是TDinsight 的详细使用说明，以帮助你充分利用这一强大工具。 以下是TDinsight 的详细使用说明，以帮助你充分利用这一强大工具。
#### 前置条件 #### 前置条件
@ -76,314 +32,31 @@ cachemodel = "none"
- TDengine 已安装并正常运行。 - TDengine 已安装并正常运行。
- taosAdapter 已经安装并正常运行。 - taosAdapter 已经安装并正常运行。
- taosKeeper 已经安装并正常运行。 - taosKeeper 已经安装并正常运行。
- Grafana 已安装并正常运行，以下介绍以 Grafna 10.4.0 为例。 - Grafana 已安装并正常运行，以下介绍以 Grafna 11.0.0 为例。
同时记录以下信息。 同时记录以下信息。
- taosAdapter 的 RESTful 接口地址，如 http://www.example.com:6041。 - taosAdapter 的 RESTful 接口地址，如 `http://www.example.com:6041`。
- TDengine 集群的认证信息，包括用户名及密码。 - TDengine 集群的认证信息，包括用户名及密码。
#### 导入仪表盘 #### 导入仪表盘
TDengine 数据源插件已被提交至 Grafana 官网，完成插件的安装和数据源的创建后，可以进行 TDinsight 仪表盘的导入。 TDengine 数据源插件已被提交至 Grafana 官网，完成插件的安装和数据源的创建后，可以进行 TDinsight 仪表盘的导入。
在 Grafana 的 Home-Dashboards 页面，点击位于右上角的 New → mport 按钮，即可进入 Dashboard 的导入页面，它支持以下两种导入方式。 在 Grafana 的 ”Home“ -> ”Dashboards“ 页面，点击位于右上角的 ”New“ -> ”import“ 按钮，即可进入 Dashboard 的导入页面，它支持以下两种导入方式。
- Dashboard ID：18180。 - Dashboard ID：18180。
- Dashboard URL：https://grafana.com/grafana/dashboards/18180-tdinsight-for-3-x/ - Dashboard URL：https://grafana.com/grafana/dashboards/18180-tdinsight-for-3-x/
填写以上 Dashboard ID 或 Dashboard URL 以后，点击 Load 按钮，按照向导操作，即可完成导入。导入成功后，Dashboards 列表页面会出现 TDinsight for 3.x 仪盘，点击进入后，就可以看到 TDinsight 中已创建的各个指标的面板，如下图所示： 填写以上 Dashboard ID 或 Dashboard URL 以后，点击 ”Load“ 按钮，按照向导操作，即可完成导入。导入成功后，Dashboards 列表页面会出现 ”TDinsight for 3.x“ 仪表盘，点击进入后，就可以看到 TDinsight 中已创建的各个指标的面板，如下图所示：
![TDinsight 界面示例](./tdinsight.png) ![TDinsight 界面示例](./TDinsight-1-cluster-status.webp)
**注意** 在 TDinsight 界面左上角的 Log from 下拉列表中可以选择 log 数据库。 **注意** 在 TDinsight 界面左上角的 ”Log from“ 下拉列表中可以选择 `log` 数据库。
### taosd 监控数据 ### TDengine V3 监控数据
TDinsight dashboard 数据来源于 log 库（存放监控数据的默认 db，可以在 taoskeeper 配置文件中修改。以下是由 taosd 上报由 taosKeeper 存储在 log 库中的数据。 TDinsight dashboard 数据来源于 `log` 库（存放监控数据的默认数据库，可以在 taoskeeper 配置文件中修改）。”TDinsight for 3.x“ 仪表盘查询了 taosd 和 TaosAdapter 的监控指标。
- taosd 的监控指标请参考 [taosd 监控指标](../../reference/components/taosd/#taosd-监控指标)
1. taosd\_cluster\_basic 表 - taosAdapter 的监控指标请参考 [taosAdapter 监控指标](../../reference/components/taosadapter/#taosadapter-监控指标)
`taosd_cluster_basic` 表记录集群基础信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|first\_ep|VARCHAR||集群 first ep|
|first\_ep\_dnode\_id|INT||集群 first ep 的 dnode id|
|cluster_version|VARCHAR||tdengine version。例如：3.0.4.0|
|cluster\_id|VARCHAR|TAG|cluster id|
2. taosd\_cluster\_info 表
`taosd_cluster_info` 表记录集群信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|cluster_uptime|DOUBLE||当前 master 节点的uptime。单位：秒|
|dbs\_total|DOUBLE||database 总数|
|tbs\_total|DOUBLE||当前集群 table 总数|
|stbs\_total|DOUBLE||当前集群 stable 总数|
|dnodes\_total|DOUBLE||当前集群 dnode 总数|
|dnodes\_alive|DOUBLE||当前集群 dnode 存活总数|
|mnodes\_total|DOUBLE||当前集群 mnode 总数|
|mnodes\_alive|DOUBLE||当前集群 mnode 存活总数|
|vgroups\_total|DOUBLE||当前集群 vgroup 总数|
|vgroups\_alive|DOUBLE||当前集群 vgroup 存活总数|
|vnodes\_total|DOUBLE||当前集群 vnode 总数|
|vnodes\_alive|DOUBLE||当前集群 vnode 存活总数|
|connections\_total|DOUBLE||当前集群连接总数|
|topics\_total|DOUBLE||当前集群 topic 总数|
|streams\_total|DOUBLE||当前集群 stream 总数|
|grants_expire\_time|DOUBLE||认证过期时间，企业版有效，社区版为 DOUBLE 最大值|
|grants_timeseries\_used|DOUBLE||已用测点数|
|grants_timeseries\_total|DOUBLE||总测点数，开源版本为 DOUBLE 最大值|
|cluster\_id|VARCHAR|TAG|cluster id|
3. taosd\_vgroups\_info 表
`taosd_vgroups_info` 表记录虚拟节点组信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|tables\_num|DOUBLE||vgroup 中 table 数量|
|status|DOUBLE||vgroup 状态, 取值范围：unsynced = 0, ready = 1|
|vgroup\_id|VARCHAR|TAG|vgroup id|
|database\_name|VARCHAR|TAG|vgroup 所属的 database 名字|
|cluster\_id|VARCHAR|TAG|cluster id|
4. taosd\_dnodes\_info 表
`taosd_dnodes_info` 记录 dnode 信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|uptime|DOUBLE||dnode uptime，单位：秒|
|cpu\_engine|DOUBLE||taosd cpu 使用率，从 `/proc/<taosd_pid>/stat` 读取|
|cpu\_system|DOUBLE||服务器 cpu 使用率，从 `/proc/stat` 读取|
|cpu\_cores|DOUBLE||服务器 cpu 核数|
|mem\_engine|DOUBLE||taosd 内存使用率，从 `/proc/<taosd_pid>/status` 读取|
|mem\_free|DOUBLE||服务器可用内存，单位 KB|
|mem\_total|DOUBLE||服务器内存总量，单位 KB|
|disk\_used|DOUBLE||data dir 挂载的磁盘使用量，单位 bytes|
|disk\_total|DOUBLE||data dir 挂载的磁盘总容量，单位 bytes|
|system\_net\_in|DOUBLE||网络吞吐率，从 `/proc/net/dev` 中读取的 received bytes。单位 byte/s|
|system\_net\_out|DOUBLE||网络吞吐率，从 `/proc/net/dev` 中读取的 transmit bytes。单位 byte/s|
|io\_read|DOUBLE||io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 rchar 与上次数值计算之后，计算得到速度。单位 byte/s|
|io\_write|DOUBLE||io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 wchar 与上次数值计算之后，计算得到速度。单位 byte/s|
|io\_read\_disk|DOUBLE||磁盘 io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 read_bytes。单位 byte/s|
|io\_write\_disk|DOUBLE||磁盘 io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 write_bytes。单位 byte/s|
|vnodes\_num|DOUBLE||dnode 上 vnodes 数量|
|masters|DOUBLE||dnode 上 master node 数量|
|has\_mnode|DOUBLE||dnode 是否包含 mnode，取值范围：包含=1,不包含=0|
|has\_qnode|DOUBLE||dnode 是否包含 qnode，取值范围：包含=1,不包含=0|
|has\_snode|DOUBLE||dnode 是否包含 snode，取值范围：包含=1,不包含=0|
|has\_bnode|DOUBLE||dnode 是否包含 bnode，取值范围：包含=1,不包含=0|
|error\_log\_count|DOUBLE||error 总数|
|info\_log\_count|DOUBLE||info 总数|
|debug\_log\_count|DOUBLE||debug 总数|
|trace\_log\_count|DOUBLE||trace 总数|
|dnode\_id|VARCHAR|TAG|dnode id|
|dnode\_ep|VARCHAR|TAG|dnode endpoint|
|cluster\_id|VARCHAR|TAG|cluster id|
5. taosd\_dnodes\_status 表
`taosd_dnodes_status` 表记录 dnode 状态信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|status|DOUBLE||dnode 状态,取值范围：ready=1，offline =0|
|dnode\_id|VARCHAR|TAG|dnode id|
|dnode\_ep|VARCHAR|TAG|dnode endpoint|
|cluster\_id|VARCHAR|TAG|cluster id|
6. taosd\_dnodes\_log\_dir 表
`taosd_dnodes_log_dir` 表记录 log 目录信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|avail|DOUBLE||log 目录可用空间。单位 byte|
|used|DOUBLE||log 目录已使用空间。单位 byte|
|total|DOUBLE||log 目录空间。单位 byte|
|name|VARCHAR|TAG|log 目录名，一般为 `/var/log/taos/`|
|dnode\_id|VARCHAR|TAG|dnode id|
|dnode\_ep|VARCHAR|TAG|dnode endpoint|
|cluster\_id|VARCHAR|TAG|cluster id|
7. taosd\_dnodes\_data\_dir 表
`taosd_dnodes_data_dir` 表记录 data 目录信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|avail|DOUBLE||data 目录可用空间。单位 byte|
|used|DOUBLE||data 目录已使用空间。单位 byte|
|total|DOUBLE||data 目录空间。单位 byte|
|level|VARCHAR|TAG|0、1、2 多级存储级别|
|name|VARCHAR|TAG|data 目录，一般为 `/var/lib/taos`|
|dnode\_id|VARCHAR|TAG|dnode id|
|dnode\_ep|VARCHAR|TAG|dnode endpoint|
|cluster\_id|VARCHAR|TAG|cluster id|
8. taosd\_mnodes\_info 表
`taosd_mnodes_info` 表记录 mnode 角色信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|role|DOUBLE||mnode 角色， 取值范围：offline = 0,follower = 100,candidate = 101,leader = 102,error = 103,learner = 104|
|mnode\_id|VARCHAR|TAG|master node id|
|mnode\_ep|VARCHAR|TAG|master node endpoint|
|cluster\_id|VARCHAR|TAG|cluster id|
9. taosd\_vnodes\_role 表
`taosd_vnodes_role` 表记录虚拟节点角色信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|vnode\_role|DOUBLE||vnode 角色，取值范围：offline = 0,follower = 100,candidate = 101,leader = 102,error = 103,learner = 104|
|vgroup\_id|VARCHAR|TAG|dnode id|
|dnode\_id|VARCHAR|TAG|dnode id|
|database\_name|VARCHAR|TAG|vgroup 所属的 database 名字|
|cluster\_id|VARCHAR|TAG|cluster id|
10. taosd\_sql\_req 表
`taosd_sql_req` 记录授权信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|count|DOUBLE||sql 数量|
|result|VARCHAR|TAG|sql的执行结果，取值范围：Success, Failed|
|username|VARCHAR|TAG|执行sql的user name|
|sql\_type|VARCHAR|TAG|sql类型，取值范围：inserted_rows|
|dnode\_id|VARCHAR|TAG|dnode id|
|dnode\_ep|VARCHAR|TAG|dnode endpoint|
|vgroup\_id|VARCHAR|TAG|dnode id|
|cluster\_id|VARCHAR|TAG|cluster id|
11. taos\_sql\_req 表
`taos_sql_req` 记录授权信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|count|DOUBLE||sql 数量|
|result|VARCHAR|TAG|sql的执行结果，取值范围：Success, Failed|
|username|VARCHAR|TAG|执行sql的user name|
|sql\_type|VARCHAR|TAG|sql类型，取值范围：select, insert，delete|
|cluster\_id|VARCHAR|TAG|cluster id|
12. taos\_slow\_sql 表
`taos_slow_sql` 记录授权信息。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|count|DOUBLE||sql 数量|
|result|VARCHAR|TAG|sql的执行结果，取值范围：Success, Failed|
|username|VARCHAR|TAG|执行sql的user name|
|duration|VARCHAR|TAG|sql执行耗时，取值范围：3-10s,10-100s,100-1000s,1000s-|
|cluster\_id|VARCHAR|TAG|cluster id|
13. keeper\_monitor 表
`keeper_monitor` 记录 taoskeeper 监控数据。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|ts|TIMESTAMP||timestamp|
|cpu|DOUBLE||cpu 使用率|
|mem|DOUBLE||内存使用率|
|identify|NCHAR|TAG||
14. taosadapter\_restful\_http\_request\_total 表
`taosadapter_restful_http_request_total` 记录 taosadapter rest 请求信息，该表为 schemaless 方式创建的表，时间戳字段名为 `_ts`。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|\_ts|TIMESTAMP||timestamp|
|gauge|DOUBLE||监控指标值|
|client\_ip|NCHAR|TAG|client ip|
|endpoint|NCHAR|TAG|taosadpater endpoint|
|request\_method|NCHAR|TAG|request method|
|request\_uri|NCHAR|TAG|request uri|
|status\_code|NCHAR|TAG|status code|
15. taosadapter\_restful\_http\_request\_fail 表
`taosadapter_restful_http_request_fail` 记录 taosadapter rest 请求失败信息，该表为 schemaless 方式创建的表，时间戳字段名为 `_ts`。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|\_ts|TIMESTAMP||timestamp|
|gauge|DOUBLE||监控指标值|
|client\_ip|NCHAR|TAG|client ip|
|endpoint|NCHAR|TAG|taosadpater endpoint|
|request\_method|NCHAR|TAG|request method|
|request\_uri|NCHAR|TAG|request uri|
|status\_code|NCHAR|TAG|status code|
16. taosadapter\_restful\_http\_request\_in\_flight 表
`taosadapter_restful_http_request_in_flight` 记录 taosadapter rest 实时请求信息，该表为 schemaless 方式创建的表，时间戳字段名为 `_ts`。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|\_ts|TIMESTAMP||timestamp|
|gauge|DOUBLE||监控指标值|
|endpoint|NCHAR|TAG|taosadpater endpoint|
17. taosadapter\_restful\_http\_request\_summary\_milliseconds 表
`taosadapter_restful_http_request_summary_milliseconds` 记录 taosadapter rest 请求汇总信息，该表为 schemaless 方式创建的表，时间戳字段名为 `_ts`。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|\_ts|TIMESTAMP||timestamp|
|count|DOUBLE|||
|sum|DOUBLE|||
|0.5|DOUBLE|||
|0.9|DOUBLE|||
|0.99|DOUBLE|||
|0.1|DOUBLE|||
|0.2|DOUBLE|||
|endpoint|NCHAR|TAG|taosadpater endpoint|
|request\_method|NCHAR|TAG|request method|
|request\_uri|NCHAR|TAG|request uri|
18. taosadapter\_system\_mem\_percent 表
`taosadapter_system_mem_percent` 表记录 taosadapter 内存使用情况，该表为 schemaless 方式创建的表，时间戳字段名为 `_ts`。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|\_ts|TIMESTAMP||timestamp|
|gauge|DOUBLE||监控指标值|
|endpoint|NCHAR|TAG|taosadpater endpoint|
19. taosadapter\_system\_cpu\_percent 表
`taosadapter_system_cpu_percent` 表记录 taosadapter cpu 使用情况，该表为 schemaless 方式创建的表，时间戳字段名为 `_ts`。
|field|type|is\_tag|comment|
|:----|:---|:-----|:------|
|\_ts|TIMESTAMP||timestamp|
|gauge|DOUBLE||监控指标值|
|endpoint|NCHAR|TAG|taosadpater endpoint|
## 监控 taosX ## 监控 taosX
@ -396,13 +69,13 @@ taosX 是 TDengine 中提供零代码数据接入能力的核心组件，对它
### 版本支持 ### 版本支持
1. TDengine 企业版本 3.2.3.0 或以上版本包含的 taosX 才包含此功能。如果单独安装 taosX，需要 taosX 1.5.0 或以上版本。 1. TDengine 企业版本 3.2.3.0 或以上版本包含的 taosX 才包含此功能。如果单独安装 taosX，需要 taosX 1.5.0 或以上版本。
1. 需要安装 Grafana 插件 [TDengie Datasource v3.5.0](https://grafana.com/grafana/plugins/tdengine-datasource/) 或以上版本。 2. 需要安装 Grafana 插件 [TDengie Datasource v3.5.0](https://grafana.com/grafana/plugins/tdengine-datasource/) 或以上版本。
### 准备工作 ### 准备工作
假设你已经部署好了 taosd，taosAdapter 和 taosAdapter。 那么还需要： 假设你已经部署好了 taosd，taosAdapter 和 taosAdapter。 那么还需要：
2. 启动 taosX 服务。 1. 启动 taosX 服务。
3. 部署 Grafana ，安装 TDengine Datasource 插件，配置好数据源。 2. 部署 Grafana ，安装 TDengine Datasource 插件，配置好数据源。
### 配置 taosX ### 配置 taosX
@ -426,38 +99,35 @@ toasX 的配置文件(默认 /etc/taos/taosx.toml) 中与 monitor 相关的配
| port | --monitor-port | MONITOR_PORT | taosKeeper 服务的端口 | | 6043 | | port | --monitor-port | MONITOR_PORT | taosKeeper 服务的端口 | | 6043 |
| interval | --monitor-interval | MONITTOR_INTERVAL | taosX 发送 metrics 数据到 taosKeeper 的时间间隔，单位秒 | 1-10 | 10 | | interval | --monitor-interval | MONITTOR_INTERVAL | taosX 发送 metrics 数据到 taosKeeper 的时间间隔，单位秒 | 1-10 | 10 |
TDinsight for taosX
"TDinsight for taosX" 专门为 taosX 监控创建的 Grafana 面板。使用前需要先导入这个面板。
### 基于 TDinsight 监控 tasoX ### 基于 TDinsight 监控 tasoX
"TDinsight for taosX" 是专门为 taosX 监控创建的 Grafana 面板。使用前需要先导入这个面板。
#### 进入面板 #### 进入面板
1. 选择 TDengine Datasource 1. 在 Grafana 界面菜单中点击 ”Data sources“， 然后选择已经配置好的 TDengine 数据源。
![TDengine Datasource](./pic/monitor-01.jpg) 2. 在数据源配置界面选择 “Dashboard” Tab, 然后导入 ”TDinsight for taosX“ 面板（第一次使用需要先导入）。 下面是一个示例图：
2. 点击 “Dashboard”, 选择 TDinsight for taosX 面板。（第一次使用需要先导入）。
![Dashboard](./pic/monitor-02.jpg)
![monitor rows](./pic/monitor-04.jpg) ![monitor rows](./pic/monitor-04.jpg)
该面板每一行代表一个或一类监控对象。最上面是 taosX 监控行，然后是 Agent 监控行, 最后是各类数据写入任务的监控。 该面板每一行代表一个或一类监控对象。最上面是 taosX 监控行，然后是 Agent 监控行, 最后是各类数据写入任务的监控。
:::note :::note
1. 如果打开这个面板后看不到任何数据，你很可能需要点击左上角的数据库列表（即 “Log from” 下拉菜单），切换到监控数据所在的数据库。 - 如果打开这个面板后看不到任何数据，你很可能需要点击左上角的数据库列表（即 “Log from” 下拉菜单），切换到监控数据所在的数据库。
2. 数据库包含多少个 Agent 的数据就会自动创建多少个 Agent 行。(如上图) - 数据库包含多少个 Agent 的数据就会自动创建多少个 Agent 行。(如上图)
::: :::
#### 监控示例 #### 监控示例
1. taosX 监控示例 1. taosX 监控示例图
![monitor taosx](./pic/monitor-03.png) ![monitor taosx](./pic/monitor-03.png)
2. Agent 监控示例 2. Agent 监控示例图
![monitor agent](./pic/monitor-09.jpg) ![monitor agent](./pic/monitor-09.jpg)
3. TDengine2 数据源监控示例 3. TDengine2 数据源监控示例图
![monitor tdengine2](./pic/monitor-05.png) ![monitor tdengine2](./pic/monitor-05.png)
@ -466,11 +136,12 @@ TDinsight for taosX
::: :::
3. TDengine3 数据源监控示例 4. TDengine3 数据源监控示例图
![monitor tdengine3](./pic/monitor-06.jpg) ![monitor tdengine3](./pic/monitor-06.jpg)
4. 其它数据源监控示例 5. 其它数据源监控示例图
![monitor task](./pic/monitor-10.jpg) ![monitor task](./pic/monitor-10.jpg)

BIN
docs/zh/07-operation/TDinsight-1-cluster-status.webp Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 104 KiB

BIN
docs/zh/07-operation/tdinsight.png
View File

Binary file not shown.
Side by Side

Before

Width:  |  Height:  |  Size: 268 KiB

46
docs/zh/08-develop/09-udf.md
View File

@ -6,7 +6,7 @@ toc_max_heading_level: 4
## UDF 简介 ## UDF 简介
在某些应用场景中，应用逻辑需要的查询功能无法直接使用 TDengine 内置的函数来实现。TDengine 允许编写用户自定义函数（UDF），以便解决特殊应用场景中的使用需求。UDF 在集群中注册成功后，可以像系统内置函数一样在 SQ L中调用，就使用角度而言没有任何区别。UDF 分为标量函数和聚合函数。标量函数对每行数据输出一个值，如求绝对值（abs）、正弦函数（sin）、字符串拼接函数（concat）等。聚合函数对多行数据输出一个值，如求平均数（avg）、取最大值（max）等。 在某些应用场景中，应用逻辑需要的查询功能无法直接使用内置函数来实现，TDengine 允许编写用户自定义函数（UDF），以便解决特殊应用场景中的使用需求。UDF 在集群中注册成功后，可以像系统内置函数一样在 SQL 中调用，就使用角度而言没有任何区别。UDF 分为标量函数和聚合函数。标量函数对每行数据输出一个值，如求绝对值（abs）、正弦函数（sin）、字符串拼接函数（concat）等。聚合函数对多行数据输出一个值，如求平均数（avg）、取最大值（max）等。
TDengine 支持用 C 和 Python 两种编程语言编写 UDF。C 语言编写的 UDF 与内置函数的性能几乎相同，Python 语言编写的 UDF 可以利用丰富的 Python 运算库。为了避免 UDF 执行中发生异常影响数据库服务，TDengine 使用了进程分离技术，把 UDF 的执行放到另一个进程中完成，即使用户编写的 UDF 崩溃，也不会影响 TDengine 的正常运行。 TDengine 支持用 C 和 Python 两种编程语言编写 UDF。C 语言编写的 UDF 与内置函数的性能几乎相同，Python 语言编写的 UDF 可以利用丰富的 Python 运算库。为了避免 UDF 执行中发生异常影响数据库服务，TDengine 使用了进程分离技术，把 UDF 的执行放到另一个进程中完成，即使用户编写的 UDF 崩溃，也不会影响 TDengine 的正常运行。
@ -15,20 +15,19 @@ TDengine 支持用 C 和 Python 两种编程语言编写 UDF。C 语言编写的
使用 C 语言实现 UDF 时，需要实现规定的接口函数 使用 C 语言实现 UDF 时，需要实现规定的接口函数
- 标量函数需要实现标量接口函数 scalarfn 。 - 标量函数需要实现标量接口函数 scalarfn 。
- 聚合函数需要实现聚合接口函数 aggfn_start、aggfn、aggfn_finish。 - 聚合函数需要实现聚合接口函数 aggfn_start、aggfn、aggfn_finish。
- 如果需要初始化，实现 udf_init；如果需要清理工作，实现 udf_destroy。 - 如果需要初始化，实现 udf_init。
- 如果需要清理工作，实现 udf_destroy。
接口函数的名称是 UDF 名称，或者是 UDF 名称和特定后缀（`_start`、`_finish`、`_init`、`_destroy`）的连接。
### 接口定义 ### 接口定义
在 TDengine 中，UDF 的接口函数名称可以是 UDF 名称，也可以是 UDF 名称和特定后缀（如_start、_finish、_init、_destroy）的连接。后面内容中描述的函数名称，例如 scalarfn、aggfn，需要替换成 UDF 名称。 接口函数的名称是 UDF 名称，或者是 UDF 名称和特定后缀（_start、_finish、_init、_destroy）的连接。后面内容中描述的函数名称，例如 scalarfn、aggfn，需要替换成 UDF 名称。
#### 标量函数接口 #### 标量函数接口
标量函数是一种将输入数据转换为输出数据的函数，通常用于对单个数据值进行计算和转换。标量函数的接口函数原型如下。 标量函数是一种将输入数据转换为输出数据的函数，通常用于对单个数据值进行计算和转换。标量函数的接口函数原型如下。
```c ```c
int32_t scalarfn(SUdfDataBlock* inputDataBlock, SUdfColumn *resultColumn) int32_t scalarfn(SUdfDataBlock* inputDataBlock, SUdfColumn *resultColumn);
``` ```
主要参数说明如下。 主要参数说明如下。
- inputDataBlock：输入的数据块。 - inputDataBlock：输入的数据块。
@ -47,9 +46,9 @@ int32_t scalarfn(SUdfDataBlock* inputDataBlock, SUdfColumn *resultColumn)
聚合函数的接口函数原型如下。 聚合函数的接口函数原型如下。
```c ```c
int32_t aggfn_start(SUdfInterBuf *interBuf) int32_t aggfn_start(SUdfInterBuf *interBuf);
int32_t aggfn(SUdfDataBlock* inputBlock, SUdfInterBuf *interBuf, SUdfInterBuf *newInterBuf) int32_t aggfn(SUdfDataBlock* inputBlock, SUdfInterBuf *interBuf, SUdfInterBuf *newInterBuf);
int32_t aggfn_finish(SUdfInterBuf* interBuf, SUdfInterBuf *result) int32_t aggfn_finish(SUdfInterBuf* interBuf, SUdfInterBuf *result);
``` ```
其中 aggfn 是函数名的占位符。首先调用 aggfn_start 生成结果 buffer，然后相关的数据会被分为多个行数据块，对每个数据块调用 aggfn 用数据块更新中间结果，最后再调用 aggfn_finish 从中间结果产生最终结果，最终结果只能含 0 或 1 条结果数据。 其中 aggfn 是函数名的占位符。首先调用 aggfn_start 生成结果 buffer，然后相关的数据会被分为多个行数据块，对每个数据块调用 aggfn 用数据块更新中间结果，最后再调用 aggfn_finish 从中间结果产生最终结果，最终结果只能含 0 或 1 条结果数据。
@ -79,15 +78,16 @@ int32_t udf_destroy()
#include "taoserror.h" #include "taoserror.h"
#include "taosudf.h" #include "taosudf.h"
// Initialization function. If no initialization, we can skip definition of it. // Initialization function.
// The initialization function shall be concatenation of the udf name and _init suffix // If no initialization, we can skip definition of it.
// The initialization function shall be concatenation of the udf name and _init suffix.
// @return error number defined in taoserror.h // @return error number defined in taoserror.h
int32_t scalarfn_init() { int32_t scalarfn_init() {
// initialization. // initialization.
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
// Scalar function main computation function // Scalar function main computation function.
// @param inputDataBlock, input data block composed of multiple columns with each column defined by SUdfColumn // @param inputDataBlock, input data block composed of multiple columns with each column defined by SUdfColumn
// @param resultColumn, output column // @param resultColumn, output column
// @return error number defined in taoserror.h // @return error number defined in taoserror.h
@ -96,7 +96,8 @@ int32_t scalarfn(SUdfDataBlock* inputDataBlock, SUdfColumn* resultColumn) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
// Cleanup function. If no cleanup related processing, we can skip definition of it. // Cleanup function.
// If no cleanup related processing, we can skip definition of it.
// The destroy function shall be concatenation of the udf name and _destroy suffix. // The destroy function shall be concatenation of the udf name and _destroy suffix.
// @return error number defined in taoserror.h // @return error number defined in taoserror.h
int32_t scalarfn_destroy() { int32_t scalarfn_destroy() {
@ -112,16 +113,18 @@ int32_t scalarfn_destroy() {
#include "taoserror.h" #include "taoserror.h"
#include "taosudf.h" #include "taosudf.h"
// Initialization function. If no initialization, we can skip definition of it. // Initialization function.
// The initialization function shall be concatenation of the udf name and _init suffix // If no initialization, we can skip definition of it.
// The initialization function shall be concatenation of the udf name and _init suffix.
// @return error number defined in taoserror.h // @return error number defined in taoserror.h
int32_t aggfn_init() { int32_t aggfn_init() {
// initialization. // initialization.
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
// Aggregate start function. The intermediate value or the state(@interBuf) is initialized in this function. // Aggregate start function.
// The function name shall be concatenation of udf name and _start suffix // The intermediate value or the state(@interBuf) is initialized in this function.
// The function name shall be concatenation of udf name and _start suffix.
// @param interbuf intermediate value to initialize // @param interbuf intermediate value to initialize
// @return error number defined in taoserror.h // @return error number defined in taoserror.h
int32_t aggfn_start(SUdfInterBuf* interBuf) { int32_t aggfn_start(SUdfInterBuf* interBuf) {
@ -129,7 +132,8 @@ int32_t aggfn_start(SUdfInterBuf* interBuf) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
// Aggregate reduce function. This function aggregate old state(@interbuf) and one data bock(inputBlock) and output a new state(@newInterBuf). // Aggregate reduce function.
// This function aggregate old state(@interbuf) and one data bock(inputBlock) and output a new state(@newInterBuf).
// @param inputBlock input data block // @param inputBlock input data block
// @param interBuf old state // @param interBuf old state
// @param newInterBuf new state // @param newInterBuf new state
@ -139,7 +143,8 @@ int32_t aggfn(SUdfDataBlock* inputBlock, SUdfInterBuf *interBuf, SUdfInterBuf *n
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
// Aggregate function finish function. This function transforms the intermediate value(@interBuf) into the final output(@result). // Aggregate function finish function.
// This function transforms the intermediate value(@interBuf) into the final output(@result).
// The function name must be concatenation of aggfn and _finish suffix. // The function name must be concatenation of aggfn and _finish suffix.
// @interBuf : intermediate value // @interBuf : intermediate value
// @result: final result // @result: final result
@ -149,7 +154,8 @@ int32_t int32_t aggfn_finish(SUdfInterBuf* interBuf, SUdfInterBuf *result) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
// Cleanup function. If no cleanup related processing, we can skip definition of it. // Cleanup function.
// If no cleanup related processing, we can skip definition of it.
// The destroy function shall be concatenation of the udf name and _destroy suffix. // The destroy function shall be concatenation of the udf name and _destroy suffix.
// @return error number defined in taoserror.h // @return error number defined in taoserror.h
int32_t aggfn_destroy() { int32_t aggfn_destroy() {

224
docs/zh/14-reference/01-components/01-taosd.md
View File

@ -28,7 +28,7 @@ taosd 命令行参数如下
### 连接相关 ### 连接相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :--------------------: | :-------------------------------------------------------------------------------------: |
| firstEp | taosd 启动时，主动连接的集群中首个 dnode 的 end point，缺省值：localhost:6030 | | firstEp | taosd 启动时，主动连接的集群中首个 dnode 的 end point，缺省值：localhost:6030 |
| secondEp | taosd 启动时，如果 firstEp 连接不上，尝试连接集群中第二个 dnode 的 endpoint，缺省值：无 | | secondEp | taosd 启动时，如果 firstEp 连接不上，尝试连接集群中第二个 dnode 的 endpoint，缺省值：无 |
| fqdn | 启动 taosd 后所监听的服务地址，缺省值：所在服务器上配置的第一个 hostname | | fqdn | 启动 taosd 后所监听的服务地址，缺省值：所在服务器上配置的第一个 hostname |
@ -39,7 +39,7 @@ taosd 命令行参数如下
### 监控相关 ### 监控相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :----------------: | :------------------------------------------------------------------------------------: |
| monitor | 是否收集监控数据并上报，0: 关闭；1:打开；缺省值：0 | | monitor | 是否收集监控数据并上报，0: 关闭；1:打开；缺省值：0 |
| monitorFqdn | taosKeeper 服务所在服务器的 FQDN，缺省值：无 | | monitorFqdn | taosKeeper 服务所在服务器的 FQDN，缺省值：无 |
| monitorPort | taosKeeper 服务所监听的端口号，缺省值：6043 | | monitorPort | taosKeeper 服务所监听的端口号，缺省值：6043 |
@ -50,7 +50,7 @@ taosd 命令行参数如下
### 查询相关 ### 查询相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :--------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| queryPolicy | 查询策略，1: 只使用 vnode，不使用 qnode; 2: 没有扫描算子的子任务在 qnode 执行，带扫描算子的子任务在 vnode 执行; 3: vnode 只运行扫描算子，其余算子均在 qnode 执行 ；缺省值：1 | | queryPolicy | 查询策略，1: 只使用 vnode，不使用 qnode; 2: 没有扫描算子的子任务在 qnode 执行，带扫描算子的子任务在 vnode 执行; 3: vnode 只运行扫描算子，其余算子均在 qnode 执行 ；缺省值：1 |
| maxNumOfDistinctRes | 允许返回的 distinct 结果最大行数，默认值 10 万，最大允许值 1 亿 | | maxNumOfDistinctRes | 允许返回的 distinct 结果最大行数，默认值 10 万，最大允许值 1 亿 |
| countAlwaysReturnValue | ount/hyperloglog函数在输入数据为空或者NULL的情况下是否返回值，0: 返回空行，1: 返回；该参数设置为 1 时，如果查询中含有 INTERVAL 子句或者该查询使用了TSMA时, 且相应的组或窗口内数据为空或者NULL， 对应的组或窗口将不返回查询结果. 注意此参数客户端和服务端值应保持一致. | | countAlwaysReturnValue | ount/hyperloglog函数在输入数据为空或者NULL的情况下是否返回值，0: 返回空行，1: 返回；该参数设置为 1 时，如果查询中含有 INTERVAL 子句或者该查询使用了TSMA时, 且相应的组或窗口内数据为空或者NULL， 对应的组或窗口将不返回查询结果. 注意此参数客户端和服务端值应保持一致. |
@ -59,7 +59,7 @@ taosd 命令行参数如下
### 区域相关 ### 区域相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :------: | :------------------------------------------------------------------------------------------------------: |
| timezone | 时区，缺省值：当前服务器所配置的时区 | | timezone | 时区，缺省值：当前服务器所配置的时区 |
| locale | 系统区位信息及编码格式 ，缺省值：系统中动态获取，如果自动获取失败，需要用户在配置文件设置或通过 API 设置 | | locale | 系统区位信息及编码格式 ，缺省值：系统中动态获取，如果自动获取失败，需要用户在配置文件设置或通过 API 设置 |
| charset | 字符集编码，缺省值：系统自动获取 | | charset | 字符集编码，缺省值：系统自动获取 |
@ -142,7 +142,7 @@ charset 的有效值是 UTF-8。
### 存储相关 ### 存储相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :--------------: | :--------------------------------------------------------------------: |
| dataDir | 数据文件目录，所有的数据文件都将写入该目录，缺省值：/var/lib | | dataDir | 数据文件目录，所有的数据文件都将写入该目录，缺省值：/var/lib |
| tempDir | 指定所有系统运行过程中的临时文件生成的目录，缺省值：/tmp | | tempDir | 指定所有系统运行过程中的临时文件生成的目录，缺省值：/tmp |
| minimalTmpDirGB | tempDir 所指定的临时文件目录所需要保留的最小空间，单位 GB，缺省值: 1 | | minimalTmpDirGB | tempDir 所指定的临时文件目录所需要保留的最小空间，单位 GB，缺省值: 1 |
@ -151,19 +151,19 @@ charset 的有效值是 UTF-8。
### 集群相关 ### 集群相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :-----------: | :-------------------------------------------------------------------------: |
| supportVnodes | dnode 支持的最大 vnode 数目，取值范围：0-4096，缺省值： CPU 核数的 2 倍 + 5 | | supportVnodes | dnode 支持的最大 vnode 数目，取值范围：0-4096，缺省值： CPU 核数的 2 倍 + 5 |
### 性能调优 ### 性能调优
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :----------------: | :---------------------------------------------: |
| numOfCommitThreads | 写入线程的最大数量，取值范围 0-1024，缺省值为 4 | | numOfCommitThreads | 写入线程的最大数量，取值范围 0-1024，缺省值为 4 |
### 日志相关 ### 日志相关
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :--------------: | :----------------------------------------------------------------------------------------------------------------------------------------------------------: |
| logDir | 日志文件目录，运行日志将写入该目录，缺省值：/var/log/taos | | logDir | 日志文件目录，运行日志将写入该目录，缺省值：/var/log/taos |
| minimalLogDirGB | 当日志文件夹所在磁盘可用空间大小小于该值时，停止写日志，单位GB，缺省值：1 | | minimalLogDirGB | 当日志文件夹所在磁盘可用空间大小小于该值时，停止写日志，单位GB，缺省值：1 |
| numOfLogLines | 单个日志文件允许的最大行数，缺省值：10,000,000 | | numOfLogLines | 单个日志文件允许的最大行数，缺省值：10,000,000 |
@ -193,7 +193,7 @@ charset 的有效值是 UTF-8。
### 压缩参数 ### 压缩参数
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :-------------: | :----------------------------------------------------------------------------------------------------------------------------------------------: |
| compressMsgSize | 是否对 RPC 消息进行压缩；-1: 所有消息都不压缩; 0: 所有消息都压缩; N (N>0): 只有大于 N 个字节的消息才压缩；缺省值 -1 | | compressMsgSize | 是否对 RPC 消息进行压缩；-1: 所有消息都不压缩; 0: 所有消息都压缩; N (N>0): 只有大于 N 个字节的消息才压缩；缺省值 -1 |
| fPrecision | 设置 float 类型浮点数压缩精度 ，取值范围：0.1 ~ 0.00000001 ，默认值 0.00000001 , 小于此值的浮点数尾数部分将被截断 | | fPrecision | 设置 float 类型浮点数压缩精度 ，取值范围：0.1 ~ 0.00000001 ，默认值 0.00000001 , 小于此值的浮点数尾数部分将被截断 |
| dPrecision | 设置 double 类型浮点数压缩精度 , 取值范围：0.1 ~ 0.0000000000000001 ， 缺省值 0.0000000000000001 ， 小于此值的浮点数尾数部分将被截取 | | dPrecision | 设置 double 类型浮点数压缩精度 , 取值范围：0.1 ~ 0.0000000000000001 ， 缺省值 0.0000000000000001 ， 小于此值的浮点数尾数部分将被截取 |
@ -217,9 +217,213 @@ lossyColumns float|double
### 其他参数 ### 其他参数
| 参数名称 | 参数说明 | | 参数名称 | 参数说明 |
|:-------------:|:----------------------------------------------------------------:| | :--------------: | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |
| enableCoreFile | crash 时是否生成 core 文件；0: 不生成，1：生成；默认值 为 1; 不同的启动方式，生成 core 文件的目录如下：1、systemctl start taosd 启动：生成的 core 在根目录下 <br/> 2、手动启动，就在 taosd 执行目录下。 | | enableCoreFile | crash 时是否生成 core 文件；0: 不生成，1：生成；默认值 为 1; 不同的启动方式，生成 core 文件的目录如下：1、systemctl start taosd 启动：生成的 core 在根目录下 <br/> 2、手动启动，就在 taosd 执行目录下。 |
| udf | 是否启动 UDF 服务；0: 不启动，1：启动；默认值 为 0 | | udf | 是否启动 UDF 服务；0: 不启动，1：启动；默认值 为 0 |
| ttlChangeOnWrite | ttl 到期时间是否伴随表的修改操作改变; 0: 不改变，1：改变 ；默认值 为 | | ttlChangeOnWrite | ttl 到期时间是否伴随表的修改操作改变; 0: 不改变，1：改变 ；默认值 为 |
| tmqMaxTopicNum | 订阅最多可建立的 topic 数量; 取值范围 1-10000；缺省值 为20 | | tmqMaxTopicNum | 订阅最多可建立的 topic 数量; 取值范围 1-10000；缺省值 为20 |
| maxTsmaNum | 集群内可创建的TSMA个数；取值范围：0-3；缺省值: 3 | | maxTsmaNum | 集群内可创建的TSMA个数；取值范围：0-3；缺省值: 3 |
## taosd 监控指标
taosd 会将监控指标上报给 taosKeeper，这些监控指标会被 taosKeeper 写入监控数据库，默认是 `log` 库，可以在 taoskeeper 配置文件中修改。以下是这些监控指标的详细介绍。
### taosd\_cluster\_basic 表
`taosd_cluster_basic` 表记录集群基础信息。
| field | type | is\_tag | comment |
| :------------------- | :-------- | :------ | :------------------------------ |
| ts | TIMESTAMP | | timestamp |
| first\_ep | VARCHAR | | 集群 first ep |
| first\_ep\_dnode\_id | INT | | 集群 first ep 的 dnode id |
| cluster_version | VARCHAR | | tdengine version。例如：3.0.4.0 |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_cluster\_info 表
`taosd_cluster_info` 表记录集群信息。
| field | type | is\_tag | comment |
| :----------------------- | :-------- | :------ | :----------------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| cluster_uptime | DOUBLE | | 当前 master 节点的uptime。单位：秒 |
| dbs\_total | DOUBLE | | database 总数 |
| tbs\_total | DOUBLE | | 当前集群 table 总数 |
| stbs\_total | DOUBLE | | 当前集群 stable 总数 |
| dnodes\_total | DOUBLE | | 当前集群 dnode 总数 |
| dnodes\_alive | DOUBLE | | 当前集群 dnode 存活总数 |
| mnodes\_total | DOUBLE | | 当前集群 mnode 总数 |
| mnodes\_alive | DOUBLE | | 当前集群 mnode 存活总数 |
| vgroups\_total | DOUBLE | | 当前集群 vgroup 总数 |
| vgroups\_alive | DOUBLE | | 当前集群 vgroup 存活总数 |
| vnodes\_total | DOUBLE | | 当前集群 vnode 总数 |
| vnodes\_alive | DOUBLE | | 当前集群 vnode 存活总数 |
| connections\_total | DOUBLE | | 当前集群连接总数 |
| topics\_total | DOUBLE | | 当前集群 topic 总数 |
| streams\_total | DOUBLE | | 当前集群 stream 总数 |
| grants_expire\_time | DOUBLE | | 认证过期时间，企业版有效，社区版为 DOUBLE 最大值 |
| grants_timeseries\_used | DOUBLE | | 已用测点数 |
| grants_timeseries\_total | DOUBLE | | 总测点数，开源版本为 DOUBLE 最大值 |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_vgroups\_info 表
`taosd_vgroups_info` 表记录虚拟节点组信息。
| field | type | is\_tag | comment |
| :------------- | :-------- | :------ | :--------------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| tables\_num | DOUBLE | | vgroup 中 table 数量 |
| status | DOUBLE | | vgroup 状态, 取值范围：unsynced = 0, ready = 1 |
| vgroup\_id | VARCHAR | TAG | vgroup id |
| database\_name | VARCHAR | TAG | vgroup 所属的 database 名字 |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_dnodes\_info 表
`taosd_dnodes_info` 记录 dnode 信息。
| field | type | is\_tag | comment |
| :---------------- | :-------- | :------ | :------------------------------------------------------------------------------------------------ |
| \_ts | TIMESTAMP | | timestamp |
| uptime | DOUBLE | | dnode uptime，单位：秒 |
| cpu\_engine | DOUBLE | | taosd cpu 使用率，从 `/proc/<taosd_pid>/stat` 读取 |
| cpu\_system | DOUBLE | | 服务器 cpu 使用率，从 `/proc/stat` 读取 |
| cpu\_cores | DOUBLE | | 服务器 cpu 核数 |
| mem\_engine | DOUBLE | | taosd 内存使用率，从 `/proc/<taosd_pid>/status` 读取 |
| mem\_free | DOUBLE | | 服务器可用内存，单位 KB |
| mem\_total | DOUBLE | | 服务器内存总量，单位 KB |
| disk\_used | DOUBLE | | data dir 挂载的磁盘使用量，单位 bytes |
| disk\_total | DOUBLE | | data dir 挂载的磁盘总容量，单位 bytes |
| system\_net\_in | DOUBLE | | 网络吞吐率，从 `/proc/net/dev` 中读取的 received bytes。单位 byte/s |
| system\_net\_out | DOUBLE | | 网络吞吐率，从 `/proc/net/dev` 中读取的 transmit bytes。单位 byte/s |
| io\_read | DOUBLE | | io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 rchar 与上次数值计算之后，计算得到速度。单位 byte/s |
| io\_write | DOUBLE | | io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 wchar 与上次数值计算之后，计算得到速度。单位 byte/s |
| io\_read\_disk | DOUBLE | | 磁盘 io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 read_bytes。单位 byte/s |
| io\_write\_disk | DOUBLE | | 磁盘 io 吞吐率，从 `/proc/<taosd_pid>/io` 中读取的 write_bytes。单位 byte/s |
| vnodes\_num | DOUBLE | | dnode 上 vnodes 数量 |
| masters | DOUBLE | | dnode 上 master node 数量 |
| has\_mnode | DOUBLE | | dnode 是否包含 mnode，取值范围：包含=1,不包含=0 |
| has\_qnode | DOUBLE | | dnode 是否包含 qnode，取值范围：包含=1,不包含=0 |
| has\_snode | DOUBLE | | dnode 是否包含 snode，取值范围：包含=1,不包含=0 |
| has\_bnode | DOUBLE | | dnode 是否包含 bnode，取值范围：包含=1,不包含=0 |
| error\_log\_count | DOUBLE | | error 总数 |
| info\_log\_count | DOUBLE | | info 总数 |
| debug\_log\_count | DOUBLE | | debug 总数 |
| trace\_log\_count | DOUBLE | | trace 总数 |
| dnode\_id | VARCHAR | TAG | dnode id |
| dnode\_ep | VARCHAR | TAG | dnode endpoint |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_dnodes\_status 表
`taosd_dnodes_status` 表记录 dnode 状态信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :--------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| status | DOUBLE | | dnode 状态,取值范围：ready=1，offline =0 |
| dnode\_id | VARCHAR | TAG | dnode id |
| dnode\_ep | VARCHAR | TAG | dnode endpoint |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_dnodes\_log\_dir 表
`taosd_dnodes_log_dir` 表记录 log 目录信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :---------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| avail | DOUBLE | | log 目录可用空间。单位 byte |
| used | DOUBLE | | log 目录已使用空间。单位 byte |
| total | DOUBLE | | log 目录空间。单位 byte |
| name | VARCHAR | TAG | log 目录名，一般为 `/var/log/taos/` |
| dnode\_id | VARCHAR | TAG | dnode id |
| dnode\_ep | VARCHAR | TAG | dnode endpoint |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_dnodes\_data\_dir 表
`taosd_dnodes_data_dir` 表记录 data 目录信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :-------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| avail | DOUBLE | | data 目录可用空间。单位 byte |
| used | DOUBLE | | data 目录已使用空间。单位 byte |
| total | DOUBLE | | data 目录空间。单位 byte |
| level | VARCHAR | TAG | 0、1、2 多级存储级别 |
| name | VARCHAR | TAG | data 目录，一般为 `/var/lib/taos` |
| dnode\_id | VARCHAR | TAG | dnode id |
| dnode\_ep | VARCHAR | TAG | dnode endpoint |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_mnodes\_info 表
`taosd_mnodes_info` 表记录 mnode 角色信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :------------------------------------------------------------------------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| role | DOUBLE | | mnode 角色， 取值范围：offline = 0,follower = 100,candidate = 101,leader = 102,error = 103,learner = 104 |
| mnode\_id | VARCHAR | TAG | master node id |
| mnode\_ep | VARCHAR | TAG | master node endpoint |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_vnodes\_role 表
`taosd_vnodes_role` 表记录虚拟节点角色信息。
| field | type | is\_tag | comment |
| :------------- | :-------- | :------ | :------------------------------------------------------------------------------------------------------ |
| \_ts | TIMESTAMP | | timestamp |
| vnode\_role | DOUBLE | | vnode 角色，取值范围：offline = 0,follower = 100,candidate = 101,leader = 102,error = 103,learner = 104 |
| vgroup\_id | VARCHAR | TAG | dnode id |
| dnode\_id | VARCHAR | TAG | dnode id |
| database\_name | VARCHAR | TAG | vgroup 所属的 database 名字 |
| cluster\_id | VARCHAR | TAG | cluster id |
### taosd\_sql\_req 表
`taosd_sql_req` 记录服务端 sql 请求信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :--------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| count | DOUBLE | | sql 数量 |
| result | VARCHAR | TAG | sql的执行结果，取值范围：Success, Failed |
| username | VARCHAR | TAG | 执行sql的user name |
| sql\_type | VARCHAR | TAG | sql类型，取值范围：inserted_rows |
| dnode\_id | VARCHAR | TAG | dnode id |
| dnode\_ep | VARCHAR | TAG | dnode endpoint |
| vgroup\_id | VARCHAR | TAG | dnode id |
| cluster\_id | VARCHAR | TAG | cluster id |
### taos\_sql\_req 表
`taos_sql_req` 记录客户端 sql 请求信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :---------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| count | DOUBLE | | sql 数量 |
| result | VARCHAR | TAG | sql的执行结果，取值范围：Success, Failed |
| username | VARCHAR | TAG | 执行sql的user name |
| sql\_type | VARCHAR | TAG | sql类型，取值范围：select, insert，delete |
| cluster\_id | VARCHAR | TAG | cluster id |
### taos\_slow\_sql 表
`taos_slow_sql` 记录客户端慢查询信息。
| field | type | is\_tag | comment |
| :---------- | :-------- | :------ | :---------------------------------------------------- |
| \_ts | TIMESTAMP | | timestamp |
| count | DOUBLE | | sql 数量 |
| result | VARCHAR | TAG | sql的执行结果，取值范围：Success, Failed |
| username | VARCHAR | TAG | 执行sql的user name |
| duration | VARCHAR | TAG | sql执行耗时，取值范围：3-10s,10-100s,100-1000s,1000s- |
| cluster\_id | VARCHAR | TAG | cluster id |

47
docs/zh/14-reference/01-components/03-taosadapter.md
View File

@ -289,31 +289,32 @@ http 返回内容：
## taosAdapter 监控指标 ## taosAdapter 监控指标
taosAdapter 采集 http 相关指标、CPU 百分比和内存百分比。 taosAdapter 采集 REST/Websocket 相关请求的监控指标。将监控指标上报给 taosKeeper，这些监控指标会被 taosKeeper 写入监控数据库，默认是 `log` 库，可以在 taoskeeper 配置文件中修改。以下是这些监控指标的详细介绍。
### http 接口 #### adapter\_requests 表
提供符合 [OpenMetrics](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md) 接口： `adapter_requests` 记录 taosadapter 监控数据。
```text | field | type | is\_tag | comment |
http://<fqdn>:6041/metrics | :----------------- | :----------- | :------ | :---------------------------------- |
``` | ts | TIMESTAMP | | timestamp |
| total | INT UNSIGNED | | 总请求数 |
### 写入 TDengine | query | INT UNSIGNED | | 查询请求数 |
| write | INT UNSIGNED | | 写入请求数 |
taosAdapter 支持将 http 监控、CPU 百分比和内存百分比写入 TDengine。 | other | INT UNSIGNED | | 其他请求数 |
| in\_process | INT UNSIGNED | | 正在处理请求数 |
有关配置参数 | success | INT UNSIGNED | | 成功请求数 |
| fail | INT UNSIGNED | | 失败请求数 |
| **配置项** | **描述** | **默认值** | | query\_success | INT UNSIGNED | | 查询成功请求数 |
|-------------------------|--------------------------------------------|----------| | query\_fail | INT UNSIGNED | | 查询失败请求数 |
| monitor.collectDuration | CPU 和内存采集间隔 | 3s | | write\_success | INT UNSIGNED | | 写入成功请求数 |
| monitor.identity | 当前taosadapter 的标识符如果不设置将使用 'hostname:port' | | | write\_fail | INT UNSIGNED | | 写入失败请求数 |
| monitor.incgroup | 是否是 cgroup 中运行(容器中运行设置为 true) | false | | other\_success | INT UNSIGNED | | 其他成功请求数 |
| monitor.writeToTD | 是否写入到 TDengine | false | | other\_fail | INT UNSIGNED | | 其他失败请求数 |
| monitor.user | TDengine 连接用户名 | root | | query\_in\_process | INT UNSIGNED | | 正在处理查询请求数 |
| monitor.password | TDengine 连接密码 | taosdata | | write\_in\_process | INT UNSIGNED | | 正在处理写入请求数 |
| monitor.writeInterval | 写入TDengine 间隔 | 30s | | endpoint | VARCHAR | | 请求端点 |
| req\_type | NCHAR | TAG | 请求类型：0 为 REST，1 为 Websocket |
## 结果返回条数限制 ## 结果返回条数限制
@ -343,7 +344,7 @@ taosAdapter 从 3.0.4.0 版本开始，提供参数 `smlAutoCreateDB` 来控制
在 TDengine server 2.2.x.x 或更早期版本中，taosd 进程包含一个内嵌的 http 服务。如前面所述，taosAdapter 是一个使用 systemd 管理的独立软件，拥有自己的进程。并且两者有一些配置参数和行为是不同的，请见下表： 在 TDengine server 2.2.x.x 或更早期版本中，taosd 进程包含一个内嵌的 http 服务。如前面所述，taosAdapter 是一个使用 systemd 管理的独立软件，拥有自己的进程。并且两者有一些配置参数和行为是不同的，请见下表：
| **#** | **embedded httpd** | **taosAdapter** | **comment** | | **#** | **embedded httpd** | **taosAdapter** | **comment** |
|-------|---------------------|-------------------------------|------------------------------------------------------------------------------------------------| | ----- | ------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | httpEnableRecordSql | --logLevel=debug | | | 1 | httpEnableRecordSql | --logLevel=debug | |
| 2 | httpMaxThreads | n/a | taosAdapter 自动管理线程池，无需此参数 | | 2 | httpMaxThreads | n/a | taosAdapter 自动管理线程池，无需此参数 |
| 3 | telegrafUseFieldNum | 请参考 taosAdapter telegraf 配置方法 | | 3 | telegrafUseFieldNum | 请参考 taosAdapter telegraf 配置方法 |

129
docs/zh/14-reference/01-components/04-taosx.md
View File

@ -91,7 +91,7 @@ taosx privileges -i ./user-pass-privileges-backup.json -t "taos:///"
可用参数列表： 可用参数列表：
| 参数 | 说明 | | 参数 | 说明 |
| ---- | ---- | | ---- | ------------------------------------ |
| -u | 包含用户基本信息（密码、是否启用等） | | -u | 包含用户基本信息（密码、是否启用等） |
| -p | 包含权限信息 | | -p | 包含权限信息 |
| -w | 包含白名单信息 | | -w | 包含白名单信息 |
@ -325,3 +325,130 @@ Linux 下 `journalctl` 查看日志的命令如下：
```bash ```bash
journalctl -u taosx [-f] journalctl -u taosx [-f]
``` ```
## taosX 监控指标
taosX 会将监控指标上报给 taosKeeper，这些监控指标会被 taosKeeper 写入监控数据库，默认是 `log` 库，可以在 taoskeeper 配置文件中修改。以下是这些监控指标的详细介绍。
### taosX 服务
| 字段 | 描述 |
| -------------------------- | ----------------------------------------------------------------------------- |
| sys_cpu_cores | 系统 CPU 核数 |
| sys_total_memory | 系统总内存，单位：字节 |
| sys_used_memory | 系统已用内存, 单位：字节 |
| sys_available_memory | 系统可用内存, 单位：字节 |
| process_uptime | taosX 运行时长，单位：秒 |
| process_id | taosX 进程 ID |
| running_tasks | taosX 当前执行任务数 |
| completed_tasks | taosX 进程在一个监控周期（比如10s）内完成的任务数 |
| failed_tasks | taosX 进程在一个监控周期（比如10s）内失败的任务数 |
| process_cpu_percent | taosX 进程占用 CPU 百分比， 单位 % |
| process_memory_percent | taosX 进程占用内存百分比， 单位 % |
| process_disk_read_bytes | taosX 进程在一个监控周期（比如10s）内从硬盘读取的字节数的平均值，单位 bytes/s |
| process_disk_written_bytes | taosX 进程在一个监控周期（比如10s）内写到硬盘的字节数的平均值，单位 bytres/s |
### Agent
| 字段 | 描述 |
| -------------------------- | ----------------------------------------------------------------------------- |
| sys_cpu_cores | 系统 CPU 核数 |
| sys_total_memory | 系统总内存，单位：字节 |
| sys_used_memory | 系统已用内存, 单位：字节 |
| sys_available_memory | 系统可用内存, 单位：字节 |
| process_uptime | agent 运行时长，单位：秒 |
| process_id | agent 进程 id |
| process_cpu_percent | agent 进程占用 CPU 百分比 |
| process_memory_percent | agent 进程占用内存百分比 |
| process_uptime | 进程启动时间，单位秒 |
| process_disk_read_bytes | agent 进程在一个监控周期（比如10s）内从硬盘读取的字节数的平均值，单位 bytes/s |
| process_disk_written_bytes | agent 进程在一个监控周期（比如10s）内写到硬盘的字节数的平均值，单位 bytes/s |
### Connector
| 字段 | 描述 |
| -------------------------- | --------------------------------------------------------------------------------- |
| process_id | connector 进程 id |
| process_uptime | 进程启动时间，单位秒 |
| process_cpu_percent | 进程占用 CPU 百分比， 单位 % |
| process_memory_percent | 进程占用内存百分比， 单位 % |
| process_disk_read_bytes | connector 进程在一个监控周期（比如10s）内从硬盘读取的字节数的平均值，单位 bytes/s |
| process_disk_written_bytes | connector 进程在一个监控周期（比如10s）内写到硬盘的字节数的平均值，单位 bytes/s |
### taosX 通用数据源任务
| 字段 | 描述 |
| -------------------- | --------------------------------------------------------------- |
| total_execute_time | 任务累计运行时间，单位毫秒 |
| total_written_rowsls | 成功写入 TDengine 的总行数（包括重复记录） |
| total_written_points | 累计写入成功点数 (等于数据块包含的行数乘以数据块包含的列数) |
| start_time | 任务启动时间 (每次重启任务会被重置) |
| written_rows | 本次运行此任务成功写入 TDengine 的总行数（包括重复记录） |
| written_points | 本次运行写入成功点数 (等于数据块包含的行数乘以数据块包含的列数) |
| execute_time | 任务本次运行时间，单位秒 |
### taosX TDengine V2 任务
| 字段 | 描述 |
| --------------------- | -------------------------------------------------------------------- |
| read_concurrency | 并发读取数据源的数据 worker 数, 也等于并发写入 TDengine 的 worker 数 |
| total_stables | 需要迁移的超级表数据数量 |
| total_updated_tags | 累计更新 tag 数 |
| total_created_tables | 累计创建子表数 |
| total_tables | 需要迁移的子表数量 |
| total_finished_tables | 完成数据迁移的子表数 (任务中断重启可能大于实际值) |
| total_success_blocks | 累计写入成功的数据块数 |
| finished_tables | 本次运行完成迁移子表数 |
| success_blocks | 本次写入成功的数据块数 |
| created_tables | 本次运行创建子表数 |
| updated_tags | 本次运行更新 tag 数 |
### taosX TDengine V3 任务
| 字段 | 描述 |
| ---------------------- | ------------------------------------------------------- |
| total_messages | 通过 TMQ 累计收到的消息总数 |
| total_messages_of_meta | 通过 TMQ 累计收到的 Meta 类型的消息总数 |
| total_messages_of_data | 通过 TMQ 累计收到的 Data 和 MetaData 类型的消息总数 |
| total_write_raw_fails | 累计写入 raw meta 失败的次数 |
| total_success_blocks | 累计写入成功的数据块数 |
| topics | 通过 TMQ 订阅的主题数 |
| consumers | TMQ 消费者数 |
| messages | 本次运行通过 TMQ 收到的消息总数 |
| messages_of_meta | 本次运行通过 TMQ 收到的 Meta 类型的消息总数 |
| messages_of_data | 本次运行通过 TMQ 收到的 Data 和 MetaData 类型的消息总数 |
| write_raw_fails | 本次运行写入 raw meta 失败的次数 |
| success_blocks | 本次写入成功的数据块数 |
### taosX 其他数据源 任务
这些数据源包括： InfluxDB，OpenTSDB，OPC UA，OPC DA，PI，CSV，MQTT，AVEVA Historian 和 Kafka。
| 字段 | 描述 |
| ----------------------- | ----------------------------------------------------------- |
| total_received_batches | 通过 IPC Stream 收到的数据总批数 |
| total_processed_batches | 已经处理的批数 |
| total_processed_rows | 已经处理的总行数（等于每批包含数据行数之和） |
| total_inserted_sqls | 执行的 INSERT SQL 总条数 |
| total_failed_sqls | 执行失败的 INSERT SQL 总条数 |
| total_created_stables | 创建的超级表总数（可能大于实际值） |
| total_created_tables | 尝试创建子表总数(可能大于实际值) |
| total_failed_rows | 写入失败的总行数 |
| total_failed_point | 写入失败的总点数 |
| total_written_blocks | 写入成功的 raw block 总数 |
| total_failed_blocks | 写入失败的 raw block 总数 |
| received_batches | 本次运行此任务通过 IPC Stream 收到的数据总批数 |
| processed_batches | 本次运行已处理批数 |
| processed_rows | 本次处理的总行数（等于包含数据的 batch 包含的数据行数之和） |
| received_records | 本次运行此任务通过 IPC Stream 收到的数据总行数 |
| inserted_sqls | 本次运行此任务执行的 INSERT SQL 总条数 |
| failed_sqls | 本次运行此任务执行失败的 INSERT SQL 总条数 |
| created_stables | 本次运行此任务尝试创建超级表数（可能大于实际值） |
| created_tables | 本次运行此任务尝试创建子表数(可能大于实际值) |
| failed_rows | 本次运行此任务写入失败的行数 |
| failed_points | 本次运行此任务写入失败的点数 |
| written_blocks | 本次运行此任务写人成功的 raw block 数 |
| failed_blocks | 本次运行此任务写入失败的 raw block 数 |

16
docs/zh/14-reference/01-components/06-taoskeeper.md
View File

@ -317,3 +317,19 @@ scrape_configs:
在 Grafana Dashboard 菜单点击 `import`，dashboard ID 填写 `18587`，点击 `Load` 按钮即可导入 `TaosKeeper Prometheus Dashboard for 3.x` dashboard。 在 Grafana Dashboard 菜单点击 `import`，dashboard ID 填写 `18587`，点击 `Load` 按钮即可导入 `TaosKeeper Prometheus Dashboard for 3.x` dashboard。
## taosKeeper 监控指标
taosKeeper 也会将自己采集的监控数据写入监控数据库，默认是 `log` 库，可以在 taoskeeper 配置文件中修改。
### keeper\_monitor 表
`keeper_monitor` 记录 taoskeeper 监控数据。
| field | type | is\_tag | comment |
| :------- | :-------- | :------ | :----------- |
| ts | TIMESTAMP | | timestamp |
| cpu | DOUBLE | | cpu 使用率 |
| mem | DOUBLE | | 内存使用率 |
| identify | NCHAR | TAG | 身份标识信息 |

4
docs/zh/14-reference/01-components/12-tdinsight/index.mdx
View File

@ -15,8 +15,8 @@ TDengine 通过 taosKeeper 将服务器的 CPU、内存、硬盘空间、带宽
首先检查下面服务： 首先检查下面服务：
- TDengine 已经安装并正常运行，此仪表盘需要 TDengine 3.0.0.0 及以上，并开启监控上报配置，具体配置请参考：[TDengine 监控配置](../taosd/#监控相关)。 - TDengine 已经安装并正常运行，此仪表盘需要 TDengine 3.0.0.0 及以上，并开启监控上报配置，具体配置请参考：[TDengine 监控配置](../taosd/#监控相关)。
- taosAdapter 已经安装并正常运行。具体细节请参考：[taosAdapter 使用手册](../taosadapter) - taosAdapter 已经安装并正常运行。具体细节请参考：[taosAdapter 参考手册](../taosadapter)
- taosKeeper 已安装并正常运行。具体细节请参考：[taosKeeper 使用手册](../taoskeeper) - taosKeeper 已安装并正常运行。具体细节请参考：[taosKeeper 参考手册](../taoskeeper)
然后记录以下信息： 然后记录以下信息：

11
docs/zh/14-reference/01-components/index.md
View File

@ -1,6 +1,13 @@
--- ---
sidebar_label: 产品组件
title: 产品组件 title: 产品组件
toc_max_heading_level: 4
description: TDengine 产品组件参考手册 description: TDengine 产品组件参考手册
--- ---
本节详细说明 TDengine 中的主要产品组件的功能、命令行参数、配置参数等。
```mdx-code-block
import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
<DocCardList items={useCurrentSidebarCategory().items}/>
```

6
docs/zh/14-reference/05-connector/index.md
View File

@ -118,3 +118,9 @@ import VerifyMacOS from "./_verify_macos.mdx";
</TabItem> </TabItem>
</Tabs> </Tabs>
```mdx-code-block
import DocCardList from '@theme/DocCardList';
import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
<DocCardList items={useCurrentSidebarCategory().items}/>
```

2
docs/zh/26-tdinternal/01-arch.md
View File

@ -301,7 +301,7 @@ TDengine 采用了一种数据驱动的策略来实现缓存数据的持久化
对于采集的数据，通常会有一定的保留期限，该期限由数据库参数 keep 指定。超出设定天数的数据文件将被集群自动移除，并释放相应的存储空间。 对于采集的数据，通常会有一定的保留期限，该期限由数据库参数 keep 指定。超出设定天数的数据文件将被集群自动移除，并释放相应的存储空间。
当设置 duration 和 keep 两个参数后，一个处于典型工作状态的 vnode 中，总的数据文件数量应为向上取整 (keep/duration)+1 个。数据文件的总个数应保持在一个合理的范围内，不宜过多也不宜过少，通常介于 10 到 100 之间较为适宜。基于这一原则，可以合理设置 duration 参数。在本书编写时的版本中，可以调整参数 keep，但参数 duration 一旦设定，则无法更改。 当设置 duration 和 keep 两个参数后，一个处于典型工作状态的 vnode 中，总的数据文件数量应为向上取整 (keep/duration)+1 个。数据文件的总个数应保持在一个合理的范围内，不宜过多也不宜过少，通常介于 10 到 100 之间较为适宜。基于这一原则，可以合理设置 duration 参数。可以调整参数 keep，但参数 duration 一旦设定，则无法更改。
在每个数据文件中，表的数据是以块的形式存储的。一张表可能包含一到多个数据文件块。在一个文件块内，数据采用列式存储，占据连续的存储空间，这有助于显著提高读取度。文件块的大小由数据库参数 maxRows（每块最大记录条数）控制，默认值为 4096。这个值应适中，过大可能导致定位特定时间段数据的搜索时间变长，影响读取速度；过小则可能导致数据文件块的索引过大，压缩效率降低，同样影响读取速度。 在每个数据文件中，表的数据是以块的形式存储的。一张表可能包含一到多个数据文件块。在一个文件块内，数据采用列式存储，占据连续的存储空间，这有助于显著提高读取度。文件块的大小由数据库参数 maxRows（每块最大记录条数）控制，默认值为 4096。这个值应适中，过大可能导致定位特定时间段数据的搜索时间变长，影响读取速度；过小则可能导致数据文件块的索引过大，压缩效率降低，同样影响读取速度。

5
include/client/taos.h
View File

@ -270,7 +270,10 @@ DLL_EXPORT TAOS_RES *taos_schemaless_insert_raw_ttl(TAOS *taos, char *lines, int
int precision, int32_t ttl); int precision, int32_t ttl);
DLL_EXPORT TAOS_RES *taos_schemaless_insert_raw_ttl_with_reqid(TAOS *taos, char *lines, int len, int32_t *totalRows, DLL_EXPORT TAOS_RES *taos_schemaless_insert_raw_ttl_with_reqid(TAOS *taos, char *lines, int len, int32_t *totalRows,
int protocol, int precision, int32_t ttl, int64_t reqid); int protocol, int precision, int32_t ttl, int64_t reqid);
DLL_EXPORT TAOS_RES *taos_schemaless_insert_raw_ttl_with_reqid_tbname_key(TAOS *taos, char *lines, int len, int32_t *totalRows,
int protocol, int precision, int32_t ttl, int64_t reqid, char *tbnameKey);
DLL_EXPORT TAOS_RES *taos_schemaless_insert_ttl_with_reqid_tbname_key(TAOS *taos, char *lines[], int numLines, int protocol,
int precision, int32_t ttl, int64_t reqid, char *tbnameKey);
/* --------------------------TMQ INTERFACE------------------------------- */ /* --------------------------TMQ INTERFACE------------------------------- */
typedef struct tmq_t tmq_t; typedef struct tmq_t tmq_t;

2
include/libs/executor/storageapi.h
View File

@ -248,7 +248,7 @@ typedef struct SStoreMeta {
SMTbCursor* (*openTableMetaCursor)(void* pVnode); // metaOpenTbCursor SMTbCursor* (*openTableMetaCursor)(void* pVnode); // metaOpenTbCursor
void (*closeTableMetaCursor)(SMTbCursor* pTbCur); // metaCloseTbCursor void (*closeTableMetaCursor)(SMTbCursor* pTbCur); // metaCloseTbCursor
void (*pauseTableMetaCursor)(SMTbCursor* pTbCur); // metaPauseTbCursor void (*pauseTableMetaCursor)(SMTbCursor* pTbCur); // metaPauseTbCursor
void (*resumeTableMetaCursor)(SMTbCursor* pTbCur, int8_t first, int8_t move); // metaResumeTbCursor int32_t (*resumeTableMetaCursor)(SMTbCursor* pTbCur, int8_t first, int8_t move); // metaResumeTbCursor
int32_t (*cursorNext)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorNext int32_t (*cursorNext)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorNext
int32_t (*cursorPrev)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorPrev int32_t (*cursorPrev)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorPrev

24
include/libs/function/taosudf.h
View File

@ -320,6 +320,30 @@ typedef int32_t (*TScriptUdfDestoryFunc)(void *udfCtx);
typedef int32_t (*TScriptOpenFunc)(SScriptUdfEnvItem *items, int numItems); typedef int32_t (*TScriptOpenFunc)(SScriptUdfEnvItem *items, int numItems);
typedef int32_t (*TScriptCloseFunc)(); typedef int32_t (*TScriptCloseFunc)();
// clang-format off
#ifdef WINDOWS
#define fnFatal(...) {}
#define fnError(...) {}
#define fnWarn(...) {}
#define fnInfo(...) {}
#define fnDebug(...) {}
#define fnTrace(...) {}
#else
DLL_EXPORT void taosPrintLog(const char *flags, int32_t level, int32_t dflag, const char *format, ...)
#ifdef __GNUC__
__attribute__((format(printf, 4, 5)))
#endif
;
extern int32_t udfDebugFlag;
#define udfFatal(...) { if (udfDebugFlag & 1) { taosPrintLog("UDF FATAL ", 1, 255, __VA_ARGS__); }}
#define udfError(...) { if (udfDebugFlag & 1) { taosPrintLog("UDF ERROR ", 1, 255, __VA_ARGS__); }}
#define udfWarn(...) { if (udfDebugFlag & 2) { taosPrintLog("UDF WARN ", 2, 255, __VA_ARGS__); }}
#define udfInfo(...) { if (udfDebugFlag & 2) { taosPrintLog("UDF ", 2, 255, __VA_ARGS__); }}
#define udfDebug(...) { if (udfDebugFlag & 4) { taosPrintLog("UDF ", 4, udfDebugFlag, __VA_ARGS__); }}
#define udfTrace(...) { if (udfDebugFlag & 8) { taosPrintLog("UDF ", 8, udfDebugFlag, __VA_ARGS__); }}
#endif
// clang-format on
#ifdef __cplusplus #ifdef __cplusplus
} }
#endif #endif

1
include/util/tcompare.h
View File

@ -49,6 +49,7 @@ int32_t InitRegexCache();
void DestroyRegexCache(); void DestroyRegexCache();
int32_t patternMatch(const char *pattern, size_t psize, const char *str, size_t ssize, const SPatternCompareInfo *pInfo); int32_t patternMatch(const char *pattern, size_t psize, const char *str, size_t ssize, const SPatternCompareInfo *pInfo);
int32_t checkRegexPattern(const char *pPattern); int32_t checkRegexPattern(const char *pPattern);
void DestoryThreadLocalRegComp();
int32_t wcsPatternMatch(const TdUcs4 *pattern, size_t psize, const TdUcs4 *str, size_t ssize, const SPatternCompareInfo *pInfo); int32_t wcsPatternMatch(const TdUcs4 *pattern, size_t psize, const TdUcs4 *str, size_t ssize, const SPatternCompareInfo *pInfo);

4
include/util/tlog.h
View File

@ -74,13 +74,13 @@ void taosCloseLog();
void taosResetLog(); void taosResetLog();
void taosDumpData(uint8_t *msg, int32_t len); void taosDumpData(uint8_t *msg, int32_t len);
void taosPrintLog(const char *flags, ELogLevel level, int32_t dflag, const char *format, ...) void taosPrintLog(const char *flags, int32_t level, int32_t dflag, const char *format, ...)
#ifdef __GNUC__ #ifdef __GNUC__
__attribute__((format(printf, 4, 5))) __attribute__((format(printf, 4, 5)))
#endif #endif
; ;
void taosPrintLongString(const char *flags, ELogLevel level, int32_t dflag, const char *format, ...) void taosPrintLongString(const char *flags, int32_t level, int32_t dflag, const char *format, ...)
#ifdef __GNUC__ #ifdef __GNUC__
__attribute__((format(printf, 4, 5))) __attribute__((format(printf, 4, 5)))
#endif #endif

4
source/client/inc/clientSml.h
View File

@ -204,6 +204,7 @@ typedef struct {
STableMeta *currSTableMeta; STableMeta *currSTableMeta;
STableDataCxt *currTableDataCtx; STableDataCxt *currTableDataCtx;
bool needModifySchema; bool needModifySchema;
char *tbnameKey;
} SSmlHandle; } SSmlHandle;
extern int64_t smlFactorNS[]; extern int64_t smlFactorNS[];
@ -219,9 +220,10 @@ bool smlParseNumberOld(SSmlKv *kvVal, SSmlMsgBuf *msg);
void smlBuildInvalidDataMsg(SSmlMsgBuf *pBuf, const char *msg1, const char *msg2); void smlBuildInvalidDataMsg(SSmlMsgBuf *pBuf, const char *msg1, const char *msg2);
int32_t smlParseNumber(SSmlKv *kvVal, SSmlMsgBuf *msg); int32_t smlParseNumber(SSmlKv *kvVal, SSmlMsgBuf *msg);
int64_t smlGetTimeValue(const char *value, int32_t len, uint8_t fromPrecision, uint8_t toPrecision); int64_t smlGetTimeValue(const char *value, int32_t len, uint8_t fromPrecision, uint8_t toPrecision);
int32_t smlBuildTableInfo(int numRows, const char* measure, int32_t measureLen, SSmlTableInfo** tInfo); int32_t smlBuildTableInfo(int numRows, const char* measure, int32_t measureLen, SSmlTableInfo** tInfo);
int32_t smlBuildSTableMeta(bool isDataFormat, SSmlSTableMeta** sMeta); int32_t smlBuildSTableMeta(bool isDataFormat, SSmlSTableMeta** sMeta);
int32_t smlSetCTableName(SSmlTableInfo *oneTable); int32_t smlSetCTableName(SSmlTableInfo *oneTable, char *tbnameKey);
int32_t getTableUid(SSmlHandle *info, SSmlLineInfo *currElement, SSmlTableInfo *tinfo); int32_t getTableUid(SSmlHandle *info, SSmlLineInfo *currElement, SSmlTableInfo *tinfo);
int32_t smlGetMeta(SSmlHandle *info, const void* measure, int32_t measureLen, STableMeta **pTableMeta); int32_t smlGetMeta(SSmlHandle *info, const void* measure, int32_t measureLen, STableMeta **pTableMeta);
int32_t is_same_child_table_telnet(const void *a, const void *b); int32_t is_same_child_table_telnet(const void *a, const void *b);

6
source/client/src/clientEnv.c
View File

@ -374,8 +374,8 @@ int32_t openTransporter(const char *user, const char *auth, int32_t numOfThread,
*pDnodeConn = rpcOpen(&rpcInit); *pDnodeConn = rpcOpen(&rpcInit);
if (*pDnodeConn == NULL) { if (*pDnodeConn == NULL) {
tscError("failed to init connection to server."); tscError("failed to init connection to server since %s", tstrerror(terrno));
code = TSDB_CODE_FAILED; code = terrno;
} }
return code; return code;
@ -871,7 +871,7 @@ _return:
TSC_ERR_RET(terrno); TSC_ERR_RET(terrno);
} }
return TSDB_CODE_SUCCESS; return code;
} }
void tscStopCrashReport() { void tscStopCrashReport() {

2
source/client/src/clientHb.c
View File

@ -1405,7 +1405,7 @@ _return:
TSC_ERR_RET(terrno); TSC_ERR_RET(terrno);
} }
return TSDB_CODE_SUCCESS; return code;
} }
static void hbStopThread() { static void hbStopThread() {

3
source/client/src/clientImpl.c
View File

@ -2467,7 +2467,8 @@ TSDB_SERVER_STATUS taos_check_server_status(const char* fqdn, int port, char* de
clientRpc = rpcOpen(&rpcInit); clientRpc = rpcOpen(&rpcInit);
if (clientRpc == NULL) { if (clientRpc == NULL) {
tscError("failed to init server status client"); code = terrno;
tscError("failed to init server status client since %s", tstrerror(code));
goto _OVER; goto _OVER;
} }

36
source/client/src/clientSml.c
View File

@ -402,7 +402,7 @@ int32_t smlProcessChildTable(SSmlHandle *info, SSmlLineInfo *elements) {
if (kv->valueEscaped) kv->value = NULL; if (kv->valueEscaped) kv->value = NULL;
} }
code = smlSetCTableName(tinfo); code = smlSetCTableName(tinfo, info->tbnameKey);
if (code != TSDB_CODE_SUCCESS){ if (code != TSDB_CODE_SUCCESS){
smlDestroyTableInfo(&tinfo); smlDestroyTableInfo(&tinfo);
return code; return code;
@ -486,10 +486,10 @@ int32_t smlParseEndLine(SSmlHandle *info, SSmlLineInfo *elements, SSmlKv *kvTs)
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static int32_t smlParseTableName(SArray *tags, char *childTableName) { static int32_t smlParseTableName(SArray *tags, char *childTableName, char *tbnameKey) {
bool autoChildName = false; bool autoChildName = false;
size_t delimiter = strlen(tsSmlAutoChildTableNameDelimiter); size_t delimiter = strlen(tsSmlAutoChildTableNameDelimiter);
if (delimiter > 0) { if(delimiter > 0 && tbnameKey == NULL){
size_t totalNameLen = delimiter * (taosArrayGetSize(tags) - 1); size_t totalNameLen = delimiter * (taosArrayGetSize(tags) - 1);
for (int i = 0; i < taosArrayGetSize(tags); i++) { for (int i = 0; i < taosArrayGetSize(tags); i++) {
SSmlKv *tag = (SSmlKv *)taosArrayGet(tags, i); SSmlKv *tag = (SSmlKv *)taosArrayGet(tags, i);
@ -518,7 +518,10 @@ static int32_t smlParseTableName(SArray *tags, char *childTableName) {
smlStrReplace(childTableName, strlen(childTableName)); smlStrReplace(childTableName, strlen(childTableName));
} }
}else{ }else{
size_t childTableNameLen = strlen(tsSmlChildTableName); if (tbnameKey == NULL){
tbnameKey = tsSmlChildTableName;
}
size_t childTableNameLen = strlen(tbnameKey);
if (childTableNameLen <= 0) return TSDB_CODE_SUCCESS; if (childTableNameLen <= 0) return TSDB_CODE_SUCCESS;
for (int i = 0; i < taosArrayGetSize(tags); i++) { for (int i = 0; i < taosArrayGetSize(tags); i++) {
@ -527,7 +530,7 @@ static int32_t smlParseTableName(SArray *tags, char *childTableName) {
return TSDB_CODE_SML_INVALID_DATA; return TSDB_CODE_SML_INVALID_DATA;
} }
// handle child table name // handle child table name
if (childTableNameLen == tag->keyLen && strncmp(tag->key, tsSmlChildTableName, tag->keyLen) == 0) { if (childTableNameLen == tag->keyLen && strncmp(tag->key, tbnameKey, tag->keyLen) == 0) {
(void)memset(childTableName, 0, TSDB_TABLE_NAME_LEN); (void)memset(childTableName, 0, TSDB_TABLE_NAME_LEN);
(void)strncpy(childTableName, tag->value, (tag->length < TSDB_TABLE_NAME_LEN ? tag->length : TSDB_TABLE_NAME_LEN)); (void)strncpy(childTableName, tag->value, (tag->length < TSDB_TABLE_NAME_LEN ? tag->length : TSDB_TABLE_NAME_LEN));
if (tsSmlDot2Underline) { if (tsSmlDot2Underline) {
@ -542,8 +545,8 @@ static int32_t smlParseTableName(SArray *tags, char *childTableName) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
int32_t smlSetCTableName(SSmlTableInfo *oneTable) { int32_t smlSetCTableName(SSmlTableInfo *oneTable, char *tbnameKey) {
int32_t code = smlParseTableName(oneTable->tags, oneTable->childTableName); int32_t code = smlParseTableName(oneTable->tags, oneTable->childTableName, tbnameKey);
if(code != TSDB_CODE_SUCCESS){ if(code != TSDB_CODE_SUCCESS){
return code; return code;
} }
@ -2127,7 +2130,7 @@ void smlSetReqSQL(SRequestObj *request, char *lines[], char *rawLine, char *rawL
} }
TAOS_RES *taos_schemaless_insert_inner(TAOS *taos, char *lines[], char *rawLine, char *rawLineEnd, int numLines, TAOS_RES *taos_schemaless_insert_inner(TAOS *taos, char *lines[], char *rawLine, char *rawLineEnd, int numLines,
int protocol, int precision, int32_t ttl, int64_t reqid) { int protocol, int precision, int32_t ttl, int64_t reqid, char *tbnameKey) {
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
if (NULL == taos) { if (NULL == taos) {
uError("SML:taos_schemaless_insert error taos is null"); uError("SML:taos_schemaless_insert error taos is null");
@ -2159,6 +2162,7 @@ TAOS_RES *taos_schemaless_insert_inner(TAOS *taos, char *lines[], char *rawLine,
info->msgBuf.buf = info->pRequest->msgBuf; info->msgBuf.buf = info->pRequest->msgBuf;
info->msgBuf.len = ERROR_MSG_BUF_DEFAULT_SIZE; info->msgBuf.len = ERROR_MSG_BUF_DEFAULT_SIZE;
info->lineNum = numLines; info->lineNum = numLines;
info->tbnameKey = tbnameKey;
smlSetReqSQL(request, lines, rawLine, rawLineEnd); smlSetReqSQL(request, lines, rawLine, rawLineEnd);
@ -2237,9 +2241,14 @@ end:
* @return TAOS_RES * @return TAOS_RES
*/ */
TAOS_RES *taos_schemaless_insert_ttl_with_reqid_tbname_key(TAOS *taos, char *lines[], int numLines, int protocol,
int precision, int32_t ttl, int64_t reqid, char *tbnameKey){
return taos_schemaless_insert_inner(taos, lines, NULL, NULL, numLines, protocol, precision, ttl, reqid, tbnameKey);
}
TAOS_RES *taos_schemaless_insert_ttl_with_reqid(TAOS *taos, char *lines[], int numLines, int protocol, int precision, TAOS_RES *taos_schemaless_insert_ttl_with_reqid(TAOS *taos, char *lines[], int numLines, int protocol, int precision,
int32_t ttl, int64_t reqid) { int32_t ttl, int64_t reqid) {
return taos_schemaless_insert_inner(taos, lines, NULL, NULL, numLines, protocol, precision, ttl, reqid); return taos_schemaless_insert_ttl_with_reqid_tbname_key(taos, lines, numLines, protocol, precision, ttl, reqid, NULL);
} }
TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLines, int protocol, int precision) { TAOS_RES *taos_schemaless_insert(TAOS *taos, char *lines[], int numLines, int protocol, int precision) {
@ -2272,10 +2281,15 @@ static void getRawLineLen(char *lines, int len, int32_t *totalRows, int protocol
} }
} }
TAOS_RES *taos_schemaless_insert_raw_ttl_with_reqid_tbname_key(TAOS *taos, char *lines, int len, int32_t *totalRows,
int protocol, int precision, int32_t ttl, int64_t reqid, char *tbnameKey){
getRawLineLen(lines, len, totalRows, protocol);
return taos_schemaless_insert_inner(taos, NULL, lines, lines + len, *totalRows, protocol, precision, ttl, reqid, tbnameKey);
}
TAOS_RES *taos_schemaless_insert_raw_ttl_with_reqid(TAOS *taos, char *lines, int len, int32_t *totalRows, int protocol, TAOS_RES *taos_schemaless_insert_raw_ttl_with_reqid(TAOS *taos, char *lines, int len, int32_t *totalRows, int protocol,
int precision, int32_t ttl, int64_t reqid) { int precision, int32_t ttl, int64_t reqid) {
getRawLineLen(lines, len, totalRows, protocol); return taos_schemaless_insert_raw_ttl_with_reqid_tbname_key(taos, lines, len, totalRows, protocol, precision, ttl, reqid, NULL);
return taos_schemaless_insert_inner(taos, NULL, lines, lines + len, *totalRows, protocol, precision, ttl, reqid);
} }
TAOS_RES *taos_schemaless_insert_raw_with_reqid(TAOS *taos, char *lines, int len, int32_t *totalRows, int protocol, TAOS_RES *taos_schemaless_insert_raw_with_reqid(TAOS *taos, char *lines, int len, int32_t *totalRows, int protocol,

2
source/client/src/clientTmq.c
View File

@ -1275,7 +1275,7 @@ tmq_t* tmq_consumer_new(tmq_conf_t* conf, char* errstr, int32_t errstrLen) {
// init semaphore // init semaphore
if (tsem2_init(&pTmq->rspSem, 0, 0) != 0) { if (tsem2_init(&pTmq->rspSem, 0, 0) != 0) {
tscError("consumer:0x %" PRIx64 " setup failed since %s, consumer group %s", pTmq->consumerId, terrstr(), tscError("consumer:0x %" PRIx64 " setup failed since %s, consumer group %s", pTmq->consumerId, tstrerror(TAOS_SYSTEM_ERROR(errno)),
pTmq->groupId); pTmq->groupId);
SET_ERROR_MSG_TMQ("init t_sem failed") SET_ERROR_MSG_TMQ("init t_sem failed")
goto _failed; goto _failed;

14
source/dnode/mgmt/node_mgmt/src/dmTransport.c
View File

@ -393,8 +393,8 @@ int32_t dmInitClient(SDnode *pDnode) {
pTrans->clientRpc = rpcOpen(&rpcInit); pTrans->clientRpc = rpcOpen(&rpcInit);
if (pTrans->clientRpc == NULL) { if (pTrans->clientRpc == NULL) {
dError("failed to init dnode rpc client"); dError("failed to init dnode rpc client since:%s", tstrerror(terrno));
return -1; return terrno;
} }
dDebug("dnode rpc client is initialized"); dDebug("dnode rpc client is initialized");
@ -437,8 +437,8 @@ int32_t dmInitStatusClient(SDnode *pDnode) {
pTrans->statusRpc = rpcOpen(&rpcInit); pTrans->statusRpc = rpcOpen(&rpcInit);
if (pTrans->statusRpc == NULL) { if (pTrans->statusRpc == NULL) {
dError("failed to init dnode rpc status client"); dError("failed to init dnode rpc status client since %s", tstrerror(terrno));
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
dDebug("dnode rpc status client is initialized"); dDebug("dnode rpc status client is initialized");
@ -482,8 +482,8 @@ int32_t dmInitSyncClient(SDnode *pDnode) {
pTrans->syncRpc = rpcOpen(&rpcInit); pTrans->syncRpc = rpcOpen(&rpcInit);
if (pTrans->syncRpc == NULL) { if (pTrans->syncRpc == NULL) {
dError("failed to init dnode rpc sync client"); dError("failed to init dnode rpc sync client since %s", tstrerror(terrno));
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
dDebug("dnode rpc sync client is initialized"); dDebug("dnode rpc sync client is initialized");
@ -532,7 +532,7 @@ int32_t dmInitServer(SDnode *pDnode) {
(void)taosVersionStrToInt(version, &(rpcInit.compatibilityVer)); (void)taosVersionStrToInt(version, &(rpcInit.compatibilityVer));
pTrans->serverRpc = rpcOpen(&rpcInit); pTrans->serverRpc = rpcOpen(&rpcInit);
if (pTrans->serverRpc == NULL) { if (pTrans->serverRpc == NULL) {
dError("failed to init dnode rpc server"); dError("failed to init dnode rpc server since:%s", tstrerror(terrno));
return terrno; return terrno;
} }

2
source/dnode/mnode/impl/inc/mndConsumer.h
View File

@ -46,7 +46,7 @@ const char *mndConsumerStatusName(int status);
#define MND_TMQ_NULL_CHECK(c) \ #define MND_TMQ_NULL_CHECK(c) \
do { \ do { \
if (c == NULL) { \ if (c == NULL) { \
code = TSDB_CODE_OUT_OF_MEMORY; \ code = TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); \
goto END; \ goto END; \
} \ } \
} while (0) } while (0)

6
source/dnode/mnode/impl/src/mndSubscribe.c
View File

@ -882,7 +882,7 @@ static int32_t buildRebOutput(SMnode *pMnode, SMqRebInputObj *rebInput, SMqRebOu
rebInput->oldConsumerNum = 0; rebInput->oldConsumerNum = 0;
code = mndCreateSubscription(pMnode, pTopic, key, &rebOutput->pSub); code = mndCreateSubscription(pMnode, pTopic, key, &rebOutput->pSub);
if (code != 0) { if (code != 0) {
mError("[rebalance] mq rebalance %s failed create sub since %s, ignore", key, terrstr()); mError("[rebalance] mq rebalance %s failed create sub since %s, ignore", key, tstrerror(code));
taosRUnLockLatch(&pTopic->lock); taosRUnLockLatch(&pTopic->lock);
mndReleaseTopic(pMnode, pTopic); mndReleaseTopic(pMnode, pTopic);
return code; return code;
@ -1067,7 +1067,7 @@ static int32_t mndProcessDropCgroupReq(SRpcMsg *pMsg) {
return 0; return 0;
} else { } else {
code = TSDB_CODE_MND_SUBSCRIBE_NOT_EXIST; code = TSDB_CODE_MND_SUBSCRIBE_NOT_EXIST;
mError("topic:%s, cgroup:%s, failed to drop since %s", dropReq.topic, dropReq.cgroup, terrstr()); mError("topic:%s, cgroup:%s, failed to drop since %s", dropReq.topic, dropReq.cgroup, tstrerror(code));
return code; return code;
} }
} }
@ -1075,7 +1075,7 @@ static int32_t mndProcessDropCgroupReq(SRpcMsg *pMsg) {
taosWLockLatch(&pSub->lock); taosWLockLatch(&pSub->lock);
if (taosHashGetSize(pSub->consumerHash) != 0) { if (taosHashGetSize(pSub->consumerHash) != 0) {
code = TSDB_CODE_MND_CGROUP_USED; code = TSDB_CODE_MND_CGROUP_USED;
mError("cgroup:%s on topic:%s, failed to drop since %s", dropReq.cgroup, dropReq.topic, terrstr()); mError("cgroup:%s on topic:%s, failed to drop since %s", dropReq.cgroup, dropReq.topic, tstrerror(code));
goto END; goto END;
} }

6
source/dnode/mnode/impl/src/mndTopic.c
View File

@ -574,7 +574,7 @@ static int32_t mndProcessCreateTopicReq(SRpcMsg *pReq) {
END: END:
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) { if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS) {
mError("failed to create topic:%s since %s", createTopicReq.name, terrstr()); mError("failed to create topic:%s since %s", createTopicReq.name, tstrerror(code));
} }
mndReleaseTopic(pMnode, pTopic); mndReleaseTopic(pMnode, pTopic);
@ -699,7 +699,7 @@ static int32_t mndProcessDropTopicReq(SRpcMsg *pReq) {
tFreeSMDropTopicReq(&dropReq); tFreeSMDropTopicReq(&dropReq);
return 0; return 0;
} else { } else {
mError("topic:%s, failed to drop since %s", dropReq.name, terrstr()); mError("topic:%s, failed to drop since %s", dropReq.name, tstrerror(code));
tFreeSMDropTopicReq(&dropReq); tFreeSMDropTopicReq(&dropReq);
return code; return code;
} }
@ -727,7 +727,7 @@ END:
mndReleaseTopic(pMnode, pTopic); mndReleaseTopic(pMnode, pTopic);
mndTransDrop(pTrans); mndTransDrop(pTrans);
if (code != 0) { if (code != 0) {
mError("topic:%s, failed to drop since %s", dropReq.name, terrstr()); mError("topic:%s, failed to drop since %s", dropReq.name, tstrerror(code));
tFreeSMDropTopicReq(&dropReq); tFreeSMDropTopicReq(&dropReq);
return code; return code;
} }

2
source/dnode/vnode/src/inc/vnodeInt.h
View File

@ -131,7 +131,7 @@ typedef SVCreateTSmaReq SSmaCfg;
SMTbCursor* metaOpenTbCursor(void* pVnode); SMTbCursor* metaOpenTbCursor(void* pVnode);
void metaCloseTbCursor(SMTbCursor* pTbCur); void metaCloseTbCursor(SMTbCursor* pTbCur);
void metaPauseTbCursor(SMTbCursor* pTbCur); void metaPauseTbCursor(SMTbCursor* pTbCur);
void metaResumeTbCursor(SMTbCursor* pTbCur, int8_t first, int8_t move); int32_t metaResumeTbCursor(SMTbCursor* pTbCur, int8_t first, int8_t move);
int32_t metaTbCursorNext(SMTbCursor* pTbCur, ETableType jumpTableType); int32_t metaTbCursorNext(SMTbCursor* pTbCur, ETableType jumpTableType);
int32_t metaTbCursorPrev(SMTbCursor* pTbCur, ETableType jumpTableType); int32_t metaTbCursorPrev(SMTbCursor* pTbCur, ETableType jumpTableType);

43
source/dnode/vnode/src/meta/metaQuery.c
View File

@ -231,6 +231,7 @@ _exit:
#if 1 // =================================================== #if 1 // ===================================================
SMTbCursor *metaOpenTbCursor(void *pVnode) { SMTbCursor *metaOpenTbCursor(void *pVnode) {
SMTbCursor *pTbCur = NULL; SMTbCursor *pTbCur = NULL;
int32_t code;
pTbCur = (SMTbCursor *)taosMemoryCalloc(1, sizeof(*pTbCur)); pTbCur = (SMTbCursor *)taosMemoryCalloc(1, sizeof(*pTbCur));
if (pTbCur == NULL) { if (pTbCur == NULL) {
@ -241,7 +242,12 @@ SMTbCursor *metaOpenTbCursor(void *pVnode) {
// tdbTbcMoveToFirst((TBC *)pTbCur->pDbc); // tdbTbcMoveToFirst((TBC *)pTbCur->pDbc);
pTbCur->pMeta = pVnodeObj->pMeta; pTbCur->pMeta = pVnodeObj->pMeta;
pTbCur->paused = 1; pTbCur->paused = 1;
metaResumeTbCursor(pTbCur, 1, 0); code = metaResumeTbCursor(pTbCur, 1, 0);
if (code) {
terrno = code;
taosMemoryFree(pTbCur);
return NULL;
}
return pTbCur; return pTbCur;
} }
@ -266,28 +272,39 @@ void metaPauseTbCursor(SMTbCursor *pTbCur) {
pTbCur->paused = 1; pTbCur->paused = 1;
} }
} }
void metaResumeTbCursor(SMTbCursor *pTbCur, int8_t first, int8_t move) { int32_t metaResumeTbCursor(SMTbCursor *pTbCur, int8_t first, int8_t move) {
int32_t code = 0;
int32_t lino;
if (pTbCur->paused) { if (pTbCur->paused) {
metaReaderDoInit(&pTbCur->mr, pTbCur->pMeta, META_READER_LOCK); metaReaderDoInit(&pTbCur->mr, pTbCur->pMeta, META_READER_LOCK);
(void)tdbTbcOpen(((SMeta *)pTbCur->pMeta)->pUidIdx, (TBC **)&pTbCur->pDbc, NULL); code = tdbTbcOpen(((SMeta *)pTbCur->pMeta)->pUidIdx, (TBC **)&pTbCur->pDbc, NULL);
TSDB_CHECK_CODE(code, lino, _exit);
if (first) { if (first) {
(void)tdbTbcMoveToFirst((TBC *)pTbCur->pDbc); code = tdbTbcMoveToFirst((TBC *)pTbCur->pDbc);
TSDB_CHECK_CODE(code, lino, _exit);
} else { } else {
int c = 1; int c = 1;
(void)tdbTbcMoveTo(pTbCur->pDbc, pTbCur->pKey, pTbCur->kLen, &c); code = tdbTbcMoveTo(pTbCur->pDbc, pTbCur->pKey, pTbCur->kLen, &c);
TSDB_CHECK_CODE(code, lino, _exit);
if (c == 0) { if (c == 0) {
if (move) tdbTbcMoveToNext(pTbCur->pDbc); if (move) tdbTbcMoveToNext(pTbCur->pDbc);
} else if (c < 0) { } else if (c < 0) {
(void)tdbTbcMoveToPrev(pTbCur->pDbc); code = tdbTbcMoveToPrev(pTbCur->pDbc);
TSDB_CHECK_CODE(code, lino, _exit);
} else { } else {
(void)tdbTbcMoveToNext(pTbCur->pDbc); code = tdbTbcMoveToNext(pTbCur->pDbc);
TSDB_CHECK_CODE(code, lino, _exit);
} }
} }
pTbCur->paused = 0; pTbCur->paused = 0;
} }
_exit:
return code;
} }
int32_t metaTbCursorNext(SMTbCursor *pTbCur, ETableType jumpTableType) { int32_t metaTbCursorNext(SMTbCursor *pTbCur, ETableType jumpTableType) {
@ -355,7 +372,9 @@ _query:
version = ((SUidIdxVal *)pData)[0].version; version = ((SUidIdxVal *)pData)[0].version;
(void)tdbTbGet(pMeta->pTbDb, &(STbDbKey){.uid = uid, .version = version}, sizeof(STbDbKey), &pData, &nData); if (tdbTbGet(pMeta->pTbDb, &(STbDbKey){.uid = uid, .version = version}, sizeof(STbDbKey), &pData, &nData) != 0) {
goto _err;
}
SMetaEntry me = {0}; SMetaEntry me = {0};
tDecoderInit(&dc, pData, nData); tDecoderInit(&dc, pData, nData);
@ -385,7 +404,9 @@ _query:
} }
tDecoderInit(&dc, pData, nData); tDecoderInit(&dc, pData, nData);
(void)tDecodeSSchemaWrapperEx(&dc, &schema); if (tDecodeSSchemaWrapperEx(&dc, &schema) != 0) {
goto _err;
}
pSchema = tCloneSSchemaWrapper(&schema); pSchema = tCloneSSchemaWrapper(&schema);
tDecoderClear(&dc); tDecoderClear(&dc);
@ -588,6 +609,7 @@ STSchema *metaGetTbTSchema(SMeta *pMeta, tb_uid_t uid, int32_t sver, int lock) {
int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sver, STSchema **ppTSchema) { int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sver, STSchema **ppTSchema) {
int32_t code = 0; int32_t code = 0;
int32_t lino;
void *pData = NULL; void *pData = NULL;
int nData = 0; int nData = 0;
@ -603,7 +625,8 @@ int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sv
skmDbKey.uid = suid ? suid : uid; skmDbKey.uid = suid ? suid : uid;
skmDbKey.sver = INT32_MAX; skmDbKey.sver = INT32_MAX;
(void)tdbTbcOpen(pMeta->pSkmDb, &pSkmDbC, NULL); code = tdbTbcOpen(pMeta->pSkmDb, &pSkmDbC, NULL);
TSDB_CHECK_CODE(code, lino, _exit);
metaRLock(pMeta); metaRLock(pMeta);
if (tdbTbcMoveTo(pSkmDbC, &skmDbKey, sizeof(skmDbKey), &c) < 0) { if (tdbTbcMoveTo(pSkmDbC, &skmDbKey, sizeof(skmDbKey), &c) < 0) {

50
source/dnode/vnode/src/meta/metaTable.c
View File

@ -312,6 +312,7 @@ int metaDropSTable(SMeta *pMeta, int64_t verison, SVDropStbReq *pReq, SArray *tb
int nData = 0; int nData = 0;
int c = 0; int c = 0;
int rc = 0; int rc = 0;
int32_t lino;
// check if super table exists // check if super table exists
rc = tdbTbGet(pMeta->pNameIdx, pReq->name, strlen(pReq->name) + 1, &pData, &nData); rc = tdbTbGet(pMeta->pNameIdx, pReq->name, strlen(pReq->name) + 1, &pData, &nData);
@ -323,7 +324,11 @@ int metaDropSTable(SMeta *pMeta, int64_t verison, SVDropStbReq *pReq, SArray *tb
// drop all child tables // drop all child tables
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
(void)(void)tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL); rc = tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL);
if (rc) {
return (terrno = rc);
}
rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = pReq->suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = pReq->suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) { if (rc < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
@ -379,20 +384,20 @@ _exit:
return 0; return 0;
} }
static void metaGetSubtables(SMeta *pMeta, int64_t suid, SArray *uids) { static int32_t metaGetSubtables(SMeta *pMeta, int64_t suid, SArray *uids) {
if (!uids) return; if (!uids) return TSDB_CODE_INVALID_PARA;
int c = 0; int c = 0;
void *pKey = NULL; void *pKey = NULL;
int nKey = 0; int nKey = 0;
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
(void)tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL));
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) { if (rc < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
metaWLock(pMeta); metaWLock(pMeta);
return; return 0;
} }
for (;;) { for (;;) {
@ -405,12 +410,17 @@ static void metaGetSubtables(SMeta *pMeta, int64_t suid, SArray *uids) {
break; break;
} }
(void)taosArrayPush(uids, &(((SCtbIdxKey *)pKey)->uid)); if (taosArrayPush(uids, &(((SCtbIdxKey *)pKey)->uid)) == NULL) {
tdbFree(pKey);
(void)tdbTbcClose(pCtbIdxc);
return terrno;
}
} }
tdbFree(pKey); tdbFree(pKey);
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
return 0;
} }
int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
@ -425,7 +435,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
int32_t ret; int32_t ret;
int32_t c = -2; int32_t c = -2;
(void)tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL));
ret = tdbTbcMoveTo(pUidIdxc, &pReq->suid, sizeof(tb_uid_t), &c); ret = tdbTbcMoveTo(pUidIdxc, &pReq->suid, sizeof(tb_uid_t), &c);
if (ret < 0 || c) { if (ret < 0 || c) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -442,7 +452,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
oversion = ((SUidIdxVal *)pData)[0].version; oversion = ((SUidIdxVal *)pData)[0].version;
(void)tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL));
ret = tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = pReq->suid, .version = oversion}), sizeof(STbDbKey), &c); ret = tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = pReq->suid, .version = oversion}), sizeof(STbDbKey), &c);
if (!(ret == 0 && c == 0)) { if (!(ret == 0 && c == 0)) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -486,7 +496,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
int16_t cid = pReq->schemaRow.pSchema[nCols - 1].colId; int16_t cid = pReq->schemaRow.pSchema[nCols - 1].colId;
int8_t col_type = pReq->schemaRow.pSchema[nCols - 1].type; int8_t col_type = pReq->schemaRow.pSchema[nCols - 1].type;
metaGetSubtables(pMeta, pReq->suid, uids); TAOS_CHECK_RETURN(metaGetSubtables(pMeta, pReq->suid, uids));
(void)tsdbCacheNewSTableColumn(pTsdb, uids, cid, col_type); (void)tsdbCacheNewSTableColumn(pTsdb, uids, cid, col_type);
} else if (deltaCol == -1) { } else if (deltaCol == -1) {
int16_t cid = -1; int16_t cid = -1;
@ -502,7 +512,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
} }
if (cid != -1) { if (cid != -1) {
metaGetSubtables(pMeta, pReq->suid, uids); TAOS_CHECK_RETURN(metaGetSubtables(pMeta, pReq->suid, uids));
(void)tsdbCacheDropSTableColumn(pTsdb, uids, cid, hasPrimaryKey); (void)tsdbCacheDropSTableColumn(pTsdb, uids, cid, hasPrimaryKey);
} }
} }
@ -619,7 +629,7 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
* iterator all pTdDbc by uid and version * iterator all pTdDbc by uid and version
*/ */
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
(void)tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL));
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) { if (rc < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
@ -756,7 +766,7 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq)
* iterator all pTdDbc by uid and version * iterator all pTdDbc by uid and version
*/ */
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
(void)tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL));
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) { if (rc < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
@ -1424,7 +1434,7 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl
// search uid index // search uid index
TBC *pUidIdxc = NULL; TBC *pUidIdxc = NULL;
(void)tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL));
(void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); (void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c);
if (c != 0) { if (c != 0) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -1438,7 +1448,7 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl
// search table.db // search table.db
TBC *pTbDbc = NULL; TBC *pTbDbc = NULL;
(void)tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL));
(void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); (void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c);
if (c != 0) { if (c != 0) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -1689,7 +1699,7 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA
// search uid index // search uid index
TBC *pUidIdxc = NULL; TBC *pUidIdxc = NULL;
(void)tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL));
(void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); (void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c);
if (c != 0) { if (c != 0) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -1706,7 +1716,7 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA
SDecoder dc2 = {0}; SDecoder dc2 = {0};
/* get ctbEntry */ /* get ctbEntry */
(void)tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL));
(void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); (void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c);
if (c != 0) { if (c != 0) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -1869,7 +1879,7 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p
// search uid index // search uid index
TBC *pUidIdxc = NULL; TBC *pUidIdxc = NULL;
(void)tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL));
(void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); (void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c);
if (c != 0) { if (c != 0) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -1883,7 +1893,7 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p
// search table.db // search table.db
TBC *pTbDbc = NULL; TBC *pTbDbc = NULL;
(void)tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL));
(void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); (void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c);
if (c != 0) { if (c != 0) {
(void)tdbTbcClose(pUidIdxc); (void)tdbTbcClose(pUidIdxc);
@ -2018,7 +2028,7 @@ static int metaAddTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTb
* iterator all pTdDbc by uid and version * iterator all pTdDbc by uid and version
*/ */
TBC *pCtbIdxc = NULL; TBC *pCtbIdxc = NULL;
(void)tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL));
int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c);
if (rc < 0) { if (rc < 0) {
(void)tdbTbcClose(pCtbIdxc); (void)tdbTbcClose(pCtbIdxc);
@ -2157,7 +2167,7 @@ static int metaDropTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterT
SArray *tagIdxList = taosArrayInit(512, sizeof(SMetaPair)); SArray *tagIdxList = taosArrayInit(512, sizeof(SMetaPair));
TBC *pTagIdxc = NULL; TBC *pTagIdxc = NULL;
(void)tdbTbcOpen(pMeta->pTagIdx, &pTagIdxc, NULL); TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTagIdx, &pTagIdxc, NULL));
int rc = int rc =
tdbTbcMoveTo(pTagIdxc, &(STagIdxKey){.suid = suid, .cid = INT32_MIN, .type = pCol->type}, sizeof(STagIdxKey), &c); tdbTbcMoveTo(pTagIdxc, &(STagIdxKey){.suid = suid, .cid = INT32_MIN, .type = pCol->type}, sizeof(STagIdxKey), &c);
for (;;) { for (;;) {

3
source/dnode/vnode/src/tq/tqScan.c
View File

@ -101,6 +101,7 @@ int32_t tqScanData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, STqOffsetVal*
TSDB_CHECK_CODE(code, line, END); TSDB_CHECK_CODE(code, line, END);
qStreamSetSourceExcluded(task, pRequest->sourceExcluded); qStreamSetSourceExcluded(task, pRequest->sourceExcluded);
uint64_t st = taosGetTimestampMs();
while (1) { while (1) {
SSDataBlock* pDataBlock = NULL; SSDataBlock* pDataBlock = NULL;
code = getDataBlock(task, pHandle, vgId, &pDataBlock); code = getDataBlock(task, pHandle, vgId, &pDataBlock);
@ -160,7 +161,7 @@ int32_t tqScanData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, STqOffsetVal*
pRsp->common.blockNum++; pRsp->common.blockNum++;
totalRows += pDataBlock->info.rows; totalRows += pDataBlock->info.rows;
if (totalRows >= tmqRowSize) { if (totalRows >= tmqRowSize || (taosGetTimestampMs() - st > 1000)) {
break; break;
} }
} }

6
source/libs/command/src/command.c
View File

@ -38,7 +38,7 @@ static int32_t buildRetrieveTableRsp(SSDataBlock* pBlock, int32_t numOfCols, SRe
size_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock) + PAYLOAD_PREFIX_LEN; size_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock) + PAYLOAD_PREFIX_LEN;
*pRsp = taosMemoryCalloc(1, rspSize); *pRsp = taosMemoryCalloc(1, rspSize);
if (NULL == *pRsp) { if (NULL == *pRsp) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
(*pRsp)->useconds = 0; (*pRsp)->useconds = 0;
@ -289,7 +289,7 @@ static int32_t buildRetension(SArray* pRetension, char **ppRetentions ) {
char* p1 = taosMemoryCalloc(1, 100); char* p1 = taosMemoryCalloc(1, 100);
if(NULL == p1) { if(NULL == p1) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
int32_t len = 0; int32_t len = 0;
@ -849,7 +849,7 @@ _return:
static int32_t buildLocalVariablesResultDataBlock(SSDataBlock** pOutput) { static int32_t buildLocalVariablesResultDataBlock(SSDataBlock** pOutput) {
SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));
if (NULL == pBlock) { if (NULL == pBlock) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
pBlock->info.hasVarCol = true; pBlock->info.hasVarCol = true;

2
source/libs/command/src/explain.c
View File

@ -227,7 +227,7 @@ int32_t qExplainGenerateResNode(SPhysiNode *pNode, SExplainGroup *group, SExplai
SExplainResNode *resNode = taosMemoryCalloc(1, sizeof(SExplainResNode)); SExplainResNode *resNode = taosMemoryCalloc(1, sizeof(SExplainResNode));
if (NULL == resNode) { if (NULL == resNode) {
qError("calloc SPhysiNodeExplainRes failed"); qError("calloc SPhysiNodeExplainRes failed");
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
int32_t code = 0; int32_t code = 0;

3
source/libs/executor/src/aggregateoperator.c
View File

@ -343,6 +343,7 @@ int32_t doAggregateImpl(SOperatorInfo* pOperator, SqlFunctionCtx* pCtx) {
static int32_t createDataBlockForEmptyInput(SOperatorInfo* pOperator, SSDataBlock** ppBlock) { static int32_t createDataBlockForEmptyInput(SOperatorInfo* pOperator, SSDataBlock** ppBlock) {
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0; int32_t lino = 0;
SSDataBlock* pBlock = NULL;
if (!tsCountAlwaysReturnValue) { if (!tsCountAlwaysReturnValue) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
@ -366,7 +367,6 @@ static int32_t createDataBlockForEmptyInput(SOperatorInfo* pOperator, SSDataBloc
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
SSDataBlock* pBlock = NULL;
code = createDataBlock(&pBlock); code = createDataBlock(&pBlock);
if (code) { if (code) {
return code; return code;
@ -414,6 +414,7 @@ static int32_t createDataBlockForEmptyInput(SOperatorInfo* pOperator, SSDataBloc
_end: _end:
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
blockDataDestroy(pBlock);
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
} }
return code; return code;

3
source/libs/executor/src/countwindowoperator.c
View File

@ -294,10 +294,11 @@ int32_t createCountwindowOperatorInfo(SOperatorInfo* downstream, SPhysiNode* phy
SSDataBlock* pResBlock = createDataBlockFromDescNode(pCountWindowNode->window.node.pOutputDataBlockDesc); SSDataBlock* pResBlock = createDataBlockFromDescNode(pCountWindowNode->window.node.pOutputDataBlockDesc);
QUERY_CHECK_NULL(pResBlock, code, lino, _error, terrno); QUERY_CHECK_NULL(pResBlock, code, lino, _error, terrno);
initBasicInfo(&pInfo->binfo, pResBlock);
code = blockDataEnsureCapacity(pResBlock, pOperator->resultInfo.capacity); code = blockDataEnsureCapacity(pResBlock, pOperator->resultInfo.capacity);
QUERY_CHECK_CODE(code, lino, _error); QUERY_CHECK_CODE(code, lino, _error);
initBasicInfo(&pInfo->binfo, pResBlock);
initResultRowInfo(&pInfo->binfo.resultRowInfo); initResultRowInfo(&pInfo->binfo.resultRowInfo);
pInfo->binfo.inputTsOrder = physiNode->inputTsOrder; pInfo->binfo.inputTsOrder = physiNode->inputTsOrder;
pInfo->binfo.outputTsOrder = physiNode->outputTsOrder; pInfo->binfo.outputTsOrder = physiNode->outputTsOrder;

2
source/libs/executor/src/eventwindowoperator.c
View File

@ -110,11 +110,11 @@ int32_t createEventwindowOperatorInfo(SOperatorInfo* downstream, SPhysiNode* phy
SSDataBlock* pResBlock = createDataBlockFromDescNode(pEventWindowNode->window.node.pOutputDataBlockDesc); SSDataBlock* pResBlock = createDataBlockFromDescNode(pEventWindowNode->window.node.pOutputDataBlockDesc);
QUERY_CHECK_NULL(pResBlock, code, lino, _error, terrno); QUERY_CHECK_NULL(pResBlock, code, lino, _error, terrno);
initBasicInfo(&pInfo->binfo, pResBlock);
code = blockDataEnsureCapacity(pResBlock, pOperator->resultInfo.capacity); code = blockDataEnsureCapacity(pResBlock, pOperator->resultInfo.capacity);
QUERY_CHECK_CODE(code, lino, _error); QUERY_CHECK_CODE(code, lino, _error);
initBasicInfo(&pInfo->binfo, pResBlock);
initResultRowInfo(&pInfo->binfo.resultRowInfo); initResultRowInfo(&pInfo->binfo.resultRowInfo);
pInfo->binfo.inputTsOrder = physiNode->inputTsOrder; pInfo->binfo.inputTsOrder = physiNode->inputTsOrder;
pInfo->binfo.outputTsOrder = physiNode->outputTsOrder; pInfo->binfo.outputTsOrder = physiNode->outputTsOrder;

1
source/libs/executor/src/exchangeoperator.c
View File

@ -735,6 +735,7 @@ int32_t extractDataBlockFromFetchRsp(SSDataBlock* pRes, char* pData, SArray* pCo
pStart += sizeof(SSysTableSchema); pStart += sizeof(SSysTableSchema);
} }
pBlock = NULL;
code = createDataBlock(&pBlock); code = createDataBlock(&pBlock);
QUERY_CHECK_CODE(code, lino, _end); QUERY_CHECK_CODE(code, lino, _end);

6
source/libs/executor/src/executil.c
View File

@ -278,7 +278,7 @@ SSDataBlock* createDataBlockFromDescNode(SDataBlockDescNode* pNode) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
blockDataDestroy(pBlock); blockDataDestroy(pBlock);
pBlock = NULL; pBlock = NULL;
terrno = code; terrno = TSDB_CODE_INVALID_PARA;
break; break;
} }
SColumnInfoData idata = SColumnInfoData idata =
@ -1094,7 +1094,7 @@ SSDataBlock* createTagValBlockForFilter(SArray* pColList, int32_t numOfTables, S
code = blockDataEnsureCapacity(pResBlock, numOfTables); code = blockDataEnsureCapacity(pResBlock, numOfTables);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
terrno = code; terrno = code;
taosMemoryFree(pResBlock); blockDataDestroy(pResBlock);
return NULL; return NULL;
} }
@ -1166,7 +1166,7 @@ SSDataBlock* createTagValBlockForFilter(SArray* pColList, int32_t numOfTables, S
_end: _end:
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
taosMemoryFree(pResBlock); blockDataDestroy(pResBlock);
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
terrno = code; terrno = code;
return NULL; return NULL;

28
source/libs/executor/src/scanoperator.c
View File

@ -4319,11 +4319,12 @@ static int32_t tagScanFilterByTagCond(SArray* aUidTags, SNode* pTagCond, SArray*
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0; int32_t lino = 0;
int32_t numOfTables = taosArrayGetSize(aUidTags); int32_t numOfTables = taosArrayGetSize(aUidTags);
SArray* pBlockList = NULL;
SSDataBlock* pResBlock = createTagValBlockForFilter(pInfo->filterCtx.cInfoList, numOfTables, aUidTags, pVnode, pAPI); SSDataBlock* pResBlock = createTagValBlockForFilter(pInfo->filterCtx.cInfoList, numOfTables, aUidTags, pVnode, pAPI);
QUERY_CHECK_NULL(pResBlock, code, lino, _end, terrno); QUERY_CHECK_NULL(pResBlock, code, lino, _end, terrno);
SArray* pBlockList = taosArrayInit(1, POINTER_BYTES); pBlockList = taosArrayInit(1, POINTER_BYTES);
QUERY_CHECK_NULL(pBlockList, code, lino, _end, terrno); QUERY_CHECK_NULL(pBlockList, code, lino, _end, terrno);
void* tmp = taosArrayPush(pBlockList, &pResBlock); void* tmp = taosArrayPush(pBlockList, &pResBlock);
@ -5836,7 +5837,8 @@ int32_t createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SR
SOperatorInfo** pOptrInfo) { SOperatorInfo** pOptrInfo) {
QRY_OPTR_CHECK(pOptrInfo); QRY_OPTR_CHECK(pOptrInfo);
int32_t code = 0; int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
STableMergeScanInfo* pInfo = taosMemoryCalloc(1, sizeof(STableMergeScanInfo)); STableMergeScanInfo* pInfo = taosMemoryCalloc(1, sizeof(STableMergeScanInfo));
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
if (pInfo == NULL || pOperator == NULL) { if (pInfo == NULL || pOperator == NULL) {
@ -5849,16 +5851,10 @@ int32_t createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SR
int32_t numOfCols = 0; int32_t numOfCols = 0;
code = extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID, code = extractColMatchInfo(pTableScanNode->scan.pScanCols, pDescNode, &numOfCols, COL_MATCH_FROM_COL_ID,
&pInfo->base.matchInfo); &pInfo->base.matchInfo);
int32_t lino = 0; QUERY_CHECK_CODE(code, lino, _error);
if (code != TSDB_CODE_SUCCESS) {
goto _error;
}
code = initQueryTableDataCond(&pInfo->base.cond, pTableScanNode, readHandle); code = initQueryTableDataCond(&pInfo->base.cond, pTableScanNode, readHandle);
if (code != TSDB_CODE_SUCCESS) { QUERY_CHECK_CODE(code, lino, _error);
taosArrayDestroy(pInfo->base.matchInfo.pList);
goto _error;
}
if (pTableScanNode->scan.pScanPseudoCols != NULL) { if (pTableScanNode->scan.pScanPseudoCols != NULL) {
SExprSupp* pSup = &pInfo->base.pseudoSup; SExprSupp* pSup = &pInfo->base.pseudoSup;
@ -5873,10 +5869,7 @@ int32_t createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SR
pInfo->scanInfo = (SScanInfo){.numOfAsc = pTableScanNode->scanSeq[0], .numOfDesc = pTableScanNode->scanSeq[1]}; pInfo->scanInfo = (SScanInfo){.numOfAsc = pTableScanNode->scanSeq[0], .numOfDesc = pTableScanNode->scanSeq[1]};
pInfo->base.metaCache.pTableMetaEntryCache = taosLRUCacheInit(1024 * 128, -1, .5); pInfo->base.metaCache.pTableMetaEntryCache = taosLRUCacheInit(1024 * 128, -1, .5);
if (pInfo->base.metaCache.pTableMetaEntryCache == NULL) { QUERY_CHECK_NULL(pInfo->base.metaCache.pTableMetaEntryCache, code, lino, _error, terrno);
code = terrno;
goto _error;
}
pInfo->base.readerAPI = pTaskInfo->storageAPI.tsdReader; pInfo->base.readerAPI = pTaskInfo->storageAPI.tsdReader;
pInfo->base.dataBlockLoadFlag = FUNC_DATA_REQUIRED_DATA_LOAD; pInfo->base.dataBlockLoadFlag = FUNC_DATA_REQUIRED_DATA_LOAD;
@ -5893,9 +5886,7 @@ int32_t createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SR
pInfo->sample.seed = taosGetTimestampSec(); pInfo->sample.seed = taosGetTimestampSec();
code = filterInitFromNode((SNode*)pTableScanNode->scan.node.pConditions, &pOperator->exprSupp.pFilterInfo, 0); code = filterInitFromNode((SNode*)pTableScanNode->scan.node.pConditions, &pOperator->exprSupp.pFilterInfo, 0);
if (code != TSDB_CODE_SUCCESS) { QUERY_CHECK_CODE(code, lino, _error);
goto _error;
}
initLimitInfo(pTableScanNode->scan.node.pLimit, pTableScanNode->scan.node.pSlimit, &pInfo->limitInfo); initLimitInfo(pTableScanNode->scan.node.pLimit, pTableScanNode->scan.node.pSlimit, &pInfo->limitInfo);
@ -5957,6 +5948,9 @@ int32_t createTableMergeScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SR
return code; return code;
_error: _error:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
pTaskInfo->code = code; pTaskInfo->code = code;
pInfo->base.pTableListInfo = NULL; pInfo->base.pTableListInfo = NULL;
if (pInfo != NULL) destroyTableMergeScanOperatorInfo(pInfo); if (pInfo != NULL) destroyTableMergeScanOperatorInfo(pInfo);

3
source/libs/executor/src/streamtimewindowoperator.c
View File

@ -5203,6 +5203,8 @@ int32_t createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiNode*
SSDataBlock* pResBlock = createDataBlockFromDescNode(pPhyNode->pOutputDataBlockDesc); SSDataBlock* pResBlock = createDataBlockFromDescNode(pPhyNode->pOutputDataBlockDesc);
QUERY_CHECK_NULL(pResBlock, code, lino, _error, terrno); QUERY_CHECK_NULL(pResBlock, code, lino, _error, terrno);
initBasicInfo(&pInfo->binfo, pResBlock);
pInfo->interval = (SInterval){ pInfo->interval = (SInterval){
.interval = pIntervalPhyNode->interval, .interval = pIntervalPhyNode->interval,
.sliding = pIntervalPhyNode->sliding, .sliding = pIntervalPhyNode->sliding,
@ -5230,7 +5232,6 @@ int32_t createStreamIntervalOperatorInfo(SOperatorInfo* downstream, SPhysiNode*
SExprSupp* pSup = &pOperator->exprSupp; SExprSupp* pSup = &pOperator->exprSupp;
pSup->hasWindowOrGroup = true; pSup->hasWindowOrGroup = true;
initBasicInfo(&pInfo->binfo, pResBlock);
code = initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window); code = initExecTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pTaskInfo->window);
QUERY_CHECK_CODE(code, lino, _error); QUERY_CHECK_CODE(code, lino, _error);

16
source/libs/executor/src/sysscanoperator.c
View File

@ -568,7 +568,7 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
if (pInfo->pCur == NULL) { if (pInfo->pCur == NULL) {
pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode); pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode);
} else { } else {
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 0); (void)pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 0);
} }
if (pInfo->pSchema == NULL) { if (pInfo->pSchema == NULL) {
@ -780,8 +780,9 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
int32_t ret = 0; int32_t ret = 0;
if (pInfo->pCur == NULL) { if (pInfo->pCur == NULL) {
pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode); pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode);
QUERY_CHECK_NULL(pInfo->pCur, code, lino, _end, terrno);
} else { } else {
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 0); (void)pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 0);
} }
while ((ret = pAPI->metaFn.cursorNext(pInfo->pCur, TSDB_SUPER_TABLE)) == 0) { while ((ret = pAPI->metaFn.cursorNext(pInfo->pCur, TSDB_SUPER_TABLE)) == 0) {
@ -1578,10 +1579,11 @@ static SSDataBlock* sysTableBuildUserTables(SOperatorInfo* pOperator) {
SSysTableScanInfo* pInfo = pOperator->info; SSysTableScanInfo* pInfo = pOperator->info;
if (pInfo->pCur == NULL) { if (pInfo->pCur == NULL) {
pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode); pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode);
QUERY_CHECK_NULL(pInfo->pCur, code, lino, _end, terrno);
firstMetaCursor = 1; firstMetaCursor = 1;
} }
if (!firstMetaCursor) { if (!firstMetaCursor) {
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 1); (void)pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 1);
} }
blockDataCleanup(pInfo->pRes); blockDataCleanup(pInfo->pRes);
@ -2151,8 +2153,8 @@ static SSDataBlock* sysTableScanFromMNode(SOperatorInfo* pOperator, SSysTableSca
} }
} }
int32_t createSysTableScanOperatorInfo(void* readHandle, SSystemTableScanPhysiNode* pScanPhyNode, int32_t createSysTableScanOperatorInfo(void* readHandle, SSystemTableScanPhysiNode* pScanPhyNode, const char* pUser,
const char* pUser, SExecTaskInfo* pTaskInfo, SOperatorInfo** pOptrInfo) { SExecTaskInfo* pTaskInfo, SOperatorInfo** pOptrInfo) {
QRY_OPTR_CHECK(pOptrInfo); QRY_OPTR_CHECK(pOptrInfo);
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
@ -2695,7 +2697,6 @@ static int32_t doBlockInfoScanNext(SOperatorInfo* pOperator, SSDataBlock** ppRes
SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, slotId); SColumnInfoData* pColInfo = taosArrayGet(pBlock->pDataBlock, slotId);
QUERY_CHECK_NULL(pColInfo, code, lino, _end, terrno); QUERY_CHECK_NULL(pColInfo, code, lino, _end, terrno);
int32_t len = tSerializeBlockDistInfo(NULL, 0, &blockDistInfo); int32_t len = tSerializeBlockDistInfo(NULL, 0, &blockDistInfo);
char* p = taosMemoryCalloc(1, len + VARSTR_HEADER_SIZE); char* p = taosMemoryCalloc(1, len + VARSTR_HEADER_SIZE);
QUERY_CHECK_NULL(p, code, lino, _end, terrno); QUERY_CHECK_NULL(p, code, lino, _end, terrno);
@ -2778,7 +2779,8 @@ static int32_t initTableblockDistQueryCond(uint64_t uid, SQueryTableDataCond* pC
} }
int32_t createDataBlockInfoScanOperator(SReadHandle* readHandle, SBlockDistScanPhysiNode* pBlockScanNode, int32_t createDataBlockInfoScanOperator(SReadHandle* readHandle, SBlockDistScanPhysiNode* pBlockScanNode,
STableListInfo* pTableListInfo, SExecTaskInfo* pTaskInfo, SOperatorInfo** pOptrInfo) { STableListInfo* pTableListInfo, SExecTaskInfo* pTaskInfo,
SOperatorInfo** pOptrInfo) {
QRY_OPTR_CHECK(pOptrInfo); QRY_OPTR_CHECK(pOptrInfo);
int32_t code = 0; int32_t code = 0;

36
source/libs/executor/src/tsort.c
View File

@ -152,8 +152,10 @@ static void destoryAllocatedTuple(void* t) { taosMemoryFree(t); }
* @param colIndex the columnIndex, for setting null bitmap * @param colIndex the columnIndex, for setting null bitmap
* @return the next offset to add field * @return the next offset to add field
* */ * */
static inline size_t tupleAddField(char** t, uint32_t colNum, uint32_t offset, uint32_t colIdx, void* data, static inline int32_t tupleAddField(char** t, uint32_t colNum, uint32_t offset, uint32_t colIdx, void* data,
size_t length, bool isNull, uint32_t tupleLen) { size_t length, bool isNull, uint32_t tupleLen, uint32_t* pOffset) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
tupleSetOffset(*t, colIdx, offset); tupleSetOffset(*t, colIdx, offset);
if (isNull) { if (isNull) {
@ -161,16 +163,20 @@ static inline size_t tupleAddField(char** t, uint32_t colNum, uint32_t offset, u
} else { } else {
if (offset + length > tupleLen + tupleGetDataStartOffset(colNum)) { if (offset + length > tupleLen + tupleGetDataStartOffset(colNum)) {
void* px = taosMemoryRealloc(*t, offset + length); void* px = taosMemoryRealloc(*t, offset + length);
if (px == NULL) { QUERY_CHECK_NULL(px, code, lino, _end, terrno);
return terrno;
}
*t = px; *t = px;
} }
tupleSetData(*t, offset, data, length); tupleSetData(*t, offset, data, length);
} }
return offset + length; (*pOffset) = offset + length;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
} }
static void* tupleGetField(char* t, uint32_t colIdx, uint32_t colNum) { static void* tupleGetField(char* t, uint32_t colIdx, uint32_t colNum) {
@ -200,6 +206,7 @@ typedef struct ReferencedTuple {
} ReferencedTuple; } ReferencedTuple;
static int32_t createAllocatedTuple(SSDataBlock* pBlock, size_t colNum, uint32_t tupleLen, size_t rowIdx, TupleDesc** pDesc) { static int32_t createAllocatedTuple(SSDataBlock* pBlock, size_t colNum, uint32_t tupleLen, size_t rowIdx, TupleDesc** pDesc) {
int32_t code = TSDB_CODE_SUCCESS;
TupleDesc* t = taosMemoryCalloc(1, sizeof(TupleDesc)); TupleDesc* t = taosMemoryCalloc(1, sizeof(TupleDesc));
if (t == NULL) { if (t == NULL) {
return terrno; return terrno;
@ -216,15 +223,20 @@ static int32_t createAllocatedTuple(SSDataBlock* pBlock, size_t colNum, uint32_t
for (size_t colIdx = 0; colIdx < colNum; ++colIdx) { for (size_t colIdx = 0; colIdx < colNum; ++colIdx) {
SColumnInfoData* pCol = taosArrayGet(pBlock->pDataBlock, colIdx); SColumnInfoData* pCol = taosArrayGet(pBlock->pDataBlock, colIdx);
if (pCol == NULL) { if (pCol == NULL) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno));
return terrno; return terrno;
} }
if (colDataIsNull_s(pCol, rowIdx)) { if (colDataIsNull_s(pCol, rowIdx)) {
offset = tupleAddField((char**)&pTuple, colNum, offset, colIdx, 0, 0, true, tupleLen); code = tupleAddField((char**)&pTuple, colNum, offset, colIdx, 0, 0, true, tupleLen, &offset);
} else { } else {
colLen = colDataGetRowLength(pCol, rowIdx); colLen = colDataGetRowLength(pCol, rowIdx);
offset = code =
tupleAddField((char**)&pTuple, colNum, offset, colIdx, colDataGetData(pCol, rowIdx), colLen, false, tupleLen); tupleAddField((char**)&pTuple, colNum, offset, colIdx, colDataGetData(pCol, rowIdx), colLen, false, tupleLen, &offset);
}
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code));
return code;
} }
} }
@ -232,7 +244,7 @@ static int32_t createAllocatedTuple(SSDataBlock* pBlock, size_t colNum, uint32_t
t->data = pTuple; t->data = pTuple;
*pDesc = t; *pDesc = t;
return 0; return code;
} }
int32_t tupleDescGetField(const TupleDesc* pDesc, int32_t colIdx, uint32_t colNum, void** pResult) { int32_t tupleDescGetField(const TupleDesc* pDesc, int32_t colIdx, uint32_t colNum, void** pResult) {
@ -259,7 +271,7 @@ int32_t tupleDescGetField(const TupleDesc* pDesc, int32_t colIdx, uint32_t colNu
void destroyTuple(void* t) { void destroyTuple(void* t) {
TupleDesc* pDesc = t; TupleDesc* pDesc = t;
if (pDesc->type == AllocatedTupleType) { if (pDesc != NULL && pDesc->type == AllocatedTupleType) {
destoryAllocatedTuple(pDesc->data); destoryAllocatedTuple(pDesc->data);
taosMemoryFree(pDesc); taosMemoryFree(pDesc);
} }
@ -1686,6 +1698,7 @@ static int32_t initRowIdSort(SSortHandle* pHandle) {
biTs.compFn = getKeyComparFunc(TSDB_DATA_TYPE_TIMESTAMP, biTs.order); biTs.compFn = getKeyComparFunc(TSDB_DATA_TYPE_TIMESTAMP, biTs.order);
void* p = taosArrayPush(pOrderInfoList, &biTs); void* p = taosArrayPush(pOrderInfoList, &biTs);
if (p == NULL) { if (p == NULL) {
taosArrayDestroy(pOrderInfoList);
return terrno; return terrno;
} }
@ -1698,6 +1711,7 @@ static int32_t initRowIdSort(SSortHandle* pHandle) {
void* px = taosArrayPush(pOrderInfoList, &biPk); void* px = taosArrayPush(pOrderInfoList, &biPk);
if (px == NULL) { if (px == NULL) {
taosArrayDestroy(pOrderInfoList);
return terrno; return terrno;
} }
} }

27
source/libs/function/src/tudf.c
View File

@ -162,7 +162,7 @@ static int32_t udfSpawnUdfd(SUdfdData *pData) {
fnInfo("[UDFD]Succsess to set TAOS_FQDN:%s", taosFqdn); fnInfo("[UDFD]Succsess to set TAOS_FQDN:%s", taosFqdn);
} else { } else {
fnError("[UDFD]Failed to allocate memory for TAOS_FQDN"); fnError("[UDFD]Failed to allocate memory for TAOS_FQDN");
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
} }
@ -837,10 +837,13 @@ int32_t convertDataBlockToUdfDataBlock(SSDataBlock *block, SUdfDataBlock *udfBlo
udfBlock->numOfRows = block->info.rows; udfBlock->numOfRows = block->info.rows;
udfBlock->numOfCols = taosArrayGetSize(block->pDataBlock); udfBlock->numOfCols = taosArrayGetSize(block->pDataBlock);
udfBlock->udfCols = taosMemoryCalloc(taosArrayGetSize(block->pDataBlock), sizeof(SUdfColumn *)); udfBlock->udfCols = taosMemoryCalloc(taosArrayGetSize(block->pDataBlock), sizeof(SUdfColumn *));
if((udfBlock->udfCols) == NULL) {
return terrno;
}
for (int32_t i = 0; i < udfBlock->numOfCols; ++i) { for (int32_t i = 0; i < udfBlock->numOfCols; ++i) {
udfBlock->udfCols[i] = taosMemoryCalloc(1, sizeof(SUdfColumn)); udfBlock->udfCols[i] = taosMemoryCalloc(1, sizeof(SUdfColumn));
if(udfBlock->udfCols[i] == NULL) { if(udfBlock->udfCols[i] == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
SColumnInfoData *col = (SColumnInfoData *)taosArrayGet(block->pDataBlock, i); SColumnInfoData *col = (SColumnInfoData *)taosArrayGet(block->pDataBlock, i);
SUdfColumn *udfCol = udfBlock->udfCols[i]; SUdfColumn *udfCol = udfBlock->udfCols[i];
@ -854,13 +857,13 @@ int32_t convertDataBlockToUdfDataBlock(SSDataBlock *block, SUdfDataBlock *udfBlo
udfCol->colData.varLenCol.varOffsetsLen = sizeof(int32_t) * udfBlock->numOfRows; udfCol->colData.varLenCol.varOffsetsLen = sizeof(int32_t) * udfBlock->numOfRows;
udfCol->colData.varLenCol.varOffsets = taosMemoryMalloc(udfCol->colData.varLenCol.varOffsetsLen); udfCol->colData.varLenCol.varOffsets = taosMemoryMalloc(udfCol->colData.varLenCol.varOffsetsLen);
if(udfCol->colData.varLenCol.varOffsets == NULL) { if(udfCol->colData.varLenCol.varOffsets == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
memcpy(udfCol->colData.varLenCol.varOffsets, col->varmeta.offset, udfCol->colData.varLenCol.varOffsetsLen); memcpy(udfCol->colData.varLenCol.varOffsets, col->varmeta.offset, udfCol->colData.varLenCol.varOffsetsLen);
udfCol->colData.varLenCol.payloadLen = colDataGetLength(col, udfBlock->numOfRows); udfCol->colData.varLenCol.payloadLen = colDataGetLength(col, udfBlock->numOfRows);
udfCol->colData.varLenCol.payload = taosMemoryMalloc(udfCol->colData.varLenCol.payloadLen); udfCol->colData.varLenCol.payload = taosMemoryMalloc(udfCol->colData.varLenCol.payloadLen);
if(udfCol->colData.varLenCol.payload == NULL) { if(udfCol->colData.varLenCol.payload == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
if (col->reassigned) { if (col->reassigned) {
for (int32_t row = 0; row < udfCol->colData.numOfRows; ++row) { for (int32_t row = 0; row < udfCol->colData.numOfRows; ++row) {
@ -882,7 +885,7 @@ int32_t convertDataBlockToUdfDataBlock(SSDataBlock *block, SUdfDataBlock *udfBlo
int32_t bitmapLen = udfCol->colData.fixLenCol.nullBitmapLen; int32_t bitmapLen = udfCol->colData.fixLenCol.nullBitmapLen;
udfCol->colData.fixLenCol.nullBitmap = taosMemoryMalloc(udfCol->colData.fixLenCol.nullBitmapLen); udfCol->colData.fixLenCol.nullBitmap = taosMemoryMalloc(udfCol->colData.fixLenCol.nullBitmapLen);
if(udfCol->colData.fixLenCol.nullBitmap == NULL) { if(udfCol->colData.fixLenCol.nullBitmap == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
char *bitmap = udfCol->colData.fixLenCol.nullBitmap; char *bitmap = udfCol->colData.fixLenCol.nullBitmap;
memcpy(bitmap, col->nullbitmap, bitmapLen); memcpy(bitmap, col->nullbitmap, bitmapLen);
@ -985,7 +988,7 @@ int32_t convertDataBlockToScalarParm(SSDataBlock *input, SScalarParam *output) {
output->columnData = taosMemoryMalloc(sizeof(SColumnInfoData)); output->columnData = taosMemoryMalloc(sizeof(SColumnInfoData));
if(output->columnData == NULL) { if(output->columnData == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
memcpy(output->columnData, taosArrayGet(input->pDataBlock, 0), sizeof(SColumnInfoData)); memcpy(output->columnData, taosArrayGet(input->pDataBlock, 0), sizeof(SColumnInfoData));
output->colAlloced = true; output->colAlloced = true;
@ -1724,7 +1727,7 @@ int32_t udfcStartUvTask(SClientUvTaskNode *uvTask) {
if(conn == NULL) { if(conn == NULL) {
fnError("udfc event loop start connect task malloc conn failed."); fnError("udfc event loop start connect task malloc conn failed.");
taosMemoryFree(pipe); taosMemoryFree(pipe);
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
conn->pipe = pipe; conn->pipe = pipe;
conn->readBuf.len = 0; conn->readBuf.len = 0;
@ -1954,7 +1957,7 @@ int32_t udfcRunUdfUvTask(SClientUdfTask *task, int8_t uvTaskType) {
SClientUvTaskNode *uvTask = taosMemoryCalloc(1, sizeof(SClientUvTaskNode)); SClientUvTaskNode *uvTask = taosMemoryCalloc(1, sizeof(SClientUvTaskNode));
if(uvTask == NULL) { if(uvTask == NULL) {
fnError("udfc client task: %p failed to allocate memory for uvTask", task); fnError("udfc client task: %p failed to allocate memory for uvTask", task);
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
fnDebug("udfc client task: %p created uvTask: %p. pipe: %p", task, uvTask, task->session->udfUvPipe); fnDebug("udfc client task: %p created uvTask: %p. pipe: %p", task, uvTask, task->session->udfUvPipe);
@ -1986,13 +1989,13 @@ int32_t doSetupUdf(char udfName[], UdfcFuncHandle *funcHandle) {
SClientUdfTask *task = taosMemoryCalloc(1, sizeof(SClientUdfTask)); SClientUdfTask *task = taosMemoryCalloc(1, sizeof(SClientUdfTask));
if(task == NULL) { if(task == NULL) {
fnError("doSetupUdf, failed to allocate memory for task"); fnError("doSetupUdf, failed to allocate memory for task");
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
task->session = taosMemoryCalloc(1, sizeof(SUdfcUvSession)); task->session = taosMemoryCalloc(1, sizeof(SUdfcUvSession));
if(task->session == NULL) { if(task->session == NULL) {
fnError("doSetupUdf, failed to allocate memory for session"); fnError("doSetupUdf, failed to allocate memory for session");
taosMemoryFree(task); taosMemoryFree(task);
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
task->session->udfc = &gUdfcProxy; task->session->udfc = &gUdfcProxy;
task->type = UDF_TASK_SETUP; task->type = UDF_TASK_SETUP;
@ -2037,7 +2040,7 @@ int32_t callUdf(UdfcFuncHandle handle, int8_t callType, SSDataBlock *input, SUdf
SClientUdfTask *task = taosMemoryCalloc(1, sizeof(SClientUdfTask)); SClientUdfTask *task = taosMemoryCalloc(1, sizeof(SClientUdfTask));
if(task == NULL) { if(task == NULL) {
fnError("udfc call udf. failed to allocate memory for task"); fnError("udfc call udf. failed to allocate memory for task");
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
task->session = (SUdfcUvSession *)handle; task->session = (SUdfcUvSession *)handle;
task->type = UDF_TASK_CALL; task->type = UDF_TASK_CALL;
@ -2169,7 +2172,7 @@ int32_t doTeardownUdf(UdfcFuncHandle handle) {
if(task == NULL) { if(task == NULL) {
fnError("doTeardownUdf, failed to allocate memory for task"); fnError("doTeardownUdf, failed to allocate memory for task");
taosMemoryFree(session); taosMemoryFree(session);
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
task->session = session; task->session = session;
task->type = UDF_TASK_TEARDOWN; task->type = UDF_TASK_TEARDOWN;

76
source/libs/function/src/udfd.c
View File

@ -409,6 +409,10 @@ int32_t udfdInitializePythonPlugin(SUdfScriptPlugin *plugin) {
int16_t lenPythonPath = int16_t lenPythonPath =
strlen(tsUdfdLdLibPath) + strlen(global.udfDataDir) + 1 + 1; // global.udfDataDir:tsUdfdLdLibPath strlen(tsUdfdLdLibPath) + strlen(global.udfDataDir) + 1 + 1; // global.udfDataDir:tsUdfdLdLibPath
char *pythonPath = taosMemoryMalloc(lenPythonPath); char *pythonPath = taosMemoryMalloc(lenPythonPath);
if(pythonPath == NULL) {
uv_dlclose(&plugin->lib);
return terrno;
}
#ifdef WINDOWS #ifdef WINDOWS
snprintf(pythonPath, lenPythonPath, "%s;%s", global.udfDataDir, tsUdfdLdLibPath); snprintf(pythonPath, lenPythonPath, "%s;%s", global.udfDataDir, tsUdfdLdLibPath);
#else #else
@ -705,6 +709,10 @@ void udfdProcessSetupRequest(SUvUdfWork *uvUdf, SUdfRequest *request) {
uv_mutex_unlock(&udf->lock); uv_mutex_unlock(&udf->lock);
} }
SUdfcFuncHandle *handle = taosMemoryMalloc(sizeof(SUdfcFuncHandle)); SUdfcFuncHandle *handle = taosMemoryMalloc(sizeof(SUdfcFuncHandle));
if(handle == NULL) {
fnError("udfdProcessSetupRequest: malloc failed.");
code = terrno;
}
handle->udf = udf; handle->udf = udf;
_send: _send:
@ -775,7 +783,7 @@ void udfdProcessCallRequest(SUvUdfWork *uvUdf, SUdfRequest *request) {
if (outBuf.buf != NULL) { if (outBuf.buf != NULL) {
code = udf->scriptPlugin->udfAggStartFunc(&outBuf, udf->scriptUdfCtx); code = udf->scriptPlugin->udfAggStartFunc(&outBuf, udf->scriptUdfCtx);
} else { } else {
code = TSDB_CODE_OUT_OF_MEMORY; code = terrno;
} }
subRsp->resultBuf = outBuf; subRsp->resultBuf = outBuf;
break; break;
@ -784,9 +792,13 @@ void udfdProcessCallRequest(SUvUdfWork *uvUdf, SUdfRequest *request) {
SUdfDataBlock input = {0}; SUdfDataBlock input = {0};
if (convertDataBlockToUdfDataBlock(&call->block, &input) == TSDB_CODE_SUCCESS) { if (convertDataBlockToUdfDataBlock(&call->block, &input) == TSDB_CODE_SUCCESS) {
SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0}; SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0};
if (outBuf.buf != NULL) {
code = udf->scriptPlugin->udfAggProcFunc(&input, &call->interBuf, &outBuf, udf->scriptUdfCtx); code = udf->scriptPlugin->udfAggProcFunc(&input, &call->interBuf, &outBuf, udf->scriptUdfCtx);
freeUdfInterBuf(&call->interBuf); freeUdfInterBuf(&call->interBuf);
subRsp->resultBuf = outBuf; subRsp->resultBuf = outBuf;
} else {
code = terrno;
}
} }
freeUdfDataDataBlock(&input); freeUdfDataDataBlock(&input);
@ -794,18 +806,27 @@ void udfdProcessCallRequest(SUvUdfWork *uvUdf, SUdfRequest *request) {
} }
case TSDB_UDF_CALL_AGG_MERGE: { case TSDB_UDF_CALL_AGG_MERGE: {
SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0}; SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0};
if (outBuf.buf != NULL) {
code = udf->scriptPlugin->udfAggMergeFunc(&call->interBuf, &call->interBuf2, &outBuf, udf->scriptUdfCtx); code = udf->scriptPlugin->udfAggMergeFunc(&call->interBuf, &call->interBuf2, &outBuf, udf->scriptUdfCtx);
freeUdfInterBuf(&call->interBuf); freeUdfInterBuf(&call->interBuf);
freeUdfInterBuf(&call->interBuf2); freeUdfInterBuf(&call->interBuf2);
subRsp->resultBuf = outBuf; subRsp->resultBuf = outBuf;
} else {
code = terrno;
}
break; break;
} }
case TSDB_UDF_CALL_AGG_FIN: { case TSDB_UDF_CALL_AGG_FIN: {
SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0}; SUdfInterBuf outBuf = {.buf = taosMemoryMalloc(udf->bufSize), .bufLen = udf->bufSize, .numOfResult = 0};
if (outBuf.buf != NULL) {
code = udf->scriptPlugin->udfAggFinishFunc(&call->interBuf, &outBuf, udf->scriptUdfCtx); code = udf->scriptPlugin->udfAggFinishFunc(&call->interBuf, &outBuf, udf->scriptUdfCtx);
freeUdfInterBuf(&call->interBuf); freeUdfInterBuf(&call->interBuf);
subRsp->resultBuf = outBuf; subRsp->resultBuf = outBuf;
} else {
code = terrno;
}
break; break;
} }
default: default:
@ -820,19 +841,24 @@ void udfdProcessCallRequest(SUvUdfWork *uvUdf, SUdfRequest *request) {
int32_t len = encodeUdfResponse(NULL, rsp); int32_t len = encodeUdfResponse(NULL, rsp);
if(len < 0) { if(len < 0) {
fnError("udfdProcessCallRequest: encode udf response failed. len %d", len); fnError("udfdProcessCallRequest: encode udf response failed. len %d", len);
return; goto _exit;
} }
rsp->msgLen = len; rsp->msgLen = len;
void *bufBegin = taosMemoryMalloc(len); void *bufBegin = taosMemoryMalloc(len);
if (bufBegin == NULL) {
fnError("udfdProcessCallRequest: malloc failed. len %d", len);
goto _exit;
}
void *buf = bufBegin; void *buf = bufBegin;
if(encodeUdfResponse(&buf, rsp) < 0) { if(encodeUdfResponse(&buf, rsp) < 0) {
fnError("udfdProcessCallRequest: encode udf response failed. len %d", len); fnError("udfdProcessCallRequest: encode udf response failed. len %d", len);
taosMemoryFree(bufBegin); taosMemoryFree(bufBegin);
return; goto _exit;
} }
uvUdf->output = uv_buf_init(bufBegin, len); uvUdf->output = uv_buf_init(bufBegin, len);
_exit:
switch (call->callType) { switch (call->callType) {
case TSDB_UDF_CALL_SCALA_PROC: { case TSDB_UDF_CALL_SCALA_PROC: {
blockDataFreeRes(&call->block); blockDataFreeRes(&call->block);
@ -906,6 +932,10 @@ _send:
} }
rsp->msgLen = len; rsp->msgLen = len;
void *bufBegin = taosMemoryMalloc(len); void *bufBegin = taosMemoryMalloc(len);
if(bufBegin == NULL) {
fnError("udfdProcessTeardownRequest: malloc failed. len %d", len);
return;
}
void *buf = bufBegin; void *buf = bufBegin;
if (encodeUdfResponse(&buf, rsp) < 0) { if (encodeUdfResponse(&buf, rsp) < 0) {
fnError("udfdProcessTeardownRequest: encode udf response failed. len %d", len); fnError("udfdProcessTeardownRequest: encode udf response failed. len %d", len);
@ -1173,7 +1203,7 @@ int32_t udfdOpenClientRpc() {
global.clientRpc = rpcOpen(&rpcInit); global.clientRpc = rpcOpen(&rpcInit);
if (global.clientRpc == NULL) { if (global.clientRpc == NULL) {
fnError("failed to init dnode rpc client"); fnError("failed to init dnode rpc client");
return -1; return terrno;
} }
return 0; return 0;
} }
@ -1210,6 +1240,11 @@ void udfdSendResponse(uv_work_t *work, int status) {
if (udfWork->conn != NULL) { if (udfWork->conn != NULL) {
uv_write_t *write_req = taosMemoryMalloc(sizeof(uv_write_t)); uv_write_t *write_req = taosMemoryMalloc(sizeof(uv_write_t));
if(write_req == NULL) {
fnError("udfd send response error, malloc failed");
taosMemoryFree(work);
return;
}
write_req->data = udfWork; write_req->data = udfWork;
int32_t code = uv_write(write_req, udfWork->conn->client, &udfWork->output, 1, udfdOnWrite); int32_t code = uv_write(write_req, udfWork->conn->client, &udfWork->output, 1, udfdOnWrite);
if (code != 0) { if (code != 0) {
@ -1269,7 +1304,16 @@ void udfdHandleRequest(SUdfdUvConn *conn) {
int32_t inputLen = conn->inputLen; int32_t inputLen = conn->inputLen;
uv_work_t *work = taosMemoryMalloc(sizeof(uv_work_t)); uv_work_t *work = taosMemoryMalloc(sizeof(uv_work_t));
if(work == NULL) {
fnError("udfd malloc work failed");
return;
}
SUvUdfWork *udfWork = taosMemoryMalloc(sizeof(SUvUdfWork)); SUvUdfWork *udfWork = taosMemoryMalloc(sizeof(SUvUdfWork));
if(udfWork == NULL) {
fnError("udfd malloc udf work failed");
taosMemoryFree(work);
return;
}
udfWork->conn = conn; udfWork->conn = conn;
udfWork->pWorkNext = conn->pWorkList; udfWork->pWorkNext = conn->pWorkList;
conn->pWorkList = udfWork; conn->pWorkList = udfWork;
@ -1334,6 +1378,10 @@ void udfdOnNewConnection(uv_stream_t *server, int status) {
int32_t code = 0; int32_t code = 0;
uv_pipe_t *client = (uv_pipe_t *)taosMemoryMalloc(sizeof(uv_pipe_t)); uv_pipe_t *client = (uv_pipe_t *)taosMemoryMalloc(sizeof(uv_pipe_t));
if(client == NULL) {
fnError("udfd pipe malloc failed");
return;
}
code = uv_pipe_init(global.loop, client, 0); code = uv_pipe_init(global.loop, client, 0);
if (code) { if (code) {
fnError("udfd pipe init error %s", uv_strerror(code)); fnError("udfd pipe init error %s", uv_strerror(code));
@ -1342,6 +1390,10 @@ void udfdOnNewConnection(uv_stream_t *server, int status) {
} }
if (uv_accept(server, (uv_stream_t *)client) == 0) { if (uv_accept(server, (uv_stream_t *)client) == 0) {
SUdfdUvConn *ctx = taosMemoryMalloc(sizeof(SUdfdUvConn)); SUdfdUvConn *ctx = taosMemoryMalloc(sizeof(SUdfdUvConn));
if(ctx == NULL) {
fnError("udfd conn malloc failed");
goto _exit;
}
ctx->pWorkList = NULL; ctx->pWorkList = NULL;
ctx->client = (uv_stream_t *)client; ctx->client = (uv_stream_t *)client;
ctx->inputBuf = 0; ctx->inputBuf = 0;
@ -1356,9 +1408,11 @@ void udfdOnNewConnection(uv_stream_t *server, int status) {
taosMemoryFree(ctx); taosMemoryFree(ctx);
taosMemoryFree(client); taosMemoryFree(client);
} }
} else { return;
uv_close((uv_handle_t *)client, NULL);
} }
_exit:
uv_close((uv_handle_t *)client, NULL);
taosMemoryFree(client);
} }
void udfdIntrSignalHandler(uv_signal_t *handle, int signum) { void udfdIntrSignalHandler(uv_signal_t *handle, int signum) {
@ -1411,6 +1465,10 @@ static int32_t udfdInitLog() {
void udfdCtrlAllocBufCb(uv_handle_t *handle, size_t suggested_size, uv_buf_t *buf) { void udfdCtrlAllocBufCb(uv_handle_t *handle, size_t suggested_size, uv_buf_t *buf) {
buf->base = taosMemoryMalloc(suggested_size); buf->base = taosMemoryMalloc(suggested_size);
if (buf->base == NULL) {
fnError("udfd ctrl pipe alloc buffer failed");
return;
}
buf->len = suggested_size; buf->len = suggested_size;
} }
@ -1477,13 +1535,13 @@ static int32_t udfdGlobalDataInit() {
uv_loop_t *loop = taosMemoryMalloc(sizeof(uv_loop_t)); uv_loop_t *loop = taosMemoryMalloc(sizeof(uv_loop_t));
if (loop == NULL) { if (loop == NULL) {
fnError("udfd init uv loop failed, mem overflow"); fnError("udfd init uv loop failed, mem overflow");
return -1; return terrno;
} }
global.loop = loop; global.loop = loop;
if (uv_mutex_init(&global.scriptPluginsMutex) != 0) { if (uv_mutex_init(&global.scriptPluginsMutex) != 0) {
fnError("udfd init script plugins mutex failed"); fnError("udfd init script plugins mutex failed");
return -1; return TSDB_CODE_UDF_UV_EXEC_FAILURE;
} }
global.udfsHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK); global.udfsHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
@ -1494,7 +1552,7 @@ static int32_t udfdGlobalDataInit() {
if (uv_mutex_init(&global.udfsMutex) != 0) { if (uv_mutex_init(&global.udfsMutex) != 0) {
fnError("udfd init udfs mutex failed"); fnError("udfd init udfs mutex failed");
return -2; return TSDB_CODE_UDF_UV_EXEC_FAILURE;
} }
return 0; return 0;

8
source/libs/qcom/src/querymsg.c
View File

@ -34,7 +34,7 @@ int32_t queryBuildUseDbOutput(SUseDbOutput *pOut, SUseDbRsp *usedbRsp) {
pOut->dbVgroup = taosMemoryCalloc(1, sizeof(SDBVgInfo)); pOut->dbVgroup = taosMemoryCalloc(1, sizeof(SDBVgInfo));
if (NULL == pOut->dbVgroup) { if (NULL == pOut->dbVgroup) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
pOut->dbVgroup->vgVersion = usedbRsp->vgVersion; pOut->dbVgroup->vgVersion = usedbRsp->vgVersion;
@ -509,7 +509,7 @@ int32_t queryCreateTableMetaFromMsg(STableMetaRsp *msg, bool isStb, STableMeta *
STableMeta *pTableMeta = taosMemoryCalloc(1, metaSize + schemaExtSize); STableMeta *pTableMeta = taosMemoryCalloc(1, metaSize + schemaExtSize);
if (NULL == pTableMeta) { if (NULL == pTableMeta) {
qError("calloc size[%d] failed", metaSize); qError("calloc size[%d] failed", metaSize);
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
SSchemaExt *pSchemaExt = (SSchemaExt *)((char *)pTableMeta + metaSize); SSchemaExt *pSchemaExt = (SSchemaExt *)((char *)pTableMeta + metaSize);
@ -764,7 +764,7 @@ int32_t queryProcessGetTbCfgRsp(void *output, char *msg, int32_t msgSize) {
STableCfgRsp *out = taosMemoryCalloc(1, sizeof(STableCfgRsp)); STableCfgRsp *out = taosMemoryCalloc(1, sizeof(STableCfgRsp));
if(out == NULL) { if(out == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
if (tDeserializeSTableCfgRsp(msg, msgSize, out) != 0) { if (tDeserializeSTableCfgRsp(msg, msgSize, out) != 0) {
qError("tDeserializeSTableCfgRsp failed, msgSize:%d", msgSize); qError("tDeserializeSTableCfgRsp failed, msgSize:%d", msgSize);
@ -785,7 +785,7 @@ int32_t queryProcessGetViewMetaRsp(void *output, char *msg, int32_t msgSize) {
SViewMetaRsp *out = taosMemoryCalloc(1, sizeof(SViewMetaRsp)); SViewMetaRsp *out = taosMemoryCalloc(1, sizeof(SViewMetaRsp));
if (out == NULL) { if (out == NULL) {
return TSDB_CODE_OUT_OF_MEMORY; return terrno;
} }
if (tDeserializeSViewMetaRsp(msg, msgSize, out) != 0) { if (tDeserializeSViewMetaRsp(msg, msgSize, out) != 0) {
qError("tDeserializeSViewMetaRsp failed, msgSize:%d", msgSize); qError("tDeserializeSViewMetaRsp failed, msgSize:%d", msgSize);

40
source/libs/scalar/src/filter.c
View File

@ -2343,7 +2343,7 @@ _return:
(void)filterFreeRangeCtx(ctx); // No need to handle the return value. (void)filterFreeRangeCtx(ctx); // No need to handle the return value.
return TSDB_CODE_SUCCESS; return code;
} }
int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t *gResNum) { int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t *gResNum) {
@ -2671,7 +2671,7 @@ _return:
(void)filterFreeRangeCtx(ctx); // No need to handle the return value. (void)filterFreeRangeCtx(ctx); // No need to handle the return value.
return TSDB_CODE_SUCCESS; return code;
} }
int32_t filterMergeGroups(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t *gResNum) { int32_t filterMergeGroups(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t *gResNum) {
@ -2758,7 +2758,7 @@ _return:
FILTER_SET_FLAG(info->status, FI_STATUS_ALL); FILTER_SET_FLAG(info->status, FI_STATUS_ALL);
return TSDB_CODE_SUCCESS; return code;
} }
int32_t filterConvertGroupFromArray(SFilterInfo *info, SArray *group) { int32_t filterConvertGroupFromArray(SFilterInfo *info, SArray *group) {
@ -2958,7 +2958,7 @@ _return:
taosMemoryFreeClear(idxNum); taosMemoryFreeClear(idxNum);
taosMemoryFreeClear(idxs); taosMemoryFreeClear(idxs);
return TSDB_CODE_SUCCESS; return code;
} }
int32_t filterPostProcessRange(SFilterInfo *info) { int32_t filterPostProcessRange(SFilterInfo *info) {
@ -3601,12 +3601,12 @@ int32_t filterPreprocess(SFilterInfo *info) {
if (FILTER_GET_FLAG(info->status, FI_STATUS_ALL)) { if (FILTER_GET_FLAG(info->status, FI_STATUS_ALL)) {
fltInfo("Final - FilterInfo: [ALL]"); fltInfo("Final - FilterInfo: [ALL]");
goto _return; goto _return1;
} }
if (FILTER_GET_FLAG(info->status, FI_STATUS_EMPTY)) { if (FILTER_GET_FLAG(info->status, FI_STATUS_EMPTY)) {
fltInfo("Final - FilterInfo: [EMPTY]"); fltInfo("Final - FilterInfo: [EMPTY]");
goto _return; goto _return1;
} }
FLT_ERR_JRET(filterGenerateColRange(info, gRes, gResNum)); FLT_ERR_JRET(filterGenerateColRange(info, gRes, gResNum));
@ -3619,10 +3619,10 @@ int32_t filterPreprocess(SFilterInfo *info) {
FLT_ERR_JRET(filterGenerateComInfo(info)); FLT_ERR_JRET(filterGenerateComInfo(info));
_return1:
FLT_ERR_JRET(filterSetExecFunc(info));
_return: _return:
FLT_ERR_RET(filterSetExecFunc(info));
for (int32_t i = 0; i < gResNum; ++i) { for (int32_t i = 0; i < gResNum; ++i) {
filterFreeGroupCtx(gRes[i]); filterFreeGroupCtx(gRes[i]);
} }
@ -3660,15 +3660,25 @@ int32_t fltInitFromNode(SNode *tree, SFilterInfo *info, uint32_t options) {
FLT_ERR_JRET(terrno); FLT_ERR_JRET(terrno);
} }
FLT_ERR_JRET(filterInitUnitsFields(info)); code = filterInitUnitsFields(info);
if(TSDB_CODE_SUCCESS != code) {
taosArrayDestroy(group);
goto _return;
}
SFltBuildGroupCtx tctx = {.info = info, .group = group}; SFltBuildGroupCtx tctx = {.info = info, .group = group};
nodesWalkExpr(tree, fltTreeToGroup, (void *)&tctx); nodesWalkExpr(tree, fltTreeToGroup, (void *)&tctx);
FLT_ERR_JRET(tctx.code); if (TSDB_CODE_SUCCESS != tctx.code) {
taosArrayDestroy(group);
FLT_ERR_JRET(filterConvertGroupFromArray(info, group)); code = tctx.code;
goto _return;
}
code = filterConvertGroupFromArray(info, group);
if (TSDB_CODE_SUCCESS != code) {
taosArrayDestroy(group);
goto _return;
}
taosArrayDestroy(group); taosArrayDestroy(group);
FLT_ERR_JRET(fltInitValFieldData(info)); FLT_ERR_JRET(fltInitValFieldData(info));
if (!FILTER_GET_FLAG(info->options, FLT_OPTION_NO_REWRITE)) { if (!FILTER_GET_FLAG(info->options, FLT_OPTION_NO_REWRITE)) {
@ -4993,7 +5003,7 @@ int32_t fltOptimizeNodes(SFilterInfo *pInfo, SNode **pNode, SFltTreeStat *pStat)
} }
_return: _return:
taosArrayDestroy(sclOpList); taosArrayDestroy(sclOpList);
return TSDB_CODE_SUCCESS; return code;
} }
int32_t fltGetDataFromColId(void *param, int32_t id, void **data) { int32_t fltGetDataFromColId(void *param, int32_t id, void **data) {

3
source/libs/scalar/src/scalar.c
View File

@ -904,9 +904,8 @@ int32_t sclExecOperator(SOperatorNode *node, SScalarCtx *ctx, SScalarParam *outp
terrno = TSDB_CODE_SUCCESS; terrno = TSDB_CODE_SUCCESS;
SCL_ERR_JRET(OperatorFn(pLeft, pRight, output, TSDB_ORDER_ASC)); SCL_ERR_JRET(OperatorFn(pLeft, pRight, output, TSDB_ORDER_ASC));
SCL_ERR_JRET(terrno);
_return:
_return:
sclFreeParamList(params, paramNum); sclFreeParamList(params, paramNum);
SCL_RET(code); SCL_RET(code);
} }

2
source/libs/scalar/src/sclfunc.c
View File

@ -2913,7 +2913,7 @@ int32_t histogramScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarP
_return: _return:
taosMemoryFree(bins); taosMemoryFree(bins);
return TSDB_CODE_SUCCESS; return code;
} }
int32_t selectScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput) { int32_t selectScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput) {

4
source/libs/sync/test/sync_test_lib/src/syncIO.c
View File

@ -193,7 +193,7 @@ static int32_t syncIOStartInternal(SSyncIO *io) {
io->clientRpc = rpcOpen(&rpcInit); io->clientRpc = rpcOpen(&rpcInit);
if (io->clientRpc == NULL) { if (io->clientRpc == NULL) {
sError("failed to initialize RPC"); sError("failed to initialize RPC");
return -1; return terrno;
} }
} }
@ -214,7 +214,7 @@ static int32_t syncIOStartInternal(SSyncIO *io) {
void *pRpc = rpcOpen(&rpcInit); void *pRpc = rpcOpen(&rpcInit);
if (pRpc == NULL) { if (pRpc == NULL) {
sError("failed to start RPC server"); sError("failed to start RPC server");
return -1; return terrno;
} }
} }

18
source/libs/tdb/src/db/tdbBtree.c
View File

@ -194,7 +194,11 @@ int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, in
int idx; int idx;
int c; int c;
(void)tdbBtcOpen(&btc, pBt, pTxn); ret = tdbBtcOpen(&btc, pBt, pTxn);
if (ret) {
tdbError("tdb/btree-insert: btc open failed with ret: %d.", ret);
return ret;
}
tdbTrace("tdb insert, btc: %p, pTxn: %p", &btc, pTxn); tdbTrace("tdb insert, btc: %p, pTxn: %p", &btc, pTxn);
@ -235,7 +239,11 @@ int tdbBtreeDelete(SBTree *pBt, const void *pKey, int kLen, TXN *pTxn) {
int c; int c;
int ret; int ret;
(void)tdbBtcOpen(&btc, pBt, pTxn); ret = tdbBtcOpen(&btc, pBt, pTxn);
if (ret) {
tdbError("tdb/btree-delete: btc open failed with ret: %d.", ret);
return ret;
}
/* /*
btc.coder.ofps = taosArrayInit(8, sizeof(SPage *)); btc.coder.ofps = taosArrayInit(8, sizeof(SPage *));
// btc.coder.ofps = taosArrayInit(8, sizeof(SPgno)); // btc.coder.ofps = taosArrayInit(8, sizeof(SPgno));
@ -337,7 +345,11 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL
void *pTVal = NULL; void *pTVal = NULL;
SCellDecoder cd = {0}; SCellDecoder cd = {0};
(void)tdbBtcOpen(&btc, pBt, NULL); ret = tdbBtcOpen(&btc, pBt, NULL);
if (ret) {
tdbError("tdb/btree-pget: btc open failed with ret: %d.", ret);
return ret;
}
tdbTrace("tdb pget, btc: %p", &btc); tdbTrace("tdb pget, btc: %p", &btc);

2
source/libs/transport/test/cliBench.c
View File

@ -160,7 +160,7 @@ int main(int argc, char *argv[]) {
void *pRpc = rpcOpen(&rpcInit); void *pRpc = rpcOpen(&rpcInit);
if (pRpc == NULL) { if (pRpc == NULL) {
tError("failed to initialize RPC"); tError("failed to initialize RPC");
return -1; return terrno;
} }
tInfo("client is initialized"); tInfo("client is initialized");

117
source/util/src/tcompare.c
View File

@ -1208,20 +1208,28 @@ typedef struct UsingRegex {
regex_t pRegex; regex_t pRegex;
int32_t lastUsedTime; int32_t lastUsedTime;
} UsingRegex; } UsingRegex;
typedef UsingRegex* HashRegexPtr;
typedef struct RegexCache { typedef struct RegexCache {
SHashObj *regexHash; SHashObj *regexHash;
void *regexCacheTmr; void *regexCacheTmr;
void *timer; void *timer;
SRWLatch mutex;
bool exit;
} RegexCache; } RegexCache;
static RegexCache sRegexCache; static RegexCache sRegexCache;
#define MAX_REGEX_CACHE_SIZE 20 #define MAX_REGEX_CACHE_SIZE 20
#define REGEX_CACHE_CLEAR_TIME 30 #define REGEX_CACHE_CLEAR_TIME 30
static void checkRegexCache(void* param, void* tmrId) { static void checkRegexCache(void* param, void* tmrId) {
int32_t code = 0;
taosRLockLatch(&sRegexCache.mutex);
if(sRegexCache.exit) {
goto _exit;
}
(void)taosTmrReset(checkRegexCache, REGEX_CACHE_CLEAR_TIME * 1000, param, sRegexCache.regexCacheTmr, &tmrId); (void)taosTmrReset(checkRegexCache, REGEX_CACHE_CLEAR_TIME * 1000, param, sRegexCache.regexCacheTmr, &tmrId);
if (taosHashGetSize(sRegexCache.regexHash) < MAX_REGEX_CACHE_SIZE) { if (taosHashGetSize(sRegexCache.regexHash) < MAX_REGEX_CACHE_SIZE) {
return; goto _exit;
} }
if (taosHashGetSize(sRegexCache.regexHash) >= MAX_REGEX_CACHE_SIZE) { if (taosHashGetSize(sRegexCache.regexHash) >= MAX_REGEX_CACHE_SIZE) {
@ -1235,6 +1243,8 @@ static void checkRegexCache(void* param, void* tmrId) {
ppUsingRegex = taosHashIterate(sRegexCache.regexHash, ppUsingRegex); ppUsingRegex = taosHashIterate(sRegexCache.regexHash, ppUsingRegex);
} }
} }
_exit:
taosRUnLockLatch(&sRegexCache.mutex);
} }
void regexCacheFree(void *ppUsingRegex) { void regexCacheFree(void *ppUsingRegex) {
@ -1246,30 +1256,35 @@ int32_t InitRegexCache() {
sRegexCache.regexHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK); sRegexCache.regexHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK);
if (sRegexCache.regexHash == NULL) { if (sRegexCache.regexHash == NULL) {
uError("failed to create RegexCache"); uError("failed to create RegexCache");
return -1; return terrno;
} }
taosHashSetFreeFp(sRegexCache.regexHash, regexCacheFree); taosHashSetFreeFp(sRegexCache.regexHash, regexCacheFree);
sRegexCache.regexCacheTmr = taosTmrInit(0, 0, 0, "REGEXCACHE"); sRegexCache.regexCacheTmr = taosTmrInit(0, 0, 0, "REGEXCACHE");
if (sRegexCache.regexCacheTmr == NULL) { if (sRegexCache.regexCacheTmr == NULL) {
uError("failed to create regex cache check timer"); uError("failed to create regex cache check timer");
terrno = TSDB_CODE_OUT_OF_MEMORY; return terrno;
return -1;
} }
sRegexCache.exit = false;
taosInitRWLatch(&sRegexCache.mutex);
sRegexCache.timer = taosTmrStart(checkRegexCache, REGEX_CACHE_CLEAR_TIME * 1000, NULL, sRegexCache.regexCacheTmr); sRegexCache.timer = taosTmrStart(checkRegexCache, REGEX_CACHE_CLEAR_TIME * 1000, NULL, sRegexCache.regexCacheTmr);
if (sRegexCache.timer == NULL) { if (sRegexCache.timer == NULL) {
uError("failed to start regex cache timer"); uError("failed to start regex cache timer");
return -1; return terrno;
} }
return 0; return TSDB_CODE_SUCCESS;
} }
void DestroyRegexCache(){ void DestroyRegexCache(){
int32_t code = 0;
uInfo("[regex cache] destory regex cache"); uInfo("[regex cache] destory regex cache");
(void)taosTmrStopA(&sRegexCache.timer); (void)taosTmrStopA(&sRegexCache.timer);
taosWLockLatch(&sRegexCache.mutex);
sRegexCache.exit = true;
taosHashCleanup(sRegexCache.regexHash); taosHashCleanup(sRegexCache.regexHash);
taosTmrCleanUp(sRegexCache.regexCacheTmr); taosTmrCleanUp(sRegexCache.regexCacheTmr);
taosWUnLockLatch(&sRegexCache.mutex);
} }
int32_t checkRegexPattern(const char *pPattern) { int32_t checkRegexPattern(const char *pPattern) {
@ -1290,18 +1305,17 @@ int32_t checkRegexPattern(const char *pPattern) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
static UsingRegex **getRegComp(const char *pPattern) { int32_t getRegComp(const char *pPattern, HashRegexPtr **regexRet) {
UsingRegex **ppUsingRegex = (UsingRegex **)taosHashAcquire(sRegexCache.regexHash, pPattern, strlen(pPattern)); HashRegexPtr* ppUsingRegex = (HashRegexPtr*)taosHashAcquire(sRegexCache.regexHash, pPattern, strlen(pPattern));
if (ppUsingRegex != NULL) { if (ppUsingRegex != NULL) {
(*ppUsingRegex)->lastUsedTime = taosGetTimestampSec(); (*ppUsingRegex)->lastUsedTime = taosGetTimestampSec();
return ppUsingRegex; *regexRet = ppUsingRegex;
return TSDB_CODE_SUCCESS;
} }
UsingRegex *pUsingRegex = taosMemoryMalloc(sizeof(UsingRegex)); UsingRegex *pUsingRegex = taosMemoryMalloc(sizeof(UsingRegex));
if (pUsingRegex == NULL) { if (pUsingRegex == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
uError("Failed to Malloc when compile regex pattern %s.", pPattern); uError("Failed to Malloc when compile regex pattern %s.", pPattern);
return NULL; return terrno;
} }
int32_t cflags = REG_EXTENDED; int32_t cflags = REG_EXTENDED;
int32_t ret = regcomp(&pUsingRegex->pRegex, pPattern, cflags); int32_t ret = regcomp(&pUsingRegex->pRegex, pPattern, cflags);
@ -1310,8 +1324,7 @@ static UsingRegex **getRegComp(const char *pPattern) {
(void)regerror(ret, &pUsingRegex->pRegex, msgbuf, tListLen(msgbuf)); (void)regerror(ret, &pUsingRegex->pRegex, msgbuf, tListLen(msgbuf));
uError("Failed to compile regex pattern %s. reason %s", pPattern, msgbuf); uError("Failed to compile regex pattern %s. reason %s", pPattern, msgbuf);
taosMemoryFree(pUsingRegex); taosMemoryFree(pUsingRegex);
terrno = TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR; return TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR;
return NULL;
} }
while (true) { while (true) {
@ -1319,8 +1332,9 @@ static UsingRegex **getRegComp(const char *pPattern) {
if (code != 0 && code != TSDB_CODE_DUP_KEY) { if (code != 0 && code != TSDB_CODE_DUP_KEY) {
regexCacheFree(&pUsingRegex); regexCacheFree(&pUsingRegex);
uError("Failed to put regex pattern %s into cache, exception internal error.", pPattern); uError("Failed to put regex pattern %s into cache, exception internal error.", pPattern);
terrno = code; return code;
return NULL; } else if (code == TSDB_CODE_DUP_KEY) {
terrno = 0;
} }
ppUsingRegex = (UsingRegex **)taosHashAcquire(sRegexCache.regexHash, pPattern, strlen(pPattern)); ppUsingRegex = (UsingRegex **)taosHashAcquire(sRegexCache.regexHash, pPattern, strlen(pPattern));
if (ppUsingRegex) { if (ppUsingRegex) {
@ -1334,27 +1348,68 @@ static UsingRegex **getRegComp(const char *pPattern) {
} }
} }
pUsingRegex->lastUsedTime = taosGetTimestampSec(); pUsingRegex->lastUsedTime = taosGetTimestampSec();
return ppUsingRegex; *regexRet = ppUsingRegex;
return TSDB_CODE_SUCCESS;
} }
void releaseRegComp(UsingRegex **regex){ void releaseRegComp(UsingRegex **regex){
taosHashRelease(sRegexCache.regexHash, regex); taosHashRelease(sRegexCache.regexHash, regex);
} }
static threadlocal UsingRegex ** ppUsingRegex;
static threadlocal regex_t * pRegex;
static threadlocal char *pOldPattern = NULL;
void DestoryThreadLocalRegComp() {
if (NULL != pOldPattern) {
releaseRegComp(ppUsingRegex);
taosMemoryFree(pOldPattern);
ppUsingRegex = NULL;
pRegex = NULL;
pOldPattern = NULL;
}
}
int32_t threadGetRegComp(regex_t **regex, const char *pPattern) {
if (NULL != pOldPattern) {
if (strcmp(pOldPattern, pPattern) == 0) {
*regex = pRegex;
return 0;
} else {
DestoryThreadLocalRegComp();
}
}
HashRegexPtr *ppRegex = NULL;
int32_t code = getRegComp(pPattern, &ppRegex);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
pOldPattern = taosStrdup(pPattern);
if (NULL == pOldPattern) {
uError("Failed to Malloc when compile regex pattern %s.", pPattern);
return terrno;
}
ppUsingRegex = ppRegex;
pRegex = &((*ppUsingRegex)->pRegex);
*regex = &(*ppRegex)->pRegex;
return 0;
}
static int32_t doExecRegexMatch(const char *pString, const char *pPattern) { static int32_t doExecRegexMatch(const char *pString, const char *pPattern) {
int32_t ret = 0; int32_t ret = 0;
char msgbuf[256] = {0}; char msgbuf[256] = {0};
UsingRegex **pUsingRegex = getRegComp(pPattern);
if (pUsingRegex == NULL) { regex_t *regex = NULL;
return 1; ret = threadGetRegComp(&regex, pPattern);
if (ret != 0) {
return ret;
} }
regmatch_t pmatch[1]; regmatch_t pmatch[1];
ret = regexec(&(*pUsingRegex)->pRegex, pString, 1, pmatch, 0); ret = regexec(regex, pString, 1, pmatch, 0);
releaseRegComp(pUsingRegex);
if (ret != 0 && ret != REG_NOMATCH) { if (ret != 0 && ret != REG_NOMATCH) {
terrno = TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR; terrno = TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR;
(void)regerror(ret, &(*pUsingRegex)->pRegex, msgbuf, sizeof(msgbuf)); (void)regerror(ret, regex, msgbuf, sizeof(msgbuf));
uDebug("Failed to match %s with pattern %s, reason %s", pString, pPattern, msgbuf) uDebug("Failed to match %s with pattern %s, reason %s", pString, pPattern, msgbuf)
} }
@ -1365,8 +1420,7 @@ int32_t comparestrRegexMatch(const void *pLeft, const void *pRight) {
size_t sz = varDataLen(pRight); size_t sz = varDataLen(pRight);
char *pattern = taosMemoryMalloc(sz + 1); char *pattern = taosMemoryMalloc(sz + 1);
if (NULL == pattern) { if (NULL == pattern) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return 1; // terrno has been set
return 1;
} }
(void)memcpy(pattern, varDataVal(pRight), varDataLen(pRight)); (void)memcpy(pattern, varDataVal(pRight), varDataLen(pRight));
@ -1376,8 +1430,7 @@ int32_t comparestrRegexMatch(const void *pLeft, const void *pRight) {
char *str = taosMemoryMalloc(sz + 1); char *str = taosMemoryMalloc(sz + 1);
if (NULL == str) { if (NULL == str) {
taosMemoryFree(pattern); taosMemoryFree(pattern);
terrno = TSDB_CODE_OUT_OF_MEMORY; return 1; // terrno has been set
return 1;
} }
(void)memcpy(str, varDataVal(pLeft), sz); (void)memcpy(str, varDataVal(pLeft), sz);
@ -1395,14 +1448,13 @@ int32_t comparewcsRegexMatch(const void *pString, const void *pPattern) {
size_t len = varDataLen(pPattern); size_t len = varDataLen(pPattern);
char *pattern = taosMemoryMalloc(len + 1); char *pattern = taosMemoryMalloc(len + 1);
if (NULL == pattern) { if (NULL == pattern) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return 1; // terrno has been set
return 1;
} }
int convertLen = taosUcs4ToMbs((TdUcs4 *)varDataVal(pPattern), len, pattern); int convertLen = taosUcs4ToMbs((TdUcs4 *)varDataVal(pPattern), len, pattern);
if (convertLen < 0) { if (convertLen < 0) {
taosMemoryFree(pattern); taosMemoryFree(pattern);
return (terrno = TSDB_CODE_APP_ERROR); return 1; // terrno has been set
} }
pattern[convertLen] = 0; pattern[convertLen] = 0;
@ -1411,15 +1463,14 @@ int32_t comparewcsRegexMatch(const void *pString, const void *pPattern) {
char *str = taosMemoryMalloc(len + 1); char *str = taosMemoryMalloc(len + 1);
if (NULL == str) { if (NULL == str) {
taosMemoryFree(pattern); taosMemoryFree(pattern);
terrno = TSDB_CODE_OUT_OF_MEMORY; return 1; // terrno has been set
return 1;
} }
convertLen = taosUcs4ToMbs((TdUcs4 *)varDataVal(pString), len, str); convertLen = taosUcs4ToMbs((TdUcs4 *)varDataVal(pString), len, str);
if (convertLen < 0) { if (convertLen < 0) {
taosMemoryFree(str); taosMemoryFree(str);
taosMemoryFree(pattern); taosMemoryFree(pattern);
return (terrno = TSDB_CODE_APP_ERROR); return 1; // terrno has been set
} }
str[convertLen] = 0; str[convertLen] = 0;

4
source/util/src/tlog.c
View File

@ -621,7 +621,7 @@ static inline void taosPrintLogImp(ELogLevel level, int32_t dflag, const char *b
} }
} }
void taosPrintLog(const char *flags, ELogLevel level, int32_t dflag, const char *format, ...) { void taosPrintLog(const char *flags, int32_t level, int32_t dflag, const char *format, ...) {
if (!(dflag & DEBUG_FILE) && !(dflag & DEBUG_SCREEN)) return; if (!(dflag & DEBUG_FILE) && !(dflag & DEBUG_SCREEN)) return;
char buffer[LOG_MAX_LINE_BUFFER_SIZE]; char buffer[LOG_MAX_LINE_BUFFER_SIZE];
@ -644,7 +644,7 @@ void taosPrintLog(const char *flags, ELogLevel level, int32_t dflag, const char
} }
} }
void taosPrintLongString(const char *flags, ELogLevel level, int32_t dflag, const char *format, ...) { void taosPrintLongString(const char *flags, int32_t level, int32_t dflag, const char *format, ...) {
if (!osLogSpaceSufficient()) return; if (!osLogSpaceSufficient()) return;
if (!(dflag & DEBUG_FILE) && !(dflag & DEBUG_SCREEN)) return; if (!(dflag & DEBUG_FILE) && !(dflag & DEBUG_SCREEN)) return;

3
source/util/src/tworker.c
View File

@ -106,6 +106,7 @@ static void *tQWorkerThreadFp(SQueueWorker *worker) {
} }
destroyThreadLocalGeosCtx(); destroyThreadLocalGeosCtx();
DestoryThreadLocalRegComp();
return NULL; return NULL;
} }
@ -237,6 +238,7 @@ static void *tAutoQWorkerThreadFp(SQueueWorker *worker) {
taosUpdateItemSize(qinfo.queue, 1); taosUpdateItemSize(qinfo.queue, 1);
} }
DestoryThreadLocalRegComp();
return NULL; return NULL;
} }
@ -664,6 +666,7 @@ static void *tQueryAutoQWorkerThreadFp(SQueryAutoQWorker *worker) {
} }
destroyThreadLocalGeosCtx(); destroyThreadLocalGeosCtx();
DestoryThreadLocalRegComp();
return NULL; return NULL;
} }

7
source/util/test/CMakeLists.txt
View File

@ -119,6 +119,13 @@ add_test(
COMMAND bufferTest COMMAND bufferTest
) )
add_executable(regexTest "regexTest.cpp")
target_link_libraries(regexTest os util gtest_main )
add_test(
NAME regexTest
COMMAND regexTest
)
#add_executable(decompressTest "decompressTest.cpp") #add_executable(decompressTest "decompressTest.cpp")
#target_link_libraries(decompressTest os util common gtest_main) #target_link_libraries(decompressTest os util common gtest_main)
#add_test( #add_test(

344
source/util/test/regexTest.cpp Normal file
View File

@ -0,0 +1,344 @@
#include <chrono>
#include <cstdio>
#include <gtest/gtest.h>
#include <limits.h>
#include <taosdef.h>
#include "os.h"
#include "tutil.h"
#include "regex.h"
#include "osDef.h"
#include "tcompare.h"
extern "C" {
typedef struct UsingRegex UsingRegex;
typedef struct HashRegexPtr HashRegexPtr;
int32_t getRegComp(const char *pPattern, HashRegexPtr **regexRet);
int32_t threadGetRegComp(regex_t **regex, const char *pPattern);
}
class regexTest {
public:
regexTest() { (void)InitRegexCache(); }
~regexTest() { (void)DestroyRegexCache(); }
};
static regexTest test;
static threadlocal regex_t pRegex;
static threadlocal char *pOldPattern = NULL;
void DestoryThreadLocalRegComp1() {
if (NULL != pOldPattern) {
regfree(&pRegex);
taosMemoryFree(pOldPattern);
pOldPattern = NULL;
}
}
static regex_t *threadGetRegComp1(const char *pPattern) {
if (NULL != pOldPattern) {
if( strcmp(pOldPattern, pPattern) == 0) {
return &pRegex;
} else {
DestoryThreadLocalRegComp1();
}
}
pOldPattern = (char*)taosMemoryMalloc(strlen(pPattern) + 1);
if (NULL == pOldPattern) {
uError("Failed to Malloc when compile regex pattern %s.", pPattern);
return NULL;
}
strcpy(pOldPattern, pPattern);
int32_t cflags = REG_EXTENDED;
int32_t ret = regcomp(&pRegex, pPattern, cflags);
if (ret != 0) {
char msgbuf[256] = {0};
regerror(ret, &pRegex, msgbuf, tListLen(msgbuf));
uError("Failed to compile regex pattern %s. reason %s", pPattern, msgbuf);
DestoryThreadLocalRegComp1();
return NULL;
}
return &pRegex;
}
TEST(testCase, regexCacheTest1) {
int times = 100000;
char s1[] = "abc";
auto start = std::chrono::high_resolution_clock::now();
uint64_t t0 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s1, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s1;
}
}
uint64_t t1 = taosGetTimestampUs();
printf("%s regex(current) %d times:%" PRIu64 " us.\n", s1, times, t1 - t0);
uint64_t t2 = taosGetTimestampUs();
for(int i = 0; i < times; i++) {
regex_t* rex = threadGetRegComp1(s1);
}
uint64_t t3 = taosGetTimestampUs();
printf("%s regex(before) %d times:%" PRIu64 " us.\n", s1, times, t3 - t2);
t2 = taosGetTimestampUs();
for(int i = 0; i < times; i++) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s1);
}
t3 = taosGetTimestampUs();
printf("%s regex(new) %d times:%" PRIu64 " us.\n", s1, times, t3 - t2);
}
TEST(testCase, regexCacheTest2) {
int times = 100000;
char s1[] = "abc%*";
auto start = std::chrono::high_resolution_clock::now();
uint64_t t0 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s1, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s1;
}
}
uint64_t t1 = taosGetTimestampUs();
printf("%s regex(current) %d times:%" PRIu64 " us.\n", s1, times, t1 - t0);
uint64_t t2 = taosGetTimestampUs();
for(int i = 0; i < times; i++) {
regex_t* rex = threadGetRegComp1(s1);
}
uint64_t t3 = taosGetTimestampUs();
printf("%s regex(before) %d times:%" PRIu64 " us.\n", s1, times, t3 - t2);
t2 = taosGetTimestampUs();
for(int i = 0; i < times; i++) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s1);
}
t3 = taosGetTimestampUs();
printf("%s regex(new) %d times:%" PRIu64 " us.\n", s1, times, t3 - t2);
}
TEST(testCase, regexCacheTest3) {
int times = 100000;
char s1[] = "abc%*";
char s2[] = "abc";
auto start = std::chrono::high_resolution_clock::now();
uint64_t t0 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s1, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s1;
}
}
uint64_t t1 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn regex(current) %d times:%" PRIu64 " us.\n", s1, s2, times, t1 - t0);
uint64_t t2 = taosGetTimestampUs();
for(int i = 0; i < times; i++) {
regex_t* rex = threadGetRegComp1(s1);
rex = threadGetRegComp1(s2);
}
uint64_t t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn regex(before) %d times:%" PRIu64 " us.\n", s1, s2, times, t3 - t2);
t2 = taosGetTimestampUs();
for(int i = 0; i < times; i++) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s1);
(void)threadGetRegComp(&rex, s2);
}
t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn regex(new) %d times:%" PRIu64 " us.\n", s1, s2, times, t3 - t2);
}
TEST(testCase, regexCacheTest4) {
int times = 100;
int count = 1000;
char s1[] = "abc%*";
char s2[] = "abc";
auto start = std::chrono::high_resolution_clock::now();
uint64_t t0 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s1, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s1;
}
}
for (int j = 0; j < count; ++j) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s2, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s2;
}
}
}
uint64_t t1 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(current) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t1 - t0);
uint64_t t2 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
regex_t* rex = threadGetRegComp1(s1);
}
for (int j = 0; j < count; ++j) {
regex_t* rex = threadGetRegComp1(s2);
}
}
uint64_t t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(before) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t3 - t2);
t2 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s1);
}
for (int j = 0; j < count; ++j) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s2);
}
}
t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(new) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t3 - t2);
}
// It is not a good idea to test this case, because it will take a long time.
/*
TEST(testCase, regexCacheTest5) {
int times = 10000;
int count = 10000;
char s1[] = "abc%*";
char s2[] = "abc";
auto start = std::chrono::high_resolution_clock::now();
uint64_t t0 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s1, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s1;
}
}
for (int j = 0; j < count; ++j) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s2, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s2;
}
}
}
uint64_t t1 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(current) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t1 - t0);
uint64_t t2 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
regex_t* rex = threadGetRegComp1(s1);
}
for (int j = 0; j < count; ++j) {
regex_t* rex = threadGetRegComp1(s2);
}
}
uint64_t t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(before) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t3 - t2);
t2 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s1);
}
for (int j = 0; j < count; ++j) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s2);
}
}
t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(new) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t3 - t2);
}
TEST(testCase, regexCacheTest6) {
int times = 10000;
int count = 1000;
char s1[] = "abc%*";
char s2[] = "abc";
auto start = std::chrono::high_resolution_clock::now();
uint64_t t0 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s1, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s1;
}
}
for (int j = 0; j < count; ++j) {
HashRegexPtr* ret = NULL;
int32_t code = getRegComp(s2, &ret);
if (code != 0) {
FAIL() << "Failed to compile regex pattern " << s2;
}
}
}
uint64_t t1 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(current) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t1 - t0);
uint64_t t2 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
regex_t* rex = threadGetRegComp1(s1);
}
for (int j = 0; j < count; ++j) {
regex_t* rex = threadGetRegComp1(s2);
}
}
uint64_t t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(before) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t3 - t2);
t2 = taosGetTimestampUs();
for (int i = 0; i < times; i++) {
for (int j = 0; j < count; ++j) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s1);
}
for (int j = 0; j < count; ++j) {
regex_t* rex = NULL;
(void)threadGetRegComp(&rex, s2);
}
}
t3 = taosGetTimestampUs();
printf("'%s' and '%s' take place by turn(per %d count) regex(new) %d times:%" PRIu64 " us.\n", s1, s2, count, times, t3 - t2);
}
*/

18
tests/script/sh/bit_and.c
View File

@ -8,13 +8,17 @@ DLL_EXPORT int32_t bit_and_init() { return 0; }
DLL_EXPORT int32_t bit_and_destroy() { return 0; } DLL_EXPORT int32_t bit_and_destroy() { return 0; }
DLL_EXPORT int32_t bit_and(SUdfDataBlock* block, SUdfColumn* resultCol) { DLL_EXPORT int32_t bit_and(SUdfDataBlock* block, SUdfColumn* resultCol) {
udfTrace("block:%p, processing begins, rows:%d cols:%d", block, block->numOfRows, block->numOfCols);
if (block->numOfCols < 2) { if (block->numOfCols < 2) {
udfError("block:%p, cols:%d needs to be greater than 2", block, block->numOfCols);
return TSDB_CODE_UDF_INVALID_INPUT; return TSDB_CODE_UDF_INVALID_INPUT;
} }
for (int32_t i = 0; i < block->numOfCols; ++i) { for (int32_t i = 0; i < block->numOfCols; ++i) {
SUdfColumn* col = block->udfCols[i]; SUdfColumn* col = block->udfCols[i];
if (!(col->colMeta.type == TSDB_DATA_TYPE_INT)) { if (col->colMeta.type != TSDB_DATA_TYPE_INT) {
udfError("block:%p, col:%d type:%d should be int(%d)", block, i, col->colMeta.type, TSDB_DATA_TYPE_INT);
return TSDB_CODE_UDF_INVALID_INPUT; return TSDB_CODE_UDF_INVALID_INPUT;
} }
} }
@ -23,25 +27,35 @@ DLL_EXPORT int32_t bit_and(SUdfDataBlock* block, SUdfColumn* resultCol) {
for (int32_t i = 0; i < block->numOfRows; ++i) { for (int32_t i = 0; i < block->numOfRows; ++i) {
if (udfColDataIsNull(block->udfCols[0], i)) { if (udfColDataIsNull(block->udfCols[0], i)) {
udfTrace("block:%p, row:%d result is null since col:0 is null", block, i);
udfColDataSetNull(resultCol, i); udfColDataSetNull(resultCol, i);
continue; continue;
} }
int32_t result = *(int32_t*)udfColDataGetData(block->udfCols[0], i); int32_t result = *(int32_t*)udfColDataGetData(block->udfCols[0], i);
int j = 1; udfTrace("block:%p, row:%d col:0 data:%d", block, i, result);
int32_t j = 1;
for (; j < block->numOfCols; ++j) { for (; j < block->numOfCols; ++j) {
if (udfColDataIsNull(block->udfCols[j], i)) { if (udfColDataIsNull(block->udfCols[j], i)) {
udfTrace("block:%p, row:%d result is null since col:%d is null", block, i, j);
udfColDataSetNull(resultCol, i); udfColDataSetNull(resultCol, i);
break; break;
} }
char* colData = udfColDataGetData(block->udfCols[j], i); char* colData = udfColDataGetData(block->udfCols[j], i);
result &= *(int32_t*)colData; result &= *(int32_t*)colData;
udfTrace("block:%p, row:%d col:%d data:%d", block, i, j, *(int32_t*)colData);
} }
if (j == block->numOfCols) { if (j == block->numOfCols) {
udfColDataSet(resultCol, i, (char*)&result, false); udfColDataSet(resultCol, i, (char*)&result, false);
udfTrace("block:%p, row:%d result is %d", block, i, result);
} }
} }
resultData->numOfRows = block->numOfRows; resultData->numOfRows = block->numOfRows;
udfTrace("block:%p, processing completed, rows:%d, cols:%d,", block, block->numOfRows, block->numOfCols);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }

2
tests/system-test/7-tmq/tmqParamsTest.py
View File

@ -104,7 +104,7 @@ class TDTestCase:
stop_flag = 0 stop_flag = 0
try: try:
while True: while True:
res = consumer.poll(1) res = consumer.poll(3)
tdSql.query('show consumers;') tdSql.query('show consumers;')
consumer_info = tdSql.queryResult[0][-1] consumer_info = tdSql.queryResult[0][-1]
if offset_value == "latest": if offset_value == "latest":