diff --git a/docs/zh/06-advanced/03-stream.md b/docs/zh/06-advanced/03-stream.md index 548fdc1821..a219f86750 100644 --- a/docs/zh/06-advanced/03-stream.md +++ b/docs/zh/06-advanced/03-stream.md @@ -72,7 +72,7 @@ SELECT _wstart, count(*), avg(voltage) FROM power.meters PARTITION BY tbname INT - stb_name 是保存计算结果的超级表的表名，如果该超级表不存在，则会自动创建；如果已存在，则检查列的 schema 信息。详见 6.3.8 节。 - tags 子句定义了流计算中创建标签的规则。通过 tags 字段可以为每个分区对应的子表生成自定义的标签值。 -## 流试计算的规则和策略 +## 流式计算的规则和策略 ### 流计算的分区 diff --git a/docs/zh/06-advanced/05-data-in/07-mqtt.md b/docs/zh/06-advanced/05-data-in/07-mqtt.md index 84f31636b2..eb6753b7b0 100644 --- a/docs/zh/06-advanced/05-data-in/07-mqtt.md +++ b/docs/zh/06-advanced/05-data-in/07-mqtt.md @@ -55,7 +55,7 @@ TDengine 可以通过 MQTT 连接器从 MQTT 代理订阅数据并将其写入 T 在 **MQTT 协议** 下拉列表中选择 MQTT 协议版本。有三个选项：`3.1`、`3.1.1`、`5.0`。 默认值为 3.1。 -在 **Client ID** 中填写客户端 ID。建多个任务时，同个 MQTT 地址下，Client ID 必须是唯一的。 +在 **Client ID** 中填写客户端标识，填写后会生成带有 `taosx` 前缀的客户端 id （例如，如果填写的标识为 `foo`，则生成的客户端 id 为 `taosxfoo`）。如果打开末尾处的开关，则会把当前任务的任务 id 拼接到 `taosx` 之后，输入的标识之前（生成的客户端 id 形如 `taosx100foo`）。连接到同一个 MQTT 地址的所有客户端 id 必须保证唯一。 在 **Keep Alive** 中输入保持活动间隔。如果代理在保持活动间隔内没有收到来自客户端的任何消息，它将假定客户端已断开连接，并关闭连接。 保持活动间隔是指客户端和代理之间协商的时间间隔，用于检测客户端是否活动。如果客户端在保持活动间隔内没有向代理发送消息，则代理将断开连接。 diff --git a/docs/zh/06-advanced/05-data-in/08-kafka.md b/docs/zh/06-advanced/05-data-in/08-kafka.md index dd5512d615..92bfc031ec 100644 --- a/docs/zh/06-advanced/05-data-in/08-kafka.md +++ b/docs/zh/06-advanced/05-data-in/08-kafka.md @@ -60,6 +60,10 @@ TDengine 可以高效地从 Kafka 读取数据并将其写入 TDengine，以实 在 **主题** 中填写要消费的 Topic 名称。可以配置多个 Topic ， Topic 之间用逗号分隔。例如：`tp1,tp2`。 +在 **Client ID** 中填写客户端标识，填写后会生成带有 `taosx` 前缀的客户端 ID （例如，如果填写的标识为 `foo`，则生成的客户端 ID 为 `taosxfoo`）。如果打开末尾处的开关，则会把当前任务的任务 ID 拼接到 `taosx` 之后，输入的标识之前（生成的客户端 ID 形如 `taosx100foo`）。连接到同一个 Kafka 集群的所有客户端 ID 必须保证唯一。 + +在 **消费者组 ID** 中填写消费者组标识，填写后会生成带有 `taosx` 前缀的消费者组 ID （例如，如果填写的标识为 `foo`，则生成的消费者组 ID 为 `taosxfoo`）。如果打开末尾处的开关，则会把当前任务的任务 ID 拼接到 `taosx` 之后，输入的标识之前（生成的消费者组 ID 形如 `taosx100foo`）。 + 在 **Offset** 的下拉列表中选择从哪个 Offset 开始消费数据。有三个选项：`Earliest`、`Latest`、`ByTime(ms)`。 默认值为Earliest。 * Earliest：用于请求最早的 offset。 diff --git a/docs/zh/06-advanced/05-data-in/kafka-06.png b/docs/zh/06-advanced/05-data-in/kafka-06.png index 5e3d414025..43efe834a6 100644 Binary files a/docs/zh/06-advanced/05-data-in/kafka-06.png and b/docs/zh/06-advanced/05-data-in/kafka-06.png differ diff --git a/docs/zh/06-advanced/05-data-in/mqtt-05.png b/docs/zh/06-advanced/05-data-in/mqtt-05.png index e1f6bfd04d..d362e8be86 100644 Binary files a/docs/zh/06-advanced/05-data-in/mqtt-05.png and b/docs/zh/06-advanced/05-data-in/mqtt-05.png differ diff --git a/docs/zh/07-operation/06-gui.md b/docs/zh/07-operation/06-gui.md index 0caecf527c..bb99b2cf43 100644 --- a/docs/zh/07-operation/06-gui.md +++ b/docs/zh/07-operation/06-gui.md @@ -229,4 +229,12 @@ toc_max_heading_level: 4 - 权限 - 白名单 -![management-01-importInfo.jpeg](./pic/management-01-importInfo.jpeg)。 \ No newline at end of file +![management-01-importInfo.jpeg](./pic/management-01-importInfo.jpeg) + +### 慢 SQL +点击“系统管理”后，点击“慢 SQL”标签页，可以查看慢 SQL 执行语句日志统计与明细。 + +- 慢 SQL 明细：默认展示的是开始执行时间是一天内和执行耗时大于等于10秒的数据 +![management-01-slowsql.jpeg](./pic/management-01-slowsql.jpeg) +- 慢 SQL 统计：默认展示所有的数据，可根据开始执行时间进行过滤 +![management-02-slowsql.jpeg](./pic/management-02-slowsql.jpeg) diff --git a/docs/zh/07-operation/12-multi.md b/docs/zh/07-operation/12-multi.md index e156af55f8..8f11ee4326 100644 --- a/docs/zh/07-operation/12-multi.md +++ b/docs/zh/07-operation/12-multi.md @@ -69,7 +69,7 @@ dataDir /mnt/data6 2 0 在配置文件 /etc/taos/taos.cfg 中，添加用于 S3 访问的参数： |参数名称 | 参数含义 | -|:-------------:|:-----------------------------------------------:| +|:-------------|:-----------------------------------------------| |s3EndPoint | 用户所在地域的 COS 服务域名，支持 http 和 https，bucket 的区域需要与 endpoint 的保持一致，否则无法访问。例如：http://cos.ap-beijing.myqcloud.com | |s3AccessKey |冒号分隔的用户 SecretId:SecretKey。例如：AKIDsQmwsfKxTo2A6nGVXZN0UlofKn6JRRSJ:lIdoy99ygEacU7iHfogaN2Xq0yumSm1E | |s3BucketName | 存储桶名称，减号后面是用户注册 COS 服务的 AppId。其中 AppId 是 COS 特有，AWS 和阿里云都没有，配置时需要作为 bucket name 的一部分，使用减号分隔。参数值均为字符串类型，但不需要引号。例如：test0711-1309024725 | @@ -111,3 +111,27 @@ s3migrate database ; | 1 | s3_keeplocal | 3650 | 1 | 365000 | 数据在本地保留的天数，即 data 文件在本地磁盘保留多长时间后可以上传到 S3。默认单位：天，支持 m（分钟）、h（小时）和 d（天）三个单位 | | 2 | s3_chunksize | 262144 | 131072 | 1048576 | 上传对象的大小阈值，与 TSDB_PAGESIZE 参数一样，不可修改，单位为 TSDB 页 | | 3 | s3_compact | 0 | 0 | 1 | TSDB 文件组首次上传 S3 时，是否自动进行 compact 操作。 | + +## Azure Blob 存储 +本节介绍在 TDengine Enterprise 如何使用微软 Azure Blob 对象存储。本功能是上一小节‘对象存储’功能的扩展，需额外依赖 Flexify 服务提供的 S3 网关。通过适当的参数配置，可以把大部分较冷的时序数据存储到 Azure Blob 服务中。 + +### Flexify 服务 +Flexify 是 Azure Marketplace 中的一款应用程序，允许兼容 S3 的应用程序通过标准 S3 API 在 Azure Blob Storage 中存储数据。可使用多个 Flexify 服务对同一个 Blob 存储建立多个 S3 网关。 + +部署方式请参考 [Flexify](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/flexify.azure-s3-api?tab=Overview) 应用页面说明。 + +### 配置方式 + +在配置文件 /etc/taos/taos.cfg 中，添加用于 S3 访问的参数： + +``` +s3EndPoint http //20.191.157.23,http://20.191.157.24,http://20.191.157.25 +s3AccessKey FLIOMMNL0:uhRNdeZMLD4wo,ABCIOMMN:uhRNdeZMD4wog,DEFOMMNL049ba:uhRNdeZMLD4wogXd +s3BucketName td-test +``` + +- 允许对 s3EndPoint、s3AccessKey 配置多项，但要求二者项数一致。多个配置项间使用 ',' 分隔。s3BucketName 仅允许配置一项 +- 认为每一组 `{s3EndPoint、s3AccessKey}` 配置对应一个 S3 服务，每次发起 S3 请求时将随机选择一个服务 +- 认为全部 S3 服务均指向同一数据源，对各个 S3 服务操作完全等价 +- 在某一 S3 服务上操作失败后会切换至其他服务，全部服务都失败后将返回最后产生的错误码 +- 最大支持的 S3 服务配置数为 10 diff --git a/docs/zh/07-operation/pic/management-01-slowsql.jpeg b/docs/zh/07-operation/pic/management-01-slowsql.jpeg new file mode 100644 index 0000000000..01e3d5b886 Binary files /dev/null and b/docs/zh/07-operation/pic/management-01-slowsql.jpeg differ diff --git a/docs/zh/07-operation/pic/management-02-slowsql.jpeg b/docs/zh/07-operation/pic/management-02-slowsql.jpeg new file mode 100644 index 0000000000..3aa3dbade6 Binary files /dev/null and b/docs/zh/07-operation/pic/management-02-slowsql.jpeg differ diff --git a/docs/zh/08-develop/07-tmq.md b/docs/zh/08-develop/07-tmq.md index 7eb8628e56..96f34c6d5d 100644 --- a/docs/zh/08-develop/07-tmq.md +++ b/docs/zh/08-develop/07-tmq.md @@ -34,13 +34,15 @@ TDengine 消费者的概念跟 Kafka 类似，消费者通过订阅主题来接 | `td.connect.user` | string | 用户名 | | | `td.connect.pass` | string | 密码 | | | `td.connect.port` | integer | 服务端的端口号 | | -| `group.id` | string | 消费组 ID，同一消费组共享消费进度 |
**必填项**。最大长度：192。
每个topic最多可建立100个 consumer group | -| `client.id` | string | 客户端 ID | 最大长度：192。 | +| `group.id` | string | 消费组 ID，同一消费组共享消费进度 |
**必填项**。最大长度：192。
每个topic最多可建立 100 个 consumer group | +| `client.id` | string | 客户端 ID | 最大长度：192 | | `auto.offset.reset` | enum | 消费组订阅的初始位置 |
`earliest`: default(version < 3.2.0.0);从头开始订阅;
`latest`: default(version >= 3.2.0.0);仅从最新数据开始订阅;
`none`: 没有提交的 offset 无法订阅 | | `enable.auto.commit` | boolean | 是否启用消费位点自动提交，true: 自动提交，客户端应用无需commit；false：客户端应用需要自行commit | 默认值为 true | | `auto.commit.interval.ms` | integer | 消费记录自动提交消费位点时间间隔，单位为毫秒 | 默认值为 5000 | | `msg.with.table.name` | boolean | 是否允许从消息中解析表名, 不适用于列订阅（列订阅时可将 tbname 作为列写入 subquery 语句）（从3.2.0.0版本该参数废弃，恒为true） | 默认关闭 | | `enable.replay` | boolean | 是否开启数据回放功能 | 默认关闭 | +| `session.timeout.ms` | integer | consumer 心跳丢失后超时时间，超时后会触发 rebalance 逻辑，成功后该 consumer 会被删除（从3.3.3.0版本开始支持） | 默认值为 12000，取值范围 [6000， 1800000] | +| `max.poll.interval.ms` | integer | consumer poll 拉取数据间隔的最长时间，超过该时间，会认为该 consumer 离线，触发rebalance 逻辑，成功后该 consumer 会被删除（从3.3.3.0版本开始支持） | 默认值为 300000，[1000，INT32_MAX] | 下面是各语言连接器创建参数： diff --git a/docs/zh/14-reference/01-components/03-taosadapter.md b/docs/zh/14-reference/01-components/03-taosadapter.md index 443b769dde..a8e8fc3418 100644 --- a/docs/zh/14-reference/01-components/03-taosadapter.md +++ b/docs/zh/14-reference/01-components/03-taosadapter.md @@ -187,8 +187,8 @@ AllowWebSockets - 兼容 InfluxDB v1 写接口 [https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/](https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/) - 兼容 OpenTSDB JSON 和 telnet 格式写入 - - \ - - \ + - [http://opentsdb.net/docs/build/html/api_http/put.html](http://opentsdb.net/docs/build/html/api_http/put.html) + - [http://opentsdb.net/docs/build/html/api_telnet/put.html](http://opentsdb.net/docs/build/html/api_telnet/put.html) - 与 collectd 无缝连接。 collectd 是一个系统统计收集守护程序，请访问 [https://collectd.org/](https://collectd.org/) 了解更多信息。 - Seamless connection with StatsD。 diff --git a/docs/zh/14-reference/01-components/06-taoskeeper.md b/docs/zh/14-reference/01-components/06-taoskeeper.md index 6f0684bcbc..c3d22d25f1 100644 --- a/docs/zh/14-reference/01-components/06-taoskeeper.md +++ b/docs/zh/14-reference/01-components/06-taoskeeper.md @@ -194,7 +194,7 @@ Active: inactive (dead) 可以访问 taosKeeper 的 `check_health` 接口来判断服务是否存活，如果服务正常则会返回 HTTP 200 状态码： ``` -$ curl -i http://127.0.0.1:6043/check_health +curl -i http://127.0.0.1:6043/check_health ``` 返回结果： diff --git a/docs/zh/14-reference/03-taos-sql/12-distinguished.md b/docs/zh/14-reference/03-taos-sql/12-distinguished.md index 0eaeb0dfa7..e149c2c82e 100644 --- a/docs/zh/14-reference/03-taos-sql/12-distinguished.md +++ b/docs/zh/14-reference/03-taos-sql/12-distinguished.md @@ -54,7 +54,7 @@ window_clause: { ``` 其中，interval_val 和 sliding_val 都表示时间段，interval_offset 表示窗口偏移量，interval_offset 必须小于 interval_val，语法上支持三种方式，举例说明如下: - - INTERVAL(1s, 500a) SLIDING(1s), 自带时间单位的形式，其中的时间单位是单字符表示, 分别为: a (毫秒), b (纳秒), d (天), h (小时), m (分钟), n (月), s (秒), u (微妙), w (周), y (年). + - INTERVAL(1s, 500a) SLIDING(1s), 自带时间单位的形式，其中的时间单位是单字符表示, 分别为: a (毫秒), b (纳秒), d (天), h (小时), m (分钟), n (月), s (秒), u (微秒), w (周), y (年). - INTERVAL(1000, 500) SLIDING(1000), 不带时间单位的形式，将使用查询库的时间精度作为默认时间单位，当存在多个库时默认采用精度更高的库. - INTERVAL('1s', '500a') SLIDING('1s'), 自带时间单位的字符串形式，字符串内部不能有任何空格等其它字符. diff --git a/docs/zh/14-reference/03-taos-sql/22-meta.md b/docs/zh/14-reference/03-taos-sql/22-meta.md index 397b3d3362..b61e06f759 100644 --- a/docs/zh/14-reference/03-taos-sql/22-meta.md +++ b/docs/zh/14-reference/03-taos-sql/22-meta.md @@ -296,8 +296,10 @@ TDengine 内置了一个名为 `INFORMATION_SCHEMA` 的数据库，提供对数 | 2 | consumer_group | VARCHAR(193) | 订阅者的消费者组 | | 3 | vgroup_id | INT | 消费者被分配的 vgroup id | | 4 | consumer_id | BIGINT | 消费者的唯一 id | -| 5 | offset | VARCHAR(64) | 消费者的消费进度 | -| 6 | rows | BIGINT | 消费者的消费的数据条数 | +| 5 | user | VARCHAR(24) | 消费者的登录的用户名 | +| 6 | fqdn | VARCHAR(128) | 消费者的所在机器的 fqdn | +| 7 | offset | VARCHAR(64) | 消费者的消费进度 | +| 8 | rows | BIGINT | 消费者的消费的数据条数 | ## INS_STREAMS diff --git a/docs/zh/14-reference/03-taos-sql/28-index.md b/docs/zh/14-reference/03-taos-sql/28-index.md index fef64e978b..14c45b6585 100644 --- a/docs/zh/14-reference/03-taos-sql/28-index.md +++ b/docs/zh/14-reference/03-taos-sql/28-index.md @@ -18,7 +18,7 @@ time_duration: number unit ``` -创建 TSMA 时需要指定 TSMA 名字, 表名字, 函数列表以及窗口大小. 当基于一个已经存在的 TSMA 创建新的 TSMA 时, 需要使用 `RECURSIVE` 关键字但不能指定 `FUNCTION()`， 新创建的 TSMA 已有 TSMA 拥有相同的函数列表， 且此种情况下所指定的 INTERVAL 必须为所基于的 TSMA 窗口长度的整数倍。 +创建 TSMA 时需要指定 TSMA 名字, 表名字, 函数列表以及窗口大小. 当基于一个已经存在的 TSMA 创建新的 TSMA 时, 需要使用 `RECURSIVE` 关键字但不能指定 `FUNCTION()`， 新创建的 TSMA 已有 TSMA 拥有相同的函数列表， 且此种情况下所指定的 INTERVAL 必须至少为所基于的 TSMA 窗口长度的整数倍, 并且天不能基于2h或3h建立, 只能基于1h建立, 月也只能基于1d而非2d,3d建立。 其中 TSMA 命名规则与表名字类似, 长度最大限制为表名长度限制减去输出表后缀长度, 表名长度限制为193, 输出表后缀为`_tsma_res_stb_`, TSMA 名字最大长度为178. @@ -28,7 +28,7 @@ TSMA只能基于超级表和普通表创建, 不能基于子表创建. 由于TSMA输出为一张超级表, 因此输出表的行长度受最大行长度限制, 不同函数的`中间结果`大小各异, 一般都大于原始数据大小, 若输出表的行长度大于最大行长度限制, 将会报`Row length exceeds max length`错误. 此时需要减少函数个数或者将常用的函数进行分组拆分到多个TSMA中. -窗口大小的限制为[1m ~ 1h]. INTERVAL 的单位与查询中INTERVAL子句相同, 如 a (毫秒), b (纳秒), h (小时), m (分钟), s (秒), u (微妙). +窗口大小的限制为[1m ~ 1y/12n]. INTERVAL 的单位与查询中INTERVAL子句相同, 如 a (毫秒), b (纳秒), h (小时), m (分钟), s (秒), u (微妙), d (天), w(周), n(月), y(年). TSMA为库内对象, 但名字全局唯一. 集群内一共可创建TSMA个数受参数`maxTsmaNum`限制, 参数默认值为3, 范围: [0-3]. 注意, 由于TSMA后台计算使用流计算, 因此每创建一条TSMA, 将会创建一条流, 因此能够创建的TSMA条数也受当前已经存在的流条数和最大可创建流条数限制. diff --git a/docs/zh/14-reference/05-connector/45-php.mdx b/docs/zh/14-reference/05-connector/45-php.mdx index 0b453218f6..70b3d760fc 100644 --- a/docs/zh/14-reference/05-connector/45-php.mdx +++ b/docs/zh/14-reference/05-connector/45-php.mdx @@ -8,7 +8,7 @@ toc_max_heading_level: 4 PHP 连接器依赖 TDengine 客户端驱动。 -项目地址：\ +项目地址：[https://github.com/Yurunsoft/php-tdengine](https://github.com/Yurunsoft/php-tdengine) TDengine 服务端或客户端安装后，`taos.h` 位于： diff --git a/docs/zh/20-third-party/01-collection/09-emq-broker.md b/docs/zh/20-third-party/01-collection/09-emq-broker.md index 33710384f0..3d920345c1 100644 --- a/docs/zh/20-third-party/01-collection/09-emq-broker.md +++ b/docs/zh/20-third-party/01-collection/09-emq-broker.md @@ -16,7 +16,7 @@ MQTT 是流行的物联网数据传输协议，[EMQX](https://github.com/emqx/em ## 安装并启动 EMQX -用户可以根据当前的操作系统，到 EMQX 官网下载安装包，并执行安装。下载地址如下：\。安装后使用 `sudo emqx start` 或 `sudo systemctl start emqx` 启动 EMQX 服务。 +用户可以根据当前的操作系统，到 [EMQX 官网下载安装包](https://www.emqx.io/zh/downloads)，并执行安装。安装后使用 `sudo emqx start` 或 `sudo systemctl start emqx` 启动 EMQX 服务。 注意：本文基于 EMQX v4.4.5 版本，其他版本由于相关配置界面、配置方法以及功能可能随着版本升级有所区别。 diff --git a/docs/zh/26-tdinternal/07-topic.md b/docs/zh/26-tdinternal/07-topic.md index 3fe12c943d..4299b3d503 100644 --- a/docs/zh/26-tdinternal/07-topic.md +++ b/docs/zh/26-tdinternal/07-topic.md @@ -61,37 +61,37 @@ TDengine 会为 WAL 文件自动创建索引以支持快速随机访问。通过 ![数据订阅架构图](./topicarch.png) -在客户端成功建立与服务器的连接之后，用户须首先指定消费组和主题，以创建相应的消费者实例。随后，客户端便会向服务器提交订阅请求。此刻，消费者的状态被标记为 modify，表示正处于配置阶段。消费者随后会定期向服务器发送请求，以检索并获取待消费的 vnode 列表，直至服务器为其分配相应的 vnode。一旦分配完成，消费者的状态便更新为 ready，标志着订阅流程的成功完成。此刻，客户端便可正式启动向 vnode 发送消费数据请求的过程。 +在客户端成功建立与服务器的连接之后，用户须首先指定消费组和主题，以创建相应的消费者实例。随后，客户端便会向服务器提交订阅请求。此刻，消费者的状态被标记为 rebalancing，表示正处于 rebalance 阶段。消费者随后会定期向服务器发送请求，以检索并获取待消费的 vnode 列表，直至服务器为其分配相应的 vnode。一旦分配完成，消费者的状态便更新为 ready，标志着订阅流程的成功完成。此刻，客户端便可正式启动向 vnode 发送消费数据请求的过程。 在消费数据的过程中，消费者会不断地向每个分配到的 vnode 发送请求，以尝试获取新的数据。一旦收到数据，消费者在完成消费后会继续向该 vnode 发送请求，以便持续消费。若在预设时间内未收到任何数据，消费者便会在 vnode 上注册一个消费 handle。这样一来，一旦 vnode 上有新数据产生，便会立即推送给消费者，从而确保数据消费的即时性，并有效减少消费者频繁主动拉取数据所导致的性能损耗。因此，可以看出，消费者从 vnode 获取数据的方式是一种拉取（pull）与推送（push）相结合的高效模式。 -消费者在收到数据时，会同时收到数据的版本号，并将其记录为当前在每个 vnode上的消费进度。这一进度仅在消费者内部以内存形式存储，确保仅对该消费者有效。这种设计保证了消费者在每次启动时能够从上次的消费中断处继续，避免了数据的重复处理。然而，若消费者需要退出并希望在之后恢复上次的消费进度，就必须在退出前将消费进度提交至服务器，执行所谓的 commit 操作。这一操作会将消费进度在服务器进行持久化存储，支持自动提交或手动提交两种方式。 +消费者在收到数据时，会同时收到数据的版本号，并将其记录为当前在每个 vnode 上的消费进度。这一进度仅在消费者内部以内存形式存储，确保仅对该消费者有效。这种设计保证了消费者在每次启动时能够从上次的消费中断处继续，避免了数据的重复处理。然而，若消费者需要退出并希望在之后恢复上次的消费进度，就必须在退出前将消费进度提交至服务器，执行所谓的 commit 操作。这一操作会将消费进度在服务器进行持久化存储，支持自动提交或手动提交两种方式。 -此外，为了维持消费者的活跃状态，客户端还实施了心跳保活机制。通过定期向服务器发送心跳信号，消费者能够向服务器证明自己仍然在线。若服务器在一定时间内未收到消费者的心跳，便会将其标记为 lost 状态，即认为消费者已离线。服务器依赖心跳机制来监控所有消费者的状态，进而有效地管理整个消费者群体。 +此外，为了维持消费者的活跃状态，客户端还实施了心跳保活机制。通过定期向服务器发送心跳信号，消费者能够向服务器证明自己仍然在线。若服务器在一定时间内未收到消费者的心跳，便会认为消费者已离线。对于一定时间（可以通过参数控制）不拉取数据的 consumer，服务器也会标记为离线，并从消费组中删除该消费者。服务器依赖心跳机制来监控所有消费者的状态，进而有效地管理整个消费者群体。 -mnode 主要负责处理订阅过程中的控制消息，包括创建和删除主题、订阅消息、查询 endpoint 消息以及心跳消息等。vnode 则专注于处理消费消息和 commit 消息。当mnode 收到消费者的订阅消息时，如果该消费者尚未订阅过，其状态将被设置为 modify。如果消费者已经订阅过，但订阅的主题发生变更，消费者的状态同样会被设置为 modify。等待再平衡的计时器到来时，mnode 会对 modify 状态的消费者执行再平衡操作，将心跳超过固定时间的消费者设置为 lost 状态，并对关闭或 lost 状态超过一定时间的消费者进行删除操作。 +mnode 主要负责处理订阅过程中的控制消息，包括创建和删除主题、订阅消息、查询 endpoint 消息以及心跳消息等。vnode 则专注于处理消费消息和 commit 消息。当 mnode 收到消费者的订阅消息时，如果该消费者尚未订阅过，其状态将被设置为 rebalancing。如果消费者已经订阅过，但订阅的主题发生变更，消费者的状态同样会被设置为 rebalancing。然后 mnode 会对 rebalancing 状态的消费者执行 rebalance 操作。心跳超过固定时间的消费者或主动关闭的消费者将被删除。 消费者会定期向 mnode 发送查询 endpoint 消息，以获取再平衡后的最新 vnode 分配结果。同时，消费者还会定期发送心跳消息，通知 mnode 自身处于活跃状态。此外，消费者的一些信息也会通过心跳消息上报至 mnode，用户可以查询 mnode 上的这些信息以监测各个消费者的状态。这种设计有助于实现对消费者的有效管理和监控。 -## 再平衡过程 +## rebalance 过程 -每个主题的数据可能分散在多个 vnode 上。服务器通过执行再平衡过程，将这些vnode 合理地分配给各个消费者，确保数据的均匀分布和高效消费。 +每个主题的数据可能分散在多个 vnode 上。服务器通过执行 rebalance 过程，将这些 vnode 合理地分配给各个消费者，确保数据的均匀分布和高效消费。 -如下图所示，c1 表示消费者 1，c2 表示消费者 2，g1 表示消费组 1。起初 g1 中只有 c1 消费数据，c1 发送订阅信息到 mnode，mnode 把数据所在的所有 4 个vnode 分配给 c1。当 c2 增加到 g1 后，c2 将订阅信息发送给 mnode，mnode 将在下一次再平衡计时器到来时检测到这个 g1 需要重新分配，就会启动再平衡过程，随后 c2 分得其中两个 vnode 消费。分配信息还会通过 mnode 发送给 vnode，同时 c1 和 c2 也会获取自己消费的 vnode 信息并开始消费。 +如下图所示，c1 表示消费者 1，c2 表示消费者 2，g1 表示消费组 1。起初 g1 中只有 c1 消费数据，c1 发送订阅信息到 mnode，mnode 把数据所在的所有 4 个vnode 分配给 c1。当 c2 增加到 g1 后，c2 将订阅信息发送给 mnode，mnode 检测到这个 g1 需要重新分配，就会启动 rebalance 过程，随后 c2 分得其中两个 vnode 消费。分配信息还会通过 mnode 发送给 vnode，同时 c1 和 c2 也会获取自己消费的 vnode 信息并开始消费。 -![再平衡过程](./rebalance.png) +![rebalance 过程](./rebalance.png) 再平衡计时器每 2s 检测一次是否需要再平衡。在再平衡过程中，如果消费者获取的状态是 not ready，则不能进行消费。只有再平衡正常结束后，消费者获取分配 vnode 的offset 后才可正常消费，否则消费者会重试指定次数后报错。 ## 消费者状态处理 -消费者的状态转换过程如下图所示。初始状态下，刚发起订阅的消费者处于modify 状态，此时客户端获取的消费者状态为 not ready，表明消费者尚未准备好进行数据消费。一旦 mnode 在再平衡计时器中检测到处于 modify 状态的消费者，便会启动再平衡过程。再平衡成功后，消费者的状态将转变为 ready，表示消费者已准备就绪。随后，当消费者通过定时的查询 endpoint 消息获取自身的 ready 状态以及分配的 vnode 列表后，便能正式开始消费数据。 +消费者的状态转换过程如下图所示。初始状态下，刚发起订阅的消费者处于 rebalancing 状态，表明消费者尚未准备好进行数据消费。一旦 mnode 检测到处于 rebalancing 状态的消费者，便会启动 rebalance 过程，成功后，消费者的状态将转变为 ready，表示消费者已准备就绪。随后，当消费者通过定时的查询 endpoint 消息获取自身的 ready 状态以及分配的 vnode 列表后，便能正式开始消费数据。 ![消费者状态转换](./statetransition.png) -若消费者的心跳丢失时间超过 12s，经过再平衡过程，其状态将被更新为 lost，表明消费者被认为已离线。如果心跳丢失时间超过 1 天，消费者的状态将进一步转变为 clear，此时消费者将被系统删除。然而，如果在上述过程中消费者重新发送心跳信号，其状态将恢复为 modify，并重新进入下一轮的再平衡过程。 +若消费者的心跳丢失时间超过 12s，经过 rebalance 过程，其状态将被更新为 clear，然后消费者将被系统删除。 -当消费者主动退出时，会发送 unsubscribe 消息。该消息将清除消费者订阅的所有主题，并将消费者的状态设置为 modify。随后，在再平衡过程中，一旦消费者的状态变为ready 且无订阅的主题，mnode 将在再平衡的计时器中检测到此状态变化，并据此删除该消费者。这一系列措施确保了消费者退出的有序性和系统的稳定性。 +当消费者主动退出时，会发送 unsubscribe 消息。该消息将清除消费者订阅的所有主题，并将消费者的状态设置为 rebalancing。随后，检测到处于 rebalancing 状态的消费者，便会启动 rebalance 过程，成功后，其状态将被更新为 clear，然后消费者将被系统删除。这一系列措施确保了消费者退出的有序性和系统的稳定性。 ## 消费数据 diff --git a/docs/zh/26-tdinternal/consumer.png b/docs/zh/26-tdinternal/consumer.png index cc996a431a..0c9a061107 100644 Binary files a/docs/zh/26-tdinternal/consumer.png and b/docs/zh/26-tdinternal/consumer.png differ diff --git a/docs/zh/26-tdinternal/offset.png b/docs/zh/26-tdinternal/offset.png index d8146b8d50..0ddb005d66 100644 Binary files a/docs/zh/26-tdinternal/offset.png and b/docs/zh/26-tdinternal/offset.png differ diff --git a/docs/zh/26-tdinternal/rebalance.png b/docs/zh/26-tdinternal/rebalance.png index 216ba5ec3d..fbc2147c91 100644 Binary files a/docs/zh/26-tdinternal/rebalance.png and b/docs/zh/26-tdinternal/rebalance.png differ diff --git a/docs/zh/26-tdinternal/statetransition.png b/docs/zh/26-tdinternal/statetransition.png index 1ec8ae22f7..dd2b472890 100644 Binary files a/docs/zh/26-tdinternal/statetransition.png and b/docs/zh/26-tdinternal/statetransition.png differ diff --git a/docs/zh/26-tdinternal/topic.png b/docs/zh/26-tdinternal/topic.png index 82f3df6b73..d9306cdb74 100644 Binary files a/docs/zh/26-tdinternal/topic.png and b/docs/zh/26-tdinternal/topic.png differ diff --git a/docs/zh/26-tdinternal/topicarch.png b/docs/zh/26-tdinternal/topicarch.png index 2442158ebb..1bb4bb3014 100644 Binary files a/docs/zh/26-tdinternal/topicarch.png and b/docs/zh/26-tdinternal/topicarch.png differ diff --git a/include/libs/executor/storageapi.h b/include/libs/executor/storageapi.h index 07cf69171b..7e0ec3e11d 100644 --- a/include/libs/executor/storageapi.h +++ b/include/libs/executor/storageapi.h @@ -198,7 +198,7 @@ typedef struct SStoreCacheReader { SArray *pFuncTypeList, SColumnInfo* pPkCol, int32_t numOfPks); void (*closeReader)(void *pReader); int32_t (*retrieveRows)(void *pReader, SSDataBlock *pResBlock, const int32_t *slotIds, const int32_t *dstSlotIds, - SArray *pTableUidList); + SArray *pTableUidList, bool* pGotAllRows); int32_t (*reuseReader)(void *pReader, void *pTableIdList, int32_t numOfTables); } SStoreCacheReader; diff --git a/source/client/src/clientEnv.c b/source/client/src/clientEnv.c index fec1060042..53be1fccf4 100644 --- a/source/client/src/clientEnv.c +++ b/source/client/src/clientEnv.c @@ -552,6 +552,9 @@ int32_t createRequest(uint64_t connId, int32_t type, int64_t reqid, SRequestObj (*pRequest)->allocatorRefId = -1; (*pRequest)->pDb = getDbOfConnection(pTscObj); + if (NULL == (*pRequest)->pDb) { + TSC_ERR_JRET(terrno); + } (*pRequest)->pTscObj = pTscObj; (*pRequest)->inCallback = false; (*pRequest)->msgBuf = taosMemoryCalloc(1, ERROR_MSG_BUF_DEFAULT_SIZE); diff --git a/source/client/src/clientHb.c b/source/client/src/clientHb.c index 82ebb5a9fd..6ee6d753e4 100644 --- a/source/client/src/clientHb.c +++ b/source/client/src/clientHb.c @@ -1359,6 +1359,14 @@ static void *hbThreadFunc(void *param) { pInfo->msgInfo.len = tlen; pInfo->msgType = TDMT_MND_HEARTBEAT; pInfo->param = taosMemoryMalloc(sizeof(int32_t)); + if (pInfo->param == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; + tFreeClientHbBatchReq(pReq); + // hbClearReqInfo(pAppHbMgr); + taosMemoryFree(buf); + taosMemoryFree(pInfo); + break; + } *(int32_t *)pInfo->param = i; pInfo->paramFreeFp = taosMemoryFree; pInfo->requestId = generateRequestId(); diff --git a/source/client/src/clientImpl.c b/source/client/src/clientImpl.c index 2a5266b7ed..1c0be6b3ac 100644 --- a/source/client/src/clientImpl.c +++ b/source/client/src/clientImpl.c @@ -1291,7 +1291,7 @@ SRequestObj* launchQueryImpl(SRequestObj* pRequest, SQuery* pQuery, bool keepQue } if (TSDB_CODE_SUCCESS != code) { - pRequest->code = terrno; + pRequest->code = code; } if (res) { @@ -1586,6 +1586,8 @@ int32_t taosConnectImpl(const char* user, const char* auth, const char* db, __ta pRequest->sqlstr = taosStrdup("taos_connect"); if (pRequest->sqlstr) { pRequest->sqlLen = strlen(pRequest->sqlstr); + } else { + return terrno; } SMsgSendInfo* body = NULL; @@ -1649,6 +1651,9 @@ static int32_t buildConnectMsg(SRequestObj* pRequest, SMsgSendInfo** pMsgSendInf char* db = getDbOfConnection(pObj); if (db != NULL) { tstrncpy(connectReq.db, db, sizeof(connectReq.db)); + } else if (terrno) { + taosMemoryFree(*pMsgSendInfo); + return terrno; } taosMemoryFreeClear(db); @@ -2396,11 +2401,16 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32 } char* getDbOfConnection(STscObj* pObj) { + terrno = TSDB_CODE_SUCCESS; char* p = NULL; (void)taosThreadMutexLock(&pObj->mutex); size_t len = strlen(pObj->db); if (len > 0) { p = strndup(pObj->db, tListLen(pObj->db)); + if (p == NULL) { + tscError("failed to strndup db name"); + terrno = TSDB_CODE_OUT_OF_MEMORY; + } } (void)taosThreadMutexUnlock(&pObj->mutex); diff --git a/source/client/src/clientStmt.c b/source/client/src/clientStmt.c index d556ada1d2..ee6e2d71a0 100644 --- a/source/client/src/clientStmt.c +++ b/source/client/src/clientStmt.c @@ -924,6 +924,9 @@ int stmtPrepare(TAOS_STMT* stmt, const char* sql, unsigned long length) { } pStmt->sql.sqlStr = strndup(sql, length); + if (!pStmt->sql.sqlStr) { + return TSDB_CODE_OUT_OF_MEMORY; + } pStmt->sql.sqlLen = length; pStmt->sql.stbInterlaceMode = pStmt->stbInterlaceMode; diff --git a/source/client/src/clientStmt2.c b/source/client/src/clientStmt2.c index 66430a2647..a9738e4e9c 100644 --- a/source/client/src/clientStmt2.c +++ b/source/client/src/clientStmt2.c @@ -869,6 +869,9 @@ int stmtPrepare2(TAOS_STMT2* stmt, const char* sql, unsigned long length) { } pStmt->sql.sqlStr = strndup(sql, length); + if (!pStmt->sql.sqlStr) { + return TSDB_CODE_OUT_OF_MEMORY; + } pStmt->sql.sqlLen = length; pStmt->sql.stbInterlaceMode = pStmt->stbInterlaceMode; @@ -998,7 +1001,7 @@ int stmtSetTbTags2(TAOS_STMT2* stmt, TAOS_STMT2_BIND* tags) { STMT_ERR_RET(TSDB_CODE_APP_ERROR); } - if (pStmt->bInfo.inExecCache && (!pStmt->sql.autoCreateTbl || (*pDataBlock)->pData->pCreateTbReq)) { + if (pStmt->bInfo.inExecCache && !pStmt->sql.autoCreateTbl) { return TSDB_CODE_SUCCESS; } diff --git a/source/common/src/tdatablock.c b/source/common/src/tdatablock.c index 7e957357a9..d8a66f82bf 100644 --- a/source/common/src/tdatablock.c +++ b/source/common/src/tdatablock.c @@ -2620,11 +2620,15 @@ int32_t dumpBlockData(SSDataBlock* pDataBlock, const char* flag, char** pDataBuf } len += snprintf(dumpBuf + len, size - len, "%s |end\n", flag); - *pDataBuf = dumpBuf; - dumpBuf = NULL; _exit: - if (dumpBuf) { - taosMemoryFree(dumpBuf); + if (code == TSDB_CODE_SUCCESS) { + *pDataBuf = dumpBuf; + dumpBuf = NULL; + } else { + uError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + if (dumpBuf) { + taosMemoryFree(dumpBuf); + } } return code; } diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c index 488a4d3b99..dd921c615b 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c @@ -360,6 +360,12 @@ static int32_t vmOpenVnodes(SVnodeMgmt *pMgmt) { int32_t vnodesPerThread = numOfVnodes / threadNum + 1; SVnodeThread *threads = taosMemoryCalloc(threadNum, sizeof(SVnodeThread)); + if (threads == NULL) { + dError("failed to allocate memory for threads since %s", terrstr()); + taosMemoryFree(pCfgs); + return terrno; + } + for (int32_t t = 0; t < threadNum; ++t) { threads[t].threadIndex = t; threads[t].pMgmt = pMgmt; diff --git a/source/dnode/vnode/inc/vnode.h b/source/dnode/vnode/inc/vnode.h index 827f7e2044..443e4433cc 100644 --- a/source/dnode/vnode/inc/vnode.h +++ b/source/dnode/vnode/inc/vnode.h @@ -186,7 +186,7 @@ int32_t tsdbCacherowsReaderOpen(void *pVnode, int32_t type, void *pTableIdList, SArray *pCidList, int32_t *pSlotIds, uint64_t suid, void **pReader, const char *idstr, SArray *pFuncTypeList, SColumnInfo *pkCol, int32_t numOfPks); int32_t tsdbRetrieveCacheRows(void *pReader, SSDataBlock *pResBlock, const int32_t *slotIds, const int32_t *dstSlotIds, - SArray *pTableUids); + SArray *pTableUids, bool *pGotAllRows); void tsdbCacherowsReaderClose(void *pReader); void tsdbCacheSetCapacity(SVnode *pVnode, size_t capacity); diff --git a/source/dnode/vnode/src/inc/vnodeInt.h b/source/dnode/vnode/src/inc/vnodeInt.h index 7b1f3716c4..e50ced2ebb 100644 --- a/source/dnode/vnode/src/inc/vnodeInt.h +++ b/source/dnode/vnode/src/inc/vnodeInt.h @@ -319,7 +319,7 @@ int32_t tsdbSnapRead(STsdbSnapReader* pReader, uint8_t** ppData); // STsdbSnapWriter ======================================== int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, void* pRanges, STsdbSnapWriter** ppWriter); int32_t tsdbSnapWrite(STsdbSnapWriter* pWriter, SSnapDataHdr* pHdr); -int32_t tsdbSnapWriterPrepareClose(STsdbSnapWriter* pWriter); +int32_t tsdbSnapWriterPrepareClose(STsdbSnapWriter* pWriter, bool rollback); int32_t tsdbSnapWriterClose(STsdbSnapWriter** ppWriter, int8_t rollback); // STsdbSnapRAWReader ======================================== int32_t tsdbSnapRAWReaderOpen(STsdb* pTsdb, int64_t ever, int8_t type, STsdbSnapRAWReader** ppReader); @@ -373,7 +373,7 @@ int32_t rsmaSnapRead(SRSmaSnapReader* pReader, uint8_t** ppData); // SRSmaSnapWriter ======================================== int32_t rsmaSnapWriterOpen(SSma* pSma, int64_t sver, int64_t ever, void** ppRanges, SRSmaSnapWriter** ppWriter); int32_t rsmaSnapWrite(SRSmaSnapWriter* pWriter, uint8_t* pData, uint32_t nData); -int32_t rsmaSnapWriterPrepareClose(SRSmaSnapWriter* pWriter); +int32_t rsmaSnapWriterPrepareClose(SRSmaSnapWriter* pWriter, bool rollback); int32_t rsmaSnapWriterClose(SRSmaSnapWriter** ppWriter, int8_t rollback); typedef struct { diff --git a/source/dnode/vnode/src/meta/metaOpen.c b/source/dnode/vnode/src/meta/metaOpen.c index 1917c09526..591c40332a 100644 --- a/source/dnode/vnode/src/meta/metaOpen.c +++ b/source/dnode/vnode/src/meta/metaOpen.c @@ -27,15 +27,15 @@ static int taskIdxKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int k static int btimeIdxCmpr(const void *pKey1, int kLen1, const void *pKey2, int kLen2); static int ncolIdxCmpr(const void *pKey1, int kLen1, const void *pKey2, int kLen2); -static int32_t metaInitLock(SMeta *pMeta) { +static void metaInitLock(SMeta *pMeta) { TdThreadRwlockAttr attr; (void)taosThreadRwlockAttrInit(&attr); (void)taosThreadRwlockAttrSetKindNP(&attr, PTHREAD_RWLOCK_PREFER_WRITER_NONRECURSIVE_NP); (void)taosThreadRwlockInit(&pMeta->lock, &attr); (void)taosThreadRwlockAttrDestroy(&attr); - return 0; + return; } -static int32_t metaDestroyLock(SMeta *pMeta) { return taosThreadRwlockDestroy(&pMeta->lock); } +static void metaDestroyLock(SMeta *pMeta) { (void)taosThreadRwlockDestroy(&pMeta->lock); } static void metaCleanup(SMeta **ppMeta); @@ -56,7 +56,7 @@ int32_t metaOpen(SVnode *pVnode, SMeta **ppMeta, int8_t rollback) { TSDB_CHECK_CODE(code = terrno, lino, _exit); } - (void)metaInitLock(pMeta); + metaInitLock(pMeta); pMeta->path = (char *)&pMeta[1]; strcpy(pMeta->path, path); @@ -188,17 +188,23 @@ int metaAlterCache(SMeta *pMeta, int32_t nPage) { void metaRLock(SMeta *pMeta) { metaTrace("meta rlock %p", &pMeta->lock); - (void)taosThreadRwlockRdlock(&pMeta->lock); + if (taosThreadRwlockRdlock(&pMeta->lock) != 0) { + metaError("vgId:%d failed to lock %p", TD_VID(pMeta->pVnode), &pMeta->lock); + } } void metaWLock(SMeta *pMeta) { metaTrace("meta wlock %p", &pMeta->lock); - (void)taosThreadRwlockWrlock(&pMeta->lock); + if (taosThreadRwlockWrlock(&pMeta->lock) != 0) { + metaError("vgId:%d failed to lock %p", TD_VID(pMeta->pVnode), &pMeta->lock); + } } void metaULock(SMeta *pMeta) { metaTrace("meta ulock %p", &pMeta->lock); - (void)taosThreadRwlockUnlock(&pMeta->lock); + if (taosThreadRwlockUnlock(&pMeta->lock) != 0) { + metaError("vgId:%d failed to unlock %p", TD_VID(pMeta->pVnode), &pMeta->lock); + } } static void metaCleanup(SMeta **ppMeta) { @@ -223,7 +229,7 @@ static void metaCleanup(SMeta **ppMeta) { if (pMeta->pSkmDb) tdbTbClose(pMeta->pSkmDb); if (pMeta->pTbDb) tdbTbClose(pMeta->pTbDb); if (pMeta->pEnv) tdbClose(pMeta->pEnv); - (void)metaDestroyLock(pMeta); + metaDestroyLock(pMeta); taosMemoryFreeClear(*ppMeta); } diff --git a/source/dnode/vnode/src/meta/metaQuery.c b/source/dnode/vnode/src/meta/metaQuery.c index acee24e494..b78b141ba9 100644 --- a/source/dnode/vnode/src/meta/metaQuery.c +++ b/source/dnode/vnode/src/meta/metaQuery.c @@ -98,7 +98,7 @@ int metaReaderGetTableEntryByUidCache(SMetaReader *pReader, tb_uid_t uid) { SMeta *pMeta = pReader->pMeta; SMetaInfo info; - int32_t code = metaGetInfo(pMeta, uid, &info, pReader); + int32_t code = metaGetInfo(pMeta, uid, &info, pReader); if (TSDB_CODE_SUCCESS != code) { return terrno = (TSDB_CODE_NOT_FOUND == code ? TSDB_CODE_PAR_TABLE_NOT_EXIST : code); } @@ -260,7 +260,7 @@ void metaCloseTbCursor(SMTbCursor *pTbCur) { if (!pTbCur->paused) { metaReaderClear(&pTbCur->mr); if (pTbCur->pDbc) { - (void)tdbTbcClose((TBC *)pTbCur->pDbc); + tdbTbcClose((TBC *)pTbCur->pDbc); } } taosMemoryFree(pTbCur); @@ -270,7 +270,7 @@ void metaCloseTbCursor(SMTbCursor *pTbCur) { void metaPauseTbCursor(SMTbCursor *pTbCur) { if (!pTbCur->paused) { metaReaderClear(&pTbCur->mr); - (void)tdbTbcClose((TBC *)pTbCur->pDbc); + tdbTbcClose((TBC *)pTbCur->pDbc); pTbCur->paused = 1; } } @@ -353,7 +353,11 @@ int32_t metaTbCursorPrev(SMTbCursor *pTbCur, ETableType jumpTableType) { tDecoderClear(&pTbCur->mr.coder); - (void)metaGetTableEntryByVersion(&pTbCur->mr, ((SUidIdxVal *)pTbCur->pVal)[0].version, *(tb_uid_t *)pTbCur->pKey); + ret = metaGetTableEntryByVersion(&pTbCur->mr, ((SUidIdxVal *)pTbCur->pVal)[0].version, *(tb_uid_t *)pTbCur->pKey); + if (ret < 0) { + return ret; + } + if (pTbCur->mr.me.type == jumpTableType) { continue; } @@ -387,7 +391,10 @@ _query: SMetaEntry me = {0}; tDecoderInit(&dc, pData, nData); - (void)metaDecodeEntry(&dc, &me); + int32_t code = metaDecodeEntry(&dc, &me); + if (code) { + goto _err; + } if (me.type == TSDB_SUPER_TABLE) { if (sver == -1 || sver == me.stbEntry.schemaRow.version) { pSchema = tCloneSSchemaWrapper(&me.stbEntry.schemaRow); @@ -463,7 +470,7 @@ void metaCloseCtbCursor(SMCtbCursor *pCtbCur) { if (!pCtbCur->paused) { if (pCtbCur->pMeta && pCtbCur->lock) metaULock(pCtbCur->pMeta); if (pCtbCur->pCur) { - (void)tdbTbcClose(pCtbCur->pCur); + tdbTbcClose(pCtbCur->pCur); } } tdbFree(pCtbCur->pKey); @@ -474,7 +481,7 @@ void metaCloseCtbCursor(SMCtbCursor *pCtbCur) { void metaPauseCtbCursor(SMCtbCursor *pCtbCur) { if (!pCtbCur->paused) { - (void)tdbTbcClose((TBC *)pCtbCur->pCur); + tdbTbcClose((TBC *)pCtbCur->pCur); if (pCtbCur->lock) { metaULock(pCtbCur->pMeta); } @@ -502,17 +509,17 @@ int32_t metaResumeCtbCursor(SMCtbCursor *pCtbCur, int8_t first) { ctbIdxKey.suid = pCtbCur->suid; ctbIdxKey.uid = INT64_MIN; int c = 0; - (void)tdbTbcMoveTo(pCtbCur->pCur, &ctbIdxKey, sizeof(ctbIdxKey), &c); + ret = tdbTbcMoveTo(pCtbCur->pCur, &ctbIdxKey, sizeof(ctbIdxKey), &c); if (c > 0) { - (void)tdbTbcMoveToNext(pCtbCur->pCur); + ret = tdbTbcMoveToNext(pCtbCur->pCur); } } else { int c = 0; ret = tdbTbcMoveTo(pCtbCur->pCur, pCtbCur->pKey, pCtbCur->kLen, &c); if (c < 0) { - (void)tdbTbcMoveToPrev(pCtbCur->pCur); + ret = tdbTbcMoveToPrev(pCtbCur->pCur); } else { - (void)tdbTbcMoveToNext(pCtbCur->pCur); + ret = tdbTbcMoveToNext(pCtbCur->pCur); } } } @@ -570,9 +577,9 @@ SMStbCursor *metaOpenStbCursor(SMeta *pMeta, tb_uid_t suid) { } // move to the suid - (void)tdbTbcMoveTo(pStbCur->pCur, &suid, sizeof(suid), &c); + ret = tdbTbcMoveTo(pStbCur->pCur, &suid, sizeof(suid), &c); if (c > 0) { - (void)tdbTbcMoveToNext(pStbCur->pCur); + ret = tdbTbcMoveToNext(pStbCur->pCur); } return pStbCur; @@ -582,7 +589,7 @@ void metaCloseStbCursor(SMStbCursor *pStbCur) { if (pStbCur) { if (pStbCur->pMeta) metaULock(pStbCur->pMeta); if (pStbCur->pCur) { - (void)tdbTbcClose(pStbCur->pCur); + tdbTbcClose(pStbCur->pCur); tdbFree(pStbCur->pKey); tdbFree(pStbCur->pVal); @@ -616,17 +623,17 @@ STSchema *metaGetTbTSchema(SMeta *pMeta, tb_uid_t uid, int32_t sver, int lock) { return pTSchema; } -int32_t metaGetTbTSchemaNotNull(SMeta *pMeta, tb_uid_t uid, int32_t sver, int lock, STSchema** ppTSchema) { +int32_t metaGetTbTSchemaNotNull(SMeta *pMeta, tb_uid_t uid, int32_t sver, int lock, STSchema **ppTSchema) { *ppTSchema = metaGetTbTSchema(pMeta, uid, sver, lock); - if(*ppTSchema == NULL) { + if (*ppTSchema == NULL) { return terrno; } return TSDB_CODE_SUCCESS; } -int32_t metaGetTbTSchemaMaybeNull(SMeta *pMeta, tb_uid_t uid, int32_t sver, int lock, STSchema** ppTSchema) { +int32_t metaGetTbTSchemaMaybeNull(SMeta *pMeta, tb_uid_t uid, int32_t sver, int lock, STSchema **ppTSchema) { *ppTSchema = metaGetTbTSchema(pMeta, uid, sver, lock); - if(*ppTSchema == NULL && terrno == TSDB_CODE_OUT_OF_MEMORY) { + if (*ppTSchema == NULL && terrno == TSDB_CODE_OUT_OF_MEMORY) { return terrno; } return TSDB_CODE_SUCCESS; @@ -656,30 +663,30 @@ int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sv if (tdbTbcMoveTo(pSkmDbC, &skmDbKey, sizeof(skmDbKey), &c) < 0) { metaULock(pMeta); - (void)tdbTbcClose(pSkmDbC); + tdbTbcClose(pSkmDbC); code = TSDB_CODE_NOT_FOUND; goto _exit; } if (c == 0) { metaULock(pMeta); - (void)tdbTbcClose(pSkmDbC); + tdbTbcClose(pSkmDbC); code = TSDB_CODE_FAILED; metaError("meta/query: incorrect c: %" PRId32 ".", c); goto _exit; } if (c < 0) { - (void)tdbTbcMoveToPrev(pSkmDbC); + int32_t ret = tdbTbcMoveToPrev(pSkmDbC); } const void *pKey = NULL; int32_t nKey = 0; - (void)tdbTbcGet(pSkmDbC, &pKey, &nKey, NULL, NULL); + int32_t ret = tdbTbcGet(pSkmDbC, &pKey, &nKey, NULL, NULL); if (((SSkmDbKey *)pKey)->uid != skmDbKey.uid) { metaULock(pMeta); - (void)tdbTbcClose(pSkmDbC); + tdbTbcClose(pSkmDbC); code = TSDB_CODE_NOT_FOUND; goto _exit; } @@ -687,7 +694,7 @@ int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sv sver = ((SSkmDbKey *)pKey)->sver; metaULock(pMeta); - (void)tdbTbcClose(pSkmDbC); + tdbTbcClose(pSkmDbC); } } @@ -805,9 +812,9 @@ SMSmaCursor *metaOpenSmaCursor(SMeta *pMeta, tb_uid_t uid) { // move to the suid smaIdxKey.uid = uid; smaIdxKey.smaUid = INT64_MIN; - (void)tdbTbcMoveTo(pSmaCur->pCur, &smaIdxKey, sizeof(smaIdxKey), &c); + ret = tdbTbcMoveTo(pSmaCur->pCur, &smaIdxKey, sizeof(smaIdxKey), &c); if (c > 0) { - (void)tdbTbcMoveToNext(pSmaCur->pCur); + ret = tdbTbcMoveToNext(pSmaCur->pCur); } return pSmaCur; @@ -817,7 +824,7 @@ void metaCloseSmaCursor(SMSmaCursor *pSmaCur) { if (pSmaCur) { if (pSmaCur->pMeta) metaULock(pSmaCur->pMeta); if (pSmaCur->pCur) { - (void)tdbTbcClose(pSmaCur->pCur); + tdbTbcClose(pSmaCur->pCur); pSmaCur->pCur = NULL; tdbFree(pSmaCur->pKey); @@ -916,7 +923,7 @@ STSmaWrapper *metaGetSmaInfoByTable(SMeta *pMeta, tb_uid_t uid, bool deepCopy) { _err: metaReaderClear(&mr); taosArrayDestroy(pSmaIds); - (void)tFreeTSmaWrapper(pSW, deepCopy); + pSW = tFreeTSmaWrapper(pSW, deepCopy); return NULL; } @@ -1036,36 +1043,6 @@ const void *metaGetTableTagVal(const void *pTag, int16_t type, STagVal *val) { return NULL; } -#ifdef TAG_FILTER_DEBUG - if (IS_VAR_DATA_TYPE(val->type)) { - char *buf = taosMemoryCalloc(val->nData + 1, 1); - memcpy(buf, val->pData, val->nData); - metaDebug("metaTag table val varchar index:%d cid:%d type:%d value:%s", 1, val->cid, val->type, buf); - taosMemoryFree(buf); - } else { - double dval = 0; - GET_TYPED_DATA(dval, double, val->type, &val->i64); - metaDebug("metaTag table val number index:%d cid:%d type:%d value:%f", 1, val->cid, val->type, dval); - } - - SArray *pTagVals = NULL; - tTagToValArray((STag *)pTag, &pTagVals); - for (int i = 0; i < taosArrayGetSize(pTagVals); i++) { - STagVal *pTagVal = (STagVal *)taosArrayGet(pTagVals, i); - - if (IS_VAR_DATA_TYPE(pTagVal->type)) { - char *buf = taosMemoryCalloc(pTagVal->nData + 1, 1); - memcpy(buf, pTagVal->pData, pTagVal->nData); - metaDebug("metaTag table varchar index:%d cid:%d type:%d value:%s", i, pTagVal->cid, pTagVal->type, buf); - taosMemoryFree(buf); - } else { - double dval = 0; - GET_TYPED_DATA(dval, double, pTagVal->type, &pTagVal->i64); - metaDebug("metaTag table number index:%d cid:%d type:%d value:%f", i, pTagVal->cid, pTagVal->type, dval); - } - } -#endif - return val; } @@ -1088,6 +1065,9 @@ int32_t metaFilterCreateTime(void *pVnode, SMetaFltParam *arg, SArray *pUids) { SIdxCursor *pCursor = NULL; pCursor = (SIdxCursor *)taosMemoryCalloc(1, sizeof(SIdxCursor)); + if (pCursor == NULL) { + return terrno; + } pCursor->pMeta = pMeta; pCursor->suid = param->suid; pCursor->cid = param->cid; @@ -1144,7 +1124,7 @@ int32_t metaFilterCreateTime(void *pVnode, SMetaFltParam *arg, SArray *pUids) { END: if (pCursor->pMeta) metaULock(pCursor->pMeta); - if (pCursor->pCur) (void)tdbTbcClose(pCursor->pCur); + if (pCursor->pCur) tdbTbcClose(pCursor->pCur); taosMemoryFree(pCursor); return ret; } @@ -1160,6 +1140,9 @@ int32_t metaFilterTableName(void *pVnode, SMetaFltParam *arg, SArray *pUids) { SIdxCursor *pCursor = NULL; pCursor = (SIdxCursor *)taosMemoryCalloc(1, sizeof(SIdxCursor)); + if (pCursor == NULL) { + return terrno; + } pCursor->pMeta = pMeta; pCursor->suid = param->suid; pCursor->cid = param->cid; @@ -1216,7 +1199,7 @@ int32_t metaFilterTableName(void *pVnode, SMetaFltParam *arg, SArray *pUids) { END: if (pCursor->pMeta) metaULock(pCursor->pMeta); - if (pCursor->pCur) (void)tdbTbcClose(pCursor->pCur); + if (pCursor->pCur) tdbTbcClose(pCursor->pCur); taosMemoryFree(buf); taosMemoryFree(pKey); @@ -1235,6 +1218,9 @@ int32_t metaFilterTtl(void *pVnode, SMetaFltParam *arg, SArray *pUids) { SIdxCursor *pCursor = NULL; pCursor = (SIdxCursor *)taosMemoryCalloc(1, sizeof(SIdxCursor)); + if (pCursor == NULL) { + return terrno; + } pCursor->pMeta = pMeta; pCursor->suid = param->suid; pCursor->cid = param->cid; @@ -1245,7 +1231,7 @@ int32_t metaFilterTtl(void *pVnode, SMetaFltParam *arg, SArray *pUids) { END: if (pCursor->pMeta) metaULock(pCursor->pMeta); - if (pCursor->pCur) (void)tdbTbcClose(pCursor->pCur); + if (pCursor->pCur) tdbTbcClose(pCursor->pCur); taosMemoryFree(buf); taosMemoryFree(pKey); @@ -1422,7 +1408,7 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) { END: if (pCursor->pMeta) metaULock(pCursor->pMeta); - if (pCursor->pCur) (void)tdbTbcClose(pCursor->pCur); + if (pCursor->pCur) tdbTbcClose(pCursor->pCur); if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf); tDecoderClear(&dc); tdbFree(pData); @@ -1604,7 +1590,10 @@ int32_t metaGetInfo(SMeta *pMeta, int64_t uid, SMetaInfo *pInfo, SMetaReader *pR } // upsert the cache metaWLock(pMeta); - (void)metaCacheUpsert(pMeta, pInfo); + int32_t ret = metaCacheUpsert(pMeta, pInfo); + if (ret != 0) { + metaError("vgId:%d, failed to upsert cache, uid:%" PRId64, TD_VID(pMeta->pVnode), uid); + } metaULock(pMeta); if (lock) { @@ -1654,7 +1643,12 @@ int32_t metaGetStbStats(void *pVnode, int64_t uid, int64_t *numOfTables, int32_t // upsert the cache metaWLock(pVnodeObj->pMeta); - (void)metaStatsCacheUpsert(pVnodeObj->pMeta, &state); + + int32_t ret = metaStatsCacheUpsert(pVnodeObj->pMeta, &state); + if (ret) { + metaError("failed to upsert stats, uid:%" PRId64 ", ctbNum:%" PRId64 ", colNum:%d", uid, ctbNum, colNum); + } + metaULock(pVnodeObj->pMeta); _exit: @@ -1667,6 +1661,10 @@ void metaUpdateStbStats(SMeta *pMeta, int64_t uid, int64_t deltaCtb, int32_t del if (metaStatsCacheGet(pMeta, uid, &stats) == TSDB_CODE_SUCCESS) { stats.ctbNum += deltaCtb; stats.colNum += deltaCol; - (void)metaStatsCacheUpsert(pMeta, &stats); + int32_t code = metaStatsCacheUpsert(pMeta, &stats); + if (code) { + metaError("vgId:%d, failed to update stats, uid:%" PRId64 ", ctbNum:%" PRId64 ", colNum:%d", + TD_VID(pMeta->pVnode), uid, deltaCtb, deltaCol); + } } } diff --git a/source/dnode/vnode/src/meta/metaSnapshot.c b/source/dnode/vnode/src/meta/metaSnapshot.c index a55be126c6..0f77ff5e01 100644 --- a/source/dnode/vnode/src/meta/metaSnapshot.c +++ b/source/dnode/vnode/src/meta/metaSnapshot.c @@ -59,7 +59,7 @@ _exit: void metaSnapReaderClose(SMetaSnapReader** ppReader) { if (ppReader && *ppReader) { - (void)tdbTbcClose((*ppReader)->pTbc); + tdbTbcClose((*ppReader)->pTbc); taosMemoryFree(*ppReader); *ppReader = NULL; } @@ -221,7 +221,7 @@ static void destroySTableInfoForChildTable(void* data) { static int32_t MoveToSnapShotVersion(SSnapContext* ctx) { int32_t code = 0; - (void)tdbTbcClose((TBC*)ctx->pCur); + tdbTbcClose((TBC*)ctx->pCur); code = tdbTbcOpen(ctx->pMeta->pTbDb, (TBC**)&ctx->pCur, NULL); if (code != 0) { return TAOS_GET_TERRNO(code); @@ -239,7 +239,7 @@ static int32_t MoveToSnapShotVersion(SSnapContext* ctx) { } static int32_t MoveToPosition(SSnapContext* ctx, int64_t ver, int64_t uid) { - (void)tdbTbcClose((TBC*)ctx->pCur); + tdbTbcClose((TBC*)ctx->pCur); int32_t code = tdbTbcOpen(ctx->pMeta->pTbDb, (TBC**)&ctx->pCur, NULL); if (code != 0) { return TAOS_GET_TERRNO(code); @@ -254,7 +254,7 @@ static int32_t MoveToPosition(SSnapContext* ctx, int64_t ver, int64_t uid) { } static int32_t MoveToFirst(SSnapContext* ctx) { - (void)tdbTbcClose((TBC*)ctx->pCur); + tdbTbcClose((TBC*)ctx->pCur); int32_t code = tdbTbcOpen(ctx->pMeta->pTbDb, (TBC**)&ctx->pCur, NULL); if (code != 0) { return TAOS_GET_TERRNO(code); @@ -455,7 +455,7 @@ int32_t buildSnapContext(SVnode* pVnode, int64_t snapVersion, int64_t suid, int8 } void destroySnapContext(SSnapContext* ctx) { - (void)tdbTbcClose((TBC*)ctx->pCur); + tdbTbcClose((TBC*)ctx->pCur); taosArrayDestroy(ctx->idList); taosHashCleanup(ctx->idVersion); taosHashCleanup(ctx->suidInfo); diff --git a/source/dnode/vnode/src/meta/metaTable.c b/source/dnode/vnode/src/meta/metaTable.c index b568b03c2c..fd7e6171c2 100644 --- a/source/dnode/vnode/src/meta/metaTable.c +++ b/source/dnode/vnode/src/meta/metaTable.c @@ -89,12 +89,15 @@ static void metaGetEntryInfo(const SMetaEntry *pEntry, SMetaInfo *pInfo) { static int metaUpdateMetaRsp(tb_uid_t uid, char *tbName, SSchemaWrapper *pSchema, STableMetaRsp *pMetaRsp) { pMetaRsp->pSchemas = taosMemoryMalloc(pSchema->nCols * sizeof(SSchema)); - if (NULL == pMetaRsp->pSchemas) { - return terrno = TSDB_CODE_OUT_OF_MEMORY; + return terrno; } pMetaRsp->pSchemaExt = taosMemoryMalloc(pSchema->nCols * sizeof(SSchemaExt)); + if (pMetaRsp->pSchemaExt == NULL) { + taosMemoryFree(pMetaRsp->pSchemas); + return terrno; + } tstrncpy(pMetaRsp->tbName, tbName, TSDB_TABLE_NAME_LEN); pMetaRsp->numOfColumns = pSchema->nCols; @@ -172,7 +175,11 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const } if (term != NULL) { - (void)indexMultiTermAdd(terms, term); + int32_t ret = indexMultiTermAdd(terms, term); + if (ret < 0) { + metaError("vgId:%d, failed to add term to multi term, uid: %" PRId64 ", key: %s, type: %d, ret: %d", + TD_VID(pMeta->pVnode), tuid, key, type, ret); + } } else { code = terrno; goto _exception; @@ -251,7 +258,11 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche term = indexTermCreate(suid, DEL_VALUE, TSDB_DATA_TYPE_BOOL, key, nKey, (const char *)&val, len); } if (term != NULL) { - (void)indexMultiTermAdd(terms, term); + int32_t ret = indexMultiTermAdd(terms, term); + if (ret < 0) { + metaError("vgId:%d, failed to add term to multi term, uid: %" PRId64 ", key: %s, type: %d, ret: %d", + TD_VID(pMeta->pVnode), tuid, key, type, ret); + } } else { code = terrno; goto _exception; @@ -348,6 +359,7 @@ int metaDropSTable(SMeta *pMeta, int64_t verison, SVDropStbReq *pReq, SArray *tb int c = 0; int rc = 0; int32_t lino; + int32_t ret; // check if super table exists rc = tdbTbGet(pMeta->pNameIdx, pReq->name, strlen(pReq->name) + 1, &pData, &nData); @@ -366,7 +378,7 @@ int metaDropSTable(SMeta *pMeta, int64_t verison, SVDropStbReq *pReq, SArray *tb rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = pReq->suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (rc < 0) { - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); metaWLock(pMeta); goto _drop_super_table; } @@ -383,32 +395,63 @@ int metaDropSTable(SMeta *pMeta, int64_t verison, SVDropStbReq *pReq, SArray *tb if (taosArrayPush(tbUidList, &(((SCtbIdxKey *)pKey)->uid)) == NULL) { tdbFree(pKey); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); return terrno; } } - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); - (void)tsdbCacheDropSubTables(pMeta->pVnode->pTsdb, tbUidList, pReq->suid); + ret = tsdbCacheDropSubTables(pMeta->pVnode->pTsdb, tbUidList, pReq->suid); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid, + tstrerror(terrno)); + } metaWLock(pMeta); for (int32_t iChild = 0; iChild < taosArrayGetSize(tbUidList); iChild++) { tb_uid_t uid = *(tb_uid_t *)taosArrayGet(tbUidList, iChild); - (void)metaDropTableByUid(pMeta, uid, NULL, NULL, NULL); + ret = metaDropTableByUid(pMeta, uid, NULL, NULL, NULL); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(terrno)); + } } // drop super table _drop_super_table: tdbTbGet(pMeta->pUidIdx, &pReq->suid, sizeof(tb_uid_t), &pData, &nData); - (void)tdbTbDelete(pMeta->pTbDb, &(STbDbKey){.version = ((SUidIdxVal *)pData)[0].version, .uid = pReq->suid}, + ret = tdbTbDelete(pMeta->pTbDb, &(STbDbKey){.version = ((SUidIdxVal *)pData)[0].version, .uid = pReq->suid}, sizeof(STbDbKey), pMeta->txn); - (void)tdbTbDelete(pMeta->pNameIdx, pReq->name, strlen(pReq->name) + 1, pMeta->txn); - (void)tdbTbDelete(pMeta->pUidIdx, &pReq->suid, sizeof(tb_uid_t), pMeta->txn); - (void)tdbTbDelete(pMeta->pSuidIdx, &pReq->suid, sizeof(tb_uid_t), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid, + tstrerror(terrno)); + } - (void)metaStatsCacheDrop(pMeta, pReq->suid); + ret = tdbTbDelete(pMeta->pNameIdx, pReq->name, strlen(pReq->name) + 1, pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid, + tstrerror(terrno)); + } + + ret = tdbTbDelete(pMeta->pUidIdx, &pReq->suid, sizeof(tb_uid_t), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid, + tstrerror(terrno)); + } + + ret = tdbTbDelete(pMeta->pSuidIdx, &pReq->suid, sizeof(tb_uid_t), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid, + tstrerror(terrno)); + } + + ret = metaStatsCacheDrop(pMeta, pReq->suid); + if (ret < 0) { + metaError("vgId:%d, failed to drop stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, pReq->suid, + tstrerror(terrno)); + } metaULock(pMeta); @@ -434,7 +477,7 @@ static int32_t metaGetSubtables(SMeta *pMeta, int64_t suid, SArray *uids) { TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL)); int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (rc < 0) { - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); metaWLock(pMeta); return 0; } @@ -451,14 +494,14 @@ static int32_t metaGetSubtables(SMeta *pMeta, int64_t suid, SArray *uids) { if (taosArrayPush(uids, &(((SCtbIdxKey *)pKey)->uid)) == NULL) { tdbFree(pKey); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); return terrno; } } tdbFree(pKey); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); return 0; } @@ -477,14 +520,14 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL)); ret = tdbTbcMoveTo(pUidIdxc, &pReq->suid, sizeof(tb_uid_t), &c); if (ret < 0 || c) { - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pUidIdxc); return terrno = TSDB_CODE_TDB_STB_NOT_EXIST; } ret = tdbTbcGet(pUidIdxc, NULL, NULL, &pData, &nData); if (ret < 0) { - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pUidIdxc); return terrno = TSDB_CODE_TDB_STB_NOT_EXIST; } @@ -494,8 +537,8 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL)); ret = tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = pReq->suid, .version = oversion}), sizeof(STbDbKey), &c); if (!(ret == 0 && c == 0)) { - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); metaError("meta/table: invalide ret: %" PRId32 " or c: %" PRId32 "alter stb failed.", ret, c); return terrno = TSDB_CODE_TDB_STB_NOT_EXIST; @@ -503,20 +546,27 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { ret = tdbTbcGet(pTbDbc, NULL, NULL, &pData, &nData); if (ret < 0) { - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); return terrno = TSDB_CODE_TDB_STB_NOT_EXIST; } if ((oStbEntry.pBuf = taosMemoryMalloc(nData)) == NULL) { - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); return terrno; } memcpy(oStbEntry.pBuf, pData, nData); tDecoderInit(&dc, oStbEntry.pBuf, nData); - (void)metaDecodeEntry(&dc, &oStbEntry); + ret = metaDecodeEntry(&dc, &oStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to decode stb:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + return terrno; + } nStbEntry.version = version; nStbEntry.type = TSDB_SUPER_TABLE; @@ -538,8 +588,8 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { if (uids == NULL) { if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf); tDecoderClear(&dc); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); return terrno; } if (deltaCol == 1) { @@ -547,7 +597,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { int8_t col_type = pReq->schemaRow.pSchema[nCols - 1].type; TAOS_CHECK_RETURN(metaGetSubtables(pMeta, pReq->suid, uids)); - (void)tsdbCacheNewSTableColumn(pTsdb, uids, cid, col_type); + TAOS_CHECK_RETURN(tsdbCacheNewSTableColumn(pTsdb, uids, cid, col_type)); } else if (deltaCol == -1) { int16_t cid = -1; bool hasPrimaryKey = false; @@ -563,7 +613,7 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { if (cid != -1) { TAOS_CHECK_RETURN(metaGetSubtables(pMeta, pReq->suid, uids)); - (void)tsdbCacheDropSTableColumn(pTsdb, uids, cid, hasPrimaryKey); + TAOS_CHECK_RETURN(tsdbCacheDropSTableColumn(pTsdb, uids, cid, hasPrimaryKey)); } } if (uids) taosArrayDestroy(uids); @@ -572,25 +622,41 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { metaWLock(pMeta); // compare two entry if (oStbEntry.stbEntry.schemaRow.version != pReq->schemaRow.version) { - (void)metaSaveToSkmDb(pMeta, &nStbEntry); + ret = metaSaveToSkmDb(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to save skm db:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } } // update table.db - (void)metaSaveToTbDb(pMeta, &nStbEntry); + ret = metaSaveToTbDb(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to save tb db:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } // update uid index - (void)metaUpdateUidIdx(pMeta, &nStbEntry); + ret = metaUpdateUidIdx(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to update uid idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } // metaStatsCacheDrop(pMeta, nStbEntry.uid); if (updStat) { - (void)metaUpdateStbStats(pMeta, pReq->suid, 0, deltaCol); + metaUpdateStbStats(pMeta, pReq->suid, 0, deltaCol); } metaULock(pMeta); if (updStat) { int64_t ctbNum; - (void)metaGetStbStats(pMeta->pVnode, pReq->suid, &ctbNum, NULL); + ret = metaGetStbStats(pMeta->pVnode, pReq->suid, &ctbNum, NULL); + if (ret < 0) { + metaError("vgId:%d, failed to get stb stats:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } pMeta->pVnode->config.vndStats.numOfTimeSeries += (ctbNum * deltaCol); metaTimeSeriesNotifyCheck(pMeta); } @@ -600,8 +666,8 @@ int metaAlterSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { _exit: if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf); tDecoderClear(&dc); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); return 0; } int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { @@ -692,7 +758,7 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { code = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (code < 0) { - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } for (;;) { @@ -702,7 +768,7 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { if (code < 0) { tdbFree(pKey); tdbFree(pVal); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); pCtbIdxc = NULL; break; } @@ -737,12 +803,16 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { tdbFree(pVal); if (code < 0) { metaDestroyTagIdxKey(pTagIdxKey); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } metaWLock(pMeta); - (void)tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn); + ret = tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to upsert tag idx key:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } metaULock(pMeta); metaDestroyTagIdxKey(pTagIdxKey); @@ -759,16 +829,24 @@ int metaAddIndexToSTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) { metaWLock(pMeta); // update table.db - (void)metaSaveToTbDb(pMeta, &nStbEntry); + ret = metaSaveToTbDb(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to save tb db:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } // update uid index - (void)metaUpdateUidIdx(pMeta, &nStbEntry); + ret = metaUpdateUidIdx(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to update uid idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->suid, tstrerror(ret)); + } metaULock(pMeta); if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf); tDecoderClear(&dc); tdbFree(pData); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); return TSDB_CODE_SUCCESS; _err: if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf); @@ -841,7 +919,7 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) code = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (code < 0) { - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } for (;;) { @@ -852,7 +930,7 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) if (code < 0) { tdbFree(pKey); tdbFree(pVal); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); pCtbIdxc = NULL; break; } @@ -888,12 +966,16 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) tdbFree(pVal); if (code < 0) { metaDestroyTagIdxKey(pTagIdxKey); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } metaWLock(pMeta); - (void)tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn); + ret = tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete tag idx key:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->stb, + pReq->stbUid, tstrerror(ret)); + } metaULock(pMeta); metaDestroyTagIdxKey(pTagIdxKey); pTagIdxKey = NULL; @@ -916,7 +998,7 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) tDeleteSColCmprWrapper(cmpr); code = TSDB_CODE_OUT_OF_MEMORY; - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } @@ -929,9 +1011,17 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) metaWLock(pMeta); // update table.db - (void)metaSaveToTbDb(pMeta, &nStbEntry); + ret = metaSaveToTbDb(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to save tb db:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->stb, + pReq->stbUid, tstrerror(ret)); + } // update uid index - (void)metaUpdateUidIdx(pMeta, &nStbEntry); + ret = metaUpdateUidIdx(pMeta, &nStbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to update uid idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->stb, + pReq->stbUid, tstrerror(ret)); + } metaULock(pMeta); tDeleteSchemaWrapper(tag); @@ -942,7 +1032,7 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) tDecoderClear(&dc); tdbFree(pData); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); return TSDB_CODE_SUCCESS; _err: if (oStbEntry.pBuf) taosMemoryFree(oStbEntry.pBuf); @@ -955,6 +1045,7 @@ _err: int metaCreateTable(SMeta *pMeta, int64_t ver, SVCreateTbReq *pReq, STableMetaRsp **pMetaRsp) { SMetaEntry me = {0}; SMetaReader mr = {0}; + int32_t ret; // validate message if (pReq->type != TSDB_CHILD_TABLE && pReq->type != TSDB_NORMAL_TABLE) { @@ -1030,18 +1121,34 @@ int metaCreateTable(SMeta *pMeta, int64_t ver, SVCreateTbReq *pReq, STableMetaRs if (!sysTbl) { int32_t nCols = 0; - (void)metaGetStbStats(pMeta->pVnode, me.ctbEntry.suid, 0, &nCols); + ret = metaGetStbStats(pMeta->pVnode, me.ctbEntry.suid, 0, &nCols); + if (ret < 0) { + metaError("vgId:%d, failed to get stb stats:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->ctb.suid, tstrerror(ret)); + } pStats->numOfTimeSeries += nCols - 1; } metaWLock(pMeta); - (void)metaUpdateStbStats(pMeta, me.ctbEntry.suid, 1, 0); - (void)metaUidCacheClear(pMeta, me.ctbEntry.suid); - (void)metaTbGroupCacheClear(pMeta, me.ctbEntry.suid); + metaUpdateStbStats(pMeta, me.ctbEntry.suid, 1, 0); + ret = metaUidCacheClear(pMeta, me.ctbEntry.suid); + if (ret < 0) { + metaError("vgId:%d, failed to clear uid cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->ctb.suid, tstrerror(ret)); + } + ret = metaTbGroupCacheClear(pMeta, me.ctbEntry.suid); + if (ret < 0) { + metaError("vgId:%d, failed to clear group cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, + pReq->ctb.suid, tstrerror(ret)); + } metaULock(pMeta); if (!TSDB_CACHE_NO(pMeta->pVnode->config)) { - (void)tsdbCacheNewTable(pMeta->pVnode->pTsdb, me.uid, me.ctbEntry.suid, NULL); + ret = tsdbCacheNewTable(pMeta->pVnode->pTsdb, me.uid, me.ctbEntry.suid, NULL); + if (ret < 0) { + metaError("vgId:%d, failed to create table:%s since %s", TD_VID(pMeta->pVnode), pReq->name, tstrerror(ret)); + goto _err; + } } } else { me.ntbEntry.btime = pReq->btime; @@ -1057,7 +1164,11 @@ int metaCreateTable(SMeta *pMeta, int64_t ver, SVCreateTbReq *pReq, STableMetaRs pStats->numOfNTimeSeries += me.ntbEntry.schemaRow.nCols - 1; if (!TSDB_CACHE_NO(pMeta->pVnode->config)) { - (void)tsdbCacheNewTable(pMeta->pVnode->pTsdb, me.uid, -1, &me.ntbEntry.schemaRow); + ret = tsdbCacheNewTable(pMeta->pVnode->pTsdb, me.uid, -1, &me.ntbEntry.schemaRow); + if (ret < 0) { + metaError("vgId:%d, failed to create table:%s since %s", TD_VID(pMeta->pVnode), pReq->name, tstrerror(ret)); + goto _err; + } } } @@ -1075,7 +1186,11 @@ int metaCreateTable(SMeta *pMeta, int64_t ver, SVCreateTbReq *pReq, STableMetaRs (*pMetaRsp)->suid = pReq->ctb.suid; strcpy((*pMetaRsp)->tbName, pReq->name); } else { - (void)metaUpdateMetaRsp(pReq->uid, pReq->name, &pReq->ntb.schemaRow, *pMetaRsp); + ret = metaUpdateMetaRsp(pReq->uid, pReq->name, &pReq->ntb.schemaRow, *pMetaRsp); + if (ret < 0) { + metaError("vgId:%d, failed to update meta rsp:%s since %s", TD_VID(pMeta->pVnode), pReq->name, + tstrerror(ret)); + } for (int32_t i = 0; i < pReq->colCmpr.nCols; i++) { SColCmpr *p = &pReq->colCmpr.pColCmpr[i]; (*pMetaRsp)->pSchemaExt[i].colId = p->id; @@ -1132,7 +1247,11 @@ int metaDropTable(SMeta *pMeta, int64_t version, SVDropTbReq *pReq, SArray *tbUi } if (!TSDB_CACHE_NO(pMeta->pVnode->config)) { - (void)tsdbCacheDropTable(pMeta->pVnode->pTsdb, uid, suid, NULL); + int32_t ret = tsdbCacheDropTable(pMeta->pVnode->pTsdb, uid, suid, NULL); + if (ret < 0) { + metaError("vgId:%d, failed to drop table:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), pReq->name, uid, + tstrerror(ret)); + } } } @@ -1214,7 +1333,7 @@ static int32_t metaFilterTableByHash(SMeta *pMeta, SArray *uidList) { code = tdbTbcMoveToFirst(pCur); if (code) { - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return code; } @@ -1230,7 +1349,11 @@ static int32_t metaFilterTableByHash(SMeta *pMeta, SArray *uidList) { SMetaEntry me = {0}; SDecoder dc = {0}; tDecoderInit(&dc, pData, nData); - (void)metaDecodeEntry(&dc, &me); + code = metaDecodeEntry(&dc, &me); + if (code < 0) { + tDecoderClear(&dc); + return code; + } if (me.type != TSDB_SUPER_TABLE) { char tbFName[TSDB_TABLE_FNAME_LEN + 1]; @@ -1248,7 +1371,7 @@ static int32_t metaFilterTableByHash(SMeta *pMeta, SArray *uidList) { } tdbFree(pData); tdbFree(pKey); - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return 0; } @@ -1337,6 +1460,7 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p int rc = 0; SMetaEntry e = {0}; SDecoder dc = {0}; + int32_t ret = 0; rc = tdbTbGet(pMeta->pUidIdx, &uid, sizeof(uid), &pData, &nData); if (rc < 0) { @@ -1344,7 +1468,11 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p } int64_t version = ((SUidIdxVal *)pData)[0].version; - (void)tdbTbGet(pMeta->pTbDb, &(STbDbKey){.version = version, .uid = uid}, sizeof(STbDbKey), &pData, &nData); + rc = tdbTbGet(pMeta->pTbDb, &(STbDbKey){.version = version, .uid = uid}, sizeof(STbDbKey), &pData, &nData); + if (rc < 0) { + tdbFree(pData); + return rc; + } tDecoderInit(&dc, pData, nData); rc = metaDecodeEntry(&dc, &e); @@ -1367,7 +1495,12 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p SMetaEntry stbEntry = {0}; tDecoderInit(&tdc, tData, tLen); - (void)metaDecodeEntry(&tdc, &stbEntry); + int32_t ret = metaDecodeEntry(&tdc, &stbEntry); + if (ret < 0) { + metaError("vgId:%d, failed to decode child table:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, + e.ctbEntry.suid, tstrerror(ret)); + return ret; + } if (pSysTbl) *pSysTbl = metaTbInFilterCache(pMeta, stbEntry.name, 1) ? 1 : 0; @@ -1375,7 +1508,11 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p SSchemaWrapper *pTagSchema = &stbEntry.stbEntry.schemaTag; if (pTagSchema->nCols == 1 && pTagSchema->pSchema[0].type == TSDB_DATA_TYPE_JSON) { pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[0]; - (void)metaDelJsonVarFromIdx(pMeta, &e, pTagColumn); + ret = metaDelJsonVarFromIdx(pMeta, &e, pTagColumn); + if (ret < 0) { + metaError("vgId:%d, failed to delete json var from idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), + e.name, e.uid, tstrerror(ret)); + } } else { for (int i = 0; i < pTagSchema->nCols; i++) { pTagColumn = &stbEntry.stbEntry.schemaTag.pSchema[i]; @@ -1403,7 +1540,11 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p if (metaCreateTagIdxKey(e.ctbEntry.suid, pTagColumn->colId, pTagData, nTagData, pTagColumn->type, uid, &pTagIdxKey, &nTagIdxKey) == 0) { - (void)tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn); + ret = tdbTbDelete(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete tag idx key:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), + e.name, e.uid, tstrerror(ret)); + } } metaDestroyTagIdxKey(pTagIdxKey); pTagIdxKey = NULL; @@ -1415,9 +1556,21 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p } } - (void)tdbTbDelete(pMeta->pTbDb, &(STbDbKey){.version = version, .uid = uid}, sizeof(STbDbKey), pMeta->txn); - (void)tdbTbDelete(pMeta->pNameIdx, e.name, strlen(e.name) + 1, pMeta->txn); - (void)tdbTbDelete(pMeta->pUidIdx, &uid, sizeof(uid), pMeta->txn); + ret = tdbTbDelete(pMeta->pTbDb, &(STbDbKey){.version = version, .uid = uid}, sizeof(STbDbKey), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete table:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } + ret = tdbTbDelete(pMeta->pNameIdx, e.name, strlen(e.name) + 1, pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete name idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } + ret = tdbTbDelete(pMeta->pUidIdx, &uid, sizeof(uid), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete uid idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } if (e.type == TSDB_CHILD_TABLE || e.type == TSDB_NORMAL_TABLE) metaDeleteBtimeIdx(pMeta, &e); if (e.type == TSDB_NORMAL_TABLE) metaDeleteNcolIdx(pMeta, &e); @@ -1425,13 +1578,25 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p if (e.type != TSDB_SUPER_TABLE) metaDeleteTtl(pMeta, &e); if (e.type == TSDB_CHILD_TABLE) { - (void)tdbTbDelete(pMeta->pCtbIdx, &(SCtbIdxKey){.suid = e.ctbEntry.suid, .uid = uid}, sizeof(SCtbIdxKey), - pMeta->txn); + ret = + tdbTbDelete(pMeta->pCtbIdx, &(SCtbIdxKey){.suid = e.ctbEntry.suid, .uid = uid}, sizeof(SCtbIdxKey), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete ctb idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } --pMeta->pVnode->config.vndStats.numOfCTables; - (void)metaUpdateStbStats(pMeta, e.ctbEntry.suid, -1, 0); - (void)metaUidCacheClear(pMeta, e.ctbEntry.suid); - (void)metaTbGroupCacheClear(pMeta, e.ctbEntry.suid); + metaUpdateStbStats(pMeta, e.ctbEntry.suid, -1, 0); + ret = metaUidCacheClear(pMeta, e.ctbEntry.suid); + if (ret < 0) { + metaError("vgId:%d, failed to clear uid cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, + e.ctbEntry.suid, tstrerror(ret)); + } + ret = metaTbGroupCacheClear(pMeta, e.ctbEntry.suid); + if (ret < 0) { + metaError("vgId:%d, failed to clear group cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, + e.ctbEntry.suid, tstrerror(ret)); + } /* if (!TSDB_CACHE_NO(pMeta->pVnode->config)) { tsdbCacheDropTable(pMeta->pVnode->pTsdb, e.uid, e.ctbEntry.suid, NULL); @@ -1449,16 +1614,36 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p } */ } else if (e.type == TSDB_SUPER_TABLE) { - (void)tdbTbDelete(pMeta->pSuidIdx, &e.uid, sizeof(tb_uid_t), pMeta->txn); + ret = tdbTbDelete(pMeta->pSuidIdx, &e.uid, sizeof(tb_uid_t), pMeta->txn); + if (ret < 0) { + metaError("vgId:%d, failed to delete suid idx:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } // drop schema.db (todo) - (void)metaStatsCacheDrop(pMeta, uid); - (void)metaUidCacheClear(pMeta, uid); - (void)metaTbGroupCacheClear(pMeta, uid); + ret = metaStatsCacheDrop(pMeta, uid); + if (ret < 0) { + metaError("vgId:%d, failed to drop stats cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } + ret = metaUidCacheClear(pMeta, uid); + if (ret < 0) { + metaError("vgId:%d, failed to clear uid cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } + ret = metaTbGroupCacheClear(pMeta, uid); + if (ret < 0) { + metaError("vgId:%d, failed to clear group cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, + e.uid, tstrerror(ret)); + } --pMeta->pVnode->config.vndStats.numOfSTables; } - (void)metaCacheDrop(pMeta, uid); + ret = metaCacheDrop(pMeta, uid); + if (ret < 0) { + metaError("vgId:%d, failed to drop cache:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name, e.uid, + tstrerror(ret)); + } tDecoderClear(&dc); tdbFree(pData); @@ -1532,39 +1717,43 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl TBC *pUidIdxc = NULL; TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL)); - (void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); + ret = tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); if (c != 0) { - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pUidIdxc); metaError("meta/table: invalide c: %" PRId32 " alt tb column failed.", c); return TSDB_CODE_FAILED; } - (void)tdbTbcGet(pUidIdxc, NULL, NULL, &pData, &nData); + ret = tdbTbcGet(pUidIdxc, NULL, NULL, &pData, &nData); oversion = ((SUidIdxVal *)pData)[0].version; // search table.db TBC *pTbDbc = NULL; TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL)); - (void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); + ret = tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); if (c != 0) { - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); metaError("meta/table: invalide c: %" PRId32 " alt tb column failed.", c); return TSDB_CODE_FAILED; } - (void)tdbTbcGet(pTbDbc, NULL, NULL, &pData, &nData); + ret = tdbTbcGet(pTbDbc, NULL, NULL, &pData, &nData); // get table entry SDecoder dc = {0}; - entry.pBuf = taosMemoryMalloc(nData); + if ((entry.pBuf = taosMemoryMalloc(nData)) == NULL) { + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + return terrno; + } memcpy(entry.pBuf, pData, nData); tDecoderInit(&dc, entry.pBuf, nData); ret = metaDecodeEntry(&dc, &entry); if (ret != 0) { - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); tDecoderClear(&dc); metaError("meta/table: invalide ret: %" PRId32 " alt tb column failed.", ret); return ret; @@ -1625,6 +1814,9 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl pSchema->version++; pSchema->nCols++; pNewSchema = taosMemoryMalloc(sizeof(SSchema) * pSchema->nCols); + if (pNewSchema == NULL) { + goto _err; + } memcpy(pNewSchema, pSchema->pSchema, sizeof(SSchema) * (pSchema->nCols - 1)); pSchema->pSchema = pNewSchema; pSchema->pSchema[entry.ntbEntry.schemaRow.nCols - 1].bytes = pAlterTbReq->bytes; @@ -1760,16 +1952,16 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl if (pNewSchema) taosMemoryFree(pNewSchema); if (freeColCmpr) taosMemoryFree(entry.colCmpr.pColCmpr); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); tDecoderClear(&dc); return 0; _err: if (entry.pBuf) taosMemoryFree(entry.pBuf); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); tDecoderClear(&dc); return terrno != 0 ? terrno : TSDB_CODE_FAILED; @@ -1807,7 +1999,7 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL)); (void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); if (c != 0) { - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pUidIdxc); metaError("meta/table: invalide c: %" PRId32 " update tb tag val failed.", c); return terrno = TSDB_CODE_TDB_TABLE_NOT_EXIST; } @@ -1824,15 +2016,19 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL)); (void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); if (c != 0) { - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); metaError("meta/table: invalide c: %" PRId32 " update tb tag val failed.", c); return terrno = TSDB_CODE_TDB_TABLE_NOT_EXIST; } (void)tdbTbcGet(pTbDbc, NULL, NULL, &pData, &nData); - ctbEntry.pBuf = taosMemoryMalloc(nData); + if ((ctbEntry.pBuf = taosMemoryMalloc(nData)) == NULL) { + (void)tdbTbcClose(pUidIdxc); + (void)tdbTbcClose(pTbDbc); + return terrno; + } memcpy(ctbEntry.pBuf, pData, nData); tDecoderInit(&dc1, ctbEntry.pBuf, nData); (void)metaDecodeEntry(&dc1, &ctbEntry); @@ -1954,8 +2150,8 @@ static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pA taosMemoryFree((void *)ctbEntry.ctbEntry.pTags); if (ctbEntry.pBuf) taosMemoryFree(ctbEntry.pBuf); if (stbEntry.pBuf) tdbFree(stbEntry.pBuf); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); return 0; _err: @@ -1963,8 +2159,8 @@ _err: tDecoderClear(&dc2); if (ctbEntry.pBuf) taosMemoryFree(ctbEntry.pBuf); if (stbEntry.pBuf) tdbFree(stbEntry.pBuf); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); return -1; } @@ -1995,7 +2191,7 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pUidIdx, &pUidIdxc, NULL)); (void)tdbTbcMoveTo(pUidIdxc, &uid, sizeof(uid), &c); if (c != 0) { - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pUidIdxc); metaError("meta/table: invalide c: %" PRId32 " update tb options failed.", c); return TSDB_CODE_FAILED; } @@ -2009,8 +2205,8 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pTbDb, &pTbDbc, NULL)); (void)tdbTbcMoveTo(pTbDbc, &((STbDbKey){.uid = uid, .version = oversion}), sizeof(STbDbKey), &c); if (c != 0) { - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); metaError("meta/table: invalide c: %" PRId32 " update tb options failed.", c); return TSDB_CODE_FAILED; } @@ -2019,14 +2215,18 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p // get table entry SDecoder dc = {0}; - entry.pBuf = taosMemoryMalloc(nData); + if ((entry.pBuf = taosMemoryMalloc(nData)) == NULL) { + (void)tdbTbcClose(pUidIdxc); + (void)tdbTbcClose(pTbDbc); + return terrno; + } memcpy(entry.pBuf, pData, nData); tDecoderInit(&dc, entry.pBuf, nData); ret = metaDecodeEntry(&dc, &entry); if (ret != 0) { tDecoderClear(&dc); - (void)tdbTbcClose(pUidIdxc); - (void)tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); metaError("meta/table: invalide ret: %" PRId32 " alt tb options failed.", ret); return TSDB_CODE_FAILED; } @@ -2063,8 +2263,8 @@ static int metaUpdateTableOptions(SMeta *pMeta, int64_t version, SVAlterTbReq *p metaULock(pMeta); - (void)tdbTbcClose(pTbDbc); - (void)tdbTbcClose(pUidIdxc); + tdbTbcClose(pTbDbc); + tdbTbcClose(pUidIdxc); tDecoderClear(&dc); if (entry.pBuf) taosMemoryFree(entry.pBuf); return 0; @@ -2144,7 +2344,7 @@ static int metaAddTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTb TAOS_CHECK_RETURN(tdbTbcOpen(pMeta->pCtbIdx, &pCtbIdxc, NULL)); int rc = tdbTbcMoveTo(pCtbIdxc, &(SCtbIdxKey){.suid = suid, .uid = INT64_MIN}, sizeof(SCtbIdxKey), &c); if (rc < 0) { - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } for (;;) { @@ -2181,14 +2381,14 @@ static int metaAddTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTb tdbFree(pKey); tdbFree(pVal); metaDestroyTagIdxKey(pTagIdxKey); - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); goto _err; } (void)tdbTbUpsert(pMeta->pTagIdx, pTagIdxKey, nTagIdxKey, NULL, 0, pMeta->txn); metaDestroyTagIdxKey(pTagIdxKey); pTagIdxKey = NULL; } - (void)tdbTbcClose(pCtbIdxc); + tdbTbcClose(pCtbIdxc); return 0; _err: @@ -2302,7 +2502,7 @@ static int metaDropTagIndex(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterT goto _err; } } - (void)tdbTbcClose(pTagIdxc); + tdbTbcClose(pTagIdxc); metaWLock(pMeta); for (int i = 0; i < taosArrayGetSize(tagIdxList); i++) { diff --git a/source/dnode/vnode/src/sma/smaSnapshot.c b/source/dnode/vnode/src/sma/smaSnapshot.c index bf0b1f0b9d..881c8ac96d 100644 --- a/source/dnode/vnode/src/sma/smaSnapshot.c +++ b/source/dnode/vnode/src/sma/smaSnapshot.c @@ -163,11 +163,11 @@ _exit: TAOS_RETURN(code); } -int32_t rsmaSnapWriterPrepareClose(SRSmaSnapWriter* pWriter) { +int32_t rsmaSnapWriterPrepareClose(SRSmaSnapWriter* pWriter, bool rollback) { int32_t code = 0; for (int32_t i = 0; i < TSDB_RETENTION_L2; ++i) { if (pWriter->pDataWriter[i]) { - code = tsdbSnapWriterPrepareClose(pWriter->pDataWriter[i]); + code = tsdbSnapWriterPrepareClose(pWriter->pDataWriter[i], rollback); if (code) { smaError("vgId:%d, failed to prepare close tsdbSnapWriter since %s. i: %d", SMA_VID(pWriter->pSma), tstrerror(code), i); diff --git a/source/dnode/vnode/src/tq/tqMeta.c b/source/dnode/vnode/src/tq/tqMeta.c index 5ad1680794..460ed89a4b 100644 --- a/source/dnode/vnode/src/tq/tqMeta.c +++ b/source/dnode/vnode/src/tq/tqMeta.c @@ -389,10 +389,7 @@ static int32_t tqMetaTransformInfo(TDB* pMetaDB, TTB* pOld, TTB* pNew) { END: tdbFree(pKey); tdbFree(pVal); - int32_t ret = tdbTbcClose(pCur); - if (ret != 0) { - tqError("failed to close tbc, ret:%d", ret); - } + tdbTbcClose(pCur); return code; } @@ -464,10 +461,7 @@ static int32_t tqMetaRestoreCheckInfo(STQ* pTq) { END: tdbFree(pKey); tdbFree(pVal); - int32_t ret = tdbTbcClose(pCur); - if (ret != 0) { - tqError("failed to close tbc, ret:%d", ret); - } + tdbTbcClose(pCur); tDeleteSTqCheckInfo(&info); return code; } diff --git a/source/dnode/vnode/src/tq/tqSnapshot.c b/source/dnode/vnode/src/tq/tqSnapshot.c index 12b1e5ec19..e06301ef53 100644 --- a/source/dnode/vnode/src/tq/tqSnapshot.c +++ b/source/dnode/vnode/src/tq/tqSnapshot.c @@ -77,10 +77,7 @@ _err: } void tqSnapReaderClose(STqSnapReader** ppReader) { - int32_t ret = tdbTbcClose((*ppReader)->pCur); - if (ret != 0){ - tqError("vgId:%d, vnode snapshot tq reader close failed since %s", TD_VID((*ppReader)->pTq->pVnode), tstrerror(ret)); - } + tdbTbcClose((*ppReader)->pCur); taosMemoryFree(*ppReader); *ppReader = NULL; } diff --git a/source/dnode/vnode/src/tq/tqStreamTaskSnap.c b/source/dnode/vnode/src/tq/tqStreamTaskSnap.c index 420fbf9c56..c49efdb042 100644 --- a/source/dnode/vnode/src/tq/tqStreamTaskSnap.c +++ b/source/dnode/vnode/src/tq/tqStreamTaskSnap.c @@ -97,12 +97,8 @@ int32_t streamTaskSnapReaderClose(SStreamTaskReader* pReader) { int32_t vgId = TD_VID(pReader->pTq->pVnode); taosArrayDestroy(pReader->tdbTbList); - code = tdbTbcClose(pReader->pCur); - if (code) { - tqError("vgId:%d failed to close stream meta reader, code:%s", vgId, tstrerror(code)); - } else { - tqInfo("vgId:%d, vnode stream-task snapshot reader closed", vgId); - } + tdbTbcClose(pReader->pCur); + tqInfo("vgId:%d, vnode stream-task snapshot reader closed", vgId); taosMemoryFree(pReader); return code; @@ -146,7 +142,7 @@ NextTbl: } if (except == 1) { if (pReader->pos + 1 < taosArrayGetSize(pReader->tdbTbList)) { - (void)tdbTbcClose(pReader->pCur); + tdbTbcClose(pReader->pCur); pReader->pos += 1; pPair = taosArrayGet(pReader->tdbTbList, pReader->pos); diff --git a/source/dnode/vnode/src/tsdb/tsdbCacheRead.c b/source/dnode/vnode/src/tsdb/tsdbCacheRead.c index 51109a616b..a7a705fa78 100644 --- a/source/dnode/vnode/src/tsdb/tsdbCacheRead.c +++ b/source/dnode/vnode/src/tsdb/tsdbCacheRead.c @@ -433,7 +433,7 @@ static int32_t tsdbCacheQueryReseek(void* pQHandle) { } int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32_t* slotIds, const int32_t* dstSlotIds, - SArray* pTableUidList) { + SArray* pTableUidList, bool* pGotAll) { if (pReader == NULL || pResBlock == NULL) { return TSDB_CODE_INVALID_PARA; } @@ -642,7 +642,8 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 taosArrayDestroyEx(pLastCols, tsdbCacheFreeSLastColItem); } else if (HASTYPE(pr->type, CACHESCAN_RETRIEVE_TYPE_ALL)) { - for (int32_t i = pr->tableIndex; i < pr->numOfTables; ++i) { + int32_t i = pr->tableIndex; + for (; i < pr->numOfTables; ++i) { tb_uid_t uid = pTableList[i].uid; if ((code = tsdbCacheGetBatch(pr->pTsdb, uid, pRow, pr, ltype)) != 0) { @@ -673,9 +674,13 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 ++pr->tableIndex; if (pResBlock->info.rows >= pResBlock->info.capacity) { - goto _end; + break; } } + + if (pGotAll && i == pr->numOfTables) { + *pGotAll = true; + } } else { code = TSDB_CODE_INVALID_PARA; } diff --git a/source/dnode/vnode/src/tsdb/tsdbMergeTree.c b/source/dnode/vnode/src/tsdb/tsdbMergeTree.c index 3191ddb447..85ed223c6c 100644 --- a/source/dnode/vnode/src/tsdb/tsdbMergeTree.c +++ b/source/dnode/vnode/src/tsdb/tsdbMergeTree.c @@ -1051,7 +1051,9 @@ _end: return code; } -void tMergeTreeAddIter(SMergeTree *pMTree, SLDataIter *pIter) { (void)tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pIter); } +void tMergeTreeAddIter(SMergeTree *pMTree, SLDataIter *pIter) { + SRBTreeNode *node = tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pIter); +} bool tMergeTreeIgnoreEarlierTs(SMergeTree *pMTree) { return pMTree->ignoreEarlierTs; } diff --git a/source/dnode/vnode/src/tsdb/tsdbSnapshot.c b/source/dnode/vnode/src/tsdb/tsdbSnapshot.c index 15930353bf..7d4bcb6914 100644 --- a/source/dnode/vnode/src/tsdb/tsdbSnapshot.c +++ b/source/dnode/vnode/src/tsdb/tsdbSnapshot.c @@ -924,6 +924,31 @@ _exit: return code; } +static int32_t tsdbSnapWriteFileSetAbort(STsdbSnapWriter* writer) { + if (!writer->ctx->fsetWriteBegin) return 0; + + int32_t code = 0; + int32_t lino = 0; + + // close write + code = tsdbSnapWriteFileSetCloseWriter(writer); + TSDB_CHECK_CODE(code, lino, _exit); + + code = tsdbSnapWriteFileSetCloseIter(writer); + TSDB_CHECK_CODE(code, lino, _exit); + + code = tsdbSnapWriteFileSetCloseReader(writer); + TSDB_CHECK_CODE(code, lino, _exit); + + writer->ctx->fsetWriteBegin = false; + +_exit: + if (code) { + TSDB_ERROR_LOG(TD_VID(writer->tsdb->pVnode), lino, code); + } + return code; +} + static int32_t tsdbSnapWriteTimeSeriesData(STsdbSnapWriter* writer, SSnapDataHdr* hdr) { int32_t code = 0; int32_t lino = 0; @@ -1075,15 +1100,23 @@ _exit: return code; } -int32_t tsdbSnapWriterPrepareClose(STsdbSnapWriter* writer) { +int32_t tsdbSnapWriterPrepareClose(STsdbSnapWriter* writer, bool rollback) { int32_t code = 0; int32_t lino = 0; - code = tsdbSnapWriteFileSetEnd(writer); - TSDB_CHECK_CODE(code, lino, _exit); + if (!rollback) { + code = tsdbSnapWriteFileSetEnd(writer); + TSDB_CHECK_CODE(code, lino, _exit); - code = tsdbFSEditBegin(writer->tsdb->pFS, writer->fopArr, TSDB_FEDIT_COMMIT); - TSDB_CHECK_CODE(code, lino, _exit); + code = tsdbFSEditBegin(writer->tsdb->pFS, writer->fopArr, TSDB_FEDIT_COMMIT); + TSDB_CHECK_CODE(code, lino, _exit); + } else { + code = tsdbSnapWriteFileSetAbort(writer); + TSDB_CHECK_CODE(code, lino, _exit); + + code = tsdbFSEditBegin(writer->tsdb->pFS, writer->fopArr, TSDB_FEDIT_COMMIT); + TSDB_CHECK_CODE(code, lino, _exit); + } _exit: if (code) { diff --git a/source/dnode/vnode/src/vnd/vnodeSnapshot.c b/source/dnode/vnode/src/vnd/vnodeSnapshot.c index 202c6dd312..4b7cd38913 100644 --- a/source/dnode/vnode/src/vnd/vnodeSnapshot.c +++ b/source/dnode/vnode/src/vnd/vnodeSnapshot.c @@ -661,7 +661,7 @@ int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback, SSnapshot * // prepare if (pWriter->pTsdbSnapWriter) { - (void)tsdbSnapWriterPrepareClose(pWriter->pTsdbSnapWriter); + (void)tsdbSnapWriterPrepareClose(pWriter->pTsdbSnapWriter, rollback); } if (pWriter->pTsdbSnapRAWWriter) { @@ -669,7 +669,7 @@ int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback, SSnapshot * } if (pWriter->pRsmaSnapWriter) { - (void)rsmaSnapWriterPrepareClose(pWriter->pRsmaSnapWriter); + (void)rsmaSnapWriterPrepareClose(pWriter->pRsmaSnapWriter, rollback); } // commit json diff --git a/source/dnode/vnode/src/vnd/vnodeSvr.c b/source/dnode/vnode/src/vnd/vnodeSvr.c index 2604e2262f..957b328807 100644 --- a/source/dnode/vnode/src/vnd/vnodeSvr.c +++ b/source/dnode/vnode/src/vnd/vnodeSvr.c @@ -1412,6 +1412,10 @@ static int32_t vnodeProcessDropTbReq(SVnode *pVnode, int64_t ver, void *pReq, in if (tsEnableAuditCreateTable) { char *str = taosMemoryCalloc(1, TSDB_TABLE_FNAME_LEN); + if (str == NULL) { + pRsp->code = terrno; + goto _exit; + } strcpy(str, pDropTbReq->name); if (taosArrayPush(tbNames, &str) == NULL) { terrno = TSDB_CODE_OUT_OF_MEMORY; diff --git a/source/libs/executor/src/cachescanoperator.c b/source/libs/executor/src/cachescanoperator.c index b575613efe..7c1ca294e7 100644 --- a/source/libs/executor/src/cachescanoperator.c +++ b/source/libs/executor/src/cachescanoperator.c @@ -48,6 +48,7 @@ typedef struct SCacheRowsScanInfo { SArray* pFuncTypeList; int32_t numOfPks; SColumnInfo pkCol; + bool gotAll; } SCacheRowsScanInfo; static int32_t doScanCacheNext(SOperatorInfo* pOperator, SSDataBlock** ppRes); @@ -292,12 +293,12 @@ static int32_t doScanCacheNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) { T_LONG_JMP(pTaskInfo->env, pTaskInfo->code); } - if (pInfo->indexOfBufferedRes >= pBufRes->info.rows) { + if (pInfo->indexOfBufferedRes >= pBufRes->info.rows && !pInfo->gotAll) { blockDataCleanup(pBufRes); taosArrayClear(pInfo->pUidList); - code = - pReaderFn->retrieveRows(pInfo->pLastrowReader, pBufRes, pInfo->pSlotIds, pInfo->pDstSlotIds, pInfo->pUidList); + code = pReaderFn->retrieveRows(pInfo->pLastrowReader, pBufRes, pInfo->pSlotIds, pInfo->pDstSlotIds, + pInfo->pUidList, &pInfo->gotAll); QUERY_CHECK_CODE(code, lino, _end); // check for tag values @@ -394,7 +395,7 @@ static int32_t doScanCacheNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) { taosArrayClear(pInfo->pUidList); code = pReaderFn->retrieveRows(pInfo->pLastrowReader, pInfo->pRes, pInfo->pSlotIds, pInfo->pDstSlotIds, - pInfo->pUidList); + pInfo->pUidList, NULL); QUERY_CHECK_CODE(code, lino, _end); pInfo->currentGroupIndex += 1; diff --git a/source/libs/executor/src/dynqueryctrloperator.c b/source/libs/executor/src/dynqueryctrloperator.c index bf38a893e3..638536349d 100644 --- a/source/libs/executor/src/dynqueryctrloperator.c +++ b/source/libs/executor/src/dynqueryctrloperator.c @@ -697,7 +697,7 @@ static void freeStbJoinTableList(SStbJoinTableList* pList) { static int32_t appendStbJoinTableList(SStbJoinPrevJoinCtx* pCtx, int64_t rows, int32_t* pLeftVg, int64_t* pLeftUid, int32_t* pRightVg, int64_t* pRightUid) { int32_t code = TSDB_CODE_SUCCESS; - SStbJoinTableList* pNew = taosMemoryMalloc(sizeof(SStbJoinTableList)); + SStbJoinTableList* pNew = taosMemoryCalloc(1, sizeof(SStbJoinTableList)); if (NULL == pNew) { return terrno; } diff --git a/source/libs/executor/src/executil.c b/source/libs/executor/src/executil.c index f3f421628d..5225e8e4cd 100644 --- a/source/libs/executor/src/executil.c +++ b/source/libs/executor/src/executil.c @@ -2835,11 +2835,13 @@ void printDataBlock(SSDataBlock* pBlock, const char* flag, const char* taskIdStr qDebug("%s===stream===%s: Block is Empty. block type %d", taskIdStr, flag, pBlock->info.type); return; } - char* pBuf = NULL; - int32_t code = dumpBlockData(pBlock, flag, &pBuf, taskIdStr); - if (code == 0) { - qDebug("%s", pBuf); - taosMemoryFree(pBuf); + if (qDebugFlag & DEBUG_DEBUG) { + char* pBuf = NULL; + int32_t code = dumpBlockData(pBlock, flag, &pBuf, taskIdStr); + if (code == 0) { + qDebug("%s", pBuf); + taosMemoryFree(pBuf); + } } } diff --git a/source/libs/executor/src/executorInt.c b/source/libs/executor/src/executorInt.c index 1804f0ce26..4fef157984 100644 --- a/source/libs/executor/src/executorInt.c +++ b/source/libs/executor/src/executorInt.c @@ -314,6 +314,8 @@ static int32_t doCreateConstantValColumnInfo(SInputColumnInfoData* pInput, SFunc } } else if (type == TSDB_DATA_TYPE_VARCHAR || type == TSDB_DATA_TYPE_GEOMETRY) { char* tmp = taosMemoryMalloc(pFuncParam->param.nLen + VARSTR_HEADER_SIZE); + QUERY_CHECK_NULL(tmp, code, lino, _end, terrno); + STR_WITH_SIZE_TO_VARSTR(tmp, pFuncParam->param.pz, pFuncParam->param.nLen); for (int32_t i = 0; i < numOfRows; ++i) { code = colDataSetVal(pColInfo, i, tmp, false); diff --git a/source/libs/executor/src/groupcacheoperator.c b/source/libs/executor/src/groupcacheoperator.c index f16f341c5f..17e18710f2 100644 --- a/source/libs/executor/src/groupcacheoperator.c +++ b/source/libs/executor/src/groupcacheoperator.c @@ -1037,16 +1037,13 @@ static int32_t getCacheBlkFromDownstreamOperator(struct SOperatorInfo* pOperator SGroupCacheOperatorInfo* pGCache = pOperator->info; while (continueFetch && TSDB_CODE_SUCCESS == code) { - int32_t code = getBlkFromDownstreamOperator(pOperator, pSession->downstreamIdx, ppRes); - if (TSDB_CODE_SUCCESS != code) { - return code; - } + QRY_ERR_RET(getBlkFromDownstreamOperator(pOperator, pSession->downstreamIdx, ppRes)); if (NULL == *ppRes) { - code = handleDownstreamFetchDone(pOperator, pSession); + QRY_ERR_RET(handleDownstreamFetchDone(pOperator, pSession)); break; } else { - code = handleGroupCacheRetrievedBlk(pOperator, *ppRes, pSession, &continueFetch); + QRY_ERR_RET(handleGroupCacheRetrievedBlk(pOperator, *ppRes, pSession, &continueFetch)); } } @@ -1129,6 +1126,11 @@ static int32_t getBlkFromSessionCacheImpl(struct SOperatorInfo* pOperator, int64 static int32_t groupCacheSessionWait(struct SOperatorInfo* pOperator, SGcDownstreamCtx* pCtx, int64_t sessionId, SGcSessionCtx* pSession, SSDataBlock** ppRes) { + // FOR NOW, IT'S ERROR TO REACH HERE +#if 1 + qError("should not enter session wait"); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; +#else SGroupCacheOperatorInfo* pGCache = pOperator->info; SGroupCacheData* pGroup = pSession->pGroupData; int32_t code = TSDB_CODE_SUCCESS; @@ -1181,6 +1183,7 @@ _return: } return code; +#endif } @@ -1206,6 +1209,10 @@ static int32_t getBlkFromSessionCache(struct SOperatorInfo* pOperator, int64_t s code = getCacheBlkFromDownstreamOperator(pOperator, pCtx, sessionId, pSession, ppRes); goto _return; + } else { + // FOR NOW, SHOULD NOT REACH HERE + qError("Invalid fetchSessionId:%" PRId64 ", currentSessionId:%" PRId64, pCtx->fetchSessionId, sessionId); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; } if (locked) { @@ -1292,6 +1299,9 @@ static int32_t initGroupCacheSession(struct SOperatorInfo* pOperator, SOperatorP qError("fail to get session %" PRId64 " from pSessions", pGcParam->sessionId); QRY_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); } + + qDebug("session:%" PRId64 " initialized, downstreamIdx:%d, vgId:%d, tbUid:%" PRId64 ", needCache:%d", + pGcParam->sessionId, pGcParam->downstreamIdx, pGcParam->vgId, pGcParam->tbUid, pGcParam->needCache); return TSDB_CODE_SUCCESS; } @@ -1323,7 +1333,7 @@ static int32_t getBlkFromGroupCache(struct SOperatorInfo* pOperator, SSDataBlock } } - code = getBlkFromSessionCache(pOperator, pGcParam->sessionId, pSession, ppRes); + QRY_ERR_RET(getBlkFromSessionCache(pOperator, pGcParam->sessionId, pSession, ppRes)); if (NULL == *ppRes) { qDebug("session %" PRId64 " in downstream %d total got %" PRId64 " rows", pGcParam->sessionId, pCtx->id, pSession->resRows); code = taosHashRemove(pCtx->pSessions, &pGcParam->sessionId, sizeof(pGcParam->sessionId)); diff --git a/source/libs/function/src/builtinsimpl.c b/source/libs/function/src/builtinsimpl.c index 5fc257db5b..87e6030898 100644 --- a/source/libs/function/src/builtinsimpl.c +++ b/source/libs/function/src/builtinsimpl.c @@ -4828,6 +4828,9 @@ int32_t histogramFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResul pInfo->normalized = 0; char* binTypeStr = strndup(varDataVal(pCtx->param[1].param.pz), varDataLen(pCtx->param[1].param.pz)); + if (binTypeStr == NULL) { + return TSDB_CODE_OUT_OF_MEMORY; + } int8_t binType = getHistogramBinType(binTypeStr); taosMemoryFree(binTypeStr); @@ -4835,6 +4838,9 @@ int32_t histogramFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResul return TSDB_CODE_FUNC_FUNTION_PARA_VALUE; } char* binDesc = strndup(varDataVal(pCtx->param[2].param.pz), varDataLen(pCtx->param[2].param.pz)); + if (binDesc == NULL) { + return TSDB_CODE_OUT_OF_MEMORY; + } int64_t normalized = pCtx->param[3].param.i; if (normalized != 0 && normalized != 1) { taosMemoryFree(binDesc); diff --git a/source/libs/function/src/thistogram.c b/source/libs/function/src/thistogram.c index 6dfad36542..2283845cfb 100644 --- a/source/libs/function/src/thistogram.c +++ b/source/libs/function/src/thistogram.c @@ -43,6 +43,10 @@ int32_t tHistogramCreate(int32_t numOfEntries, SHistogramInfo** pHisto) { #if !defined(USE_ARRAYLIST) pHisto->pList = SSkipListCreate(MAX_SKIP_LIST_LEVEL, TSDB_DATA_TYPE_DOUBLE, sizeof(double)); SInsertSupporter* pss = taosMemoryMalloc(sizeof(SInsertSupporter)); + if (NULL == pss) { + taosMemoryFree(*pHisto); + return terrno; + } pss->numOfEntries = pHisto->maxEntries; pss->pSkipList = pHisto->pList; @@ -119,6 +123,9 @@ int32_t tHistogramAdd(SHistogramInfo** pHisto, double val) { #else tSkipListKey key = tSkipListCreateKey(TSDB_DATA_TYPE_DOUBLE, &val, tDataTypes[TSDB_DATA_TYPE_DOUBLE].nSize); SHistBin* entry = taosMemoryCalloc(1, sizeof(SHistBin)); + if (entry == NULL) { + return terrno; + } entry->val = val; tSkipListNode* pResNode = SSkipListPut((*pHisto)->pList, entry, &key, 0); diff --git a/source/libs/parser/src/parInsertStmt.c b/source/libs/parser/src/parInsertStmt.c index 9f3d5b5a81..7d68e80df9 100644 --- a/source/libs/parser/src/parInsertStmt.c +++ b/source/libs/parser/src/parInsertStmt.c @@ -155,6 +155,10 @@ int32_t qBindStmtTagsValue(void* pBlock, void* boundTags, int64_t suid, const ch SSchema* pTagSchema = &pSchema[tags->pColIndex[c]]; int32_t colLen = pTagSchema->bytes; if (IS_VAR_DATA_TYPE(pTagSchema->type)) { + if (!bind[c].length) { + code = buildInvalidOperationMsg(&pBuf, "var tag length is null"); + goto end; + } colLen = bind[c].length[0]; if ((colLen + VARSTR_HEADER_SIZE) > pTagSchema->bytes) { code = buildInvalidOperationMsg(&pBuf, "tag length is too big"); @@ -173,6 +177,10 @@ int32_t qBindStmtTagsValue(void* pBlock, void* boundTags, int64_t suid, const ch isJson = true; char* tmp = taosMemoryCalloc(1, colLen + 1); + if (!tmp) { + code = TSDB_CODE_OUT_OF_MEMORY; + goto end; + } memcpy(tmp, bind[c].buffer, colLen); code = parseJsontoTagData(tmp, pTagArray, &pTag, &pBuf); taosMemoryFree(tmp); @@ -495,6 +503,10 @@ int32_t qBindStmtTagsValue2(void* pBlock, void* boundTags, int64_t suid, const c SSchema* pTagSchema = &pSchema[tags->pColIndex[c]]; int32_t colLen = pTagSchema->bytes; if (IS_VAR_DATA_TYPE(pTagSchema->type)) { + if (!bind[c].length) { + code = buildInvalidOperationMsg(&pBuf, "var tag length is null"); + goto end; + } colLen = bind[c].length[0]; if ((colLen + VARSTR_HEADER_SIZE) > pTagSchema->bytes) { code = buildInvalidOperationMsg(&pBuf, "tag length is too big"); @@ -513,6 +525,10 @@ int32_t qBindStmtTagsValue2(void* pBlock, void* boundTags, int64_t suid, const c isJson = true; char* tmp = taosMemoryCalloc(1, colLen + 1); + if (!tmp) { + code = TSDB_CODE_OUT_OF_MEMORY; + goto end; + } memcpy(tmp, bind[c].buffer, colLen); code = parseJsontoTagData(tmp, pTagArray, &pTag, &pBuf); taosMemoryFree(tmp); diff --git a/source/libs/scalar/src/sclfunc.c b/source/libs/scalar/src/sclfunc.c index 3b9cad68ad..965e029c8f 100644 --- a/source/libs/scalar/src/sclfunc.c +++ b/source/libs/scalar/src/sclfunc.c @@ -4353,10 +4353,16 @@ int32_t histogramScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarP int32_t totalCount = 0; char *binTypeStr = strndup(varDataVal(pInput[1].columnData->pData), varDataLen(pInput[1].columnData->pData)); + if (NULL == binTypeStr) { + SCL_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); + } int8_t binType = getHistogramBinType(binTypeStr); taosMemoryFree(binTypeStr); char *binDesc = strndup(varDataVal(pInput[2].columnData->pData), varDataLen(pInput[2].columnData->pData)); + if (NULL == binDesc) { + SCL_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); + } int64_t normalized = *(int64_t *)(pInput[3].columnData->pData); int32_t type = GET_PARAM_TYPE(pInput); diff --git a/source/libs/scalar/src/sclvector.c b/source/libs/scalar/src/sclvector.c index d8cc620b1e..2bfad71ddb 100644 --- a/source/libs/scalar/src/sclvector.c +++ b/source/libs/scalar/src/sclvector.c @@ -596,6 +596,11 @@ int32_t ncharTobinary(void *buf, void **out) { // todo need to remove , if tobi int32_t inputLen = varDataTLen(buf); *out = taosMemoryCalloc(1, inputLen); + if (NULL == *out) { + sclError("charset:%s to %s. val:%s convert ncharTobinary failed, since memory alloc failed.", + DEFAULT_UNICODE_ENCODEC, tsCharset, (char *)varDataVal(buf)); + SCL_ERR_RET(terrno); + } int32_t len = taosUcs4ToMbs((TdUcs4 *)varDataVal(buf), varDataLen(buf), varDataVal(*out)); if (len < 0) { sclError("charset:%s to %s. val:%s convert ncharTobinary failed.", DEFAULT_UNICODE_ENCODEC, tsCharset, diff --git a/source/libs/stream/src/streamMeta.c b/source/libs/stream/src/streamMeta.c index f4202667ff..67fc5af013 100644 --- a/source/libs/stream/src/streamMeta.c +++ b/source/libs/stream/src/streamMeta.c @@ -191,11 +191,7 @@ int32_t streamMetaCheckBackendCompatible(SStreamMeta* pMeta) { if (code) { stError("vgId:%d failed to open stream meta file cursor, not perform compatible check, code:%s", pMeta->vgId, tstrerror(code)); - code = tdbTbcClose(pCur); - if (code) { - stError("vgId:%d failed to close meta file cursor, code:%s", pMeta->vgId, tstrerror(code)); - } - + tdbTbcClose(pCur); return ret; } @@ -223,10 +219,7 @@ int32_t streamMetaCheckBackendCompatible(SStreamMeta* pMeta) { tdbFree(pKey); tdbFree(pVal); - code = tdbTbcClose(pCur); - if (code != 0) { - stError("vgId:%d failed to close meta file cursor, code:%s", pMeta->vgId, tstrerror(code)); - } + tdbTbcClose(pCur); return ret; } @@ -977,10 +970,7 @@ int64_t streamMetaGetLatestCheckpointId(SStreamMeta* pMeta) { code = tdbTbcMoveToFirst(pCur); if (code) { stError("failed to move stream meta file cursor, the latest checkpointId is 0, vgId:%d", pMeta->vgId); - int32_t ret = tdbTbcClose(pCur); - if (ret != 0) { - stError("vgId:%d failed to close meta file cursor, code:%s", pMeta->vgId, tstrerror(ret)); - } + tdbTbcClose(pCur); return checkpointId; } @@ -1003,11 +993,7 @@ int64_t streamMetaGetLatestCheckpointId(SStreamMeta* pMeta) { tdbFree(pKey); tdbFree(pVal); - int32_t ret = tdbTbcClose(pCur); - if (ret != 0) { - stError("vgId:%d failed to close meta file cursor, code:%s", pMeta->vgId, tstrerror(ret)); - } - + tdbTbcClose(pCur); return checkpointId; } @@ -1047,10 +1033,7 @@ void streamMetaLoadAllTasks(SStreamMeta* pMeta) { if (code) { stError("vgId:%d failed to open stream meta cursor, code:%s, not load any stream tasks", vgId, tstrerror(terrno)); taosArrayDestroy(pRecycleList); - int32_t ret = tdbTbcClose(pCur); - if (ret != 0) { - stError("vgId:%d failed to close meta file cursor, code:%s", pMeta->vgId, tstrerror(ret)); - } + tdbTbcClose(pCur); return; } @@ -1137,9 +1120,7 @@ void streamMetaLoadAllTasks(SStreamMeta* pMeta) { tdbFree(pKey); tdbFree(pVal); - if (tdbTbcClose(pCur) < 0) { - stError("vgId:%d failed to close meta-file cursor, code:%s, continue", vgId, tstrerror(terrno)); - } + tdbTbcClose(pCur); if (taosArrayGetSize(pRecycleList) > 0) { for (int32_t i = 0; i < taosArrayGetSize(pRecycleList); ++i) { diff --git a/source/libs/stream/src/streamSessionState.c b/source/libs/stream/src/streamSessionState.c index a556c7c6b4..b2907037e6 100644 --- a/source/libs/stream/src/streamSessionState.c +++ b/source/libs/stream/src/streamSessionState.c @@ -79,6 +79,9 @@ bool inSessionWindow(SSessionKey* pKey, TSKEY ts, int64_t gap) { SStreamStateCur* createStateCursor(SStreamFileState* pFileState) { SStreamStateCur* pCur = createStreamStateCursor(); + if (pCur == NULL) { + return NULL; + } pCur->pStreamFileState = pFileState; return pCur; } @@ -531,6 +534,9 @@ static SStreamStateCur* seekKeyCurrentPrev_buff(SStreamFileState* pFileState, co if (index >= 0) { pCur = createStateCursor(pFileState); + if (pCur == NULL) { + return NULL; + } pCur->buffIndex = index; if (pIndex) { *pIndex = index; @@ -632,6 +638,9 @@ SStreamStateCur* countWinStateSeekKeyPrev(SStreamFileState* pFileState, const SS pBuffCur->buffIndex = 0; } else if (taosArrayGetSize(pWinStates) > 0) { pBuffCur = createStateCursor(pFileState); + if (pBuffCur == NULL) { + return NULL; + } pBuffCur->buffIndex = 0; } diff --git a/source/libs/stream/src/streamState.c b/source/libs/stream/src/streamState.c index 19db90bc69..ae19c8b42f 100644 --- a/source/libs/stream/src/streamState.c +++ b/source/libs/stream/src/streamState.c @@ -536,6 +536,9 @@ void streamStateCopyBackend(SStreamState* src, SStreamState* dst) { } SStreamStateCur* createStreamStateCursor() { SStreamStateCur* pCur = taosMemoryCalloc(1, sizeof(SStreamStateCur)); + if (pCur == NULL) { + return NULL; + } pCur->buffIndex = -1; return pCur; } diff --git a/source/libs/stream/src/streamUpdate.c b/source/libs/stream/src/streamUpdate.c index 155af0f2bf..f93559401f 100644 --- a/source/libs/stream/src/streamUpdate.c +++ b/source/libs/stream/src/streamUpdate.c @@ -630,7 +630,11 @@ int32_t updateInfoDeserialize(void* buf, int32_t bufLen, SUpdateInfo* pInfo) { if (tDecodeI8(&decoder, &pInfo->pkColType) < 0) return -1; pInfo->pKeyBuff = taosMemoryCalloc(1, sizeof(TSKEY) + sizeof(int64_t) + pInfo->pkColLen); + QUERY_CHECK_NULL(pInfo->pKeyBuff, code, lino, _error, terrno); + pInfo->pValueBuff = taosMemoryCalloc(1, sizeof(TSKEY) + pInfo->pkColLen); + QUERY_CHECK_NULL(pInfo->pValueBuff, code, lino, _error, terrno); + if (pInfo->pkColLen != 0) { pInfo->comparePkRowFn = compareKeyTsAndPk; pInfo->comparePkCol = getKeyComparFunc(pInfo->pkColType, TSDB_ORDER_ASC); diff --git a/source/libs/stream/src/tstreamFileState.c b/source/libs/stream/src/tstreamFileState.c index 923aaa945b..855a42a5f8 100644 --- a/source/libs/stream/src/tstreamFileState.c +++ b/source/libs/stream/src/tstreamFileState.c @@ -101,6 +101,10 @@ int32_t intervalFileGetFn(SStreamFileState* pFileState, void* pKey, void** data, void* intervalCreateStateKey(SRowBuffPos* pPos, int64_t num) { SStateKey* pStateKey = taosMemoryCalloc(1, sizeof(SStateKey)); + if (pStateKey == NULL) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno)); + return NULL; + } SWinKey* pWinKey = pPos->pKey; pStateKey->key = *pWinKey; pStateKey->opNum = num; @@ -117,6 +121,10 @@ int32_t sessionFileGetFn(SStreamFileState* pFileState, void* pKey, void** data, void* sessionCreateStateKey(SRowBuffPos* pPos, int64_t num) { SStateSessionKey* pStateKey = taosMemoryCalloc(1, sizeof(SStateSessionKey)); + if (pStateKey == NULL) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno)); + return NULL; + } SSessionKey* pWinKey = pPos->pKey; pStateKey->key = *pWinKey; pStateKey->opNum = num; @@ -125,11 +133,16 @@ void* sessionCreateStateKey(SRowBuffPos* pPos, int64_t num) { static void streamFileStateDecode(TSKEY* pKey, void* pBuff, int32_t len) { pBuff = taosDecodeFixedI64(pBuff, pKey); } -static void streamFileStateEncode(TSKEY* pKey, void** pVal, int32_t* pLen) { +static int32_t streamFileStateEncode(TSKEY* pKey, void** pVal, int32_t* pLen) { *pLen = sizeof(TSKEY); (*pVal) = taosMemoryCalloc(1, *pLen); + if ((*pVal) == NULL) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno)); + return terrno; + } void* buff = *pVal; int32_t tmp = taosEncodeFixedI64(&buff, *pKey); + return TSDB_CODE_SUCCESS; } int32_t streamFileStateInit(int64_t memSize, uint32_t keySize, uint32_t rowSize, uint32_t selectRowSize, GetTsFun fp, @@ -518,11 +531,10 @@ SRowBuffPos* getNewRowPos(SStreamFileState* pFileState) { if (pFileState->curRowCount < pFileState->maxRowCount) { pBuff = taosMemoryCalloc(1, pFileState->rowSize); - if (pBuff) { - pPos->pRowBuff = pBuff; - pFileState->curRowCount++; - goto _end; - } + QUERY_CHECK_NULL(pBuff, code, lino, _error, terrno); + pPos->pRowBuff = pBuff; + pFileState->curRowCount++; + goto _end; } code = clearRowBuff(pFileState); @@ -770,6 +782,8 @@ void flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot, boo } void* pSKey = pFileState->stateBuffCreateStateKeyFn(pPos, ((SStreamState*)pFileState->pFileStore)->number); + QUERY_CHECK_NULL(pSKey, code, lino, _end, terrno); + code = streamStatePutBatchOptimize(pFileState->pFileStore, idx, batch, pSKey, pPos->pRowBuff, pFileState->rowSize, 0, buf); taosMemoryFreeClear(pSKey); @@ -798,7 +812,9 @@ void flushSnapshot(SStreamFileState* pFileState, SStreamSnapshot* pSnapshot, boo if (flushState) { void* valBuf = NULL; int32_t len = 0; - streamFileStateEncode(&pFileState->flushMark, &valBuf, &len); + code = streamFileStateEncode(&pFileState->flushMark, &valBuf, &len); + QUERY_CHECK_CODE(code, lino, _end); + qDebug("===stream===flushMark write:%" PRId64, pFileState->flushMark); code = streamStatePutBatch(pFileState->pFileStore, "default", batch, STREAM_STATE_INFO_NAME, valBuf, len, 0); taosMemoryFree(valBuf); diff --git a/source/libs/sync/src/syncMain.c b/source/libs/sync/src/syncMain.c index 8081de60c9..4dce54fc1a 100644 --- a/source/libs/sync/src/syncMain.c +++ b/source/libs/sync/src/syncMain.c @@ -146,6 +146,10 @@ void syncStop(int64_t rid) { void syncPreStop(int64_t rid) { SSyncNode* pSyncNode = syncNodeAcquire(rid); if (pSyncNode != NULL) { + if (snapshotReceiverIsStart(pSyncNode->pNewNodeReceiver)) { + sInfo("vgId:%d, stop snapshot receiver", pSyncNode->vgId); + snapshotReceiverStop(pSyncNode->pNewNodeReceiver); + } syncNodePreClose(pSyncNode); syncNodeRelease(pSyncNode); } diff --git a/source/libs/tdb/inc/tdb.h b/source/libs/tdb/inc/tdb.h index 0935e87ab0..374c3e0dc3 100644 --- a/source/libs/tdb/inc/tdb.h +++ b/source/libs/tdb/inc/tdb.h @@ -60,7 +60,7 @@ int32_t tdbTbTraversal(TTB *pTb, void *data, // TBC int32_t tdbTbcOpen(TTB *pTb, TBC **ppTbc, TXN *pTxn); -int32_t tdbTbcClose(TBC *pTbc); +void tdbTbcClose(TBC *pTbc); int32_t tdbTbcIsValid(TBC *pTbc); int32_t tdbTbcMoveTo(TBC *pTbc, const void *pKey, int kLen, int *c); int32_t tdbTbcMoveToFirst(TBC *pTbc); diff --git a/source/libs/tdb/src/db/tdbBtree.c b/source/libs/tdb/src/db/tdbBtree.c index 93c214545f..7c342c0527 100644 --- a/source/libs/tdb/src/db/tdbBtree.c +++ b/source/libs/tdb/src/db/tdbBtree.c @@ -172,12 +172,11 @@ int tdbBtreeOpen(int keyLen, int valLen, SPager *pPager, char const *tbname, SPg return 0; } -int tdbBtreeClose(SBTree *pBt) { +void tdbBtreeClose(SBTree *pBt) { if (pBt) { tdbFree(pBt->pBuf); tdbOsFree(pBt); } - return 0; } int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, int vLen, TXN *pTxn) { @@ -201,7 +200,7 @@ int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, in // move to the position to insert ret = tdbBtcMoveTo(&btc, pKey, kLen, &c); if (ret < 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-insert: btc move to failed with ret: %d.", ret); return ret; } @@ -213,7 +212,7 @@ int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, in btc.idx++; } else if (c == 0) { // dup key not allowed with insert - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-insert: dup key. pKey: %p, kLen: %d, btc: %p, pTxn: %p", pKey, kLen, &btc, pTxn); return TSDB_CODE_DUP_KEY; } @@ -221,12 +220,12 @@ int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, in ret = tdbBtcUpsert(&btc, pKey, kLen, pVal, vLen, 1); if (ret < 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-insert: btc upsert failed with ret: %d.", ret); return ret; } - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); return 0; } @@ -250,20 +249,20 @@ int tdbBtreeDelete(SBTree *pBt, const void *pKey, int kLen, TXN *pTxn) { // move the cursor ret = tdbBtcMoveTo(&btc, pKey, kLen, &c); if (ret < 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-delete: btc move to failed with ret: %d.", ret); return ret; } if (btc.idx < 0 || c != 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); return TSDB_CODE_NOT_FOUND; } // delete the key ret = tdbBtcDelete(&btc); if (ret < 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); return ret; } /* @@ -278,7 +277,7 @@ int tdbBtreeDelete(SBTree *pBt, const void *pKey, int kLen, TXN *pTxn) { btc.coder.ofps = NULL; } */ - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); return 0; } @@ -351,20 +350,20 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL ret = tdbBtcMoveTo(&btc, pKey, kLen, &cret); if (ret < 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-pget: btc move to failed with ret: %d.", ret); return ret; } if (btc.idx < 0 || cret) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); return TSDB_CODE_NOT_FOUND; } pCell = tdbPageGetCell(btc.pPage, btc.idx); ret = tdbBtreeDecodeCell(btc.pPage, pCell, &cd, btc.pTxn, pBt); if (ret < 0) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-pget: decode cell failed with ret: %d.", ret); return ret; } @@ -372,7 +371,7 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL if (ppKey) { pTKey = tdbRealloc(*ppKey, cd.kLen); if (pTKey == NULL) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-pget: realloc pTKey failed."); return terrno; } @@ -384,7 +383,7 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL if (ppVal) { pTVal = tdbRealloc(*ppVal, cd.vLen); if (pTVal == NULL) { - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); tdbError("tdb/btree-pget: realloc pTVal failed."); return terrno; } @@ -405,7 +404,7 @@ int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkL tdbTrace("tdb pget end, btc decoder: %p/0x%x, local decoder:%p", &btc.coder, btc.coder.freeKV, &cd); - (void)tdbBtcClose(&btc); + tdbBtcClose(&btc); return 0; } @@ -597,14 +596,20 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx if (sIdx + i < TDB_PAGE_TOTAL_CELLS(pParent)) { pCell = tdbPageGetCell(pParent, sIdx + i); szDivCell[i] = tdbBtreeCellSize(pParent, pCell, 0, NULL, NULL); - pDivCell[i] = tdbOsMalloc(szDivCell[i]); + if ((pDivCell[i] = tdbOsMalloc(szDivCell[i])) == NULL) { + return terrno; + } memcpy(pDivCell[i], pCell, szDivCell[i]); } if (i < nOlds - 1) { ((SPgno *)pDivCell[i])[0] = ((SIntHdr *)pOlds[i]->pData)->pgno; ((SIntHdr *)pOlds[i]->pData)->pgno = 0; - (void)tdbPageInsertCell(pOlds[i], TDB_PAGE_TOTAL_CELLS(pOlds[i]), pDivCell[i], szDivCell[i], 1); + ret = tdbPageInsertCell(pOlds[i], TDB_PAGE_TOTAL_CELLS(pOlds[i]), pDivCell[i], szDivCell[i], 1); + if (ret < 0) { + tdbError("tdb/btree-balance: insert cell failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } } } rPgno = ((SIntHdr *)pOlds[nOlds - 1]->pData)->pgno; @@ -791,15 +796,31 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx iarg.pBt = pBt; iarg.flags = TDB_BTREE_PAGE_GET_FLAGS(pOlds[0]); for (int i = 0; i < nOlds; i++) { - (void)tdbPageCreate(pOlds[0]->pageSize, &pOldsCopy[i], tdbDefaultMalloc, NULL); - (void)tdbBtreeInitPage(pOldsCopy[i], &iarg, 0); - (void)tdbPageCopy(pOlds[i], pOldsCopy[i], 0); + ret = tdbPageCreate(pOlds[0]->pageSize, &pOldsCopy[i], tdbDefaultMalloc, NULL); + if (ret < 0) { + tdbError("tdb/btree-balance: create page failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } + ret = tdbBtreeInitPage(pOldsCopy[i], &iarg, 0); + if (ret < 0) { + tdbError("tdb/btree-balance: init page failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } + ret = tdbPageCopy(pOlds[i], pOldsCopy[i], 0); + if (ret < 0) { + tdbError("tdb/btree-balance: copy page failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } pOlds[i]->nOverflow = 0; } iNew = 0; nNewCells = 0; - (void)tdbBtreeInitPage(pNews[iNew], &iarg, 0); + ret = tdbBtreeInitPage(pNews[iNew], &iarg, 0); + if (ret < 0) { + tdbError("tdb/btree-balance: init page failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } for (int iOld = 0; iOld < nOlds; iOld++) { SPage *pPage; @@ -818,7 +839,11 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx } if (nNewCells < infoNews[iNew].cnt) { - (void)tdbPageInsertCell(pNews[iNew], nNewCells, pCell, szCell, 0); + ret = tdbPageInsertCell(pNews[iNew], nNewCells, pCell, szCell, 0); + if (ret < 0) { + tdbError("tdb/btree-balance: insert cell failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } nNewCells++; // insert parent page @@ -832,12 +857,19 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx // TODO: pCell here may be inserted as an overflow cell, handle it SCell *pNewCell = tdbOsMalloc(cd.kLen + 9); - int szNewCell; - SPgno pgno; + if (pNewCell == NULL) { + return terrno; + } + int szNewCell; + SPgno pgno; pgno = TDB_PAGE_PGNO(pNews[iNew]); (void)tdbBtreeEncodeCell(pParent, cd.pKey, cd.kLen, (void *)&pgno, sizeof(SPgno), pNewCell, &szNewCell, pTxn, pBt); - (void)tdbPageInsertCell(pParent, sIdx++, pNewCell, szNewCell, 0); + ret = tdbPageInsertCell(pParent, sIdx++, pNewCell, szNewCell, 0); + if (ret) { + tdbError("tdb/btree-balance: insert cell failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } tdbOsFree(pNewCell); if (TDB_CELLDECODER_FREE_VAL(&cd)) { @@ -850,7 +882,11 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx iNew++; nNewCells = 0; if (iNew < nNews) { - (void)tdbBtreeInitPage(pNews[iNew], &iarg, 0); + ret = tdbBtreeInitPage(pNews[iNew], &iarg, 0); + if (ret < 0) { + tdbError("tdb/btree-balance: init page failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } } } } else { @@ -869,13 +905,21 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx return TSDB_CODE_FAILED; } ((SPgno *)pCell)[0] = TDB_PAGE_PGNO(pNews[iNew]); - (void)tdbPageInsertCell(pParent, sIdx++, pCell, szCell, 0); + ret = tdbPageInsertCell(pParent, sIdx++, pCell, szCell, 0); + if (ret) { + tdbError("tdb/btree-balance: insert cell failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } // move to next new page iNew++; nNewCells = 0; if (iNew < nNews) { - (void)tdbBtreeInitPage(pNews[iNew], &iarg, 0); + ret = tdbBtreeInitPage(pNews[iNew], &iarg, 0); + if (ret < 0) { + tdbError("tdb/btree-balance: init page failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } } } } @@ -892,26 +936,39 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx pIntHdr->pgno = TDB_PAGE_PGNO(pNews[nNews - 1]); } else { ((SPgno *)pDivCell[nOlds - 1])[0] = TDB_PAGE_PGNO(pNews[nNews - 1]); - (void)tdbPageInsertCell(pParent, sIdx, pDivCell[nOlds - 1], szDivCell[nOlds - 1], 0); + ret = tdbPageInsertCell(pParent, sIdx, pDivCell[nOlds - 1], szDivCell[nOlds - 1], 0); + if (ret) { + tdbError("tdb/btree-balance: insert cell failed with ret: %d.", ret); + return TSDB_CODE_FAILED; + } } } for (int i = 0; i < nOlds; i++) { - (void)tdbPageDestroy(pOldsCopy[i], tdbDefaultFree, NULL); + tdbPageDestroy(pOldsCopy[i], tdbDefaultFree, NULL); } } if (TDB_BTREE_PAGE_IS_ROOT(pParent) && TDB_PAGE_TOTAL_CELLS(pParent) == 0) { i8 flags = TDB_BTREE_ROOT | TDB_BTREE_PAGE_IS_LEAF(pNews[0]); // copy content to the parent page - (void)tdbBtreeInitPage(pParent, &(SBtreeInitPageArg){.flags = flags, .pBt = pBt}, 0); - (void)tdbPageCopy(pNews[0], pParent, 1); + ret = tdbBtreeInitPage(pParent, &(SBtreeInitPageArg){.flags = flags, .pBt = pBt}, 0); + if (ret < 0) { + return ret; + } + ret = tdbPageCopy(pNews[0], pParent, 1); + if (ret < 0) { + return ret; + } if (!TDB_BTREE_PAGE_IS_LEAF(pNews[0])) { ((SIntHdr *)(pParent->pData))->pgno = ((SIntHdr *)(pNews[0]->pData))->pgno; } - (void)tdbPagerInsertFreePage(pBt->pPager, pNews[0], pTxn); + ret = tdbPagerInsertFreePage(pBt->pPager, pNews[0], pTxn); + if (ret < 0) { + return ret; + } } for (int i = 0; i < 3; i++) { @@ -1986,7 +2043,11 @@ int tdbBtcMoveToNext(SBTC *pBtc) { return 0; } - (void)tdbBtcMoveUpward(pBtc); + ret = tdbBtcMoveUpward(pBtc); + if (ret < 0) { + tdbError("tdb/btc-move-to-next: btc move upward failed with ret: %d.", ret); + return ret; + } pBtc->idx++; if (TDB_BTREE_PAGE_IS_LEAF(pBtc->pPage)) { @@ -2015,6 +2076,7 @@ int tdbBtcMoveToNext(SBTC *pBtc) { } int tdbBtcMoveToPrev(SBTC *pBtc) { + int32_t ret = 0; if (pBtc->idx < 0) return TSDB_CODE_FAILED; pBtc->idx--; @@ -2029,7 +2091,11 @@ int tdbBtcMoveToPrev(SBTC *pBtc) { return 0; } - (void)tdbBtcMoveUpward(pBtc); + ret = tdbBtcMoveUpward(pBtc); + if (ret < 0) { + tdbError("tdb/btc-move-to-prev: btc move upward failed with ret: %d.", ret); + return ret; + } pBtc->idx--; if (pBtc->idx >= 0) { break; @@ -2040,7 +2106,11 @@ int tdbBtcMoveToPrev(SBTC *pBtc) { for (;;) { if (TDB_BTREE_PAGE_IS_LEAF(pBtc->pPage)) break; - (void)tdbBtcMoveDownward(pBtc); + ret = tdbBtcMoveDownward(pBtc); + if (ret < 0) { + tdbError("tdb/btc-move-to-prev: btc move downward failed with ret: %d.", ret); + return ret; + } if (TDB_BTREE_PAGE_IS_LEAF(pBtc->pPage)) { pBtc->idx = TDB_PAGE_TOTAL_CELLS(pBtc->pPage) - 1; } else { @@ -2162,6 +2232,9 @@ int tdbBtcDelete(SBTC *pBtc) { bool destroyOfps = false; if (!pBtc->pPage->pPager->ofps) { pBtc->pPage->pPager->ofps = taosArrayInit(8, sizeof(SPage *)); + if (pBtc->pPage->pPager->ofps == NULL) { + return terrno; + } destroyOfps = true; } @@ -2184,7 +2257,11 @@ int tdbBtcDelete(SBTC *pBtc) { if (idx == nCells - 1) { if (idx) { pBtc->idx--; - (void)tdbBtcGet(pBtc, &pKey, &nKey, NULL, NULL); + ret = tdbBtcGet(pBtc, &pKey, &nKey, NULL, NULL); + if (ret) { + tdbError("tdb/btc-delete: btc get failed with ret: %d.", ret); + return ret; + } // loop to update the interial page pgno = TDB_PAGE_PGNO(pBtc->pPage); @@ -2201,7 +2278,10 @@ int tdbBtcDelete(SBTC *pBtc) { } // update the cell with new key - pCell = tdbOsMalloc(nKey + 9); + if ((pCell = tdbOsMalloc(nKey + 9)) == NULL) { + tdbError("tdb/btc-delete: malloc failed."); + return terrno; + } (void)tdbBtreeEncodeCell(pPage, pKey, nKey, &pgno, sizeof(pgno), pCell, &szCell, pBtc->pTxn, pBtc->pBt); ret = tdbPageUpdateCell(pPage, idx, pCell, szCell, pBtc->pTxn, pBtc->pBt); @@ -2432,7 +2512,11 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { // compare first cell pBtc->idx = lidx; - (void)tdbBtcGet(pBtc, &pTKey, &tkLen, NULL, NULL); + ret = tdbBtcGet(pBtc, &pTKey, &tkLen, NULL, NULL); + if (ret < 0) { + tdbError("tdb/btc-move-to: btc get failed with ret: %d.", ret); + return ret; + } c = pBt->kcmpr(pKey, kLen, pTKey, tkLen); if (c <= 0) { ridx = lidx - 1; @@ -2442,7 +2526,11 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { // compare last cell if (lidx <= ridx) { pBtc->idx = ridx; - (void)tdbBtcGet(pBtc, &pTKey, &tkLen, NULL, NULL); + ret = tdbBtcGet(pBtc, &pTKey, &tkLen, NULL, NULL); + if (ret < 0) { + tdbError("tdb/btc-move-to: btc get failed with ret: %d.", ret); + return ret; + } c = pBt->kcmpr(pKey, kLen, pTKey, tkLen); if (c >= 0) { lidx = ridx + 1; @@ -2457,7 +2545,11 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { if (lidx > ridx) break; pBtc->idx = (lidx + ridx) >> 1; - (void)tdbBtcGet(pBtc, &pTKey, &tkLen, NULL, NULL); + ret = tdbBtcGet(pBtc, &pTKey, &tkLen, NULL, NULL); + if (ret < 0) { + tdbError("tdb/btc-move-to: btc get failed with ret: %d.", ret); + return ret; + } c = pBt->kcmpr(pKey, kLen, pTKey, tkLen); if (c < 0) { // pKey < cd.pKey @@ -2491,18 +2583,18 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { return 0; } -int tdbBtcClose(SBTC *pBtc) { +void tdbBtcClose(SBTC *pBtc) { if (pBtc->iPage < 0) { if (pBtc->freeTxn) { tdbTxnClose(pBtc->pTxn); } - return 0; + return; } for (;;) { if (NULL == pBtc->pPage) { tdbError("tdb/btc-close: null ptr pPage."); - return TSDB_CODE_FAILED; + return; } tdbPagerReturnPage(pBtc->pBt->pPager, pBtc->pPage, pBtc->pTxn); @@ -2528,7 +2620,7 @@ int tdbBtcClose(SBTC *pBtc) { tdbTxnClose(pBtc->pTxn); } - return 0; + return; } int tdbBtcIsValid(SBTC *pBtc) { diff --git a/source/libs/tdb/src/db/tdbPCache.c b/source/libs/tdb/src/db/tdbPCache.c index c1a2943bee..53c1284faa 100644 --- a/source/libs/tdb/src/db/tdbPCache.c +++ b/source/libs/tdb/src/db/tdbPCache.c @@ -149,7 +149,7 @@ static int tdbPCacheAlterImpl(SPCache *pCache, int32_t nPage) { SPage *pPage = *ppPage; *ppPage = pPage->pFreeNext; pCache->aPage[pPage->id] = NULL; - (void)tdbPageDestroy(pPage, tdbDefaultFree, NULL); + tdbPageDestroy(pPage, tdbDefaultFree, NULL); pCache->nFree--; } else { ppPage = &(*ppPage)->pFreeNext; @@ -209,7 +209,7 @@ static void tdbPCacheFreePage(SPCache *pCache, SPage *pPage) { tdbTrace("pcache/free2 page: %p/%d, pgno:%d, ", pPage, pPage->id, TDB_PAGE_PGNO(pPage)); tdbPCacheRemovePageFromHash(pCache, pPage); - (void)tdbPageDestroy(pPage, tdbDefaultFree, NULL); + tdbPageDestroy(pPage, tdbDefaultFree, NULL); } } @@ -268,7 +268,7 @@ void tdbPCacheRelease(SPCache *pCache, SPage *pPage, TXN *pTxn) { tdbPCacheRemovePageFromHash(pCache, pPage); } - (void)tdbPageDestroy(pPage, pTxn->xFree, pTxn->xArg); + tdbPageDestroy(pPage, pTxn->xFree, pTxn->xArg); } // } } @@ -432,7 +432,7 @@ static void tdbPCacheUnpinPage(SPCache *pCache, SPage *pPage) { tdbTrace("pcache destroy page: %p/%d/%d", pPage, TDB_PAGE_PGNO(pPage), pPage->id); tdbPCacheRemovePageFromHash(pCache, pPage); - (void)tdbPageDestroy(pPage, tdbDefaultFree, NULL); + tdbPageDestroy(pPage, tdbDefaultFree, NULL); } } @@ -518,14 +518,14 @@ static int tdbPCacheCloseImpl(SPCache *pCache) { // free free page for (SPage *pPage = pCache->pFree; pPage;) { SPage *pPageT = pPage->pFreeNext; - (void)tdbPageDestroy(pPage, tdbDefaultFree, NULL); + tdbPageDestroy(pPage, tdbDefaultFree, NULL); pPage = pPageT; } for (int32_t iBucket = 0; iBucket < pCache->nHash; iBucket++) { for (SPage *pPage = pCache->pgHash[iBucket]; pPage;) { SPage *pPageT = pPage->pHashNext; - (void)tdbPageDestroy(pPage, tdbDefaultFree, NULL); + tdbPageDestroy(pPage, tdbDefaultFree, NULL); pPage = pPageT; } } diff --git a/source/libs/tdb/src/db/tdbPage.c b/source/libs/tdb/src/db/tdbPage.c index eab8f6ef19..7888a08bb3 100644 --- a/source/libs/tdb/src/db/tdbPage.c +++ b/source/libs/tdb/src/db/tdbPage.c @@ -79,19 +79,19 @@ int tdbPageCreate(int pageSize, SPage **ppPage, void *(*xMalloc)(void *, size_t) return 0; } -int tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg) { +void tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg) { u8 *ptr; tdbTrace("tdb/page-destroy: %p/%d %p", pPage, pPage->id, xFree); if (pPage->isDirty) { tdbError("tdb/page-destroy: dirty page: %" PRIu8 ".", pPage->isDirty); - return TSDB_CODE_INVALID_PARA; + return; } if (!xFree) { tdbError("tdb/page-destroy: null xFree."); - return TSDB_CODE_INVALID_PARA; + return; } for (int iOvfl = 0; iOvfl < pPage->nOverflow; iOvfl++) { @@ -102,7 +102,7 @@ int tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg) ptr = pPage->pData; xFree(arg, ptr); - return 0; + return; } void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, TXN *, SBTree *pBt)) { @@ -156,6 +156,7 @@ int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl int iOvfl; int lidx; // local idx SCell *pNewCell; + int ret; if (szCell > TDB_PAGE_MAX_FREE_BLOCK(pPage, pPage->pPageHdr - pPage->pData)) { tdbError("tdb/page-insert-cell: invalid page, szCell: %d, max free: %lu", szCell, @@ -184,6 +185,10 @@ int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl // TODO: here has memory leak pNewCell = (SCell *)tdbOsMalloc(szCell); + if (pNewCell == NULL) { + tdbError("tdb/page-insert-cell: malloc failed."); + return terrno; + } memcpy(pNewCell, pCell, szCell); tdbTrace("tdbPage/insert/new ovfl cell: %p/%p", pNewCell, pPage); @@ -194,7 +199,11 @@ int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl iOvfl++; } else { // page must has enough space to hold the cell locally - (void)tdbPageAllocate(pPage, szCell, &pNewCell); + ret = tdbPageAllocate(pPage, szCell, &pNewCell); + if (ret) { + tdbError("tdb/page-insert-cell: allocate cell failed, idx: %d, szCell: %d", idx, szCell); + return ret; + } memcpy(pNewCell, pCell, szCell); @@ -220,7 +229,9 @@ int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl } int tdbPageUpdateCell(SPage *pPage, int idx, SCell *pCell, int szCell, TXN *pTxn, SBTree *pBt) { - (void)tdbPageDropCell(pPage, idx, pTxn, pBt); + int ret = tdbPageDropCell(pPage, idx, pTxn, pBt); + if (ret) return ret; + return tdbPageInsertCell(pPage, idx, pCell, szCell, 0); } @@ -230,6 +241,7 @@ int tdbPageDropCell(SPage *pPage, int idx, TXN *pTxn, SBTree *pBt) { int szCell; int nCells; int iOvfl; + int ret; nCells = TDB_PAGE_NCELLS(pPage); @@ -259,9 +271,12 @@ int tdbPageDropCell(SPage *pPage, int idx, TXN *pTxn, SBTree *pBt) { lidx = idx - iOvfl; pCell = TDB_PAGE_CELL_AT(pPage, lidx); szCell = (*pPage->xCellSize)(pPage, pCell, 1, pTxn, pBt); - (void)tdbPageFree(pPage, lidx, pCell, szCell); + ret = tdbPageFree(pPage, lidx, pCell, szCell); + if (ret) { + tdbError("tdb/page-drop-cell: free cell failed, idx: %d, lidx: %d, szCell: %d", idx, lidx, szCell); + return ret; + } TDB_PAGE_NCELLS_SET(pPage, nCells - 1); - for (; iOvfl < pPage->nOverflow; iOvfl++) { pPage->aiOvfl[iOvfl]--; if (pPage->aiOvfl[iOvfl] <= 0) { diff --git a/source/libs/tdb/src/db/tdbPager.c b/source/libs/tdb/src/db/tdbPager.c index 4a5ad9d663..64b892692d 100644 --- a/source/libs/tdb/src/db/tdbPager.c +++ b/source/libs/tdb/src/db/tdbPager.c @@ -802,7 +802,7 @@ static int tdbPagerRemoveFreePage(SPager *pPager, SPgno *pPgno, TXN *pTxn) { code = tdbTbcMoveToFirst(pCur); if (code) { tdbError("tdb/remove-free-page: moveto first failed with ret: %d.", code); - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return 0; } @@ -812,7 +812,7 @@ static int tdbPagerRemoveFreePage(SPager *pPager, SPgno *pPgno, TXN *pTxn) { code = tdbTbcGet(pCur, (const void **)&pKey, &nKey, NULL, NULL); if (code < 0) { // tdbError("tdb/remove-free-page: tbc get failed with ret: %d.", code); - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return 0; } @@ -823,10 +823,10 @@ static int tdbPagerRemoveFreePage(SPager *pPager, SPgno *pPgno, TXN *pTxn) { code = tdbTbcDelete(pCur); if (code < 0) { tdbError("tdb/remove-free-page: tbc delete failed with ret: %d.", code); - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return 0; } - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return 0; } @@ -1165,14 +1165,14 @@ int tdbPagerRestoreJournals(SPager *pPager) { code = tdbPagerRestore(pPager, jname); if (code) { taosArrayDestroy(pTxnList); - (void)tdbCloseDir(&pDir); + tdbCloseDir(&pDir); tdbError("failed to restore file due to %s. jFileName:%s", tstrerror(code), jname); return code; } } taosArrayDestroy(pTxnList); - (void)tdbCloseDir(&pDir); + tdbCloseDir(&pDir); return 0; } @@ -1195,7 +1195,7 @@ int tdbPagerRollback(SPager *pPager) { jname[dirLen] = '/'; memcpy(jname + dirLen + 1, name, strlen(name)); if (tdbOsRemove(jname) < 0 && errno != ENOENT) { - (void)tdbCloseDir(&pDir); + tdbCloseDir(&pDir); tdbError("failed to remove file due to %s. jFileName:%s", strerror(errno), name); return terrno = TAOS_SYSTEM_ERROR(errno); @@ -1203,7 +1203,7 @@ int tdbPagerRollback(SPager *pPager) { } } - (void)tdbCloseDir(&pDir); + tdbCloseDir(&pDir); return 0; } diff --git a/source/libs/tdb/src/db/tdbTable.c b/source/libs/tdb/src/db/tdbTable.c index 11275c9ec8..f62e211586 100644 --- a/source/libs/tdb/src/db/tdbTable.c +++ b/source/libs/tdb/src/db/tdbTable.c @@ -128,7 +128,7 @@ int tdbTbOpen(const char *tbname, int keyLen, int valLen, tdb_cmpr_fn_t keyCmprF int tdbTbClose(TTB *pTb) { if (pTb) { - (void)tdbBtreeClose(pTb->pBt); + tdbBtreeClose(pTb->pBt); tdbOsFree(pTb); } return 0; @@ -260,7 +260,7 @@ int32_t tdbTbTraversal(TTB *pTb, void *data, } tdbFree(pKey); tdbFree(pValue); - (void)tdbTbcClose(pCur); + tdbTbcClose(pCur); return 0; } @@ -293,13 +293,11 @@ int tdbTbcUpsert(TBC *pTbc, const void *pKey, int nKey, const void *pData, int n return tdbBtcUpsert(&pTbc->btc, pKey, nKey, pData, nData, insert); } -int tdbTbcClose(TBC *pTbc) { +void tdbTbcClose(TBC *pTbc) { if (pTbc) { - (void)tdbBtcClose(&pTbc->btc); + tdbBtcClose(&pTbc->btc); tdbOsFree(pTbc); } - - return 0; } int tdbTbcIsValid(TBC *pTbc) { return tdbBtcIsValid(&pTbc->btc); } diff --git a/source/libs/tdb/src/inc/tdbInt.h b/source/libs/tdb/src/inc/tdbInt.h index 7e97be962b..a4498ee1ea 100644 --- a/source/libs/tdb/src/inc/tdbInt.h +++ b/source/libs/tdb/src/inc/tdbInt.h @@ -27,17 +27,6 @@ extern "C" { #endif -// clang-format off -extern int32_t tdbDebugFlag; - -#define tdbFatal(...) do { if (tdbDebugFlag & DEBUG_FATAL) { taosPrintLog("TDB FATAL ", DEBUG_FATAL, 255, __VA_ARGS__); }} while(0) -#define tdbError(...) do { if (tdbDebugFlag & DEBUG_ERROR) { taosPrintLog("TDB ERROR ", DEBUG_ERROR, 255, __VA_ARGS__); }} while(0) -#define tdbWarn(...) do { if (tdbDebugFlag & DEBUG_WARN) { taosPrintLog("TDB WARN ", DEBUG_WARN, 255, __VA_ARGS__); }} while(0) -#define tdbInfo(...) do { if (tdbDebugFlag & DEBUG_INFO) { taosPrintLog("TDB ", DEBUG_INFO, 255, __VA_ARGS__); }} while(0) -#define tdbDebug(...) do { if (tdbDebugFlag & DEBUG_DEBUG) { taosPrintLog("TDB ", DEBUG_DEBUG, tdbDebugFlag, __VA_ARGS__); }} while(0) -#define tdbTrace(...) do { if (tdbDebugFlag & DEBUG_TRACE) { taosPrintLog("TDB ", DEBUG_TRACE, tdbDebugFlag, __VA_ARGS__); }} while(0) -// clang-format on - typedef int8_t i8; typedef int16_t i16; typedef int32_t i32; @@ -156,11 +145,11 @@ struct SBTC { }; // SBTree -int tdbBtreeOpen(int keyLen, int valLen, SPager *pFile, char const *tbname, SPgno pgno, tdb_cmpr_fn_t kcmpr, TDB *pEnv, - SBTree **ppBt); -int tdbBtreeClose(SBTree *pBt); -int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, int vLen, TXN *pTxn); -int tdbBtreeDelete(SBTree *pBt, const void *pKey, int kLen, TXN *pTxn); +int tdbBtreeOpen(int keyLen, int valLen, SPager *pFile, char const *tbname, SPgno pgno, tdb_cmpr_fn_t kcmpr, TDB *pEnv, + SBTree **ppBt); +void tdbBtreeClose(SBTree *pBt); +int tdbBtreeInsert(SBTree *pBt, const void *pKey, int kLen, const void *pVal, int vLen, TXN *pTxn); +int tdbBtreeDelete(SBTree *pBt, const void *pKey, int kLen, TXN *pTxn); // int tdbBtreeUpsert(SBTree *pBt, const void *pKey, int nKey, const void *pData, int nData, TXN *pTxn); int tdbBtreeGet(SBTree *pBt, const void *pKey, int kLen, void **ppVal, int *vLen); int tdbBtreePGet(SBTree *pBt, const void *pKey, int kLen, void **ppKey, int *pkLen, void **ppVal, int *vLen); @@ -173,19 +162,19 @@ typedef struct { int tdbBtreeInitPage(SPage *pPage, void *arg, int init); // SBTC -int tdbBtcOpen(SBTC *pBtc, SBTree *pBt, TXN *pTxn); -int tdbBtcClose(SBTC *pBtc); -int tdbBtcIsValid(SBTC *pBtc); -int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst); -int tdbBtcMoveToFirst(SBTC *pBtc); -int tdbBtcMoveToLast(SBTC *pBtc); -int tdbBtcMoveToNext(SBTC *pBtc); -int tdbBtcMoveToPrev(SBTC *pBtc); -int tdbBtreeNext(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen); -int tdbBtreePrev(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen); -int tdbBtcGet(SBTC *pBtc, const void **ppKey, int *kLen, const void **ppVal, int *vLen); -int tdbBtcDelete(SBTC *pBtc); -int tdbBtcUpsert(SBTC *pBtc, const void *pKey, int kLen, const void *pData, int nData, int insert); +int tdbBtcOpen(SBTC *pBtc, SBTree *pBt, TXN *pTxn); +void tdbBtcClose(SBTC *pBtc); +int tdbBtcIsValid(SBTC *pBtc); +int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst); +int tdbBtcMoveToFirst(SBTC *pBtc); +int tdbBtcMoveToLast(SBTC *pBtc); +int tdbBtcMoveToNext(SBTC *pBtc); +int tdbBtcMoveToPrev(SBTC *pBtc); +int tdbBtreeNext(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen); +int tdbBtreePrev(SBTC *pBtc, void **ppKey, int *kLen, void **ppVal, int *vLen); +int tdbBtcGet(SBTC *pBtc, const void **ppKey, int *kLen, const void **ppVal, int *vLen); +int tdbBtcDelete(SBTC *pBtc); +int tdbBtcUpsert(SBTC *pBtc, const void *pKey, int kLen, const void *pData, int nData, int insert); // tdbPager.c ==================================== @@ -339,7 +328,7 @@ static inline int tdbTryLockPage(tdb_spinlock_t *pLock) { #define TDB_PAGE_OFFSET_SIZE(pPage) ((pPage)->pPageMethods->szOffset) int tdbPageCreate(int pageSize, SPage **ppPage, void *(*xMalloc)(void *, size_t), void *arg); -int tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg); +void tdbPageDestroy(SPage *pPage, void (*xFree)(void *arg, void *ptr), void *arg); void tdbPageZero(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, TXN *, SBTree *pBt)); void tdbPageInit(SPage *pPage, u8 szAmHdr, int (*xCellSize)(const SPage *, SCell *, int, TXN *, SBTree *pBt)); int tdbPageInsertCell(SPage *pPage, int idx, SCell *pCell, int szCell, u8 asOvfl); diff --git a/source/libs/tdb/src/inc/tdbOs.h b/source/libs/tdb/src/inc/tdbOs.h index db4283e267..993c3ffab8 100644 --- a/source/libs/tdb/src/inc/tdbOs.h +++ b/source/libs/tdb/src/inc/tdbOs.h @@ -27,6 +27,17 @@ extern "C" { #include "os.h" #include "thash.h" +// clang-format off +extern int32_t tdbDebugFlag; + +#define tdbFatal(...) do { if (tdbDebugFlag & DEBUG_FATAL) { taosPrintLog("TDB FATAL ", DEBUG_FATAL, 255, __VA_ARGS__); }} while(0) +#define tdbError(...) do { if (tdbDebugFlag & DEBUG_ERROR) { taosPrintLog("TDB ERROR ", DEBUG_ERROR, 255, __VA_ARGS__); }} while(0) +#define tdbWarn(...) do { if (tdbDebugFlag & DEBUG_WARN) { taosPrintLog("TDB WARN ", DEBUG_WARN, 255, __VA_ARGS__); }} while(0) +#define tdbInfo(...) do { if (tdbDebugFlag & DEBUG_INFO) { taosPrintLog("TDB ", DEBUG_INFO, 255, __VA_ARGS__); }} while(0) +#define tdbDebug(...) do { if (tdbDebugFlag & DEBUG_DEBUG) { taosPrintLog("TDB ", DEBUG_DEBUG, tdbDebugFlag, __VA_ARGS__); }} while(0) +#define tdbTrace(...) do { if (tdbDebugFlag & DEBUG_TRACE) { taosPrintLog("TDB ", DEBUG_TRACE, tdbDebugFlag, __VA_ARGS__); }} while(0) +// clang-format on + // For memory ----------------- #define tdbOsMalloc taosMemoryMalloc #define tdbOsCalloc taosMemoryCalloc @@ -59,9 +70,16 @@ typedef TdFilePtr tdb_fd_t; #define tdbReadDir taosReadDir #define tdbGetDirEntryName taosGetDirEntryName #define tdbDirEntryBaseName taosDirEntryBaseName -#define tdbCloseDir taosCloseDir -#define tdbOsRemove remove -#define tdbOsFileSize(FD, PSIZE) taosFStatFile(FD, PSIZE, NULL) + +static FORCE_INLINE void tdbCloseDir(TdDirPtr *ppDir) { + int32_t ret = taosCloseDir(ppDir); + if (ret) { + tdbError("failed to close directory, reason:%s", tstrerror(ret)); + } +} + +#define tdbOsRemove remove +#define tdbOsFileSize(FD, PSIZE) taosFStatFile(FD, PSIZE, NULL) /* directory */ #define tdbOsMkdir taosMkDir diff --git a/source/util/src/tbloomfilter.c b/source/util/src/tbloomfilter.c index 841657e628..0889017cde 100644 --- a/source/util/src/tbloomfilter.c +++ b/source/util/src/tbloomfilter.c @@ -177,6 +177,8 @@ int32_t tBloomFilterDecode(SDecoder* pDecoder, SBloomFilter** ppBF) { QUERY_CHECK_CODE(code, lino, _error); } pBF->buffer = taosMemoryCalloc(pBF->numUnits, sizeof(uint64_t)); + QUERY_CHECK_NULL(pBF->buffer, code, lino, _error, terrno); + for (int32_t i = 0; i < pBF->numUnits; i++) { uint64_t* pUnits = (uint64_t*)pBF->buffer; if (tDecodeU64(pDecoder, pUnits + i) < 0) { diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 87bac4cce3..9150527f14 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -353,7 +353,7 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/splitVGroupWal.py -N 3 -n 3 ,,n,system-test,python3 ./test.py -f 0-others/timeRangeWise.py -N 3 ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/delete_check.py -#,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/test_hot_refresh_configurations.py +,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/test_hot_refresh_configurations.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/subscribe_stream_privilege.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/empty_identifier.py diff --git a/tests/system-test/0-others/test_hot_refresh_configurations.py b/tests/system-test/0-others/test_hot_refresh_configurations.py index 15f291787a..ad21e8ce7c 100644 --- a/tests/system-test/0-others/test_hot_refresh_configurations.py +++ b/tests/system-test/0-others/test_hot_refresh_configurations.py @@ -13,18 +13,19 @@ from util.common import * class TDTestCase: """This test case is used to veirfy hot refresh configurations """ + def init(self, conn, logSql, replicaVar=1): self.replicaVar = int(replicaVar) tdLog.debug(f"start to excute {__file__}") tdSql.init(conn.cursor()) - + self.configration_dic = { "cli": [ - { - "name": "keepAliveIdle", - "values": [1, 100, 7200000], - "except_values": [0, 7200001] - }, + # { + # "name": "keepAliveIdle", + # "values": [1, 100, 7200000], + # "except_values": [0, 7200001] + # }, { "name": "queryPolicy", "values": [1, 2, 4], @@ -42,12 +43,12 @@ class TDTestCase: } ], "svr": [ - { - "name": "keepAliveIdle", - "alias": "tsKeepAliveIdle", - "values": [1, 100, 7200000], - "except_values": [0, 7200001] - }, + # { + # "name": "keepAliveIdle", + # "alias": "tsKeepAliveIdle", + # "values": [1, 100, 7200000], + # "except_values": [0, 7200001] + # }, { "name": "mndSdbWriteDelta", "alias": "tsMndSdbWriteDelta", @@ -100,6 +101,7 @@ class TDTestCase: "name": "minDiskFreeSize", "alias": "tsMinDiskFreeSize", "values": ["51200K", "100M", "1G"], + "check_values": ["52428800", "104857600", "1073741824"], "except_values": ["1024K", "1.1G", "1T"] }, { @@ -159,27 +161,52 @@ class TDTestCase: ] } - def get_param_value(self, config_name): + def cli_get_param_value(self, config_name): tdSql.query("show local variables;") for row in tdSql.queryResult: if config_name == row[0]: tdLog.debug("Found variable '{}'".format(row[0])) return row[1] - def cli_check(self, name, values, except_values=False): + def svr_get_param_value(self, config_name): + tdSql.query("show dnode 1 variables;") + for row in tdSql.queryResult: + if config_name == row[1]: + tdLog.debug("Found variable '{}'".format(row[1])) + return row[2] + + def cli_check(self, item, except_values=False): + name = item["name"] + if except_values: + values = item["except_values"] + else: + values = item["values"] + check_values = item.get("check_values", []) if not except_values: - for v in values: + for i in range(len(values)): + v = values[i] tdLog.debug("Set {} to {}".format(name, v)) tdSql.execute(f'alter local "{name} {v}";') - value = self.get_param_value(name) + value = self.cli_get_param_value(name) tdLog.debug("Get {} value: {}".format(name, value)) - assert(v == int(value)) + if check_values: + tdLog.debug(f"assert {check_values[i]} == {str(value)}") + assert str(check_values[i]) == str(value) + else: + tdLog.debug(f"assert {v} == {str(value)}") + assert str(v) == str(value) else: for v in values: tdLog.debug("Set {} to {}".format(name, v)) tdSql.error(f'alter local "{name} {v}";') - def svr_check(self, name, alias, values, except_values=False): + def svr_check(self, item, except_values=False): + name = item["name"] + if except_values: + values = item["except_values"] + else: + values = item["values"] + check_values = item.get("check_values", []) p_list = ["dnode 1", "all dnodes"] # check bool param value if len(values) == 2 and [0, 1] == values and name != "queryRspPolicy": @@ -188,13 +215,18 @@ class TDTestCase: is_bool = False tdLog.debug(f"{name} is_bool: {is_bool}") if not except_values: - for v in values: + for i in range(len(values)): + v = values[i] dnode = random.choice(p_list) tdSql.execute(f'alter {dnode} "{name} {v}";') - value = self.get_param_value(alias) - if value: - tdLog.debug(f"value: {value}") - assert(value == str(bool(v)).lower() if is_bool else str(v)) + value = self.svr_get_param_value(name) + tdLog.debug(f"value: {value}") + if check_values: + tdLog.debug(f"assert {check_values[i]} == {str(value)}") + assert str(check_values[i]) == str(value) + else: + tdLog.debug(f"assert {v} == {str(value)}") + assert str(v) == str(value) else: for v in values: dnode = random.choice(p_list) @@ -205,28 +237,29 @@ class TDTestCase: # reset log taosdLogAbsoluteFilename = tdCom.getTaosdPath() + "/log/" + "taosdlog*" tdSql.execute("alter all dnodes 'resetlog';") - r = subprocess.Popen("cat {} | grep 'reset log file'".format(taosdLogAbsoluteFilename), shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + r = subprocess.Popen("cat {} | grep 'reset log file'".format(taosdLogAbsoluteFilename), shell=True, + stdout=subprocess.PIPE, stderr=subprocess.PIPE) stdout, stderr = r.communicate() - assert('reset log file' in stdout.decode()) + assert ('reset log file' in stdout.decode()) for key in self.configration_dic: if "cli" == key: for item in self.configration_dic[key]: - self.cli_check(item["name"], item["values"]) + self.cli_check(item) if "except_values" in item: - self.cli_check(item["name"], item["except_values"], True) + self.cli_check(item, True) elif "svr" == key: for item in self.configration_dic[key]: - self.svr_check(item["name"], item["alias"], item["values"]) + self.svr_check(item) if "except_values" in item: - self.svr_check(item["name"], item["alias"], item["except_values"], True) + self.svr_check(item, True) else: raise Exception(f"unknown key: {key}") - def stop(self): tdSql.close() tdLog.success(f"{__file__} successfully executed") + tdCases.addLinux(__file__, TDTestCase()) tdCases.addWindows(__file__, TDTestCase())