xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge remote-tracking branch 'origin/3.0' into fix/addErrorToMgmt

This commit is contained in:
Yihao Deng 2024-07-18 00:57:10 +00:00
parent 71ec35a7da a3c0acb78a
commit 690c6e1cf1
42 changed files with 1953 additions and 1209 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
docs/en/12-taos-sql/03-table.md
View File

@ -49,6 +49,7 @@ table_option: {
7. Escape character "\`" can be used to avoid the conflict between table names and reserved keywords, above rules will be bypassed when using escape character on table names, but the upper limit for the name length is still valid. The table names specified using escape character are case sensitive. 7. Escape character "\`" can be used to avoid the conflict between table names and reserved keywords, above rules will be bypassed when using escape character on table names, but the upper limit for the name length is still valid. The table names specified using escape character are case sensitive.
For example \`aBc\` and \`abc\` are different table names but `abc` and `aBc` are same table names because they are both converted to `abc` internally. For example \`aBc\` and \`abc\` are different table names but `abc` and `aBc` are same table names because they are both converted to `abc` internally.
Only ASCII visible characters can be used with escape character. Only ASCII visible characters can be used with escape character.
8. For the details of using `ENCODE` and `COMPRESS`, please refer to [Encode and Compress for Column](../compress).
**Parameter description** **Parameter description**

20
docs/en/12-taos-sql/04-stable.md
View File

@ -13,17 +13,29 @@ create_definition:
col_name column_definition col_name column_definition
column_definition: column_definition:
type_name type_name [comment 'string_value'] [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type']
table_options:
table_option ...
table_option: {
COMMENT 'string_value'
| SMA(col_name [, col_name] ...)
| TTL value
}
``` ```
**More explanations** **More explanations**
- Each supertable can have a maximum of 4096 columns, including tags. The minimum number of columns is 3: a timestamp column used as the key, one tag column, and one data column. 1. Each supertable can have a maximum of 4096 columns, including tags. The minimum number of columns is 3: a timestamp column used as the key, one tag column, and one data column.
- The TAGS keyword defines the tag columns for the supertable. The following restrictions apply to tag columns: 2. Since version 3.3.0.0, besides the timestamp, you can specify another column as primary key using `PRIMARY KEY` keyword, the column specified using `primary key` must be type of integer or varchar.
2. The TAGS keyword defines the tag columns for the supertable. The following restrictions apply to tag columns:
- A tag column can use the TIMESTAMP data type, but the values in the column must be fixed numbers. Timestamps including formulae, such as "now + 10s", cannot be stored in a tag column. - A tag column can use the TIMESTAMP data type, but the values in the column must be fixed numbers. Timestamps including formulae, such as "now + 10s", cannot be stored in a tag column.
- The name of a tag column cannot be the same as the name of any other column. - The name of a tag column cannot be the same as the name of any other column.
- The name of a tag column cannot be a reserved keyword. - The name of a tag column cannot be a reserved keyword.
- Each supertable must contain between 1 and 128 tags. The total length of the TAGS keyword cannot exceed 16 KB. - Each supertable must contain between 1 and 128 tags. The total length of the TAGS keyword cannot exceed 16 KB.
- For more information about table parameters, see Create a Table. 3. Regarding how to use `ENCODE` and `COMPRESS`, please refer to [Encode and Compress for Column](../compress).
3. For more information about table parameters, see [Create a Table](../table).
## View a Supertable ## View a Supertable

0
docs/en/21-tdinternal/08-compress.md → docs/en/12-taos-sql/31-compress.md
View File

51
docs/en/14-reference/12-config/index.md
View File

@ -729,6 +729,57 @@ The charset that takes effect is UTF-8.
| Value Range | -1: none message is compressed; 0: all messages are compressed; N (N>0): messages exceeding N bytes are compressed | | Value Range | -1: none message is compressed; 0: all messages are compressed; N (N>0): messages exceeding N bytes are compressed |
| Default | -1 | | Default | -1 |
### fPrecision
| Attribute | Description |
| -------- | -------------------------------- |
| Application | Server Only |
| Meaning | Compression precision for float data type |
| Value Range | 0.1 ~ 0.00000001 |
| Default | 0.00000001 |
| Note | The floating value below this setting will be cut off |
### dPrecision
| Attribute | Description |
| -------- | -------------------------------- |
| Applicable | Server Only |
| Meaning | Compression precision for double data type |
| Value Range | 0.1 ~ 0.0000000000000001 |
| Default | 0.0000000000000001 |
| Note | The floating value below this setting will be cut off |
### lossyColumn
| Attribute | Description |
| -------- | -------------------------------- |
| Applicable | Server Only |
| Meaning | Enable TSZ lossy compression for float and/or double |
| Value Range | float, double |
| Default | none: disable TSZ lossy compression |
**补充说明**
1. It's only available since 3.2.0.0 version, and can't downgrade to previous version once upgrading to 3.2.0.0 and enabling this parameter
2. TSZ compression algorithm compresses data based on data prediction technique, so it's more suitable for data with specific pattern
3. TSZ compression algorithm may take longer time but it has better compression ratio, so it's suitable when you have enough CPU resources and are more sensitive to disk occupation
4. Example: enable TSZ for both float and double
```shell
lossyColumns float|double
```
5. After configuring, taosd service needs to restarted. After restarting, if you see the following output in taosd logfile, it means the function has been enabled
```sql
02/22 10:49:27.607990 00002933 UTL lossyColumns float|double
```
### ifAdtFse
| Attribute | Description |
| -------- | -------------------------------- |
| Applicable | Server Only |
| Meaning | Replace HUFFMAN with FSE in TSZ, FSE is faster when compressing but slower when uncompressing |
| Value Range | 0: Use HUFFMAN, 1: Use FSE |
| Default | 0: Use HUFFMAN |
## Other Parameters ## Other Parameters

1
docs/zh/12-taos-sql/03-table.md
View File

@ -49,6 +49,7 @@ table_option: {
6. 使用数据类型 BINARY/NCHAR/GEOMETRY，需指定其最长的字节数，如 BINARY(20)，表示 20 字节。 6. 使用数据类型 BINARY/NCHAR/GEOMETRY，需指定其最长的字节数，如 BINARY(20)，表示 20 字节。
7. 为了兼容支持更多形式的表名，TDengine 引入新的转义符 "\`"，可以让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查。但是同样具有长度限制要求。使用转义字符以后，不再对转义字符中的内容进行大小写统一， 7. 为了兼容支持更多形式的表名，TDengine 引入新的转义符 "\`"，可以让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查。但是同样具有长度限制要求。使用转义字符以后，不再对转义字符中的内容进行大小写统一，
例如：\`aBc\` 和 \`abc\` 是不同的表名，但是 abc 和 aBc 是相同的表名。 例如：\`aBc\` 和 \`abc\` 是不同的表名，但是 abc 和 aBc 是相同的表名。
8. 关于 `ENCODE` 和 `COMPRESS` 的使用，请参考[按列压缩](../compress)
**参数说明** **参数说明**

19
docs/zh/12-taos-sql/04-stable.md
View File

@ -13,17 +13,28 @@ create_definition:
col_name column_definition col_name column_definition
column_definition: column_definition:
type_name type_name [comment 'string_value'] [PRIMARY KEY] [ENCODE 'encode_type'] [COMPRESS 'compress_type'] [LEVEL 'level_type']
table_options:
table_option ...
table_option: {
COMMENT 'string_value'
| SMA(col_name [, col_name] ...)
| TTL value
}
``` ```
**使用说明** **使用说明**
- 超级表中列的最大个数为 4096，需要注意，这里的 4096 是包含 TAG 列在内的，最小个数为 3，包含一个时间戳主键、一个 TAG 列和一个数据列。 1. 超级表中列的最大个数为 4096，需要注意，这里的 4096 是包含 TAG 列在内的，最小个数为 3，包含一个时间戳主键、一个 TAG 列和一个数据列。
- TAGS语法指定超级表的标签列，标签列需要遵循以下约定： 2. 除时间戳主键列之外，还可以通过 PRIMARY KEY 关键字指定第二列为额外的主键列。被指定为主键列的第二列必须为整型或字符串类型（varchar）
3. TAGS语法指定超级表的标签列，标签列需要遵循以下约定：
- TAGS 中的 TIMESTAMP 列写入数据时需要提供给定值，而暂不支持四则运算，例如 NOW + 10s 这类表达式。 - TAGS 中的 TIMESTAMP 列写入数据时需要提供给定值，而暂不支持四则运算，例如 NOW + 10s 这类表达式。
- TAGS 列名不能与其他列名相同。 - TAGS 列名不能与其他列名相同。
- TAGS 列名不能为预留关键字。 - TAGS 列名不能为预留关键字。
- TAGS 最多允许 128 个，至少 1 个，总长度不超过 16 KB。 - TAGS 最多允许 128 个，至少 1 个，总长度不超过 16 KB。
- 关于表参数的详细说明，参见 CREATE TABLE 中的介绍。 4. 关于 `ENCODE` 和 `COMPRESS` 的使用，请参考 [按列压缩](../compress)
5. 关于 table_option 中的参数说明，请参考 [建表 SQL 说明](../table)
## 查看超级表 ## 查看超级表

1
docs/zh/21-tdinternal/08-compress.md → docs/zh/12-taos-sql/31-compress.md
View File

@ -1,5 +1,6 @@
--- ---
title: 可配置压缩算法 title: 可配置压缩算法
sidebar_label: 可配置压缩
description: 可配置压缩算法 description: 可配置压缩算法
--- ---

54
docs/zh/14-reference/12-config/index.md
View File

@ -784,6 +784,57 @@ charset 的有效值是 UTF-8。
| 取值范围 | -1: 所有消息都不压缩; 0: 所有消息都压缩; N (N>0): 只有大于 N 个字节的消息才压缩 | | 取值范围 | -1: 所有消息都不压缩; 0: 所有消息都压缩; N (N>0): 只有大于 N 个字节的消息才压缩 |
| 缺省值 | -1 | | 缺省值 | -1 |
### fPrecision
FLOAT 类型压缩精度控制：
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 设置 float 类型浮点数压缩精度 |
| 取值范围 | 0.1 ~ 0.00000001 |
| 缺省值 | 0.00000001 |
| 补充说明 | 小于此值的浮点数尾数部分将被截取 |
### dPrecision
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 设置 double 类型浮点数压缩精度 |
| 取值范围 | 0.1 ~ 0.0000000000000001 |
| 缺省值 | 0.0000000000000001 |
| 补充说明 | 小于此值的浮点数尾数部分将被截取 |
### lossyColumn
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 对 float 和/或 double 类型启用 TSZ 有损压缩 |
| 取值范围 | float, double |
| 缺省值 | none：表示关闭无损压缩 |
**补充说明**
1. 在 3.2.0.0 ~ 3.3.0.0（不包含）版本生效，启用该参数后不能回退到升级前的版本
2. TSZ 压缩算法是通过数据预测技术完成的压缩，所以更适合有规律变化的数据
3. TSZ 压缩时间会更长一些，如果您的服务器 CPU 空闲多，存储空间小的情况下适合选用
4. 示例：对 float 和 double 类型都启用有损压缩
```shell
lossyColumns float|double
```
5. 配置需重启服务生效，重启如果在 taosd 日志中看到以下内容，表明配置已生效：
```sql
02/22 10:49:27.607990 00002933 UTL lossyColumns float|double
```
### ifAdtFse
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 在启用 TSZ 有损压缩时，使用 FSE 算法替换 HUFFMAN 算法， FSE 算法压缩速度更快，但解压稍慢，追求压缩速度可选用此算法 |
| 取值范围 | 0：关闭 1：打开 |
| 缺省值 | 0：关闭 |
## 3.0 中有效的配置参数列表 ## 3.0 中有效的配置参数列表
| # | **参数** | **适用于 2.X ** | **适用于 3.0 ** | 3.0 版本的当前行为 | | # | **参数** | **适用于 2.X ** | **适用于 3.0 ** | 3.0 版本的当前行为 |
@ -922,8 +973,5 @@ charset 的有效值是 UTF-8。
| 73 | probeSeconds | 是 | 否 | 3.0 行为未知 | | 73 | probeSeconds | 是 | 否 | 3.0 行为未知 |
| 74 | probeKillSeconds | 是 | 否 | 3.0 行为未知 | | 74 | probeKillSeconds | 是 | 否 | 3.0 行为未知 |
| 75 | probeInterval | 是 | 否 | 3.0 行为未知 | | 75 | probeInterval | 是 | 否 | 3.0 行为未知 |
| 76 | lossyColumns | 是 | 否 | 3.0 行为未知 |
| 77 | fPrecision | 是 | 否 | 3.0 行为未知 |
| 78 | dPrecision | 是 | 否 | 3.0 行为未知 |
| 79 | maxRange | 是 | 否 | 3.0 行为未知 | | 79 | maxRange | 是 | 否 | 3.0 行为未知 |
| 80 | range | 是 | 否 | 3.0 行为未知 | | 80 | range | 是 | 否 | 3.0 行为未知 |

69
docs/zh/21-tdinternal/07-tsz.md
View File

@ -1,69 +0,0 @@
---
title: TSZ 压缩算法
description: TDengine 对浮点数进行高效压缩的算法
---
TSZ 压缩算法是 TDengine 为浮点数据类型提供的可选压缩算法，可以实现浮点数有损至无损全状态压缩，相比默认压缩算法， TSZ 压缩算法压缩率更高，即使切至无损状态，压缩率也会比默认压缩高一倍。
## 适合场景
- TSZ 压缩算法是通过数据预测技术完成的压缩，所以更适合有规律变化的数据
- TSZ 压缩时间会更长一些，如果您的服务器 CPU 空闲多，存储空间小的情况下适合选用
## 使用步骤
- TDengine 支持版本为 3.2.0.0 或以上
- 开启选项
在 taos.cfg 配置中增加以下内容，即可开启 TSZ 压缩算法，功能打开后，会替换默认算法。
以下表示字段类型是 float 及 double 类型都使用此压缩算法，也可以单独只配置一个
```sql
lossyColumns float|double
```
- 配置需重启服务生效
- Taosd 日志输出以下内容，表明功能已生效：
```sql
02/22 10:49:27.607990 00002933 UTL lossyColumns float|double
```
## 配置参数
### fPrecision
FLOAT 类型精度控制：
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 设置 float 类型浮点数压缩精度 |
| 取值范围 | 0.1 ~ 0.00000001 |
| 缺省值 | 0.00000001 |
| 补充说明 | 小于此值的浮点数尾数部分将被截取 |
### dPrecision
DOUBLE 类型精度控制：
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 设置 double 类型浮点数压缩精度 |
| 取值范围 | 0.1 ~ 0.0000000000000001 |
| 缺省值 | 0.0000000000000001 |
| 补充说明 | 小于此值的浮点数尾数部分将被截取 |
### ifAdtFse
TSZ 压缩中可选择的算法 FSE，默认为 HUFFMAN：
| 属性 | 说明 |
| -------- | -------------------------------- |
| 适用范围 | 服务器端 |
| 含义 | 使用 FSE 算法替换 HUFFMAN 算法， FSE 算法压缩速度更快，但解压稍慢，追求压缩速度可选用此算法 |
| 取值范围 | 0：关闭 1：打开 |
| 缺省值 | 0：关闭 |
## 注意事项
- 打开 TSZ 后生成的存储数据格式，回退至 3.2.0.0 之前的版本，数据将不能被识别

4
include/libs/parser/parser.h
View File

@ -163,8 +163,8 @@ int rawBlockBindData(SQuery *query, STableMeta* pTableMeta, void* data, SVCr
int numFields, bool needChangeLength, char* errstr, int32_t errstrLen); int numFields, bool needChangeLength, char* errstr, int32_t errstrLen);
int32_t rewriteToVnodeModifyOpStmt(SQuery* pQuery, SArray* pBufArray); int32_t rewriteToVnodeModifyOpStmt(SQuery* pQuery, SArray* pBufArray);
SArray* serializeVgroupsCreateTableBatch(SHashObj* pVgroupHashmap); int32_t serializeVgroupsCreateTableBatch(SHashObj* pVgroupHashmap, SArray** pOut);
SArray* serializeVgroupsDropTableBatch(SHashObj* pVgroupHashmap); int32_t serializeVgroupsDropTableBatch(SHashObj* pVgroupHashmap, SArray** pOut);
void destoryCatalogReq(SCatalogReq *pCatalogReq); void destoryCatalogReq(SCatalogReq *pCatalogReq);
bool isPrimaryKeyImpl(SNode* pExpr); bool isPrimaryKeyImpl(SNode* pExpr);
int32_t insAppendStmtTableDataCxt(SHashObj* pAllVgHash, STableColsData* pTbData, STableDataCxt* pTbCtx, SStbInterlaceInfo* pBuildInfo); int32_t insAppendStmtTableDataCxt(SHashObj* pAllVgHash, STableColsData* pTbData, STableDataCxt* pTbCtx, SStbInterlaceInfo* pBuildInfo);

2
include/libs/stream/tstream.h
View File

@ -237,6 +237,8 @@ typedef struct {
int64_t stbUid; int64_t stbUid;
char stbFullName[TSDB_TABLE_FNAME_LEN]; char stbFullName[TSDB_TABLE_FNAME_LEN];
SSchemaWrapper* pSchemaWrapper; SSchemaWrapper* pSchemaWrapper;
SSchemaWrapper* pTagSchema;
bool autoCreateCtb;
void* vnode; // not available to encoder and decoder void* vnode; // not available to encoder and decoder
FTbSink* tbSinkFunc; FTbSink* tbSinkFunc;
STSchema* pTSchema; STSchema* pTSchema;

57
include/util/tqueue.h
View File

@ -73,37 +73,36 @@ struct STaosQnode {
char item[]; char item[];
}; };
STaosQueue *taosOpenQueue(); int32_t taosOpenQueue(STaosQueue **queue);
void taosCloseQueue(STaosQueue *queue); void taosCloseQueue(STaosQueue *queue);
void taosSetQueueFp(STaosQueue *queue, FItem itemFp, FItems itemsFp); void taosSetQueueFp(STaosQueue *queue, FItem itemFp, FItems itemsFp);
void *taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize); int32_t taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize, void **item);
int32_t taosAllocateQitemWrapper(int32_t size, EQItype itype, int64_t dataSize, void **pItem); void taosFreeQitem(void *pItem);
void taosFreeQitem(void *pItem); int32_t taosWriteQitem(STaosQueue *queue, void *pItem);
int32_t taosWriteQitem(STaosQueue *queue, void *pItem); int32_t taosReadQitem(STaosQueue *queue, void **ppItem);
int32_t taosReadQitem(STaosQueue *queue, void **ppItem); bool taosQueueEmpty(STaosQueue *queue);
bool taosQueueEmpty(STaosQueue *queue); void taosUpdateItemSize(STaosQueue *queue, int32_t items);
void taosUpdateItemSize(STaosQueue *queue, int32_t items); int32_t taosQueueItemSize(STaosQueue *queue);
int32_t taosQueueItemSize(STaosQueue *queue); int64_t taosQueueMemorySize(STaosQueue *queue);
int64_t taosQueueMemorySize(STaosQueue *queue); void taosSetQueueCapacity(STaosQueue *queue, int64_t size);
void taosSetQueueCapacity(STaosQueue *queue, int64_t size); void taosSetQueueMemoryCapacity(STaosQueue *queue, int64_t mem);
void taosSetQueueMemoryCapacity(STaosQueue *queue, int64_t mem);
STaosQall *taosAllocateQall(); int32_t taosAllocateQall(STaosQall **qall);
void taosFreeQall(STaosQall *qall); void taosFreeQall(STaosQall *qall);
int32_t taosReadAllQitems(STaosQueue *queue, STaosQall *qall); int32_t taosReadAllQitems(STaosQueue *queue, STaosQall *qall);
int32_t taosGetQitem(STaosQall *qall, void **ppItem); int32_t taosGetQitem(STaosQall *qall, void **ppItem);
void taosResetQitems(STaosQall *qall); void taosResetQitems(STaosQall *qall);
int32_t taosQallItemSize(STaosQall *qall); int32_t taosQallItemSize(STaosQall *qall);
int64_t taosQallMemSize(STaosQall *qll); int64_t taosQallMemSize(STaosQall *qll);
int64_t taosQallUnAccessedItemSize(STaosQall *qall); int64_t taosQallUnAccessedItemSize(STaosQall *qall);
int64_t taosQallUnAccessedMemSize(STaosQall *qall); int64_t taosQallUnAccessedMemSize(STaosQall *qall);
STaosQset *taosOpenQset(); int32_t taosOpenQset(STaosQset **qset);
void taosCloseQset(STaosQset *qset); void taosCloseQset(STaosQset *qset);
void taosQsetThreadResume(STaosQset *qset); void taosQsetThreadResume(STaosQset *qset);
int32_t taosAddIntoQset(STaosQset *qset, STaosQueue *queue, void *ahandle); int32_t taosAddIntoQset(STaosQset *qset, STaosQueue *queue, void *ahandle);
void taosRemoveFromQset(STaosQset *qset, STaosQueue *queue); void taosRemoveFromQset(STaosQset *qset, STaosQueue *queue);
int32_t taosGetQueueNumber(STaosQset *qset); int32_t taosGetQueueNumber(STaosQset *qset);
int32_t taosReadQitemFromQset(STaosQset *qset, void **ppItem, SQueueInfo *qinfo); int32_t taosReadQitemFromQset(STaosQset *qset, void **ppItem, SQueueInfo *qinfo);
int32_t taosReadAllQitemsFromQset(STaosQset *qset, STaosQall *qall, SQueueInfo *qinfo); int32_t taosReadAllQitemsFromQset(STaosQset *qset, STaosQall *qall, SQueueInfo *qinfo);

350
source/client/src/clientMonitor.c
View File

@ -1,13 +1,13 @@
#include "clientMonitor.h" #include "clientMonitor.h"
#include "clientLog.h"
#include "os.h"
#include "tmisce.h"
#include "ttime.h"
#include "ttimer.h"
#include "tglobal.h"
#include "tqueue.h"
#include "cJSON.h" #include "cJSON.h"
#include "clientInt.h" #include "clientInt.h"
#include "clientLog.h"
#include "os.h"
#include "tglobal.h"
#include "tmisce.h"
#include "tqueue.h"
#include "ttime.h"
#include "ttimer.h"
SRWLatch monitorLock; SRWLatch monitorLock;
void* monitorTimer; void* monitorTimer;
@ -20,41 +20,41 @@ STaosQueue* monitorQueue;
SHashObj* monitorSlowLogHash; SHashObj* monitorSlowLogHash;
char tmpSlowLogPath[PATH_MAX] = {0}; char tmpSlowLogPath[PATH_MAX] = {0};
static int32_t getSlowLogTmpDir(char* tmpPath, int32_t size){ static int32_t getSlowLogTmpDir(char* tmpPath, int32_t size) {
if (tsTempDir == NULL) { if (tsTempDir == NULL) {
return -1; return -1;
} }
int ret = snprintf(tmpPath, size, "%s/tdengine_slow_log/", tsTempDir); int ret = snprintf(tmpPath, size, "%s/tdengine_slow_log/", tsTempDir);
if (ret < 0){ if (ret < 0) {
tscError("failed to get tmp path ret:%d", ret); tscError("failed to get tmp path ret:%d", ret);
return ret; return ret;
} }
return 0; return 0;
} }
static void processFileInTheEnd(TdFilePtr pFile, char* path){ static void processFileInTheEnd(TdFilePtr pFile, char* path) {
if(pFile == NULL){ if (pFile == NULL) {
return; return;
} }
if(taosFtruncateFile(pFile, 0) != 0){ if (taosFtruncateFile(pFile, 0) != 0) {
tscError("failed to truncate file:%s, errno:%d", path, errno); tscError("failed to truncate file:%s, errno:%d", path, errno);
return; return;
} }
if(taosUnLockFile(pFile) != 0){ if (taosUnLockFile(pFile) != 0) {
tscError("failed to unlock file:%s, errno:%d", path, errno); tscError("failed to unlock file:%s, errno:%d", path, errno);
return; return;
} }
if(taosCloseFile(&(pFile)) != 0){ if (taosCloseFile(&(pFile)) != 0) {
tscError("failed to close file:%s, errno:%d", path, errno); tscError("failed to close file:%s, errno:%d", path, errno);
return; return;
} }
if(taosRemoveFile(path) != 0){ if (taosRemoveFile(path) != 0) {
tscError("failed to remove file:%s, errno:%d", path, errno); tscError("failed to remove file:%s, errno:%d", path, errno);
return; return;
} }
} }
static void destroySlowLogClient(void* data){ static void destroySlowLogClient(void* data) {
if (data == NULL) { if (data == NULL) {
return; return;
} }
@ -63,40 +63,40 @@ static void destroySlowLogClient(void* data){
taosMemoryFree(slowLogClient); taosMemoryFree(slowLogClient);
} }
static void destroyMonitorClient(void* data){ static void destroyMonitorClient(void* data) {
if (data == NULL) { if (data == NULL) {
return; return;
} }
MonitorClient* pMonitor = *(MonitorClient**)data; MonitorClient* pMonitor = *(MonitorClient**)data;
if(pMonitor == NULL){ if (pMonitor == NULL) {
return; return;
} }
taosTmrStopA(&pMonitor->timer); taosTmrStopA(&pMonitor->timer);
taosHashCleanup(pMonitor->counters); taosHashCleanup(pMonitor->counters);
taos_collector_registry_destroy(pMonitor->registry); taos_collector_registry_destroy(pMonitor->registry);
// taos_collector_destroy(pMonitor->colector); // taos_collector_destroy(pMonitor->colector);
taosMemoryFree(pMonitor); taosMemoryFree(pMonitor);
} }
static void monitorFreeSlowLogData(void *paras) { static void monitorFreeSlowLogData(void* paras) {
MonitorSlowLogData* pData = (MonitorSlowLogData*)paras; MonitorSlowLogData* pData = (MonitorSlowLogData*)paras;
if (pData == NULL) { if (pData == NULL) {
return; return;
} }
taosMemoryFreeClear(pData->data); taosMemoryFreeClear(pData->data);
if (pData->type == SLOW_LOG_READ_BEGINNIG){ if (pData->type == SLOW_LOG_READ_BEGINNIG) {
taosMemoryFree(pData->fileName); taosMemoryFree(pData->fileName);
} }
} }
static void monitorFreeSlowLogDataEx(void *paras) { static void monitorFreeSlowLogDataEx(void* paras) {
monitorFreeSlowLogData(paras); monitorFreeSlowLogData(paras);
taosMemoryFree(paras); taosMemoryFree(paras);
} }
static SAppInstInfo* getAppInstByClusterId(int64_t clusterId) { static SAppInstInfo* getAppInstByClusterId(int64_t clusterId) {
void *p = taosHashGet(appInfo.pInstMapByClusterId, &clusterId, LONG_BYTES); void* p = taosHashGet(appInfo.pInstMapByClusterId, &clusterId, LONG_BYTES);
if(p == NULL){ if (p == NULL) {
tscError("failed to get app inst, clusterId:%" PRIx64, clusterId); tscError("failed to get app inst, clusterId:%" PRIx64, clusterId);
return NULL; return NULL;
} }
@ -111,21 +111,25 @@ static int32_t monitorReportAsyncCB(void* param, SDataBuf* pMsg, int32_t code) {
taosMemoryFree(pMsg->pData); taosMemoryFree(pMsg->pData);
taosMemoryFree(pMsg->pEpSet); taosMemoryFree(pMsg->pEpSet);
} }
if(param != NULL){ if (param != NULL) {
MonitorSlowLogData* p = (MonitorSlowLogData*)param; MonitorSlowLogData* p = (MonitorSlowLogData*)param;
if(code != 0){ if (code != 0) {
tscError("failed to send slow log:%s, clusterId:%" PRIx64, p->data, p->clusterId); tscError("failed to send slow log:%s, clusterId:%" PRIx64, p->data, p->clusterId);
} }
MonitorSlowLogData tmp = {.clusterId = p->clusterId, .type = p->type, .fileName = p->fileName, MonitorSlowLogData tmp = {.clusterId = p->clusterId,
.pFile= p->pFile, .offset = p->offset, .data = NULL}; .type = p->type,
if(monitorPutData2MonitorQueue(tmp) == 0){ .fileName = p->fileName,
.pFile = p->pFile,
.offset = p->offset,
.data = NULL};
if (monitorPutData2MonitorQueue(tmp) == 0) {
p->fileName = NULL; p->fileName = NULL;
} }
} }
return code; return code;
} }
static int32_t sendReport(void* pTransporter, SEpSet *epSet, char* pCont, MONITOR_TYPE type, void* param) { static int32_t sendReport(void* pTransporter, SEpSet* epSet, char* pCont, MONITOR_TYPE type, void* param) {
SStatisReq sStatisReq; SStatisReq sStatisReq;
sStatisReq.pCont = pCont; sStatisReq.pCont = pCont;
sStatisReq.contLen = strlen(pCont); sStatisReq.contLen = strlen(pCont);
@ -167,11 +171,11 @@ FAILED:
return -1; return -1;
} }
static void generateClusterReport(taos_collector_registry_t* registry, void* pTransporter, SEpSet *epSet) { static void generateClusterReport(taos_collector_registry_t* registry, void* pTransporter, SEpSet* epSet) {
char ts[50] = {0}; char ts[50] = {0};
sprintf(ts, "%" PRId64, taosGetTimestamp(TSDB_TIME_PRECISION_MILLI)); sprintf(ts, "%" PRId64, taosGetTimestamp(TSDB_TIME_PRECISION_MILLI));
char* pCont = (char*)taos_collector_registry_bridge_new(registry, ts, "%" PRId64, NULL); char* pCont = (char*)taos_collector_registry_bridge_new(registry, ts, "%" PRId64, NULL);
if(NULL == pCont) { if (NULL == pCont) {
tscError("generateClusterReport failed, get null content."); tscError("generateClusterReport failed, get null content.");
return; return;
} }
@ -190,8 +194,8 @@ static void reportSendProcess(void* param, void* tmrId) {
} }
MonitorClient* pMonitor = (MonitorClient*)param; MonitorClient* pMonitor = (MonitorClient*)param;
SAppInstInfo* pInst = getAppInstByClusterId(pMonitor->clusterId); SAppInstInfo* pInst = getAppInstByClusterId(pMonitor->clusterId);
if(pInst == NULL){ if (pInst == NULL) {
taosRUnLockLatch(&monitorLock); taosRUnLockLatch(&monitorLock);
return; return;
} }
@ -202,15 +206,15 @@ static void reportSendProcess(void* param, void* tmrId) {
taosRUnLockLatch(&monitorLock); taosRUnLockLatch(&monitorLock);
} }
static void sendAllCounter(){ static void sendAllCounter() {
MonitorClient** ppMonitor = NULL; MonitorClient** ppMonitor = NULL;
while ((ppMonitor = taosHashIterate(monitorSlowLogHash, ppMonitor))) { while ((ppMonitor = taosHashIterate(monitorSlowLogHash, ppMonitor))) {
MonitorClient* pMonitor = *ppMonitor; MonitorClient* pMonitor = *ppMonitor;
if (pMonitor == NULL){ if (pMonitor == NULL) {
continue; continue;
} }
SAppInstInfo* pInst = getAppInstByClusterId(pMonitor->clusterId); SAppInstInfo* pInst = getAppInstByClusterId(pMonitor->clusterId);
if(pInst == NULL){ if (pInst == NULL) {
taosHashCancelIterate(monitorCounterHash, ppMonitor); taosHashCancelIterate(monitorCounterHash, ppMonitor);
break; break;
} }
@ -231,46 +235,48 @@ void monitorCreateClient(int64_t clusterId) {
} }
pMonitor->clusterId = clusterId; pMonitor->clusterId = clusterId;
char clusterKey[32] = {0}; char clusterKey[32] = {0};
if(snprintf(clusterKey, sizeof(clusterKey), "%"PRId64, clusterId) < 0){ if (snprintf(clusterKey, sizeof(clusterKey), "%" PRId64, clusterId) < 0) {
tscError("failed to create cluster key"); tscError("failed to create cluster key");
goto fail; goto fail;
} }
pMonitor->registry = taos_collector_registry_new(clusterKey); pMonitor->registry = taos_collector_registry_new(clusterKey);
if(pMonitor->registry == NULL){ if (pMonitor->registry == NULL) {
tscError("failed to create registry"); tscError("failed to create registry");
goto fail; goto fail;
} }
pMonitor->colector = taos_collector_new(clusterKey); pMonitor->colector = taos_collector_new(clusterKey);
if(pMonitor->colector == NULL){ if (pMonitor->colector == NULL) {
tscError("failed to create collector"); tscError("failed to create collector");
goto fail; goto fail;
} }
taos_collector_registry_register_collector(pMonitor->registry, pMonitor->colector); taos_collector_registry_register_collector(pMonitor->registry, pMonitor->colector);
pMonitor->counters = (SHashObj*)taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK); pMonitor->counters =
(SHashObj*)taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK);
if (pMonitor->counters == NULL) { if (pMonitor->counters == NULL) {
tscError("failed to create monitor counters"); tscError("failed to create monitor counters");
goto fail; goto fail;
} }
// taosHashSetFreeFp(pMonitor->counters, destroyCounter); // taosHashSetFreeFp(pMonitor->counters, destroyCounter);
if(taosHashPut(monitorCounterHash, &clusterId, LONG_BYTES, &pMonitor, POINTER_BYTES) != 0){ if (taosHashPut(monitorCounterHash, &clusterId, LONG_BYTES, &pMonitor, POINTER_BYTES) != 0) {
tscError("failed to put monitor client to hash"); tscError("failed to put monitor client to hash");
goto fail; goto fail;
} }
SAppInstInfo* pInst = getAppInstByClusterId(clusterId); SAppInstInfo* pInst = getAppInstByClusterId(clusterId);
if(pInst == NULL){ if (pInst == NULL) {
tscError("failed to get app instance by cluster id"); tscError("failed to get app instance by cluster id");
pMonitor = NULL; pMonitor = NULL;
goto fail; goto fail;
} }
pMonitor->timer = taosTmrStart(reportSendProcess, pInst->monitorParas.tsMonitorInterval * 1000, (void*)pMonitor, monitorTimer); pMonitor->timer =
if(pMonitor->timer == NULL){ taosTmrStart(reportSendProcess, pInst->monitorParas.tsMonitorInterval * 1000, (void*)pMonitor, monitorTimer);
if (pMonitor->timer == NULL) {
tscError("failed to start timer"); tscError("failed to start timer");
goto fail; goto fail;
} }
tscInfo("[monitor] monitorCreateClient for %"PRIx64 "finished %p.", clusterId, pMonitor); tscInfo("[monitor] monitorCreateClient for %" PRIx64 "finished %p.", clusterId, pMonitor);
} }
taosWUnLockLatch(&monitorLock); taosWUnLockLatch(&monitorLock);
if (-1 != atomic_val_compare_exchange_32(&monitorFlag, -1, 0)) { if (-1 != atomic_val_compare_exchange_32(&monitorFlag, -1, 0)) {
@ -283,7 +289,8 @@ fail:
taosWUnLockLatch(&monitorLock); taosWUnLockLatch(&monitorLock);
} }
void monitorCreateClientCounter(int64_t clusterId, const char* name, const char* help, size_t label_key_count, const char** label_keys) { void monitorCreateClientCounter(int64_t clusterId, const char* name, const char* help, size_t label_key_count,
const char** label_keys) {
taosWLockLatch(&monitorLock); taosWLockLatch(&monitorLock);
MonitorClient** ppMonitor = (MonitorClient**)taosHashGet(monitorCounterHash, &clusterId, LONG_BYTES); MonitorClient** ppMonitor = (MonitorClient**)taosHashGet(monitorCounterHash, &clusterId, LONG_BYTES);
if (ppMonitor == NULL || *ppMonitor == NULL) { if (ppMonitor == NULL || *ppMonitor == NULL) {
@ -291,16 +298,16 @@ void monitorCreateClientCounter(int64_t clusterId, const char* name, const char*
goto end; goto end;
} }
taos_counter_t* newCounter = taos_counter_new(name, help, label_key_count, label_keys); taos_counter_t* newCounter = taos_counter_new(name, help, label_key_count, label_keys);
if (newCounter == NULL) if (newCounter == NULL) return;
return; MonitorClient* pMonitor = *ppMonitor;
MonitorClient* pMonitor = *ppMonitor;
taos_collector_add_metric(pMonitor->colector, newCounter); taos_collector_add_metric(pMonitor->colector, newCounter);
if(taosHashPut(pMonitor->counters, name, strlen(name), &newCounter, POINTER_BYTES) != 0){ if (taosHashPut(pMonitor->counters, name, strlen(name), &newCounter, POINTER_BYTES) != 0) {
tscError("failed to put counter to monitor"); tscError("failed to put counter to monitor");
taos_counter_destroy(newCounter); taos_counter_destroy(newCounter);
goto end; goto end;
} }
tscInfo("[monitor] monitorCreateClientCounter %"PRIx64"(%p):%s : %p.", pMonitor->clusterId, pMonitor, name, newCounter); tscInfo("[monitor] monitorCreateClientCounter %" PRIx64 "(%p):%s : %p.", pMonitor->clusterId, pMonitor, name,
newCounter);
end: end:
taosWUnLockLatch(&monitorLock); taosWUnLockLatch(&monitorLock);
@ -315,18 +322,18 @@ void monitorCounterInc(int64_t clusterId, const char* counterName, const char**
MonitorClient** ppMonitor = (MonitorClient**)taosHashGet(monitorCounterHash, &clusterId, LONG_BYTES); MonitorClient** ppMonitor = (MonitorClient**)taosHashGet(monitorCounterHash, &clusterId, LONG_BYTES);
if (ppMonitor == NULL || *ppMonitor == NULL) { if (ppMonitor == NULL || *ppMonitor == NULL) {
tscError("monitorCounterInc not found pMonitor %"PRId64, clusterId); tscError("monitorCounterInc not found pMonitor %" PRId64, clusterId);
goto end; goto end;
} }
MonitorClient* pMonitor = *ppMonitor; MonitorClient* pMonitor = *ppMonitor;
taos_counter_t** ppCounter = (taos_counter_t**)taosHashGet(pMonitor->counters, counterName, strlen(counterName)); taos_counter_t** ppCounter = (taos_counter_t**)taosHashGet(pMonitor->counters, counterName, strlen(counterName));
if (ppCounter == NULL || *ppCounter == NULL) { if (ppCounter == NULL || *ppCounter == NULL) {
tscError("monitorCounterInc not found pCounter %"PRIx64":%s.", clusterId, counterName); tscError("monitorCounterInc not found pCounter %" PRIx64 ":%s.", clusterId, counterName);
goto end; goto end;
} }
taos_counter_inc(*ppCounter, label_values); taos_counter_inc(*ppCounter, label_values);
tscDebug("[monitor] monitorCounterInc %"PRIx64"(%p):%s", pMonitor->clusterId, pMonitor, counterName); tscDebug("[monitor] monitorCounterInc %" PRIx64 "(%p):%s", pMonitor->clusterId, pMonitor, counterName);
end: end:
taosWUnLockLatch(&monitorLock); taosWUnLockLatch(&monitorLock);
@ -339,13 +346,13 @@ const char* monitorResultStr(SQL_RESULT_CODE code) {
static void monitorThreadFuncUnexpectedStopped(void) { atomic_store_32(&slowLogFlag, -1); } static void monitorThreadFuncUnexpectedStopped(void) { atomic_store_32(&slowLogFlag, -1); }
static void monitorWriteSlowLog2File(MonitorSlowLogData* slowLogData, char *tmpPath){ static void monitorWriteSlowLog2File(MonitorSlowLogData* slowLogData, char* tmpPath) {
TdFilePtr pFile = NULL; TdFilePtr pFile = NULL;
void* tmp = taosHashGet(monitorSlowLogHash, &slowLogData->clusterId, LONG_BYTES); void* tmp = taosHashGet(monitorSlowLogHash, &slowLogData->clusterId, LONG_BYTES);
if (tmp == NULL){ if (tmp == NULL) {
char path[PATH_MAX] = {0}; char path[PATH_MAX] = {0};
char clusterId[32] = {0}; char clusterId[32] = {0};
if (snprintf(clusterId, sizeof(clusterId), "%" PRIx64, slowLogData->clusterId) < 0){ if (snprintf(clusterId, sizeof(clusterId), "%" PRIx64, slowLogData->clusterId) < 0) {
tscError("failed to generate clusterId:%" PRIx64, slowLogData->clusterId); tscError("failed to generate clusterId:%" PRIx64, slowLogData->clusterId);
return; return;
} }
@ -358,8 +365,8 @@ static void monitorWriteSlowLog2File(MonitorSlowLogData* slowLogData, char *tmpP
return; return;
} }
SlowLogClient *pClient = taosMemoryCalloc(1, sizeof(SlowLogClient)); SlowLogClient* pClient = taosMemoryCalloc(1, sizeof(SlowLogClient));
if (pClient == NULL){ if (pClient == NULL) {
tscError("failed to allocate memory for slow log client"); tscError("failed to allocate memory for slow log client");
taosCloseFile(&pFile); taosCloseFile(&pFile);
return; return;
@ -368,58 +375,58 @@ static void monitorWriteSlowLog2File(MonitorSlowLogData* slowLogData, char *tmpP
strcpy(pClient->path, path); strcpy(pClient->path, path);
pClient->offset = 0; pClient->offset = 0;
pClient->pFile = pFile; pClient->pFile = pFile;
if (taosHashPut(monitorSlowLogHash, &slowLogData->clusterId, LONG_BYTES, &pClient, POINTER_BYTES) != 0){ if (taosHashPut(monitorSlowLogHash, &slowLogData->clusterId, LONG_BYTES, &pClient, POINTER_BYTES) != 0) {
tscError("failed to put clusterId:%" PRId64 " to hash table", slowLogData->clusterId); tscError("failed to put clusterId:%" PRId64 " to hash table", slowLogData->clusterId);
taosCloseFile(&pFile); taosCloseFile(&pFile);
taosMemoryFree(pClient); taosMemoryFree(pClient);
return; return;
} }
if(taosLockFile(pFile) < 0){ if (taosLockFile(pFile) < 0) {
tscError("failed to lock file:%p since %s", pFile, terrstr()); tscError("failed to lock file:%p since %s", pFile, terrstr());
return; return;
} }
}else{ } else {
pFile = (*(SlowLogClient**)tmp)->pFile; pFile = (*(SlowLogClient**)tmp)->pFile;
} }
if(taosLSeekFile(pFile, 0, SEEK_END) < 0){ if (taosLSeekFile(pFile, 0, SEEK_END) < 0) {
tscError("failed to seek file:%p code: %d", pFile, errno); tscError("failed to seek file:%p code: %d", pFile, errno);
return; return;
} }
if (taosWriteFile(pFile, slowLogData->data, strlen(slowLogData->data) + 1) < 0){ if (taosWriteFile(pFile, slowLogData->data, strlen(slowLogData->data) + 1) < 0) {
tscError("failed to write len to file:%p since %s", pFile, terrstr()); tscError("failed to write len to file:%p since %s", pFile, terrstr());
} }
tscDebug("[monitor] write slow log to file:%p, clusterId:%"PRIx64, pFile, slowLogData->clusterId); tscDebug("[monitor] write slow log to file:%p, clusterId:%" PRIx64, pFile, slowLogData->clusterId);
} }
static char* readFile(TdFilePtr pFile, int64_t *offset, int64_t size){ static char* readFile(TdFilePtr pFile, int64_t* offset, int64_t size) {
tscDebug("[monitor] readFile slow begin pFile:%p, offset:%"PRId64 ", size:%"PRId64, pFile, *offset, size); tscDebug("[monitor] readFile slow begin pFile:%p, offset:%" PRId64 ", size:%" PRId64, pFile, *offset, size);
if(taosLSeekFile(pFile, *offset, SEEK_SET) < 0){ if (taosLSeekFile(pFile, *offset, SEEK_SET) < 0) {
tscError("failed to seek file:%p code: %d", pFile, errno); tscError("failed to seek file:%p code: %d", pFile, errno);
return NULL; return NULL;
} }
ASSERT(size > *offset); ASSERT(size > *offset);
char* pCont = NULL; char* pCont = NULL;
int64_t totalSize = 0; int64_t totalSize = 0;
if (size - *offset >= SLOW_LOG_SEND_SIZE_MAX) { if (size - *offset >= SLOW_LOG_SEND_SIZE_MAX) {
pCont = taosMemoryCalloc(1, 4 + SLOW_LOG_SEND_SIZE_MAX); //4 reserved for [] pCont = taosMemoryCalloc(1, 4 + SLOW_LOG_SEND_SIZE_MAX); // 4 reserved for []
totalSize = 4 + SLOW_LOG_SEND_SIZE_MAX; totalSize = 4 + SLOW_LOG_SEND_SIZE_MAX;
}else{ } else {
pCont = taosMemoryCalloc(1, 4 + (size - *offset)); pCont = taosMemoryCalloc(1, 4 + (size - *offset));
totalSize = 4 + (size - *offset); totalSize = 4 + (size - *offset);
} }
if(pCont == NULL){ if (pCont == NULL) {
tscError("failed to allocate memory for slow log, size:%" PRId64, totalSize); tscError("failed to allocate memory for slow log, size:%" PRId64, totalSize);
return NULL; return NULL;
} }
char* buf = pCont; char* buf = pCont;
strcat(buf++, "["); strcat(buf++, "[");
int64_t readSize = taosReadFile(pFile, buf, SLOW_LOG_SEND_SIZE_MAX); int64_t readSize = taosReadFile(pFile, buf, SLOW_LOG_SEND_SIZE_MAX);
if (readSize <= 0) { if (readSize <= 0) {
if (readSize < 0){ if (readSize < 0) {
tscError("failed to read len from file:%p since %s", pFile, terrstr()); tscError("failed to read len from file:%p since %s", pFile, terrstr());
} }
taosMemoryFree(pCont); taosMemoryFree(pCont);
@ -427,24 +434,24 @@ static char* readFile(TdFilePtr pFile, int64_t *offset, int64_t size){
} }
totalSize = 0; totalSize = 0;
while(1){ while (1) {
size_t len = strlen(buf); size_t len = strlen(buf);
totalSize += (len+1); totalSize += (len + 1);
if (totalSize > readSize || len == 0) { if (totalSize > readSize || len == 0) {
*(buf-1) = ']'; *(buf - 1) = ']';
*buf = '\0'; *buf = '\0';
break; break;
} }
buf[len] = ','; // replace '\0' with ',' buf[len] = ','; // replace '\0' with ','
buf += (len + 1); buf += (len + 1);
*offset += (len+1); *offset += (len + 1);
} }
tscDebug("[monitor] readFile slow log end, data:%s, offset:%"PRId64, pCont, *offset); tscDebug("[monitor] readFile slow log end, data:%s, offset:%" PRId64, pCont, *offset);
return pCont; return pCont;
} }
static int64_t getFileSize(char* path){ static int64_t getFileSize(char* path) {
int64_t fileSize = 0; int64_t fileSize = 0;
if (taosStatFile(path, &fileSize, NULL, NULL) < 0) { if (taosStatFile(path, &fileSize, NULL, NULL) < 0) {
return -1; return -1;
@ -453,13 +460,14 @@ static int64_t getFileSize(char* path){
return fileSize; return fileSize;
} }
static int32_t sendSlowLog(int64_t clusterId, char* data, TdFilePtr pFile, int64_t offset, SLOW_LOG_QUEUE_TYPE type, char* fileName, void* pTransporter, SEpSet *epSet){ static int32_t sendSlowLog(int64_t clusterId, char* data, TdFilePtr pFile, int64_t offset, SLOW_LOG_QUEUE_TYPE type,
if (data == NULL){ char* fileName, void* pTransporter, SEpSet* epSet) {
if (data == NULL) {
taosMemoryFree(fileName); taosMemoryFree(fileName);
return -1; return -1;
} }
MonitorSlowLogData* pParam = taosMemoryMalloc(sizeof(MonitorSlowLogData)); MonitorSlowLogData* pParam = taosMemoryMalloc(sizeof(MonitorSlowLogData));
if(pParam == NULL){ if (pParam == NULL) {
taosMemoryFree(data); taosMemoryFree(data);
taosMemoryFree(fileName); taosMemoryFree(fileName);
return -1; return -1;
@ -473,121 +481,124 @@ static int32_t sendSlowLog(int64_t clusterId, char* data, TdFilePtr pFile, int64
return sendReport(pTransporter, epSet, data, MONITOR_TYPE_SLOW_LOG, pParam); return sendReport(pTransporter, epSet, data, MONITOR_TYPE_SLOW_LOG, pParam);
} }
static int32_t monitorReadSend(int64_t clusterId, TdFilePtr pFile, int64_t* offset, int64_t size, SLOW_LOG_QUEUE_TYPE type, char* fileName){ static int32_t monitorReadSend(int64_t clusterId, TdFilePtr pFile, int64_t* offset, int64_t size,
SLOW_LOG_QUEUE_TYPE type, char* fileName) {
SAppInstInfo* pInst = getAppInstByClusterId(clusterId); SAppInstInfo* pInst = getAppInstByClusterId(clusterId);
if(pInst == NULL){ if (pInst == NULL) {
tscError("failed to get app instance by clusterId:%" PRId64, clusterId); tscError("failed to get app instance by clusterId:%" PRId64, clusterId);
return -1; return -1;
} }
SEpSet ep = getEpSet_s(&pInst->mgmtEp); SEpSet ep = getEpSet_s(&pInst->mgmtEp);
char* data = readFile(pFile, offset, size); char* data = readFile(pFile, offset, size);
return sendSlowLog(clusterId, data, (type == SLOW_LOG_READ_BEGINNIG ? pFile : NULL), *offset, type, fileName, pInst->pTransporter, &ep); return sendSlowLog(clusterId, data, (type == SLOW_LOG_READ_BEGINNIG ? pFile : NULL), *offset, type, fileName,
pInst->pTransporter, &ep);
} }
static void monitorSendSlowLogAtBeginning(int64_t clusterId, char** fileName, TdFilePtr pFile, int64_t offset){ static void monitorSendSlowLogAtBeginning(int64_t clusterId, char** fileName, TdFilePtr pFile, int64_t offset) {
int64_t size = getFileSize(*fileName); int64_t size = getFileSize(*fileName);
if(size <= offset){ if (size <= offset) {
processFileInTheEnd(pFile, *fileName); processFileInTheEnd(pFile, *fileName);
tscDebug("[monitor] monitorSendSlowLogAtBeginning delete file:%s", *fileName); tscDebug("[monitor] monitorSendSlowLogAtBeginning delete file:%s", *fileName);
}else{ } else {
int32_t code = monitorReadSend(clusterId, pFile, &offset, size, SLOW_LOG_READ_BEGINNIG, *fileName); int32_t code = monitorReadSend(clusterId, pFile, &offset, size, SLOW_LOG_READ_BEGINNIG, *fileName);
tscDebug("[monitor] monitorSendSlowLogAtBeginning send slow log clusterId:%"PRId64",ret:%d", clusterId, code); tscDebug("[monitor] monitorSendSlowLogAtBeginning send slow log clusterId:%" PRId64 ",ret:%d", clusterId, code);
*fileName = NULL; *fileName = NULL;
} }
} }
static void monitorSendSlowLogAtRunning(int64_t clusterId){ static void monitorSendSlowLogAtRunning(int64_t clusterId) {
void* tmp = taosHashGet(monitorSlowLogHash, &clusterId, LONG_BYTES); void* tmp = taosHashGet(monitorSlowLogHash, &clusterId, LONG_BYTES);
if (tmp == NULL){ if (tmp == NULL) {
return; return;
} }
SlowLogClient* pClient = (*(SlowLogClient**)tmp); SlowLogClient* pClient = (*(SlowLogClient**)tmp);
if (pClient == NULL){ if (pClient == NULL) {
return; return;
} }
int64_t size = getFileSize(pClient->path); int64_t size = getFileSize(pClient->path);
if(size <= pClient->offset){ if (size <= pClient->offset) {
if(taosFtruncateFile(pClient->pFile, 0) < 0){ if (taosFtruncateFile(pClient->pFile, 0) < 0) {
tscError("failed to truncate file:%p code: %d", pClient->pFile, errno); tscError("failed to truncate file:%p code: %d", pClient->pFile, errno);
} }
tscDebug("[monitor] monitorSendSlowLogAtRunning truncate file to 0 file:%p", pClient->pFile); tscDebug("[monitor] monitorSendSlowLogAtRunning truncate file to 0 file:%p", pClient->pFile);
pClient->offset = 0; pClient->offset = 0;
}else{ } else {
int32_t code = monitorReadSend(clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_RUNNING, NULL); int32_t code = monitorReadSend(clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_RUNNING, NULL);
tscDebug("[monitor] monitorSendSlowLogAtRunning send slow log clusterId:%"PRId64",ret:%d", clusterId, code); tscDebug("[monitor] monitorSendSlowLogAtRunning send slow log clusterId:%" PRId64 ",ret:%d", clusterId, code);
} }
} }
static bool monitorSendSlowLogAtQuit(int64_t clusterId) { static bool monitorSendSlowLogAtQuit(int64_t clusterId) {
void* tmp = taosHashGet(monitorSlowLogHash, &clusterId, LONG_BYTES); void* tmp = taosHashGet(monitorSlowLogHash, &clusterId, LONG_BYTES);
if (tmp == NULL){ if (tmp == NULL) {
return true; return true;
} }
SlowLogClient* pClient = (*(SlowLogClient**)tmp); SlowLogClient* pClient = (*(SlowLogClient**)tmp);
if (pClient == NULL){ if (pClient == NULL) {
return true; return true;
} }
int64_t size = getFileSize(pClient->path); int64_t size = getFileSize(pClient->path);
if(size <= pClient->offset){ if (size <= pClient->offset) {
processFileInTheEnd(pClient->pFile, pClient->path); processFileInTheEnd(pClient->pFile, pClient->path);
pClient->pFile = NULL; pClient->pFile = NULL;
tscInfo("[monitor] monitorSendSlowLogAtQuit remove file:%s", pClient->path); tscInfo("[monitor] monitorSendSlowLogAtQuit remove file:%s", pClient->path);
if((--quitCnt) == 0){ if ((--quitCnt) == 0) {
return true; return true;
} }
}else{ } else {
int32_t code = monitorReadSend(clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_QUIT, NULL); int32_t code = monitorReadSend(clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_QUIT, NULL);
tscDebug("[monitor] monitorSendSlowLogAtQuit send slow log clusterId:%"PRId64",ret:%d", clusterId, code); tscDebug("[monitor] monitorSendSlowLogAtQuit send slow log clusterId:%" PRId64 ",ret:%d", clusterId, code);
} }
return false; return false;
} }
static void monitorSendAllSlowLogAtQuit(){ static void monitorSendAllSlowLogAtQuit() {
void* pIter = NULL; void* pIter = NULL;
while ((pIter = taosHashIterate(monitorSlowLogHash, pIter))) { while ((pIter = taosHashIterate(monitorSlowLogHash, pIter))) {
SlowLogClient* pClient = (*(SlowLogClient**)pIter); SlowLogClient* pClient = (*(SlowLogClient**)pIter);
if(pClient == NULL) { if (pClient == NULL) {
continue; continue;
} }
int64_t size = getFileSize(pClient->path); int64_t size = getFileSize(pClient->path);
if(size <= pClient->offset){ if (size <= pClient->offset) {
processFileInTheEnd(pClient->pFile, pClient->path); processFileInTheEnd(pClient->pFile, pClient->path);
pClient->pFile = NULL; pClient->pFile = NULL;
}else if(pClient->offset == 0){ } else if (pClient->offset == 0) {
int64_t* clusterId = (int64_t*)taosHashGetKey(pIter, NULL); int64_t* clusterId = (int64_t*)taosHashGetKey(pIter, NULL);
int32_t code = monitorReadSend(*clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_QUIT, NULL); int32_t code = monitorReadSend(*clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_QUIT, NULL);
tscDebug("[monitor] monitorSendAllSlowLogAtQuit send slow log clusterId:%"PRId64",ret:%d", *clusterId, code); tscDebug("[monitor] monitorSendAllSlowLogAtQuit send slow log clusterId:%" PRId64 ",ret:%d", *clusterId, code);
if (code == 0){ if (code == 0) {
quitCnt ++; quitCnt++;
} }
} }
} }
} }
static void processFileRemoved(SlowLogClient* pClient){ static void processFileRemoved(SlowLogClient* pClient) {
taosUnLockFile(pClient->pFile); taosUnLockFile(pClient->pFile);
taosCloseFile(&(pClient->pFile)); taosCloseFile(&(pClient->pFile));
TdFilePtr pFile = taosOpenFile(pClient->path, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND | TD_FILE_READ | TD_FILE_TRUNC); TdFilePtr pFile =
taosOpenFile(pClient->path, TD_FILE_CREATE | TD_FILE_WRITE | TD_FILE_APPEND | TD_FILE_READ | TD_FILE_TRUNC);
if (pFile == NULL) { if (pFile == NULL) {
terrno = TAOS_SYSTEM_ERROR(errno); terrno = TAOS_SYSTEM_ERROR(errno);
tscError("failed to open file:%s since %s", pClient->path, terrstr()); tscError("failed to open file:%s since %s", pClient->path, terrstr());
}else{ } else {
pClient->pFile = pFile; pClient->pFile = pFile;
} }
} }
static void monitorSendAllSlowLog(){ static void monitorSendAllSlowLog() {
int64_t t = taosGetMonoTimestampMs(); int64_t t = taosGetMonoTimestampMs();
void* pIter = NULL; void* pIter = NULL;
while ((pIter = taosHashIterate(monitorSlowLogHash, pIter))) { while ((pIter = taosHashIterate(monitorSlowLogHash, pIter))) {
int64_t* clusterId = (int64_t*)taosHashGetKey(pIter, NULL); int64_t* clusterId = (int64_t*)taosHashGetKey(pIter, NULL);
SAppInstInfo* pInst = getAppInstByClusterId(*clusterId); SAppInstInfo* pInst = getAppInstByClusterId(*clusterId);
SlowLogClient* pClient = (*(SlowLogClient**)pIter); SlowLogClient* pClient = (*(SlowLogClient**)pIter);
if (pClient == NULL){ if (pClient == NULL) {
taosHashCancelIterate(monitorSlowLogHash, pIter); taosHashCancelIterate(monitorSlowLogHash, pIter);
return; return;
} }
if (t - pClient->lastCheckTime > pInst->monitorParas.tsMonitorInterval * 1000){ if (t - pClient->lastCheckTime > pInst->monitorParas.tsMonitorInterval * 1000) {
pClient->lastCheckTime = t; pClient->lastCheckTime = t;
} else { } else {
continue; continue;
@ -595,35 +606,35 @@ static void monitorSendAllSlowLog(){
if (pInst != NULL && pClient->offset == 0) { if (pInst != NULL && pClient->offset == 0) {
int64_t size = getFileSize(pClient->path); int64_t size = getFileSize(pClient->path);
if(size <= 0){ if (size <= 0) {
if(size < 0){ if (size < 0) {
tscError("[monitor] monitorSendAllSlowLog failed to get file size:%s, err:%d", pClient->path, errno); tscError("[monitor] monitorSendAllSlowLog failed to get file size:%s, err:%d", pClient->path, errno);
if(errno == ENOENT){ if (errno == ENOENT) {
processFileRemoved(pClient); processFileRemoved(pClient);
} }
} }
continue; continue;
} }
int32_t code = monitorReadSend(*clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_RUNNING, NULL); int32_t code = monitorReadSend(*clusterId, pClient->pFile, &pClient->offset, size, SLOW_LOG_READ_RUNNING, NULL);
tscDebug("[monitor] monitorSendAllSlowLog send slow log clusterId:%"PRId64",ret:%d", *clusterId, code); tscDebug("[monitor] monitorSendAllSlowLog send slow log clusterId:%" PRId64 ",ret:%d", *clusterId, code);
} }
} }
} }
static void monitorSendAllSlowLogFromTempDir(int64_t clusterId){ static void monitorSendAllSlowLogFromTempDir(int64_t clusterId) {
SAppInstInfo* pInst = getAppInstByClusterId((int64_t)clusterId); SAppInstInfo* pInst = getAppInstByClusterId((int64_t)clusterId);
if(pInst == NULL || !pInst->monitorParas.tsEnableMonitor){ if (pInst == NULL || !pInst->monitorParas.tsEnableMonitor) {
tscInfo("[monitor] monitor is disabled, skip send slow log"); tscInfo("[monitor] monitor is disabled, skip send slow log");
return; return;
} }
char namePrefix[PATH_MAX] = {0}; char namePrefix[PATH_MAX] = {0};
if (snprintf(namePrefix, sizeof(namePrefix), "%s%"PRIx64, TD_TMP_FILE_PREFIX, clusterId) < 0) { if (snprintf(namePrefix, sizeof(namePrefix), "%s%" PRIx64, TD_TMP_FILE_PREFIX, clusterId) < 0) {
tscError("failed to generate slow log file name prefix"); tscError("failed to generate slow log file name prefix");
return; return;
} }
char tmpPath[PATH_MAX] = {0}; char tmpPath[PATH_MAX] = {0};
if (getSlowLogTmpDir(tmpPath, sizeof(tmpPath)) < 0) { if (getSlowLogTmpDir(tmpPath, sizeof(tmpPath)) < 0) {
return; return;
} }
@ -639,11 +650,9 @@ static void monitorSendAllSlowLogFromTempDir(int64_t clusterId){
continue; continue;
} }
char *name = taosGetDirEntryName(de); char* name = taosGetDirEntryName(de);
if (strcmp(name, ".") == 0 || if (strcmp(name, ".") == 0 || strcmp(name, "..") == 0 || strstr(name, namePrefix) == NULL) {
strcmp(name, "..") == 0 || tscInfo("skip file:%s, for cluster id:%" PRIx64, name, clusterId);
strstr(name, namePrefix) == NULL) {
tscInfo("skip file:%s, for cluster id:%"PRIx64, name, clusterId);
continue; continue;
} }
@ -659,7 +668,7 @@ static void monitorSendAllSlowLogFromTempDir(int64_t clusterId){
taosCloseFile(&pFile); taosCloseFile(&pFile);
continue; continue;
} }
char *tmp = taosStrdup(filename); char* tmp = taosStrdup(filename);
monitorSendSlowLogAtBeginning(clusterId, &tmp, pFile, 0); monitorSendSlowLogAtBeginning(clusterId, &tmp, pFile, 0);
taosMemoryFree(tmp); taosMemoryFree(tmp);
} }
@ -667,7 +676,7 @@ static void monitorSendAllSlowLogFromTempDir(int64_t clusterId){
taosCloseDir(&pDir); taosCloseDir(&pDir);
} }
static void* monitorThreadFunc(void *param){ static void* monitorThreadFunc(void* param) {
setThreadName("client-monitor-slowlog"); setThreadName("client-monitor-slowlog");
#ifdef WINDOWS #ifdef WINDOWS
@ -680,18 +689,18 @@ static void* monitorThreadFunc(void *param){
return NULL; return NULL;
} }
tscDebug("monitorThreadFunc start"); tscDebug("monitorThreadFunc start");
int64_t quitTime = 0; int64_t quitTime = 0;
while (1) { while (1) {
if (atomic_load_32(&slowLogFlag) > 0 > 0) { if (atomic_load_32(&slowLogFlag) > 0 > 0) {
if(quitCnt == 0){ if (quitCnt == 0) {
monitorSendAllSlowLogAtQuit(); monitorSendAllSlowLogAtQuit();
if(quitCnt == 0){ if (quitCnt == 0) {
tscInfo("monitorThreadFunc quit since no slow log to send"); tscInfo("monitorThreadFunc quit since no slow log to send");
break; break;
} }
quitTime = taosGetMonoTimestampMs(); quitTime = taosGetMonoTimestampMs();
} }
if(taosGetMonoTimestampMs() - quitTime > 500){ //quit at most 500ms if (taosGetMonoTimestampMs() - quitTime > 500) { // quit at most 500ms
tscInfo("monitorThreadFunc quit since timeout"); tscInfo("monitorThreadFunc quit since timeout");
break; break;
} }
@ -700,18 +709,19 @@ static void* monitorThreadFunc(void *param){
MonitorSlowLogData* slowLogData = NULL; MonitorSlowLogData* slowLogData = NULL;
taosReadQitem(monitorQueue, (void**)&slowLogData); taosReadQitem(monitorQueue, (void**)&slowLogData);
if (slowLogData != NULL) { if (slowLogData != NULL) {
if (slowLogData->type == SLOW_LOG_READ_BEGINNIG){ if (slowLogData->type == SLOW_LOG_READ_BEGINNIG) {
if(slowLogData->pFile != NULL){ if (slowLogData->pFile != NULL) {
monitorSendSlowLogAtBeginning(slowLogData->clusterId, &(slowLogData->fileName), slowLogData->pFile, slowLogData->offset); monitorSendSlowLogAtBeginning(slowLogData->clusterId, &(slowLogData->fileName), slowLogData->pFile,
}else{ slowLogData->offset);
} else {
monitorSendAllSlowLogFromTempDir(slowLogData->clusterId); monitorSendAllSlowLogFromTempDir(slowLogData->clusterId);
} }
} else if(slowLogData->type == SLOW_LOG_WRITE){ } else if (slowLogData->type == SLOW_LOG_WRITE) {
monitorWriteSlowLog2File(slowLogData, tmpSlowLogPath); monitorWriteSlowLog2File(slowLogData, tmpSlowLogPath);
} else if(slowLogData->type == SLOW_LOG_READ_RUNNING){ } else if (slowLogData->type == SLOW_LOG_READ_RUNNING) {
monitorSendSlowLogAtRunning(slowLogData->clusterId); monitorSendSlowLogAtRunning(slowLogData->clusterId);
} else if(slowLogData->type == SLOW_LOG_READ_QUIT){ } else if (slowLogData->type == SLOW_LOG_READ_QUIT) {
if(monitorSendSlowLogAtQuit(slowLogData->clusterId)){ if (monitorSendSlowLogAtQuit(slowLogData->clusterId)) {
tscInfo("monitorThreadFunc quit since all slow log sended"); tscInfo("monitorThreadFunc quit since all slow log sended");
monitorFreeSlowLogData(slowLogData); monitorFreeSlowLogData(slowLogData);
taosFreeQitem(slowLogData); taosFreeQitem(slowLogData);
@ -757,8 +767,11 @@ static void tscMonitorStop() {
} }
int32_t monitorInit() { int32_t monitorInit() {
int32_t code;
tscInfo("[monitor] tscMonitor init"); tscInfo("[monitor] tscMonitor init");
monitorCounterHash = (SHashObj*)taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_ENTRY_LOCK); monitorCounterHash =
(SHashObj*)taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_ENTRY_LOCK);
if (monitorCounterHash == NULL) { if (monitorCounterHash == NULL) {
tscError("failed to create monitorCounterHash"); tscError("failed to create monitorCounterHash");
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
@ -766,7 +779,8 @@ int32_t monitorInit() {
} }
taosHashSetFreeFp(monitorCounterHash, destroyMonitorClient); taosHashSetFreeFp(monitorCounterHash, destroyMonitorClient);
monitorSlowLogHash = (SHashObj*)taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_ENTRY_LOCK); monitorSlowLogHash =
(SHashObj*)taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_ENTRY_LOCK);
if (monitorSlowLogHash == NULL) { if (monitorSlowLogHash == NULL) {
tscError("failed to create monitorSlowLogHash"); tscError("failed to create monitorSlowLogHash");
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
@ -781,7 +795,7 @@ int32_t monitorInit() {
return -1; return -1;
} }
if (getSlowLogTmpDir(tmpSlowLogPath, sizeof(tmpSlowLogPath)) < 0){ if (getSlowLogTmpDir(tmpSlowLogPath, sizeof(tmpSlowLogPath)) < 0) {
terrno = TSDB_CODE_TSC_INTERNAL_ERROR; terrno = TSDB_CODE_TSC_INTERNAL_ERROR;
return -1; return -1;
} }
@ -798,14 +812,15 @@ int32_t monitorInit() {
return -1; return -1;
} }
monitorQueue = taosOpenQueue(); code = taosOpenQueue(&monitorQueue);
if(monitorQueue == NULL){ if (code) {
terrno = code;
tscError("open queue error since %s", terrstr()); tscError("open queue error since %s", terrstr());
return -1; return -1;
} }
taosInitRWLatch(&monitorLock); taosInitRWLatch(&monitorLock);
if (tscMonitortInit() != 0){ if (tscMonitortInit() != 0) {
return -1; return -1;
} }
return 0; return 0;
@ -828,21 +843,26 @@ void monitorClose() {
taosWUnLockLatch(&monitorLock); taosWUnLockLatch(&monitorLock);
} }
int32_t monitorPutData2MonitorQueue(MonitorSlowLogData data){ int32_t monitorPutData2MonitorQueue(MonitorSlowLogData data) {
int32_t code;
MonitorSlowLogData* slowLogData;
if (atomic_load_32(&slowLogFlag) == -2) { if (atomic_load_32(&slowLogFlag) == -2) {
tscError("[monitor] slow log thread is exiting"); tscError("[monitor] slow log thread is exiting");
return -1; return -1;
} }
MonitorSlowLogData* slowLogData = taosAllocateQitem(sizeof(MonitorSlowLogData), DEF_QITEM, 0);
if (slowLogData == NULL) { code = taosAllocateQitem(sizeof(MonitorSlowLogData), DEF_QITEM, 0, (void**)&slowLogData);
if (code) {
tscError("[monitor] failed to allocate slow log data"); tscError("[monitor] failed to allocate slow log data");
return -1; return terrno = code;
} }
*slowLogData = data; *slowLogData = data;
tscDebug("[monitor] write slow log to queue, clusterId:%"PRIx64 " type:%s, data:%s", slowLogData->clusterId, queueTypeStr[slowLogData->type], slowLogData->data); tscDebug("[monitor] write slow log to queue, clusterId:%" PRIx64 " type:%s, data:%s", slowLogData->clusterId,
if (taosWriteQitem(monitorQueue, slowLogData) == 0){ queueTypeStr[slowLogData->type], slowLogData->data);
if (taosWriteQitem(monitorQueue, slowLogData) == 0) {
tsem2_post(&monitorSem); tsem2_post(&monitorSem);
}else{ } else {
monitorFreeSlowLogData(slowLogData); monitorFreeSlowLogData(slowLogData);
taosFreeQitem(slowLogData); taosFreeQitem(slowLogData);
} }

12
source/client/src/clientRawBlockWrite.c
View File

@ -1116,9 +1116,9 @@ static int32_t taosCreateTable(TAOS* taos, void* meta, int32_t metaLen) {
if (taosHashGetSize(pVgroupHashmap) == 0) { if (taosHashGetSize(pVgroupHashmap) == 0) {
goto end; goto end;
} }
SArray* pBufArray = serializeVgroupsCreateTableBatch(pVgroupHashmap); SArray* pBufArray = NULL;
if (NULL == pBufArray) { code = serializeVgroupsCreateTableBatch(pVgroupHashmap, &pBufArray);
code = TSDB_CODE_OUT_OF_MEMORY; if (TSDB_CODE_SUCCESS != code) {
goto end; goto end;
} }
@ -1264,9 +1264,9 @@ static int32_t taosDropTable(TAOS* taos, void* meta, int32_t metaLen) {
if (taosHashGetSize(pVgroupHashmap) == 0) { if (taosHashGetSize(pVgroupHashmap) == 0) {
goto end; goto end;
} }
SArray* pBufArray = serializeVgroupsDropTableBatch(pVgroupHashmap); SArray* pBufArray = NULL;
if (NULL == pBufArray) { code = serializeVgroupsDropTableBatch(pVgroupHashmap, &pBufArray);
code = TSDB_CODE_OUT_OF_MEMORY; if (TSDB_CODE_SUCCESS != code) {
goto end; goto end;
} }

102
source/client/src/clientTmq.c
View File

@ -733,13 +733,17 @@ end:
} }
static void generateTimedTask(int64_t refId, int32_t type) { static void generateTimedTask(int64_t refId, int32_t type) {
tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, refId); tmq_t* tmq;
int8_t* pTaskType;
int32_t code;
tmq = taosAcquireRef(tmqMgmt.rsetId, refId);
if (tmq == NULL) return; if (tmq == NULL) return;
int8_t* pTaskType = taosAllocateQitem(sizeof(int8_t), DEF_QITEM, 0); code = taosAllocateQitem(sizeof(int8_t), DEF_QITEM, 0, (void**)&pTaskType);
if (pTaskType != NULL){ if (code == TSDB_CODE_SUCCESS) {
*pTaskType = type; *pTaskType = type;
if (taosWriteQitem(tmq->delayedTask, pTaskType) == 0){ if (taosWriteQitem(tmq->delayedTask, pTaskType) == 0) {
tsem2_post(&tmq->rspSem); tsem2_post(&tmq->rspSem);
} }
} }
@ -877,7 +881,7 @@ void tmqSendHbReq(void* param, void* tmrId) {
OVER: OVER:
tDestroySMqHbReq(&req); tDestroySMqHbReq(&req);
if(tmrId != NULL){ if (tmrId != NULL) {
taosTmrReset(tmqSendHbReq, DEFAULT_HEARTBEAT_INTERVAL, param, tmqMgmt.timer, &tmq->hbLiveTimer); taosTmrReset(tmqSendHbReq, DEFAULT_HEARTBEAT_INTERVAL, param, tmqMgmt.timer, &tmq->hbLiveTimer);
} }
taosReleaseRef(tmqMgmt.rsetId, refId); taosReleaseRef(tmqMgmt.rsetId, refId);
@ -890,7 +894,12 @@ static void defaultCommitCbFn(tmq_t* pTmq, int32_t code, void* param) {
} }
int32_t tmqHandleAllDelayedTask(tmq_t* pTmq) { int32_t tmqHandleAllDelayedTask(tmq_t* pTmq) {
STaosQall* qall = taosAllocateQall(); STaosQall* qall;
int32_t code;
code = taosAllocateQall(&qall);
if (code) return code;
taosReadAllQitems(pTmq->delayedTask, qall); taosReadAllQitems(pTmq->delayedTask, qall);
int32_t numOfItems = taosQallItemSize(qall); int32_t numOfItems = taosQallItemSize(qall);
@ -914,7 +923,8 @@ int32_t tmqHandleAllDelayedTask(tmq_t* pTmq) {
asyncCommitAllOffsets(pTmq, pCallbackFn, pTmq->commitCbUserParam); asyncCommitAllOffsets(pTmq, pCallbackFn, pTmq->commitCbUserParam);
tscDebug("consumer:0x%" PRIx64 " next commit to vnode(s) in %.2fs", pTmq->consumerId, tscDebug("consumer:0x%" PRIx64 " next commit to vnode(s) in %.2fs", pTmq->consumerId,
pTmq->autoCommitInterval / 1000.0); pTmq->autoCommitInterval / 1000.0);
taosTmrReset(tmqAssignDelayedCommitTask, pTmq->autoCommitInterval, (void*)(pTmq->refId), tmqMgmt.timer, &pTmq->commitTimer); taosTmrReset(tmqAssignDelayedCommitTask, pTmq->autoCommitInterval, (void*)(pTmq->refId), tmqMgmt.timer,
&pTmq->commitTimer);
} else { } else {
tscError("consumer:0x%" PRIx64 " invalid task type:%d", pTmq->consumerId, *pTaskType); tscError("consumer:0x%" PRIx64 " invalid task type:%d", pTmq->consumerId, *pTaskType);
} }
@ -977,14 +987,14 @@ void tmqClearUnhandleMsg(tmq_t* tmq) {
} }
int32_t tmqSubscribeCb(void* param, SDataBuf* pMsg, int32_t code) { int32_t tmqSubscribeCb(void* param, SDataBuf* pMsg, int32_t code) {
if(param == NULL) { if (param == NULL) {
return code; return code;
} }
SMqSubscribeCbParam* pParam = (SMqSubscribeCbParam*)param; SMqSubscribeCbParam* pParam = (SMqSubscribeCbParam*)param;
pParam->rspErr = code; pParam->rspErr = code;
if(pMsg){ if (pMsg) {
taosMemoryFree(pMsg->pEpSet); taosMemoryFree(pMsg->pEpSet);
} }
tsem_post(&pParam->rspSem); tsem_post(&pParam->rspSem);
@ -1057,13 +1067,13 @@ void tmqFreeImpl(void* handle) {
taosArrayDestroyEx(tmq->clientTopics, freeClientVgImpl); taosArrayDestroyEx(tmq->clientTopics, freeClientVgImpl);
taos_close_internal(tmq->pTscObj); taos_close_internal(tmq->pTscObj);
if(tmq->commitTimer) { if (tmq->commitTimer) {
taosTmrStopA(&tmq->commitTimer); taosTmrStopA(&tmq->commitTimer);
} }
if(tmq->epTimer) { if (tmq->epTimer) {
taosTmrStopA(&tmq->epTimer); taosTmrStopA(&tmq->epTimer);
} }
if(tmq->hbLiveTimer) { if (tmq->hbLiveTimer) {
taosTmrStopA(&tmq->hbLiveTimer); taosTmrStopA(&tmq->hbLiveTimer);
} }
taosMemoryFree(tmq); taosMemoryFree(tmq);
@ -1101,6 +1111,8 @@ void tmqMgmtClose(void) {
if (errstr != NULL) snprintf(errstr, errstrLen, MSG); if (errstr != NULL) snprintf(errstr, errstrLen, MSG);
tmq_t* tmq_consumer_new(tmq_conf_t* conf, char* errstr, int32_t errstrLen) { tmq_t* tmq_consumer_new(tmq_conf_t* conf, char* errstr, int32_t errstrLen) {
int32_t code;
if (conf == NULL) { if (conf == NULL) {
SET_ERROR_MSG_TMQ("configure is null") SET_ERROR_MSG_TMQ("configure is null")
return NULL; return NULL;
@ -1124,12 +1136,31 @@ tmq_t* tmq_consumer_new(tmq_conf_t* conf, char* errstr, int32_t errstrLen) {
const char* pass = conf->pass == NULL ? TSDB_DEFAULT_PASS : conf->pass; const char* pass = conf->pass == NULL ? TSDB_DEFAULT_PASS : conf->pass;
pTmq->clientTopics = taosArrayInit(0, sizeof(SMqClientTopic)); pTmq->clientTopics = taosArrayInit(0, sizeof(SMqClientTopic));
pTmq->mqueue = taosOpenQueue(); code = taosOpenQueue(&pTmq->mqueue);
pTmq->delayedTask = taosOpenQueue(); if (code) {
pTmq->qall = taosAllocateQall(); terrno = code;
tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, terrstr(), pTmq->groupId);
SET_ERROR_MSG_TMQ("open queue failed")
goto _failed;
}
if (pTmq->clientTopics == NULL || pTmq->mqueue == NULL || pTmq->qall == NULL || pTmq->delayedTask == NULL || code = taosOpenQueue(&pTmq->delayedTask);
conf->groupId[0] == 0) { if (code) {
terrno = code;
tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, terrstr(), pTmq->groupId);
SET_ERROR_MSG_TMQ("open delayed task queue failed")
goto _failed;
}
code = taosAllocateQall(&pTmq->qall);
if (code) {
terrno = code;
tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, terrstr(), pTmq->groupId);
SET_ERROR_MSG_TMQ("allocate qall failed")
goto _failed;
}
if (conf->groupId[0] == 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, terrstr(), pTmq->groupId); tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, terrstr(), pTmq->groupId);
SET_ERROR_MSG_TMQ("malloc tmq element failed or group is empty") SET_ERROR_MSG_TMQ("malloc tmq element failed or group is empty")
@ -1315,7 +1346,8 @@ int32_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) {
// init ep timer // init ep timer
tmq->epTimer = taosTmrStart(tmqAssignAskEpTask, DEFAULT_ASKEP_INTERVAL, (void*)(tmq->refId), tmqMgmt.timer); tmq->epTimer = taosTmrStart(tmqAssignAskEpTask, DEFAULT_ASKEP_INTERVAL, (void*)(tmq->refId), tmqMgmt.timer);
// init auto commit timer // init auto commit timer
tmq->commitTimer = taosTmrStart(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, (void*)(tmq->refId), tmqMgmt.timer); tmq->commitTimer =
taosTmrStart(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, (void*)(tmq->refId), tmqMgmt.timer);
FAIL: FAIL:
taosArrayDestroyP(req.topicNames, taosMemoryFree); taosArrayDestroyP(req.topicNames, taosMemoryFree);
@ -1372,20 +1404,20 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) {
if (pParam == NULL || pMsg == NULL) { if (pParam == NULL || pMsg == NULL) {
goto FAIL; goto FAIL;
} }
int64_t refId = pParam->refId; int64_t refId = pParam->refId;
int32_t vgId = pParam->vgId; int32_t vgId = pParam->vgId;
uint64_t requestId = pParam->requestId; uint64_t requestId = pParam->requestId;
tmq = taosAcquireRef(tmqMgmt.rsetId, refId); tmq = taosAcquireRef(tmqMgmt.rsetId, refId);
if (tmq == NULL) { if (tmq == NULL) {
code = TSDB_CODE_TMQ_CONSUMER_CLOSED; code = TSDB_CODE_TMQ_CONSUMER_CLOSED;
goto FAIL; goto FAIL;
} }
SMqPollRspWrapper* pRspWrapper = taosAllocateQitem(sizeof(SMqPollRspWrapper), DEF_QITEM, 0); SMqPollRspWrapper* pRspWrapper;
if (pRspWrapper == NULL) { code = taosAllocateQitem(sizeof(SMqPollRspWrapper), DEF_QITEM, 0, (void**)&pRspWrapper);
if (code) {
tscWarn("consumer:0x%" PRIx64 " msg discard from vgId:%d, since out of memory", tmq->consumerId, vgId); tscWarn("consumer:0x%" PRIx64 " msg discard from vgId:%d, since out of memory", tmq->consumerId, vgId);
taosReleaseRef(tmqMgmt.rsetId, refId); taosReleaseRef(tmqMgmt.rsetId, refId);
code = TSDB_CODE_OUT_OF_MEMORY;
goto FAIL; goto FAIL;
} }
@ -2575,7 +2607,7 @@ end:
} }
int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) { int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) {
if(param == NULL) return code; if (param == NULL) return code;
SMqAskEpCbParam* pParam = (SMqAskEpCbParam*)param; SMqAskEpCbParam* pParam = (SMqAskEpCbParam*)param;
tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, pParam->refId); tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, pParam->refId);
if (tmq == NULL) { if (tmq == NULL) {
@ -2600,9 +2632,9 @@ int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) {
doUpdateLocalEp(tmq, head->epoch, &rsp); doUpdateLocalEp(tmq, head->epoch, &rsp);
tDeleteSMqAskEpRsp(&rsp); tDeleteSMqAskEpRsp(&rsp);
} else { } else {
SMqAskEpRspWrapper* pWrapper = taosAllocateQitem(sizeof(SMqAskEpRspWrapper), DEF_QITEM, 0); SMqAskEpRspWrapper* pWrapper;
if (pWrapper == NULL) { code = taosAllocateQitem(sizeof(SMqAskEpRspWrapper), DEF_QITEM, 0, (void**)&pWrapper);
code = TSDB_CODE_OUT_OF_MEMORY; if (code) {
goto END; goto END;
} }
@ -2620,13 +2652,13 @@ END:
FAIL: FAIL:
if (pParam->sync) { if (pParam->sync) {
SAskEpInfo* pInfo = pParam->pParam; SAskEpInfo* pInfo = pParam->pParam;
if(pInfo) { if (pInfo) {
pInfo->code = code; pInfo->code = code;
tsem_post(&pInfo->sem); tsem_post(&pInfo->sem);
} }
} }
if(pMsg){ if (pMsg) {
taosMemoryFree(pMsg->pEpSet); taosMemoryFree(pMsg->pEpSet);
taosMemoryFree(pMsg->pData); taosMemoryFree(pMsg->pData);
} }
@ -2636,11 +2668,11 @@ FAIL:
int32_t syncAskEp(tmq_t* pTmq) { int32_t syncAskEp(tmq_t* pTmq) {
SAskEpInfo* pInfo = taosMemoryMalloc(sizeof(SAskEpInfo)); SAskEpInfo* pInfo = taosMemoryMalloc(sizeof(SAskEpInfo));
if(pInfo == NULL) return TSDB_CODE_OUT_OF_MEMORY; if (pInfo == NULL) return TSDB_CODE_OUT_OF_MEMORY;
tsem_init(&pInfo->sem, 0, 0); tsem_init(&pInfo->sem, 0, 0);
int32_t code = askEp(pTmq, pInfo, true, false); int32_t code = askEp(pTmq, pInfo, true, false);
if(code == 0){ if (code == 0) {
tsem_wait(&pInfo->sem); tsem_wait(&pInfo->sem);
code = pInfo->code; code = pInfo->code;
} }
@ -2778,7 +2810,7 @@ SReqResultInfo* tmqGetNextResInfo(TAOS_RES* res, bool convertUcs4) {
} }
static int32_t tmqGetWalInfoCb(void* param, SDataBuf* pMsg, int32_t code) { static int32_t tmqGetWalInfoCb(void* param, SDataBuf* pMsg, int32_t code) {
if(param == NULL) { if (param == NULL) {
return code; return code;
} }
SMqVgWalInfoParam* pParam = param; SMqVgWalInfoParam* pParam = param;
@ -2812,7 +2844,7 @@ static int32_t tmqGetWalInfoCb(void* param, SDataBuf* pMsg, int32_t code) {
tsem_post(&pCommon->rsp); tsem_post(&pCommon->rsp);
} }
if(pMsg){ if (pMsg) {
taosMemoryFree(pMsg->pData); taosMemoryFree(pMsg->pData);
taosMemoryFree(pMsg->pEpSet); taosMemoryFree(pMsg->pEpSet);
} }
@ -3248,7 +3280,7 @@ static int32_t tmqSeekCb(void* param, SDataBuf* pMsg, int32_t code) {
taosMemoryFree(pMsg->pData); taosMemoryFree(pMsg->pData);
taosMemoryFree(pMsg->pEpSet); taosMemoryFree(pMsg->pEpSet);
} }
if(param == NULL) { if (param == NULL) {
return code; return code;
} }
SMqSeekParam* pParam = param; SMqSeekParam* pParam = param;

5
source/common/src/tname.c
View File

@ -98,7 +98,7 @@ SName* toName(int32_t acctId, const char* pDbName, const char* pTableName, SName
int32_t tNameExtractFullName(const SName* name, char* dst) { int32_t tNameExtractFullName(const SName* name, char* dst) {
// invalid full name format, abort // invalid full name format, abort
if (!tNameIsValid(name)) { if (!tNameIsValid(name)) {
return -1; return TSDB_CODE_INVALID_PARA;
} }
int32_t len = snprintf(dst, TSDB_DB_FNAME_LEN, "%d.%s", name->acctId, name->dbname); int32_t len = snprintf(dst, TSDB_DB_FNAME_LEN, "%d.%s", name->acctId, name->dbname);
@ -154,8 +154,7 @@ int32_t tNameGetDbName(const SName* name, char* dst) {
const char* tNameGetDbNameP(const SName* name) { return &name->dbname[0]; } const char* tNameGetDbNameP(const SName* name) { return &name->dbname[0]; }
int32_t tNameGetFullDbName(const SName* name, char* dst) { int32_t tNameGetFullDbName(const SName* name, char* dst) {
snprintf(dst, TSDB_DB_FNAME_LEN, "%d.%s", name->acctId, name->dbname); return snprintf(dst, TSDB_DB_FNAME_LEN, "%d.%s", name->acctId, name->dbname);
return 0;
} }
bool tNameIsEmpty(const SName* name) { return name->type == 0 || name->acctId == 0; } bool tNameIsEmpty(const SName* name) { return name->type == 0 || name->acctId == 0; }

15
source/dnode/mgmt/mgmt_mnode/src/mmWorker.c
View File

@ -153,6 +153,8 @@ int32_t mmPutMsgToFetchQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) {
} }
int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
int32_t code;
SSingleWorker *pWorker = NULL; SSingleWorker *pWorker = NULL;
switch (qtype) { switch (qtype) {
case WRITE_QUEUE: case WRITE_QUEUE:
@ -177,18 +179,21 @@ int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
pWorker = &pMgmt->syncRdWorker; pWorker = &pMgmt->syncRdWorker;
break; break;
default: default:
terrno = TSDB_CODE_INVALID_PARA; code = TSDB_CODE_INVALID_PARA;
} }
if (pWorker == NULL) return -1; if (pWorker == NULL) return code;
SRpcMsg *pMsg = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM, pRpc->contLen);
if (pMsg == NULL) return -1; SRpcMsg *pMsg;
code = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM, pRpc->contLen, (void **)&pMsg);
if (code) return code;
memcpy(pMsg, pRpc, sizeof(SRpcMsg)); memcpy(pMsg, pRpc, sizeof(SRpcMsg));
pRpc->pCont = NULL; pRpc->pCont = NULL;
dTrace("msg:%p, is created and will put into %s queue, type:%s len:%d", pMsg, pWorker->name, TMSG_INFO(pRpc->msgType), dTrace("msg:%p, is created and will put into %s queue, type:%s len:%d", pMsg, pWorker->name, TMSG_INFO(pRpc->msgType),
pRpc->contLen); pRpc->contLen);
int32_t code = mmPutMsgToWorker(pMgmt, pWorker, pMsg); code = mmPutMsgToWorker(pMgmt, pWorker, pMsg);
if (code != 0) { if (code != 0) {
dTrace("msg:%p, is freed", pMsg); dTrace("msg:%p, is freed", pMsg);
rpcFreeCont(pMsg->pCont); rpcFreeCont(pMsg->pCont);

7
source/dnode/mgmt/mgmt_qnode/src/qmWorker.c
View File

@ -58,8 +58,11 @@ int32_t qmPutNodeMsgToFetchQueue(SQnodeMgmt *pMgmt, SRpcMsg *pMsg) {
} }
int32_t qmPutRpcMsgToQueue(SQnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { int32_t qmPutRpcMsgToQueue(SQnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
SRpcMsg *pMsg = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM, pRpc->contLen); int32_t code;
if (pMsg == NULL) return -1; SRpcMsg *pMsg;
code = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM, pRpc->contLen, (void **)&pMsg);
if (code) return code;
memcpy(pMsg, pRpc, sizeof(SRpcMsg)); memcpy(pMsg, pRpc, sizeof(SRpcMsg));
pRpc->pCont = NULL; pRpc->pCont = NULL;

8
source/dnode/mgmt/mgmt_snode/src/smWorker.c
View File

@ -127,9 +127,11 @@ void smStopWorker(SSnodeMgmt *pMgmt) {
} }
int32_t smPutMsgToQueue(SSnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { int32_t smPutMsgToQueue(SSnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
int32_t code = 0; int32_t code;
SRpcMsg *pMsg = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM, pRpc->contLen); SRpcMsg *pMsg;
if (pMsg == NULL) {
code = taosAllocateQitem(sizeof(SRpcMsg), RPC_QITEM, pRpc->contLen, (void **)&pMsg);
if (code) {
rpcFreeCont(pRpc->pCont); rpcFreeCont(pRpc->pCont);
pRpc->pCont = NULL; pRpc->pCont = NULL;
return code = terrno; return code = terrno;

8
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
View File

@ -312,7 +312,7 @@ int32_t vmPutMsgToMgmtQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
} }
int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) { int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
int32_t code = 0; int32_t code;
if (pRpc->contLen < sizeof(SMsgHead)) { if (pRpc->contLen < sizeof(SMsgHead)) {
dError("invalid rpc msg with no msg head at pCont. pRpc:%p, type:%s, len:%d", pRpc, TMSG_INFO(pRpc->msgType), dError("invalid rpc msg with no msg head at pCont. pRpc:%p, type:%s, len:%d", pRpc, TMSG_INFO(pRpc->msgType),
pRpc->contLen); pRpc->contLen);
@ -322,9 +322,9 @@ int32_t vmPutRpcMsgToQueue(SVnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
} }
EQItype itype = APPLY_QUEUE == qtype ? DEF_QITEM : RPC_QITEM; EQItype itype = APPLY_QUEUE == qtype ? DEF_QITEM : RPC_QITEM;
SRpcMsg *pMsg = NULL; SRpcMsg *pMsg;
code = taosAllocateQitemWrapper(sizeof(SRpcMsg), itype, pRpc->contLen, (void **)&pMsg); code = taosAllocateQitem(sizeof(SRpcMsg), itype, pRpc->contLen, (void **)&pMsg);
if (code != 0) { if (code) {
rpcFreeCont(pRpc->pCont); rpcFreeCont(pRpc->pCont);
pRpc->pCont = NULL; pRpc->pCont = NULL;
return code; return code;

18
source/dnode/mgmt/node_mgmt/src/dmTransport.c
View File

@ -211,10 +211,8 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
pRpc->info.wrapper = pWrapper; pRpc->info.wrapper = pWrapper;
EQItype itype = IsReq(pRpc) ? RPC_QITEM : DEF_QITEM; // rsp msg is not restricted by tsRpcQueueMemoryUsed EQItype itype = IsReq(pRpc) ? RPC_QITEM : DEF_QITEM; // rsp msg is not restricted by tsRpcQueueMemoryUsed
code = taosAllocateQitemWrapper(sizeof(SRpcMsg), itype, pRpc->contLen, (void **)&pMsg); code = taosAllocateQitem(sizeof(SRpcMsg), itype, pRpc->contLen, (void **)&pMsg);
if (code != 0) { if (code) goto _OVER;
goto _OVER;
}
memcpy(pMsg, pRpc, sizeof(SRpcMsg)); memcpy(pMsg, pRpc, sizeof(SRpcMsg));
dGTrace("msg:%p, is created, type:%s handle:%p len:%d", pMsg, TMSG_INFO(pRpc->msgType), pMsg->info.handle, dGTrace("msg:%p, is created, type:%s handle:%p len:%d", pMsg, TMSG_INFO(pRpc->msgType), pMsg->info.handle,
@ -228,7 +226,7 @@ _OVER:
if (pMsg) { if (pMsg) {
dGTrace("msg:%p, failed to process %s since %s", pMsg, TMSG_INFO(pMsg->msgType), tstrerror(code)); dGTrace("msg:%p, failed to process %s since %s", pMsg, TMSG_INFO(pMsg->msgType), tstrerror(code));
} else { } else {
dGTrace("msg:%p, failed to process empty msg since %s", pMsg, terrstr()); dGTrace("msg:%p, failed to process empty msg since %s", pMsg, tstrerror(code));
} }
if (IsReq(pRpc)) { if (IsReq(pRpc)) {
@ -302,17 +300,19 @@ static inline int32_t dmSendReq(const SEpSet *pEpSet, SRpcMsg *pMsg) {
} }
} }
static inline int32_t dmSendSyncReq(const SEpSet *pEpSet, SRpcMsg *pMsg) { static inline int32_t dmSendSyncReq(const SEpSet *pEpSet, SRpcMsg *pMsg) {
int32_t code = 0;
SDnode *pDnode = dmInstance(); SDnode *pDnode = dmInstance();
if (pDnode->status != DND_STAT_RUNNING && pMsg->msgType < TDMT_SYNC_MSG_MIN) { if (pDnode->status != DND_STAT_RUNNING && pMsg->msgType < TDMT_SYNC_MSG_MIN) {
rpcFreeCont(pMsg->pCont); rpcFreeCont(pMsg->pCont);
pMsg->pCont = NULL; pMsg->pCont = NULL;
if (pDnode->status == DND_STAT_INIT) { if (pDnode->status == DND_STAT_INIT) {
terrno = TSDB_CODE_APP_IS_STARTING; code = TSDB_CODE_APP_IS_STARTING;
} else { } else {
terrno = TSDB_CODE_APP_IS_STOPPING; code = TSDB_CODE_APP_IS_STOPPING;
} }
dError("failed to send rpc msg:%s since %s, handle:%p", TMSG_INFO(pMsg->msgType), terrstr(), pMsg->info.handle); dError("failed to send rpc msg:%s since %s, handle:%p", TMSG_INFO(pMsg->msgType), tstrerror(code),
return -1; pMsg->info.handle);
return code;
} else { } else {
rpcSendRequest(pDnode->trans.syncRpc, pEpSet, pMsg, NULL); rpcSendRequest(pDnode->trans.syncRpc, pEpSet, pMsg, NULL);
return 0; return 0;

1
source/dnode/mnode/impl/src/mndStreamUtil.c
View File

@ -105,6 +105,7 @@ SArray *mndTakeVgroupSnapshot(SMnode *pMnode, bool *allReady) {
mInfo("vgId:%d replica:%d inconsistent with other vgroups replica:%d, not ready for stream operations", mInfo("vgId:%d replica:%d inconsistent with other vgroups replica:%d, not ready for stream operations",
pVgroup->vgId, pVgroup->replica, replica); pVgroup->vgId, pVgroup->replica, replica);
*allReady = false; *allReady = false;
sdbRelease(pSdb, pVgroup);
break; break;
} }
} }

75
source/dnode/vnode/src/sma/smaRollup.c
View File

@ -61,7 +61,7 @@ struct SRSmaQTaskInfoItem {
int32_t len; int32_t len;
int8_t type; int8_t type;
int64_t suid; int64_t suid;
void * qTaskInfo; void *qTaskInfo;
}; };
static void tdRSmaQTaskInfoFree(qTaskInfo_t *taskHandle, int32_t vgId, int32_t level) { static void tdRSmaQTaskInfoFree(qTaskInfo_t *taskHandle, int32_t vgId, int32_t level) {
@ -185,7 +185,7 @@ int32_t tdUpdateTbUidList(SSma *pSma, STbUidStore *pStore, bool isAdd) {
void *pIter = NULL; void *pIter = NULL;
while ((pIter = taosHashIterate(pStore->uidHash, pIter))) { while ((pIter = taosHashIterate(pStore->uidHash, pIter))) {
tb_uid_t *pTbSuid = (tb_uid_t *)taosHashGetKey(pIter, NULL); tb_uid_t *pTbSuid = (tb_uid_t *)taosHashGetKey(pIter, NULL);
SArray * pTbUids = *(SArray **)pIter; SArray *pTbUids = *(SArray **)pIter;
if (tdUpdateTbUidListImpl(pSma, pTbSuid, pTbUids, isAdd) != TSDB_CODE_SUCCESS) { if (tdUpdateTbUidListImpl(pSma, pTbSuid, pTbUids, isAdd) != TSDB_CODE_SUCCESS) {
taosHashCancelIterate(pStore->uidHash, pIter); taosHashCancelIterate(pStore->uidHash, pIter);
@ -213,7 +213,7 @@ int32_t tdFetchTbUidList(SSma *pSma, STbUidStore **ppStore, tb_uid_t suid, tb_ui
} }
SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv); SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv);
SHashObj * infoHash = NULL; SHashObj *infoHash = NULL;
if (!pStat || !(infoHash = RSMA_INFO_HASH(pStat))) { if (!pStat || !(infoHash = RSMA_INFO_HASH(pStat))) {
terrno = TSDB_CODE_RSMA_INVALID_STAT; terrno = TSDB_CODE_RSMA_INVALID_STAT;
return TSDB_CODE_FAILED; return TSDB_CODE_FAILED;
@ -264,11 +264,11 @@ static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaStat
int8_t idx) { int8_t idx) {
if ((param->qmsgLen > 0) && param->qmsg[idx]) { if ((param->qmsgLen > 0) && param->qmsg[idx]) {
SRSmaInfoItem *pItem = &(pRSmaInfo->items[idx]); SRSmaInfoItem *pItem = &(pRSmaInfo->items[idx]);
SRetention * pRetention = SMA_RETENTION(pSma); SRetention *pRetention = SMA_RETENTION(pSma);
STsdbCfg * pTsdbCfg = SMA_TSDB_CFG(pSma); STsdbCfg *pTsdbCfg = SMA_TSDB_CFG(pSma);
SVnode * pVnode = pSma->pVnode; SVnode *pVnode = pSma->pVnode;
char taskInfDir[TSDB_FILENAME_LEN] = {0}; char taskInfDir[TSDB_FILENAME_LEN] = {0};
void * pStreamState = NULL; void *pStreamState = NULL;
// set the backend of stream state // set the backend of stream state
tdRSmaQTaskInfoGetFullPath(pVnode, pRSmaInfo->suid, idx + 1, pVnode->pTfs, taskInfDir); tdRSmaQTaskInfoGetFullPath(pVnode, pRSmaInfo->suid, idx + 1, pVnode->pTfs, taskInfDir);
@ -362,6 +362,8 @@ static int32_t tdSetRSmaInfoItemParams(SSma *pSma, SRSmaParam *param, SRSmaStat
* @return int32_t * @return int32_t
*/ */
int32_t tdRSmaProcessCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, const char *tbName) { int32_t tdRSmaProcessCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, const char *tbName) {
int32_t code;
if ((param->qmsgLen[0] == 0) && (param->qmsgLen[1] == 0)) { if ((param->qmsgLen[0] == 0) && (param->qmsgLen[1] == 0)) {
smaDebug("vgId:%d, no qmsg1/qmsg2 for rollup table %s %" PRIi64, SMA_VID(pSma), tbName, suid); smaDebug("vgId:%d, no qmsg1/qmsg2 for rollup table %s %" PRIi64, SMA_VID(pSma), tbName, suid);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -374,7 +376,7 @@ int32_t tdRSmaProcessCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, con
} }
#endif #endif
SSmaEnv * pEnv = SMA_RSMA_ENV(pSma); SSmaEnv *pEnv = SMA_RSMA_ENV(pSma);
SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv); SRSmaStat *pStat = (SRSmaStat *)SMA_ENV_STAT(pEnv);
SRSmaInfo *pRSmaInfo = NULL; SRSmaInfo *pRSmaInfo = NULL;
@ -401,8 +403,13 @@ int32_t tdRSmaProcessCreateImpl(SSma *pSma, SRSmaParam *param, int64_t suid, con
pRSmaInfo->suid = suid; pRSmaInfo->suid = suid;
T_REF_INIT_VAL(pRSmaInfo, 1); T_REF_INIT_VAL(pRSmaInfo, 1);
if (!(pRSmaInfo->queue = taosOpenQueue()) || !(pRSmaInfo->qall = taosAllocateQall()) || code = taosOpenQueue(&pRSmaInfo->queue);
tdSetRSmaInfoItemParams(pSma, param, pStat, pRSmaInfo, 0) < 0 || if (code) goto _err;
code = taosAllocateQall(&pRSmaInfo->qall);
if (code) goto _err;
if (tdSetRSmaInfoItemParams(pSma, param, pStat, pRSmaInfo, 0) < 0 ||
tdSetRSmaInfoItemParams(pSma, param, pStat, pRSmaInfo, 1) < 0) { tdSetRSmaInfoItemParams(pSma, param, pStat, pRSmaInfo, 1) < 0) {
goto _err; goto _err;
} }
@ -673,8 +680,8 @@ static int32_t tdRSmaExecAndSubmitResult(SSma *pSma, qTaskInfo_t taskInfo, SRSma
int32_t code = 0; int32_t code = 0;
int32_t lino = 0; int32_t lino = 0;
SSDataBlock *output = NULL; SSDataBlock *output = NULL;
SArray * pResList = pItem->pResList; SArray *pResList = pItem->pResList;
STSchema * pTSchema = pInfo->pTSchema; STSchema *pTSchema = pInfo->pTSchema;
int64_t suid = pInfo->suid; int64_t suid = pInfo->suid;
while (1) { while (1) {
@ -733,7 +740,7 @@ static int32_t tdRSmaExecAndSubmitResult(SSma *pSma, qTaskInfo_t taskInfo, SRSma
} }
} }
STsdb * sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb[0] : pSma->pRSmaTsdb[1]); STsdb *sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb[0] : pSma->pRSmaTsdb[1]);
SSubmitReq2 *pReq = NULL; SSubmitReq2 *pReq = NULL;
if (buildSubmitReqFromDataBlock(&pReq, output, pTSchema, output->info.id.groupId, SMA_VID(pSma), suid) < 0) { if (buildSubmitReqFromDataBlock(&pReq, output, pTSchema, output->info.id.groupId, SMA_VID(pSma), suid) < 0) {
@ -795,11 +802,13 @@ _exit:
*/ */
static int32_t tdExecuteRSmaImplAsync(SSma *pSma, int64_t version, const void *pMsg, int32_t len, int32_t inputType, static int32_t tdExecuteRSmaImplAsync(SSma *pSma, int64_t version, const void *pMsg, int32_t len, int32_t inputType,
SRSmaInfo *pInfo, tb_uid_t suid) { SRSmaInfo *pInfo, tb_uid_t suid) {
int32_t code;
int32_t size = RSMA_EXEC_MSG_HLEN + len; // header + payload int32_t size = RSMA_EXEC_MSG_HLEN + len; // header + payload
void * qItem = taosAllocateQitem(size, DEF_QITEM, 0); void *qItem;
if (!qItem) { code = taosAllocateQitem(size, DEF_QITEM, 0, (void **)&qItem);
return TSDB_CODE_FAILED; if (code) {
return code;
} }
void *pItem = qItem; void *pItem = qItem;
@ -874,7 +883,7 @@ static int32_t tdRsmaPrintSubmitReq(SSma *pSma, SSubmitReq *pReq) {
static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t msgSize, int64_t version, int32_t inputType, static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t msgSize, int64_t version, int32_t inputType,
SRSmaInfo *pInfo, ERsmaExecType type, int8_t level) { SRSmaInfo *pInfo, ERsmaExecType type, int8_t level) {
int32_t idx = level - 1; int32_t idx = level - 1;
void * qTaskInfo = RSMA_INFO_QTASK(pInfo, idx); void *qTaskInfo = RSMA_INFO_QTASK(pInfo, idx);
SRSmaInfoItem *pItem = RSMA_INFO_ITEM(pInfo, idx); SRSmaInfoItem *pItem = RSMA_INFO_ITEM(pInfo, idx);
if (!qTaskInfo) { if (!qTaskInfo) {
@ -914,7 +923,7 @@ static int32_t tdExecuteRSmaImpl(SSma *pSma, const void *pMsg, int32_t msgSize,
static SRSmaInfo *tdAcquireRSmaInfoBySuid(SSma *pSma, int64_t suid) { static SRSmaInfo *tdAcquireRSmaInfoBySuid(SSma *pSma, int64_t suid) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0; int32_t lino = 0;
SSmaEnv * pEnv = SMA_RSMA_ENV(pSma); SSmaEnv *pEnv = SMA_RSMA_ENV(pSma);
SRSmaStat *pStat = NULL; SRSmaStat *pStat = NULL;
SRSmaInfo *pRSmaInfo = NULL; SRSmaInfo *pRSmaInfo = NULL;
@ -1069,8 +1078,8 @@ _err:
static int32_t tdRSmaRestoreQTaskInfoInit(SSma *pSma, int64_t *nTables) { static int32_t tdRSmaRestoreQTaskInfoInit(SSma *pSma, int64_t *nTables) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0; int32_t lino = 0;
SVnode * pVnode = pSma->pVnode; SVnode *pVnode = pSma->pVnode;
SArray * suidList = NULL; SArray *suidList = NULL;
STbUidStore uidStore = {0}; STbUidStore uidStore = {0};
SMetaReader mr = {0}; SMetaReader mr = {0};
tb_uid_t suid = 0; tb_uid_t suid = 0;
@ -1198,7 +1207,7 @@ int32_t tdRSmaPersistExecImpl(SRSmaStat *pRSmaStat, SHashObj *pInfoHash) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0; int32_t lino = 0;
int32_t nTaskInfo = 0; int32_t nTaskInfo = 0;
SSma * pSma = pRSmaStat->pSma; SSma *pSma = pRSmaStat->pSma;
SVnode *pVnode = pSma->pVnode; SVnode *pVnode = pSma->pVnode;
if (taosHashGetSize(pInfoHash) <= 0) { if (taosHashGetSize(pInfoHash) <= 0) {
@ -1231,7 +1240,7 @@ int32_t tdRSmaPersistExecImpl(SRSmaStat *pRSmaStat, SHashObj *pInfoHash) {
do { do {
int32_t nStreamFlushed = 0; int32_t nStreamFlushed = 0;
int32_t nSleep = 0; int32_t nSleep = 0;
void * infoHash = NULL; void *infoHash = NULL;
while (true) { while (true) {
while ((infoHash = taosHashIterate(pInfoHash, infoHash))) { while ((infoHash = taosHashIterate(pInfoHash, infoHash))) {
SRSmaInfo *pRSmaInfo = *(SRSmaInfo **)infoHash; SRSmaInfo *pRSmaInfo = *(SRSmaInfo **)infoHash;
@ -1273,7 +1282,7 @@ _checkpoint:
SStreamMeta *pMeta = NULL; SStreamMeta *pMeta = NULL;
int64_t checkpointId = taosGetTimestampNs(); int64_t checkpointId = taosGetTimestampNs();
bool checkpointBuilt = false; bool checkpointBuilt = false;
void * infoHash = NULL; void *infoHash = NULL;
while ((infoHash = taosHashIterate(pInfoHash, infoHash))) { while ((infoHash = taosHashIterate(pInfoHash, infoHash))) {
SRSmaInfo *pRSmaInfo = *(SRSmaInfo **)infoHash; SRSmaInfo *pRSmaInfo = *(SRSmaInfo **)infoHash;
if (RSMA_INFO_IS_DEL(pRSmaInfo)) { if (RSMA_INFO_IS_DEL(pRSmaInfo)) {
@ -1346,10 +1355,10 @@ _exit:
* @param tmrId * @param tmrId
*/ */
static void tdRSmaFetchTrigger(void *param, void *tmrId) { static void tdRSmaFetchTrigger(void *param, void *tmrId) {
SRSmaRef * pRSmaRef = NULL; SRSmaRef *pRSmaRef = NULL;
SSma * pSma = NULL; SSma *pSma = NULL;
SRSmaStat * pStat = NULL; SRSmaStat *pStat = NULL;
SRSmaInfo * pRSmaInfo = NULL; SRSmaInfo *pRSmaInfo = NULL;
SRSmaInfoItem *pItem = NULL; SRSmaInfoItem *pItem = NULL;
if (!(pRSmaRef = taosHashGet(smaMgmt.refHash, &param, POINTER_BYTES))) { if (!(pRSmaRef = taosHashGet(smaMgmt.refHash, &param, POINTER_BYTES))) {
@ -1517,7 +1526,7 @@ _err:
} }
static int32_t tdRSmaBatchExec(SSma *pSma, SRSmaInfo *pInfo, STaosQall *qall, SArray *pSubmitArr, ERsmaExecType type) { static int32_t tdRSmaBatchExec(SSma *pSma, SRSmaInfo *pInfo, STaosQall *qall, SArray *pSubmitArr, ERsmaExecType type) {
void * msg = NULL; void *msg = NULL;
int8_t resume = 0; int8_t resume = 0;
int32_t nSubmit = 0; int32_t nSubmit = 0;
int32_t nDelete = 0; int32_t nDelete = 0;
@ -1548,7 +1557,7 @@ static int32_t tdRSmaBatchExec(SSma *pSma, SRSmaInfo *pInfo, STaosQall *qall, SA
_resume_delete: _resume_delete:
version = RSMA_EXEC_MSG_VER(msg); version = RSMA_EXEC_MSG_VER(msg);
if ((terrno = tqExtractDelDataBlock(RSMA_EXEC_MSG_BODY(msg), RSMA_EXEC_MSG_LEN(msg), version, if ((terrno = tqExtractDelDataBlock(RSMA_EXEC_MSG_BODY(msg), RSMA_EXEC_MSG_LEN(msg), version,
&packData.pDataBlock, 1))) { &packData.pDataBlock, 1))) {
taosFreeQitem(msg); taosFreeQitem(msg);
goto _err; goto _err;
} }
@ -1621,11 +1630,11 @@ _err:
int32_t tdRSmaProcessExecImpl(SSma *pSma, ERsmaExecType type) { int32_t tdRSmaProcessExecImpl(SSma *pSma, ERsmaExecType type) {
int32_t code = 0; int32_t code = 0;
int32_t lino = 0; int32_t lino = 0;
SVnode * pVnode = pSma->pVnode; SVnode *pVnode = pSma->pVnode;
SSmaEnv * pEnv = SMA_RSMA_ENV(pSma); SSmaEnv *pEnv = SMA_RSMA_ENV(pSma);
SRSmaStat *pRSmaStat = (SRSmaStat *)SMA_ENV_STAT(pEnv); SRSmaStat *pRSmaStat = (SRSmaStat *)SMA_ENV_STAT(pEnv);
SHashObj * infoHash = NULL; SHashObj *infoHash = NULL;
SArray * pSubmitArr = NULL; SArray *pSubmitArr = NULL;
bool isFetchAll = false; bool isFetchAll = false;
if (!pRSmaStat || !(infoHash = RSMA_INFO_HASH(pRSmaStat))) { if (!pRSmaStat || !(infoHash = RSMA_INFO_HASH(pRSmaStat))) {

77
source/dnode/vnode/src/tq/tqSink.c
View File

@ -725,24 +725,33 @@ int32_t setDstTableDataUid(SVnode* pVnode, SStreamTask* pTask, SSDataBlock* pDat
if (metaGetTableEntryByName(&mr, dstTableName) < 0) { if (metaGetTableEntryByName(&mr, dstTableName) < 0) {
metaReaderClear(&mr); metaReaderClear(&mr);
tqDebug("s-task:%s stream write into table:%s, table auto created", id, dstTableName); if (pTask->outputInfo.tbSink.autoCreateCtb) {
tqDebug("s-task:%s stream write into table:%s, table auto created", id, dstTableName);
SArray* pTagArray = taosArrayInit(pTSchema->numOfCols + 1, sizeof(STagVal)); SArray* pTagArray = taosArrayInit(pTSchema->numOfCols + 1, sizeof(STagVal));
pTableData->flags = SUBMIT_REQ_AUTO_CREATE_TABLE; pTableData->flags = SUBMIT_REQ_AUTO_CREATE_TABLE;
pTableData->pCreateTbReq = pTableData->pCreateTbReq =
buildAutoCreateTableReq(stbFullName, suid, pTSchema->numOfCols + 1, pDataBlock, pTagArray, buildAutoCreateTableReq(stbFullName, suid, pTSchema->numOfCols + 1, pDataBlock, pTagArray,
pTask->ver >= SSTREAM_TASK_SUBTABLE_CHANGED_VER && pTask->subtableWithoutMd5 != 1); pTask->ver >= SSTREAM_TASK_SUBTABLE_CHANGED_VER && pTask->subtableWithoutMd5 != 1);
taosArrayDestroy(pTagArray); taosArrayDestroy(pTagArray);
if (pTableData->pCreateTbReq == NULL) { if (pTableData->pCreateTbReq == NULL) {
tqError("s-task:%s failed to build auto create table req, code:%s", id, tstrerror(terrno)); tqError("s-task:%s failed to build auto create dst-table req:%s, code:%s", id, dstTableName,
taosMemoryFree(pTableSinkInfo); tstrerror(terrno));
return terrno; taosMemoryFree(pTableSinkInfo);
return terrno;
}
pTableSinkInfo->uid = 0;
doPutIntoCache(pTask->outputInfo.tbSink.pTblInfo, pTableSinkInfo, groupId, id);
} else {
metaReaderClear(&mr);
tqError("s-task:%s vgId:%d dst-table:%s not auto-created, and not create in tsdb, discard data", id,
vgId, dstTableName);
return TSDB_CODE_TDB_TABLE_NOT_EXIST;
} }
pTableSinkInfo->uid = 0;
doPutIntoCache(pTask->outputInfo.tbSink.pTblInfo, pTableSinkInfo, groupId, id);
} else { } else {
bool isValid = isValidDstChildTable(&mr, vgId, dstTableName, suid); bool isValid = isValidDstChildTable(&mr, vgId, dstTableName, suid);
if (!isValid) { if (!isValid) {
@ -788,16 +797,34 @@ int32_t tqSetDstTableDataPayload(uint64_t suid, const STSchema *pTSchema, int32_
} }
void tqSinkDataIntoDstTable(SStreamTask* pTask, void* vnode, void* data) { void tqSinkDataIntoDstTable(SStreamTask* pTask, void* vnode, void* data) {
const SArray* pBlocks = (const SArray*)data; const SArray* pBlocks = (const SArray*)data;
SVnode* pVnode = (SVnode*)vnode; SVnode* pVnode = (SVnode*)vnode;
int64_t suid = pTask->outputInfo.tbSink.stbUid; int64_t suid = pTask->outputInfo.tbSink.stbUid;
char* stbFullName = pTask->outputInfo.tbSink.stbFullName; char* stbFullName = pTask->outputInfo.tbSink.stbFullName;
STSchema* pTSchema = pTask->outputInfo.tbSink.pTSchema; STSchema* pTSchema = pTask->outputInfo.tbSink.pTSchema;
int32_t vgId = TD_VID(pVnode); int32_t vgId = TD_VID(pVnode);
int32_t numOfBlocks = taosArrayGetSize(pBlocks); int32_t numOfBlocks = taosArrayGetSize(pBlocks);
int32_t code = TSDB_CODE_SUCCESS; int32_t code = TSDB_CODE_SUCCESS;
const char* id = pTask->id.idStr; const char* id = pTask->id.idStr;
int64_t earlyTs = tsdbGetEarliestTs(pVnode->pTsdb); int64_t earlyTs = tsdbGetEarliestTs(pVnode->pTsdb);
STaskOutputInfo* pOutputInfo = &pTask->outputInfo;
if (pTask->outputInfo.tbSink.pTagSchema == NULL) {
SMetaReader mer1 = {0};
metaReaderDoInit(&mer1, pVnode->pMeta, META_READER_LOCK);
code = metaReaderGetTableEntryByUid(&mer1, pOutputInfo->tbSink.stbUid);
pOutputInfo->tbSink.pTagSchema = tCloneSSchemaWrapper(&mer1.me.stbEntry.schemaTag);
metaReaderClear(&mer1);
SSchemaWrapper* pTagSchema = pOutputInfo->tbSink.pTagSchema;
SSchema* pCol1 = &pTagSchema->pSchema[0];
if (pTagSchema->nCols == 1 && pCol1->type == TSDB_DATA_TYPE_UBIGINT && strcmp(pCol1->name, "group_id") == 0) {
pOutputInfo->tbSink.autoCreateCtb = true;
} else {
pOutputInfo->tbSink.autoCreateCtb = false;
}
}
bool onlySubmitData = hasOnlySubmitData(pBlocks, numOfBlocks); bool onlySubmitData = hasOnlySubmitData(pBlocks, numOfBlocks);
if (!onlySubmitData) { if (!onlySubmitData) {
@ -829,6 +856,7 @@ void tqSinkDataIntoDstTable(SStreamTask* pTask, void* vnode, void* data) {
SSubmitTbData tbData = {.suid = suid, .uid = 0, .sver = pTSchema->version, .flags = TD_REQ_FROM_APP}; SSubmitTbData tbData = {.suid = suid, .uid = 0, .sver = pTSchema->version, .flags = TD_REQ_FROM_APP};
code = setDstTableDataUid(pVnode, pTask, pDataBlock, stbFullName, &tbData); code = setDstTableDataUid(pVnode, pTask, pDataBlock, stbFullName, &tbData);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
tqError("vgId:%d s-task:%s dst-table not exist, stb:%s discard stream results", vgId, id, stbFullName);
continue; continue;
} }
@ -882,6 +910,7 @@ void tqSinkDataIntoDstTable(SStreamTask* pTask, void* vnode, void* data) {
if (index == NULL) { // no data yet, append it if (index == NULL) { // no data yet, append it
code = setDstTableDataUid(pVnode, pTask, pDataBlock, stbFullName, &tbData); code = setDstTableDataUid(pVnode, pTask, pDataBlock, stbFullName, &tbData);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
tqError("vgId:%d dst-table gid:%" PRId64 " not exist, discard stream results", vgId, groupId);
continue; continue;
} }

66
source/dnode/vnode/src/tq/tqUtil.c
View File

@ -164,7 +164,8 @@ static int32_t extractDataAndRspForNormalSubscribe(STQ* pTq, STqHandle* pHandle,
taosWUnLockLatch(&pTq->lock); taosWUnLockLatch(&pTq->lock);
} }
tOffsetCopy(&dataRsp.common.reqOffset, pOffset); // reqOffset represents the current date offset, may be changed if wal not exists tOffsetCopy(&dataRsp.common.reqOffset,
pOffset); // reqOffset represents the current date offset, may be changed if wal not exists
code = tqSendDataRsp(pHandle, pMsg, pRequest, &dataRsp, TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); code = tqSendDataRsp(pHandle, pMsg, pRequest, &dataRsp, TMQ_MSG_TYPE__POLL_DATA_RSP, vgId);
end : { end : {
@ -178,25 +179,25 @@ end : {
} }
} }
#define PROCESS_EXCLUDED_MSG(TYPE, DECODE_FUNC, DELETE_FUNC) \ #define PROCESS_EXCLUDED_MSG(TYPE, DECODE_FUNC, DELETE_FUNC) \
SDecoder decoder = {0};\ SDecoder decoder = {0}; \
TYPE req = {0}; \ TYPE req = {0}; \
void* data = POINTER_SHIFT(pHead->body, sizeof(SMsgHead)); \ void* data = POINTER_SHIFT(pHead->body, sizeof(SMsgHead)); \
int32_t len = pHead->bodyLen - sizeof(SMsgHead); \ int32_t len = pHead->bodyLen - sizeof(SMsgHead); \
tDecoderInit(&decoder, data, len); \ tDecoderInit(&decoder, data, len); \
if (DECODE_FUNC(&decoder, &req) == 0 && (req.source & TD_REQ_FROM_TAOX) != 0) { \ if (DECODE_FUNC(&decoder, &req) == 0 && (req.source & TD_REQ_FROM_TAOX) != 0) { \
tqDebug("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, jump meta for, vgId:%d offset %" PRId64 " msgType %d", \ tqDebug("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, jump meta for, vgId:%d offset %" PRId64 \
pRequest->consumerId, pRequest->epoch, vgId, fetchVer, pHead->msgType); \ " msgType %d", \
fetchVer++; \ pRequest->consumerId, pRequest->epoch, vgId, fetchVer, pHead->msgType); \
DELETE_FUNC(&req); \ fetchVer++; \
tDecoderClear(&decoder); \ DELETE_FUNC(&req); \
continue; \ tDecoderClear(&decoder); \
} \ continue; \
DELETE_FUNC(&req); \ } \
DELETE_FUNC(&req); \
tDecoderClear(&decoder); tDecoderClear(&decoder);
static void tDeleteCommon(void* parm) { static void tDeleteCommon(void* parm) {}
}
static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest,
SRpcMsg* pMsg, STqOffsetVal* offset) { SRpcMsg* pMsg, STqOffsetVal* offset) {
@ -313,9 +314,9 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle,
tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); tqError("tmq extract meta from log, tEncodeMqMetaRsp error");
continue; continue;
} }
int32_t tLen = sizeof(SMqRspHead) + len; int32_t tLen = sizeof(SMqRspHead) + len;
void* tBuf = taosMemoryCalloc(1, tLen); void* tBuf = taosMemoryCalloc(1, tLen);
void* metaBuff = POINTER_SHIFT(tBuf, sizeof(SMqRspHead)); void* metaBuff = POINTER_SHIFT(tBuf, sizeof(SMqRspHead));
SEncoder encoder = {0}; SEncoder encoder = {0};
tEncoderInit(&encoder, metaBuff, len); tEncoderInit(&encoder, metaBuff, len);
code = tEncodeMqMetaRsp(&encoder, &tmpMetaRsp); code = tEncodeMqMetaRsp(&encoder, &tmpMetaRsp);
@ -374,13 +375,13 @@ end:
} }
int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg) { int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg) {
int32_t code = 0; int32_t code = 0;
STqOffsetVal reqOffset = {0}; STqOffsetVal reqOffset = {0};
tOffsetCopy(&reqOffset, &pRequest->reqOffset); tOffsetCopy(&reqOffset, &pRequest->reqOffset);
// reset the offset if needed // reset the offset if needed
if (IS_OFFSET_RESET_TYPE(pRequest->reqOffset.type)) { if (IS_OFFSET_RESET_TYPE(pRequest->reqOffset.type)) {
bool blockReturned = false; bool blockReturned = false;
code = extractResetOffsetVal(&reqOffset, pTq, pHandle, pRequest, pMsg, &blockReturned); code = extractResetOffsetVal(&reqOffset, pTq, pHandle, pRequest, pMsg, &blockReturned);
if (code != 0) { if (code != 0) {
goto END; goto END;
@ -392,7 +393,7 @@ int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequ
} }
} else if (reqOffset.type == 0) { // use the consumer specified offset } else if (reqOffset.type == 0) { // use the consumer specified offset
uError("req offset type is 0"); uError("req offset type is 0");
code = TSDB_CODE_TMQ_INVALID_MSG; code = TSDB_CODE_TMQ_INVALID_MSG;
goto END; goto END;
} }
@ -416,8 +417,8 @@ static void initMqRspHead(SMqRspHead* pMsgHead, int32_t type, int32_t epoch, int
pMsgHead->walever = ever; pMsgHead->walever = ever;
} }
int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* pReq, const SMqBatchMetaRsp* pRsp, int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* pReq,
int32_t vgId) { const SMqBatchMetaRsp* pRsp, int32_t vgId) {
int32_t len = 0; int32_t len = 0;
int32_t code = 0; int32_t code = 0;
tEncodeSize(tEncodeMqBatchMetaRsp, pRsp, len, code); tEncodeSize(tEncodeMqBatchMetaRsp, pRsp, len, code);
@ -444,8 +445,8 @@ int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SM
SRpcMsg resp = {.info = pMsg->info, .pCont = buf, .contLen = tlen, .code = 0}; SRpcMsg resp = {.info = pMsg->info, .pCont = buf, .contLen = tlen, .code = 0};
tmsgSendRsp(&resp); tmsgSendRsp(&resp);
tqDebug("vgId:%d, from consumer:0x%" PRIx64 " (epoch %d) send rsp, res msg type: batch meta, size:%ld offset type:%d", vgId, tqDebug("vgId:%d, from consumer:0x%" PRIx64 " (epoch %d) send rsp, res msg type: batch meta, size:%ld offset type:%d",
pReq->consumerId, pReq->epoch, taosArrayGetSize(pRsp->batchMetaReq), pRsp->rspOffset.type); vgId, pReq->consumerId, pReq->epoch, taosArrayGetSize(pRsp->batchMetaReq), pRsp->rspOffset.type);
return 0; return 0;
} }
@ -527,6 +528,7 @@ int32_t tqDoSendDataRsp(const SRpcHandleInfo* pRpcHandleInfo, const void* pRsp,
} }
int32_t tqExtractDelDataBlock(const void* pData, int32_t len, int64_t ver, void** pRefBlock, int32_t type) { int32_t tqExtractDelDataBlock(const void* pData, int32_t len, int64_t ver, void** pRefBlock, int32_t type) {
int32_t code;
SDecoder* pCoder = &(SDecoder){0}; SDecoder* pCoder = &(SDecoder){0};
SDeleteRes* pRes = &(SDeleteRes){0}; SDeleteRes* pRes = &(SDeleteRes){0};
@ -570,11 +572,11 @@ int32_t tqExtractDelDataBlock(const void* pData, int32_t len, int64_t ver, void*
taosArrayDestroy(pRes->uidList); taosArrayDestroy(pRes->uidList);
if (type == 0) { if (type == 0) {
*pRefBlock = taosAllocateQitem(sizeof(SStreamRefDataBlock), DEF_QITEM, 0); code = taosAllocateQitem(sizeof(SStreamRefDataBlock), DEF_QITEM, 0, pRefBlock);
if (*pRefBlock == NULL) { if (code) {
blockDataCleanup(pDelBlock); blockDataCleanup(pDelBlock);
taosMemoryFree(pDelBlock); taosMemoryFree(pDelBlock);
return TSDB_CODE_OUT_OF_MEMORY; return code;
} }
((SStreamRefDataBlock*)(*pRefBlock))->type = STREAM_INPUT__REF_DATA_BLOCK; ((SStreamRefDataBlock*)(*pRefBlock))->type = STREAM_INPUT__REF_DATA_BLOCK;
@ -614,7 +616,7 @@ int32_t tqGetStreamExecInfo(SVnode* pVnode, int64_t streamId, int64_t* pDelay, b
continue; continue;
} }
STaskId id = {.streamId = pId->streamId, .taskId = pId->taskId}; STaskId id = {.streamId = pId->streamId, .taskId = pId->taskId};
SStreamTask** ppTask = taosHashGet(pMeta->pTasksMap, &id, sizeof(id)); SStreamTask** ppTask = taosHashGet(pMeta->pTasksMap, &id, sizeof(id));
if (ppTask == NULL) { if (ppTask == NULL) {
tqError("vgId:%d failed to acquire task:0x%x in retrieving progress", pMeta->vgId, pId->taskId); tqError("vgId:%d failed to acquire task:0x%x in retrieving progress", pMeta->vgId, pId->taskId);

17
source/libs/executor/src/dataDeleter.c
View File

@ -130,9 +130,11 @@ static int32_t getStatus(SDataDeleterHandle* pDeleter) {
static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput, bool* pContinue) { static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput, bool* pContinue) {
SDataDeleterHandle* pDeleter = (SDataDeleterHandle*)pHandle; SDataDeleterHandle* pDeleter = (SDataDeleterHandle*)pHandle;
SDataDeleterBuf* pBuf = taosAllocateQitem(sizeof(SDataDeleterBuf), DEF_QITEM, 0); SDataDeleterBuf* pBuf;
if (NULL == pBuf) {
return TSDB_CODE_OUT_OF_MEMORY; int32_t code = taosAllocateQitem(sizeof(SDataDeleterBuf), DEF_QITEM, 0, (void**)&pBuf);
if (code) {
return code;
} }
if (!allocBuf(pDeleter, pInput, pBuf)) { if (!allocBuf(pDeleter, pInput, pBuf)) {
@ -227,7 +229,7 @@ static int32_t destroyDataSinker(SDataSinkHandle* pHandle) {
} }
taosCloseQueue(pDeleter->pDataBlocks); taosCloseQueue(pDeleter->pDataBlocks);
taosThreadMutexDestroy(&pDeleter->mutex); taosThreadMutexDestroy(&pDeleter->mutex);
taosMemoryFree(pDeleter->pManager); taosMemoryFree(pDeleter->pManager);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
@ -270,12 +272,11 @@ int32_t createDataDeleter(SDataSinkManager* pManager, const SDataSinkNode* pData
deleter->pParam = pParam; deleter->pParam = pParam;
deleter->status = DS_BUF_EMPTY; deleter->status = DS_BUF_EMPTY;
deleter->queryEnd = false; deleter->queryEnd = false;
deleter->pDataBlocks = taosOpenQueue(); code = taosOpenQueue(&deleter->pDataBlocks);
taosThreadMutexInit(&deleter->mutex, NULL); if (code) {
if (NULL == deleter->pDataBlocks) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _end; goto _end;
} }
taosThreadMutexInit(&deleter->mutex, NULL);
*pHandle = deleter; *pHandle = deleter;
return code; return code;

20
source/libs/executor/src/dataDispatcher.c
View File

@ -104,7 +104,8 @@ static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pIn
} }
int32_t dataLen = blockEncode(pInput->pData, pHandle->pCompressBuf, numOfCols); int32_t dataLen = blockEncode(pInput->pData, pHandle->pCompressBuf, numOfCols);
int32_t len = tsCompressString(pHandle->pCompressBuf, dataLen, 1, pEntry->data, pBuf->allocSize, ONE_STAGE_COMP, NULL, 0); int32_t len =
tsCompressString(pHandle->pCompressBuf, dataLen, 1, pEntry->data, pBuf->allocSize, ONE_STAGE_COMP, NULL, 0);
if (len < dataLen) { if (len < dataLen) {
pEntry->compressed = 1; pEntry->compressed = 1;
pEntry->dataLen = len; pEntry->dataLen = len;
@ -168,9 +169,11 @@ static int32_t getStatus(SDataDispatchHandle* pDispatcher) {
static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput, bool* pContinue) { static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput, bool* pContinue) {
int32_t code = 0; int32_t code = 0;
SDataDispatchHandle* pDispatcher = (SDataDispatchHandle*)pHandle; SDataDispatchHandle* pDispatcher = (SDataDispatchHandle*)pHandle;
SDataDispatchBuf* pBuf = taosAllocateQitem(sizeof(SDataDispatchBuf), DEF_QITEM, 0); SDataDispatchBuf* pBuf;
if (NULL == pBuf) {
return TSDB_CODE_OUT_OF_MEMORY; code = taosAllocateQitem(sizeof(SDataDispatchBuf), DEF_QITEM, 0, (void**)&pBuf);
if (code) {
return code;
} }
if (!allocBuf(pDispatcher, pInput, pBuf)) { if (!allocBuf(pDispatcher, pInput, pBuf)) {
@ -228,7 +231,6 @@ static void getDataLength(SDataSinkHandle* pHandle, int64_t* pLen, int64_t* pRow
((SDataCacheEntry*)(pDispatcher->nextOutput.pData))->numOfRows); ((SDataCacheEntry*)(pDispatcher->nextOutput.pData))->numOfRows);
} }
static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) { static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) {
SDataDispatchHandle* pDispatcher = (SDataDispatchHandle*)pHandle; SDataDispatchHandle* pDispatcher = (SDataDispatchHandle*)pHandle;
if (NULL == pDispatcher->nextOutput.pData) { if (NULL == pDispatcher->nextOutput.pData) {
@ -291,6 +293,8 @@ static int32_t getCacheSize(struct SDataSinkHandle* pHandle, uint64_t* size) {
} }
int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pDataSink, DataSinkHandle* pHandle) { int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pDataSink, DataSinkHandle* pHandle) {
int32_t code;
SDataDispatchHandle* dispatcher = taosMemoryCalloc(1, sizeof(SDataDispatchHandle)); SDataDispatchHandle* dispatcher = taosMemoryCalloc(1, sizeof(SDataDispatchHandle));
if (NULL == dispatcher) { if (NULL == dispatcher) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
@ -309,7 +313,11 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, const SDataSinkNode* pD
dispatcher->pSchema = pDataSink->pInputDataBlockDesc; dispatcher->pSchema = pDataSink->pInputDataBlockDesc;
dispatcher->status = DS_BUF_EMPTY; dispatcher->status = DS_BUF_EMPTY;
dispatcher->queryEnd = false; dispatcher->queryEnd = false;
dispatcher->pDataBlocks = taosOpenQueue(); code = taosOpenQueue(&dispatcher->pDataBlocks);
if (code) {
terrno = code;
goto _return;
}
taosThreadMutexInit(&dispatcher->mutex, NULL); taosThreadMutexInit(&dispatcher->mutex, NULL);
if (NULL == dispatcher->pDataBlocks) { if (NULL == dispatcher->pDataBlocks) {

4
source/libs/parser/inc/parTranslater.h
View File

@ -47,7 +47,7 @@ typedef struct STranslateContext {
SNode* pPostRoot; SNode* pPostRoot;
} STranslateContext; } STranslateContext;
bool biRewriteToTbnameFunc(STranslateContext* pCxt, SNode** ppNode); int32_t biRewriteToTbnameFunc(STranslateContext* pCxt, SNode** ppNode, bool* pRet);
int32_t biRewriteSelectStar(STranslateContext* pCxt, SSelectStmt* pSelect); int32_t biRewriteSelectStar(STranslateContext* pCxt, SSelectStmt* pSelect);
int32_t biCheckCreateTableTbnameCol(STranslateContext* pCxt, SCreateTableStmt* pStmt); int32_t biCheckCreateTableTbnameCol(STranslateContext* pCxt, SCreateTableStmt* pStmt);
int32_t findTable(STranslateContext* pCxt, const char* pTableAlias, STableNode** pOutput); int32_t findTable(STranslateContext* pCxt, const char* pTableAlias, STableNode** pOutput);
@ -57,4 +57,4 @@ int32_t getTargetMetaImpl(SParseContext* pParCxt, SParseMetaCache* pMetaCache, c
} }
#endif #endif
#endif /*_TD_PARSER_TRANS_H_*/ #endif /*_TD_PARSER_TRANS_H_*/

2
source/libs/parser/inc/sql.y
View File

@ -509,7 +509,7 @@ cmd ::= SHOW USERS FULL.
cmd ::= SHOW USER PRIVILEGES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_USER_PRIVILEGES_STMT); } cmd ::= SHOW USER PRIVILEGES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_USER_PRIVILEGES_STMT); }
cmd ::= SHOW db_kind_opt(A) DATABASES. { cmd ::= SHOW db_kind_opt(A) DATABASES. {
pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DATABASES_STMT); pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_DATABASES_STMT);
setShowKind(pCxt, pCxt->pRootNode, A); (void)setShowKind(pCxt, pCxt->pRootNode, A);
} }
cmd ::= SHOW table_kind_db_name_cond_opt(A) TABLES like_pattern_opt(B). { cmd ::= SHOW table_kind_db_name_cond_opt(A) TABLES like_pattern_opt(B). {
pCxt->pRootNode = createShowTablesStmt(pCxt, A, B, OP_TYPE_LIKE); pCxt->pRootNode = createShowTablesStmt(pCxt, A, B, OP_TYPE_LIKE);

43
source/libs/parser/src/parAstCreater.c
View File

@ -101,7 +101,7 @@ static bool checkPassword(SAstCreateContext* pCxt, const SToken* pPasswordToken,
pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NAME_OR_PASSWD_TOO_LONG); pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NAME_OR_PASSWD_TOO_LONG);
} else { } else {
strncpy(pPassword, pPasswordToken->z, pPasswordToken->n); strncpy(pPassword, pPasswordToken->z, pPasswordToken->n);
strdequote(pPassword); (void)strdequote(pPassword);
if (strtrim(pPassword) <= 0) { if (strtrim(pPassword) <= 0) {
pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_PASSWD_EMPTY); pCxt->errCode = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_PASSWD_EMPTY);
} else if (invalidPassword(pPassword)) { } else if (invalidPassword(pPassword)) {
@ -126,8 +126,8 @@ static int32_t parseEndpoint(SAstCreateContext* pCxt, const SToken* pEp, char* p
char ep[TSDB_FQDN_LEN + 1 + 5] = {0}; char ep[TSDB_FQDN_LEN + 1 + 5] = {0};
COPY_STRING_FORM_ID_TOKEN(ep, pEp); COPY_STRING_FORM_ID_TOKEN(ep, pEp);
strdequote(ep); (void)strdequote(ep);
strtrim(ep); (void)strtrim(ep);
if (NULL == pPort) { if (NULL == pPort) {
strcpy(pFqdn, ep); strcpy(pFqdn, ep);
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
@ -371,7 +371,7 @@ SNode* createValueNode(SAstCreateContext* pCxt, int32_t dataType, const SToken*
val->literal = strndup(pLiteral->z, pLiteral->n); val->literal = strndup(pLiteral->z, pLiteral->n);
if (TK_NK_ID != pLiteral->type && TK_TIMEZONE != pLiteral->type && if (TK_NK_ID != pLiteral->type && TK_TIMEZONE != pLiteral->type &&
(IS_VAR_DATA_TYPE(dataType) || TSDB_DATA_TYPE_TIMESTAMP == dataType)) { (IS_VAR_DATA_TYPE(dataType) || TSDB_DATA_TYPE_TIMESTAMP == dataType)) {
trimString(pLiteral->z, pLiteral->n, val->literal, pLiteral->n); (void)trimString(pLiteral->z, pLiteral->n, val->literal, pLiteral->n);
} }
CHECK_OUT_OF_MEM(val->literal); CHECK_OUT_OF_MEM(val->literal);
val->node.resType.type = dataType; val->node.resType.type = dataType;
@ -784,7 +784,10 @@ SNode* createPlaceholderValueNode(SAstCreateContext* pCxt, const SToken* pLitera
return NULL; return NULL;
} }
} }
taosArrayPush(pCxt->pPlaceholderValues, &val); if (NULL == taosArrayPush(pCxt->pPlaceholderValues, &val)) {
pCxt->errCode = TSDB_CODE_OUT_OF_MEMORY;
}
CHECK_PARSER_STATUS(pCxt);
return (SNode*)val; return (SNode*)val;
} }
@ -905,7 +908,8 @@ SNode* createCastFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SDataType d
} else if (TSDB_DATA_TYPE_NCHAR == dt.type) { } else if (TSDB_DATA_TYPE_NCHAR == dt.type) {
func->node.resType.bytes = func->node.resType.bytes * TSDB_NCHAR_SIZE + VARSTR_HEADER_SIZE; func->node.resType.bytes = func->node.resType.bytes * TSDB_NCHAR_SIZE + VARSTR_HEADER_SIZE;
} }
nodesListMakeAppend(&func->pParameterList, pExpr); pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)func; return (SNode*)func;
} }
@ -923,8 +927,10 @@ SNode* createNodeListNodeEx(SAstCreateContext* pCxt, SNode* p1, SNode* p2) {
CHECK_OUT_OF_MEM(list); CHECK_OUT_OF_MEM(list);
list->pNodeList = nodesMakeList(); list->pNodeList = nodesMakeList();
CHECK_OUT_OF_MEM(list->pNodeList); CHECK_OUT_OF_MEM(list->pNodeList);
nodesListAppend(list->pNodeList, p1); pCxt->errCode = nodesListAppend(list->pNodeList, p1);
nodesListAppend(list->pNodeList, p2); CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListAppend(list->pNodeList, p2);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)list; return (SNode*)list;
} }
@ -1121,7 +1127,8 @@ SNode* createGroupingSetNode(SAstCreateContext* pCxt, SNode* pNode) {
CHECK_OUT_OF_MEM(groupingSet); CHECK_OUT_OF_MEM(groupingSet);
groupingSet->groupingSetType = GP_TYPE_NORMAL; groupingSet->groupingSetType = GP_TYPE_NORMAL;
groupingSet->pParameterList = nodesMakeList(); groupingSet->pParameterList = nodesMakeList();
nodesListAppend(groupingSet->pParameterList, pNode); pCxt->errCode = nodesListAppend(groupingSet->pParameterList, pNode);
CHECK_PARSER_STATUS(pCxt);
return (SNode*)groupingSet; return (SNode*)groupingSet;
} }
@ -1842,7 +1849,7 @@ SNode* createCreateSubTableFromFileClause(SAstCreateContext* pCxt, bool ignoreEx
pStmt->ignoreExists = ignoreExists; pStmt->ignoreExists = ignoreExists;
pStmt->pSpecificTags = pSpecificTags; pStmt->pSpecificTags = pSpecificTags;
if (TK_NK_STRING == pFilePath->type) { if (TK_NK_STRING == pFilePath->type) {
trimString(pFilePath->z, pFilePath->n, pStmt->filePath, PATH_MAX); (void)trimString(pFilePath->z, pFilePath->n, pStmt->filePath, PATH_MAX);
} else { } else {
strncpy(pStmt->filePath, pFilePath->z, pFilePath->n); strncpy(pStmt->filePath, pFilePath->z, pFilePath->n);
} }
@ -2087,7 +2094,7 @@ SNode* createShowTablesStmt(SAstCreateContext* pCxt, SShowTablesOption option, S
pDbName = createIdentifierValueNode(pCxt, &option.dbName); pDbName = createIdentifierValueNode(pCxt, &option.dbName);
} }
SNode* pStmt = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_TABLES_STMT, pDbName, pTbName, tableCondType); SNode* pStmt = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_TABLES_STMT, pDbName, pTbName, tableCondType);
setShowKind(pCxt, pStmt, option.kind); (void)setShowKind(pCxt, pStmt, option.kind);
return pStmt; return pStmt;
} }
@ -2408,9 +2415,9 @@ SNode* createAlterDnodeStmt(SAstCreateContext* pCxt, const SToken* pDnode, const
} else { } else {
pStmt->dnodeId = -1; pStmt->dnodeId = -1;
} }
trimString(pConfig->z, pConfig->n, pStmt->config, sizeof(pStmt->config)); (void)trimString(pConfig->z, pConfig->n, pStmt->config, sizeof(pStmt->config));
if (NULL != pValue) { if (NULL != pValue) {
trimString(pValue->z, pValue->n, pStmt->value, sizeof(pStmt->value)); (void)trimString(pValue->z, pValue->n, pStmt->value, sizeof(pStmt->value));
} }
return (SNode*)pStmt; return (SNode*)pStmt;
} }
@ -2591,9 +2598,9 @@ SNode* createAlterClusterStmt(SAstCreateContext* pCxt, const SToken* pConfig, co
CHECK_PARSER_STATUS(pCxt); CHECK_PARSER_STATUS(pCxt);
SAlterClusterStmt* pStmt = (SAlterClusterStmt*)nodesMakeNode(QUERY_NODE_ALTER_CLUSTER_STMT); SAlterClusterStmt* pStmt = (SAlterClusterStmt*)nodesMakeNode(QUERY_NODE_ALTER_CLUSTER_STMT);
CHECK_OUT_OF_MEM(pStmt); CHECK_OUT_OF_MEM(pStmt);
trimString(pConfig->z, pConfig->n, pStmt->config, sizeof(pStmt->config)); (void)trimString(pConfig->z, pConfig->n, pStmt->config, sizeof(pStmt->config));
if (NULL != pValue) { if (NULL != pValue) {
trimString(pValue->z, pValue->n, pStmt->value, sizeof(pStmt->value)); (void)trimString(pValue->z, pValue->n, pStmt->value, sizeof(pStmt->value));
} }
return (SNode*)pStmt; return (SNode*)pStmt;
} }
@ -2602,9 +2609,9 @@ SNode* createAlterLocalStmt(SAstCreateContext* pCxt, const SToken* pConfig, cons
CHECK_PARSER_STATUS(pCxt); CHECK_PARSER_STATUS(pCxt);
SAlterLocalStmt* pStmt = (SAlterLocalStmt*)nodesMakeNode(QUERY_NODE_ALTER_LOCAL_STMT); SAlterLocalStmt* pStmt = (SAlterLocalStmt*)nodesMakeNode(QUERY_NODE_ALTER_LOCAL_STMT);
CHECK_OUT_OF_MEM(pStmt); CHECK_OUT_OF_MEM(pStmt);
trimString(pConfig->z, pConfig->n, pStmt->config, sizeof(pStmt->config)); (void)trimString(pConfig->z, pConfig->n, pStmt->config, sizeof(pStmt->config));
if (NULL != pValue) { if (NULL != pValue) {
trimString(pValue->z, pValue->n, pStmt->value, sizeof(pStmt->value)); (void)trimString(pValue->z, pValue->n, pStmt->value, sizeof(pStmt->value));
} }
return (SNode*)pStmt; return (SNode*)pStmt;
} }
@ -2869,7 +2876,7 @@ SNode* createKillQueryStmt(SAstCreateContext* pCxt, const SToken* pQueryId) {
CHECK_PARSER_STATUS(pCxt); CHECK_PARSER_STATUS(pCxt);
SKillQueryStmt* pStmt = (SKillQueryStmt*)nodesMakeNode(QUERY_NODE_KILL_QUERY_STMT); SKillQueryStmt* pStmt = (SKillQueryStmt*)nodesMakeNode(QUERY_NODE_KILL_QUERY_STMT);
CHECK_OUT_OF_MEM(pStmt); CHECK_OUT_OF_MEM(pStmt);
trimString(pQueryId->z, pQueryId->n, pStmt->queryId, sizeof(pStmt->queryId) - 1); (void)trimString(pQueryId->z, pQueryId->n, pStmt->queryId, sizeof(pStmt->queryId) - 1);
return (SNode*)pStmt; return (SNode*)pStmt;
} }

1600
source/libs/parser/src/parTranslater.c
View File

File diff suppressed because it is too large Load Diff

4
source/libs/qcom/src/queryUtil.c
View File

@ -158,7 +158,9 @@ int32_t cleanupTaskQueue() {
} }
int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code) { int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code) {
SSchedMsg* pSchedMsg = taosAllocateQitem(sizeof(SSchedMsg), DEF_QITEM, 0); SSchedMsg* pSchedMsg;
int32_t rc = taosAllocateQitem(sizeof(SSchedMsg), DEF_QITEM, 0, (void **)&pSchedMsg);
if (rc) return rc;
pSchedMsg->fp = NULL; pSchedMsg->fp = NULL;
pSchedMsg->ahandle = execFn; pSchedMsg->ahandle = execFn;
pSchedMsg->thandle = execParam; pSchedMsg->thandle = execParam;

10
source/libs/stream/src/streamCheckpoint.c
View File

@ -31,9 +31,11 @@ static void checkpointTriggerMonitorFn(void* param, void* tmrId);
SStreamDataBlock* createChkptTriggerBlock(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId, SStreamDataBlock* createChkptTriggerBlock(SStreamTask* pTask, int32_t checkpointType, int64_t checkpointId,
int32_t transId, int32_t srcTaskId) { int32_t transId, int32_t srcTaskId) {
SStreamDataBlock* pChkpoint = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SSDataBlock)); SStreamDataBlock* pChkpoint;
if (pChkpoint == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; int32_t code = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SSDataBlock), (void**)&pChkpoint);
if (code) {
terrno = code;
return NULL; return NULL;
} }
@ -425,7 +427,7 @@ int32_t streamTaskProcessCheckpointReadyRsp(SStreamTask* pTask, int32_t upstream
} }
void streamTaskClearCheckInfo(SStreamTask* pTask, bool clearChkpReadyMsg) { void streamTaskClearCheckInfo(SStreamTask* pTask, bool clearChkpReadyMsg) {
pTask->chkInfo.startTs = 0; // clear the recorded start time pTask->chkInfo.startTs = 0; // clear the recorded start time
streamTaskOpenAllUpstreamInput(pTask); // open inputQ for all upstream tasks streamTaskOpenAllUpstreamInput(pTask); // open inputQ for all upstream tasks
taosThreadMutexLock(&pTask->chkInfo.pActiveInfo->lock); taosThreadMutexLock(&pTask->chkInfo.pActiveInfo->lock);

44
source/libs/stream/src/streamData.c
View File

@ -16,8 +16,11 @@
#include "streamInt.h" #include "streamInt.h"
SStreamDataBlock* createStreamBlockFromDispatchMsg(const SStreamDispatchReq* pReq, int32_t blockType, int32_t srcVg) { SStreamDataBlock* createStreamBlockFromDispatchMsg(const SStreamDispatchReq* pReq, int32_t blockType, int32_t srcVg) {
SStreamDataBlock* pData = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, pReq->totalLen); SStreamDataBlock* pData;
if (pData == NULL) {
int32_t code = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, pReq->totalLen, (void**)&pData);
if (code) {
terrno = code;
return NULL; return NULL;
} }
@ -34,7 +37,7 @@ SStreamDataBlock* createStreamBlockFromDispatchMsg(const SStreamDispatchReq* pRe
ASSERT((pReq->blockNum == taosArrayGetSize(pReq->data)) && (pReq->blockNum == taosArrayGetSize(pReq->dataLen))); ASSERT((pReq->blockNum == taosArrayGetSize(pReq->data)) && (pReq->blockNum == taosArrayGetSize(pReq->dataLen)));
for (int32_t i = 0; i < blockNum; i++) { for (int32_t i = 0; i < blockNum; i++) {
SRetrieveTableRsp* pRetrieve = (SRetrieveTableRsp*) taosArrayGetP(pReq->data, i); SRetrieveTableRsp* pRetrieve = (SRetrieveTableRsp*)taosArrayGetP(pReq->data, i);
SSDataBlock* pDataBlock = taosArrayGet(pArray, i); SSDataBlock* pDataBlock = taosArrayGet(pArray, i);
int32_t compLen = *(int32_t*)pRetrieve->data; int32_t compLen = *(int32_t*)pRetrieve->data;
@ -42,7 +45,7 @@ SStreamDataBlock* createStreamBlockFromDispatchMsg(const SStreamDispatchReq* pRe
char* pInput = pRetrieve->data + PAYLOAD_PREFIX_LEN; char* pInput = pRetrieve->data + PAYLOAD_PREFIX_LEN;
if (pRetrieve->compressed && compLen < fullLen) { if (pRetrieve->compressed && compLen < fullLen) {
char* p = taosMemoryMalloc(fullLen); char* p = taosMemoryMalloc(fullLen);
int32_t len = tsDecompressString(pInput, compLen, 1, p, fullLen, ONE_STAGE_COMP, NULL, 0); int32_t len = tsDecompressString(pInput, compLen, 1, p, fullLen, ONE_STAGE_COMP, NULL, 0);
ASSERT(len == fullLen); ASSERT(len == fullLen);
pInput = p; pInput = p;
@ -69,10 +72,14 @@ SStreamDataBlock* createStreamBlockFromDispatchMsg(const SStreamDispatchReq* pRe
return pData; return pData;
} }
SStreamDataBlock* createStreamBlockFromResults(SStreamQueueItem* pItem, SStreamTask* pTask, int64_t resultSize, SArray* pRes) { SStreamDataBlock* createStreamBlockFromResults(SStreamQueueItem* pItem, SStreamTask* pTask, int64_t resultSize,
SStreamDataBlock* pStreamBlocks = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, resultSize); SArray* pRes) {
if (pStreamBlocks == NULL) { SStreamDataBlock* pStreamBlocks;
int32_t code = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, resultSize, (void**)&pStreamBlocks);
if (code) {
taosArrayClearEx(pRes, (FDelete)blockDataFreeRes); taosArrayClearEx(pRes, (FDelete)blockDataFreeRes);
terrno = code;
return NULL; return NULL;
} }
@ -132,8 +139,10 @@ int32_t streamRetrieveReqToData(const SStreamRetrieveReq* pReq, SStreamDataBlock
} }
SStreamDataSubmit* streamDataSubmitNew(SPackedData* pData, int32_t type) { SStreamDataSubmit* streamDataSubmitNew(SPackedData* pData, int32_t type) {
SStreamDataSubmit* pDataSubmit = (SStreamDataSubmit*)taosAllocateQitem(sizeof(SStreamDataSubmit), DEF_QITEM, pData->msgLen); SStreamDataSubmit* pDataSubmit;
if (pDataSubmit == NULL) { int32_t code = taosAllocateQitem(sizeof(SStreamDataSubmit), DEF_QITEM, pData->msgLen, (void**)&pDataSubmit);
if (code) {
terrno = code;
return NULL; return NULL;
} }
@ -151,8 +160,11 @@ void streamDataSubmitDestroy(SStreamDataSubmit* pDataSubmit) {
} }
SStreamMergedSubmit* streamMergedSubmitNew() { SStreamMergedSubmit* streamMergedSubmitNew() {
SStreamMergedSubmit* pMerged = (SStreamMergedSubmit*)taosAllocateQitem(sizeof(SStreamMergedSubmit), DEF_QITEM, 0); SStreamMergedSubmit* pMerged;
if (pMerged == NULL) {
int32_t code = taosAllocateQitem(sizeof(SStreamMergedSubmit), DEF_QITEM, 0, (void**)&pMerged);
if (code) {
terrno = code;
return NULL; return NULL;
} }
@ -178,7 +190,7 @@ int32_t streamMergeSubmit(SStreamMergedSubmit* pMerged, SStreamDataSubmit* pSubm
// todo handle memory error // todo handle memory error
SStreamQueueItem* streamQueueMergeQueueItem(SStreamQueueItem* dst, SStreamQueueItem* pElem) { SStreamQueueItem* streamQueueMergeQueueItem(SStreamQueueItem* dst, SStreamQueueItem* pElem) {
terrno = 0; terrno = 0;
if (dst->type == STREAM_INPUT__DATA_BLOCK && pElem->type == STREAM_INPUT__DATA_BLOCK) { if (dst->type == STREAM_INPUT__DATA_BLOCK && pElem->type == STREAM_INPUT__DATA_BLOCK) {
SStreamDataBlock* pBlock = (SStreamDataBlock*)dst; SStreamDataBlock* pBlock = (SStreamDataBlock*)dst;
SStreamDataBlock* pBlockSrc = (SStreamDataBlock*)pElem; SStreamDataBlock* pBlockSrc = (SStreamDataBlock*)pElem;
@ -212,7 +224,8 @@ SStreamQueueItem* streamQueueMergeQueueItem(SStreamQueueItem* dst, SStreamQueueI
taosFreeQitem(pElem); taosFreeQitem(pElem);
return (SStreamQueueItem*)pMerged; return (SStreamQueueItem*)pMerged;
} else { } else {
stDebug("block type:%s not merged with existed blocks list, type:%d", streamQueueItemGetTypeStr(pElem->type), dst->type); stDebug("block type:%s not merged with existed blocks list, type:%d", streamQueueItemGetTypeStr(pElem->type),
dst->type);
return NULL; return NULL;
} }
} }
@ -245,8 +258,9 @@ void streamFreeQitem(SStreamQueueItem* data) {
SStreamRefDataBlock* pRefBlock = (SStreamRefDataBlock*)data; SStreamRefDataBlock* pRefBlock = (SStreamRefDataBlock*)data;
blockDataDestroy(pRefBlock->pBlock); blockDataDestroy(pRefBlock->pBlock);
taosFreeQitem(pRefBlock); taosFreeQitem(pRefBlock);
} else if (type == STREAM_INPUT__CHECKPOINT || type == STREAM_INPUT__CHECKPOINT_TRIGGER || type == STREAM_INPUT__TRANS_STATE) { } else if (type == STREAM_INPUT__CHECKPOINT || type == STREAM_INPUT__CHECKPOINT_TRIGGER ||
SStreamDataBlock* pBlock = (SStreamDataBlock*) data; type == STREAM_INPUT__TRANS_STATE) {
SStreamDataBlock* pBlock = (SStreamDataBlock*)data;
taosArrayDestroyEx(pBlock->blocks, freeItems); taosArrayDestroyEx(pBlock->blocks, freeItems);
taosFreeQitem(pBlock); taosFreeQitem(pBlock);
} }

56
source/libs/stream/src/streamDispatch.c
View File

@ -1162,10 +1162,10 @@ void streamClearChkptReadyMsg(SActiveCheckpointInfo* pActiveInfo) {
static int32_t handleDispatchSuccessRsp(SStreamTask* pTask, int32_t downstreamId, int32_t downstreamNodeId) { static int32_t handleDispatchSuccessRsp(SStreamTask* pTask, int32_t downstreamId, int32_t downstreamNodeId) {
stDebug("s-task:%s destroy dispatch msg:%p", pTask->id.idStr, pTask->msgInfo.pData); stDebug("s-task:%s destroy dispatch msg:%p", pTask->id.idStr, pTask->msgInfo.pData);
bool delayDispatch = (pTask->msgInfo.dispatchMsgType == STREAM_INPUT__CHECKPOINT_TRIGGER);
clearBufferedDispatchMsg(pTask);
int64_t el = taosGetTimestampMs() - pTask->msgInfo.startTs; int64_t el = taosGetTimestampMs() - pTask->msgInfo.startTs;
bool delayDispatch = (pTask->msgInfo.dispatchMsgType == STREAM_INPUT__CHECKPOINT_TRIGGER);
clearBufferedDispatchMsg(pTask);
// put data into inputQ of current task is also allowed // put data into inputQ of current task is also allowed
if (pTask->inputq.status == TASK_INPUT_STATUS__BLOCKED) { if (pTask->inputq.status == TASK_INPUT_STATUS__BLOCKED) {
@ -1189,13 +1189,24 @@ static int32_t handleDispatchSuccessRsp(SStreamTask* pTask, int32_t downstreamId
return 0; return 0;
} }
static int32_t setDispatchRspInfo(SDispatchMsgInfo* pMsgInfo, int32_t vgId, int32_t code, int64_t now, const char* id) { static bool setDispatchRspInfo(SDispatchMsgInfo* pMsgInfo, int32_t vgId, int32_t code, int64_t now, int32_t* pNotRsp, const char* id) {
int32_t numOfRsp = 0; int32_t numOfRsp = 0;
bool alreadySet = false; bool alreadySet = false;
bool updated = false; bool updated = false;
bool allRsp = false;
*pNotRsp = 0;
taosThreadMutexLock(&pMsgInfo->lock); taosThreadMutexLock(&pMsgInfo->lock);
for (int32_t j = 0; j < taosArrayGetSize(pMsgInfo->pSendInfo); ++j) { int32_t numOfDispatchBranch = taosArrayGetSize(pMsgInfo->pSendInfo);
for(int32_t i = 0; i < numOfDispatchBranch; ++i) {
SDispatchEntry* pEntry = taosArrayGet(pMsgInfo->pSendInfo, i);
if (pEntry->rspTs != -1) {
numOfRsp += 1;
}
}
for (int32_t j = 0; j < numOfDispatchBranch; ++j) {
SDispatchEntry* pEntry = taosArrayGet(pMsgInfo->pSendInfo, j); SDispatchEntry* pEntry = taosArrayGet(pMsgInfo->pSendInfo, j);
if (pEntry->nodeId == vgId) { if (pEntry->nodeId == vgId) {
ASSERT(!alreadySet); ASSERT(!alreadySet);
@ -1203,18 +1214,20 @@ static int32_t setDispatchRspInfo(SDispatchMsgInfo* pMsgInfo, int32_t vgId, int3
pEntry->status = code; pEntry->status = code;
alreadySet = true; alreadySet = true;
updated = true; updated = true;
stDebug("s-task:%s record the rsp recv, ts:%" PRId64 " code:%d, idx:%d", id, now, code, j);
}
if (pEntry->rspTs != -1) {
numOfRsp += 1; numOfRsp += 1;
stDebug("s-task:%s record the rsp recv, ts:%" PRId64 " code:%d, idx:%d, total recv:%d/%d", id, now, code, j,
numOfRsp, numOfDispatchBranch);
} }
} }
taosThreadMutexUnlock(&pMsgInfo->lock); *pNotRsp = numOfDispatchBranch - numOfRsp;
ASSERT(updated); allRsp = (numOfRsp == numOfDispatchBranch);
return numOfRsp; taosThreadMutexUnlock(&pMsgInfo->lock);
ASSERT(updated);
return allRsp;
} }
bool isDispatchRspTimeout(SDispatchEntry* pEntry, int64_t now) { bool isDispatchRspTimeout(SDispatchEntry* pEntry, int64_t now) {
@ -1240,7 +1253,8 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
int32_t vgId = pTask->pMeta->vgId; int32_t vgId = pTask->pMeta->vgId;
SDispatchMsgInfo* pMsgInfo = &pTask->msgInfo; SDispatchMsgInfo* pMsgInfo = &pTask->msgInfo;
int64_t now = taosGetTimestampMs(); int64_t now = taosGetTimestampMs();
int32_t totalRsp = 0; bool allRsp = false;
int32_t notRsp = 0;
taosThreadMutexLock(&pMsgInfo->lock); taosThreadMutexLock(&pMsgInfo->lock);
int32_t msgId = pMsgInfo->msgId; int32_t msgId = pMsgInfo->msgId;
@ -1269,18 +1283,18 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
if (code == TSDB_CODE_STREAM_TASK_NOT_EXIST) { // destination task does not exist, not retry anymore if (code == TSDB_CODE_STREAM_TASK_NOT_EXIST) { // destination task does not exist, not retry anymore
stError("s-task:%s failed to dispatch msg to task:0x%x(vgId:%d), msgId:%d no retry, since task destroyed already", stError("s-task:%s failed to dispatch msg to task:0x%x(vgId:%d), msgId:%d no retry, since task destroyed already",
id, pRsp->downstreamTaskId, pRsp->downstreamNodeId, msgId); id, pRsp->downstreamTaskId, pRsp->downstreamNodeId, msgId);
totalRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, TSDB_CODE_SUCCESS, now, id); allRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, TSDB_CODE_SUCCESS, now, &notRsp, id);
} else { } else {
stError("s-task:%s failed to dispatch msgId:%d to task:0x%x(vgId:%d), code:%s, add to retry list", id, msgId, stError("s-task:%s failed to dispatch msgId:%d to task:0x%x(vgId:%d), code:%s, add to retry list", id, msgId,
pRsp->downstreamTaskId, pRsp->downstreamNodeId, tstrerror(code)); pRsp->downstreamTaskId, pRsp->downstreamNodeId, tstrerror(code));
totalRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, code, now, id); allRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, code, now, &notRsp, id);
} }
} else { // code == 0 } else { // code == 0
if (pRsp->inputStatus == TASK_INPUT_STATUS__BLOCKED) { if (pRsp->inputStatus == TASK_INPUT_STATUS__BLOCKED) {
pTask->inputq.status = TASK_INPUT_STATUS__BLOCKED; pTask->inputq.status = TASK_INPUT_STATUS__BLOCKED;
// block the input of current task, to push pressure to upstream // block the input of current task, to push pressure to upstream
totalRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, pRsp->inputStatus, now, id); allRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, pRsp->inputStatus, now, &notRsp, id);
stTrace("s-task:%s inputQ of downstream task:0x%x(vgId:%d) is full, wait for retry dispatch", id, stTrace("s-task:%s inputQ of downstream task:0x%x(vgId:%d) is full, wait for retry dispatch", id,
pRsp->downstreamTaskId, pRsp->downstreamNodeId); pRsp->downstreamTaskId, pRsp->downstreamNodeId);
} else { } else {
@ -1292,7 +1306,7 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
id, pRsp->downstreamTaskId, pRsp->downstreamNodeId); id, pRsp->downstreamTaskId, pRsp->downstreamNodeId);
} }
totalRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, TSDB_CODE_SUCCESS, now, id); allRsp = setDispatchRspInfo(pMsgInfo, pRsp->downstreamNodeId, TSDB_CODE_SUCCESS, now, &notRsp, id);
{ {
bool delayDispatch = (pMsgInfo->dispatchMsgType == STREAM_INPUT__CHECKPOINT_TRIGGER); bool delayDispatch = (pMsgInfo->dispatchMsgType == STREAM_INPUT__CHECKPOINT_TRIGGER);
@ -1317,13 +1331,11 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
} }
} }
int32_t notRsp = taosArrayGetSize(pMsgInfo->pSendInfo) - totalRsp;
if (pTask->outputInfo.type == TASK_OUTPUT__SHUFFLE_DISPATCH) { if (pTask->outputInfo.type == TASK_OUTPUT__SHUFFLE_DISPATCH) {
if (notRsp > 0) { if (!allRsp) {
stDebug( stDebug(
"s-task:%s recv dispatch rsp, msgId:%d from 0x%x(vgId:%d), downstream task input status:%d code:%s, " "s-task:%s recv dispatch rsp, msgId:%d from 0x%x(vgId:%d), downstream task input status:%d code:%s, "
"waiting " "waiting for %d rsp",
"for %d rsp",
id, msgId, pRsp->downstreamTaskId, pRsp->downstreamNodeId, pRsp->inputStatus, tstrerror(code), notRsp); id, msgId, pRsp->downstreamTaskId, pRsp->downstreamNodeId, pRsp->inputStatus, tstrerror(code), notRsp);
} else { } else {
stDebug( stDebug(
@ -1337,7 +1349,7 @@ int32_t streamProcessDispatchRsp(SStreamTask* pTask, SStreamDispatchRsp* pRsp, i
} }
// all msg rsp already, continue // all msg rsp already, continue
if (notRsp == 0) { if (allRsp) {
ASSERT(pTask->outputq.status == TASK_OUTPUT_STATUS__WAIT); ASSERT(pTask->outputq.status == TASK_OUTPUT_STATUS__WAIT);
// we need to re-try send dispatch msg to downstream tasks // we need to re-try send dispatch msg to downstream tasks

66
source/libs/stream/src/streamQueue.c
View File

@ -15,8 +15,8 @@
#include "streamInt.h" #include "streamInt.h"
#define MAX_STREAM_EXEC_BATCH_NUM 32 #define MAX_STREAM_EXEC_BATCH_NUM 32
#define MAX_SMOOTH_BURST_RATIO 5 // 5 sec #define MAX_SMOOTH_BURST_RATIO 5 // 5 sec
// todo refactor: // todo refactor:
// read data from input queue // read data from input queue
@ -42,18 +42,26 @@ static void streamQueueCleanup(SStreamQueue* pQueue) {
static void* streamQueueCurItem(SStreamQueue* queue) { return queue->qItem; } static void* streamQueueCurItem(SStreamQueue* queue) { return queue->qItem; }
SStreamQueue* streamQueueOpen(int64_t cap) { SStreamQueue* streamQueueOpen(int64_t cap) {
int32_t code;
SStreamQueue* pQueue = taosMemoryCalloc(1, sizeof(SStreamQueue)); SStreamQueue* pQueue = taosMemoryCalloc(1, sizeof(SStreamQueue));
if (pQueue == NULL) { if (pQueue == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL; return NULL;
} }
pQueue->pQueue = taosOpenQueue(); code = taosOpenQueue(&pQueue->pQueue);
pQueue->qall = taosAllocateQall(); if (code) {
taosMemoryFreeClear(pQueue);
terrno = code;
return NULL;
}
if (pQueue->pQueue == NULL || pQueue->qall == NULL) { code = taosAllocateQall(&pQueue->qall);
if (pQueue->pQueue) taosCloseQueue(pQueue->pQueue); if (code) {
if (pQueue->qall) taosFreeQall(pQueue->qall); taosCloseQueue(pQueue->pQueue);
taosMemoryFree(pQueue); taosMemoryFree(pQueue);
terrno = code;
return NULL; return NULL;
} }
@ -64,7 +72,8 @@ SStreamQueue* streamQueueOpen(int64_t cap) {
} }
void streamQueueClose(SStreamQueue* pQueue, int32_t taskId) { void streamQueueClose(SStreamQueue* pQueue, int32_t taskId) {
stDebug("s-task:0x%x free the queue:%p, items in queue:%d", taskId, pQueue->pQueue, taosQueueItemSize(pQueue->pQueue)); stDebug("s-task:0x%x free the queue:%p, items in queue:%d", taskId, pQueue->pQueue,
taosQueueItemSize(pQueue->pQueue));
streamQueueCleanup(pQueue); streamQueueCleanup(pQueue);
taosFreeQall(pQueue->qall); taosFreeQall(pQueue->qall);
@ -129,12 +138,12 @@ int32_t streamQueueGetItemSize(const SStreamQueue* pQueue) {
} }
int32_t streamQueueItemGetSize(const SStreamQueueItem* pItem) { int32_t streamQueueItemGetSize(const SStreamQueueItem* pItem) {
STaosQnode* p = (STaosQnode*)((char*) pItem - sizeof(STaosQnode)); STaosQnode* p = (STaosQnode*)((char*)pItem - sizeof(STaosQnode));
return p->dataSize; return p->dataSize;
} }
void streamQueueItemIncSize(const SStreamQueueItem* pItem, int32_t size) { void streamQueueItemIncSize(const SStreamQueueItem* pItem, int32_t size) {
STaosQnode* p = (STaosQnode*)((char*) pItem - sizeof(STaosQnode)); STaosQnode* p = (STaosQnode*)((char*)pItem - sizeof(STaosQnode));
p->dataSize += size; p->dataSize += size;
} }
@ -152,7 +161,7 @@ const char* streamQueueItemGetTypeStr(int32_t type) {
} }
EExtractDataCode streamTaskGetDataFromInputQ(SStreamTask* pTask, SStreamQueueItem** pInput, int32_t* numOfBlocks, EExtractDataCode streamTaskGetDataFromInputQ(SStreamTask* pTask, SStreamQueueItem** pInput, int32_t* numOfBlocks,
int32_t* blockSize) { int32_t* blockSize) {
const char* id = pTask->id.idStr; const char* id = pTask->id.idStr;
int32_t taskLevel = pTask->info.taskLevel; int32_t taskLevel = pTask->info.taskLevel;
@ -174,7 +183,6 @@ EExtractDataCode streamTaskGetDataFromInputQ(SStreamTask* pTask, SStreamQueueIte
SStreamQueueItem* qItem = streamQueueNextItem(pTask->inputq.queue); SStreamQueueItem* qItem = streamQueueNextItem(pTask->inputq.queue);
if (qItem == NULL) { if (qItem == NULL) {
// restore the token to bucket // restore the token to bucket
if (*numOfBlocks > 0) { if (*numOfBlocks > 0) {
*blockSize = streamQueueItemGetSize(*pInput); *blockSize = streamQueueItemGetSize(*pInput);
@ -203,7 +211,7 @@ EExtractDataCode streamTaskGetDataFromInputQ(SStreamTask* pTask, SStreamQueueIte
*numOfBlocks = 1; *numOfBlocks = 1;
*pInput = qItem; *pInput = qItem;
return EXEC_CONTINUE; return EXEC_CONTINUE;
} else { // previous existed blocks needs to be handle, before handle the checkpoint msg block } else { // previous existed blocks needs to be handle, before handle the checkpoint msg block
stDebug("s-task:%s %s msg extracted, handle previous blocks, numOfBlocks:%d", id, p, *numOfBlocks); stDebug("s-task:%s %s msg extracted, handle previous blocks, numOfBlocks:%d", id, p, *numOfBlocks);
*blockSize = streamQueueItemGetSize(*pInput); *blockSize = streamQueueItemGetSize(*pInput);
if (taskLevel == TASK_LEVEL__SINK) { if (taskLevel == TASK_LEVEL__SINK) {
@ -223,7 +231,7 @@ EExtractDataCode streamTaskGetDataFromInputQ(SStreamTask* pTask, SStreamQueueIte
if (newRet == NULL) { if (newRet == NULL) {
if (terrno != 0) { if (terrno != 0) {
stError("s-task:%s failed to merge blocks from inputQ, numOfBlocks:%d, code:%s", id, *numOfBlocks, stError("s-task:%s failed to merge blocks from inputQ, numOfBlocks:%d, code:%s", id, *numOfBlocks,
tstrerror(terrno)); tstrerror(terrno));
} }
*blockSize = streamQueueItemGetSize(*pInput); *blockSize = streamQueueItemGetSize(*pInput);
@ -284,14 +292,14 @@ int32_t streamTaskPutDataIntoInputQ(SStreamTask* pTask, SStreamQueueItem* pItem)
// use the local variable to avoid the pItem be freed by other threads, since it has been put into queue already. // use the local variable to avoid the pItem be freed by other threads, since it has been put into queue already.
stDebug("s-task:%s submit enqueue msgLen:%d ver:%" PRId64 ", total in queue:%d, size:%.2fMiB", pTask->id.idStr, stDebug("s-task:%s submit enqueue msgLen:%d ver:%" PRId64 ", total in queue:%d, size:%.2fMiB", pTask->id.idStr,
msgLen, ver, total, size + SIZE_IN_MiB(msgLen)); msgLen, ver, total, size + SIZE_IN_MiB(msgLen));
} else if (type == STREAM_INPUT__DATA_BLOCK || type == STREAM_INPUT__DATA_RETRIEVE || } else if (type == STREAM_INPUT__DATA_BLOCK || type == STREAM_INPUT__DATA_RETRIEVE ||
type == STREAM_INPUT__REF_DATA_BLOCK) { type == STREAM_INPUT__REF_DATA_BLOCK) {
if (streamQueueIsFull(pTask->inputq.queue)) { if (streamQueueIsFull(pTask->inputq.queue)) {
double size = SIZE_IN_MiB(taosQueueMemorySize(pQueue)); double size = SIZE_IN_MiB(taosQueueMemorySize(pQueue));
stTrace("s-task:%s input queue is full, capacity:%d size:%d MiB, current(blocks:%d, size:%.2fMiB) abort", stTrace("s-task:%s input queue is full, capacity:%d size:%d MiB, current(blocks:%d, size:%.2fMiB) abort",
pTask->id.idStr, STREAM_TASK_QUEUE_CAPACITY, STREAM_TASK_QUEUE_CAPACITY_IN_SIZE, total, size); pTask->id.idStr, STREAM_TASK_QUEUE_CAPACITY, STREAM_TASK_QUEUE_CAPACITY_IN_SIZE, total, size);
streamFreeQitem(pItem); streamFreeQitem(pItem);
return -1; return -1;
} }
@ -314,7 +322,7 @@ int32_t streamTaskPutDataIntoInputQ(SStreamTask* pTask, SStreamQueueItem* pItem)
double size = SIZE_IN_MiB(taosQueueMemorySize(pQueue)); double size = SIZE_IN_MiB(taosQueueMemorySize(pQueue));
stDebug("s-task:%s level:%d %s blockdata enqueue, total in queue:%d, size:%.2fMiB", pTask->id.idStr, stDebug("s-task:%s level:%d %s blockdata enqueue, total in queue:%d, size:%.2fMiB", pTask->id.idStr,
pTask->info.taskLevel, streamQueueItemGetTypeStr(type), total, size); pTask->info.taskLevel, streamQueueItemGetTypeStr(type), total, size);
} else if (type == STREAM_INPUT__GET_RES) { } else if (type == STREAM_INPUT__GET_RES) {
// use the default memory limit, refactor later. // use the default memory limit, refactor later.
int32_t code = taosWriteQitem(pQueue, pItem); int32_t code = taosWriteQitem(pQueue, pItem);
@ -332,16 +340,20 @@ int32_t streamTaskPutDataIntoInputQ(SStreamTask* pTask, SStreamQueueItem* pItem)
if (type != STREAM_INPUT__GET_RES && type != STREAM_INPUT__CHECKPOINT && type != STREAM_INPUT__CHECKPOINT_TRIGGER && if (type != STREAM_INPUT__GET_RES && type != STREAM_INPUT__CHECKPOINT && type != STREAM_INPUT__CHECKPOINT_TRIGGER &&
(pTask->info.delaySchedParam != 0)) { (pTask->info.delaySchedParam != 0)) {
atomic_val_compare_exchange_8(&pTask->schedInfo.status, TASK_TRIGGER_STATUS__INACTIVE, TASK_TRIGGER_STATUS__ACTIVE); atomic_val_compare_exchange_8(&pTask->schedInfo.status, TASK_TRIGGER_STATUS__INACTIVE, TASK_TRIGGER_STATUS__ACTIVE);
stDebug("s-task:%s new data arrived, active the sched-trigger, triggerStatus:%d", pTask->id.idStr, pTask->schedInfo.status); stDebug("s-task:%s new data arrived, active the sched-trigger, triggerStatus:%d", pTask->id.idStr,
pTask->schedInfo.status);
} }
return 0; return 0;
} }
int32_t streamTaskPutTranstateIntoInputQ(SStreamTask* pTask) { int32_t streamTaskPutTranstateIntoInputQ(SStreamTask* pTask) {
SStreamDataBlock* pTranstate = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SSDataBlock)); int32_t code;
if (pTranstate == NULL) { SStreamDataBlock* pTranstate;
return TSDB_CODE_OUT_OF_MEMORY;
code = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SSDataBlock), (void**)&pTranstate);
if (code) {
return code;
} }
SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock)); SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));
@ -371,13 +383,13 @@ int32_t streamTaskPutTranstateIntoInputQ(SStreamTask* pTask) {
// the result should be put into the outputQ in any cases, the result may be lost otherwise. // the result should be put into the outputQ in any cases, the result may be lost otherwise.
int32_t streamTaskPutDataIntoOutputQ(SStreamTask* pTask, SStreamDataBlock* pBlock) { int32_t streamTaskPutDataIntoOutputQ(SStreamTask* pTask, SStreamDataBlock* pBlock) {
STaosQueue* pQueue = pTask->outputq.queue->pQueue; STaosQueue* pQueue = pTask->outputq.queue->pQueue;
int32_t code = taosWriteQitem(pQueue, pBlock); int32_t code = taosWriteQitem(pQueue, pBlock);
int32_t total = streamQueueGetNumOfItems(pTask->outputq.queue); int32_t total = streamQueueGetNumOfItems(pTask->outputq.queue);
double size = SIZE_IN_MiB(taosQueueMemorySize(pQueue)); double size = SIZE_IN_MiB(taosQueueMemorySize(pQueue));
if (code != 0) { if (code != 0) {
stError("s-task:%s failed to put res into outputQ, outputQ items:%d, size:%.2fMiB code:%s, result lost", stError("s-task:%s failed to put res into outputQ, outputQ items:%d, size:%.2fMiB code:%s, result lost",
pTask->id.idStr, total + 1, size, tstrerror(code)); pTask->id.idStr, total + 1, size, tstrerror(code));
} else { } else {
if (streamQueueIsFull(pTask->outputq.queue)) { if (streamQueueIsFull(pTask->outputq.queue)) {
stWarn( stWarn(
@ -427,7 +439,7 @@ static void fillTokenBucket(STokenBucket* pBucket, const char* id) {
// increase the new available quota as time goes on // increase the new available quota as time goes on
int64_t deltaQuota = now - pBucket->quotaFillTimestamp; int64_t deltaQuota = now - pBucket->quotaFillTimestamp;
double incSize = (deltaQuota / 1000.0) * pBucket->quotaRate; double incSize = (deltaQuota / 1000.0) * pBucket->quotaRate;
if (incSize > 0) { if (incSize > 0) {
pBucket->quotaRemain = TMIN(pBucket->quotaRemain + incSize, pBucket->quotaCapacity); pBucket->quotaRemain = TMIN(pBucket->quotaRemain + incSize, pBucket->quotaCapacity);
pBucket->quotaFillTimestamp = now; pBucket->quotaFillTimestamp = now;
@ -447,7 +459,7 @@ bool streamTaskExtractAvailableToken(STokenBucket* pBucket, const char* id) {
if (pBucket->quotaRemain > 0) { if (pBucket->quotaRemain > 0) {
pBucket->numOfToken -= 1; pBucket->numOfToken -= 1;
return true; return true;
} else { // no available size quota now } else { // no available size quota now
return false; return false;
} }
} else { } else {
@ -460,8 +472,6 @@ void streamTaskPutbackToken(STokenBucket* pBucket) {
} }
// size in KB // size in KB
void streamTaskConsumeQuota(STokenBucket* pBucket, int32_t bytes) { void streamTaskConsumeQuota(STokenBucket* pBucket, int32_t bytes) { pBucket->quotaRemain -= SIZE_IN_MiB(bytes); }
pBucket->quotaRemain -= SIZE_IN_MiB(bytes);
}
void streamTaskInputFail(SStreamTask* pTask) { atomic_store_8(&pTask->inputq.status, TASK_INPUT_STATUS__FAILED); } void streamTaskInputFail(SStreamTask* pTask) { atomic_store_8(&pTask->inputq.status, TASK_INPUT_STATUS__FAILED); }

9
source/libs/stream/src/streamSched.c
View File

@ -134,11 +134,14 @@ void streamTaskSchedHelper(void* param, void* tmrId) {
stDebug("s-task:%s in checkpoint procedure, not retrieve result, next:%dms", id, nextTrigger); stDebug("s-task:%s in checkpoint procedure, not retrieve result, next:%dms", id, nextTrigger);
} else { } else {
if (status == TASK_TRIGGER_STATUS__ACTIVE) { if (status == TASK_TRIGGER_STATUS__ACTIVE) {
SStreamTrigger* pTrigger = taosAllocateQitem(sizeof(SStreamTrigger), DEF_QITEM, 0); SStreamTrigger* pTrigger;
if (pTrigger == NULL) {
int32_t code = taosAllocateQitem(sizeof(SStreamTrigger), DEF_QITEM, 0, (void**)&pTrigger);
if (code) {
stError("s-task:%s failed to prepare retrieve data trigger, code:%s, try again in %dms", id, "out of memory", stError("s-task:%s failed to prepare retrieve data trigger, code:%s, try again in %dms", id, "out of memory",
nextTrigger); nextTrigger);
taosTmrReset(streamTaskSchedHelper, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pDelayTimer); taosTmrReset(streamTaskSchedHelper, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pDelayTimer);
terrno = code;
return; return;
} }
@ -156,7 +159,7 @@ void streamTaskSchedHelper(void* param, void* tmrId) {
atomic_store_8(&pTask->schedInfo.status, TASK_TRIGGER_STATUS__INACTIVE); atomic_store_8(&pTask->schedInfo.status, TASK_TRIGGER_STATUS__INACTIVE);
pTrigger->pBlock->info.type = STREAM_GET_ALL; pTrigger->pBlock->info.type = STREAM_GET_ALL;
int32_t code = streamTaskPutDataIntoInputQ(pTask, (SStreamQueueItem*)pTrigger); code = streamTaskPutDataIntoInputQ(pTask, (SStreamQueueItem*)pTrigger);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
taosTmrReset(streamTaskSchedHelper, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pDelayTimer); taosTmrReset(streamTaskSchedHelper, nextTrigger, pTask, streamTimer, &pTask->schedInfo.pDelayTimer);
return; return;

12
source/libs/stream/src/streamTask.c
View File

@ -263,6 +263,7 @@ void tFreeStreamTask(SStreamTask* pTask) {
tDeleteSchemaWrapper(pTask->outputInfo.tbSink.pSchemaWrapper); tDeleteSchemaWrapper(pTask->outputInfo.tbSink.pSchemaWrapper);
taosMemoryFree(pTask->outputInfo.tbSink.pTSchema); taosMemoryFree(pTask->outputInfo.tbSink.pTSchema);
tSimpleHashCleanup(pTask->outputInfo.tbSink.pTblInfo); tSimpleHashCleanup(pTask->outputInfo.tbSink.pTblInfo);
tDeleteSchemaWrapper(pTask->outputInfo.tbSink.pTagSchema);
} else if (pTask->outputInfo.type == TASK_OUTPUT__SHUFFLE_DISPATCH) { } else if (pTask->outputInfo.type == TASK_OUTPUT__SHUFFLE_DISPATCH) {
taosArrayDestroy(pTask->outputInfo.shuffleDispatcher.dbInfo.pVgroupInfos); taosArrayDestroy(pTask->outputInfo.shuffleDispatcher.dbInfo.pVgroupInfos);
} }
@ -979,10 +980,13 @@ char* createStreamTaskIdStr(int64_t streamId, int32_t taskId) {
} }
static int32_t streamTaskEnqueueRetrieve(SStreamTask* pTask, SStreamRetrieveReq* pReq) { static int32_t streamTaskEnqueueRetrieve(SStreamTask* pTask, SStreamRetrieveReq* pReq) {
SStreamDataBlock* pData = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SStreamDataBlock)); int32_t code;
if (pData == NULL) { SStreamDataBlock* pData;
code = taosAllocateQitem(sizeof(SStreamDataBlock), DEF_QITEM, sizeof(SStreamDataBlock), (void**)&pData);
if (code) {
stError("s-task:%s failed to allocated retrieve-block", pTask->id.idStr); stError("s-task:%s failed to allocated retrieve-block", pTask->id.idStr);
return terrno; return terrno = code;
} }
// enqueue // enqueue
@ -992,7 +996,7 @@ static int32_t streamTaskEnqueueRetrieve(SStreamTask* pTask, SStreamRetrieveReq*
pData->type = STREAM_INPUT__DATA_RETRIEVE; pData->type = STREAM_INPUT__DATA_RETRIEVE;
pData->srcVgId = 0; pData->srcVgId = 0;
int32_t code = streamRetrieveReqToData(pReq, pData, pTask->id.idStr); code = streamRetrieveReqToData(pReq, pData, pTask->id.idStr);
if (code != TSDB_CODE_SUCCESS) { if (code != TSDB_CODE_SUCCESS) {
taosFreeQitem(pData); taosFreeQitem(pData);
return code; return code;

8
source/libs/transport/test/svrBench.c
View File

@ -67,7 +67,7 @@ void *processShellMsg(void *arg) {
int type; int type;
SQueueInfo qinfo = {0}; SQueueInfo qinfo = {0};
qall = taosAllocateQall(); taosAllocateQall(&qall);
while (1) { while (1) {
int numOfMsgs = taosReadAllQitemsFromQset(multiQ->qset[idx], qall, &qinfo); int numOfMsgs = taosReadAllQitemsFromQset(multiQ->qset[idx], qall, &qinfo);
@ -129,7 +129,7 @@ void *processShellMsg(void *arg) {
void processRequestMsg(void *pParent, SRpcMsg *pMsg, SEpSet *pEpSet) { void processRequestMsg(void *pParent, SRpcMsg *pMsg, SEpSet *pEpSet) {
SRpcMsg *pTemp; SRpcMsg *pTemp;
pTemp = taosAllocateQitem(sizeof(SRpcMsg), DEF_QITEM, 0); taosAllocateQitem(sizeof(SRpcMsg), DEF_QITEM, 0, (void **)&pTemp);
memcpy(pTemp, pMsg, sizeof(SRpcMsg)); memcpy(pTemp, pMsg, sizeof(SRpcMsg));
int32_t idx = balance % multiQ->numOfThread; int32_t idx = balance % multiQ->numOfThread;
@ -212,8 +212,8 @@ int main(int argc, char *argv[]) {
multiQ->qset = (STaosQset **)taosMemoryMalloc(sizeof(STaosQset *) * numOfAthread); multiQ->qset = (STaosQset **)taosMemoryMalloc(sizeof(STaosQset *) * numOfAthread);
for (int i = 0; i < numOfAthread; i++) { for (int i = 0; i < numOfAthread; i++) {
multiQ->qhandle[i] = taosOpenQueue(); taosOpenQueue(&multiQ->qhandle[i]);
multiQ->qset[i] = taosOpenQset(); taosOpenQset(&multiQ->qset[i]);
taosAddIntoQset(multiQ->qset[i], multiQ->qhandle[i], NULL); taosAddIntoQset(multiQ->qset[i], multiQ->qhandle[i], NULL);
} }
TThread *threads = taosMemoryMalloc(sizeof(TThread) * numOfAthread); TThread *threads = taosMemoryMalloc(sizeof(TThread) * numOfAthread);

4
source/util/src/thash.c
View File

@ -675,7 +675,11 @@ void taosHashTableResize(SHashObj *pHashObj) {
} }
SHashNode *doCreateHashNode(const void *key, size_t keyLen, const void *pData, size_t dsize, uint32_t hashVal) { SHashNode *doCreateHashNode(const void *key, size_t keyLen, const void *pData, size_t dsize, uint32_t hashVal) {
#ifdef _TD_LOONGARCH_64
SHashNode *pNewNode = taosMemoryCalloc(1, sizeof(SHashNode) + keyLen + dsize + 1);
#else
SHashNode *pNewNode = taosMemoryMalloc(sizeof(SHashNode) + keyLen + dsize + 1); SHashNode *pNewNode = taosMemoryMalloc(sizeof(SHashNode) + keyLen + dsize + 1);
#endif
if (pNewNode == NULL) { if (pNewNode == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;

84
source/util/src/tqueue.c
View File

@ -17,6 +17,7 @@
#include "tqueue.h" #include "tqueue.h"
#include "taoserror.h" #include "taoserror.h"
#include "tlog.h" #include "tlog.h"
#include "tutil.h"
int64_t tsRpcQueueMemoryAllowed = 0; int64_t tsRpcQueueMemoryAllowed = 0;
int64_t tsRpcQueueMemoryUsed = 0; int64_t tsRpcQueueMemoryUsed = 0;
@ -58,20 +59,20 @@ struct STaosQall {
void taosSetQueueMemoryCapacity(STaosQueue *queue, int64_t cap) { queue->memLimit = cap; } void taosSetQueueMemoryCapacity(STaosQueue *queue, int64_t cap) { queue->memLimit = cap; }
void taosSetQueueCapacity(STaosQueue *queue, int64_t size) { queue->itemLimit = size; } void taosSetQueueCapacity(STaosQueue *queue, int64_t size) { queue->itemLimit = size; }
STaosQueue *taosOpenQueue() { int32_t taosOpenQueue(STaosQueue **queue) {
STaosQueue *queue = taosMemoryCalloc(1, sizeof(STaosQueue)); *queue = taosMemoryCalloc(1, sizeof(STaosQueue));
if (queue == NULL) { if (*queue == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return (terrno = TSDB_CODE_OUT_OF_MEMORY);
return NULL;
} }
if (taosThreadMutexInit(&queue->mutex, NULL) != 0) { int32_t code = taosThreadMutexInit(&(*queue)->mutex, NULL);
terrno = TSDB_CODE_OUT_OF_MEMORY; if (code) {
return NULL; taosMemoryFreeClear(*queue);
return (terrno = TAOS_SYSTEM_ERROR(code));
} }
uDebug("queue:%p is opened", queue); uDebug("queue:%p is opened", queue);
return queue; return 0;
} }
void taosSetQueueFp(STaosQueue *queue, FItem itemFp, FItems itemsFp) { void taosSetQueueFp(STaosQueue *queue, FItem itemFp, FItems itemsFp) {
@ -146,20 +147,12 @@ int64_t taosQueueMemorySize(STaosQueue *queue) {
return memOfItems; return memOfItems;
} }
int32_t taosAllocateQitemWrapper(int32_t size, EQItype itype, int64_t dataSize, void **pItem) { int32_t taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize, void **item) {
int32_t code = 0; *item = NULL;
*pItem = taosAllocateQitem(size, itype, dataSize);
if (*pItem == NULL) {
return terrno;
}
return code;
}
void *taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize) {
STaosQnode *pNode = taosMemoryCalloc(1, sizeof(STaosQnode) + size); STaosQnode *pNode = taosMemoryCalloc(1, sizeof(STaosQnode) + size);
if (pNode == NULL) { if (pNode == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL;
} }
pNode->dataSize = dataSize; pNode->dataSize = dataSize;
@ -174,15 +167,15 @@ void *taosAllocateQitem(int32_t size, EQItype itype, int64_t dataSize) {
tsRpcQueueMemoryAllowed); tsRpcQueueMemoryAllowed);
atomic_sub_fetch_64(&tsRpcQueueMemoryUsed, size + dataSize); atomic_sub_fetch_64(&tsRpcQueueMemoryUsed, size + dataSize);
taosMemoryFree(pNode); taosMemoryFree(pNode);
terrno = TSDB_CODE_OUT_OF_RPC_MEMORY_QUEUE; return (terrno = TSDB_CODE_OUT_OF_RPC_MEMORY_QUEUE);
return NULL;
} }
uTrace("item:%p, node:%p is allocated, alloc:%" PRId64, pNode->item, pNode, alloced); uTrace("item:%p, node:%p is allocated, alloc:%" PRId64, pNode->item, pNode, alloced);
} else { } else {
uTrace("item:%p, node:%p is allocated", pNode->item, pNode); uTrace("item:%p, node:%p is allocated", pNode->item, pNode);
} }
return pNode->item; *item = pNode->item;
return 0;
} }
void taosFreeQitem(void *pItem) { void taosFreeQitem(void *pItem) {
@ -230,13 +223,17 @@ int32_t taosWriteQitem(STaosQueue *queue, void *pItem) {
} }
queue->numOfItems++; queue->numOfItems++;
queue->memOfItems += (pNode->size + pNode->dataSize); queue->memOfItems += (pNode->size + pNode->dataSize);
if (queue->qset) atomic_add_fetch_32(&queue->qset->numOfItems, 1); if (queue->qset) {
atomic_add_fetch_32(&queue->qset->numOfItems, 1);
}
uTrace("item:%p is put into queue:%p, items:%d mem:%" PRId64, pItem, queue, queue->numOfItems, queue->memOfItems); uTrace("item:%p is put into queue:%p, items:%d mem:%" PRId64, pItem, queue, queue->numOfItems, queue->memOfItems);
taosThreadMutexUnlock(&queue->mutex); taosThreadMutexUnlock(&queue->mutex);
if (queue->qset) tsem_post(&queue->qset->sem); if (queue->qset) {
tsem_post(&queue->qset->sem);
}
return code; return code;
} }
@ -250,10 +247,14 @@ int32_t taosReadQitem(STaosQueue *queue, void **ppItem) {
pNode = queue->head; pNode = queue->head;
*ppItem = pNode->item; *ppItem = pNode->item;
queue->head = pNode->next; queue->head = pNode->next;
if (queue->head == NULL) queue->tail = NULL; if (queue->head == NULL) {
queue->tail = NULL;
}
queue->numOfItems--; queue->numOfItems--;
queue->memOfItems -= (pNode->size + pNode->dataSize); queue->memOfItems -= (pNode->size + pNode->dataSize);
if (queue->qset) atomic_sub_fetch_32(&queue->qset->numOfItems, 1); if (queue->qset) {
atomic_sub_fetch_32(&queue->qset->numOfItems, 1);
}
code = 1; code = 1;
uTrace("item:%p is read out from queue:%p, items:%d mem:%" PRId64, *ppItem, queue, queue->numOfItems, uTrace("item:%p is read out from queue:%p, items:%d mem:%" PRId64, *ppItem, queue, queue->numOfItems,
queue->memOfItems); queue->memOfItems);
@ -264,12 +265,12 @@ int32_t taosReadQitem(STaosQueue *queue, void **ppItem) {
return code; return code;
} }
STaosQall *taosAllocateQall() { int32_t taosAllocateQall(STaosQall **qall) {
STaosQall *qall = taosMemoryCalloc(1, sizeof(STaosQall)); *qall = taosMemoryCalloc(1, sizeof(STaosQall));
if (qall != NULL) { if (*qall == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return terrno = TSDB_CODE_OUT_OF_MEMORY;
} }
return qall; return 0;
} }
void taosFreeQall(STaosQall *qall) { taosMemoryFree(qall); } void taosFreeQall(STaosQall *qall) { taosMemoryFree(qall); }
@ -299,7 +300,9 @@ int32_t taosReadAllQitems(STaosQueue *queue, STaosQall *qall) {
queue->memOfItems = 0; queue->memOfItems = 0;
uTrace("read %d items from queue:%p, items:%d mem:%" PRId64, numOfItems, queue, queue->numOfItems, uTrace("read %d items from queue:%p, items:%d mem:%" PRId64, numOfItems, queue, queue->numOfItems,
queue->memOfItems); queue->memOfItems);
if (queue->qset) atomic_sub_fetch_32(&queue->qset->numOfItems, qall->numOfItems); if (queue->qset) {
atomic_sub_fetch_32(&queue->qset->numOfItems, qall->numOfItems);
}
} }
taosThreadMutexUnlock(&queue->mutex); taosThreadMutexUnlock(&queue->mutex);
@ -335,18 +338,17 @@ int32_t taosGetQitem(STaosQall *qall, void **ppItem) {
return num; return num;
} }
STaosQset *taosOpenQset() { int32_t taosOpenQset(STaosQset **qset) {
STaosQset *qset = taosMemoryCalloc(sizeof(STaosQset), 1); *qset = taosMemoryCalloc(sizeof(STaosQset), 1);
if (qset == NULL) { if (*qset == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; return terrno = TSDB_CODE_OUT_OF_MEMORY;
return NULL;
} }
taosThreadMutexInit(&qset->mutex, NULL); taosThreadMutexInit(&(*qset)->mutex, NULL);
tsem_init(&qset->sem, 0, 0); tsem_init(&(*qset)->sem, 0, 0);
uDebug("qset:%p is opened", qset); uDebug("qset:%p is opened", qset);
return qset; return 0;
} }
void taosCloseQset(STaosQset *qset) { void taosCloseQset(STaosQset *qset) {

156
source/util/src/tworker.c
View File

@ -16,20 +16,22 @@
#define _DEFAULT_SOURCE #define _DEFAULT_SOURCE
#include "tworker.h" #include "tworker.h"
#include "taoserror.h" #include "taoserror.h"
#include "tcompare.h"
#include "tgeosctx.h" #include "tgeosctx.h"
#include "tlog.h" #include "tlog.h"
#include "tcompare.h"
#define QUEUE_THRESHOLD (1000 * 1000) #define QUEUE_THRESHOLD (1000 * 1000)
typedef void *(*ThreadFp)(void *param); typedef void *(*ThreadFp)(void *param);
int32_t tQWorkerInit(SQWorkerPool *pool) { int32_t tQWorkerInit(SQWorkerPool *pool) {
pool->qset = taosOpenQset(); int32_t code = taosOpenQset(&pool->qset);
if (code) return code;
pool->workers = taosMemoryCalloc(pool->max, sizeof(SQueueWorker)); pool->workers = taosMemoryCalloc(pool->max, sizeof(SQueueWorker));
if (pool->workers == NULL) { if (pool->workers == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; taosCloseQset(pool->qset);
return -1; return terrno = TSDB_CODE_OUT_OF_MEMORY;
} }
(void)taosThreadMutexInit(&pool->mutex, NULL); (void)taosThreadMutexInit(&pool->mutex, NULL);
@ -109,8 +111,14 @@ static void *tQWorkerThreadFp(SQueueWorker *worker) {
} }
STaosQueue *tQWorkerAllocQueue(SQWorkerPool *pool, void *ahandle, FItem fp) { STaosQueue *tQWorkerAllocQueue(SQWorkerPool *pool, void *ahandle, FItem fp) {
STaosQueue *queue = taosOpenQueue(); int32_t code;
if (queue == NULL) return NULL; STaosQueue *queue;
code = taosOpenQueue(&queue);
if (code) {
terrno = code;
return NULL;
}
taosThreadMutexLock(&pool->mutex); taosThreadMutexLock(&pool->mutex);
taosSetQueueFp(queue, fp, NULL); taosSetQueueFp(queue, fp, NULL);
@ -150,11 +158,17 @@ void tQWorkerFreeQueue(SQWorkerPool *pool, STaosQueue *queue) {
} }
int32_t tAutoQWorkerInit(SAutoQWorkerPool *pool) { int32_t tAutoQWorkerInit(SAutoQWorkerPool *pool) {
pool->qset = taosOpenQset(); int32_t code;
code = taosOpenQset(&pool->qset);
if (code) {
return terrno = code;
}
pool->workers = taosArrayInit(2, sizeof(SQueueWorker *)); pool->workers = taosArrayInit(2, sizeof(SQueueWorker *));
if (pool->workers == NULL) { if (pool->workers == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY; taosCloseQset(pool->qset);
return -1; return terrno = TSDB_CODE_OUT_OF_MEMORY;
} }
(void)taosThreadMutexInit(&pool->mutex, NULL); (void)taosThreadMutexInit(&pool->mutex, NULL);
@ -228,8 +242,14 @@ static void *tAutoQWorkerThreadFp(SQueueWorker *worker) {
} }
STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem fp) { STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem fp) {
STaosQueue *queue = taosOpenQueue(); int32_t code;
if (queue == NULL) return NULL; STaosQueue *queue;
code = taosOpenQueue(&queue);
if (code) {
terrno = code;
return NULL;
}
taosThreadMutexLock(&pool->mutex); taosThreadMutexLock(&pool->mutex);
taosSetQueueFp(queue, fp, NULL); taosSetQueueFp(queue, fp, NULL);
@ -375,20 +395,21 @@ static void *tWWorkerThreadFp(SWWorker *worker) {
STaosQueue *tWWorkerAllocQueue(SWWorkerPool *pool, void *ahandle, FItems fp) { STaosQueue *tWWorkerAllocQueue(SWWorkerPool *pool, void *ahandle, FItems fp) {
taosThreadMutexLock(&pool->mutex); taosThreadMutexLock(&pool->mutex);
SWWorker *worker = pool->workers + pool->nextId; SWWorker *worker = pool->workers + pool->nextId;
int32_t code = -1; int32_t code = -1;
STaosQueue *queue;
STaosQueue *queue = taosOpenQueue(); code = taosOpenQueue(&queue);
if (queue == NULL) goto _OVER; if (code) goto _OVER;
taosSetQueueFp(queue, NULL, fp); taosSetQueueFp(queue, NULL, fp);
if (worker->qset == NULL) { if (worker->qset == NULL) {
worker->qset = taosOpenQset(); code = taosOpenQset(&worker->qset);
if (worker->qset == NULL) goto _OVER; if (code) goto _OVER;
taosAddIntoQset(worker->qset, queue, ahandle); taosAddIntoQset(worker->qset, queue, ahandle);
worker->qall = taosAllocateQall(); code = taosAllocateQall(&worker->qall);
if (worker->qall == NULL) goto _OVER; if (code) goto _OVER;
TdThreadAttr thAttr; TdThreadAttr thAttr;
taosThreadAttrInit(&thAttr); taosThreadAttrInit(&thAttr);
@ -406,12 +427,10 @@ STaosQueue *tWWorkerAllocQueue(SWWorkerPool *pool, void *ahandle, FItems fp) {
pool->nextId = (pool->nextId + 1) % pool->max; pool->nextId = (pool->nextId + 1) % pool->max;
} }
code = 0;
_OVER: _OVER:
taosThreadMutexUnlock(&pool->mutex); taosThreadMutexUnlock(&pool->mutex);
if (code == -1) { if (code) {
if (queue != NULL) taosCloseQueue(queue); if (queue != NULL) taosCloseQueue(queue);
if (worker->qset != NULL) taosCloseQset(worker->qset); if (worker->qset != NULL) taosCloseQset(worker->qset);
if (worker->qall != NULL) taosFreeQall(worker->qall); if (worker->qall != NULL) taosFreeQall(worker->qall);
@ -517,13 +536,13 @@ void tMultiWorkerCleanup(SMultiWorker *pWorker) {
tWWorkerFreeQueue(&pWorker->pool, pWorker->queue); tWWorkerFreeQueue(&pWorker->pool, pWorker->queue);
} }
static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool* pool); static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool *pool);
static int32_t tQueryAutoQWorkerBeforeBlocking(void *p); static int32_t tQueryAutoQWorkerBeforeBlocking(void *p);
static int32_t tQueryAutoQWorkerRecoverFromBlocking(void *p); static int32_t tQueryAutoQWorkerRecoverFromBlocking(void *p);
static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool* pPool); static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool *pPool);
static bool tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool* pPool, SQueryAutoQWorker* pWorker); static bool tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool *pPool, SQueryAutoQWorker *pWorker);
#define GET_ACTIVE_N(int64_val) (int32_t)((int64_val) >> 32) #define GET_ACTIVE_N(int64_val) (int32_t)((int64_val) >> 32)
#define GET_RUNNING_N(int64_val) (int32_t)(int64_val & 0xFFFFFFFF) #define GET_RUNNING_N(int64_val) (int32_t)(int64_val & 0xFFFFFFFF)
static int32_t atomicFetchSubActive(int64_t *ptr, int32_t val) { static int32_t atomicFetchSubActive(int64_t *ptr, int32_t val) {
@ -544,7 +563,7 @@ static int32_t atomicFetchAddActive(int64_t *ptr, int32_t val) {
static int32_t atomicFetchAddRunning(int64_t *ptr, int32_t val) { return GET_RUNNING_N(atomic_fetch_add_64(ptr, val)); } static int32_t atomicFetchAddRunning(int64_t *ptr, int32_t val) { return GET_RUNNING_N(atomic_fetch_add_64(ptr, val)); }
static bool atomicCompareExchangeActive(int64_t* ptr, int32_t* expectedVal, int32_t newVal) { static bool atomicCompareExchangeActive(int64_t *ptr, int32_t *expectedVal, int32_t newVal) {
int64_t oldVal64 = *expectedVal, newVal64 = newVal; int64_t oldVal64 = *expectedVal, newVal64 = newVal;
int32_t running = GET_RUNNING_N(*ptr); int32_t running = GET_RUNNING_N(*ptr);
oldVal64 <<= 32; oldVal64 <<= 32;
@ -560,7 +579,7 @@ static bool atomicCompareExchangeActive(int64_t* ptr, int32_t* expectedVal, int3
} }
} }
static int64_t atomicCompareExchangeRunning(int64_t* ptr, int32_t* expectedVal, int32_t newVal) { static int64_t atomicCompareExchangeRunning(int64_t *ptr, int32_t *expectedVal, int32_t newVal) {
int64_t oldVal64 = *expectedVal, newVal64 = newVal; int64_t oldVal64 = *expectedVal, newVal64 = newVal;
int64_t activeShifted = GET_ACTIVE_N(*ptr); int64_t activeShifted = GET_ACTIVE_N(*ptr);
activeShifted <<= 32; activeShifted <<= 32;
@ -576,7 +595,7 @@ static int64_t atomicCompareExchangeRunning(int64_t* ptr, int32_t* expectedVal,
} }
static int64_t atomicCompareExchangeActiveAndRunning(int64_t *ptr, int32_t *expectedActive, int32_t newActive, static int64_t atomicCompareExchangeActiveAndRunning(int64_t *ptr, int32_t *expectedActive, int32_t newActive,
int32_t *expectedRunning, int32_t newRunning) { int32_t *expectedRunning, int32_t newRunning) {
int64_t oldVal64 = *expectedActive, newVal64 = newActive; int64_t oldVal64 = *expectedActive, newVal64 = newActive;
oldVal64 <<= 32; oldVal64 <<= 32;
oldVal64 |= *expectedRunning; oldVal64 |= *expectedRunning;
@ -641,7 +660,7 @@ static void *tQueryAutoQWorkerThreadFp(SQueryAutoQWorker *worker) {
static bool tQueryAutoQWorkerTrySignalWaitingAfterBlock(void *p) { static bool tQueryAutoQWorkerTrySignalWaitingAfterBlock(void *p) {
SQueryAutoQWorkerPool *pPool = p; SQueryAutoQWorkerPool *pPool = p;
bool ret = false; bool ret = false;
int32_t waiting = pPool->waitingAfterBlockN; int32_t waiting = pPool->waitingAfterBlockN;
while (waiting > 0) { while (waiting > 0) {
int32_t waitingNew = atomic_val_compare_exchange_32(&pPool->waitingAfterBlockN, waiting, waiting - 1); int32_t waitingNew = atomic_val_compare_exchange_32(&pPool->waitingAfterBlockN, waiting, waiting - 1);
if (waitingNew == waiting) { if (waitingNew == waiting) {
@ -656,10 +675,10 @@ static bool tQueryAutoQWorkerTrySignalWaitingAfterBlock(void *p) {
return ret; return ret;
} }
static bool tQueryAutoQWorkerTrySignalWaitingBeforeProcess(void* p) { static bool tQueryAutoQWorkerTrySignalWaitingBeforeProcess(void *p) {
SQueryAutoQWorkerPool *pPool = p; SQueryAutoQWorkerPool *pPool = p;
bool ret = false; bool ret = false;
int32_t waiting = pPool->waitingBeforeProcessMsgN; int32_t waiting = pPool->waitingBeforeProcessMsgN;
while (waiting > 0) { while (waiting > 0) {
int32_t waitingNew = atomic_val_compare_exchange_32(&pPool->waitingBeforeProcessMsgN, waiting, waiting - 1); int32_t waitingNew = atomic_val_compare_exchange_32(&pPool->waitingBeforeProcessMsgN, waiting, waiting - 1);
if (waitingNew == waiting) { if (waitingNew == waiting) {
@ -674,10 +693,10 @@ static bool tQueryAutoQWorkerTrySignalWaitingBeforeProcess(void* p) {
return ret; return ret;
} }
static bool tQueryAutoQWorkerTryDecActive(void* p, int32_t minActive) { static bool tQueryAutoQWorkerTryDecActive(void *p, int32_t minActive) {
SQueryAutoQWorkerPool *pPool = p; SQueryAutoQWorkerPool *pPool = p;
bool ret = false; bool ret = false;
int64_t val64 = pPool->activeRunningN; int64_t val64 = pPool->activeRunningN;
int32_t active = GET_ACTIVE_N(val64), running = GET_RUNNING_N(val64); int32_t active = GET_ACTIVE_N(val64), running = GET_RUNNING_N(val64);
while (active > minActive) { while (active > minActive) {
if (atomicCompareExchangeActiveAndRunning(&pPool->activeRunningN, &active, active - 1, &running, running - 1)) if (atomicCompareExchangeActiveAndRunning(&pPool->activeRunningN, &active, active - 1, &running, running - 1))
@ -687,7 +706,7 @@ static bool tQueryAutoQWorkerTryDecActive(void* p, int32_t minActive) {
return false; return false;
} }
static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool* pPool) { static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool *pPool) {
while (1) { while (1) {
int64_t val64 = pPool->activeRunningN; int64_t val64 = pPool->activeRunningN;
int32_t running = GET_RUNNING_N(val64), active = GET_ACTIVE_N(val64); int32_t running = GET_RUNNING_N(val64), active = GET_ACTIVE_N(val64);
@ -709,17 +728,17 @@ static int32_t tQueryAutoQWorkerWaitingCheck(SQueryAutoQWorkerPool* pPool) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;
} }
bool tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool* pPool, SQueryAutoQWorker* pWorker) { bool tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool *pPool, SQueryAutoQWorker *pWorker) {
if (tQueryAutoQWorkerTrySignalWaitingAfterBlock(pPool) || tQueryAutoQWorkerTrySignalWaitingBeforeProcess(pPool) || if (tQueryAutoQWorkerTrySignalWaitingAfterBlock(pPool) || tQueryAutoQWorkerTrySignalWaitingBeforeProcess(pPool) ||
tQueryAutoQWorkerTryDecActive(pPool, pPool->num)) { tQueryAutoQWorkerTryDecActive(pPool, pPool->num)) {
taosThreadMutexLock(&pPool->poolLock); taosThreadMutexLock(&pPool->poolLock);
SListNode* pNode = listNode(pWorker); SListNode *pNode = listNode(pWorker);
tdListPopNode(pPool->workers, pNode); tdListPopNode(pPool->workers, pNode);
// reclaim some workers // reclaim some workers
if (pWorker->id >= pPool->maxInUse) { if (pWorker->id >= pPool->maxInUse) {
while (listNEles(pPool->exitedWorkers) > pPool->maxInUse - pPool->num) { while (listNEles(pPool->exitedWorkers) > pPool->maxInUse - pPool->num) {
SListNode* head = tdListPopHead(pPool->exitedWorkers); SListNode *head = tdListPopHead(pPool->exitedWorkers);
SQueryAutoQWorker* pWorker = (SQueryAutoQWorker*)head->data; SQueryAutoQWorker *pWorker = (SQueryAutoQWorker *)head->data;
if (pWorker && taosCheckPthreadValid(pWorker->thread)) { if (pWorker && taosCheckPthreadValid(pWorker->thread)) {
taosThreadJoin(pWorker->thread, NULL); taosThreadJoin(pWorker->thread, NULL);
taosThreadClear(&pWorker->thread); taosThreadClear(&pWorker->thread);
@ -758,8 +777,9 @@ bool tQueryAutoQWorkerTryRecycleWorker(SQueryAutoQWorkerPool* pPool, SQueryAutoQ
} }
int32_t tQueryAutoQWorkerInit(SQueryAutoQWorkerPool *pool) { int32_t tQueryAutoQWorkerInit(SQueryAutoQWorkerPool *pool) {
pool->qset = taosOpenQset(); int32_t code;
if (!pool->qset) return terrno; code = taosOpenQset(&pool->qset);
if (code) return terrno = code;
pool->workers = tdListNew(sizeof(SQueryAutoQWorker)); pool->workers = tdListNew(sizeof(SQueryAutoQWorker));
if (!pool->workers) return TSDB_CODE_OUT_OF_MEMORY; if (!pool->workers) return TSDB_CODE_OUT_OF_MEMORY;
pool->backupWorkers = tdListNew(sizeof(SQueryAutoQWorker)); pool->backupWorkers = tdListNew(sizeof(SQueryAutoQWorker));
@ -812,16 +832,16 @@ void tQueryAutoQWorkerCleanup(SQueryAutoQWorkerPool *pPool) {
taosThreadCondBroadcast(&pPool->waitingBeforeProcessMsgCond); taosThreadCondBroadcast(&pPool->waitingBeforeProcessMsgCond);
taosThreadMutexUnlock(&pPool->waitingBeforeProcessMsgLock); taosThreadMutexUnlock(&pPool->waitingBeforeProcessMsgLock);
int32_t idx = 0; int32_t idx = 0;
SQueryAutoQWorker* worker = NULL; SQueryAutoQWorker *worker = NULL;
while (true) { while (true) {
taosThreadMutexLock(&pPool->poolLock); taosThreadMutexLock(&pPool->poolLock);
if (listNEles(pPool->workers) == 0) { if (listNEles(pPool->workers) == 0) {
taosThreadMutexUnlock(&pPool->poolLock); taosThreadMutexUnlock(&pPool->poolLock);
break; break;
} }
SListNode* pNode = tdListPopHead(pPool->workers); SListNode *pNode = tdListPopHead(pPool->workers);
worker = (SQueryAutoQWorker*)pNode->data; worker = (SQueryAutoQWorker *)pNode->data;
taosThreadMutexUnlock(&pPool->poolLock); taosThreadMutexUnlock(&pPool->poolLock);
if (worker && taosCheckPthreadValid(worker->thread)) { if (worker && taosCheckPthreadValid(worker->thread)) {
taosThreadJoin(worker->thread, NULL); taosThreadJoin(worker->thread, NULL);
@ -831,8 +851,8 @@ void tQueryAutoQWorkerCleanup(SQueryAutoQWorkerPool *pPool) {
} }
while (listNEles(pPool->backupWorkers) > 0) { while (listNEles(pPool->backupWorkers) > 0) {
SListNode* pNode = tdListPopHead(pPool->backupWorkers); SListNode *pNode = tdListPopHead(pPool->backupWorkers);
worker = (SQueryAutoQWorker*)pNode->data; worker = (SQueryAutoQWorker *)pNode->data;
if (worker && taosCheckPthreadValid(worker->thread)) { if (worker && taosCheckPthreadValid(worker->thread)) {
taosThreadJoin(worker->thread, NULL); taosThreadJoin(worker->thread, NULL);
taosThreadClear(&worker->thread); taosThreadClear(&worker->thread);
@ -841,8 +861,8 @@ void tQueryAutoQWorkerCleanup(SQueryAutoQWorkerPool *pPool) {
} }
while (listNEles(pPool->exitedWorkers) > 0) { while (listNEles(pPool->exitedWorkers) > 0) {
SListNode* pNode = tdListPopHead(pPool->exitedWorkers); SListNode *pNode = tdListPopHead(pPool->exitedWorkers);
worker = (SQueryAutoQWorker*)pNode->data; worker = (SQueryAutoQWorker *)pNode->data;
if (worker && taosCheckPthreadValid(worker->thread)) { if (worker && taosCheckPthreadValid(worker->thread)) {
taosThreadJoin(worker->thread, NULL); taosThreadJoin(worker->thread, NULL);
taosThreadClear(&worker->thread); taosThreadClear(&worker->thread);
@ -867,14 +887,18 @@ void tQueryAutoQWorkerCleanup(SQueryAutoQWorkerPool *pPool) {
} }
STaosQueue *tQueryAutoQWorkerAllocQueue(SQueryAutoQWorkerPool *pool, void *ahandle, FItem fp) { STaosQueue *tQueryAutoQWorkerAllocQueue(SQueryAutoQWorkerPool *pool, void *ahandle, FItem fp) {
STaosQueue *queue = taosOpenQueue(); STaosQueue *queue;
if (queue == NULL) return NULL; int32_t code = taosOpenQueue(&queue);
if (code) {
terrno = code;
return NULL;
}
taosThreadMutexLock(&pool->poolLock); taosThreadMutexLock(&pool->poolLock);
taosSetQueueFp(queue, fp, NULL); taosSetQueueFp(queue, fp, NULL);
taosAddIntoQset(pool->qset, queue, ahandle); taosAddIntoQset(pool->qset, queue, ahandle);
SQueryAutoQWorker worker = {0}; SQueryAutoQWorker worker = {0};
SQueryAutoQWorker* pWorker = NULL; SQueryAutoQWorker *pWorker = NULL;
// spawn a thread to process queue // spawn a thread to process queue
if (pool->num < pool->max) { if (pool->num < pool->max) {
@ -882,14 +906,14 @@ STaosQueue *tQueryAutoQWorkerAllocQueue(SQueryAutoQWorkerPool *pool, void *ahand
worker.id = listNEles(pool->workers); worker.id = listNEles(pool->workers);
worker.backupIdx = -1; worker.backupIdx = -1;
worker.pool = pool; worker.pool = pool;
SListNode* pNode = tdListAdd(pool->workers, &worker); SListNode *pNode = tdListAdd(pool->workers, &worker);
if (!pNode) { if (!pNode) {
taosCloseQueue(queue); taosCloseQueue(queue);
queue = NULL; queue = NULL;
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
break; break;
} }
pWorker = (SQueryAutoQWorker*)pNode->data; pWorker = (SQueryAutoQWorker *)pNode->data;
TdThreadAttr thAttr; TdThreadAttr thAttr;
taosThreadAttrInit(&thAttr); taosThreadAttrInit(&thAttr);
@ -915,11 +939,9 @@ STaosQueue *tQueryAutoQWorkerAllocQueue(SQueryAutoQWorkerPool *pool, void *ahand
return queue; return queue;
} }
void tQueryAutoQWorkerFreeQueue(SQueryAutoQWorkerPool *pPool, STaosQueue *pQ) { void tQueryAutoQWorkerFreeQueue(SQueryAutoQWorkerPool *pPool, STaosQueue *pQ) { taosCloseQueue(pQ); }
taosCloseQueue(pQ);
}
static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool* pool) { static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool *pool) {
// try backup pool // try backup pool
int32_t backup = pool->backupNum; int32_t backup = pool->backupNum;
while (backup > 0) { while (backup > 0) {
@ -931,20 +953,20 @@ static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool* pool) {
backup = backupNew; backup = backupNew;
} }
// backup pool is empty, create new // backup pool is empty, create new
SQueryAutoQWorker* pWorker = NULL; SQueryAutoQWorker *pWorker = NULL;
SQueryAutoQWorker worker = {0}; SQueryAutoQWorker worker = {0};
worker.pool = pool; worker.pool = pool;
worker.backupIdx = -1; worker.backupIdx = -1;
taosThreadMutexLock(&pool->poolLock); taosThreadMutexLock(&pool->poolLock);
worker.id = listNEles(pool->workers); worker.id = listNEles(pool->workers);
SListNode* pNode = tdListAdd(pool->workers, &worker); SListNode *pNode = tdListAdd(pool->workers, &worker);
if (!pNode) { if (!pNode) {
taosThreadMutexUnlock(&pool->poolLock); taosThreadMutexUnlock(&pool->poolLock);
terrno = TSDB_CODE_OUT_OF_MEMORY; terrno = TSDB_CODE_OUT_OF_MEMORY;
return terrno; return terrno;
} }
taosThreadMutexUnlock(&pool->poolLock); taosThreadMutexUnlock(&pool->poolLock);
pWorker = (SQueryAutoQWorker*)pNode->data; pWorker = (SQueryAutoQWorker *)pNode->data;
TdThreadAttr thAttr; TdThreadAttr thAttr;
taosThreadAttrInit(&thAttr); taosThreadAttrInit(&thAttr);
@ -960,7 +982,7 @@ static int32_t tQueryAutoQWorkerAddWorker(SQueryAutoQWorkerPool* pool) {
} }
static int32_t tQueryAutoQWorkerBeforeBlocking(void *p) { static int32_t tQueryAutoQWorkerBeforeBlocking(void *p) {
SQueryAutoQWorkerPool* pPool = p; SQueryAutoQWorkerPool *pPool = p;
if (tQueryAutoQWorkerTrySignalWaitingAfterBlock(p) || tQueryAutoQWorkerTrySignalWaitingBeforeProcess(p) || if (tQueryAutoQWorkerTrySignalWaitingAfterBlock(p) || tQueryAutoQWorkerTrySignalWaitingBeforeProcess(p) ||
tQueryAutoQWorkerTryDecActive(p, 1)) { tQueryAutoQWorkerTryDecActive(p, 1)) {
} else { } else {
@ -974,9 +996,9 @@ static int32_t tQueryAutoQWorkerBeforeBlocking(void *p) {
} }
static int32_t tQueryAutoQWorkerRecoverFromBlocking(void *p) { static int32_t tQueryAutoQWorkerRecoverFromBlocking(void *p) {
SQueryAutoQWorkerPool* pPool = p; SQueryAutoQWorkerPool *pPool = p;
int64_t val64 = pPool->activeRunningN; int64_t val64 = pPool->activeRunningN;
int32_t running = GET_RUNNING_N(val64), active = GET_ACTIVE_N(val64); int32_t running = GET_RUNNING_N(val64), active = GET_ACTIVE_N(val64);
while (running < pPool->num) { while (running < pPool->num) {
if (atomicCompareExchangeActiveAndRunning(&pPool->activeRunningN, &active, active + 1, &running, running + 1)) { if (atomicCompareExchangeActiveAndRunning(&pPool->activeRunningN, &active, active + 1, &running, running + 1)) {
return TSDB_CODE_SUCCESS; return TSDB_CODE_SUCCESS;