xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch 'main' into merge/mainto3.0

This commit is contained in:
Shengliang Guan 2024-12-09 19:06:06 +08:00
parent 704a395134 a444263f09
commit e99b888654
55 changed files with 985 additions and 769 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
docs/en/07-develop/01-connect.md
View File

@ -164,9 +164,6 @@ If you are using Maven to manage your project, simply add the following dependen
pip3 install taospy[ws]
```
</TabItem>
</Tabs>
- **Installation Verification**
<Tabs defaultValue="rest">
@ -199,8 +196,8 @@ import taosws
</TabItem>
</Tabs>
</TabItem>
<Tabs>
<TabItem label="Go" value="go">
Edit `go.mod` to add the `driver-go` dependency.

2
docs/en/14-reference/03-taos-sql/03-table.md
View File

@ -194,7 +194,7 @@ alter_table_option: {
### Modify Subtable Tag Value
```sql
ALTER TABLE tb_name SET tag tag_name=new_tag_value;
ALTER TABLE tb_name SET TAG tag_name1=new_tag_value1,tag_name2=new_tag_value2...;
```
### Modify Table Lifespan

2
docs/zh/07-develop/01-connect/index.md
View File

@ -37,7 +37,7 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速
关键不同点在于：
1. 使用 原生连接，需要保证客户端的驱动程序 taosc 和服务端的 TDengine 版本配套。
1. 使用 原生连接，需要保证客户端的驱动程序 taosc 和服务端的 TDengine 版本保持一致。
2. 使用 REST 连接，用户无需安装客户端驱动程序 taosc，具有跨平台易用的优势，但是无法体验数据订阅和二进制数据类型等功能。另外与 原生连接 和 WebSocket 连接相比，REST连接的性能最低。REST 接口是无状态的。在使用 REST 连接时，需要在 SQL 中指定表、超级表的数据库名称。
3. 使用 WebSocket 连接，用户也无需安装客户端驱动程序 taosc。
4. 连接云服务实例，必须使用 REST 连接 或 WebSocket 连接。

BIN
docs/zh/14-reference/01-components/12-tdinsight/assets/Alert-Used-Only.webp Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 212 KiB

BIN
docs/zh/14-reference/01-components/12-tdinsight/assets/Alert-rules.webp Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 222 KiB

BIN
docs/zh/14-reference/01-components/12-tdinsight/assets/TDengine-Alert.webp Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 95 KiB

45
docs/zh/14-reference/01-components/12-tdinsight/index.mdx
View File

@ -145,6 +145,44 @@ TDinsight 仪表盘旨在提供 TDengine 相关资源的使用情况和状态，
还有上述分类的细分维度折线图。
### 预配置告警规则自动导入
涛思总结用户使用经验，整理出 14 个常用的告警规则（alert rule），能够对集群关键指标进行监测并及时上报指标异常、超限等告警信息。
从 TDengine-server 3.3.4.3 版本（tdengine-datasource 3.6.3）开始，TDengine Datasource 支持预配置告警规则自动导入功能，用户可将 14 个告警规则一键导入 Grafana（11 及以上版本），直接使用。
预配置告警规则导入方法如下图所示，在 tdengine-datasource setting 界面，打开 “Load Tengine Alert” 开关，点击 “Save & test” 按钮后，插件会自动加载上述告警规则, 规则会放入以数据源名称 + “-alert” 的 grafana 告警目录中。如不需要，关闭 “Load TDengine Alert” 开关，点击 “Clear TDengine Alert” 旁边的按钮则会清除此数据源已导入的所有告警规则。
![TDengine Alert](./assets/TDengine-Alert.webp)
导入后，点击 Grafana 左侧 “Alert rules” ，可查看当前所有告警规则。
用户只需配置联络点（Contact points），即可获取告警通知。联络点配置方法见[告警配置](https://docs.taosdata.com/third-party/visual/grafana/#%E5%91%8A%E8%AD%A6%E9%85%8D%E7%BD%AE)。
![Alert-rules](./assets/Alert-rules.webp)
14 个告警规则具体配置如下：
| 规则名称| 规则阈值| 无监控数据时的行为 | 数据扫描间隔 |持续时间 | 执行SQL |
| ------ | --------- | ---------------- | ----------- |------- |----------------------|
|dnode 节点的CPU负载|均值 > 80%|触发告警|5分钟|5分钟 |`select now(), dnode_id, last(cpu_system) as cup_use from log.taosd_dnodes_info where _ts >= (now- 5m) and _ts < now partition by dnode_id having first(_ts) > 0 `|
|dnode 节点的的内存 |均值 > 60%|触发告警|5分钟|5分钟|`select now(), dnode_id, last(mem_engine) / last(mem_total) * 100 as taosd from log.taosd_dnodes_info where _ts >= (now- 5m) and _ts <now partition by dnode_id`|
|dnode 节点的磁盘容量占用 | > 80%|触发告警|5分钟|5分钟|`select now(), dnode_id, data_dir_level, data_dir_name, last(used) / last(total) * 100 as used from log.taosd_dnodes_data_dirs where _ts >= (now - 5m) and _ts < now partition by dnode_id, data_dir_level, data_dir_name`|
|集群授权到期 |< 60天|触发告警|1天|0秒|`select now(), cluster_id, last(grants_expire_time) / 86400 as expire_time from log.taosd_cluster_info where _ts >= (now - 24h) and _ts < now partition by cluster_id having first(_ts) > 0 `|
|测点数达到授权测点数|>= 90%|触发告警|1天|0秒|`select now(), cluster_id, CASE WHEN max(grants_timeseries_total) > 0.0 THEN max(grants_timeseries_used) /max(grants_timeseries_total) * 100.0 ELSE 0.0 END AS result from log.taosd_cluster_info where _ts >= (now - 30s) and _ts < now partition by cluster_id having timetruncate(first(_ts), 1m) > 0`|
|查询并发请求数 | > 100|不触发报警|1分钟|0秒|`select now() as ts, count(*) as slow_count from performance_schema.perf_queries`|
|慢查询执行最长时间 （无时间窗口） |> 300秒|不触发报警|1分钟|0秒|`select now() as ts, count(*) as slow_count from performance_schema.perf_queries where exec_usec>300000000`|
|dnode下线 |total != alive|触发告警|30秒|0秒|`select now(), cluster_id, last(dnodes_total) - last(dnodes_alive) as dnode_offline from log.taosd_cluster_info where _ts >= (now -30s) and _ts < now partition by cluster_id having first(_ts) > 0`|
|vnode下线 |total != alive|触发告警|30秒|0秒|`select now(), cluster_id, last(vnodes_total) - last(vnodes_alive) as vnode_offline from log.taosd_cluster_info where _ts >= (now - 30s) and _ts < now partition by cluster_id having first(_ts) > 0 `|
|数据删除请求数 |> 0|不触发报警|30秒|0秒|``select now(), count(`count`) as `delete_count` from log.taos_sql_req where sql_type = 'delete' and _ts >= (now -30s) and _ts < now``|
|Adapter RESTful 请求失败 |> 5|不触发报警|30秒|0秒|``select now(), sum(`fail`) as `Failed` from log.adapter_requests where req_type=0 and ts >= (now -30s) and ts < now``|
|Adapter WebSocket 请求失败 |> 5|不触发报警|30秒|0秒|``select now(), sum(`fail`) as `Failed` from log.adapter_requests where req_type=1 and ts >= (now -30s) and ts < now``|
|dnode 数据上报缺少 |< 3|触发告警|180秒|0秒|`select now(), cluster_id, count(*) as dnode_report from log.taosd_cluster_info where _ts >= (now -180s) and _ts < now partition by cluster_id having timetruncate(first(_ts), 1h) > 0`|
|dnode 重启 |max(update_time) > last(update_time)|触发告警|90秒|0秒|`select now(), dnode_id, max(uptime) - last(uptime) as dnode_restart from log.taosd_dnodes_info where _ts >= (now - 90s) and _ts < now partition by dnode_id`|
用户可参考上述告警规则，根据自己业务需求进行修改与完善。
Grafana7.5 及以下版本，Dashboards 与 Alert rules 功能合在一起，而之后的新版本两个功能是分开的。为兼容 Grafana7.5 及以下版本，TDinsight 面板中增加了 Alert Used Only 面板，仅 Grafana7.5 及以下版本需要使用。
![Alert Used Only](./assets/Alert-Used-Only.webp)
## 升级
下面三种方式都可以进行升级：
- 用图形界面，若有新版本，可以在 ”TDengine Datasource“ 插件页面点击 update 升级。
@ -155,10 +193,11 @@ TDinsight 仪表盘旨在提供 TDengine 相关资源的使用情况和状态，
针对不同的安装方式，卸载时：
- 用图形界面，在 ”TDengine Datasource“ 插件页面点击 ”Uninstall“ 卸载。
- 通过 `TDinsight.sh` 脚本安装的 TDinsight，可以使用命令行 `TDinsight.sh -R` 清理相关资源。
- 手动安装的 TDinsight，要完全卸载，需要清理以下内容：
- 手动安装的 TDinsight，要完全卸载，需要按照顺序清理以下内容：
1. Grafana 中的 TDinsight Dashboard。
2. Grafana 中的 Data Source 数据源。
3. 从插件安装目录删除 `tdengine-datasource` 插件。
2. Grafana 中的 Alert rules 告警规则。
3. Grafana 中的 Data Source 数据源。
4. 从插件安装目录删除 `tdengine-datasource` 插件。
## 附录

2
docs/zh/14-reference/03-taos-sql/03-table.md
View File

@ -195,7 +195,7 @@ alter_table_option: {
### 修改子表标签值
```
ALTER TABLE tb_name SET TAG tag_name=new_tag_value;
ALTER TABLE tb_name SET TAG tag_name1=new_tag_value1,tag_name2=new_tag_value2...;
```
### 修改表生命周期

71
packaging/delete_ref_lock.py
View File

@ -1,59 +1,80 @@
import subprocess
import re
# 执行 git fetch 命令并捕获输出
def git_fetch():
result = subprocess.run(['git', 'fetch'], capture_output=True, text=True)
return result
# 解析分支名称
def git_prune():
# git remote prune origin
print("git remote prune origin")
result = subprocess.run(['git', 'remote', 'prune', 'origin'], capture_output=True, text=True)
return result
def parse_branch_name_type1(error_output):
# 使用正则表达式匹配 'is at' 前的分支名称
# error: cannot lock ref 'refs/remotes/origin/fix/3.0/TD-32817': is at 7af5 but expected eaba
# match the branch name before ‘is at’ with a regular expression
match = re.search(r"error: cannot lock ref '(refs/remotes/origin/[^']+)': is at", error_output)
if match:
return match.group(1)
return None
# 解析第二种错误中的分支名称
def parse_branch_name_type2(error_output):
# 使用正则表达式匹配 'exists' 前的第一个引号内的分支名称
# match the branch name before ‘exists; cannot create’ with a regular expression
match = re.search(r"'(refs/remotes/origin/[^']+)' exists;", error_output)
if match:
return match.group(1)
return None
# 执行 git update-ref -d 命令
# parse branch name from error output of git remote prune origin
def parse_branch_name_type3(error_output):
# match the branch name before the first single quote before 'Unable to' with a regular expression
# git error: could not delete references: cannot lock ref 'refs/remotes/origin/test/3.0/TS-4893': Unable to create 'D:/workspace/main/TDinternal/community/.git/refs/remotes/origin/test/3.0/TS-4893.lock': File exists
match = re.search(r"references: cannot lock ref '(refs/remotes/origin/[^']+)': Unable to", error_output)
if match:
return match.group(1)
return None
# execute git update-ref -d <branch_name> to delete the ref
def git_update_ref(branch_name):
if branch_name:
subprocess.run(['git', 'update-ref', '-d', f'{branch_name}'], check=True)
# 解析错误类型并执行相应的修复操作
# parse error type and execute corresponding repair operation
def handle_error(error_output):
# 错误类型1：本地引用的提交ID与远程不一致
if "is at" in error_output and "but expected" in error_output:
branch_name = parse_branch_name_type1(error_output)
if branch_name:
print(f"Detected error type 1, attempting to delete ref for branch: {branch_name}")
git_update_ref(branch_name)
else:
print("Error parsing branch name for type 1.")
# 错误类型2：尝试创建新的远程引用时，本地已经存在同名的引用
elif "exists; cannot create" in error_output:
branch_name = parse_branch_name_type2(error_output)
if branch_name:
print(f"Detected error type 2, attempting to delete ref for branch: {branch_name}")
git_update_ref(branch_name)
else:
print("Error parsing branch name for type 2.")
error_types = [
("is at", "but expected", parse_branch_name_type1, "type 1"),
("exists; cannot create", None, parse_branch_name_type2, "type 2"),
("Unable to create", "File exists", parse_branch_name_type3, "type 3")
]
for error_type in error_types:
if error_type[0] in error_output and (error_type[1] is None or error_type[1] in error_output):
branch_name = error_type[2](error_output)
if branch_name:
print(f"Detected error {error_type[3]}, attempting to delete ref for branch: {branch_name}")
git_update_ref(branch_name)
else:
print(f"Error parsing branch name for {error_type[3]}.")
break
# 主函数
def main():
fetch_result = git_fetch()
if fetch_result.returncode != 0: # 如果 git fetch 命令失败
if fetch_result.returncode != 0:
error_output = fetch_result.stderr
handle_error(error_output)
else:
print("Git fetch successful.")
prune_result = git_prune()
print(prune_result.returncode)
if prune_result.returncode != 0:
error_output = prune_result.stderr
print(error_output)
handle_error(error_output)
else:
print("Git prune successful.")
if __name__ == "__main__":
main()

16
source/dnode/mnode/impl/src/mndSma.c
View File

@ -797,7 +797,7 @@ static int32_t mndGetStreamNameFromSmaName(char *streamName, char *smaName) {
if (TSDB_CODE_SUCCESS != code) {
return code;
}
sprintf(streamName, "%d.%s", n.acctId, n.tname);
snprintf(streamName, TSDB_TABLE_FNAME_LEN,"%d.%s", n.acctId, n.tname);
return TSDB_CODE_SUCCESS;
}
@ -1222,7 +1222,7 @@ static int32_t mndGetSma(SMnode *pMnode, SUserIndexReq *indexReq, SUserIndexRsp
memcpy(rsp->dbFName, pSma->db, sizeof(pSma->db));
memcpy(rsp->tblFName, pSma->stb, sizeof(pSma->stb));
strcpy(rsp->indexType, TSDB_INDEX_TYPE_SMA);
tstrncpy(rsp->indexType, TSDB_INDEX_TYPE_SMA, TSDB_INDEX_TYPE_LEN);
SNodeList *pList = NULL;
int32_t extOffset = 0;
@ -1255,8 +1255,8 @@ int32_t mndGetTableSma(SMnode *pMnode, char *tbFName, STableIndexRsp *rsp, bool
return TSDB_CODE_SUCCESS;
}
strcpy(rsp->dbFName, pStb->db);
strcpy(rsp->tbName, pStb->name + strlen(pStb->db) + 1);
tstrncpy(rsp->dbFName, pStb->db, TSDB_DB_FNAME_LEN);
tstrncpy(rsp->tbName, pStb->name + strlen(pStb->db) + 1, TSDB_TABLE_NAME_LEN);
rsp->suid = pStb->uid;
rsp->version = pStb->smaVer;
mndReleaseStb(pMnode, pStb);
@ -1632,7 +1632,7 @@ static int32_t mndCreateTSMABuildCreateStreamReq(SCreateTSMACxt *pCxt) {
f.bytes = pExprNode->resType.bytes;
f.type = pExprNode->resType.type;
f.flags = COL_SMA_ON;
strcpy(f.name, pExprNode->userAlias);
tstrncpy(f.name, pExprNode->userAlias, TSDB_COL_NAME_LEN);
if (NULL == taosArrayPush(pCxt->pCreateStreamReq->pCols, &f)) {
code = terrno;
break;
@ -1735,7 +1735,7 @@ static int32_t mndCreateTSMATxnPrepare(SCreateTSMACxt* pCxt) {
}
dropStbReq.igNotExists = true;
strncpy(dropStbReq.name, pCxt->targetStbFullName, TSDB_TABLE_FNAME_LEN);
tstrncpy(dropStbReq.name, pCxt->targetStbFullName, TSDB_TABLE_FNAME_LEN);
dropStbUndoAction.epSet = createStreamRedoAction.epSet;
dropStbUndoAction.acceptableCode = TSDB_CODE_MND_STB_NOT_EXIST;
dropStbUndoAction.retryCode = TSDB_CODE_MND_STREAM_MUST_BE_DELETED;
@ -1836,7 +1836,7 @@ static int32_t mndTSMAGenerateOutputName(const char* tsmaName, char* streamName,
if (TSDB_CODE_SUCCESS != code) {
return code;
}
sprintf(streamName, "%d.%s", smaName.acctId, smaName.tname);
snprintf(streamName, TSDB_TABLE_FNAME_LEN, "%d.%s", smaName.acctId, smaName.tname);
snprintf(targetStbName, TSDB_TABLE_FNAME_LEN, "%s"TSMA_RES_STB_POSTFIX, tsmaName);
return TSDB_CODE_SUCCESS;
}
@ -2486,7 +2486,7 @@ static int32_t mndGetSomeTsmas(SMnode* pMnode, STableTSMAInfoRsp* pRsp, tsmaFilt
mndReleaseStb(pMnode, pStb);
TAOS_RETURN(code);
}
sprintf(streamName, "%d.%s", smaName.acctId, smaName.tname);
snprintf(streamName, TSDB_TABLE_FNAME_LEN, "%d.%s", smaName.acctId, smaName.tname);
pStream = NULL;
code = mndAcquireStream(pMnode, streamName, &pStream);

11
source/dnode/mnode/impl/src/mndStream.c
View File

@ -1931,11 +1931,6 @@ static int32_t mndProcessVgroupChange(SMnode *pMnode, SVgroupChangeInfo *pChange
sdbCancelFetch(pSdb, pIter);
return terrno = code;
}
code = mndStreamRegisterTrans(pTrans, MND_STREAM_TASK_UPDATE_NAME, pStream->uid);
if (code) {
mError("failed to register trans, transId:%d, and continue", pTrans->id);
}
}
if (!includeAllNodes) {
@ -1951,6 +1946,12 @@ static int32_t mndProcessVgroupChange(SMnode *pMnode, SVgroupChangeInfo *pChange
mDebug("stream:0x%" PRIx64 " %s involved node changed, create update trans, transId:%d", pStream->uid,
pStream->name, pTrans->id);
// NOTE: for each stream, we register one trans entry for task update
code = mndStreamRegisterTrans(pTrans, MND_STREAM_TASK_UPDATE_NAME, pStream->uid);
if (code) {
mError("failed to register trans, transId:%d, and continue", pTrans->id);
}
code = mndStreamSetUpdateEpsetAction(pMnode, pStream, pChangeInfo, pTrans);
// todo: not continue, drop all and retry again

54
source/dnode/mnode/impl/src/mndStreamTrans.c
View File

@ -35,7 +35,11 @@ int32_t mndStreamClearFinishedTrans(SMnode *pMnode, int32_t *pNumOfActiveChkpt)
size_t keyLen = 0;
void *pIter = NULL;
SArray *pList = taosArrayInit(4, sizeof(SKeyInfo));
int32_t num = 0;
int32_t numOfChkpt = 0;
if (pNumOfActiveChkpt != NULL) {
*pNumOfActiveChkpt = 0;
}
if (pList == NULL) {
return terrno;
@ -50,15 +54,15 @@ int32_t mndStreamClearFinishedTrans(SMnode *pMnode, int32_t *pNumOfActiveChkpt)
void *pKey = taosHashGetKey(pEntry, &keyLen);
// key is the name of src/dst db name
SKeyInfo info = {.pKey = pKey, .keyLen = keyLen};
mDebug("transId:%d %s startTs:%" PRId64 " cleared since finished", pEntry->transId, pEntry->name,
pEntry->startTime);
mDebug("transId:%d stream:0x%" PRIx64 " %s startTs:%" PRId64 " cleared since finished", pEntry->transId,
pEntry->streamId, pEntry->name, pEntry->startTime);
void* p = taosArrayPush(pList, &info);
if (p == NULL) {
return terrno;
}
} else {
if (strcmp(pEntry->name, MND_STREAM_CHECKPOINT_NAME) == 0) {
num++;
numOfChkpt++;
}
mndReleaseTrans(pMnode, pTrans);
}
@ -78,48 +82,34 @@ int32_t mndStreamClearFinishedTrans(SMnode *pMnode, int32_t *pNumOfActiveChkpt)
}
}
mDebug("clear %d finished stream-trans, remained:%d, active checkpoint trans:%d", size,
taosHashGetSize(execInfo.transMgmt.pDBTrans), num);
mDebug("clear %d finished stream-trans, active trans:%d, active checkpoint trans:%d", size,
taosHashGetSize(execInfo.transMgmt.pDBTrans), numOfChkpt);
taosArrayDestroy(pList);
if (pNumOfActiveChkpt != NULL) {
*pNumOfActiveChkpt = num;
*pNumOfActiveChkpt = numOfChkpt;
}
return 0;
}
// * Transactions of different streams are not related. Here only check the conflict of transaction for a given stream.
// For a given stream:
// 1. checkpoint trans is conflict with any other trans except for the drop and reset trans.
// 2. create/drop/reset/update trans are conflict with any other trans.
int32_t mndStreamTransConflictCheck(SMnode *pMnode, int64_t streamId, const char *pTransName, bool lock) {
if (lock) {
streamMutexLock(&execInfo.lock);
}
static int32_t doStreamTransConflictCheck(SMnode *pMnode, int64_t streamId, const char *pTransName) {
int32_t num = taosHashGetSize(execInfo.transMgmt.pDBTrans);
if (num <= 0) {
if (lock) {
streamMutexUnlock(&execInfo.lock);
}
return 0;
}
// if any task updates exist, any other stream trans are not allowed to be created
int32_t code = mndStreamClearFinishedTrans(pMnode, NULL);
if (code) {
mError("failed to clear finish trans, code:%s", tstrerror(code));
mError("failed to clear finish trans, code:%s, and continue", tstrerror(code));
}
SStreamTransInfo *pEntry = taosHashGet(execInfo.transMgmt.pDBTrans, &streamId, sizeof(streamId));
if (pEntry != NULL) {
SStreamTransInfo tInfo = *pEntry;
if (lock) {
streamMutexUnlock(&execInfo.lock);
}
if (strcmp(tInfo.name, MND_STREAM_CHECKPOINT_NAME) == 0) {
if ((strcmp(pTransName, MND_STREAM_DROP_NAME) != 0) && (strcmp(pTransName, MND_STREAM_TASK_RESET_NAME) != 0) &&
(strcmp(pTransName, MND_STREAM_RESTART_NAME) != 0)) {
@ -141,11 +131,25 @@ int32_t mndStreamTransConflictCheck(SMnode *pMnode, int64_t streamId, const char
mDebug("stream:0x%" PRIx64 " no conflict trans existed, continue create trans", streamId);
}
return TSDB_CODE_SUCCESS;
}
// * Transactions of different streams are not related. Here only check the conflict of transaction for a given stream.
// For a given stream:
// 1. checkpoint trans is conflict with any other trans except for the drop and reset trans.
// 2. create/drop/reset/update trans are conflict with any other trans.
int32_t mndStreamTransConflictCheck(SMnode *pMnode, int64_t streamId, const char *pTransName, bool lock) {
if (lock) {
streamMutexLock(&execInfo.lock);
}
int32_t code = doStreamTransConflictCheck(pMnode, streamId, pTransName);
if (lock) {
streamMutexUnlock(&execInfo.lock);
}
return 0;
return code;
}
int32_t mndStreamGetRelTrans(SMnode *pMnode, int64_t streamId) {

67
source/dnode/mnode/impl/src/mndTrans.c
View File

@ -52,10 +52,17 @@ static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans, bool t
static bool mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans, bool topHalf);
static bool mndTransPerformFinishStage(SMnode *pMnode, STrans *pTrans, bool topHalf);
static bool mndCannotExecuteTransAction(SMnode *pMnode, bool topHalf) {
return (!pMnode->deploy && !mndIsLeader(pMnode)) || !topHalf;
static inline bool mndTransIsInSyncContext(bool topHalf) { return !topHalf; }
static bool mndCannotExecuteTrans(SMnode *pMnode, bool topHalf) {
bool isLeader = mndIsLeader(pMnode);
bool ret = (!pMnode->deploy && !isLeader) || mndTransIsInSyncContext(topHalf);
if (ret) mDebug("cannot execute trans action, deploy:%d, isLeader:%d, topHalf:%d", pMnode->deploy, isLeader, topHalf);
return ret;
}
static inline char *mndStrExecutionContext(bool topHalf) { return topHalf ? "transContext" : "syncContext"; }
static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans);
static int32_t mndProcessTransTimer(SRpcMsg *pReq);
static int32_t mndProcessTtl(SRpcMsg *pReq);
@ -1327,7 +1334,7 @@ static int32_t mndTransWriteSingleLog(SMnode *pMnode, STrans *pTrans, STransActi
// execute in trans context
static int32_t mndTransSendSingleMsg(SMnode *pMnode, STrans *pTrans, STransAction *pAction, bool topHalf) {
if (pAction->msgSent) return 0;
if (mndCannotExecuteTransAction(pMnode, topHalf)) {
if (mndCannotExecuteTrans(pMnode, topHalf)) {
TAOS_RETURN(TSDB_CODE_MND_TRANS_CTX_SWITCH);
}
@ -1473,8 +1480,8 @@ static int32_t mndTransExecuteActions(SMnode *pMnode, STrans *pTrans, SArray *pA
static int32_t mndTransExecuteRedoActions(SMnode *pMnode, STrans *pTrans, bool topHalf) {
int32_t code = mndTransExecuteActions(pMnode, pTrans, pTrans->redoActions, topHalf);
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS && code != TSDB_CODE_MND_TRANS_CTX_SWITCH) {
mError("trans:%d, failed to execute redoActions since:%s, code:0x%x, topHalf(TransContext):%d", pTrans->id,
terrstr(), terrno, topHalf);
mError("trans:%d, failed to execute redoActions since:%s, code:0x%x, in %s", pTrans->id, terrstr(), terrno,
mndStrExecutionContext(topHalf));
}
return code;
}
@ -1482,8 +1489,8 @@ static int32_t mndTransExecuteRedoActions(SMnode *pMnode, STrans *pTrans, bool t
static int32_t mndTransExecuteUndoActions(SMnode *pMnode, STrans *pTrans, bool topHalf) {
int32_t code = mndTransExecuteActions(pMnode, pTrans, pTrans->undoActions, topHalf);
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS && code != TSDB_CODE_MND_TRANS_CTX_SWITCH) {
mError("trans:%d, failed to execute undoActions since %s. topHalf(TransContext):%d", pTrans->id, terrstr(),
topHalf);
mError("trans:%d, failed to execute undoActions since %s. in %s", pTrans->id, terrstr(),
mndStrExecutionContext(topHalf));
}
return code;
}
@ -1491,8 +1498,8 @@ static int32_t mndTransExecuteUndoActions(SMnode *pMnode, STrans *pTrans, bool t
static int32_t mndTransExecuteCommitActions(SMnode *pMnode, STrans *pTrans, bool topHalf) {
int32_t code = mndTransExecuteActions(pMnode, pTrans, pTrans->commitActions, topHalf);
if (code != 0 && code != TSDB_CODE_ACTION_IN_PROGRESS && code != TSDB_CODE_MND_TRANS_CTX_SWITCH) {
mError("trans:%d, failed to execute commitActions since %s. topHalf(TransContext):%d", pTrans->id, terrstr(),
topHalf);
mError("trans:%d, failed to execute commitActions since %s. in %s", pTrans->id, terrstr(),
mndStrExecutionContext(topHalf));
}
return code;
}
@ -1512,7 +1519,7 @@ static int32_t mndTransExecuteActionsSerial(SMnode *pMnode, STrans *pTrans, SArr
for (int32_t action = pTrans->actionPos; action < numOfActions; ++action) {
STransAction *pAction = taosArrayGet(pActions, action);
mInfo("trans:%d, current action:%d, stage:%s, actionType(0:log,1:msg):%d", pTrans->id, pTrans->actionPos,
mInfo("trans:%d, current action:%d, stage:%s, actionType(1:msg,2:log):%d", pTrans->id, pTrans->actionPos,
mndTransStr(pAction->stage), pAction->actionType);
code = mndTransExecSingleAction(pMnode, pTrans, pAction, topHalf);
@ -1543,11 +1550,11 @@ static int32_t mndTransExecuteActionsSerial(SMnode *pMnode, STrans *pTrans, SArr
}
mndSetTransLastAction(pTrans, pAction);
if (mndCannotExecuteTransAction(pMnode, topHalf)) {
if (mndCannotExecuteTrans(pMnode, topHalf)) {
pTrans->lastErrorNo = code;
pTrans->code = code;
mInfo("trans:%d, %s:%d, topHalf(TransContext):%d, not execute next action, code:%s", pTrans->id,
mndTransStr(pAction->stage), action, topHalf, tstrerror(code));
mInfo("trans:%d, %s:%d, cannot execute next action in %s, code:%s", pTrans->id, mndTransStr(pAction->stage),
action, mndStrExecutionContext(topHalf), tstrerror(code));
break;
}
@ -1648,20 +1655,21 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans, bool
code = mndTransExecuteRedoActions(pMnode, pTrans, topHalf);
}
if (mndCannotExecuteTransAction(pMnode, topHalf)) {
if (code != 0 && code != TSDB_CODE_MND_TRANS_CTX_SWITCH && mndTransIsInSyncContext(topHalf)) {
pTrans->lastErrorNo = code;
pTrans->code = code;
bool continueExec = true;
if (code != 0 && code != TSDB_CODE_MND_TRANS_CTX_SWITCH) {
mInfo(
"trans:%d, failed to execute, will retry redo action stage in 100 ms , in %s, "
"continueExec:%d, code:%s",
pTrans->id, mndStrExecutionContext(topHalf), continueExec, tstrerror(code));
taosMsleep(100);
continueExec = true;
return true;
} else {
continueExec = false;
if (mndCannotExecuteTrans(pMnode, topHalf)) {
mInfo("trans:%d, cannot continue to execute redo action stage in %s, continueExec:%d, code:%s", pTrans->id,
mndStrExecutionContext(topHalf), continueExec, tstrerror(code));
return false;
}
mInfo("trans:%d, cannot execute redo action stage, topHalf(TransContext):%d, continueExec:%d, code:%s", pTrans->id,
topHalf, continueExec, tstrerror(code));
return continueExec;
}
terrno = code;
@ -1704,9 +1712,9 @@ static bool mndTransPerformRedoActionStage(SMnode *pMnode, STrans *pTrans, bool
return continueExec;
}
// in trans context
// execute in trans context
static bool mndTransPerformCommitStage(SMnode *pMnode, STrans *pTrans, bool topHalf) {
if (mndCannotExecuteTransAction(pMnode, topHalf)) return false;
if (mndCannotExecuteTrans(pMnode, topHalf)) return false;
bool continueExec = true;
int32_t code = mndTransCommit(pMnode, pTrans);
@ -1760,7 +1768,7 @@ static bool mndTransPerformUndoActionStage(SMnode *pMnode, STrans *pTrans, bool
code = mndTransExecuteUndoActions(pMnode, pTrans, topHalf);
}
if (mndCannotExecuteTransAction(pMnode, topHalf)) return false;
if (mndCannotExecuteTrans(pMnode, topHalf)) return false;
terrno = code;
if (code == 0) {
@ -1781,7 +1789,7 @@ static bool mndTransPerformUndoActionStage(SMnode *pMnode, STrans *pTrans, bool
// in trans context
static bool mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans, bool topHalf) {
if (mndCannotExecuteTransAction(pMnode, topHalf)) return false;
if (mndCannotExecuteTrans(pMnode, topHalf)) return false;
bool continueExec = true;
int32_t code = mndTransRollback(pMnode, pTrans);
@ -1798,8 +1806,9 @@ static bool mndTransPerformRollbackStage(SMnode *pMnode, STrans *pTrans, bool to
return continueExec;
}
// excute in trans context
static bool mndTransPerformPreFinishStage(SMnode *pMnode, STrans *pTrans, bool topHalf) {
if (mndCannotExecuteTransAction(pMnode, topHalf)) return false;
if (mndCannotExecuteTrans(pMnode, topHalf)) return false;
bool continueExec = true;
int32_t code = mndTransPreFinish(pMnode, pTrans);
@ -1842,8 +1851,8 @@ void mndTransExecuteImp(SMnode *pMnode, STrans *pTrans, bool topHalf) {
bool continueExec = true;
while (continueExec) {
mInfo("trans:%d, continue to execute, stage:%s createTime:%" PRId64 " topHalf(TransContext):%d", pTrans->id,
mndTransStr(pTrans->stage), pTrans->createdTime, topHalf);
mInfo("trans:%d, continue to execute stage:%s in %s, createTime:%" PRId64 "", pTrans->id,
mndTransStr(pTrans->stage), mndStrExecutionContext(topHalf), pTrans->createdTime);
pTrans->lastExecTime = taosGetTimestampMs();
switch (pTrans->stage) {
case TRN_STAGE_PREPARE:

3
source/dnode/vnode/src/sma/smaRollup.c
View File

@ -254,11 +254,12 @@ static void tdRSmaTaskInit(SStreamMeta *pMeta, SRSmaInfoItem *pItem, SStreamTask
}
static void tdRSmaTaskRemove(SStreamMeta *pMeta, int64_t streamId, int32_t taskId) {
streamMetaWLock(pMeta);
int32_t code = streamMetaUnregisterTask(pMeta, streamId, taskId);
if (code != 0) {
smaError("vgId:%d, rsma task:%" PRIi64 ",%d drop failed since %s", pMeta->vgId, streamId, taskId, tstrerror(code));
}
streamMetaWLock(pMeta);
int32_t numOfTasks = streamMetaGetNumOfTasks(pMeta);
if (streamMetaCommit(pMeta) < 0) {
// persist to disk

104
source/dnode/vnode/src/tq/tq.c
View File

@ -1122,6 +1122,18 @@ _OVER:
return code;
}
// always return success to mnode
//todo: handle failure of build and send msg to mnode
static void doSendChkptSourceRsp(SStreamCheckpointSourceReq* pReq, SRpcHandleInfo* pRpcInfo, int32_t code,
int32_t taskId) {
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(pReq, pRpcInfo, &rsp, code);
if (ret) { // suppress the error in build checkpoint source rsp
tqError("s-task:0x%x failed to build checkpoint-source rsp, code:%s", taskId, tstrerror(ret));
}
tmsgSendRsp(&rsp); // error occurs
}
// no matter what kinds of error happened, make sure the mnode will receive the success execution code.
int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp) {
int32_t vgId = TD_VID(pTq->pVnode);
@ -1129,84 +1141,57 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
char* msg = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead));
int32_t len = pMsg->contLen - sizeof(SMsgHead);
int32_t code = 0;
SStreamCheckpointSourceReq req = {0};
SDecoder decoder = {0};
SStreamTask* pTask = NULL;
int64_t checkpointId = 0;
// disable auto rsp to mnode
pRsp->info.handle = NULL;
SStreamCheckpointSourceReq req = {0};
SDecoder decoder;
tDecoderInit(&decoder, (uint8_t*)msg, len);
if (tDecodeStreamCheckpointSourceReq(&decoder, &req) < 0) {
code = TSDB_CODE_MSG_DECODE_ERROR;
tDecoderClear(&decoder);
tqError("vgId:%d failed to decode checkpoint-source msg, code:%s", vgId, tstrerror(code));
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:0x%x failed to build checkpoint-source rsp, code:%s", req.taskId, tstrerror(code));
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS; // always return success to mnode,
}
tmsgSendRsp(&rsp); // error occurs
return TSDB_CODE_SUCCESS; // always return success to mnode, todo: handle failure of build and send msg to mnode
}
tDecoderClear(&decoder);
if (!vnodeIsRoleLeader(pTq->pVnode)) {
tqDebug("vgId:%d not leader, ignore checkpoint-source msg, s-task:0x%x", vgId, req.taskId);
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:0x%x failed to build checkpoint-source rsp, code:%s", req.taskId, tstrerror(code));
}
tmsgSendRsp(&rsp); // error occurs
return TSDB_CODE_SUCCESS; // always return success to mnode, todo: handle failure of build and send msg to mnode
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS; // always return success to mnode
}
if (!pTq->pVnode->restored) {
tqDebug("vgId:%d checkpoint-source msg received during restoring, checkpointId:%" PRId64
", transId:%d s-task:0x%x ignore it",
vgId, req.checkpointId, req.transId, req.taskId);
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:0x%x failed to build checkpoint-source rsp, code:%s", req.taskId, tstrerror(code));
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS; // always return success to mnode
}
tmsgSendRsp(&rsp); // error occurs
return TSDB_CODE_SUCCESS; // always return success to mnode, , todo: handle failure of build and send msg to mnode
}
SStreamTask* pTask = NULL;
code = streamMetaAcquireTask(pMeta, req.streamId, req.taskId, &pTask);
if (pTask == NULL || code != 0) {
tqError("vgId:%d failed to find s-task:0x%x, ignore checkpoint msg. checkpointId:%" PRId64
" transId:%d it may have been destroyed",
vgId, req.taskId, req.checkpointId, req.transId);
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:%s failed to build checkpoint-source rsp, code:%s", pTask->id.idStr, tstrerror(code));
}
tmsgSendRsp(&rsp); // error occurs
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS;
}
if (pTask->status.downstreamReady != 1) {
streamTaskSetFailedChkptInfo(pTask, req.transId, req.checkpointId); // record the latest failed checkpoint id
// record the latest failed checkpoint id
streamTaskSetFailedChkptInfo(pTask, req.transId, req.checkpointId);
tqError("s-task:%s not ready for checkpoint, since downstream not ready, ignore this checkpointId:%" PRId64
", transId:%d set it failed",
pTask->id.idStr, req.checkpointId, req.transId);
streamMetaReleaseTask(pMeta, pTask);
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:%s failed to build checkpoint-source rsp, code:%s", pTask->id.idStr, tstrerror(code));
}
tmsgSendRsp(&rsp); // error occurs
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS; // todo retry handle error
}
@ -1221,14 +1206,7 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
streamMutexUnlock(&pTask->lock);
streamMetaReleaseTask(pMeta, pTask);
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:%s failed to build checkpoint-source rsp, code:%s", pTask->id.idStr, tstrerror(code));
}
tmsgSendRsp(&rsp); // error occurs
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS;
}
} else {
@ -1240,7 +1218,6 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
// check if the checkpoint msg already sent or not.
if (status == TASK_STATUS__CK) {
int64_t checkpointId = 0;
streamTaskGetActiveCheckpointInfo(pTask, NULL, &checkpointId);
tqWarn("s-task:%s repeatly recv checkpoint-source msg checkpointId:%" PRId64
@ -1249,7 +1226,7 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
streamMutexUnlock(&pTask->lock);
streamMetaReleaseTask(pMeta, pTask);
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SYN_PROPOSE_NOT_READY, req.taskId);
return TSDB_CODE_SUCCESS;
} else { // checkpoint already finished, and not in checkpoint status
if (req.checkpointId <= pTask->chkInfo.checkpointId) {
@ -1259,15 +1236,7 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
streamMutexUnlock(&pTask->lock);
streamMetaReleaseTask(pMeta, pTask);
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:%s failed to build checkpoint-source rsp, code:%s", pTask->id.idStr, tstrerror(code));
}
tmsgSendRsp(&rsp); // error occurs
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS;
}
}
@ -1278,7 +1247,9 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
if (code) {
qError("s-task:%s (vgId:%d) failed to process checkpoint-source req, code:%s", pTask->id.idStr, vgId,
tstrerror(code));
return code;
streamMetaReleaseTask(pMeta, pTask);
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
return TSDB_CODE_SUCCESS;
}
if (req.mndTrigger) {
@ -1293,13 +1264,8 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp)
code = streamAddCheckpointSourceRspMsg(&req, &pMsg->info, pTask);
if (code != TSDB_CODE_SUCCESS) {
SRpcMsg rsp = {0};
int32_t ret = streamTaskBuildCheckpointSourceRsp(&req, &pMsg->info, &rsp, TSDB_CODE_SUCCESS);
if (ret) { // suppress the error in build checkpointsource rsp
tqError("s-task:%s failed to build checkpoint-source rsp, code:%s", pTask->id.idStr, tstrerror(code));
}
tmsgSendRsp(&rsp); // error occurs
return TSDB_CODE_SUCCESS;
streamTaskSetCheckpointFailed(pTask); // set the checkpoint failed
doSendChkptSourceRsp(&req, &pMsg->info, TSDB_CODE_SUCCESS, req.taskId);
}
streamMetaReleaseTask(pMeta, pTask);

8
source/dnode/vnode/src/tq/tqSink.c
View File

@ -13,9 +13,7 @@
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include <common/tmsg.h>
#include "tcommon.h"
#include "tmsg.h"
#include "tq.h"
#define IS_NEW_SUBTB_RULE(_t) (((_t)->ver >= SSTREAM_TASK_SUBTABLE_CHANGED_VER) && ((_t)->subtableWithoutMd5 != 1))
@ -50,7 +48,7 @@ static int32_t doPutSinkTableInfoIntoCache(SSHashObj* pSinkTableMap, STableSinkI
static bool doGetSinkTableInfoFromCache(SSHashObj* pTableInfoMap, uint64_t groupId, STableSinkInfo** pInfo);
static int32_t doRemoveSinkTableInfoInCache(SSHashObj* pSinkTableMap, uint64_t groupId, const char* id);
static int32_t checkTagSchema(SStreamTask* pTask, SVnode* pVnode);
static void reubuildAndSendMultiResBlock(SStreamTask* pTask, const SArray* pBlocks, SVnode* pVnode, int64_t earlyTs);
static void rebuildAndSendMultiResBlock(SStreamTask* pTask, const SArray* pBlocks, SVnode* pVnode, int64_t earlyTs);
static int32_t handleResultBlockMsg(SStreamTask* pTask, SSDataBlock* pDataBlock, int32_t index, SVnode* pVnode,
int64_t earlyTs);
static int32_t doWaitForDstTableDropped(SVnode* pVnode, SStreamTask* pTask, const char* dstTableName);
@ -1187,7 +1185,7 @@ void tqSinkDataIntoDstTable(SStreamTask* pTask, void* vnode, void* data) {
return;
}
reubuildAndSendMultiResBlock(pTask, pBlocks, pVnode, earlyTs);
rebuildAndSendMultiResBlock(pTask, pBlocks, pVnode, earlyTs);
}
}
@ -1290,7 +1288,7 @@ int32_t doBuildAndSendDeleteMsg(SVnode* pVnode, char* stbFullName, SSDataBlock*
return TSDB_CODE_SUCCESS;
}
void reubuildAndSendMultiResBlock(SStreamTask* pTask, const SArray* pBlocks, SVnode* pVnode, int64_t earlyTs) {
void rebuildAndSendMultiResBlock(SStreamTask* pTask, const SArray* pBlocks, SVnode* pVnode, int64_t earlyTs) {
int32_t code = 0;
const char* id = pTask->id.idStr;
int32_t vgId = pTask->pMeta->vgId;

106
source/dnode/vnode/src/tq/tqStreamTask.c
View File

@ -17,19 +17,20 @@
#include "vnd.h"
#define MAX_REPEAT_SCAN_THRESHOLD 3
#define SCAN_WAL_IDLE_DURATION 100
#define SCAN_WAL_IDLE_DURATION 500 // idle for 500ms to do next wal scan
typedef struct SBuildScanWalMsgParam {
int64_t metaId;
int32_t numOfTasks;
} SBuildScanWalMsgParam;
static int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta, bool* pScanIdle);
static int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta);
static int32_t setWalReaderStartOffset(SStreamTask* pTask, int32_t vgId);
static bool handleFillhistoryScanComplete(SStreamTask* pTask, int64_t ver);
static bool taskReadyForDataFromWal(SStreamTask* pTask);
static int32_t doPutDataIntoInputQ(SStreamTask* pTask, int64_t maxVer, int32_t* numOfItems, bool* pSucc);
static int32_t tqScanWalInFuture(STQ* pTq, int32_t numOfTasks, int32_t idleDuration);
static int32_t doScanWalAsync(STQ* pTq, bool ckPause);
// extract data blocks(submit/delete) from WAL, and add them into the input queue for all the sources tasks.
int32_t tqScanWal(STQ* pTq) {
@ -37,12 +38,11 @@ int32_t tqScanWal(STQ* pTq) {
int32_t vgId = pMeta->vgId;
int64_t st = taosGetTimestampMs();
int32_t numOfTasks = 0;
bool shouldIdle = true;
tqDebug("vgId:%d continue to check if data in wal are available, scanCounter:%d", vgId, pMeta->scanInfo.scanCounter);
// check all tasks
int32_t code = doScanWalForAllTasks(pMeta, &shouldIdle);
int32_t code = doScanWalForAllTasks(pMeta);
if (code) {
tqError("vgId:%d failed to start all tasks, try next time, code:%s", vgId, tstrerror(code));
return code;
@ -133,10 +133,9 @@ int32_t tqScanWalInFuture(STQ* pTq, int32_t numOfTasks, int32_t idleDuration) {
}
int32_t tqScanWalAsync(STQ* pTq, bool ckPause) {
int32_t vgId = TD_VID(pTq->pVnode);
SStreamMeta* pMeta = pTq->pStreamMeta;
bool alreadyRestored = pTq->pVnode->restored;
int32_t numOfTasks = 0;
int32_t code = 0;
// do not launch the stream tasks, if it is a follower or not restored vnode.
if (!(vnodeIsRoleLeader(pTq->pVnode) && alreadyRestored)) {
@ -144,47 +143,8 @@ int32_t tqScanWalAsync(STQ* pTq, bool ckPause) {
}
streamMetaWLock(pMeta);
numOfTasks = taosArrayGetSize(pMeta->pTaskList);
if (numOfTasks == 0) {
tqDebug("vgId:%d no stream tasks existed to run", vgId);
code = doScanWalAsync(pTq, ckPause);
streamMetaWUnLock(pMeta);
return 0;
}
if (pMeta->startInfo.startAllTasks) {
tqTrace("vgId:%d in restart procedure, not scan wal", vgId);
streamMetaWUnLock(pMeta);
return 0;
}
pMeta->scanInfo.scanCounter += 1;
if (pMeta->scanInfo.scanCounter > MAX_REPEAT_SCAN_THRESHOLD) {
pMeta->scanInfo.scanCounter = MAX_REPEAT_SCAN_THRESHOLD;
}
if (pMeta->scanInfo.scanCounter > 1) {
tqDebug("vgId:%d wal read task has been launched, remain scan times:%d", vgId, pMeta->scanInfo.scanCounter);
streamMetaWUnLock(pMeta);
return 0;
}
int32_t numOfPauseTasks = pMeta->numOfPausedTasks;
if (ckPause && numOfTasks == numOfPauseTasks) {
tqDebug("vgId:%d ignore all submit, all streams had been paused, reset the walScanCounter", vgId);
// reset the counter value, since we do not launch the scan wal operation.
pMeta->scanInfo.scanCounter = 0;
streamMetaWUnLock(pMeta);
return 0;
}
tqDebug("vgId:%d create msg to start wal scan to launch stream tasks, numOfTasks:%d, vnd restored:%d", vgId,
numOfTasks, alreadyRestored);
int32_t code = streamTaskSchedTask(&pTq->pVnode->msgCb, vgId, 0, 0, STREAM_EXEC_T_EXTRACT_WAL_DATA);
streamMetaWUnLock(pMeta);
return code;
}
@ -368,11 +328,8 @@ int32_t doPutDataIntoInputQ(SStreamTask* pTask, int64_t maxVer, int32_t* numOfIt
return code;
}
int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta, bool* pScanIdle) {
*pScanIdle = true;
bool noDataInWal = true;
int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta) {
int32_t vgId = pStreamMeta->vgId;
int32_t numOfTasks = taosArrayGetSize(pStreamMeta->pTaskList);
if (numOfTasks == 0) {
return TSDB_CODE_SUCCESS;
@ -410,8 +367,6 @@ int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta, bool* pScanIdle) {
continue;
}
*pScanIdle = false;
// seek the stored version and extract data from WAL
code = setWalReaderStartOffset(pTask, vgId);
if (code != TSDB_CODE_SUCCESS) {
@ -437,7 +392,6 @@ int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta, bool* pScanIdle) {
streamMutexUnlock(&pTask->lock);
if ((numOfItems > 0) || hasNewData) {
noDataInWal = false;
code = streamTrySchedExec(pTask);
if (code != TSDB_CODE_SUCCESS) {
streamMetaReleaseTask(pStreamMeta, pTask);
@ -449,11 +403,47 @@ int32_t doScanWalForAllTasks(SStreamMeta* pStreamMeta, bool* pScanIdle) {
streamMetaReleaseTask(pStreamMeta, pTask);
}
// all wal are checked, and no new data available in wal.
if (noDataInWal) {
*pScanIdle = true;
}
taosArrayDestroy(pTaskList);
return TSDB_CODE_SUCCESS;
}
int32_t doScanWalAsync(STQ* pTq, bool ckPause) {
SStreamMeta* pMeta = pTq->pStreamMeta;
bool alreadyRestored = pTq->pVnode->restored;
int32_t vgId = pMeta->vgId;
int32_t numOfTasks = taosArrayGetSize(pMeta->pTaskList);
if (numOfTasks == 0) {
tqDebug("vgId:%d no stream tasks existed to run", vgId);
return 0;
}
if (pMeta->startInfo.startAllTasks) {
tqTrace("vgId:%d in restart procedure, not scan wal", vgId);
return 0;
}
pMeta->scanInfo.scanCounter += 1;
if (pMeta->scanInfo.scanCounter > MAX_REPEAT_SCAN_THRESHOLD) {
pMeta->scanInfo.scanCounter = MAX_REPEAT_SCAN_THRESHOLD;
}
if (pMeta->scanInfo.scanCounter > 1) {
tqDebug("vgId:%d wal read task has been launched, remain scan times:%d", vgId, pMeta->scanInfo.scanCounter);
return 0;
}
int32_t numOfPauseTasks = pMeta->numOfPausedTasks;
if (ckPause && numOfTasks == numOfPauseTasks) {
tqDebug("vgId:%d ignore all submit, all streams had been paused, reset the walScanCounter", vgId);
// reset the counter value, since we do not launch the scan wal operation.
pMeta->scanInfo.scanCounter = 0;
return 0;
}
tqDebug("vgId:%d create msg to start wal scan to launch stream tasks, numOfTasks:%d, vnd restored:%d", vgId,
numOfTasks, alreadyRestored);
return streamTaskSchedTask(&pTq->pVnode->msgCb, vgId, 0, 0, STREAM_EXEC_T_EXTRACT_WAL_DATA);
}

3
source/dnode/vnode/src/tqCommon/tqCommon.c
View File

@ -718,8 +718,6 @@ int32_t tqStreamTaskProcessDropReq(SStreamMeta* pMeta, char* msg, int32_t msgLen
}
}
streamMetaWUnLock(pMeta);
// drop the related fill-history task firstly
if (hTaskId.taskId != 0 && hTaskId.streamId != 0) {
tqDebug("s-task:0x%x vgId:%d drop rel fill-history task:0x%x firstly", pReq->taskId, vgId, (int32_t)hTaskId.taskId);
@ -737,7 +735,6 @@ int32_t tqStreamTaskProcessDropReq(SStreamMeta* pMeta, char* msg, int32_t msgLen
}
// commit the update
streamMetaWLock(pMeta);
int32_t numOfTasks = streamMetaGetNumOfTasks(pMeta);
tqDebug("vgId:%d task:0x%x dropped, remain tasks:%d", vgId, pReq->taskId, numOfTasks);

9
source/libs/executor/src/streamfilloperator.c
View File

@ -987,7 +987,14 @@ _end:
}
void resetStreamFillSup(SStreamFillSupporter* pFillSup) {
_hash_fn_t hashFn = taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY);
SSHashObj* pNewMap = tSimpleHashInit(16, hashFn);
if (pNewMap != NULL) {
tSimpleHashCleanup(pFillSup->pResMap);
pFillSup->pResMap = pNewMap;
} else {
tSimpleHashClear(pFillSup->pResMap);
}
pFillSup->hasDelete = false;
}
void resetStreamFillInfo(SStreamFillOperatorInfo* pInfo) {
@ -1406,6 +1413,7 @@ static int32_t doStreamForceFillNext(SOperatorInfo* pOperator, SSDataBlock** ppR
}
pInfo->stateStore.streamStateClearExpiredState(pInfo->pState);
resetStreamFillInfo(pInfo);
setStreamOperatorCompleted(pOperator);
(*ppRes) = NULL;
goto _end;
@ -1477,6 +1485,7 @@ static int32_t doStreamForceFillNext(SOperatorInfo* pOperator, SSDataBlock** ppR
if ((*ppRes) == NULL) {
pInfo->stateStore.streamStateClearExpiredState(pInfo->pState);
resetStreamFillInfo(pInfo);
setStreamOperatorCompleted(pOperator);
}

66
source/libs/executor/src/streamintervalsliceoperator.c
View File

@ -120,6 +120,36 @@ void initIntervalSlicePoint(SStreamAggSupporter* pAggSup, STimeWindow* pTWin, in
pPoint->pLastRow = POINTER_SHIFT(pPoint->pFinished, sizeof(bool));
}
int32_t getIntervalSlicePrevStateBuf(SStreamAggSupporter* pAggSup, SInterval* pInterval, SWinKey* pCurKey,
SInervalSlicePoint* pPrevPoint) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SWinKey prevKey = {.groupId = pCurKey->groupId};
SET_WIN_KEY_INVALID(prevKey.ts);
int32_t prevVLen = 0;
int32_t prevWinCode = TSDB_CODE_SUCCESS;
code = pAggSup->stateStore.streamStateGetPrev(pAggSup->pState, pCurKey, &prevKey, (void**)&pPrevPoint->pResPos,
&prevVLen, &prevWinCode);
QUERY_CHECK_CODE(code, lino, _end);
if (prevWinCode == TSDB_CODE_SUCCESS) {
STimeWindow prevSTW = {.skey = prevKey.ts};
prevSTW.ekey = taosTimeGetIntervalEnd(prevSTW.skey, pInterval);
initIntervalSlicePoint(pAggSup, &prevSTW, pCurKey->groupId, pPrevPoint);
qDebug("===stream=== set stream twa prev point buf.ts:%" PRId64 ", groupId:%" PRIu64 ", res:%d",
pPrevPoint->winKey.win.skey, pPrevPoint->winKey.groupId, prevWinCode);
} else {
SET_WIN_KEY_INVALID(pPrevPoint->winKey.win.skey);
SET_WIN_KEY_INVALID(pPrevPoint->winKey.win.ekey);
}
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
static int32_t getIntervalSliceCurStateBuf(SStreamAggSupporter* pAggSup, SInterval* pInterval, bool needPrev, STimeWindow* pTWin, int64_t groupId,
SInervalSlicePoint* pCurPoint, SInervalSlicePoint* pPrevPoint, int32_t* pWinCode) {
int32_t code = TSDB_CODE_SUCCESS;
@ -136,24 +166,8 @@ static int32_t getIntervalSliceCurStateBuf(SStreamAggSupporter* pAggSup, SInterv
initIntervalSlicePoint(pAggSup, pTWin, groupId, pCurPoint);
if (needPrev) {
SWinKey prevKey = {.groupId = groupId};
SET_WIN_KEY_INVALID(prevKey.ts);
int32_t prevVLen = 0;
int32_t prevWinCode = TSDB_CODE_SUCCESS;
code = pAggSup->stateStore.streamStateGetPrev(pAggSup->pState, &curKey, &prevKey, (void**)&pPrevPoint->pResPos,
&prevVLen, &prevWinCode);
code = getIntervalSlicePrevStateBuf(pAggSup, pInterval, &curKey, pPrevPoint);
QUERY_CHECK_CODE(code, lino, _end);
if (prevWinCode == TSDB_CODE_SUCCESS) {
STimeWindow prevSTW = {.skey = prevKey.ts};
prevSTW.ekey = taosTimeGetIntervalEnd(prevSTW.skey, pInterval);
initIntervalSlicePoint(pAggSup, &prevSTW, groupId, pPrevPoint);
qDebug("===stream=== set stream twa prev point buf.ts:%" PRId64 ", groupId:%" PRIu64 ", res:%d", pPrevPoint->winKey.win.skey,
pPrevPoint->winKey.groupId, prevWinCode);
} else {
SET_WIN_KEY_INVALID(pPrevPoint->winKey.win.skey);
SET_WIN_KEY_INVALID(pPrevPoint->winKey.win.ekey);
}
}
_end:
@ -265,13 +279,15 @@ static int32_t doStreamIntervalSliceAggImpl(SOperatorInfo* pOperator, SSDataBloc
STimeWindow curWin =
getActiveTimeWindow(NULL, pResultRowInfo, curTs, &pInfo->interval, TSDB_ORDER_ASC);
while (1) {
if (curTs > pInfo->endTs) {
break;
}
int32_t winCode = TSDB_CODE_SUCCESS;
if (curTs <= pInfo->endTs) {
code = getIntervalSliceCurStateBuf(&pInfo->streamAggSup, &pInfo->interval, pInfo->hasInterpoFunc, &curWin, groupId, &curPoint, &prevPoint, &winCode);
QUERY_CHECK_CODE(code, lino, _end);
} else if (pInfo->hasInterpoFunc) {
SWinKey curKey = {.ts = curWin.skey, .groupId = groupId};
code = getIntervalSlicePrevStateBuf(&pInfo->streamAggSup, &pInfo->interval, &curKey, &prevPoint);
QUERY_CHECK_CODE(code, lino, _end);
}
if (pInfo->hasInterpoFunc && IS_VALID_WIN_KEY(prevPoint.winKey.win.skey) && isInterpoWindowFinished(&prevPoint) == false) {
code = setIntervalSliceOutputBuf(&prevPoint, pSup->pCtx, numOfOutput, pSup->rowEntryInfoOffset);
@ -288,6 +304,12 @@ static int32_t doStreamIntervalSliceAggImpl(SOperatorInfo* pOperator, SSDataBloc
code = saveWinResult(&prevKey, prevPoint.pResPos, pInfo->pUpdatedMap);
QUERY_CHECK_CODE(code, lino, _end);
setInterpoWindowFinished(&prevPoint);
} else if (IS_VALID_WIN_KEY(prevPoint.winKey.win.skey)) {
releaseOutputBuf(pInfo->streamAggSup.pState, prevPoint.pResPos, &pInfo->streamAggSup.stateStore);
}
if (curTs > pInfo->endTs) {
break;
}
code = setIntervalSliceOutputBuf(&curPoint, pSup->pCtx, numOfOutput, pSup->rowEntryInfoOffset);
@ -300,7 +322,7 @@ static int32_t doStreamIntervalSliceAggImpl(SOperatorInfo* pOperator, SSDataBloc
forwardRows = getNumOfRowsInTimeWindow(&pBlock->info, tsCols, startPos, curWin.ekey, binarySearchForKey, NULL,
TSDB_ORDER_ASC);
int32_t prevEndPos = (forwardRows - 1) + startPos;
if (pInfo->hasInterpoFunc && winCode != TSDB_CODE_SUCCESS) {
if (pInfo->hasInterpoFunc) {
int32_t endRowId = getQualifiedRowNumDesc(pSup, pBlock, tsCols, prevEndPos, false);
TSKEY endRowTs = tsCols[endRowId];
transBlockToSliceResultRow(pBlock, endRowId, endRowTs, curPoint.pLastRow, 0, NULL, NULL, pInfo->pOffsetInfo);

2
source/libs/function/src/builtins.c
View File

@ -4943,7 +4943,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = {
.paramInfoPattern = 0,
.outputParaInfo = {.validDataType = FUNC_PARAM_SUPPORT_ALL_TYPE}},
.translateFunc = translateSelectValue,
.getEnvFunc = getSelectivityFuncEnv,
.getEnvFunc = getGroupKeyFuncEnv,
.initFunc = functionSetup,
.processFunc = groupConstValueFunction,
.finalizeFunc = groupConstValueFinalize,

2
source/libs/nodes/src/nodesCloneFuncs.c
View File

@ -28,7 +28,7 @@
#define COPY_CHAR_ARRAY_FIELD(fldname) \
do { \
strcpy((pDst)->fldname, (pSrc)->fldname); \
tstrncpy((pDst)->fldname, (pSrc)->fldname, sizeof((pDst)->fldname)); \
} while (0)
#define COPY_OBJECT_FIELD(fldname, size) \

17
source/libs/nodes/src/nodesUtilFuncs.c
View File

@ -2341,7 +2341,7 @@ char* nodesGetStrValueFromNode(SValueNode* pNode) {
return NULL;
}
sprintf(buf, "%s", pNode->datum.b ? "true" : "false");
snprintf(buf, MAX_NUM_STR_SIZE, "%s", pNode->datum.b ? "true" : "false");
return buf;
}
case TSDB_DATA_TYPE_TINYINT:
@ -2354,7 +2354,7 @@ char* nodesGetStrValueFromNode(SValueNode* pNode) {
return NULL;
}
sprintf(buf, "%" PRId64, pNode->datum.i);
snprintf(buf, MAX_NUM_STR_SIZE, "%" PRId64, pNode->datum.i);
return buf;
}
case TSDB_DATA_TYPE_UTINYINT:
@ -2366,7 +2366,7 @@ char* nodesGetStrValueFromNode(SValueNode* pNode) {
return NULL;
}
sprintf(buf, "%" PRIu64, pNode->datum.u);
snprintf(buf, MAX_NUM_STR_SIZE, "%" PRIu64, pNode->datum.u);
return buf;
}
case TSDB_DATA_TYPE_FLOAT:
@ -2376,7 +2376,7 @@ char* nodesGetStrValueFromNode(SValueNode* pNode) {
return NULL;
}
sprintf(buf, "%e", pNode->datum.d);
snprintf(buf, MAX_NUM_STR_SIZE, "%e", pNode->datum.d);
return buf;
}
case TSDB_DATA_TYPE_NCHAR:
@ -2532,7 +2532,7 @@ static EDealRes doCollect(SCollectColumnsCxt* pCxt, SColumnNode* pCol, SNode* pN
}
if (pCol->projRefIdx > 0) {
len = taosHashBinary(name, strlen(name));
len += sprintf(name + len, "_%d", pCol->projRefIdx);
len += tsnprintf(name + len, TSDB_TABLE_NAME_LEN + TSDB_COL_NAME_LEN - len, "_%d", pCol->projRefIdx);
}
SNode** pNodeFound = taosHashGet(pCxt->pColHash, name, len);
if (pNodeFound == NULL) {
@ -2952,17 +2952,16 @@ int32_t nodesValueNodeToVariant(const SValueNode* pNode, SVariant* pVal) {
case TSDB_DATA_TYPE_VARCHAR:
case TSDB_DATA_TYPE_VARBINARY:
case TSDB_DATA_TYPE_GEOMETRY:
pVal->pz = taosMemoryMalloc(pVal->nLen + 1);
pVal->pz = taosMemoryCalloc(1, pVal->nLen + 1);
if (pVal->pz) {
memcpy(pVal->pz, pNode->datum.p, pVal->nLen);
pVal->pz[pVal->nLen] = 0;
memcpy(pVal->pz, pNode->datum.p, varDataTLen(pNode->datum.p));
} else {
code = terrno;
}
break;
case TSDB_DATA_TYPE_JSON:
pVal->nLen = getJsonValueLen(pNode->datum.p);
pVal->pz = taosMemoryMalloc(pVal->nLen);
pVal->pz = taosMemoryCalloc(1, pVal->nLen);
if (pVal->pz) {
memcpy(pVal->pz, pNode->datum.p, pVal->nLen);
} else {

112
source/libs/parser/src/parAstCreater.c
View File

@ -142,13 +142,13 @@ static int32_t parseEndpoint(SAstCreateContext* pCxt, const SToken* pEp, char* p
(void)strdequote(ep);
(void)strtrim(ep);
if (NULL == pPort) {
strcpy(pFqdn, ep);
tstrncpy(pFqdn, ep, TSDB_FQDN_LEN);
return TSDB_CODE_SUCCESS;
}
char* pColon = strchr(ep, ':');
if (NULL == pColon) {
*pPort = tsServerPort;
strcpy(pFqdn, ep);
tstrncpy(pFqdn, ep, TSDB_FQDN_LEN);
return TSDB_CODE_SUCCESS;
}
strncpy(pFqdn, ep, pColon - ep);
@ -320,12 +320,12 @@ SNode* releaseRawExprNode(SAstCreateContext* pCxt, SNode* pNode) {
if (nodesIsExprNode(pRealizedExpr)) {
SExprNode* pExpr = (SExprNode*)pRealizedExpr;
if (QUERY_NODE_COLUMN == nodeType(pExpr)) {
strcpy(pExpr->aliasName, ((SColumnNode*)pExpr)->colName);
strcpy(pExpr->userAlias, ((SColumnNode*)pExpr)->colName);
tstrncpy(pExpr->aliasName, ((SColumnNode*)pExpr)->colName, TSDB_COL_NAME_LEN);
tstrncpy(pExpr->userAlias, ((SColumnNode*)pExpr)->colName, TSDB_COL_NAME_LEN);
} else if (pRawExpr->isPseudoColumn) {
// all pseudo column are translate to function with same name
strcpy(pExpr->userAlias, ((SFunctionNode*)pExpr)->functionName);
strcpy(pExpr->aliasName, ((SFunctionNode*)pExpr)->functionName);
tstrncpy(pExpr->userAlias, ((SFunctionNode*)pExpr)->functionName, TSDB_COL_NAME_LEN);
tstrncpy(pExpr->aliasName, ((SFunctionNode*)pExpr)->functionName, TSDB_COL_NAME_LEN);
} else {
int32_t len = TMIN(sizeof(pExpr->aliasName) - 1, pRawExpr->n);
@ -333,9 +333,9 @@ SNode* releaseRawExprNode(SAstCreateContext* pCxt, SNode* pNode) {
// Len of pRawExpr->p could be larger than len of aliasName[TSDB_COL_NAME_LEN].
// If aliasName is truncated, hash value of aliasName could be the same.
uint64_t hashVal = MurmurHash3_64(pRawExpr->p, pRawExpr->n);
sprintf(pExpr->aliasName, "%" PRIu64, hashVal);
snprintf(pExpr->aliasName, TSDB_COL_NAME_LEN, "%" PRIu64, hashVal);
strncpy(pExpr->userAlias, pRawExpr->p, len);
pExpr->userAlias[len] = '\0';
pExpr->userAlias[len] = 0;
}
}
pRawExpr->pNode = NULL;
@ -946,11 +946,11 @@ SNode* createOperatorNode(SAstCreateContext* pCxt, EOperatorType type, SNode* pL
goto _err;
}
if ('+' == pVal->literal[0]) {
sprintf(pNewLiteral, "-%s", pVal->literal + 1);
snprintf(pNewLiteral, strlen(pVal->literal) + 2, "-%s", pVal->literal + 1);
} else if ('-' == pVal->literal[0]) {
sprintf(pNewLiteral, "%s", pVal->literal + 1);
snprintf(pNewLiteral, strlen(pVal->literal) + 2, "%s", pVal->literal + 1);
} else {
sprintf(pNewLiteral, "-%s", pVal->literal);
snprintf(pNewLiteral, strlen(pVal->literal) + 2, "-%s", pVal->literal);
}
taosMemoryFree(pVal->literal);
pVal->literal = pNewLiteral;
@ -1021,7 +1021,7 @@ static SNode* createPrimaryKeyCol(SAstCreateContext* pCxt, const SToken* pFuncNa
CHECK_MAKE_NODE(pCol);
pCol->colId = PRIMARYKEY_TIMESTAMP_COL_ID;
if (NULL == pFuncName) {
strcpy(pCol->colName, ROWTS_PSEUDO_COLUMN_NAME);
tstrncpy(pCol->colName, ROWTS_PSEUDO_COLUMN_NAME, TSDB_COL_NAME_LEN);
} else {
strncpy(pCol->colName, pFuncName->z, pFuncName->n);
}
@ -1052,7 +1052,7 @@ SNode* createCastFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SDataType d
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "cast");
tstrncpy(func->functionName, "cast", TSDB_FUNC_NAME_LEN);
func->node.resType = dt;
if (TSDB_DATA_TYPE_VARCHAR == dt.type || TSDB_DATA_TYPE_GEOMETRY == dt.type || TSDB_DATA_TYPE_VARBINARY == dt.type) {
func->node.resType.bytes = func->node.resType.bytes + VARSTR_HEADER_SIZE;
@ -1073,7 +1073,7 @@ SNode* createPositionFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SNode*
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "position");
tstrncpy(func->functionName, "position", TSDB_FUNC_NAME_LEN);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
@ -1091,7 +1091,7 @@ SNode* createTrimFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, ETrimType t
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "trim");
tstrncpy(func->functionName, "trim", TSDB_FUNC_NAME_LEN);
func->trimType = type;
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
@ -1107,7 +1107,7 @@ SNode* createTrimFunctionNodeExt(SAstCreateContext* pCxt, SNode* pExpr, SNode* p
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "trim");
tstrncpy(func->functionName, "trim", TSDB_FUNC_NAME_LEN);
func->trimType = type;
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
@ -1126,7 +1126,7 @@ SNode* createSubstrFunctionNode(SAstCreateContext* pCxt, SNode* pExpr, SNode* pE
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "substr");
tstrncpy(func->functionName, "substr", TSDB_FUNC_NAME_LEN);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
@ -1144,7 +1144,7 @@ SNode* createSubstrFunctionNodeExt(SAstCreateContext* pCxt, SNode* pExpr, SNode*
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&func);
CHECK_MAKE_NODE(func);
strcpy(func->functionName, "substr");
tstrncpy(func->functionName, "substr", TSDB_FUNC_NAME_LEN);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr);
CHECK_PARSER_STATUS(pCxt);
pCxt->errCode = nodesListMakeAppend(&func->pParameterList, pExpr2);
@ -1228,10 +1228,10 @@ SNode* createTempTableNode(SAstCreateContext* pCxt, SNode* pSubquery, SToken* pT
taosRandStr(tempTable->table.tableAlias, 8);
}
if (QUERY_NODE_SELECT_STMT == nodeType(pSubquery)) {
strcpy(((SSelectStmt*)pSubquery)->stmtName, tempTable->table.tableAlias);
tstrncpy(((SSelectStmt*)pSubquery)->stmtName, tempTable->table.tableAlias, TSDB_TABLE_NAME_LEN);
((SSelectStmt*)pSubquery)->isSubquery = true;
} else if (QUERY_NODE_SET_OPERATOR == nodeType(pSubquery)) {
strcpy(((SSetOperator*)pSubquery)->stmtName, tempTable->table.tableAlias);
tstrncpy(((SSetOperator*)pSubquery)->stmtName, tempTable->table.tableAlias, TSDB_TABLE_NAME_LEN);
}
return (SNode*)tempTable;
_err:
@ -1434,7 +1434,7 @@ SNode* createFillNode(SAstCreateContext* pCxt, EFillMode mode, SNode* pValues) {
fill->pWStartTs = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&(fill->pWStartTs));
CHECK_MAKE_NODE(fill->pWStartTs);
strcpy(((SFunctionNode*)fill->pWStartTs)->functionName, "_wstart");
tstrncpy(((SFunctionNode*)fill->pWStartTs)->functionName, "_wstart", TSDB_FUNC_NAME_LEN);
return (SNode*)fill;
_err:
nodesDestroyNode((SNode*)fill);
@ -1764,7 +1764,7 @@ SNode* createSetOperator(SAstCreateContext* pCxt, ESetOperatorType type, SNode*
setSubquery(setOp->pLeft);
setOp->pRight = pRight;
setSubquery(setOp->pRight);
sprintf(setOp->stmtName, "%p", setOp);
snprintf(setOp->stmtName, TSDB_TABLE_NAME_LEN, "%p", setOp);
return (SNode*)setOp;
_err:
nodesDestroyNode(pLeft);
@ -2322,8 +2322,8 @@ SNode* createCreateTableStmt(SAstCreateContext* pCxt, bool ignoreExists, SNode*
SCreateTableStmt* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_CREATE_TABLE_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
pStmt->ignoreExists = ignoreExists;
pStmt->pCols = pCols;
pStmt->pTags = pTags;
@ -2344,10 +2344,10 @@ SNode* createCreateSubTableClause(SAstCreateContext* pCxt, bool ignoreExists, SN
SCreateSubTableClause* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_CREATE_SUBTABLE_CLAUSE, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
strcpy(pStmt->useDbName, ((SRealTableNode*)pUseRealTable)->table.dbName);
strcpy(pStmt->useTableName, ((SRealTableNode*)pUseRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pStmt->useDbName, ((SRealTableNode*)pUseRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->useTableName, ((SRealTableNode*)pUseRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
pStmt->ignoreExists = ignoreExists;
pStmt->pSpecificTags = pSpecificTags;
pStmt->pValsOfTags = pValsOfTags;
@ -2370,8 +2370,8 @@ SNode* createCreateSubTableFromFileClause(SAstCreateContext* pCxt, bool ignoreEx
SCreateSubTableFromFileClause* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_CREATE_SUBTABLE_FROM_FILE_CLAUSE, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->useDbName, ((SRealTableNode*)pUseRealTable)->table.dbName);
strcpy(pStmt->useTableName, ((SRealTableNode*)pUseRealTable)->table.tableName);
tstrncpy(pStmt->useDbName, ((SRealTableNode*)pUseRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->useTableName, ((SRealTableNode*)pUseRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
pStmt->ignoreExists = ignoreExists;
pStmt->pSpecificTags = pSpecificTags;
if (TK_NK_STRING == pFilePath->type) {
@ -2405,8 +2405,8 @@ SNode* createDropTableClause(SAstCreateContext* pCxt, bool ignoreNotExists, SNod
SDropTableClause* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_DROP_TABLE_CLAUSE, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
pStmt->ignoreNotExists = ignoreNotExists;
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
@ -2433,8 +2433,8 @@ SNode* createDropSuperTableStmt(SAstCreateContext* pCxt, bool withOpt, bool igno
SDropSuperTableStmt* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_DROP_SUPER_TABLE_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
pStmt->ignoreNotExists = ignoreNotExists;
pStmt->withOpt = withOpt;
nodesDestroyNode(pRealTable);
@ -2445,8 +2445,8 @@ _err:
}
static SNode* createAlterTableStmtFinalize(SNode* pRealTable, SAlterTableStmt* pStmt) {
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
}
@ -2765,8 +2765,8 @@ SNode* createShowCreateTableStmt(SAstCreateContext* pCxt, ENodeType type, SNode*
SShowCreateTableStmt* pStmt = NULL;
pCxt->errCode = nodesMakeNode(type, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
_err:
@ -2779,8 +2779,8 @@ SNode* createShowCreateViewStmt(SAstCreateContext* pCxt, ENodeType type, SNode*
SShowCreateViewStmt* pStmt = NULL;
pCxt->errCode = nodesMakeNode(type, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->viewName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->viewName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
_err:
@ -2793,8 +2793,8 @@ SNode* createShowTableDistributedStmt(SAstCreateContext* pCxt, SNode* pRealTable
SShowTableDistributedStmt* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_SHOW_TABLE_DISTRIBUTED_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
_err:
@ -2950,7 +2950,7 @@ SNode* createCreateUserStmt(SAstCreateContext* pCxt, SToken* pUserName, const ST
pCxt->errCode = nodesMakeNode(QUERY_NODE_CREATE_USER_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
COPY_STRING_FORM_ID_TOKEN(pStmt->userName, pUserName);
strcpy(pStmt->password, password);
tstrncpy(pStmt->password, password, TSDB_USET_PASSWORD_LEN);
pStmt->sysinfo = sysinfo;
pStmt->createDb = createDb;
pStmt->isImport = is_import;
@ -2972,7 +2972,7 @@ SNode* createAlterUserStmt(SAstCreateContext* pCxt, SToken* pUserName, int8_t al
char password[TSDB_USET_PASSWORD_LEN] = {0};
SToken* pVal = pAlterInfo;
CHECK_NAME(checkPassword(pCxt, pVal, password));
strcpy(pStmt->password, password);
tstrncpy(pStmt->password, password, TSDB_USET_PASSWORD_LEN);
break;
}
case TSDB_ALTER_USER_ENABLE: {
@ -3282,8 +3282,8 @@ SNode* createCreateTopicStmtUseTable(SAstCreateContext* pCxt, bool ignoreExists,
pStmt->withMeta = withMeta;
pStmt->pWhere = pWhere;
strcpy(pStmt->subDbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->subSTbName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->subDbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->subSTbName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
_err:
@ -3396,8 +3396,8 @@ SNode* createDescribeStmt(SAstCreateContext* pCxt, SNode* pRealTable) {
SDescribeStmt* pStmt = NULL;
pCxt->errCode = nodesMakeNode(QUERY_NODE_DESCRIBE_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
strcpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->dbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->tableName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
return (SNode*)pStmt;
_err:
@ -3477,8 +3477,8 @@ SNode* createCreateViewStmt(SAstCreateContext* pCxt, bool orReplace, SNode* pVie
}
pStmt->pQuerySql = tstrdup(pAs->z + i);
CHECK_OUT_OF_MEM(pStmt->pQuerySql);
strcpy(pStmt->dbName, ((SViewNode*)pView)->table.dbName);
strcpy(pStmt->viewName, ((SViewNode*)pView)->table.tableName);
tstrncpy(pStmt->dbName, ((SViewNode*)pView)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->viewName, ((SViewNode*)pView)->table.tableName, TSDB_VIEW_NAME_LEN);
nodesDestroyNode(pView);
pStmt->orReplace = orReplace;
pStmt->pQuery = pQuery;
@ -3496,8 +3496,8 @@ SNode* createDropViewStmt(SAstCreateContext* pCxt, bool ignoreNotExists, SNode*
pCxt->errCode = nodesMakeNode(QUERY_NODE_DROP_VIEW_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
pStmt->ignoreNotExists = ignoreNotExists;
strcpy(pStmt->dbName, ((SViewNode*)pView)->table.dbName);
strcpy(pStmt->viewName, ((SViewNode*)pView)->table.tableName);
tstrncpy(pStmt->dbName, ((SViewNode*)pView)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->viewName, ((SViewNode*)pView)->table.tableName, TSDB_VIEW_NAME_LEN);
nodesDestroyNode(pView);
return (SNode*)pStmt;
_err:
@ -3582,8 +3582,8 @@ SNode* createCreateStreamStmt(SAstCreateContext* pCxt, bool ignoreExists, SToken
pCxt->errCode = nodesMakeNode(QUERY_NODE_CREATE_STREAM_STMT, (SNode**)&pStmt);
CHECK_MAKE_NODE(pStmt);
COPY_STRING_FORM_ID_TOKEN(pStmt->streamName, pStreamName);
strcpy(pStmt->targetDbName, ((SRealTableNode*)pRealTable)->table.dbName);
strcpy(pStmt->targetTabName, ((SRealTableNode*)pRealTable)->table.tableName);
tstrncpy(pStmt->targetDbName, ((SRealTableNode*)pRealTable)->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pStmt->targetTabName, ((SRealTableNode*)pRealTable)->table.tableName, TSDB_TABLE_NAME_LEN);
nodesDestroyNode(pRealTable);
pStmt->ignoreExists = ignoreExists;
pStmt->pOptions = (SStreamOptions*)pOptions;
@ -3837,9 +3837,9 @@ SNode* createInsertStmt(SAstCreateContext* pCxt, SNode* pTable, SNodeList* pCols
pStmt->pCols = pCols;
pStmt->pQuery = pQuery;
if (QUERY_NODE_SELECT_STMT == nodeType(pQuery)) {
strcpy(((SSelectStmt*)pQuery)->stmtName, ((STableNode*)pTable)->tableAlias);
tstrncpy(((SSelectStmt*)pQuery)->stmtName, ((STableNode*)pTable)->tableAlias, TSDB_TABLE_NAME_LEN);
} else if (QUERY_NODE_SET_OPERATOR == nodeType(pQuery)) {
strcpy(((SSetOperator*)pQuery)->stmtName, ((STableNode*)pTable)->tableAlias);
tstrncpy(((SSetOperator*)pQuery)->stmtName, ((STableNode*)pTable)->tableAlias, TSDB_TABLE_NAME_LEN);
}
return (SNode*)pStmt;
_err:

16
source/libs/parser/src/parAstParser.c
View File

@ -480,8 +480,8 @@ static int32_t collectMetaKeyFromExplain(SCollectMetaKeyCxt* pCxt, SExplainStmt*
static int32_t collectMetaKeyFromDescribe(SCollectMetaKeyCxt* pCxt, SDescribeStmt* pStmt) {
SName name = {.type = TSDB_TABLE_NAME_T, .acctId = pCxt->pParseCxt->acctId};
strcpy(name.dbname, pStmt->dbName);
strcpy(name.tname, pStmt->tableName);
tstrncpy(name.dbname, pStmt->dbName, TSDB_DB_NAME_LEN);
tstrncpy(name.tname, pStmt->tableName, TSDB_TABLE_NAME_LEN);
int32_t code = catalogRemoveTableMeta(pCxt->pParseCxt->pCatalog, &name);
#ifdef TD_ENTERPRISE
if (TSDB_CODE_SUCCESS == code) {
@ -781,8 +781,8 @@ static int32_t collectMetaKeyFromShowCreateDatabase(SCollectMetaKeyCxt* pCxt, SS
static int32_t collectMetaKeyFromShowCreateTable(SCollectMetaKeyCxt* pCxt, SShowCreateTableStmt* pStmt) {
SName name = {.type = TSDB_TABLE_NAME_T, .acctId = pCxt->pParseCxt->acctId};
strcpy(name.dbname, pStmt->dbName);
strcpy(name.tname, pStmt->tableName);
tstrncpy(name.dbname, pStmt->dbName, TSDB_DB_NAME_LEN);
tstrncpy(name.tname, pStmt->tableName, TSDB_TABLE_NAME_LEN);
int32_t code = catalogRemoveTableMeta(pCxt->pParseCxt->pCatalog, &name);
if (TSDB_CODE_SUCCESS == code) {
code = reserveTableCfgInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->tableName, pCxt->pMetaCache);
@ -799,8 +799,8 @@ static int32_t collectMetaKeyFromShowCreateTable(SCollectMetaKeyCxt* pCxt, SShow
static int32_t collectMetaKeyFromShowCreateView(SCollectMetaKeyCxt* pCxt, SShowCreateViewStmt* pStmt) {
SName name = {.type = TSDB_TABLE_NAME_T, .acctId = pCxt->pParseCxt->acctId};
strcpy(name.dbname, pStmt->dbName);
strcpy(name.tname, pStmt->viewName);
tstrncpy(name.dbname, pStmt->dbName, TSDB_DB_NAME_LEN);
tstrncpy(name.tname, pStmt->viewName, TSDB_TABLE_NAME_LEN);
char dbFName[TSDB_DB_FNAME_LEN];
(void)tNameGetFullDbName(&name, dbFName);
int32_t code = catalogRemoveViewMeta(pCxt->pParseCxt->pCatalog, dbFName, 0, pStmt->viewName, 0);
@ -841,8 +841,8 @@ static int32_t collectMetaKeyFromInsert(SCollectMetaKeyCxt* pCxt, SInsertStmt* p
static int32_t collectMetaKeyFromShowBlockDist(SCollectMetaKeyCxt* pCxt, SShowTableDistributedStmt* pStmt) {
SName name = {.type = TSDB_TABLE_NAME_T, .acctId = pCxt->pParseCxt->acctId};
strcpy(name.dbname, pStmt->dbName);
strcpy(name.tname, pStmt->tableName);
tstrncpy(name.dbname, pStmt->dbName, TSDB_DB_NAME_LEN);
tstrncpy(name.tname, pStmt->tableName, TSDB_TABLE_NAME_LEN);
int32_t code = catalogRemoveTableMeta(pCxt->pParseCxt->pCatalog, &name);
if (TSDB_CODE_SUCCESS == code) {
code = collectMetaKeyFromRealTableImpl(pCxt, pStmt->dbName, pStmt->tableName, AUTH_TYPE_READ);

4
source/libs/parser/src/parAuthenticator.c
View File

@ -134,8 +134,8 @@ EDealRes rewriteAuthTable(SNode* pNode, void* pContext) {
if (QUERY_NODE_COLUMN == nodeType(pNode)) {
SColumnNode* pCol = (SColumnNode*)pNode;
SAuthRewriteCxt* pCxt = (SAuthRewriteCxt*)pContext;
strcpy(pCol->tableName, pCxt->pTarget->tableName);
strcpy(pCol->tableAlias, pCxt->pTarget->tableAlias);
tstrncpy(pCol->tableName, pCxt->pTarget->tableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableAlias, pCxt->pTarget->tableAlias, TSDB_TABLE_NAME_LEN);
}
return DEAL_RES_CONTINUE;

8
source/libs/parser/src/parCalcConst.c
View File

@ -178,14 +178,14 @@ static EDealRes doFindAndReplaceNode(SNode** pNode, void* pContext) {
SCalcConstContext* pCxt = pContext;
if (pCxt->replaceCxt.pTarget == *pNode) {
char aliasName[TSDB_COL_NAME_LEN] = {0};
strcpy(aliasName, ((SExprNode*)*pNode)->aliasName);
tstrncpy(aliasName, ((SExprNode*)*pNode)->aliasName, TSDB_COL_NAME_LEN);
nodesDestroyNode(*pNode);
*pNode = NULL;
pCxt->code = nodesCloneNode(pCxt->replaceCxt.pNew, pNode);
if (NULL == *pNode) {
return DEAL_RES_ERROR;
}
strcpy(((SExprNode*)*pNode)->aliasName, aliasName);
tstrncpy(((SExprNode*)*pNode)->aliasName, aliasName, TSDB_COL_NAME_LEN);
pCxt->replaceCxt.replaced = true;
return DEAL_RES_END;
@ -228,14 +228,14 @@ static int32_t calcConstProject(SCalcConstContext* pCxt, SNode* pProject, bool d
SAssociationNode* pAssNode = taosArrayGet(pAssociation, i);
SNode** pCol = pAssNode->pPlace;
if (*pCol == pAssNode->pAssociationNode) {
strcpy(aliasName, ((SExprNode*)*pCol)->aliasName);
tstrncpy(aliasName, ((SExprNode*)*pCol)->aliasName, TSDB_COL_NAME_LEN);
SArray* pOrigAss = NULL;
TSWAP(((SExprNode*)*pCol)->pAssociation, pOrigAss);
nodesDestroyNode(*pCol);
*pCol = NULL;
code = nodesCloneNode(*pNew, pCol);
if (TSDB_CODE_SUCCESS == code) {
strcpy(((SExprNode*)*pCol)->aliasName, aliasName);
tstrncpy(((SExprNode*)*pCol)->aliasName, aliasName, TSDB_COL_NAME_LEN);
TSWAP(pOrigAss, ((SExprNode*)*pCol)->pAssociation);
}
taosArrayDestroy(pOrigAss);

2
source/libs/parser/src/parInsertStmt.c
View File

@ -927,7 +927,7 @@ int32_t buildBoundFields(int32_t numOfBound, int16_t* boundColumns, SSchema* pSc
for (int32_t i = 0; i < numOfBound; ++i) {
schema = &pSchema[boundColumns[i]];
strcpy((*fields)[i].name, schema->name);
tstrncpy((*fields)[i].name, schema->name, 65);
(*fields)[i].type = schema->type;
(*fields)[i].bytes = schema->bytes;
}

364
source/libs/parser/src/parTranslater.c
View File

File diff suppressed because it is too large Load Diff

18
source/libs/parser/src/parUtil.c
View File

@ -250,7 +250,7 @@ int32_t generateSyntaxErrMsgExt(SMsgBuf* pBuf, int32_t errCode, const char* pFor
int32_t buildInvalidOperationMsg(SMsgBuf* pBuf, const char* msg) {
if (pBuf->buf) {
strncpy(pBuf->buf, msg, pBuf->len);
tstrncpy(pBuf->buf, msg, pBuf->len);
}
return TSDB_CODE_TSC_INVALID_OPERATION;
@ -277,7 +277,7 @@ int32_t buildSyntaxErrMsg(SMsgBuf* pBuf, const char* additionalInfo, const char*
}
char buf[64] = {0}; // only extract part of sql string
strncpy(buf, sourceStr, tListLen(buf) - 1);
tstrncpy(buf, sourceStr, tListLen(buf));
if (additionalInfo != NULL) {
snprintf(pBuf->buf, pBuf->len, msgFormat2, buf, additionalInfo);
@ -454,7 +454,7 @@ int32_t parseJsontoTagData(const char* json, SArray* pTagVals, STag** ppTag, voi
continue;
}
STagVal val = {0};
// strcpy(val.colName, colName);
// TSDB_DB_FNAME_LENme, colName);
val.pKey = jsonKey;
retCode = taosHashPut(keyHash, jsonKey, keyLen, &keyLen,
CHAR_BYTES); // add key to hash to remove dumplicate, value is useless
@ -595,15 +595,15 @@ int32_t getVnodeSysTableTargetName(int32_t acctId, SNode* pWhere, SName* pName)
static int32_t userAuthToString(int32_t acctId, const char* pUser, const char* pDb, const char* pTable, AUTH_TYPE type,
char* pStr, bool isView) {
return sprintf(pStr, "%s*%d*%s*%s*%d*%d", pUser, acctId, pDb, (NULL == pTable || '\0' == pTable[0]) ? "``" : pTable,
type, isView);
return snprintf(pStr, USER_AUTH_KEY_MAX_LEN, "%s*%d*%s*%s*%d*%d", pUser, acctId, pDb,
(NULL == pTable || '\0' == pTable[0]) ? "``" : pTable, type, isView);
}
static int32_t getIntegerFromAuthStr(const char* pStart, char** pNext) {
char* p = strchr(pStart, '*');
char buf[10] = {0};
if (NULL == p) {
strcpy(buf, pStart);
tstrncpy(buf, pStart, 10);
*pNext = NULL;
} else {
strncpy(buf, pStart, p - pStart);
@ -615,7 +615,7 @@ static int32_t getIntegerFromAuthStr(const char* pStart, char** pNext) {
static void getStringFromAuthStr(const char* pStart, char* pStr, char** pNext) {
char* p = strchr(pStart, '*');
if (NULL == p) {
strcpy(pStr, pStart);
tstrncpy(pStr, pStart, strlen(pStart) + 1);
*pNext = NULL;
} else {
strncpy(pStr, pStart, p - pStart);
@ -707,7 +707,7 @@ static int32_t buildTableReqFromDb(SHashObj* pDbsHash, SArray** pDbs) {
SParseTablesMetaReq* p = taosHashIterate(pDbsHash, NULL);
while (NULL != p) {
STablesReq req = {0};
strcpy(req.dbFName, p->dbFName);
tstrncpy(req.dbFName, p->dbFName, TSDB_DB_FNAME_LEN);
int32_t code = buildTableReq(p->pTables, &req.pTables);
if (TSDB_CODE_SUCCESS == code) {
if (NULL == taosArrayPush(*pDbs, &req)) {
@ -831,7 +831,7 @@ int32_t createSelectStmtImpl(bool isDistinct, SNodeList* pProjectionList, SNode*
select->isDistinct = isDistinct;
select->pProjectionList = pProjectionList;
select->pFromTable = pTable;
sprintf(select->stmtName, "%p", select);
snprintf(select->stmtName, TSDB_TABLE_NAME_LEN, "%p", select);
select->timeLineResMode = select->isDistinct ? TIME_LINE_NONE : TIME_LINE_GLOBAL;
select->timeLineCurMode = TIME_LINE_GLOBAL;
select->onlyHasKeepOrderFunc = true;

5
source/libs/parser/src/parser.c
View File

@ -221,7 +221,7 @@ static int32_t setValueByBindParam(SValueNode* pVal, TAOS_MULTI_BIND* pParam) {
static EDealRes rewriteQueryExprAliasImpl(SNode* pNode, void* pContext) {
if (nodesIsExprNode(pNode) && QUERY_NODE_COLUMN != nodeType(pNode)) {
sprintf(((SExprNode*)pNode)->aliasName, "#%d", *(int32_t*)pContext);
snprintf(((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN, "#%d", *(int32_t*)pContext);
++(*(int32_t*)pContext);
}
return DEAL_RES_CONTINUE;
@ -528,9 +528,6 @@ int32_t qStmtBindParams2(SQuery* pQuery, TAOS_STMT2_BIND* pParams, int32_t colId
nodesDestroyNode(pQuery->pRoot);
pQuery->pRoot = NULL;
code = nodesCloneNode(pQuery->pPrepareRoot, &pQuery->pRoot);
if (NULL == pQuery->pRoot) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {
rewriteExprAlias(pQuery->pRoot);

39
source/libs/planner/src/planLogicCreater.c
View File

@ -119,9 +119,9 @@ static EDealRes doRewriteExpr(SNode** pNode, void* pContext) {
}
SExprNode* pToBeRewrittenExpr = (SExprNode*)(*pNode);
pCol->node.resType = pToBeRewrittenExpr->resType;
strcpy(pCol->node.aliasName, pToBeRewrittenExpr->aliasName);
strcpy(pCol->node.userAlias, ((SExprNode*)pExpr)->userAlias);
strcpy(pCol->colName, ((SExprNode*)pExpr)->aliasName);
tstrncpy(pCol->node.aliasName, pToBeRewrittenExpr->aliasName, TSDB_COL_NAME_LEN);
tstrncpy(pCol->node.userAlias, ((SExprNode*)pExpr)->userAlias, TSDB_COL_NAME_LEN);
tstrncpy(pCol->colName, ((SExprNode*)pExpr)->aliasName, TSDB_COL_NAME_LEN);
pCol->node.projIdx = ((SExprNode*)(*pNode))->projIdx;
if (QUERY_NODE_FUNCTION == nodeType(pExpr)) {
setColumnInfo((SFunctionNode*)pExpr, pCol, pCxt->isPartitionBy);
@ -150,7 +150,7 @@ static EDealRes doNameExpr(SNode* pNode, void* pContext) {
case QUERY_NODE_LOGIC_CONDITION:
case QUERY_NODE_FUNCTION: {
if ('\0' == ((SExprNode*)pNode)->aliasName[0]) {
sprintf(((SExprNode*)pNode)->aliasName, "#expr_%p", pNode);
snprintf(((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN, "#expr_%p", pNode);
}
return DEAL_RES_IGNORE_CHILD;
}
@ -311,12 +311,12 @@ static SNode* createFirstCol(SRealTableNode* pTable, const SSchema* pSchema) {
pCol->tableId = pTable->pMeta->uid;
pCol->colId = pSchema->colId;
pCol->colType = COLUMN_TYPE_COLUMN;
strcpy(pCol->tableAlias, pTable->table.tableAlias);
strcpy(pCol->tableName, pTable->table.tableName);
tstrncpy(pCol->tableAlias, pTable->table.tableAlias, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableName, pTable->table.tableName, TSDB_TABLE_NAME_LEN);
pCol->isPk = pSchema->flags & COL_IS_KEY;
pCol->tableHasPk = hasPkInTable(pTable->pMeta);
pCol->numOfPKs = pTable->pMeta->tableInfo.numOfPKs;
strcpy(pCol->colName, pSchema->name);
tstrncpy(pCol->colName, pSchema->name, TSDB_COL_NAME_LEN);
return (SNode*)pCol;
}
@ -392,8 +392,8 @@ static int32_t makeScanLogicNode(SLogicPlanContext* pCxt, SRealTableNode* pRealT
pScan->scanRange = TSWINDOW_INITIALIZER;
pScan->tableName.type = TSDB_TABLE_NAME_T;
pScan->tableName.acctId = pCxt->pPlanCxt->acctId;
strcpy(pScan->tableName.dbname, pRealTable->table.dbName);
strcpy(pScan->tableName.tname, pRealTable->table.tableName);
tstrncpy(pScan->tableName.dbname, pRealTable->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pScan->tableName.tname, pRealTable->table.tableName, TSDB_TABLE_NAME_LEN);
pScan->showRewrite = pCxt->pPlanCxt->showRewrite;
pScan->ratio = pRealTable->ratio;
pScan->dataRequired = FUNC_DATA_REQUIRED_DATA_LOAD;
@ -785,12 +785,12 @@ static int32_t addWinJoinPrimKeyToAggFuncs(SSelectStmt* pSelect, SNodeList** pLi
}
SSchema* pColSchema = &pProbeTable->pMeta->schema[0];
strcpy(pCol->dbName, pProbeTable->table.dbName);
strcpy(pCol->tableAlias, pProbeTable->table.tableAlias);
strcpy(pCol->tableName, pProbeTable->table.tableName);
strcpy(pCol->colName, pColSchema->name);
strcpy(pCol->node.aliasName, pColSchema->name);
strcpy(pCol->node.userAlias, pColSchema->name);
tstrncpy(pCol->dbName, pProbeTable->table.dbName, TSDB_DB_NAME_LEN);
tstrncpy(pCol->tableAlias, pProbeTable->table.tableAlias, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableName, pProbeTable->table.tableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->colName, pColSchema->name, TSDB_COL_NAME_LEN);
tstrncpy(pCol->node.aliasName, pColSchema->name, TSDB_COL_NAME_LEN);
tstrncpy(pCol->node.userAlias, pColSchema->name, TSDB_COL_NAME_LEN);
pCol->tableId = pProbeTable->pMeta->uid;
pCol->tableType = pProbeTable->pMeta->tableType;
pCol->colId = pColSchema->colId;
@ -1556,7 +1556,7 @@ static int32_t createSortLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSelect
if (TSDB_CODE_SUCCESS == code) {
pSort->pSortKeys = NULL;
code = nodesCloneList(pSelect->pOrderByList, &pSort->pSortKeys);
if (NULL != pSort->pSortKeys) {
if (TSDB_CODE_SUCCESS == code) {
SNode* pNode = NULL;
SOrderByExprNode* firstSortKey = (SOrderByExprNode*)nodesListGetNode(pSort->pSortKeys, 0);
if (isPrimaryKeySort(pSelect->pOrderByList)) pSort->node.outputTsOrder = firstSortKey->order;
@ -1623,10 +1623,7 @@ static int32_t createProjectLogicNode(SLogicPlanContext* pCxt, SSelectStmt* pSel
pProject->pProjections = NULL;
code = nodesCloneList(pSelect->pProjectionList, &pProject->pProjections);
if (NULL == pProject->pProjections) {
code = code;
}
strcpy(pProject->stmtName, pSelect->stmtName);
tstrncpy(pProject->stmtName, pSelect->stmtName, TSDB_TABLE_NAME_LEN);
if (TSDB_CODE_SUCCESS == code) {
code = createColumnByProjections(pCxt, pSelect->stmtName, pSelect->pProjectionList, &pProject->node.pTargets);
@ -2116,7 +2113,7 @@ static int32_t createVnodeModifLogicNodeByDelete(SLogicPlanContext* pCxt, SDelet
pModify->tableId = pRealTable->pMeta->uid;
pModify->tableType = pRealTable->pMeta->tableType;
snprintf(pModify->tableName, sizeof(pModify->tableName), "%s", pRealTable->table.tableName);
strcpy(pModify->tsColName, pRealTable->pMeta->schema->name);
tstrncpy(pModify->tsColName, pRealTable->pMeta->schema->name, TSDB_COL_NAME_LEN);
pModify->deleteTimeRange = pDelete->timeRange;
pModify->pAffectedRows = NULL;
code = nodesCloneNode(pDelete->pCountFunc, &pModify->pAffectedRows);

105
source/libs/planner/src/planOptimizer.c
View File

@ -188,7 +188,7 @@ static EDealRes optRebuildTbanme(SNode** pNode, void* pContext) {
*(int32_t*)pContext = code;
return DEAL_RES_ERROR;
}
strcpy(pFunc->functionName, "tbname");
tstrncpy(pFunc->functionName, "tbname", TSDB_FUNC_NAME_LEN);
pFunc->funcType = FUNCTION_TYPE_TBNAME;
pFunc->node.resType = ((SColumnNode*)*pNode)->node.resType;
nodesDestroyNode(*pNode);
@ -1302,9 +1302,7 @@ static int32_t pdcJoinAddPreFilterColsToTarget(SOptimizeContext* pCxt, SJoinLogi
SNodeList* pCondCols = NULL;
code = nodesMakeList(&pCondCols);
SNodeList* pTargets = NULL;
if (NULL == pCondCols) {
code = code;
} else {
if (TSDB_CODE_SUCCESS == code) {
code = nodesCollectColumnsFromNode(pJoin->pColOnCond, NULL, COLLECT_COL_TYPE_ALL, &pCondCols);
}
if (TSDB_CODE_SUCCESS == code) {
@ -2958,9 +2956,9 @@ static int32_t smaIndexOptCreateSmaCol(SNode* pFunc, uint64_t tableId, int32_t c
pCol->tableType = TSDB_SUPER_TABLE;
pCol->colId = colId;
pCol->colType = COLUMN_TYPE_COLUMN;
strcpy(pCol->colName, ((SExprNode*)pFunc)->aliasName);
tstrncpy(pCol->colName, ((SExprNode*)pFunc)->aliasName, TSDB_COL_NAME_LEN);
pCol->node.resType = ((SExprNode*)pFunc)->resType;
strcpy(pCol->node.aliasName, ((SExprNode*)pFunc)->aliasName);
tstrncpy(pCol->node.aliasName, ((SExprNode*)pFunc)->aliasName, TSDB_COL_NAME_LEN);
*ppNode = pCol;
return code;
}
@ -3029,7 +3027,7 @@ static int32_t smaIndexOptCreateSmaCols(SNodeList* pFuncs, uint64_t tableId, SNo
}
SExprNode exprNode;
exprNode.resType = ((SExprNode*)pWsNode)->resType;
sprintf(exprNode.aliasName, "#expr_%d", index + 1);
snprintf(exprNode.aliasName, TSDB_COL_NAME_LEN, "#expr_%d", index + 1);
SColumnNode* pkNode = NULL;
code = smaIndexOptCreateSmaCol((SNode*)&exprNode, tableId, PRIMARYKEY_TIMESTAMP_COL_ID, &pkNode);
if (TSDB_CODE_SUCCESS != code) {
@ -3199,7 +3197,7 @@ static void partTagsSetAlias(char* pAlias, const char* pTableAlias, const char*
int32_t len = tsnprintf(name, TSDB_COL_FNAME_LEN, "%s.%s", pTableAlias, pColName);
(void)taosHashBinary(name, len);
strncpy(pAlias, name, TSDB_COL_NAME_LEN - 1);
tstrncpy(pAlias, name, TSDB_COL_NAME_LEN);
}
static int32_t partTagsCreateWrapperFunc(const char* pFuncName, SNode* pNode, SFunctionNode** ppNode) {
@ -3217,7 +3215,7 @@ static int32_t partTagsCreateWrapperFunc(const char* pFuncName, SNode* pNode, SF
SColumnNode* pCol = (SColumnNode*)pNode;
partTagsSetAlias(pFunc->node.aliasName, pCol->tableAlias, pCol->colName);
} else {
strcpy(pFunc->node.aliasName, ((SExprNode*)pNode)->aliasName);
tstrncpy(pFunc->node.aliasName, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN);
}
code = nodesCloneNode(pNode, &pNew);
if (TSDB_CODE_SUCCESS == code) {
@ -3506,7 +3504,7 @@ static EDealRes eliminateProjOptRewriteScanTableAlias(SNode* pNode, void* pConte
if (QUERY_NODE_COLUMN == nodeType(pNode)) {
SColumnNode* pCol = (SColumnNode*)pNode;
RewriteTableAliasCxt* pCtx = (RewriteTableAliasCxt*)pContext;
strncpy(pCol->tableAlias, pCtx->newTableAlias, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableAlias, pCtx->newTableAlias, TSDB_TABLE_NAME_LEN);
}
return DEAL_RES_CONTINUE;
}
@ -3760,7 +3758,7 @@ static int32_t rewriteTailOptCreateProjectExpr(SFunctionNode* pFunc, SNode** ppN
if (NULL == pExpr) {
return code;
}
strcpy(((SExprNode*)pExpr)->aliasName, pFunc->node.aliasName);
tstrncpy(((SExprNode*)pExpr)->aliasName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
*ppNode = pExpr;
return code;
}
@ -3916,15 +3914,15 @@ static int32_t rewriteUniqueOptCreateFirstFunc(SFunctionNode* pSelectValue, SNod
return code;
}
strcpy(pFunc->functionName, "first");
tstrncpy(pFunc->functionName, "first", TSDB_FUNC_NAME_LEN);
if (NULL != pSelectValue) {
strcpy(pFunc->node.aliasName, pSelectValue->node.aliasName);
tstrncpy(pFunc->node.aliasName, pSelectValue->node.aliasName, TSDB_COL_NAME_LEN);
} else {
int64_t pointer = (int64_t)pFunc;
char name[TSDB_FUNC_NAME_LEN + TSDB_POINTER_PRINT_BYTES + TSDB_NAME_DELIMITER_LEN + 1] = {0};
int32_t len = tsnprintf(name, sizeof(name) - 1, "%s.%" PRId64 "", pFunc->functionName, pointer);
(void)taosHashBinary(name, len);
strncpy(pFunc->node.aliasName, name, TSDB_COL_NAME_LEN - 1);
tstrncpy(pFunc->node.aliasName, name, TSDB_COL_NAME_LEN);
}
SNode* pNew = NULL;
code = nodesCloneNode(pCol, &pNew);
@ -4022,15 +4020,15 @@ static int32_t rewriteUniqueOptCreateProjectCol(SFunctionNode* pFunc, SNode** pp
if (FUNCTION_TYPE_UNIQUE == pFunc->funcType) {
SExprNode* pExpr = (SExprNode*)nodesListGetNode(pFunc->pParameterList, 0);
if (QUERY_NODE_COLUMN == nodeType(pExpr)) {
strcpy(pCol->tableAlias, ((SColumnNode*)pExpr)->tableAlias);
strcpy(pCol->colName, ((SColumnNode*)pExpr)->colName);
tstrncpy(pCol->tableAlias, ((SColumnNode*)pExpr)->tableAlias, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->colName, ((SColumnNode*)pExpr)->colName, TSDB_COL_NAME_LEN);
} else {
strcpy(pCol->colName, pExpr->aliasName);
tstrncpy(pCol->colName, pExpr->aliasName, TSDB_COL_NAME_LEN);
}
} else {
strcpy(pCol->colName, pFunc->node.aliasName);
tstrncpy(pCol->colName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
}
strcpy(pCol->node.aliasName, pFunc->node.aliasName);
tstrncpy(pCol->node.aliasName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
*ppNode = (SNode*)pCol;
return code;
}
@ -4404,7 +4402,7 @@ static int32_t lastRowScanBuildFuncTypes(SScanLogicNode* pScan, SColumnNode* pCo
return terrno;
}
pFuncTypeParam->pCol->colId = pColNode->colId;
strcpy(pFuncTypeParam->pCol->name, pColNode->colName);
tstrncpy(pFuncTypeParam->pCol->name, pColNode->colName, TSDB_COL_NAME_LEN);
if (NULL == taosArrayPush(pScan->pFuncTypes, pFuncTypeParam)) {
taosMemoryFree(pFuncTypeParam);
return terrno;
@ -4481,15 +4479,15 @@ static int32_t lastRowScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogic
if (TSDB_CODE_SUCCESS != code) {
break;
}
sprintf(((SColumnNode*)newColNode)->colName, "#dup_col.%p", newColNode);
sprintf(((SColumnNode*)pParamNode)->colName, "#dup_col.%p", newColNode);
snprintf(((SColumnNode*)newColNode)->colName, TSDB_COL_NAME_LEN, "#dup_col.%p", newColNode);
snprintf(((SColumnNode*)pParamNode)->colName, TSDB_COL_NAME_LEN, "#dup_col.%p", newColNode);
if (FUNCTION_TYPE_LAST_ROW == funcType &&
((SColumnNode*)pParamNode)->colId == PRIMARYKEY_TIMESTAMP_COL_ID) {
if (!adjLastRowTsColName) {
adjLastRowTsColName = true;
strncpy(tsColName, ((SColumnNode*)pParamNode)->colName, TSDB_COL_NAME_LEN);
tstrncpy(tsColName, ((SColumnNode*)pParamNode)->colName, TSDB_COL_NAME_LEN);
} else {
strncpy(((SColumnNode*)pParamNode)->colName, tsColName, TSDB_COL_NAME_LEN);
tstrncpy(((SColumnNode*)pParamNode)->colName, tsColName, TSDB_COL_NAME_LEN);
nodesDestroyNode(newColNode);
continue;
}
@ -4613,7 +4611,7 @@ static int32_t lastRowScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogic
((cxt.pLastCols->length == 1 && nodesEqualNode((SNode*)pPKTsCol, nodesListGetNode(cxt.pLastCols, 0))) ||
(pScan->node.pTargets->length == 2 && cxt.pkBytes > 0))) {
// when select last(ts),tbname,ts from ..., we add another ts to targets
sprintf(pPKTsCol->colName, "#sel_val.%p", pPKTsCol);
snprintf(pPKTsCol->colName, TSDB_COL_NAME_LEN, "#sel_val.%p", pPKTsCol);
SNode* pNew = NULL;
code = nodesCloneNode((SNode*)pPKTsCol, &pNew);
if (TSDB_CODE_SUCCESS == code) {
@ -4630,7 +4628,7 @@ static int32_t lastRowScanOptimize(SOptimizeContext* pCxt, SLogicSubplan* pLogic
if (pNonPKCol && cxt.pLastCols->length == 1 &&
nodesEqualNode((SNode*)pNonPKCol, nodesListGetNode(cxt.pLastCols, 0))) {
// when select last(c1), c1 from ..., we add c1 to targets
sprintf(pNonPKCol->colName, "#sel_val.%p", pNonPKCol);
snprintf(pNonPKCol->colName, TSDB_COL_NAME_LEN, "#sel_val.%p", pNonPKCol);
SNode* pNew = NULL;
code = nodesCloneNode((SNode*)pNonPKCol, &pNew);
if (TSDB_CODE_SUCCESS == code) {
@ -5402,10 +5400,10 @@ static bool tbCntScanOptIsEligibleLogicCond(STbCntScanOptInfo* pInfo, SLogicCond
}
if (!hasDbCond && 0 == strcmp(pCol->colName, "db_name")) {
hasDbCond = true;
strcpy(pInfo->table.dbname, pVal->literal);
tstrncpy(pInfo->table.dbname, pVal->literal, TSDB_DB_NAME_LEN);
} else if (!hasStbCond && 0 == strcmp(pCol->colName, "stable_name")) {
hasStbCond = true;
strcpy(pInfo->table.tname, pVal->literal);
tstrncpy(pInfo->table.tname, pVal->literal, TSDB_TABLE_NAME_LEN);
} else {
return false;
}
@ -5469,8 +5467,8 @@ static int32_t tbCntScanOptCreateTableCountFunc(SNode** ppNode) {
if (NULL == pFunc) {
return code;
}
strcpy(pFunc->functionName, "_table_count");
strcpy(pFunc->node.aliasName, "_table_count");
tstrncpy(pFunc->functionName, "_table_count", TSDB_FUNC_NAME_LEN);
tstrncpy(pFunc->node.aliasName, "_table_count", TSDB_COL_NAME_LEN);
code = fmGetFuncInfo(pFunc, NULL, 0);
if (TSDB_CODE_SUCCESS != code) {
nodesDestroyNode((SNode*)pFunc);
@ -5482,8 +5480,8 @@ static int32_t tbCntScanOptCreateTableCountFunc(SNode** ppNode) {
static int32_t tbCntScanOptRewriteScan(STbCntScanOptInfo* pInfo) {
pInfo->pScan->scanType = SCAN_TYPE_TABLE_COUNT;
strcpy(pInfo->pScan->tableName.dbname, pInfo->table.dbname);
strcpy(pInfo->pScan->tableName.tname, pInfo->table.tname);
tstrncpy(pInfo->pScan->tableName.dbname, pInfo->table.dbname, TSDB_DB_NAME_LEN);
tstrncpy(pInfo->pScan->tableName.tname, pInfo->table.tname, TSDB_TABLE_NAME_LEN);
NODES_DESTORY_LIST(pInfo->pScan->node.pTargets);
NODES_DESTORY_LIST(pInfo->pScan->pScanCols);
NODES_DESTORY_NODE(pInfo->pScan->node.pConditions);
@ -5533,8 +5531,8 @@ static int32_t tbCntScanOptCreateSumFunc(SFunctionNode* pCntFunc, SNode* pParam,
if (NULL == pFunc) {
return code;
}
strcpy(pFunc->functionName, "sum");
strcpy(pFunc->node.aliasName, pCntFunc->node.aliasName);
tstrncpy(pFunc->functionName, "sum", TSDB_FUNC_NAME_LEN);
tstrncpy(pFunc->node.aliasName, pCntFunc->node.aliasName, TSDB_COL_NAME_LEN);
code = createColumnByRewriteExpr(pParam, &pFunc->pParameterList);
if (TSDB_CODE_SUCCESS == code) {
code = fmGetFuncInfo(pFunc, NULL, 0);
@ -6311,29 +6309,22 @@ static int32_t stbJoinOptCreateDynQueryCtrlNode(SLogicNode* pRoot, SLogicNode* p
if (TSDB_CODE_SUCCESS == code) {
pDynCtrl->node.pChildren = NULL;
code = nodesMakeList(&pDynCtrl->node.pChildren);
if (NULL == pDynCtrl->node.pChildren) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {
pDynCtrl->stbJoin.pVgList = NULL;
code = nodesMakeList(&pDynCtrl->stbJoin.pVgList);
if (NULL == pDynCtrl->stbJoin.pVgList) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {
pDynCtrl->stbJoin.pUidList = NULL;
code = nodesMakeList(&pDynCtrl->stbJoin.pUidList);
if (NULL == pDynCtrl->stbJoin.pUidList) {
code = code;
}
}
SJoinLogicNode* pHJoin = (SJoinLogicNode*)pPrev;
if (TSDB_CODE_SUCCESS == code) {
code = nodesListStrictAppend(pDynCtrl->stbJoin.pUidList, nodesListGetNode(pHJoin->node.pTargets, 0));
}
if (TSDB_CODE_SUCCESS == code) {
code = nodesListStrictAppend(pDynCtrl->stbJoin.pUidList, nodesListGetNode(pHJoin->node.pTargets, 2));
}
@ -6352,9 +6343,6 @@ static int32_t stbJoinOptCreateDynQueryCtrlNode(SLogicNode* pRoot, SLogicNode* p
if (TSDB_CODE_SUCCESS == code) {
pDynCtrl->node.pTargets = NULL;
code = nodesCloneList(pPost->pTargets, &pDynCtrl->node.pTargets);
if (!pDynCtrl->node.pTargets) {
code = code;
}
}
}
@ -7101,10 +7089,9 @@ int32_t tsmaOptCreateTsmaScanCols(const STSMAOptUsefulTsma* pTsma, const SNodeLi
pCol->tableType = TSDB_SUPER_TABLE;
pCol->tableId = pTsma->targetTbUid;
pCol->colType = COLUMN_TYPE_COLUMN;
strcpy(pCol->tableName, pTsma->targetTbName);
strcpy(pCol->dbName, pTsma->pTsma->targetDbFName);
strcpy(pCol->colName, pFunc->node.aliasName);
strcpy(pCol->node.aliasName, pFunc->node.aliasName);
tstrncpy(pCol->tableName, pTsma->targetTbName, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->colName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
tstrncpy(pCol->node.aliasName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
pCol->node.resType.type = TSDB_DATA_TYPE_BINARY;
code = nodesListMakeStrictAppend(&pScanCols, (SNode*)pCol);
}
@ -7128,8 +7115,8 @@ static int32_t tsmaOptRewriteTag(const STSMAOptCtx* pTsmaOptCtx, const STSMAOptU
for (int32_t i = 0; i < pTsma->pTsma->pTags->size; ++i) {
const SSchema* pSchema = taosArrayGet(pTsma->pTsma->pTags, i);
if (strcmp(pTagCol->colName, pSchema->name) == 0) {
strcpy(pTagCol->tableName, pTsma->targetTbName);
strcpy(pTagCol->tableAlias, pTsma->targetTbName);
tstrncpy(pTagCol->tableName, pTsma->targetTbName, TSDB_TABLE_NAME_LEN);
tstrncpy(pTagCol->tableAlias, pTsma->targetTbName, TSDB_TABLE_NAME_LEN);
pTagCol->tableId = pTsma->targetTbUid;
pTagCol->tableType = TSDB_SUPER_TABLE;
pTagCol->colId = pSchema->colId;
@ -7154,8 +7141,8 @@ static int32_t tsmaOptRewriteTbname(const STSMAOptCtx* pTsmaOptCtx, SNode** pTbN
nodesDestroyNode(*pTbNameNode);
SColumnNode* pCol = (SColumnNode*)pRewrittenFunc;
const SSchema* pSchema = taosArrayGet(pTsma->pTsma->pTags, pTsma->pTsma->pTags->size - 1);
strcpy(pCol->tableName, pTsma->targetTbName);
strcpy(pCol->tableAlias, pTsma->targetTbName);
tstrncpy(pCol->tableName, pTsma->targetTbName, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableAlias, pTsma->targetTbName, TSDB_TABLE_NAME_LEN);
pCol->tableId = pTsma->targetTbUid;
pCol->tableType = TSDB_SUPER_TABLE;
pCol->colId = pSchema->colId;
@ -7280,7 +7267,7 @@ static int32_t tsmaOptRewriteScan(STSMAOptCtx* pTsmaOptCtx, SScanLogicNode* pNew
if (code == TSDB_CODE_SUCCESS) {
pNewScan->stableId = pTsma->pTsma->destTbUid;
pNewScan->tableId = pTsma->targetTbUid;
strcpy(pNewScan->tableName.tname, pTsma->targetTbName);
tstrncpy(pNewScan->tableName.tname, pTsma->targetTbName, TSDB_TABLE_NAME_LEN);
}
if (code == TSDB_CODE_SUCCESS) {
code = tsmaOptRewriteNodeList(pNewScan->pScanPseudoCols, pTsmaOptCtx, pTsma, true, true);
@ -7333,12 +7320,12 @@ static int32_t tsmaOptCreateWStart(int8_t precision, SFunctionNode** pWStartOut)
if (NULL == pWStart) {
return code;
}
strcpy(pWStart->functionName, "_wstart");
tstrncpy(pWStart->functionName, "_wstart", TSDB_FUNC_NAME_LEN);
int64_t pointer = (int64_t)pWStart;
char name[TSDB_COL_NAME_LEN + TSDB_POINTER_PRINT_BYTES + TSDB_NAME_DELIMITER_LEN + 1] = {0};
int32_t len = tsnprintf(name, sizeof(name) - 1, "%s.%" PRId64 "", pWStart->functionName, pointer);
(void)taosHashBinary(name, len);
strncpy(pWStart->node.aliasName, name, TSDB_COL_NAME_LEN - 1);
tstrncpy(pWStart->node.aliasName, name, TSDB_COL_NAME_LEN);
pWStart->node.resType.precision = precision;
code = fmGetFuncInfo(pWStart, NULL, 0);
@ -7441,12 +7428,12 @@ static int32_t tsmaOptGeneratePlan(STSMAOptCtx* pTsmaOptCtx) {
for (int32_t j = 0; j < pTsmaOptCtx->pScan->pTsmas->size; ++j) {
if (taosArrayGetP(pTsmaOptCtx->pScan->pTsmas, j) == pTsma->pTsma) {
const STsmaTargetTbInfo* ptbInfo = taosArrayGet(pTsmaOptCtx->pScan->pTsmaTargetTbInfo, j);
strcpy(pTsma->targetTbName, ptbInfo->tableName);
tstrncpy(pTsma->targetTbName, ptbInfo->tableName, TSDB_TABLE_NAME_LEN);
pTsma->targetTbUid = ptbInfo->uid;
}
}
} else {
strcpy(pTsma->targetTbName, pTsma->pTsma->targetTb);
tstrncpy(pTsma->targetTbName, pTsma->pTsma->targetTb, TSDB_TABLE_NAME_LEN);
pTsma->targetTbUid = pTsma->pTsma->destTbUid;
}
}

70
source/libs/planner/src/planPhysiCreater.c
View File

@ -40,9 +40,9 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pStmtName);
strcat(*ppKey, ".");
strcat(*ppKey, pCol->node.aliasName);
TAOS_STRNCAT(*ppKey, pStmtName, TSDB_TABLE_NAME_LEN);
TAOS_STRNCAT(*ppKey, ".", 2);
TAOS_STRNCAT(*ppKey, pCol->node.aliasName, TSDB_COL_NAME_LEN);
*pLen = taosHashBinary(*ppKey, strlen(*ppKey));
return code;
} else {
@ -50,7 +50,7 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pCol->node.aliasName);
TAOS_STRNCAT(*ppKey, pCol->node.aliasName, TSDB_COL_NAME_LEN);
*pLen = strlen(*ppKey);
return code;
}
@ -60,7 +60,7 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pCol->colName);
TAOS_STRNCAT(*ppKey, pCol->colName, TSDB_COL_NAME_LEN);
*pLen = strlen(*ppKey);
return code;
}
@ -69,9 +69,9 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pCol->tableAlias);
strcat(*ppKey, ".");
strcat(*ppKey, pCol->colName);
TAOS_STRNCAT(*ppKey, pCol->tableAlias, TSDB_TABLE_NAME_LEN);
TAOS_STRNCAT(*ppKey, ".", 2);
TAOS_STRNCAT(*ppKey, pCol->colName, TSDB_COL_NAME_LEN);
*pLen = taosHashBinary(*ppKey, strlen(*ppKey));
return code;
} else if (QUERY_NODE_FUNCTION == nodeType(pNode)) {
@ -84,9 +84,9 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pStmtName);
strcat(*ppKey, ".");
strcat(*ppKey, ((SExprNode*)pNode)->aliasName);
TAOS_STRNCAT(*ppKey, pStmtName, TSDB_TABLE_NAME_LEN);
TAOS_STRNCAT(*ppKey, ".", 2);
TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN);
*pLen = taosHashBinary(*ppKey, strlen(*ppKey));
return code;
}
@ -94,9 +94,9 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pVal->literal);
strcat(*ppKey, ".");
strcat(*ppKey, ((SExprNode*)pNode)->aliasName);
TAOS_STRNCAT(*ppKey, pVal->literal, strlen(pVal->literal));
TAOS_STRNCAT(*ppKey, ".", 2);
TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN);
*pLen = taosHashBinary(*ppKey, strlen(*ppKey));
return code;
}
@ -108,9 +108,9 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, pStmtName);
strcat(*ppKey, ".");
strcat(*ppKey, ((SExprNode*)pNode)->aliasName);
TAOS_STRNCAT(*ppKey, pStmtName, TSDB_TABLE_NAME_LEN);
TAOS_STRNCAT(*ppKey, ".", 2);
TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN);
*pLen = taosHashBinary(*ppKey, strlen(*ppKey));
return code;
}
@ -119,7 +119,7 @@ static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int
if (!*ppKey) {
return terrno;
}
strcat(*ppKey, ((SExprNode*)pNode)->aliasName);
TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN);
*pLen = strlen(*ppKey);
return code;
}
@ -228,7 +228,7 @@ static int32_t buildDataBlockSlots(SPhysiPlanContext* pCxt, SNodeList* pList, SD
code = putSlotToHash(name, len, pDataBlockDesc->dataBlockId, slotId, pNode, pHash);
if (TSDB_CODE_SUCCESS == code) {
if (nodeType(pNode) == QUERY_NODE_COLUMN && ((SColumnNode*)pNode)->resIdx > 0) {
sprintf(name + strlen(name), "_%d", ((SColumnNode*)pNode)->resIdx);
snprintf(name + strlen(name), 16, "_%d", ((SColumnNode*)pNode)->resIdx);
code = putSlotToHash(name, strlen(name), pDataBlockDesc->dataBlockId, slotId, pNode, pProjIdxDescHash);
}
}
@ -395,7 +395,7 @@ static EDealRes doSetSlotId(SNode* pNode, void* pContext) {
}
SSlotIndex* pIndex = NULL;
if (((SColumnNode*)pNode)->projRefIdx > 0) {
sprintf(name + strlen(name), "_%d", ((SColumnNode*)pNode)->projRefIdx);
snprintf(name + strlen(name), 16, "_%d", ((SColumnNode*)pNode)->projRefIdx);
pIndex = taosHashGet(pCxt->pLeftProjIdxHash, name, strlen(name));
if (!pIndex) {
pIndex = taosHashGet(pCxt->pRightProdIdxHash, name, strlen(name));
@ -566,9 +566,6 @@ static int32_t createScanPhysiNodeFinalize(SPhysiPlanContext* pCxt, SSubplan* pS
if (TSDB_CODE_SUCCESS == code && NULL != pScanLogicNode->pScanPseudoCols) {
pScanPhysiNode->pScanPseudoCols = NULL;
code = nodesCloneList(pScanLogicNode->pScanPseudoCols, &pScanPhysiNode->pScanPseudoCols);
if (NULL == pScanPhysiNode->pScanPseudoCols) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {
@ -588,9 +585,6 @@ static int32_t createScanPhysiNodeFinalize(SPhysiPlanContext* pCxt, SSubplan* pS
if (NULL != pScanLogicNode->pTagCond) {
pSubplan->pTagCond = NULL;
code = nodesCloneNode(pScanLogicNode->pTagCond, &pSubplan->pTagCond);
if (NULL == pSubplan->pTagCond) {
code = code;
}
}
}
@ -598,9 +592,6 @@ static int32_t createScanPhysiNodeFinalize(SPhysiPlanContext* pCxt, SSubplan* pS
if (NULL != pScanLogicNode->pTagIndexCond) {
pSubplan->pTagIndexCond = NULL;
code = nodesCloneNode(pScanLogicNode->pTagIndexCond, &pSubplan->pTagIndexCond);
if (NULL == pSubplan->pTagIndexCond) {
code = code;
}
}
}
@ -1690,11 +1681,11 @@ static EDealRes collectAndRewrite(SRewritePrecalcExprsCxt* pCxt, SNode** pNode)
SExprNode* pRewrittenExpr = (SExprNode*)pExpr;
pCol->node.resType = pRewrittenExpr->resType;
if ('\0' != pRewrittenExpr->aliasName[0]) {
strcpy(pCol->colName, pRewrittenExpr->aliasName);
tstrncpy(pCol->colName, pRewrittenExpr->aliasName, TSDB_COL_NAME_LEN);
} else {
snprintf(pRewrittenExpr->aliasName, sizeof(pRewrittenExpr->aliasName), "#expr_%d_%d", pCxt->planNodeId,
pCxt->rewriteId);
strcpy(pCol->colName, pRewrittenExpr->aliasName);
tstrncpy(pCol->colName, pRewrittenExpr->aliasName, TSDB_COL_NAME_LEN);
}
nodesDestroyNode(*pNode);
*pNode = (SNode*)pCol;
@ -1715,7 +1706,7 @@ static int32_t rewriteValueToOperator(SRewritePrecalcExprsCxt* pCxt, SNode** pNo
}
SValueNode* pVal = (SValueNode*)*pNode;
pOper->node.resType = pVal->node.resType;
strcpy(pOper->node.aliasName, pVal->node.aliasName);
tstrncpy(pOper->node.aliasName, pVal->node.aliasName, TSDB_COL_NAME_LEN);
pOper->opType = OP_TYPE_ASSIGN;
pOper->pRight = *pNode;
*pNode = (SNode*)pOper;
@ -1976,9 +1967,6 @@ static int32_t createInterpFuncPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pCh
pInterpFunc->rangeInterval = pFuncLogicNode->rangeInterval;
pInterpFunc->rangeIntervalUnit = pFuncLogicNode->rangeIntervalUnit;
code = nodesCloneNode(pFuncLogicNode->pFillValues, &pInterpFunc->pFillValues);
if (TSDB_CODE_SUCCESS != code) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {
@ -2893,7 +2881,7 @@ static int32_t createQueryInserter(SPhysiPlanContext* pCxt, SVnodeModifyLogicNod
pInserter->tableId = pModify->tableId;
pInserter->stableId = pModify->stableId;
pInserter->tableType = pModify->tableType;
strcpy(pInserter->tableName, pModify->tableName);
tstrncpy(pInserter->tableName, pModify->tableName, TSDB_TABLE_NAME_LEN);
pInserter->explain = (QUERY_NODE_EXPLAIN_STMT == nodeType(pCxt->pPlanCxt->pAstRoot) ? true : false);
if (pModify->pVgroupList) {
pInserter->vgId = pModify->pVgroupList->vgroups[0].vgId;
@ -2905,9 +2893,6 @@ static int32_t createQueryInserter(SPhysiPlanContext* pCxt, SVnodeModifyLogicNod
if (TSDB_CODE_SUCCESS == code) {
pInserter->sink.pInputDataBlockDesc = NULL;
code = nodesCloneNode((SNode*)pSubplan->pNode->pOutputDataBlockDesc, (SNode**)&pInserter->sink.pInputDataBlockDesc);
if (NULL == pInserter->sink.pInputDataBlockDesc) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {
@ -2946,8 +2931,8 @@ static int32_t createDataDeleter(SPhysiPlanContext* pCxt, SVnodeModifyLogicNode*
pDeleter->tableId = pModify->tableId;
pDeleter->tableType = pModify->tableType;
strcpy(pDeleter->tableFName, pModify->tableName);
strcpy(pDeleter->tsColName, pModify->tsColName);
tstrncpy(pDeleter->tableFName, pModify->tableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pDeleter->tsColName, pModify->tsColName, TSDB_COL_NAME_LEN);
pDeleter->deleteTimeRange = pModify->deleteTimeRange;
code = setNodeSlotId(pCxt, pRoot->pOutputDataBlockDesc->dataBlockId, -1, pModify->pAffectedRows,
@ -2961,9 +2946,6 @@ static int32_t createDataDeleter(SPhysiPlanContext* pCxt, SVnodeModifyLogicNode*
if (TSDB_CODE_SUCCESS == code) {
pDeleter->sink.pInputDataBlockDesc = NULL;
code = nodesCloneNode((SNode*)pRoot->pOutputDataBlockDesc, (SNode**)&pDeleter->sink.pInputDataBlockDesc);
if (NULL == pDeleter->sink.pInputDataBlockDesc) {
code = code;
}
}
if (TSDB_CODE_SUCCESS == code) {

32
source/libs/planner/src/planSpliter.c
View File

@ -428,12 +428,12 @@ static int32_t stbSplAppendWStart(SNodeList* pFuncs, int32_t* pIndex, uint8_t pr
if (NULL == pWStart) {
return code;
}
strcpy(pWStart->functionName, "_wstart");
tstrncpy(pWStart->functionName, "_wstart", TSDB_FUNC_NAME_LEN);
int64_t pointer = (int64_t)pWStart;
char name[TSDB_COL_NAME_LEN + TSDB_POINTER_PRINT_BYTES + TSDB_NAME_DELIMITER_LEN + 1] = {0};
int32_t len = tsnprintf(name, sizeof(name) - 1, "%s.%" PRId64 "", pWStart->functionName, pointer);
(void)taosHashBinary(name, len);
strncpy(pWStart->node.aliasName, name, TSDB_COL_NAME_LEN - 1);
tstrncpy(pWStart->node.aliasName, name, TSDB_COL_NAME_LEN);
pWStart->node.resType.precision = precision;
code = fmGetFuncInfo(pWStart, NULL, 0);
@ -460,12 +460,12 @@ static int32_t stbSplAppendWEnd(SWindowLogicNode* pWin, int32_t* pIndex) {
if (NULL == pWEnd) {
return code;
}
strcpy(pWEnd->functionName, "_wend");
tstrncpy(pWEnd->functionName, "_wend", TSDB_FUNC_NAME_LEN);
int64_t pointer = (int64_t)pWEnd;
char name[TSDB_COL_NAME_LEN + TSDB_POINTER_PRINT_BYTES + TSDB_NAME_DELIMITER_LEN + 1] = {0};
int32_t len = tsnprintf(name, sizeof(name) - 1, "%s.%" PRId64 "", pWEnd->functionName, pointer);
(void)taosHashBinary(name, len);
strncpy(pWEnd->node.aliasName, name, TSDB_COL_NAME_LEN - 1);
tstrncpy(pWEnd->node.aliasName, name, TSDB_COL_NAME_LEN);
code = fmGetFuncInfo(pWEnd, NULL, 0);
if (TSDB_CODE_SUCCESS == code) {
@ -1135,8 +1135,8 @@ static int32_t stbSplAggNodeCreateMerge(SSplitContext* pCtx, SStableSplitInfo* p
if (!nodesEqualNode(pParam, (SNode*)pCol)) continue;
// use the colName of group_key func to make sure finding the right slot id for merge keys.
strcpy(pCol->colName, pFunc->node.aliasName);
strcpy(pCol->node.aliasName, pFunc->node.aliasName);
tstrncpy(pCol->colName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
tstrncpy(pCol->node.aliasName, pFunc->node.aliasName, TSDB_COL_NAME_LEN);
memset(pCol->tableAlias, 0, TSDB_TABLE_NAME_LEN);
break;
}
@ -1265,15 +1265,15 @@ static int32_t stbSplCreateColumnNode(SExprNode* pExpr, SNode** ppNode) {
return code;
}
if (QUERY_NODE_COLUMN == nodeType(pExpr)) {
strcpy(pCol->dbName, ((SColumnNode*)pExpr)->dbName);
strcpy(pCol->tableName, ((SColumnNode*)pExpr)->tableName);
strcpy(pCol->tableAlias, ((SColumnNode*)pExpr)->tableAlias);
strcpy(pCol->colName, ((SColumnNode*)pExpr)->colName);
tstrncpy(pCol->dbName, ((SColumnNode*)pExpr)->dbName, TSDB_DB_NAME_LEN);
tstrncpy(pCol->tableName, ((SColumnNode*)pExpr)->tableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableAlias, ((SColumnNode*)pExpr)->tableAlias, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->colName, ((SColumnNode*)pExpr)->colName, TSDB_COL_NAME_LEN);
} else {
strcpy(pCol->colName, pExpr->aliasName);
tstrncpy(pCol->colName, pExpr->aliasName, TSDB_COL_NAME_LEN);
}
strcpy(pCol->node.aliasName, pExpr->aliasName);
strcpy(pCol->node.userAlias, pExpr->userAlias);
tstrncpy(pCol->node.aliasName, pExpr->aliasName, TSDB_COL_NAME_LEN);
tstrncpy(pCol->node.userAlias, pExpr->userAlias, TSDB_COL_NAME_LEN);
pCol->node.resType = pExpr->resType;
*ppNode = (SNode*)pCol;
return code;
@ -1358,9 +1358,6 @@ static int32_t stbSplCreatePartSortNode(SSortLogicNode* pSort, SLogicNode** pOut
int32_t code = TSDB_CODE_SUCCESS;
SSortLogicNode* pPartSort = NULL;
code = nodesCloneNode((SNode*)pSort, (SNode**)&pPartSort);
if (NULL == pPartSort) {
code = code;
}
SNodeList* pMergeKeys = NULL;
if (TSDB_CODE_SUCCESS == code) {
@ -1541,9 +1538,6 @@ static int32_t stbSplCreateMergeScanNode(SScanLogicNode* pScan, SLogicNode** pOu
int32_t code = TSDB_CODE_SUCCESS;
SScanLogicNode* pMergeScan = NULL;
code = nodesCloneNode((SNode*)pScan, (SNode**)&pMergeScan);
if (NULL == pMergeScan) {
code = code;
}
SNodeList* pMergeKeys = NULL;
if (TSDB_CODE_SUCCESS == code) {

14
source/libs/planner/src/planUtil.c
View File

@ -68,14 +68,14 @@ static EDealRes doCreateColumn(SNode* pNode, void* pContext) {
return DEAL_RES_ERROR;
}
pCol->node.resType = pExpr->resType;
strcpy(pCol->colName, pExpr->aliasName);
tstrncpy(pCol->colName, pExpr->aliasName, TSDB_COL_NAME_LEN);
if (QUERY_NODE_FUNCTION == nodeType(pNode)) {
SFunctionNode* pFunc = (SFunctionNode*)pNode;
if (pFunc->funcType == FUNCTION_TYPE_TBNAME) {
SValueNode* pVal = (SValueNode*)nodesListGetNode(pFunc->pParameterList, 0);
if (NULL != pVal) {
strcpy(pCol->tableAlias, pVal->literal);
strcpy(pCol->tableName, pVal->literal);
tstrncpy(pCol->tableAlias, pVal->literal, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->tableName, pVal->literal, TSDB_TABLE_NAME_LEN);
}
}
}
@ -636,9 +636,9 @@ SFunctionNode* createGroupKeyAggFunc(SColumnNode* pGroupCol) {
SFunctionNode* pFunc = NULL;
int32_t code = nodesMakeNode(QUERY_NODE_FUNCTION, (SNode**)&pFunc);
if (pFunc) {
strcpy(pFunc->functionName, "_group_key");
strcpy(pFunc->node.aliasName, pGroupCol->node.aliasName);
strcpy(pFunc->node.userAlias, pGroupCol->node.userAlias);
tstrncpy(pFunc->functionName, "_group_key", TSDB_FUNC_NAME_LEN);
tstrncpy(pFunc->node.aliasName, pGroupCol->node.aliasName, TSDB_COL_NAME_LEN);
tstrncpy(pFunc->node.userAlias, pGroupCol->node.userAlias, TSDB_COL_NAME_LEN);
SNode* pNew = NULL;
code = nodesCloneNode((SNode*)pGroupCol, &pNew);
if (TSDB_CODE_SUCCESS == code) {
@ -655,7 +655,7 @@ SFunctionNode* createGroupKeyAggFunc(SColumnNode* pGroupCol) {
char name[TSDB_FUNC_NAME_LEN + TSDB_NAME_DELIMITER_LEN + TSDB_POINTER_PRINT_BYTES + 1] = {0};
int32_t len = tsnprintf(name, sizeof(name) - 1, "%s.%p", pFunc->functionName, pFunc);
(void)taosHashBinary(name, len);
strncpy(pFunc->node.aliasName, name, TSDB_COL_NAME_LEN - 1);
tstrncpy(pFunc->node.aliasName, name, TSDB_COL_NAME_LEN);
}
}
if (TSDB_CODE_SUCCESS != code) {

1
source/libs/scalar/src/scalar.c
View File

@ -1241,7 +1241,6 @@ EDealRes sclRewriteFunction(SNode **pNode, SScalarCtx *ctx) {
ctx->code = TSDB_CODE_OUT_OF_MEMORY;
return DEAL_RES_ERROR;
}
res->node.resType.bytes = varDataTLen(output.columnData->pData);
(void)memcpy(res->datum.p, output.columnData->pData, varDataTLen(output.columnData->pData));
} else {
ctx->code = nodesSetValueNodeValue(res, output.columnData->pData);

6
source/libs/scheduler/src/schTask.c
View File

@ -1129,7 +1129,11 @@ int32_t schLaunchRemoteTask(SSchJob *pJob, SSchTask *pTask) {
int32_t schLaunchLocalTask(SSchJob *pJob, SSchTask *pTask) {
// SCH_ERR_JRET(schSetTaskCandidateAddrs(pJob, pTask));
if (NULL == schMgmt.queryMgmt) {
SCH_ERR_RET(qWorkerInit(NODE_TYPE_CLIENT, CLIENT_HANDLE, (void **)&schMgmt.queryMgmt, NULL));
void* p = NULL;
SCH_ERR_RET(qWorkerInit(NODE_TYPE_CLIENT, CLIENT_HANDLE, &p, NULL));
if (atomic_val_compare_exchange_ptr(&schMgmt.queryMgmt, NULL, p)) {
qWorkerDestroy(&p);
}
}
SArray *explainRes = NULL;

2
source/libs/stream/inc/streamInt.h
View File

@ -37,7 +37,7 @@ extern "C" {
#define META_HB_CHECK_INTERVAL 200
#define META_HB_SEND_IDLE_COUNTER 25 // send hb every 5 sec
#define STREAM_TASK_KEY_LEN ((sizeof(int64_t)) << 1)
#define STREAM_TASK_QUEUE_CAPACITY 20480
#define STREAM_TASK_QUEUE_CAPACITY 5120
#define STREAM_TASK_QUEUE_CAPACITY_IN_SIZE (30)
// clang-format off

1
source/libs/stream/src/streamBackendRocksdb.c
View File

@ -4378,7 +4378,6 @@ void streamStateParTagSeekKeyNext_rocksdb(SStreamState* pState, const int64_t gr
}
int32_t streamStateParTagGetKVByCur_rocksdb(SStreamStateCur* pCur, int64_t* pGroupId, const void** pVal, int32_t* pVLen) {
stDebug("streamStateFillGetKVByCur_rocksdb");
if (!pCur) {
return -1;
}

14
source/libs/stream/src/streamCheckpoint.c
View File

@ -613,7 +613,6 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV
{ // destroy the related fill-history tasks
// drop task should not in the meta-lock, and drop the related fill-history task now
streamMetaWUnLock(pMeta);
if (pReq->dropRelHTask) {
code = streamMetaUnregisterTask(pMeta, pReq->hStreamId, pReq->hTaskId);
int32_t numOfTasks = streamMetaGetNumOfTasks(pMeta);
@ -621,7 +620,6 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV
id, vgId, pReq->taskId, numOfTasks);
}
streamMetaWLock(pMeta);
if (pReq->dropRelHTask) {
code = streamMetaCommit(pMeta);
}
@ -656,9 +654,11 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV
pInfo->processedVer <= pReq->checkpointVer);
if (!valid) {
stFatal("invalid checkpoint id check, current checkpointId:%" PRId64 " checkpointVer:%" PRId64
" processedVer:%" PRId64 " req checkpointId:%" PRId64 " checkpointVer:%" PRId64,
pInfo->checkpointId, pInfo->checkpointVer, pInfo->processedVer, pReq->checkpointId, pReq->checkpointVer);
stFatal("s-task:%s invalid checkpointId update info recv, current checkpointId:%" PRId64 " checkpointVer:%" PRId64
" processedVer:%" PRId64 " req checkpointId:%" PRId64 " checkpointVer:%" PRId64 " discard it",
id, pInfo->checkpointId, pInfo->checkpointVer, pInfo->processedVer, pReq->checkpointId,
pReq->checkpointVer);
streamMutexUnlock(&pTask->lock);
return TSDB_CODE_STREAM_INTERNAL_ERROR;
}
@ -695,8 +695,6 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV
return TSDB_CODE_SUCCESS;
}
streamMetaWUnLock(pMeta);
// drop task should not in the meta-lock, and drop the related fill-history task now
if (pReq->dropRelHTask) {
code = streamMetaUnregisterTask(pMeta, pReq->hStreamId, pReq->hTaskId);
@ -705,9 +703,7 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV
(int32_t)pReq->hTaskId, numOfTasks);
}
streamMetaWLock(pMeta);
code = streamMetaCommit(pMeta);
return TSDB_CODE_SUCCESS;
}

65
source/libs/stream/src/streamExec.c
View File

@ -522,7 +522,10 @@ static int32_t doSetStreamInputBlock(SStreamTask* pTask, const void* pInput, int
if (pItem->type == STREAM_INPUT__GET_RES) {
const SStreamTrigger* pTrigger = (const SStreamTrigger*)pInput;
code = qSetMultiStreamInput(pExecutor, pTrigger->pBlock, 1, STREAM_INPUT__DATA_BLOCK);
if (pTask->info.trigger == STREAM_TRIGGER_FORCE_WINDOW_CLOSE) {
stDebug("s-task:%s set force_window_close as source block, skey:%"PRId64, id, pTrigger->pBlock->info.window.skey);
(*pVer) = pTrigger->pBlock->info.window.skey;
}
} else if (pItem->type == STREAM_INPUT__DATA_SUBMIT) {
const SStreamDataSubmit* pSubmit = (const SStreamDataSubmit*)pInput;
code = qSetMultiStreamInput(pExecutor, &pSubmit->submit, 1, STREAM_INPUT__DATA_SUBMIT);
@ -671,7 +674,7 @@ static int32_t doStreamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pBlock
doRecordThroughput(&pTask->execInfo, totalBlocks, totalSize, blockSize, st, pTask->id.idStr);
// update the currentVer if processing the submit blocks.
// update the currentVer if processing the submitted blocks.
if (!(pInfo->checkpointVer <= pInfo->nextProcessVer && ver >= pInfo->checkpointVer)) {
stError("s-task:%s invalid info, checkpointVer:%" PRId64 ", nextProcessVer:%" PRId64 " currentVer:%" PRId64, id,
pInfo->checkpointVer, pInfo->nextProcessVer, ver);
@ -688,6 +691,34 @@ static int32_t doStreamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pBlock
return code;
}
// do nothing after sync executor state to storage backend, untill checkpoint is completed.
static int32_t doHandleChkptBlock(SStreamTask* pTask) {
int32_t code = 0;
const char* id = pTask->id.idStr;
streamMutexLock(&pTask->lock);
SStreamTaskState pState = streamTaskGetStatus(pTask);
if (pState.state == TASK_STATUS__CK) { // todo other thread may change the status
stDebug("s-task:%s checkpoint block received, set status:%s", id, pState.name);
code = streamTaskBuildCheckpoint(pTask); // ignore this error msg, and continue
} else { // todo refactor
if (pTask->info.taskLevel == TASK_LEVEL__SOURCE) {
code = streamTaskSendCheckpointSourceRsp(pTask);
} else {
code = streamTaskSendCheckpointReadyMsg(pTask);
}
if (code != TSDB_CODE_SUCCESS) {
// todo: let's retry send rsp to upstream/mnode
stError("s-task:%s failed to send checkpoint rsp to upstream, checkpointId:%d, code:%s", id, 0,
tstrerror(code));
}
}
streamMutexUnlock(&pTask->lock);
return code;
}
int32_t flushStateDataInExecutor(SStreamTask* pTask, SStreamQueueItem* pCheckpointBlock) {
const char* id = pTask->id.idStr;
@ -832,36 +863,16 @@ static int32_t doStreamExecTask(SStreamTask* pTask) {
}
}
if (type != STREAM_INPUT__CHECKPOINT) {
if (type == STREAM_INPUT__CHECKPOINT) {
code = doHandleChkptBlock(pTask);
streamFreeQitem(pInput);
return code;
} else {
code = doStreamTaskExecImpl(pTask, pInput, numOfBlocks);
streamFreeQitem(pInput);
if (code) {
return code;
}
} else { // todo other thread may change the status
// do nothing after sync executor state to storage backend, untill the vnode-level checkpoint is completed.
streamMutexLock(&pTask->lock);
SStreamTaskState pState = streamTaskGetStatus(pTask);
if (pState.state == TASK_STATUS__CK) {
stDebug("s-task:%s checkpoint block received, set status:%s", id, pState.name);
code = streamTaskBuildCheckpoint(pTask); // ignore this error msg, and continue
} else { // todo refactor
if (pTask->info.taskLevel == TASK_LEVEL__SOURCE) {
code = streamTaskSendCheckpointSourceRsp(pTask);
} else {
code = streamTaskSendCheckpointReadyMsg(pTask);
}
if (code != TSDB_CODE_SUCCESS) {
// todo: let's retry send rsp to upstream/mnode
stError("s-task:%s failed to send checkpoint rsp to upstream, checkpointId:%d, code:%s", id, 0,
tstrerror(code));
}
}
streamMutexUnlock(&pTask->lock);
streamFreeQitem(pInput);
return code;
}
}
}

6
source/libs/stream/src/streamMeta.c
View File

@ -501,8 +501,6 @@ _err:
void streamMetaInitBackend(SStreamMeta* pMeta) {
pMeta->streamBackend = streamBackendInit(pMeta->path, pMeta->chkpId, pMeta->vgId);
if (pMeta->streamBackend == NULL) {
streamMetaWUnLock(pMeta);
while (1) {
streamMetaWLock(pMeta);
pMeta->streamBackend = streamBackendInit(pMeta->path, pMeta->chkpId, pMeta->vgId);
@ -908,8 +906,6 @@ int32_t streamMetaUnregisterTask(SStreamMeta* pMeta, int64_t streamId, int32_t t
int32_t code = 0;
STaskId id = {.streamId = streamId, .taskId = taskId};
streamMetaWLock(pMeta);
code = streamMetaAcquireTaskUnsafe(pMeta, &id, &pTask);
if (code == 0) {
// desc the paused task counter
@ -958,10 +954,8 @@ int32_t streamMetaUnregisterTask(SStreamMeta* pMeta, int64_t streamId, int32_t t
}
streamMetaReleaseTask(pMeta, pTask);
streamMetaWUnLock(pMeta);
} else {
stDebug("vgId:%d failed to find the task:0x%x, it may have been dropped already", vgId, taskId);
streamMetaWUnLock(pMeta);
}
return 0;

19
source/libs/stream/src/streamSched.c
View File

@ -211,6 +211,7 @@ static int32_t doCreateForceWindowTrigger(SStreamTask* pTask, int32_t* pNextTrig
const char* id = pTask->id.idStr;
int8_t precision = pTask->info.interval.precision;
SStreamTrigger* pTrigger = NULL;
bool isFull = false;
while (1) {
code = streamCreateForcewindowTrigger(&pTrigger, pTask->info.delaySchedParam, &pTask->info.interval,
@ -233,7 +234,6 @@ static int32_t doCreateForceWindowTrigger(SStreamTask* pTask, int32_t* pNextTrig
// check whether the time window gaps exist or not
int64_t now = taosGetTimestamp(precision);
int64_t ekey = pTrigger->pBlock->info.window.skey + pTask->info.interval.interval;
// there are gaps, needs to be filled
STimeWindow w = pTrigger->pBlock->info.window;
@ -245,13 +245,18 @@ static int32_t doCreateForceWindowTrigger(SStreamTask* pTask, int32_t* pNextTrig
}
pTask->status.latestForceWindow = w;
if (ekey + pTask->info.watermark + pTask->info.interval.interval > now) {
int64_t prev = convertTimePrecision(*pNextTrigger, precision, TSDB_TIME_PRECISION_MILLI);
isFull = streamQueueIsFull(pTask->inputq.queue);
if ((w.ekey + pTask->info.watermark + pTask->info.interval.interval > now) || isFull) {
int64_t prev = convertTimePrecision(*pNextTrigger, precision, TSDB_TIME_PRECISION_MILLI);
if (!isFull) {
*pNextTrigger = w.ekey + pTask->info.watermark + pTask->info.interval.interval - now;
}
*pNextTrigger = ekey + pTask->info.watermark + pTask->info.interval.interval - now;
*pNextTrigger = convertTimePrecision(*pNextTrigger, precision, TSDB_TIME_PRECISION_MILLI);
stDebug("s-task:%s generate %d time window(s), trigger delay adjust from %" PRId64 " to %d", id, num, prev,
*pNextTrigger);
pTask->chkInfo.nextProcessVer = w.ekey + pTask->info.interval.interval;
stDebug("s-task:%s generate %d time window(s), trigger delay adjust from %" PRId64 " to %d, set ver:%" PRId64, id,
num, prev, *pNextTrigger, pTask->chkInfo.nextProcessVer);
return code;
} else {
stDebug("s-task:%s gap exist for force_window_close, current force_window_skey:%" PRId64, id, w.skey);
@ -308,7 +313,7 @@ void streamTaskSchedHelper(void* param, void* tmrId) {
}
if (streamTaskGetStatus(pTask).state == TASK_STATUS__CK) {
nextTrigger = TRIGGER_RECHECK_INTERVAL; // retry in 10 seec
nextTrigger = TRIGGER_RECHECK_INTERVAL; // retry in 10 sec
stDebug("s-task:%s in checkpoint procedure, not retrieve result, next:%dms", id, TRIGGER_RECHECK_INTERVAL);
} else {
if (pTask->info.trigger == STREAM_TRIGGER_FORCE_WINDOW_CLOSE && pTask->info.taskLevel == TASK_LEVEL__SOURCE) {

13
source/libs/stream/src/tstreamFileState.c
View File

@ -947,6 +947,7 @@ int32_t recoverSession(SStreamFileState* pFileState, int64_t ckId) {
}
SRowBuffPos* pPos = createSessionWinBuff(pFileState, &key, pVal, &vlen);
pPos->beUsed = false;
winRes = putSessionWinResultBuff(pFileState, pPos);
if (winRes != TSDB_CODE_SUCCESS) {
break;
@ -1008,6 +1009,7 @@ int32_t recoverSnapshot(SStreamFileState* pFileState, int64_t ckId) {
memcpy(pNewPos->pRowBuff, pVal, vlen);
taosMemoryFreeClear(pVal);
pNewPos->beFlushed = true;
pNewPos->beUsed = false;
qDebug("===stream=== read checkpoint state from disc. %s", __func__);
code = tSimpleHashPut(pFileState->rowStateBuff, pNewPos->pKey, pFileState->keyLen, &pNewPos, POINTER_BYTES);
if (code != TSDB_CODE_SUCCESS) {
@ -1090,6 +1092,7 @@ int32_t recoverFillSnapshot(SStreamFileState* pFileState, int64_t ckId) {
if (vlen != pFileState->rowSize) {
qError("row size mismatch, expect:%d, actual:%d", pFileState->rowSize, vlen);
destroyRowBuffPos(pNewPos);
code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
taosMemoryFreeClear(pVal);
QUERY_CHECK_CODE(code, lino, _end);
@ -1098,6 +1101,7 @@ int32_t recoverFillSnapshot(SStreamFileState* pFileState, int64_t ckId) {
memcpy(pNewPos->pRowBuff, pVal, vlen);
taosMemoryFreeClear(pVal);
pNewPos->beFlushed = true;
pNewPos->beUsed = false;
qDebug("===stream=== read checkpoint state from disc. %s", __func__);
winRes = tSimpleHashPut(pFileState->rowStateBuff, pNewPos->pKey, pFileState->keyLen, &pNewPos, POINTER_BYTES);
if (winRes != TSDB_CODE_SUCCESS) {
@ -1228,6 +1232,8 @@ void setFillInfo(SStreamFileState* pFileState) {
}
void clearExpiredState(SStreamFileState* pFileState) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SSHashObj* pSearchBuff = pFileState->searchBuff;
void* pIte = NULL;
int32_t iter = 0;
@ -1246,6 +1252,13 @@ void clearExpiredState(SStreamFileState* pFileState) {
}
taosArrayRemoveBatch(pWinStates, 0, size - 1, NULL);
}
code = clearRowBuff(pFileState);
QUERY_CHECK_CODE(code, lino, _end);
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
}
int32_t getStateSearchRowBuff(SStreamFileState* pFileState, const SWinKey* pKey, void** pVal, int32_t* pVLen,

28
source/os/src/osSocket.c
View File

@ -233,15 +233,19 @@ int32_t taosBlockSIGPIPE() {
}
int32_t taosGetIpv4FromFqdn(const char *fqdn, uint32_t *ip) {
int32_t code = 0;
OS_PARAM_CHECK(fqdn);
OS_PARAM_CHECK(ip);
int64_t limitMs = 1000;
int64_t st = taosGetTimestampMs(), cost = 0;
#ifdef WINDOWS
// Initialize Winsock
WSADATA wsaData;
int iResult;
iResult = WSAStartup(MAKEWORD(2, 2), &wsaData);
if (iResult != 0) {
return TAOS_SYSTEM_WINSOCKET_ERROR(WSAGetLastError());
code = TAOS_SYSTEM_WINSOCKET_ERROR(WSAGetLastError());
goto _err;
}
#endif
@ -260,12 +264,12 @@ int32_t taosGetIpv4FromFqdn(const char *fqdn, uint32_t *ip) {
inRetry = true;
continue;
} else if (EAI_SYSTEM == ret) {
terrno = TAOS_SYSTEM_ERROR(errno);
return terrno;
code = TAOS_SYSTEM_ERROR(errno);
goto _err;
}
terrno = TAOS_SYSTEM_ERROR(errno);
return terrno;
code = TAOS_SYSTEM_ERROR(errno);
goto _err;
}
struct sockaddr *sa = result->ai_addr;
@ -275,8 +279,7 @@ int32_t taosGetIpv4FromFqdn(const char *fqdn, uint32_t *ip) {
*ip = ia.s_addr;
freeaddrinfo(result);
return 0;
goto _err;
}
#else
struct addrinfo hints = {0};
@ -292,7 +295,7 @@ int32_t taosGetIpv4FromFqdn(const char *fqdn, uint32_t *ip) {
struct in_addr ia = si->sin_addr;
*ip = ia.s_addr;
freeaddrinfo(result);
return 0;
goto _err;
} else {
#ifdef EAI_SYSTEM
if (ret == EAI_SYSTEM) {
@ -305,9 +308,16 @@ int32_t taosGetIpv4FromFqdn(const char *fqdn, uint32_t *ip) {
#endif
*ip = 0xFFFFFFFF;
return TSDB_CODE_RPC_FQDN_ERROR;
code = TSDB_CODE_RPC_FQDN_ERROR;
goto _err;
}
#endif
_err:
cost = taosGetTimestampMs() - st;
if (cost >= limitMs) {
uWarn("get ip from fqdn:%s, cost:%" PRId64 "ms", fqdn, cost);
}
return code;
}
int32_t taosGetFqdn(char *fqdn) {

2
tests/ci/container_build_newmachine.sh
View File

@ -80,7 +80,7 @@ else
fi
mv ${REP_REAL_PATH}/debug ${WORKDIR}/debugSan
rm -rf ${REP_REAL_PATH}/debug
date
ret=$?

1
tests/run_all_ci_cases.sh
View File

@ -27,7 +27,6 @@ function runCasesOneByOne () {
|| echo -e "${RED}$case failed${NC}" | tee -a $TDENGINE_ALLCI_REPORT
end_time=`date +%s`
echo execution time of $case was `expr $end_time - $start_time`s. | tee -a $TDENGINE_ALLCI_REPORT
if $case failed
elif [[ "$line" == *"$2"* ]]; then
if [[ "$cmd" == *"pytest.sh"* ]]; then

58
tests/script/tsim/stream/streamTwaFwcInterval.sim
View File

@ -289,6 +289,64 @@ if $data51 != $query1_data51 then
goto loop3
endi
print ======step3
sql create database test3 vgroups 1;
sql use test3;
sql create stable st(ts timestamp, a int, b int , c int)tags(ta int,tb int,tc int);
sql create table t1 using st tags(1,1,1);
sql create table t2 using st tags(2,2,2);
sql create stream streams3 trigger force_window_close IGNORE EXPIRED 1 IGNORE UPDATE 1 into streamt3 as select _wstart, twa(a), ta from st partition by tbname,ta interval(10s);
run tsim/stream/checkTaskStatus.sim
sql insert into t1 values(now + 3000a,1,1,1);
sql flush database test;
sql insert into t1 values(now + 3001a,10,10,10);
sql insert into t1 values(now + 13s,50,50,50);
sleep 1000
print sql select _wstart, twa(a), ta from st partition by tbname,ta interval(10s) order by 1;
sql select _wstart, twa(a), ta from st partition by tbname,ta interval(10s) order by 1;
$query_data01 = $data01
print $data00 $data01 $data02 $data03 $data04
print $data10 $data11 $data12 $data13 $data14
print $data20 $data21 $data22 $data23 $data24
print $data30 $data31 $data32 $data33 $data34
print $data40 $data41 $data42 $data43 $data44
print $data50 $data51 $data52 $data53 $data54
$loop_count = 0
loop4:
sleep 2000
$loop_count = $loop_count + 1
if $loop_count == 20 then
return -1
endi
print 2 sql select * from streamt3 order by 1;
sql select * from streamt3 order by 1;
print $data00 $data01 $data02 $data03 $data04
print $data10 $data11 $data12 $data13 $data14
print $data20 $data21 $data22 $data23 $data24
print $data30 $data31 $data32 $data33 $data34
print $data40 $data41 $data42 $data43 $data44
print $data50 $data51 $data52 $data53 $data54
if $data01 != $query_data01 then
print ======data01======$data01
print ====query_data01=$query_data01
goto loop4
endi
print end
system sh/exec.sh -n dnode1 -s stop -x SIGINT

12
tests/system-test/2-query/and_or_for_byte.py
View File

@ -515,6 +515,18 @@ class TDTestCase:
tdSql.query(f"select t1 from {dbname}.stb1 where abs(c1+t1)=1")
tdSql.checkRows(1)
tdSql.checkData(0,0,0)
tdSql.query(f"select * from {dbname}.stb1")
rows = tdSql.queryRows
tdSql.query(f"select t1 from {dbname}.stb1 where abs(c1+t1)=1 or (1<2)")
tdSql.checkRows(rows)
tdSql.query(f"select t1 from {dbname}.stb1 where abs(c1+t1)=1 and (1<2)")
tdSql.checkRows(1)
tdSql.checkData(0,0,0)
tdSql.query(f"select t1 from {dbname}.stb1 where abs(c1+t1)=1 and (1>2)")
tdSql.checkRows(0)
tdSql.query(f"select t1 from {dbname}.stb1 where abs(c1+t1)=1 or (1>2)")
tdSql.checkRows(1)
tdSql.checkData(0,0,0)
tdSql.query(
f"select abs(c1+t1)*t1 from {dbname}.stb1 where abs(c1)/floor(abs(ceil(t1))) ==1")

69
tests/system-test/2-query/group_partition.py
View File

@ -452,6 +452,73 @@ class TDTestCase:
tdSql.query(sql, queryTimes=1)
tdSql.checkRows(5)
def test_TS_5727(self):
tdSql.execute(f" use {self.dbname} ")
stableName = "test5727"
sql = f"CREATE STABLE {self.dbname}.{stableName} (`ts` TIMESTAMP, `WaterConsumption` FLOAT, \
`ElecConsumption` INT, `Status` BOOL, `status2` BOOL, `online` BOOL) \
TAGS (`ActivationTime` TIMESTAMP, `ProductId` INT, \
`ProductMac` VARCHAR(24), `location` INT)"
tdSql.execute(sql)
sql = f'CREATE TABLE {self.dbname}.`d00` USING {self.dbname}.{stableName} \
(`ActivationTime`, `ProductId`, `ProductMac`, `location`) \
TAGS (1733124710578, 1001, "00:11:22:33:44:55", 100000)'
tdSql.execute(sql)
sql = f'CREATE TABLE {self.dbname}.`d01` USING {self.dbname}.{stableName} \
(`ActivationTime`, `ProductId`, `ProductMac`, `location`) \
TAGS (1733124723572, 1002, "00:12:22:33:44:55", 200000)'
tdSql.execute(sql)
sql = f'CREATE TABLE {self.dbname}.`d02` USING {self.dbname}.{stableName} \
(`ActivationTime`, `ProductId`, `ProductMac`, `location`) \
TAGS (1733124730908, 1003, "00:11:23:33:44:55", 100000)'
tdSql.execute(sql)
sql = f'insert into {self.dbname}.d00 values(now - 2s, 5, 5, true, true, false);'
tdSql.execute(sql)
sql = f'insert into {self.dbname}.d01 values(now - 1s, 6, 5, true, true, true);'
tdSql.execute(sql)
sql = f'insert into {self.dbname}.d02 values(now, 6, 7, true, true, true);'
tdSql.execute(sql)
sql = f'select `location`, tbname from {self.dbname}.{stableName} where ts < now group by tbname order by tbname;'
tdSql.query(sql)
tdSql.checkRows(3)
tdSql.checkData(0, 0, 100000)
tdSql.checkData(1, 0, 200000)
tdSql.checkData(2, 0, 100000)
tdSql.checkData(0, 1, "d00")
tdSql.checkData(1, 1, "d01")
tdSql.checkData(2, 1, "d02")
sql = f'select tbname,last(online) as online,location from {self.dbname}.{stableName} where ts < now group by tbname order by tbname;'
tdSql.query(sql)
tdSql.checkRows(3)
tdSql.checkData(0, 0, "d00")
tdSql.checkData(1, 0, "d01")
tdSql.checkData(2, 0, "d02")
tdSql.checkData(0, 1, False)
tdSql.checkData(1, 1, True)
tdSql.checkData(2, 1, True)
tdSql.checkData(0, 2, 100000)
tdSql.checkData(1, 2, 200000)
tdSql.checkData(2, 2, 100000)
sql = f'select location,tbname,last_row(online) as online from {self.dbname}.{stableName} where ts < now group by tbname order by tbname;'
tdSql.query(sql)
tdSql.checkRows(3)
tdSql.checkData(0, 0, 100000)
tdSql.checkData(1, 0, 200000)
tdSql.checkData(2, 0, 100000)
tdSql.checkData(0, 1, "d00")
tdSql.checkData(1, 1, "d01")
tdSql.checkData(2, 1, "d02")
tdSql.checkData(0, 2, False)
tdSql.checkData(1, 2, True)
tdSql.checkData(2, 2, True)
def run(self):
tdSql.prepare()
self.prepare_db()
@ -494,6 +561,8 @@ class TDTestCase:
self.test_error()
self.test_TS_5727()
def stop(self):
tdSql.close()

4
tests/system-test/2-query/max_partition.py
View File

@ -64,6 +64,10 @@ class TDTestCase:
tdSql.query(f"select tbname , max(c1) from {dbname}.sub_stb_1 where c1 is null group by c1 order by c1 desc ")
tdSql.checkRows(1)
tdSql.checkData(0,0,"sub_stb_1")
tdSql.query(f"select tbname , max(c1) from {dbname}.sub_stb_1 group by c1 order by c1 desc ")
rows = tdSql.queryRows
tdSql.query(f"select tbname , max(c1) from {dbname}.sub_stb_1 where c1 is null or (1<2) group by c1 order by c1 desc ")
tdSql.checkRows(rows)
tdSql.query(f"select max(c1) ,c2 ,t2,tbname from {dbname}.stb group by abs(c1) order by abs(c1)")
tdSql.checkRows(self.row_nums+1)

21
tests/system-test/2-query/normal.py
View File

@ -75,6 +75,25 @@ class TDTestCase:
tdSql.checkData(1, 1, 3)
tdSql.checkData(2, 1, 9)
tdSql.query(f"select * from {dbname}.stb_1 order by ts")
rows = tdSql.queryRows
tdSql.query(f"select * from {dbname}.stb_1 where col1 in (1, 2) or (1<2) order by ts")
tdSql.checkRows(rows)
tdSql.query(f"select * from (select * from {dbname}.stb_1 where col1 in (1, 9, 3) or (1<2) order by ts)")
tdSql.checkRows(rows)
tdSql.query(f"select * from {dbname}.stb_1 where col1 in (1, 2) and (1<2) order by ts")
tdSql.checkRows(2)
tdSql.checkData(0, 1, 1)
tdSql.checkData(1, 1, 2)
tdSql.query(f"select * from {dbname}.stb_1 where col1 in (1, 9, 3) and (1<2) order by ts")
tdSql.checkRows(3)
tdSql.checkData(0, 1, 1)
tdSql.checkData(1, 1, 3)
tdSql.checkData(2, 1, 9)
tdSql.query(f"select * from {dbname}.stb_1 where col1 in (1, 9, 3, 'xy') order by ts")
tdSql.checkRows(3)
tdSql.checkData(0, 1, 1)
@ -160,7 +179,7 @@ class TDTestCase:
tdSql.prepare()
self.timeZoneTest()
# self.inAndNotinTest()
self.inAndNotinTest()
def stop(self):

9
tests/system-test/2-query/not.py
View File

@ -28,6 +28,10 @@ class TDTestCase:
for type_name in stype:
tdsql.execute(f"drop table if exists {dbname}.{stbname}")
tdsql.execute(f"create table if not exists {dbname}.{stbname} (ts timestamp, v1 {type_name}) tags(t1 {type_name})")
tdsql.query(f"select t1, * from {dbname}.{stbname} where t1 not in (1, 2) or (1<2) order by t1")
tdsql.checkRows(0)
tdsql.execute(f"insert into {dbname}.sub_1 using {dbname}.{stbname} tags(1) values({self.ts}, 10)")
tdsql.execute(f"insert into {dbname}.sub_2 using {dbname}.{stbname} tags(2) values({self.ts + 1000}, 20)")
tdsql.execute(f"insert into {dbname}.sub_3 using {dbname}.{stbname} tags(3) values({self.ts + 2000}, 30)")
@ -36,6 +40,11 @@ class TDTestCase:
tdsql.query(f"select t1, * from {dbname}.{stbname} where t1 not in (1, 2) order by t1")
tdsql.checkRows(1)
tdsql.checkData(0, 0, 3)
tdsql.query(f"select * from {dbname}.{stbname} where t1 not in (1, 2) or (1<2) order by t1")
tdsql.checkRows(3)
tdsql.checkData(0, 1, 10)
tdsql.checkData(1, 1, 20)
tdsql.checkData(2, 1, 30)
# Test case 2: NOT BETWEEN
tdsql.query(f"select * from {dbname}.{stbname} where v1 not between 10 and 20 order by t1")

9
tests/system-test/2-query/union.py
View File

@ -426,6 +426,15 @@ class TDTestCase:
tdLog.printNoPrefix("==========step4:after wal, all check again ")
self.all_test()
self.test_TD_33137()
def test_TD_33137(self):
sql = "select 'asd' union all select 'asdasd'"
tdSql.query(sql, queryTimes=1)
tdSql.checkRows(2)
sql = "select db_name `TABLE_CAT`, '' `TABLE_SCHEM`, stable_name `TABLE_NAME`, 'TABLE' `TABLE_TYPE`, table_comment `REMARKS` from information_schema.ins_stables union all select db_name `TABLE_CAT`, '' `TABLE_SCHEM`, table_name `TABLE_NAME`, case when `type`='SYSTEM_TABLE' then 'TABLE' when `type`='NORMAL_TABLE' then 'TABLE' when `type`='CHILD_TABLE' then 'TABLE' else 'UNKNOWN' end `TABLE_TYPE`, table_comment `REMARKS` from information_schema.ins_tables union all select db_name `TABLE_CAT`, '' `TABLE_SCHEM`, view_name `TABLE_NAME`, 'VIEW' `TABLE_TYPE`, NULL `REMARKS` from information_schema.ins_views"
tdSql.query(sql, queryTimes=1)
tdSql.checkRows(47)
def stop(self):
tdSql.close()