Merge branch 'main' into merge/mainto3.0

docs/en/10-third-party/05-bi/11-superset.md

@@ -7,7 +7,7 @@ Apache Superset is a modern enterprise level business intelligence (BI) web appl
 It is supported by the Apache Software Foundation and is an open source project with an active community and rich ecosystem.
 Apache Superset provides an intuitive user interface that makes creating, sharing, and visualizing data simple, while supporting multiple data sources and rich visualization options.  
 
-Through the Python connector of TDengine, Superset can support TDengine data sources and provide functions such as data presentation and analysis  
+Through the Python connector of TDengine, Superset can support TDengine data sources and provide functions such as data presentation and analysis.  
 
 ## Prerequisites
 

docs/en/10-third-party/05-bi/12-tableau.md

@@ -10,8 +10,8 @@ Tableau is a well-known business intelligence tool that supports multiple data s
 
 Prepare the following environment:
 
-- TDengine 3.3.5.4 and above version is installed and running normally (both Enterprise and Community versions are available)
-- taosAdapter is running normally, refer to [taosAdapter Reference](../../../tdengine-reference/components/taosadapter/)
+- TDengine 3.3.5.8 and above version is installed and running normally (both Enterprise and Community versions are available).
+- taosAdapter is running normally, refer to [taosAdapter Reference](../../../tdengine-reference/components/taosadapter/).
 - Install and run Tableau Desktop (if not installed, please download and install Windows operating system 64-bit [Download Tableau Desktop](https://www.tableau.com/products/desktop/download)). Install Tableau please refer to [Tableau Desktop](https://www.tableau.com).
 - Download the latest Windows operating system X64 client driver from the TDengine official website and install it, refer to [Install ODBC Driver](../../../tdengine-reference/client-libraries/odbc/#Installation).
 
@@ -19,6 +19,10 @@ Prepare the following environment:
 
 **Step 1**, Search and open the "ODBC Data Source (64 bit)" management tool in the Start menu of the Windows operating system and configure it, refer to [Install ODBC Driver](../../../tdengine-reference/client-libraries/odbc/#Installation).
 
+:::tip
+It should be noted that when configuring the ODBC data source for Tableau, the [Database] configuration item on the TDengine ODBC data source configuration page is required. You need to select a database that can be successfully connected.
+:::
+
 **Step 2**, Start Tableau in the Windows system environment, then search for "ODBC" on its connection page and select "Other Databases (ODBC)".
 
 **Step 3**, Click the `DSN` radio button, then select the configured data source (MyTDengine), and click the `Connect` button. After the connection is successful, delete the content of the string attachment, and finally click the `Sign In` button.  

docs/en/10-third-party/05-bi/13-excel.md

@@ -10,7 +10,7 @@ toc_max_heading_level: 4
 
 Prepare the following environment:
 
-- TDengine 3.3.5.7 and above version is installed and running normally (both Enterprise and Community versions are available).
+- TDengine 3.3.5.8 and above version is installed and running normally (both Enterprise and Community versions are available).
 - taosAdapter is running normally, refer to [taosAdapter Reference](../../../tdengine-reference/components/taosadapter/).
 - Install and run Excel. If not installed, please download and install it. For specific instructions, please refer to Microsoft's official documentation.
 - Download the latest Windows operating system X64 client driver from the TDengine official website and install it, refer to [Install ODBC Driver](../../../tdengine-reference/client-libraries/odbc/#Installation).

docs/zh/08-operation/09-backup.md

@@ -76,6 +76,17 @@ taosExplorer 服务页面中，进入“系统管理 - 备份”页面，在“
 8. 备份文件大小：备份文件的大小限制。当备份文件大小达到此限制时，会自动创建新的备份文件。
 9. 文件压缩等级：备份文件的压缩等级。支持：最快速度、最佳压缩比、兼具速度和压缩比。
 
+用户可以通过开启 S3 转储，将备份文件上传至 S3 存储服务上。开启 S3 转储，需要填写以下信息：
+
+1. S3 节点：S3 节点的地址。
+2. 访问密钥 ID：访问密钥 ID。
+3. 访问密钥：访问密钥。
+4. 存储桶：存储桶名称。
+5. 区域：存储桶所在的区域。
+6. 对象前缀：备份文件的对象前缀，类似于 S3 上的目录。
+7. 本地备份文件的保留时长：本地备份的保留时间，所有早于`当前时间 - backup_retention_period`的文件都需要上传到 S3。
+8. 本地备份文件的保留个数：本地备份文件的保留个数，本地只保留最新的`backup_retention_size`个备份文件。
+
 创建成功后，备份计划会开始按照配置的参数运行。在“备份计划”下的列表中，可以查看已创建的备份计划。
 
 备份计划支持以下操作：

docs/zh/10-third-party/05-bi/12-tableau.md

@@ -8,8 +8,8 @@ Tableau 是一款知名的商业智能工具，它支持多种数据源，可方
 ## 前置条件
 
 准备以下环境：
-- TDengine 3.3.5.4 以上版本集群已部署并正常运行（企业及社区版均可）
-- taosAdapter 能够正常运行。详细参考 [taosAdapter 参考手册](../../../reference/components/taosadapter)
+- TDengine 3.3.5.8 以上版本集群已部署并正常运行（企业及社区版均可）。
+- taosAdapter 能够正常运行。详细参考 [taosAdapter 参考手册](../../../reference/components/taosadapter)。
 - Tableau 桌面版安装并运行（如未安装，请下载并安装 Windows 操作系统 64 位 [Tableau 桌面版](https://www.tableau.com/products/desktop/download) ）。安装 Tableau 桌面版请参考 [官方文档](https://www.tableau.com)。
 - 从 TDengine 官网下载最新的 Windows 操作系统 X64 客户端驱动程序，并进行安装。详细参考 [安装 ODBC 驱动](../../../reference/connector/odbc/#安装)。
 
@@ -18,7 +18,11 @@ Tableau 是一款知名的商业智能工具，它支持多种数据源，可方
 
 **第 1 步**，在Windows操作系统的开始菜单中搜索并打开“ODBC数据源（64位）”管理工具并进行配置。详细参考[配置ODBC数据源](../../../reference/connector/odbc/#配置数据源)。
 
-**第 2 步**，在 Windows 系统环境下启动 Tableau，之后在其连接页面中搜索 “ODBC”，并选择 “其他数据库 (ODBC)”。 
+:::tip
+需要注意的是，在为 Tableau 配置 ODBC 数据源时，TDengine ODBC 数据源配置页面中的【数据库】配置项为必填项，需选择一个可成功连接的数据库。
+:::
+
+**第 2 步**，在 Windows 系统环境下启动 Tableau，之后在其连接页面中搜索 “ODBC”，并选择 “其他数据库 (ODBC)”。 对于 Tableau 的使用的ODBC数据源，在其 TDengine ODBC 数据源配置页面的【数据库】的配置项为必填，需要选择可以连接的数据库。
  
 **第 3 步**，点击 `DSN` 单选框，接着选择已配置好的数据源(MyTDengine)，然后点击`连接`按钮。待连接成功后，删除字符串附加部分的内容，最后点击`登录`按钮即可。
 

docs/zh/10-third-party/05-bi/13-excel.md

@@ -8,7 +8,7 @@ title: 与 Excel 集成
 ## 前置条件
 
 准备以下环境：
-- TDengine 3.3.5.7 以上版本集群已部署并正常运行（企业及社区版均可）。
+- TDengine 3.3.5.8 以上版本集群已部署并正常运行（企业及社区版均可）。
 - taosAdapter 能够正常运行，详细参考 [taosAdapter 参考手册](../../../reference/components/taosadapter)。
 - Excel 安装并运行, 如未安装，请下载并安装, 具体操作请参考 Microsoft 官方文档。
 - 从 TDengine 官网下载最新的 Windows 操作系统 X64 客户端驱动程序并进行安装，详细参考 [安装 ODBC 驱动](../../../reference/connector/odbc/#安装)。

include/libs/stream/tstream.h

@@ -805,6 +805,7 @@ void    streamMetaClearSetUpdateTaskListComplete(SStreamMeta* pMeta);
 bool    streamMetaInitUpdateTaskList(SStreamMeta* pMeta, int32_t transId);
 
 void    streamMetaRLock(SStreamMeta* pMeta);
+int32_t streamMetaTryRlock(SStreamMeta* pMeta);
 void    streamMetaRUnLock(SStreamMeta* pMeta);
 void    streamMetaWLock(SStreamMeta* pMeta);
 void    streamMetaWUnLock(SStreamMeta* pMeta);

packaging/docker/Dockerfile

@@ -15,8 +15,10 @@ ENV DEBIAN_FRONTEND=noninteractive
 ADD https://github.com/krallin/tini/releases/download/${TINI_VERSION}/tini-${cpuType} /tini
 RUN chmod +x /tini
 
-RUN tar -zxf ${pkgFile} && \
+RUN tar -zxf /root/${pkgFile} && \
+    cd /root/${dirName}/ && \
     /bin/bash /root/${dirName}/install.sh -e no && \
+    cd /root/ && \
     rm /root/${pkgFile} && \
     rm -rf /root/${dirName} && \
     apt-get update && \
@@ -46,4 +48,4 @@ COPY ./bin/* /usr/bin/
 ENTRYPOINT ["/tini", "--", "/usr/bin/entrypoint.sh"]
 CMD ["taosd"]
 
-VOLUME [ "/var/lib/taos", "/var/log/taos", "/corefile" ]
+VOLUME [ "/var/lib/taos", "/var/log/taos", "/corefile" ]

source/dnode/mnode/impl/src/mndConfig.c

@@ -299,6 +299,8 @@ _OVER:
   }
   sdbRelease(pMnode->pSdb, vObj);
   cfgArrayCleanUp(array);
+
+  tFreeSConfigReq(&configReq);
   return code;
 }
 

source/dnode/vnode/src/tq/tqSink.c

@@ -1428,6 +1428,7 @@ int32_t handleResultBlockMsg(SStreamTask* pTask, SSDataBlock* pDataBlock, int32_
   code = setDstTableDataUid(pVnode, pTask, pDataBlock, stbFullName, &tbData);
   if (code != TSDB_CODE_SUCCESS) {
     tqError("vgId:%d s-task:%s dst-table not exist, stb:%s discard stream results", vgId, id, stbFullName);
+    tDestroySubmitReq(&submitReq, TSDB_MSG_FLG_ENCODE);
     return code;
   }
 
@@ -1439,12 +1440,14 @@ int32_t handleResultBlockMsg(SStreamTask* pTask, SSDataBlock* pDataBlock, int32_
       tbData.pCreateTbReq = NULL;
     }
 
+    tDestroySubmitReq(&submitReq, TSDB_MSG_FLG_ENCODE);
     return code;
   }
 
   void* p = taosArrayPush(submitReq.aSubmitTbData, &tbData);
   if (p == NULL) {
     tqDebug("vgId:%d, s-task:%s failed to build submit msg, code:%s, data lost", vgId, id, tstrerror(terrno));
+    tDestroySubmitReq(&submitReq, TSDB_MSG_FLG_ENCODE);
     return terrno;
   }
 

source/dnode/vnode/src/tq/tqStreamTask.c

@@ -148,9 +148,13 @@ static void doStartScanWal(void* param, void* tmrId) {
     return;
   }
 
-  streamMetaRLock(pMeta);
-  numOfTasks = taosArrayGetSize(pMeta->pTaskList);
-  streamMetaRUnLock(pMeta);
+  code = streamMetaTryRlock(pMeta);
+  if (code == 0) {
+    numOfTasks = taosArrayGetSize(pMeta->pTaskList);
+    streamMetaRUnLock(pMeta);
+  } else {
+    numOfTasks = 0;
+  }
 
   if (numOfTasks == 0) {
     goto _end;
@@ -169,7 +173,6 @@ static void doStartScanWal(void* param, void* tmrId) {
   }
 
 _end:
-
   streamTmrStart(doStartScanWal, SCAN_WAL_IDLE_DURATION, pParam, pTimer, &pMeta->scanInfo.scanTimer, vgId, "scan-wal");
   tqDebug("vgId:%d scan-wal will start in %dms", vgId, SCAN_WAL_IDLE_DURATION*SCAN_WAL_WAIT_COUNT);
 

source/libs/stream/src/streamCheckpoint.c

@@ -1585,10 +1585,13 @@ int32_t streamTaskSendNegotiateChkptIdMsg(SStreamTask* pTask) {
     pTask->pBackend = NULL;
   }
 
+  streamMetaWLock(pTask->pMeta);
   if (pTask->exec.pExecutor != NULL) {
     qDestroyTask(pTask->exec.pExecutor);
     pTask->exec.pExecutor = NULL;
   }
+  streamMetaWUnLock(pTask->pMeta);
+
   return 0;
 }
 

source/libs/stream/src/streamHb.c

@@ -363,13 +363,24 @@ void streamMetaHbToMnode(void* param, void* tmrId) {
     pMeta->pHbInfo->hbStart = taosGetTimestampMs();
   }
 
-  streamMetaRLock(pMeta);
-  code = streamMetaSendHbHelper(pMeta);
-  if (code) {
-    stError("vgId:%d failed to send hmMsg to mnode, try again in 5s, code:%s", pMeta->vgId, tstrerror(code));
+  // NOTE: stream task in restart procedure. not generate the hb now, try to acquire the lock may cause stuck this timer.
+  int32_t count = 30;
+  bool    send = false;
+  while ((--count) >= 0) {
+    int32_t ret = streamMetaTryRlock(pMeta);
+    if (ret != 0) {
+      taosMsleep(10);
+    } else {
+      send = true;
+      code = streamMetaSendHbHelper(pMeta);
+      streamMetaRUnLock(pMeta);
+      break;
+    }
   }
 
-  streamMetaRUnLock(pMeta);
+  if (!send) {
+    stError("vgId:%d failed to send hmMsg to mnode, retry again in 5s, code:%s", pMeta->vgId, tstrerror(code));
+  }
 
   streamTmrStart(streamMetaHbToMnode, META_HB_CHECK_INTERVAL, param, streamTimer, &pMeta->pHbInfo->hbTmr, pMeta->vgId,
                  "meta-hb-tmr");

source/libs/stream/src/streamQueue.c

@@ -144,21 +144,11 @@ void streamQueueNextItemInSourceQ(SStreamQueue* pQueue, SStreamQueueItem** pItem
 
   // let's try the ordinary input q
   pQueue->qItem = NULL;
-  int32_t code = taosGetQitem(pQueue->qall, &pQueue->qItem);
-  if (code) {
-    stError("s-task:%s failed to get item in inputq, code:%s", id, tstrerror(code));
-  }
+  int32_t num = taosGetQitem(pQueue->qall, &pQueue->qItem);
 
   if (pQueue->qItem == NULL) {
-    code = taosReadAllQitems(pQueue->pQueue, pQueue->qall);
-    if (code) {
-      stError("s-task:%s failed to get all items in inputq, code:%s", id, tstrerror(code));
-    }
-
-    code = taosGetQitem(pQueue->qall, &pQueue->qItem);
-    if (code) {
-      stError("s-task:%s failed to get item in inputq, code:%s", id, tstrerror(code));
-    }
+    num = taosReadAllQitems(pQueue->pQueue, pQueue->qall);
+    num = taosGetQitem(pQueue->qall, &pQueue->qItem);
   }
 
   *pItem = streamQueueCurItem(pQueue);

source/libs/stream/src/streamUtil.c

@@ -54,6 +54,15 @@ void streamMetaRUnLock(SStreamMeta* pMeta) {
   }
 }
 
+int32_t streamMetaTryRlock(SStreamMeta* pMeta) {
+  int32_t code = taosThreadRwlockTryRdlock(&pMeta->lock);
+  if (code) {
+    stError("vgId:%d try meta-rlock failed, code:%s", pMeta->vgId, tstrerror(code));
+  }
+
+  return code;
+}
+
 void streamMetaWLock(SStreamMeta* pMeta) {
   //  stTrace("vgId:%d meta-wlock", pMeta->vgId);
   int32_t code = taosThreadRwlockWrlock(&pMeta->lock);