From b1e9c7e6e6c145f78efe1de591d0e1a1a7a4719f Mon Sep 17 00:00:00 2001 From: sheyanjie-qq <249478495@qq.com> Date: Wed, 25 Dec 2024 16:43:09 +0800 Subject: [PATCH 01/73] update jdbc stmt2 doc and sample code --- docs/en/07-develop/01-connect.md | 2 +- docs/en/07-develop/05-stmt.md | 9 +- docs/en/14-reference/05-connector/14-java.md | 155 +++++++++--------- docs/examples/JDBC/JDBCDemo/pom.xml | 2 +- docs/examples/JDBC/SpringJdbcTemplate/pom.xml | 2 +- docs/examples/JDBC/connectionPools/pom.xml | 2 +- docs/examples/JDBC/consumer-demo/pom.xml | 2 +- docs/examples/JDBC/mybatisplus-demo/pom.xml | 2 +- docs/examples/JDBC/springbootdemo/pom.xml | 2 +- docs/examples/JDBC/taosdemo/pom.xml | 2 +- docs/examples/java/pom.xml | 2 +- ...WSParameterBindingExtendInterfaceDemo.java | 87 ++++++++++ ...> WSParameterBindingStdInterfaceDemo.java} | 38 ++--- .../src/test/java/com/taos/test/TestAll.java | 9 +- docs/zh/07-develop/01-connect/index.md | 2 +- docs/zh/07-develop/05-stmt.md | 8 +- docs/zh/14-reference/05-connector/14-java.mdx | 1 + 17 files changed, 215 insertions(+), 112 deletions(-) create mode 100644 docs/examples/java/src/main/java/com/taos/example/WSParameterBindingExtendInterfaceDemo.java rename docs/examples/java/src/main/java/com/taos/example/{WSParameterBindingBasicDemo.java => WSParameterBindingStdInterfaceDemo.java} (61%) diff --git a/docs/en/07-develop/01-connect.md b/docs/en/07-develop/01-connect.md index af5f171f8c..9b1fbad6dd 100644 --- a/docs/en/07-develop/01-connect.md +++ b/docs/en/07-develop/01-connect.md @@ -109,7 +109,7 @@ If you are using Maven to manage your project, simply add the following dependen com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 ``` diff --git a/docs/en/07-develop/05-stmt.md b/docs/en/07-develop/05-stmt.md index 4503bb8bd3..4e49145628 100644 --- a/docs/en/07-develop/05-stmt.md +++ b/docs/en/07-develop/05-stmt.md @@ -28,8 +28,15 @@ Next, we continue to use smart meters as an example to demonstrate the efficient + +There are two kinds of interfaces for parameter binding: one is the standard JDBC interface, and the other is an extended interface. The extended interface offers better performance. + ```java -{{#include docs/examples/java/src/main/java/com/taos/example/WSParameterBindingBasicDemo.java:para_bind}} +{{#include docs/examples/java/src/main/java/com/taos/example/WSParameterBindingStdInterfaceDemo.java:para_bind}} +``` + +```java +{{#include docs/examples/java/src/main/java/com/taos/example/WSParameterBindingExtendInterfaceDemo.java:para_bind}} ``` This is a [more detailed parameter binding example](https://github.com/taosdata/TDengine/blob/main/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingFullDemo.java) diff --git a/docs/en/14-reference/05-connector/14-java.md b/docs/en/14-reference/05-connector/14-java.md index 10e4ec6d42..f52b7e71f7 100644 --- a/docs/en/14-reference/05-connector/14-java.md +++ b/docs/en/14-reference/05-connector/14-java.md @@ -30,33 +30,34 @@ The JDBC driver implementation for TDengine strives to be consistent with relati ## Version History -| taos-jdbcdriver Version | Major Changes | TDengine Version | -| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -| 3.4.0 | 1. Replaced fastjson library with jackson.
2. WebSocket uses a separate protocol identifier.
3. Optimized background thread usage to avoid user misuse leading to timeouts. | - | -| 3.3.4 | Fixed getInt error when data type is float. | - | -| 3.3.3 | Fixed memory leak caused by closing WebSocket statement. | - | -| 3.3.2 | 1. Optimized parameter binding performance under WebSocket connection.
2. Improved support for mybatis. | - | -| 3.3.0 | 1. Optimized data transmission performance under WebSocket connection.
2. Supports skipping SSL verification, off by default. | 3.3.2.0 and higher | -| 3.2.11 | Fixed a bug in closing result set in Native connection. | - | -| 3.2.10 | 1. REST/WebSocket connections support data compression during transmission.
2. WebSocket automatic reconnection mechanism, off by default.
3. Connection class provides methods for schemaless writing.
4. Optimized data fetching performance for native connections.
5. Fixed some known issues.
6. Metadata retrieval functions can return a list of supported functions. | - | -| 3.2.9 | Fixed bug in closing WebSocket prepareStatement. | - | -| 3.2.8 | 1. Optimized auto-commit.
2. Fixed manual commit bug in WebSocket.
3. Optimized WebSocket prepareStatement using a single connection.
4. Metadata supports views. | - | -| 3.2.7 | 1. Supports VARBINARY and GEOMETRY types.
2. Added timezone setting support for native connections.
3. Added WebSocket automatic reconnection feature. | 3.2.0.0 and higher | -| 3.2.5 | Data subscription adds committed() and assignment() methods. | 3.1.0.3 and higher | -| 3.2.4 | Data subscription adds enable.auto.commit parameter under WebSocket connection, as well as unsubscribe() method. | - | -| 3.2.3 | Fixed ResultSet data parsing failure in some cases. | - | -| 3.2.2 | New feature: Data subscription supports seek function. | 3.0.5.0 and higher | -| 3.2.1 | 1. WebSocket connection supports schemaless and prepareStatement writing.
2. Consumer poll returns result set as ConsumerRecord, which can be accessed through value() method. | 3.0.3.0 and higher | -| 3.2.0 | Connection issues, not recommended for use. | - | -| 3.1.0 | WebSocket connection supports subscription function. | - | -| 3.0.1 - 3.0.4 | Fixed data parsing errors in result sets under some conditions. 3.0.1 compiled in JDK 11 environment, other versions recommended for JDK 8. | - | -| 3.0.0 | Supports TDengine 3.0 | 3.0.0.0 and higher | -| 2.0.42 | Fixed wasNull interface return value in WebSocket connection. | - | -| 2.0.41 | Fixed username and password encoding method in REST connection. | - | -| 2.0.39 - 2.0.40 | Added REST connection/request timeout settings. | - | -| 2.0.38 | JDBC REST connection adds batch fetching function. | - | -| 2.0.37 | Added support for json tag. | - | -| 2.0.36 | Added support for schemaless writing. | - | +| taos-jdbcdriver Version | Major Changes | TDengine Version | +| ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | +| 3.5.0 | 1. Optimized the performance of WebSocket connection parameter binding, supporting parameter binding queries using binary data.
2. Optimized the performance of small queries in WebSocket connection.
3. Added support for setting time zone on WebSocket connection. | 3.3.5.0 and higher | +| 3.4.0 | 1. Replaced fastjson library with jackson.
2. WebSocket uses a separate protocol identifier.
3. Optimized background thread usage to avoid user misuse leading to timeouts. | - | +| 3.3.4 | Fixed getInt error when data type is float. | - | +| 3.3.3 | Fixed memory leak caused by closing WebSocket statement. | - | +| 3.3.2 | 1. Optimized parameter binding performance under WebSocket connection.
2. Improved support for mybatis. | - | +| 3.3.0 | 1. Optimized data transmission performance under WebSocket connection.
2. Supports skipping SSL verification, off by default. | 3.3.2.0 and higher | +| 3.2.11 | Fixed a bug in closing result set in Native connection. | - | +| 3.2.10 | 1. REST/WebSocket connections support data compression during transmission.
2. WebSocket automatic reconnection mechanism, off by default.
3. Connection class provides methods for schemaless writing.
4. Optimized data fetching performance for native connections.
5. Fixed some known issues.
6. Metadata retrieval functions can return a list of supported functions. | - | +| 3.2.9 | Fixed bug in closing WebSocket prepareStatement. | - | +| 3.2.8 | 1. Optimized auto-commit.
2. Fixed manual commit bug in WebSocket.
3. Optimized WebSocket prepareStatement using a single connection.
4. Metadata supports views. | - | +| 3.2.7 | 1. Supports VARBINARY and GEOMETRY types.
2. Added timezone setting support for native connections.
3. Added WebSocket automatic reconnection feature. | 3.2.0.0 and higher | +| 3.2.5 | Data subscription adds committed() and assignment() methods. | 3.1.0.3 and higher | +| 3.2.4 | Data subscription adds enable.auto.commit parameter under WebSocket connection, as well as unsubscribe() method. | - | +| 3.2.3 | Fixed ResultSet data parsing failure in some cases. | - | +| 3.2.2 | New feature: Data subscription supports seek function. | 3.0.5.0 and higher | +| 3.2.1 | 1. WebSocket connection supports schemaless and prepareStatement writing.
2. Consumer poll returns result set as ConsumerRecord, which can be accessed through value() method. | 3.0.3.0 and higher | +| 3.2.0 | Connection issues, not recommended for use. | - | +| 3.1.0 | WebSocket connection supports subscription function. | - | +| 3.0.1 - 3.0.4 | Fixed data parsing errors in result sets under some conditions. 3.0.1 compiled in JDK 11 environment, other versions recommended for JDK 8. | - | +| 3.0.0 | Supports TDengine 3.0 | 3.0.0.0 and higher | +| 2.0.42 | Fixed wasNull interface return value in WebSocket connection. | - | +| 2.0.41 | Fixed username and password encoding method in REST connection. | - | +| 2.0.39 - 2.0.40 | Added REST connection/request timeout settings. | - | +| 2.0.38 | JDBC REST connection adds batch fetching function. | - | +| 2.0.37 | Added support for json tag. | - | +| 2.0.36 | Added support for schemaless writing. | - | ## Exceptions and Error Codes @@ -75,47 +76,47 @@ The error codes that the JDBC connector may report include 4 types: Please refer to the specific error codes: -| Error Code | Description | Suggested Actions | -| ---------- | --------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | -| 0x2301 | connection already closed | The connection is already closed, check the connection status, or recreate the connection to execute related commands. | -| 0x2302 | this operation is NOT supported currently! | The current interface is not supported, consider switching to another connection method. | -| 0x2303 | invalid variables | Invalid parameters, please check the interface specifications and adjust the parameter types and sizes. | -| 0x2304 | statement is closed | The statement is already closed, check if the statement was used after being closed, or if the connection is normal. | -| 0x2305 | resultSet is closed | The resultSet has been released, check if the resultSet was used after being released. | -| 0x2306 | Batch is empty! | Add parameters to prepareStatement before executing executeBatch. | -| 0x2307 | Can not issue data manipulation statements with executeQuery() | Use executeUpdate() for update operations, not executeQuery(). | -| 0x2308 | Can not issue SELECT via executeUpdate() | Use executeQuery() for query operations, not executeUpdate(). | -| 0x230d | parameter index out of range | Parameter out of bounds, check the reasonable range of parameters. | -| 0x230e | connection already closed | The connection is already closed, check if the Connection was used after being closed, or if the connection is normal. | -| 0x230f | unknown sql type in tdengine | Check the Data Type types supported by TDengine. | -| 0x2310 | can't register JDBC-JNI driver | Cannot register JNI driver, check if the url is correctly filled. | -| 0x2312 | url is not set | Check if the REST connection url is correctly filled. | -| 0x2314 | numeric value out of range | Check if the correct interface was used for numeric types in the result set. | -| 0x2315 | unknown taos type in tdengine | When converting TDengine data types to JDBC data types, check if the correct TDengine data type was specified. | -| 0x2317 | | Incorrect request type used in REST connection. | -| 0x2318 | | Data transmission error occurred in REST connection, check the network situation and retry. | -| 0x2319 | user is required | Username information is missing when creating a connection. | -| 0x231a | password is required | Password information is missing when creating a connection. | -| 0x231c | httpEntity is null, sql: | An exception occurred in REST connection execution. | -| 0x231d | can't create connection with server within | Increase the httpConnectTimeout parameter to extend the connection time, or check the connection with taosAdapter. | -| 0x231e | failed to complete the task within the specified time | Increase the messageWaitTimeout parameter to extend the execution time, or check the connection with taosAdapter. | -| 0x2350 | unknown error | Unknown exception, please provide feedback to the developers on github. | -| 0x2352 | Unsupported encoding | An unsupported character encoding set was specified in the local connection. | -| 0x2353 | internal error of database, please see taoslog for more details | An error occurred while executing prepareStatement in local connection, check taos log for troubleshooting. | -| 0x2354 | JNI connection is NULL | The Connection was already closed when executing commands in local connection. Check the connection with TDengine. | -| 0x2355 | JNI result set is NULL | The result set is abnormal in local connection, check the connection and retry. | -| 0x2356 | invalid num of fields | The meta information of the result set obtained in local connection does not match. | -| 0x2357 | empty sql string | Fill in the correct SQL for execution. | -| 0x2359 | JNI alloc memory failed, please see taoslog for more details | Memory allocation error in local connection, check taos log for troubleshooting. | -| 0x2371 | consumer properties must not be null! | Parameters are null when creating a subscription, fill in the correct parameters. | -| 0x2372 | configs contain empty key, failed to set consumer property | The parameter key contains empty values, fill in the correct parameters. | -| 0x2373 | failed to set consumer property, | The parameter value contains empty values, fill in the correct parameters. | -| 0x2375 | topic reference has been destroyed | During the data subscription process, the topic reference was released. Check the connection with TDengine. | +| Error Code | Description | Suggested Actions | +| ---------- | --------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| 0x2301 | connection already closed | The connection is already closed, check the connection status, or recreate the connection to execute related commands. | +| 0x2302 | this operation is NOT supported currently! | The current interface is not supported, consider switching to another connection method. | +| 0x2303 | invalid variables | Invalid parameters, please check the interface specifications and adjust the parameter types and sizes. | +| 0x2304 | statement is closed | The statement is already closed, check if the statement was used after being closed, or if the connection is normal. | +| 0x2305 | resultSet is closed | The resultSet has been released, check if the resultSet was used after being released. | +| 0x2306 | Batch is empty! | Add parameters to prepareStatement before executing executeBatch. | +| 0x2307 | Can not issue data manipulation statements with executeQuery() | Use executeUpdate() for update operations, not executeQuery(). | +| 0x2308 | Can not issue SELECT via executeUpdate() | Use executeQuery() for query operations, not executeUpdate(). | +| 0x230d | parameter index out of range | Parameter out of bounds, check the reasonable range of parameters. | +| 0x230e | connection already closed | The connection is already closed, check if the Connection was used after being closed, or if the connection is normal. | +| 0x230f | unknown sql type in tdengine | Check the Data Type types supported by TDengine. | +| 0x2310 | can't register JDBC-JNI driver | Cannot register JNI driver, check if the url is correctly filled. | +| 0x2312 | url is not set | Check if the REST connection url is correctly filled. | +| 0x2314 | numeric value out of range | Check if the correct interface was used for numeric types in the result set. | +| 0x2315 | unknown taos type in tdengine | When converting TDengine data types to JDBC data types, check if the correct TDengine data type was specified. | +| 0x2317 | | Incorrect request type used in REST connection. | +| 0x2318 | | Data transmission error occurred in REST connection, check the network situation and retry. | +| 0x2319 | user is required | Username information is missing when creating a connection. | +| 0x231a | password is required | Password information is missing when creating a connection. | +| 0x231c | httpEntity is null, sql: | An exception occurred in REST connection execution. | +| 0x231d | can't create connection with server within | Increase the httpConnectTimeout parameter to extend the connection time, or check the connection with taosAdapter. | +| 0x231e | failed to complete the task within the specified time | Increase the messageWaitTimeout parameter to extend the execution time, or check the connection with taosAdapter. | +| 0x2350 | unknown error | Unknown exception, please provide feedback to the developers on github. | +| 0x2352 | Unsupported encoding | An unsupported character encoding set was specified in the local connection. | +| 0x2353 | internal error of database, please see taoslog for more details | An error occurred while executing prepareStatement in local connection, check taos log for troubleshooting. | +| 0x2354 | JNI connection is NULL | The Connection was already closed when executing commands in local connection. Check the connection with TDengine. | +| 0x2355 | JNI result set is NULL | The result set is abnormal in local connection, check the connection and retry. | +| 0x2356 | invalid num of fields | The meta information of the result set obtained in local connection does not match. | +| 0x2357 | empty sql string | Fill in the correct SQL for execution. | +| 0x2359 | JNI alloc memory failed, please see taoslog for more details | Memory allocation error in local connection, check taos log for troubleshooting. | +| 0x2371 | consumer properties must not be null! | Parameters are null when creating a subscription, fill in the correct parameters. | +| 0x2372 | configs contain empty key, failed to set consumer property | The parameter key contains empty values, fill in the correct parameters. | +| 0x2373 | failed to set consumer property, | The parameter value contains empty values, fill in the correct parameters. | +| 0x2375 | topic reference has been destroyed | During the data subscription process, the topic reference was released. Check the connection with TDengine. | | 0x2376 | failed to set consumer topic, topic name is empty | During the data subscription process, the subscription topic name is empty. Check if the specified topic name is correctly filled. | -| 0x2377 | consumer reference has been destroyed | The data transmission channel for the subscription has been closed, check the connection with TDengine. | -| 0x2378 | consumer create error | Data subscription creation failed, check the error information and taos log for troubleshooting. | -| 0x2379 | seek offset must not be a negative number | The seek interface parameter must not be negative, use the correct parameters. | -| 0x237a | vGroup not found in result set | VGroup not assigned to the current consumer, due to the Rebalance mechanism causing the Consumer and VGroup to be unbound. | +| 0x2377 | consumer reference has been destroyed | The data transmission channel for the subscription has been closed, check the connection with TDengine. | +| 0x2378 | consumer create error | Data subscription creation failed, check the error information and taos log for troubleshooting. | +| 0x2379 | seek offset must not be a negative number | The seek interface parameter must not be negative, use the correct parameters. | +| 0x237a | vGroup not found in result set | VGroup not assigned to the current consumer, due to the Rebalance mechanism causing the Consumer and VGroup to be unbound. | - [TDengine Java Connector Error Code](https://github.com/taosdata/taos-connector-jdbc/blob/main/src/main/java/com/taosdata/jdbc/TSDBErrorNumbers.java) @@ -489,16 +490,16 @@ For example: if the password is specified as taosdata in the URL and as taosdemo List of interface methods that return `true` for supported features, others not explicitly mentioned return `false`. -| Interface Method | Description | -|--------------------------------------------------------|-----------------------------------------------------| -| `boolean nullsAreSortedAtStart()` | Determines if `NULL` values are sorted at the start | -| `boolean storesLowerCaseIdentifiers()` | Determines if the database stores identifiers in lowercase | -| `boolean supportsAlterTableWithAddColumn()` | Determines if the database supports adding columns with `ALTER TABLE` | -| `boolean supportsAlterTableWithDropColumn()` | Determines if the database supports dropping columns with `ALTER TABLE` | -| `boolean supportsColumnAliasing()` | Determines if the database supports column aliasing | -| `boolean supportsGroupBy()` | Determines if the database supports `GROUP BY` statements | -| `boolean isCatalogAtStart()` | Determines if the catalog name appears at the start of the fully qualified name in the database | -| `boolean supportsCatalogsInDataManipulation()` | Determines if the database supports catalog names in data manipulation statements | +| Interface Method | Description | +| ---------------------------------------------- | ----------------------------------------------------------------------------------------------- | +| `boolean nullsAreSortedAtStart()` | Determines if `NULL` values are sorted at the start | +| `boolean storesLowerCaseIdentifiers()` | Determines if the database stores identifiers in lowercase | +| `boolean supportsAlterTableWithAddColumn()` | Determines if the database supports adding columns with `ALTER TABLE` | +| `boolean supportsAlterTableWithDropColumn()` | Determines if the database supports dropping columns with `ALTER TABLE` | +| `boolean supportsColumnAliasing()` | Determines if the database supports column aliasing | +| `boolean supportsGroupBy()` | Determines if the database supports `GROUP BY` statements | +| `boolean isCatalogAtStart()` | Determines if the catalog name appears at the start of the fully qualified name in the database | +| `boolean supportsCatalogsInDataManipulation()` | Determines if the database supports catalog names in data manipulation statements | ### Connection Features diff --git a/docs/examples/JDBC/JDBCDemo/pom.xml b/docs/examples/JDBC/JDBCDemo/pom.xml index 4b3e1ab675..e0c17ffbac 100644 --- a/docs/examples/JDBC/JDBCDemo/pom.xml +++ b/docs/examples/JDBC/JDBCDemo/pom.xml @@ -19,7 +19,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 org.locationtech.jts diff --git a/docs/examples/JDBC/SpringJdbcTemplate/pom.xml b/docs/examples/JDBC/SpringJdbcTemplate/pom.xml index 34719dc135..b48f17acce 100644 --- a/docs/examples/JDBC/SpringJdbcTemplate/pom.xml +++ b/docs/examples/JDBC/SpringJdbcTemplate/pom.xml @@ -47,7 +47,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 diff --git a/docs/examples/JDBC/connectionPools/pom.xml b/docs/examples/JDBC/connectionPools/pom.xml index e3ef30d2f8..8fca6ce1b2 100644 --- a/docs/examples/JDBC/connectionPools/pom.xml +++ b/docs/examples/JDBC/connectionPools/pom.xml @@ -18,7 +18,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 diff --git a/docs/examples/JDBC/consumer-demo/pom.xml b/docs/examples/JDBC/consumer-demo/pom.xml index 709f87d9c1..997643c3de 100644 --- a/docs/examples/JDBC/consumer-demo/pom.xml +++ b/docs/examples/JDBC/consumer-demo/pom.xml @@ -17,7 +17,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 com.google.guava diff --git a/docs/examples/JDBC/mybatisplus-demo/pom.xml b/docs/examples/JDBC/mybatisplus-demo/pom.xml index 2077e31d8d..27e62695bc 100644 --- a/docs/examples/JDBC/mybatisplus-demo/pom.xml +++ b/docs/examples/JDBC/mybatisplus-demo/pom.xml @@ -47,7 +47,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 diff --git a/docs/examples/JDBC/springbootdemo/pom.xml b/docs/examples/JDBC/springbootdemo/pom.xml index df8a3f5d61..f42799119b 100644 --- a/docs/examples/JDBC/springbootdemo/pom.xml +++ b/docs/examples/JDBC/springbootdemo/pom.xml @@ -70,7 +70,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 diff --git a/docs/examples/JDBC/taosdemo/pom.xml b/docs/examples/JDBC/taosdemo/pom.xml index c36973947b..936a133e31 100644 --- a/docs/examples/JDBC/taosdemo/pom.xml +++ b/docs/examples/JDBC/taosdemo/pom.xml @@ -67,7 +67,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 diff --git a/docs/examples/java/pom.xml b/docs/examples/java/pom.xml index e1a9504249..9b3a8c147d 100644 --- a/docs/examples/java/pom.xml +++ b/docs/examples/java/pom.xml @@ -22,7 +22,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 diff --git a/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingExtendInterfaceDemo.java b/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingExtendInterfaceDemo.java new file mode 100644 index 0000000000..8a83c27bbc --- /dev/null +++ b/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingExtendInterfaceDemo.java @@ -0,0 +1,87 @@ +package com.taos.example; + +import com.taosdata.jdbc.ws.TSWSPreparedStatement; + +import java.sql.*; +import java.util.ArrayList; +import java.util.Random; + +// ANCHOR: para_bind +public class WSParameterBindingExtendInterfaceDemo { + + // modify host to your own + private static final String host = "127.0.0.1"; + private static final Random random = new Random(System.currentTimeMillis()); + private static final int numOfSubTable = 10, numOfRow = 10; + + public static void main(String[] args) throws SQLException { + + String jdbcUrl = "jdbc:TAOS-WS://" + host + ":6041"; + try (Connection conn = DriverManager.getConnection(jdbcUrl, "root", "taosdata")) { + init(conn); + + String sql = "INSERT INTO ? USING power.meters TAGS(?,?) VALUES (?,?,?,?)"; + + try (TSWSPreparedStatement pstmt = conn.prepareStatement(sql).unwrap(TSWSPreparedStatement.class)) { + + for (int i = 1; i <= numOfSubTable; i++) { + // set table name + pstmt.setTableName("d_bind_" + i); + + // set tags + pstmt.setTagInt(0, i); + pstmt.setTagString(1, "location_" + i); + + // set column ts + ArrayList tsList = new ArrayList<>(); + long current = System.currentTimeMillis(); + for (int j = 0; j < numOfRow; j++) + tsList.add(current + j); + pstmt.setTimestamp(0, tsList); + + // set column current + ArrayList currentList = new ArrayList<>(); + for (int j = 0; j < numOfRow; j++) + currentList.add(random.nextFloat() * 30); + pstmt.setFloat(1, currentList); + + // set column voltage + ArrayList voltageList = new ArrayList<>(); + for (int j = 0; j < numOfRow; j++) + voltageList.add(random.nextInt(300)); + pstmt.setInt(2, voltageList); + + // set column phase + ArrayList phaseList = new ArrayList<>(); + for (int j = 0; j < numOfRow; j++) + phaseList.add(random.nextFloat()); + pstmt.setFloat(3, phaseList); + // add column + pstmt.columnDataAddBatch(); + } + // execute column + pstmt.columnDataExecuteBatch(); + // you can check exeResult here + System.out.println("Successfully inserted " + (numOfSubTable * numOfRow) + " rows to power.meters."); + } + } catch (Exception ex) { + // please refer to the JDBC specifications for detailed exceptions info + System.out.printf("Failed to insert to table meters using stmt, %sErrMessage: %s%n", + ex instanceof SQLException ? "ErrCode: " + ((SQLException) ex).getErrorCode() + ", " : "", + ex.getMessage()); + // Print stack trace for context in examples. Use logging in production. + ex.printStackTrace(); + throw ex; + } + } + + private static void init(Connection conn) throws SQLException { + try (Statement stmt = conn.createStatement()) { + stmt.execute("CREATE DATABASE IF NOT EXISTS power"); + stmt.execute("USE power"); + stmt.execute( + "CREATE STABLE IF NOT EXISTS power.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (groupId INT, location BINARY(24))"); + } + } +} +// ANCHOR_END: para_bind diff --git a/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingBasicDemo.java b/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingStdInterfaceDemo.java similarity index 61% rename from docs/examples/java/src/main/java/com/taos/example/WSParameterBindingBasicDemo.java rename to docs/examples/java/src/main/java/com/taos/example/WSParameterBindingStdInterfaceDemo.java index 1353ebbddc..7f0e523b97 100644 --- a/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingBasicDemo.java +++ b/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingStdInterfaceDemo.java @@ -1,12 +1,10 @@ package com.taos.example; -import com.taosdata.jdbc.ws.TSWSPreparedStatement; - import java.sql.*; import java.util.Random; // ANCHOR: para_bind -public class WSParameterBindingBasicDemo { +public class WSParameterBindingStdInterfaceDemo { // modify host to your own private static final String host = "127.0.0.1"; @@ -19,31 +17,29 @@ public class WSParameterBindingBasicDemo { try (Connection conn = DriverManager.getConnection(jdbcUrl, "root", "taosdata")) { init(conn); - String sql = "INSERT INTO ? USING power.meters TAGS(?,?) VALUES (?,?,?,?)"; + // If you are certain that the child table exists, you can avoid binding the tag column to improve performance. + String sql = "INSERT INTO power.meters (tbname, groupid, location, ts, current, voltage, phase) VALUES (?,?,?,?,?,?,?)"; - try (TSWSPreparedStatement pstmt = conn.prepareStatement(sql).unwrap(TSWSPreparedStatement.class)) { + try (PreparedStatement pstmt = conn.prepareStatement(sql)) { + long current = System.currentTimeMillis(); for (int i = 1; i <= numOfSubTable; i++) { - // set table name - pstmt.setTableName("d_bind_" + i); - - // set tags - pstmt.setTagInt(0, i); - pstmt.setTagString(1, "location_" + i); - - // set columns - long current = System.currentTimeMillis(); for (int j = 0; j < numOfRow; j++) { - pstmt.setTimestamp(1, new Timestamp(current + j)); - pstmt.setFloat(2, random.nextFloat() * 30); - pstmt.setInt(3, random.nextInt(300)); - pstmt.setFloat(4, random.nextFloat()); + pstmt.setString(1, "d_bind_" + i); + + pstmt.setInt(2, i); + pstmt.setString(3, "location_" + i); + + pstmt.setTimestamp(4, new Timestamp(current + j)); + pstmt.setFloat(5, random.nextFloat() * 30); + pstmt.setInt(6, random.nextInt(300)); + pstmt.setFloat(7, random.nextFloat()); pstmt.addBatch(); } - int[] exeResult = pstmt.executeBatch(); - // you can check exeResult here - System.out.println("Successfully inserted " + exeResult.length + " rows to power.meters."); } + int[] exeResult = pstmt.executeBatch(); + // you can check exeResult here + System.out.println("Successfully inserted " + exeResult.length + " rows to power.meters."); } } catch (Exception ex) { // please refer to the JDBC specifications for detailed exceptions info diff --git a/docs/examples/java/src/test/java/com/taos/test/TestAll.java b/docs/examples/java/src/test/java/com/taos/test/TestAll.java index a92ddd116c..f73b9d42ee 100644 --- a/docs/examples/java/src/test/java/com/taos/test/TestAll.java +++ b/docs/examples/java/src/test/java/com/taos/test/TestAll.java @@ -118,9 +118,14 @@ public class TestAll { } @Test - public void testWsStmtBasic() throws Exception { + public void testWsStmtStd() throws Exception { dropDB("power"); - WSParameterBindingBasicDemo.main(args); + WSParameterBindingStdInterfaceDemo.main(args); + } + @Test + public void testWsStmtExtend() throws Exception { + dropDB("power"); + WSParameterBindingExtendInterfaceDemo.main(args); } @Test diff --git a/docs/zh/07-develop/01-connect/index.md b/docs/zh/07-develop/01-connect/index.md index 2381c49d93..e81a9e08c7 100644 --- a/docs/zh/07-develop/01-connect/index.md +++ b/docs/zh/07-develop/01-connect/index.md @@ -89,7 +89,7 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速 com.taosdata.jdbc taos-jdbcdriver - 3.4.0 + 3.5.0 ``` diff --git a/docs/zh/07-develop/05-stmt.md b/docs/zh/07-develop/05-stmt.md index 74b44ba8e6..f6c67f98fe 100644 --- a/docs/zh/07-develop/05-stmt.md +++ b/docs/zh/07-develop/05-stmt.md @@ -26,10 +26,16 @@ import TabItem from "@theme/TabItem"; ## WebSocket 连接 + +参数绑定有两种接口使用方式，一种是 JDBC 标准接口，一种是扩展接口，扩展接口性能更好一些。 + ```java -{{#include docs/examples/java/src/main/java/com/taos/example/WSParameterBindingBasicDemo.java:para_bind}} +{{#include docs/examples/java/src/main/java/com/taos/example/WSParameterBindingStdInterfaceDemo.java:para_bind}} ``` +```java +{{#include docs/examples/java/src/main/java/com/taos/example/WSParameterBindingExtendInterfaceDemo.java:para_bind}} +``` 这是一个[更详细的参数绑定示例](https://github.com/taosdata/TDengine/blob/main/docs/examples/java/src/main/java/com/taos/example/WSParameterBindingFullDemo.java) diff --git a/docs/zh/14-reference/05-connector/14-java.mdx b/docs/zh/14-reference/05-connector/14-java.mdx index 27f43676f3..ab7494707b 100644 --- a/docs/zh/14-reference/05-connector/14-java.mdx +++ b/docs/zh/14-reference/05-connector/14-java.mdx @@ -33,6 +33,7 @@ TDengine 的 JDBC 驱动实现尽可能与关系型数据库驱动保持一致 | taos-jdbcdriver 版本 | 主要变化 | TDengine 版本 | | ------------------| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | +| 3.5.0 | 1. 优化了 WebSocket 连接参数绑定性能，支持参数绑定查询使用二进制数据
2. 优化了 WebSocket 连接在小查询上的性能
3. WebSocket 连接上支持设置时区 | 3.3.5.0 及更高版本 | | 3.4.0 | 1. 使用 jackson 库替换 fastjson 库
2. WebSocket 采用独立协议标识
3. 优化后台拉取线程使用，避免用户误用导致超时 | - | | 3.3.4 | 解决了 getInt 在数据类型为 float 报错 | - | | 3.3.3 | 解决了 WebSocket statement 关闭导致的内存泄漏 | - | From 29089af5926186af06cec79b6fcca665cff25b82 Mon Sep 17 00:00:00 2001 From: sheyanjie-qq <249478495@qq.com> Date: Fri, 27 Dec 2024 15:26:41 +0800 Subject: [PATCH 02/73] add jdbc time zone and app info doc --- docs/en/14-reference/05-connector/14-java.md | 19 +++++++++++-------- docs/zh/14-reference/05-connector/14-java.mdx | 18 +++++++++++------- 2 files changed, 22 insertions(+), 15 deletions(-) diff --git a/docs/en/14-reference/05-connector/14-java.md b/docs/en/14-reference/05-connector/14-java.md index f52b7e71f7..48302b9d3b 100644 --- a/docs/en/14-reference/05-connector/14-java.md +++ b/docs/en/14-reference/05-connector/14-java.md @@ -32,7 +32,7 @@ The JDBC driver implementation for TDengine strives to be consistent with relati | taos-jdbcdriver Version | Major Changes | TDengine Version | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | -| 3.5.0 | 1. Optimized the performance of WebSocket connection parameter binding, supporting parameter binding queries using binary data.
2. Optimized the performance of small queries in WebSocket connection.
3. Added support for setting time zone on WebSocket connection. | 3.3.5.0 and higher | +| 3.5.0 | 1. Optimized the performance of WebSocket connection parameter binding, supporting parameter binding queries using binary data.
2. Optimized the performance of small queries in WebSocket connection.
3. Added support for setting time zone and app info on WebSocket connection. | 3.3.5.0 and higher | | 3.4.0 | 1. Replaced fastjson library with jackson.
2. WebSocket uses a separate protocol identifier.
3. Optimized background thread usage to avoid user misuse leading to timeouts. | - | | 3.3.4 | Fixed getInt error when data type is float. | - | | 3.3.3 | Fixed memory leak caused by closing WebSocket statement. | - | @@ -245,13 +245,13 @@ For WebSocket connections, the configuration parameters in the URL are as follow - user: Login username for TDengine, default value 'root'. - password: User login password, default value 'taosdata'. -- charset: Specifies the character set for parsing string data when batch fetching is enabled. - batchErrorIgnore: true: Continues executing the following SQL if one SQL fails during the execution of Statement's executeBatch. false: Does not execute any statements after a failed SQL. Default value: false. - httpConnectTimeout: Connection timeout in ms, default value 60000. - messageWaitTimeout: Message timeout in ms, default value 60000. - useSSL: Whether SSL is used in the connection. +- timezone: Client timezone, default is the system current timezone. Recommended not to set, using the system time zone provides better performance. -**Note**: Some configuration items (such as: locale, timezone) do not take effect in WebSocket connections. +**Note**: Some configuration items (such as: locale, charset) do not take effect in WebSocket connections. **REST Connection** Using JDBC REST connection does not depend on the client driver. Compared to native JDBC connections, you only need to: @@ -264,14 +264,13 @@ For REST connections, the configuration parameters in the URL are as follows: - user: Login username for TDengine, default value 'root'. - password: User login password, default value 'taosdata'. -- charset: Specifies the character set for parsing string data when batch fetching is enabled. - batchErrorIgnore: true: Continues executing the following SQL if one SQL fails during the execution of Statement's executeBatch. false: Does not execute any statements after a failed SQL. Default value: false. - httpConnectTimeout: Connection timeout in ms, default value 60000. - httpSocketTimeout: Socket timeout in ms, default value 60000. - useSSL: Whether SSL is used in the connection. - httpPoolSize: REST concurrent request size, default 20. -**Note**: Some configuration items (such as: locale, timezone) do not take effect in REST connections. +**Note**: Some configuration items (such as: locale, charset and timezone) do not take effect in REST connections. :::note @@ -295,7 +294,9 @@ The configuration parameters in properties are as follows: - TSDBDriver.PROPERTY_KEY_CONFIG_DIR: Effective only when using native JDBC connections. Client configuration file directory path, default value on Linux OS is `/etc/taos`, on Windows OS is `C:/TDengine/cfg`. - TSDBDriver.PROPERTY_KEY_CHARSET: Character set used by the client, default value is the system character set. - TSDBDriver.PROPERTY_KEY_LOCALE: Effective only when using native JDBC connections. Client locale, default value is the current system locale. -- TSDBDriver.PROPERTY_KEY_TIME_ZONE: Effective only when using native JDBC connections. Client time zone, default value is the current system time zone. Due to historical reasons, we only support part of the POSIX standard, such as UTC-8 (representing Shanghai, China), GMT-8, Asia/Shanghai. +- TSDBDriver.PROPERTY_KEY_TIME_ZONE: + - Native connections: Client time zone, default value is the current system time zone. Effective globally. Due to historical reasons, we only support part of the POSIX standard, such as UTC-8 (representing Shanghai, China), GMT-8, Asia/Shanghai. + - WebSocket connections. Client time zone, default value is the current system time zone. Effective on the connection. Only IANA time zones are supported, such as Asia/Shanghai. It is recommended not to set this parameter, as using the system time zone provides better performance. - TSDBDriver.HTTP_CONNECT_TIMEOUT: Connection timeout, in ms, default value is 60000. Effective only in REST connections. - TSDBDriver.HTTP_SOCKET_TIMEOUT: Socket timeout, in ms, default value is 60000. Effective only in REST connections and when batchfetch is set to false. - TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: Message timeout, in ms, default value is 60000. Effective only under WebSocket connections. @@ -304,12 +305,14 @@ The configuration parameters in properties are as follows: - TSDBDriver.PROPERTY_KEY_ENABLE_COMPRESSION: Whether to enable compression during transmission. Effective only when using REST/WebSocket connections. true: enabled, false: not enabled. Default is false. - TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: Whether to enable auto-reconnect. Effective only when using WebSocket connections. true: enabled, false: not enabled. Default is false. -> **Note**: Enabling auto-reconnect is only effective for simple SQL execution, schema-less writing, and data subscription. It is ineffective for parameter binding. Auto-reconnect is only effective for connections established through parameters specifying the database, and ineffective for later `use db` statements to switch databases. + > **Note**: Enabling auto-reconnect is only effective for simple SQL execution, schema-less writing, and data subscription. It is ineffective for parameter binding. Auto-reconnect is only effective for connections established through parameters specifying the database, and ineffective for later `use db` statements to switch databases. - TSDBDriver.PROPERTY_KEY_RECONNECT_INTERVAL_MS: Auto-reconnect retry interval, in milliseconds, default value 2000. Effective only when PROPERTY_KEY_ENABLE_AUTO_RECONNECT is true. - TSDBDriver.PROPERTY_KEY_RECONNECT_RETRY_COUNT: Auto-reconnect retry count, default value 3, effective only when PROPERTY_KEY_ENABLE_AUTO_RECONNECT is true. - TSDBDriver.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: Disable SSL certificate validation. Effective only when using WebSocket connections. true: enabled, false: not enabled. Default is false. - +- TSDBDriver.PROPERTY_KEY_APP_NAME: App name, can be used for display in the `show connections` query result. Effective only when using WebSocket connections. Default value is java. +- TSDBDriver.PROPERTY_KEY_APP_IP: App IP, can be used for display in the `show connections` query result. Effective only when using WebSocket connections. Default value is empty. + Additionally, for native JDBC connections, other parameters such as log level and SQL length can be specified by specifying the URL and Properties. **Priority of Configuration Parameters** diff --git a/docs/zh/14-reference/05-connector/14-java.mdx b/docs/zh/14-reference/05-connector/14-java.mdx index ab7494707b..c61bf51c82 100644 --- a/docs/zh/14-reference/05-connector/14-java.mdx +++ b/docs/zh/14-reference/05-connector/14-java.mdx @@ -33,7 +33,7 @@ TDengine 的 JDBC 驱动实现尽可能与关系型数据库驱动保持一致 | taos-jdbcdriver 版本 | 主要变化 | TDengine 版本 | | ------------------| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | -| 3.5.0 | 1. 优化了 WebSocket 连接参数绑定性能，支持参数绑定查询使用二进制数据
2. 优化了 WebSocket 连接在小查询上的性能
3. WebSocket 连接上支持设置时区 | 3.3.5.0 及更高版本 | +| 3.5.0 | 1. 优化了 WebSocket 连接参数绑定性能，支持参数绑定查询使用二进制数据
2. 优化了 WebSocket 连接在小查询上的性能
3. WebSocket 连接上支持设置时区和应用信息 | 3.3.5.0 及更高版本 | | 3.4.0 | 1. 使用 jackson 库替换 fastjson 库
2. WebSocket 采用独立协议标识
3. 优化后台拉取线程使用，避免用户误用导致超时 | - | | 3.3.4 | 解决了 getInt 在数据类型为 float 报错 | - | | 3.3.3 | 解决了 WebSocket statement 关闭导致的内存泄漏 | - | @@ -244,13 +244,13 @@ TDengine 中，只要保证 firstEp 和 secondEp 中一个节点有效，就可 对于 WebSocket 连接，url 中的配置参数如下： - user：登录 TDengine 用户名，默认值 'root'。 - password：用户登录密码，默认值 'taosdata'。 -- charset: 当开启批量拉取数据时，指定解析字符串数据的字符集。 - batchErrorIgnore：true：在执行 Statement 的 executeBatch 时，如果中间有一条 SQL 执行失败，继续执行下面的 SQL 了。false：不再执行失败 SQL 后的任何语句。默认值为：false。 - httpConnectTimeout: 连接超时时间，单位 ms， 默认值为 60000。 - messageWaitTimeout: 消息超时时间, 单位 ms， 默认值为 60000。 - useSSL: 连接中是否使用 SSL。 +- timezone：客户端使用的时区，连接上生效，默认值为系统时区。推荐不设置，使用系统时区性能更好。 -**注意**：部分配置项（比如：locale、timezone）在 WebSocket 连接中不生效。 +**注意**：部分配置项（比如：locale、charset）在 WebSocket 连接中不生效。 **REST 连接** 使用 JDBC REST 连接，不需要依赖客户端驱动。与 JDBC 原生连接相比，仅需要： @@ -262,14 +262,13 @@ TDengine 中，只要保证 firstEp 和 secondEp 中一个节点有效，就可 对于 REST 连接，url 中的配置参数如下： - user：登录 TDengine 用户名，默认值 'root'。 - password：用户登录密码，默认值 'taosdata'。 -- charset: 当开启批量拉取数据时，指定解析字符串数据的字符集。 - batchErrorIgnore：true：在执行 Statement 的 executeBatch 时，如果中间有一条 SQL 执行失败，继续执行下面的 SQL 了。false：不再执行失败 SQL 后的任何语句。默认值为：false。 - httpConnectTimeout: 连接超时时间，单位 ms， 默认值为 60000。 - httpSocketTimeout: socket 超时时间，单位 ms，默认值为 60000。 - useSSL: 连接中是否使用 SSL。 - httpPoolSize: REST 并发请求大小，默认 20。 -**注意**：部分配置项（比如：locale、timezone）在 REST 连接中不生效。 +**注意**：部分配置项（比如：locale、charset 和 timezone）在 REST 连接中不生效。 :::note @@ -292,7 +291,9 @@ properties 中的配置参数如下： - TSDBDriver.PROPERTY_KEY_CONFIG_DIR：仅在使用 JDBC 原生连接时生效。客户端配置文件目录路径，Linux OS 上默认值 `/etc/taos`，Windows OS 上默认值 `C:/TDengine/cfg`。 - TSDBDriver.PROPERTY_KEY_CHARSET：客户端使用的字符集，默认值为系统字符集。 - TSDBDriver.PROPERTY_KEY_LOCALE：仅在使用 JDBC 原生连接时生效。 客户端语言环境，默认值系统当前 locale。 -- TSDBDriver.PROPERTY_KEY_TIME_ZONE：仅在使用 JDBC 原生连接时生效。 客户端使用的时区，默认值为系统当前时区。因为历史的原因，我们只支持POSIX标准的部分规范，如UTC-8(代表中国上上海), GMT-8，Asia/Shanghai 这几种形式。 +- TSDBDriver.PROPERTY_KEY_TIME_ZONE： + - 原生连接：客户端使用的时区，默认值为系统当前时区，全局生效。因为历史的原因，我们只支持POSIX标准的部分规范，如UTC-8(代表中国上上海), GMT-8，Asia/Shanghai 这几种形式。 + - WebSocket 连接：客户端使用的时区，连接上生效，默认值为系统时区。仅支持 IANA 时区，即 Asia/Shanghai 这种形式。推荐不设置，使用系统时区性能更好。 - TSDBDriver.HTTP_CONNECT_TIMEOUT: 连接超时时间，单位 ms， 默认值为 60000。仅在 REST 连接时生效。 - TSDBDriver.HTTP_SOCKET_TIMEOUT: socket 超时时间，单位 ms，默认值为 60000。仅在 REST 连接且 batchfetch 设置为 false 时生效。 - TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms， 默认值为 60000。 仅 WebSocket 连接下有效。 @@ -300,12 +301,15 @@ properties 中的配置参数如下： - TSDBDriver.HTTP_POOL_SIZE: REST 并发请求大小，默认 20。 - TSDBDriver.PROPERTY_KEY_ENABLE_COMPRESSION: 传输过程是否启用压缩。仅在使用 REST/WebSocket 连接时生效。true: 启用，false: 不启用。默认为 false。 - TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: 是否启用自动重连。仅在使用 WebSocket 连接时生效。true: 启用，false: 不启用。默认为 false。 -> **注意**：启用自动重连仅对简单执行 SQL 语句以及 无模式写入、数据订阅有效。对于参数绑定无效。自动重连仅对连接建立时通过参数指定数据库有效，对后面的 `use db` 语句切换数据库无效。 + > **注意**：启用自动重连仅对简单执行 SQL 语句以及 无模式写入、数据订阅有效。对于参数绑定无效。自动重连仅对连接建立时通过参数指定数据库有效，对后面的 `use db` 语句切换数据库无效。 - TSDBDriver.PROPERTY_KEY_RECONNECT_INTERVAL_MS: 自动重连重试间隔，单位毫秒，默认值 2000。仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。 - TSDBDriver.PROPERTY_KEY_RECONNECT_RETRY_COUNT: 自动重连重试次数，默认值 3，仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。 - TSDBDriver.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: 关闭 SSL 证书验证 。仅在使用 WebSocket 连接时生效。true: 启用，false: 不启用。默认为 false。 +- TSDBDriver.PROPERTY_KEY_APP_NAME: App 名称，可用于 `show connections` 查询结果显示。仅在使用 WebSocket 连接时生效。默认值为 java。 +- TSDBDriver.PROPERTY_KEY_APP_IP: App IP，可用于 `show connections` 查询结果显示。仅在使用 WebSocket 连接时生效。默认值为空。 + 此外对 JDBC 原生连接，通过指定 URL 和 Properties 还可以指定其他参数，比如日志级别、SQL 长度等。 **配置参数的优先级** From 014363954c39fc2021224431fd5be47422a809a6 Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Sun, 29 Dec 2024 22:27:39 +0800 Subject: [PATCH 03/73] feat: add setup scripts for build env --- packaging/setup_env.sh | 2055 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 2055 insertions(+) create mode 100644 packaging/setup_env.sh diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh new file mode 100644 index 0000000000..cce54b2839 --- /dev/null +++ b/packaging/setup_env.sh @@ -0,0 +1,2055 @@ +#!/bin/bash +# define default timezone +DEFAULT_TIMEZONE="Asia/Shanghai" + +# Define default DNS server +DEFAULT_DNS="192.168.1.252" + +# Define the packages to be installed +SYSTEM_APT_TOOLS="git wget vim gdb screen tmux ntp tree atop iotop sysstat fio tcpdump iperf3 qemu-guest-agent dstat linux-tools-common linux-tools-generic jq zip unzip cloud-guest-utils nfs-kernel-server nfs-common" +SYSTEM_YUM_TOOLS="git wget vim gdb screen tmux ntp tree atop iotop sysstat fio tcpdump iperf3 qemu-guest-agent dstat jq zip unzip cloud-utils-growpart python3-devel nfs-utils rpm-build automake autoconf libevent-devel ncurses-devel" + +# Define the packages to be installed for build TDinternal +BUILD_APT_TOOLS="llvm gcc make cmake perl g++ lzma curl locales psmisc libgeos-dev libgoogle-glog-dev valgrind rsync libjemalloc-dev openssh-server sshpass net-tools dirmngr gnupg apt-transport-https \ + ca-certificates software-properties-common r-base iputils-ping build-essential git libssl-dev libgflags2.2 libgflags-dev libjansson-dev libsnappy-dev liblzma-dev libz-dev zlib1g pkg-config" +BUILD_YUM_TOOLS="gcc make cmake3 perl gcc-c++ xz curl psmisc geos glog valgrind rsync jemalloc openssh-server sshpass net-tools gnupg2 libarchive snappy-devel pkgconfig libatomic perl-IPC-Cmd libcurl-devel libxml2-devel\ + ca-certificates libicu-devel R-core iputils bison flex glibc-static libstdc++-static libstdc++-devel openssl-devel gflags jansson jansson-devel snappy xz-devel zlib-devel zlib bzip2-devel zlib-static libs3" + +# Define the packages to be installed via pip +PIP_PKGS="wheel setuptools-rust pandas psutil fabric2 requests faker simplejson toml pexpect tzlocal distro decorator loguru hyperloglog taospy numpy poetry" + +# Gcc version to be updated +GCC_VERSION="9.5.0" + +# Define the version of the Ubuntu release +# Define jdk version to be installed +if [ -f /etc/debian_version ]; then + DIST_VERSION=$(lsb_release -sr) + JDK_VERSION="openjdk-17-jdk" +elif [ -f /etc/redhat-release ]; then + DIST_VERSION=$(grep -oP '\d+\.\d+' < /etc/redhat-release) + JDK_VERSION="java-1.8.0-openjdk" +else + echo "Unsupported Linux distribution." + exit 1 +fi + +# Define the path where the core dump files should be stored +COREPATH="/corefile" + +# Define the path where the repository should be cloned +REPOPATH="$HOME/repos" + +# Define the path to the script directory +SCRIPT_DIR=$(dirname "$(realpath "$0")") + +# Define the path to the .bashrc file +BASH_RC=$HOME/.bashrc + +# Define the path to the Cargo configuration file +CARGO_CONFIG_FILE=$HOME/.cargo/config.toml + +# Define jmeter version to be installed +JMETER_VERSION="5.6.3" + +# Define the path where the Prometheus binary should exist +PROMETHEUS_BINARY="/usr/local/bin/prometheus" + +# Define the path where the Node Exporter binary should exist +NODE_EXPORTER_BINARY="/usr/local/bin/node_exporter" + +# Define the path where the Process Exporter binary should exist +PROCESS_EXPORTER_BINARY="/usr/local/bin/process-exporter" + +# Define fstab input +FSTAB_LINE="share-server.platform.tdengine.dev:/mnt/share_server /mnt/share_server nfs rw,sync,_netdev 0 0" + +# ANSI color codes +GREEN='\033[0;32m' # Green color +RED='\033[0;31m' # Red color +NO_COLOR='\033[0m' # Reset to default color +YELLOW='\033[0;33m' # Yellow color + +# read -r -d '' CLOUD_INIT_CONFIG << 'EOF' +# datasource: +# NoCloud: +# seedfrom: /var/lib/cloud/seed/nocloud/ +# meta-data: {} +# user-data: {} +# vendor-data: {} +# ConfigDrive: {} +# None: {} +# datasource_list: [ NoCloud, ConfigDrive, None ] +# EOF + +read -r -d '' CUSTOM_SETTINGS <<'EOF' +export LC_CTYPE="en_US.UTF-8" +export LANG="en_US.UTF-8" +export HISTTIMEFORMAT="%d/%m/%y %T " +parse_git_branch() { + git branch 2> /dev/null | sed -e '/^[^*]/d' -e 's/* \(.*\)/(\1)/' +} +export PS1="\u@\h \[\e[32m\]\w \[\e[91m\]\$(parse_git_branch)\[\e[00m\]$ " +EOF + +read -r -d '' CARGO_CONFIG <<'EOF' +[source.crates-io] +replace-with = 'rsproxy-sparse' +[source.rsproxy] +registry = "https://rsproxy.cn/crates.io-index" +[source.rsproxy-sparse] +registry = "sparse+https://rsproxy.cn/index/" +[registries.rsproxy] +index = "https://rsproxy.cn/crates.io-index" +[net] +git-fetch-with-cli = true +[source.tuna] +registry = "https://mirrors.tuna.tsinghua.edu.cn/git/crates.io-index.git" +[source.ustc] +registry = "git://mirrors.ustc.edu.cn/crates.io-index" +[source.sjtu] +registry = "https://mirrors.sjtug.sjtu.edu.cn/git/crates.io-index" +[source.rustcc] +registry = "git://crates.rustcc.cn/crates.io-index" +EOF + +# Help function to display usage information +help() { + echo "Usage: $0 [option]" + echo "Options:" + echo " --help - Display this help and exit" + echo " setup_all - Setup all configurations and installations" + echo " TDasset - Prepare TDasset env" + echo " TDinternal - Prepare TDinternal env" + echo " TDgpt - Prepare TDgpt env" + echo " taostest - Prepare taostest env" + echo " system_config - Perform system configuration" + echo " deploy_pure - Deploy Pure environment" + echo " deploy_dev - Deploy development environment" + echo " deploy_cmake - Deploy CMake" + echo " update_redhat_gcc - Update GCC on Red Hat or CentOS" + echo " update_redhat_tmux - Update tmux on Red Hat or CentOS" + echo " deploy_tmux - Deploy tmux" + echo " config_ssh - Configure SSH settings" + echo " disable_firewalld - Disable firewalld" + echo " config_cloud_init - Set cloud initialization parameters" + echo " deploy_git - Deploy git repositories" + echo " replace_sources - Replace package sources" + echo " update - Update the system" + echo " upgrade - Upgrade the system" + echo " config_timezone - Configure the system timezone" + echo " config_dns - Set DNS configurations" + echo " config_custom_settings - Add custom settings to your shell configuration" + echo " config_share_server - Configure share server" + echo " install_packages - Install specified packages" + echo " config_system_limits - Configure system limits and kernel parameters" + echo " config_coredump - Configure core dump settings" + echo " disable_service - Disable specified services" + echo " install_python - Install Python and pip" + echo " install_java - Install Java" + echo " install_maven - Install Maven" + echo " deploy_go - Deploy Go environment" + echo " deploy_rust - Deploy Rust environment" + echo " install_node - Install Node via package manager or binary" + echo " install_node_via_nvm - Install Node via NVM" + echo " install_pnpm - Install PNPM, node version >=v18.12.00 required" + echo " deploy_node_exporter - Deploy Node Exporter for Prometheus" + echo " deploy_process_exporter - Deploy Process Exporter" + echo " deploy_prometheus - Deploy Prometheus" + echo " deploy_grafana - Deploy Grafana" + echo " deploy_jmeter - Deploy JMeter" + echo " install_nginx - Install NGINX" + echo " config_qemu_guest_agent - Configure QEMU guest agent" + echo " deploy_docker - Deploy Docker" + echo " deploy_docker_compose - Deploy Docker Compose" + echo " clone_enterprise - Clone the enterprise repository" + echo " clone_community - Clone the community repository" + echo " clone_taosx - Clone TaosX repository" + echo " clone_taoskeeper - Clone TaosKeeper repository" + echo " clone_taostest - Clone TaosTest repository" + echo " clone_operation - Clone operation tools repository" +} + +replace_apt_sources() { + # Define the codename of the Ubuntu release + local CODENAME + CODENAME=$(lsb_release -sc) + if grep -q "mirrors.aliyun.com" /etc/apt/sources.list; then + echo "The Aliyun mirror is already set." + else + echo "Backing up the original sources.list..." + cp /etc/apt/sources.list /etc/apt/sources.list.bak + + echo "Replacing sources.list with the Aliyun mirror..." + tee /etc/apt/sources.list << EOF +deb http://mirrors.aliyun.com/ubuntu/ $CODENAME main restricted universe multiverse +deb http://mirrors.aliyun.com/ubuntu/ $CODENAME-security main restricted universe multiverse +deb http://mirrors.aliyun.com/ubuntu/ $CODENAME-updates main restricted universe multiverse +deb http://mirrors.aliyun.com/ubuntu/ $CODENAME-proposed main restricted universe multiverse +deb http://mirrors.aliyun.com/ubuntu/ $CODENAME-backports main restricted universe multiverse +deb-src http://mirrors.aliyun.com/ubuntu/ $CODENAME main restricted universe multiverse +deb-src http://mirrors.aliyun.com/ubuntu/ $CODENAME-security main restricted universe multiverse +deb-src http://mirrors.aliyun.com/ubuntu/ $CODENAME-updates main restricted universe multiverse +deb-src http://mirrors.aliyun.com/ubuntu/ $CODENAME-proposed main restricted universe multiverse +deb-src http://mirrors.aliyun.com/ubuntu/ $CODENAME-backports main restricted universe multiverse +EOF + fi + echo "Updating repositories..." + apt-get update -y + echo "The sources have been replaced and updated successfully." +} + +replace_yum_sources() { + if grep -q "mirrors.aliyun.com" /etc/yum.repos.d/CentOS-Base.repo; then + echo "The Aliyun mirror is already set." + else + echo "Backing up the original CentOS-Base.repo..." + cp /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak + + echo "Replacing CentOS-Base.repo with the Aliyun mirror..." + tee /etc/yum.repos.d/CentOS-Base.repo << 'EOF' +[base] +name=CentOS-$releasever - Base - Aliyun +baseurl=http://mirrors.aliyun.com/centos/$releasever/os/$basearch/ +gpgcheck=1 +gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7 + +#released updates +[updates] +name=CentOS-$releasever - Updates - Aliyun +baseurl=http://mirrors.aliyun.com/centos/$releasever/updates/$basearch/ +gpgcheck=1 +gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7 + +#additional packages that may be useful +[extras] +name=CentOS-$releasever - Extras - Aliyun +baseurl=http://mirrors.aliyun.com/centos/$releasever/extras/$basearch/ +gpgcheck=1 +gpgkey=http://mirrors.aliyun.com/centos/RPM-GPG-KEY-CentOS-7 +EOF + fi + echo "Updating repositories..." + yum makecache fast + yum install epel-release -y + yum update -y + + echo "The sources have been replaced and updated successfully." +} + +replace_sources() { + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Replacing sources for APT package manager." + replace_apt_sources + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Replacing sources for YUM package manager." + replace_yum_sources + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +update() { + echo "Updating ..." + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Using APT package manager." + apt update -y + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Using YUM package manager." + yum install epel-release -y + yum update -y + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +upgrade() { + echo "Upgrading ..." + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Using APT package manager." + apt upgrade -y + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Using YUM package manager." + yum upgrade -y + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +config_frontend() { + # only ubuntu need + if [ -f /etc/debian_version ]; then + echo "Configuring frontend..." + add_config_if_not_exist "export DEBIAN_FRONTEND=noninteractive" "$BASH_RC" + fi + update + upgrade + systemctl restart dbus.service network-dispatcher.service + +} + +# Adds a configuration to a file if it does not already exist +add_config_if_not_exist() { + local config="$1" + local file="$2" + grep -qF -- "$config" "$file" || echo "$config" >> "$file" +} + +# General error handling function +check_status() { + local message_on_failure="$1" + local message_on_success="$2" + local exit_code="$3" + + if [ "${exit_code:-0}" -ne 0 ]; then + echo -e "${RED}${message_on_failure}${NO_COLOR}" + exit 1 + else + echo -e "${GREEN}${message_on_success}${NO_COLOR}" + fi +} + +# Config Share-NFS server +config_share_server() { + echo "Configuring share server..." + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Using APT package manager." + install_package "nfs-common" + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Using YUM package manager." + install_package "nfs-utils" + else + echo "Unsupported Linux distribution." + exit 1 + fi + mkdir -p /mnt/share_server + add_config_if_not_exist "$FSTAB_LINE" /etc/fstab + mount -a + check_status "Failed to configure share server" "Share server configured successfully." $? +} + +# Init environment +init_env() { + export DEBIAN_FRONTEND=noninteractive + export LC_CTYPE="en_US.UTF-8" +} + +# Install packages +install_packages() { + echo "Installing $package..." + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Using APT package manager." + install_package $SYSTEM_APT_TOOLS + install_package $BUILD_APT_TOOLS + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Using YUM package manager." + yum install epel-release -y + install_package $SYSTEM_YUM_TOOLS + install_package $BUILD_YUM_TOOLS + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +# Install package +install_package() { + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Using APT package manager." + install_via_apt "$@" + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Using YUM package manager." + install_via_yum "$@" + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +# Install package via apt +install_via_apt() { + echo -e "${YELLOW}Installing packages: $*...${NO_COLOR}" + if DEBIAN_FRONTEND=noninteractive apt-get install -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" "$@"; then + echo -e "${GREEN}Installed packages successfully.${NO_COLOR}" + else + echo -e "${RED}Failed to install packages.${NO_COLOR}" + return 1 + fi +} + +# Install package via yum +install_via_yum() { + echo -e "${YELLOW}Installing packages: $*...${NO_COLOR}" + if yum install -y "$@"; then + echo -e "${GREEN}Installed packages successfully.${NO_COLOR}" + else + echo -e "${RED}Failed to install packages.${NO_COLOR}" + return 1 + fi +} + +# disable and stop firewalld +disable_firewalld() { + if [ -f /etc/debian_version ]; then + # Only Debian or Ubuntu + echo "ufw disable..." + ufw disable + check_status "Failed to disable ufw" "Ufw disabled successfully." $? + elif [ -f /etc/redhat-release ]; then + # Only Red Hat or CentOS + echo "Disabling firewalld..." + systemctl stop firewalld + systemctl disable firewalld + check_status "Failed to disable firewalld" "Firewalld disabled successfully." $? + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +# Modifies SSH configuration and sets the root password +config_ssh() { + echo "Configuring SSH settings..." + sed -i 's/^#PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config + sed -i 's/^PermitRootLogin.*/PermitRootLogin yes/' /etc/ssh/sshd_config + + echo "Restarting SSH..." + if [ "$DIST_VERSION" = "24.04" ]; then + systemctl restart ssh + else + systemctl restart sshd + fi + check_status "Failed to restart SSH" "SSH restarted successfully." $? +} + +# Sets the timezone +config_timezone() { + echo "Setting timezone to $DEFAULT_TIMEZONE..." + timedatectl set-timezone "$DEFAULT_TIMEZONE" + check_status "Failed to set timezone" "Timezone set to $DEFAULT_TIMEZONE successfully." $? +} + +# Disables service +disable_service() { + if [ -f /etc/debian_version ]; then + # Only Debian or Ubuntu + echo "Stop and disable and related services..." + systemctl stop apt-daily.service apt-daily-upgrade.service apt-daily-upgrade.timer apt-daily.timer unattended-upgrades + systemctl disable apt-daily.service apt-daily-upgrade.service apt-daily-upgrade.timer apt-daily.timer unattended-upgrades + check_status "Failed to disable related services" "Related services disabled successfully." $? + fi +} + +# Config dns for Red Hat or CentOS +config_redhat_dns() { + local DEFAULT_DNS="192.168.2.99" + echo "Configuring DNS settings to use $INTERNAL_DNS and $DEFAULT_DNS..." + echo "nameserver $INTERNAL_DNS" > /etc/resolv.conf + echo "nameserver $DEFAULT_DNS" >> /etc/resolv.conf + check_status "Failed to configure DNS" "DNS configured to use $INTERNAL_DNS and $DEFAULT_DNS successfully." $? +} + +# Config dns for Debian or Ubuntu +config_debian_dns() { + local DEFAULT_DNS="192.168.2.99" + echo "Configuring DNS settings to use $INTERNAL_DNS and $DEFAULT_DNS..." + systemctl stop systemd-resolved.service + echo "[Resolve]" > /etc/systemd/resolved.conf + echo "DNS=$INTERNAL_DNS" >> /etc/systemd/resolved.conf + echo "DNS=$DEFAULT_DNS" >> /etc/systemd/resolved.conf + systemctl restart systemd-resolved.service + ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf + check_status "Failed to configure DNS" "DNS configured to use $INTERNAL_DNS and $DEFAULT_DNS successfully." $? +} + +# Config DNS settings +config_dns() { + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Configuring DNS settings for Debian or Ubuntu..." + config_debian_dns + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Configuring DNS settings for Red Hat or CentOS..." + config_redhat_dns + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +# Config qemu-guest-agent +config_qemu_guest_agent() { + install_package "qemu-guest-agent" + echo "Configuring qemu-guest-agent..." + systemctl enable qemu-guest-agent + systemctl start qemu-guest-agent + check_status "Failed to configure qemu-guest-agent" "Qemu-guest-agent configured successfully." $? +} + +# Config custom settings +config_custom_settings() { + echo "Configuring custom settings..." + marker="parse_git_branch" + + if grep -qF "$marker" "$BASH_RC"; then + echo "Configuration already exists in ""$BASH_RC""." + else + echo "Adding configuration to ""$BASH_RC""." + echo "$CUSTOM_SETTINGS" >>"$BASH_RC" + echo "Custom settings have been updated in your $BASH_RC file." + fi + check_status "Failed to apply custom settings" "Custom settings configured successfully." $? +} + +# Config core dump settings +config_coredump() { + echo "Configuring core dump directory..." + mkdir -p $COREPATH + add_config_if_not_exist "kernel.core_pattern=$COREPATH/core_%e-%p" /etc/sysctl.conf + add_config_if_not_exist "ulimit -n 600000" "$BASH_RC" + add_config_if_not_exist "ulimit -c unlimited" "$BASH_RC" + sysctl -p + check_status "Failed to apply core dump settings" "Core path:$COREPATH applied successfully." $? +} + +# Modifies system resource limits and TCP/IP core parameters +config_system_limits() { + echo "Configuring system limits and kernel parameters..." + local sysctl_conf="/etc/sysctl.conf" + local limits_conf="/etc/security/limits.conf" + + add_config_if_not_exist "fs.nr_open = 1048576" $sysctl_conf + add_config_if_not_exist "net.core.somaxconn=10240" $sysctl_conf + add_config_if_not_exist "net.core.netdev_max_backlog=20480" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_max_syn_backlog=10240" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_retries2=5" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_syn_retries=2" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_synack_retries=2" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_tw_reuse=1" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_keepalive_time=600" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_abort_on_overflow=1" $sysctl_conf + add_config_if_not_exist "net.ipv4.tcp_max_tw_buckets=5000" $sysctl_conf + + sysctl -p + check_status "Failed to apply sysctl settings" "Apply sysctl settings successfully." $? + + for limit in "soft nproc 65536" "soft nofile 65536" "soft stack 65536" "hard nproc 65536" "hard nofile 65536" "hard stack 65536"; do + add_config_if_not_exist "* $limit" $limits_conf + add_config_if_not_exist "root $limit" $limits_conf + done + check_status "Failed to apply limits settings" "Apply limits settings successfully." $? +} + +# Check the operating system version +centos_skip_check() { + # Check if the operating system is CentOS 7 + if [[ -f /etc/redhat-release ]]; then + if grep -q "CentOS Linux release 7" /etc/redhat-release; then + echo "This platform requires you to manually upgrade gcc and glibc." + exit 1 + fi + fi +} + +# Other logic can go here... + +# Deploy cmake +deploy_cmake() { + # Check if cmake is installed + if command -v cmake >/dev/null 2>&1; then + echo "Cmake is already installed. Skipping installation." + cmake --version + return + fi + install_package "cmake3" + ln -sf /usr/bin/cmake3 /usr/bin/cmake + check_status "Failed to install cmake" "Install cmake successfully" $? +} + + +# install pkg via pip +install_pip_pkg() { + if [ "$DIST_VERSION" != "24.04" ]; then + echo "Installing $PIP_PKGS ..." + pip3 install --upgrade pip setuptools -i https://pypi.tuna.tsinghua.edu.cn/simple + install_via_pip "$PIP_PKGS" + fi +} + +install_via_pip() { + echo "pip install $*..." + if pip3 install $* -i https://pypi.tuna.tsinghua.edu.cn/simple; then + echo "pip install packages successfully." + else + echo "Failed to install packages." + return 1 + fi +} + + +# Complie python +download_and_compile_python() { + if [ -f /etc/debian_version ]; then + install_package gcc make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev + elif [ -f /etc/redhat-release ]; then + # install_package gcc patch libffi libffi-devel python-devel zlib zlib-devel bzip2-devel openssl-devel openssl11 openssl11-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel + install_package gcc zlib zlib-devel libffi libffi-devel readline-devel openssl-devel openssl11 openssl11-devel + # CFLAGS=$(pkg-config --cflags openssl11) + # export CFLAGS + # LDFLAGS=$(pkg-config --libs openssl11) + # export LDFLAGS + export CFLAGS=$(pkg-config --cflags openssl11) + export LDFLAGS=$(pkg-config --libs openssl11) + else + echo "Unsupported Linux distribution." + exit 1 + fi + + local VERSION="$1" + local DOWNLOAD_URL="https://www.python.org/ftp/python/$VERSION/Python-$VERSION.tgz" + + + echo "Downloading Python $VERSION from $DOWNLOAD_URL..." + wget "$DOWNLOAD_URL" -O "/tmp/Python-$VERSION.tgz" + check_status "Failed to download Python source." "Python source downloaded successfully." $? + + echo "Extracting Python $VERSION..." + cd /tmp || exit + tar -xzf "Python-$VERSION.tgz" + cd "Python-$VERSION" || exit + + echo "Compiling and installing Python $VERSION..." + ./configure --enable-optimizations --prefix=/usr/local/python"$VERSION" + make -j"$(nproc)" + make altinstall + + cd .. + rm -rf "Python-$VERSION" "Python-$VERSION.tgz" + + MAJOR_MINOR_VERSION=$(echo "$VERSION" | cut -d '.' -f 1-2) + + # files=( + # "/usr/bin/python3" + # "/usr/bin/python" + # "/usr/bin/pip3" + # "/usr/bin/pip" + # ) + + # for file in "${files[@]}"; do + # if [ -e "$file" ]; then + # backup_file="$file.bak" + # echo "Backing up $file to $backup_file" + # cp "$file" "$backup_file" + # else + # echo "$file does not exist, skipping." + # fi + # done + + ln -sf /usr/local/python"$VERSION"/bin/python"$MAJOR_MINOR_VERSION" /usr/local/bin/python3 + ln -sf /usr/local/python"$VERSION"/bin/pip"$MAJOR_MINOR_VERSION" /usr/local/bin/pip3 + python3 --version + check_status "Failed to install Python $VERSION" "Python $VERSION installed successfully." $? +} + +upgrage_pip() { + echo "Upgrading pip..." + python3 -m pip install --upgrade pip + check_status "Failed to upgrade pip" "Pip upgraded successfully." $? +} + +# Install Python via package_manager +install_python_via_package_manager() { + if [ -n "$1" ]; then + PYTHON_PACKAGE="$1" + else + PYTHON_PACKAGE="python3" + fi + install_package "$PYTHON_PACKAGE" + install_package "python3-pip" + upgrage_pip + python3 --version +} + +# Install Python and pip +# shellcheck disable=SC2120 +install_python() { + echo -e "${YELLOW}Installing Python...${NO_COLOR}" + # Specify the major python version to search for; default is set to 3.10 if not specified + if [ -n "$1" ]; then + PYTHON_VERSION="$1" + else + install_python_via_package_manager "$PYTHON_PACKAGE" + exit 0 + fi + MAJOR_MINOR_VERSION=$(echo "$PYTHON_VERSION" | cut -d '.' -f 1-2) + # Check if the JDK package is available in the repository + if [ -f /etc/debian_version ]; then + PYTHON_PACKAGE="python${MAJOR_MINOR_VERSION}" + if apt-cache search "$PYTHON_PACKAGE" | grep -w "$PYTHON_PACKAGE"; then + install_python_via_package_manager "$PYTHON_PACKAGE" + exit 0 + else + echo -e "${RED}Failed to install Python using package manager.${NO_COLOR}" + fi + elif [ -f /etc/redhat-release ]; then + PYTHON_PACKAGE="python${MAJOR_MINOR_VERSION//./}" + if yum list available | grep -w "$PYTHON_PACKAGE"; then + install_python_via_package_manager "$PYTHON_PACKAGE" + exit 0 + else + echo -e "${RED}Failed to install Python using package manager.${NO_COLOR}" + fi + else + echo "Unsupported Linux distribution." + exit 1 + fi + + echo -e "${YELLOW}$PYTHON_PACKAGE not found in source repository. Attempting to download and install manually....${NO_COLOR}" + download_and_compile_python "$PYTHON_VERSION" + upgrage_pip + + # Check installation status + INSTALLED_VERSION=$(python3 --version 2>&1) + if echo "$INSTALLED_VERSION" | grep -q "$MAJOR_MINOR_VERSION"; then + echo -e "${GREEN}Python $MAJOR_MINOR_VERSION installed successfully.${NO_COLOR}" + else + echo -e "${YELLOW}Python version not match.${NO_COLOR}" + exit 1 + fi +} + +update_redhat_gcc() { + echo "Updating the system..." + update + + echo "Installing dependencies..." + yum groupinstall -y "Development Tools" + install_package gmp-devel mpfr-devel libmpc-devel wget + + echo "Downloading GCC $GCC_VERSION..." + cd /usr/local/src || exit + wget https://ftp.gnu.org/gnu/gcc/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.gz + wget https://ftp.gnu.org/gnu/gcc/gcc-$GCC_VERSION/gcc-$GCC_VERSION.tar.gz.sig + + echo "Extracting GCC $GCC_VERSION..." + tar -xzf gcc-$GCC_VERSION.tar.gz + cd gcc-$GCC_VERSION || exit + + echo "Downloading necessary dependencies for GCC..." + ./contrib/download_prerequisites + + mkdir build + cd build || exit + + echo "Configuring GCC..." + ../configure --enable-languages=c,c++ --disable-multilib --prefix=/usr + + echo "Compiling GCC, this may take a while..." + make -j"$(nproc)" + make install + + echo "Cleaning up downloaded files..." + cd /usr/local/src || exit + rm -rf gcc-$GCC_VERSION gcc-$GCC_VERSION.tar.gz gcc-$GCC_VERSION.tar.gz.sig + echo "Cleanup completed." + + echo "GCC installation completed. Verifying installation..." + gcc --version + check_status "Failed to install GCC" "GCC $GCC_VERSION installed successfully." $? + + # Backup + if [ -f "/lib64/libstdc++.so.6.0.28-gdb.py" ]; then + # Copy the file + mv -f /lib64/libstdc++.so.6.0.28-gdb.py /tmp/libstdc++.so.6.0.28-gdb.py + echo "File has been successfully moved to /tmp/libstdc++.so.6.0.28-gdb.py" + else + echo "File /lib64/libstdc++.so.6.0.28-gdb.py does not exist, cannot perform copy operation." + fi +} + +update_redhat_tmux() { + echo "Downloading the latest version of tmux..." + cd /usr/local/src || exit + latest_tmux_version=$(curl -s https://api.github.com/repos/tmux/tmux/releases/latest | grep -Po '"tag_name": "\K.*?(?=")') + wget https://github.com/tmux/tmux/releases/download/"${latest_tmux_version}"/tmux-"${latest_tmux_version}".tar.gz + + echo "Extracting tmux ${latest_tmux_version}..." + tar -xzf tmux-"${latest_tmux_version}".tar.gz + cd tmux-"${latest_tmux_version}" || exit + + echo "Configuring tmux..." + ./configure --prefix=/usr + + echo "Compiling tmux, this may take a while..." + make -j"$(nproc)" + make install + + echo "Cleaning up downloaded files..." + cd /usr/local/src || exit + rm -rf tmux-"${latest_tmux_version}" tmux-"${latest_tmux_version}".tar.gz + + echo "Cleanup completed." + + echo "tmux installation completed. Verifying installation..." + tmux -V + check_status "Failed to install tmux" "tmux ${latest_tmux_version} installed successfully." $? +} + +deploy_tmux() { + if [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + update_redhat_tmux + fi + echo "Copying configuration file..." + + cp "$SCRIPT_DIR/../conf/tmux.conf" ~/.tmux.conf + + echo "Configuration file copied to ~/.tmux.conf." +} + + +# Install Java +# install_java() { +# if command -v java >/dev/null 2>&1; then +# echo "Java is already installed. Skipping installation." +# java -version +# return +# else +# echo "Installing $JDK_VERSION..." +# install_package "$JDK_VERSION" +# check_status "Failed to install Java" "Install Java successfully" $? +# fi +# } + +# Install Java +install_java() { + echo -e "${YELLOW}Installing Java...${NO_COLOR}" + # Specify the major JDK version to search for; default is set to 17 if not specified + if [ -n "$1" ]; then + DEFAULT_JDK_VERSION="$1" + else + DEFAULT_JDK_VERSION="17" + fi + + # Check if the JDK package is available in the repository + if [ -f /etc/debian_version ]; then + JDK_PACKAGE="openjdk-$DEFAULT_JDK_VERSION-jdk" + if apt-cache search "$JDK_PACKAGE" | grep -q "$JDK_PACKAGE"; then + echo "Installing $JDK_PACKAGE using apt..." + fi + elif [ -f /etc/redhat-release ]; then + JDK_PACKAGE="java-$DEFAULT_JDK_VERSION-openjdk" + if yum list available | grep -q "$JDK_PACKAGE"; then + echo "Installing $JDK_PACKAGE using yum..." + fi + else + echo "Unsupported Linux distribution." + exit 1 + fi + + # Check installation status + if ! install_package "$JDK_PACKAGE"; then + echo -e "${RED}Failed to install Java using package manager.${NO_COLOR}" + else + echo -e "${GREEN}Java installed successfully.${NO_COLOR}" + java -version + return + fi + + echo -e "${YELLOW}$JDK_PACKAGE not found in $PACKAGE_MANAGER repository. Attempting to download and install manually....${NO_COLOR}" + + # URL of the archive page to search + ARCHIVE_URL="https://jdk.java.net/archive/" + + # Get the latest minor version number + LATEST_VERSION=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s "$ARCHIVE_URL" | \ + grep -o "jdk${DEFAULT_JDK_VERSION}\.[0-9]\+\.[0-9]\+" | \ + sort -V | \ + tail -n 1) + JDK_VERSION_NUM="${LATEST_VERSION#jdk}" + # Confirm the latest version found + if [[ $LATEST_VERSION =~ jdk([0-9]+)\.([0-9]+)\.([0-9]+) ]]; then + # Print the latest version found + echo -e "${YELLOW}Latest JDK version found: $LATEST_VERSION${NO_COLOR}" + MATCH_URL="https://download.java.net/java/GA/${LATEST_VERSION}/[^ ]*linux-x64_bin.tar.gz" + JDK_DOWNLOAD_URL=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s "$ARCHIVE_URL" | \ + grep -o "$MATCH_URL" | \ + head -n 1) + else + echo -e "${RED}Failed to find the JDK version: $LATEST_VERSION.${NO_COLOR}" + exit 1 + fi + # Download JDK + echo "Downloading OpenJDK $LATEST_VERSION from $JDK_DOWNLOAD_URL..." + wget "$JDK_DOWNLOAD_URL" -O /tmp/"${LATEST_VERSION}"_linux-x64_bin.tar.gz + check_status "Failed to download OpenJDK." "OpenJDK downloaded successfully." $? + + # Extract and install + echo "Extracting OpenJDK..." + if [ -d "/usr/local/jdk-${JDK_VERSION_NUM}" ]; then + rm -rf "/usr/local/jdk-${JDK_VERSION_NUM}" + fi + tar -xzf /tmp/"${LATEST_VERSION}"_linux-x64_bin.tar.gz -C /usr/local/ + rm -rf /tmp/"${LATEST_VERSION}"_linux-x64_bin.tar.gz + # Configure environment variables + echo "Configuring environment variables..." + add_config_if_not_exist "export JAVA_HOME=/usr/local/jdk-${JDK_VERSION_NUM}" "$BASH_RC" + add_config_if_not_exist "export PATH=\$PATH:\$JAVA_HOME/bin" "$BASH_RC" + # shellcheck source=/dev/null + export JAVA_HOME=/usr/local/jdk-${JDK_VERSION_NUM} + export PATH=$PATH:$JAVA_HOME/bin + INSTALLED_VERSION=$("$JAVA_HOME"/bin/java --version 2>&1) + if echo "$INSTALLED_VERSION" | grep -q "openjdk $DEFAULT_JDK_VERSION"; then + echo -e "${GREEN}Java installed successfully.${NO_COLOR}" + else + echo -e "${YELLOW}Java version not match.${NO_COLOR}" + exit 1 + fi +} + +# Install sdkman +install_sdkman() { + install_package zip unzip + if [ -d "$HOME/.sdkman" ]; then + echo -e "${GREEN}SDKMAN is already installed.${NO_COLOR}" + else + echo -e "${YELLOW}Installing SDKMAN...${NO_COLOR}" + curl -s "https://get.sdkman.io" | bash + fi + +} + +# Install Maven +# shellcheck disable=SC2120 +install_maven() { + echo -e "${YELLOW}Installing maven...${NO_COLOR}" + if [ -n "$1" ]; then + DEFAULT_MVN_VERSION="$1" + install_sdkman + if [ -f "$HOME/.sdkman/bin/sdkman-init.sh" ]; then + source "$HOME/.sdkman/bin/sdkman-init.sh" + fi + # 3.2.5 + sdk install maven "$DEFAULT_MVN_VERSION" + else + install_package "maven" + fi + mvn -version + check_status "Failed to install maven" "Maven installed successfully." $? +} + +# Install Go +deploy_go() { + # Define the installation location for Go + GO_INSTALL_DIR="/usr/local/go" + GOPATH_DIR="/root/go" + + # Check if Go is installed + if command -v go >/dev/null 2>&1; then + echo "Go is already installed. Skipping installation." + return + fi + + # Fetch the latest version number of Go + GO_LATEST_DATA=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s https://golang.google.cn/VERSION?m=text) + GO_LATEST_VERSION=$(echo "$GO_LATEST_DATA" | grep -oP 'go[0-9]+\.[0-9]+\.[0-9]+') + # Download and install the latest version of Go + echo "Installing $GO_LATEST_VERSION..." + wget https://golang.google.cn/dl/$GO_LATEST_VERSION.linux-amd64.tar.gz -O go.tar.gz + + # Extract to the specified directory + tar -C /usr/local -xzf go.tar.gz + rm -rf go.tar.gz + + # Configure environment variables using the helper function + add_config_if_not_exist "export GOROOT=$GO_INSTALL_DIR" "$BASH_RC" + add_config_if_not_exist "export GOPATH=$GOPATH_DIR" "$BASH_RC" + add_config_if_not_exist "export PATH=\$PATH:\$GOROOT/bin" "$BASH_RC" + add_config_if_not_exist "export GO111MODULE=on" "$BASH_RC" + add_config_if_not_exist "export GOPROXY=https://goproxy.cn,direct" "$BASH_RC" + + # Apply the environment variables + $GO_INSTALL_DIR/bin/go version + check_status "Failed to install GO" "Install GO successfully" $? +} + +# Function to install Rust and Cargo +deploy_rust() { + # Check if Rust is already installed + if ! command -v rustc &> /dev/null; then + # add_config_if_not_exist "export RUSTUP_DIST_SERVER=http://mirrors.ustc.edu.cn/rust-static" $BASH_RC + # export RUSTUP_DIST_SERVER=http://mirrors.ustc.edu.cn/rust-static + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + echo "Using APT package manager." + install_package build-essential + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Using YUM package manager." + yum groupinstall -y "Development Tools" + else + echo "Unsupported Linux distribution." + exit 1 + fi + + add_config_if_not_exist "export RUSTUP_DIST_SERVER=\"https://rsproxy.cn\"" "$BASH_RC" + add_config_if_not_exist "export RUSTUP_UPDATE_ROOT=\"https://rsproxy.cn/rustup\"" "$BASH_RC" + export RUSTUP_DIST_SERVER="https://rsproxy.cn" + export RUSTUP_UPDATE_ROOT="https://rsproxy.cn/rustup" + + echo "Rust is not installed. Installing Rust and Cargo..." + # Download and install Rust and Cargo + curl --retry 10 --retry-delay 5 --retry-max-time 120 --proto '=https' --tlsv1.2 -sSf https://rsproxy.cn/rustup-init.sh | sh -s -- -y + + echo "Cargo settings..." + marker="git-fetch-with-cli" + + if grep -qF "$marker" "$CARGO_CONFIG_FILE"; then + echo "Configuration already exists in ""$CARGO_CONFIG_FILE""." + else + echo "Adding configuration to ""$CARGO_CONFIG_FILE""." + echo "$CARGO_CONFIG" >>"$CARGO_CONFIG_FILE" + echo "Cargo config have been updated in your $CARGO_CONFIG_FILE file." + fi + + # Source the Cargo environment script to update the current shell + if [ -f "$HOME/.cargo/env" ]; then + source "$HOME/.cargo/env" + fi + # Check if the installation was successful + rustc --version + # Install cargo-make + cargo install cargo-make + check_status "Failed to install Rust" "Install Rust successfully" $? + else + echo "Rust is already installed." + fi +} + +# Update GCC for Ubuntu 18.04 +update_ubuntu_gcc_18.04() { + echo -e "${YELLOW}Updating GCC for Ubuntu 18.04...${NO_COLOR}" + install_package software-properties-common + yes | add-apt-repository ppa:ubuntu-toolchain-r/test + update + install_package gcc-9 g++-9 + update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-9 60 --slave /usr/bin/g++ g++ /usr/bin/g++-9 + check_status "Failed to update GCC" "GCC updated successfully." $? +} + +install_node_in_ubuntu18.04() { + # very slow and not test + if [ -n "$1" ]; then + DEFAULT_NODE_VERSION="$1" + else + DEFAULT_NODE_VERSION="14" + fi + NODE_DISTRO="node-v$DEFAULT_NODE_VERSION-linux-x64" + update_ubuntu_gcc_18.04 + echo "Installing Node..." + curl -O https://nodejs.org/dist/v22.0.0/node-v22.0.0.tar.gz + tar -xzf node-v22.0.0.tar.gz + cd node-v22.0.0 || exit + ./configure + make + make install +} + + +# Install pnpm +install_pnpm() { + echo -e "${YELLOW}Installing pnpm...${NO_COLOR}" + centos_skip_check + NODE_VERSION=$(node -v) + + MAJOR_VERSION=$(echo "$NODE_VERSION" | cut -d '.' -f 1 | tr -d 'v') + MINOR_VERSION=$(echo "$NODE_VERSION" | cut -d '.' -f 2) + PATCH_VERSION=$(echo "$NODE_VERSION" | cut -d '.' -f 3) + + VERSION_NUMBER=$((MAJOR_VERSION * 10000 + MINOR_VERSION * 100 + PATCH_VERSION)) + + REQUIRED_VERSION=181200 # v18.12.00 + + if [ $VERSION_NUMBER -ge $REQUIRED_VERSION ]; then + echo "Node version is $NODE_VERSION, installing pnpm..." + npm install --global pnpm + pnpm --version + check_status "Failed to install pnpm" "pnpm installed successfully." $? + else + echo "Node version is $NODE_VERSION, skipping pnpm installation." + fi + +} + +# Install Node via nvm +# shellcheck disable=SC2120 +install_node_via_nvm () { + echo -e "${YELLOW}Installing Node via NVM...${NO_COLOR}" + if [ -n "$1" ]; then + DEFAULT_NODE_VERSION="$1" + else + DEFAULT_NODE_VERSION="" + fi + + if [[ -f /etc/redhat-release ]]; then + if [[ "$1" != "16.20.2" ]]; then + centos_skip_check + fi + fi + + # Install NVM + if ! command -v nvm &> /dev/null; then + NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")') + curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/"$NVM_VERSION"/install.sh | bash + export NVM_DIR="$HOME/.nvm" + [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" + [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" + echo -e "${GREEN}NVM installed successfully.${NO_COLOR}" + else + echo -e "${GREEN}NVM is already installed.${NO_COLOR}" + fi + + # Check if version is specified + if [ -n "$1" ]; then + NODE_VERSION="$1" # use specified version + echo -e "${YELLOW}Installing Node version $NODE_VERSION...${NO_COLOR}" + nvm install "$NODE_VERSION" + else + echo -e "${YELLOW}Installing the latest version of Node...${NO_COLOR}" + nvm install node # use latest version + fi + nvm alias default node # set default version + + echo -e "${GREEN}Node installed successfully.${NO_COLOR}" + + npm config set fetch-retry-maxtimeout 120000 + npm config set fetch-retry-factor 5 + npm config set registry=https://registry.npmmirror.com + npm install --global yarn + npm install --global pnpm + # NPM_BIN_DIR=$(npm bin -g) + # if [ -d "$NPM_BIN_DIR" ]; then + # ln -sf "$NPM_BIN_DIR/yarn" /usr/bin/yarn + # ln -sf "$NPM_BIN_DIR/yarnpkg" /usr/bin/yarnpkg + # echo -e "${GREEN}Yarn installed successfully.${NO_COLOR}" + # else + # echo -e "${RED}Failed to find npm global bin directory. Yarn installation may not be complete.${NO_COLOR}" + # fi + node --version + check_status "Failed to install Node" "Node installed successfully." $? + npm --version + check_status "Failed to install npm" "npm installed successfully." $? + yarn --version + check_status "Failed to install Yarn" "Yarn installed successfully." $? +} + +# Install Node npm Yarn +# shellcheck disable=SC2120 +install_node() { + echo -e "${YELLOW}Installing Node...${NO_COLOR}" + if [ -n "$1" ]; then + DEFAULT_NODE_VERSION="$1" + if [[ -f /etc/redhat-release ]]; then + if [[ "$1" != "16.20.2" ]]; then + centos_skip_check + fi + fi + echo -e "${YELLOW}Installing Node version $DEFAULT_NODE_VERSION from source...${NO_COLOR}" + NODE_DISTRO="node-v$DEFAULT_NODE_VERSION-linux-x64" + wget "https://nodejs.org/dist/v$DEFAULT_NODE_VERSION/$NODE_DISTRO.tar.xz" -O "/tmp/$NODE_DISTRO.tar.xz" + tar -xf "/tmp/$NODE_DISTRO.tar.xz" -C /usr/local/lib/ + ln -sf "/usr/local/lib/$NODE_DISTRO/bin/node" /usr/bin/node + ln -sf "/usr/local/lib/$NODE_DISTRO/bin/npm" /usr/bin/npm + ln -sf "/usr/local/lib/$NODE_DISTRO/bin/npx" /usr/bin/npx + rm -rf "/tmp/$NODE_DISTRO.tar.xz" + node --version + echo -e "${GREEN}Node version $DEFAULT_NODE_VERSION installed successfully.${NO_COLOR}" + else + install_package "nodejs" + install_package "npm" + fi + node --version + check_status "Failed to install Node" "Node installed successfully." $? + + npm --version + check_status "Failed to install npm" "npm installed successfully." $? + npm config set fetch-retry-maxtimeout 120000 + npm config set fetch-retry-attempts + npm config set fetch-retry-factor 5 + npm config set registry=https://registry.npmmirror.com + npm install --global yarn + NPM_BIN_DIR=$(npm bin -g) + if [ -d "$NPM_BIN_DIR" ]; then + ln -sf "$NPM_BIN_DIR/yarn" /usr/bin/yarn + ln -sf "$NPM_BIN_DIR/yarnpkg" /usr/bin/yarnpkg + echo -e "${GREEN}Yarn installed successfully.${NO_COLOR}" + else + echo -e "${RED}Failed to find npm global bin directory. Yarn installation may not be complete.${NO_COLOR}" + fi + yarn --version + check_status "Failed to install Yarn" "Yarn installed successfully." $? +} + + + +# Deploy Git +deploy_git() { + install_package "git" + git --version + check_status "Failed to install Git" "Git installed successfully." $? + git config --global user.name "taos-support" + git config --global user.email "it@taosdata.com" + git config --global credential.helper store +} + +deploy_node_exporter() { + if [ ! -f "$NODE_EXPORTER_BINARY" ]; then + echo "Node Exporter is not installed. Installing now..." + + echo "Fetching the latest version of Node Exporter..." + LATEST_URL=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s https://api.github.com/repos/prometheus/node_exporter/releases/latest | jq -r '.assets[] | select(.name | test("node_exporter-.*linux-amd64.tar.gz")) | .browser_download_url') + + if [ -z "$LATEST_URL" ]; then + echo "Failed to fetch the latest Node Exporter release URL. Exiting." + exit 1 + fi + + echo "Downloading Node Exporter from $LATEST_URL..." + wget "$LATEST_URL" -O node_exporter.tar.gz + + echo "Extracting Node Exporter..." + tar -xzf node_exporter.tar.gz + cd node_exporter-*.linux-amd64 || exit + + echo "Copying binary..." + cp node_exporter /usr/local/bin/ + + echo "Creating systemd service..." + cat < /etc/systemd/system/node_exporter.service +[Unit] +Description=Node Exporter + +[Service] +ExecStart=/usr/local/bin/node_exporter + +[Install] +WantedBy=default.target +EOF + + # Start Node Exporter and enable it to run on startup + systemctl daemon-reload + systemctl start node_exporter + systemctl enable node_exporter + + # Clean up the downloaded tar to save space + cd .. + rm -rf node_exporter*.tar.gz node_exporter-*.linux-amd64 + node_exporter --version + check_status "Failed to install Node Exporter" "Node Exporter installed successfully." $? + else + echo "Node Exporter is already installed." + fi +} + +deploy_process_exporter() { + if [ ! -f "$PROCESS_EXPORTER_BINARY" ]; then + echo "Process Exporter is not installed. Installing now..." + + echo "Fetching the latest version of Process Exporter..." + LATEST_URL=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s https://api.github.com/repos/ncabatoff/process-exporter/releases/latest | jq -r '.assets[] | select(.name | test("process-exporter-.*linux-amd64.tar.gz")) | .browser_download_url') + + if [ -z "$LATEST_URL" ]; then + echo "Failed to fetch the latest Process Exporter release URL. Exiting." + exit 1 + fi + + echo "Downloading Process Exporter from $LATEST_URL..." + wget "$LATEST_URL" -O process-exporter.tar.gz + + echo "Extracting Process Exporter..." + tar -xzf process-exporter.tar.gz + cd process-exporter-*.linux-amd64 || exit + + echo "Copying binary..." + cp process-exporter /usr/local/bin/process-exporter + + echo "Creating configuration file..." + cat < /etc/process_exporter.yml +process_names: + - name: "{{.Comm}}" + cmdline: + - taosd +EOF + + echo "Creating systemd service..." + cat < /etc/systemd/system/process_exporter.service +[Unit] +Description=Process Exporter + +[Service] +ExecStart=/usr/local/bin/process-exporter --config.path /etc/process_exporter.yml + +[Install] +WantedBy=default.target +EOF + + # Start Process Exporter and enable it to run on startup + systemctl daemon-reload + systemctl start process_exporter + systemctl enable process_exporter + + # Clean up the downloaded tar to save space + cd .. + rm -rf process-exporter*.tar.gz process-exporter-*.linux-amd64 + process-exporter --version + check_status "Failed to install Process Exporter" "Process Exporter installed successfully." $? + else + echo "Process Exporter is already installed." + fi +} + +deploy_prometheus() { + # Check if Prometheus binary exists + if [ ! -f "$PROMETHEUS_BINARY" ]; then + echo "Prometheus is not installed. Installing now..." + + echo "Fetching the latest version of Prometheus..." + LATEST_URL=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s https://api.github.com/repos/prometheus/prometheus/releases/latest | jq -r '.assets[] | select(.name | test("prometheus-.*linux-amd64.tar.gz")) | .browser_download_url') + + if [ -z "$LATEST_URL" ]; then + echo "Failed to fetch the latest Prometheus release URL. Exiting." + exit 1 + fi + + echo "Downloading Prometheus from $LATEST_URL..." + wget "$LATEST_URL" -O prometheus.tar.gz + + echo "Extracting Prometheus..." + tar -xzf prometheus.tar.gz + cd prometheus-*.linux-amd64 || exit + + echo "Creating directories..." + mkdir -p /etc/prometheus /var/lib/prometheus + + echo "Copying binaries and configuration..." + cp prometheus promtool /usr/local/bin/ + + echo "Setting up Prometheus configuration..." + cat < /etc/prometheus/prometheus.yml +global: + scrape_interval: 15s +scrape_configs: + - job_name: 'node_exporter' + static_configs: + - targets: ['localhost:9100'] +EOF + + echo "Creating systemd service..." + cat < /etc/systemd/system/prometheus.service +[Unit] +Description=Prometheus Service + +[Service] +ExecStart=/usr/local/bin/prometheus \\ + --config.file /etc/prometheus/prometheus.yml \\ + --storage.tsdb.path /var/lib/prometheus/ \\ + --web.console.templates=/etc/prometheus/consoles \\ + --web.console.libraries=/etc/prometheus/console_libraries + +[Install] +WantedBy=default.target +EOF + + # Start Prometheus and enable it to run on startup + systemctl daemon-reload + systemctl start prometheus + systemctl enable prometheus + + # Clean up the downloaded tar to save space + cd .. + rm -rf prometheus*.tar.gz prometheus-*.linux-amd64 + prometheus --version + check_status "Failed to install Prometheus" "Prometheus installed successfully." $? + else + echo "Prometheus is already installed." + fi +} + +# Install Grafana using a downloaded .deb package +deploy_grafana() { + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + deploy_debian_grafana + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + deploy_redhat_grafana + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +# Install Grafana for ubuntu/debian +deploy_debian_grafana() { + # Check if Grafana is already installed + if ! dpkg -s "grafana" &> /dev/null; then + echo "Downloading the latest Grafana .deb package..." + # Download the latest Grafana .deb package + wget https://dl.grafana.com/oss/release/grafana_latest_amd64.deb -O grafana.deb + # install the required fontconfig package + install_package libfontconfig1 + echo "Installing Grafana..." + # Install the .deb package + dpkg -i grafana.deb + + # Clean up the downloaded .deb package to save space + rm -rf grafana.deb + + # Start the Grafana server and enable it to run on startup + systemctl start grafana-server + systemctl enable grafana-server + + # Check if Grafana was installed successfully + if [ $? -eq 0 ]; then + echo "Grafana was installed successfully." + else + echo "Failed to install Grafana." + fi + else + echo "Grafana is already installed." + fi +} + +# Install Grafana for centos/redhat +deploy_redhat_grafana() { + # Check if Grafana is already installed + if ! rpm -q grafana &> /dev/null; then + echo "Downloading the latest Grafana .rpm package..." + # Download the latest Grafana .rpm package + wget https://dl.grafana.com/oss/release/grafana-8.5.2-1.x86_64.rpm -O grafana.rpm + + # Install the required fontconfig package + yum install -y fontconfig + + echo "Installing Grafana..." + # Install the .rpm package + rpm -ivh grafana.rpm + + # Clean up the downloaded .rpm package to save space + rm -rf grafana.rpm + + # Start the Grafana server and enable it to run on startup + systemctl start grafana-server + systemctl enable grafana-server + + # Check if Grafana was installed successfully + if [ $? -eq 0 ]; then + echo "Grafana was installed successfully." + else + echo "Failed to install Grafana." + fi + else + echo "Grafana is already installed." + fi +} + +# Install Nginx +install_nginx() { + install_package "nginx" + nginx -v + check_status "Failed to install Nginx" "Nginx installed successfully." $? +} + +# Deploy JMeter +deploy_jmeter() { + if ! command -v jmeter &> /dev/null; then + echo "Installing JMeter..." + install_java + wget -P /opt https://mirrors.aliyun.com/apache/jmeter/binaries/apache-jmeter-$JMETER_VERSION.tgz + tar -xvzf /opt/apache-jmeter-$JMETER_VERSION.tgz -C /opt/ + ln -sf /opt/apache-jmeter-$JMETER_VERSION/bin/jmeter /usr/local/bin/jmeter + rm -rf /opt/apache-jmeter-$JMETER_VERSION.tgz + jmeter --version + check_status "Failed to install JMeter" "JMeter installed successfully." $? + else + echo "JMeter is already installed." + fi +} + +# Deploy Docker +deploy_docker() { + if [ -f /etc/debian_version ]; then + # Debian or Ubuntu + deploy_debian_docker + elif [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + deploy_redhat_docker + else + echo "Unsupported Linux distribution." + exit 1 + fi +} + +# Deploy Docker for centos/redhat +deploy_redhat_docker() { + # Check if Docker is already installed + if ! command -v docker &> /dev/null; then + echo "Docker is not installed. Installing now..." + + # Set up the repository for Docker + echo "Setting up the Docker repository..." + install_package yum-utils + + echo "Adding Docker's official repository..." + yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo + + # Install Docker CE + echo "Installing Docker CE..." + install_package docker-ce docker-ce-cli containerd.io + + # Enable and start Docker + echo "Enabling and starting Docker..." + systemctl enable docker + systemctl start docker + + # Adding current user to the Docker group + usermod -aG docker "$USER" + + # Print Docker version + docker --version + + # Check the installation status + if [ $? -eq 0 ]; then + echo "Docker installed successfully." + else + echo "Failed to install Docker." + fi + else + echo "Docker is already installed." + fi +} + +# Deploy docker for ubuntu/debian +deploy_debian_docker() { + # Check if Docker is already installed + if ! command -v docker &> /dev/null; then + echo "Docker is not installed. Installing now..." + + # Set up the repository for Docker + echo "Setting up the Docker repository..." + update + install_package apt-transport-https ca-certificates curl software-properties-common gnupg lsb-release + + echo "Adding Docker's official GPG key from Aliyun..." + curl --retry 10 --retry-delay 5 --retry-max-time 120 -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | { + if [ -f /usr/share/keyrings/docker-archive-keyring.gpg ]; then + rm /usr/share/keyrings/docker-archive-keyring.gpg + fi + gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg + } + echo "Setting up stable repository using Aliyun..." + echo \ + "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://mirrors.aliyun.com/docker-ce/linux/ubuntu \ + $(lsb_release -cs) stable" | tee /etc/apt/sources.list.d/docker.list > /dev/null + + # Install Docker CE + echo "Installing Docker CE..." + update + install_package docker-ce docker-ce-cli containerd.io + + # Enable and start Docker + echo "Enabling and starting Docker..." + systemctl enable docker + systemctl start docker + + # Adding current user to the Docker group + usermod -aG docker "$USER" + + # Print Docker version + docker --version + check_status "Failed to install Docker" "Docker installed successfully." $? + else + echo "Docker is already installed." + fi +} + +# Deploy Docker Compose +deploy_docker_compose() { + # Check if Docker Compose is installed + if ! command -v docker-compose &> /dev/null; then + echo "Docker Compose is not installed. Installing now..." + + # Install Docker Compose + curl --retry 10 --retry-delay 5 --retry-max-time 120 -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose + chmod +x /usr/local/bin/docker-compose + + # Print Docker Compose version + docker-compose --version + check_status "Failed to install Docker Compose" "Docker Compose installed successfully." $? + else + echo "Docker Compose is already installed." + fi +} + +# Reconfigure cloud-init +reconfig_cloud_init() { + echo "Reconfiguring cloud-init..." + apt remove -y cloud-init && apt purge -y cloud-init + rm -rf /var/lib/cloud /etc/cloud + apt update -y + install_package cloud-init + sed -i '/package[-_]update[-_]upgrade[-_]install/s/^/#/' /etc/cloud/cloud.cfg +} + +# Config cloud-init +config_cloud_init() { + if [ "$DIST_VERSION" = "7.9" ];then + install_package "cloud-init" + sed -i '/ssh_pwauth.*/s/^/#/' /etc/cloud/cloud.cfg + else + reconfig_cloud_init + fi + check_status "Failed to configure cloud-init" "Cloud-init configured successfully and you need reboot manually." $? + # if [ "$DIST_VERSION" = "18.04" ] || [ "$DIST_VERSION" = "20.04" ]; then + # reconfig_cloud_init + # elif [ "$DIST_VERSION" = "7.9" ];then + # install_package "cloud-init" + # sed -i '/ssh_pwauth.*/s/^/#/' /etc/cloud/cloud.cfg + # else + # echo "Configuring cloud-init..." + # add_config_if_not_exist "$CLOUD_INIT_CONFIG" "/etc/cloud/cloud.cfg" + + # marker="NoCloud" + + # if grep -qF "$marker" "/etc/cloud/cloud.cfg"; then + # echo "cloud-init settings already exists in /etc/cloud/cloud.cfg." + # else + # echo "Adding configuration to /etc/cloud/cloud.cfg." + # echo "$CLOUD_INIT_CONFIG" >> "/etc/cloud/cloud.cfg" + # echo "cloud-init settings have been updated in /etc/cloud/cloud.cfg." + # fi + + # mkdir -p /var/lib/cloud/seed/nocloud/ + # cd /var/lib/cloud/seed/nocloud/ || exit + # touch meta-data + # touch user-data + # add_config_if_not_exist "hostname: \${name}" user-data + # add_config_if_not_exist "manage_etc_hosts: true" user-data + # fi + # cloud-init clean --logs +} + +# Clone a repository with a specified target directory +clone_repo_with_rename() { + local repo_url="$1" + local target_dir="$2" + local branch_name="$3" + + if [ -z "$target_dir" ]; then + target_dir=$(basename -s .git "$repo_url") + fi + + cd "$REPOPATH" || exit + + if [ -d "$target_dir" ]; then + echo "Directory $target_dir already exists. Skipping clone." + else + echo "Cloning into $target_dir..." + if [ -n "$branch_name" ]; then + git clone -b "$branch_name" "$repo_url" "$target_dir" + else + git clone "$repo_url" "$target_dir" + fi + fi +} + +# Clone enterprise repository +clone_enterprise() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com/taosdata/TDinternal + clone_repo_with_rename git@github.com:taosdata/TDengine.git TDinternal/community +} + +# Clone community repository +clone_community() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com:taosdata/TDengine.git +} + +# Clone TaosX repository +clone_taosx() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com/taosdata/taosx +} + +# Clone TaosKeeper repository +clone_taoskeeper() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com/taosdata/taoskeeper +} + +# Clone TaosTest repository +clone_taostest() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com/taosdata/taos-test-framework "" "master" +} + +# Clone TestNG repository +clone_testng() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com/taosdata/TestNG "" "master" +} + +# Clone operation tools repository +clone_operation() { + cd "$REPOPATH" || exit + clone_repo_with_rename https://github.com/taosdata/operation.git +} + +# init system +system_config() { + disable_service + config_dns + replace_sources + config_cloud_init + config_ssh + config_custom_settings + config_timezone + config_share_server + disable_firewalld + config_frontend + config_system_limits + config_coredump + check_status "Failed to config system" "Config system successfully" $? +} + +# Clone all the repositories +clone_repos() { + clone_enterprise + clone_community + clone_taosx + clone_taoskeeper + clone_taostest + clone_operation +} + +new_funcs() { + echo "Adding test..." + install_python 3.10.12 + # install_java 21 + # install_node 16.20.2 + # install_maven 3.2.5 +} + +# deploy TDasset +TDasset() { + install_java 21 + install_maven 3.9.9 + # not supported in centos7 because of the old version of glibc + # install_node 22.0.0 + install_node_via_nvm 22.0.0 + install_pnpm +} + +# deploy TDinternal/TDengine/taosx +TDinternal() { + deploy_go + deploy_rust + install_java 17 + install_node_via_nvm 16.20.2 + install_python 3.10.12 +} + +# deploy TDgpt +TDgpt() { + install_python 3.10.12 +} + +# deploy taos-test-framework +taostest() { + if [ ! -d "$REPOPATH/taos-test-framework" ]; then + echo "Cloning TaosTest repository..." + clone_taostest + else + echo "TaosTest repository already exists. Skipping clone." + fi + check_status "Failed to clone TaosTest repository" "TaosTest repository cloned successfully." $? + + if [ ! -d "$REPOPATH/TestNG" ]; then + echo "Cloning TestNG repository..." + clone_testng + else + echo "TestNG repository already exists. Skipping clone." + fi + check_status "Failed to clone TestNG repository" "TestNG repository cloned successfully." $? + + # Configure environment variables + echo "Configuring TaosTest environment variables..." + mkdir -p "$HOME"/.taostest + add_config_if_not_exist "TEST_ROOT=$REPOPATH/TestNG" "$HOME"/.taostest/.env + + # Install TaosTest + echo "Installing TaosTest..." + cd "$REPOPATH"/taos-test-framework || exit + install_package "python3-pip" + install_via_pip "poetry" + yes | ./reinstall.sh + check_status "Failed to install TaosTest" "TaosTest installed successfully." $? + + # Configure passwdless login + echo "Configuring passwdless login..." + yes | ssh-keygen -t rsa -b 2048 -N "" -f "$HOME/.ssh/testng" + cat "$HOME"/.ssh/testng.pub >> "$HOME"/.ssh/authorized_keys +} + + +# Deploy pure environment +deploy_pure() { + disable_service + config_dns + if [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + echo "Replacing sources for YUM package manager." + replace_yum_sources + fi + config_cloud_init + config_ssh + config_custom_settings + config_timezone + config_share_server + disable_firewalld + install_package "jq" + install_package "wget" + deploy_node_exporter + check_status "Failed to config pure system" "Config pure system successfully" $? +} + +# Deploy development environment +deploy_dev() { + install_packages + deploy_cmake + if [ -f /etc/redhat-release ]; then + # Red Hat or CentOS + update_redhat_gcc + fi + deploy_tmux + deploy_git + install_python + install_pip_pkg + install_java + install_maven + deploy_go + deploy_rust + install_node + deploy_node_exporter + deploy_process_exporter + deploy_prometheus + deploy_grafana + deploy_jmeter + install_nginx + deploy_docker + deploy_docker_compose + check_status "Failed to deploy some tools" "Deploy all tools successfully" $? +} + +# Setup all configurations +setup_all() { + system_config + deploy_dev +} + +# More installation functions can be added here following the above examples + +# Main execution function +main() { + # Check if at least one argument is provided + if [ $# -eq 0 ]; then + echo "Error: No arguments provided." + echo "Please try $0 --help" + exit 1 + fi + init_env + for arg in "$@"; do + case $arg in + --help) + help + exit 0 + ;; + setup_all) + setup_all + ;; + config_ssh) + config_ssh + ;; + disable_firewalld) + disable_firewalld + ;; + config_cloud_init) + config_cloud_init + ;; + deploy_git) + deploy_git + ;; + replace_sources) + replace_sources + ;; + upgrade) + upgrade + ;; + config_timezone) + config_timezone + ;; + config_dns) + config_dns + ;; + config_custom_settings) + config_custom_settings + ;; + install_packages) + install_packages + ;; + config_system_limits) + config_system_limits + ;; + config_coredump) + config_coredump + ;; + disable_service) + disable_service + ;; + install_python) + install_python + ;; + install_pip_pkg) + install_pip_pkg + ;; + install_java) + install_java + ;; + install_maven) + install_maven + ;; + deploy_cmake) + deploy_cmake + ;; + update_redhat_gcc) + update_redhat_gcc + ;; + update_redhat_tmux) + update_redhat_tmux + ;; + deploy_tmux) + deploy_tmux + ;; + deploy_go) + deploy_go + ;; + deploy_rust) + deploy_rust + ;; + install_node) + install_node + ;; + install_node_via_nvm) + install_node_via_nvm + ;; + install_pnpm) + install_pnpm + ;; + deploy_node_exporter) + deploy_node_exporter + ;; + deploy_process_exporter) + deploy_process_exporter + ;; + deploy_prometheus) + deploy_prometheus + ;; + deploy_grafana) + deploy_grafana + ;; + deploy_jmeter) + deploy_jmeter + ;; + install_nginx) + install_nginx + ;; + config_qemu_guest_agent) + config_qemu_guest_agent + ;; + config_share_server) + config_share_server + ;; + deploy_docker) + deploy_docker + ;; + deploy_docker_compose) + deploy_docker_compose + ;; + clone_enterprise) + clone_enterprise + ;; + clone_community) + clone_community + ;; + clone_taosx) + clone_taosx + ;; + clone_taoskeeper) + clone_taoskeeper + ;; + clone_taostest) + clone_taostest + ;; + clone_operation) + clone_operation + ;; + system_config) + system_config + ;; + deploy_pure) + deploy_pure + ;; + deploy_dev) + deploy_dev + ;; + TDasset) + TDasset + ;; + TDinternal) + TDinternal + ;; + TDgpt) + TDgpt + ;; + taostest) + taostest + ;; + new_funcs) + new_funcs + ;; + *) + echo "Unknown function: $arg" + ;; + esac + done +} + +# Execute the script with specified function arguments +main "$@" From 8dedf8f21d723177aef8441785791747224523aa Mon Sep 17 00:00:00 2001 From: Shengliang Guan Date: Sun, 29 Dec 2024 22:34:24 +0800 Subject: [PATCH 04/73] minor changes --- source/libs/parser/src/parTranslater.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/libs/parser/src/parTranslater.c b/source/libs/parser/src/parTranslater.c index 87c648989e..8f23daf8dd 100755 --- a/source/libs/parser/src/parTranslater.c +++ b/source/libs/parser/src/parTranslater.c @@ -7782,7 +7782,7 @@ static int32_t buildCreateDbReq(STranslateContext* pCxt, SCreateDatabaseStmt* pS static int32_t checkRangeOption(STranslateContext* pCxt, int32_t code, const char* pName, int64_t val, int64_t minVal, int64_t maxVal, bool skipUndef) { - if (skipUndef ? ((val >= 0 | val < -2) && (val < minVal || val > maxVal)) : (val < minVal || val > maxVal)) { + if (skipUndef ? ((val >= 0 || val < -2) && (val < minVal || val > maxVal)) : (val < minVal || val > maxVal)) { return generateSyntaxErrMsgExt(&pCxt->msgBuf, code, "Invalid option %s: %" PRId64 ", valid range: [%" PRId64 ", %" PRId64 "]", pName, val, minVal, maxVal); From 65330a5b47637a6546e2c3d50836e02f3be53b62 Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Sun, 29 Dec 2024 22:41:49 +0800 Subject: [PATCH 05/73] enh: update --- packaging/setup_env.sh | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh index cce54b2839..42ae6b3814 100644 --- a/packaging/setup_env.sh +++ b/packaging/setup_env.sh @@ -1761,8 +1761,7 @@ new_funcs() { TDasset() { install_java 21 install_maven 3.9.9 - # not supported in centos7 because of the old version of glibc - # install_node 22.0.0 + # not supported in centos7/ubuntu18 because of the old version of glibc install_node_via_nvm 22.0.0 install_pnpm } From cf7701af6722f23a2ce56b14391cf54a669e8f91 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 09:47:03 +0800 Subject: [PATCH 06/73] fix: limit can not over childtable_count --- .../5-taos-tools/taosbenchmark/json/custom_col_tag.json | 2 +- .../5-taos-tools/taosbenchmark/json/taosc_insert_alltypes.json | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/develop-test/5-taos-tools/taosbenchmark/json/custom_col_tag.json b/tests/develop-test/5-taos-tools/taosbenchmark/json/custom_col_tag.json index fec5775cd6..b844f7b1b8 100644 --- a/tests/develop-test/5-taos-tools/taosbenchmark/json/custom_col_tag.json +++ b/tests/develop-test/5-taos-tools/taosbenchmark/json/custom_col_tag.json @@ -36,7 +36,7 @@ "insert_mode": "taosc", "line_protocol": "line", "childtable_limit": -10, - "childtable_offset": 10, + "childtable_offset": 0, "insert_rows": 20, "insert_interval": 0, "interlace_rows": 0, diff --git a/tests/develop-test/5-taos-tools/taosbenchmark/json/taosc_insert_alltypes.json b/tests/develop-test/5-taos-tools/taosbenchmark/json/taosc_insert_alltypes.json index 5694b58407..5ba870a3bd 100644 --- a/tests/develop-test/5-taos-tools/taosbenchmark/json/taosc_insert_alltypes.json +++ b/tests/develop-test/5-taos-tools/taosbenchmark/json/taosc_insert_alltypes.json @@ -36,7 +36,7 @@ "insert_mode": "taosc", "line_protocol": "line", "childtable_limit": -10, - "childtable_offset": 10, + "childtable_offset": 0, "insert_rows": 20, "insert_interval": 0, "interlace_rows": 0, From 118b2b98779b2f248db8a4a2586a5ecb3c6e6702 Mon Sep 17 00:00:00 2001 From: t_max <1172915550@qq.com> Date: Mon, 30 Dec 2024 09:34:50 +0800 Subject: [PATCH 07/73] docs: go connector support native stmt2 binding --- docs/en/07-develop/05-stmt.md | 10 +++ docs/en/14-reference/05-connector/20-go.md | 37 +++++++++ docs/examples/go/stmt2/native/main.go | 84 +++++++++++++++++++++ docs/zh/07-develop/05-stmt.md | 11 +++ docs/zh/14-reference/05-connector/20-go.mdx | 31 ++++++++ 5 files changed, 173 insertions(+) create mode 100644 docs/examples/go/stmt2/native/main.go diff --git a/docs/en/07-develop/05-stmt.md b/docs/en/07-develop/05-stmt.md index 4503bb8bd3..614d867c9e 100644 --- a/docs/en/07-develop/05-stmt.md +++ b/docs/en/07-develop/05-stmt.md @@ -96,9 +96,19 @@ This is a [more detailed parameter binding example](https://github.com/taosdata/ + +The example code for binding parameters with stmt2 (Go connector v3.6.0 and above, TDengine v3.3.5.0 and above) is as follows: + +```go +{{#include docs/examples/go/stmt2/native/main.go}} +``` + +The example code for binding parameters with stmt is as follows: + ```go {{#include docs/examples/go/stmt/native/main.go}} ``` + diff --git a/docs/en/14-reference/05-connector/20-go.md b/docs/en/14-reference/05-connector/20-go.md index dd32df2c5b..aecad76a2e 100644 --- a/docs/en/14-reference/05-connector/20-go.md +++ b/docs/en/14-reference/05-connector/20-go.md @@ -493,6 +493,43 @@ The `af` package provides more interfaces using native connections for parameter * **Interface Description**: Closes the statement. * **Return Value**: Error information. +From version 3.6.0, the `stmt2` interface for binding parameters is provided. + +* `func (conn *Connector) Stmt2(reqID int64, singleTableBindOnce bool) *Stmt2` + * **Interface Description**: Returns a Stmt2 object bound to this connection. + * **Parameter Description**: + * `reqID`: Request ID. + * `singleTableBindOnce`: Indicates whether a single child table is bound only once during a single execution. + * **Return Value**: Stmt2 object. + +* `func (s *Stmt2) Prepare(sql string) error` + * **Interface Description**: Prepares an SQL. + * **Parameter Description**: + * `sql`: The statement for parameter binding. + * **Return Value**: Error information. + +* `func (s *Stmt2) Bind(params []*stmt.TaosStmt2BindData) error` + * **Interface Description**: Binds data to the prepared statement. + * **Parameter Description**: + * `params`: The data to bind. + * **Return Value**: Error information. + +* `func (s *Stmt2) Execute() error` + * **Interface Description**: Executes the batch. + * **Return Value**: Error information. + +* `func (s *Stmt2) GetAffectedRows() int` + * **Interface Description**: Gets the number of affected rows (only valid for insert statements). + * **Return Value**: Number of affected rows. + +* `func (s *Stmt2) UseResult() (driver.Rows, error)` + * **Interface Description**: Retrieves the result set (only valid for query statements). + * **Return Value**: Result set Rows object, error information. + +* `func (s *Stmt2) Close() error` + * **Interface Description**: Closes the statement. + * **Return Value**: Error information. + The `ws/stmt` package provides interfaces for parameter binding via WebSocket * `func (c *Connector) Init() (*Stmt, error)` diff --git a/docs/examples/go/stmt2/native/main.go b/docs/examples/go/stmt2/native/main.go new file mode 100644 index 0000000000..fc3ab763c2 --- /dev/null +++ b/docs/examples/go/stmt2/native/main.go @@ -0,0 +1,84 @@ +package main + +import ( + "database/sql/driver" + "fmt" + "log" + "math/rand" + "time" + + "github.com/taosdata/driver-go/v3/af" + "github.com/taosdata/driver-go/v3/common" + "github.com/taosdata/driver-go/v3/common/stmt" +) + +func main() { + host := "127.0.0.1" + numOfSubTable := 10 + numOfRow := 10 + db, err := af.Open(host, "root", "taosdata", "", 0) + if err != nil { + log.Fatalln("Failed to connect to " + host + "; ErrMessage: " + err.Error()) + } + defer db.Close() + // prepare database and table + _, err = db.Exec("CREATE DATABASE IF NOT EXISTS power") + if err != nil { + log.Fatalln("Failed to create database power, ErrMessage: " + err.Error()) + } + _, err = db.Exec("USE power") + if err != nil { + log.Fatalln("Failed to use database power, ErrMessage: " + err.Error()) + } + _, err = db.Exec("CREATE STABLE IF NOT EXISTS meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (groupId INT, location BINARY(24))") + if err != nil { + log.Fatalln("Failed to create stable meters, ErrMessage: " + err.Error()) + } + // prepare statement + sql := "INSERT INTO ? USING meters TAGS(?,?) VALUES (?,?,?,?)" + reqID := common.GetReqID() + stmt2 := db.Stmt2(reqID, false) + err = stmt2.Prepare(sql) + if err != nil { + log.Fatalln("Failed to prepare sql, sql: " + sql + ", ErrMessage: " + err.Error()) + } + for i := 1; i <= numOfSubTable; i++ { + // generate column data + current := time.Now() + columns := make([][]driver.Value, 4) + for j := 0; j < numOfRow; j++ { + columns[0] = append(columns[0], current.Add(time.Millisecond*time.Duration(j))) + columns[1] = append(columns[1], rand.Float32()*30) + columns[2] = append(columns[2], rand.Int31n(300)) + columns[3] = append(columns[3], rand.Float32()) + } + // generate bind data + tableName := fmt.Sprintf("d_bind_%d", i) + tags := []driver.Value{int32(i), []byte(fmt.Sprintf("location_%d", i))} + bindData := []*stmt.TaosStmt2BindData{ + { + TableName: tableName, + Tags: tags, + Cols: columns, + }, + } + // bind params + err = stmt2.Bind(bindData) + if err != nil { + log.Fatalln("Failed to bind params, ErrMessage: " + err.Error()) + } + // execute batch + err = stmt2.Execute() + if err != nil { + log.Fatalln("Failed to exec, ErrMessage: " + err.Error()) + } + // get affected rows + affected := stmt2.GetAffectedRows() + // you can check exeResult here + fmt.Printf("Successfully inserted %d rows to %s.\n", affected, tableName) + } + err = stmt2.Close() + if err != nil { + log.Fatal("failed to close statement, err:", err) + } +} diff --git a/docs/zh/07-develop/05-stmt.md b/docs/zh/07-develop/05-stmt.md index 74b44ba8e6..2cc5413a03 100644 --- a/docs/zh/07-develop/05-stmt.md +++ b/docs/zh/07-develop/05-stmt.md @@ -91,9 +91,20 @@ import TabItem from "@theme/TabItem"; ``` + +stmt2 绑定参数的示例代码如下（go 连接器 v3.6.0 及以上，TDengine v3.3.5.0 及以上）： + +```go +{{#include docs/examples/go/stmt2/native/main.go}} +``` + +stmt 绑定参数的示例代码如下： + ```go {{#include docs/examples/go/stmt/native/main.go}} ``` + + diff --git a/docs/zh/14-reference/05-connector/20-go.mdx b/docs/zh/14-reference/05-connector/20-go.mdx index 85c65a5fb8..a4d7b9be71 100644 --- a/docs/zh/14-reference/05-connector/20-go.mdx +++ b/docs/zh/14-reference/05-connector/20-go.mdx @@ -494,6 +494,37 @@ Prepare 允许使用预编译的 SQL 语句，可以提高性能并提供参数 - **接口说明**：关闭语句。 - **返回值**：错误信息。 +从 3.6.0 版本开始，提供 stmt2 绑定参数的接口 + +- `func (conn *Connector) Stmt2(reqID int64, singleTableBindOnce bool) *Stmt2` + - **接口说明**：从连接创建 stmt2。 + - **参数说明**： + - `reqID`：请求 ID。 + - `singleTableBindOnce`：单个子表在单次执行中只有一次数据绑定。 + - **返回值**：stmt2 对象。 +- `func (s *Stmt2) Prepare(sql string) error` + - **接口说明**：绑定 sql 语句。 + - **参数说明**： + - `sql`：要绑定的 sql 语句。 + - **返回值**：错误信息。 +- `func (s *Stmt2) Bind(params []*stmt.TaosStmt2BindData) error` + - **接口说明**：绑定数据。 + - **参数说明**： + - params要绑定的数据。 + - **返回值**：错误信息。 +- `func (s *Stmt2) Execute() error` + - **接口说明**：执行语句。 + - **返回值**：错误信息。 +- `func (s *Stmt2) GetAffectedRows() int` + - **接口说明**：获取受影响行数（只在插入语句有效）。 + - **返回值**：受影响行数。 +- `func (s *Stmt2) UseResult() (driver.Rows, error)` + - **接口说明**：获取结果集（只在查询语句有效）。 + - **返回值**：结果集 Rows 对象，错误信息。 +- `func (s *Stmt2) Close() error` + - **接口说明**：关闭stmt2。 + - **返回值**：错误信息。 + `ws/stmt` 包提供了通过 WebSocket 进行参数绑定的接口 - `func (c *Connector) Init() (*Stmt, error)` From 922766c1457c9af2df3cedfe5d2fed1c1c4fdee3 Mon Sep 17 00:00:00 2001 From: Haolin Wang Date: Fri, 27 Dec 2024 17:16:00 +0800 Subject: [PATCH 08/73] fix: infinite loop when reading CSV file EOF on Windows --- source/os/src/osFile.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/os/src/osFile.c b/source/os/src/osFile.c index 8a2606c4c2..b1198e1cb2 100644 --- a/source/os/src/osFile.c +++ b/source/os/src/osFile.c @@ -1403,7 +1403,7 @@ int64_t taosGetLineFile(TdFilePtr pFile, char **__restrict ptrBuf) { } (*ptrBuf)[totalBytesRead] = '\0'; - ret = totalBytesRead; + ret = (totalBytesRead > 0 ? totalBytesRead : -1); // -1 means EOF #else size_t len = 0; ret = getline(ptrBuf, &len, pFile->fp); From af434a28318d69e8c16dc46e485fd840877e39b4 Mon Sep 17 00:00:00 2001 From: t_max <1172915550@qq.com> Date: Mon, 30 Dec 2024 10:49:41 +0800 Subject: [PATCH 09/73] docs: go connector v3.6.0 --- docs/en/14-reference/05-connector/20-go.md | 39 +++++++++++---------- docs/zh/14-reference/05-connector/20-go.mdx | 39 +++++++++++---------- 2 files changed, 42 insertions(+), 36 deletions(-) diff --git a/docs/en/14-reference/05-connector/20-go.md b/docs/en/14-reference/05-connector/20-go.md index aecad76a2e..bf0e6dd979 100644 --- a/docs/en/14-reference/05-connector/20-go.md +++ b/docs/en/14-reference/05-connector/20-go.md @@ -21,24 +21,25 @@ Supports Go 1.14 and above. ## Version History -| driver-go Version | Major Changes | TDengine Version | -|------------------|------------------------------------------------------------------|-------------------| -| v3.5.8 | Fixed null pointer exception. | - | -| v3.5.7 | taosWS and taosRestful support passing request id. | - | -| v3.5.6 | Improved websocket query and insert performance. | 3.3.2.0 and higher | -| v3.5.5 | Restful supports skipping SSL certificate check. | - | -| v3.5.4 | Compatible with TDengine 3.3.0.0 tmq raw data. | - | -| v3.5.3 | Refactored taosWS. | - | -| v3.5.2 | Websocket compression and optimized tmq subscription performance. | 3.2.3.0 and higher | -| v3.5.1 | Native stmt query and geometry type support. | 3.2.1.0 and higher | -| v3.5.0 | Support tmq get assignment and seek offset. | 3.0.5.0 and higher | -| v3.3.1 | Schemaless protocol insert based on websocket. | 3.0.4.1 and higher | -| v3.1.0 | Provided Kafka-like subscription API. | - | -| v3.0.4 | Added request id related interfaces. | 3.0.2.2 and higher | -| v3.0.3 | Websocket-based statement insert. | - | -| v3.0.2 | Websocket-based data query and insert. | 3.0.1.5 and higher | -| v3.0.1 | Websocket-based message subscription. | - | -| v3.0.0 | Adapted to TDengine 3.0 query and insert. | 3.0.0.0 and higher | +| driver-go Version | Major Changes | TDengine Version | +|-------------------|-------------------------------------------------------------------------------------------------|--------------------| +| v3.6.0 | stmt2 native interface, DSN supports passwords containing special characters (url.QueryEscape). | 3.3.5.0 and higher | +| v3.5.8 | Fixed null pointer exception. | - | +| v3.5.7 | taosWS and taosRestful support passing request id. | - | +| v3.5.6 | Improved websocket query and insert performance. | 3.3.2.0 and higher | +| v3.5.5 | Restful supports skipping SSL certificate check. | - | +| v3.5.4 | Compatible with TDengine 3.3.0.0 tmq raw data. | - | +| v3.5.3 | Refactored taosWS. | - | +| v3.5.2 | Websocket compression and optimized tmq subscription performance. | 3.2.3.0 and higher | +| v3.5.1 | Native stmt query and geometry type support. | 3.2.1.0 and higher | +| v3.5.0 | Support tmq get assignment and seek offset. | 3.0.5.0 and higher | +| v3.3.1 | Schemaless protocol insert based on websocket. | 3.0.4.1 and higher | +| v3.1.0 | Provided Kafka-like subscription API. | - | +| v3.0.4 | Added request id related interfaces. | 3.0.2.2 and higher | +| v3.0.3 | Websocket-based statement insert. | - | +| v3.0.2 | Websocket-based data query and insert. | 3.0.1.5 and higher | +| v3.0.1 | Websocket-based message subscription. | - | +| v3.0.0 | Adapted to TDengine 3.0 query and insert. | 3.0.0.0 and higher | ## Exceptions and Error Codes @@ -136,6 +137,8 @@ Full form of DSN: username:password@protocol(address)/dbname?param=value ``` +When the password contains special characters, it needs to be escaped using url.QueryEscape. + ##### Native Connection Import the driver: diff --git a/docs/zh/14-reference/05-connector/20-go.mdx b/docs/zh/14-reference/05-connector/20-go.mdx index a4d7b9be71..210b0e438d 100644 --- a/docs/zh/14-reference/05-connector/20-go.mdx +++ b/docs/zh/14-reference/05-connector/20-go.mdx @@ -23,24 +23,25 @@ import RequestId from "./_request_id.mdx"; ## 版本历史 -| driver-go 版本 | 主要变化 | TDengine 版本 | -|-------------|-------------------------------------|---------------| -| v3.5.8 | 修复空指针异常 | - | -| v3.5.7 | taosWS 和 taosRestful 支持传入 request id | - | -| v3.5.6 | 提升 websocket 查询和写入性能 | 3.3.2.0 及更高版本 | -| v3.5.5 | restful 支持跳过 ssl 证书检查 | - | -| v3.5.4 | 兼容 TDengine 3.3.0.0 tmq raw data | - | -| v3.5.3 | 重构 taosWS | - | -| v3.5.2 | websocket 压缩和优化消息订阅性能 | 3.2.3.0 及更高版本 | -| v3.5.1 | 原生 stmt 查询和 geometry 类型支持 | 3.2.1.0 及更高版本 | -| v3.5.0 | 获取消费进度及按照指定进度开始消费 | 3.0.5.0 及更高版本 | -| v3.3.1 | 基于 websocket 的 schemaless 协议写入 | 3.0.4.1 及更高版本 | -| v3.1.0 | 提供贴近 kafka 的订阅 api | - | -| v3.0.4 | 新增 request id 相关接口 | 3.0.2.2 及更高版本 | -| v3.0.3 | 基于 websocket 的 statement 写入 | - | -| v3.0.2 | 基于 websocket 的数据查询和写入 | 3.0.1.5 及更高版本 | -| v3.0.1 | 基于 websocket 的消息订阅 | - | -| v3.0.0 | 适配 TDengine 3.0 查询和写入 | 3.0.0.0 及更高版本 | +| driver-go 版本 | 主要变化 | TDengine 版本 | +|--------------|--------------------------------------------|---------------| +| v3.6.0 | stmt2 原生接口，DSN 支持密码包含特殊字符（url.QueryEscape） | 3.3.5.0 及更高版本 | +| v3.5.8 | 修复空指针异常 | - | +| v3.5.7 | taosWS 和 taosRestful 支持传入 request id | - | +| v3.5.6 | 提升 websocket 查询和写入性能 | 3.3.2.0 及更高版本 | +| v3.5.5 | restful 支持跳过 ssl 证书检查 | - | +| v3.5.4 | 兼容 TDengine 3.3.0.0 tmq raw data | - | +| v3.5.3 | 重构 taosWS | - | +| v3.5.2 | websocket 压缩和优化消息订阅性能 | 3.2.3.0 及更高版本 | +| v3.5.1 | 原生 stmt 查询和 geometry 类型支持 | 3.2.1.0 及更高版本 | +| v3.5.0 | 获取消费进度及按照指定进度开始消费 | 3.0.5.0 及更高版本 | +| v3.3.1 | 基于 websocket 的 schemaless 协议写入 | 3.0.4.1 及更高版本 | +| v3.1.0 | 提供贴近 kafka 的订阅 api | - | +| v3.0.4 | 新增 request id 相关接口 | 3.0.2.2 及更高版本 | +| v3.0.3 | 基于 websocket 的 statement 写入 | - | +| v3.0.2 | 基于 websocket 的数据查询和写入 | 3.0.1.5 及更高版本 | +| v3.0.1 | 基于 websocket 的消息订阅 | - | +| v3.0.0 | 适配 TDengine 3.0 查询和写入 | 3.0.0.0 及更高版本 | ## 异常和错误码 @@ -137,6 +138,8 @@ WKB规范请参考[Well-Known Binary (WKB)](https://libgeos.org/specifications/w username:password@protocol(address)/dbname?param=value ``` +当密码中包含特殊字符时，需要使用 `url.QueryEscape` 进行转义。 + ##### 原生连接 导入驱动： From 4167f0d40f1e461cbe94858a7a0adb84dc26ca16 Mon Sep 17 00:00:00 2001 From: t_max <1172915550@qq.com> Date: Mon, 30 Dec 2024 11:06:33 +0800 Subject: [PATCH 10/73] test: add go stmt2 example test --- docs/examples/go/go.mod | 2 +- docs/examples/go/go.sum | 4 ++-- tests/docs-examples-test/go.sh | 5 +++++ 3 files changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/examples/go/go.mod b/docs/examples/go/go.mod index ed8fde2d9f..2dd5d4ec2b 100644 --- a/docs/examples/go/go.mod +++ b/docs/examples/go/go.mod @@ -2,7 +2,7 @@ module goexample go 1.17 -require github.com/taosdata/driver-go/v3 v3.5.6 +require github.com/taosdata/driver-go/v3 v3.6.0 require ( github.com/google/uuid v1.3.0 // indirect diff --git a/docs/examples/go/go.sum b/docs/examples/go/go.sum index 61841429ee..ee2ad55588 100644 --- a/docs/examples/go/go.sum +++ b/docs/examples/go/go.sum @@ -18,8 +18,8 @@ github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+ github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI= github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY= github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg= -github.com/taosdata/driver-go/v3 v3.5.6 h1:LDVtMyT3B9p2VREsd5KKM91D4Y7P4kSdh2SQumXi8bk= -github.com/taosdata/driver-go/v3 v3.5.6/go.mod h1:H2vo/At+rOPY1aMzUV9P49SVX7NlXb3LAbKw+MCLrmU= +github.com/taosdata/driver-go/v3 v3.6.0 h1:4dRXMl01DhIS5xBXUvtkkB+MjL8g64zN674xKd+ojTE= +github.com/taosdata/driver-go/v3 v3.6.0/go.mod h1:H2vo/At+rOPY1aMzUV9P49SVX7NlXb3LAbKw+MCLrmU= gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo= gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/tests/docs-examples-test/go.sh b/tests/docs-examples-test/go.sh index 606265435d..fa02b33b10 100644 --- a/tests/docs-examples-test/go.sh +++ b/tests/docs-examples-test/go.sh @@ -61,6 +61,11 @@ check_transactions || exit 1 reset_cache || exit 1 go run ./stmt/ws/main.go +taos -s "drop database if exists power" +check_transactions || exit 1 +reset_cache || exit 1 +go run ./stmt2/native/main.go + taos -s "drop database if exists power" check_transactions || exit 1 reset_cache || exit 1 From b80a20a4a13b7ec9768e92ffb0af8837078a75dc Mon Sep 17 00:00:00 2001 From: dapan1121 Date: Mon, 30 Dec 2024 12:11:11 +0800 Subject: [PATCH 11/73] fix: session list issue --- source/util/inc/tmempoolInt.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/source/util/inc/tmempoolInt.h b/source/util/inc/tmempoolInt.h index 8d4cb57ddc..013838e6df 100755 --- a/source/util/inc/tmempoolInt.h +++ b/source/util/inc/tmempoolInt.h @@ -222,7 +222,7 @@ typedef struct SMPSessionChunk { } SMPSessionChunk; typedef struct SMPSession { - // SMPListNode list; + SMPListNode list; char* sessionId; SMPJob* pJob; From 549d26be95add84aef6cb416e8a6dda95c2ab8b3 Mon Sep 17 00:00:00 2001 From: dapan1121 Date: Mon, 30 Dec 2024 13:12:10 +0800 Subject: [PATCH 12/73] fix: abort preprocess query issue --- source/libs/qworker/src/qworker.c | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/source/libs/qworker/src/qworker.c b/source/libs/qworker/src/qworker.c index 4a9eea66e2..5dd43ca064 100644 --- a/source/libs/qworker/src/qworker.c +++ b/source/libs/qworker/src/qworker.c @@ -746,9 +746,19 @@ _return: } int32_t qwAbortPrerocessQuery(QW_FPARAMS_DEF) { - QW_ERR_RET(qwDropTask(QW_FPARAMS())); + int32_t code = TSDB_CODE_SUCCESS; + SQWTaskCtx *ctx = NULL; - return TSDB_CODE_SUCCESS; + QW_ERR_RET(qwAcquireTaskCtx(QW_FPARAMS(), &ctx)); + + QW_LOCK(QW_WRITE, &ctx->lock); + QW_ERR_JRET(qwDropTask(QW_FPARAMS())); + +_return: + + QW_UNLOCK(QW_WRITE, &ctx->lock); + + return code; } int32_t qwPreprocessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg) { From 8c7ee17dd14a72616eb5f066c6875b4e8bd4358c Mon Sep 17 00:00:00 2001 From: Haojun Liao Date: Mon, 30 Dec 2024 13:36:57 +0800 Subject: [PATCH 13/73] fix(stream): replace functions with safty ones. --- source/libs/stream/src/streamBackendRocksdb.c | 2 +- source/libs/stream/src/streamMeta.c | 7 +- source/libs/stream/src/streamSnapshot.c | 85 ++++++++++++++----- source/libs/stream/src/streamTask.c | 28 ++++-- 4 files changed, 89 insertions(+), 33 deletions(-) diff --git a/source/libs/stream/src/streamBackendRocksdb.c b/source/libs/stream/src/streamBackendRocksdb.c index 37fb081a87..6031710bf9 100644 --- a/source/libs/stream/src/streamBackendRocksdb.c +++ b/source/libs/stream/src/streamBackendRocksdb.c @@ -1808,7 +1808,7 @@ int stateKeyDecode(void* k, char* buf) { return p - buf; } -int stateKeyToString(void* k, char* buf) { +int32_t stateKeyToString(void* k, char* buf) { SStateKey* key = k; int n = 0; n += sprintf(buf + n, "[groupId:%" PRIu64 ",", key->key.groupId); diff --git a/source/libs/stream/src/streamMeta.c b/source/libs/stream/src/streamMeta.c index 8e8e300ee6..3955343fdb 100644 --- a/source/libs/stream/src/streamMeta.c +++ b/source/libs/stream/src/streamMeta.c @@ -250,9 +250,10 @@ _EXIT: streamBackendCleanup((void*)pBackend); if (code == 0) { - char* state = taosMemoryCalloc(1, strlen(pMeta->path) + 32); + int32_t len = strlen(pMeta->path) + 32; + char* state = taosMemoryCalloc(1, len); if (state != NULL) { - sprintf(state, "%s%s%s", pMeta->path, TD_DIRSEP, "state"); + (void) snprintf(state, len, "%s%s%s", pMeta->path, TD_DIRSEP, "state"); taosRemoveDir(state); taosMemoryFree(state); } else { @@ -379,7 +380,7 @@ int32_t streamMetaOpen(const char* path, void* ahandle, FTaskBuild buildTaskFn, char* tpath = taosMemoryCalloc(1, len); TSDB_CHECK_NULL(tpath, code, lino, _err, terrno); - sprintf(tpath, "%s%s%s", path, TD_DIRSEP, "stream"); + (void) snprintf(tpath, len, "%s%s%s", path, TD_DIRSEP, "stream"); pMeta->path = tpath; code = streamMetaOpenTdb(pMeta); diff --git a/source/libs/stream/src/streamSnapshot.c b/source/libs/stream/src/streamSnapshot.c index ae8a71d988..1730e2c7c2 100644 --- a/source/libs/stream/src/streamSnapshot.c +++ b/source/libs/stream/src/streamSnapshot.c @@ -119,23 +119,21 @@ static int64_t kBlockSize = 64 * 1024; int32_t streamSnapHandleInit(SStreamSnapHandle* handle, char* path, void* pMeta); void streamSnapHandleDestroy(SStreamSnapHandle* handle); -// static void streamBuildFname(char* path, char* file, char* fullname) - -#define STREAM_ROCKSDB_BUILD_FULLNAME(path, file, fullname) \ - do { \ - sprintf(fullname, "%s%s%s", path, TD_DIRSEP, file); \ - } while (0) - int32_t streamGetFileSize(char* path, char* name, int64_t* sz) { int32_t ret = 0; + int32_t len = strlen(path) + 32; - char* fullname = taosMemoryCalloc(1, strlen(path) + 32); + char* fullname = taosMemoryCalloc(1, len); if (fullname == NULL) { stError("failed to get file:%s size, code: out of memory", name); return terrno; } - sprintf(fullname, "%s%s%s", path, TD_DIRSEP, name); + ret = snprintf(fullname, len, "%s%s%s", path, TD_DIRSEP, name); + if (ret < 0) { + stError("%s failed to set the file path for get the file size, code: out of buffer", name); + return TSDB_CODE_OUT_OF_BUFFER; + } ret = taosStatFile(fullname, sz, NULL, NULL); taosMemoryFree(fullname); @@ -146,7 +144,7 @@ int32_t streamGetFileSize(char* path, char* name, int64_t* sz) { TdFilePtr streamOpenFile(char* path, char* name, int32_t opt) { char fullname[256] = {0}; - STREAM_ROCKSDB_BUILD_FULLNAME(path, name, fullname); + (void) snprintf(fullname, tListLen(fullname),"%s%s%s", path, TD_DIRSEP, name); return taosOpenFile(fullname, opt); } @@ -155,9 +153,9 @@ int32_t streamCreateTaskDbSnapInfo(void* arg, char* path, SArray* pSnap) { retur int32_t streamDestroyTaskDbSnapInfo(void* arg, SArray* snap) { return taskDbDestroySnap(arg, snap); } void snapFileDebugInfo(SBackendSnapFile2* pSnapFile) { - if (qDebugFlag & DEBUG_DEBUG) { - int16_t cap = 512; + int16_t cap = 512; + if (qDebugFlag & DEBUG_DEBUG) { char* buf = taosMemoryCalloc(1, cap); if (buf == NULL) { stError("%s failed to alloc memory, reason:%s", STREAM_STATE_TRANSFER, tstrerror(TSDB_CODE_OUT_OF_MEMORY)); @@ -171,19 +169,57 @@ void snapFileDebugInfo(SBackendSnapFile2* pSnapFile) { return; } - if (pSnapFile->pCurrent) sprintf(buf, "current: %s,", pSnapFile->pCurrent); - if (pSnapFile->pMainfest) sprintf(buf + strlen(buf), "MANIFEST: %s,", pSnapFile->pMainfest); - if (pSnapFile->pOptions) sprintf(buf + strlen(buf), "options: %s,", pSnapFile->pOptions); + int32_t wlen = 0; + int32_t len = 0; + if (pSnapFile->pCurrent) { + len = snprintf(buf, cap,"current: %s,", pSnapFile->pCurrent); + if (len > 0) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + } + } + + if (pSnapFile->pMainfest) { + len = snprintf(buf + wlen, cap - wlen, "MANIFEST: %s,", pSnapFile->pMainfest); + if (len > 0) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + } + } + + if (pSnapFile->pOptions) { + len = snprintf(buf + wlen, cap - wlen, "options: %s,", pSnapFile->pOptions); + if (len > 0) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + } + } + if (pSnapFile->pSst) { for (int32_t i = 0; i < taosArrayGetSize(pSnapFile->pSst); i++) { char* name = taosArrayGetP(pSnapFile->pSst, i); - if (strlen(buf) + strlen(name) < cap) sprintf(buf + strlen(buf), "%s,", name); + if (strlen(buf) + strlen(name) < cap) { + len = snprintf(buf + wlen, cap - wlen, "%s,", name); + if (len > 0) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + } + } } } - if ((strlen(buf)) < cap) sprintf(buf + strlen(buf) - 1, "]"); + + if ((strlen(buf)) < cap) { + buf[wlen++] = ']'; + buf[wlen - 1] = '\0'; + } stInfo("%s %" PRId64 "-%" PRId64 " get file list: %s", STREAM_STATE_TRANSFER, pSnapFile->snapInfo.streamId, pSnapFile->snapInfo.taskId, buf); + taosMemoryFree(buf); } } @@ -771,16 +807,23 @@ int32_t streamSnapWrite(SStreamSnapWriter* pWriter, uint8_t* pData, uint32_t nDa SBackendSnapFile2* pDbSnapFile = taosArrayGet(pHandle->pDbSnapSet, pHandle->currIdx); if (pDbSnapFile->inited == 0) { char idstr[64] = {0}; - sprintf(idstr, "0x%" PRIx64 "-0x%x", snapInfo.streamId, (int32_t)(snapInfo.taskId)); + (void)snprintf(idstr, tListLen(idstr), "0x%" PRIx64 "-0x%x", snapInfo.streamId, (int32_t)(snapInfo.taskId)); - char* path = taosMemoryCalloc(1, strlen(pHandle->metaPath) + 256); + int32_t bufLen = strlen(pHandle->metaPath) + 256; + char* path = taosMemoryCalloc(1, bufLen); if (path == NULL) { stError("s-task:0x%x failed to prepare meta header buffer, code:Out of memory", (int32_t) snapInfo.taskId); return terrno; } - sprintf(path, "%s%s%s%s%s%s%s%" PRId64 "", pHandle->metaPath, TD_DIRSEP, idstr, TD_DIRSEP, "checkpoints", TD_DIRSEP, - "checkpoint", snapInfo.chkpId); + int32_t ret = snprintf(path, bufLen, "%s%s%s%s%s%s%s%" PRId64 "", pHandle->metaPath, TD_DIRSEP, idstr, TD_DIRSEP, + "checkpoints", TD_DIRSEP, "checkpoint", snapInfo.chkpId); + if (ret < 0) { + stError("s-task:0x%x failed to set the path for take snapshot, code: out of buffer, %s", (int32_t)snapInfo.taskId, + pHandle->metaPath); + return TSDB_CODE_OUT_OF_BUFFER; + } + if (!taosIsDir(path)) { code = taosMulMkDir(path); stInfo("%s mkdir %s", STREAM_STATE_TRANSFER, path); diff --git a/source/libs/stream/src/streamTask.c b/source/libs/stream/src/streamTask.c index f46228fd47..e3a1f5f94a 100644 --- a/source/libs/stream/src/streamTask.c +++ b/source/libs/stream/src/streamTask.c @@ -132,8 +132,12 @@ int32_t tNewStreamTask(int64_t streamId, int8_t taskLevel, SEpSet* pEpset, bool return code; } - char buf[128] = {0}; - sprintf(buf, "0x%" PRIx64 "-0x%x", pTask->id.streamId, pTask->id.taskId); + char buf[128] = {0}; + int32_t ret = snprintf(buf, tListLen(buf), "0x%" PRIx64 "-0x%x", pTask->id.streamId, pTask->id.taskId); + if (ret < 0) { + stError("s-task:0x%x failed to set the taskIdstr, code: out of buffer", pTask->id.taskId); + return TSDB_CODE_OUT_OF_BUFFER; + } pTask->id.idStr = taosStrdup(buf); if (pTask->id.idStr == NULL) { @@ -402,7 +406,7 @@ int32_t streamTaskSetBackendPath(SStreamTask* pTask) { } char id[128] = {0}; - int32_t nBytes = sprintf(id, "0x%" PRIx64 "-0x%x", streamId, taskId); + int32_t nBytes = snprintf(id, tListLen(id), "0x%" PRIx64 "-0x%x", streamId, taskId); if (nBytes < 0 || nBytes >= sizeof(id)) { return TSDB_CODE_OUT_OF_BUFFER; } @@ -413,10 +417,14 @@ int32_t streamTaskSetBackendPath(SStreamTask* pTask) { return terrno; } - (void)sprintf(pTask->backendPath, "%s%s%s", pTask->pMeta->path, TD_DIRSEP, id); - stDebug("s-task:%s set backend path:%s", pTask->id.idStr, pTask->backendPath); - - return 0; + int32_t code = snprintf(pTask->backendPath, len + nBytes + 2, "%s%s%s", pTask->pMeta->path, TD_DIRSEP, id); + if (code < 0) { + stError("s-task:%s failed to set backend path:%s, code: out of buffer", pTask->id.idStr, pTask->backendPath); + return TSDB_CODE_OUT_OF_BUFFER; + } else { + stDebug("s-task:%s set backend path:%s", pTask->id.idStr, pTask->backendPath); + return 0; + } } int32_t streamTaskInit(SStreamTask* pTask, SStreamMeta* pMeta, SMsgCb* pMsgCb, int64_t ver) { @@ -1129,7 +1137,11 @@ SEpSet* streamTaskGetDownstreamEpInfo(SStreamTask* pTask, int32_t taskId) { int32_t createStreamTaskIdStr(int64_t streamId, int32_t taskId, const char** pId) { char buf[128] = {0}; - sprintf(buf, "0x%" PRIx64 "-0x%x", streamId, taskId); + int32_t code = snprintf(buf, tListLen(buf),"0x%" PRIx64 "-0x%x", streamId, taskId); + if (code < 0) { + return TSDB_CODE_OUT_OF_BUFFER; + } + *pId = taosStrdup(buf); if (*pId == NULL) { From 44cfecc4023fbbe7b6b407c71ae7dd3cdbd41c13 Mon Sep 17 00:00:00 2001 From: Haojun Liao Date: Mon, 30 Dec 2024 13:41:34 +0800 Subject: [PATCH 14/73] refactor: remove unused codes. --- source/libs/stream/src/streamSnapshot.c | 1 - 1 file changed, 1 deletion(-) diff --git a/source/libs/stream/src/streamSnapshot.c b/source/libs/stream/src/streamSnapshot.c index 1730e2c7c2..9bd0ab0ff2 100644 --- a/source/libs/stream/src/streamSnapshot.c +++ b/source/libs/stream/src/streamSnapshot.c @@ -214,7 +214,6 @@ void snapFileDebugInfo(SBackendSnapFile2* pSnapFile) { if ((strlen(buf)) < cap) { buf[wlen++] = ']'; - buf[wlen - 1] = '\0'; } stInfo("%s %" PRId64 "-%" PRId64 " get file list: %s", STREAM_STATE_TRANSFER, pSnapFile->snapInfo.streamId, From 49e1e05f2166e39177feca8a10edffc8150bbafd Mon Sep 17 00:00:00 2001 From: Shengliang Guan Date: Mon, 30 Dec 2024 14:08:07 +0800 Subject: [PATCH 15/73] docs: username --- docs/zh/14-reference/03-taos-sql/19-limit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/14-reference/03-taos-sql/19-limit.md b/docs/zh/14-reference/03-taos-sql/19-limit.md index 4e2fa69a14..e5c03db2fd 100644 --- a/docs/zh/14-reference/03-taos-sql/19-limit.md +++ b/docs/zh/14-reference/03-taos-sql/19-limit.md @@ -37,6 +37,6 @@ description: 合法字符集和命名中的限制规则 - 库的数目，超级表的数目、表的数目，系统不做限制，仅受系统资源限制 - 数据库的副本数只能设置为 1 或 3 - 用户名的最大长度是 23 字节 -- 用户密码的最大长度是 31 字节 +- 用户密码的长度范围是 8-16 字节 - 总数据行数取决于可用资源 - 单个数据库的虚拟结点数上限为 1024 From 1742dd3c0f1dbf5281270c008498c779f7d172fb Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Mon, 30 Dec 2024 14:15:24 +0800 Subject: [PATCH 16/73] ci: add paths-ignore --- .github/workflows/taosd-ci-build.yml | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/.github/workflows/taosd-ci-build.yml b/.github/workflows/taosd-ci-build.yml index 0876f5b731..8d8a120d76 100644 --- a/.github/workflows/taosd-ci-build.yml +++ b/.github/workflows/taosd-ci-build.yml @@ -6,6 +6,10 @@ on: - 'main' - '3.0' - '3.1' + paths-ignore: + - 'docs/**' + - 'packaging/**' + - 'tests/**' concurrency: group: ${{ github.workflow }}-${{ github.ref }} @@ -14,7 +18,7 @@ concurrency: jobs: build: runs-on: ubuntu-latest - name: Run unit tests + name: Build and test steps: - name: Checkout the repository @@ -32,12 +36,11 @@ jobs: libgeos-dev libjansson-dev libsnappy-dev liblzma-dev libz-dev \ zlib1g pkg-config libssl-dev gawk - - name: Build and install TDengine run: | mkdir debug && cd debug cmake .. -DBUILD_HTTP=false -DBUILD_JDBC=false \ - -DBUILD_TOOLS=true -DBUILD_TEST=off \ + -DBUILD_TOOLS=true -DBUILD_TEST=false \ -DBUILD_KEEPER=true -DBUILD_DEPENDENCY_TESTS=false make -j 4 sudo make install From aea32365978df8defb117d2b9372a260f4bd2abb Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 14:44:49 +0800 Subject: [PATCH 17/73] stmt2 document for python connector --- docs/examples/python/insert_with_stmt2.py | 626 ++++++++++++++++++ .../14-reference/05-connector/30-python.mdx | 131 ++-- tests/docs-examples-test/python.sh | 2 + 3 files changed, 688 insertions(+), 71 deletions(-) create mode 100644 docs/examples/python/insert_with_stmt2.py diff --git a/docs/examples/python/insert_with_stmt2.py b/docs/examples/python/insert_with_stmt2.py new file mode 100644 index 0000000000..433b283fe3 --- /dev/null +++ b/docs/examples/python/insert_with_stmt2.py @@ -0,0 +1,626 @@ +# encoding:UTF-8 +from ctypes import * +from datetime import datetime +# geometry support +from shapely.wkb import dumps, loads +from shapely.wkt import dumps as wkt_dumps, loads as wkt_loads + +import taos +import math +import traceback +from taos.statement2 import * +from taos.constants import FieldType +from taos import log +from taos import bind2 + +# input WKT return WKB (bytes object) +def WKB(wkt, hex = False): + if wkt is None: + return None + wkb = wkt_loads(wkt) + wkb_bytes = dumps(wkb, hex) + return wkb_bytes + +def compareLine(oris, rows): + n = len(oris) + if len(rows) != n: + return False + log.debug(f" len is {n} oris={oris} rows={rows}") + for i in range(n): + if oris[i] != rows[i]: + if type(rows[i]) == bool: + if bool(oris[i]) != rows[i]: + log.debug1(f" diff bool i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") + return False + else: + log.debug1(f" float i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") + elif type(rows[i]) == float: + if math.isclose(oris[i], rows[i], rel_tol=1e-3) is False: + log.debug1(f" diff float i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") + return False + else: + log.debug1(f" float i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") + else: + log.debug1(f" diff i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") + return False + else: + log.debug1(f" i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") + + return True + + +def checkResultCorrect(conn, sql, tagsTb, datasTb): + # column to rows + log.debug(f"check sql correct: {sql}\n") + oris = [] + ncol = len(datasTb) + nrow = len(datasTb[0]) + + for i in range(nrow): + row = [] + for j in range(ncol): + if j == 0: + # ts column + c0 = datasTb[j][i] + if type(c0) is int : + row.append(datasTb[j][i]) + else: + ts = int(bind2.datetime_to_timestamp(c0, PrecisionEnum.Milliseconds).value) + row.append(ts) + else: + row.append(datasTb[j][i]) + + if tagsTb is not None: + row += tagsTb + oris.append(row) + + # fetch all + lres = [] + log.debug(sql) + res = conn.query(sql) + i = 0 + for row in res: + lrow = list(row) + lrow[0] = int(lrow[0].timestamp()*1000) + if compareLine(oris[i], lrow) is False: + log.info(f"insert data differet. i={i} expect ori data={oris[i]} query from db ={lrow}") + raise(BaseException("check insert data correct failed.")) + else: + log.debug(f"i={i} origin data same with get from db\n") + log.debug(f" origin data = {oris[i]} \n") + log.debug(f" get from db = {lrow} \n") + i += 1 + + +def checkResultCorrects(conn, dbname, stbname, tbnames, tags, datas): + count = len(tbnames) + for i in range(count): + if stbname is None: + sql = f"select * from {dbname}.{tbnames[i]} " + else: + sql = f"select * from {dbname}.{stbname} where tbname='{tbnames[i]}' " + + checkResultCorrect(conn, sql, tags[i], datas[i]) + + print("insert data check correct ..................... ok\n") + + +def prepare(conn, dbname, stbname, ntb1, ntb2): + conn.execute("drop database if exists %s" % dbname) + conn.execute("create database if not exists %s precision 'ms' " % dbname) + conn.select_db(dbname) + # stable + sql = f"create table if not exists {dbname}.{stbname}(ts timestamp, name binary(32), sex bool, score int) tags(grade nchar(8), class int)" + conn.execute(sql) + # normal table + sql = f"create table if not exists {dbname}.{ntb1} (ts timestamp, name varbinary(32), sex bool, score float, geo geometry(128))" + conn.execute(sql) + sql = f"create table if not exists {dbname}.{ntb2} (ts timestamp, name varbinary(32), sex bool, score float, geo geometry(128))" + conn.execute(sql) + + +# performace is high +def insert_bind_param(conn, stmt2, dbname, stbname): + # + # table info , write 5 lines to 3 child tables d0, d1, d2 with super table + # + tbnames = ["d1","d2","d3"] + + tags = [ + ["grade1", 1], + ["grade1", None], + [None , 3] + ] + datas = [ + # class 1 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], + ["Mary", "Tom", "Jack", "Jane", "alex" ,None ], + [0, 1, 1, 0, 1 ,None ], + [98, 80, 60, 100, 99 ,None ] + ], + # class 2 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], + ["Mary2", "Tom2", "Jack2", "Jane2", "alex2" ,None ], + [0, 1, 1, 0, 1 ,0 ], + [298, 280, 260, 2100, 299 ,None ] + ], + # class 3 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], + ["Mary3", "Tom3", "Jack3", "Jane3", "alex3" ,"Mark" ], + [0, 1, 1, 0, 1 ,None ], + [398, 380, 360, 3100, 399 ,None ] + ] + ] + + stmt2.bind_param(tbnames, tags, datas) + stmt2.execute() + + # check correct + checkResultCorrects(conn, dbname, stbname, tbnames, tags, datas) + + +def insert_bind_param_normal_tables(conn, stmt2, dbname, ntb): + tbnames = [ntb] + tags = None + wkts = [None, b"POINT(121.213 31.234)", b"POINT(122.22 32.222)", None, b"POINT(124.22 34.222)"] + wkbs = [WKB(wkt) for wkt in wkts] + + datas = [ + # table 1 + [ + # student + [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], + [b"Mary", b"tom", b"Jack", b"Jane", None ], + [0, 3.14, True, 0, 1 ], + [98, 99.87, 60, 100, 99 ], + wkbs + ] + ] + + stmt2.bind_param(tbnames, tags, datas) + stmt2.execute() + + # check correct + checkResultCorrects(conn, dbname, None, tbnames, [None], datas) + +def insert_bind_param_with_table(conn, stmt2, dbname, stbname, ctb): + + tbnames = None + tags = [ + ["grade2", 1] + ] + + # prepare data + datas = [ + # table 1 + [ + # student + [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], + ["Mary", "Tom", "Jack", "Jane", "alex" ], + [0, 1, 1, 0, 1 ], + [98, 80, 60, 100, 99 ] + ] + ] + + stmt2.bind_param(tbnames, tags, datas) + stmt2.execute() + + # check correct + checkResultCorrects(conn, dbname, stbname, [ctb], tags, datas) + + +# insert with single table (performance is lower) +def insert_bind_param_with_tables(conn, stmt2, dbname, stbname): + + tbnames = ["t1", "t2", "t3"] + tags = [ + ["grade2", 1], + ["grade2", 2], + ["grade2", 3] + ] + + # prepare data + datas = [ + # table 1 + [ + # student + [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], + ["Mary", "Tom", "Jack", "Jane", "alex" ], + [0, 1, 1, 0, 1 ], + [98, 80, 60, 100, 99 ] + ], + # table 2 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], + ["Mary2", "Tom2", "Jack2", "Jane2", "alex2" ], + [0, 1, 1, 0, 1 ], + [298, 280, 260, 2100, 299 ] + ], + # table 3 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], + ["Mary3", "Tom3", "Jack3", "Jane3", "alex3" ], + [0, 1, 1, 0, 1 ], + [398, 380, 360, 3100, 399 ] + ] + ] + + table0 = BindTable(tbnames[0], tags[0]) + table1 = BindTable(tbnames[1], tags[1]) + table2 = BindTable(tbnames[2], tags[2]) + + for data in datas[0]: + table0.add_col_data(data) + for data in datas[1]: + table1.add_col_data(data) + for data in datas[2]: + table2.add_col_data(data) + + # bind with single table + stmt2.bind_param_with_tables([table0, table1, table2]) + stmt2.execute() + + # check correct + checkResultCorrects(conn, dbname, stbname, tbnames, tags, datas) + + +def do_check_invalid(stmt2, tbnames, tags, datas): + table0 = BindTable(tbnames[0], tags[0]) + table1 = BindTable(tbnames[1], tags[1]) + table2 = BindTable(tbnames[2], tags[2]) + + for data in datas[0]: + table0.add_col_data(data) + for data in datas[1]: + table1.add_col_data(data) + for data in datas[2]: + table2.add_col_data(data) + + # bind with single table + try: + stmt2.bind_param_with_tables([table0, table1, table2]) + stmt2.execute() + except Exception as err: + #traceback.print_stack() + print(f"failed to do_check_invalid. err={err}") + return + + print(f"input invalid data passed , unexpect. \ntbnames={tbnames}\ntags={tags} \ndatas={datas} \n") + assert False + + +def check_input_invalid_param(conn, stmt2, dbname, stbname): + + tbnames = ["t1", "t2", "t3"] + tags = [ + ["grade2", 1], + ["grade2", 2], + ["grade2", 3] + ] + + # prepare data + datas = [ + # table 1 + [ + # student + [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], + ["Mary", "Tom", "Jack", "Jane", "alex" ], + [0, 1, 1, 0, 1 ], + [98, 80, 60, 100, 99 ] + ], + # table 2 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], + ["Mary2", "Tom2", "Jack2", "Jane2", "alex2" ], + [0, 1, 1, 0, 1 ], + [298, 280, 260, 2100, 299 ] + ], + # table 3 + [ + # student + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], + ["Mary3", "Tom3", "Jack3", "Jane3", "alex3" ], + [0, 1, 1, 0, 1 ], + [398, 380, 360, 3100, 399 ] + ] + ] + + # some tags is none + tags1 = [ ["grade2", 1], None, ["grade2", 3] ] + do_check_invalid(stmt2, tbnames, tags1, datas) + + # timestamp is over range + origin = datas[0][0][0] + datas[0][0][0] = 100000000000000000000000 + do_check_invalid(stmt2, tbnames, tags, datas) + datas[0][0][0] = origin # restore + + +# insert with single table (performance is lower) +def insert_with_normal_tables(conn, stmt2, dbname, ntb): + + tbnames = [ntb] + tags = [None] + # prepare data + + wkts = [None, "POINT(121.213 31.234)", "POINT(122.22 32.222)", None, "POINT(124.22 34.222)"] + wkbs = [WKB(wkt) for wkt in wkts] + + datas = [ + # table 1 + [ + # student + [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], + [b"Mary", b"tom", b"Jack", b"Jane", None ], + [0, 3.14, True, 0, 1 ], + [98, 99.87, 60, 100, 99 ], + wkbs + ] + ] + + table0 = BindTable(tbnames[0], tags[0]) + for data in datas[0]: + table0.add_col_data(data) + + # bind with single table + stmt2.bind_param_with_tables([table0]) + stmt2.execute() + + # check correct + checkResultCorrects(conn, dbname, None, tbnames, tags, datas) + + +def test_stmt2_prepare_empty_sql(conn): + if not IS_V3: + print(" test_stmt2_prepare_empty_sql not support TDengine 2.X version.") + return + + try: + # prepare + stmt2 = conn.statement2() + stmt2.prepare(sql='') + + # should not run here + conn.close() + print("prepare empty sql ............................. failed\n") + assert False + + except StatementError as err: + print("prepare empty sql ............................. ok\n") + conn.close() + + +def test_bind_invalid_tbnames_type(): + if not IS_V3: + print(" test_bind_invalid_tbnames_type not support TDengine 2.X version.") + return + + dbname = "stmt2" + stbname = "stmt2_stable" + subtbname = "stmt2_subtable" + + try: + conn = taos.connect() + conn.execute(f"drop database if exists {dbname}") + conn.execute(f"create database {dbname}") + conn.select_db(dbname) + conn.execute(f"create stable {stbname} (ts timestamp, a int) tags (b int);") + conn.execute(f"create table {subtbname} using {stbname} tags(0);") + + stmt2 = conn.statement2(f"insert into ? using {dbname}.{stbname} tags(?) values(?,?)") + + tags = [[1]] + datas = [[[1626861392589], [1]]] + + stmt2.bind_param(subtbname, tags, datas) + + # should not run here + conn.close() + print("bind invalid tbnames type ..................... failed\n") + assert False + + except StatementError as err: + print("bind invalid tbnames type ..................... ok\n") + conn.close() + + +# +# insert +# +def test_stmt2_insert(conn): + if not IS_V3: + print(" test_stmt2_query not support TDengine 2.X version.") + return + + dbname = "stmt2" + stbname = "meters" + ntb1 = "ntb1" + ntb2 = "ntb2" + + try: + prepare(conn, dbname, stbname, ntb1, ntb2) + + ctb = 'ctb' # child table + stmt2 = conn.statement2(f"insert into {dbname}.{ctb} using {dbname}.{stbname} tags (?,?) values(?,?,?,?)") + insert_bind_param_with_table(conn, stmt2, dbname, stbname, ctb) + print("insert child table ........................... ok\n") + stmt2.close() + + # # prepare + # stmt2 = conn.statement2(f"insert into ? using {dbname}.{stbname} tags(?,?) values(?,?,?,?)") + # print("insert prepare sql ............................ ok\n") + # + # # insert with table + # insert_bind_param_with_tables(conn, stmt2, dbname, stbname) + # print("insert bind with tables ....................... ok\n") + # check_input_invalid_param(conn, stmt2, dbname, stbname) + # print("check input invalid params .................... ok\n") + # + # # insert with split args + # insert_bind_param(conn, stmt2, dbname, stbname) + # print("insert bind ................................... ok\n") + # print("insert execute ................................ ok\n") + # stmt2.close() + + # ntb1 + stmt2 = conn.statement2(f"insert into {dbname}.{ntb1} values(?,?,?,?,?)") + insert_with_normal_tables(conn, stmt2, dbname, ntb1) + print("insert normal tables .......................... ok\n") + stmt2.close() + + # ntb2 + stmt2 = conn.statement2(f"insert into {dbname}.{ntb2} values(?,?,?,?,?)") + insert_bind_param_normal_tables(conn, stmt2, dbname, ntb2) + print("insert normal tables (bind param) ............. ok\n") + stmt2.close() + + conn.close() + print("test_stmt2_insert ............................. [passed]\n") + except Exception as err: + #conn.execute("drop database if exists %s" % dbname) + print("test_stmt2_insert ............................. failed\n") + conn.close() + raise err + + +# +# ------------------------ query ------------------- +# +def query_bind_param(conn, stmt2): + # set param + #tbnames = ["d2"] + tbnames = None + tags = None + datas = [ + # class 1 + [ + # where name in ('Tom2','alex2') or score > 1000;" + ["Tom2"], + [1000] + ] + ] + + # set param + types = [FieldType.C_BINARY, FieldType.C_INT] + stmt2.set_columns_type(types) + + # bind + stmt2.bind_param(tbnames, tags, datas) + + +# compare +def compare_result(conn, sql2, res2): + lres1 = [] + lres2 = [] + + # shor res2 + for row in res2: + log.debug(f" res2 rows = {row} \n") + lres2.append(row) + + res1 = conn.query(sql2) + for row in res1: + log.debug(f" res1 rows = {row} \n") + lres1.append(row) + + row1 = len(lres1) + row2 = len(lres2) + col1 = len(lres1[0]) + col2 = len(lres2[0]) + + # check number + if row1 != row2: + err = f"two results row count different. row1={row1} row2={row2}" + raise(BaseException(err)) + if col1 != col2: + err = f" two results column count different. col1={col1} col2={col2}" + raise(BaseException(err)) + + for i in range(row1): + for j in range(col1): + if lres1[i][j] != lres2[i][j]: + raise(f" two results data different. i={i} j={j} data1={res1[i][j]} data2={res2[i][j]}\n") + +# query +def test_stmt2_query(conn): + if not IS_V3: + print(" test_stmt2_query not support TDengine 2.X version.") + return + + dbname = "stmt2" + stbname = "meters" + ntb1 = "ntb1" + ntb2 = "ntb2" + sql1 = f"select * from {dbname}.d2 where name in (?) or score > ? ;" + sql2 = f"select * from {dbname}.d2 where name in ('Tom2') or score > 1000;" + + try: + # prepare + prepare(conn, dbname, stbname, ntb1, ntb2) + + # prepare + # stmt2 = conn.statement2(f"insert into ? using {dbname}.{stbname} tags(?,?) values(?,?,?,?)") + # insert_bind_param_with_tables(conn, stmt2, dbname, stbname) + # insert_bind_param(conn, stmt2, dbname, stbname) + # stmt2.close() + # print("insert bind & execute ......................... ok\n") + + conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.000', 'Mary2', false, 298)") + conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.001', 'Tom2', true, 280)") + conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.002', 'Jack2', true, 260)") + conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.003', 'Jane2', false, 2100)") + conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.004', 'alex2', true, 299)") + conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.005', NULL, false, NULL)") + + + # statement2 + stmt2 = conn.statement2(sql1) + print("query prepare sql ............................. ok\n") + + + # insert with table + #insert_bind_param_with_tables(conn, stmt2) + + + # bind + query_bind_param(conn, stmt2) + print("query bind param .............................. ok\n") + + # query execute + stmt2.execute() + + # fetch result + res2 = stmt2.result() + + # check result + compare_result(conn, sql2, res2) + print("query check corrent ........................... ok\n") + + #conn.execute("drop database if exists %s" % dbname) + stmt2.close() + conn.close() + print("test_stmt2_query .............................. [passed]\n") + + except Exception as err: + print("query ......................................... failed\n") + conn.close() + raise err + + +if __name__ == "__main__": + print("start stmt2 test case...\n") + taos.log.setting(True, True, True, True, True, False) + # insert + test_stmt2_insert(taos.connect()) + # query + test_stmt2_query(taos.connect()) + print("end stmt2 test case.\n") \ No newline at end of file diff --git a/docs/zh/14-reference/05-connector/30-python.mdx b/docs/zh/14-reference/05-connector/30-python.mdx index d724fc796c..c91e9775f4 100644 --- a/docs/zh/14-reference/05-connector/30-python.mdx +++ b/docs/zh/14-reference/05-connector/30-python.mdx @@ -246,43 +246,43 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - `reqId`: 用于问题追踪。 - **异常**：操作失败抛出 `DataError` 或 `OperationalError` 异常。 -#### 参数绑定 -- `fn statement(&self) -> PyResult` - - **接口说明**：使用 连接 对象创建 stmt 对象。 - - **返回值**：stmt 对象。 +#### 参数绑定 STMT2 +- `def statement2(self, sql=None, option=None)` + - **接口说明**：使用连接对象创建 stmt2 对象 + - **参数说明** + - `sql`: 绑定的 SQL 语句，如果不为空会调用`prepare`函数 + - `option` 传入 TaosStmt2Option 类实例选项 + - **返回值**：stmt2 对象。 - **异常**：操作失败抛出 `ConnectionError` 异常。 -- `fn prepare(&mut self, sql: &str) -> PyResult<()>` - - **接口说明**：绑定预编译 sql 语句。 +- `def prepare(self, sql)` + - **接口说明**：绑定预编译 sql 语句 - **参数说明**： - - `sql`: 预编译的 SQL 语句。 + - `sql`: 绑定的 SQL 语句 - **异常**：操作失败抛出 `ProgrammingError` 异常。 -- `fn set_tbname(&mut self, table_name: &str) -> PyResult<()>` - - **接口说明**：设置将要写入数据的表名。 +- `def bind_param(self, tbnames, tags, datas)` + - **接口说明**：以独立数组方式绑定数据 - **参数说明**： - - `tableName`: 表名，如果需要指定数据库, 例如： `db_name.table_name` 即可。 - - **异常**：操作失败抛出 `ProgrammingError` 异常。 -- `fn set_tags(&mut self, tags: Vec) -> PyResult<()>` - - **接口说明**：设置表 Tags 数据, 用于自动建表。 + - `tbnames`: 绑定表名数组，数据类型为 list + - `tags`: 绑定 tag 列值数组，数据类型为 list + - `tags`: 绑定普通列值数组，数据类型为 list + - **异常**：操作失败抛出 `StatementError` 异常 +- `def bind_param_with_tables(self, tables)` + - **接口说明**：以独立表方式绑定数据，独立表是以表为组织单位，每张表中有表名，TAG 值及普通列数值属性 - **参数说明**： - - `paramsArray`: Tags 数据。 - - **异常**：操作失败抛出 `ProgrammingError` 异常。 -- `fn bind_param(&mut self, params: Vec) -> PyResult<()>` - - **接口说明**：绑定数据。 - - **参数说明**： - - `paramsArray`: 绑定数据。 - - **异常**：操作失败抛出 `ProgrammingError` 异常。 -- `fn add_batch(&mut self) -> PyResult<()>` - - **接口说明**：提交绑定数据。 - - **异常**：操作失败抛出 `ProgrammingError` 异常。 -- `fn execute(&mut self) -> PyResult` - - **接口说明**：执行将绑定的数据全部写入。 - - **返回值**：写入条数。 + - `tables`: `BindTable` 独立表对象数组 + - **异常**：操作失败抛出 `StatementError` 异常。 +- `def execute(self) -> int:` + - **接口说明**：执行将绑定数据全部写入 + - **返回值**：影响行数 - **异常**：操作失败抛出 `QueryError` 异常。 -- `fn affect_rows(&mut self) -> PyResult` - - **接口说明**： 获取写入条数。 - - **返回值**：写入条数。 -- `fn close(&self) -> PyResult<()>` - - **接口说明**： 关闭 stmt 对象。 +- `def result(self)` + - **接口说明**： 获取参数绑定查询结果集 + - **返回值**：返回 TaosResult 对象 +- `def close(self)` + - **接口说明**： 关闭 stmt2 对象 + +[示例](http://example.code.stmt2) + #### 数据订阅 - **创建消费者支持属性列表**： @@ -423,51 +423,40 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - **返回值**：影响的条数。 - **异常**：操作失败抛出 `SchemalessError` 异常。 -#### 参数绑定 -- `def statement(self, sql=None)` - - **接口说明**：使用连接对象创建 stmt 对象, 如果 sql 不空会进行调用 prepare。 - - `sql`: 预编译的 SQL 语句。 - - **返回值**：stmt 对象。 - - **异常**：操作失败抛出 `StatementError` 异常。 +#### 参数绑定 STMT2 +- `def statement2(self, sql=None, option=None)` + - **接口说明**：使用连接对象创建 stmt2 对象 + - **参数说明** + - `sql`: 绑定的 SQL 语句，如果不为空会调用`prepare`函数 + - `option` 传入 TaosStmt2Option 类实例选项 + - **返回值**：stmt2 对象。 + - **异常**：操作失败抛出 `ConnectionError` 异常。 - `def prepare(self, sql)` - - **接口说明**：绑定预编译 sql 语句。 + - **接口说明**：绑定预编译 sql 语句 - **参数说明**： - - `sql`: 预编译的 SQL 语句。 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def set_tbname(self, name)` - - **接口说明**：设置将要写入数据的表名。 + - `sql`: 绑定的 SQL 语句 + - **异常**：操作失败抛出 `ProgrammingError` 异常。 +- `def bind_param(self, tbnames, tags, datas)` + - **接口说明**：以独立数组方式绑定数据 - **参数说明**： - - `name`: 表名，如果需要指定数据库, 例如： `db_name.table_name` 即可。 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def set_tbname_tags(self, name, tags):` - - **接口说明**：设置表和 Tags 数据, 用于自动建表。 + - `tbnames`: 绑定表名数组，数据类型为 list + - `tags`: 绑定 tag 列值数组，数据类型为 list + - `tags`: 绑定普通列值数组，数据类型为 list + - **异常**：操作失败抛出 `StatementError` 异常 +- `def bind_param_with_tables(self, tables)` + - **接口说明**：以独立表方式绑定数据，独立表是以表为组织单位，每张表中有表名，TAG 值及普通列数值属性 - **参数说明**： - - `name`: 表名，如果需要指定数据库, 例如： `db_name.table_name` 即可。 - - `tags`: Tags 数据。 + - `tables`: `BindTable` 独立表对象数组 - **异常**：操作失败抛出 `StatementError` 异常。 -- `def bind_param(self, params, add_batch=True)` - - **接口说明**：绑定一组数据并提交。 - - **参数说明**： - - `params`: 绑定数据。 - - `add_batch`: 是否提交绑定数据。 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def bind_param_batch(self, binds, add_batch=True)` - - **接口说明**：绑定多组数据并提交。 - - **参数说明**： - - `binds`: 绑定数据。 - - `add_batch`: 是否提交绑定数据。 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def add_batch(self)` - - **接口说明**：提交绑定数据。 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def execute(self)` - - **接口说明**：执行将绑定的数据全部写入。 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def affected_rows(self)` - - **接口说明**： 获取写入条数。 - - **返回值**：写入条数。 -- `def close(&self)` - - **接口说明**： 关闭 stmt 对象。 +- `def execute(self) -> int:` + - **接口说明**：执行将绑定数据全部写入 + - **返回值**：影响行数 + - **异常**：操作失败抛出 `QueryError` 异常。 +- `def result(self)` + - **接口说明**： 获取参数绑定查询结果集 + - **返回值**：返回 TaosResult 对象 +- `def close(self)` + - **接口说明**： 关闭 stmt2 对象 #### 数据订阅 - **创建消费者支持属性列表**： diff --git a/tests/docs-examples-test/python.sh b/tests/docs-examples-test/python.sh index 3a9812637c..f7f94db1f2 100644 --- a/tests/docs-examples-test/python.sh +++ b/tests/docs-examples-test/python.sh @@ -196,3 +196,5 @@ check_transactions || exit 1 reset_cache || exit 1 python3 tmq_websocket_example.py +# stmt2 +python3 insert_with_stmt2.py \ No newline at end of file From e6ec05e155e8ef7f0088595ca7187176db664a59 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 14:59:26 +0800 Subject: [PATCH 18/73] docs: update document for python --- docs/zh/14-reference/05-connector/30-python.mdx | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/docs/zh/14-reference/05-connector/30-python.mdx b/docs/zh/14-reference/05-connector/30-python.mdx index c91e9775f4..0c15d866d5 100644 --- a/docs/zh/14-reference/05-connector/30-python.mdx +++ b/docs/zh/14-reference/05-connector/30-python.mdx @@ -258,7 +258,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - **接口说明**：绑定预编译 sql 语句 - **参数说明**： - `sql`: 绑定的 SQL 语句 - - **异常**：操作失败抛出 `ProgrammingError` 异常。 + - **异常**：操作失败抛出 `StatementError` 异常。 - `def bind_param(self, tbnames, tags, datas)` - **接口说明**：以独立数组方式绑定数据 - **参数说明**： @@ -281,7 +281,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - `def close(self)` - **接口说明**： 关闭 stmt2 对象 -[示例](http://example.code.stmt2) +[使用示例](http://https://github.com/taosdata/TDengine/tree/main/docs/examples/python/insert_with_stmt2.py) #### 数据订阅 @@ -435,7 +435,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - **接口说明**：绑定预编译 sql 语句 - **参数说明**： - `sql`: 绑定的 SQL 语句 - - **异常**：操作失败抛出 `ProgrammingError` 异常。 + - **异常**：操作失败抛出 `StatementError` 异常。 - `def bind_param(self, tbnames, tags, datas)` - **接口说明**：以独立数组方式绑定数据 - **参数说明**： @@ -458,6 +458,8 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - `def close(self)` - **接口说明**： 关闭 stmt2 对象 +[使用示例](http://https://github.com/taosdata/TDengine/tree/main/docs/examples/python/insert_with_stmt2.py) + #### 数据订阅 - **创建消费者支持属性列表**： - td.connect.ip：主机地址。 From defa7ffc42c8bd2ae08ac7106dea5566c815bf03 Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Mon, 30 Dec 2024 15:00:41 +0800 Subject: [PATCH 19/73] chore: remove wrong build option skip-checks: true --- .github/workflows/taosd-ci-build.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/.github/workflows/taosd-ci-build.yml b/.github/workflows/taosd-ci-build.yml index 8d8a120d76..cd5f1eeeae 100644 --- a/.github/workflows/taosd-ci-build.yml +++ b/.github/workflows/taosd-ci-build.yml @@ -39,9 +39,11 @@ jobs: - name: Build and install TDengine run: | mkdir debug && cd debug - cmake .. -DBUILD_HTTP=false -DBUILD_JDBC=false \ - -DBUILD_TOOLS=true -DBUILD_TEST=false \ - -DBUILD_KEEPER=true -DBUILD_DEPENDENCY_TESTS=false + cmake .. -DBUILD_TOOLS=true \ + -DBUILD_KEEPER=true \ + -DBUILD_HTTP=false \ + -DBUILD_TEST=false \ + -DBUILD_DEPENDENCY_TESTS=false make -j 4 sudo make install which taosd From f0627fd7e8042102a9843f205d741928062a0342 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 15:25:25 +0800 Subject: [PATCH 20/73] fix: remove tdengine case to python-connector repo --- docs/examples/python/insert_with_stmt2.py | 626 ------------------ .../14-reference/05-connector/30-python.mdx | 8 +- tests/docs-examples-test/python.sh | 2 - 3 files changed, 4 insertions(+), 632 deletions(-) delete mode 100644 docs/examples/python/insert_with_stmt2.py diff --git a/docs/examples/python/insert_with_stmt2.py b/docs/examples/python/insert_with_stmt2.py deleted file mode 100644 index 433b283fe3..0000000000 --- a/docs/examples/python/insert_with_stmt2.py +++ /dev/null @@ -1,626 +0,0 @@ -# encoding:UTF-8 -from ctypes import * -from datetime import datetime -# geometry support -from shapely.wkb import dumps, loads -from shapely.wkt import dumps as wkt_dumps, loads as wkt_loads - -import taos -import math -import traceback -from taos.statement2 import * -from taos.constants import FieldType -from taos import log -from taos import bind2 - -# input WKT return WKB (bytes object) -def WKB(wkt, hex = False): - if wkt is None: - return None - wkb = wkt_loads(wkt) - wkb_bytes = dumps(wkb, hex) - return wkb_bytes - -def compareLine(oris, rows): - n = len(oris) - if len(rows) != n: - return False - log.debug(f" len is {n} oris={oris} rows={rows}") - for i in range(n): - if oris[i] != rows[i]: - if type(rows[i]) == bool: - if bool(oris[i]) != rows[i]: - log.debug1(f" diff bool i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") - return False - else: - log.debug1(f" float i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") - elif type(rows[i]) == float: - if math.isclose(oris[i], rows[i], rel_tol=1e-3) is False: - log.debug1(f" diff float i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") - return False - else: - log.debug1(f" float i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") - else: - log.debug1(f" diff i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") - return False - else: - log.debug1(f" i={i} oris[i] == rows[i] {oris[i]} == {rows[i]}") - - return True - - -def checkResultCorrect(conn, sql, tagsTb, datasTb): - # column to rows - log.debug(f"check sql correct: {sql}\n") - oris = [] - ncol = len(datasTb) - nrow = len(datasTb[0]) - - for i in range(nrow): - row = [] - for j in range(ncol): - if j == 0: - # ts column - c0 = datasTb[j][i] - if type(c0) is int : - row.append(datasTb[j][i]) - else: - ts = int(bind2.datetime_to_timestamp(c0, PrecisionEnum.Milliseconds).value) - row.append(ts) - else: - row.append(datasTb[j][i]) - - if tagsTb is not None: - row += tagsTb - oris.append(row) - - # fetch all - lres = [] - log.debug(sql) - res = conn.query(sql) - i = 0 - for row in res: - lrow = list(row) - lrow[0] = int(lrow[0].timestamp()*1000) - if compareLine(oris[i], lrow) is False: - log.info(f"insert data differet. i={i} expect ori data={oris[i]} query from db ={lrow}") - raise(BaseException("check insert data correct failed.")) - else: - log.debug(f"i={i} origin data same with get from db\n") - log.debug(f" origin data = {oris[i]} \n") - log.debug(f" get from db = {lrow} \n") - i += 1 - - -def checkResultCorrects(conn, dbname, stbname, tbnames, tags, datas): - count = len(tbnames) - for i in range(count): - if stbname is None: - sql = f"select * from {dbname}.{tbnames[i]} " - else: - sql = f"select * from {dbname}.{stbname} where tbname='{tbnames[i]}' " - - checkResultCorrect(conn, sql, tags[i], datas[i]) - - print("insert data check correct ..................... ok\n") - - -def prepare(conn, dbname, stbname, ntb1, ntb2): - conn.execute("drop database if exists %s" % dbname) - conn.execute("create database if not exists %s precision 'ms' " % dbname) - conn.select_db(dbname) - # stable - sql = f"create table if not exists {dbname}.{stbname}(ts timestamp, name binary(32), sex bool, score int) tags(grade nchar(8), class int)" - conn.execute(sql) - # normal table - sql = f"create table if not exists {dbname}.{ntb1} (ts timestamp, name varbinary(32), sex bool, score float, geo geometry(128))" - conn.execute(sql) - sql = f"create table if not exists {dbname}.{ntb2} (ts timestamp, name varbinary(32), sex bool, score float, geo geometry(128))" - conn.execute(sql) - - -# performace is high -def insert_bind_param(conn, stmt2, dbname, stbname): - # - # table info , write 5 lines to 3 child tables d0, d1, d2 with super table - # - tbnames = ["d1","d2","d3"] - - tags = [ - ["grade1", 1], - ["grade1", None], - [None , 3] - ] - datas = [ - # class 1 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], - ["Mary", "Tom", "Jack", "Jane", "alex" ,None ], - [0, 1, 1, 0, 1 ,None ], - [98, 80, 60, 100, 99 ,None ] - ], - # class 2 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], - ["Mary2", "Tom2", "Jack2", "Jane2", "alex2" ,None ], - [0, 1, 1, 0, 1 ,0 ], - [298, 280, 260, 2100, 299 ,None ] - ], - # class 3 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], - ["Mary3", "Tom3", "Jack3", "Jane3", "alex3" ,"Mark" ], - [0, 1, 1, 0, 1 ,None ], - [398, 380, 360, 3100, 399 ,None ] - ] - ] - - stmt2.bind_param(tbnames, tags, datas) - stmt2.execute() - - # check correct - checkResultCorrects(conn, dbname, stbname, tbnames, tags, datas) - - -def insert_bind_param_normal_tables(conn, stmt2, dbname, ntb): - tbnames = [ntb] - tags = None - wkts = [None, b"POINT(121.213 31.234)", b"POINT(122.22 32.222)", None, b"POINT(124.22 34.222)"] - wkbs = [WKB(wkt) for wkt in wkts] - - datas = [ - # table 1 - [ - # student - [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], - [b"Mary", b"tom", b"Jack", b"Jane", None ], - [0, 3.14, True, 0, 1 ], - [98, 99.87, 60, 100, 99 ], - wkbs - ] - ] - - stmt2.bind_param(tbnames, tags, datas) - stmt2.execute() - - # check correct - checkResultCorrects(conn, dbname, None, tbnames, [None], datas) - -def insert_bind_param_with_table(conn, stmt2, dbname, stbname, ctb): - - tbnames = None - tags = [ - ["grade2", 1] - ] - - # prepare data - datas = [ - # table 1 - [ - # student - [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], - ["Mary", "Tom", "Jack", "Jane", "alex" ], - [0, 1, 1, 0, 1 ], - [98, 80, 60, 100, 99 ] - ] - ] - - stmt2.bind_param(tbnames, tags, datas) - stmt2.execute() - - # check correct - checkResultCorrects(conn, dbname, stbname, [ctb], tags, datas) - - -# insert with single table (performance is lower) -def insert_bind_param_with_tables(conn, stmt2, dbname, stbname): - - tbnames = ["t1", "t2", "t3"] - tags = [ - ["grade2", 1], - ["grade2", 2], - ["grade2", 3] - ] - - # prepare data - datas = [ - # table 1 - [ - # student - [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], - ["Mary", "Tom", "Jack", "Jane", "alex" ], - [0, 1, 1, 0, 1 ], - [98, 80, 60, 100, 99 ] - ], - # table 2 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], - ["Mary2", "Tom2", "Jack2", "Jane2", "alex2" ], - [0, 1, 1, 0, 1 ], - [298, 280, 260, 2100, 299 ] - ], - # table 3 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], - ["Mary3", "Tom3", "Jack3", "Jane3", "alex3" ], - [0, 1, 1, 0, 1 ], - [398, 380, 360, 3100, 399 ] - ] - ] - - table0 = BindTable(tbnames[0], tags[0]) - table1 = BindTable(tbnames[1], tags[1]) - table2 = BindTable(tbnames[2], tags[2]) - - for data in datas[0]: - table0.add_col_data(data) - for data in datas[1]: - table1.add_col_data(data) - for data in datas[2]: - table2.add_col_data(data) - - # bind with single table - stmt2.bind_param_with_tables([table0, table1, table2]) - stmt2.execute() - - # check correct - checkResultCorrects(conn, dbname, stbname, tbnames, tags, datas) - - -def do_check_invalid(stmt2, tbnames, tags, datas): - table0 = BindTable(tbnames[0], tags[0]) - table1 = BindTable(tbnames[1], tags[1]) - table2 = BindTable(tbnames[2], tags[2]) - - for data in datas[0]: - table0.add_col_data(data) - for data in datas[1]: - table1.add_col_data(data) - for data in datas[2]: - table2.add_col_data(data) - - # bind with single table - try: - stmt2.bind_param_with_tables([table0, table1, table2]) - stmt2.execute() - except Exception as err: - #traceback.print_stack() - print(f"failed to do_check_invalid. err={err}") - return - - print(f"input invalid data passed , unexpect. \ntbnames={tbnames}\ntags={tags} \ndatas={datas} \n") - assert False - - -def check_input_invalid_param(conn, stmt2, dbname, stbname): - - tbnames = ["t1", "t2", "t3"] - tags = [ - ["grade2", 1], - ["grade2", 2], - ["grade2", 3] - ] - - # prepare data - datas = [ - # table 1 - [ - # student - [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], - ["Mary", "Tom", "Jack", "Jane", "alex" ], - [0, 1, 1, 0, 1 ], - [98, 80, 60, 100, 99 ] - ], - # table 2 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], - ["Mary2", "Tom2", "Jack2", "Jane2", "alex2" ], - [0, 1, 1, 0, 1 ], - [298, 280, 260, 2100, 299 ] - ], - # table 3 - [ - # student - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004], - ["Mary3", "Tom3", "Jack3", "Jane3", "alex3" ], - [0, 1, 1, 0, 1 ], - [398, 380, 360, 3100, 399 ] - ] - ] - - # some tags is none - tags1 = [ ["grade2", 1], None, ["grade2", 3] ] - do_check_invalid(stmt2, tbnames, tags1, datas) - - # timestamp is over range - origin = datas[0][0][0] - datas[0][0][0] = 100000000000000000000000 - do_check_invalid(stmt2, tbnames, tags, datas) - datas[0][0][0] = origin # restore - - -# insert with single table (performance is lower) -def insert_with_normal_tables(conn, stmt2, dbname, ntb): - - tbnames = [ntb] - tags = [None] - # prepare data - - wkts = [None, "POINT(121.213 31.234)", "POINT(122.22 32.222)", None, "POINT(124.22 34.222)"] - wkbs = [WKB(wkt) for wkt in wkts] - - datas = [ - # table 1 - [ - # student - [1601481600000,1601481600004,"2024-09-19 10:00:00", "2024-09-19 10:00:01.123", datetime(2024,9,20,10,11,12,456)], - [b"Mary", b"tom", b"Jack", b"Jane", None ], - [0, 3.14, True, 0, 1 ], - [98, 99.87, 60, 100, 99 ], - wkbs - ] - ] - - table0 = BindTable(tbnames[0], tags[0]) - for data in datas[0]: - table0.add_col_data(data) - - # bind with single table - stmt2.bind_param_with_tables([table0]) - stmt2.execute() - - # check correct - checkResultCorrects(conn, dbname, None, tbnames, tags, datas) - - -def test_stmt2_prepare_empty_sql(conn): - if not IS_V3: - print(" test_stmt2_prepare_empty_sql not support TDengine 2.X version.") - return - - try: - # prepare - stmt2 = conn.statement2() - stmt2.prepare(sql='') - - # should not run here - conn.close() - print("prepare empty sql ............................. failed\n") - assert False - - except StatementError as err: - print("prepare empty sql ............................. ok\n") - conn.close() - - -def test_bind_invalid_tbnames_type(): - if not IS_V3: - print(" test_bind_invalid_tbnames_type not support TDengine 2.X version.") - return - - dbname = "stmt2" - stbname = "stmt2_stable" - subtbname = "stmt2_subtable" - - try: - conn = taos.connect() - conn.execute(f"drop database if exists {dbname}") - conn.execute(f"create database {dbname}") - conn.select_db(dbname) - conn.execute(f"create stable {stbname} (ts timestamp, a int) tags (b int);") - conn.execute(f"create table {subtbname} using {stbname} tags(0);") - - stmt2 = conn.statement2(f"insert into ? using {dbname}.{stbname} tags(?) values(?,?)") - - tags = [[1]] - datas = [[[1626861392589], [1]]] - - stmt2.bind_param(subtbname, tags, datas) - - # should not run here - conn.close() - print("bind invalid tbnames type ..................... failed\n") - assert False - - except StatementError as err: - print("bind invalid tbnames type ..................... ok\n") - conn.close() - - -# -# insert -# -def test_stmt2_insert(conn): - if not IS_V3: - print(" test_stmt2_query not support TDengine 2.X version.") - return - - dbname = "stmt2" - stbname = "meters" - ntb1 = "ntb1" - ntb2 = "ntb2" - - try: - prepare(conn, dbname, stbname, ntb1, ntb2) - - ctb = 'ctb' # child table - stmt2 = conn.statement2(f"insert into {dbname}.{ctb} using {dbname}.{stbname} tags (?,?) values(?,?,?,?)") - insert_bind_param_with_table(conn, stmt2, dbname, stbname, ctb) - print("insert child table ........................... ok\n") - stmt2.close() - - # # prepare - # stmt2 = conn.statement2(f"insert into ? using {dbname}.{stbname} tags(?,?) values(?,?,?,?)") - # print("insert prepare sql ............................ ok\n") - # - # # insert with table - # insert_bind_param_with_tables(conn, stmt2, dbname, stbname) - # print("insert bind with tables ....................... ok\n") - # check_input_invalid_param(conn, stmt2, dbname, stbname) - # print("check input invalid params .................... ok\n") - # - # # insert with split args - # insert_bind_param(conn, stmt2, dbname, stbname) - # print("insert bind ................................... ok\n") - # print("insert execute ................................ ok\n") - # stmt2.close() - - # ntb1 - stmt2 = conn.statement2(f"insert into {dbname}.{ntb1} values(?,?,?,?,?)") - insert_with_normal_tables(conn, stmt2, dbname, ntb1) - print("insert normal tables .......................... ok\n") - stmt2.close() - - # ntb2 - stmt2 = conn.statement2(f"insert into {dbname}.{ntb2} values(?,?,?,?,?)") - insert_bind_param_normal_tables(conn, stmt2, dbname, ntb2) - print("insert normal tables (bind param) ............. ok\n") - stmt2.close() - - conn.close() - print("test_stmt2_insert ............................. [passed]\n") - except Exception as err: - #conn.execute("drop database if exists %s" % dbname) - print("test_stmt2_insert ............................. failed\n") - conn.close() - raise err - - -# -# ------------------------ query ------------------- -# -def query_bind_param(conn, stmt2): - # set param - #tbnames = ["d2"] - tbnames = None - tags = None - datas = [ - # class 1 - [ - # where name in ('Tom2','alex2') or score > 1000;" - ["Tom2"], - [1000] - ] - ] - - # set param - types = [FieldType.C_BINARY, FieldType.C_INT] - stmt2.set_columns_type(types) - - # bind - stmt2.bind_param(tbnames, tags, datas) - - -# compare -def compare_result(conn, sql2, res2): - lres1 = [] - lres2 = [] - - # shor res2 - for row in res2: - log.debug(f" res2 rows = {row} \n") - lres2.append(row) - - res1 = conn.query(sql2) - for row in res1: - log.debug(f" res1 rows = {row} \n") - lres1.append(row) - - row1 = len(lres1) - row2 = len(lres2) - col1 = len(lres1[0]) - col2 = len(lres2[0]) - - # check number - if row1 != row2: - err = f"two results row count different. row1={row1} row2={row2}" - raise(BaseException(err)) - if col1 != col2: - err = f" two results column count different. col1={col1} col2={col2}" - raise(BaseException(err)) - - for i in range(row1): - for j in range(col1): - if lres1[i][j] != lres2[i][j]: - raise(f" two results data different. i={i} j={j} data1={res1[i][j]} data2={res2[i][j]}\n") - -# query -def test_stmt2_query(conn): - if not IS_V3: - print(" test_stmt2_query not support TDengine 2.X version.") - return - - dbname = "stmt2" - stbname = "meters" - ntb1 = "ntb1" - ntb2 = "ntb2" - sql1 = f"select * from {dbname}.d2 where name in (?) or score > ? ;" - sql2 = f"select * from {dbname}.d2 where name in ('Tom2') or score > 1000;" - - try: - # prepare - prepare(conn, dbname, stbname, ntb1, ntb2) - - # prepare - # stmt2 = conn.statement2(f"insert into ? using {dbname}.{stbname} tags(?,?) values(?,?,?,?)") - # insert_bind_param_with_tables(conn, stmt2, dbname, stbname) - # insert_bind_param(conn, stmt2, dbname, stbname) - # stmt2.close() - # print("insert bind & execute ......................... ok\n") - - conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.000', 'Mary2', false, 298)") - conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.001', 'Tom2', true, 280)") - conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.002', 'Jack2', true, 260)") - conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.003', 'Jane2', false, 2100)") - conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.004', 'alex2', true, 299)") - conn.execute(f"insert into d2 using {stbname} tags('grade1', 2) values('2020-10-01 00:00:00.005', NULL, false, NULL)") - - - # statement2 - stmt2 = conn.statement2(sql1) - print("query prepare sql ............................. ok\n") - - - # insert with table - #insert_bind_param_with_tables(conn, stmt2) - - - # bind - query_bind_param(conn, stmt2) - print("query bind param .............................. ok\n") - - # query execute - stmt2.execute() - - # fetch result - res2 = stmt2.result() - - # check result - compare_result(conn, sql2, res2) - print("query check corrent ........................... ok\n") - - #conn.execute("drop database if exists %s" % dbname) - stmt2.close() - conn.close() - print("test_stmt2_query .............................. [passed]\n") - - except Exception as err: - print("query ......................................... failed\n") - conn.close() - raise err - - -if __name__ == "__main__": - print("start stmt2 test case...\n") - taos.log.setting(True, True, True, True, True, False) - # insert - test_stmt2_insert(taos.connect()) - # query - test_stmt2_query(taos.connect()) - print("end stmt2 test case.\n") \ No newline at end of file diff --git a/docs/zh/14-reference/05-connector/30-python.mdx b/docs/zh/14-reference/05-connector/30-python.mdx index 0c15d866d5..cdf63f918c 100644 --- a/docs/zh/14-reference/05-connector/30-python.mdx +++ b/docs/zh/14-reference/05-connector/30-python.mdx @@ -52,7 +52,8 @@ Python Connector 历史版本（建议使用最新版本的 `taospy`）： | Python Connector 版本 | 主要变化 | TDengine 版本 | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | -| 2.7.18 | 支持 Apache Superset BI 产品 | - | +| 2.7.19 | 支持 Apache Superset 连接 TDengine Cloud 数据源 | - | +| 2.7.18 | 支持 Apache Superset 产品连接本地 TDengine 数据源 | - | | 2.7.16 | 新增订阅配置 (session.timeout.ms, max.poll.interval.ms) | - | | 2.7.15 | 新增 VARBINARY 和 GEOMETRY 类型支持 | - | | 2.7.14 | 修复已知问题 | - | @@ -131,7 +132,8 @@ TDengine 目前支持时间戳、数字、字符、布尔类型，与 Python 对 | [json_tag.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/json-tag.py) | 使用 JSON 类型的标签 | | [tmq_consumer.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/tmq_consumer.py) | tmq 订阅 | | [native_all_type_query.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_query.py) | 支持全部类型示例 | -| [native_all_type_stmt.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_stmt.py) | 参数绑定支持全部类型示例 | +| [native_all_type_stmt.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_stmt.py) | 参数绑定 stmt 全部类型示例 | +| [insert_with_stmt2.py](https://github.com/taosdata/taos-connector-python/blob/main/tests/test_stmt2.py) | 参数绑定 stmt2 写入示例 | 示例程序源码请参考： @@ -281,7 +283,6 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - `def close(self)` - **接口说明**： 关闭 stmt2 对象 -[使用示例](http://https://github.com/taosdata/TDengine/tree/main/docs/examples/python/insert_with_stmt2.py) #### 数据订阅 @@ -458,7 +459,6 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - `def close(self)` - **接口说明**： 关闭 stmt2 对象 -[使用示例](http://https://github.com/taosdata/TDengine/tree/main/docs/examples/python/insert_with_stmt2.py) #### 数据订阅 - **创建消费者支持属性列表**： diff --git a/tests/docs-examples-test/python.sh b/tests/docs-examples-test/python.sh index f7f94db1f2..3a9812637c 100644 --- a/tests/docs-examples-test/python.sh +++ b/tests/docs-examples-test/python.sh @@ -196,5 +196,3 @@ check_transactions || exit 1 reset_cache || exit 1 python3 tmq_websocket_example.py -# stmt2 -python3 insert_with_stmt2.py \ No newline at end of file From bbfa55a90ffb3a2ed09441419580fa8bfbb008e8 Mon Sep 17 00:00:00 2001 From: Shengliang Guan Date: Mon, 30 Dec 2024 16:05:02 +0800 Subject: [PATCH 21/73] doc: password --- docs/zh/08-operation/14-user.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/08-operation/14-user.md b/docs/zh/08-operation/14-user.md index a894570b46..3a080619aa 100644 --- a/docs/zh/08-operation/14-user.md +++ b/docs/zh/08-operation/14-user.md @@ -17,7 +17,7 @@ create user user_name pass'password' [sysinfo {1|0}] [createdb {1|0}] 相关参数说明如下。 - user_name：用户名最长不超过 23 个字节。 -- password：密码长度必须为 8 到 16 位，并且至少包含大写字母、小写字母、数字、特殊字符中的三类。特殊字符包括 `! @ # $ % ^ & * ( ) - _ + = [ ] { } : ; > < ? | ~ , .`。 +- password：密码长度必须为 8 到 16 位，并且至少包含大写字母、小写字母、数字、特殊字符中的三类。特殊字符包括 `! @ # $ % ^ & * ( ) - _ + = [ ] { } : ; > < ? | ~ , .`。（始自 3.3.5.0 版本） - sysinfo ：用户是否可以查看系统信息。1 表示可以查看，0 表示不可以查看。系统信息包括服务端配置信息、服务端各种节点信息，如 dnode、查询节点（qnode）等，以及与存储相关的信息等。默认为可以查看系统信息。 - createdb：用户是否可以创建数据库。1 表示可以创建，0 表示不可以创建。缺省值为 0。// 从 TDengine 企业版 3.3.2.0 开始支持 From 2c35ec8ae9a8859871cc4330add7872e9f257dcb Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 16:11:11 +0800 Subject: [PATCH 22/73] docs: add stmt2_native.py example --- docs/en/07-develop/05-stmt.md | 2 +- docs/examples/python/stmt2_native.py | 78 ++++++++++++++++++++++++++++ docs/zh/07-develop/05-stmt.md | 2 +- tests/docs-examples-test/python.sh | 1 + 4 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 docs/examples/python/stmt2_native.py diff --git a/docs/en/07-develop/05-stmt.md b/docs/en/07-develop/05-stmt.md index 485315bc93..11b055bcf9 100644 --- a/docs/en/07-develop/05-stmt.md +++ b/docs/en/07-develop/05-stmt.md @@ -98,7 +98,7 @@ This is a [more detailed parameter binding example](https://github.com/taosdata/ ```python -{{#include docs/examples/python/stmt_native.py}} +{{#include docs/examples/python/stmt2_native.py}} ``` diff --git a/docs/examples/python/stmt2_native.py b/docs/examples/python/stmt2_native.py new file mode 100644 index 0000000000..7e6d59466e --- /dev/null +++ b/docs/examples/python/stmt2_native.py @@ -0,0 +1,78 @@ +import taos +from datetime import datetime +import random + +conn = None +stmt2 = None +host="localhost" +port=6030 +try: + # 1 connect + conn = taos.connect( + user="root", + password="taosdata", + host=host, + port=port, + ) + + # 2 create db and table + conn.execute("CREATE DATABASE IF NOT EXISTS power") + conn.execute("USE power") + conn.execute( + "CREATE TABLE IF NOT EXISTS `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) TAGS (`groupid` INT, `location` BINARY(16))" + ) + + # 3 prepare + sql = "INSERT INTO ? USING meters (groupid, location) TAGS(?,?) VALUES (?,?,?,?)" + stmt2 = conn.statement2(sql) + + # table name array + tbnames = ["d0","d1","d2"] + # tag data array + tags = [ + [1, "BeiJing"], + [2, None], + [3, "ShangHai"] + ] + # column data array + datas = [ + # d0 tabled + [ + [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], + [10.1, 10.2, 10.3, 10.4, 10.5 ,None ], + [98, None, 60, 100, 99 ,128 ], + [0, 1, 0, 0, 1 ,0 ] + ], + # d1 tabled + [ + [1601481700000,1601481700001,1601481700002,1601481700003,1601481700004,1601481700005], + [10.1, 10.2, 10.3, 10.4, 10.5 ,11.2 ], + [98, 80, 60, 100, 99 ,128 ], + [0, 1, 0, 0, 1 ,0 ] + ], + # d2 tabled + [ + [1601481800000,1601481800001,1601481800002,1601481800003,1601481800004,1601481800005], + [10.1, 10.2, 10.3, 10.4, 10.5 ,13.4 ], + [98, 80, 60, 100, 99 ,128 ], + [0, 1, 0, None, 1 ,0 ] + ], + ] + + # 4 bind param + stmt2.bind_param(tbnames, tags, datas) + + # 5 execute + stmt2.execute() + + # show + print(f"Successfully inserted with stmt2 to power.meters.") + +except Exception as err: + print(f"Failed to insert to table meters using stmt2, ErrMessage:{err}") + raise err +finally: + if stmt2: + stmt2.close() + if conn: + conn.close() diff --git a/docs/zh/07-develop/05-stmt.md b/docs/zh/07-develop/05-stmt.md index 045126b333..1917a86e74 100644 --- a/docs/zh/07-develop/05-stmt.md +++ b/docs/zh/07-develop/05-stmt.md @@ -93,7 +93,7 @@ import TabItem from "@theme/TabItem"; ```python -{{#include docs/examples/python/stmt_native.py}} +{{#include docs/examples/python/stmt2_native.py}} ``` diff --git a/tests/docs-examples-test/python.sh b/tests/docs-examples-test/python.sh index 3a9812637c..536155437b 100644 --- a/tests/docs-examples-test/python.sh +++ b/tests/docs-examples-test/python.sh @@ -196,3 +196,4 @@ check_transactions || exit 1 reset_cache || exit 1 python3 tmq_websocket_example.py +python3 stmt2_native.py \ No newline at end of file From 5fe895c6d228938f3c98083586057f3786819257 Mon Sep 17 00:00:00 2001 From: qevolg <2227465945@qq.com> Date: Mon, 30 Dec 2024 16:31:11 +0800 Subject: [PATCH 23/73] fix(keeper): add gitinfo --- tools/CMakeLists.txt | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/CMakeLists.txt b/tools/CMakeLists.txt index d058d7a52f..1ee2bc4ce6 100644 --- a/tools/CMakeLists.txt +++ b/tools/CMakeLists.txt @@ -251,7 +251,7 @@ IF(TD_BUILD_KEEPER) PATCH_COMMAND COMMAND git clean -f -d BUILD_COMMAND - COMMAND go build -a -ldflags "-X 'github.com/taosdata/taoskeeper/version.Version=${taos_version}' -X 'github.com/taosdata/taoskeeper/version.CommitID=${taoskeeper_commit_sha1}' -X 'github.com/taosdata/taoskeeper/version.BuildInfo=${TD_VER_OSTYPE}-${TD_VER_CPUTYPE} ${TD_VER_DATE}'" + COMMAND go build -a -ldflags "-X 'github.com/taosdata/taoskeeper/version.Version=${taos_version}' -X 'github.com/taosdata/taoskeeper/version.Gitinfo=${taoskeeper_commit_sha1}' -X 'github.com/taosdata/taoskeeper/version.CommitID=${taoskeeper_commit_sha1}' -X 'github.com/taosdata/taoskeeper/version.BuildInfo=${TD_VER_OSTYPE}-${TD_VER_CPUTYPE} ${TD_VER_DATE}'" INSTALL_COMMAND COMMAND cmake -E echo "Comparessing taoskeeper.exe" COMMAND cmake -E time upx taoskeeper.exe @@ -278,7 +278,7 @@ IF(TD_BUILD_KEEPER) PATCH_COMMAND COMMAND git clean -f -d BUILD_COMMAND - COMMAND go build -a -ldflags "-X 'github.com/taosdata/taoskeeper/version.Version=${taos_version}' -X 'github.com/taosdata/taoskeeper/version.CommitID=${taoskeeper_commit_sha1}' -X 'github.com/taosdata/taoskeeper/version.BuildInfo=${TD_VER_OSTYPE}-${TD_VER_CPUTYPE} ${TD_VER_DATE}'" + COMMAND go build -a -ldflags "-X 'github.com/taosdata/taoskeeper/version.Version=${taos_version}' -X 'github.com/taosdata/taoskeeper/version.Gitinfo=${taoskeeper_commit_sha1}' -X 'github.com/taosdata/taoskeeper/version.CommitID=${taoskeeper_commit_sha1}' -X 'github.com/taosdata/taoskeeper/version.BuildInfo=${TD_VER_OSTYPE}-${TD_VER_CPUTYPE} ${TD_VER_DATE}'" INSTALL_COMMAND COMMAND cmake -E echo "Copy taoskeeper" COMMAND cmake -E copy taoskeeper ${CMAKE_BINARY_DIR}/build/bin From f6b180ff569aab9ba859a6ec84be019a7fc5c4fd Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 16:57:16 +0800 Subject: [PATCH 24/73] fix: use old case model --- docs/examples/python/stmt2_native.py | 59 ++++++++++++---------------- 1 file changed, 26 insertions(+), 33 deletions(-) diff --git a/docs/examples/python/stmt2_native.py b/docs/examples/python/stmt2_native.py index 7e6d59466e..72f01c9038 100644 --- a/docs/examples/python/stmt2_native.py +++ b/docs/examples/python/stmt2_native.py @@ -2,6 +2,9 @@ import taos from datetime import datetime import random +numOfSubTable = 10 +numOfRow = 10 + conn = None stmt2 = None host="localhost" @@ -26,38 +29,28 @@ try: sql = "INSERT INTO ? USING meters (groupid, location) TAGS(?,?) VALUES (?,?,?,?)" stmt2 = conn.statement2(sql) - # table name array - tbnames = ["d0","d1","d2"] - # tag data array - tags = [ - [1, "BeiJing"], - [2, None], - [3, "ShangHai"] - ] - # column data array - datas = [ - # d0 tabled - [ - [1601481600000,1601481600001,1601481600002,1601481600003,1601481600004,1601481600005], - [10.1, 10.2, 10.3, 10.4, 10.5 ,None ], - [98, None, 60, 100, 99 ,128 ], - [0, 1, 0, 0, 1 ,0 ] - ], - # d1 tabled - [ - [1601481700000,1601481700001,1601481700002,1601481700003,1601481700004,1601481700005], - [10.1, 10.2, 10.3, 10.4, 10.5 ,11.2 ], - [98, 80, 60, 100, 99 ,128 ], - [0, 1, 0, 0, 1 ,0 ] - ], - # d2 tabled - [ - [1601481800000,1601481800001,1601481800002,1601481800003,1601481800004,1601481800005], - [10.1, 10.2, 10.3, 10.4, 10.5 ,13.4 ], - [98, 80, 60, 100, 99 ,128 ], - [0, 1, 0, None, 1 ,0 ] - ], - ] + tbnames = [] + tags = [] + datas = [] + + for i in range(numOfSubTable): + # tbnames + tbnames.append(f"d_bind_{i}") + # tags + tags.append([i, f"location_{i}"]) + # datas + current = int(datetime.now().timestamp() * 1000) + timestamps = [] + currents = [] + voltages = [] + phases = [] + for j in range (numOfRow): + timestamps.append(current + i*1000 + j) + currents.append(float(random.random() * 30)) + voltages.append(random.randint(100, 300)) + phases.append(float(random.random())) + data = [timestamps, currents, voltages, phases] + datas.append(data) # 4 bind param stmt2.bind_param(tbnames, tags, datas) @@ -66,7 +59,7 @@ try: stmt2.execute() # show - print(f"Successfully inserted with stmt2 to power.meters.") + print(f"Successfully inserted with stmt2 to power.meters. child={numOfSubTable} rows={numOfRow} \n") except Exception as err: print(f"Failed to insert to table meters using stmt2, ErrMessage:{err}") From 62d3df49d1d5513433b114dc02cc20dd22417c13 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 17:16:23 +0800 Subject: [PATCH 25/73] docs: remove STMT2 --- .../14-reference/05-connector/30-python.mdx | 65 ++++++++++--------- 1 file changed, 34 insertions(+), 31 deletions(-) diff --git a/docs/zh/14-reference/05-connector/30-python.mdx b/docs/zh/14-reference/05-connector/30-python.mdx index cdf63f918c..ad43ce19df 100644 --- a/docs/zh/14-reference/05-connector/30-python.mdx +++ b/docs/zh/14-reference/05-connector/30-python.mdx @@ -52,6 +52,7 @@ Python Connector 历史版本（建议使用最新版本的 `taospy`）： | Python Connector 版本 | 主要变化 | TDengine 版本 | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | +| 2.7.20 | Native 支持 STMT2 写入 | - | | 2.7.19 | 支持 Apache Superset 连接 TDengine Cloud 数据源 | - | | 2.7.18 | 支持 Apache Superset 产品连接本地 TDengine 数据源 | - | | 2.7.16 | 新增订阅配置 (session.timeout.ms, max.poll.interval.ms) | - | @@ -248,41 +249,43 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - `reqId`: 用于问题追踪。 - **异常**：操作失败抛出 `DataError` 或 `OperationalError` 异常。 -#### 参数绑定 STMT2 -- `def statement2(self, sql=None, option=None)` - - **接口说明**：使用连接对象创建 stmt2 对象 - - **参数说明** - - `sql`: 绑定的 SQL 语句，如果不为空会调用`prepare`函数 - - `option` 传入 TaosStmt2Option 类实例选项 - - **返回值**：stmt2 对象。 +#### 参数绑定 +- `fn statement(&self) -> PyResult` + - **接口说明**：使用 连接 对象创建 stmt 对象。 + - **返回值**：stmt 对象。 - **异常**：操作失败抛出 `ConnectionError` 异常。 -- `def prepare(self, sql)` - - **接口说明**：绑定预编译 sql 语句 +- `fn prepare(&mut self, sql: &str) -> PyResult<()>` + - **接口说明**：绑定预编译 sql 语句。 - **参数说明**： - - `sql`: 绑定的 SQL 语句 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def bind_param(self, tbnames, tags, datas)` - - **接口说明**：以独立数组方式绑定数据 + - `sql`: 预编译的 SQL 语句。 + - **异常**：操作失败抛出 `ProgrammingError` 异常。 +- `fn set_tbname(&mut self, table_name: &str) -> PyResult<()>` + - **接口说明**：设置将要写入数据的表名。 - **参数说明**： - - `tbnames`: 绑定表名数组，数据类型为 list - - `tags`: 绑定 tag 列值数组，数据类型为 list - - `tags`: 绑定普通列值数组，数据类型为 list - - **异常**：操作失败抛出 `StatementError` 异常 -- `def bind_param_with_tables(self, tables)` - - **接口说明**：以独立表方式绑定数据，独立表是以表为组织单位，每张表中有表名，TAG 值及普通列数值属性 + - `tableName`: 表名，如果需要指定数据库, 例如： `db_name.table_name` 即可。 + - **异常**：操作失败抛出 `ProgrammingError` 异常。 +- `fn set_tags(&mut self, tags: Vec) -> PyResult<()>` + - **接口说明**：设置表 Tags 数据, 用于自动建表。 - **参数说明**： - - `tables`: `BindTable` 独立表对象数组 - - **异常**：操作失败抛出 `StatementError` 异常。 -- `def execute(self) -> int:` - - **接口说明**：执行将绑定数据全部写入 - - **返回值**：影响行数 + - `paramsArray`: Tags 数据。 + - **异常**：操作失败抛出 `ProgrammingError` 异常。 +- `fn bind_param(&mut self, params: Vec) -> PyResult<()>` + - **接口说明**：绑定数据。 + - **参数说明**： + - `paramsArray`: 绑定数据。 + - **异常**：操作失败抛出 `ProgrammingError` 异常。 +- `fn add_batch(&mut self) -> PyResult<()>` + - **接口说明**：提交绑定数据。 + - **异常**：操作失败抛出 `ProgrammingError` 异常。 +- `fn execute(&mut self) -> PyResult` + - **接口说明**：执行将绑定的数据全部写入。 + - **返回值**：写入条数。 - **异常**：操作失败抛出 `QueryError` 异常。 -- `def result(self)` - - **接口说明**： 获取参数绑定查询结果集 - - **返回值**：返回 TaosResult 对象 -- `def close(self)` - - **接口说明**： 关闭 stmt2 对象 - +- `fn affect_rows(&mut self) -> PyResult` + - **接口说明**： 获取写入条数。 + - **返回值**：写入条数。 +- `fn close(&self) -> PyResult<()>` + - **接口说明**： 关闭 stmt 对象。 #### 数据订阅 @@ -424,7 +427,7 @@ TaosResult 对象可以通过循环遍历获取查询到的数据。 - **返回值**：影响的条数。 - **异常**：操作失败抛出 `SchemalessError` 异常。 -#### 参数绑定 STMT2 +#### 参数绑定 - `def statement2(self, sql=None, option=None)` - **接口说明**：使用连接对象创建 stmt2 对象 - **参数说明** From b63787aa09622fbab86f7dfd9596a61c3e98bc09 Mon Sep 17 00:00:00 2001 From: wangmm0220 Date: Mon, 30 Dec 2024 17:23:11 +0800 Subject: [PATCH 26/73] fix:[TD-33396]add log for tmq --- source/dnode/vnode/src/tq/tq.c | 2 +- source/dnode/vnode/src/tq/tqRead.c | 107 +++---- source/dnode/vnode/src/tq/tqScan.c | 438 ++++++++++++++--------------- source/dnode/vnode/src/tq/tqUtil.c | 178 +++++------- 4 files changed, 346 insertions(+), 379 deletions(-) diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index 73052c1e5e..3bfc50fcb2 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -207,7 +207,7 @@ int32_t tqSendDataRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* (void)tFormatOffset(buf1, TSDB_OFFSET_LEN, &(pRsp->reqOffset)); (void)tFormatOffset(buf2, TSDB_OFFSET_LEN, &(pRsp->rspOffset)); - tqDebug("tmq poll vgId:%d consumer:0x%" PRIx64 " (epoch %d) send rsp, block num:%d, req:%s, rsp:%s,QID:0x%" PRIx64, + tqDebug("tmq poll vgId:%d consumer:0x%" PRIx64 " (epoch %d) start to send rsp, block num:%d, req:%s, rsp:%s,QID:0x%" PRIx64, vgId, pReq->consumerId, pReq->epoch, pRsp->blockNum, buf1, buf2, pReq->reqId); return tqDoSendDataRsp(&pMsg->info, pRsp, pReq->epoch, pReq->consumerId, type, sver, ever); diff --git a/source/dnode/vnode/src/tq/tqRead.c b/source/dnode/vnode/src/tq/tqRead.c index a2b6194375..61b59e4cd0 100644 --- a/source/dnode/vnode/src/tq/tqRead.c +++ b/source/dnode/vnode/src/tq/tqRead.c @@ -197,7 +197,9 @@ bool isValValidForTable(STqHandle* pHandle, SWalCont* pHead) { end: tDecoderClear(&dcoder); - return tbSuid == realTbSuid; + bool tmp = tbSuid == realTbSuid; + tqDebug("%s suid:%"PRId64" realSuid:%"PRId64" return:%d", __FUNCTION__, tbSuid, realTbSuid, tmp); + return tmp; } int32_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, uint64_t reqId) { @@ -262,6 +264,8 @@ int32_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, uint64_t END: *fetchOffset = offset; + tqDebug("vgId:%d, end to fetch wal, code:%d , index:%" PRId64 ", last:%" PRId64 " commit:%" PRId64 ", applied:%" PRId64 ", 0x%" PRIx64, + vgId, code, offset, lastVer, committedVer, appliedVer, id); return code; } @@ -273,6 +277,8 @@ bool tqGetTablePrimaryKey(STqReader* pReader) { } void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) { + tqDebug("%s:%p uid:%"PRId64, __FUNCTION__ , pReader, uid); + if (pReader == NULL) { return; } @@ -286,6 +292,7 @@ void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) { } STqReader* tqReaderOpen(SVnode* pVnode) { + tqDebug("%s:%p", __FUNCTION__ , pVnode); if (pVnode == NULL) { return NULL; } @@ -317,6 +324,7 @@ STqReader* tqReaderOpen(SVnode* pVnode) { } void tqReaderClose(STqReader* pReader) { + tqDebug("%s:%p", __FUNCTION__ , pReader); if (pReader == NULL) return; // close wal reader @@ -485,7 +493,7 @@ bool tqNextBlockInWal(STqReader* pReader, const char* id, int sourceExcluded) { } int32_t tqReaderSetSubmitMsg(STqReader* pReader, void* msgStr, int32_t msgLen, int64_t ver) { -if (pReader == NULL) { + if (pReader == NULL) { return TSDB_CODE_INVALID_PARA; } pReader->msg.msgStr = msgStr; @@ -497,14 +505,13 @@ if (pReader == NULL) { tDecoderInit(&decoder, pReader->msg.msgStr, pReader->msg.msgLen); int32_t code = tDecodeSubmitReq(&decoder, &pReader->submit); + tDecoderClear(&decoder); + if (code != 0) { - tDecoderClear(&decoder); tqError("DecodeSSubmitReq2 error, msgLen:%d, ver:%" PRId64, msgLen, ver); - return code; } - tDecoderClear(&decoder); - return 0; + return code; } SWalReader* tqGetWalReader(STqReader* pReader) { @@ -529,32 +536,25 @@ int64_t tqGetResultBlockTime(STqReader* pReader) { } bool tqNextBlockImpl(STqReader* pReader, const char* idstr) { - if (pReader == NULL || pReader->msg.msgStr == NULL) { - return false; - } + int32_t code = false; + int32_t lino = 0; + int64_t uid = 0; + + TSDB_CHECK_NULL(pReader, code, lino, END, false); + TSDB_CHECK_NULL(pReader->msg.msgStr, code, lino, END, false); + TSDB_CHECK_NULL(pReader->tbIdHash, code, lino, END, true); int32_t numOfBlocks = taosArrayGetSize(pReader->submit.aSubmitTbData); while (pReader->nextBlk < numOfBlocks) { - tqDebug("try next data block, len:%d ver:%" PRId64 " index:%d/%d, %s", pReader->msg.msgLen, pReader->msg.ver, - (pReader->nextBlk + 1), numOfBlocks, idstr); + tqDebug("try next data block, len:%d ver:%" PRId64 " index:%d/%d, %s", pReader->msg.msgLen, pReader->msg.ver, (pReader->nextBlk + 1), numOfBlocks, idstr); SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); - if (pSubmitTbData == NULL) { - return false; - } - if (pReader->tbIdHash == NULL) { - return true; - } - + TSDB_CHECK_NULL(pSubmitTbData, code, lino, END, false); + uid = pSubmitTbData->uid; void* ret = taosHashGet(pReader->tbIdHash, &pSubmitTbData->uid, sizeof(int64_t)); - if (ret != NULL) { - tqDebug("block found, ver:%" PRId64 ", uid:%" PRId64 ", %s", pReader->msg.ver, pSubmitTbData->uid, idstr); - return true; - } else { - tqDebug("discard submit block, uid:%" PRId64 ", total queried tables:%d continue %s", pSubmitTbData->uid, - taosHashGetSize(pReader->tbIdHash), idstr); - } + TSDB_CHECK_CONDITION(ret == NULL, code, lino, END, true); + tqDebug("discard submit block, uid:%" PRId64 ", total queried tables:%d continue %s", pSubmitTbData->uid, taosHashGetSize(pReader->tbIdHash), idstr); pReader->nextBlk++; } @@ -562,30 +562,40 @@ bool tqNextBlockImpl(STqReader* pReader, const char* idstr) { pReader->nextBlk = 0; pReader->msg.msgStr = NULL; - return false; +END: + tqDebug("%s:%d, uid:%"PRId64",code:%d", __FUNCTION__, lino, uid, code); + return code; } bool tqNextDataBlockFilterOut(STqReader* pReader, SHashObj* filterOutUids) { - if (pReader == NULL || pReader->msg.msgStr == NULL) return false; + int32_t code = false; + int32_t lino = 0; + int64_t uid = 0; + + TSDB_CHECK_NULL(pReader, code, lino, END, false); + TSDB_CHECK_NULL(pReader->msg.msgStr, code, lino, END, false); + TSDB_CHECK_NULL(filterOutUids, code, lino, END, true); int32_t blockSz = taosArrayGetSize(pReader->submit.aSubmitTbData); while (pReader->nextBlk < blockSz) { SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); - if (pSubmitTbData == NULL) return false; - if (filterOutUids == NULL) return true; - + TSDB_CHECK_NULL(pSubmitTbData, code, lino, END, false); + uid = pSubmitTbData->uid; void* ret = taosHashGet(filterOutUids, &pSubmitTbData->uid, sizeof(int64_t)); - if (ret == NULL) { - return true; - } + TSDB_CHECK_NULL(ret, code, lino, END, true); + pReader->nextBlk++; + tqDebug("discard submit block, uid:%" PRId64 ", continue", pSubmitTbData->uid); } tDestroySubmitReq(&pReader->submit, TSDB_MSG_FLG_DECODE); pReader->nextBlk = 0; pReader->msg.msgStr = NULL; + tqDebug("all data blocks are filtered out"); - return false; +END: + tqDebug("%s:%d, uid:%"PRId64",code:%d", __FUNCTION__, lino, uid, code); + return code; } int32_t tqMaskBlock(SSchemaWrapper* pDst, SSDataBlock* pBlock, const SSchemaWrapper* pSrc, char* mask) { @@ -709,7 +719,7 @@ int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* if (pReader == NULL || pRes == NULL) { return TSDB_CODE_INVALID_PARA; } - tqTrace("tq reader retrieve data block %p, index:%d", pReader->msg.msgStr, pReader->nextBlk); + tqDebug("tq reader retrieve data block %p, index:%d", pReader->msg.msgStr, pReader->nextBlk); int32_t code = 0; int32_t line = 0; STSchema* pTSchema = NULL; @@ -876,10 +886,6 @@ END: static int32_t processBuildNew(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas, SSchemaWrapper* pSchemaWrapper, char* assigned, int32_t numOfRows, int32_t curRow, int32_t* lastRow) { - if (pReader == NULL || pSubmitTbData == NULL || blocks == NULL || schemas == NULL || pSchemaWrapper == NULL || - assigned == NULL || lastRow == NULL) { - return TSDB_CODE_INVALID_PARA; - } int32_t code = 0; SSchemaWrapper* pSW = NULL; SSDataBlock* block = NULL; @@ -909,15 +915,15 @@ static int32_t processBuildNew(STqReader* pReader, SSubmitTbData* pSubmitTbData, taosMemoryFreeClear(block); END: + if (code != 0) { + tqError("processBuildNew failed, code:%d", code); + } tDeleteSchemaWrapper(pSW); blockDataFreeRes(block); taosMemoryFree(block); return code; } static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas) { - if (pReader == NULL || pSubmitTbData == NULL || blocks == NULL || schemas == NULL) { - return TSDB_CODE_INVALID_PARA; - } int32_t code = 0; int32_t curRow = 0; int32_t lastRow = 0; @@ -931,6 +937,7 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData TQ_NULL_GO_TO_END(pCol); int32_t numOfRows = pCol->nVal; int32_t numOfCols = taosArrayGetSize(pCols); + tqDebug("vgId:%d, tqProcessColData, col num: %d, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols, numOfRows); for (int32_t i = 0; i < numOfRows; i++) { bool buildNew = false; @@ -972,14 +979,14 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData pLastBlock->info.rows = curRow - lastRow; END: + if (code != TSDB_CODE_SUCCESS) { + tqError("vgId:%d, process col data failed, code:%d", pReader->pWalReader->pWal->cfg.vgId, code); + } taosMemoryFree(assigned); return code; } int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas) { - if (pReader == NULL || pSubmitTbData == NULL || blocks == NULL || schemas == NULL) { - return TSDB_CODE_INVALID_PARA; - } int32_t code = 0; STSchema* pTSchema = NULL; @@ -992,6 +999,8 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra SArray* pRows = pSubmitTbData->aRowP; int32_t numOfRows = taosArrayGetSize(pRows); pTSchema = tBuildTSchema(pSchemaWrapper->pSchema, pSchemaWrapper->nCols, pSchemaWrapper->version); + TQ_NULL_GO_TO_END(pTSchema); + tqDebug("vgId:%d, tqProcessRowData, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfRows); for (int32_t i = 0; i < numOfRows; i++) { bool buildNew = false; @@ -1031,16 +1040,16 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra pLastBlock->info.rows = curRow - lastRow; END: + if (code != TSDB_CODE_SUCCESS) { + tqError("vgId:%d, process row data failed, code:%d", pReader->pWalReader->pWal->cfg.vgId, code); + } taosMemoryFreeClear(pTSchema); taosMemoryFree(assigned); return code; } int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas, SSubmitTbData** pSubmitTbDataRet, int64_t *createTime) { - if (pReader == NULL || blocks == NULL || schemas == NULL) { - return TSDB_CODE_INVALID_PARA; - } - tqTrace("tq reader retrieve data block %p, %d", pReader->msg.msgStr, pReader->nextBlk); + tqDebug("tq reader retrieve data block %p, %d", pReader->msg.msgStr, pReader->nextBlk); SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); if (pSubmitTbData == NULL) { return terrno; diff --git a/source/dnode/vnode/src/tq/tqScan.c b/source/dnode/vnode/src/tq/tqScan.c index 3419cd0020..ec200ba144 100644 --- a/source/dnode/vnode/src/tq/tqScan.c +++ b/source/dnode/vnode/src/tq/tqScan.c @@ -16,15 +16,16 @@ #include "tq.h" int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp, int32_t numOfCols, int8_t precision) { - if (pBlock == NULL || pRsp == NULL) { - return TSDB_CODE_INVALID_PARA; - } + int32_t code = TDB_CODE_SUCCESS; + int32_t lino = 0; + void* buf = NULL; + TSDB_CHECK_NULL(pBlock, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); + size_t dataEncodeBufSize = blockGetEncodeSize(pBlock); int32_t dataStrLen = sizeof(SRetrieveTableRspForTmq) + dataEncodeBufSize; - void* buf = taosMemoryCalloc(1, dataStrLen); - if (buf == NULL) { - return terrno; - } + buf = taosMemoryCalloc(1, dataStrLen); + TSDB_CHECK_NULL(buf, code, lino, END, terrno); SRetrieveTableRspForTmq* pRetrieve = (SRetrieveTableRspForTmq*)buf; pRetrieve->version = 1; @@ -33,49 +34,49 @@ int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp, int32_t pRetrieve->numOfRows = htobe64((int64_t)pBlock->info.rows); int32_t actualLen = blockEncode(pBlock, pRetrieve->data, dataEncodeBufSize, numOfCols); - if(actualLen < 0){ - taosMemoryFree(buf); - return terrno; - } + TSDB_CHECK_CONDITION(actualLen >= 0, code, lino, END, terrno); actualLen += sizeof(SRetrieveTableRspForTmq); - if (taosArrayPush(pRsp->blockDataLen, &actualLen) == NULL){ - taosMemoryFree(buf); - return terrno; - } - if (taosArrayPush(pRsp->blockData, &buf) == NULL) { - taosMemoryFree(buf); - return terrno; - } + TSDB_CHECK_NULL(taosArrayPush(pRsp->blockDataLen, &actualLen), code, lino, END, terrno); + TSDB_CHECK_NULL(taosArrayPush(pRsp->blockData, &buf), code, lino, END, terrno); - return TSDB_CODE_SUCCESS; + tqDebug("add block data to response success:%p, blockDataLen:%d, blockData:%p", pRsp->blockDataLen, actualLen, pRsp->blockData); +END: + if (code != TSDB_CODE_SUCCESS) { + taosMemoryFree(buf); + tqError("%s failed at %d, failed to add block data to response:%s", __FUNCTION__, lino, tstrerror(code)); + } + return code; } static int32_t tqAddTbNameToRsp(const STQ* pTq, int64_t uid, SMqDataRsp* pRsp, int32_t n) { - if (pRsp == NULL || pTq == NULL) { - return TSDB_CODE_INVALID_PARA; - } + int32_t code = TDB_CODE_SUCCESS; + int32_t lino = 0; SMetaReader mr = {0}; + + TSDB_CHECK_NULL(pTq, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); + metaReaderDoInit(&mr, pTq->pVnode->pMeta, META_READER_LOCK); - int32_t code = metaReaderGetTableEntryByUidCache(&mr, uid); - if (code < 0) { - metaReaderClear(&mr); - return code; - } + code = metaReaderGetTableEntryByUidCache(&mr, uid); + TSDB_CHECK_CODE(code, lino, END); for (int32_t i = 0; i < n; i++) { char* tbName = taosStrdup(mr.me.name); - if (tbName == NULL) { - metaReaderClear(&mr); - return terrno; - } + TSDB_CHECK_NULL(tbName, code, lino, END, terrno); if(taosArrayPush(pRsp->blockTbName, &tbName) == NULL){ - tqError("failed to push tbName to blockTbName:%s", tbName); + tqError("failed to push tbName to blockTbName:%s, uid:%"PRId64, tbName, uid); continue; } + tqDebug("add tbName to response success tbname:%s, uid:%"PRId64, tbName, uid); + } + +END: + if (code != TSDB_CODE_SUCCESS) { + tqError("%s failed at %d, failed to add tbName to response:%s, uid:%"PRId64, __FUNCTION__, lino, tstrerror(code), uid); } metaReaderClear(&mr); - return 0; + return code; } int32_t getDataBlock(qTaskInfo_t task, const STqHandle* pHandle, int32_t vgId, SSDataBlock** res) { @@ -96,109 +97,130 @@ int32_t getDataBlock(qTaskInfo_t task, const STqHandle* pHandle, int32_t vgId, S return 0; } -int32_t tqScanData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, STqOffsetVal* pOffset, const SMqPollReq* pRequest) { - if (pTq == NULL || pHandle == NULL || pRsp == NULL || pOffset == NULL || pRequest == NULL){ - return TSDB_CODE_INVALID_PARA; - } - int32_t vgId = TD_VID(pTq->pVnode); +static int32_t tqProcessReplayRsp(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, const SMqPollReq* pRequest, SSDataBlock* pDataBlock, qTaskInfo_t task){ int32_t code = 0; - int32_t line = 0; + int32_t lino = 0; + + if (IS_OFFSET_RESET_TYPE(pRequest->reqOffset.type) && pHandle->block != NULL) { + blockDataDestroy(pHandle->block); + pHandle->block = NULL; + } + if (pHandle->block == NULL) { + if (pDataBlock == NULL) { + goto END; + } + + STqOffsetVal offset = {0}; + code = qStreamExtractOffset(task, &offset); + TSDB_CHECK_CODE(code, lino, END); + + pHandle->block = NULL; + + code = createOneDataBlock(pDataBlock, true, &pHandle->block); + TSDB_CHECK_CODE(code, lino, END); + + pHandle->blockTime = offset.ts; + tOffsetDestroy(&offset); + int32_t vgId = TD_VID(pTq->pVnode); + code = getDataBlock(task, pHandle, vgId, &pDataBlock); + TSDB_CHECK_CODE(code, lino, END); + } + + const STqExecHandle* pExec = &pHandle->execHandle; + code = tqAddBlockDataToRsp(pHandle->block, pRsp, pExec->numOfCols, pTq->pVnode->config.tsdbCfg.precision); + TSDB_CHECK_CODE(code, lino, END); + + pRsp->blockNum++; + if (pDataBlock == NULL) { + blockDataDestroy(pHandle->block); + pHandle->block = NULL; + } else { + code = copyDataBlock(pHandle->block, pDataBlock); + TSDB_CHECK_CODE(code, lino, END); + + STqOffsetVal offset = {0}; + code = qStreamExtractOffset(task, &offset); + TSDB_CHECK_CODE(code, lino, END); + + pRsp->sleepTime = offset.ts - pHandle->blockTime; + pHandle->blockTime = offset.ts; + tOffsetDestroy(&offset); + } + +END: + if (code != TSDB_CODE_SUCCESS) { + tqError("%s failed at %d, failed to process replay response:%s", __FUNCTION__, lino, tstrerror(code)); + } + return code; +} + +int32_t tqScanData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, STqOffsetVal* pOffset, const SMqPollReq* pRequest) { + int32_t code = 0; + int32_t lino = 0; + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pTq, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pHandle, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pOffset, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pRequest, code, lino, END, TSDB_CODE_INVALID_PARA); + + int32_t vgId = TD_VID(pTq->pVnode); int32_t totalRows = 0; const STqExecHandle* pExec = &pHandle->execHandle; qTaskInfo_t task = pExec->task; code = qStreamPrepareScan(task, pOffset, pHandle->execHandle.subType); - TSDB_CHECK_CODE(code, line, END); + TSDB_CHECK_CODE(code, lino, END); qStreamSetSourceExcluded(task, pRequest->sourceExcluded); uint64_t st = taosGetTimestampMs(); while (1) { SSDataBlock* pDataBlock = NULL; code = getDataBlock(task, pHandle, vgId, &pDataBlock); - TSDB_CHECK_CODE(code, line, END); + TSDB_CHECK_CODE(code, lino, END); if (pRequest->enableReplay) { - if (IS_OFFSET_RESET_TYPE(pRequest->reqOffset.type) && pHandle->block != NULL) { - blockDataDestroy(pHandle->block); - pHandle->block = NULL; - } - if (pHandle->block == NULL) { - if (pDataBlock == NULL) { - break; - } - - STqOffsetVal offset = {0}; - code = qStreamExtractOffset(task, &offset); - TSDB_CHECK_CODE(code, line, END); - - pHandle->block = NULL; - - code = createOneDataBlock(pDataBlock, true, &pHandle->block); - TSDB_CHECK_CODE(code, line, END); - - pHandle->blockTime = offset.ts; - tOffsetDestroy(&offset); - code = getDataBlock(task, pHandle, vgId, &pDataBlock); - TSDB_CHECK_CODE(code, line, END); - } - - code = tqAddBlockDataToRsp(pHandle->block, pRsp, pExec->numOfCols, pTq->pVnode->config.tsdbCfg.precision); - TSDB_CHECK_CODE(code, line, END); - - pRsp->blockNum++; - if (pDataBlock == NULL) { - blockDataDestroy(pHandle->block); - pHandle->block = NULL; - } else { - code = copyDataBlock(pHandle->block, pDataBlock); - TSDB_CHECK_CODE(code, line, END); - - STqOffsetVal offset = {0}; - code = qStreamExtractOffset(task, &offset); - TSDB_CHECK_CODE(code, line, END); - - pRsp->sleepTime = offset.ts - pHandle->blockTime; - pHandle->blockTime = offset.ts; - tOffsetDestroy(&offset); - } + code = tqProcessReplayRsp(pTq, pHandle, pRsp, pRequest, pDataBlock, task); + TSDB_CHECK_CODE(code, lino, END); break; - } else { - if (pDataBlock == NULL) { - break; - } - code = tqAddBlockDataToRsp(pDataBlock, pRsp, pExec->numOfCols, pTq->pVnode->config.tsdbCfg.precision); - TSDB_CHECK_CODE(code, line, END); + } + if (pDataBlock == NULL) { + break; + } + code = tqAddBlockDataToRsp(pDataBlock, pRsp, pExec->numOfCols, pTq->pVnode->config.tsdbCfg.precision); + TSDB_CHECK_CODE(code, lino, END); - pRsp->blockNum++; - totalRows += pDataBlock->info.rows; - if (totalRows >= tmqRowSize || (taosGetTimestampMs() - st > 1000)) { - break; - } + pRsp->blockNum++; + totalRows += pDataBlock->info.rows; + if (totalRows >= tmqRowSize || (taosGetTimestampMs() - st > 1000)) { + break; } } - tqDebug("consumer:0x%" PRIx64 " vgId:%d tmq task executed finished, total blocks:%d, totalRows:%d", - pHandle->consumerId, vgId, pRsp->blockNum, totalRows); + tqDebug("consumer:0x%" PRIx64 " vgId:%d tmq task executed finished, total blocks:%d, totalRows:%d", pHandle->consumerId, vgId, pRsp->blockNum, totalRows); code = qStreamExtractOffset(task, &pRsp->rspOffset); + END: if (code != 0) { - tqError("consumer:0x%" PRIx64 " vgId:%d tmq task executed error, line:%d code:%d", pHandle->consumerId, vgId, line, - code); + tqError("%s failed at %d, tmq task executed error msg:%s", __FUNCTION__, lino, tstrerror(code)); } return code; } int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, SMqBatchMetaRsp* pBatchMetaRsp, STqOffsetVal* pOffset) { - if (pTq == NULL || pHandle == NULL || pRsp == NULL || pBatchMetaRsp == NULL || pOffset == NULL) { - return TSDB_CODE_INVALID_PARA; - } + int32_t code = 0; + int32_t lino = 0; + char* tbName = NULL; + SSchemaWrapper* pSW = NULL; + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pTq, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pHandle, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pOffset, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pBatchMetaRsp, code, lino, END, TSDB_CODE_INVALID_PARA); const STqExecHandle* pExec = &pHandle->execHandle; qTaskInfo_t task = pExec->task; - int code = qStreamPrepareScan(task, pOffset, pHandle->execHandle.subType); - if (code != 0) { - return code; - } + code = qStreamPrepareScan(task, pOffset, pHandle->execHandle.subType); + TSDB_CHECK_CODE(code, lino, END); int32_t rowCnt = 0; while (1) { @@ -206,52 +228,37 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, SMqBat uint64_t ts = 0; tqDebug("tmqsnap task start to execute"); code = qExecTask(task, &pDataBlock, &ts); - if (code != 0) { - tqError("vgId:%d, task exec error since %s", pTq->pVnode->config.vgId, tstrerror(code)); - return code; - } - + TSDB_CHECK_CODE(code, lino, END); tqDebug("tmqsnap task execute end, get %p", pDataBlock); if (pDataBlock != NULL && pDataBlock->info.rows > 0) { if (pRsp->withTbName) { - char* tbName = taosStrdup(qExtractTbnameFromTask(task)); - if (tbName == NULL) { - tqError("vgId:%d, failed to add tbname to rsp msg, null", pTq->pVnode->config.vgId); - return terrno; - } - if (taosArrayPush(pRsp->blockTbName, &tbName) == NULL){ - tqError("vgId:%d, failed to add tbname to rsp msg", pTq->pVnode->config.vgId); - continue; - } + tbName = taosStrdup(qExtractTbnameFromTask(task)); + TSDB_CHECK_NULL(tbName, code, lino, END, terrno); + TSDB_CHECK_NULL(taosArrayPush(pRsp->blockTbName, &tbName), code, lino, END, terrno); + tbName = NULL; } if (pRsp->withSchema) { - SSchemaWrapper* pSW = tCloneSSchemaWrapper(qExtractSchemaFromTask(task)); - if(taosArrayPush(pRsp->blockSchema, &pSW) == NULL){ - tqError("vgId:%d, failed to add schema to rsp msg", pTq->pVnode->config.vgId); - continue; - } + pSW = tCloneSSchemaWrapper(qExtractSchemaFromTask(task)); + TSDB_CHECK_NULL(pSW, code, lino, END, terrno); + TSDB_CHECK_NULL(taosArrayPush(pRsp->blockSchema, &pSW), code, lino, END, terrno); + pSW = NULL; } - if (tqAddBlockDataToRsp(pDataBlock, pRsp, taosArrayGetSize(pDataBlock->pDataBlock), - pTq->pVnode->config.tsdbCfg.precision) != 0) { - tqError("vgId:%d, failed to add block to rsp msg", pTq->pVnode->config.vgId); - continue; - } + code = tqAddBlockDataToRsp(pDataBlock, pRsp, taosArrayGetSize(pDataBlock->pDataBlock), pTq->pVnode->config.tsdbCfg.precision); + TSDB_CHECK_CODE(code, lino, END); pRsp->blockNum++; rowCnt += pDataBlock->info.rows; - if (rowCnt <= tmqRowSize) continue; - + if (rowCnt <= tmqRowSize) { + continue; + } } // get meta SMqBatchMetaRsp* tmp = qStreamExtractMetaMsg(task); if (taosArrayGetSize(tmp->batchMetaReq) > 0) { code = qStreamExtractOffset(task, &tmp->rspOffset); - if (code) { - return code; - } - + TSDB_CHECK_CODE(code, lino, END); *pBatchMetaRsp = *tmp; tqDebug("tmqsnap task get meta"); break; @@ -259,16 +266,13 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, SMqBat if (pDataBlock == NULL) { code = qStreamExtractOffset(task, pOffset); - if (code) { - break; - } + TSDB_CHECK_CODE(code, lino, END); if (pOffset->type == TMQ_OFFSET__SNAPSHOT_DATA) { continue; } - tqDebug("tmqsnap vgId: %d, tsdb consume over, switch to wal, ver %" PRId64, TD_VID(pTq->pVnode), - pHandle->snapshotVer + 1); + tqDebug("tmqsnap vgId: %d, tsdb consume over, switch to wal, ver %" PRId64, TD_VID(pTq->pVnode), pHandle->snapshotVer + 1); code = qStreamExtractOffset(task, &pRsp->rspOffset); break; } @@ -280,119 +284,93 @@ int32_t tqScanTaosx(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, SMqBat } } + tqDebug("%s:%d success", __FUNCTION__, lino); +END: + if (code != 0){ + tqDebug("%s:%d failed, code:%s", __FUNCTION__, lino, tstrerror(code) ); + } + taosMemoryFree(pSW); + taosMemoryFree(tbName); return code; } static int32_t buildCreateTbInfo(SMqDataRsp* pRsp, SVCreateTbReq* pCreateTbReq){ - if (pRsp == NULL || pCreateTbReq == NULL) { - return TSDB_CODE_INVALID_PARA; - } int32_t code = 0; + int32_t lino = 0; void* createReq = NULL; + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pCreateTbReq, code, lino, END, TSDB_CODE_INVALID_PARA); + if (pRsp->createTableNum == 0) { pRsp->createTableLen = taosArrayInit(0, sizeof(int32_t)); - if (pRsp->createTableLen == NULL) { - code = terrno; - goto END; - } + TSDB_CHECK_NULL(pRsp->createTableLen, code, lino, END, terrno); pRsp->createTableReq = taosArrayInit(0, sizeof(void*)); - if (pRsp->createTableReq == NULL) { - code = terrno; - goto END; - } + TSDB_CHECK_NULL(pRsp->createTableReq, code, lino, END, terrno); } uint32_t len = 0; tEncodeSize(tEncodeSVCreateTbReq, pCreateTbReq, len, code); - if (TSDB_CODE_SUCCESS != code) { - goto END; - } + TSDB_CHECK_CODE(code, lino, END); createReq = taosMemoryCalloc(1, len); - if (createReq == NULL){ - code = terrno; - goto END; - } + TSDB_CHECK_NULL(createReq, code, lino, END, terrno); + SEncoder encoder = {0}; tEncoderInit(&encoder, createReq, len); code = tEncodeSVCreateTbReq(&encoder, pCreateTbReq); tEncoderClear(&encoder); - if (code < 0) { - goto END; - } - if (taosArrayPush(pRsp->createTableLen, &len) == NULL){ - code = terrno; - goto END; - } - if (taosArrayPush(pRsp->createTableReq, &createReq) == NULL){ - code = terrno; - goto END; - } + TSDB_CHECK_CODE(code, lino, END); + TSDB_CHECK_NULL(taosArrayPush(pRsp->createTableLen, &len), code, lino, END, terrno); + TSDB_CHECK_NULL(taosArrayPush(pRsp->createTableReq, &createReq), code, lino, END, terrno); pRsp->createTableNum++; + tqDebug("build create table info msg success"); - return 0; END: - taosMemoryFree(createReq); + if (code != 0){ + tqError("%s failed at %d, failed to build create table info msg:%s", __FUNCTION__, lino, tstrerror(code)); + taosMemoryFree(createReq); + } return code; } -static void tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded){ - if (pTq == NULL || pHandle == NULL || pRsp == NULL || totalRows == NULL) { - return; - } +static int32_t tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded){ int32_t code = 0; - STqExecHandle* pExec = &pHandle->execHandle; - STqReader* pReader = pExec->pTqReader; + int32_t lino = 0; SArray* pBlocks = NULL; SArray* pSchemas = NULL; + + STqExecHandle* pExec = &pHandle->execHandle; + STqReader* pReader = pExec->pTqReader; + pBlocks = taosArrayInit(0, sizeof(SSDataBlock)); - if (pBlocks == NULL) { - code = terrno; - goto END; - } + TSDB_CHECK_NULL(pBlocks, code, lino, END, terrno); pSchemas = taosArrayInit(0, sizeof(void*)); - if(pSchemas == NULL){ - code = terrno; - goto END; - } + TSDB_CHECK_NULL(pSchemas, code, lino, END, terrno); SSubmitTbData* pSubmitTbDataRet = NULL; int64_t createTime = INT64_MAX; code = tqRetrieveTaosxBlock(pReader, pBlocks, pSchemas, &pSubmitTbDataRet, &createTime); - if (code != 0) { - tqError("vgId:%d, failed to retrieve block", pTq->pVnode->config.vgId); - goto END; - } - - if ((pSubmitTbDataRet->flags & sourceExcluded) != 0) { - goto END; - } + TSDB_CHECK_CODE(code, lino, END); + bool tmp = (pSubmitTbDataRet->flags & sourceExcluded) != 0; + TSDB_CHECK_CONDITION(!tmp, code, lino, END, TSDB_CODE_SUCCESS); if (pRsp->withTbName) { int64_t uid = pExec->pTqReader->lastBlkUid; code = tqAddTbNameToRsp(pTq, uid, pRsp, taosArrayGetSize(pBlocks)); - if (code != 0) { - tqError("vgId:%d, failed to add tbname to rsp msg", pTq->pVnode->config.vgId); - goto END; - } + TSDB_CHECK_CODE(code, lino, END); } if (pHandle->fetchMeta != WITH_DATA && pSubmitTbDataRet->pCreateTbReq != NULL) { if (pSubmitTbDataRet->ctimeMs - createTime <= 1000) { // judge if table is already created to avoid sending crateTbReq code = buildCreateTbInfo(pRsp, pSubmitTbDataRet->pCreateTbReq); - if (code != 0){ - tqError("vgId:%d, failed to build create table info", pTq->pVnode->config.vgId); - goto END; - } + TSDB_CHECK_CODE(code, lino, END); } } - if (pHandle->fetchMeta == ONLY_META && pSubmitTbDataRet->pCreateTbReq == NULL) { - goto END; - } + tmp = (pHandle->fetchMeta == ONLY_META && pSubmitTbDataRet->pCreateTbReq == NULL); + TSDB_CHECK_CONDITION(!tmp, code, lino, END, TSDB_CODE_SUCCESS); for (int32_t i = 0; i < taosArrayGetSize(pBlocks); i++) { SSDataBlock* pBlock = taosArrayGet(pBlocks, i); if (pBlock == NULL) { continue; } - if (tqAddBlockDataToRsp(pBlock, pRsp, taosArrayGetSize(pBlock->pDataBlock), - pTq->pVnode->config.tsdbCfg.precision) != 0){ + if (tqAddBlockDataToRsp(pBlock, pRsp, taosArrayGetSize(pBlock->pDataBlock), pTq->pVnode->config.tsdbCfg.precision) != 0){ tqError("vgId:%d, failed to add block to rsp msg", pTq->pVnode->config.vgId); continue; } @@ -405,38 +383,46 @@ static void tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int } pRsp->blockNum++; } - - taosArrayDestroy(pBlocks); - taosArrayDestroy(pSchemas); - return; - + tqDebug("vgId:%d, process sub data success", pTq->pVnode->config.vgId); END: - taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes); - taosArrayDestroyP(pSchemas, (FDelete)tDeleteSchemaWrapper); + if (code != 0){ + tqError("%s failed at %d, failed to process sub data:%s", __FUNCTION__, lino, tstrerror(code)); + taosArrayDestroyEx(pBlocks, (FDelete)blockDataFreeRes); + taosArrayDestroyP(pSchemas, (FDelete)tDeleteSchemaWrapper); + } else { + taosArrayDestroy(pBlocks); + taosArrayDestroy(pSchemas); + } + return code; } -int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SPackedData submit, SMqDataRsp* pRsp, int32_t* totalRows, - int8_t sourceExcluded) { - if (pTq == NULL || pHandle == NULL || pRsp == NULL || totalRows == NULL) { - return TSDB_CODE_INVALID_PARA; - } +int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SPackedData submit, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded) { + int32_t code = 0; + int32_t lino = 0; + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pTq, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(pHandle, code, lino, END, TSDB_CODE_INVALID_PARA); + TSDB_CHECK_NULL(totalRows, code, lino, END, TSDB_CODE_INVALID_PARA); STqExecHandle* pExec = &pHandle->execHandle; - int32_t code = 0; STqReader* pReader = pExec->pTqReader; code = tqReaderSetSubmitMsg(pReader, submit.msgStr, submit.msgLen, submit.ver); - if (code != 0) { - return code; - } + TSDB_CHECK_CODE(code, lino, END); if (pExec->subType == TOPIC_SUB_TYPE__TABLE) { while (tqNextBlockImpl(pReader, NULL)) { - tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); + code = tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); + TSDB_CHECK_CODE(code, lino, END); } } else if (pExec->subType == TOPIC_SUB_TYPE__DB) { while (tqNextDataBlockFilterOut(pReader, pExec->execDb.pFilterOutTbUid)) { - tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); + code = tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); + TSDB_CHECK_CODE(code, lino, END); } } +END: + if (code != 0){ + tqError("%s failed at %d, failed to scan log:%s", __FUNCTION__, lino, tstrerror(code)); + } return code; } diff --git a/source/dnode/vnode/src/tq/tqUtil.c b/source/dnode/vnode/src/tq/tqUtil.c index f6a8563c70..2c27f7e515 100644 --- a/source/dnode/vnode/src/tq/tqUtil.c +++ b/source/dnode/vnode/src/tq/tqUtil.c @@ -21,21 +21,27 @@ static int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, c const SMqBatchMetaRsp* pRsp, int32_t vgId); int32_t tqInitDataRsp(SMqDataRsp* pRsp, STqOffsetVal pOffset) { - if (pRsp == NULL) { - return TSDB_CODE_INVALID_PARA; - } - pRsp->blockData = taosArrayInit(0, sizeof(void*)); - pRsp->blockDataLen = taosArrayInit(0, sizeof(int32_t)); + int32_t code = TDB_CODE_SUCCESS; + int32_t lino = 0; + tqDebug("%s called", __FUNCTION__ ); + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); - if (pRsp->blockData == NULL || pRsp->blockDataLen == NULL) { - return terrno; - } + pRsp->blockData = taosArrayInit(0, sizeof(void*)); + TSDB_CHECK_NULL(pRsp->blockData, code, lino, END, terrno); + + pRsp->blockDataLen = taosArrayInit(0, sizeof(int32_t)); + TSDB_CHECK_NULL(pRsp->blockDataLen, code, lino, END, terrno); tOffsetCopy(&pRsp->reqOffset, &pOffset); tOffsetCopy(&pRsp->rspOffset, &pOffset); pRsp->withTbName = 0; pRsp->withSchema = false; - return 0; + +END: + if (code != 0){ + tqError("%s failed at:%d, code:%s", __FUNCTION__ , lino, tstrerror(code)); + } + return code; } void tqUpdateNodeStage(STQ* pTq, bool isLeader) { @@ -44,45 +50,37 @@ void tqUpdateNodeStage(STQ* pTq, bool isLeader) { } static int32_t tqInitTaosxRsp(SMqDataRsp* pRsp, STqOffsetVal pOffset) { - if (pRsp == NULL) { - return TSDB_CODE_INVALID_PARA; - } + int32_t code = TDB_CODE_SUCCESS; + int32_t lino = 0; + tqDebug("%s called", __FUNCTION__ ); + TSDB_CHECK_NULL(pRsp, code, lino, END, TSDB_CODE_INVALID_PARA); tOffsetCopy(&pRsp->reqOffset, &pOffset); tOffsetCopy(&pRsp->rspOffset, &pOffset); pRsp->withTbName = 1; pRsp->withSchema = 1; pRsp->blockData = taosArrayInit(0, sizeof(void*)); + TSDB_CHECK_NULL(pRsp->blockData, code, lino, END, terrno);\ + pRsp->blockDataLen = taosArrayInit(0, sizeof(int32_t)); + TSDB_CHECK_NULL(pRsp->blockDataLen, code, lino, END, terrno); + pRsp->blockTbName = taosArrayInit(0, sizeof(void*)); + TSDB_CHECK_NULL(pRsp->blockTbName, code, lino, END, terrno); + pRsp->blockSchema = taosArrayInit(0, sizeof(void*)); + TSDB_CHECK_NULL(pRsp->blockSchema, code, lino, END, terrno); - if (pRsp->blockData == NULL || pRsp->blockDataLen == NULL || - pRsp->blockTbName == NULL || pRsp->blockSchema == NULL) { - if (pRsp->blockData != NULL) { - taosArrayDestroy(pRsp->blockData); - pRsp->blockData = NULL; - } - if (pRsp->blockDataLen != NULL) { - taosArrayDestroy(pRsp->blockDataLen); - pRsp->blockDataLen = NULL; - } - - if (pRsp->blockTbName != NULL) { - taosArrayDestroy(pRsp->blockTbName); - pRsp->blockTbName = NULL; - } - - if (pRsp->blockSchema != NULL) { - taosArrayDestroy(pRsp->blockSchema); - pRsp->blockSchema = NULL; - } - - return terrno; +END: + if (code != 0){ + tqError("%s failed at:%d, code:%s", __FUNCTION__ , lino, tstrerror(code)); + taosArrayDestroy(pRsp->blockData); + taosArrayDestroy(pRsp->blockDataLen); + taosArrayDestroy(pRsp->blockTbName); + taosArrayDestroy(pRsp->blockSchema); } - - return 0; + return code; } static int32_t extractResetOffsetVal(STqOffsetVal* pOffsetVal, STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, @@ -151,23 +149,19 @@ static int32_t extractResetOffsetVal(STqOffsetVal* pOffsetVal, STQ* pTq, STqHand static int32_t extractDataAndRspForNormalSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg, STqOffsetVal* pOffset) { - if (pTq == NULL || pHandle == NULL || pRequest == NULL || pMsg == NULL || pOffset == NULL) { - return TSDB_CODE_INVALID_PARA; - } + int32_t code = TDB_CODE_SUCCESS; + int32_t lino = 0; + tqDebug("%s called", __FUNCTION__ ); uint64_t consumerId = pRequest->consumerId; int32_t vgId = TD_VID(pTq->pVnode); terrno = 0; SMqDataRsp dataRsp = {0}; - int code = tqInitDataRsp(&dataRsp, *pOffset); - if (code != 0) { - goto end; - } + code = tqInitDataRsp(&dataRsp, *pOffset); + TSDB_CHECK_CODE(code, lino, end); code = qSetTaskId(pHandle->execHandle.task, consumerId, pRequest->reqId); - if (code != 0) { - goto end; - } + TSDB_CHECK_CODE(code, lino, end); code = tqScanData(pTq, pHandle, &dataRsp, pOffset, pRequest); if (code != 0 && terrno != TSDB_CODE_WAL_LOG_NOT_EXIST) { @@ -191,15 +185,21 @@ static int32_t extractDataAndRspForNormalSubscribe(STQ* pTq, STqHandle* pHandle, tOffsetCopy(&dataRsp.reqOffset, pOffset); code = tqSendDataRsp(pHandle, pMsg, pRequest, &dataRsp, TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); -end : { - char buf[TSDB_OFFSET_LEN] = {0}; - tFormatOffset(buf, TSDB_OFFSET_LEN, &dataRsp.rspOffset); - tqDebug("tmq poll: consumer:0x%" PRIx64 ", subkey %s, vgId:%d, rsp block:%d, rsp offset type:%s,QID:0x%" PRIx64 - " code:%d", - consumerId, pHandle->subKey, vgId, dataRsp.blockNum, buf, pRequest->reqId, code); - tDeleteMqDataRsp(&dataRsp); - return code; -} +end: + { + char buf[TSDB_OFFSET_LEN] = {0}; + tFormatOffset(buf, TSDB_OFFSET_LEN, &dataRsp.rspOffset); + if (code != 0){ + tqError("tmq poll: consumer:0x%" PRIx64 ", subkey %s, vgId:%d, rsp block:%d, rsp offset type:%s,QID:0x%" PRIx64 " error msg:%s, line:%d", + consumerId, pHandle->subKey, vgId, dataRsp.blockNum, buf, pRequest->reqId, tstrerror(code), lino); + } else { + tqDebug("tmq poll: consumer:0x%" PRIx64 ", subkey %s, vgId:%d, rsp block:%d, rsp offset type:%s,QID:0x%" PRIx64 " success", + consumerId, pHandle->subKey, vgId, dataRsp.blockNum, buf, pRequest->reqId); + } + + tDeleteMqDataRsp(&dataRsp); + return code; + } } #define PROCESS_EXCLUDED_MSG(TYPE, DECODE_FUNC, DELETE_FUNC) \ @@ -224,9 +224,6 @@ static void tDeleteCommon(void* parm) {} static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg, STqOffsetVal* offset) { - if (pTq == NULL || pHandle == NULL || pRequest == NULL || pMsg == NULL || offset == NULL) { - return TSDB_CODE_INVALID_PARA; - } int32_t vgId = TD_VID(pTq->pVnode); SMqDataRsp taosxRsp = {0}; SMqBatchMetaRsp btMetaRsp = {0}; @@ -238,17 +235,13 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, if (taosArrayGetSize(btMetaRsp.batchMetaReq) > 0) { code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId); - tqDebug("tmq poll: consumer:0x%" PRIx64 " subkey:%s vgId:%d, send meta offset type:%d,uid:%" PRId64 - ",ts:%" PRId64, - pRequest->consumerId, pHandle->subKey, vgId, btMetaRsp.rspOffset.type, btMetaRsp.rspOffset.uid, - btMetaRsp.rspOffset.ts); + tqDebug("tmq poll: consumer:0x%" PRIx64 " subkey:%s vgId:%d, send meta offset type:%d,uid:%" PRId64 ",ts:%" PRId64, + pRequest->consumerId, pHandle->subKey, vgId, btMetaRsp.rspOffset.type, btMetaRsp.rspOffset.uid,btMetaRsp.rspOffset.ts); goto END; } - tqDebug("taosx poll: consumer:0x%" PRIx64 " subkey:%s vgId:%d, send data blockNum:%d, offset type:%d,uid:%" PRId64 - ",ts:%" PRId64, - pRequest->consumerId, pHandle->subKey, vgId, taosxRsp.blockNum, taosxRsp.rspOffset.type, - taosxRsp.rspOffset.uid, taosxRsp.rspOffset.ts); + tqDebug("taosx poll: consumer:0x%" PRIx64 " subkey:%s vgId:%d, send data blockNum:%d, offset type:%d,uid:%" PRId64",ts:%" PRId64, + pRequest->consumerId, pHandle->subKey, vgId, taosxRsp.blockNum, taosxRsp.rspOffset.type, taosxRsp.rspOffset.uid, taosxRsp.rspOffset.ts); if (taosxRsp.blockNum > 0) { code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); goto END; @@ -277,16 +270,10 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, if (totalMetaRows > 0) { tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer); code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId); - if (totalRows != 0) { - tqError("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, totalRows error, vgId:%d offset %" PRId64, - pRequest->consumerId, pRequest->epoch, vgId, fetchVer); - code = code == 0 ? TSDB_CODE_TQ_INTERNAL_ERROR : code; - } goto END; } tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer); - code = tqSendDataRsp( - pHandle, pMsg, pRequest, &taosxRsp, + code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, taosxRsp.createTableNum > 0 ? TMQ_MSG_TYPE__POLL_DATA_META_RSP : TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); goto END; } @@ -299,8 +286,7 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, if (pHead->msgType != TDMT_VND_SUBMIT) { if (totalRows > 0) { tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer); - code = tqSendDataRsp( - pHandle, pMsg, pRequest, &taosxRsp, + code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, taosxRsp.createTableNum > 0 ? TMQ_MSG_TYPE__POLL_DATA_META_RSP : TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); goto END; } @@ -331,15 +317,9 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, if (!btMetaRsp.batchMetaReq) { btMetaRsp.batchMetaReq = taosArrayInit(4, POINTER_BYTES); - if (btMetaRsp.batchMetaReq == NULL) { - code = TAOS_GET_TERRNO(terrno); - goto END; - } + TQ_NULL_GO_TO_END(btMetaRsp.batchMetaReq); btMetaRsp.batchMetaLen = taosArrayInit(4, sizeof(int32_t)); - if (btMetaRsp.batchMetaLen == NULL) { - code = TAOS_GET_TERRNO(terrno); - goto END; - } + TQ_NULL_GO_TO_END(btMetaRsp.batchMetaLen); } fetchVer++; @@ -355,10 +335,7 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, } int32_t tLen = sizeof(SMqRspHead) + len; void* tBuf = taosMemoryCalloc(1, tLen); - if (tBuf == NULL) { - code = TAOS_GET_TERRNO(terrno); - goto END; - } + TQ_NULL_GO_TO_END(tBuf); void* metaBuff = POINTER_SHIFT(tBuf, sizeof(SMqRspHead)); SEncoder encoder = {0}; tEncoderInit(&encoder, metaBuff, len); @@ -369,14 +346,8 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); continue; } - if (taosArrayPush(btMetaRsp.batchMetaReq, &tBuf) == NULL) { - code = TAOS_GET_TERRNO(terrno); - goto END; - } - if (taosArrayPush(btMetaRsp.batchMetaLen, &tLen) == NULL) { - code = TAOS_GET_TERRNO(terrno); - goto END; - } + TQ_NULL_GO_TO_END (taosArrayPush(btMetaRsp.batchMetaReq, &tBuf)); + TQ_NULL_GO_TO_END (taosArrayPush(btMetaRsp.batchMetaLen, &tLen)); totalMetaRows++; if ((taosArrayGetSize(btMetaRsp.batchMetaReq) >= tmqRowSize) || (taosGetTimestampMs() - st > 1000)) { tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer); @@ -399,17 +370,11 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, .ver = pHead->version, }; - code = tqTaosxScanLog(pTq, pHandle, submit, &taosxRsp, &totalRows, pRequest->sourceExcluded); - if (code < 0) { - tqError("tmq poll: tqTaosxScanLog error %" PRId64 ", in vgId:%d, subkey %s", pRequest->consumerId, vgId, - pRequest->subKey); - goto END; - } + TQ_ERR_GO_TO_END(tqTaosxScanLog(pTq, pHandle, submit, &taosxRsp, &totalRows, pRequest->sourceExcluded)); if (totalRows >= tmqRowSize || (taosGetTimestampMs() - st > 1000)) { tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer + 1); - code = tqSendDataRsp( - pHandle, pMsg, pRequest, &taosxRsp, + code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, taosxRsp.createTableNum > 0 ? TMQ_MSG_TYPE__POLL_DATA_META_RSP : TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); goto END; } else { @@ -419,6 +384,10 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, } END: + if (code != 0){ + tqError("tmq poll: tqTaosxScanLog error %" PRId64 ", in vgId:%d, subkey %s", pRequest->consumerId, vgId, + pRequest->subKey); + } tDeleteMqBatchMetaRsp(&btMetaRsp); tDeleteSTaosxRsp(&taosxRsp); return code; @@ -457,6 +426,9 @@ int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequ } END: + if (code != 0){ + uError("failed to extract data for mq, code:%d", code); + } tOffsetDestroy(&reqOffset); return code; } From e8232129651cd56ae9a7ce4cf7a142b90c09741a Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 17:53:28 +0800 Subject: [PATCH 27/73] docs: update english documents --- .../en/14-reference/05-connector/30-python.md | 81 +++++++++---------- .../14-reference/05-connector/30-python.mdx | 2 +- 2 files changed, 37 insertions(+), 46 deletions(-) diff --git a/docs/en/14-reference/05-connector/30-python.md b/docs/en/14-reference/05-connector/30-python.md index 58cff75c90..d1e922da8d 100644 --- a/docs/en/14-reference/05-connector/30-python.md +++ b/docs/en/14-reference/05-connector/30-python.md @@ -55,6 +55,8 @@ Python Connector historical versions (it is recommended to use the latest versio |Python Connector Version | Major Changes | TDengine Version| | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | +|2.7.20 | Native supports STMT2 writing | - | +|2.7.19 | Support Apache Superset connection to TDengine Cloud data source | - | |2.7.18 | Support Apache SuperSet BI Tools. | - | |2.7.16 | Add subscription configuration (session. timeout. ms, Max. roll. interval. ms). | - | |2.7.15 | Added support for VARBINRY and GEOMETRY types. | - | @@ -136,7 +138,7 @@ TDengine currently supports timestamp, numeric, character, boolean types, and th | [tmq_consumer.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/tmq_consumer.py) | tmq subscription | | [native_all_type_query.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_query.py) | Example supporting all types | | [native_all_type_stmt.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_stmt.py) | Parameter binding example supporting all types | - +| [test_stmt2.py](https://github.com/taosdata/taos-connector-python/blob/main/tests/test_stmt2.py) | Example of STMT2 writing | Example program source code can be found at: 1. [More native example programs](https://github.com/taosdata/taos-connector-python/tree/main/examples) @@ -429,51 +431,40 @@ TaosResult object can be iterated over to retrieve queried data. - **Exceptions**: Throws `SchemalessError` if operation fails. #### Parameter Binding - -- `def statement(self, sql=None)` - - **Interface Description**: Creates a stmt object using the connection object, if sql is not empty it will call prepare. - - `sql`: Precompiled SQL statement. - - **Return Value**: stmt object. - - **Exception**: Throws `StatementError` exception on failure. +- `def statement2(self, sql=None, option=None)` + - **Interface Description**：Creating an STMT2 object using a connection object + - **Parameter Description** + - `sql`: The bound SQL statement will call the `prepare` function if it is not empty + - `option` Pass in TaoStmt2Option class instance options + - **Return Value**：stmt2 object + - **Exception**：Throws `ConnectionError` on failure - `def prepare(self, sql)` - - **Interface Description**: Binds a precompiled sql statement. - - **Parameter Description**: - - `sql`: Precompiled SQL statement. - - **Exception**: Throws `StatementError` exception on failure. -- `def set_tbname(self, name)` - - **Interface Description**: Sets the table name for data to be written to. - - **Parameter Description**: - - `name`: Table name, if you need to specify a database, for example: `db_name.table_name`. - - **Exception**: Throws `StatementError` exception on failure. -- `def set_tbname_tags(self, name, tags):` - - **Interface Description**: Sets the table and Tags data, used for automatic table creation. - - **Parameter Description**: - - `name`: Table name, if you need to specify a database, for example: `db_name.table_name`. - - `tags`: Tags data. - - **Exception**: Throws `StatementError` exception on failure. -- `def bind_param(self, params, add_batch=True)` - - **Interface Description**: Binds a set of data and submits. - - **Parameter Description**: - - `params`: Data to bind. - - `add_batch`: Whether to submit the bound data. - - **Exception**: Throws `StatementError` exception on failure. -- `def bind_param_batch(self, binds, add_batch=True)` - - **Interface Description**: Binds multiple sets of data and submits. - - **Parameter Description**: - - `binds`: Data to bind. - - `add_batch`: Whether to submit the bound data. - - **Exception**: Throws `StatementError` exception on failure. -- `def add_batch(self)` - - **Interface Description**: Submits the bound data. - - **Exception**: Throws `StatementError` exception on failure. -- `def execute(self)` - - **Interface Description**: Executes and writes all the bound data. - - **Exception**: Throws `StatementError` exception on failure. -- `def affected_rows(self)` - - **Interface Description**: Gets the number of rows written. - - **Return Value**: Number of rows written. -- `def close(&self)` - - **Interface Description**: Closes the stmt object. + - **Interface Description**：Bind a precompiled SQL statement + - **Parameter Description**： + - `sql`: Precompiled SQL statement + - **Exception**：Throws `StatementError` on failure +- `def bind_param(self, tbnames, tags, datas)` + - **Interface Description**：Binding Data as an Independent Array + - **Parameter Description**： + - `tbnames`:Bind table name array, data type is list + - `tags`: Bind tag column value array, data type is list + - `tags`: Bind a regular column value array with a data type of list + - **Exception**：Throws `StatementError` on failure +- `def bind_param_with_tables(self, tables)` + - **Interface Description**：Bind data in an independent table format. Independent tables are organized by table units, with table names, TAG values, and regular column numerical attributes in each table + - **Parameter Description**： + - `tables`: `BindTable` Independent Table Object Array + - **Exception**：Throws `StatementError` on failure +- `def execute(self) -> int:` + - **Interface Description**：Execute to write all bound data + - **Return Value**：Affects the number of rows + - **Exception**：Throws `QueryError` on failure +- `def result(self)` + - **Interface Description**：Get parameter binding query result set + - **Return Value**：Returns the TaosResult object +- `def close(self)` + - **Interface Description**： close the STMT2 object + #### Data Subscription diff --git a/docs/zh/14-reference/05-connector/30-python.mdx b/docs/zh/14-reference/05-connector/30-python.mdx index ad43ce19df..a1fedfd92c 100644 --- a/docs/zh/14-reference/05-connector/30-python.mdx +++ b/docs/zh/14-reference/05-connector/30-python.mdx @@ -134,7 +134,7 @@ TDengine 目前支持时间戳、数字、字符、布尔类型，与 Python 对 | [tmq_consumer.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/tmq_consumer.py) | tmq 订阅 | | [native_all_type_query.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_query.py) | 支持全部类型示例 | | [native_all_type_stmt.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/native_all_type_stmt.py) | 参数绑定 stmt 全部类型示例 | -| [insert_with_stmt2.py](https://github.com/taosdata/taos-connector-python/blob/main/tests/test_stmt2.py) | 参数绑定 stmt2 写入示例 | +| [test_stmt2.py](https://github.com/taosdata/taos-connector-python/blob/main/tests/test_stmt2.py) | 参数绑定 stmt2 写入示例 | 示例程序源码请参考： From eb42773328c7d25baf286e1f8dae50b9c0ff79dd Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Mon, 30 Dec 2024 18:08:47 +0800 Subject: [PATCH 28/73] chore: remove useless ci config --- .appveyor.yml | 49 -------- .circleci/config.yml | 13 --- .drone.yml | 266 ------------------------------------------- 3 files changed, 328 deletions(-) delete mode 100644 .appveyor.yml delete mode 100644 .circleci/config.yml delete mode 100644 .drone.yml diff --git a/.appveyor.yml b/.appveyor.yml deleted file mode 100644 index e7802b3d0d..0000000000 --- a/.appveyor.yml +++ /dev/null @@ -1,49 +0,0 @@ -version: 1.0.{build} -image: - - Visual Studio 2015 - - macos -environment: - matrix: - - ARCH: amd64 - - ARCH: x86 -matrix: - exclude: - - image: macos - ARCH: x86 -for: - - - matrix: - only: - - image: Visual Studio 2015 - clone_folder: c:\dev\TDengine - clone_depth: 1 - - init: - - call "C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\vcvarsall.bat" %ARCH% - - before_build: - - cd c:\dev\TDengine - - md build - - build_script: - - cd build - - cmake -G "NMake Makefiles" .. -DBUILD_JDBC=false - - nmake install - - - matrix: - only: - - image: macos - clone_depth: 1 - - build_script: - - mkdir debug - - cd debug - - cmake .. > /dev/null - - make > /dev/null -notifications: -- provider: Email - to: - - sangshuduo@gmail.com - on_build_success: true - on_build_failure: true - on_build_status_changed: true diff --git a/.circleci/config.yml b/.circleci/config.yml deleted file mode 100644 index 6f98693add..0000000000 --- a/.circleci/config.yml +++ /dev/null @@ -1,13 +0,0 @@ -# Use the latest 2.1 version of CircleCI pipeline process engine. See: https://circleci.com/docs/2.0/configuration-reference -version: 2.1 -# Use a package of configuration called an orb. -orbs: - # Declare a dependency on the welcome-orb - welcome: circleci/welcome-orb@0.4.1 -# Orchestrate or schedule a set of jobs -workflows: - # Name the workflow "welcome" - welcome: - # Run the welcome/run job in its own container - jobs: - - welcome/run diff --git a/.drone.yml b/.drone.yml deleted file mode 100644 index d35c104830..0000000000 --- a/.drone.yml +++ /dev/null @@ -1,266 +0,0 @@ ---- -kind: pipeline -name: test_amd64 - -platform: - os: linux - arch: amd64 - -steps: -- name: build - image: gcc - commands: - - apt-get update - - apt-get install -y cmake build-essential - - mkdir debug - - cd debug - - cmake .. - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: test_arm64_bionic - -platform: - os: linux - arch: arm64 -steps: -- name: build - image: arm64v8/ubuntu:bionic - commands: - - apt-get update - - apt-get install -y cmake build-essential - - mkdir debug - - cd debug - - cmake .. -DCPUTYPE=aarch64 > /dev/null - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: test_arm64_focal - -platform: - os: linux - arch: arm64 - -steps: -- name: build - image: arm64v8/ubuntu:focal - commands: - - echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections - - apt-get update - - apt-get install -y -qq cmake build-essential - - mkdir debug - - cd debug - - cmake .. -DCPUTYPE=aarch64 > /dev/null - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: test_arm64_centos7 - -platform: - os: linux - arch: arm64 - -steps: -- name: build - image: arm64v8/centos:7 - commands: - - yum install -y gcc gcc-c++ make cmake git - - mkdir debug - - cd debug - - cmake .. -DCPUTYPE=aarch64 > /dev/null - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: test_arm64_centos8 - -platform: - os: linux - arch: arm64 - -steps: -- name: build - image: arm64v8/centos:8 - commands: - - dnf install -y gcc gcc-c++ make cmake epel-release git libarchive - - mkdir debug - - cd debug - - cmake .. -DCPUTYPE=aarch64 > /dev/null - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: test_arm_bionic - -platform: - os: linux - arch: arm - -steps: -- name: build - image: arm32v7/ubuntu:bionic - commands: - - apt-get update - - apt-get install -y cmake build-essential - - mkdir debug - - cd debug - - cmake .. -DCPUTYPE=aarch32 > /dev/null - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: build_trusty - -platform: - os: linux - arch: amd64 - -steps: -- name: build - image: ubuntu:trusty - commands: - - apt-get update - - apt-get install -y gcc cmake3 build-essential git binutils-2.26 - - - mkdir debug - - cd debug - - cmake .. - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: build_xenial - -platform: - os: linux - arch: amd64 - -steps: -- name: build - image: ubuntu:xenial - commands: - - apt-get update - - apt-get install -y gcc cmake build-essential - - mkdir debug - - cd debug - - cmake .. - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: build_bionic -platform: - os: linux - arch: amd64 - -steps: -- name: build - image: ubuntu:bionic - commands: - - apt-get update - - apt-get install -y gcc cmake build-essential - - mkdir debug - - cd debug - - cmake .. - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 ---- -kind: pipeline -name: build_centos7 -platform: - os: linux - arch: amd64 - -steps: -- name: build - image: ansible/centos7-ansible - commands: - - yum install -y gcc gcc-c++ make cmake - - mkdir debug - - cd debug - - cmake .. - - make -j4 - trigger: - event: - - pull_request - when: - branch: - - develop - - master - - 2.0 - - 3.0 \ No newline at end of file From 3e55011e983ce2724ad0c3bf79fc91a7a6281f55 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 18:13:59 +0800 Subject: [PATCH 29/73] fix: add top level to 4 --- docs/en/14-reference/05-connector/30-python.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/en/14-reference/05-connector/30-python.md b/docs/en/14-reference/05-connector/30-python.md index d1e922da8d..05ae323f9a 100644 --- a/docs/en/14-reference/05-connector/30-python.md +++ b/docs/en/14-reference/05-connector/30-python.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: Python title: Python Client Library slug: /tdengine-reference/client-libraries/python From 9e701eaf2a47006835ac661849370e2cb3c1bcae Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 18:18:43 +0800 Subject: [PATCH 30/73] add toc_max_heading_level 4 --- docs/en/14-reference/05-connector/10-cpp.md | 1 + docs/en/14-reference/05-connector/14-java.md | 1 + docs/en/14-reference/05-connector/20-go.md | 1 + docs/en/14-reference/05-connector/26-rust.md | 1 + docs/en/14-reference/05-connector/35-node.md | 1 + docs/en/14-reference/05-connector/40-csharp.md | 1 + 6 files changed, 6 insertions(+) diff --git a/docs/en/14-reference/05-connector/10-cpp.md b/docs/en/14-reference/05-connector/10-cpp.md index fe7574d416..940d4c359e 100644 --- a/docs/en/14-reference/05-connector/10-cpp.md +++ b/docs/en/14-reference/05-connector/10-cpp.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: C/C++ title: C/C++ Client Library slug: /tdengine-reference/client-libraries/cpp diff --git a/docs/en/14-reference/05-connector/14-java.md b/docs/en/14-reference/05-connector/14-java.md index 48302b9d3b..2781f26c24 100644 --- a/docs/en/14-reference/05-connector/14-java.md +++ b/docs/en/14-reference/05-connector/14-java.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: Java title: Java Client Library slug: /tdengine-reference/client-libraries/java diff --git a/docs/en/14-reference/05-connector/20-go.md b/docs/en/14-reference/05-connector/20-go.md index bf0e6dd979..578150f0fa 100644 --- a/docs/en/14-reference/05-connector/20-go.md +++ b/docs/en/14-reference/05-connector/20-go.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: Go title: Go Client Library slug: /tdengine-reference/client-libraries/go diff --git a/docs/en/14-reference/05-connector/26-rust.md b/docs/en/14-reference/05-connector/26-rust.md index 637d009b8c..8de5d628de 100644 --- a/docs/en/14-reference/05-connector/26-rust.md +++ b/docs/en/14-reference/05-connector/26-rust.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: Rust title: Rust Client Library slug: /tdengine-reference/client-libraries/rust diff --git a/docs/en/14-reference/05-connector/35-node.md b/docs/en/14-reference/05-connector/35-node.md index 19dae0357f..49b1d200cf 100644 --- a/docs/en/14-reference/05-connector/35-node.md +++ b/docs/en/14-reference/05-connector/35-node.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: Node.js title: Node.js Client Library slug: /tdengine-reference/client-libraries/node diff --git a/docs/en/14-reference/05-connector/40-csharp.md b/docs/en/14-reference/05-connector/40-csharp.md index 8e51cb319b..c9c9f95228 100644 --- a/docs/en/14-reference/05-connector/40-csharp.md +++ b/docs/en/14-reference/05-connector/40-csharp.md @@ -1,4 +1,5 @@ --- +toc_max_heading_level: 4 sidebar_label: C# title: C# Client Library slug: /tdengine-reference/client-libraries/csharp From a5bafca5fbe2443a6d680349c6e599a7ba477fda Mon Sep 17 00:00:00 2001 From: Haolin Wang Date: Mon, 30 Dec 2024 14:18:55 +0800 Subject: [PATCH 31/73] test: insert into table from csv file --- tests/parallel_test/cases.task | 1 + tests/system-test/1-insert/insert_from_csv.py | 47 +++++++++++++++++++ .../1-insert/test_insert_from_csv.csv | 5 ++ tests/system-test/simpletest.bat | 1 + 4 files changed, 54 insertions(+) create mode 100644 tests/system-test/1-insert/insert_from_csv.py create mode 100644 tests/system-test/1-insert/test_insert_from_csv.csv diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 9047b0f3af..3eeb583c11 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -443,6 +443,7 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/InsertFuturets.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_wide_column.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_column_value.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_from_csv.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k_benchmark.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k.py -R diff --git a/tests/system-test/1-insert/insert_from_csv.py b/tests/system-test/1-insert/insert_from_csv.py new file mode 100644 index 0000000000..38f6a8b09d --- /dev/null +++ b/tests/system-test/1-insert/insert_from_csv.py @@ -0,0 +1,47 @@ +import taos +import sys +import datetime +import inspect +import random +from util.dnodes import TDDnode +from util.dnodes import tdDnodes + +from util.log import * +from util.sql import * +from util.cases import * + +class TDTestCase: + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), True) + + self.testcasePath = os.path.split(__file__)[0] + self.testcasePath = self.testcasePath.replace('\\', '//') + self.database = "test_insert_csv_db" + self.table = "test_insert_csv_tbl" + + def insert_from_csv(self): + tdSql.execute(f"drop database if exists {self.database}") + tdSql.execute(f"create database {self.database}") + tdSql.execute(f"use {self.database}") + tdSql.execute(f"create table {self.table} (ts timestamp, c1 nchar(16), c2 double, c3 int)") + tdSql.execute(f"insert into {self.table} file '{self.testcasePath}//test_insert_from_csv.csv'") + tdSql.query(f"select count(*) from {self.table}") + tdSql.checkData(0, 0, 5) + + def run(self): + tdSql.prepare() + + startTime_all = time.time() + self.insert_from_csv() + endTime_all = time.time() + print("total time %ds" % (endTime_all - startTime_all)) + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/1-insert/test_insert_from_csv.csv b/tests/system-test/1-insert/test_insert_from_csv.csv new file mode 100644 index 0000000000..966af8c27a --- /dev/null +++ b/tests/system-test/1-insert/test_insert_from_csv.csv @@ -0,0 +1,5 @@ +'2024-12-13 09:30:00.050','ABCDEF68',24.774736842805263,200 +'2024-12-13 09:30:00.060','ABCDEF68',24.774736842805263,201 +'2024-12-13 09:30:00.080','ABCDEF68',24.774736842805263,202 +'2024-12-13 09:30:00.100','ABCDEF68',24.774736842805263,203 +'2024-12-13 09:30:00.110','ABCDEF68',24.774736842805263,204 diff --git a/tests/system-test/simpletest.bat b/tests/system-test/simpletest.bat index a1f7273ad4..5948c7fc80 100644 --- a/tests/system-test/simpletest.bat +++ b/tests/system-test/simpletest.bat @@ -13,6 +13,7 @@ python3 .\test.py -f 0-others\cachemodel.py @REM python3 .\test.py -f 0-others\fsync.py python3 .\test.py -f 1-insert\influxdb_line_taosc_insert.py +python3 .\test.py -f 1-insert\insert_from_csv.py @REM python3 .\test.py -f 1-insert\opentsdb_telnet_line_taosc_insert.py @REM python3 .\test.py -f 1-insert\opentsdb_json_taosc_insert.py @REM #python3 .\test.py -f 1-insert\test_stmt_muti_insert_query.py From 1126f97853d59627982bf6fe729642d704bed77a Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Mon, 30 Dec 2024 18:25:20 +0800 Subject: [PATCH 32/73] chore: add two more badges [skip ci] --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index ff72412434..ed04fafee1 100644 --- a/README.md +++ b/README.md @@ -12,6 +12,7 @@ [![Build Status](https://github.com/taosdata/TDengine/actions/workflows/taosd-ci-build.yml/badge.svg)](https://github.com/taosdata/TDengine/actions/workflows/taosd-ci-build.yml) [![Coverage Status](https://coveralls.io/repos/github/taosdata/TDengine/badge.svg?branch=3.0)](https://coveralls.io/github/taosdata/TDengine?branch=3.0) +![GitHub License](https://img.shields.io/github/license/taosdata/tdengine) [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4201/badge)](https://bestpractices.coreinfrastructure.org/projects/4201)
[![Twitter Follow](https://img.shields.io/twitter/follow/tdenginedb?label=TDengine&style=social)](https://twitter.com/tdenginedb) @@ -19,6 +20,9 @@ [![Discord Community](https://img.shields.io/badge/Join_Discord--white?logo=discord&style=social)](https://discord.com/invite/VZdSuUg4pS) [![LinkedIn](https://img.shields.io/badge/Follow_LinkedIn--white?logo=linkedin&style=social)](https://www.linkedin.com/company/tdengine) [![StackOverflow](https://img.shields.io/badge/Ask_StackOverflow--white?logo=stackoverflow&style=social&logoColor=orange)](https://stackoverflow.com/questions/tagged/tdengine) +
+![GitHub Release](https://img.shields.io/github/v/release/taosdata/tdengine) +![GitHub commit activity](https://img.shields.io/github/commit-activity/m/taosdata/tdengine) English | [简体中文](README-CN.md) | [TDengine Cloud](https://cloud.tdengine.com) | [Learn more about TSDB](https://tdengine.com/tsdb/) From 24f6828b9f90b5ef571756dbeabb6881455b78ba Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 18:27:39 +0800 Subject: [PATCH 33/73] docs: fix tags double mistaken --- docs/en/14-reference/05-connector/30-python.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/en/14-reference/05-connector/30-python.md b/docs/en/14-reference/05-connector/30-python.md index 05ae323f9a..7cf5858c2f 100644 --- a/docs/en/14-reference/05-connector/30-python.md +++ b/docs/en/14-reference/05-connector/30-python.md @@ -436,8 +436,8 @@ TaosResult object can be iterated over to retrieve queried data. - **Interface Description**：Creating an STMT2 object using a connection object - **Parameter Description** - `sql`: The bound SQL statement will call the `prepare` function if it is not empty - - `option` Pass in TaoStmt2Option class instance options - - **Return Value**：stmt2 object + - `option` Pass in `TaoStmt2Option` class instance + - **Return Value**：STMT2 object - **Exception**：Throws `ConnectionError` on failure - `def prepare(self, sql)` - **Interface Description**：Bind a precompiled SQL statement @@ -449,12 +449,12 @@ TaosResult object can be iterated over to retrieve queried data. - **Parameter Description**： - `tbnames`:Bind table name array, data type is list - `tags`: Bind tag column value array, data type is list - - `tags`: Bind a regular column value array with a data type of list + - `datas`: Bind data column value array, data type of list - **Exception**：Throws `StatementError` on failure - `def bind_param_with_tables(self, tables)` - - **Interface Description**：Bind data in an independent table format. Independent tables are organized by table units, with table names, TAG values, and regular column numerical attributes in each table + - **Interface Description**：Bind data in an independent table format. Independent tables are organized by table units, with table name, TAG value, and data column attributes in table object - **Parameter Description**： - - `tables`: `BindTable` Independent Table Object Array + - `tables`: `BindTable` Independent table object array - **Exception**：Throws `StatementError` on failure - `def execute(self) -> int:` - **Interface Description**：Execute to write all bound data From 7c36eda698dabe5064a42d856e5c08cbc904fc01 Mon Sep 17 00:00:00 2001 From: Shengliang Guan Date: Mon, 30 Dec 2024 19:36:25 +0800 Subject: [PATCH 34/73] Revert "Fix infinite loop when insert from CSV file on Windows" --- source/os/src/osFile.c | 2 +- tests/parallel_test/cases.task | 1 - tests/system-test/1-insert/insert_from_csv.py | 47 ------------------- .../1-insert/test_insert_from_csv.csv | 5 -- tests/system-test/simpletest.bat | 1 - 5 files changed, 1 insertion(+), 55 deletions(-) delete mode 100644 tests/system-test/1-insert/insert_from_csv.py delete mode 100644 tests/system-test/1-insert/test_insert_from_csv.csv diff --git a/source/os/src/osFile.c b/source/os/src/osFile.c index b1198e1cb2..8a2606c4c2 100644 --- a/source/os/src/osFile.c +++ b/source/os/src/osFile.c @@ -1403,7 +1403,7 @@ int64_t taosGetLineFile(TdFilePtr pFile, char **__restrict ptrBuf) { } (*ptrBuf)[totalBytesRead] = '\0'; - ret = (totalBytesRead > 0 ? totalBytesRead : -1); // -1 means EOF + ret = totalBytesRead; #else size_t len = 0; ret = getline(ptrBuf, &len, pFile->fp); diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 3eeb583c11..9047b0f3af 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -443,7 +443,6 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/InsertFuturets.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_wide_column.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_column_value.py -,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_from_csv.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k_benchmark.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k.py -R diff --git a/tests/system-test/1-insert/insert_from_csv.py b/tests/system-test/1-insert/insert_from_csv.py deleted file mode 100644 index 38f6a8b09d..0000000000 --- a/tests/system-test/1-insert/insert_from_csv.py +++ /dev/null @@ -1,47 +0,0 @@ -import taos -import sys -import datetime -import inspect -import random -from util.dnodes import TDDnode -from util.dnodes import tdDnodes - -from util.log import * -from util.sql import * -from util.cases import * - -class TDTestCase: - def init(self, conn, logSql, replicaVar=1): - self.replicaVar = int(replicaVar) - tdLog.debug(f"start to excute {__file__}") - tdSql.init(conn.cursor(), True) - - self.testcasePath = os.path.split(__file__)[0] - self.testcasePath = self.testcasePath.replace('\\', '//') - self.database = "test_insert_csv_db" - self.table = "test_insert_csv_tbl" - - def insert_from_csv(self): - tdSql.execute(f"drop database if exists {self.database}") - tdSql.execute(f"create database {self.database}") - tdSql.execute(f"use {self.database}") - tdSql.execute(f"create table {self.table} (ts timestamp, c1 nchar(16), c2 double, c3 int)") - tdSql.execute(f"insert into {self.table} file '{self.testcasePath}//test_insert_from_csv.csv'") - tdSql.query(f"select count(*) from {self.table}") - tdSql.checkData(0, 0, 5) - - def run(self): - tdSql.prepare() - - startTime_all = time.time() - self.insert_from_csv() - endTime_all = time.time() - print("total time %ds" % (endTime_all - startTime_all)) - - def stop(self): - tdSql.close() - tdLog.success("%s successfully executed" % __file__) - - -tdCases.addWindows(__file__, TDTestCase()) -tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/1-insert/test_insert_from_csv.csv b/tests/system-test/1-insert/test_insert_from_csv.csv deleted file mode 100644 index 966af8c27a..0000000000 --- a/tests/system-test/1-insert/test_insert_from_csv.csv +++ /dev/null @@ -1,5 +0,0 @@ -'2024-12-13 09:30:00.050','ABCDEF68',24.774736842805263,200 -'2024-12-13 09:30:00.060','ABCDEF68',24.774736842805263,201 -'2024-12-13 09:30:00.080','ABCDEF68',24.774736842805263,202 -'2024-12-13 09:30:00.100','ABCDEF68',24.774736842805263,203 -'2024-12-13 09:30:00.110','ABCDEF68',24.774736842805263,204 diff --git a/tests/system-test/simpletest.bat b/tests/system-test/simpletest.bat index 5948c7fc80..a1f7273ad4 100644 --- a/tests/system-test/simpletest.bat +++ b/tests/system-test/simpletest.bat @@ -13,7 +13,6 @@ python3 .\test.py -f 0-others\cachemodel.py @REM python3 .\test.py -f 0-others\fsync.py python3 .\test.py -f 1-insert\influxdb_line_taosc_insert.py -python3 .\test.py -f 1-insert\insert_from_csv.py @REM python3 .\test.py -f 1-insert\opentsdb_telnet_line_taosc_insert.py @REM python3 .\test.py -f 1-insert\opentsdb_json_taosc_insert.py @REM #python3 .\test.py -f 1-insert\test_stmt_muti_insert_query.py From 977a452ea17e3d163abe6e55e4bc2ee2e921bc70 Mon Sep 17 00:00:00 2001 From: Haolin Wang Date: Mon, 30 Dec 2024 19:41:41 +0800 Subject: [PATCH 35/73] Revert "Revert "Fix infinite loop when insert from CSV file on Windows"" --- source/os/src/osFile.c | 2 +- tests/parallel_test/cases.task | 1 + tests/system-test/1-insert/insert_from_csv.py | 47 +++++++++++++++++++ .../1-insert/test_insert_from_csv.csv | 5 ++ tests/system-test/simpletest.bat | 1 + 5 files changed, 55 insertions(+), 1 deletion(-) create mode 100644 tests/system-test/1-insert/insert_from_csv.py create mode 100644 tests/system-test/1-insert/test_insert_from_csv.csv diff --git a/source/os/src/osFile.c b/source/os/src/osFile.c index 8a2606c4c2..b1198e1cb2 100644 --- a/source/os/src/osFile.c +++ b/source/os/src/osFile.c @@ -1403,7 +1403,7 @@ int64_t taosGetLineFile(TdFilePtr pFile, char **__restrict ptrBuf) { } (*ptrBuf)[totalBytesRead] = '\0'; - ret = totalBytesRead; + ret = (totalBytesRead > 0 ? totalBytesRead : -1); // -1 means EOF #else size_t len = 0; ret = getline(ptrBuf, &len, pFile->fp); diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 9047b0f3af..3eeb583c11 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -443,6 +443,7 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/InsertFuturets.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_wide_column.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_column_value.py +,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/insert_from_csv.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k_benchmark.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k.py ,,y,system-test,./pytest.sh python3 ./test.py -f 1-insert/rowlength64k.py -R diff --git a/tests/system-test/1-insert/insert_from_csv.py b/tests/system-test/1-insert/insert_from_csv.py new file mode 100644 index 0000000000..38f6a8b09d --- /dev/null +++ b/tests/system-test/1-insert/insert_from_csv.py @@ -0,0 +1,47 @@ +import taos +import sys +import datetime +import inspect +import random +from util.dnodes import TDDnode +from util.dnodes import tdDnodes + +from util.log import * +from util.sql import * +from util.cases import * + +class TDTestCase: + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), True) + + self.testcasePath = os.path.split(__file__)[0] + self.testcasePath = self.testcasePath.replace('\\', '//') + self.database = "test_insert_csv_db" + self.table = "test_insert_csv_tbl" + + def insert_from_csv(self): + tdSql.execute(f"drop database if exists {self.database}") + tdSql.execute(f"create database {self.database}") + tdSql.execute(f"use {self.database}") + tdSql.execute(f"create table {self.table} (ts timestamp, c1 nchar(16), c2 double, c3 int)") + tdSql.execute(f"insert into {self.table} file '{self.testcasePath}//test_insert_from_csv.csv'") + tdSql.query(f"select count(*) from {self.table}") + tdSql.checkData(0, 0, 5) + + def run(self): + tdSql.prepare() + + startTime_all = time.time() + self.insert_from_csv() + endTime_all = time.time() + print("total time %ds" % (endTime_all - startTime_all)) + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/1-insert/test_insert_from_csv.csv b/tests/system-test/1-insert/test_insert_from_csv.csv new file mode 100644 index 0000000000..966af8c27a --- /dev/null +++ b/tests/system-test/1-insert/test_insert_from_csv.csv @@ -0,0 +1,5 @@ +'2024-12-13 09:30:00.050','ABCDEF68',24.774736842805263,200 +'2024-12-13 09:30:00.060','ABCDEF68',24.774736842805263,201 +'2024-12-13 09:30:00.080','ABCDEF68',24.774736842805263,202 +'2024-12-13 09:30:00.100','ABCDEF68',24.774736842805263,203 +'2024-12-13 09:30:00.110','ABCDEF68',24.774736842805263,204 diff --git a/tests/system-test/simpletest.bat b/tests/system-test/simpletest.bat index a1f7273ad4..5948c7fc80 100644 --- a/tests/system-test/simpletest.bat +++ b/tests/system-test/simpletest.bat @@ -13,6 +13,7 @@ python3 .\test.py -f 0-others\cachemodel.py @REM python3 .\test.py -f 0-others\fsync.py python3 .\test.py -f 1-insert\influxdb_line_taosc_insert.py +python3 .\test.py -f 1-insert\insert_from_csv.py @REM python3 .\test.py -f 1-insert\opentsdb_telnet_line_taosc_insert.py @REM python3 .\test.py -f 1-insert\opentsdb_json_taosc_insert.py @REM #python3 .\test.py -f 1-insert\test_stmt_muti_insert_query.py From 6bcbf6cc37a0243e28a868d0dd5b92d2d0e17a92 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 20:22:29 +0800 Subject: [PATCH 36/73] fix: taospy version update to 2.7.21 --- docs/en/14-reference/05-connector/30-python.md | 2 +- docs/zh/14-reference/05-connector/30-python.mdx | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/14-reference/05-connector/30-python.md b/docs/en/14-reference/05-connector/30-python.md index 7cf5858c2f..19247e5364 100644 --- a/docs/en/14-reference/05-connector/30-python.md +++ b/docs/en/14-reference/05-connector/30-python.md @@ -56,7 +56,7 @@ Python Connector historical versions (it is recommended to use the latest versio |Python Connector Version | Major Changes | TDengine Version| | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | -|2.7.20 | Native supports STMT2 writing | - | +|2.7.21 | Native supports STMT2 writing | - | |2.7.19 | Support Apache Superset connection to TDengine Cloud data source | - | |2.7.18 | Support Apache SuperSet BI Tools. | - | |2.7.16 | Add subscription configuration (session. timeout. ms, Max. roll. interval. ms). | - | diff --git a/docs/zh/14-reference/05-connector/30-python.mdx b/docs/zh/14-reference/05-connector/30-python.mdx index a1fedfd92c..42ed67e927 100644 --- a/docs/zh/14-reference/05-connector/30-python.mdx +++ b/docs/zh/14-reference/05-connector/30-python.mdx @@ -52,7 +52,7 @@ Python Connector 历史版本（建议使用最新版本的 `taospy`）： | Python Connector 版本 | 主要变化 | TDengine 版本 | | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | -| 2.7.20 | Native 支持 STMT2 写入 | - | +| 2.7.21 | Native 支持 STMT2 写入 | - | | 2.7.19 | 支持 Apache Superset 连接 TDengine Cloud 数据源 | - | | 2.7.18 | 支持 Apache Superset 产品连接本地 TDengine 数据源 | - | | 2.7.16 | 新增订阅配置 (session.timeout.ms, max.poll.interval.ms) | - | From 9bf4e232895c384582569cd91a46aefe459b4cd4 Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Mon, 30 Dec 2024 20:53:06 +0800 Subject: [PATCH 37/73] fix: set latest taospy version --- tests/parallel_test/cases.task | 20 ++++++++++---------- tests/parallel_test/run_case.sh | 2 +- 2 files changed, 11 insertions(+), 11 deletions(-) diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 9047b0f3af..a3e87a5334 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -6,6 +6,16 @@ ,,n,unit-test,bash test.sh +#docs-examples test +,,n,docs-examples-test,bash c.sh +,,n,docs-examples-test,bash python.sh +,,n,docs-examples-test,bash node.sh +,,n,docs-examples-test,bash csharp.sh +,,n,docs-examples-test,bash jdbc.sh +,,n,docs-examples-test,bash rust.sh +,,n,docs-examples-test,bash go.sh +,,n,docs-examples-test,bash test_R.sh + # # army-test # @@ -1661,13 +1671,3 @@ ,,n,develop-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/sml_json_alltypes.py ,,n,develop-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/taosdemoTestQueryWithJson.py -R ,,n,develop-test,python3 ./test.py -f 5-taos-tools/taosbenchmark/telnet_tcp.py -R - -#docs-examples test -,,n,docs-examples-test,bash c.sh -,,n,docs-examples-test,bash python.sh -,,n,docs-examples-test,bash node.sh -,,n,docs-examples-test,bash csharp.sh -,,n,docs-examples-test,bash jdbc.sh -,,n,docs-examples-test,bash rust.sh -,,n,docs-examples-test,bash go.sh -,,n,docs-examples-test,bash test_R.sh diff --git a/tests/parallel_test/run_case.sh b/tests/parallel_test/run_case.sh index a78d0aa4a4..5dc1cef673 100755 --- a/tests/parallel_test/run_case.sh +++ b/tests/parallel_test/run_case.sh @@ -77,7 +77,7 @@ md5sum /usr/lib/libtaos.so.1 md5sum /home/TDinternal/debug/build/lib/libtaos.so #get python connector and update: taospy 2.7.16 taos-ws-py 0.3.5 -pip3 install taospy==2.7.16 +pip3 install taospy==2.7.21 pip3 install taos-ws-py==0.3.5 $TIMEOUT_CMD $cmd RET=$? From 88a938d00822cf4ad27b46b8f81905e84382db72 Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Tue, 31 Dec 2024 01:10:39 +0800 Subject: [PATCH 38/73] feat: add source msg to cleanup function when exit --- packaging/setup_env.sh | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh index 42ae6b3814..3899ddc4fd 100644 --- a/packaging/setup_env.sh +++ b/packaging/setup_env.sh @@ -1,4 +1,8 @@ #!/bin/bash + +# Run cleanup function on exit +trap cleanup EXIT + # define default timezone DEFAULT_TIMEZONE="Asia/Shanghai" @@ -64,6 +68,9 @@ PROCESS_EXPORTER_BINARY="/usr/local/bin/process-exporter" # Define fstab input FSTAB_LINE="share-server.platform.tdengine.dev:/mnt/share_server /mnt/share_server nfs rw,sync,_netdev 0 0" +# Results need to be stored when source +SOURCE_RESULTS="" + # ANSI color codes GREEN='\033[0;32m' # Green color RED='\033[0;31m' # Red color @@ -917,6 +924,7 @@ install_java() { INSTALLED_VERSION=$("$JAVA_HOME"/bin/java --version 2>&1) if echo "$INSTALLED_VERSION" | grep -q "openjdk $DEFAULT_JDK_VERSION"; then echo -e "${GREEN}Java installed successfully.${NO_COLOR}" + SOURCE_RESULTS+="Java: source /root/.bashrc\n" else echo -e "${YELLOW}Java version not match.${NO_COLOR}" exit 1 @@ -952,6 +960,7 @@ install_maven() { fi mvn -version check_status "Failed to install maven" "Maven installed successfully." $? + SOURCE_RESULTS+="Sdkman: source $HOME/.sdkman/bin/sdkman-init.sh\n" } # Install Go @@ -987,6 +996,7 @@ deploy_go() { # Apply the environment variables $GO_INSTALL_DIR/bin/go version check_status "Failed to install GO" "Install GO successfully" $? + SOURCE_RESULTS+="Golang: source $BASH_RC\n" } # Function to install Rust and Cargo @@ -1037,6 +1047,7 @@ deploy_rust() { # Install cargo-make cargo install cargo-make check_status "Failed to install Rust" "Install Rust successfully" $? + SOURCE_RESULTS+="Rust: source $BASH_RC && source $HOME/.cargo/env\n" else echo "Rust is already installed." fi @@ -1121,6 +1132,7 @@ install_node_via_nvm () { [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" echo -e "${GREEN}NVM installed successfully.${NO_COLOR}" + SOURCE_RESULTS+="NVM: source $NVM_DIR/nvm.sh && source $NVM_DIR/bash_completion\n" else echo -e "${GREEN}NVM is already installed.${NO_COLOR}" fi @@ -1655,6 +1667,15 @@ config_cloud_init() { # cloud-init clean --logs } +cleanup() { + echo -e "${GREEN}===========================================\n${NO_COLOR}" + echo -e "${GREEN}Installation complete! \n${NO_COLOR}" + echo -e "${GREEN}Some tools require you to manually source${NO_COLOR}" + echo -e "${GREEN}or restart your terminal to take effect.\n${NO_COLOR}" + echo -e "${GREEN}===========================================\n${NO_COLOR}" + echo -e "${GREEN}$SOURCE_RESULTS${NO_COLOR}" +} + # Clone a repository with a specified target directory clone_repo_with_rename() { local repo_url="$1" @@ -1752,9 +1773,10 @@ clone_repos() { new_funcs() { echo "Adding test..." install_python 3.10.12 - # install_java 21 - # install_node 16.20.2 - # install_maven 3.2.5 + install_java 21 + install_node 16.20.2 + install_maven 3.2.5 + deploy_rust } # deploy TDasset @@ -1908,6 +1930,9 @@ main() { replace_sources) replace_sources ;; + update) + update + ;; upgrade) upgrade ;; From 7304750f47b5c4e0b0caa7b1efa208c0fe4045f7 Mon Sep 17 00:00:00 2001 From: Shengliang Guan Date: Tue, 31 Dec 2024 09:07:39 +0800 Subject: [PATCH 39/73] doc: update s3 support --- docs/zh/08-operation/12-multi.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/08-operation/12-multi.md b/docs/zh/08-operation/12-multi.md index d18957d7d2..122fb9c2f3 100644 --- a/docs/zh/08-operation/12-multi.md +++ b/docs/zh/08-operation/12-multi.md @@ -60,7 +60,7 @@ dataDir /mnt/data6 2 0 ## 对象存储 -本节介绍在 TDengine Enterprise 如何使用 S3 对象存储，本功能基于通用 S3 SDK 实现，对各个 S3 平台的访问参数进行了兼容适配，可以访问如 minio，腾讯云 COS，Amazon S3 等对象存储服务。通过适当的参数配置，可以把大部分较冷的时序数据存储到 S3 服务中。 +本节介绍在 TDengine Enterprise 如何使用 S3 对象存储，本功能基于通用 S3 SDK 实现，对各个 S3 平台的访问参数进行了兼容适配，可以访问如 Amazon S3、Azure Blob、华为 OBS、腾讯云 COS、阿里云 OSS、minio等对象存储服务。通过适当的参数配置，可以把大部分较冷的时序数据存储到 S3 服务中。 **注意** 在配合多级存储使用时，每一级存储介质上保存的数据都有可能被按规则备份到远程对象存储中并删除本地数据文件。 From 8967d8f2096d57209621fa43ab4bb37438270aa4 Mon Sep 17 00:00:00 2001 From: menshibin Date: Tue, 31 Dec 2024 11:09:33 +0800 Subject: [PATCH 40/73] add flink doc --- docs/examples/flink/Main.java | 586 ++++++++++++++++++ .../10-third-party/01-collection/12-flink.md | 366 +++++++++++ 2 files changed, 952 insertions(+) create mode 100644 docs/examples/flink/Main.java create mode 100644 docs/zh/10-third-party/01-collection/12-flink.md diff --git a/docs/examples/flink/Main.java b/docs/examples/flink/Main.java new file mode 100644 index 0000000000..b601778776 --- /dev/null +++ b/docs/examples/flink/Main.java @@ -0,0 +1,586 @@ +package com.taosdata.flink.example; + +import com.taosdata.flink.cdc.TDengineCdcSource; +import com.taosdata.flink.common.TDengineCdcParams; +import com.taosdata.flink.common.TDengineConfigParams; +import com.taosdata.flink.sink.TDengineSink; +import com.taosdata.flink.source.TDengineSource; +import com.taosdata.flink.source.entity.SourceSplitSql; +import com.taosdata.flink.source.entity.SplitType; +import com.taosdata.flink.source.entity.TimestampSplitInfo; +import com.taosdata.jdbc.TSDBDriver; +import com.taosdata.jdbc.tmq.ConsumerRecords; +import org.apache.flink.api.common.eventtime.WatermarkStrategy; +import org.apache.flink.core.execution.JobClient; +import org.apache.flink.shaded.curator5.com.google.common.base.Strings; +import org.apache.flink.streaming.api.datastream.DataStreamSource; +import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; +import org.apache.flink.table.api.EnvironmentSettings; +import org.apache.flink.table.api.TableResult; +import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; +import org.apache.flink.table.data.RowData; +import org.junit.Assert; + +import java.sql.Connection; +import java.sql.DriverManager; +import java.sql.SQLException; +import java.sql.Statement; +import java.text.SimpleDateFormat; +import java.time.Duration; + +import java.time.ZoneId; +import java.util.Arrays; +import java.util.List; +import java.util.Properties; + +import javax.xml.transform.Source; + +import org.apache.flink.streaming.api.CheckpointingMode; + + + +public class Main { + static String jdbcUrl = "jdbc:TAOS-WS://localhost:6041?user=root&password=taosdata"; + static void prepare() throws ClassNotFoundException, SQLException { + Properties properties = new Properties(); + properties.setProperty(TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT, "true"); + properties.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8"); + properties.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8"); + String insertQuery = "INSERT INTO " + + "power.d1001 USING power.meters TAGS('California.SanFrancisco', 1) " + + "VALUES " + + "('2024-12-19 19:12:45.642', 50.30000, 201, 0.31000) " + + "('2024-12-19 19:12:46.642', 82.60000, 202, 0.33000) " + + "('2024-12-19 19:12:47.642', 92.30000, 203, 0.31000) " + + "('2024-12-19 18:12:45.642', 50.30000, 201, 0.31000) " + + "('2024-12-19 18:12:46.642', 82.60000, 202, 0.33000) " + + "('2024-12-19 18:12:47.642', 92.30000, 203, 0.31000) " + + "('2024-12-19 17:12:45.642', 50.30000, 201, 0.31000) " + + "('2024-12-19 17:12:46.642', 82.60000, 202, 0.33000) " + + "('2024-12-19 17:12:47.642', 92.30000, 203, 0.31000) " + + "power.d1002 USING power.meters TAGS('Alabama.Montgomery', 2) " + + "VALUES " + + "('2024-12-19 19:12:45.642', 50.30000, 204, 0.25000) " + + "('2024-12-19 19:12:46.642', 62.60000, 205, 0.33000) " + + "('2024-12-19 19:12:47.642', 72.30000, 206, 0.31000) " + + "('2024-12-19 18:12:45.642', 50.30000, 204, 0.25000) " + + "('2024-12-19 18:12:46.642', 62.60000, 205, 0.33000) " + + "('2024-12-19 18:12:47.642', 72.30000, 206, 0.31000) " + + "('2024-12-19 17:12:45.642', 50.30000, 204, 0.25000) " + + "('2024-12-19 17:12:46.642', 62.60000, 205, 0.33000) " + + "('2024-12-19 17:12:47.642', 72.30000, 206, 0.31000) "; + + Class.forName("com.taosdata.jdbc.ws.WebSocketDriver"); + try (Connection connection = DriverManager.getConnection(jdbcUrl, properties); + Statement stmt = connection.createStatement()) { + + stmt.executeUpdate("DROP TOPIC IF EXISTS topic_meters"); + + stmt.executeUpdate("DROP database IF EXISTS power"); + // create database + int rowsAffected = stmt.executeUpdate("CREATE DATABASE IF NOT EXISTS power vgroups 5"); + + stmt.executeUpdate("use power"); + // you can check rowsAffected here + System.out.println("Create database power successfully, rowsAffected: " + rowsAffected); + // create table + rowsAffected = stmt.executeUpdate("CREATE STABLE IF NOT EXISTS meters (ts timestamp, current float, voltage int, phase float) TAGS (location binary(64), groupId int);"); + // you can check rowsAffected here + System.out.println("Create stable power.meters successfully, rowsAffected: " + rowsAffected); + + stmt.executeUpdate("CREATE TOPIC topic_meters as SELECT ts, `current`, voltage, phase, location, groupid, tbname FROM meters"); + + int affectedRows = stmt.executeUpdate(insertQuery); + // you can check affectedRows here + System.out.println("Successfully inserted " + affectedRows + " rows to power.meters."); + + stmt.executeUpdate("DROP database IF EXISTS power_sink"); + // create database + stmt.executeUpdate("CREATE DATABASE IF NOT EXISTS power_sink vgroups 5"); + + stmt.executeUpdate("use power_sink"); + // you can check rowsAffected here + System.out.println("Create database power successfully, rowsAffected: " + rowsAffected); + // create table + stmt.executeUpdate("CREATE STABLE IF NOT EXISTS sink_meters (ts timestamp, current float, voltage int, phase float) TAGS (location binary(64), groupId int);"); + // you can check rowsAffected here + + stmt.executeUpdate("CREATE TABLE IF NOT EXISTS sink_normal (ts timestamp, current float, voltage int, phase float);"); + // you can check rowsAffected here + + + } catch (Exception ex) { + // please refer to the JDBC specifications for detailed exceptions info + System.out.printf("Failed to create database power or stable meters, %sErrMessage: %s%n", + ex instanceof SQLException ? "ErrCode: " + ((SQLException) ex).getErrorCode() + ", " : "", + ex.getMessage()); + // Print stack trace for context in examples. Use logging in production. + throw ex; + } + } + + public static void main(String[] args) throws Exception { + prepare(); + if (args != null && args.length > 0 && args[0].equals("source")) { + testSource(); + } else if (args != null && args.length > 0 && args[0].equals("table")) { + testTableToSink(); + } else if (args != null && args.length > 0 && args[0].equals("cdc")) { + testCustomTypeCdc(); + }else if (args != null && args.length > 0 && args[0].equals("table-cdc")) { + testCdcTableToSink(); + } + } + + static SourceSplitSql getTimeSplit() { +// ANCHOR: time_interval +SourceSplitSql splitSql = new SourceSplitSql(); +splitSql.setSql("select ts, `current`, voltage, phase, groupid, location, tbname from meters") +.setSplitType(SplitType.SPLIT_TYPE_TIMESTAMP) +.setTimestampSplitInfo(new TimestampSplitInfo( + "2024-12-19 16:12:48.000", + "2024-12-19 19:12:48.000", + "ts", + Duration.ofHours(1), + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS"), + ZoneId.of("Asia/Shanghai"))); +// ANCHOR_END: time_interval +return splitSql; + } + + static SourceSplitSql getTagSplit() throws Exception { +// ANCHOR: tag_split +SourceSplitSql splitSql = new SourceSplitSql(); +splitSql.setSql("select ts, current, voltage, phase, groupid, location from meters where voltage > 100") + .setTagList(Arrays.asList("groupid >100 and location = 'Shanghai'", + "groupid >50 and groupid < 100 and location = 'Guangzhou'", + "groupid >0 and groupid < 50 and location = 'Beijing'")) + .setSplitType(SplitType.SPLIT_TYPE_TAG); +// ANCHOR_END: tag_split +return splitSql; + } + + static SourceSplitSql getTableSqlit() { +// ANCHOR: table_split +SourceSplitSql splitSql = new SourceSplitSql(); +splitSql.setSelect("ts, current, voltage, phase, groupid, location") + .setTableList(Arrays.asList("d1001", "d1002")) + .setOther("order by ts limit 100") + .setSplitType(SplitType.SPLIT_TYPE_TABLE); +// ANCHOR_END: table_split + } + + //ANCHOR: source_test + static void testSource() throws Exception { + Properties connProps = new Properties(); + connProps.setProperty(TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT, "true"); + connProps.setProperty(TDengineConfigParams.PROPERTY_KEY_TIME_ZONE, "UTC-8"); + connProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); + connProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + + splitSql.setSql("select ts, `current`, voltage, phase, groupid, location, tbname from meters") + .setSplitType(SplitType.SPLIT_TYPE_TIMESTAMP) + .setTimestampSplitInfo(new TimestampSplitInfo( + "2024-12-19 16:12:48.000", + "2024-12-19 19:12:48.000", + "ts", + Duration.ofHours(1), + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS"), + ZoneId.of("Asia/Shanghai"))); + + TDengineSource source = new TDengineSource<>(connProps, sql, RowData.class); + DataStreamSource input = env.fromSource(source, WatermarkStrategy.noWatermarks(), "tdengine-source"); + DataStream resultStream = input.map((MapFunction) rowData -> { + StringBuilder sb = new StringBuilder(); + sb.append("ts: " + rowData.getTimestamp(0, 0) + + ", current: " + rowData.getFloat(1) + + ", voltage: " + rowData.getInt(2) + + ", phase: " + rowData.getFloat(3) + + ", location: " + new String(rowData.getBinary(4))); + sb.append("\n"); + return sb.toString(); + }); + resultStream.print(); + env.execute("tdengine flink source"); + + } + //ANCHOR_END: source_test + + //ANCHOR: source_custom_type_test + void testCustomTypeSource() throws Exception { + System.out.println("testTDengineSourceByTimeSplit start！"); + Properties connProps = new Properties(); + connProps.setProperty(TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT, "true"); + connProps.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8"); + connProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "com.taosdata.flink.entity.ResultSoureDeserialization"); + connProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); + SourceSplitSql splitSql = new SourceSplitSql(); + splitSql.setSql("select ts, `current`, voltage, phase, groupid, location, tbname from meters") + .setSplitType(SplitType.SPLIT_TYPE_TIMESTAMP) + //按照时间分片 + .setTimestampSplitInfo(new TimestampSplitInfo( + "2024-12-19 16:12:48.000", + "2024-12-19 19:12:48.000", + "ts", + Duration.ofHours(1), + new SimpleDateFormat("yyyy-MM-dd HH:mm:ss.SSS"), + ZoneId.of("Asia/Shanghai"))); + + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + TDengineSource source = new TDengineSource<>(connProps, splitSql, ResultBean.class); + DataStreamSource input = env.fromSource(source, WatermarkStrategy.noWatermarks(), "tdengine-source"); + DataStream resultStream = input.map((MapFunction) rowData -> { + StringBuilder sb = new StringBuilder(); + sb.append("ts: " + rowData.getTs() + + ", current: " + rowData.getCurrent() + + ", voltage: " + rowData.getVoltage() + + ", phase: " + rowData.getPhase() + + ", groupid: " + rowData.getGroupid() + + ", location" + rowData.getLocation() + + ", tbname: " + rowData.getTbname()); + sb.append("\n"); + totalVoltage.addAndGet(rowData.getVoltage()); + return sb.toString(); + }); + resultStream.print(); + env.execute("flink tdengine source"); + } + //ANCHOR_END: source_custom_type_test + + //ANCHOR: source_batch_test + void testBatchSource() throws Exception { + Properties connProps = new Properties(); + connProps.setProperty(TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT, "true"); + connProps.setProperty(TDengineConfigParams.PROPERTY_KEY_TIME_ZONE, "UTC-8"); + connProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); + connProps.setProperty(TDengineConfigParams.TD_BATCH_MODE, "true"); + connProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + Class> typeClass = (Class>) (Class) SourceRecords.class; + SourceSplitSql sql = new SourceSplitSql("select ts, `current`, voltage, phase, tbname from meters"); + TDengineSource> source = new TDengineSource<>(connProps, sql, typeClass); + DataStreamSource> input = env.fromSource(source, WatermarkStrategy.noWatermarks(), "kafka-source"); + DataStream resultStream = input.map((MapFunction, String>) records -> { + StringBuilder sb = new StringBuilder(); + Iterator iterator = records.iterator(); + while (iterator.hasNext()) { + GenericRowData row = (GenericRowData) iterator.next(); + sb.append("ts: " + row.getTimestamp(0, 0) + + ", current: " + row.getFloat(1) + + ", voltage: " + row.getInt(2) + + ", phase: " + row.getFloat(3) + + ", location: " + new String(row.getBinary(4))); + sb.append("\n"); + totalVoltage.addAndGet(row.getInt(2)); + } + return sb.toString(); + }); + resultStream.print(); + env.execute("flink tdengine source"); + + } + //ANCHOR_END: source_batch_test + + //ANCHOR: cdc_source + void testTDengineCdc() throws Exception { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + env.enableCheckpointing(100, AT_LEAST_ONCE); + env.getConfig().setRestartStrategy(RestartStrategies.noRestart()); + Properties config = new Properties(); + config.setProperty(TDengineCdcParams.CONNECT_TYPE, "ws"); + config.setProperty(TDengineCdcParams.BOOTSTRAP_SERVERS, "localhost:6041"); + config.setProperty(TDengineCdcParams.AUTO_OFFSET_RESET, "earliest"); + config.setProperty(TDengineCdcParams.MSG_WITH_TABLE_NAME, "true"); + config.setProperty(TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS, "1000"); + config.setProperty(TDengineCdcParams.GROUP_ID, "group_1"); + config.setProperty(TDengineCdcParams.ENABLE_AUTO_COMMIT, "true"); + config.setProperty(TDengineCdcParams.CONNECT_USER, "root"); + config.setProperty(TDengineCdcParams.CONNECT_PASS, "taosdata"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER, "RowData"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER_ENCODING, "UTF-8"); + TDengineCdcSource tdengineSource = new TDengineCdcSource<>("topic_meters", config, RowData.class); + DataStreamSource input = env.fromSource(tdengineSource, WatermarkStrategy.noWatermarks(), "kafka-source"); + DataStream resultStream = input.map((MapFunction) rowData -> { + StringBuilder sb = new StringBuilder(); + sb.append("tsxx: " + rowData.getTimestamp(0, 0) + + ", current: " + rowData.getFloat(1) + + ", voltage: " + rowData.getInt(2) + + ", phase: " + rowData.getFloat(3) + + ", location: " + new String(rowData.getBinary(4))); + sb.append("\n"); + totalVoltage.addAndGet(rowData.getInt(2)); + return sb.toString(); + }); + resultStream.print(); + JobClient jobClient = env.executeAsync("Flink test cdc Example"); + Thread.sleep(5000L); + // The task submitted by Flink UI cannot be cancle and needs to be stopped on the UI page. + jobClient.cancel().get(); + } + //ANCHOR_END: cdc_source + + //ANCHOR: cdc_batch_source + void testTDengineCdcBatch() throws Exception { + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + Properties config = new Properties(); + config.setProperty(TDengineCdcParams.CONNECT_TYPE, "ws"); + config.setProperty(TDengineCdcParams.BOOTSTRAP_SERVERS, "localhost:6041"); + config.setProperty(TDengineCdcParams.AUTO_OFFSET_RESET, "earliest"); + config.setProperty(TDengineCdcParams.MSG_WITH_TABLE_NAME, "true"); + config.setProperty(TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS, "1000"); + config.setProperty(TDengineCdcParams.GROUP_ID, "group_1"); + config.setProperty(TDengineCdcParams.CONNECT_USER, "root"); + config.setProperty(TDengineCdcParams.CONNECT_PASS, "taosdata"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER, "RowData"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER_ENCODING, "UTF-8"); + config.setProperty(TDengineCdcParams.TMQ_BATCH_MODE, "true"); + + Class> typeClass = (Class>) (Class) ConsumerRecords.class; + TDengineCdcSource> tdengineSource = new TDengineCdcSource<>("topic_meters", config, typeClass); + DataStreamSource> input = env.fromSource(tdengineSource, WatermarkStrategy.noWatermarks(), "kafka-source"); + DataStream resultStream = input.map((MapFunction, String>) records -> { + Iterator> iterator = records.iterator(); + StringBuilder sb = new StringBuilder(); + while (iterator.hasNext()) { + GenericRowData row = (GenericRowData) iterator.next().value(); + sb.append("tsxx: " + row.getTimestamp(0, 0) + + ", current: " + row.getFloat(1) + + ", voltage: " + row.getInt(2) + + ", phase: " + row.getFloat(3) + + ", location: " + new String(row.getBinary(4))); + sb.append("\n"); + totalVoltage.addAndGet(row.getInt(2)); + } + return sb.toString(); + + }); + + resultStream.print(); + JobClient jobClient = env.executeAsync("Flink test cdc Example"); + Thread.sleep(5000L); + jobClient.cancel().get(); + } + //ANCHOR_END: cdc_batch_source + + //ANCHOR: cdc_custom_type_test + static void testCustomTypeCdc() throws Exception { + System.out.println("testCustomTypeTDengineCdc start！"); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + env.enableCheckpointing(100, AT_LEAST_ONCE); + env.getConfig().setRestartStrategy(RestartStrategies.noRestart()); + env.getCheckpointConfig().setTolerableCheckpointFailureNumber(4); + Properties config = new Properties(); + config.setProperty(TDengineCdcParams.CONNECT_TYPE, "ws"); + config.setProperty(TDengineCdcParams.BOOTSTRAP_SERVERS, "localhost:6041"); + config.setProperty(TDengineCdcParams.AUTO_OFFSET_RESET, "earliest"); + config.setProperty(TDengineCdcParams.MSG_WITH_TABLE_NAME, "true"); + config.setProperty(TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS, "1000"); + config.setProperty(TDengineCdcParams.GROUP_ID, "group_1"); + config.setProperty(TDengineCdcParams.CONNECT_USER, "root"); + config.setProperty(TDengineCdcParams.CONNECT_PASS, "taosdata"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER, "com.taosdata.flink.entity.ResultDeserializer"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER_ENCODING, "UTF-8"); + TDengineCdcSource tdengineSource = new TDengineCdcSource<>("topic_meters", config, ResultBean.class); + DataStreamSource input = env.fromSource(tdengineSource, WatermarkStrategy.noWatermarks(), "kafka-source"); + DataStream resultStream = input.map((MapFunction) rowData -> { + StringBuilder sb = new StringBuilder(); + sb.append("ts: " + rowData.getTs() + + ", current: " + rowData.getCurrent() + + ", voltage: " + rowData.getVoltage() + + ", phase: " + rowData.getPhase() + + ", groupid: " + rowData.getGroupid() + + ", location" + rowData.getLocation() + + ", tbname: " + rowData.getTbname()); + sb.append("\n"); + totalVoltage.addAndGet(rowData.getVoltage()); + return sb.toString(); + }); + resultStream.print(); + JobClient jobClient = env.executeAsync("Flink test cdc Example"); + Thread.sleep(5000L); + jobClient.cancel().get(); + } + //ANCHOR_END: cdc_custom_type_test + + //ANCHOR: RowDataToSink + static void testRowDataToSink() throws Exception { + Properties connProps = new Properties(); + connProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); + connProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); + SourceSplitSql splitSql = getTimeSplit(); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(1); + env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE); + TDengineSource source = new TDengineSource<>(connProps, sql, RowData.class); + DataStreamSource input = env.fromSource(source, WatermarkStrategy.noWatermarks(), "tdengine-source"); + Properties sinkProps = new Properties(); + sinkProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); + sinkProps.setProperty(TDengineConfigParams.TD_SOURCE_TYPE, "tdengine_source"); + sinkProps.setProperty(TDengineConfigParams.TD_DATABASE_NAME, "power_sink"); + sinkProps.setProperty(TDengineConfigParams.TD_SUPERTABLE_NAME, "sink_meters"); + sinkProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power_sink?user=root&password=taosdata"); + sinkProps.setProperty(TDengineConfigParams.TD_BATCH_SIZE, "2000"); + + // Arrays.asList The list of target table field names needs to be consistent with the data order + TDengineSink sink = new TDengineSink<>(sinkProps, + Arrays.asList("ts", "current", "voltage", "phase", "groupid", "location", "tbname")); + + input.sinkTo(sink); + env.execute("flink tdengine source"); + } + //ANCHOR_END: RowDataToSink + + //ANCHOR: BatchRowDataToSink + static void testBatchToTdSink() throws Exception { + System.out.println("testTDengineCdcToTdSink start！"); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + env.enableCheckpointing(500, CheckpointingMode.AT_LEAST_ONCE); + env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500); + env.getCheckpointConfig().setCheckpointTimeout(5000); + Properties config = new Properties(); + config.setProperty(TDengineCdcParams.CONNECT_TYPE, "ws"); + config.setProperty(TDengineCdcParams.BOOTSTRAP_SERVERS, "localhost:6041"); + config.setProperty(TDengineCdcParams.AUTO_OFFSET_RESET, "earliest"); + config.setProperty(TDengineCdcParams.MSG_WITH_TABLE_NAME, "true"); + config.setProperty(TDengineCdcParams.AUTO_COMMIT_INTERVAL, "1000"); + config.setProperty(TDengineCdcParams.GROUP_ID, "group_1"); + config.setProperty(TDengineCdcParams.CONNECT_USER, "root"); + config.setProperty(TDengineCdcParams.CONNECT_PASS, "taosdata"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER, "RowData"); + config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER_ENCODING, "UTF-8"); + config.setProperty(TDengineCdcParams.TMQ_BATCH_MODE, "true"); + + Class> typeClass = (Class>) (Class) ConsumerRecords.class; + TDengineCdcSource> tdengineSource = new TDengineCdcSource<>("topic_meters", config, typeClass); + DataStreamSource> input = env.fromSource(tdengineSource, WatermarkStrategy.noWatermarks(), "tdengine-source"); + + Properties sinkProps = new Properties(); + sinkProps.setProperty(TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT, "true"); + sinkProps.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8"); + sinkProps.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8"); + sinkProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); + sinkProps.setProperty(TDengineConfigParams.TD_BATCH_MODE, "true"); + sinkProps.setProperty(TDengineConfigParams.TD_SOURCE_TYPE, "tdengine_cdc"); + sinkProps.setProperty(TDengineConfigParams.TD_DATABASE_NAME, "power_sink"); + sinkProps.setProperty(TDengineConfigParams.TD_SUPERTABLE_NAME, "sink_meters"); + sinkProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); + sinkProps.setProperty(TDengineConfigParams.TD_BATCH_SIZE, "2000"); + + TDengineSink> sink = new TDengineSink<>(sinkProps, Arrays.asList("ts", "current", "voltage", "phase", "location", "groupid", "tbname")); + input.sinkTo(sink); + JobClient jobClient = env.executeAsync("Flink test cdc Example"); + Thread.sleep(6000L); + jobClient.cancel().get(); + System.out.println("testTDengineCdcToTdSink finish！"); + } + //ANCHOR_END: BatchRowDataToSink + + //ANCHOR: source_table + static void testTableToSink() throws Exception { + System.out.println("testTableToSink start！"); + EnvironmentSettings fsSettings = EnvironmentSettings.newInstance().inStreamingMode().build(); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(3); + env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE); + StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, fsSettings); + String tdengineSourceTableDDL = "CREATE TABLE `meters` (" + + " ts TIMESTAMP," + + " `current` FLOAT," + + " voltage INT," + + " phase FLOAT," + + " location VARBINARY," + + " groupid INT," + + " tbname VARBINARY" + + ") WITH (" + + " 'connector' = 'tdengine-connector'," + + " 'td.jdbc.url' = 'jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata'," + + " 'td.jdbc.mode' = 'source'," + + " 'table-name' = 'meters'," + + " 'scan.query' = 'SELECT ts, `current`, voltage, phase, location, groupid, tbname FROM `meters`'" + + ")"; + + + String tdengineSinkTableDDL = "CREATE TABLE `sink_meters` (" + + " ts TIMESTAMP," + + " `current` FLOAT," + + " voltage INT," + + " phase FLOAT," + + " location VARBINARY," + + " groupid INT," + + " tbname VARBINARY" + + ") WITH (" + + " 'connector' = 'tdengine-connector'," + + " 'td.jdbc.mode' = 'sink'," + + " 'td.jdbc.url' = 'jdbc:TAOS-WS://localhost:6041/power_sink?user=root&password=taosdata'," + + " 'sink.db.name' = 'power_sink'," + + " 'sink.supertable.name' = 'sink_meters'" + + ")"; + + tableEnv.executeSql(tdengineSourceTableDDL); + tableEnv.executeSql(tdengineSinkTableDDL); + tableEnv.executeSql("INSERT INTO sink_meters SELECT ts, `current`, voltage, phase, location, groupid, tbname FROM `meters`"); + } + //ANCHOR_END: source_table + + //ANCHOR: cdc_table + static void testCdcTableToSink() throws Exception { + EnvironmentSettings fsSettings = EnvironmentSettings.newInstance().inStreamingMode().build(); + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.setParallelism(5); + env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE); + StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env, fsSettings); + String tdengineSourceTableDDL = "CREATE TABLE `meters` (" + + " ts TIMESTAMP," + + " `current` FLOAT," + + " voltage INT," + + " phase FLOAT," + + " location VARBINARY," + + " groupid INT," + + " tbname VARBINARY" + + ") WITH (" + + " 'connector' = 'tdengine-connector'," + + " 'bootstrap.servers' = 'localhost:6041'," + + " 'td.jdbc.mode' = 'cdc'," + + " 'group.id' = 'group_22'," + + " 'auto.offset.reset' = 'earliest'," + + " 'enable.auto.commit' = 'false'," + + " 'topic' = 'topic_meters'" + + ")"; + + + String tdengineSinkTableDDL = "CREATE TABLE `sink_meters` (" + + " ts TIMESTAMP," + + " `current` FLOAT," + + " voltage INT," + + " phase FLOAT," + + " location VARBINARY," + + " groupid INT," + + " tbname VARBINARY" + + ") WITH (" + + " 'connector' = 'tdengine-connector'," + + " 'td.jdbc.mode' = 'cdc'," + + " 'td.jdbc.url' = 'jdbc:TAOS-WS://localhost:6041/power_sink?user=root&password=taosdata'," + + " 'sink.db.name' = 'power_sink'," + + " 'sink.supertable.name' = 'sink_meters'" + + ")"; + + tableEnv.executeSql(tdengineSourceTableDDL); + tableEnv.executeSql(tdengineSinkTableDDL); + + TableResult tableResult = tableEnv.executeSql("INSERT INTO sink_meters SELECT ts, `current`, voltage, phase, location, groupid, tbname FROM `meters`"); + + Thread.sleep(5000L); + tableResult.getJobClient().get().cancel().get(); + } + //ANCHOR_END: cdc_table + + +} diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md new file mode 100644 index 0000000000..8b2e5422b1 --- /dev/null +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -0,0 +1,366 @@ +--- +sidebar_label: Flink +title: TDengine Flink Connector +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +Apache Flink 是一款由 Apache 软件基金会支持的开源分布式流批一体化处理框架，可用于流处理、批处理、复杂事件处理、实时数据仓库构建及为机器学习提供实时数据支持等诸多大数据处理场景。与此同时，Flink 拥有丰富的连接器与各类工具，可对接众多不同类型的数据源实现数据的读取与写入。在数据处理的过程中，Flink 还提供了一系列可靠的容错机制，有力保障任务即便遭遇意外状况，依然能稳定、持续运行。 + +借助 TDengine 的 Flink 连接器，Apache Flink 得以与 TDengine 数据库无缝对接，一方面能够将经过复杂运算和深度分析后所得到的结果精准存入 TDengine 数据库，实现数据的高效存储与管理；另一方面，也可以从 TDengine 数据库中快速、稳定地读取海量数据，并在此基础上进行全面、深入的分析处理，充分挖掘数据的潜在价值，为企业的决策制定提供有力的数据支持和科学依据，极大地提升数据处理的效率和质量，增强企业在数字化时代的竞争力和创新能力。 + +## 前置条件 + +准备以下环境： +- TDengine 集群已部署并正常运行（企业及社区版均可） +- taosAdapter 能够正常运行。详细参考 [taosAdapter 使用手册](../../../reference/components/taosadapter) +- Apache Flink v1.19.0 或以上版本已安装。安装 Apache Flink 请参考 [官方文档](https://flink.apache.org/) + +## JRE 版本兼容性 + +- JRE: 支持 JRE 8 及以上版本。 + +# 支持的平台 + +Flink Connector 支持所有能运行 Flink 1.19 及以上版本的平台。 + +## 版本历史 +| Flink Connector 版本 | 主要变化 | TDengine 版本 | +| ------------------| ------------------------------------ | ---------------- | +| 1.1.0 | 1. 支持 SQL 查询 TDengine 数据库中的数据
2. 支持 CDC 订阅 TDengine 数据库中的数据
3. 支持 Table SQL 方式读取和写入 TDengine 数据库| 3.3.5.0 及以上版本 | + +## 异常和错误码 + +在任务执行失败后，查看 Flink 任务执行日志确认失败原因 + +具体的错误码请参考： + +| Error Code | Description | Suggested Actions | +| ---------------- |------------------------------------------------------- | -------------------- | +| 0xa000 |connection param error |连接器参数错误 +| 0xa001 |the groupid parameter of CDC is incorrect |CDC 的 groupid 参数错误。| +| 0xa002 |wrong topic parameter for CDC |CDC 的 topic 参数错误。| +| 0xa010 |database name configuration error |数据库名配置错误| +| 0xa011 |table name configuration error |表名配置错误| +| 0xa012 |no data was obtained from the data source |从数据源中获取数据失败| +| 0xa013 |value.deserializer parameter not set |未设置序列化方式| +| 0xa014 |list of column names for target table not set |未设置目标表的列名列表| +| 0x2301 |connection already closed |连接已经关闭，检查连接情况，或重新创建连接去执行相关指令。| +| 0x2302 |this operation is NOT supported currently! |当前使用接口不支持，可以更换其他连接方式。| +| 0x2303 |invalid variables |参数不合法，请检查相应接口规范，调整参数类型及大小。| +| 0x2304 |statement is closed |statement 已经关闭，请检查 statement 是否关闭后再次使用，或是连接是否正常。| +| 0x2305 |resultSet is closed |resultSet 结果集已经释放，请检查 resultSet 是否释放后再次使用。| +| 0x230d |parameter index out of range |参数越界，请检查参数的合理范围。| +| 0x230e |connection already closed |连接已经关闭，请检查 Connection 是否关闭后再次使用，或是连接是否正常。| +| 0x230f |unknown sql type in tdengine |请检查 TDengine 支持的 Data Type 类型。| +| 0x2315 |unknown taos type in tdengine |在 TDengine 数据类型与 JDBC 数据类型转换时，是否指定了正确的 TDengine 数据类型。| +| 0x2319 |user is required |创建连接时缺少用户名信息。| +| 0x231a |password is required |创建连接时缺少密码信息。| +| 0x231d |can't create connection with server within |通过增加参数 httpConnectTimeout 增加连接耗时，或是请检查与 taosAdapter 之间的连接情况。| +| 0x231e |failed to complete the task within the specified time |通过增加参数 messageWaitTimeout 增加执行耗时，或是请检查与 taosAdapter 之间的连接情况。| +| 0x2352 |Unsupported encoding |本地连接下指定了不支持的字符编码集。| +| 0x2353 |internal error of database, please see taoslog for more details |本地连接执行 prepareStatement 时出现错误，请检查 taos log 进行问题定位。| +| 0x2354 |connection is NULL |本地连接执行命令时，Connection 已经关闭。请检查与 TDengine 的连接情况。| +| 0x2355 |result set is NULL |本地连接获取结果集，结果集异常，请检查连接情况，并重试。| +| 0x2356 |invalid num of fields |本地连接获取结果集的 meta 信息不匹配。| +| 0x2357 |empty sql string |填写正确的 SQL 进行执行。| +| 0x2371 |consumer properties must not be null! |创建订阅时参数为空，请填写正确的参数。| +| 0x2375 |topic reference has been destroyed |创建数据订阅过程中，topic 引用被释放。请检查与 TDengine 的连接情况。| +| 0x2376 |failed to set consumer topic, topic name is empty |创建数据订阅过程中，订阅 topic 名称为空。请检查指定的 topic 名称是否填写正确。| +| 0x2377 |consumer reference has been destroyed |订阅数据传输通道已经关闭，请检查与 TDengine 的连接情况。| +| 0x2378 |consumer create error |创建数据订阅失败，请根据错误信息检查 taos log 进行问题定位。| +| 0x237a |vGroup not found in result set VGroup |没有分配给当前 consumer，由于 Rebalance 机制导致 Consumer 与 VGroup 不是绑定的关系。| + +## 数据类型映射 +TDengine 目前支持时间戳、数字、字符、布尔类型，与 Flink RowData Type 对应类型转换如下： + +| TDengine DataType | Flink RowDataType | +| ----------------- | ------------------ | +| TIMESTAMP | TimestampData | +| INT | Integer | +| BIGINT | Long | +| FLOAT | Float | +| DOUBLE | Double | +| SMALLINT | Short | +| TINYINT | Byte | +| BOOL | Boolean | +| BINARY | byte[] | +| NCHAR | StringData | +| JSON | StringData | +| VARBINARY | byte[] | +| GEOMETRY | byte[] | + +## 使用说明 + +### Flink 语义选择说明 + +采用 At-Least-Once（至少一次）语义原因： + - TDengine 目前不支持事务，不能进行频繁的检查点操作和复杂的事务协调。 + - 由于 TDengine 采用时间戳作为主键，重复数据下游算子可以进行过滤操作，避免重复计算。 + - 采用 At-Least-Once（至少一次）确保达到较高的数据处理的性能和较低的数据延时，设置方式如下： + ```java + StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); + env.enableCheckpointing(5000); + env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE); + ``` + +如果使用 Maven 管理项目，只需在 pom.xml 中加入以下依赖。 + +```xml + + com.taosdata.flink + flink-connector-tdengine + 1.0.0 + +``` + +### 连接参数 + +建立连接的参数有 URL 和 Properties。 +URL 规范格式为： +`jdbc:TAOS-WS://[host_name]:[port]/[database_name]?[user={user}|&password={password}|&timezone={timezone}]` + +参数说明： +- user：登录 TDengine 用户名，默认值 'root'。 +- password：用户登录密码，默认值 'taosdata'。 +- batchErrorIgnore：true：在执行 Statement 的 executeBatch 时，如果中间有一条 SQL 执行失败，继续执行下面的 SQL 了。false：不再执行失败 SQL 后的任何语句。默认值为：false。 +- httpConnectTimeout: 连接超时时间，单位 ms， 默认值为 60000。 +- messageWaitTimeout: 消息超时时间，单位 ms， 默认值为 60000。 +- useSSL: 连接中是否使用 SSL。 + +### Source + +Source 拉取 TDengine 数据库中的数据，并将获取到的数据转换为 Flink 内部可处理的格式和类型，并以并行的方式进行读取和分发，为后续的数据处理提供高效的输入。 +通过设置数据源的并行度，实现多个线程并行地从数据源中读取数据，提高数据读取的效率和吞吐量，充分利用集群资源进行大规模数据处理能力。 + +#### Source Properties + +Properties 中配置参数如下： + +| 参数名称 | 类型 | 参数说明 | 备注 | +| ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'| | +| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'| | +| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径| +| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineSource 对象时需要指定数据类型为 SourceRecords\<类型\> | 此处的类型为用下游算子接收数据的类型| +| TDengineConfigParams.PROPERTY_KEY_CHARSET | string | 客户端使用的字符集，默认值为系统字符集。| | +| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false|| +| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效| +| TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| + +#### 按时间分片 + +用户可以对查询的 SQL 按照时间拆分为多个子任务，输入：开始时间，结束时间，拆分间隔，时间字段名称，系统会按照设置的间隔（时间左闭右开）进行拆分并行获取数据。 + +```java +{{#include docs/examples/flink/Main.java:time_interval}} +``` + +#### 按超级表 TAG 分片 + +用户可以按照超级表的 TAG 字段将查询的 SQL 拆分为多个查询条件，系统会以一个查询条件对应一个子任务的方式对其进行拆分，进而并行获取数据。 + +```java +{{#include docs/examples/flink/Main.java:tag_split}} +``` + +#### 按表名分片 + +支持输入多个相同表结构的超级表或普通表进行分片，系统会按照一个表一个任务的方式进行拆分，进而并行获取数据。 + +```java +{{#include docs/examples/flink/Main.java:table_split}} +``` + +#### 使用 Source 连接器 + +查询结果为 RowData 数据类型示例： + +
+RowData Source +```java +{{#include docs/examples/flink/Main.java:source_test}} +``` +
+ +批量查询结果示例： + +
+Batch Source +```java +{{#include docs/examples/flink/Main.java:source_batch_test}} +``` +
+ +查询结果为自定义数据类型示例： + +
+Custom Type Source +```java +{{#include docs/examples/flink/Main.java:source_custom_type_test}} +``` +
+ +- ResultBean 自定义的一个内部类，用于定义 Source 查询结果的数据类型。 +- ResultSoureDeserialization 是自定义的一个内部类，通过继承 TdengineRecordDeserialization 并实现 convert 和 getProducedType 方法。 + +### CDC 数据订阅 + +Flink CDC 主要用于提供数据订阅功能，能实时监控 TDengine 数据库的数据变化，并将这些变更以数据流形式传输到 Flink 中进行处理，同时确保数据的一致性和完整性。 + +#### 参数说明 +| 参数名称 | 类型 | 参数说明 | 备注 | +| ----------------------- | :-----: | ------------------------- | -------------------------- | +| TDengineCdcParams.BOOTSTRAP_SERVERS| string | 服务端的 IP 地址 | | +| TDengineCdcParams.CONNECT_USER| string | 用户名 | | +| TDengineCdcParams.CONNECT_PASS| string | 密码 | | +| TDengineCdcParams.POLL_INTERVAL_MS|int| 拉取数据间隔, 默认 500ms| | +| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径| +| TDengineCdcParams.TMQ_BATCH_MODE| boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineCdcSource 对象时需要指定数据类型为 ConsumerRecords\<类型\> | 此处的类型为用下游算子接收数据的类型| +| TDengineCdcParams.GROUP_ID| string | 消费组 ID，同一消费组共享消费进度 |
**必填项**。最大长度：192。
每个 topic 最多可建立 100 个 consumer +| TDengineCdcParams.AUTO_OFFSET_RESET| string | 消费组订阅的初始位置 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅| +| TDengineCdcParams.ENABLE_AUTO_COMMIT| boolean | 是否自动提交，true: 启用（用于下游均为无状态算子） ；false：由 checkpoint 触发 commit | 默认 false| +| TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS| integer|消费记录自动提交消费位点时间间隔，单位为毫秒| 默认值为 5000, 此参数在 AUTO_OFFSET_RESET 为 true 生效| +| TDengineCdcParams.TMQ_SESSION_TIMEOUT_MS| integer | consumer 心跳丢失后超时时间，超时后会触发 rebalance 逻辑，成功后该 consumer 会被删除（从 TDengine 3.3.3.0 版本开始支持）| 默认值为 12000，取值范围 [6000， 1800000] | +| TDengineCdcParams.TMQ_MAX_POLL_INTERVAL_MS| integer | consumer poll 拉取数据间隔的最长时间，超过该时间，会认为该 consumer 离线，触发 rebalance 逻辑，成功后该 consumer 会被删除（从 3.3.3.0 版本开始支持） | 默认值为 300000，[1000，INT32_MAX] + +#### 使用 CDC 连接器 + +CDC 连接器会根据用户设置的并行度进行创建 consumer，因此用户根据资源情况合理设置并行度。 + +订阅结果为 RowData 数据类型示例： + +
+CDC Source +```java +{{#include docs/examples/flink/Main.java:cdc_source}} +``` +
+ +批量查询结果示例： + +
+CDC Batch Source +```java +{{#include docs/examples/flink/Main.java:cdc_batch_source}} +``` +
+ +查询结果为自定义数据类型示例： + +
+CDC Custom Type +```java +{{#include docs/examples/flink/Main.java:cdc_custom_type_test}} +``` +
+ +- ResultBean 是自定义的一个内部类，其字段名和数据类型与列的名称和数据类型一一对应，这样根据 value.deserializer 属性对应的反序列化类可以反序列化出 ResultBean 类型的对象。 + +### Sink + +Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自不同数据源或算子的数据写入 TDengine。在这一过程中，TDengine 所具备的高效写入机制发挥了至关重要的作用，有力保障了数据的快速和稳定存储。 + +#### Sink Properties + +| 参数名称 | 类型 | 参数说明 | 备注 | +| ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'| | +| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'| | +| TDengineConfigParams.PROPERTY_KEY_DBNAME| string | 写入的数据库名称|| +| TDengineConfigParams.TD_SUPERTABLE_NAME| string | 写入的超级表名称| 如果是超级表接收的数据必须有 tbname 字段，确定写入那张子表| +| TDengineConfigParams.TD_TABLE_NAME| string | 写入的表名，此参数和TD_SUPERTABLE_NAME 仅需要设置一个即可| 用于确定写入那张子表或普通表| +| TDengineConfigParams.TD_BATCH_SIZE| integer | 设置批大小 | 当到达批的数量后进行写入，或是一个checkpoint的时间也会触发写入数据库| +| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径| +| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于设置接收批量数据 | 如果设置为 True:
来源是 TDengine Source， 使用SourceRecords\<类型\> 创建 TDengineSink 对象
来源是 TDengine CDC 使用 ConsumerRecords\<类型\> 创建 TDengineSink 对象 | 此处的类型为接收数据的类型| +| TDengineConfigParams.TD_SOURCE_TYPE | string | 如果数据是表示数据来源是，source 或者 cdc 等 | TDengine source 设置为 "tdengine_source", TDengine cdc 设置为 "tdengine_cdc"| +| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false|| +| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效| +| TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| + +#### 使用 Sink 连接器 + +将接收的 RowData 类型数据写入 TDengine 示例： + +
+Sink RowData +```java +{{#include docs/examples/flink/Main.java:RowDataToSink}} +``` +
+ + +将批量接收的 RowData 类型数据写入 TDengine 示例： + +
+Sink RowData +```java +{{#include docs/examples/flink/Main.java:BatchRowDataToSink}} +``` +
+ +### Table SQL + +数据处理 ETL（Extract，Transform，Load）：可以使用 Flink SQL With JDBC 从多个不同的数据源数据库（如 TDengine、MySQL、Oracle 等）中提取数据，在 Flink 中进行转换操作（如数据清洗、格式转换、关联不同表的数据等），然后将处理后的结果加载到目标数据源（如 TDengine、Mysql 等）中。 + + +#### Source 连接器 + +参数配置说明： + +| 参数名称 | 类型 | 参数说明 | 备注 | +| ----------------------- | :-----: | ------------ | ------ | +| connector | string | 连接器标识，设置 tdengine-connector || +| td.jdbc.url| string | 连接的 url | | +| td.jdbc.mode | strng | 连接器类型, 设置 source, cdc, sink| | +| table.name| string| 原表或目标表名称| | +| scan.query| string| 获取数据的 SQL 语句|| +| sink.db.name|string| 目标数据库名称|| +| sink.supertable.name|string |写入的超级表名称|| +| sink.batch.size | integer | 写入的批大小|| +| sink.table.name|string|写入的普通表或子表名称|| + +#### Source 连接器使用示例 + +
+Table Source +```java +{{#include docs/examples/flink/Main.java:source_table}} +``` +
+ +#### CDC 连接器 + +参数配置说明： + +| 参数名称 | 类型 | 参数说明 | 备注 | +| ----------------------- | :-----: | ------------ |-------| +| connector | string | 连接器标识，设置 tdengine-connector || +| user| string | 用户名， 默认 root| | +| password | string | 密码， 默认taosdata| | +| bootstrap.servers| string | 服务器地址 | | +| topic | string | 订阅主题 || +| td.jdbc.mode | strng | 连接器类型, cdc, sink| | +| group.id| string| 消费组 ID，同一消费组共享消费进度 | | +| auto.offset.reset| string| 消费组订阅的初始位置 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅| +| poll.interval_ms| integer| 拉取数据间隔, 默认 500ms| | +| sink.db.name|string| 目标数据库名称|| +| sink.supertable.name|string |写入的超级表名称|| +| sink.batch.size | integer | 写入的批大小|| +| sink.table.name|string|写入的普通表或子表名称|| + + + +#### CDC 连接器使用示例 + +
+Table CDC +```java +{{#include docs/examples/flink/Main.java:cdc_table}} +``` +
From 85aad90d3a64d0ab818646be0811d4b5c699db27 Mon Sep 17 00:00:00 2001 From: wangmm0220 Date: Tue, 31 Dec 2024 11:22:56 +0800 Subject: [PATCH 41/73] fix:[TD-33396]add log for tmq --- source/dnode/vnode/src/tq/tqRead.c | 32 ++++++++++++++---------------- source/dnode/vnode/src/tq/tqScan.c | 13 +++++------- source/dnode/vnode/src/tq/tqUtil.c | 8 ++++---- 3 files changed, 24 insertions(+), 29 deletions(-) diff --git a/source/dnode/vnode/src/tq/tqRead.c b/source/dnode/vnode/src/tq/tqRead.c index 61b59e4cd0..937ca80bcc 100644 --- a/source/dnode/vnode/src/tq/tqRead.c +++ b/source/dnode/vnode/src/tq/tqRead.c @@ -500,7 +500,7 @@ int32_t tqReaderSetSubmitMsg(STqReader* pReader, void* msgStr, int32_t msgLen, i pReader->msg.msgLen = msgLen; pReader->msg.ver = ver; - tqDebug("tq reader set msg %p %d", msgStr, msgLen); + tqDebug("tq reader set msg pointer:%p, msg len:%d", msgStr, msgLen); SDecoder decoder = {0}; tDecoderInit(&decoder, pReader->msg.msgStr, pReader->msg.msgLen); @@ -539,31 +539,29 @@ bool tqNextBlockImpl(STqReader* pReader, const char* idstr) { int32_t code = false; int32_t lino = 0; int64_t uid = 0; - TSDB_CHECK_NULL(pReader, code, lino, END, false); TSDB_CHECK_NULL(pReader->msg.msgStr, code, lino, END, false); TSDB_CHECK_NULL(pReader->tbIdHash, code, lino, END, true); - int32_t numOfBlocks = taosArrayGetSize(pReader->submit.aSubmitTbData); - while (pReader->nextBlk < numOfBlocks) { - tqDebug("try next data block, len:%d ver:%" PRId64 " index:%d/%d, %s", pReader->msg.msgLen, pReader->msg.ver, (pReader->nextBlk + 1), numOfBlocks, idstr); - + int32_t blockSz = taosArrayGetSize(pReader->submit.aSubmitTbData); + while (pReader->nextBlk < blockSz) { SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); TSDB_CHECK_NULL(pSubmitTbData, code, lino, END, false); uid = pSubmitTbData->uid; void* ret = taosHashGet(pReader->tbIdHash, &pSubmitTbData->uid, sizeof(int64_t)); TSDB_CHECK_CONDITION(ret == NULL, code, lino, END, true); - tqDebug("discard submit block, uid:%" PRId64 ", total queried tables:%d continue %s", pSubmitTbData->uid, taosHashGetSize(pReader->tbIdHash), idstr); + tqDebug("iterator data block in hash continue, progress:%d/%d, total queried tables:%d, uid:%"PRId64, pReader->nextBlk, blockSz, taosHashGetSize(pReader->tbIdHash), uid); pReader->nextBlk++; } tDestroySubmitReq(&pReader->submit, TSDB_MSG_FLG_DECODE); pReader->nextBlk = 0; pReader->msg.msgStr = NULL; + tqDebug("iterator data block end, block progress:%d/%d, uid:%"PRId64, pReader->nextBlk, blockSz, uid); END: - tqDebug("%s:%d, uid:%"PRId64",code:%d", __FUNCTION__, lino, uid, code); + tqDebug("%s:%d return:%s, uid:%"PRId64, __FUNCTION__, lino, code?"true":"false", uid); return code; } @@ -583,18 +581,17 @@ bool tqNextDataBlockFilterOut(STqReader* pReader, SHashObj* filterOutUids) { uid = pSubmitTbData->uid; void* ret = taosHashGet(filterOutUids, &pSubmitTbData->uid, sizeof(int64_t)); TSDB_CHECK_NULL(ret, code, lino, END, true); - + tqDebug("iterator data block in hash continue, progress:%d/%d, uid:%" PRId64 "", pReader->nextBlk, blockSz, uid); pReader->nextBlk++; - tqDebug("discard submit block, uid:%" PRId64 ", continue", pSubmitTbData->uid); } tDestroySubmitReq(&pReader->submit, TSDB_MSG_FLG_DECODE); pReader->nextBlk = 0; pReader->msg.msgStr = NULL; - tqDebug("all data blocks are filtered out"); + tqDebug("iterator data block end, block progress:%d/%d, uid:%"PRId64, pReader->nextBlk, blockSz, uid); END: - tqDebug("%s:%d, uid:%"PRId64",code:%d", __FUNCTION__, lino, uid, code); + tqDebug("%s:%d return:%s, uid:%"PRId64, __FUNCTION__, lino, code?"true":"false", uid); return code; } @@ -856,7 +853,7 @@ int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* END: if (code != 0) { - tqError("tqRetrieveDataBlock failed, line:%d, code:%d", line, code); + tqError("tqRetrieveDataBlock failed, line:%d, msg:%s", line, tstrerror(code)); } taosMemoryFreeClear(pTSchema); return code; @@ -937,7 +934,7 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData TQ_NULL_GO_TO_END(pCol); int32_t numOfRows = pCol->nVal; int32_t numOfCols = taosArrayGetSize(pCols); - tqDebug("vgId:%d, tqProcessColData, col num: %d, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols, numOfRows); + tqDebug("vgId:%d, tqProcessColData start, col num: %d, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols, numOfRows); for (int32_t i = 0; i < numOfRows; i++) { bool buildNew = false; @@ -977,7 +974,7 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData } SSDataBlock* pLastBlock = taosArrayGetLast(blocks); pLastBlock->info.rows = curRow - lastRow; - + tqDebug("vgId:%d, tqProcessColData end, col num: %d, rows:%d, block num:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols, numOfRows, (int)taosArrayGetSize(blocks)); END: if (code != TSDB_CODE_SUCCESS) { tqError("vgId:%d, process col data failed, code:%d", pReader->pWalReader->pWal->cfg.vgId, code); @@ -1000,7 +997,7 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra int32_t numOfRows = taosArrayGetSize(pRows); pTSchema = tBuildTSchema(pSchemaWrapper->pSchema, pSchemaWrapper->nCols, pSchemaWrapper->version); TQ_NULL_GO_TO_END(pTSchema); - tqDebug("vgId:%d, tqProcessRowData, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfRows); + tqDebug("vgId:%d, tqProcessRowData start, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfRows); for (int32_t i = 0; i < numOfRows; i++) { bool buildNew = false; @@ -1039,6 +1036,7 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra SSDataBlock* pLastBlock = taosArrayGetLast(blocks); pLastBlock->info.rows = curRow - lastRow; + tqDebug("vgId:%d, tqProcessRowData end, rows:%d, block num:%d", pReader->pWalReader->pWal->cfg.vgId, numOfRows, (int)taosArrayGetSize(blocks)); END: if (code != TSDB_CODE_SUCCESS) { tqError("vgId:%d, process row data failed, code:%d", pReader->pWalReader->pWal->cfg.vgId, code); @@ -1049,7 +1047,7 @@ END: } int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas, SSubmitTbData** pSubmitTbDataRet, int64_t *createTime) { - tqDebug("tq reader retrieve data block %p, %d", pReader->msg.msgStr, pReader->nextBlk); + tqDebug("tq reader retrieve data block msg pointer:%p, index:%d", pReader->msg.msgStr, pReader->nextBlk); SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); if (pSubmitTbData == NULL) { return terrno; diff --git a/source/dnode/vnode/src/tq/tqScan.c b/source/dnode/vnode/src/tq/tqScan.c index ec200ba144..b4dc610a6a 100644 --- a/source/dnode/vnode/src/tq/tqScan.c +++ b/source/dnode/vnode/src/tq/tqScan.c @@ -39,7 +39,7 @@ int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp, int32_t TSDB_CHECK_NULL(taosArrayPush(pRsp->blockDataLen, &actualLen), code, lino, END, terrno); TSDB_CHECK_NULL(taosArrayPush(pRsp->blockData, &buf), code, lino, END, terrno); - tqDebug("add block data to response success:%p, blockDataLen:%d, blockData:%p", pRsp->blockDataLen, actualLen, pRsp->blockData); + tqDebug("add block data to block array, blockDataLen:%d, blockData:%p", actualLen, buf); END: if (code != TSDB_CODE_SUCCESS) { taosMemoryFree(buf); @@ -332,7 +332,7 @@ END: return code; } -static int32_t tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded){ +static void tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded){ int32_t code = 0; int32_t lino = 0; SArray* pBlocks = NULL; @@ -383,7 +383,7 @@ static int32_t tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, } pRsp->blockNum++; } - tqDebug("vgId:%d, process sub data success", pTq->pVnode->config.vgId); + tqDebug("vgId:%d, process sub data success, response blocknum:%d, rows:%d", pTq->pVnode->config.vgId, pRsp->blockNum, *totalRows); END: if (code != 0){ tqError("%s failed at %d, failed to process sub data:%s", __FUNCTION__, lino, tstrerror(code)); @@ -393,7 +393,6 @@ END: taosArrayDestroy(pBlocks); taosArrayDestroy(pSchemas); } - return code; } int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SPackedData submit, SMqDataRsp* pRsp, int32_t* totalRows, int8_t sourceExcluded) { @@ -410,13 +409,11 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SPackedData submit, SMqData if (pExec->subType == TOPIC_SUB_TYPE__TABLE) { while (tqNextBlockImpl(pReader, NULL)) { - code = tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); - TSDB_CHECK_CODE(code, lino, END); + tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); } } else if (pExec->subType == TOPIC_SUB_TYPE__DB) { while (tqNextDataBlockFilterOut(pReader, pExec->execDb.pFilterOutTbUid)) { - code = tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); - TSDB_CHECK_CODE(code, lino, END); + tqProcessSubData(pTq, pHandle, pRsp, totalRows, sourceExcluded); } } diff --git a/source/dnode/vnode/src/tq/tqUtil.c b/source/dnode/vnode/src/tq/tqUtil.c index 2c27f7e515..6ecb1b1b4d 100644 --- a/source/dnode/vnode/src/tq/tqUtil.c +++ b/source/dnode/vnode/src/tq/tqUtil.c @@ -279,8 +279,8 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, } SWalCont* pHead = &pHandle->pWalReader->pHead->head; - tqDebug("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, vgId:%d offset %" PRId64 " msgType %d", - pRequest->consumerId, pRequest->epoch, vgId, fetchVer, pHead->msgType); + tqDebug("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, vgId:%d offset %" PRId64 " msgType %s", + pRequest->consumerId, pRequest->epoch, vgId, fetchVer, TMSG_INFO(pHead->msgType)); // process meta if (pHead->msgType != TDMT_VND_SUBMIT) { @@ -385,7 +385,7 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, END: if (code != 0){ - tqError("tmq poll: tqTaosxScanLog error %" PRId64 ", in vgId:%d, subkey %s", pRequest->consumerId, vgId, + tqError("tmq poll: tqTaosxScanLog error. consumerId:0x%" PRIx64 ", in vgId:%d, subkey %s", pRequest->consumerId, vgId, pRequest->subKey); } tDeleteMqBatchMetaRsp(&btMetaRsp); @@ -427,7 +427,7 @@ int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequ END: if (code != 0){ - uError("failed to extract data for mq, code:%d", code); + uError("failed to extract data for mq, msg:%s", tstrerror(code)); } tOffsetDestroy(&reqOffset); return code; From 77566ebc29d248afb331701c891bdb89ce74afe5 Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Tue, 31 Dec 2024 11:26:32 +0800 Subject: [PATCH 42/73] enh: add pkg manager for java/go --- packaging/setup_env.sh | 122 ++++++++++++++++++++++++++++++++--------- 1 file changed, 96 insertions(+), 26 deletions(-) diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh index 3899ddc4fd..30e3b9cda3 100644 --- a/packaging/setup_env.sh +++ b/packaging/setup_env.sh @@ -154,8 +154,11 @@ help() { echo " disable_service - Disable specified services" echo " install_python - Install Python and pip" echo " install_java - Install Java" - echo " install_maven - Install Maven" + echo " install_java_via_sdkman - Install Java via sdkman" + echo " install_maven_via_sdkman - Install Maven via sdkman" echo " deploy_go - Deploy Go environment" + echo " install_gvm - Install GVM" + echo " install_go_via_gvm - Install Go via GVM" echo " deploy_rust - Deploy Rust environment" echo " install_node - Install Node via package manager or binary" echo " install_node_via_nvm - Install Node via NVM" @@ -793,7 +796,7 @@ update_redhat_gcc() { update_redhat_tmux() { echo "Downloading the latest version of tmux..." cd /usr/local/src || exit - latest_tmux_version=$(curl -s https://api.github.com/repos/tmux/tmux/releases/latest | grep -Po '"tag_name": "\K.*?(?=")') + latest_tmux_version=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s https://api.github.com/repos/tmux/tmux/releases/latest | grep -Po '"tag_name": "\K.*?(?=")') wget https://github.com/tmux/tmux/releases/download/"${latest_tmux_version}"/tmux-"${latest_tmux_version}".tar.gz echo "Extracting tmux ${latest_tmux_version}..." @@ -845,6 +848,7 @@ deploy_tmux() { # } # Install Java +# shellcheck disable=SC2120 install_java() { echo -e "${YELLOW}Installing Java...${NO_COLOR}" # Specify the major JDK version to search for; default is set to 17 if not specified @@ -933,33 +937,62 @@ install_java() { # Install sdkman install_sdkman() { + echo -e "${YELLOW}Installing SDKMAN...${NO_COLOR}" install_package zip unzip if [ -d "$HOME/.sdkman" ]; then echo -e "${GREEN}SDKMAN is already installed.${NO_COLOR}" else echo -e "${YELLOW}Installing SDKMAN...${NO_COLOR}" - curl -s "https://get.sdkman.io" | bash + curl --retry 10 --retry-delay 5 --retry-max-time 120 -s "https://get.sdkman.io" | bash fi +} +# Install gvm +install_gvm() { + echo -e "${YELLOW}Installing GVM...${NO_COLOR}" + install_package bison gcc make + bash < <(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer) + source $HOME/.gvm/scripts/gvm + gvm version + check_status "Failed to install GVM" "GVM installed successfully." $? + add_config_if_not_exist "export GO111MODULE=on" "$BASH_RC" + add_config_if_not_exist "export GOPROXY=https://goproxy.cn,direct" "$BASH_RC" + add_config_if_not_exist "export GO_BINARY_BASE_URL=https://mirrors.aliyun.com/golang/" "$BASH_RC" + add_config_if_not_exist "export GOROOT_BOOTSTRAP=$GOROOT" "$BASH_RC" + SOURCE_RESULTS+="GVM: source $HOME/.gvm/scripts/gvm\n" } # Install Maven # shellcheck disable=SC2120 -install_maven() { +install_maven_via_sdkman() { echo -e "${YELLOW}Installing maven...${NO_COLOR}" if [ -n "$1" ]; then DEFAULT_MVN_VERSION="$1" install_sdkman - if [ -f "$HOME/.sdkman/bin/sdkman-init.sh" ]; then - source "$HOME/.sdkman/bin/sdkman-init.sh" - fi + [[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh" # 3.2.5 - sdk install maven "$DEFAULT_MVN_VERSION" + yes | sdk install maven "$DEFAULT_MVN_VERSION" else install_package "maven" fi + [[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh" mvn -version check_status "Failed to install maven" "Maven installed successfully." $? +} + +install_java_via_sdkman() { + echo -e "${YELLOW}Installing java...${NO_COLOR}" + if [ -n "$1" ]; then + DEFAULT_JDK_VERSION="$1" + else + DEFAULT_JDK_VERSION="17" + fi + install_sdkman + [[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh" + yes | sdk install java "$DEFAULT_JDK_VERSION-open" + [[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh" + java -version + check_status "Failed to install java" "Java installed successfully." $? SOURCE_RESULTS+="Sdkman: source $HOME/.sdkman/bin/sdkman-init.sh\n" } @@ -999,6 +1032,29 @@ deploy_go() { SOURCE_RESULTS+="Golang: source $BASH_RC\n" } +# Install Go via gvm +install_go_via_gvm() { + echo -e "${YELLOW}Installing Go...${NO_COLOR}" + if [ -n "$1" ]; then + DEFAULT_GO_VERSION="$1" + else + DEFAULT_GO_VERSION="1.23.0" + fi + install_gvm + export GO111MODULE=on + export GOPROXY=https://goproxy.cn,direct + export GO_BINARY_BASE_URL=https://mirrors.aliyun.com/golang/ + export GOROOT_BOOTSTRAP=$GOROOT + + gvm install go"$DEFAULT_GO_VERSION" -B + gvm use go"$DEFAULT_GO_VERSION" + gvm use go"$DEFAULT_GO_VERSION" --default + + go version + check_status "Failed to install Go" "Go installed successfully." $? + SOURCE_RESULTS+="Golang: source $BASH_RC\n" +} + # Function to install Rust and Cargo deploy_rust() { # Check if Rust is already installed @@ -1074,7 +1130,7 @@ install_node_in_ubuntu18.04() { NODE_DISTRO="node-v$DEFAULT_NODE_VERSION-linux-x64" update_ubuntu_gcc_18.04 echo "Installing Node..." - curl -O https://nodejs.org/dist/v22.0.0/node-v22.0.0.tar.gz + curl --retry 10 --retry-delay 5 --retry-max-time 120 -O https://nodejs.org/dist/v22.0.0/node-v22.0.0.tar.gz tar -xzf node-v22.0.0.tar.gz cd node-v22.0.0 || exit ./configure @@ -1126,8 +1182,8 @@ install_node_via_nvm () { # Install NVM if ! command -v nvm &> /dev/null; then - NVM_VERSION=$(curl -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")') - curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/"$NVM_VERSION"/install.sh | bash + NVM_VERSION=$(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s https://api.github.com/repos/nvm-sh/nvm/releases/latest | grep -oP '"tag_name": "\K(.*)(?=")') + curl --retry 10 --retry-delay 5 --retry-max-time 120 -o- https://raw.githubusercontent.com/nvm-sh/nvm/"$NVM_VERSION"/install.sh | bash export NVM_DIR="$HOME/.nvm" [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" @@ -1668,12 +1724,16 @@ config_cloud_init() { } cleanup() { - echo -e "${GREEN}===========================================\n${NO_COLOR}" - echo -e "${GREEN}Installation complete! \n${NO_COLOR}" - echo -e "${GREEN}Some tools require you to manually source${NO_COLOR}" - echo -e "${GREEN}or restart your terminal to take effect.\n${NO_COLOR}" - echo -e "${GREEN}===========================================\n${NO_COLOR}" - echo -e "${GREEN}$SOURCE_RESULTS${NO_COLOR}" + if [ -n "$SOURCE_RESULTS" ]; then + echo -e "${YELLOW}===========================================\n${NO_COLOR}" + echo -e "${YELLOW}Installation complete! \n${NO_COLOR}" + echo -e "${YELLOW}Some tools require you to manually source${NO_COLOR}" + echo -e "${YELLOW}or restart your terminal to take effect.\n${NO_COLOR}" + echo -e "${YELLOW}===========================================\n${NO_COLOR}" + echo -e "${YELLOW}$SOURCE_RESULTS${NO_COLOR}" + else + echo -e "${YELLOW}Installation complete \n${NO_COLOR}" + fi } # Clone a repository with a specified target directory @@ -1773,16 +1833,16 @@ clone_repos() { new_funcs() { echo "Adding test..." install_python 3.10.12 - install_java 21 + install_java_via_sdkman 21.0.2 install_node 16.20.2 - install_maven 3.2.5 + install_maven_via_sdkman 3.2.5 deploy_rust } # deploy TDasset TDasset() { - install_java 21 - install_maven 3.9.9 + install_java_via_sdkman 21.0.2 + install_maven_via_sdkman 3.9.9 # not supported in centos7/ubuntu18 because of the old version of glibc install_node_via_nvm 22.0.0 install_pnpm @@ -1790,9 +1850,10 @@ TDasset() { # deploy TDinternal/TDengine/taosx TDinternal() { - deploy_go + install_go_via_gvm 1.23.3 deploy_rust - install_java 17 + install_java_via_sdkman 17 + install_maven_via_sdkman 3.9.9 install_node_via_nvm 16.20.2 install_python 3.10.12 } @@ -1874,7 +1935,7 @@ deploy_dev() { install_python install_pip_pkg install_java - install_maven + install_maven_via_sdkman deploy_go deploy_rust install_node @@ -1966,8 +2027,11 @@ main() { install_java) install_java ;; - install_maven) - install_maven + install_java_via_sdkman) + install_java_via_sdkman + ;; + install_maven_via_sdkman) + install_maven_via_sdkman ;; deploy_cmake) deploy_cmake @@ -1984,6 +2048,12 @@ main() { deploy_go) deploy_go ;; + install_gvm) + install_gvm + ;; + install_go_via_gvm) + install_go_via_gvm + ;; deploy_rust) deploy_rust ;; From 5a1e5b1549e8c50f97c6e1ee52071e78ec93b89b Mon Sep 17 00:00:00 2001 From: Haojun Liao Date: Tue, 31 Dec 2024 11:52:06 +0800 Subject: [PATCH 43/73] fix(stream): check return value for snprintf --- source/libs/stream/src/streamSnapshot.c | 86 +++++++++++++------------ 1 file changed, 44 insertions(+), 42 deletions(-) diff --git a/source/libs/stream/src/streamSnapshot.c b/source/libs/stream/src/streamSnapshot.c index 9bd0ab0ff2..cb4386067b 100644 --- a/source/libs/stream/src/streamSnapshot.c +++ b/source/libs/stream/src/streamSnapshot.c @@ -162,59 +162,61 @@ void snapFileDebugInfo(SBackendSnapFile2* pSnapFile) { return; } - int32_t nBytes = snprintf(buf + strlen(buf), cap, "["); - if (nBytes <= 0 || nBytes >= cap) { - taosMemoryFree(buf); - stError("%s failed to write buf, reason:%s", STREAM_STATE_TRANSFER, tstrerror(TSDB_CODE_OUT_OF_RANGE)); - return; - } - - int32_t wlen = 0; int32_t len = 0; - if (pSnapFile->pCurrent) { - len = snprintf(buf, cap,"current: %s,", pSnapFile->pCurrent); - if (len > 0) { - wlen += len; - } else { - stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); - } - } + int32_t wlen = 1; - if (pSnapFile->pMainfest) { - len = snprintf(buf + wlen, cap - wlen, "MANIFEST: %s,", pSnapFile->pMainfest); - if (len > 0) { - wlen += len; - } else { - stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + do { + buf[0] = '['; + if (pSnapFile->pCurrent) { + len = snprintf(buf + wlen, cap - wlen, "current: %s,", pSnapFile->pCurrent); + if (len > 0 && len < (cap - wlen)) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + break; + } } - } - if (pSnapFile->pOptions) { - len = snprintf(buf + wlen, cap - wlen, "options: %s,", pSnapFile->pOptions); - if (len > 0) { - wlen += len; - } else { - stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + if (pSnapFile->pMainfest) { + len = snprintf(buf + wlen, cap - wlen, "MANIFEST: %s,", pSnapFile->pMainfest); + if (len > 0 && len < (cap - wlen)) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + break; + } } - } - if (pSnapFile->pSst) { - for (int32_t i = 0; i < taosArrayGetSize(pSnapFile->pSst); i++) { - char* name = taosArrayGetP(pSnapFile->pSst, i); - if (strlen(buf) + strlen(name) < cap) { - len = snprintf(buf + wlen, cap - wlen, "%s,", name); - if (len > 0) { - wlen += len; - } else { - stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + if (pSnapFile->pOptions) { + len = snprintf(buf + wlen, cap - wlen, "options: %s,", pSnapFile->pOptions); + if (len > 0 && len < (cap - wlen)) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + break; + } + } + + if (pSnapFile->pSst) { + for (int32_t i = 0; i < taosArrayGetSize(pSnapFile->pSst); i++) { + char* name = taosArrayGetP(pSnapFile->pSst, i); + if (strlen(buf) + strlen(name) < cap) { + len = snprintf(buf + wlen, cap - wlen, "%s,", name); + if (len > 0 && len < (cap - wlen)) { + wlen += len; + } else { + stError("%s failed to build buf for debug, code: out of buffer", STREAM_STATE_TRANSFER); + break; + } } } } - } + } while (0); - if ((strlen(buf)) < cap) { - buf[wlen++] = ']'; + if (wlen < cap) { + buf[wlen] = ']'; } + buf[cap - 1] = '\0'; stInfo("%s %" PRId64 "-%" PRId64 " get file list: %s", STREAM_STATE_TRANSFER, pSnapFile->snapInfo.streamId, pSnapFile->snapInfo.taskId, buf); From 1d0ab80e75a78445268acef2f3a310755f71227c Mon Sep 17 00:00:00 2001 From: Haojun Liao Date: Tue, 31 Dec 2024 11:59:13 +0800 Subject: [PATCH 44/73] fix(stream): check return value for snprintf --- source/libs/stream/src/streamSnapshot.c | 4 ++-- source/libs/stream/src/streamTask.c | 6 +++--- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/source/libs/stream/src/streamSnapshot.c b/source/libs/stream/src/streamSnapshot.c index cb4386067b..88495ac760 100644 --- a/source/libs/stream/src/streamSnapshot.c +++ b/source/libs/stream/src/streamSnapshot.c @@ -130,7 +130,7 @@ int32_t streamGetFileSize(char* path, char* name, int64_t* sz) { } ret = snprintf(fullname, len, "%s%s%s", path, TD_DIRSEP, name); - if (ret < 0) { + if (ret < 0 || ret >= len) { stError("%s failed to set the file path for get the file size, code: out of buffer", name); return TSDB_CODE_OUT_OF_BUFFER; } @@ -819,7 +819,7 @@ int32_t streamSnapWrite(SStreamSnapWriter* pWriter, uint8_t* pData, uint32_t nDa int32_t ret = snprintf(path, bufLen, "%s%s%s%s%s%s%s%" PRId64 "", pHandle->metaPath, TD_DIRSEP, idstr, TD_DIRSEP, "checkpoints", TD_DIRSEP, "checkpoint", snapInfo.chkpId); - if (ret < 0) { + if (ret < 0 || ret >= bufLen) { stError("s-task:0x%x failed to set the path for take snapshot, code: out of buffer, %s", (int32_t)snapInfo.taskId, pHandle->metaPath); return TSDB_CODE_OUT_OF_BUFFER; diff --git a/source/libs/stream/src/streamTask.c b/source/libs/stream/src/streamTask.c index e3a1f5f94a..d27ed520c6 100644 --- a/source/libs/stream/src/streamTask.c +++ b/source/libs/stream/src/streamTask.c @@ -134,7 +134,7 @@ int32_t tNewStreamTask(int64_t streamId, int8_t taskLevel, SEpSet* pEpset, bool char buf[128] = {0}; int32_t ret = snprintf(buf, tListLen(buf), "0x%" PRIx64 "-0x%x", pTask->id.streamId, pTask->id.taskId); - if (ret < 0) { + if (ret < 0 || ret >= tListLen(buf)) { stError("s-task:0x%x failed to set the taskIdstr, code: out of buffer", pTask->id.taskId); return TSDB_CODE_OUT_OF_BUFFER; } @@ -418,7 +418,7 @@ int32_t streamTaskSetBackendPath(SStreamTask* pTask) { } int32_t code = snprintf(pTask->backendPath, len + nBytes + 2, "%s%s%s", pTask->pMeta->path, TD_DIRSEP, id); - if (code < 0) { + if (code < 0 || code >= len + nBytes + 2) { stError("s-task:%s failed to set backend path:%s, code: out of buffer", pTask->id.idStr, pTask->backendPath); return TSDB_CODE_OUT_OF_BUFFER; } else { @@ -1138,7 +1138,7 @@ SEpSet* streamTaskGetDownstreamEpInfo(SStreamTask* pTask, int32_t taskId) { int32_t createStreamTaskIdStr(int64_t streamId, int32_t taskId, const char** pId) { char buf[128] = {0}; int32_t code = snprintf(buf, tListLen(buf),"0x%" PRIx64 "-0x%x", streamId, taskId); - if (code < 0) { + if (code < 0 || code >= tListLen(buf)) { return TSDB_CODE_OUT_OF_BUFFER; } From abbede02293766f54ed58c0b26442c0e7480720e Mon Sep 17 00:00:00 2001 From: Minglei Jin Date: Tue, 31 Dec 2024 11:21:00 +0800 Subject: [PATCH 45/73] fix(docs/s3): make OBS explicitly --- docs/zh/08-operation/12-multi.md | 28 ++++++++++++++++------------ 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/docs/zh/08-operation/12-multi.md b/docs/zh/08-operation/12-multi.md index 122fb9c2f3..994192e8fc 100644 --- a/docs/zh/08-operation/12-multi.md +++ b/docs/zh/08-operation/12-multi.md @@ -60,11 +60,15 @@ dataDir /mnt/data6 2 0 ## 对象存储 -本节介绍在 TDengine Enterprise 如何使用 S3 对象存储，本功能基于通用 S3 SDK 实现，对各个 S3 平台的访问参数进行了兼容适配，可以访问如 Amazon S3、Azure Blob、华为 OBS、腾讯云 COS、阿里云 OSS、minio等对象存储服务。通过适当的参数配置，可以把大部分较冷的时序数据存储到 S3 服务中。 +本节介绍在 TDengine Enterprise 版本中如何使用对象存储功能，如 Amazon S3、Azure Blob Storage、华为 OBS、腾讯云 COS、阿里云 OSS、MinIO 等对象存储服务。 **注意** 在配合多级存储使用时，每一级存储介质上保存的数据都有可能被按规则备份到远程对象存储中并删除本地数据文件。 -### 配置方式 +### S3 对象存储 + +本功能基于通用 S3 SDK 实现，对各个 S3 平台的访问参数进行了兼容适配，通过适当的参数配置，可以把大部分较冷的时序数据存储到 S3 服务中。 + +#### 配置方式 在配置文件 /etc/taos/taos.cfg 中，添加用于 S3 访问的参数： @@ -78,7 +82,7 @@ dataDir /mnt/data6 2 0 | s3MigrateIntervalSec | 本地数据文件自动上传 S3 的触发周期，单位为秒。最小值：600；最大值：100000。默认值 3600 | | s3MigrateEnabled | 是否自动进行 S3 迁移，默认值为 0，表示关闭自动 S3 迁移，可配置为 1。 | -### 检查配置参数可用性 +#### 检查配置参数可用性 在 taos.cfg 中完成对 S3 的配置后，通过 taosd 命令的 checks3 参数可以检查所配置的 S3 服务是否可用： @@ -88,7 +92,7 @@ taosd --checks3 如果配置的 S3 服务无法访问，此命令会在运行过程中输出相应的错误信息。 -### 创建使用 S3 的 DB +#### 创建使用 S3 的 DB 完成配置后，即可启动 TDengine 集群，创建使用 S3 的数据库，比如： @@ -112,11 +116,11 @@ s3migrate database ; | 2 | s3_chunkpages | 131072 | 131072 | 1048576 | 上传对象的大小阈值，与 tsdb_pagesize 参数一样，不可修改，单位为 TSDB 页 | | 3 | s3_compact | 1 | 0 | 1 | TSDB 文件组首次上传 S3 时，是否自动进行 compact 操作 | -### 对象存储读写次数估算 +#### 对象存储读写次数估算 对象存储服务的使用成本与存储的数据量及请求次数相关，下面分别介绍数据的上传及下载过程。 -#### 数据上传 +##### 数据上传 当 TSDB 时序数据超过 `s3_keeplocal` 参数指定的时间，相关的数据文件会被切分成多个文件块，每个文件块的默认大小是 512M 字节 (`s3_chunkpages * tsdb_pagesize`)。除了最后一个文件块保留在本地文件系统外，其余的文件块会被上传到对象存储服务。 @@ -128,7 +132,7 @@ s3migrate database ; 其它类型的文件如 head, stt, sma 等，保留在本地文件系统，以加速预计算相关查询。 -#### 数据下载 +##### 数据下载 在查询操作中，如果需要访问对象存储中的数据，TSDB 不会下载整个数据文件，而是计算所需数据在文件中的位置，只下载相应的数据到 TSDB 页缓存中，然后将数据返回给查询执行引擎。后续查询首先检查页缓存，查看数据是否已被缓存。如果数据已缓存，则直接使用缓存中的数据，而无需重复从对象存储下载，从而有效降低从对象存储下载数据的次数。 @@ -140,15 +144,15 @@ s3migrate database ; 页缓存是内存缓存，节点重启后，再次查询需要重新下载数据。缓存采用 LRU (Least Recently Used) 策略，当缓存空间不足时，最近最少使用的数据将被淘汰。缓存的大小可以通过 `s3PageCacheSize` 参数进行调整，通常来说，缓存越大，下载次数越少。 -## Azure Blob 存储 -本节介绍在 TDengine Enterprise 如何使用微软 Azure Blob 对象存储。本功能是上一小节‘对象存储’功能的扩展，需额外依赖 Flexify 服务提供的 S3 网关。通过适当的参数配置，可以把大部分较冷的时序数据存储到 Azure Blob 服务中。 +### Azure Blob 存储 +本节介绍在 TDengine Enterprise 版本中如何使用微软 Azure Blob 存储。本功能可以通过两个方式使用：利用 Flexify 服务提供的 S3 网关功能和不依赖 Flexify 服务。通过配置参数，可以把大部分较冷的时序数据存储到 Azure Blob 服务中。 -### Flexify 服务 +#### Flexify 服务 Flexify 是 Azure Marketplace 中的一款应用程序，允许兼容 S3 的应用程序通过标准 S3 API 在 Azure Blob Storage 中存储数据。可使用多个 Flexify 服务对同一个 Blob 存储建立多个 S3 网关。 部署方式请参考 [Flexify](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/flexify.azure-s3-api?tab=Overview) 应用页面说明。 -### 配置方式 +##### 配置方式 在配置文件 /etc/taos/taos.cfg 中，添加用于 S3 访问的参数： @@ -164,7 +168,7 @@ s3BucketName td-test - 在某一 S3 服务上操作失败后会切换至其他服务，全部服务都失败后将返回最后产生的错误码 - 最大支持的 S3 服务配置数为 10 -### 不依赖 Flexify 服务 +#### 不依赖 Flexify 服务 用户界面同 S3，不同的地方在于下面三个参数的配置： From 626dec071855ee09f0a4d106184b1b64d252a343 Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Tue, 31 Dec 2024 12:55:42 +0800 Subject: [PATCH 46/73] enh: add pkg manager for java/go/python --- packaging/setup_env.sh | 88 ++++++++++++++++++++++++++++++++---------- 1 file changed, 68 insertions(+), 20 deletions(-) diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh index 30e3b9cda3..07c5d45337 100644 --- a/packaging/setup_env.sh +++ b/packaging/setup_env.sh @@ -928,7 +928,7 @@ install_java() { INSTALLED_VERSION=$("$JAVA_HOME"/bin/java --version 2>&1) if echo "$INSTALLED_VERSION" | grep -q "openjdk $DEFAULT_JDK_VERSION"; then echo -e "${GREEN}Java installed successfully.${NO_COLOR}" - SOURCE_RESULTS+="Java: source /root/.bashrc\n" + SOURCE_RESULTS+="source /root/.bashrc # For OpenJDK\n" else echo -e "${YELLOW}Java version not match.${NO_COLOR}" exit 1 @@ -938,11 +938,11 @@ install_java() { # Install sdkman install_sdkman() { echo -e "${YELLOW}Installing SDKMAN...${NO_COLOR}" - install_package zip unzip if [ -d "$HOME/.sdkman" ]; then echo -e "${GREEN}SDKMAN is already installed.${NO_COLOR}" else echo -e "${YELLOW}Installing SDKMAN...${NO_COLOR}" + install_package zip unzip curl --retry 10 --retry-delay 5 --retry-max-time 120 -s "https://get.sdkman.io" | bash fi } @@ -950,16 +950,64 @@ install_sdkman() { # Install gvm install_gvm() { echo -e "${YELLOW}Installing GVM...${NO_COLOR}" - install_package bison gcc make - bash < <(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer) - source $HOME/.gvm/scripts/gvm - gvm version - check_status "Failed to install GVM" "GVM installed successfully." $? - add_config_if_not_exist "export GO111MODULE=on" "$BASH_RC" - add_config_if_not_exist "export GOPROXY=https://goproxy.cn,direct" "$BASH_RC" - add_config_if_not_exist "export GO_BINARY_BASE_URL=https://mirrors.aliyun.com/golang/" "$BASH_RC" - add_config_if_not_exist "export GOROOT_BOOTSTRAP=$GOROOT" "$BASH_RC" - SOURCE_RESULTS+="GVM: source $HOME/.gvm/scripts/gvm\n" + if [ -d "$HOME/.gvm" ]; then + echo -e "${GREEN}GVM is already installed.${NO_COLOR}" + else + install_package bison gcc make + bash < <(curl --retry 10 --retry-delay 5 --retry-max-time 120 -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer) + source $HOME/.gvm/scripts/gvm + gvm version + check_status "Failed to install GVM" "GVM installed successfully." $? + add_config_if_not_exist "export GO111MODULE=on" "$BASH_RC" + add_config_if_not_exist "export GOPROXY=https://goproxy.cn,direct" "$BASH_RC" + add_config_if_not_exist "export GO_BINARY_BASE_URL=https://mirrors.aliyun.com/golang/" "$BASH_RC" + add_config_if_not_exist "export GOROOT_BOOTSTRAP=$GOROOT" "$BASH_RC" + SOURCE_RESULTS+="source $HOME/.gvm/scripts/gvm # For GVM\n" + fi +} + +# Install pyvenv +install_pyenv() { + echo -e "${YELLOW}Installing Pyenv...${NO_COLOR}" + if [ -d "$HOME/.pyenv" ]; then + echo -e "${GREEN}Pyenv is already installed.${NO_COLOR}" + else + curl -L https://gitee.com/xinghuipeng/pyenv-installer/raw/master/bin/pyenv-installer | bash + export PATH="$HOME/.pyenv/bin:$PATH" + eval "$(pyenv init --path)" + eval "$(pyenv init -)" + add_config_if_not_exist "export PATH=\"\$HOME/.pyenv/bin:\$PATH\"" "$BASH_RC" + add_config_if_not_exist "eval \"\$(pyenv init --path)\"" "$BASH_RC" + add_config_if_not_exist "eval \"\$(pyenv init -)\"" "$BASH_RC" + pyenv --version + check_status "Failed to install Pyenv" "Pyenv installed successfully." $? + SOURCE_RESULTS+="source $BASH_RC For: Pyenv/python\n" + fi +} + +# Install python via pyenv +install_python_via_pyenv() { + echo -e "${YELLOW}Installing Python via Pyenv...${NO_COLOR}" + if [ -f /etc/debian_version ]; then + install_package gcc make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils tk-dev libffi-dev liblzma-dev + elif [ -f /etc/redhat-release ]; then + install_package gcc zlib zlib-devel libffi libffi-devel readline-devel openssl-devel openssl11 openssl11-devel + else + echo "Unsupported Linux distribution." + exit 1 + fi + + if [ -n "$1" ]; then + DEFAULT_PYTHON_VERSION="$1" + else + DEFAULT_PYTHON_VERSION="3.10.12" + fi + install_pyenv + + pyenv install "$DEFAULT_PYTHON_VERSION" + pyenv global "$DEFAULT_PYTHON_VERSION" + python --version + check_status "Failed to install Python" "Python installed successfully." $? } # Install Maven @@ -993,7 +1041,7 @@ install_java_via_sdkman() { [[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh" java -version check_status "Failed to install java" "Java installed successfully." $? - SOURCE_RESULTS+="Sdkman: source $HOME/.sdkman/bin/sdkman-init.sh\n" + SOURCE_RESULTS+="source $HOME/.sdkman/bin/sdkman-init.sh # For Sdkman/java/maven\n" } # Install Go @@ -1029,7 +1077,7 @@ deploy_go() { # Apply the environment variables $GO_INSTALL_DIR/bin/go version check_status "Failed to install GO" "Install GO successfully" $? - SOURCE_RESULTS+="Golang: source $BASH_RC\n" + SOURCE_RESULTS+="source $BASH_RC # For Golang\n" } # Install Go via gvm @@ -1052,7 +1100,7 @@ install_go_via_gvm() { go version check_status "Failed to install Go" "Go installed successfully." $? - SOURCE_RESULTS+="Golang: source $BASH_RC\n" + SOURCE_RESULTS+="source $BASH_RC # For Golang\n" } # Function to install Rust and Cargo @@ -1103,7 +1151,7 @@ deploy_rust() { # Install cargo-make cargo install cargo-make check_status "Failed to install Rust" "Install Rust successfully" $? - SOURCE_RESULTS+="Rust: source $BASH_RC && source $HOME/.cargo/env\n" + SOURCE_RESULTS+="source $BASH_RC && source $HOME/.cargo/env # For Rust\n" else echo "Rust is already installed." fi @@ -1188,7 +1236,7 @@ install_node_via_nvm () { [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" echo -e "${GREEN}NVM installed successfully.${NO_COLOR}" - SOURCE_RESULTS+="NVM: source $NVM_DIR/nvm.sh && source $NVM_DIR/bash_completion\n" + SOURCE_RESULTS+="source $NVM_DIR/nvm.sh && source $NVM_DIR/bash_completion # For NVM/node/npm/yarn/pnpm\n" else echo -e "${GREEN}NVM is already installed.${NO_COLOR}" fi @@ -1832,7 +1880,7 @@ clone_repos() { new_funcs() { echo "Adding test..." - install_python 3.10.12 + install_python_via_pyenv 3.10.12 install_java_via_sdkman 21.0.2 install_node 16.20.2 install_maven_via_sdkman 3.2.5 @@ -1855,12 +1903,12 @@ TDinternal() { install_java_via_sdkman 17 install_maven_via_sdkman 3.9.9 install_node_via_nvm 16.20.2 - install_python 3.10.12 + install_python_via_pyenv 3.10.12 } # deploy TDgpt TDgpt() { - install_python 3.10.12 + install_python_via_pyenv 3.10.12 } # deploy taos-test-framework From fd31350f2a24700e06c3f3bdae90dd530be285a0 Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Tue, 31 Dec 2024 12:58:01 +0800 Subject: [PATCH 47/73] enh: update --- packaging/setup_env.sh | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh index 07c5d45337..d84e40294d 100644 --- a/packaging/setup_env.sh +++ b/packaging/setup_env.sh @@ -153,6 +153,8 @@ help() { echo " config_coredump - Configure core dump settings" echo " disable_service - Disable specified services" echo " install_python - Install Python and pip" + echo " install_pyenv - Install Pyenv" + echo " install_python_via_pyenv - Install Python via pyenv" echo " install_java - Install Java" echo " install_java_via_sdkman - Install Java via sdkman" echo " install_maven_via_sdkman - Install Maven via sdkman" @@ -2069,6 +2071,12 @@ main() { install_python) install_python ;; + install_pyenv) + install_pyenv + ;; + install_python_via_pyenv) + install_python_via_pyenv + ;; install_pip_pkg) install_pip_pkg ;; From a17ac95893009dbea69755434a2bd78abf71f1e0 Mon Sep 17 00:00:00 2001 From: WANG MINGMING Date: Tue, 31 Dec 2024 13:41:29 +0800 Subject: [PATCH 48/73] Update 01-taosd.md --- docs/zh/14-reference/01-components/01-taosd.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/14-reference/01-components/01-taosd.md b/docs/zh/14-reference/01-components/01-taosd.md index c08581fdc1..ffc4d8b3c1 100644 --- a/docs/zh/14-reference/01-components/01-taosd.md +++ b/docs/zh/14-reference/01-components/01-taosd.md @@ -256,7 +256,7 @@ charset 的有效值是 UTF-8。 |slowLogMaxLen |3.3.3.0 后|支持动态修改 立即生效 |慢查询日志最大长度，取值范围 1-16384，默认值 4096| |slowLogScope |3.3.3.0 后|支持动态修改 立即生效 |慢查询记录类型，取值范围 ALL/QUERY/INSERT/OTHERS/NONE，默认值 QUERY| |slowLogExceptDb |3.3.3.0 后|支持动态修改 立即生效 |指定的数据库不上报慢查询，仅支持配置换一个数据库| -|debugFlag | |支持动态修改 立即生效 |运行日志开关，131（输出错误和警告日志），135（输出错误、警告和调试日志），143（输出错误、警告、调试和跟踪日志）；默认值 131 或 135 （取决于不同模块）| +|debugFlag | |支持动态修改 立即生效 |运行日志开关，131（输出错误和警告日志），135（输出错误、警告和调试日志），143（输出错误、警告、调试和跟踪日志）；默认值 131 或 135 （取决于不同模块），该参数的设置会影响所有模块的开关，后设置的参数起效| |tmrDebugFlag | |支持动态修改 立即生效 |定时器模块的日志开关，取值范围同上| |uDebugFlag | |支持动态修改 立即生效 |共用功能模块的日志开关，取值范围同上| |rpcDebugFlag | |支持动态修改 立即生效 |rpc 模块的日志开关，取值范围同上| From 2550b26c85e8cc803a00455195bbc60a67bfdc59 Mon Sep 17 00:00:00 2001 From: sheyanjie-qq <249478495@qq.com> Date: Tue, 31 Dec 2024 13:41:51 +0800 Subject: [PATCH 49/73] update jdbc version to 3.5.1 --- docs/en/07-develop/01-connect.md | 2 +- docs/en/14-reference/05-connector/14-java.md | 1 + docs/examples/JDBC/JDBCDemo/pom.xml | 2 +- docs/examples/JDBC/SpringJdbcTemplate/pom.xml | 2 +- docs/examples/JDBC/connectionPools/pom.xml | 2 +- docs/examples/JDBC/consumer-demo/pom.xml | 2 +- docs/examples/JDBC/mybatisplus-demo/pom.xml | 2 +- docs/examples/JDBC/springbootdemo/pom.xml | 2 +- docs/examples/JDBC/taosdemo/pom.xml | 2 +- docs/examples/java/pom.xml | 2 +- docs/zh/07-develop/01-connect/index.md | 2 +- docs/zh/14-reference/05-connector/14-java.mdx | 1 + 12 files changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/en/07-develop/01-connect.md b/docs/en/07-develop/01-connect.md index 9b1fbad6dd..c14eed311a 100644 --- a/docs/en/07-develop/01-connect.md +++ b/docs/en/07-develop/01-connect.md @@ -109,7 +109,7 @@ If you are using Maven to manage your project, simply add the following dependen com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 ``` diff --git a/docs/en/14-reference/05-connector/14-java.md b/docs/en/14-reference/05-connector/14-java.md index 2781f26c24..c28702440a 100644 --- a/docs/en/14-reference/05-connector/14-java.md +++ b/docs/en/14-reference/05-connector/14-java.md @@ -33,6 +33,7 @@ The JDBC driver implementation for TDengine strives to be consistent with relati | taos-jdbcdriver Version | Major Changes | TDengine Version | | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | +| 3.5.1 | Fixed the getObject issue in data subscription. | - | | 3.5.0 | 1. Optimized the performance of WebSocket connection parameter binding, supporting parameter binding queries using binary data.
2. Optimized the performance of small queries in WebSocket connection.
3. Added support for setting time zone and app info on WebSocket connection. | 3.3.5.0 and higher | | 3.4.0 | 1. Replaced fastjson library with jackson.
2. WebSocket uses a separate protocol identifier.
3. Optimized background thread usage to avoid user misuse leading to timeouts. | - | | 3.3.4 | Fixed getInt error when data type is float. | - | diff --git a/docs/examples/JDBC/JDBCDemo/pom.xml b/docs/examples/JDBC/JDBCDemo/pom.xml index e0c17ffbac..78262712e9 100644 --- a/docs/examples/JDBC/JDBCDemo/pom.xml +++ b/docs/examples/JDBC/JDBCDemo/pom.xml @@ -19,7 +19,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 org.locationtech.jts diff --git a/docs/examples/JDBC/SpringJdbcTemplate/pom.xml b/docs/examples/JDBC/SpringJdbcTemplate/pom.xml index b48f17acce..7ff4a72f5e 100644 --- a/docs/examples/JDBC/SpringJdbcTemplate/pom.xml +++ b/docs/examples/JDBC/SpringJdbcTemplate/pom.xml @@ -47,7 +47,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 diff --git a/docs/examples/JDBC/connectionPools/pom.xml b/docs/examples/JDBC/connectionPools/pom.xml index 8fca6ce1b2..70be6ed527 100644 --- a/docs/examples/JDBC/connectionPools/pom.xml +++ b/docs/examples/JDBC/connectionPools/pom.xml @@ -18,7 +18,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 diff --git a/docs/examples/JDBC/consumer-demo/pom.xml b/docs/examples/JDBC/consumer-demo/pom.xml index 997643c3de..c9537a93bf 100644 --- a/docs/examples/JDBC/consumer-demo/pom.xml +++ b/docs/examples/JDBC/consumer-demo/pom.xml @@ -17,7 +17,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 com.google.guava diff --git a/docs/examples/JDBC/mybatisplus-demo/pom.xml b/docs/examples/JDBC/mybatisplus-demo/pom.xml index 27e62695bc..effb13cfe8 100644 --- a/docs/examples/JDBC/mybatisplus-demo/pom.xml +++ b/docs/examples/JDBC/mybatisplus-demo/pom.xml @@ -47,7 +47,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 diff --git a/docs/examples/JDBC/springbootdemo/pom.xml b/docs/examples/JDBC/springbootdemo/pom.xml index f42799119b..25b503b0e6 100644 --- a/docs/examples/JDBC/springbootdemo/pom.xml +++ b/docs/examples/JDBC/springbootdemo/pom.xml @@ -70,7 +70,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 diff --git a/docs/examples/JDBC/taosdemo/pom.xml b/docs/examples/JDBC/taosdemo/pom.xml index 936a133e31..a80deeff94 100644 --- a/docs/examples/JDBC/taosdemo/pom.xml +++ b/docs/examples/JDBC/taosdemo/pom.xml @@ -67,7 +67,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 diff --git a/docs/examples/java/pom.xml b/docs/examples/java/pom.xml index 9b3a8c147d..63ce3159e6 100644 --- a/docs/examples/java/pom.xml +++ b/docs/examples/java/pom.xml @@ -22,7 +22,7 @@ com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 diff --git a/docs/zh/07-develop/01-connect/index.md b/docs/zh/07-develop/01-connect/index.md index e81a9e08c7..fa22f750f5 100644 --- a/docs/zh/07-develop/01-connect/index.md +++ b/docs/zh/07-develop/01-connect/index.md @@ -89,7 +89,7 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速 com.taosdata.jdbc taos-jdbcdriver - 3.5.0 + 3.5.1 ``` diff --git a/docs/zh/14-reference/05-connector/14-java.mdx b/docs/zh/14-reference/05-connector/14-java.mdx index c61bf51c82..7d5096bb66 100644 --- a/docs/zh/14-reference/05-connector/14-java.mdx +++ b/docs/zh/14-reference/05-connector/14-java.mdx @@ -33,6 +33,7 @@ TDengine 的 JDBC 驱动实现尽可能与关系型数据库驱动保持一致 | taos-jdbcdriver 版本 | 主要变化 | TDengine 版本 | | ------------------| ---------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------- | +| 3.5.1 | 解决了数据订阅获取时间戳对象类型问题 | - | | 3.5.0 | 1. 优化了 WebSocket 连接参数绑定性能，支持参数绑定查询使用二进制数据
2. 优化了 WebSocket 连接在小查询上的性能
3. WebSocket 连接上支持设置时区和应用信息 | 3.3.5.0 及更高版本 | | 3.4.0 | 1. 使用 jackson 库替换 fastjson 库
2. WebSocket 采用独立协议标识
3. 优化后台拉取线程使用，避免用户误用导致超时 | - | | 3.3.4 | 解决了 getInt 在数据类型为 float 报错 | - | From fcab40ad958fea42996871678fdc2fc7848c6f4c Mon Sep 17 00:00:00 2001 From: jiajingbin Date: Tue, 31 Dec 2024 14:10:39 +0800 Subject: [PATCH 50/73] enh: update --- packaging/setup_env.sh | 32 +++++++++++++++++++------------- 1 file changed, 19 insertions(+), 13 deletions(-) diff --git a/packaging/setup_env.sh b/packaging/setup_env.sh index d84e40294d..e8b69c964e 100644 --- a/packaging/setup_env.sh +++ b/packaging/setup_env.sh @@ -930,7 +930,7 @@ install_java() { INSTALLED_VERSION=$("$JAVA_HOME"/bin/java --version 2>&1) if echo "$INSTALLED_VERSION" | grep -q "openjdk $DEFAULT_JDK_VERSION"; then echo -e "${GREEN}Java installed successfully.${NO_COLOR}" - SOURCE_RESULTS+="source /root/.bashrc # For OpenJDK\n" + SOURCE_RESULTS+="source /root/.bashrc # For openjdk\n" else echo -e "${YELLOW}Java version not match.${NO_COLOR}" exit 1 @@ -964,27 +964,32 @@ install_gvm() { add_config_if_not_exist "export GOPROXY=https://goproxy.cn,direct" "$BASH_RC" add_config_if_not_exist "export GO_BINARY_BASE_URL=https://mirrors.aliyun.com/golang/" "$BASH_RC" add_config_if_not_exist "export GOROOT_BOOTSTRAP=$GOROOT" "$BASH_RC" - SOURCE_RESULTS+="source $HOME/.gvm/scripts/gvm # For GVM\n" fi + SOURCE_RESULTS+="source $HOME/.gvm/scripts/gvm # For gvm\n" } -# Install pyvenv +# enable pyenv +enable_pyenv() { + export PATH="$HOME/.pyenv/bin:$PATH" + eval "$(pyenv init --path)" + eval "$(pyenv init -)" +} + +# Install pyenv install_pyenv() { echo -e "${YELLOW}Installing Pyenv...${NO_COLOR}" if [ -d "$HOME/.pyenv" ]; then echo -e "${GREEN}Pyenv is already installed.${NO_COLOR}" else curl -L https://gitee.com/xinghuipeng/pyenv-installer/raw/master/bin/pyenv-installer | bash - export PATH="$HOME/.pyenv/bin:$PATH" - eval "$(pyenv init --path)" - eval "$(pyenv init -)" + enable_pyenv add_config_if_not_exist "export PATH=\"\$HOME/.pyenv/bin:\$PATH\"" "$BASH_RC" add_config_if_not_exist "eval \"\$(pyenv init --path)\"" "$BASH_RC" add_config_if_not_exist "eval \"\$(pyenv init -)\"" "$BASH_RC" pyenv --version check_status "Failed to install Pyenv" "Pyenv installed successfully." $? - SOURCE_RESULTS+="source $BASH_RC For: Pyenv/python\n" fi + SOURCE_RESULTS+="source $BASH_RC For: pyenv/python\n" } # Install python via pyenv @@ -1005,7 +1010,7 @@ install_python_via_pyenv() { DEFAULT_PYTHON_VERSION="3.10.12" fi install_pyenv - + enable_pyenv pyenv install "$DEFAULT_PYTHON_VERSION" pyenv global "$DEFAULT_PYTHON_VERSION" python --version @@ -1043,7 +1048,7 @@ install_java_via_sdkman() { [[ -s "$HOME/.sdkman/bin/sdkman-init.sh" ]] && source "$HOME/.sdkman/bin/sdkman-init.sh" java -version check_status "Failed to install java" "Java installed successfully." $? - SOURCE_RESULTS+="source $HOME/.sdkman/bin/sdkman-init.sh # For Sdkman/java/maven\n" + SOURCE_RESULTS+="source $HOME/.sdkman/bin/sdkman-init.sh # For sdkman/java/maven\n" } # Install Go @@ -1079,7 +1084,7 @@ deploy_go() { # Apply the environment variables $GO_INSTALL_DIR/bin/go version check_status "Failed to install GO" "Install GO successfully" $? - SOURCE_RESULTS+="source $BASH_RC # For Golang\n" + SOURCE_RESULTS+="source $BASH_RC # For golang\n" } # Install Go via gvm @@ -1091,6 +1096,7 @@ install_go_via_gvm() { DEFAULT_GO_VERSION="1.23.0" fi install_gvm + source $HOME/.gvm/scripts/gvm export GO111MODULE=on export GOPROXY=https://goproxy.cn,direct export GO_BINARY_BASE_URL=https://mirrors.aliyun.com/golang/ @@ -1102,7 +1108,7 @@ install_go_via_gvm() { go version check_status "Failed to install Go" "Go installed successfully." $? - SOURCE_RESULTS+="source $BASH_RC # For Golang\n" + SOURCE_RESULTS+="source $BASH_RC # For golang\n" } # Function to install Rust and Cargo @@ -1153,7 +1159,7 @@ deploy_rust() { # Install cargo-make cargo install cargo-make check_status "Failed to install Rust" "Install Rust successfully" $? - SOURCE_RESULTS+="source $BASH_RC && source $HOME/.cargo/env # For Rust\n" + SOURCE_RESULTS+="source $BASH_RC && source $HOME/.cargo/env # For cargo/rust\n" else echo "Rust is already installed." fi @@ -1238,7 +1244,7 @@ install_node_via_nvm () { [ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh" [ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion" echo -e "${GREEN}NVM installed successfully.${NO_COLOR}" - SOURCE_RESULTS+="source $NVM_DIR/nvm.sh && source $NVM_DIR/bash_completion # For NVM/node/npm/yarn/pnpm\n" + SOURCE_RESULTS+="source $NVM_DIR/nvm.sh && source $NVM_DIR/bash_completion # For nvm/node/npm/yarn/pnpm\n" else echo -e "${GREEN}NVM is already installed.${NO_COLOR}" fi From 6dfd0747264be360ded119fbbdfce713adf80376 Mon Sep 17 00:00:00 2001 From: menshibin Date: Tue, 31 Dec 2024 14:13:08 +0800 Subject: [PATCH 51/73] modify flink connector version --- docs/zh/10-third-party/01-collection/12-flink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md index 8b2e5422b1..d411b728cb 100644 --- a/docs/zh/10-third-party/01-collection/12-flink.md +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -111,7 +111,7 @@ TDengine 目前支持时间戳、数字、字符、布尔类型，与 Flink RowD com.taosdata.flink flink-connector-tdengine - 1.0.0 + 2.0.0 ``` From e5bd690c7a9c16b76bb36cc3de92c1274cf332de Mon Sep 17 00:00:00 2001 From: menshibin Date: Tue, 31 Dec 2024 14:15:59 +0800 Subject: [PATCH 52/73] modify flink connector history version --- docs/zh/10-third-party/01-collection/12-flink.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md index d411b728cb..46374f8069 100644 --- a/docs/zh/10-third-party/01-collection/12-flink.md +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -28,7 +28,7 @@ Flink Connector 支持所有能运行 Flink 1.19 及以上版本的平台。 ## 版本历史 | Flink Connector 版本 | 主要变化 | TDengine 版本 | | ------------------| ------------------------------------ | ---------------- | -| 1.1.0 | 1. 支持 SQL 查询 TDengine 数据库中的数据
2. 支持 CDC 订阅 TDengine 数据库中的数据
3. 支持 Table SQL 方式读取和写入 TDengine 数据库| 3.3.5.0 及以上版本 | +| 2.0.0 | 1. 支持 SQL 查询 TDengine 数据库中的数据
2. 支持 CDC 订阅 TDengine 数据库中的数据
3. 支持 Table SQL 方式读取和写入 TDengine 数据库| 3.3.5.0 及以上版本 | ## 异常和错误码 From 511b83f187695e0cd6f49c9b0ffccd8ea40e811b Mon Sep 17 00:00:00 2001 From: menshibin Date: Tue, 31 Dec 2024 17:00:27 +0800 Subject: [PATCH 53/73] modify flink connector docs format --- .../10-third-party/01-collection/12-flink.md | 122 +++++++++--------- 1 file changed, 61 insertions(+), 61 deletions(-) diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md index 46374f8069..b58a013f38 100644 --- a/docs/zh/10-third-party/01-collection/12-flink.md +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -38,14 +38,14 @@ Flink Connector 支持所有能运行 Flink 1.19 及以上版本的平台。 | Error Code | Description | Suggested Actions | | ---------------- |------------------------------------------------------- | -------------------- | -| 0xa000 |connection param error |连接器参数错误 +| 0xa000 |connection param error |连接器参数错误。 | 0xa001 |the groupid parameter of CDC is incorrect |CDC 的 groupid 参数错误。| | 0xa002 |wrong topic parameter for CDC |CDC 的 topic 参数错误。| -| 0xa010 |database name configuration error |数据库名配置错误| -| 0xa011 |table name configuration error |表名配置错误| -| 0xa012 |no data was obtained from the data source |从数据源中获取数据失败| -| 0xa013 |value.deserializer parameter not set |未设置序列化方式| -| 0xa014 |list of column names for target table not set |未设置目标表的列名列表| +| 0xa010 |database name configuration error |数据库名配置错误。| +| 0xa011 |table name configuration error |表名配置错误。| +| 0xa012 |no data was obtained from the data source |从数据源中获取数据失败。| +| 0xa013 |value.deserializer parameter not set |未设置序列化方式。| +| 0xa014 |list of column names for target table not set |未设置目标表的列名列表。| | 0x2301 |connection already closed |连接已经关闭，检查连接情况，或重新创建连接去执行相关指令。| | 0x2302 |this operation is NOT supported currently! |当前使用接口不支持，可以更换其他连接方式。| | 0x2303 |invalid variables |参数不合法，请检查相应接口规范，调整参数类型及大小。| @@ -140,15 +140,15 @@ Properties 中配置参数如下： | 参数名称 | 类型 | 参数说明 | 备注 | | ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'| | -| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'| | -| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径| -| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineSource 对象时需要指定数据类型为 SourceRecords\<类型\> | 此处的类型为用下游算子接收数据的类型| +| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'。| | +| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'。| | +| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径。| +| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineSource 对象时需要指定数据类型为 SourceRecords\<类型\> 。 | 此处的类型为用下游算子接收数据的类型。| | TDengineConfigParams.PROPERTY_KEY_CHARSET | string | 客户端使用的字符集，默认值为系统字符集。| | -| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false|| -| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效| +| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000。| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false。|| +| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。| | TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| #### 按时间分片 @@ -214,18 +214,18 @@ Flink CDC 主要用于提供数据订阅功能，能实时监控 TDengine 数据 #### 参数说明 | 参数名称 | 类型 | 参数说明 | 备注 | | ----------------------- | :-----: | ------------------------- | -------------------------- | -| TDengineCdcParams.BOOTSTRAP_SERVERS| string | 服务端的 IP 地址 | | -| TDengineCdcParams.CONNECT_USER| string | 用户名 | | -| TDengineCdcParams.CONNECT_PASS| string | 密码 | | -| TDengineCdcParams.POLL_INTERVAL_MS|int| 拉取数据间隔, 默认 500ms| | -| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径| -| TDengineCdcParams.TMQ_BATCH_MODE| boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineCdcSource 对象时需要指定数据类型为 ConsumerRecords\<类型\> | 此处的类型为用下游算子接收数据的类型| -| TDengineCdcParams.GROUP_ID| string | 消费组 ID，同一消费组共享消费进度 |
**必填项**。最大长度：192。
每个 topic 最多可建立 100 个 consumer -| TDengineCdcParams.AUTO_OFFSET_RESET| string | 消费组订阅的初始位置 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅| -| TDengineCdcParams.ENABLE_AUTO_COMMIT| boolean | 是否自动提交，true: 启用（用于下游均为无状态算子） ；false：由 checkpoint 触发 commit | 默认 false| -| TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS| integer|消费记录自动提交消费位点时间间隔，单位为毫秒| 默认值为 5000, 此参数在 AUTO_OFFSET_RESET 为 true 生效| -| TDengineCdcParams.TMQ_SESSION_TIMEOUT_MS| integer | consumer 心跳丢失后超时时间，超时后会触发 rebalance 逻辑，成功后该 consumer 会被删除（从 TDengine 3.3.3.0 版本开始支持）| 默认值为 12000，取值范围 [6000， 1800000] | -| TDengineCdcParams.TMQ_MAX_POLL_INTERVAL_MS| integer | consumer poll 拉取数据间隔的最长时间，超过该时间，会认为该 consumer 离线，触发 rebalance 逻辑，成功后该 consumer 会被删除（从 3.3.3.0 版本开始支持） | 默认值为 300000，[1000，INT32_MAX] +| TDengineCdcParams.BOOTSTRAP_SERVERS| string | 服务端的 IP 地址。 | | +| TDengineCdcParams.CONNECT_USER| string | 用户名。 | | +| TDengineCdcParams.CONNECT_PASS| string | 密码。 | | +| TDengineCdcParams.POLL_INTERVAL_MS|int| 拉取数据间隔, 默认 500ms。| | +| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型。 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径。| +| TDengineCdcParams.TMQ_BATCH_MODE| boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineCdcSource 对象时需要指定数据类型为 ConsumerRecords\<类型\>。| 此处的类型为用下游算子接收数据的类型。| +| TDengineCdcParams.GROUP_ID| string | 消费组 ID，同一消费组共享消费进度。 |
**必填项**。最大长度：192。
每个 topic 最多可建立 100 个 consumer 。| +| TDengineCdcParams.AUTO_OFFSET_RESET| string | 消费组订阅的初始位置。 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅。| +| TDengineCdcParams.ENABLE_AUTO_COMMIT| boolean | 是否自动提交，true: 启用（用于下游均为无状态算子） ；false：由 checkpoint 触发 commit 。| 默认 false。| +| TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS| integer|消费记录自动提交消费位点时间间隔，单位为毫秒。| 默认值为 5000, 此参数在 AUTO_OFFSET_RESET 为 true 生效。| +| TDengineCdcParams.TMQ_SESSION_TIMEOUT_MS| integer | consumer 心跳丢失后超时时间，超时后会触发 rebalance 逻辑，成功后该 consumer 会被删除（从 TDengine 3.3.3.0 版本开始支持）。| 默认值为 12000，取值范围 [6000， 1800000]。 | +| TDengineCdcParams.TMQ_MAX_POLL_INTERVAL_MS| integer | consumer poll 拉取数据间隔的最长时间，超过该时间，会认为该 consumer 离线，触发 rebalance 逻辑，成功后该 consumer 会被删除（从 3.3.3.0 版本开始支持）。 | 默认值为 300000，[1000，INT32_MAX]。| #### 使用 CDC 连接器 @@ -268,19 +268,19 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 | 参数名称 | 类型 | 参数说明 | 备注 | | ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'| | -| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'| | -| TDengineConfigParams.PROPERTY_KEY_DBNAME| string | 写入的数据库名称|| -| TDengineConfigParams.TD_SUPERTABLE_NAME| string | 写入的超级表名称| 如果是超级表接收的数据必须有 tbname 字段，确定写入那张子表| -| TDengineConfigParams.TD_TABLE_NAME| string | 写入的表名，此参数和TD_SUPERTABLE_NAME 仅需要设置一个即可| 用于确定写入那张子表或普通表| -| TDengineConfigParams.TD_BATCH_SIZE| integer | 设置批大小 | 当到达批的数量后进行写入，或是一个checkpoint的时间也会触发写入数据库| -| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径| -| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于设置接收批量数据 | 如果设置为 True:
来源是 TDengine Source， 使用SourceRecords\<类型\> 创建 TDengineSink 对象
来源是 TDengine CDC 使用 ConsumerRecords\<类型\> 创建 TDengineSink 对象 | 此处的类型为接收数据的类型| -| TDengineConfigParams.TD_SOURCE_TYPE | string | 如果数据是表示数据来源是，source 或者 cdc 等 | TDengine source 设置为 "tdengine_source", TDengine cdc 设置为 "tdengine_cdc"| -| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false|| -| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效| +| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'。| | +| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'。| | +| TDengineConfigParams.PROPERTY_KEY_DBNAME| string | 写入的数据库名称。|| +| TDengineConfigParams.TD_SUPERTABLE_NAME| string | 写入的超级表名称。| 如果是超级表接收的数据必须有 tbname 字段，确定写入那张子表。| +| TDengineConfigParams.TD_TABLE_NAME| string | 写入的表名，此参数和TD_SUPERTABLE_NAME 仅需要设置一个即可。| 用于确定写入那张子表或普通表。| +| TDengineConfigParams.TD_BATCH_SIZE| integer | 设置批大小 | 当到达批的数量后进行写入，或是一个checkpoint的时间也会触发写入数据库。| +| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型。 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径。| +| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于设置接收批量数据。 | 如果设置为 True:
来源是 TDengine Source， 使用SourceRecords\<类型\> 创建 TDengineSink 对象
来源是 TDengine CDC 使用 ConsumerRecords\<类型\> 创建 TDengineSink 对象。| 此处的类型为接收数据的类型。| +| TDengineConfigParams.TD_SOURCE_TYPE | string | 如果数据是表示数据来源是，source 或者 cdc 等。 | TDengine source 设置为 "tdengine_source", TDengine cdc 设置为 "tdengine_cdc"。| +| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000。| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。| | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false。|| +| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3。 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。| | TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| #### 使用 Sink 连接器 @@ -315,15 +315,15 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 | 参数名称 | 类型 | 参数说明 | 备注 | | ----------------------- | :-----: | ------------ | ------ | -| connector | string | 连接器标识，设置 tdengine-connector || -| td.jdbc.url| string | 连接的 url | | -| td.jdbc.mode | strng | 连接器类型, 设置 source, cdc, sink| | -| table.name| string| 原表或目标表名称| | -| scan.query| string| 获取数据的 SQL 语句|| -| sink.db.name|string| 目标数据库名称|| -| sink.supertable.name|string |写入的超级表名称|| -| sink.batch.size | integer | 写入的批大小|| -| sink.table.name|string|写入的普通表或子表名称|| +| connector | string | 连接器标识，设置 `tdengine-connector` 。|| +| td.jdbc.url| string | 连接的 url 。| | +| td.jdbc.mode | strng | 连接器类型, 设置 `source`, `cdc`, `sink`。| | +| table.name| string| 原表或目标表名称。| | +| scan.query| string| 获取数据的 SQL 语句。|| +| sink.db.name|string| 目标数据库名称。|| +| sink.supertable.name|string |写入的超级表名称。|| +| sink.batch.size | integer | 写入的批大小。|| +| sink.table.name|string|写入的普通表或子表名称。|| #### Source 连接器使用示例 @@ -340,19 +340,19 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 | 参数名称 | 类型 | 参数说明 | 备注 | | ----------------------- | :-----: | ------------ |-------| -| connector | string | 连接器标识，设置 tdengine-connector || -| user| string | 用户名， 默认 root| | -| password | string | 密码， 默认taosdata| | -| bootstrap.servers| string | 服务器地址 | | -| topic | string | 订阅主题 || -| td.jdbc.mode | strng | 连接器类型, cdc, sink| | -| group.id| string| 消费组 ID，同一消费组共享消费进度 | | -| auto.offset.reset| string| 消费组订阅的初始位置 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅| -| poll.interval_ms| integer| 拉取数据间隔, 默认 500ms| | -| sink.db.name|string| 目标数据库名称|| -| sink.supertable.name|string |写入的超级表名称|| -| sink.batch.size | integer | 写入的批大小|| -| sink.table.name|string|写入的普通表或子表名称|| +| connector | string | 连接器标识，设置 `tdengine-connector`。|| +| user| string | 用户名， 默认 root。| | +| password | string | 密码， 默认taosdata。| | +| bootstrap.servers| string | 服务器地址。| | +| topic | string | 订阅主题。|| +| td.jdbc.mode | strng | 连接器类型, cdc, sink。| | +| group.id| string| 消费组 ID，同一消费组共享消费进度。 | | +| auto.offset.reset| string| 消费组订阅的初始位置。 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅。| +| poll.interval_ms| integer| 拉取数据间隔, 默认 500ms。| | +| sink.db.name|string| 目标数据库名称。|| +| sink.supertable.name|string |写入的超级表名称。|| +| sink.batch.size | integer | 写入的批大小。|| +| sink.table.name|string|写入的普通表或子表名称。|| From db459aca3776e400420c4605b099d9e29d0a6b5f Mon Sep 17 00:00:00 2001 From: menshibin Date: Tue, 31 Dec 2024 17:34:45 +0800 Subject: [PATCH 54/73] add english flink connector docs --- docs/en/10-third-party/01-collection/flink.md | 358 ++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 docs/en/10-third-party/01-collection/flink.md diff --git a/docs/en/10-third-party/01-collection/flink.md b/docs/en/10-third-party/01-collection/flink.md new file mode 100644 index 0000000000..f14ffc33a6 --- /dev/null +++ b/docs/en/10-third-party/01-collection/flink.md @@ -0,0 +1,358 @@ +--- +sidebar_label: Flink +title: TDengine Flink Connector +--- + +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + +## Preconditions + +Prepare the following environment: + +- TDengine cluster has been deployed and is running normally (both enterprise and community versions are available) +- TaosAdapter can run normally. +- Apache Flink v1.19.0 or above is installed. Please refer to the installation of Apache Flink [Official documents](https://flink.apache.org/) + +## JRE version compatibility + +JRE: Supports JRE 8 and above versions. + +## Supported platforms + +Flink Connector supports all platforms that can run Flink 1.19 and above versions. + +## Version History + + +| Flink Connector Version | Major Changes | TDengine Version| +|-------------------------| ------------------------------------ | ---------------- | +| 2.0.0 | 1. Support SQL queries on data in TDengine database
2 Support CDC subscription to data in TDengine database
3 Supports reading and writing to TDengine database using Table SQL | 3.3.5.0 and above versions| + +## Exception and error codes + +After the task execution fails, check the Flink task execution log to confirm the reason for the failure +Please refer to: + +| Error Code | Description | Suggested Actions | +| ---------------- |------------------------------------------------------- | -------------------- | +|0xa000 | connection param error | connector parameter error +|0xa001 | The groupid parameter of CDC is incorrect | The groupid parameter of CDC is incorrect| +|0xa002 | wrong topic parameter for CDC | The topic parameter for CDC is incorrect| +|0xa010 | database name configuration error | database name configuration error| +|0xa011 | Table name configuration error | Table name configuration error| +|0xa012 | No data was obtained from the data source | Failed to retrieve data from the data source| +|0xa013 | value.deserializer parameter not set | No serialization method set| +|0xa014 | List of column names for target table not set | List of column names for target table not set || +|0x2301 | Connection already closed | The connection has been closed. Check the connection status or create a new connection to execute the relevant instructions| +|0x2302 | this operation is NOT supported currently | The current interface is not supported, you can switch to other connection methods| +|0x2303 | invalid variables | The parameter is invalid. Please check the corresponding interface specification and adjust the parameter type and size| +|0x2304 | Statement is closed | Statement has already been closed. Please check if the statement is closed and reused, or if the connection is working properly| +|0x2305 | ResultSet is closed | The ResultSet has been released. Please check if the ResultSet has been released and used again| +|0x230d | parameter index out of range | parameter out of range, please check the reasonable range of the parameter| +|0x230e | Connection already closed | The connection has been closed. Please check if the connection is closed and used again, or if the connection is working properly| +|0x230f | unknown SQL type in TDengine | Please check the Data Type types supported by TDengine| +|0x2315 | unknown tao type in TDengine | Did the correct TDengine data type be specified when converting TDengine data type to JDBC data type| +|0x2319 | user is required | Username information is missing when creating a connection| +|0x231a | password is required | Password information is missing when creating a connection| +|0x231d | can't create connection with server within | Increase connection time by adding the parameter httpConnectTimeout, or check the connection status with taosAdapter| +|0x231e | failed to complete the task within the specified time | Increase execution time by adding the parameter messageWaitTimeout, or check the connection with taosAdapter| +|0x2352 | Unsupported encoding | An unsupported character encoding set was specified under the local connection| +|0x2353 |internal error of database, Please see taoslog for more details | An error occurred while executing prepareStatement on the local connection. Please check the taoslog for problem localization| +|0x2354 | Connection is NULL | Connection has already been closed while executing the command on the local connection. Please check the connection with TDengine| +|0x2355 | result set is NULL | Local connection to obtain result set, result set exception, please check connection status and retry| +|0x2356 | invalid num of fields | The meta information obtained from the local connection result set does not match| +|0x2357 | empty SQL string | Fill in the correct SQL for execution| +|0x2371 |consumer properties must not be null | When creating a subscription, the parameter is empty. Please fill in the correct parameter| +|0x2375 | Topic reference has been destroyed | During the process of creating a data subscription, the topic reference was released. Please check the connection with TDengine| +|0x2376 |failed to set consumer topic, Topic name is empty | During the process of creating a data subscription, the subscription topic name is empty. Please check if the specified topic name is filled in correctly| +|0x2377 | Consumer reference has been destroyed | The subscription data transmission channel has been closed, please check the connection with TDengine| +|0x2378 | Consumer create error | Failed to create data subscription. Please check the taos log based on the error message to locate the problem| +|0x237a | vGroup not found in result set VGroup | Not assigned to the current consumer, due to the Rebalance mechanism, the relationship between Consumer and VGroup is not bound| + +## Data type mapping + +TDengine currently supports timestamp, number, character, and boolean types, and the corresponding type conversions with Flink RowData Type are as follows: + +| TDengine DataType | Flink RowDataType | +| ----------------- | ------------------ | +| TIMESTAMP | TimestampData | +| INT | Integer | +| BIGINT | Long | +| FLOAT | Float | +| DOUBLE | Double | +| SMALLINT | Short | +| TINYINT | Byte | +| BOOL | Boolean | +| BINARY | byte[] | +| NCHAR | StringData | +| JSON | StringData | +| VARBINARY | byte[] | +| GEOMETRY | byte[] | + +## Instructions for use +### Flink Semantic Selection Instructions + +The semantic reason for using At Least One (at least once) is: +-TDengine currently does not support transactions and cannot perform frequent checkpoint operations and complex transaction coordination. +-Due to TDengine's use of timestamps as primary keys, downstream operators of duplicate data can perform filtering operations to avoid duplicate calculations. +-Using At Least One (at least once) to ensure high data processing performance and low data latency, the setting method is as follows: + +```text +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +env.enableCheckpointing(5000); +env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE); +``` + +If using Maven to manage a project, simply add the following dependencies in pom.xml. + +```xml + + com.taosdata.flink + flink-connector-tdengine + 2.0.0 + +``` + +The parameters for establishing a connection include URL and Properties. +The URL specification format is: + +`jdbc: TAOS-WS://[host_name]:[port]/[database_name]? [user={user}|&password={password}|&timezone={timezone}]` + +Parameter description: + +- User: Login TDengine username, default value is' root '. +- Password: User login password, default value 'taosdata'. +- batchErrorIgnore： true： If there is an SQL execution failure in the middle of the ExecutBatch of Statement, continue to execute the following SQL. false： Do not execute any statements after failed SQL. The default value is: false。 +- HttpConnectTimeout: The connection timeout time, measured in milliseconds, with a default value of 60000. +- MessageWaitTimeout: The timeout period for a message, measured in milliseconds, with a default value of 60000. +- UseSSL: Whether SSL is used in the connection. + +### Source + +Source retrieves data from the TDengine database, converts it into a format and type that Flink can handle internally, and reads and distributes it in parallel, providing efficient input for subsequent data processing. +By setting the parallelism of the data source, multiple threads can read data from the data source in parallel, improving the efficiency and throughput of data reading, and fully utilizing cluster resources for large-scale data processing capabilities. + +#### Source Properties + +The configuration parameters in Properties are as follows: + +|Parameter Name | Type | Parameter Description | Remarks| +| ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| TDengineConfigParams.PROPERTYKEYUSER | string | Login TDengine username, default value 'root' || +| TDengineConfigParams.PROPERTYKEY-PASSWORD | string | User login password, default value 'taosdata' || +| TDengineConfigParams.If the downstream operator receives data of RowData type, it only needs to be set to RowData. If the user needs to customize the type, the complete class path needs to be set here| +| TDengineConfigParams.TD_STACTMODE | boolean | This parameter is used to batch push data to downstream operators. If set to True, when creating a TDengine Source object, the data type needs to be specified as SourceRecords \| The type here is the type used to receive data from downstream operators| +| TDengineConfigParams.PROPERTYKEY_CARSET | string | The character set used by the client, with the default value being the system character set. || +| TDengineConfigParams.PROPERTYKEY.MSSAGE_maIT_TIMEOUT | integer | Message timeout, in milliseconds, default value is 60000 || +| TDengineConfigParams.Whether compression is enabled during the transmission process. true: Enable, false: Not enabled. Default is false || +| TDengineConfigParams.Whether to enable automatic reconnection or not. true: Enable, false: Not enabled. Default to false|| +| TDengineConfigParams.PROPERTYKEY-RECONNECT-RETR_COUNT | integer | number of automatic reconnection retries, default value 3 | only takes effect when PROPERTYKEY-INABLE AUTO-RECONNECT is true| +| TDengineConfigParams.PROPERTYKEYDISABLE_SSL_CERTVNet | boolean | Disable SSL certificate verification. true: close, false: Not closed. The default is false|| + +#### Split by time + +Users can split the SQL query into multiple subtasks based on time, entering: start time, end time, split interval, time field name. The system will split and obtain data in parallel according to the set interval (time left closed and right open). + +```java +{{#include docs/examples/flink/Main.java:time_interval}} +``` + +Splitting by Super Table TAG + +Users can split the query SQL into multiple query conditions based on the TAG field of the super table, and the system will split them into subtasks corresponding to each query condition, thereby obtaining data in parallel. + +```java +{{#include docs/examples/flink/Main.java:tag_split}} +``` + +Classify by table + +Support sharding by inputting multiple super tables or regular tables with the same table structure. The system will split them according to the method of one table, one task, and then obtain data in parallel. + +```java +{{#include docs/examples/flink/Main.java:table_split}} +``` + +Use Source connector + +The query result is RowData data type example: + +
+RowData Source +```java +{{#include docs/examples/flink/Main.java:source_test}} +``` +
+ +Example of batch query results: + +
+Batch Source +```java +{{#include docs/examples/flink/Main.java:source_batch_test}} +``` +
+ +Example of custom data type query result: + +
+Custom Type Source +```java +{{#include docs/examples/flink/Main.java:source_custom_type_test}} +``` +
+ +- ResultBean is a custom inner class used to define the data type of the Source query results. +- ResultSoureDeserialization is a custom inner class that inherits Tdengine RecordDesrialization and implements convert and getProducedType methods. + +### CDC Data Subscription +Flink CDC is mainly used to provide data subscription functionality, which can monitor real-time changes in TDengine database data and transmit these changes in the form of data streams to Flink for processing, while ensuring data consistency and integrity. + +#### Parameter Description + +| Parameter Name | Type | Parameter Description | Remarks | +|-------------------------------------------|:-------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| TDengineCdcParams.BOOTSTRAP_SERVER | string | ip address of the server | | +| TDengineCdcParams.CONNECT-USER | string | username | | +| TDengineCdcParams.CONNECT-PASS | string | password | | +| TDengineCdcParams.POLL_INTERVAL_MS | integer | pull data interval, default 500ms | | +| TDengineConfigParams.VALUE_DESERIALIZER | string | the type of data received by the operator| If the downstream operator receives data of RowData type, it only needs to be set to RowData. If the user needs to customize the type, the complete class path needs to be set here | +| TDengineCdcParams.TMQ_STACTMODE | boolean | this parameter is used to batch push data to downstream operators. If set to True, when creating a TDengine CdcSource object, the data type needs to be specified as ConsumerRecords \ | The type here is the type used to receive data from downstream operators | +| TDengineCdcParams.GROUP ID | string | consumption group ID, shared consumption progress within the same consumption group |
* * Required field * *. Maximum length: 192< Each topic can create up to 100 consumers +| TDengineCdcParams.AUTO-OFFSET-REET | string | initial position of consumer group subscription | early: subscribe from scratch
latest: default; Subscribe only from the latest data | +| TDengineCdcParams.ENABLEAUTO_CMMIT | boolean | whether to automatically submit, true: Enable (for downstream stateless operators); false： Commit triggered by checkpoint; default false | | +| TDengineCdcParams.AUTO_CMMIT_INTERVAL_S | integer | the time interval for automatically submitting consumption records to consumption sites, in milliseconds | The default value is 5000, and this parameter takes effect when AUTO_oFFSET-REET is set to true | +| TDengineCdcParams.TMQ_SSSION_TIMEOUT_SS | integer | timeout after consumer heartbeat loss, which triggers rebalancing logic. After success, the consumer will be deleted (supported from TDengine 3.3.3.0 version) | default value is 12000, value range [60001800000] | +| TDengineCdcParams.TMQ_maX_POLL_INTERVAL_S | integer | the longest time interval for pulling data from a consumer poll. If this time is exceeded, the consumer will be considered offline and the rebalancing logic will be triggered. After success, the consumer will be deleted (supported from version 3.3.3.0) | The default value is 300000, [1000，INT32_MAX] + +#### Use CDC connector + +The CDC connector will create consumers based on the parallelism set by the user, so the user should set the parallelism reasonably according to the resource situation. +The subscription result is RowData data type example: + +
+CDC Source +```java +{{#include docs/examples/flink/Main.java:cdc_source}} +``` +
+ +Example of batch query results: + +
+CDC Batch Source +```java +{{#include docs/examples/flink/Main.java:cdc_batch_source}} +``` +
+ +Example of custom data type query result: + +
+CDC Custom Type +```java +{{#include docs/examples/flink/Main.java:cdc_custom_type_test}} +``` +
+ +- ResultBean is a custom inner class whose field names and data types correspond one-to-one with column names and data types. This allows the deserialization class corresponding to the value.ddeserializer property to deserialize objects of ResultBean type. + +### Sink + +The core function of Sink is to efficiently and accurately write Flink processed data from different data sources or operators into TDengine. In this process, the efficient write mechanism possessed by TDengine played a crucial role, effectively ensuring the fast and stable storage of data. + +#### Sink Properties + +| Parameter Name | Type | Parameter Description | Remarks| +|---------------------------------------------------------|:----------------------------------------------------------------------------------------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| TDengineConfigParams.PROPERTYKEYUSER | string | Login TDengine username, default value 'root' || +| TDengineConfigParams.PROPERTYKEY-PASSWORD | string | User login password, default value 'taosdata' || +| TDengineConfigParams.PROPERTYKEYDBNAME | string | Database name written || +| TDengineConfigParams.TD_SUPERTABLeNAME | string | Name of the super table to be written | If the data received by the super table must have a tbname field, determine which sub table to write to| +| TDengineConfigParams.TD_TABLeNAME | string | The name of the table to be written, this parameter only needs to be set together with TD_SUPERTABLeNAME | Used to determine which sub table or regular table to write to| +| TDengineConfigParams.TD_STACTISZE | integer | Set batch size | Write when the batch quantity is reached, or a checkpoint time will also trigger writing to the database| +| TDengineConfigParams.VALUE_DESERIALIZER | string | If the downstream operator receives data of RowData type, it only needs to be set to RowData. If the user needs to customize the type, the complete class path needs to be set here | +| TDengineConfigParams.TD_STACTMODE | boolean | This parameter is used to set the reception of batch data | If set to True:< The source is TDengine Source, using SourceRecords \to create TDengine Sink object
The source is TDengine CDC, using ConsumerRecords \to create TDengine Sink object | The type here is the type that receives data| +| TDengineConfigParams.TD_SOURCETYPE | string | If the data is from a source, such as source or cdc | TDengine source is set to "tdengine_stource", TDengine cdc is set to "tdengine_cdc"| +| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | Message timeout, in milliseconds, default value is 60000 || +| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | Whether compression is enabled during the transmission process. true: Enable, false: Not enabled. Default is false | | +| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT | integer| to enable automatic reconnection or not. true: Enable, false: Not enabled. Default to false | | +| TDengineConfigParams.PROPERTYKEY_RECONNECT_RETR_COUNT | integer | number of automatic reconnection retries, default value 3 | only takes effect when PROPERTYKEY-INABLE AUTO-RECONNECT is true| +| TDengineConfigParams.PROPERTYKEYDISABLE_SSL_CERTVNet | boolean | Disable SSL certificate verification. true: close, false: Not closed. The default is false || + +#### Use Sink connector +Write the received RowData type data into TDengine example: + +
+Sink RowData +```java +{{#include docs/examples/flink/Main.java:RowDataToSink}} +``` +
+ +Write batch received RowData data into TDengine example: + +
+Sink RowData +```java +{{#include docs/examples/flink/Main.java:BatchRowDataToSink}} +``` +
+ +### Table SQL + +ETL (Extract, Transform, Load) data processing: Flink SQL with JDBC can be used to extract data from multiple different data source databases (such as TDengine, MySQL, Oracle, etc.), perform transformation operations (such as data cleaning, format conversion, associating data from different tables, etc.) in Flink, and then load the processed results into the target data source (such as TDengine, MySQL, etc.). +#### Source connector + +Parameter configuration instructions: + +| Parameter Name | Type | Parameter Description | Remarks| +|-----------------------| :-----: | ------------ | ------ | +| connector | string | connector identifier, set `tdengine-connector`|| +| td.jdbc.url | string | url of the connection || +| td.jdbc.mode | strng | connector type: `source`, `cdc`, `sink`| | +| table.name | string | original or target table name || +| scan.query | string | SQL statement to retrieve data|| +| sink.db.name | string | target database name|| +| sink.superstable.name | string | write the name of the superstable|| +| sink.batch.size | integer| batch size written|| +| sink.table.name | string | name of the regular table or sub table written|| + +#### Example of using Source connector + +
+Table Source +```java +{{#include docs/examples/flink/Main.java:source_table}} +``` +
+ +#### CDC connector +Parameter configuration instructions: + +| Parameter Name | Type | Parameter Description | Remarks| +|-------------------| :-----: |--------------------------------------------------------------------------------------|-------| +| connector | string | connector identifier, set `tdengine-connector` || +| user | string | username, default root || +| password | string | password, default taosdata || +| bootstrap. servers| string | server address || +| topic | string | subscribe to topic || +| td.jdbc.mode | strng | connector type: `cdc`, `sink` | | +| group.id | string | Consumption group ID, sharing consumption progress within the same consumption group || +| auto.offset.reset | string | initial position for consumer group subscription | earliest: subscribe from scratch
latest: default; Subscribe only from the latest data| +| poll.interval_mas | integer | Pull data interval, default 500ms || +| sink.db.name | string | Target database name || +| sink.superstable.name | string | Write the name of the superstable || +| sink.batch.size | integer | batch size written || +| sink.table.name | string | Name of the regular table or sub table written || + +#### Example of using CDC connector + +
+Table CDC +```java +{{#include docs/examples/flink/Main.java:cdc_table}} +``` +
+ From fa78cda0ad2708d673fd5fd2d5265911ddfe4d12 Mon Sep 17 00:00:00 2001 From: menshibin Date: Tue, 31 Dec 2024 18:58:06 +0800 Subject: [PATCH 55/73] add english flink connector docs --- docs/en/10-third-party/01-collection/flink.md | 78 ++++++++------ docs/examples/flink/Main.java | 21 ++-- .../10-third-party/01-collection/12-flink.md | 102 +++++++++--------- 3 files changed, 106 insertions(+), 95 deletions(-) diff --git a/docs/en/10-third-party/01-collection/flink.md b/docs/en/10-third-party/01-collection/flink.md index f14ffc33a6..e716d5a757 100644 --- a/docs/en/10-third-party/01-collection/flink.md +++ b/docs/en/10-third-party/01-collection/flink.md @@ -6,7 +6,11 @@ title: TDengine Flink Connector import Tabs from '@theme/Tabs'; import TabItem from '@theme/TabItem'; -## Preconditions +Apache Flink is an open-source distributed stream batch integrated processing framework supported by the Apache Software Foundation, which can be used for many big data processing scenarios such as stream processing, batch processing, complex event processing, real-time data warehouse construction, and providing real-time data support for machine learning. At the same time, Flink has a wealth of connectors and various tools that can interface with numerous different types of data sources to achieve data reading and writing. In the process of data processing, Flink also provides a series of reliable fault-tolerant mechanisms, effectively ensuring that tasks can run stably and continuously even in the event of unexpected situations. + +With the help of TDengine's Flink connector, Apache Flink can seamlessly integrate with the TDengine database. On the one hand, it can accurately store the results obtained after complex calculations and deep analysis into the TDengine database, achieving efficient storage and management of data; On the other hand, it is also possible to quickly and stably read massive amounts of data from the TDengine database, and conduct comprehensive and in-depth analysis and processing on this basis, fully tapping into the potential value of the data, providing strong data support and scientific basis for enterprise decision-making, greatly improving the efficiency and quality of data processing, and enhancing the competitiveness and innovation ability of enterprises in the digital age. + +## Prerequisites Prepare the following environment: @@ -14,20 +18,16 @@ Prepare the following environment: - TaosAdapter can run normally. - Apache Flink v1.19.0 or above is installed. Please refer to the installation of Apache Flink [Official documents](https://flink.apache.org/) -## JRE version compatibility - -JRE: Supports JRE 8 and above versions. - ## Supported platforms Flink Connector supports all platforms that can run Flink 1.19 and above versions. ## Version History - | Flink Connector Version | Major Changes | TDengine Version| |-------------------------| ------------------------------------ | ---------------- | | 2.0.0 | 1. Support SQL queries on data in TDengine database
2 Support CDC subscription to data in TDengine database
3 Supports reading and writing to TDengine database using Table SQL | 3.3.5.0 and above versions| +| 1.0.0 | Support Sink function to write data from other sources to TDengine in the future| 3.3.2.0 and above versions| ## Exception and error codes @@ -91,13 +91,17 @@ TDengine currently supports timestamp, number, character, and boolean types, and | GEOMETRY | byte[] | ## Instructions for use + ### Flink Semantic Selection Instructions The semantic reason for using At Least One (at least once) is: + -TDengine currently does not support transactions and cannot perform frequent checkpoint operations and complex transaction coordination. -Due to TDengine's use of timestamps as primary keys, downstream operators of duplicate data can perform filtering operations to avoid duplicate calculations. -Using At Least One (at least once) to ensure high data processing performance and low data latency, the setting method is as follows: +Instructions: + ```text StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(5000); @@ -123,7 +127,8 @@ Parameter description: - User: Login TDengine username, default value is' root '. - Password: User login password, default value 'taosdata'. -- batchErrorIgnore： true： If there is an SQL execution failure in the middle of the ExecutBatch of Statement, continue to execute the following SQL. false： Do not execute any statements after failed SQL. The default value is: false。 +- database_name: database name。 +- timezone: time zone。 - HttpConnectTimeout: The connection timeout time, measured in milliseconds, with a default value of 60000. - MessageWaitTimeout: The timeout period for a message, measured in milliseconds, with a default value of 60000. - UseSSL: Whether SSL is used in the connection. @@ -281,8 +286,9 @@ The core function of Sink is to efficiently and accurately write Flink processed | TDengineConfigParams.PROPERTYKEY_RECONNECT_RETR_COUNT | integer | number of automatic reconnection retries, default value 3 | only takes effect when PROPERTYKEY-INABLE AUTO-RECONNECT is true| | TDengineConfigParams.PROPERTYKEYDISABLE_SSL_CERTVNet | boolean | Disable SSL certificate verification. true: close, false: Not closed. The default is false || -#### Use Sink connector -Write the received RowData type data into TDengine example: +Usage example: + +Write the sub table data of the meters table in the power database into the corresponding sub table of the sink_meters super table in the power_stink database.
Sink RowData @@ -291,18 +297,21 @@ Write the received RowData type data into TDengine example: ```
-Write batch received RowData data into TDengine example: +Usage example: + +Subscribe to the sub table data of the meters super table in the power database and write it to the corresponding sub table of the sink_meters super table in the power_stink database.
-Sink RowData +Cdc Sink ```java -{{#include docs/examples/flink/Main.java:BatchRowDataToSink}} +{{#include docs/examples/flink/Main.java:CdcRowDataToSink}} ```
### Table SQL -ETL (Extract, Transform, Load) data processing: Flink SQL with JDBC can be used to extract data from multiple different data source databases (such as TDengine, MySQL, Oracle, etc.), perform transformation operations (such as data cleaning, format conversion, associating data from different tables, etc.) in Flink, and then load the processed results into the target data source (such as TDengine, MySQL, etc.). +Extract data from multiple different data source databases (such as TDengine, MySQL, Oracle, etc.) using Table SQL, perform custom operator operations (such as data cleaning, format conversion, associating data from different tables, etc.), and then load the processed results into the target data source (such as TDengine, MySQL, etc.). + #### Source connector Parameter configuration instructions: @@ -311,7 +320,7 @@ Parameter configuration instructions: |-----------------------| :-----: | ------------ | ------ | | connector | string | connector identifier, set `tdengine-connector`|| | td.jdbc.url | string | url of the connection || -| td.jdbc.mode | strng | connector type: `source`, `cdc`, `sink`| | +| td.jdbc.mode | strng | connector type: `source`, `sink`| | | table.name | string | original or target table name || | scan.query | string | SQL statement to retrieve data|| | sink.db.name | string | target database name|| @@ -319,7 +328,9 @@ Parameter configuration instructions: | sink.batch.size | integer| batch size written|| | sink.table.name | string | name of the regular table or sub table written|| -#### Example of using Source connector +Usage example: + +Write the sub table data of the meters table in the power database into the corresponding sub table of the sink_meters super table in the power_stink database.
Table Source @@ -328,26 +339,29 @@ Parameter configuration instructions: ```
-#### CDC connector +#### Table CDC connector + Parameter configuration instructions: -| Parameter Name | Type | Parameter Description | Remarks| -|-------------------| :-----: |--------------------------------------------------------------------------------------|-------| -| connector | string | connector identifier, set `tdengine-connector` || -| user | string | username, default root || -| password | string | password, default taosdata || -| bootstrap. servers| string | server address || -| topic | string | subscribe to topic || -| td.jdbc.mode | strng | connector type: `cdc`, `sink` | | -| group.id | string | Consumption group ID, sharing consumption progress within the same consumption group || -| auto.offset.reset | string | initial position for consumer group subscription | earliest: subscribe from scratch
latest: default; Subscribe only from the latest data| -| poll.interval_mas | integer | Pull data interval, default 500ms || -| sink.db.name | string | Target database name || -| sink.superstable.name | string | Write the name of the superstable || -| sink.batch.size | integer | batch size written || -| sink.table.name | string | Name of the regular table or sub table written || +| Parameter Name | Type | Parameter Description | +|-------------------| :-----: |--------------------------------------------------------------------------------------| +| connector | string | connector identifier, set `tdengine-connector` | +| user | string | username, default root | +| password | string | password, default taosdata | +| bootstrap. servers| string | server address | +| topic | string | subscribe to topic | +| td.jdbc.mode | strng | connector type: `cdc`, `sink` | +| group.id | string | Consumption group ID, sharing consumption progress within the same consumption group | +| auto.offset.reset | string | initial position for consumer group subscription.
earliest: subscribe from scratch
latest: default; Subscribe only from the latest data| +| poll.interval_mas | integer | Pull data interval, default 500ms | +| sink.db.name | string | Target database name | +| sink.superstable.name | string | Write the name of the superstable | +| sink.batch.size | integer | batch size written | +| sink.table.name | string | Name of the regular table or sub table written | -#### Example of using CDC connector +Usage example: + +Subscribe to the sub table data of the meters super table in the power database and write it to the corresponding sub table of the sink_meters super table in the power_stink database.
Table CDC diff --git a/docs/examples/flink/Main.java b/docs/examples/flink/Main.java index b601778776..12d79126cf 100644 --- a/docs/examples/flink/Main.java +++ b/docs/examples/flink/Main.java @@ -414,7 +414,7 @@ splitSql.setSelect("ts, current, voltage, phase, groupid, location") Properties connProps = new Properties(); connProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); connProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); - SourceSplitSql splitSql = getTimeSplit(); + SourceSplitSql sql = new SourceSplitSql("select ts, `current`, voltage, phase, tbname from meters"); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(1); env.enableCheckpointing(1000, CheckpointingMode.AT_LEAST_ONCE); @@ -437,8 +437,8 @@ splitSql.setSelect("ts, current, voltage, phase, groupid, location") } //ANCHOR_END: RowDataToSink - //ANCHOR: BatchRowDataToSink - static void testBatchToTdSink() throws Exception { + //ANCHOR: CdcRowDataToSink + static void testCdcToSink() throws Exception { System.out.println("testTDengineCdcToTdSink start！"); StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.setParallelism(3); @@ -449,39 +449,32 @@ splitSql.setSelect("ts, current, voltage, phase, groupid, location") config.setProperty(TDengineCdcParams.CONNECT_TYPE, "ws"); config.setProperty(TDengineCdcParams.BOOTSTRAP_SERVERS, "localhost:6041"); config.setProperty(TDengineCdcParams.AUTO_OFFSET_RESET, "earliest"); - config.setProperty(TDengineCdcParams.MSG_WITH_TABLE_NAME, "true"); - config.setProperty(TDengineCdcParams.AUTO_COMMIT_INTERVAL, "1000"); config.setProperty(TDengineCdcParams.GROUP_ID, "group_1"); config.setProperty(TDengineCdcParams.CONNECT_USER, "root"); config.setProperty(TDengineCdcParams.CONNECT_PASS, "taosdata"); config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER, "RowData"); config.setProperty(TDengineCdcParams.VALUE_DESERIALIZER_ENCODING, "UTF-8"); - config.setProperty(TDengineCdcParams.TMQ_BATCH_MODE, "true"); - Class> typeClass = (Class>) (Class) ConsumerRecords.class; - TDengineCdcSource> tdengineSource = new TDengineCdcSource<>("topic_meters", config, typeClass); - DataStreamSource> input = env.fromSource(tdengineSource, WatermarkStrategy.noWatermarks(), "tdengine-source"); + TDengineCdcSource tdengineSource = new TDengineCdcSource<>("topic_meters", config, RowData.class); + DataStreamSource input = env.fromSource(tdengineSource, WatermarkStrategy.noWatermarks(), "tdengine-source"); Properties sinkProps = new Properties(); sinkProps.setProperty(TSDBDriver.PROPERTY_KEY_ENABLE_AUTO_RECONNECT, "true"); - sinkProps.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8"); sinkProps.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8"); sinkProps.setProperty(TDengineConfigParams.VALUE_DESERIALIZER, "RowData"); - sinkProps.setProperty(TDengineConfigParams.TD_BATCH_MODE, "true"); - sinkProps.setProperty(TDengineConfigParams.TD_SOURCE_TYPE, "tdengine_cdc"); sinkProps.setProperty(TDengineConfigParams.TD_DATABASE_NAME, "power_sink"); sinkProps.setProperty(TDengineConfigParams.TD_SUPERTABLE_NAME, "sink_meters"); sinkProps.setProperty(TDengineConfigParams.TD_JDBC_URL, "jdbc:TAOS-WS://localhost:6041/power?user=root&password=taosdata"); sinkProps.setProperty(TDengineConfigParams.TD_BATCH_SIZE, "2000"); - TDengineSink> sink = new TDengineSink<>(sinkProps, Arrays.asList("ts", "current", "voltage", "phase", "location", "groupid", "tbname")); + TDengineSink sink = new TDengineSink<>(sinkProps, Arrays.asList("ts", "current", "voltage", "phase", "location", "groupid", "tbname")); input.sinkTo(sink); JobClient jobClient = env.executeAsync("Flink test cdc Example"); Thread.sleep(6000L); jobClient.cancel().get(); System.out.println("testTDengineCdcToTdSink finish！"); } - //ANCHOR_END: BatchRowDataToSink + //ANCHOR_END: CdcRowDataToSink //ANCHOR: source_table static void testTableToSink() throws Exception { diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md index b58a013f38..af5434e0b3 100644 --- a/docs/zh/10-third-party/01-collection/12-flink.md +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -17,10 +17,6 @@ Apache Flink 是一款由 Apache 软件基金会支持的开源分布式流批 - taosAdapter 能够正常运行。详细参考 [taosAdapter 使用手册](../../../reference/components/taosadapter) - Apache Flink v1.19.0 或以上版本已安装。安装 Apache Flink 请参考 [官方文档](https://flink.apache.org/) -## JRE 版本兼容性 - -- JRE: 支持 JRE 8 及以上版本。 - # 支持的平台 Flink Connector 支持所有能运行 Flink 1.19 及以上版本的平台。 @@ -29,6 +25,7 @@ Flink Connector 支持所有能运行 Flink 1.19 及以上版本的平台。 | Flink Connector 版本 | 主要变化 | TDengine 版本 | | ------------------| ------------------------------------ | ---------------- | | 2.0.0 | 1. 支持 SQL 查询 TDengine 数据库中的数据
2. 支持 CDC 订阅 TDengine 数据库中的数据
3. 支持 Table SQL 方式读取和写入 TDengine 数据库| 3.3.5.0 及以上版本 | +| 1.0.0 | 支持 Sink 功能，将来着其他数据源的数据写入到 TDengine| 3.3.2.0 及以上版本| ## 异常和错误码 @@ -53,8 +50,8 @@ Flink Connector 支持所有能运行 Flink 1.19 及以上版本的平台。 | 0x2305 |resultSet is closed |resultSet 结果集已经释放，请检查 resultSet 是否释放后再次使用。| | 0x230d |parameter index out of range |参数越界，请检查参数的合理范围。| | 0x230e |connection already closed |连接已经关闭，请检查 Connection 是否关闭后再次使用，或是连接是否正常。| -| 0x230f |unknown sql type in tdengine |请检查 TDengine 支持的 Data Type 类型。| -| 0x2315 |unknown taos type in tdengine |在 TDengine 数据类型与 JDBC 数据类型转换时，是否指定了正确的 TDengine 数据类型。| +| 0x230f |unknown sql type in TDengine |请检查 TDengine 支持的 Data Type 类型。| +| 0x2315 |unknown taos type in TDengine |在 TDengine 数据类型与 JDBC 数据类型转换时，是否指定了正确的 TDengine 数据类型。| | 0x2319 |user is required |创建连接时缺少用户名信息。| | 0x231a |password is required |创建连接时缺少密码信息。| | 0x231d |can't create connection with server within |通过增加参数 httpConnectTimeout 增加连接耗时，或是请检查与 taosAdapter 之间的连接情况。| @@ -99,11 +96,14 @@ TDengine 目前支持时间戳、数字、字符、布尔类型，与 Flink RowD - TDengine 目前不支持事务，不能进行频繁的检查点操作和复杂的事务协调。 - 由于 TDengine 采用时间戳作为主键，重复数据下游算子可以进行过滤操作，避免重复计算。 - 采用 At-Least-Once（至少一次）确保达到较高的数据处理的性能和较低的数据延时，设置方式如下： - ```java - StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); - env.enableCheckpointing(5000); - env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE); - ``` + +使用方式: + +```java +StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); +env.enableCheckpointing(5000); +env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE); +``` 如果使用 Maven 管理项目，只需在 pom.xml 中加入以下依赖。 @@ -124,7 +124,8 @@ URL 规范格式为： 参数说明： - user：登录 TDengine 用户名，默认值 'root'。 - password：用户登录密码，默认值 'taosdata'。 -- batchErrorIgnore：true：在执行 Statement 的 executeBatch 时，如果中间有一条 SQL 执行失败，继续执行下面的 SQL 了。false：不再执行失败 SQL 后的任何语句。默认值为：false。 +- database_name: 数据库名称。 +- timezone: 时区设置。 - httpConnectTimeout: 连接超时时间，单位 ms， 默认值为 60000。 - messageWaitTimeout: 消息超时时间，单位 ms， 默认值为 60000。 - useSSL: 连接中是否使用 SSL。 @@ -240,7 +241,7 @@ CDC 连接器会根据用户设置的并行度进行创建 consumer，因此用 ```
-批量查询结果示例： +将订阅结果批量下发到算子的示例：
CDC Batch Source @@ -249,7 +250,7 @@ CDC 连接器会根据用户设置的并行度进行创建 consumer，因此用 ```
-查询结果为自定义数据类型示例： +订阅结果为自定义数据类型示例：
CDC Custom Type @@ -283,9 +284,9 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 | TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3。 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。| | TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| -#### 使用 Sink 连接器 +使用示例： -将接收的 RowData 类型数据写入 TDengine 示例： +将 power 库的 meters 表的子表数据，写入 power_sink 库的 sink_meters 超级表对应的子表中。
Sink RowData @@ -294,38 +295,41 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 ```
+使用示例： -将批量接收的 RowData 类型数据写入 TDengine 示例： +订阅 power 库的 meters 超级表的子表数据，写入 power_sink 库的 sink_meters 超级表对应的子表中。
-Sink RowData +Cdc Sink ```java -{{#include docs/examples/flink/Main.java:BatchRowDataToSink}} +{{#include docs/examples/flink/Main.java:CdcRowDataToSink}} ```
### Table SQL -数据处理 ETL（Extract，Transform，Load）：可以使用 Flink SQL With JDBC 从多个不同的数据源数据库（如 TDengine、MySQL、Oracle 等）中提取数据，在 Flink 中进行转换操作（如数据清洗、格式转换、关联不同表的数据等），然后将处理后的结果加载到目标数据源（如 TDengine、Mysql 等）中。 +使用 Table SQL 的方式从多个不同的数据源数据库（如 TDengine、MySQL、Oracle 等）中提取数据后， 再进行自定义的算子操作（如数据清洗、格式转换、关联不同表的数据等），然后将处理后的结果加载到目标数据源（如 TDengine、Mysql 等）中。 - -#### Source 连接器 +#### Table Source 连接器 参数配置说明： | 参数名称 | 类型 | 参数说明 | 备注 | | ----------------------- | :-----: | ------------ | ------ | -| connector | string | 连接器标识，设置 `tdengine-connector` 。|| -| td.jdbc.url| string | 连接的 url 。| | -| td.jdbc.mode | strng | 连接器类型, 设置 `source`, `cdc`, `sink`。| | -| table.name| string| 原表或目标表名称。| | -| scan.query| string| 获取数据的 SQL 语句。|| -| sink.db.name|string| 目标数据库名称。|| -| sink.supertable.name|string |写入的超级表名称。|| -| sink.batch.size | integer | 写入的批大小。|| -| sink.table.name|string|写入的普通表或子表名称。|| +| connector | string | 连接器标识，设置 `tdengine-connector` 。| +| td.jdbc.url| string | 连接的 url 。| +| td.jdbc.mode | strng | 连接器类型, 设置 `source`, `sink`。| +| table.name| string| 原表或目标表名称。| +| scan.query| string| 获取数据的 SQL 语句。| +| sink.db.name|string| 目标数据库名称。| +| sink.supertable.name|string |写入的超级表名称。| +| sink.batch.size | integer | 写入的批大小。| +| sink.table.name|string|写入的普通表或子表名称。| -#### Source 连接器使用示例 + +使用示例： + +将 power 库的 meters 表的子表数据，写入 power_sink 库的 sink_meters 超级表对应的子表中。
Table Source @@ -334,29 +338,29 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 ```
-#### CDC 连接器 +#### Table CDC 连接器 参数配置说明： -| 参数名称 | 类型 | 参数说明 | 备注 | -| ----------------------- | :-----: | ------------ |-------| -| connector | string | 连接器标识，设置 `tdengine-connector`。|| -| user| string | 用户名， 默认 root。| | -| password | string | 密码， 默认taosdata。| | -| bootstrap.servers| string | 服务器地址。| | +| 参数名称 | 类型 | 参数说明 | +| ----------------------- | :-----: | ------------ | +| connector | string | 连接器标识，设置 `tdengine-connector`。| +| user| string | 用户名， 默认 root。| +| password | string | 密码， 默认taosdata。| +| bootstrap.servers| string | 服务器地址。| | topic | string | 订阅主题。|| -| td.jdbc.mode | strng | 连接器类型, cdc, sink。| | -| group.id| string| 消费组 ID，同一消费组共享消费进度。 | | -| auto.offset.reset| string| 消费组订阅的初始位置。 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅。| -| poll.interval_ms| integer| 拉取数据间隔, 默认 500ms。| | -| sink.db.name|string| 目标数据库名称。|| -| sink.supertable.name|string |写入的超级表名称。|| -| sink.batch.size | integer | 写入的批大小。|| -| sink.table.name|string|写入的普通表或子表名称。|| +| td.jdbc.mode | strng | 连接器类型, cdc, sink。| +| group.id| string| 消费组 ID，同一消费组共享消费进度。 | +| auto.offset.reset| string| 消费组订阅的初始位置。
earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅。| +| poll.interval_ms| integer| 拉取数据间隔, 默认 500ms。| +| sink.db.name|string| 目标数据库名称。| +| sink.supertable.name|string |写入的超级表名称。| +| sink.batch.size | integer | 写入的批大小。| +| sink.table.name|string|写入的普通表或子表名称。| +使用示例： - -#### CDC 连接器使用示例 +订阅 power 库的 meters 超级表的子表数据，写入 power_sink 库的 sink_meters 超级表对应的子表中。
Table CDC From 6dbb3d1cb67dae7ac69aa5892ca4eb4d2bb4b95b Mon Sep 17 00:00:00 2001 From: Haojun Liao Date: Tue, 31 Dec 2024 19:06:26 +0800 Subject: [PATCH 56/73] test(stream): add test cases. --- tests/perf-test/stream.py | 286 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 286 insertions(+) create mode 100644 tests/perf-test/stream.py diff --git a/tests/perf-test/stream.py b/tests/perf-test/stream.py new file mode 100644 index 0000000000..a34fe5381b --- /dev/null +++ b/tests/perf-test/stream.py @@ -0,0 +1,286 @@ +import json +import subprocess + +import psutil +import time +import taos + + +class MonitorSystemLoad: + + def __init__(self, name, count) -> None: + self.pid = self.get_pid_by_name(name) + self.count = count + + def get_pid_by_name(self, name): + for proc in psutil.process_iter(['pid', 'name']): + if proc.info['name'] == name: + return proc.info['pid'] + return None + + def get_proc_status(self): + process = psutil.Process(self.pid) + + while True: + cpu_percent = process.cpu_percent(interval=1) + + memory_info = process.memory_info() + memory_percent = process.memory_percent() + + io_counters = process.io_counters() + sys_load = psutil.getloadavg() + + print("load: %s, CPU:%s, Mem:%.2f MiB(%.2f%%), Read: %.2fMiB(%d), Write: %.2fMib (%d)" % ( + sys_load, cpu_percent, memory_info.rss / 1048576.0, + memory_percent, io_counters.read_bytes / 1048576.0, io_counters.read_count, + io_counters.write_bytes / 1048576.0, io_counters.write_count)) + + time.sleep(1) + self.count -= 1 + + if self.count <= 0: + break + + +class StreamStarter: + def __init__(self) -> None: + self.sql = None + self.host='127.0.0.1' + self.user = 'root' + self.passwd = 'taosdata' + self.conf = '/etc/taos/taos.cfg' + self.tz = 'Asia/Shanghai' + + def prepare_data(self) -> dict: + json_data = { + "filetype": "insert", + "cfgdir": "/etc/taos/cfg", + "host": "127.0.0.1", + "port": 6030, + "rest_port": 6041, + "user": "root", + "password": "taosdata", + "thread_count": 20, + "create_table_thread_count": 40, + "result_file": "/tmp/taosBenchmark_result.log", + "confirm_parameter_prompt": "no", + "insert_interval": 0, + "num_of_records_per_req": 10000, + "max_sql_len": 1024000, + "databases": [ + { + "dbinfo": { + "name": "stream_test", + "drop": "yes", + "replica": 1, + "duration": 10, + "precision": "ms", + "keep": 3650, + "minRows": 100, + "maxRows": 4096, + "comp": 2, + "vgroups": 10, + "stt_trigger": 1, + "WAL_RETENTION_PERIOD": 86400 + }, + "super_tables": [ + { + "name": "stb", + "child_table_exists": "yes", + "childtable_count": 500, + "childtable_prefix": "ctb0_", + "escape_character": "no", + "auto_create_table": "yes", + "batch_create_tbl_num": 1000, + "data_source": "rand", + "insert_mode": "taosc", + "interlace_rows": 400, + "tcp_transfer": "no", + "insert_rows": 10000, + "partial_col_num": 0, + "childtable_limit": 0, + "childtable_offset": 0, + "rows_per_tbl": 0, + "max_sql_len": 1024000, + "disorder_ratio": 0, + "disorder_range": 1000, + "keep_trying": -1, + "timestamp_step": 1000, + "trying_interval": 10, + "start_timestamp": "2021-01-01 00:00:00", + "sample_format": "csv", + "sample_file": "./sample.csv", + "tags_file": "", + "columns": [ + { + "type": "INT", + "count": 1 + }, + { + "type": "TINYINT", + "count": 0 + }, + { + "type": "DOUBLE", + "count": 0 + }, + { + "type": "VARCHAR", + "count": 0, + "len": 16 + }, + { + "type": "NCHAR", + "count": 0, + "len": 4 + }, + { + "type": "SMALLINT", + "count": 0 + }, + { + "type": "BIGINT", + "count": 0 + }, + { + "type": "UTINYINT", + "count": 0 + }, + { + "type": "USMALLINT", + "count": 0 + }, + { + "type": "UINT", + "count": 0 + }, + { + "type": "UBIGINT", + "count": 0 + }, + { + "type": "FLOAT", + "count": 2 + }, + { + "type": "BINARY", + "count": 0, + "len": 8 + }, + { + "type": "BOOL", + "count": 0 + }, + { + "type": "TIMESTAMP", + "count": 1 + } + ], + "tags": [ + { + "type": "INT", + "count": 0 + }, + { + "type": "TINYINT", + "count": 1 + }, + { + "type": "DOUBLE", + "count": 0 + }, + { + "type": "VARCHAR", + "count": 0, + "len": 8 + }, + { + "type": "NCHAR", + "count": 0, + "len": 16 + }, + { + "type": "SMALLINT", + "count": 0 + }, + { + "type": "BIGINT", + "count": 0 + }, + { + "type": "UTINYINT", + "count": 0 + }, + { + "type": "USMALLINT", + "count": 0 + }, + { + "type": "UINT", + "count": 0 + }, + { + "type": "UBIGINT", + "count": 0 + }, + { + "type": "FLOAT", + "count": 0 + }, + { + "type": "BINARY", + "count": 1, + "len": 16 + }, + { + "type": "BOOL", + "count": 0 + }, + { + "type": "TIMESTAMP", + "count": 0 + } + ] + } + ] + } + ], + "prepare_rand": 10000, + "chinese": "no", + "test_log": "/tmp/testlog/" + } + + with open('/tmp/stream.json', 'w+') as f: + json.dump(json_data, f, indent=4) + + def do_start(self): + self.prepare_data() + + try: + subprocess.Popen('taosBenchmark --f /tmp/stream.json', stdout=subprocess.PIPE, shell=True, text=True) + except subprocess.CalledProcessError as e: + print(f"Error running Bash command: {e}") + + conn = taos.connect( + host=self.host, user=self.user, password=self.passwd, config=self.conf, timezone=self.tz + ) + + time.sleep(10) + + print("start to connect db") + cursor = conn.cursor() + + cursor.execute('use stream_test') + + sql = "create stream str1 ignore update 0 into str1_dst as select _wstart as wstart, min(c1),max(c2), count(c3) from stream_test.stb partition by cast(t1 as int) t1,tbname interval(5s)" + cursor.execute(sql) + + print("create stream completed, start to monitor system load") + conn.close() + + loader = MonitorSystemLoad('taosd', 80) + loader.get_proc_status() + + +if __name__ == "__main__": + StreamStarter().do_start() From 687406d5569ccf024d7d557deffecbf06a6a7c21 Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Wed, 1 Jan 2025 13:04:21 +0800 Subject: [PATCH 57/73] chore: adjust badges --- README.md | 10 ++++------ 1 file changed, 4 insertions(+), 6 deletions(-) diff --git a/README.md b/README.md index ed04fafee1..e4814cee67 100644 --- a/README.md +++ b/README.md @@ -1,4 +1,3 @@ -

-

-[![Build Status](https://github.com/taosdata/TDengine/actions/workflows/taosd-ci-build.yml/badge.svg)](https://github.com/taosdata/TDengine/actions/workflows/taosd-ci-build.yml) +[![GitHub Actions Workflow Status](https://img.shields.io/github/actions/workflow/status/taosdata/tdengine/taosd-ci-build.yml)](https://github.com/taosdata/TDengine/actions/workflows/taosd-ci-build.yml) [![Coverage Status](https://coveralls.io/repos/github/taosdata/TDengine/badge.svg?branch=3.0)](https://coveralls.io/github/taosdata/TDengine?branch=3.0) +![GitHub commit activity](https://img.shields.io/github/commit-activity/m/taosdata/tdengine) +
+![GitHub Release](https://img.shields.io/github/v/release/taosdata/tdengine) ![GitHub License](https://img.shields.io/github/license/taosdata/tdengine) [![CII Best Practices](https://bestpractices.coreinfrastructure.org/projects/4201/badge)](https://bestpractices.coreinfrastructure.org/projects/4201)
@@ -20,9 +21,6 @@ [![Discord Community](https://img.shields.io/badge/Join_Discord--white?logo=discord&style=social)](https://discord.com/invite/VZdSuUg4pS) [![LinkedIn](https://img.shields.io/badge/Follow_LinkedIn--white?logo=linkedin&style=social)](https://www.linkedin.com/company/tdengine) [![StackOverflow](https://img.shields.io/badge/Ask_StackOverflow--white?logo=stackoverflow&style=social&logoColor=orange)](https://stackoverflow.com/questions/tagged/tdengine) -
-![GitHub Release](https://img.shields.io/github/v/release/taosdata/tdengine) -![GitHub commit activity](https://img.shields.io/github/commit-activity/m/taosdata/tdengine) English | [简体中文](README-CN.md) | [TDengine Cloud](https://cloud.tdengine.com) | [Learn more about TSDB](https://tdengine.com/tsdb/) From b999303ac868d76f0e05f3496836b1a1e702035e Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Wed, 1 Jan 2025 13:14:35 +0800 Subject: [PATCH 58/73] docs: add 33~35 FAQ problem --- docs/en/27-train-faq/index.md | 19 +++++++++++++++++++ docs/zh/27-train-faq/01-faq.md | 10 ++++++++++ 2 files changed, 29 insertions(+) diff --git a/docs/en/27-train-faq/index.md b/docs/en/27-train-faq/index.md index ca6cd91714..aa85729102 100644 --- a/docs/en/27-train-faq/index.md +++ b/docs/en/27-train-faq/index.md @@ -297,3 +297,22 @@ Reporting this error indicates that the first connection to the cluster was succ Therefore, first, check whether all ports on the server and cluster (default 6030 for native connections and 6041 for HTTP connections) are open; Next, check if the client's hosts file has configured the fqdn and IP information for all dnodes in the cluster. If the issue still cannot be resolved, it is necessary to contact Taos technical personnel for support. + +### 32 Why is the original database lost and the cluster ID changed when the data directory dataDir of the database remains unchanged on the same server? +Background: When the TDengine server process (taosd) starts, if there are no valid data file subdirectories (such as mnode, dnode, and vnode) under the data directory (dataDir, which is specified in the configuration file taos.cfg), these directories will be created automatically.When a new mnode directory is created, a new cluster ID will be allocated to generate a new cluster. + +Cause analysis: The data directory dataDir of taosd can point to multiple different mount points.If these mount points are not configured for automatic mounting in the fstab file, after the server restarts, dataDir will only exist as a normal directory of the local disk, and it will not point to the mounted disk as expected.At this point, if the taosd service is started, it will create a new directory under dataDir to generate a new cluster. + +Impact of the problem: After the server is restarted, the original database is lost (note: it is not really lost, but the original data disk is not attached and cannot be seen for the time being) and the cluster ID changes, resulting in the inability to access the original database. For enterprise users, if they have been authorized for the cluster ID, they will also find that the machine code of the cluster server has not changed, but the original authorization has expired.If the problem is not monitored or found and handled in time, the user will not notice that the original database has been lost, resulting in losses and increased operation and maintenance costs. + +Problem solving: You should configure the automatic mount of the dataDir directory in the fstab file to ensure that the dataDir always points to the expected mount point and directory. At this point, restarting the server will retrieve the original database and cluster. In the subsequent version, we will develop a function to enable taosd to exit in the startup phase when it detects that the dataDir changes before and after startup, and provide corresponding error prompts. + +### 33 How to solve MVCP1400.DLL loss when running TDengine on Windows platform? +1. Reinstall Microsoft Visual C++ Redistributable: As msvcp140.dll is part of Microsoft Visual C++Redistributable, reinstalling this package usually resolves most issues. You can download the corresponding version from the official Microsoft website for installation +2. Manually download and replace the msvcp140.dll file online: You can download the msvcp140.dll file from a reliable source and copy it to the corresponding directory in the system. Ensure that the downloaded files match your system architecture (32-bit or 64 bit) and ensure the security of the source + +### 34 Which fast query data from super table with TAG filter or child table ? +Directly querying from child table is fast. The query from super table with TAG filter is designed to meet the convenience of querying. It can filter data from multiple child tables at the same time. If the goal is to pursue performance and the child table has been clearly queried, directly querying from the sub table can achieve higher performance + +### 35 How to view data compression ratio indicators? +Currently, TDengine only provides compression ratios based on tables, not databases or the entire system. To view the compression ratios, execute the `SHOW TABLE DISTRIBUTED table_name;` command in the client taos-CLI. The table_name can be a super table, regular table, or subtable. For details [Click Here](https://docs.tdengine.com/tdengine-reference/sql-manual/show-commands/#show-table-distributed) \ No newline at end of file diff --git a/docs/zh/27-train-faq/01-faq.md b/docs/zh/27-train-faq/01-faq.md index 8b4cabf520..1b2f2c9fcc 100644 --- a/docs/zh/27-train-faq/01-faq.md +++ b/docs/zh/27-train-faq/01-faq.md @@ -302,3 +302,13 @@ TDinsight插件中展示的数据是通过taosKeeper和taosAdapter服务收集 问题影响：服务器重启后，原有数据库丢失（注：并非真正丢失，只是原有的数据磁盘未挂载，暂时看不到）且集群 ID 发生变化，导致无法访问原有数据库。对于企业版用户，如果已针对集群 ID 进行授权，还会发现集群服务器的机器码未变，但原有的授权已失效。如果未针对该问题进行监控或者未及时发现并进行处理，则用户不会注意到原有数据库已经丢失，从而造成损失，增加运维成本。 问题解决：应在 fstab 文件中配置 dataDir 目录的自动挂载，确保 dataDir 始终指向预期的挂载点和目录，此时，再重启服务器，会找回原有的数据库和集群。在后续的版本中，我们将开发一个功能，使 taosd 在检测到启动前后 dataDir 发生变化时，在启动阶段退出，同时提供相应的错误提示。 + +### 33 Windows 平台运行 TDengine 出现丢失 MVCP1400.DLL 解决方法？ +1. 重新安装 Microsoft Visual C++ Redistributable‌：由于 msvcp140.dll 是 Microsoft Visual C++ Redistributable 的一部分，重新安装这个包通常可以解决大部分问题。可以从 Microsoft 官方网站下载相应的版本进行安装‌ +2. 手动上网下载并替换 msvcp140.dll 文件‌：可以从可靠的源下载 msvcp140.dll 文件，并将其复制到系统的相应目录下。确保下载的文件与您的系统架构（32位或64位）相匹配，并确保来源的安全性‌ + +### 34 超级表带 TAG 过滤查子查数据与直接查子表哪个块？ +直接查子表更快。超级表带 TAG 过滤查询子查数据是为满足查询方便性，同时可对多个子表中数据进行过滤，如果目的是追求性能并已明确查询子表，直接从子表查性能更高 + +### 35 如何查看数据压缩率指标？ +TDengine 目前只提供以表为统计单位的压缩率，数据库及整体还未提供，查看命令是在客户端 taos-CLI 中执行 `SHOW TABLE DISTRIBUTED table_name;` 命令，table_name 为要查看压缩率的表，可以为超级表、普通表及子表，详细可[查看此处](https://docs.taosdata.com/reference/taos-sql/show/#show-table-distributed) \ No newline at end of file From 06a28cbe330f683154c0f3c89d06058bff08cbbf Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Wed, 1 Jan 2025 13:26:09 +0800 Subject: [PATCH 59/73] chore: correct typo in workflow name --- .github/workflows/taoskeeper-ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/taoskeeper-ci.yml b/.github/workflows/taoskeeper-ci.yml index fbc662ffb2..7f84eaa401 100644 --- a/.github/workflows/taoskeeper-ci.yml +++ b/.github/workflows/taoskeeper-ci.yml @@ -1,4 +1,4 @@ -name: TaosKeeper CI +name: taosKeeper CI on: push: From b941f3654312d860d86352d07bcb2de04e5436a3 Mon Sep 17 00:00:00 2001 From: WANG Xu Date: Wed, 1 Jan 2025 13:26:38 +0800 Subject: [PATCH 60/73] chore: delete lgtm config --- .lgtm.yml | 402 ------------------------------------------------------ 1 file changed, 402 deletions(-) delete mode 100644 .lgtm.yml diff --git a/.lgtm.yml b/.lgtm.yml deleted file mode 100644 index fbcedead43..0000000000 --- a/.lgtm.yml +++ /dev/null @@ -1,402 +0,0 @@ -########################################################################################## -# Customize file classifications. # -# Results from files under any classifier will be excluded from LGTM # -# statistics. # -########################################################################################## - -########################################################################################## -# Use the `path_classifiers` block to define changes to the default classification of # -# files. # -########################################################################################## - -path_classifiers: - # docs: - # Identify the top-level file called `generate_javadoc.py` as documentation-related. - test: - # Override LGTM's default classification of test files by excluding all files. - - exclude: / - # Classify all files in the top-level directories tests/ and testsuites/ as test code. - - tests - # - testsuites - # Classify all files with suffix `.test` as test code. - # Note: use only forward slash / as a path separator. - # Use ** to indicate an arbitrary parent path. - # Use * to indicate any sequence of characters excluding /. - # Always enclose the expression in double quotes if it includes *. - # - "**/*.test" - # Refine the classifications above by excluding files in test/util/. - # - exclude: test/util - # The default behavior is to tag all files created during the - # build as `generated`. Results are hidden for generated code. You can tag - # further files as being generated by adding them to the `generated` section. - generated: - # Exclude all `*.c` files under the `ui/` directory from classification as - # generated code. - # - exclude: ui/**/*.c - # By default, all files not checked into the repository are considered to be - # 'generated'. - # The default behavior is to tag library code as `library`. Results are hidden - # for library code. You can tag further files as being library code by adding them - # to the `library` section. - library: - - exclude: deps/ - # The default behavior is to tag template files as `template`. Results are hidden - # for template files. You can tag further files as being template files by adding - # them to the `template` section. - template: - #- exclude: path/to/template/code/**/*.c - # Define your own category, for example: 'some_custom_category'. - some_custom_category: - # Classify all files in the top-level directory tools/ (or the top-level file - # called tools). - # - tools - -######################################################################################### -# Use the `queries` block to change the default display of query results. # -######################################################################################### - - # queries: - # Start by hiding the results of all queries. - # - exclude: "*" - # Then include all queries tagged 'security' and 'correctness', and with a severity of - # 'error'. - # - include: - # tags: - # - "security" - # - "correctness" - # severity: "error" - # Specifically hide the results of two queries. - # - exclude: cpp/use-of-goto - # - exclude: java/equals-on-unrelated-types - # Refine by including the `java/command-line-injection` query. - # - include: java/command-line-injection - -######################################################################################### -# Define changes to the default code extraction process. # -# Each block configures the extraction of a single language, and modifies actions in a # -# named step. Every named step includes automatic default actions, # -# except for the 'prepare' step. The steps are performed in the following sequence: # -# prepare # -# after_prepare # -# configure (C/C++ only) # -# python_setup (Python only) # -# before_index # -# index # -########################################################################################## - -######################################################################################### -# Environment variables available to the steps: # -######################################################################################### - -# LGTM_SRC -# The root of the source tree. -# LGTM_WORKSPACE -# An existing (initially empty) folder outside the source tree. -# Used for temporary download and setup commands. - -######################################################################################### -# Use the extraction block to define changes to the default code extraction process # -# for one or more languages. The settings for each language are defined in a child # -# block, with one or more steps. # -######################################################################################### - -extraction: - # Define settings for C/C++ analysis - ##################################### - cpp: - # The `prepare` step exists for customization on LGTM.com only. - prepare: - # # The `packages` section is valid for LGTM.com only. It names Ubuntu packages to - # # be installed. - packages: - - cmake - # Add an `after-prepare` step if you need to run commands after the prepare step. - # Each command should be listed on a separate line. - # This step is useful for C/C++ analysis where you want to prepare the environment - # for the `configure` step without changing the default behavior for that step. - # after_prepare: - #- export GNU_MAKE=make - #- export GIT=true - # The `configure` step generates build configuration files which the `index` step - # then uses to build the codebase. - configure: - command: - - mkdir build - - cd build - - cmake .. - # - ./prepare_deps - # Optional step. You should add a `before_index` step if you need to run commands - # before the `index` step. - # before_index: - # - export BOOST_DIR=$LGTM_SRC/boost - # - export GTEST_DIR=$LGTM_SRC/googletest - # - export HUNSPELL_DIR=$LGTM_SRC/hunspell - # - export CRYPTOPP_DIR=$LGTM_SRC/cryptopp - # The `index` step builds the code and extracts information during the build - # process. - index: - # Override the autobuild process by specifying a list of custom build commands - # to use instead. - build_command: - - cd build - - make - # - $GNU_MAKE -j2 -s - # Specify that all project or solution files should be used for extraction. - # Default: false. - # all_solutions: true - # Specify a list of one or more project or solution files for extraction. - # Default: LGTM chooses the file closest to the root of the repository (this may - # fail if there are multiple candidates). - # solution: - # - myProject.sln - # Specify MSBuild settings - # msbuild: - # Specify a list of additional arguments to MSBuild. Default: empty. - # arguments: /p:Platform=x64 /p:Configuration=Release - # Specify the MSBuild configuration to use, for example, debug or release. - # Default: read from the solution file or files. - # configuration: - # Specify the platform to target, for example: x86, x64, or Any CPU. - # Default: read from the solution file or files. - # platform: - # Specify the MSBuild target. Default: rebuild. - # target: - # Specify whether or not to perform a NuGet restore for extraction. Default: true. - # nuget_restore: false - # Specify a version of Microsoft Visual Studio to use for MSBuild or any custom - # build commands (build_command). For example: - # 10 for Visual Studio 2010 - # 12 for Visual Studio 2012 - # 14 for Visual Studio 2015 - # 15 for Visual Studio 2017 - # Default: read from project files. - # vstools_version: 10 - - # Define settings for C# analysis - ################################## - # csharp: - # The `prepare` step exists for customization on LGTM.com only. - # prepare: - # packages: - # - example_package - # Add an `after-prepare` step if you need to run commands after the `prepare` step. - # Each command should be listed on a separate line. - # after_prepare: - # - export PATH=$LGTM_WORKSPACE/tools:$PATH - # The `index` step builds the code and extracts information during the build - # process. - #index: - # Specify that all project or solution files should be used for extraction. - # Default: false. - # all_solutions: true - # Specify a list of one or more project or solution files for extraction. - # Default: LGTM chooses the file closest to the root of the repository (this may - # fail if there are multiple candidates). - # solution: - # - myProject.sln - # Override the autobuild process by specifying a list of custom build commands - # to use instead. - # build_command: - # - ./example-compile-all.sh - # By default, LGTM analyzes the code by building it. You can override this, - # and tell LGTM not to build the code. Beware that this can lead - # to less accurate results. - # buildless: true - # Specify .NET Core settings. - # dotnet: - # Specify additional arguments to `dotnet build`. - # Default: empty. - # arguments: "example_arg" - # Specify the version of .NET Core SDK to use. - # Default: The version installed on the build machine. - # version: 2.1 - # Specify MSBuild settings. - # msbuild: - # Specify a list of additional arguments to MSBuild. Default: empty. - # arguments: /P:WarningLevel=2 - # Specify the MSBuild configuration to use, for example, debug or release. - # Default: read from the solution file or files. - # configuration: release - # Specify the platform to target, for example: x86, x64, or Any CPU. - # Default: read from the solution file or files. - # platform: x86 - # Specify the MSBuild target. Default: rebuild. - # target: notest - # Specify whether or not to perform a NuGet restore for extraction. Default: true. - # nuget_restore: false - # Specify a version of Microsoft Visual Studio to use for MSBuild or any custom - # build commands (build_command). For example: - # 10 for Visual Studio 2010 - # 12 for Visual Studio 2012 - # 14 for Visual Studio 2015 - # 15 for Visual Studio 2017 - # Default: read from project files - # vstools_version: 10 - # Specify additional options for the extractor, - # for example --fast to perform a faster extraction that produces a smaller - # database. - # extractor: "--fast" - - # Define settings for Go analysis - ################################## - # go: - # The `prepare` step exists for customization on LGTM.com only. - # prepare: - # packages: - # - example_package - # Add an `after-prepare` step if you need to run commands after the `prepare` step. - # Each command should be listed on a separate line. - # after_prepare: - # - export PATH=$LGTM_WORKSPACE/tools:$PATH - # The `index` step builds the code and extracts information during the build - # process. - # index: - # Override the autobuild process by specifying a list of custom build commands - # to use instead. - # build_command: - # - ./compile-all.sh - - # Define settings for Java analysis - #################################### - # java: - # The `prepare` step exists for customization on LGTM.com only. - # prepare: - # packages: - # - example_package - # Add an `after-prepare` step if you need to run commands after the prepare step. - # Each command should be listed on a separate line. - # after_prepare: - # - export PATH=$LGTM_WORKSPACE/tools:$PATH - # The `index` step extracts information from the files in the codebase. - # index: - # Specify Gradle settings. - # gradle: - # Specify the required Gradle version. - # Default: determined automatically. - # version: 4.4 - # Override the autobuild process by specifying a list of custom build commands - # to use instead. - # build_command: ./compile-all.sh - # Specify the Java version required to build the project. - # java_version: 11 - # Specify whether to extract Java .properties files - # Default: false - # properties_files: true - # Specify Maven settings. - # maven: - # Specify the path (absolute or relative) of a Maven settings file to use. - # Default: Maven uses a settings file in the default location, if it exists. - # settings_file: /opt/share/settings.xml - # Specify the path of a Maven toolchains file. - # Default: Maven uses a toolchains file in the default location, if it exists. - # toolchains_file: /opt/share/toolchains.xml - # Specify the required Maven version. - # Default: the Maven version is determined automatically, where feasible. - # version: 3.5.2 - # Specify how XML files should be extracted: - # all = extract all XML files. - # default = only extract XML files named `AndroidManifest.xml`, `pom.xml`, and `web.xml`. - # disabled = do not extract any XML files. - # xml_mode: all - - # Define settings for JavaScript analysis - ########################################## - # javascript: - # The `prepare` step exists for customization on LGTM.com only. - # prepare: - # packages: - # - example_package - # Add an `after-prepare` step if you need to run commands after the prepare step. - # Each command should be listed on a separate line. - # after_prepare: - # - export PATH=$LGTM_WORKSPACE/tools:$PATH - # The `index` step extracts information from the files in the codebase. - # index: - # Specify a list of files and folders to extract. - # Default: The project root directory. - # include: - # - src/js - # Specify a list of files and folders to exclude from extraction. - # exclude: - # - thirdparty/lib - # You can add additional file types for LGTM to extract, by mapping file - # extensions (including the leading dot) to file types. The usual - # include/exclude patterns apply, so, for example, `.jsm` files under - # `thirdparty/lib` will not be extracted. - # filetypes: - # ".jsm": "js" - # ".tmpl": "html" - # Specify a list of glob patterns to include/exclude files from extraction; this - # is applied on top of the include/exclude paths from above; patterns are - # processed in the same way as for path classifiers above. - # Default: include all files with known extensions (such as .js, .ts and .html), - # but exclude files ending in `-min.js` or `.min.js` and folders named `node_modules` - # or `bower_components` - # filters: - # exclude any *.ts files anywhere. - # - exclude: "**/*.ts" - # but include *.ts files under src/js/typescript. - # - include: "src/js/typescript/**/*.ts" - # Specify how TypeScript files should be extracted: - # none = exclude all TypeScript files. - # basic = extract syntactic information from TypeScript files. - # full = extract syntactic and type information from TypeScript files. - # Default: full. - # typescript: basic - # By default, LGTM doesn't extract any XML files. You can override this by - # using the `xml_mode` property and setting it to `all`. - # xml_mode: all - - # Define settings for Python analysis - ###################################### - # python: - # # The `prepare` step exists for customization on LGTM.com only. - # # prepare: - # # # The `packages` section is valid for LGTM.com only. It names packages to - # # # be installed. - # # packages: libpng-dev - # # This step is useful for Python analysis where you want to prepare the - # # environment for the `python_setup` step without changing the default behavior - # # for that step. - # after_prepare: - # - export PATH=$LGTM_WORKSPACE/tools:$PATH - # # This sets up the Python interpreter and virtual environment, ready for the - # # `index` step to extract the codebase. - # python_setup: - # # Specify packages that should NOT be installed despite being mentioned in the - # # requirements.txt file. - # # Default: no package marked for exclusion. - # exclude_requirements: - # - pywin32 - # # Specify a list of pip packages to install. - # # If any of these packages cannot be installed, the extraction will fail. - # requirements: - # - Pillow - # # Specify a list of requirements text files to use to set up the environment, - # # or false for none. Default: any requirements.txt, test-requirements.txt, - # # and similarly named files identified in the codebase are used. - # requirements_files: - # - required-packages.txt - # # Specify a setup.py file to use to set up the environment, or false for none. - # # Default: any setup.py files identified in the codebase are used in preference - # # to any requirements text files. - # setup_py: new-setup.py - # # Override the version of the Python interpreter used for setup and extraction - # # Default: Python 3. - # version: 2 - # # Optional step. You should add a `before_index` step if you need to run commands - # # before the `index` step. - # before_index: - # - antlr4 -Dlanguage=Python3 Grammar.g4 - # # The `index` step extracts information from the files in the codebase. - # index: - # # Specify a list of files and folders to exclude from extraction. - # # Default: Git submodules and Subversion externals. - # exclude: - # - legacy-implementation - # - thirdparty/libs - # filters: - # - exclude: "**/documentation/examples/snippets/*.py" - # - include: "**/documentation/examples/test_application/*" - # include: - # - example/to/include From 5bef1dc3bd4c7a5d96307e4809ab68fe7f544bf5 Mon Sep 17 00:00:00 2001 From: menshibin Date: Wed, 1 Jan 2025 16:24:35 +0800 Subject: [PATCH 61/73] modify flink connector docs param note --- docs/en/10-third-party/01-collection/flink.md | 205 +++++++++--------- .../10-third-party/01-collection/12-flink.md | 107 ++++----- 2 files changed, 158 insertions(+), 154 deletions(-) diff --git a/docs/en/10-third-party/01-collection/flink.md b/docs/en/10-third-party/01-collection/flink.md index e716d5a757..f586b36d0e 100644 --- a/docs/en/10-third-party/01-collection/flink.md +++ b/docs/en/10-third-party/01-collection/flink.md @@ -26,7 +26,7 @@ Flink Connector supports all platforms that can run Flink 1.19 and above version | Flink Connector Version | Major Changes | TDengine Version| |-------------------------| ------------------------------------ | ---------------- | -| 2.0.0 | 1. Support SQL queries on data in TDengine database
2 Support CDC subscription to data in TDengine database
3 Supports reading and writing to TDengine database using Table SQL | 3.3.5.0 and above versions| +| 2.0.0 | 1.Support SQL queries on data in TDengine database
2 Support CDC subscription to data in TDengine database
3 Supports reading and writing to TDengine database using Table SQL | 3.3.5.0 and above versions| | 1.0.0 | Support Sink function to write data from other sources to TDengine in the future| 3.3.2.0 and above versions| ## Exception and error codes @@ -36,39 +36,39 @@ Please refer to: | Error Code | Description | Suggested Actions | | ---------------- |------------------------------------------------------- | -------------------- | -|0xa000 | connection param error | connector parameter error -|0xa001 | The groupid parameter of CDC is incorrect | The groupid parameter of CDC is incorrect| -|0xa002 | wrong topic parameter for CDC | The topic parameter for CDC is incorrect| -|0xa010 | database name configuration error | database name configuration error| -|0xa011 | Table name configuration error | Table name configuration error| -|0xa012 | No data was obtained from the data source | Failed to retrieve data from the data source| -|0xa013 | value.deserializer parameter not set | No serialization method set| -|0xa014 | List of column names for target table not set | List of column names for target table not set || -|0x2301 | Connection already closed | The connection has been closed. Check the connection status or create a new connection to execute the relevant instructions| -|0x2302 | this operation is NOT supported currently | The current interface is not supported, you can switch to other connection methods| -|0x2303 | invalid variables | The parameter is invalid. Please check the corresponding interface specification and adjust the parameter type and size| -|0x2304 | Statement is closed | Statement has already been closed. Please check if the statement is closed and reused, or if the connection is working properly| -|0x2305 | ResultSet is closed | The ResultSet has been released. Please check if the ResultSet has been released and used again| -|0x230d | parameter index out of range | parameter out of range, please check the reasonable range of the parameter| -|0x230e | Connection already closed | The connection has been closed. Please check if the connection is closed and used again, or if the connection is working properly| -|0x230f | unknown SQL type in TDengine | Please check the Data Type types supported by TDengine| -|0x2315 | unknown tao type in TDengine | Did the correct TDengine data type be specified when converting TDengine data type to JDBC data type| -|0x2319 | user is required | Username information is missing when creating a connection| -|0x231a | password is required | Password information is missing when creating a connection| -|0x231d | can't create connection with server within | Increase connection time by adding the parameter httpConnectTimeout, or check the connection status with taosAdapter| -|0x231e | failed to complete the task within the specified time | Increase execution time by adding the parameter messageWaitTimeout, or check the connection with taosAdapter| -|0x2352 | Unsupported encoding | An unsupported character encoding set was specified under the local connection| -|0x2353 |internal error of database, Please see taoslog for more details | An error occurred while executing prepareStatement on the local connection. Please check the taoslog for problem localization| -|0x2354 | Connection is NULL | Connection has already been closed while executing the command on the local connection. Please check the connection with TDengine| -|0x2355 | result set is NULL | Local connection to obtain result set, result set exception, please check connection status and retry| -|0x2356 | invalid num of fields | The meta information obtained from the local connection result set does not match| -|0x2357 | empty SQL string | Fill in the correct SQL for execution| -|0x2371 |consumer properties must not be null | When creating a subscription, the parameter is empty. Please fill in the correct parameter| -|0x2375 | Topic reference has been destroyed | During the process of creating a data subscription, the topic reference was released. Please check the connection with TDengine| -|0x2376 |failed to set consumer topic, Topic name is empty | During the process of creating a data subscription, the subscription topic name is empty. Please check if the specified topic name is filled in correctly| -|0x2377 | Consumer reference has been destroyed | The subscription data transmission channel has been closed, please check the connection with TDengine| -|0x2378 | Consumer create error | Failed to create data subscription. Please check the taos log based on the error message to locate the problem| -|0x237a | vGroup not found in result set VGroup | Not assigned to the current consumer, due to the Rebalance mechanism, the relationship between Consumer and VGroup is not bound| +|0xa000 | connection param error | Connector parameter error. +|0xa001 | the groupid parameter of CDC is incorrect | The groupid parameter of CDC is incorrect.| +|0xa002 | wrong topic parameter for CDC | The topic parameter for CDC is incorrect.| +|0xa010 | database name configuration error | database name configuration error.| +|0xa011 | table name configuration error | Table name configuration error.| +|0xa012 | no data was obtained from the data source | Failed to retrieve data from the data source.| +|0xa013 | value.deserializer parameter not set | No serialization method set.| +|0xa014 | list of column names set incorrectly | List of column names for target table not set. | +|0x2301 | connection already closed | The connection has been closed. Check the connection status or create a new connection to execute the relevant instructions.| +|0x2302 | this operation is NOT supported currently | The current interface is not supported, you can switch to other connection methods.| +|0x2303 | invalid variables | The parameter is invalid. Please check the corresponding interface specification and adjust the parameter type and size.| +|0x2304 | statement is closed | Statement has already been closed. Please check if the statement is closed and reused, or if the connection is working properly.| +|0x2305 | resultSet is closed | The ResultSet has been released. Please check if the ResultSet has been released and used again.| +|0x230d | parameter index out of range | parameter out of range, please check the reasonable range of the parameter.| +|0x230e | connection already closed | The connection has been closed. Please check if the connection is closed and used again, or if the connection is working properly.| +|0x230f | unknown SQL type in TDengine | Please check the Data Type types supported by TDengine.| +|0x2315 | unknown tao type in TDengine | Did the correct TDengine data type be specified when converting TDengine data type to JDBC data type.| +|0x2319 | user is required | Username information is missing when creating a connection.| +|0x231a | password is required | Password information is missing when creating a connection.| +|0x231d | can't create connection with server within | Increase connection time by adding the parameter httpConnectTimeout, or check the connection status with taosAdapter.| +|0x231e | failed to complete the task within the specified time | Increase execution time by adding the parameter messageWaitTimeout, or check the connection with taosAdapter.| +|0x2352 | unsupported encoding | An unsupported character encoding set was specified under the local connection.| +|0x2353 | internal error of database, Please see taoslog for more details | An error occurred while executing prepareStatement on the local connection. Please check the taoslog for problem localization.| +|0x2354 | connection is NULL | Connection has already been closed while executing the command on the local connection. Please check the connection with TDengine.| +|0x2355 | result set is NULL | Local connection to obtain result set, result set exception, please check connection status and retry.| +|0x2356 | invalid num of fields | The meta information obtained from the local connection result set does not match.| +|0x2357 | empty SQL string | Fill in the correct SQL for execution.| +|0x2371 | consumer properties must not be null | When creating a subscription, the parameter is empty. Please fill in the correct parameter.| +|0x2375 | topic reference has been destroyed | During the process of creating a data subscription, the topic reference was released. Please check the connection with TDengine.| +|0x2376 | failed to set consumer topic, Topic name is empty | During the process of creating a data subscription, the subscription topic name is empty. Please check if the specified topic name is filled in correctly.| +|0x2377 | consumer reference has been destroyed | The subscription data transmission channel has been closed, please check the connection with TDengine.| +|0x2378 | consumer create error | Failed to create data subscription. Please check the taos log based on the error message to locate the problem.| +|0x237a | vGroup not found in result set VGroup | Not assigned to the current consumer, due to the Rebalance mechanism, the relationship between Consumer and VGroup is not bound.| ## Data type mapping @@ -96,13 +96,13 @@ TDengine currently supports timestamp, number, character, and boolean types, and The semantic reason for using At Least One (at least once) is: --TDengine currently does not support transactions and cannot perform frequent checkpoint operations and complex transaction coordination. --Due to TDengine's use of timestamps as primary keys, downstream operators of duplicate data can perform filtering operations to avoid duplicate calculations. --Using At Least One (at least once) to ensure high data processing performance and low data latency, the setting method is as follows: +- TDengine currently does not support transactions and cannot perform frequent checkpoint operations and complex transaction coordination. +- Due to TDengine's use of timestamps as primary keys, downstream operators of duplicate data can perform filtering operations to avoid duplicate calculations. +- Using At Least One (at least once) to ensure high data processing performance and low data latency, the setting method is as follows: Instructions: -```text +```java StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment(); env.enableCheckpointing(5000); env.getCheckpointConfig().setCheckpointingMode(CheckpointingMode.AT_LEAST_ONCE); @@ -121,7 +121,7 @@ If using Maven to manage a project, simply add the following dependencies in pom The parameters for establishing a connection include URL and Properties. The URL specification format is: -`jdbc: TAOS-WS://[host_name]:[port]/[database_name]? [user={user}|&password={password}|&timezone={timezone}]` +`jdbc: TAOS-WS://[host_name]:[port]/[database_name]?[user={user}|&password={password}|&timezone={timezone}]` Parameter description: @@ -142,18 +142,17 @@ By setting the parallelism of the data source, multiple threads can read data fr The configuration parameters in Properties are as follows: -|Parameter Name | Type | Parameter Description | Remarks| -| ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| TDengineConfigParams.PROPERTYKEYUSER | string | Login TDengine username, default value 'root' || -| TDengineConfigParams.PROPERTYKEY-PASSWORD | string | User login password, default value 'taosdata' || -| TDengineConfigParams.If the downstream operator receives data of RowData type, it only needs to be set to RowData. If the user needs to customize the type, the complete class path needs to be set here| -| TDengineConfigParams.TD_STACTMODE | boolean | This parameter is used to batch push data to downstream operators. If set to True, when creating a TDengine Source object, the data type needs to be specified as SourceRecords \| The type here is the type used to receive data from downstream operators| -| TDengineConfigParams.PROPERTYKEY_CARSET | string | The character set used by the client, with the default value being the system character set. || -| TDengineConfigParams.PROPERTYKEY.MSSAGE_maIT_TIMEOUT | integer | Message timeout, in milliseconds, default value is 60000 || -| TDengineConfigParams.Whether compression is enabled during the transmission process. true: Enable, false: Not enabled. Default is false || -| TDengineConfigParams.Whether to enable automatic reconnection or not. true: Enable, false: Not enabled. Default to false|| -| TDengineConfigParams.PROPERTYKEY-RECONNECT-RETR_COUNT | integer | number of automatic reconnection retries, default value 3 | only takes effect when PROPERTYKEY-INABLE AUTO-RECONNECT is true| -| TDengineConfigParams.PROPERTYKEYDISABLE_SSL_CERTVNet | boolean | Disable SSL certificate verification. true: close, false: Not closed. The default is false|| +- TDengineConfigParams.PROPERTY_KEY_USER: Login to TDengine username, default value is 'root '. +- TDengineConfigParams.PROPERTY_KEY_PASSWORD: User login password, default value 'taosdata'. +- TDengineConfigParams.VALUE_DESERIALIZER: The downstream operator receives the result set deserialization method. If the received result set type is `RowData` of `Flink`, it only needs to be set to `RowData`. It is also possible to inherit `TDengineRecordDeserialization` and implement `convert` and `getProducedType` methods, customizing the deserialization method based on `ResultSet` of `SQL`. +- TDengineConfigParams.TD_BATCH_MODE: This parameter is used to batch push data to downstream operators. If set to True, when creating the `TDengine Source` object, it is necessary to specify the data type as a `Template` form of the `SourceRecords` type. +- TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: Message timeout time, in milliseconds, default value is 60000. +- TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION: Is compression enabled during the transmission process. true: Enable, false: Not enabled. The default is false. +- TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: Whether to enable automatic reconnection. true: Enable, false: Not enabled. The default is false. +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_INTERVAL_MS: Automatic reconnection retry interval, in milliseconds, default value 2000. It only takes effect when `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` is true. +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT: The default value for automatic reconnection retry is 3, which only takes effect when `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` is true. +- TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: Turn off SSL certificate verification. true: Enable, false: Not enabled. The default is false. + #### Split by time @@ -209,27 +208,32 @@ Example of custom data type query result:

- ResultBean is a custom inner class used to define the data type of the Source query results. -- ResultSoureDeserialization is a custom inner class that inherits Tdengine RecordDesrialization and implements convert and getProducedType methods. +- ResultSoureDeserialization is a custom inner class that inherits `TDengine` RecordDesrialization and implements convert and getProducedType methods. ### CDC Data Subscription + Flink CDC is mainly used to provide data subscription functionality, which can monitor real-time changes in TDengine database data and transmit these changes in the form of data streams to Flink for processing, while ensuring data consistency and integrity. -#### Parameter Description +Parameter Description -| Parameter Name | Type | Parameter Description | Remarks | -|-------------------------------------------|:-------:|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| TDengineCdcParams.BOOTSTRAP_SERVER | string | ip address of the server | | -| TDengineCdcParams.CONNECT-USER | string | username | | -| TDengineCdcParams.CONNECT-PASS | string | password | | -| TDengineCdcParams.POLL_INTERVAL_MS | integer | pull data interval, default 500ms | | -| TDengineConfigParams.VALUE_DESERIALIZER | string | the type of data received by the operator| If the downstream operator receives data of RowData type, it only needs to be set to RowData. If the user needs to customize the type, the complete class path needs to be set here | -| TDengineCdcParams.TMQ_STACTMODE | boolean | this parameter is used to batch push data to downstream operators. If set to True, when creating a TDengine CdcSource object, the data type needs to be specified as ConsumerRecords \ | The type here is the type used to receive data from downstream operators | -| TDengineCdcParams.GROUP ID | string | consumption group ID, shared consumption progress within the same consumption group |
* * Required field * *. Maximum length: 192< Each topic can create up to 100 consumers -| TDengineCdcParams.AUTO-OFFSET-REET | string | initial position of consumer group subscription | early: subscribe from scratch
latest: default; Subscribe only from the latest data | -| TDengineCdcParams.ENABLEAUTO_CMMIT | boolean | whether to automatically submit, true: Enable (for downstream stateless operators); false： Commit triggered by checkpoint; default false | | -| TDengineCdcParams.AUTO_CMMIT_INTERVAL_S | integer | the time interval for automatically submitting consumption records to consumption sites, in milliseconds | The default value is 5000, and this parameter takes effect when AUTO_oFFSET-REET is set to true | -| TDengineCdcParams.TMQ_SSSION_TIMEOUT_SS | integer | timeout after consumer heartbeat loss, which triggers rebalancing logic. After success, the consumer will be deleted (supported from TDengine 3.3.3.0 version) | default value is 12000, value range [60001800000] | -| TDengineCdcParams.TMQ_maX_POLL_INTERVAL_S | integer | the longest time interval for pulling data from a consumer poll. If this time is exceeded, the consumer will be considered offline and the rebalancing logic will be triggered. After success, the consumer will be deleted (supported from version 3.3.3.0) | The default value is 300000, [1000，INT32_MAX] +- TDengineCdcParams.BOOTSTRAP_SERVERS: `ip:port` of the TDengine server, if using WebSocket connection, then it is the `ip:port` where taosAdapter is located. +- TDengineCdcParams.CONNECT_USER: Login to TDengine username, default value is 'root '. +- TDengineCdcParams.CONNECT_PASS: User login password, default value 'taosdata'. +- TDengineCdcParams.POLL_INTERVAL_MS: Pull data interval, default 500ms. +- TDengineCdcParams. VALUE_DESERIALIZER: Result set deserialization method, If the received result set type is `RowData` of `Flink`, simply set it to 'RowData'. You can inherit `com.taosdata.jdbc.tmq.ReferenceDeserializer`, specify the result set bean, and implement deserialization. You can also inherit `com.taosdata.jdbc.tmq.Deserializer` and customize the deserialization method based on the SQL resultSet. +- TDengineCdcParams.TMQ_BATCH_MODE: This parameter is used to batch push data to downstream operators. If set to True, when creating the `TDengineCdcSource` object, it is necessary to specify the data type as a template form of the `ConsumerRecords` type. +- TDengineCdcParams.GROUP_ID: Consumer group ID, the same consumer group shares consumption progress。Maximum length: 192. +- TDengineCdcParams.AUTO_OFFSET_RESET: Initial position of the consumer group subscription （ `earliest` subscribe from the beginning, `latest` subscribe from the latest data, default `latest`）。 +- TDengineCdcParams.ENABLE_AUTO_COMMIT: Whether to enable automatic consumption point submission，true: automatic submission；false：submit based on the `checkpoint` time, default to false. +> **Note**：The automatic submission mode of the reader automatically submits data after obtaining it, regardless of whether the downstream operator has processed the data correctly. There is a risk of data loss, and it is mainly used for efficient stateless operator scenarios or scenarios with low data consistency requirements. + +- TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS: Time interval for automatically submitting consumption records, in milliseconds, default 5000. This parameter takes effect when `ENABLE_AUTO_COMMIT` is set to true. +- TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION: Is compression enabled during the transmission process. true: Enable, false: Not enabled. The default is false. +- TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: Whether to enable automatic reconnection. true: Enable, false: Not enabled. The default is false. +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_INTERVAL_MS: Automatic reconnection retry interval, in milliseconds, default value 2000. It only takes effect when `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` is true. +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT: The default value for automatic reconnection retry is 3, which only takes effect when `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` is true. +- TDengineCdcParams.TMQ_SESSION_TIMEOUT_MS: Timeout after consumer heartbeat is lost, after which rebalance logic is triggered, and upon success, that consumer will be removed (supported from version 3.3.3.0)，Default is 12000, range [6000, 1800000]. +- TDengineCdcParams.TMQ_MAX_POLL_INTERVAL_MS: The longest time interval for consumer poll data fetching, exceeding this time will be considered as the consumer being offline, triggering rebalance logic, and upon success, that consumer will be removed (supported from version 3.3.3.0) Default is 300000, range [1000, INT32_MAX]. #### Use CDC connector @@ -267,24 +271,23 @@ Example of custom data type query result: The core function of Sink is to efficiently and accurately write Flink processed data from different data sources or operators into TDengine. In this process, the efficient write mechanism possessed by TDengine played a crucial role, effectively ensuring the fast and stable storage of data. -#### Sink Properties +Sink Properties -| Parameter Name | Type | Parameter Description | Remarks| -|---------------------------------------------------------|:----------------------------------------------------------------------------------------------------:|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| TDengineConfigParams.PROPERTYKEYUSER | string | Login TDengine username, default value 'root' || -| TDengineConfigParams.PROPERTYKEY-PASSWORD | string | User login password, default value 'taosdata' || -| TDengineConfigParams.PROPERTYKEYDBNAME | string | Database name written || -| TDengineConfigParams.TD_SUPERTABLeNAME | string | Name of the super table to be written | If the data received by the super table must have a tbname field, determine which sub table to write to| -| TDengineConfigParams.TD_TABLeNAME | string | The name of the table to be written, this parameter only needs to be set together with TD_SUPERTABLeNAME | Used to determine which sub table or regular table to write to| -| TDengineConfigParams.TD_STACTISZE | integer | Set batch size | Write when the batch quantity is reached, or a checkpoint time will also trigger writing to the database| -| TDengineConfigParams.VALUE_DESERIALIZER | string | If the downstream operator receives data of RowData type, it only needs to be set to RowData. If the user needs to customize the type, the complete class path needs to be set here | -| TDengineConfigParams.TD_STACTMODE | boolean | This parameter is used to set the reception of batch data | If set to True:< The source is TDengine Source, using SourceRecords \to create TDengine Sink object
The source is TDengine CDC, using ConsumerRecords \to create TDengine Sink object | The type here is the type that receives data| -| TDengineConfigParams.TD_SOURCETYPE | string | If the data is from a source, such as source or cdc | TDengine source is set to "tdengine_stource", TDengine cdc is set to "tdengine_cdc"| -| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | Message timeout, in milliseconds, default value is 60000 || -| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | Whether compression is enabled during the transmission process. true: Enable, false: Not enabled. Default is false | | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT | integer| to enable automatic reconnection or not. true: Enable, false: Not enabled. Default to false | | -| TDengineConfigParams.PROPERTYKEY_RECONNECT_RETR_COUNT | integer | number of automatic reconnection retries, default value 3 | only takes effect when PROPERTYKEY-INABLE AUTO-RECONNECT is true| -| TDengineConfigParams.PROPERTYKEYDISABLE_SSL_CERTVNet | boolean | Disable SSL certificate verification. true: close, false: Not closed. The default is false || +- TDengineConfigParams.PROPERTY_KEY_USER: Login to TDengine username, default value is 'root '. +- TDengineConfigParams.PROPERTY_KEY_PASSWORD: User login password, default value 'taosdata'. +- TDengineConfigParams.PROPERTY_KEY_DBNAME: The database name. +- TDengineConfigParams.TD_SUPERTABLE_NAME:The name of the super table. The received data must have a tbname field to determine which sub table to write to. +- TDengineConfigParams.TD_TABLE_NAME: The table name of a sub table or a normal table. This parameter only needs to be set together with `TD_SUPERTABLE_NAME`. +- TDengineConfigParams.VALUE_DESERIALIZER: The deserialization method for receiving result sets. If the type of the received result set is RowData of Flink, it only needs to be set to RowData. It is also possible to inherit 'TDengine SinkRecordSequencer' and implement the 'serialize' method, customizing the deserialization method based on the received data type. +- TDengineConfigParams.TD_BATCH_SIZE: Set the batch size for writing to the `TDengine` database once | Writing will be triggered when the number of batches is reached, or when a checkpoint is set. +- TDengineConfigParams.TD_BATCH_MODE: When set to True for receiving batch data, if the data source is `TDengine Source` , use the `SourceRecords Template` type to create a `TDengineSink` object; If the source is `TDengine CDC`, use the `ConsumerRecords Template` to create a `TDengineSink` object. +- TDengineConfigParams.TD_SOURCE_TYPE: Set the data source. When the data source is `TDengine Source`, it is set to 'tdengine_stource', and when the source is `TDengine CDC`, it is set to 'tdengine_cdc'. When the configuration of `TD_BATCH_MODE` is set to True, it takes effect. +- TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: Message timeout time, in milliseconds, default value is 60000. +- TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION: Is compression enabled during the transmission process. true: Enable, false: Not enabled. The default is false. +- TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: Whether to enable automatic reconnection. true: Enable, false: Not enabled. The default is false. +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_INTERVAL_MS: Automatic reconnection retry interval, in milliseconds, default value 2000. It only takes effect when `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` is true. +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT: The default value for automatic reconnection retry is 3, which only takes effect when `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` is true. +- TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: Turn off SSL certificate verification. true: Enable, false: Not enabled. The default is false. Usage example: @@ -316,17 +319,17 @@ Extract data from multiple different data source databases (such as TDengine, My Parameter configuration instructions: -| Parameter Name | Type | Parameter Description | Remarks| -|-----------------------| :-----: | ------------ | ------ | -| connector | string | connector identifier, set `tdengine-connector`|| -| td.jdbc.url | string | url of the connection || -| td.jdbc.mode | strng | connector type: `source`, `sink`| | -| table.name | string | original or target table name || -| scan.query | string | SQL statement to retrieve data|| -| sink.db.name | string | target database name|| -| sink.superstable.name | string | write the name of the superstable|| -| sink.batch.size | integer| batch size written|| -| sink.table.name | string | name of the regular table or sub table written|| +| Parameter Name | Type | Parameter Description | +|-----------------------| :-----: | ------------ | +| connector | string | connector identifier, set `tdengine-connector`| +| td.jdbc.url | string | url of the connection | +| td.jdbc.mode | strng | connector type: `source`, `sink`| +| table.name | string | original or target table name | +| scan.query | string | SQL statement to retrieve data| +| sink.db.name | string | target database name| +| sink.supertable.name | string | name of the supertable| +| sink.batch.size | integer| batch size written| +| sink.table.name | string | the table name of a sub table or a normal table | Usage example: @@ -351,13 +354,13 @@ Parameter configuration instructions: | bootstrap. servers| string | server address | | topic | string | subscribe to topic | | td.jdbc.mode | strng | connector type: `cdc`, `sink` | -| group.id | string | Consumption group ID, sharing consumption progress within the same consumption group | -| auto.offset.reset | string | initial position for consumer group subscription.
earliest: subscribe from scratch
latest: default; Subscribe only from the latest data| -| poll.interval_mas | integer | Pull data interval, default 500ms | -| sink.db.name | string | Target database name | -| sink.superstable.name | string | Write the name of the superstable | +| group.id | string | consumption group ID, sharing consumption progress within the same consumption group | +| auto.offset.reset | string | initial position for consumer group subscription.
`earliest`: subscribe from the beginning
`latest` subscribe from the latest data
default `latest`| +| poll.interval_mas | integer | pull data interval, default 500ms | +| sink.db.name | string | target database name | +| sink.supertable.name | string | name of the supertable | | sink.batch.size | integer | batch size written | -| sink.table.name | string | Name of the regular table or sub table written | +| sink.table.name | string | the table name of a sub table or a normal table | Usage example: diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md index af5434e0b3..a8e9ec6edf 100644 --- a/docs/zh/10-third-party/01-collection/12-flink.md +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -122,6 +122,7 @@ URL 规范格式为： `jdbc:TAOS-WS://[host_name]:[port]/[database_name]?[user={user}|&password={password}|&timezone={timezone}]` 参数说明： + - user：登录 TDengine 用户名，默认值 'root'。 - password：用户登录密码，默认值 'taosdata'。 - database_name: 数据库名称。 @@ -135,22 +136,18 @@ URL 规范格式为： Source 拉取 TDengine 数据库中的数据，并将获取到的数据转换为 Flink 内部可处理的格式和类型，并以并行的方式进行读取和分发，为后续的数据处理提供高效的输入。 通过设置数据源的并行度，实现多个线程并行地从数据源中读取数据，提高数据读取的效率和吞吐量，充分利用集群资源进行大规模数据处理能力。 -#### Source Properties - Properties 中配置参数如下： -| 参数名称 | 类型 | 参数说明 | 备注 | -| ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'。| | -| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'。| | -| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径。| -| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineSource 对象时需要指定数据类型为 SourceRecords\<类型\> 。 | 此处的类型为用下游算子接收数据的类型。| -| TDengineConfigParams.PROPERTY_KEY_CHARSET | string | 客户端使用的字符集，默认值为系统字符集。| | -| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000。| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false。|| -| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。| -| TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| +- TDengineConfigParams.PROPERTY_KEY_USER：登录 TDengine 用户名，默认值 'root'。 +- TDengineConfigParams.PROPERTY_KEY_PASSWORD：用户登录密码，默认值 'taosdata'。 +- TDengineConfigParams.VALUE_DESERIALIZER：下游算子接收结果集反序列化方法, 如果接收结果集类型是 `Flink` 的 `RowData`，仅需要设置为 `RowData`即可。也可继承 `TDengineRecordDeserialization` 并实现 `convert` 和 `getProducedType` 方法，根据 `SQL` 的 `ResultSet` 自定义反序列化方式。 +- TDengineConfigParams.TD_BATCH_MODE：此参数用于批量将数据推送给下游算子，如果设置为 True，创建 `TDengineSource` 对象时需要指定数据类型为 `SourceRecords` 类型的泛型形式。 +- TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms， 默认值为 60000。 +- TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION: 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。 +- TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: 是否启用自动重连。true: 启用，false: 不启用。默认为 false。 +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_INTERVAL_MS: 自动重连重试间隔，单位毫秒，默认值 2000。仅在 `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` 为 true 时生效。 +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT: 自动重连重试次数，默认值 3，仅在 `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` 为 true 时生效。 +- TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: 关闭 SSL 证书验证 。true: 启用，false: 不启用。默认为 false。 #### 按时间分片 @@ -206,27 +203,32 @@ Properties 中配置参数如下：
- ResultBean 自定义的一个内部类，用于定义 Source 查询结果的数据类型。 -- ResultSoureDeserialization 是自定义的一个内部类，通过继承 TdengineRecordDeserialization 并实现 convert 和 getProducedType 方法。 +- ResultSoureDeserialization 是自定义的一个内部类，通过继承 `TDengineRecordDeserialization` 并实现 `convert` 和 `getProducedType` 方法。 ### CDC 数据订阅 -Flink CDC 主要用于提供数据订阅功能，能实时监控 TDengine 数据库的数据变化，并将这些变更以数据流形式传输到 Flink 中进行处理，同时确保数据的一致性和完整性。 +Flink CDC 主要用于提供数据订阅功能，能实时监控 `TDengine` 数据库的数据变化，并将这些变更以数据流形式传输到 `Flink` 中进行处理，同时确保数据的一致性和完整性。 -#### 参数说明 -| 参数名称 | 类型 | 参数说明 | 备注 | -| ----------------------- | :-----: | ------------------------- | -------------------------- | -| TDengineCdcParams.BOOTSTRAP_SERVERS| string | 服务端的 IP 地址。 | | -| TDengineCdcParams.CONNECT_USER| string | 用户名。 | | -| TDengineCdcParams.CONNECT_PASS| string | 密码。 | | -| TDengineCdcParams.POLL_INTERVAL_MS|int| 拉取数据间隔, 默认 500ms。| | -| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型。 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径。| -| TDengineCdcParams.TMQ_BATCH_MODE| boolean | 此参数用于批量将数据推送给下游算子，如果设置为 True，创建 TDengineCdcSource 对象时需要指定数据类型为 ConsumerRecords\<类型\>。| 此处的类型为用下游算子接收数据的类型。| -| TDengineCdcParams.GROUP_ID| string | 消费组 ID，同一消费组共享消费进度。 |
**必填项**。最大长度：192。
每个 topic 最多可建立 100 个 consumer 。| -| TDengineCdcParams.AUTO_OFFSET_RESET| string | 消费组订阅的初始位置。 | earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅。| -| TDengineCdcParams.ENABLE_AUTO_COMMIT| boolean | 是否自动提交，true: 启用（用于下游均为无状态算子） ；false：由 checkpoint 触发 commit 。| 默认 false。| -| TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS| integer|消费记录自动提交消费位点时间间隔，单位为毫秒。| 默认值为 5000, 此参数在 AUTO_OFFSET_RESET 为 true 生效。| -| TDengineCdcParams.TMQ_SESSION_TIMEOUT_MS| integer | consumer 心跳丢失后超时时间，超时后会触发 rebalance 逻辑，成功后该 consumer 会被删除（从 TDengine 3.3.3.0 版本开始支持）。| 默认值为 12000，取值范围 [6000， 1800000]。 | -| TDengineCdcParams.TMQ_MAX_POLL_INTERVAL_MS| integer | consumer poll 拉取数据间隔的最长时间，超过该时间，会认为该 consumer 离线，触发 rebalance 逻辑，成功后该 consumer 会被删除（从 3.3.3.0 版本开始支持）。 | 默认值为 300000，[1000，INT32_MAX]。| +Properties 中配置参数如下： + +- TDengineCdcParams.BOOTSTRAP_SERVERS：TDengine 服务端所在的`ip:port`，如果使用 `WebSocket` 连接，则为 taosAdapter 所在的`ip:port`。 +- TDengineCdcParams.CONNECT_USER：登录 TDengine 用户名，默认值 'root'。 +- TDengineCdcParams.CONNECT_PASS：用户登录密码，默认值 'taosdata'。 +- TDengineCdcParams.POLL_INTERVAL_MS：拉取数据间隔, 默认 500ms。 +- TDengineCdcParams.VALUE_DESERIALIZER：结果集反序列化方法，如果接收结果集类型是 `Flink` 的 `RowData`，仅需要设置为 `RowData`即可。可以继承 `com.taosdata.jdbc.tmq.ReferenceDeserializer`，并指定结果集 bean，实现反序列化。 +- TDengineCdcParams.TMQ_BATCH_MODE：此参数用于批量将数据推送给下游算子，如果设置为 True，创建 `TDengineCdcSource` 对象时需要指定数据类型为 `ConsumerRecords` 类型的泛型形式。 +- TDengineCdcParams.GROUP_ID：消费组 ID，同一消费组共享消费进度。最大长度：192。 +- TDengineCdcParams.AUTO_OFFSET_RESET： 消费组订阅的初始位置 （ `earliest` 从头开始订阅, `latest` 仅从最新数据开始订阅, 默认 `latest`）。 +- TDengineCdcParams.ENABLE_AUTO_COMMIT：是否启用消费位点自动提交，true: 自动提交；false：依赖 `checkpoint` 时间来提交， 默认 false。 +> **注意**：自动提交模式reader获取完成数据后自动提交，不管下游算子是否正确的处理了数据，存在数据丢失的风险，主要用于为了追求高效的无状态算子场景或是数据一致性要求不高的场景。 + +- TDengineCdcParams.AUTO_COMMIT_INTERVAL_MS：消费记录自动提交消费位点时间间隔，单位为毫秒。默认值为 5000, 此参数在 `ENABLE_AUTO_COMMIT` 为 true 生效。 +- TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION：传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。 +- TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT：是否启用自动重连。true: 启用，false: 不启用。默认为 false。 +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_INTERVAL_MS：自动重连重试间隔，单位毫秒，默认值 2000。仅在 `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` 为 true 时生效。 +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT：自动重连重试次数，默认值 3，仅在 `PROPERTY_KEY_ENABLE_AUTO_RECONNECT` 为 true 时生效。 +- TDengineCdcParams.TMQ_SESSION_TIMEOUT_MS：`consumer` 心跳丢失后超时时间，超时后会触发 `rebalance` 逻辑，成功后该 `consumer` 会被删除（从3.3.3.0版本开始支持）， 默认值为 12000，取值范围 [6000， 1800000]。 +- TDengineCdcParams.TMQ_MAX_POLL_INTERVAL_MS：`consumer poll` 拉取数据间隔的最长时间，超过该时间，会认为该 `consumer` 离线，触发 `rebalance` 逻辑，成功后该 `consumer` 会被删除。 默认值为 300000，[1000，INT32_MAX]。 #### 使用 CDC 连接器 @@ -259,30 +261,29 @@ CDC 连接器会根据用户设置的并行度进行创建 consumer，因此用 ``` -- ResultBean 是自定义的一个内部类，其字段名和数据类型与列的名称和数据类型一一对应，这样根据 value.deserializer 属性对应的反序列化类可以反序列化出 ResultBean 类型的对象。 +- ResultBean 是自定义的一个内部类，其字段名和数据类型与列的名称和数据类型一一对应，这样根据 `TDengineCdcParams.VALUE_DESERIALIZER` 属性对应的反序列化类可以反序列化出 ResultBean 类型的对象。 ### Sink -Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自不同数据源或算子的数据写入 TDengine。在这一过程中，TDengine 所具备的高效写入机制发挥了至关重要的作用，有力保障了数据的快速和稳定存储。 +Sink 的核心功能在于高效且精准地将经过 `Flink` 处理的、源自不同数据源或算子的数据写入 `TDengine`。在这一过程中，`TDengine` 所具备的高效写入机制发挥了至关重要的作用，有力保障了数据的快速和稳定存储。 -#### Sink Properties +Properties 中配置参数如下： -| 参数名称 | 类型 | 参数说明 | 备注 | -| ----------------------- | :-----: | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| TDengineConfigParams.PROPERTY_KEY_USER | string | 登录 TDengine 用户名，默认值 'root'。| | -| TDengineConfigParams.PROPERTY_KEY_PASSWORD| string | 用户登录密码，默认值 'taosdata'。| | -| TDengineConfigParams.PROPERTY_KEY_DBNAME| string | 写入的数据库名称。|| -| TDengineConfigParams.TD_SUPERTABLE_NAME| string | 写入的超级表名称。| 如果是超级表接收的数据必须有 tbname 字段，确定写入那张子表。| -| TDengineConfigParams.TD_TABLE_NAME| string | 写入的表名，此参数和TD_SUPERTABLE_NAME 仅需要设置一个即可。| 用于确定写入那张子表或普通表。| -| TDengineConfigParams.TD_BATCH_SIZE| integer | 设置批大小 | 当到达批的数量后进行写入，或是一个checkpoint的时间也会触发写入数据库。| -| TDengineConfigParams.VALUE_DESERIALIZER| string | 游算子接收数据的类型。 | 如果下游算子接收数据的类型是 RowData 仅需要设置为 RowData, 如果用户需要自定义类型这里需要设置完整的类路径。| -| TDengineConfigParams.TD_BATCH_MODE | boolean | 此参数用于设置接收批量数据。 | 如果设置为 True:
来源是 TDengine Source， 使用SourceRecords\<类型\> 创建 TDengineSink 对象
来源是 TDengine CDC 使用 ConsumerRecords\<类型\> 创建 TDengineSink 对象。| 此处的类型为接收数据的类型。| -| TDengineConfigParams.TD_SOURCE_TYPE | string | 如果数据是表示数据来源是，source 或者 cdc 等。 | TDengine source 设置为 "tdengine_source", TDengine cdc 设置为 "tdengine_cdc"。| -| TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT | integer | 消息超时时间，单位 ms， 默认值为 60000。| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION | boolean | 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。| | -| TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT| boolean| 是否启用自动重连。true: 启用，false: 不启用。默认为 false。|| -| TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT| integer| 自动重连重试次数，默认值 3。 | 仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。| -| TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION| boolean| 关闭 SSL 证书验证 。true: 关闭，false: 不关闭。默认为 false。|| +- TDengineConfigParams.PROPERTY_KEY_USER：登录 `TDengine` 用户名，默认值 'root'。 +- TDengineConfigParams.PROPERTY_KEY_PASSWORD：用户登录密码，默认值 'taosdata'。 +- TDengineConfigParams.PROPERTY_KEY_DBNAME：写入的数据库名称。 +- TDengineConfigParams.TD_SUPERTABLE_NAME：写入的超级表名称。接收的数据必须有 tbname 字段，确定写入那张子表。 +- TDengineConfigParams.TD_TABLE_NAME：写入子表或普通表的表名，此参数和TD_SUPERTABLE_NAME 仅需要设置一个即可。 +- TDengineConfigParams.VALUE_DESERIALIZER：接收结果集反序列化方法, 如果接收结果集类型是 `Flink` 的 `RowData`，仅需要设置为 `RowData`即可。也可继承 `TDengineSinkRecordSerializer` 并实现 `serialize` 方法，根据 接收的数据类型自定义反序列化方式。 +- TDengineConfigParams.TD_BATCH_SIZE：设置一次写入 `TDengine` 数据库的批大小 | 当到达批的数量后进行写入，或是一个checkpoint的时间也会触发写入数据库。 +- TDengineConfigParams.TD_BATCH_MODE：接收批量数据当设置为 True 时，如果数据来源是 `TDengine Source`，则使用 `SourceRecords` 泛型类型来创建 `TDengineSink` 对象；若来源是 `TDengine CDC`，则使用 `ConsumerRecords` 泛型来创建 `TDengineSink` 对象。 +- TDengineConfigParams.TD_SOURCE_TYPE：设置数据来源。 当数据来源是 `TDengine Source` 是设置为 'tdengine_source', 当来源是 `TDengine CDC` 设置为 'tdengine_cdc'。当配置 `TD_BATCH_MODE` 为 True 生效。 +- TDengineConfigParams.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms， 默认值为 60000。 +- TDengineConfigParams.PROPERTY_KEY_ENABLE_COMPRESSION: 传输过程是否启用压缩。true: 启用，false: 不启用。默认为 false。 +- TDengineConfigParams.PROPERTY_KEY_ENABLE_AUTO_RECONNECT: 是否启用自动重连。true: 启用，false: 不启用。默认为 false。 +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_INTERVAL_MS: 自动重连重试间隔，单位毫秒，默认值 2000。仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。 +- TDengineConfigParams.PROPERTY_KEY_RECONNECT_RETRY_COUNT: 自动重连重试次数，默认值 3，仅在 PROPERTY_KEY_ENABLE_AUTO_RECONNECT 为 true 时生效。 +- TDengineConfigParams.PROPERTY_KEY_DISABLE_SSL_CERT_VALIDATION: 关闭 SSL 证书验证 。true: 启用，false: 不启用。默认为 false。 使用示例： @@ -314,8 +315,8 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 参数配置说明： -| 参数名称 | 类型 | 参数说明 | 备注 | -| ----------------------- | :-----: | ------------ | ------ | +| 参数名称 | 类型 | 参数说明 | +| ----------------------- | :-----: | ------------ | | connector | string | 连接器标识，设置 `tdengine-connector` 。| | td.jdbc.url| string | 连接的 url 。| | td.jdbc.mode | strng | 连接器类型, 设置 `source`, `sink`。| @@ -351,7 +352,7 @@ Sink 的核心功能在于高效且精准地将经过 Flink 处理的、源自 | topic | string | 订阅主题。|| | td.jdbc.mode | strng | 连接器类型, cdc, sink。| | group.id| string| 消费组 ID，同一消费组共享消费进度。 | -| auto.offset.reset| string| 消费组订阅的初始位置。
earliest: 从头开始订阅
latest: default; 仅从最新数据开始订阅。| +| auto.offset.reset| string| 消费组订阅的初始位置。
`earliest`: 从头开始订阅
`latest`: 仅从最新数据开始订阅。
默认 `latest`。| | poll.interval_ms| integer| 拉取数据间隔, 默认 500ms。| | sink.db.name|string| 目标数据库名称。| | sink.supertable.name|string |写入的超级表名称。| From 3ef9c225ca158a2869331fb79487f68549b5e825 Mon Sep 17 00:00:00 2001 From: menshibin Date: Wed, 1 Jan 2025 16:32:56 +0800 Subject: [PATCH 62/73] modify Number error --- docs/en/10-third-party/01-collection/flink.md | 2 +- docs/zh/10-third-party/01-collection/12-flink.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en/10-third-party/01-collection/flink.md b/docs/en/10-third-party/01-collection/flink.md index f586b36d0e..dea8fedc05 100644 --- a/docs/en/10-third-party/01-collection/flink.md +++ b/docs/en/10-third-party/01-collection/flink.md @@ -26,7 +26,7 @@ Flink Connector supports all platforms that can run Flink 1.19 and above version | Flink Connector Version | Major Changes | TDengine Version| |-------------------------| ------------------------------------ | ---------------- | -| 2.0.0 | 1.Support SQL queries on data in TDengine database
2 Support CDC subscription to data in TDengine database
3 Supports reading and writing to TDengine database using Table SQL | 3.3.5.0 and above versions| +| 2.0.0 | 1.Support SQL queries on data in TDengine database
2. Support CDC subscription to data in TDengine database
3. Supports reading and writing to TDengine database using Table SQL | 3.3.5.0 and above versions| | 1.0.0 | Support Sink function to write data from other sources to TDengine in the future| 3.3.2.0 and above versions| ## Exception and error codes diff --git a/docs/zh/10-third-party/01-collection/12-flink.md b/docs/zh/10-third-party/01-collection/12-flink.md index a8e9ec6edf..e589e36c9a 100644 --- a/docs/zh/10-third-party/01-collection/12-flink.md +++ b/docs/zh/10-third-party/01-collection/12-flink.md @@ -352,7 +352,7 @@ Properties 中配置参数如下： | topic | string | 订阅主题。|| | td.jdbc.mode | strng | 连接器类型, cdc, sink。| | group.id| string| 消费组 ID，同一消费组共享消费进度。 | -| auto.offset.reset| string| 消费组订阅的初始位置。
`earliest`: 从头开始订阅
`latest`: 仅从最新数据开始订阅。
默认 `latest`。| +| auto.offset.reset| string| 消费组订阅的初始位置。
`earliest`: 从头开始订阅
`latest`: 仅从最新数据开始订阅。
默认 `latest`。| | poll.interval_ms| integer| 拉取数据间隔, 默认 500ms。| | sink.db.name|string| 目标数据库名称。| | sink.supertable.name|string |写入的超级表名称。| From da3c5a8e0c43a4d95f2b8733241dd34c9add8fec Mon Sep 17 00:00:00 2001 From: Alex Duan <417921451@qq.com> Date: Wed, 1 Jan 2025 17:50:52 +0800 Subject: [PATCH 63/73] docs: add blank before link --- docs/zh/27-train-faq/01-faq.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/27-train-faq/01-faq.md b/docs/zh/27-train-faq/01-faq.md index 1b2f2c9fcc..24254c8807 100644 --- a/docs/zh/27-train-faq/01-faq.md +++ b/docs/zh/27-train-faq/01-faq.md @@ -311,4 +311,4 @@ TDinsight插件中展示的数据是通过taosKeeper和taosAdapter服务收集 直接查子表更快。超级表带 TAG 过滤查询子查数据是为满足查询方便性，同时可对多个子表中数据进行过滤，如果目的是追求性能并已明确查询子表，直接从子表查性能更高 ### 35 如何查看数据压缩率指标？ -TDengine 目前只提供以表为统计单位的压缩率，数据库及整体还未提供，查看命令是在客户端 taos-CLI 中执行 `SHOW TABLE DISTRIBUTED table_name;` 命令，table_name 为要查看压缩率的表，可以为超级表、普通表及子表，详细可[查看此处](https://docs.taosdata.com/reference/taos-sql/show/#show-table-distributed) \ No newline at end of file +TDengine 目前只提供以表为统计单位的压缩率，数据库及整体还未提供，查看命令是在客户端 taos-CLI 中执行 `SHOW TABLE DISTRIBUTED table_name;` 命令，table_name 为要查看压缩率的表，可以为超级表、普通表及子表，详细可 [查看此处](https://docs.taosdata.com/reference/taos-sql/show/#show-table-distributed) \ No newline at end of file From 055441d8fbbb498dc95422cc73d2fc2dbe2abcd3 Mon Sep 17 00:00:00 2001 From: Haojun Liao Date: Thu, 2 Jan 2025 09:55:05 +0800 Subject: [PATCH 64/73] fix(stream): fix dead-lock caused by invalid repeatly unlock. --- source/dnode/vnode/src/tq/tqSink.c | 2 -- 1 file changed, 2 deletions(-) diff --git a/source/dnode/vnode/src/tq/tqSink.c b/source/dnode/vnode/src/tq/tqSink.c index 5405ace89b..7ba77cf813 100644 --- a/source/dnode/vnode/src/tq/tqSink.c +++ b/source/dnode/vnode/src/tq/tqSink.c @@ -1047,8 +1047,6 @@ int32_t setDstTableDataUid(SVnode* pVnode, SStreamTask* pTask, SSDataBlock* pDat pTableSinkInfo->uid = 0; code = doPutSinkTableInfoIntoCache(pTask->outputInfo.tbSink.pTbInfo, pTableSinkInfo, groupId, id); } else { - metaReaderClear(&mr); - tqError("s-task:%s vgId:%d dst-table:%s not auto-created, and not create in tsdb, discard data", id, vgId, dstTableName); return TSDB_CODE_TDB_TABLE_NOT_EXIST; From b87ec025804fc7a1a19ca55c1d66a8460cc65b9b Mon Sep 17 00:00:00 2001 From: wangjiaming0909 Date: Thu, 2 Jan 2025 13:49:09 +0800 Subject: [PATCH 65/73] add test for drop view crash --- .../0-others/view/non_marterial_view/test_view.py | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tests/system-test/0-others/view/non_marterial_view/test_view.py b/tests/system-test/0-others/view/non_marterial_view/test_view.py index 7e062143d2..059b016c3d 100644 --- a/tests/system-test/0-others/view/non_marterial_view/test_view.py +++ b/tests/system-test/0-others/view/non_marterial_view/test_view.py @@ -566,8 +566,18 @@ class TDTestCase: tdSql.execute(f"drop database {paraDict['dbName']}") tdSql.execute("drop database cdb;") tdLog.debug("Finish test case 'test_tmq_from_view'") + def test_TD_33390(self): + tdSql.execute('create database test') + tdSql.execute('create table test.nt(ts timestamp, c1 int)') + for i in range(0, 200): + tdSql.execute(f'create view test.view{i} as select * from test.nt') + tdSql.query("show test.views") + + for i in range(0, 200): + tdSql.execute(f'drop view test.view{i}') def run(self): + self.test_TD_33390() self.test_create_view_from_one_database() self.test_create_view_from_multi_database() self.test_create_view_name_params() From e066c1ed2909b4b9494bd1e2bedfd6c85e0b7be8 Mon Sep 17 00:00:00 2001 From: wangjiaming0909 Date: Thu, 2 Jan 2025 13:49:28 +0800 Subject: [PATCH 66/73] modify Jenkinsfile2 --- Jenkinsfile2 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Jenkinsfile2 b/Jenkinsfile2 index 1b2f28908c..52658434a4 100644 --- a/Jenkinsfile2 +++ b/Jenkinsfile2 @@ -165,7 +165,7 @@ def pre_test(){ script { sh ''' cd ${WK} - git checkout ''' + env.CHANGE_TARGET + ''' + git checkout fix/main/TD-33390 cd ${WKC} git checkout ''' + env.CHANGE_TARGET + ''' ''' @@ -274,7 +274,7 @@ def pre_test_win(){ script { bat ''' cd %WIN_INTERNAL_ROOT% - git checkout ''' + env.CHANGE_TARGET + ''' + git checkout fix/main/TD-33390 ''' bat ''' cd %WIN_COMMUNITY_ROOT% From ae034718c55fb9986b0b9982d338ab2adaea5792 Mon Sep 17 00:00:00 2001 From: Ping Xiao Date: Thu, 2 Jan 2025 14:46:45 +0800 Subject: [PATCH 67/73] test:update 3.3.5.0 release notes in office web --- docs/en/28-releases/01-tdengine.md | 4 ++ docs/en/28-releases/03-notes/3.3.5.0.md | 85 +++++++++++++++++++++++++ docs/en/28-releases/03-notes/index.md | 1 + docs/zh/28-releases/01-tdengine.md | 4 ++ docs/zh/28-releases/03-notes/3.3.5.0.md | 84 ++++++++++++++++++++++++ docs/zh/28-releases/03-notes/index.md | 1 + 6 files changed, 179 insertions(+) create mode 100755 docs/en/28-releases/03-notes/3.3.5.0.md create mode 100755 docs/zh/28-releases/03-notes/3.3.5.0.md diff --git a/docs/en/28-releases/01-tdengine.md b/docs/en/28-releases/01-tdengine.md index 3c57cea0b1..12cf5484d4 100644 --- a/docs/en/28-releases/01-tdengine.md +++ b/docs/en/28-releases/01-tdengine.md @@ -25,6 +25,10 @@ Download links for TDengine 3.x version installation packages are as follows: import Release from "/components/ReleaseV3"; +## 3.3.5.0 + + + ## 3.3.4.8 diff --git a/docs/en/28-releases/03-notes/3.3.5.0.md b/docs/en/28-releases/03-notes/3.3.5.0.md new file mode 100755 index 0000000000..3326b3f0f0 --- /dev/null +++ b/docs/en/28-releases/03-notes/3.3.5.0.md @@ -0,0 +1,85 @@ +--- +title: TDengine 3.3.5.0 Release Notes +sidebar_label: 3.3.5.0 +description: Version 3.3.5.0 Notes +slug: /release-history/release-notes/3.3.5.0 +--- + +## Features + 1. feat: refactor MQTT to improve stability and performance + 2. feat: refactor taosX incremental backup-restore + 3. feat: add stmt2 apis in JDBC via websocket connection + 4. feat: add stmt2 api in Rust connector + 5. feat: Adding error codes in error prompts in taos-CLI + 6. feat: SuperSet can connect TDengine with python connector + 7. feat: configurable grafana dashboards in explorer management + 8. feat: add taosX-agent in-memory cache queu capacity option + +## Enhancements + 1. enh: adjust the reporting mechanism of telemetry. + 2. enh: support for SQL-based statistics of disk space for a specified DB. + 3. enh: add memory management for SQL queries on the server side + 4. enh: INTERVAL clause allows the use of the AUTO keyword to specify the window offset. + 5. enh: reduce the impact on data write performance during data migration across multi-level storage + 6. enh: migrate from angular to react for grafana 11.3+ + 7. enh: refactor taosAdapter websocket api for a slightly better perf + 8. enh: add health state in taosX task status + 9. enh: taosX add configurations to handle exceptions + 10. enh: support setting options for client connections, including time zone, character set, user IP, and user name. + 11. enh: taosdump support retry after connection timeout or broken + 12. enh: allow creating index for tags that already subscribed + 13. enh: taosX now support literal special chars in password + 14. enh: improve data write performance when Last Cache is activated. + 15. enh: COMPACT command supports automatic execution, concurrency setting, and progress observation. + 16. enh: support update global configuration parameters through SQL statements and persisting them. + 17. enh: update the default compression method for all data types to improve the compression ratio in most scenarios. + 18. enh: taosBenchmark --nodrop fix for mac/window + 19. enh: prohibit the simultaneous execution of DB compaction and replica change operations (Enterpris). + 20. enh: taosdump support primary key tables + 21. enh: display user IP and name in the results of the SHOW QUERIES and SHOW CONNECTIONS statements. + 22. enh: (JDBC)support batch insertion into multiple tables + 23. enh: support for dynamically modifying the dataDir parameter for multi-level storage. + 24. enh: prefer db file under data_dir + 25. enh: enforce users to set strong passwords, which must be 8 to 16 characters in length and include at least three types of characters from the following: uppercase letters, lowercase letters, numbers, and special characters. + 26. enh: improve the speed at which clients acquire the new Leader. + 27. enh: support negative regex pattern in opc point selector + +## Fixes + 1. fix: the potential for deadlocks when updating checkpoints in stream computing under high-load scenarios. + 2. fix: write tmq data into target error when terrno already set + 3. fix: taosd cannot start when there is data corruption in a block within the WAL + 4. fix: taosBenchmark fails when taosd disconnected in replica 2/3 + 5. fix: log files being lost when they are switched frequently. + 6. fix: he stream computing stops due to the data update within the window. + 7. fix: libtaosws.so sets an incorrect error code when the connection is terminated while fetch data. + 8. fix: taosX opc error in case of @-prefixed name + 9. fix: fix permission denied with show vgroups sql in cloud + 10. fix: fix sql syntax error when migrating from large stables with compress options + 11. fix: incorrect memory estimation for vnode usage + 12. fix: failed to perform UNION ALL query on constant strings of the varchar type. + 13. fix: leader transfer during the execution of transaction may cause deadlock. + 14. fix: rust connector invliad pointer addr in ws_stmt_get_tag_fields + 15. fix: UNION statement fails when executing with subqueries containing multiple NULLs. + 16. fix: the pause operation of stream computing might fail. + 17. fix: when writing data into a sub-table with a table name length of 192 characters using an SQL statement, errors may occur if the table name is enclosed in backticks (`). + 18. fix: when performing a join query on super tables across different databases, if each database contains only one vnode, the query will return an error. + 19. fix: no enough disk space cause taosX panic + 20. fix: when write data to a super table, using both bound and unbound simultaneously will trigger an exception. + 21. fix: metrics non-exist cause panic when connect with agent + 22. fix: when creating indexes for tag with a large character length, taosd may crash. + 23. fix: when the input parameters for the functions first, last, last_row, and char exceed 127, the taosd may crash. https://github.com/taosdata/TDengine/issues/29241 + 24. fix: when the number of rows in the result set of the LIMIT statement exceeds the size of a single data block, the returned count does not match the expectation. + 25. fix: when synchronizing data between clusters, if the target task is deleted, the source cluster may run out of memory + 26. fix: metadata read-write lock misconfiguration leads to a very small chance of blocking writes. + 27. fix: when importing CSV files using the INSERT INTO statement on the Windows platform, the absence of a newline character at the end of the file may lead to an issue of infinite loop reading. + 28. fix: after the tags of the table are updated, the stream computing fails to recognize and apply the ne values. + 29. fix: fix kafka timeout issue and improve performance and stability + 30. fix: in SQL queries, when both 'is null' and invalid 'in' filter conditions are included simultaneously, the query results are incorrect. https://github.com/taosdata/TDengine/issues/29067 + 31. fix: SQL queries containing both 'IN' and 'BETWEEN' filter conditions result in incorrect query results. https://github.com/taosdata/TDengine/issues/#28989 + 32. fix: when performing multiplication or division operations between timestamp and numeric types, the results are incorrect. https://github.com/taosdata/TDengine/issues/28339 + 33. fix: data type conversion error in the IN statement leads to incorrect query results. https://github.com/taosdata/TDengine/issues/29047 https://github.com/taosdata/TDengine/issues/28902 + 34. fix: the error in filtering results when constant conditions are combined with OR operators. https://github.com/taosdata/TDengine/issues/28904 + 35. fix: when performing subtraction operation on timestamp type, the negative value is not considered. https://github.com/taosdata/TDengine/issues/28906 + 36. fix: tag values may display incorrectly when using GROUP BY tag synatix + 37. fix: gcc < 10 bug cause taosX compile error + diff --git a/docs/en/28-releases/03-notes/index.md b/docs/en/28-releases/03-notes/index.md index f8c44ef2af..efdedb6c07 100644 --- a/docs/en/28-releases/03-notes/index.md +++ b/docs/en/28-releases/03-notes/index.md @@ -5,6 +5,7 @@ slug: /release-history/release-notes [3.3.4.8](./3-3-4-8/) +[3.3.5.0](./3.3.5.0) [3.3.4.3](./3-3-4-3/) [3.3.3.0](./3-3-3-0/) diff --git a/docs/zh/28-releases/01-tdengine.md b/docs/zh/28-releases/01-tdengine.md index 43129fc242..356777acdc 100644 --- a/docs/zh/28-releases/01-tdengine.md +++ b/docs/zh/28-releases/01-tdengine.md @@ -24,6 +24,10 @@ TDengine 3.x 各版本安装包下载链接如下： import Release from "/components/ReleaseV3"; +## 3.3.5.0 + + + ## 3.3.4.8 diff --git a/docs/zh/28-releases/03-notes/3.3.5.0.md b/docs/zh/28-releases/03-notes/3.3.5.0.md new file mode 100755 index 0000000000..8b5c639813 --- /dev/null +++ b/docs/zh/28-releases/03-notes/3.3.5.0.md @@ -0,0 +1,84 @@ +--- +title: 3.3.5.0 版本说明 +sidebar_label: 3.3.5.0 +description: 3.3.5.0 版本说明 +--- + +## 特性 + 1. 特性：MQTT 稳定性和性能提升 + 2. 特性：taosX 增量备份与恢复 + 3. 特性：JDBC WebSocket 连接支持 STMT2 接口 + 4. 特性：Rust 连接器支持 STMT2 接口 + 5. 特性: taos-CLI 中在错误提示中增加错误码 + 6. 特性：Python 连接器对接 SuperSet + 7. 特性：Explorer 可配置 Grafana Dashboard + 8. 特性：taosX-agent 支持配置内存缓存队列长度 + +## 优化 + 1. 优化：调整 telemerty 的上报机制 + 2. 优化：支持通过 SQL 统计指定 DB 的磁盘空间 + 3. 优化：在服务端增加查询内存管控 + 4. 优化：INTERVAL 子句允许使用 AUTO 关键字来指定自动计算窗口偏移量 + 5. 优化：减少在多级存储迁移数据时对数据写入性能的影响 + 6. 优化：Grafana 插件 UI 转为 React 以完整适配 11.3.0 版本 + 7. 优化：taosAdapter websocket 接口优化 + 8. 优化：taosX 添加健康状态 + 9. 优化：taosX 支持可配置的异常数据处理 + 10. 优化：支持为客户端连接设置选项，包括时区、字符集、用户 IP、用户名称 + 11. 优化：taosdump 支持查询超时或连接端开时自动重连 + 12. 优化：允许为已经订阅的 tag 列创建索引 + 13. 优化：taosX 支持密码中包含特殊字符 + 14. 优化：提升开启 Last 缓存时的数据写入性能 + 15. 优化：COMPACT 命令支持自动执行、并发度设置及执行进度观测 + 16. 优化：支持通过 SQL 语句修改全局配置参数并持久化 + 17. 优化：更新各数据类型的压缩算法默认值，提高大部分场景的压缩比 + 18. 优化：taosBenchmark 在 Mac/Windows 上 --nodrop 参数行为修复 + 19. 优化：禁止 DB 内 Compact 和副本变更相关操作同时进行（企业版） + 20. 优化：taosdump 支持复合主键表导出 + 21. 优化：支持在 show queries 和 show connections 语句的返回结果中显示用户 IP 和用户名称 + 22. 优化：JDBC 支持多表批量写入 + 23. 优化：支持对多级存储中的 dataDir 参数进行动态调整。 + 24. 优化：taosX 数据库文件默认使用 data_dir + 25. 优化：强制要求用户设置强密码，密码长度必须为 8 到 16 位，并且至少包含大写字母、小写字母、数字、特殊字符中的三类 + 26. 优化：提高客户端获取新 Leader 的速度 + 27. 优化：OPC 点位正则匹配支持 “非” + +## 修复 + 1. 修复：流计算在高负载场景下更新 checkpoint 时可能出现死锁 + 2. 修复：TMQ 同步目标端报错无法恢复的问题 + 3. 修复：WAL 中的某个数据块校验失败时 taosd 无法启动的问题 + 4. 修复：taosBenchmark 多副本下节点宕机写入失败 + 5. 修复：日志文件切换频繁时日志文件可能丢失的问题 + 6. 修复：窗口内数据更新导致流计算停止的问题 + 7. 修复：libtaosws.so 读取数据时如果连接中端，错误码设置有误 + 8. 修复：OPC 数据点位以 @ 开头时解析错误 + 9. 修复：taosbechmark 解决云服务下show vgroups 权限问题 + 10. 修复：taosX 迁移支持列压缩的超宽超级表时报语法错误 + 11. 修复：Vnode 占用内存估算错误的问题 + 12. 修复：对 varchar 类型的常量字符串进行 union all 查询时失败的问题 + 13. 修复：执行事务期间切换 leader 会导致 mnode 死锁 + 14. 修复：ws_stmt_get_tag_fields 返回地址非法 + 15. 修复：UNION 语句中存在包含多个 NULL 的子查询时执行报错 + 16. 修复：流计算的暂停操作可能失败的问题 + 17. 修复：通过 SQL 语句将数据写入一个表名长度为192个字符的子表时，如果表名被反引号（`）包围，可能会导致错误 + 18. 修复：在对不同数据库的超级表进行联合查询时，如果两个数据库各自仅包含一个虚拟节点，查询将返回错误。 + 19. 修复：磁盘空间不足时 taosX panic 导致任务无法恢复 + 20. 修复：在向超级表中写入数据的过程中，如果同时使用了绑定和非绑定的方式，将会引发异常 + 21. 修复：特殊情况下 taosX agent 连接时 metrics 不存在导致panic + 22. 修复：为字符长度较大的标签数据创建索引时，taosd 可能 crash + 23. 修复：当函数 first、last、last_row 和 char 的输入参数超过 127 时，taosd 可能会崩溃的问题 https://github.com/taosdata/TDengine/issues/29241 + 24. 修复：Limit 语句的结果集条数超过一个数据块时，返回条数和预期不符合 + 25. 修复：集群间数据同步时，如果删除掉目标集群任务，源集群可能 OOM + 26. 修复：元数据读写锁设置错误导致有极小几率阻塞写入的问题 + 27. 修复：在 Windows 平台上使用 INSERT INTO 语句导入 CSV 文件时，如果文件末尾没有换行符，可能会导致无限循环读取的问题 + 28. 修复：在源表的标签更新后，流计算未能识别并应用新的标签值 + 29. 修复：调整 Kafka 订阅参数提升 Kakfa 数据写入性能和稳定性 + 30. 修复：SQL 查询中同时包含 is null 和无效的 in 筛选条件时，查询结果有误 https://github.com/taosdata/TDengine/issues/29067 + 31. 修复：SQL 查询中同时包含 in 和 between 筛选条件时，查询结果有误 https://github.com/taosdata/TDengine/issues/#28989 + 32. 修复：timestamp 类型和数值类型进行乘除运算时结果有误 https://github.com/taosdata/TDengine/issues/28339 + 33. 修复：IN 条件中的数据类型转换错误导致查询结果不正确 https://github.com/taosdata/TDengine/issues/29047 https://github.com/taosdata/TDengine/issues/28902 + 34. 修复：常量条件和 OR 运算符结合时筛选结果错误 https://github.com/taosdata/TDengine/issues/28904 + 35. 修复：对时间戳类型进行减法运算时未考虑负值的情况 https://github.com/taosdata/TDengine/issues/28906 + 36. 修复：GROUP BY tag 时某些标签值显示错误的问题 + 37. 修复：旧版本 GCC Bug 导致编译失败 + diff --git a/docs/zh/28-releases/03-notes/index.md b/docs/zh/28-releases/03-notes/index.md index 03984f130f..27898aa2df 100644 --- a/docs/zh/28-releases/03-notes/index.md +++ b/docs/zh/28-releases/03-notes/index.md @@ -4,6 +4,7 @@ sidebar_label: 版本说明 description: 各版本版本说明 --- +[3.3.5.0](./3.3.5.0) [3.3.4.8](./3.3.4.8) [3.3.4.3](./3.3.4.3) [3.3.3.0](./3.3.3.0) From 1fea561e6c5b6b1a7e310c45738bee69df1ae0c3 Mon Sep 17 00:00:00 2001 From: haoranchen Date: Thu, 2 Jan 2025 16:07:29 +0800 Subject: [PATCH 68/73] Update 3.3.5.0.md --- docs/en/28-releases/03-notes/3.3.5.0.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/en/28-releases/03-notes/3.3.5.0.md b/docs/en/28-releases/03-notes/3.3.5.0.md index 3326b3f0f0..087b55d050 100755 --- a/docs/en/28-releases/03-notes/3.3.5.0.md +++ b/docs/en/28-releases/03-notes/3.3.5.0.md @@ -7,11 +7,11 @@ slug: /release-history/release-notes/3.3.5.0 ## Features 1. feat: refactor MQTT to improve stability and performance - 2. feat: refactor taosX incremental backup-restore + 2. feat: refactor taosX incremental backup-restore 3. feat: add stmt2 apis in JDBC via websocket connection 4. feat: add stmt2 api in Rust connector - 5. feat: Adding error codes in error prompts in taos-CLI - 6. feat: SuperSet can connect TDengine with python connector + 5. feat: adding error codes in error prompts in taos-CLI + 6. feat: superSet can connect TDengine with python connector 7. feat: configurable grafana dashboards in explorer management 8. feat: add taosX-agent in-memory cache queu capacity option @@ -19,7 +19,7 @@ slug: /release-history/release-notes/3.3.5.0 1. enh: adjust the reporting mechanism of telemetry. 2. enh: support for SQL-based statistics of disk space for a specified DB. 3. enh: add memory management for SQL queries on the server side - 4. enh: INTERVAL clause allows the use of the AUTO keyword to specify the window offset. + 4. enh: interval clause allows the use of the AUTO keyword to specify the window offset. 5. enh: reduce the impact on data write performance during data migration across multi-level storage 6. enh: migrate from angular to react for grafana 11.3+ 7. enh: refactor taosAdapter websocket api for a slightly better perf @@ -30,7 +30,7 @@ slug: /release-history/release-notes/3.3.5.0 12. enh: allow creating index for tags that already subscribed 13. enh: taosX now support literal special chars in password 14. enh: improve data write performance when Last Cache is activated. - 15. enh: COMPACT command supports automatic execution, concurrency setting, and progress observation. + 15. enh: compact command supports automatic execution, concurrency setting, and progress observation. 16. enh: support update global configuration parameters through SQL statements and persisting them. 17. enh: update the default compression method for all data types to improve the compression ratio in most scenarios. 18. enh: taosBenchmark --nodrop fix for mac/window @@ -59,7 +59,7 @@ slug: /release-history/release-notes/3.3.5.0 12. fix: failed to perform UNION ALL query on constant strings of the varchar type. 13. fix: leader transfer during the execution of transaction may cause deadlock. 14. fix: rust connector invliad pointer addr in ws_stmt_get_tag_fields - 15. fix: UNION statement fails when executing with subqueries containing multiple NULLs. + 15. fix: union statement fails when executing with subqueries containing multiple NULLs. 16. fix: the pause operation of stream computing might fail. 17. fix: when writing data into a sub-table with a table name length of 192 characters using an SQL statement, errors may occur if the table name is enclosed in backticks (`). 18. fix: when performing a join query on super tables across different databases, if each database contains only one vnode, the query will return an error. @@ -74,8 +74,8 @@ slug: /release-history/release-notes/3.3.5.0 27. fix: when importing CSV files using the INSERT INTO statement on the Windows platform, the absence of a newline character at the end of the file may lead to an issue of infinite loop reading. 28. fix: after the tags of the table are updated, the stream computing fails to recognize and apply the ne values. 29. fix: fix kafka timeout issue and improve performance and stability - 30. fix: in SQL queries, when both 'is null' and invalid 'in' filter conditions are included simultaneously, the query results are incorrect. https://github.com/taosdata/TDengine/issues/29067 - 31. fix: SQL queries containing both 'IN' and 'BETWEEN' filter conditions result in incorrect query results. https://github.com/taosdata/TDengine/issues/#28989 + 30. fix: in sql queries, when both 'is null' and invalid 'in' filter conditions are included simultaneously, the query results are incorrect. https://github.com/taosdata/TDengine/issues/29067 + 31. fix: sql queries containing both 'IN' and 'BETWEEN' filter conditions result in incorrect query results. https://github.com/taosdata/TDengine/issues/#28989 32. fix: when performing multiplication or division operations between timestamp and numeric types, the results are incorrect. https://github.com/taosdata/TDengine/issues/28339 33. fix: data type conversion error in the IN statement leads to incorrect query results. https://github.com/taosdata/TDengine/issues/29047 https://github.com/taosdata/TDengine/issues/28902 34. fix: the error in filtering results when constant conditions are combined with OR operators. https://github.com/taosdata/TDengine/issues/28904 From 2cc688cc9974c2f64ec7643d864a52dd8472bdbe Mon Sep 17 00:00:00 2001 From: Shengliang Guan Date: Thu, 2 Jan 2025 16:13:15 +0800 Subject: [PATCH 69/73] fix typos --- docs/en/28-releases/03-notes/3.3.5.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/28-releases/03-notes/3.3.5.0.md b/docs/en/28-releases/03-notes/3.3.5.0.md index 087b55d050..a82ba082b0 100755 --- a/docs/en/28-releases/03-notes/3.3.5.0.md +++ b/docs/en/28-releases/03-notes/3.3.5.0.md @@ -10,7 +10,7 @@ slug: /release-history/release-notes/3.3.5.0 2. feat: refactor taosX incremental backup-restore 3. feat: add stmt2 apis in JDBC via websocket connection 4. feat: add stmt2 api in Rust connector - 5. feat: adding error codes in error prompts in taos-CLI + 5. feat: add error codes in error prompts in taos-CLI 6. feat: superSet can connect TDengine with python connector 7. feat: configurable grafana dashboards in explorer management 8. feat: add taosX-agent in-memory cache queu capacity option @@ -22,7 +22,7 @@ slug: /release-history/release-notes/3.3.5.0 4. enh: interval clause allows the use of the AUTO keyword to specify the window offset. 5. enh: reduce the impact on data write performance during data migration across multi-level storage 6. enh: migrate from angular to react for grafana 11.3+ - 7. enh: refactor taosAdapter websocket api for a slightly better perf + 7. enh: refactor taosAdapter websocket api for a slightly better perf 8. enh: add health state in taosX task status 9. enh: taosX add configurations to handle exceptions 10. enh: support setting options for client connections, including time zone, character set, user IP, and user name. @@ -50,7 +50,7 @@ slug: /release-history/release-notes/3.3.5.0 3. fix: taosd cannot start when there is data corruption in a block within the WAL 4. fix: taosBenchmark fails when taosd disconnected in replica 2/3 5. fix: log files being lost when they are switched frequently. - 6. fix: he stream computing stops due to the data update within the window. + 6. fix: the stream computing stops due to the data update within the window. 7. fix: libtaosws.so sets an incorrect error code when the connection is terminated while fetch data. 8. fix: taosX opc error in case of @-prefixed name 9. fix: fix permission denied with show vgroups sql in cloud From 72e9973000db814649f6c66416cd724bded72480 Mon Sep 17 00:00:00 2001 From: haoranchen Date: Thu, 2 Jan 2025 16:14:43 +0800 Subject: [PATCH 70/73] Update 3.3.5.0.md --- docs/en/28-releases/03-notes/3.3.5.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/28-releases/03-notes/3.3.5.0.md b/docs/en/28-releases/03-notes/3.3.5.0.md index a82ba082b0..a740daa76b 100755 --- a/docs/en/28-releases/03-notes/3.3.5.0.md +++ b/docs/en/28-releases/03-notes/3.3.5.0.md @@ -75,7 +75,7 @@ slug: /release-history/release-notes/3.3.5.0 28. fix: after the tags of the table are updated, the stream computing fails to recognize and apply the ne values. 29. fix: fix kafka timeout issue and improve performance and stability 30. fix: in sql queries, when both 'is null' and invalid 'in' filter conditions are included simultaneously, the query results are incorrect. https://github.com/taosdata/TDengine/issues/29067 - 31. fix: sql queries containing both 'IN' and 'BETWEEN' filter conditions result in incorrect query results. https://github.com/taosdata/TDengine/issues/#28989 + 31. fix: sql queries containing both 'IN' and 'BETWEEN' filter conditions result in incorrect query results. https://github.com/taosdata/TDengine/issues/28989 32. fix: when performing multiplication or division operations between timestamp and numeric types, the results are incorrect. https://github.com/taosdata/TDengine/issues/28339 33. fix: data type conversion error in the IN statement leads to incorrect query results. https://github.com/taosdata/TDengine/issues/29047 https://github.com/taosdata/TDengine/issues/28902 34. fix: the error in filtering results when constant conditions are combined with OR operators. https://github.com/taosdata/TDengine/issues/28904 From f80a1705d8402e242889cc765de1be3f201d2c69 Mon Sep 17 00:00:00 2001 From: huohong <346479823@qq.com> Date: Thu, 2 Jan 2025 16:17:48 +0800 Subject: [PATCH 71/73] Update 3.3.5.0.md --- docs/zh/28-releases/03-notes/3.3.5.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/zh/28-releases/03-notes/3.3.5.0.md b/docs/zh/28-releases/03-notes/3.3.5.0.md index 8b5c639813..f3bd9ea4a1 100755 --- a/docs/zh/28-releases/03-notes/3.3.5.0.md +++ b/docs/zh/28-releases/03-notes/3.3.5.0.md @@ -52,7 +52,7 @@ description: 3.3.5.0 版本说明 6. 修复：窗口内数据更新导致流计算停止的问题 7. 修复：libtaosws.so 读取数据时如果连接中端，错误码设置有误 8. 修复：OPC 数据点位以 @ 开头时解析错误 - 9. 修复：taosbechmark 解决云服务下show vgroups 权限问题 + 9. 修复：taosBenchmark 解决云服务下show vgroups 权限问题 10. 修复：taosX 迁移支持列压缩的超宽超级表时报语法错误 11. 修复：Vnode 占用内存估算错误的问题 12. 修复：对 varchar 类型的常量字符串进行 union all 查询时失败的问题 @@ -74,7 +74,7 @@ description: 3.3.5.0 版本说明 28. 修复：在源表的标签更新后，流计算未能识别并应用新的标签值 29. 修复：调整 Kafka 订阅参数提升 Kakfa 数据写入性能和稳定性 30. 修复：SQL 查询中同时包含 is null 和无效的 in 筛选条件时，查询结果有误 https://github.com/taosdata/TDengine/issues/29067 - 31. 修复：SQL 查询中同时包含 in 和 between 筛选条件时，查询结果有误 https://github.com/taosdata/TDengine/issues/#28989 + 31. 修复：SQL 查询中同时包含 in 和 between 筛选条件时，查询结果有误 https://github.com/taosdata/TDengine/issues/28989 32. 修复：timestamp 类型和数值类型进行乘除运算时结果有误 https://github.com/taosdata/TDengine/issues/28339 33. 修复：IN 条件中的数据类型转换错误导致查询结果不正确 https://github.com/taosdata/TDengine/issues/29047 https://github.com/taosdata/TDengine/issues/28902 34. 修复：常量条件和 OR 运算符结合时筛选结果错误 https://github.com/taosdata/TDengine/issues/28904 From 99173dfcd8bcd5c5084930337a9112dc3876baae Mon Sep 17 00:00:00 2001 From: huohong <346479823@qq.com> Date: Thu, 2 Jan 2025 16:18:49 +0800 Subject: [PATCH 72/73] Update 3.3.5.0.md --- docs/zh/28-releases/03-notes/3.3.5.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh/28-releases/03-notes/3.3.5.0.md b/docs/zh/28-releases/03-notes/3.3.5.0.md index f3bd9ea4a1..7e16516344 100755 --- a/docs/zh/28-releases/03-notes/3.3.5.0.md +++ b/docs/zh/28-releases/03-notes/3.3.5.0.md @@ -9,7 +9,7 @@ description: 3.3.5.0 版本说明 2. 特性：taosX 增量备份与恢复 3. 特性：JDBC WebSocket 连接支持 STMT2 接口 4. 特性：Rust 连接器支持 STMT2 接口 - 5. 特性: taos-CLI 中在错误提示中增加错误码 + 5. 特性：taos-CLI 中在错误提示中增加错误码 6. 特性：Python 连接器对接 SuperSet 7. 特性：Explorer 可配置 Grafana Dashboard 8. 特性：taosX-agent 支持配置内存缓存队列长度 From e24fab999c108c0ad5447b55aaf7fc972bd6267b Mon Sep 17 00:00:00 2001 From: wangjiaming0909 Date: Thu, 2 Jan 2025 16:33:42 +0800 Subject: [PATCH 73/73] Revert "modify Jenkinsfile2" This reverts commit e066c1ed2909b4b9494bd1e2bedfd6c85e0b7be8. --- Jenkinsfile2 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Jenkinsfile2 b/Jenkinsfile2 index 52658434a4..1b2f28908c 100644 --- a/Jenkinsfile2 +++ b/Jenkinsfile2 @@ -165,7 +165,7 @@ def pre_test(){ script { sh ''' cd ${WK} - git checkout fix/main/TD-33390 + git checkout ''' + env.CHANGE_TARGET + ''' cd ${WKC} git checkout ''' + env.CHANGE_TARGET + ''' ''' @@ -274,7 +274,7 @@ def pre_test_win(){ script { bat ''' cd %WIN_INTERNAL_ROOT% - git checkout fix/main/TD-33390 + git checkout ''' + env.CHANGE_TARGET + ''' ''' bat ''' cd %WIN_COMMUNITY_ROOT%