diff --git a/docs/en/14-reference/03-taos-sql/10-function.md b/docs/en/14-reference/03-taos-sql/10-function.md index e130442279..72ca878ce8 100644 --- a/docs/en/14-reference/03-taos-sql/10-function.md +++ b/docs/en/14-reference/03-taos-sql/10-function.md @@ -656,6 +656,7 @@ use_current_timezone: { For example, if the time zone configured by the Client is UTC + 0800, TIMETRUNCATE ('2020-01-01 23:00:00', 1d, 0) returns the result of '2020-01-01 08:00:00'. When using TIMETRUNCATE ('2020-01-01 23:00:00', 1d, 1), the result is 2020-01-01 00:00:00 '. When use_current_timezone is not specified, use_current_timezone defaults to 1. +- When truncating a time value to the week (1w), weeks are determined using the Unix epoch (1970-01-01T00:00:00Z UTC). The Unix epoch was on a Thursday, so all calculated weeks begin on Thursday. #### TIMEZONE diff --git a/docs/en/14-reference/05-connectors/10-cpp.mdx b/docs/en/14-reference/05-connectors/10-cpp.mdx index b429afde97..6a570b2490 100644 --- a/docs/en/14-reference/05-connectors/10-cpp.mdx +++ b/docs/en/14-reference/05-connectors/10-cpp.mdx @@ -64,64 +64,20 @@ In the above example code, `taos_connect()` establishes a connection to port 603 ## Sample program -This section shows sample code for standard access methods to TDengine clusters using the client driver. +This section shows sample code for standard access methods to TDengine clusters using the client driver. -### Synchronous query example +- [Synchronous query example](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/demo.c) -
-Synchronous query +- [Asynchronous query example](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/asyncdemo.c) -```c -{{#include examples/c/demo.c}} -``` +- [Parameter binding example](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/prepare.c) -
+- [Schemaless writing example](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/schemaless.c) -### Asynchronous query example - -
-Asynchronous queries - -```c -{{#include examples/c/asyncdemo.c}} -``` - -
- -### Parameter binding example - -
-Parameter Binding - -```c -{{#include examples/c/prepare.c}} -``` - -
- -### Pattern-free writing example - -
-Mode free write - -```c -{{#include examples/c/schemaless.c}} -``` - -
- -### Subscription and consumption example - -
-Subscribe and consume - -```c -``` - -
+- [Subscription and consumption example](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/tmq.c) :::info -More example code and downloads are available at [GitHub](https://github.com/taosdata/TDengine/tree/develop/examples/c). +More example code and downloads are available at [GitHub](https://github.com/taosdata/TDengine/tree/main/docs/examples/c). You can find it in the installation directory under the `examples/c` path. This directory has a makefile and can be compiled under Linux/macOS by executing `make` directly. **Hint:** When compiling in an ARM environment, please remove `-msse4.2` from the makefile. This option is only supported on the x64/x86 hardware platforms. @@ -311,7 +267,7 @@ Starting with versions 2.1.1.0 and 2.1.2.0, TDengine has significantly improved Note: If `taos_stmt_execute()` succeeds, you can reuse the parsed result of `taos_stmt_prepare()` to bind new data in steps 3 to 6 if you don't need to change the SQL command. However, if there is an execution error, it is not recommended to continue working in the current context but release the resources and start again with `taos_stmt_init()` steps. -The specific functions related to the interface are as follows (see also the [prepare.c](https://github.com/taosdata/TDengine/blob/develop/examples/c/prepare.c) file for the way to use the corresponding functions) +The specific functions related to the interface are as follows (see also the [prepare.c](https://github.com/taosdata/TDengine/blob/develop/docs/examples/c/prepare.c) file for the way to use the corresponding functions) - `TAOS_STMT* taos_stmt_init(TAOS *taos)` @@ -661,4 +617,4 @@ In addition to writing data using the SQL method or the parameter binding API, w - tmq_get_table_name : table name of result, NULL if failed - tmq_get_res_type : result type tmq_res_t - tmq_get_topic_name : topic name of result, NULL if failed - - tmq_get_db_name : db name of result, NULL if failed \ No newline at end of file + - tmq_get_db_name : db name of result, NULL if failed diff --git a/docs/en/14-reference/05-connectors/14-java.mdx b/docs/en/14-reference/05-connectors/14-java.mdx index 8192807c6d..aa7e91b7fa 100644 --- a/docs/en/14-reference/05-connectors/14-java.mdx +++ b/docs/en/14-reference/05-connectors/14-java.mdx @@ -150,7 +150,7 @@ TDengine currently supports timestamp, number, character, Boolean type, and the Due to historical reasons, the BINARY type data in TDengine is not truly binary data and is no longer recommended for use. Please use VARBINARY type instead. GEOMETRY type is binary data in little endian byte order, which complies with the WKB specification. For detailed information, please refer to [Data Type](../../taos-sql/data-type/) For WKB specifications, please refer to [Well Known Binary (WKB)](https://libgeos.org/specifications/wkb/) -For Java connector, the jts library can be used to easily create GEOMETRY type objects, serialize them, and write them to TDengine. Here is an example [Geometry example](https://github.com/taosdata/TDengine/blob/3.0/docs/examples/java/src/main/java/com/taos/example/GeometryDemo.java) +For Java connector, the jts library can be used to easily create GEOMETRY type objects, serialize them, and write them to TDengine. Here is an example [Geometry example](https://github.com/taosdata/TDengine/blob/main/docs/examples/java/src/main/java/com/taos/example/GeometryDemo.java) ## Installation Steps @@ -662,7 +662,7 @@ Example usage is as follows. ### More sample programs -The source code of the sample application is under `TDengine/examples/JDBC`: +The source code of the sample application is under `TDengine/docs/examples/JDBC`: - JDBCDemo: JDBC sample source code. - connectionPools: using taos-jdbcdriver in connection pools such as HikariCP, Druid, dbcp, c3p0, etc. @@ -671,7 +671,7 @@ The source code of the sample application is under `TDengine/examples/JDBC`: - springbootdemo: using taos-jdbcdriver in Springboot. - consumer-demo: consumer TDengine data example, the consumption rate can be controlled by parameters. -[JDBC example](https://github.com/taosdata/TDengine/tree/3.0/examples/JDBC) +[JDBC example](https://github.com/taosdata/TDengine/tree/main/docs/examples/JDBC) ## Frequently Asked Questions diff --git a/docs/en/14-reference/05-connectors/20-go.mdx b/docs/en/14-reference/05-connectors/20-go.mdx index 7be57baf6b..39ee97687c 100644 --- a/docs/en/14-reference/05-connectors/20-go.mdx +++ b/docs/en/14-reference/05-connectors/20-go.mdx @@ -368,7 +368,7 @@ The TDengine Go client library supports subscription functionality with the foll ### More sample programs -* [sample program](https://github.com/taosdata/driver-go/tree/3.0/examples) +* [sample program](https://github.com/taosdata/driver-go/tree/main/examples) ## Frequently Asked Questions diff --git a/docs/en/14-reference/05-connectors/26-rust.mdx b/docs/en/14-reference/05-connectors/26-rust.mdx index 92a8e24dbb..310f9760a5 100644 --- a/docs/en/14-reference/05-connectors/26-rust.mdx +++ b/docs/en/14-reference/05-connectors/26-rust.mdx @@ -403,7 +403,7 @@ The following parameters can be configured for the TMQ DSN. Only `group.id` is m #### Full Sample Code -For more information, see [GitHub sample file](https://github.com/taosdata/TDengine/blob/3.0/docs/examples/rust/nativeexample/examples/subscribe_demo.rs). +For more information, see [GitHub sample file](https://github.com/taosdata/TDengine/blob/main/docs/examples/rust/nativeexample/examples/subscribe_demo.rs). ### Use with connection pool @@ -437,9 +437,9 @@ let taos = pool.get()?; ### More sample programs -The source code of the sample application is under `TDengine/examples/rust` : +The source code of the sample application is under `TDengine/docs/examples/rust` : -[rust example](https://github.com/taosdata/TDengine/tree/3.0/examples/rust) +[rust example](https://github.com/taosdata/TDengine/tree/main/docs/examples/rust) ## Frequently Asked Questions diff --git a/examples/JDBC/JDBCDemo/README-jdbc-windows.md b/docs/examples/JDBC/JDBCDemo/README-jdbc-windows.md similarity index 100% rename from examples/JDBC/JDBCDemo/README-jdbc-windows.md rename to docs/examples/JDBC/JDBCDemo/README-jdbc-windows.md diff --git a/examples/JDBC/JDBCDemo/pom.xml b/docs/examples/JDBC/JDBCDemo/pom.xml similarity index 100% rename from examples/JDBC/JDBCDemo/pom.xml rename to docs/examples/JDBC/JDBCDemo/pom.xml diff --git a/examples/JDBC/JDBCDemo/readme.md b/docs/examples/JDBC/JDBCDemo/readme.md similarity index 100% rename from examples/JDBC/JDBCDemo/readme.md rename to docs/examples/JDBC/JDBCDemo/readme.md diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ConsumerLoop.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ConsumerLoop.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ConsumerLoop.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ConsumerLoop.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/GeometryDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/GeometryDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/GeometryDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/GeometryDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcBasicDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcBasicDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcBasicDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcBasicDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcRestfulDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcRestfulDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcRestfulDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/JdbcRestfulDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingFullDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingFullDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingFullDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/ParameterBindingFullDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/Util.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/Util.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/Util.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/Util.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingDemo.java diff --git a/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingFullDemo.java b/docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingFullDemo.java similarity index 100% rename from examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingFullDemo.java rename to docs/examples/JDBC/JDBCDemo/src/main/java/com/taosdata/example/WSParameterBindingFullDemo.java diff --git a/examples/JDBC/SpringJdbcTemplate/.gitignore b/docs/examples/JDBC/SpringJdbcTemplate/.gitignore similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/.gitignore rename to docs/examples/JDBC/SpringJdbcTemplate/.gitignore diff --git a/examples/JDBC/SpringJdbcTemplate/pom.xml b/docs/examples/JDBC/SpringJdbcTemplate/pom.xml similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/pom.xml rename to docs/examples/JDBC/SpringJdbcTemplate/pom.xml diff --git a/examples/JDBC/SpringJdbcTemplate/readme.md b/docs/examples/JDBC/SpringJdbcTemplate/readme.md similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/readme.md rename to docs/examples/JDBC/SpringJdbcTemplate/readme.md diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/App.java b/docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/App.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/App.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/App.java diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatement.java b/docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatement.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatement.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatement.java diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatementImpl.java b/docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatementImpl.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatementImpl.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/ExecuteAsStatementImpl.java diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDao.java b/docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDao.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDao.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDao.java diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDaoImpl.java b/docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDaoImpl.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDaoImpl.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/dao/WeatherDaoImpl.java diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/domain/Weather.java b/docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/domain/Weather.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/domain/Weather.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/java/com/taosdata/example/jdbcTemplate/domain/Weather.java diff --git a/examples/JDBC/SpringJdbcTemplate/src/main/resources/applicationContext.xml b/docs/examples/JDBC/SpringJdbcTemplate/src/main/resources/applicationContext.xml similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/main/resources/applicationContext.xml rename to docs/examples/JDBC/SpringJdbcTemplate/src/main/resources/applicationContext.xml diff --git a/examples/JDBC/SpringJdbcTemplate/src/test/java/com/taosdata/example/jdbcTemplate/BatcherInsertTest.java b/docs/examples/JDBC/SpringJdbcTemplate/src/test/java/com/taosdata/example/jdbcTemplate/BatcherInsertTest.java similarity index 100% rename from examples/JDBC/SpringJdbcTemplate/src/test/java/com/taosdata/example/jdbcTemplate/BatcherInsertTest.java rename to docs/examples/JDBC/SpringJdbcTemplate/src/test/java/com/taosdata/example/jdbcTemplate/BatcherInsertTest.java diff --git a/examples/JDBC/connectionPools/README-cn.md b/docs/examples/JDBC/connectionPools/README-cn.md similarity index 100% rename from examples/JDBC/connectionPools/README-cn.md rename to docs/examples/JDBC/connectionPools/README-cn.md diff --git a/examples/JDBC/connectionPools/pom.xml b/docs/examples/JDBC/connectionPools/pom.xml similarity index 100% rename from examples/JDBC/connectionPools/pom.xml rename to docs/examples/JDBC/connectionPools/pom.xml diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ConnectionPoolDemo.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ConnectionPoolDemo.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ConnectionPoolDemo.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ConnectionPoolDemo.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/DruidDemo.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/DruidDemo.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/DruidDemo.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/DruidDemo.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/HikariDemo.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/HikariDemo.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/HikariDemo.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/HikariDemo.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ProxoolDemo.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ProxoolDemo.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ProxoolDemo.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/ProxoolDemo.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/common/InsertTask.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/common/InsertTask.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/common/InsertTask.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/common/InsertTask.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/C3p0Builder.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/C3p0Builder.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/C3p0Builder.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/C3p0Builder.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DbcpBuilder.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DbcpBuilder.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DbcpBuilder.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DbcpBuilder.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DruidPoolBuilder.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DruidPoolBuilder.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DruidPoolBuilder.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/DruidPoolBuilder.java diff --git a/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/HikariCpBuilder.java b/docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/HikariCpBuilder.java similarity index 100% rename from examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/HikariCpBuilder.java rename to docs/examples/JDBC/connectionPools/src/main/java/com/taosdata/example/pool/HikariCpBuilder.java diff --git a/examples/JDBC/connectionPools/src/main/resources/log4j.properties b/docs/examples/JDBC/connectionPools/src/main/resources/log4j.properties similarity index 100% rename from examples/JDBC/connectionPools/src/main/resources/log4j.properties rename to docs/examples/JDBC/connectionPools/src/main/resources/log4j.properties diff --git a/examples/JDBC/connectionPools/src/main/resources/proxool.xml b/docs/examples/JDBC/connectionPools/src/main/resources/proxool.xml similarity index 100% rename from examples/JDBC/connectionPools/src/main/resources/proxool.xml rename to docs/examples/JDBC/connectionPools/src/main/resources/proxool.xml diff --git a/examples/JDBC/consumer-demo/pom.xml b/docs/examples/JDBC/consumer-demo/pom.xml similarity index 100% rename from examples/JDBC/consumer-demo/pom.xml rename to docs/examples/JDBC/consumer-demo/pom.xml diff --git a/examples/JDBC/consumer-demo/readme.md b/docs/examples/JDBC/consumer-demo/readme.md similarity index 100% rename from examples/JDBC/consumer-demo/readme.md rename to docs/examples/JDBC/consumer-demo/readme.md diff --git a/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Bean.java b/docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Bean.java similarity index 100% rename from examples/JDBC/consumer-demo/src/main/java/com/taosdata/Bean.java rename to docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Bean.java diff --git a/examples/JDBC/consumer-demo/src/main/java/com/taosdata/BeanDeserializer.java b/docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/BeanDeserializer.java similarity index 100% rename from examples/JDBC/consumer-demo/src/main/java/com/taosdata/BeanDeserializer.java rename to docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/BeanDeserializer.java diff --git a/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Config.java b/docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Config.java similarity index 100% rename from examples/JDBC/consumer-demo/src/main/java/com/taosdata/Config.java rename to docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Config.java diff --git a/examples/JDBC/consumer-demo/src/main/java/com/taosdata/ConsumerDemo.java b/docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/ConsumerDemo.java similarity index 100% rename from examples/JDBC/consumer-demo/src/main/java/com/taosdata/ConsumerDemo.java rename to docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/ConsumerDemo.java diff --git a/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Worker.java b/docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Worker.java similarity index 100% rename from examples/JDBC/consumer-demo/src/main/java/com/taosdata/Worker.java rename to docs/examples/JDBC/consumer-demo/src/main/java/com/taosdata/Worker.java diff --git a/examples/JDBC/mybatisplus-demo/.gitignore b/docs/examples/JDBC/mybatisplus-demo/.gitignore similarity index 100% rename from examples/JDBC/mybatisplus-demo/.gitignore rename to docs/examples/JDBC/mybatisplus-demo/.gitignore diff --git a/examples/JDBC/mybatisplus-demo/.mvn/wrapper/MavenWrapperDownloader.java b/docs/examples/JDBC/mybatisplus-demo/.mvn/wrapper/MavenWrapperDownloader.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/.mvn/wrapper/MavenWrapperDownloader.java rename to docs/examples/JDBC/mybatisplus-demo/.mvn/wrapper/MavenWrapperDownloader.java diff --git a/examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.jar b/docs/examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.jar similarity index 100% rename from examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.jar rename to docs/examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.jar diff --git a/examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.properties b/docs/examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.properties similarity index 100% rename from examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.properties rename to docs/examples/JDBC/mybatisplus-demo/.mvn/wrapper/maven-wrapper.properties diff --git a/examples/JDBC/mybatisplus-demo/mvnw b/docs/examples/JDBC/mybatisplus-demo/mvnw similarity index 100% rename from examples/JDBC/mybatisplus-demo/mvnw rename to docs/examples/JDBC/mybatisplus-demo/mvnw diff --git a/examples/JDBC/mybatisplus-demo/mvnw.cmd b/docs/examples/JDBC/mybatisplus-demo/mvnw.cmd similarity index 100% rename from examples/JDBC/mybatisplus-demo/mvnw.cmd rename to docs/examples/JDBC/mybatisplus-demo/mvnw.cmd diff --git a/examples/JDBC/mybatisplus-demo/pom.xml b/docs/examples/JDBC/mybatisplus-demo/pom.xml similarity index 100% rename from examples/JDBC/mybatisplus-demo/pom.xml rename to docs/examples/JDBC/mybatisplus-demo/pom.xml diff --git a/examples/JDBC/mybatisplus-demo/readme b/docs/examples/JDBC/mybatisplus-demo/readme similarity index 100% rename from examples/JDBC/mybatisplus-demo/readme rename to docs/examples/JDBC/mybatisplus-demo/readme diff --git a/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/MybatisplusDemoApplication.java b/docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/MybatisplusDemoApplication.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/MybatisplusDemoApplication.java rename to docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/MybatisplusDemoApplication.java diff --git a/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/config/MybatisPlusConfig.java b/docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/config/MybatisPlusConfig.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/config/MybatisPlusConfig.java rename to docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/config/MybatisPlusConfig.java diff --git a/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Temperature.java b/docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Temperature.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Temperature.java rename to docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Temperature.java diff --git a/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Weather.java b/docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Weather.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Weather.java rename to docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/domain/Weather.java diff --git a/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapper.java b/docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapper.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapper.java rename to docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapper.java diff --git a/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapper.java b/docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapper.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapper.java rename to docs/examples/JDBC/mybatisplus-demo/src/main/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapper.java diff --git a/examples/JDBC/mybatisplus-demo/src/main/resources/application.yml b/docs/examples/JDBC/mybatisplus-demo/src/main/resources/application.yml similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/main/resources/application.yml rename to docs/examples/JDBC/mybatisplus-demo/src/main/resources/application.yml diff --git a/examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapperTest.java b/docs/examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapperTest.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapperTest.java rename to docs/examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/TemperatureMapperTest.java diff --git a/examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapperTest.java b/docs/examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapperTest.java similarity index 100% rename from examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapperTest.java rename to docs/examples/JDBC/mybatisplus-demo/src/test/java/com/taosdata/example/mybatisplusdemo/mapper/WeatherMapperTest.java diff --git a/examples/JDBC/readme.md b/docs/examples/JDBC/readme.md similarity index 100% rename from examples/JDBC/readme.md rename to docs/examples/JDBC/readme.md diff --git a/examples/JDBC/springbootdemo/.gitignore b/docs/examples/JDBC/springbootdemo/.gitignore similarity index 100% rename from examples/JDBC/springbootdemo/.gitignore rename to docs/examples/JDBC/springbootdemo/.gitignore diff --git a/examples/JDBC/springbootdemo/mvnw b/docs/examples/JDBC/springbootdemo/mvnw similarity index 100% rename from examples/JDBC/springbootdemo/mvnw rename to docs/examples/JDBC/springbootdemo/mvnw diff --git a/examples/JDBC/springbootdemo/mvnw.cmd b/docs/examples/JDBC/springbootdemo/mvnw.cmd similarity index 100% rename from examples/JDBC/springbootdemo/mvnw.cmd rename to docs/examples/JDBC/springbootdemo/mvnw.cmd diff --git a/examples/JDBC/springbootdemo/pom.xml b/docs/examples/JDBC/springbootdemo/pom.xml similarity index 100% rename from examples/JDBC/springbootdemo/pom.xml rename to docs/examples/JDBC/springbootdemo/pom.xml diff --git a/examples/JDBC/springbootdemo/readme.md b/docs/examples/JDBC/springbootdemo/readme.md similarity index 100% rename from examples/JDBC/springbootdemo/readme.md rename to docs/examples/JDBC/springbootdemo/readme.md diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/SpringbootdemoApplication.java b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/SpringbootdemoApplication.java similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/SpringbootdemoApplication.java rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/SpringbootdemoApplication.java diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/controller/WeatherController.java b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/controller/WeatherController.java similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/controller/WeatherController.java rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/controller/WeatherController.java diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.java b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.java similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.java rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.java diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.xml b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.xml similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.xml rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/dao/WeatherMapper.xml diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/domain/Weather.java b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/domain/Weather.java similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/domain/Weather.java rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/domain/Weather.java diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/service/WeatherService.java b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/service/WeatherService.java similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/service/WeatherService.java rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/service/WeatherService.java diff --git a/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/util/TaosAspect.java b/docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/util/TaosAspect.java similarity index 100% rename from examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/util/TaosAspect.java rename to docs/examples/JDBC/springbootdemo/src/main/java/com/taosdata/example/springbootdemo/util/TaosAspect.java diff --git a/examples/JDBC/springbootdemo/src/main/resources/application.properties b/docs/examples/JDBC/springbootdemo/src/main/resources/application.properties similarity index 100% rename from examples/JDBC/springbootdemo/src/main/resources/application.properties rename to docs/examples/JDBC/springbootdemo/src/main/resources/application.properties diff --git a/examples/JDBC/taosdemo/.gitignore b/docs/examples/JDBC/taosdemo/.gitignore similarity index 100% rename from examples/JDBC/taosdemo/.gitignore rename to docs/examples/JDBC/taosdemo/.gitignore diff --git a/examples/JDBC/taosdemo/.mvn/wrapper/MavenWrapperDownloader.java b/docs/examples/JDBC/taosdemo/.mvn/wrapper/MavenWrapperDownloader.java similarity index 100% rename from examples/JDBC/taosdemo/.mvn/wrapper/MavenWrapperDownloader.java rename to docs/examples/JDBC/taosdemo/.mvn/wrapper/MavenWrapperDownloader.java diff --git a/examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.jar b/docs/examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.jar similarity index 100% rename from examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.jar rename to docs/examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.jar diff --git a/examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.properties b/docs/examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.properties similarity index 100% rename from examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.properties rename to docs/examples/JDBC/taosdemo/.mvn/wrapper/maven-wrapper.properties diff --git a/examples/JDBC/taosdemo/mvnw b/docs/examples/JDBC/taosdemo/mvnw similarity index 100% rename from examples/JDBC/taosdemo/mvnw rename to docs/examples/JDBC/taosdemo/mvnw diff --git a/examples/JDBC/taosdemo/mvnw.cmd b/docs/examples/JDBC/taosdemo/mvnw.cmd similarity index 100% rename from examples/JDBC/taosdemo/mvnw.cmd rename to docs/examples/JDBC/taosdemo/mvnw.cmd diff --git a/examples/JDBC/taosdemo/pom.xml b/docs/examples/JDBC/taosdemo/pom.xml similarity index 100% rename from examples/JDBC/taosdemo/pom.xml rename to docs/examples/JDBC/taosdemo/pom.xml diff --git a/examples/JDBC/taosdemo/readme.md b/docs/examples/JDBC/taosdemo/readme.md similarity index 100% rename from examples/JDBC/taosdemo/readme.md rename to docs/examples/JDBC/taosdemo/readme.md diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/TaosDemoApplication.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/TaosDemoApplication.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/TaosDemoApplication.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/TaosDemoApplication.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/DataSourceFactory.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/DataSourceFactory.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/DataSourceFactory.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/DataSourceFactory.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JdbcTaosdemoConfig.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JdbcTaosdemoConfig.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JdbcTaosdemoConfig.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JdbcTaosdemoConfig.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JsonConfig.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JsonConfig.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JsonConfig.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/components/JsonConfig.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapper.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapper.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapper.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapper.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapperImpl.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapperImpl.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapperImpl.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/DatabaseMapperImpl.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapper.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapper.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapper.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapper.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapperImpl.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapperImpl.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapperImpl.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SubTableMapperImpl.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapper.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapper.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapper.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapper.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapperImpl.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapperImpl.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapperImpl.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/SuperTableMapperImpl.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapper.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapper.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapper.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapper.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapperImpl.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapperImpl.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapperImpl.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/dao/TableMapperImpl.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldMeta.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldMeta.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldMeta.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldMeta.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldValue.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldValue.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldValue.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/FieldValue.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/RowValue.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/RowValue.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/RowValue.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/RowValue.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableMeta.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableMeta.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableMeta.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableMeta.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableValue.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableValue.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableValue.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SubTableValue.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SuperTableMeta.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SuperTableMeta.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SuperTableMeta.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/SuperTableMeta.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableMeta.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableMeta.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableMeta.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableMeta.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableValue.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableValue.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableValue.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TableValue.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagMeta.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagMeta.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagMeta.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagMeta.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagValue.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagValue.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagValue.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/domain/TagValue.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/AbstractService.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/AbstractService.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/AbstractService.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/AbstractService.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/DatabaseService.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/DatabaseService.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/DatabaseService.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/DatabaseService.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/QueryService.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/QueryService.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/QueryService.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/QueryService.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SqlExecuteTask.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SqlExecuteTask.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SqlExecuteTask.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SqlExecuteTask.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SubTableService.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SubTableService.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SubTableService.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SubTableService.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SuperTableService.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SuperTableService.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SuperTableService.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/SuperTableService.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/TableService.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/TableService.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/TableService.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/TableService.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/FieldValueGenerator.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/FieldValueGenerator.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/FieldValueGenerator.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/FieldValueGenerator.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableMetaGenerator.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableMetaGenerator.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableMetaGenerator.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableMetaGenerator.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableValueGenerator.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableValueGenerator.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableValueGenerator.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SubTableValueGenerator.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SuperTableMetaGenerator.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SuperTableMetaGenerator.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SuperTableMetaGenerator.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/SuperTableMetaGenerator.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/TagValueGenerator.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/TagValueGenerator.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/TagValueGenerator.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/service/data/TagValueGenerator.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/DataGenerator.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/DataGenerator.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/DataGenerator.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/DataGenerator.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/Printer.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/Printer.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/Printer.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/Printer.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/SqlSpeller.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/SqlSpeller.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/SqlSpeller.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/SqlSpeller.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TaosConstants.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TaosConstants.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TaosConstants.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TaosConstants.java diff --git a/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TimeStampUtil.java b/docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TimeStampUtil.java similarity index 100% rename from examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TimeStampUtil.java rename to docs/examples/JDBC/taosdemo/src/main/java/com/taosdata/taosdemo/utils/TimeStampUtil.java diff --git a/examples/JDBC/taosdemo/src/main/resources/application.properties b/docs/examples/JDBC/taosdemo/src/main/resources/application.properties similarity index 100% rename from examples/JDBC/taosdemo/src/main/resources/application.properties rename to docs/examples/JDBC/taosdemo/src/main/resources/application.properties diff --git a/examples/JDBC/taosdemo/src/main/resources/insert.json b/docs/examples/JDBC/taosdemo/src/main/resources/insert.json similarity index 100% rename from examples/JDBC/taosdemo/src/main/resources/insert.json rename to docs/examples/JDBC/taosdemo/src/main/resources/insert.json diff --git a/examples/JDBC/taosdemo/src/main/resources/log4j.properties b/docs/examples/JDBC/taosdemo/src/main/resources/log4j.properties similarity index 100% rename from examples/JDBC/taosdemo/src/main/resources/log4j.properties rename to docs/examples/JDBC/taosdemo/src/main/resources/log4j.properties diff --git a/examples/JDBC/taosdemo/src/main/resources/query.json b/docs/examples/JDBC/taosdemo/src/main/resources/query.json similarity index 100% rename from examples/JDBC/taosdemo/src/main/resources/query.json rename to docs/examples/JDBC/taosdemo/src/main/resources/query.json diff --git a/examples/JDBC/taosdemo/src/main/resources/templates/index.html b/docs/examples/JDBC/taosdemo/src/main/resources/templates/index.html similarity index 100% rename from examples/JDBC/taosdemo/src/main/resources/templates/index.html rename to docs/examples/JDBC/taosdemo/src/main/resources/templates/index.html diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/DatabaseServiceTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/DatabaseServiceTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/DatabaseServiceTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/DatabaseServiceTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/QueryServiceTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/QueryServiceTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/QueryServiceTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/QueryServiceTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SubTableServiceTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SubTableServiceTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SubTableServiceTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SubTableServiceTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SuperTableServiceTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SuperTableServiceTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SuperTableServiceTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/SuperTableServiceTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/FieldValueGeneratorTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/FieldValueGeneratorTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/FieldValueGeneratorTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/FieldValueGeneratorTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SubTableMetaGeneratorTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SubTableMetaGeneratorTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SubTableMetaGeneratorTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SubTableMetaGeneratorTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SuperTableMetaGeneratorImplTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SuperTableMetaGeneratorImplTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SuperTableMetaGeneratorImplTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/SuperTableMetaGeneratorImplTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/TagValueGeneratorTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/TagValueGeneratorTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/TagValueGeneratorTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/service/data/TagValueGeneratorTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/DataGeneratorTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/DataGeneratorTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/DataGeneratorTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/DataGeneratorTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/SqlSpellerTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/SqlSpellerTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/SqlSpellerTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/SqlSpellerTest.java diff --git a/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/TimeStampUtilTest.java b/docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/TimeStampUtilTest.java similarity index 100% rename from examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/TimeStampUtilTest.java rename to docs/examples/JDBC/taosdemo/src/test/java/com/taosdata/taosdemo/utils/TimeStampUtilTest.java diff --git a/docs/examples/c/asyncdemo.c b/docs/examples/c/asyncdemo.c new file mode 100644 index 0000000000..91ec6f24b1 --- /dev/null +++ b/docs/examples/c/asyncdemo.c @@ -0,0 +1,293 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +// TAOS asynchronous API example +// this example opens multiple tables, insert/retrieve multiple tables +// it is used by TAOS internally for one performance testing +// to compiple: gcc -o asyncdemo asyncdemo.c -ltaos + +#include +#include +#include +#include +#include +#include +#include "taos.h" + +int points = 5; +int numOfTables = 3; +int tablesInsertProcessed = 0; +int tablesSelectProcessed = 0; +int64_t st, et; + +typedef struct { + int id; + TAOS *taos; + char name[32]; + time_t timeStamp; + int value; + int rowsInserted; + int rowsTried; + int rowsRetrieved; +} STable; + +void taos_insert_call_back(void *param, TAOS_RES *tres, int code); +void taos_select_call_back(void *param, TAOS_RES *tres, int code); +void shellPrintError(TAOS *taos); + +static void queryDB(TAOS *taos, char *command) { + int i; + TAOS_RES *pSql = NULL; + int32_t code = -1; + + for (i = 0; i < 5; i++) { + if (NULL != pSql) { + taos_free_result(pSql); + pSql = NULL; + } + + pSql = taos_query(taos, command); + code = taos_errno(pSql); + if (0 == code) { + break; + } + } + + if (code != 0) { + fprintf(stderr, "Failed to run %s, reason: %s\n", command, taos_errstr(pSql)); + taos_free_result(pSql); + taos_close(taos); + taos_cleanup(); + exit(EXIT_FAILURE); + } + + taos_free_result(pSql); +} + +int main(int argc, char *argv[]) +{ + TAOS *taos; + struct timeval systemTime; + int i; + char sql[1024] = { 0 }; + char prefix[20] = { 0 }; + char db[128] = { 0 }; + STable *tableList; + + if (argc != 5) { + printf("usage: %s server-ip dbname rowsPerTable numOfTables\n", argv[0]); + exit(0); + } + + // a simple way to parse input parameters + if (argc >= 3) strncpy(db, argv[2], sizeof(db) - 1); + if (argc >= 4) points = atoi(argv[3]); + if (argc >= 5) numOfTables = atoi(argv[4]); + + size_t size = sizeof(STable) * (size_t)numOfTables; + tableList = (STable *)malloc(size); + memset(tableList, 0, size); + + taos = taos_connect(argv[1], "root", "taosdata", NULL, 0); + if (taos == NULL) + shellPrintError(taos); + + printf("success to connect to server\n"); + + sprintf(sql, "drop database if exists %s", db); + queryDB(taos, sql); + + sprintf(sql, "create database %s", db); + queryDB(taos, sql); + + sprintf(sql, "use %s", db); + queryDB(taos, sql); + + strcpy(prefix, "asytbl_"); + for (i = 0; i < numOfTables; ++i) { + tableList[i].id = i; + tableList[i].taos = taos; + sprintf(tableList[i].name, "%s%d", prefix, i); + sprintf(sql, "create table %s%d (ts timestamp, volume bigint)", prefix, i); + queryDB(taos, sql); + } + + gettimeofday(&systemTime, NULL); + for (i = 0; i < numOfTables; ++i) + tableList[i].timeStamp = (time_t)(systemTime.tv_sec) * 1000 + systemTime.tv_usec / 1000; + + printf("success to create tables, press any key to insert\n"); + getchar(); + + printf("start to insert...\n"); + gettimeofday(&systemTime, NULL); + st = systemTime.tv_sec * 1000000 + systemTime.tv_usec; + + tablesInsertProcessed = 0; + tablesSelectProcessed = 0; + + for (i = 0; irowsTried++; + + if (code < 0) { + printf("%s insert failed, code:%d, rows:%d\n", pTable->name, code, pTable->rowsTried); + } + else if (code == 0) { + printf("%s not inserted\n", pTable->name); + } + else { + pTable->rowsInserted++; + } + + if (pTable->rowsTried < points) { + // for this demo, insert another record + sprintf(sql, "insert into %s values(%ld, %d)", pTable->name, 1546300800000+pTable->rowsTried*1000, pTable->rowsTried); + taos_query_a(pTable->taos, sql, taos_insert_call_back, (void *)pTable); + } + else { + printf("%d rows data are inserted into %s\n", points, pTable->name); + tablesInsertProcessed++; + if (tablesInsertProcessed >= numOfTables) { + gettimeofday(&systemTime, NULL); + et = systemTime.tv_sec * 1000000 + systemTime.tv_usec; + printf("%" PRId64 " mseconds to insert %d data points\n", (et - st) / 1000, points*numOfTables); + } + } + + taos_free_result(tres); +} + +void taos_retrieve_call_back(void *param, TAOS_RES *tres, int numOfRows) +{ + STable *pTable = (STable *)param; + struct timeval systemTime; + + if (numOfRows > 0) { + + for (int i = 0; irowsRetrieved += numOfRows; + + // retrieve next batch of rows + taos_fetch_rows_a(tres, taos_retrieve_call_back, pTable); + + } + else { + if (numOfRows < 0) + printf("%s retrieve failed, code:%d\n", pTable->name, numOfRows); + + //taos_free_result(tres); + printf("%d rows data retrieved from %s\n", pTable->rowsRetrieved, pTable->name); + + tablesSelectProcessed++; + if (tablesSelectProcessed >= numOfTables) { + gettimeofday(&systemTime, NULL); + et = systemTime.tv_sec * 1000000 + systemTime.tv_usec; + printf("%" PRId64 " mseconds to query %d data rows\n", (et - st) / 1000, points * numOfTables); + } + + taos_free_result(tres); + } + + +} + +void taos_select_call_back(void *param, TAOS_RES *tres, int code) +{ + STable *pTable = (STable *)param; + + if (code == 0 && tres) { + // asynchronous API to fetch a batch of records + taos_fetch_rows_a(tres, taos_retrieve_call_back, pTable); + } + else { + printf("%s select failed, code:%d\n", pTable->name, code); + taos_free_result(tres); + taos_cleanup(); + exit(1); + } +} diff --git a/docs/examples/c/demo.c b/docs/examples/c/demo.c new file mode 100644 index 0000000000..0351aecbd1 --- /dev/null +++ b/docs/examples/c/demo.c @@ -0,0 +1,133 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +// TAOS standard API example. The same syntax as MySQL, but only a subset +// to compile: gcc -o demo demo.c -ltaos + +#include +#include +#include +#include +#include "taos.h" // TAOS header file + +static void queryDB(TAOS *taos, char *command) { + int i; + TAOS_RES *pSql = NULL; + int32_t code = -1; + + for (i = 0; i < 5; i++) { + if (NULL != pSql) { + taos_free_result(pSql); + pSql = NULL; + } + + pSql = taos_query(taos, command); + code = taos_errno(pSql); + if (0 == code) { + break; + } + } + + if (code != 0) { + fprintf(stderr, "Failed to run %s, reason: %s\n", command, taos_errstr(pSql)); + taos_free_result(pSql); + taos_close(taos); + exit(EXIT_FAILURE); + } + + taos_free_result(pSql); +} + +void Test(TAOS *taos, char *qstr, int i); + +int main(int argc, char *argv[]) { + char qstr[1024]; + + // connect to server + if (argc < 2) { + printf("please input server-ip \n"); + return 0; + } + + TAOS *taos = taos_connect(argv[1], "root", "taosdata", NULL, 0); + if (taos == NULL) { + printf("failed to connect to server, reason:%s\n", taos_errstr(NULL)); + exit(1); + } + for (int i = 0; i < 100; i++) { + Test(taos, qstr, i); + } + taos_close(taos); + taos_cleanup(); +} +void Test(TAOS *taos, char *qstr, int index) { + printf("==================test at %d\n================================", index); + queryDB(taos, "drop database if exists demo"); + queryDB(taos, "create database demo"); + TAOS_RES *result; + queryDB(taos, "use demo"); + + queryDB(taos, "create table m1 (ts timestamp, ti tinyint, si smallint, i int, bi bigint, f float, d double, b binary(10))"); + printf("success to create table\n"); + + int i = 0; + for (i = 0; i < 10; ++i) { + sprintf(qstr, "insert into m1 values (%" PRId64 ", %d, %d, %d, %d, %f, %lf, '%s')", (uint64_t)(1546300800000 + i * 1000), i, i, i, i*10000000, i*1.0, i*2.0, "hello"); + printf("qstr: %s\n", qstr); + + // note: how do you wanna do if taos_query returns non-NULL + // if (taos_query(taos, qstr)) { + // printf("insert row: %i, reason:%s\n", i, taos_errstr(taos)); + // } + TAOS_RES *result1 = taos_query(taos, qstr); + if (result1 == NULL || taos_errno(result1) != 0) { + printf("failed to insert row, reason:%s\n", taos_errstr(result1)); + taos_free_result(result1); + exit(1); + } else { + printf("insert row: %i\n", i); + } + taos_free_result(result1); + } + printf("success to insert rows, total %d rows\n", i); + + // query the records + sprintf(qstr, "SELECT * FROM m1"); + result = taos_query(taos, qstr); + if (result == NULL || taos_errno(result) != 0) { + printf("failed to select, reason:%s\n", taos_errstr(result)); + taos_free_result(result); + exit(1); + } + + TAOS_ROW row; + int rows = 0; + int num_fields = taos_field_count(result); + TAOS_FIELD *fields = taos_fetch_fields(result); + + printf("num_fields = %d\n", num_fields); + printf("select * from table, result:\n"); + // fetch the records row by row + while ((row = taos_fetch_row(result))) { + char temp[1024] = {0}; + rows++; + taos_print_row(temp, row, fields, num_fields); + printf("%s\n", temp); + } + + taos_free_result(result); + printf("====demo end====\n\n"); +} + diff --git a/docs/examples/c/prepare.c b/docs/examples/c/prepare.c new file mode 100644 index 0000000000..aee8400663 --- /dev/null +++ b/docs/examples/c/prepare.c @@ -0,0 +1,235 @@ +// TAOS standard API example. The same syntax as MySQL, but only a subet +// to compile: gcc -o prepare prepare.c -ltaos + +#include +#include +#include +#include "taos.h" + +void taosMsleep(int mseconds); + +int main(int argc, char *argv[]) +{ + TAOS *taos; + TAOS_RES *result; + int code; + TAOS_STMT *stmt; + + // connect to server + if (argc < 2) { + printf("please input server ip \n"); + return 0; + } + + taos = taos_connect(argv[1], "root", "taosdata", NULL, 0); + if (taos == NULL) { + printf("failed to connect to db, reason:%s\n", taos_errstr(taos)); + exit(1); + } + + result = taos_query(taos, "drop database demo"); + taos_free_result(result); + + result = taos_query(taos, "create database demo"); + code = taos_errno(result); + if (code != 0) { + printf("failed to create database, reason:%s\n", taos_errstr(result)); + taos_free_result(result); + exit(1); + } + taos_free_result(result); + + result = taos_query(taos, "use demo"); + taos_free_result(result); + + // create table + const char* sql = "create table m1 (ts timestamp, b bool, v1 tinyint, v2 smallint, v4 int, v8 bigint, f4 float, f8 double, bin binary(40), blob nchar(10), varbin varbinary(16))"; + result = taos_query(taos, sql); + code = taos_errno(result); + if (code != 0) { + printf("failed to create table, reason:%s\n", taos_errstr(result)); + taos_free_result(result); + exit(1); + } + taos_free_result(result); + + // sleep for one second to make sure table is created on data node + // taosMsleep(1000); + + // insert 10 records + struct { + int64_t ts; + int8_t b; + int8_t v1; + int16_t v2; + int32_t v4; + int64_t v8; + float f4; + double f8; + char bin[40]; + char blob[80]; + int8_t varbin[16]; + } v = {0}; + + int32_t boolLen = sizeof(int8_t); + int32_t sintLen = sizeof(int16_t); + int32_t intLen = sizeof(int32_t); + int32_t bintLen = sizeof(int64_t); + int32_t floatLen = sizeof(float); + int32_t doubleLen = sizeof(double); + int32_t binLen = sizeof(v.bin); + int32_t ncharLen = 30; + + stmt = taos_stmt_init(taos); + TAOS_MULTI_BIND params[11]; + params[0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP; + params[0].buffer_length = sizeof(v.ts); + params[0].buffer = &v.ts; + params[0].length = &bintLen; + params[0].is_null = NULL; + params[0].num = 1; + + params[1].buffer_type = TSDB_DATA_TYPE_BOOL; + params[1].buffer_length = sizeof(v.b); + params[1].buffer = &v.b; + params[1].length = &boolLen; + params[1].is_null = NULL; + params[1].num = 1; + + params[2].buffer_type = TSDB_DATA_TYPE_TINYINT; + params[2].buffer_length = sizeof(v.v1); + params[2].buffer = &v.v1; + params[2].length = &boolLen; + params[2].is_null = NULL; + params[2].num = 1; + + params[3].buffer_type = TSDB_DATA_TYPE_SMALLINT; + params[3].buffer_length = sizeof(v.v2); + params[3].buffer = &v.v2; + params[3].length = &sintLen; + params[3].is_null = NULL; + params[3].num = 1; + + params[4].buffer_type = TSDB_DATA_TYPE_INT; + params[4].buffer_length = sizeof(v.v4); + params[4].buffer = &v.v4; + params[4].length = &intLen; + params[4].is_null = NULL; + params[4].num = 1; + + params[5].buffer_type = TSDB_DATA_TYPE_BIGINT; + params[5].buffer_length = sizeof(v.v8); + params[5].buffer = &v.v8; + params[5].length = &bintLen; + params[5].is_null = NULL; + params[5].num = 1; + + params[6].buffer_type = TSDB_DATA_TYPE_FLOAT; + params[6].buffer_length = sizeof(v.f4); + params[6].buffer = &v.f4; + params[6].length = &floatLen; + params[6].is_null = NULL; + params[6].num = 1; + + params[7].buffer_type = TSDB_DATA_TYPE_DOUBLE; + params[7].buffer_length = sizeof(v.f8); + params[7].buffer = &v.f8; + params[7].length = &doubleLen; + params[7].is_null = NULL; + params[7].num = 1; + + params[8].buffer_type = TSDB_DATA_TYPE_BINARY; + params[8].buffer_length = sizeof(v.bin); + params[8].buffer = v.bin; + params[8].length = &binLen; + params[8].is_null = NULL; + params[8].num = 1; + + strcpy(v.blob, "一二三四五六七八九十"); + params[9].buffer_type = TSDB_DATA_TYPE_NCHAR; + params[9].buffer_length = sizeof(v.blob); + params[9].buffer = v.blob; + params[9].length = &ncharLen; + params[9].is_null = NULL; + params[9].num = 1; + + int8_t tmp[16] = {'a', 0, 1, 13, '1'}; + int32_t vbinLen = 5; + memcpy(v.varbin, tmp, sizeof(v.varbin)); + params[10].buffer_type = TSDB_DATA_TYPE_VARBINARY; + params[10].buffer_length = sizeof(v.varbin); + params[10].buffer = v.varbin; + params[10].length = &vbinLen; + params[10].is_null = NULL; + params[10].num = 1; + + char is_null = 1; + + sql = "insert into m1 values(?,?,?,?,?,?,?,?,?,?,?)"; + code = taos_stmt_prepare(stmt, sql, 0); + if (code != 0){ + printf("failed to execute taos_stmt_prepare. code:0x%x\n", code); + } + v.ts = 1591060628000; + for (int i = 0; i < 10; ++i) { + v.ts += 1; + for (int j = 1; j < 11; ++j) { + params[j].is_null = ((i == j) ? &is_null : 0); + } + v.b = (int8_t)i % 2; + v.v1 = (int8_t)i; + v.v2 = (int16_t)(i * 2); + v.v4 = (int32_t)(i * 4); + v.v8 = (int64_t)(i * 8); + v.f4 = (float)(i * 40); + v.f8 = (double)(i * 80); + for (int j = 0; j < sizeof(v.bin); ++j) { + v.bin[j] = (char)(i + '0'); + } + + taos_stmt_bind_param(stmt, params); + taos_stmt_add_batch(stmt); + } + if (taos_stmt_execute(stmt) != 0) { + printf("failed to execute insert statement.\n"); + exit(1); + } + taos_stmt_close(stmt); + + // query the records + stmt = taos_stmt_init(taos); + taos_stmt_prepare(stmt, "SELECT * FROM m1 WHERE v1 > ? AND v2 < ?", 0); + v.v1 = 5; + v.v2 = 15; + taos_stmt_bind_param(stmt, params + 2); + if (taos_stmt_execute(stmt) != 0) { + printf("failed to execute select statement.\n"); + exit(1); + } + + result = taos_stmt_use_result(stmt); + + TAOS_ROW row; + int rows = 0; + int num_fields = taos_num_fields(result); + TAOS_FIELD *fields = taos_fetch_fields(result); + + // fetch the records row by row + while ((row = taos_fetch_row(result))) { + char temp[256] = {0}; + rows++; + taos_print_row(temp, row, fields, num_fields); + printf("%s\n", temp); + } + if (rows == 2) { + printf("two rows are fetched as expectation\n"); + } else { + printf("expect two rows, but %d rows are fetched\n", rows); + } + +// taos_free_result(result); + taos_stmt_close(stmt); + + return 0; +} + diff --git a/docs/examples/c/schemaless.c b/docs/examples/c/schemaless.c new file mode 100644 index 0000000000..328a663ae7 --- /dev/null +++ b/docs/examples/c/schemaless.c @@ -0,0 +1,73 @@ +#include "taos.h" +#include +#include +#include +#include +#include +#include +#include + + +int numSuperTables = 8; +int numChildTables = 4; +int numRowsPerChildTable = 2048; + +static int64_t getTimeInUs() { + struct timeval systemTime; + gettimeofday(&systemTime, NULL); + return (int64_t)systemTime.tv_sec * 1000000L + (int64_t)systemTime.tv_usec; +} + +int main(int argc, char* argv[]) { + TAOS_RES *result; + const char* host = "127.0.0.1"; + const char* user = "root"; + const char* passwd = "taosdata"; + + taos_options(TSDB_OPTION_TIMEZONE, "GMT-8"); + TAOS* taos = taos_connect(host, user, passwd, "", 0); + if (taos == NULL) { + printf("\033[31mfailed to connect to db, reason:%s\033[0m\n", taos_errstr(taos)); + exit(1); + } + + const char* info = taos_get_server_info(taos); + printf("server info: %s\n", info); + info = taos_get_client_info(taos); + printf("client info: %s\n", info); + result = taos_query(taos, "drop database if exists db;"); + taos_free_result(result); + usleep(100000); + result = taos_query(taos, "create database db precision 'ms';"); + taos_free_result(result); + usleep(100000); + + (void)taos_select_db(taos, "db"); + + time_t ct = time(0); + int64_t ts = ct * 1000; + char* lineFormat = "sta%d,t0=true,t1=127i8,t2=32767i16,t3=%di32,t4=9223372036854775807i64,t9=11.12345f32,t10=22.123456789f64,t11=\"binaryTagValue\",t12=L\"ncharTagValue\" c0=true,c1=127i8,c2=32767i16,c3=2147483647i32,c4=9223372036854775807i64,c5=254u8,c6=32770u16,c7=2147483699u32,c8=9223372036854775899u64,c9=11.12345f32,c10=22.123456789f64,c11=\"binaryValue\",c12=L\"ncharValue\" %" PRId64; + + int lineNum = numSuperTables * numChildTables * numRowsPerChildTable; + char** lines = calloc((size_t)lineNum, sizeof(char*)); + int l = 0; + for (int i = 0; i < numSuperTables; ++i) { + for (int j = 0; j < numChildTables; ++j) { + for (int k = 0; k < numRowsPerChildTable; ++k) { + char* line = calloc(512, 1); + snprintf(line, 512, lineFormat, i, j, ts + 10 * l); + lines[l] = line; + ++l; + } + } + } + + printf("%s\n", "begin taos_insert_lines"); + int64_t begin = getTimeInUs(); + TAOS_RES *res = taos_schemaless_insert(taos, lines, lineNum, TSDB_SML_LINE_PROTOCOL, TSDB_SML_TIMESTAMP_MILLI_SECONDS); + int64_t end = getTimeInUs(); + printf("code: %s. time used: %" PRId64 "\n", taos_errstr(res), end-begin); + taos_free_result(res); + + return 0; +} diff --git a/docs/examples/c/stream_demo.c b/docs/examples/c/stream_demo.c new file mode 100644 index 0000000000..46107c7e13 --- /dev/null +++ b/docs/examples/c/stream_demo.c @@ -0,0 +1,120 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +// clang-format off +#include +#include +#include +#include +#include "taos.h" + +int32_t init_env() { + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return -1; + } + + TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 2"); + if (taos_errno(pRes) != 0) { + printf("error in create db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + +#if 0 + pRes = taos_query(pConn, "create database if not exists abc2 vgroups 20"); + if (taos_errno(pRes) != 0) { + printf("error in create db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); +#endif + + pRes = taos_query(pConn, "use abc1"); + if (taos_errno(pRes) != 0) { + printf("error in use db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create stable if not exists st1 (ts timestamp, k int, j varchar(20)) tags(a varchar(20))"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table st1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table if not exists tu1 using st1 tags('c1')"); + if (taos_errno(pRes) != 0) { + printf("failed to create child table tu1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table if not exists tu2 using st1 tags('c2')"); + if (taos_errno(pRes) != 0) { + printf("failed to create child table tu2, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table if not exists tu3 using st1 tags('c3')"); + if (taos_errno(pRes) != 0) { + printf("failed to create child table tu3, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + return 0; +} + +int32_t create_stream() { + printf("create stream\n"); + TAOS_RES* pRes; + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return -1; + } + + pRes = taos_query(pConn, "use abc1"); + if (taos_errno(pRes) != 0) { + printf("error in use db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, + /*"create stream stream1 trigger at_once watermark 10s into outstb as select _wstart start, avg(k) from st1 partition by tbname interval(10s)");*/ + "create stream stream2 into outstb subtable(concat(concat(concat('prefix_', tname), '_suffix_'), cast(k1 as varchar(20)))) as select _wstart wstart, avg(k) from st1 partition by tbname tname, a k1 interval(10s);"); + if (taos_errno(pRes) != 0) { + printf("failed to create stream stream1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + taos_close(pConn); + return 0; +} + +int main(int argc, char* argv[]) { + if (argc > 1) { + printf("env init\n"); + int code = init_env(); + if (code) { + return code; + } + + } + create_stream(); +} diff --git a/docs/examples/c/tmq.c b/docs/examples/c/tmq.c new file mode 100644 index 0000000000..46dd3bb551 --- /dev/null +++ b/docs/examples/c/tmq.c @@ -0,0 +1,333 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +#include +#include +#include +#include +#include +#include "taos.h" + +static int running = 1; +const char* topic_name = "topicname"; + +static int32_t msg_process(TAOS_RES* msg) { + char buf[1024]; + int32_t rows = 0; + + const char* topicName = tmq_get_topic_name(msg); + const char* dbName = tmq_get_db_name(msg); + int32_t vgroupId = tmq_get_vgroup_id(msg); + + printf("topic: %s\n", topicName); + printf("db: %s\n", dbName); + printf("vgroup id: %d\n", vgroupId); + + while (1) { + TAOS_ROW row = taos_fetch_row(msg); + if (row == NULL) break; + + TAOS_FIELD* fields = taos_fetch_fields(msg); + int32_t numOfFields = taos_field_count(msg); + // int32_t* length = taos_fetch_lengths(msg); + int32_t precision = taos_result_precision(msg); + rows++; + taos_print_row(buf, row, fields, numOfFields); + printf("precision: %d, row content: %s\n", precision, buf); + } + + return rows; +} + +static int32_t init_env() { + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return -1; + } + + TAOS_RES* pRes; + // drop database if exists + printf("create database\n"); + pRes = taos_query(pConn, "drop topic topicname"); + if (taos_errno(pRes) != 0) { + printf("error in drop topicname, reason:%s\n", taos_errstr(pRes)); + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "drop database if exists tmqdb"); + if (taos_errno(pRes) != 0) { + printf("error in drop tmqdb, reason:%s\n", taos_errstr(pRes)); + } + taos_free_result(pRes); + + // create database + pRes = taos_query(pConn, "create database tmqdb precision 'ns' WAL_RETENTION_PERIOD 3600"); + if (taos_errno(pRes) != 0) { + printf("error in create tmqdb, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + // create super table + printf("create super table\n"); + pRes = taos_query( + pConn, "create table tmqdb.stb (ts timestamp, c1 int, c2 float, c3 varchar(16)) tags(t1 int, t3 varchar(16))"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table stb, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + // create sub tables + printf("create sub tables\n"); + pRes = taos_query(pConn, "create table tmqdb.ctb0 using tmqdb.stb tags(0, 'subtable0')"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table ctb0, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table tmqdb.ctb1 using tmqdb.stb tags(1, 'subtable1')"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table ctb1, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table tmqdb.ctb2 using tmqdb.stb tags(2, 'subtable2')"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table ctb2, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table tmqdb.ctb3 using tmqdb.stb tags(3, 'subtable3')"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table ctb3, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + // insert data + printf("insert data into sub tables\n"); + pRes = taos_query(pConn, "insert into tmqdb.ctb0 values(now, 0, 0, 'a0')(now+1s, 0, 0, 'a00')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ctb0, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "insert into tmqdb.ctb1 values(now, 1, 1, 'a1')(now+1s, 11, 11, 'a11')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ctb0, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "insert into tmqdb.ctb2 values(now, 2, 2, 'a1')(now+1s, 22, 22, 'a22')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ctb0, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "insert into tmqdb.ctb3 values(now, 3, 3, 'a1')(now+1s, 33, 33, 'a33')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ctb0, reason:%s\n", taos_errstr(pRes)); + goto END; + } + taos_free_result(pRes); + taos_close(pConn); + return 0; + +END: + taos_free_result(pRes); + taos_close(pConn); + return -1; +} + +int32_t create_topic() { + printf("create topic\n"); + TAOS_RES* pRes; + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return -1; + } + + pRes = taos_query(pConn, "use tmqdb"); + if (taos_errno(pRes) != 0) { + printf("error in use tmqdb, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create topic topicname as select ts, c1, c2, c3, tbname from tmqdb.stb where c1 > 1"); + if (taos_errno(pRes) != 0) { + printf("failed to create topic topicname, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + taos_close(pConn); + return 0; +} + +void tmq_commit_cb_print(tmq_t* tmq, int32_t code, void* param) { + printf("tmq_commit_cb_print() code: %d, tmq: %p, param: %p\n", code, tmq, param); +} + +tmq_t* build_consumer() { + tmq_conf_res_t code; + tmq_t* tmq = NULL; + + tmq_conf_t* conf = tmq_conf_new(); + code = tmq_conf_set(conf, "enable.auto.commit", "true"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + code = tmq_conf_set(conf, "auto.commit.interval.ms", "1000"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + code = tmq_conf_set(conf, "group.id", "cgrpName"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + code = tmq_conf_set(conf, "client.id", "user defined name"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + code = tmq_conf_set(conf, "td.connect.user", "root"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + code = tmq_conf_set(conf, "td.connect.pass", "taosdata"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + code = tmq_conf_set(conf, "auto.offset.reset", "earliest"); + if (TMQ_CONF_OK != code) { + tmq_conf_destroy(conf); + return NULL; + } + + tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, NULL); + tmq = tmq_consumer_new(conf, NULL, 0); + +_end: + tmq_conf_destroy(conf); + return tmq; +} + +tmq_list_t* build_topic_list() { + tmq_list_t* topicList = tmq_list_new(); + int32_t code = tmq_list_append(topicList, topic_name); + if (code) { + tmq_list_destroy(topicList); + return NULL; + } + return topicList; +} + +void basic_consume_loop(tmq_t* tmq) { + int32_t totalRows = 0; + int32_t msgCnt = 0; + int32_t timeout = 5000; + while (running) { + TAOS_RES* tmqmsg = tmq_consumer_poll(tmq, timeout); + if (tmqmsg) { + msgCnt++; + totalRows += msg_process(tmqmsg); + taos_free_result(tmqmsg); + } else { + break; + } + } + + fprintf(stderr, "%d msg consumed, include %d rows\n", msgCnt, totalRows); +} + +void consume_repeatly(tmq_t* tmq) { + int32_t numOfAssignment = 0; + tmq_topic_assignment* pAssign = NULL; + + int32_t code = tmq_get_topic_assignment(tmq, topic_name, &pAssign, &numOfAssignment); + if (code != 0) { + fprintf(stderr, "failed to get assignment, reason:%s", tmq_err2str(code)); + } + + // seek to the earliest offset + for(int32_t i = 0; i < numOfAssignment; ++i) { + tmq_topic_assignment* p = &pAssign[i]; + + code = tmq_offset_seek(tmq, topic_name, p->vgId, p->begin); + if (code != 0) { + fprintf(stderr, "failed to seek to %d, reason:%s", (int)p->begin, tmq_err2str(code)); + } + } + + tmq_free_assignment(pAssign); + + // let's do it again + basic_consume_loop(tmq); +} + +int main(int argc, char* argv[]) { + int32_t code; + + if (init_env() < 0) { + return -1; + } + + if (create_topic() < 0) { + return -1; + } + + tmq_t* tmq = build_consumer(); + if (NULL == tmq) { + fprintf(stderr, "build_consumer() fail!\n"); + return -1; + } + + tmq_list_t* topic_list = build_topic_list(); + if (NULL == topic_list) { + return -1; + } + + if ((code = tmq_subscribe(tmq, topic_list))) { + fprintf(stderr, "Failed to tmq_subscribe(): %s\n", tmq_err2str(code)); + } + + tmq_list_destroy(topic_list); + + basic_consume_loop(tmq); + + consume_repeatly(tmq); + + code = tmq_consumer_close(tmq); + if (code) { + fprintf(stderr, "Failed to close consumer: %s\n", tmq_err2str(code)); + } else { + fprintf(stderr, "Consumer closed\n"); + } + + return 0; +} diff --git a/docs/examples/c/with_reqid_demo.c b/docs/examples/c/with_reqid_demo.c index 1a1a53acc6..0cfedf3b3d 100644 --- a/docs/examples/c/with_reqid_demo.c +++ b/docs/examples/c/with_reqid_demo.c @@ -33,7 +33,8 @@ static int DemoWithReqId() { // connect TAOS *taos = taos_connect(host, user, password, NULL, port); if (taos == NULL) { - fprintf(stderr, "Failed to connect to %s:%hu, ErrCode: 0x%x, ErrMessage: %s.\n", host, port, taos_errno(NULL), taos_errstr(NULL)); + fprintf(stderr, "Failed to connect to %s:%hu, ErrCode: 0x%x, ErrMessage: %s.\n", host, port, taos_errno(NULL), + taos_errstr(NULL)); taos_cleanup(); return -1; } @@ -44,7 +45,8 @@ static int DemoWithReqId() { TAOS_RES *result = taos_query_with_reqid(taos, sql, reqid); code = taos_errno(result); if (code != 0) { - fprintf(stderr, "Failed to execute sql with reqId: %ld, ErrCode: 0x%x, ErrMessage: %s\n.", reqid, code, taos_errstr(result)); + fprintf(stderr, "Failed to execute sql with qid: %ld, ErrCode: 0x%x, ErrMessage: %s\n.", reqid, code, + taos_errstr(result)); taos_close(taos); taos_cleanup(); return -1; @@ -73,6 +75,4 @@ static int DemoWithReqId() { // ANCHOR_END: with_reqid } -int main(int argc, char *argv[]) { - return DemoWithReqId(); -} +int main(int argc, char *argv[]) { return DemoWithReqId(); } diff --git a/docs/examples/rust/nativeexample/examples/bind-tags.rs b/docs/examples/rust/nativeexample/examples/bind-tags.rs new file mode 100644 index 0000000000..a1f7286625 --- /dev/null +++ b/docs/examples/rust/nativeexample/examples/bind-tags.rs @@ -0,0 +1,80 @@ +use anyhow::Result; +use serde::Deserialize; +use taos::*; + +#[tokio::main] +async fn main() -> Result<()> { + let taos = TaosBuilder::from_dsn("taos://")?.build()?; + taos.exec_many([ + "drop database if exists test", + "create database test keep 36500", + "use test", + "create table tb1 (ts timestamp, c1 bool, c2 tinyint, c3 smallint, c4 int, c5 bigint, + c6 tinyint unsigned, c7 smallint unsigned, c8 int unsigned, c9 bigint unsigned, + c10 float, c11 double, c12 varchar(100), c13 nchar(100)) tags(t1 varchar(100))", + ]) + .await?; + let mut stmt = Stmt::init(&taos)?; + stmt.prepare( + "insert into ? using tb1 tags(?) values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)", + )?; + stmt.set_tbname("d0")?; + stmt.set_tags(&[Value::VarChar("涛思".to_string())])?; + + let params = vec![ + ColumnView::from_millis_timestamp(vec![164000000000]), + ColumnView::from_bools(vec![true]), + ColumnView::from_tiny_ints(vec![i8::MAX]), + ColumnView::from_small_ints(vec![i16::MAX]), + ColumnView::from_ints(vec![i32::MAX]), + ColumnView::from_big_ints(vec![i64::MAX]), + ColumnView::from_unsigned_tiny_ints(vec![u8::MAX]), + ColumnView::from_unsigned_small_ints(vec![u16::MAX]), + ColumnView::from_unsigned_ints(vec![u32::MAX]), + ColumnView::from_unsigned_big_ints(vec![u64::MAX]), + ColumnView::from_floats(vec![f32::MAX]), + ColumnView::from_doubles(vec![f64::MAX]), + ColumnView::from_varchar(vec!["ABC"]), + ColumnView::from_nchar(vec!["涛思数据"]), + ]; + let rows = stmt.bind(¶ms)?.add_batch()?.execute()?; + assert_eq!(rows, 1); + + #[derive(Debug, Deserialize)] + #[allow(dead_code)] + struct Row { + ts: String, + c1: bool, + c2: i8, + c3: i16, + c4: i32, + c5: i64, + c6: u8, + c7: u16, + c8: u32, + c9: u64, + c10: Option, + c11: f64, + c12: String, + c13: String, + t1: serde_json::Value, + } + + let rows: Vec = taos + .query("select * from tb1") + .await? + .deserialize() + .try_collect() + .await?; + let row = &rows[0]; + dbg!(&row); + assert_eq!(row.c5, i64::MAX); + assert_eq!(row.c8, u32::MAX); + assert_eq!(row.c9, u64::MAX); + assert_eq!(row.c10.unwrap(), f32::MAX); + // assert_eq!(row.c11, f64::MAX); + assert_eq!(row.c12, "ABC"); + assert_eq!(row.c13, "涛思数据"); + + Ok(()) +} diff --git a/docs/examples/rust/nativeexample/examples/bind.rs b/docs/examples/rust/nativeexample/examples/bind.rs new file mode 100644 index 0000000000..194938a319 --- /dev/null +++ b/docs/examples/rust/nativeexample/examples/bind.rs @@ -0,0 +1,74 @@ +use anyhow::Result; +use serde::Deserialize; +use taos::*; + +#[tokio::main] +async fn main() -> Result<()> { + let taos = TaosBuilder::from_dsn("taos://")?.build()?; + taos.exec_many([ + "drop database if exists test_bindable", + "create database test_bindable keep 36500", + "use test_bindable", + "create table tb1 (ts timestamp, c1 bool, c2 tinyint, c3 smallint, c4 int, c5 bigint, + c6 tinyint unsigned, c7 smallint unsigned, c8 int unsigned, c9 bigint unsigned, + c10 float, c11 double, c12 varchar(100), c13 nchar(100))", + ]) + .await?; + let mut stmt = Stmt::init(&taos)?; + stmt.prepare("insert into tb1 values(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)")?; + let params = vec![ + ColumnView::from_millis_timestamp(vec![0]), + ColumnView::from_bools(vec![true]), + ColumnView::from_tiny_ints(vec![i8::MAX]), + ColumnView::from_small_ints(vec![i16::MAX]), + ColumnView::from_ints(vec![i32::MAX]), + ColumnView::from_big_ints(vec![i64::MAX]), + ColumnView::from_unsigned_tiny_ints(vec![u8::MAX]), + ColumnView::from_unsigned_small_ints(vec![u16::MAX]), + ColumnView::from_unsigned_ints(vec![u32::MAX]), + ColumnView::from_unsigned_big_ints(vec![u64::MAX]), + ColumnView::from_floats(vec![f32::MAX]), + ColumnView::from_doubles(vec![f64::MAX]), + ColumnView::from_varchar(vec!["ABC"]), + ColumnView::from_nchar(vec!["涛思数据"]), + ]; + let rows = stmt.bind(¶ms)?.add_batch()?.execute()?; + assert_eq!(rows, 1); + + #[derive(Debug, Deserialize)] + #[allow(dead_code)] + struct Row { + ts: String, + c1: bool, + c2: i8, + c3: i16, + c4: i32, + c5: i64, + c6: u8, + c7: u16, + c8: u32, + c9: u64, + c10: Option, + c11: f64, + c12: String, + c13: String, + } + + let rows: Vec = taos + .query("select * from tb1") + .await? + .deserialize() + .try_collect() + .await?; + let row = &rows[0]; + dbg!(&row); + assert_eq!(row.c5, i64::MAX); + assert_eq!(row.c8, u32::MAX); + assert_eq!(row.c9, u64::MAX); + assert_eq!(row.c10.unwrap(), f32::MAX); + // assert_eq!(row.c11, f64::MAX); + assert_eq!(row.c12, "ABC"); + assert_eq!(row.c13, "涛思数据"); + + Ok(()) +} diff --git a/docs/examples/rust/nativeexample/examples/query2.rs b/docs/examples/rust/nativeexample/examples/query2.rs new file mode 100644 index 0000000000..016b291abc --- /dev/null +++ b/docs/examples/rust/nativeexample/examples/query2.rs @@ -0,0 +1,106 @@ +use std::time::Duration; + +use chrono::{DateTime, Local}; +use taos::*; + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + let dsn = "taos://"; + + let opts = PoolBuilder::new() + .max_size(5000) // max connections + .max_lifetime(Some(Duration::from_secs(60 * 60))) // lifetime of each connection + .min_idle(Some(1000)) // minimal idle connections + .connection_timeout(Duration::from_secs(2)); + + let pool = TaosBuilder::from_dsn(dsn)?.with_pool_builder(opts)?; + + let taos = pool.get()?; + + let db = "query"; + + // prepare database + taos.exec_many([ + format!("DROP DATABASE IF EXISTS `{db}`"), + format!("CREATE DATABASE `{db}`"), + format!("USE `{db}`"), + ]) + .await?; + + let inserted = taos.exec_many([ + // create super table + "CREATE TABLE `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) TAGS (`groupid` INT, `location` BINARY(16))", + // create child table + "CREATE TABLE `d0` USING `meters` TAGS(0, 'Los Angles')", + // insert into child table + "INSERT INTO `d0` values(now - 10s, 10, 116, 0.32)", + // insert with NULL values + "INSERT INTO `d0` values(now - 8s, NULL, NULL, NULL)", + // insert and automatically create table with tags if not exists + "INSERT INTO `d1` USING `meters` TAGS(1, 'San Francisco') values(now - 9s, 10.1, 119, 0.33)", + // insert many records in a single sql + "INSERT INTO `d1` values (now-8s, 10, 120, 0.33) (now - 6s, 10, 119, 0.34) (now - 4s, 11.2, 118, 0.322)", + ]).await?; + + assert_eq!(inserted, 6); + loop { + let count: usize = taos + .query_one("select count(*) from `meters`") + .await? + .unwrap_or_default(); + + if count >= 6 { + break; + } else { + println!("waiting for data"); + } + } + + let mut result = taos.query("select tbname, * from `meters`").await?; + + for field in result.fields() { + println!("got field: {}", field.name()); + } + + // Query option 1, use rows stream. + let mut rows = result.rows(); + let mut nrows = 0; + while let Some(row) = rows.try_next().await? { + for (col, (name, value)) in row.enumerate() { + println!( + "[{}] got value in col {} (named `{:>8}`): {}", + nrows, col, name, value + ); + } + nrows += 1; + } + + // Query options 2, use deserialization with serde. + #[derive(Debug, serde::Deserialize)] + #[allow(dead_code)] + struct Record { + tbname: String, + // deserialize timestamp to chrono::DateTime + ts: DateTime, + // float to f32 + current: Option, + // int to i32 + voltage: Option, + phase: Option, + groupid: i32, + // binary/varchar to String + location: String, + } + + let records: Vec = taos + .query("select tbname, * from `meters`") + .await? + .deserialize() + .try_collect() + .await?; + + dbg!(result.summary()); + assert_eq!(records.len(), 6); + dbg!(records); + Ok(()) +} diff --git a/docs/examples/rust/nativeexample/examples/subscribe.rs b/docs/examples/rust/nativeexample/examples/subscribe.rs new file mode 100644 index 0000000000..9e2e890405 --- /dev/null +++ b/docs/examples/rust/nativeexample/examples/subscribe.rs @@ -0,0 +1,103 @@ +use std::time::Duration; + +use chrono::{DateTime, Local}; +use taos::*; + +// Query options 2, use deserialization with serde. +#[derive(Debug, serde::Deserialize)] +#[allow(dead_code)] +struct Record { + // deserialize timestamp to chrono::DateTime + ts: DateTime, + // float to f32 + current: Option, + // int to i32 + voltage: Option, + phase: Option, +} + +async fn prepare(taos: Taos) -> anyhow::Result<()> { + let inserted = taos.exec_many([ + // create child table + "CREATE TABLE `d0` USING `meters` TAGS(0, 'Los Angles')", + // insert into child table + "INSERT INTO `d0` values(now - 10s, 10, 116, 0.32)", + // insert with NULL values + "INSERT INTO `d0` values(now - 8s, NULL, NULL, NULL)", + // insert and automatically create table with tags if not exists + "INSERT INTO `d1` USING `meters` TAGS(1, 'San Francisco') values(now - 9s, 10.1, 119, 0.33)", + // insert many records in a single sql + "INSERT INTO `d1` values (now-8s, 10, 120, 0.33) (now - 6s, 10, 119, 0.34) (now - 4s, 11.2, 118, 0.322)", + ]).await?; + assert_eq!(inserted, 6); + Ok(()) +} + +#[tokio::main] +async fn main() -> anyhow::Result<()> { + // std::env::set_var("RUST_LOG", "debug"); + pretty_env_logger::init(); + let dsn = "taos://localhost:6030"; + let builder = TaosBuilder::from_dsn(dsn)?; + + let taos = builder.build()?; + let db = "tmq"; + + // prepare database + taos.exec_many([ + "DROP TOPIC IF EXISTS tmq_meters".to_string(), + format!("DROP DATABASE IF EXISTS `{db}`"), + format!("CREATE DATABASE `{db}`"), + format!("USE `{db}`"), + // create super table + "CREATE TABLE `meters` (`ts` TIMESTAMP, `current` FLOAT, `voltage` INT, `phase` FLOAT) TAGS (`groupid` INT, `location` BINARY(16))".to_string(), + // create topic for subscription + format!("CREATE TOPIC tmq_meters with META AS DATABASE {db}") + ]) + .await?; + + let task = tokio::spawn(prepare(taos)); + + tokio::time::sleep(Duration::from_secs(1)).await; + + // subscribe + let tmq = TmqBuilder::from_dsn("taos://localhost:6030/?group.id=test")?; + + let mut consumer = tmq.build()?; + consumer.subscribe(["tmq_meters"]).await?; + + { + let mut stream = consumer.stream(); + + while let Some((offset, message)) = stream.try_next().await? { + // get information from offset + + // the topic + let topic = offset.topic(); + // the vgroup id, like partition id in kafka. + let vgroup_id = offset.vgroup_id(); + println!("* in vgroup id {vgroup_id} of topic {topic}\n"); + + if let Some(data) = message.into_data() { + while let Some(block) = data.fetch_raw_block().await? { + // one block for one table, get table name if needed + let name = block.table_name(); + let records: Vec = block.deserialize().try_collect()?; + println!( + "** table: {}, got {} records: {:#?}\n", + name.unwrap(), + records.len(), + records + ); + } + } + consumer.commit(offset).await?; + } + } + + consumer.unsubscribe().await; + + task.await??; + + Ok(()) +} diff --git a/docs/examples/rust/nativeexample/examples/tmq.rs b/docs/examples/rust/nativeexample/examples/tmq.rs index 800b66e8fe..d5feec1150 100644 --- a/docs/examples/rust/nativeexample/examples/tmq.rs +++ b/docs/examples/rust/nativeexample/examples/tmq.rs @@ -1,9 +1,9 @@ -use std::time::Duration; -use std::str::FromStr; -use chrono::Local; use chrono::DateTime; -use taos::*; +use chrono::Local; +use std::str::FromStr; use std::thread; +use std::time::Duration; +use taos::*; use tokio::runtime::Runtime; #[tokio::main] @@ -12,7 +12,7 @@ async fn main() -> anyhow::Result<()> { .filter_level(log::LevelFilter::Info) .init(); use taos_query::prelude::*; - // ANCHOR: create_consumer_dsn + // ANCHOR: create_consumer_dsn let dsn = "taos://localhost:6030".to_string(); println!("dsn: {}", dsn); let mut dsn = Dsn::from_str(&dsn)?; @@ -41,17 +41,25 @@ async fn main() -> anyhow::Result<()> { // ANCHOR: create_consumer_ac let group_id = "group1".to_string(); let client_id = "client1".to_string(); - dsn.params.insert("auto.offset.reset".to_string(), "latest".to_string()); - dsn.params.insert("msg.with.table.name".to_string(), "true".to_string()); - dsn.params.insert("enable.auto.commit".to_string(), "true".to_string()); - dsn.params.insert("auto.commit.interval.ms".to_string(), "1000".to_string()); + dsn.params + .insert("auto.offset.reset".to_string(), "latest".to_string()); + dsn.params + .insert("msg.with.table.name".to_string(), "true".to_string()); + dsn.params + .insert("enable.auto.commit".to_string(), "true".to_string()); + dsn.params + .insert("auto.commit.interval.ms".to_string(), "1000".to_string()); dsn.params.insert("group.id".to_string(), group_id.clone()); - dsn.params.insert("client.id".to_string(), client_id.clone()); + dsn.params + .insert("client.id".to_string(), client_id.clone()); let builder = TmqBuilder::from_dsn(&dsn)?; - let mut consumer = match builder.build().await{ + let mut consumer = match builder.build().await { Ok(consumer) => { - println!("Create consumer successfully, dsn: {}, groupId: {}, clientId: {}.", dsn, group_id, client_id); + println!( + "Create consumer successfully, dsn: {}, groupId: {}, clientId: {}.", + dsn, group_id, client_id + ); consumer } Err(err) => { @@ -61,10 +69,10 @@ async fn main() -> anyhow::Result<()> { }; // ANCHOR_END: create_consumer_ac - thread::spawn(move || { + let handle = thread::spawn(move || { let rt = Runtime::new().unwrap(); - rt.block_on(async { + tokio::time::sleep(Duration::from_secs(1)).await; let taos_insert = TaosBuilder::from_dsn(&dsn).unwrap().build().await.unwrap(); for i in 0..50 { let insert_sql = format!(r#"INSERT INTO @@ -77,16 +85,17 @@ async fn main() -> anyhow::Result<()> { tokio::time::sleep(Duration::from_millis(10)).await; } }); - - }).join().unwrap(); - + }); // ANCHOR: consume let topic = "topic_meters"; - match consumer.subscribe([topic]).await{ + match consumer.subscribe([topic]).await { Ok(_) => println!("Subscribe topics successfully."), Err(err) => { - eprintln!("Failed to subscribe topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", topic, group_id, client_id, err); + eprintln!( + "Failed to subscribe topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", + topic, group_id, client_id, err + ); return Err(err.into()); } } @@ -107,32 +116,36 @@ async fn main() -> anyhow::Result<()> { } consumer - .stream() - .try_for_each(|(offset, message)| async move { - let topic = offset.topic(); - // the vgroup id, like partition id in kafka. - let vgroup_id = offset.vgroup_id(); - println!("* in vgroup id {vgroup_id} of topic {topic}\n"); + .stream_with_timeout(Timeout::from_secs(10)) + .try_for_each(|(offset, message)| async move { + let topic = offset.topic(); + // the vgroup id, like partition id in kafka. + let vgroup_id = offset.vgroup_id(); + println!("* in vgroup id {vgroup_id} of topic {topic}\n"); - if let Some(data) = message.into_data() { - while let Some(block) = data.fetch_raw_block().await? { - let records: Vec = block.deserialize().try_collect()?; - // Add your data processing logic here - println!("** read {} records: {:#?}\n", records.len(), records); + if let Some(data) = message.into_data() { + while let Some(block) = data.fetch_raw_block().await? { + let records: Vec = block.deserialize().try_collect()?; + // Add your data processing logic here + println!("** read {} records: {:#?}\n", records.len(), records); + } } - } - Ok(()) - }) - .await.map_err(|e| { - eprintln!("Failed to poll data, topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", topic, group_id, client_id, e); - e - })?; + Ok(()) + }) + .await + .map_err(|e| { + eprintln!( + "Failed to poll data, topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", + topic, group_id, client_id, e + ); + e + })?; // ANCHOR_END: consume - // ANCHOR: consumer_commit_manually + // ANCHOR: consumer_commit_manually consumer - .stream() + .stream_with_timeout(Timeout::from_secs(10)) .try_for_each(|(offset, message)| async { // the vgroup id, like partition id in kafka. let vgroup_id = offset.vgroup_id(); @@ -162,12 +175,14 @@ async fn main() -> anyhow::Result<()> { })?; // ANCHOR_END: consumer_commit_manually - // ANCHOR: seek_offset - let assignments = match consumer.assignments().await{ + let assignments = match consumer.assignments().await { Some(assignments) => assignments, None => { - let error_message = format!("Failed to get assignments. topic: {}, groupId: {}, clientId: {}", topic, group_id, client_id); + let error_message = format!( + "Failed to get assignments. topic: {}, groupId: {}, clientId: {}", + topic, group_id, client_id + ); eprintln!("{}", error_message); return Err(anyhow::anyhow!(error_message)); } @@ -185,14 +200,10 @@ async fn main() -> anyhow::Result<()> { let end = assignment.end(); println!( "topic: {}, vgroup_id: {}, current offset: {}, begin {}, end: {}", - topic, - vgroup_id, - current, - begin, - end + topic, vgroup_id, current, begin, end ); - match consumer.offset_seek(topic, vgroup_id, begin).await{ + match consumer.offset_seek(topic, vgroup_id, begin).await { Ok(_) => (), Err(err) => { eprintln!("Failed to seek offset, topic: {}, groupId: {}, clientId: {}, vGroupId: {}, begin: {}, ErrMessage: {:?}", @@ -207,10 +218,13 @@ async fn main() -> anyhow::Result<()> { } println!("Assignment seek to beginning successfully."); // after seek offset - let assignments = match consumer.assignments().await{ + let assignments = match consumer.assignments().await { Some(assignments) => assignments, None => { - let error_message = format!("Failed to get assignments. topic: {}, groupId: {}, clientId: {}", topic, group_id, client_id); + let error_message = format!( + "Failed to get assignments. topic: {}, groupId: {}, clientId: {}", + topic, group_id, client_id + ); eprintln!("{}", error_message); return Err(anyhow::anyhow!(error_message)); } @@ -225,10 +239,9 @@ async fn main() -> anyhow::Result<()> { tokio::time::sleep(Duration::from_secs(1)).await; - taos.exec_many([ - "drop topic topic_meters", - "drop database power", - ]) - .await?; + handle.join().unwrap(); + + taos.exec_many(["drop topic topic_meters", "drop database power"]) + .await?; Ok(()) } diff --git a/docs/examples/rust/restexample/examples/tmq.rs b/docs/examples/rust/restexample/examples/tmq.rs index 0a41025955..61416133e3 100644 --- a/docs/examples/rust/restexample/examples/tmq.rs +++ b/docs/examples/rust/restexample/examples/tmq.rs @@ -1,9 +1,9 @@ -use std::time::Duration; -use std::str::FromStr; -use chrono::Local; use chrono::DateTime; -use taos::*; +use chrono::Local; +use std::str::FromStr; use std::thread; +use std::time::Duration; +use taos::*; use tokio::runtime::Runtime; #[tokio::main] @@ -12,7 +12,7 @@ async fn main() -> anyhow::Result<()> { .filter_level(log::LevelFilter::Info) .init(); use taos_query::prelude::*; - // ANCHOR: create_consumer_dsn + // ANCHOR: create_consumer_dsn let dsn = "ws://localhost:6041".to_string(); println!("dsn: {}", dsn); let mut dsn = Dsn::from_str(&dsn)?; @@ -41,17 +41,25 @@ async fn main() -> anyhow::Result<()> { // ANCHOR: create_consumer_ac let group_id = "group1".to_string(); let client_id = "client1".to_string(); - dsn.params.insert("auto.offset.reset".to_string(), "latest".to_string()); - dsn.params.insert("msg.with.table.name".to_string(), "true".to_string()); - dsn.params.insert("enable.auto.commit".to_string(), "true".to_string()); - dsn.params.insert("auto.commit.interval.ms".to_string(), "1000".to_string()); + dsn.params + .insert("auto.offset.reset".to_string(), "latest".to_string()); + dsn.params + .insert("msg.with.table.name".to_string(), "true".to_string()); + dsn.params + .insert("enable.auto.commit".to_string(), "true".to_string()); + dsn.params + .insert("auto.commit.interval.ms".to_string(), "1000".to_string()); dsn.params.insert("group.id".to_string(), group_id.clone()); - dsn.params.insert("client.id".to_string(), client_id.clone()); + dsn.params + .insert("client.id".to_string(), client_id.clone()); let builder = TmqBuilder::from_dsn(&dsn)?; - let mut consumer = match builder.build().await{ + let mut consumer = match builder.build().await { Ok(consumer) => { - println!("Create consumer successfully, dsn: {}, groupId: {}, clientId: {}.", dsn, group_id, client_id); + println!( + "Create consumer successfully, dsn: {}, groupId: {}, clientId: {}.", + dsn, group_id, client_id + ); consumer } Err(err) => { @@ -61,10 +69,10 @@ async fn main() -> anyhow::Result<()> { }; // ANCHOR_END: create_consumer_ac - thread::spawn(move || { + let handle = thread::spawn(move || { let rt = Runtime::new().unwrap(); - rt.block_on(async { + tokio::time::sleep(Duration::from_secs(1)).await; let taos_insert = TaosBuilder::from_dsn(&dsn).unwrap().build().await.unwrap(); for i in 0..50 { let insert_sql = format!(r#"INSERT INTO @@ -75,18 +83,20 @@ async fn main() -> anyhow::Result<()> { eprintln!("Failed to execute insert: {:?}", e); } tokio::time::sleep(Duration::from_millis(10)).await; + println!("Succed to execute insert 1 row"); } }); - - }).join().unwrap(); - + }); // ANCHOR: consume let topic = "topic_meters"; - match consumer.subscribe([topic]).await{ + match consumer.subscribe([topic]).await { Ok(_) => println!("Subscribe topics successfully."), Err(err) => { - eprintln!("Failed to subscribe topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", topic, group_id, client_id, err); + eprintln!( + "Failed to subscribe topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", + topic, group_id, client_id, err + ); return Err(err.into()); } } @@ -107,32 +117,36 @@ async fn main() -> anyhow::Result<()> { } consumer - .stream() - .try_for_each(|(offset, message)| async move { - let topic = offset.topic(); - // the vgroup id, like partition id in kafka. - let vgroup_id = offset.vgroup_id(); - println!("* in vgroup id {vgroup_id} of topic {topic}\n"); + .stream_with_timeout(Timeout::from_secs(10)) + .try_for_each(|(offset, message)| async move { + let topic = offset.topic(); + // the vgroup id, like partition id in kafka. + let vgroup_id = offset.vgroup_id(); + println!("* in vgroup id {vgroup_id} of topic {topic}\n"); - if let Some(data) = message.into_data() { - while let Some(block) = data.fetch_raw_block().await? { - let records: Vec = block.deserialize().try_collect()?; - // Add your data processing logic here - println!("** read {} records: {:#?}\n", records.len(), records); + if let Some(data) = message.into_data() { + while let Some(block) = data.fetch_raw_block().await? { + let records: Vec = block.deserialize().try_collect()?; + // Add your data processing logic here + println!("** read {} records: {:#?}\n", records.len(), records); + } } - } - Ok(()) - }) - .await.map_err(|e| { - eprintln!("Failed to poll data, topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", topic, group_id, client_id, e); - e - })?; + Ok(()) + }) + .await + .map_err(|e| { + eprintln!( + "Failed to poll data, topic: {}, groupId: {}, clientId: {}, ErrMessage: {:?}", + topic, group_id, client_id, e + ); + e + })?; // ANCHOR_END: consume - // ANCHOR: consumer_commit_manually + // ANCHOR: consumer_commit_manually consumer - .stream() + .stream_with_timeout(Timeout::from_secs(10)) .try_for_each(|(offset, message)| async { // the vgroup id, like partition id in kafka. let vgroup_id = offset.vgroup_id(); @@ -162,12 +176,14 @@ async fn main() -> anyhow::Result<()> { })?; // ANCHOR_END: consumer_commit_manually - // ANCHOR: seek_offset - let assignments = match consumer.assignments().await{ + let assignments = match consumer.assignments().await { Some(assignments) => assignments, None => { - let error_message = format!("Failed to get assignments. topic: {}, groupId: {}, clientId: {}", topic, group_id, client_id); + let error_message = format!( + "Failed to get assignments. topic: {}, groupId: {}, clientId: {}", + topic, group_id, client_id + ); eprintln!("{}", error_message); return Err(anyhow::anyhow!(error_message)); } @@ -185,14 +201,10 @@ async fn main() -> anyhow::Result<()> { let end = assignment.end(); println!( "topic: {}, vgroup_id: {}, current offset: {}, begin {}, end: {}", - topic, - vgroup_id, - current, - begin, - end + topic, vgroup_id, current, begin, end ); - match consumer.offset_seek(topic, vgroup_id, begin).await{ + match consumer.offset_seek(topic, vgroup_id, begin).await { Ok(_) => (), Err(err) => { eprintln!("Failed to seek offset, topic: {}, groupId: {}, clientId: {}, vGroupId: {}, begin: {}, ErrMessage: {:?}", @@ -207,10 +219,13 @@ async fn main() -> anyhow::Result<()> { } println!("Assignment seek to beginning successfully."); // after seek offset - let assignments = match consumer.assignments().await{ + let assignments = match consumer.assignments().await { Some(assignments) => assignments, None => { - let error_message = format!("Failed to get assignments. topic: {}, groupId: {}, clientId: {}", topic, group_id, client_id); + let error_message = format!( + "Failed to get assignments. topic: {}, groupId: {}, clientId: {}", + topic, group_id, client_id + ); eprintln!("{}", error_message); return Err(anyhow::anyhow!(error_message)); } @@ -225,10 +240,9 @@ async fn main() -> anyhow::Result<()> { tokio::time::sleep(Duration::from_secs(1)).await; - taos.exec_many([ - "drop topic topic_meters", - "drop database power", - ]) - .await?; + handle.join().unwrap(); + + taos.exec_many(["drop topic topic_meters", "drop database power"]) + .await?; Ok(()) } diff --git a/docs/zh/07-operation/12-multi.md b/docs/zh/07-operation/12-multi.md index 9b91a47e39..e156af55f8 100644 --- a/docs/zh/07-operation/12-multi.md +++ b/docs/zh/07-operation/12-multi.md @@ -16,7 +16,7 @@ toc_max_heading_level: 4 ### 配置方式 -多级存储支持 3 级，每级最多可配置 16 个挂载点。 +多级存储支持 3 级，每级最多可配置 128 个挂载点。 **Tips** 典型的配置方案有：0 级配置多个挂载点，每个挂载点对应单块 SAS 硬盘；1 级配置多个挂载点，每个挂载点对应单块或多块 SATA 硬盘；2 级可配置 S3 存储或其他廉价网络存储。 @@ -110,4 +110,4 @@ s3migrate database ; | :--- | :----------- | :----- | :----- | :------ | :----------------------------------------------------------- | | 1 | s3_keeplocal | 3650 | 1 | 365000 | 数据在本地保留的天数，即 data 文件在本地磁盘保留多长时间后可以上传到 S3。默认单位：天，支持 m（分钟）、h（小时）和 d（天）三个单位 | | 2 | s3_chunksize | 262144 | 131072 | 1048576 | 上传对象的大小阈值，与 TSDB_PAGESIZE 参数一样，不可修改，单位为 TSDB 页 | -| 3 | s3_compact | 0 | 0 | 1 | TSDB 文件组首次上传 S3 时，是否自动进行 compact 操作。 | \ No newline at end of file +| 3 | s3_compact | 0 | 0 | 1 | TSDB 文件组首次上传 S3 时，是否自动进行 compact 操作。 | diff --git a/docs/zh/14-reference/01-components/01-taosd.md b/docs/zh/14-reference/01-components/01-taosd.md index 3746a16c54..ba17126a92 100644 --- a/docs/zh/14-reference/01-components/01-taosd.md +++ b/docs/zh/14-reference/01-components/01-taosd.md @@ -53,7 +53,7 @@ taosd 命令行参数如下 | :--------------------: | :-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: | | queryPolicy | 查询策略，1: 只使用 vnode，不使用 qnode; 2: 没有扫描算子的子任务在 qnode 执行，带扫描算子的子任务在 vnode 执行; 3: vnode 只运行扫描算子，其余算子均在 qnode 执行 ；缺省值：1 | | maxNumOfDistinctRes | 允许返回的 distinct 结果最大行数，默认值 10 万，最大允许值 1 亿 | -| countAlwaysReturnValue | ount/hyperloglog函数在输入数据为空或者NULL的情况下是否返回值，0: 返回空行，1: 返回；该参数设置为 1 时，如果查询中含有 INTERVAL 子句或者该查询使用了TSMA时, 且相应的组或窗口内数据为空或者NULL， 对应的组或窗口将不返回查询结果. 注意此参数客户端和服务端值应保持一致. | +| countAlwaysReturnValue | count/hyperloglog函数在输入数据为空或者NULL的情况下是否返回值，0: 返回空行，1: 返回；该参数设置为 1 时，如果查询中含有 INTERVAL 子句或者该查询使用了TSMA时, 且相应的组或窗口内数据为空或者NULL， 对应的组或窗口将不返回查询结果. 注意此参数客户端和服务端值应保持一致. | ### 区域相关 diff --git a/docs/zh/14-reference/03-taos-sql/10-function.md b/docs/zh/14-reference/03-taos-sql/10-function.md index ca6ba378ca..ac9311184e 100644 --- a/docs/zh/14-reference/03-taos-sql/10-function.md +++ b/docs/zh/14-reference/03-taos-sql/10-function.md @@ -656,7 +656,8 @@ use_current_timezone: { 例如客户端所配置时区为 UTC+0800, 则 TIMETRUNCATE('2020-01-01 23:00:00', 1d, 0) 返回结果为东八区时间 '2020-01-01 08:00:00'。 而使用 TIMETRUNCATE('2020-01-01 23:00:00', 1d, 1) 时，返回结果为东八区时间 '2020-01-01 00:00:00'。 当不指定 use_current_timezone 时，use_current_timezone 默认值为 1 。 - +- 当将时间值截断到一周（1w）时，timetruncate 的计算是基于 Unix 时间戳（1970年1月1日00:00:00 UTC）进行的。Unix 时间戳始于星期四， + 因此所有截断后的日期都是星期四。 #### TIMEZONE diff --git a/docs/zh/14-reference/05-connector/10-cpp.mdx b/docs/zh/14-reference/05-connector/10-cpp.mdx index be7e44812c..7c0da088a6 100644 --- a/docs/zh/14-reference/05-connector/10-cpp.mdx +++ b/docs/zh/14-reference/05-connector/10-cpp.mdx @@ -40,151 +40,18 @@ TDengine 客户端驱动的版本号与 TDengine 服务端的版本号是一一 本节展示了使用客户端驱动访问 TDengine 集群的常见访问方式的示例代码。 -### 同步查询示例 +- 同步查询示例：[同步查询](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/demo.c) -
-同步查询 +- 异步查询示例：[异步查询](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/asyncdemo.c) -```c -{{#include examples/c/demo.c}} -``` -格式化输出不同类型字段函数 taos_print_row -```c -int taos_print_row(char *str, TAOS_ROW row, TAOS_FIELD *fields, int num_fields) { - int32_t len = 0; - for (int i = 0; i < num_fields; ++i) { - if (i > 0) { - str[len++] = ' '; - } +- 参数绑定示例：[参数绑定](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/prepare.c) - if (row[i] == NULL) { - len += sprintf(str + len, "%s", TSDB_DATA_NULL_STR); - continue; - } +- 无模式写入示例：[无模式写入](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/schemaless.c) - switch (fields[i].type) { - case TSDB_DATA_TYPE_TINYINT: - len += sprintf(str + len, "%d", *((int8_t *)row[i])); - break; - - case TSDB_DATA_TYPE_UTINYINT: - len += sprintf(str + len, "%u", *((uint8_t *)row[i])); - break; - - case TSDB_DATA_TYPE_SMALLINT: - len += sprintf(str + len, "%d", *((int16_t *)row[i])); - break; - - case TSDB_DATA_TYPE_USMALLINT: - len += sprintf(str + len, "%u", *((uint16_t *)row[i])); - break; - - case TSDB_DATA_TYPE_INT: - len += sprintf(str + len, "%d", *((int32_t *)row[i])); - break; - - case TSDB_DATA_TYPE_UINT: - len += sprintf(str + len, "%u", *((uint32_t *)row[i])); - break; - - case TSDB_DATA_TYPE_BIGINT: - len += sprintf(str + len, "%" PRId64, *((int64_t *)row[i])); - break; - - case TSDB_DATA_TYPE_UBIGINT: - len += sprintf(str + len, "%" PRIu64, *((uint64_t *)row[i])); - break; - - case TSDB_DATA_TYPE_FLOAT: { - float fv = 0; - fv = GET_FLOAT_VAL(row[i]); - len += sprintf(str + len, "%f", fv); - } break; - - case TSDB_DATA_TYPE_DOUBLE: { - double dv = 0; - dv = GET_DOUBLE_VAL(row[i]); - len += sprintf(str + len, "%lf", dv); - } break; - - case TSDB_DATA_TYPE_BINARY: - case TSDB_DATA_TYPE_NCHAR: { - int32_t charLen = varDataLen((char *)row[i] - VARSTR_HEADER_SIZE); - if (fields[i].type == TSDB_DATA_TYPE_BINARY) { - assert(charLen <= fields[i].bytes && charLen >= 0); - } else { - assert(charLen <= fields[i].bytes * TSDB_NCHAR_SIZE && charLen >= 0); - } - - memcpy(str + len, row[i], charLen); - len += charLen; - } break; - - case TSDB_DATA_TYPE_TIMESTAMP: - len += sprintf(str + len, "%" PRId64, *((int64_t *)row[i])); - break; - - case TSDB_DATA_TYPE_BOOL: - len += sprintf(str + len, "%d", *((int8_t *)row[i])); - default: - break; - } - } - str[len] = 0; - - return len; -} - -``` - -
- -### 异步查询示例 - -
-异步查询 - -```c -{{#include examples/c/asyncdemo.c}} -``` - -
- -### 参数绑定示例 - -
-参数绑定 - -```c -{{#include examples/c/prepare.c}} -``` - -
- -### 无模式写入示例 - -
-无模式写入 - -```c -{{#include examples/c/schemaless.c}} -``` - -
- -### 订阅和消费示例 - -
-订阅和消费 - -```c - {{#include examples/c/tmq.c}} -``` - -
+- 订阅和消费示例：[订阅和消费](https://github.com/taosdata/TDengine/tree/main/docs/examples/c/tmq.c) :::info -更多示例代码及下载请见 [GitHub](https://github.com/taosdata/TDengine/tree/develop/examples/c)。 +更多示例代码及下载请见 [GitHub](https://github.com/taosdata/TDengine/tree/main/docs/examples/c)。 也可以在安装目录下的 `examples/c` 路径下找到。 该目录下有 makefile，在 Linux/macOS 环境下，直接执行 make 就可以编译得到执行文件。 **提示：**在 ARM 环境下编译时，请将 makefile 中的 `-msse4.2` 去掉，这个选项只有在 x64/x86 硬件平台上才能支持。 @@ -403,7 +270,7 @@ TDengine 的异步 API 均采用非阻塞调用模式。应用程序可以用多 说明：如果 `taos_stmt_execute()` 执行成功，假如不需要改变 SQL 语句的话，那么是可以复用 `taos_stmt_prepare()` 的解析结果，直接进行第 3 ～ 6 步绑定新数据的。但如果执行出错，那么并不建议继续在当前的环境上下文下继续工作，而是建议释放资源，然后从 `taos_stmt_init()` 步骤重新开始。 -接口相关的具体函数如下（也可以参考 [prepare.c](https://github.com/taosdata/TDengine/blob/develop/examples/c/prepare.c) 文件中使用对应函数的方式）： +接口相关的具体函数如下（也可以参考 [prepare.c](https://github.com/taosdata/TDengine/blob/develop/docs/examples/c/prepare.c) 文件中使用对应函数的方式）： - `TAOS_STMT* taos_stmt_init(TAOS *taos)` - **接口说明**：初始化一个预编译的 SQL 语句对象。 diff --git a/docs/zh/14-reference/05-connector/14-java.mdx b/docs/zh/14-reference/05-connector/14-java.mdx index a752867b3f..5f95233163 100644 --- a/docs/zh/14-reference/05-connector/14-java.mdx +++ b/docs/zh/14-reference/05-connector/14-java.mdx @@ -145,7 +145,7 @@ WKB规范请参考[Well-Known Binary (WKB)](https://libgeos.org/specifications/w ## 示例程序汇总 -示例程序源码位于 `TDengine/examples/JDBC` 下: +示例程序源码位于 `TDengine/docs/examples/JDBC` 下: - JDBCDemo：JDBC 示例源程序。 - connectionPools：HikariCP, Druid, dbcp, c3p0 等连接池中使用 taos-jdbcdriver。 @@ -154,7 +154,7 @@ WKB规范请参考[Well-Known Binary (WKB)](https://libgeos.org/specifications/w - springbootdemo: Springboot 中使用 taos-jdbcdriver。 - consumer-demo：Consumer 消费 TDengine 数据示例，可通过参数控制消费速度。 -请参考：[JDBC example](https://github.com/taosdata/TDengine/tree/3.0/examples/JDBC) +请参考：[JDBC example](https://github.com/taosdata/TDengine/tree/main/docs/examples/JDBC) ## 常见问题 diff --git a/docs/zh/14-reference/05-connector/20-go.mdx b/docs/zh/14-reference/05-connector/20-go.mdx index 45782a7284..8d65fba321 100644 --- a/docs/zh/14-reference/05-connector/20-go.mdx +++ b/docs/zh/14-reference/05-connector/20-go.mdx @@ -66,6 +66,9 @@ TDengine 其他功能模块的报错，请参考 [错误码](../../../reference/ **注意**：JSON 类型仅在 tag 中支持。 +## 示例程序汇总 +示例程序源码请参考：[示例程序](https://github.com/taosdata/driver-go/tree/main/examples) + ## 常见问题 1. database/sql 中 stmt（参数绑定）相关接口崩溃 @@ -894,5 +897,4 @@ type TopicPartition struct { ## 附录 * [driver-go 文档](https://pkg.go.dev/github.com/taosdata/driver-go/v3)。 -* [示例程序](https://github.com/taosdata/driver-go/tree/3.0/examples)。 * [视频教程](https://www.taosdata.com/blog/2020/11/11/1951.html)。 diff --git a/docs/zh/14-reference/05-connector/26-rust.mdx b/docs/zh/14-reference/05-connector/26-rust.mdx index 6b0ec4a68d..88be297ac6 100644 --- a/docs/zh/14-reference/05-connector/26-rust.mdx +++ b/docs/zh/14-reference/05-connector/26-rust.mdx @@ -85,9 +85,7 @@ TDengine 目前支持时间戳、数字、字符、布尔类型，与 Rust 对 ## 示例程序汇总 -示例程序源码位于 `TDengine/examples/rust` 下: - -请参考：[rust example](https://github.com/taosdata/TDengine/tree/3.0/examples/rust) +示例程序源码请参考：[rust example](https://github.com/taosdata/TDengine/tree/main/docs/examples/rust) ## 常见问题 diff --git a/examples/c/makefile b/examples/c/makefile deleted file mode 100644 index 244d13fad7..0000000000 --- a/examples/c/makefile +++ /dev/null @@ -1,27 +0,0 @@ -# Copyright (c) 2017 by TAOS Technologies, Inc. -# todo: library dependency, header file dependency - -ROOT=./ -TARGET=exe -LFLAGS = '-Wl,-rpath,/usr/local/taos/driver/' -ltaos -lpthread -lm -lrt -CFLAGS = -O3 -g -Wall -Wno-deprecated -fPIC -Wno-unused-result -Wconversion \ - -Wno-char-subscripts -D_REENTRANT -Wno-format -D_REENTRANT -DLINUX \ - -Wno-unused-function -D_M_X64 -I/usr/local/taos/include -std=gnu99 \ - -I/usr/local/include/cjson -all: $(TARGET) - -exe: - gcc $(CFLAGS) ./asyncdemo.c -o $(ROOT)asyncdemo $(LFLAGS) - gcc $(CFLAGS) ./demo.c -o $(ROOT)demo $(LFLAGS) - gcc $(CFLAGS) ./prepare.c -o $(ROOT)prepare $(LFLAGS) - gcc $(CFLAGS) ./stream_demo.c -o $(ROOT)stream_demo $(LFLAGS) - gcc $(CFLAGS) ./tmq.c -o $(ROOT)tmq $(LFLAGS) - gcc $(CFLAGS) ./schemaless.c -o $(ROOT)schemaless $(LFLAGS) - -clean: - rm $(ROOT)asyncdemo - rm $(ROOT)demo - rm $(ROOT)prepare - rm $(ROOT)stream_demo - rm $(ROOT)tmq - rm $(ROOT)schemaless diff --git a/include/common/tdataformat.h b/include/common/tdataformat.h index 0acb28fabd..c14704ae19 100644 --- a/include/common/tdataformat.h +++ b/include/common/tdataformat.h @@ -193,7 +193,7 @@ int32_t tColDataDecompress(void *input, SColDataCompressInfo *info, SColData *co // for stmt bind int32_t tColDataAddValueByBind(SColData *pColData, TAOS_MULTI_BIND *pBind, int32_t buffMaxLen); -void tColDataSortMerge(SArray *colDataArr); +int32_t tColDataSortMerge(SArray *colDataArr); // for raw block int32_t tColDataAddValueByDataBlock(SColData *pColData, int8_t type, int32_t bytes, int32_t nRows, char *lengthOrbitmap, diff --git a/include/common/trow.h b/include/common/trow.h index e43c67f931..6b836add21 100644 --- a/include/common/trow.h +++ b/include/common/trow.h @@ -213,7 +213,6 @@ static FORCE_INLINE SKvRowIdx *tdKvRowColIdxAt(STSRow *pRow, col_id_t idx) { } static FORCE_INLINE int16_t tdKvRowColIdAt(STSRow *pRow, col_id_t idx) { - ASSERT(idx >= 0); if (idx == 0) { return PRIMARYKEY_TIMESTAMP_COL_ID; } @@ -288,9 +287,7 @@ int32_t tdGetBitmapValType(const void *pBitmap, int16_t colIdx, TDRowValT *pValT * */ -static FORCE_INLINE void tdSRowInit(SRowBuilder *pBuilder, int16_t sver) { - pBuilder->sver = sver; -} +static FORCE_INLINE void tdSRowInit(SRowBuilder *pBuilder, int16_t sver) { pBuilder->sver = sver; } int32_t tdSRowSetInfo(SRowBuilder *pBuilder, int32_t nCols, int32_t nBoundCols, int32_t flen); int32_t tdSRowSetTpInfo(SRowBuilder *pBuilder, int32_t nCols, int32_t flen); int32_t tdSRowSetExtendedInfo(SRowBuilder *pBuilder, int32_t nCols, int32_t nBoundCols, int32_t flen, @@ -313,7 +310,7 @@ int32_t tdAppendColValToRow(SRowBuilder *pBuilder, col_id_t colId, int8_t colTyp int32_t tdGetTpRowValOfCol(SCellVal *output, STSRow *pRow, void *pBitmap, int8_t colType, int32_t offset, int16_t colIdx); int32_t tdGetKvRowValOfCol(SCellVal *output, STSRow *pRow, void *pBitmap, int32_t offset, int16_t colIdx); -void tTSRowGetVal(STSRow *pRow, STSchema *pTSchema, int16_t iCol, SColVal *pColVal); +int32_t tTSRowGetVal(STSRow *pRow, STSchema *pTSchema, int16_t iCol, SColVal *pColVal); typedef struct { STSchema *pSchema; diff --git a/include/common/tvariant.h b/include/common/tvariant.h index 6f88eba027..9a0585dada 100644 --- a/include/common/tvariant.h +++ b/include/common/tvariant.h @@ -46,14 +46,14 @@ int32_t toUInteger(const char *z, int32_t n, int32_t base, uint64_t *value); /** * non floating point integers -*/ + */ int32_t toIntegerPure(const char *z, int32_t n, int32_t base, int64_t *value); void taosVariantCreateFromBinary(SVariant *pVar, const char *pz, size_t len, uint32_t type); void taosVariantDestroy(SVariant *pV); -void taosVariantAssign(SVariant *pDst, const SVariant *pSrc); +int32_t taosVariantAssign(SVariant *pDst, const SVariant *pSrc); int32_t taosVariantCompare(const SVariant *p1, const SVariant *p2); diff --git a/include/libs/monitor/monitor.h b/include/libs/monitor/monitor.h index 6007d52bb4..96ac53ef4e 100644 --- a/include/libs/monitor/monitor.h +++ b/include/libs/monitor/monitor.h @@ -30,6 +30,8 @@ extern "C" { #define MON_VER_LEN 12 #define MON_LOG_LEN 1024 +#define VNODE_METRIC_TAG_VALUE_INSERT_AFFECTED_ROWS "inserted_rows" + typedef struct { int64_t ts; ELogLevel level; @@ -216,6 +218,7 @@ typedef struct { } SDmNotifyHandle; int32_t monInit(const SMonCfg *pCfg); +void monInitVnode(); void monCleanup(); void monRecordLog(int64_t ts, ELogLevel level, const char *content); int32_t monGetLogs(SMonLogs *logs); diff --git a/include/os/osFile.h b/include/os/osFile.h index 8bacb1bf7c..a56c54b086 100644 --- a/include/os/osFile.h +++ b/include/os/osFile.h @@ -120,15 +120,29 @@ int32_t taosSetFileHandlesLimit(); int32_t taosLinkFile(char *src, char *dst); -FILE* taosOpenCFile(const char* filename, const char* mode); -int taosSeekCFile(FILE* file, int64_t offset, int whence); -size_t taosReadFromCFile(void *buffer, size_t size, size_t count, FILE *stream ); -size_t taosWriteToCFile(const void* ptr, size_t size, size_t nitems, FILE* stream); -int taosCloseCFile(FILE *); -int taosSetAutoDelFile(char* path); +FILE *taosOpenCFile(const char *filename, const char *mode); +int taosSeekCFile(FILE *file, int64_t offset, int whence); +size_t taosReadFromCFile(void *buffer, size_t size, size_t count, FILE *stream); +size_t taosWriteToCFile(const void *ptr, size_t size, size_t nitems, FILE *stream); +int taosCloseCFile(FILE *); +int taosSetAutoDelFile(char *path); bool lastErrorIsFileNotExist(); +#ifdef BUILD_WITH_RAND_ERR +#define STUB_RAND_NETWORK_ERR(status) \ + do { \ + if (tsEnableRandErr && (tsRandErrScope & RAND_ERR_NETWORK)) { \ + uint32_t r = taosRand() % tsRandErrDivisor; \ + if ((r + 1) <= tsRandErrChance) { \ + status = TSDB_CODE_RPC_NETWORK_UNAVAIL; \ + } \ + } \ + while (0) +#else +#define STUB_RAND_NETWORK_ERR(status) +#endif + #ifdef __cplusplus } #endif diff --git a/include/util/taoserror.h b/include/util/taoserror.h index a5737277ff..02e6a49bb6 100644 --- a/include/util/taoserror.h +++ b/include/util/taoserror.h @@ -91,7 +91,7 @@ int32_t taosGetErrSize(); #define TSDB_CODE_RPC_NETWORK_BUSY TAOS_DEF_ERROR_CODE(0, 0x0024) #define TSDB_CODE_HTTP_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0025) #define TSDB_CODE_RPC_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0026) -#define TSDB_CODE_RPC_ASYNC_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0027) +#define TSDB_CODE_RPC_ASYNC_MODULE_QUIT TAOS_DEF_ERROR_CODE(0, 0x0027) @@ -152,6 +152,7 @@ int32_t taosGetErrSize(); #define TSDB_CODE_FAILED_TO_CONNECT_S3 TAOS_DEF_ERROR_CODE(0, 0x0135) #define TSDB_CODE_MSG_PREPROCESSED TAOS_DEF_ERROR_CODE(0, 0x0136) // internal #define TSDB_CODE_OUT_OF_BUFFER TAOS_DEF_ERROR_CODE(0, 0x0137) +#define TSDB_CODE_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x0138) //client #define TSDB_CODE_TSC_INVALID_OPERATION TAOS_DEF_ERROR_CODE(0, 0x0200) @@ -467,6 +468,8 @@ int32_t taosGetErrSize(); #define TSDB_CODE_DNODE_INVALID_TTL_CHG_ON_WR TAOS_DEF_ERROR_CODE(0, 0x0427) #define TSDB_CODE_DNODE_INVALID_EN_WHITELIST TAOS_DEF_ERROR_CODE(0, 0x0428) #define TSDB_CODE_DNODE_INVALID_MONITOR_PARAS TAOS_DEF_ERROR_CODE(0, 0x0429) +#define TSDB_CODE_MNODE_STOPPED TAOS_DEF_ERROR_CODE(0, 0x042A) + // mnode-sma #define TSDB_CODE_MND_SMA_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0480) @@ -530,6 +533,7 @@ int32_t taosGetErrSize(); #define TSDB_CODE_VND_COLUMN_COMPRESS_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0536) #define TSDB_CODE_VND_ARB_NOT_SYNCED TAOS_DEF_ERROR_CODE(0, 0x0537) // internal #define TSDB_CODE_VND_WRITE_DISABLED TAOS_DEF_ERROR_CODE(0, 0x0538) // internal +#define TSDB_CODE_VND_TTL_FLUSH_INCOMPLETION TAOS_DEF_ERROR_CODE(0, 0x0539) // internal // tsdb #define TSDB_CODE_TDB_INVALID_TABLE_ID TAOS_DEF_ERROR_CODE(0, 0x0600) @@ -700,6 +704,7 @@ int32_t taosGetErrSize(); #define TSDB_CODE_TQ_GROUP_NOT_SET TAOS_DEF_ERROR_CODE(0, 0x0A0B) #define TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND TAOS_DEF_ERROR_CODE(0, 0x0A0C) #define TSDB_CODE_TQ_NO_COMMITTED_OFFSET TAOS_DEF_ERROR_CODE(0, 0x0A0D) +#define TSDB_CODE_TQ_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x0A0E) // wal // #define TSDB_CODE_WAL_APP_ERROR TAOS_DEF_ERROR_CODE(0, 0x1000) // 2.x diff --git a/include/util/tdef.h b/include/util/tdef.h index 35c4adab50..f087c28684 100644 --- a/include/util/tdef.h +++ b/include/util/tdef.h @@ -294,7 +294,7 @@ typedef enum ELogicConditionType { #define TSDB_SHOW_SUBQUERY_LEN 1000 #define TSDB_LOG_VAR_LEN 32 -#define TSDB_MAX_EP_NUM 10 +#define TSDB_MAX_EP_NUM 10 #define TSDB_ARB_GROUP_MEMBER_NUM 2 #define TSDB_ARB_TOKEN_SIZE 32 @@ -568,12 +568,7 @@ enum { SND_WORKER_TYPE__UNIQUE, }; -enum { - RAND_ERR_MEMORY = 1, - RAND_ERR_FILE = 2, - // RAND_ERR_SCOPE_XXX... = 4, - // ... -}; +enum { RAND_ERR_MEMORY = 1, RAND_ERR_FILE = 2, RAND_ERR_NETWORK = 4 }; #define DEFAULT_HANDLE 0 #define MNODE_HANDLE 1 diff --git a/include/util/texception.h b/include/util/texception.h deleted file mode 100644 index 576545d96c..0000000000 --- a/include/util/texception.h +++ /dev/null @@ -1,129 +0,0 @@ -/* - * Copyright (c) 2020 TAOS Data, Inc. - * - * This program is free software: you can use, redistribute, and/or modify - * it under the terms of the GNU Affero General Public License, version 3 - * or later ("AGPL"), as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. - * - * You should have received a copy of the GNU Affero General Public License - * along with this program. If not, see . - */ - -#ifndef _TD_UTIL_EXCEPTION_H_ -#define _TD_UTIL_EXCEPTION_H_ - -#include "os.h" - -#ifdef __cplusplus -extern "C" { -#endif - -/* - * cleanup actions - */ -typedef struct SCleanupAction { - bool failOnly; - uint8_t wrapper; - uint16_t reserved; - void* func; - union { - void* Ptr; - bool Bool; - char Char; - int8_t Int8; - uint8_t Uint8; - int16_t Int16; - uint16_t Uint16; - int32_t Int; - uint32_t Uint; - int32_t Int32; - uint32_t Uint32; - int64_t Int64; - uint64_t Uint64; - float Float; - double Double; - } arg1, arg2; -} SCleanupAction; - -/* - * exception hander registration - */ -typedef struct SExceptionNode { - struct SExceptionNode* prev; - jmp_buf jb; - int32_t code; - int32_t maxCleanupAction; - int32_t numCleanupAction; - SCleanupAction* cleanupActions; -} SExceptionNode; - -// functions & macros for auto-cleanup - -void cleanupPush_void_ptr_ptr(bool failOnly, void* func, void* arg1, void* arg2); -void cleanupPush_void_ptr_bool(bool failOnly, void* func, void* arg1, bool arg2); -void cleanupPush_void_ptr(bool failOnly, void* func, void* arg); -void cleanupPush_int_int(bool failOnly, void* func, int32_t arg); -void cleanupPush_void(bool failOnly, void* func); -void cleanupPush_int_ptr(bool failOnly, void* func, void* arg); - -int32_t cleanupGetActionCount(); -void cleanupExecuteTo(int32_t anchor, bool failed); -void cleanupExecute(SExceptionNode* node, bool failed); -bool cleanupExceedLimit(); - -#define CLEANUP_PUSH_VOID_PTR_PTR(failOnly, func, arg1, arg2) \ - cleanupPush_void_ptr_ptr((failOnly), (void*)(func), (void*)(arg1), (void*)(arg2)) -#define CLEANUP_PUSH_VOID_PTR_BOOL(failOnly, func, arg1, arg2) \ - cleanupPush_void_ptr_bool((failOnly), (void*)(func), (void*)(arg1), (bool)(arg2)) -#define CLEANUP_PUSH_VOID_PTR(failOnly, func, arg) cleanupPush_void_ptr((failOnly), (void*)(func), (void*)(arg)) -#define CLEANUP_PUSH_INT_INT(failOnly, func, arg) cleanupPush_void_ptr((failOnly), (void*)(func), (int32_t)(arg)) -#define CLEANUP_PUSH_VOID(failOnly, func) cleanupPush_void((failOnly), (void*)(func)) -#define CLEANUP_PUSH_INT_PTR(failOnly, func, arg) cleanupPush_int_ptr((failOnly), (void*)(func), (void*)(arg)) -#define CLEANUP_PUSH_FREE(failOnly, arg) cleanupPush_void_ptr((failOnly), free, (void*)(arg)) -#define CLEANUP_PUSH_CLOSE(failOnly, arg) cleanupPush_int_int((failOnly), close, (int32_t)(arg)) -#define CLEANUP_PUSH_FCLOSE(failOnly, arg) cleanupPush_int_ptr((failOnly), fclose, (void*)(arg)) - -#define CLEANUP_GET_ANCHOR() cleanupGetActionCount() -#define CLEANUP_EXECUTE_TO(anchor, failed) cleanupExecuteTo((anchor), (failed)) -#define CLEANUP_EXCEED_LIMIT() cleanupExceedLimit() - -// functions & macros for exception handling - -void exceptionPushNode(SExceptionNode* node); -int32_t exceptionPopNode(); -void exceptionThrow(int32_t code); - -#define TRY(maxCleanupActions) \ - do { \ - SExceptionNode exceptionNode = {0}; \ - SCleanupAction cleanupActions[(maxCleanupActions) > 0 ? (maxCleanupActions) : 1]; \ - exceptionNode.maxCleanupAction = (maxCleanupActions) > 0 ? (maxCleanupActions) : 1; \ - exceptionNode.cleanupActions = cleanupActions; \ - exceptionPushNode(&exceptionNode); \ - int32_t caughtException = setjmp(exceptionNode.jb); \ - if (caughtException == 0) - -#define CATCH(code) \ - int32_t code = exceptionPopNode(); \ - if (caughtException == 1) - -#define FINALLY(code) int32_t code = exceptionPopNode(); - -#define END_TRY \ - } \ - while (0) \ - ; - -#define THROW(x) exceptionThrow((x)) -#define CAUGHT_EXCEPTION() ((bool)(caughtException == 1)) -#define CLEANUP_EXECUTE() cleanupExecute(&exceptionNode, CAUGHT_EXCEPTION()) - -#ifdef __cplusplus -} -#endif - -#endif /*_TD_UTIL_EXCEPTION_H_*/ diff --git a/include/util/tlosertree.h b/include/util/tlosertree.h index b3aa37a537..c82eff4bd7 100644 --- a/include/util/tlosertree.h +++ b/include/util/tlosertree.h @@ -45,9 +45,9 @@ int32_t tMergeTreeCreate(SMultiwayMergeTreeInfo **pTree, uint32_t numOfEntries, void tMergeTreeDestroy(SMultiwayMergeTreeInfo **pTree); -void tMergeTreeAdjust(SMultiwayMergeTreeInfo *pTree, int32_t idx); +int32_t tMergeTreeAdjust(SMultiwayMergeTreeInfo *pTree, int32_t idx); -void tMergeTreeRebuild(SMultiwayMergeTreeInfo *pTree); +int32_t tMergeTreeRebuild(SMultiwayMergeTreeInfo *pTree); void tMergeTreePrint(const SMultiwayMergeTreeInfo *pTree); diff --git a/packaging/cfg/taos.cfg b/packaging/cfg/taos.cfg index 6d25db843d..ebbab8697b 100644 --- a/packaging/cfg/taos.cfg +++ b/packaging/cfg/taos.cfg @@ -98,11 +98,6 @@ # enable/disable system monitor # monitor 1 -# enable/disable audit log -# audit 1 - -# enable/disable audit create table -# auditCreateTable 1 # The following parameter is used to limit the maximum number of lines in log files. # max number of lines per log filters diff --git a/packaging/tools/install.sh b/packaging/tools/install.sh index cc3868d2f0..8a6b159a22 100755 --- a/packaging/tools/install.sh +++ b/packaging/tools/install.sh @@ -229,6 +229,9 @@ function install_bin() { if [ -d ${script_dir}/${xname}/bin ]; then ${csudo}cp -r ${script_dir}/${xname}/bin/* ${install_main_dir}/bin fi + if [ -e ${script_dir}/${xname}/uninstall_${xname}.sh ]; then + ${csudo}cp -r ${script_dir}/${xname}/uninstall_${xname}.sh ${install_main_dir}/uninstall_${xname}.sh + fi fi if [ -f ${script_dir}/bin/quick_deploy.sh ]; then @@ -250,6 +253,8 @@ function install_bin() { for service in "${services[@]}"; do [ -x ${install_main_dir}/bin/${service} ] && ${csudo}ln -sf ${install_main_dir}/bin/${service} ${bin_link_dir}/${service} || : done + + [ -x ${install_main_dir}/uninstall_${xname}.sh ] && ${csudo}ln -sf ${install_main_dir}/uninstall_${xname}.sh ${bin_link_dir}/uninstall_${xname}.sh || : } function install_lib() { diff --git a/packaging/tools/makeclient.sh b/packaging/tools/makeclient.sh index f46a9adeff..d67d436fa7 100755 --- a/packaging/tools/makeclient.sh +++ b/packaging/tools/makeclient.sh @@ -182,15 +182,16 @@ if [[ $productName == "TDengine" ]] && [ "$verMode" != "cloud" ]; then # Copy example code mkdir -p ${install_dir}/examples examples_dir="${top_dir}/examples" + new_example_dir="${top_dir}/docs/examples" cp -r ${examples_dir}/c ${install_dir}/examples if [[ "$pagMode" != "lite" ]] && [[ "$cpuType" != "aarch32" ]]; then - cp -r ${examples_dir}/JDBC ${install_dir}/examples - cp -r ${examples_dir}/matlab ${install_dir}/examples - cp -r ${examples_dir}/python ${install_dir}/examples - cp -r ${examples_dir}/R ${install_dir}/examples - cp -r ${examples_dir}/go ${install_dir}/examples - cp -r ${examples_dir}/nodejs ${install_dir}/examples - cp -r ${examples_dir}/C# ${install_dir}/examples + cp -r ${new_example_dir}/JDBC ${install_dir}/examples ||: + cp -r ${examples_dir}/matlab ${install_dir}/examples ||: + cp -r ${examples_dir}/python ${install_dir}/examples ||: + cp -r ${examples_dir}/R ${install_dir}/examples ||: + cp -r ${examples_dir}/go ${install_dir}/examples ||: + cp -r ${examples_dir}/nodejs ${install_dir}/examples ||: + cp -r ${examples_dir}/C# ${install_dir}/examples ||: mkdir -p ${install_dir}/examples/taosbenchmark-json && cp ${examples_dir}/../tools/taos-tools/example/* ${install_dir}/examples/taosbenchmark-json fi diff --git a/packaging/tools/makepkg.sh b/packaging/tools/makepkg.sh index 9e1cc73238..798d73d0f3 100755 --- a/packaging/tools/makepkg.sh +++ b/packaging/tools/makepkg.sh @@ -284,34 +284,35 @@ if [[ $dbName == "taos" ]]; then # Copy example code mkdir -p ${install_dir}/examples examples_dir="${top_dir}/examples" + new_example_dir="${top_dir}/docs/examples" cp -r ${examples_dir}/c ${install_dir}/examples if [[ "$pagMode" != "lite" ]] && [[ "$cpuType" != "aarch32" ]]; then - if [ -d ${examples_dir}/JDBC/connectionPools/target ]; then - rm -rf ${examples_dir}/JDBC/connectionPools/target + if [ -d ${new_example_dir}/JDBC/connectionPools/target ]; then + rm -rf ${new_example_dir}/JDBC/connectionPools/target fi - if [ -d ${examples_dir}/JDBC/JDBCDemo/target ]; then - rm -rf ${examples_dir}/JDBC/JDBCDemo/target + if [ -d ${new_example_dir}/JDBC/JDBCDemo/target ]; then + rm -rf ${new_example_dir}/JDBC/JDBCDemo/target fi - if [ -d ${examples_dir}/JDBC/mybatisplus-demo/target ]; then - rm -rf ${examples_dir}/JDBC/mybatisplus-demo/target + if [ -d ${new_example_dir}/JDBC/mybatisplus-demo/target ]; then + rm -rf ${new_example_dir}/JDBC/mybatisplus-demo/target fi - if [ -d ${examples_dir}/JDBC/springbootdemo/target ]; then - rm -rf ${examples_dir}/JDBC/springbootdemo/target + if [ -d ${new_example_dir}/JDBC/springbootdemo/target ]; then + rm -rf ${new_example_dir}/JDBC/springbootdemo/target fi - if [ -d ${examples_dir}/JDBC/SpringJdbcTemplate/target ]; then - rm -rf ${examples_dir}/JDBC/SpringJdbcTemplate/target + if [ -d ${new_example_dir}/JDBC/SpringJdbcTemplate/target ]; then + rm -rf ${new_example_dir}/JDBC/SpringJdbcTemplate/target fi - if [ -d ${examples_dir}/JDBC/taosdemo/target ]; then - rm -rf ${examples_dir}/JDBC/taosdemo/target + if [ -d ${new_example_dir}/JDBC/taosdemo/target ]; then + rm -rf ${new_example_dir}/JDBC/taosdemo/target fi - cp -r ${examples_dir}/JDBC ${install_dir}/examples - cp -r ${examples_dir}/matlab ${install_dir}/examples - cp -r ${examples_dir}/python ${install_dir}/examples - cp -r ${examples_dir}/R ${install_dir}/examples - cp -r ${examples_dir}/go ${install_dir}/examples - cp -r ${examples_dir}/nodejs ${install_dir}/examples - cp -r ${examples_dir}/C# ${install_dir}/examples + cp -r ${new_example_dir}/JDBC ${install_dir}/examples ||: + cp -r ${examples_dir}/matlab ${install_dir}/examples ||: + cp -r ${examples_dir}/python ${install_dir}/examples ||: + cp -r ${examples_dir}/R ${install_dir}/examples ||: + cp -r ${examples_dir}/go ${install_dir}/examples ||: + cp -r ${examples_dir}/nodejs ${install_dir}/examples ||: + cp -r ${examples_dir}/C# ${install_dir}/examples ||: mkdir -p ${install_dir}/examples/taosbenchmark-json && cp ${examples_dir}/../tools/taos-tools/example/* ${install_dir}/examples/taosbenchmark-json fi @@ -363,8 +364,8 @@ if [ "$verMode" == "cluster" ]; then # copy taosx if [ -d ${top_dir}/../enterprise/src/plugins/taosx/release/taosx ]; then cp -r ${top_dir}/../enterprise/src/plugins/taosx/release/taosx ${install_dir} - cp ${top_dir}/../enterprise/src/plugins/taosx/packaging/uninstall.sh ${install_dir}/taosx - sed -i 's/target=\"\"/target=\"taosx\"/g' ${install_dir}/taosx/uninstall.sh + cp ${top_dir}/../enterprise/src/plugins/taosx/packaging/uninstall.sh ${install_dir}/taosx/uninstall_taosx.sh + sed -i "s/uninstall.sh/uninstall_taosx.sh/g" ${install_dir}/taosx/uninstall_taosx.sh fi fi fi diff --git a/packaging/tools/remove.sh b/packaging/tools/remove.sh index 7af64fab1e..58a17e2a50 100755 --- a/packaging/tools/remove.sh +++ b/packaging/tools/remove.sh @@ -56,7 +56,11 @@ local_bin_link_dir="/usr/local/bin" service_config_dir="/etc/systemd/system" config_dir="/etc/${PREFIX}" -services=(${PREFIX}"d" ${PREFIX}"adapter" ${PREFIX}"x" ${PREFIX}"-explorer" ${PREFIX}"keeper") +if [ "${verMode}" == "cluster" ]; then + services=(${PREFIX}"d" ${PREFIX}"adapter" ${PREFIX}"keeper") +else + services=(${PREFIX}"d" ${PREFIX}"adapter" ${PREFIX}"keeper" ${PREFIX}"-explorer") +fi tools=(${PREFIX} ${PREFIX}"Benchmark" ${PREFIX}"dump" ${PREFIX}"demo" udfd set_core.sh TDinsight.sh $uninstallScript start-all.sh stop-all.sh) csudo="" @@ -222,6 +226,30 @@ function remove_data_and_config() { [ -d "${log_dir}" ] && ${csudo}rm -rf ${log_dir} } +echo +echo "Do you want to remove all the data, log and configuration files? [y/n]" +read answer +remove_flag=false +if [ X$answer == X"y" ] || [ X$answer == X"Y" ]; then + confirmMsg="I confirm that I would like to delete all data, log and configuration files" + echo "Please enter '${confirmMsg}' to continue" + read answer + if [ X"$answer" == X"${confirmMsg}" ]; then + remove_flag=true + else + echo "answer doesn't match, skip this step" + fi +fi +echo + +if [ -e ${install_main_dir}/uninstall_${PREFIX}x.sh ]; then + if [ X$remove_flag == X"true" ]; then + bash ${install_main_dir}/uninstall_${PREFIX}x.sh --clean-all true + else + bash ${install_main_dir}/uninstall_${PREFIX}x.sh --clean-all false + fi +fi + remove_bin clean_header # Remove lib file @@ -232,6 +260,11 @@ clean_log clean_config # Remove data link directory ${csudo}rm -rf ${data_link_dir} || : + +if [ X$remove_flag == X"true" ]; then + remove_data_and_config +fi + ${csudo}rm -rf ${install_main_dir} || : if [[ -e /etc/os-release ]]; then osinfo=$(awk -F= '/^NAME/{print $2}' /etc/os-release) @@ -254,20 +287,6 @@ if [ "$osType" = "Darwin" ]; then ${csudo}rm -rf /Applications/TDengine.app fi -echo -echo "Do you want to remove all the data, log and configuration files? [y/n]" -read answer -if [ X$answer == X"y" ] || [ X$answer == X"Y" ]; then - confirmMsg="I confirm that I would like to delete all data, log and configuration files" - echo "Please enter '${confirmMsg}' to continue" - read answer - if [ X"$answer" == X"${confirmMsg}" ]; then - remove_data_and_config - else - echo "answer doesn't match, skip this step" - fi -fi - command -v systemctl >/dev/null 2>&1 && ${csudo}systemctl daemon-reload >/dev/null 2>&1 || true echo echo "${productName} is removed successfully!" diff --git a/packaging/tools/remove_client.sh b/packaging/tools/remove_client.sh index de14cb38d9..31b1053a42 100755 --- a/packaging/tools/remove_client.sh +++ b/packaging/tools/remove_client.sh @@ -31,6 +31,8 @@ bin_link_dir="/usr/bin" lib_link_dir="/usr/lib" lib64_link_dir="/usr/lib64" inc_link_dir="/usr/include" +log_dir="/var/log/${clientName2}" +cfg_dir="/etc/${clientName2}" csudo="" if command -v sudo > /dev/null; then @@ -92,6 +94,24 @@ function clean_log() { ${csudo}rm -rf ${log_link_dir} || : } +function clean_config_and_log_dir() { + # Remove link + echo "Do you want to remove all the log and configuration files? [y/n]" + read answer + if [ X$answer == X"y" ] || [ X$answer == X"Y" ]; then + confirmMsg="I confirm that I would like to delete all log and configuration files" + echo "Please enter '${confirmMsg}' to continue" + read answer + if [ X"$answer" == X"${confirmMsg}" ]; then + # Remove dir + rm -rf ${cfg_dir} || : + rm -rf ${log_dir} || : + else + echo "answer doesn't match, skip this step" + fi + fi +} + # Stop client. kill_client # Remove binary file and links @@ -104,6 +124,8 @@ clean_lib clean_log # Remove link configuration file clean_config +# Remove dir +clean_config_and_log_dir ${csudo}rm -rf ${install_main_dir} diff --git a/source/client/src/clientEnv.c b/source/client/src/clientEnv.c index d63701c529..d5287d5062 100644 --- a/source/client/src/clientEnv.c +++ b/source/client/src/clientEnv.c @@ -26,6 +26,7 @@ #include "qworker.h" #include "scheduler.h" #include "tcache.h" +#include "tcompare.h" #include "tglobal.h" #include "thttp.h" #include "tmsg.h" @@ -35,7 +36,6 @@ #include "tsched.h" #include "ttime.h" #include "tversion.h" -#include "tcompare.h" #if defined(CUS_NAME) || defined(CUS_PROMPT) || defined(CUS_EMAIL) #include "cus_name.h" @@ -44,16 +44,16 @@ #define TSC_VAR_NOT_RELEASE 1 #define TSC_VAR_RELEASED 0 -#define ENV_JSON_FALSE_CHECK(c) \ - do { \ - if (!c) { \ - tscError("faild to add item to JSON object");\ - code = TSDB_CODE_TSC_FAIL_GENERATE_JSON; \ - goto _end; \ - } \ +#define ENV_JSON_FALSE_CHECK(c) \ + do { \ + if (!c) { \ + tscError("faild to add item to JSON object"); \ + code = TSDB_CODE_TSC_FAIL_GENERATE_JSON; \ + goto _end; \ + } \ } while (0) -#define ENV_ERR_RET(c,info) \ +#define ENV_ERR_RET(c, info) \ do { \ int32_t _code = c; \ if (_code != TSDB_CODE_SUCCESS) { \ @@ -94,109 +94,110 @@ static int32_t registerRequest(SRequestObj *pRequest, STscObj *pTscObj) { int32_t total = atomic_add_fetch_64((int64_t *)&pSummary->totalRequests, 1); int32_t currentInst = atomic_add_fetch_64((int64_t *)&pSummary->currentRequests, 1); tscDebug("0x%" PRIx64 " new Request from connObj:0x%" PRIx64 - ", current:%d, app current:%d, total:%d, reqId:0x%" PRIx64, + ", current:%d, app current:%d, total:%d, qid:0x%" PRIx64, pRequest->self, pRequest->pTscObj->id, num, currentInst, total, pRequest->requestId); } return code; } -static void concatStrings(SArray *list, char* buf, int size){ - int len = 0; - for(int i = 0; i < taosArrayGetSize(list); i++){ - char* db = taosArrayGet(list, i); +static void concatStrings(SArray *list, char *buf, int size) { + int len = 0; + for (int i = 0; i < taosArrayGetSize(list); i++) { + char *db = taosArrayGet(list, i); if (NULL == db) { tscError("get dbname failed, buf:%s", buf); break; } - char* dot = strchr(db, '.'); + char *dot = strchr(db, '.'); if (dot != NULL) { db = dot + 1; } - if (i != 0){ + if (i != 0) { (void)strcat(buf, ","); len += 1; } int ret = snprintf(buf + len, size - len, "%s", db); - if (ret < 0) { + if (ret < 0) { tscError("snprintf failed, buf:%s, ret:%d", buf, ret); break; } len += ret; - if (len >= size){ + if (len >= size) { tscInfo("dbList is truncated, buf:%s, len:%d", buf, len); break; } } } -static int32_t generateWriteSlowLog(STscObj *pTscObj, SRequestObj *pRequest, int32_t reqType, int64_t duration){ - cJSON* json = cJSON_CreateObject(); +static int32_t generateWriteSlowLog(STscObj *pTscObj, SRequestObj *pRequest, int32_t reqType, int64_t duration) { + cJSON *json = cJSON_CreateObject(); int32_t code = TSDB_CODE_SUCCESS; if (json == NULL) { tscError("[monitor] cJSON_CreateObject failed"); return TSDB_CODE_OUT_OF_MEMORY; } char clusterId[32] = {0}; - if (snprintf(clusterId, sizeof(clusterId), "%" PRId64, pTscObj->pAppInfo->clusterId) < 0){ + if (snprintf(clusterId, sizeof(clusterId), "%" PRId64, pTscObj->pAppInfo->clusterId) < 0) { tscError("failed to generate clusterId:%" PRId64, pTscObj->pAppInfo->clusterId); code = TSDB_CODE_FAILED; goto _end; } char startTs[32] = {0}; - if (snprintf(startTs, sizeof(startTs), "%" PRId64, pRequest->metric.start/1000) < 0){ - tscError("failed to generate startTs:%" PRId64, pRequest->metric.start/1000); + if (snprintf(startTs, sizeof(startTs), "%" PRId64, pRequest->metric.start / 1000) < 0) { + tscError("failed to generate startTs:%" PRId64, pRequest->metric.start / 1000); code = TSDB_CODE_FAILED; goto _end; } char requestId[32] = {0}; - if (snprintf(requestId, sizeof(requestId), "%" PRIu64, pRequest->requestId) < 0){ + if (snprintf(requestId, sizeof(requestId), "%" PRIu64, pRequest->requestId) < 0) { tscError("failed to generate requestId:%" PRIu64, pRequest->requestId); code = TSDB_CODE_FAILED; goto _end; } - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "cluster_id", cJSON_CreateString(clusterId))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "start_ts", cJSON_CreateString(startTs))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "request_id", cJSON_CreateString(requestId))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "query_time", cJSON_CreateNumber(duration/1000))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "code", cJSON_CreateNumber(pRequest->code))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "error_info", cJSON_CreateString(tstrerror(pRequest->code)))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "type", cJSON_CreateNumber(reqType))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "rows_num", cJSON_CreateNumber(pRequest->body.resInfo.numOfRows + pRequest->body.resInfo.totalRows))); - if(pRequest->sqlstr != NULL && strlen(pRequest->sqlstr) > pTscObj->pAppInfo->monitorParas.tsSlowLogMaxLen){ + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "cluster_id", cJSON_CreateString(clusterId))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "start_ts", cJSON_CreateString(startTs))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "request_id", cJSON_CreateString(requestId))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "query_time", cJSON_CreateNumber(duration / 1000))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "code", cJSON_CreateNumber(pRequest->code))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "error_info", cJSON_CreateString(tstrerror(pRequest->code)))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "type", cJSON_CreateNumber(reqType))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject( + json, "rows_num", cJSON_CreateNumber(pRequest->body.resInfo.numOfRows + pRequest->body.resInfo.totalRows))); + if (pRequest->sqlstr != NULL && strlen(pRequest->sqlstr) > pTscObj->pAppInfo->monitorParas.tsSlowLogMaxLen) { char tmp = pRequest->sqlstr[pTscObj->pAppInfo->monitorParas.tsSlowLogMaxLen]; pRequest->sqlstr[pTscObj->pAppInfo->monitorParas.tsSlowLogMaxLen] = '\0'; - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "sql", cJSON_CreateString(pRequest->sqlstr))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "sql", cJSON_CreateString(pRequest->sqlstr))); pRequest->sqlstr[pTscObj->pAppInfo->monitorParas.tsSlowLogMaxLen] = tmp; - }else{ - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "sql", cJSON_CreateString(pRequest->sqlstr))); + } else { + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "sql", cJSON_CreateString(pRequest->sqlstr))); } - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "user", cJSON_CreateString(pTscObj->user))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "user", cJSON_CreateString(pTscObj->user))); ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "process_name", cJSON_CreateString(appInfo.appName))); - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "ip", cJSON_CreateString(tsLocalFqdn))); + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "ip", cJSON_CreateString(tsLocalFqdn))); char pid[32] = {0}; - if (snprintf(pid, sizeof(pid), "%d", appInfo.pid) < 0){ + if (snprintf(pid, sizeof(pid), "%d", appInfo.pid) < 0) { tscError("failed to generate pid:%d", appInfo.pid); code = TSDB_CODE_FAILED; goto _end; } - ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "process_id", cJSON_CreateString(pid))); - if(pRequest->dbList != NULL){ + ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "process_id", cJSON_CreateString(pid))); + if (pRequest->dbList != NULL) { char dbList[1024] = {0}; concatStrings(pRequest->dbList, dbList, sizeof(dbList) - 1); ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "db", cJSON_CreateString(dbList))); - }else if(pRequest->pDb != NULL){ + } else if (pRequest->pDb != NULL) { ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "db", cJSON_CreateString(pRequest->pDb))); - }else{ + } else { ENV_JSON_FALSE_CHECK(cJSON_AddItemToObject(json, "db", cJSON_CreateString(""))); } - char* value = cJSON_PrintUnformatted(json); + char *value = cJSON_PrintUnformatted(json); MonitorSlowLogData data = {0}; data.clusterId = pTscObj->pAppInfo->clusterId; data.type = SLOW_LOG_WRITE; @@ -212,7 +213,7 @@ _end: return code; } -static bool checkSlowLogExceptDb(SRequestObj *pRequest, char* exceptDb) { +static bool checkSlowLogExceptDb(SRequestObj *pRequest, char *exceptDb) { if (pRequest->pDb != NULL) { return strcmp(pRequest->pDb, exceptDb) != 0; } @@ -227,7 +228,7 @@ static bool checkSlowLogExceptDb(SRequestObj *pRequest, char* exceptDb) { if (dot != NULL) { db = dot + 1; } - if(strcmp(db, exceptDb) == 0){ + if (strcmp(db, exceptDb) == 0) { return false; } } @@ -248,15 +249,14 @@ static void deregisterRequest(SRequestObj *pRequest) { int32_t reqType = SLOW_LOG_TYPE_OTHERS; int64_t duration = taosGetTimestampUs() - pRequest->metric.start; - tscDebug("0x%" PRIx64 " free Request from connObj: 0x%" PRIx64 ", reqId:0x%" PRIx64 + tscDebug("0x%" PRIx64 " free Request from connObj: 0x%" PRIx64 ", qid:0x%" PRIx64 " elapsed:%.2f ms, " "current:%d, app current:%d", pRequest->self, pTscObj->id, pRequest->requestId, duration / 1000.0, num, currentInst); if (TSDB_CODE_SUCCESS == nodesSimAcquireAllocator(pRequest->allocatorRefId)) { - if ((pRequest->pQuery && pRequest->pQuery->pRoot && - QUERY_NODE_VNODE_MODIFY_STMT == pRequest->pQuery->pRoot->type && - (0 == ((SVnodeModifyOpStmt *)pRequest->pQuery->pRoot)->sqlNodeType)) || + if ((pRequest->pQuery && pRequest->pQuery->pRoot && QUERY_NODE_VNODE_MODIFY_STMT == pRequest->pQuery->pRoot->type && + (0 == ((SVnodeModifyOpStmt *)pRequest->pQuery->pRoot)->sqlNodeType)) || QUERY_NODE_VNODE_MODIFY_STMT == pRequest->stmtType) { tscDebug("insert duration %" PRId64 "us: parseCost:%" PRId64 "us, ctgCost:%" PRId64 "us, analyseCost:%" PRId64 "us, planCost:%" PRId64 "us, exec:%" PRId64 "us", @@ -279,7 +279,7 @@ static void deregisterRequest(SRequestObj *pRequest) { } } - if(pTscObj->pAppInfo->monitorParas.tsEnableMonitor){ + if (pTscObj->pAppInfo->monitorParas.tsEnableMonitor) { if (QUERY_NODE_VNODE_MODIFY_STMT == pRequest->stmtType || QUERY_NODE_INSERT_STMT == pRequest->stmtType) { sqlReqLog(pTscObj->id, pRequest->killed, pRequest->code, MONITORSQLTYPEINSERT); } else if (QUERY_NODE_SELECT_STMT == pRequest->stmtType) { @@ -289,14 +289,15 @@ static void deregisterRequest(SRequestObj *pRequest) { } } - if ((duration >= pTscObj->pAppInfo->monitorParas.tsSlowLogThreshold * 1000000UL || duration >= pTscObj->pAppInfo->monitorParas.tsSlowLogThresholdTest * 1000000UL) && + if ((duration >= pTscObj->pAppInfo->monitorParas.tsSlowLogThreshold * 1000000UL || + duration >= pTscObj->pAppInfo->monitorParas.tsSlowLogThresholdTest * 1000000UL) && checkSlowLogExceptDb(pRequest, pTscObj->pAppInfo->monitorParas.tsSlowLogExceptDb)) { (void)atomic_add_fetch_64((int64_t *)&pActivity->numOfSlowQueries, 1); if (pTscObj->pAppInfo->monitorParas.tsSlowLogScope & reqType) { - taosPrintSlowLog("PID:%d, Conn:%u, QID:0x%" PRIx64 ", Start:%" PRId64 " us, Duration:%" PRId64 "us, SQL:%s", + taosPrintSlowLog("PID:%d, Conn:%u, qid:0x%" PRIx64 ", Start:%" PRId64 " us, Duration:%" PRId64 "us, SQL:%s", taosGetPId(), pTscObj->connId, pRequest->requestId, pRequest->metric.start, duration, pRequest->sqlstr); - if(pTscObj->pAppInfo->monitorParas.tsEnableMonitor){ + if (pTscObj->pAppInfo->monitorParas.tsEnableMonitor) { slowQueryLog(pTscObj->id, pRequest->killed, pRequest->code, duration); if (TSDB_CODE_SUCCESS != generateWriteSlowLog(pTscObj, pRequest, reqType, duration)) { tscError("failed to generate write slow log"); @@ -389,7 +390,7 @@ void destroyAllRequests(SHashObj *pRequests) { SRequestObj *pRequest = acquireRequest(*rid); if (pRequest) { destroyRequest(pRequest); - (void)releaseRequest(*rid); // ignore error + (void)releaseRequest(*rid); // ignore error } pIter = taosHashIterate(pRequests, pIter); @@ -404,7 +405,7 @@ void stopAllRequests(SHashObj *pRequests) { SRequestObj *pRequest = acquireRequest(*rid); if (pRequest) { taos_stop_query(pRequest); - (void)releaseRequest(*rid); // ignore error + (void)releaseRequest(*rid); // ignore error } pIter = taosHashIterate(pRequests, pIter); @@ -412,7 +413,7 @@ void stopAllRequests(SHashObj *pRequests) { } void destroyAppInst(void *info) { - SAppInstInfo* pAppInfo = *(SAppInstInfo**)info; + SAppInstInfo *pAppInfo = *(SAppInstInfo **)info; tscDebug("destroy app inst mgr %p", pAppInfo); int32_t code = taosThreadMutexLock(&appInfo.mutex); @@ -597,28 +598,28 @@ int32_t releaseRequest(int64_t rid) { return taosReleaseRef(clientReqRefPool, ri int32_t removeRequest(int64_t rid) { return taosRemoveRef(clientReqRefPool, rid); } /// return the most previous req ref id -int64_t removeFromMostPrevReq(SRequestObj* pRequest) { - int64_t mostPrevReqRefId = pRequest->self; - SRequestObj* pTmp = pRequest; +int64_t removeFromMostPrevReq(SRequestObj *pRequest) { + int64_t mostPrevReqRefId = pRequest->self; + SRequestObj *pTmp = pRequest; while (pTmp->relation.prevRefId) { pTmp = acquireRequest(pTmp->relation.prevRefId); if (pTmp) { mostPrevReqRefId = pTmp->self; - (void)releaseRequest(mostPrevReqRefId); // ignore error + (void)releaseRequest(mostPrevReqRefId); // ignore error } else { break; } } - (void)removeRequest(mostPrevReqRefId); // ignore error + (void)removeRequest(mostPrevReqRefId); // ignore error return mostPrevReqRefId; } void destroyNextReq(int64_t nextRefId) { if (nextRefId) { - SRequestObj* pObj = acquireRequest(nextRefId); + SRequestObj *pObj = acquireRequest(nextRefId); if (pObj) { - (void)releaseRequest(nextRefId); // ignore error - (void)releaseRequest(nextRefId); // ignore error + (void)releaseRequest(nextRefId); // ignore error + (void)releaseRequest(nextRefId); // ignore error } } } @@ -638,7 +639,7 @@ void destroySubRequests(SRequestObj *pRequest) { pTmp = acquireRequest(tmpRefId); if (pTmp) { pReqList[++reqIdx] = pTmp; - (void)releaseRequest(tmpRefId); // ignore error + (void)releaseRequest(tmpRefId); // ignore error } else { tscError("prev req ref 0x%" PRIx64 " is not there", tmpRefId); break; @@ -646,7 +647,7 @@ void destroySubRequests(SRequestObj *pRequest) { } for (int32_t i = reqIdx; i >= 0; i--) { - (void)removeRequest(pReqList[i]->self); // ignore error + (void)removeRequest(pReqList[i]->self); // ignore error } tmpRefId = pRequest->relation.nextRefId; @@ -654,8 +655,8 @@ void destroySubRequests(SRequestObj *pRequest) { pTmp = acquireRequest(tmpRefId); if (pTmp) { tmpRefId = pTmp->relation.nextRefId; - (void)removeRequest(pTmp->self); // ignore error - (void)releaseRequest(pTmp->self); // ignore error + (void)removeRequest(pTmp->self); // ignore error + (void)releaseRequest(pTmp->self); // ignore error } else { tscError("next req ref 0x%" PRIx64 " is not there", tmpRefId); break; @@ -758,7 +759,7 @@ void stopAllQueries(SRequestObj *pRequest) { for (int32_t i = reqIdx; i >= 0; i--) { taosStopQueryImpl(pReqList[i]); - (void)releaseRequest(pReqList[i]->self); // ignore error + (void)releaseRequest(pReqList[i]->self); // ignore error } taosStopQueryImpl(pRequest); @@ -769,7 +770,7 @@ void stopAllQueries(SRequestObj *pRequest) { if (pTmp) { tmpRefId = pTmp->relation.nextRefId; taosStopQueryImpl(pTmp); - (void)releaseRequest(pTmp->self); // ignore error + (void)releaseRequest(pTmp->self); // ignore error } else { tscError("next req ref 0x%" PRIx64 " is not there", tmpRefId); break; @@ -929,7 +930,8 @@ void taos_init_imp(void) { appInfo.pid = taosGetPId(); appInfo.startTime = taosGetTimestampMs(); appInfo.pInstMap = taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_ENTRY_LOCK); - appInfo.pInstMapByClusterId = taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_ENTRY_LOCK); + appInfo.pInstMapByClusterId = + taosHashInit(4, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_ENTRY_LOCK); if (NULL == appInfo.pInstMap || NULL == appInfo.pInstMapByClusterId) { tscError("failed to allocate memory when init appInfo"); tscInitRes = TSDB_CODE_OUT_OF_MEMORY; @@ -993,8 +995,8 @@ int taos_init() { return tscInitRes; } -const char* getCfgName(TSDB_OPTION option) { - const char* name = NULL; +const char *getCfgName(TSDB_OPTION option) { + const char *name = NULL; switch (option) { case TSDB_OPTION_SHELL_ACTIVITY_TIMER: @@ -1025,12 +1027,12 @@ int taos_options_imp(TSDB_OPTION option, const char *str) { char newstr[PATH_MAX]; int len = strlen(str); if (len > 1 && str[0] != '"' && str[0] != '\'') { - if (len + 2 >= PATH_MAX) { + if (len + 2 >= PATH_MAX) { tscError("Too long path %s", str); return -1; } newstr[0] = '"'; - (void)memcpy(newstr+1, str, len); + (void)memcpy(newstr + 1, str, len); newstr[len + 1] = '"'; newstr[len + 2] = '\0'; str = newstr; diff --git a/source/client/src/clientImpl.c b/source/client/src/clientImpl.c index 563d70ab08..ab007deb7f 100644 --- a/source/client/src/clientImpl.c +++ b/source/client/src/clientImpl.c @@ -15,8 +15,8 @@ #include "cJSON.h" #include "clientInt.h" -#include "clientMonitor.h" #include "clientLog.h" +#include "clientMonitor.h" #include "command.h" #include "scheduler.h" #include "tdatablock.h" @@ -28,7 +28,7 @@ #include "tref.h" #include "tsched.h" #include "tversion.h" -static int32_t initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* pEpSet); +static int32_t initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* pEpSet); static int32_t buildConnectMsg(SRequestObj* pRequest, SMsgSendInfo** pMsgSendInfo); static bool stringLengthCheck(const char* str, size_t maxsize) { @@ -68,15 +68,13 @@ bool chkRequestKilled(void* param) { return killed; } -void cleanupAppInfo() { - taosHashCleanup(appInfo.pInstMap); -} +void cleanupAppInfo() { taosHashCleanup(appInfo.pInstMap); } static int32_t taosConnectImpl(const char* user, const char* auth, const char* db, __taos_async_fn_t fp, void* param, SAppInstInfo* pAppInfo, int connType, STscObj** pTscObj); int32_t taos_connect_internal(const char* ip, const char* user, const char* pass, const char* auth, const char* db, - uint16_t port, int connType, STscObj** pObj) { + uint16_t port, int connType, STscObj** pObj) { TSC_ERR_RET(taos_init()); if (!validateUserName(user)) { TSC_ERR_RET(TSDB_CODE_TSC_INVALID_USER_LENGTH); @@ -126,7 +124,7 @@ int32_t taos_connect_internal(const char* ip, const char* user, const char* pass } SAppInstInfo** pInst = NULL; - int32_t code = taosThreadMutexLock(&appInfo.mutex); + int32_t code = taosThreadMutexLock(&appInfo.mutex); if (TSDB_CODE_SUCCESS != code) { tscError("failed to lock app info, code:%s", tstrerror(TAOS_SYSTEM_ERROR(code))); TSC_ERR_RET(code); @@ -176,25 +174,29 @@ int32_t taos_connect_internal(const char* ip, const char* user, const char* pass _return: - code = taosThreadMutexUnlock(&appInfo.mutex); if (TSDB_CODE_SUCCESS != code) { - tscError("failed to unlock app info, code:%s", tstrerror(TAOS_SYSTEM_ERROR(code))); + (void)taosThreadMutexUnlock(&appInfo.mutex); + taosMemoryFreeClear(key); return code; + } else { + code = taosThreadMutexUnlock(&appInfo.mutex); + taosMemoryFreeClear(key); + if (TSDB_CODE_SUCCESS != code) { + tscError("failed to unlock app info, code:%s", tstrerror(TAOS_SYSTEM_ERROR(code))); + return code; + } + return taosConnectImpl(user, &secretEncrypt[0], localDb, NULL, NULL, *pInst, connType, pObj); } - - taosMemoryFreeClear(key); - - return taosConnectImpl(user, &secretEncrypt[0], localDb, NULL, NULL, *pInst, connType, pObj); } -//SAppInstInfo* getAppInstInfo(const char* clusterKey) { -// SAppInstInfo** ppAppInstInfo = taosHashGet(appInfo.pInstMap, clusterKey, strlen(clusterKey)); -// if (ppAppInstInfo != NULL && *ppAppInstInfo != NULL) { -// return *ppAppInstInfo; -// } else { -// return NULL; -// } -//} +// SAppInstInfo* getAppInstInfo(const char* clusterKey) { +// SAppInstInfo** ppAppInstInfo = taosHashGet(appInfo.pInstMap, clusterKey, strlen(clusterKey)); +// if (ppAppInstInfo != NULL && *ppAppInstInfo != NULL) { +// return *ppAppInstInfo; +// } else { +// return NULL; +// } +// } void freeQueryParam(SSyncQueryParam* param) { if (param == NULL) return; @@ -230,7 +232,7 @@ int32_t buildRequest(uint64_t connId, const char* sql, int sqlLen, void* param, int32_t err = taosHashPut(pTscObj->pRequests, &(*pRequest)->self, sizeof((*pRequest)->self), &(*pRequest)->self, sizeof((*pRequest)->self)); if (err) { - tscError("%" PRId64 " failed to add to request container, reqId:0x%" PRIx64 ", conn:%" PRId64 ", %s", + tscError("%" PRId64 " failed to add to request container, qid:0x%" PRIx64 ", conn:%" PRId64 ", %s", (*pRequest)->self, (*pRequest)->requestId, pTscObj->id, sql); destroyRequest(*pRequest); *pRequest = NULL; @@ -241,7 +243,7 @@ int32_t buildRequest(uint64_t connId, const char* sql, int sqlLen, void* param, if (tsQueryUseNodeAllocator && !qIsInsertValuesSql((*pRequest)->sqlstr, (*pRequest)->sqlLen)) { if (TSDB_CODE_SUCCESS != nodesCreateAllocator((*pRequest)->requestId, tsQueryNodeChunkSize, &((*pRequest)->allocatorRefId))) { - tscError("%" PRId64 " failed to create node allocator, reqId:0x%" PRIx64 ", conn:%" PRId64 ", %s", + tscError("%" PRId64 " failed to create node allocator, qid:0x%" PRIx64 ", conn:%" PRId64 ", %s", (*pRequest)->self, (*pRequest)->requestId, pTscObj->id, sql); destroyRequest(*pRequest); *pRequest = NULL; @@ -249,7 +251,7 @@ int32_t buildRequest(uint64_t connId, const char* sql, int sqlLen, void* param, } } - tscDebugL("0x%" PRIx64 " SQL: %s, reqId:0x%" PRIx64, (*pRequest)->self, (*pRequest)->sqlstr, (*pRequest)->requestId); + tscDebugL("0x%" PRIx64 " SQL: %s, qid:0x%" PRIx64, (*pRequest)->self, (*pRequest)->sqlstr, (*pRequest)->requestId); return TSDB_CODE_SUCCESS; } @@ -363,10 +365,10 @@ void asyncExecLocalCmd(SRequestObj* pRequest, SQuery* pQuery) { if (pRequest->code != TSDB_CODE_SUCCESS) { pResultInfo->numOfRows = 0; - tscError("0x%" PRIx64 " fetch results failed, code:%s, reqId:0x%" PRIx64, pRequest->self, tstrerror(code), + tscError("0x%" PRIx64 " fetch results failed, code:%s, qid:0x%" PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); } else { - tscDebug("0x%" PRIx64 " fetch results, numOfRows:%" PRId64 " total Rows:%" PRId64 ", complete:%d, reqId:0x%" PRIx64, + tscDebug("0x%" PRIx64 " fetch results, numOfRows:%" PRId64 " total Rows:%" PRId64 ", complete:%d, qid:0x%" PRIx64, pRequest->self, pResultInfo->numOfRows, pResultInfo->totalRows, pResultInfo->completed, pRequest->requestId); } @@ -432,7 +434,7 @@ int32_t updateQnodeList(SAppInstInfo* pInfo, SArray* pNodeList) { return TSDB_CODE_SUCCESS; } -int32_t qnodeRequired(SRequestObj* pRequest, bool *required) { +int32_t qnodeRequired(SRequestObj* pRequest, bool* required) { if (QUERY_POLICY_VNODE == tsQueryPolicy || QUERY_POLICY_CLIENT == tsQueryPolicy) { *required = false; return TSDB_CODE_SUCCESS; @@ -554,7 +556,7 @@ int32_t buildVnodePolicyNodeList(SRequestObj* pRequest, SArray** pNodeList, SArr if (NULL == nodeList) { return terrno; } - char* policy = (tsQueryPolicy == QUERY_POLICY_VNODE) ? "vnode" : "client"; + char* policy = (tsQueryPolicy == QUERY_POLICY_VNODE) ? "vnode" : "client"; int32_t dbNum = taosArrayGetSize(pDbVgList); for (int32_t i = 0; i < dbNum; ++i) { @@ -568,7 +570,7 @@ int32_t buildVnodePolicyNodeList(SRequestObj* pRequest, SArray** pNodeList, SArr } for (int32_t j = 0; j < vgNum; ++j) { - SVgroupInfo* pInfo = taosArrayGet(pVg, j); + SVgroupInfo* pInfo = taosArrayGet(pVg, j); if (NULL == pInfo) { taosArrayDestroy(nodeList); return TSDB_CODE_OUT_OF_RANGE; @@ -972,8 +974,8 @@ int32_t handleQueryExecRsp(SRequestObj* pRequest) { break; } default: - tscError("0x%" PRIx64 ", invalid exec result for request type %d, reqId:0x%" PRIx64, pRequest->self, - pRequest->type, pRequest->requestId); + tscError("0x%" PRIx64 ", invalid exec result for request type %d, qid:0x%" PRIx64, pRequest->self, pRequest->type, + pRequest->requestId); code = TSDB_CODE_APP_ERROR; } @@ -1017,12 +1019,12 @@ void returnToUser(SRequestObj* pRequest) { (void)releaseRequest(pRequest->relation.userRefId); return; } else { - tscError("0x%" PRIx64 ", user ref 0x%" PRIx64 " is not there, reqId:0x%" PRIx64, pRequest->self, + tscError("0x%" PRIx64 ", user ref 0x%" PRIx64 " is not there, qid:0x%" PRIx64, pRequest->self, pRequest->relation.userRefId, pRequest->requestId); } } -static int32_t createResultBlock(TAOS_RES* pRes, int32_t numOfRows, SSDataBlock**pBlock) { +static int32_t createResultBlock(TAOS_RES* pRes, int32_t numOfRows, SSDataBlock** pBlock) { int64_t lastTs = 0; TAOS_FIELD* pResFields = taos_fetch_fields(pRes); int32_t numOfFields = taos_num_fields(pRes); @@ -1032,7 +1034,7 @@ static int32_t createResultBlock(TAOS_RES* pRes, int32_t numOfRows, SSDataBlock* return code; } - for(int32_t i = 0; i < numOfFields; ++i) { + for (int32_t i = 0; i < numOfFields; ++i) { SColumnInfoData colInfoData = createColumnInfoData(pResFields[i].type, pResFields[i].bytes, i + 1); code = blockDataAppendColInfo(*pBlock, &colInfoData); if (TSDB_CODE_SUCCESS != code) { @@ -1049,12 +1051,16 @@ static int32_t createResultBlock(TAOS_RES* pRes, int32_t numOfRows, SSDataBlock* for (int32_t i = 0; i < numOfRows; ++i) { TAOS_ROW pRow = taos_fetch_row(pRes); + if(NULL == pRow[0] || NULL == pRow[1] || NULL == pRow[2]) { + tscError("invalid data from vnode"); + return TSDB_CODE_TSC_INTERNAL_ERROR; + } int64_t ts = *(int64_t*)pRow[0]; if (lastTs < ts) { lastTs = ts; } - for(int32_t j = 0; j < numOfFields; ++j) { + for (int32_t j = 0; j < numOfFields; ++j) { SColumnInfoData* pColInfoData = taosArrayGet((*pBlock)->pDataBlock, j); code = colDataSetVal(pColInfoData, i, pRow[j], false); if (TSDB_CODE_SUCCESS != code) { @@ -1069,7 +1075,7 @@ static int32_t createResultBlock(TAOS_RES* pRes, int32_t numOfRows, SSDataBlock* (*pBlock)->info.window.ekey = lastTs; (*pBlock)->info.rows = numOfRows; - tscDebug("lastKey:%"PRId64" numOfRows:%d from all vgroups", lastTs, numOfRows); + tscDebug("lastKey:%" PRId64 " numOfRows:%d from all vgroups", lastTs, numOfRows); return TSDB_CODE_SUCCESS; } @@ -1082,7 +1088,7 @@ void postSubQueryFetchCb(void* param, TAOS_RES* res, int32_t rowNum) { SSDataBlock* pBlock = NULL; if (TSDB_CODE_SUCCESS != createResultBlock(res, rowNum, &pBlock)) { - tscError("0x%" PRIx64 ", create result block failed, reqId:0x%" PRIx64, pRequest->self, pRequest->requestId); + tscError("0x%" PRIx64 ", create result block failed, qid:0x%" PRIx64, pRequest->self, pRequest->requestId); return; } @@ -1091,7 +1097,7 @@ void postSubQueryFetchCb(void* param, TAOS_RES* res, int32_t rowNum) { continuePostSubQuery(pNextReq, pBlock); (void)releaseRequest(pRequest->relation.nextRefId); } else { - tscError("0x%" PRIx64 ", next req ref 0x%" PRIx64 " is not there, reqId:0x%" PRIx64, pRequest->self, + tscError("0x%" PRIx64 ", next req ref 0x%" PRIx64 " is not there, qid:0x%" PRIx64, pRequest->self, pRequest->relation.nextRefId, pRequest->requestId); } @@ -1110,7 +1116,7 @@ void handlePostSubQuery(SSqlCallbackWrapper* pWrapper) { continuePostSubQuery(pNextReq, NULL); (void)releaseRequest(pRequest->relation.nextRefId); } else { - tscError("0x%" PRIx64 ", next req ref 0x%" PRIx64 " is not there, reqId:0x%" PRIx64, pRequest->self, + tscError("0x%" PRIx64 ", next req ref 0x%" PRIx64 " is not there, qid:0x%" PRIx64, pRequest->self, pRequest->relation.nextRefId, pRequest->requestId); } } @@ -1141,11 +1147,11 @@ void schedulerExecCb(SExecResult* pResult, void* param, int32_t code) { } taosMemoryFree(pResult); - tscDebug("0x%" PRIx64 " enter scheduler exec cb, code:%s, reqId:0x%" PRIx64, pRequest->self, tstrerror(code), + tscDebug("0x%" PRIx64 " enter scheduler exec cb, code:%s, qid:0x%" PRIx64, pRequest->self, tstrerror(code), pRequest->requestId); if (code != TSDB_CODE_SUCCESS && NEED_CLIENT_HANDLE_ERROR(code) && pRequest->sqlstr != NULL) { - tscDebug("0x%" PRIx64 " client retry to handle the error, code:%s, tryCount:%d, reqId:0x%" PRIx64, pRequest->self, + tscDebug("0x%" PRIx64 " client retry to handle the error, code:%s, tryCount:%d, qid:0x%" PRIx64, pRequest->self, tstrerror(code), pRequest->retry, pRequest->requestId); (void)removeMeta(pTscObj, pRequest->targetTableList, IS_VIEW_REQUEST(pRequest->type)); restartAsyncQuery(pRequest, code); @@ -1215,7 +1221,7 @@ SRequestObj* launchQueryImpl(SRequestObj* pRequest, SQuery* pQuery, bool keepQue } break; case QUERY_EXEC_MODE_SCHEDULE: { - SArray* pMnodeList = taosArrayInit(4, sizeof(SQueryNodeLoad)); + SArray* pMnodeList = taosArrayInit(4, sizeof(SQueryNodeLoad)); if (NULL == pMnodeList) { code = terrno; break; @@ -1533,7 +1539,7 @@ int32_t initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* p } int32_t taosConnectImpl(const char* user, const char* auth, const char* db, __taos_async_fn_t fp, void* param, - SAppInstInfo* pAppInfo, int connType, STscObj** pTscObj) { + SAppInstInfo* pAppInfo, int connType, STscObj** pTscObj) { *pTscObj = NULL; int32_t code = createTscObj(user, auth, db, connType, pAppInfo, pTscObj); if (TSDB_CODE_SUCCESS != code) { @@ -1560,7 +1566,8 @@ int32_t taosConnectImpl(const char* user, const char* auth, const char* db, __ta } int64_t transporterId = 0; - code = asyncSendMsgToServer((*pTscObj)->pAppInfo->pTransporter, &(*pTscObj)->pAppInfo->mgmtEp.epSet, &transporterId, body); + code = asyncSendMsgToServer((*pTscObj)->pAppInfo->pTransporter, &(*pTscObj)->pAppInfo->mgmtEp.epSet, &transporterId, + body); if (TSDB_CODE_SUCCESS != code) { destroyTscObj(*pTscObj); tscError("failed to send connect msg to server, code:%s", tstrerror(code)); @@ -1577,7 +1584,7 @@ int32_t taosConnectImpl(const char* user, const char* auth, const char* db, __ta *pTscObj = NULL; return terrno; } else { - tscDebug("0x%" PRIx64 " connection is opening, connId:%u, dnodeConn:%p, reqId:0x%" PRIx64, (*pTscObj)->id, + tscDebug("0x%" PRIx64 " connection is opening, connId:%u, dnodeConn:%p, qid:0x%" PRIx64, (*pTscObj)->id, (*pTscObj)->connId, (*pTscObj)->pAppInfo->pTransporter, pRequest->requestId); destroyRequest(pRequest); } @@ -1708,7 +1715,7 @@ int32_t doProcessMsgFromServer(void* param) { char tbuf[40] = {0}; TRACE_TO_STR(trace, tbuf); - tscDebug("processMsgFromServer handle %p, message: %s, size:%d, code: %s, gtid: %s", pMsg->info.handle, + tscDebug("processMsgFromServer handle %p, message: %s, size:%d, code: %s, qid:%s", pMsg->info.handle, TMSG_INFO(pMsg->msgType), pMsg->contLen, tstrerror(pMsg->code), tbuf); if (pSendInfo->requestObjRefId != 0) { @@ -1792,6 +1799,7 @@ void processMsgFromServer(void* parent, SRpcMsg* pMsg, SEpSet* pEpSet) { AsyncArg* arg = taosMemoryCalloc(1, sizeof(AsyncArg)); if (NULL == arg) { pMsg->code = TSDB_CODE_OUT_OF_MEMORY; + taosMemoryFree(tEpSet); rpcFreeCont(pMsg->pCont); destroySendMsgInfo(pMsg->info.ahandle); return; @@ -1820,7 +1828,7 @@ TAOS* taos_connect_auth(const char* ip, const char* user, const char* auth, cons } STscObj* pObj = NULL; - int32_t code = taos_connect_internal(ip, user, NULL, auth, db, port, CONN_TYPE__QUERY, &pObj); + int32_t code = taos_connect_internal(ip, user, NULL, auth, db, port, CONN_TYPE__QUERY, &pObj); if (TSDB_CODE_SUCCESS == code) { int64_t* rid = taosMemoryCalloc(1, sizeof(int64_t)); if (NULL == rid) { @@ -1890,7 +1898,7 @@ void* doFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) } SReqResultInfo* pResInfo = &pRequest->body.resInfo; - SSchedulerReq req = { .syncReq = true, .pFetchRes = (void**)&pResInfo->pData }; + SSchedulerReq req = {.syncReq = true, .pFetchRes = (void**)&pResInfo->pData}; pRequest->code = schedulerFetchRows(pRequest->body.queryJob, &req); if (pRequest->code != TSDB_CODE_SUCCESS) { @@ -1905,7 +1913,7 @@ void* doFetchRows(SRequestObj* pRequest, bool setupOneRowPtr, bool convertUcs4) return NULL; } - tscDebug("0x%" PRIx64 " fetch results, numOfRows:%" PRId64 " total Rows:%" PRId64 ", complete:%d, reqId:0x%" PRIx64, + tscDebug("0x%" PRIx64 " fetch results, numOfRows:%" PRId64 " total Rows:%" PRId64 ", complete:%d, qid:0x%" PRIx64, pRequest->self, pResInfo->numOfRows, pResInfo->totalRows, pResInfo->completed, pRequest->requestId); STscObj* pTscObj = pRequest->pTscObj; @@ -2033,7 +2041,7 @@ int32_t getVersion1BlockMetaSize(const char* p, int32_t numOfCols) { } static int32_t estimateJsonLen(SReqResultInfo* pResultInfo, int32_t numOfCols, int32_t numOfRows) { - char* p = (char*)pResultInfo->pData; + char* p = (char*)pResultInfo->pData; int32_t blockVersion = *(int32_t*)p; // | version | total length | total rows | total columns | flag seg| block group id | column schema | each column @@ -2298,7 +2306,7 @@ int32_t setResultDataPtr(SReqResultInfo* pResultInfo, TAOS_FIELD* pFields, int32 char* pStart = p; for (int32_t i = 0; i < numOfCols; ++i) { - if(blockVersion == BLOCK_VERSION_1){ + if (blockVersion == BLOCK_VERSION_1) { colLength[i] = htonl(colLength[i]); } if (colLength[i] >= dataLen) { @@ -2733,7 +2741,8 @@ void syncQueryFn(void* param, void* res, int32_t code) { (void)tsem_post(&pParam->sem); } -void taosAsyncQueryImpl(uint64_t connId, const char* sql, __taos_async_fn_t fp, void* param, bool validateOnly, int8_t source) { +void taosAsyncQueryImpl(uint64_t connId, const char* sql, __taos_async_fn_t fp, void* param, bool validateOnly, + int8_t source) { if (sql == NULL || NULL == fp) { terrno = TSDB_CODE_INVALID_PARA; if (fp) { @@ -2854,7 +2863,7 @@ static void fetchCallback(void* pResult, void* param, int32_t code) { SReqResultInfo* pResultInfo = &pRequest->body.resInfo; - tscDebug("0x%" PRIx64 " enter scheduler fetch cb, code:%d - %s, reqId:0x%" PRIx64, pRequest->self, code, + tscDebug("0x%" PRIx64 " enter scheduler fetch cb, code:%d - %s, qid:0x%" PRIx64, pRequest->self, code, tstrerror(code), pRequest->requestId); pResultInfo->pData = pResult; @@ -2877,10 +2886,10 @@ static void fetchCallback(void* pResult, void* param, int32_t code) { setQueryResultFromRsp(pResultInfo, (const SRetrieveTableRsp*)pResultInfo->pData, pResultInfo->convertUcs4); if (pRequest->code != TSDB_CODE_SUCCESS) { pResultInfo->numOfRows = 0; - tscError("0x%" PRIx64 " fetch results failed, code:%s, reqId:0x%" PRIx64, pRequest->self, tstrerror(pRequest->code), + tscError("0x%" PRIx64 " fetch results failed, code:%s, qid:0x%" PRIx64, pRequest->self, tstrerror(pRequest->code), pRequest->requestId); } else { - tscDebug("0x%" PRIx64 " fetch results, numOfRows:%" PRId64 " total Rows:%" PRId64 ", complete:%d, reqId:0x%" PRIx64, + tscDebug("0x%" PRIx64 " fetch results, numOfRows:%" PRId64 " total Rows:%" PRId64 ", complete:%d, qid:0x%" PRIx64, pRequest->self, pResultInfo->numOfRows, pResultInfo->totalRows, pResultInfo->completed, pRequest->requestId); @@ -2932,7 +2941,7 @@ void taosAsyncFetchImpl(SRequestObj* pRequest, __taos_async_fn_t fp, void* param int32_t code = schedulerFetchRows(pRequest->body.queryJob, &req); if (TSDB_CODE_SUCCESS != code) { tscError("0x%" PRIx64 " failed to schedule fetch rows", pRequest->requestId); - pRequest->body.fetchFp(param, pRequest, code); + // pRequest->body.fetchFp(param, pRequest, code); } } diff --git a/source/client/src/clientMain.c b/source/client/src/clientMain.c index de56a4844a..5732df8013 100644 --- a/source/client/src/clientMain.c +++ b/source/client/src/clientMain.c @@ -16,19 +16,19 @@ #include "catalog.h" #include "clientInt.h" #include "clientLog.h" -#include "clientStmt.h" #include "clientMonitor.h" +#include "clientStmt.h" #include "functionMgt.h" #include "os.h" #include "query.h" #include "scheduler.h" +#include "tcompare.h" #include "tdatablock.h" #include "tglobal.h" #include "tmsg.h" #include "tref.h" #include "trpc.h" #include "version.h" -#include "tcompare.h" #define TSC_VAR_NOT_RELEASE 1 #define TSC_VAR_RELEASED 0 @@ -120,7 +120,7 @@ TAOS *taos_connect(const char *ip, const char *user, const char *pass, const cha } STscObj *pObj = NULL; - int32_t code = taos_connect_internal(ip, user, pass, NULL, db, port, CONN_TYPE__QUERY, &pObj); + int32_t code = taos_connect_internal(ip, user, pass, NULL, db, port, CONN_TYPE__QUERY, &pObj); if (TSDB_CODE_SUCCESS == code) { int64_t *rid = taosMemoryCalloc(1, sizeof(int64_t)); if (NULL == rid) { @@ -183,15 +183,15 @@ int taos_set_notify_cb(TAOS *taos, __taos_notify_fn_t fp, void *param, int type) return 0; } -typedef struct SFetchWhiteListInfo{ - int64_t connId; +typedef struct SFetchWhiteListInfo { + int64_t connId; __taos_async_whitelist_fn_t userCbFn; - void* userParam; + void *userParam; } SFetchWhiteListInfo; -int32_t fetchWhiteListCallbackFn(void* param, SDataBuf* pMsg, int32_t code) { - SFetchWhiteListInfo* pInfo = (SFetchWhiteListInfo*)param; - TAOS* taos = &pInfo->connId; +int32_t fetchWhiteListCallbackFn(void *param, SDataBuf *pMsg, int32_t code) { + SFetchWhiteListInfo *pInfo = (SFetchWhiteListInfo *)param; + TAOS *taos = &pInfo->connId; if (code != TSDB_CODE_SUCCESS) { pInfo->userCbFn(pInfo->userParam, code, taos, 0, NULL); taosMemoryFree(pMsg->pData); @@ -209,7 +209,7 @@ int32_t fetchWhiteListCallbackFn(void* param, SDataBuf* pMsg, int32_t code) { return terrno; } - uint64_t* pWhiteLists = taosMemoryMalloc(wlRsp.numWhiteLists * sizeof(uint64_t)); + uint64_t *pWhiteLists = taosMemoryMalloc(wlRsp.numWhiteLists * sizeof(uint64_t)); if (pWhiteLists == NULL) { taosMemoryFree(pMsg->pData); taosMemoryFree(pMsg->pEpSet); @@ -238,7 +238,7 @@ void taos_fetch_whitelist_a(TAOS *taos, __taos_async_whitelist_fn_t fp, void *pa return; } - int64_t connId = *(int64_t*)taos; + int64_t connId = *(int64_t *)taos; STscObj *pTsc = acquireTscObj(connId); if (NULL == pTsc) { @@ -255,7 +255,7 @@ void taos_fetch_whitelist_a(TAOS *taos, __taos_async_whitelist_fn_t fp, void *pa return; } - void* pReq = taosMemoryMalloc(msgLen); + void *pReq = taosMemoryMalloc(msgLen); if (pReq == NULL) { fp(param, TSDB_CODE_OUT_OF_MEMORY, taos, 0, NULL); releaseTscObj(connId); @@ -269,7 +269,7 @@ void taos_fetch_whitelist_a(TAOS *taos, __taos_async_whitelist_fn_t fp, void *pa return; } - SFetchWhiteListInfo* pParam = taosMemoryMalloc(sizeof(SFetchWhiteListInfo)); + SFetchWhiteListInfo *pParam = taosMemoryMalloc(sizeof(SFetchWhiteListInfo)); if (pParam == NULL) { fp(param, TSDB_CODE_OUT_OF_MEMORY, taos, 0, NULL); taosMemoryFree(pReq); @@ -280,9 +280,9 @@ void taos_fetch_whitelist_a(TAOS *taos, __taos_async_whitelist_fn_t fp, void *pa pParam->connId = connId; pParam->userCbFn = fp; pParam->userParam = param; - SMsgSendInfo* pSendInfo = taosMemoryCalloc(1, sizeof(SMsgSendInfo)); + SMsgSendInfo *pSendInfo = taosMemoryCalloc(1, sizeof(SMsgSendInfo)); if (pSendInfo == NULL) { - fp(param, TSDB_CODE_OUT_OF_MEMORY, taos, 0, NULL); + fp(param, TSDB_CODE_OUT_OF_MEMORY, taos, 0, NULL); taosMemoryFree(pParam); taosMemoryFree(pReq); releaseTscObj(connId); @@ -297,7 +297,7 @@ void taos_fetch_whitelist_a(TAOS *taos, __taos_async_whitelist_fn_t fp, void *pa pSendInfo->msgType = TDMT_MND_GET_USER_WHITELIST; int64_t transportId = 0; - SEpSet epSet = getEpSet_s(&pTsc->pAppInfo->mgmtEp); + SEpSet epSet = getEpSet_s(&pTsc->pAppInfo->mgmtEp); if (TSDB_CODE_SUCCESS != asyncSendMsgToServer(pTsc->pAppInfo->pTransporter, &epSet, &transportId, pSendInfo)) { tscWarn("failed to async send msg to server"); } @@ -443,7 +443,7 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) { return NULL; } - if(pRequest->inCallback) { + if (pRequest->inCallback) { tscError("can not call taos_fetch_row before query callback ends."); terrno = TSDB_CODE_TSC_INVALID_OPERATION; return NULL; @@ -454,7 +454,7 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) { SMqRspObj *msg = ((SMqRspObj *)res); SReqResultInfo *pResultInfo = NULL; if (msg->common.resIter == -1) { - if(tmqGetNextResInfo(res, true, &pResultInfo) != 0){ + if (tmqGetNextResInfo(res, true, &pResultInfo) != 0) { return NULL; } } else { @@ -466,7 +466,7 @@ TAOS_ROW taos_fetch_row(TAOS_RES *res) { pResultInfo->current += 1; return pResultInfo->row; } else { - if (tmqGetNextResInfo(res, true, &pResultInfo) != 0){ + if (tmqGetNextResInfo(res, true, &pResultInfo) != 0) { return NULL; } @@ -540,22 +540,23 @@ int taos_print_row(char *str, TAOS_ROW row, TAOS_FIELD *fields, int num_fields) len += sprintf(str + len, "%lf", dv); } break; - case TSDB_DATA_TYPE_VARBINARY:{ - void* data = NULL; + case TSDB_DATA_TYPE_VARBINARY: { + void *data = NULL; uint32_t size = 0; - int32_t charLen = varDataLen((char *)row[i] - VARSTR_HEADER_SIZE); - if(taosAscii2Hex(row[i], charLen, &data, &size) < 0){ + int32_t charLen = varDataLen((char *)row[i] - VARSTR_HEADER_SIZE); + if (taosAscii2Hex(row[i], charLen, &data, &size) < 0) { break; } (void)memcpy(str + len, data, size); len += size; taosMemoryFree(data); - }break; + } break; case TSDB_DATA_TYPE_BINARY: case TSDB_DATA_TYPE_NCHAR: case TSDB_DATA_TYPE_GEOMETRY: { int32_t charLen = varDataLen((char *)row[i] - VARSTR_HEADER_SIZE); - if (fields[i].type == TSDB_DATA_TYPE_BINARY || fields[i].type == TSDB_DATA_TYPE_VARBINARY || fields[i].type == TSDB_DATA_TYPE_GEOMETRY) { + if (fields[i].type == TSDB_DATA_TYPE_BINARY || fields[i].type == TSDB_DATA_TYPE_VARBINARY || + fields[i].type == TSDB_DATA_TYPE_GEOMETRY) { if (charLen > fields[i].bytes || charLen < 0) { tscError("taos_print_row error binary. charLen:%d, fields[i].bytes:%d", charLen, fields[i].bytes); break; @@ -664,7 +665,8 @@ const char *taos_get_client_info() { return version; } // return int32_t int taos_affected_rows(TAOS_RES *res) { - if (res == NULL || TD_RES_TMQ(res) || TD_RES_TMQ_META(res) || TD_RES_TMQ_METADATA(res) || TD_RES_TMQ_BATCH_META(res)) { + if (res == NULL || TD_RES_TMQ(res) || TD_RES_TMQ_META(res) || TD_RES_TMQ_METADATA(res) || + TD_RES_TMQ_BATCH_META(res)) { return 0; } @@ -675,7 +677,8 @@ int taos_affected_rows(TAOS_RES *res) { // return int64_t int64_t taos_affected_rows64(TAOS_RES *res) { - if (res == NULL || TD_RES_TMQ(res) || TD_RES_TMQ_META(res) || TD_RES_TMQ_METADATA(res) || TD_RES_TMQ_BATCH_META(res)) { + if (res == NULL || TD_RES_TMQ(res) || TD_RES_TMQ_META(res) || TD_RES_TMQ_METADATA(res) || + TD_RES_TMQ_BATCH_META(res)) { return 0; } @@ -725,7 +728,8 @@ int taos_select_db(TAOS *taos, const char *db) { } void taos_stop_query(TAOS_RES *res) { - if (res == NULL || TD_RES_TMQ(res) || TD_RES_TMQ_META(res) || TD_RES_TMQ_METADATA(res) || TD_RES_TMQ_BATCH_META(res)) { + if (res == NULL || TD_RES_TMQ(res) || TD_RES_TMQ_META(res) || TD_RES_TMQ_METADATA(res) || + TD_RES_TMQ_BATCH_META(res)) { return; } @@ -784,7 +788,7 @@ int taos_fetch_block_s(TAOS_RES *res, int *numOfRows, TAOS_ROW *rows) { return pRequest->code; } else if (TD_RES_TMQ(res) || TD_RES_TMQ_METADATA(res)) { SReqResultInfo *pResultInfo = NULL; - int32_t code = tmqGetNextResInfo(res, true, &pResultInfo); + int32_t code = tmqGetNextResInfo(res, true, &pResultInfo); if (code != 0) return code; pResultInfo->current = pResultInfo->numOfRows; @@ -807,7 +811,7 @@ int taos_fetch_raw_block(TAOS_RES *res, int *numOfRows, void **pData) { if (TD_RES_TMQ(res) || TD_RES_TMQ_METADATA(res)) { SReqResultInfo *pResultInfo = NULL; - int32_t code = tmqGetNextResInfo(res, false, &pResultInfo); + int32_t code = tmqGetNextResInfo(res, false, &pResultInfo); if (code != 0) { (*numOfRows) = 0; return 0; @@ -937,7 +941,7 @@ static void doAsyncQueryFromAnalyse(SMetaData *pResultMeta, void *param, int32_t SRequestObj *pRequest = pWrapper->pRequest; SQuery *pQuery = pRequest->pQuery; - qDebug("0x%" PRIx64 " start to semantic analysis, reqId:0x%" PRIx64, pRequest->self, pRequest->requestId); + qDebug("0x%" PRIx64 " start to semantic analysis, qid:0x%" PRIx64, pRequest->self, pRequest->requestId); int64_t analyseStart = taosGetTimestampUs(); pRequest->metric.ctgCostUs = analyseStart - pRequest->metric.ctgStart; @@ -953,7 +957,7 @@ static void doAsyncQueryFromAnalyse(SMetaData *pResultMeta, void *param, int32_t (void)memcpy(&pRequest->parseMeta, pResultMeta, sizeof(*pResultMeta)); (void)memset(pResultMeta, 0, sizeof(*pResultMeta)); } - + handleQueryAnslyseRes(pWrapper, pResultMeta, code); } @@ -999,7 +1003,7 @@ void handleSubQueryFromAnalyse(SSqlCallbackWrapper *pWrapper, SMetaData *pResult } pNewRequest->pQuery = NULL; - code = nodesMakeNode(QUERY_NODE_QUERY, (SNode**)&pNewRequest->pQuery); + code = nodesMakeNode(QUERY_NODE_QUERY, (SNode **)&pNewRequest->pQuery); if (pNewRequest->pQuery) { pNewRequest->pQuery->pRoot = pRoot; pRoot = NULL; @@ -1056,15 +1060,15 @@ void handleQueryAnslyseRes(SSqlCallbackWrapper *pWrapper, SMetaData *pResultMeta pRequest->pQuery = NULL; if (NEED_CLIENT_HANDLE_ERROR(code)) { - tscDebug("0x%" PRIx64 " client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%" PRIx64, + tscDebug("0x%" PRIx64 " client retry to handle the error, code:%d - %s, tryCount:%d, qid:0x%" PRIx64, pRequest->self, code, tstrerror(code), pRequest->retry, pRequest->requestId); restartAsyncQuery(pRequest, code); return; } // return to app directly - tscError("0x%" PRIx64 " error occurs, code:%s, return to user app, reqId:0x%" PRIx64, pRequest->self, - tstrerror(code), pRequest->requestId); + tscError("0x%" PRIx64 " error occurs, code:%s, return to user app, qid:0x%" PRIx64, pRequest->self, tstrerror(code), + pRequest->requestId); pRequest->code = code; returnToUser(pRequest); } @@ -1113,7 +1117,7 @@ static void doAsyncQueryFromParse(SMetaData *pResultMeta, void *param, int32_t c SQuery *pQuery = pRequest->pQuery; pRequest->metric.ctgCostUs += taosGetTimestampUs() - pRequest->metric.ctgStart; - qDebug("0x%" PRIx64 " start to continue parse, reqId:0x%" PRIx64 ", code:%s", pRequest->self, pRequest->requestId, + qDebug("0x%" PRIx64 " start to continue parse, qid:0x%" PRIx64 ", code:%s", pRequest->self, pRequest->requestId, tstrerror(code)); if (code == TSDB_CODE_SUCCESS) { @@ -1126,7 +1130,7 @@ static void doAsyncQueryFromParse(SMetaData *pResultMeta, void *param, int32_t c } if (TSDB_CODE_SUCCESS != code) { - tscError("0x%" PRIx64 " error happens, code:%d - %s, reqId:0x%" PRIx64, pWrapper->pRequest->self, code, + tscError("0x%" PRIx64 " error happens, code:%d - %s, qid:0x%" PRIx64, pWrapper->pRequest->self, code, tstrerror(code), pWrapper->pRequest->requestId); destorySqlCallbackWrapper(pWrapper); pRequest->pWrapper = NULL; @@ -1143,7 +1147,7 @@ void continueInsertFromCsv(SSqlCallbackWrapper *pWrapper, SRequestObj *pRequest) } if (TSDB_CODE_SUCCESS != code) { - tscError("0x%" PRIx64 " error happens, code:%d - %s, reqId:0x%" PRIx64, pWrapper->pRequest->self, code, + tscError("0x%" PRIx64 " error happens, code:%d - %s, qid:0x%" PRIx64, pWrapper->pRequest->self, code, tstrerror(code), pWrapper->pRequest->requestId); destorySqlCallbackWrapper(pWrapper); pRequest->pWrapper = NULL; @@ -1261,7 +1265,7 @@ void doAsyncQuery(SRequestObj *pRequest, bool updateMetaForce) { } if (TSDB_CODE_SUCCESS != code) { - tscError("0x%" PRIx64 " error happens, code:%d - %s, reqId:0x%" PRIx64, pRequest->self, code, tstrerror(code), + tscError("0x%" PRIx64 " error happens, code:%d - %s, qid:0x%" PRIx64, pRequest->self, code, tstrerror(code), pRequest->requestId); destorySqlCallbackWrapper(pWrapper); pRequest->pWrapper = NULL; @@ -1269,9 +1273,9 @@ void doAsyncQuery(SRequestObj *pRequest, bool updateMetaForce) { pRequest->pQuery = NULL; if (NEED_CLIENT_HANDLE_ERROR(code)) { - tscDebug("0x%" PRIx64 " client retry to handle the error, code:%d - %s, tryCount:%d, reqId:0x%" PRIx64, + tscDebug("0x%" PRIx64 " client retry to handle the error, code:%d - %s, tryCount:%d, qid:0x%" PRIx64, pRequest->self, code, tstrerror(code), pRequest->retry, pRequest->requestId); - (void)refreshMeta(pRequest->pTscObj, pRequest); //ignore return code,try again + (void)refreshMeta(pRequest->pTscObj, pRequest); // ignore return code,try again pRequest->prevCode = code; doAsyncQuery(pRequest, true); return; @@ -1285,7 +1289,7 @@ void doAsyncQuery(SRequestObj *pRequest, bool updateMetaForce) { void restartAsyncQuery(SRequestObj *pRequest, int32_t code) { tscInfo("restart request: %s p: %p", pRequest->sqlstr, pRequest); - SRequestObj* pUserReq = pRequest; + SRequestObj *pUserReq = pRequest; (void)acquireRequest(pRequest->self); while (pUserReq) { if (pUserReq->self == pUserReq->relation.userRefId || pUserReq->relation.userRefId == 0) { @@ -1631,7 +1635,6 @@ TAOS_STMT *taos_stmt_init_with_options(TAOS *taos, TAOS_STMT_OPTIONS *options) { return pStmt; } - int taos_stmt_prepare(TAOS_STMT *stmt, const char *sql, unsigned long length) { if (stmt == NULL || sql == NULL) { tscError("NULL parameter for %s", __FUNCTION__); @@ -1874,7 +1877,7 @@ int taos_stmt_close(TAOS_STMT *stmt) { return stmtClose(stmt); } -int taos_set_conn_mode(TAOS* taos, int mode, int value) { +int taos_set_conn_mode(TAOS *taos, int mode, int value) { if (taos == NULL) { terrno = TSDB_CODE_INVALID_PARA; return terrno; @@ -1897,6 +1900,4 @@ int taos_set_conn_mode(TAOS* taos, int mode, int value) { return 0; } -char* getBuildInfo(){ - return buildinfo; -} +char *getBuildInfo() { return buildinfo; } diff --git a/source/client/src/clientMonitor.c b/source/client/src/clientMonitor.c index 4bb29f8d97..4492073620 100644 --- a/source/client/src/clientMonitor.c +++ b/source/client/src/clientMonitor.c @@ -120,9 +120,13 @@ static int32_t monitorReportAsyncCB(void* param, SDataBuf* pMsg, int32_t code) { .data = NULL}; if (monitorPutData2MonitorQueue(tmp) == 0) { p->fileName = NULL; + } else { + if(taosCloseFile(&(p->pFile)) != 0) { + tscError("failed to close file:%p", p->pFile); + } } } - return code; + return TSDB_CODE_SUCCESS; } static int32_t sendReport(void* pTransporter, SEpSet* epSet, char* pCont, MONITOR_TYPE type, void* param) { @@ -164,7 +168,10 @@ static int32_t sendReport(void* pTransporter, SEpSet* epSet, char* pCont, MONITO int64_t transporterId = 0; return asyncSendMsgToServer(pTransporter, epSet, &transporterId, pInfo); - FAILED: +FAILED: + if (taosCloseFile(&(((MonitorSlowLogData*)param)->pFile)) != 0) { + tscError("failed to close file:%p", ((MonitorSlowLogData*)param)->pFile); + } monitorFreeSlowLogDataEx(param); return TAOS_GET_TERRNO(TSDB_CODE_TSC_INTERNAL_ERROR); } @@ -315,7 +322,7 @@ void monitorCreateClientCounter(int64_t clusterId, const char* name, const char* void monitorCounterInc(int64_t clusterId, const char* counterName, const char** label_values) { taosWLockLatch(&monitorLock); if (atomic_load_32(&monitorFlag) == 1) { - taosRUnLockLatch(&monitorLock); + taosWUnLockLatch(&monitorLock); return; } @@ -406,7 +413,11 @@ static char* readFile(TdFilePtr pFile, int64_t* offset, int64_t size) { return NULL; } - ASSERT(size > *offset); + if((size <= *offset)){ + tscError("invalid size:%" PRId64 ", offset:%" PRId64, size, *offset); + terrno = TSDB_CODE_TSC_INTERNAL_ERROR; + return NULL; + } char* pCont = NULL; int64_t totalSize = 0; if (size - *offset >= SLOW_LOG_SEND_SIZE_MAX) { @@ -462,11 +473,17 @@ static int64_t getFileSize(char* path) { static int32_t sendSlowLog(int64_t clusterId, char* data, TdFilePtr pFile, int64_t offset, SLOW_LOG_QUEUE_TYPE type, char* fileName, void* pTransporter, SEpSet* epSet) { if (data == NULL) { + if (taosCloseFile(&pFile) != 0) { + tscError("failed to close file:%p", pFile); + } taosMemoryFree(fileName); return TSDB_CODE_INVALID_PARA; } MonitorSlowLogData* pParam = taosMemoryMalloc(sizeof(MonitorSlowLogData)); if (pParam == NULL) { + if (taosCloseFile(&pFile) != 0) { + tscError("failed to close file:%p", pFile); + } taosMemoryFree(data); taosMemoryFree(fileName); return terrno; @@ -485,11 +502,15 @@ static int32_t monitorReadSend(int64_t clusterId, TdFilePtr pFile, int64_t* offs SAppInstInfo* pInst = getAppInstByClusterId(clusterId); if (pInst == NULL) { tscError("failed to get app instance by clusterId:%" PRId64, clusterId); + if (taosCloseFile(&pFile) != 0) { + tscError("failed to close file:%p", pFile); + } taosMemoryFree(fileName); return terrno; } SEpSet ep = getEpSet_s(&pInst->mgmtEp); char* data = readFile(pFile, offset, size); + if(data == NULL) return terrno; return sendSlowLog(clusterId, data, (type == SLOW_LOG_READ_BEGINNIG ? pFile : NULL), *offset, type, fileName, pInst->pTransporter, &ep); } @@ -710,7 +731,7 @@ static void* monitorThreadFunc(void* param) { MonitorSlowLogData* slowLogData = NULL; (void)taosReadQitem(monitorQueue, (void**)&slowLogData); if (slowLogData != NULL) { - if (slowLogData->type == SLOW_LOG_READ_BEGINNIG) { + if (slowLogData->type == SLOW_LOG_READ_BEGINNIG && quitCnt == 0) { if (slowLogData->pFile != NULL) { monitorSendSlowLogAtBeginning(slowLogData->clusterId, &(slowLogData->fileName), slowLogData->pFile, slowLogData->offset); @@ -729,9 +750,9 @@ static void* monitorThreadFunc(void* param) { break; } } + monitorFreeSlowLogData(slowLogData); + taosFreeQitem(slowLogData); } - monitorFreeSlowLogData(slowLogData); - taosFreeQitem(slowLogData); if (quitCnt == 0) { monitorSendAllSlowLog(); @@ -853,6 +874,9 @@ int32_t monitorPutData2MonitorQueue(MonitorSlowLogData data) { if (taosWriteQitem(monitorQueue, slowLogData) == 0) { (void)tsem2_post(&monitorSem); } else { + if (taosCloseFile(&(slowLogData->pFile)) != 0) { + tscError("failed to close file:%p", slowLogData->pFile); + } monitorFreeSlowLogData(slowLogData); taosFreeQitem(slowLogData); } diff --git a/source/client/src/clientMsgHandler.c b/source/client/src/clientMsgHandler.c index 4dea9c17b0..771a22b7e3 100644 --- a/source/client/src/clientMsgHandler.c +++ b/source/client/src/clientMsgHandler.c @@ -576,8 +576,8 @@ static int32_t buildShowVariablesRsp(SArray* pVars, SRetrieveTableRsp** pRsp) { size_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock) + PAYLOAD_PREFIX_LEN; *pRsp = taosMemoryCalloc(1, rspSize); if (NULL == *pRsp) { - blockDataDestroy(pBlock); - return TSDB_CODE_OUT_OF_MEMORY; + code = terrno; + goto _exit; } (*pRsp)->useconds = 0; @@ -589,6 +589,11 @@ static int32_t buildShowVariablesRsp(SArray* pVars, SRetrieveTableRsp** pRsp) { (*pRsp)->numOfCols = htonl(SHOW_VARIABLES_RESULT_COLS); int32_t len = blockEncode(pBlock, (*pRsp)->data + PAYLOAD_PREFIX_LEN, SHOW_VARIABLES_RESULT_COLS); + if(len < 0) { + uError("buildShowVariablesRsp error, len:%d", len); + code = terrno; + goto _exit; + } blockDataDestroy(pBlock); SET_PAYLOAD_LEN((*pRsp)->data, len, len); @@ -600,10 +605,21 @@ static int32_t buildShowVariablesRsp(SArray* pVars, SRetrieveTableRsp** pRsp) { if (payloadLen != rspSize - sizeof(SRetrieveTableRsp)) { uError("buildShowVariablesRsp error, len:%d != rspSize - sizeof(SRetrieveTableRsp):%" PRIu64, len, (uint64_t)(rspSize - sizeof(SRetrieveTableRsp))); - return TSDB_CODE_TSC_INVALID_INPUT; + code = TSDB_CODE_TSC_INVALID_INPUT; + goto _exit; } return TSDB_CODE_SUCCESS; +_exit: + if(*pRsp) { + taosMemoryFree(*pRsp); + *pRsp = NULL; + } + if(pBlock) { + blockDataDestroy(pBlock); + pBlock = NULL; + } + return code; } int32_t processShowVariablesRsp(void* param, SDataBuf* pMsg, int32_t code) { @@ -711,8 +727,8 @@ static int32_t buildRetriveTableRspForCompactDb(SCompactDbRsp* pCompactDb, SRetr size_t rspSize = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock) + PAYLOAD_PREFIX_LEN; *pRsp = taosMemoryCalloc(1, rspSize); if (NULL == *pRsp) { - blockDataDestroy(pBlock); - return TSDB_CODE_OUT_OF_MEMORY; + code = terrno; + goto _exit; } (*pRsp)->useconds = 0; @@ -725,6 +741,11 @@ static int32_t buildRetriveTableRspForCompactDb(SCompactDbRsp* pCompactDb, SRetr (*pRsp)->numOfCols = htonl(COMPACT_DB_RESULT_COLS); int32_t len = blockEncode(pBlock, (*pRsp)->data + PAYLOAD_PREFIX_LEN, COMPACT_DB_RESULT_COLS); + if(len < 0) { + uError("buildRetriveTableRspForCompactDb error, len:%d", len); + code = terrno; + goto _exit; + } blockDataDestroy(pBlock); SET_PAYLOAD_LEN((*pRsp)->data, len, len); @@ -736,10 +757,21 @@ static int32_t buildRetriveTableRspForCompactDb(SCompactDbRsp* pCompactDb, SRetr if (payloadLen != rspSize - sizeof(SRetrieveTableRsp)) { uError("buildRetriveTableRspForCompactDb error, len:%d != rspSize - sizeof(SRetrieveTableRsp):%" PRIu64, len, (uint64_t)(rspSize - sizeof(SRetrieveTableRsp))); - return TSDB_CODE_TSC_INVALID_INPUT; + code = TSDB_CODE_TSC_INVALID_INPUT; + goto _exit; } return TSDB_CODE_SUCCESS; +_exit: + if(*pRsp) { + taosMemoryFree(*pRsp); + *pRsp = NULL; + } + if(pBlock) { + blockDataDestroy(pBlock); + pBlock = NULL; + } + return code; } diff --git a/source/client/src/clientRawBlockWrite.c b/source/client/src/clientRawBlockWrite.c index f05120b366..8eb2f46dfb 100644 --- a/source/client/src/clientRawBlockWrite.c +++ b/source/client/src/clientRawBlockWrite.c @@ -23,32 +23,31 @@ #include "tglobal.h" #include "tmsgtype.h" -#define RAW_NULL_CHECK(c) \ - do { \ - if (c == NULL) { \ - code = TSDB_CODE_OUT_OF_MEMORY; \ - goto end; \ - } \ +#define RAW_NULL_CHECK(c) \ + do { \ + if (c == NULL) { \ + code = TSDB_CODE_OUT_OF_MEMORY; \ + goto end; \ + } \ } while (0) -#define RAW_FALSE_CHECK(c) \ - do { \ - if (!c) { \ - code = TSDB_CODE_INVALID_PARA; \ - goto end; \ - } \ +#define RAW_FALSE_CHECK(c) \ + do { \ + if (!c) { \ + code = TSDB_CODE_INVALID_PARA; \ + goto end; \ + } \ } while (0) -#define RAW_RETURN_CHECK(c) \ - do { \ - code = c; \ - if (code != 0) { \ - goto end; \ - } \ +#define RAW_RETURN_CHECK(c) \ + do { \ + code = c; \ + if (code != 0) { \ + goto end; \ + } \ } while (0) - -#define LOG_ID_TAG "connId:0x%" PRIx64 ",reqId:0x%" PRIx64 +#define LOG_ID_TAG "connId:0x%" PRIx64 ",qid:0x%" PRIx64 #define LOG_ID_VALUE *(int64_t*)taos, pRequest->requestId #define TMQ_META_VERSION "1.0" @@ -57,10 +56,10 @@ static int32_t tmqWriteBatchMetaDataImpl(TAOS* taos, void* meta, int32_t metaLen static tb_uid_t processSuid(tb_uid_t suid, char* db) { return suid + MurmurHash3_32(db, strlen(db)); } -static void buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id, - int8_t t, SColCmprWrapper* pColCmprRow, cJSON** pJson) { +static void buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id, int8_t t, + SColCmprWrapper* pColCmprRow, cJSON** pJson) { int32_t code = TSDB_CODE_SUCCESS; - int8_t buildDefaultCompress = 0; + int8_t buildDefaultCompress = 0; if (pColCmprRow->nCols <= 0) { buildDefaultCompress = 1; } @@ -82,7 +81,7 @@ static void buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* sche cJSON* columns = cJSON_CreateArray(); RAW_NULL_CHECK(columns); for (int i = 0; i < schemaRow->nCols; i++) { - cJSON* column = cJSON_CreateObject(); + cJSON* column = cJSON_CreateObject(); RAW_NULL_CHECK(column); SSchema* s = schemaRow->pSchema + i; cJSON* cname = cJSON_CreateString(s->name); @@ -142,7 +141,7 @@ static void buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* sche cJSON* tags = cJSON_CreateArray(); RAW_NULL_CHECK(tags); for (int i = 0; schemaTag && i < schemaTag->nCols; i++) { - cJSON* tag = cJSON_CreateObject(); + cJSON* tag = cJSON_CreateObject(); RAW_NULL_CHECK(tag); SSchema* s = schemaTag->pSchema + i; cJSON* tname = cJSON_CreateString(s->name); @@ -176,7 +175,7 @@ static int32_t setCompressOption(cJSON* json, uint32_t para) { if (encode != 0) { const char* encodeStr = columnEncodeStr(encode); RAW_NULL_CHECK(encodeStr); - cJSON* encodeJson = cJSON_CreateString(encodeStr); + cJSON* encodeJson = cJSON_CreateString(encodeStr); RAW_NULL_CHECK(encodeJson); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "encode", encodeJson)); return code; @@ -185,7 +184,7 @@ static int32_t setCompressOption(cJSON* json, uint32_t para) { if (compress != 0) { const char* compressStr = columnCompressStr(compress); RAW_NULL_CHECK(compressStr); - cJSON* compressJson = cJSON_CreateString(compressStr); + cJSON* compressJson = cJSON_CreateString(compressStr); RAW_NULL_CHECK(compressJson); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "compress", compressJson)); return code; @@ -194,7 +193,7 @@ static int32_t setCompressOption(cJSON* json, uint32_t para) { if (level != 0) { const char* levelStr = columnLevelStr(level); RAW_NULL_CHECK(levelStr); - cJSON* levelJson = cJSON_CreateString(levelStr); + cJSON* levelJson = cJSON_CreateString(levelStr); RAW_NULL_CHECK(levelJson); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "level", levelJson)); return code; @@ -235,7 +234,7 @@ static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** case TSDB_ALTER_TABLE_ADD_COLUMN: { TAOS_FIELD* field = taosArrayGet(req.pFields, 0); RAW_NULL_CHECK(field); - cJSON* colName = cJSON_CreateString(field->name); + cJSON* colName = cJSON_CreateString(field->name); RAW_NULL_CHECK(colName); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "colName", colName)); cJSON* colType = cJSON_CreateNumber(field->type); @@ -259,7 +258,7 @@ static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** case TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COMPRESS_OPTION: { SFieldWithOptions* field = taosArrayGet(req.pFields, 0); RAW_NULL_CHECK(field); - cJSON* colName = cJSON_CreateString(field->name); + cJSON* colName = cJSON_CreateString(field->name); RAW_NULL_CHECK(colName); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "colName", colName)); cJSON* colType = cJSON_CreateNumber(field->type); @@ -285,7 +284,7 @@ static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** case TSDB_ALTER_TABLE_DROP_COLUMN: { TAOS_FIELD* field = taosArrayGet(req.pFields, 0); RAW_NULL_CHECK(field); - cJSON* colName = cJSON_CreateString(field->name); + cJSON* colName = cJSON_CreateString(field->name); RAW_NULL_CHECK(colName); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "colName", colName)); break; @@ -294,7 +293,7 @@ static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** case TSDB_ALTER_TABLE_UPDATE_COLUMN_BYTES: { TAOS_FIELD* field = taosArrayGet(req.pFields, 0); RAW_NULL_CHECK(field); - cJSON* colName = cJSON_CreateString(field->name); + cJSON* colName = cJSON_CreateString(field->name); RAW_NULL_CHECK(colName); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "colName", colName)); cJSON* colType = cJSON_CreateNumber(field->type); @@ -320,7 +319,7 @@ static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** RAW_NULL_CHECK(oldField); TAOS_FIELD* newField = taosArrayGet(req.pFields, 1); RAW_NULL_CHECK(newField); - cJSON* colName = cJSON_CreateString(oldField->name); + cJSON* colName = cJSON_CreateString(oldField->name); RAW_NULL_CHECK(colName); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "colName", colName)); cJSON* colNewName = cJSON_CreateString(newField->name); @@ -331,7 +330,7 @@ static void buildAlterSTableJson(void* alterData, int32_t alterDataLen, cJSON** case TSDB_ALTER_TABLE_UPDATE_COLUMN_COMPRESS: { TAOS_FIELD* field = taosArrayGet(req.pFields, 0); RAW_NULL_CHECK(field); - cJSON* colName = cJSON_CreateString(field->name); + cJSON* colName = cJSON_CreateString(field->name); RAW_NULL_CHECK(colName); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "colName", colName)); RAW_RETURN_CHECK(setCompressOption(json, field->bytes)); @@ -405,7 +404,7 @@ static void buildChildElement(cJSON* json, SVCreateTbReq* pCreateReq) { RAW_NULL_CHECK(tagNumJson); RAW_FALSE_CHECK(cJSON_AddItemToObject(json, "tagNum", tagNumJson)); - cJSON* tags = cJSON_CreateArray(); + cJSON* tags = cJSON_CreateArray(); RAW_NULL_CHECK(tags); SArray* pTagVals = NULL; RAW_RETURN_CHECK(tTagToValArray(pTag, &pTagVals)); @@ -416,13 +415,13 @@ static void buildChildElement(cJSON* json, SVCreateTbReq* pCreateReq) { uError("p->nTag == 0"); goto end; } - char* pJson = NULL; + char* pJson = NULL; parseTagDatatoJson(pTag, &pJson); - cJSON* tag = cJSON_CreateObject(); + cJSON* tag = cJSON_CreateObject(); RAW_NULL_CHECK(tag); STagVal* pTagVal = taosArrayGet(pTagVals, 0); RAW_NULL_CHECK(pTagVal); - char* ptname = taosArrayGet(tagName, 0); + char* ptname = taosArrayGet(tagName, 0); RAW_NULL_CHECK(ptname); cJSON* tname = cJSON_CreateString(ptname); RAW_NULL_CHECK(tname); @@ -443,7 +442,7 @@ static void buildChildElement(cJSON* json, SVCreateTbReq* pCreateReq) { RAW_NULL_CHECK(pTagVal); cJSON* tag = cJSON_CreateObject(); RAW_NULL_CHECK(tag); - char* ptname = taosArrayGet(tagName, i); + char* ptname = taosArrayGet(tagName, i); RAW_NULL_CHECK(ptname); cJSON* tname = cJSON_CreateString(ptname); RAW_NULL_CHECK(tname); @@ -463,7 +462,7 @@ static void buildChildElement(cJSON* json, SVCreateTbReq* pCreateReq) { RAW_NULL_CHECK(buf); if (!buf) goto end; - if(dataConverToStr(buf, pTagVal->type, pTagVal->pData, pTagVal->nData, NULL) != TSDB_CODE_SUCCESS) { + if (dataConverToStr(buf, pTagVal->type, pTagVal->pData, pTagVal->nData, NULL) != TSDB_CODE_SUCCESS) { taosMemoryFree(buf); goto end; } @@ -489,8 +488,8 @@ end: static void buildCreateCTableJson(SVCreateTbReq* pCreateReq, int32_t nReqs, cJSON** pJson) { int32_t code = 0; - char* string = NULL; - cJSON* json = cJSON_CreateObject(); + char* string = NULL; + cJSON* json = cJSON_CreateObject(); RAW_NULL_CHECK(json); cJSON* type = cJSON_CreateString("create"); RAW_NULL_CHECK(type); @@ -534,8 +533,8 @@ static void processCreateTable(SMqMetaRsp* metaRsp, cJSON** pJson) { if (pCreateReq->type == TSDB_CHILD_TABLE) { buildCreateCTableJson(req.pReqs, req.nReqs, pJson); } else if (pCreateReq->type == TSDB_NORMAL_TABLE) { - buildCreateTableJson(&pCreateReq->ntb.schemaRow, NULL, pCreateReq->name, pCreateReq->uid, - TSDB_NORMAL_TABLE, &pCreateReq->colCmpr, pJson); + buildCreateTableJson(&pCreateReq->ntb.schemaRow, NULL, pCreateReq->name, pCreateReq->uid, TSDB_NORMAL_TABLE, + &pCreateReq->colCmpr, pJson); } } @@ -548,7 +547,7 @@ end: static void processAutoCreateTable(STaosxRsp* rsp, char** string) { SDecoder* decoder = NULL; SVCreateTbReq* pCreateReq = NULL; - int32_t code = 0; + int32_t code = 0; uDebug("auto create table data:%p", rsp); if (rsp->createTableNum <= 0) { uError("processAutoCreateTable rsp->createTableNum <= 0"); @@ -563,7 +562,7 @@ static void processAutoCreateTable(STaosxRsp* rsp, char** string) { // loop to create table for (int32_t iReq = 0; iReq < rsp->createTableNum; iReq++) { // decode - void** data = taosArrayGet(rsp->createTableReq, iReq); + void** data = taosArrayGet(rsp->createTableReq, iReq); RAW_NULL_CHECK(data); int32_t* len = taosArrayGet(rsp->createTableLen, iReq); RAW_NULL_CHECK(len); @@ -735,7 +734,8 @@ static void processAlterTable(SMqMetaRsp* metaRsp, cJSON** pJson) { buf = taosMemoryCalloc(vAlterTbReq.nTagVal + 3, 1); } RAW_NULL_CHECK(buf); - if(dataConverToStr(buf, vAlterTbReq.tagType, vAlterTbReq.pTagVal, vAlterTbReq.nTagVal, NULL) != TSDB_CODE_SUCCESS) { + if (dataConverToStr(buf, vAlterTbReq.tagType, vAlterTbReq.pTagVal, vAlterTbReq.nTagVal, NULL) != + TSDB_CODE_SUCCESS) { taosMemoryFree(buf); goto end; } @@ -823,7 +823,7 @@ static void processDeleteTable(SMqMetaRsp* metaRsp, cJSON** pJson) { // getTbName(req.tableFName); char sql[256] = {0}; (void)snprintf(sql, sizeof(sql), "delete from `%s` where `%s` >= %" PRId64 " and `%s` <= %" PRId64, req.tableFName, - req.tsColName, req.skey, req.tsColName, req.ekey); + req.tsColName, req.skey, req.tsColName, req.ekey); json = cJSON_CreateObject(); RAW_NULL_CHECK(json); @@ -865,7 +865,7 @@ static void processDropTable(SMqMetaRsp* metaRsp, cJSON** pJson) { RAW_NULL_CHECK(tableNameList); for (int32_t iReq = 0; iReq < req.nReqs; iReq++) { SVDropTbReq* pDropTbReq = req.pReqs + iReq; - cJSON* tableName = cJSON_CreateString(pDropTbReq->name); + cJSON* tableName = cJSON_CreateString(pDropTbReq->name); RAW_NULL_CHECK(tableName); RAW_FALSE_CHECK(cJSON_AddItemToArray(tableNameList, tableName)); } @@ -910,7 +910,7 @@ static int32_t taosCreateStb(TAOS* taos, void* meta, int32_t metaLen) { } // build create stable pReq.pColumns = taosArrayInit(req.schemaRow.nCols, sizeof(SFieldWithOptions)); - RAW_NULL_CHECK (pReq.pColumns); + RAW_NULL_CHECK(pReq.pColumns); for (int32_t i = 0; i < req.schemaRow.nCols; i++) { SSchema* pSchema = req.schemaRow.pSchema + i; SFieldWithOptions field = {.type = pSchema->type, .flags = pSchema->flags, .bytes = pSchema->bytes}; @@ -957,7 +957,7 @@ static int32_t taosCreateStb(TAOS* taos, void* meta, int32_t metaLen) { } pCmdMsg.pMsg = taosMemoryMalloc(pCmdMsg.msgLen); RAW_NULL_CHECK(pCmdMsg.pMsg); - if (tSerializeSMCreateStbReq(pCmdMsg.pMsg, pCmdMsg.msgLen, &pReq) <= 0){ + if (tSerializeSMCreateStbReq(pCmdMsg.pMsg, pCmdMsg.msgLen, &pReq) <= 0) { code = TSDB_CODE_INVALID_PARA; goto end; } @@ -968,7 +968,7 @@ static int32_t taosCreateStb(TAOS* taos, void* meta, int32_t metaLen) { pQuery.msgType = pQuery.pCmdMsg->msgType; pQuery.stableQuery = true; - (void)launchQueryImpl(pRequest, &pQuery, true, NULL); //ignore, because return value is pRequest + (void)launchQueryImpl(pRequest, &pQuery, true, NULL); // ignore, because return value is pRequest if (pRequest->code == TSDB_CODE_SUCCESS) { SCatalog* pCatalog = NULL; @@ -1021,7 +1021,8 @@ static int32_t taosDropStb(TAOS* taos, void* meta, int32_t metaLen) { .requestObjRefId = pRequest->self, .mgmtEps = getEpSet_s(&pRequest->pTscObj->pAppInfo->mgmtEp)}; SName pName = {0}; - (void)toName(pRequest->pTscObj->acctId, pRequest->pDb, req.name, &pName); // ignore the return value, always return pName + (void)toName(pRequest->pTscObj->acctId, pRequest->pDb, req.name, + &pName); // ignore the return value, always return pName STableMeta* pTableMeta = NULL; code = catalogGetTableMeta(pCatalog, &conn, &pName, &pTableMeta); if (code == TSDB_CODE_PAR_TABLE_NOT_EXIST) { @@ -1059,19 +1060,18 @@ static int32_t taosDropStb(TAOS* taos, void* meta, int32_t metaLen) { } pCmdMsg.pMsg = taosMemoryMalloc(pCmdMsg.msgLen); RAW_NULL_CHECK(pCmdMsg.pMsg); - if (tSerializeSMDropStbReq(pCmdMsg.pMsg, pCmdMsg.msgLen, &pReq) <= 0){ + if (tSerializeSMDropStbReq(pCmdMsg.pMsg, pCmdMsg.msgLen, &pReq) <= 0) { code = TSDB_CODE_INVALID_PARA; goto end; } - SQuery pQuery = {0}; pQuery.execMode = QUERY_EXEC_MODE_RPC; pQuery.pCmdMsg = &pCmdMsg; pQuery.msgType = pQuery.pCmdMsg->msgType; pQuery.stableQuery = true; - (void)launchQueryImpl(pRequest, &pQuery, true, NULL); //ignore, because return value is pRequest + (void)launchQueryImpl(pRequest, &pQuery, true, NULL); // ignore, because return value is pRequest if (pRequest->code == TSDB_CODE_SUCCESS) { // ignore the error code @@ -1391,9 +1391,9 @@ static int32_t taosDeleteData(TAOS* taos, void* meta, int32_t metaLen) { } (void)snprintf(sql, sizeof(sql), "delete from `%s` where `%s` >= %" PRId64 " and `%s` <= %" PRId64, req.tableFName, - req.tsColName, req.skey, req.tsColName, req.ekey); + req.tsColName, req.skey, req.tsColName, req.ekey); - TAOS_RES* res = taosQueryImpl(taos, sql, false, TD_REQ_FROM_TAOX); + TAOS_RES* res = taosQueryImpl(taos, sql, false, TD_REQ_FROM_TAOX); RAW_NULL_CHECK(res); SRequestObj* pRequest = (SRequestObj*)res; code = pRequest->code; @@ -1500,7 +1500,6 @@ static int32_t taosAlterTable(TAOS* taos, void* meta, int32_t metaLen) { if (TSDB_CODE_SUCCESS != code) goto end; RAW_RETURN_CHECK(rewriteToVnodeModifyOpStmt(pQuery, pArray)); - (void)launchQueryImpl(pRequest, pQuery, true, NULL); pVgData = NULL; @@ -1573,7 +1572,8 @@ int taos_write_raw_block_with_fields_with_reqid(TAOS* taos, int rows, char* pDat RAW_RETURN_CHECK(smlInitHandle(&pQuery)); pVgHash = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), true, HASH_NO_LOCK); RAW_NULL_CHECK(pVgHash); - RAW_RETURN_CHECK(taosHashPut(pVgHash, (const char*)&vgData.vgId, sizeof(vgData.vgId), (char*)&vgData, sizeof(vgData))); + RAW_RETURN_CHECK( + taosHashPut(pVgHash, (const char*)&vgData.vgId, sizeof(vgData.vgId), (char*)&vgData, sizeof(vgData))); RAW_RETURN_CHECK(rawBlockBindData(pQuery, pTableMeta, pData, NULL, fields, numFields, false, NULL, 0)); RAW_RETURN_CHECK(smlBuildOutput(pQuery, pVgHash)); @@ -1632,7 +1632,8 @@ int taos_write_raw_block_with_reqid(TAOS* taos, int rows, char* pData, const cha RAW_RETURN_CHECK(smlInitHandle(&pQuery)); pVgHash = taosHashInit(16, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), true, HASH_NO_LOCK); RAW_NULL_CHECK(pVgHash); - RAW_RETURN_CHECK(taosHashPut(pVgHash, (const char*)&vgData.vgId, sizeof(vgData.vgId), (char*)&vgData, sizeof(vgData))); + RAW_RETURN_CHECK( + taosHashPut(pVgHash, (const char*)&vgData.vgId, sizeof(vgData.vgId), (char*)&vgData, sizeof(vgData))); RAW_RETURN_CHECK(rawBlockBindData(pQuery, pTableMeta, pData, NULL, NULL, 0, false, NULL, 0)); RAW_RETURN_CHECK(smlBuildOutput(pQuery, pVgHash)); @@ -1656,7 +1657,6 @@ static void* getRawDataFromRes(void* pRetrieve) { } else if (*(int64_t*)pRetrieve == 1) { rawData = ((SRetrieveTableRspForTmq*)pRetrieve)->data; } - ASSERT(rawData != NULL); return rawData; } @@ -1736,7 +1736,7 @@ static int32_t tmqWriteRawDataImpl(TAOS* taos, void* data, int32_t dataLen) { SSchemaWrapper* pSW = (SSchemaWrapper*)taosArrayGetP(rspObj.rsp.common.blockSchema, rspObj.common.resIter); RAW_NULL_CHECK(pSW); - TAOS_FIELD* fields = taosMemoryCalloc(pSW->nCols, sizeof(TAOS_FIELD)); + TAOS_FIELD* fields = taosMemoryCalloc(pSW->nCols, sizeof(TAOS_FIELD)); RAW_NULL_CHECK(fields); for (int i = 0; i < pSW->nCols; i++) { fields[i].type = pSW->pSchema[i].type; @@ -1845,7 +1845,7 @@ static int32_t tmqWriteRawMetaDataImpl(TAOS* taos, void* data, int32_t dataLen) // find schema data info for (int j = 0; j < rspObj.rsp.createTableNum; j++) { - void** dataTmp = taosArrayGet(rspObj.rsp.createTableReq, j); + void** dataTmp = taosArrayGet(rspObj.rsp.createTableReq, j); RAW_NULL_CHECK(dataTmp); int32_t* lenTmp = taosArrayGet(rspObj.rsp.createTableLen, j); RAW_NULL_CHECK(dataTmp); @@ -1897,7 +1897,7 @@ static int32_t tmqWriteRawMetaDataImpl(TAOS* taos, void* data, int32_t dataLen) SSchemaWrapper* pSW = (SSchemaWrapper*)taosArrayGetP(rspObj.rsp.common.blockSchema, rspObj.common.resIter); RAW_NULL_CHECK(pSW); - TAOS_FIELD* fields = taosMemoryCalloc(pSW->nCols, sizeof(TAOS_FIELD)); + TAOS_FIELD* fields = taosMemoryCalloc(pSW->nCols, sizeof(TAOS_FIELD)); if (fields == NULL) { SET_ERROR_MSG("calloc fields failed"); code = TSDB_CODE_OUT_OF_MEMORY; @@ -1980,10 +1980,10 @@ static void processBatchMetaToJson(SMqBatchMetaRsp* pMsgRsp, char** string) { RAW_NULL_CHECK(len); void* tmpBuf = taosArrayGetP(rsp.batchMetaReq, i); RAW_NULL_CHECK(tmpBuf); - SDecoder metaCoder = {0}; + SDecoder metaCoder = {0}; SMqMetaRsp metaRsp = {0}; tDecoderInit(&metaCoder, POINTER_SHIFT(tmpBuf, sizeof(SMqRspHead)), *len - sizeof(SMqRspHead)); - if(tDecodeMqMetaRsp(&metaCoder, &metaRsp) < 0 ) { + if (tDecodeMqMetaRsp(&metaCoder, &metaRsp) < 0) { goto end; } cJSON* pItem = NULL; @@ -2013,18 +2013,18 @@ char* tmq_get_json_meta(TAOS_RES* res) { if (TD_RES_TMQ_METADATA(res)) { SMqTaosxRspObj* pMetaDataRspObj = (SMqTaosxRspObj*)res; - char* string = NULL; + char* string = NULL; processAutoCreateTable(&pMetaDataRspObj->rsp, &string); return string; } else if (TD_RES_TMQ_BATCH_META(res)) { SMqBatchMetaRspObj* pBatchMetaRspObj = (SMqBatchMetaRspObj*)res; - char* string = NULL; + char* string = NULL; processBatchMetaToJson(&pBatchMetaRspObj->rsp, &string); return string; } SMqMetaRspObj* pMetaRspObj = (SMqMetaRspObj*)res; - cJSON* pJson = NULL; + cJSON* pJson = NULL; processSimpleMeta(&pMetaRspObj->metaRsp, &pJson); char* string = cJSON_PrintUnformatted(pJson); cJSON_Delete(pJson); @@ -2103,7 +2103,7 @@ int32_t tmq_get_raw(TAOS_RES* res, tmq_raw_data* raw) { uDebug("tmq get raw type meta:%p", raw); } else if (TD_RES_TMQ(res)) { SMqRspObj* rspObj = ((SMqRspObj*)res); - int32_t code = encodeMqDataRsp(tEncodeMqDataRsp, &rspObj->rsp, raw); + int32_t code = encodeMqDataRsp(tEncodeMqDataRsp, &rspObj->rsp, raw); if (code != 0) { uError("tmq get raw type error:%d", terrno); return code; diff --git a/source/client/src/clientSml.c b/source/client/src/clientSml.c index 55be514e40..c8fd5281ef 100644 --- a/source/client/src/clientSml.c +++ b/source/client/src/clientSml.c @@ -360,7 +360,10 @@ int32_t smlProcessSuperTable(SSmlHandle *info, SSmlLineInfo *elements) { } else { sMeta = *tmp; } - ASSERT(sMeta != NULL); + if (sMeta == NULL) { + uError("smlProcessSuperTable failed to get super table meta"); + return TSDB_CODE_SML_INTERNAL_ERROR; + } info->currSTableMeta = sMeta->tableMeta; info->maxTagKVs = sMeta->tags; info->maxColKVs = sMeta->cols; @@ -424,7 +427,10 @@ int32_t smlProcessChildTable(SSmlHandle *info, SSmlLineInfo *elements) { } else { tinfo = *oneTable; } - ASSERT(tinfo != NULL); + if (tinfo == NULL) { + uError("smlProcessChildTable failed to get child table info"); + return TSDB_CODE_SML_INTERNAL_ERROR; + } if (info->dataFormat) info->currTableDataCtx = tinfo->tableDataCtx; return TSDB_CODE_SUCCESS; } @@ -556,7 +562,11 @@ int32_t smlSetCTableName(SSmlTableInfo *oneTable, char *tbnameKey) { if (dst == NULL) { return TSDB_CODE_OUT_OF_MEMORY; } - ASSERT(oneTable->sTableNameLen < TSDB_TABLE_NAME_LEN); + if(oneTable->sTableNameLen >= TSDB_TABLE_NAME_LEN){ + uError("SML:smlSetCTableName super table name is too long"); + taosArrayDestroy(dst); + return TSDB_CODE_SML_INTERNAL_ERROR; + } char superName[TSDB_TABLE_NAME_LEN] = {0}; RandTableName rName = {dst, NULL, (uint8_t)oneTable->sTableNameLen, oneTable->childTableName}; if (tsSmlDot2Underline) { diff --git a/source/client/src/clientSmlLine.c b/source/client/src/clientSmlLine.c index 3456b7df17..f3efee0805 100644 --- a/source/client/src/clientSmlLine.c +++ b/source/client/src/clientSmlLine.c @@ -437,7 +437,11 @@ static int32_t smlParseColLine(SSmlHandle *info, char **sql, char *sqlEnd, SSmlL } (void)memcpy(tmp, kv.value, kv.length); PROCESS_SLASH_IN_FIELD_VALUE(tmp, kv.length); - ASSERT(kv.type != TSDB_DATA_TYPE_GEOMETRY); + if(kv.type == TSDB_DATA_TYPE_GEOMETRY) { + uError("SML:0x%" PRIx64 " smlParseColLine error, invalid GEOMETRY type.", info->id); + taosMemoryFree((void*)kv.value); + return TSDB_CODE_TSC_INVALID_VALUE; + } if(kv.type == TSDB_DATA_TYPE_VARBINARY){ taosMemoryFree((void*)kv.value); } diff --git a/source/client/src/clientTmq.c b/source/client/src/clientTmq.c index c9f166e565..d67f3565b9 100644 --- a/source/client/src/clientTmq.c +++ b/source/client/src/clientTmq.c @@ -37,7 +37,7 @@ struct SMqMgmt { static TdThreadOnce tmqInit = PTHREAD_ONCE_INIT; // initialize only once volatile int32_t tmqInitRes = 0; // initialize rsp code static struct SMqMgmt tmqMgmt = {0}; -static int8_t pollFlag = 0; +static int8_t pollFlag = 0; typedef struct { int32_t code; @@ -121,7 +121,7 @@ struct tmq_t { typedef struct SAskEpInfo { int32_t code; - tsem2_t sem; + tsem2_t sem; } SAskEpInfo; enum { @@ -191,7 +191,7 @@ typedef struct { } SMqPollRspWrapper; typedef struct { - tsem2_t rspSem; + tsem2_t rspSem; int32_t rspErr; } SMqSubscribeCbParam; @@ -219,12 +219,12 @@ typedef struct SMqVgCommon { } SMqVgCommon; typedef struct SMqSeekParam { - tsem2_t sem; + tsem2_t sem; int32_t code; } SMqSeekParam; typedef struct SMqCommittedParam { - tsem2_t sem; + tsem2_t sem; int32_t code; SMqVgOffset vgOffset; } SMqCommittedParam; @@ -242,18 +242,18 @@ typedef struct { int32_t waitingRspNum; int32_t code; tmq_commit_cb* callbackFn; - void* userParam; + void* userParam; } SMqCommitCbParamSet; typedef struct { SMqCommitCbParamSet* params; - char topicName[TSDB_TOPIC_FNAME_LEN]; - int32_t vgId; - int64_t consumerId; + char topicName[TSDB_TOPIC_FNAME_LEN]; + int32_t vgId; + int64_t consumerId; } SMqCommitCbParam; typedef struct SSyncCommitInfo { - tsem2_t sem; + tsem2_t sem; int32_t code; } SSyncCommitInfo; @@ -334,7 +334,7 @@ tmq_conf_res_t tmq_conf_set(tmq_conf_t* conf, const char* key, const char* value if (strcasecmp(key, "session.timeout.ms") == 0) { int64_t tmp = taosStr2int64(value); - if (tmp < 6000 || tmp > 1800000){ + if (tmp < 6000 || tmp > 1800000) { return TMQ_CONF_INVALID; } conf->sessionTimeoutMs = tmp; @@ -343,7 +343,7 @@ tmq_conf_res_t tmq_conf_set(tmq_conf_t* conf, const char* key, const char* value if (strcasecmp(key, "heartbeat.interval.ms") == 0) { int64_t tmp = taosStr2int64(value); - if (tmp < 1000 || tmp >= conf->sessionTimeoutMs){ + if (tmp < 1000 || tmp >= conf->sessionTimeoutMs) { return TMQ_CONF_INVALID; } conf->heartBeatIntervalMs = tmp; @@ -352,7 +352,7 @@ tmq_conf_res_t tmq_conf_set(tmq_conf_t* conf, const char* key, const char* value if (strcasecmp(key, "max.poll.interval.ms") == 0) { int64_t tmp = taosStr2int64(value); - if (tmp < 1000 || tmp > INT32_MAX){ + if (tmp < 1000 || tmp > INT32_MAX) { return TMQ_CONF_INVALID; } conf->maxPollIntervalMs = tmp; @@ -515,7 +515,7 @@ static int32_t doSendCommitMsg(tmq_t* tmq, int32_t vgId, SEpSet* epSet, STqOffse SEncoder encoder = {0}; tEncoderInit(&encoder, abuf, len); - if(tEncodeMqVgOffset(&encoder, &pOffset) < 0) { + if (tEncodeMqVgOffset(&encoder, &pOffset) < 0) { tEncoderClear(&encoder); taosMemoryFree(buf); return TSDB_CODE_INVALID_PARA; @@ -562,7 +562,7 @@ static int32_t doSendCommitMsg(tmq_t* tmq, int32_t vgId, SEpSet* epSet, STqOffse return code; } -static int32_t getTopicByName(tmq_t* tmq, const char* pTopicName, SMqClientTopic **topic) { +static int32_t getTopicByName(tmq_t* tmq, const char* pTopicName, SMqClientTopic** topic) { int32_t numOfTopics = taosArrayGetSize(tmq->clientTopics); for (int32_t i = 0; i < numOfTopics; ++i) { SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); @@ -577,8 +577,8 @@ static int32_t getTopicByName(tmq_t* tmq, const char* pTopicName, SMqClientTopic return TSDB_CODE_TMQ_INVALID_TOPIC; } -static int32_t prepareCommitCbParamSet(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* userParam, - int32_t rspNum, SMqCommitCbParamSet** ppParamSet) { +static int32_t prepareCommitCbParamSet(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* userParam, int32_t rspNum, + SMqCommitCbParamSet** ppParamSet) { SMqCommitCbParamSet* pParamSet = taosMemoryCalloc(1, sizeof(SMqCommitCbParamSet)); if (pParamSet == NULL) { return TSDB_CODE_OUT_OF_MEMORY; @@ -595,7 +595,7 @@ static int32_t prepareCommitCbParamSet(tmq_t* tmq, tmq_commit_cb* pCommitFp, voi static int32_t getClientVg(tmq_t* tmq, char* pTopicName, int32_t vgId, SMqClientVg** pVg) { SMqClientTopic* pTopic = NULL; - int32_t code = getTopicByName(tmq, pTopicName, &pTopic); + int32_t code = getTopicByName(tmq, pTopicName, &pTopic); if (code != 0) { tscError("consumer:0x%" PRIx64 " invalid topic name:%s", tmq->consumerId, pTopicName); return code; @@ -723,7 +723,7 @@ static void asyncCommitAllOffsets(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* us taosRUnLockLatch(&tmq->lock); goto end; } - int32_t numOfVgroups = taosArrayGetSize(pTopic->vgs); + int32_t numOfVgroups = taosArrayGetSize(pTopic->vgs); tscDebug("consumer:0x%" PRIx64 " commit offset for topics:%s, numOfVgs:%d", tmq->consumerId, pTopic->topicName, numOfVgroups); for (int32_t j = 0; j < numOfVgroups; j++) { @@ -769,7 +769,7 @@ static void asyncCommitAllOffsets(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* us if (pParamSet->waitingRspNum != 1) { // count down since waiting rsp num init as 1 code = commitRspCountDown(pParamSet, tmq->consumerId, "", 0); - if (code != 0){ + if (code != 0) { tscError("consumer:0x%" PRIx64 " commit rsp count down failed, code:%s", tmq->consumerId, tstrerror(code)); pParamSet = NULL; goto end; @@ -824,14 +824,14 @@ void tmqAssignDelayedCommitTask(void* param, void* tmrId) { } int32_t tmqHbCb(void* param, SDataBuf* pMsg, int32_t code) { - if (code != 0){ + if (code != 0) { goto _return; } if (pMsg == NULL || param == NULL) { code = TSDB_CODE_INVALID_PARA; goto _return; } - + SMqHbRsp rsp = {0}; code = tDeserializeSMqHbRsp(pMsg->pData, pMsg->len, &rsp); if (code != 0) { @@ -858,7 +858,7 @@ int32_t tmqHbCb(void* param, SDataBuf* pMsg, int32_t code) { taosWUnLockLatch(&tmq->lock); (void)taosReleaseRef(tmqMgmt.rsetId, refId); } - + tDestroySMqHbRsp(&rsp); _return: @@ -881,32 +881,32 @@ void tmqSendHbReq(void* param, void* tmrId) { req.epoch = tmq->epoch; req.pollFlag = atomic_load_8(&pollFlag); req.topics = taosArrayInit(taosArrayGetSize(tmq->clientTopics), sizeof(TopicOffsetRows)); - if (req.topics == NULL){ + if (req.topics == NULL) { return; } taosRLockLatch(&tmq->lock); for (int i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) { - SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); + SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); if (pTopic == NULL) { continue; } int32_t numOfVgroups = taosArrayGetSize(pTopic->vgs); TopicOffsetRows* data = taosArrayReserve(req.topics, 1); - if (data == NULL){ + if (data == NULL) { continue; } (void)strcpy(data->topicName, pTopic->topicName); data->offsetRows = taosArrayInit(numOfVgroups, sizeof(OffsetRows)); - if (data->offsetRows == NULL){ + if (data->offsetRows == NULL) { continue; } for (int j = 0; j < numOfVgroups; j++) { SMqClientVg* pVg = taosArrayGet(pTopic->vgs, j); - if (pVg == NULL){ + if (pVg == NULL) { continue; } - OffsetRows* offRows = taosArrayReserve(data->offsetRows, 1); - if (offRows == NULL){ + OffsetRows* offRows = taosArrayReserve(data->offsetRows, 1); + if (offRows == NULL) { continue; } offRows->vgId = pVg->vgId; @@ -964,7 +964,7 @@ void tmqSendHbReq(void* param, void* tmrId) { (void)atomic_val_compare_exchange_8(&pollFlag, 1, 0); OVER: tDestroySMqHbReq(&req); - if(tmrId != NULL){ + if (tmrId != NULL) { (void)taosTmrReset(tmqSendHbReq, tmq->heartBeatIntervalMs, param, tmqMgmt.timer, &tmq->hbLiveTimer); } (void)taosReleaseRef(tmqMgmt.rsetId, refId); @@ -1006,14 +1006,15 @@ void tmqHandleAllDelayedTask(tmq_t* pTmq) { continue; } tscDebug("consumer:0x%" PRIx64 " retrieve ep from mnode in 1s", pTmq->consumerId); - (void)taosTmrReset(tmqAssignAskEpTask, DEFAULT_ASKEP_INTERVAL, (void*)(pTmq->refId), tmqMgmt.timer, &pTmq->epTimer); + (void)taosTmrReset(tmqAssignAskEpTask, DEFAULT_ASKEP_INTERVAL, (void*)(pTmq->refId), tmqMgmt.timer, + &pTmq->epTimer); } else if (*pTaskType == TMQ_DELAYED_TASK__COMMIT) { tmq_commit_cb* pCallbackFn = pTmq->commitCb ? pTmq->commitCb : defaultCommitCbFn; asyncCommitAllOffsets(pTmq, pCallbackFn, pTmq->commitCbUserParam); tscDebug("consumer:0x%" PRIx64 " next commit to vnode(s) in %.2fs", pTmq->consumerId, pTmq->autoCommitInterval / 1000.0); (void)taosTmrReset(tmqAssignDelayedCommitTask, pTmq->autoCommitInterval, (void*)(pTmq->refId), tmqMgmt.timer, - &pTmq->commitTimer); + &pTmq->commitTimer); } else { tscError("consumer:0x%" PRIx64 " invalid task type:%d", pTmq->consumerId, *pTaskType); } @@ -1100,16 +1101,16 @@ int32_t tmq_subscription(tmq_t* tmq, tmq_list_t** topics) { taosRLockLatch(&tmq->lock); for (int i = 0; i < taosArrayGetSize(tmq->clientTopics); i++) { SMqClientTopic* topic = taosArrayGet(tmq->clientTopics, i); - if(topic == NULL) { + if (topic == NULL) { tscError("topic is null"); continue; } char* tmp = strchr(topic->topicName, '.'); - if(tmp == NULL) { + if (tmp == NULL) { tscError("topic name is invalid:%s", topic->topicName); continue; } - if(tmq_list_append(*topics, tmp+ 1) != 0) { + if (tmq_list_append(*topics, tmp + 1) != 0) { tscError("failed to append topic:%s", tmp + 1); continue; } @@ -1227,27 +1228,31 @@ tmq_t* tmq_consumer_new(tmq_conf_t* conf, char* errstr, int32_t errstrLen) { } code = taosOpenQueue(&pTmq->mqueue); if (code) { - tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), pTmq->groupId); + tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), + pTmq->groupId); SET_ERROR_MSG_TMQ("open queue failed") goto _failed; } code = taosOpenQueue(&pTmq->delayedTask); if (code) { - tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), pTmq->groupId); + tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), + pTmq->groupId); SET_ERROR_MSG_TMQ("open delayed task queue failed") goto _failed; } code = taosAllocateQall(&pTmq->qall); if (code) { - tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), pTmq->groupId); + tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), + pTmq->groupId); SET_ERROR_MSG_TMQ("allocate qall failed") goto _failed; } if (conf->groupId[0] == 0) { - tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), pTmq->groupId); + tscError("consumer:0x%" PRIx64 " setup failed since %s, groupId:%s", pTmq->consumerId, tstrerror(code), + pTmq->groupId); SET_ERROR_MSG_TMQ("malloc tmq element failed or group is empty") goto _failed; } @@ -1287,8 +1292,8 @@ tmq_t* tmq_consumer_new(tmq_conf_t* conf, char* errstr, int32_t errstrLen) { // init semaphore if (tsem2_init(&pTmq->rspSem, 0, 0) != 0) { - tscError("consumer:0x %" PRIx64 " setup failed since %s, consumer group %s", pTmq->consumerId, tstrerror(TAOS_SYSTEM_ERROR(errno)), - pTmq->groupId); + tscError("consumer:0x %" PRIx64 " setup failed since %s, consumer group %s", pTmq->consumerId, + tstrerror(TAOS_SYSTEM_ERROR(errno)), pTmq->groupId); SET_ERROR_MSG_TMQ("init t_sem failed") goto _failed; } @@ -1371,7 +1376,8 @@ int32_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { SName name = {0}; code = tNameSetDbName(&name, tmq->pTscObj->acctId, topic, strlen(topic)); if (code) { - tscError("consumer:0x%" PRIx64 " cgroup:%s, failed to set topic name, code:%d", tmq->consumerId, tmq->groupId, code); + tscError("consumer:0x%" PRIx64 " cgroup:%s, failed to set topic name, code:%d", tmq->consumerId, tmq->groupId, + code); goto FAIL; } char* topicFName = taosMemoryCalloc(1, TSDB_TOPIC_FNAME_LEN); @@ -1382,7 +1388,8 @@ int32_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { code = tNameExtractFullName(&name, topicFName); if (code) { - tscError("consumer:0x%" PRIx64 " cgroup:%s, failed to extract topic name, code:%d", tmq->consumerId, tmq->groupId, code); + tscError("consumer:0x%" PRIx64 " cgroup:%s, failed to extract topic name, code:%d", tmq->consumerId, tmq->groupId, + code); taosMemoryFree(topicFName); goto FAIL; } @@ -1459,7 +1466,8 @@ int32_t tmq_subscribe(tmq_t* tmq, const tmq_list_t* topic_list) { } tmq->epTimer = taosTmrStart(tmqAssignAskEpTask, DEFAULT_ASKEP_INTERVAL, (void*)(tmq->refId), tmqMgmt.timer); - tmq->commitTimer =taosTmrStart(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, (void*)(tmq->refId), tmqMgmt.timer); + tmq->commitTimer = + taosTmrStart(tmqAssignDelayedCommitTask, tmq->autoCommitInterval, (void*)(tmq->refId), tmqMgmt.timer); if (tmq->epTimer == NULL || tmq->commitTimer == NULL) { code = TSDB_CODE_TSC_INTERNAL_ERROR; goto FAIL; @@ -1516,12 +1524,12 @@ static void setVgIdle(tmq_t* tmq, char* topicName, int32_t vgId) { } int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) { - tmq_t* tmq = NULL; + tmq_t* tmq = NULL; SMqPollRspWrapper* pRspWrapper = NULL; - int8_t rspType = 0; - int32_t vgId = 0; - uint64_t requestId = 0; - SMqPollCbParam* pParam = (SMqPollCbParam*)param; + int8_t rspType = 0; + int32_t vgId = 0; + uint64_t requestId = 0; + SMqPollCbParam* pParam = (SMqPollCbParam*)param; if (pMsg == NULL) { return TSDB_CODE_TSC_INTERNAL_ERROR; } @@ -1530,7 +1538,7 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) { taosMemoryFreeClear(pMsg->pEpSet); return TSDB_CODE_TSC_INTERNAL_ERROR; } - int64_t refId = pParam->refId; + int64_t refId = pParam->refId; vgId = pParam->vgId; requestId = pParam->requestId; tmq = taosAcquireRef(tmqMgmt.rsetId, refId); @@ -1562,13 +1570,19 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) { if (msgEpoch < clientEpoch) { // do not write into queue since updating epoch reset tscWarn("consumer:0x%" PRIx64 - " msg discard from vgId:%d since from earlier epoch, rsp epoch %d, current epoch %d, reqId:0x%" PRIx64, + " msg discard from vgId:%d since from earlier epoch, rsp epoch %d, current epoch %d, qid:0x%" PRIx64, tmq->consumerId, vgId, msgEpoch, clientEpoch, requestId); code = TSDB_CODE_TMQ_CONSUMER_MISMATCH; goto END; } - ASSERT(msgEpoch == clientEpoch); + if (msgEpoch != clientEpoch) { + tscError("consumer:0x%" PRIx64 + " msg discard from vgId:%d since from earlier epoch, rsp epoch %d, current epoch %d, reqId:0x%" PRIx64, + tmq->consumerId, vgId, msgEpoch, clientEpoch, requestId); + code = TSDB_CODE_TMQ_CONSUMER_MISMATCH; + goto END; + } // handle meta rsp rspType = ((SMqRspHead*)pMsg->pData)->mqMsgType; pRspWrapper->tmqRspType = rspType; @@ -1589,7 +1603,7 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) { char buf[TSDB_OFFSET_LEN] = {0}; tFormatOffset(buf, TSDB_OFFSET_LEN, &pRspWrapper->dataRsp.common.rspOffset); - tscDebug("consumer:0x%" PRIx64 " recv poll rsp, vgId:%d, req ver:%" PRId64 ", rsp:%s type %d, reqId:0x%" PRIx64, + tscDebug("consumer:0x%" PRIx64 " recv poll rsp, vgId:%d, req ver:%" PRId64 ", rsp:%s type %d, qid:0x%" PRIx64, tmq->consumerId, vgId, pRspWrapper->dataRsp.common.reqOffset.version, buf, rspType, requestId); } else if (rspType == TMQ_MSG_TYPE__POLL_META_RSP) { SDecoder decoder = {0}; @@ -1621,23 +1635,24 @@ int32_t tmqPollCb(void* param, SDataBuf* pMsg, int32_t code) { } tDecoderClear(&decoder); (void)memcpy(&pRspWrapper->batchMetaRsp, pMsg->pData, sizeof(SMqRspHead)); - tscDebug("consumer:0x%" PRIx64 " recv poll batchmeta rsp, vgId:%d, reqId:0x%" PRIx64, tmq->consumerId, vgId,requestId); + tscDebug("consumer:0x%" PRIx64 " recv poll batchmeta rsp, vgId:%d, qid:0x%" PRIx64, tmq->consumerId, vgId, + requestId); } else { // invalid rspType tscError("consumer:0x%" PRIx64 " invalid rsp msg received, type:%d ignored", tmq->consumerId, rspType); } END: - if (pRspWrapper){ + if (pRspWrapper) { pRspWrapper->code = code; pRspWrapper->vgId = vgId; (void)strcpy(pRspWrapper->topicName, pParam->topicName); code = taosWriteQitem(tmq->mqueue, pRspWrapper); - if(code != 0){ + if (code != 0) { tscError("consumer:0x%" PRIx64 " put poll res into mqueue failed, code:%d", tmq->consumerId, code); } } int32_t total = taosQueueItemSize(tmq->mqueue); - tscDebug("consumer:0x%" PRIx64 " put poll res into mqueue, type:%d, vgId:%d, total in queue:%d, reqId:0x%" PRIx64, + tscDebug("consumer:0x%" PRIx64 " put poll res into mqueue, type:%d, vgId:%d, total in queue:%d, qid:0x%" PRIx64, tmq ? tmq->consumerId : 0, rspType, vgId, total, requestId); if (tmq) (void)tsem2_post(&tmq->rspSem); @@ -1676,7 +1691,7 @@ static void initClientTopicFromRsp(SMqClientTopic* pTopic, SMqSubTopicEp* pTopic } for (int32_t j = 0; j < vgNumGet; j++) { SMqSubVgEp* pVgEp = taosArrayGet(pTopicEp->vgs, j); - if (pVgEp == NULL){ + if (pVgEp == NULL) { continue; } (void)sprintf(vgKey, "%s:%d", pTopic->topicName, pVgEp->vgId); @@ -1712,7 +1727,7 @@ static void initClientTopicFromRsp(SMqClientTopic* pTopic, SMqSubTopicEp* pTopic clientVg.offsetInfo.committedOffset = offsetNew; clientVg.offsetInfo.beginOffset = offsetNew; } - if (taosArrayPush(pTopic->vgs, &clientVg) == NULL){ + if (taosArrayPush(pTopic->vgs, &clientVg) == NULL) { tscError("consumer:0x%" PRIx64 ", failed to push vg:%d into topic:%s", tmq->consumerId, pVgEp->vgId, pTopic->topicName); freeClientVg(&clientVg); @@ -1773,7 +1788,7 @@ static bool doUpdateLocalEp(tmq_t* tmq, int32_t epoch, const SMqAskEpRsp* pRsp) .commitOffset = pVgCur->offsetInfo.committedOffset, .numOfRows = pVgCur->numOfRows, .vgStatus = pVgCur->vgStatus}; - if(taosHashPut(pVgOffsetHashMap, vgKey, strlen(vgKey), &info, sizeof(SVgroupSaveInfo)) != 0){ + if (taosHashPut(pVgOffsetHashMap, vgKey, strlen(vgKey), &info, sizeof(SVgroupSaveInfo)) != 0) { tscError("consumer:0x%" PRIx64 ", failed to put vg:%d into hashmap", tmq->consumerId, pVgCur->vgId); } } @@ -1787,7 +1802,7 @@ static bool doUpdateLocalEp(tmq_t* tmq, int32_t epoch, const SMqAskEpRsp* pRsp) continue; } initClientTopicFromRsp(&topic, pTopicEp, pVgOffsetHashMap, tmq); - if(taosArrayPush(newTopics, &topic) == NULL){ + if (taosArrayPush(newTopics, &topic) == NULL) { tscError("consumer:0x%" PRIx64 ", failed to push topic:%s into new topics", tmq->consumerId, topic.topicName); freeClientTopic(&topic); } @@ -1865,7 +1880,10 @@ void changeByteEndian(char* pData) { // | version | total length | total rows | total columns | flag seg| block group id | column schema | each column // length | version: int32_t blockVersion = *(int32_t*)p; - ASSERT(blockVersion == BLOCK_VERSION_1); + if (blockVersion != BLOCK_VERSION_1) { + tscError("invalid block version:%d", blockVersion); + return; + } *(int32_t*)p = BLOCK_VERSION_2; p += sizeof(int32_t); @@ -1919,7 +1937,7 @@ static void tmqBuildRspFromWrapperInner(SMqPollRspWrapper* pWrapper, SMqClientVg if (!pDataRsp->withSchema) { // withSchema is false if subscribe subquery, true if subscribe db or stable pDataRsp->withSchema = true; pDataRsp->blockSchema = taosArrayInit(pDataRsp->blockNum, sizeof(void*)); - if (pDataRsp->blockSchema == NULL){ + if (pDataRsp->blockSchema == NULL) { tscError("failed to allocate memory for blockSchema"); return; } @@ -1938,7 +1956,7 @@ static void tmqBuildRspFromWrapperInner(SMqPollRspWrapper* pWrapper, SMqClientVg if (needTransformSchema) { // withSchema is false if subscribe subquery, true if subscribe db or stable SSchemaWrapper* schema = tCloneSSchemaWrapper(&pWrapper->topicHandle->schema); if (schema) { - if (taosArrayPush(pDataRsp->blockSchema, &schema) == NULL){ + if (taosArrayPush(pDataRsp->blockSchema, &schema) == NULL) { tscError("failed to push schema into blockSchema"); continue; } @@ -1947,7 +1965,8 @@ static void tmqBuildRspFromWrapperInner(SMqPollRspWrapper* pWrapper, SMqClientVg } } -int32_t tmqBuildRspFromWrapper(SMqPollRspWrapper* pWrapper, SMqClientVg* pVg, int64_t* numOfRows, SMqRspObj** ppRspObj) { +int32_t tmqBuildRspFromWrapper(SMqPollRspWrapper* pWrapper, SMqClientVg* pVg, int64_t* numOfRows, + SMqRspObj** ppRspObj) { SMqRspObj* pRspObj = taosMemoryCalloc(1, sizeof(SMqRspObj)); if (pRspObj == NULL) { return TSDB_CODE_OUT_OF_MEMORY; @@ -1959,7 +1978,8 @@ int32_t tmqBuildRspFromWrapper(SMqPollRspWrapper* pWrapper, SMqClientVg* pVg, in return 0; } -int32_t tmqBuildTaosxRspFromWrapper(SMqPollRspWrapper* pWrapper, SMqClientVg* pVg, int64_t* numOfRows, SMqTaosxRspObj** ppRspObj) { +int32_t tmqBuildTaosxRspFromWrapper(SMqPollRspWrapper* pWrapper, SMqClientVg* pVg, int64_t* numOfRows, + SMqTaosxRspObj** ppRspObj) { SMqTaosxRspObj* pRspObj = taosMemoryCalloc(1, sizeof(SMqTaosxRspObj)); if (pRspObj == NULL) { return TSDB_CODE_OUT_OF_MEMORY; @@ -2030,7 +2050,7 @@ static int32_t doTmqPollImpl(tmq_t* pTmq, SMqClientTopic* pTopic, SMqClientVg* p char offsetFormatBuf[TSDB_OFFSET_LEN] = {0}; tFormatOffset(offsetFormatBuf, tListLen(offsetFormatBuf), &pVg->offsetInfo.endOffset); code = asyncSendMsgToServer(pTmq->pTscObj->pAppInfo->pTransporter, &pVg->epSet, &transporterId, sendInfo); - tscDebug("consumer:0x%" PRIx64 " send poll to %s vgId:%d, code:%d, epoch %d, req:%s, reqId:0x%" PRIx64, + tscDebug("consumer:0x%" PRIx64 " send poll to %s vgId:%d, code:%d, epoch %d, req:%s, qid:0x%" PRIx64, pTmq->consumerId, pTopic->topicName, pVg->vgId, code, pTmq->epoch, offsetFormatBuf, req.reqId); if (code != 0) { return code; @@ -2056,10 +2076,10 @@ static int32_t tmqPollImpl(tmq_t* tmq, int64_t timeout) { for (int i = 0; i < numOfTopics; i++) { SMqClientTopic* pTopic = taosArrayGet(tmq->clientTopics, i); - if (pTopic == NULL){ + if (pTopic == NULL) { continue; } - int32_t numOfVg = taosArrayGetSize(pTopic->vgs); + int32_t numOfVg = taosArrayGetSize(pTopic->vgs); if (pTopic->noPrivilege) { tscDebug("consumer:0x%" PRIx64 " has no privilegr for topic:%s", tmq->consumerId, pTopic->topicName); continue; @@ -2069,7 +2089,7 @@ static int32_t tmqPollImpl(tmq_t* tmq, int64_t timeout) { if (pVg == NULL) { continue; } - int64_t elapsed = taosGetTimestampMs() - pVg->emptyBlockReceiveTs; + int64_t elapsed = taosGetTimestampMs() - pVg->emptyBlockReceiveTs; if (elapsed < EMPTY_BLOCK_POLL_IDLE_DURATION && elapsed >= 0) { // less than 10ms tscDebug("consumer:0x%" PRIx64 " epoch %d, vgId:%d idle for 10ms before start next poll", tmq->consumerId, tmq->epoch, pVg->vgId); @@ -2203,7 +2223,7 @@ static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) { tFormatOffset(buf, TSDB_OFFSET_LEN, &pDataRsp->rspOffset); if (pDataRsp->blockNum == 0) { tscDebug("consumer:0x%" PRIx64 " empty block received, vgId:%d, offset:%s, vg total:%" PRId64 - ", total:%" PRId64 ", reqId:0x%" PRIx64, + ", total:%" PRId64 ", qid:0x%" PRIx64, tmq->consumerId, pVg->vgId, buf, pVg->numOfRows, tmq->totalRows, pollRspWrapper->reqId); pVg->emptyBlockReceiveTs = taosGetTimestampMs(); tmqFreeRspWrapper(pRspWrapper); @@ -2220,13 +2240,13 @@ static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) { pVg->blockSleepForReplay = pRsp->rsp.sleepTime; if (pVg->blockSleepForReplay > 0) { if (taosTmrStart(tmqReplayTask, pVg->blockSleepForReplay, (void*)(tmq->refId), tmqMgmt.timer) == NULL) { - tscError("consumer:0x%" PRIx64 " failed to start replay timer, vgId:%d, sleep:%"PRId64, tmq->consumerId, - pVg->vgId, pVg->blockSleepForReplay); + tscError("consumer:0x%" PRIx64 " failed to start replay timer, vgId:%d, sleep:%" PRId64, + tmq->consumerId, pVg->vgId, pVg->blockSleepForReplay); } } } tscDebug("consumer:0x%" PRIx64 " process poll rsp, vgId:%d, offset:%s, blocks:%d, rows:%" PRId64 - ", vg total:%" PRId64 ", total:%" PRId64 ", reqId:0x%" PRIx64, + ", vg total:%" PRId64 ", total:%" PRId64 ", qid:0x%" PRIx64, tmq->consumerId, pVg->vgId, buf, pDataRsp->blockNum, numOfRows, pVg->numOfRows, tmq->totalRows, pollRspWrapper->reqId); taosFreeQitem(pRspWrapper); @@ -2302,7 +2322,7 @@ static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) { pollRspWrapper->batchMetaRsp.head.walsver, pollRspWrapper->batchMetaRsp.head.walever, tmq->consumerId, true); SMqBatchMetaRspObj* pRsp = NULL; - (void)tmqBuildBatchMetaRspFromWrapper(pollRspWrapper, &pRsp) ; + (void)tmqBuildBatchMetaRspFromWrapper(pollRspWrapper, &pRsp); taosFreeQitem(pRspWrapper); taosWUnLockLatch(&tmq->lock); return pRsp; @@ -2337,7 +2357,7 @@ static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) { tmq->consumerId, pDataRsp->blockNum != 0); if (pDataRsp->blockNum == 0) { - tscDebug("consumer:0x%" PRIx64 " taosx empty block received, vgId:%d, vg total:%" PRId64 ", reqId:0x%" PRIx64, + tscDebug("consumer:0x%" PRIx64 " taosx empty block received, vgId:%d, vg total:%" PRId64 ", qid:0x%" PRIx64, tmq->consumerId, pVg->vgId, pVg->numOfRows, pollRspWrapper->reqId); pVg->emptyBlockReceiveTs = taosGetTimestampMs(); tmqFreeRspWrapper(pRspWrapper); @@ -2349,9 +2369,9 @@ static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) { } // build rsp - int64_t numOfRows = 0; - SMqTaosxRspObj* pRsp = NULL; - if (tmqBuildTaosxRspFromWrapper(pollRspWrapper, pVg, &numOfRows, &pRsp) !=0 ) { + int64_t numOfRows = 0; + SMqTaosxRspObj* pRsp = NULL; + if (tmqBuildTaosxRspFromWrapper(pollRspWrapper, pVg, &numOfRows, &pRsp) != 0) { tscError("consumer:0x%" PRIx64 " build taosx rsp failed, vgId:%d", tmq->consumerId, pVg->vgId); } tmq->totalRows += numOfRows; @@ -2359,7 +2379,7 @@ static void* tmqHandleAllRsp(tmq_t* tmq, int64_t timeout) { char buf[TSDB_OFFSET_LEN] = {0}; tFormatOffset(buf, TSDB_OFFSET_LEN, &pVg->offsetInfo.endOffset); tscDebug("consumer:0x%" PRIx64 " process taosx poll rsp, vgId:%d, offset:%s, blocks:%d, rows:%" PRId64 - ", vg total:%" PRId64 ", total:%" PRId64 ", reqId:0x%" PRIx64, + ", vg total:%" PRId64 ", total:%" PRId64 ", qid:0x%" PRIx64, tmq->consumerId, pVg->vgId, buf, pDataRsp->blockNum, numOfRows, pVg->numOfRows, tmq->totalRows, pollRspWrapper->reqId); @@ -2471,7 +2491,7 @@ static void displayConsumeStatistics(tmq_t* pTmq) { tscDebug("consumer:0x%" PRIx64 " rows dist end", pTmq->consumerId); } -static int32_t innerClose(tmq_t* tmq){ +static int32_t innerClose(tmq_t* tmq) { if (tmq->status != TMQ_CONSUMER_STATUS__READY) { tscInfo("consumer:0x%" PRIx64 " not in ready state, unsubscribe it directly", tmq->consumerId); return 0; @@ -2485,7 +2505,7 @@ static int32_t innerClose(tmq_t* tmq){ tmqSendHbReq((void*)(tmq->refId), NULL); tmq_list_t* lst = tmq_list_new(); - if (lst == NULL){ + if (lst == NULL) { return TSDB_CODE_OUT_OF_MEMORY; } int32_t code = tmq_subscribe(tmq, lst); @@ -2499,7 +2519,7 @@ int32_t tmq_unsubscribe(tmq_t* tmq) { int32_t code = 0; if (atomic_load_8(&tmq->status) != TMQ_CONSUMER_STATUS__CLOSED) { code = innerClose(tmq); - if(code == 0){ + if (code == 0) { atomic_store_8(&tmq->status, TMQ_CONSUMER_STATUS__CLOSED); } } @@ -2514,12 +2534,12 @@ int32_t tmq_consumer_close(tmq_t* tmq) { int32_t code = 0; if (atomic_load_8(&tmq->status) != TMQ_CONSUMER_STATUS__CLOSED) { code = innerClose(tmq); - if(code == 0){ + if (code == 0) { atomic_store_8(&tmq->status, TMQ_CONSUMER_STATUS__CLOSED); } } - if (code == 0){ + if (code == 0) { (void)taosRemoveRef(tmqMgmt.rsetId, tmq->refId); } return code; @@ -2562,13 +2582,13 @@ const char* tmq_get_topic_name(TAOS_RES* res) { return NULL; } if (TD_RES_TMQ(res) || TD_RES_TMQ_METADATA(res) || TD_RES_TMQ_BATCH_META(res)) { - char *tmp = strchr(((SMqRspObjCommon*)res)->topic, '.'); + char* tmp = strchr(((SMqRspObjCommon*)res)->topic, '.'); if (tmp == NULL) { return NULL; } return tmp + 1; } else if (TD_RES_TMQ_META(res)) { - char *tmp = strchr(((SMqMetaRspObj*)res)->topic, '.'); + char* tmp = strchr(((SMqMetaRspObj*)res)->topic, '.'); if (tmp == NULL) { return NULL; } @@ -2584,13 +2604,13 @@ const char* tmq_get_db_name(TAOS_RES* res) { } if (TD_RES_TMQ(res) || TD_RES_TMQ_METADATA(res) || TD_RES_TMQ_BATCH_META(res)) { - char *tmp = strchr(((SMqRspObjCommon*)res)->db, '.'); + char* tmp = strchr(((SMqRspObjCommon*)res)->db, '.'); if (tmp == NULL) { return NULL; } return tmp + 1; } else if (TD_RES_TMQ_META(res)) { - char *tmp = strchr(((SMqMetaRspObj*)res)->db, '.'); + char* tmp = strchr(((SMqMetaRspObj*)res)->db, '.'); if (tmp == NULL) { return NULL; } @@ -2690,7 +2710,7 @@ int32_t tmq_commit_sync(tmq_t* tmq, const TAOS_RES* pRes) { int32_t code = 0; SSyncCommitInfo* pInfo = taosMemoryMalloc(sizeof(SSyncCommitInfo)); - if(pInfo == NULL) { + if (pInfo == NULL) { tscError("failed to allocate memory for sync commit"); return TSDB_CODE_OUT_OF_MEMORY; } @@ -2836,7 +2856,7 @@ int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) { if (param == NULL) { goto FAIL; } - + SMqAskEpCbParam* pParam = (SMqAskEpCbParam*)param; tmq_t* tmq = taosAcquireRef(tmqMgmt.rsetId, pParam->refId); if (tmq == NULL) { @@ -2857,7 +2877,7 @@ int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) { tscDebug("consumer:0x%" PRIx64 ", recv ep, msg epoch %d, current epoch %d", tmq->consumerId, head->epoch, epoch); if (pParam->sync) { SMqAskEpRsp rsp = {0}; - if(tDecodeSMqAskEpRsp(POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), &rsp) != NULL){ + if (tDecodeSMqAskEpRsp(POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), &rsp) != NULL) { (void)doUpdateLocalEp(tmq, head->epoch, &rsp); } tDeleteSMqAskEpRsp(&rsp); @@ -2871,10 +2891,10 @@ int32_t askEpCb(void* param, SDataBuf* pMsg, int32_t code) { pWrapper->tmqRspType = TMQ_MSG_TYPE__EP_RSP; pWrapper->epoch = head->epoch; (void)memcpy(&pWrapper->msg, pMsg->pData, sizeof(SMqRspHead)); - if (tDecodeSMqAskEpRsp(POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), &pWrapper->msg) == NULL){ + if (tDecodeSMqAskEpRsp(POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), &pWrapper->msg) == NULL) { tmqFreeRspWrapper((SMqRspWrapper*)pWrapper); taosFreeQitem(pWrapper); - }else{ + } else { (void)taosWriteQitem(tmq->mqueue, pWrapper); } } @@ -2972,7 +2992,7 @@ int32_t askEp(tmq_t* pTmq, void* param, bool sync, bool updateEpSet) { sendInfo->msgType = TDMT_MND_TMQ_ASK_EP; SEpSet epSet = getEpSet_s(&pTmq->pTscObj->pAppInfo->mgmtEp); - tscDebug("consumer:0x%" PRIx64 " ask ep from mnode, reqId:0x%" PRIx64, pTmq->consumerId, sendInfo->requestId); + tscDebug("consumer:0x%" PRIx64 " ask ep from mnode, qid:0x%" PRIx64, pTmq->consumerId, sendInfo->requestId); return asyncSendMsgToServer(pTmq->pTscObj->pAppInfo->pTransporter, &epSet, NULL, sendInfo); } @@ -3015,7 +3035,7 @@ int32_t tmqGetNextResInfo(TAOS_RES* res, bool convertUcs4, SReqResultInfo** pRes if (common->withSchema) { doFreeReqResultInfo(&pRspObj->resInfo); SSchemaWrapper* pSW = (SSchemaWrapper*)taosArrayGetP(common->blockSchema, pRspObj->resIter); - if (pSW){ + if (pSW) { TAOS_CHECK_RETURN(setResSchemaInfo(&pRspObj->resInfo, pSW->pSchema, pSW->nCols)); } } @@ -3032,9 +3052,9 @@ int32_t tmqGetNextResInfo(TAOS_RES* res, bool convertUcs4, SReqResultInfo** pRes pRspObj->resInfo.precision = precision; pRspObj->resInfo.totalRows += pRspObj->resInfo.numOfRows; - int32_t code = setResultDataPtr(&pRspObj->resInfo, pRspObj->resInfo.fields, pRspObj->resInfo.numOfCols, pRspObj->resInfo.numOfRows, - convertUcs4); - if (code != 0){ + int32_t code = setResultDataPtr(&pRspObj->resInfo, pRspObj->resInfo.fields, pRspObj->resInfo.numOfCols, + pRspObj->resInfo.numOfRows, convertUcs4); + if (code != 0) { return code; } *pResInfo = &pRspObj->resInfo; @@ -3062,18 +3082,18 @@ static int32_t tmqGetWalInfoCb(void* param, SDataBuf* pMsg, int32_t code) { tDecoderInit(&decoder, POINTER_SHIFT(pMsg->pData, sizeof(SMqRspHead)), pMsg->len - sizeof(SMqRspHead)); code = tDecodeMqDataRsp(&decoder, &rsp); tDecoderClear(&decoder); - if (code != 0){ + if (code != 0) { goto END; } - SMqRspHead* pHead = pMsg->pData; + SMqRspHead* pHead = pMsg->pData; tmq_topic_assignment assignment = {.begin = pHead->walsver, .end = pHead->walever + 1, .currentOffset = rsp.common.rspOffset.version, .vgId = pParam->vgId}; (void)taosThreadMutexLock(&pCommon->mutex); - if(taosArrayPush(pCommon->pList, &assignment) == NULL){ + if (taosArrayPush(pCommon->pList, &assignment) == NULL) { tscError("consumer:0x%" PRIx64 " failed to push the wal info from vgId:%d for topic:%s", pCommon->consumerId, pParam->vgId, pCommon->pTopicName); code = TSDB_CODE_TSC_INTERNAL_ERROR; @@ -3184,7 +3204,7 @@ int64_t getCommittedFromServer(tmq_t* tmq, char* tname, int32_t vgId, SEpSet* ep taosMemoryFree(sendInfo); return TSDB_CODE_OUT_OF_MEMORY; } - if (tsem2_init(&pParam->sem, 0, 0) != 0){ + if (tsem2_init(&pParam->sem, 0, 0) != 0) { taosMemoryFree(buf); taosMemoryFree(sendInfo); taosMemoryFree(pParam); @@ -3348,7 +3368,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a taosWLockLatch(&tmq->lock); SMqClientTopic* pTopic = NULL; - int32_t code = getTopicByName(tmq, tname, &pTopic); + int32_t code = getTopicByName(tmq, tname, &pTopic); if (code != 0) { tscError("consumer:0x%" PRIx64 " invalid topic name:%s", tmq->consumerId, pTopicName); goto end; @@ -3358,10 +3378,10 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a *numOfAssignment = taosArrayGetSize(pTopic->vgs); for (int32_t j = 0; j < (*numOfAssignment); ++j) { SMqClientVg* pClientVg = taosArrayGet(pTopic->vgs, j); - if (pClientVg == NULL){ + if (pClientVg == NULL) { continue; } - int32_t type = pClientVg->offsetInfo.beginOffset.type; + int32_t type = pClientVg->offsetInfo.beginOffset.type; if (isInSnapshotMode(type, tmq->useSnapshot)) { tscError("consumer:0x%" PRIx64 " offset type:%d not wal version, assignment not allowed", tmq->consumerId, type); code = TSDB_CODE_TMQ_SNAPSHOT_ERROR; @@ -3381,7 +3401,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a for (int32_t j = 0; j < (*numOfAssignment); ++j) { SMqClientVg* pClientVg = taosArrayGet(pTopic->vgs, j); - if (pClientVg == NULL){ + if (pClientVg == NULL) { continue; } if (pClientVg->offsetInfo.beginOffset.type != TMQ_OFFSET__LOG) { @@ -3410,7 +3430,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a code = TSDB_CODE_OUT_OF_MEMORY; goto end; } - if (tsem2_init(&pCommon->rsp, 0, 0) != 0){ + if (tsem2_init(&pCommon->rsp, 0, 0) != 0) { code = TSDB_CODE_OUT_OF_MEMORY; goto end; } @@ -3420,7 +3440,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a for (int32_t i = 0; i < (*numOfAssignment); ++i) { SMqClientVg* pClientVg = taosArrayGet(pTopic->vgs, i); - if (pClientVg == NULL){ + if (pClientVg == NULL) { continue; } SMqVgWalInfoParam* pParam = taosMemoryMalloc(sizeof(SMqVgWalInfoParam)); @@ -3479,8 +3499,8 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a char offsetFormatBuf[TSDB_OFFSET_LEN] = {0}; tFormatOffset(offsetFormatBuf, tListLen(offsetFormatBuf), &pClientVg->offsetInfo.beginOffset); - tscInfo("consumer:0x%" PRIx64 " %s retrieve wal info vgId:%d, epoch %d, req:%s, reqId:0x%" PRIx64, - tmq->consumerId, pTopic->topicName, pClientVg->vgId, tmq->epoch, offsetFormatBuf, req.reqId); + tscInfo("consumer:0x%" PRIx64 " %s retrieve wal info vgId:%d, epoch %d, req:%s, qid:0x%" PRIx64, tmq->consumerId, + pTopic->topicName, pClientVg->vgId, tmq->epoch, offsetFormatBuf, req.reqId); code = asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &pClientVg->epSet, &transporterId, sendInfo); if (code != 0) { goto end; @@ -3504,7 +3524,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a for (int32_t i = 0; i < taosArrayGetSize(pTopic->vgs); ++i) { SMqClientVg* pClientVg = taosArrayGet(pTopic->vgs, i); - if (pClientVg == NULL){ + if (pClientVg == NULL) { continue; } if (pClientVg->vgId != p->vgId) { @@ -3631,7 +3651,7 @@ int32_t tmq_offset_seek(tmq_t* tmq, const char* pTopicName, int32_t vgId, int64_ taosMemoryFree(sendInfo); return TSDB_CODE_OUT_OF_MEMORY; } - if (tsem2_init(&pParam->sem, 0, 0) != 0){ + if (tsem2_init(&pParam->sem, 0, 0) != 0) { taosMemoryFree(msg); taosMemoryFree(sendInfo); taosMemoryFree(pParam); diff --git a/source/client/test/clientTests.cpp b/source/client/test/clientTests.cpp index 3e9f16a2a1..307ef7e06f 100644 --- a/source/client/test/clientTests.cpp +++ b/source/client/test/clientTests.cpp @@ -44,7 +44,10 @@ void printSubResults(void* pRes, int32_t* totalRows) { } TAOS_FIELD* fields = taos_fetch_fields(pRes); - assert(fields != NULL); + if(fields == NULL) { + std::cout << "fields is null" << std::endl; + break; + } int32_t numOfFields = taos_field_count(pRes); int32_t precision = taos_result_precision(pRes); (void)taos_print_row(buf, row, fields, numOfFields); @@ -61,7 +64,6 @@ void showDB(TAOS* pConn) { TAOS_ROW pRow = NULL; TAOS_FIELD* pFields = taos_fetch_fields(pRes); - assert(pFields != NULL); int32_t numOfFields = taos_num_fields(pRes); char str[512] = {0}; @@ -173,7 +175,6 @@ void createNewTable(TAOS* pConn, int32_t index, int32_t numOfRows, int64_t start i + 14, pVarchar, startTs + 15, i + 15, pVarchar, startTs + 16, i + 16, pVarchar, startTs + 17, i + 17, pVarchar, startTs + 18, i + 18, pVarchar, startTs + 19, i + 19, pVarchar); TAOS_RES* p = taos_query(pConn, sql); - assert(p != NULL); if (taos_errno(p) != 0) { (void)printf("failed to insert data, reason:%s\n", taos_errstr(p)); } @@ -223,9 +224,7 @@ void tmq_commit_cb_print(tmq_t* pTmq, int32_t code, void* param) { void* doConsumeData(void* param) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); tmq_conf_t* conf = tmq_conf_new(); - assert(conf != NULL); (void)tmq_conf_set(conf, "enable.auto.commit", "true"); (void)tmq_conf_set(conf, "auto.commit.interval.ms", "1000"); (void)tmq_conf_set(conf, "group.id", "cgrpName41"); @@ -237,12 +236,10 @@ void* doConsumeData(void* param) { (void)tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, NULL); tmq_t* tmq = tmq_consumer_new(conf, NULL, 0); - assert(tmq != NULL); tmq_conf_destroy(conf); // 创建订阅 topics 列表 tmq_list_t* topicList = tmq_list_new(); - assert(topicList != NULL); (void)tmq_list_append(topicList, "topic_t2"); // 启动订阅 @@ -279,7 +276,6 @@ void* doConsumeData(void* param) { } fields = taos_fetch_fields(pRes); - assert(fields != NULL); numOfFields = taos_field_count(pRes); precision = taos_result_precision(pRes); (void)taos_print_row(buf, row, fields, numOfFields); @@ -336,7 +332,7 @@ TEST(clientCase, connect_Test) { TEST(clientCase, create_user_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "create user abc pass 'abc'"); if (taos_errno(pRes) != TSDB_CODE_SUCCESS) { @@ -349,7 +345,7 @@ TEST(clientCase, create_user_Test) { TEST(clientCase, create_account_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "create account aabc pass 'abc'"); if (taos_errno(pRes) != TSDB_CODE_SUCCESS) { @@ -362,7 +358,7 @@ TEST(clientCase, create_account_Test) { TEST(clientCase, drop_account_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "drop account aabc"); if (taos_errno(pRes) != TSDB_CODE_SUCCESS) { @@ -375,13 +371,13 @@ TEST(clientCase, drop_account_Test) { TEST(clientCase, show_user_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "show users"); TAOS_ROW pRow = NULL; TAOS_FIELD* pFields = taos_fetch_fields(pRes); - assert(pFields != NULL); + ASSERT_NE(pFields, nullptr); int32_t numOfFields = taos_num_fields(pRes); char str[512] = {0}; @@ -396,7 +392,7 @@ TEST(clientCase, show_user_Test) { TEST(clientCase, drop_user_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "drop user abc"); if (taos_errno(pRes) != TSDB_CODE_SUCCESS) { @@ -409,13 +405,13 @@ TEST(clientCase, drop_user_Test) { TEST(clientCase, show_db_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "show databases"); TAOS_ROW pRow = NULL; TAOS_FIELD* pFields = taos_fetch_fields(pRes); - assert(pFields != NULL); + ASSERT_NE(pFields, nullptr); int32_t numOfFields = taos_num_fields(pRes); char str[512] = {0}; @@ -429,7 +425,7 @@ TEST(clientCase, show_db_Test) { TEST(clientCase, create_db_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "create database abc1 vgroups 2"); if (taos_errno(pRes) != 0) { @@ -454,7 +450,7 @@ TEST(clientCase, create_db_Test) { TEST(clientCase, create_dnode_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "create dnode abc1 port 7000"); if (taos_errno(pRes) != 0) { @@ -473,7 +469,7 @@ TEST(clientCase, create_dnode_Test) { TEST(clientCase, drop_dnode_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "drop dnode 3"); if (taos_errno(pRes) != 0) { @@ -498,7 +494,7 @@ TEST(clientCase, drop_dnode_Test) { TEST(clientCase, use_db_test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { @@ -517,7 +513,7 @@ TEST(clientCase, use_db_test) { TEST(clientCase, create_stable_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "create database if not exists abc1 vgroups 2"); if (taos_errno(pRes) != 0) { @@ -545,7 +541,7 @@ TEST(clientCase, create_stable_Test) { TEST(clientCase, create_table_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "use abc1"); taos_free_result(pRes); @@ -564,7 +560,7 @@ TEST(clientCase, create_table_Test) { TEST(clientCase, create_ctable_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { @@ -589,7 +585,7 @@ TEST(clientCase, create_ctable_Test) { TEST(clientCase, show_stable_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != nullptr); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "show abc1.stables"); if (taos_errno(pRes) != 0) { @@ -614,7 +610,7 @@ TEST(clientCase, show_stable_Test) { TEST(clientCase, show_vgroup_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { @@ -716,7 +712,7 @@ TEST(clientCase, create_multiple_tables) { TEST(clientCase, show_table_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "show tables"); if (taos_errno(pRes) != 0) { @@ -760,7 +756,7 @@ TEST(clientCase, generated_request_id_test) { if (result != nullptr) { // (void)printf("0x%llx, index:%d\n", v, i); } - assert(result == nullptr); + ASSERT_EQ(result, nullptr); (void)taosHashPut(phash, &v, sizeof(v), NULL, 0); } @@ -1090,7 +1086,7 @@ TEST(clientCase, sub_db_test) { if (row == NULL) break; fields = taos_fetch_fields(pRes); - assert(fields != NULL); + ASSERT_NE(fields, nullptr); numOfFields = taos_field_count(pRes); precision = taos_result_precision(pRes); rows++; @@ -1226,7 +1222,7 @@ TEST(clientCase, td_25129) { (void)tmq_conf_set(conf, "msg.with.table.name", "true"); tmq_t* tmq = tmq_consumer_new(conf, NULL, 0); - assert(tmq != NULL); + ASSERT_NE(tmq, nullptr); tmq_conf_destroy(conf); char topicName[128] = "tp"; @@ -1393,7 +1389,7 @@ TEST(clientCase, sub_tb_test) { (void)tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, NULL); tmq_t* tmq = tmq_consumer_new(conf, NULL, 0); - assert(tmq != NULL); + ASSERT_NE(tmq, nullptr); tmq_conf_destroy(conf); // 创建订阅 topics 列表 diff --git a/source/client/test/tmqTest.cpp b/source/client/test/tmqTest.cpp index 00ed16944c..c7a75e352a 100644 --- a/source/client/test/tmqTest.cpp +++ b/source/client/test/tmqTest.cpp @@ -40,7 +40,7 @@ TEST(testCase, driverInit_Test) { TEST(testCase, create_topic_ctb_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { @@ -64,7 +64,7 @@ TEST(testCase, create_topic_ctb_Test) { TEST(testCase, create_topic_stb_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, nullptr); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { @@ -90,7 +90,7 @@ TEST(testCase, create_topic_stb_Test) { #if 0 TEST(testCase, tmq_subscribe_ctb_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, NULL); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { @@ -116,7 +116,7 @@ TEST(testCase, tmq_subscribe_ctb_Test) { TEST(testCase, tmq_subscribe_stb_Test) { TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); - assert(pConn != NULL); + ASSERT_NE(pConn, NULL); TAOS_RES* pRes = taos_query(pConn, "use abc1"); if (taos_errno(pRes) != 0) { diff --git a/source/common/src/cos.c b/source/common/src/cos.c index 6e9c7dd50d..3b5fca0fea 100644 --- a/source/common/src/cos.c +++ b/source/common/src/cos.c @@ -1024,7 +1024,7 @@ int32_t s3PutObjectFromFile2(const char *file, const char *object_name, int8_t w } static int32_t s3PutObjectFromFileOffsetByEp(const char *file, const char *object_name, int64_t offset, int64_t size, - int8_t epIndex) { + int8_t epIndex) { int32_t code = 0; int32_t lmtime = 0; const char *filename = 0; @@ -1136,7 +1136,10 @@ static S3Status listBucketCallback(int isTruncated, const char *nextMarker, int const S3ListBucketContent *content = &(contents[i]); // printf("%-50s", content->key); char *object_key = strdup(content->key); - (void)taosArrayPush(data->objectArray, &object_key); + if (!taosArrayPush(data->objectArray, &object_key)) { + taosMemoryFree(object_key); + return S3StatusOutOfMemory; + } } data->keyCount += contentsCount; diff --git a/source/common/src/tdatablock.c b/source/common/src/tdatablock.c index 2181496635..d0bae5d6a7 100644 --- a/source/common/src/tdatablock.c +++ b/source/common/src/tdatablock.c @@ -973,6 +973,9 @@ int32_t blockDataToBuf(char* buf, const SSDataBlock* pBlock) { int32_t blockDataFromBuf(SSDataBlock* pBlock, const char* buf) { int32_t numOfRows = *(int32_t*)buf; + if (numOfRows == 0) { + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } int32_t code = blockDataEnsureCapacity(pBlock, numOfRows); if (code) { return code; @@ -1670,6 +1673,8 @@ int32_t assignOneDataBlock(SSDataBlock* dst, const SSDataBlock* src) { int32_t code = 0; dst->info = src->info; + dst->info.pks[0].pData = NULL; + dst->info.pks[1].pData = NULL; dst->info.rows = 0; dst->info.capacity = 0; @@ -1707,6 +1712,8 @@ int32_t assignOneDataBlock(SSDataBlock* dst, const SSDataBlock* src) { uint32_t cap = dst->info.capacity; dst->info = src->info; + dst->info.pks[0].pData = NULL; + dst->info.pks[1].pData = NULL; dst->info.capacity = cap; return code; } @@ -1737,6 +1744,8 @@ int32_t copyDataBlock(SSDataBlock* pDst, const SSDataBlock* pSrc) { uint32_t cap = pDst->info.capacity; pDst->info = pSrc->info; + pDst->info.pks[0].pData = NULL; + pDst->info.pks[1].pData = NULL; code = copyPkVal(&pDst->info, &pSrc->info); if (code != TSDB_CODE_SUCCESS) { uError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); @@ -1854,6 +1863,8 @@ int32_t blockCopyOneRow(const SSDataBlock* pDataBlock, int32_t rowIdx, SSDataBlo } pBlock->info = pDataBlock->info; + pBlock->info.pks[0].pData = NULL; + pBlock->info.pks[1].pData = NULL; pBlock->info.rows = 0; pBlock->info.capacity = 0; @@ -1916,20 +1927,20 @@ int32_t copyPkVal(SDataBlockInfo* pDst, const SDataBlockInfo* pSrc) { // prepare the pk buffer if needed SValue* p = &pDst->pks[0]; - p->type = pDst->pks[0].type; - p->pData = taosMemoryCalloc(1, pDst->pks[0].nData); + p->type = pSrc->pks[0].type; + p->pData = taosMemoryCalloc(1, pSrc->pks[0].nData); QUERY_CHECK_NULL(p->pData, code, lino, _end, terrno); - p->nData = pDst->pks[0].nData; - memcpy(p->pData, pDst->pks[0].pData, p->nData); + p->nData = pSrc->pks[0].nData; + memcpy(p->pData, pSrc->pks[0].pData, p->nData); p = &pDst->pks[1]; - p->type = pDst->pks[1].type; - p->pData = taosMemoryCalloc(1, pDst->pks[1].nData); + p->type = pSrc->pks[1].type; + p->pData = taosMemoryCalloc(1, pSrc->pks[1].nData); QUERY_CHECK_NULL(p->pData, code, lino, _end, terrno); - p->nData = pDst->pks[1].nData; - memcpy(p->pData, pDst->pks[1].pData, p->nData); + p->nData = pSrc->pks[1].nData; + memcpy(p->pData, pSrc->pks[1].pData, p->nData); _end: if (code != TSDB_CODE_SUCCESS) { @@ -1951,6 +1962,8 @@ int32_t createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData, SSDataB } pDstBlock->info = pDataBlock->info; + pDstBlock->info.pks[0].pData = NULL; + pDstBlock->info.pks[1].pData = NULL; pDstBlock->info.rows = 0; pDstBlock->info.capacity = 0; @@ -2878,6 +2891,7 @@ int32_t buildCtbNameByGroupIdImpl(const char* stbFullName, uint64_t groupId, cha return code; } +// return length of encoded data, return -1 if failed int32_t blockEncode(const SSDataBlock* pBlock, char* data, int32_t numOfCols) { int32_t dataLen = 0; @@ -2911,7 +2925,7 @@ int32_t blockEncode(const SSDataBlock* pBlock, char* data, int32_t numOfCols) { for (int32_t i = 0; i < numOfCols; ++i) { SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, i); if (pColInfoData == NULL) { - return terrno; + return -1; } *((int8_t*)data) = pColInfoData->info.type; @@ -2930,7 +2944,7 @@ int32_t blockEncode(const SSDataBlock* pBlock, char* data, int32_t numOfCols) { for (int32_t col = 0; col < numOfCols; ++col) { SColumnInfoData* pColRes = (SColumnInfoData*)taosArrayGet(pBlock->pDataBlock, col); if (pColRes == NULL) { - return terrno; + return -1; } // copy the null bitmap @@ -2981,7 +2995,6 @@ int32_t blockEncode(const SSDataBlock* pBlock, char* data, int32_t numOfCols) { *actualLen = dataLen; *groupId = pBlock->info.id.groupId; - ASSERT(dataLen > 0); return dataLen; } diff --git a/source/common/src/tdataformat.c b/source/common/src/tdataformat.c index 5caf93de65..8e44c9c4b4 100644 --- a/source/common/src/tdataformat.c +++ b/source/common/src/tdataformat.c @@ -119,7 +119,6 @@ static FORCE_INLINE void tRowBuildScanAddValue(SRowBuildScanInfo *sinfo, SColVal bool isPK = ((pTColumn->flags & COL_IS_KEY) != 0); if (isPK) { - ASSERTS(sinfo->numOfPKs < TD_MAX_PK_COLS, "too many primary keys"); sinfo->tupleIndices[sinfo->numOfPKs].type = colVal->value.type; sinfo->tupleIndices[sinfo->numOfPKs].offset = IS_VAR_DATA_TYPE(pTColumn->type) ? sinfo->tupleVarSize + sinfo->tupleFixedSize : pTColumn->offset; @@ -149,9 +148,15 @@ static int32_t tRowBuildScan(SArray *colVals, const STSchema *schema, SRowBuildS int32_t numOfColVals = TARRAY_SIZE(colVals); SColVal *colValArray = (SColVal *)TARRAY_DATA(colVals); - ASSERT(numOfColVals > 0); - ASSERT(colValArray[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID); - ASSERT(colValArray[0].value.type == TSDB_DATA_TYPE_TIMESTAMP); + if (!(numOfColVals > 0)) { + return TSDB_CODE_INVALID_PARA; + } + if (!(colValArray[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID)) { + return TSDB_CODE_INVALID_PARA; + } + if (!(colValArray[0].value.type == TSDB_DATA_TYPE_TIMESTAMP)) { + return TSDB_CODE_INVALID_PARA; + } *sinfo = (SRowBuildScanInfo){ .tupleFixedSize = schema->flen, @@ -166,7 +171,10 @@ static int32_t tRowBuildScan(SArray *colVals, const STSchema *schema, SRowBuildS } if (colValArray[colValIndex].cid == schema->columns[i].colId) { - ASSERT(colValArray[colValIndex].value.type == schema->columns[i].type); + if (!(colValArray[colValIndex].value.type == schema->columns[i].type)) { + code = TSDB_CODE_INVALID_PARA; + goto _exit; + } if (COL_VAL_IS_VALUE(&colValArray[colValIndex])) { tRowBuildScanAddValue(sinfo, &colValArray[colValIndex], schema->columns + i); @@ -272,7 +280,6 @@ static int32_t tRowBuildTupleRow(SArray *aColVal, const SRowBuildScanInfo *sinfo (*ppRow)->ts = colValArray[0].value.val; if (sinfo->tupleFlag == HAS_NONE || sinfo->tupleFlag == HAS_NULL) { - ASSERT(sinfo->tupleRowSize == sizeof(SRow)); return 0; } @@ -285,7 +292,6 @@ static int32_t tRowBuildTupleRow(SArray *aColVal, const SRowBuildScanInfo *sinfo for (int32_t i = 0; i < sinfo->numOfPKs; i++) { primaryKeys += tPutPrimaryKeyIndex(primaryKeys, sinfo->tupleIndices + i); } - ASSERT(primaryKeys == bitmap); // bitmap + fixed + varlen int32_t numOfColVals = TARRAY_SIZE(aColVal); @@ -356,7 +362,9 @@ static int32_t tRowBuildKVRow(SArray *aColVal, const SRowBuildScanInfo *sinfo, c (*ppRow)->len = sinfo->kvRowSize; (*ppRow)->ts = colValArray[0].value.val; - ASSERT(sinfo->flag != HAS_NONE && sinfo->flag != HAS_NULL); + if (!(sinfo->flag != HAS_NONE && sinfo->flag != HAS_NULL)) { + return TSDB_CODE_INVALID_PARA; + } uint8_t *primaryKeys = (*ppRow)->data; SKVIdx *indices = (SKVIdx *)(primaryKeys + sinfo->kvPKSize); @@ -367,7 +375,6 @@ static int32_t tRowBuildKVRow(SArray *aColVal, const SRowBuildScanInfo *sinfo, c for (int32_t i = 0; i < sinfo->numOfPKs; i++) { primaryKeys += tPutPrimaryKeyIndex(primaryKeys, sinfo->kvIndices + i); } - ASSERT(primaryKeys == (uint8_t *)indices); int32_t numOfColVals = TARRAY_SIZE(aColVal); int32_t colValIndex = 1; @@ -504,8 +511,8 @@ _exit: } int32_t tRowGet(SRow *pRow, STSchema *pTSchema, int32_t iCol, SColVal *pColVal) { - ASSERT(iCol < pTSchema->numOfCols); - ASSERT(pRow->sver == pTSchema->version); + if (!(iCol < pTSchema->numOfCols)) return TSDB_CODE_INVALID_PARA; + if (!(pRow->sver == pTSchema->version)) return TSDB_CODE_INVALID_PARA; STColumn *pTColumn = pTSchema->columns + iCol; @@ -786,7 +793,7 @@ struct SRowIter { }; int32_t tRowIterOpen(SRow *pRow, STSchema *pTSchema, SRowIter **ppIter) { - ASSERT(pRow->sver == pTSchema->version); + if (!(pRow->sver == pTSchema->version)) return TSDB_CODE_INVALID_PARA; int32_t code = 0; @@ -839,7 +846,6 @@ int32_t tRowIterOpen(SRow *pRow, STSchema *pTSchema, SRowIter **ppIter) { pIter->pv = pIter->pf + pTSchema->flen; break; default: - ASSERT(0); break; } } @@ -928,7 +934,8 @@ SColVal *tRowIterNext(SRowIter *pIter) { pIter->cv = COL_VAL_NONE(pTColumn->colId, pTColumn->type); goto _exit; } else { - ASSERT(0); + uError("unexpected column id %d, %d", cid, pTColumn->colId); + goto _exit; } } else { pIter->cv = COL_VAL_NONE(pTColumn->colId, pTColumn->type); @@ -952,7 +959,6 @@ SColVal *tRowIterNext(SRowIter *pIter) { bv = GET_BIT2(pIter->pb, pIter->iTColumn - 1); break; default: - ASSERT(0); break; } @@ -1070,14 +1076,15 @@ static int32_t tRowTupleUpsertColData(SRow *pRow, STSchema *pTSchema, SColData * pv = pf + pTSchema->flen; break; default: - ASSERTS(0, "Invalid row flag"); return TSDB_CODE_INVALID_DATA_FMT; } while (pColData) { if (pTColumn) { if (pTColumn->colId == pColData->cid) { - ASSERT(pTColumn->type == pColData->type); + if (!(pTColumn->type == pColData->type)) { + return TSDB_CODE_INVALID_PARA; + } if (pb) { uint8_t bv; switch (pRow->flag) { @@ -1095,7 +1102,6 @@ static int32_t tRowTupleUpsertColData(SRow *pRow, STSchema *pTSchema, SColData * bv = GET_BIT2(pb, iTColumn - 1); break; default: - ASSERTS(0, "Invalid row flag"); return TSDB_CODE_INVALID_DATA_FMT; } @@ -1178,7 +1184,7 @@ static int32_t tRowKVUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *aCo } else if (pRow->flag & KV_FLG_BIG) { pv = pKVIdx->idx + (pKVIdx->nCol << 2); } else { - ASSERT(0); + return TSDB_CODE_INVALID_PARA; } while (pColData) { @@ -1193,7 +1199,6 @@ static int32_t tRowKVUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *aCo } else if (pRow->flag & KV_FLG_BIG) { pData = pv + ((uint32_t *)pKVIdx->idx)[iCol]; } else { - ASSERTS(0, "Invalid KV row format"); return TSDB_CODE_INVALID_DATA_FMT; } @@ -1256,8 +1261,8 @@ _exit: * flag < 0: backward update */ int32_t tRowUpsertColData(SRow *pRow, STSchema *pTSchema, SColData *aColData, int32_t nColData, int32_t flag) { - ASSERT(pRow->sver == pTSchema->version); - ASSERT(nColData > 0); + if (!(pRow->sver == pTSchema->version)) return TSDB_CODE_INVALID_PARA; + if (!(nColData > 0)) return TSDB_CODE_INVALID_PARA; if (pRow->flag == HAS_NONE) { return tRowNoneUpsertColData(aColData, nColData, flag); @@ -1277,8 +1282,6 @@ void tRowGetPrimaryKey(SRow *row, SRowKey *key) { return; } - ASSERT(row->numOfPKs <= TD_MAX_PK_COLS); - SPrimaryKeyIndex indices[TD_MAX_PK_COLS]; uint8_t *data = row->data; @@ -1317,8 +1320,6 @@ void tRowGetPrimaryKey(SRow *row, SRowKey *key) { } while (0) int32_t tValueCompare(const SValue *tv1, const SValue *tv2) { - ASSERT(tv1->type == tv2->type); - switch (tv1->type) { case TSDB_DATA_TYPE_BOOL: case TSDB_DATA_TYPE_TINYINT: @@ -1356,11 +1357,8 @@ int32_t tValueCompare(const SValue *tv1, const SValue *tv2) { int32_t ret = memcmp(tv1->pData, tv2->pData, tv1->nData < tv2->nData ? tv1->nData : tv2->nData); return ret ? ret : (tv1->nData < tv2->nData ? -1 : (tv1->nData > tv2->nData ? 1 : 0)); } - case TSDB_DATA_TYPE_DECIMAL: - ASSERT(0); - break; default: - ASSERT(0); + break; } return 0; @@ -1403,7 +1401,6 @@ void tRowKeyAssign(SRowKey *pDst, SRowKey *pSrc) { pVal->val = pSrc->pks[i].val; } else { pVal->nData = pSrc->pks[i].nData; - ASSERT(pSrc->pks[i].pData != NULL); (void)memcpy(pVal->pData, pSrc->pks[i].pData, pVal->nData); } } @@ -1476,7 +1473,6 @@ static void debugPrintTagVal(int8_t type, const void *val, int32_t vlen, const c printf("%s:%d type:%d vlen:%d, val:%" PRIi8 "\n", tag, ln, (int32_t)type, vlen, *(int8_t *)val); break; default: - ASSERT(0); break; } } @@ -1526,7 +1522,6 @@ static int32_t tPutTagVal(uint8_t *p, STagVal *pTagVal, int8_t isJson) { n += tPutCStr(p ? p + n : p, pTagVal->pKey); } else { n += tPutI16v(p ? p + n : p, pTagVal->cid); - ASSERTS(pTagVal->cid > 0, "Invalid tag cid:%" PRIi16, pTagVal->cid); } // type @@ -1602,8 +1597,6 @@ int32_t tTagNew(SArray *pArray, int32_t version, int8_t isJson, STag **ppTag) { isLarge = 1; } - ASSERT(szTag <= INT16_MAX); - // build tag (*ppTag) = (STag *)taosMemoryCalloc(szTag, 1); if ((*ppTag) == NULL) { @@ -1806,8 +1799,14 @@ STSchema *tBuildTSchema(SSchema *aSchema, int32_t numOfCols, int32_t version) { pTSchema->version = version; // timestamp column - ASSERT(aSchema[0].type == TSDB_DATA_TYPE_TIMESTAMP); - ASSERT(aSchema[0].colId == PRIMARYKEY_TIMESTAMP_COL_ID); + if (!(aSchema[0].type == TSDB_DATA_TYPE_TIMESTAMP)) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } + if (!(aSchema[0].colId == PRIMARYKEY_TIMESTAMP_COL_ID)) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } pTSchema->columns[0].colId = aSchema[0].colId; pTSchema->columns[0].type = aSchema[0].type; pTSchema->columns[0].flags = aSchema[0].flags; @@ -1910,7 +1909,9 @@ static FORCE_INLINE int32_t tColDataPutValue(SColData *pColData, uint8_t *pData, pColData->nData += nData; } } else { - ASSERT(pColData->nData == tDataTypes[pColData->type].bytes * pColData->nVal); + if (!(pColData->nData == tDataTypes[pColData->type].bytes * pColData->nVal)) { + return TSDB_CODE_INVALID_PARA; + } code = tRealloc(&pColData->pData, pColData->nData + tDataTypes[pColData->type].bytes); if (code) goto _exit; if (pData) { @@ -2259,7 +2260,9 @@ static int32_t (*tColDataAppendValueImpl[8][3])(SColData *pColData, uint8_t *pDa // VALUE NONE NULL }; int32_t tColDataAppendValue(SColData *pColData, SColVal *pColVal) { - ASSERT(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type); + if (!(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type)) { + return TSDB_CODE_INVALID_PARA; + } return tColDataAppendValueImpl[pColData->flag][pColVal->flag]( pColData, IS_VAR_DATA_TYPE(pColData->type) ? pColVal->value.pData : (uint8_t *)&pColVal->value.val, pColVal->value.nData); @@ -2511,7 +2514,7 @@ static FORCE_INLINE int32_t tColDataUpdateValue70(SColData *pColData, uint8_t *p return tColDataPutValue(pColData, pData, nData); } } else { - ASSERT(0); + return TSDB_CODE_INVALID_PARA; } return 0; } @@ -2569,8 +2572,8 @@ static int32_t (*tColDataUpdateValueImpl[8][3])(SColData *pColData, uint8_t *pDa // VALUE NONE NULL }; int32_t tColDataUpdateValue(SColData *pColData, SColVal *pColVal, bool forward) { - ASSERT(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type); - ASSERT(pColData->nVal > 0); + if (!(pColData->cid == pColVal->cid && pColData->type == pColVal->value.type)) return TSDB_CODE_INVALID_PARA; + if (!(pColData->nVal > 0)) return TSDB_CODE_INVALID_PARA; if (tColDataUpdateValueImpl[pColData->flag][pColVal->flag] == NULL) return 0; @@ -2595,7 +2598,7 @@ static FORCE_INLINE void tColDataGetValue3(SColData *pColData, int32_t iVal, *pColVal = COL_VAL_NULL(pColData->cid, pColData->type); break; default: - ASSERT(0); + break; } } static FORCE_INLINE void tColDataGetValue4(SColData *pColData, int32_t iVal, SColVal *pColVal) { // HAS_VALUE @@ -2623,7 +2626,7 @@ static FORCE_INLINE void tColDataGetValue5(SColData *pColData, int32_t iVal, tColDataGetValue4(pColData, iVal, pColVal); break; default: - ASSERT(0); + break; } } static FORCE_INLINE void tColDataGetValue6(SColData *pColData, int32_t iVal, @@ -2636,7 +2639,7 @@ static FORCE_INLINE void tColDataGetValue6(SColData *pColData, int32_t iVal, tColDataGetValue4(pColData, iVal, pColVal); break; default: - ASSERT(0); + break; } } static FORCE_INLINE void tColDataGetValue7(SColData *pColData, int32_t iVal, @@ -2652,7 +2655,7 @@ static FORCE_INLINE void tColDataGetValue7(SColData *pColData, int32_t iVal, tColDataGetValue4(pColData, iVal, pColVal); break; default: - ASSERT(0); + break; } } static void (*tColDataGetValueImpl[])(SColData *pColData, int32_t iVal, SColVal *pColVal) = { @@ -2666,7 +2669,6 @@ static void (*tColDataGetValueImpl[])(SColData *pColData, int32_t iVal, SColVal tColDataGetValue7 // HAS_VALUE | HAS_NULL | HAS_NONE }; void tColDataGetValue(SColData *pColData, int32_t iVal, SColVal *pColVal) { - ASSERT(iVal >= 0 && iVal < pColData->nVal && pColData->flag); tColDataGetValueImpl[pColData->flag](pColData, iVal, pColVal); } @@ -2687,7 +2689,6 @@ uint8_t tColDataGetBitValue(const SColData *pColData, int32_t iVal) { case (HAS_VALUE | HAS_NULL | HAS_NONE): return GET_BIT2(pColData->pBitMap, iVal); default: - ASSERTS(0, "not possible"); return 0; } } @@ -2755,7 +2756,9 @@ int32_t tColDataCompress(SColData *colData, SColDataCompressInfo *info, SBuffer int32_t code; SBuffer local; - ASSERT(colData->nVal > 0); + if (!(colData->nVal > 0)) { + return TSDB_CODE_INVALID_PARA; + } (*info) = (SColDataCompressInfo){ .cmprAlg = info->cmprAlg, @@ -3053,7 +3056,9 @@ int32_t tColDataAddValueByBind(SColData *pColData, TAOS_MULTI_BIND *pBind, int32 int32_t code = 0; if (!(pBind->num == 1 && pBind->is_null && *pBind->is_null)) { - ASSERT(pColData->type == pBind->buffer_type); + if (!(pColData->type == pBind->buffer_type)) { + return TSDB_CODE_INVALID_PARA; + } } if (IS_VAR_DATA_TYPE(pColData->type)) { // var-length data type @@ -3210,7 +3215,6 @@ void tColDataArrGetRowKey(SColData *aColData, int32_t nColData, int32_t iRow, SR for (int i = 1; i < nColData; i++) { if (aColData[i].cflag & COL_IS_KEY) { - ASSERT(aColData->flag == HAS_VALUE); tColDataGetValue4(&aColData[i], iRow, &cv); key->pks[key->numOfPKs++] = cv.value; } else { @@ -3327,7 +3331,8 @@ static void tColDataMergeImpl(SColData *pColData, int32_t iStart, int32_t iEnd / } else if (bv == BIT_FLG_NULL) { flag |= HAS_NULL; } else { - ASSERT(0); + uError("invalid bit value:%d", bv); + return; } if (flag == pColData->flag) break; @@ -3372,8 +3377,6 @@ static void tColDataMergeImpl(SColData *pColData, int32_t iStart, int32_t iEnd / iv < (pColData->nVal - 1) ? pColData->aOffset[iv + 1] - pColData->aOffset[iv] : pColData->nData - pColData->aOffset[iv]); } - // TODO - ASSERT(0); } else { if (iv != iStart) { (void)memcpy(&pColData->pData[TYPE_BYTES[pColData->type] * iStart], @@ -3521,15 +3524,21 @@ static void tColDataMerge(SColData *aColData, int32_t nColData) { } } -void tColDataSortMerge(SArray *colDataArr) { +int32_t tColDataSortMerge(SArray *colDataArr) { int32_t nColData = TARRAY_SIZE(colDataArr); SColData *aColData = (SColData *)TARRAY_DATA(colDataArr); - if (aColData[0].nVal <= 1) goto _exit; + if (!(aColData[0].type == TSDB_DATA_TYPE_TIMESTAMP)) { + return TSDB_CODE_INVALID_PARA; + } + if (!(aColData[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID)) { + return TSDB_CODE_INVALID_PARA; + } + if (!(aColData[0].flag == HAS_VALUE)) { + return TSDB_CODE_INVALID_PARA; + } - ASSERT(aColData[0].type == TSDB_DATA_TYPE_TIMESTAMP); - ASSERT(aColData[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID); - ASSERT(aColData[0].flag == HAS_VALUE); + if (aColData[0].nVal <= 1) goto _exit; int8_t doSort = 0; int8_t doMerge = 0; @@ -3578,7 +3587,7 @@ void tColDataSortMerge(SArray *colDataArr) { } _exit: - return; + return 0; } static int32_t tPutColDataVersion0(uint8_t *pBuf, SColData *pColData) { @@ -3685,8 +3694,7 @@ int32_t tPutColData(uint8_t version, uint8_t *pBuf, SColData *pColData) { } else if (version == 1) { return tPutColDataVersion1(pBuf, pColData); } else { - ASSERT(0); - return -1; + return TSDB_CODE_INVALID_PARA; } } @@ -3696,8 +3704,7 @@ int32_t tGetColData(uint8_t version, uint8_t *pBuf, SColData *pColData) { } else if (version == 1) { return tGetColDataVersion1(pBuf, pColData); } else { - ASSERT(0); - return -1; + return TSDB_CODE_INVALID_PARA; } } @@ -3733,7 +3740,6 @@ static FORCE_INLINE void tColDataCalcSMABool(SColData *pColData, int64_t *sum, i CALC_SUM_MAX_MIN(*sum, *max, *min, val); break; default: - ASSERT(0); break; } } @@ -3765,7 +3771,6 @@ static FORCE_INLINE void tColDataCalcSMATinyInt(SColData *pColData, int64_t *sum CALC_SUM_MAX_MIN(*sum, *max, *min, val); break; default: - ASSERT(0); break; } } @@ -3797,7 +3802,6 @@ static FORCE_INLINE void tColDataCalcSMATinySmallInt(SColData *pColData, int64_t CALC_SUM_MAX_MIN(*sum, *max, *min, val); break; default: - ASSERT(0); break; } } @@ -3829,7 +3833,6 @@ static FORCE_INLINE void tColDataCalcSMAInt(SColData *pColData, int64_t *sum, in CALC_SUM_MAX_MIN(*sum, *max, *min, val); break; default: - ASSERT(0); break; } } @@ -3861,7 +3864,6 @@ static FORCE_INLINE void tColDataCalcSMABigInt(SColData *pColData, int64_t *sum, CALC_SUM_MAX_MIN(*sum, *max, *min, val); break; default: - ASSERT(0); break; } } @@ -3893,7 +3895,6 @@ static FORCE_INLINE void tColDataCalcSMAFloat(SColData *pColData, int64_t *sum, CALC_SUM_MAX_MIN(*(double *)sum, *(double *)max, *(double *)min, val); break; default: - ASSERT(0); break; } } @@ -3925,7 +3926,6 @@ static FORCE_INLINE void tColDataCalcSMADouble(SColData *pColData, int64_t *sum, CALC_SUM_MAX_MIN(*(double *)sum, *(double *)max, *(double *)min, val); break; default: - ASSERT(0); break; } } @@ -3957,7 +3957,6 @@ static FORCE_INLINE void tColDataCalcSMAUTinyInt(SColData *pColData, int64_t *su CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val); break; default: - ASSERT(0); break; } } @@ -3989,7 +3988,6 @@ static FORCE_INLINE void tColDataCalcSMATinyUSmallInt(SColData *pColData, int64_ CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val); break; default: - ASSERT(0); break; } } @@ -4021,7 +4019,6 @@ static FORCE_INLINE void tColDataCalcSMAUInt(SColData *pColData, int64_t *sum, i CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val); break; default: - ASSERT(0); break; } } @@ -4053,7 +4050,6 @@ static FORCE_INLINE void tColDataCalcSMAUBigInt(SColData *pColData, int64_t *sum CALC_SUM_MAX_MIN(*(uint64_t *)sum, *(uint64_t *)max, *(uint64_t *)min, val); break; default: - ASSERT(0); break; } } @@ -4092,7 +4088,6 @@ static FORCE_INLINE void tColDataCalcSMAVarType(SColData *pColData, int64_t *sum } break; default: - ASSERT(0); break; } } @@ -4153,7 +4148,9 @@ int32_t tValueColumnAppend(SValueColumn *valCol, const SValue *value) { valCol->type = value->type; } - ASSERT(value->type == valCol->type); + if (!(value->type == valCol->type)) { + return TSDB_CODE_INVALID_PARA; + } if (IS_VAR_DATA_TYPE(value->type)) { if ((code = tBufferPutI32(&valCol->offsets, tBufferGetSize(&valCol->data)))) { @@ -4230,7 +4227,9 @@ int32_t tValueColumnGet(SValueColumn *valCol, int32_t idx, SValue *value) { int32_t tValueColumnCompress(SValueColumn *valCol, SValueColumnCompressInfo *info, SBuffer *output, SBuffer *assist) { int32_t code; - ASSERT(valCol->numOfValues > 0); + if (!(valCol->numOfValues > 0)) { + return TSDB_CODE_INVALID_PARA; + } (*info) = (SValueColumnCompressInfo){ .cmprAlg = info->cmprAlg, @@ -4344,7 +4343,7 @@ int32_t tValueColumnCompressInfoDecode(SBufferReader *reader, SValueColumnCompre if ((code = tBufferGetI32v(reader, &info->dataOriginalSize))) return code; if ((code = tBufferGetI32v(reader, &info->dataCompressedSize))) return code; } else { - ASSERT(0); + return TSDB_CODE_INVALID_PARA; } return 0; @@ -4360,7 +4359,9 @@ int32_t tCompressData(void *input, // input int32_t code; extraSizeNeeded = (info->cmprAlg == NO_COMPRESSION) ? info->originalSize : info->originalSize + COMP_OVERFLOW_BYTES; - ASSERT(outputSize >= extraSizeNeeded); + if (!(outputSize >= extraSizeNeeded)) { + return TSDB_CODE_INVALID_PARA; + } if (info->cmprAlg == NO_COMPRESSION) { (void)memcpy(output, input, info->originalSize); @@ -4442,10 +4443,14 @@ int32_t tDecompressData(void *input, // input ) { int32_t code; - ASSERT(outputSize >= info->originalSize); + if (!(outputSize >= info->originalSize)) { + return TSDB_CODE_INVALID_PARA; + } if (info->cmprAlg == NO_COMPRESSION) { - ASSERT(info->compressedSize == info->originalSize); + if (!(info->compressedSize == info->originalSize)) { + return TSDB_CODE_INVALID_PARA; + } (void)memcpy(output, input, info->compressedSize); } else if (info->cmprAlg == ONE_STAGE_COMP || info->cmprAlg == TWO_STAGE_COMP) { SBuffer local; @@ -4478,7 +4483,9 @@ int32_t tDecompressData(void *input, // input return TSDB_CODE_COMPRESS_ERROR; } - ASSERT(decompressedSize == info->originalSize); + if (!(decompressedSize == info->originalSize)) { + return TSDB_CODE_COMPRESS_ERROR; + } tBufferDestroy(&local); } else { DEFINE_VAR(info->cmprAlg); @@ -4512,7 +4519,9 @@ int32_t tDecompressData(void *input, // input return TSDB_CODE_COMPRESS_ERROR; } - ASSERT(decompressedSize == info->originalSize); + if (!(decompressedSize == info->originalSize)) { + return TSDB_CODE_COMPRESS_ERROR; + } tBufferDestroy(&local); } diff --git a/source/common/src/tglobal.c b/source/common/src/tglobal.c index cf0a4725c1..73b9786f99 100644 --- a/source/common/src/tglobal.c +++ b/source/common/src/tglobal.c @@ -1599,6 +1599,12 @@ static int32_t taosSetServerCfg(SConfig *pCfg) { TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "minDiskFreeSize"); tsMinDiskFreeSize = pItem->i64; + TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3MigrateIntervalSec"); + tsS3MigrateIntervalSec = pItem->i32; + + TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3MigrateEnabled"); + tsS3MigrateEnabled = (bool)pItem->bval; + TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "s3PageCacheSize"); tsS3PageCacheSize = pItem->i32; @@ -2323,7 +2329,7 @@ int8_t taosGranted(int8_t type) { case TSDB_GRANT_VIEW: return atomic_load_8(&tsGrant) & GRANT_FLAG_VIEW; default: - ASSERTS(0, "undefined grant type:%" PRIi8, type); + uWarn("undefined grant type:%" PRIi8, type); break; } return 0; diff --git a/source/common/src/tmsg.c b/source/common/src/tmsg.c index f02144a1d1..58a7f8f448 100644 --- a/source/common/src/tmsg.c +++ b/source/common/src/tmsg.c @@ -103,7 +103,9 @@ int32_t tInitSubmitMsgIter(const SSubmitReq *pMsg, SSubmitMsgIter *pIter) { pIter->totalLen = htonl(pMsg->length); pIter->numOfBlocks = htonl(pMsg->numOfBlocks); - ASSERT(pIter->totalLen > 0); + if (!(pIter->totalLen > 0)) { + return terrno = TSDB_CODE_TDB_SUBMIT_MSG_MSSED_UP; + } pIter->len = 0; pIter->pMsg = pMsg; if (pIter->totalLen <= sizeof(SSubmitReq)) { @@ -114,17 +116,21 @@ int32_t tInitSubmitMsgIter(const SSubmitReq *pMsg, SSubmitMsgIter *pIter) { } int32_t tGetSubmitMsgNext(SSubmitMsgIter *pIter, SSubmitBlk **pPBlock) { - ASSERT(pIter->len >= 0); + if (!(pIter->len >= 0)) { + return terrno = TSDB_CODE_INVALID_MSG_LEN; + } if (pIter->len == 0) { pIter->len += sizeof(SSubmitReq); } else { if (pIter->len >= pIter->totalLen) { - ASSERT(0); + return terrno = TSDB_CODE_INVALID_MSG_LEN; } pIter->len += (sizeof(SSubmitBlk) + pIter->dataLen + pIter->schemaLen); - ASSERT(pIter->len > 0); + if (!(pIter->len > 0)) { + return terrno = TSDB_CODE_INVALID_MSG_LEN; + } } if (pIter->len > pIter->totalLen) { @@ -377,7 +383,9 @@ static int32_t tDeserializeSClientHbReq(SDecoder *pDecoder, SClientHbReq *pReq) } } - ASSERT(desc.subPlanNum == taosArrayGetSize(desc.subDesc)); + if (!(desc.subPlanNum == taosArrayGetSize(desc.subDesc))) { + return TSDB_CODE_INVALID_MSG; + } if (!taosArrayPush(pReq->query->queryDesc, &desc)) { return terrno; @@ -8476,7 +8484,7 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) { } else if (pReq->type == TSDB_NORMAL_TABLE) { if (tEncodeSSchemaWrapper(pCoder, &pReq->ntb.schemaRow) < 0) return -1; } else { - ASSERT(0); + return TSDB_CODE_INVALID_MSG; } // ENCODESQL @@ -8528,7 +8536,7 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) { } else if (pReq->type == TSDB_NORMAL_TABLE) { if (tDecodeSSchemaWrapperEx(pCoder, &pReq->ntb.schemaRow) < 0) return -1; } else { - ASSERT(0); + return TSDB_CODE_INVALID_MSG; } // DECODESQL @@ -9311,7 +9319,6 @@ bool tOffsetEqual(const STqOffsetVal *pLeft, const STqOffsetVal *pRight) { return pLeft->uid == pRight->uid; } else { uError("offset type:%d", pLeft->type); - ASSERT(0); } } return false; @@ -9727,7 +9734,9 @@ static int32_t tEncodeSSubmitTbData(SEncoder *pCoder, const SSubmitTbData *pSubm // auto create table if (pSubmitTbData->flags & SUBMIT_REQ_AUTO_CREATE_TABLE) { - ASSERT(pSubmitTbData->pCreateTbReq); + if (!(pSubmitTbData->pCreateTbReq)) { + return TSDB_CODE_INVALID_MSG; + } if (tEncodeSVCreateTbReq(pCoder, pSubmitTbData->pCreateTbReq) < 0) return -1; } diff --git a/source/common/src/tname.c b/source/common/src/tname.c index 491c203a58..8d0f324509 100644 --- a/source/common/src/tname.c +++ b/source/common/src/tname.c @@ -105,7 +105,6 @@ int32_t tNameExtractFullName(const SName* name, char* dst) { size_t tnameLen = strlen(name->tname); if (tnameLen > 0) { - /*ASSERT(name->type == TSDB_TABLE_NAME_T);*/ dst[len] = TS_PATH_DELIMITER[0]; memcpy(dst + len + 1, name->tname, tnameLen); @@ -160,7 +159,10 @@ int32_t tNameGetFullDbName(const SName* name, char* dst) { bool tNameIsEmpty(const SName* name) { return name->type == 0 || name->acctId == 0; } const char* tNameGetTableName(const SName* name) { - ASSERT(name != NULL && name->type == TSDB_TABLE_NAME_T); + if (!(name != NULL && name->type == TSDB_TABLE_NAME_T)) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } return &name->tname[0]; } diff --git a/source/common/src/trow.c b/source/common/src/trow.c index 942255cd7c..3e0e52a860 100644 --- a/source/common/src/trow.c +++ b/source/common/src/trow.c @@ -232,7 +232,6 @@ void tdSRowPrint(STSRow *row, STSchema *pSchema, const char *tag) { if (!tdSTSRowIterNext(&iter, &sVal)) { break; } - ASSERT(sVal.valType == 0 || sVal.valType == 1 || sVal.valType == 2); tdSCellValPrint(&sVal, cols[iter.colIdx - 1].type); } printf("\n"); @@ -389,7 +388,7 @@ bool tdSTSRowIterNext(STSRowIter *pIter, SCellVal *pVal) { } else if (TD_IS_KV_ROW(pIter->pRow)) { (void)tdSTSRowIterGetKvVal(pIter, pCol->colId, &pIter->kvIdx, pVal); } else { - ASSERT(0); + return false; } ++pIter->colIdx; @@ -409,7 +408,9 @@ int32_t tdSTSRowNew(SArray *pArray, STSchema *pTSchema, STSRow **ppRow, int8_t r void *varBuf = NULL; bool isAlloc = false; - ASSERT(nColVal > 1); + if(nColVal <= 1) { + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } for (int32_t iColumn = 0; iColumn < pTSchema->numOfCols; ++iColumn) { pTColumn = &pTSchema->columns[iColumn]; @@ -423,9 +424,10 @@ int32_t tdSTSRowNew(SArray *pArray, STSchema *pTSchema, STSRow **ppRow, int8_t r } if (iColumn == 0) { - ASSERT(pColVal && pColVal->cid == pTColumn->colId); - ASSERT(pTColumn->type == TSDB_DATA_TYPE_TIMESTAMP); - ASSERT(pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID); + if ((pColVal && pColVal->cid != pTColumn->colId) || (pTColumn->type != TSDB_DATA_TYPE_TIMESTAMP) || + (pTColumn->colId != PRIMARYKEY_TIMESTAMP_COL_ID)) { + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } } else { if (IS_VAR_DATA_TYPE(pTColumn->type)) { if (pColVal && COL_VAL_IS_VALUE(pColVal)) { @@ -598,7 +600,10 @@ bool tdSTSRowGetVal(STSRowIter *pIter, col_id_t colId, col_type_t colType, SCell int32_t tdGetKvRowValOfCol(SCellVal *output, STSRow *pRow, void *pBitmap, int32_t offset, int16_t colIdx) { #ifdef TD_SUPPORT_BITMAP - ASSERT(colIdx < tdRowGetNCols(pRow) - 1); + if (!(colIdx < tdRowGetNCols(pRow) - 1)) { + output->valType = TD_VTYPE_NONE; + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } int32_t code = 0; if ((code = tdGetBitmapValType(pBitmap, colIdx, &output->valType, 0)) != TSDB_CODE_SUCCESS) { output->valType = TD_VTYPE_NONE; @@ -699,7 +704,6 @@ int32_t tdAppendColValToRow(SRowBuilder *pBuilder, col_id_t colId, int8_t colTyp int32_t tdAppendColValToKvRow(SRowBuilder *pBuilder, TDRowValT valType, const void *val, bool isCopyVarData, int8_t colType, int16_t colIdx, int32_t offset, col_id_t colId) { if (colIdx < 1) { - ASSERTS(0, "colIdx is %" PRIi64, colIdx); TAOS_RETURN(TSDB_CODE_INVALID_PARA); } --colIdx; @@ -775,7 +779,9 @@ int32_t tdSRowResetBuf(SRowBuilder *pBuilder, void *pBuf) { TD_ROW_SET_INFO(pBuilder->pBuf, 0); TD_ROW_SET_TYPE(pBuilder->pBuf, pBuilder->rowType); - ASSERT(pBuilder->nBitmaps > 0 && pBuilder->flen > 0); + if(!(pBuilder->nBitmaps > 0 && pBuilder->flen > 0)) { + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } uint32_t len = 0; switch (pBuilder->rowType) { @@ -814,7 +820,9 @@ int32_t tdSRowGetBuf(SRowBuilder *pBuilder, void *pBuf) { TAOS_RETURN(TSDB_CODE_INVALID_PARA); } - ASSERT(pBuilder->nBitmaps > 0 && pBuilder->flen > 0); + if(!(pBuilder->nBitmaps > 0 && pBuilder->flen > 0)) { + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } uint32_t len = 0; switch (pBuilder->rowType) { @@ -930,18 +938,20 @@ int32_t tdSetBitmapValType(void *pBitmap, int16_t colIdx, TDRowValT valType, int TAOS_RETURN(TSDB_CODE_SUCCESS); } -void tTSRowGetVal(STSRow *pRow, STSchema *pTSchema, int16_t iCol, SColVal *pColVal) { +int32_t tTSRowGetVal(STSRow *pRow, STSchema *pTSchema, int16_t iCol, SColVal *pColVal) { STColumn *pTColumn = &pTSchema->columns[iCol]; SCellVal cv = {0}; - ASSERT((pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID) || (iCol > 0)); + if (!((pTColumn->colId == PRIMARYKEY_TIMESTAMP_COL_ID) || (iCol > 0))) { + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } if (TD_IS_TP_ROW(pRow)) { (void)tdSTpRowGetVal(pRow, pTColumn->colId, pTColumn->type, pTSchema->flen, pTColumn->offset, iCol - 1, &cv); } else if (TD_IS_KV_ROW(pRow)) { (void)tdSKvRowGetVal(pRow, pTColumn->colId, iCol - 1, &cv); } else { - ASSERT(0); + TAOS_RETURN(TSDB_CODE_INVALID_PARA); } if (tdValTypeIsNone(cv.valType)) { @@ -960,4 +970,5 @@ void tTSRowGetVal(STSRow *pRow, STSchema *pTSchema, int16_t iCol, SColVal *pColV (void)memcpy(&pColVal->value.val, cv.val, tDataTypes[pTColumn->type].bytes); } } + return 0; } \ No newline at end of file diff --git a/source/common/src/ttime.c b/source/common/src/ttime.c index 2a8e7951b1..79847d4e4a 100644 --- a/source/common/src/ttime.c +++ b/source/common/src/ttime.c @@ -397,11 +397,6 @@ char getPrecisionUnit(int32_t precision) { } int64_t convertTimePrecision(int64_t utime, int32_t fromPrecision, int32_t toPrecision) { - ASSERT(fromPrecision == TSDB_TIME_PRECISION_MILLI || fromPrecision == TSDB_TIME_PRECISION_MICRO || - fromPrecision == TSDB_TIME_PRECISION_NANO); - ASSERT(toPrecision == TSDB_TIME_PRECISION_MILLI || toPrecision == TSDB_TIME_PRECISION_MICRO || - toPrecision == TSDB_TIME_PRECISION_NANO); - switch (fromPrecision) { case TSDB_TIME_PRECISION_MILLI: { switch (toPrecision) { @@ -418,7 +413,6 @@ int64_t convertTimePrecision(int64_t utime, int32_t fromPrecision, int32_t toPre } return utime * 1000000; default: - ASSERT(0); return utime; } } // end from milli @@ -434,7 +428,6 @@ int64_t convertTimePrecision(int64_t utime, int32_t fromPrecision, int32_t toPre } return utime * 1000; default: - ASSERT(0); return utime; } } // end from micro @@ -447,12 +440,10 @@ int64_t convertTimePrecision(int64_t utime, int32_t fromPrecision, int32_t toPre case TSDB_TIME_PRECISION_NANO: return utime; default: - ASSERT(0); return utime; } } // end from nano default: { - ASSERT(0); return utime; // only to pass windows compilation } } // end switch fromPrecision @@ -463,10 +454,6 @@ int64_t convertTimePrecision(int64_t utime, int32_t fromPrecision, int32_t toPre // !!!!notice:there are precision problems, double lose precison if time is too large, for example: // 1626006833631000000*1.0 = double = 1626006833631000064 // int64_t convertTimePrecision(int64_t time, int32_t fromPrecision, int32_t toPrecision) { -// assert(fromPrecision == TSDB_TIME_PRECISION_MILLI || fromPrecision == TSDB_TIME_PRECISION_MICRO || -// fromPrecision == TSDB_TIME_PRECISION_NANO); -// assert(toPrecision == TSDB_TIME_PRECISION_MILLI || toPrecision == TSDB_TIME_PRECISION_MICRO || -// toPrecision == TSDB_TIME_PRECISION_NANO); // static double factors[3][3] = {{1., 1000., 1000000.}, {1.0 / 1000, 1., 1000.}, {1.0 / 1000000, 1.0 / 1000, 1.}}; // ((double)time * factors[fromPrecision][toPrecision]); //} @@ -783,7 +770,6 @@ int32_t taosTimeCountIntervalForFill(int64_t skey, int64_t ekey, int64_t interva int64_t taosTimeTruncate(int64_t ts, const SInterval* pInterval) { if (pInterval->sliding == 0) { - ASSERT(pInterval->interval == 0); return ts; } @@ -814,7 +800,6 @@ int64_t taosTimeTruncate(int64_t ts, const SInterval* pInterval) { } else { if (IS_CALENDAR_TIME_DURATION(pInterval->intervalUnit)) { int64_t news = (ts / pInterval->sliding) * pInterval->sliding; - ASSERT(news <= ts); if (pInterval->slidingUnit == 'd' || pInterval->slidingUnit == 'w') { #if defined(WINDOWS) && _MSC_VER >= 1900 int64_t timezone = _timezone; @@ -887,8 +872,6 @@ int64_t taosTimeTruncate(int64_t ts, const SInterval* pInterval) { } } - ASSERT(pInterval->offset >= 0); - if (pInterval->offset > 0) { // try to move current window to the left-hande-side, due to the offset effect. int64_t newe = taosTimeAdd(start, pInterval->interval, pInterval->intervalUnit, precision) - 1; @@ -1284,7 +1267,6 @@ static int32_t parseTsFormat(const char* formatStr, SArray* formats) { } if (last) { // expand - assert(last->type == TS_FORMAT_NODE_TYPE_CHAR); last->len++; formatStr++; } else { @@ -1311,7 +1293,6 @@ static int32_t parseTsFormat(const char* formatStr, SArray* formats) { } } if (lastOtherFormat) { - assert(lastOtherFormat->type == TS_FORMAT_NODE_TYPE_CHAR); lastOtherFormat->len++; formatStr++; } else { @@ -1664,7 +1645,6 @@ static int32_t char2ts(const char* s, SArray* formats, int64_t* ts, int32_t prec } continue; } - assert(node->type == TS_FORMAT_NODE_TYPE_KEYWORD); switch (node->key->id) { case TSFKW_A_M: case TSFKW_P_M: @@ -1929,7 +1909,7 @@ static int32_t char2ts(const char* s, SArray* formats, int64_t* ts, int32_t prec int32_t taosTs2Char(const char* format, SArray** formats, int64_t ts, int32_t precision, char* out, int32_t outLen) { if (!*formats) { *formats = taosArrayInit(8, sizeof(TSFormatNode)); - if (!*formats){ + if (!*formats) { TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); } TAOS_CHECK_RETURN(parseTsFormat(format, *formats)); @@ -2002,7 +1982,7 @@ static int8_t UNIT_INDEX[26] = {/*a*/ 2, 0, -1, 6, -1, -1, -1, /*o*/ -1, -1, -1, -1, 3, -1, /*u*/ 1, -1, 7, -1, 9, -1}; -#define GET_UNIT_INDEX(idx) UNIT_INDEX[(idx) - 97] +#define GET_UNIT_INDEX(idx) UNIT_INDEX[(idx)-97] // clang-format off static int64_t UNIT_MATRIX[10][11] = { /* ns, us, ms, s, min, h, d, w, month, y*/ diff --git a/source/common/src/tvariant.c b/source/common/src/tvariant.c index cd238fef25..3cb8d5f41e 100644 --- a/source/common/src/tvariant.c +++ b/source/common/src/tvariant.c @@ -459,8 +459,8 @@ void taosVariantDestroy(SVariant *pVar) { } } -void taosVariantAssign(SVariant *pDst, const SVariant *pSrc) { - if (pSrc == NULL || pDst == NULL) return; +int32_t taosVariantAssign(SVariant *pDst, const SVariant *pSrc) { + if (pSrc == NULL || pDst == NULL) return 0; pDst->nType = pSrc->nType; if (pSrc->nType == TSDB_DATA_TYPE_BINARY || pSrc->nType == TSDB_DATA_TYPE_VARBINARY || @@ -468,19 +468,20 @@ void taosVariantAssign(SVariant *pDst, const SVariant *pSrc) { pSrc->nType == TSDB_DATA_TYPE_GEOMETRY) { int32_t len = pSrc->nLen + TSDB_NCHAR_SIZE; char *p = taosMemoryRealloc(pDst->pz, len); - ASSERT(p); + if (!p) return terrno; (void)memset(p, 0, len); pDst->pz = p; (void)memcpy(pDst->pz, pSrc->pz, pSrc->nLen); pDst->nLen = pSrc->nLen; - return; + return 0; } if (IS_NUMERIC_TYPE(pSrc->nType) || (pSrc->nType == TSDB_DATA_TYPE_BOOL)) { pDst->i = pSrc->i; } + return 0; } int32_t taosVariantCompare(const SVariant *p1, const SVariant *p2) { diff --git a/source/dnode/mgmt/mgmt_dnode/src/dmHandle.c b/source/dnode/mgmt/mgmt_dnode/src/dmHandle.c index 70f258a362..0de0a34c25 100644 --- a/source/dnode/mgmt/mgmt_dnode/src/dmHandle.c +++ b/source/dnode/mgmt/mgmt_dnode/src/dmHandle.c @@ -479,6 +479,12 @@ int32_t dmProcessRetrieve(SDnodeMgmt *pMgmt, SRpcMsg *pMsg) { } int32_t len = blockEncode(pBlock, pStart, numOfCols); + if(len < 0) { + dError("failed to retrieve data since %s", tstrerror(code)); + blockDataDestroy(pBlock); + rpcFreeCont(pRsp); + return terrno; + } pRsp->numOfRows = htonl(pBlock->info.rows); pRsp->precision = TSDB_TIME_PRECISION_MILLI; // millisecond time precision diff --git a/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c b/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c index fc070d0d05..e8e7a75889 100644 --- a/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c +++ b/source/dnode/mgmt/mgmt_mnode/src/mmWorker.c @@ -22,7 +22,7 @@ static inline int32_t mmAcquire(SMnodeMgmt *pMgmt) { int32_t code = 0; (void)taosThreadRwlockRdlock(&pMgmt->lock); if (pMgmt->stopped) { - code = -1; + code = TSDB_CODE_MNODE_STOPPED; } else { (void)atomic_add_fetch_32(&pMgmt->refCount, 1); } @@ -134,16 +134,19 @@ int32_t mmPutMsgToReadQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) { } int32_t mmPutMsgToQueryQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) { + int32_t code = 0; if (NULL == pMgmt->pMnode) { const STraceId *trace = &pMsg->info.traceId; - dGError("msg:%p, stop to pre-process in mnode since mnode is NULL, type:%s", pMsg, TMSG_INFO(pMsg->msgType)); - return -1; + code = TSDB_CODE_MNODE_NOT_FOUND; + dGError("msg:%p, stop to pre-process in mnode since %s, type:%s", pMsg, tstrerror(code), TMSG_INFO(pMsg->msgType)); + return code; } pMsg->info.node = pMgmt->pMnode; - if (mndPreProcessQueryMsg(pMsg) != 0) { + if ((code = mndPreProcessQueryMsg(pMsg)) != 0) { const STraceId *trace = &pMsg->info.traceId; - dGError("msg:%p, failed to pre-process in mnode since %s, type:%s", pMsg, terrstr(), TMSG_INFO(pMsg->msgType)); - return -1; + dGError("msg:%p, failed to pre-process in mnode since %s, type:%s", pMsg, tstrerror(code), + TMSG_INFO(pMsg->msgType)); + return code; } return mmPutMsgToWorker(pMgmt, &pMgmt->queryWorker, pMsg); } diff --git a/source/dnode/mgmt/node_util/inc/dmUtil.h b/source/dnode/mgmt/node_util/inc/dmUtil.h index 5be41f830d..95556bba86 100644 --- a/source/dnode/mgmt/node_util/inc/dmUtil.h +++ b/source/dnode/mgmt/node_util/inc/dmUtil.h @@ -83,12 +83,12 @@ extern "C" { }\ } -#define dGFatal(param, ...) {if (dDebugFlag & DEBUG_FATAL) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dFatal(param ", gtid:%s", __VA_ARGS__, buf);}} -#define dGError(param, ...) {if (dDebugFlag & DEBUG_ERROR) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dError(param ", gtid:%s", __VA_ARGS__, buf);}} -#define dGWarn(param, ...) {if (dDebugFlag & DEBUG_WARN) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dWarn(param ", gtid:%s", __VA_ARGS__, buf);}} -#define dGInfo(param, ...) {if (dDebugFlag & DEBUG_INFO) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dInfo(param ", gtid:%s", __VA_ARGS__, buf);}} -#define dGDebug(param, ...) {if (dDebugFlag & DEBUG_DEBUG) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dDebug(param ", gtid:%s", __VA_ARGS__, buf);}} -#define dGTrace(param, ...) {if (dDebugFlag & DEBUG_TRACE) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dTrace(param ", gtid:%s", __VA_ARGS__, buf);}} +#define dGFatal(param, ...) {if (dDebugFlag & DEBUG_FATAL) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dFatal(param ", qid:%s", __VA_ARGS__, buf);}} +#define dGError(param, ...) {if (dDebugFlag & DEBUG_ERROR) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dError(param ", qid:%s", __VA_ARGS__, buf);}} +#define dGWarn(param, ...) {if (dDebugFlag & DEBUG_WARN) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dWarn(param ", qid:%s", __VA_ARGS__, buf);}} +#define dGInfo(param, ...) {if (dDebugFlag & DEBUG_INFO) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dInfo(param ", qid:%s", __VA_ARGS__, buf);}} +#define dGDebug(param, ...) {if (dDebugFlag & DEBUG_DEBUG) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dDebug(param ", qid:%s", __VA_ARGS__, buf);}} +#define dGTrace(param, ...) {if (dDebugFlag & DEBUG_TRACE) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); dTrace(param ", qid:%s", __VA_ARGS__, buf);}} // clang-format on @@ -208,7 +208,7 @@ void dmGetMonitorSystemInfo(SMonSysInfo *pInfo); int32_t dmReadFile(const char *path, const char *name, bool *pDeployed); int32_t dmWriteFile(const char *path, const char *name, bool deployed); int32_t dmCheckRunning(const char *dataDir, TdFilePtr *pFile); -//int32_t dmCheckRunningWrapper(const char *dataDir, TdFilePtr *pFile); +// int32_t dmCheckRunningWrapper(const char *dataDir, TdFilePtr *pFile); // dmodule.c int32_t dmInitDndInfo(SDnodeData *pData); diff --git a/source/dnode/mnode/impl/inc/mndArbGroup.h b/source/dnode/mnode/impl/inc/mndArbGroup.h index fcd11310e7..779d64c7e2 100644 --- a/source/dnode/mnode/impl/inc/mndArbGroup.h +++ b/source/dnode/mnode/impl/inc/mndArbGroup.h @@ -29,7 +29,7 @@ void mndReleaseArbGroup(SMnode *pMnode, SArbGroup *pObj); SSdbRaw *mndArbGroupActionEncode(SArbGroup *pGroup); SSdbRow *mndArbGroupActionDecode(SSdbRaw *pRaw); -void mndArbGroupInitFromVgObj(SVgObj *pVgObj, SArbGroup *outGroup); +int32_t mndArbGroupInitFromVgObj(SVgObj *pVgObj, SArbGroup *outGroup); int32_t mndSetCreateArbGroupRedoLogs(STrans *pTrans, SArbGroup *pGroup); int32_t mndSetCreateArbGroupUndoLogs(STrans *pTrans, SArbGroup *pGroup); diff --git a/source/dnode/mnode/impl/inc/mndInt.h b/source/dnode/mnode/impl/inc/mndInt.h index 072568eb0f..5a4062daf6 100644 --- a/source/dnode/mnode/impl/inc/mndInt.h +++ b/source/dnode/mnode/impl/inc/mndInt.h @@ -41,12 +41,12 @@ extern "C" { #define mDebug(...) { if (mDebugFlag & DEBUG_DEBUG) { taosPrintLog("MND ", DEBUG_DEBUG, mDebugFlag, __VA_ARGS__); }} #define mTrace(...) { if (mDebugFlag & DEBUG_TRACE) { taosPrintLog("MND ", DEBUG_TRACE, mDebugFlag, __VA_ARGS__); }} -#define mGFatal(param, ...) { if (mDebugFlag & DEBUG_FATAL){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mFatal(param ", gtid:%s", __VA_ARGS__, buf);}} -#define mGError(param, ...) { if (mDebugFlag & DEBUG_ERROR){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mError(param ", gtid:%s", __VA_ARGS__, buf);}} -#define mGWarn(param, ...) { if (mDebugFlag & DEBUG_WARN){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mWarn (param ", gtid:%s", __VA_ARGS__, buf);}} -#define mGInfo(param, ...) { if (mDebugFlag & DEBUG_INFO){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mInfo (param ", gtid:%s", __VA_ARGS__, buf);}} -#define mGDebug(param, ...) { if (mDebugFlag & DEBUG_DEBUG){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mDebug(param ", gtid:%s", __VA_ARGS__, buf);}} -#define mGTrace(param, ...) { if (mDebugFlag & DEBUG_TRACE){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mTrace(param ", gtid:%s", __VA_ARGS__, buf);}} +#define mGFatal(param, ...) { if (mDebugFlag & DEBUG_FATAL){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mFatal(param ", qid:%s", __VA_ARGS__, buf);}} +#define mGError(param, ...) { if (mDebugFlag & DEBUG_ERROR){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mError(param ", qid:%s", __VA_ARGS__, buf);}} +#define mGWarn(param, ...) { if (mDebugFlag & DEBUG_WARN){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mWarn (param ", qid:%s", __VA_ARGS__, buf);}} +#define mGInfo(param, ...) { if (mDebugFlag & DEBUG_INFO){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mInfo (param ", qid:%s", __VA_ARGS__, buf);}} +#define mGDebug(param, ...) { if (mDebugFlag & DEBUG_DEBUG){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mDebug(param ", qid:%s", __VA_ARGS__, buf);}} +#define mGTrace(param, ...) { if (mDebugFlag & DEBUG_TRACE){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); mTrace(param ", qid:%s", __VA_ARGS__, buf);}} // clang-format on #define SYSTABLE_SCH_TABLE_NAME_LEN ((TSDB_TABLE_NAME_LEN - 1) + VARSTR_HEADER_SIZE) @@ -54,7 +54,7 @@ extern "C" { #define SYSTABLE_SCH_COL_NAME_LEN ((TSDB_COL_NAME_LEN - 1) + VARSTR_HEADER_SIZE) typedef int32_t (*MndMsgFp)(SRpcMsg *pMsg); -typedef int32_t (*MndMsgFpExt)(SRpcMsg *pMsg, SQueueInfo* pInfo); +typedef int32_t (*MndMsgFpExt)(SRpcMsg *pMsg, SQueueInfo *pInfo); typedef int32_t (*MndInitFp)(SMnode *pMnode); typedef void (*MndCleanupFp)(SMnode *pMnode); typedef int32_t (*ShowRetrieveFp)(SRpcMsg *pMsg, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows); diff --git a/source/dnode/mnode/impl/src/mndArbGroup.c b/source/dnode/mnode/impl/src/mndArbGroup.c index 32b62422e9..23434e5ee3 100644 --- a/source/dnode/mnode/impl/src/mndArbGroup.c +++ b/source/dnode/mnode/impl/src/mndArbGroup.c @@ -102,8 +102,10 @@ void mndReleaseArbGroup(SMnode *pMnode, SArbGroup *pGroup) { sdbRelease(pSdb, pGroup); } -void mndArbGroupInitFromVgObj(SVgObj *pVgObj, SArbGroup *outGroup) { - ASSERT(pVgObj->replica == 2); +int32_t mndArbGroupInitFromVgObj(SVgObj *pVgObj, SArbGroup *outGroup) { + if (pVgObj->replica != 2) { + TAOS_RETURN(TSDB_CODE_INVALID_PARA); + } (void)memset(outGroup, 0, sizeof(SArbGroup)); outGroup->dbUid = pVgObj->dbUid; outGroup->vgId = pVgObj->vgId; @@ -111,6 +113,8 @@ void mndArbGroupInitFromVgObj(SVgObj *pVgObj, SArbGroup *outGroup) { SArbGroupMember *pMember = &outGroup->members[i]; pMember->info.dnodeId = pVgObj->vnodeGid[i].dnodeId; } + + TAOS_RETURN(TSDB_CODE_SUCCESS); } SSdbRaw *mndArbGroupActionEncode(SArbGroup *pGroup) { diff --git a/source/dnode/mnode/impl/src/mndDb.c b/source/dnode/mnode/impl/src/mndDb.c index d0eed37f99..9b83fa8a1d 100644 --- a/source/dnode/mnode/impl/src/mndDb.c +++ b/source/dnode/mnode/impl/src/mndDb.c @@ -629,7 +629,7 @@ static int32_t mndSetCreateDbRedoLogs(SMnode *pMnode, STrans *pTrans, SDbObj *pD for (int32_t v = 0; v < pDb->cfg.numOfVgroups; ++v) { SVgObj *pVgObj = pVgroups + v; SArbGroup arbGroup = {0}; - mndArbGroupInitFromVgObj(pVgObj, &arbGroup); + TAOS_CHECK_RETURN(mndArbGroupInitFromVgObj(pVgObj, &arbGroup)); TAOS_CHECK_RETURN(mndSetCreateArbGroupRedoLogs(pTrans, &arbGroup)); } } @@ -663,7 +663,7 @@ static int32_t mndSetCreateDbUndoLogs(SMnode *pMnode, STrans *pTrans, SDbObj *pD for (int32_t v = 0; v < pDb->cfg.numOfVgroups; ++v) { SVgObj *pVgObj = pVgroups + v; SArbGroup arbGroup = {0}; - mndArbGroupInitFromVgObj(pVgObj, &arbGroup); + TAOS_CHECK_RETURN(mndArbGroupInitFromVgObj(pVgObj, &arbGroup)); TAOS_CHECK_RETURN(mndSetCreateArbGroupUndoLogs(pTrans, &arbGroup)); } } @@ -698,7 +698,7 @@ static int32_t mndSetCreateDbCommitLogs(SMnode *pMnode, STrans *pTrans, SDbObj * for (int32_t v = 0; v < pDb->cfg.numOfVgroups; ++v) { SVgObj *pVgObj = pVgroups + v; SArbGroup arbGroup = {0}; - mndArbGroupInitFromVgObj(pVgObj, &arbGroup); + TAOS_CHECK_RETURN(mndArbGroupInitFromVgObj(pVgObj, &arbGroup)); TAOS_CHECK_RETURN(mndSetCreateArbGroupCommitLogs(pTrans, &arbGroup)); } } @@ -1156,44 +1156,30 @@ static int32_t mndSetAlterDbCommitLogs(SMnode *pMnode, STrans *pTrans, SDbObj *p } static int32_t mndSetAlterDbRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj *pOldDb, SDbObj *pNewDb) { - int32_t code = 0; + int32_t code = 0, lino = 0; SSdb *pSdb = pMnode->pSdb; void *pIter = NULL; + SVgObj *pVgroup = NULL; SArray *pArray = mndBuildDnodesArray(pMnode, 0); while (1) { - SVgObj *pVgroup = NULL; pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void **)&pVgroup); if (pIter == NULL) break; if (mndVgroupInDb(pVgroup, pNewDb->uid)) { SVgObj newVgroup = {0}; - if ((code = mndBuildAlterVgroupAction(pMnode, pTrans, pOldDb, pNewDb, pVgroup, pArray, &newVgroup)) != 0) { - sdbCancelFetch(pSdb, pIter); - sdbRelease(pSdb, pVgroup); - taosArrayDestroy(pArray); - TAOS_RETURN(code); - } + TAOS_CHECK_GOTO(mndBuildAlterVgroupAction(pMnode, pTrans, pOldDb, pNewDb, pVgroup, pArray, &newVgroup), &lino, + _err); + if (pNewDb->cfg.withArbitrator != pOldDb->cfg.withArbitrator) { if (pNewDb->cfg.withArbitrator) { SArbGroup arbGroup = {0}; - mndArbGroupInitFromVgObj(&newVgroup, &arbGroup); - if ((code = mndSetCreateArbGroupCommitLogs(pTrans, &arbGroup)) != 0) { - sdbCancelFetch(pSdb, pIter); - sdbRelease(pSdb, pVgroup); - taosArrayDestroy(pArray); - TAOS_RETURN(code); - } - + TAOS_CHECK_GOTO(mndArbGroupInitFromVgObj(&newVgroup, &arbGroup), &lino, _err); + TAOS_CHECK_GOTO(mndSetCreateArbGroupCommitLogs(pTrans, &arbGroup), &lino, _err); } else { SArbGroup arbGroup = {0}; - mndArbGroupInitFromVgObj(pVgroup, &arbGroup); - if ((code = mndSetDropArbGroupCommitLogs(pTrans, &arbGroup)) != 0) { - sdbCancelFetch(pSdb, pIter); - sdbRelease(pSdb, pVgroup); - taosArrayDestroy(pArray); - TAOS_RETURN(code); - } + TAOS_CHECK_GOTO(mndArbGroupInitFromVgObj(pVgroup, &arbGroup), &lino, _err); + TAOS_CHECK_GOTO(mndSetDropArbGroupCommitLogs(pTrans, &arbGroup), &lino, _err); } } } @@ -1203,6 +1189,14 @@ static int32_t mndSetAlterDbRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj * taosArrayDestroy(pArray); TAOS_RETURN(code); + +_err: + mError("db:%s, %s failed at %d since %s", pNewDb->name, __func__, lino, tstrerror(code)); + + sdbCancelFetch(pSdb, pIter); + sdbRelease(pSdb, pVgroup); + taosArrayDestroy(pArray); + TAOS_RETURN(code); } static int32_t mndAlterDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pOld, SDbObj *pNew) { diff --git a/source/dnode/mnode/impl/src/mndMain.c b/source/dnode/mnode/impl/src/mndMain.c index 25872801e4..9b80370fd4 100644 --- a/source/dnode/mnode/impl/src/mndMain.c +++ b/source/dnode/mnode/impl/src/mndMain.c @@ -785,7 +785,7 @@ int32_t mndProcessSyncMsg(SRpcMsg *pMsg) { int32_t code = syncProcessMsg(pMgmt->sync, pMsg); if (code != 0) { - mGError("vgId:1, failed to process sync msg:%p type:%s, errno: %s, code:0x%x", pMsg, TMSG_INFO(pMsg->msgType), + mGError("vgId:1, failed to process sync msg:%p type:%s, reason: %s, code:0x%x", pMsg, TMSG_INFO(pMsg->msgType), tstrerror(code), code); } diff --git a/source/dnode/mnode/impl/src/mndShow.c b/source/dnode/mnode/impl/src/mndShow.c index 6989e1e4f1..86802914b8 100644 --- a/source/dnode/mnode/impl/src/mndShow.c +++ b/source/dnode/mnode/impl/src/mndShow.c @@ -180,7 +180,11 @@ static void mndFreeShowObj(SShowObj *pShow) { ShowFreeIterFp freeFp = pMgmt->freeIterFps[pShow->type]; if (freeFp != NULL) { if (pShow->pIter != NULL) { + mTrace("show:0x%" PRIx64 ", is destroying, data:%p, pIter:%p, ", pShow->id, pShow, pShow->pIter); + (*freeFp)(pMnode, pShow->pIter); + + pShow->pIter = NULL; } } @@ -330,11 +334,9 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) { SRetrieveMetaTableRsp *pRsp = rpcMallocCont(size); if (pRsp == NULL) { - mndReleaseShowObj(pShow, false); - code = TSDB_CODE_OUT_OF_MEMORY; mError("show:0x%" PRIx64 ", failed to retrieve data since %s", pShow->id, tstrerror(code)); - blockDataDestroy(pBlock); - TAOS_RETURN(code); + code = terrno; + goto _exit; } pRsp->handle = htobe64(pShow->id); @@ -356,6 +358,11 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) { } int32_t len = blockEncode(pBlock, pStart, pShow->pMeta->numOfColumns); + if(len < 0){ + mError("show:0x%" PRIx64 ", failed to retrieve data since %s", pShow->id, tstrerror(code)); + code = terrno; + return code; + } } pRsp->numOfRows = htonl(rowsRead); @@ -374,6 +381,13 @@ static int32_t mndProcessRetrieveSysTableReq(SRpcMsg *pReq) { blockDataDestroy(pBlock); return TSDB_CODE_SUCCESS; +_exit: + mndReleaseShowObj(pShow, false); + blockDataDestroy(pBlock); + if(pRsp) { + rpcFreeCont(pRsp); + } + return code; } static bool mndCheckRetrieveFinished(SShowObj *pShow) { diff --git a/source/dnode/mnode/impl/src/mndStb.c b/source/dnode/mnode/impl/src/mndStb.c index a76dfa1b51..9a1804ce6a 100644 --- a/source/dnode/mnode/impl/src/mndStb.c +++ b/source/dnode/mnode/impl/src/mndStb.c @@ -1774,7 +1774,7 @@ static int32_t mndUpdateSuperTableColumnCompress(SMnode *pMnode, const SStbObj * int32_t nCols) { // if (pColCmpr == NULL || colName == NULL) return -1; - ASSERT(taosArrayGetSize(pField) == nCols); + if (taosArrayGetSize(pField) != nCols) return TSDB_CODE_FAILED; TAOS_FIELD *p = taosArrayGet(pField, 0); int32_t code = 0; diff --git a/source/dnode/mnode/impl/src/mndSync.c b/source/dnode/mnode/impl/src/mndSync.c index 3c5724dde3..da5873039b 100644 --- a/source/dnode/mnode/impl/src/mndSync.c +++ b/source/dnode/mnode/impl/src/mndSync.c @@ -189,13 +189,16 @@ int32_t mndProcessWriteMsg(SMnode *pMnode, SRpcMsg *pMsg, SFsmCbMeta *pMeta) { goto _OUT; } + (void)taosThreadMutexLock(&pMnode->pSdb->filelock); code = sdbWriteWithoutFree(pMnode->pSdb, pRaw); if (code != 0) { mError("trans:%d, failed to write to sdb since %s", transId, terrstr()); // code = 0; + (void)taosThreadMutexUnlock(&pMnode->pSdb->filelock); pMeta->code = code; goto _OUT; } + (void)taosThreadMutexUnlock(&pMnode->pSdb->filelock); pTrans = mndAcquireTrans(pMnode, transId); if (pTrans == NULL) { diff --git a/source/dnode/mnode/impl/src/mndUser.c b/source/dnode/mnode/impl/src/mndUser.c index c4823dc62e..49cacb3dce 100644 --- a/source/dnode/mnode/impl/src/mndUser.c +++ b/source/dnode/mnode/impl/src/mndUser.c @@ -3147,7 +3147,7 @@ int32_t mndValidateUserAuthInfo(SMnode *pMnode, SUserAuthVersion *pUsers, int32_ if (TSDB_CODE_MND_USER_NOT_EXIST == code) { SGetUserAuthRsp rsp = {.dropped = 1}; (void)memcpy(rsp.user, pUsers[i].user, TSDB_USER_LEN); - (void)taosArrayPush(batchRsp.pArray, &rsp); + TSDB_CHECK_NULL(taosArrayPush(batchRsp.pArray, &rsp), code, lino, _OVER, TSDB_CODE_OUT_OF_MEMORY); } mError("user:%s, failed to auth user since %s", pUsers[i].user, tstrerror(code)); code = 0; @@ -3168,7 +3168,12 @@ int32_t mndValidateUserAuthInfo(SMnode *pMnode, SUserAuthVersion *pUsers, int32_ TAOS_CHECK_GOTO(code, &lino, _OVER); } - (void)taosArrayPush(batchRsp.pArray, &rsp); + if (!(taosArrayPush(batchRsp.pArray, &rsp))) { + code = TSDB_CODE_OUT_OF_MEMORY; + mndReleaseUser(pMnode, pUser); + tFreeSGetUserAuthRsp(&rsp); + TAOS_CHECK_GOTO(code, &lino, _OVER); + } mndReleaseUser(pMnode, pUser); } diff --git a/source/dnode/mnode/impl/test/topic/topic.cpp b/source/dnode/mnode/impl/test/topic/topic.cpp index 4c2213b700..6808fdd04d 100644 --- a/source/dnode/mnode/impl/test/topic/topic.cpp +++ b/source/dnode/mnode/impl/test/topic/topic.cpp @@ -53,7 +53,6 @@ void* MndTestTopic::BuildCreateDbReq(const char* dbname, int32_t* pContLen) { int32_t contLen = tSerializeSCreateDbReq(NULL, 0, &createReq); void* pReq = rpcMallocCont(contLen); - assert(pReq != NULL); (void)tSerializeSCreateDbReq(pReq, contLen, &createReq); *pContLen = contLen; @@ -69,7 +68,6 @@ void* MndTestTopic::BuildCreateTopicReq(const char* topicName, const char* sql, int32_t contLen = tSerializeSCMCreateTopicReq(NULL, 0, &createReq); void* pReq = rpcMallocCont(contLen); - assert(pReq != NULL); (void)tSerializeSCMCreateTopicReq(pReq, contLen, &createReq); *pContLen = contLen; @@ -82,7 +80,6 @@ void* MndTestTopic::BuildDropTopicReq(const char* topicName, int32_t* pContLen) int32_t contLen = tSerializeSMDropTopicReq(NULL, 0, &dropReq); void* pReq = rpcMallocCont(contLen); - assert(pReq != NULL); (void)tSerializeSMDropTopicReq(pReq, contLen, &dropReq); *pContLen = contLen; diff --git a/source/dnode/mnode/sdb/src/sdbFile.c b/source/dnode/mnode/sdb/src/sdbFile.c index 6fb395e63f..9073032738 100644 --- a/source/dnode/mnode/sdb/src/sdbFile.c +++ b/source/dnode/mnode/sdb/src/sdbFile.c @@ -52,6 +52,8 @@ static void sdbResetData(SSdb *pSdb) { SHashObj *hash = pSdb->hashObjs[i]; if (hash == NULL) continue; + sdbWriteLock(pSdb, i); + SSdbRow **ppRow = taosHashIterate(hash, NULL); while (ppRow != NULL) { SSdbRow *pRow = *ppRow; @@ -60,15 +62,13 @@ static void sdbResetData(SSdb *pSdb) { sdbFreeRow(pSdb, pRow, true); ppRow = taosHashIterate(hash, ppRow); } - } - - for (ESdbType i = 0; i < SDB_MAX; ++i) { - SHashObj *hash = pSdb->hashObjs[i]; - if (hash == NULL) continue; taosHashClear(pSdb->hashObjs[i]); pSdb->tableVer[i] = 0; pSdb->maxId[i] = 0; + + sdbUnLock(pSdb, i); + mInfo("sdb:%s is reset", sdbTableName(i)); } diff --git a/source/dnode/vnode/src/inc/vnd.h b/source/dnode/vnode/src/inc/vnd.h index 7bdc8e1c33..dbcda45ef5 100644 --- a/source/dnode/vnode/src/inc/vnd.h +++ b/source/dnode/vnode/src/inc/vnd.h @@ -32,12 +32,12 @@ extern "C" { #define vDebug(...) do { if (vDebugFlag & DEBUG_DEBUG) { taosPrintLog("VND ", DEBUG_DEBUG, vDebugFlag, __VA_ARGS__); }} while(0) #define vTrace(...) do { if (vDebugFlag & DEBUG_TRACE) { taosPrintLog("VND ", DEBUG_TRACE, vDebugFlag, __VA_ARGS__); }} while(0) -#define vGTrace(param, ...) do { if (vDebugFlag & DEBUG_TRACE) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vTrace(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) -#define vGFatal(param, ...) do { if (vDebugFlag & DEBUG_FATAL) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vFatal(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) -#define vGError(param, ...) do { if (vDebugFlag & DEBUG_ERROR) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vError(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) -#define vGWarn(param, ...) do { if (vDebugFlag & DEBUG_WARN) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vWarn(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) -#define vGInfo(param, ...) do { if (vDebugFlag & DEBUG_INFO) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vInfo(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) -#define vGDebug(param, ...) do { if (vDebugFlag & DEBUG_DEBUG) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vDebug(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) +#define vGTrace(param, ...) do { if (vDebugFlag & DEBUG_TRACE) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vTrace(param ", qid:%s", __VA_ARGS__, buf);}} while(0) +#define vGFatal(param, ...) do { if (vDebugFlag & DEBUG_FATAL) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vFatal(param ", qid:%s", __VA_ARGS__, buf);}} while(0) +#define vGError(param, ...) do { if (vDebugFlag & DEBUG_ERROR) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vError(param ", qid:%s", __VA_ARGS__, buf);}} while(0) +#define vGWarn(param, ...) do { if (vDebugFlag & DEBUG_WARN) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vWarn(param ", qid:%s", __VA_ARGS__, buf);}} while(0) +#define vGInfo(param, ...) do { if (vDebugFlag & DEBUG_INFO) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vInfo(param ", qid:%s", __VA_ARGS__, buf);}} while(0) +#define vGDebug(param, ...) do { if (vDebugFlag & DEBUG_DEBUG) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); vDebug(param ", qid:%s", __VA_ARGS__, buf);}} while(0) // clang-format on diff --git a/source/dnode/vnode/src/inc/vnodeInt.h b/source/dnode/vnode/src/inc/vnodeInt.h index e81b858d0e..6f562cef4a 100644 --- a/source/dnode/vnode/src/inc/vnodeInt.h +++ b/source/dnode/vnode/src/inc/vnodeInt.h @@ -101,20 +101,6 @@ typedef struct SQueryNode SQueryNode; #define VND_INFO_FNAME "vnode.json" #define VND_INFO_FNAME_TMP "vnode_tmp.json" -#define VNODE_METRIC_SQL_COUNT "taosd_sql_req:count" - -#define VNODE_METRIC_TAG_NAME_SQL_TYPE "sql_type" -#define VNODE_METRIC_TAG_NAME_CLUSTER_ID "cluster_id" -#define VNODE_METRIC_TAG_NAME_DNODE_ID "dnode_id" -#define VNODE_METRIC_TAG_NAME_DNODE_EP "dnode_ep" -#define VNODE_METRIC_TAG_NAME_VGROUP_ID "vgroup_id" -#define VNODE_METRIC_TAG_NAME_USERNAME "username" -#define VNODE_METRIC_TAG_NAME_RESULT "result" - -#define VNODE_METRIC_TAG_VALUE_INSERT_AFFECTED_ROWS "inserted_rows" -// #define VNODE_METRIC_TAG_VALUE_INSERT "insert" -// #define VNODE_METRIC_TAG_VALUE_DELETE "delete" - // vnd.h typedef int32_t (*_query_reseek_func_t)(void* pQHandle); struct SQueryNode { @@ -452,7 +438,6 @@ typedef struct SVMonitorObj { char strClusterId[TSDB_CLUSTER_ID_LEN]; char strDnodeId[TSDB_NODE_ID_LEN]; char strVgId[TSDB_VGROUP_ID_LEN]; - taos_counter_t* insertCounter; } SVMonitorObj; typedef struct { diff --git a/source/dnode/vnode/src/meta/metaCache.c b/source/dnode/vnode/src/meta/metaCache.c index 98f5615f5e..5cab16c63c 100644 --- a/source/dnode/vnode/src/meta/metaCache.c +++ b/source/dnode/vnode/src/meta/metaCache.c @@ -503,7 +503,6 @@ static int checkAllEntriesInCache(const STagFilterResEntry* pEntry, SArray* pInv } static FORCE_INLINE void setMD5DigestInKey(uint64_t* pBuf, const char* key, int32_t keyLen) { - // ASSERT(keyLen == sizeof(int64_t) * 2); memcpy(&pBuf[2], key, keyLen); } @@ -513,7 +512,6 @@ static void initCacheKey(uint64_t* buf, const SHashObj* pHashMap, uint64_t suid, buf[0] = (uint64_t)pHashMap; buf[1] = suid; setMD5DigestInKey(buf, key, keyLen); - ASSERT(keyLen == sizeof(uint64_t) * 2); } int32_t metaGetCachedTableUidList(void* pVnode, tb_uid_t suid, const uint8_t* pKey, int32_t keyLen, SArray* pList1, diff --git a/source/dnode/vnode/src/meta/metaOpen.c b/source/dnode/vnode/src/meta/metaOpen.c index 13163aef05..2ba35a3987 100644 --- a/source/dnode/vnode/src/meta/metaOpen.c +++ b/source/dnode/vnode/src/meta/metaOpen.c @@ -332,6 +332,10 @@ static int tagIdxKeyCmpr(const void *pKey1, int kLen1, const void *pKey2, int kL } else if (!pTagIdxKey1->isNull && !pTagIdxKey2->isNull) { // all not NULL, compr tag vals __compar_fn_t func = getComparFunc(pTagIdxKey1->type, 0); + if (func == NULL) { + metaError("meta/open: %s", terrstr()); + return TSDB_CODE_FAILED; + } c = func(pTagIdxKey1->data, pTagIdxKey2->data); if (c) return c; } diff --git a/source/dnode/vnode/src/meta/metaQuery.c b/source/dnode/vnode/src/meta/metaQuery.c index 3b10182c90..1668f699ae 100644 --- a/source/dnode/vnode/src/meta/metaQuery.c +++ b/source/dnode/vnode/src/meta/metaQuery.c @@ -275,10 +275,10 @@ void metaPauseTbCursor(SMTbCursor *pTbCur) { int32_t metaResumeTbCursor(SMTbCursor *pTbCur, int8_t first, int8_t move) { int32_t code = 0; int32_t lino; - int8_t locked = 0; + int8_t locked = 0; if (pTbCur->paused) { metaReaderDoInit(&pTbCur->mr, pTbCur->pMeta, META_READER_LOCK); - locked = 1; + locked = 1; code = tdbTbcOpen(((SMeta *)pTbCur->pMeta)->pUidIdx, (TBC **)&pTbCur->pDbc, NULL); if (code != 0) { TSDB_CHECK_CODE(code, lino, _exit); @@ -673,7 +673,7 @@ int32_t metaGetTbTSchemaEx(SMeta *pMeta, tb_uid_t suid, tb_uid_t uid, int32_t sv } } - if (ASSERTS(sver > 0, "failed to get table schema version: %d", sver)) { + if (!(sver > 0)) { code = TSDB_CODE_NOT_FOUND; goto _exit; } @@ -1103,7 +1103,12 @@ int32_t metaFilterCreateTime(void *pVnode, SMetaFltParam *arg, SArray *pUids) { SBtimeIdxKey *p = entryKey; if (count > TRY_ERROR_LIMIT) break; + terrno = TSDB_CODE_SUCCESS; int32_t cmp = (*param->filterFunc)((void *)&p->btime, (void *)&pBtimeKey->btime, param->type); + if (terrno != TSDB_CODE_SUCCESS) { + ret = terrno; + break; + } if (cmp == 0) { if (taosArrayPush(pUids, &p->uid) == NULL) { ret = terrno; @@ -1167,7 +1172,12 @@ int32_t metaFilterTableName(void *pVnode, SMetaFltParam *arg, SArray *pUids) { if (count > TRY_ERROR_LIMIT) break; char *pTableKey = (char *)pEntryKey; + terrno = TSDB_CODE_SUCCESS; cmp = (*param->filterFunc)(pTableKey, pName, pCursor->type); + if (terrno != TSDB_CODE_SUCCESS) { + ret = terrno; + goto END; + } if (cmp == 0) { tb_uid_t tuid = *(tb_uid_t *)pEntryVal; if (taosArrayPush(pUids, &tuid) == NULL) { @@ -1361,7 +1371,12 @@ int32_t metaFilterTableIds(void *pVnode, SMetaFltParam *arg, SArray *pUids) { } } + terrno = TSDB_CODE_SUCCESS; int32_t cmp = (*param->filterFunc)(p->data, pKey->data, pKey->type); + if (terrno != TSDB_CODE_SUCCESS) { + TAOS_CHECK_GOTO(terrno, NULL, END); + break; + } if (cmp == 0) { // match tb_uid_t tuid = 0; @@ -1593,8 +1608,6 @@ int32_t metaGetStbStats(void *pVnode, int64_t uid, int64_t *numOfTables, int32_t metaULock(pVnodeObj->pMeta); if (numOfTables) *numOfTables = state.ctbNum; if (numOfCols) *numOfCols = state.colNum; - ASSERTS(state.colNum > 0, "vgId:%d, suid:%" PRIi64 " nCols:%d <= 0 in metaCache", TD_VID(pVnodeObj), uid, - state.colNum); goto _exit; } diff --git a/source/dnode/vnode/src/meta/metaTable.c b/source/dnode/vnode/src/meta/metaTable.c index 21602627e9..0334990365 100644 --- a/source/dnode/vnode/src/meta/metaTable.c +++ b/source/dnode/vnode/src/meta/metaTable.c @@ -129,7 +129,11 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const } SIndexMultiTerm *terms = indexMultiTermCreate(); - int16_t nCols = taosArrayGetSize(pTagVals); + if (terms == NULL) { + return terrno; + } + + int16_t nCols = taosArrayGetSize(pTagVals); for (int i = 0; i < nCols; i++) { STagVal *pTagVal = (STagVal *)taosArrayGet(pTagVals, i); char type = pTagVal->type; @@ -142,8 +146,14 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const term = indexTermCreate(suid, ADD_VALUE, TSDB_DATA_TYPE_VARCHAR, key, nKey, NULL, 0); } else if (type == TSDB_DATA_TYPE_NCHAR) { if (pTagVal->nData > 0) { - char *val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE); + char *val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE); + if (val == NULL) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, NULL, _exception); + } int32_t len = taosUcs4ToMbs((TdUcs4 *)pTagVal->pData, pTagVal->nData, val + VARSTR_HEADER_SIZE); + if (len < 0) { + TAOS_CHECK_GOTO(len, NULL, _exception); + } memcpy(val, (uint16_t *)&len, VARSTR_HEADER_SIZE); type = TSDB_DATA_TYPE_VARCHAR; term = indexTermCreate(suid, ADD_VALUE, type, key, nKey, val, len); @@ -160,16 +170,24 @@ static int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const int len = sizeof(val); term = indexTermCreate(suid, ADD_VALUE, TSDB_DATA_TYPE_BOOL, key, nKey, (const char *)&val, len); } + if (term != NULL) { (void)indexMultiTermAdd(terms, term); + } else { + code = terrno; + goto _exception; } } - (void)indexJsonPut(pMeta->pTagIvtIdx, terms, tuid); + code = indexJsonPut(pMeta->pTagIvtIdx, terms, tuid); indexMultiTermDestroy(terms); taosArrayDestroy(pTagVals); #endif - return 0; + return code; +_exception: + indexMultiTermDestroy(terms); + taosArrayDestroy(pTagVals); + return code; } int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSchema *pSchema) { #ifdef USE_INVERTED_INDEX @@ -191,7 +209,11 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche } SIndexMultiTerm *terms = indexMultiTermCreate(); - int16_t nCols = taosArrayGetSize(pTagVals); + if (terms == NULL) { + return TSDB_CODE_OUT_OF_MEMORY; + } + + int16_t nCols = taosArrayGetSize(pTagVals); for (int i = 0; i < nCols; i++) { STagVal *pTagVal = (STagVal *)taosArrayGet(pTagVals, i); char type = pTagVal->type; @@ -204,8 +226,14 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche term = indexTermCreate(suid, DEL_VALUE, TSDB_DATA_TYPE_VARCHAR, key, nKey, NULL, 0); } else if (type == TSDB_DATA_TYPE_NCHAR) { if (pTagVal->nData > 0) { - char *val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE); + char *val = taosMemoryCalloc(1, pTagVal->nData + VARSTR_HEADER_SIZE); + if (val == NULL) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, NULL, _exception); + } int32_t len = taosUcs4ToMbs((TdUcs4 *)pTagVal->pData, pTagVal->nData, val + VARSTR_HEADER_SIZE); + if (len < 0) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, NULL, _exception); + } memcpy(val, (uint16_t *)&len, VARSTR_HEADER_SIZE); type = TSDB_DATA_TYPE_VARCHAR; term = indexTermCreate(suid, DEL_VALUE, type, key, nKey, val, len); @@ -224,13 +252,20 @@ int metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSche } if (term != NULL) { (void)indexMultiTermAdd(terms, term); + } else { + code = terrno; + goto _exception; } } - (void)indexJsonPut(pMeta->pTagIvtIdx, terms, tuid); + code = indexJsonPut(pMeta->pTagIvtIdx, terms, tuid); indexMultiTermDestroy(terms); taosArrayDestroy(pTagVals); #endif - return 0; + return code; +_exception: + indexMultiTermDestroy(terms); + taosArrayDestroy(pTagVals); + return code; } static inline void metaTimeSeriesNotifyCheck(SMeta *pMeta) { @@ -867,7 +902,6 @@ int metaDropIndexFromSTable(SMeta *pMeta, int64_t version, SDropIndexReq *pReq) goto _err; } - nStbEntry.stbEntry.schemaRow = *row; nStbEntry.stbEntry.schemaTag = *tag; nStbEntry.stbEntry.rsmaParam = oStbEntry.stbEntry.rsmaParam; @@ -1580,7 +1614,12 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl : pAlterTbReq->compress; (void)updataTableColCmpr(&entry.colCmpr, pCol, 1, compress); freeColCmpr = true; - ASSERT(entry.colCmpr.nCols == pSchema->nCols); + if (entry.colCmpr.nCols != pSchema->nCols) { + if (pNewSchema) taosMemoryFree(pNewSchema); + if (freeColCmpr) taosMemoryFree(entry.colCmpr.pColCmpr); + terrno = TSDB_CODE_VND_INVALID_TABLE_ACTION; + goto _err; + } break; case TSDB_ALTER_TABLE_DROP_COLUMN: if (pColumn == NULL) { @@ -1617,7 +1656,10 @@ static int metaAlterTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pAl } (void)updataTableColCmpr(&entry.colCmpr, &tScheam, 0, 0); - ASSERT(entry.colCmpr.nCols == pSchema->nCols); + if (entry.colCmpr.nCols != pSchema->nCols) { + terrno = TSDB_CODE_VND_INVALID_TABLE_ACTION; + goto _err; + } break; case TSDB_ALTER_TABLE_UPDATE_COLUMN_BYTES: if (pColumn == NULL) { @@ -1698,7 +1740,7 @@ _err: (void)tdbTbcClose(pUidIdxc); tDecoderClear(&dc); - return TSDB_CODE_FAILED; + return terrno != 0 ? terrno : TSDB_CODE_FAILED; } static int metaUpdateTableTagVal(SMeta *pMeta, int64_t version, SVAlterTbReq *pAlterTbReq) { @@ -2514,8 +2556,7 @@ static int metaUpdateTagIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry) { if (tdbTbGet(pMeta->pUidIdx, &pCtbEntry->ctbEntry.suid, sizeof(tb_uid_t), &pData, &nData) != 0) { metaError("vgId:%d, failed to get stable suid for update. version:%" PRId64, TD_VID(pMeta->pVnode), pCtbEntry->version); - terrno = TSDB_CODE_TDB_INVALID_TABLE_ID; - ret = -1; + ret = TSDB_CODE_TDB_INVALID_TABLE_ID; goto end; } tbDbKey.uid = pCtbEntry->ctbEntry.suid; @@ -2529,6 +2570,7 @@ static int metaUpdateTagIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry) { } if (stbEntry.stbEntry.schemaTag.pSchema == NULL) { + ret = TSDB_CODE_INVALID_PARA; goto end; } @@ -2573,7 +2615,9 @@ static int metaUpdateTagIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry) { } } end: - // metaDestroyTagIdxKey(pTagIdxKey); + if (terrno != 0) { + ret = terrno; + } tDecoderClear(&dc); tdbFree(pData); return ret; diff --git a/source/dnode/vnode/src/meta/metaTtl.c b/source/dnode/vnode/src/meta/metaTtl.c index cd1aa7bcad..0b5b9280df 100644 --- a/source/dnode/vnode/src/meta/metaTtl.c +++ b/source/dnode/vnode/src/meta/metaTtl.c @@ -408,7 +408,7 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) { int64_t startNs = taosGetTimestampNs(); int64_t endNs = startNs; - metaTrace("%s, ttl mgr flush start. dirty uids:%d", pTtlMgr->logPrefix, taosHashGetSize(pTtlMgr->pDirtyUids)); + metaTrace("%s, ttl mgr flush start. num of dirty uids:%d", pTtlMgr->logPrefix, taosHashGetSize(pTtlMgr->pDirtyUids)); int32_t code = TSDB_CODE_SUCCESS; @@ -419,13 +419,8 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) { STtlCacheEntry *cacheEntry = taosHashGet(pTtlMgr->pTtlCache, pUid, sizeof(*pUid)); if (cacheEntry == NULL) { - metaInfo("%s, ttlMgr flush failed to get ttl cache, uid: %" PRId64 ", type: %d", pTtlMgr->logPrefix, *pUid, + metaError("%s, ttlMgr flush failed to get ttl cache, uid: %" PRId64 ", type: %d", pTtlMgr->logPrefix, *pUid, pEntry->type); - code = taosHashRemove(pTtlMgr->pDirtyUids, pUid, sizeof(*pUid)); - if (TSDB_CODE_SUCCESS != code) { - metaError("%s, ttlMgr flush failed to remove dirty uid since %s", pTtlMgr->logPrefix, tstrerror(code)); - goto _out; - } continue; } @@ -437,31 +432,35 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) { if (pEntry->type == ENTRY_TYPE_UPSERT) { // delete old key & upsert new key - (void)tdbTbDelete(pTtlMgr->pTtlIdx, &ttlKey, sizeof(ttlKey), pTxn); // maybe first insert, ignore error + code = tdbTbDelete(pTtlMgr->pTtlIdx, &ttlKey, sizeof(ttlKey), pTxn); // maybe first insert, ignore error + if (TSDB_CODE_SUCCESS != code && TSDB_CODE_NOT_FOUND != code) { + metaError("%s, ttlMgr flush failed to delete since %s", pTtlMgr->logPrefix, tstrerror(code)); + continue; + } code = tdbTbUpsert(pTtlMgr->pTtlIdx, &ttlKeyDirty, sizeof(ttlKeyDirty), &cacheEntry->ttlDaysDirty, sizeof(cacheEntry->ttlDaysDirty), pTxn); if (TSDB_CODE_SUCCESS != code) { metaError("%s, ttlMgr flush failed to upsert since %s", pTtlMgr->logPrefix, tstrerror(code)); - goto _out; + continue; } cacheEntry->ttlDays = cacheEntry->ttlDaysDirty; cacheEntry->changeTimeMs = cacheEntry->changeTimeMsDirty; } else if (pEntry->type == ENTRY_TYPE_DELETE) { code = tdbTbDelete(pTtlMgr->pTtlIdx, &ttlKey, sizeof(ttlKey), pTxn); - if (TSDB_CODE_SUCCESS != code) { + if (TSDB_CODE_SUCCESS != code && TSDB_CODE_NOT_FOUND != code) { metaError("%s, ttlMgr flush failed to delete since %s", pTtlMgr->logPrefix, tstrerror(code)); - goto _out; + continue; } code = taosHashRemove(pTtlMgr->pTtlCache, pUid, sizeof(*pUid)); if (TSDB_CODE_SUCCESS != code) { metaError("%s, ttlMgr flush failed to remove cache since %s", pTtlMgr->logPrefix, tstrerror(code)); - goto _out; + continue; } } else { metaError("%s, ttlMgr flush failed, unknown type: %d", pTtlMgr->logPrefix, pEntry->type); - goto _out; + continue; } metaDebug("isdel:%d", pEntry->type == ENTRY_TYPE_DELETE); @@ -471,11 +470,18 @@ int32_t ttlMgrFlush(STtlManger *pTtlMgr, TXN *pTxn) { code = taosHashRemove(pTtlMgr->pDirtyUids, pUid, sizeof(*pUid)); if (TSDB_CODE_SUCCESS != code) { metaError("%s, ttlMgr flush failed to remove dirty uid since %s", pTtlMgr->logPrefix, tstrerror(code)); - goto _out; + continue; } } - taosHashClear(pTtlMgr->pDirtyUids); + int32_t count = taosHashGetSize(pTtlMgr->pDirtyUids); + if (count != 0) { + taosHashClear(pTtlMgr->pDirtyUids); + metaError("%s, ttlMgr flush failed, dirty uids not empty, count: %d", pTtlMgr->logPrefix, count); + code = TSDB_CODE_VND_TTL_FLUSH_INCOMPLETION; + + goto _out; + } code = TSDB_CODE_SUCCESS; diff --git a/source/dnode/vnode/src/sma/smaEnv.c b/source/dnode/vnode/src/sma/smaEnv.c index d90e869bd4..264758bf3e 100644 --- a/source/dnode/vnode/src/sma/smaEnv.c +++ b/source/dnode/vnode/src/sma/smaEnv.c @@ -213,7 +213,7 @@ static int32_t tdInitSmaStat(SSmaStat **pSmaStat, int8_t smaType, const SSma *pS TAOS_CHECK_GOTO(code, &lino, _exit); } SSDataBlock datablock = {.info.type = STREAM_CHECKPOINT}; - (void)taosArrayPush(pRSmaStat->blocks, &datablock); + TSDB_CHECK_NULL(taosArrayPush(pRSmaStat->blocks, &datablock), code, lino, _exit, TSDB_CODE_OUT_OF_MEMORY); // init smaMgmt TAOS_CHECK_GOTO(smaInit(), &lino, _exit); diff --git a/source/dnode/vnode/src/sma/smaRollup.c b/source/dnode/vnode/src/sma/smaRollup.c index ebff03ff99..d77b933e28 100644 --- a/source/dnode/vnode/src/sma/smaRollup.c +++ b/source/dnode/vnode/src/sma/smaRollup.c @@ -928,7 +928,7 @@ static int32_t tdAcquireRSmaInfoBySuid(SSma *pSma, int64_t suid, SRSmaInfo **ppR tdRefRSmaInfo(pSma, pRSmaInfo); taosRUnLockLatch(SMA_ENV_LOCK(pEnv)); - if (ASSERTS(pRSmaInfo->suid == suid, "suid:%" PRIi64 " != %" PRIi64, pRSmaInfo->suid, suid)) { + if (pRSmaInfo->suid != suid) { TAOS_RETURN(TSDB_CODE_APP_ERROR); } *ppRSmaInfo = pRSmaInfo; @@ -1430,7 +1430,7 @@ static void tdFreeRSmaSubmitItems(SArray *pItems, int32_t type) { blockDataDestroy(packData->pDataBlock); } } else { - ASSERTS(0, "unknown type:%d", type); + smaWarn("%s:%d unknown type:%d", __func__, __LINE__, type); } taosArrayClear(pItems); } @@ -1540,14 +1540,13 @@ static int32_t tdRSmaBatchExec(SSma *pSma, SRSmaInfo *pInfo, STaosQall *qall, SA ++nDelete; } } else { - ASSERTS(0, "unknown msg type:%d", inputType); + smaWarn("%s:%d unknown msg type:%d", __func__, __LINE__, inputType); break; } } if (nSubmit > 0 || nDelete > 0) { int32_t size = TARRAY_SIZE(pSubmitArr); - ASSERTS(size > 0, "size is %d", size); int32_t inputType = nSubmit > 0 ? STREAM_INPUT__MERGED_SUBMIT : STREAM_INPUT__REF_DATA_BLOCK; for (int32_t i = 1; i <= TSDB_RETENTION_L2; ++i) { TAOS_CHECK_EXIT(tdExecuteRSmaImpl(pSma, pSubmitArr->pData, size, version, inputType, pInfo, type, i)); @@ -1645,7 +1644,7 @@ int32_t tdRSmaProcessExecImpl(SSma *pSma, ERsmaExecType type) { ((oldStat == 2) && atomic_load_8(RSMA_TRIGGER_STAT(pRSmaStat)) < TASK_TRIGGER_STAT_PAUSED)) { int32_t oldVal = atomic_fetch_add_32(&pRSmaStat->nFetchAll, 1); - if (ASSERTS(oldVal >= 0, "oldVal of nFetchAll: %d < 0", oldVal)) { + if (oldVal < 0) { code = TSDB_CODE_APP_ERROR; taosHashCancelIterate(infoHash, pIter); TSDB_CHECK_CODE(code, lino, _exit); @@ -1677,7 +1676,7 @@ int32_t tdRSmaProcessExecImpl(SSma *pSma, ERsmaExecType type) { } } } else { - ASSERTS(0, "unknown rsma exec type:%d", (int32_t)type); + smaWarn("%s:%d unknown rsma exec type:%d", __func__, __LINE__, (int32_t)type); code = TSDB_CODE_APP_ERROR; TSDB_CHECK_CODE(code, lino, _exit); } diff --git a/source/dnode/vnode/src/sma/smaTimeRange.c b/source/dnode/vnode/src/sma/smaTimeRange.c index 96010728c2..9498e8c5ff 100644 --- a/source/dnode/vnode/src/sma/smaTimeRange.c +++ b/source/dnode/vnode/src/sma/smaTimeRange.c @@ -350,8 +350,7 @@ static int32_t tdProcessTSmaInsertImpl(SSma *pSma, int64_t indexUid, const char } } - if (ASSERTS(pTsmaStat->pTSma->indexUid == indexUid, "indexUid:%" PRIi64 " != %" PRIi64, pTsmaStat->pTSma->indexUid, - indexUid)) { + if (pTsmaStat->pTSma->indexUid != indexUid) { code = TSDB_CODE_APP_ERROR; TSDB_CHECK_CODE(code, lino, _exit); } diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index 5fc550da32..b280d62b3a 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -173,7 +173,7 @@ void tqPushEmptyDataRsp(STqHandle* pHandle, int32_t vgId) { dataRsp.common.blockNum = 0; char buf[TSDB_OFFSET_LEN] = {0}; (void)tFormatOffset(buf, TSDB_OFFSET_LEN, &dataRsp.common.reqOffset); - tqInfo("tqPushEmptyDataRsp to consumer:0x%" PRIx64 " vgId:%d, offset:%s, reqId:0x%" PRIx64, req.consumerId, vgId, buf, + tqInfo("tqPushEmptyDataRsp to consumer:0x%" PRIx64 " vgId:%d, offset:%s, qid:0x%" PRIx64, req.consumerId, vgId, buf, req.reqId); code = tqSendDataRsp(pHandle, pHandle->msg, &req, &dataRsp, TMQ_MSG_TYPE__POLL_DATA_RSP, vgId); @@ -193,7 +193,7 @@ int32_t tqSendDataRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SMqPollReq* (void)tFormatOffset(buf1, TSDB_OFFSET_LEN, &((SMqDataRspCommon*)pRsp)->reqOffset); (void)tFormatOffset(buf2, TSDB_OFFSET_LEN, &((SMqDataRspCommon*)pRsp)->rspOffset); - tqDebug("tmq poll vgId:%d consumer:0x%" PRIx64 " (epoch %d) send rsp, block num:%d, req:%s, rsp:%s, reqId:0x%" PRIx64, + tqDebug("tmq poll vgId:%d consumer:0x%" PRIx64 " (epoch %d) send rsp, block num:%d, req:%s, rsp:%s, qid:0x%" PRIx64, vgId, pReq->consumerId, pReq->epoch, ((SMqDataRspCommon*)pRsp)->blockNum, buf1, buf2, pReq->reqId); return tqDoSendDataRsp(&pMsg->info, pRsp, pReq->epoch, pReq->consumerId, type, sver, ever); @@ -335,7 +335,7 @@ int32_t tqProcessPollPush(STQ* pTq, SRpcMsg* pMsg) { STqHandle* pHandle = *(STqHandle**)pIter; tqDebug("vgId:%d start set submit for pHandle:%p, consumer:0x%" PRIx64, vgId, pHandle, pHandle->consumerId); - if (ASSERT(pHandle->msg != NULL)) { + if (pHandle->msg == NULL) { tqError("pHandle->msg should not be null"); taosHashCancelIterate(pTq->pPushMgr, pIter); break; @@ -421,7 +421,7 @@ int32_t tqProcessPollReq(STQ* pTq, SRpcMsg* pMsg) { char buf[TSDB_OFFSET_LEN] = {0}; (void)tFormatOffset(buf, TSDB_OFFSET_LEN, &reqOffset); - tqDebug("tmq poll: consumer:0x%" PRIx64 " (epoch %d), subkey %s, recv poll req vgId:%d, req:%s, reqId:0x%" PRIx64, + tqDebug("tmq poll: consumer:0x%" PRIx64 " (epoch %d), subkey %s, recv poll req vgId:%d, req:%s, qid:0x%" PRIx64, consumerId, req.epoch, pHandle->subKey, vgId, buf, req.reqId); code = tqExtractDataForMq(pTq, pHandle, &req, pMsg); @@ -777,7 +777,11 @@ int32_t tqBuildStreamTask(void* pTqObj, SStreamTask* pTask, int64_t nextProcessV pTask->info.selfChildId, pTask->info.taskLevel, p, pNext, pTask->info.fillHistory, (int32_t)pTask->hTaskInfo.id.taskId, pTask->info.delaySchedParam, nextProcessVer); - ASSERT(pChkInfo->checkpointVer <= pChkInfo->nextProcessVer); + if (pChkInfo->checkpointVer > pChkInfo->nextProcessVer) { + tqError("vgId:%d build stream task, s-task:%s, checkpointVer:%" PRId64 " > nextProcessVer:%" PRId64, vgId, + pTask->id.idStr, pChkInfo->checkpointVer, pChkInfo->nextProcessVer); + return TSDB_CODE_STREAM_INTERNAL_ERROR; + } } return 0; @@ -817,7 +821,9 @@ static void doStartFillhistoryStep2(SStreamTask* pTask, SStreamTask* pStreamTask ", do secondary scan-history from WAL after halt the related stream task:%s", id, pTask->info.taskLevel, pStep2Range->minVer, pStep2Range->maxVer, pWindow->skey, pWindow->ekey, pStreamTask->id.idStr); - ASSERT(pTask->status.schedStatus == TASK_SCHED_STATUS__WAITING); + if (pTask->status.schedStatus != TASK_SCHED_STATUS__WAITING) { + tqError("s-task:%s level:%d unexpected sched-status:%d", id, pTask->info.taskLevel, pTask->status.schedStatus); + } int32_t code = streamSetParamForStreamScannerStep2(pTask, pStep2Range, pWindow); if (code) { @@ -950,7 +956,10 @@ int32_t tqProcessTaskScanHistory(STQ* pTq, SRpcMsg* pMsg) { // the following procedure should be executed, no matter status is stop/pause or not tqDebug("s-task:%s scan-history(step 1) ended, elapsed time:%.2fs", id, pTask->execInfo.step1El); - ASSERT(pTask->info.fillHistory == 1); + if (pTask->info.fillHistory != 1) { + tqError("s-task:%s fill-history is disabled, unexpected", id); + return TSDB_CODE_STREAM_INTERNAL_ERROR; + } // 1. get the related stream task SStreamTask* pStreamTask = NULL; @@ -967,7 +976,10 @@ int32_t tqProcessTaskScanHistory(STQ* pTq, SRpcMsg* pMsg) { return code; // todo: handle failure } - ASSERT(pStreamTask->info.taskLevel == TASK_LEVEL__SOURCE); + if (pStreamTask->info.taskLevel != TASK_LEVEL__SOURCE) { + tqError("s-task:%s fill-history task related stream task level:%d, unexpected", id, pStreamTask->info.taskLevel); + return TSDB_CODE_STREAM_INTERNAL_ERROR; + } code = streamTaskHandleEventAsync(pStreamTask->status.pSM, TASK_EVENT_HALT, handleStep2Async, pTq); streamMetaReleaseTask(pMeta, pStreamTask); @@ -1204,13 +1216,13 @@ int32_t tqProcessTaskCheckPointSourceReq(STQ* pTq, SRpcMsg* pMsg, SRpcMsg* pRsp) } if (req.mndTrigger) { - tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 ", transId:%d, ", pTask->id.idStr, - vgId, pTask->info.taskLevel, req.checkpointId, req.transId); + tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 ", transId:%d, ", + pTask->id.idStr, vgId, pTask->info.taskLevel, req.checkpointId, req.transId); } else { const char* pPrevStatus = streamTaskGetStatusStr(streamTaskGetPrevStatus(pTask)); tqInfo("s-task:%s (vgId:%d) level:%d receive checkpoint-source msg chkpt:%" PRId64 - ", transId:%d after transfer-state, prev status:%s", - pTask->id.idStr, vgId, pTask->info.taskLevel, req.checkpointId, req.transId, pPrevStatus); + ", transId:%d after transfer-state, prev status:%s", + pTask->id.idStr, vgId, pTask->info.taskLevel, req.checkpointId, req.transId, pPrevStatus); } code = streamAddCheckpointSourceRspMsg(&req, &pMsg->info, pTask); diff --git a/source/dnode/vnode/src/tq/tqRead.c b/source/dnode/vnode/src/tq/tqRead.c index c779a17301..98cfc6888c 100644 --- a/source/dnode/vnode/src/tq/tqRead.c +++ b/source/dnode/vnode/src/tq/tqRead.c @@ -74,7 +74,7 @@ bool isValValidForTable(STqHandle* pHandle, SWalCont* pHead) { pCreateReq = req.pReqs + iReq; if (pCreateReq->type == TSDB_CHILD_TABLE && pCreateReq->ctb.suid == tbSuid) { reqNew.nReqs++; - if (taosArrayPush(reqNew.pArray, pCreateReq) == NULL){ + if (taosArrayPush(reqNew.pArray, pCreateReq) == NULL) { taosArrayDestroy(reqNew.pArray); tDeleteSVCreateTbBatchReq(&req); goto end; @@ -155,7 +155,7 @@ bool isValValidForTable(STqHandle* pHandle, SWalCont* pHead) { pDropReq = req.pReqs + iReq; if (pDropReq->suid == tbSuid) { reqNew.nReqs++; - if (taosArrayPush(reqNew.pArray, pDropReq) == NULL){ + if (taosArrayPush(reqNew.pArray, pDropReq) == NULL) { taosArrayDestroy(reqNew.pArray); goto end; } @@ -214,12 +214,12 @@ int32_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, uint64_t while (offset <= appliedVer) { if (walFetchHead(pHandle->pWalReader, offset) < 0) { tqDebug("tmq poll: consumer:0x%" PRIx64 ", (epoch %d) vgId:%d offset %" PRId64 - ", no more log to return, reqId:0x%" PRIx64 " 0x%" PRIx64, + ", no more log to return, qid:0x%" PRIx64 " 0x%" PRIx64, pHandle->consumerId, pHandle->epoch, vgId, offset, reqId, id); goto END; } - tqDebug("vgId:%d, consumer:0x%" PRIx64 " taosx get msg ver %" PRId64 ", type: %s, reqId:0x%" PRIx64 " 0x%" PRIx64, + tqDebug("vgId:%d, consumer:0x%" PRIx64 " taosx get msg ver %" PRId64 ", type: %s, qid:0x%" PRIx64 " 0x%" PRIx64, vgId, pHandle->consumerId, offset, TMSG_INFO(pHandle->pWalReader->pHead->head.msgType), reqId, id); if (pHandle->pWalReader->pHead->head.msgType == TDMT_VND_SUBMIT) { @@ -261,10 +261,10 @@ END: bool tqGetTablePrimaryKey(STqReader* pReader) { return pReader->hasPrimaryKey; } -void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid){ - bool ret = false; - SSchemaWrapper *schema = metaGetTableSchema(pReader->pVnodeMeta, uid, -1, 1); - if (schema && schema->nCols >= 2 && schema->pSchema[1].flags & COL_IS_KEY){ +void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) { + bool ret = false; + SSchemaWrapper* schema = metaGetTableSchema(pReader->pVnodeMeta, uid, -1, 1); + if (schema && schema->nCols >= 2 && schema->pSchema[1].flags & COL_IS_KEY) { ret = true; } tDeleteSchemaWrapper(schema); @@ -386,7 +386,8 @@ int32_t extractMsgFromWal(SWalReader* pReader, void** pItem, int64_t maxVer, con } } else { - ASSERT(0); + tqError("s-task:%s invalid msg type:%d, ver:%" PRId64, id, pCont->msgType, ver); + return TSDB_CODE_STREAM_INTERNAL_ERROR; } return code; @@ -486,7 +487,7 @@ bool tqNextBlockImpl(STqReader* pReader, const char* idstr) { (pReader->nextBlk + 1), numOfBlocks, idstr); SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); - if (pSubmitTbData == NULL){ + if (pSubmitTbData == NULL) { return false; } if (pReader->tbIdHash == NULL) { @@ -639,9 +640,9 @@ static int32_t doSetVal(SColumnInfoData* pColumnInfoData, int32_t rowIndex, SCol int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* id) { tqTrace("tq reader retrieve data block %p, index:%d", pReader->msg.msgStr, pReader->nextBlk); - int32_t code = 0; - int32_t line = 0; - STSchema* pTSchema = NULL; + int32_t code = 0; + int32_t line = 0; + STSchema* pTSchema = NULL; SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk++); TSDB_CHECK_NULL(pSubmitTbData, code, line, END, terrno); SSDataBlock* pBlock = pReader->pResBlock; @@ -675,7 +676,11 @@ int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* pReader->cachedSchemaSuid = suid; pReader->cachedSchemaVer = sversion; - ASSERT(pReader->cachedSchemaVer == pReader->pSchemaWrapper->version); + if (pReader->cachedSchemaVer != pReader->pSchemaWrapper->version) { + tqError("vgId:%d, schema version mismatch, suid:%" PRId64 ", uid:%" PRId64 ", version:%d, cached version:%d", + vgId, suid, uid, sversion, pReader->pSchemaWrapper->version); + return TSDB_CODE_TQ_INTERNAL_ERROR; + } if (blockDataGetNumOfCols(pBlock) == 0) { code = buildResSDataBlock(pReader->pResBlock, pReader->pSchemaWrapper, pReader->pColIdList); TSDB_CHECK_CODE(code, line, END); @@ -712,10 +717,11 @@ int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* continue; } - SColData* pCol = taosArrayGet(pCols, sourceIdx); + SColData* pCol = taosArrayGet(pCols, sourceIdx); TSDB_CHECK_NULL(pCol, code, line, END, terrno); - SColVal colVal = {0}; - tqTrace("lostdata colActual:%d, sourceIdx:%d, targetIdx:%d, numOfCols:%d, source cid:%d, dst cid:%d", colActual, sourceIdx, targetIdx, numOfCols, pCol->cid, pColData->info.colId); + SColVal colVal = {0}; + tqTrace("lostdata colActual:%d, sourceIdx:%d, targetIdx:%d, numOfCols:%d, source cid:%d, dst cid:%d", colActual, + sourceIdx, targetIdx, numOfCols, pCol->cid, pColData->info.colId); if (pCol->cid < pColData->info.colId) { sourceIdx++; } else if (pCol->cid == pColData->info.colId) { @@ -738,7 +744,7 @@ int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* TSDB_CHECK_NULL(pTSchema, code, line, END, terrno); for (int32_t i = 0; i < numOfRows; i++) { - SRow* pRow = taosArrayGetP(pRows, i); + SRow* pRow = taosArrayGetP(pRows, i); TSDB_CHECK_NULL(pRow, code, line, END, terrno); int32_t sourceIdx = 0; for (int32_t j = 0; j < colActual; j++) { @@ -765,44 +771,43 @@ int32_t tqRetrieveDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* } } } - } END: - if (code != 0){ + if (code != 0) { tqError("tqRetrieveDataBlock failed, line:%d, code:%d", line, code); } taosMemoryFreeClear(pTSchema); return code; } -#define PROCESS_VAL \ - if (curRow == 0) {\ - assigned[j] = !COL_VAL_IS_NONE(&colVal);\ - buildNew = true;\ - } else {\ - bool currentRowAssigned = !COL_VAL_IS_NONE(&colVal);\ - if (currentRowAssigned != assigned[j]) {\ - assigned[j] = currentRowAssigned;\ - buildNew = true;\ - }\ +#define PROCESS_VAL \ + if (curRow == 0) { \ + assigned[j] = !COL_VAL_IS_NONE(&colVal); \ + buildNew = true; \ + } else { \ + bool currentRowAssigned = !COL_VAL_IS_NONE(&colVal); \ + if (currentRowAssigned != assigned[j]) { \ + assigned[j] = currentRowAssigned; \ + buildNew = true; \ + } \ } -#define SET_DATA \ - if (colVal.cid < pColData->info.colId) {\ - sourceIdx++;\ - } else if (colVal.cid == pColData->info.colId) {\ - TQ_ERR_GO_TO_END(doSetVal(pColData, curRow - lastRow, &colVal));\ - sourceIdx++;\ - targetIdx++;\ +#define SET_DATA \ + if (colVal.cid < pColData->info.colId) { \ + sourceIdx++; \ + } else if (colVal.cid == pColData->info.colId) { \ + TQ_ERR_GO_TO_END(doSetVal(pColData, curRow - lastRow, &colVal)); \ + sourceIdx++; \ + targetIdx++; \ } -static int32_t processBuildNew(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, - SArray* schemas, SSchemaWrapper* pSchemaWrapper, char* assigned, - int32_t numOfRows, int32_t curRow, int32_t* lastRow){ - int32_t code = 0; - SSchemaWrapper* pSW = NULL; - SSDataBlock* block = NULL; +static int32_t processBuildNew(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas, + SSchemaWrapper* pSchemaWrapper, char* assigned, int32_t numOfRows, int32_t curRow, + int32_t* lastRow) { + int32_t code = 0; + SSchemaWrapper* pSW = NULL; + SSDataBlock* block = NULL; if (taosArrayGetSize(blocks) > 0) { SSDataBlock* pLastBlock = taosArrayGetLast(blocks); TQ_NULL_GO_TO_END(pLastBlock); @@ -834,33 +839,34 @@ END: taosMemoryFree(block); return code; } -static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas){ - int32_t code = 0; - int32_t curRow = 0; - int32_t lastRow = 0; +static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas) { + int32_t code = 0; + int32_t curRow = 0; + int32_t lastRow = 0; SSchemaWrapper* pSchemaWrapper = pReader->pSchemaWrapper; - char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols); + char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols); TQ_NULL_GO_TO_END(assigned); - SArray* pCols = pSubmitTbData->aCol; - SColData* pCol = taosArrayGet(pCols, 0); + SArray* pCols = pSubmitTbData->aCol; + SColData* pCol = taosArrayGet(pCols, 0); TQ_NULL_GO_TO_END(pCol); - int32_t numOfRows = pCol->nVal; - int32_t numOfCols = taosArrayGetSize(pCols); + int32_t numOfRows = pCol->nVal; + int32_t numOfCols = taosArrayGetSize(pCols); for (int32_t i = 0; i < numOfRows; i++) { bool buildNew = false; for (int32_t j = 0; j < numOfCols; j++) { pCol = taosArrayGet(pCols, j); TQ_NULL_GO_TO_END(pCol); - SColVal colVal = {0}; + SColVal colVal = {0}; tColDataGetValue(pCol, i, &colVal); PROCESS_VAL } if (buildNew) { - TQ_ERR_GO_TO_END(processBuildNew(pReader, pSubmitTbData, blocks, schemas, pSchemaWrapper, assigned, numOfRows, curRow, &lastRow)); + TQ_ERR_GO_TO_END(processBuildNew(pReader, pSubmitTbData, blocks, schemas, pSchemaWrapper, assigned, numOfRows, + curRow, &lastRow)); } SSDataBlock* pBlock = taosArrayGetLast(blocks); @@ -877,7 +883,7 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData TQ_NULL_GO_TO_END(pCol); SColumnInfoData* pColData = taosArrayGet(pBlock->pDataBlock, targetIdx); TQ_NULL_GO_TO_END(pColData); - SColVal colVal = {0}; + SColVal colVal = {0}; tColDataGetValue(pCol, i, &colVal); SET_DATA } @@ -885,30 +891,30 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData curRow++; } SSDataBlock* pLastBlock = taosArrayGetLast(blocks); - pLastBlock->info.rows = curRow - lastRow; + pLastBlock->info.rows = curRow - lastRow; END: taosMemoryFree(assigned); return code; } -int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas){ - int32_t code = 0; - STSchema* pTSchema = NULL; +int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArray* blocks, SArray* schemas) { + int32_t code = 0; + STSchema* pTSchema = NULL; SSchemaWrapper* pSchemaWrapper = pReader->pSchemaWrapper; - char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols); + char* assigned = taosMemoryCalloc(1, pSchemaWrapper->nCols); TQ_NULL_GO_TO_END(assigned); - int32_t curRow = 0; - int32_t lastRow = 0; - SArray* pRows = pSubmitTbData->aRowP; - int32_t numOfRows = taosArrayGetSize(pRows); - pTSchema = tBuildTSchema(pSchemaWrapper->pSchema, pSchemaWrapper->nCols, pSchemaWrapper->version); + int32_t curRow = 0; + int32_t lastRow = 0; + SArray* pRows = pSubmitTbData->aRowP; + int32_t numOfRows = taosArrayGetSize(pRows); + pTSchema = tBuildTSchema(pSchemaWrapper->pSchema, pSchemaWrapper->nCols, pSchemaWrapper->version); for (int32_t i = 0; i < numOfRows; i++) { bool buildNew = false; - SRow* pRow = taosArrayGetP(pRows, i); + SRow* pRow = taosArrayGetP(pRows, i); TQ_NULL_GO_TO_END(pRow); for (int32_t j = 0; j < pTSchema->numOfCols; j++) { @@ -918,7 +924,8 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra } if (buildNew) { - TQ_ERR_GO_TO_END(processBuildNew(pReader, pSubmitTbData, blocks, schemas, pSchemaWrapper, assigned, numOfRows, curRow, &lastRow)); + TQ_ERR_GO_TO_END(processBuildNew(pReader, pSubmitTbData, blocks, schemas, pSchemaWrapper, assigned, numOfRows, + curRow, &lastRow)); } SSDataBlock* pBlock = taosArrayGetLast(blocks); @@ -932,7 +939,7 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra int32_t colActual = blockDataGetNumOfCols(pBlock); while (targetIdx < colActual) { SColumnInfoData* pColData = taosArrayGet(pBlock->pDataBlock, targetIdx); - SColVal colVal = {0}; + SColVal colVal = {0}; TQ_ERR_GO_TO_END(tRowGet(pRow, pTSchema, sourceIdx, &colVal)); SET_DATA } @@ -940,7 +947,7 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra curRow++; } SSDataBlock* pLastBlock = taosArrayGetLast(blocks); - pLastBlock->info.rows = curRow - lastRow; + pLastBlock->info.rows = curRow - lastRow; END: taosMemoryFreeClear(pTSchema); @@ -950,10 +957,10 @@ END: int32_t tqRetrieveTaosxBlock(STqReader* pReader, SArray* blocks, SArray* schemas, SSubmitTbData** pSubmitTbDataRet) { tqDebug("tq reader retrieve data block %p, %d", pReader->msg.msgStr, pReader->nextBlk); - SSDataBlock* block = NULL; + SSDataBlock* block = NULL; SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk); - if(pSubmitTbData == NULL){ + if (pSubmitTbData == NULL) { return terrno; } pReader->nextBlk++; @@ -1034,7 +1041,7 @@ bool tqCurrentBlockConsumed(const STqReader* pReader) { return pReader->msg.msgS void tqReaderRemoveTbUidList(STqReader* pReader, const SArray* tbUidList) { for (int32_t i = 0; i < taosArrayGetSize(tbUidList); i++) { int64_t* pKey = (int64_t*)taosArrayGet(tbUidList, i); - if (pKey && taosHashRemove(pReader->tbIdHash, pKey, sizeof(int64_t)) != 0){ + if (pKey && taosHashRemove(pReader->tbIdHash, pKey, sizeof(int64_t)) != 0) { tqError("failed to remove table uid:%" PRId64 " from hash", *pKey); } } @@ -1064,7 +1071,8 @@ int32_t tqUpdateTbUidList(STQ* pTq, const SArray* tbUidList, bool isAdd) { int32_t sz = taosArrayGetSize(tbUidList); for (int32_t i = 0; i < sz; i++) { int64_t* tbUid = (int64_t*)taosArrayGet(tbUidList, i); - if (tbUid && taosHashPut(pTqHandle->execHandle.execDb.pFilterOutTbUid, tbUid, sizeof(int64_t), NULL, 0) != 0){ + if (tbUid && + taosHashPut(pTqHandle->execHandle.execDb.pFilterOutTbUid, tbUid, sizeof(int64_t), NULL, 0) != 0) { tqError("failed to add table uid:%" PRId64 " to hash", *tbUid); continue; } diff --git a/source/dnode/vnode/src/tq/tqScan.c b/source/dnode/vnode/src/tq/tqScan.c index 4357456790..8ca50f9de7 100644 --- a/source/dnode/vnode/src/tq/tqScan.c +++ b/source/dnode/vnode/src/tq/tqScan.c @@ -29,6 +29,10 @@ int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, void* pRsp, int32_t numOf pRetrieve->numOfRows = htobe64((int64_t)pBlock->info.rows); int32_t actualLen = blockEncode(pBlock, pRetrieve->data, numOfCols); + if(actualLen < 0){ + taosMemoryFree(buf); + return terrno; + } actualLen += sizeof(SRetrieveTableRspForTmq); if (taosArrayPush(((SMqDataRspCommon*)pRsp)->blockDataLen, &actualLen) == NULL){ taosMemoryFree(buf); diff --git a/source/dnode/vnode/src/tq/tqUtil.c b/source/dnode/vnode/src/tq/tqUtil.c index b3d2300996..fc97a542a7 100644 --- a/source/dnode/vnode/src/tq/tqUtil.c +++ b/source/dnode/vnode/src/tq/tqUtil.c @@ -92,7 +92,7 @@ static int32_t extractResetOffsetVal(STqOffsetVal* pOffsetVal, STQ* pTq, STqHand char formatBuf[TSDB_OFFSET_LEN] = {0}; tFormatOffset(formatBuf, TSDB_OFFSET_LEN, pOffsetVal); tqDebug("tmq poll: consumer:0x%" PRIx64 - ", subkey %s, vgId:%d, existed offset found, offset reset to %s and continue. reqId:0x%" PRIx64, + ", subkey %s, vgId:%d, existed offset found, offset reset to %s and continue. qid:0x%" PRIx64, consumerId, pHandle->subKey, vgId, formatBuf, pRequest->reqId); return 0; } else { @@ -117,7 +117,7 @@ static int32_t extractResetOffsetVal(STqOffsetVal* pOffsetVal, STQ* pTq, STqHand tqOffsetResetToLog(pOffsetVal, pHandle->pRef->refVer + 1); code = tqInitDataRsp(&dataRsp.common, *pOffsetVal); - if (code != 0){ + if (code != 0) { return code; } tqDebug("tmq poll: consumer:0x%" PRIx64 ", subkey %s, vgId:%d, (latest) offset reset to %" PRId64, consumerId, @@ -145,7 +145,7 @@ static int32_t extractDataAndRspForNormalSubscribe(STQ* pTq, STqHandle* pHandle, terrno = 0; SMqDataRsp dataRsp = {0}; - int code = tqInitDataRsp(&dataRsp.common, *pOffset); + int code = tqInitDataRsp(&dataRsp.common, *pOffset); if (code != 0) { goto end; } @@ -176,7 +176,7 @@ static int32_t extractDataAndRspForNormalSubscribe(STQ* pTq, STqHandle* pHandle, end : { char buf[TSDB_OFFSET_LEN] = {0}; tFormatOffset(buf, TSDB_OFFSET_LEN, &dataRsp.common.rspOffset); - tqDebug("tmq poll: consumer:0x%" PRIx64 ", subkey %s, vgId:%d, rsp block:%d, rsp offset type:%s, reqId:0x%" PRIx64 + tqDebug("tmq poll: consumer:0x%" PRIx64 ", subkey %s, vgId:%d, rsp block:%d, rsp offset type:%s, qid:0x%" PRIx64 " code:%d", consumerId, pHandle->subKey, vgId, dataRsp.common.blockNum, buf, pRequest->reqId, code); tDeleteMqDataRsp(&dataRsp); @@ -206,10 +206,10 @@ static void tDeleteCommon(void* parm) {} static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg, STqOffsetVal* offset) { - int32_t vgId = TD_VID(pTq->pVnode); - STaosxRsp taosxRsp = {0}; - SMqBatchMetaRsp btMetaRsp = {0}; - int32_t code = 0; + int32_t vgId = TD_VID(pTq->pVnode); + STaosxRsp taosxRsp = {0}; + SMqBatchMetaRsp btMetaRsp = {0}; + int32_t code = 0; TQ_ERR_GO_TO_END(tqInitTaosxRsp(&taosxRsp.common, *offset)); if (offset->type != TMQ_OFFSET__LOG) { @@ -245,13 +245,22 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, int32_t totalMetaRows = 0; while (1) { int32_t savedEpoch = atomic_load_32(&pHandle->epoch); - ASSERT(savedEpoch <= pRequest->epoch); + if (savedEpoch > pRequest->epoch) { + tqError("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, savedEpoch error, vgId:%d offset %" PRId64, + pRequest->consumerId, pRequest->epoch, vgId, fetchVer); + code = TSDB_CODE_TQ_INTERNAL_ERROR; + goto END; + } if (tqFetchLog(pTq, pHandle, &fetchVer, pRequest->reqId) < 0) { if (totalMetaRows > 0) { tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer); code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId); - ASSERT(totalRows == 0); + if (totalRows != 0) { + tqError("tmq poll: consumer:0x%" PRIx64 " (epoch %d) iter log, totalRows error, vgId:%d offset %" PRId64, + pRequest->consumerId, pRequest->epoch, vgId, fetchVer); + code = code == 0 ? TSDB_CODE_TQ_INTERNAL_ERROR : code; + } goto END; } tqOffsetResetToLog(&taosxRsp.common.rspOffset, fetchVer); @@ -302,12 +311,12 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, if (!btMetaRsp.batchMetaReq) { btMetaRsp.batchMetaReq = taosArrayInit(4, POINTER_BYTES); if (btMetaRsp.batchMetaReq == NULL) { - code = TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); + code = TAOS_GET_TERRNO(terrno); goto END; } btMetaRsp.batchMetaLen = taosArrayInit(4, sizeof(int32_t)); if (btMetaRsp.batchMetaLen == NULL) { - code = TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); + code = TAOS_GET_TERRNO(terrno); goto END; } } @@ -323,10 +332,10 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); continue; } - int32_t tLen = sizeof(SMqRspHead) + len; - void* tBuf = taosMemoryCalloc(1, tLen); - if (tBuf == NULL){ - code = TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); + int32_t tLen = sizeof(SMqRspHead) + len; + void* tBuf = taosMemoryCalloc(1, tLen); + if (tBuf == NULL) { + code = TAOS_GET_TERRNO(terrno); goto END; } void* metaBuff = POINTER_SHIFT(tBuf, sizeof(SMqRspHead)); @@ -339,12 +348,12 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); continue; } - if (taosArrayPush(btMetaRsp.batchMetaReq, &tBuf) == NULL){ - code = TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); + if (taosArrayPush(btMetaRsp.batchMetaReq, &tBuf) == NULL) { + code = TAOS_GET_TERRNO(terrno); goto END; } - if (taosArrayPush(btMetaRsp.batchMetaLen, &tLen) == NULL){ - code = TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); + if (taosArrayPush(btMetaRsp.batchMetaLen, &tLen) == NULL) { + code = TAOS_GET_TERRNO(terrno); goto END; } totalMetaRows++; @@ -448,7 +457,7 @@ int32_t tqSendBatchMetaPollRsp(STqHandle* pHandle, const SRpcMsg* pMsg, const SM int32_t tlen = sizeof(SMqRspHead) + len; void* buf = rpcMallocCont(tlen); if (buf == NULL) { - return TAOS_GET_TERRNO(TSDB_CODE_OUT_OF_MEMORY); + return TAOS_GET_TERRNO(terrno); } int64_t sver = 0, ever = 0; @@ -601,7 +610,7 @@ int32_t tqExtractDelDataBlock(const void* pData, int32_t len, int64_t ver, void* SColumnInfoData* pUidCol = taosArrayGet(pDelBlock->pDataBlock, UID_COLUMN_INDEX); TSDB_CHECK_NULL(pUidCol, code, line, END, terrno) - int64_t* pUid = taosArrayGet(pRes->uidList, i); + int64_t* pUid = taosArrayGet(pRes->uidList, i); code = colDataSetVal(pUidCol, i, (const char*)pUid, false); TSDB_CHECK_CODE(code, line, END); void* tmp = taosArrayGet(pDelBlock->pDataBlock, GROUPID_COLUMN_INDEX); @@ -628,11 +637,12 @@ int32_t tqExtractDelDataBlock(const void* pData, int32_t len, int64_t ver, void* } else if (type == 1) { *pRefBlock = pDelBlock; } else { - ASSERTS(0, "unknown type:%d", type); + tqError("unknown type:%d", type); + code = TSDB_CODE_TMQ_CONSUMER_ERROR; } END: - if (code != 0){ + if (code != 0) { tqError("failed to extract delete data block, line:%d code:%d", line, code); } tDecoderClear(pCoder); @@ -662,7 +672,7 @@ int32_t tqGetStreamExecInfo(SVnode* pVnode, int64_t streamId, int64_t* pDelay, b for (int32_t i = 0; i < numOfTasks; ++i) { SStreamTaskId* pId = taosArrayGet(pMeta->pTaskList, i); - if (pId == NULL){ + if (pId == NULL) { continue; } if (pId->streamId != streamId) { diff --git a/source/dnode/vnode/src/tqCommon/tqCommon.c b/source/dnode/vnode/src/tqCommon/tqCommon.c index 422ca16e50..a63c15edfb 100644 --- a/source/dnode/vnode/src/tqCommon/tqCommon.c +++ b/source/dnode/vnode/src/tqCommon/tqCommon.c @@ -207,7 +207,7 @@ int32_t tqStreamTaskProcessUpdateReq(SStreamMeta* pMeta, SMsgCb* cb, SRpcMsg* pM updated = streamTaskUpdateEpsetInfo(pTask, req.pNodeList); // send the checkpoint-source-rsp for source task to end the checkpoint trans in mnode - (void) streamTaskSendCheckpointsourceRsp(pTask); + (void)streamTaskSendCheckpointsourceRsp(pTask); streamTaskResetStatus(pTask); streamTaskStopMonitorCheckRsp(&pTask->taskCheckInfo, pTask->id.idStr); @@ -303,7 +303,7 @@ int32_t tqStreamTaskProcessUpdateReq(SStreamMeta* pMeta, SMsgCb* cb, SRpcMsg* pM streamMetaWUnLock(pMeta); taosArrayDestroy(req.pNodeList); - return rsp.code; // always return true + return rsp.code; // always return true } int32_t tqStreamTaskProcessDispatchReq(SStreamMeta* pMeta, SRpcMsg* pMsg) { @@ -324,7 +324,7 @@ int32_t tqStreamTaskProcessDispatchReq(SStreamMeta* pMeta, SRpcMsg* pMsg) { tqDebug("s-task:0x%x recv dispatch msg from 0x%x(vgId:%d)", req.taskId, req.upstreamTaskId, req.upstreamNodeId); SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, req.streamId, req.taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, req.streamId, req.taskId, &pTask); if (pTask && (code == 0)) { SRpcMsg rsp = {.info = pMsg->info, .code = 0}; if (streamProcessDispatchMsg(pTask, &req, &rsp) != 0) { @@ -384,7 +384,7 @@ int32_t tqStreamTaskProcessDispatchRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) { pRsp->downstreamTaskId, pRsp->downstreamNodeId); SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->upstreamTaskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->upstreamTaskId, &pTask); if (pTask && (code == 0)) { code = streamProcessDispatchRsp(pTask, pRsp, pMsg->code); streamMetaReleaseTask(pMeta, pTask); @@ -482,8 +482,8 @@ int32_t tqStreamTaskProcessCheckRsp(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLe } tDecoderClear(&decoder); - tqDebug("tq task:0x%x (vgId:%d) recv check rsp(reqId:0x%" PRIx64 ") from 0x%x (vgId:%d) status %d", - rsp.upstreamTaskId, rsp.upstreamNodeId, rsp.reqId, rsp.downstreamTaskId, rsp.downstreamNodeId, rsp.status); + tqDebug("tq task:0x%x (vgId:%d) recv check rsp(qid:0x%" PRIx64 ") from 0x%x (vgId:%d) status %d", rsp.upstreamTaskId, + rsp.upstreamNodeId, rsp.reqId, rsp.downstreamTaskId, rsp.downstreamNodeId, rsp.status); if (!isLeader) { tqError("vgId:%d not leader, task:0x%x not handle the check rsp, downstream:0x%x (vgId:%d)", vgId, @@ -681,7 +681,8 @@ int32_t tqStreamTaskProcessDropReq(SStreamMeta* pMeta, char* msg, int32_t msgLen tqDebug("s-task:0x%x vgId:%d drop rel fill-history task:0x%x firstly", pReq->taskId, vgId, (int32_t)hTaskId.taskId); code = streamMetaUnregisterTask(pMeta, hTaskId.streamId, hTaskId.taskId); if (code) { - tqDebug("s-task:0x%x vgId:%d drop rel fill-history task:0x%x failed", pReq->taskId, vgId, (int32_t)hTaskId.taskId); + tqDebug("s-task:0x%x vgId:%d drop rel fill-history task:0x%x failed", pReq->taskId, vgId, + (int32_t)hTaskId.taskId); } } @@ -701,7 +702,7 @@ int32_t tqStreamTaskProcessDropReq(SStreamMeta* pMeta, char* msg, int32_t msgLen } streamMetaWUnLock(pMeta); - return 0; // always return success + return 0; // always return success } int32_t tqStreamTaskProcessUpdateCheckpointReq(SStreamMeta* pMeta, bool restored, char* msg) { @@ -793,23 +794,23 @@ int32_t tqStreamTaskProcessRunReq(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLead int32_t vgId = pMeta->vgId; if (type == STREAM_EXEC_T_START_ONE_TASK) { - (void) streamMetaStartOneTask(pMeta, pReq->streamId, pReq->taskId); + (void)streamMetaStartOneTask(pMeta, pReq->streamId, pReq->taskId); return 0; } else if (type == STREAM_EXEC_T_START_ALL_TASKS) { - (void) streamMetaStartAllTasks(pMeta); + (void)streamMetaStartAllTasks(pMeta); return 0; } else if (type == STREAM_EXEC_T_RESTART_ALL_TASKS) { - (void) restartStreamTasks(pMeta, isLeader); + (void)restartStreamTasks(pMeta, isLeader); return 0; } else if (type == STREAM_EXEC_T_STOP_ALL_TASKS) { - (void) streamMetaStopAllTasks(pMeta); + (void)streamMetaStopAllTasks(pMeta); return 0; } else if (type == STREAM_EXEC_T_ADD_FAILED_TASK) { int32_t code = streamMetaAddFailedTask(pMeta, pReq->streamId, pReq->taskId); return code; } else if (type == STREAM_EXEC_T_RESUME_TASK) { // task resume to run after idle for a while SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); if (pTask != NULL && (code == 0)) { char* pStatus = NULL; @@ -831,13 +832,13 @@ int32_t tqStreamTaskProcessRunReq(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLead } SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); if ((pTask != NULL) && (code == 0)) { // even in halt status, the data in inputQ must be processed char* p = NULL; if (streamTaskReadyToRun(pTask, &p)) { tqDebug("vgId:%d s-task:%s status:%s start to process block from inputQ, next checked ver:%" PRId64, vgId, pTask->id.idStr, p, pTask->chkInfo.nextProcessVer); - (void) streamExecTask(pTask); + (void)streamExecTask(pTask); } else { int8_t status = streamTaskSetSchedStatusInactive(pTask); tqDebug("vgId:%d s-task:%s ignore run req since not in ready state, status:%s, sched-status:%d", vgId, @@ -900,7 +901,7 @@ int32_t tqStreamTaskProcessTaskResetReq(SStreamMeta* pMeta, char* pMsg) { SVPauseStreamTaskReq* pReq = (SVPauseStreamTaskReq*)pMsg; SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); if (pTask == NULL || (code != 0)) { tqError("vgId:%d process task-reset req, failed to acquire task:0x%x, it may have been dropped already", pMeta->vgId, pReq->taskId); @@ -924,9 +925,9 @@ int32_t tqStreamTaskProcessTaskResetReq(SStreamMeta* pMeta, char* pMsg) { streamTaskSetStatusReady(pTask); } else if (pState.state == TASK_STATUS__UNINIT) { -// tqDebug("s-task:%s start task by checking downstream tasks", pTask->id.idStr); -// ASSERT(pTask->status.downstreamReady == 0); -// tqStreamTaskRestoreCheckpoint(pMeta, pTask->id.streamId, pTask->id.taskId); + // tqDebug("s-task:%s start task by checking downstream tasks", pTask->id.idStr); + // ASSERT(pTask->status.downstreamReady == 0); + // tqStreamTaskRestoreCheckpoint(pMeta, pTask->id.streamId, pTask->id.taskId); tqDebug("s-task:%s status:%s do nothing after receiving reset-task from mnode", pTask->id.idStr, pState.name); } else { tqDebug("s-task:%s status:%s do nothing after receiving reset-task from mnode", pTask->id.idStr, pState.name); @@ -942,7 +943,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerReq(SStreamMeta* pMeta, SRpcMsg* pMsg) SRetrieveChkptTriggerReq* pReq = (SRetrieveChkptTriggerReq*)pMsg->pCont; SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->upstreamTaskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->upstreamTaskId, &pTask); if (pTask == NULL || (code != 0)) { tqError("vgId:%d process retrieve checkpoint trigger, checkpointId:%" PRId64 " from s-task:0x%x, failed to acquire task:0x%x, it may have been dropped already", @@ -958,7 +959,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerReq(SStreamMeta* pMeta, SRpcMsg* pMsg) pTask->id.idStr, (int32_t)pReq->downstreamTaskId); code = streamTaskSendCheckpointTriggerMsg(pTask, pReq->downstreamTaskId, pReq->downstreamNodeId, &pMsg->info, - TSDB_CODE_STREAM_TASK_IVLD_STATUS); + TSDB_CODE_STREAM_TASK_IVLD_STATUS); streamMetaReleaseTask(pMeta, pTask); return code; } @@ -996,7 +997,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerReq(SStreamMeta* pMeta, SRpcMsg* pMsg) pTask->id.idStr, recv, total); } code = streamTaskSendCheckpointTriggerMsg(pTask, pReq->downstreamTaskId, pReq->downstreamNodeId, &pMsg->info, - TSDB_CODE_ACTION_IN_PROGRESS); + TSDB_CODE_ACTION_IN_PROGRESS); } } else { // upstream not recv the checkpoint-source/trigger till now ASSERT(pState.state == TASK_STATUS__READY || pState.state == TASK_STATUS__HALT); @@ -1005,7 +1006,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerReq(SStreamMeta* pMeta, SRpcMsg* pMsg) "upstream sending checkpoint-source/trigger", pTask->id.idStr); code = streamTaskSendCheckpointTriggerMsg(pTask, pReq->downstreamTaskId, pReq->downstreamNodeId, &pMsg->info, - TSDB_CODE_ACTION_IN_PROGRESS); + TSDB_CODE_ACTION_IN_PROGRESS); } streamMetaReleaseTask(pMeta, pTask); @@ -1016,7 +1017,7 @@ int32_t tqStreamTaskProcessRetrieveTriggerRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) SCheckpointTriggerRsp* pRsp = POINTER_SHIFT(pMsg->pCont, sizeof(SMsgHead)); SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->taskId, &pTask); if (pTask == NULL || (code != 0)) { tqError( "vgId:%d process retrieve checkpoint-trigger, failed to acquire task:0x%x, it may have been dropped already", @@ -1038,7 +1039,7 @@ int32_t tqStreamTaskProcessTaskPauseReq(SStreamMeta* pMeta, char* pMsg) { SVPauseStreamTaskReq* pReq = (SVPauseStreamTaskReq*)pMsg; SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); if (pTask == NULL || (code != 0)) { tqError("vgId:%d process pause req, failed to acquire task:0x%x, it may have been dropped already", pMeta->vgId, pReq->taskId); @@ -1122,7 +1123,7 @@ int32_t tqStreamTaskProcessTaskResumeReq(void* handle, int64_t sversion, char* m SStreamMeta* pMeta = fromVnode ? ((STQ*)handle)->pStreamMeta : handle; SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); if (pTask == NULL || (code != 0)) { tqError("s-task:0x%x failed to acquire task to resume, it may have been dropped or stopped", pReq->taskId); return TSDB_CODE_STREAM_TASK_IVLD_STATUS; @@ -1169,13 +1170,15 @@ int32_t tqStreamProcessReqCheckpointRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) { ret int32_t tqStreamProcessChkptReportRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) { return doProcessDummyRspMsg(pMeta, pMsg); } -int32_t tqStreamProcessConsensusChkptRsp2(SStreamMeta* pMeta, SRpcMsg* pMsg) { return doProcessDummyRspMsg(pMeta, pMsg); } +int32_t tqStreamProcessConsensusChkptRsp2(SStreamMeta* pMeta, SRpcMsg* pMsg) { + return doProcessDummyRspMsg(pMeta, pMsg); +} int32_t tqStreamProcessCheckpointReadyRsp(SStreamMeta* pMeta, SRpcMsg* pMsg) { SMStreamCheckpointReadyRspMsg* pRsp = pMsg->pCont; SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->downstreamTaskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pRsp->streamId, pRsp->downstreamTaskId, &pTask); if (pTask == NULL || (code != 0)) { tqError("vgId:%d failed to acquire task:0x%x when handling checkpoint-ready msg, it may have been dropped", pRsp->downstreamNodeId, pRsp->downstreamTaskId); @@ -1211,8 +1214,9 @@ int32_t tqStreamTaskProcessConsenChkptIdReq(SStreamMeta* pMeta, SRpcMsg* pMsg) { SStreamTask* pTask = NULL; code = streamMetaAcquireTask(pMeta, req.streamId, req.taskId, &pTask); if (pTask == NULL || (code != 0)) { - tqError("vgId:%d process set consensus checkpointId req, failed to acquire task:0x%x, it may have been dropped already", - pMeta->vgId, req.taskId); + tqError( + "vgId:%d process set consensus checkpointId req, failed to acquire task:0x%x, it may have been dropped already", + pMeta->vgId, req.taskId); (void)streamMetaAddFailedTask(pMeta, req.streamId, req.taskId); return code; } @@ -1221,7 +1225,8 @@ int32_t tqStreamTaskProcessConsenChkptIdReq(SStreamMeta* pMeta, SRpcMsg* pMsg) { if (req.startTs < pTask->execInfo.created) { tqWarn("s-task:%s vgId:%d create time:%" PRId64 " recv expired consensus checkpointId:%" PRId64 " from task createTs:%" PRId64 " < task createTs:%" PRId64 ", discard", - pTask->id.idStr, pMeta->vgId, pTask->execInfo.created, req.checkpointId, req.startTs, pTask->execInfo.created); + pTask->id.idStr, pMeta->vgId, pTask->execInfo.created, req.checkpointId, req.startTs, + pTask->execInfo.created); streamMetaAddFailedTaskSelf(pTask, now); streamMetaReleaseTask(pMeta, pTask); return TSDB_CODE_SUCCESS; @@ -1234,15 +1239,15 @@ int32_t tqStreamTaskProcessConsenChkptIdReq(SStreamMeta* pMeta, SRpcMsg* pMsg) { ASSERT(pTask->chkInfo.checkpointId >= req.checkpointId); if (pTask->chkInfo.consensusTransId >= req.transId) { - tqDebug("s-task:%s vgId:%d latest consensus transId:%d, expired consensus trans:%d, discard", - pTask->id.idStr, vgId, pTask->chkInfo.consensusTransId, req.transId); + tqDebug("s-task:%s vgId:%d latest consensus transId:%d, expired consensus trans:%d, discard", pTask->id.idStr, vgId, + pTask->chkInfo.consensusTransId, req.transId); streamMutexUnlock(&pTask->lock); streamMetaReleaseTask(pMeta, pTask); return TSDB_CODE_SUCCESS; } if (pTask->chkInfo.checkpointId != req.checkpointId) { - tqDebug("s-task:%s vgId:%d update the checkpoint from %" PRId64 " to %" PRId64" transId:%d", pTask->id.idStr, vgId, + tqDebug("s-task:%s vgId:%d update the checkpoint from %" PRId64 " to %" PRId64 " transId:%d", pTask->id.idStr, vgId, pTask->chkInfo.checkpointId, req.checkpointId, req.transId); pTask->chkInfo.checkpointId = req.checkpointId; tqSetRestoreVersionInfo(pTask); diff --git a/source/dnode/vnode/src/tsdb/tsdbCache.c b/source/dnode/vnode/src/tsdb/tsdbCache.c index fb72784229..e6c52d8bed 100644 --- a/source/dnode/vnode/src/tsdb/tsdbCache.c +++ b/source/dnode/vnode/src/tsdb/tsdbCache.c @@ -534,13 +534,14 @@ int32_t tsdbCacheCommit(STsdb *pTsdb) { static int32_t reallocVarDataVal(SValue *pValue) { if (IS_VAR_DATA_TYPE(pValue->type)) { uint8_t *pVal = pValue->pData; - if (pValue->nData > 0) { - uint8_t *p = taosMemoryMalloc(pValue->nData); + uint32_t nData = pValue->nData; + if (nData > 0) { + uint8_t *p = taosMemoryMalloc(nData); if (!p) { TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); } pValue->pData = p; - memcpy(pValue->pData, pVal, pValue->nData); + (void)memcpy(pValue->pData, pVal, nData); } else { pValue->pData = NULL; } @@ -551,6 +552,54 @@ static int32_t reallocVarDataVal(SValue *pValue) { static int32_t reallocVarData(SColVal *pColVal) { return reallocVarDataVal(&pColVal->value); } +// realloc pk data and col data. +static int32_t tsdbCacheReallocSLastCol(SLastCol *pCol, size_t *pCharge) { + int32_t code = TSDB_CODE_SUCCESS, lino = 0; + size_t charge = sizeof(SLastCol); + + int8_t i = 0; + for (; i < pCol->rowKey.numOfPKs; i++) { + SValue *pValue = &pCol->rowKey.pks[i]; + if (IS_VAR_DATA_TYPE(pValue->type)) { + TAOS_CHECK_EXIT(reallocVarDataVal(pValue)); + charge += pValue->nData; + } + } + + if (IS_VAR_DATA_TYPE(pCol->colVal.value.type)) { + TAOS_CHECK_EXIT(reallocVarData(&pCol->colVal)); + charge += pCol->colVal.value.nData; + } + + if (pCharge) { + *pCharge = charge; + } + +_exit: + if (TSDB_CODE_SUCCESS != code) { + for (int8_t j = 0; j < i; j++) { + if (IS_VAR_DATA_TYPE(pCol->rowKey.pks[j].type)) { + taosMemoryFree(pCol->rowKey.pks[j].pData); + } + } + } + + TAOS_RETURN(code); +} + +void tsdbCacheFreeSLastColItem(void *pItem) { + SLastCol *pCol = (SLastCol *)pItem; + for (int i = 0; i < pCol->rowKey.numOfPKs; i++) { + if (IS_VAR_DATA_TYPE(pCol->rowKey.pks[i].type)) { + taosMemoryFree(pCol->rowKey.pks[i].pData); + } + } + + if (IS_VAR_DATA_TYPE(pCol->colVal.value.type) && pCol->colVal.value.pData) { + taosMemoryFree(pCol->colVal.value.pData); + } +} + static void tsdbCacheDeleter(const void *key, size_t klen, void *value, void *ud) { SLastCol *pLastCol = (SLastCol *)value; @@ -573,36 +622,22 @@ static void tsdbCacheDeleter(const void *key, size_t klen, void *value, void *ud } static int32_t tsdbCacheNewTableColumn(STsdb *pTsdb, int64_t uid, int16_t cid, int8_t col_type, int8_t lflag) { - int32_t code = 0; + int32_t code = 0, lino = 0; SLRUCache *pCache = pTsdb->lruCache; rocksdb_writebatch_t *wb = pTsdb->rCache.writebatch; SRowKey emptyRowKey = {.ts = TSKEY_MIN, .numOfPKs = 0}; SLastCol emptyCol = { .rowKey = emptyRowKey, .colVal = COL_VAL_NONE(cid, col_type), .dirty = 1, .cacheStatus = TSDB_LAST_CACHE_VALID}; - SLastCol *pLastCol = &emptyCol; - SLastCol *pTmpLastCol = taosMemoryCalloc(1, sizeof(SLastCol)); - if (!pTmpLastCol) { + SLastCol *pLastCol = taosMemoryCalloc(1, sizeof(SLastCol)); + if (!pLastCol) { TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); } - *pTmpLastCol = *pLastCol; - pLastCol = pTmpLastCol; - size_t charge = sizeof(*pLastCol); - - for (int8_t i = 0; i < pLastCol->rowKey.numOfPKs; i++) { - SValue *pValue = &pLastCol->rowKey.pks[i]; - if (IS_VAR_DATA_TYPE(pValue->type)) { - TAOS_CHECK_RETURN(reallocVarDataVal(pValue)); - charge += pValue->nData; - } - } - - if (IS_VAR_DATA_TYPE(pLastCol->colVal.value.type)) { - TAOS_CHECK_RETURN(reallocVarData(&pLastCol->colVal)); - charge += pLastCol->colVal.value.nData; - } + size_t charge = 0; + *pLastCol = emptyCol; + TAOS_CHECK_EXIT(tsdbCacheReallocSLastCol(pLastCol, &charge)); SLastKey *pLastKey = &(SLastKey){.lflag = lflag, .uid = uid, .cid = cid}; LRUStatus status = taosLRUCacheInsert(pCache, pLastKey, ROCKS_KEY_LEN, pLastCol, charge, tsdbCacheDeleter, NULL, @@ -611,6 +646,11 @@ static int32_t tsdbCacheNewTableColumn(STsdb *pTsdb, int64_t uid, int16_t cid, i // code = -1; } +_exit: + if (TSDB_CODE_SUCCESS != code) { + taosMemoryFree(pLastCol); + } + TAOS_RETURN(code); } @@ -1062,21 +1102,9 @@ static int32_t tsdbCachePutToLRU(STsdb *pTsdb, SLastKey *pLastKey, SLastCol *pLa TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); } + size_t charge = 0; *pLRULastCol = *pLastCol; - - size_t charge = sizeof(*pLRULastCol); - for (int8_t i = 0; i < pLRULastCol->rowKey.numOfPKs; i++) { - SValue *pValue = &pLRULastCol->rowKey.pks[i]; - if (IS_VAR_DATA_TYPE(pValue->type)) { - TAOS_CHECK_GOTO(reallocVarDataVal(pValue), &lino, _exit); - charge += pValue->nData; - } - } - - if (IS_VAR_DATA_TYPE(pLRULastCol->colVal.value.type)) { - TAOS_CHECK_GOTO(reallocVarData(&pLRULastCol->colVal), &lino, _exit); - charge += pLRULastCol->colVal.value.nData; - } + TAOS_CHECK_EXIT(tsdbCacheReallocSLastCol(pLRULastCol, &charge)); LRUStatus status = taosLRUCacheInsert(pTsdb->lruCache, pLastKey, ROCKS_KEY_LEN, pLRULastCol, charge, tsdbCacheDeleter, NULL, TAOS_LRU_PRIORITY_LOW, &pTsdb->flushState); @@ -1134,8 +1162,13 @@ static int32_t tsdbCacheUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, SArray } else { if (!remainCols) { remainCols = taosArrayInit(num_keys * 2, sizeof(SIdxKey)); + if (!remainCols) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } + } + if (!taosArrayPush(remainCols, &(SIdxKey){i, *key})) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); } - (void)taosArrayPush(remainCols, &(SIdxKey){i, *key}); } } @@ -1281,14 +1314,20 @@ int32_t tsdbCacheRowFormatUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, int6 int32_t iCol = 0; for (SColVal *pColVal = tsdbRowIterNext(&iter); pColVal && iCol < nCol; pColVal = tsdbRowIterNext(&iter), iCol++) { SLastUpdateCtx updateCtx = {.lflag = LFLAG_LAST_ROW, .tsdbRowKey = tsdbRowKey, .colVal = *pColVal}; - (void)taosArrayPush(ctxArray, &updateCtx); + if (!taosArrayPush(ctxArray, &updateCtx)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } if (!COL_VAL_IS_VALUE(pColVal)) { - (void)tSimpleHashPut(iColHash, &iCol, sizeof(iCol), NULL, 0); + if (tSimpleHashPut(iColHash, &iCol, sizeof(iCol), NULL, 0)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } continue; } updateCtx.lflag = LFLAG_LAST; - (void)taosArrayPush(ctxArray, &updateCtx); + if (!taosArrayPush(ctxArray, &updateCtx)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } } tsdbRowClose(&iter); @@ -1312,7 +1351,9 @@ int32_t tsdbCacheRowFormatUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, int6 if (COL_VAL_IS_VALUE(&colVal)) { SLastUpdateCtx updateCtx = {.lflag = LFLAG_LAST, .tsdbRowKey = tsdbRowKey, .colVal = colVal}; - (void)taosArrayPush(ctxArray, &updateCtx); + if (!taosArrayPush(ctxArray, &updateCtx)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } (void)tSimpleHashIterateRemove(iColHash, &iCol, sizeof(iCol), &pIte, &iter); } } @@ -1330,7 +1371,7 @@ _exit: } int32_t tsdbCacheColFormatUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, SBlockData *pBlockData) { - int32_t code = 0; + int32_t code = 0, lino = 0; TSDBROW lRow = tsdbRowFromBlockData(pBlockData, pBlockData->nRow - 1); @@ -1352,7 +1393,9 @@ int32_t tsdbCacheColFormatUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, SBlo .tsdbRowKey = tsdbRowKey, .colVal = COL_VAL_VALUE(PRIMARYKEY_TIMESTAMP_COL_ID, ((SValue){.type = TSDB_DATA_TYPE_TIMESTAMP, .val = lRow.pBlockData->aTSKEY[lRow.iRow]}))}; - (void)taosArrayPush(ctxArray, &updateCtx); + if (!taosArrayPush(ctxArray, &updateCtx)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } } TSDBROW tRow = tsdbRowFromBlockData(pBlockData, 0); @@ -1373,7 +1416,9 @@ int32_t tsdbCacheColFormatUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, SBlo tColDataGetValue(pColData, tRow.iRow, &colVal); SLastUpdateCtx updateCtx = {.lflag = LFLAG_LAST, .tsdbRowKey = tsdbRowKey, .colVal = colVal}; - (void)taosArrayPush(ctxArray, &updateCtx); + if (!taosArrayPush(ctxArray, &updateCtx)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } break; } } @@ -1384,7 +1429,9 @@ int32_t tsdbCacheColFormatUpdate(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, SBlo (void)tsdbRowIterOpen(&iter, &lRow, pTSchema); for (SColVal *pColVal = tsdbRowIterNext(&iter); pColVal; pColVal = tsdbRowIterNext(&iter)) { SLastUpdateCtx updateCtx = {.lflag = LFLAG_LAST_ROW, .tsdbRowKey = tsdbRowKey, .colVal = *pColVal}; - (void)taosArrayPush(ctxArray, &updateCtx); + if (!taosArrayPush(ctxArray, &updateCtx)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _exit); + } } tsdbRowClose(&iter); @@ -1413,11 +1460,13 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr SIdxKey *idxKey = taosArrayGet(remainCols, 0); if (idxKey->key.cid != PRIMARYKEY_TIMESTAMP_COL_ID) { // ignore 'ts' loaded from cache and load it from tsdb - SLastCol* pLastCol = taosArrayGet(pLastArray, 0); + SLastCol *pLastCol = taosArrayGet(pLastArray, 0); tsdbCacheUpdateLastColToNone(pLastCol, TSDB_LAST_CACHE_NO_CACHE); SLastKey *key = &(SLastKey){.lflag = ltype, .uid = uid, .cid = PRIMARYKEY_TIMESTAMP_COL_ID}; - (void)taosArrayInsert(remainCols, 0, &(SIdxKey){0, *key}); + if (!taosArrayInsert(remainCols, 0, &(SIdxKey){0, *key})) { + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } int num_keys = TARRAY_SIZE(remainCols); @@ -1453,7 +1502,9 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); } } - (void)taosArrayPush(lastTmpIndexArray, &(i)); + if (!taosArrayPush(lastTmpIndexArray, &(i))) { + TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); + } lastColIds[lastIndex] = idxKey->key.cid; lastSlotIds[lastIndex] = pr->pSlotIds[idxKey->idx]; lastIndex++; @@ -1464,7 +1515,9 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); } } - (void)taosArrayPush(lastrowTmpIndexArray, &(i)); + if (!taosArrayPush(lastrowTmpIndexArray, &(i))) { + TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); + } lastrowColIds[lastrowIndex] = idxKey->key.cid; lastrowSlotIds[lastrowIndex] = pr->pSlotIds[idxKey->idx]; lastrowIndex++; @@ -1479,16 +1532,20 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr if (lastTmpIndexArray != NULL) { TAOS_CHECK_EXIT(mergeLastCid(uid, pTsdb, &lastTmpColArray, pr, lastColIds, lastIndex, lastSlotIds)); for (int i = 0; i < taosArrayGetSize(lastTmpColArray); i++) { - (void)taosArrayInsert(pTmpColArray, *(int32_t *)taosArrayGet(lastTmpIndexArray, i), - taosArrayGet(lastTmpColArray, i)); + if (!taosArrayInsert(pTmpColArray, *(int32_t *)taosArrayGet(lastTmpIndexArray, i), + taosArrayGet(lastTmpColArray, i))) { + TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); + } } } if (lastrowTmpIndexArray != NULL) { TAOS_CHECK_EXIT(mergeLastRowCid(uid, pTsdb, &lastrowTmpColArray, pr, lastrowColIds, lastrowIndex, lastrowSlotIds)); for (int i = 0; i < taosArrayGetSize(lastrowTmpColArray); i++) { - (void)taosArrayInsert(pTmpColArray, *(int32_t *)taosArrayGet(lastrowTmpIndexArray, i), - taosArrayGet(lastrowTmpColArray, i)); + if (!taosArrayInsert(pTmpColArray, *(int32_t *)taosArrayGet(lastrowTmpIndexArray, i), + taosArrayGet(lastrowTmpColArray, i))) { + TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); + } } } @@ -1507,7 +1564,6 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr .cacheStatus = TSDB_LAST_CACHE_VALID}; if (!pLastCol) { pLastCol = &noneCol; - TAOS_CHECK_EXIT(reallocVarData(&pLastCol->colVal)); } taosArraySet(pLastArray, idxKey->idx, pLastCol); @@ -1524,20 +1580,14 @@ static int32_t tsdbCacheLoadFromRaw(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArr if (!pTmpLastCol) { TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); } + + size_t charge = 0; *pTmpLastCol = *pLastCol; pLastCol = pTmpLastCol; - - size_t charge = sizeof(*pLastCol); - for (int8_t i = 0; i < pLastCol->rowKey.numOfPKs; i++) { - SValue *pValue = &pLastCol->rowKey.pks[i]; - if (IS_VAR_DATA_TYPE(pValue->type)) { - TAOS_CHECK_EXIT(reallocVarDataVal(pValue)); - charge += pValue->nData; - } - } - if (IS_VAR_DATA_TYPE(pLastCol->colVal.value.type)) { - TAOS_CHECK_EXIT(reallocVarData(&pLastCol->colVal)); - charge += pLastCol->colVal.value.nData; + code = tsdbCacheReallocSLastCol(pLastCol, &charge); + if (TSDB_CODE_SUCCESS != code) { + taosMemoryFree(pTmpLastCol); + TAOS_CHECK_EXIT(code); } LRUStatus status = taosLRUCacheInsert(pCache, &idxKey->key, ROCKS_KEY_LEN, pLastCol, charge, tsdbCacheDeleter, NULL, @@ -1585,7 +1635,7 @@ _exit: static int32_t tsdbCacheLoadFromRocks(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SArray *remainCols, SArray *ignoreFromRocks, SCacheRowsReader *pr, int8_t ltype) { - int32_t code = 0; + int32_t code = 0, lino = 0; int num_keys = TARRAY_SIZE(remainCols); char **keys_list = taosMemoryMalloc(num_keys * sizeof(char *)); size_t *keys_list_sizes = taosMemoryMalloc(num_keys * sizeof(size_t)); @@ -1631,20 +1681,15 @@ static int32_t tsdbCacheLoadFromRocks(STsdb *pTsdb, tb_uid_t uid, SArray *pLastA code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; } + + size_t charge = 0; *pTmpLastCol = *pLastCol; pLastCol = pTmpLastCol; - - size_t charge = sizeof(*pLastCol); - for (int8_t i = 0; i < pLastCol->rowKey.numOfPKs; i++) { - SValue *pValue = &pLastCol->rowKey.pks[i]; - if (IS_VAR_DATA_TYPE(pValue->type)) { - TAOS_CHECK_RETURN(reallocVarDataVal(pValue)); - charge += pValue->nData; - } - } - if (IS_VAR_DATA_TYPE(pLastCol->colVal.value.type)) { - TAOS_CHECK_RETURN(reallocVarData(&pLastCol->colVal)); - charge += pLastCol->colVal.value.nData; + code = tsdbCacheReallocSLastCol(pLastCol, &charge); + if (TSDB_CODE_SUCCESS != code) { + taosMemoryFree(pTmpLastCol); + taosMemoryFreeClear(PToFree); + goto _exit; } LRUStatus status = taosLRUCacheInsert(pCache, &idxKey->key, ROCKS_KEY_LEN, pLastCol, charge, tsdbCacheDeleter, @@ -1654,10 +1699,8 @@ static int32_t tsdbCacheLoadFromRocks(STsdb *pTsdb, tb_uid_t uid, SArray *pLastA } SLastCol lastCol = *pLastCol; - for (int8_t i = 0; i < lastCol.rowKey.numOfPKs; i++) { - TAOS_CHECK_RETURN(reallocVarDataVal(&lastCol.rowKey.pks[i])); - } - TAOS_CHECK_RETURN(reallocVarData(&lastCol.colVal)); + TAOS_CHECK_EXIT(tsdbCacheReallocSLastCol(&lastCol, NULL)); + taosArraySet(pLastArray, idxKey->idx, &lastCol); taosArrayRemove(remainCols, j); taosArrayRemove(ignoreFromRocks, j); @@ -1715,10 +1758,8 @@ int32_t tsdbCacheGetBatch(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SCache SLastCol *pLastCol = h ? (SLastCol *)taosLRUCacheValue(pCache, h) : NULL; if (h && pLastCol->cacheStatus != TSDB_LAST_CACHE_NO_CACHE) { SLastCol lastCol = *pLastCol; - for (int8_t j = 0; j < lastCol.rowKey.numOfPKs; j++) { - TAOS_CHECK_RETURN(reallocVarDataVal(&lastCol.rowKey.pks[j])); - } - TAOS_CHECK_RETURN(reallocVarData(&lastCol.colVal)); + TAOS_CHECK_GOTO(tsdbCacheReallocSLastCol(&lastCol, NULL), NULL, _exit); + if (taosArrayPush(pLastArray, &lastCol) == NULL) { code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; @@ -1745,12 +1786,12 @@ int32_t tsdbCacheGetBatch(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SCache goto _exit; } } - if (taosArrayPush(remainCols, &(SIdxKey){i, key}) ==NULL) { + if (taosArrayPush(remainCols, &(SIdxKey){i, key}) == NULL) { code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; } bool ignoreRocks = pLastCol ? (pLastCol->cacheStatus == TSDB_LAST_CACHE_NO_CACHE) : false; - if (taosArrayPush(ignoreFromRocks, &ignoreRocks) ==NULL) { + if (taosArrayPush(ignoreFromRocks, &ignoreRocks) == NULL) { code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; } @@ -1769,18 +1810,12 @@ int32_t tsdbCacheGetBatch(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SCache SLastCol *pLastCol = h ? (SLastCol *)taosLRUCacheValue(pCache, h) : NULL; if (h && pLastCol->cacheStatus != TSDB_LAST_CACHE_NO_CACHE) { SLastCol lastCol = *pLastCol; - for (int8_t j = 0; j < lastCol.rowKey.numOfPKs; j++) { - code = reallocVarDataVal(&lastCol.rowKey.pks[j]); - if (code) { - (void)taosThreadMutexUnlock(&pTsdb->lruMutex); - TAOS_RETURN(code); - } - } - code = reallocVarData(&lastCol.colVal); + code = tsdbCacheReallocSLastCol(&lastCol, NULL); if (code) { (void)taosThreadMutexUnlock(&pTsdb->lruMutex); TAOS_RETURN(code); } + taosArraySet(pLastArray, idxKey->idx, &lastCol); taosArrayRemove(remainCols, i); @@ -1802,18 +1837,18 @@ int32_t tsdbCacheGetBatch(STsdb *pTsdb, tb_uid_t uid, SArray *pLastArray, SCache } _exit: - if (remainCols) { - taosArrayDestroy(remainCols); - } - if (ignoreFromRocks) { - taosArrayDestroy(ignoreFromRocks); - } + if (remainCols) { + taosArrayDestroy(remainCols); + } + if (ignoreFromRocks) { + taosArrayDestroy(ignoreFromRocks); + } TAOS_RETURN(code); } int32_t tsdbCacheDel(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKEY eKey) { - int32_t code = 0; + int32_t code = 0, lino = 0; // fetch schema STSchema *pTSchema = NULL; int sver = -1; @@ -1844,7 +1879,9 @@ int32_t tsdbCacheDel(STsdb *pTsdb, tb_uid_t suid, tb_uid_t uid, TSKEY sKey, TSKE if (!remainCols) { remainCols = taosArrayInit(numCols * 2, sizeof(SLastKey)); } - (void)taosArrayPush(remainCols, &lastKey); + if (!taosArrayPush(remainCols, &lastKey)) { + TAOS_CHECK_EXIT(TSDB_CODE_OUT_OF_MEMORY); + } } } } @@ -2028,10 +2065,12 @@ static int32_t getTableDelDataFromTbData(STbData *pTbData, SArray *aDelData) { SDelData *pDelData = pTbData ? pTbData->pHead : NULL; for (; pDelData; pDelData = pDelData->pNext) { - (void)taosArrayPush(aDelData, pDelData); + if (!taosArrayPush(aDelData, pDelData)) { + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } - return code; + TAOS_RETURN(code); } static void freeTableInfoFunc(void *param) { @@ -2042,6 +2081,9 @@ static void freeTableInfoFunc(void *param) { static STableLoadInfo *getTableLoadInfo(SCacheRowsReader *pReader, uint64_t uid) { if (!pReader->pTableMap) { pReader->pTableMap = tSimpleHashInit(pReader->numOfTables, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT)); + if (!pReader->pTableMap) { + return NULL; + } tSimpleHashSetFreeFp(pReader->pTableMap, freeTableInfoFunc); } @@ -2051,7 +2093,9 @@ static STableLoadInfo *getTableLoadInfo(SCacheRowsReader *pReader, uint64_t uid) if (!ppInfo) { pInfo = taosMemoryCalloc(1, sizeof(STableLoadInfo)); if (pInfo) { - (void)tSimpleHashPut(pReader->pTableMap, &uid, sizeof(uint64_t), &pInfo, POINTER_BYTES); + if (tSimpleHashPut(pReader->pTableMap, &uid, sizeof(uint64_t), &pInfo, POINTER_BYTES)) { + return NULL; + } } return pInfo; @@ -2163,7 +2207,9 @@ static int32_t loadTombFromBlk(const TTombBlkArray *pTombBlkArray, SCacheRowsRea TD_VID(pReader->pTsdb->pVnode), pReader->pCurFileSet->fid, record.skey, record.ekey, uid);*/ SDelData delData = {.version = record.version, .sKey = record.skey, .eKey = record.ekey}; - (void)taosArrayPush(pInfo->pTombData, &delData); + if (!taosArrayPush(pInfo->pTombData, &delData)) { + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } } @@ -2368,6 +2414,9 @@ static int32_t getNextRowFromFS(void *iter, TSDBROW **ppRow, bool *pIgnoreEarlie if (!state->pIndexList) { state->pIndexList = taosArrayInit(1, sizeof(SBrinBlk)); + if (!state->pIndexList) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err); + } } else { taosArrayClear(state->pIndexList); } @@ -2378,7 +2427,9 @@ static int32_t getNextRowFromFS(void *iter, TSDBROW **ppRow, bool *pIgnoreEarlie SBrinBlk *pBrinBlk = &pBlkArray->data[i]; if (state->suid >= pBrinBlk->minTbid.suid && state->suid <= pBrinBlk->maxTbid.suid) { if (state->uid >= pBrinBlk->minTbid.uid && state->uid <= pBrinBlk->maxTbid.uid) { - (void)taosArrayPush(state->pIndexList, pBrinBlk); + if (!taosArrayPush(state->pIndexList, pBrinBlk)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err); + } } } else if (state->suid > pBrinBlk->maxTbid.suid || (state->suid == pBrinBlk->maxTbid.suid && state->uid > pBrinBlk->maxTbid.uid)) { @@ -2653,7 +2704,6 @@ static int32_t getNextRowFromMem(void *iter, TSDBROW **ppRow, bool *pIgnoreEarli TAOS_RETURN(code); } default: - ASSERT(0); break; } @@ -2949,7 +2999,9 @@ static int32_t nextRowIterGet(CacheNextRowIter *pIter, TSDBROW **ppRow, bool *pI TSDB_CHECK_NULL(pInfo->pTombData, code, lino, _err, TSDB_CODE_OUT_OF_MEMORY); } - (void)taosArrayAddAll(pInfo->pTombData, pIter->pMemDelData); + if (!taosArrayAddAll(pInfo->pTombData, pIter->pMemDelData)) { + TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err); + } size_t delSize = TARRAY_SIZE(pInfo->pTombData); if (delSize > 0) { @@ -2996,7 +3048,9 @@ static int32_t initLastColArrayPartial(STSchema *pTSchema, SArray **ppColArray, int16_t slotId = slotIds[i]; SLastCol col = {.rowKey.ts = 0, .colVal = COL_VAL_NULL(pTSchema->columns[slotId].colId, pTSchema->columns[slotId].type)}; - (void)taosArrayPush(pColArray, &col); + if (!taosArrayPush(pColArray, &col)) { + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } *ppColArray = pColArray; @@ -3050,7 +3104,11 @@ static int32_t mergeLastCid(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray, SC } for (int i = 0; i < nCols; ++i) { - (void)taosArrayPush(aColArray, &aCols[i]); + if (!taosArrayPush(aColArray, &aCols[i])) { + taosArrayDestroy(pColArray); + + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } STsdbRowKey lastRowKey = {.key.ts = TSKEY_MAX}; @@ -3102,26 +3160,18 @@ static int32_t mergeLastCid(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray, SC } if (slotIds[iCol] == 0) { STColumn *pTColumn = &pTSchema->columns[0]; - *pColVal = COL_VAL_VALUE(pTColumn->colId, ((SValue){.type = pTColumn->type, .val = rowKey.key.ts})); - taosArraySet(pColArray, 0, - &(SLastCol){.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}); + + SLastCol colTmp = {.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}; + TAOS_CHECK_GOTO(tsdbCacheReallocSLastCol(&colTmp, NULL), &lino, _err); + + taosArraySet(pColArray, 0, &colTmp); continue; } tsdbRowGetColVal(pRow, pTSchema, slotIds[iCol], pColVal); *pCol = (SLastCol){.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}; - if (IS_VAR_DATA_TYPE(pColVal->value.type) /*&& pColVal->value.nData > 0*/) { - if (pColVal->value.nData > 0) { - pCol->colVal.value.pData = taosMemoryMalloc(pCol->colVal.value.nData); - if (pCol->colVal.value.pData == NULL) { - TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err); - } - memcpy(pCol->colVal.value.pData, pColVal->value.pData, pColVal->value.nData); - } else { - pCol->colVal.value.pData = NULL; - } - } + TAOS_CHECK_GOTO(tsdbCacheReallocSLastCol(pCol, NULL), &lino, _err); if (!COL_VAL_IS_VALUE(pColVal)) { if (!setNoneCol) { @@ -3160,21 +3210,9 @@ static int32_t mergeLastCid(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray, SC tsdbRowGetColVal(pRow, pTSchema, slotIds[iCol], pColVal); if (COL_VAL_IS_VALUE(pColVal)) { SLastCol lastCol = {.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}; - if (IS_VAR_DATA_TYPE(pColVal->value.type) /* && pColVal->value.nData > 0 */) { - SLastCol *pLastCol = (SLastCol *)taosArrayGet(pColArray, iCol); - taosMemoryFree(pLastCol->colVal.value.pData); - - if (pColVal->value.nData > 0) { - lastCol.colVal.value.pData = taosMemoryMalloc(lastCol.colVal.value.nData); - if (lastCol.colVal.value.pData == NULL) { - TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err); - } - memcpy(lastCol.colVal.value.pData, pColVal->value.pData, pColVal->value.nData); - } else { - lastCol.colVal.value.pData = NULL; - } - } + TAOS_CHECK_GOTO(tsdbCacheReallocSLastCol(&lastCol, NULL), &lino, _err); + tsdbCacheFreeSLastColItem(lastColVal); taosArraySet(pColArray, iCol, &lastCol); int32_t aColIndex = taosArraySearchIdx(aColArray, &lastCol.colVal.cid, compareInt16Val, TD_EQ); if (aColIndex >= 0) { @@ -3206,7 +3244,7 @@ _err: nextRowIterClose(&iter); // taosMemoryFreeClear(pTSchema); *ppLastArray = NULL; - taosArrayDestroy(pColArray); + taosArrayDestroyEx(pColArray, tsdbCacheFreeSLastColItem); taosArrayDestroy(aColArray); if (code) { @@ -3239,7 +3277,11 @@ static int32_t mergeLastRowCid(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray, } for (int i = 0; i < nCols; ++i) { - (void)taosArrayPush(aColArray, &aCols[i]); + if (!taosArrayPush(aColArray, &aCols[i])) { + taosArrayDestroy(pColArray); + + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } // inverse iterator @@ -3282,28 +3324,18 @@ static int32_t mergeLastRowCid(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray, } if (slotIds[iCol] == 0) { STColumn *pTColumn = &pTSchema->columns[0]; - *pColVal = COL_VAL_VALUE(pTColumn->colId, ((SValue){.type = pTColumn->type, .val = rowKey.key.ts})); - taosArraySet(pColArray, 0, - &(SLastCol){.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}); + + SLastCol colTmp = {.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}; + TAOS_CHECK_GOTO(tsdbCacheReallocSLastCol(&colTmp, NULL), &lino, _err); + + taosArraySet(pColArray, 0, &colTmp); continue; } tsdbRowGetColVal(pRow, pTSchema, slotIds[iCol], pColVal); *pCol = (SLastCol){.rowKey = rowKey.key, .colVal = *pColVal, .cacheStatus = TSDB_LAST_CACHE_VALID}; - if (IS_VAR_DATA_TYPE(pColVal->value.type) /*&& pColVal->value.nData > 0*/) { - if (pColVal->value.nData > 0) { - pCol->colVal.value.pData = taosMemoryMalloc(pCol->colVal.value.nData); - if (pCol->colVal.value.pData == NULL) { - TAOS_CHECK_GOTO(TSDB_CODE_OUT_OF_MEMORY, &lino, _err); - } - if (pColVal->value.nData > 0) { - memcpy(pCol->colVal.value.pData, pColVal->value.pData, pColVal->value.nData); - } - } else { - pCol->colVal.value.pData = NULL; - } - } + TAOS_CHECK_GOTO(tsdbCacheReallocSLastCol(pCol, NULL), &lino, _err); int32_t aColIndex = taosArraySearchIdx(aColArray, &pColVal->cid, compareInt16Val, TD_EQ); if (aColIndex >= 0) { @@ -3333,7 +3365,7 @@ _err: nextRowIterClose(&iter); *ppLastArray = NULL; - taosArrayDestroy(pColArray); + taosArrayDestroyEx(pColArray, tsdbCacheFreeSLastColItem); taosArrayDestroy(aColArray); if (code) { diff --git a/source/dnode/vnode/src/tsdb/tsdbCacheRead.c b/source/dnode/vnode/src/tsdb/tsdbCacheRead.c index af5b3523e8..842b2cfa90 100644 --- a/source/dnode/vnode/src/tsdb/tsdbCacheRead.c +++ b/source/dnode/vnode/src/tsdb/tsdbCacheRead.c @@ -404,26 +404,6 @@ void tsdbCacherowsReaderClose(void* pReader) { taosMemoryFree(pReader); } -static void freeItemOfRow(void* pItem) { - SLastCol* pCol = (SLastCol*)pItem; - if (IS_VAR_DATA_TYPE(pCol->colVal.value.type) && pCol->colVal.value.pData) { - taosMemoryFree(pCol->colVal.value.pData); - } -} - -static void freeItemWithPk(void* pItem) { - SLastCol* pCol = (SLastCol*)pItem; - for (int i = 0; i < pCol->rowKey.numOfPKs; i++) { - if (IS_VAR_DATA_TYPE(pCol->rowKey.pks[i].type)) { - taosMemoryFree(pCol->rowKey.pks[i].pData); - } - } - - if (IS_VAR_DATA_TYPE(pCol->colVal.value.type) && pCol->colVal.value.pData) { - taosMemoryFree(pCol->colVal.value.pData); - } -} - static int32_t tsdbCacheQueryReseek(void* pQHandle) { int32_t code = 0; SCacheRowsReader* pReader = pQHandle; @@ -559,7 +539,7 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 } if (TARRAY_SIZE(pRow) <= 0 || COL_VAL_IS_NONE(&((SLastCol*)TARRAY_DATA(pRow))[0].colVal)) { - taosArrayClearEx(pRow, freeItemOfRow); + taosArrayClearEx(pRow, tsdbCacheFreeSLastColItem); continue; } @@ -642,7 +622,7 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 } } - taosArrayClearEx(pRow, freeItemOfRow); + taosArrayClearEx(pRow, tsdbCacheFreeSLastColItem); } if (hasRes) { @@ -652,7 +632,7 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 } } - taosArrayDestroyEx(pLastCols, freeItemWithPk); + taosArrayDestroyEx(pLastCols, tsdbCacheFreeSLastColItem); } else if (HASTYPE(pr->type, CACHESCAN_RETRIEVE_TYPE_ALL)) { for (int32_t i = pr->tableIndex; i < pr->numOfTables; ++i) { tb_uid_t uid = pTableList[i].uid; @@ -666,7 +646,7 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 } if (TARRAY_SIZE(pRow) <= 0 || COL_VAL_IS_NONE(&((SLastCol*)TARRAY_DATA(pRow))[0].colVal)) { - taosArrayClearEx(pRow, freeItemOfRow); + taosArrayClearEx(pRow, tsdbCacheFreeSLastColItem); continue; } @@ -675,7 +655,7 @@ int32_t tsdbRetrieveCacheRows(void* pReader, SSDataBlock* pResBlock, const int32 goto _end; } - taosArrayClearEx(pRow, freeItemOfRow); + taosArrayClearEx(pRow, tsdbCacheFreeSLastColItem); void* px = taosArrayPush(pTableUidList, &uid); if (px == NULL) { diff --git a/source/dnode/vnode/src/tsdb/tsdbCommit2.c b/source/dnode/vnode/src/tsdb/tsdbCommit2.c index 3c407b31cf..df572ff852 100644 --- a/source/dnode/vnode/src/tsdb/tsdbCommit2.c +++ b/source/dnode/vnode/src/tsdb/tsdbCommit2.c @@ -208,8 +208,6 @@ static int32_t tsdbCommitOpenReader(SCommitter2 *committer) { int32_t code = 0; int32_t lino = 0; - ASSERT(TARRAY2_SIZE(committer->sttReaderArray) == 0); - if (committer->ctx->info->fset == NULL // || committer->sttTrigger > 1 // || TARRAY2_SIZE(committer->ctx->info->fset->lvlArr) == 0 // @@ -264,11 +262,6 @@ static int32_t tsdbCommitOpenIter(SCommitter2 *committer) { int32_t code = 0; int32_t lino = 0; - ASSERT(TARRAY2_SIZE(committer->dataIterArray) == 0); - ASSERT(committer->dataIterMerger == NULL); - ASSERT(TARRAY2_SIZE(committer->tombIterArray) == 0); - ASSERT(committer->tombIterMerger == NULL); - STsdbIter *iter; STsdbIterConfig config = {0}; @@ -342,10 +335,6 @@ static int32_t tsdbCommitFileSetBegin(SCommitter2 *committer) { committer->ctx->tbid->suid = 0; committer->ctx->tbid->uid = 0; - ASSERT(TARRAY2_SIZE(committer->dataIterArray) == 0); - ASSERT(committer->dataIterMerger == NULL); - ASSERT(committer->writer == NULL); - TAOS_CHECK_GOTO(tsdbCommitOpenReader(committer), &lino, _exit); TAOS_CHECK_GOTO(tsdbCommitOpenIter(committer), &lino, _exit); TAOS_CHECK_GOTO(tsdbCommitOpenWriter(committer), &lino, _exit); @@ -637,13 +626,10 @@ static int32_t tsdbCloseCommitter(SCommitter2 *committer, int32_t eno) { if (eno == 0) { TAOS_CHECK_GOTO(tsdbFSEditBegin(committer->tsdb->pFS, committer->fopArray, TSDB_FEDIT_COMMIT), &lino, _exit); } else { - // TODO - ASSERT(0); + tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(committer->tsdb->pVnode), __func__, __FILE__, lino, + tstrerror(eno)); } - ASSERT(committer->writer == NULL); - ASSERT(committer->dataIterMerger == NULL); - ASSERT(committer->tombIterMerger == NULL); TARRAY2_DESTROY(committer->dataIterArray, NULL); TARRAY2_DESTROY(committer->tombIterArray, NULL); TARRAY2_DESTROY(committer->sttReaderArray, NULL); diff --git a/source/dnode/vnode/src/tsdb/tsdbDataFileRAW.c b/source/dnode/vnode/src/tsdb/tsdbDataFileRAW.c index 6469160536..886a6f31a2 100644 --- a/source/dnode/vnode/src/tsdb/tsdbDataFileRAW.c +++ b/source/dnode/vnode/src/tsdb/tsdbDataFileRAW.c @@ -113,7 +113,7 @@ _exit: } static int32_t tsdbDataFileRAWWriterCloseAbort(SDataFileRAWWriter *writer) { - ASSERT(0); + tsdbError("vgId:%d %s failed since not implemented", TD_VID(writer->config->tsdb->pVnode), __func__); return 0; } @@ -122,8 +122,6 @@ static int32_t tsdbDataFileRAWWriterDoClose(SDataFileRAWWriter *writer) { return static int32_t tsdbDataFileRAWWriterCloseCommit(SDataFileRAWWriter *writer, TFileOpArray *opArr) { int32_t code = 0; int32_t lino = 0; - ASSERT(writer->ctx->offset <= writer->file.size); - ASSERT(writer->config->fid == writer->file.fid); STFileOp op = (STFileOp){ .optype = TSDB_FOP_CREATE, diff --git a/source/dnode/vnode/src/tsdb/tsdbDataFileRW.c b/source/dnode/vnode/src/tsdb/tsdbDataFileRW.c index 7e7ea59a5b..f80c898f68 100644 --- a/source/dnode/vnode/src/tsdb/tsdbDataFileRW.c +++ b/source/dnode/vnode/src/tsdb/tsdbDataFileRW.c @@ -283,7 +283,9 @@ int32_t tsdbDataFileReadBrinBlock(SDataFileReader *reader, const SBrinBlk *brinB } } - ASSERT(br.offset == br.buffer->size); + if (br.offset != br.buffer->size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { @@ -313,7 +315,10 @@ int32_t tsdbDataFileReadBlockData(SDataFileReader *reader, const SBrinRecord *re // decompress SBufferReader br = BUFFER_READER_INITIALIZER(0, buffer); TAOS_CHECK_GOTO(tBlockDataDecompress(&br, bData, assist), &lino, _exit); - ASSERT(br.offset == buffer->size); + + if (br.offset != buffer->size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { @@ -345,7 +350,9 @@ int32_t tsdbDataFileReadBlockDataByColumn(SDataFileReader *reader, const SBrinRe SBufferReader br = BUFFER_READER_INITIALIZER(0, buffer0); TAOS_CHECK_GOTO(tGetDiskDataHdr(&br, &hdr), &lino, _exit); - ASSERT(hdr.delimiter == TSDB_FILE_DLMT); + if (hdr.delimiter != TSDB_FILE_DLMT) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } tBlockDataReset(bData); bData->suid = hdr.suid; @@ -354,7 +361,9 @@ int32_t tsdbDataFileReadBlockDataByColumn(SDataFileReader *reader, const SBrinRe // Key part TAOS_CHECK_GOTO(tBlockDataDecompressKeyPart(&hdr, &br, bData, assist), &lino, _exit); - ASSERT(br.offset == buffer0->size); + if (br.offset != buffer0->size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } int extraColIdx = -1; for (int i = 0; i < ncid; i++) { @@ -526,7 +535,9 @@ int32_t tsdbDataFileReadBlockSma(SDataFileReader *reader, const SBrinRecord *rec TAOS_CHECK_GOTO(tGetColumnDataAgg(&br, sma), &lino, _exit); TAOS_CHECK_GOTO(TARRAY2_APPEND_PTR(columnDataAggArray, sma), &lino, _exit); } - ASSERT(br.offset == record->smaSize); + if (br.offset != record->smaSize) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } } _exit: @@ -661,7 +672,8 @@ struct SDataFileWriter { }; static int32_t tsdbDataFileWriterCloseAbort(SDataFileWriter *writer) { - ASSERT(0); + tsdbError("vgId:%d %s failed at %s:%d since %s", TD_VID(writer->config->tsdb->pVnode), __func__, __FILE__, __LINE__, + "not implemented"); return 0; } @@ -980,7 +992,9 @@ static int32_t tsdbDataFileDoWriteBlockData(SDataFileWriter *writer, SBlockData return 0; } - ASSERT(bData->uid); + if (!bData->uid) { + return TSDB_CODE_INVALID_PARA; + } int32_t code = 0; int32_t lino = 0; @@ -1022,7 +1036,9 @@ static int32_t tsdbDataFileDoWriteBlockData(SDataFileWriter *writer, SBlockData code = metaGetColCmpr(writer->config->tsdb->pVnode->pMeta, bData->suid != 0 ? bData->suid : bData->uid, &cmprInfo.pColCmpr); - ASSERT(code == TSDB_CODE_SUCCESS); + if (code) { + tsdbWarn("vgId:%d failed to get column compress algrithm", TD_VID(writer->config->tsdb->pVnode)); + } TAOS_CHECK_GOTO(tBlockDataCompress(bData, &cmprInfo, buffers, assist), &lino, _exit); @@ -1234,7 +1250,6 @@ static int32_t tsdbDataFileWriteTableDataEnd(SDataFileWriter *writer) { if (writer->ctx->tbHasOldData) { TAOS_CHECK_GOTO(tsdbDataFileDoWriteTableOldData(writer, NULL /* as the largest key */), &lino, _exit); - ASSERT(writer->ctx->tbHasOldData == false); } TAOS_CHECK_GOTO(tsdbDataFileDoWriteBlockData(writer, writer->blockData), &lino, _exit); @@ -1251,9 +1266,6 @@ static int32_t tsdbDataFileWriteTableDataBegin(SDataFileWriter *writer, const TA int32_t code = 0; int32_t lino = 0; - ASSERT(writer->ctx->blockDataIdx == writer->ctx->blockData->nRow); - ASSERT(writer->blockData->nRow == 0); - SMetaInfo info; bool drop = false; TABLEID tbid1[1]; @@ -1451,7 +1463,9 @@ int32_t tsdbFileWriteTombBlk(STsdbFD *fd, const TTombBlkArray *tombBlkArray, SFD } static int32_t tsdbDataFileDoWriteTombBlk(SDataFileWriter *writer) { - ASSERT(TARRAY2_SIZE(writer->tombBlkArray) > 0); + if (TARRAY2_SIZE(writer->tombBlkArray) <= 0) { + return TSDB_CODE_INVALID_PARA; + } int32_t code = 0; int32_t lino = 0; @@ -1523,7 +1537,9 @@ static int32_t tsdbDataFileDoWriteTombRecord(SDataFileWriter *writer, const STom TAOS_CHECK_GOTO(tsdbDataFileDoWriteTombBlock(writer), &lino, _exit); } } else { - ASSERT(0); + tsdbError("vgId:%d duplicate tomb record, cid:%" PRId64 ", suid:%" PRId64 ", uid:%" PRId64 ", version:%" PRId64, + TD_VID(writer->config->tsdb->pVnode), writer->config->cid, record->suid, record->uid, + record->version); } } @@ -1566,7 +1582,9 @@ _exit: int32_t tsdbFileWriteBrinBlk(STsdbFD *fd, TBrinBlkArray *brinBlkArray, SFDataPtr *ptr, int64_t *fileSize, int32_t encryptAlgorithm, char *encryptKey) { - ASSERT(TARRAY2_SIZE(brinBlkArray) > 0); + if (TARRAY2_SIZE(brinBlkArray) <= 0) { + return TSDB_CODE_INVALID_PARA; + } ptr->offset = *fileSize; ptr->size = TARRAY2_DATA_LEN(brinBlkArray); @@ -1852,7 +1870,9 @@ int32_t tsdbDataFileWriteBlockData(SDataFileWriter *writer, SBlockData *bData) { int32_t code = 0; int32_t lino = 0; - ASSERT(bData->uid); + if (!bData->uid) { + return TSDB_CODE_INVALID_PARA; + } if (!writer->ctx->opened) { TAOS_CHECK_GOTO(tsdbDataFileWriterDoOpen(writer), &lino, _exit); @@ -1895,7 +1915,9 @@ _exit: } int32_t tsdbDataFileFlush(SDataFileWriter *writer) { - ASSERT(writer->ctx->opened); + if (!writer->ctx->opened) { + return TSDB_CODE_INVALID_PARA; + } if (writer->blockData->nRow == 0) return 0; if (writer->ctx->tbHasOldData) return 0; @@ -1910,8 +1932,6 @@ static int32_t tsdbDataFileWriterOpenTombFD(SDataFileWriter *writer) { char fname[TSDB_FILENAME_LEN]; int32_t ftype = TSDB_FTYPE_TOMB; - ASSERT(writer->files[ftype].size == 0); - int32_t flag = (TD_FILE_READ | TD_FILE_WRITE | TD_FILE_CREATE | TD_FILE_TRUNC); int32_t lcn = writer->files[ftype].lcn; diff --git a/source/dnode/vnode/src/tsdb/tsdbFS.c b/source/dnode/vnode/src/tsdb/tsdbFS.c index cc89b106da..a516114c29 100644 --- a/source/dnode/vnode/src/tsdb/tsdbFS.c +++ b/source/dnode/vnode/src/tsdb/tsdbFS.c @@ -83,7 +83,10 @@ static int32_t tsdbBinaryToFS(uint8_t *pData, int64_t nData, STsdbFS *pFS) { } } - ASSERT(n + sizeof(TSCKSUM) == nData); + if (n + sizeof(TSCKSUM) != nData) { + code = TSDB_CODE_FILE_CORRUPTED; + goto _exit; + } _exit: return code; @@ -450,8 +453,9 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe taosMemoryFree(pHeadF); } } else { - ASSERT(pHeadF->offset == pSetNew->pHeadF->offset); - ASSERT(pHeadF->size == pSetNew->pHeadF->size); + if (pHeadF->offset != pSetNew->pHeadF->offset || pHeadF->size != pSetNew->pHeadF->size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } } // data @@ -499,7 +503,6 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe // stt if (sameDisk) { if (pSetNew->nSttF > pSetOld->nSttF) { - ASSERT(pSetNew->nSttF == pSetOld->nSttF + 1); pSetOld->aSttF[pSetOld->nSttF] = (SSttFile *)taosMemoryMalloc(sizeof(SSttFile)); if (pSetOld->aSttF[pSetOld->nSttF] == NULL) { code = TSDB_CODE_OUT_OF_MEMORY; @@ -509,7 +512,6 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe pSetOld->aSttF[pSetOld->nSttF]->nRef = 1; pSetOld->nSttF++; } else if (pSetNew->nSttF < pSetOld->nSttF) { - ASSERT(pSetNew->nSttF == 1); for (int32_t iStt = 0; iStt < pSetOld->nSttF; iStt++) { SSttFile *pSttFile = pSetOld->aSttF[iStt]; nRef = atomic_sub_fetch_32(&pSttFile->nRef, 1); @@ -548,8 +550,10 @@ static int32_t tsdbMergeFileSet(STsdb *pTsdb, SDFileSet *pSetOld, SDFileSet *pSe *pSetOld->aSttF[iStt] = *pSetNew->aSttF[iStt]; pSetOld->aSttF[iStt]->nRef = 1; } else { - ASSERT(pSetOld->aSttF[iStt]->size == pSetOld->aSttF[iStt]->size); - ASSERT(pSetOld->aSttF[iStt]->offset == pSetOld->aSttF[iStt]->offset); + if (pSetOld->aSttF[iStt]->size != pSetOld->aSttF[iStt]->size || + pSetOld->aSttF[iStt]->offset == pSetOld->aSttF[iStt]->offset) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } } } } @@ -777,8 +781,6 @@ int32_t tsdbFSOpen(STsdb *pTsdb, int8_t rollback) { // empty one code = tsdbSaveFSToFile(&pTsdb->fs, current); TSDB_CHECK_CODE(code, lino, _exit); - - ASSERT(!rollback); } // scan and fix FS @@ -796,7 +798,6 @@ int32_t tsdbFSClose(STsdb *pTsdb) { int32_t code = 0; if (pTsdb->fs.pDelFile) { - ASSERT(pTsdb->fs.pDelFile->nRef == 1); taosMemoryFree(pTsdb->fs.pDelFile); } @@ -804,20 +805,16 @@ int32_t tsdbFSClose(STsdb *pTsdb) { SDFileSet *pSet = (SDFileSet *)taosArrayGet(pTsdb->fs.aDFileSet, iSet); // head - ASSERT(pSet->pHeadF->nRef == 1); taosMemoryFree(pSet->pHeadF); // data - ASSERT(pSet->pDataF->nRef == 1); taosMemoryFree(pSet->pDataF); // sma - ASSERT(pSet->pSmaF->nRef == 1); taosMemoryFree(pSet->pSmaF); // stt for (int32_t iStt = 0; iStt < pSet->nSttF; iStt++) { - ASSERT(pSet->aSttF[iStt]->nRef == 1); taosMemoryFree(pSet->aSttF[iStt]); } } diff --git a/source/dnode/vnode/src/tsdb/tsdbFS2.c b/source/dnode/vnode/src/tsdb/tsdbFS2.c index c3b612b227..6dec456be7 100644 --- a/source/dnode/vnode/src/tsdb/tsdbFS2.c +++ b/source/dnode/vnode/src/tsdb/tsdbFS2.c @@ -208,7 +208,9 @@ static int32_t load_fs(STsdb *pTsdb, const char *fname, TFileSetArray *arr) { /* fmtv */ item1 = cJSON_GetObjectItem(json, "fmtv"); if (cJSON_IsNumber(item1)) { - ASSERT(item1->valuedouble == 1); + if (item1->valuedouble != 1) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } } else { TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); } @@ -892,7 +894,6 @@ int32_t tsdbFSEditCommit(STFileSystem *fs) { // commit code = commit_edit(fs); - ASSERT(code == 0); TSDB_CHECK_CODE(code, lino, _exit); // schedule merge @@ -1050,7 +1051,6 @@ static SHashObj *tsdbFSetRangeArrayToHash(TFileSetRangeArray *pRanges) { STFileSetRange *u = TARRAY2_GET(pRanges, i); int32_t fid = u->fid; int32_t code = taosHashPut(pHash, &fid, sizeof(fid), u, sizeof(*u)); - ASSERT(code == 0); tsdbDebug("range diff hash fid:%d, sver:%" PRId64 ", ever:%" PRId64, u->fid, u->sver, u->ever); } return pHash; @@ -1190,10 +1190,8 @@ int32_t tsdbBeginTaskOnFileSet(STsdb *tsdb, int32_t fid, STFileSet **fset) { (void)taosThreadCondWait(&(*fset)->beginTask, &tsdb->mutex); (void)tsdbFSGetFSet(tsdb->pFS, fid, fset); - ASSERT(fset != NULL); (*fset)->numWaitTask--; - ASSERT((*fset)->numWaitTask >= 0); } else { (*fset)->taskRunning = true; break; diff --git a/source/dnode/vnode/src/tsdb/tsdbFSet2.c b/source/dnode/vnode/src/tsdb/tsdbFSet2.c index 78e1a6a13d..05ce1b23f5 100644 --- a/source/dnode/vnode/src/tsdb/tsdbFSet2.c +++ b/source/dnode/vnode/src/tsdb/tsdbFSet2.c @@ -106,7 +106,9 @@ static void tsdbSttLvlRemove(SSttLvl **lvl) { static int32_t tsdbSttLvlApplyEdit(STsdb *pTsdb, const SSttLvl *lvl1, SSttLvl *lvl2) { int32_t code = 0; - ASSERT(lvl1->level == lvl2->level); + if (lvl1->level != lvl2->level) { + return TSDB_CODE_INVALID_PARA; + } int32_t i1 = 0, i2 = 0; while (i1 < TARRAY2_SIZE(lvl1->fobjArr) || i2 < TARRAY2_SIZE(lvl2->fobjArr)) { @@ -331,29 +333,24 @@ int32_t tsdbTFileSetEdit(STsdb *pTsdb, STFileSet *fset, const STFileOp *op) { code = TARRAY2_SORT_INSERT(lvl->fobjArr, fobj, tsdbTFileObjCmpr); if (code) return code; } else { - ASSERT(fset->farr[fobj->f->type] == NULL); fset->farr[fobj->f->type] = fobj; } } else if (op->optype == TSDB_FOP_REMOVE) { // delete a file if (op->of.type == TSDB_FTYPE_STT) { SSttLvl *lvl = tsdbTFileSetGetSttLvl(fset, op->of.stt->level); - ASSERT(lvl); STFileObj tfobj = {.f[0] = {.cid = op->of.cid}}; STFileObj *tfobjp = &tfobj; int32_t idx = TARRAY2_SEARCH_IDX(lvl->fobjArr, &tfobjp, tsdbTFileObjCmpr, TD_EQ); - ASSERT(idx >= 0); TARRAY2_REMOVE(lvl->fobjArr, idx, tsdbSttLvlClearFObj); } else { - ASSERT(tsdbIsSameTFile(&op->of, fset->farr[op->of.type]->f)); (void)tsdbTFileObjUnref(fset->farr[op->of.type]); fset->farr[op->of.type] = NULL; } } else { if (op->nf.type == TSDB_FTYPE_STT) { SSttLvl *lvl = tsdbTFileSetGetSttLvl(fset, op->of.stt->level); - ASSERT(lvl); STFileObj tfobj = {.f[0] = {.cid = op->of.cid}}, *tfobjp = &tfobj; STFileObj **fobjPtr = TARRAY2_SEARCH(lvl->fobjArr, &tfobjp, tsdbTFileObjCmpr, TD_EQ); @@ -374,7 +371,9 @@ int32_t tsdbTFileSetEdit(STsdb *pTsdb, STFileSet *fset, const STFileOp *op) { int32_t tsdbTFileSetApplyEdit(STsdb *pTsdb, const STFileSet *fset1, STFileSet *fset2) { int32_t code = 0; - ASSERT(fset1->fid == fset2->fid); + if (fset1->fid != fset2->fid) { + return TSDB_CODE_INVALID_PARA; + } for (tsdb_ftype_t ftype = TSDB_FTYPE_MIN; ftype < TSDB_FTYPE_MAX; ++ftype) { if (!fset1->farr[ftype] && !fset2->farr[ftype]) continue; diff --git a/source/dnode/vnode/src/tsdb/tsdbFSetRAW.c b/source/dnode/vnode/src/tsdb/tsdbFSetRAW.c index 1adca31347..09db2ef4ef 100644 --- a/source/dnode/vnode/src/tsdb/tsdbFSetRAW.c +++ b/source/dnode/vnode/src/tsdb/tsdbFSetRAW.c @@ -154,13 +154,11 @@ _exit: return code; } -int32_t tsdbFSetRAWWriteBlockData(SFSetRAWWriter *writer, STsdbDataRAWBlockHeader *bHdr, int32_t encryptAlgorithm, - char* encryptKey) { +int32_t tsdbFSetRAWWriteBlockData(SFSetRAWWriter *writer, STsdbDataRAWBlockHeader *bHdr, int32_t encryptAlgorithm, + char *encryptKey) { int32_t code = 0; int32_t lino = 0; - ASSERT(writer->ctx->offset >= 0 && writer->ctx->offset <= writer->ctx->file.size); - if (writer->ctx->offset == writer->ctx->file.size) { code = tsdbFSetRAWWriteFileDataEnd(writer); TSDB_CHECK_CODE(code, lino, _exit); @@ -173,7 +171,6 @@ int32_t tsdbFSetRAWWriteBlockData(SFSetRAWWriter *writer, STsdbDataRAWBlockHeade TSDB_CHECK_CODE(code, lino, _exit); writer->ctx->offset += bHdr->dataLength; - ASSERT(writer->ctx->offset == writer->dataWriter->ctx->offset); _exit: if (code) { diff --git a/source/dnode/vnode/src/tsdb/tsdbFSetRW.c b/source/dnode/vnode/src/tsdb/tsdbFSetRW.c index 3ccf833ddd..4a35316f35 100644 --- a/source/dnode/vnode/src/tsdb/tsdbFSetRW.c +++ b/source/dnode/vnode/src/tsdb/tsdbFSetRW.c @@ -73,8 +73,6 @@ static int32_t tsdbFSetWriteTableDataEnd(SFSetWriter *writer) { int32_t numRow = ((writer->blockData[pidx].nRow + writer->blockData[cidx].nRow) >> 1); if (writer->blockData[pidx].nRow > 0 && numRow >= writer->config->minRow) { - ASSERT(writer->blockData[pidx].nRow == writer->config->maxRow); - SRowInfo row = { .suid = writer->ctx->tbid->suid, .uid = writer->ctx->tbid->uid, diff --git a/source/dnode/vnode/src/tsdb/tsdbFile2.c b/source/dnode/vnode/src/tsdb/tsdbFile2.c index 3d6a259254..50272a689c 100644 --- a/source/dnode/vnode/src/tsdb/tsdbFile2.c +++ b/source/dnode/vnode/src/tsdb/tsdbFile2.c @@ -242,19 +242,30 @@ int32_t tsdbTFileObjInit(STsdb *pTsdb, const STFile *f, STFileObj **fobj) { int32_t tsdbTFileObjRef(STFileObj *fobj) { int32_t nRef; - (void)(void)taosThreadMutexLock(&fobj->mutex); - ASSERT(fobj->ref > 0 && fobj->state == TSDB_FSTATE_LIVE); + (void)taosThreadMutexLock(&fobj->mutex); + + if (fobj->ref <= 0 || fobj->state != TSDB_FSTATE_LIVE) { + (void)taosThreadMutexUnlock(&fobj->mutex); + tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, fobj->ref); + return TSDB_CODE_FAILED; + } + nRef = ++fobj->ref; - (void)(void)taosThreadMutexUnlock(&fobj->mutex); + (void)taosThreadMutexUnlock(&fobj->mutex); tsdbTrace("ref file %s, fobj:%p ref %d", fobj->fname, fobj, nRef); return 0; } int32_t tsdbTFileObjUnref(STFileObj *fobj) { - (void)(void)taosThreadMutexLock(&fobj->mutex); + (void)taosThreadMutexLock(&fobj->mutex); int32_t nRef = --fobj->ref; - (void)(void)taosThreadMutexUnlock(&fobj->mutex); - ASSERT(nRef >= 0); + (void)taosThreadMutexUnlock(&fobj->mutex); + + if (nRef < 0) { + tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, nRef); + return TSDB_CODE_FAILED; + } + tsdbTrace("unref file %s, fobj:%p ref %d", fobj->fname, fobj, nRef); if (nRef == 0) { if (fobj->state == TSDB_FSTATE_DEAD) { @@ -319,7 +330,11 @@ static void tsdbTFileObjRemoveLC(STFileObj *fobj, bool remove_all) { int32_t tsdbTFileObjRemove(STFileObj *fobj) { (void)taosThreadMutexLock(&fobj->mutex); - ASSERT(fobj->state == TSDB_FSTATE_LIVE && fobj->ref > 0); + if (fobj->state != TSDB_FSTATE_LIVE || fobj->ref <= 0) { + (void)taosThreadMutexUnlock(&fobj->mutex); + tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, fobj->ref); + return TSDB_CODE_FAILED; + } fobj->state = TSDB_FSTATE_DEAD; int32_t nRef = --fobj->ref; (void)taosThreadMutexUnlock(&fobj->mutex); @@ -333,7 +348,13 @@ int32_t tsdbTFileObjRemove(STFileObj *fobj) { int32_t tsdbTFileObjRemoveUpdateLC(STFileObj *fobj) { (void)taosThreadMutexLock(&fobj->mutex); - ASSERT(fobj->state == TSDB_FSTATE_LIVE && fobj->ref > 0); + + if (fobj->state != TSDB_FSTATE_LIVE || fobj->ref <= 0) { + (void)taosThreadMutexUnlock(&fobj->mutex); + tsdbError("file %s, fobj:%p ref %d", fobj->fname, fobj, fobj->ref); + return TSDB_CODE_FAILED; + } + fobj->state = TSDB_FSTATE_DEAD; int32_t nRef = --fobj->ref; (void)taosThreadMutexUnlock(&fobj->mutex); diff --git a/source/dnode/vnode/src/tsdb/tsdbIter.c b/source/dnode/vnode/src/tsdb/tsdbIter.c index 0de9ba9822..5f9e838ccc 100644 --- a/source/dnode/vnode/src/tsdb/tsdbIter.c +++ b/source/dnode/vnode/src/tsdb/tsdbIter.c @@ -547,8 +547,7 @@ int32_t tsdbIterOpen(const STsdbIterConfig *config, STsdbIter **iter) { code = tsdbMemTombIterOpen(iter[0]); break; default: - code = TSDB_CODE_INVALID_PARA; - ASSERTS(false, "Not implemented"); + return TSDB_CODE_INVALID_PARA; } if (code) { @@ -588,7 +587,7 @@ int32_t tsdbIterClose(STsdbIter **iter) { case TSDB_ITER_TYPE_MEMT_TOMB: break; default: - ASSERT(false); + return TSDB_CODE_INVALID_PARA; } taosMemoryFree(iter[0]); iter[0] = NULL; @@ -610,7 +609,7 @@ int32_t tsdbIterNext(STsdbIter *iter) { case TSDB_ITER_TYPE_MEMT_TOMB: return tsdbMemTombIterNext(iter, NULL); default: - ASSERT(false); + return TSDB_CODE_INVALID_PARA; } return 0; } @@ -630,7 +629,7 @@ static int32_t tsdbIterSkipTableData(STsdbIter *iter, const TABLEID *tbid) { case TSDB_ITER_TYPE_MEMT_TOMB: return tsdbMemTombIterNext(iter, tbid); default: - ASSERT(false); + return TSDB_CODE_INVALID_PARA; } return 0; } @@ -691,7 +690,10 @@ int32_t tsdbIterMergerOpen(const TTsdbIterArray *iterArray, SIterMerger **merger TARRAY2_FOREACH(iterArray, iter) { if (iter->noMoreData) continue; node = tRBTreePut(merger[0]->iterTree, iter->node); - ASSERT(node); + if (node == NULL) { + taosMemoryFree(merger[0]); + return TSDB_CODE_INVALID_PARA; + } } return tsdbIterMergerNext(merger[0]); @@ -718,10 +720,8 @@ int32_t tsdbIterMergerNext(SIterMerger *merger) { merger->iter = NULL; } else if ((node = tRBTreeMin(merger->iterTree))) { c = merger->iterTree->cmprFn(merger->iter->node, node); - ASSERT(c); if (c > 0) { node = tRBTreePut(merger->iterTree, merger->iter->node); - ASSERT(node); merger->iter = NULL; } } @@ -734,15 +734,9 @@ int32_t tsdbIterMergerNext(SIterMerger *merger) { return 0; } -SRowInfo *tsdbIterMergerGetData(SIterMerger *merger) { - ASSERT(!merger->isTomb); - return merger->iter ? merger->iter->row : NULL; -} +SRowInfo *tsdbIterMergerGetData(SIterMerger *merger) { return merger->iter ? merger->iter->row : NULL; } -STombRecord *tsdbIterMergerGetTombRecord(SIterMerger *merger) { - ASSERT(merger->isTomb); - return merger->iter ? merger->iter->record : NULL; -} +STombRecord *tsdbIterMergerGetTombRecord(SIterMerger *merger) { return merger->iter ? merger->iter->record : NULL; } int32_t tsdbIterMergerSkipTableData(SIterMerger *merger, const TABLEID *tbid) { int32_t code; @@ -757,10 +751,8 @@ int32_t tsdbIterMergerSkipTableData(SIterMerger *merger, const TABLEID *tbid) { merger->iter = NULL; } else if ((node = tRBTreeMin(merger->iterTree))) { c = merger->iterTree->cmprFn(merger->iter->node, node); - ASSERT(c); if (c > 0) { node = tRBTreePut(merger->iterTree, merger->iter->node); - ASSERT(node); merger->iter = NULL; } } diff --git a/source/dnode/vnode/src/tsdb/tsdbMemTable.c b/source/dnode/vnode/src/tsdb/tsdbMemTable.c index d8ec014985..79565f887b 100644 --- a/source/dnode/vnode/src/tsdb/tsdbMemTable.c +++ b/source/dnode/vnode/src/tsdb/tsdbMemTable.c @@ -170,7 +170,6 @@ int32_t tsdbDeleteTableData(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid goto _err; } - ASSERT(pPool != NULL); // do delete SDelData *pDelData = (SDelData *)vnodeBufPoolMalloc(pPool, sizeof(*pDelData)); if (pDelData == NULL) { @@ -183,7 +182,6 @@ int32_t tsdbDeleteTableData(STsdb *pTsdb, int64_t version, tb_uid_t suid, tb_uid pDelData->pNext = NULL; taosWLockLatch(&pTbData->lock); if (pTbData->pHead == NULL) { - ASSERT(pTbData->pTail == NULL); pTbData->pHead = pTbData->pTail = pDelData; } else { pTbData->pTail->pNext = pDelData; @@ -263,8 +261,6 @@ void tsdbTbDataIterOpen(STbData *pTbData, STsdbRowKey *pFrom, int8_t backward, S bool tsdbTbDataIterNext(STbDataIter *pIter) { pIter->pRow = NULL; if (pIter->backward) { - ASSERT(pIter->pNode != pIter->pTbData->sl.pTail); - if (pIter->pNode == pIter->pTbData->sl.pHead) { return false; } @@ -274,8 +270,6 @@ bool tsdbTbDataIterNext(STbDataIter *pIter) { return false; } } else { - ASSERT(pIter->pNode != pIter->pTbData->sl.pHead); - if (pIter->pNode == pIter->pTbData->sl.pTail) { return false; } @@ -366,7 +360,6 @@ static int32_t tsdbGetOrCreateTbData(SMemTable *pMemTable, tb_uid_t suid, tb_uid SVBufPool *pPool = pMemTable->pTsdb->pVnode->inUse; int8_t maxLevel = pMemTable->pTsdb->pVnode->config.tsdbCfg.slLevel; - ASSERT(pPool != NULL); pTbData = vnodeBufPoolMallocAligned(pPool, sizeof(*pTbData) + SL_NODE_SIZE(maxLevel) * 2); if (pTbData == NULL) { code = terrno; @@ -516,8 +509,6 @@ static int32_t tbDataDoPut(SMemTable *pMemTable, STbData *pTbData, SMemSkipListN pNode = (SMemSkipListNode *)vnodeBufPoolMallocAligned(pPool, nSize + pRow->pTSRow->len); } else if (pRow->type == TSDBROW_COL_FMT) { pNode = (SMemSkipListNode *)vnodeBufPoolMallocAligned(pPool, nSize); - } else { - ASSERT(0); } if (pNode == NULL) { code = terrno; @@ -582,10 +573,6 @@ static int32_t tsdbInsertColDataToTable(SMemTable *pMemTable, STbData *pTbData, int32_t nColData = TARRAY_SIZE(pSubmitTbData->aCol); SColData *aColData = (SColData *)TARRAY_DATA(pSubmitTbData->aCol); - ASSERT(aColData[0].cid == PRIMARYKEY_TIMESTAMP_COL_ID); - ASSERT(aColData[0].type == TSDB_DATA_TYPE_TIMESTAMP); - ASSERT(aColData[0].flag == HAS_VALUE); - // copy and construct block data SBlockData *pBlockData = vnodeBufPoolMalloc(pPool, sizeof(*pBlockData)); if (pBlockData == NULL) { @@ -740,7 +727,9 @@ int32_t tsdbRefMemTable(SMemTable *pMemTable, SQueryNode *pQNode) { int32_t code = 0; int32_t nRef = atomic_fetch_add_32(&pMemTable->nRef, 1); - ASSERT(nRef > 0); + if (nRef <= 0) { + tsdbError("vgId:%d, memtable ref count is invalid, ref:%d", TD_VID(pMemTable->pTsdb->pVnode), nRef); + } (void)vnodeBufPoolRegisterQuery(pMemTable->pPool, pQNode); diff --git a/source/dnode/vnode/src/tsdb/tsdbMerge.c b/source/dnode/vnode/src/tsdb/tsdbMerge.c index 1bbda8a249..5587f29440 100644 --- a/source/dnode/vnode/src/tsdb/tsdbMerge.c +++ b/source/dnode/vnode/src/tsdb/tsdbMerge.c @@ -69,13 +69,6 @@ static int32_t tsdbMergerClose(SMerger *merger) { int32_t lino = 0; SVnode *pVnode = merger->tsdb->pVnode; - ASSERT(merger->writer == NULL); - ASSERT(merger->dataIterMerger == NULL); - ASSERT(merger->tombIterMerger == NULL); - ASSERT(TARRAY2_SIZE(merger->dataIterArr) == 0); - ASSERT(TARRAY2_SIZE(merger->tombIterArr) == 0); - ASSERT(TARRAY2_SIZE(merger->sttReaderArr) == 0); - // clear the merge TARRAY2_DESTROY(merger->tombIterArr, NULL); TARRAY2_DESTROY(merger->dataIterArr, NULL); @@ -156,8 +149,6 @@ static int32_t tsdbMergeFileSetBeginOpenReader(SMerger *merger) { } } - ASSERT(merger->ctx->level > 0); - if (merger->ctx->level <= TSDB_MAX_LEVEL) { TARRAY2_FOREACH_REVERSE(merger->ctx->fset->lvlArr, lvl) { if (TARRAY2_SIZE(lvl->fobjArr) == 0) { @@ -180,8 +171,6 @@ static int32_t tsdbMergeFileSetBeginOpenReader(SMerger *merger) { numFile = numFile - TARRAY2_SIZE(lvl->fobjArr) * pow(merger->sttTrigger, lvl->level); } - ASSERT(numFile >= 0); - // get file system operations TARRAY2_FOREACH(merger->ctx->fset->lvlArr, lvl) { if (lvl->level >= merger->ctx->level) { @@ -323,11 +312,6 @@ static int32_t tsdbMergeFileSetBegin(SMerger *merger) { int32_t code = 0; int32_t lino = 0; - ASSERT(TARRAY2_SIZE(merger->sttReaderArr) == 0); - ASSERT(TARRAY2_SIZE(merger->dataIterArr) == 0); - ASSERT(merger->dataIterMerger == NULL); - ASSERT(merger->writer == NULL); - TARRAY2_CLEAR(merger->fopArr, NULL); merger->ctx->tbid->suid = 0; diff --git a/source/dnode/vnode/src/tsdb/tsdbMergeTree.c b/source/dnode/vnode/src/tsdb/tsdbMergeTree.c index 901632894e..21c5d33ec9 100644 --- a/source/dnode/vnode/src/tsdb/tsdbMergeTree.c +++ b/source/dnode/vnode/src/tsdb/tsdbMergeTree.c @@ -319,7 +319,7 @@ static int32_t extractSttBlockInfo(SLDataIter *pIter, const TSttBlkArray *pArray } else { // all blocks are qualified taosArrayClear(pBlockLoadInfo->aSttBlk); px = taosArrayAddBatch(pBlockLoadInfo->aSttBlk, pArray->data, pArray->size); - if (px == NULL){ + if (px == NULL) { return terrno; } } @@ -336,7 +336,7 @@ static int32_t extractSttBlockInfo(SLDataIter *pIter, const TSttBlkArray *pArray } if (p->suid == suid) { - void* px = taosArrayPush(pTmp, p); + void *px = taosArrayPush(pTmp, p); if (px == NULL) { code = terrno; break; @@ -372,7 +372,7 @@ static int32_t loadSttStatisticsBlockData(SSttFileReader *pSttFileReader, SSttBl TStatisBlkArray *pStatisBlkArray, uint64_t suid, const char *id) { int32_t code = TSDB_CODE_SUCCESS; int32_t lino = 0; - void* px = NULL; + void *px = NULL; int32_t startIndex = 0; int32_t numOfBlocks = TARRAY2_SIZE(pStatisBlkArray); @@ -415,7 +415,7 @@ static int32_t loadSttStatisticsBlockData(SSttFileReader *pSttFileReader, SSttBl // existed if (i < rows) { - SSttTableRowsInfo* pInfo = &pBlockLoadInfo->info; + SSttTableRowsInfo *pInfo = &pBlockLoadInfo->info; if (pInfo->pUid == NULL) { pInfo->pUid = taosArrayInit(rows, sizeof(int64_t)); @@ -530,7 +530,7 @@ static int32_t loadSttStatisticsBlockData(SSttFileReader *pSttFileReader, SSttBl } } - _end: +_end: (void)tStatisBlockDestroy(&block); double el = (taosGetTimestampUs() - st) / 1000.0; @@ -672,7 +672,7 @@ int32_t tLDataIterOpen2(SLDataIter *pIter, SSttFileReader *pSttFileReader, int32 } void tLDataIterClose2(SLDataIter *pIter) { - (void) tsdbSttFileReaderClose(&pIter->pReader); // always return 0 + (void)tsdbSttFileReaderClose(&pIter->pReader); // always return 0 pIter->pReader = NULL; } @@ -826,7 +826,7 @@ static int32_t findNextValidRow(SLDataIter *pIter, const char *idStr) { return code; } -int32_t tLDataIterNextRow(SLDataIter *pIter, const char *idStr, bool* hasNext) { +int32_t tLDataIterNextRow(SLDataIter *pIter, const char *idStr, bool *hasNext) { int32_t step = pIter->backward ? -1 : 1; int32_t code = 0; int32_t iBlockL = pIter->iSttBlk; @@ -835,7 +835,7 @@ int32_t tLDataIterNextRow(SLDataIter *pIter, const char *idStr, bool* hasNext) { *hasNext = false; terrno = 0; - + // no qualified last file block in current file, no need to fetch row if (pIter->pSttBlk == NULL) { return code; @@ -1020,7 +1020,7 @@ int32_t tMergeTreeOpen2(SMergeTree *pMTree, SMergeTreeConf *pConf, SSttDataInfoF // let's record the time window for current table of uid in the stt files if (pSttDataInfo != NULL && numOfRows > 0) { - void* px = taosArrayPush(pSttDataInfo->pKeyRangeList, &range); + void *px = taosArrayPush(pSttDataInfo->pKeyRangeList, &range); if (px == NULL) { return terrno; } @@ -1041,7 +1041,7 @@ _end: return code; } -void tMergeTreeAddIter(SMergeTree *pMTree, SLDataIter *pIter) { (void) tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pIter); } +void tMergeTreeAddIter(SMergeTree *pMTree, SLDataIter *pIter) { (void)tRBTreePut(&pMTree->rbt, (SRBTreeNode *)pIter); } bool tMergeTreeIgnoreEarlierTs(SMergeTree *pMTree) { return pMTree->ignoreEarlierTs; } @@ -1050,14 +1050,12 @@ static void tLDataIterPinSttBlock(SLDataIter *pIter, const char *id) { if (pInfo->blockData[0].sttBlockIndex == pIter->iSttBlk) { pInfo->blockData[0].pin = true; - ASSERT(!pInfo->blockData[1].pin); tsdbTrace("pin stt-block, blockIndex:%d, stt-fileVer:%" PRId64 " %s", pIter->iSttBlk, pIter->cid, id); return; } if (pInfo->blockData[1].sttBlockIndex == pIter->iSttBlk) { pInfo->blockData[1].pin = true; - ASSERT(!pInfo->blockData[0].pin); tsdbTrace("pin stt-block, blockIndex:%d, stt-fileVer:%" PRId64 " %s", pIter->iSttBlk, pIter->cid, id); return; } @@ -1068,14 +1066,12 @@ static void tLDataIterPinSttBlock(SLDataIter *pIter, const char *id) { static void tLDataIterUnpinSttBlock(SLDataIter *pIter, const char *id) { SSttBlockLoadInfo *pInfo = pIter->pBlockLoadInfo; if (pInfo->blockData[0].pin) { - ASSERT(!pInfo->blockData[1].pin); pInfo->blockData[0].pin = false; tsdbTrace("unpin stt-block:%d, stt-fileVer:%" PRId64 " %s", pInfo->blockData[0].sttBlockIndex, pIter->cid, id); return; } if (pInfo->blockData[1].pin) { - ASSERT(!pInfo->blockData[0].pin); pInfo->blockData[1].pin = false; tsdbTrace("unpin stt-block:%d, stt-fileVer:%" PRId64 " %s", pInfo->blockData[1].sttBlockIndex, pIter->cid, id); return; @@ -1117,7 +1113,7 @@ int32_t tMergeTreeNext(SMergeTree *pMTree, bool *pHasNext) { code = tLDataIterNextRow(pIter, pMTree->idStr, &hasVal); if (!hasVal || (code != 0)) { if (code == TSDB_CODE_FILE_CORRUPTED) { - code = 0; // suppress the file corrupt error to enable all queries within this cluster can run without failed. + code = 0; // suppress the file corrupt error to enable all queries within this cluster can run without failed. } pMTree->pIter = NULL; diff --git a/source/dnode/vnode/src/tsdb/tsdbRead2.c b/source/dnode/vnode/src/tsdb/tsdbRead2.c index 9be2c3b3f6..391b7f636d 100644 --- a/source/dnode/vnode/src/tsdb/tsdbRead2.c +++ b/source/dnode/vnode/src/tsdb/tsdbRead2.c @@ -137,7 +137,6 @@ static int32_t tGetPrimaryKeyIndex(uint8_t* p, SPrimaryKeyIndex* index) { static void tRowGetPrimaryKeyDeepCopy(SRow* pRow, SRowKey* pKey) { SPrimaryKeyIndex indices[TD_MAX_PK_COLS]; - ASSERT(pKey->numOfPKs <= TD_MAX_PK_COLS); uint8_t* data = pRow->data; for (int32_t i = 0; i < pRow->numOfPKs; i++) { @@ -673,7 +672,10 @@ static int32_t doLoadBlockIndex(STsdbReader* pReader, SDataFileReader* pFileRead break; } - ASSERT(pBrinBlk->minTbid.suid <= pReader->info.suid && pBrinBlk->maxTbid.suid >= pReader->info.suid); + if (!(pBrinBlk->minTbid.suid <= pReader->info.suid && pBrinBlk->maxTbid.suid >= pReader->info.suid)) { + tsdbError("tsdb failed at: %s %d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } if (pBrinBlk->maxTbid.suid == pReader->info.suid && pBrinBlk->maxTbid.uid < pList->tableUidList[0]) { i += 1; continue; @@ -753,7 +755,10 @@ static int32_t loadFileBlockBrinInfo(STsdbReader* pReader, SArray* pIndexList, S continue; } - ASSERT(pRecord->suid == pReader->info.suid && uid == pRecord->uid); + if (!(pRecord->suid == pReader->info.suid && uid == pRecord->uid)) { + tsdbError("tsdb failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } STableBlockScanInfo* pScanInfo = NULL; code = getTableBlockScanInfo(pReader->status.pTableMap, uid, &pScanInfo, pReader->idStr); @@ -924,21 +929,23 @@ static int32_t getCurrentBlockInfo(SDataBlockIter* pBlockIter, SFileDataBlockInf size_t num = TARRAY_SIZE(pBlockIter->blockList); if (num == 0) { - ASSERT(pBlockIter->numOfBlocks == num); - return TSDB_CODE_FAILED; + tsdbError("tsdb read failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; } *pInfo = taosArrayGet(pBlockIter->blockList, pBlockIter->index); return (*pInfo) != NULL? TSDB_CODE_SUCCESS:TSDB_CODE_FAILED; } -static int doBinarySearchKey(const TSKEY* keyList, int num, int pos, TSKEY key, int order) { +static int32_t doBinarySearchKey(const TSKEY* keyList, int num, int pos, TSKEY key, int order) { // start end position int s, e; s = pos; // check - ASSERT(pos >= 0 && pos < num && num > 0); + if (!(pos >= 0 && pos < num && num > 0)) { + return -1; + } if (order == TSDB_ORDER_ASC) { // find the first position which is smaller than the key e = num - 1; @@ -1242,7 +1249,10 @@ static int32_t copyBlockDataToSDataBlock(STsdbReader* pReader, SRowKey* pLastPro } pDumpInfo->rowIndex = findFirstPos(pBlockData->aTSKEY, pRecord->numRow, pDumpInfo->rowIndex, (!asc)); - ASSERT(pReader->info.verRange.minVer <= pRecord->maxVer && pReader->info.verRange.maxVer >= pRecord->minVer); + if (!(pReader->info.verRange.minVer <= pRecord->maxVer && pReader->info.verRange.maxVer >= pRecord->minVer)) { + tsdbError("tsdb failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INVALID_PARA; + } // find the appropriate start position that satisfies the version requirement. if ((pReader->info.verRange.maxVer >= pRecord->minVer && pReader->info.verRange.maxVer < pRecord->maxVer) || @@ -1382,7 +1392,12 @@ static int32_t copyBlockDataToSDataBlock(STsdbReader* pReader, SRowKey* pLastPro } static FORCE_INLINE STSchema* getTableSchemaImpl(STsdbReader* pReader, uint64_t uid) { - ASSERT(pReader->info.pSchema == NULL); + if (pReader->info.pSchema != NULL) { + terrno = TSDB_CODE_INVALID_PARA; + tsdbError("tsdb invalid input param at: %s:%d", __func__, __LINE__); + return NULL; + } + int32_t code = metaGetTbTSchemaEx(pReader->pTsdb->pVnode->pMeta, pReader->info.suid, uid, -1, &pReader->info.pSchema); if (code != TSDB_CODE_SUCCESS || pReader->info.pSchema == NULL) { terrno = code; @@ -1663,7 +1678,6 @@ static void getBlockToLoadInfo(SDataBlockToLoadInfo* pInfo, SFileDataBlockInfo* pInfo->overlapWithDelInfo = overlapWithDelSkyline(pScanInfo, &pRecord, order); // todo handle the primary key overlap case - ASSERT(pScanInfo->sttKeyInfo.status != STT_FILE_READER_UNINIT); if (pScanInfo->sttKeyInfo.status == STT_FILE_HAS_DATA) { int64_t nextProcKeyInStt = pScanInfo->sttKeyInfo.nextProcKey.ts; pInfo->overlapWithSttBlock = !(pBlockInfo->lastKey < nextProcKeyInStt || pBlockInfo->firstKey > nextProcKeyInStt); @@ -1921,7 +1935,10 @@ static int32_t doMergeBufAndFileRows(STsdbReader* pReader, STableBlockScanInfo* // merge is not initialized yet, due to the fact that the pReader->info.pSchema is not initialized if (pMerger->pArray == NULL) { - ASSERT(pReader->info.pSchema == NULL); + if (pReader->info.pSchema != NULL) { + tsdbError("tsdb failed at %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } STSchema* ps = getTableSchemaImpl(pReader, pBlockScanInfo->uid); if (ps == NULL) { return terrno; @@ -2014,7 +2031,10 @@ static int32_t mergeFileBlockAndSttBlock(STsdbReader* pReader, SSttBlockReader* // merge is not initialized yet, due to the fact that the pReader->info.pSchema is not initialized if (pMerger->pArray == NULL) { - ASSERT(pReader->info.pSchema == NULL); + if (pReader->info.pSchema) { + tsdbError("tsdb failed at %s %d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } STSchema* ps = getTableSchemaImpl(pReader, pBlockScanInfo->uid); if (ps == NULL) { return terrno; @@ -2147,7 +2167,10 @@ static int32_t doMergeMultiLevelRows(STsdbReader* pReader, STableBlockScanInfo* // merge is not initialized yet, due to the fact that the pReader->info.pSchema is not initialized if (pMerger->pArray == NULL) { - ASSERT(pReader->info.pSchema == NULL); + if (pReader->info.pSchema != NULL) { + tsdbError("tsdb read failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } STSchema* ps = getTableSchemaImpl(pReader, pBlockScanInfo->uid); if (ps == NULL) { return terrno; @@ -2548,7 +2571,10 @@ int32_t mergeRowsInFileBlocks(SBlockData* pBlockData, STableBlockScanInfo* pBloc // merge is not initialized yet, due to the fact that the pReader->info.pSchema is not initialized if (pMerger->pArray == NULL) { - ASSERT(pReader->info.pSchema == NULL); + if (pReader->info.pSchema != NULL) { + tsdbError("tsdb reader failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } STSchema* ps = getTableSchemaImpl(pReader, pBlockScanInfo->uid); if (ps == NULL) { return terrno; @@ -3267,8 +3293,6 @@ static int32_t doLoadSttBlockSequentially(STsdbReader* pReader) { // current active data block not overlap with the stt-files/stt-blocks static bool notOverlapWithFiles(SFileDataBlockInfo* pBlockInfo, STableBlockScanInfo* pScanInfo, bool asc) { - ASSERT(pScanInfo->sttKeyInfo.status != STT_FILE_READER_UNINIT); - if ((!hasDataInSttBlock(pScanInfo)) || (pScanInfo->cleanSttBlocks == true)) { return true; } else { @@ -3336,7 +3360,10 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) { // data block, so the overlap check is invalid actually. buildCleanBlockFromDataFiles(pReader, pScanInfo, pBlockInfo, pBlockIter->index); } else { // clean stt block - ASSERT(pReader->info.execMode == READER_EXEC_ROWS && pSttBlockReader->mergeTree.pIter == NULL); + if (!(pReader->info.execMode == READER_EXEC_ROWS && pSttBlockReader->mergeTree.pIter == NULL)) { + tsdbError("tsdb reader failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } code = buildCleanBlockFromSttFiles(pReader, pScanInfo); return code; } @@ -3357,7 +3384,10 @@ static int32_t doBuildDataBlock(STsdbReader* pReader) { // no data in stt block, no need to proceed. while (hasDataInSttBlock(pScanInfo)) { - ASSERT(pScanInfo->sttKeyInfo.status == STT_FILE_HAS_DATA); + if (pScanInfo->sttKeyInfo.status != STT_FILE_HAS_DATA) { + tsdbError("tsdb reader failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } code = buildComposedDataBlockImpl(pReader, pScanInfo, &pReader->status.fileBlockData, pSttBlockReader); if (code != TSDB_CODE_SUCCESS) { @@ -3580,7 +3610,11 @@ static ERetrieveType doReadDataFromSttFiles(STsdbReader* pReader) { } // all data blocks are checked in this stt file, now let's try the next file set - ASSERT(pReader->status.pTableIter == NULL); + if (pReader->status.pTableIter != NULL) { + terrno = TSDB_CODE_INTERNAL_ERROR; + tsdbError("tsdb reader failed at: %s:%d", __func__, __LINE__); + return TSDB_READ_RETURN; + } code = initForFirstBlockInFile(pReader, pBlockIter); // error happens or all the data files are completely checked @@ -3794,7 +3828,7 @@ bool hasBeenDropped(const SArray* pDelList, int32_t* index, int64_t key, int64_t return false; } - ASSERT(key >= last->ts); + // ASSERT(key >= last->ts); if (key > last->ts) { return false; } else if (key == last->ts) { @@ -3857,7 +3891,7 @@ bool hasBeenDropped(const SArray* pDelList, int32_t* index, int64_t key, int64_t } else if (key == pFirst->ts) { return pFirst->version >= ver; } else { - ASSERT(0); + // ASSERT(0); } } else { TSDBKEY* pCurrent = taosArrayGet(pDelList, *index); diff --git a/source/dnode/vnode/src/tsdb/tsdbReadUtil.c b/source/dnode/vnode/src/tsdb/tsdbReadUtil.c index 9f5f72e0aa..5d4dc94431 100644 --- a/source/dnode/vnode/src/tsdb/tsdbReadUtil.c +++ b/source/dnode/vnode/src/tsdb/tsdbReadUtil.c @@ -194,7 +194,7 @@ int32_t initRowKey(SRowKey* pKey, int64_t ts, int32_t numOfPks, int32_t type, in break; } default: - ASSERT(0); + return TSDB_CODE_INVALID_PARA; } } else { switch (type) { @@ -223,7 +223,7 @@ int32_t initRowKey(SRowKey* pKey, int64_t ts, int32_t numOfPks, int32_t type, in pKey->pks[0].val = UINT8_MAX; break; default: - ASSERT(0); + return TSDB_CODE_INVALID_PARA; } } } else { @@ -741,7 +741,10 @@ int32_t initBlockIterator(STsdbReader* pReader, SDataBlockIter* pBlockIter, int3 } numOfTotal += 1; - tMergeTreeAdjust(pTree, tMergeTreeGetAdjustIndex(pTree)); + code = tMergeTreeAdjust(pTree, tMergeTreeGetAdjustIndex(pTree)); + if (TSDB_CODE_SUCCESS != code) { + return code; + } } for (int32_t i = 0; i < numOfTables; ++i) { @@ -843,7 +846,10 @@ static int32_t doCheckTombBlock(STombBlock* pBlock, STsdbReader* pReader, int32_ continue; } - ASSERT(record.suid == pReader->info.suid && uid == record.uid); + if (!(record.suid == pReader->info.suid && uid == record.uid)) { + tsdbError("tsdb reader failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } if (record.version <= pReader->info.verRange.maxVer) { SDelData delData = {.version = record.version, .sKey = record.skey, .eKey = record.ekey}; @@ -877,7 +883,10 @@ static int32_t doLoadTombDataFromTombBlk(const TTombBlkArray* pTombBlkArray, STs break; } - ASSERT(pTombBlk->minTbid.suid <= pReader->info.suid && pTombBlk->maxTbid.suid >= pReader->info.suid); + if (!(pTombBlk->minTbid.suid <= pReader->info.suid && pTombBlk->maxTbid.suid >= pReader->info.suid)) { + tsdbError("tsdb reader failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } if (pTombBlk->maxTbid.suid == pReader->info.suid && pTombBlk->maxTbid.uid < pList->tableUidList[0]) { i += 1; continue; diff --git a/source/dnode/vnode/src/tsdb/tsdbReadUtil.h b/source/dnode/vnode/src/tsdb/tsdbReadUtil.h index 865e8e2d41..ed895b7d27 100644 --- a/source/dnode/vnode/src/tsdb/tsdbReadUtil.h +++ b/source/dnode/vnode/src/tsdb/tsdbReadUtil.h @@ -398,6 +398,7 @@ typedef struct SCacheRowsReader { } SCacheRowsReader; int32_t tsdbCacheGetBatch(STsdb* pTsdb, tb_uid_t uid, SArray* pLastArray, SCacheRowsReader* pr, int8_t ltype); +void tsdbCacheFreeSLastColItem(void* pItem); #ifdef __cplusplus } diff --git a/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c b/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c index 8e52074807..8632603fd6 100644 --- a/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c +++ b/source/dnode/vnode/src/tsdb/tsdbReaderWriter.c @@ -77,7 +77,9 @@ static int32_t tsdbOpenFileImpl(STsdbFD *pFD) { } } - ASSERT(pFD->szFile % szPage == 0); + if (pFD->szFile % szPage != 0) { + TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit); + } pFD->szFile = pFD->szFile / szPage; _exit: @@ -149,7 +151,6 @@ static int32_t tsdbWriteFilePage(STsdbFD *pFD, int32_t encryptAlgorithm, char *e offset -= chunkoffset; } - ASSERT(offset >= 0); int64_t n = taosLSeekFile(pFD->pFD, offset, SEEK_SET); if (n < 0) { @@ -202,7 +203,6 @@ static int32_t tsdbReadFilePage(STsdbFD *pFD, int64_t pgno, int32_t encryptAlgor int32_t code = 0; int32_t lino; - // ASSERT(pgno <= pFD->szFile); if (!pFD->pFD) { code = tsdbOpenFileImpl(pFD); TSDB_CHECK_CODE(code, lino, _exit); @@ -216,7 +216,6 @@ static int32_t tsdbReadFilePage(STsdbFD *pFD, int64_t pgno, int32_t encryptAlgor offset -= chunkoffset; } - ASSERT(offset >= 0); // seek int64_t n = taosLSeekFile(pFD->pFD, offset, SEEK_SET); @@ -317,8 +316,9 @@ static int32_t tsdbReadFileImp(STsdbFD *pFD, int64_t offset, uint8_t *pBuf, int6 int32_t szPgCont = PAGE_CONTENT_SIZE(pFD->szPage); int64_t bOffset = fOffset % pFD->szPage; - // ASSERT(pgno && pgno <= pFD->szFile); - ASSERT(bOffset < szPgCont); + if (bOffset >= szPgCont) { + TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit); + } while (n < size) { if (pFD->pgno != pgno) { @@ -417,7 +417,9 @@ static int32_t tsdbReadFileS3(STsdbFD *pFD, int64_t offset, uint8_t *pBuf, int64 int64_t pgno = OFFSET_PGNO(fOffset, pFD->szPage); int64_t bOffset = fOffset % pFD->szPage; - ASSERT(bOffset < szPgCont); + if (bOffset >= szPgCont) { + TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit); + } // 1, find pgnoStart & pgnoEnd to fetch from s3, if all pgs are local, no need to fetch // 2, fetch pgnoStart ~ pgnoEnd from s3 @@ -690,7 +692,9 @@ int32_t tsdbReadBlockIdx(SDataFReader *pReader, SArray *aBlockIdx) { TSDB_CHECK_CODE(code = TSDB_CODE_OUT_OF_MEMORY, lino, _exit); } } - ASSERT(n == size); + if (n != size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { @@ -731,7 +735,9 @@ int32_t tsdbReadSttBlk(SDataFReader *pReader, int32_t iStt, SArray *aSttBlk) { TSDB_CHECK_CODE(code = TSDB_CODE_OUT_OF_MEMORY, lino, _exit); } } - ASSERT(n == size); + if (n != size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { @@ -760,7 +766,9 @@ int32_t tsdbReadDataBlk(SDataFReader *pReader, SBlockIdx *pBlockIdx, SMapData *m int32_t n; code = tGetMapData(pReader->aBuf[0], mDataBlk, &n); if (code) goto _exit; - ASSERT(n == size); + if (n != size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { @@ -861,7 +869,9 @@ int32_t tsdbReadDelDatav1(SDelFReader *pReader, SDelIdx *pDelIdx, SArray *aDelDa } } - ASSERT(n == size); + if (n != size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { @@ -901,7 +911,9 @@ int32_t tsdbReadDelIdx(SDelFReader *pReader, SArray *aDelIdx) { } } - ASSERT(n == size); + if (n != size) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } _exit: if (code) { diff --git a/source/dnode/vnode/src/tsdb/tsdbRetention.c b/source/dnode/vnode/src/tsdb/tsdbRetention.c index bdd174a8f8..96b752104d 100644 --- a/source/dnode/vnode/src/tsdb/tsdbRetention.c +++ b/source/dnode/vnode/src/tsdb/tsdbRetention.c @@ -403,8 +403,6 @@ static int32_t tsdbS3FidLevel(int32_t fid, STsdbKeepCfg *pKeepCfg, int32_t s3Kee nowSec = nowSec * 1000000l; } else if (pKeepCfg->precision == TSDB_TIME_PRECISION_NANO) { nowSec = nowSec * 1000000000l; - } else { - ASSERT(0); } nowSec = nowSec - pKeepCfg->keepTimeOffset * tsTickPerHour[pKeepCfg->precision]; diff --git a/source/dnode/vnode/src/tsdb/tsdbSnapInfo.c b/source/dnode/vnode/src/tsdb/tsdbSnapInfo.c index decf276bc6..4a73f53c77 100644 --- a/source/dnode/vnode/src/tsdb/tsdbSnapInfo.c +++ b/source/dnode/vnode/src/tsdb/tsdbSnapInfo.c @@ -92,7 +92,6 @@ static int32_t tsdbTFileSetToFSetPartition(STFileSet* fset, STsdbFSetPartition** for (int32_t ftype = TSDB_FTYPE_MIN; ftype < TSDB_FTYPE_MAX; ++ftype) { if (fset->farr[ftype] == NULL) continue; typ = tsdbFTypeToFRangeType(ftype); - ASSERT(typ < TSDB_FSET_RANGE_TYP_MAX); STFile* f = fset->farr[ftype]->f; if (f->maxVer > fset->maxVerValid) { corrupt = true; @@ -103,8 +102,7 @@ static int32_t tsdbTFileSetToFSetPartition(STFileSet* fset, STsdbFSetPartition** } count++; SVersionRange vr = {.minVer = f->minVer, .maxVer = f->maxVer}; - code = TARRAY2_SORT_INSERT(&p->verRanges[typ], vr, tVersionRangeCmprFn); - ASSERT(code == 0); + (void)TARRAY2_SORT_INSERT(&p->verRanges[typ], vr, tVersionRangeCmprFn); } typ = TSDB_FSET_RANGE_TYP_STT; @@ -122,14 +120,12 @@ static int32_t tsdbTFileSetToFSetPartition(STFileSet* fset, STsdbFSetPartition** } count++; SVersionRange vr = {.minVer = f->minVer, .maxVer = f->maxVer}; - code = TARRAY2_SORT_INSERT(&p->verRanges[typ], vr, tVersionRangeCmprFn); - ASSERT(code == 0); + (void)TARRAY2_SORT_INSERT(&p->verRanges[typ], vr, tVersionRangeCmprFn); } } if (corrupt && count == 0) { SVersionRange vr = {.minVer = VERSION_MIN, .maxVer = fset->maxVerValid}; - code = TARRAY2_SORT_INSERT(&p->verRanges[typ], vr, tVersionRangeCmprFn); - ASSERT(code == 0); + (void)TARRAY2_SORT_INSERT(&p->verRanges[typ], vr, tVersionRangeCmprFn); } ppSP[0] = p; return 0; @@ -186,8 +182,7 @@ int32_t tsdbFSetPartListToRangeDiff(STsdbFSetPartList* pList, TFileSetRangeArray r->sver = maxVerValid + 1; r->ever = VERSION_MAX; tsdbDebug("range diff fid:%" PRId64 ", sver:%" PRId64 ", ever:%" PRId64, part->fid, r->sver, r->ever); - int32_t code = TARRAY2_SORT_INSERT(pDiff, r, tsdbTFileSetRangeCmprFn); - ASSERT(code == 0); + (void)TARRAY2_SORT_INSERT(pDiff, r, tsdbTFileSetRangeCmprFn); } ppRanges[0] = pDiff; @@ -360,9 +355,7 @@ static STsdbFSetPartList* tsdbSnapGetFSetPartList(STFileSystem* fs) { terrno = code; break; } - ASSERT(pItem != NULL); - code = TARRAY2_SORT_INSERT(pList, pItem, tsdbFSetPartCmprFn); - ASSERT(code == 0); + (void)TARRAY2_SORT_INSERT(pList, pItem, tsdbFSetPartCmprFn); } (void)taosThreadMutexUnlock(&fs->tsdb->mutex); @@ -400,7 +393,9 @@ static int32_t tsdbPartitionInfoInit(SVnode* pVnode, STsdbPartitionInfo* pInfo) pInfo->vgId = TD_VID(pVnode); pInfo->tsdbMaxCnt = (!VND_IS_RSMA(pVnode) ? 1 : TSDB_RETENTION_MAX); - ASSERT(sizeof(pInfo->subTyps) == sizeof(subTyps)); + if (!(sizeof(pInfo->subTyps) == sizeof(subTyps))) { + return TSDB_CODE_INVALID_PARA; + } memcpy(pInfo->subTyps, (char*)subTyps, sizeof(subTyps)); // fset partition list @@ -437,8 +432,7 @@ static int32_t tsdbPartitionInfoSerialize(STsdbPartitionInfo* pInfo, uint8_t* bu for (int32_t j = 0; j < pInfo->tsdbMaxCnt; ++j) { SSyncTLV* pSubHead = (void*)((char*)buf + offset); int32_t valOffset = offset + sizeof(*pSubHead); - ASSERT(pSubHead->val == (char*)buf + valOffset); - int32_t code = tSerializeTsdbFSetPartList(pSubHead->val, bufLen - valOffset, pInfo->pLists[j], &tlen); + int32_t code = tSerializeTsdbFSetPartList(pSubHead->val, bufLen - valOffset, pInfo->pLists[j], &tlen); if (code) { tsdbError("vgId:%d, failed to serialize fset partition list of tsdb %d since %s", pInfo->vgId, j, terrstr()); return code; @@ -574,7 +568,6 @@ static int32_t tsdbSnapPrepDealWithSnapInfo(SVnode* pVnode, SSnapshot* pSnap, ST } int32_t tsdbSnapPrepDescription(SVnode* pVnode, SSnapshot* pSnap) { - ASSERT(pSnap->type == TDMT_SYNC_PREP_SNAPSHOT || pSnap->type == TDMT_SYNC_PREP_SNAPSHOT_REPLY); STsdbPartitionInfo partitionInfo = {0}; int code = 0; STsdbPartitionInfo* pInfo = &partitionInfo; @@ -616,7 +609,6 @@ int32_t tsdbSnapPrepDescription(SVnode* pVnode, SSnapshot* pSnap) { goto _out; } offset += tlen; - ASSERT(offset <= bufLen); if ((tlen = tsdbRepOptsSerialize(&opts, buf + offset, bufLen - offset)) < 0) { code = tlen; @@ -624,7 +616,6 @@ int32_t tsdbSnapPrepDescription(SVnode* pVnode, SSnapshot* pSnap) { goto _out; } offset += tlen; - ASSERT(offset <= bufLen); // set header of info data SSyncTLV* pHead = pSnap->data; diff --git a/source/dnode/vnode/src/tsdb/tsdbSnapshot.c b/source/dnode/vnode/src/tsdb/tsdbSnapshot.c index 2b63d5b6e6..680b1250f9 100644 --- a/source/dnode/vnode/src/tsdb/tsdbSnapshot.c +++ b/source/dnode/vnode/src/tsdb/tsdbSnapshot.c @@ -67,9 +67,6 @@ static int32_t tsdbSnapReadFileSetOpenReader(STsdbSnapReader* reader) { int32_t code = 0; int32_t lino = 0; - ASSERT(reader->dataReader == NULL); - ASSERT(TARRAY2_SIZE(reader->sttReaderArr) == 0); - // data SDataFileReaderConfig config = { .tsdb = reader->tsdb, @@ -125,11 +122,6 @@ static int32_t tsdbSnapReadFileSetOpenIter(STsdbSnapReader* reader) { int32_t code = 0; int32_t lino = 0; - ASSERT(reader->dataIterMerger == NULL); - ASSERT(reader->tombIterMerger == NULL); - ASSERT(TARRAY2_SIZE(reader->dataIterArr) == 0); - ASSERT(TARRAY2_SIZE(reader->tombIterArr) == 0); - STsdbIter* iter; STsdbIterConfig config = { .filterByVersion = true, @@ -210,8 +202,6 @@ static int32_t tsdbSnapReadRangeBegin(STsdbSnapReader* reader) { int32_t code = 0; int32_t lino = 0; - ASSERT(reader->ctx->fsr == NULL); - if (reader->ctx->fsrArrIdx < TARRAY2_SIZE(reader->fsrArr)) { reader->ctx->fsr = TARRAY2_GET(reader->fsrArr, reader->ctx->fsrArrIdx++); reader->ctx->isDataDone = false; @@ -335,7 +325,6 @@ static int32_t tsdbSnapReadTimeSeriesData(STsdbSnapReader* reader, uint8_t** dat } if (reader->blockData->nRow > 0) { - ASSERT(reader->blockData->suid || reader->blockData->uid); code = tsdbSnapCmprData(reader, data); TSDB_CHECK_CODE(code, lino, _exit); } @@ -616,7 +605,6 @@ static int32_t tsdbSnapWriteTimeSeriesRow(STsdbSnapWriter* writer, SRowInfo* row } if (row->suid == INT64_MAX) { - ASSERT(writer->ctx->hasData == false); goto _exit; } @@ -634,9 +622,6 @@ static int32_t tsdbSnapWriteFileSetOpenReader(STsdbSnapWriter* writer) { int32_t code = 0; int32_t lino = 0; - ASSERT(writer->ctx->dataReader == NULL); - ASSERT(TARRAY2_SIZE(writer->ctx->sttReaderArr) == 0); - if (writer->ctx->fset) { // open data reader SDataFileReaderConfig dataFileReaderConfig = { @@ -825,8 +810,6 @@ static int32_t tsdbSnapWriteFileSetBegin(STsdbSnapWriter* writer, int32_t fid) { int32_t code = 0; int32_t lino = 0; - ASSERT(writer->ctx->fsetWriteBegin == false); - STFileSet* fset = &(STFileSet){.fid = fid}; writer->ctx->fid = fid; @@ -885,7 +868,6 @@ static int32_t tsdbSnapWriteTombRecord(STsdbSnapWriter* writer, const STombRecor } if (record->suid == INT64_MAX) { - ASSERT(writer->ctx->hasTomb == false); goto _exit; } @@ -996,7 +978,6 @@ static int32_t tsdbSnapWriteDecmprTombBlock(SSnapDataHdr* hdr, STombBlock* tombB TAOS_UNUSED(tTombBlockClear(tombBlock)); int64_t size = hdr->size; - ASSERT(size % TOMB_RECORD_ELEM_NUM == 0); size = size / TOMB_RECORD_ELEM_NUM; tombBlock->numOfRecords = size / sizeof(int64_t); @@ -1043,8 +1024,6 @@ static int32_t tsdbSnapWriteTombData(STsdbSnapWriter* writer, SSnapDataHdr* hdr) TSDB_CHECK_CODE(code, lino, _exit); } - ASSERT(writer->ctx->hasData == false); - for (int32_t i = 0; i < TOMB_BLOCK_SIZE(tombBlock); ++i) { code = tTombBlockGet(tombBlock, i, &record); TSDB_CHECK_CODE(code, lino, _exit); @@ -1177,7 +1156,7 @@ int32_t tsdbSnapWrite(STsdbSnapWriter* writer, SSnapDataHdr* hdr) { code = tsdbSnapWriteTombData(writer, hdr); TSDB_CHECK_CODE(code, lino, _exit); } else { - ASSERT(0); + TSDB_CHECK_CODE(code = TSDB_CODE_INVALID_PARA, lino, _exit); } _exit: diff --git a/source/dnode/vnode/src/tsdb/tsdbSnapshotRAW.c b/source/dnode/vnode/src/tsdb/tsdbSnapshotRAW.c index 5b69638b36..55ddf64421 100644 --- a/source/dnode/vnode/src/tsdb/tsdbSnapshotRAW.c +++ b/source/dnode/vnode/src/tsdb/tsdbSnapshotRAW.c @@ -170,11 +170,8 @@ static int64_t tsdbSnapRAWReadPeek(SDataFileRAWReader* reader) { } static SDataFileRAWReader* tsdbSnapRAWReaderIterNext(STsdbSnapRAWReader* reader) { - ASSERT(reader->dataIter->idx <= reader->dataIter->count); - while (reader->dataIter->idx < reader->dataIter->count) { SDataFileRAWReader* dataReader = TARRAY2_GET(reader->dataReaderArr, reader->dataIter->idx); - ASSERT(dataReader); if (dataReader->ctx->offset < dataReader->config->file.size) { return dataReader; } @@ -196,7 +193,6 @@ static int32_t tsdbSnapRAWReadNext(STsdbSnapRAWReader* reader, SSnapDataHdr** pp // prepare int64_t dataLength = tsdbSnapRAWReadPeek(dataReader); - ASSERT(dataLength > 0); void* pBuf = taosMemoryCalloc(1, sizeof(SSnapDataHdr) + sizeof(STsdbDataRAWBlockHeader) + dataLength); if (pBuf == NULL) { @@ -217,7 +213,6 @@ static int32_t tsdbSnapRAWReadNext(STsdbSnapRAWReader* reader, SSnapDataHdr** pp // finish dataReader->ctx->offset += pBlock->dataLength; - ASSERT(dataReader->ctx->offset <= dataReader->config->file.size); ppData[0] = pBuf; _exit: @@ -247,8 +242,6 @@ static int32_t tsdbSnapRAWReadBegin(STsdbSnapRAWReader* reader) { int32_t code = 0; int32_t lino = 0; - ASSERT(reader->ctx->fset == NULL); - if (reader->ctx->fsetArrIdx < TARRAY2_SIZE(reader->fsetArr)) { reader->ctx->fset = TARRAY2_GET(reader->fsetArr, reader->ctx->fsetArrIdx++); reader->ctx->isDataDone = false; @@ -409,8 +402,6 @@ static int32_t tsdbSnapRAWWriteFileSetBegin(STsdbSnapRAWWriter* writer, int32_t int32_t code = 0; int32_t lino = 0; - ASSERT(writer->ctx->fsetWriteBegin == false); - STFileSet* fset = &(STFileSet){.fid = fid}; writer->ctx->fid = fid; @@ -555,8 +546,6 @@ _exit: } int32_t tsdbSnapRAWWrite(STsdbSnapRAWWriter* writer, SSnapDataHdr* hdr) { - ASSERT(hdr->type == SNAP_DATA_RAW); - int32_t code = 0; int32_t lino = 0; diff --git a/source/dnode/vnode/src/tsdb/tsdbUtil.c b/source/dnode/vnode/src/tsdb/tsdbUtil.c index 667d7ffb7c..8820a026e9 100644 --- a/source/dnode/vnode/src/tsdb/tsdbUtil.c +++ b/source/dnode/vnode/src/tsdb/tsdbUtil.c @@ -106,7 +106,6 @@ _exit: #endif void tMapDataGetItemByIdx(SMapData *pMapData, int32_t idx, void *pItem, int32_t (*tGetItemFn)(uint8_t *, void *)) { - ASSERT(idx >= 0 && idx < pMapData->nItem); (void)tGetItemFn(pMapData->pData + pMapData->aOffset[idx], pItem); } @@ -404,8 +403,6 @@ static const int32_t BLOCK_WITH_ALG_VER = 2; int32_t tPutBlockCol(SBuffer *buffer, const SBlockCol *pBlockCol, int32_t ver, uint32_t defaultCmprAlg) { int32_t code; - ASSERT(pBlockCol->flag && (pBlockCol->flag != HAS_NONE)); - if ((code = tBufferPutI16v(buffer, pBlockCol->cid))) return code; if ((code = tBufferPutI8(buffer, pBlockCol->type))) return code; if ((code = tBufferPutI8(buffer, pBlockCol->cflag))) return code; @@ -444,8 +441,6 @@ int32_t tGetBlockCol(SBufferReader *br, SBlockCol *pBlockCol, int32_t ver, uint3 if ((code = tBufferGetI8(br, &pBlockCol->flag))) return code; if ((code = tBufferGetI32v(br, &pBlockCol->szOrigin))) return code; - ASSERT(pBlockCol->flag && (pBlockCol->flag != HAS_NONE)); - pBlockCol->szBitmap = 0; pBlockCol->szOffset = 0; pBlockCol->szValue = 0; @@ -584,7 +579,8 @@ int32_t tsdbFidLevel(int32_t fid, STsdbKeepCfg *pKeepCfg, int64_t nowSec) { } else if (pKeepCfg->precision == TSDB_TIME_PRECISION_NANO) { nowSec = nowSec * 1000000000l; } else { - ASSERT(0); + tsdbError("invalid time precision:%d", pKeepCfg->precision); + return 0; } nowSec = nowSec - pKeepCfg->keepTimeOffset * tsTickPerHour[pKeepCfg->precision]; @@ -628,8 +624,6 @@ void tsdbRowGetColVal(TSDBROW *pRow, STSchema *pTSchema, int32_t iCol, SColVal * *pColVal = COL_VAL_NONE(pTColumn->colId, pTColumn->type); } } - } else { - ASSERT(0); } } @@ -649,7 +643,6 @@ void tColRowGetPrimaryKey(SBlockData *pBlock, int32_t irow, SRowKey *key) { if (pColData->cflag & COL_IS_KEY) { SColVal cv; tColDataGetValue(pColData, irow, &cv); - ASSERT(COL_VAL_IS_VALUE(&cv)); key->pks[key->numOfPKs] = cv.value; key->numOfPKs++; } else { @@ -697,8 +690,6 @@ int32_t tsdbRowIterOpen(STSDBRowIter *pIter, TSDBROW *pRow, STSchema *pTSchema) if (code) return code; } else if (pRow->type == TSDBROW_COL_FMT) { pIter->iColData = 0; - } else { - ASSERT(0); } return 0; @@ -730,8 +721,8 @@ SColVal *tsdbRowIterNext(STSDBRowIter *pIter) { return NULL; } } else { - ASSERT(0); - return NULL; // suppress error report by compiler + tsdbError("invalid row type:%d", pIter->pRow->type); + return NULL; } } @@ -752,8 +743,6 @@ int32_t tsdbRowMergerAdd(SRowMerger *pMerger, TSDBROW *pRow, STSchema *pTSchema) jCol = 0; pTColumn = &pTSchema->columns[jCol++]; - ASSERT(pTColumn->type == TSDB_DATA_TYPE_TIMESTAMP); - *pColVal = COL_VAL_VALUE(pTColumn->colId, ((SValue){.type = pTColumn->type, .val = key.ts})); if (taosArrayPush(pMerger->pArray, pColVal) == NULL) { code = terrno; @@ -804,8 +793,6 @@ int32_t tsdbRowMergerAdd(SRowMerger *pMerger, TSDBROW *pRow, STSchema *pTSchema) pMerger->version = key.version; return 0; } else { - ASSERT(((SColVal *)pMerger->pArray->pData)->value.val == key.ts); - for (iCol = 1; iCol < pMerger->pTSchema->numOfCols && jCol < pTSchema->numOfCols; ++iCol) { pTColumn = &pMerger->pTSchema->columns[iCol]; if (pTSchema->columns[jCol].colId < pTColumn->colId) { @@ -856,7 +843,7 @@ int32_t tsdbRowMergerAdd(SRowMerger *pMerger, TSDBROW *pRow, STSchema *pTSchema) } } } else { - ASSERT(0 && "dup versions not allowed"); + return TSDB_CODE_INVALID_PARA; } } @@ -902,76 +889,6 @@ int32_t tsdbRowMergerGetRow(SRowMerger *pMerger, SRow **ppRow) { return tRowBuild(pMerger->pArray, pMerger->pTSchema, ppRow); } -/* -// delete skyline ====================================================== -static int32_t tsdbMergeSkyline2(SArray *aSkyline1, SArray *aSkyline2, SArray *aSkyline) { - int32_t code = 0; - int32_t i1 = 0; - int32_t n1 = taosArrayGetSize(aSkyline1); - int32_t i2 = 0; - int32_t n2 = taosArrayGetSize(aSkyline2); - TSDBKEY *pSkyline1; - TSDBKEY *pSkyline2; - TSDBKEY item; - int64_t version1 = 0; - int64_t version2 = 0; - - ASSERT(n1 > 0 && n2 > 0); - - taosArrayClear(aSkyline); - - while (i1 < n1 && i2 < n2) { - pSkyline1 = (TSDBKEY *)taosArrayGet(aSkyline1, i1); - pSkyline2 = (TSDBKEY *)taosArrayGet(aSkyline2, i2); - - if (pSkyline1->ts < pSkyline2->ts) { - version1 = pSkyline1->version; - i1++; - } else if (pSkyline1->ts > pSkyline2->ts) { - version2 = pSkyline2->version; - i2++; - } else { - version1 = pSkyline1->version; - version2 = pSkyline2->version; - i1++; - i2++; - } - - item.ts = TMIN(pSkyline1->ts, pSkyline2->ts); - item.version = TMAX(version1, version2); - if (taosArrayPush(aSkyline, &item) == NULL) { - code = TSDB_CODE_OUT_OF_MEMORY; - goto _exit; - } - } - - while (i1 < n1) { - pSkyline1 = (TSDBKEY *)taosArrayGet(aSkyline1, i1); - item.ts = pSkyline1->ts; - item.version = pSkyline1->version; - if (taosArrayPush(aSkyline, &item) == NULL) { - code = TSDB_CODE_OUT_OF_MEMORY; - goto _exit; - } - i1++; - } - - while (i2 < n2) { - pSkyline2 = (TSDBKEY *)taosArrayGet(aSkyline2, i2); - item.ts = pSkyline2->ts; - item.version = pSkyline2->version; - if (taosArrayPush(aSkyline, &item) == NULL) { - code = TSDB_CODE_OUT_OF_MEMORY; - goto _exit; - } - i2++; - } - -_exit: - return code; -} -*/ - // delete skyline ====================================================== static void tsdbMergeSkyline(SArray *pSkyline1, SArray *pSkyline2, SArray *pSkyline) { int32_t i1 = 0; @@ -983,8 +900,6 @@ static void tsdbMergeSkyline(SArray *pSkyline1, SArray *pSkyline2, SArray *pSkyl int64_t version1 = 0; int64_t version2 = 0; - ASSERT(n1 > 0 && n2 > 0); - taosArrayClear(pSkyline); TSDBKEY **pItem = TARRAY_GET_ELEM(pSkyline, 0); @@ -1217,7 +1132,9 @@ _exit: int32_t tBlockDataInit(SBlockData *pBlockData, TABLEID *pId, STSchema *pTSchema, int16_t *aCid, int32_t nCid) { int32_t code = 0; - ASSERT(pId->suid || pId->uid); + if (!pId->suid && !pId->uid) { + return TSDB_CODE_INVALID_PARA; + } pBlockData->suid = pId->suid; pBlockData->uid = pId->uid; @@ -1238,7 +1155,9 @@ int32_t tBlockDataInit(SBlockData *pBlockData, TABLEID *pId, STSchema *pTSchema, while (pTColumn->colId < aCid[iCid]) { iColumn++; - ASSERT(iColumn < pTSchema->numOfCols); + if (!(iColumn < pTSchema->numOfCols)) { + return TSDB_CODE_INVALID_PARA; + } pTColumn = &pTSchema->columns[iColumn]; } @@ -1277,8 +1196,6 @@ void tBlockDataReset(SBlockData *pBlockData) { } void tBlockDataClear(SBlockData *pBlockData) { - ASSERT(pBlockData->suid || pBlockData->uid); - pBlockData->nRow = 0; for (int32_t iColData = 0; iColData < pBlockData->nColData; iColData++) { tColDataClear(tBlockDataGetColDataByIdx(pBlockData, iColData)); @@ -1286,7 +1203,9 @@ void tBlockDataClear(SBlockData *pBlockData) { } int32_t tBlockDataAddColData(SBlockData *pBlockData, int16_t cid, int8_t type, int8_t cflag, SColData **ppColData) { - ASSERT(pBlockData->nColData == 0 || pBlockData->aColData[pBlockData->nColData - 1].cid < cid); + if (pBlockData->nColData != 0 && pBlockData->aColData[pBlockData->nColData - 1].cid >= cid) { + return TSDB_CODE_INVALID_PARA; + } SColData *newColData = taosMemoryRealloc(pBlockData->aColData, sizeof(SColData) * (pBlockData->nColData + 1)); if (newColData == NULL) { @@ -1346,11 +1265,15 @@ _exit: int32_t tBlockDataAppendRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTSchema, int64_t uid) { int32_t code = 0; - ASSERT(pBlockData->suid || pBlockData->uid); + if (!(pBlockData->suid || pBlockData->uid)) { + return TSDB_CODE_INVALID_PARA; + } // uid if (pBlockData->uid == 0) { - ASSERT(uid); + if (!uid) { + return TSDB_CODE_INVALID_PARA; + } code = tRealloc((uint8_t **)&pBlockData->aUid, sizeof(int64_t) * (pBlockData->nRow + 1)); if (code) goto _exit; pBlockData->aUid[pBlockData->nRow] = uid; @@ -1371,7 +1294,7 @@ int32_t tBlockDataAppendRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS code = tBlockDataUpsertBlockRow(pBlockData, pRow->pBlockData, pRow->iRow, 0 /* append */); if (code) goto _exit; } else { - ASSERT(0); + return TSDB_CODE_INVALID_PARA; } pBlockData->nRow++; @@ -1384,7 +1307,9 @@ int32_t tBlockDataUpdateRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS // version int64_t lversion = pBlockData->aVersion[pBlockData->nRow - 1]; int64_t rversion = TSDBROW_VERSION(pRow); - ASSERT(lversion != rversion); + if (lversion == rversion) { + return TSDB_CODE_INVALID_PARA; + } if (rversion > lversion) { pBlockData->aVersion[pBlockData->nRow - 1] = rversion; } @@ -1398,7 +1323,8 @@ int32_t tBlockDataUpdateRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS code = tBlockDataUpsertBlockRow(pBlockData, pRow->pBlockData, pRow->iRow, (rversion > lversion) ? 1 : -1); if (code) goto _exit; } else { - ASSERT(0); + code = TSDB_CODE_INVALID_PARA; + goto _exit; } _exit: @@ -1426,7 +1352,6 @@ int32_t tBlockDataUpsertRow(SBlockData *pBlockData, TSDBROW *pRow, STSchema *pTS #endif SColData *tBlockDataGetColData(SBlockData *pBlockData, int16_t cid) { - ASSERT(cid != PRIMARYKEY_TIMESTAMP_COL_ID); int32_t lidx = 0; int32_t ridx = pBlockData->nColData - 1; @@ -1696,7 +1621,9 @@ static int32_t tBlockDataCompressKeyPart(SBlockData *bData, SDiskDataHdr *hdr, S // primary keys for (hdr->numOfPKs = 0; hdr->numOfPKs < bData->nColData; hdr->numOfPKs++) { - ASSERT(hdr->numOfPKs <= TD_MAX_PK_COLS); + if (!(hdr->numOfPKs <= TD_MAX_PK_COLS)) { + return TSDB_CODE_INVALID_PARA; + } SBlockCol *blockCol = &hdr->primaryBlockCols[hdr->numOfPKs]; SColData *colData = tBlockDataGetColDataByIdx(bData, hdr->numOfPKs); @@ -1745,8 +1672,6 @@ int32_t tBlockDataDecompressColData(const SDiskDataHdr *hdr, const SBlockCol *bl code = tBlockDataAddColData(blockData, blockCol->cid, blockCol->type, blockCol->cflag, &colData); TSDB_CHECK_CODE(code, lino, _exit); - // ASSERT(blockCol->flag != HAS_NONE); - SColDataCompressInfo info = { .cmprAlg = blockCol->alg, .columnFlag = blockCol->cflag, @@ -1833,8 +1758,12 @@ int32_t tBlockDataDecompressKeyPart(const SDiskDataHdr *hdr, SBufferReader *br, for (int i = 0; i < hdr->numOfPKs; i++) { const SBlockCol *blockCol = &hdr->primaryBlockCols[i]; - ASSERT(blockCol->flag == HAS_VALUE); - ASSERT(blockCol->cflag & COL_IS_KEY); + if (!(blockCol->flag == HAS_VALUE)) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } + if (!(blockCol->cflag & COL_IS_KEY)) { + TSDB_CHECK_CODE(code = TSDB_CODE_FILE_CORRUPTED, lino, _exit); + } code = tBlockDataDecompressColData(hdr, blockCol, br, blockData, assist); TSDB_CHECK_CODE(code, lino, _exit); diff --git a/source/dnode/vnode/src/tsdb/tsdbUtil2.c b/source/dnode/vnode/src/tsdb/tsdbUtil2.c index e13e520cbf..d87a05bcf4 100644 --- a/source/dnode/vnode/src/tsdb/tsdbUtil2.c +++ b/source/dnode/vnode/src/tsdb/tsdbUtil2.c @@ -276,7 +276,9 @@ int32_t tBrinBlockClear(SBrinBlock *brinBlock) { } int32_t tBrinBlockPut(SBrinBlock *brinBlock, const SBrinRecord *record) { - ASSERT(record->firstKey.key.numOfPKs == record->lastKey.key.numOfPKs); + if (record->firstKey.key.numOfPKs != record->lastKey.key.numOfPKs) { + return TSDB_CODE_INVALID_PARA; + } if (brinBlock->numOfRecords == 0) { // the first row brinBlock->numOfPKs = record->firstKey.key.numOfPKs; diff --git a/source/dnode/vnode/src/vnd/vnodeBufPool.c b/source/dnode/vnode/src/vnd/vnodeBufPool.c index cbd6fbbe52..2071125175 100644 --- a/source/dnode/vnode/src/vnd/vnodeBufPool.c +++ b/source/dnode/vnode/src/vnd/vnodeBufPool.c @@ -103,17 +103,18 @@ int vnodeCloseBufPool(SVnode *pVnode) { } void vnodeBufPoolReset(SVBufPool *pPool) { - ASSERT(pPool->nQuery == 0); + if (pPool->nQuery != 0) { + vError("vgId:%d, buffer pool %p of id %d has %d queries, reset it may cause problem", TD_VID(pPool->pVnode), pPool, + pPool->id, pPool->nQuery); + } + for (SVBufPoolNode *pNode = pPool->pTail; pNode->prev; pNode = pPool->pTail) { - ASSERT(pNode->pnext == &pPool->pTail); pNode->prev->pnext = &pPool->pTail; pPool->pTail = pNode->prev; pPool->size = pPool->size - sizeof(*pNode) - pNode->size; taosMemoryFree(pNode); } - ASSERT(pPool->size == pPool->ptr - pPool->node.data); - pPool->size = 0; pPool->ptr = pPool->node.data; } @@ -123,7 +124,11 @@ void *vnodeBufPoolMallocAligned(SVBufPool *pPool, int size) { void *p = NULL; uint8_t *ptr = NULL; int paddingLen = 0; - ASSERT(pPool != NULL); + + if (pPool == NULL) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } if (pPool->lock) taosThreadSpinLock(pPool->lock); @@ -162,7 +167,11 @@ void *vnodeBufPoolMallocAligned(SVBufPool *pPool, int size) { void *vnodeBufPoolMalloc(SVBufPool *pPool, int size) { SVBufPoolNode *pNode; void *p = NULL; - ASSERT(pPool != NULL); + + if (pPool == NULL) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } if (pPool->lock) taosThreadSpinLock(pPool->lock); if (pPool->node.size >= pPool->ptr - pPool->node.data + size) { @@ -207,7 +216,9 @@ void vnodeBufPoolFree(SVBufPool *pPool, void *p) { void vnodeBufPoolRef(SVBufPool *pPool) { int32_t nRef = atomic_fetch_add_32(&pPool->nRef, 1); - ASSERT(nRef > 0); + if (nRef <= 0) { + vError("vgId:%d, buffer pool %p of id %d is referenced by %d", TD_VID(pPool->pVnode), pPool, pPool->id, nRef); + } } void vnodeBufPoolAddToFreeList(SVBufPool *pPool) { diff --git a/source/dnode/vnode/src/vnd/vnodeCfg.c b/source/dnode/vnode/src/vnd/vnodeCfg.c index e2791d8a00..d90badc34c 100644 --- a/source/dnode/vnode/src/vnd/vnodeCfg.c +++ b/source/dnode/vnode/src/vnd/vnodeCfg.c @@ -248,7 +248,6 @@ int vnodeDecodeConfig(const SJson *pJson, void *pObj) { } for (int32_t i = 0; i < nRetention; ++i) { SJson *pNodeRetention = tjsonGetArrayItem(pNodeRetentions, i); - ASSERT(pNodeRetention != NULL); tjsonGetNumberValue(pNodeRetention, "freq", (pCfg->tsdbCfg.retentions)[i].freq, code); if (code) return code; tjsonGetNumberValue(pNodeRetention, "freqUnit", (pCfg->tsdbCfg.retentions)[i].freqUnit, code); diff --git a/source/dnode/vnode/src/vnd/vnodeModule.c b/source/dnode/vnode/src/vnd/vnodeModule.c index 709bfa19bc..2a2d01ada8 100644 --- a/source/dnode/vnode/src/vnd/vnodeModule.c +++ b/source/dnode/vnode/src/vnd/vnodeModule.c @@ -14,6 +14,7 @@ */ #include "cos.h" +#include "monitor.h" #include "vnd.h" static volatile int32_t VINIT = 0; @@ -26,6 +27,8 @@ int vnodeInit(int nthreads, StopDnodeFp stopDnodeFp) { TAOS_CHECK_RETURN(vnodeAsyncOpen(nthreads)); TAOS_CHECK_RETURN(walInit(stopDnodeFp)); + monInitVnode(); + return 0; } diff --git a/source/dnode/vnode/src/vnd/vnodeOpen.c b/source/dnode/vnode/src/vnd/vnodeOpen.c index 989faa3a0f..2964306092 100644 --- a/source/dnode/vnode/src/vnd/vnodeOpen.c +++ b/source/dnode/vnode/src/vnd/vnodeOpen.c @@ -494,24 +494,6 @@ SVnode *vnodeOpen(const char *path, int32_t diskPrimary, STfs *pTfs, SMsgCb msgC snprintf(pVnode->monitor.strDnodeId, TSDB_NODE_ID_LEN, "%" PRId32, pVnode->config.syncCfg.nodeInfo[0].nodeId); snprintf(pVnode->monitor.strVgId, TSDB_VGROUP_ID_LEN, "%" PRId32, pVnode->config.vgId); - if (tsEnableMonitor && pVnode->monitor.insertCounter == NULL) { - taos_counter_t *counter = NULL; - int32_t label_count = 7; - const char *sample_labels[] = {VNODE_METRIC_TAG_NAME_SQL_TYPE, VNODE_METRIC_TAG_NAME_CLUSTER_ID, - VNODE_METRIC_TAG_NAME_DNODE_ID, VNODE_METRIC_TAG_NAME_DNODE_EP, - VNODE_METRIC_TAG_NAME_VGROUP_ID, VNODE_METRIC_TAG_NAME_USERNAME, - VNODE_METRIC_TAG_NAME_RESULT}; - counter = taos_counter_new(VNODE_METRIC_SQL_COUNT, "counter for insert sql", label_count, sample_labels); - vInfo("vgId:%d, new metric:%p", TD_VID(pVnode), counter); - if (taos_collector_registry_register_metric(counter) == 1) { - (void)taos_counter_destroy(counter); - counter = taos_collector_registry_get_metric(VNODE_METRIC_SQL_COUNT); - vInfo("vgId:%d, get metric from registry:%p", TD_VID(pVnode), counter); - } - pVnode->monitor.insertCounter = counter; - vInfo("vgId:%d, succeed to set metric:%p", TD_VID(pVnode), counter); - } - return pVnode; _err: diff --git a/source/dnode/vnode/src/vnd/vnodeQuery.c b/source/dnode/vnode/src/vnd/vnodeQuery.c index 904b29bf43..e080463fc8 100644 --- a/source/dnode/vnode/src/vnd/vnodeQuery.c +++ b/source/dnode/vnode/src/vnd/vnodeQuery.c @@ -19,7 +19,6 @@ #define VNODE_GET_LOAD_RESET_VALS(pVar, oVal, vType, tags) \ do { \ int##vType##_t newVal = atomic_sub_fetch_##vType(&(pVar), (oVal)); \ - ASSERT(newVal >= 0); \ if (newVal < 0) { \ vWarn("vgId:%d, %s, abnormal val:%" PRIi64 ", old val:%" PRIi64, TD_VID(pVnode), tags, newVal, (oVal)); \ } \ @@ -37,7 +36,10 @@ int32_t fillTableColCmpr(SMetaReader *reader, SSchemaExt *pExt, int32_t numOfCol int8_t tblType = reader->me.type; if (useCompress(tblType)) { SColCmprWrapper *p = &(reader->me.colCmpr); - ASSERT(numOfCol == p->nCols); + if (numOfCol != p->nCols) { + vError("fillTableColCmpr table type:%d, col num:%d, col cmpr num:%d mismatch", tblType, numOfCol, p->nCols); + return TSDB_CODE_APP_ERROR; + } for (int i = 0; i < p->nCols; i++) { SColCmpr *pCmpr = &p->pColCmpr[i]; pExt[i].colId = pCmpr->id; @@ -104,7 +106,8 @@ int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) { } else if (mer1.me.type == TSDB_NORMAL_TABLE) { schema = mer1.me.ntbEntry.schemaRow; } else { - ASSERT(0); + vError("vnodeGetTableMeta get invalid table type:%d", mer1.me.type); + return TSDB_CODE_APP_ERROR; } metaRsp.numOfTags = schemaTag.nCols; @@ -126,7 +129,6 @@ int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) { SMetaReader *pReader = mer1.me.type == TSDB_CHILD_TABLE ? &mer2 : &mer1; code = fillTableColCmpr(pReader, metaRsp.pSchemaExt, metaRsp.numOfColumns); if (code < 0) { - code = TSDB_CODE_INVALID_MSG; goto _exit; } } else { @@ -262,7 +264,8 @@ int32_t vnodeGetTableCfg(SVnode *pVnode, SRpcMsg *pMsg, bool direct) { } } } else { - ASSERT(0); + vError("vnodeGetTableCfg get invalid table type:%d", mer1.me.type); + return TSDB_CODE_APP_ERROR; } cfgRsp.numOfTags = schemaTag.nCols; @@ -541,7 +544,7 @@ int32_t vnodeGetTableList(void *pVnode, int8_t type, SArray *pList) { } int32_t vnodeGetAllTableList(SVnode *pVnode, uint64_t uid, SArray *list) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; SMCtbCursor *pCur = metaOpenCtbCursor(pVnode, uid, 1); if (NULL == pCur) { qError("vnode get all table list failed"); @@ -571,7 +574,7 @@ int32_t vnodeGetCtbIdListByFilter(SVnode *pVnode, int64_t suid, SArray *list, bo } int32_t vnodeGetCtbIdList(void *pVnode, int64_t suid, SArray *list) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; SVnode *pVnodeObj = pVnode; SMCtbCursor *pCur = metaOpenCtbCursor(pVnodeObj, suid, 1); if (NULL == pCur) { @@ -585,7 +588,7 @@ int32_t vnodeGetCtbIdList(void *pVnode, int64_t suid, SArray *list) { break; } - if (NULL == taosArrayPush(list, &id)) { + if (NULL == taosArrayPush(list, &id)) { qError("taosArrayPush failed"); code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; @@ -598,7 +601,7 @@ _exit: } int32_t vnodeGetStbIdList(SVnode *pVnode, int64_t suid, SArray *list) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; SMStbCursor *pCur = metaOpenStbCursor(pVnode->pMeta, suid); if (!pCur) { return TSDB_CODE_OUT_OF_MEMORY; @@ -610,7 +613,7 @@ int32_t vnodeGetStbIdList(SVnode *pVnode, int64_t suid, SArray *list) { break; } - if (NULL == taosArrayPush(list, &id)) { + if (NULL == taosArrayPush(list, &id)) { qError("taosArrayPush failed"); code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; @@ -624,7 +627,7 @@ _exit: int32_t vnodeGetStbIdListByFilter(SVnode *pVnode, int64_t suid, SArray *list, bool (*filter)(void *arg, void *arg1), void *arg) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; SMStbCursor *pCur = metaOpenStbCursor(pVnode->pMeta, suid); if (!pCur) { return terrno; @@ -640,7 +643,7 @@ int32_t vnodeGetStbIdListByFilter(SVnode *pVnode, int64_t suid, SArray *list, bo continue; } - if (NULL == taosArrayPush(list, &id)) { + if (NULL == taosArrayPush(list, &id)) { qError("taosArrayPush failed"); code = TSDB_CODE_OUT_OF_MEMORY; goto _exit; diff --git a/source/dnode/vnode/src/vnd/vnodeSnapshot.c b/source/dnode/vnode/src/vnd/vnodeSnapshot.c index 28e7ae97ca..9a5b98e31b 100644 --- a/source/dnode/vnode/src/vnd/vnodeSnapshot.c +++ b/source/dnode/vnode/src/vnd/vnodeSnapshot.c @@ -73,7 +73,10 @@ struct SVSnapReader { }; static TFileSetRangeArray **vnodeSnapReaderGetTsdbRanges(SVSnapReader *pReader, int32_t tsdbTyp) { - ASSERTS(sizeof(pReader->pRsmaRanges) / sizeof(pReader->pRsmaRanges[0]) == 2, "Unexpected array size"); + if (!(sizeof(pReader->pRsmaRanges) / sizeof(pReader->pRsmaRanges[0]) == 2)) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } switch (tsdbTyp) { case SNAP_DATA_TSDB: return &pReader->pRanges; @@ -147,7 +150,6 @@ static int32_t vnodeSnapReaderDealWithSnapInfo(SVSnapReader *pReader, SSnapshotP pReader->tsdbRAWDone = true; } - ASSERT(pReader->tsdbDone != pReader->tsdbRAWDone); vInfo("vgId:%d, vnode snap writer enabled replication mode: %s", TD_VID(pVnode), (pReader->tsdbDone ? "raw" : "normal")); } @@ -177,7 +179,6 @@ int32_t vnodeSnapReaderOpen(SVnode *pVnode, SSnapshotParam *pParam, SVSnapReader // open tsdb snapshot raw reader if (!pReader->tsdbRAWDone) { - ASSERT(pReader->sver == 0); code = tsdbSnapRAWReaderOpen(pVnode->pTsdb, ever, SNAP_DATA_RAW, &pReader->pTsdbRAWReader); if (code) goto _exit; } @@ -339,7 +340,6 @@ int32_t vnodeSnapRead(SVSnapReader *pReader, uint8_t **ppData, uint32_t *nData) if (!pReader->tsdbRAWDone) { // open if not if (pReader->pTsdbRAWReader == NULL) { - ASSERT(pReader->sver == 0); code = tsdbSnapRAWReaderOpen(pReader->pVnode->pTsdb, pReader->ever, SNAP_DATA_RAW, &pReader->pTsdbRAWReader); TSDB_CHECK_CODE(code, lino, _exit); } @@ -518,7 +518,6 @@ struct SVSnapWriter { }; TFileSetRangeArray **vnodeSnapWriterGetTsdbRanges(SVSnapWriter *pWriter, int32_t tsdbTyp) { - ASSERTS(sizeof(pWriter->pRsmaRanges) / sizeof(pWriter->pRsmaRanges[0]) == 2, "Unexpected array size"); switch (tsdbTyp) { case SNAP_DATA_TSDB: return &pWriter->pRanges; @@ -674,7 +673,6 @@ int32_t vnodeSnapWriterClose(SVSnapWriter *pWriter, int8_t rollback, SSnapshot * // commit json if (!rollback) { - ASSERT(pVnode->config.vgId == pWriter->info.config.vgId); pWriter->info.state.committed = pWriter->ever; pVnode->config = pWriter->info.config; pVnode->state = (SVState){.committed = pWriter->info.state.committed, @@ -794,7 +792,9 @@ int32_t vnodeSnapWrite(SVSnapWriter *pWriter, uint8_t *pData, uint32_t nData) { SSnapDataHdr *pHdr = (SSnapDataHdr *)pData; SVnode *pVnode = pWriter->pVnode; - ASSERT(pHdr->size + sizeof(SSnapDataHdr) == nData); + if (!(pHdr->size + sizeof(SSnapDataHdr) == nData)) { + return TSDB_CODE_INVALID_PARA; + } if (pHdr->index != pWriter->index + 1) { vError("vgId:%d, unexpected vnode snapshot msg. index:%" PRId64 ", expected index:%" PRId64, TD_VID(pVnode), @@ -837,7 +837,6 @@ int32_t vnodeSnapWrite(SVSnapWriter *pWriter, uint8_t *pData, uint32_t nData) { case SNAP_DATA_RAW: { // tsdb if (pWriter->pTsdbSnapRAWWriter == NULL) { - ASSERT(pWriter->sver == 0); code = tsdbSnapRAWWriterOpen(pVnode->pTsdb, pWriter->ever, &pWriter->pTsdbSnapRAWWriter); TSDB_CHECK_CODE(code, lino, _exit); } diff --git a/source/dnode/vnode/src/vnd/vnodeSvr.c b/source/dnode/vnode/src/vnd/vnodeSvr.c index dc84b73c10..75db6e2925 100644 --- a/source/dnode/vnode/src/vnd/vnodeSvr.c +++ b/source/dnode/vnode/src/vnd/vnodeSvr.c @@ -15,6 +15,7 @@ #include "audit.h" #include "cos.h" +#include "monitor.h" #include "tencode.h" #include "tglobal.h" #include "tmsg.h" @@ -23,6 +24,8 @@ #include "vnode.h" #include "vnodeInt.h" +extern taos_counter_t *tsInsertCounter; + static int32_t vnodeProcessCreateStbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp); static int32_t vnodeProcessAlterStbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp); static int32_t vnodeProcessDropStbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp); @@ -537,8 +540,13 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t ver, SRpcMsg TD_VID(pVnode), TMSG_INFO(pMsg->msgType), ver, pVnode->state.applied, pVnode->state.applyTerm, pMsg->info.conn.applyTerm); - ASSERT(pVnode->state.applyTerm <= pMsg->info.conn.applyTerm); - ASSERTS(pVnode->state.applied + 1 == ver, "applied:%" PRId64 ", ver:%" PRId64, pVnode->state.applied, ver); + if (!(pVnode->state.applyTerm <= pMsg->info.conn.applyTerm)) { + return terrno = TSDB_CODE_INTERNAL_ERROR; + } + + if (!(pVnode->state.applied + 1 == ver)) { + return terrno = TSDB_CODE_INTERNAL_ERROR; + } atomic_store_64(&pVnode->state.applied, ver); atomic_store_64(&pVnode->state.applyTerm, pMsg->info.conn.applyTerm); @@ -934,20 +942,23 @@ _exit: } static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp) { + int ret = 0; SVDropTtlTableReq ttlReq = {0}; if (tDeserializeSVDropTtlTableReq(pReq, len, &ttlReq) != 0) { - terrno = TSDB_CODE_INVALID_MSG; + ret = TSDB_CODE_INVALID_MSG; goto end; } - ASSERT(ttlReq.nUids == taosArrayGetSize(ttlReq.pTbUids)); + if (ttlReq.nUids != taosArrayGetSize(ttlReq.pTbUids)) { + ret = TSDB_CODE_INVALID_MSG; + goto end; + } if (ttlReq.nUids != 0) { vInfo("vgId:%d, drop ttl table req will be processed, time:%d, ntbUids:%d", pVnode->config.vgId, ttlReq.timestampSec, ttlReq.nUids); } - int ret = 0; if (ttlReq.nUids > 0) { metaDropTables(pVnode->pMeta, ttlReq.pTbUids); (void)tqUpdateTbUidList(pVnode->pTq, ttlReq.pTbUids, false); @@ -975,7 +986,10 @@ static int32_t vnodeProcessFetchTtlExpiredTbs(SVnode *pVnode, int64_t ver, void goto _end; } - ASSERT(ttlReq.nUids == taosArrayGetSize(ttlReq.pTbUids)); + if (!(ttlReq.nUids == taosArrayGetSize(ttlReq.pTbUids))) { + terrno = TSDB_CODE_INVALID_MSG; + goto _end; + } tb_uid_t suid; char ctbName[TSDB_TABLE_NAME_LEN]; @@ -1787,8 +1801,7 @@ static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t ver, void *pReq, in } if (info.suid) { - code = metaGetInfo(pVnode->pMeta, info.suid, &info, NULL); - ASSERT(code == 0); + (void)metaGetInfo(pVnode->pMeta, info.suid, &info, NULL); } if (pSubmitTbData->sver != info.skmVer) { @@ -1901,7 +1914,8 @@ _exit: (void)atomic_add_fetch_64(&pVnode->statis.nInsertSuccess, pSubmitRsp->affectedRows); (void)atomic_add_fetch_64(&pVnode->statis.nBatchInsert, 1); - if (tsEnableMonitor && pSubmitRsp->affectedRows > 0 && strlen(pOriginalMsg->info.conn.user) > 0) { + if (tsEnableMonitor && tsMonitorFqdn[0] != 0 && tsMonitorPort != 0 && pSubmitRsp->affectedRows > 0 && + strlen(pOriginalMsg->info.conn.user) > 0 && tsInsertCounter != NULL) { const char *sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT_AFFECTED_ROWS, pVnode->monitor.strClusterId, pVnode->monitor.strDnodeId, @@ -1909,7 +1923,7 @@ _exit: pVnode->monitor.strVgId, pOriginalMsg->info.conn.user, "Success"}; - (void)taos_counter_add(pVnode->monitor.insertCounter, pSubmitRsp->affectedRows, sample_labels); + (void)taos_counter_add(tsInsertCounter, pSubmitRsp->affectedRows, sample_labels); } if (code == 0) { diff --git a/source/dnode/vnode/src/vnd/vnodeSync.c b/source/dnode/vnode/src/vnd/vnodeSync.c index 31e44f5912..d8a29c8e73 100644 --- a/source/dnode/vnode/src/vnd/vnodeSync.c +++ b/source/dnode/vnode/src/vnd/vnodeSync.c @@ -117,7 +117,9 @@ static int32_t inline vnodeProposeMsg(SVnode *pVnode, SRpcMsg *pMsg, bool isWeak int32_t code = syncPropose(pVnode->sync, pMsg, isWeak, &seq); bool wait = (code == 0 && vnodeIsMsgBlock(pMsg->msgType)); if (wait) { - ASSERT(!pVnode->blocked); + if (pVnode->blocked) { + return TSDB_CODE_INTERNAL_ERROR; + } pVnode->blocked = true; pVnode->blockSec = taosGetTimestampSec(); pVnode->blockSeq = seq; @@ -175,7 +177,6 @@ static void inline vnodeProposeBatchMsg(SVnode *pVnode, SRpcMsg **pMsgArr, bool int32_t code = syncProposeBatch(pVnode->sync, pMsgArr, pIsWeakArr, *arrSize); bool wait = (code == 0 && vnodeIsBlockMsg(pLastMsg->msgType)); if (wait) { - ASSERT(!pVnode->blocked); pVnode->blocked = true; } (void)taosThreadMutexUnlock(&pVnode->lock); @@ -372,8 +373,8 @@ int32_t vnodeProcessSyncMsg(SVnode *pVnode, SRpcMsg *pMsg, SRpcMsg **pRsp) { int32_t code = syncProcessMsg(pVnode->sync, pMsg); if (code != 0) { - vGError("vgId:%d, failed to process sync msg:%p type:%s, errno: %s, code:0x%x", pVnode->config.vgId, pMsg, - TMSG_INFO(pMsg->msgType), terrstr(), code); + vGError("vgId:%d, failed to process sync msg:%p type:%s, reason: %s", pVnode->config.vgId, pMsg, + TMSG_INFO(pMsg->msgType), tstrerror(code)); } return code; @@ -543,7 +544,11 @@ static void vnodeRestoreFinish(const SSyncFSM *pFsm, const SyncIndex commitIdx) do { appliedIdx = vnodeSyncAppliedIndex(pFsm); - ASSERT(appliedIdx <= commitIdx); + if (appliedIdx > commitIdx) { + vError("vgId:%d, restore failed since applied-index:%" PRId64 " is larger than commit-index:%" PRId64, vgId, + appliedIdx, commitIdx); + break; + } if (appliedIdx == commitIdx) { vInfo("vgId:%d, no items to be applied, restore finish", pVnode->config.vgId); break; @@ -555,7 +560,6 @@ static void vnodeRestoreFinish(const SSyncFSM *pFsm, const SyncIndex commitIdx) } } while (true); - ASSERT(commitIdx == vnodeSyncAppliedIndex(pFsm)); (void)walApplyVer(pVnode->pWal, commitIdx); pVnode->restored = true; diff --git a/source/libs/catalog/inc/catalogInt.h b/source/libs/catalog/inc/catalogInt.h index f3b1852ce1..380e5224e1 100644 --- a/source/libs/catalog/inc/catalogInt.h +++ b/source/libs/catalog/inc/catalogInt.h @@ -21,11 +21,11 @@ extern "C" { #endif #include "catalog.h" +#include "os.h" #include "query.h" #include "tcommon.h" -#include "ttimer.h" #include "tglobal.h" -#include "os.h" +#include "ttimer.h" #define CTG_DEFAULT_CACHE_CLUSTER_NUMBER 6 #define CTG_DEFAULT_CACHE_VGROUP_NUMBER 100 @@ -40,7 +40,7 @@ extern "C" { #define CTG_DEFAULT_FETCH_NUM 8 #define CTG_MAX_COMMAND_LEN 512 #define CTG_DEFAULT_CACHE_MON_MSEC 5000 -#define CTG_CLEAR_CACHE_ROUND_TB_NUM 3000 +#define CTG_CLEAR_CACHE_ROUND_TB_NUM 3000 #define CTG_RENT_SLOT_SECOND 1.5 @@ -292,7 +292,7 @@ typedef struct SCtgTSMAFetch { typedef struct SCtgTbTSMACtx { int32_t fetchNum; - SArray* pNames; // SArray + SArray* pNames; // SArray SArray* pResList; SArray* pFetches; } SCtgTbTSMACtx; @@ -301,10 +301,10 @@ typedef STableIndexRsp STableIndex; typedef STableTSMAInfo STSMACache; typedef struct SCtgTbCache { - SRWLatch metaLock; - SRWLatch indexLock; - STableMeta* pMeta; - STableIndex* pIndex; + SRWLatch metaLock; + SRWLatch indexLock; + STableMeta* pMeta; + STableIndex* pIndex; } SCtgTbCache; typedef struct SCtgVgCache { @@ -318,13 +318,13 @@ typedef struct SCtgCfgCache { } SCtgCfgCache; typedef struct SCtgViewCache { - SRWLatch viewLock; - SViewMeta* pMeta; + SRWLatch viewLock; + SViewMeta* pMeta; } SCtgViewCache; typedef struct SCtgTSMACache { SRWLatch tsmaLock; - SArray* pTsmas; // SArray + SArray* pTsmas; // SArray } SCtgTSMACache; typedef struct SCtgDBCache { @@ -333,9 +333,9 @@ typedef struct SCtgDBCache { int8_t deleted; SCtgVgCache vgCache; SCtgCfgCache cfgCache; - SHashObj* viewCache; // key:viewname, value:SCtgViewCache - SHashObj* tbCache; // key:tbname, value:SCtgTbCache - SHashObj* stbCache; // key:suid, value:char* + SHashObj* viewCache; // key:viewname, value:SCtgViewCache + SHashObj* tbCache; // key:tbname, value:SCtgTbCache + SHashObj* stbCache; // key:suid, value:char* SHashObj* tsmaCache; // key:tbname, value: SCtgTSMACache int32_t tsmaVersion; uint64_t dbCacheNum[CTG_CI_MAX_VALUE]; @@ -365,7 +365,7 @@ typedef struct SCtgUserAuth { } SCtgUserAuth; typedef struct SCatalog { - int64_t clusterId; + int64_t clusterId; bool stopUpdate; SDynViewVersion dynViewVer; SHashObj* userCache; // key:user, value:SCtgUserAuth @@ -420,7 +420,7 @@ typedef struct SCtgJob { int32_t svrVerNum; int32_t viewNum; int32_t tbTsmaNum; - int32_t tsmaNum; // currently, only 1 is possible + int32_t tsmaNum; // currently, only 1 is possible } SCtgJob; typedef struct SCtgMsgCtx { @@ -587,8 +587,8 @@ typedef struct SCtgUpdateEpsetMsg { } SCtgUpdateEpsetMsg; typedef struct SCtgUpdateViewMetaMsg { - SCatalog* pCtg; - SViewMetaRsp* pRsp; + SCatalog* pCtg; + SViewMetaRsp* pRsp; } SCtgUpdateViewMetaMsg; typedef struct SCtgDropViewMetaMsg { @@ -618,7 +618,6 @@ typedef struct SCtgDropTbTSMAMsg { bool dropAllForTb; } SCtgDropTbTSMAMsg; - typedef struct SCtgCacheOperation { int32_t opId; void* data; @@ -647,7 +646,7 @@ typedef struct SCatalogMgmt { int32_t jobPool; SRWLatch lock; SCtgQueue queue; - void *timer; + void* timer; tmr_h cacheTimer; TdThread updateThread; SHashObj* pCluster; // key: clusterId, value: SCatalog* @@ -665,8 +664,8 @@ typedef struct SCtgOperation { } SCtgOperation; typedef struct SCtgCacheItemInfo { - char* name; - int32_t flag; + char* name; + int32_t flag; } SCtgCacheItemInfo; #define CTG_AUTH_READ(_t) ((_t) == AUTH_TYPE_READ || (_t) == AUTH_TYPE_READ_OR_WRITE) @@ -800,8 +799,8 @@ typedef struct SCtgCacheItemInfo { ((CTG_TASK_GET_TB_META_BATCH == (_taskType)) || (CTG_TASK_GET_TB_HASH_BATCH == (_taskType)) || \ (CTG_TASK_GET_VIEW == (_taskType)) || (CTG_TASK_GET_TB_TSMA == (_taskType))) -#define CTG_GET_TASK_MSGCTX(_task, _id) \ - (CTG_IS_BATCH_TASK((_task)->type) ? taosArrayGet((_task)->msgCtxs, (_id)) : &(_task)->msgCtx) +#define CTG_GET_TASK_MSGCTX(_task, _id) \ + (CTG_IS_BATCH_TASK((_task)->type) ? taosArrayGet((_task)->msgCtxs, (_id)) : &(_task)->msgCtx) #define CTG_META_SIZE(pMeta) \ (sizeof(STableMeta) + ((pMeta)->tableInfo.numOfTags + (pMeta)->tableInfo.numOfColumns) * sizeof(SSchema)) @@ -810,9 +809,8 @@ typedef struct SCtgCacheItemInfo { #define CTG_DB_NOT_EXIST(code) \ (code == TSDB_CODE_MND_DB_NOT_EXIST || code == TSDB_CODE_MND_DB_IN_CREATING || code == TSDB_CODE_MND_DB_IN_DROPPING) -#define CTG_CACHE_OVERFLOW(_csize, _maxsize) ((_maxsize >= 0) ? ((_csize) >= (_maxsize) * 1048576L * 0.9) : false) -#define CTG_CACHE_LOW(_csize, _maxsize) ((_maxsize >= 0) ? ((_csize) <= (_maxsize) * 1048576L * 0.75) : true) - +#define CTG_CACHE_OVERFLOW(_csize, _maxsize) ((_maxsize >= 0) ? ((_csize) >= (_maxsize)*1048576L * 0.9) : false) +#define CTG_CACHE_LOW(_csize, _maxsize) ((_maxsize >= 0) ? ((_csize) <= (_maxsize)*1048576L * 0.75) : true) #define ctgFatal(param, ...) qFatal("CTG:%p " param, pCtg, __VA_ARGS__) #define ctgError(param, ...) qError("CTG:%p " param, pCtg, __VA_ARGS__) @@ -821,12 +819,12 @@ typedef struct SCtgCacheItemInfo { #define ctgDebug(param, ...) qDebug("CTG:%p " param, pCtg, __VA_ARGS__) #define ctgTrace(param, ...) qTrace("CTG:%p " param, pCtg, __VA_ARGS__) -#define ctgTaskFatal(param, ...) qFatal("QID:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) -#define ctgTaskError(param, ...) qError("QID:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) -#define ctgTaskWarn(param, ...) qWarn("QID:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) -#define ctgTaskInfo(param, ...) qInfo("QID:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) -#define ctgTaskDebug(param, ...) qDebug("QID:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) -#define ctgTaskTrace(param, ...) qTrace("QID:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) +#define ctgTaskFatal(param, ...) qFatal("qid:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) +#define ctgTaskError(param, ...) qError("qid:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) +#define ctgTaskWarn(param, ...) qWarn("qid:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) +#define ctgTaskInfo(param, ...) qInfo("qid:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) +#define ctgTaskDebug(param, ...) qDebug("qid:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) +#define ctgTaskTrace(param, ...) qTrace("qid:%" PRIx64 " CTG:%p " param, pTask->pJob->queryId, pCtg, __VA_ARGS__) #define CTG_LOCK_DEBUG(...) \ do { \ @@ -849,38 +847,62 @@ typedef struct SCtgCacheItemInfo { #define TD_RWLATCH_WRITE_FLAG_COPY 0x40000000 -#define CTG_LOCK(type, _lock) \ - do { \ - if (CTG_READ == (type)) { \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value before read lock"); \ - CTG_LOCK_DEBUG("CTG RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosRLockLatch(_lock); \ - CTG_LOCK_DEBUG("CTG RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value after read lock"); \ - } else { \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value before write lock"); \ - CTG_LOCK_DEBUG("CTG WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosWLockLatch(_lock); \ - CTG_LOCK_DEBUG("CTG WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value after write lock"); \ - } \ +#define CTG_LOCK(type, _lock) \ + do { \ + if (CTG_READ == (type)) { \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value before read lock"); \ + break; \ + } \ + CTG_LOCK_DEBUG("CTG RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosRLockLatch(_lock); \ + CTG_LOCK_DEBUG("CTG RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) <= 0) { \ + qError("invalid lock value after read lock"); \ + break; \ + } \ + } else { \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value before write lock"); \ + break; \ + } \ + CTG_LOCK_DEBUG("CTG WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosWLockLatch(_lock); \ + CTG_LOCK_DEBUG("CTG WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \ + qError("invalid lock value after write lock"); \ + break; \ + } \ + } \ } while (0) -#define CTG_UNLOCK(type, _lock) \ - do { \ - if (CTG_READ == (type)) { \ - ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value before read unlock"); \ - CTG_LOCK_DEBUG("CTG RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosRUnLockLatch(_lock); \ - CTG_LOCK_DEBUG("CTG RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after read unlock"); \ - } else { \ - ASSERTS(atomic_load_32((_lock)) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value before write unlock"); \ - CTG_LOCK_DEBUG("CTG WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosWUnLockLatch(_lock); \ - CTG_LOCK_DEBUG("CTG WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after write unlock"); \ - } \ +#define CTG_UNLOCK(type, _lock) \ + do { \ + if (CTG_READ == (type)) { \ + if (atomic_load_32((_lock)) <= 0) { \ + qError("invalid lock value before read unlock"); \ + break; \ + } \ + CTG_LOCK_DEBUG("CTG RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosRUnLockLatch(_lock); \ + CTG_LOCK_DEBUG("CTG RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value after read unlock"); \ + break; \ + } \ + } else { \ + if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \ + qError("invalid lock value before write unlock"); \ + break; \ + } \ + CTG_LOCK_DEBUG("CTG WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosWUnLockLatch(_lock); \ + CTG_LOCK_DEBUG("CTG WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value after write unlock"); \ + break; \ + } \ + } \ } while (0) #define CTG_ERR_RET(c) \ @@ -921,7 +943,7 @@ typedef struct SCtgCacheItemInfo { CTG_UNLOCK(CTG_READ, &gCtgMgmt.lock); \ CTG_API_DEBUG("CTG API leave %s", __FUNCTION__); \ return; \ - } while (0) + } while (0) #define CTG_API_ENTER() \ do { \ @@ -939,7 +961,7 @@ typedef struct SCtgCacheItemInfo { if (atomic_load_8((int8_t*)&gCtgMgmt.exit)) { \ CTG_API_NLEAVE(); \ } \ - } while (0) + } while (0) #define CTG_API_JENTER() \ do { \ @@ -977,7 +999,7 @@ int32_t ctgGetTbMetasFromCache(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgTbMe int32_t ctgCloneDbCfgInfo(void* pSrc, SDbCfgInfo** ppDst); int32_t ctgOpUpdateVgroup(SCtgCacheOperation* action); -int32_t ctgOpUpdateDbCfg(SCtgCacheOperation *operation); +int32_t ctgOpUpdateDbCfg(SCtgCacheOperation* operation); int32_t ctgOpUpdateTbMeta(SCtgCacheOperation* action); int32_t ctgOpDropDbCache(SCtgCacheOperation* action); int32_t ctgOpDropDbVgroup(SCtgCacheOperation* action); @@ -1000,29 +1022,30 @@ int32_t ctgDropStbMetaEnqueue(SCatalog* pCtg, const char* dbFName, int64_t dbId, bool syncReq); int32_t ctgDropTbMetaEnqueue(SCatalog* pCtg, const char* dbFName, int64_t dbId, const char* tbName, bool syncReq); int32_t ctgUpdateVgroupEnqueue(SCatalog* pCtg, const char* dbFName, int64_t dbId, SDBVgInfo* dbInfo, bool syncReq); -int32_t ctgUpdateDbCfgEnqueue(SCatalog *pCtg, const char *dbFName, int64_t dbId, SDbCfgInfo *cfgInfo, bool syncOp); +int32_t ctgUpdateDbCfgEnqueue(SCatalog* pCtg, const char* dbFName, int64_t dbId, SDbCfgInfo* cfgInfo, bool syncOp); int32_t ctgUpdateTbMetaEnqueue(SCatalog* pCtg, STableMetaOutput* output, bool syncReq); int32_t ctgUpdateUserEnqueue(SCatalog* pCtg, SGetUserAuthRsp* pAuth, bool syncReq); int32_t ctgUpdateVgEpsetEnqueue(SCatalog* pCtg, char* dbFName, int32_t vgId, SEpSet* pEpSet); int32_t ctgUpdateTbIndexEnqueue(SCatalog* pCtg, STableIndex** pIndex, bool syncOp); -int32_t ctgDropViewMetaEnqueue(SCatalog *pCtg, const char *dbFName, uint64_t dbId, const char *viewName, uint64_t viewId, bool syncOp); +int32_t ctgDropViewMetaEnqueue(SCatalog* pCtg, const char* dbFName, uint64_t dbId, const char* viewName, + uint64_t viewId, bool syncOp); int32_t ctgClearCacheEnqueue(SCatalog* pCtg, bool clearMeta, bool freeCtg, bool stopQueue, bool syncOp); int32_t ctgMetaRentInit(SCtgRentMgmt* mgmt, uint32_t rentSec, int8_t type, int32_t size); int32_t ctgMetaRentAdd(SCtgRentMgmt* mgmt, void* meta, int64_t id, int32_t size); -int32_t ctgMetaRentUpdate(SCtgRentMgmt *mgmt, void *meta, int64_t id, int32_t size, __compar_fn_t sortCompare, +int32_t ctgMetaRentUpdate(SCtgRentMgmt* mgmt, void* meta, int64_t id, int32_t size, __compar_fn_t sortCompare, __compar_fn_t searchCompare); int32_t ctgMetaRentGet(SCtgRentMgmt* mgmt, void** res, uint32_t* num, int32_t size); -int32_t ctgMetaRentRemove(SCtgRentMgmt *mgmt, int64_t id, __compar_fn_t sortCompare, __compar_fn_t searchCompare); -void ctgRemoveStbRent(SCatalog *pCtg, SCtgDBCache *dbCache); -void ctgRemoveViewRent(SCatalog *pCtg, SCtgDBCache *dbCache); +int32_t ctgMetaRentRemove(SCtgRentMgmt* mgmt, int64_t id, __compar_fn_t sortCompare, __compar_fn_t searchCompare); +void ctgRemoveStbRent(SCatalog* pCtg, SCtgDBCache* dbCache); +void ctgRemoveViewRent(SCatalog* pCtg, SCtgDBCache* dbCache); void ctgRemoveTSMARent(SCatalog* pCtg, SCtgDBCache* dbCache); -int32_t ctgUpdateRentStbVersion(SCatalog *pCtg, char *dbFName, char *tbName, uint64_t dbId, uint64_t suid, - SCtgTbCache *pCache); -int32_t ctgUpdateRentViewVersion(SCatalog *pCtg, char *dbFName, char *viewName, uint64_t dbId, uint64_t viewId, - SCtgViewCache *pCache); +int32_t ctgUpdateRentStbVersion(SCatalog* pCtg, char* dbFName, char* tbName, uint64_t dbId, uint64_t suid, + SCtgTbCache* pCache); +int32_t ctgUpdateRentViewVersion(SCatalog* pCtg, char* dbFName, char* viewName, uint64_t dbId, uint64_t viewId, + SCtgViewCache* pCache); int32_t ctgUpdateRentTSMAVersion(SCatalog* pCtg, char* dbFName, const STSMACache* pCache); int32_t ctgUpdateTbMetaToCache(SCatalog* pCtg, STableMetaOutput* pOut, bool syncReq); -int32_t ctgUpdateViewMetaToCache(SCatalog *pCtg, SViewMetaRsp *pRsp, bool syncReq); +int32_t ctgUpdateViewMetaToCache(SCatalog* pCtg, SViewMetaRsp* pRsp, bool syncReq); int32_t ctgStartUpdateThread(); int32_t ctgRelaunchGetTbMetaTask(SCtgTask* pTask); void ctgReleaseVgInfoToCache(SCatalog* pCtg, SCtgDBCache* dbCache); @@ -1031,11 +1054,11 @@ int32_t ctgDropTbIndexEnqueue(SCatalog* pCtg, SName* pName, bool syncOp); int32_t ctgOpDropTbIndex(SCtgCacheOperation* operation); int32_t ctgOpUpdateTbIndex(SCtgCacheOperation* operation); int32_t ctgOpClearCache(SCtgCacheOperation* operation); -int32_t ctgOpUpdateViewMeta(SCtgCacheOperation *operation); +int32_t ctgOpUpdateViewMeta(SCtgCacheOperation* operation); int32_t ctgReadTbTypeFromCache(SCatalog* pCtg, char* dbFName, char* tableName, int32_t* tbType); int32_t ctgGetTbHashVgroupFromCache(SCatalog* pCtg, const SName* pTableName, SVgroupInfo** pVgroup); -int32_t ctgGetViewsFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgViewsCtx *ctx, int32_t dbIdx, - int32_t *fetchIdx, int32_t baseResIdx, SArray *pList); +int32_t ctgGetViewsFromCache(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgViewsCtx* ctx, int32_t dbIdx, + int32_t* fetchIdx, int32_t baseResIdx, SArray* pList); int32_t ctgProcessRspMsg(void* out, int32_t reqType, char* msg, int32_t msgSize, int32_t rspCode, char* target); int32_t ctgGetDBVgInfoFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, SBuildUseDBInput* input, SUseDbOutput* out, SCtgTaskReq* tReq); @@ -1082,9 +1105,10 @@ int32_t ctgCloneMetaOutput(STableMetaOutput* output, STableMetaOutput** pOutput) int32_t ctgGenerateVgList(SCatalog* pCtg, SHashObj* vgHash, SArray** pList); void ctgFreeJob(void* job); void ctgFreeHandleImpl(SCatalog* pCtg); -int32_t ctgGetVgInfoFromHashValue(SCatalog* pCtg, SEpSet* pMgmtEps, SDBVgInfo* dbInfo, const SName* pTableName, SVgroupInfo* pVgroup); -int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskReq* tReq, SDBVgInfo* dbInfo, SCtgTbHashsCtx* pCtx, - char* dbFName, SArray* pNames, bool update); +int32_t ctgGetVgInfoFromHashValue(SCatalog* pCtg, SEpSet* pMgmtEps, SDBVgInfo* dbInfo, const SName* pTableName, + SVgroupInfo* pVgroup); +int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskReq* tReq, SDBVgInfo* dbInfo, + SCtgTbHashsCtx* pCtx, char* dbFName, SArray* pNames, bool update); int32_t ctgGetVgIdsFromHashValue(SCatalog* pCtg, SDBVgInfo* dbInfo, char* dbFName, const char* pTbs[], int32_t tbNum, int32_t* vgId); void ctgResetTbMetaTask(SCtgTask* pTask); @@ -1119,8 +1143,8 @@ int32_t ctgGetFetchName(SArray* pNames, SCtgFetch* pFetch, SName** ppName); int32_t ctgdGetOneHandle(SCatalog** pHandle); int ctgVgInfoComp(const void* lp, const void* rp); int32_t ctgMakeVgArray(SDBVgInfo* dbInfo); -int32_t ctgChkSetAuthRes(SCatalog *pCtg, SCtgAuthReq *req, SCtgAuthRsp* res); -int32_t ctgReadDBCfgFromCache(SCatalog *pCtg, const char* dbFName, SDbCfgInfo* pDbCfg); +int32_t ctgChkSetAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res); +int32_t ctgReadDBCfgFromCache(SCatalog* pCtg, const char* dbFName, SDbCfgInfo* pDbCfg); int32_t ctgAcquireVgMetaFromCache(SCatalog* pCtg, const char* dbFName, const char* tbName, SCtgDBCache** pDb, SCtgTbCache** pTb); @@ -1133,20 +1157,20 @@ int32_t ctgChkSetAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res); int32_t ctgBuildViewNullRes(SCtgTask* pTask, SCtgViewsCtx* pCtx); int32_t dupViewMetaFromRsp(SViewMetaRsp* pRsp, SViewMeta* pViewMeta); void ctgDestroySMetaData(SMetaData* pData); -void ctgGetGlobalCacheSize(uint64_t *pSize); -uint64_t ctgGetTbIndexCacheSize(STableIndex *pIndex); -uint64_t ctgGetViewMetaCacheSize(SViewMeta *pMeta); -uint64_t ctgGetTbMetaCacheSize(STableMeta *pMeta); -uint64_t ctgGetDbVgroupCacheSize(SDBVgInfo *pVg); -uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth); -uint64_t ctgGetClusterCacheSize(SCatalog *pCtg); -void ctgClearHandleMeta(SCatalog* pCtg, int64_t *pClearedSize, int64_t *pCleardNum, bool *roundDone); -void ctgClearAllHandleMeta(int64_t *clearedSize, int64_t *clearedNum, bool *roundDone); -void ctgProcessTimerEvent(void *param, void *tmrId); +void ctgGetGlobalCacheSize(uint64_t* pSize); +uint64_t ctgGetTbIndexCacheSize(STableIndex* pIndex); +uint64_t ctgGetViewMetaCacheSize(SViewMeta* pMeta); +uint64_t ctgGetTbMetaCacheSize(STableMeta* pMeta); +uint64_t ctgGetDbVgroupCacheSize(SDBVgInfo* pVg); +uint64_t ctgGetUserCacheSize(SGetUserAuthRsp* pAuth); +uint64_t ctgGetClusterCacheSize(SCatalog* pCtg); +void ctgClearHandleMeta(SCatalog* pCtg, int64_t* pClearedSize, int64_t* pCleardNum, bool* roundDone); +void ctgClearAllHandleMeta(int64_t* clearedSize, int64_t* clearedNum, bool* roundDone); +void ctgProcessTimerEvent(void* param, void* tmrId); int32_t ctgBuildUseDbOutput(SUseDbOutput** ppOut, SDBVgInfo* vgInfo); int32_t ctgGetTbMeta(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgTbMetaCtx* ctx, STableMeta** pTableMeta); -int32_t ctgGetCachedStbNameFromSuid(SCatalog* pCtg, char* dbFName, uint64_t suid, char **stbName); +int32_t ctgGetCachedStbNameFromSuid(SCatalog* pCtg, char* dbFName, uint64_t suid, char** stbName); int32_t ctgGetTbTagCb(SCtgTask* pTask); int32_t ctgGetUserCb(SCtgTask* pTask); @@ -1170,7 +1194,8 @@ int32_t ctgGetStreamProgressFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, int32_t ctgAddTSMAFetch(SArray** pFetchs, int32_t dbIdx, int32_t tbIdx, int32_t* fetchIdx, int32_t resIdx, int32_t flag, CTG_TSMA_FETCH_TYPE fetchType, const SName* sourceTbName); int32_t ctgOpUpdateDbTsmaVersion(SCtgCacheOperation* pOper); -int32_t ctgUpdateDbTsmaVersionEnqueue(SCatalog* pCtg, int32_t tsmaVersion, const char* dbFName, int64_t dbId, bool syncOper); +int32_t ctgUpdateDbTsmaVersionEnqueue(SCatalog* pCtg, int32_t tsmaVersion, const char* dbFName, int64_t dbId, + bool syncOper); void ctgFreeTask(SCtgTask* pTask, bool freeRes); extern SCatalogMgmt gCtgMgmt; diff --git a/source/libs/catalog/src/catalog.c b/source/libs/catalog/src/catalog.c index d4c79a6c8d..334dce9c1a 100644 --- a/source/libs/catalog/src/catalog.c +++ b/source/libs/catalog/src/catalog.c @@ -773,8 +773,6 @@ int32_t ctgGetTsma(SCatalog* pCtg, SRequestConnInfo* pConn, const SName* pTsmaNa } CTG_ERR_JRET(code); - - ASSERT(tsmaRsp.pTsmas && tsmaRsp.pTsmas->size == 1); *pTsma = taosArrayGetP(tsmaRsp.pTsmas, 0); taosArrayDestroy(tsmaRsp.pTsmas); diff --git a/source/libs/catalog/src/ctgAsync.c b/source/libs/catalog/src/ctgAsync.c index 4caa66445a..7e15fd5f4f 100644 --- a/source/libs/catalog/src/ctgAsync.c +++ b/source/libs/catalog/src/ctgAsync.c @@ -20,12 +20,11 @@ #include "tref.h" #include "trpc.h" - void ctgIsTaskDone(SCtgJob* pJob, CTG_TASK_TYPE type, bool* done) { SCtgTask* pTask = NULL; *done = true; - + CTG_LOCK(CTG_READ, &pJob->taskLock); int32_t taskNum = taosArrayGetSize(pJob->pTasks); @@ -46,8 +45,8 @@ void ctgIsTaskDone(SCtgJob* pJob, CTG_TASK_TYPE type, bool* done) { int32_t ctgInitGetTbMetaTask(SCtgJob* pJob, int32_t taskIdx, void* param) { SCtgTbMetaParam* pParam = (SCtgTbMetaParam*)param; - SName* name = pParam->pName; - SCtgTask task = {0}; + SName* name = pParam->pName; + SCtgTask task = {0}; task.type = CTG_TASK_GET_TB_META; task.taskId = taskIdx; @@ -73,7 +72,7 @@ int32_t ctgInitGetTbMetaTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name->tname); return TSDB_CODE_SUCCESS; @@ -95,17 +94,18 @@ int32_t ctgInitGetTbMetasTask(SCtgJob* pJob, int32_t taskIdx, void* param) { ctx->pNames = param; ctx->pResList = taosArrayInit(pJob->tbMetaNum, sizeof(SMetaRes)); if (NULL == ctx->pResList) { - qError("QID:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tbMetaNum, (int32_t)sizeof(SMetaRes)); + qError("qid:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tbMetaNum, + (int32_t)sizeof(SMetaRes)); ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + if (NULL == taosArrayPush(pJob->pTasks, &task)) { ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, dbNum:%lu, tbNum:%d", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, dbNum:%lu, tbNum:%d", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), taosArrayGetSize(ctx->pNames), pJob->tbMetaNum); return TSDB_CODE_SUCCESS; @@ -133,7 +133,7 @@ int32_t ctgInitGetDbVgTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, dbFName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, dbFName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), dbFName); return TSDB_CODE_SUCCESS; @@ -161,7 +161,7 @@ int32_t ctgInitGetDbCfgTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, dbFName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, dbFName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), dbFName); return TSDB_CODE_SUCCESS; @@ -189,7 +189,7 @@ int32_t ctgInitGetDbInfoTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, dbFName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, dbFName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), dbFName); return TSDB_CODE_SUCCESS; @@ -223,7 +223,7 @@ int32_t ctgInitGetTbHashTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, tableName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, tableName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name->tname); return TSDB_CODE_SUCCESS; @@ -245,17 +245,18 @@ int32_t ctgInitGetTbHashsTask(SCtgJob* pJob, int32_t taskIdx, void* param) { ctx->pNames = param; ctx->pResList = taosArrayInit(pJob->tbHashNum, sizeof(SMetaRes)); if (NULL == ctx->pResList) { - qError("QID:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tbHashNum, (int32_t)sizeof(SMetaRes)); + qError("qid:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tbHashNum, + (int32_t)sizeof(SMetaRes)); ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + if (NULL == taosArrayPush(pJob->pTasks, &task)) { ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, dbNum:%lu, tbNum:%d", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, dbNum:%lu, tbNum:%d", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), taosArrayGetSize(ctx->pNames), pJob->tbHashNum); return TSDB_CODE_SUCCESS; @@ -274,7 +275,7 @@ int32_t ctgInitGetQnodeTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type)); + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type)); return TSDB_CODE_SUCCESS; } @@ -292,7 +293,7 @@ int32_t ctgInitGetDnodeTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type)); + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type)); return TSDB_CODE_SUCCESS; } @@ -319,7 +320,7 @@ int32_t ctgInitGetIndexTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, indexFName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, indexFName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name); return TSDB_CODE_SUCCESS; @@ -347,7 +348,7 @@ int32_t ctgInitGetUdfTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, udfName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, udfName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name); return TSDB_CODE_SUCCESS; @@ -375,7 +376,7 @@ int32_t ctgInitGetUserTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, user:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, user:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), user->user); return TSDB_CODE_SUCCESS; @@ -393,7 +394,7 @@ int32_t ctgInitGetSvrVerTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type)); + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type)); return TSDB_CODE_SUCCESS; } @@ -425,7 +426,7 @@ int32_t ctgInitGetTbIndexTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name->tname); return TSDB_CODE_SUCCESS; @@ -458,7 +459,7 @@ int32_t ctgInitGetTbCfgTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name->tname); return TSDB_CODE_SUCCESS; @@ -491,7 +492,7 @@ int32_t ctgInitGetTbTagTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, tbName:%s", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), name->tname); return TSDB_CODE_SUCCESS; @@ -513,7 +514,8 @@ int32_t ctgInitGetViewsTask(SCtgJob* pJob, int32_t taskIdx, void* param) { ctx->pNames = param; ctx->pResList = taosArrayInit(pJob->viewNum, sizeof(SMetaRes)); if (NULL == ctx->pResList) { - qError("QID:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->viewNum, (int32_t)sizeof(SMetaRes)); + qError("qid:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->viewNum, + (int32_t)sizeof(SMetaRes)); ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } @@ -523,7 +525,7 @@ int32_t ctgInitGetViewsTask(SCtgJob* pJob, int32_t taskIdx, void* param) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - qDebug("QID:0x%" PRIx64 " the %dth task type %s initialized, dbNum:%lu, viewNum:%d", pJob->queryId, taskIdx, + qDebug("qid:0x%" PRIx64 " the %dth task type %s initialized, dbNum:%lu, viewNum:%d", pJob->queryId, taskIdx, ctgTaskTypeStr(task.type), taosArrayGetSize(ctx->pNames), pJob->viewNum); return TSDB_CODE_SUCCESS; @@ -539,21 +541,22 @@ int32_t ctgInitGetTbTSMATask(SCtgJob* pJob, int32_t taskId, void* param) { if (NULL == pTaskCtx) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + task.taskCtx = pTaskCtx; pTaskCtx->pNames = param; pTaskCtx->pResList = taosArrayInit(pJob->tbTsmaNum, sizeof(SMetaRes)); if (NULL == pTaskCtx->pResList) { - qError("QID:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tbTsmaNum, (int32_t)sizeof(SMetaRes)); + qError("qid:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tbTsmaNum, + (int32_t)sizeof(SMetaRes)); ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + if (NULL == taosArrayPush(pJob->pTasks, &task)) { ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + return TSDB_CODE_SUCCESS; } @@ -571,7 +574,8 @@ int32_t ctgInitGetTSMATask(SCtgJob* pJob, int32_t taskId, void* param) { pTaskCtx->pNames = param; pTaskCtx->pResList = taosArrayInit(pJob->tsmaNum, sizeof(SMetaRes)); if (NULL == pTaskCtx->pResList) { - qError("QID:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tsmaNum, (int32_t)sizeof(SMetaRes)); + qError("qid:0x%" PRIx64 " taosArrayInit %d SMetaRes %d failed", pJob->queryId, pJob->tsmaNum, + (int32_t)sizeof(SMetaRes)); ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } @@ -580,11 +584,10 @@ int32_t ctgInitGetTSMATask(SCtgJob* pJob, int32_t taskId, void* param) { ctgFreeTask(&task, true); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + return TSDB_CODE_SUCCESS; } - int32_t ctgHandleForceUpdateView(SCatalog* pCtg, const SCatalogReq* pReq) { int32_t viewNum = taosArrayGetSize(pReq->pView); for (int32_t i = 0; i < viewNum; ++i) { @@ -593,20 +596,19 @@ int32_t ctgHandleForceUpdateView(SCatalog* pCtg, const SCatalogReq* pReq) { qError("taosArrayGet the %dth view in req failed", i); CTG_ERR_RET(TSDB_CODE_CTG_INVALID_INPUT); } - + int32_t viewNum = taosArrayGetSize(p->pTables); for (int32_t m = 0; m < viewNum; ++m) { SName* name = taosArrayGet(p->pTables, m); CTG_ERR_RET(ctgDropViewMetaEnqueue(pCtg, p->dbFName, 0, name->tname, 0, true)); } } - + return TSDB_CODE_SUCCESS; } - int32_t ctgHandleForceUpdate(SCatalog* pCtg, int32_t taskNum, SCtgJob* pJob, const SCatalogReq* pReq) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; SHashObj* pDb = taosHashInit(taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK); SHashObj* pTb = taosHashInit(taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK); if (NULL == pDb || NULL == pTb) { @@ -650,7 +652,7 @@ int32_t ctgHandleForceUpdate(SCatalog* pCtg, int32_t taskNum, SCtgJob* pJob, con CTG_ERR_JRET(TSDB_CODE_CTG_INVALID_INPUT); } CTG_ERR_JRET(taosHashPut(pDb, p->dbFName, strlen(p->dbFName), p->dbFName, TSDB_DB_FNAME_LEN)); - + int32_t tbNum = taosArrayGetSize(p->pTables); for (int32_t m = 0; m < tbNum; ++m) { SName* name = taosArrayGet(p->pTables, m); @@ -671,7 +673,7 @@ int32_t ctgHandleForceUpdate(SCatalog* pCtg, int32_t taskNum, SCtgJob* pJob, con } CTG_ERR_JRET(taosHashPut(pDb, p->dbFName, strlen(p->dbFName), p->dbFName, TSDB_DB_FNAME_LEN)); - + int32_t tbNum = taosArrayGetSize(p->pTables); for (int32_t m = 0; m < tbNum; ++m) { SName* name = taosArrayGet(p->pTables, m); @@ -700,7 +702,7 @@ int32_t ctgHandleForceUpdate(SCatalog* pCtg, int32_t taskNum, SCtgJob* pJob, con CTG_ERR_JRET(TSDB_CODE_CTG_INVALID_INPUT); } - char dbFName[TSDB_DB_FNAME_LEN]; + char dbFName[TSDB_DB_FNAME_LEN]; (void)tNameGetFullDbName(name, dbFName); CTG_ERR_JRET(taosHashPut(pDb, dbFName, strlen(dbFName), dbFName, TSDB_DB_FNAME_LEN)); CTG_ERR_JRET(taosHashPut(pTb, name, sizeof(SName), name, sizeof(SName))); @@ -713,7 +715,7 @@ int32_t ctgHandleForceUpdate(SCatalog* pCtg, int32_t taskNum, SCtgJob* pJob, con CTG_ERR_JRET(TSDB_CODE_CTG_INVALID_INPUT); } - char dbFName[TSDB_DB_FNAME_LEN]; + char dbFName[TSDB_DB_FNAME_LEN]; (void)tNameGetFullDbName(name, dbFName); CTG_ERR_JRET(taosHashPut(pDb, dbFName, strlen(dbFName), dbFName, TSDB_DB_FNAME_LEN)); CTG_ERR_JRET(taosHashPut(pTb, name, sizeof(SName), name, sizeof(SName))); @@ -776,7 +778,7 @@ _return: taosHashCleanup(pDb); taosHashCleanup(pTb); - + return code; } @@ -826,7 +828,7 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const *job = taosMemoryCalloc(1, sizeof(SCtgJob)); if (NULL == *job) { - ctgError("failed to calloc, size:%d, reqId:0x%" PRIx64, (int32_t)sizeof(SCtgJob), pConn->requestId); + ctgError("failed to calloc, size:%d, qid:0x%" PRIx64, (int32_t)sizeof(SCtgJob), pConn->requestId); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } @@ -953,7 +955,6 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const CTG_ERR_JRET(ctgInitTask(pJob, CTG_TASK_GET_TB_TAG, name, NULL)); } - for (int32_t i = 0; i < indexNum; ++i) { char* indexName = taosArrayGet(pReq->pIndex, i); if (NULL == indexName) { @@ -1009,7 +1010,7 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const (void)taosAcquireRef(gCtgMgmt.jobPool, pJob->refId); double el = (taosGetTimestampUs() - st) / 1000.0; - qDebug("QID:0x%" PRIx64 ", jobId: 0x%" PRIx64 " initialized, task num %d, forceUpdate %d, elapsed time:%.2f ms", + qDebug("qid:0x%" PRIx64 ", jobId: 0x%" PRIx64 " initialized, task num %d, forceUpdate %d, elapsed time:%.2f ms", pJob->queryId, pJob->refId, taskNum, pReq->forceUpdate, el); return TSDB_CODE_SUCCESS; @@ -1017,7 +1018,7 @@ _return: ctgFreeJob(*job); *job = NULL; - + CTG_RET(code); } @@ -1025,7 +1026,7 @@ int32_t ctgDumpTbMetaRes(SCtgTask* pTask) { if (pTask->subTask) { return TSDB_CODE_SUCCESS; } - + SCtgJob* pJob = pTask->pJob; if (NULL == pJob->jobRes.pTableMeta) { pJob->jobRes.pTableMeta = taosArrayInit(pJob->tbMetaNum, sizeof(SMetaRes)); @@ -1038,7 +1039,7 @@ int32_t ctgDumpTbMetaRes(SCtgTask* pTask) { if (NULL == taosArrayPush(pJob->jobRes.pTableMeta, &res)) { CTG_ERR_RET(terrno); } - + return TSDB_CODE_SUCCESS; } @@ -1169,7 +1170,7 @@ int32_t ctgDumpTbTagRes(SCtgTask* pTask) { if (atomic_val_compare_exchange_ptr(&pJob->jobRes.pTableTag, NULL, pRes)) { taosArrayDestroy(pRes); } - + if (NULL == pJob->jobRes.pTableTag) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } @@ -1183,7 +1184,6 @@ int32_t ctgDumpTbTagRes(SCtgTask* pTask) { return TSDB_CODE_SUCCESS; } - int32_t ctgDumpIndexRes(SCtgTask* pTask) { if (pTask->subTask) { return TSDB_CODE_SUCCESS; @@ -1350,7 +1350,6 @@ int32_t ctgDumpSvrVer(SCtgTask* pTask) { return TSDB_CODE_SUCCESS; } - int32_t ctgDumpViewsRes(SCtgTask* pTask) { if (pTask->subTask) { return TSDB_CODE_SUCCESS; @@ -1363,7 +1362,6 @@ int32_t ctgDumpViewsRes(SCtgTask* pTask) { return TSDB_CODE_SUCCESS; } - int32_t ctgCallSubCb(SCtgTask* pTask) { int32_t code = 0; @@ -1377,7 +1375,7 @@ int32_t ctgCallSubCb(SCtgTask* pTask) { qError("taosArrayGetP the %dth parent failed", i); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + pParent->subRes.code = pTask->code; if (TSDB_CODE_SUCCESS == pTask->code) { code = (*gCtgAsyncFps[pTask->type].cloneFp)(pTask, &pParent->subRes.res); @@ -1402,11 +1400,11 @@ _return: int32_t ctgCallUserCb(void* param) { SCtgJob* pJob = (SCtgJob*)param; - qDebug("QID:0x%" PRIx64 " ctg start to call user cb with rsp %s", pJob->queryId, tstrerror(pJob->jobResCode)); + qDebug("qid:0x%" PRIx64 " ctg start to call user cb with rsp %s", pJob->queryId, tstrerror(pJob->jobResCode)); (*pJob->userFp)(&pJob->jobRes, pJob->userParam, pJob->jobResCode); - qDebug("QID:0x%" PRIx64 " ctg end to call user cb", pJob->queryId); + qDebug("qid:0x%" PRIx64 " ctg end to call user cb", pJob->queryId); (void)taosRemoveRef(gCtgMgmt.jobPool, pJob->refId); @@ -1417,7 +1415,7 @@ void ctgUpdateJobErrCode(SCtgJob* pJob, int32_t errCode) { if (!NEED_CLIENT_REFRESH_VG_ERROR(errCode) || errCode == TSDB_CODE_SUCCESS) return; atomic_store_32(&pJob->jobResCode, errCode); - qDebug("QID:0x%" PRIx64 " ctg job errCode updated to %s", pJob->queryId, tstrerror(errCode)); + qDebug("qid:0x%" PRIx64 " ctg job errCode updated to %s", pJob->queryId, tstrerror(errCode)); return; } @@ -1429,7 +1427,7 @@ int32_t ctgHandleTaskEnd(SCtgTask* pTask, int32_t rspCode) { return TSDB_CODE_SUCCESS; } - qDebug("QID:0x%" PRIx64 " task %d end with res %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); + qDebug("qid:0x%" PRIx64 " task %d end with res %s", pJob->queryId, pTask->taskId, tstrerror(rspCode)); pTask->code = rspCode; pTask->status = CTG_TASK_DONE; @@ -1438,7 +1436,7 @@ int32_t ctgHandleTaskEnd(SCtgTask* pTask, int32_t rspCode) { int32_t taskDone = atomic_add_fetch_32(&pJob->taskDone, 1); if (taskDone < taosArrayGetSize(pJob->pTasks)) { - qDebug("QID:0x%" PRIx64 " task done: %d, total: %d", pJob->queryId, taskDone, + qDebug("qid:0x%" PRIx64 " task done: %d, total: %d", pJob->queryId, taskDone, (int32_t)taosArrayGetSize(pJob->pTasks)); ctgUpdateJobErrCode(pJob, rspCode); @@ -1525,7 +1523,7 @@ int32_t ctgHandleGetTbMetaRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf } ctgError("no tbmeta got, tbName:%s", tNameGetTableName(pName)); - (void)ctgRemoveTbMetaFromCache(pCtg, pName, false); // update cache not fatal error + (void)ctgRemoveTbMetaFromCache(pCtg, pName, false); // update cache not fatal error CTG_ERR_JRET(CTG_ERR_CODE_TABLE_NOT_EXIST); } @@ -1644,23 +1642,23 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, tReq->msgIdx); SCtgTbMetasCtx* ctx = (SCtgTbMetasCtx*)pTask->taskCtx; bool taskDone = false; - + if (NULL == pMsgCtx) { ctgError("fail to get task msgCtx, taskType:%d", pTask->type); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SCtgFetch* pFetch = taosArrayGet(ctx->pFetchs, tReq->msgIdx); + SCtgFetch* pFetch = taosArrayGet(ctx->pFetchs, tReq->msgIdx); if (NULL == pFetch) { ctgError("fail to get the %dth fetch, fetchNum:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(ctx->pFetchs)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - - int32_t flag = pFetch->flag; - int32_t* vgId = &pFetch->vgId; - SName* pName = NULL; + + int32_t flag = pFetch->flag; + int32_t* vgId = &pFetch->vgId; + SName* pName = NULL; CTG_ERR_JRET(ctgGetFetchName(ctx->pNames, pFetch, &pName)); - + CTG_ERR_JRET(ctgProcessRspMsg(pMsgCtx->out, reqType, pMsg->pData, pMsg->len, rspCode, pMsgCtx->target)); switch (reqType) { @@ -1710,7 +1708,7 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu } ctgTaskError("no tbmeta got, tbName:%s", tNameGetTableName(pName)); - (void)ctgRemoveTbMetaFromCache(pCtg, pName, false); // cache update not fatal error + (void)ctgRemoveTbMetaFromCache(pCtg, pName, false); // cache update not fatal error CTG_ERR_JRET(CTG_ERR_CODE_TABLE_NOT_EXIST); } @@ -1728,7 +1726,7 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu if (CTG_IS_META_NULL(pOut->metaType)) { ctgTaskError("no tbmeta got, tbName:%s", tNameGetTableName(pName)); - (void)ctgRemoveTbMetaFromCache(pCtg, pName, false); // cache update not fatal error + (void)ctgRemoveTbMetaFromCache(pCtg, pName, false); // cache update not fatal error CTG_ERR_JRET(CTG_ERR_CODE_TABLE_NOT_EXIST); } @@ -1778,7 +1776,7 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu STableMetaOutput* pOut = (STableMetaOutput*)pMsgCtx->out; - (void)ctgUpdateTbMetaToCache(pCtg, pOut, false); // cache update not fatal error + (void)ctgUpdateTbMetaToCache(pCtg, pOut, false); // cache update not fatal error if (CTG_IS_META_BOTH(pOut->metaType)) { TAOS_MEMCPY(pOut->tbMeta, &pOut->ctbMeta, sizeof(pOut->ctbMeta)); @@ -1806,10 +1804,11 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu SMetaRes* pRes = taosArrayGet(ctx->pResList, pFetch->resIdx); if (NULL == pRes) { - ctgTaskError("fail to get the %dth res in pResList, resNum:%d", pFetch->resIdx, (int32_t)taosArrayGetSize(ctx->pResList)); + ctgTaskError("fail to get the %dth res in pResList, resNum:%d", pFetch->resIdx, + (int32_t)taosArrayGetSize(ctx->pResList)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + pRes->code = 0; pRes->pRes = pOut->tbMeta; pOut->tbMeta = NULL; @@ -1827,7 +1826,8 @@ _return: if (code) { SMetaRes* pRes = taosArrayGet(ctx->pResList, pFetch->resIdx); if (NULL == pRes) { - ctgTaskError("fail to get the %dth res in pResList, resNum:%d", pFetch->resIdx, (int32_t)taosArrayGetSize(ctx->pResList)); + ctgTaskError("fail to get the %dth res in pResList, resNum:%d", pFetch->resIdx, + (int32_t)taosArrayGetSize(ctx->pResList)); } else { pRes->code = code; pRes->pRes = NULL; @@ -1904,7 +1904,8 @@ int32_t ctgHandleGetTbHashRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf CTG_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } - CTG_ERR_JRET(ctgGetVgInfoFromHashValue(pCtg, &pTask->pJob->conn.mgmtEps, pOut->dbVgroup, ctx->pName, (SVgroupInfo*)pTask->res)); + CTG_ERR_JRET(ctgGetVgInfoFromHashValue(pCtg, &pTask->pJob->conn.mgmtEps, pOut->dbVgroup, ctx->pName, + (SVgroupInfo*)pTask->res)); CTG_ERR_JRET(ctgUpdateVgroupEnqueue(pCtg, ctx->dbFName, pOut->dbId, pOut->dbVgroup, false)); pOut->dbVgroup = NULL; @@ -1938,7 +1939,7 @@ int32_t ctgHandleGetTbHashsRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SCtgFetch* pFetch = taosArrayGet(ctx->pFetchs, tReq->msgIdx); + SCtgFetch* pFetch = taosArrayGet(ctx->pFetchs, tReq->msgIdx); if (NULL == pFetch) { ctgError("fail to get the %dth fetch, fetchNum:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(ctx->pFetchs)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); @@ -1952,11 +1953,13 @@ int32_t ctgHandleGetTbHashsRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu STablesReq* pReq = taosArrayGet(ctx->pNames, pFetch->dbIdx); if (NULL == pReq) { - ctgError("fail to get the %dth tb in ctx->pNames, reqNum:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(ctx->pNames)); + ctgError("fail to get the %dth tb in ctx->pNames, reqNum:%d", pFetch->dbIdx, + (int32_t)taosArrayGetSize(ctx->pNames)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - - CTG_ERR_JRET(ctgGetVgInfosFromHashValue(pCtg, &pTask->pJob->conn.mgmtEps, tReq, pOut->dbVgroup, ctx, pMsgCtx->target, pReq->pTables, true)); + + CTG_ERR_JRET(ctgGetVgInfosFromHashValue(pCtg, &pTask->pJob->conn.mgmtEps, tReq, pOut->dbVgroup, ctx, + pMsgCtx->target, pReq->pTables, true)); CTG_ERR_JRET(ctgUpdateVgroupEnqueue(pCtg, pMsgCtx->target, pOut->dbId, pOut->dbVgroup, false)); pOut->dbVgroup = NULL; @@ -1978,13 +1981,15 @@ _return: if (code) { STablesReq* pReq = taosArrayGet(ctx->pNames, pFetch->dbIdx); if (NULL == pReq) { - ctgError("fail to get the %dth tb in ctx->pNames, reqNum:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(ctx->pNames)); + ctgError("fail to get the %dth tb in ctx->pNames, reqNum:%d", pFetch->dbIdx, + (int32_t)taosArrayGetSize(ctx->pNames)); } else { - int32_t num = taosArrayGetSize(pReq->pTables); + int32_t num = taosArrayGetSize(pReq->pTables); for (int32_t i = 0; i < num; ++i) { SMetaRes* pRes = taosArrayGet(ctx->pResList, pFetch->resIdx + i); if (NULL == pRes) { - ctgError("fail to get the %dth res in ctx->pResList, resNum:%d", pFetch->resIdx + i, (int32_t)taosArrayGetSize(ctx->pResList)); + ctgError("fail to get the %dth res in ctx->pResList, resNum:%d", pFetch->resIdx + i, + (int32_t)taosArrayGetSize(ctx->pResList)); } else { pRes->code = code; pRes->pRes = NULL; @@ -2027,7 +2032,7 @@ _return: if (TSDB_CODE_MND_DB_INDEX_NOT_EXIST == code) { code = TSDB_CODE_SUCCESS; } - + newCode = ctgHandleTaskEnd(pTask, code); if (newCode && TSDB_CODE_SUCCESS == code) { code = newCode; @@ -2055,7 +2060,6 @@ _return: CTG_RET(code); } - int32_t ctgHandleGetTbTagRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* pMsg, int32_t rspCode) { int32_t code = 0; SCtgTask* pTask = tReq->pTask; @@ -2090,11 +2094,11 @@ int32_t ctgHandleGetTbTagRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* taosMemoryFree(pJson); taosArrayDestroy(pTagVals); CTG_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); - } + } } else { CTG_ERR_JRET(tTagToValArray((const STag*)pRsp->pTags, &pTagVals)); } - + pTask->res = pTagVals; _return: @@ -2107,13 +2111,12 @@ _return: CTG_RET(code); } - int32_t ctgHandleGetDbCfgRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* pMsg, int32_t rspCode) { - int32_t code = 0; - SCtgTask* pTask = tReq->pTask; + int32_t code = 0; + SCtgTask* pTask = tReq->pTask; SCtgDbCfgCtx* ctx = pTask->taskCtx; - int32_t newCode = TSDB_CODE_SUCCESS; - + int32_t newCode = TSDB_CODE_SUCCESS; + CTG_ERR_JRET(ctgProcessRspMsg(pTask->msgCtx.out, reqType, pMsg->pData, pMsg->len, rspCode, pTask->msgCtx.target)); SDbCfgInfo* pCfg = NULL; @@ -2219,7 +2222,7 @@ int32_t ctgHandleGetUserRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* CTG_ERR_JRET(ctgProcessRspMsg(pTask->msgCtx.out, reqType, pMsg->pData, pMsg->len, rspCode, pTask->msgCtx.target)); - (void)ctgUpdateUserEnqueue(pCtg, pOut, true); // cache update not fatal error + (void)ctgUpdateUserEnqueue(pCtg, pOut, true); // cache update not fatal error taosMemoryFreeClear(pTask->msgCtx.out); CTG_ERR_JRET((*gCtgAsyncFps[pTask->type].launchFp)(pTask)); @@ -2275,9 +2278,9 @@ int32_t ctgHandleGetViewsRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - int32_t flag = pFetch->flag; - int32_t* vgId = &pFetch->vgId; - SName* pName = NULL; + int32_t flag = pFetch->flag; + int32_t* vgId = &pFetch->vgId; + SName* pName = NULL; CTG_ERR_JRET(ctgGetFetchName(ctx->pNames, pFetch, &pName)); @@ -2297,16 +2300,17 @@ int32_t ctgHandleGetViewsRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* } ctgDebug("start to update view meta to cache, view:%s, querySQL:%s", pRsp->name, pRsp->querySql); - (void)ctgUpdateViewMetaToCache(pCtg, pRsp, false); // cache update not fatal error + (void)ctgUpdateViewMetaToCache(pCtg, pRsp, false); // cache update not fatal error taosMemoryFreeClear(pMsgCtx->out); pRsp = NULL; - + SMetaRes* pRes = taosArrayGet(ctx->pResList, pFetch->resIdx); if (NULL == pRes) { - ctgTaskError("fail to get the %dth res in ctx->pResList, totalResNum:%d", pFetch->resIdx, (int32_t)taosArrayGetSize(ctx->pResList)); + ctgTaskError("fail to get the %dth res in ctx->pResList, totalResNum:%d", pFetch->resIdx, + (int32_t)taosArrayGetSize(ctx->pResList)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + pRes->code = 0; pRes->pRes = pViewMeta; if (0 == atomic_sub_fetch_32(&ctx->fetchNum, 1)) { @@ -2319,7 +2323,8 @@ _return: if (code) { SMetaRes* pRes = taosArrayGet(ctx->pResList, pFetch->resIdx); if (NULL == pRes) { - ctgTaskError("fail to get the %dth res in ctx->pResList, totalResNum:%d", pFetch->resIdx, (int32_t)taosArrayGetSize(ctx->pResList)); + ctgTaskError("fail to get the %dth res in ctx->pResList, totalResNum:%d", pFetch->resIdx, + (int32_t)taosArrayGetSize(ctx->pResList)); } else { pRes->code = code; pRes->pRes = NULL; @@ -2347,7 +2352,6 @@ _return: CTG_RET(code); } - static int32_t ctgTsmaFetchStreamProgress(SCtgTaskReq* tReq, SHashObj* pVgHash, const STableTSMAInfoRsp* pTsmas) { int32_t code = 0; SCtgTask* pTask = tReq->pTask; @@ -2358,17 +2362,18 @@ static int32_t ctgTsmaFetchStreamProgress(SCtgTaskReq* tReq, SHashObj* pVgHash, SVgroupInfo* pVgInfo = NULL; SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); if (NULL == pFetch) { - ctgError("fail to get the %dth SCtgTSMAFetch, totalNum:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetches)); + ctgError("fail to get the %dth SCtgTSMAFetch, totalNum:%d", tReq->msgIdx, + (int32_t)taosArrayGetSize(pCtx->pFetches)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); + STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); if (NULL == pTbReq) { ctgError("fail to get the %dth STablesReq, totalNum:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - const SName* pTbName = taosArrayGet(pTbReq->pTables, pFetch->tbIdx); + const SName* pTbName = taosArrayGet(pTbReq->pTables, pFetch->tbIdx); if (NULL == pTbName) { ctgError("fail to get the %dth SName, totalNum:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pTbReq->pTables)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); @@ -2392,42 +2397,41 @@ static int32_t ctgTsmaFetchStreamProgress(SCtgTaskReq* tReq, SHashObj* pVgHash, .vgId = pVgInfo->vgId}; CTG_ERR_JRET(ctgGetStreamProgressFromVnode(pCtg, pConn, pTbName, pVgInfo, NULL, tReq, &req)); pFetch->subFetchNum++; - + pVgInfo = taosHashIterate(pVgHash, pVgInfo); } } - + _return: CTG_RET(code); } - int32_t ctgHandleGetTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* pMsg, int32_t rspCode) { - int32_t code = 0; - SCtgTask* pTask = tReq->pTask; - SCatalog* pCtg = pTask->pJob->pCtg; - SCtgTbTSMACtx* pCtx = pTask->taskCtx; - int32_t newCode = TSDB_CODE_SUCCESS; - SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, tReq->msgIdx); + int32_t code = 0; + SCtgTask* pTask = tReq->pTask; + SCatalog* pCtg = pTask->pJob->pCtg; + SCtgTbTSMACtx* pCtx = pTask->taskCtx; + int32_t newCode = TSDB_CODE_SUCCESS; + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, tReq->msgIdx); if (NULL == pMsgCtx) { ctgError("fail to get the %dth SCtgMsgCtx", tReq->msgIdx); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - STablesReq* pTbReq = taosArrayGet(pCtx->pNames, 0); + STablesReq* pTbReq = taosArrayGet(pCtx->pNames, 0); if (NULL == pTbReq) { ctgError("fail to get the 0th STablesReq, totalNum:%d", (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - - SName* pName = taosArrayGet(pTbReq->pTables, 0); + + SName* pName = taosArrayGet(pTbReq->pTables, 0); if (NULL == pName) { ctgError("fail to get the 0th SName, totalNum:%d", (int32_t)taosArrayGetSize(pTbReq->pTables)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SRequestConnInfo* pConn = &pTask->pJob->conn; + SRequestConnInfo* pConn = &pTask->pJob->conn; CTG_ERR_JRET(ctgProcessRspMsg(pMsgCtx->out, reqType, pMsg->pData, pMsg->len, rspCode, pMsgCtx->target)); switch (reqType) { @@ -2436,9 +2440,9 @@ int32_t ctgHandleGetTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* if (!CTG_IS_META_NULL(pOut->metaType)) { CTG_ERR_JRET(ctgUpdateTbMetaToCache(pCtg, pOut, CTG_FLAG_SYNC_OP)); } - + break; - } + } case TDMT_MND_GET_TSMA: { SMetaRes* pRes = taosArrayGet(pCtx->pResList, 0); if (NULL == pRes) { @@ -2448,30 +2452,33 @@ int32_t ctgHandleGetTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* STableTSMAInfoRsp* pOut = pMsgCtx->out; pRes->code = 0; - if (pOut->pTsmas->size > 0) { - ASSERT(pOut->pTsmas->size == 1); - pRes->pRes = pOut; - pMsgCtx->out = NULL; - TSWAP(pTask->res, pCtx->pResList); - - STableTSMAInfo* pTsma = taosArrayGetP(pOut->pTsmas, 0); - if (NULL == pTsma) { - ctgError("fail to get the 0th STableTSMAInfo, totalNum:%d", (int32_t)taosArrayGetSize(pOut->pTsmas)); - CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); - } - - int32_t exists = false; - CTG_ERR_JRET(ctgTbMetaExistInCache(pCtg, pTsma->targetDbFName, pTsma->targetTb, &exists)); - if (!exists) { - TSWAP(pMsgCtx->lastOut, pMsgCtx->out); - CTG_RET(ctgGetTbMetaFromMnodeImpl(pCtg, pConn, pTsma->targetDbFName, pTsma->targetTb, NULL, tReq)); - } + if (1 != pOut->pTsmas->size) { + ctgError("invalid tsma num:%d", (int32_t)pOut->pTsmas->size); + CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + + pRes->pRes = pOut; + pMsgCtx->out = NULL; + TSWAP(pTask->res, pCtx->pResList); + + STableTSMAInfo* pTsma = taosArrayGetP(pOut->pTsmas, 0); + if (NULL == pTsma) { + ctgError("fail to get the 0th STableTSMAInfo, totalNum:%d", (int32_t)taosArrayGetSize(pOut->pTsmas)); + CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); + } + + int32_t exists = false; + CTG_ERR_JRET(ctgTbMetaExistInCache(pCtg, pTsma->targetDbFName, pTsma->targetTb, &exists)); + if (!exists) { + TSWAP(pMsgCtx->lastOut, pMsgCtx->out); + CTG_RET(ctgGetTbMetaFromMnodeImpl(pCtg, pConn, pTsma->targetDbFName, pTsma->targetTb, NULL, tReq)); + } + break; - } + } default: - ASSERT(0); + ctgError("invalid reqType:%d while getting tsma rsp", reqType); + CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } _return: @@ -2484,53 +2491,53 @@ _return: tstrerror(code)); } } - + newCode = ctgHandleTaskEnd(pTask, code); if (newCode && TSDB_CODE_SUCCESS == code) { code = newCode; } - + CTG_RET(code); } - int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* pMsg, int32_t rspCode) { - bool taskDone = false; - int32_t code = 0; - SCtgTask* pTask = tReq->pTask; - SCatalog* pCtg = pTask->pJob->pCtg; - SCtgTbTSMACtx* pCtx = pTask->taskCtx; - SArray* pTsmas = NULL; - SHashObj* pVgHash = NULL; - SCtgDBCache* pDbCache = NULL; - STableTSMAInfo* pTsma = NULL; - SRequestConnInfo* pConn = &pTask->pJob->conn; + bool taskDone = false; + int32_t code = 0; + SCtgTask* pTask = tReq->pTask; + SCatalog* pCtg = pTask->pJob->pCtg; + SCtgTbTSMACtx* pCtx = pTask->taskCtx; + SArray* pTsmas = NULL; + SHashObj* pVgHash = NULL; + SCtgDBCache* pDbCache = NULL; + STableTSMAInfo* pTsma = NULL; + SRequestConnInfo* pConn = &pTask->pJob->conn; - SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, tReq->msgIdx); + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, tReq->msgIdx); if (NULL == pMsgCtx) { ctgError("fail to get the %dth SCtgMsgCtx", tReq->msgIdx); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); + SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); if (NULL == pFetch) { - ctgError("fail to get the %dth SCtgTSMAFetch, totalNum:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetches)); + ctgError("fail to get the %dth SCtgTSMAFetch, totalNum:%d", tReq->msgIdx, + (int32_t)taosArrayGetSize(pCtx->pFetches)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx); + SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx); if (NULL == pRes) { ctgError("fail to get the %dth SMetaRes, totalNum:%d", pFetch->resIdx, (int32_t)taosArrayGetSize(pCtx->pResList)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); + STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); if (NULL == pTbReq) { ctgError("fail to get the %dth STablesReq, totalNum:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SName* pTbName = taosArrayGet(pTbReq->pTables, pFetch->tbIdx); + SName* pTbName = taosArrayGet(pTbReq->pTables, pFetch->tbIdx); if (NULL == pTbName) { ctgError("fail to get the %dth SName, totalNum:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pTbReq->pTables)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); @@ -2548,8 +2555,8 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf if (pOut->pTsmas && taosArrayGetSize(pOut->pTsmas) > 0) { // fetch progress - (void)ctgAcquireVgInfoFromCache(pCtg, pTbReq->dbFName, &pDbCache); // ignore cache error - + (void)ctgAcquireVgInfoFromCache(pCtg, pTbReq->dbFName, &pDbCache); // ignore cache error + if (!pDbCache) { // do not know which vnodes to fetch, fetch vnode list first SBuildUseDBInput input = {0}; @@ -2569,13 +2576,13 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf taskDone = true; } } - + break; - } + } case TDMT_VND_GET_STREAM_PROGRESS: { SStreamProgressRsp rsp = {0}; CTG_ERR_JRET(ctgProcessRspMsg(&rsp, reqType, pMsg->pData, pMsg->len, rspCode, pMsgCtx->target)); - + // update progress into res STableTSMAInfoRsp* pTsmasRsp = pRes->pRes; SArray* pTsmas = pTsmasRsp->pTsmas; @@ -2586,15 +2593,15 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf ctgError("fail to get the %dth STableTSMAInfo, totalNum:%d", tsmaIdx, (int32_t)taosArrayGetSize(pTsmas)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + if (pTsmaInfo->rspTs == 0) { pTsmaInfo->fillHistoryFinished = true; } - + pTsmaInfo->rspTs = taosGetTimestampMs(); pTsmaInfo->delayDuration = TMAX(pRsp->progressDelay, pTsmaInfo->delayDuration); pTsmaInfo->fillHistoryFinished = pTsmaInfo->fillHistoryFinished && pRsp->fillHisFinished; - + qDebug("received stream progress for tsma %s rsp history: %d vnode: %d, delay: %" PRId64, pTsmaInfo->name, pRsp->fillHisFinished, pRsp->subFetchIdx, pRsp->progressDelay); @@ -2613,7 +2620,7 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf } break; - } + } case TDMT_MND_USE_DB: { SUseDbOutput* pOut = (SUseDbOutput*)pMsgCtx->out; @@ -2626,44 +2633,45 @@ int32_t ctgHandleGetTbTSMARsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf CTG_ERR_JRET(ctgGetTbMetaFromVnode(pCtg, pConn, pTbName, &vgInfo, NULL, tReq)); break; - } + } case FETCH_TSMA_STREAM_PROGRESS: { STableTSMAInfoRsp* pTsmas = pRes->pRes; TSWAP(pOut->dbVgroup->vgHash, pVgHash); CTG_ERR_JRET(ctgTsmaFetchStreamProgress(tReq, pVgHash, pTsmas)); break; - } + } default: - ASSERT(0); + ctgError("invalid fetchType:%d while getting tb tsma rsp", pFetch->fetchType); + CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } break; - } + } case TDMT_VND_TABLE_META: { // handle source tb meta - ASSERT(pFetch->fetchType == FETCH_TSMA_SOURCE_TB_META); STableMetaOutput* pOut = (STableMetaOutput*)pMsgCtx->out; pFetch->fetchType = FETCH_TB_TSMA; pFetch->tsmaSourceTbName = *pTbName; - + if (CTG_IS_META_NULL(pOut->metaType)) { ctgTaskError("no tbmeta found when fetching tsma source tb meta: %s.%s", pTbName->dbname, pTbName->tname); - (void)ctgRemoveTbMetaFromCache(pCtg, pTbName, false); // ignore cache error + (void)ctgRemoveTbMetaFromCache(pCtg, pTbName, false); // ignore cache error CTG_ERR_JRET(CTG_ERR_CODE_TABLE_NOT_EXIST); } - + if (META_TYPE_BOTH_TABLE == pOut->metaType) { // rewrite tsma fetch table with it's super table name (void)sprintf(pFetch->tsmaSourceTbName.tname, "%s", pOut->tbName); } - + CTG_ERR_JRET(ctgGetTbTSMAFromMnode(pCtg, pConn, &pFetch->tsmaSourceTbName, NULL, tReq, TDMT_MND_GET_TABLE_TSMA)); break; - } + } default: - ASSERT(0); + ctgError("invalid reqType:%d while getting tsma rsp", reqType); + CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } _return: @@ -2679,7 +2687,7 @@ _return: taosHashCleanup(pVgHash); } if (code) { - int32_t newCode = TSDB_CODE_SUCCESS; + int32_t newCode = TSDB_CODE_SUCCESS; SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx); if (NULL == pRes) { ctgError("fail to get the %dth SMetaRes, totalNum:%d", pFetch->resIdx, (int32_t)taosArrayGetSize(pCtx->pResList)); @@ -2697,7 +2705,7 @@ _return: if (newCode && TSDB_CODE_SUCCESS == code) { code = newCode; } - + bool allSubFetchFinished = false; if (pMsgCtx->reqType == TDMT_VND_GET_STREAM_PROGRESS) { allSubFetchFinished = atomic_add_fetch_32(&pFetch->finishedSubFetchNum, 1) >= pFetch->subFetchNum; @@ -2707,7 +2715,7 @@ _return: taskDone = true; } } - + if (pTask->res && taskDone) { int32_t newCode = ctgHandleTaskEnd(pTask, code); if (newCode && TSDB_CODE_SUCCESS == code) { @@ -2718,7 +2726,6 @@ _return: CTG_RET(code); } - int32_t ctgAsyncRefreshTbMeta(SCtgTaskReq* tReq, int32_t flag, SName* pName, int32_t* vgId) { SCtgTask* pTask = tReq->pTask; SCatalog* pCtg = pTask->pJob->pCtg; @@ -2814,7 +2821,7 @@ int32_t ctgLaunchGetTbMetasTask(SCtgTask* pTask) { ctgError("fail to get the %dth STablesReq, num:%d", i, dbNum); CTG_ERR_RET(TSDB_CODE_CTG_INVALID_INPUT); } - + ctgDebug("start to check tb metas in db %s, tbNum %ld", pReq->dbFName, taosArrayGetSize(pReq->pTables)); CTG_ERR_RET(ctgGetTbMetasFromCache(pCtg, pConn, pCtx, i, &fetchIdx, baseResIdx, pReq->pTables)); baseResIdx += taosArrayGetSize(pReq->pTables); @@ -2833,16 +2840,16 @@ int32_t ctgLaunchGetTbMetasTask(SCtgTask* pTask) { ctgError("taosArrayInit_s %d SCtgMsgCtx %d failed", pCtx->fetchNum, (int32_t)sizeof(SCtgMsgCtx)); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + for (int32_t i = 0; i < pCtx->fetchNum; ++i) { SCtgFetch* pFetch = taosArrayGet(pCtx->pFetchs, i); if (NULL == pFetch) { ctgError("fail to get the %dth fetch in pCtx->pFetchs, fetchNum:%d", i, pCtx->fetchNum); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + CTG_ERR_RET(ctgGetFetchName(pCtx->pNames, pFetch, &pName)); - + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, i); if (NULL == pMsgCtx) { ctgError("fail to get the %dth pMsgCtx", i); @@ -2885,7 +2892,7 @@ int32_t ctgLaunchGetDbVgTask(SCtgTask* pTask) { pMsgCtx->reqType = TDMT_MND_USE_DB; CTG_ERR_JRET(ctgBuildUseDbOutput((SUseDbOutput**)&pMsgCtx->out, dbCache->vgCache.vgInfo)); } - + CTG_ERR_JRET(ctgGenerateVgList(pCtg, dbCache->vgCache.vgInfo->vgHash, (SArray**)&pTask->res)); ctgReleaseVgInfoToCache(pCtg, dbCache); @@ -2936,7 +2943,8 @@ int32_t ctgLaunchGetTbHashTask(SCtgTask* pTask) { if (NULL == pTask->res) { CTG_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } - CTG_ERR_JRET(ctgGetVgInfoFromHashValue(pCtg, &pConn->mgmtEps, dbCache->vgCache.vgInfo, pCtx->pName, (SVgroupInfo*)pTask->res)); + CTG_ERR_JRET(ctgGetVgInfoFromHashValue(pCtg, &pConn->mgmtEps, dbCache->vgCache.vgInfo, pCtx->pName, + (SVgroupInfo*)pTask->res)); ctgReleaseVgInfoToCache(pCtg, dbCache); dbCache = NULL; @@ -2987,8 +2995,8 @@ int32_t ctgLaunchGetTbHashsTask(SCtgTask* pTask) { SCtgTaskReq tReq; tReq.pTask = pTask; tReq.msgIdx = -1; - CTG_ERR_JRET( - ctgGetVgInfosFromHashValue(pCtg, &pConn->mgmtEps, &tReq, dbCache->vgCache.vgInfo, pCtx, pReq->dbFName, pReq->pTables, false)); + CTG_ERR_JRET(ctgGetVgInfosFromHashValue(pCtg, &pConn->mgmtEps, &tReq, dbCache->vgCache.vgInfo, pCtx, + pReq->dbFName, pReq->pTables, false)); ctgReleaseVgInfoToCache(pCtg, dbCache); dbCache = NULL; @@ -2999,7 +3007,7 @@ int32_t ctgLaunchGetTbHashsTask(SCtgTask* pTask) { baseResIdx += taosArrayGetSize(pReq->pTables); int32_t inc = baseResIdx - taosArrayGetSize(pCtx->pResList); - for(int32_t j = 0; j < inc; ++j) { + for (int32_t j = 0; j < inc; ++j) { if (NULL == taosArrayPush(pCtx->pResList, &(SMetaRes){0})) { CTG_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } @@ -3022,12 +3030,12 @@ int32_t ctgLaunchGetTbHashsTask(SCtgTask* pTask) { } for (int32_t i = 0; i < pCtx->fetchNum; ++i) { - SCtgFetch* pFetch = taosArrayGet(pCtx->pFetchs, i); + SCtgFetch* pFetch = taosArrayGet(pCtx->pFetchs, i); if (NULL == pFetch) { ctgError("fail to get the %dth SCtgFetch, fetchNum:%d", i, pCtx->fetchNum); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + STablesReq* pReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); if (NULL == pFetch) { ctgError("fail to get the %dth SCtgFetch, fetchNum:%d", i, pCtx->fetchNum); @@ -3102,7 +3110,7 @@ int32_t ctgLaunchGetTbCfgTask(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; char dbFName[TSDB_DB_FNAME_LEN]; (void)tNameGetFullDbName(pCtx->pName, dbFName); - + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, -1); if (NULL == pMsgCtx) { ctgError("fail to get the %dth pMsgCtx", -1); @@ -3154,7 +3162,6 @@ _return: CTG_RET(code); } - int32_t ctgLaunchGetTbTagTask(SCtgTask* pTask) { int32_t code = 0; SCatalog* pCtg = pTask->pJob->pCtg; @@ -3164,7 +3171,7 @@ int32_t ctgLaunchGetTbTagTask(SCtgTask* pTask) { SCtgJob* pJob = pTask->pJob; char dbFName[TSDB_DB_FNAME_LEN]; (void)tNameGetFullDbName(pCtx->pName, dbFName); - + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, -1); if (NULL == pMsgCtx) { ctgError("fail to get the %dth pMsgCtx", -1); @@ -3201,7 +3208,6 @@ _return: CTG_RET(code); } - int32_t ctgLaunchGetQnodeTask(SCtgTask* pTask) { SCatalog* pCtg = pTask->pJob->pCtg; SRequestConnInfo* pConn = &pTask->pJob->conn; @@ -3218,7 +3224,7 @@ int32_t ctgLaunchGetQnodeTask(SCtgTask* pTask) { CTG_CACHE_NHIT_INC(CTG_CI_QNODE, 1); CTG_ERR_RET(ctgGetQnodeListFromMnode(pCtg, pConn, NULL, pTask)); - + return TSDB_CODE_SUCCESS; } @@ -3238,7 +3244,7 @@ int32_t ctgLaunchGetDnodeTask(SCtgTask* pTask) { CTG_CACHE_NHIT_INC(CTG_CI_DNODE, 1); CTG_ERR_RET(ctgGetDnodeListFromMnode(pCtg, pConn, NULL, pTask)); - + return TSDB_CODE_SUCCESS; } @@ -3257,7 +3263,7 @@ int32_t ctgLaunchGetDbCfgTask(SCtgTask* pTask) { pMsgCtx->pBatchs = pJob->pBatchs; } - SDbCfgInfo cfgInfo; + SDbCfgInfo cfgInfo; CTG_ERR_RET(ctgReadDBCfgFromCache(pCtg, pCtx->dbFName, &cfgInfo)); if (cfgInfo.cfgVersion < 0) { @@ -3397,13 +3403,13 @@ int32_t ctgLaunchGetUserTask(SCtgTask* pTask) { CTG_ERR_RET(pCtx->subTaskCode); } } - + CTG_ERR_RET(ctgChkAuthFromCache(pCtg, &pCtx->user, tbNotExists, &inCache, &rsp)); if (inCache) { pTask->res = rsp.pRawRes; - ctgTaskDebug("Final res got, pass:[%d,%d], pCond:[%p,%p]", - rsp.pRawRes->pass[0], rsp.pRawRes->pass[1], rsp.pRawRes->pCond[0], rsp.pRawRes->pCond[1]); + ctgTaskDebug("Final res got, pass:[%d,%d], pCond:[%p,%p]", rsp.pRawRes->pass[0], rsp.pRawRes->pass[1], + rsp.pRawRes->pCond[0], rsp.pRawRes->pCond[1]); CTG_ERR_RET(ctgHandleTaskEnd(pTask, 0)); return TSDB_CODE_SUCCESS; @@ -3419,7 +3425,7 @@ int32_t ctgLaunchGetUserTask(SCtgTask* pTask) { } else { CTG_ERR_RET(ctgGetUserDbAuthFromMnode(pCtg, pConn, pCtx->user.user, NULL, pTask)); } - + return TSDB_CODE_SUCCESS; } @@ -3452,16 +3458,16 @@ int32_t ctgLaunchGetViewsTask(SCtgTask* pTask) { bool tbMetaDone = false; SName* pName = NULL; -/* - ctgIsTaskDone(pJob, CTG_TASK_GET_TB_META_BATCH, &tbMetaDone); - if (tbMetaDone) { - CTG_ERR_RET(ctgBuildViewNullRes(pTask, pCtx)); - TSWAP(pTask->res, pCtx->pResList); + /* + ctgIsTaskDone(pJob, CTG_TASK_GET_TB_META_BATCH, &tbMetaDone); + if (tbMetaDone) { + CTG_ERR_RET(ctgBuildViewNullRes(pTask, pCtx)); + TSWAP(pTask->res, pCtx->pResList); - CTG_ERR_RET(ctgHandleTaskEnd(pTask, 0)); - return TSDB_CODE_SUCCESS; - } -*/ + CTG_ERR_RET(ctgHandleTaskEnd(pTask, 0)); + return TSDB_CODE_SUCCESS; + } + */ int32_t dbNum = taosArrayGetSize(pCtx->pNames); int32_t fetchIdx = 0; @@ -3493,14 +3499,14 @@ int32_t ctgLaunchGetViewsTask(SCtgTask* pTask) { } for (int32_t i = 0; i < pCtx->fetchNum; ++i) { - SCtgFetch* pFetch = taosArrayGet(pCtx->pFetchs, i); + SCtgFetch* pFetch = taosArrayGet(pCtx->pFetchs, i); if (NULL == pFetch) { ctgError("fail to get the %dth SCtgFetch, fetchNum:%d", i, pCtx->fetchNum); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } CTG_ERR_RET(ctgGetFetchName(pCtx->pNames, pFetch, &pName)); - + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, i); if (NULL == pMsgCtx) { ctgError("fail to get the %dth SCtgMsgCtx", i); @@ -3528,13 +3534,14 @@ int32_t ctgAsyncRefreshTbTsma(SCtgTaskReq* pReq, const SCtgTSMAFetch* pFetch) { SCtgTbTSMACtx* pTaskCtx = pTask->taskCtx; SCtgDBCache* pDbCache = NULL; - STablesReq* pTbReq = taosArrayGet(pTaskCtx->pNames, pFetch->dbIdx); + STablesReq* pTbReq = taosArrayGet(pTaskCtx->pNames, pFetch->dbIdx); if (NULL == pTbReq) { - ctgError("fail to get the %dth STablesReq, totalNum:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pTaskCtx->pNames)); + ctgError("fail to get the %dth STablesReq, totalNum:%d", pFetch->dbIdx, + (int32_t)taosArrayGetSize(pTaskCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - (void)ctgAcquireVgInfoFromCache(pCtg, pTbReq->dbFName, &pDbCache); // ignore error + (void)ctgAcquireVgInfoFromCache(pCtg, pTbReq->dbFName, &pDbCache); // ignore error if (pDbCache) { ctgReleaseVgInfoToCache(pCtg, pDbCache); } else { @@ -3544,7 +3551,7 @@ int32_t ctgAsyncRefreshTbTsma(SCtgTaskReq* pReq, const SCtgTSMAFetch* pFetch) { CTG_ERR_JRET(ctgGetDBVgInfoFromMnode(pCtg, pConn, &input, NULL, pReq)); } - + _return: return code; @@ -3566,11 +3573,11 @@ int32_t ctgLaunchGetTbTSMATask(SCtgTask* pTask) { ctgError("fail to get the %dth STablesReq, dbNum:%d", idx, dbNum); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + CTG_ERR_RET(ctgGetTbTSMAFromCache(pCtg, pCtx, idx, &fetchIdx, baseResIdx, pReq->pTables)); baseResIdx += taosArrayGetSize(pReq->pTables); } - + pCtx->fetchNum = taosArrayGetSize(pCtx->pFetches); if (pCtx->fetchNum <= 0) { TSWAP(pTask->res, pCtx->pResList); @@ -3583,7 +3590,7 @@ int32_t ctgLaunchGetTbTSMATask(SCtgTask* pTask) { ctgError("taosArrayInit_s %d SCtgMsgCtx %d failed", pCtx->fetchNum, (int32_t)sizeof(SCtgMsgCtx)); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + for (int32_t i = 0; i < pCtx->fetchNum; ++i) { SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, i); if (NULL == pFetch) { @@ -3591,19 +3598,19 @@ int32_t ctgLaunchGetTbTSMATask(SCtgTask* pTask) { CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - STablesReq* pReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); + STablesReq* pReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); if (NULL == pReq) { ctgError("fail to get the %dth STablesReq, totalNum:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SName* pName = taosArrayGet(pReq->pTables, pFetch->tbIdx); + SName* pName = taosArrayGet(pReq->pTables, pFetch->tbIdx); if (NULL == pName) { ctgError("fail to get the %dth SName, totalNum:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pReq->pTables)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, i); + SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, i); if (NULL == pMsgCtx) { ctgError("fail to get the %dth pMsgCtx", i); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); @@ -3628,7 +3635,8 @@ int32_t ctgLaunchGetTbTSMATask(SCtgTask* pTask) { break; } default: - ASSERT(0); + ctgError("invalid fetchType:%d in getting tb tsma task", pFetch->fetchType); + CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); break; } } @@ -3642,16 +3650,14 @@ int32_t ctgLaunchGetTSMATask(SCtgTask* pTask) { SRequestConnInfo* pConn = &pTask->pJob->conn; SArray* pRes = NULL; SCtgJob* pJob = pTask->pJob; - + // currently, only support fetching one tsma - ASSERT(pCtx->pNames->size == 1); STablesReq* pReq = taosArrayGet(pCtx->pNames, 0); if (NULL == pReq) { ctgError("fail to get the 0th STablesReq, totalNum:%d", (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - - ASSERT(pReq->pTables->size == 1); + SName* pTsmaName = taosArrayGet(pReq->pTables, 0); if (NULL == pReq) { ctgError("fail to get the 0th SName, totalNum:%d", (int32_t)taosArrayGetSize(pReq->pTables)); @@ -3666,17 +3672,17 @@ int32_t ctgLaunchGetTSMATask(SCtgTask* pTask) { ctgError("fail to get the 0th SCtgMsgCtx, taskType:%d", pTask->type); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + if (!pMsgCtx->pBatchs) { pMsgCtx->pBatchs = pJob->pBatchs; } - + SCtgTaskReq tReq = {.pTask = pTask, .msgIdx = 0}; if (NULL == taosArrayPush(pCtx->pResList, &(SMetaRes){0})) { ctgError("taosArrayPush SMetaRes failed, code:%x", terrno); CTG_ERR_RET(terrno); } - + CTG_ERR_RET(ctgGetTbTSMAFromMnode(pCtg, pConn, pTsmaName, NULL, &tReq, TDMT_MND_GET_TSMA)); } else { SMetaRes* pRes = taosArrayGet(pCtx->pResList, 0); @@ -3686,14 +3692,13 @@ int32_t ctgLaunchGetTSMATask(SCtgTask* pTask) { } STableTSMAInfoRsp* pRsp = (STableTSMAInfoRsp*)pRes->pRes; - ASSERT(pRsp->pTsmas->size == 1); - + const STSMACache* pTsma = taosArrayGetP(pRsp->pTsmas, 0); if (NULL == pTsma) { ctgError("fail to get the 0th STSMACache, totalNum:%d", (int32_t)taosArrayGetSize(pRsp->pTsmas)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + TSWAP(pTask->res, pCtx->pResList); // get tsma target stable meta if not existed in cache int32_t exists = false; @@ -3706,7 +3711,7 @@ int32_t ctgLaunchGetTSMATask(SCtgTask* pTask) { } else { CTG_ERR_RET(ctgHandleTaskEnd(pTask, 0)); } - + return TSDB_CODE_SUCCESS; } @@ -3756,8 +3761,9 @@ int32_t ctgGetTbCfgCb(SCtgTask* pTask) { if (NULL == pCtx->pVgInfo) { CTG_ERR_JRET(terrno); } - - CTG_ERR_JRET(ctgGetVgInfoFromHashValue(pTask->pJob->pCtg, &pTask->pJob->conn.mgmtEps, pDb, pCtx->pName, pCtx->pVgInfo)); + + CTG_ERR_JRET( + ctgGetVgInfoFromHashValue(pTask->pJob->pCtg, &pTask->pJob->conn.mgmtEps, pDb, pCtx->pName, pCtx->pVgInfo)); } CTG_RET(ctgLaunchGetTbCfgTask(pTask)); @@ -3773,17 +3779,18 @@ int32_t ctgGetTbTagCb(SCtgTask* pTask) { CTG_ERR_JRET(pTask->subRes.code); SCtgTbTagCtx* pCtx = (SCtgTbTagCtx*)pTask->taskCtx; - SDBVgInfo* pDb = (SDBVgInfo*)pTask->subRes.res; + SDBVgInfo* pDb = (SDBVgInfo*)pTask->subRes.res; if (NULL == pCtx->pVgInfo) { pCtx->pVgInfo = taosMemoryCalloc(1, sizeof(SVgroupInfo)); if (NULL == pCtx->pVgInfo) { CTG_ERR_JRET(terrno); } - - CTG_ERR_JRET(ctgGetVgInfoFromHashValue(pTask->pJob->pCtg, &pTask->pJob->conn.mgmtEps, pDb, pCtx->pName, pCtx->pVgInfo)); + + CTG_ERR_JRET( + ctgGetVgInfoFromHashValue(pTask->pJob->pCtg, &pTask->pJob->conn.mgmtEps, pDb, pCtx->pName, pCtx->pVgInfo)); } - + CTG_RET(ctgLaunchGetTbTagTask(pTask)); _return: @@ -3791,14 +3798,12 @@ _return: CTG_RET(ctgHandleTaskEnd(pTask, pTask->subRes.code)); } - - int32_t ctgGetUserCb(SCtgTask* pTask) { int32_t code = 0; SCtgUserCtx* pCtx = (SCtgUserCtx*)pTask->taskCtx; pCtx->subTaskCode = pTask->subRes.code; - + CTG_RET(ctgLaunchGetUserTask(pTask)); _return: @@ -3806,7 +3811,6 @@ _return: CTG_RET(ctgHandleTaskEnd(pTask, pTask->subRes.code)); } - int32_t ctgCompDbVgTasks(SCtgTask* pTask, void* param, bool* equal) { SCtgDbVgCtx* ctx = pTask->taskCtx; @@ -3816,7 +3820,7 @@ int32_t ctgCompDbVgTasks(SCtgTask* pTask, void* param, bool* equal) { } int32_t ctgCompTbMetaTasks(SCtgTask* pTask, void* param, bool* equal) { - SCtgTbMetaCtx* ctx = pTask->taskCtx; + SCtgTbMetaCtx* ctx = pTask->taskCtx; SCtgTbMetaParam* pParam = (SCtgTbMetaParam*)param; *equal = tNameTbNameEqual(ctx->pName, (SName*)pParam->pName); @@ -3888,7 +3892,7 @@ int32_t ctgSearchExistingTask(SCtgJob* pJob, CTG_TASK_TYPE type, void* param, in qError("fail to get the %dth task, taskNum:%d", i, taskNum); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + if (type != pTask->type) { continue; } @@ -3921,7 +3925,7 @@ int32_t ctgSetSubTaskCb(SCtgTask* pSub, SCtgTask* pTask) { qError("fail to get the -1th SCtgMsgCtx"); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + SCtgMsgCtx* pSubMsgCtx = CTG_GET_TASK_MSGCTX(pSub, -1); if (NULL == pSubMsgCtx) { qError("fail to get the -1th sub SCtgMsgCtx"); @@ -3964,7 +3968,6 @@ SCtgTask* ctgGetTask(SCtgJob* pJob, int32_t taskId) { return NULL; } - int32_t ctgLaunchSubTask(SCtgTask** ppTask, CTG_TASK_TYPE type, ctgSubTaskCbFp fp, void* param) { SCtgJob* pJob = (*ppTask)->pJob; int32_t subTaskId = -1; @@ -3987,7 +3990,7 @@ int32_t ctgLaunchSubTask(SCtgTask** ppTask, CTG_TASK_TYPE type, ctgSubTaskCbFp f qError("fail to get the %dth sub SCtgTask, taskNum:%d", subTaskId, (int32_t)taosArrayGetSize(pJob->pTasks)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + if (newTask) { pSub->subTask = true; } @@ -4025,8 +4028,8 @@ int32_t ctgLaunchJob(SCtgJob* pJob) { qError("fail to get the %dth task, taskNum:%d", i, taskNum); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - - qDebug("QID:0x%" PRIx64 " ctg launch [%dth] task", pJob->queryId, pTask->taskId); + + qDebug("qid:0x%" PRIx64 " ctg launch [%dth] task", pJob->queryId, pTask->taskId); CTG_ERR_RET((*gCtgAsyncFps[pTask->type].launchFp)(pTask)); pTask = taosArrayGet(pJob->pTasks, i); @@ -4039,7 +4042,7 @@ int32_t ctgLaunchJob(SCtgJob* pJob) { } if (taskNum <= 0) { - qDebug("QID:0x%" PRIx64 " ctg call user callback with rsp %s", pJob->queryId, tstrerror(pJob->jobResCode)); + qDebug("qid:0x%" PRIx64 " ctg call user callback with rsp %s", pJob->queryId, tstrerror(pJob->jobResCode)); CTG_ERR_RET(taosAsyncExec(ctgCallUserCb, pJob, NULL)); #if CTG_BATCH_FETCH diff --git a/source/libs/catalog/src/ctgRemote.c b/source/libs/catalog/src/ctgRemote.c index a312dce164..fb444ed9d9 100644 --- a/source/libs/catalog/src/ctgRemote.c +++ b/source/libs/catalog/src/ctgRemote.c @@ -42,11 +42,12 @@ int32_t ctgHandleBatchRsp(SCtgJob* pJob, SCtgTaskCallbackParam* cbParam, SDataBu msgNum = taosArrayGetSize(batchRsp.pRsps); } - if (ASSERTS(taskNum == msgNum || 0 == msgNum, "taskNum %d mis-match msgNum %d", taskNum, msgNum)) { + if (taskNum != msgNum && 0 != msgNum) { + ctgError("taskNum %d mis-match msgNum %d", taskNum, msgNum); msgNum = 0; } - ctgDebug("QID:0x%" PRIx64 " ctg got batch %d rsp %s", pJob->queryId, cbParam->batchId, + ctgDebug("qid:0x%" PRIx64 " ctg got batch %d rsp %s", pJob->queryId, cbParam->batchId, TMSG_INFO(cbParam->reqType + 1)); SHashObj* pBatchs = taosHashInit(taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), false, HASH_NO_LOCK); @@ -56,13 +57,13 @@ int32_t ctgHandleBatchRsp(SCtgJob* pJob, SCtgTaskCallbackParam* cbParam, SDataBu } for (int32_t i = 0; i < taskNum; ++i) { - int32_t* taskId = taosArrayGet(cbParam->taskId, i); + int32_t* taskId = taosArrayGet(cbParam->taskId, i); if (NULL == taskId) { ctgError("taosArrayGet %d taskId failed, total:%d", i, (int32_t)taosArrayGetSize(cbParam->taskId)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - int32_t* msgIdx = taosArrayGet(cbParam->msgIdx, i); + int32_t* msgIdx = taosArrayGet(cbParam->msgIdx, i); if (NULL == msgIdx) { ctgError("taosArrayGet %d msgIdx failed, total:%d", i, (int32_t)taosArrayGetSize(cbParam->msgIdx)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); @@ -77,7 +78,9 @@ int32_t ctgHandleBatchRsp(SCtgJob* pJob, SCtgTaskCallbackParam* cbParam, SDataBu if (msgNum > 0) { pRsp = taosArrayGet(batchRsp.pRsps, i); - if (ASSERTS(pRsp->msgIdx == *msgIdx, "rsp msgIdx %d mis-match msgIdx %d", pRsp->msgIdx, *msgIdx)) { + if (pRsp->msgIdx != *msgIdx) { + ctgError("rsp msgIdx %d mis-match msgIdx %d", pRsp->msgIdx, *msgIdx); + pRsp = &rsp; pRsp->msgIdx = *msgIdx; pRsp->reqType = -1; @@ -111,10 +114,11 @@ int32_t ctgHandleBatchRsp(SCtgJob* pJob, SCtgTaskCallbackParam* cbParam, SDataBu pMsgCtx->pBatchs = pBatchs; - ctgDebug("QID:0x%" PRIx64 " ctg task %d idx %d start to handle rsp %s, pBatchs: %p", pJob->queryId, pTask->taskId, + ctgDebug("qid:0x%" PRIx64 " ctg task %d idx %d start to handle rsp %s, pBatchs: %p", pJob->queryId, pTask->taskId, pRsp->msgIdx, TMSG_INFO(taskMsg.msgType + 1), pBatchs); - (void)(*gCtgAsyncFps[pTask->type].handleRspFp)(&tReq, pRsp->reqType, &taskMsg, (pRsp->rspCode ? pRsp->rspCode : rspCode)); // error handled internal + (void)(*gCtgAsyncFps[pTask->type].handleRspFp)( + &tReq, pRsp->reqType, &taskMsg, (pRsp->rspCode ? pRsp->rspCode : rspCode)); // error handled internal } CTG_ERR_JRET(ctgLaunchBatchs(pJob->pCtg, pJob, pBatchs)); @@ -417,19 +421,19 @@ int32_t ctgHandleMsgCallback(void* param, SDataBuf* pMsg, int32_t rspCode) { if (TDMT_VND_BATCH_META == cbParam->reqType || TDMT_MND_BATCH_META == cbParam->reqType) { CTG_ERR_JRET(ctgHandleBatchRsp(pJob, cbParam, pMsg, rspCode)); } else { - int32_t* taskId = taosArrayGet(cbParam->taskId, 0); + int32_t* taskId = taosArrayGet(cbParam->taskId, 0); if (NULL == taskId) { ctgError("taosArrayGet %d taskId failed, total:%d", 0, (int32_t)taosArrayGetSize(cbParam->taskId)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + SCtgTask* pTask = taosArrayGet(pJob->pTasks, *taskId); if (NULL == pTask) { ctgError("taosArrayGet %d SCtgTask failed, total:%d", *taskId, (int32_t)taosArrayGetSize(pJob->pTasks)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - qDebug("QID:0x%" PRIx64 " ctg task %d start to handle rsp %s", pJob->queryId, pTask->taskId, + qDebug("qid:0x%" PRIx64 " ctg task %d start to handle rsp %s", pJob->queryId, pTask->taskId, TMSG_INFO(cbParam->reqType + 1)); #if CTG_BATCH_FETCH @@ -445,7 +449,7 @@ int32_t ctgHandleMsgCallback(void* param, SDataBuf* pMsg, int32_t rspCode) { ctgError("get task %d SCtgMsgCtx failed, taskType:%d", -1, pTask->type); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + pMsgCtx->pBatchs = pBatchs; #endif @@ -534,7 +538,7 @@ int32_t ctgAsyncSendMsg(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob* pJob, CTG_ERR_JRET(code); } - ctgDebug("ctg req msg sent, reqId:0x%" PRIx64 ", msg type:%d, %s", pJob->queryId, msgType, TMSG_INFO(msgType)); + ctgDebug("ctg req msg sent, qid:0x%" PRIx64 ", msg type:%d, %s", pJob->queryId, msgType, TMSG_INFO(msgType)); return TSDB_CODE_SUCCESS; _return: @@ -558,9 +562,9 @@ int32_t ctgAddBatch(SCatalog* pCtg, int32_t vgId, SRequestConnInfo* pConn, SCtgT ctgError("get task %d SCtgMsgCtx failed, taskType:%d", tReq->msgIdx, pTask->type); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - - SHashObj* pBatchs = pMsgCtx->pBatchs; - SCtgBatch* pBatch = taosHashGet(pBatchs, &vgId, sizeof(vgId)); + + SHashObj* pBatchs = pMsgCtx->pBatchs; + SCtgBatch* pBatch = taosHashGet(pBatchs, &vgId, sizeof(vgId)); if (NULL == pBatch) { newBatch.pMsgs = taosArrayInit(pJob->subTaskNum, sizeof(SBatchMsg)); newBatch.pTaskIds = taosArrayInit(pJob->subTaskNum, sizeof(int32_t)); @@ -599,7 +603,7 @@ int32_t ctgAddBatch(SCatalog* pCtg, int32_t vgId, SRequestConnInfo* pConn, SCtgT SCtgTbMetasCtx* ctx = (SCtgTbMetasCtx*)pTask->taskCtx; SCtgFetch* fetch = taosArrayGet(ctx->pFetchs, tReq->msgIdx); CTG_ERR_JRET(ctgGetFetchName(ctx->pNames, fetch, &pName)); - } else if (CTG_TASK_GET_TB_TSMA == pTask->type){ + } else if (CTG_TASK_GET_TB_TSMA == pTask->type) { SCtgTbTSMACtx* pCtx = pTask->taskCtx; SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); @@ -616,10 +620,11 @@ int32_t ctgAddBatch(SCatalog* pCtg, int32_t vgId, SRequestConnInfo* pConn, SCtgT SCtgTbTSMACtx* pCtx = pTask->taskCtx; SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); if (NULL == pFetch) { - ctgError("fail to get %d SCtgTSMAFetch, totalFetchs:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetches)); + ctgError("fail to get %d SCtgTSMAFetch, totalFetchs:%d", tReq->msgIdx, + (int32_t)taosArrayGetSize(pCtx->pFetches)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); } - STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); + STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); if (NULL == pTbReq) { ctgError("fail to get %d STablesReq, totalTables:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); @@ -675,7 +680,7 @@ int32_t ctgAddBatch(SCatalog* pCtg, int32_t vgId, SRequestConnInfo* pConn, SCtgT SCtgTbMetasCtx* ctx = (SCtgTbMetasCtx*)pTask->taskCtx; SCtgFetch* fetch = taosArrayGet(ctx->pFetchs, tReq->msgIdx); CTG_ERR_JRET(ctgGetFetchName(ctx->pNames, fetch, &pName)); - } else if (CTG_TASK_GET_TB_TSMA == pTask->type){ + } else if (CTG_TASK_GET_TB_TSMA == pTask->type) { SCtgTbTSMACtx* pCtx = pTask->taskCtx; SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); @@ -689,22 +694,23 @@ int32_t ctgAddBatch(SCatalog* pCtg, int32_t vgId, SRequestConnInfo* pConn, SCtgT pName = ctx->pName; } } else if (TDMT_VND_GET_STREAM_PROGRESS == msgType) { - SCtgTbTSMACtx* pCtx = pTask->taskCtx; - SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); - if (NULL == pFetch) { - ctgError("fail to get %d SCtgTSMAFetch, totalFetchs:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetches)); - CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); - } - STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); - if (NULL == pTbReq) { - ctgError("fail to get %d STablesReq, totalTables:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pCtx->pNames)); - CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); - } - pName = taosArrayGet(pTbReq->pTables, pFetch->tbIdx); - if (NULL == pName) { - ctgError("fail to get %d SName, totalTables:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pTbReq->pTables)); - CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); - } + SCtgTbTSMACtx* pCtx = pTask->taskCtx; + SCtgTSMAFetch* pFetch = taosArrayGet(pCtx->pFetches, tReq->msgIdx); + if (NULL == pFetch) { + ctgError("fail to get %d SCtgTSMAFetch, totalFetchs:%d", tReq->msgIdx, + (int32_t)taosArrayGetSize(pCtx->pFetches)); + CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); + } + STablesReq* pTbReq = taosArrayGet(pCtx->pNames, pFetch->dbIdx); + if (NULL == pTbReq) { + ctgError("fail to get %d STablesReq, totalTables:%d", pFetch->dbIdx, (int32_t)taosArrayGetSize(pCtx->pNames)); + CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); + } + pName = taosArrayGet(pTbReq->pTables, pFetch->tbIdx); + if (NULL == pName) { + ctgError("fail to get %d SName, totalTables:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pTbReq->pTables)); + CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR); + } } else { ctgError("invalid vnode msgType %d", msgType); CTG_ERR_JRET(TSDB_CODE_APP_ERROR); @@ -771,7 +777,7 @@ int32_t ctgLaunchBatchs(SCatalog* pCtg, SCtgJob* pJob, SHashObj* pBatchs) { SCtgBatch* pBatch = (SCtgBatch*)p; int32_t msgSize = 0; - ctgDebug("QID:0x%" PRIx64 " ctg start to launch batch %d", pJob->queryId, pBatch->batchId); + ctgDebug("qid:0x%" PRIx64 " ctg start to launch batch %d", pJob->queryId, pBatch->batchId); CTG_ERR_JRET(ctgBuildBatchReqMsg(pBatch, *vgId, &msg, &msgSize)); code = ctgAsyncSendMsg(pCtg, &pBatch->conn, pJob, pBatch->pTaskIds, pBatch->batchId, pBatch->pMsgIdxs, @@ -1629,9 +1635,9 @@ int32_t ctgGetViewInfoFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, SName* } int32_t ctgGetTbTSMAFromMnode(SCatalog* pCtg, SRequestConnInfo* pConn, const SName* name, STableTSMAInfoRsp* out, - SCtgTaskReq* tReq, int32_t reqType) { - char* msg = NULL; - int32_t msgLen = 0; + SCtgTaskReq* tReq, int32_t reqType) { + char* msg = NULL; + int32_t msgLen = 0; SCtgTask* pTask = tReq ? tReq->pTask : NULL; void* (*mallocFp)(int64_t) = pTask ? (MallocType)taosMemoryMalloc : (MallocType)rpcMallocCont; char tbFName[TSDB_TABLE_FNAME_LEN]; @@ -1720,7 +1726,7 @@ int32_t ctgGetStreamProgressFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, c #if CTG_BATCH_FETCH CTG_RET(ctgAddBatch(pCtg, vgroupInfo->vgId, &vConn, tReq, reqType, msg, msgLen)); #else - char dbFName[TSDB_DB_FNAME_LEN]; + char dbFName[TSDB_DB_FNAME_LEN]; (void)tNameGetFullDbName(pTbName, dbFName); SArray* pTaskId = taosArrayInit(1, sizeof(int32_t)); if (NULL == pTaskId) { @@ -1731,7 +1737,8 @@ int32_t ctgGetStreamProgressFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, c CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - CTG_RET(ctgAsyncSendMsg(pCtg, &vConn, pTask->pJob, pTaskId, -1, NULL, dbFName, vgroupInfo->vgId, reqType, msg, msgLen)); + CTG_RET( + ctgAsyncSendMsg(pCtg, &vConn, pTask->pJob, pTaskId, -1, NULL, dbFName, vgroupInfo->vgId, reqType, msg, msgLen)); #endif } diff --git a/source/libs/catalog/src/ctgUtil.c b/source/libs/catalog/src/ctgUtil.c index fa4df70f59..daa2199421 100644 --- a/source/libs/catalog/src/ctgUtil.c +++ b/source/libs/catalog/src/ctgUtil.c @@ -245,36 +245,36 @@ void ctgFreeStbMetaCache(SCtgDBCache* dbCache) { void ctgFreeTbCacheImpl(SCtgTbCache* pCache, bool lock) { if (pCache->pMeta) { - if (lock) { + if (lock) { CTG_LOCK(CTG_WRITE, &pCache->metaLock); } taosMemoryFreeClear(pCache->pMeta); - if (lock) { + if (lock) { CTG_UNLOCK(CTG_WRITE, &pCache->metaLock); } } if (pCache->pIndex) { - if (lock) { + if (lock) { CTG_LOCK(CTG_WRITE, &pCache->indexLock); } taosArrayDestroyEx(pCache->pIndex->pIndex, tFreeSTableIndexInfo); taosMemoryFreeClear(pCache->pIndex); - if (lock) { + if (lock) { CTG_UNLOCK(CTG_WRITE, &pCache->indexLock); } } } void ctgFreeViewCacheImpl(SCtgViewCache* pCache, bool lock) { - if (lock) { + if (lock) { CTG_LOCK(CTG_WRITE, &pCache->viewLock); } if (pCache->pMeta) { ctgFreeSViewMeta(pCache->pMeta); taosMemoryFreeClear(pCache->pMeta); } - if (lock) { + if (lock) { CTG_UNLOCK(CTG_WRITE, &pCache->viewLock); } } @@ -398,10 +398,10 @@ void ctgFreeHandleImpl(SCatalog* pCtg) { } int32_t ctgRemoveCacheUser(SCatalog* pCtg, SCtgUserAuth* pUser, const char* user) { - CTG_LOCK(CTG_WRITE, &pUser->lock); + CTG_LOCK(CTG_WRITE, &pUser->lock); ctgFreeSCtgUserAuth(pUser); - CTG_UNLOCK(CTG_WRITE, &pUser->lock); - + CTG_UNLOCK(CTG_WRITE, &pUser->lock); + if (taosHashRemove(pCtg->userCache, user, strlen(user)) == 0) { return 0; // user found and removed } @@ -431,12 +431,12 @@ void ctgFreeHandle(SCatalog* pCtg) { ctgInfo("handle freed, clusterId:0x%" PRIx64, clusterId); } -void ctgClearHandleMeta(SCatalog* pCtg, int64_t *pClearedSize, int64_t *pCleardNum, bool *roundDone) { +void ctgClearHandleMeta(SCatalog* pCtg, int64_t* pClearedSize, int64_t* pCleardNum, bool* roundDone) { int64_t cacheSize = 0; - void* pIter = taosHashIterate(pCtg->dbCache, NULL); + void* pIter = taosHashIterate(pCtg->dbCache, NULL); while (pIter) { SCtgDBCache* dbCache = pIter; - + SCtgTbCache* pCache = taosHashIterate(dbCache->tbCache, NULL); while (NULL != pCache) { size_t len = 0; @@ -446,10 +446,11 @@ void ctgClearHandleMeta(SCatalog* pCtg, int64_t *pClearedSize, int64_t *pCleardN pCache = taosHashIterate(dbCache->tbCache, pCache); continue; } - + (void)taosHashRemove(dbCache->tbCache, key, len); - - cacheSize = len + sizeof(SCtgTbCache) + ctgGetTbMetaCacheSize(pCache->pMeta) + ctgGetTbIndexCacheSize(pCache->pIndex); + + cacheSize = + len + sizeof(SCtgTbCache) + ctgGetTbMetaCacheSize(pCache->pMeta) + ctgGetTbIndexCacheSize(pCache->pIndex); (void)atomic_sub_fetch_64(&dbCache->dbCacheSize, cacheSize); *pClearedSize += cacheSize; (*pCleardNum)++; @@ -457,14 +458,14 @@ void ctgClearHandleMeta(SCatalog* pCtg, int64_t *pClearedSize, int64_t *pCleardN if (pCache->pMeta) { CTG_META_NUM_DEC(pCache->pMeta->tableType); } - + ctgFreeTbCacheImpl(pCache, true); - + if (*pCleardNum >= CTG_CLEAR_CACHE_ROUND_TB_NUM) { taosHashCancelIterate(dbCache->tbCache, pCache); goto _return; } - + pCache = taosHashIterate(dbCache->tbCache, pCache); } @@ -478,12 +479,12 @@ _return: } } -void ctgClearAllHandleMeta(int64_t *clearedSize, int64_t *clearedNum, bool *roundDone) { - SCatalog *pCtg = NULL; +void ctgClearAllHandleMeta(int64_t* clearedSize, int64_t* clearedNum, bool* roundDone) { + SCatalog* pCtg = NULL; - void *pIter = taosHashIterate(gCtgMgmt.pCluster, NULL); + void* pIter = taosHashIterate(gCtgMgmt.pCluster, NULL); while (pIter) { - pCtg = *(SCatalog **)pIter; + pCtg = *(SCatalog**)pIter; if (pCtg) { ctgClearHandleMeta(pCtg, clearedSize, clearedNum, roundDone); @@ -503,7 +504,7 @@ void ctgClearHandle(SCatalog* pCtg) { } int64_t clusterId = pCtg->clusterId; - + ctgFreeMetaRent(&pCtg->dbRent); ctgFreeMetaRent(&pCtg->stbRent); ctgFreeMetaRent(&pCtg->viewRent); @@ -725,7 +726,7 @@ void ctgFreeTbTSMARes(void* res) { SMetaRes* pRes = res; if (pRes->pRes) { - STableTSMAInfoRsp * pTsmaRsp = pRes->pRes; + STableTSMAInfoRsp* pTsmaRsp = pRes->pRes; tFreeTableTSMAInfoRsp(pTsmaRsp); taosMemoryFree(pTsmaRsp); pRes->pRes = NULL; @@ -1059,7 +1060,7 @@ void ctgFreeJob(void* job) { taosMemoryFree(job); - qDebug("QID:0x%" PRIx64 ", ctg job 0x%" PRIx64 " freed", qid, rid); + qDebug("qid:0x%" PRIx64 ", ctg job 0x%" PRIx64 " freed", qid, rid); } int32_t ctgUpdateMsgCtx(SCtgMsgCtx* pCtx, int32_t reqType, void* out, char* target) { @@ -1175,7 +1176,8 @@ int32_t ctgHashValueComp(void const* lp, void const* rp) { return 0; } -int32_t ctgGetVgInfoFromHashValue(SCatalog* pCtg, SEpSet* pMgmtEps, SDBVgInfo* dbInfo, const SName* pTableName, SVgroupInfo* pVgroup) { +int32_t ctgGetVgInfoFromHashValue(SCatalog* pCtg, SEpSet* pMgmtEps, SDBVgInfo* dbInfo, const SName* pTableName, + SVgroupInfo* pVgroup) { int32_t code = 0; CTG_ERR_RET(ctgMakeVgArray(dbInfo)); @@ -1188,7 +1190,7 @@ int32_t ctgGetVgInfoFromHashValue(SCatalog* pCtg, SEpSet* pMgmtEps, SDBVgInfo* d if (pMgmtEps) { TAOS_MEMCPY(&pVgroup->epSet, pMgmtEps, sizeof(pVgroup->epSet)); } - + return TSDB_CODE_SUCCESS; } @@ -1235,16 +1237,16 @@ int32_t ctgGetVgInfoFromHashValue(SCatalog* pCtg, SEpSet* pMgmtEps, SDBVgInfo* d CTG_RET(code); } -int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskReq* tReq, SDBVgInfo* dbInfo, SCtgTbHashsCtx* pCtx, - char* dbFName, SArray* pNames, bool update) { - int32_t code = 0; - SCtgTask* pTask = tReq->pTask; - SMetaRes res = {0}; +int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskReq* tReq, SDBVgInfo* dbInfo, + SCtgTbHashsCtx* pCtx, char* dbFName, SArray* pNames, bool update) { + int32_t code = 0; + SCtgTask* pTask = tReq->pTask; + SMetaRes res = {0}; SVgroupInfo* vgInfo = NULL; CTG_ERR_RET(ctgMakeVgArray(dbInfo)); - int32_t tbNum = taosArrayGetSize(pNames); + int32_t tbNum = taosArrayGetSize(pNames); char* pSep = strchr(dbFName, '.'); if (pSep && IS_SYS_DBNAME(pSep + 1)) { @@ -1253,7 +1255,7 @@ int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskR if (pMgmgEpSet) { TAOS_MEMCPY(&mgmtInfo.epSet, pMgmgEpSet, sizeof(mgmtInfo.epSet)); } - + for (int32_t i = 0; i < tbNum; ++i) { vgInfo = taosMemoryMalloc(sizeof(SVgroupInfo)); if (NULL == vgInfo) { @@ -1271,12 +1273,13 @@ int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskR ctgError("fail to get the %dth SCtgFetch, total:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetchs)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx + i); + SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx + i); if (NULL == pFetch) { - ctgError("fail to get the %dth SMetaRes, total:%d", pFetch->resIdx + i, (int32_t)taosArrayGetSize(pCtx->pResList)); + ctgError("fail to get the %dth SMetaRes, total:%d", pFetch->resIdx + i, + (int32_t)taosArrayGetSize(pCtx->pResList)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + pRes->pRes = vgInfo; } else { res.pRes = vgInfo; @@ -1285,7 +1288,7 @@ int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskR } } } - + return TSDB_CODE_SUCCESS; } @@ -1319,9 +1322,10 @@ int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskR ctgError("fail to get the %dth SCtgFetch, total:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetchs)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx + i); + SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx + i); if (NULL == pRes) { - ctgError("fail to get the %dth SMetaRes, total:%d", pFetch->resIdx + i, (int32_t)taosArrayGetSize(pCtx->pResList)); + ctgError("fail to get the %dth SMetaRes, total:%d", pFetch->resIdx + i, + (int32_t)taosArrayGetSize(pCtx->pResList)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -1380,9 +1384,10 @@ int32_t ctgGetVgInfosFromHashValue(SCatalog* pCtg, SEpSet* pMgmgEpSet, SCtgTaskR ctgError("fail to get the %dth SCtgFetch, total:%d", tReq->msgIdx, (int32_t)taosArrayGetSize(pCtx->pFetchs)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx + i); + SMetaRes* pRes = taosArrayGet(pCtx->pResList, pFetch->resIdx + i); if (NULL == pRes) { - ctgError("fail to get the %dth SMetaRes, total:%d", pFetch->resIdx + i, (int32_t)taosArrayGetSize(pCtx->pResList)); + ctgError("fail to get the %dth SMetaRes, total:%d", pFetch->resIdx + i, + (int32_t)taosArrayGetSize(pCtx->pResList)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } @@ -1531,7 +1536,7 @@ int32_t ctgMakeVgArray(SDBVgInfo* dbInfo) { taosHashCancelIterate(dbInfo->vgHash, pIter); CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + pIter = taosHashIterate(dbInfo->vgHash, pIter); } @@ -1581,7 +1586,7 @@ int32_t ctgCloneVgInfo(SDBVgInfo* src, SDBVgInfo** dst) { if (NULL == (*dst)->vgArray) { taosHashCleanup((*dst)->vgHash); taosMemoryFreeClear(*dst); - CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); + CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } } @@ -1603,7 +1608,7 @@ int32_t ctgCloneMetaOutput(STableMetaOutput* output, STableMetaOutput** pOutput) if (useCompress(output->tbMeta->tableType) && (*pOutput)->tbMeta->schemaExt) { schemaExtSize = output->tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt); } - + (*pOutput)->tbMeta = taosMemoryMalloc(metaSize + schemaExtSize); qDebug("tbMeta cloned, size:%d, p:%p", metaSize, (*pOutput)->tbMeta); if (NULL == (*pOutput)->tbMeta) { @@ -1614,7 +1619,7 @@ int32_t ctgCloneMetaOutput(STableMetaOutput* output, STableMetaOutput** pOutput) TAOS_MEMCPY((*pOutput)->tbMeta, output->tbMeta, metaSize); if (useCompress(output->tbMeta->tableType) && (*pOutput)->tbMeta->schemaExt) { - (*pOutput)->tbMeta->schemaExt = (SSchemaExt *)((char *)(*pOutput)->tbMeta + metaSize); + (*pOutput)->tbMeta->schemaExt = (SSchemaExt*)((char*)(*pOutput)->tbMeta + metaSize); TAOS_MEMCPY((*pOutput)->tbMeta->schemaExt, output->tbMeta->schemaExt, schemaExtSize); } else { (*pOutput)->tbMeta->schemaExt = NULL; @@ -1683,7 +1688,7 @@ int32_t ctgGetTablesReqNum(SArray* pList) { qError("fail to get the %dth STablesReq, total:%d", i, (int32_t)taosArrayGetSize(pList)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + total += taosArrayGetSize(pReq->pTables); } @@ -1718,7 +1723,7 @@ int32_t ctgGetFetchName(SArray* pNames, SCtgFetch* pFetch, SName** ppName) { qError("fail to get the %dth tb in pTables, tbNum:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pReq->pTables)); return TSDB_CODE_CTG_INTERNAL_ERROR; } - + *ppName = (SName*)taosArrayGet(pReq->pTables, pFetch->tbIdx); if (NULL == *ppName) { qError("fail to get the %dth tb in pTables, tbNum:%d", pFetch->tbIdx, (int32_t)taosArrayGetSize(pReq->pTables)); @@ -1728,7 +1733,7 @@ int32_t ctgGetFetchName(SArray* pNames, SCtgFetch* pFetch, SName** ppName) { return TSDB_CODE_SUCCESS; } -static int32_t ctgCloneDbVgroup(void* pSrc, void** ppDst) { +static int32_t ctgCloneDbVgroup(void* pSrc, void** ppDst) { #if 0 if (NULL == pSrc) { *ppDst = NULL; @@ -1749,10 +1754,10 @@ int32_t ctgCloneDbCfgInfo(void* pSrc, SDbCfgInfo** ppDst) { if (NULL == pDst) { return terrno; } - + TAOS_MEMCPY(pDst, pSrc, sizeof(SDbCfgInfo)); - if (((SDbCfgInfo *)pSrc)->pRetensions) { - pDst->pRetensions = taosArrayDup(((SDbCfgInfo *)pSrc)->pRetensions, NULL); + if (((SDbCfgInfo*)pSrc)->pRetensions) { + pDst->pRetensions = taosArrayDup(((SDbCfgInfo*)pSrc)->pRetensions, NULL); if (NULL == pDst->pRetensions) { taosMemoryFree(pDst); return terrno; @@ -1760,12 +1765,12 @@ int32_t ctgCloneDbCfgInfo(void* pSrc, SDbCfgInfo** ppDst) { } *ppDst = pDst; - + return TSDB_CODE_SUCCESS; } -static void ctgFreeDbCfgInfo(void* p) { - SDbCfgInfo* pDst = (SDbCfgInfo *)((SMetaRes*)p)->pRes; +static void ctgFreeDbCfgInfo(void* p) { + SDbCfgInfo* pDst = (SDbCfgInfo*)((SMetaRes*)p)->pRes; freeDbCfgInfo(pDst); } @@ -1829,9 +1834,9 @@ static int32_t ctgCloneVgroupInfo(void* pSrc, void** ppDst) { static void ctgFreeVgroupInfo(void* p) { taosMemoryFree(((SMetaRes*)p)->pRes); } -static int32_t ctgCloneTableIndexs(void* pSrc, void** ppDst) { +static int32_t ctgCloneTableIndexs(void* pSrc, void** ppDst) { #if 0 - return taosArrayDup((const SArray*)pSrc, NULL); + return taosArrayDup((const SArray*)pSrc, NULL); #else return TSDB_CODE_CTG_INTERNAL_ERROR; #endif @@ -1884,9 +1889,9 @@ static int32_t ctgCloneUserAuth(void* pSrc) { static void ctgFreeUserAuth(void* p) { taosMemoryFree(((SMetaRes*)p)->pRes); } -static int32_t ctgCloneQnodeList(void* pSrc) { +static int32_t ctgCloneQnodeList(void* pSrc) { #if 0 - return taosArrayDup((const SArray*)pSrc, NULL); + return taosArrayDup((const SArray*)pSrc, NULL); #else return TSDB_CODE_CTG_INTERNAL_ERROR; #endif @@ -1905,9 +1910,9 @@ static void ctgFreeQnodeList(void* p) { taosArrayDestroy((SArray*)((SMetaRes*)p) static void ctgFreeTableCfg(void* p) { taosMemoryFree(((SMetaRes*)p)->pRes); } -static int32_t ctgCloneDnodeList(void* pSrc) { +static int32_t ctgCloneDnodeList(void* pSrc) { #if 0 - return taosArrayDup((const SArray*)pSrc, NULL); + return taosArrayDup((const SArray*)pSrc, NULL); #else return TSDB_CODE_CTG_INTERNAL_ERROR; #endif @@ -1935,14 +1940,14 @@ static int32_t ctgCloneViewMeta(void* pSrc) { #endif } -static void ctgFreeViewMeta(void* p) { - SViewMeta* pMeta = ((SMetaRes*)p)->pRes; +static void ctgFreeViewMeta(void* p) { + SViewMeta* pMeta = ((SMetaRes*)p)->pRes; if (NULL == pMeta) { return; } taosMemoryFree(pMeta->user); taosMemoryFree(pMeta->querySql); - taosMemoryFree(pMeta->pSchema); + taosMemoryFree(pMeta->pSchema); taosMemoryFree(pMeta); } @@ -2054,8 +2059,8 @@ int32_t ctgChkSetBasicAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res } if (req->tbNotExists) { - //pRes->pass[AUTH_RES_BASIC] = true; - //return TSDB_CODE_SUCCESS; + // pRes->pass[AUTH_RES_BASIC] = true; + // return TSDB_CODE_SUCCESS; pReq->tbName.type = TSDB_DB_NAME_T; } @@ -2152,7 +2157,7 @@ int32_t ctgChkSetViewAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res) switch (pReq->type) { case AUTH_TYPE_READ: { - char *value = taosHashGet(pInfo->readViews, viewFName, len); + char* value = taosHashGet(pInfo->readViews, viewFName, len); if (NULL != value) { pRes->pass[AUTH_RES_VIEW] = true; return TSDB_CODE_SUCCESS; @@ -2160,7 +2165,7 @@ int32_t ctgChkSetViewAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res) break; } case AUTH_TYPE_WRITE: { - char *value = taosHashGet(pInfo->writeViews, viewFName, len); + char* value = taosHashGet(pInfo->writeViews, viewFName, len); if (NULL != value) { pRes->pass[AUTH_RES_VIEW] = true; return TSDB_CODE_SUCCESS; @@ -2168,7 +2173,7 @@ int32_t ctgChkSetViewAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res) break; } case AUTH_TYPE_ALTER: { - char *value = taosHashGet(pInfo->alterViews, viewFName, len); + char* value = taosHashGet(pInfo->alterViews, viewFName, len); if (NULL != value) { pRes->pass[AUTH_RES_VIEW] = true; return TSDB_CODE_SUCCESS; @@ -2288,7 +2293,7 @@ void ctgDestroySMetaData(SMetaData* pData) { taosMemoryFreeClear(pData->pSvrVer); } -uint64_t ctgGetTbIndexCacheSize(STableIndex *pIndex) { +uint64_t ctgGetTbIndexCacheSize(STableIndex* pIndex) { if (NULL == pIndex) { return 0; } @@ -2296,7 +2301,7 @@ uint64_t ctgGetTbIndexCacheSize(STableIndex *pIndex) { return sizeof(*pIndex) + pIndex->indexSize; } -uint64_t ctgGetViewMetaCacheSize(SViewMeta *pMeta) { +uint64_t ctgGetViewMetaCacheSize(SViewMeta* pMeta) { if (NULL == pMeta) { return 0; } @@ -2304,8 +2309,7 @@ uint64_t ctgGetViewMetaCacheSize(SViewMeta *pMeta) { return sizeof(*pMeta) + strlen(pMeta->querySql) + 1 + strlen(pMeta->user) + 1 + pMeta->numOfCols * sizeof(SSchema); } - -FORCE_INLINE uint64_t ctgGetTbMetaCacheSize(STableMeta *pMeta) { +FORCE_INLINE uint64_t ctgGetTbMetaCacheSize(STableMeta* pMeta) { if (NULL == pMeta) { return 0; } @@ -2322,22 +2326,22 @@ FORCE_INLINE uint64_t ctgGetTbMetaCacheSize(STableMeta *pMeta) { return 0; } -uint64_t ctgGetDbVgroupCacheSize(SDBVgInfo *pVg) { +uint64_t ctgGetDbVgroupCacheSize(SDBVgInfo* pVg) { if (NULL == pVg) { return 0; } - return sizeof(*pVg) + taosHashGetSize(pVg->vgHash) * (sizeof(SVgroupInfo) + sizeof(int32_t)) - + taosArrayGetSize(pVg->vgArray) * sizeof(SVgroupInfo); + return sizeof(*pVg) + taosHashGetSize(pVg->vgHash) * (sizeof(SVgroupInfo) + sizeof(int32_t)) + + taosArrayGetSize(pVg->vgArray) * sizeof(SVgroupInfo); } -uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { +uint64_t ctgGetUserCacheSize(SGetUserAuthRsp* pAuth) { if (NULL == pAuth) { return 0; } uint64_t cacheSize = 0; - char* p = taosHashIterate(pAuth->createdDbs, NULL); + char* p = taosHashIterate(pAuth->createdDbs, NULL); while (p != NULL) { size_t len = 0; void* key = taosHashGetKey(p, &len); @@ -2353,7 +2357,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->readDbs, p); - } + } p = taosHashIterate(pAuth->writeDbs, NULL); while (p != NULL) { @@ -2362,7 +2366,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->writeDbs, p); - } + } p = taosHashIterate(pAuth->readTbs, NULL); while (p != NULL) { @@ -2371,7 +2375,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->readTbs, p); - } + } p = taosHashIterate(pAuth->writeTbs, NULL); while (p != NULL) { @@ -2380,7 +2384,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->writeTbs, p); - } + } p = taosHashIterate(pAuth->alterTbs, NULL); while (p != NULL) { @@ -2389,7 +2393,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->alterTbs, p); - } + } p = taosHashIterate(pAuth->readViews, NULL); while (p != NULL) { @@ -2398,7 +2402,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->readViews, p); - } + } p = taosHashIterate(pAuth->writeViews, NULL); while (p != NULL) { @@ -2407,7 +2411,7 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->writeViews, p); - } + } p = taosHashIterate(pAuth->alterViews, NULL); while (p != NULL) { @@ -2416,23 +2420,23 @@ uint64_t ctgGetUserCacheSize(SGetUserAuthRsp *pAuth) { cacheSize += len + strlen(p) + 1; p = taosHashIterate(pAuth->alterViews, p); - } + } - int32_t *ref = taosHashIterate(pAuth->useDbs, NULL); + int32_t* ref = taosHashIterate(pAuth->useDbs, NULL); while (ref != NULL) { size_t len = 0; void* key = taosHashGetKey(ref, &len); cacheSize += len + sizeof(*ref); ref = taosHashIterate(pAuth->useDbs, ref); - } + } return cacheSize; } -uint64_t ctgGetClusterCacheSize(SCatalog *pCtg) { +uint64_t ctgGetClusterCacheSize(SCatalog* pCtg) { uint64_t cacheSize = sizeof(SCatalog); - + SCtgUserAuth* pAuth = taosHashIterate(pCtg->userCache, NULL); while (pAuth != NULL) { size_t len = 0; @@ -2523,7 +2527,7 @@ void ctgGetGlobalCacheStat(SCtgCacheStat* pStat) { TAOS_MEMCPY(pStat, &gCtgMgmt.statInfo.cache, sizeof(gCtgMgmt.statInfo.cache)); } -void ctgGetGlobalCacheSize(uint64_t *pSize) { +void ctgGetGlobalCacheSize(uint64_t* pSize) { *pSize = 0; SCatalog* pCtg = NULL; @@ -2531,8 +2535,8 @@ void ctgGetGlobalCacheSize(uint64_t *pSize) { while (pIter) { size_t len = 0; void* key = taosHashGetKey(pIter, &len); - *pSize += len + POINTER_BYTES; - + *pSize += len + POINTER_BYTES; + pCtg = *(SCatalog**)pIter; if (pCtg) { *pSize += ctgGetClusterCacheSize(pCtg); @@ -2544,18 +2548,18 @@ void ctgGetGlobalCacheSize(uint64_t *pSize) { int32_t ctgBuildViewNullRes(SCtgTask* pTask, SCtgViewsCtx* pCtx) { SCatalog* pCtg = pTask->pJob->pCtg; - int32_t dbNum = taosArrayGetSize(pCtx->pNames); + int32_t dbNum = taosArrayGetSize(pCtx->pNames); for (int32_t i = 0; i < dbNum; ++i) { STablesReq* pReq = taosArrayGet(pCtx->pNames, i); if (NULL == pReq) { qError("fail to get the %dth STablesReq, total:%d", i, (int32_t)taosArrayGetSize(pCtx->pNames)); CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR); } - + int32_t viewNum = taosArrayGetSize(pReq->pTables); - + ctgDebug("start to check views in db %s, viewNum %d", pReq->dbFName, viewNum); - + for (int32_t m = 0; m < viewNum; ++m) { if (NULL == taosArrayPush(pCtx->pResList, &(SMetaData){0})) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); @@ -2584,13 +2588,12 @@ int32_t dupViewMetaFromRsp(SViewMetaRsp* pRsp, SViewMeta* pViewMeta) { if (pViewMeta->pSchema == NULL) { CTG_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); } - + TAOS_MEMCPY(pViewMeta->pSchema, pRsp->pSchema, pViewMeta->numOfCols * sizeof(SSchema)); return TSDB_CODE_SUCCESS; } - int32_t ctgBuildUseDbOutput(SUseDbOutput** ppOut, SDBVgInfo* vgInfo) { *ppOut = taosMemoryCalloc(1, sizeof(SUseDbOutput)); if (NULL == *ppOut) { @@ -2620,7 +2623,7 @@ uint64_t ctgGetTbTSMACacheSize(STableTSMAInfo* pTsmaInfo) { if (pTsmaInfo->pUsedCols) { size += sizeof(SSchema) * pTsmaInfo->pUsedCols->size; } - + return size; } @@ -2631,13 +2634,13 @@ bool hasOutOfDateTSMACache(SArray* pTsmas) { for (int32_t i = 0; i < pTsmas->size; ++i) { STSMACache* pTsmaInfo = taosArrayGetP(pTsmas, i); if (NULL == pTsmaInfo) { - ASSERT(0); + continue; } if (isCtgTSMACacheOutOfDate(pTsmaInfo)) { return true; } } - + return false; } diff --git a/source/libs/command/src/command.c b/source/libs/command/src/command.c index 11ddc89d4c..138782495f 100644 --- a/source/libs/command/src/command.c +++ b/source/libs/command/src/command.c @@ -24,13 +24,13 @@ #include "tgrant.h" #define COL_DATA_SET_VAL_AND_CHECK(pCol, rows, buf, isNull) \ - do { \ - int _code = colDataSetVal(pCol, rows, buf, isNull);\ - if (TSDB_CODE_SUCCESS != _code) { \ - terrno = _code; \ - return _code; \ - } \ - } while(0) + do { \ + int _code = colDataSetVal(pCol, rows, buf, isNull); \ + if (TSDB_CODE_SUCCESS != _code) { \ + terrno = _code; \ + return _code; \ + } \ + } while (0) extern SConfig* tsCfg; @@ -50,6 +50,10 @@ static int32_t buildRetrieveTableRsp(SSDataBlock* pBlock, int32_t numOfCols, SRe (*pRsp)->numOfCols = htonl(numOfCols); int32_t len = blockEncode(pBlock, (*pRsp)->data + PAYLOAD_PREFIX_LEN, numOfCols); + if(len < 0) { + taosMemoryFree(*pRsp); + return terrno; + } SET_PAYLOAD_LEN((*pRsp)->data, len, len); int32_t payloadLen = len + PAYLOAD_PREFIX_LEN; @@ -77,7 +81,7 @@ static int32_t buildDescResultDataBlock(SSDataBlock** pOutput) { QRY_OPTR_CHECK(pOutput); SSDataBlock* pBlock = NULL; - int32_t code = createDataBlock(&pBlock); + int32_t code = createDataBlock(&pBlock); if (code) { return code; } @@ -235,7 +239,7 @@ static int32_t buildCreateDBResultDataBlock(SSDataBlock** pOutput) { QRY_OPTR_CHECK(pOutput); SSDataBlock* pBlock = NULL; - int32_t code = createDataBlock(&pBlock); + int32_t code = createDataBlock(&pBlock); if (code) { return code; } @@ -280,15 +284,15 @@ int64_t getValOfDiffPrecision(int8_t unit, int64_t val) { return v; } -static int32_t buildRetension(SArray* pRetension, char **ppRetentions ) { +static int32_t buildRetension(SArray* pRetension, char** ppRetentions) { size_t size = taosArrayGetSize(pRetension); if (size == 0) { *ppRetentions = NULL; return TSDB_CODE_SUCCESS; } - char* p1 = taosMemoryCalloc(1, 100); - if(NULL == p1) { + char* p1 = taosMemoryCalloc(1, 100); + if (NULL == p1) { return terrno; } int32_t len = 0; @@ -340,6 +344,29 @@ static const char* encryptAlgorithmStr(int8_t encryptAlgorithm) { return TSDB_CACHE_MODEL_NONE_STR; } +static int32_t formatDurationOrKeep(char** buffer, int32_t timeInMinutes) { + int len = 0; + if (timeInMinutes % 1440 == 0) { + int days = timeInMinutes / 1440; + len = snprintf(NULL, 0, "%dd", days); + *buffer = (char*)taosMemoryCalloc(len + 1, sizeof(char)); + if(*buffer == NULL) return terrno; + sprintf(*buffer, "%dd", days); + } else if (timeInMinutes % 60 == 0) { + int hours = timeInMinutes / 60; + len = snprintf(NULL, 0, "%dh", hours); + *buffer = (char*)taosMemoryCalloc(len + 1, sizeof(char)); + if(*buffer == NULL) return terrno; + sprintf(*buffer, "%dh", hours); + } else { + len = snprintf(NULL, 0, "%dm", timeInMinutes); + *buffer = (char*)taosMemoryCalloc(len + 1, sizeof(char)); + if(*buffer == NULL) return terrno; + sprintf(*buffer, "%dm", timeInMinutes); + } + return TSDB_CODE_SUCCESS; +} + static int32_t setCreateDBResultIntoDataBlock(SSDataBlock* pBlock, char* dbName, char* dbFName, SDbCfgInfo* pCfg) { QRY_ERR_RET(blockDataEnsureCapacity(pBlock, 1)); pBlock->info.rows = 1; @@ -368,7 +395,7 @@ static int32_t setCreateDBResultIntoDataBlock(SSDataBlock* pBlock, char* dbName, break; } - char* pRetentions = NULL; + char* pRetentions = NULL; QRY_ERR_RET(buildRetension(pCfg->pRetensions, &pRetentions)); int32_t dbFNameLen = strlen(dbFName); int32_t hashPrefix = 0; @@ -377,20 +404,40 @@ static int32_t setCreateDBResultIntoDataBlock(SSDataBlock* pBlock, char* dbName, } else if (pCfg->hashPrefix < 0) { hashPrefix = pCfg->hashPrefix + dbFNameLen + 1; } - + char* durationStr = NULL; + char* keep0Str = NULL; + char* keep1Str = NULL; + char* keep2Str = NULL; + int32_t codeDuration = formatDurationOrKeep(&durationStr, pCfg->daysPerFile); + int32_t codeKeep0 = formatDurationOrKeep(&keep0Str, pCfg->daysToKeep0); + int32_t codeKeep1 = formatDurationOrKeep(&keep1Str, pCfg->daysToKeep1); + int32_t codeKeep2 = formatDurationOrKeep(&keep2Str, pCfg->daysToKeep2); + if(codeDuration != TSDB_CODE_SUCCESS || codeKeep0 != TSDB_CODE_SUCCESS || codeKeep1 != TSDB_CODE_SUCCESS || codeKeep2 != TSDB_CODE_SUCCESS) { + int32_t firstErrorCode = codeDuration != TSDB_CODE_SUCCESS ? codeDuration : + codeKeep0 != TSDB_CODE_SUCCESS ? codeKeep0 : + codeKeep1 != TSDB_CODE_SUCCESS ? codeKeep1 : codeKeep2; + taosMemoryFree(pRetentions); + taosMemoryFree(durationStr); + taosMemoryFree(keep0Str); + taosMemoryFree(keep1Str); + taosMemoryFree(keep2Str); + return firstErrorCode; + } if (IS_SYS_DBNAME(dbName)) { len += sprintf(buf2 + VARSTR_HEADER_SIZE, "CREATE DATABASE `%s`", dbName); } else { len += sprintf(buf2 + VARSTR_HEADER_SIZE, - "CREATE DATABASE `%s` BUFFER %d CACHESIZE %d CACHEMODEL '%s' COMP %d DURATION %dm " - "WAL_FSYNC_PERIOD %d MAXROWS %d MINROWS %d STT_TRIGGER %d KEEP %dm,%dm,%dm PAGES %d PAGESIZE %d " + "CREATE DATABASE `%s` BUFFER %d CACHESIZE %d CACHEMODEL '%s' COMP %d DURATION %s " + "WAL_FSYNC_PERIOD %d MAXROWS %d MINROWS %d STT_TRIGGER %d KEEP %s,%s,%s PAGES %d PAGESIZE %d " "PRECISION '%s' REPLICA %d " "WAL_LEVEL %d VGROUPS %d SINGLE_STABLE %d TABLE_PREFIX %d TABLE_SUFFIX %d TSDB_PAGESIZE %d " "WAL_RETENTION_PERIOD %d WAL_RETENTION_SIZE %" PRId64 " KEEP_TIME_OFFSET %d ENCRYPT_ALGORITHM '%s' S3_CHUNKSIZE %d S3_KEEPLOCAL %dm S3_COMPACT %d", dbName, pCfg->buffer, pCfg->cacheSize, cacheModelStr(pCfg->cacheLast), pCfg->compression, - pCfg->daysPerFile, pCfg->walFsyncPeriod, pCfg->maxRows, pCfg->minRows, pCfg->sstTrigger, - pCfg->daysToKeep0, pCfg->daysToKeep1, pCfg->daysToKeep2, pCfg->pages, pCfg->pageSize, prec, + durationStr, + pCfg->walFsyncPeriod, pCfg->maxRows, pCfg->minRows, pCfg->sstTrigger, + keep0Str, keep1Str, keep2Str, + pCfg->pages, pCfg->pageSize, prec, pCfg->replications, pCfg->walLevel, pCfg->numOfVgroups, 1 == pCfg->numOfStables, hashPrefix, pCfg->hashSuffix, pCfg->tsdbPageSize, pCfg->walRetentionPeriod, pCfg->walRetentionSize, pCfg->keepTimeOffset, encryptAlgorithmStr(pCfg->encryptAlgorithm), pCfg->s3ChunkSize, @@ -402,6 +449,10 @@ static int32_t setCreateDBResultIntoDataBlock(SSDataBlock* pBlock, char* dbName, } taosMemoryFree(pRetentions); + taosMemoryFree(durationStr); + taosMemoryFree(keep0Str); + taosMemoryFree(keep1Str); + taosMemoryFree(keep2Str); (varDataLen(buf2)) = len; @@ -427,7 +478,7 @@ static int32_t buildCreateTbResultDataBlock(SSDataBlock** pOutput) { QRY_OPTR_CHECK(pOutput); SSDataBlock* pBlock = NULL; - int32_t code = createDataBlock(&pBlock); + int32_t code = createDataBlock(&pBlock); if (code) { return code; } @@ -451,7 +502,7 @@ static int32_t buildCreateViewResultDataBlock(SSDataBlock** pOutput) { QRY_OPTR_CHECK(pOutput); SSDataBlock* pBlock = NULL; - int32_t code = createDataBlock(&pBlock); + int32_t code = createDataBlock(&pBlock); if (code) { return code; } @@ -578,36 +629,7 @@ int32_t appendTagValues(char* buf, int32_t* len, STableCfg* pCfg) { } else { *len += sprintf(buf + VARSTR_HEADER_SIZE + *len, "NULL"); } - - /* - if (type == TSDB_DATA_TYPE_BINARY || type == TSDB_DATA_TYPE_GEOMETRY) { - if (pTagVal->nData > 0) { - if (num) { - *len += sprintf(buf + VARSTR_HEADER_SIZE + *len, ", "); - } - - memcpy(buf + VARSTR_HEADER_SIZE + *len, pTagVal->pData, pTagVal->nData); - *len += pTagVal->nData; - } - } else if (type == TSDB_DATA_TYPE_NCHAR) { - if (pTagVal->nData > 0) { - if (num) { - *len += sprintf(buf + VARSTR_HEADER_SIZE + *len, ", "); - } - int32_t tlen = taosUcs4ToMbs((TdUcs4 *)pTagVal->pData, pTagVal->nData, buf + VARSTR_HEADER_SIZE + *len); - } - } else if (type == TSDB_DATA_TYPE_DOUBLE) { - double val = *(double *)(&pTagVal->i64); - int len = 0; - term = indexTermCreate(suid, ADD_VALUE, type, key, nKey, (const char *)&val, len); - } else if (type == TSDB_DATA_TYPE_BOOL) { - int val = *(int *)(&pTagVal->i64); - int len = 0; - term = indexTermCreate(suid, ADD_VALUE, TSDB_DATA_TYPE_INT, key, nKey, (const char *)&val, len); - } - */ } - _exit: taosArrayDestroy(pTagVals); @@ -740,11 +762,14 @@ static int32_t setCreateViewResultIntoDataBlock(SSDataBlock* pBlock, SShowCreate SColumnInfoData* pCol2 = taosArrayGet(pBlock->pDataBlock, 1); char* buf2 = taosMemoryMalloc(SHOW_CREATE_VIEW_RESULT_FIELD2_LEN); if (NULL == buf2) { - QRY_ERR_RET(TSDB_CODE_OUT_OF_MEMORY); + return terrno; } SViewMeta* pMeta = pStmt->pViewMeta; - ASSERT(pMeta); + if(NULL == pMeta) { + qError("exception: view meta is null"); + return TSDB_CODE_APP_ERROR; + } snprintf(varDataVal(buf2), SHOW_CREATE_VIEW_RESULT_FIELD2_LEN - VARSTR_HEADER_SIZE, "CREATE VIEW `%s`.`%s` AS %s", pStmt->dbName, pStmt->viewName, pMeta->querySql); int32_t len = strlen(varDataVal(buf2)); @@ -864,26 +889,26 @@ static int32_t buildLocalVariablesResultDataBlock(SSDataBlock** pOutput) { infoData.info.type = TSDB_DATA_TYPE_VARCHAR; infoData.info.bytes = SHOW_LOCAL_VARIABLES_RESULT_FIELD1_LEN; - if(taosArrayPush(pBlock->pDataBlock, &infoData) == NULL) { + if (taosArrayPush(pBlock->pDataBlock, &infoData) == NULL) { goto _exit; } infoData.info.type = TSDB_DATA_TYPE_VARCHAR; infoData.info.bytes = SHOW_LOCAL_VARIABLES_RESULT_FIELD2_LEN; - if(taosArrayPush(pBlock->pDataBlock, &infoData) == NULL) { + if (taosArrayPush(pBlock->pDataBlock, &infoData) == NULL) { goto _exit; } infoData.info.type = TSDB_DATA_TYPE_VARCHAR; infoData.info.bytes = SHOW_LOCAL_VARIABLES_RESULT_FIELD3_LEN; - if(taosArrayPush(pBlock->pDataBlock, &infoData) == NULL) { + if (taosArrayPush(pBlock->pDataBlock, &infoData) == NULL) { goto _exit; } *pOutput = pBlock; _exit: - if(terrno != TSDB_CODE_SUCCESS) { + if (terrno != TSDB_CODE_SUCCESS) { taosMemoryFree(pBlock); taosArrayDestroy(pBlock->pDataBlock); } @@ -907,7 +932,7 @@ static int32_t createSelectResultDataBlock(SNodeList* pProjects, SSDataBlock** p QRY_OPTR_CHECK(pOutput); SSDataBlock* pBlock = NULL; - int32_t code = createDataBlock(&pBlock); + int32_t code = createDataBlock(&pBlock); if (code) { return code; } @@ -942,7 +967,8 @@ int32_t buildSelectResultDataBlock(SNodeList* pProjects, SSDataBlock* pBlock) { if (((SValueNode*)pProj)->isNull) { QRY_ERR_RET(colDataSetVal(taosArrayGet(pBlock->pDataBlock, index++), 0, NULL, true)); } else { - QRY_ERR_RET(colDataSetVal(taosArrayGet(pBlock->pDataBlock, index++), 0, nodesGetValueFromNode((SValueNode*)pProj), false)); + QRY_ERR_RET(colDataSetVal(taosArrayGet(pBlock->pDataBlock, index++), 0, + nodesGetValueFromNode((SValueNode*)pProj), false)); } } } diff --git a/source/libs/command/src/explain.c b/source/libs/command/src/explain.c index 3a73c05de2..b9f79d1e00 100644 --- a/source/libs/command/src/explain.c +++ b/source/libs/command/src/explain.c @@ -1978,6 +1978,10 @@ int32_t qExplainGetRspFromCtx(void *ctx, SRetrieveTableRsp **pRsp) { rsp->numOfRows = htobe64((int64_t)rowNum); int32_t len = blockEncode(pBlock, rsp->data + PAYLOAD_PREFIX_LEN, taosArrayGetSize(pBlock->pDataBlock)); + if(len < 0) { + qError("qExplainGetRspFromCtx: blockEncode failed"); + QRY_ERR_JRET(terrno); + } rsp->compLen = htonl(len); rsp->payloadLen = htonl(len); diff --git a/source/libs/executor/inc/executil.h b/source/libs/executor/inc/executil.h index b68ea5a781..e35b26627b 100644 --- a/source/libs/executor/inc/executil.h +++ b/source/libs/executor/inc/executil.h @@ -126,7 +126,7 @@ uint64_t tableListGetTableGroupId(const STableListInfo* pTableList, uint6 int32_t tableListAddTableInfo(STableListInfo* pTableList, uint64_t uid, uint64_t gid); int32_t tableListGetGroupList(const STableListInfo* pTableList, int32_t ordinalIndex, STableKeyInfo** pKeyInfo, int32_t* num); -uint64_t tableListGetSize(const STableListInfo* pTableList); +int32_t tableListGetSize(const STableListInfo* pTableList, int32_t* pRes); uint64_t tableListGetSuid(const STableListInfo* pTableList); STableKeyInfo* tableListGetInfo(const STableListInfo* pTableList, int32_t index); int32_t tableListFind(const STableListInfo* pTableList, uint64_t uid, int32_t startIndex); diff --git a/source/libs/executor/inc/tsort.h b/source/libs/executor/inc/tsort.h index 474f3eedbf..6f6384af5b 100644 --- a/source/libs/executor/inc/tsort.h +++ b/source/libs/executor/inc/tsort.h @@ -66,7 +66,7 @@ typedef struct SMsortComparParam { typedef struct SSortHandle SSortHandle; typedef struct STupleHandle STupleHandle; -typedef SSDataBlock* (*_sort_fetch_block_fn_t)(void* param); +typedef int32_t (*_sort_fetch_block_fn_t)(void* param, SSDataBlock** ppBlock); typedef int32_t (*_sort_merge_compar_fn_t)(const void* p1, const void* p2, void* param); /** diff --git a/source/libs/executor/src/cachescanoperator.c b/source/libs/executor/src/cachescanoperator.c index 6b5fadbaa1..a219a5b5f0 100644 --- a/source/libs/executor/src/cachescanoperator.c +++ b/source/libs/executor/src/cachescanoperator.c @@ -176,7 +176,10 @@ int32_t createCacherowsScanOperator(SLastRowScanPhysiNode* pScanNode, SReadHandl code = extractCacheScanSlotId(pInfo->matchInfo.pList, pTaskInfo, &pInfo->pSlotIds, &pInfo->pDstSlotIds); QUERY_CHECK_CODE(code, lino, _error); - int32_t totalTables = tableListGetSize(pTableListInfo); + int32_t totalTables = 0; + code = tableListGetSize(pTableListInfo, &totalTables); + QUERY_CHECK_CODE(code, lino, _error); + int32_t capacity = 0; pInfo->pUidList = taosArrayInit(4, sizeof(int64_t)); @@ -271,7 +274,10 @@ int32_t doScanCacheNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) { SSDataBlock* pBufRes = pInfo->pBufferedRes; uint64_t suid = tableListGetSuid(pTableList); - int32_t size = tableListGetSize(pTableList); + int32_t size = 0; + code = tableListGetSize(pTableList, &size); + QUERY_CHECK_CODE(code, lino, _end); + if (size == 0) { setOperatorCompleted(pOperator); (*ppRes) = NULL; diff --git a/source/libs/executor/src/countwindowoperator.c b/source/libs/executor/src/countwindowoperator.c index f4eb0dd87e..ba07e666a0 100644 --- a/source/libs/executor/src/countwindowoperator.c +++ b/source/libs/executor/src/countwindowoperator.c @@ -32,6 +32,7 @@ typedef struct SCountWindowResult { typedef struct SCountWindowSupp { SArray* pWinStates; int32_t stateIndex; + int32_t curStateIndex; } SCountWindowSupp; typedef struct SCountWindowOperatorInfo { @@ -45,6 +46,8 @@ typedef struct SCountWindowOperatorInfo { int32_t windowCount; int32_t windowSliding; SCountWindowSupp countSup; + SSDataBlock* pPreDataBlock; + int32_t preStateIndex; } SCountWindowOperatorInfo; void destroyCountWindowOperatorInfo(void* param) { @@ -65,6 +68,7 @@ static void clearWinStateBuff(SCountWindowResult* pBuff) { pBuff->winRows = 0; } static SCountWindowResult* getCountWinStateInfo(SCountWindowSupp* pCountSup) { SCountWindowResult* pBuffInfo = taosArrayGet(pCountSup->pWinStates, pCountSup->stateIndex); + pCountSup->curStateIndex = pCountSup->stateIndex; if (!pBuffInfo) { terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno)); @@ -113,6 +117,19 @@ void doCountWindowAggImpl(SOperatorInfo* pOperator, SSDataBlock* pBlock) { SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, pInfo->tsSlotId); QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno); TSKEY* tsCols = (TSKEY*)pColInfoData->pData; + int32_t numOfBuff = taosArrayGetSize(pInfo->countSup.pWinStates); + if (numOfBuff == 0) { + code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + T_LONG_JMP(pTaskInfo->env, code); + } + pInfo->countSup.stateIndex = (pInfo->preStateIndex + 1) % numOfBuff; + + int32_t newSize = pRes->info.rows + pBlock->info.rows / pInfo->windowSliding + 1; + if (newSize > pRes->info.capacity) { + code = blockDataEnsureCapacity(pRes, newSize); + QUERY_CHECK_CODE(code, lino, _end); + } for (int32_t i = 0; i < pBlock->info.rows;) { SCountWindowResult* pBuffInfo = NULL; @@ -132,14 +149,6 @@ void doCountWindowAggImpl(SOperatorInfo* pOperator, SSDataBlock* pBlock) { updateTimeWindowInfo(&pInfo->twAggSup.timeWindowData, &pInfo->pRow->win, 0); applyAggFunctionOnPartialTuples(pTaskInfo, pExprSup->pCtx, &pInfo->twAggSup.timeWindowData, i, num, pBlock->info.rows, pExprSup->numOfExprs); - if (pBuffInfo->winRows == pInfo->windowCount) { - doUpdateNumOfRows(pExprSup->pCtx, pInfo->pRow, pExprSup->numOfExprs, pExprSup->rowEntryInfoOffset); - copyResultrowToDataBlock(pExprSup->pExprInfo, pExprSup->numOfExprs, pInfo->pRow, pExprSup->pCtx, pRes, - pExprSup->rowEntryInfoOffset, pTaskInfo); - pRes->info.rows += pInfo->pRow->numOfRows; - clearWinStateBuff(pBuffInfo); - clearResultRowInitFlag(pExprSup->pCtx, pExprSup->numOfExprs); - } if (pInfo->windowCount != pInfo->windowSliding) { if (prevRows <= pInfo->windowSliding) { if (pBuffInfo->winRows > pInfo->windowSliding) { @@ -151,9 +160,21 @@ void doCountWindowAggImpl(SOperatorInfo* pOperator, SSDataBlock* pBlock) { step = 0; } } + if (pBuffInfo->winRows == pInfo->windowCount) { + doUpdateNumOfRows(pExprSup->pCtx, pInfo->pRow, pExprSup->numOfExprs, pExprSup->rowEntryInfoOffset); + copyResultrowToDataBlock(pExprSup->pExprInfo, pExprSup->numOfExprs, pInfo->pRow, pExprSup->pCtx, pRes, + pExprSup->rowEntryInfoOffset, pTaskInfo); + pRes->info.rows += pInfo->pRow->numOfRows; + clearWinStateBuff(pBuffInfo); + pInfo->preStateIndex = pInfo->countSup.curStateIndex; + clearResultRowInitFlag(pExprSup->pCtx, pExprSup->numOfExprs); + } i += step; } + code = doFilter(pRes, pOperator->exprSupp.pFilterInfo, NULL); + QUERY_CHECK_CODE(code, lino, _end); + _end: if (code != TSDB_CODE_SUCCESS) { qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); @@ -163,11 +184,18 @@ _end: } static void buildCountResult(SExprSupp* pExprSup, SCountWindowSupp* pCountSup, SExecTaskInfo* pTaskInfo, - SFilterInfo* pFilterInfo, SSDataBlock* pBlock) { + SFilterInfo* pFilterInfo, int32_t preStateIndex, SSDataBlock* pBlock) { SResultRow* pResultRow = NULL; int32_t code = TSDB_CODE_SUCCESS; int32_t lino = 0; - for (int32_t i = 0; i < taosArrayGetSize(pCountSup->pWinStates); i++) { + int32_t numOfBuff = taosArrayGetSize(pCountSup->pWinStates); + int32_t newSize = pBlock->info.rows + numOfBuff; + if (newSize > pBlock->info.capacity) { + code = blockDataEnsureCapacity(pBlock, newSize); + QUERY_CHECK_CODE(code, lino, _end); + } + pCountSup->stateIndex = (preStateIndex + 1) % numOfBuff; + for (int32_t i = 0; i < numOfBuff; i++) { SCountWindowResult* pBuff = NULL; code = setCountWindowOutputBuff(pExprSup, pCountSup, &pResultRow, &pBuff); QUERY_CHECK_CODE(code, lino, _end); @@ -204,7 +232,14 @@ static int32_t countWindowAggregateNext(SOperatorInfo* pOperator, SSDataBlock** blockDataCleanup(pRes); while (1) { - SSDataBlock* pBlock = getNextBlockFromDownstream(pOperator, 0); + SSDataBlock* pBlock = NULL; + if (pInfo->pPreDataBlock == NULL) { + pBlock = getNextBlockFromDownstream(pOperator, 0); + } else { + pBlock = pInfo->pPreDataBlock; + pInfo->pPreDataBlock = NULL; + } + if (pBlock == NULL) { break; } @@ -226,18 +261,26 @@ static int32_t countWindowAggregateNext(SOperatorInfo* pOperator, SSDataBlock** if (pInfo->groupId == 0) { pInfo->groupId = pBlock->info.id.groupId; } else if (pInfo->groupId != pBlock->info.id.groupId) { - buildCountResult(pExprSup, &pInfo->countSup, pTaskInfo, pOperator->exprSupp.pFilterInfo, pRes); + pInfo->pPreDataBlock = pBlock; + pRes->info.id.groupId = pInfo->groupId; + buildCountResult(pExprSup, &pInfo->countSup, pTaskInfo, pOperator->exprSupp.pFilterInfo, pInfo->preStateIndex, pRes); pInfo->groupId = pBlock->info.id.groupId; + if (pRes->info.rows > 0) { + (*ppRes) = pRes; + return code; + } } doCountWindowAggImpl(pOperator, pBlock); if (pRes->info.rows >= pOperator->resultInfo.threshold) { + pRes->info.id.groupId = pInfo->groupId; (*ppRes) = pRes; return code; } } - buildCountResult(pExprSup, &pInfo->countSup, pTaskInfo, pOperator->exprSupp.pFilterInfo, pRes); + pRes->info.id.groupId = pInfo->groupId; + buildCountResult(pExprSup, &pInfo->countSup, pTaskInfo, pOperator->exprSupp.pFilterInfo, pInfo->preStateIndex, pRes); _end: if (code != TSDB_CODE_SUCCESS) { @@ -320,6 +363,8 @@ int32_t createCountwindowOperatorInfo(SOperatorInfo* downstream, SPhysiNode* phy } pInfo->countSup.stateIndex = 0; + pInfo->pPreDataBlock = NULL; + pInfo->preStateIndex = 0; code = filterInitFromNode((SNode*)pCountWindowNode->window.node.pConditions, &pOperator->exprSupp.pFilterInfo, 0); QUERY_CHECK_CODE(code, lino, _error); diff --git a/source/libs/executor/src/dataDeleter.c b/source/libs/executor/src/dataDeleter.c index 9f0ea0a87f..60bfb58ef5 100644 --- a/source/libs/executor/src/dataDeleter.c +++ b/source/libs/executor/src/dataDeleter.c @@ -87,7 +87,10 @@ static int32_t toDataCacheEntry(SDataDeleterHandle* pHandle, const SInputData* p if (pRes->affectedRows) { pRes->skey = *(int64_t*)pColSKey->pData; pRes->ekey = *(int64_t*)pColEKey->pData; - ASSERT(pRes->skey <= pRes->ekey); + if (pRes->skey > pRes->ekey) { + qError("data delter skey:%" PRId64 " is bigger than ekey:%" PRId64, pRes->skey, pRes->ekey); + QRY_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); + } } else { pRes->skey = pHandle->pDeleter->deleteTimeRange.skey; pRes->ekey = pHandle->pDeleter->deleteTimeRange.ekey; @@ -205,7 +208,10 @@ static void getDataLength(SDataSinkHandle* pHandle, int64_t* pLen, int64_t* pRaw static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) { SDataDeleterHandle* pDeleter = (SDataDeleterHandle*)pHandle; if (NULL == pDeleter->nextOutput.pData) { - ASSERT(pDeleter->queryEnd); + if (!pDeleter->queryEnd) { + qError("empty res while query not end in data deleter"); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } pOutput->useconds = pDeleter->useconds; pOutput->precision = pDeleter->pSchema->precision; pOutput->bufStatus = DS_BUF_EMPTY; diff --git a/source/libs/executor/src/dataDispatcher.c b/source/libs/executor/src/dataDispatcher.c index f616cb05da..fb3ed06224 100644 --- a/source/libs/executor/src/dataDispatcher.c +++ b/source/libs/executor/src/dataDispatcher.c @@ -106,6 +106,10 @@ static int32_t toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* } int32_t dataLen = blockEncode(pInput->pData, pHandle->pCompressBuf, numOfCols); + if(dataLen < 0) { + qError("failed to encode data block, code: %d", dataLen); + return terrno; + } int32_t len = tsCompressString(pHandle->pCompressBuf, dataLen, 1, pEntry->data, pBuf->allocSize, ONE_STAGE_COMP, NULL, 0); if (len < dataLen) { @@ -120,6 +124,10 @@ static int32_t toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* } } else { pEntry->dataLen = blockEncode(pInput->pData, pEntry->data, numOfCols); + if(pEntry->dataLen < 0) { + qError("failed to encode data block, code: %d", pEntry->dataLen); + return terrno; + } pEntry->rawLen = pEntry->dataLen; } } @@ -242,7 +250,11 @@ static void getDataLength(SDataSinkHandle* pHandle, int64_t* pLen, int64_t* pRow static int32_t getDataBlock(SDataSinkHandle* pHandle, SOutputData* pOutput) { SDataDispatchHandle* pDispatcher = (SDataDispatchHandle*)pHandle; if (NULL == pDispatcher->nextOutput.pData) { - ASSERT(pDispatcher->queryEnd); + if (!pDispatcher->queryEnd) { + qError("empty res while query not end in data dispatcher"); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } + pOutput->useconds = pDispatcher->useconds; pOutput->precision = pDispatcher->pSchema->precision; pOutput->bufStatus = DS_BUF_EMPTY; diff --git a/source/libs/executor/src/dataInserter.c b/source/libs/executor/src/dataInserter.c index 6f226ecb21..44342b0ac9 100644 --- a/source/libs/executor/src/dataInserter.c +++ b/source/libs/executor/src/dataInserter.c @@ -234,7 +234,11 @@ int32_t buildSubmitReqFromBlock(SDataInserterHandle* pInserter, SSubmitReq2** pp case TSDB_DATA_TYPE_NCHAR: case TSDB_DATA_TYPE_VARBINARY: case TSDB_DATA_TYPE_VARCHAR: { // TSDB_DATA_TYPE_BINARY - ASSERT(pColInfoData->info.type == pCol->type); + if (pColInfoData->info.type != pCol->type) { + qError("column:%d type:%d in block dismatch with schema col:%d type:%d", colIdx, pColInfoData->info.type, k, pCol->type); + terrno = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + goto _end; + } if (colDataIsNull_s(pColInfoData, j)) { SColVal cv = COL_VAL_NULL(pCol->colId, pCol->type); if (NULL == taosArrayPush(pVals, &cv)) { diff --git a/source/libs/executor/src/dynqueryctrloperator.c b/source/libs/executor/src/dynqueryctrloperator.c index 772222f9ed..6268fa0268 100644 --- a/source/libs/executor/src/dynqueryctrloperator.c +++ b/source/libs/executor/src/dynqueryctrloperator.c @@ -310,39 +310,34 @@ static int32_t buildBatchExchangeOperatorParam(SOperatorParam** ppRes, int32_t d } -static int32_t buildMergeJoinOperatorParam(SOperatorParam** ppRes, bool initParam, SOperatorParam* pChild0, SOperatorParam* pChild1) { +static int32_t buildMergeJoinOperatorParam(SOperatorParam** ppRes, bool initParam, SOperatorParam** ppChild0, SOperatorParam** ppChild1) { int32_t code = TSDB_CODE_SUCCESS; *ppRes = taosMemoryMalloc(sizeof(SOperatorParam)); if (NULL == *ppRes) { code = terrno; - freeOperatorParam(pChild0, OP_GET_PARAM); - freeOperatorParam(pChild1, OP_GET_PARAM); return code; } (*ppRes)->pChildren = taosArrayInit(2, POINTER_BYTES); if (NULL == (*ppRes)->pChildren) { code = terrno; - freeOperatorParam(pChild0, OP_GET_PARAM); - freeOperatorParam(pChild1, OP_GET_PARAM); freeOperatorParam(*ppRes, OP_GET_PARAM); *ppRes = NULL; return code; } - if (NULL == taosArrayPush((*ppRes)->pChildren, &pChild0)) { + if (NULL == taosArrayPush((*ppRes)->pChildren, ppChild0)) { code = terrno; - freeOperatorParam(pChild0, OP_GET_PARAM); - freeOperatorParam(pChild1, OP_GET_PARAM); freeOperatorParam(*ppRes, OP_GET_PARAM); *ppRes = NULL; return code; } - if (NULL == taosArrayPush((*ppRes)->pChildren, &pChild1)) { + *ppChild0 = NULL; + if (NULL == taosArrayPush((*ppRes)->pChildren, ppChild1)) { code = terrno; - freeOperatorParam(pChild1, OP_GET_PARAM); freeOperatorParam(*ppRes, OP_GET_PARAM); *ppRes = NULL; return code; } + *ppChild1 = NULL; SSortMergeJoinOperatorParam* pJoin = taosMemoryMalloc(sizeof(SSortMergeJoinOperatorParam)); if (NULL == pJoin) { @@ -493,7 +488,7 @@ static int32_t buildSeqStbJoinOperatorParam(SDynQueryCtrlOperatorInfo* pInfo, SS pSrcParam1 = NULL; } if (TSDB_CODE_SUCCESS == code) { - code = buildMergeJoinOperatorParam(ppParam, initParam, pGcParam0, pGcParam1); + code = buildMergeJoinOperatorParam(ppParam, initParam, &pGcParam0, &pGcParam1); } if (TSDB_CODE_SUCCESS != code) { if (pSrcParam0) { @@ -823,12 +818,14 @@ static void postProcessStbJoinTableHash(SOperatorInfo* pOperator) { pStbJoin->execInfo.leftCacheNum = tSimpleHashGetSize(pStbJoin->ctx.prev.leftCache); qDebug("more than 1 ref build table num %" PRId64, (int64_t)tSimpleHashGetSize(pStbJoin->ctx.prev.leftCache)); +/* // debug only iter = 0; uint32_t* num = NULL; while (NULL != (num = tSimpleHashIterate(pStbJoin->ctx.prev.leftCache, num, &iter))) { - ASSERT(*num > 1); + A S S E R T(*num > 1); } +*/ } static void buildStbJoinTableList(SOperatorInfo* pOperator) { diff --git a/source/libs/executor/src/eventwindowoperator.c b/source/libs/executor/src/eventwindowoperator.c index b80ea74006..7218291f8c 100644 --- a/source/libs/executor/src/eventwindowoperator.c +++ b/source/libs/executor/src/eventwindowoperator.c @@ -36,6 +36,7 @@ typedef struct SEventWindowOperatorInfo { SFilterInfo* pEndCondInfo; bool inWindow; SResultRow* pRow; + SSDataBlock* pPreDataBlock; } SEventWindowOperatorInfo; static SSDataBlock* eventWindowAggregate(SOperatorInfo* pOperator); @@ -126,6 +127,7 @@ int32_t createEventwindowOperatorInfo(SOperatorInfo* downstream, SPhysiNode* phy QUERY_CHECK_CODE(code, lino, _error); pInfo->tsSlotId = tsSlotId; + pInfo->pPreDataBlock = NULL; setOperatorInfo(pOperator, "EventWindowOperator", QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE, true, OP_NOT_OPENED, pInfo, pTaskInfo); @@ -199,7 +201,14 @@ static int32_t eventWindowAggregateNext(SOperatorInfo* pOperator, SSDataBlock** SOperatorInfo* downstream = pOperator->pDownstream[0]; while (1) { - SSDataBlock* pBlock = getNextBlockFromDownstream(pOperator, 0); + SSDataBlock* pBlock = NULL; + if (pInfo->pPreDataBlock == NULL) { + pBlock = getNextBlockFromDownstream(pOperator, 0); + } else { + pBlock = pInfo->pPreDataBlock; + pInfo->pPreDataBlock = NULL; + } + if (pBlock == NULL) { break; } @@ -224,7 +233,8 @@ static int32_t eventWindowAggregateNext(SOperatorInfo* pOperator, SSDataBlock** code = doFilter(pRes, pSup->pFilterInfo, NULL); QUERY_CHECK_CODE(code, lino, _end); - if (pRes->info.rows >= pOperator->resultInfo.threshold) { + if (pRes->info.rows >= pOperator->resultInfo.threshold || + (pRes->info.id.groupId != pInfo->groupId && pRes->info.rows > 0)) { (*ppRes) = pRes; return code; } @@ -303,7 +313,10 @@ int32_t eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* p // this is a new group, reset the info pInfo->inWindow = false; pInfo->groupId = gid; + pInfo->pPreDataBlock = pBlock; + goto _return; } + pRes->info.id.groupId = pInfo->groupId; SFilterColumnParam param1 = {.numOfCols = taosArrayGetSize(pBlock->pDataBlock), .pDataBlock = pBlock->pDataBlock}; diff --git a/source/libs/executor/src/exchangeoperator.c b/source/libs/executor/src/exchangeoperator.c index 60faba3e3a..f354c9a1cc 100644 --- a/source/libs/executor/src/exchangeoperator.c +++ b/source/libs/executor/src/exchangeoperator.c @@ -521,6 +521,7 @@ void doDestroyExchangeOperatorInfo(void* param) { int32_t loadRemoteDataCallback(void* param, SDataBuf* pMsg, int32_t code) { SFetchRspHandleWrapper* pWrapper = (SFetchRspHandleWrapper*)param; + taosMemoryFreeClear(pMsg->pEpSet); SExchangeInfo* pExchangeInfo = taosAcquireRef(exchangeObjRefPool, pWrapper->exchangeId); if (pExchangeInfo == NULL) { qWarn("failed to acquire exchange operator, since it may have been released, %p", pExchangeInfo); @@ -893,12 +894,12 @@ int32_t doExtractResultBlocks(SExchangeInfo* pExchangeInfo, SSourceDataInfo* pDa int32_t rawLen = *(int32_t*)pStart; pStart += sizeof(int32_t); - ASSERT(compLen <= rawLen && compLen != 0); + QUERY_CHECK_CONDITION((compLen <= rawLen && compLen != 0), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); pNextStart = pStart + compLen; if (pRetrieveRsp->compressed && (compLen < rawLen)) { int32_t t = tsDecompressString(pStart, compLen, 1, pDataInfo->decompBuf, rawLen, ONE_STAGE_COMP, NULL, 0); - ASSERT(t == rawLen); + QUERY_CHECK_CONDITION((t == rawLen), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); pStart = pDataInfo->decompBuf; } diff --git a/source/libs/executor/src/executil.c b/source/libs/executor/src/executil.c index 24eafe7d57..29ffd900f2 100644 --- a/source/libs/executor/src/executil.c +++ b/source/libs/executor/src/executil.c @@ -197,7 +197,6 @@ void initMultiResInfoFromArrayList(SGroupResInfo* pGroupResInfo, SArray* pArrayL pGroupResInfo->freeItem = true; pGroupResInfo->pRows = pArrayList; pGroupResInfo->index = 0; - ASSERT(pGroupResInfo->index <= getNumOfTotalRes(pGroupResInfo)); } bool hasRemainResults(SGroupResInfo* pGroupResInfo) { @@ -1560,7 +1559,12 @@ int32_t getGroupIdFromTagsVal(void* pVnode, uint64_t uid, SNodeList* pGroupNode, return code; } - ASSERT(nodeType(pNew) == QUERY_NODE_VALUE); + if (nodeType(pNew) != QUERY_NODE_VALUE) { + nodesDestroyList(groupNew); + pAPI->metaReaderFn.clearReader(&mr); + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR)); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } SValueNode* pValue = (SValueNode*)pNew; if (pValue->node.resType.type == TSDB_DATA_TYPE_NULL || pValue->isNull) { @@ -1879,7 +1883,8 @@ int32_t createExprFromOneNode(SExprInfo* pExp, SNode* pNode, int16_t slotId) { pExp->base.resSchema = createResSchema(pType->type, pType->bytes, slotId, pType->scale, pType->precision, pCond->node.aliasName); pExp->pExpr->_optrRoot.pRootNode = pNode; } else { - ASSERT(0); + code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + QUERY_CHECK_CODE(code, lino, _end); } _end: @@ -2149,7 +2154,8 @@ int32_t relocateColumnData(SSDataBlock* pBlock, const SArray* pColMatchInfo, SAr } else if (p->info.colId < pmInfo->colId) { i++; } else { - ASSERT(0); + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR)); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; } } return code; @@ -2383,9 +2389,13 @@ void resetLimitInfoForNextGroup(SLimitInfo* pLimitInfo) { pLimitInfo->remainOffset = pLimitInfo->limit.offset; } -uint64_t tableListGetSize(const STableListInfo* pTableList) { - ASSERT(taosArrayGetSize(pTableList->pTableList) == taosHashGetSize(pTableList->map)); - return taosArrayGetSize(pTableList->pTableList); +int32_t tableListGetSize(const STableListInfo* pTableList, int32_t* pRes) { + if (taosArrayGetSize(pTableList->pTableList) != taosHashGetSize(pTableList->map)) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR)); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } + (*pRes) = taosArrayGetSize(pTableList->pTableList); + return TSDB_CODE_SUCCESS; } uint64_t tableListGetSuid(const STableListInfo* pTableList) { return pTableList->idInfo.suid; } @@ -2430,7 +2440,6 @@ uint64_t tableListGetTableGroupId(const STableListInfo* pTableList, uint64_t tab } STableKeyInfo* pKeyInfo = taosArrayGet(pTableList->pTableList, *slot); - ASSERT(pKeyInfo->uid == tableUid); return pKeyInfo->groupId; } @@ -2457,7 +2466,8 @@ int32_t tableListAddTableInfo(STableListInfo* pTableList, uint64_t uid, uint64_t int32_t slot = (int32_t)taosArrayGetSize(pTableList->pTableList) - 1; code = taosHashPut(pTableList->map, &uid, sizeof(uid), &slot, sizeof(slot)); if (code != TSDB_CODE_SUCCESS) { - ASSERT(code != TSDB_CODE_DUP_KEY); // we have checked the existence of uid in hash map above + // we have checked the existence of uid in hash map above + QUERY_CHECK_CONDITION((code != TSDB_CODE_DUP_KEY), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); taosArrayPopTailBatch(pTableList->pTableList, 1); // let's pop the last element in the array list } @@ -2474,7 +2484,12 @@ _end: int32_t tableListGetGroupList(const STableListInfo* pTableList, int32_t ordinalGroupIndex, STableKeyInfo** pKeyInfo, int32_t* size) { int32_t totalGroups = tableListGetOutputGroups(pTableList); - int32_t numOfTables = tableListGetSize(pTableList); + int32_t numOfTables = 0; + int32_t code = tableListGetSize(pTableList, &numOfTables); + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + return code; + } if (ordinalGroupIndex < 0 || ordinalGroupIndex >= totalGroups) { return TSDB_CODE_INVALID_PARA; diff --git a/source/libs/executor/src/executor.c b/source/libs/executor/src/executor.c index b2cbef8919..f908ef5984 100644 --- a/source/libs/executor/src/executor.c +++ b/source/libs/executor/src/executor.c @@ -492,7 +492,7 @@ int32_t qUpdateTableListForStreamScanner(qTaskInfo_t tinfo, const SArray* tableI taosWLockLatch(&pTaskInfo->lock); for (int32_t i = 0; i < numOfQualifiedTables; ++i) { - uint64_t* uid = taosArrayGet(qa, i); + uint64_t* uid = taosArrayGet(qa, i); if (!uid) { taosMemoryFree(keyBuf); taosArrayDestroy(qa); @@ -600,7 +600,7 @@ int32_t qCreateExecTask(SReadHandle* readHandle, int32_t vgId, uint64_t taskId, SExecTaskInfo** pTask = (SExecTaskInfo**)pTaskInfo; (void)taosThreadOnce(&initPoolOnce, initRefPool); - qDebug("start to create task, TID:0x%" PRIx64 " QID:0x%" PRIx64 ", vgId:%d", taskId, pSubplan->id.queryId, vgId); + qDebug("start to create task, TID:0x%" PRIx64 " qid:0x%" PRIx64 ", vgId:%d", taskId, pSubplan->id.queryId, vgId); int32_t code = createExecTaskInfo(pSubplan, pTask, readHandle, taskId, vgId, sql, model); if (code != TSDB_CODE_SUCCESS || NULL == *pTask) { @@ -629,7 +629,7 @@ int32_t qCreateExecTask(SReadHandle* readHandle, int32_t vgId, uint64_t taskId, code = dsCreateDataSinker(pSinkManager, pSubplan->pDataSink, handle, pSinkParam, (*pTask)->id.str); } - qDebug("subplan task create completed, TID:0x%" PRIx64 " QID:0x%" PRIx64, taskId, pSubplan->id.queryId); + qDebug("subplan task create completed, TID:0x%" PRIx64 " qid:0x%" PRIx64, taskId, pSubplan->id.queryId); _error: // if failed to add ref for all tables in this query, abort current query @@ -887,7 +887,7 @@ void qStopTaskOperators(SExecTaskInfo* pTaskInfo) { qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno)); continue; } - SExchangeInfo* pExchangeInfo = taosAcquireRef(exchangeObjRefPool, pStop->refId); + SExchangeInfo* pExchangeInfo = taosAcquireRef(exchangeObjRefPool, pStop->refId); if (pExchangeInfo) { (void)tsem_post(&pExchangeInfo->ready); (void)taosReleaseRef(exchangeObjRefPool, pStop->refId); @@ -1295,7 +1295,13 @@ int32_t qStreamPrepareScan(qTaskInfo_t tinfo, STqOffsetVal* pOffset, int8_t subT // this value may be changed if new tables are created taosRLockLatch(&pTaskInfo->lock); - int32_t numOfTables = tableListGetSize(pTableListInfo); + int32_t numOfTables = 0; + code = tableListGetSize(pTableListInfo, &numOfTables); + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + taosRUnLockLatch(&pTaskInfo->lock); + return code; + } if (uid == 0) { if (numOfTables != 0) { @@ -1383,7 +1389,7 @@ int32_t qStreamPrepareScan(qTaskInfo_t tinfo, STqOffsetVal* pOffset, int8_t subT if (pOffset->type == TMQ_OFFSET__SNAPSHOT_DATA) { SStreamRawScanInfo* pInfo = pOperator->info; SSnapContext* sContext = pInfo->sContext; - SOperatorInfo* p = NULL; + SOperatorInfo* p = NULL; code = extractOperatorInTree(pOperator, QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN, id, &p); if (code != 0) { @@ -1399,7 +1405,7 @@ int32_t qStreamPrepareScan(qTaskInfo_t tinfo, STqOffsetVal* pOffset, int8_t subT SMetaTableInfo mtInfo = {0}; code = pTaskInfo->storageAPI.snapshotFn.getMetaTableInfoFromSnapshot(sContext, &mtInfo); - if (code != 0){ + if (code != 0) { return code; } pTaskInfo->storageAPI.tsdReader.tsdReaderClose(pInfo->dataReader); @@ -1439,7 +1445,13 @@ int32_t qStreamPrepareScan(qTaskInfo_t tinfo, STqOffsetVal* pOffset, int8_t subT tDeleteSchemaWrapper(mtInfo.schema); return code; } - int32_t size = tableListGetSize(pTableListInfo); + int32_t size = 0; + code = tableListGetSize(pTableListInfo, &size); + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + tDeleteSchemaWrapper(mtInfo.schema); + return code; + } code = pTaskInfo->storageAPI.tsdReader.tsdReaderOpen(pInfo->vnode, &pTaskInfo->streamInfo.tableCond, pList, size, NULL, (void**)&pInfo->dataReader, NULL, NULL); @@ -1520,11 +1532,14 @@ SArray* qGetQueriedTableListInfo(qTaskInfo_t tinfo) { SArray* pUidList = taosArrayInit(10, sizeof(uint64_t)); QUERY_CHECK_NULL(pUidList, code, lino, _end, terrno); - int32_t numOfTables = tableListGetSize(pTableListInfo); + int32_t numOfTables = 0; + code = tableListGetSize(pTableListInfo, &numOfTables); + QUERY_CHECK_CODE(code, lino, _end); + for (int32_t i = 0; i < numOfTables; ++i) { STableKeyInfo* pKeyInfo = tableListGetInfo(pTableListInfo, i); QUERY_CHECK_NULL(pKeyInfo, code, lino, _end, terrno); - void* tmp = taosArrayPush(pUidList, &pKeyInfo->uid); + void* tmp = taosArrayPush(pUidList, &pKeyInfo->uid); QUERY_CHECK_NULL(tmp, code, lino, _end, terrno); } diff --git a/source/libs/executor/src/executorInt.c b/source/libs/executor/src/executorInt.c index 031ffbb50e..966bda382b 100644 --- a/source/libs/executor/src/executorInt.c +++ b/source/libs/executor/src/executorInt.c @@ -1115,7 +1115,14 @@ int32_t createDataSinkParam(SDataSinkNode* pNode, void** pParam, SExecTaskInfo* pDeleterParam->suid = tableListGetSuid(pTableListInfo); // TODO extract uid list - int32_t numOfTables = tableListGetSize(pTableListInfo); + int32_t numOfTables = 0; + code = tableListGetSize(pTableListInfo, &numOfTables); + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + taosMemoryFree(pDeleterParam); + return code; + } + pDeleterParam->pUidList = taosArrayInit(numOfTables, sizeof(uint64_t)); if (NULL == pDeleterParam->pUidList) { taosMemoryFree(pDeleterParam); diff --git a/source/libs/executor/src/filloperator.c b/source/libs/executor/src/filloperator.c index fe9d0d3cf0..63f7667890 100644 --- a/source/libs/executor/src/filloperator.c +++ b/source/libs/executor/src/filloperator.c @@ -415,7 +415,8 @@ static int32_t createPrimaryTsExprIfNeeded(SFillOperatorInfo* pInfo, SFillPhysiN int32_t code = createExprFromTargetNode(&pExpr[pExprSupp->numOfExprs], (STargetNode*)pPhyFillNode->pWStartTs); if (code != TSDB_CODE_SUCCESS) { qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); - taosMemoryFreeClear(pExpr); + pExprSupp->numOfExprs += 1; + pExprSupp->pExprInfo = pExpr; return code; } @@ -447,6 +448,7 @@ int32_t createFillOperatorInfo(SOperatorInfo* downstream, SFillPhysiNode* pPhyFi QUERY_CHECK_CODE(code, lino, _error); pOperator->exprSupp.pExprInfo = pExprInfo; + pOperator->exprSupp.numOfExprs = pInfo->numOfExpr; SExprSupp* pNoFillSupp = &pInfo->noFillExprSupp; code = createExprInfo(pPhyFillNode->pNotFillExprs, NULL, &pNoFillSupp->pExprInfo, &pNoFillSupp->numOfExprs); @@ -511,7 +513,6 @@ int32_t createFillOperatorInfo(SOperatorInfo* downstream, SFillPhysiNode* pPhyFi } setOperatorInfo(pOperator, "FillOperator", QUERY_NODE_PHYSICAL_PLAN_FILL, false, OP_NOT_OPENED, pInfo, pTaskInfo); - pOperator->exprSupp.numOfExprs = pInfo->numOfExpr; pOperator->fpSet = createOperatorFpSet(optrDummyOpenFn, doFill, NULL, destroyFillOperatorInfo, optrDefaultBufFn, NULL, optrDefaultGetNextExtFn, NULL); diff --git a/source/libs/executor/src/groupoperator.c b/source/libs/executor/src/groupoperator.c index e5289fa216..3f203e7a95 100644 --- a/source/libs/executor/src/groupoperator.c +++ b/source/libs/executor/src/groupoperator.c @@ -213,7 +213,6 @@ static void recordNewGroupKeys(SArray* pGroupCols, SArray* pGroupColVals, SSData memcpy(pkey->pData, val, dataLen); } else if (IS_VAR_DATA_TYPE(pkey->type)) { memcpy(pkey->pData, val, varDataTLen(val)); - ASSERT(varDataTLen(val) <= pkey->bytes); } else { memcpy(pkey->pData, val, pkey->bytes); } @@ -241,7 +240,6 @@ static int32_t buildGroupKeys(void* pKey, const SArray* pGroupColVals) { } else if (IS_VAR_DATA_TYPE(pkey->type)) { varDataCopy(pStart, pkey->pData); pStart += varDataTLen(pkey->pData); - ASSERT(varDataTLen(pkey->pData) <= pkey->bytes); } else { memcpy(pStart, pkey->pData, pkey->bytes); pStart += pkey->bytes; @@ -548,7 +546,7 @@ int32_t createGroupOperatorInfo(SOperatorInfo* downstream, SAggPhysiNode* pAggNo SGroupbyOperatorInfo* pInfo = taosMemoryCalloc(1, sizeof(SGroupbyOperatorInfo)); SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo)); if (pInfo == NULL || pOperator == NULL) { - code = TSDB_CODE_OUT_OF_MEMORY; + code = terrno; goto _error; } @@ -740,7 +738,7 @@ static void doHashPartition(SOperatorInfo* pOperator, SSDataBlock* pBlock) { memcpy(data + (*columnLen), src, dataLen); int32_t v = (data + (*columnLen) + dataLen - (char*)pPage); - ASSERT(v > 0); + QUERY_CHECK_CONDITION((v > 0), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); contentLen = dataLen; } else { @@ -748,7 +746,7 @@ static void doHashPartition(SOperatorInfo* pOperator, SSDataBlock* pBlock) { char* src = colDataGetData(pColInfoData, j); memcpy(data + (*columnLen), src, varDataTLen(src)); int32_t v = (data + (*columnLen) + varDataTLen(src) - (char*)pPage); - ASSERT(v > 0); + QUERY_CHECK_CONDITION((v > 0), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); contentLen = varDataTLen(src); } @@ -762,7 +760,8 @@ static void doHashPartition(SOperatorInfo* pOperator, SSDataBlock* pBlock) { colDataSetNull_f(bitmap, (*rows)); } else { memcpy(data + (*columnLen), colDataGetData(pColInfoData, j), bytes); - ASSERT((data + (*columnLen) + bytes - (char*)pPage) <= getBufPageSize(pInfo->pBuf)); + QUERY_CHECK_CONDITION(((data + (*columnLen) + bytes - (char*)pPage) <= getBufPageSize(pInfo->pBuf)), code, + lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); } contentLen = bytes; } @@ -1299,7 +1298,7 @@ static SSDataBlock* buildStreamPartitionResult(SOperatorInfo* pOperator) { SStreamPartitionOperatorInfo* pInfo = pOperator->info; SSDataBlock* pDest = pInfo->binfo.pRes; - ASSERT(hasRemainPartion(pInfo)); + QUERY_CHECK_CONDITION((hasRemainPartion(pInfo)), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); SPartitionDataInfo* pParInfo = (SPartitionDataInfo*)pInfo->parIte; blockDataCleanup(pDest); int32_t rows = taosArrayGetSize(pParInfo->rowIds); @@ -1343,7 +1342,7 @@ static SSDataBlock* buildStreamPartitionResult(SOperatorInfo* pOperator) { pDest->info.id.groupId = pParInfo->groupId; pOperator->resultInfo.totalRows += pDest->info.rows; pInfo->parIte = taosHashIterate(pInfo->pPartitions, pInfo->parIte); - ASSERT(pDest->info.rows > 0); + QUERY_CHECK_CONDITION((pDest->info.rows > 0), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); _end: if (code != TSDB_CODE_SUCCESS) { @@ -1549,7 +1548,8 @@ static int32_t doStreamHashPartitionNext(SOperatorInfo* pOperator, SSDataBlock** return code; } default: - ASSERTS(0, "invalid SSDataBlock type"); + code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + QUERY_CHECK_CODE(code, lino, _end); } // there is an scalar expression that needs to be calculated right before apply the group aggregation. diff --git a/source/libs/executor/src/hashjoin.c b/source/libs/executor/src/hashjoin.c index d4a84afea2..f63b4093db 100755 --- a/source/libs/executor/src/hashjoin.c +++ b/source/libs/executor/src/hashjoin.c @@ -60,7 +60,7 @@ int32_t hInnerJoinDo(struct SOperatorInfo* pOperator) { /* size_t keySize = 0; int32_t* pKey = tSimpleHashGetKey(pGroup, &keySize); - ASSERT(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen)); + A S S E R T(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen)); int64_t rows = getSingleKeyRowsNum(pGroup->rows); pJoin->execInfo.expectRows += rows; qTrace("hash_key:%d, rows:%" PRId64, *pKey, rows); @@ -145,7 +145,7 @@ int32_t hLeftJoinHandleSeqProbeRows(struct SOperatorInfo* pOperator, SHJoinOpera /* size_t keySize = 0; int32_t* pKey = tSimpleHashGetKey(pGroup, &keySize); - ASSERT(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen)); + A S S E R T(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen)); int64_t rows = getSingleKeyRowsNum(pGroup->rows); pJoin->execInfo.expectRows += rows; qTrace("hash_key:%d, rows:%" PRId64, *pKey, rows); @@ -248,7 +248,7 @@ int32_t hLeftJoinHandleProbeRows(struct SOperatorInfo* pOperator, SHJoinOperator /* size_t keySize = 0; int32_t* pKey = tSimpleHashGetKey(pGroup, &keySize); - ASSERT(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen)); + A S S E R T(keySize == bufLen && 0 == memcmp(pKey, pProbe->keyData, bufLen)); int64_t rows = getSingleKeyRowsNum(pGroup->rows); pJoin->execInfo.expectRows += rows; qTrace("hash_key:%d, rows:%" PRId64, *pKey, rows); diff --git a/source/libs/executor/src/hashjoinoperator.c b/source/libs/executor/src/hashjoinoperator.c index 807d6b9785..15819cd94a 100644 --- a/source/libs/executor/src/hashjoinoperator.c +++ b/source/libs/executor/src/hashjoinoperator.c @@ -38,8 +38,6 @@ bool hJoinBlkReachThreshold(SHJoinOperatorInfo* pInfo, int64_t blkRows) { } int32_t hJoinHandleMidRemains(SHJoinOperatorInfo* pJoin, SHJoinCtx* pCtx) { - ASSERT(0 < pJoin->midBlk->info.rows); - TSWAP(pJoin->midBlk, pJoin->finBlk); pCtx->midRemains = false; @@ -1153,7 +1151,6 @@ int32_t hJoinInitResBlocks(SHJoinOperatorInfo* pJoin, SHashJoinPhysiNode* pJoinN if (NULL == pJoin->finBlk) { QRY_ERR_RET(terrno); } - ASSERT(pJoinNode->node.pOutputDataBlockDesc->totalRowSize > 0); int32_t code = blockDataEnsureCapacity(pJoin->finBlk, hJoinGetFinBlkCapacity(pJoin, pJoinNode)); if (TSDB_CODE_SUCCESS != code) { diff --git a/source/libs/executor/src/mergejoin.c b/source/libs/executor/src/mergejoin.c index d3abaaab6d..302dd31788 100755 --- a/source/libs/executor/src/mergejoin.c +++ b/source/libs/executor/src/mergejoin.c @@ -321,7 +321,7 @@ static int32_t mOuterJoinMergeSeqCart(SMJoinMergeCtx* pCtx) { int32_t buildEndIdx = buildGrp->endIdx; buildGrp->endIdx = buildGrp->readIdx + rowsLeft - 1; - ASSERT(buildGrp->endIdx >= buildGrp->readIdx); + //A S S E R T(buildGrp->endIdx >= buildGrp->readIdx); MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->midBlk, true, probeGrp, buildGrp)); buildGrp->readIdx += rowsLeft; buildGrp->endIdx = buildEndIdx; @@ -1383,9 +1383,9 @@ static int32_t mSemiJoinHashGrpCartFilter(SMJoinMergeCtx* pCtx, SMJoinGrpRows* p continue; } - ASSERT(1 == pCtx->midBlk->info.rows); + //A S S E R T(1 == pCtx->midBlk->info.rows); MJ_ERR_RET(mJoinCopyMergeMidBlk(pCtx, &pCtx->midBlk, &pCtx->finBlk)); - ASSERT(false == pCtx->midRemains); + //A S S E R T(false == pCtx->midRemains); break; } while (true); @@ -1449,10 +1449,10 @@ static int32_t mSemiJoinHashFullCart(SMJoinMergeCtx* pCtx) { } build->pHashCurGrp = *(SArray**)pGrp; - ASSERT(1 == taosArrayGetSize(build->pHashCurGrp)); + //A S S E R T(1 == taosArrayGetSize(build->pHashCurGrp)); build->grpRowIdx = 0; MJ_ERR_RET(mJoinHashGrpCart(pCtx->finBlk, probeGrp, true, probe, build, NULL)); - ASSERT(build->grpRowIdx < 0); + //A S S E R T(build->grpRowIdx < 0); } pCtx->grpRemains = probeGrp->readIdx <= probeGrp->endIdx; @@ -1497,7 +1497,7 @@ static int32_t mSemiJoinMergeSeqCart(SMJoinMergeCtx* pCtx) { int32_t buildEndIdx = buildGrp->endIdx; buildGrp->endIdx = buildGrp->readIdx + rowsLeft - 1; - ASSERT(buildGrp->endIdx >= buildGrp->readIdx); + //A S S E R T(buildGrp->endIdx >= buildGrp->readIdx); MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->midBlk, true, probeGrp, buildGrp)); buildGrp->readIdx += rowsLeft; buildGrp->endIdx = buildEndIdx; @@ -1513,9 +1513,9 @@ static int32_t mSemiJoinMergeSeqCart(SMJoinMergeCtx* pCtx) { continue; } } else { - ASSERT(1 == pCtx->midBlk->info.rows); + //A S S E R T(1 == pCtx->midBlk->info.rows); MJ_ERR_RET(mJoinCopyMergeMidBlk(pCtx, &pCtx->midBlk, &pCtx->finBlk)); - ASSERT(false == pCtx->midRemains); + //A S S E R T(false == pCtx->midRemains); if (build->grpIdx == buildGrpNum) { continue; @@ -1555,8 +1555,8 @@ static int32_t mSemiJoinMergeFullCart(SMJoinMergeCtx* pCtx) { int32_t probeRows = GRP_REMAIN_ROWS(probeGrp); int32_t probeEndIdx = probeGrp->endIdx; - ASSERT(1 == taosArrayGetSize(build->eqGrps)); - ASSERT(buildGrp->beginIdx == buildGrp->endIdx); + //A S S E R T(1 == taosArrayGetSize(build->eqGrps)); + //A S S E R T(buildGrp->beginIdx == buildGrp->endIdx); if (probeRows <= rowsLeft) { MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->finBlk, true, probeGrp, buildGrp)); @@ -1826,7 +1826,7 @@ static int32_t mAntiJoinMergeSeqCart(SMJoinMergeCtx* pCtx) { int32_t buildEndIdx = buildGrp->endIdx; buildGrp->endIdx = buildGrp->readIdx + rowsLeft - 1; - ASSERT(buildGrp->endIdx >= buildGrp->readIdx); + //A S S E R T(buildGrp->endIdx >= buildGrp->readIdx); MJ_ERR_RET(mJoinMergeGrpCart(pCtx->pJoin, pCtx->midBlk, true, probeGrp, buildGrp)); buildGrp->readIdx += rowsLeft; buildGrp->endIdx = buildEndIdx; @@ -2381,7 +2381,7 @@ int32_t mAsofForwardTrimCacheBlk(SMJoinWindowCtx* pCtx) { if (pGrp->blk == pCtx->cache.outBlk && pCtx->pJoin->build->blkRowIdx > 0) { MJ_ERR_RET(blockDataTrimFirstRows(pGrp->blk, pCtx->pJoin->build->blkRowIdx)); pCtx->pJoin->build->blkRowIdx = 0; - ASSERT(pCtx->pJoin->build->blk == pGrp->blk); + //A S S E R T(pCtx->pJoin->build->blk == pGrp->blk); MJOIN_SAVE_TB_BLK(&pCtx->cache, pCtx->pJoin->build); } @@ -2419,16 +2419,16 @@ int32_t mAsofForwardChkFillGrpCache(SMJoinWindowCtx* pCtx) { MJ_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); } - ASSERT(pGrp->blk == pCache->outBlk); + //A S S E R T(pGrp->blk == pCache->outBlk); //pGrp->endIdx = pGrp->blk->info.rows - pGrp->beginIdx; } - //ASSERT((pGrp->endIdx - pGrp->beginIdx + 1) == pCtx->cache.rowNum); + //A S S E R T((pGrp->endIdx - pGrp->beginIdx + 1) == pCtx->cache.rowNum); } - ASSERT(taosArrayGetSize(pCache->grps) == 1); - ASSERT(pGrp->blk->info.rows - pGrp->beginIdx == pCtx->cache.rowNum); + //A S S E R T(taosArrayGetSize(pCache->grps) == 1); + //A S S E R T(pGrp->blk->info.rows - pGrp->beginIdx == pCtx->cache.rowNum); } do { @@ -2473,7 +2473,7 @@ int32_t mAsofForwardUpdateBuildGrpEndIdx(SMJoinWindowCtx* pCtx) { return TSDB_CODE_SUCCESS; } - ASSERT(pCtx->jLimit > (pGrp->blk->info.rows - pGrp->beginIdx)); + //A S S E R T(pCtx->jLimit > (pGrp->blk->info.rows - pGrp->beginIdx)); pGrp->endIdx = pGrp->blk->info.rows - 1; int64_t remainRows = pCtx->jLimit - (pGrp->endIdx - pGrp->beginIdx + 1); @@ -2550,8 +2550,8 @@ int32_t mAsofForwardSkipAllEqRows(SMJoinWindowCtx* pCtx, int64_t timestamp) { MJOIN_RESTORE_TB_BLK(cache, pTable); } while (!MJOIN_BUILD_TB_ROWS_DONE(pTable)); - ASSERT(pCtx->cache.rowNum == 0); - ASSERT(taosArrayGetSize(pCtx->cache.grps) == 0); + //A S S E R T(pCtx->cache.rowNum == 0); + //A S S E R T(taosArrayGetSize(pCtx->cache.grps) == 0); if (pTable->dsFetchDone) { return TSDB_CODE_SUCCESS; @@ -2648,7 +2648,7 @@ static int32_t mAsofForwardRetrieve(SOperatorInfo* pOperator, SMJoinOperatorInfo if ((probeGot || MJOIN_DS_NEED_INIT(pOperator, pJoin->build)) && pCtx->cache.rowNum < pCtx->jLimit) { pJoin->build->newBlk = false; MJOIN_SAVE_TB_BLK(&pCtx->cache, pCtx->pJoin->build); - ASSERT(taosArrayGetSize(pCtx->cache.grps) <= 1); + //A S S E R T(taosArrayGetSize(pCtx->cache.grps) <= 1); buildGot = mJoinRetrieveBlk(pJoin, &pJoin->build->blkRowIdx, &pJoin->build->blk, pJoin->build); } @@ -3375,7 +3375,7 @@ int32_t mWinJoinMoveAscWinEnd(SMJoinWindowCtx* pCtx) { continue; } - ASSERT(pGrp->endIdx > startIdx); + //A S S E R T(pGrp->endIdx > startIdx); pGrp->endIdx--; break; @@ -3419,7 +3419,7 @@ int32_t mWinJoinMoveDescWinEnd(SMJoinWindowCtx* pCtx) { continue; } - ASSERT(pGrp->endIdx > startIdx); + //A S S E R T(pGrp->endIdx > startIdx); pGrp->endIdx--; break; @@ -3676,7 +3676,7 @@ int32_t mJoinInitMergeCtx(SMJoinOperatorInfo* pJoin, SSortMergeJoinPhysiNode* pJ MJ_ERR_RET(terrno); } - ASSERT(pJoinNode->node.pOutputDataBlockDesc->totalRowSize > 0); + //A S S E R T(pJoinNode->node.pOutputDataBlockDesc->totalRowSize > 0); MJ_ERR_RET(blockDataEnsureCapacity(pCtx->finBlk, mJoinGetFinBlkCapacity(pJoin, pJoinNode))); diff --git a/source/libs/executor/src/mergejoinoperator.c b/source/libs/executor/src/mergejoinoperator.c index 52b0da7c92..fced682312 100644 --- a/source/libs/executor/src/mergejoinoperator.c +++ b/source/libs/executor/src/mergejoinoperator.c @@ -490,8 +490,6 @@ int32_t mJoinCopyMergeMidBlk(SMJoinMergeCtx* pCtx, SSDataBlock** ppMid, SSDataBl int32_t mJoinHandleMidRemains(SMJoinMergeCtx* pCtx) { - ASSERT(0 < pCtx->midBlk->info.rows); - TSWAP(pCtx->midBlk, pCtx->finBlk); pCtx->midRemains = false; @@ -567,7 +565,6 @@ int32_t mJoinMergeGrpCart(SMJoinOperatorInfo* pJoin, SSDataBlock* pRes, bool app int32_t currRows = append ? pRes->info.rows : 0; int32_t firstRows = GRP_REMAIN_ROWS(pFirst); int32_t secondRows = GRP_REMAIN_ROWS(pSecond); - ASSERT(secondRows > 0); for (int32_t c = 0; c < probe->finNum; ++c) { SMJoinColMap* pFirstCol = probe->finCols + c; @@ -581,9 +578,15 @@ int32_t mJoinMergeGrpCart(SMJoinOperatorInfo* pJoin, SSDataBlock* pRes, bool app if (colDataIsNull_s(pInCol, pFirst->readIdx + r)) { colDataSetNItemsNull(pOutCol, currRows + r * secondRows, secondRows); } else { - ASSERT(pRes->info.capacity >= (pRes->info.rows + firstRows * secondRows)); + if (pRes->info.capacity < (pRes->info.rows + firstRows * secondRows)) { + qError("capacity:%d not enough, rows:%" PRId64 ", firstRows:%d, secondRows:%d", pRes->info.capacity, pRes->info.rows, firstRows, secondRows); + MJ_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); + } uint32_t startOffset = (IS_VAR_DATA_TYPE(pOutCol->info.type)) ? pOutCol->varmeta.length : ((currRows + r * secondRows) * pOutCol->info.bytes); - ASSERT((startOffset + 1 * pOutCol->info.bytes) <= pRes->info.capacity * pOutCol->info.bytes); + if ((startOffset + 1 * pOutCol->info.bytes) > pRes->info.capacity * pOutCol->info.bytes) { + qError("col buff not enough, startOffset:%d, bytes:%d, capacity:%d", startOffset, pOutCol->info.bytes, pRes->info.capacity); + MJ_ERR_RET(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); + } MJ_ERR_RET(colDataSetNItems(pOutCol, currRows + r * secondRows, colDataGetData(pInCol, pFirst->readIdx + r), secondRows, true)); } } @@ -1097,7 +1100,6 @@ SSDataBlock* mJoinGrpRetrieveImpl(SMJoinOperatorInfo* pJoin, SMJoinTableCtx* pTa } SMJoinTableCtx* pProbe = pJoin->probe; - ASSERT(pProbe->lastInGid); while (true) { if (pTable->remainInBlk) { diff --git a/source/libs/executor/src/mergeoperator.c b/source/libs/executor/src/mergeoperator.c index 8bc7c2db50..3f85324f57 100644 --- a/source/libs/executor/src/mergeoperator.c +++ b/source/libs/executor/src/mergeoperator.c @@ -65,10 +65,10 @@ static SSDataBlock* doNonSortMerge1(SOperatorInfo* pOperator); static SSDataBlock* doColsMerge1(SOperatorInfo* pOperator); static int32_t doColsMerge(SOperatorInfo* pOperator, SSDataBlock** pResBlock); -SSDataBlock* sortMergeloadNextDataBlock(void* param) { +int32_t sortMergeloadNextDataBlock(void* param, SSDataBlock** ppBlock) { SOperatorInfo* pOperator = (SOperatorInfo*)param; - SSDataBlock* pBlock = pOperator->fpSet.getNextFn(pOperator); - return pBlock; + *ppBlock = pOperator->fpSet.getNextFn(pOperator); + return TSDB_CODE_SUCCESS; } int32_t openSortMergeOperator(SOperatorInfo* pOperator) { diff --git a/source/libs/executor/src/operator.c b/source/libs/executor/src/operator.c index af52c31364..0ff6870405 100644 --- a/source/libs/executor/src/operator.c +++ b/source/libs/executor/src/operator.c @@ -240,7 +240,12 @@ int32_t getTableScanInfo(SOperatorInfo* pOperator, int32_t* order, int32_t* scan *order = info.order; *scanFlag = info.scanFlag; - ASSERT(*order == TSDB_ORDER_ASC || *order == TSDB_ORDER_DESC); + if (p.code == TSDB_CODE_SUCCESS) { + if (!(*order == TSDB_ORDER_ASC || *order == TSDB_ORDER_DESC)) { + qError("operator failed at: %s:%d", __func__, __LINE__); + p.code = TSDB_CODE_INVALID_PARA; + } + } return p.code; } diff --git a/source/libs/executor/src/projectoperator.c b/source/libs/executor/src/projectoperator.c index 4d2bdc62f8..bf523af918 100644 --- a/source/libs/executor/src/projectoperator.c +++ b/source/libs/executor/src/projectoperator.c @@ -218,7 +218,10 @@ static int32_t discardGroupDataBlock(SSDataBlock* pBlock, SLimitInfo* pLimitInfo static int32_t setInfoForNewGroup(SSDataBlock* pBlock, SLimitInfo* pLimitInfo, SOperatorInfo* pOperator) { // remainGroupOffset == 0 // here check for a new group data, we need to handle the data of the previous group. - ASSERT(pLimitInfo->remainGroupOffset == 0 || pLimitInfo->remainGroupOffset == -1); + if (!(pLimitInfo->remainGroupOffset == 0 || pLimitInfo->remainGroupOffset == -1)) { + qError("project failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INVALID_PARA; + } bool newGroup = false; if (0 == pBlock->info.id.groupId) { @@ -681,6 +684,7 @@ int32_t doApplyIndefinitFunction(SOperatorInfo* pOperator, SSDataBlock** pResBlo } int32_t initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size) { + int32_t code = TSDB_CODE_SUCCESS; for (int32_t j = 0; j < size; ++j) { struct SResultRowEntryInfo* pResInfo = GET_RES_INFO(&pCtx[j]); if (isRowEntryInitialized(pResInfo) || fmIsPseudoColumnFunc(pCtx[j].functionId) || pCtx[j].functionId == -1 || @@ -688,7 +692,10 @@ int32_t initCtxOutputBuffer(SqlFunctionCtx* pCtx, int32_t size) { continue; } - (void)pCtx[j].fpSet.init(&pCtx[j], pCtx[j].resultInfo); + code = pCtx[j].fpSet.init(&pCtx[j], pCtx[j].resultInfo); + if (code) { + return code; + } } return 0; @@ -814,7 +821,10 @@ int32_t doGenerateSourceData(SOperatorInfo* pOperator) { } int32_t startOffset = pRes->info.rows; - ASSERT(pRes->info.capacity > 0); + if (pRes->info.capacity <= 0) { + qError("project failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } code = colDataAssign(pResColData, &idata, dest.numOfRows, &pRes->info); if (code) { return code; @@ -871,7 +881,11 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc for (int32_t k = 0; k < numOfOutput; ++k) { int32_t outputSlotId = pExpr[k].base.resSchema.slotId; - ASSERT(pExpr[k].pExpr->nodeType == QUERY_NODE_VALUE); + if (pExpr[k].pExpr->nodeType != QUERY_NODE_VALUE) { + qError("project failed at: %s:%d", __func__, __LINE__); + code = TSDB_CODE_INVALID_PARA; + TSDB_CHECK_CODE(code, lino, _exit); + } SColumnInfoData* pColInfoData = taosArrayGet(pResult->pDataBlock, outputSlotId); if (pColInfoData == NULL) { code = terrno; @@ -1015,7 +1029,11 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc } int32_t startOffset = createNewColModel ? 0 : pResult->info.rows; - ASSERT(pResult->info.capacity > 0); + if (pResult->info.capacity <= 0) { + qError("project failed at: %s:%d", __func__, __LINE__); + code = TSDB_CODE_INVALID_PARA; + TSDB_CHECK_CODE(code, lino, _exit); + } int32_t ret = colDataMergeCol(pResColData, startOffset, (int32_t*)&pResult->info.capacity, &idata, dest.numOfRows); if (ret < 0) { @@ -1033,8 +1051,8 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc // do nothing } else if (fmIsIndefiniteRowsFunc(pfCtx->functionId)) { SResultRowEntryInfo* pResInfo = GET_RES_INFO(pfCtx); - (void) pfCtx->fpSet.init(pfCtx, pResInfo); - + code = pfCtx->fpSet.init(pfCtx, pResInfo); + TSDB_CHECK_CODE(code, lino, _exit); pfCtx->pOutput = taosArrayGet(pResult->pDataBlock, outputSlotId); if (pfCtx->pOutput == NULL) { code = terrno; @@ -1142,7 +1160,11 @@ int32_t projectApplyFunctions(SExprInfo* pExpr, SSDataBlock* pResult, SSDataBloc } int32_t startOffset = createNewColModel ? 0 : pResult->info.rows; - ASSERT(pResult->info.capacity > 0); + if (pResult->info.capacity <= 0) { + qError("project failed at: %s:%d", __func__, __LINE__); + code = TSDB_CODE_INVALID_PARA; + TSDB_CHECK_CODE(code, lino, _exit); + } int32_t ret = colDataMergeCol(pResColData, startOffset, (int32_t*)&pResult->info.capacity, &idata, dest.numOfRows); if (ret < 0) { code = ret; diff --git a/source/libs/executor/src/querytask.c b/source/libs/executor/src/querytask.c index 881b4f9316..c39ccd6ba5 100644 --- a/source/libs/executor/src/querytask.c +++ b/source/libs/executor/src/querytask.c @@ -84,7 +84,7 @@ bool isTaskKilled(void* pTaskInfo) { return (0 != ((SExecTaskInfo*)pTaskInfo)->c void setTaskKilled(SExecTaskInfo* pTaskInfo, int32_t rspCode) { pTaskInfo->code = rspCode; - (void) stopTableScanOperator(pTaskInfo->pRoot, pTaskInfo->id.str, &pTaskInfo->storageAPI); + (void)stopTableScanOperator(pTaskInfo->pRoot, pTaskInfo->id.str, &pTaskInfo->storageAPI); } void setTaskStatus(SExecTaskInfo* pTaskInfo, int8_t status) { @@ -128,14 +128,15 @@ int32_t createExecTaskInfo(SSubplan* pPlan, SExecTaskInfo** pTaskInfo, SReadHand void cleanupQueriedTableScanInfo(void* p) { SSchemaInfo* pSchemaInfo = p; - + taosMemoryFreeClear(pSchemaInfo->dbname); taosMemoryFreeClear(pSchemaInfo->tablename); tDeleteSchemaWrapper(pSchemaInfo->sw); tDeleteSchemaWrapper(pSchemaInfo->qsw); } -int32_t initQueriedTableSchemaInfo(SReadHandle* pHandle, SScanPhysiNode* pScanNode, const char* dbName, SExecTaskInfo* pTaskInfo) { +int32_t initQueriedTableSchemaInfo(SReadHandle* pHandle, SScanPhysiNode* pScanNode, const char* dbName, + SExecTaskInfo* pTaskInfo) { SMetaReader mr = {0}; if (pHandle == NULL) { return TSDB_CODE_INVALID_PARA; @@ -286,7 +287,7 @@ void buildTaskId(uint64_t taskId, uint64_t queryId, char* dst) { memcpy(p, "TID:0x", offset); offset += tintToHex(taskId, &p[offset]); - memcpy(&p[offset], " QID:0x", 7); + memcpy(&p[offset], " qid:0x", 7); offset += 7; offset += tintToHex(queryId, &p[offset]); diff --git a/source/libs/executor/src/scanoperator.c b/source/libs/executor/src/scanoperator.c index 09a6c7ca1e..93a03bdffa 100644 --- a/source/libs/executor/src/scanoperator.c +++ b/source/libs/executor/src/scanoperator.c @@ -592,6 +592,7 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int SMetaReader mr = {0}; const char* idStr = pTask->id.str; int32_t insertRet = TAOS_LRU_STATUS_OK; + STableCachedVal* pVal = NULL; // currently only the tbname pseudo column if (numOfExpr <= 0) { @@ -656,19 +657,11 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int pHandle->api.metaReaderFn.readerReleaseLock(&mr); - STableCachedVal* pVal = NULL; code = createTableCacheVal(&mr, &pVal); QUERY_CHECK_CODE(code, lino, _end); val = *pVal; freeReader = true; - - insertRet = taosLRUCacheInsert(pCache->pTableMetaEntryCache, &pBlock->info.id.uid, sizeof(uint64_t), pVal, - sizeof(STableCachedVal), freeCachedMetaItem, NULL, TAOS_LRU_PRIORITY_LOW, NULL); - if (insertRet != TAOS_LRU_STATUS_OK) { - qError("failed to put meta into lru cache, code:%d, %s", insertRet, idStr); - taosMemoryFreeClear(pVal); - } } else { pCache->cacheHit += 1; STableCachedVal* pVal = taosLRUCacheValue(pCache->pTableMetaEntryCache, h); @@ -742,9 +735,17 @@ int32_t addTagPseudoColumnData(SReadHandle* pHandle, const SExprInfo* pExpr, int pBlock->info.rows = backupRows; _end: - if (insertRet != TAOS_LRU_STATUS_OK) { - freeTableCachedValObj(&val); + + if (NULL != pVal) { + insertRet = taosLRUCacheInsert(pCache->pTableMetaEntryCache, &pBlock->info.id.uid, sizeof(uint64_t), pVal, + sizeof(STableCachedVal), freeCachedMetaItem, NULL, TAOS_LRU_PRIORITY_LOW, NULL); + if (insertRet != TAOS_LRU_STATUS_OK) { + qError("failed to put meta into lru cache, code:%d, %s", insertRet, idStr); + taosMemoryFreeClear(pVal); + freeTableCachedValObj(&val); + } } + if (freeReader) { pHandle->api.metaReaderFn.clearReader(&mr); } @@ -1082,7 +1083,10 @@ static SSDataBlock* doGroupedTableScan(SOperatorInfo* pOperator) { return getBlockForEmptyTable(pOperator, pStart); } } else { // group by tag + no sort - int32_t numOfTables = tableListGetSize(pTableListInfo); + int32_t numOfTables = 0; + code = tableListGetSize(pTableListInfo, &numOfTables); + QUERY_CHECK_CODE(code, lino, _end); + if (pTableScanInfo->tableEndIndex + 1 >= numOfTables) { // get empty group, mark processed & rm from hash void* pIte = taosHashIterate(pTableListInfo->remainGroups, NULL); @@ -1171,7 +1175,10 @@ static SSDataBlock* startNextGroupScan(SOperatorInfo* pOperator) { STableScanInfo* pInfo = pOperator->info; SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SStorageAPI* pAPI = &pTaskInfo->storageAPI; - int32_t numOfTables = tableListGetSize(pInfo->base.pTableListInfo); + int32_t numOfTables = 0; + code = tableListGetSize(pInfo->base.pTableListInfo, &numOfTables); + QUERY_CHECK_CODE(code, lino, _end); + if ((++pInfo->currentGroupId) >= tableListGetOutputGroups(pInfo->base.pTableListInfo)) { setOperatorCompleted(pOperator); if (pOperator->dynamicTask) { @@ -1304,7 +1311,7 @@ static int32_t doTableScanNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) { // scan table one by one sequentially if (pInfo->scanMode == TABLE_SCAN__TABLE_ORDER) { - int32_t numOfTables = 0; // tableListGetSize(pTaskInfo->pTableListInfo); + int32_t numOfTables = 0; STableKeyInfo tInfo = {0}; pInfo->countState = TABLE_COUNT_STATE_END; @@ -1319,7 +1326,13 @@ static int32_t doTableScanNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) { pInfo->currentTable++; taosRLockLatch(&pTaskInfo->lock); - numOfTables = tableListGetSize(pInfo->base.pTableListInfo); + numOfTables = 0; + code = tableListGetSize(pInfo->base.pTableListInfo, &numOfTables); + if (code != TSDB_CODE_SUCCESS) { + taosRUnLockLatch(&pTaskInfo->lock); + lino = __LINE__; + goto _end; + } if (pInfo->currentTable >= numOfTables) { qDebug("all table checked in table list, total:%d, return NULL, %s", numOfTables, GET_TASKID(pTaskInfo)); @@ -2592,6 +2605,7 @@ static int32_t doBlockDataPrimaryKeyFilter(SSDataBlock* pBlock, STqOffsetVal* of TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); __compar_fn_t func = getComparFunc(pColPk->info.type, 0); + QUERY_CHECK_NULL(func, code, lino, _end, terrno); for (int32_t i = 0; i < pBlock->info.rows; ++i) { int64_t* ts = (int64_t*)colDataGetData(pColTs, i); void* data = colDataGetData(pColPk, i); @@ -3611,7 +3625,9 @@ static int32_t extractTableIdList(const STableListInfo* pTableListInfo, SArray** QUERY_CHECK_NULL(tableIdList, code, lino, _end, terrno); // Transfer the Array of STableKeyInfo into uid list. - size_t size = tableListGetSize(pTableListInfo); + int32_t size = 0; + code = tableListGetSize(pTableListInfo, &size); + QUERY_CHECK_CODE(code, lino, _end); for (int32_t i = 0; i < size; ++i) { STableKeyInfo* pkeyInfo = tableListGetInfo(pTableListInfo, i); QUERY_CHECK_NULL(pkeyInfo, code, lino, _end, terrno); @@ -4627,7 +4643,13 @@ static int32_t doTagScanFromMetaEntryNext(SOperatorInfo* pOperator, SSDataBlock* SSDataBlock* pRes = pInfo->pRes; blockDataCleanup(pRes); - int32_t size = tableListGetSize(pInfo->pTableListInfo); + int32_t size = 0; + code = tableListGetSize(pInfo->pTableListInfo, &size); + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + return code; + } + if (size == 0) { setTaskStatus(pTaskInfo, TASK_COMPLETED); (*ppRes) = NULL; @@ -4750,7 +4772,7 @@ int32_t createTagScanOperatorInfo(SReadHandle* pReadHandle, STagScanPhysiNode* p QUERY_CHECK_NULL(pInfo->filterCtx.colHash, code, lino, _error, terrno); pInfo->filterCtx.cInfoList = taosArrayInit(4, sizeof(SColumnInfo)); - QUERY_CHECK_NULL(pInfo->pRes, code, lino, _error, terrno); + QUERY_CHECK_NULL(pInfo->filterCtx.cInfoList, code, lino, _error, terrno); if (pInfo->pTagCond != NULL) { nodesRewriteExprPostOrder(&pTagCond, tagScanRewriteTagColumn, (void*)&pInfo->filterCtx); @@ -4925,7 +4947,13 @@ static int32_t fetchNextSubTableBlockFromReader(SOperatorInfo* pOperator, STmsSu static int32_t setGroupStartEndIndex(STableMergeScanInfo* pInfo) { pInfo->bGroupProcessed = false; - size_t numOfTables = tableListGetSize(pInfo->base.pTableListInfo); + int32_t numOfTables = 0; + int32_t code = tableListGetSize(pInfo->base.pTableListInfo, &numOfTables); + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + return code; + } + int32_t i = pInfo->tableStartIndex + 1; for (; i < numOfTables; ++i) { STableKeyInfo* tableKeyInfo = tableListGetInfo(pInfo->base.pTableListInfo, i); @@ -5117,9 +5145,7 @@ static int32_t adjustSubTableForNextRow(SOperatorInfo* pOperatorInfo, STmsSubTab } } - tMergeTreeAdjust(pSubTblsInfo->pTree, tMergeTreeGetAdjustIndex(pSubTblsInfo->pTree)); - - return TSDB_CODE_SUCCESS; + return tMergeTreeAdjust(pSubTblsInfo->pTree, tMergeTreeGetAdjustIndex(pSubTblsInfo->pTree)); } static int32_t appendChosenRowToDataBlock(STmsSubTablesMergeInfo* pSubTblsInfo, SSDataBlock* pBlock) { @@ -5266,7 +5292,10 @@ int32_t doTableMergeScanParaSubTablesNext(SOperatorInfo* pOperator, SSDataBlock* int64_t st = taosGetTimestampUs(); - size_t tableListSize = tableListGetSize(pInfo->base.pTableListInfo); + int32_t tableListSize = 0; + code = tableListGetSize(pInfo->base.pTableListInfo, &tableListSize); + QUERY_CHECK_CODE(code, lino, _end); + if (!pInfo->hasGroupId) { pInfo->hasGroupId = true; @@ -5413,7 +5442,7 @@ static void doGetBlockForTableMergeScan(SOperatorInfo* pOperator, bool* pFinishe return; } -static SSDataBlock* getBlockForTableMergeScan(void* param) { +static int32_t getBlockForTableMergeScan(void* param, SSDataBlock** ppBlock) { STableMergeScanSortSourceParam* source = param; SOperatorInfo* pOperator = source->pOperator; @@ -5421,6 +5450,7 @@ static SSDataBlock* getBlockForTableMergeScan(void* param) { SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo; SSDataBlock* pBlock = NULL; int64_t st = taosGetTimestampUs(); + int32_t code = TSDB_CODE_SUCCESS; while (true) { if (pInfo->rtnNextDurationBlocks) { @@ -5455,10 +5485,11 @@ static SSDataBlock* getBlockForTableMergeScan(void* param) { if (pInfo->bNextDurationBlockEvent || pInfo->bNewFilesetEvent) { if (!bSkipped) { - int32_t code = createOneDataBlock(pBlock, true, &pInfo->nextDurationBlocks[pInfo->numNextDurationBlocks]); + code = createOneDataBlock(pBlock, true, &pInfo->nextDurationBlocks[pInfo->numNextDurationBlocks]); if (code) { terrno = code; - return NULL; + *ppBlock = NULL; + return code; } ++pInfo->numNextDurationBlocks; @@ -5472,7 +5503,8 @@ static SSDataBlock* getBlockForTableMergeScan(void* param) { if (pInfo->bNewFilesetEvent) { pInfo->rtnNextDurationBlocks = true; - return NULL; + *ppBlock = NULL; + return code; } if (pInfo->bNextDurationBlockEvent) { @@ -5487,11 +5519,13 @@ static SSDataBlock* getBlockForTableMergeScan(void* param) { pOperator->resultInfo.totalRows += pBlock->info.rows; pInfo->base.readRecorder.elapsedTime += (taosGetTimestampUs() - st) / 1000.0; - - return pBlock; + *ppBlock = pBlock; + + return code; } - return NULL; + *ppBlock = NULL; + return code; } int32_t generateSortByTsPkInfo(SArray* colMatchInfo, int32_t order, SArray** ppSortArray) { @@ -5638,7 +5672,10 @@ void startGroupTableMergeScan(SOperatorInfo* pOperator) { qDebug("%s table merge scan start group %" PRIu64, GET_TASKID(pTaskInfo), pInfo->groupId); { - size_t numOfTables = tableListGetSize(pInfo->base.pTableListInfo); + int32_t numOfTables = 0; + code = tableListGetSize(pInfo->base.pTableListInfo, &numOfTables); + QUERY_CHECK_CODE(code, lino, _end); + int32_t i = pInfo->tableStartIndex + 1; for (; i < numOfTables; ++i) { STableKeyInfo* tableKeyInfo = tableListGetInfo(pInfo->base.pTableListInfo, i); @@ -5712,7 +5749,11 @@ SSDataBlock* getSortedTableMergeScanBlockData(SSortHandle* pHandle, SSDataBlock* while (1) { pTupleHandle = NULL; int32_t code = tsortNextTuple(pHandle, &pTupleHandle); - if (pTupleHandle == NULL || code != 0) { + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + T_LONG_JMP(pOperator->pTaskInfo->env, code); + } + if (pTupleHandle == NULL) { break; } @@ -5758,7 +5799,10 @@ int32_t doTableMergeScanNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) { int64_t st = taosGetTimestampUs(); - size_t tableListSize = tableListGetSize(pInfo->base.pTableListInfo); + int32_t tableListSize = 0; + code = tableListGetSize(pInfo->base.pTableListInfo, &tableListSize); + QUERY_CHECK_CODE(code, lino, _end); + if (!pInfo->hasGroupId) { pInfo->hasGroupId = true; diff --git a/source/libs/executor/src/sortoperator.c b/source/libs/executor/src/sortoperator.c index cb15c3c836..d9b1e40510 100644 --- a/source/libs/executor/src/sortoperator.c +++ b/source/libs/executor/src/sortoperator.c @@ -295,6 +295,9 @@ static int32_t getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, break; } } + if (TSDB_CODE_SUCCESS != code) { + return code; + } if (p->info.rows > 0) { code = blockDataEnsureCapacity(pDataBlock, capacity); @@ -337,10 +340,10 @@ static int32_t getSortedBlockData(SSortHandle* pHandle, SSDataBlock* pDataBlock, return code; } -SSDataBlock* loadNextDataBlock(void* param) { +int32_t loadNextDataBlock(void* param, SSDataBlock** ppBlock) { SOperatorInfo* pOperator = (SOperatorInfo*)param; - SSDataBlock* pBlock = pOperator->fpSet.getNextFn(pOperator); - return pBlock; + *ppBlock = pOperator->fpSet.getNextFn(pOperator); + return TSDB_CODE_SUCCESS; } // todo refactor: merged with fetch fp @@ -609,30 +612,32 @@ typedef struct SGroupSortSourceParam { SGroupSortOperatorInfo* grpSortOpInfo; } SGroupSortSourceParam; -SSDataBlock* fetchNextGroupSortDataBlock(void* param) { +int32_t fetchNextGroupSortDataBlock(void* param, SSDataBlock** ppBlock) { + *ppBlock = NULL; + SGroupSortSourceParam* source = param; SGroupSortOperatorInfo* grpSortOpInfo = source->grpSortOpInfo; if (grpSortOpInfo->prefetchedSortInput) { SSDataBlock* block = grpSortOpInfo->prefetchedSortInput; grpSortOpInfo->prefetchedSortInput = NULL; - return block; + *ppBlock = block; } else { SOperatorInfo* childOp = source->childOpInfo; SSDataBlock* block = childOp->fpSet.getNextFn(childOp); if (block != NULL) { if (block->info.id.groupId == grpSortOpInfo->currGroupId) { grpSortOpInfo->childOpStatus = CHILD_OP_SAME_GROUP; - return block; + *ppBlock = block; } else { grpSortOpInfo->childOpStatus = CHILD_OP_NEW_GROUP; grpSortOpInfo->prefetchedSortInput = block; - return NULL; } } else { grpSortOpInfo->childOpStatus = CHILD_OP_FINISHED; - return NULL; } } + + return TSDB_CODE_SUCCESS; } int32_t beginSortGroup(SOperatorInfo* pOperator) { diff --git a/source/libs/executor/src/sysscanoperator.c b/source/libs/executor/src/sysscanoperator.c index 0a8d0b4cbd..70cb6f4be4 100644 --- a/source/libs/executor/src/sysscanoperator.c +++ b/source/libs/executor/src/sysscanoperator.c @@ -343,27 +343,45 @@ int optSysDoCompare(__compar_fn_t func, int8_t comparType, void* a, void* b) { static int optSysFilterFuncImpl__LowerThan(void* a, void* b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return optSysDoCompare(func, OP_TYPE_LOWER_THAN, a, b); } static int optSysFilterFuncImpl__LowerEqual(void* a, void* b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return optSysDoCompare(func, OP_TYPE_LOWER_EQUAL, a, b); } static int optSysFilterFuncImpl__GreaterThan(void* a, void* b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return optSysDoCompare(func, OP_TYPE_GREATER_THAN, a, b); } static int optSysFilterFuncImpl__GreaterEqual(void* a, void* b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return optSysDoCompare(func, OP_TYPE_GREATER_EQUAL, a, b); } static int optSysFilterFuncImpl__Equal(void* a, void* b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return optSysDoCompare(func, OP_TYPE_EQUAL, a, b); } static int optSysFilterFuncImpl__NoEqual(void* a, void* b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return optSysDoCompare(func, OP_TYPE_NOT_EQUAL, a, b); } @@ -1337,8 +1355,8 @@ _end: } static int32_t doSetUserTableMetaInfo(SStoreMetaReader* pMetaReaderFn, SStoreMeta* pMetaFn, void* pVnode, - SMTbCursor* pCur, SMetaReader* pMReader, int64_t uid, const char* dbname, - int32_t vgId, SSDataBlock* p, int32_t rowIndex, const char* idStr) { + SMetaReader* pMReader, int64_t uid, const char* dbname, int32_t vgId, + SSDataBlock* p, int32_t rowIndex, const char* idStr) { char n[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0}; int32_t lino = 0; int32_t code = pMetaReaderFn->getTableEntryByUid(pMReader, uid); @@ -1383,7 +1401,7 @@ static int32_t doSetUserTableMetaInfo(SStoreMetaReader* pMetaReaderFn, SStoreMet int64_t suid = pMReader->me.ctbEntry.suid; code = pMetaReaderFn->getTableEntryByUid(&mr1, suid); if (code != TSDB_CODE_SUCCESS) { - qError("failed to get super table meta, cname:%s, suid:0x%" PRIx64 ", code:%s, %s", pCur->mr.me.name, suid, + qError("failed to get super table meta, cname:%s, suid:0x%" PRIx64 ", code:%s, %s", pMReader->me.name, suid, tstrerror(code), idStr); pMetaReaderFn->clearReader(&mr1); QUERY_CHECK_CODE(code, lino, _end); @@ -1448,14 +1466,14 @@ static int32_t doSetUserTableMetaInfo(SStoreMetaReader* pMetaReaderFn, SStoreMet // create time pColInfoData = taosArrayGet(p->pDataBlock, 2); QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno); - code = colDataSetVal(pColInfoData, rowIndex, (char*)&pCur->mr.me.ntbEntry.btime, false); + code = colDataSetVal(pColInfoData, rowIndex, (char*)&pMReader->me.ntbEntry.btime, false); QUERY_CHECK_CODE(code, lino, _end); // number of columns pColInfoData = taosArrayGet(p->pDataBlock, 3); QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno); - code = colDataSetVal(pColInfoData, rowIndex, (char*)&pCur->mr.me.ntbEntry.schemaRow.nCols, false); + code = colDataSetVal(pColInfoData, rowIndex, (char*)&pMReader->me.ntbEntry.schemaRow.nCols, false); QUERY_CHECK_CODE(code, lino, _end); // super table name @@ -1542,8 +1560,8 @@ static SSDataBlock* sysTableBuildUserTablesByUids(SOperatorInfo* pOperator) { SMetaReader mr = {0}; pAPI->metaReaderFn.initReader(&mr, pInfo->readHandle.vnode, META_READER_LOCK, &pAPI->metaFn); - doSetUserTableMetaInfo(&pAPI->metaReaderFn, &pAPI->metaFn, pInfo->readHandle.vnode, pInfo->pCur, &mr, *uid, dbname, - vgId, p, numOfRows, GET_TASKID(pTaskInfo)); + doSetUserTableMetaInfo(&pAPI->metaReaderFn, &pAPI->metaFn, pInfo->readHandle.vnode, &mr, *uid, dbname, vgId, p, + numOfRows, GET_TASKID(pTaskInfo)); pAPI->metaReaderFn.clearReader(&mr); @@ -2852,7 +2870,10 @@ int32_t createDataBlockInfoScanOperator(SReadHandle* readHandle, SBlockDistScanP QUERY_CHECK_CODE(code, lino, _error); pInfo->pTableListInfo = pTableListInfo; - size_t num = tableListGetSize(pTableListInfo); + int32_t num = 0; + code = tableListGetSize(pTableListInfo, &num); + QUERY_CHECK_CODE(code, lino, _error); + void* pList = tableListGetInfo(pTableListInfo, 0); code = readHandle->api.tsdReader.tsdReaderOpen(readHandle->vnode, &cond, pList, num, pInfo->pResBlock, diff --git a/source/libs/executor/src/tfill.c b/source/libs/executor/src/tfill.c index 59c19a706c..cdfbd7a850 100644 --- a/source/libs/executor/src/tfill.c +++ b/source/libs/executor/src/tfill.c @@ -324,8 +324,11 @@ _end: static void saveColData(SArray* rowBuf, int32_t columnIndex, const char* src, bool isNull); -static void copyCurrentRowIntoBuf(SFillInfo* pFillInfo, int32_t rowIndex, SRowVal* pRowVal, bool reset) { +static int32_t copyCurrentRowIntoBuf(SFillInfo* pFillInfo, int32_t rowIndex, SRowVal* pRowVal, bool reset) { + int32_t code = TSDB_CODE_SUCCESS; + int32_t lino = 0; SColumnInfoData* pTsCol = taosArrayGet(pFillInfo->pSrcBlock->pDataBlock, pFillInfo->srcTsSlotId); + QUERY_CHECK_NULL(pTsCol, code, lino, _end, terrno); pRowVal->key = ((int64_t*)pTsCol->pData)[rowIndex]; for (int32_t i = 0; i < pFillInfo->numOfCols; ++i) { @@ -342,15 +345,24 @@ static void copyCurrentRowIntoBuf(SFillInfo* pFillInfo, int32_t rowIndex, SRowVa } SColumnInfoData* pSrcCol = taosArrayGet(pFillInfo->pSrcBlock->pDataBlock, srcSlotId); + QUERY_CHECK_NULL(pSrcCol, code, lino, _end, terrno); bool isNull = colDataIsNull_s(pSrcCol, rowIndex); char* p = colDataGetData(pSrcCol, rowIndex); saveColData(pRowVal->pRowVal, i, p, reset ? true : isNull); } else { - ASSERT(0); + code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(code)); + QUERY_CHECK_CODE(code, lino, _end); } } + +_end: + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); + } + return code; } static int32_t fillResultImpl(SFillInfo* pFillInfo, SSDataBlock* pBlock, int32_t outputRows) { @@ -362,20 +374,18 @@ static int32_t fillResultImpl(SFillInfo* pFillInfo, SSDataBlock* pBlock, int32_t int32_t step = GET_FORWARD_DIRECTION_FACTOR(pFillInfo->order); bool ascFill = FILL_IS_ASC_FILL(pFillInfo); -#if 0 - ASSERT(ascFill && (pFillInfo->currentKey >= pFillInfo->start) || (!ascFill && (pFillInfo->currentKey <= pFillInfo->start))); -#endif - while (pFillInfo->numOfCurrent < outputRows) { int64_t ts = ((int64_t*)pTsCol->pData)[pFillInfo->index]; // set the next value for interpolation if (pFillInfo->currentKey < ts && ascFill) { SRowVal* pRVal = pFillInfo->type == TSDB_FILL_NEXT ? &pFillInfo->next : &pFillInfo->prev; - copyCurrentRowIntoBuf(pFillInfo, pFillInfo->index, pRVal, false); + code = copyCurrentRowIntoBuf(pFillInfo, pFillInfo->index, pRVal, false); + QUERY_CHECK_CODE(code, lino, _end); } else if (pFillInfo->currentKey > ts && !ascFill) { SRowVal* pRVal = pFillInfo->type == TSDB_FILL_NEXT ? &pFillInfo->prev : &pFillInfo->next; - copyCurrentRowIntoBuf(pFillInfo, pFillInfo->index, pRVal, false); + code = copyCurrentRowIntoBuf(pFillInfo, pFillInfo->index, pRVal, false); + QUERY_CHECK_CODE(code, lino, _end); } if (((pFillInfo->currentKey < ts && ascFill) || (pFillInfo->currentKey > ts && !ascFill)) && @@ -392,21 +402,24 @@ static int32_t fillResultImpl(SFillInfo* pFillInfo, SSDataBlock* pBlock, int32_t goto _end; } } else { - ASSERT(pFillInfo->currentKey == ts); + QUERY_CHECK_CONDITION((pFillInfo->currentKey == ts), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); int32_t index = pBlock->info.rows; int32_t nextRowIndex = pFillInfo->index + 1; if (pFillInfo->type == TSDB_FILL_NEXT) { if ((pFillInfo->index + 1) < pFillInfo->numOfRows) { - copyCurrentRowIntoBuf(pFillInfo, nextRowIndex, &pFillInfo->next, false); + code = copyCurrentRowIntoBuf(pFillInfo, nextRowIndex, &pFillInfo->next, false); + QUERY_CHECK_CODE(code, lino, _end); } else { // reset to null after last row - copyCurrentRowIntoBuf(pFillInfo, nextRowIndex, &pFillInfo->next, true); + code = copyCurrentRowIntoBuf(pFillInfo, nextRowIndex, &pFillInfo->next, true); + QUERY_CHECK_CODE(code, lino, _end); } } if (pFillInfo->type == TSDB_FILL_PREV) { if (nextRowIndex + 1 >= pFillInfo->numOfRows && !FILL_IS_ASC_FILL(pFillInfo)) { - copyCurrentRowIntoBuf(pFillInfo, nextRowIndex, &pFillInfo->next, true); + code = copyCurrentRowIntoBuf(pFillInfo, nextRowIndex, &pFillInfo->next, true); + QUERY_CHECK_CODE(code, lino, _end); } } @@ -500,7 +513,9 @@ static void saveColData(SArray* rowBuf, int32_t columnIndex, const char* src, bo } } -static void appendFilledResult(SFillInfo* pFillInfo, SSDataBlock* pBlock, int64_t resultCapacity) { +static int32_t appendFilledResult(SFillInfo* pFillInfo, SSDataBlock* pBlock, int64_t resultCapacity) { + int32_t code = TSDB_CODE_SUCCESS; + int32_t lino = 0; /* * These data are generated according to fill strategy, since the current timestamp is out of the time window of * real result set. Note that we need to keep the direct previous result rows, to generated the filled data. @@ -512,7 +527,14 @@ static void appendFilledResult(SFillInfo* pFillInfo, SSDataBlock* pBlock, int64_ pFillInfo->numOfTotal += pFillInfo->numOfCurrent; - ASSERT(pFillInfo->numOfCurrent == resultCapacity); + QUERY_CHECK_CONDITION((pFillInfo->numOfCurrent == resultCapacity), code, lino, _end, + TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); + +_end: + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); + } + return code; } static int32_t taosNumOfRemainRows(SFillInfo* pFillInfo) { @@ -635,15 +657,6 @@ void taosFillSetStartInfo(SFillInfo* pFillInfo, int32_t numOfRows, TSKEY endKey) // the endKey is now the aligned time window value. truncate time window isn't correct. pFillInfo->end = endKey; - -#if 0 - if (pFillInfo->order == TSDB_ORDER_ASC) { - ASSERT(pFillInfo->start <= pFillInfo->end); - } else { - ASSERT(pFillInfo->start >= pFillInfo->end); - } -#endif - pFillInfo->index = 0; pFillInfo->numOfRows = numOfRows; } @@ -687,7 +700,6 @@ int64_t getNumOfResultsAfterFillGap(SFillInfo* pFillInfo, TSKEY ekey, int32_t ma numOfRes = taosTimeCountIntervalForFill(lastKey, pFillInfo->currentKey, pFillInfo->interval.sliding, pFillInfo->interval.slidingUnit, pFillInfo->interval.precision, pFillInfo->order); - ASSERT(numOfRes >= numOfRows); } else { // reach the end of data if ((ekey1 < pFillInfo->currentKey && FILL_IS_ASC_FILL(pFillInfo)) || (ekey1 > pFillInfo->currentKey && !FILL_IS_ASC_FILL(pFillInfo))) { @@ -719,23 +731,30 @@ void taosGetLinearInterpolationVal(SPoint* point, int32_t outputType, SPoint* po int32_t taosFillResultDataBlock(SFillInfo* pFillInfo, SSDataBlock* p, int32_t capacity) { int32_t code = TSDB_CODE_SUCCESS; + int32_t lino = 0; int32_t remain = taosNumOfRemainRows(pFillInfo); int64_t numOfRes = getNumOfResultsAfterFillGap(pFillInfo, pFillInfo->end, capacity); - ASSERT(numOfRes <= capacity); + QUERY_CHECK_CONDITION((numOfRes <= capacity), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); // no data existed for fill operation now, append result according to the fill strategy if (remain == 0) { - appendFilledResult(pFillInfo, p, numOfRes); + code = appendFilledResult(pFillInfo, p, numOfRes); + QUERY_CHECK_CODE(code, lino, _end); } else { code = fillResultImpl(pFillInfo, p, (int32_t)numOfRes); - ASSERT(numOfRes == pFillInfo->numOfCurrent); + QUERY_CHECK_CODE(code, lino, _end); + QUERY_CHECK_CONDITION((numOfRes == pFillInfo->numOfCurrent), code, lino, _end, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR); } qDebug("fill:%p, generated fill result, src block:%d, index:%d, brange:%" PRId64 "-%" PRId64 ", currentKey:%" PRId64 ", current : % d, total : % d, %s", pFillInfo, pFillInfo->numOfRows, pFillInfo->index, pFillInfo->start, pFillInfo->end, pFillInfo->currentKey, pFillInfo->numOfCurrent, pFillInfo->numOfTotal, pFillInfo->id); +_end: + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); + } return code; } diff --git a/source/libs/executor/src/timesliceoperator.c b/source/libs/executor/src/timesliceoperator.c index f48393273f..7f23255257 100644 --- a/source/libs/executor/src/timesliceoperator.c +++ b/source/libs/executor/src/timesliceoperator.c @@ -1247,6 +1247,9 @@ void destroyTimeSliceOperatorInfo(void* param) { taosMemoryFree(pInfo->pPrevGroupKey->pData); taosMemoryFree(pInfo->pPrevGroupKey); } + if (pInfo->hasPk && IS_VAR_DATA_TYPE(pInfo->pkCol.type)) { + taosMemoryFreeClear(pInfo->prevKey.pks[0].pData); + } cleanupExprSupp(&pInfo->scalarSup); if (pInfo->pFillColInfo != NULL) { diff --git a/source/libs/executor/src/timewindowoperator.c b/source/libs/executor/src/timewindowoperator.c index 95d415f85c..c9bb1e69e0 100644 --- a/source/libs/executor/src/timewindowoperator.c +++ b/source/libs/executor/src/timewindowoperator.c @@ -1256,7 +1256,7 @@ void destroyIntervalOperatorInfo(void* param) { taosMemoryFreeClear(param); } -static bool timeWindowinterpNeeded(SqlFunctionCtx* pCtx, int32_t numOfCols, SIntervalAggOperatorInfo* pInfo, +static int32_t timeWindowinterpNeeded(SqlFunctionCtx* pCtx, int32_t numOfCols, SIntervalAggOperatorInfo* pInfo, bool* pRes) { // the primary timestamp column bool needed = false; diff --git a/source/libs/executor/src/tlinearhash.c b/source/libs/executor/src/tlinearhash.c index 0c70428e78..55628c18f9 100644 --- a/source/libs/executor/src/tlinearhash.c +++ b/source/libs/executor/src/tlinearhash.c @@ -14,6 +14,7 @@ */ #include "tlinearhash.h" +#include "query.h" #include "taoserror.h" #include "tdef.h" #include "tpagedbuf.h" @@ -59,7 +60,11 @@ static int32_t doGetBucketIdFromHashVal(int32_t hashv, int32_t bits) { return ha static int32_t doGetAlternativeBucketId(int32_t bucketId, int32_t bits, int32_t numOfBuckets) { int32_t v = bucketId - (1ul << (bits - 1)); - ASSERT(v < numOfBuckets); + if (v >= numOfBuckets) { + qError("tlinearhash failed at: %s:%d", __func__, __LINE__); + terrno = TSDB_CODE_INTERNAL_ERROR; + return -1; + } return v; } @@ -85,11 +90,15 @@ static int32_t doAddToBucket(SLHashObj* pHashObj, SLHashBucket* pBucket, int32_t int32_t pageId = *(int32_t*)taosArrayGetLast(pBucket->pPageIdList); SFilePage* pPage = getBufPage(pHashObj->pBuf, pageId); - ASSERT(pPage != NULL); + if (pPage == NULL) { + return TSDB_CODE_INVALID_PARA; + } // put to current buf page size_t nodeSize = sizeof(SLHashNode) + keyLen + size; - ASSERT(nodeSize + sizeof(SFilePage) <= getBufPageSize(pHashObj->pBuf)); + if (nodeSize + sizeof(SFilePage) > getBufPageSize(pHashObj->pBuf)) { + return TSDB_CODE_INVALID_PARA; + } if (pPage->num + nodeSize > getBufPageSize(pHashObj->pBuf)) { releaseBufPage(pHashObj->pBuf, pPage); @@ -174,7 +183,7 @@ static void doTrimBucketPages(SLHashObj* pHashObj, SLHashBucket* pBucket) { setBufPageDirty(pFirst, true); setBufPageDirty(pLast, true); - ASSERT(pLast->num >= nodeSize + sizeof(SFilePage)); + // ASSERT(pLast->num >= nodeSize + sizeof(SFilePage)); pFirst->num += nodeSize; pLast->num -= nodeSize; @@ -320,6 +329,9 @@ int32_t tHashPut(SLHashObj* pHashObj, const void* key, size_t keyLen, void* data // printf("bucketId: 0x%x not exists, put it into 0x%x instead\n", v, newBucketId); v = newBucketId; } + if (v < 0) { + return terrno; + } SLHashBucket* pBucket = pHashObj->pBucket[v]; code = doAddToBucket(pHashObj, pBucket, v, key, keyLen, data, size); @@ -343,7 +355,10 @@ int32_t tHashPut(SLHashObj* pHashObj, const void* key, size_t keyLen, void* data int32_t numOfBits = ceil(log(pHashObj->numOfBuckets) / log(2)); if (numOfBits > pHashObj->bits) { // printf("extend the bits from %d to %d, new bucket:%d\n", pHashObj->bits, numOfBits, newBucketId); - ASSERT(numOfBits == pHashObj->bits + 1); + if (numOfBits != pHashObj->bits + 1) { + qError("linear hash faield at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } pHashObj->bits = numOfBits; } @@ -360,14 +375,20 @@ int32_t tHashPut(SLHashObj* pHashObj, const void* key, size_t keyLen, void* data char* pStart = p->data; while (pStart - ((char*)p) < p->num) { SLHashNode* pNode = (SLHashNode*)pStart; - ASSERT(pNode->keyLen > 0); + if (pNode->keyLen <= 0) { + qError("linear hash faield at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } char* k = GET_LHASH_NODE_KEY(pNode); int32_t hashv = pHashObj->hashFn(k, pNode->keyLen); int32_t v1 = doGetBucketIdFromHashVal(hashv, pHashObj->bits); if (v1 != splitBucketId) { // place it into the new bucket - ASSERT(v1 == newBucketId); + if (v1 != newBucketId) { + qError("linear hash failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INTERNAL_ERROR; + } // printf("move key:%d to 0x%x bucket, remain items:%d\n", *(int32_t*)k, v1, pBucket->size - 1); SLHashBucket* pNewBucket = pHashObj->pBucket[newBucketId]; code = doAddToBucket(pHashObj, pNewBucket, newBucketId, (void*)GET_LHASH_NODE_KEY(pNode), pNode->keyLen, @@ -395,6 +416,9 @@ char* tHashGet(SLHashObj* pHashObj, const void* key, size_t keyLen) { if (bucketId >= pHashObj->numOfBuckets) { bucketId = doGetAlternativeBucketId(bucketId, pHashObj->bits, pHashObj->numOfBuckets); } + if (bucketId < 0) { + return NULL; + } SLHashBucket* pBucket = pHashObj->pBucket[bucketId]; int32_t num = taosArrayGetSize(pBucket->pPageIdList); diff --git a/source/libs/executor/src/tsort.c b/source/libs/executor/src/tsort.c index 5bd11489c9..c1edf700d1 100644 --- a/source/libs/executor/src/tsort.c +++ b/source/libs/executor/src/tsort.c @@ -468,7 +468,11 @@ static int32_t doAddNewExternalMemSource(SDiskbasedBuf* pBuf, SArray* pAllSource // The value of numOfRows must be greater than 0, which is guaranteed by the previous memory allocation int32_t numOfRows = (getBufPageSize(pBuf) - blockDataGetSerialMetaSize(taosArrayGetSize(pBlock->pDataBlock))) / rowSize; - ASSERT(numOfRows > 0); + if (numOfRows <= 0) { + qError("sort failed at: %s:%d", __func__, __LINE__); + taosArrayDestroy(pPageIdList); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } return blockDataEnsureCapacity(pSource->src.pBlock, numOfRows); } @@ -528,7 +532,10 @@ static int32_t doAddToBuf(SSDataBlock* pDataBlock, SSortHandle* pHandle) { } int32_t size = blockDataGetSize(p) + sizeof(int32_t) + taosArrayGetSize(p->pDataBlock) * sizeof(int32_t); - ASSERT(size <= getBufPageSize(pHandle->pBuf)); + if (size > getBufPageSize(pHandle->pBuf)) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } code = blockDataToBuf(pPage, p); if (code) { @@ -619,7 +626,7 @@ static int32_t sortComparInit(SMsortComparParam* pParam, SArray* pSources, int32 for (int32_t i = 0; i < pParam->numOfSources; ++i) { SSortSource* pSource = pParam->pSources[i]; - pSource->src.pBlock = pHandle->fetchfp(pSource->param); + TAOS_CHECK_RETURN(pHandle->fetchfp(pSource->param, &pSource->src.pBlock)); // set current source is done if (pSource->src.pBlock == NULL) { @@ -711,7 +718,7 @@ static int32_t adjustMergeTreeForNextTuple(SSortSource* pSource, SMultiwayMergeT } } else { int64_t st = taosGetTimestampUs(); - pSource->src.pBlock = pHandle->fetchfp(((SSortSource*)pSource)->param); + TAOS_CHECK_RETURN(pHandle->fetchfp(((SSortSource*)pSource)->param, &pSource->src.pBlock)); pSource->fetchUs += taosGetTimestampUs() - st; pSource->fetchNum++; if (pSource->src.pBlock == NULL) { @@ -733,7 +740,10 @@ static int32_t adjustMergeTreeForNextTuple(SSortSource* pSource, SMultiwayMergeT tMergeTreePrint(pTree); #endif - tMergeTreeAdjust(pTree, leafNodeIndex); + int32_t code = tMergeTreeAdjust(pTree, leafNodeIndex); + if (TSDB_CODE_SUCCESS != code) { + return code; + } #ifdef _DEBUG_VIEW printf("\nafter adjust:\t"); @@ -1040,7 +1050,10 @@ static int32_t doInternalMergeSort(SSortHandle* pHandle) { int32_t size = blockDataGetSize(pDataBlock) + sizeof(int32_t) + taosArrayGetSize(pDataBlock->pDataBlock) * sizeof(int32_t); - ASSERT(size <= getBufPageSize(pHandle->pBuf)); + if (size > getBufPageSize(pHandle->pBuf)) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } code= blockDataToBuf(pPage, pDataBlock); if (code) { @@ -1310,7 +1323,10 @@ static int32_t getRowBufFromExtMemFile(SSortHandle* pHandle, int32_t regionId, i } pRegion->bufLen = readBytes; } - ASSERT(pRegion->bufRegOffset <= tupleOffset); + if (pRegion->bufRegOffset > tupleOffset) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } if (pRegion->bufRegOffset + pRegion->bufLen >= tupleOffset + rowLen) { *pFreeRow = false; *ppRow = pRegion->buf + tupleOffset - pRegion->bufRegOffset; @@ -1350,7 +1366,7 @@ static int32_t createSortMemFile(SSortHandle* pHandle) { } if (code == TSDB_CODE_SUCCESS) { taosGetTmpfilePath(tsTempDir, "sort-ext-mem", pMemFile->memFilePath); - pMemFile->pTdFile = taosOpenCFile(pMemFile->memFilePath, "w+"); + pMemFile->pTdFile = taosOpenCFile(pMemFile->memFilePath, "w+b"); if (pMemFile->pTdFile == NULL) { code = terrno = TAOS_SYSTEM_ERROR(errno); } @@ -1487,7 +1503,10 @@ static int32_t tsortCloseRegion(SSortHandle* pHandle) { static int32_t tsortFinalizeRegions(SSortHandle* pHandle) { SSortMemFile* pMemFile = pHandle->pExtRowsMemFile; size_t numRegions = taosArrayGetSize(pMemFile->aFileRegions); - ASSERT(numRegions == (pMemFile->currRegionId + 1)); + if (numRegions != (pMemFile->currRegionId + 1)) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } if (numRegions == 0) { return TSDB_CODE_SUCCESS; } @@ -1812,7 +1831,10 @@ static int32_t appendDataBlockToPageBuf(SSortHandle* pHandle, SSDataBlock* blk, } int32_t size = blockDataGetSize(blk) + sizeof(int32_t) + taosArrayGetSize(blk->pDataBlock) * sizeof(int32_t); - ASSERT(size <= getBufPageSize(pHandle->pBuf)); + if (size > getBufPageSize(pHandle->pBuf)) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } int32_t code = blockDataToBuf(pPage, blk); @@ -1863,12 +1885,18 @@ static int32_t getPageBufIncForRowIdSort(SSDataBlock* pDstBlock, int32_t srcRowI int32_t numOfCols = blockDataGetNumOfCols(pDstBlock); if (pPkCol == NULL) { // no var column - ASSERT((numOfCols == 4) && (!pDstBlock->info.hasVarCol)); + if (!((numOfCols == 4) && (!pDstBlock->info.hasVarCol))) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } size += numOfCols * ((dstRowIndex & 0x7) == 0 ? 1: 0); size += blockDataGetRowSize(pDstBlock); } else { - ASSERT(numOfCols == 5); + if (numOfCols != 5) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } size += (numOfCols - 1) * (((dstRowIndex & 0x7) == 0)? 1:0); for(int32_t i = 0; i < numOfCols - 1; ++i) { @@ -2018,7 +2046,7 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* SArray* aPgId = taosArrayInit(8, sizeof(int32_t)); if (aPgId == NULL) { - return terrno; + goto _error; } int32_t nRows = 0; @@ -2039,10 +2067,7 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* lastPageBufTs = ((int64_t*)tsCol->pData)[pHandle->pDataBlock->info.rows - 1]; code = appendDataBlockToPageBuf(pHandle, pHandle->pDataBlock, aPgId); if (code != TSDB_CODE_SUCCESS) { - taosMemoryFree(pTree); - taosArrayDestroy(aPgId); - cleanupMergeSup(&sup); - return code; + goto _error; } nMergedRows += pHandle->pDataBlock->info.rows; @@ -2064,7 +2089,7 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* code = blockDataEnsureCapacity(pHandle->pDataBlock, pHandle->pDataBlock->info.rows + 1); if (code) { - return code; + goto _error; } if (pHandle->bSortByRowId) { @@ -2074,11 +2099,14 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* } if (code) { - return code; + goto _error; } blkPgSz += bufInc; - ASSERT(blkPgSz == blockDataGetSize(pHandle->pDataBlock) + pageHeaderSize); + if (blkPgSz != blockDataGetSize(pHandle->pDataBlock) + pageHeaderSize) { + qError("sort failed at: %s:%d", __func__, __LINE__); + goto _error; + } ++nRows; @@ -2087,7 +2115,10 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* } else { ++sup.aRowIdx[minIdx]; } - tMergeTreeAdjust(pTree, tMergeTreeGetAdjustIndex(pTree)); + code = tMergeTreeAdjust(pTree, tMergeTreeGetAdjustIndex(pTree)); + if (TSDB_CODE_SUCCESS != code) { + goto _error; + } } if (pHandle->pDataBlock->info.rows > 0) { @@ -2096,10 +2127,7 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* lastPageBufTs = ((int64_t*)tsCol->pData)[pHandle->pDataBlock->info.rows - 1]; code = appendDataBlockToPageBuf(pHandle, pHandle->pDataBlock, aPgId); if (code != TSDB_CODE_SUCCESS) { - taosArrayDestroy(aPgId); - taosMemoryFree(pTree); - cleanupMergeSup(&sup); - return code; + goto _error; } nMergedRows += pHandle->pDataBlock->info.rows; if ((pHandle->mergeLimit != -1) && (nMergedRows >= pHandle->mergeLimit)) { @@ -2115,11 +2143,7 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* SSDataBlock* pMemSrcBlk = NULL; code = createOneDataBlock(pHandle->pDataBlock, false, &pMemSrcBlk); - if (code) { - cleanupMergeSup(&sup); - tMergeTreeDestroy(&pTree); - return code; - } + if (code) goto _error; code = doAddNewExternalMemSource(pHandle->pBuf, aExtSrc, pMemSrcBlk, &pHandle->sourceId, aPgId); @@ -2127,6 +2151,12 @@ static int32_t sortBlocksToExtSource(SSortHandle* pHandle, SArray* aBlk, SArray* tMergeTreeDestroy(&pTree); return code; + +_error: + tMergeTreeDestroy(&pTree); + cleanupMergeSup(&sup); + if (aPgId) taosArrayDestroy(aPgId); + return code; } static int32_t getRowsBlockWithinMergeLimit(const SSortHandle* pHandle, SSHashObj* mTableNumRows, SSDataBlock* pOrigBlk, @@ -2236,8 +2266,9 @@ static int32_t createBlocksMergeSortInitialSources(SSortHandle* pHandle) { while (1) { bool bExtractedBlock = false; bool bSkipBlock = false; - - SSDataBlock* pBlk = pHandle->fetchfp(pSrc->param); + SSDataBlock* pBlk = NULL; + + TAOS_CHECK_RETURN(pHandle->fetchfp(pSrc->param, &pBlk)); if (pBlk != NULL && pHandle->mergeLimit > 0) { SSDataBlock* p = NULL; code = getRowsBlockWithinMergeLimit(pHandle, mTableNumRows, pBlk, &bExtractedBlock, &bSkipBlock, &p); @@ -2390,7 +2421,8 @@ static int32_t createBlocksQuickSortInitialSources(SSortHandle* pHandle) { tsortClearOrderdSource(pHandle->pOrderedSource, NULL, NULL); while (1) { - SSDataBlock* pBlock = pHandle->fetchfp(source->param); + SSDataBlock* pBlock = NULL; + TAOS_CHECK_RETURN(pHandle->fetchfp(source->param, &pBlock)); if (pBlock == NULL) { break; } @@ -2500,7 +2532,10 @@ static int32_t tsortOpenForBufMergeSort(SSortHandle* pHandle) { int32_t numOfSources = taosArrayGetSize(pHandle->pOrderedSource); if (pHandle->pBuf != NULL) { - ASSERT(numOfSources <= getNumOfInMemBufPages(pHandle->pBuf)); + if (numOfSources > getNumOfInMemBufPages(pHandle->pBuf)) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } } if (numOfSources == 0) { @@ -2594,7 +2629,10 @@ static int32_t tsortBufMergeSortNextTuple(SSortHandle* pHandle, STupleHandle** p index = tMergeTreeGetChosenIndex(pHandle->pMergeTree); pSource = pHandle->cmpParam.pSources[index]; - ASSERT(pSource->src.pBlock != NULL); + if (pSource->src.pBlock == NULL) { + qError("sort failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } pHandle->tupleHandle.rowIndex = pSource->src.rowIndex; pHandle->tupleHandle.pBlock = pSource->src.pBlock; @@ -2701,7 +2739,8 @@ static int32_t tsortOpenForPQSort(SSortHandle* pHandle) { while (1) { // fetch data - SSDataBlock* pBlock = pHandle->fetchfp(source->param); + SSDataBlock* pBlock = NULL; + TAOS_CHECK_RETURN(pHandle->fetchfp(source->param, &pBlock)); if (NULL == pBlock) { break; } @@ -2828,7 +2867,9 @@ static int32_t tsortSingleTableMergeNextTuple(SSortHandle* pHandle, STupleHandle } SSortSource* source = *pSource; - SSDataBlock* pBlock = pHandle->fetchfp(source->param); + SSDataBlock* pBlock = NULL; + TAOS_CHECK_RETURN(pHandle->fetchfp(source->param, &pBlock)); + if (!pBlock || pBlock->info.rows == 0) { setCurrentSourceDone(source, pHandle); pHandle->tupleHandle.pBlock = NULL; diff --git a/source/libs/function/src/tudf.c b/source/libs/function/src/tudf.c index d5ecf09cee..44b6ba4722 100644 --- a/source/libs/function/src/tudf.c +++ b/source/libs/function/src/tudf.c @@ -1168,6 +1168,10 @@ void cleanupExpiredUdfs() { void cleanupNotExpiredUdfs() { SArray *udfStubs = taosArrayInit(16, sizeof(SUdfcFuncStub)); + if (udfStubs == NULL) { + fnError("cleanupNotExpiredUdfs: failed to init array"); + return; + } int32_t i = 0; while (i < taosArrayGetSize(gUdfcProxy.udfStubs)) { SUdfcFuncStub *stub = taosArrayGet(gUdfcProxy.udfStubs, i); diff --git a/source/libs/geometry/src/geosWrapper.c b/source/libs/geometry/src/geosWrapper.c index dde34edc91..b349ab5cbe 100644 --- a/source/libs/geometry/src/geosWrapper.c +++ b/source/libs/geometry/src/geosWrapper.c @@ -331,18 +331,26 @@ int32_t doGeosRelation(const GEOSGeometry *geom1, const GEOSPreparedGeometry *pr if (!preparedGeom1) { if (!swapped) { - ASSERT(relationFn); + if (!relationFn) { + return TSDB_CODE_FUNC_FUNTION_PARA_VALUE; + } *res = relationFn(geosCtx->handle, geom1, geom2); } else { - ASSERT(swappedRelationFn); + if (!swappedRelationFn) { + return TSDB_CODE_FUNC_FUNTION_PARA_VALUE; + } *res = swappedRelationFn(geosCtx->handle, geom1, geom2); } } else { if (!swapped) { - ASSERT(preparedRelationFn); + if (!preparedRelationFn) { + return TSDB_CODE_FUNC_FUNTION_PARA_VALUE; + } *res = preparedRelationFn(geosCtx->handle, preparedGeom1, geom2); } else { - ASSERT(swappedPreparedRelationFn); + if (!swappedPreparedRelationFn) { + return TSDB_CODE_FUNC_FUNTION_PARA_VALUE; + } *res = swappedPreparedRelationFn(geosCtx->handle, preparedGeom1, geom2); } } @@ -391,7 +399,9 @@ int32_t readGeometry(const unsigned char *input, GEOSGeometry **outputGeom, const GEOSPreparedGeometry **outputPreparedGeom) { SGeosContext *geosCtx = getThreadLocalGeosCtx(); - ASSERT(outputGeom); // it is not allowed if outputGeom is NULL + if (!outputGeom) { + return TSDB_CODE_FUNC_FUNTION_PARA_VALUE; + } *outputGeom = NULL; if (outputPreparedGeom) { // it means not to generate PreparedGeometry if outputPreparedGeom is NULL diff --git a/source/libs/geometry/test/geomFuncTestUtil.cpp b/source/libs/geometry/test/geomFuncTestUtil.cpp index 0918781499..25b2dd7c8d 100644 --- a/source/libs/geometry/test/geomFuncTestUtil.cpp +++ b/source/libs/geometry/test/geomFuncTestUtil.cpp @@ -66,7 +66,7 @@ void setScalarParam(SScalarParam *sclParam, int32_t type, void *valueArray, TDRo break; } default: { - ASSERT(0); + ASSERT_TRUE(false); break; } } diff --git a/source/libs/index/inc/indexTfile.h b/source/libs/index/inc/indexTfile.h index 51d3cc0e7b..115d5b3894 100644 --- a/source/libs/index/inc/indexTfile.h +++ b/source/libs/index/inc/indexTfile.h @@ -115,7 +115,7 @@ int32_t tfileWriterOpen(char* path, uint64_t suid, int64_t version, const char* void tfileWriterClose(TFileWriter* tw); int32_t tfileWriterCreate(IFileCtx* ctx, TFileHeader* header, TFileWriter** pWriter); void tfileWriterDestroy(TFileWriter* tw); -int tfileWriterPut(TFileWriter* tw, void* data, bool order); +int32_t tfileWriterPut(TFileWriter* tw, void* data, bool order); int tfileWriterFinish(TFileWriter* tw); // diff --git a/source/libs/index/src/index.c b/source/libs/index/src/index.c index b881d2cac8..e2c74f0dbb 100644 --- a/source/libs/index/src/index.c +++ b/source/libs/index/src/index.c @@ -237,8 +237,7 @@ int32_t indexPut(SIndex* index, SIndexMultiTerm* fVals, uint64_t uid) { indexDebug("w suid:%" PRIu64 ", colName:%s, colType:%d", key.suid, key.colName, key.colType); IndexCache** cache = taosHashGet(index->colObj, buf, sz); - ASSERTS(*cache != NULL, "index-cache already release"); - if (*cache == NULL) return -1; + if (*cache == NULL) return TSDB_CODE_INVALID_PTR; int ret = idxCachePut(*cache, p, uid); if (ret != 0) { @@ -295,7 +294,9 @@ void indexMultiTermQueryDestroy(SIndexMultiTermQuery* pQuery) { }; int32_t indexMultiTermQueryAdd(SIndexMultiTermQuery* pQuery, SIndexTerm* term, EIndexQueryType qType) { SIndexTermQuery q = {.qType = qType, .term = term}; - (void)taosArrayPush(pQuery->query, &q); + if (taosArrayPush(pQuery->query, &q) == NULL) { + return terrno; + } return 0; } @@ -303,6 +304,7 @@ SIndexTerm* indexTermCreate(int64_t suid, SIndexOperOnColumn oper, uint8_t colTy int32_t nColName, const char* colVal, int32_t nColVal) { SIndexTerm* tm = (SIndexTerm*)taosMemoryCalloc(1, (sizeof(SIndexTerm))); if (tm == NULL) { + terrno = TSDB_CODE_OUT_OF_MEMORY; return NULL; } @@ -311,6 +313,10 @@ SIndexTerm* indexTermCreate(int64_t suid, SIndexOperOnColumn oper, uint8_t colTy tm->colType = colType; tm->colName = (char*)taosMemoryCalloc(1, nColName + 1); + if (tm->colName == NULL) { + taosMemoryFree(tm); + return NULL; + } memcpy(tm->colName, colName, nColName); tm->nColName = nColName; @@ -326,8 +332,23 @@ SIndexTerm* indexTermCreate(int64_t suid, SIndexOperOnColumn oper, uint8_t colTy buf = strndup(emptyStr, (int32_t)strlen(emptyStr)); len = (int32_t)strlen(emptyStr); } + tm->colVal = buf; + if (tm->colVal == NULL) { + taosMemoryFree(tm->colName); + taosMemoryFree(tm); + terrno = TSDB_CODE_OUT_OF_MEMORY; + return NULL; + } + tm->nColVal = len; + if (tm->nColVal < 0) { + taosMemoryFree(tm->colName); + taosMemoryFree(tm->colVal); + taosMemoryFree(tm); + terrno = len; + return NULL; + } return tm; } @@ -677,6 +698,7 @@ static int32_t idxGenTFile(SIndex* sIdx, IndexCache* cache, SArray* batch) { code = tfileWriterOpen(sIdx->path, cache->suid, version, cache->colName, cache->type, &tw); if (code != 0) { indexError("failed to open file to write since %s", tstrerror(code)); + return code; } code = tfileWriterPut(tw, batch, true); diff --git a/source/libs/index/src/indexCache.c b/source/libs/index/src/indexCache.c index adf555edd2..734c4067c6 100644 --- a/source/libs/index/src/indexCache.c +++ b/source/libs/index/src/indexCache.c @@ -123,6 +123,7 @@ static int32_t cacheSearchCompareFunc(void* cache, SIndexTerm* term, SIdxTRslt* if (cache == NULL) { return 0; } + int32_t code = TSDB_CODE_SUCCESS; MemTable* mem = cache; IndexCache* pCache = mem->pCache; @@ -146,7 +147,12 @@ static int32_t cacheSearchCompareFunc(void* cache, SIndexTerm* term, SIdxTRslt* break; } CacheTerm* c = (CacheTerm*)SL_GET_NODE_DATA(node); + terrno = TSDB_CODE_SUCCESS; TExeCond cond = cmpFn(c->colVal, pCt->colVal, pCt->colType); + if (terrno != TSDB_CODE_SUCCESS) { + code = terrno; + goto _return; + } if (cond == MATCH) { if (c->operaType == ADD_VALUE) { INDEX_MERGE_ADD_DEL(tr->del, tr->add, c->uid) @@ -161,9 +167,11 @@ static int32_t cacheSearchCompareFunc(void* cache, SIndexTerm* term, SIdxTRslt* break; } } + +_return: taosMemoryFree(pCt); (void)tSkipListDestroyIter(iter); - return TSDB_CODE_SUCCESS; + return code; } static int32_t cacheSearchLessThan(void* cache, SIndexTerm* term, SIdxTRslt* tr, STermValueType* s) { return cacheSearchCompareFunc(cache, term, tr, s, LT); @@ -265,6 +273,7 @@ static int32_t cacheSearchCompareFunc_JSON(void* cache, SIndexTerm* term, SIdxTR } _cache_range_compare cmpFn = idxGetCompare(type); + int32_t code = TSDB_CODE_SUCCESS; MemTable* mem = cache; IndexCache* pCache = mem->pCache; @@ -308,9 +317,18 @@ static int32_t cacheSearchCompareFunc_JSON(void* cache, SIndexTerm* term, SIdxTR continue; } else { char* p = taosMemoryCalloc(1, strlen(c->colVal) + 1); + if (NULL == p) { + code = terrno; + goto _return; + } memcpy(p, c->colVal, strlen(c->colVal)); + terrno = TSDB_CODE_SUCCESS; cond = cmpFn(p + skip, term->colVal, dType); taosMemoryFree(p); + if (terrno != TSDB_CODE_SUCCESS) { + code = terrno; + goto _return; + } } } if (cond == MATCH) { @@ -327,11 +345,12 @@ static int32_t cacheSearchCompareFunc_JSON(void* cache, SIndexTerm* term, SIdxTR } } +_return: taosMemoryFree(pCt); taosMemoryFree(exBuf); (void)tSkipListDestroyIter(iter); - return TSDB_CODE_SUCCESS; + return code; } static int32_t cacheSearchRange(void* cache, SIndexTerm* term, SIdxTRslt* tr, STermValueType* s) { // impl later diff --git a/source/libs/index/src/indexComm.c b/source/libs/index/src/indexComm.c index 6337b2c556..8dc63ed105 100644 --- a/source/libs/index/src/indexComm.c +++ b/source/libs/index/src/indexComm.c @@ -84,27 +84,45 @@ __compar_fn_t idxGetCompar(int8_t type) { } static FORCE_INLINE TExeCond tCompareLessThan(void* a, void* b, int8_t type) { __compar_fn_t func = idxGetCompar(type); + if (func == NULL) { + return BREAK; + } return tCompare(func, QUERY_LESS_THAN, a, b, type); } static FORCE_INLINE TExeCond tCompareLessEqual(void* a, void* b, int8_t type) { __compar_fn_t func = idxGetCompar(type); + if (func == NULL) { + return BREAK; + } return tCompare(func, QUERY_LESS_EQUAL, a, b, type); } static FORCE_INLINE TExeCond tCompareGreaterThan(void* a, void* b, int8_t type) { __compar_fn_t func = idxGetCompar(type); + if (func == NULL) { + return BREAK; + } return tCompare(func, QUERY_GREATER_THAN, a, b, type); } static FORCE_INLINE TExeCond tCompareGreaterEqual(void* a, void* b, int8_t type) { __compar_fn_t func = idxGetCompar(type); + if (func == NULL) { + return BREAK; + } return tCompare(func, QUERY_GREATER_EQUAL, a, b, type); } static FORCE_INLINE TExeCond tCompareContains(void* a, void* b, int8_t type) { __compar_fn_t func = idxGetCompar(type); + if (func == NULL) { + return BREAK; + } return tCompare(func, QUERY_TERM, a, b, type); } static FORCE_INLINE TExeCond tCompareEqual(void* a, void* b, int8_t type) { __compar_fn_t func = idxGetCompar(type); + if (func == NULL) { + return BREAK; + } return tCompare(func, QUERY_TERM, a, b, type); } TExeCond tCompare(__compar_fn_t func, int8_t cmptype, void* a, void* b, int8_t dtype) { @@ -290,90 +308,6 @@ int idxUidCompare(const void* a, const void* b) { uint64_t r = *(uint64_t*)b; return l - r; } -#ifdef BUILD_NO_CALL -int32_t idxConvertData(void* src, int8_t type, void** dst) { - int tlen = -1; - switch (type) { - case TSDB_DATA_TYPE_TIMESTAMP: - tlen = taosEncodeFixedI64(NULL, *(int64_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedI64(dst, *(int64_t*)src); - break; - case TSDB_DATA_TYPE_BOOL: - case TSDB_DATA_TYPE_UTINYINT: - tlen = taosEncodeFixedU8(NULL, *(uint8_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedU8(dst, *(uint8_t*)src); - break; - case TSDB_DATA_TYPE_TINYINT: - tlen = taosEncodeFixedI8(NULL, *(uint8_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedI8(dst, *(uint8_t*)src); - break; - case TSDB_DATA_TYPE_SMALLINT: - tlen = taosEncodeFixedI16(NULL, *(int16_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedI16(dst, *(int16_t*)src); - break; - case TSDB_DATA_TYPE_USMALLINT: - tlen = taosEncodeFixedU16(NULL, *(uint16_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedU16(dst, *(uint16_t*)src); - break; - case TSDB_DATA_TYPE_INT: - tlen = taosEncodeFixedI32(NULL, *(int32_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedI32(dst, *(int32_t*)src); - break; - case TSDB_DATA_TYPE_FLOAT: - tlen = taosEncodeBinary(NULL, src, sizeof(float)); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeBinary(dst, src, sizeof(float)); - break; - case TSDB_DATA_TYPE_UINT: - tlen = taosEncodeFixedU32(NULL, *(uint32_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedU32(dst, *(uint32_t*)src); - break; - case TSDB_DATA_TYPE_BIGINT: - tlen = taosEncodeFixedI64(NULL, *(int64_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedI64(dst, *(int64_t*)src); - break; - case TSDB_DATA_TYPE_DOUBLE: - tlen = taosEncodeBinary(NULL, src, sizeof(double)); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeBinary(dst, src, sizeof(double)); - break; - case TSDB_DATA_TYPE_UBIGINT: - tlen = taosEncodeFixedU64(NULL, *(uint64_t*)src); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeFixedU64(dst, *(uint64_t*)src); - break; - case TSDB_DATA_TYPE_NCHAR: { - tlen = taosEncodeBinary(NULL, varDataVal(src), varDataLen(src)); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeBinary(dst, varDataVal(src), varDataLen(src)); - - break; - } - case TSDB_DATA_TYPE_VARCHAR: // TSDB_DATA_TYPE_BINARY - case TSDB_DATA_TYPE_VARBINARY: - case TSDB_DATA_TYPE_GEOMETRY: { - tlen = taosEncodeBinary(NULL, src, strlen(src)); - *dst = taosMemoryCalloc(1, tlen + 1); - tlen = taosEncodeBinary(dst, src, strlen(src)); - break; - } - default: - ASSERTS(0, "index invalid input type"); - break; - } - *dst = (char*)*dst - tlen; - // indexMayFillNumbericData(*dst, tlen); - return tlen; -} -#endif int32_t idxConvertDataToStr(void* src, int8_t type, void** dst) { if (src == NULL) { @@ -495,7 +429,7 @@ int32_t idxConvertDataToStr(void* src, int8_t type, void** dst) { break; } default: - ASSERTS(0, "index invalid input type"); + tlen = TSDB_CODE_INVALID_DATA_FMT; break; } return tlen; diff --git a/source/libs/index/src/indexFilter.c b/source/libs/index/src/indexFilter.c index c362932da3..03474c0143 100644 --- a/source/libs/index/src/indexFilter.c +++ b/source/libs/index/src/indexFilter.c @@ -432,22 +432,37 @@ typedef int (*FilterFunc)(void *a, void *b, int16_t dtype); static FORCE_INLINE int sifGreaterThan(void *a, void *b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return tDoCompare(func, QUERY_GREATER_THAN, a, b); } static FORCE_INLINE int sifGreaterEqual(void *a, void *b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return tDoCompare(func, QUERY_GREATER_EQUAL, a, b); } static FORCE_INLINE int sifLessEqual(void *a, void *b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return tDoCompare(func, QUERY_LESS_EQUAL, a, b); } static FORCE_INLINE int sifLessThan(void *a, void *b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } return (int)tDoCompare(func, QUERY_LESS_THAN, a, b); } static FORCE_INLINE int sifEqual(void *a, void *b, int16_t dtype) { __compar_fn_t func = getComparFunc(dtype, 0); + if (func == NULL) { + return -1; + } //__compar_fn_t func = idxGetCompar(dtype); return (int)tDoCompare(func, QUERY_TERM, a, b); } @@ -646,7 +661,16 @@ static int32_t sifDoIndex(SIFParam *left, SIFParam *right, int8_t operType, SIFP } SIndexMultiTermQuery *mtm = indexMultiTermQueryCreate(MUST); - (void)indexMultiTermQueryAdd(mtm, tm, qtype); + if (mtm == NULL) { + indexTermDestroy(tm); + return TSDB_CODE_OUT_OF_MEMORY; + } + + if ((ret = indexMultiTermQueryAdd(mtm, tm, qtype)) != 0) { + indexMultiTermQueryDestroy(mtm); + return ret; + } + ret = indexJsonSearch(arg->ivtIdx, mtm, output->result); indexMultiTermQueryDestroy(mtm); } else { @@ -1104,7 +1128,6 @@ int32_t doFilterTag(SNode *pFilterNode, SIndexMetaArg *metaArg, SArray *result, *status = st; } - if (taosArrayAddAll(result, param.result) == NULL) { sifFreeParam(¶m); return TSDB_CODE_OUT_OF_MEMORY; diff --git a/source/libs/index/src/indexFst.c b/source/libs/index/src/indexFst.c index 54d28f4784..3edd61bd08 100644 --- a/source/libs/index/src/indexFst.c +++ b/source/libs/index/src/indexFst.c @@ -98,7 +98,7 @@ void fstUnFinishedNodesAddSuffix(FstUnFinishedNodes* nodes, FstSlice bs, Output } int32_t sz = taosArrayGetSize(nodes->stack) - 1; FstBuilderNodeUnfinished* un = taosArrayGet(nodes->stack, sz); - ASSERTS(un->last == NULL, "index-fst meet unexpected node"); + if (un->last != NULL) return; // FstLastTransition *trn = taosMemoryMalloc(sizeof(FstLastTransition)); @@ -318,8 +318,6 @@ void fstStateCompileForAnyTrans(IdxFstFile* w, CompiledAddr addr, FstBuilderNode // set_comm_input void fstStateSetCommInput(FstState* s, uint8_t inp) { - ASSERT(s->state == OneTransNext || s->state == OneTrans); - uint8_t val; COMMON_INDEX(inp, 0b111111, val); s->val = (s->val & fstStateDict[s->state].val) | val; @@ -327,7 +325,6 @@ void fstStateSetCommInput(FstState* s, uint8_t inp) { // comm_input uint8_t fstStateCommInput(FstState* s, bool* null) { - ASSERT(s->state == OneTransNext || s->state == OneTrans); uint8_t v = s->val & 0b00111111; if (v == 0) { *null = true; @@ -340,7 +337,6 @@ uint8_t fstStateCommInput(FstState* s, bool* null) { // input_len uint64_t fstStateInputLen(FstState* s) { - ASSERT(s->state == OneTransNext || s->state == OneTrans); bool null = false; (void)fstStateCommInput(s, &null); return null ? 1 : 0; @@ -348,11 +344,9 @@ uint64_t fstStateInputLen(FstState* s) { // end_addr uint64_t fstStateEndAddrForOneTransNext(FstState* s, FstSlice* data) { - ASSERT(s->state == OneTransNext); return FST_SLICE_LEN(data) - 1 - fstStateInputLen(s); } uint64_t fstStateEndAddrForOneTrans(FstState* s, FstSlice* data, PackSizes sizes) { - ASSERT(s->state == OneTrans); return FST_SLICE_LEN(data) - 1 - fstStateInputLen(s) - 1 // pack size - FST_GET_TRANSITION_PACK_SIZE(sizes) - FST_GET_OUTPUT_PACK_SIZE(sizes); } @@ -366,7 +360,6 @@ uint64_t fstStateEndAddrForAnyTrans(FstState* state, uint64_t version, FstSlice* } // input uint8_t fstStateInput(FstState* s, FstNode* node) { - ASSERT(s->state == OneTransNext || s->state == OneTrans); FstSlice* slice = &node->data; bool null = false; uint8_t inp = fstStateCommInput(s, &null); @@ -374,7 +367,6 @@ uint8_t fstStateInput(FstState* s, FstNode* node) { return null == false ? inp : data[node->start - 1]; } uint8_t fstStateInputForAnyTrans(FstState* s, FstNode* node, uint64_t i) { - ASSERT(s->state == AnyTrans); FstSlice* slice = &node->data; uint64_t at = node->start - fstStateNtransLen(s) - 1 // pack size @@ -386,7 +378,6 @@ uint8_t fstStateInputForAnyTrans(FstState* s, FstNode* node, uint64_t i) { // trans_addr CompiledAddr fstStateTransAddr(FstState* s, FstNode* node) { - ASSERT(s->state == OneTransNext || s->state == OneTrans); FstSlice* slice = &node->data; if (s->state == OneTransNext) { return (CompiledAddr)(node->end) - 1; @@ -402,8 +393,6 @@ CompiledAddr fstStateTransAddr(FstState* s, FstNode* node) { } } CompiledAddr fstStateTransAddrForAnyTrans(FstState* s, FstNode* node, uint64_t i) { - ASSERT(s->state == AnyTrans); - FstSlice* slice = &node->data; uint8_t tSizes = FST_GET_TRANSITION_PACK_SIZE(node->sizes); uint64_t at = node->start - fstStateNtransLen(s) - 1 - fstStateTransIndexSize(s, node->version, node->nTrans) - @@ -414,7 +403,6 @@ CompiledAddr fstStateTransAddrForAnyTrans(FstState* s, FstNode* node, uint64_t i // sizes PackSizes fstStateSizes(FstState* s, FstSlice* slice) { - ASSERT(s->state == OneTrans || s->state == AnyTrans); uint64_t i; if (s->state == OneTrans) { i = FST_SLICE_LEN(slice) - 1 - fstStateInputLen(s) - 1; @@ -427,8 +415,6 @@ PackSizes fstStateSizes(FstState* s, FstSlice* slice) { } // Output Output fstStateOutput(FstState* s, FstNode* node) { - ASSERT(s->state == OneTrans); - uint8_t oSizes = FST_GET_OUTPUT_PACK_SIZE(node->sizes); if (oSizes == 0) { return 0; @@ -441,8 +427,6 @@ Output fstStateOutput(FstState* s, FstNode* node) { return unpackUint64(data + i, oSizes); } Output fstStateOutputForAnyTrans(FstState* s, FstNode* node, uint64_t i) { - ASSERT(s->state == AnyTrans); - uint8_t oSizes = FST_GET_OUTPUT_PACK_SIZE(node->sizes); if (oSizes == 0) { return 0; @@ -458,19 +442,14 @@ Output fstStateOutputForAnyTrans(FstState* s, FstNode* node, uint64_t i) { // anyTrans specify function void fstStateSetFinalState(FstState* s, bool yes) { - ASSERT(s->state == AnyTrans); if (yes) { s->val |= 0b01000000; } return; } -bool fstStateIsFinalState(FstState* s) { - ASSERT(s->state == AnyTrans); - return (s->val & 0b01000000) == 0b01000000; -} +bool fstStateIsFinalState(FstState* s) { return (s->val & 0b01000000) == 0b01000000; } void fstStateSetStateNtrans(FstState* s, uint8_t n) { - ASSERT(s->state == AnyTrans); if (n <= 0b00111111) { s->val = (s->val & 0b11000000) | n; } @@ -478,7 +457,6 @@ void fstStateSetStateNtrans(FstState* s, uint8_t n) { } // state_ntrans uint8_t fstStateStateNtrans(FstState* s, bool* null) { - ASSERT(s->state == AnyTrans); *null = false; uint8_t n = s->val & 0b00111111; @@ -488,16 +466,13 @@ uint8_t fstStateStateNtrans(FstState* s, bool* null) { return n; } uint64_t fstStateTotalTransSize(FstState* s, uint64_t version, PackSizes sizes, uint64_t nTrans) { - ASSERT(s->state == AnyTrans); uint64_t idxSize = fstStateTransIndexSize(s, version, nTrans); return nTrans + (nTrans * FST_GET_TRANSITION_PACK_SIZE(sizes)) + idxSize; } uint64_t fstStateTransIndexSize(FstState* s, uint64_t version, uint64_t nTrans) { - ASSERT(s->state == AnyTrans); return (version >= 2 && nTrans > TRANS_INDEX_THRESHOLD) ? 256 : 0; } uint64_t fstStateNtransLen(FstState* s) { - ASSERT(s->state == AnyTrans); bool null = false; (void)fstStateStateNtrans(s, &null); return null == true ? 1 : 0; @@ -526,7 +501,6 @@ Output fstStateFinalOutput(FstState* s, uint64_t version, FstSlice* slice, PackS return unpackUint64(data + at, (uint8_t)oSizes); } uint64_t fstStateFindInput(FstState* s, FstNode* node, uint8_t b, bool* null) { - ASSERT(s->state == AnyTrans); FstSlice* slice = &node->data; if (node->version >= 2 && node->nTrans > TRANS_INDEX_THRESHOLD) { uint64_t at = node->start - fstStateNtransLen(s) - 1 // pack size @@ -672,17 +646,14 @@ bool fstNodeGetTransitionAddrAt(FstNode* node, uint64_t i, CompiledAddr* res) { bool s = true; FstState* st = &node->state; if (st->state == OneTransNext) { - ASSERT(i == 0); (void)fstStateTransAddr(st, node); } else if (st->state == OneTrans) { - ASSERT(i == 0); (void)fstStateTransAddr(st, node); } else if (st->state == AnyTrans) { (void)fstStateTransAddrForAnyTrans(st, node, i); } else if (FST_STATE_EMPTY_FINAL(node)) { s = false; } else { - ASSERT(0); } return s; } @@ -718,7 +689,6 @@ bool fstNodeFindInput(FstNode* node, uint8_t b, uint64_t* res) { bool fstNodeCompile(FstNode* node, void* w, CompiledAddr lastAddr, CompiledAddr addr, FstBuilderNode* builderNode) { int32_t sz = taosArrayGetSize(builderNode->trans); - ASSERT(sz < 256); if (sz == 0 && builderNode->isFinal && builderNode->finalOutput == 0) { return true; } else if (sz != 1 || builderNode->isFinal) { @@ -800,7 +770,6 @@ void fstBuilderInsertOutput(FstBuilder* b, FstSlice bs, Output in) { uint64_t prefixLen = fstUnFinishedNodesFindCommPrefixAndSetOutput(b->unfinished, bs, in, &out); if (prefixLen == FST_SLICE_LEN(s)) { - ASSERT(out == 0); return; } @@ -844,7 +813,6 @@ void fstBuilderCompileFrom(FstBuilder* b, uint64_t istate) { addr = fstBuilderCompile(b, bn); fstBuilderNodeDestroy(bn); - ASSERT(addr != NONE_ADDRESS); } fstUnFinishedNodesTopLastFreeze(b->unfinished, addr); return; diff --git a/source/libs/index/src/indexFstFile.c b/source/libs/index/src/indexFstFile.c index 69fee8d29d..5c6a39d13c 100644 --- a/source/libs/index/src/indexFstFile.c +++ b/source/libs/index/src/indexFstFile.c @@ -104,7 +104,10 @@ static int idxFileCtxDoReadFrom(IFileCtx* ctx, uint8_t* buf, int len, int32_t of do { char key[1024] = {0}; - ASSERT(strlen(ctx->file.buf) + 1 + 64 < sizeof(key)); + if (strlen(ctx->file.buf) + 1 + 64 >= sizeof(key)) { + return TSDB_CODE_INDEX_INVALID_FILE; + } + idxGenLRUKey(key, ctx->file.buf, blkId); LRUHandle* h = taosLRUCacheLookup(ctx->lru, key, strlen(key)); @@ -118,8 +121,10 @@ static int idxFileCtxDoReadFrom(IFileCtx* ctx, uint8_t* buf, int len, int32_t of if (left < kBlockSize) { nread = TMIN(left, len); int32_t bytes = taosPReadFile(ctx->file.pFile, buf + total, nread, offset); - ASSERTS(bytes == nread, "index read incomplete data"); - if (bytes != nread) break; + if (bytes != nread) { + total = TSDB_CODE_INDEX_INVALID_FILE; + break; + } total += bytes; return total; @@ -129,7 +134,6 @@ static int idxFileCtxDoReadFrom(IFileCtx* ctx, uint8_t* buf, int len, int32_t of SDataBlock* blk = taosMemoryCalloc(1, cacheMemSize); blk->blockId = blkId; blk->nread = taosPReadFile(ctx->file.pFile, blk->buf, kBlockSize, blkId * kBlockSize); - ASSERTS(blk->nread <= kBlockSize, "index read incomplete data"); if (blk->nread < kBlockSize && blk->nread < len) { taosMemoryFree(blk); break; @@ -288,7 +292,6 @@ int32_t idxFileWrite(IdxFstFile* write, uint8_t* buf, uint32_t len) { // update checksum IFileCtx* ctx = write->wrt; int nWrite = ctx->write(ctx, buf, len); - ASSERTS(nWrite == len, "index write incomplete data"); if (nWrite != len) { code = TAOS_SYSTEM_ERROR(errno); return code; diff --git a/source/libs/index/src/indexFstNode.c b/source/libs/index/src/indexFstNode.c index b934cb0536..7f1a86bc55 100644 --- a/source/libs/index/src/indexFstNode.c +++ b/source/libs/index/src/indexFstNode.c @@ -97,9 +97,7 @@ void fstBuilderNodeCloneFrom(FstBuilderNode* dst, FstBuilderNode* src) { // bool fstBuilderNodeCompileTo(FstBuilderNode *b, IdxFile *wrt, CompiledAddr lastAddr, CompiledAddr // startAddr) { - // size_t sz = taosArrayGetSize(b->trans); -// assert(sz < 256); // if (FST_BUILDER_NODE_IS_FINAL(b) // && FST_BUILDER_NODE_TRANS_ISEMPTY(b) // && FST_BUILDER_NODE_FINALOUTPUT_ISZERO(b)) { diff --git a/source/libs/index/src/indexFstRegister.c b/source/libs/index/src/indexFstRegister.c index e27ab939b3..a8c4365a44 100644 --- a/source/libs/index/src/indexFstRegister.c +++ b/source/libs/index/src/indexFstRegister.c @@ -57,8 +57,9 @@ static void fstRegistryCellPromote(SArray* arr, uint32_t start, uint32_t end) { if (start >= sz && end >= sz) { return; } - ASSERTS(start >= end, "index-fst start lower than end"); - if (start < end) return; + if (start < end) { + return; + } int32_t s = (int32_t)start; int32_t e = (int32_t)end; diff --git a/source/libs/index/src/indexTfile.c b/source/libs/index/src/indexTfile.c index 99c7b6bc76..8c06296732 100644 --- a/source/libs/index/src/indexTfile.c +++ b/source/libs/index/src/indexTfile.c @@ -315,7 +315,7 @@ static int32_t tfSearchRegex(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { } static int32_t tfSearchCompareFunc(void* reader, SIndexTerm* tem, SIdxTRslt* tr, RangeType type) { - int ret = 0; + int32_t code = TSDB_CODE_SUCCESS; char* p = tem->colVal; int skip = 0; _cache_range_compare cmpFn = idxGetCompare(type); @@ -335,7 +335,13 @@ static int32_t tfSearchCompareFunc(void* reader, SIndexTerm* tem, SIdxTRslt* tr, FstSlice* s = &rt->data; char* ch = (char*)fstSliceData(s, NULL); + terrno = TSDB_CODE_SUCCESS; TExeCond cond = cmpFn(ch, p, tem->colType); + if (TSDB_CODE_SUCCESS != terrno) { + swsResultDestroy(rt); + code = terrno; + goto _return; + } if (MATCH == cond) { (void)tfileReaderLoadTableIds((TFileReader*)reader, rt->out.out, tr->total); } else if (CONTINUE == cond) { @@ -345,10 +351,11 @@ static int32_t tfSearchCompareFunc(void* reader, SIndexTerm* tem, SIdxTRslt* tr, } swsResultDestroy(rt); } +_return: stmStDestroy(st); stmBuilderDestroy(sb); taosArrayDestroy(offsets); - return TSDB_CODE_SUCCESS; + return code; } static int32_t tfSearchLessThan(void* reader, SIndexTerm* tem, SIdxTRslt* tr) { return tfSearchCompareFunc(reader, tem, tr, LT); @@ -427,8 +434,8 @@ static int32_t tfSearchRange_JSON(void* reader, SIndexTerm* tem, SIdxTRslt* tr) } static int32_t tfSearchCompareFunc_JSON(void* reader, SIndexTerm* tem, SIdxTRslt* tr, RangeType ctype) { - int ret = 0; - int skip = 0; + int32_t code = TSDB_CODE_SUCCESS; + int skip = 0; char* p = NULL; if (ctype == CONTAINS) { @@ -469,9 +476,20 @@ static int32_t tfSearchCompareFunc_JSON(void* reader, SIndexTerm* tem, SIdxTRslt continue; } char* tBuf = taosMemoryCalloc(1, sz + 1); + if (NULL == tBuf) { + swsResultDestroy(rt); + code = terrno; + goto _return; + } memcpy(tBuf, ch, sz); + terrno = TSDB_CODE_SUCCESS; cond = cmpFn(tBuf + skip, tem->colVal, IDX_TYPE_GET_TYPE(tem->colType)); taosMemoryFree(tBuf); + if (TSDB_CODE_SUCCESS != terrno) { + swsResultDestroy(rt); + code = terrno; + goto _return; + } } if (MATCH == cond) { (void)tfileReaderLoadTableIds((TFileReader*)reader, rt->out.out, tr->total); @@ -482,12 +500,14 @@ static int32_t tfSearchCompareFunc_JSON(void* reader, SIndexTerm* tem, SIdxTRslt } swsResultDestroy(rt); } + +_return: stmStDestroy(st); stmBuilderDestroy(sb); taosArrayDestroy(offsets); taosMemoryFree(p); - return TSDB_CODE_SUCCESS; + return code; } int tfileReaderSearch(TFileReader* reader, SIndexTermQuery* query, SIdxTRslt* tr) { int ret = 0; @@ -523,8 +543,13 @@ int32_t tfileWriterOpen(char* path, uint64_t suid, int64_t version, const char* memcpy(tfh.colName, colName, strlen(colName)); } - return tfileWriterCreate(wcx, &tfh, pWriter); + code = tfileWriterCreate(wcx, &tfh, pWriter); + if (code != 0) { + idxFileCtxDestroy(wcx, true); + } + return code; } + int32_t tfileReaderOpen(SIndex* idx, uint64_t suid, int64_t version, const char* colName, TFileReader** pReader) { int32_t code = 0; char fullname[256] = {0}; @@ -552,13 +577,17 @@ int32_t tfileWriterCreate(IFileCtx* ctx, TFileHeader* header, TFileWriter** pWri } tw->ctx = ctx; tw->header = *header; - (void)tfileWriteHeader(tw); + code = tfileWriteHeader(tw); + if (code != 0) { + taosMemoryFree(tw); + return code; + } *pWriter = tw; return code; } -int tfileWriterPut(TFileWriter* tw, void* data, bool order) { +int32_t tfileWriterPut(TFileWriter* tw, void* data, bool order) { // sort by coltype and write to tindex if (order == false) { __compar_fn_t fn; @@ -571,6 +600,9 @@ int tfileWriterPut(TFileWriter* tw, void* data, bool order) { } else { fn = getComparFunc(colType, 0); } + if (fn == NULL) { + return terrno; + } (void)taosArraySortPWithExt((SArray*)(data), tfileValueCompare, &fn); } @@ -867,7 +899,8 @@ static int tfileWriteFstOffset(TFileWriter* tw, int32_t offset) { return 0; } static int tfileWriteHeader(TFileWriter* writer) { - char buf[TFILE_HEADER_NO_FST] = {0}; + int32_t code = 0; + char buf[TFILE_HEADER_NO_FST] = {0}; TFileHeader* header = &writer->header; memcpy(buf, (char*)header, sizeof(buf)); @@ -875,7 +908,9 @@ static int tfileWriteHeader(TFileWriter* writer) { indexInfo("tfile pre write header size: %d", writer->ctx->size(writer->ctx)); int nwrite = writer->ctx->write(writer->ctx, buf, sizeof(buf)); if (sizeof(buf) != nwrite) { - return -1; + code = TAOS_SYSTEM_ERROR(errno); + indexError("failed to write header, code:0x%x, filename: %s", code, writer->ctx->file.buf); + return code; } indexInfo("tfile after write header size: %d", writer->ctx->size(writer->ctx)); @@ -898,17 +933,20 @@ static int tfileWriteFooter(TFileWriter* write) { char buf[sizeof(FILE_MAGIC_NUMBER) + 1] = {0}; void* pBuf = (void*)buf; (void)taosEncodeFixedU64((void**)(void*)&pBuf, FILE_MAGIC_NUMBER); - int nwrite = write->ctx->write(write->ctx, buf, (int32_t)strlen(buf)); + int nwrite = write->ctx->write(write->ctx, (uint8_t*)buf, (int32_t)strlen(buf)); indexInfo("tfile write footer size: %d", write->ctx->size(write->ctx)); - ASSERTS(nwrite == sizeof(FILE_MAGIC_NUMBER), "index write incomplete data"); - return nwrite; + if (nwrite != sizeof(FILE_MAGIC_NUMBER)) { + return TAOS_SYSTEM_ERROR(errno); + } else { + return nwrite; + } } static int tfileReaderLoadHeader(TFileReader* reader) { // TODO simple tfile header later char buf[TFILE_HEADER_SIZE] = {0}; - int64_t nread = reader->ctx->readFrom(reader->ctx, buf, sizeof(buf), 0); + int64_t nread = reader->ctx->readFrom(reader->ctx, (uint8_t*)buf, sizeof(buf), 0); if (nread == -1) { indexError("actual Read: %d, to read: %d, code:0x%x, filename: %s", (int)(nread), (int)sizeof(buf), errno, @@ -932,14 +970,13 @@ static int tfileReaderLoadFst(TFileReader* reader) { } int64_t ts = taosGetTimestampUs(); - int32_t nread = ctx->readFrom(ctx, buf, fstSize, reader->header.fstOffset); + int32_t nread = ctx->readFrom(ctx, (uint8_t*)buf, fstSize, reader->header.fstOffset); int64_t cost = taosGetTimestampUs() - ts; indexInfo("nread = %d, and fst offset=%d, fst size: %d, filename: %s, file size: %d, time cost: %" PRId64 "us", nread, reader->header.fstOffset, fstSize, ctx->file.buf, size, cost); // we assuse fst size less than FST_MAX_SIZE - ASSERTS(nread > 0 && nread <= fstSize, "index read incomplete fst"); if (nread <= 0 || nread > fstSize) { - return -1; + return TSDB_CODE_INDEX_INVALID_FILE; } FstSlice st = fstSliceCreate((uint8_t*)buf, nread); @@ -954,8 +991,10 @@ static int tfileReaderLoadTableIds(TFileReader* reader, int32_t offset, SArray* IFileCtx* ctx = reader->ctx; // add block cache char block[4096] = {0}; - int32_t nread = ctx->readFrom(ctx, block, sizeof(block), offset); - ASSERT(nread >= sizeof(uint32_t)); + int32_t nread = ctx->readFrom(ctx, (uint8_t*)block, sizeof(block), offset); + if (nread < sizeof(uint32_t)) { + return TSDB_CODE_INDEX_INVALID_FILE; + } char* p = block; int32_t nid = *(int32_t*)p; @@ -972,7 +1011,7 @@ static int tfileReaderLoadTableIds(TFileReader* reader, int32_t offset, SArray* memset(block, 0, sizeof(block)); offset += sizeof(block); - nread = ctx->readFrom(ctx, block, sizeof(block), offset); + nread = ctx->readFrom(ctx, (uint8_t*)block, sizeof(block), offset); memcpy(buf + left, block, sizeof(uint64_t) - left); (void)taosArrayPush(result, (uint64_t*)buf); @@ -991,13 +1030,14 @@ static int tfileReaderVerify(TFileReader* reader) { int size = ctx->size(ctx); if (size < sizeof(tMagicNumber) || size <= sizeof(reader->header)) { - return -1; - } else if (ctx->readFrom(ctx, buf, sizeof(tMagicNumber), size - sizeof(tMagicNumber)) != sizeof(tMagicNumber)) { - return -1; + return TSDB_CODE_INDEX_INVALID_FILE; + } else if (ctx->readFrom(ctx, (uint8_t*)buf, sizeof(tMagicNumber), size - sizeof(tMagicNumber)) != + sizeof(tMagicNumber)) { + return TSDB_CODE_INDEX_INVALID_FILE; } (void)taosDecodeFixedU64(buf, &tMagicNumber); - return tMagicNumber == FILE_MAGIC_NUMBER ? 0 : -1; + return tMagicNumber == FILE_MAGIC_NUMBER ? 0 : TSDB_CODE_INDEX_INVALID_FILE; } void tfileReaderRef(TFileReader* rd) { @@ -1072,12 +1112,12 @@ static int tfileParseFileName(const char* filename, uint64_t* suid, char* col, i } // tfile name suid-colId-version.tindex static void tfileGenFileName(char* filename, uint64_t suid, const char* col, int64_t version) { - sprintf(filename, "%" PRIu64 "-%s-%" PRId64 ".tindex", suid, col, version); + (void)sprintf(filename, "%" PRIu64 "-%s-%" PRId64 ".tindex", suid, col, version); return; } static void FORCE_INLINE tfileGenFileFullName(char* fullname, const char* path, uint64_t suid, const char* col, int64_t version) { char filename[128] = {0}; tfileGenFileName(filename, suid, col, version); - sprintf(fullname, "%s/%s", path, filename); + (void)sprintf(fullname, "%s/%s", path, filename); } diff --git a/source/libs/monitor/src/monMain.c b/source/libs/monitor/src/monMain.c index 9669eac508..4abc8958df 100644 --- a/source/libs/monitor/src/monMain.c +++ b/source/libs/monitor/src/monMain.c @@ -21,11 +21,25 @@ #include "thttp.h" #include "ttime.h" +#define VNODE_METRIC_SQL_COUNT "taosd_sql_req:count" + +#define VNODE_METRIC_TAG_NAME_SQL_TYPE "sql_type" +#define VNODE_METRIC_TAG_NAME_CLUSTER_ID "cluster_id" +#define VNODE_METRIC_TAG_NAME_DNODE_ID "dnode_id" +#define VNODE_METRIC_TAG_NAME_DNODE_EP "dnode_ep" +#define VNODE_METRIC_TAG_NAME_VGROUP_ID "vgroup_id" +#define VNODE_METRIC_TAG_NAME_USERNAME "username" +#define VNODE_METRIC_TAG_NAME_RESULT "result" + +// #define VNODE_METRIC_TAG_VALUE_INSERT "insert" +// #define VNODE_METRIC_TAG_VALUE_DELETE "delete" + SMonitor tsMonitor = {0}; char *tsMonUri = "/report"; char *tsMonFwUri = "/general-metric"; char *tsMonSlowLogUri = "/slow-sql-detail-batch"; char *tsMonFwBasicUri = "/taosd-cluster-basic"; +taos_counter_t *tsInsertCounter = NULL; void monRecordLog(int64_t ts, ELogLevel level, const char *content) { (void)taosThreadMutexLock(&tsMonitor.lock); @@ -117,6 +131,28 @@ int32_t monInit(const SMonCfg *pCfg) { return 0; } +void monInitVnode() { + if (tsEnableMonitor && tsMonitorFqdn[0] != 0 && tsMonitorPort != 0 && tsInsertCounter == NULL) { + taos_counter_t *counter = NULL; + int32_t label_count = 7; + const char *sample_labels[] = {VNODE_METRIC_TAG_NAME_SQL_TYPE, VNODE_METRIC_TAG_NAME_CLUSTER_ID, + VNODE_METRIC_TAG_NAME_DNODE_ID, VNODE_METRIC_TAG_NAME_DNODE_EP, + VNODE_METRIC_TAG_NAME_VGROUP_ID, VNODE_METRIC_TAG_NAME_USERNAME, + VNODE_METRIC_TAG_NAME_RESULT}; + counter = taos_counter_new(VNODE_METRIC_SQL_COUNT, "counter for insert sql", label_count, sample_labels); + uDebug("new metric:%p", counter); + if (taos_collector_registry_register_metric(counter) == 1) { + (void)taos_counter_destroy(counter); + uError("failed to register metric:%p", counter); + } else { + tsInsertCounter = counter; + uInfo("succeed to set inserted row metric:%p", tsInsertCounter); + } + } else { + uError("failed to set insert counter, already set"); + } +} + void monCleanup() { tsLogFp = NULL; taosArrayDestroy(tsMonitor.logs); diff --git a/source/libs/monitorfw/inc/taos_assert.h b/source/libs/monitorfw/inc/taos_assert.h index 9d2732acc5..b73e6572b6 100644 --- a/source/libs/monitorfw/inc/taos_assert.h +++ b/source/libs/monitorfw/inc/taos_assert.h @@ -16,13 +16,15 @@ #include -#ifndef TAOS_ASSERT_H -#define TAOS_ASSERT_H +#ifndef TAOS_TEST_H +#define TAOS_TEST_H -#ifdef TAOS_ASSERT_ENABLE -#define TAOS_ASSERT(i) assert(i); +#ifdef TAOS_TEST_PARA_ENABLE #else -#define TAOS_ASSERT(i) +#define TAOS_TEST_PARA(i) \ + if (!(i)) return 1; +#define TAOS_TEST_PARA_NULL(i) \ + if (!(i)) return NULL; #endif // TAOS_TEST -#endif // TAOS_ASSERT_H +#endif // TAOS_TEST_H diff --git a/source/libs/monitorfw/src/taos_collector.c b/source/libs/monitorfw/src/taos_collector.c index 01e37b6d76..6ba34725b7 100644 --- a/source/libs/monitorfw/src/taos_collector.c +++ b/source/libs/monitorfw/src/taos_collector.c @@ -59,7 +59,7 @@ taos_collector_t *taos_collector_new(const char *name) { } int taos_collector_destroy(taos_collector_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; int r = 0; @@ -97,14 +97,14 @@ void taos_collector_free_generic(void *gen) { } int taos_collector_set_collect_fn(taos_collector_t *self, taos_collect_fn *fn) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; self->collect_fn = fn; return 0; } int taos_collector_add_metric(taos_collector_t *self, taos_metric_t *metric) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (taos_map_get(self->metrics, metric->name) != NULL) { TAOS_LOG("metric already found in collector"); @@ -114,13 +114,13 @@ int taos_collector_add_metric(taos_collector_t *self, taos_metric_t *metric) { } int taos_collector_remove_metric(taos_collector_t *self, const char* key){ - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; return taos_map_delete(self->metrics, key); } taos_metric_t* taos_collector_get_metric(taos_collector_t *self, char *metric_name){ - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); if (self == NULL) return NULL; return taos_map_get(self->metrics, metric_name); } \ No newline at end of file diff --git a/source/libs/monitorfw/src/taos_collector_registry.c b/source/libs/monitorfw/src/taos_collector_registry.c index 88f56549e5..f028dabce8 100644 --- a/source/libs/monitorfw/src/taos_collector_registry.c +++ b/source/libs/monitorfw/src/taos_collector_registry.c @@ -113,7 +113,7 @@ int taos_collector_registry_destroy(taos_collector_registry_t *self) { } int taos_collector_registry_register_metric(taos_metric_t *metric) { - TAOS_ASSERT(metric != NULL); + TAOS_TEST_PARA(metric != NULL); taos_collector_t *default_collector = (taos_collector_t *)taos_map_get(TAOS_COLLECTOR_REGISTRY_DEFAULT->collectors, "default"); @@ -126,7 +126,7 @@ int taos_collector_registry_register_metric(taos_metric_t *metric) { } int taos_collector_registry_deregister_metric(const char *key) { - TAOS_ASSERT(metric != NULL); + TAOS_TEST_PARA(key != NULL); taos_collector_t *default_collector = (taos_collector_t *)taos_map_get(TAOS_COLLECTOR_REGISTRY_DEFAULT->collectors, "default"); @@ -139,7 +139,7 @@ int taos_collector_registry_deregister_metric(const char *key) { } taos_metric_t *taos_collector_registry_get_metric(char* metric_name){ - TAOS_ASSERT(metric != NULL); + TAOS_TEST_PARA_NULL(metric_name != NULL); taos_collector_t *default_collector = (taos_collector_t *)taos_map_get(TAOS_COLLECTOR_REGISTRY_DEFAULT->collectors, "default"); @@ -161,7 +161,7 @@ taos_metric_t *taos_collector_registry_must_register_metric(taos_metric_t *metri } int taos_collector_registry_register_collector(taos_collector_registry_t *self, taos_collector_t *collector) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; diff --git a/source/libs/monitorfw/src/taos_counter.c b/source/libs/monitorfw/src/taos_counter.c index b4c867c66f..5588286988 100644 --- a/source/libs/monitorfw/src/taos_counter.c +++ b/source/libs/monitorfw/src/taos_counter.c @@ -33,7 +33,7 @@ taos_counter_t *taos_counter_new(const char *name, const char *help, size_t labe } int taos_counter_destroy(taos_counter_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; int r = 0; r = taos_metric_destroy(self); @@ -42,7 +42,7 @@ int taos_counter_destroy(taos_counter_t *self) { } int taos_counter_inc(taos_counter_t *self, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_COUNTER) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); @@ -54,7 +54,7 @@ int taos_counter_inc(taos_counter_t *self, const char **label_values) { } int taos_counter_add(taos_counter_t *self, double r_value, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_COUNTER) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); diff --git a/source/libs/monitorfw/src/taos_gauge.c b/source/libs/monitorfw/src/taos_gauge.c index 1f9fa6fc9c..5b3eedeb84 100644 --- a/source/libs/monitorfw/src/taos_gauge.c +++ b/source/libs/monitorfw/src/taos_gauge.c @@ -33,7 +33,7 @@ taos_gauge_t *taos_gauge_new(const char *name, const char *help, size_t label_ke } int taos_gauge_destroy(taos_gauge_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; r = taos_metric_destroy(self); self = NULL; @@ -41,7 +41,7 @@ int taos_gauge_destroy(taos_gauge_t *self) { } /* int taos_gauge_inc(taos_gauge_t *self, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_GAUGE) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); @@ -53,7 +53,7 @@ int taos_gauge_inc(taos_gauge_t *self, const char **label_values) { } int taos_gauge_dec(taos_gauge_t *self, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_GAUGE) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); @@ -65,7 +65,7 @@ int taos_gauge_dec(taos_gauge_t *self, const char **label_values) { } int taos_gauge_add(taos_gauge_t *self, double r_value, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_GAUGE) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); @@ -77,7 +77,7 @@ int taos_gauge_add(taos_gauge_t *self, double r_value, const char **label_values } int taos_gauge_sub(taos_gauge_t *self, double r_value, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_GAUGE) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); @@ -89,7 +89,7 @@ int taos_gauge_sub(taos_gauge_t *self, double r_value, const char **label_values } */ int taos_gauge_set(taos_gauge_t *self, double r_value, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (self->type != TAOS_GAUGE) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); diff --git a/source/libs/monitorfw/src/taos_linked_list.c b/source/libs/monitorfw/src/taos_linked_list.c index 7c46e89ab0..eab1d3f5db 100644 --- a/source/libs/monitorfw/src/taos_linked_list.c +++ b/source/libs/monitorfw/src/taos_linked_list.c @@ -34,8 +34,7 @@ taos_linked_list_t *taos_linked_list_new(void) { } int taos_linked_list_purge(taos_linked_list_t *self) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); taos_linked_list_node_t *node = self->head; while (node != NULL) { taos_linked_list_node_t *next = node->next; @@ -57,7 +56,7 @@ int taos_linked_list_purge(taos_linked_list_t *self) { } int taos_linked_list_destroy(taos_linked_list_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; int ret = 0; @@ -69,7 +68,7 @@ int taos_linked_list_destroy(taos_linked_list_t *self) { } void *taos_linked_list_first(taos_linked_list_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); if (self->head) { return self->head->item; } else { @@ -78,7 +77,7 @@ void *taos_linked_list_first(taos_linked_list_t *self) { } void *taos_linked_list_last(taos_linked_list_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); if (self->tail) { return self->tail->item; } else { @@ -87,8 +86,7 @@ void *taos_linked_list_last(taos_linked_list_t *self) { } int taos_linked_list_append(taos_linked_list_t *self, void *item) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); taos_linked_list_node_t *node = (taos_linked_list_node_t *)taos_malloc(sizeof(taos_linked_list_node_t)); node->item = item; @@ -104,8 +102,7 @@ int taos_linked_list_append(taos_linked_list_t *self, void *item) { } int taos_linked_list_push(taos_linked_list_t *self, void *item) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); taos_linked_list_node_t *node = (taos_linked_list_node_t *)taos_malloc(sizeof(taos_linked_list_node_t)); node->item = item; @@ -120,7 +117,7 @@ int taos_linked_list_push(taos_linked_list_t *self, void *item) { /* void *taos_linked_list_pop(taos_linked_list_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return NULL; taos_linked_list_node_t *node = self->head; void *item = NULL; @@ -146,8 +143,7 @@ void *taos_linked_list_pop(taos_linked_list_t *self) { */ int taos_linked_list_remove(taos_linked_list_t *self, void *item) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); taos_linked_list_node_t *node; taos_linked_list_node_t *prev_node = NULL; #ifdef TAOS_LOG_ENABLE @@ -225,8 +221,7 @@ int taos_linked_list_remove(taos_linked_list_t *self, void *item) { } taos_linked_list_compare_t taos_linked_list_compare(taos_linked_list_t *self, void *item_a, void *item_b) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); if (self->compare_fn) { return (*self->compare_fn)(item_a, item_b); } else { @@ -235,20 +230,18 @@ taos_linked_list_compare_t taos_linked_list_compare(taos_linked_list_t *self, vo } size_t taos_linked_list_size(taos_linked_list_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); return self->size; } int taos_linked_list_set_free_fn(taos_linked_list_t *self, taos_linked_list_free_item_fn free_fn) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); self->free_fn = free_fn; return 0; } int taos_linked_list_set_compare_fn(taos_linked_list_t *self, taos_linked_list_compare_item_fn compare_fn) { - TAOS_ASSERT(self != NULL); - if (self == NULL) return 1; + TAOS_TEST_PARA(self != NULL); self->compare_fn = compare_fn; return 0; } diff --git a/source/libs/monitorfw/src/taos_map.c b/source/libs/monitorfw/src/taos_map.c index 64d5475af2..ba325ac517 100644 --- a/source/libs/monitorfw/src/taos_map.c +++ b/source/libs/monitorfw/src/taos_map.c @@ -46,7 +46,7 @@ taos_map_node_t *taos_map_node_new(const char *key, void *value, taos_map_node_f } int taos_map_node_destroy(taos_map_node_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; taos_free((void *)self->key); self->key = NULL; @@ -120,7 +120,7 @@ taos_map_t *taos_map_new() { } int taos_map_destroy(taos_map_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; int ret = 0; @@ -199,7 +199,7 @@ static void *taos_map_get_internal(const char *key, size_t *size, size_t *max_si } void *taos_map_get(taos_map_t *self, const char *key) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); int r = 0; r = pthread_rwlock_wrlock(self->rwlock); if (r) { @@ -217,7 +217,7 @@ void *taos_map_get(taos_map_t *self, const char *key) { } void *taos_map_get_withoutlock(taos_map_t *self, const char *key) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); int r = 0; void *payload = taos_map_get_internal(key, &self->size, &self->max_size, self->keys, self->addrs, self->free_value_fn); @@ -255,7 +255,7 @@ static int taos_map_set_internal(const char *key, void *value, size_t *size, siz } int taos_map_ensure_space(taos_map_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; if (self->size <= self->max_size / 2) { @@ -329,7 +329,7 @@ int taos_map_ensure_space(taos_map_t *self) { } int taos_map_set(taos_map_t *self, const char *key, void *value) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; r = pthread_rwlock_wrlock(self->rwlock); if (r) { @@ -394,7 +394,7 @@ static int taos_map_delete_internal(const char *key, size_t *size, size_t *max_s } int taos_map_delete(taos_map_t *self, const char *key) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; int ret = 0; r = pthread_rwlock_wrlock(self->rwlock); @@ -413,12 +413,12 @@ int taos_map_delete(taos_map_t *self, const char *key) { } int taos_map_set_free_value_fn(taos_map_t *self, taos_map_node_free_value_fn free_value_fn) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); self->free_value_fn = free_value_fn; return 0; } size_t taos_map_size(taos_map_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); return self->size; } diff --git a/source/libs/monitorfw/src/taos_metric.c b/source/libs/monitorfw/src/taos_metric.c index f31cef79b0..7d35dc2421 100644 --- a/source/libs/monitorfw/src/taos_metric.c +++ b/source/libs/monitorfw/src/taos_metric.c @@ -89,7 +89,7 @@ taos_metric_t *taos_metric_new(taos_metric_type_t metric_type, const char *name, } int taos_metric_destroy(taos_metric_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; int r = 0; @@ -144,7 +144,7 @@ void taos_metric_free_generic(void *item) { } taos_metric_sample_t *taos_metric_sample_from_labels(taos_metric_t *self, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); int r = 0; r = pthread_rwlock_wrlock(self->rwlock); if (r) { diff --git a/source/libs/monitorfw/src/taos_metric_formatter.c b/source/libs/monitorfw/src/taos_metric_formatter.c index e57c809815..3d3b456aad 100644 --- a/source/libs/monitorfw/src/taos_metric_formatter.c +++ b/source/libs/monitorfw/src/taos_metric_formatter.c @@ -46,7 +46,7 @@ taos_metric_formatter_t *taos_metric_formatter_new() { } int taos_metric_formatter_destroy(taos_metric_formatter_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; int r = 0; @@ -66,7 +66,7 @@ int taos_metric_formatter_destroy(taos_metric_formatter_t *self) { } /* int taos_metric_formatter_load_help(taos_metric_formatter_t *self, const char *name, const char *help) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -87,7 +87,7 @@ int taos_metric_formatter_load_help(taos_metric_formatter_t *self, const char *n } int taos_metric_formatter_load_type(taos_metric_formatter_t *self, const char *name, taos_metric_type_t metric_type) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -109,7 +109,7 @@ int taos_metric_formatter_load_type(taos_metric_formatter_t *self, const char *n */ int taos_metric_formatter_load_l_value(taos_metric_formatter_t *self, const char *name, const char *suffix, size_t label_count, const char **label_keys, const char **label_values) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -158,9 +158,9 @@ int taos_metric_formatter_load_l_value(taos_metric_formatter_t *self, const char return 0; } /* -int taos_metric_formatter_load_sample(taos_metric_formatter_t *self, taos_metric_sample_t *sample, +int taos_metric_formatter_load_sample(taos_metric_formatter_t *self, taos_metric_sample_t *sample, char *ts, char *format) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -188,12 +188,12 @@ int taos_metric_formatter_load_sample(taos_metric_formatter_t *self, taos_metric } */ int taos_metric_formatter_clear(taos_metric_formatter_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); return taos_string_builder_clear(self->string_builder); } char *taos_metric_formatter_dump(taos_metric_formatter_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); int r = 0; if (self == NULL) return NULL; char *data = taos_string_builder_dump(self->string_builder); @@ -207,7 +207,7 @@ char *taos_metric_formatter_dump(taos_metric_formatter_t *self) { } /* int taos_metric_formatter_load_metric(taos_metric_formatter_t *self, taos_metric_t *metric, char *ts, char *format) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -234,7 +234,7 @@ int taos_metric_formatter_load_metric(taos_metric_formatter_t *self, taos_metric } int taos_metric_formatter_load_metrics(taos_metric_formatter_t *self, taos_map_t *collectors, char *ts, char *format) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; for (taos_linked_list_node_t *current_node = collectors->keys->head; current_node != NULL; current_node = current_node->next) { diff --git a/source/libs/monitorfw/src/taos_metric_formatter_custom.c b/source/libs/monitorfw/src/taos_metric_formatter_custom.c index 0287641311..1cf8560d21 100644 --- a/source/libs/monitorfw/src/taos_metric_formatter_custom.c +++ b/source/libs/monitorfw/src/taos_metric_formatter_custom.c @@ -29,7 +29,7 @@ int taos_metric_formatter_load_sample_new(taos_metric_formatter_t *self, taos_metric_sample_t *sample, char *ts, char *format, char *metricName, int32_t metric_type, SJson *arrayMetricGroups) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -129,7 +129,7 @@ int taos_metric_formatter_load_sample_new(taos_metric_formatter_t *self, taos_me int taos_metric_formatter_load_metric_new(taos_metric_formatter_t *self, taos_metric_t *metric, char *ts, char *format, SJson* tableArray) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; int r = 0; @@ -195,7 +195,7 @@ int taos_metric_formatter_load_metric_new(taos_metric_formatter_t *self, taos_me int taos_metric_formatter_load_metrics_new(taos_metric_formatter_t *self, taos_map_t *collectors, char *ts, char *format, SJson* tableArray) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; for (taos_linked_list_node_t *current_node = collectors->keys->head; current_node != NULL; diff --git a/source/libs/monitorfw/src/taos_metric_sample.c b/source/libs/monitorfw/src/taos_metric_sample.c index ccddefa17e..e45671bd79 100644 --- a/source/libs/monitorfw/src/taos_metric_sample.c +++ b/source/libs/monitorfw/src/taos_metric_sample.c @@ -43,7 +43,7 @@ taos_metric_sample_t *taos_metric_sample_new(taos_metric_type_t type, const char } int taos_metric_sample_destroy(taos_metric_sample_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; taos_free((void *)self->l_value); self->l_value = NULL; @@ -67,7 +67,7 @@ void taos_metric_sample_free_generic(void *gen) { } int taos_metric_sample_add(taos_metric_sample_t *self, double r_value) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (r_value < 0) { return 1; } @@ -94,14 +94,14 @@ int taos_metric_sample_add(taos_metric_sample_t *self, double r_value) { /* int taos_metric_sample_sub(taos_metric_sample_t *self, double r_value) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self->type != TAOS_GAUGE) { TAOS_LOG(TAOS_METRIC_INCORRECT_TYPE); return 1; } #ifdef C11_ATOMIC - ///_Atomic/ + ///_Atomic/ double old = atomic_load(&self->r_value); for (;;) { _Atomic double new = ATOMIC_VAR_INIT(old - r_value); diff --git a/source/libs/monitorfw/src/taos_string_builder.c b/source/libs/monitorfw/src/taos_string_builder.c index 208f3f6d51..5180a72d5c 100644 --- a/source/libs/monitorfw/src/taos_string_builder.c +++ b/source/libs/monitorfw/src/taos_string_builder.c @@ -52,7 +52,7 @@ taos_string_builder_t *taos_string_builder_new(void) { } int taos_string_builder_init(taos_string_builder_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; self->str = (char *)taos_malloc(self->init_size); *self->str = '\0'; @@ -62,7 +62,7 @@ int taos_string_builder_init(taos_string_builder_t *self) { } int taos_string_builder_destroy(taos_string_builder_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 0; taos_free(self->str); self->str = NULL; @@ -78,7 +78,7 @@ int taos_string_builder_destroy(taos_string_builder_t *self) { * is called in methods that need to add one or more characters to the underlying string. */ static int taos_string_builder_ensure_space(taos_string_builder_t *self, size_t add_len) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (add_len == 0 || self->allocated >= self->len + add_len + 1) return 0; while (self->allocated < self->len + add_len + 1) self->allocated <<= 1; @@ -87,7 +87,7 @@ static int taos_string_builder_ensure_space(taos_string_builder_t *self, size_t } int taos_string_builder_add_str(taos_string_builder_t *self, const char *str) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; if (self == NULL) return 1; @@ -104,7 +104,7 @@ int taos_string_builder_add_str(taos_string_builder_t *self, const char *str) { } int taos_string_builder_add_char(taos_string_builder_t *self, char c) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); int r = 0; if (self == NULL) return 1; @@ -118,7 +118,7 @@ int taos_string_builder_add_char(taos_string_builder_t *self, char c) { } int taos_string_builder_truncate(taos_string_builder_t *self, size_t len) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); if (self == NULL) return 1; if (len >= self->len) return 0; @@ -128,19 +128,19 @@ int taos_string_builder_truncate(taos_string_builder_t *self, size_t len) { } int taos_string_builder_clear(taos_string_builder_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); taos_free(self->str); self->str = NULL; return taos_string_builder_init(self); } size_t taos_string_builder_len(taos_string_builder_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA(self != NULL); return self->len; } char *taos_string_builder_dump(taos_string_builder_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); // +1 to accommodate \0 char *out = (char *)taos_malloc((self->len + 1) * sizeof(char)); memcpy(out, self->str, self->len + 1); @@ -148,6 +148,6 @@ char *taos_string_builder_dump(taos_string_builder_t *self) { } char *taos_string_builder_str(taos_string_builder_t *self) { - TAOS_ASSERT(self != NULL); + TAOS_TEST_PARA_NULL(self != NULL); return self->str; } diff --git a/source/libs/parser/src/parAstCreater.c b/source/libs/parser/src/parAstCreater.c index 2bf82ba98f..20081b7535 100644 --- a/source/libs/parser/src/parAstCreater.c +++ b/source/libs/parser/src/parAstCreater.c @@ -1905,10 +1905,11 @@ SNode* setTableOption(SAstCreateContext* pCxt, SNode* pOptions, ETableOptionType case TABLE_OPTION_TTL: { int64_t ttl = taosStr2Int64(((SToken*)pVal)->z, NULL, 10); if (ttl > INT32_MAX) { - ttl = INT32_MAX; + pCxt->errCode = TSDB_CODE_TSC_VALUE_OUT_OF_RANGE; + } else { + // ttl can not be smaller than 0, because there is a limitation in sql.y (TTL NK_INTEGER) + ((STableOptions*)pOptions)->ttl = ttl; } - // ttl can not be smaller than 0, because there is a limitation in sql.y (TTL NK_INTEGER) - ((STableOptions*)pOptions)->ttl = ttl; break; } case TABLE_OPTION_SMA: @@ -2516,7 +2517,7 @@ SNode* addCreateUserStmtWhiteList(SAstCreateContext* pCxt, SNode* pCreateUserStm return pCreateUserStmt; } -SNode* createCreateUserStmt(SAstCreateContext* pCxt, SToken* pUserName, const SToken* pPassword, int8_t sysinfo, +SNode* createCreateUserStmt(SAstCreateContext* pCxt, SToken* pUserName, const SToken* pPassword, int8_t sysinfo, int8_t createDb, int8_t is_import) { CHECK_PARSER_STATUS(pCxt); char password[TSDB_USET_PASSWORD_LEN + 3] = {0}; diff --git a/source/libs/parser/src/parInsertUtil.c b/source/libs/parser/src/parInsertUtil.c index 3eb69afa40..280a5c9e71 100644 --- a/source/libs/parser/src/parInsertUtil.c +++ b/source/libs/parser/src/parInsertUtil.c @@ -745,7 +745,7 @@ int32_t insMergeTableDataCxt(SHashObj* pTableHash, SArray** pVgDataBlocks, bool taosArraySort(pTableCxt->pData->aCol, insColDataComp); - tColDataSortMerge(pTableCxt->pData->aCol); + code = tColDataSortMerge(pTableCxt->pData->aCol); } else { // skip the table has no data to insert // eg: import a csv without valid data @@ -883,6 +883,10 @@ static bool findFileds(SSchema* pSchema, TAOS_FIELD* fields, int numFields) { int rawBlockBindData(SQuery* query, STableMeta* pTableMeta, void* data, SVCreateTbReq** pCreateTb, TAOS_FIELD* tFields, int numFields, bool needChangeLength, char* errstr, int32_t errstrLen) { + if(data == NULL) { + uError("rawBlockBindData, data is NULL"); + return TSDB_CODE_APP_ERROR; + } void* tmp = taosHashGet(((SVnodeModifyOpStmt*)(query->pRoot))->pTableBlockHashObj, &pTableMeta->uid, sizeof(pTableMeta->uid)); STableDataCxt* pTableCxt = NULL; diff --git a/source/libs/parser/src/parTranslater.c b/source/libs/parser/src/parTranslater.c index 05cceb656e..7e84ef8482 100755 --- a/source/libs/parser/src/parTranslater.c +++ b/source/libs/parser/src/parTranslater.c @@ -4014,7 +4014,6 @@ static int32_t setTableTsmas(STranslateContext* pCxt, SName* pName, SRealTableNo if (TSDB_CODE_SUCCESS == code) code = catalogGetCachedTableHashVgroup(pCxt->pParseCxt->pCatalog, &tsmaTargetTbName, &vgInfo, &exists); if (TSDB_CODE_SUCCESS == code) { - ASSERT(exists); if (!pRealTable->tsmaTargetTbVgInfo) { pRealTable->tsmaTargetTbVgInfo = taosArrayInit(pRealTable->pTsmas->size, POINTER_BYTES); if (!pRealTable->tsmaTargetTbVgInfo) { @@ -7320,7 +7319,10 @@ static int32_t checkDbRetentionsOption(STranslateContext* pCxt, SNodeList* pRete SValueNode* pFreq = (SValueNode*)nodesListGetNode(((SNodeListNode*)pRetention)->pNodeList, 0); SValueNode* pKeep = (SValueNode*)nodesListGetNode(((SNodeListNode*)pRetention)->pNodeList, 1); - ASSERTS(IS_DURATION_VAL(pFreq->flag) && IS_DURATION_VAL(pKeep->flag), "Retentions freq/keep should have unit"); + if (!IS_DURATION_VAL(pFreq->flag) || !IS_DURATION_VAL(pKeep->flag)) { + return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_DB_OPTION, + "Retentions freq/keep should have unit"); + } // check unit if (IS_DURATION_VAL(pFreq->flag) && TIME_UNIT_SECOND != pFreq->unit && TIME_UNIT_MINUTE != pFreq->unit && @@ -9427,7 +9429,9 @@ static int32_t buildCreateTagIndexReq(STranslateContext* pCxt, SCreateIndexStmt* (void)tNameGetFullDbName(&name, pReq->dbFName); SNode* pNode = NULL; - ASSERT(LIST_LENGTH(pStmt->pCols) == 1); + if (LIST_LENGTH(pStmt->pCols) != 1) { + return TSDB_CODE_PAR_INVALID_TAGS_NUM; + } FOREACH(pNode, pStmt->pCols) { SColumnNode* p = (SColumnNode*)pNode; memcpy(pReq->colName, p->colName, sizeof(p->colName)); @@ -11068,7 +11072,10 @@ static int32_t createStreamReqVersionInfo(SSDataBlock* pBlock, SArray** pArray, for (int32_t i = 0; i < pBlock->info.rows; ++i) { SVgroupVer v = {.vgId = *(int32_t*)colDataGetData(pCol1, i), .ver = *(int64_t*)colDataGetData(pCol2, i)}; - (void)taosArrayPush(*pArray, &v); + if((taosArrayPush(*pArray, &v)) == NULL) { + taosArrayDestroy(*pArray); + return terrno; + } } } else { int32_t precision = (pInterval->interval > 0) ? pInterval->precision : TSDB_TIME_PRECISION_MILLI; diff --git a/source/libs/parser/src/parUtil.c b/source/libs/parser/src/parUtil.c index 6a36ea7b85..250e9385b4 100644 --- a/source/libs/parser/src/parUtil.c +++ b/source/libs/parser/src/parUtil.c @@ -1281,8 +1281,10 @@ int32_t getTsmaFromCache(SParseMetaCache* pMetaCache, const SName* pTsmaName, ST } STableTSMAInfoRsp* pTsmaRsp = NULL; code = getMetaDataFromHash(tsmaFName, strlen(tsmaFName), pMetaCache->pTSMAs, (void**)&pTsmaRsp); - if (TSDB_CODE_SUCCESS == code && pTsmaRsp) { - ASSERT(pTsmaRsp->pTsmas->size == 1); + if (TSDB_CODE_SUCCESS == code) { + if (!pTsmaRsp || pTsmaRsp->pTsmas->size != 1) { + return TSDB_CODE_PAR_INTERNAL_ERROR; + } *pTsma = taosArrayGetP(pTsmaRsp->pTsmas, 0); } else if (code == TSDB_CODE_PAR_INTERNAL_ERROR){ code = TSDB_CODE_MND_SMA_NOT_EXIST; diff --git a/source/libs/planner/src/planOptimizer.c b/source/libs/planner/src/planOptimizer.c index c7ce9aed76..d48871fd70 100644 --- a/source/libs/planner/src/planOptimizer.c +++ b/source/libs/planner/src/planOptimizer.c @@ -6493,6 +6493,7 @@ static int32_t partitionColsOpt(SOptimizeContext* pCxt, SLogicSubplan* pLogicSub pSort->calcGroupId = true; code = replaceLogicNode(pLogicSubplan, (SLogicNode*)pNode, (SLogicNode*)pSort); if (code == TSDB_CODE_SUCCESS) { + nodesDestroyNode((SNode*)pNode); pCxt->optimized = true; } else { nodesDestroyNode((SNode*)pSort); @@ -6641,7 +6642,6 @@ static int32_t fillTSMAOptCtx(STSMAOptCtx* pTsmaOptCtx, SScanLogicNode* pScan) { pTsmaOptCtx->pAggFuncs = pWindow->pFuncs; pTsmaOptCtx->ppParentTsmaSubplans = &pWindow->pTsmaSubplans; } else { - ASSERT(nodeType(pTsmaOptCtx->pParent) == QUERY_NODE_LOGIC_PLAN_AGG); SAggLogicNode* pAgg = (SAggLogicNode*)pTsmaOptCtx->pParent; pTsmaOptCtx->pAggFuncs = pAgg->pAggFuncs; pTsmaOptCtx->ppParentTsmaSubplans = &pAgg->pTsmaSubplans; @@ -6939,8 +6939,9 @@ static int32_t tsmaOptSplitWindows(STSMAOptCtx* pTsmaOptCtx, const STimeWindow* } int32_t tsmaOptCreateTsmaScanCols(const STSMAOptUsefulTsma* pTsma, const SNodeList* pAggFuncs, SNodeList** ppList) { - ASSERT(pTsma->pTsma); - ASSERT(pTsma->pTsmaScanCols); + if (!pTsma->pTsma || !pTsma->pTsmaScanCols) { + return TSDB_CODE_PLAN_INTERNAL_ERROR; + } int32_t code; SNode* pNode; SNodeList* pScanCols = NULL; @@ -6996,8 +6997,7 @@ static int32_t tsmaOptRewriteTag(const STSMAOptCtx* pTsmaOptCtx, const STSMAOptU break; } } - ASSERT(found); - return 0; + return found ? TSDB_CODE_SUCCESS : TSDB_CODE_PLAN_INTERNAL_ERROR; } static int32_t tsmaOptRewriteTbname(const STSMAOptCtx* pTsmaOptCtx, SNode** pTbNameNode, @@ -7116,7 +7116,6 @@ static int32_t tsmaOptRewriteScan(STSMAOptCtx* pTsmaOptCtx, SScanLogicNode* pNew SColumnNode* pPkTsCol = NULL; FOREACH(pNode, pNewScan->pScanCols) { SColumnNode* pCol = (SColumnNode*)pNode; - ASSERT(pTsma->pTsmaScanCols); if (pCol->colId == PRIMARYKEY_TIMESTAMP_COL_ID) { pPkTsCol = NULL; code = nodesCloneNode((SNode*)pCol, (SNode**)&pPkTsCol); @@ -7302,7 +7301,6 @@ static int32_t tsmaOptGeneratePlan(STSMAOptCtx* pTsmaOptCtx) { for (int32_t j = 0; j < pTsmaOptCtx->pScan->pTsmas->size; ++j) { if (taosArrayGetP(pTsmaOptCtx->pScan->pTsmas, j) == pTsma->pTsma) { const STsmaTargetTbInfo* ptbInfo = taosArrayGet(pTsmaOptCtx->pScan->pTsmaTargetTbInfo, j); - ASSERT(ptbInfo->uid != 0); strcpy(pTsma->targetTbName, ptbInfo->tableName); pTsma->targetTbUid = ptbInfo->uid; } diff --git a/source/libs/planner/src/planner.c b/source/libs/planner/src/planner.c index 1b53819c7b..43159bce20 100644 --- a/source/libs/planner/src/planner.c +++ b/source/libs/planner/src/planner.c @@ -20,7 +20,7 @@ #include "tglobal.h" static int32_t debugPrintNode(SNode* pNode) { - char* pStr = NULL; + char* pStr = NULL; int32_t code = nodesNodeToString(pNode, false, &pStr, NULL); if (TSDB_CODE_SUCCESS == code) { (void)printf("%s\n", pStr); @@ -37,7 +37,7 @@ static int32_t dumpQueryPlan(SQueryPlan* pPlan) { char* pStr = NULL; code = nodesNodeToString((SNode*)pPlan, false, &pStr, NULL); if (TSDB_CODE_SUCCESS == code) { - planDebugL("QID:0x%" PRIx64 " Query Plan, JsonPlan: %s", pPlan->queryId, pStr); + planDebugL("qid:0x%" PRIx64 " Query Plan, JsonPlan: %s", pPlan->queryId, pStr); taosMemoryFree(pStr); } return code; @@ -117,14 +117,13 @@ static int32_t setSubplanExecutionNode(SPhysiNode* pNode, int32_t groupId, SDown return code; } -int32_t qContinuePlanPostQuery(void *pPostPlan) { - //TODO +int32_t qContinuePlanPostQuery(void* pPostPlan) { + // TODO return TSDB_CODE_SUCCESS; } - int32_t qSetSubplanExecutionNode(SSubplan* subplan, int32_t groupId, SDownstreamSourceNode* pSource) { - planDebug("QID:0x%" PRIx64 " set subplan execution node, groupId:%d", subplan->id.queryId, groupId); + planDebug("qid:0x%" PRIx64 " set subplan execution node, groupId:%d", subplan->id.queryId, groupId); return setSubplanExecutionNode(subplan->pNode, groupId, pSource); } @@ -144,7 +143,7 @@ static void clearSubplanExecutionNode(SPhysiNode* pNode) { } void qClearSubplanExecutionNode(SSubplan* pSubplan) { - planDebug("QID:0x%" PRIx64 " clear subplan execution node, groupId:%d", pSubplan->id.queryId, pSubplan->id.groupId); + planDebug("qid:0x%" PRIx64 " clear subplan execution node, groupId:%d", pSubplan->id.queryId, pSubplan->id.groupId); clearSubplanExecutionNode(pSubplan->pNode); } diff --git a/source/libs/qworker/inc/qwInt.h b/source/libs/qworker/inc/qwInt.h index 93bbc44a25..7a902bdd66 100644 --- a/source/libs/qworker/inc/qwInt.h +++ b/source/libs/qworker/inc/qwInt.h @@ -142,10 +142,10 @@ typedef struct SQWTaskCtx { int8_t events[QW_EVENT_MAX]; - SArray *explainRes; - void *taskHandle; - void *sinkHandle; - SArray *tbInfo; // STbVerInfo + SArray *explainRes; + void *taskHandle; + void *sinkHandle; + SArray *tbInfo; // STbVerInfo } SQWTaskCtx; typedef struct SQWSchStatus { @@ -201,7 +201,7 @@ typedef struct SQWorker { SQWStat stat; int32_t *destroyed; - int8_t nodeStopped; + int8_t nodeStopped; } SQWorker; typedef struct SQWorkerMgmt { @@ -212,7 +212,8 @@ typedef struct SQWorkerMgmt { int32_t paramIdx; } SQWorkerMgmt; -#define QW_CTX_NOT_EXISTS_ERR_CODE(mgmt) (atomic_load_8(&(mgmt)->nodeStopped) ? TSDB_CODE_VND_STOPPED : TSDB_CODE_QRY_TASK_CTX_NOT_EXIST) +#define QW_CTX_NOT_EXISTS_ERR_CODE(mgmt) \ + (atomic_load_8(&(mgmt)->nodeStopped) ? TSDB_CODE_VND_STOPPED : TSDB_CODE_QRY_TASK_CTX_NOT_EXIST) #define QW_FPARAMS_DEF SQWorker *mgmt, uint64_t sId, uint64_t qId, uint64_t tId, int64_t rId, int32_t eId #define QW_IDS() sId, qId, tId, rId, eId @@ -229,31 +230,31 @@ typedef struct SQWorkerMgmt { #define QW_SET_EVENT_PROCESSED(ctx, event) atomic_store_8(&(ctx)->events[event], QW_EVENT_PROCESSED) #define QW_GET_PHASE(ctx) atomic_load_8(&(ctx)->phase) -#define QW_SET_PHASE(ctx, _value) \ - do { \ - switch (_value) { \ - case QW_PHASE_PRE_FETCH: \ - ctx->inFetch = 1; \ - break; \ - case QW_PHASE_POST_FETCH: \ - ctx->inFetch = 0; \ - break; \ - case QW_PHASE_PRE_QUERY: \ - case QW_PHASE_POST_QUERY: \ - case QW_PHASE_PRE_CQUERY: \ - case QW_PHASE_POST_CQUERY: \ - atomic_store_8(&(ctx)->phase, _value); \ - break; \ - default: \ - break; \ - } \ +#define QW_SET_PHASE(ctx, _value) \ + do { \ + switch (_value) { \ + case QW_PHASE_PRE_FETCH: \ + ctx->inFetch = 1; \ + break; \ + case QW_PHASE_POST_FETCH: \ + ctx->inFetch = 0; \ + break; \ + case QW_PHASE_PRE_QUERY: \ + case QW_PHASE_POST_QUERY: \ + case QW_PHASE_PRE_CQUERY: \ + case QW_PHASE_POST_CQUERY: \ + atomic_store_8(&(ctx)->phase, _value); \ + break; \ + default: \ + break; \ + } \ } while (0) #define QW_SET_RSP_CODE(ctx, code) atomic_store_32(&(ctx)->rspCode, code) #define QW_UPDATE_RSP_CODE(ctx, code) (void)atomic_val_compare_exchange_32(&(ctx)->rspCode, 0, code) -#define QW_QUERY_RUNNING(ctx) (QW_GET_PHASE(ctx) == QW_PHASE_PRE_QUERY || QW_GET_PHASE(ctx) == QW_PHASE_PRE_CQUERY) -#define QW_FETCH_RUNNING(ctx) ((ctx)->inFetch) +#define QW_QUERY_RUNNING(ctx) (QW_GET_PHASE(ctx) == QW_PHASE_PRE_QUERY || QW_GET_PHASE(ctx) == QW_PHASE_PRE_CQUERY) +#define QW_FETCH_RUNNING(ctx) ((ctx)->inFetch) #define QW_QUERY_NOT_STARTED(ctx) (QW_GET_PHASE(ctx) == -1) #define QW_SET_QTID(id, qId, tId, eId) \ @@ -309,24 +310,24 @@ typedef struct SQWorkerMgmt { #define QW_SCH_ELOG(param, ...) qError("QW:%p SID:%" PRIx64 " " param, mgmt, sId, __VA_ARGS__) #define QW_SCH_DLOG(param, ...) qDebug("QW:%p SID:%" PRIx64 " " param, mgmt, sId, __VA_ARGS__) -#define QW_TASK_ELOG(param, ...) qError("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) -#define QW_TASK_WLOG(param, ...) qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) -#define QW_TASK_DLOG(param, ...) qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) +#define QW_TASK_ELOG(param, ...) qError("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) +#define QW_TASK_WLOG(param, ...) qWarn("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) +#define QW_TASK_DLOG(param, ...) qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) #define QW_TASK_DLOGL(param, ...) \ - qDebugL("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) + qDebugL("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId, __VA_ARGS__) -#define QW_TASK_ELOG_E(param) qError("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId) -#define QW_TASK_WLOG_E(param) qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId) -#define QW_TASK_DLOG_E(param) qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId) +#define QW_TASK_ELOG_E(param) qError("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId) +#define QW_TASK_WLOG_E(param) qWarn("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId) +#define QW_TASK_DLOG_E(param) qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, qId, tId, eId) #define QW_SCH_TASK_ELOG(param, ...) \ - qError("QW:%p SID:0x%" PRIx64 ",QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, mgmt, sId, qId, tId, eId, \ + qError("QW:%p SID:0x%" PRIx64 ",qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, mgmt, sId, qId, tId, eId, \ __VA_ARGS__) #define QW_SCH_TASK_WLOG(param, ...) \ - qWarn("QW:%p SID:0x%" PRIx64 ",QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, mgmt, sId, qId, tId, eId, \ + qWarn("QW:%p SID:0x%" PRIx64 ",qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, mgmt, sId, qId, tId, eId, \ __VA_ARGS__) #define QW_SCH_TASK_DLOG(param, ...) \ - qDebug("QW:%p SID:0x%" PRIx64 ",QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, mgmt, sId, qId, tId, eId, \ + qDebug("QW:%p SID:0x%" PRIx64 ",qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, mgmt, sId, qId, tId, eId, \ __VA_ARGS__) #define QW_LOCK_DEBUG(...) \ @@ -341,34 +342,58 @@ typedef struct SQWorkerMgmt { #define QW_LOCK(type, _lock) \ do { \ if (QW_READ == (type)) { \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value before read lock"); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value before read lock"); \ + break; \ + } \ QW_LOCK_DEBUG("QW RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosRLockLatch(_lock); \ QW_LOCK_DEBUG("QW RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value after read lock"); \ + if (atomic_load_32((_lock)) <= 0) { \ + qError("invalid lock value after read lock"); \ + break; \ + } \ } else { \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value before write lock"); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value before write lock"); \ + break; \ + } \ QW_LOCK_DEBUG("QW WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosWLockLatch(_lock); \ QW_LOCK_DEBUG("QW WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value after write lock"); \ + if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \ + qError("invalid lock value after write lock"); \ + break; \ + } \ } \ } while (0) #define QW_UNLOCK(type, _lock) \ do { \ if (QW_READ == (type)) { \ - ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value before read unlock"); \ + if (atomic_load_32((_lock)) <= 0) { \ + qError("invalid lock value before read unlock"); \ + break; \ + } \ QW_LOCK_DEBUG("QW RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosRUnLockLatch(_lock); \ QW_LOCK_DEBUG("QW RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after read unlock"); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value after read unlock"); \ + break; \ + } \ } else { \ - ASSERTS(atomic_load_32((_lock)) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value before write unlock"); \ + if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \ + qError("invalid lock value before write unlock"); \ + break; \ + } \ QW_LOCK_DEBUG("QW WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ taosWUnLockLatch(_lock); \ QW_LOCK_DEBUG("QW WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after write unlock"); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value after write unlock"); \ + break; \ + } \ } \ } while (0) diff --git a/source/libs/scalar/inc/filterInt.h b/source/libs/scalar/inc/filterInt.h index 4d45fb344c..9adc9ee99c 100644 --- a/source/libs/scalar/inc/filterInt.h +++ b/source/libs/scalar/inc/filterInt.h @@ -464,7 +464,7 @@ struct SFilterInfo { (colInfo).type = RANGE_TYPE_UNIT; \ (colInfo).dataType = FILTER_UNIT_DATA_TYPE(u); \ if (taosArrayPush((SArray *)((colInfo).info), &u) == NULL) { \ - FLT_ERR_RET(terrno); \ + FLT_ERR_JRET(terrno); \ } \ } while (0) #define FILTER_PUSH_VAR_HASH(colInfo, ha) \ @@ -482,7 +482,7 @@ struct SFilterInfo { do { \ *(dst) = taosMemoryMalloc(sizeof(uint32_t) * n); \ if (NULL == *(dst)) { \ - FLT_ERR_JRET(terrno); \ + FLT_ERR_JRET(terrno); \ } \ (void)memcpy(*(dst), src, sizeof(uint32_t) * n); \ } while (0) diff --git a/source/libs/scalar/src/filter.c b/source/libs/scalar/src/filter.c index 988aeba46f..cbc7ca77bb 100644 --- a/source/libs/scalar/src/filter.c +++ b/source/libs/scalar/src/filter.c @@ -521,6 +521,10 @@ int32_t filterInitRangeCtx(int32_t type, int32_t options, SFilterRangeCtx **ctx) (*ctx)->type = type; (*ctx)->options = options; (*ctx)->pCompareFunc = getComparFunc(type, 0); + if ((*ctx)->pCompareFunc == NULL) { + taosMemoryFree(*ctx); + return terrno; + } return TSDB_CODE_SUCCESS; } @@ -556,6 +560,9 @@ int32_t filterReuseRangeCtx(SFilterRangeCtx *ctx, int32_t type, int32_t options) ctx->options = options; ctx->pCompareFunc = getComparFunc(type, 0); + if (ctx->pCompareFunc == NULL) { + return terrno; + } return TSDB_CODE_SUCCESS; } @@ -1454,6 +1461,9 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan if ((!FILTER_GET_FLAG(ra->sflag, RANGE_FLG_NULL)) && (!FILTER_GET_FLAG(ra->eflag, RANGE_FLG_NULL))) { __compar_fn_t func = getComparFunc(type, 0); + if (func == NULL) { + FLT_ERR_RET(terrno); + } if (func(&ra->s, &ra->e) == 0) { void *data = taosMemoryMalloc(sizeof(int64_t)); if (data == NULL) { @@ -1565,6 +1575,9 @@ int32_t filterAddGroupUnitFromCtx(SFilterInfo *dst, SFilterInfo *src, SFilterRan if ((!FILTER_GET_FLAG(r->ra.sflag, RANGE_FLG_NULL)) && (!FILTER_GET_FLAG(r->ra.eflag, RANGE_FLG_NULL))) { __compar_fn_t func = getComparFunc(type, 0); + if (func == NULL) { + FLT_ERR_RET(terrno); + } if (func(&r->ra.s, &r->ra.e) == 0) { void *data = taosMemoryMalloc(sizeof(int64_t)); if (data == NULL) { @@ -2433,8 +2446,7 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t } gRes[gResIdx]->colInfo = taosMemoryCalloc(info->fields[FLD_TYPE_COLUMN].num, sizeof(SFilterColInfo)); if (gRes[gResIdx]->colInfo == NULL) { - filterFreeGroupCtx(gRes[gResIdx]); - FLT_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); + FLT_ERR_JRET(terrno); } colIdxi = 0; empty = false; @@ -2446,7 +2458,6 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t if (gRes[gResIdx]->colInfo[cidx].info == NULL) { gRes[gResIdx]->colInfo[cidx].info = (SArray *)taosArrayInit(4, POINTER_BYTES); if (gRes[gResIdx]->colInfo[cidx].info == NULL) { - filterFreeGroupCtx(gRes[gResIdx]); FLT_ERR_JRET(terrno); } colIdx[colIdxi++] = cidx; @@ -2461,7 +2472,11 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t } if (colIdxi > 1) { - taosSort(colIdx, colIdxi, sizeof(uint32_t), getComparFunc(TSDB_DATA_TYPE_USMALLINT, 0)); + __compar_fn_t cmpFn = getComparFunc(TSDB_DATA_TYPE_USMALLINT, 0); + if (cmpFn == NULL) { + FLT_ERR_JRET(terrno); + } + taosSort(colIdx, colIdxi, sizeof(uint32_t), cmpFn); } for (uint32_t l = 0; l < colIdxi; ++l) { @@ -2470,12 +2485,7 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t if (FILTER_NO_MERGE_DATA_TYPE(type)) { continue; } - - code = filterMergeUnits(info, gRes[gResIdx], colIdx[l], &empty); - if (TSDB_CODE_SUCCESS != code) { - filterFreeGroupCtx(gRes[gResIdx]); - SCL_ERR_JRET(code); - } + SCL_ERR_JRET(filterMergeUnits(info, gRes[gResIdx], colIdx[l], &empty)); if (empty) { break; @@ -2501,6 +2511,9 @@ int32_t filterMergeGroupUnits(SFilterInfo *info, SFilterGroupCtx **gRes, int32_t } _return: + if (code) { + filterFreeGroupCtx(gRes[gResIdx]); + } taosMemoryFreeClear(colIdx); FLT_RET(code); } diff --git a/source/libs/scalar/src/sclfunc.c b/source/libs/scalar/src/sclfunc.c index 12f0137fc5..ed55bde663 100644 --- a/source/libs/scalar/src/sclfunc.c +++ b/source/libs/scalar/src/sclfunc.c @@ -860,31 +860,27 @@ int32_t concatFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOu int32_t numOfRows = 0; bool hasNchar = (GET_PARAM_TYPE(pOutput) == TSDB_DATA_TYPE_NCHAR) ? true : false; for (int32_t i = 0; i < inputNum; ++i) { - if (pInput[i].numOfRows > numOfRows) { - numOfRows = pInput[i].numOfRows; - } + numOfRows = TMAX(pInput[i].numOfRows, numOfRows); } + int32_t outputLen = VARSTR_HEADER_SIZE; for (int32_t i = 0; i < inputNum; ++i) { + if (IS_NULL_TYPE(GET_PARAM_TYPE(&pInput[i]))) { + colDataSetNNULL(pOutputData, 0, numOfRows); + pOutput->numOfRows = numOfRows; + goto _return; + } pInputData[i] = pInput[i].columnData; int32_t factor = 1; if (hasNchar && (GET_PARAM_TYPE(&pInput[i]) == TSDB_DATA_TYPE_VARCHAR)) { factor = TSDB_NCHAR_SIZE; } - - int32_t numOfNulls = getNumOfNullEntries(pInputData[i], pInput[i].numOfRows); - if (pInput[i].numOfRows == 1) { - inputLen += (pInputData[i]->varmeta.length - VARSTR_HEADER_SIZE) * factor * (numOfRows - numOfNulls); - } else { - inputLen += (pInputData[i]->varmeta.length - (numOfRows - numOfNulls) * VARSTR_HEADER_SIZE) * factor; - } + outputLen += pInputData[i]->info.bytes * factor; } - int32_t outputLen = inputLen + numOfRows * VARSTR_HEADER_SIZE; outputBuf = taosMemoryCalloc(outputLen, 1); if (NULL == outputBuf) { SCL_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } - char *output = outputBuf; for (int32_t k = 0; k < numOfRows; ++k) { bool hasNull = false; @@ -901,6 +897,7 @@ int32_t concatFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOu } VarDataLenT dataLen = 0; + char *output = outputBuf; for (int32_t i = 0; i < inputNum; ++i) { int32_t rowIdx = (pInput[i].numOfRows == 1) ? 0 : k; input[i] = colDataGetData(pInputData[i], rowIdx); @@ -908,8 +905,7 @@ int32_t concatFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOu SCL_ERR_JRET(concatCopyHelper(input[i], output, hasNchar, GET_PARAM_TYPE(&pInput[i]), &dataLen)); } varDataSetLen(output, dataLen); - SCL_ERR_JRET(colDataSetVal(pOutputData, k, output, false)); - output += varDataTLen(output); + SCL_ERR_JRET(colDataSetVal(pOutputData, k, outputBuf, false)); } pOutput->numOfRows = numOfRows; @@ -926,51 +922,44 @@ int32_t concatWsFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *p int32_t code = TSDB_CODE_SUCCESS; SColumnInfoData **pInputData = taosMemoryCalloc(inputNum, sizeof(SColumnInfoData *)); SColumnInfoData *pOutputData = pOutput->columnData; - char **input = taosMemoryCalloc(inputNum, POINTER_BYTES); char *outputBuf = NULL; if (NULL == pInputData) { SCL_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } - if (NULL == input) { - SCL_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); - } int32_t inputLen = 0; int32_t numOfRows = 0; bool hasNchar = (GET_PARAM_TYPE(pOutput) == TSDB_DATA_TYPE_NCHAR) ? true : false; - for (int32_t i = 1; i < inputNum; ++i) { - if (pInput[i].numOfRows > numOfRows) { - numOfRows = pInput[i].numOfRows; - } - } for (int32_t i = 0; i < inputNum; ++i) { + numOfRows = TMAX(pInput[i].numOfRows, numOfRows); + } + int32_t outputLen = VARSTR_HEADER_SIZE; + for (int32_t i = 0; i < inputNum; ++i) { + if (IS_NULL_TYPE(GET_PARAM_TYPE(&pInput[i]))) { + colDataSetNNULL(pOutputData, 0, numOfRows); + pOutput->numOfRows = numOfRows; + goto _return; + } pInputData[i] = pInput[i].columnData; int32_t factor = 1; if (hasNchar && (GET_PARAM_TYPE(&pInput[i]) == TSDB_DATA_TYPE_VARCHAR)) { factor = TSDB_NCHAR_SIZE; } - int32_t numOfNulls = getNumOfNullEntries(pInputData[i], pInput[i].numOfRows); if (i == 0) { // calculate required separator space - inputLen += - (pInputData[0]->varmeta.length - VARSTR_HEADER_SIZE) * (numOfRows - numOfNulls) * (inputNum - 2) * factor; - } else if (pInput[i].numOfRows == 1) { - inputLen += (pInputData[i]->varmeta.length - VARSTR_HEADER_SIZE) * (numOfRows - numOfNulls) * factor; + outputLen += pInputData[i]->info.bytes * factor * (inputNum - 2); } else { - inputLen += (pInputData[i]->varmeta.length - (numOfRows - numOfNulls) * VARSTR_HEADER_SIZE) * factor; + outputLen += pInputData[i]->info.bytes * factor; } } - int32_t outputLen = inputLen + numOfRows * VARSTR_HEADER_SIZE; outputBuf = taosMemoryCalloc(outputLen, 1); if (NULL == outputBuf) { SCL_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } - char *output = outputBuf; - for (int32_t k = 0; k < numOfRows; ++k) { if (colDataIsNull_s(pInputData[0], k) || IS_NULL_TYPE(GET_PARAM_TYPE(&pInput[0]))) { colDataSetNULL(pOutputData, k); @@ -979,6 +968,7 @@ int32_t concatWsFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *p VarDataLenT dataLen = 0; bool hasNull = false; + char *output = outputBuf; for (int32_t i = 1; i < inputNum; ++i) { if (colDataIsNull_s(pInputData[i], k) || IS_NULL_TYPE(GET_PARAM_TYPE(&pInput[i]))) { hasNull = true; @@ -986,9 +976,8 @@ int32_t concatWsFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *p } int32_t rowIdx = (pInput[i].numOfRows == 1) ? 0 : k; - input[i] = colDataGetData(pInputData[i], rowIdx); - SCL_ERR_JRET(concatCopyHelper(input[i], output, hasNchar, GET_PARAM_TYPE(&pInput[i]), &dataLen)); + SCL_ERR_JRET(concatCopyHelper(colDataGetData(pInputData[i], rowIdx), output, hasNchar, GET_PARAM_TYPE(&pInput[i]), &dataLen)); if (i < inputNum - 1) { // insert the separator @@ -1003,14 +992,12 @@ int32_t concatWsFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *p } else { varDataSetLen(output, dataLen); SCL_ERR_JRET(colDataSetVal(pOutputData, k, output, false)); - output += varDataTLen(output); } } pOutput->numOfRows = numOfRows; _return: - taosMemoryFree(input); taosMemoryFree(outputBuf); taosMemoryFree(pInputData); @@ -1557,9 +1544,15 @@ int32_t replaceFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pO needFreeFrom = true; } if (GET_PARAM_TYPE(&pInput[2]) != GET_PARAM_TYPE(&pInput[0])) { - SCL_ERR_JRET(convBetweenNcharAndVarchar(varDataVal(colDataGetData(pInputData[2], colIdx3)), &toStr, - varDataLen(colDataGetData(pInputData[2], colIdx3)), &toLen, - GET_PARAM_TYPE(&pInput[0]))); + code = convBetweenNcharAndVarchar(varDataVal(colDataGetData(pInputData[2], colIdx3)), &toStr, + varDataLen(colDataGetData(pInputData[2], colIdx3)), &toLen, + GET_PARAM_TYPE(&pInput[0])); + if (TSDB_CODE_SUCCESS != code) { + if (needFreeFrom) { + taosMemoryFree(fromStr); + } + goto _return; + } needFreeTo = true; } @@ -1673,9 +1666,9 @@ int32_t substrIdxFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam * SCL_ERR_JRET(colDataSetVal(pOutputData, k, output, false)); } - -_return: pOutput->numOfRows = numOfRows; +_return: + taosMemoryFree(outputBuf); return code; } diff --git a/source/libs/scheduler/inc/schInt.h b/source/libs/scheduler/inc/schInt.h index 5ea79c6ae9..c723360b36 100644 --- a/source/libs/scheduler/inc/schInt.h +++ b/source/libs/scheduler/inc/schInt.h @@ -410,22 +410,22 @@ extern SSchedulerMgmt schMgmt; #define SCH_SWITCH_EPSET(_addr) ((_addr)->epSet.inUse = ((_addr)->epSet.inUse + 1) % (_addr)->epSet.numOfEps) #define SCH_TASK_NUM_OF_EPS(_addr) ((_addr)->epSet.numOfEps) -#define SCH_LOG_TASK_START_TS(_task) \ - do { \ - int64_t us = taosGetTimestampUs(); \ +#define SCH_LOG_TASK_START_TS(_task) \ + do { \ + int64_t us = taosGetTimestampUs(); \ (void)taosArrayPush((_task)->profile.execTime, &us); \ - if (0 == (_task)->execId) { \ - (_task)->profile.startTs = us; \ - } \ + if (0 == (_task)->execId) { \ + (_task)->profile.startTs = us; \ + } \ } while (0) -#define SCH_LOG_TASK_WAIT_TS(_task) \ - do { \ - int64_t us = taosGetTimestampUs(); \ - int64_t* startus = (int64_t*)taosArrayGet((_task)->profile.execTime, (_task)->execId); \ - if (NULL != startus) { \ - (_task)->profile.waitTime += us - *startus; \ - } \ +#define SCH_LOG_TASK_WAIT_TS(_task) \ + do { \ + int64_t us = taosGetTimestampUs(); \ + int64_t *startus = (int64_t *)taosArrayGet((_task)->profile.execTime, (_task)->execId); \ + if (NULL != startus) { \ + (_task)->profile.waitTime += us - *startus; \ + } \ } while (0) #define SCH_LOG_TASK_END_TS(_task) \ @@ -434,28 +434,28 @@ extern SSchedulerMgmt schMgmt; int32_t idx = (_task)->execId % (_task)->maxExecTimes; \ int64_t *startts = taosArrayGet((_task)->profile.execTime, (_task)->execId); \ if (NULL != startts) { \ - *startts = us - *startts; \ + *startts = us - *startts; \ } \ (_task)->profile.endTs = us; \ } while (0) -#define SCH_JOB_ELOG(param, ...) qError("QID:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__) -#define SCH_JOB_DLOG(param, ...) qDebug("QID:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__) +#define SCH_JOB_ELOG(param, ...) qError("qid:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__) +#define SCH_JOB_DLOG(param, ...) qDebug("qid:0x%" PRIx64 " " param, pJob->queryId, __VA_ARGS__) #define SCH_TASK_ELOG(param, ...) \ - qError("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ + qError("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ __VA_ARGS__) #define SCH_TASK_DLOG(param, ...) \ - qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ + qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ __VA_ARGS__) #define SCH_TASK_TLOG(param, ...) \ - qTrace("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ + qTrace("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ __VA_ARGS__) #define SCH_TASK_DLOGL(param, ...) \ - qDebugL("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ + qDebugL("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ __VA_ARGS__) #define SCH_TASK_WLOG(param, ...) \ - qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ + qWarn("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d " param, pJob->queryId, SCH_TASK_ID(pTask), SCH_TASK_EID(pTask), \ __VA_ARGS__) #define SCH_SET_ERRNO(_err) \ @@ -498,38 +498,62 @@ extern SSchedulerMgmt schMgmt; #define TD_RWLATCH_WRITE_FLAG_COPY 0x40000000 -#define SCH_LOCK(type, _lock) \ - do { \ - if (SCH_READ == (type)) { \ - ASSERTS(atomic_load_32(_lock) >= 0, "invalid lock value before read lock"); \ - SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosRLockLatch(_lock); \ - SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32(_lock) > 0, "invalid lock value after read lock"); \ - } else { \ - ASSERTS(atomic_load_32(_lock) >= 0, "invalid lock value before write lock"); \ - SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosWLockLatch(_lock); \ - SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32(_lock) == TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value after write lock"); \ - } \ +#define SCH_LOCK(type, _lock) \ + do { \ + if (SCH_READ == (type)) { \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value before read lock"); \ + break; \ + } \ + SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosRLockLatch(_lock); \ + SCH_LOCK_DEBUG("SCH RLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) <= 0) { \ + qError("invalid lock value after read lock"); \ + break; \ + } \ + } else { \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value before write lock"); \ + break; \ + } \ + SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosWLockLatch(_lock); \ + SCH_LOCK_DEBUG("SCH WLOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \ + qError("invalid lock value after write lock"); \ + break; \ + } \ + } \ } while (0) -#define SCH_UNLOCK(type, _lock) \ - do { \ - if (SCH_READ == (type)) { \ - ASSERTS(atomic_load_32((_lock)) > 0, "invalid lock value before read unlock"); \ - SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosRUnLockLatch(_lock); \ - SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after read unlock"); \ - } else { \ - ASSERTS(atomic_load_32((_lock)) & TD_RWLATCH_WRITE_FLAG_COPY, "invalid lock value before write unlock"); \ - SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - taosWUnLockLatch(_lock); \ - SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ - ASSERTS(atomic_load_32((_lock)) >= 0, "invalid lock value after write unlock"); \ - } \ +#define SCH_UNLOCK(type, _lock) \ + do { \ + if (SCH_READ == (type)) { \ + if (atomic_load_32((_lock)) <= 0) { \ + qError("invalid lock value before read unlock"); \ + break; \ + } \ + SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosRUnLockLatch(_lock); \ + SCH_LOCK_DEBUG("SCH RULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value after read unlock"); \ + break; \ + } \ + } else { \ + if (atomic_load_32((_lock)) != TD_RWLATCH_WRITE_FLAG_COPY) { \ + qError("invalid lock value before write unlock"); \ + break; \ + } \ + SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d B", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + taosWUnLockLatch(_lock); \ + SCH_LOCK_DEBUG("SCH WULOCK%p:%d, %s:%d E", (_lock), atomic_load_32(_lock), __FILE__, __LINE__); \ + if (atomic_load_32((_lock)) < 0) { \ + qError("invalid lock value after write unlock"); \ + break; \ + } \ + } \ } while (0) #define SCH_RESET_JOB_LEVEL_IDX(_job) \ @@ -583,7 +607,7 @@ int32_t schJobFetchRows(SSchJob *pJob); int32_t schJobFetchRowsA(SSchJob *pJob); int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, void *handle, int32_t execId); int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId *pEpId, SArray *pStatusList); -int32_t schDumpEpSet(SEpSet *pEpSet, char** ppRes); +int32_t schDumpEpSet(SEpSet *pEpSet, char **ppRes); char *schGetOpStr(SCH_OP_TYPE type); int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync); int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq); diff --git a/source/libs/scheduler/src/schJob.c b/source/libs/scheduler/src/schJob.c index 58a0706223..cf4a132eb5 100644 --- a/source/libs/scheduler/src/schJob.c +++ b/source/libs/scheduler/src/schJob.c @@ -87,7 +87,7 @@ int32_t schUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { if (JOB_TASK_STATUS_FETCH == newStatus) { return code; } - + SCH_ERR_JRET(TSDB_CODE_SCH_IGNORE_ERROR); } @@ -125,7 +125,7 @@ int32_t schUpdateJobStatus(SSchJob *pJob, int8_t newStatus) { newStatus != JOB_TASK_STATUS_FETCH) { SCH_ERR_JRET(TSDB_CODE_APP_ERROR); } - + break; case JOB_TASK_STATUS_SUCC: case JOB_TASK_STATUS_FAIL: @@ -171,14 +171,14 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { SCH_JOB_ELOG("fail to get the %dth level, levelNum: %d", i, pJob->levelNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + for (int32_t m = 0; m < pLevel->taskNum; ++m) { SSchTask *pTask = taosArrayGet(pLevel->subTasks, m); if (NULL == pTask) { SCH_JOB_ELOG("fail to get the %dth task in level %d, taskNum: %d", m, pLevel->level, pLevel->taskNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SSubplan *pPlan = pTask->plan; int32_t childNum = pPlan->pChildren ? (int32_t)LIST_LENGTH(pPlan->pChildren) : 0; int32_t parentNum = pPlan->pParents ? (int32_t)LIST_LENGTH(pPlan->pParents) : 0; @@ -197,12 +197,12 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { } for (int32_t n = 0; n < childNum; ++n) { - SSubplan *child = (SSubplan *)nodesListGetNode(pPlan->pChildren, n); + SSubplan *child = (SSubplan *)nodesListGetNode(pPlan->pChildren, n); if (NULL == child) { SCH_JOB_ELOG("fail to get the %dth child subplan, childNum: %d", n, childNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SSchTask **childTask = taosHashGet(planToTask, &child, POINTER_BYTES); if (NULL == childTask || NULL == *childTask) { SCH_TASK_ELOG("subplan children relationship error, level:%d, taskIdx:%d, childIdx:%d", i, m, n); @@ -236,12 +236,12 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { } for (int32_t n = 0; n < parentNum; ++n) { - SSubplan *parent = (SSubplan *)nodesListGetNode(pPlan->pParents, n); + SSubplan *parent = (SSubplan *)nodesListGetNode(pPlan->pParents, n); if (NULL == parent) { SCH_JOB_ELOG("fail to get the %dth parent subplan, parentNum: %d", n, parentNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SSchTask **parentTask = taosHashGet(planToTask, &parent, POINTER_BYTES); if (NULL == parentTask || NULL == *parentTask) { SCH_TASK_ELOG("subplan parent relationship error, level:%d, taskIdx:%d, childIdx:%d", i, m, n); @@ -265,7 +265,7 @@ int32_t schBuildTaskRalation(SSchJob *pJob, SHashObj *planToTask) { SCH_JOB_ELOG("fail to get level 0 level, levelNum:%d", (int32_t)taosArrayGetSize(pJob->levels)); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + if (SCH_IS_QUERY_JOB(pJob)) { if (pLevel->taskNum > 1) { SCH_JOB_ELOG("invalid query plan, level:0, taskNum:%d", pLevel->taskNum); @@ -354,7 +354,7 @@ int32_t schValidateAndBuildJob(SQueryPlan *pDag, SSchJob *pJob) { SCH_JOB_ELOG("fail to get the %dth level, levelNum: %d", i, levelNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + pLevel->level = i; plans = (SNodeListNode *)nodesListGetNode(pDag->pSubplans, i); @@ -427,7 +427,7 @@ _return: void schDumpJobExecRes(SSchJob *pJob, SExecResult *pRes) { pRes->code = atomic_load_32(&pJob->errCode); pRes->numOfRows = pJob->resNumOfRows; - + SCH_LOCK(SCH_WRITE, &pJob->resLock); pRes->res = pJob->execRes.res; pRes->msgType = pJob->execRes.msgType; @@ -519,7 +519,7 @@ void schPostJobRes(SSchJob *pJob, SCH_OP_TYPE op) { goto _return; } - if (op && pJob->opStatus.op != op) { + if (SCH_OP_NULL != op && pJob->opStatus.op != op) { SCH_JOB_ELOG("job in operation %s mis-match with expected %s", schGetOpStr(pJob->opStatus.op), schGetOpStr(op)); goto _return; } @@ -547,6 +547,7 @@ _return: int32_t schProcessOnJobFailure(SSchJob *pJob, int32_t errCode) { if (TSDB_CODE_SCH_IGNORE_ERROR == errCode) { + schPostJobRes(pJob, 0); return TSDB_CODE_SCH_IGNORE_ERROR; } @@ -567,8 +568,8 @@ int32_t schHandleJobFailure(SSchJob *pJob, int32_t errCode) { return TSDB_CODE_SCH_IGNORE_ERROR; } - (void)schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, &errCode); // ignore error - + (void)schSwitchJobStatus(pJob, JOB_TASK_STATUS_FAIL, &errCode); // ignore error + return TSDB_CODE_SCH_IGNORE_ERROR; } @@ -579,8 +580,8 @@ int32_t schHandleJobDrop(SSchJob *pJob, int32_t errCode) { return TSDB_CODE_SCH_IGNORE_ERROR; } - (void)schSwitchJobStatus(pJob, JOB_TASK_STATUS_DROP, &errCode); // ignore error - + (void)schSwitchJobStatus(pJob, JOB_TASK_STATUS_DROP, &errCode); // ignore error + return TSDB_CODE_SCH_IGNORE_ERROR; } @@ -626,14 +627,14 @@ int32_t schLaunchJobLowerLevel(SSchJob *pJob, SSchTask *pTask) { SCH_JOB_ELOG("fail to get the %dth level, levelNum:%d", pJob->levelIdx, (int32_t)taosArrayGetSize(pJob->levels)); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + for (int32_t i = 0; i < pLevel->taskNum; ++i) { SSchTask *pTask = taosArrayGet(pLevel->subTasks, i); if (NULL == pTask) { SCH_JOB_ELOG("fail to get the %dth task in level %d, taskNum:%d", i, pLevel->level, pLevel->taskNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + if (pTask->children && taosArrayGetSize(pTask->children) > 0) { continue; } @@ -661,13 +662,14 @@ int32_t schSaveJobExecRes(SSchJob *pJob, SQueryTableRsp *rsp) { } } - if (NULL == taosArrayAddBatch((SArray *)pJob->execRes.res, taosArrayGet(rsp->tbVerInfo, 0), taosArrayGetSize(rsp->tbVerInfo))) { + if (NULL == taosArrayAddBatch((SArray *)pJob->execRes.res, taosArrayGet(rsp->tbVerInfo, 0), + taosArrayGetSize(rsp->tbVerInfo))) { SCH_UNLOCK(SCH_WRITE, &pJob->resLock); SCH_ERR_RET(terrno); } - + taosArrayDestroy(rsp->tbVerInfo); - + pJob->execRes.msgType = TDMT_SCH_QUERY; SCH_UNLOCK(SCH_WRITE, &pJob->resLock); @@ -696,7 +698,7 @@ int32_t schLaunchJob(SSchJob *pJob) { SCH_JOB_ELOG("fail to get the %dth level, levelNum:%d", pJob->levelIdx, (int32_t)taosArrayGetSize(pJob->levels)); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SCH_ERR_RET(schLaunchLevelTasks(pJob, level)); } @@ -722,7 +724,7 @@ void schFreeJobImpl(void *job) { uint64_t queryId = pJob->queryId; int64_t refId = pJob->refId; - qDebug("QID:0x%" PRIx64 " begin to free sch job, refId:0x%" PRIx64 ", pointer:%p", queryId, refId, pJob); + qDebug("qid:0x%" PRIx64 " begin to free sch job, refId:0x%" PRIx64 ", pointer:%p", queryId, refId, pJob); schDropJobAllTasks(pJob); @@ -764,12 +766,12 @@ void schFreeJobImpl(void *job) { destroyQueryExecRes(&pJob->execRes); qDestroyQueryPlan(pJob->pDag); - (void)nodesReleaseAllocatorWeakRef(pJob->allocatorRefId); // ignore error + (void)nodesReleaseAllocatorWeakRef(pJob->allocatorRefId); // ignore error taosMemoryFreeClear(pJob->userRes.execRes); taosMemoryFreeClear(pJob->fetchRes); taosMemoryFreeClear(pJob->sql); - (void)tsem_destroy(&pJob->rspSem); // ignore error + (void)tsem_destroy(&pJob->rspSem); // ignore error taosMemoryFree(pJob); int32_t jobNum = atomic_sub_fetch_32(&schMgmt.jobNum, 1); @@ -777,7 +779,7 @@ void schFreeJobImpl(void *job) { schCloseJobRef(); } - qDebug("QID:0x%" PRIx64 " sch job freed, refId:0x%" PRIx64 ", pointer:%p", queryId, refId, pJob); + qDebug("qid:0x%" PRIx64 " sch job freed, refId:0x%" PRIx64 ", pointer:%p", queryId, refId, pJob); } int32_t schJobFetchRows(SSchJob *pJob) { @@ -788,7 +790,7 @@ int32_t schJobFetchRows(SSchJob *pJob) { if (schChkCurrentOp(pJob, SCH_OP_FETCH, true)) { SCH_JOB_DLOG("sync wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob)); - (void)tsem_wait(&pJob->rspSem); // ignore error + (void)tsem_wait(&pJob->rspSem); // ignore error SCH_RET(schDumpJobFetchRes(pJob, pJob->userRes.fetchRes)); } } else { @@ -807,7 +809,7 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) { int64_t refId = -1; SSchJob *pJob = taosMemoryCalloc(1, sizeof(SSchJob)); if (NULL == pJob) { - qError("QID:0x%" PRIx64 " calloc %d failed", pReq->pDag->queryId, (int32_t)sizeof(SSchJob)); + qError("qid:0x%" PRIx64 " calloc %d failed", pReq->pDag->queryId, (int32_t)sizeof(SSchJob)); SCH_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } @@ -817,7 +819,7 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) { if (pReq->sql) { pJob->sql = taosStrdup(pReq->sql); if (NULL == pJob->sql) { - qError("QID:0x%" PRIx64 " strdup sql %s failed", pReq->pDag->queryId, pReq->sql); + qError("qid:0x%" PRIx64 " strdup sql %s failed", pReq->pDag->queryId, pReq->sql); SCH_ERR_JRET(TSDB_CODE_OUT_OF_MEMORY); } } @@ -825,7 +827,7 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) { if (pReq->allocatorRefId > 0) { pJob->allocatorRefId = nodesMakeAllocatorWeakRef(pReq->allocatorRefId); if (pJob->allocatorRefId <= 0) { - qError("QID:0x%" PRIx64 " nodesMakeAllocatorWeakRef failed", pReq->pDag->queryId); + qError("qid:0x%" PRIx64 " nodesMakeAllocatorWeakRef failed", pReq->pDag->queryId); SCH_ERR_JRET(terrno); } } @@ -836,11 +838,12 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) { pJob->source = pReq->source; if (pReq->pNodeList == NULL || taosArrayGetSize(pReq->pNodeList) <= 0) { - qDebug("QID:0x%" PRIx64 " input exec nodeList is empty", pReq->pDag->queryId); + qDebug("qid:0x%" PRIx64 " input exec nodeList is empty", pReq->pDag->queryId); } else { pJob->nodeList = taosArrayDup(pReq->pNodeList, NULL); if (NULL == pJob->nodeList) { - qError("QID:0x%" PRIx64 " taosArrayDup failed, origNum:%d", pReq->pDag->queryId, (int32_t)taosArrayGetSize(pReq->pNodeList)); + qError("qid:0x%" PRIx64 " taosArrayDup failed, origNum:%d", pReq->pDag->queryId, + (int32_t)taosArrayGetSize(pReq->pNodeList)); SCH_ERR_JRET(terrno); } } @@ -891,7 +894,7 @@ _return: } else if (pJob->refId < 0) { schFreeJobImpl(pJob); } else { - (void)taosRemoveRef(schMgmt.jobRef, pJob->refId); // ignore error + (void)taosRemoveRef(schMgmt.jobRef, pJob->refId); // ignore error } SCH_RET(code); @@ -899,13 +902,13 @@ _return: int32_t schExecJob(SSchJob *pJob, SSchedulerReq *pReq) { int32_t code = 0; - qDebug("QID:0x%" PRIx64 " sch job refId 0x%" PRIx64 " started", pReq->pDag->queryId, pJob->refId); + qDebug("qid:0x%" PRIx64 " sch job refId 0x%" PRIx64 " started", pReq->pDag->queryId, pJob->refId); SCH_ERR_RET(schLaunchJob(pJob)); if (pReq->syncReq) { SCH_JOB_DLOG("sync wait for rsp now, job status:%s", SCH_GET_JOB_STATUS_STR(pJob)); - (void)tsem_wait(&pJob->rspSem); // ignore error + (void)tsem_wait(&pJob->rspSem); // ignore error } SCH_JOB_DLOG("job exec done, job status:%s, jobId:0x%" PRIx64, SCH_GET_JOB_STATUS_STR(pJob), pJob->refId); @@ -950,7 +953,7 @@ int32_t schResetJobForRetry(SSchJob *pJob, int32_t rspCode, bool *inRetry) { } *inRetry = true; - + SCH_ERR_RET(schChkResetJobRetry(pJob, rspCode)); int32_t code = 0; @@ -990,10 +993,9 @@ int32_t schResetJobForRetry(SSchJob *pJob, int32_t rspCode, bool *inRetry) { return TSDB_CODE_SUCCESS; } - int32_t schHandleJobRetry(SSchJob *pJob, SSchTask *pTask, SDataBuf *pMsg, int32_t rspCode) { int32_t code = 0; - bool inRetry = false; + bool inRetry = false; taosMemoryFreeClear(pMsg->pData); taosMemoryFreeClear(pMsg->pEpSet); @@ -1073,7 +1075,7 @@ void schProcessOnOpEnd(SSchJob *pJob, SCH_OP_TYPE type, SSchedulerReq *pReq, int } if (errCode) { - (void)schHandleJobFailure(pJob, errCode); // handle internal + (void)schHandleJobFailure(pJob, errCode); // handle internal } SCH_JOB_DLOG("job end %s operation with code %s", schGetOpStr(type), tstrerror(errCode)); @@ -1152,11 +1154,11 @@ void schProcessOnCbEnd(SSchJob *pJob, SSchTask *pTask, int32_t errCode) { } if (errCode) { - (void)schHandleJobFailure(pJob, errCode); // ignore error + (void)schHandleJobFailure(pJob, errCode); // ignore error } if (pJob) { - (void)schReleaseJob(pJob->refId); // ignore error + (void)schReleaseJob(pJob->refId); // ignore error } } @@ -1169,7 +1171,7 @@ int32_t schProcessOnCbBegin(SSchJob **job, SSchTask **task, uint64_t qId, int64_ (void)schAcquireJob(rId, &pJob); if (NULL == pJob) { - qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, qId, tId, rId); + qWarn("qid:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, qId, tId, rId); SCH_ERR_RET(TSDB_CODE_QRY_JOB_NOT_EXIST); } @@ -1193,7 +1195,7 @@ _return: SCH_UNLOCK_TASK(pTask); } if (pJob) { - (void)schReleaseJob(rId); // ignore error + (void)schReleaseJob(rId); // ignore error } SCH_RET(code); diff --git a/source/libs/scheduler/src/schRemote.c b/source/libs/scheduler/src/schRemote.c index 9215254f9c..ad720e15f5 100644 --- a/source/libs/scheduler/src/schRemote.c +++ b/source/libs/scheduler/src/schRemote.c @@ -499,7 +499,7 @@ _return: int32_t schHandleDropCallback(void *param, SDataBuf *pMsg, int32_t code) { SSchTaskCallbackParam *pParam = (SSchTaskCallbackParam *)param; - qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 " drop task rsp received, code:0x%x", pParam->queryId, pParam->taskId, + qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 " drop task rsp received, code:0x%x", pParam->queryId, pParam->taskId, code); // called if drop task rsp received code (void)rpcReleaseHandle(pMsg->handle, TAOS_CONN_CLIENT); // ignore error @@ -512,7 +512,7 @@ int32_t schHandleDropCallback(void *param, SDataBuf *pMsg, int32_t code) { int32_t schHandleNotifyCallback(void *param, SDataBuf *pMsg, int32_t code) { SSchTaskCallbackParam *pParam = (SSchTaskCallbackParam *)param; - qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 " task notify rsp received, code:0x%x", pParam->queryId, pParam->taskId, + qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 " task notify rsp received, code:0x%x", pParam->queryId, pParam->taskId, code); if (pMsg) { taosMemoryFree(pMsg->pData); diff --git a/source/libs/scheduler/src/schTask.c b/source/libs/scheduler/src/schTask.c index 6dd6aa9aae..729c5066ac 100644 --- a/source/libs/scheduler/src/schTask.c +++ b/source/libs/scheduler/src/schTask.c @@ -301,7 +301,7 @@ int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask) { SCH_TASK_ELOG("fail to get task %d parent, parentNum: %d", i, parentNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SCH_LOCK(SCH_WRITE, &parent->planLock); SDownstreamSourceNode source = { .type = QUERY_NODE_DOWNSTREAM_SOURCE, @@ -319,7 +319,7 @@ int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask) { SCH_UNLOCK(SCH_WRITE, &parent->planLock); SCH_ERR_RET(code); - + int32_t readyNum = atomic_add_fetch_32(&parent->childReady, 1); if (SCH_TASK_READY_FOR_LAUNCH(readyNum, parent)) { @@ -331,7 +331,7 @@ int32_t schProcessOnTaskSuccess(SSchJob *pJob, SSchTask *pTask) { if (taskDone == pTask->level->taskNum) { SCH_ERR_RET(schLaunchJobLowerLevel(pJob, pTask)); } - + return TSDB_CODE_SUCCESS; } @@ -422,10 +422,10 @@ void schResetTaskForRetry(SSchJob *pJob, SSchTask *pTask) { schDropTaskOnExecNode(pJob, pTask); if (pTask->delayTimer) { - (void)taosTmrStopA(&pTask->delayTimer); // ignore error + (void)taosTmrStopA(&pTask->delayTimer); // ignore error } taosHashClear(pTask->execNodes); - (void)schRemoveTaskFromExecList(pJob, pTask); // ignore error + (void)schRemoveTaskFromExecList(pJob, pTask); // ignore error schDeregisterTaskHb(pJob, pTask); taosMemoryFreeClear(pTask->msg); pTask->msgLen = 0; @@ -453,17 +453,19 @@ int32_t schDoTaskRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf *pData, int32 } else if (SYNC_SELF_LEADER_REDIRECT_ERROR(rspCode)) { SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx); if (NULL == addr) { - SCH_TASK_ELOG("fail to get the %dth condidateAddr, totalNum:%d", pTask->candidateIdx, (int32_t)taosArrayGetSize(pTask->candidateAddrs)); + SCH_TASK_ELOG("fail to get the %dth condidateAddr, totalNum:%d", pTask->candidateIdx, + (int32_t)taosArrayGetSize(pTask->candidateAddrs)); SCH_ERR_JRET(TSDB_CODE_SCH_INTERNAL_ERROR); } - - SEp *pEp = &addr->epSet.eps[addr->epSet.inUse]; + + SEp *pEp = &addr->epSet.eps[addr->epSet.inUse]; SCH_TASK_DLOG("task retry node %d current ep, idx:%d/%d,%s:%d, code:%s", addr->nodeId, addr->epSet.inUse, addr->epSet.numOfEps, pEp->fqdn, pEp->port, tstrerror(rspCode)); } else { SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx); if (NULL == addr) { - SCH_TASK_ELOG("fail to get the %dth condidateAddr, totalNum:%d", pTask->candidateIdx, (int32_t)taosArrayGetSize(pTask->candidateAddrs)); + SCH_TASK_ELOG("fail to get the %dth condidateAddr, totalNum:%d", pTask->candidateIdx, + (int32_t)taosArrayGetSize(pTask->candidateAddrs)); SCH_ERR_JRET(TSDB_CODE_SCH_INTERNAL_ERROR); } @@ -495,7 +497,7 @@ int32_t schDoTaskRedirect(SSchJob *pJob, SSchTask *pTask, SDataBuf *pData, int32 for (int32_t i = 0; i < childrenNum; ++i) { SSchTask *pChild = taosArrayGetP(pTask->children, i); SCH_LOCK_TASK(pChild); - (void)schDoTaskRedirect(pJob, pChild, NULL, rspCode); // error handled internal + (void)schDoTaskRedirect(pJob, pChild, NULL, rspCode); // error handled internal SCH_UNLOCK_TASK(pChild); } @@ -509,13 +511,13 @@ _return: int32_t schResetTaskSetLevelInfo(SSchJob *pJob, SSchTask *pTask) { SSchLevel *pLevel = pTask->level; - SCH_TASK_DLOG("start to reset level for current task set, execDone:%d, launched:%d", - atomic_load_32(&pLevel->taskExecDoneNum), atomic_load_32(&pLevel->taskLaunchedNum)); + SCH_TASK_DLOG("start to reset level for current task set, execDone:%d, launched:%d", + atomic_load_32(&pLevel->taskExecDoneNum), atomic_load_32(&pLevel->taskLaunchedNum)); if (SCH_GET_TASK_STATUS(pTask) >= JOB_TASK_STATUS_PART_SUCC) { (void)atomic_sub_fetch_32(&pLevel->taskExecDoneNum, 1); } - + (void)atomic_sub_fetch_32(&pLevel->taskLaunchedNum, 1); int32_t childrenNum = taosArrayGetSize(pTask->children); @@ -525,16 +527,16 @@ int32_t schResetTaskSetLevelInfo(SSchJob *pJob, SSchTask *pTask) { SCH_TASK_ELOG("fail to get the %dth child, childrenNum:%d", i, childrenNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SCH_LOCK_TASK(pChild); pLevel = pChild->level; (void)atomic_sub_fetch_32(&pLevel->taskExecDoneNum, 1); (void)atomic_sub_fetch_32(&pLevel->taskLaunchedNum, 1); SCH_UNLOCK_TASK(pChild); - } + } - SCH_TASK_DLOG("end to reset level for current task set, execDone:%d, launched:%d", - atomic_load_32(&pLevel->taskExecDoneNum), atomic_load_32(&pLevel->taskLaunchedNum)); + SCH_TASK_DLOG("end to reset level for current task set, execDone:%d, launched:%d", + atomic_load_32(&pLevel->taskExecDoneNum), atomic_load_32(&pLevel->taskLaunchedNum)); return TSDB_CODE_SUCCESS; } @@ -737,7 +739,7 @@ int32_t schTaskCheckSetRetry(SSchJob *pJob, SSchTask *pTask, int32_t errCode, bo int32_t schHandleTaskRetry(SSchJob *pJob, SSchTask *pTask) { (void)atomic_sub_fetch_32(&pTask->level->taskLaunchedNum, 1); - (void)schRemoveTaskFromExecList(pJob, pTask); // ignore error + (void)schRemoveTaskFromExecList(pJob, pTask); // ignore error SCH_SET_TASK_STATUS(pTask, JOB_TASK_STATUS_INIT); if (SCH_TASK_NEED_FLOW_CTRL(pJob, pTask)) { @@ -749,10 +751,11 @@ int32_t schHandleTaskRetry(SSchJob *pJob, SSchTask *pTask) { if (SCH_IS_DATA_BIND_TASK(pTask)) { SQueryNodeAddr *addr = taosArrayGet(pTask->candidateAddrs, pTask->candidateIdx); if (NULL == addr) { - SCH_TASK_ELOG("fail to the %dth condidateAddr, totalNum:%d", pTask->candidateIdx, (int32_t)taosArrayGetSize(pTask->candidateAddrs)); + SCH_TASK_ELOG("fail to the %dth condidateAddr, totalNum:%d", pTask->candidateIdx, + (int32_t)taosArrayGetSize(pTask->candidateAddrs)); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SCH_SWITCH_EPSET(addr); } else { SCH_ERR_RET(schSwitchTaskCandidateAddr(pJob, pTask)); @@ -776,7 +779,7 @@ int32_t schSetAddrsFromNodeList(SSchJob *pJob, SSchTask *pTask) { SCH_TASK_ELOG("fail to get the %dth node in nodeList, nodeNum:%d", i, nodeNum); SCH_ERR_RET(TSDB_CODE_SCH_INTERNAL_ERROR); } - + SQueryNodeAddr *naddr = &nload->addr; if (NULL == taosArrayPush(pTask->candidateAddrs, naddr)) { @@ -859,7 +862,7 @@ int32_t schUpdateTaskCandidateAddr(SSchJob *pJob, SSchTask *pTask, SEpSet *pEpSe char *origEpset = NULL; char *newEpset = NULL; - + SCH_ERR_RET(schDumpEpSet(&pAddr->epSet, &origEpset)); SCH_ERR_JRET(schDumpEpSet(pEpSet, &newEpset)); @@ -932,7 +935,7 @@ void schDropTaskOnExecNode(SSchJob *pJob, SSchTask *pTask) { if (nodeInfo->handle) { SCH_SET_TASK_HANDLE(pTask, nodeInfo->handle); void *pExecId = taosHashGetKey(nodeInfo, NULL); - (void)schBuildAndSendMsg(pJob, pTask, &nodeInfo->addr, TDMT_SCH_DROP_TASK, pExecId); // ignore error and continue + (void)schBuildAndSendMsg(pJob, pTask, &nodeInfo->addr, TDMT_SCH_DROP_TASK, pExecId); // ignore error and continue SCH_TASK_DLOG("start to drop task's %dth execNode", i); } else { @@ -986,10 +989,10 @@ int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId *pEpId, SArray *pStatusList) { qError("fail to get the %dth task status in hb rsp, taskNum:%d", i, taskNum); continue; } - - int32_t code = 0; - qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d task status in server: %s", pStatus->queryId, pStatus->taskId, + int32_t code = 0; + + qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ",EID:%d task status in server: %s", pStatus->queryId, pStatus->taskId, pStatus->execId, jobTaskStatusStr(pStatus->status)); if (schProcessOnCbBegin(&pJob, &pTask, pStatus->queryId, pStatus->refId, pStatus->taskId)) { @@ -1036,12 +1039,12 @@ int32_t schHandleExplainRes(SArray *pExplainRes) { continue; } - qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ", begin to handle LOCAL explain rsp msg", localRsp->qId, localRsp->tId); + qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ", begin to handle LOCAL explain rsp msg", localRsp->qId, localRsp->tId); pJob = NULL; (void)schAcquireJob(localRsp->rId, &pJob); if (NULL == pJob) { - qWarn("QID:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, localRsp->qId, + qWarn("qid:0x%" PRIx64 ",TID:0x%" PRIx64 "job no exist, may be dropped, refId:0x%" PRIx64, localRsp->qId, localRsp->tId, localRsp->rId); SCH_ERR_JRET(TSDB_CODE_QRY_JOB_NOT_EXIST); } @@ -1061,7 +1064,7 @@ int32_t schHandleExplainRes(SArray *pExplainRes) { (void)schReleaseJob(pJob->refId); - qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 ", end to handle LOCAL explain rsp msg, code:%x", localRsp->qId, + qDebug("qid:0x%" PRIx64 ",TID:0x%" PRIx64 ", end to handle LOCAL explain rsp msg, code:%x", localRsp->qId, localRsp->tId, code); SCH_ERR_JRET(code); @@ -1079,7 +1082,7 @@ _return: qError("in _return fail to get the %dth LOCAL explain rsp msg, total:%d", i, resNum); continue; } - + tFreeSExplainRsp(&localRsp->rsp); } diff --git a/source/libs/stream/src/streamBackendRocksdb.c b/source/libs/stream/src/streamBackendRocksdb.c index 83e55791d2..29fb18ef07 100644 --- a/source/libs/stream/src/streamBackendRocksdb.c +++ b/source/libs/stream/src/streamBackendRocksdb.c @@ -420,7 +420,10 @@ int32_t remoteChkpGetDelFile(char* path, SArray* toDel) { } void cleanDir(const char* pPath, const char* id) { - ASSERT(pPath != NULL); + if (pPath == NULL) { + stError("%s try to clean dir, but path is NULL", id); + return; + } if (taosIsDir(pPath)) { taosRemoveDir(pPath); @@ -2603,7 +2606,7 @@ void taskDbDestroy(void* pDb, bool flush) { stDebug("succ to destroy stream backend:%p", wrapper); int8_t nCf = tListLen(ginitDict); - if (flush && wrapper->removeAllFiles == 0) { + if (flush && wrapper->removeAllFiles == 0) { if (wrapper->db && wrapper->pCf) { rocksdb_flushoptions_t* flushOpt = rocksdb_flushoptions_create(); rocksdb_flushoptions_set_wait(flushOpt, 1); diff --git a/source/libs/stream/src/streamCheckStatus.c b/source/libs/stream/src/streamCheckStatus.c index b7fb48ddc6..2d04f2cb34 100644 --- a/source/libs/stream/src/streamCheckStatus.c +++ b/source/libs/stream/src/streamCheckStatus.c @@ -36,7 +36,7 @@ static void setCheckDownstreamReqInfo(SStreamTaskCheckReq* pReq, int64_t reqId, static void getCheckRspStatus(STaskCheckInfo* pInfo, int64_t el, int32_t* numOfReady, int32_t* numOfFault, int32_t* numOfNotRsp, SArray* pTimeoutList, SArray* pNotReadyList, const char* id); static int32_t addDownstreamFailedStatusResultAsync(SMsgCb* pMsgCb, int32_t vgId, int64_t streamId, int32_t taskId); -static void findCheckRspStatus(STaskCheckInfo* pInfo, int32_t taskId, SDownstreamStatusInfo** pStatusInfo); +static void findCheckRspStatus(STaskCheckInfo* pInfo, int32_t taskId, SDownstreamStatusInfo** pStatusInfo); int32_t streamTaskCheckStatus(SStreamTask* pTask, int32_t upstreamTaskId, int32_t vgId, int64_t stage, int64_t* oldStage) { @@ -107,11 +107,12 @@ void streamTaskSendCheckMsg(SStreamTask* pTask) { streamTaskAddReqInfo(&pTask->taskCheckInfo, req.reqId, pDispatch->taskId, pDispatch->nodeId, idstr); stDebug("s-task:%s (vgId:%d) stage:%" PRId64 " check single downstream task:0x%x(vgId:%d) ver:%" PRId64 "-%" PRId64 - " window:%" PRId64 "-%" PRId64 " reqId:0x%" PRIx64, + " window:%" PRId64 "-%" PRId64 " qid:0x%" PRIx64, idstr, pTask->info.nodeId, req.stage, req.downstreamTaskId, req.downstreamNodeId, pRange->range.minVer, pRange->range.maxVer, pWindow->skey, pWindow->ekey, req.reqId); - (void) streamSendCheckMsg(pTask, &req, pTask->outputInfo.fixedDispatcher.nodeId, &pTask->outputInfo.fixedDispatcher.epSet); + (void)streamSendCheckMsg(pTask, &req, pTask->outputInfo.fixedDispatcher.nodeId, + &pTask->outputInfo.fixedDispatcher.epSet); } else if (pTask->outputInfo.type == TASK_OUTPUT__SHUFFLE_DISPATCH) { streamTaskStartMonitorCheckRsp(pTask); @@ -132,9 +133,9 @@ void streamTaskSendCheckMsg(SStreamTask* pTask) { streamTaskAddReqInfo(&pTask->taskCheckInfo, req.reqId, pVgInfo->taskId, pVgInfo->vgId, idstr); stDebug("s-task:%s (vgId:%d) stage:%" PRId64 - " check downstream task:0x%x (vgId:%d) (shuffle), idx:%d, reqId:0x%" PRIx64, + " check downstream task:0x%x (vgId:%d) (shuffle), idx:%d, qid:0x%" PRIx64, idstr, pTask->info.nodeId, req.stage, req.downstreamTaskId, req.downstreamNodeId, i, req.reqId); - (void) streamSendCheckMsg(pTask, &req, pVgInfo->vgId, &pVgInfo->epSet); + (void)streamSendCheckMsg(pTask, &req, pVgInfo->vgId, &pVgInfo->epSet); } } else { // for sink task, set it ready directly. stDebug("s-task:%s (vgId:%d) set downstream ready, since no downstream", idstr, pTask->info.nodeId); @@ -164,20 +165,20 @@ void streamTaskProcessCheckMsg(SStreamMeta* pMeta, SStreamTaskCheckReq* pReq, SS pRsp->status = TASK_DOWNSTREAM_NOT_LEADER; } else { SStreamTask* pTask = NULL; - int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, taskId, &pTask); + int32_t code = streamMetaAcquireTask(pMeta, pReq->streamId, taskId, &pTask); if (pTask != NULL) { pRsp->status = streamTaskCheckStatus(pTask, pReq->upstreamTaskId, pReq->upstreamNodeId, pReq->stage, &pRsp->oldStage); SStreamTaskState pState = streamTaskGetStatus(pTask); - stDebug("s-task:%s status:%s, stage:%" PRId64 " recv task check req(reqId:0x%" PRIx64 + stDebug("s-task:%s status:%s, stage:%" PRId64 " recv task check req(qid:0x%" PRIx64 ") task:0x%x (vgId:%d), check_status:%d", pTask->id.idStr, pState.name, pRsp->oldStage, pRsp->reqId, pRsp->upstreamTaskId, pRsp->upstreamNodeId, pRsp->status); streamMetaReleaseTask(pMeta, pTask); } else { pRsp->status = TASK_DOWNSTREAM_NOT_READY; - stDebug("tq recv task check(taskId:0x%" PRIx64 "-0x%x not built yet) req(reqId:0x%" PRIx64 + stDebug("tq recv task check(taskId:0x%" PRIx64 "-0x%x not built yet) req(qid:0x%" PRIx64 ") from task:0x%x (vgId:%d), rsp check_status %d", pReq->streamId, taskId, pRsp->reqId, pRsp->upstreamTaskId, pRsp->upstreamNodeId, pRsp->status); } @@ -261,7 +262,7 @@ int32_t streamTaskSendCheckRsp(const SStreamMeta* pMeta, int32_t vgId, SStreamTa void* abuf = POINTER_SHIFT(buf, sizeof(SMsgHead)); tEncoderInit(&encoder, (uint8_t*)abuf, len); - (void) tEncodeStreamTaskCheckRsp(&encoder, pRsp); + (void)tEncodeStreamTaskCheckRsp(&encoder, pRsp); tEncoderClear(&encoder); SRpcMsg rspMsg = {.code = 0, .pCont = buf, .contLen = sizeof(SMsgHead) + len, .info = *pRpcInfo}; @@ -300,7 +301,8 @@ void streamTaskStartMonitorCheckRsp(SStreamTask* pTask) { if (pInfo->checkRspTmr == NULL) { pInfo->checkRspTmr = taosTmrStart(rspMonitorFn, CHECK_RSP_CHECK_INTERVAL, pTask, streamTimer); } else { - streamTmrReset(rspMonitorFn, CHECK_RSP_CHECK_INTERVAL, pTask, streamTimer, &pInfo->checkRspTmr, vgId, "check-status-monitor"); + streamTmrReset(rspMonitorFn, CHECK_RSP_CHECK_INTERVAL, pTask, streamTimer, &pInfo->checkRspTmr, vgId, + "check-status-monitor"); } streamMutexUnlock(&pInfo->checkInfoLock); @@ -319,7 +321,7 @@ void streamTaskCleanupCheckInfo(STaskCheckInfo* pInfo) { pInfo->pList = NULL; if (pInfo->checkRspTmr != NULL) { - (void) taosTmrStop(pInfo->checkRspTmr); + (void)taosTmrStop(pInfo->checkRspTmr); pInfo->checkRspTmr = NULL; } @@ -329,11 +331,11 @@ void streamTaskCleanupCheckInfo(STaskCheckInfo* pInfo) { /////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// void processDownstreamReadyRsp(SStreamTask* pTask) { EStreamTaskEvent event = (pTask->info.fillHistory == 0) ? TASK_EVENT_INIT : TASK_EVENT_INIT_SCANHIST; - (void) streamTaskOnHandleEventSuccess(pTask->status.pSM, event, NULL, NULL); + (void)streamTaskOnHandleEventSuccess(pTask->status.pSM, event, NULL, NULL); int64_t checkTs = pTask->execInfo.checkTs; int64_t readyTs = pTask->execInfo.readyTs; - (void) streamMetaAddTaskLaunchResult(pTask->pMeta, pTask->id.streamId, pTask->id.taskId, checkTs, readyTs, true); + (void)streamMetaAddTaskLaunchResult(pTask->pMeta, pTask->id.streamId, pTask->id.taskId, checkTs, readyTs, true); if (pTask->status.taskStatus == TASK_STATUS__HALT) { if (!HAS_RELATED_FILLHISTORY_TASK(pTask) || (pTask->info.fillHistory != 0)) { @@ -355,7 +357,7 @@ void processDownstreamReadyRsp(SStreamTask* pTask) { // todo: let's retry if (HAS_RELATED_FILLHISTORY_TASK(pTask)) { stDebug("s-task:%s try to launch related fill-history task", pTask->id.idStr); - (void) streamLaunchFillHistoryTask(pTask); + (void)streamLaunchFillHistoryTask(pTask); } } @@ -379,7 +381,7 @@ void addIntoNodeUpdateList(SStreamTask* pTask, int32_t nodeId) { if (!existed) { SDownstreamTaskEpset t = {.nodeId = nodeId}; - void* p = taosArrayPush(pTask->outputInfo.pNodeEpsetUpdateList, &t); + void* p = taosArrayPush(pTask->outputInfo.pNodeEpsetUpdateList, &t); if (p == NULL) { // todo let's retry } @@ -430,7 +432,7 @@ int32_t streamTaskUpdateCheckInfo(STaskCheckInfo* pInfo, int32_t taskId, int32_t findCheckRspStatus(pInfo, taskId, &p); if (p != NULL) { if (reqId != p->reqId) { - stError("s-task:%s reqId:0x%" PRIx64 " expected:0x%" PRIx64 + stError("s-task:%s qid:0x%" PRIx64 " expected:0x%" PRIx64 " expired check-rsp recv from downstream task:0x%x, discarded", id, reqId, p->reqId, taskId); streamMutexUnlock(&pInfo->checkInfoLock); @@ -452,7 +454,7 @@ int32_t streamTaskUpdateCheckInfo(STaskCheckInfo* pInfo, int32_t taskId, int32_t } streamMutexUnlock(&pInfo->checkInfoLock); - stError("s-task:%s unexpected check rsp msg, invalid downstream task:0x%x, reqId:%" PRIx64 " discarded", id, taskId, + stError("s-task:%s unexpected check rsp msg, invalid downstream task:0x%x, qid:%" PRIx64 " discarded", id, taskId, reqId); return TSDB_CODE_FAILED; } @@ -539,10 +541,10 @@ void doSendCheckMsg(SStreamTask* pTask, SDownstreamStatusInfo* p) { STaskDispatcherFixed* pDispatch = &pOutputInfo->fixedDispatcher; setCheckDownstreamReqInfo(&req, p->reqId, pDispatch->taskId, pDispatch->nodeId); - stDebug("s-task:%s (vgId:%d) stage:%" PRId64 " re-send check downstream task:0x%x(vgId:%d) reqId:0x%" PRIx64, id, + stDebug("s-task:%s (vgId:%d) stage:%" PRId64 " re-send check downstream task:0x%x(vgId:%d) qid:0x%" PRIx64, id, pTask->info.nodeId, req.stage, req.downstreamTaskId, req.downstreamNodeId, req.reqId); - (void) streamSendCheckMsg(pTask, &req, pOutputInfo->fixedDispatcher.nodeId, &pOutputInfo->fixedDispatcher.epSet); + (void)streamSendCheckMsg(pTask, &req, pOutputInfo->fixedDispatcher.nodeId, &pOutputInfo->fixedDispatcher.epSet); } else if (pOutputInfo->type == TASK_OUTPUT__SHUFFLE_DISPATCH) { SArray* vgInfo = pOutputInfo->shuffleDispatcher.dbInfo.pVgroupInfos; int32_t numOfVgs = taosArrayGetSize(vgInfo); @@ -557,9 +559,9 @@ void doSendCheckMsg(SStreamTask* pTask, SDownstreamStatusInfo* p) { setCheckDownstreamReqInfo(&req, p->reqId, pVgInfo->taskId, pVgInfo->vgId); stDebug("s-task:%s (vgId:%d) stage:%" PRId64 - " re-send check downstream task:0x%x(vgId:%d) (shuffle), idx:%d reqId:0x%" PRIx64, + " re-send check downstream task:0x%x(vgId:%d) (shuffle), idx:%d qid:0x%" PRIx64, id, pTask->info.nodeId, req.stage, req.downstreamTaskId, req.downstreamNodeId, i, p->reqId); - (void) streamSendCheckMsg(pTask, &req, pVgInfo->vgId, &pVgInfo->epSet); + (void)streamSendCheckMsg(pTask, &req, pVgInfo->vgId, &pVgInfo->epSet); break; } } @@ -580,15 +582,15 @@ void getCheckRspStatus(STaskCheckInfo* pInfo, int64_t el, int32_t* numOfReady, i stDebug("s-task:%s recv status:NEW_STAGE/NOT_LEADER from downstream, task:0x%x, quit from check downstream", id, p->taskId); (*numOfFault) += 1; - } else { // TASK_DOWNSTREAM_NOT_READY - if (p->rspTs == 0) { // not response yet + } else { // TASK_DOWNSTREAM_NOT_READY + if (p->rspTs == 0) { // not response yet if (el >= CHECK_NOT_RSP_DURATION) { // not receive info for 10 sec. - (void) taosArrayPush(pTimeoutList, &p->taskId); + (void)taosArrayPush(pTimeoutList, &p->taskId); } else { // el < CHECK_NOT_RSP_DURATION (*numOfNotRsp) += 1; // do nothing and continue waiting for their rsp } } else { - (void) taosArrayPush(pNotReadyList, &p->taskId); + (void)taosArrayPush(pNotReadyList, &p->taskId); } } } @@ -613,14 +615,13 @@ void handleTimeoutDownstreamTasks(SStreamTask* pTask, SArray* pTimeoutList) { continue; } - int32_t taskId = *px; + int32_t taskId = *px; SDownstreamStatusInfo* p = NULL; findCheckRspStatus(pInfo, taskId, &p); if (p != NULL) { - if (p->status != -1 || p->rspTs != 0) { - stError("s-task:%s invalid rsp record entry, index:%d, status:%d, rspTs:%"PRId64, pTask->id.idStr, i, p->status, - p->rspTs); + stError("s-task:%s invalid rsp record entry, index:%d, status:%d, rspTs:%" PRId64, pTask->id.idStr, i, + p->status, p->rspTs); continue; } @@ -692,24 +693,24 @@ int32_t addDownstreamFailedStatusResultAsync(SMsgCb* pMsgCb, int32_t vgId, int64 // this function is executed in timer thread void rspMonitorFn(void* param, void* tmrId) { - SStreamTask* pTask = param; - SStreamMeta* pMeta = pTask->pMeta; - STaskCheckInfo* pInfo = &pTask->taskCheckInfo; - int32_t vgId = pTask->pMeta->vgId; - int64_t now = taosGetTimestampMs(); - int64_t timeoutDuration = now - pInfo->timeoutStartTs; - const char* id = pTask->id.idStr; - int32_t numOfReady = 0; - int32_t numOfFault = 0; - int32_t numOfNotRsp = 0; - int32_t numOfNotReady = 0; - int32_t numOfTimeout = 0; - int32_t total = taosArrayGetSize(pInfo->pList); + SStreamTask* pTask = param; + SStreamMeta* pMeta = pTask->pMeta; + STaskCheckInfo* pInfo = &pTask->taskCheckInfo; + int32_t vgId = pTask->pMeta->vgId; + int64_t now = taosGetTimestampMs(); + int64_t timeoutDuration = now - pInfo->timeoutStartTs; + const char* id = pTask->id.idStr; + int32_t numOfReady = 0; + int32_t numOfFault = 0; + int32_t numOfNotRsp = 0; + int32_t numOfNotReady = 0; + int32_t numOfTimeout = 0; + int32_t total = taosArrayGetSize(pInfo->pList); stDebug("s-task:%s start to do check-downstream-rsp check in tmr", id); streamMutexLock(&pTask->lock); - SStreamTaskState state = streamTaskGetStatus(pTask); + SStreamTaskState state = streamTaskGetStatus(pTask); streamMutexUnlock(&pTask->lock); if (state.state == TASK_STATUS__STOP) { @@ -721,7 +722,7 @@ void rspMonitorFn(void* param, void* tmrId) { // not record the failed of the current task if try to close current vnode // otherwise, the put of message operation may incur invalid read of message queue. if (!pMeta->closeFlag) { - (void) addDownstreamFailedStatusResultAsync(pTask->pMsgCb, vgId, pTask->id.streamId, pTask->id.taskId); + (void)addDownstreamFailedStatusResultAsync(pTask->pMsgCb, vgId, pTask->id.streamId, pTask->id.taskId); } streamMetaReleaseTask(pMeta, pTask); @@ -740,8 +741,8 @@ void rspMonitorFn(void* param, void* tmrId) { streamMutexLock(&pInfo->checkInfoLock); if (pInfo->notReadyTasks == 0) { int32_t ref = atomic_sub_fetch_32(&pTask->status.timerActive, 1); - stDebug("s-task:%s status:%s vgId:%d all downstream ready, quit from monitor rsp tmr, ref:%d", id, state.name, - vgId, ref); + stDebug("s-task:%s status:%s vgId:%d all downstream ready, quit from monitor rsp tmr, ref:%d", id, state.name, vgId, + ref); streamTaskCompleteCheckRsp(pInfo, false, id); streamMutexUnlock(&pInfo->checkInfoLock); @@ -803,7 +804,7 @@ void rspMonitorFn(void* param, void* tmrId) { streamTaskCompleteCheckRsp(pInfo, false, id); streamMutexUnlock(&pInfo->checkInfoLock); - (void) addDownstreamFailedStatusResultAsync(pTask->pMsgCb, vgId, pTask->id.streamId, pTask->id.taskId); + (void)addDownstreamFailedStatusResultAsync(pTask->pMsgCb, vgId, pTask->id.streamId, pTask->id.taskId); streamMetaReleaseTask(pMeta, pTask); taosArrayDestroy(pNotReadyList); @@ -819,7 +820,8 @@ void rspMonitorFn(void* param, void* tmrId) { handleTimeoutDownstreamTasks(pTask, pTimeoutList); } - streamTmrReset(rspMonitorFn, CHECK_RSP_CHECK_INTERVAL, pTask, streamTimer, &pInfo->checkRspTmr, vgId, "check-status-monitor"); + streamTmrReset(rspMonitorFn, CHECK_RSP_CHECK_INTERVAL, pTask, streamTimer, &pInfo->checkRspTmr, vgId, + "check-status-monitor"); streamMutexUnlock(&pInfo->checkInfoLock); stDebug( diff --git a/source/libs/stream/src/streamDispatch.c b/source/libs/stream/src/streamDispatch.c index 918a6b6cb1..a6c2ae73e5 100644 --- a/source/libs/stream/src/streamDispatch.c +++ b/source/libs/stream/src/streamDispatch.c @@ -118,7 +118,7 @@ int32_t streamTaskBroadcastRetrieveReq(SStreamTask* pTask, SStreamRetrieveReq* r void* abuf = POINTER_SHIFT(buf, sizeof(SMsgHead)); SEncoder encoder; tEncoderInit(&encoder, abuf, len); - (void) tEncodeStreamRetrieveReq(&encoder, req); + (void)tEncodeStreamRetrieveReq(&encoder, req); tEncoderClear(&encoder); SRpcMsg rpcMsg = {0}; @@ -130,7 +130,7 @@ int32_t streamTaskBroadcastRetrieveReq(SStreamTask* pTask, SStreamRetrieveReq* r return code; } - stDebug("s-task:%s (child %d) send retrieve req to task:0x%x (vgId:%d), reqId:0x%" PRIx64, pTask->id.idStr, + stDebug("s-task:%s (child %d) send retrieve req to task:0x%x (vgId:%d), qid:0x%" PRIx64, pTask->id.idStr, pTask->info.selfChildId, pEpInfo->taskId, pEpInfo->nodeId, req->reqId); } @@ -158,6 +158,10 @@ static int32_t buildStreamRetrieveReq(SStreamTask* pTask, const SSDataBlock* pBl pRetrieve->version = htobe64(pBlock->info.version); int32_t actualLen = blockEncode(pBlock, pRetrieve->data + PAYLOAD_PREFIX_LEN, numOfCols); + if (actualLen < 0) { + taosMemoryFree(pRetrieve); + return terrno; + } SET_PAYLOAD_LEN(pRetrieve->data, actualLen, actualLen); int32_t payloadLen = actualLen + PAYLOAD_PREFIX_LEN; @@ -254,7 +258,7 @@ static SStreamDispatchReq* createDispatchDataReq(SStreamTask* pTask, const SStre int32_t type = pTask->outputInfo.type; int32_t num = streamTaskGetNumOfDownstream(pTask); - if(type != TASK_OUTPUT__SHUFFLE_DISPATCH && type != TASK_OUTPUT__FIXED_DISPATCH) { + if (type != TASK_OUTPUT__SHUFFLE_DISPATCH && type != TASK_OUTPUT__FIXED_DISPATCH) { terrno = TSDB_CODE_INVALID_PARA; stError("s-task:%s invalid dispatch type:%d not dispatch data", pTask->id.idStr, type); return NULL; @@ -283,7 +287,7 @@ static SStreamDispatchReq* createDispatchDataReq(SStreamTask* pTask, const SStre return NULL; } } - } else { // shuffle dispatch + } else { // shuffle dispatch int32_t numOfBlocks = taosArrayGetSize(pData->blocks); int32_t downstreamTaskId = pTask->outputInfo.fixedDispatcher.taskId; @@ -470,7 +474,7 @@ static void addDispatchEntry(SDispatchMsgInfo* pMsgInfo, int32_t nodeId, int64_t streamMutexLock(&pMsgInfo->lock); } - (void) taosArrayPush(pMsgInfo->pSendInfo, &entry); + (void)taosArrayPush(pMsgInfo->pSendInfo, &entry); if (lock) { streamMutexUnlock(&pMsgInfo->lock); @@ -591,7 +595,7 @@ static void doMonitorDispatchData(void* param, void* tmrId) { SEpSet* pEpSet = &pTask->outputInfo.fixedDispatcher.epSet; int32_t downstreamTaskId = pTask->outputInfo.fixedDispatcher.taskId; - int32_t s = taosArrayGetSize(pTask->msgInfo.pSendInfo); + int32_t s = taosArrayGetSize(pTask->msgInfo.pSendInfo); SDispatchEntry* pEntry = taosArrayGet(pTask->msgInfo.pSendInfo, 0); if (pEntry != NULL) { setResendInfo(pEntry, now); @@ -617,7 +621,8 @@ static void doMonitorDispatchData(void* param, void* tmrId) { void streamStartMonitorDispatchData(SStreamTask* pTask, int64_t waitDuration) { int32_t vgId = pTask->pMeta->vgId; if (pTask->msgInfo.pRetryTmr != NULL) { - streamTmrReset(doMonitorDispatchData, waitDuration, pTask, streamTimer, &pTask->msgInfo.pRetryTmr, vgId, "dispatch-monitor-tmr"); + streamTmrReset(doMonitorDispatchData, waitDuration, pTask, streamTimer, &pTask->msgInfo.pRetryTmr, vgId, + "dispatch-monitor-tmr"); } else { pTask->msgInfo.pRetryTmr = taosTmrStart(doMonitorDispatchData, waitDuration, pTask, streamTimer); } @@ -651,7 +656,8 @@ int32_t streamSearchAndAddBlock(SStreamTask* pTask, SStreamDispatchReq* pReqs, S } } } else { - (void) buildCtbNameByGroupIdImpl(pTask->outputInfo.shuffleDispatcher.stbFullName, groupId, pDataBlock->info.parTbName); + (void)buildCtbNameByGroupIdImpl(pTask->outputInfo.shuffleDispatcher.stbFullName, groupId, + pDataBlock->info.parTbName); } snprintf(ctbName, TSDB_TABLE_NAME_LEN, "%s.%s", pTask->outputInfo.shuffleDispatcher.dbInfo.db, @@ -666,7 +672,7 @@ int32_t streamSearchAndAddBlock(SStreamTask* pTask, SStreamDispatchReq* pReqs, S // failed to put into name buffer, no need to do anything if (tSimpleHashGetSize(pTask->pNameMap) < MAX_BLOCK_NAME_NUM) { - (void) tSimpleHashPut(pTask->pNameMap, &groupId, sizeof(int64_t), &bln, sizeof(SBlockName)); + (void)tSimpleHashPut(pTask->pNameMap, &groupId, sizeof(int64_t), &bln, sizeof(SBlockName)); } } @@ -848,7 +854,8 @@ static void checkpointReadyMsgSendMonitorFn(void* param, void* tmrId) { } if (++pTmrInfo->activeCounter < 50) { - streamTmrReset(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer, &pTmrInfo->tmrHandle, vgId, "chkpt-ready-monitor"); + streamTmrReset(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer, &pTmrInfo->tmrHandle, vgId, + "chkpt-ready-monitor"); return; } @@ -907,7 +914,7 @@ static void checkpointReadyMsgSendMonitorFn(void* param, void* tmrId) { continue; } - (void) taosArrayPush(pNotRspList, &pInfo->upstreamTaskId); + (void)taosArrayPush(pNotRspList, &pInfo->upstreamTaskId); stDebug("s-task:%s vgId:%d level:%d checkpoint-ready rsp from upstream:0x%x not confirmed yet", id, vgId, pTask->info.taskLevel, pInfo->upstreamTaskId); } @@ -931,8 +938,8 @@ static void checkpointReadyMsgSendMonitorFn(void* param, void* tmrId) { if (*pTaskId == pReadyInfo->upstreamTaskId) { // send msg again SRpcMsg msg = {0}; - int32_t code = initCheckpointReadyMsg(pTask, pReadyInfo->upstreamNodeId, pReadyInfo->upstreamTaskId, pReadyInfo->childId, - checkpointId, &msg); + int32_t code = initCheckpointReadyMsg(pTask, pReadyInfo->upstreamNodeId, pReadyInfo->upstreamTaskId, + pReadyInfo->childId, checkpointId, &msg); if (code == TSDB_CODE_SUCCESS) { code = tmsgSendReq(&pReadyInfo->upstreamNodeEpset, &msg); if (code == TSDB_CODE_SUCCESS) { @@ -948,7 +955,8 @@ static void checkpointReadyMsgSendMonitorFn(void* param, void* tmrId) { } } - streamTmrReset(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer, &pTmrInfo->tmrHandle, vgId, "chkpt-ready-monitor"); + streamTmrReset(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer, &pTmrInfo->tmrHandle, vgId, + "chkpt-ready-monitor"); streamMutexUnlock(&pActiveInfo->lock); } else { int32_t ref = streamCleanBeforeQuitTmr(pTmrInfo, pTask); @@ -1015,7 +1023,8 @@ int32_t streamTaskSendCheckpointReadyMsg(SStreamTask* pTask) { if (pTmrInfo->tmrHandle == NULL) { pTmrInfo->tmrHandle = taosTmrStart(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer); } else { - streamTmrReset(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer, &pTmrInfo->tmrHandle, vgId, "chkpt-ready-monitor"); + streamTmrReset(checkpointReadyMsgSendMonitorFn, 200, pTask, streamTimer, &pTmrInfo->tmrHandle, vgId, + "chkpt-ready-monitor"); } // mark the timer monitor checkpointId @@ -1059,7 +1068,7 @@ int32_t streamAddBlockIntoDispatchMsg(const SSDataBlock* pBlock, SStreamDispatch void* buf = taosMemoryCalloc(1, dataStrLen); if (buf == NULL) { - return -1; + return terrno; } SRetrieveTableRsp* pRetrieve = (SRetrieveTableRsp*)buf; @@ -1079,6 +1088,10 @@ int32_t streamAddBlockIntoDispatchMsg(const SSDataBlock* pBlock, SStreamDispatch pRetrieve->numOfCols = htonl(numOfCols); int32_t actualLen = blockEncode(pBlock, pRetrieve->data + PAYLOAD_PREFIX_LEN, numOfCols); + if (actualLen < 0) { + taosMemoryFree(buf); + return terrno; + } SET_PAYLOAD_LEN(pRetrieve->data, actualLen, actualLen); int32_t payloadLen = actualLen + PAYLOAD_PREFIX_LEN; @@ -1087,8 +1100,8 @@ int32_t streamAddBlockIntoDispatchMsg(const SSDataBlock* pBlock, SStreamDispatch payloadLen += sizeof(SRetrieveTableRsp); - (void) taosArrayPush(pReq->dataLen, &payloadLen); - (void) taosArrayPush(pReq->data, &buf); + (void)taosArrayPush(pReq->dataLen, &payloadLen); + (void)taosArrayPush(pReq->data, &buf); pReq->totalLen += dataStrLen; return 0; @@ -1169,7 +1182,7 @@ int32_t streamTaskBuildCheckpointSourceRsp(SStreamCheckpointSourceReq* pReq, SRp void* abuf = POINTER_SHIFT(pBuf, sizeof(SMsgHead)); tEncoderInit(&encoder, (uint8_t*)abuf, len); - (void) tEncodeStreamCheckpointSourceRsp(&encoder, &rsp); + (void)tEncodeStreamCheckpointSourceRsp(&encoder, &rsp); tEncoderClear(&encoder); initRpcMsg(pMsg, 0, pBuf, sizeof(SMsgHead) + len); @@ -1185,7 +1198,7 @@ int32_t streamAddCheckpointSourceRspMsg(SStreamCheckpointSourceReq* pReq, SRpcHa .recvTs = taosGetTimestampMs(), .transId = pReq->transId, .checkpointId = pReq->checkpointId}; // todo retry until it success - (void) streamTaskBuildCheckpointSourceRsp(pReq, pRpcInfo, &info.msg, TSDB_CODE_SUCCESS); + (void)streamTaskBuildCheckpointSourceRsp(pReq, pRpcInfo, &info.msg, TSDB_CODE_SUCCESS); SActiveCheckpointInfo* pActiveInfo = pTask->chkInfo.pActiveInfo; streamMutexLock(&pActiveInfo->lock); @@ -1208,7 +1221,7 @@ int32_t streamAddCheckpointSourceRspMsg(SStreamCheckpointSourceReq* pReq, SRpcHa pTask->id.idStr, pReady->checkpointId, pReady->transId, pReq->transId, pReq->checkpointId); } } else { - (void) taosArrayPush(pActiveInfo->pReadyMsgList, &info); + (void)taosArrayPush(pActiveInfo->pReadyMsgList, &info); stDebug("s-task:%s add checkpoint source rsp msg, total:%d", pTask->id.idStr, size + 1); } @@ -1217,7 +1230,7 @@ int32_t streamAddCheckpointSourceRspMsg(SStreamCheckpointSourceReq* pReq, SRpcHa } void initCheckpointReadyInfo(STaskCheckpointReadyInfo* pReadyInfo, int32_t upstreamNodeId, int32_t upstreamTaskId, - int32_t childId, SEpSet* pEpset, int64_t checkpointId) { + int32_t childId, SEpSet* pEpset, int64_t checkpointId) { pReadyInfo->upstreamTaskId = upstreamTaskId; pReadyInfo->upstreamNodeEpset = *pEpset; pReadyInfo->upstreamNodeId = upstreamNodeId; @@ -1246,7 +1259,7 @@ int32_t streamAddCheckpointReadyMsg(SStreamTask* pTask, int32_t upstreamTaskId, SActiveCheckpointInfo* pActiveInfo = pTask->chkInfo.pActiveInfo; streamMutexLock(&pActiveInfo->lock); - (void) taosArrayPush(pActiveInfo->pReadyMsgList, &info); + (void)taosArrayPush(pActiveInfo->pReadyMsgList, &info); int32_t numOfRecv = taosArrayGetSize(pActiveInfo->pReadyMsgList); int32_t total = streamTaskGetNumOfUpstream(pTask); @@ -1282,7 +1295,7 @@ static int32_t handleDispatchSuccessRsp(SStreamTask* pTask, int32_t downstreamId stDebug("s-task:%s destroy dispatch msg:%p", pTask->id.idStr, pTask->msgInfo.pData); int64_t el = taosGetTimestampMs() - pTask->msgInfo.startTs; - bool delayDispatch = (pTask->msgInfo.dispatchMsgType == STREAM_INPUT__CHECKPOINT_TRIGGER); + bool delayDispatch = (pTask->msgInfo.dispatchMsgType == STREAM_INPUT__CHECKPOINT_TRIGGER); clearBufferedDispatchMsg(pTask); diff --git a/source/libs/stream/src/streamExec.c b/source/libs/stream/src/streamExec.c index cd69c9168c..7cb800c756 100644 --- a/source/libs/stream/src/streamExec.c +++ b/source/libs/stream/src/streamExec.c @@ -18,10 +18,10 @@ // maximum allowed processed block batches. One block may include several submit blocks #define MAX_STREAM_EXEC_BATCH_NUM 32 #define STREAM_RESULT_DUMP_THRESHOLD 300 -#define STREAM_RESULT_DUMP_SIZE_THRESHOLD (1048576 * 1) // 1MiB result data -#define STREAM_SCAN_HISTORY_TIMESLICE 1000 // 1000 ms -#define MIN_INVOKE_INTERVAL 50 // 50ms -#define FILL_HISTORY_TASK_EXEC_INTERVAL 5000 // 5 sec +#define STREAM_RESULT_DUMP_SIZE_THRESHOLD (1048576 * 1) // 1MiB result data +#define STREAM_SCAN_HISTORY_TIMESLICE 1000 // 1000 ms +#define MIN_INVOKE_INTERVAL 50 // 50ms +#define FILL_HISTORY_TASK_EXEC_INTERVAL 5000 // 5 sec static int32_t streamTransferStateDoPrepare(SStreamTask* pTask); static void streamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pItem, int64_t* totalSize, int32_t* totalBlocks); @@ -54,7 +54,7 @@ static int32_t doOutputResultBlockImpl(SStreamTask* pTask, SStreamDataBlock* pBl // not handle error, if dispatch failed, try next time. // checkpoint trigger will be checked - (void) streamDispatchStreamBlock(pTask); + (void)streamDispatchStreamBlock(pTask); } return code; @@ -78,7 +78,7 @@ static int32_t doDumpResult(SStreamTask* pTask, SStreamQueueItem* pItem, SArray* } stDebug("s-task:%s dump stream result data blocks, num:%d, size:%.2fMiB", pTask->id.idStr, numOfBlocks, - SIZE_IN_MiB(size)); + SIZE_IN_MiB(size)); code = doOutputResultBlockImpl(pTask, pStreamBlocks); if (code != TSDB_CODE_SUCCESS) { // back pressure and record position @@ -99,7 +99,7 @@ void streamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pItem, int64_t* to *totalSize = 0; int32_t size = 0; - int32_t numOfBlocks= 0; + int32_t numOfBlocks = 0; SArray* pRes = NULL; while (1) { @@ -129,7 +129,7 @@ void streamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pItem, int64_t* to const SStreamDataBlock* pRetrieveBlock = (const SStreamDataBlock*)pItem; ASSERT(taosArrayGetSize(pRetrieveBlock->blocks) == 1); - (void) assignOneDataBlock(&block, taosArrayGet(pRetrieveBlock->blocks, 0)); + (void)assignOneDataBlock(&block, taosArrayGet(pRetrieveBlock->blocks, 0)); block.info.type = STREAM_PULL_OVER; block.info.childId = pTask->info.selfChildId; @@ -140,8 +140,8 @@ void streamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pItem, int64_t* to stError("s-task:%s failed to add retrieve block", pTask->id.idStr); } - stDebug("s-task:%s(child %d) retrieve process completed, reqId:0x%" PRIx64 " dump results", pTask->id.idStr, - pTask->info.selfChildId, pRetrieveBlock->reqId); + stDebug("s-task:%s(child %d) retrieve process completed, qid:0x%" PRIx64 " dump results", pTask->id.idStr, + pTask->info.selfChildId, pRetrieveBlock->reqId); } break; @@ -174,7 +174,7 @@ void streamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pItem, int64_t* to } stDebug("s-task:%s (child %d) executed and get %d result blocks, size:%.2fMiB", pTask->id.idStr, - pTask->info.selfChildId, numOfBlocks, SIZE_IN_MiB(size)); + pTask->info.selfChildId, numOfBlocks, SIZE_IN_MiB(size)); // current output should be dispatched to down stream nodes if (numOfBlocks >= STREAM_RESULT_DUMP_THRESHOLD || size >= STREAM_RESULT_DUMP_SIZE_THRESHOLD) { @@ -251,7 +251,7 @@ static void streamScanHistoryDataImpl(SStreamTask* pTask, SArray* pRes, int32_t* } SSDataBlock block = {0}; - (void) assignOneDataBlock(&block, output); + (void)assignOneDataBlock(&block, output); block.info.childId = pTask->info.selfChildId; void* p = taosArrayPush(pRes, &block); @@ -259,7 +259,8 @@ static void streamScanHistoryDataImpl(SStreamTask* pTask, SArray* pRes, int32_t* stError("s-task:%s failed to add computing results, the final res may be incorrect", pTask->id.idStr); } - (*pSize) += blockDataGetSize(output) + sizeof(SSDataBlock) + sizeof(SColumnInfoData) * blockDataGetNumOfCols(&block); + (*pSize) += + blockDataGetSize(output) + sizeof(SSDataBlock) + sizeof(SColumnInfoData) * blockDataGetNumOfCols(&block); numOfBlocks += 1; if (numOfBlocks >= STREAM_RESULT_DUMP_THRESHOLD || (*pSize) >= STREAM_RESULT_DUMP_SIZE_THRESHOLD) { @@ -283,7 +284,7 @@ SScanhistoryDataInfo streamScanHistoryData(SStreamTask* pTask, int64_t st) { const char* id = pTask->id.idStr; if (!pTask->hTaskInfo.operatorOpen) { - (void) qSetStreamOpOpen(exec); + (void)qSetStreamOpOpen(exec); pTask->hTaskInfo.operatorOpen = true; } @@ -315,13 +316,13 @@ SScanhistoryDataInfo streamScanHistoryData(SStreamTask* pTask, int64_t st) { int32_t size = 0; streamScanHistoryDataImpl(pTask, pRes, &size, &finished); - if(streamTaskShouldStop(pTask)) { + if (streamTaskShouldStop(pTask)) { taosArrayDestroyEx(pRes, (FDelete)blockDataFreeRes); return buildScanhistoryExecRet(TASK_SCANHISTORY_QUIT, 0); } // dispatch the generated results, todo fix error - (void) handleScanhistoryResultBlocks(pTask, pRes, size); + (void)handleScanhistoryResultBlocks(pTask, pRes, size); if (finished) { return buildScanhistoryExecRet(TASK_SCANHISTORY_CONT, 0); @@ -346,11 +347,11 @@ int32_t streamTransferStateDoPrepare(SStreamTask* pTask) { stError( "s-task:%s failed to find related stream task:0x%x, it may have been destroyed or closed, destroy the related " "fill-history task", - id, (int32_t) pTask->streamTaskId.taskId); + id, (int32_t)pTask->streamTaskId.taskId); // 1. free it and remove fill-history task from disk meta-store // todo: this function should never be failed. - (void) streamBuildAndSendDropTaskMsg(pTask->pMsgCb, pMeta->vgId, &pTask->id, 0); + (void)streamBuildAndSendDropTaskMsg(pTask->pMsgCb, pMeta->vgId, &pTask->id, 0); // 2. save to disk streamMetaWLock(pMeta); @@ -367,7 +368,7 @@ int32_t streamTransferStateDoPrepare(SStreamTask* pTask) { id, streamTaskGetStatus(pTask).name, el, pStreamTask->id.idStr); } - ETaskStatus status = streamTaskGetStatus(pStreamTask).state; + ETaskStatus status = streamTaskGetStatus(pStreamTask).state; STimeWindow* pTimeWindow = &pStreamTask->dataRange.window; // It must be halted for a source stream task, since when the related scan-history-data task start scan the history @@ -408,14 +409,14 @@ int32_t streamTransferStateDoPrepare(SStreamTask* pTask) { pStreamTask->id.idStr, TASK_LEVEL__SOURCE, pTimeWindow->skey, pTimeWindow->ekey, INT64_MIN, pTimeWindow->ekey, p, pStreamTask->status.schedStatus); - (void) streamTaskResetTimewindowFilter(pStreamTask); + (void)streamTaskResetTimewindowFilter(pStreamTask); } else { stDebug("s-task:%s no need to update/reset filter time window for non-source tasks", pStreamTask->id.idStr); } // NOTE: transfer the ownership of executor state before handle the checkpoint block during stream exec // 2. send msg to mnode to launch a checkpoint to keep the state for current stream - (void) streamTaskSendCheckpointReq(pStreamTask); + (void)streamTaskSendCheckpointReq(pStreamTask); // 3. assign the status to the value that will be kept in disk pStreamTask->status.taskStatus = streamTaskGetStatus(pStreamTask).state; @@ -429,12 +430,12 @@ int32_t streamTransferStateDoPrepare(SStreamTask* pTask) { static int32_t haltCallback(SStreamTask* pTask, void* param) { streamTaskOpenAllUpstreamInput(pTask); - (void) streamTaskSendCheckpointReq(pTask); + (void)streamTaskSendCheckpointReq(pTask); return TSDB_CODE_SUCCESS; } int32_t streamTransferStatePrepare(SStreamTask* pTask) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; SStreamMeta* pMeta = pTask->pMeta; ASSERT(pTask->status.appendTranstateBlock == 1); @@ -466,7 +467,7 @@ int32_t streamTransferStatePrepare(SStreamTask* pTask) { // set input static int32_t doSetStreamInputBlock(SStreamTask* pTask, const void* pInput, int64_t* pVer, const char* id) { - void* pExecutor = pTask->exec.pExecutor; + void* pExecutor = pTask->exec.pExecutor; int32_t code = 0; const SStreamQueueItem* pItem = pInput; @@ -479,7 +480,7 @@ static int32_t doSetStreamInputBlock(SStreamTask* pTask, const void* pInput, int const SStreamDataSubmit* pSubmit = (const SStreamDataSubmit*)pInput; code = qSetMultiStreamInput(pExecutor, &pSubmit->submit, 1, STREAM_INPUT__DATA_SUBMIT); stDebug("s-task:%s set submit blocks as source block completed, %p %p len:%d ver:%" PRId64, id, pSubmit, - pSubmit->submit.msgStr, pSubmit->submit.msgLen, pSubmit->submit.ver); + pSubmit->submit.msgStr, pSubmit->submit.msgLen, pSubmit->submit.ver); ASSERT((*pVer) <= pSubmit->submit.ver); (*pVer) = pSubmit->submit.ver; @@ -497,7 +498,7 @@ static int32_t doSetStreamInputBlock(SStreamTask* pTask, const void* pInput, int SArray* pBlockList = pMerged->submits; int32_t numOfBlocks = taosArrayGetSize(pBlockList); stDebug("s-task:%s %p set (merged) submit blocks as a batch, numOfBlocks:%d, ver:%" PRId64, id, pTask, numOfBlocks, - pMerged->ver); + pMerged->ver); code = qSetMultiStreamInput(pExecutor, pBlockList->pData, numOfBlocks, STREAM_INPUT__MERGED_SUBMIT); ASSERT((*pVer) <= pMerged->ver); (*pVer) = pMerged->ver; @@ -549,7 +550,7 @@ void streamProcessTransstateBlock(SStreamTask* pTask, SStreamDataBlock* pBlock) pBlock->srcVgId = pTask->pMeta->vgId; code = taosWriteQitem(pTask->outputq.queue->pQueue, pBlock); if (code == 0) { - (void) streamDispatchStreamBlock(pTask); + (void)streamDispatchStreamBlock(pTask); } else { // todo put into queue failed, retry streamFreeQitem((SStreamQueueItem*)pBlock); } @@ -568,7 +569,7 @@ void streamProcessTransstateBlock(SStreamTask* pTask, SStreamDataBlock* pBlock) } } -//static void streamTaskSetIdleInfo(SStreamTask* pTask, int32_t idleTime) { pTask->status.schedIdleTime = idleTime; } +// static void streamTaskSetIdleInfo(SStreamTask* pTask, int32_t idleTime) { pTask->status.schedIdleTime = idleTime; } static void setLastExecTs(SStreamTask* pTask, int64_t ts) { pTask->status.lastExecTs = ts; } static void doStreamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pBlock, int32_t num) { @@ -581,7 +582,7 @@ static void doStreamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pBlock, i stDebug("s-task:%s start to process batch blocks, num:%d, type:%s", id, num, streamQueueItemGetTypeStr(pBlock->type)); int32_t code = doSetStreamInputBlock(pTask, pBlock, &ver, id); - if(code) { + if (code) { stError("s-task:%s failed to set input block, not exec for these blocks", id); return; } @@ -609,7 +610,7 @@ static void doStreamTaskExecImpl(SStreamTask* pTask, SStreamQueueItem* pBlock, i if (ver != pInfo->processedVer) { stDebug("s-task:%s update processedVer(unsaved) from %" PRId64 " to %" PRId64 " nextProcessVer:%" PRId64 - " ckpt:%" PRId64, + " ckpt:%" PRId64, id, pInfo->processedVer, ver, pInfo->nextProcessVer, pInfo->checkpointVer); pInfo->processedVer = ver; } @@ -625,10 +626,10 @@ void flushStateDataInExecutor(SStreamTask* pTask, SStreamQueueItem* pCheckpointB STaskId* pHTaskId = &pTask->hTaskInfo.id; SStreamTask* pHTask = NULL; - int32_t code = streamMetaAcquireTask(pTask->pMeta, pHTaskId->streamId, pHTaskId->taskId, &pHTask); - if (code == TSDB_CODE_SUCCESS) { // ignore the error code. - (void) streamTaskReleaseState(pHTask); - (void) streamTaskReloadState(pTask); + int32_t code = streamMetaAcquireTask(pTask->pMeta, pHTaskId->streamId, pHTaskId->taskId, &pHTask); + if (code == TSDB_CODE_SUCCESS) { // ignore the error code. + (void)streamTaskReleaseState(pHTask); + (void)streamTaskReloadState(pTask); stDebug("s-task:%s transfer state from fill-history task:%s, status:%s completed", id, pHTask->id.idStr, streamTaskGetStatus(pHTask).name); // todo execute qExecTask to fetch the reload-generated result, if this is stream is for session window query. @@ -707,7 +708,7 @@ static int32_t doStreamExecTask(SStreamTask* pTask) { // dispatch checkpoint msg to all downstream tasks int32_t type = pInput->type; if (type == STREAM_INPUT__CHECKPOINT_TRIGGER) { - (void) streamProcessCheckpointTriggerBlock(pTask, (SStreamDataBlock*)pInput); + (void)streamProcessCheckpointTriggerBlock(pTask, (SStreamDataBlock*)pInput); continue; } @@ -744,14 +745,14 @@ static int32_t doStreamExecTask(SStreamTask* pTask) { if (type != STREAM_INPUT__CHECKPOINT) { doStreamTaskExecImpl(pTask, pInput, numOfBlocks); streamFreeQitem(pInput); - } else { // todo other thread may change the status - // do nothing after sync executor state to storage backend, untill the vnode-level checkpoint is completed. + } else { // todo other thread may change the status + // do nothing after sync executor state to storage backend, untill the vnode-level checkpoint is completed. streamMutexLock(&pTask->lock); SStreamTaskState pState = streamTaskGetStatus(pTask); if (pState.state == TASK_STATUS__CK) { stDebug("s-task:%s checkpoint block received, set status:%s", id, pState.name); - (void) streamTaskBuildCheckpoint(pTask); // ignore this error msg, and continue - } else { // todo refactor + (void)streamTaskBuildCheckpoint(pTask); // ignore this error msg, and continue + } else { // todo refactor int32_t code = 0; if (pTask->info.taskLevel == TASK_LEVEL__SOURCE) { code = streamTaskSendCheckpointSourceRsp(pTask); @@ -804,7 +805,7 @@ void streamResumeTask(SStreamTask* pTask) { const char* id = pTask->id.idStr; while (1) { - (void) doStreamExecTask(pTask); + (void)doStreamExecTask(pTask); // check if continue streamMutexLock(&pTask->lock); diff --git a/source/libs/stream/src/streamTask.c b/source/libs/stream/src/streamTask.c index b99c9f5f65..a184314714 100644 --- a/source/libs/stream/src/streamTask.c +++ b/source/libs/stream/src/streamTask.c @@ -1092,7 +1092,7 @@ static int32_t streamTaskEnqueueRetrieve(SStreamTask* pTask, SStreamRetrieveReq* } // enqueue - stDebug("s-task:%s (vgId:%d level:%d) recv retrieve req from task:0x%x(vgId:%d), reqId:0x%" PRIx64, pTask->id.idStr, + stDebug("s-task:%s (vgId:%d level:%d) recv retrieve req from task:0x%x(vgId:%d), qid:0x%" PRIx64, pTask->id.idStr, pTask->pMeta->vgId, pTask->info.taskLevel, pReq->srcTaskId, pReq->srcNodeId, pReq->reqId); pData->type = STREAM_INPUT__DATA_RETRIEVE; diff --git a/source/libs/sync/src/syncAppendEntriesReply.c b/source/libs/sync/src/syncAppendEntriesReply.c index 6ea5402c28..005cf4337d 100644 --- a/source/libs/sync/src/syncAppendEntriesReply.c +++ b/source/libs/sync/src/syncAppendEntriesReply.c @@ -57,14 +57,12 @@ int32_t syncNodeOnAppendEntriesReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) { } if (ths->state == TAOS_SYNC_STATE_LEADER || ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) { - if (pMsg->term > raftStoreGetTerm(ths)) { + if (pMsg->term != raftStoreGetTerm(ths)) { syncLogRecvAppendEntriesReply(ths, pMsg, "error term"); syncNodeStepDown(ths, pMsg->term); return TSDB_CODE_SYN_WRONG_TERM; } - ASSERT(pMsg->term == raftStoreGetTerm(ths)); - sTrace("vgId:%d, received append entries reply. srcId:0x%016" PRIx64 ", term:%" PRId64 ", matchIndex:%" PRId64 "", pMsg->vgId, pMsg->srcId.addr, pMsg->term, pMsg->matchIndex); diff --git a/source/libs/sync/src/syncCommit.c b/source/libs/sync/src/syncCommit.c index 3e2b98c35b..5054339e8e 100644 --- a/source/libs/sync/src/syncCommit.c +++ b/source/libs/sync/src/syncCommit.c @@ -45,9 +45,6 @@ // static inline int64_t syncNodeAbs64(int64_t a, int64_t b) { - ASSERT(a >= 0); - ASSERT(b >= 0); - int64_t c = a > b ? a - b : b - a; return c; } @@ -57,7 +54,10 @@ int32_t syncNodeDynamicQuorum(const SSyncNode* pSyncNode) { return pSyncNode->qu bool syncNodeAgreedUpon(SSyncNode* pNode, SyncIndex index) { int count = 0; SSyncIndexMgr* pMatches = pNode->pMatchIndex; - ASSERT(pNode->replicaNum == pMatches->replicaNum); + if (pNode->replicaNum != pMatches->replicaNum) { + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return false; + }; for (int i = 0; i < pNode->totalReplicaNum; i++) { if(pNode->raftCfg.cfg.nodeInfo[i].nodeRole == TAOS_SYNC_ROLE_VOTER){ diff --git a/source/libs/sync/src/syncElection.c b/source/libs/sync/src/syncElection.c index 3b595f464d..901b43023a 100644 --- a/source/libs/sync/src/syncElection.c +++ b/source/libs/sync/src/syncElection.c @@ -100,7 +100,11 @@ int32_t syncNodeElect(SSyncNode* pSyncNode) { if (voteGrantedMajority(pSyncNode->pVotesGranted)) { // only myself, to leader - ASSERT(!pSyncNode->pVotesGranted->toLeader); + if (pSyncNode->pVotesGranted->toLeader) { + ret = TSDB_CODE_SYN_INTERNAL_ERROR; + sError("vgId:%d, failed to elect since already be to leader", pSyncNode->vgId); + return ret; + } syncNodeCandidate2Leader(pSyncNode); pSyncNode->pVotesGranted->toLeader = true; return ret; @@ -119,7 +123,7 @@ int32_t syncNodeElect(SSyncNode* pSyncNode) { } ret = syncNodeRequestVotePeers(pSyncNode); - ASSERT(ret == 0); + if (ret != 0) return ret; syncNodeResetElectTimer(pSyncNode); return ret; diff --git a/source/libs/sync/src/syncMain.c b/source/libs/sync/src/syncMain.c index ffd180ee01..215c8f9e58 100644 --- a/source/libs/sync/src/syncMain.c +++ b/source/libs/sync/src/syncMain.c @@ -269,8 +269,8 @@ int32_t syncProcessMsg(int64_t rid, SRpcMsg* pMsg) { syncNodeRelease(pSyncNode); if (code != 0) { - sDebug("vgId:%d, failed to process sync msg:%p type:%s since 0x%x", pSyncNode->vgId, pMsg, TMSG_INFO(pMsg->msgType), - code); + sDebug("vgId:%d, failed to process sync msg:%p type:%s since %s", pSyncNode->vgId, pMsg, TMSG_INFO(pMsg->msgType), + tstrerror(code)); } TAOS_RETURN(code); } @@ -726,7 +726,12 @@ int32_t syncUpdateArbTerm(int64_t rid, SyncTerm arbTerm) { } SyncIndex syncNodeGetSnapshotConfigIndex(SSyncNode* pSyncNode, SyncIndex snapshotLastApplyIndex) { - ASSERT(pSyncNode->raftCfg.configIndexCount >= 1); + if (!(pSyncNode->raftCfg.configIndexCount >= 1)) { + sError("vgId:%d, failed get snapshot config index, configIndexCount:%d", pSyncNode->vgId, + pSyncNode->raftCfg.configIndexCount); + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return -2; + } SyncIndex lastIndex = (pSyncNode->raftCfg.configIndexArr)[0]; for (int32_t i = 0; i < pSyncNode->raftCfg.configIndexCount; ++i) { @@ -988,9 +993,18 @@ static int32_t syncHbTimerStop(SSyncNode* pSyncNode, SSyncTimer* pSyncTimer) { int32_t syncNodeLogStoreRestoreOnNeed(SSyncNode* pNode) { int32_t code = 0; - ASSERTS(pNode->pLogStore != NULL, "log store not created"); - ASSERTS(pNode->pFsm != NULL, "pFsm not registered"); - ASSERTS(pNode->pFsm->FpGetSnapshotInfo != NULL, "FpGetSnapshotInfo not registered"); + if (pNode->pLogStore == NULL) { + sError("vgId:%d, log store not created", pNode->vgId); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pNode->pFsm == NULL) { + sError("vgId:%d, pFsm not registered", pNode->vgId); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pNode->pFsm->FpGetSnapshotInfo == NULL) { + sError("vgId:%d, FpGetSnapshotInfo not registered", pNode->vgId); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } SSnapshot snapshot = {0}; // TODO check return value (void)pNode->pFsm->FpGetSnapshotInfo(pNode->pFsm, &snapshot); @@ -1384,8 +1398,14 @@ void syncNodeMaybeUpdateCommitBySnapshot(SSyncNode* pSyncNode) { int32_t syncNodeRestore(SSyncNode* pSyncNode) { int32_t code = 0; - ASSERTS(pSyncNode->pLogStore != NULL, "log store not created"); - ASSERTS(pSyncNode->pLogBuf != NULL, "ring log buffer not created"); + if (pSyncNode->pLogStore == NULL) { + sError("vgId:%d, log store not created", pSyncNode->vgId); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pSyncNode->pLogBuf == NULL) { + sError("vgId:%d, ring log buffer not created", pSyncNode->vgId); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } (void)taosThreadMutexLock(&pSyncNode->pLogBuf->mutex); SyncIndex lastVer = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore); @@ -1400,7 +1420,7 @@ int32_t syncNodeRestore(SSyncNode* pSyncNode) { TAOS_RETURN(code); } - ASSERT(endIndex == lastVer + 1); + if (endIndex != lastVer + 1) return TSDB_CODE_SYN_INTERNAL_ERROR; pSyncNode->commitIndex = TMAX(pSyncNode->commitIndex, commitIndex); sInfo("vgId:%d, restore sync until commitIndex:%" PRId64, pSyncNode->vgId, pSyncNode->commitIndex); @@ -1462,9 +1482,18 @@ int32_t syncNodeStartStandBy(SSyncNode* pSyncNode) { #endif void syncNodePreClose(SSyncNode* pSyncNode) { - ASSERT(pSyncNode != NULL); - ASSERT(pSyncNode->pFsm != NULL); - ASSERT(pSyncNode->pFsm->FpApplyQueueItems != NULL); + if (pSyncNode == NULL) { + sError("failed to pre close sync node since sync node is null"); + return; + } + if (pSyncNode->pFsm == NULL) { + sError("failed to pre close sync node since fsm is null"); + return; + } + if (pSyncNode->pFsm->FpApplyQueueItems == NULL) { + sError("failed to pre close sync node since FpApplyQueueItems is null"); + return; + } // stop elect timer (void)syncNodeStopElectTimer(pSyncNode); @@ -1743,7 +1772,10 @@ inline bool syncNodeInConfig(SSyncNode* pNode, const SSyncCfg* pCfg) { } } - ASSERT(b1 == b2); + if (b1 != b2) { + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return false; + } return b1; } @@ -2062,7 +2094,10 @@ void syncNodeBecomeLeader(SSyncNode* pSyncNode, const char* debugStr) { SyncIndex lastIndex; SyncTerm lastTerm; int32_t code = syncNodeGetLastIndexTerm(pSyncNode, &lastIndex, &lastTerm); - ASSERT(code == 0); + if (code != 0) { + sError("vgId:%d, failed to become leader since %s", pSyncNode->vgId, tstrerror(code)); + return; + } pSyncNode->pNextIndex->index[i] = lastIndex + 1; } @@ -2135,7 +2170,10 @@ void syncNodeBecomeAssignedLeader(SSyncNode* pSyncNode) { SyncIndex lastIndex; SyncTerm lastTerm; int32_t code = syncNodeGetLastIndexTerm(pSyncNode, &lastIndex, &lastTerm); - ASSERT(code == 0); + if (code != 0) { + sError("vgId:%d, failed to become assigned leader since %s", pSyncNode->vgId, tstrerror(code)); + return; + } pSyncNode->pNextIndex->index[i] = lastIndex + 1; } @@ -2178,7 +2216,10 @@ void syncNodeBecomeAssignedLeader(SSyncNode* pSyncNode) { } void syncNodeCandidate2Leader(SSyncNode* pSyncNode) { - ASSERT(pSyncNode->state == TAOS_SYNC_STATE_CANDIDATE); + if (pSyncNode->state != TAOS_SYNC_STATE_CANDIDATE) { + sError("vgId:%d, failed leader from candidate since node state is wrong:%d", pSyncNode->vgId, pSyncNode->state); + return; + } bool granted = voteGrantedMajority(pSyncNode->pVotesGranted); if (!granted) { sError("vgId:%d, not granted by majority.", pSyncNode->vgId); @@ -2194,7 +2235,7 @@ void syncNodeCandidate2Leader(SSyncNode* pSyncNode) { } SyncIndex lastIndex = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore); - // ASSERT(lastIndex >= 0); + sInfo("vgId:%d, become leader. term:%" PRId64 ", commit index:%" PRId64 ", last index:%" PRId64 "", pSyncNode->vgId, raftStoreGetTerm(pSyncNode), pSyncNode->commitIndex, lastIndex); } @@ -2211,7 +2252,10 @@ int32_t syncNodePeerStateInit(SSyncNode* pSyncNode) { } void syncNodeFollower2Candidate(SSyncNode* pSyncNode) { - ASSERT(pSyncNode->state == TAOS_SYNC_STATE_FOLLOWER); + if (pSyncNode->state != TAOS_SYNC_STATE_FOLLOWER) { + sError("vgId:%d, failed candidate from follower since node state is wrong:%d", pSyncNode->vgId, pSyncNode->state); + return; + } pSyncNode->state = TAOS_SYNC_STATE_CANDIDATE; pSyncNode->roleTimeMs = taosGetTimestampMs(); SyncIndex lastIndex = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore); @@ -2222,7 +2266,7 @@ void syncNodeFollower2Candidate(SSyncNode* pSyncNode) { } int32_t syncNodeAssignedLeader2Leader(SSyncNode* pSyncNode) { - ASSERT(pSyncNode->state == TAOS_SYNC_STATE_ASSIGNED_LEADER); + if (pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) return TSDB_CODE_SYN_INTERNAL_ERROR; syncNodeBecomeLeader(pSyncNode, "assigned leader to leader"); sNTrace(pSyncNode, "assigned leader to leader"); @@ -2241,11 +2285,17 @@ int32_t syncNodeAssignedLeader2Leader(SSyncNode* pSyncNode) { } // just called by syncNodeVoteForSelf -// need assert void syncNodeVoteForTerm(SSyncNode* pSyncNode, SyncTerm term, SRaftId* pRaftId) { - ASSERT(term == raftStoreGetTerm(pSyncNode)); + SyncTerm storeTerm = raftStoreGetTerm(pSyncNode); + if (term != storeTerm) { + sError("vgId:%d, failed to vote for term, term:%" PRId64 ", storeTerm:%" PRId64, pSyncNode->vgId, term, storeTerm); + return; + } bool voted = raftStoreHasVoted(pSyncNode); - ASSERT(!voted); + if (voted) { + sError("vgId:%d, failed to vote for term since not voted", pSyncNode->vgId); + return; + } raftStoreVote(pSyncNode, pRaftId); } @@ -2389,7 +2439,7 @@ SyncTerm syncNodeGetPreTerm(SSyncNode* pSyncNode, SyncIndex index) { .lastConfigIndex = SYNC_INDEX_INVALID}; if (code == 0) { - ASSERT(pPreEntry != NULL); + if (pPreEntry == NULL) return -1; preTerm = pPreEntry->term; if (h) { @@ -3220,7 +3270,6 @@ int32_t syncNodeAppend(SSyncNode* ths, SSyncRaftEntry* pEntry) { // append to log buffer if ((code = syncLogBufferAppend(ths->pLogBuf, ths, pEntry)) < 0) { sError("vgId:%d, failed to enqueue sync log buffer, index:%" PRId64, ths->vgId, pEntry->index); - ASSERT(terrno != 0); (void)syncFsmExecute(ths, ths->pFsm, ths->state, raftStoreGetTerm(ths), pEntry, terrno, false); syncEntryDestroy(pEntry); goto _out; @@ -3329,7 +3378,7 @@ static int32_t syncNodeAppendNoopOld(SSyncNode* ths) { SyncIndex index = ths->pLogStore->syncLogWriteIndex(ths->pLogStore); SyncTerm term = raftStoreGetTerm(ths); SSyncRaftEntry* pEntry = syncEntryBuildNoop(term, index, ths->vgId); - ASSERT(pEntry != NULL); + if (pEntry == NULL) return -1; LRUHandle* h = NULL; @@ -3505,6 +3554,9 @@ int32_t syncNodeOnLocalCmd(SSyncNode* ths, const SRpcMsg* pRpcMsg) { return 0; } SyncTerm matchTerm = syncLogBufferGetLastMatchTerm(ths->pLogBuf); + if (matchTerm < 0) { + return TSDB_CODE_SYN_INTERNAL_ERROR; + } if (pMsg->currentTerm == matchTerm) { (void)syncNodeUpdateCommitIndex(ths, pMsg->commitIndex); } diff --git a/source/libs/sync/src/syncPipeline.c b/source/libs/sync/src/syncPipeline.c index 782d97f789..a6e9c7de32 100644 --- a/source/libs/sync/src/syncPipeline.c +++ b/source/libs/sync/src/syncPipeline.c @@ -46,8 +46,8 @@ int64_t syncLogBufferGetEndIndex(SSyncLogBuffer* pBuf) { int32_t syncLogBufferAppend(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEntry* pEntry) { int32_t code = 0; + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); (void)taosThreadMutexLock(&pBuf->mutex); - (void)syncLogBufferValidate(pBuf); SyncIndex index = pEntry->index; if (index - pBuf->startIndex >= pBuf->size) { @@ -71,28 +71,44 @@ int32_t syncLogBufferAppend(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEnt goto _err; } - ASSERT(index == pBuf->endIndex); + if (index != pBuf->endIndex) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _err; + }; SSyncRaftEntry* pExist = pBuf->entries[index % pBuf->size].pItem; - ASSERT(pExist == NULL); + if (pExist != NULL) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _err; + } // initial log buffer with at least one item, e.g. commitIndex SSyncRaftEntry* pMatch = pBuf->entries[(index - 1 + pBuf->size) % pBuf->size].pItem; - ASSERTS(pMatch != NULL, "no matched log entry"); - ASSERT(pMatch->index + 1 == index); - ASSERT(pMatch->term <= pEntry->term); + if (pMatch == NULL) { + sError("vgId:%d, no matched log entry", pNode->vgId); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _err; + } + if (pMatch->index + 1 != index) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _err; + } + if (!(pMatch->term <= pEntry->term)) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _err; + } SSyncLogBufEntry tmp = {.pItem = pEntry, .prevLogIndex = pMatch->index, .prevLogTerm = pMatch->term}; pBuf->entries[index % pBuf->size] = tmp; pBuf->endIndex = index + 1; - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); return 0; _err: - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); taosMsleep(1); TAOS_RETURN(code); } @@ -114,11 +130,15 @@ int32_t syncLogReplGetPrevLogTerm(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncI TAOS_RETURN(TSDB_CODE_WAL_LOG_NOT_EXIST); } - ASSERT(index - 1 == prevIndex); + if (index - 1 != prevIndex) return TSDB_CODE_SYN_INTERNAL_ERROR; if (prevIndex >= pBuf->startIndex) { pEntry = pBuf->entries[(prevIndex + pBuf->size) % pBuf->size].pItem; - ASSERTS(pEntry != NULL, "no log entry found"); + if (pEntry == NULL) { + sError("vgId:%d, failed to get pre log term since no log entry found", pNode->vgId); + *pSyncTerm = -1; + TAOS_RETURN(TSDB_CODE_SYN_INTERNAL_ERROR); + } prevLogTerm = pEntry->term; *pSyncTerm = prevLogTerm; return 0; @@ -126,9 +146,18 @@ int32_t syncLogReplGetPrevLogTerm(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncI if (pMgr && pMgr->startIndex <= prevIndex && prevIndex < pMgr->endIndex) { int64_t timeMs = pMgr->states[(prevIndex + pMgr->size) % pMgr->size].timeMs; - ASSERTS(timeMs != 0, "no log entry found"); + if (timeMs == 0) { + sError("vgId:%d, failed to get pre log term since timeMs is 0", pNode->vgId); + *pSyncTerm = -1; + TAOS_RETURN(TSDB_CODE_SYN_INTERNAL_ERROR); + } prevLogTerm = pMgr->states[(prevIndex + pMgr->size) % pMgr->size].term; - ASSERT(prevIndex == 0 || prevLogTerm != 0); + if (!(prevIndex == 0 || prevLogTerm != 0)) { + sError("vgId:%d, failed to get pre log term prevIndex:%" PRId64 ", prevLogTerm:%" PRId64, pNode->vgId, prevIndex, + prevLogTerm); + *pSyncTerm = -1; + TAOS_RETURN(TSDB_CODE_SYN_INTERNAL_ERROR); + } *pSyncTerm = prevLogTerm; return 0; } @@ -178,9 +207,18 @@ int32_t syncLogValidateAlignmentOfCommit(SSyncNode* pNode, SyncIndex commitIndex } int32_t syncLogBufferInitWithoutLock(SSyncLogBuffer* pBuf, SSyncNode* pNode) { - ASSERTS(pNode->pLogStore != NULL, "log store not created"); - ASSERTS(pNode->pFsm != NULL, "pFsm not registered"); - ASSERTS(pNode->pFsm->FpGetSnapshotInfo != NULL, "FpGetSnapshotInfo not registered"); + if (pNode->pLogStore == NULL) { + sError("log store not created"); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pNode->pFsm == NULL) { + sError("pFsm not registered"); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pNode->pFsm->FpGetSnapshotInfo == NULL) { + sError("FpGetSnapshotInfo not registered"); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } int32_t code = 0, lino = 0; SSnapshot snapshot = {0}; @@ -191,7 +229,8 @@ int32_t syncLogBufferInitWithoutLock(SSyncLogBuffer* pBuf, SSyncNode* pNode) { TAOS_CHECK_EXIT(syncLogValidateAlignmentOfCommit(pNode, commitIndex)); SyncIndex lastVer = pNode->pLogStore->syncLogLastIndex(pNode->pLogStore); - ASSERT(lastVer >= commitIndex); + if (lastVer < commitIndex) return TSDB_CODE_SYN_INTERNAL_ERROR; + ; SyncIndex toIndex = lastVer; // update match index pBuf->commitIndex = commitIndex; @@ -239,7 +278,7 @@ int32_t syncLogBufferInitWithoutLock(SSyncLogBuffer* pBuf, SSyncNode* pNode) { // put a dummy record at commitIndex if present in log buffer if (takeDummy) { - ASSERT(index == pBuf->commitIndex); + if (index != pBuf->commitIndex) return TSDB_CODE_SYN_INTERNAL_ERROR; SSyncRaftEntry* pDummy = syncEntryBuildDummy(commitTerm, commitIndex, pNode->vgId); if (pDummy == NULL) { @@ -263,7 +302,7 @@ int32_t syncLogBufferInitWithoutLock(SSyncLogBuffer* pBuf, SSyncNode* pNode) { pBuf->startIndex, pBuf->commitIndex, pBuf->matchIndex, pBuf->endIndex); // validate - (void)syncLogBufferValidate(pBuf); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); return 0; _exit: @@ -281,8 +320,8 @@ int32_t syncLogBufferInit(SSyncLogBuffer* pBuf, SSyncNode* pNode) { } int32_t syncLogBufferReInit(SSyncLogBuffer* pBuf, SSyncNode* pNode) { + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); (void)taosThreadMutexLock(&pBuf->mutex); - (void)syncLogBufferValidate(pBuf); for (SyncIndex index = pBuf->startIndex; index < pBuf->endIndex; index++) { SSyncRaftEntry* pEntry = pBuf->entries[(index + pBuf->size) % pBuf->size].pItem; if (pEntry == NULL) continue; @@ -295,15 +334,19 @@ int32_t syncLogBufferReInit(SSyncLogBuffer* pBuf, SSyncNode* pNode) { if (code < 0) { sError("vgId:%d, failed to re-initialize sync log buffer since %s.", pNode->vgId, tstrerror(code)); } - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); return code; } FORCE_INLINE SyncTerm syncLogBufferGetLastMatchTermWithoutLock(SSyncLogBuffer* pBuf) { SyncIndex index = pBuf->matchIndex; SSyncRaftEntry* pEntry = pBuf->entries[(index + pBuf->size) % pBuf->size].pItem; - ASSERT(pEntry != NULL); + if (pEntry == NULL) { + sError("failed to get last match term since entry is null"); + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return -1; + } return pEntry->term; } @@ -322,8 +365,8 @@ bool syncLogBufferIsEmpty(SSyncLogBuffer* pBuf) { } int32_t syncLogBufferAccept(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEntry* pEntry, SyncTerm prevTerm) { + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); (void)taosThreadMutexLock(&pBuf->mutex); - (void)syncLogBufferValidate(pBuf); int32_t code = 0; SyncIndex index = pEntry->index; SyncIndex prevIndex = pEntry->index - 1; @@ -331,6 +374,12 @@ int32_t syncLogBufferAccept(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEnt SSyncRaftEntry* pExist = NULL; bool inBuf = true; + if (lastMatchTerm < 0) { + sError("vgId:%d, failed to accept, lastMatchTerm:%" PRId64, pNode->vgId, lastMatchTerm); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } + if (index <= pBuf->commitIndex) { sTrace("vgId:%d, already committed. index:%" PRId64 ", term:%" PRId64 ". log buffer: [%" PRId64 " %" PRId64 " %" PRId64 ", %" PRId64 ")", @@ -338,7 +387,11 @@ int32_t syncLogBufferAccept(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEnt pBuf->endIndex); SyncTerm term = -1; code = syncLogReplGetPrevLogTerm(NULL, pNode, index + 1, &term); - ASSERT(pEntry->term >= 0); + if (pEntry->term < 0) { + sError("vgId:%d, failed to accept, pEntry->term:%" PRId64, pNode->vgId, pEntry->term); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } if (term == pEntry->term) { code = 0; } @@ -375,7 +428,12 @@ int32_t syncLogBufferAccept(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEnt // check current in buffer code = syncLogBufferGetOneEntry(pBuf, pNode, index, &inBuf, &pExist); if (pExist != NULL) { - ASSERT(pEntry->index == pExist->index); + if (pEntry->index != pExist->index) { + sError("vgId:%d, failed to accept, pEntry->index:%" PRId64 ", pExist->index:%" PRId64, pNode->vgId, pEntry->index, + pExist->index); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } if (pEntry->term != pExist->term) { (void)syncLogBufferRollback(pBuf, pNode, index); } else { @@ -385,16 +443,32 @@ int32_t syncLogBufferAccept(SSyncLogBuffer* pBuf, SSyncNode* pNode, SSyncRaftEnt pBuf->endIndex); SyncTerm existPrevTerm = -1; (void)syncLogReplGetPrevLogTerm(NULL, pNode, index, &existPrevTerm); - ASSERT(pEntry->term == pExist->term && (pEntry->index > pBuf->matchIndex || prevTerm == existPrevTerm)); + if (!(pEntry->term == pExist->term && (pEntry->index > pBuf->matchIndex || prevTerm == existPrevTerm))) { + sError("vgId:%d, failed to accept, pEntry->term:%" PRId64 ", pExist->indexpExist->term:%" PRId64 + ", pEntry->index:%" PRId64 ", pBuf->matchIndex:%" PRId64 ", prevTerm:%" PRId64 + ", existPrevTerm:%" PRId64, + pNode->vgId, pEntry->term, pExist->term, pEntry->index, pBuf->matchIndex, prevTerm, existPrevTerm); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } code = 0; goto _out; } } // update - ASSERT(pBuf->startIndex < index); - ASSERT(index - pBuf->startIndex < pBuf->size); - ASSERT(pBuf->entries[index % pBuf->size].pItem == NULL); + if (!(pBuf->startIndex < index)) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + }; + if (!(index - pBuf->startIndex < pBuf->size)) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } + if (pBuf->entries[index % pBuf->size].pItem != NULL) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } SSyncLogBufEntry tmp = {.pItem = pEntry, .prevLogIndex = prevIndex, .prevLogTerm = prevTerm}; pEntry = NULL; pBuf->entries[index % pBuf->size] = tmp; @@ -411,8 +485,8 @@ _out: syncEntryDestroy(pExist); pExist = NULL; } - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); TAOS_RETURN(code); } @@ -422,14 +496,14 @@ static inline bool syncLogStoreNeedFlush(SSyncRaftEntry* pEntry, int32_t replica int32_t syncLogStorePersist(SSyncLogStore* pLogStore, SSyncNode* pNode, SSyncRaftEntry* pEntry) { int32_t code = 0; - ASSERT(pEntry->index >= 0); + if (pEntry->index < 0) return TSDB_CODE_SYN_INTERNAL_ERROR; SyncIndex lastVer = pLogStore->syncLogLastIndex(pLogStore); if (lastVer >= pEntry->index && (code = pLogStore->syncLogTruncate(pLogStore, pEntry->index)) < 0) { sError("failed to truncate log store since %s. from index:%" PRId64 "", tstrerror(code), pEntry->index); TAOS_RETURN(code); } lastVer = pLogStore->syncLogLastIndex(pLogStore); - ASSERT(pEntry->index == lastVer + 1); + if (pEntry->index != lastVer + 1) return TSDB_CODE_SYN_INTERNAL_ERROR; bool doFsync = syncLogStoreNeedFlush(pEntry, pNode->replicaNum); if ((code = pLogStore->syncLogAppendEntry(pLogStore, pEntry, doFsync)) < 0) { @@ -439,13 +513,13 @@ int32_t syncLogStorePersist(SSyncLogStore* pLogStore, SSyncNode* pNode, SSyncRaf } lastVer = pLogStore->syncLogLastIndex(pLogStore); - ASSERT(pEntry->index == lastVer); + if (pEntry->index != lastVer) return TSDB_CODE_SYN_INTERNAL_ERROR; return 0; } int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* pMatchTerm, char* str) { + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); (void)taosThreadMutexLock(&pBuf->mutex); - (void)syncLogBufferValidate(pBuf); SSyncLogStore* pLogStore = pNode->pLogStore; int64_t matchIndex = pBuf->matchIndex; @@ -453,7 +527,11 @@ int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* p while (pBuf->matchIndex + 1 < pBuf->endIndex) { int64_t index = pBuf->matchIndex + 1; - ASSERT(index >= 0); + if (index < 0) { + sError("vgId:%d, failed to proceed index:%" PRId64, pNode->vgId, index); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } // try to proceed SSyncLogBufEntry* pBufEntry = &pBuf->entries[index % pBuf->size]; @@ -466,14 +544,37 @@ int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* p goto _out; } - ASSERT(index == pEntry->index); + if (index != pEntry->index) { + sError("vgId:%d, failed to proceed index:%" PRId64 ", pEntry->index:%" PRId64, pNode->vgId, index, pEntry->index); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } // match SSyncRaftEntry* pMatch = pBuf->entries[(pBuf->matchIndex + pBuf->size) % pBuf->size].pItem; - ASSERT(pMatch != NULL); - ASSERT(pMatch->index == pBuf->matchIndex); - ASSERT(pMatch->index + 1 == pEntry->index); - ASSERT(prevLogIndex == pMatch->index); + if (pMatch == NULL) { + sError("vgId:%d, failed to proceed since pMatch is null", pNode->vgId); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } + if (pMatch->index != pBuf->matchIndex) { + sError("vgId:%d, failed to proceed, pMatch->index:%" PRId64 ", pBuf->matchIndex:%" PRId64, pNode->vgId, + pMatch->index, pBuf->matchIndex); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } + if (pMatch->index + 1 != pEntry->index) { + sError("vgId:%d, failed to proceed, pMatch->index:%" PRId64 ", pEntry->index:%" PRId64, pNode->vgId, + pMatch->index, pEntry->index); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } + if (prevLogIndex != pMatch->index) { + sError("vgId:%d, failed to proceed, prevLogIndex:%" PRId64 ", pMatch->index:%" PRId64, pNode->vgId, prevLogIndex, + pMatch->index); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } if (pMatch->term != prevLogTerm) { sInfo( @@ -532,7 +633,12 @@ int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* p // replicate on demand (void)syncNodeReplicateWithoutLock(pNode); - ASSERT(pEntry->index == pBuf->matchIndex); + if (pEntry->index != pBuf->matchIndex) { + sError("vgId:%d, failed to proceed, pEntry->index:%" PRId64 ", pBuf->matchIndex:%" PRId64, pNode->vgId, + pEntry->index, pBuf->matchIndex); + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } // update my match index matchIndex = pBuf->matchIndex; @@ -544,8 +650,8 @@ _out: if (pMatchTerm) { *pMatchTerm = pBuf->entries[(matchIndex + pBuf->size) % pBuf->size].pItem->term; } - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); return matchIndex; } @@ -576,12 +682,18 @@ int32_t syncFsmExecute(SSyncNode* pNode, SSyncFSM* pFsm, ESyncState role, SyncTe int32_t code = 0, lino = 0; bool retry = false; do { + SFsmCbMeta cbMeta = {0}; + cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(pNode, pEntry->index); + if (cbMeta.lastConfigIndex < -1) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + if (terrno != 0) code = terrno; + return code; + } + SRpcMsg rpcMsg = {.code = applyCode}; TAOS_CHECK_EXIT(syncEntry2OriginalRpc(pEntry, &rpcMsg)); - SFsmCbMeta cbMeta = {0}; cbMeta.index = pEntry->index; - cbMeta.lastConfigIndex = syncNodeGetSnapshotConfigIndex(pNode, pEntry->index); cbMeta.isWeak = pEntry->isWeak; cbMeta.code = applyCode; cbMeta.state = role; @@ -608,17 +720,36 @@ _exit: } int32_t syncLogBufferValidate(SSyncLogBuffer* pBuf) { - ASSERT(pBuf->startIndex <= pBuf->matchIndex); - ASSERT(pBuf->commitIndex <= pBuf->matchIndex); - ASSERT(pBuf->matchIndex < pBuf->endIndex); - ASSERT(pBuf->endIndex - pBuf->startIndex <= pBuf->size); - ASSERT(pBuf->entries[(pBuf->matchIndex + pBuf->size) % pBuf->size].pItem); + if (pBuf->startIndex > pBuf->matchIndex) { + sError("failed to validate, pBuf->startIndex:%" PRId64 ", pBuf->matchIndex:%" PRId64, pBuf->startIndex, + pBuf->matchIndex); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pBuf->commitIndex > pBuf->matchIndex) { + sError("failed to validate, pBuf->commitIndex:%" PRId64 ", pBuf->matchIndex:%" PRId64, pBuf->commitIndex, + pBuf->matchIndex); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pBuf->matchIndex >= pBuf->endIndex) { + sError("failed to validate, pBuf->matchIndex:%" PRId64 ", pBuf->endIndex:%" PRId64, pBuf->matchIndex, + pBuf->endIndex); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pBuf->endIndex - pBuf->startIndex > pBuf->size) { + sError("failed to validate, pBuf->endIndex:%" PRId64 ", pBuf->startIndex:%" PRId64 ", pBuf->size:%" PRId64, + pBuf->endIndex, pBuf->startIndex, pBuf->size); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } + if (pBuf->entries[(pBuf->matchIndex + pBuf->size) % pBuf->size].pItem == NULL) { + sError("failed to validate since pItem is null"); + return TSDB_CODE_SYN_INTERNAL_ERROR; + } return 0; } int32_t syncLogBufferCommit(SSyncLogBuffer* pBuf, SSyncNode* pNode, int64_t commitIndex) { + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); (void)taosThreadMutexLock(&pBuf->mutex); - (void)syncLogBufferValidate(pBuf); SSyncLogStore* pLogStore = pNode->pLogStore; SSyncFSM* pFsm = pNode->pFsm; @@ -718,7 +849,7 @@ int32_t syncLogBufferCommit(SSyncLogBuffer* pBuf, SSyncNode* pNode, int64_t comm SyncIndex until = pBuf->commitIndex - TSDB_SYNC_LOG_BUFFER_RETENTION; for (SyncIndex index = pBuf->startIndex; index < until; index++) { SSyncRaftEntry* pEntry = pBuf->entries[(index + pBuf->size) % pBuf->size].pItem; - ASSERT(pEntry != NULL); + if (pEntry == NULL) return TSDB_CODE_SYN_INTERNAL_ERROR; syncEntryDestroy(pEntry); (void)memset(&pBuf->entries[(index + pBuf->size) % pBuf->size], 0, sizeof(pBuf->entries[0])); pBuf->startIndex = index + 1; @@ -743,15 +874,18 @@ _out: syncEntryDestroy(pNextEntry); pNextEntry = NULL; } - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); TAOS_RETURN(code); } void syncLogReplReset(SSyncLogReplMgr* pMgr) { if (pMgr == NULL) return; - ASSERT(pMgr->startIndex >= 0); + if (pMgr->startIndex < 0) { + sError("failed to reset, pMgr->startIndex:%" PRId64, pMgr->startIndex); + return; + } for (SyncIndex index = pMgr->startIndex; index < pMgr->endIndex; index++) { (void)memset(&pMgr->states[index % pMgr->size], 0, sizeof(pMgr->states[0])); } @@ -786,7 +920,10 @@ int32_t syncLogReplRetryOnNeed(SSyncLogReplMgr* pMgr, SSyncNode* pNode) { for (SyncIndex index = pMgr->startIndex; index < pMgr->endIndex; index++) { int64_t pos = index % pMgr->size; - ASSERT(!pMgr->states[pos].barrier || (index == pMgr->startIndex || index + 1 == pMgr->endIndex)); + if (!(!pMgr->states[pos].barrier || (index == pMgr->startIndex || index + 1 == pMgr->endIndex))) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } if (nowMs < pMgr->states[pos].timeMs + retryWaitMs) { break; @@ -809,7 +946,10 @@ int32_t syncLogReplRetryOnNeed(SSyncLogReplMgr* pMgr, SSyncNode* pNode) { tstrerror(code), index, pDestId->addr); goto _out; } - ASSERT(barrier == pMgr->states[pos].barrier); + if (barrier != pMgr->states[pos].barrier) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } pMgr->states[pos].timeMs = nowMs; pMgr->states[pos].term = term; pMgr->states[pos].acked = false; @@ -839,11 +979,11 @@ int32_t syncLogReplRecover(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncAppendEn SSyncLogBuffer* pBuf = pNode->pLogBuf; SRaftId destId = pMsg->srcId; int32_t code = 0; - ASSERT(pMgr->restored == false); + if (pMgr->restored != false) return TSDB_CODE_SYN_INTERNAL_ERROR; if (pMgr->endIndex == 0) { - ASSERT(pMgr->startIndex == 0); - ASSERT(pMgr->matchIndex == 0); + if (pMgr->startIndex != 0) return TSDB_CODE_SYN_INTERNAL_ERROR; + if (pMgr->matchIndex != 0) return TSDB_CODE_SYN_INTERNAL_ERROR; if (pMsg->matchIndex < 0) { pMgr->restored = true; sInfo("vgId:%d, sync log repl restored. peer: dnode:%d (%" PRIx64 "), repl-mgr:[%" PRId64 " %" PRId64 ", %" PRId64 @@ -909,7 +1049,7 @@ int32_t syncLogReplRecover(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncAppendEn if ((index + 1 < firstVer) || (term < 0) || (term != pMsg->lastMatchTerm && (index + 1 == firstVer || index == firstVer))) { - ASSERT(term >= 0 || terrno == TSDB_CODE_WAL_LOG_NOT_EXIST); + if (!(term >= 0 || terrno == TSDB_CODE_WAL_LOG_NOT_EXIST)) return TSDB_CODE_SYN_INTERNAL_ERROR; if ((code = syncNodeStartSnapshot(pNode, &destId)) < 0) { sError("vgId:%d, failed to start snapshot for peer dnode:%d", pNode->vgId, DID(&destId)); TAOS_RETURN(code); @@ -918,13 +1058,13 @@ int32_t syncLogReplRecover(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncAppendEn return 0; } - ASSERT(index + 1 >= firstVer); + if (!(index + 1 >= firstVer)) return TSDB_CODE_SYN_INTERNAL_ERROR; if (term == pMsg->lastMatchTerm) { index = index + 1; - ASSERT(index <= pNode->pLogBuf->matchIndex); + if (!(index <= pNode->pLogBuf->matchIndex)) return TSDB_CODE_SYN_INTERNAL_ERROR; } else { - ASSERT(index > firstVer); + if (!(index > firstVer)) return TSDB_CODE_SYN_INTERNAL_ERROR; } } @@ -975,8 +1115,8 @@ int32_t syncLogReplStart(SSyncLogReplMgr* pMgr, SSyncNode* pNode) { } int32_t syncLogReplProbe(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncIndex index) { - ASSERT(!pMgr->restored); - ASSERT(pMgr->startIndex >= 0); + if (pMgr->restored) return TSDB_CODE_SYN_INTERNAL_ERROR; + if (!(pMgr->startIndex >= 0)) return TSDB_CODE_SYN_INTERNAL_ERROR; int64_t retryMaxWaitMs = syncGetRetryMaxWaitMs(); int64_t nowMs = taosGetMonoTimestampMs(); int32_t code = 0; @@ -996,7 +1136,7 @@ int32_t syncLogReplProbe(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncIndex inde TAOS_RETURN(code); } - ASSERT(index >= 0); + if (!(index >= 0)) return TSDB_CODE_SYN_INTERNAL_ERROR; pMgr->states[index % pMgr->size].barrier = barrier; pMgr->states[index % pMgr->size].timeMs = nowMs; pMgr->states[index % pMgr->size].term = term; @@ -1014,7 +1154,7 @@ int32_t syncLogReplProbe(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncIndex inde } int32_t syncLogReplAttempt(SSyncLogReplMgr* pMgr, SSyncNode* pNode) { - ASSERT(pMgr->restored); + if (!pMgr->restored) return TSDB_CODE_SYN_INTERNAL_ERROR; SRaftId* pDestId = &pNode->replicasId[pMgr->peerId]; int32_t batchSize = TMAX(1, pMgr->size >> (4 + pMgr->retryBackoff)); @@ -1070,7 +1210,7 @@ int32_t syncLogReplAttempt(SSyncLogReplMgr* pMgr, SSyncNode* pNode) { } int32_t syncLogReplContinue(SSyncLogReplMgr* pMgr, SSyncNode* pNode, SyncAppendEntriesReply* pMsg) { - ASSERT(pMgr->restored == true); + if (pMgr->restored != true) return TSDB_CODE_SYN_INTERNAL_ERROR; if (pMgr->startIndex <= pMsg->lastSendIndex && pMsg->lastSendIndex < pMgr->endIndex) { if (pMgr->startIndex < pMgr->matchIndex && pMgr->retryBackoff > 0) { int64_t firstMs = pMgr->states[pMgr->startIndex % pMgr->size].timeMs; @@ -1100,7 +1240,10 @@ SSyncLogReplMgr* syncLogReplCreate() { pMgr->size = sizeof(pMgr->states) / sizeof(pMgr->states[0]); - ASSERT(pMgr->size == TSDB_SYNC_LOG_BUFFER_SIZE); + if (pMgr->size != TSDB_SYNC_LOG_BUFFER_SIZE) { + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return NULL; + } return pMgr; } @@ -1115,7 +1258,7 @@ void syncLogReplDestroy(SSyncLogReplMgr* pMgr) { int32_t syncNodeLogReplInit(SSyncNode* pNode) { for (int i = 0; i < TSDB_MAX_REPLICA + TSDB_MAX_LEARNER_REPLICA; i++) { - ASSERT(pNode->logReplMgrs[i] == NULL); + if (pNode->logReplMgrs[i] != NULL) return TSDB_CODE_SYN_INTERNAL_ERROR; pNode->logReplMgrs[i] = syncLogReplCreate(); if (pNode->logReplMgrs[i] == NULL) { TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); @@ -1141,7 +1284,10 @@ int32_t syncLogBufferCreate(SSyncLogBuffer** ppBuf) { pBuf->size = sizeof(pBuf->entries) / sizeof(pBuf->entries[0]); - ASSERT(pBuf->size == TSDB_SYNC_LOG_BUFFER_SIZE); + if (pBuf->size != TSDB_SYNC_LOG_BUFFER_SIZE) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _exit; + } if (taosThreadMutexAttrInit(&pBuf->attr) < 0) { code = TAOS_SYSTEM_ERROR(errno); @@ -1194,7 +1340,7 @@ void syncLogBufferDestroy(SSyncLogBuffer* pBuf) { int32_t syncLogBufferRollback(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncIndex toIndex) { int32_t code = 0; - ASSERT(pBuf->commitIndex < toIndex && toIndex <= pBuf->endIndex); + if (!(pBuf->commitIndex < toIndex && toIndex <= pBuf->endIndex)) return TSDB_CODE_SYN_INTERNAL_ERROR; if (toIndex == pBuf->endIndex) { return 0; @@ -1217,7 +1363,7 @@ int32_t syncLogBufferRollback(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncIndex } pBuf->endIndex = toIndex; pBuf->matchIndex = TMIN(pBuf->matchIndex, index); - ASSERT(index + 1 == toIndex); + if (index + 1 != toIndex) return TSDB_CODE_SYN_INTERNAL_ERROR; // trunc wal SyncIndex lastVer = pNode->pLogStore->syncLogLastIndex(pNode->pLogStore); @@ -1227,7 +1373,7 @@ int32_t syncLogBufferRollback(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncIndex TAOS_RETURN(code); } lastVer = pNode->pLogStore->syncLogLastIndex(pNode->pLogStore); - ASSERT(toIndex == lastVer + 1); + if (toIndex != lastVer + 1) return TSDB_CODE_SYN_INTERNAL_ERROR; // refill buffer on need if (toIndex <= pBuf->startIndex) { @@ -1237,16 +1383,16 @@ int32_t syncLogBufferRollback(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncIndex } } - ASSERT(pBuf->endIndex == toIndex); - (void)syncLogBufferValidate(pBuf); + if (pBuf->endIndex != toIndex) return TSDB_CODE_SYN_INTERNAL_ERROR; + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); return 0; } int32_t syncLogBufferReset(SSyncLogBuffer* pBuf, SSyncNode* pNode) { + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); (void)taosThreadMutexLock(&pBuf->mutex); - (void)syncLogBufferValidate(pBuf); SyncIndex lastVer = pNode->pLogStore->syncLogLastIndex(pNode->pLogStore); - ASSERT(lastVer == pBuf->matchIndex); + if (lastVer != pBuf->matchIndex) return TSDB_CODE_SYN_INTERNAL_ERROR; SyncIndex index = pBuf->endIndex - 1; (void)syncLogBufferRollback(pBuf, pNode, pBuf->matchIndex + 1); @@ -1261,8 +1407,8 @@ int32_t syncLogBufferReset(SSyncLogBuffer* pBuf, SSyncNode* pNode) { SSyncLogReplMgr* pMgr = pNode->logReplMgrs[i]; syncLogReplReset(pMgr); } - (void)syncLogBufferValidate(pBuf); (void)taosThreadMutexUnlock(&pBuf->mutex); + TAOS_CHECK_RETURN(syncLogBufferValidate(pBuf)); return 0; } diff --git a/source/libs/sync/src/syncRaftEntry.c b/source/libs/sync/src/syncRaftEntry.c index fd6781f354..56a702a9d5 100644 --- a/source/libs/sync/src/syncRaftEntry.c +++ b/source/libs/sync/src/syncRaftEntry.c @@ -70,7 +70,10 @@ SSyncRaftEntry* syncEntryBuildFromAppendEntries(const SyncAppendEntries* pMsg) { return NULL; } memcpy(pEntry, pMsg->data, pMsg->dataLen); - ASSERT(pEntry->bytes == pMsg->dataLen); + if (pEntry->bytes != pMsg->dataLen) { + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return NULL; + } return pEntry; } diff --git a/source/libs/sync/src/syncRaftLog.c b/source/libs/sync/src/syncRaftLog.c index ecca777806..e7ecb67e69 100644 --- a/source/libs/sync/src/syncRaftLog.c +++ b/source/libs/sync/src/syncRaftLog.c @@ -61,7 +61,10 @@ SSyncLogStore* logStoreCreate(SSyncNode* pSyncNode) { SSyncLogStoreData* pData = pLogStore->data; pData->pSyncNode = pSyncNode; pData->pWal = pSyncNode->pWal; - ASSERT(pData->pWal != NULL); + if (pData->pWal == NULL) { + terrno = TSDB_CODE_SYN_INTERNAL_ERROR; + return NULL; + } (void)taosThreadMutexInit(&(pData->mutex), NULL); pData->pWalHandle = walOpenReader(pData->pWal, NULL, 0); @@ -115,7 +118,7 @@ void logStoreDestory(SSyncLogStore* pLogStore) { // log[m .. n] static int32_t raftLogRestoreFromSnapshot(struct SSyncLogStore* pLogStore, SyncIndex snapshotIndex) { - ASSERT(snapshotIndex >= 0); + if (!(snapshotIndex >= 0)) return TSDB_CODE_SYN_INTERNAL_ERROR; SSyncLogStoreData* pData = pLogStore->data; SWal* pWal = pData->pWal; @@ -303,14 +306,14 @@ int32_t raftLogGetEntry(struct SSyncLogStore* pLogStore, SyncIndex index, SSyncR } *ppEntry = syncEntryBuild(pWalHandle->pHead->head.bodyLen); - ASSERT(*ppEntry != NULL); + if (*ppEntry == NULL) return TSDB_CODE_SYN_INTERNAL_ERROR; (*ppEntry)->msgType = TDMT_SYNC_CLIENT_REQUEST; (*ppEntry)->originalRpcType = pWalHandle->pHead->head.msgType; (*ppEntry)->seqNum = pWalHandle->pHead->head.syncMeta.seqNum; (*ppEntry)->isWeak = pWalHandle->pHead->head.syncMeta.isWeek; (*ppEntry)->term = pWalHandle->pHead->head.syncMeta.term; (*ppEntry)->index = index; - ASSERT((*ppEntry)->dataLen == pWalHandle->pHead->head.bodyLen); + if ((*ppEntry)->dataLen != pWalHandle->pHead->head.bodyLen) return TSDB_CODE_SYN_INTERNAL_ERROR; (void)memcpy((*ppEntry)->data, pWalHandle->pHead->head.body, pWalHandle->pHead->head.bodyLen); /* @@ -362,14 +365,14 @@ static int32_t raftLogTruncate(struct SSyncLogStore* pLogStore, SyncIndex fromIn static int32_t raftLogGetLastEntry(SSyncLogStore* pLogStore, SSyncRaftEntry** ppLastEntry) { SSyncLogStoreData* pData = pLogStore->data; SWal* pWal = pData->pWal; - ASSERT(ppLastEntry != NULL); + if (ppLastEntry == NULL) return TSDB_CODE_SYN_INTERNAL_ERROR; *ppLastEntry = NULL; if (walIsEmpty(pWal)) { TAOS_RETURN(TSDB_CODE_WAL_LOG_NOT_EXIST); } else { SyncIndex lastIndex = raftLogLastIndex(pLogStore); - ASSERT(lastIndex >= SYNC_INDEX_BEGIN); + if (!(lastIndex >= SYNC_INDEX_BEGIN)) return TSDB_CODE_SYN_INTERNAL_ERROR; int32_t code = raftLogGetEntry(pLogStore, lastIndex, ppLastEntry); TAOS_RETURN(code); diff --git a/source/libs/sync/src/syncRequestVote.c b/source/libs/sync/src/syncRequestVote.c index a30b9a4930..738e558033 100644 --- a/source/libs/sync/src/syncRequestVote.c +++ b/source/libs/sync/src/syncRequestVote.c @@ -95,7 +95,7 @@ int32_t syncNodeOnRequestVote(SSyncNode* ths, const SRpcMsg* pRpcMsg) { if (!syncNodeInRaftGroup(ths, &pMsg->srcId)) { syncLogRecvRequestVote(ths, pMsg, -1, "not in my config"); - TAOS_RETURN(TSDB_CODE_FAILED); + TAOS_RETURN(TSDB_CODE_SYN_MISMATCHED_SIGNATURE); } bool logOK = syncNodeOnRequestVoteLogOK(ths, pMsg); @@ -104,7 +104,7 @@ int32_t syncNodeOnRequestVote(SSyncNode* ths, const SRpcMsg* pRpcMsg) { syncNodeStepDown(ths, pMsg->term); } SyncTerm currentTerm = raftStoreGetTerm(ths); - ASSERT(pMsg->term <= currentTerm); + if (!(pMsg->term <= currentTerm)) return TSDB_CODE_SYN_INTERNAL_ERROR; bool grant = (pMsg->term == currentTerm) && logOK && ((!raftStoreHasVoted(ths)) || (syncUtilSameId(&ths->raftStore.voteFor, &pMsg->srcId))); @@ -130,7 +130,7 @@ int32_t syncNodeOnRequestVote(SSyncNode* ths, const SRpcMsg* pRpcMsg) { pReply->destId = pMsg->srcId; pReply->term = currentTerm; pReply->voteGranted = grant; - ASSERT(!grant || pMsg->term == pReply->term); + if (!(!grant || pMsg->term == pReply->term)) return TSDB_CODE_SYN_INTERNAL_ERROR; // trace log syncLogRecvRequestVote(ths, pMsg, pReply->voteGranted, ""); diff --git a/source/libs/sync/src/syncRequestVoteReply.c b/source/libs/sync/src/syncRequestVoteReply.c index 10d9a6c96b..add237c2d0 100644 --- a/source/libs/sync/src/syncRequestVoteReply.c +++ b/source/libs/sync/src/syncRequestVoteReply.c @@ -46,7 +46,7 @@ int32_t syncNodeOnRequestVoteReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) { if (!syncNodeInRaftGroup(ths, &(pMsg->srcId))) { syncLogRecvRequestVoteReply(ths, pMsg, "not in my config"); - TAOS_RETURN(TSDB_CODE_FAILED); + TAOS_RETURN(TSDB_CODE_SYN_MISMATCHED_SIGNATURE); } SyncTerm currentTerm = raftStoreGetTerm(ths); // drop stale response @@ -64,7 +64,7 @@ int32_t syncNodeOnRequestVoteReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) { } syncLogRecvRequestVoteReply(ths, pMsg, ""); - ASSERT(pMsg->term == currentTerm); + if (pMsg->term != currentTerm) return TSDB_CODE_SYN_INTERNAL_ERROR; // This tallies votes even when the current state is not Candidate, // but they won't be looked at, so it doesn't matter. diff --git a/source/libs/sync/src/syncSnapshot.c b/source/libs/sync/src/syncSnapshot.c index 8a2c53997b..2a3b164f03 100644 --- a/source/libs/sync/src/syncSnapshot.c +++ b/source/libs/sync/src/syncSnapshot.c @@ -57,7 +57,7 @@ static int32_t syncSnapBufferCreate(SSyncSnapBuffer **ppBuf) { TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); } pBuf->size = sizeof(pBuf->entries) / sizeof(void *); - ASSERT(pBuf->size == TSDB_SYNC_SNAP_BUFFER_SIZE); + if (pBuf->size != TSDB_SYNC_SNAP_BUFFER_SIZE) return TSDB_CODE_SYN_INTERNAL_ERROR; (void)taosThreadMutexInit(&pBuf->mutex, NULL); *ppBuf = pBuf; TAOS_RETURN(0); @@ -311,7 +311,10 @@ static int32_t snapshotSend(SSyncSnapshotSender *pSender) { } } - ASSERT(pSender->seq >= SYNC_SNAPSHOT_SEQ_BEGIN && pSender->seq <= SYNC_SNAPSHOT_SEQ_END); + if (!(pSender->seq >= SYNC_SNAPSHOT_SEQ_BEGIN && pSender->seq <= SYNC_SNAPSHOT_SEQ_END)) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _OUT; + } // send msg int32_t blockLen = (pBlk) ? pBlk->blockLen : 0; @@ -323,7 +326,10 @@ static int32_t snapshotSend(SSyncSnapshotSender *pSender) { // put in buffer int64_t nowMs = taosGetTimestampMs(); if (pBlk) { - ASSERT(pBlk->seq > SYNC_SNAPSHOT_SEQ_BEGIN && pBlk->seq < SYNC_SNAPSHOT_SEQ_END); + if (!(pBlk->seq > SYNC_SNAPSHOT_SEQ_BEGIN && pBlk->seq < SYNC_SNAPSHOT_SEQ_END)) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _OUT; + } pBlk->sendTimeMs = nowMs; pSender->pSndBuf->entries[pSender->seq % pSender->pSndBuf->size] = pBlk; pBlk = NULL; @@ -351,7 +357,10 @@ int32_t snapshotReSend(SSyncSnapshotSender *pSender) { for (int32_t seq = pSndBuf->cursor + 1; seq < pSndBuf->end; ++seq) { SyncSnapBlock *pBlk = pSndBuf->entries[seq % pSndBuf->size]; - ASSERT(pBlk); + if (!pBlk) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } int64_t nowMs = taosGetTimestampMs(); if (pBlk->acked || nowMs < pBlk->sendTimeMs + SYNC_SNAP_RESEND_MS) { continue; @@ -682,7 +691,7 @@ SyncIndex syncNodeGetSnapBeginIndex(SSyncNode *ths) { static int32_t syncSnapReceiverExchgSnapInfo(SSyncNode *pSyncNode, SSyncSnapshotReceiver *pReceiver, SyncSnapshotSend *pMsg, SSnapshot *pInfo) { - ASSERT(pMsg->payloadType == TDMT_SYNC_PREP_SNAPSHOT); + if (pMsg->payloadType != TDMT_SYNC_PREP_SNAPSHOT) return TSDB_CODE_SYN_INTERNAL_ERROR; int32_t code = 0, lino = 0; // copy snap info from leader @@ -878,7 +887,7 @@ static int32_t syncSnapBufferRecv(SSyncSnapshotReceiver *pReceiver, SyncSnapshot goto _out; } - ASSERT(pRcvBuf->start <= pRcvBuf->cursor + 1 && pRcvBuf->cursor < pRcvBuf->end); + if (!(pRcvBuf->start <= pRcvBuf->cursor + 1 && pRcvBuf->cursor < pRcvBuf->end)) return TSDB_CODE_SYN_INTERNAL_ERROR; if (pMsg->seq > pRcvBuf->cursor) { if (pRcvBuf->entries[pMsg->seq % pRcvBuf->size]) { @@ -922,7 +931,7 @@ static int32_t syncNodeOnSnapshotReceive(SSyncNode *pSyncNode, SyncSnapshotSend // condition 4 // transfering SyncSnapshotSend *pMsg = ppMsg[0]; - ASSERT(pMsg); + if (!pMsg) return TSDB_CODE_SYN_INTERNAL_ERROR; SSyncSnapshotReceiver *pReceiver = pSyncNode->pNewNodeReceiver; int64_t timeNow = taosGetTimestampMs(); int32_t code = 0; @@ -1071,7 +1080,7 @@ _out:; } static int32_t syncSnapSenderExchgSnapInfo(SSyncNode *pSyncNode, SSyncSnapshotSender *pSender, SyncSnapshotRsp *pMsg) { - ASSERT(pMsg->payloadType == TDMT_SYNC_PREP_SNAPSHOT_REPLY); + if (pMsg->payloadType != TDMT_SYNC_PREP_SNAPSHOT_REPLY) return TSDB_CODE_SYN_INTERNAL_ERROR; SSyncTLV *datHead = (void *)pMsg->data; if (datHead->typ != pMsg->payloadType) { @@ -1168,11 +1177,17 @@ static int32_t syncSnapBufferSend(SSyncSnapshotSender *pSender, SyncSnapshotRsp goto _out; } - ASSERT(pSndBuf->start <= pSndBuf->cursor + 1 && pSndBuf->cursor < pSndBuf->end); + if (!(pSndBuf->start <= pSndBuf->cursor + 1 && pSndBuf->cursor < pSndBuf->end)) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } if (pMsg->ack > pSndBuf->cursor && pMsg->ack < pSndBuf->end) { SyncSnapBlock *pBlk = pSndBuf->entries[pMsg->ack % pSndBuf->size]; - ASSERT(pBlk); + if (!pBlk) { + code = TSDB_CODE_SYN_INTERNAL_ERROR; + goto _out; + } pBlk->acked = 1; } diff --git a/source/libs/tdb/src/db/tdbBtree.c b/source/libs/tdb/src/db/tdbBtree.c index 59b7f4af2e..ff40616d70 100644 --- a/source/libs/tdb/src/db/tdbBtree.c +++ b/source/libs/tdb/src/db/tdbBtree.c @@ -42,10 +42,6 @@ struct SBTree { #define TDB_BTREE_PAGE_IS_ROOT(PAGE) (TDB_BTREE_PAGE_GET_FLAGS(PAGE) & TDB_BTREE_ROOT) #define TDB_BTREE_PAGE_IS_LEAF(PAGE) (TDB_BTREE_PAGE_GET_FLAGS(PAGE) & TDB_BTREE_LEAF) #define TDB_BTREE_PAGE_IS_OVFL(PAGE) (TDB_BTREE_PAGE_GET_FLAGS(PAGE) & TDB_BTREE_OVFL) -#define TDB_BTREE_ASSERT_FLAG(flags) \ - ASSERT(TDB_FLAG_IS(flags, TDB_BTREE_ROOT) || TDB_FLAG_IS(flags, TDB_BTREE_LEAF) || \ - TDB_FLAG_IS(flags, TDB_BTREE_ROOT | TDB_BTREE_LEAF) || TDB_FLAG_IS(flags, 0) || \ - TDB_FLAG_IS(flags, TDB_BTREE_OVFL)) #pragma pack(push, 1) typedef struct { @@ -161,7 +157,7 @@ int tdbBtreeOpen(int keyLen, int valLen, SPager *pPager, char const *tbname, SPg if (pgno == 0) { tdbError("tdb/btree-open: pgno cannot be zero."); tdbOsFree(pBt); - ASSERT(0); + return TSDB_CODE_INTERNAL_ERROR; } pBt->root = pgno; /* @@ -418,10 +414,6 @@ static int tdbDefaultKeyCmprFn(const void *pKey1, int keyLen1, const void *pKey2 int mlen; int cret; - if (ASSERT(keyLen1 > 0 && keyLen2 > 0 && pKey1 != NULL && pKey2 != NULL)) { - // -1 is less than - } - mlen = keyLen1 < keyLen2 ? keyLen1 : keyLen2; cret = memcmp(pKey1, pKey2, mlen); if (cret == 0) { @@ -447,14 +439,12 @@ int tdbBtreeInitPage(SPage *pPage, void *arg, int init) { // init page flags = TDB_BTREE_PAGE_GET_FLAGS(pPage); leaf = TDB_BTREE_PAGE_IS_LEAF(pPage); - TDB_BTREE_ASSERT_FLAG(flags); tdbPageInit(pPage, leaf ? sizeof(SLeafHdr) : sizeof(SIntHdr), tdbBtreeCellSize); } else { // zero page flags = ((SBtreeInitPageArg *)arg)->flags; leaf = flags & TDB_BTREE_LEAF; - TDB_BTREE_ASSERT_FLAG(flags); tdbPageZero(pPage, leaf ? sizeof(SLeafHdr) : sizeof(SIntHdr), tdbBtreeCellSize); @@ -571,14 +561,14 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx nOlds = 3; } for (int i = 0; i < nOlds; i++) { - if (ASSERT(sIdx + i <= nCells)) { + if (!(sIdx + i <= nCells)) { return TSDB_CODE_FAILED; } SPgno pgno; if (sIdx + i == nCells) { - if (ASSERT(!TDB_BTREE_PAGE_IS_LEAF(pParent))) { - return TSDB_CODE_FAILED; + if (TDB_BTREE_PAGE_IS_LEAF(pParent)) { + return TSDB_CODE_INTERNAL_ERROR; } pgno = ((SIntHdr *)(pParent->pData))->pgno; } else { @@ -685,8 +675,6 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx // page is full, use a new page nNews++; - ASSERT(infoNews[nNews].size + cellBytes <= TDB_PAGE_USABLE_SIZE(pPage)); - if (childNotLeaf) { // for non-child page, this cell is used as the right-most child, // the divider cell to parent as well @@ -732,7 +720,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx szRCell = tdbBtreeCellSize(pPage, pCell, 0, NULL, NULL); } - if (ASSERT(infoNews[iNew - 1].cnt > 0)) { + if (!(infoNews[iNew - 1].cnt > 0)) { return TSDB_CODE_FAILED; } @@ -822,10 +810,10 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx pCell = tdbPageGetCell(pPage, oIdx); szCell = tdbBtreeCellSize(pPage, pCell, 0, NULL, NULL); - if (ASSERT(nNewCells <= infoNews[iNew].cnt)) { + if (!(nNewCells <= infoNews[iNew].cnt)) { return TSDB_CODE_FAILED; } - if (ASSERT(iNew < nNews)) { + if (!(iNew < nNews)) { return TSDB_CODE_FAILED; } @@ -866,10 +854,10 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx } } } else { - if (ASSERT(childNotLeaf)) { + if (!(childNotLeaf)) { return TSDB_CODE_FAILED; } - if (ASSERT(iNew < nNews - 1)) { + if (!(iNew < nNews - 1)) { return TSDB_CODE_FAILED; } @@ -877,7 +865,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx ((SIntHdr *)pNews[iNew]->pData)->pgno = ((SPgno *)pCell)[0]; // insert to parent as divider cell - if (ASSERT(iNew < nNews - 1)) { + if (!(iNew < nNews - 1)) { return TSDB_CODE_FAILED; } ((SPgno *)pCell)[0] = TDB_PAGE_PGNO(pNews[iNew]); @@ -894,7 +882,7 @@ static int tdbBtreeBalanceNonRoot(SBTree *pBt, SPage *pParent, int idx, TXN *pTx } if (childNotLeaf) { - if (ASSERT(TDB_PAGE_TOTAL_CELLS(pNews[nNews - 1]) == infoNews[nNews - 1].cnt)) { + if (!(TDB_PAGE_TOTAL_CELLS(pNews[nNews - 1]) == infoNews[nNews - 1].cnt)) { return TSDB_CODE_FAILED; } ((SIntHdr *)(pNews[nNews - 1]->pData))->pgno = rPgno; @@ -1091,7 +1079,7 @@ static int tdbBtreeEncodePayload(SPage *pPage, SCell *pCell, int nHeader, const nLeft -= kLen; // pack partial val to local if any space left if (nLocal > nHeader + kLen + sizeof(SPgno)) { - if (ASSERT(pVal != NULL && vLen != 0)) { + if (!(pVal != NULL && vLen != 0)) { tdbFree(pBuf); return TSDB_CODE_FAILED; } @@ -1145,7 +1133,7 @@ static int tdbBtreeEncodePayload(SPage *pPage, SCell *pCell, int nHeader, const memcpy(pCell + nLocal - sizeof(pgno), &pgno, sizeof(pgno)); - int lastKeyPageSpace = 0; + size_t lastKeyPageSpace = 0; // pack left key & val to ovpages do { // cal key to cpy @@ -1259,14 +1247,14 @@ static int tdbBtreeEncodeCell(SPage *pPage, const void *pKey, int kLen, const vo int nPayload; int ret; - if (ASSERT(pPage->kLen == TDB_VARIANT_LEN || pPage->kLen == kLen)) { - return TSDB_CODE_FAILED; + if (!(pPage->kLen == TDB_VARIANT_LEN || pPage->kLen == kLen)) { + return TSDB_CODE_INVALID_PARA; } - if (ASSERT(pPage->vLen == TDB_VARIANT_LEN || pPage->vLen == vLen)) { - return TSDB_CODE_FAILED; + if (!(pPage->vLen == TDB_VARIANT_LEN || pPage->vLen == vLen)) { + return TSDB_CODE_INVALID_PARA; } - if (ASSERT(pKey != NULL && kLen > 0)) { - return TSDB_CODE_FAILED; + if (!(pKey != NULL && kLen > 0)) { + return TSDB_CODE_INVALID_PARA; } nPayload = 0; @@ -1645,7 +1633,6 @@ static int tdbBtreeCellSize(const SPage *pPage, SCell *pCell, int dropOfp, TXN * SArray *ofps = pPage->pPager->ofps; if (ofps) { if (taosArrayPush(ofps, &ofp) == NULL) { - ASSERT(0); return terrno; } } @@ -2438,7 +2425,10 @@ int tdbBtcMoveTo(SBTC *pBtc, const void *pKey, int kLen, int *pCRst) { lidx = 0; ridx = nCells - 1; - ASSERT(nCells > 0); + if (nCells <= 0) { + tdbError("tdb/btc-move-to: empty page."); + return TSDB_CODE_FAILED; + } // compare first cell pBtc->idx = lidx; diff --git a/source/libs/tdb/src/db/tdbPage.c b/source/libs/tdb/src/db/tdbPage.c index 26c1c108d2..eab8f6ef19 100644 --- a/source/libs/tdb/src/db/tdbPage.c +++ b/source/libs/tdb/src/db/tdbPage.c @@ -522,7 +522,9 @@ static int tdbPageDefragment(SPage *pPage) { SCell *pCell = TDB_PAGE_CELL_AT(pPage, aCellIdx[iCell].iCell); int32_t szCell = pPage->xCellSize(pPage, pCell, 0, NULL, NULL); - ASSERT(pNextCell - szCell >= pCell); + if (pNextCell - szCell < pCell) { + return TSDB_CODE_INTERNAL_ERROR; + } pNextCell -= szCell; if (pNextCell > pCell) { @@ -535,7 +537,11 @@ static int tdbPageDefragment(SPage *pPage) { TDB_PAGE_FCELL_SET(pPage, 0); tdbOsFree(aCellIdx); - ASSERT(pPage->pFreeEnd - pPage->pFreeStart == nFree); + if (pPage->pFreeEnd - pPage->pFreeStart != nFree) { + tdbError("tdb/page-defragment: nFree: %d, pFreeStart: %p, pFreeEnd: %p.", nFree, pPage->pFreeStart, + pPage->pFreeEnd); + return TSDB_CODE_INTERNAL_ERROR; + } return 0; } diff --git a/source/libs/tdb/src/db/tdbPager.c b/source/libs/tdb/src/db/tdbPager.c index a650847e1e..b0f32136a3 100644 --- a/source/libs/tdb/src/db/tdbPager.c +++ b/source/libs/tdb/src/db/tdbPager.c @@ -16,19 +16,7 @@ #include "crypt.h" #include "tdbInt.h" #include "tglobal.h" -/* -#pragma pack(push, 1) -typedef struct { - u8 hdrString[16]; - u16 pageSize; - SPgno freePage; - u32 nFreePages; - u8 reserved[102]; -} SFileHdr; -#pragma pack(pop) -TDB_STATIC_ASSERT(sizeof(SFileHdr) == 128, "Size of file header is not correct"); -*/ struct hashset_st { size_t nbits; size_t mask; @@ -450,7 +438,6 @@ static char *tdbEncryptPage(SPager *pPager, char *pPageData, int32_t pageSize, c if (encryptAlgorithm == DND_CA_SM4) { // tdbInfo("CBC_Encrypt key:%d %s %s", encryptAlgorithm, encryptKey, __FUNCTION__); - // ASSERT(strlen(encryptKey) > 0); // tdbInfo("CBC tdb offset:%" PRId64 ", flag:%d before Encrypt", offset, pPage->pData[0]); @@ -915,7 +902,6 @@ static int tdbPagerInitPage(SPager *pPager, SPage *pPage, int (*initPage)(SPage if (encryptAlgorithm == DND_CA_SM4) { // tdbInfo("CBC_Decrypt key:%d %s %s", encryptAlgorithm, encryptKey, __FUNCTION__); - // ASSERT(strlen(encryptKey) > 0); // uint8_t flags = pPage->pData[0]; // tdbInfo("CBC tdb offset:%" PRId64 ", flag:%d before Decrypt", ((i64)pPage->pageSize) * (pgno - 1), flags); diff --git a/source/libs/tdb/src/db/tdbTxn.c b/source/libs/tdb/src/db/tdbTxn.c index 24a70f62b2..71560e3e85 100644 --- a/source/libs/tdb/src/db/tdbTxn.c +++ b/source/libs/tdb/src/db/tdbTxn.c @@ -40,7 +40,6 @@ int tdbTxnCloseImpl(TXN *pTxn) { if (pTxn->jfd) { TAOS_UNUSED(tdbOsClose(pTxn->jfd)); - ASSERT(pTxn->jfd == NULL); } tdbOsFree(pTxn); diff --git a/source/libs/tdb/src/inc/tdbInt.h b/source/libs/tdb/src/inc/tdbInt.h index 605fe6a1a4..7e97be962b 100644 --- a/source/libs/tdb/src/inc/tdbInt.h +++ b/source/libs/tdb/src/inc/tdbInt.h @@ -319,7 +319,6 @@ static inline int tdbTryLockPage(tdb_spinlock_t *pLock) { } else if (ret == EBUSY) { return P_LOCK_BUSY; } else { - ASSERT(0); return P_LOCK_FAIL; } } @@ -354,7 +353,10 @@ static inline SCell *tdbPageGetCell(SPage *pPage, int idx) { int iOvfl; int lidx; - ASSERT(idx >= 0 && idx < TDB_PAGE_TOTAL_CELLS(pPage)); + if (idx < 0 || idx >= TDB_PAGE_TOTAL_CELLS(pPage)) { + terrno = TSDB_CODE_INVALID_PARA; + return NULL; + } iOvfl = 0; for (; iOvfl < pPage->nOverflow; iOvfl++) { @@ -367,7 +369,6 @@ static inline SCell *tdbPageGetCell(SPage *pPage, int idx) { } lidx = idx - iOvfl; - ASSERT(lidx >= 0 && lidx < pPage->pPageMethods->getCellNum(pPage)); pCell = pPage->pData + pPage->pPageMethods->getCellOffset(pPage, lidx); return pCell; diff --git a/source/libs/tdb/src/inc/tdbUtil.h b/source/libs/tdb/src/inc/tdbUtil.h index 4382513f73..22468e6579 100644 --- a/source/libs/tdb/src/inc/tdbUtil.h +++ b/source/libs/tdb/src/inc/tdbUtil.h @@ -22,12 +22,6 @@ extern "C" { #endif -#if __STDC_VERSION__ >= 201112LL -#define TDB_STATIC_ASSERT(op, info) static_assert(op, info) -#else -#define TDB_STATIC_ASSERT(op, info) -#endif - #define TDB_ROUND8(x) (((x) + 7) & ~7) int tdbGnrtFileID(tdb_fd_t fd, uint8_t *fileid, bool unique); diff --git a/source/libs/tfs/src/tfs.c b/source/libs/tfs/src/tfs.c index f81bbbdeb7..4954e81837 100644 --- a/source/libs/tfs/src/tfs.c +++ b/source/libs/tfs/src/tfs.c @@ -554,7 +554,7 @@ static int32_t tfsCheckAndFormatCfg(STfs *pTfs, SDiskCfg *pCfg) { } STfsDisk *pDisk = NULL; - if ((code = tfsGetDiskByName(pTfs, dirName, NULL)) != 0) { + if ((code = tfsGetDiskByName(pTfs, dirName, &pDisk)) != 0) { fError("failed to mount %s to FS since %s", pCfg->dir, tstrerror(code)); TAOS_RETURN(code); } diff --git a/source/libs/transport/inc/transComm.h b/source/libs/transport/inc/transComm.h index 820075787f..e8172385b2 100644 --- a/source/libs/transport/inc/transComm.h +++ b/source/libs/transport/inc/transComm.h @@ -240,6 +240,7 @@ typedef struct { void* pThrd; queue qmsg; TdThreadMutex mtx; // protect qmsg; + int64_t num; } SAsyncItem; typedef struct { diff --git a/source/libs/transport/inc/transLog.h b/source/libs/transport/inc/transLog.h index 121939787c..278356b42e 100644 --- a/source/libs/transport/inc/transLog.h +++ b/source/libs/transport/inc/transLog.h @@ -32,12 +32,12 @@ extern "C" { #define tTrace(...) { if (rpcDebugFlag & DEBUG_TRACE) { taosPrintLog("RPC ", DEBUG_TRACE, rpcDebugFlag, __VA_ARGS__); }} #define tDump(x, y) { if (rpcDebugFlag & DEBUG_DUMP) { taosDumpData((unsigned char *)x, y); } } -#define tGTrace(param, ...) do { if (rpcDebugFlag & DEBUG_TRACE){char buf[40] = {0}; TRACE_TO_STR(trace, buf); tTrace(param ", gtid:%s", __VA_ARGS__, buf);}} while(0) -#define tGFatal(param, ...) do {if (rpcDebugFlag & DEBUG_FATAL){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); tFatal(param ", gtid:%s", __VA_ARGS__, buf); }} while (0) -#define tGError(param, ...) do { if (rpcDebugFlag & DEBUG_ERROR){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); tError(param ", gtid:%s", __VA_ARGS__, buf);} } while(0) -#define tGWarn(param, ...) do { if (rpcDebugFlag & DEBUG_WARN) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); tWarn(param ", gtid:%s", __VA_ARGS__, buf); }} while(0) -#define tGInfo(param, ...) do { if (rpcDebugFlag & DEBUG_INFO) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); tInfo(param ", gtid:%s", __VA_ARGS__, buf); }} while(0) -#define tGDebug(param,...) do {if (rpcDebugFlag & DEBUG_DEBUG){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); tDebug(param ", gtid:%s", __VA_ARGS__, buf); }} while(0) +#define tGTrace(param, ...) do { if (rpcDebugFlag & DEBUG_TRACE){char buf[40] = {0}; TRACE_TO_STR(trace, buf); tTrace(param ", qid:%s", __VA_ARGS__, buf);}} while(0) +#define tGFatal(param, ...) do {if (rpcDebugFlag & DEBUG_FATAL){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); tFatal(param ", qid:%s", __VA_ARGS__, buf); }} while (0) +#define tGError(param, ...) do { if (rpcDebugFlag & DEBUG_ERROR){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); tError(param ", qid:%s", __VA_ARGS__, buf);} } while(0) +#define tGWarn(param, ...) do { if (rpcDebugFlag & DEBUG_WARN) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); tWarn(param ", qid:%s", __VA_ARGS__, buf); }} while(0) +#define tGInfo(param, ...) do { if (rpcDebugFlag & DEBUG_INFO) { char buf[40] = {0}; TRACE_TO_STR(trace, buf); tInfo(param ", qid:%s", __VA_ARGS__, buf); }} while(0) +#define tGDebug(param,...) do {if (rpcDebugFlag & DEBUG_DEBUG){ char buf[40] = {0}; TRACE_TO_STR(trace, buf); tDebug(param ", qid:%s", __VA_ARGS__, buf); }} while(0) // clang-format on diff --git a/source/libs/transport/src/thttp.c b/source/libs/transport/src/thttp.c index 908468f094..6096f69ca2 100644 --- a/source/libs/transport/src/thttp.c +++ b/source/libs/transport/src/thttp.c @@ -30,6 +30,8 @@ static int32_t FAST_FAILURE_LIMIT = 1; static int64_t httpDefaultChanId = -1; +static int64_t httpSeqNum = 0; + typedef struct SHttpModule { uv_loop_t* loop; SAsyncPool* asyncPool; @@ -50,6 +52,7 @@ typedef struct SHttpMsg { EHttpCompFlag flag; int8_t quit; int64_t chanId; + int64_t seq; } SHttpMsg; typedef struct SHttpClient { @@ -62,6 +65,7 @@ typedef struct SHttpClient { uint16_t port; struct sockaddr_in dest; int64_t chanId; + int64_t seq; } SHttpClient; typedef struct SHttpConnList { @@ -193,14 +197,14 @@ static FORCE_INLINE int32_t taosBuildDstAddr(const char* server, uint16_t port, uint32_t ip = 0; int32_t code = taosGetIpv4FromFqdn(server, &ip); if (code) { - tError("http-report failed to resolving domain names: %s", server); + tError("http-report failed to resolving domain names %s, reason: %s", server, tstrerror(code)); return TSDB_CODE_RPC_FQDN_ERROR; } char buf[256] = {0}; tinet_ntoa(buf, ip); int ret = uv_ip4_addr(buf, port, dest); if (ret != 0) { - tError("http-report failed to get addr %s", uv_err_name(ret)); + tError("http-report failed to get addr, reason:%s", uv_err_name(ret)); return TSDB_CODE_THIRDPARTY_ERROR; } return 0; @@ -215,14 +219,15 @@ static void* httpThread(void* arg) { static int32_t httpCreateMsg(const char* server, const char* uri, uint16_t port, char* pCont, int32_t contLen, EHttpCompFlag flag, int64_t chanId, SHttpMsg** httpMsg) { + int64_t seqNum = atomic_fetch_add_64(&httpSeqNum, 1); if (server == NULL || uri == NULL) { - tError("http-report failed to report to invalid addr, chanId:%" PRId64 "", chanId); + tError("http-report failed to report to invalid addr, chanId:%" PRId64 ", seq:%" PRId64 "", chanId, seqNum); *httpMsg = NULL; return TSDB_CODE_INVALID_PARA; } if (pCont == NULL || contLen == 0) { - tError("http-report failed to report empty packet, chanId:%" PRId64 "", chanId); + tError("http-report failed to report empty packet, chanId:%" PRId64 ", seq:%" PRId64 "", chanId, seqNum); *httpMsg = NULL; return TSDB_CODE_INVALID_PARA; } @@ -233,6 +238,7 @@ static int32_t httpCreateMsg(const char* server, const char* uri, uint16_t port, return TSDB_CODE_OUT_OF_MEMORY; } + msg->seq = seqNum; msg->port = port; msg->server = taosStrdup(server); msg->uri = taosStrdup(uri); @@ -259,7 +265,11 @@ static void httpDestroyMsg(SHttpMsg* msg) { taosMemoryFree(msg->cont); taosMemoryFree(msg); } -static void httpDestroyMsgWrapper(void* cont, void* param) { httpDestroyMsg((SHttpMsg*)cont); } +static void httpDestroyMsgWrapper(void* cont, void* param) { + SHttpMsg* pMsg = cont; + tWarn("http-report destroy msg, chanId:%" PRId64 ", seq:%" PRId64 "", pMsg->chanId, pMsg->seq); + httpDestroyMsg(pMsg); +} static void httpMayDiscardMsg(SHttpModule* http, SAsyncItem* item) { SHttpMsg *msg = NULL, *quitMsg = NULL; @@ -272,6 +282,8 @@ static void httpMayDiscardMsg(SHttpModule* http, SAsyncItem* item) { QUEUE_REMOVE(h); msg = QUEUE_DATA(h, SHttpMsg, q); if (!msg->quit) { + tError("http-report failed to report chanId:%" PRId64 ",seq:%" PRId64 ", reason: %s", msg->chanId, msg->seq, + tstrerror(TSDB_CODE_HTTP_MODULE_QUIT)); httpDestroyMsg(msg); } else { quitMsg = msg; @@ -281,6 +293,25 @@ static void httpMayDiscardMsg(SHttpModule* http, SAsyncItem* item) { QUEUE_PUSH(&item->qmsg, &quitMsg->q); } } +static void httpTrace(queue* q) { + if (!(rpcDebugFlag & DEBUG_DEBUG) || (QUEUE_IS_EMPTY(q))) { + return; + } + + int64_t startSeq = 0, endSeq = 0; + SHttpMsg* msg = NULL; + + queue* h = QUEUE_HEAD(q); + msg = QUEUE_DATA(h, SHttpMsg, q); + startSeq = msg->seq; + + h = QUEUE_TAIL(q); + msg = QUEUE_DATA(h, SHttpMsg, q); + endSeq = msg->seq; + + tDebug("http-report process msg, start_seq:%" PRId64 ", end_seq:%" PRId64 ", max_seq:%" PRId64 "", startSeq, endSeq, + atomic_load_64(&httpSeqNum) - 1); +} static void httpAsyncCb(uv_async_t* handle) { SAsyncItem* item = handle->data; @@ -290,7 +321,7 @@ static void httpAsyncCb(uv_async_t* handle) { queue wq; QUEUE_INIT(&wq); - static int32_t BATCH_SIZE = 5; + static int32_t BATCH_SIZE = 20; int32_t count = 0; (void)taosThreadMutexLock(&item->mtx); @@ -303,6 +334,8 @@ static void httpAsyncCb(uv_async_t* handle) { } (void)taosThreadMutexUnlock(&item->mtx); + httpTrace(&wq); + while (!QUEUE_IS_EMPTY(&wq)) { queue* h = QUEUE_HEAD(&wq); QUEUE_REMOVE(h); @@ -345,47 +378,50 @@ static FORCE_INLINE void clientAllocBuffCb(uv_handle_t* handle, size_t suggested } static FORCE_INLINE void clientRecvCb(uv_stream_t* handle, ssize_t nread, const uv_buf_t* buf) { + STUB_RAND_NETWORK_ERR(nread); SHttpClient* cli = handle->data; if (nread < 0) { - tError("http-report recv error:%s", uv_strerror(nread)); + tError("http-report recv error:%s, seq:%" PRId64 "", uv_strerror(nread), cli->seq); } else { - tTrace("http-report succ to recv %d bytes", (int32_t)nread); + tTrace("http-report succ to recv %d bytes, seq:%" PRId64 "", (int32_t)nread, cli->seq); } if (!uv_is_closing((uv_handle_t*)&cli->tcp)) { uv_close((uv_handle_t*)&cli->tcp, clientCloseCb); } } static void clientSentCb(uv_write_t* req, int32_t status) { + STUB_RAND_NETWORK_ERR(status); SHttpClient* cli = req->data; if (status != 0) { - tError("http-report failed to send data, reason: %s, dst:%s:%d, chanId:%" PRId64 "", uv_strerror(status), cli->addr, - cli->port, cli->chanId); + tError("http-report failed to send data, reason: %s, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 "", + uv_strerror(status), cli->addr, cli->port, cli->chanId, cli->seq); if (!uv_is_closing((uv_handle_t*)&cli->tcp)) { uv_close((uv_handle_t*)&cli->tcp, clientCloseCb); } return; } else { - tTrace("http-report succ to send data, chanId:%" PRId64 "", cli->chanId); + tTrace("http-report succ to send data, chanId:%" PRId64 ", seq:%" PRId64 "", cli->chanId, cli->seq); } + status = uv_read_start((uv_stream_t*)&cli->tcp, clientAllocBuffCb, clientRecvCb); if (status != 0) { - tError("http-report failed to recv data,reason:%s, dst:%s:%d, chanId:%" PRId64 "", uv_strerror(status), cli->addr, - cli->port, cli->chanId); + tError("http-report failed to recv data,reason:%s, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 "", + uv_strerror(status), cli->addr, cli->port, cli->chanId, cli->seq); if (!uv_is_closing((uv_handle_t*)&cli->tcp)) { uv_close((uv_handle_t*)&cli->tcp, clientCloseCb); } } } static void clientConnCb(uv_connect_t* req, int32_t status) { + STUB_RAND_NETWORK_ERR(status); SHttpClient* cli = req->data; int64_t chanId = cli->chanId; SHttpModule* http = taosAcquireRef(httpRefMgt, chanId); if (status != 0) { httpFailFastMayUpdate(http->connStatusTable, cli->addr, cli->port, 0); - - tError("http-report failed to conn to server, reason:%s, dst:%s:%d, chanId:%" PRId64 "", uv_strerror(status), - cli->addr, cli->port, chanId); + tError("http-report failed to conn to server, reason:%s, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 "", + uv_strerror(status), cli->addr, cli->port, chanId, cli->seq); if (!uv_is_closing((uv_handle_t*)&cli->tcp)) { uv_close((uv_handle_t*)&cli->tcp, clientCloseCb); } @@ -398,8 +434,8 @@ static void clientConnCb(uv_connect_t* req, int32_t status) { status = uv_write(&cli->req, (uv_stream_t*)&cli->tcp, cli->wbuf, 2, clientSentCb); if (0 != status) { - tError("http-report failed to send data,reason:%s, dst:%s:%d, chanId:%" PRId64 "", uv_strerror(status), cli->addr, - cli->port, chanId); + tError("http-report failed to send data,reason:%s, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 "", + uv_strerror(status), cli->addr, cli->port, chanId, cli->seq); if (!uv_is_closing((uv_handle_t*)&cli->tcp)) { uv_close((uv_handle_t*)&cli->tcp, clientCloseCb); } @@ -412,7 +448,7 @@ int32_t httpSendQuit(SHttpModule* http, int64_t chanId) { if (msg == NULL) { return TSDB_CODE_OUT_OF_MEMORY; } - + msg->seq = atomic_fetch_add_64(&httpSeqNum, 1); msg->quit = 1; msg->chanId = chanId; @@ -442,10 +478,11 @@ static void httpWalkCb(uv_handle_t* handle, void* arg) { return; } static void httpHandleQuit(SHttpMsg* msg) { + int64_t seq = msg->seq; int64_t chanId = msg->chanId; taosMemoryFree(msg); - tDebug("http-report receive quit, chanId:%" PRId64 "", chanId); + tDebug("http-report receive quit, chanId:%" PRId64 ", seq:%" PRId64 "", chanId, seq); SHttpModule* http = taosAcquireRef(httpRefMgt, chanId); if (http == NULL) return; uv_walk(http->loop, httpWalkCb, NULL); @@ -542,7 +579,7 @@ static void httpHandleReq(SHttpMsg* msg) { code = TSDB_CODE_OUT_OF_MEMORY; goto END; } - + cli->seq = msg->seq; cli->conn.data = cli; cli->tcp.data = cli; cli->req.data = cli; @@ -556,8 +593,8 @@ static void httpHandleReq(SHttpMsg* msg) { cli->wbuf = wb; cli->rbuf = taosMemoryCalloc(1, HTTP_RECV_BUF_SIZE); if (cli->rbuf == NULL) { - tError("http-report failed to alloc read buf, dst:%s:%d,chanId:%" PRId64 ", reason:%s", cli->addr, cli->port, - chanId, tstrerror(TSDB_CODE_OUT_OF_MEMORY)); + tError("http-report failed to alloc read buf, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 ",reason:%s", cli->addr, + cli->port, chanId, cli->seq, tstrerror(TSDB_CODE_OUT_OF_MEMORY)); destroyHttpClient(cli); (void)taosReleaseRef(httpRefMgt, chanId); return; @@ -565,8 +602,8 @@ static void httpHandleReq(SHttpMsg* msg) { int err = uv_tcp_init(http->loop, &cli->tcp); if (err != 0) { - tError("http-report failed to init socket handle, dst:%s:%d,chanId:%" PRId64 ", reason:%s", cli->addr, cli->port, - chanId, uv_strerror(err)); + tError("http-report failed to init socket handle, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 ", reason:%s", + cli->addr, cli->port, chanId, cli->seq, uv_strerror(err)); destroyHttpClient(cli); (void)taosReleaseRef(httpRefMgt, chanId); return; @@ -575,8 +612,8 @@ static void httpHandleReq(SHttpMsg* msg) { // set up timeout to avoid stuck; int32_t fd = taosCreateSocketWithTimeout(5000); if (fd < 0) { - tError("http-report failed to open socket, dst:%s:%d, chanId:%" PRId64 ", reason:%s", cli->addr, cli->port, chanId, - tstrerror(TAOS_SYSTEM_ERROR(errno))); + tError("http-report failed to open socket, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 ", reason:%s", cli->addr, + cli->port, chanId, cli->seq, tstrerror(TAOS_SYSTEM_ERROR(errno))); destroyHttpClient(cli); (void)taosReleaseRef(httpRefMgt, chanId); return; @@ -584,8 +621,9 @@ static void httpHandleReq(SHttpMsg* msg) { int ret = uv_tcp_open((uv_tcp_t*)&cli->tcp, fd); if (ret != 0) { - tError("http-report failed to open socket, reason:%s, dst:%s:%d, chanId:%" PRId64 ",reason:%s", uv_strerror(ret), - cli->addr, cli->port, chanId, uv_strerror(ret)); + tError("http-report failed to open socket, dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 ", reason:%s", cli->addr, + cli->port, chanId, cli->seq, uv_strerror(ret)); + destroyHttpClient(cli); (void)taosReleaseRef(httpRefMgt, chanId); return; @@ -593,8 +631,8 @@ static void httpHandleReq(SHttpMsg* msg) { ret = uv_tcp_connect(&cli->conn, &cli->tcp, (const struct sockaddr*)&cli->dest, clientConnCb); if (ret != 0) { - tError("http-report failed to connect to http-server,dst:%s:%d, chanId:%" PRId64 ",reson:%s", cli->addr, cli->port, - chanId, uv_strerror(ret)); + tError("http-report failed to connect to http-server,dst:%s:%d, chanId:%" PRId64 ", seq:%" PRId64 ", reson:%s", + cli->addr, cli->port, chanId, cli->seq, uv_strerror(ret)); httpFailFastMayUpdate(http->connStatusTable, cli->addr, cli->port, 0); destroyHttpClient(cli); } @@ -603,8 +641,8 @@ static void httpHandleReq(SHttpMsg* msg) { END: if (ignore == false) { - tError("http-report failed to report to addr: %s:%d, chanId:%" PRId64 ", reason:%s", msg->server, msg->port, chanId, - tstrerror(code)); + tError("http-report failed to report to addr: %s:%d, chanId:%" PRId64 ",seq:%" PRId64 " reason:%s", msg->server, + msg->port, chanId, msg->seq, tstrerror(code)); } httpDestroyMsg(msg); taosMemoryFree(header); @@ -652,6 +690,7 @@ static int32_t taosSendHttpReportImplByChan(const char* server, const char* uri, code = TSDB_CODE_HTTP_MODULE_QUIT; goto _ERROR; } + tDebug("http-report start to report, chanId:%" PRId64 ", seq:%" PRId64 "", chanId, msg->seq); code = transAsyncSend(load->asyncPool, &(msg->q)); if (code != 0) { @@ -661,6 +700,11 @@ static int32_t taosSendHttpReportImplByChan(const char* server, const char* uri, msg = NULL; _ERROR: + + if (code != 0) { + tError("http-report failed to report reason:%s, chanId:%" PRId64 ", seq:%" PRId64 "", tstrerror(code), chanId, + msg->seq); + } httpDestroyMsg(msg); if (load != NULL) taosReleaseRef(httpRefMgt, chanId); return code; @@ -684,6 +728,7 @@ int64_t transInitHttpChanImpl(); static void transHttpEnvInit() { httpRefMgt = taosOpenRef(64, transHttpDestroyHandle); httpDefaultChanId = transInitHttpChanImpl(); + httpSeqNum = 0; } void transHttpEnvDestroy() { @@ -748,7 +793,7 @@ int64_t taosInitHttpChan() { } void taosDestroyHttpChan(int64_t chanId) { - tDebug("http-report send quit, chanId:%" PRId64 "", chanId); + tDebug("http-report send quit, chanId: %" PRId64 "", chanId); int ret = 0; SHttpModule* load = taosAcquireRef(httpRefMgt, chanId); @@ -772,4 +817,4 @@ void taosDestroyHttpChan(int64_t chanId) { (void)taosReleaseRef(httpRefMgt, chanId); (void)taosRemoveRef(httpRefMgt, chanId); -} \ No newline at end of file +} diff --git a/source/libs/transport/src/trans.c b/source/libs/transport/src/trans.c index 8b99443a84..1c783f44fe 100644 --- a/source/libs/transport/src/trans.c +++ b/source/libs/transport/src/trans.c @@ -140,7 +140,6 @@ void* rpcMallocCont(int64_t contLen) { char* start = taosMemoryCalloc(1, size); if (start == NULL) { tError("failed to malloc msg, size:%" PRId64, size); - terrno = TSDB_CODE_OUT_OF_MEMORY; return NULL; } else { tTrace("malloc mem:%p size:%" PRId64, start, size); diff --git a/source/libs/transport/src/transCli.c b/source/libs/transport/src/transCli.c index 862a74c72b..336a182153 100644 --- a/source/libs/transport/src/transCli.c +++ b/source/libs/transport/src/transCli.c @@ -923,10 +923,12 @@ static void cliAllocRecvBufferCb(uv_handle_t* handle, size_t suggested_size, uv_ } } static void cliRecvCb(uv_stream_t* handle, ssize_t nread, const uv_buf_t* buf) { - // impl later + STUB_RAND_NETWORK_ERR(nread); + if (handle->data == NULL) { return; } + SCliConn* conn = handle->data; SConnBuffer* pBuf = &conn->readBuf; if (nread > 0) { @@ -1117,6 +1119,8 @@ static bool cliHandleNoResp(SCliConn* conn) { return res; } static void cliSendCb(uv_write_t* req, int status) { + STUB_RAND_NETWORK_ERR(status); + SCliConn* pConn = transReqQueueRemove(req); if (pConn == NULL) return; @@ -1434,6 +1438,7 @@ static void cliHandleBatchReq(SCliBatch* pBatch, SCliThrd* pThrd) { cliSendBatch(conn); } static void cliSendBatchCb(uv_write_t* req, int status) { + STUB_RAND_NETWORK_ERR(status); SCliConn* conn = req->data; SCliThrd* thrd = conn->hostThrd; SCliBatch* p = conn->pBatch; @@ -1523,6 +1528,8 @@ void cliConnCb(uv_connect_t* req, int status) { pConn->timer = NULL; } + STUB_RAND_NETWORK_ERR(status); + if (status != 0) { cliMayUpdateFqdnCache(pThrd->fqdn2ipCache, pConn->dstAddr); if (timeout == false) { @@ -2193,7 +2200,6 @@ bool cliRecvReleaseReq(SCliConn* conn, STransMsgHead* pHead) { for (int i = 0; ahandle == 0 && i < transQueueSize(&conn->cliMsgs); i++) { SCliMsg* cliMsg = transQueueGet(&conn->cliMsgs, i); if (cliMsg->type == Release) { - ASSERTS(pMsg == NULL, "trans-cli recv invaid release-req"); tDebug("%s conn %p receive release request, refId:%" PRId64 ", ignore msg", CONN_GET_INST_LABEL(conn), conn, conn->refId); cliDestroyConn(conn, true); @@ -3015,7 +3021,6 @@ _exception: } int32_t transSendRequestWithId(void* shandle, const SEpSet* pEpSet, STransMsg* pReq, int64_t* transpointId) { if (transpointId == NULL) { - ASSERT(0); return TSDB_CODE_INVALID_PARA; } int32_t code = 0; diff --git a/source/libs/transport/src/transComm.c b/source/libs/transport/src/transComm.c index 5d82e157b3..a0836eae3d 100644 --- a/source/libs/transport/src/transComm.c +++ b/source/libs/transport/src/transComm.c @@ -183,7 +183,7 @@ int32_t transResetBuffer(SConnBuffer* connBuf, int8_t resetBuf) { } } } else { - ASSERTS(0, "invalid read from sock buf"); + tError("failed to reset buffer, total:%d, len:%d, reason:%s", p->total, p->len, tstrerror(TSDB_CODE_INVALID_MSG)); return TSDB_CODE_INVALID_MSG; } return 0; @@ -869,3 +869,8 @@ int32_t transUtilSWhiteListToStr(SIpWhiteList* pList, char** ppBuf) { *ppBuf = pBuf; return len; } + +// int32_t transGenRandomError(int32_t status) { +// STUB_RAND_NETWORK_ERR(status) +// return status; +// } diff --git a/source/libs/transport/src/transSvr.c b/source/libs/transport/src/transSvr.c index c1b934c812..53a7dee7be 100644 --- a/source/libs/transport/src/transSvr.c +++ b/source/libs/transport/src/transSvr.c @@ -468,8 +468,8 @@ static bool uvHandleReq(SSvrConn* pConn) { tGTrace("%s handle %p conn:%p translated to app, refId:%" PRIu64, transLabel(pTransInst), transMsg.info.handle, pConn, pConn->refId); - ASSERTS(transMsg.info.handle != NULL, "trans-svr failed to alloc handle to msg"); if (transMsg.info.handle == NULL) { + tError("%s handle %p conn:%p handle failed to init" PRIu64, transLabel(pTransInst), transMsg.info.handle, pConn); return false; } @@ -493,6 +493,8 @@ void uvOnRecvCb(uv_stream_t* cli, ssize_t nread, const uv_buf_t* buf) { SSvrConn* conn = cli->data; SWorkThrd* pThrd = conn->hostThrd; + STUB_RAND_NETWORK_ERR(nread); + if (true == pThrd->quit) { tInfo("work thread received quit msg, destroy conn"); destroyConn(conn, true); @@ -553,6 +555,7 @@ void uvOnTimeoutCb(uv_timer_t* handle) { } void uvOnSendCb(uv_write_t* req, int status) { + STUB_RAND_NETWORK_ERR(status); SSvrConn* conn = transReqQueueRemove(req); if (conn == NULL) return; @@ -602,6 +605,7 @@ void uvOnSendCb(uv_write_t* req, int status) { } } static void uvOnPipeWriteCb(uv_write_t* req, int status) { + STUB_RAND_NETWORK_ERR(status); if (status == 0) { tTrace("success to dispatch conn to work thread"); } else { @@ -949,6 +953,7 @@ void uvOnAcceptCb(uv_stream_t* stream, int status) { } } void uvOnConnectionCb(uv_stream_t* q, ssize_t nread, const uv_buf_t* buf) { + STUB_RAND_NETWORK_ERR(nread); if (nread < 0) { if (nread != UV_EOF) { tError("read error %s", uv_err_name(nread)); @@ -960,15 +965,19 @@ void uvOnConnectionCb(uv_stream_t* q, ssize_t nread, const uv_buf_t* buf) { return; } // free memory allocated by - ASSERTS(nread == strlen(notify), "trans-svr mem corrupted"); - ASSERTS(buf->base[0] == notify[0], "trans-svr mem corrupted"); + if (nread != strlen(notify) || strncmp(buf->base, notify, strlen(notify)) != 0) { + tError("failed to read pip "); + taosMemoryFree(buf->base); + uv_close((uv_handle_t*)q, NULL); + } + taosMemoryFree(buf->base); SWorkThrd* pThrd = q->data; uv_pipe_t* pipe = (uv_pipe_t*)q; if (!uv_pipe_pending_count(pipe)) { - tError("No pending count"); + tError("no pending count, unexpected error"); uv_close((uv_handle_t*)q, NULL); return; } @@ -1041,9 +1050,11 @@ void* transAcceptThread(void* arg) { return NULL; } void uvOnPipeConnectionCb(uv_connect_t* connect, int status) { + STUB_RAND_NETWORK_ERR(status); if (status != 0) { return; - } + }; + SWorkThrd* pThrd = container_of(connect, SWorkThrd, connect_req); (void)uv_read_start((uv_stream_t*)pThrd->pipe, uvAllocConnBufferCb, uvOnConnectionCb); } @@ -1344,25 +1355,32 @@ static void uvPipeListenCb(uv_stream_t* handle, int status) { uv_pipe_t* pipe = &(srv->pipe[srv->numOfWorkerReady][0]); int ret = uv_pipe_init(srv->loop, pipe, 1); - ASSERTS(ret == 0, "trans-svr failed to init pipe"); - if (ret != 0) return; + if (ret != 0) { + tError("trans-svr failed to init pipe, errmsg: %s", uv_err_name(ret)); + } ret = uv_accept((uv_stream_t*)&srv->pipeListen, (uv_stream_t*)pipe); - ASSERTS(ret == 0, "trans-svr failed to accept pipe msg"); - if (ret != 0) return; + if (ret != 0) { + tError("trans-svr failed to accept pipe, errmsg: %s", uv_err_name(ret)); + return; + } ret = uv_is_readable((uv_stream_t*)pipe); - ASSERTS(ret == 1, "trans-svr pipe status corrupted"); - if (ret != 1) return; - + if (ret != 1) { + tError("trans-svr failed to check pipe, pip not readable"); + return; + } ret = uv_is_writable((uv_stream_t*)pipe); - ASSERTS(ret == 1, "trans-svr pipe status corrupted"); - if (ret != 1) return; + if (ret != 1) { + tError("trans-svr failed to check pipe, pip not writable"); + return; + } ret = uv_is_closing((uv_handle_t*)pipe); - ASSERTS(ret == 0, "trans-svr pipe status corrupted"); - if (ret != 0) return; - + if (ret != 0) { + tError("trans-svr failed to check pipe, pip is closing"); + return; + } srv->numOfWorkerReady++; } diff --git a/source/libs/wal/src/walMeta.c b/source/libs/wal/src/walMeta.c index db1d61a023..0eb011113a 100644 --- a/source/libs/wal/src/walMeta.c +++ b/source/libs/wal/src/walMeta.c @@ -357,7 +357,10 @@ static int32_t walLogEntriesComplete(const SWal* pWal) { static int32_t walTrimIdxFile(SWal* pWal, int32_t fileIdx) { SWalFileInfo* pFileInfo = taosArrayGet(pWal->fileInfoSet, fileIdx); - ASSERT(pFileInfo != NULL); + if (!pFileInfo) { + TAOS_RETURN(TSDB_CODE_FAILED); + } + char fnameStr[WAL_FILE_LEN]; walBuildIdxName(pWal, pFileInfo->firstVer, fnameStr); @@ -414,7 +417,13 @@ int32_t walCheckAndRepairMeta(SWal* pWal) { SWalFileInfo fileInfo; (void)memset(&fileInfo, -1, sizeof(SWalFileInfo)); (void)sscanf(name, "%" PRId64 ".log", &fileInfo.firstVer); - (void)taosArrayPush(actualLog, &fileInfo); + if (!taosArrayPush(actualLog, &fileInfo)) { + regfree(&logRegPattern); + regfree(&idxRegPattern); + (void)taosCloseDir(&pDir); + + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } } } @@ -727,9 +736,12 @@ int32_t walRollFileInfo(SWal* pWal) { pNewInfo->closeTs = -1; pNewInfo->fileSize = 0; pNewInfo->syncedOffset = 0; - (void)taosArrayPush(pArray, pNewInfo); - taosMemoryFree(pNewInfo); + if (!taosArrayPush(pArray, pNewInfo)) { + taosMemoryFree(pNewInfo); + TAOS_RETURN(TSDB_CODE_OUT_OF_MEMORY); + } + taosMemoryFree(pNewInfo); TAOS_RETURN(TSDB_CODE_SUCCESS); } diff --git a/source/libs/wal/src/walWrite.c b/source/libs/wal/src/walWrite.c index 9979ddd0b0..52af3e8528 100644 --- a/source/libs/wal/src/walWrite.c +++ b/source/libs/wal/src/walWrite.c @@ -371,8 +371,11 @@ static FORCE_INLINE int32_t walCheckAndRoll(SWal *pWal) { int32_t walBeginSnapshot(SWal *pWal, int64_t ver, int64_t logRetention) { int32_t code = 0; + if (logRetention < 0) { + TAOS_RETURN(TSDB_CODE_FAILED); + } + TAOS_UNUSED(taosThreadMutexLock(&pWal->mutex)); - ASSERT(logRetention >= 0); pWal->vers.verInSnapshotting = ver; pWal->vers.logRetention = logRetention; @@ -438,7 +441,10 @@ int32_t walEndSnapshot(SWal *pWal) { if (pInfo) { wDebug("vgId:%d, wal search found file info. ver:%" PRId64 ", first:%" PRId64 " last:%" PRId64, pWal->cfg.vgId, ver, pInfo->firstVer, pInfo->lastVer); - ASSERT(ver <= pInfo->lastVer); + if (ver > pInfo->lastVer) { + TAOS_CHECK_GOTO(TSDB_CODE_FAILED, &lino, _exit); + } + if (ver == pInfo->lastVer) { pInfo++; } diff --git a/source/os/src/osFile.c b/source/os/src/osFile.c index a5df4f63f3..144baeb148 100644 --- a/source/os/src/osFile.c +++ b/source/os/src/osFile.c @@ -454,7 +454,6 @@ int64_t taosPWriteFile(TdFilePtr pFile, const void *buf, int64_t count, int64_t #if FILE_WITH_LOCK taosThreadRwlockWrlock(&(pFile->rwlock)); #endif - ASSERT(pFile->hFile != NULL); // Please check if you have closed the file. if (pFile->hFile == NULL) { #if FILE_WITH_LOCK taosThreadRwlockUnlock(&(pFile->rwlock)); @@ -867,7 +866,6 @@ int64_t taosLSeekFile(TdFilePtr pFile, int64_t offset, int32_t whence) { #endif int32_t code = 0; - ASSERT(pFile->fd >= 0); // Please check if you have closed the file. #ifdef WINDOWS int64_t ret = _lseeki64(pFile->fd, offset, whence); @@ -906,7 +904,7 @@ int32_t taosFStatFile(TdFilePtr pFile, int64_t *size, int32_t *mtime) { int32_t code = _fstat64(pFile->fd, &fileStat); #else struct stat fileStat; - int32_t code = fstat(pFile->fd, &fileStat); + int32_t code = fstat(pFile->fd, &fileStat); #endif if (-1 == code) { terrno = TAOS_SYSTEM_ERROR(errno); @@ -1479,7 +1477,6 @@ int32_t taosEOFFile(TdFilePtr pFile) { terrno = TSDB_CODE_INVALID_PARA; return -1; } - ASSERT(pFile->fp != NULL); if (pFile->fp == NULL) { terrno = TSDB_CODE_INVALID_PARA; return -1; @@ -1614,7 +1611,7 @@ int taosSeekCFile(FILE *file, int64_t offset, int whence) { #ifdef WINDOWS return _fseeki64(file, offset, whence); #else - int code = fseeko(file, offset, whence); + int code = fseeko(file, offset, whence); if (-1 == code) { terrno = TAOS_SYSTEM_ERROR(errno); } @@ -1636,7 +1633,14 @@ int taosCloseCFile(FILE *f) { return fclose(f); } int taosSetAutoDelFile(char *path) { #ifdef WINDOWS - return SetFileAttributes(path, FILE_ATTRIBUTE_TEMPORARY); + bool succ = SetFileAttributes(path, FILE_ATTRIBUTE_TEMPORARY); + if (succ) { + return 0; + } else { + DWORD error = GetLastError(); + terrno = TAOS_SYSTEM_ERROR(error); + return terrno; + } #else if (-1 == unlink(path)) { terrno = TAOS_SYSTEM_ERROR(errno); diff --git a/source/os/src/osMemory.c b/source/os/src/osMemory.c index 7a5a547354..49a5c2a2a2 100644 --- a/source/os/src/osMemory.c +++ b/source/os/src/osMemory.c @@ -24,7 +24,7 @@ int32_t tsRandErrChance = 1; int64_t tsRandErrDivisor = 10001; -int64_t tsRandErrScope = (RAND_ERR_MEMORY | RAND_ERR_FILE); +int64_t tsRandErrScope = (RAND_ERR_MEMORY | RAND_ERR_FILE | RAND_ERR_NETWORK); threadlocal bool tsEnableRandErr = 0; #if defined(USE_TD_MEMORY) || defined(USE_ADDR2LINE) @@ -326,10 +326,8 @@ void *taosMemoryRealloc(void *ptr, int64_t size) { if (ptr == NULL) return taosMemoryMalloc(size); TdMemoryInfoPtr pTdMemoryInfo = (TdMemoryInfoPtr)((char *)ptr - sizeof(TdMemoryInfo)); - ASSERT(pTdMemoryInfo->symbol == TD_MEMORY_SYMBOL); if (tpTdMemoryInfo->symbol != TD_MEMORY_SYMBOL) { - +return NULL; - + + return NULL; } TdMemoryInfo tdMemoryInfo; @@ -366,7 +364,6 @@ char *taosStrdup(const char *ptr) { if (ptr == NULL) return NULL; TdMemoryInfoPtr pTdMemoryInfo = (TdMemoryInfoPtr)((char *)ptr - sizeof(TdMemoryInfo)); - ASSERT(pTdMemoryInfo->symbol == TD_MEMORY_SYMBOL); if (pTdMemoryInfo->symbol != TD_MEMORY_SYMBOL) { return NULL; } @@ -388,7 +385,7 @@ char *taosStrdup(const char *ptr) { } #endif - return tstrdup(ptr); + return tstrdup(ptr); #endif } @@ -413,7 +410,6 @@ int64_t taosMemorySize(void *ptr) { #ifdef USE_TD_MEMORY TdMemoryInfoPtr pTdMemoryInfo = (TdMemoryInfoPtr)((char *)ptr - sizeof(TdMemoryInfo)); - ASSERT(pTdMemoryInfo->symbol == TD_MEMORY_SYMBOL); if (pTdMemoryInfo->symbol != TD_MEMORY_SYMBOL) { return NULL; } @@ -441,7 +437,7 @@ void taosMemoryTrim(int32_t size) { void *taosMemoryMallocAlign(uint32_t alignment, int64_t size) { #ifdef USE_TD_MEMORY - ASSERT(0); + return NULL; #else #if defined(LINUX) #ifdef BUILD_WITH_RAND_ERR diff --git a/source/os/src/osSocket.c b/source/os/src/osSocket.c index 081ed46c9a..2d160b277b 100644 --- a/source/os/src/osSocket.c +++ b/source/os/src/osSocket.c @@ -312,8 +312,7 @@ int32_t taosGetSockOpt(TdSocketPtr pSocket, int32_t level, int32_t optname, void return -1; } #ifdef WINDOWS - ASSERT(0); - return 0; + return -1; #else return getsockopt(pSocket->fd, level, optname, optval, (int *)optlen); #endif @@ -681,8 +680,7 @@ int32_t taosKeepTcpAlive(TdSocketPtr pSocket) { int taosGetLocalIp(const char *eth, char *ip) { #if defined(WINDOWS) // DO NOTHAING - ASSERT(0); - return 0; + return -1; #else int fd; struct ifreq ifr; @@ -708,8 +706,7 @@ int taosGetLocalIp(const char *eth, char *ip) { int taosValidIp(uint32_t ip) { #if defined(WINDOWS) // DO NOTHAING - ASSERT(0); - return 0; + return -1; #else int ret = -1; int fd; @@ -1111,7 +1108,7 @@ int32_t taosIgnSIGPIPE() { int32_t taosSetMaskSIGPIPE() { #ifdef WINDOWS - // ASSERT(0); + return -1; #else sigset_t signal_mask; (void)sigemptyset(&signal_mask); diff --git a/source/os/src/osString.c b/source/os/src/osString.c index b0a3615ee5..1c99355e34 100644 --- a/source/os/src/osString.c +++ b/source/os/src/osString.c @@ -425,10 +425,6 @@ int64_t taosStr2Int64(const char *str, char **pEnd, int32_t radix) { int64_t tmp = strtoll(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); #endif return tmp; } @@ -437,10 +433,6 @@ uint64_t taosStr2UInt64(const char *str, char **pEnd, int32_t radix) { uint64_t tmp = strtoull(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); #endif return tmp; } @@ -449,10 +441,6 @@ int32_t taosStr2Int32(const char *str, char **pEnd, int32_t radix) { int32_t tmp = strtol(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); #endif return tmp; } @@ -461,10 +449,6 @@ uint32_t taosStr2UInt32(const char *str, char **pEnd, int32_t radix) { uint32_t tmp = strtol(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); #endif return tmp; } @@ -473,12 +457,6 @@ int16_t taosStr2Int16(const char *str, char **pEnd, int32_t radix) { int32_t tmp = strtol(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); - ASSERT(tmp >= SHRT_MIN); - ASSERT(tmp <= SHRT_MAX); #endif return (int16_t)tmp; } @@ -487,23 +465,12 @@ uint16_t taosStr2UInt16(const char *str, char **pEnd, int32_t radix) { uint32_t tmp = strtoul(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); - ASSERT(tmp <= USHRT_MAX); #endif return (uint16_t)tmp; } int8_t taosStr2Int8(const char *str, char **pEnd, int32_t radix) { int32_t tmp = strtol(str, pEnd, radix); -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); - ASSERT(tmp >= SCHAR_MIN); - ASSERT(tmp <= SCHAR_MAX); -#endif return tmp; } @@ -511,33 +478,17 @@ uint8_t taosStr2UInt8(const char *str, char **pEnd, int32_t radix) { uint32_t tmp = strtoul(str, pEnd, radix); #if defined(DARWIN) || defined(_ALPINE) if (errno == EINVAL) errno = 0; -#endif -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); - ASSERT(tmp <= UCHAR_MAX); #endif return tmp; } double taosStr2Double(const char *str, char **pEnd) { double tmp = strtod(str, pEnd); -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); - ASSERT(tmp != HUGE_VAL); -#endif return tmp; } float taosStr2Float(const char *str, char **pEnd) { float tmp = strtof(str, pEnd); -#ifdef TD_CHECK_STR_TO_INT_ERROR - ASSERT(errno != ERANGE); - ASSERT(errno != EINVAL); - ASSERT(tmp != HUGE_VALF); - ASSERT(tmp != NAN); -#endif return tmp; } diff --git a/source/os/src/osSysinfo.c b/source/os/src/osSysinfo.c index 92e5967416..9a4f0f8a98 100644 --- a/source/os/src/osSysinfo.c +++ b/source/os/src/osSysinfo.c @@ -287,7 +287,7 @@ void taosGetSystemInfo() { int32_t taosGetEmail(char *email, int32_t maxLen) { #ifdef WINDOWS - // ASSERT(0); + return 0; #elif defined(_TD_DARWIN_64) #ifdef CUS_PROMPT const char *filepath = "/usr/local/"CUS_PROMPT"/email"; @@ -1040,7 +1040,6 @@ int32_t taosGetSystemUUID(char *uid, int32_t uidlen) { char *taosGetCmdlineByPID(int pid) { #ifdef WINDOWS - ASSERT(0); return ""; #elif defined(_TD_DARWIN_64) static char cmdline[1024]; diff --git a/source/os/src/osSystem.c b/source/os/src/osSystem.c index 843ee20d5b..a8a9ff681b 100644 --- a/source/os/src/osSystem.c +++ b/source/os/src/osSystem.c @@ -91,7 +91,6 @@ typedef struct FILE TdCmd; #ifdef BUILD_NO_CALL void* taosLoadDll(const char* filename) { #if defined(WINDOWS) - ASSERT(0); return NULL; #elif defined(_TD_DARWIN_64) return NULL; @@ -110,7 +109,6 @@ void* taosLoadDll(const char* filename) { void* taosLoadSym(void* handle, char* name) { #if defined(WINDOWS) - ASSERT(0); return NULL; #elif defined(_TD_DARWIN_64) return NULL; @@ -131,7 +129,6 @@ void* taosLoadSym(void* handle, char* name) { void taosCloseDll(void* handle) { #if defined(WINDOWS) - ASSERT(0); return; #elif defined(_TD_DARWIN_64) return; diff --git a/source/os/src/osThread.c b/source/os/src/osThread.c index 3e37d12759..5a24e7775f 100644 --- a/source/os/src/osThread.c +++ b/source/os/src/osThread.c @@ -784,8 +784,7 @@ int32_t taosThreadSpinDestroy(TdThreadSpinlock *lock) { int32_t taosThreadSpinInit(TdThreadSpinlock *lock, int32_t pshared) { #ifdef TD_USE_SPINLOCK_AS_MUTEX - ASSERT(pshared == 0); - if (pshared != 0) return -1; + if (pshared != 0) return TSDB_CODE_INVALID_PARA; return pthread_mutex_init((pthread_mutex_t *)lock, NULL); #else int32_t code = pthread_spin_init((pthread_spinlock_t *)lock, pshared); diff --git a/source/util/src/talgo.c b/source/util/src/talgo.c index d7cd0348f4..9c241516ca 100644 --- a/source/util/src/talgo.c +++ b/source/util/src/talgo.c @@ -39,8 +39,6 @@ static void median(void *src, int64_t size, int64_t s, int64_t e, const void *pa doswap(elePtrAt(src, size, s), elePtrAt(src, size, e), size, buf); } - ASSERT(comparFn(elePtrAt(src, size, mid), elePtrAt(src, size, s), param) <= 0 && - comparFn(elePtrAt(src, size, s), elePtrAt(src, size, e), param) <= 0); } static void tInsertSort(void *src, int64_t size, int32_t s, int32_t e, const void *param, __ext_compar_fn_t comparFn, @@ -323,7 +321,7 @@ void *taosbsearch(const void *key, const void *base, int32_t nmemb, int32_t size } else if (flags == TD_LT) { return (c > 0) ? p : (midx > 0 ? p - size : NULL); } else { - ASSERT(0); + uError("Invalid bsearch flags:%d", flags); return NULL; } } diff --git a/source/util/src/tarray.c b/source/util/src/tarray.c index 37667e2975..59505dc0c1 100644 --- a/source/util/src/tarray.c +++ b/source/util/src/tarray.c @@ -266,8 +266,9 @@ void* taosArrayInsert(SArray* pArray, size_t index, const void* pData) { } void taosArraySet(SArray* pArray, size_t index, void* pData) { - ASSERT(index < pArray->size); - memcpy(TARRAY_GET_ELEM(pArray, index), pData, pArray->elemSize); + if (index < pArray->size) { + memcpy(TARRAY_GET_ELEM(pArray, index), pData, pArray->elemSize); + } } void taosArrayPopFrontBatch(SArray* pArray, size_t cnt) { @@ -291,7 +292,9 @@ void taosArrayPopTailBatch(SArray* pArray, size_t cnt) { } void taosArrayRemove(SArray* pArray, size_t index) { - ASSERT(index < pArray->size); + if (!(index < pArray->size)) { + return; + } if (index == pArray->size - 1) { (void)taosArrayPop(pArray); @@ -305,17 +308,17 @@ void taosArrayRemove(SArray* pArray, size_t index) { } void taosArrayRemoveBatch(SArray* pArray, size_t index, size_t num, FDelete fp) { - ASSERT(index + num <= pArray->size); - - if (fp) { - for (int32_t i = 0; i < num; i++) { - fp(taosArrayGet(pArray, index + i)); + if (index + num <= pArray->size) { + if (fp) { + for (int32_t i = 0; i < num; i++) { + fp(taosArrayGet(pArray, index + i)); + } } - } - memmove((char*)pArray->pData + index * pArray->elemSize, (char*)pArray->pData + (index + num) * pArray->elemSize, - (pArray->size - index - num) * pArray->elemSize); - pArray->size -= num; + memmove((char*)pArray->pData + index * pArray->elemSize, (char*)pArray->pData + (index + num) * pArray->elemSize, + (pArray->size - index - num) * pArray->elemSize); + pArray->size -= num; + } } SArray* taosArrayFromList(const void* src, size_t size, size_t elemSize) { @@ -349,8 +352,6 @@ SArray* taosArrayDup(const SArray* pSrc, __array_item_dup_fn_t fn) { if (fn == NULL) { memcpy(dst->pData, pSrc->pData, pSrc->elemSize * pSrc->size); } else { - ASSERT(pSrc->elemSize == sizeof(void*)); - for (int32_t i = 0; i < pSrc->size; ++i) { void* p = fn(taosArrayGetP(pSrc, i)); memcpy(((char*)dst->pData) + i * dst->elemSize, &p, dst->elemSize); diff --git a/source/util/src/tbloomfilter.c b/source/util/src/tbloomfilter.c index b20fb4bf39..c87c482167 100644 --- a/source/util/src/tbloomfilter.c +++ b/source/util/src/tbloomfilter.c @@ -78,7 +78,10 @@ _error: } int32_t tBloomFilterPutHash(SBloomFilter* pBF, uint64_t hash1, uint64_t hash2) { - ASSERT(!tBloomFilterIsFull(pBF)); + if (tBloomFilterIsFull(pBF)) { + uError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(TSDB_CODE_INVALID_PARA)); + return TSDB_CODE_FAILED; + } bool hasChange = false; const register uint64_t size = pBF->numBits; uint64_t cbHash = hash1; diff --git a/source/util/src/tcache.c b/source/util/src/tcache.c index c150ca3fbd..b7ca99d26a 100644 --- a/source/util/src/tcache.c +++ b/source/util/src/tcache.c @@ -262,7 +262,7 @@ static void pushfrontNodeInEntryList(SCacheEntry *pEntry, SCacheNode *pNode) { pNode->pNext = pEntry->next; pEntry->next = pNode; pEntry->num += 1; - ASSERT((pEntry->next && pEntry->num > 0) || (NULL == pEntry->next && pEntry->num == 0)); + //A S S E R T((pEntry->next && pEntry->num > 0) || (NULL == pEntry->next && pEntry->num == 0)); } static void removeNodeInEntryList(SCacheEntry *pe, SCacheNode *prev, SCacheNode *pNode) { @@ -274,7 +274,7 @@ static void removeNodeInEntryList(SCacheEntry *pe, SCacheNode *prev, SCacheNode pNode->pNext = NULL; pe->num -= 1; - ASSERT((pe->next && pe->num > 0) || (NULL == pe->next && pe->num == 0)); + //A S S E R T((pe->next && pe->num > 0) || (NULL == pe->next && pe->num == 0)); } static FORCE_INLINE SCacheEntry *doFindEntry(SCacheObj *pCacheObj, const void *key, size_t keyLen) { @@ -499,7 +499,7 @@ void *taosCacheAcquireByData(SCacheObj *pCacheObj, void *data) { uDebug("cache:%s, data: %p acquired by data in cache, refcnt:%d", pCacheObj->name, ptNode->data, ref); // the data if referenced by at least one object, so the reference count must be greater than the value of 2. - ASSERT(ref >= 2); + //A S S E R T(ref >= 2); return data; } @@ -574,19 +574,19 @@ void taosCacheRelease(SCacheObj *pCacheObj, void **data, bool _remove) { if (ref == 1) { // If it is the last ref, remove it from trashcan linked-list first, and then destroy it.Otherwise, it may be // destroyed by refresh worker if decrease ref count before removing it from linked-list. - ASSERT(pNode->pTNodeHeader->pData == pNode); + //A S S E R T(pNode->pTNodeHeader->pData == pNode); __trashcan_wr_lock(pCacheObj); (void)doRemoveElemInTrashcan(pCacheObj, pNode->pTNodeHeader); __trashcan_unlock(pCacheObj); ref = T_REF_DEC(pNode); - ASSERT(ref == 0); + //A S S E R T(ref == 0); doDestroyTrashcanElem(pCacheObj, pNode->pTNodeHeader); } else { ref = T_REF_DEC(pNode); - ASSERT(ref >= 0); + //A S S E R T(ref >= 0); } } else { // NOTE: remove it from hash in the first place, otherwise, the pNode may have been released by other thread @@ -608,7 +608,7 @@ void taosCacheRelease(SCacheObj *pCacheObj, void **data, bool _remove) { "others already, prev must in trashcan", pCacheObj->name, pNode->key, p->data, T_REF_VAL_GET(p), pNode->data, T_REF_VAL_GET(pNode)); - ASSERT(p->pTNodeHeader == NULL && pNode->pTNodeHeader != NULL); + //A S S E R T(p->pTNodeHeader == NULL && pNode->pTNodeHeader != NULL); } else { removeNodeInEntryList(pe, prev, p); uDebug("cache:%s, key:%p, %p successfully removed from hash table, refcnt:%d", pCacheObj->name, pNode->key, @@ -668,7 +668,7 @@ void doTraverseElems(SCacheObj *pCacheObj, bool (*fp)(void *param, SCacheNode *p } else { *pPre = next; pEntry->num -= 1; - ASSERT((pEntry->next && pEntry->num > 0) || (NULL == pEntry->next && pEntry->num == 0)); + //A S S E R T((pEntry->next && pEntry->num > 0) || (NULL == pEntry->next && pEntry->num == 0)); (void)atomic_sub_fetch_ptr(&pCacheObj->numOfElems, 1); pNode = next; @@ -734,7 +734,7 @@ SCacheNode *taosCreateCacheNode(const char *key, size_t keyLen, const char *pDat void taosAddToTrashcan(SCacheObj *pCacheObj, SCacheNode *pNode) { if (pNode->inTrashcan) { /* node is already in trash */ - ASSERT(pNode->pTNodeHeader != NULL && pNode->pTNodeHeader->pData == pNode); + //A S S E R T(pNode->pTNodeHeader != NULL && pNode->pTNodeHeader->pData == pNode); return; } @@ -780,7 +780,7 @@ void taosTrashcanEmpty(SCacheObj *pCacheObj, bool force) { STrashElem *pElem = pCacheObj->pTrash; while (pElem) { T_REF_VAL_CHECK(pElem->pData); - ASSERT(pElem->next != pElem && pElem->prev != pElem); + //A S S E R T(pElem->next != pElem && pElem->prev != pElem); if (force || (T_REF_VAL_GET(pElem->pData) == 0)) { uDebug("cache:%s, key:%p, %p removed from trashcan. numOfElem in trashcan:%d", pCacheObj->name, pElem->pData->key, diff --git a/source/util/src/tcompare.c b/source/util/src/tcompare.c index 670a70a309..0dee177faf 100644 --- a/source/util/src/tcompare.c +++ b/source/util/src/tcompare.c @@ -267,7 +267,7 @@ int32_t compareJsonVal(const void *pLeft, const void *pRight) { } else if (leftType == TSDB_DATA_TYPE_NULL) { return 0; } else { - ASSERTS(0, "data type unexpected"); + uError("data type unexpected leftType:%d rightType:%d", leftType, rightType); return 0; } } @@ -1497,7 +1497,9 @@ int32_t taosArrayCompareString(const void *a, const void *b) { int32_t comparestrPatternMatch(const void *pLeft, const void *pRight) { SPatternCompareInfo pInfo = PATTERN_COMPARE_INFO_INITIALIZER; - ASSERT(varDataTLen(pRight) <= TSDB_MAX_FIELD_LEN); + if (varDataTLen(pRight) > TSDB_MAX_FIELD_LEN) { + return 1; + } size_t pLen = varDataLen(pRight); size_t sz = varDataLen(pLeft); @@ -1546,7 +1548,9 @@ __compar_fn_t getComparFunc(int32_t type, int32_t optr) { case TSDB_DATA_TYPE_TIMESTAMP: return setChkInBytes8; default: - ASSERTS(0, "data type unexpected"); + uError("getComparFunc data type unexpected type:%d, optr:%d", type, optr); + terrno = TSDB_CODE_FUNC_FUNTION_PARA_TYPE; + return NULL; } } @@ -1570,7 +1574,9 @@ __compar_fn_t getComparFunc(int32_t type, int32_t optr) { case TSDB_DATA_TYPE_TIMESTAMP: return setChkNotInBytes8; default: - ASSERTS(0, "data type unexpected"); + uError("getComparFunc data type unexpected type:%d, optr:%d", type, optr); + terrno = TSDB_CODE_FUNC_FUNTION_PARA_TYPE; + return NULL; } } diff --git a/source/util/src/tcompression.c b/source/util/src/tcompression.c index 8402d2a658..afbd5304ce 100644 --- a/source/util/src/tcompression.c +++ b/source/util/src/tcompression.c @@ -718,7 +718,9 @@ int32_t tsCompressTimestampImp(const char *const input, const int32_t nelements, int32_t _pos = 1; int32_t longBytes = LONG_BYTES; - ASSERTS(nelements >= 0, "nelements is negative"); + if (nelements < 0) { + return -1; + } if (nelements == 0) return 0; @@ -815,7 +817,7 @@ _exit_over: int32_t tsDecompressTimestampImp(const char *const input, const int32_t nelements, char *const output) { int64_t longBytes = LONG_BYTES; - ASSERTS(nelements >= 0, "nelements is negative"); + if (nelements < 0) return -1; if (nelements == 0) return 0; if (input[0] == 0) { @@ -1278,1151 +1280,6 @@ int32_t tsDecompressDoubleLossyImp(const char *input, int32_t compressedSize, co return tdszDecompress(SZ_DOUBLE, input + 1, compressedSize - 1, nelements, output); } -#ifdef BUILD_NO_CALL -/************************************************************************* - * STREAM COMPRESSION - *************************************************************************/ -#define I64_SAFE_ADD(a, b) (((a) >= 0 && (b) <= INT64_MAX - (a)) || ((a) < 0 && (b) >= INT64_MIN - (a))) - -static int32_t tCompBoolStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg); -static int32_t tCompBool(SCompressor *pCmprsor, const void *pData, int32_t nData); -static int32_t tCompBoolEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData); - -static int32_t tCompIntStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg); -static int32_t tCompInt(SCompressor *pCmprsor, const void *pData, int32_t nData); -static int32_t tCompIntEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData); - -static int32_t tCompFloatStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg); -static int32_t tCompFloat(SCompressor *pCmprsor, const void *pData, int32_t nData); -static int32_t tCompFloatEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData); - -static int32_t tCompDoubleStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg); -static int32_t tCompDouble(SCompressor *pCmprsor, const void *pData, int32_t nData); -static int32_t tCompDoubleEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData); - -static int32_t tCompTimestampStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg); -static int32_t tCompTimestamp(SCompressor *pCmprsor, const void *pData, int32_t nData); -static int32_t tCompTimestampEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData); - -static int32_t tCompBinaryStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg); -static int32_t tCompBinary(SCompressor *pCmprsor, const void *pData, int32_t nData); -static int32_t tCompBinaryEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData); - -static FORCE_INLINE int64_t tGetI64OfI8(const void *pData) { return *(int8_t *)pData; } -static FORCE_INLINE int64_t tGetI64OfI16(const void *pData) { return *(int16_t *)pData; } -static FORCE_INLINE int64_t tGetI64OfI32(const void *pData) { return *(int32_t *)pData; } -static FORCE_INLINE int64_t tGetI64OfI64(const void *pData) { return *(int64_t *)pData; } - -static FORCE_INLINE void tPutI64OfI8(int64_t v, void *pData) { *(int8_t *)pData = v; } -static FORCE_INLINE void tPutI64OfI16(int64_t v, void *pData) { *(int16_t *)pData = v; } -static FORCE_INLINE void tPutI64OfI32(int64_t v, void *pData) { *(int32_t *)pData = v; } -static FORCE_INLINE void tPutI64OfI64(int64_t v, void *pData) { *(int64_t *)pData = v; } - -static struct { - int8_t type; - int32_t bytes; - int8_t isVarLen; - int32_t (*startFn)(SCompressor *, int8_t type, int8_t cmprAlg); - int32_t (*cmprFn)(SCompressor *, const void *, int32_t nData); - int32_t (*endFn)(SCompressor *, const uint8_t **, int32_t *); - int64_t (*getI64)(const void *pData); - void (*putI64)(int64_t v, void *pData); -} DATA_TYPE_INFO[] = { - {.type = TSDB_DATA_TYPE_NULL, - .bytes = 0, - .isVarLen = 0, - .startFn = NULL, - .cmprFn = NULL, - .endFn = NULL, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_BOOL, - .bytes = 1, - .isVarLen = 0, - .startFn = tCompBoolStart, - .cmprFn = tCompBool, - .endFn = tCompBoolEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_TINYINT, - .bytes = 1, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI8, - .putI64 = tPutI64OfI8}, - {.type = TSDB_DATA_TYPE_SMALLINT, - .bytes = 2, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI16, - .putI64 = tPutI64OfI16}, - {.type = TSDB_DATA_TYPE_INT, - .bytes = 4, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI32, - .putI64 = tPutI64OfI32}, - {.type = TSDB_DATA_TYPE_BIGINT, - .bytes = 8, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI64, - .putI64 = tPutI64OfI64}, - {.type = TSDB_DATA_TYPE_FLOAT, - .bytes = 4, - .isVarLen = 0, - .startFn = tCompFloatStart, - .cmprFn = tCompFloat, - .endFn = tCompFloatEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_DOUBLE, - .bytes = 8, - .isVarLen = 0, - .startFn = tCompDoubleStart, - .cmprFn = tCompDouble, - .endFn = tCompDoubleEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_VARCHAR, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_TIMESTAMP, - .bytes = 8, - .isVarLen = 0, - .startFn = tCompTimestampStart, - .cmprFn = tCompTimestamp, - .endFn = tCompTimestampEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_NCHAR, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_UTINYINT, - .bytes = 1, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI8, - .putI64 = tPutI64OfI8}, - {.type = TSDB_DATA_TYPE_USMALLINT, - .bytes = 2, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI16, - .putI64 = tPutI64OfI16}, - {.type = TSDB_DATA_TYPE_UINT, - .bytes = 4, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI32, - .putI64 = tPutI64OfI32}, - {.type = TSDB_DATA_TYPE_UBIGINT, - .bytes = 8, - .isVarLen = 0, - .startFn = tCompIntStart, - .cmprFn = tCompInt, - .endFn = tCompIntEnd, - .getI64 = tGetI64OfI64, - .putI64 = tPutI64OfI64}, - {.type = TSDB_DATA_TYPE_JSON, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_VARBINARY, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_DECIMAL, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_BLOB, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_MEDIUMBLOB, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, - {.type = TSDB_DATA_TYPE_GEOMETRY, - .bytes = 1, - .isVarLen = 1, - .startFn = tCompBinaryStart, - .cmprFn = tCompBinary, - .endFn = tCompBinaryEnd, - .getI64 = NULL, - .putI64 = NULL}, -}; - -struct SCompressor { - int8_t type; - int8_t cmprAlg; - int8_t autoAlloc; - int32_t nVal; - uint8_t *pBuf; - int32_t nBuf; - uint8_t *aBuf[1]; - union { - // Timestamp ---- - struct { - int64_t ts_prev_val; - int64_t ts_prev_delta; - uint8_t *ts_flag_p; - }; - // Integer ---- - struct { - int64_t i_prev; - int32_t i_selector; - int32_t i_start; - int32_t i_end; - int32_t i_nEle; - uint64_t i_aZigzag[241]; - int8_t i_aBitN[241]; - }; - // Float ---- - struct { - uint32_t f_prev; - uint8_t *f_flag_p; - }; - // Double ---- - struct { - uint64_t d_prev; - uint8_t *d_flag_p; - }; - }; -}; - -static int32_t tTwoStageComp(SCompressor *pCmprsor, int32_t *szComp) { - int32_t code = 0; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->aBuf[0], pCmprsor->nBuf + 1))) { - return code; - } - - *szComp = LZ4_compress_default(pCmprsor->pBuf, pCmprsor->aBuf[0] + 1, pCmprsor->nBuf, pCmprsor->nBuf); - if (*szComp && *szComp < pCmprsor->nBuf) { - pCmprsor->aBuf[0][0] = 1; - *szComp += 1; - } else { - pCmprsor->aBuf[0][0] = 0; - memcpy(pCmprsor->aBuf[0] + 1, pCmprsor->pBuf, pCmprsor->nBuf); - *szComp = pCmprsor->nBuf + 1; - } - - return code; -} - -// Timestamp ===================================================== -static int32_t tCompTimestampStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - int32_t code = 0; - - pCmprsor->nBuf = 1; - - code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf); - if (code) return code; - - pCmprsor->pBuf[0] = 1; - - return code; -} - -static int32_t tCompTSSwitchToCopy(SCompressor *pCmprsor) { - int32_t code = 0; - - if (pCmprsor->nVal == 0) goto _exit; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->aBuf[0], sizeof(int64_t) * pCmprsor->nVal + 1))) { - return code; - } - - int32_t n = 1; - int32_t nBuf = 1; - int64_t value; - int64_t delta; - for (int32_t iVal = 0; iVal < pCmprsor->nVal;) { - uint8_t aN[2] = {(pCmprsor->pBuf[n] & 0xf), (pCmprsor->pBuf[n] >> 4)}; - - n++; - - for (int32_t i = 0; i < 2; i++) { - uint64_t vZigzag = 0; - for (uint8_t j = 0; j < aN[i]; j++) { - vZigzag |= (((uint64_t)pCmprsor->pBuf[n]) << (8 * j)); - n++; - } - - int64_t delta_of_delta = ZIGZAG_DECODE(int64_t, vZigzag); - if (iVal) { - delta = delta_of_delta + delta; - value = delta + value; - } else { - delta = 0; - value = delta_of_delta; - } - - memcpy(pCmprsor->aBuf[0] + nBuf, &value, sizeof(value)); - nBuf += sizeof(int64_t); - - iVal++; - if (iVal >= pCmprsor->nVal) break; - } - } - - ASSERT(n == pCmprsor->nBuf && nBuf == sizeof(int64_t) * pCmprsor->nVal + 1); - - uint8_t *pBuf = pCmprsor->pBuf; - pCmprsor->pBuf = pCmprsor->aBuf[0]; - pCmprsor->aBuf[0] = pBuf; - pCmprsor->nBuf = nBuf; - -_exit: - pCmprsor->pBuf[0] = 0; - return code; -} -static int32_t tCompTimestamp(SCompressor *pCmprsor, const void *pData, int32_t nData) { - int32_t code = 0; - - int64_t ts = *(int64_t *)pData; - ASSERT(nData == 8); - - if (pCmprsor->pBuf[0] == 1) { - if (pCmprsor->nVal == 0) { - pCmprsor->ts_prev_val = ts; - pCmprsor->ts_prev_delta = -ts; - } - - if (!I64_SAFE_ADD(ts, -pCmprsor->ts_prev_val)) { - code = tCompTSSwitchToCopy(pCmprsor); - if (code) return code; - goto _copy_cmpr; - } - int64_t delta = ts - pCmprsor->ts_prev_val; - - if (!I64_SAFE_ADD(delta, -pCmprsor->ts_prev_delta)) { - code = tCompTSSwitchToCopy(pCmprsor); - if (code) return code; - goto _copy_cmpr; - } - int64_t delta_of_delta = delta - pCmprsor->ts_prev_delta; - uint64_t vZigzag = ZIGZAG_ENCODE(int64_t, delta_of_delta); - - pCmprsor->ts_prev_val = ts; - pCmprsor->ts_prev_delta = delta; - - if ((pCmprsor->nVal & 0x1) == 0) { - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + 17))) { - return code; - } - - pCmprsor->ts_flag_p = pCmprsor->pBuf + pCmprsor->nBuf; - pCmprsor->nBuf++; - pCmprsor->ts_flag_p[0] = 0; - while (vZigzag) { - pCmprsor->pBuf[pCmprsor->nBuf] = (vZigzag & 0xff); - pCmprsor->nBuf++; - pCmprsor->ts_flag_p[0]++; - vZigzag >>= 8; - } - } else { - while (vZigzag) { - pCmprsor->pBuf[pCmprsor->nBuf] = (vZigzag & 0xff); - pCmprsor->nBuf++; - pCmprsor->ts_flag_p[0] += 0x10; - vZigzag >>= 8; - } - } - } else { - _copy_cmpr: - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + sizeof(ts)))) { - return code; - } - - memcpy(pCmprsor->pBuf + pCmprsor->nBuf, &ts, sizeof(ts)); - pCmprsor->nBuf += sizeof(ts); - } - pCmprsor->nVal++; - - return code; -} - -static int32_t tCompTimestampEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData) { - int32_t code = 0; - - if (pCmprsor->nBuf >= sizeof(int64_t) * pCmprsor->nVal + 1 && pCmprsor->pBuf[0] == 1) { - code = tCompTSSwitchToCopy(pCmprsor); - if (code) return code; - } - - if (pCmprsor->cmprAlg == TWO_STAGE_COMP) { - code = tTwoStageComp(pCmprsor, nData); - if (code) return code; - *ppData = pCmprsor->aBuf[0]; - } else if (pCmprsor->cmprAlg == ONE_STAGE_COMP) { - *ppData = pCmprsor->pBuf; - *nData = pCmprsor->nBuf; - } else { - ASSERT(0); - } - - return code; -} - -// Integer ===================================================== -#define SIMPLE8B_MAX ((uint64_t)1152921504606846974LL) -static const uint8_t BIT_PER_INTEGER[] = {0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 10, 12, 15, 20, 30, 60}; -static const int32_t SELECTOR_TO_ELEMS[] = {240, 120, 60, 30, 20, 15, 12, 10, 8, 7, 6, 5, 4, 3, 2, 1}; -static const uint8_t BIT_TO_SELECTOR[] = {0, 2, 3, 4, 5, 6, 7, 8, 9, 10, 10, 11, 11, 12, 12, 12, - 13, 13, 13, 13, 13, 14, 14, 14, 14, 14, 14, 14, 14, 14, 14, 15, - 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, - 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15, 15}; -static const int32_t NEXT_IDX[] = { - 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, - 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, - 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, - 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, - 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, - 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, - 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, - 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, - 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, - 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, - 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 0}; - -static int32_t tCompIntStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - int32_t code = 0; - - pCmprsor->i_prev = 0; - pCmprsor->i_selector = 0; - pCmprsor->i_start = 0; - pCmprsor->i_end = 0; - pCmprsor->i_nEle = 0; - pCmprsor->nBuf = 1; - - code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf); - if (code) return code; - - pCmprsor->pBuf[0] = 0; - - return code; -} - -static int32_t tCompIntSwitchToCopy(SCompressor *pCmprsor) { - int32_t code = 0; - - if (pCmprsor->nVal == 0) goto _exit; - - int32_t size = DATA_TYPE_INFO[pCmprsor->type].bytes * pCmprsor->nVal + 1; - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->aBuf[0], size))) { - return code; - } - - int32_t n = 1; - int32_t nBuf = 1; - int64_t vPrev = 0; - while (n < pCmprsor->nBuf) { - uint64_t b; - memcpy(&b, pCmprsor->pBuf + n, sizeof(b)); - n += sizeof(b); - - int32_t i_selector = (b & 0xf); - int32_t nEle = SELECTOR_TO_ELEMS[i_selector]; - uint8_t bits = BIT_PER_INTEGER[i_selector]; - uint64_t mask = (((uint64_t)1) << bits) - 1; - for (int32_t iEle = 0; iEle < nEle; iEle++) { - uint64_t vZigzag = (b >> (bits * iEle + 4)) & mask; - vPrev = ZIGZAG_DECODE(int64_t, vZigzag) + vPrev; - - DATA_TYPE_INFO[pCmprsor->type].putI64(vPrev, pCmprsor->aBuf[0] + nBuf); - nBuf += DATA_TYPE_INFO[pCmprsor->type].bytes; - } - } - - while (pCmprsor->i_nEle) { - vPrev = ZIGZAG_DECODE(int64_t, pCmprsor->i_aZigzag[pCmprsor->i_start]) + vPrev; - - memcpy(pCmprsor->aBuf[0] + nBuf, &vPrev, DATA_TYPE_INFO[pCmprsor->type].bytes); - nBuf += DATA_TYPE_INFO[pCmprsor->type].bytes; - - pCmprsor->i_start = NEXT_IDX[pCmprsor->i_start]; - pCmprsor->i_nEle--; - } - - ASSERT(n == pCmprsor->nBuf && nBuf == size); - - uint8_t *pBuf = pCmprsor->pBuf; - pCmprsor->pBuf = pCmprsor->aBuf[0]; - pCmprsor->aBuf[0] = pBuf; - pCmprsor->nBuf = size; - -_exit: - pCmprsor->pBuf[0] = 1; - return code; -} - -static int32_t tCompInt(SCompressor *pCmprsor, const void *pData, int32_t nData) { - int32_t code = 0; - - ASSERT(nData == DATA_TYPE_INFO[pCmprsor->type].bytes); - - if (pCmprsor->pBuf[0] == 0) { - int64_t val = DATA_TYPE_INFO[pCmprsor->type].getI64(pData); - - if (!I64_SAFE_ADD(val, -pCmprsor->i_prev)) { - code = tCompIntSwitchToCopy(pCmprsor); - if (code) return code; - goto _copy_cmpr; - } - - int64_t diff = val - pCmprsor->i_prev; - uint64_t vZigzag = ZIGZAG_ENCODE(int64_t, diff); - if (vZigzag >= SIMPLE8B_MAX) { - code = tCompIntSwitchToCopy(pCmprsor); - if (code) return code; - goto _copy_cmpr; - } - - int8_t nBit = (vZigzag) ? (64 - BUILDIN_CLZL(vZigzag)) : 0; - pCmprsor->i_prev = val; - - for (;;) { - if (pCmprsor->i_nEle + 1 <= SELECTOR_TO_ELEMS[pCmprsor->i_selector] && - pCmprsor->i_nEle + 1 <= SELECTOR_TO_ELEMS[BIT_TO_SELECTOR[nBit]]) { - if (pCmprsor->i_selector < BIT_TO_SELECTOR[nBit]) { - pCmprsor->i_selector = BIT_TO_SELECTOR[nBit]; - } - pCmprsor->i_aZigzag[pCmprsor->i_end] = vZigzag; - pCmprsor->i_aBitN[pCmprsor->i_end] = nBit; - pCmprsor->i_end = NEXT_IDX[pCmprsor->i_end]; - pCmprsor->i_nEle++; - break; - } else { - if (pCmprsor->i_nEle < SELECTOR_TO_ELEMS[pCmprsor->i_selector]) { - int32_t lidx = pCmprsor->i_selector + 1; - int32_t ridx = 15; - while (lidx <= ridx) { - pCmprsor->i_selector = (lidx + ridx) >> 1; - - if (pCmprsor->i_nEle < SELECTOR_TO_ELEMS[pCmprsor->i_selector]) { - lidx = pCmprsor->i_selector + 1; - } else if (pCmprsor->i_nEle > SELECTOR_TO_ELEMS[pCmprsor->i_selector]) { - ridx = pCmprsor->i_selector - 1; - } else { - break; - } - } - - if (pCmprsor->i_nEle < SELECTOR_TO_ELEMS[pCmprsor->i_selector]) pCmprsor->i_selector++; - } - int32_t nEle = SELECTOR_TO_ELEMS[pCmprsor->i_selector]; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + sizeof(uint64_t)))) { - return code; - } - - uint64_t *bp = (uint64_t *)(pCmprsor->pBuf + pCmprsor->nBuf); - pCmprsor->nBuf += sizeof(uint64_t); - bp[0] = pCmprsor->i_selector; - uint8_t bits = BIT_PER_INTEGER[pCmprsor->i_selector]; - for (int32_t iVal = 0; iVal < nEle; iVal++) { - bp[0] |= (pCmprsor->i_aZigzag[pCmprsor->i_start] << (bits * iVal + 4)); - pCmprsor->i_start = NEXT_IDX[pCmprsor->i_start]; - pCmprsor->i_nEle--; - } - - // reset and continue - pCmprsor->i_selector = 0; - for (int32_t iVal = pCmprsor->i_start; iVal < pCmprsor->i_end; iVal = NEXT_IDX[iVal]) { - if (pCmprsor->i_selector < BIT_TO_SELECTOR[pCmprsor->i_aBitN[iVal]]) { - pCmprsor->i_selector = BIT_TO_SELECTOR[pCmprsor->i_aBitN[iVal]]; - } - } - } - } - } else { - _copy_cmpr: - code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + nData); - if (code) return code; - - memcpy(pCmprsor->pBuf + pCmprsor->nBuf, pData, nData); - pCmprsor->nBuf += nData; - } - pCmprsor->nVal++; - - return code; -} - -static int32_t tCompIntEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData) { - int32_t code = 0; - - for (; pCmprsor->i_nEle;) { - if (pCmprsor->i_nEle < SELECTOR_TO_ELEMS[pCmprsor->i_selector]) { - int32_t lidx = pCmprsor->i_selector + 1; - int32_t ridx = 15; - while (lidx <= ridx) { - pCmprsor->i_selector = (lidx + ridx) >> 1; - - if (pCmprsor->i_nEle < SELECTOR_TO_ELEMS[pCmprsor->i_selector]) { - lidx = pCmprsor->i_selector + 1; - } else if (pCmprsor->i_nEle > SELECTOR_TO_ELEMS[pCmprsor->i_selector]) { - ridx = pCmprsor->i_selector - 1; - } else { - break; - } - } - - if (pCmprsor->i_nEle < SELECTOR_TO_ELEMS[pCmprsor->i_selector]) pCmprsor->i_selector++; - } - int32_t nEle = SELECTOR_TO_ELEMS[pCmprsor->i_selector]; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + sizeof(uint64_t)))) { - return code; - } - - uint64_t *bp = (uint64_t *)(pCmprsor->pBuf + pCmprsor->nBuf); - pCmprsor->nBuf += sizeof(uint64_t); - bp[0] = pCmprsor->i_selector; - uint8_t bits = BIT_PER_INTEGER[pCmprsor->i_selector]; - for (int32_t iVal = 0; iVal < nEle; iVal++) { - bp[0] |= (pCmprsor->i_aZigzag[pCmprsor->i_start] << (bits * iVal + 4)); - pCmprsor->i_start = NEXT_IDX[pCmprsor->i_start]; - pCmprsor->i_nEle--; - } - - pCmprsor->i_selector = 0; - } - - if (pCmprsor->nBuf >= DATA_TYPE_INFO[pCmprsor->type].bytes * pCmprsor->nVal + 1 && pCmprsor->pBuf[0] == 0) { - code = tCompIntSwitchToCopy(pCmprsor); - if (code) return code; - } - - if (pCmprsor->cmprAlg == TWO_STAGE_COMP) { - code = tTwoStageComp(pCmprsor, nData); - if (code) return code; - *ppData = pCmprsor->aBuf[0]; - } else if (pCmprsor->cmprAlg == ONE_STAGE_COMP) { - *ppData = pCmprsor->pBuf; - *nData = pCmprsor->nBuf; - } else { - ASSERT(0); - } - - return code; -} - -// Float ===================================================== -static int32_t tCompFloatStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - int32_t code = 0; - - pCmprsor->f_prev = 0; - pCmprsor->f_flag_p = NULL; - - pCmprsor->nBuf = 1; - - code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf); - if (code) return code; - - pCmprsor->pBuf[0] = 0; - - return code; -} - -static int32_t tCompFloatSwitchToCopy(SCompressor *pCmprsor) { - int32_t code = 0; - - if (pCmprsor->nVal == 0) goto _exit; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->aBuf[0], sizeof(float) * pCmprsor->nVal + 1))) { - return code; - } - - int32_t n = 1; - int32_t nBuf = 1; - union { - float f; - uint32_t u; - } val = {.u = 0}; - - for (int32_t iVal = 0; iVal < pCmprsor->nVal;) { - uint8_t flags[2] = {(pCmprsor->pBuf[n] & 0xf), (pCmprsor->pBuf[n] >> 4)}; - - n++; - - for (int8_t i = 0; i < 2; i++) { - uint8_t flag = flags[i]; - - uint32_t diff = 0; - int8_t nBytes = (flag & 0x7) + 1; - for (int j = 0; j < nBytes; j++) { - diff |= (((uint32_t)pCmprsor->pBuf[n]) << (8 * j)); - n++; - } - - if (flag & 0x8) { - diff <<= (32 - nBytes * 8); - } - - val.u ^= diff; - - memcpy(pCmprsor->aBuf[0] + nBuf, &val.f, sizeof(val)); - nBuf += sizeof(val); - - iVal++; - if (iVal >= pCmprsor->nVal) break; - } - } - uint8_t *pBuf = pCmprsor->pBuf; - pCmprsor->pBuf = pCmprsor->aBuf[0]; - pCmprsor->aBuf[0] = pBuf; - pCmprsor->nBuf = nBuf; - -_exit: - pCmprsor->pBuf[0] = 1; - return code; -} -static int32_t tCompFloat(SCompressor *pCmprsor, const void *pData, int32_t nData) { - int32_t code = 0; - - ASSERT(nData == sizeof(float)); - - union { - float f; - uint32_t u; - } val = {.f = *(float *)pData}; - - uint32_t diff = val.u ^ pCmprsor->f_prev; - pCmprsor->f_prev = val.u; - - int32_t clz, ctz; - if (diff) { - clz = BUILDIN_CLZ(diff); - ctz = BUILDIN_CTZ(diff); - } else { - clz = 32; - ctz = 32; - } - - uint8_t nBytes; - if (clz < ctz) { - nBytes = sizeof(uint32_t) - ctz / BITS_PER_BYTE; - if (nBytes) diff >>= (32 - nBytes * BITS_PER_BYTE); - } else { - nBytes = sizeof(uint32_t) - clz / BITS_PER_BYTE; - } - if (nBytes == 0) nBytes++; - - if ((pCmprsor->nVal & 0x1) == 0) { - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + 9))) { - return code; - } - - pCmprsor->f_flag_p = &pCmprsor->pBuf[pCmprsor->nBuf]; - pCmprsor->nBuf++; - - if (clz < ctz) { - pCmprsor->f_flag_p[0] = (0x08 | (nBytes - 1)); - } else { - pCmprsor->f_flag_p[0] = nBytes - 1; - } - } else { - if (clz < ctz) { - pCmprsor->f_flag_p[0] |= ((0x08 | (nBytes - 1)) << 4); - } else { - pCmprsor->f_flag_p[0] |= ((nBytes - 1) << 4); - } - } - for (; nBytes; nBytes--) { - pCmprsor->pBuf[pCmprsor->nBuf] = (diff & 0xff); - pCmprsor->nBuf++; - diff >>= BITS_PER_BYTE; - } - pCmprsor->nVal++; - - return code; -} - -static int32_t tCompFloatEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData) { - int32_t code = 0; - - if (pCmprsor->nBuf >= sizeof(float) * pCmprsor->nVal + 1) { - code = tCompFloatSwitchToCopy(pCmprsor); - if (code) return code; - } - - if (pCmprsor->cmprAlg == TWO_STAGE_COMP) { - code = tTwoStageComp(pCmprsor, nData); - if (code) return code; - *ppData = pCmprsor->aBuf[0]; - } else if (pCmprsor->cmprAlg == ONE_STAGE_COMP) { - *ppData = pCmprsor->pBuf; - *nData = pCmprsor->nBuf; - } else { - ASSERT(0); - } - - return code; -} - -// Double ===================================================== -static int32_t tCompDoubleStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - int32_t code = 0; - - pCmprsor->d_prev = 0; - pCmprsor->d_flag_p = NULL; - - pCmprsor->nBuf = 1; - - code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf); - if (code) return code; - - pCmprsor->pBuf[0] = 0; - - return code; -} - -static int32_t tCompDoubleSwitchToCopy(SCompressor *pCmprsor) { - int32_t code = 0; - - if (pCmprsor->nVal == 0) goto _exit; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->aBuf[0], sizeof(double) * pCmprsor->nVal + 1))) { - return code; - } - - int32_t n = 1; - int32_t nBuf = 1; - union { - double f; - uint64_t u; - } val = {.u = 0}; - - for (int32_t iVal = 0; iVal < pCmprsor->nVal;) { - uint8_t flags[2] = {(pCmprsor->pBuf[n] & 0xf), (pCmprsor->pBuf[n] >> 4)}; - - n++; - - for (int8_t i = 0; i < 2; i++) { - uint8_t flag = flags[i]; - - uint64_t diff = 0; - int8_t nBytes = (flag & 0x7) + 1; - for (int j = 0; j < nBytes; j++) { - diff |= (((uint64_t)pCmprsor->pBuf[n]) << (8 * j)); - n++; - } - - if (flag & 0x8) { - diff <<= (64 - nBytes * 8); - } - - val.u ^= diff; - - memcpy(pCmprsor->aBuf[0] + nBuf, &val.f, sizeof(val)); - nBuf += sizeof(val); - - iVal++; - if (iVal >= pCmprsor->nVal) break; - } - } - uint8_t *pBuf = pCmprsor->pBuf; - pCmprsor->pBuf = pCmprsor->aBuf[0]; - pCmprsor->aBuf[0] = pBuf; - pCmprsor->nBuf = nBuf; - -_exit: - pCmprsor->pBuf[0] = 1; - return code; -} -static int32_t tCompDouble(SCompressor *pCmprsor, const void *pData, int32_t nData) { - int32_t code = 0; - - ASSERT(nData == sizeof(double)); - - union { - double d; - uint64_t u; - } val = {.d = *(double *)pData}; - - uint64_t diff = val.u ^ pCmprsor->d_prev; - pCmprsor->d_prev = val.u; - - int32_t clz, ctz; - if (diff) { - clz = BUILDIN_CLZL(diff); - ctz = BUILDIN_CTZL(diff); - } else { - clz = 64; - ctz = 64; - } - - uint8_t nBytes; - if (clz < ctz) { - nBytes = sizeof(uint64_t) - ctz / BITS_PER_BYTE; - if (nBytes) diff >>= (64 - nBytes * BITS_PER_BYTE); - } else { - nBytes = sizeof(uint64_t) - clz / BITS_PER_BYTE; - } - if (nBytes == 0) nBytes++; - - if ((pCmprsor->nVal & 0x1) == 0) { - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + 17))) { - return code; - } - - pCmprsor->d_flag_p = &pCmprsor->pBuf[pCmprsor->nBuf]; - pCmprsor->nBuf++; - - if (clz < ctz) { - pCmprsor->d_flag_p[0] = (0x08 | (nBytes - 1)); - } else { - pCmprsor->d_flag_p[0] = nBytes - 1; - } - } else { - if (clz < ctz) { - pCmprsor->d_flag_p[0] |= ((0x08 | (nBytes - 1)) << 4); - } else { - pCmprsor->d_flag_p[0] |= ((nBytes - 1) << 4); - } - } - for (; nBytes; nBytes--) { - pCmprsor->pBuf[pCmprsor->nBuf] = (diff & 0xff); - pCmprsor->nBuf++; - diff >>= BITS_PER_BYTE; - } - pCmprsor->nVal++; - - return code; -} - -static int32_t tCompDoubleEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData) { - int32_t code = 0; - - if (pCmprsor->nBuf >= sizeof(double) * pCmprsor->nVal + 1) { - code = tCompDoubleSwitchToCopy(pCmprsor); - if (code) return code; - } - - if (pCmprsor->cmprAlg == TWO_STAGE_COMP) { - code = tTwoStageComp(pCmprsor, nData); - if (code) return code; - *ppData = pCmprsor->aBuf[0]; - } else if (pCmprsor->cmprAlg == ONE_STAGE_COMP) { - *ppData = pCmprsor->pBuf; - *nData = pCmprsor->nBuf; - } else { - ASSERT(0); - } - - return code; -} - -// Binary ===================================================== -static int32_t tCompBinaryStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - pCmprsor->nBuf = 1; - return 0; -} - -static int32_t tCompBinary(SCompressor *pCmprsor, const void *pData, int32_t nData) { - int32_t code = 0; - - if (nData) { - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf + nData))) { - return code; - } - - memcpy(pCmprsor->pBuf + pCmprsor->nBuf, pData, nData); - pCmprsor->nBuf += nData; - } - pCmprsor->nVal++; - - return code; -} - -static int32_t tCompBinaryEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData) { - int32_t code = 0; - - if (pCmprsor->nBuf == 1) return code; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->aBuf[0], pCmprsor->nBuf))) { - return code; - } - - int32_t szComp = - LZ4_compress_default(pCmprsor->pBuf + 1, pCmprsor->aBuf[0] + 1, pCmprsor->nBuf - 1, pCmprsor->nBuf - 1); - if (szComp && szComp < pCmprsor->nBuf - 1) { - pCmprsor->aBuf[0][0] = 1; - *ppData = pCmprsor->aBuf[0]; - *nData = szComp + 1; - } else { - pCmprsor->pBuf[0] = 0; - *ppData = pCmprsor->pBuf; - *nData = pCmprsor->nBuf; - } - - return code; -} - -// Bool ===================================================== -static const uint8_t BOOL_CMPR_TABLE[] = {0b01, 0b0100, 0b010000, 0b01000000}; - -static int32_t tCompBoolStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - pCmprsor->nBuf = 0; - return 0; -} - -static int32_t tCompBool(SCompressor *pCmprsor, const void *pData, int32_t nData) { - int32_t code = 0; - - bool vBool = *(int8_t *)pData; - - int32_t mod4 = (pCmprsor->nVal & 3); - if (mod4 == 0) { - pCmprsor->nBuf++; - - if (pCmprsor->autoAlloc && (code = tRealloc(&pCmprsor->pBuf, pCmprsor->nBuf))) { - return code; - } - - pCmprsor->pBuf[pCmprsor->nBuf - 1] = 0; - } - if (vBool) { - pCmprsor->pBuf[pCmprsor->nBuf - 1] |= BOOL_CMPR_TABLE[mod4]; - } - pCmprsor->nVal++; - - return code; -} - -static int32_t tCompBoolEnd(SCompressor *pCmprsor, const uint8_t **ppData, int32_t *nData) { - int32_t code = 0; - - if (pCmprsor->cmprAlg == TWO_STAGE_COMP) { - code = tTwoStageComp(pCmprsor, nData); - if (code) return code; - *ppData = pCmprsor->aBuf[0]; - } else if (pCmprsor->cmprAlg == ONE_STAGE_COMP) { - *ppData = pCmprsor->pBuf; - *nData = pCmprsor->nBuf; - } else { - ASSERT(0); - } - - return code; -} - -// SCompressor ===================================================== -int32_t tCompressorCreate(SCompressor **ppCmprsor) { - int32_t code = 0; - - *ppCmprsor = (SCompressor *)taosMemoryCalloc(1, sizeof(SCompressor)); - if ((*ppCmprsor) == NULL) { - code = TSDB_CODE_OUT_OF_MEMORY; - return code; - } - - return code; -} - -int32_t tCompressorDestroy(SCompressor *pCmprsor) { - int32_t code = 0; - - tFree(pCmprsor->pBuf); - - int32_t nBuf = sizeof(pCmprsor->aBuf) / sizeof(pCmprsor->aBuf[0]); - for (int32_t iBuf = 0; iBuf < nBuf; iBuf++) { - tFree(pCmprsor->aBuf[iBuf]); - } - - taosMemoryFree(pCmprsor); - - return code; -} - -int32_t tCompressStart(SCompressor *pCmprsor, int8_t type, int8_t cmprAlg) { - int32_t code = 0; - - pCmprsor->type = type; - pCmprsor->cmprAlg = cmprAlg; - pCmprsor->autoAlloc = 1; - pCmprsor->nVal = 0; - - if (DATA_TYPE_INFO[type].startFn) { - DATA_TYPE_INFO[type].startFn(pCmprsor, type, cmprAlg); - } - - return code; -} - -int32_t tCompressEnd(SCompressor *pCmprsor, const uint8_t **ppOut, int32_t *nOut, int32_t *nOrigin) { - int32_t code = 0; - - *ppOut = NULL; - *nOut = 0; - if (nOrigin) { - if (DATA_TYPE_INFO[pCmprsor->type].isVarLen) { - *nOrigin = pCmprsor->nBuf - 1; - } else { - *nOrigin = pCmprsor->nVal * DATA_TYPE_INFO[pCmprsor->type].bytes; - } - } - - if (pCmprsor->nVal == 0) return code; - - if (DATA_TYPE_INFO[pCmprsor->type].endFn) { - return DATA_TYPE_INFO[pCmprsor->type].endFn(pCmprsor, ppOut, nOut); - } - - return code; -} - -int32_t tCompress(SCompressor *pCmprsor, const void *pData, int64_t nData) { - return DATA_TYPE_INFO[pCmprsor->type].cmprFn(pCmprsor, pData, nData); -} -#endif /************************************************************************* * REGULAR COMPRESSION *************************************************************************/ @@ -2448,7 +1305,6 @@ int32_t tsDecompressTimestamp(void *pIn, int32_t nIn, int32_t nEle, void *pOut, if (tsDecompressStringImp(pIn, nIn, pBuf, nBuf) < 0) return -1; return tsDecompressTimestampImp(pBuf, nEle, pOut); } else { - ASSERTS(0, "compress algo invalid"); return -1; } } @@ -2467,7 +1323,6 @@ int32_t tsCompressFloat(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32_ int32_t len = tsCompressFloatImp(pIn, nEle, pBuf); return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2486,7 +1341,6 @@ int32_t tsDecompressFloat(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int3 if (tsDecompressStringImp(pIn, nIn, pBuf, nBuf) < 0) return -1; return tsDecompressFloatImp(pBuf, nEle, pOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2506,7 +1360,6 @@ int32_t tsCompressDouble(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32 int32_t len = tsCompressDoubleImp(pIn, nEle, pBuf); return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2525,7 +1378,6 @@ int32_t tsDecompressDouble(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int if (tsDecompressStringImp(pIn, nIn, pBuf, nBuf) < 0) return -1; return tsDecompressDoubleImp(pBuf, nEle, pOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2554,7 +1406,6 @@ int32_t tsCompressBool(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32_t } return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_THIRDPARTY_ERROR; } } @@ -2568,7 +1419,6 @@ int32_t tsDecompressBool(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32 if ((code = tsDecompressStringImp(pIn, nIn, pBuf, nBuf)) < 0) return code; return tsDecompressBoolImp(pBuf, nEle, pOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2585,7 +1435,6 @@ int32_t tsCompressTinyint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int3 } return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2599,7 +1448,6 @@ int32_t tsDecompressTinyint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, in if ((code = tsDecompressStringImp(pIn, nIn, pBuf, nBuf)) < 0) return code; return tsDecompressINTImp(pBuf, nEle, pOut, TSDB_DATA_TYPE_TINYINT); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2616,7 +1464,6 @@ int32_t tsCompressSmallint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int } return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2630,7 +1477,6 @@ int32_t tsDecompressSmallint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, i if ((code = tsDecompressStringImp(pIn, nIn, pBuf, nBuf)) < 0) return code; return tsDecompressINTImp(pBuf, nEle, pOut, TSDB_DATA_TYPE_SMALLINT); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2647,7 +1493,6 @@ int32_t tsCompressInt(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32_t } return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2661,7 +1506,6 @@ int32_t tsDecompressInt(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32_ if ((code = tsDecompressStringImp(pIn, nIn, pBuf, nBuf)) < 0) return code; return tsDecompressINTImp(pBuf, nEle, pOut, TSDB_DATA_TYPE_INT); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2678,7 +1522,6 @@ int32_t tsCompressBigint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int32 } return tsCompressStringImp(pBuf, len, pOut, nOut); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2692,7 +1535,6 @@ int32_t tsDecompressBigint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int if ((code = tsDecompressStringImp(pIn, nIn, pBuf, nBuf)) < 0) return code; return tsDecompressINTImp(pBuf, nEle, pOut, TSDB_DATA_TYPE_BIGINT); } else { - ASSERTS(0, "compress algo invalid"); return TSDB_CODE_INVALID_PARA; } } @@ -2734,8 +1576,6 @@ int32_t tsDecompressBigint(void *pIn, int32_t nIn, int32_t nEle, void *pOut, int tDataTypes[type].name); \ return compressL2Dict[l2].decomprFn(pIn, nIn, pOut, nOut, type); \ } \ - } else { \ - ASSERT(0); \ } \ return TSDB_CODE_INVALID_PARA; \ } while (1) diff --git a/source/util/src/tconfig.c b/source/util/src/tconfig.c index b14b0823a3..29d8b38146 100644 --- a/source/util/src/tconfig.c +++ b/source/util/src/tconfig.c @@ -121,7 +121,9 @@ int32_t cfgGetSize(SConfig *pCfg) { return taosArrayGetSize(pCfg->array); } static int32_t cfgCheckAndSetConf(SConfigItem *pItem, const char *conf) { cfgItemFreeVal(pItem); - ASSERT(pItem->str == NULL); + if (!(pItem->str == NULL)) { + return TSDB_CODE_INVALID_PARA; + } pItem->str = taosStrdup(conf); if (pItem->str == NULL) { diff --git a/source/util/src/tdigest.c b/source/util/src/tdigest.c index 51b68f8f6b..636c97dbaa 100644 --- a/source/util/src/tdigest.c +++ b/source/util/src/tdigest.c @@ -143,24 +143,20 @@ int32_t tdigestCompress(TDigest *t) { if (a->mean <= b->mean) { mergeCentroid(&args, a); - ASSERTS(args.idx < t->size, "idx over size"); i++; } else { mergeCentroid(&args, b); - ASSERTS(args.idx < t->size, "idx over size"); j++; } } while (i < num_unmerged) { mergeCentroid(&args, &unmerged_centroids[i++]); - ASSERTS(args.idx < t->size, "idx over size"); } taosMemoryFree((void *)unmerged_centroids); while (j < t->num_centroids) { mergeCentroid(&args, &t->centroids[j++]); - ASSERTS(args.idx < t->size, "idx over size"); } if (t->total_weight > 0) { diff --git a/source/util/src/tencode.c b/source/util/src/tencode.c index aa9f157f3a..99b0b2bded 100644 --- a/source/util/src/tencode.c +++ b/source/util/src/tencode.c @@ -104,7 +104,6 @@ void tEndEncode(SEncoder* pCoder) { if (pCoder->data) { pNode = pCoder->eStack; - ASSERT(pNode); pCoder->eStack = pNode->pNext; len = pCoder->pos; @@ -148,7 +147,6 @@ void tEndDecode(SDecoder* pCoder) { SDecoderNode* pNode; pNode = pCoder->dStack; - ASSERT(pNode); pCoder->dStack = pNode->pNext; pCoder->data = pNode->data; diff --git a/source/util/src/terror.c b/source/util/src/terror.c index e26538b521..6701842ec9 100644 --- a/source/util/src/terror.c +++ b/source/util/src/terror.c @@ -57,8 +57,8 @@ TAOS_DEFINE_ERROR(TSDB_CODE_RPC_MAX_SESSIONS, "rpc open too many ses TAOS_DEFINE_ERROR(TSDB_CODE_RPC_NETWORK_ERROR, "rpc network error") TAOS_DEFINE_ERROR(TSDB_CODE_RPC_NETWORK_BUSY, "rpc network busy") TAOS_DEFINE_ERROR(TSDB_CODE_HTTP_MODULE_QUIT, "http-report already quit") -TAOS_DEFINE_ERROR(TSDB_CODE_RPC_MODULE_QUIT, "rpc module already quit") -TAOS_DEFINE_ERROR(TSDB_CODE_RPC_ASYNC_MODULE_QUIT, "rpc async module already quit") +TAOS_DEFINE_ERROR(TSDB_CODE_RPC_MODULE_QUIT, "rpc module already quit") +TAOS_DEFINE_ERROR(TSDB_CODE_RPC_ASYNC_MODULE_QUIT, "rpc async module already quit") //common & util TAOS_DEFINE_ERROR(TSDB_CODE_TIME_UNSYNCED, "Client and server's time is not synchronized") @@ -111,6 +111,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_IP_NOT_IN_WHITE_LIST, "Not allowed to connec TAOS_DEFINE_ERROR(TSDB_CODE_FAILED_TO_CONNECT_S3, "Failed to connect to s3 server") TAOS_DEFINE_ERROR(TSDB_CODE_MSG_PREPROCESSED, "Message has been processed in preprocess") TAOS_DEFINE_ERROR(TSDB_CODE_OUT_OF_BUFFER, "Out of buffer") +TAOS_DEFINE_ERROR(TSDB_CODE_INTERNAL_ERROR, "Internal error") //client TAOS_DEFINE_ERROR(TSDB_CODE_TSC_INVALID_OPERATION, "Invalid operation") @@ -388,6 +389,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_DNODE_INVALID_CHARSET, "charset not match") TAOS_DEFINE_ERROR(TSDB_CODE_DNODE_INVALID_LOCALE, "locale not match") TAOS_DEFINE_ERROR(TSDB_CODE_DNODE_INVALID_TTL_CHG_ON_WR, "ttlChangeOnWrite not match") TAOS_DEFINE_ERROR(TSDB_CODE_DNODE_INVALID_EN_WHITELIST, "enableWhiteList not match") +TAOS_DEFINE_ERROR(TSDB_CODE_MNODE_STOPPED, "Mnode stopped") // vnode TAOS_DEFINE_ERROR(TSDB_CODE_VND_INVALID_VGROUP_ID, "Vnode is closed or removed") @@ -411,6 +413,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_VND_META_DATA_UNSAFE_DELETE, "Single replica vnode TAOS_DEFINE_ERROR(TSDB_CODE_VND_ARB_NOT_SYNCED, "Vgroup peer is not synced") TAOS_DEFINE_ERROR(TSDB_CODE_VND_WRITE_DISABLED, "Vnode write is disabled for snapshot") TAOS_DEFINE_ERROR(TSDB_CODE_VND_COLUMN_COMPRESS_ALREADY_EXIST,"Same with old param") +TAOS_DEFINE_ERROR(TSDB_CODE_VND_TTL_FLUSH_INCOMPLETION, "Failed to flush all ttl modification to tdb") // tsdb @@ -556,6 +559,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TQ_META_KEY_DUP_IN_TXN, "TQ met key dup in txn TAOS_DEFINE_ERROR(TSDB_CODE_TQ_GROUP_NOT_SET, "TQ group not exist") TAOS_DEFINE_ERROR(TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND, "TQ table schema not found") TAOS_DEFINE_ERROR(TSDB_CODE_TQ_NO_COMMITTED_OFFSET, "TQ no committed offset") +TAOS_DEFINE_ERROR(TSDB_CODE_TQ_INTERNAL_ERROR, "TQ internal error") // wal TAOS_DEFINE_ERROR(TSDB_CODE_WAL_FILE_CORRUPTED, "WAL file is corrupted") diff --git a/source/util/src/texception.c b/source/util/src/texception.c deleted file mode 100644 index 2c8ddf6a5f..0000000000 --- a/source/util/src/texception.c +++ /dev/null @@ -1,150 +0,0 @@ -/* - * Copyright (c) 2019 TAOS Data, Inc. - * - * This program is free software: you can use, redistribute, and/or modify - * it under the terms of the GNU Affero General Public License, version 3 - * or later ("AGPL"), as published by the Free Software Foundation. - * - * This program is distributed in the hope that it will be useful, but WITHOUT - * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or - * FITNESS FOR A PARTICULAR PURPOSE. - * - * You should have received a copy of the GNU Affero General Public License - * along with this program. If not, see . - */ - -#define _DEFAULT_SOURCE -#include "texception.h" -#include "tlog.h" - -static threadlocal SExceptionNode* expList; - -void exceptionPushNode(SExceptionNode* node) { - node->prev = expList; - expList = node; -} - -int32_t exceptionPopNode() { - SExceptionNode* node = expList; - expList = node->prev; - return node->code; -} - -void exceptionThrow(int32_t code) { - expList->code = code; - longjmp(expList->jb, 1); -} - -static void cleanupWrapper_void_ptr_ptr(SCleanupAction* ca) { - void (*func)(void*, void*) = ca->func; - func(ca->arg1.Ptr, ca->arg2.Ptr); -} - -static void cleanupWrapper_void_ptr_bool(SCleanupAction* ca) { - void (*func)(void*, bool) = ca->func; - func(ca->arg1.Ptr, ca->arg2.Bool); -} - -static void cleanupWrapper_void_ptr(SCleanupAction* ca) { - void (*func)(void*) = ca->func; - func(ca->arg1.Ptr); -} - -static void cleanupWrapper_int_int(SCleanupAction* ca) { - int32_t (*func)(int32_t) = ca->func; - (void)func(ca->arg1.Int); -} - -static void cleanupWrapper_void(SCleanupAction* ca) { - void (*func)() = ca->func; - func(); -} - -static void cleanupWrapper_int_ptr(SCleanupAction* ca) { - int32_t (*func)(void*) = ca->func; - (void)func(ca->arg1.Ptr); -} - -typedef void (*wrapper)(SCleanupAction*); -static wrapper wrappers[] = { - cleanupWrapper_void_ptr_ptr, cleanupWrapper_void_ptr_bool, cleanupWrapper_void_ptr, - cleanupWrapper_int_int, cleanupWrapper_void, cleanupWrapper_int_ptr, -}; - -void cleanupPush_void_ptr_ptr(bool failOnly, void* func, void* arg1, void* arg2) { - ASSERTS(expList->numCleanupAction < expList->maxCleanupAction, "numCleanupAction over maxCleanupAction"); - - SCleanupAction* ca = expList->cleanupActions + expList->numCleanupAction++; - ca->wrapper = 0; - ca->failOnly = failOnly; - ca->func = func; - ca->arg1.Ptr = arg1; - ca->arg2.Ptr = arg2; -} - -void cleanupPush_void_ptr_bool(bool failOnly, void* func, void* arg1, bool arg2) { - ASSERTS(expList->numCleanupAction < expList->maxCleanupAction, "numCleanupAction over maxCleanupAction"); - - SCleanupAction* ca = expList->cleanupActions + expList->numCleanupAction++; - ca->wrapper = 1; - ca->failOnly = failOnly; - ca->func = func; - ca->arg1.Ptr = arg1; - ca->arg2.Bool = arg2; -} - -void cleanupPush_void_ptr(bool failOnly, void* func, void* arg) { - ASSERTS(expList->numCleanupAction < expList->maxCleanupAction, "numCleanupAction over maxCleanupAction"); - - SCleanupAction* ca = expList->cleanupActions + expList->numCleanupAction++; - ca->wrapper = 2; - ca->failOnly = failOnly; - ca->func = func; - ca->arg1.Ptr = arg; -} - -void cleanupPush_int_int(bool failOnly, void* func, int32_t arg) { - ASSERTS(expList->numCleanupAction < expList->maxCleanupAction, "numCleanupAction over maxCleanupAction"); - - SCleanupAction* ca = expList->cleanupActions + expList->numCleanupAction++; - ca->wrapper = 3; - ca->failOnly = failOnly; - ca->func = func; - ca->arg1.Int = arg; -} - -void cleanupPush_void(bool failOnly, void* func) { - ASSERTS(expList->numCleanupAction < expList->maxCleanupAction, "numCleanupAction over maxCleanupAction"); - - SCleanupAction* ca = expList->cleanupActions + expList->numCleanupAction++; - ca->wrapper = 4; - ca->failOnly = failOnly; - ca->func = func; -} - -void cleanupPush_int_ptr(bool failOnly, void* func, void* arg) { - ASSERTS(expList->numCleanupAction < expList->maxCleanupAction, "numCleanupAction over maxCleanupAction"); - - SCleanupAction* ca = expList->cleanupActions + expList->numCleanupAction++; - ca->wrapper = 5; - ca->failOnly = failOnly; - ca->func = func; - ca->arg1.Ptr = arg; -} - -int32_t cleanupGetActionCount() { return expList->numCleanupAction; } - -static void doExecuteCleanup(SExceptionNode* node, int32_t anchor, bool failed) { - while (node->numCleanupAction > anchor) { - --node->numCleanupAction; - SCleanupAction* ca = node->cleanupActions + node->numCleanupAction; - if (failed || !(ca->failOnly)) { - wrappers[ca->wrapper](ca); - } - } -} - -void cleanupExecuteTo(int32_t anchor, bool failed) { doExecuteCleanup(expList, anchor, failed); } - -void cleanupExecute(SExceptionNode* node, bool failed) { doExecuteCleanup(node, 0, failed); } -bool cleanupExceedLimit() { return expList->numCleanupAction >= expList->maxCleanupAction; } diff --git a/source/util/src/thash.c b/source/util/src/thash.c index 5c1909b16e..aac66348e7 100644 --- a/source/util/src/thash.c +++ b/source/util/src/thash.c @@ -191,14 +191,12 @@ static FORCE_INLINE void doUpdateHashNode(SHashObj *pHashObj, SHashEntry *pe, SH (void)atomic_sub_fetch_16(&pNode->refCount, 1); if (prev != NULL) { prev->next = pNewNode; - ASSERT(prev->next != prev); } else { pe->next = pNewNode; } if (pNode->refCount <= 0) { pNewNode->next = pNode->next; - ASSERT(pNewNode->next != pNewNode); FREE_HASH_NODE(pHashObj->freeFp, pNode); } else { @@ -508,7 +506,6 @@ int32_t taosHashRemove(SHashObj *pHashObj, const void *key, size_t keyLen) { pe->next = pNode->next; } else { prevNode->next = pNode->next; - ASSERT(prevNode->next != prevNode); } pe->num--; @@ -759,12 +756,10 @@ static void *taosHashReleaseNode(SHashObj *pHashObj, void *p, int *slot) { if (pOld->refCount <= 0) { if (prevNode) { prevNode->next = pOld->next; - ASSERT(prevNode->next != prevNode); } else { pe->next = pOld->next; SHashNode *x = pe->next; if (x != NULL) { - ASSERT(x->next != x); } } diff --git a/source/util/src/thashutil.c b/source/util/src/thashutil.c index 9d65977ff1..b466e1b351 100644 --- a/source/util/src/thashutil.c +++ b/source/util/src/thashutil.c @@ -226,10 +226,12 @@ _hash_fn_t taosGetDefaultHashFunction(int32_t type) { } int32_t taosFloatEqual(const void *a, const void *b, size_t UNUSED_PARAM(sz)) { + // getComparFunc(TSDB_DATA_TYPE_FLOAT, -1) will always get function compareFloatVal, which will never be NULL. return getComparFunc(TSDB_DATA_TYPE_FLOAT, -1)(a, b); } int32_t taosDoubleEqual(const void *a, const void *b, size_t UNUSED_PARAM(sz)) { + // getComparFunc(TSDB_DATA_TYPE_DOUBLE, -1) will always get function compareDoubleVal, which will never be NULL. return getComparFunc(TSDB_DATA_TYPE_DOUBLE, -1)(a, b); } diff --git a/source/util/src/tlosertree.c b/source/util/src/tlosertree.c index 7973a84593..06ed9b7690 100644 --- a/source/util/src/tlosertree.c +++ b/source/util/src/tlosertree.c @@ -19,8 +19,11 @@ #include "tlog.h" // Set the initial value of the multiway merge tree. -static void tMergeTreeInit(SMultiwayMergeTreeInfo* pTree) { - ASSERT((pTree->totalSources & 0x01) == 0 && (pTree->numOfSources << 1 == pTree->totalSources)); +static int32_t tMergeTreeInit(SMultiwayMergeTreeInfo* pTree) { + if (!((pTree->totalSources & 0x01) == 0 && (pTree->numOfSources << 1 == pTree->totalSources))) { + uError("losertree failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INVALID_PARA; + } for (int32_t i = 0; i < pTree->totalSources; ++i) { if (i < pTree->numOfSources) { @@ -29,6 +32,7 @@ static void tMergeTreeInit(SMultiwayMergeTreeInfo* pTree) { pTree->pNode[i].index = i - pTree->numOfSources; } } + return TSDB_CODE_SUCCESS; } int32_t tMergeTreeCreate(SMultiwayMergeTreeInfo** pTree, uint32_t numOfSources, void* param, @@ -50,7 +54,11 @@ int32_t tMergeTreeCreate(SMultiwayMergeTreeInfo** pTree, uint32_t numOfSources, pTreeInfo->comparFn = compareFn; // set initial value for loser tree - tMergeTreeInit(pTreeInfo); + int32_t code = tMergeTreeInit(pTreeInfo); + if (TSDB_CODE_SUCCESS != code) { + taosMemoryFree(pTreeInfo); + return code; + } #ifdef _DEBUG_VIEW printf("the initial value of loser tree:\n"); @@ -58,7 +66,11 @@ int32_t tMergeTreeCreate(SMultiwayMergeTreeInfo** pTree, uint32_t numOfSources, #endif for (int32_t i = totalEntries - 1; i >= numOfSources; i--) { - tMergeTreeAdjust(pTreeInfo, i); + code = tMergeTreeAdjust(pTreeInfo, i); + if (TSDB_CODE_SUCCESS != code) { + taosMemoryFree(pTreeInfo); + return code; + } } #if defined(_DEBUG_VIEW) @@ -79,13 +91,17 @@ void tMergeTreeDestroy(SMultiwayMergeTreeInfo** pTree) { taosMemoryFreeClear(*pTree); } -void tMergeTreeAdjust(SMultiwayMergeTreeInfo* pTree, int32_t idx) { - ASSERT(idx <= pTree->totalSources - 1 && idx >= pTree->numOfSources && pTree->totalSources >= 2); +int32_t tMergeTreeAdjust(SMultiwayMergeTreeInfo* pTree, int32_t idx) { + int32_t code = 0; + if (!(idx <= pTree->totalSources - 1 && idx >= pTree->numOfSources && pTree->totalSources >= 2)) { + uError("losertree failed at: %s:%d", __func__, __LINE__); + return TSDB_CODE_INVALID_PARA; + } if (pTree->totalSources == 2) { pTree->pNode[0].index = 0; pTree->pNode[1].index = 0; - return; + return code; } int32_t parentId = idx >> 1; @@ -95,7 +111,7 @@ void tMergeTreeAdjust(SMultiwayMergeTreeInfo* pTree, int32_t idx) { STreeNode* pCur = &pTree->pNode[parentId]; if (pCur->index == -1) { pTree->pNode[parentId] = kLeaf; - return; + return code; } int32_t ret = pTree->comparFn(pCur, &kLeaf, pTree->param); @@ -112,13 +128,21 @@ void tMergeTreeAdjust(SMultiwayMergeTreeInfo* pTree, int32_t idx) { // winner cannot be identical to the loser, which is pTreeNode[1] pTree->pNode[0] = kLeaf; } + return code; } -void tMergeTreeRebuild(SMultiwayMergeTreeInfo* pTree) { - tMergeTreeInit(pTree); - for (int32_t i = pTree->totalSources - 1; i >= pTree->numOfSources; i--) { - tMergeTreeAdjust(pTree, i); +int32_t tMergeTreeRebuild(SMultiwayMergeTreeInfo* pTree) { + int32_t code = tMergeTreeInit(pTree); + if (TSDB_CODE_SUCCESS != code) { + return code; } + for (int32_t i = pTree->totalSources - 1; i >= pTree->numOfSources; i--) { + code = tMergeTreeAdjust(pTree, i); + if (TSDB_CODE_SUCCESS != code) { + return code; + } + } + return TSDB_CODE_SUCCESS; } /* diff --git a/source/util/src/tlrucache.c b/source/util/src/tlrucache.c index 83e0e2d811..33e0076c6e 100644 --- a/source/util/src/tlrucache.c +++ b/source/util/src/tlrucache.c @@ -87,14 +87,11 @@ struct SLRUEntry { #define TAOS_LRU_ENTRY_REF(h) (++(h)->refs) static bool taosLRUEntryUnref(SLRUEntry *entry) { - ASSERT(entry->refs > 0); --entry->refs; return entry->refs == 0; } static void taosLRUEntryFree(SLRUEntry *entry) { - ASSERT(entry->refs == 0); - if (entry->deleter) { (*entry->deleter)(entry->keyData, entry->keyLength, entry->value, entry->ud); } @@ -129,7 +126,6 @@ static void taosLRUEntryTableApply(SLRUEntryTable *table, _taos_lru_table_func_t SLRUEntry *h = table->list[i]; while (h) { SLRUEntry *n = h->nextHash; - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(h)); func(h); h = n; } @@ -155,7 +151,6 @@ static int taosLRUEntryTableApplyF(SLRUEntryTable *table, _taos_lru_functor_t fu SLRUEntry *h = table->list[i]; while (h) { SLRUEntry *n = h->nextHash; - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(h)); ret = functor(h->keyData, h->keyLength, h->value, ud); if (ret) { return ret; @@ -205,7 +200,6 @@ static void taosLRUEntryTableResize(SLRUEntryTable *table) { ++count; } } - ASSERT(table->elems == count); taosMemoryFree(table->list); table->list = newList; @@ -261,17 +255,13 @@ struct SLRUCacheShard { static void taosLRUCacheShardMaintainPoolSize(SLRUCacheShard *shard) { while (shard->highPriPoolUsage > shard->highPriPoolCapacity) { shard->lruLowPri = shard->lruLowPri->next; - ASSERT(shard->lruLowPri != &shard->lru); TAOS_LRU_ENTRY_SET_IN_HIGH_POOL(shard->lruLowPri, false); - ASSERT(shard->highPriPoolUsage >= shard->lruLowPri->totalCharge); shard->highPriPoolUsage -= shard->lruLowPri->totalCharge; } } static void taosLRUCacheShardLRUInsert(SLRUCacheShard *shard, SLRUEntry *e) { - ASSERT(e->next == NULL && e->prev == NULL); - if (shard->highPriPoolRatio > 0 && (TAOS_LRU_ENTRY_IS_HIGH_PRI(e) || TAOS_LRU_ENTRY_HAS_HIT(e))) { e->next = &shard->lru; e->prev = shard->lru.prev; @@ -297,8 +287,6 @@ static void taosLRUCacheShardLRUInsert(SLRUCacheShard *shard, SLRUEntry *e) { } static void taosLRUCacheShardLRURemove(SLRUCacheShard *shard, SLRUEntry *e) { - ASSERT(e->next && e->prev); - if (shard->lruLowPri == e) { shard->lruLowPri = e->prev; } @@ -306,10 +294,8 @@ static void taosLRUCacheShardLRURemove(SLRUCacheShard *shard, SLRUEntry *e) { e->prev->next = e->next; e->prev = e->next = NULL; - ASSERT(shard->lruUsage >= e->totalCharge); shard->lruUsage -= e->totalCharge; if (TAOS_LRU_ENTRY_IN_HIGH_POOL(e)) { - ASSERT(shard->highPriPoolUsage >= e->totalCharge); shard->highPriPoolUsage -= e->totalCharge; } } @@ -317,21 +303,24 @@ static void taosLRUCacheShardLRURemove(SLRUCacheShard *shard, SLRUEntry *e) { static void taosLRUCacheShardEvictLRU(SLRUCacheShard *shard, size_t charge, SArray *deleted) { while (shard->usage + charge > shard->capacity && shard->lru.next != &shard->lru) { SLRUEntry *old = shard->lru.next; - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(old) && !TAOS_LRU_ENTRY_HAS_REFS(old)); taosLRUCacheShardLRURemove(shard, old); (void)taosLRUEntryTableRemove(&shard->table, old->keyData, old->keyLength, old->hash); TAOS_LRU_ENTRY_SET_IN_CACHE(old, false); - ASSERT(shard->usage >= old->totalCharge); shard->usage -= old->totalCharge; - (void)taosArrayPush(deleted, &old); + if (!taosArrayPush(deleted, &old)) { + // ignore this round's eviting + } } } static void taosLRUCacheShardSetCapacity(SLRUCacheShard *shard, size_t capacity) { SArray *lastReferenceList = taosArrayInit(16, POINTER_BYTES); + if (!lastReferenceList) { + return; + } (void)taosThreadMutexLock(&shard->mutex); @@ -395,7 +384,11 @@ static LRUStatus taosLRUCacheShardInsertEntry(SLRUCacheShard *shard, SLRUEntry * if (shard->usage + e->totalCharge > shard->capacity && (shard->strictCapacity || handle == NULL)) { TAOS_LRU_ENTRY_SET_IN_CACHE(e, false); if (handle == NULL) { - (void)taosArrayPush(lastReferenceList, &e); + if (!taosArrayPush(lastReferenceList, &e)) { + (void)taosThreadMutexUnlock(&shard->mutex); + taosLRUEntryFree(e); + return status; + } } else { if (freeOnFail) { taosMemoryFree(e); @@ -411,14 +404,16 @@ static LRUStatus taosLRUCacheShardInsertEntry(SLRUCacheShard *shard, SLRUEntry * if (old != NULL) { status = TAOS_LRU_STATUS_OK_OVERWRITTEN; - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(old)); TAOS_LRU_ENTRY_SET_IN_CACHE(old, false); if (!TAOS_LRU_ENTRY_HAS_REFS(old)) { taosLRUCacheShardLRURemove(shard, old); - ASSERT(shard->usage >= old->totalCharge); shard->usage -= old->totalCharge; - (void)taosArrayPush(lastReferenceList, &old); + if (!taosArrayPush(lastReferenceList, &old)) { + (void)taosThreadMutexUnlock(&shard->mutex); + taosLRUEntryFree(old); + return status; + } } } if (handle == NULL) { @@ -476,7 +471,6 @@ static LRUHandle *taosLRUCacheShardLookup(SLRUCacheShard *shard, const void *key (void)taosThreadMutexLock(&shard->mutex); e = taosLRUEntryTableLookup(&shard->table, key, keyLen, hash); if (e != NULL) { - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(e)); if (!TAOS_LRU_ENTRY_HAS_REFS(e)) { taosLRUCacheShardLRURemove(shard, e); } @@ -495,12 +489,10 @@ static void taosLRUCacheShardErase(SLRUCacheShard *shard, const void *key, size_ SLRUEntry *e = taosLRUEntryTableRemove(&shard->table, key, keyLen, hash); if (e != NULL) { - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(e)); TAOS_LRU_ENTRY_SET_IN_CACHE(e, false); if (!TAOS_LRU_ENTRY_HAS_REFS(e)) { taosLRUCacheShardLRURemove(shard, e); - ASSERT(shard->usage >= e->totalCharge); shard->usage -= e->totalCharge; lastReference = true; } @@ -532,14 +524,15 @@ static void taosLRUCacheShardEraseUnrefEntries(SLRUCacheShard *shard) { while (shard->lru.next != &shard->lru) { SLRUEntry *old = shard->lru.next; - ASSERT(TAOS_LRU_ENTRY_IN_CACHE(old) && !TAOS_LRU_ENTRY_HAS_REFS(old)); taosLRUCacheShardLRURemove(shard, old); (void)taosLRUEntryTableRemove(&shard->table, old->keyData, old->keyLength, old->hash); TAOS_LRU_ENTRY_SET_IN_CACHE(old, false); - ASSERT(shard->usage >= old->totalCharge); shard->usage -= old->totalCharge; - (void)taosArrayPush(lastReferenceList, &old); + if (!taosArrayPush(lastReferenceList, &old)) { + taosLRUEntryFree(old); + return; + } } (void)taosThreadMutexUnlock(&shard->mutex); @@ -557,7 +550,6 @@ static bool taosLRUCacheShardRef(SLRUCacheShard *shard, LRUHandle *handle) { SLRUEntry *e = (SLRUEntry *)handle; (void)taosThreadMutexLock(&shard->mutex); - ASSERT(TAOS_LRU_ENTRY_HAS_REFS(e)); TAOS_LRU_ENTRY_REF(e); (void)taosThreadMutexUnlock(&shard->mutex); @@ -578,8 +570,6 @@ static bool taosLRUCacheShardRelease(SLRUCacheShard *shard, LRUHandle *handle, b lastReference = taosLRUEntryUnref(e); if (lastReference && TAOS_LRU_ENTRY_IN_CACHE(e)) { if (shard->usage > shard->capacity || eraseIfLastRef) { - ASSERT(shard->lru.next == &shard->lru || eraseIfLastRef); - (void)taosLRUEntryTableRemove(&shard->table, e->keyData, e->keyLength, e->hash); TAOS_LRU_ENTRY_SET_IN_CACHE(e, false); } else { @@ -590,7 +580,6 @@ static bool taosLRUCacheShardRelease(SLRUCacheShard *shard, LRUHandle *handle, b } if (lastReference && e->value) { - ASSERT(shard->usage >= e->totalCharge); shard->usage -= e->totalCharge; } @@ -628,7 +617,6 @@ static size_t taosLRUCacheShardGetPinnedUsage(SLRUCacheShard *shard) { (void)taosThreadMutexLock(&shard->mutex); - ASSERT(shard->usage >= shard->lruUsage); usage = shard->usage - shard->lruUsage; (void)taosThreadMutexUnlock(&shard->mutex); @@ -698,8 +686,11 @@ SLRUCache *taosLRUCacheInit(size_t capacity, int numShardBits, double highPriPoo size_t perShard = (capacity + (numShards - 1)) / numShards; for (int i = 0; i < numShards; ++i) { if (TSDB_CODE_SUCCESS != - taosLRUCacheShardInit(&cache->shards[i], perShard, strictCapacity, highPriPoolRatio, 32 - numShardBits)) + taosLRUCacheShardInit(&cache->shards[i], perShard, strictCapacity, highPriPoolRatio, 32 - numShardBits)) { + taosMemoryFree(cache->shards); + taosMemoryFree(cache); return NULL; + } } cache->numShards = numShards; @@ -718,7 +709,6 @@ void taosLRUCacheCleanup(SLRUCache *cache) { if (cache) { if (cache->shards) { int numShards = cache->numShards; - ASSERT(numShards > 0); for (int i = 0; i < numShards; ++i) { taosLRUCacheShardCleanup(&cache->shards[i]); } diff --git a/source/util/src/tpagedbuf.c b/source/util/src/tpagedbuf.c index a1fbe3db2a..32b9b6e18d 100644 --- a/source/util/src/tpagedbuf.c +++ b/source/util/src/tpagedbuf.c @@ -2,8 +2,8 @@ #include "tpagedbuf.h" #include "taoserror.h" #include "tcompression.h" -#include "tsimplehash.h" #include "tlog.h" +#include "tsimplehash.h" #define GET_PAYLOAD_DATA(_p) ((char*)(_p)->pData + POINTER_BYTES) #define BUF_PAGE_IN_MEM(_p) ((_p)->pData != NULL) @@ -27,24 +27,24 @@ struct SPageInfo { }; struct SDiskbasedBuf { - int32_t numOfPages; - int64_t totalBufSize; - uint64_t fileSize; // disk file size - TdFilePtr pFile; - int32_t allocateId; // allocated page id - char* path; // file path - char* prefix; // file name prefix - int32_t pageSize; // current used page size - int32_t inMemPages; // numOfPages that are allocated in memory - SList* freePgList; // free page list - SArray* pIdList; // page id list - SSHashObj*all; - SList* lruList; - void* emptyDummyIdList; // dummy id list - void* assistBuf; // assistant buffer for compress/decompress data - SArray* pFree; // free area in file - bool comp; // compressed before flushed to disk - uint64_t nextPos; // next page flush position + int32_t numOfPages; + int64_t totalBufSize; + uint64_t fileSize; // disk file size + TdFilePtr pFile; + int32_t allocateId; // allocated page id + char* path; // file path + char* prefix; // file name prefix + int32_t pageSize; // current used page size + int32_t inMemPages; // numOfPages that are allocated in memory + SList* freePgList; // free page list + SArray* pIdList; // page id list + SSHashObj* all; + SList* lruList; + void* emptyDummyIdList; // dummy id list + void* assistBuf; // assistant buffer for compress/decompress data + SArray* pFree; // free area in file + bool comp; // compressed before flushed to disk + uint64_t nextPos; // next page flush position char* id; // for debug purpose bool printStatis; // Print statistics info when closing this buffer. @@ -95,7 +95,8 @@ static int32_t doDecompressData(void* data, int32_t srcSize, int32_t* dst, SDisk } else if (*dst < 0) { return terrno; } - return code;; + return code; + ; } static uint64_t allocateNewPositionInFile(SDiskbasedBuf* pBuf, size_t size) { @@ -300,7 +301,6 @@ static SListNode* getEldestUnrefedPage(SDiskbasedBuf* pBuf) { SPageInfo* pageInfo = *(SPageInfo**)pn->data; SPageInfo* p = *(SPageInfo**)(pageInfo->pData); - ASSERT(pageInfo->pageId >= 0 && pageInfo->pn == pn && p == pageInfo); if (!pageInfo->used) { break; @@ -313,7 +313,6 @@ static SListNode* getEldestUnrefedPage(SDiskbasedBuf* pBuf) { static char* evictBufPage(SDiskbasedBuf* pBuf) { SListNode* pn = getEldestUnrefedPage(pBuf); if (pn == NULL) { // no available buffer pages now, return. - terrno = TSDB_CODE_OUT_OF_MEMORY; return NULL; } @@ -435,14 +434,14 @@ static char* doExtractPage(SDiskbasedBuf* pBuf, bool* newPage) { void* getNewBufPage(SDiskbasedBuf* pBuf, int32_t* pageId) { pBuf->statis.getPages += 1; - bool newPage = false; + bool newPage = false; char* availablePage = doExtractPage(pBuf, &newPage); if (availablePage == NULL) { return NULL; } SPageInfo* pi = NULL; - int32_t code = 0; + int32_t code = 0; if (listNEles(pBuf->freePgList) != 0) { SListNode* pItem = tdListPopHead(pBuf->freePgList); pi = *(SPageInfo**)pItem->data; @@ -538,8 +537,6 @@ void* getBufPage(SDiskbasedBuf* pBuf, int32_t id) { #endif return (void*)(GET_PAYLOAD_DATA(*pi)); } else { // not in memory - ASSERT((!BUF_PAGE_IN_MEM(*pi)) && (*pi)->pn == NULL && - (((*pi)->length >= 0 && (*pi)->offset >= 0) || ((*pi)->length == -1 && (*pi)->offset == -1))); bool newPage = false; (*pi)->pData = doExtractPage(pBuf, &newPage); @@ -700,7 +697,7 @@ void setBufPageDirty(void* pPage, bool dirty) { void setBufPageCompressOnDisk(SDiskbasedBuf* pBuf, bool comp) { pBuf->comp = comp; - if (comp && (pBuf->assistBuf == NULL)) { + if (comp && (pBuf->assistBuf == NULL)) { pBuf->assistBuf = taosMemoryMalloc(pBuf->pageSize + 2); // EXTRA BYTES } } diff --git a/source/util/src/trbtree.c b/source/util/src/trbtree.c index dddc2aea4b..fa0862a155 100644 --- a/source/util/src/trbtree.c +++ b/source/util/src/trbtree.c @@ -263,7 +263,6 @@ static void rbtree_delete_fixup(rbtree_t *rbtree, rbnode_t *child, rbnode_t *chi child_parent->color = BLACK; return; } - ASSERTS(sibling != RBTREE_NULL, "sibling is NULL"); /* get a new sibling, by rotating at sibling. See which child of sibling is red */ @@ -293,11 +292,9 @@ static void rbtree_delete_fixup(rbtree_t *rbtree, rbnode_t *child, rbnode_t *chi sibling->color = child_parent->color; child_parent->color = BLACK; if (child_parent->right == child) { - ASSERTS(sibling->left->color == RED, "slibing->left->color=%d not equal RED", sibling->left->color); sibling->left->color = BLACK; rbtree_rotate_right(rbtree, child_parent); } else { - ASSERTS(sibling->right->color == RED, "slibing->right->color=%d not equal RED", sibling->right->color); sibling->right->color = BLACK; rbtree_rotate_left(rbtree, child_parent); } @@ -320,18 +317,15 @@ static void swap_np(rbnode_t **x, rbnode_t **y) { /** Update parent pointers of child trees of 'parent' */ static void change_parent_ptr(rbtree_t *rbtree, rbnode_t *parent, rbnode_t *old, rbnode_t *new) { if (parent == RBTREE_NULL) { - ASSERTS(rbtree->root == old, "root not equal old"); if (rbtree->root == old) rbtree->root = new; return; } - ASSERT(parent->left == old || parent->right == old || parent->left == new || parent->right == new); if (parent->left == old) parent->left = new; if (parent->right == old) parent->right = new; } /** Update parent pointer of a node 'child' */ static void change_child_ptr(rbtree_t *rbtree, rbnode_t *child, rbnode_t *old, rbnode_t *new) { if (child == RBTREE_NULL) return; - ASSERT(child->parent == old || child->parent == new); if (child->parent == old) child->parent = new; } @@ -376,7 +370,6 @@ rbnode_t *rbtree_delete(rbtree_t *rbtree, void *key) { /* now delete to_delete (which is at the location where the smright previously was) */ } - ASSERT(to_delete->left == RBTREE_NULL || to_delete->right == RBTREE_NULL); if (to_delete->left != RBTREE_NULL) child = to_delete->left; diff --git a/source/util/src/tscalablebf.c b/source/util/src/tscalablebf.c index 407223e937..4fac5a2b4f 100644 --- a/source/util/src/tscalablebf.c +++ b/source/util/src/tscalablebf.c @@ -118,7 +118,8 @@ int32_t tScalableBfPut(SScalableBf* pSBf, const void* keyBuf, uint32_t len, int3 } SBloomFilter* pNormalBf = taosArrayGetP(pSBf->bfArray, size - 1); - ASSERT(pNormalBf); + QUERY_CHECK_NULL(pNormalBf, code, lino, _end, TSDB_CODE_INTERNAL_ERROR); + if (tBloomFilterIsFull(pNormalBf)) { code = tScalableBfAddFilter(pSBf, pNormalBf->expectedEntries * pSBf->growth, pNormalBf->errorRate * DEFAULT_TIGHTENING_RATIO, &pNormalBf); diff --git a/source/util/src/tsimplehash.c b/source/util/src/tsimplehash.c index e39c7364b7..d14e72822f 100644 --- a/source/util/src/tsimplehash.c +++ b/source/util/src/tsimplehash.c @@ -261,8 +261,7 @@ int32_t tSimpleHashPut(SSHashObj *pHashObj, const void *key, size_t keyLen, cons static FORCE_INLINE SHNode *doSearchInEntryList(SSHashObj *pHashObj, const void *key, size_t keyLen, int32_t index) { SHNode *pNode = pHashObj->hashList[index]; while (pNode) { - const char* p = GET_SHASH_NODE_KEY(pNode, pNode->dataLen); - ASSERT(keyLen > 0); + const char *p = GET_SHASH_NODE_KEY(pNode, pNode->dataLen); if (pNode->keyLen == keyLen && ((*(pHashObj->equalFp))(p, key, keyLen) == 0)) { break; diff --git a/source/util/src/tskiplist.c b/source/util/src/tskiplist.c index e5e52e44d9..ae01292e08 100644 --- a/source/util/src/tskiplist.c +++ b/source/util/src/tskiplist.c @@ -366,51 +366,6 @@ void *tSkipListDestroyIter(SSkipListIterator *iter) { return NULL; } -#ifdef BUILD_NO_CALL -void tSkipListPrint(SSkipList *pSkipList, int16_t nlevel) { - if (pSkipList == NULL || pSkipList->level < nlevel || nlevel <= 0) { - return; - } - - SSkipListNode *p = SL_NODE_GET_FORWARD_POINTER(pSkipList->pHead, nlevel - 1); - - int32_t id = 1; - char *prev = NULL; - - while (p != pSkipList->pTail) { - char *key = SL_GET_NODE_KEY(pSkipList, p); - if (prev != NULL) { - ASSERT(pSkipList->comparFn(prev, key) < 0); - } - - switch (pSkipList->type) { - case TSDB_DATA_TYPE_INT: - fprintf(stdout, "%d: %d\n", id++, *(int32_t *)key); - break; - case TSDB_DATA_TYPE_SMALLINT: - case TSDB_DATA_TYPE_TINYINT: - case TSDB_DATA_TYPE_BIGINT: - fprintf(stdout, "%d: %" PRId64 " \n", id++, *(int64_t *)key); - break; - case TSDB_DATA_TYPE_BINARY: - case TSDB_DATA_TYPE_VARBINARY: - case TSDB_DATA_TYPE_GEOMETRY: - fprintf(stdout, "%d: %s \n", id++, key); - break; - case TSDB_DATA_TYPE_DOUBLE: - fprintf(stdout, "%d: %lf \n", id++, *(double *)key); - break; - default: - fprintf(stdout, "\n"); - } - - prev = SL_GET_NODE_KEY(pSkipList, p); - - p = SL_NODE_GET_FORWARD_POINTER(p, nlevel - 1); - } -} -#endif - static void tSkipListDoInsert(SSkipList *pSkipList, SSkipListNode **direction, SSkipListNode *pNode, bool isForward) { for (int32_t i = 0; i < pNode->level; ++i) { SSkipListNode *x = direction[i]; @@ -538,33 +493,6 @@ static bool tSkipListGetPosToPut(SSkipList *pSkipList, SSkipListNode **backward, return hasDupKey; } -#ifdef BUILD_NO_CALL -static void tSkipListRemoveNodeImpl(SSkipList *pSkipList, SSkipListNode *pNode) { - int32_t level = pNode->level; - uint8_t dupMode = SL_DUP_MODE(pSkipList); - ASSERT(dupMode != SL_DISCARD_DUP_KEY && dupMode != SL_UPDATE_DUP_KEY); - - for (int32_t j = level - 1; j >= 0; --j) { - SSkipListNode *prev = SL_NODE_GET_BACKWARD_POINTER(pNode, j); - SSkipListNode *next = SL_NODE_GET_FORWARD_POINTER(pNode, j); - - SL_NODE_GET_FORWARD_POINTER(prev, j) = next; - SL_NODE_GET_BACKWARD_POINTER(next, j) = prev; - } - - tSkipListFreeNode(pNode); - pSkipList->size--; -} - -// Function must be called after calling tSkipListRemoveNodeImpl() function -static void tSkipListCorrectLevel(SSkipList *pSkipList) { - while (pSkipList->level > 0 && - SL_NODE_GET_FORWARD_POINTER(pSkipList->pHead, pSkipList->level - 1) == pSkipList->pTail) { - pSkipList->level -= 1; - } -} -#endif - UNUSED_FUNC static FORCE_INLINE void recordNodeEachLevel(SSkipList *pSkipList, int32_t level) { // record link count in each level #if SKIP_LIST_RECORD_PERFORMANCE diff --git a/source/util/src/tutil.c b/source/util/src/tutil.c index 94f514e208..081acda20f 100644 --- a/source/util/src/tutil.c +++ b/source/util/src/tutil.c @@ -124,7 +124,6 @@ char **strsplit(char *z, const char *delim, int32_t *num) { if (split == NULL) { return NULL; } - ASSERTS(NULL != split, "realloc memory failed. size=%d", (int32_t)POINTER_BYTES * size); } } diff --git a/tests/army/query/last/test_last.py b/tests/army/query/last/test_last.py new file mode 100644 index 0000000000..287cc97974 --- /dev/null +++ b/tests/army/query/last/test_last.py @@ -0,0 +1,192 @@ +from frame.log import * +from frame.cases import * +from frame.sql import * +from frame.caseBase import * +from frame import * +from frame.eos import * + + +class TDTestCase(TBase): + """Add test case to verify TD-30816 (last/last_row accuracy) + """ + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + + def prepare_data(self): + tdSql.execute("create database db_td30816 cachemodel 'both';") + tdSql.execute("use db_td30816;") + # create regular table + tdSql.execute("create table rt_int (ts timestamp, c1 int primary key, c2 int);") + tdSql.execute("create table rt_str (ts timestamp, c1 varchar(16) primary key, c2 varchar(16));") + + # create stable + tdSql.execute("create table st_pk_int (ts timestamp, c1 int primary key, c2 int) tags (t1 int);") + tdSql.execute("create table st_pk_str (ts timestamp, c1 varchar(16) primary key, c2 varchar(16)) tags (t1 int);") + + # create child table + tdSql.execute("create table ct1 using st_pk_int tags(1);") + tdSql.execute("create table ct2 using st_pk_int tags(2);") + + tdSql.execute("create table ct3 using st_pk_str tags(3);") + tdSql.execute("create table ct4 using st_pk_str tags(4);") + + # insert data to regular table + tdSql.execute("insert into rt_int values ('2021-01-01 00:00:00', 1, NULL);") + tdSql.execute("insert into rt_int values ('2021-01-01 00:00:01', 2, 1);") + tdSql.execute("insert into rt_str values ('2021-01-01 00:00:00', 'a', NULL);") + tdSql.execute("insert into rt_str values ('2021-01-01 00:00:01', 'b', '1');") + + # insert data to child table + tdSql.execute("insert into ct1 values ('2021-01-01 00:00:00', 1, 1);") + tdSql.execute("insert into ct1 values ('2021-01-01 00:00:01', 2, NULL);") + tdSql.execute("insert into ct2 values ('2021-01-01 00:00:00', 3, 3);") + tdSql.execute("insert into ct2 values ('2021-01-01 00:00:01', 4, NULL);") + + tdSql.execute("insert into ct3 values ('2021-01-01 00:00:00', 'a', '1');") + tdSql.execute("insert into ct3 values ('2021-01-01 00:00:01', 'b', NULL);") + tdSql.execute("insert into ct4 values ('2021-01-01 00:00:00', 'c', '3');") + tdSql.execute("insert into ct4 values ('2021-01-01 00:00:01', 'd', NULL);") + + def test_last_with_primarykey_int_ct(self): + tdSql.execute("use db_td30816;") + tdSql.query("select last(*) from st_pk_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 3) + + tdSql.query("select last_row(*) from st_pk_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, None) + + # delete and insert data + tdSql.execute("delete from ct1 where ts='2021-01-01 00:00:01';") + tdSql.execute("delete from ct2 where ts='2021-01-01 00:00:01';") + tdSql.execute("insert into ct1 values ('2021-01-01 00:00:00', 0, 5);") + tdSql.execute("insert into ct2 values ('2021-01-01 00:00:00', -1, 6);") + tdSql.query("select last(*) from st_pk_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 2, 3) + + tdSql.query("select last_row(*) from st_pk_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 2, 3) + tdLog.info("Finish test_last_with_primarykey_int_ct") + + def test_last_with_primarykey_str_ct(self): + tdSql.execute("use db_td30816;") + tdSql.query("select last(*) from st_pk_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 'd') + tdSql.checkData(0, 2, '3') + + tdSql.query("select last_row(*) from st_pk_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 'd') + tdSql.checkData(0, 2, None) + + # delete and insert data + tdSql.execute("delete from ct3 where ts='2021-01-01 00:00:01';") + tdSql.execute("delete from ct4 where ts='2021-01-01 00:00:01';") + tdSql.execute("insert into ct3 values ('2021-01-01 00:00:00', '6', '5');") + tdSql.execute("insert into ct4 values ('2021-01-01 00:00:00', '7', '6');") + + tdSql.query("select last(*) from st_pk_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 'c') + tdSql.checkData(0, 2, '3') + + tdSql.query("select last_row(*) from st_pk_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 'c') + tdSql.checkData(0, 2, 3) + tdLog.info("Finish test_last_with_primarykey_str_ct") + + def test_last_with_primarykey_int_rt(self): + tdSql.execute("use db_td30816;") + tdSql.query("select last(*) from rt_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 2) + tdSql.checkData(0, 2, 1) + + tdSql.query("select last_row(*) from rt_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 2) + tdSql.checkData(0, 2, 1) + + tdSql.execute("delete from rt_int where ts='2021-01-01 00:00:01';") + tdSql.execute("insert into rt_int values ('2021-01-01 00:00:00', 0, 5);") + + tdSql.query("select last(*) from rt_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 5) + + tdSql.query("select last_row(*) from rt_int;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, None) + tdLog.info("Finish test_last_with_primarykey_int_rt") + + def test_last_with_primarykey_str_rt(self): + tdSql.execute("use db_td30816;") + tdSql.query("select last(*) from rt_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 'b') + tdSql.checkData(0, 2, '1') + + tdSql.query("select last_row(*) from rt_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:01') + tdSql.checkData(0, 1, 'b') + tdSql.checkData(0, 2, '1') + + tdSql.execute("delete from rt_str where ts='2021-01-01 00:00:01';") + tdSql.execute("insert into rt_str values ('2021-01-01 00:00:00', '2', '5');") + + tdSql.query("select last(*) from rt_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 'a') + tdSql.checkData(0, 2, '5') + + tdSql.query("select last_row(*) from rt_str;") + tdSql.checkRows(1) + tdSql.checkData(0, 0, '2021-01-01 00:00:00') + tdSql.checkData(0, 1, 'a') + tdSql.checkData(0, 2, None) + tdLog.info("Finish test_last_with_primarykey_str_rt") + + def run(self): + self.prepare_data() + # regular table + self.test_last_with_primarykey_int_rt() + self.test_last_with_primarykey_str_rt() + # child tables + self.test_last_with_primarykey_int_ct() + self.test_last_with_primarykey_str_ct() + + def stop(self): + tdSql.execute("drop database db_td30816;") + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/develop-test/2-query/show_create_db.py b/tests/develop-test/2-query/show_create_db.py index 3fe68a6803..d7d093aa78 100644 --- a/tests/develop-test/2-query/show_create_db.py +++ b/tests/develop-test/2-query/show_create_db.py @@ -42,17 +42,17 @@ class TDTestCase: tdSql.query('show create database scd;') tdSql.checkRows(1) tdSql.checkData(0, 0, 'scd') - tdSql.checkData(0, 1, "CREATE DATABASE `scd` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 2 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") + tdSql.checkData(0, 1, "CREATE DATABASE `scd` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 10d WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 2 KEEP 3650d,3650d,3650d PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") tdSql.query('show create database scd2;') tdSql.checkRows(1) tdSql.checkData(0, 0, 'scd2') - tdSql.checkData(0, 1, "CREATE DATABASE `scd2` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 3 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") + tdSql.checkData(0, 1, "CREATE DATABASE `scd2` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 10d WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 3 KEEP 3650d,3650d,3650d PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") tdSql.query('show create database scd4') tdSql.checkRows(1) tdSql.checkData(0, 0, 'scd4') - tdSql.checkData(0, 1, "CREATE DATABASE `scd4` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 13 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") + tdSql.checkData(0, 1, "CREATE DATABASE `scd4` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 10d WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 13 KEEP 3650d,3650d,3650d PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") self.restartTaosd(1, dbname='scd') @@ -60,16 +60,16 @@ class TDTestCase: tdSql.query('show create database scd;') tdSql.checkRows(1) tdSql.checkData(0, 0, 'scd') - tdSql.checkData(0, 1, "CREATE DATABASE `scd` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 2 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") + tdSql.checkData(0, 1, "CREATE DATABASE `scd` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 10d WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 2 KEEP 3650d,3650d,3650d PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") tdSql.query('show create database scd2;') tdSql.checkRows(1) tdSql.checkData(0, 0, 'scd2') - tdSql.checkData(0, 1, "CREATE DATABASE `scd2` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 3 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") + tdSql.checkData(0, 1, "CREATE DATABASE `scd2` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 10d WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 3 KEEP 3650d,3650d,3650d PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") tdSql.query('show create database scd4') tdSql.checkRows(1) tdSql.checkData(0, 0, 'scd4') - tdSql.checkData(0, 1, "CREATE DATABASE `scd4` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 13 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") + tdSql.checkData(0, 1, "CREATE DATABASE `scd4` BUFFER 256 CACHESIZE 1 CACHEMODEL 'none' COMP 2 DURATION 10d WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 13 KEEP 3650d,3650d,3650d PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 2 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0 KEEP_TIME_OFFSET 0 ENCRYPT_ALGORITHM 'none' S3_CHUNKSIZE 262144 S3_KEEPLOCAL 5256000m S3_COMPACT 0") tdSql.execute('drop database scd') diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 3553f0d913..3ec1d7c428 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -38,6 +38,7 @@ ,,y,army,./pytest.sh python3 ./test.py -f query/queryBugs.py -N 3 ,,y,army,./pytest.sh python3 ./test.py -f tmq/tmqBugs.py -N 3 ,,y,army,./pytest.sh python3 ./test.py -f query/fill/fill_compare_asc_desc.py +,,y,army,./pytest.sh python3 ./test.py -f query/last/test_last.py # # system test @@ -310,7 +311,7 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/balance_vgroups_r1.py -N 6 ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/taosShell.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/taosShellError.py -,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/taosShellNetChk.py +,,n,system-test,python3 ./test.py -f 0-others/taosShellNetChk.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/telemetry.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/backquote_check.py ,,y,system-test,./pytest.sh python3 ./test.py -f 0-others/taosdMonitor.py diff --git a/tests/script/sh/checkAsan.sh b/tests/script/sh/checkAsan.sh index b1571aa173..aa73d4cb23 100755 --- a/tests/script/sh/checkAsan.sh +++ b/tests/script/sh/checkAsan.sh @@ -49,6 +49,7 @@ fi indirect_leak=$(cat ${LOG_DIR}/*.asan | grep "Indirect leak" | wc -l) python_error=$(cat ${LOG_DIR}/*.info | grep -w "stack" | wc -l) +python_taos_error=$(cat ${LOG_DIR}/*.info |grep "#" | grep -w "TDinternal" | wc -l) # ignore @@ -84,17 +85,18 @@ echo -e "\033[44;32;1m"asan memory_leak: $memory_leak"\033[0m" echo -e "\033[44;32;1m"asan indirect_leak: $indirect_leak"\033[0m" echo -e "\033[44;32;1m"asan runtime error: $runtime_error"\033[0m" echo -e "\033[44;32;1m"asan python error: $python_error"\033[0m" +echo -e "\033[44;32;1m"asan python taos error: $python_taos_error"\033[0m" -let "errors=$error_num+$memory_leak+$indirect_leak+$runtime_error+$python_error" +let "errors=$error_num+$memory_leak+$indirect_leak+$runtime_error+$python_error+$python_taos_error" if [ $errors -eq 0 ]; then echo -e "\033[44;32;1m"no asan errors"\033[0m" exit 0 else echo -e "\033[44;31;1m"asan total errors: $errors"\033[0m" - if [ $python_error -ne 0 ]; then - cat ${LOG_DIR}/*.info + if [ $python_error -ne 0 ] || [ $python_taos_error -ne 0 ] ; then + cat ${LOG_DIR}/*.info |grep "#" | grep -w "TDinternal" fi cat ${LOG_DIR}/*.asan exit 1 -fi +fi \ No newline at end of file diff --git a/tests/script/tsim/query/event.sim b/tests/script/tsim/query/event.sim index 16fe0f4a13..740cdb11dd 100644 --- a/tests/script/tsim/query/event.sim +++ b/tests/script/tsim/query/event.sim @@ -236,4 +236,54 @@ sql_error select count(*) from sta event_window start with f1 >0 end with f2 > 0 sql_error select count(*) from sta event_window start with f1 >0 end with f2 > 0 group by tbname; sql_error select count(*) from sta event_window start with f1 >0 end with f2 > 0 fill(NULL); +print step2 +print =============== create database +sql create database test2 vgroups 4; +sql use test2; +sql create stable st(ts timestamp,a int,b int,c int, d double) tags(ta int,tb int,tc int); +sql create table t1 using st tags(1,1,1); +sql create table t2 using st tags(2,2,2); +sql create table t3 using st tags(3,3,3); + +sql insert into t1 values(1648791213000,1,2,3,1.0); +sql insert into t1 values(1648791223001,2,2,3,1.1); +sql insert into t1 values(1648791233002,3,2,3,2.1); +sql insert into t1 values(1648791243003,4,2,3,3.1); +sql insert into t1 values(1648791253004,5,2,3,4.1); + +sql insert into t2 values(1648791213000,1,2,3,1.0); +sql insert into t2 values(1648791223001,2,2,3,1.1); +sql insert into t2 values(1648791233002,3,2,3,2.1); +sql insert into t2 values(1648791243003,4,2,3,3.1); +sql insert into t2 values(1648791253004,5,2,3,4.1); + +sql insert into t3 values(1648791213000,1,2,3,1.0); +sql insert into t3 values(1648791223001,2,2,3,1.1); +sql insert into t3 values(1648791233002,3,2,3,2.1); +sql insert into t3 values(1648791243003,4,2,3,3.1); +sql insert into t3 values(1648791253004,5,2,3,4.1); + +$loop_count = 0 +loop4: + +sleep 300 + +sql select _wstart, count(*) c1, tbname from st partition by tbname event_window start with a > 0 end with b = 2 slimit 2 limit 2; + +print $data00 $data01 $data02 $data03 +print $data10 $data11 $data12 $data13 +print $data20 $data21 $data22 $data23 +print $data30 $data31 $data32 $data33 + +$loop_count = $loop_count + 1 +if $loop_count == 10 then + return -1 +endi + +# row 0 +if $rows != 4 then + print ======rows=$rows + goto loop4 +endi + system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/script/tsim/query/query_count1.sim b/tests/script/tsim/query/query_count1.sim index 043b604263..9711a09d97 100644 --- a/tests/script/tsim/query/query_count1.sim +++ b/tests/script/tsim/query/query_count1.sim @@ -101,5 +101,57 @@ sql_error select _wstart as s, count(*) c1, sum(b), max(c) from t1 count_windo sql select _wstart as s, count(*) c1, sum(b), max(c) from t1 count_window(2); sql select _wstart as s, count(*) c1, sum(b), max(c) from t1 count_window(2147483647); + +print step3 +print =============== create database +sql create database test2 vgroups 4; +sql use test2; +sql create stable st(ts timestamp,a int,b int,c int, d double) tags(ta int,tb int,tc int); +sql create table t1 using st tags(1,1,1); +sql create table t2 using st tags(2,2,2); +sql create table t3 using st tags(3,3,3); + +sql insert into t1 values(1648791213000,1,2,3,1.0); +sql insert into t1 values(1648791223001,2,2,3,1.1); +sql insert into t1 values(1648791233002,3,2,3,2.1); +sql insert into t1 values(1648791243003,4,2,3,3.1); +sql insert into t1 values(1648791253004,5,2,3,4.1); + +sql insert into t2 values(1648791213000,1,2,3,1.0); +sql insert into t2 values(1648791223001,2,2,3,1.1); +sql insert into t2 values(1648791233002,3,2,3,2.1); +sql insert into t2 values(1648791243003,4,2,3,3.1); +sql insert into t2 values(1648791253004,5,2,3,4.1); + +sql insert into t3 values(1648791213000,1,2,3,1.0); +sql insert into t3 values(1648791223001,2,2,3,1.1); +sql insert into t3 values(1648791233002,3,2,3,2.1); +sql insert into t3 values(1648791243003,4,2,3,3.1); +sql insert into t3 values(1648791253004,5,2,3,4.1); + +$loop_count = 0 +loop4: + +sleep 300 + +sql select _wstart, count(*) c1, tbname from st partition by tbname count_window(2) slimit 2 limit 2; + +print $data00 $data01 $data02 $data03 +print $data10 $data11 $data12 $data13 +print $data20 $data21 $data22 $data23 +print $data30 $data31 $data32 $data33 + +$loop_count = $loop_count + 1 +if $loop_count == 10 then + return -1 +endi + +# row 0 +if $rows != 4 then + print ======rows=$rows + goto loop4 +endi + + print query_count0 end system sh/exec.sh -n dnode1 -s stop -x SIGINT diff --git a/tests/system-test/2-query/concat.py b/tests/system-test/2-query/concat.py index 326f6940f6..b43d4fafa8 100644 --- a/tests/system-test/2-query/concat.py +++ b/tests/system-test/2-query/concat.py @@ -144,6 +144,9 @@ class TDTestCase: f"{dbname}.ct2", f"{dbname}.ct4", ] + tdSql.query("select concat(null,null)") # TD-31571 + tdSql.checkRows(1) + tdSql.checkData(0, 0, None) for tb in tbname: for i in range(2,8): self.__concat_check(tb,i) diff --git a/tests/system-test/2-query/concat_ws.py b/tests/system-test/2-query/concat_ws.py index 26731715c1..338166c61e 100644 --- a/tests/system-test/2-query/concat_ws.py +++ b/tests/system-test/2-query/concat_ws.py @@ -156,7 +156,9 @@ class TDTestCase: f"{dbname}.ct2", f"{dbname}.ct4", ] - + tdSql.query("select concat_ws(null,null,null);") # TD-31572 + tdSql.checkRows(1) + tdSql.checkData(0, 0, None) for tb in tbname: for errsql in self.__concat_ws_err_check(tb): tdSql.error(sql=errsql) diff --git a/tests/system-test/2-query/ttl_comment.py b/tests/system-test/2-query/ttl_comment.py index ba24579611..b332cd3afb 100644 --- a/tests/system-test/2-query/ttl_comment.py +++ b/tests/system-test/2-query/ttl_comment.py @@ -36,12 +36,12 @@ class TDTestCase: tdSql.error(f"create table {dbname}.ttl_table1(ts timestamp, i int) ttl 1.1") tdSql.error(f"create table {dbname}.ttl_table2(ts timestamp, i int) ttl 1e1") tdSql.error(f"create table {dbname}.ttl_table3(ts timestamp, i int) ttl -1") + tdSql.error(f"create table {dbname}.ttl_table4(ts timestamp, i int) ttl 2147483648") print("============== STEP 1 ===== test normal table") tdSql.execute(f"create table {dbname}.normal_table1(ts timestamp, i int)") tdSql.execute(f"create table {dbname}.normal_table2(ts timestamp, i int) comment '' ttl 3") - tdSql.execute(f"create table {dbname}.normal_table3(ts timestamp, i int) ttl 2100000000020 comment 'hello'") tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table1'") tdSql.checkData(0, 0, 'normal_table1') @@ -54,12 +54,6 @@ class TDTestCase: tdSql.checkData(0, 7, 3) tdSql.checkData(0, 8, '') - - tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table3'") - tdSql.checkData(0, 0, 'normal_table3') - tdSql.checkData(0, 7, 2147483647) - tdSql.checkData(0, 8, 'hello') - tdSql.execute(f"alter table {dbname}.normal_table1 comment 'nihao'") tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table1'") tdSql.checkData(0, 0, 'normal_table1') @@ -75,19 +69,14 @@ class TDTestCase: tdSql.checkData(0, 0, 'normal_table2') tdSql.checkData(0, 8, 'fly') - tdSql.execute(f"alter table {dbname}.normal_table3 comment 'fly'") - tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table3'") - tdSql.checkData(0, 0, 'normal_table3') - tdSql.checkData(0, 8, 'fly') - tdSql.execute(f"alter table {dbname}.normal_table1 ttl 1") tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table1'") tdSql.checkData(0, 0, 'normal_table1') tdSql.checkData(0, 7, 1) - tdSql.execute(f"alter table {dbname}.normal_table3 ttl 0") - tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table3'") - tdSql.checkData(0, 0, 'normal_table3') + tdSql.execute(f"alter table {dbname}.normal_table2 ttl 0") + tdSql.query("select * from information_schema.ins_tables where table_name like 'normal_table2'") + tdSql.checkData(0, 0, 'normal_table2') tdSql.checkData(0, 7, 0) diff --git a/tools/shell/src/shellAuto.c b/tools/shell/src/shellAuto.c index 2c92d19fcf..9b1c949807 100644 --- a/tools/shell/src/shellAuto.c +++ b/tools/shell/src/shellAuto.c @@ -585,7 +585,7 @@ void showHelp() { balance vgroup ;\n\ balance vgroup leader on \n\ compact database ; \n\ - crate view as select ...\n\ + create view as select ...\n\ redistribute vgroup dnode ;\n\ split vgroup ;\n\ show compacts;\n\ diff --git a/tools/shell/src/shellEngine.c b/tools/shell/src/shellEngine.c index 0ccbd683dc..6fd5f7402f 100644 --- a/tools/shell/src/shellEngine.c +++ b/tools/shell/src/shellEngine.c @@ -164,11 +164,13 @@ int32_t shellRunCommand(char *command, bool recordHistory) { } // add help or help; - if (strncasecmp(command, "help;", 5) == 0) { - showHelp(); - return 0; + if (strncasecmp(command, "help", 4) == 0) { + if(command[4] == ';' || command[4] == ' ' || command[4] == 0) { + showHelp(); + return 0; + } } - + if (recordHistory) shellRecordCommandToHistory(command); char quote = 0, *cmd = command;