diff --git a/docs/en/10-third-party/05-bi/03-powerbi.md b/docs/en/10-third-party/05-bi/03-powerbi.md index b514a58753..85746da45b 100644 --- a/docs/en/10-third-party/05-bi/03-powerbi.md +++ b/docs/en/10-third-party/05-bi/03-powerbi.md @@ -7,36 +7,24 @@ Power BI is a business analytics tool provided by Microsoft. By configuring the ## Prerequisites -Install and run Power BI Desktop software (if not installed, please download the latest version for Windows OS 32/64 bit from its official address). +- TDengine 3.3.4.0 and above version is installed and running normally (both Enterprise and Community versions are available). +- taosAdapter is running normally, refer to [taosAdapter Reference](../../../tdengine-reference/components/taosadapter/). +- Install and run Power BI Desktop software (if not installed, please download the latest version for Windows OS 32/64 bit from its official address). +- Download the latest Windows OS X64 client driver from the TDengine official website and install it on the machine running Power BI. After successful installation, the TDengine driver can be seen in the "ODBC Data Sources (32-bit)" or "ODBC Data Sources (64-bit)" management tool. -## Install ODBC Driver +## Configure Data Source -Download the latest Windows OS X64 client driver from the TDengine official website and install it on the machine running Power BI. After successful installation, the TDengine driver can be seen in the "ODBC Data Sources (32-bit)" or "ODBC Data Sources (64-bit)" management tool. +**Step 1**, Search and open the [ODBC Data Source (64 bit)] management tool in the Start menu of the Windows operating system and configure it, refer to [Install ODBC Driver](../../../tdengine-reference/client-libraries/odbc/#Installation). + +**Step 2**, Open Power BI and log in, click [Home] -> [Get Data] -> [Other] -> [ODBC] -> [Connect], add data source. -## Configure ODBC Data Source +**Step 3**, Select the data source name just created, such as [MyTDengine], if you need to enter SQL, you can click the [Advanced options] tab, in the expanded dialog box enter the SQL statement. Click the [OK] button to connect to the configured data source. -The steps to configure the ODBC data source are as follows. +**Step 4**, Enter the [Navigator], you can browse the corresponding database's tables/views and load data. -Step 1, search and open "ODBC Data Sources (32-bit)" or "ODBC Data Sources (64-bit)" management tool from the Windows start menu. -Step 2, click the "User DSN" tab → "Add" button, enter the "Create New Data Source" dialog box. -Step 3, in the list of "Select the driver you want to install for this data source" choose "TDengine", click the "Finish" button, enter the TDengine ODBC data source configuration page. Fill in the necessary information as follows. +## Data Analysis -- DSN: Data source name, required, such as "MyTDengine". -- Connection Type: Check the "WebSocket" checkbox. -- URL: ODBC data source URL, required, such as `http://127.0.0.1:6041`. -- Database: Indicates the database to connect to, optional, such as "test". -- Username: Enter username, if not filled, default is "root". -- Password: Enter user password, if not filled, default is "taosdata". - -Step 4, click the "Test Connection" button, test the connection situation, if successfully connected, it will prompt "Successfully connected to `http://127.0.0.1:6041`". -Step 5, click the "OK" button, to save the configuration and exit. - -## Import TDengine Data into Power BI - -The steps to import TDengine data into Power BI are as follows: -Step 1, open Power BI and log in, click "Home" → "Get Data" → "Other" → "ODBC" → "Connect", add data source. -Step 2, select the data source name just created, such as "MyTDengine", if you need to enter SQL, you can click the "Advanced options" tab, in the expanded dialog box enter the SQL statement. Click the "OK" button to connect to the configured data source. -Step 3, enter the "Navigator", you can browse the corresponding database's tables/views and load data. +### Instructions for use To fully leverage Power BI's advantages in analyzing data from TDengine, users need to first understand core concepts such as dimensions, metrics, window split queries, data split queries, time-series, and correlation, then import data through custom SQL. @@ -47,7 +35,7 @@ To fully leverage Power BI's advantages in analyzing data from TDengine, users n - Time-Series: When drawing curves or aggregating data by time, it is usually necessary to introduce a date table. Date tables can be imported from Excel spreadsheets, or obtained in TDengine by executing SQL like `select _wstart date, count(*) cnt from test.meters where ts between A and B interval(1d) fill(0)`, where the fill clause represents the filling mode in case of data missing, and the pseudocolumn `_wstart` is the date column to be obtained. - Correlation: Tells how data is related, such as metrics and dimensions can be associated together through the tbname column, date tables and metrics can be associated through the date column, combined to form visual reports. -## Smart Meter Example +### Smart Meter Example TDengine employs a unique data model to optimize the storage and query performance of time-series data. This model uses supertables as templates to create an independent table for each device. Each table is designed with high scalability in mind, supporting up to 4096 data columns and 128 tag columns. This design enables TDengine to efficiently handle large volumes of time-series data while maintaining flexibility and ease of use. @@ -56,24 +44,35 @@ Taking smart meters as an example, suppose each meter generates one record per s In Power BI, users can map the tag columns in TDengine tables to dimension columns for grouping and filtering data. Meanwhile, the aggregated results of the data columns can be imported as measure columns for calculating key indicators and generating reports. In this way, Power BI helps decision-makers quickly obtain the information they need, gain a deeper understanding of business operations, and make more informed decisions. Follow the steps below to experience the functionality of generating time-series data reports through Power BI. -Step 1, Use TDengine's taosBenchMark to quickly generate data for 1,000 smart meters over 3 days, with a collection frequency of 1s. - ```shell - taosBenchmark -t 1000 -n 259200 -S 1000 -y - ``` -Step 2, Import dimension data. In Power BI, import the tag columns of the table, named as tags, using the following SQL to get the tag data of all smart meters under the supertable. - ```sql - select distinct tbname device, groupId, location from test.meters - ``` -Step 3, Import measure data. In Power BI, import the average current, voltage, and phase of each smart meter in 1-hour time windows, named as data, with the following SQL. - ```sql - select tbname, _wstart ws, avg(current), avg(voltage), avg(phase) from test.meters PARTITION by tbname interval(1h) - ``` -Step 4, Import date data. Using a 1-day time window, obtain the time range and data count of the time-series data, with the following SQL. In the Power Query editor, convert the format of the date column from "text" to "date". - ```sql - select _wstart date, count(*) from test.meters interval(1d) having count(*)>0 - ``` -Step 5, Establish the relationship between dimensions and measures. Open the model view and establish the relationship between the tags and data tables, setting tbname as the relationship data column. -Step 6, Establish the relationship between date and measures. Open the model view and establish the relationship between the date dataset and data, with the relationship data columns being date and datatime. -Step 7, Create reports. Use these data in bar charts, pie charts, and other controls. + +**Step 1**, Use TDengine's taosBenchMark to quickly generate data for 1,000 smart meters over 3 days, with a collection frequency of 1s. + +```shell +taosBenchmark -t 1000 -n 259200 -S 1000 -y +``` + +**Step 2**, Import dimension data. In Power BI, import the tag columns of the table, named as tags, using the following SQL to get the tag data of all smart meters under the supertable. + +```sql +select distinct tbname device, groupId, location from test.meters +``` + +**Step 3**, Import measure data. In Power BI, import the average current, voltage, and phase of each smart meter in 1-hour time windows, named as data, with the following SQL. + +```sql +select tbname, _wstart ws, avg(current), avg(voltage), avg(phase) from test.meters PARTITION by tbname interval(1h) +``` + +**Step 4**, Import date data. Using a 1-day time window, obtain the time range and data count of the time-series data, with the following SQL. In the Power Query editor, convert the format of the date column from "text" to "date". + +```sql +select _wstart date, count(*) from test.meters interval(1d) having count(*)>0 +``` + +**Step 5**, Establish the relationship between dimensions and measures. Open the model view and establish the relationship between the tags and data tables, setting tbname as the relationship data column. + +**Step 6**, Establish the relationship between date and measures. Open the model view and establish the relationship between the date dataset and data, with the relationship data columns being date and datatime. + +**Step 7**, Create reports. Use these data in bar charts, pie charts, and other controls. Due to TDengine's superior performance in handling time-series data, users can enjoy a very good experience during data import and daily regular data refreshes. For more information on building Power BI visual effects, please refer to the official Power BI documentation. diff --git a/docs/en/10-third-party/05-bi/09-seeq.md b/docs/en/10-third-party/05-bi/09-seeq.md index 7fb7569461..ba33d801a9 100644 --- a/docs/en/10-third-party/05-bi/09-seeq.md +++ b/docs/en/10-third-party/05-bi/09-seeq.md @@ -11,43 +11,42 @@ import imgStep04 from '../../assets/seeq-04.png'; Seeq is advanced analytics software for the manufacturing and Industrial Internet of Things (IIOT). Seeq supports innovative new features using machine learning in process manufacturing organizations. These features enable organizations to deploy their own or third-party machine learning algorithms to advanced analytics applications used by frontline process engineers and subject matter experts, thus extending the efforts of a single data scientist to many frontline staff. -Through the TDengine Java connector, Seeq can easily support querying time-series data provided by TDengine and offer data presentation, analysis, prediction, and other functions. +Through the `TDengine Java connector`, Seeq can easily support querying time-series data provided by TDengine and offer data presentation, analysis, prediction, and other functions. ## Prerequisites -- Seeq has been installed. Download the relevant software from [Seeq's official website](https://www.seeq.com/customer-download), such as Seeq Server and Seeq Data Lab, etc. Seeq Data Lab needs to be installed on a different server from Seeq Server and interconnected through configuration. For detailed installation and configuration instructions, refer to the [Seeq Knowledge Base](https://support.seeq.com/kb/latest/cloud/). +- TDengine 3.1.0.3 and above version is installed and running normally (both Enterprise and Community versions are available). +- taosAdapter is running normally, refer to [taosAdapter Reference](../../../tdengine-reference/components/taosadapter/). +- Seeq has been installed. Download the relevant software from [Seeq's official website](https://www.seeq.com/customer-download), such as `Seeq Server` and `Seeq Data Lab`, etc. `Seeq Data Lab` needs to be installed on a different server from `Seeq Server` and interconnected through configuration. For detailed installation and configuration instructions, refer to the [Seeq Knowledge Base](https://support.seeq.com/kb/latest/cloud/). +- Install the JDBC driver. Download the `TDengine JDBC connector` file `taos-jdbcdriver-3.2.5-dist.jar` or a higher version from `maven.org`. -- TDengine local instance has been installed. Please refer to the [official documentation](../../../get-started). If using TDengine Cloud, please go to https://cloud.taosdata.com apply for an account and log in to see how to access TDengine Cloud. +## Configure Data Source -## Configuring Seeq to Access TDengine +### Configuration of JDBC Connector -1. Check the data storage location +**Step 1**, Check the data storage location ```shell sudo seeq config get Folders/Data ``` -2. Download the TDengine Java connector package from maven.org, the latest version is [3.2.5](https://repo1.maven.org/maven2/com/taosdata/jdbc/taos-jdbcdriver/3.2.5/taos-jdbcdriver-3.2.5-dist.jar), and copy it to the plugins\lib in the data storage location. +**Step 2**, Download the TDengine Java connector package from `maven.org` and copy it to the `plugins\lib` directory in the data storage location. -3. Restart seeq server +**Step 3**, Restart seeq server ```shell sudo seeq restart ``` -4. Enter License +**Step 4**, Enter License Use a browser to visit ip:34216 and follow the instructions to enter the license. -## Using Seeq to Analyze TDengine Time-Series Data +## Load TDengine Time-Series Data -This section demonstrates how to use Seeq software in conjunction with TDengine for time-series data analysis. +This chapter demonstrates how to use the Seeq software to load TDengine time-series data. -### Scenario Introduction - -The example scenario is a power system where users collect electricity usage data from power station instruments daily and store it in the TDengine cluster. Now, users want to predict how power consumption will develop and purchase more equipment to support it. User power consumption varies with monthly orders, and considering seasonal changes, power consumption will differ. This city is located in the northern hemisphere, so more electricity is used in summer. We simulate data to reflect these assumptions. - -### Data Schema +**Step 1**, Create tables in TDengine. ```sql CREATE STABLE meters (ts TIMESTAMP, num INT, temperature FLOAT, goods INT) TAGS (device NCHAR(20)); @@ -58,7 +57,7 @@ CREATE TABLE goods (ts1 TIMESTAMP, ts2 TIMESTAMP, goods FLOAT); -### Data Construction Method +**Step 2**, Construct data in TDengine. ```shell python mockdata.py @@ -67,11 +66,7 @@ taos -s "insert into power.goods select _wstart, _wstart + 10d, avg(goods) from The source code is hosted on [GitHub Repository](https://github.com/sangshuduo/td-forecasting). -## Using Seeq for Data Analysis - -### Configuring Data Source - -Log in using a Seeq administrator role account and create a new data source. +**第 3 步**,Log in using a Seeq administrator role account and create a new data source. - Power @@ -251,6 +246,12 @@ Log in using a Seeq administrator role account and create a new data source. } ``` +## Data Analysis + +### Scenario Introduction + +The example scenario is a power system where users collect electricity usage data from power station instruments daily and store it in the TDengine cluster. Now, users want to predict how power consumption will develop and purchase more equipment to support it. User power consumption varies with monthly orders, and considering seasonal changes, power consumption will differ. This city is located in the northern hemisphere, so more electricity is used in summer. We simulate data to reflect these assumptions. + ### Using Seeq Workbench Log in to the Seeq service page and create a new Seeq Workbench. By selecting data sources from search results and choosing different tools as needed, you can display data or make predictions. For detailed usage methods, refer to the [official knowledge base](https://support.seeq.com/space/KB/146440193/Seeq+Workbench). @@ -330,77 +331,7 @@ Program output results: -## Configuring Seeq Data Source Connection to TDengine Cloud - -Configuring a Seeq data source connection to TDengine Cloud is essentially no different from connecting to a local TDengine installation. Simply log in to TDengine Cloud, select "Programming - Java" and copy the JDBC string with a token to fill in as the DatabaseJdbcUrl value for the Seeq Data Source. -Note that when using TDengine Cloud, the database name needs to be specified in SQL commands. - -### Configuration example using TDengine Cloud as a data source - -```json -{ - "QueryDefinitions": [ - { - "Name": "CloudVoltage", - "Type": "SIGNAL", - "Sql": "SELECT ts, voltage FROM test.meters", - "Enabled": true, - "TestMode": false, - "TestQueriesDuringSync": true, - "InProgressCapsulesEnabled": false, - "Variables": null, - "Properties": [ - { - "Name": "Name", - "Value": "Voltage", - "Sql": null, - "Uom": "string" - }, - { - "Name": "Interpolation Method", - "Value": "linear", - "Sql": null, - "Uom": "string" - }, - { - "Name": "Maximum Interpolation", - "Value": "2day", - "Sql": null, - "Uom": "string" - } - ], - "CapsuleProperties": null - } - ], - "Type": "GENERIC", - "Hostname": null, - "Port": 0, - "DatabaseName": null, - "Username": "root", - "Password": "taosdata", - "InitialSql": null, - "TimeZone": null, - "PrintRows": false, - "UseWindowsAuth": false, - "SqlFetchBatchSize": 100000, - "UseSSL": false, - "JdbcProperties": null, - "GenericDatabaseConfig": { - "DatabaseJdbcUrl": "jdbc:TAOS-RS://gw.cloud.tdengine.com?useSSL=true&token=41ac9d61d641b6b334e8b76f45f5a8XXXXXXXXXX", - "SqlDriverClassName": "com.taosdata.jdbc.rs.RestfulDriver", - "ResolutionInNanoseconds": 1000, - "ZonedColumnTypes": [] - } -} -``` - -### Example of Seeq Workbench Interface with TDengine Cloud as Data Source - -
- -
- -## Solution Summary +### Solution Summary By integrating Seeq and TDengine, users can fully leverage the efficient storage and querying capabilities of TDengine, while also benefiting from the powerful data visualization and analysis features provided by Seeq. diff --git a/docs/en/14-reference/01-components/01-taosd.md b/docs/en/14-reference/01-components/01-taosd.md index 7456593ddb..c320afcf9d 100644 --- a/docs/en/14-reference/01-components/01-taosd.md +++ b/docs/en/14-reference/01-components/01-taosd.md @@ -243,6 +243,8 @@ The effective value of charset is UTF-8. | concurrentCheckpoint | |Supported, effective immediately | Internal parameter, whether to check checkpoints concurrently | | maxStreamBackendCache | |Supported, effective immediately | Internal parameter, maximum cache used by stream computing | | streamSinkDataRate | |Supported, effective after restart| Internal parameter, used to control the write speed of stream computing results | +| streamNotifyMessageSize | After 3.3.6.0 | Not supported | Internal parameter, controls the message size for event notifications, default value is 8192 | +| streamNotifyFrameSize | After 3.3.6.0 | Not supported | Internal parameter, controls the underlying frame size when sending event notification messages, default value is 256 | ### Log Related diff --git a/docs/en/14-reference/03-taos-sql/06-select.md b/docs/en/14-reference/03-taos-sql/06-select.md index 21cb419bed..54c4248ff0 100644 --- a/docs/en/14-reference/03-taos-sql/06-select.md +++ b/docs/en/14-reference/03-taos-sql/06-select.md @@ -55,9 +55,9 @@ join_clause: window_clause: { SESSION(ts_col, tol_val) - | STATE_WINDOW(col) + | STATE_WINDOW(col) [TRUE_FOR(true_for_duration)] | INTERVAL(interval_val [, interval_offset]) [SLIDING (sliding_val)] [WATERMARK(watermark_val)] [FILL(fill_mod_and_val)] - | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition + | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition [TRUE_FOR(true_for_duration)] | COUNT_WINDOW(count_val[, sliding_val]) interp_clause: diff --git a/docs/en/14-reference/03-taos-sql/10-function.md b/docs/en/14-reference/03-taos-sql/10-function.md index 46ac6b00d5..7875501e7e 100644 --- a/docs/en/14-reference/03-taos-sql/10-function.md +++ b/docs/en/14-reference/03-taos-sql/10-function.md @@ -2125,6 +2125,28 @@ UNIQUE(expr) **Applicable to**: Tables and supertables. +### COLS​ + +```sql​ +COLS​(func(expr), output_expr1, [, output_expr2] ... )​ +``` + +**Function Description**: On the data row where the execution result of function func(expr) is located, execute the expression output_expr1, [, output_expr2], return its result, and the result of func (expr) is not output.​ + +**Return Data Type**: Returns multiple columns of data, and the data type of each column is the type of the result returned by the corresponding expression.​ + +**Applicable Data Types**: All type fields.​ + +**Applicable to**: Tables and Super Tables.​ + +**Usage Instructions**: +- Func function type: must be a single-line selection function (output result is a single-line selection function, for example, last is a single-line selection function, but top is a multi-line selection function).​ +- Mainly used to obtain the associated columns of multiple selection function results in a single SQL query. For example: select cols(max(c0), ts), cols(max(c1), ts) from ... can be used to get the different ts values of the maximum values of columns c0 and c1. +- The result of the parameter func is not returned. If you need to output the result of func, you can add additional output columns, such as: select first(ts), cols(first(ts), c1) from .. +- When there is only one column in the output, you can set an alias for the function. For example, you can do it like this: "select cols(first (ts), c1) as c11 from ...". +- Output one or more columns, and you can set an alias for each output column of the function. For example, you can do it like this: "select (first (ts), c1 as c11, c2 as c22) from ...". + + ## Time-Series Specific Functions Time-Series specific functions are tailor-made by TDengine to meet the query scenarios of time-series data. In general databases, implementing similar functionalities usually requires complex query syntax and is inefficient. TDengine has built these functionalities into functions, greatly reducing the user's cost of use. diff --git a/docs/en/14-reference/03-taos-sql/12-distinguished.md b/docs/en/14-reference/03-taos-sql/12-distinguished.md index e98b654be3..fb8c4ffe96 100644 --- a/docs/en/14-reference/03-taos-sql/12-distinguished.md +++ b/docs/en/14-reference/03-taos-sql/12-distinguished.md @@ -53,9 +53,9 @@ The syntax for the window clause is as follows: ```sql window_clause: { SESSION(ts_col, tol_val) - | STATE_WINDOW(col) + | STATE_WINDOW(col) [TRUE_FOR(true_for_duration)] | INTERVAL(interval_val [, interval_offset]) [SLIDING (sliding_val)] [FILL(fill_mod_and_val)] - | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition + | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition [TRUE_FOR(true_for_duration)] | COUNT_WINDOW(count_val[, sliding_val]) } ``` @@ -177,6 +177,12 @@ TDengine also supports using CASE expressions in state quantities, which can exp SELECT tbname, _wstart, CASE WHEN voltage >= 205 and voltage <= 235 THEN 1 ELSE 0 END status FROM meters PARTITION BY tbname STATE_WINDOW(CASE WHEN voltage >= 205 and voltage <= 235 THEN 1 ELSE 0 END); ``` +The state window supports using the TRUE_FOR parameter to set its minimum duration. If the window's duration is less than the specified value, it will be discarded automatically and no result will be returned. For example, setting the minimum duration to 3 seconds: + +``` +SELECT COUNT(*), FIRST(ts), status FROM temp_tb_1 STATE_WINDOW(status) TRUE_FOR (3s); +``` + ### Session Window The session window is determined based on the timestamp primary key values of the records. As shown in the diagram below, if the continuous interval of the timestamps is set to be less than or equal to 12 seconds, the following 6 records form 2 session windows, which are: [2019-04-28 14:22:10, 2019-04-28 14:22:30] and [2019-04-28 14:23:10, 2019-04-28 14:23:30]. This is because the interval between 2019-04-28 14:22:30 and 2019-04-28 14:23:10 is 40 seconds, exceeding the continuous interval (12 seconds). @@ -212,6 +218,12 @@ select _wstart, _wend, count(*) from t event_window start with c1 > 0 end with c +The event window supports using the TRUE_FOR parameter to set its minimum duration. If the window's duration is less than the specified value, it will be discarded automatically and no result will be returned. For example, setting the minimum duration to 3 seconds: + +``` +select _wstart, _wend, count(*) from t event_window start with c1 > 0 end with c2 < 10 true_for (3s); +``` + ### Count Window Count windows divide data into windows based on a fixed number of data rows. By default, data is sorted by timestamp, then divided into multiple windows based on the value of count_val, and aggregate calculations are performed. count_val represents the maximum number of data rows in each count window; if the total number of data rows is not divisible by count_val, the last window will have fewer rows than count_val. sliding_val is a constant that represents the number of rows the window slides, similar to the SLIDING in interval. diff --git a/docs/en/14-reference/09-error-code.md b/docs/en/14-reference/09-error-code.md index 233ac78a19..2d75dc953c 100644 --- a/docs/en/14-reference/09-error-code.md +++ b/docs/en/14-reference/09-error-code.md @@ -458,6 +458,12 @@ This document details the server error codes that may be encountered when using | 0x80002665 | The _TAGS pseudocolumn can only be used for subtable and supertable queries | Illegal tag column query | Check and correct the SQL statement | | 0x80002666 | Subquery does not output primary timestamp column | Check and correct the SQL statement | | | 0x80002667 | Invalid usage of expr: %s | Illegal expression | Check and correct the SQL statement | +| 0x80002687 | True_for duration cannot be negative | Use negative value as true_for duration | Check and correct the SQL statement | +| 0x80002688 | Cannot use 'year' or 'month' as true_for duration | Use year or month as true_for_duration | Check and correct the SQL statement | +| 0x80002689 | Invalid using cols function | Illegal using cols function | Check and correct the SQL statement | +| 0x8000268A | Cols function's first param must be a select function that output a single row | The first parameter of the cols function should be a selection function | Check and correct the SQL statement | +| 0x8000268B | Invalid using cols function with multiple output columns | Illegal using the cols function for multiple column output | Check and correct the SQL statement | +| 0x8000268C | Invalid using alias for cols function | Illegal cols function alias | Check and correct the SQL statement | | 0x800026FF | Parser internal error | Internal error in parser | Preserve the scene and logs, report issue on GitHub | | 0x80002700 | Planner internal error | Internal error in planner | Preserve the scene and logs, report issue on GitHub | | 0x80002701 | Expect ts equal | JOIN condition validation failed | Preserve the scene and logs, report issue on GitHub | diff --git a/docs/zh/07-develop/07-tmq.md b/docs/zh/07-develop/07-tmq.md index 7ae09aaeef..8f4477e88e 100644 --- a/docs/zh/07-develop/07-tmq.md +++ b/docs/zh/07-develop/07-tmq.md @@ -39,13 +39,13 @@ TDengine 消费者的概念跟 Kafka 类似,消费者通过订阅主题来接 | `auto.offset.reset` | enum | 消费组订阅的初始位置 |
`earliest`: default(version < 3.2.0.0);从头开始订阅;
`latest`: default(version >= 3.2.0.0);仅从最新数据开始订阅;
`none`: 没有提交的 offset 无法订阅 | | `enable.auto.commit` | boolean | 是否启用消费位点自动提交,true: 自动提交,客户端应用无需commit;false:客户端应用需要自行commit | 默认值为 true | | `auto.commit.interval.ms` | integer | 消费记录自动提交消费位点时间间隔,单位为毫秒 | 默认值为 5000 | -| `msg.with.table.name` | boolean | 是否允许从消息中解析表名, 不适用于列订阅(列订阅时可将 tbname 作为列写入 subquery 语句)(从3.2.0.0版本该参数废弃,恒为true) | 默认关闭 | +| `msg.with.table.name` | boolean | 是否允许从消息中解析表名, 不适用于列订阅(列订阅时可将 tbname 作为列写入 subquery 语句)(从 3.2.0.0 版本该参数废弃,恒为 true) | 默认关闭 | | `enable.replay` | boolean | 是否开启数据回放功能 | 默认关闭 | -| `session.timeout.ms` | integer | consumer 心跳丢失后超时时间,超时后会触发 rebalance 逻辑,成功后该 consumer 会被删除(从3.3.3.0版本开始支持) | 默认值为 12000,取值范围 [6000, 1800000] | -| `max.poll.interval.ms` | integer | consumer poll 拉取数据间隔的最长时间,超过该时间,会认为该 consumer 离线,触发rebalance 逻辑,成功后该 consumer 会被删除(从3.3.3.0版本开始支持) | 默认值为 300000,[1000,INT32_MAX] | -| `fetch.max.wait.ms` | integer | 服务端单次返回数据的最大耗时(从3.3.6.0版本开始支持) | 默认值为 1000,[1,INT32_MAX] | -| `min.poll.rows` | integer | 服务端单次返回数据的最小条数(从3.3.6.0版本开始支持) | 默认值为 4096,[1,INT32_MAX] | -| `msg.consume.rawdata` | integer | 消费数据时拉取数据类型为二进制类型,不可做解析操作,内部参数,只用于 taosx 数据迁移(从3.3.6.0版本开始支持) | 默认值为 0 表示不起效, 非 0 为 起效 | +| `session.timeout.ms` | integer | consumer 心跳丢失后超时时间,超时后会触发 rebalance 逻辑,成功后该 consumer 会被删除(从 3.3.3.0 版本开始支持) | 默认值为 12000,取值范围 [6000, 1800000] | +| `max.poll.interval.ms` | integer | consumer poll 拉取数据间隔的最长时间,超过该时间,会认为该 consumer 离线,触发 rebalance 逻辑,成功后该 consumer 会被删除(从 3.3.3.0 版本开始支持) | 默认值为 300000,[1000,INT32_MAX] | +| `fetch.max.wait.ms` | integer | 服务端单次返回数据的最大耗时(从 3.3.6.0 版本开始支持) | 默认值为 1000,[1,INT32_MAX] | +| `min.poll.rows` | integer | 服务端单次返回数据的最小条数(从 3.3.6.0 版本开始支持) | 默认值为 4096,[1,INT32_MAX] | +| `msg.consume.rawdata` | integer | 消费数据时拉取数据类型为二进制类型,不可做解析操作,内部参数,只用于 taosX 数据迁移(从 3.3.6.0 版本开始支持) | 默认值为 0 表示不起效, 非 0 为 起效 | 下面是各语言连接器创建参数: diff --git a/docs/zh/10-third-party/05-bi/03-powerbi.md b/docs/zh/10-third-party/05-bi/03-powerbi.md index b864fa2835..e55f6b6d88 100644 --- a/docs/zh/10-third-party/05-bi/03-powerbi.md +++ b/docs/zh/10-third-party/05-bi/03-powerbi.md @@ -1,78 +1,79 @@ --- -title: 与 PowerBI 的集成 +title: 与 PowerBI 集成 sidebar_label: PowerBI toc_max_heading_level: 4 --- -Power BI是由Microsoft提供的一种商业分析工具。通过配置使用ODBC连接器,Power BI可以快速访问TDengine的数据。用户可以将标签数据、原始时序数据或按时间聚合后的时序数据从TDengine导入到Power BI,制作报表或仪表盘,整个过程不需要任何代码编写过程。 +Power BI 是由 Microsoft 提供的一种商业分析工具。通过配置使用 ODBC 连接器,Power BI 可以快速访问 TDengine 的数据。用户可以将标签数据、原始时序数据或按时间聚合后的时序数据从 TDengine 导入到 Power BI,制作报表或仪表盘,整个过程不需要任何代码编写过程。 ## 前置条件 -安装完成Power BI Desktop软件并运行(如未安装,请从其官方地址下载最新的Windows操作系统 32/64 位版本)。 +准备以下环境: +- TDengine 3.3.4.0 以上版本集群已部署并正常运行(企业及社区版均可)。 +- taosAdapter 能够正常运行,详细参考 [taosAdapter 参考手册](../../../reference/components/taosadapter)。 +- 从 TDengine 官网下载最新的 Windows 操作系统 X64 客户端驱动程序并进行安装,详细参考 [安装 ODBC 驱动](../../../reference/connector/odbc/#安装)。 +- 安装完成 Power BI Desktop 软件并运行(如未安装,请从其官方地址下载最新的Windows操作系统 32/64 位版本)。 -## 安装 ODBC 驱动 +## 配置数据源 -从TDengine官网下载最新的Windows操作系统X64客户端驱动程序,并安装在运行Power BI的机器上。安装成功后可在“ODBC数据源(32位)”或者“ODBC数据源(64位)”管理工具中看到 TDengine 驱动程序。 +**第 1 步**,在 Windows 操作系统的开始菜单中搜索并打开【ODBC数据源(64位)】管理工具并进行配置。详细参考 [配置ODBC数据源](../../../reference/connector/odbc/#配置数据源)。 -## 配置ODBC数据源 +**第 2 步**,打开 Power BI 并登录后,点击【主页】->【获取数据】->【其他】->【ODBC】->【连接】,添加数据源。 -配置ODBC数据源的操作步骤如下。 +**第 3 步**,选择刚才创建的数据源名称,比如【MyTDengine】,如果需要输入 SQL,则可以点击【高级选项】选项卡,在展开的对话框的编辑框中输入 SQL 语句。点击【确定】按钮,即可连接到配置好的数据源。 -第1步,在Windows操作系统的开始菜单中搜索并打开“ODBC数据源(32位)”或者“ODBC数据源(64位)”管理工具。 -第2步,点击“用户DSN”选项卡→“添加”按钮,进入“创建新数据源”对话框。 -第3步,在“选择您想为其安装数据源的驱动程序”列表中选择“TDengine”,点击“完成”按钮,进入TDengine ODBC数据源配置页面。填写如下必要信息。 - - DSN:数据源名称,必填,比如“MyTDengine”。 - - 连接类型:勾选“WebSocket”复选框。 - - URL:ODBC 数据源 URL,必填,比如`http://127.0.0.1:6041`。 - - 数据库:表示需要连接的数据库,可选,比如“test”。 - - 用户名:输入用户名,如果不填,默认为“root”。 - - 密码:输入用户密码,如果不填,默认为“taosdata”。 +**第 4 步**,进入【导航器】后,可以浏览对应数据库的数据表/视图并加载数据。 -第4步,点击“测试连接”按钮,测试连接情况,如果成功连接,则会提示“成功连接到`http://127.0.0.1:6041`”。 -第5步,点击“确定”按钮,即可保存配置并退出。 +## 数据分析 -## 导入TDengine数据到Power BI +### 使用说明 -将TDengine数据导入Power BI的操作步骤如下: -第1步,打开Power BI并登录后,点击“主页”→“获取数据”→“其他”→“ODBC”→“连接”,添加数据源。 -第2步,选择刚才创建的数据源名称,比如“MyTDengine”,如果需要输入SQL,则可以点击“高级选项”选项卡,在展开的对话框的编辑框中输入SQL语句。点击“确定”按钮,即可连接到配置好的数据源。 -第3步,进入“导航器”后,可以浏览对应数据库的数据表/视图并加载数据。 +为了充分发挥 Power BI 在分析 TDengine中 数据方面的优势,用户需要先理解维度、度量、窗口切分查询、数据切分查询、时序和相关性等核心概念,之后通过自定义的 SQL 导入数据。 +- 维度:通常是分类(文本)数据,描述设备、测点、型号等类别信息。在 TDengine 的超级表中,使用标签列存储数据的维度信息,可以通过形如 “select distinct tbname, tag1, tag2 from supertable” 的SQL语法快速获得维度信息。 +- 度量:可以用于进行计算的定量(数值)字段,常见计算有求和、取平均值和最小值等。如果测点的采集周期为1s,那么一年就有 3000 多万条记录,把这些数据全部导入 Power BI 会严重影响其执行效率。在 TDengine 中,用户可以使用数据切分查询、窗口切分查询等语法,结合与窗口相关的伪列,把降采样后的数据导入Power BI 中,具体语法请参阅 TDengine 官方文档的特色查询功能部分。 +- 窗口切分查询:比如温度传感器每秒采集一次数据,但须查询每隔 10min 的温度平均值,在这种场景下可以使用窗口子句来获得需要的降采样查询结果,对应的 SQL 形如 `select tbname, _wstart date,avg(temperature) temp from table interval(10m)`,其中,`_wstart` 是伪列,表示时间窗口起始时间,10m 表示时间窗口的持续时间,`avg(temperature)` 表示时间窗口内的聚合值。 +- 数据切分查询:如果需要同时获取很多温度传感器的聚合数值,可对数据进行切分,然后在切分出的数据空间内进行一系列的计算,对应的 SQL 形如 `partition by part_list`。数据切分子句最常见的用法是在超级表查询中按标签将子表数据进行切分,将每个子表的数据独立出来,形成一条条独立的时间序列,方便针对各种时序场景的统计分析。 +- 时序:在绘制曲线或者按照时间聚合数据时,通常需要引入日期表。日期表可以从 Excel 表格中导入,也可以在 TDengine 中执行 SQL 获取,例如 `select _wstart date, count(*) cnt from test.meters where ts between A and B interval(1d) fill(0)`,其中 fill 字句表示数据缺失情况下的填充模式,伪列 _wstart 则为要获取的日期列。 +- 相关性:告诉数据之间如何关联,如度量和维度可以通过 tbname 列关联在一起,日期表和度量则可以通过 date 列关联,配合形成可视化报表。 -为了充分发挥Power BI在分析TDengine中数据方面的优势,用户需要先理解维度、度量、窗口切分查询、数据切分查询、时序和相关性等核心概念,之后通过自定义的SQL导入数据。 -- 维度:通常是分类(文本)数据,描述设备、测点、型号等类别信息。在TDengine的超级表中,使用标签列存储数据的维度信息,可以通过形如“select distinct tbname, tag1, tag2 from supertable”的SQL语法快速获得维度信息。 -- 度量:可以用于进行计算的定量(数值)字段,常见计算有求和、取平均值和最小值等。如果测点的采集周期为1s,那么一年就有3000多万条记录,把这些数据全部导入Power BI会严重影响其执行效率。在TDengine中,用户可以使用数据切分查询、窗口切分查询等语法,结合与窗口相关的伪列,把降采样后的数据导入Power BI中,具体语法请参阅TDengine官方文档的特色查询功能部分。 -- 窗口切分查询:比如温度传感器每秒采集一次数据,但须查询每隔10min的温度平均值,在这种场景下可以使用窗口子句来获得需要的降采样查询结果,对应的SQL形如`select tbname, _wstart date,avg(temperature) temp from table interval(10m)`,其中,_wstart是伪列,表示时间窗口起始时间,10m表示时间窗口的持续时间,avg(temperature)表示时间窗口内的聚合值。 -- 数据切分查询:如果需要同时获取很多温度传感器的聚合数值,可对数据进行切分,然后在切分出的数据空间内进行一系列的计算,对应的SQL形如 `partition by part_list`。数据切分子句最常见的用法是在超级表查询中按标签将子表数据进行切分,将每个子表的数据独立出来,形成一条条独立的时间序列,方便针对各种时序场景的统计分析。 -- 时序:在绘制曲线或者按照时间聚合数据时,通常需要引入日期表。日期表可以从Excel表格中导入,也可以在TDengine中执行SQL获取,例如 `select _wstart date, count(*) cnt from test.meters where ts between A and B interval(1d) fill(0)`,其中fill字句表示数据缺失情况下的填充模式,伪列_wstart则为要获取的日期列。 -- 相关性:告诉数据之间如何关联,如度量和维度可以通过tbname列关联在一起,日期表和度量则可以通过date列关联,配合形成可视化报表。 +### 智能电表样例 -## 智能电表样例 +TDengine 采用了一种独特的数据模型,以优化时序数据的存储和查询性能。该模型利用超级表作为模板,为每台设备创建一张独立的表。每张表在设计时考虑了高度的可扩展性,最多可包含 4096 个数据列和 128 个标签列。这种设计使得 TDengine 能够高效地处理大量时序数据,同时保持数据的灵活性和易用性。 -TDengine采用了一种独特的数据模型,以优化时序数据的存储和查询性能。该模型利用超级表作为模板,为每台设备创建一张独立的表。每张表在设计时考虑了高度的可扩展性,最多可包含4096个数据列和128个标签列。这种设计使得TDengine能够高效地处理大量时序数据,同时保持数据的灵活性和易用性。 +以智能电表为例,假设每块电表每秒产生一条记录,那么每天将产生 86400 条记录。对于 1000 块智能电表来说,每年产生的记录将占用大约 600GB 的存储空间。面对如此庞大的数据量,Power BI 等商业智能工具在数据分析和可视化方面发挥着重要作用。 -以智能电表为例,假设每块电表每秒产生一条记录,那么每天将产生86 400条记录。对于1000块智能电表来说,每年产生的记录将占用大约600GB的存储空间。面对如此庞大的数据量,Power BI等商业智能工具在数据分析和可视化方面发挥着重要作用。 +在 Power BI 中,用户可以将 TDengine 表中的标签列映射为维度列,以便对数据进行分组和筛选。同时,数据列的聚合结果可以导入为度量列,用于计算关键指标和生成报表。通过这种方式,Power BI 能够帮助决策者快速获取所需的信息,深入了解业务运营情况,从而制定更加明智的决策。 -在Power BI中,用户可以将TDengine表中的标签列映射为维度列,以便对数据进行分组和筛选。同时,数据列的聚合结果可以导入为度量列,用于计算关键指标和生成报表。通过这种方式,Power BI能够帮助决策者快速获取所需的信息,深入了解业务运营情况,从而制定更加明智的决策。 +根据如下步骤,便可以体验通过 Power BI 生成时序数据报表的功能。 -根据如下步骤,便可以体验通过Power BI生成时序数据报表的功能。 -第1步,使用TDengine的taosBenchMark快速生成1000块智能电表3天的数据,采集频率为1s。 - ```shell - taosBenchmark -t 1000 -n 259200 -S 1000 -y - ``` -第2步,导入维度数据。在Power BI中导入表的标签列,取名为tags,通过如下SQL获取超级表下所有智能电表的标签数据。 - ```sql - select distinct tbname device, groupId, location from test.meters - ``` -第3步,导入度量数据。在Power BI中,按照1小时的时间窗口,导入每块智能电表的电流均值、电压均值、相位均值,取名为data,SQL如下。 - ```sql - select tbname, _wstart ws, avg(current), avg(voltage), avg(phase) from test.meters PARTITION by tbname interval(1h) - ``` -第4步,导入日期数据。按照1天的时间窗口,获得时序数据的时间范围及数据计数,SQL如下。需要在Power Query编辑器中将date列的格式从“文本”转化为“日期”。 - ```sql - select _wstart date, count(*) from test.meters interval(1d) having count(*)>0 - ``` -第5步,建立维度和度量的关联关系。打开模型视图,建立表tags和data的关联关系,将tbname设置为关联数据列。 -第6步,建立日期和度量的关联关系。打开模型视图,建立数据集date和data的关联关系,关联的数据列为date和datatime。 -第7步,制作报告。在柱状图、饼图等控件中使用这些数据。 +**第 1 步**,使用 TDengine 的 taosBenchMark 快速生成1000块智能电表3天的数据,采集频率为 1s。 -由于TDengine处理时序数据的超强性能,使得用户在数据导入及每日定期刷新数据时,都可以得到非常好的体验。更多有关Power BI视觉效果的构建方法,请参照Power BI的官方文档。 \ No newline at end of file +```shell +taosBenchmark -t 1000 -n 259200 -S 1000 -y +``` + +**第 2 步**,导入维度数据。在 Power BI 中导入表的标签列,取名为 tags,通过如下 SQL 获取超级表下所有智能电表的标签数据。 + +```sql +select distinct tbname device, groupId, location from test.meters +``` + +**第 3 步**,导入度量数据。在 Power BI 中,按照 1 小时的时间窗口,导入每块智能电表的电流均值、电压均值、相位均值,取名为 data,SQL如下。 + +```sql +select tbname, _wstart ws, avg(current), avg(voltage), avg(phase) from test.meters PARTITION by tbname interval(1h) +``` + +**第 4 步**,导入日期数据。按照 1 天的时间窗口,获得时序数据的时间范围及数据计数,SQL 如下。需要在 Power Query 编辑器中将 date 列的格式从“文本”转化为“日期”。 + +```sql +select _wstart date, count(*) from test.meters interval(1d) having count(*)>0 +``` + +**第 5 步**,建立维度和度量的关联关系。打开模型视图,建立表 tags 和 data 的关联关系,将 tbname 设置为关联数据列。 + +**第 6 步**,建立日期和度量的关联关系。打开模型视图,建立数据集 date 和 data 的关联关系,关联的数据列为 date 和 datatime。 + +**第 7 步**,制作报告。在柱状图、饼图等控件中使用这些数据。 + +由于TDengine处理时序数据的超强性能,使得用户在数据导入及每日定期刷新数据时,都可以得到非常好的体验。更多有关 Power BI 视觉效果的构建方法,请参照 Power BI 的官方文档。 diff --git a/docs/zh/10-third-party/05-bi/05-yhbi.md b/docs/zh/10-third-party/05-bi/05-yhbi.md index 70dda71051..b0a25eac83 100644 --- a/docs/zh/10-third-party/05-bi/05-yhbi.md +++ b/docs/zh/10-third-party/05-bi/05-yhbi.md @@ -1,46 +1,55 @@ --- -title: 与永洪 BI 的集成 +title: 与永洪 BI 集成 sidebar_label: 永洪 BI toc_max_heading_level: 4 --- 永洪 BI 是一个专为各种规模企业打造的全业务链大数据分析解决方案,旨在帮助用户轻松发掘大数据价值,获取深入的洞察力。该平台以其灵活性和易用性而广受好评,无论企业规模大小,都能从中受益。 -为了实现与 TDengine 的高效集成,永洪 BI 提供了 JDBC 连接器。用户只须按照简单的步骤配置数据源,即可将 TDengine 作为数据源添加到永洪BI中。这一过程不仅快速便捷,还能确保数据的准确性和稳定性。 +为了实现与 TDengine 的高效集成,永洪 BI 提供了 JDBC 连接器。用户只须按照简单的步骤配置数据源,即可将 TDengine 作为数据源添加到永洪 BI 中。这一过程不仅快速便捷,还能确保数据的准确性和稳定性。 -一旦数据源配置完成,永洪BI便能直接从TDengine中读取数据,并利用其强大的数据处理和分析功能,为用户提供丰富的数据展示、分析和预测能力。这意味着用户无须编写复杂的代码或进行烦琐的数据转换工作,即可轻松获取所需的业务洞察。 +一旦数据源配置完成,永洪 BI 便能直接从 TDengine 中读取数据,并利用其强大的数据处理和分析功能,为用户提供丰富的数据展示、分析和预测能力。这意味着用户无须编写复杂的代码或进行烦琐的数据转换工作,即可轻松获取所需的业务洞察。 ## 前置条件 +准备以下环境: +- TDengine 3.3.2.0 以上版本集群已部署并正常运行(企业及社区版均可)。 +- taosAdapter 能够正常运行,详细参考 [taosAdapter 参考手册](../../../reference/components/taosadapter)。 - 确保永洪 BI 已经安装并运行(如果未安装,请到永洪科技官方下载页面下载)。 -- 安装JDBC驱动。从 maven.org 下载 TDengine JDBC 连接器文件 “taos-jdbcdriver-3.4.0-dist.jar”,并安装在永洪 BI 的机器上。 +- 安装JDBC驱动。从 maven.org 下载 TDengine JDBC 连接器文件 `taos-jdbcdriver-3.4.0-dist.jar` 及以上版本。 -## 配置JDBC数据源 +## 配置数据源 -配置JDBC数据源的步骤如下。 +配置JDBC数据源的步骤如下: -第1步,在打开的永洪BI中点击“添加数据源”按钮,选择SQL数据源中的“GENERIC”类型。 -第2步,点击“选择自定义驱动”按钮,在“驱动管理”对话框中点击“驱动列表”旁边的“+”,输入名称“MyTDengine”。然后点击“上传文件”按钮,上传刚刚下载的TDengine JDBC连接器文件“taos-jdbcdriver-3.2.7-dist.jar”,并选择“com.taosdata.jdbc. -rs.RestfulDriver”驱动,最后点击“确定”按钮,完成驱动添加步骤。 -第3步,复制下面的内容到“URL”字段。 - ```text - jdbc:TAOS-RS://127.0.0.1:6041?user=root&password=taosdata - ``` -第4步,在“认证方式”中点击“无身份认证”单选按钮。 -第5步,在数据源的高级设置中修改“Quote 符号”的值为反引号(`)。 -第6步,点击“测试连接”按钮,弹出“测试成功”对话框。点击“保存”按钮,输入“MyTDengine”来保存TDengine数据源。 +**第 1 步**,在打开的永洪 BI 中点击【添加数据源】按钮,选择 SQL 数据源中的 “GENERIC” 类型。 -## 创建TDengine数据集 +**第 2 步**,点击【选择自定义驱动】按钮,在【驱动管理】对话框中点击【驱动列表】旁边的 “+”,输入名称 “MyTDengine”。然后点击【上传文件】按钮,上传刚刚下载的 TDengine JDBC 连接器文件 `taos-jdbcdriver-3.2.7-dist.jar`,并选择 `com.taosdata.jdbc.rs.RestfulDriver` 驱动,最后点击“确定”按钮,完成驱动添加步骤。 -创建TDengine数据集的步骤如下。 +**第 3 步**,复制下面的内容到【URL】字段。 -第1步,在永洪BI中点击“添加数据源”按钮,展开刚刚创建的数据源,并浏览TDengine中的超级表。 -第2步,可以将超级表的数据全部加载到永洪BI中,也可以通过自定义SQL导入部分数据。 -第3步,当勾选“数据库内计算”复选框时,永洪BI将不再缓存TDengine的时序数据,并在处理查询时将SQL请求发送给TDengine直接处理。 +```text +jdbc:TAOS-RS://127.0.0.1:6041?user=root&password=taosdata +``` -当导入数据后,永洪BI会自动将数值类型设置为“度量”列,将文本类型设置为“维度”列。而在TDengine的超级表中,由于将普通列作为数据的度量,将标签列作为数据的维度,因此用户可能需要在创建数据集时更改部分列的属性。TDengine在支持标准SQL的基础之上还提供了一系列满足时序业务场景需求的特色查询语法,例如数据切分查询、窗口切分查询等,具体操作步骤请参阅TDengine的官方文档。通过使用这些特色查询,当永洪BI将SQL查询发送到TDengine时,可以大大提高数据访问速度,减少网络传输带宽。 +**第 4 步**,在【认证方式】中点击【无身份认证】单选按钮。 + +**第 5 步**,在数据源的高级设置中修改 “Quote 符号” 的值为反引号(`)。 + +**第 6 步**,点击【测试连接】按钮,弹出【测试成功】对话框。点击【保存】按钮,输入 “MyTDengine” 来保存 TDengine 数据源。 + +**第 7 步**,在永洪 BI 中点击【添加数据源】按钮,展开刚刚创建的数据源,并浏览 TDengine 中的超级表。 + +**第 8 步**,可以将超级表的数据全部加载到永洪 BI 中,也可以通过自定义 SQL 导入部分数据。 + +**第 9 步**,当勾选【数据库内计算】复选框时,永洪 BI 将不再缓存 TDengine 的时序数据,并在处理查询时将 SQL 请求发送给 TDengine 直接处理。 + +## 数据分析 + +当导入数据后,永洪 BI 会自动将数值类型设置为 “度量” 列,将文本类型设置为 “维度” 列。而在 TDengine 的超级表中,由于将普通列作为数据的度量,将标签列作为数据的维度,因此用户可能需要在创建数据集时更改部分列的属性。TDengine 在支持标准 SQL 的基础之上还提供了一系列满足时序业务场景需求的特色查询语法,例如数据切分查询、窗口切分查询等,具体操作步骤请参阅 TDengine 的官方文档。通过使用这些特色查询,当永洪 BI 将 SQL 查询发送到 TDengine 时,可以大大提高数据访问速度,减少网络传输带宽。 + +在永洪 BI 中,你可以创建 “参数” 并在 SQL 中使用,通过手动、定时的方式动态执行这些 SQL,即可实现可视化报告的刷新效果。如下 SQL 可以从 TDengine 实时读取数据。 -在永洪BI中,你可以创建“参数”并在SQL中使用,通过手动、定时的方式动态执行这些SQL,即可实现可视化报告的刷新效果。如下SQL可以从TDengine实时读取数据。 ```sql select _wstart ws, count(*) cnt from supertable where tbname=?{metric} and ts = ?{from} and ts < ?{to} interval(?{interval}) ``` @@ -49,17 +58,15 @@ select _wstart ws, count(*) cnt from supertable where tbname=?{metric} and ts = 1. `_wstart`:表示时间窗口起始时间。 2. `count(*)`:表示时间窗口内的聚合值。 3. `?{interval}`:表示在 SQL 语句中引入名称为 `interval` 的参数,当 BI 工具查询数据时,会给 `interval` 参数赋值,如果取值为 1m,则表示按照 1 分钟的时间窗口降采样数据。 -4. `?{metric}`:该参数用来指定查询的数据表名称,当在 BI 工具中把某个“下拉参数组件”的 ID 也设置为 metric 时,该“下拉参数组件”的被选择项将会和该参数绑定在一起,实现动态选择的效果。 -5. `?{from}` 和 `?{to}`:这两个参数用来表示查询数据集的时间范围,可以与“文本参数组件”绑定。 -您可以在 BI 工具的“编辑参数”对话框中修改“参数”的数据类型、数据范围、默认取值,并在“可视化报告”中动态设置这些参数的值。 +4. `?{metric}`:该参数用来指定查询的数据表名称,当在 BI 工具中把某个 “下拉参数组件” 的 ID 也设置为 metric 时,该 “下拉参数组件” 的被选择项将会和该参数绑定在一起,实现动态选择的效果。 +5. `?{from}` 和 `?{to}`:这两个参数用来表示查询数据集的时间范围,可以与 “文本参数组件” 绑定。 +您可以在 BI 工具的【编辑参数】对话框中修改 “参数” 的数据类型、数据范围、默认取值,并在 “可视化报告” 中动态设置这些参数的值。 -## 21.4.5 制作可视化报告 +制作可视化报告的步骤如下: -制作可视化报告的步骤如下。 - -1. 在永洪 BI 工具中点击“制作报告”,创建画布。 -2. 拖动可视化组件到画布中,例如“表格组件”。 -3. 在“数据集”侧边栏中选择待绑定的数据集,将数据列中的“维度”和“度量”按需绑定到“表格组件”。 -4. 点击“保存”后,即可查看报告。 +1. 在永洪 BI 工具中点击【制作报告】,创建画布。 +2. 拖动可视化组件到画布中,例如 “表格组件”。 +3. 在【数据集】侧边栏中选择待绑定的数据集,将数据列中的 “维度” 和 “度量” 按需绑定到 “表格组件”。 +4. 点击【保存】后,即可查看报告。 5. 更多有关永洪 BI 工具的信息,请查询永洪科技官方帮助文档。 diff --git a/docs/zh/10-third-party/05-bi/09-seeq.md b/docs/zh/10-third-party/05-bi/09-seeq.md index e01deb7e84..24d3457f87 100644 --- a/docs/zh/10-third-party/05-bi/09-seeq.md +++ b/docs/zh/10-third-party/05-bi/09-seeq.md @@ -1,48 +1,48 @@ --- sidebar_label: Seeq -title: 与 Seeq 的集成 +title: 与 Seeq 集成 toc_max_heading_level: 4 --- Seeq 是制造业和工业互联网(IIOT)高级分析软件。Seeq 支持在工艺制造组织中使用机器学习创新的新功能。这些功能使组织能够将自己或第三方机器学习算法部署到前线流程工程师和主题专家使用的高级分析应用程序,从而使单个数据科学家的努力扩展到许多前线员工。 -通过 TDengine Java connector, Seeq 可以轻松支持查询 TDengine 提供的时序数据,并提供数据展现、分析、预测等功能。 +通过 `TDengine Java connector`, Seeq 可以轻松支持查询 TDengine 提供的时序数据,并提供数据展现、分析、预测等功能。 ## 前置条件 -- Seeq 已经安装。从 [Seeq 官网](https://www.seeq.com/customer-download)下载相关软件,例如 Seeq Server 和 Seeq Data Lab 等。Seeq Data Lab 需要安装在和 Seeq Server 不同的服务器上,并通过配置和 Seeq Server 互联。详细安装配置指令参见[Seeq 知识库]( https://support.seeq.com/kb/latest/cloud/)。 +准备以下环境: +- TDengine 3.1.0.3 以上版本集群已部署并正常运行(企业及社区版均可)。 +- taosAdapter 能够正常运行,详细参考 [taosAdapter 参考手册](../../../reference/components/taosadapter)。 +- Seeq 已经安装。从 [Seeq 官网](https://www.seeq.com/customer-download)下载相关软件,例如 `Seeq Server` 和 `Seeq Data Lab` 等。`Seeq Data Lab` 需要安装在和 `Seeq Server` 不同的服务器上,并通过配置和 `Seeq Server` 互联。详细安装配置指令参见 [Seeq 知识库]( https://support.seeq.com/kb/latest/cloud/)。 +- 安装 JDBC 驱动。从 `maven.org` 下载 `TDengine JDBC` 连接器文件 `taos-jdbcdriver-3.2.5-dist.jar` 及以上版本。 -- TDengine 本地实例已安装。 请参考[官网文档](../../../get-started)。 若使用 TDengine Cloud,请在 https://cloud.taosdata.com 申请帐号并登录查看如何访问 TDengine Cloud。 +## 配置数据源 -## 配置 Seeq 访问 TDengine +### 配置 JDBC 连接器 -1. 查看 data 存储位置 +**第 1 步**,查看 data 存储位置 ``` sudo seeq config get Folders/Data ``` -2. 从 maven.org 下载 TDengine Java connector 包,目前最新版本为[3.2.5](https://repo1.maven.org/maven2/com/taosdata/jdbc/taos-jdbcdriver/3.2.5/taos-jdbcdriver-3.2.5-dist.jar),并拷贝至 data 存储位置的 plugins\lib 中。 +**第 2 步**,将 `maven.org` 下载 `TDengine Java connector` 包并拷贝至 data 存储位置的 `plugins\lib` 中。 -3. 重新启动 seeq server +**第 3 步**,重新启动 seeq server ``` sudo seeq restart ``` -4. 输入 License +**第 4 步**,输入 License 使用浏览器访问 ip:34216 并按照说明输入 license。 -## 使用 Seeq 分析 TDengine 时序数据 +### 加载 TDengine 时序数据 -本章节演示如何使用 Seeq 软件配合 TDengine 进行时序数据分析。 +本章节演示如何使用 Seeq 软件加载 TDengine 时序数据。 -### 场景介绍 - -示例场景为一个电力系统,用户每天从电站仪表收集用电量数据,并将其存储在 TDengine 集群中。现在用户想要预测电力消耗将会如何发展,并购买更多设备来支持它。用户电力消耗随着每月订单变化而不同,另外考虑到季节变化,电力消耗量会有所不同。这个城市位于北半球,所以在夏天会使用更多的电力。我们模拟数据来反映这些假定。 - -### 数据 Schema +**第 1 步**,在 TDengine 中创建表。 ``` CREATE STABLE meters (ts TIMESTAMP, num INT, temperature FLOAT, goods INT) TAGS (device NCHAR(20)); @@ -51,20 +51,16 @@ CREATE TABLE goods (ts1 TIMESTAMP, ts2 TIMESTAMP, goods FLOAT); ![Seeq demo schema](./seeq/seeq-demo-schema.webp) -### 构造数据方法 +**第 2 步**,在 TDengine 中构造数据。 ``` python mockdata.py taos -s "insert into power.goods select _wstart, _wstart + 10d, avg(goods) from power.meters interval(10d);" ``` -源代码托管在[GitHub 仓库](https://github.com/sangshuduo/td-forecasting)。 +源代码托管在 [GitHub 仓库](https://github.com/sangshuduo/td-forecasting)。 -## 使用 Seeq 进行数据分析 - -### 配置数据源(Data Source) - -使用 Seeq 管理员角色的帐号登录,并新建数据源。 +**第 3 步**,使用 Seeq 管理员角色的帐号登录,并新建数据源。 - Power @@ -244,9 +240,15 @@ taos -s "insert into power.goods select _wstart, _wstart + 10d, avg(goods) from } ``` +## 数据分析 + +### 场景介绍 + +示例场景为一个电力系统,用户每天从电站仪表收集用电量数据,并将其存储在 TDengine 集群中。现在用户想要预测电力消耗将会如何发展,并购买更多设备来支持它。用户电力消耗随着每月订单变化而不同,另外考虑到季节变化,电力消耗量会有所不同。这个城市位于北半球,所以在夏天会使用更多的电力。我们模拟数据来反映这些假定。 + ### 使用 Seeq Workbench -登录 Seeq 服务页面并新建 Seeq Workbench,通过选择数据源搜索结果和根据需要选择不同的工具,可以进行数据展现或预测,详细使用方法参见[官方知识库](https://support.seeq.com/space/KB/146440193/Seeq+Workbench)。 +登录 Seeq 服务页面并新建 Seeq Workbench,通过选择数据源搜索结果和根据需要选择不同的工具,可以进行数据展现或预测,详细使用方法参见 [官方知识库](https://support.seeq.com/space/KB/146440193/Seeq+Workbench)。 ![Seeq Workbench](./seeq/seeq-demo-workbench.webp) @@ -319,78 +321,10 @@ plt.show() ![Seeq forecast result](./seeq/seeq-forecast-result.webp) -## 配置 Seeq 数据源连接 TDengine Cloud +### 方案总结 -配置 Seeq 数据源连接 TDengine Cloud 和连接 TDengine 本地安装实例没有本质的不同,只要登录 TDengine Cloud 后选择“编程 - Java”并拷贝带 token 字符串的 JDBC 填写为 Seeq Data Source 的 DatabaseJdbcUrl 值。 -注意使用 TDengine Cloud 时 SQL 命令中需要指定数据库名称。 +通过集成 Seeq 和 TDengine,可以充分利用 TDengine 高效的存储和查询性能,同时也可以受益于 Seeq 提供给用户的强大数据可视化和分析功能。 -### 用 TDengine Cloud 作为数据源的配置内容示例: +这种集成使用户能够充分利用 TDengine 的高性能时序数据存储和检索,确保高效处理大量数据。同时,Seeq 提供高级分析功能,如数据可视化、异常检测、相关性分析和预测建模,使用户能够获得有价值的洞察并基于数据进行决策。 -``` -{ - "QueryDefinitions": [ - { - "Name": "CloudVoltage", - "Type": "SIGNAL", - "Sql": "SELECT ts, voltage FROM test.meters", - "Enabled": true, - "TestMode": false, - "TestQueriesDuringSync": true, - "InProgressCapsulesEnabled": false, - "Variables": null, - "Properties": [ - { - "Name": "Name", - "Value": "Voltage", - "Sql": null, - "Uom": "string" - }, - { - "Name": "Interpolation Method", - "Value": "linear", - "Sql": null, - "Uom": "string" - }, - { - "Name": "Maximum Interpolation", - "Value": "2day", - "Sql": null, - "Uom": "string" - } - ], - "CapsuleProperties": null - } - ], - "Type": "GENERIC", - "Hostname": null, - "Port": 0, - "DatabaseName": null, - "Username": "root", - "Password": "taosdata", - "InitialSql": null, - "TimeZone": null, - "PrintRows": false, - "UseWindowsAuth": false, - "SqlFetchBatchSize": 100000, - "UseSSL": false, - "JdbcProperties": null, - "GenericDatabaseConfig": { - "DatabaseJdbcUrl": "jdbc:TAOS-RS://gw.cloud.taosdata.com?useSSL=true&token=41ac9d61d641b6b334e8b76f45f5a8XXXXXXXXXX", - "SqlDriverClassName": "com.taosdata.jdbc.rs.RestfulDriver", - "ResolutionInNanoseconds": 1000, - "ZonedColumnTypes": [] - } -} -``` - -### TDengine Cloud 作为数据源的 Seeq Workbench 界面示例 - -![Seeq workbench with TDengine cloud](./seeq/seeq-workbench-with-tdengine-cloud.webp) - -## 方案总结 - -通过集成Seeq和TDengine,可以充分利用TDengine高效的存储和查询性能,同时也可以受益于Seeq提供给用户的强大数据可视化和分析功能。 - -这种集成使用户能够充分利用TDengine的高性能时序数据存储和检索,确保高效处理大量数据。同时,Seeq提供高级分析功能,如数据可视化、异常检测、相关性分析和预测建模,使用户能够获得有价值的洞察并基于数据进行决策。 - -综合来看,Seeq和TDengine共同为制造业、工业物联网和电力系统等各行各业的时序数据分析提供了综合解决方案。高效数据存储和先进的分析相结合,赋予用户充分发挥时序数据潜力的能力,推动运营改进,并支持预测和规划分析应用。 +综合来看,Seeq 和 TDengine 共同为制造业、工业物联网和电力系统等各行各业的时序数据分析提供了综合解决方案。高效数据存储和先进的分析相结合,赋予用户充分发挥时序数据潜力的能力,推动运营改进,并支持预测和规划分析应用。 diff --git a/docs/zh/14-reference/01-components/01-taosd.md b/docs/zh/14-reference/01-components/01-taosd.md index 06897a68d6..88c53c6a0c 100644 --- a/docs/zh/14-reference/01-components/01-taosd.md +++ b/docs/zh/14-reference/01-components/01-taosd.md @@ -1067,6 +1067,26 @@ charset 的有效值是 UTF-8。 - 动态修改:支持通过 SQL 修改,立即生效 - 支持版本:从 v3.1.0.0 版本开始引入 +#### streamNotifyMessageSize +- 说明:用于控制事件通知的消息大小 `内部参数` +- 类型:整数 +- 单位:KB +- 默认值:8192 +- 最小值:8 +- 最大值:1048576 +- 动态修改:不支持 +- 支持版本:从 v3.3.6.0 版本开始引入 + +#### streamNotifyFrameSize +- 说明:用于控制事件通知消息发送时底层的帧大小 `内部参数` +- 类型:整数 +- 单位:KB +- 默认值:256 +- 最小值:8 +- 最大值:1048576 +- 动态修改:不支持 +- 支持版本:从 v3.3.6.0 版本开始引入 + ### 日志相关 #### logDir @@ -1441,7 +1461,7 @@ charset 的有效值是 UTF-8。 - 取值范围:float/double/none - 默认值:none,表示关闭无损压缩 - 动态修改:不支持 -- 支持版本:从v3.1.0.0 版本引入,v3.3.0.0 以后废弃 +- 支持版本:从 v3.1.0.0 版本引入,v3.3.0.0 以后废弃 #### ifAdtFse - 说明:在启用 TSZ 有损压缩时,使用 FSE 算法替换 HUFFMAN 算法,FSE 算法压缩速度更快,但解压稍慢,追求压缩速度可选用此算法 @@ -1450,22 +1470,22 @@ charset 的有效值是 UTF-8。 - 最小值:0 - 最大值:1 - 动态修改:支持通过 SQL 修改,重启生效 -- 支持版本:从v3.1.0.0 版本引入,v3.3.0.0 以后废弃 +- 支持版本:从 v3.1.0.0 版本引入,v3.3.0.0 以后废弃 #### maxRange - 说明:用于有损压缩设置 `内部参数` - 动态修改:支持通过 SQL 修改,重启生效 -- 支持版本:从v3.1.0.0 版本引入,v3.3.0.0 以后废弃 +- 支持版本:从 v3.1.0.0 版本引入,v3.3.0.0 以后废弃 #### curRange - 说明:用于有损压缩设置 `内部参数` - 动态修改:支持通过 SQL 修改,重启生效 -- 支持版本:从v3.1.0.0 版本引入,v3.3.0.0 以后废弃 +- 支持版本:从 v3.1.0.0 版本引入,v3.3.0.0 以后废弃 #### compressor - 说明:用于有损压缩设置 `内部参数` - 动态修改:支持通过 SQL 修改,重启生效 -- 支持版本:从v3.1.0.0 版本引入,v3.3.0.0 以后废弃 +- 支持版本:从 v3.1.0.0 版本引入,v3.3.0.0 以后废弃 **补充说明** 1. 在 3.3.5.0 之后,所有配置参数都将被持久化到本地存储,重启数据库服务后,将默认使用持久化的配置参数列表;如果您希望继续使用 config 文件中配置的参数,需设置 forceReadConfig 为 1。 diff --git a/docs/zh/14-reference/02-tools/10-taosbenchmark.md b/docs/zh/14-reference/02-tools/10-taosbenchmark.md index 67a823ad5e..577127d93f 100644 --- a/docs/zh/14-reference/02-tools/10-taosbenchmark.md +++ b/docs/zh/14-reference/02-tools/10-taosbenchmark.md @@ -115,9 +115,9 @@ taosBenchmark -f   - **continue_if_fail** : 允许用户定义失败后行为。 - “continue_if_fail”:  “no”, 失败 taosBenchmark 自动退出,默认行为。 - “continue_if_fail”: “yes”, 失败 taosBenchmark 警告用户,并继续写入。 - “continue_if_fail”: “smart”, 如果子表不存在失败,taosBenchmark 会建立子表并继续写入。 + "continue_if_fail":  "no", 失败 taosBenchmark 自动退出,默认行为。 + "continue_if_fail": "yes", 失败 taosBenchmark 警告用户,并继续写入。 + "continue_if_fail": "smart", 如果子表不存在失败,taosBenchmark 会建立子表并继续写入。 #### 数据库相关 @@ -125,7 +125,7 @@ taosBenchmark -f - **name** : 数据库名。 -- **drop** : 数据库已存在时是否删除,可选项为 "yes" 或 "no", 默认为 “yes” 。 +- **drop** : 数据库已存在时是否删除,可选项为 "yes" 或 "no", 默认为 "yes" 。 #### 超级表相关 @@ -208,7 +208,7 @@ taosBenchmark -f - **scalingFactor** : 浮点数精度增强因子,仅当数据类型是 float/double 时生效,有效值范围为 1 至 1000000 的正整数。用于增强生成浮点数的精度,特别是在 min 或 max 值较小的情况下。此属性按 10 的幂次增强小数点后的精度:scalingFactor 为 10 表示增强 1 位小数精度,100 表示增强 2 位,依此类推。 -- **fun** : 此列数据以函数填充,目前只支持 sin 和 cos 两函数,输入参数为时间戳换算成角度值,换算公式: 角度 x = 输入的时间列ts值 % 360。同时支持系数调节,随机波动因子调节,以固定格式的表达式展现,如 fun=“10\*sin(x)+100\*random(5)” , x 表示角度,取值 0 ~ 360度,增长步长与时间列步长一致。10 表示乘的系数,100 表示加或减的系数,5 表示波动幅度在 5% 的随机范围内。目前支持的数据类型为 int, bigint, float, double 四种数据类型。注意:表达式为固定模式,不可前后颠倒。 +- **fun** : 此列数据以函数填充,目前只支持 sin 和 cos 两函数,输入参数为时间戳换算成角度值,换算公式: 角度 x = 输入的时间列ts值 % 360。同时支持系数调节,随机波动因子调节,以固定格式的表达式展现,如 fun="10\*sin(x)+100\*random(5)" , x 表示角度,取值 0 ~ 360度,增长步长与时间列步长一致。10 表示乘的系数,100 表示加或减的系数,5 表示波动幅度在 5% 的随机范围内。目前支持的数据类型为 int, bigint, float, double 四种数据类型。注意:表达式为固定模式,不可前后颠倒。 - **values** : nchar/binary 列/标签的值域,将从值中随机选择。 @@ -220,15 +220,15 @@ taosBenchmark -f - **level**: 字符串类型,指定此列两级压缩中的第二级加密算法的压缩率高低,详细参见创建超级表。 -- **gen**: 字符串类型,指定此列生成数据的方式,不指定为随机,若指定为 “order”, 会按自然数顺序增长。 +- **gen**: 字符串类型,指定此列生成数据的方式,不指定为随机,若指定为 "order", 会按自然数顺序增长。 -- **fillNull**: 字符串类型,指定此列是否随机插入 NULL 值,可指定为 “true” 或 "false", 只有当 generate_row_rule 为 2 时有效。 +- **fillNull**: 字符串类型,指定此列是否随机插入 NULL 值,可指定为 "true" 或 "false", 只有当 generate_row_rule 为 2 时有效。 #### 写入行为相关 - **thread_count** : 插入数据的线程数量,默认为 8。 -- **thread_bind_vgroup** : 写入时 vgroup 是否和写入线程绑定,绑定后可提升写入速度, 取值为 "yes" 或 "no",默认值为 “no”, 设置为 “no” 后与原来行为一致。 当设为 “yes” 时,如果 thread_count 大于写入数据库 vgroups 数量, thread_count 自动调整为 vgroups 数量;如果 thread_count 小于 vgroups 数量,写入线程数量不做调整,一个线程写完一个 vgroup 数据后再写下一个,同时保持一个 vgroup 同时只能由一个线程写入的规则。 +- **thread_bind_vgroup** : 写入时 vgroup 是否和写入线程绑定,绑定后可提升写入速度, 取值为 "yes" 或 "no",默认值为 "no", 设置为 "no" 后与原来行为一致。 当设为 "yes" 时,如果 thread_count 大于写入数据库 vgroups 数量, thread_count 自动调整为 vgroups 数量;如果 thread_count 小于 vgroups 数量,写入线程数量不做调整,一个线程写完一个 vgroup 数据后再写下一个,同时保持一个 vgroup 同时只能由一个线程写入的规则。 - **create_table_thread_count** : 建表的线程数量,默认为 8。 @@ -248,7 +248,7 @@ taosBenchmark -f - **prepare_rand** : 生成的随机数据中唯一值的数量。若为 1 则表示所有数据都相同。默认值为 10000 。 -- **pre_load_tb_meta** :是否提前加载子表的 meta 数据,取值为 “yes” or "no"。当子表数量非常多时,打开此选项可提高写入速度。 +- **pre_load_tb_meta** :是否提前加载子表的 meta 数据,取值为 "yes" or "no"。当子表数量非常多时,打开此选项可提高写入速度。 ### 查询配置参数 @@ -265,7 +265,7 @@ interval 控制休眠时间,避免持续查询慢查询消耗 CPU ,单位为 查询指定表(可以指定超级表、子表或普通表)的配置参数在 `specified_table_query` 中设置。 - **mixed_query** : 查询模式 - “yes” :`混合查询` + "yes" :`混合查询` "no"(默认值) :`普通查询` `普通查询`:`sqls` 中每个 sql 启动 `threads` 个线程查询此 sql, 执行完 `query_times` 次查询后退出,执行此 sql 的所有线程都完成后进入下一个 sql `查询总次数` = `sqls` 个数 * `query_times` * `threads` diff --git a/docs/zh/14-reference/03-taos-sql/06-select.md b/docs/zh/14-reference/03-taos-sql/06-select.md index 3375fe6134..15b9196f1a 100644 --- a/docs/zh/14-reference/03-taos-sql/06-select.md +++ b/docs/zh/14-reference/03-taos-sql/06-select.md @@ -56,9 +56,9 @@ join_clause: window_clause: { SESSION(ts_col, tol_val) - | STATE_WINDOW(col) + | STATE_WINDOW(col) [TRUE_FOR(true_for_duration)] | INTERVAL(interval_val [, interval_offset]) [SLIDING (sliding_val)] [WATERMARK(watermark_val)] [FILL(fill_mod_and_val)] - | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition + | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition [TRUE_FOR(true_for_duration)] | COUNT_WINDOW(count_val[, sliding_val]) interp_clause: diff --git a/docs/zh/14-reference/03-taos-sql/10-function.md b/docs/zh/14-reference/03-taos-sql/10-function.md index fa76693c6f..0c713c486b 100644 --- a/docs/zh/14-reference/03-taos-sql/10-function.md +++ b/docs/zh/14-reference/03-taos-sql/10-function.md @@ -2049,6 +2049,27 @@ UNIQUE(expr) **适用于**:表和超级表。 +### COLS + +```sql +COLS(func(expr), output_expr1, [, output_expr2] ... ) +``` + +**功能说明**:在选择函数 func(expr) 执行结果所在数据行上,执行表达式 output_expr1, [, output_expr2],返回其结果,func(expr)结果不输出。 + +**返回数据类型**:返回多列数据,每列数据类型为对应表达式返回结果的类型。 + +**适用数据类型**:全部类型字段。 + +**适用于**:表和超级表。 + +**使用说明:** +- func 函数类型:必须是单行选择函数(输出结果为一行的选择函数,例如 last 是单行选择函数, 但 top 是多行选择函数)。 +- 主要用于一个 sql 中获取多个选择函数结果关联列的场景,例如: select cols(max(c0), ts), cols(max(c1), ts) from ...可用于获取 c0, c1 列最大值的不同 ts 值。 +- 注意, 函数 func 的结果并没有返回,如需输出 func 结果,可额外增加输出列,如: select fist(ts), cols(first(ts), c1) from ... +- 输出只有一列时,可以对 cols 函数设置别名。例如: "select cols(first(ts), c1) as c11 from ..." +- 输出一列或者多列时,可以对 cols 函数的每个输出列设置命名。例如: "select cols(first(ts), c1 as c11, c2 as c22)" + ## 时序数据特有函数 diff --git a/docs/zh/14-reference/03-taos-sql/12-distinguished.md b/docs/zh/14-reference/03-taos-sql/12-distinguished.md index bd6705b33b..8ac941dd77 100644 --- a/docs/zh/14-reference/03-taos-sql/12-distinguished.md +++ b/docs/zh/14-reference/03-taos-sql/12-distinguished.md @@ -46,9 +46,9 @@ TDengine 支持按时间窗口切分方式进行聚合结果查询,比如温 ```sql window_clause: { SESSION(ts_col, tol_val) - | STATE_WINDOW(col) + | STATE_WINDOW(col) [TRUE_FOR(true_for_duration)] | INTERVAL(interval_val [, interval_offset]) [SLIDING (sliding_val)] [FILL(fill_mod_and_val)] - | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition + | EVENT_WINDOW START WITH start_trigger_condition END WITH end_trigger_condition [TRUE_FOR(true_for_duration)] | COUNT_WINDOW(count_val[, sliding_val]) } ``` @@ -165,6 +165,12 @@ TDengine 还支持将 CASE 表达式用在状态量,可以表达某个状态 SELECT tbname, _wstart, CASE WHEN voltage >= 205 and voltage <= 235 THEN 1 ELSE 0 END status FROM meters PARTITION BY tbname STATE_WINDOW(CASE WHEN voltage >= 205 and voltage <= 235 THEN 1 ELSE 0 END); ``` +状态窗口支持使用 TRUE_FOR 参数来设定窗口的最小持续时长。如果某个状态窗口的宽度低于该设定值,则会自动舍弃,不返回任何计算结果。例如,设置最短持续时长为 3s: + +``` +SELECT COUNT(*), FIRST(ts), status FROM temp_tb_1 STATE_WINDOW(status) TRUE_FOR (3s); +``` + ### 会话窗口 会话窗口根据记录的时间戳主键的值来确定是否属于同一个会话。如下图所示,如果设置时间戳的连续的间隔小于等于 12 秒,则以下 6 条记录构成 2 个会话窗口,分别是:[2019-04-28 14:22:10,2019-04-28 14:22:30]和[2019-04-28 14:23:10,2019-04-28 14:23:30]。因为 2019-04-28 14:22:30 与 2019-04-28 14:23:10 之间的时间间隔是 40 秒,超过了连续时间间隔(12 秒)。 @@ -196,6 +202,12 @@ select _wstart, _wend, count(*) from t event_window start with c1 > 0 end with c ![TDengine Database 事件窗口示意图](./event_window.webp) +事件窗口支持使用 TRUE_FOR 参数来设定窗口的最小持续时长。如果某个事件窗口的宽度低于该设定值,则会自动舍弃,不返回任何计算结果。例如,设置最短持续时长为 3s: + +``` +select _wstart, _wend, count(*) from t event_window start with c1 > 0 end with c2 < 10 true_for (3s); +``` + ### 计数窗口 计数窗口按固定的数据行数来划分窗口。默认将数据按时间戳排序,再按照count_val的值,将数据划分为多个窗口,然后做聚合计算。count_val表示每个count window包含的最大数据行数,总数据行数不能整除count_val时,最后一个窗口的行数会小于count_val。sliding_val是常量,表示窗口滑动的数量,类似于 interval的SLIDING。 diff --git a/docs/zh/14-reference/03-taos-sql/20-keywords.md b/docs/zh/14-reference/03-taos-sql/20-keywords.md index 9c023e0f92..1e8fabf571 100644 --- a/docs/zh/14-reference/03-taos-sql/20-keywords.md +++ b/docs/zh/14-reference/03-taos-sql/20-keywords.md @@ -77,6 +77,7 @@ description: TDengine 保留关键字的详细列表 | CLIENT_VERSION | | | CLUSTER | | | COLON | | +| COLS | | | COLUMN | | | COMMA | | | COMMENT | | @@ -427,6 +428,7 @@ description: TDengine 保留关键字的详细列表 | TRANSACTIONS | | | TRIGGER | | | TRIM | | +| TRUE_FOR | | | TSDB_PAGESIZE | | | TSERIES | | | TSMA | | diff --git a/docs/zh/14-reference/05-connector/40-csharp.mdx b/docs/zh/14-reference/05-connector/40-csharp.mdx index 89bba2e443..17120e52e9 100644 --- a/docs/zh/14-reference/05-connector/40-csharp.mdx +++ b/docs/zh/14-reference/05-connector/40-csharp.mdx @@ -60,8 +60,8 @@ TDengine 其他功能模块的报错,请参考 [错误码](../../../reference/ | GEOMETRY | byte[] | **注意**:JSON 类型仅在 tag 中支持。 -GEOMETRY类型是 little endian 字节序的二进制数据,符合 WKB 规范。详细信息请参考 [数据类型](../../taos-sql/data-type/#数据类型)。 -WKB规范请参考[Well-Known Binary (WKB)](https://libgeos.org/specifications/wkb/)。 +GEOMETRY 类型是 little endian 字节序的二进制数据,符合 WKB 规范。详细信息请参考 [数据类型](../../taos-sql/data-type/#数据类型)。 +WKB 规范请参考 [Well-Known Binary (WKB)](https://libgeos.org/specifications/wkb/)。 ## 示例程序汇总 diff --git a/docs/zh/14-reference/09-error-code.md b/docs/zh/14-reference/09-error-code.md index d29ff542eb..7326f45b34 100644 --- a/docs/zh/14-reference/09-error-code.md +++ b/docs/zh/14-reference/09-error-code.md @@ -389,20 +389,20 @@ description: TDengine 服务端的错误码列表和详细说明 | 错误码 | 错误描述 | 可能的出错场景或者可能的原因 | 建议用户采取的措施 | | ---------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------- | ------------------------------------- | -| 0x80002600 | syntax error near | SQL语法错误 | 检查并修正SQL语句 | -| 0x80002601 | Incomplete SQL statement | 不完整的SQL语句 | 检查并修正SQL语句 | -| 0x80002602 | Invalid column name | 不合法或不存在的列名 | 检查并修正SQL语句 | +| 0x80002600 | syntax error near | SQL语法错误 | 检查并修正 SQL 语句 | +| 0x80002601 | Incomplete SQL statement | 不完整的SQL语句 | 检查并修正 SQL 语句 | +| 0x80002602 | Invalid column name | 不合法或不存在的列名 | 检查并修正 SQL 语句 | | 0x80002603 | Table does not exist | 表不存在 | 检查并确认SQL语句中的表是否存在 | -| 0x80002604 | Column ambiguously defined | 列名(别名)重复定义 | 检查并修正SQL语句 | -| 0x80002605 | Invalid value type | 常量值非法 | 检查并修正SQL语句 | -| 0x80002608 | There mustn't be aggregation | 聚合函数出现在非法子句中 | 检查并修正SQL语句 | -| 0x80002609 | ORDER BY item must be the number of a SELECT-list expression | Order by指定的位置不合法 | 检查并修正SQL语句 | -| 0x8000260A | Not a GROUP BY expression | 非法group by语句 | 检查并修正SQL语句 | -| 0x8000260B | Not SELECTed expression | 非法表达式 | 检查并修正SQL语句 | -| 0x8000260C | Not a single-group group function | 非法使用列与函数 | 检查并修正SQL语句 | -| 0x8000260D | Tags number not matched | tag列个数不匹配 | 检查并修正SQL语句 | -| 0x8000260E | Invalid tag name | 无效或不存在的tag名 | 检查并修正SQL语句 | -| 0x80002610 | Value is too long | 值长度超出限制 | 检查并修正SQL语句或API参数 | +| 0x80002604 | Column ambiguously defined | 列名(别名)重复定义 | 检查并修正 SQL 语句 | +| 0x80002605 | Invalid value type | 常量值非法 | 检查并修正 SQL 语句 | +| 0x80002608 | There mustn't be aggregation | 聚合函数出现在非法子句中 | 检查并修正 SQL 语句 | +| 0x80002609 | ORDER BY item must be the number of a SELECT-list expression | Order by指定的位置不合法 | 检查并修正 SQL 语句 | +| 0x8000260A | Not a GROUP BY expression | 非法group by语句 | 检查并修正 SQL 语句 | +| 0x8000260B | Not SELECTed expression | 非法表达式 | 检查并修正 SQL 语句 | +| 0x8000260C | Not a single-group group function | 非法使用列与函数 | 检查并修正 SQL 语句 | +| 0x8000260D | Tags number not matched | tag列个数不匹配 | 检查并修正 SQL 语句 | +| 0x8000260E | Invalid tag name | 无效或不存在的tag名 | 检查并修正 SQL 语句 | +| 0x80002610 | Value is too long | 值长度超出限制 | 检查并修正 SQL 语句或API参数 | | 0x80002611 | Password too short or empty | 密码为空或少于 8 个字符 | 使用合法的密码 | | 0x80002612 | Port should be an integer that is less than 65535 and greater than 0 | 端口号非法 | 检查并修正端口号 | | 0x80002613 | Endpoint should be in the format of 'fqdn:port' | 地址格式错误 | 检查并修正地址信息 | @@ -413,72 +413,78 @@ description: TDengine 服务端的错误码列表和详细说明 | 0x80002618 | Corresponding super table not in this db | 超级表不存在 | 检查库中是否存在对应的超级表 | | 0x80002619 | Invalid database option | 数据库选项值非法 | 检查并修正数据库选项值 | | 0x8000261A | Invalid table option | 表选项值非法 | 检查并修正数据表选项值 | -| 0x80002624 | GROUP BY and WINDOW-clause can't be used together | Group by和窗口不能同时使用 | 检查并修正SQL语句 | -| 0x80002627 | Aggregate functions do not support nesting | 函数不支持嵌套使用 | 检查并修正SQL语句 | -| 0x80002628 | Only support STATE_WINDOW on integer/bool/varchar column | 不支持的STATE_WINDOW数据类型 | 检查并修正SQL语句 | -| 0x80002629 | Not support STATE_WINDOW on tag column | 不支持TAG列的STATE_WINDOW | 检查并修正SQL语句 | -| 0x8000262A | STATE_WINDOW not support for super table query | 不支持超级表的STATE_WINDOW | 检查并修正SQL语句 | -| 0x8000262B | SESSION gap should be fixed time window, and greater than 0 | SESSION窗口值非法 | 检查并修正SQL语句 | -| 0x8000262C | Only support SESSION on primary timestamp column | SESSION窗口列非法 | 检查并修正SQL语句 | -| 0x8000262D | Interval offset cannot be negative | INTERVAL offset值非法 | 检查并修正SQL语句 | -| 0x8000262E | Cannot use 'year' as offset when interval is 'month' | INTERVAL offset单位非法 | 检查并修正SQL语句 | -| 0x8000262F | Interval offset should be shorter than interval | INTERVAL offset值非法 | 检查并修正SQL语句 | -| 0x80002630 | Does not support sliding when interval is natural month/year | sliding单位非法 | 检查并修正SQL语句 | -| 0x80002631 | sliding value no larger than the interval value | sliding值非法 | 检查并修正SQL语句 | -| 0x80002632 | sliding value can not less than 1%% of interval value | sliding值非法 | 检查并修正SQL语句 | -| 0x80002633 | Only one tag if there is a json tag | 只支持单个JSON TAG列 | 检查并修正SQL语句 | -| 0x80002634 | Query block has incorrect number of result columns | 列个数不匹配 | 检查并修正SQL语句 | -| 0x80002635 | Incorrect TIMESTAMP value | 主键时间戳列值非法 | 检查并修正SQL语句 | -| 0x80002637 | soffset/offset can not be less than 0 | soffset/offset值非法 | 检查并修正SQL语句 | -| 0x80002638 | slimit/soffset only available for PARTITION/GROUP BY query | slimit/soffset只支持PARTITION BY/GROUP BY语句 | 检查并修正SQL语句 | +| 0x80002624 | GROUP BY and WINDOW-clause can't be used together | Group by和窗口不能同时使用 | 检查并修正 SQL 语句 | +| 0x80002627 | Aggregate functions do not support nesting | 函数不支持嵌套使用 | 检查并修正 SQL 语句 | +| 0x80002628 | Only support STATE_WINDOW on integer/bool/varchar column | 不支持的STATE_WINDOW数据类型 | 检查并修正 SQL 语句 | +| 0x80002629 | Not support STATE_WINDOW on tag column | 不支持TAG列的STATE_WINDOW | 检查并修正 SQL 语句 | +| 0x8000262A | STATE_WINDOW not support for super table query | 不支持超级表的STATE_WINDOW | 检查并修正 SQL 语句 | +| 0x8000262B | SESSION gap should be fixed time window, and greater than 0 | SESSION窗口值非法 | 检查并修正 SQL 语句 | +| 0x8000262C | Only support SESSION on primary timestamp column | SESSION窗口列非法 | 检查并修正 SQL 语句 | +| 0x8000262D | Interval offset cannot be negative | INTERVAL offset值非法 | 检查并修正 SQL 语句 | +| 0x8000262E | Cannot use 'year' as offset when interval is 'month' | INTERVAL offset单位非法 | 检查并修正 SQL 语句 | +| 0x8000262F | Interval offset should be shorter than interval | INTERVAL offset值非法 | 检查并修正 SQL 语句 | +| 0x80002630 | Does not support sliding when interval is natural month/year | sliding单位非法 | 检查并修正 SQL 语句 | +| 0x80002631 | sliding value no larger than the interval value | sliding值非法 | 检查并修正 SQL 语句 | +| 0x80002632 | sliding value can not less than 1%% of interval value | sliding值非法 | 检查并修正 SQL 语句 | +| 0x80002633 | Only one tag if there is a json tag | 只支持单个JSON TAG列 | 检查并修正 SQL 语句 | +| 0x80002634 | Query block has incorrect number of result columns | 列个数不匹配 | 检查并修正 SQL 语句 | +| 0x80002635 | Incorrect TIMESTAMP value | 主键时间戳列值非法 | 检查并修正 SQL 语句 | +| 0x80002637 | soffset/offset can not be less than 0 | soffset/offset值非法 | 检查并修正 SQL 语句 | +| 0x80002638 | slimit/soffset only available for PARTITION/GROUP BY query | slimit/soffset只支持PARTITION BY/GROUP BY语句 | 检查并修正 SQL 语句 | | 0x80002639 | Invalid topic query | 不支持的TOPIC查询语 | -| 0x8000263A | Cannot drop super table in batch | 不支持批量删除超级表 | 检查并修正SQL语句 | -| 0x8000263B | Start(end) time of query range required or time range too large | 窗口个数超出限制 | 检查并修正SQL语句 | -| 0x8000263C | Duplicated column names | 列名称重复 | 检查并修正SQL语句 | -| 0x8000263D | Tags length exceeds max length | TAG值长度超出最大支持范围 | 检查并修正SQL语句 | -| 0x8000263E | Row length exceeds max length | 行长度检查并修正SQL语句 | 检查并修正SQL语句 | -| 0x8000263F | Illegal number of columns | 列个数错误 | 检查并修正SQL语句 | -| 0x80002640 | Too many columns | 列个数超出上限 | 检查并修正SQL语句 | -| 0x80002641 | First column must be timestamp | 第一列必须是主键时间戳列 | 检查并修正SQL语句 | -| 0x80002642 | Invalid binary/nchar column/tag length | binary/nchar长度错误 | 检查并修正SQL语句 | -| 0x80002643 | Invalid number of tag columns | TAG列个数错误 | 检查并修正SQL语句 | +| 0x8000263A | Cannot drop super table in batch | 不支持批量删除超级表 | 检查并修正 SQL 语句 | +| 0x8000263B | Start(end) time of query range required or time range too large | 窗口个数超出限制 | 检查并修正 SQL 语句 | +| 0x8000263C | Duplicated column names | 列名称重复 | 检查并修正 SQL 语句 | +| 0x8000263D | Tags length exceeds max length | TAG值长度超出最大支持范围 | 检查并修正 SQL 语句 | +| 0x8000263E | Row length exceeds max length | 行长度检查并修正 SQL 语句 | 检查并修正 SQL 语句 | +| 0x8000263F | Illegal number of columns | 列个数错误 | 检查并修正 SQL 语句 | +| 0x80002640 | Too many columns | 列个数超出上限 | 检查并修正 SQL 语句 | +| 0x80002641 | First column must be timestamp | 第一列必须是主键时间戳列 | 检查并修正 SQL 语句 | +| 0x80002642 | Invalid binary/nchar column/tag length | binary/nchar长度错误 | 检查并修正 SQL 语句 | +| 0x80002643 | Invalid number of tag columns | TAG列个数错误 | 检查并修正 SQL 语句 | | 0x80002644 | Permission denied | 权限错误 | 检查确认用户是否有相应操作权限 | -| 0x80002645 | Invalid stream query | 非法流语句 | 检查并修正SQL语句 | -| 0x80002646 | Invalid _c0 or _rowts expression | _c0或_rowts非法使用 | 检查并修正SQL语句 | -| 0x80002647 | Invalid timeline function | 函数依赖的主键时间戳不存在 | 检查并修正SQL语句 | +| 0x80002645 | Invalid stream query | 非法流语句 | 检查并修正 SQL 语句 | +| 0x80002646 | Invalid _c0 or _rowts expression | _c0或_rowts非法使用 | 检查并修正 SQL 语句 | +| 0x80002647 | Invalid timeline function | 函数依赖的主键时间戳不存在 | 检查并修正 SQL 语句 | | 0x80002648 | Invalid password | 密码不符合规范 | 检查并修改密码 | -| 0x80002649 | Invalid alter table statement | 修改表语句不合法 | 检查并修正SQL语句 | -| 0x8000264A | Primary timestamp column cannot be dropped | 主键时间戳列不允许删除 | 检查并修正SQL语句 | -| 0x8000264B | Only binary/nchar column length could be modified, and the length can only be increased, not decreased | 非法列修改 | 检查并修正SQL语句 | -| 0x8000264C | Invalid tbname pseudo column | 非法使用tbname列 | 检查并修正SQL语句 | +| 0x80002649 | Invalid alter table statement | 修改表语句不合法 | 检查并修正 SQL 语句 | +| 0x8000264A | Primary timestamp column cannot be dropped | 主键时间戳列不允许删除 | 检查并修正 SQL 语句 | +| 0x8000264B | Only binary/nchar column length could be modified, and the length can only be increased, not decreased | 非法列修改 | 检查并修正 SQL 语句 | +| 0x8000264C | Invalid tbname pseudo column | 非法使用tbname列 | 检查并修正 SQL 语句 | | 0x8000264D | Invalid function name | 非法函数名 | 检查并修正函数名 | -| 0x8000264E | Comment too long | 注释长度超限 | 检查并修正SQL语句 | -| 0x8000264F | Function(s) only allowed in SELECT list, cannot mixed with non scalar functions or columns | 非法的函数混用 | 检查并修正SQL语句 | -| 0x80002650 | Window query not supported, since no valid timestamp column included in the result of subquery | 窗口查询依赖的主键时间戳列不存在 | 检查并修正SQL语句 | -| 0x80002651 | No columns can be dropped | 必须的列不能被删除 | 检查并修正SQL语句 | -| 0x80002652 | Only tag can be json type | 普通列不支持JSON类型 | 检查并修正SQL语句 | -| 0x80002655 | The DELETE statement must have a definite time window range | DELETE语句中存在非法WHERE条件 | 检查并修正SQL语句 | -| 0x80002656 | The REDISTRIBUTE VGROUP statement only support 1 to 3 dnodes | REDISTRIBUTE VGROUP指定的DNODE个数非法 | 检查并修正SQL语句 | -| 0x80002657 | Fill now allowed | 函数不允许FILL功能 | 检查并修正SQL语句 | -| 0x80002658 | Invalid windows pc | 非法使用窗口伪列 | 检查并修正SQL语句 | -| 0x80002659 | Window not allowed | 函数不能在窗口中使用 | 检查并修正SQL语句 | -| 0x8000265A | Stream not allowed | 函数不能在流计算中使用 | 检查并修正SQL语句 | -| 0x8000265B | Group by not allowd | 函数不能在分组中使用 | 检查并修正SQL语句 | -| 0x8000265D | Invalid interp clause | 非法INTERP或相关语句 | 检查并修正SQL语句 | -| 0x8000265E | Not valid function ion window | 非法窗口语句 | 检查并修正SQL语句 | -| 0x8000265F | Only support single table | 函数只支持在单表查询中使用 | 检查并修正SQL语句 | -| 0x80002660 | Invalid sma index | 非法创建SMA语句 | 检查并修正SQL语句 | -| 0x80002661 | Invalid SELECTed expression | 无效查询语句 | 检查并修正SQL语句 | +| 0x8000264E | Comment too long | 注释长度超限 | 检查并修正 SQL 语句 | +| 0x8000264F | Function(s) only allowed in SELECT list, cannot mixed with non scalar functions or columns | 非法的函数混用 | 检查并修正 SQL 语句 | +| 0x80002650 | Window query not supported, since no valid timestamp column included in the result of subquery | 窗口查询依赖的主键时间戳列不存在 | 检查并修正 SQL 语句 | +| 0x80002651 | No columns can be dropped | 必须的列不能被删除 | 检查并修正 SQL 语句 | +| 0x80002652 | Only tag can be json type | 普通列不支持JSON类型 | 检查并修正 SQL 语句 | +| 0x80002655 | The DELETE statement must have a definite time window range | DELETE语句中存在非法WHERE条件 | 检查并修正 SQL 语句 | +| 0x80002656 | The REDISTRIBUTE VGROUP statement only support 1 to 3 dnodes | REDISTRIBUTE VGROUP指定的DNODE个数非法 | 检查并修正 SQL 语句 | +| 0x80002657 | Fill now allowed | 函数不允许FILL功能 | 检查并修正 SQL 语句 | +| 0x80002658 | Invalid windows pc | 非法使用窗口伪列 | 检查并修正 SQL 语句 | +| 0x80002659 | Window not allowed | 函数不能在窗口中使用 | 检查并修正 SQL 语句 | +| 0x8000265A | Stream not allowed | 函数不能在流计算中使用 | 检查并修正 SQL 语句 | +| 0x8000265B | Group by not allowd | 函数不能在分组中使用 | 检查并修正 SQL 语句 | +| 0x8000265D | Invalid interp clause | 非法INTERP或相关语句 | 检查并修正 SQL 语句 | +| 0x8000265E | Not valid function ion window | 非法窗口语句 | 检查并修正 SQL 语句 | +| 0x8000265F | Only support single table | 函数只支持在单表查询中使用 | 检查并修正 SQL 语句 | +| 0x80002660 | Invalid sma index | 非法创建SMA语句 | 检查并修正 SQL 语句 | +| 0x80002661 | Invalid SELECTed expression | 无效查询语句 | 检查并修正 SQL 语句 | | 0x80002662 | Fail to get table info | 获取表元数据信息失败 | 保留现场和日志,github上报issue | -| 0x80002663 | Not unique table/alias | 表名(别名)冲突 | 检查并修正SQL语句 | -| 0x80002664 | Join requires valid time series input | 不支持子查询不含主键时间戳列输出的JOIN查询 | 检查并修正SQL语句 | -| 0x80002665 | The _TAGS pseudo column can only be used for subtable and supertable queries | 非法TAG列查询 | 检查并修正SQL语句 | -| 0x80002666 | 子查询不含主键时间戳列输出 | 检查并修正SQL语句 | -| 0x80002667 | Invalid usage of expr: %s | 非法表达式 | 检查并修正SQL语句 | +| 0x80002663 | Not unique table/alias | 表名(别名)冲突 | 检查并修正 SQL 语句 | +| 0x80002664 | Join requires valid time series input | 不支持子查询不含主键时间戳列输出的JOIN查询 | 检查并修正 SQL 语句 | +| 0x80002665 | The _TAGS pseudo column can only be used for subtable and supertable queries | 非法TAG列查询 | 检查并修正 SQL 语句 | +| 0x80002666 | 子查询不含主键时间戳列输出 | 检查并修正 SQL 语句 | +| 0x80002667 | Invalid usage of expr: %s | 非法表达式 | 检查并修正 SQL 语句 | +| 0x80002687 | True_for duration cannot be negative | true_for 的值不能是负数 | 检查并修正 SQL 语句 | +| 0x80002688 | Cannot use 'year' or 'month' as true_for duration | 不能使用 n(月), y(年) 作为 true_for 的时间单位 | 检查并修正 SQL 语句 | +| 0x80002689 | Invalid using cols function | cols函数使用错误 | 检查并修正 SQL 语句 | +| 0x8000268A | Cols function's first param must be a select function that output a single row | cols函数第一个参数应该为选择函数 | 检查并修正 SQL 语句 | +| 0x8000268B | Invalid using cols function with multiple output columns | 多列输出的 cols 函数使用错误 | 检查并修正 SQL 语句 | +| 0x8000268C | Invalid using alias for cols function | cols 函数输出列重命名错误 | 检查并修正 SQL 语句 | | 0x800026FF | Parser internal error | 解析器内部错误 | 保留现场和日志,github上报issue | | 0x80002700 | Planner internal error | 计划期内部错误 | 保留现场和日志,github上报issue | | 0x80002701 | Expect ts equal | JOIN条件校验失败 | 保留现场和日志,github上报issue | -| 0x80002702 | Cross join not support | 不支持CROSS JOIN | 检查并修正SQL语句 | +| 0x80002702 | Cross join not support | 不支持CROSS JOIN | 检查并修正 SQL 语句 | ## function diff --git a/include/common/tgrant.h b/include/common/tgrant.h index 1660cc6a1b..17a383f7c6 100644 --- a/include/common/tgrant.h +++ b/include/common/tgrant.h @@ -61,6 +61,7 @@ typedef enum { TSDB_GRANT_ACTIVE_ACTIVE, TSDB_GRANT_DUAL_REPLICA_HA, TSDB_GRANT_DB_ENCRYPTION, + TSDB_GRANT_TD_GPT, } EGrantType; int32_t checkAndGetCryptKey(const char *encryptCode, const char *machineId, char **key); diff --git a/include/common/tmsgcb.h b/include/common/tmsgcb.h index 2847f4278a..c934cb6961 100644 --- a/include/common/tmsgcb.h +++ b/include/common/tmsgcb.h @@ -38,7 +38,6 @@ typedef enum { STREAM_QUEUE, ARB_QUEUE, STREAM_CTRL_QUEUE, - STREAM_LONG_EXEC_QUEUE, QUEUE_MAX, } EQueueType; diff --git a/include/libs/executor/executor.h b/include/libs/executor/executor.h index e2bb6eefbf..f05234b82f 100644 --- a/include/libs/executor/executor.h +++ b/include/libs/executor/executor.h @@ -183,7 +183,7 @@ void qCleanExecTaskBlockBuf(qTaskInfo_t tinfo); */ int32_t qAsyncKillTask(qTaskInfo_t tinfo, int32_t rspCode); -int32_t qKillTask(qTaskInfo_t tinfo, int32_t rspCode, int64_t waitDuration); +int32_t qKillTask(qTaskInfo_t tinfo, int32_t rspCode); bool qTaskIsExecuting(qTaskInfo_t qinfo); diff --git a/include/libs/function/function.h b/include/libs/function/function.h index 126ed2c9b0..b2bde42463 100644 --- a/include/libs/function/function.h +++ b/include/libs/function/function.h @@ -276,12 +276,14 @@ typedef struct tExprNode { int32_t num; struct SFunctionNode *pFunctNode; int32_t functionType; + int32_t bindExprID; } _function; struct { struct SNode *pRootNode; } _optrRoot; }; + int32_t relatedTo; } tExprNode; struct SScalarParam { diff --git a/include/libs/function/functionMgt.h b/include/libs/function/functionMgt.h index 853f70e755..41b5d76371 100644 --- a/include/libs/function/functionMgt.h +++ b/include/libs/function/functionMgt.h @@ -155,6 +155,7 @@ typedef enum EFunctionType { FUNCTION_TYPE_FORECAST_LOW, FUNCTION_TYPE_FORECAST_HIGH, FUNCTION_TYPE_FORECAST_ROWTS, + FUNCTION_TYPE_COLS, FUNCTION_TYPE_IROWTS_ORIGIN, // internal function @@ -208,6 +209,7 @@ typedef enum EFunctionType { FUNCTION_TYPE_HYPERLOGLOG_STATE, FUNCTION_TYPE_HYPERLOGLOG_STATE_MERGE, + // geometry functions FUNCTION_TYPE_GEOM_FROM_TEXT = 4250, FUNCTION_TYPE_AS_TEXT, @@ -295,6 +297,7 @@ bool fmisSelectGroupConstValueFunc(int32_t funcId); bool fmIsElapsedFunc(int32_t funcId); bool fmIsDBUsageFunc(int32_t funcId); bool fmIsRowTsOriginFunc(int32_t funcId); +bool fmIsSelectColsFunc(int32_t funcId); void getLastCacheDataType(SDataType* pType, int32_t pkBytes); int32_t createFunction(const char* pName, SNodeList* pParameterList, SFunctionNode** pFunc); diff --git a/include/libs/nodes/plannodes.h b/include/libs/nodes/plannodes.h index 1afec35c3c..9ed565090e 100644 --- a/include/libs/nodes/plannodes.h +++ b/include/libs/nodes/plannodes.h @@ -327,6 +327,7 @@ typedef struct SWindowLogicNode { SNode* pStateExpr; SNode* pStartCond; SNode* pEndCond; + int64_t trueForLimit; int8_t triggerType; int64_t watermark; int64_t deleteMark; @@ -724,6 +725,7 @@ typedef SSessionWinodwPhysiNode SStreamFinalSessionWinodwPhysiNode; typedef struct SStateWinodwPhysiNode { SWindowPhysiNode window; SNode* pStateKey; + int64_t trueForLimit; } SStateWinodwPhysiNode; typedef SStateWinodwPhysiNode SStreamStateWinodwPhysiNode; @@ -732,6 +734,7 @@ typedef struct SEventWinodwPhysiNode { SWindowPhysiNode window; SNode* pStartCond; SNode* pEndCond; + int64_t trueForLimit; } SEventWinodwPhysiNode; typedef SEventWinodwPhysiNode SStreamEventWinodwPhysiNode; diff --git a/include/libs/nodes/querynodes.h b/include/libs/nodes/querynodes.h index c80dda863f..3c15ffa6b4 100644 --- a/include/libs/nodes/querynodes.h +++ b/include/libs/nodes/querynodes.h @@ -16,6 +16,7 @@ #ifndef _TD_QUERY_NODES_H_ #define _TD_QUERY_NODES_H_ +#include #ifdef __cplusplus extern "C" { #endif @@ -61,6 +62,8 @@ typedef struct SExprNode { bool asParam; bool asPosition; int32_t projIdx; + int32_t relatedTo; + int32_t bindExprID; } SExprNode; typedef enum EColumnType { @@ -322,6 +325,7 @@ typedef struct SStateWindowNode { ENodeType type; // QUERY_NODE_STATE_WINDOW SNode* pCol; // timestamp primary key SNode* pExpr; + SNode* pTrueForLimit; } SStateWindowNode; typedef struct SSessionWindowNode { @@ -346,6 +350,7 @@ typedef struct SEventWindowNode { SNode* pCol; // timestamp primary key SNode* pStartCond; SNode* pEndCond; + SNode* pTrueForLimit; } SEventWindowNode; typedef struct SCountWindowNode { @@ -428,6 +433,7 @@ typedef struct SSelectStmt { ENodeType type; // QUERY_NODE_SELECT_STMT bool isDistinct; SNodeList* pProjectionList; + SNodeList* pProjectionBindList; SNode* pFromTable; SNode* pWhere; SNodeList* pPartitionByList; @@ -697,6 +703,9 @@ char* getJoinSTypeString(EJoinSubType type); char* getFullJoinTypeString(EJoinType type, EJoinSubType stype); int32_t mergeJoinConds(SNode** ppDst, SNode** ppSrc); +void rewriteExprAliasName(SExprNode* pNode, int64_t num); +bool isRelatedToOtherExpr(SExprNode* pExpr); + #ifdef __cplusplus } #endif diff --git a/include/libs/stream/tstream.h b/include/libs/stream/tstream.h index 682e042915..041d888d33 100644 --- a/include/libs/stream/tstream.h +++ b/include/libs/stream/tstream.h @@ -58,7 +58,6 @@ extern "C" { #define STREAM_EXEC_T_STOP_ALL_TASKS (-5) #define STREAM_EXEC_T_RESUME_TASK (-6) #define STREAM_EXEC_T_ADD_FAILED_TASK (-7) -#define STREAM_EXEC_T_STOP_ONE_TASK (-8) typedef struct SStreamTask SStreamTask; typedef struct SStreamQueue SStreamQueue; @@ -769,19 +768,15 @@ void streamMetaCleanup(); int32_t streamMetaOpen(const char* path, void* ahandle, FTaskBuild expandFunc, FTaskExpand expandTaskFn, int32_t vgId, int64_t stage, startComplete_fn_t fn, SStreamMeta** pMeta); void streamMetaClose(SStreamMeta* streamMeta); - -int32_t streamMetaSaveTaskInMeta(SStreamMeta* pMeta, SStreamTask* pTask); // save to stream meta store -int32_t streamMetaRemoveTaskInMeta(SStreamMeta* pMeta, STaskId* pKey); - +int32_t streamMetaSaveTask(SStreamMeta* pMeta, SStreamTask* pTask); // save to stream meta store +int32_t streamMetaRemoveTask(SStreamMeta* pMeta, STaskId* pKey); int32_t streamMetaRegisterTask(SStreamMeta* pMeta, int64_t ver, SStreamTask* pTask, bool* pAdded); int32_t streamMetaUnregisterTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId); int32_t streamMetaGetNumOfTasks(SStreamMeta* pMeta); - int32_t streamMetaAcquireTaskNoLock(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, SStreamTask** pTask); int32_t streamMetaAcquireTaskUnsafe(SStreamMeta* pMeta, STaskId* pId, SStreamTask** pTask); int32_t streamMetaAcquireTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId, SStreamTask** pTask); void streamMetaReleaseTask(SStreamMeta* pMeta, SStreamTask* pTask); - void streamMetaClear(SStreamMeta* pMeta); void streamMetaInitBackend(SStreamMeta* pMeta); int32_t streamMetaCommit(SStreamMeta* pMeta); @@ -815,7 +810,6 @@ void streamMetaLoadAllTasks(SStreamMeta* pMeta); int32_t streamMetaStartAllTasks(SStreamMeta* pMeta); int32_t streamMetaStopAllTasks(SStreamMeta* pMeta); int32_t streamMetaStartOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId); -int32_t streamMetaStopOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId); bool streamMetaAllTasksReady(const SStreamMeta* pMeta); int32_t streamTaskSendNegotiateChkptIdMsg(SStreamTask* pTask); int32_t streamTaskCheckIfReqConsenChkptId(SStreamTask* pTask, int64_t ts); diff --git a/include/util/taoserror.h b/include/util/taoserror.h index 397118411c..29e95251e9 100644 --- a/include/util/taoserror.h +++ b/include/util/taoserror.h @@ -693,6 +693,7 @@ int32_t taosGetErrSize(); #define TSDB_CODE_GRANT_OBJECT_STROAGE_EXPIRED TAOS_DEF_ERROR_CODE(0, 0x082B) #define TSDB_CODE_GRANT_DUAL_REPLICA_HA_EXPIRED TAOS_DEF_ERROR_CODE(0, 0x082C) #define TSDB_CODE_GRANT_DB_ENCRYPTION_EXPIRED TAOS_DEF_ERROR_CODE(0, 0x082D) +#define TSDB_CODE_GRANT_TD_GPT_EXPIRED TAOS_DEF_ERROR_CODE(0, 0x082E) // sync // #define TSDB_CODE_SYN_INVALID_CONFIG TAOS_DEF_ERROR_CODE(0, 0x0900) // 2.x @@ -908,6 +909,12 @@ int32_t taosGetErrSize(); #define TSDB_CODE_PAR_INVALID_ANOMALY_WIN_OPT TAOS_DEF_ERROR_CODE(0, 0x2684) #define TSDB_CODE_PAR_INVALID_FORECAST_CLAUSE TAOS_DEF_ERROR_CODE(0, 0x2685) #define TSDB_CODE_PAR_INVALID_VGID_LIST TAOS_DEF_ERROR_CODE(0, 0x2686) +#define TSDB_CODE_PAR_TRUE_FOR_NEGATIVE TAOS_DEF_ERROR_CODE(0, 0x2687) +#define TSDB_CODE_PAR_TRUE_FOR_UNIT TAOS_DEF_ERROR_CODE(0, 0x2688) +#define TSDB_CODE_PAR_INVALID_COLS_FUNCTION TAOS_DEF_ERROR_CODE(0, 0x2689) +#define TSDB_CODE_PAR_INVALID_COLS_SELECTFUNC TAOS_DEF_ERROR_CODE(0, 0x268A) +#define TSDB_CODE_INVALID_MULITI_COLS_FUNC TAOS_DEF_ERROR_CODE(0, 0x268B) +#define TSDB_CODE_INVALID_COLS_ALIAS TAOS_DEF_ERROR_CODE(0, 0x268C) #define TSDB_CODE_PAR_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x26FF) //planner diff --git a/include/util/tdef.h b/include/util/tdef.h index 41cb147622..f76d57f5b5 100644 --- a/include/util/tdef.h +++ b/include/util/tdef.h @@ -266,6 +266,7 @@ typedef enum ELogicConditionType { #define TSDB_SUBSCRIBE_KEY_LEN (TSDB_CGROUP_LEN + TSDB_TOPIC_FNAME_LEN + 2) #define TSDB_PARTITION_KEY_LEN (TSDB_SUBSCRIBE_KEY_LEN + 20) #define TSDB_COL_NAME_LEN 65 +#define TSDB_COL_NAME_EXLEN 8 #define TSDB_COL_FNAME_LEN (TSDB_TABLE_NAME_LEN + TSDB_COL_NAME_LEN + TSDB_NAME_DELIMITER_LEN) #define TSDB_MAX_SAVED_SQL_LEN TSDB_MAX_COLUMNS * 64 #define TSDB_MAX_SQL_LEN TSDB_PAYLOAD_SIZE diff --git a/include/util/tworker.h b/include/util/tworker.h index bc0dde1a37..a3ba7dba6d 100644 --- a/include/util/tworker.h +++ b/include/util/tworker.h @@ -76,7 +76,7 @@ void tQWorkerFreeQueue(SQWorkerPool *pool, STaosQueue *queue); int32_t tAutoQWorkerInit(SAutoQWorkerPool *pool); void tAutoQWorkerCleanup(SAutoQWorkerPool *pool); -STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem fp, int32_t minNum); +STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem fp); void tAutoQWorkerFreeQueue(SAutoQWorkerPool *pool, STaosQueue *queue); int32_t tWWorkerInit(SWWorkerPool *pool); diff --git a/source/client/test/clientTests.cpp b/source/client/test/clientTests.cpp index 54c0e59817..8ebbe62857 100644 --- a/source/client/test/clientTests.cpp +++ b/source/client/test/clientTests.cpp @@ -1510,7 +1510,7 @@ TEST(clientCase, sub_tb_mt_test) { (void)taosThreadCreate(&qid[i], NULL, doConsumeData, NULL); } - for (int32_t i = 0; i < 4; ++i) { + for (int32_t i = 0; i < 1; ++i) { (void)taosThreadJoin(qid[i], NULL); } } diff --git a/source/common/src/tglobal.c b/source/common/src/tglobal.c index 2fd16b4f67..6369358008 100644 --- a/source/common/src/tglobal.c +++ b/source/common/src/tglobal.c @@ -188,7 +188,7 @@ char tsCheckpointBackupDir[PATH_MAX] = "/var/lib/taos/backup/checkpoint/"; // tmq int32_t tmqMaxTopicNum = 20; -int32_t tmqRowSize = 4096; +int32_t tmqRowSize = 1000; // query int32_t tsQueryPolicy = 1; bool tsQueryTbNotExistAsEmpty = false; diff --git a/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h b/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h index 9b4c11d6ae..84f5149624 100644 --- a/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h +++ b/source/dnode/mgmt/mgmt_vnode/inc/vmInt.h @@ -32,7 +32,6 @@ typedef struct SVnodeMgmt { const char *name; SQueryAutoQWorkerPool queryPool; SAutoQWorkerPool streamPool; - SAutoQWorkerPool streamLongExecPool; SWWorkerPool streamCtrlPool; SWWorkerPool fetchPool; SSingleWorker mgmtWorker; @@ -76,7 +75,6 @@ typedef struct { STaosQueue *pQueryQ; STaosQueue *pStreamQ; STaosQueue *pStreamCtrlQ; - STaosQueue *pStreamLongExecQ; STaosQueue *pFetchQ; STaosQueue *pMultiMgmQ; } SVnodeObj; @@ -139,8 +137,6 @@ int32_t vmPutMsgToQueryQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t vmPutMsgToFetchQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t vmPutMsgToStreamQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t vmPutMsgToStreamCtrlQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); -int32_t vmPutMsgToStreamLongExecQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); - int32_t vmPutMsgToMergeQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t vmPutMsgToMgmtQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); int32_t vmPutMsgToMultiMgmtQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg); diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c index 1dea7d3cad..234d4f41e1 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmHandle.c @@ -1008,29 +1008,27 @@ SArray *vmGetMsgHandles() { if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RUN, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_DISPATCH_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_CHECK, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_CHECK_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_CHECKPOINT_READY, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_CHECKPOINT_READY_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE_TRIGGER, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE_TRIGGER_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_HEARTBEAT_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_REQ_CHKPT_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_REPORT_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_SCAN_HISTORY, vmPutMsgToStreamLongExecQueue, 0) == NULL) goto _OVER; - - if (dmSetMgmtHandle(pArray, TDMT_VND_GET_STREAM_PROGRESS, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE_RSP, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER; - - if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_UPDATE_CHKPT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; - if (dmSetMgmtHandle(pArray, TDMT_STREAM_CONSEN_CHKPT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_CHECK, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_CHECK_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_PAUSE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_RESUME, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_STOP, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_CHECK_POINT_SOURCE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_CHECKPOINT_READY, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_CHECKPOINT_READY_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE_TRIGGER, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_RETRIEVE_TRIGGER_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_UPDATE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_STREAM_TASK_RESET, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_HEARTBEAT_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_REQ_CHKPT_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_MND_STREAM_CHKPT_REPORT_RSP, vmPutMsgToStreamCtrlQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_VND_GET_STREAM_PROGRESS, vmPutMsgToStreamQueue, 0) == NULL) goto _OVER; + + if (dmSetMgmtHandle(pArray, TDMT_STREAM_TASK_UPDATE_CHKPT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; + if (dmSetMgmtHandle(pArray, TDMT_STREAM_CONSEN_CHKPT, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_REPLICA, vmPutMsgToMgmtQueue, 0) == NULL) goto _OVER; if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_CONFIG, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER; diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c index 6f30977e10..d71e0b02c4 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmInt.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmInt.c @@ -398,14 +398,10 @@ void vmCloseVnode(SVnodeMgmt *pMgmt, SVnodeObj *pVnode, bool commitAndRemoveWal, dInfo("vgId:%d, wait for vnode stream queue:%p is empty, %d remains", pVnode->vgId, pVnode->pStreamQ, taosQueueItemSize(pVnode->pStreamQ)); - while (!taosQueueEmpty(pVnode->pStreamQ)) taosMsleep(50); + while (!taosQueueEmpty(pVnode->pStreamQ)) taosMsleep(10); dInfo("vgId:%d, wait for vnode stream ctrl queue:%p is empty", pVnode->vgId, pVnode->pStreamCtrlQ); - while (!taosQueueEmpty(pVnode->pStreamCtrlQ)) taosMsleep(50); - - dInfo("vgId:%d, wait for vnode stream long-exec queue:%p is empty, %d remains", pVnode->vgId, - pVnode->pStreamLongExecQ, taosQueueItemSize(pVnode->pStreamLongExecQ)); - while (!taosQueueEmpty(pVnode->pStreamLongExecQ)) taosMsleep(50); + while (!taosQueueEmpty(pVnode->pStreamCtrlQ)) taosMsleep(10); dInfo("vgId:%d, all vnode queues is empty", pVnode->vgId); diff --git a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c index 5acd06bbda..b398bdf242 100644 --- a/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c +++ b/source/dnode/mgmt/mgmt_vnode/src/vmWorker.c @@ -150,7 +150,7 @@ static void vmProcessStreamCtrlQueue(SQueueInfo *pInfo, STaosQall* pQall, int32_ SRpcMsg *pMsg = pItem; const STraceId *trace = &pMsg->info.traceId; - dGTrace("vgId:%d, msg:%p get from vnode-stream-ctrl queue", pVnode->vgId, pMsg); + dGTrace("vgId:%d, msg:%p get from vnode-ctrl-stream queue", pVnode->vgId, pMsg); code = vnodeProcessStreamCtrlMsg(pVnode->pImpl, pMsg, pInfo); if (code != 0) { terrno = code; @@ -165,26 +165,6 @@ static void vmProcessStreamCtrlQueue(SQueueInfo *pInfo, STaosQall* pQall, int32_ } } -static void vmProcessStreamLongExecQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) { - SVnodeObj *pVnode = pInfo->ahandle; - const STraceId *trace = &pMsg->info.traceId; - int32_t code = 0; - - dGTrace("vgId:%d, msg:%p get from vnode-stream long-exec queue", pVnode->vgId, pMsg); - - code = vnodeProcessStreamLongExecMsg(pVnode->pImpl, pMsg, pInfo); - if (code != 0) { - terrno = code; - dGError("vgId:%d, msg:%p failed to process stream msg %s since %s", pVnode->vgId, pMsg, TMSG_INFO(pMsg->msgType), - tstrerror(code)); - vmSendRsp(pMsg, code); - } - - dGTrace("vgId:%d, msg:%p is freed, code:0x%x", pVnode->vgId, pMsg, code); - rpcFreeCont(pMsg->pCont); - taosFreeQitem(pMsg); -} - static void vmProcessFetchQueue(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs) { SVnodeObj *pVnode = pInfo->ahandle; SRpcMsg *pMsg = NULL; @@ -294,13 +274,9 @@ static int32_t vmPutMsgToQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg, EQueueType qtyp code = taosWriteQitem(pVnode->pStreamQ, pMsg); break; case STREAM_CTRL_QUEUE: - dGTrace("vgId:%d, msg:%p put into vnode-stream-ctrl queue", pVnode->vgId, pMsg); + dGTrace("vgId:%d, msg:%p put into vnode-ctrl-stream queue", pVnode->vgId, pMsg); code = taosWriteQitem(pVnode->pStreamCtrlQ, pMsg); break; - case STREAM_LONG_EXEC_QUEUE: - dGTrace("vgId:%d, msg:%p put into vnode-stream-long-exec queue", pVnode->vgId, pMsg); - code = taosWriteQitem(pVnode->pStreamLongExecQ, pMsg); - break; case FETCH_QUEUE: dGTrace("vgId:%d, msg:%p put into vnode-fetch queue", pVnode->vgId, pMsg); code = taosWriteQitem(pVnode->pFetchQ, pMsg); @@ -359,8 +335,6 @@ int32_t vmPutMsgToStreamQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMs int32_t vmPutMsgToStreamCtrlQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsgToQueue(pMgmt, pMsg, STREAM_CTRL_QUEUE); } -int32_t vmPutMsgToStreamLongExecQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { return vmPutMsgToQueue(pMgmt, pMsg, STREAM_LONG_EXEC_QUEUE); } - int32_t vmPutMsgToMultiMgmtQueue(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) { const STraceId *trace = &pMsg->info.traceId; dGTrace("msg:%p, put into vnode-multi-mgmt queue", pMsg); @@ -435,10 +409,6 @@ int32_t vmGetQueueSize(SVnodeMgmt *pMgmt, int32_t vgId, EQueueType qtype) { break; case STREAM_CTRL_QUEUE: size = taosQueueItemSize(pVnode->pStreamCtrlQ); - break; - case STREAM_LONG_EXEC_QUEUE: - size = taosQueueItemSize(pVnode->pStreamLongExecQ); - break; default: break; } @@ -481,16 +451,13 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { } pVnode->pQueryQ = tQueryAutoQWorkerAllocQueue(&pMgmt->queryPool, pVnode, (FItem)vmProcessQueryQueue); + pVnode->pStreamQ = tAutoQWorkerAllocQueue(&pMgmt->streamPool, pVnode, (FItem)vmProcessStreamQueue); pVnode->pFetchQ = tWWorkerAllocQueue(&pMgmt->fetchPool, pVnode, (FItems)vmProcessFetchQueue); - - // init stream msg processing queue family - pVnode->pStreamQ = tAutoQWorkerAllocQueue(&pMgmt->streamPool, pVnode, (FItem)vmProcessStreamQueue, 2); pVnode->pStreamCtrlQ = tWWorkerAllocQueue(&pMgmt->streamCtrlPool, pVnode, (FItems)vmProcessStreamCtrlQueue); - pVnode->pStreamLongExecQ = tAutoQWorkerAllocQueue(&pMgmt->streamLongExecPool, pVnode, (FItem)vmProcessStreamLongExecQueue, 1); if (pVnode->pWriteW.queue == NULL || pVnode->pSyncW.queue == NULL || pVnode->pSyncRdW.queue == NULL || pVnode->pApplyW.queue == NULL || pVnode->pQueryQ == NULL || pVnode->pStreamQ == NULL || pVnode->pFetchQ == NULL - || pVnode->pStreamCtrlQ == NULL || pVnode->pStreamLongExecQ == NULL) { + || pVnode->pStreamCtrlQ == NULL) { return TSDB_CODE_OUT_OF_MEMORY; } @@ -506,7 +473,6 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { dInfo("vgId:%d, fetch-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pFetchQ, taosQueueGetThreadId(pVnode->pFetchQ)); dInfo("vgId:%d, stream-queue:%p is alloced", pVnode->vgId, pVnode->pStreamQ); - dInfo("vgId:%d, stream-long-exec-queue:%p is alloced", pVnode->vgId, pVnode->pStreamLongExecQ); dInfo("vgId:%d, stream-ctrl-queue:%p is alloced, thread:%08" PRId64, pVnode->vgId, pVnode->pStreamCtrlQ, taosQueueGetThreadId(pVnode->pStreamCtrlQ)); return 0; @@ -515,22 +481,17 @@ int32_t vmAllocQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { void vmFreeQueue(SVnodeMgmt *pMgmt, SVnodeObj *pVnode) { tQueryAutoQWorkerFreeQueue(&pMgmt->queryPool, pVnode->pQueryQ); tAutoQWorkerFreeQueue(&pMgmt->streamPool, pVnode->pStreamQ); - tAutoQWorkerFreeQueue(&pMgmt->streamLongExecPool, pVnode->pStreamLongExecQ); tWWorkerFreeQueue(&pMgmt->streamCtrlPool, pVnode->pStreamCtrlQ); tWWorkerFreeQueue(&pMgmt->fetchPool, pVnode->pFetchQ); pVnode->pQueryQ = NULL; - pVnode->pFetchQ = NULL; - pVnode->pStreamQ = NULL; pVnode->pStreamCtrlQ = NULL; - pVnode->pStreamLongExecQ = NULL; - + pVnode->pFetchQ = NULL; dDebug("vgId:%d, queue is freed", pVnode->vgId); } int32_t vmStartWorker(SVnodeMgmt *pMgmt) { - int32_t code = 0; - + int32_t code = 0; SQueryAutoQWorkerPool *pQPool = &pMgmt->queryPool; pQPool->name = "vnode-query"; pQPool->min = tsNumOfVnodeQueryThreads; @@ -544,13 +505,8 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) { pStreamPool->ratio = tsRatioOfVnodeStreamThreads; if ((code = tAutoQWorkerInit(pStreamPool)) != 0) return code; - SAutoQWorkerPool *pLongExecPool = &pMgmt->streamLongExecPool; - pLongExecPool->name = "vnode-stream-long-exec"; - pLongExecPool->ratio = tsRatioOfVnodeStreamThreads/3; - if ((code = tAutoQWorkerInit(pLongExecPool)) != 0) return code; - SWWorkerPool *pStreamCtrlPool = &pMgmt->streamCtrlPool; - pStreamCtrlPool->name = "vnode-stream-ctrl"; + pStreamCtrlPool->name = "vnode-ctrl-stream"; pStreamCtrlPool->max = 1; if ((code = tWWorkerInit(pStreamCtrlPool)) != 0) return code; @@ -585,7 +541,6 @@ int32_t vmStartWorker(SVnodeMgmt *pMgmt) { void vmStopWorker(SVnodeMgmt *pMgmt) { tQueryAutoQWorkerCleanup(&pMgmt->queryPool); tAutoQWorkerCleanup(&pMgmt->streamPool); - tAutoQWorkerCleanup(&pMgmt->streamLongExecPool); tWWorkerCleanup(&pMgmt->streamCtrlPool); tWWorkerCleanup(&pMgmt->fetchPool); dDebug("vnode workers are closed"); diff --git a/source/dnode/mnode/impl/src/mndDump.c b/source/dnode/mnode/impl/src/mndDump.c index d1dd99b319..485a89f1bd 100644 --- a/source/dnode/mnode/impl/src/mndDump.c +++ b/source/dnode/mnode/impl/src/mndDump.c @@ -41,7 +41,7 @@ int32_t sendSyncReq(const SEpSet *pEpSet, SRpcMsg *pMsg) { } char *i642str(int64_t val) { - static char str[24] = {0}; + static threadlocal char str[24] = {0}; (void)snprintf(str, sizeof(str), "%" PRId64, val); return str; } diff --git a/source/dnode/vnode/inc/vnode.h b/source/dnode/vnode/inc/vnode.h index d224f9a411..f0e7af50f3 100644 --- a/source/dnode/vnode/inc/vnode.h +++ b/source/dnode/vnode/inc/vnode.h @@ -113,7 +113,6 @@ int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo); int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo); int32_t vnodeProcessStreamMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo); int32_t vnodeProcessStreamCtrlMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo); -int32_t vnodeProcessStreamLongExecMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo); void vnodeProposeWriteMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs); void vnodeApplyWriteMsg(SQueueInfo *pInfo, STaosQall *qall, int32_t numOfMsgs); void vnodeProposeCommitOnNeed(SVnode *pVnode, bool atExit); diff --git a/source/dnode/vnode/src/sma/smaRollup.c b/source/dnode/vnode/src/sma/smaRollup.c index 1f755f816e..7d83dbcf84 100644 --- a/source/dnode/vnode/src/sma/smaRollup.c +++ b/source/dnode/vnode/src/sma/smaRollup.c @@ -1302,7 +1302,7 @@ _checkpoint: } streamMetaWLock(pMeta); - if ((code = streamMetaSaveTaskInMeta(pMeta, pTask)) != 0) { + if ((code = streamMetaSaveTask(pMeta, pTask)) != 0) { streamMetaWUnLock(pMeta); taosHashCancelIterate(pInfoHash, infoHash); TSDB_CHECK_CODE(code, lino, _exit); diff --git a/source/dnode/vnode/src/tq/tq.c b/source/dnode/vnode/src/tq/tq.c index 66e64b91d9..b9de5dce52 100644 --- a/source/dnode/vnode/src/tq/tq.c +++ b/source/dnode/vnode/src/tq/tq.c @@ -962,6 +962,7 @@ int32_t tqProcessTaskScanHistory(STQ* pTq, SRpcMsg* pMsg) { int32_t code = TSDB_CODE_SUCCESS; SStreamTask* pTask = NULL; SStreamTask* pStreamTask = NULL; + char* pStatus = NULL; code = streamMetaAcquireTask(pMeta, pReq->streamId, pReq->taskId, &pTask); if (pTask == NULL) { @@ -972,7 +973,29 @@ int32_t tqProcessTaskScanHistory(STQ* pTq, SRpcMsg* pMsg) { // do recovery step1 const char* id = pTask->id.idStr; - char* pStatus = streamTaskGetStatus(pTask).name; + streamMutexLock(&pTask->lock); + + SStreamTaskState s = streamTaskGetStatus(pTask); + pStatus = s.name; + + if ((s.state != TASK_STATUS__SCAN_HISTORY) || (pTask->status.downstreamReady == 0)) { + tqError("s-task:%s vgId:%d status:%s downstreamReady:%d not allowed/ready for scan-history data, quit", id, + pMeta->vgId, s.name, pTask->status.downstreamReady); + + streamMutexUnlock(&pTask->lock); + streamMetaReleaseTask(pMeta, pTask); + return 0; + } + + if (pTask->exec.pExecutor == NULL) { + tqError("s-task:%s vgId:%d executor is null, not executor scan history", id, pMeta->vgId); + + streamMutexUnlock(&pTask->lock); + streamMetaReleaseTask(pMeta, pTask); + return 0; + } + + streamMutexUnlock(&pTask->lock); // avoid multi-thread exec while (1) { diff --git a/source/dnode/vnode/src/tq/tqRead.c b/source/dnode/vnode/src/tq/tqRead.c index d650ae9751..ee705ed374 100644 --- a/source/dnode/vnode/src/tq/tqRead.c +++ b/source/dnode/vnode/src/tq/tqRead.c @@ -1098,6 +1098,22 @@ int32_t tqRetrieveTaosxBlock(STqReader* pReader, SMqDataRsp* pRsp, SArray* block *pSubmitTbDataRet = pSubmitTbData; } + if (fetchMeta == ONLY_META) { + if (pSubmitTbData->pCreateTbReq != NULL) { + if (pRsp->createTableReq == NULL){ + pRsp->createTableReq = taosArrayInit(0, POINTER_BYTES); + if (pRsp->createTableReq == NULL){ + return terrno; + } + } + if (taosArrayPush(pRsp->createTableReq, &pSubmitTbData->pCreateTbReq) == NULL){ + return terrno; + } + pSubmitTbData->pCreateTbReq = NULL; + } + return 0; + } + int32_t sversion = pSubmitTbData->sver; int64_t uid = pSubmitTbData->uid; pReader->lastBlkUid = uid; diff --git a/source/dnode/vnode/src/tq/tqScan.c b/source/dnode/vnode/src/tq/tqScan.c index 70a165906e..549a47d006 100644 --- a/source/dnode/vnode/src/tq/tqScan.c +++ b/source/dnode/vnode/src/tq/tqScan.c @@ -339,6 +339,9 @@ static void tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int bool tmp = (pSubmitTbData->flags & pRequest->sourceExcluded) != 0; TSDB_CHECK_CONDITION(!tmp, code, lino, END, TSDB_CODE_SUCCESS); + if (pHandle->fetchMeta == ONLY_META){ + goto END; + } int32_t blockNum = taosArrayGetSize(pBlocks) == 0 ? 1 : taosArrayGetSize(pBlocks); if (pRsp->withTbName) { @@ -347,7 +350,6 @@ static void tqProcessSubData(STQ* pTq, STqHandle* pHandle, SMqDataRsp* pRsp, int TSDB_CHECK_CODE(code, lino, END); } - tmp = (pHandle->fetchMeta == ONLY_META && pSubmitTbData->pCreateTbReq == NULL); TSDB_CHECK_CONDITION(!tmp, code, lino, END, TSDB_CODE_SUCCESS); for (int32_t i = 0; i < blockNum; i++) { if (taosArrayGetSize(pBlocks) == 0){ @@ -403,14 +405,16 @@ static void preProcessSubmitMsg(STqHandle* pHandle, const SMqPollReq* pRequest, } int64_t uid = pSubmitTbData->uid; - if (taosHashGet(pRequest->uidHash, &uid, LONG_BYTES) != NULL) { - tqDebug("poll rawdata split,uid:%" PRId64 " is already exists", uid); - terrno = TSDB_CODE_TMQ_RAW_DATA_SPLIT; - return; - } else { - int32_t code = taosHashPut(pRequest->uidHash, &uid, LONG_BYTES, &uid, LONG_BYTES); - if (code != 0){ - tqError("failed to add table uid to hash, code:%d, uid:%"PRId64, code, uid); + if (pRequest->rawData) { + if (taosHashGet(pRequest->uidHash, &uid, LONG_BYTES) != NULL) { + tqDebug("poll rawdata split,uid:%" PRId64 " is already exists", uid); + terrno = TSDB_CODE_TMQ_RAW_DATA_SPLIT; + return; + } else { + int32_t code = taosHashPut(pRequest->uidHash, &uid, LONG_BYTES, &uid, LONG_BYTES); + if (code != 0) { + tqError("failed to add table uid to hash, code:%d, uid:%" PRId64, code, uid); + } } } @@ -453,9 +457,7 @@ int32_t tqTaosxScanLog(STQ* pTq, STqHandle* pHandle, SPackedData submit, SMqData } code = tqReaderSetSubmitMsg(pReader, submit.msgStr, submit.msgLen, submit.ver, rawList); TSDB_CHECK_CODE(code, lino, END); - if (pRequest->rawData) { - preProcessSubmitMsg(pHandle, pRequest, &rawList); - } + preProcessSubmitMsg(pHandle, pRequest, &rawList); // data could not contains same uid data in rawdata mode if (pRequest->rawData != 0 && terrno == TSDB_CODE_TMQ_RAW_DATA_SPLIT){ goto END; diff --git a/source/dnode/vnode/src/tq/tqUtil.c b/source/dnode/vnode/src/tq/tqUtil.c index 9866528446..bc8b8504c4 100644 --- a/source/dnode/vnode/src/tq/tqUtil.c +++ b/source/dnode/vnode/src/tq/tqUtil.c @@ -226,6 +226,82 @@ static void tDeleteCommon(void* parm) {} taosxRsp.createTableNum > 0 ? TMQ_MSG_TYPE__POLL_DATA_META_RSP : \ (pRequest->rawData == 1 ? TMQ_MSG_TYPE__POLL_RAW_DATA_RSP : TMQ_MSG_TYPE__POLL_DATA_RSP) +static int32_t buildBatchMeta(SMqBatchMetaRsp *btMetaRsp, int16_t type, int32_t bodyLen, void* body){ + int32_t code = 0; + + if (!btMetaRsp->batchMetaReq) { + btMetaRsp->batchMetaReq = taosArrayInit(4, POINTER_BYTES); + TQ_NULL_GO_TO_END(btMetaRsp->batchMetaReq); + btMetaRsp->batchMetaLen = taosArrayInit(4, sizeof(int32_t)); + TQ_NULL_GO_TO_END(btMetaRsp->batchMetaLen); + } + + SMqMetaRsp tmpMetaRsp = {0}; + tmpMetaRsp.resMsgType = type; + tmpMetaRsp.metaRspLen = bodyLen; + tmpMetaRsp.metaRsp = body; + uint32_t len = 0; + tEncodeSize(tEncodeMqMetaRsp, &tmpMetaRsp, len, code); + if (TSDB_CODE_SUCCESS != code) { + tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); + goto END; + } + int32_t tLen = sizeof(SMqRspHead) + len; + void* tBuf = taosMemoryCalloc(1, tLen); + TQ_NULL_GO_TO_END(tBuf); + void* metaBuff = POINTER_SHIFT(tBuf, sizeof(SMqRspHead)); + SEncoder encoder = {0}; + tEncoderInit(&encoder, metaBuff, len); + code = tEncodeMqMetaRsp(&encoder, &tmpMetaRsp); + tEncoderClear(&encoder); + + if (code < 0) { + tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); + goto END; + } + TQ_NULL_GO_TO_END (taosArrayPush(btMetaRsp->batchMetaReq, &tBuf)); + TQ_NULL_GO_TO_END (taosArrayPush(btMetaRsp->batchMetaLen, &tLen)); + +END: + return code; +} + +static int32_t buildCreateTbBatchReqBinary(SMqDataRsp *taosxRsp, void** pBuf, int32_t *len){ + int32_t code = 0; + SVCreateTbBatchReq pReq = {0}; + pReq.nReqs = taosArrayGetSize(taosxRsp->createTableReq); + pReq.pArray = taosArrayInit(1, sizeof(struct SVCreateTbReq)); + TQ_NULL_GO_TO_END(pReq.pArray); + for (int i = 0; i < taosArrayGetSize(taosxRsp->createTableReq); i++){ + void *createTableReq = taosArrayGetP(taosxRsp->createTableReq, i); + TQ_NULL_GO_TO_END(taosArrayPush(pReq.pArray, createTableReq)); + } + tEncodeSize(tEncodeSVCreateTbBatchReq, &pReq, *len, code); + if (code < 0) { + goto END; + } + *len += sizeof(SMsgHead); + *pBuf = taosMemoryMalloc(*len); + TQ_NULL_GO_TO_END(pBuf); + SEncoder coder = {0}; + tEncoderInit(&coder, POINTER_SHIFT(*pBuf, sizeof(SMsgHead)), *len); + code = tEncodeSVCreateTbBatchReq(&coder, &pReq); + tEncoderClear(&coder); + +END: + taosArrayDestroy(pReq.pArray); + return code; +} + +#define SEND_BATCH_META_RSP \ +tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer);\ +code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId);\ +goto END; + +#define SEND_DATA_RSP \ +tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer);\ +code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, POLL_RSP_TYPE(pRequest, taosxRsp), vgId);\ +goto END; static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequest, SRpcMsg* pMsg, STqOffsetVal* offset) { int32_t vgId = TD_VID(pTq->pVnode); @@ -272,14 +348,9 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, if (tqFetchLog(pTq, pHandle, &fetchVer, pRequest->reqId) < 0) { if (totalMetaRows > 0) { - tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer); - code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId); - goto END; + SEND_BATCH_META_RSP } - tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer); - code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, - POLL_RSP_TYPE(pRequest, taosxRsp), vgId); - goto END; + SEND_DATA_RSP } SWalCont* pHead = &pHandle->pWalReader->pHead->head; @@ -289,10 +360,7 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, // process meta if (pHead->msgType != TDMT_VND_SUBMIT) { if (totalRows > 0) { - tqOffsetResetToLog(&taosxRsp.rspOffset, fetchVer); - code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, - POLL_RSP_TYPE(pRequest, taosxRsp), vgId); - goto END; + SEND_DATA_RSP } if ((pRequest->sourceExcluded & TD_REQ_FROM_TAOX) != 0) { @@ -318,53 +386,20 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, code = tqSendMetaPollRsp(pHandle, pMsg, pRequest, &metaRsp, vgId); goto END; } - - if (!btMetaRsp.batchMetaReq) { - btMetaRsp.batchMetaReq = taosArrayInit(4, POINTER_BYTES); - TQ_NULL_GO_TO_END(btMetaRsp.batchMetaReq); - btMetaRsp.batchMetaLen = taosArrayInit(4, sizeof(int32_t)); - TQ_NULL_GO_TO_END(btMetaRsp.batchMetaLen); - } + code = buildBatchMeta(&btMetaRsp, pHead->msgType, pHead->bodyLen, pHead->body); fetchVer++; - - SMqMetaRsp tmpMetaRsp = {0}; - tmpMetaRsp.resMsgType = pHead->msgType; - tmpMetaRsp.metaRspLen = pHead->bodyLen; - tmpMetaRsp.metaRsp = pHead->body; - uint32_t len = 0; - tEncodeSize(tEncodeMqMetaRsp, &tmpMetaRsp, len, code); - if (TSDB_CODE_SUCCESS != code) { - tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); - continue; + if (code != 0){ + goto END; } - int32_t tLen = sizeof(SMqRspHead) + len; - void* tBuf = taosMemoryCalloc(1, tLen); - TQ_NULL_GO_TO_END(tBuf); - void* metaBuff = POINTER_SHIFT(tBuf, sizeof(SMqRspHead)); - SEncoder encoder = {0}; - tEncoderInit(&encoder, metaBuff, len); - code = tEncodeMqMetaRsp(&encoder, &tmpMetaRsp); - tEncoderClear(&encoder); - - if (code < 0) { - tqError("tmq extract meta from log, tEncodeMqMetaRsp error"); - continue; - } - TQ_NULL_GO_TO_END (taosArrayPush(btMetaRsp.batchMetaReq, &tBuf)); - TQ_NULL_GO_TO_END (taosArrayPush(btMetaRsp.batchMetaLen, &tLen)); totalMetaRows++; if ((taosArrayGetSize(btMetaRsp.batchMetaReq) >= tmqRowSize) || (taosGetTimestampMs() - st > pRequest->timeout)) { - tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer); - code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId); - goto END; + SEND_BATCH_META_RSP } continue; } - if (totalMetaRows > 0) { - tqOffsetResetToLog(&btMetaRsp.rspOffset, fetchVer); - code = tqSendBatchMetaPollRsp(pHandle, pMsg, pRequest, &btMetaRsp, vgId); - goto END; + if (totalMetaRows > 0 && pHandle->fetchMeta != ONLY_META) { + SEND_BATCH_META_RSP } // process data @@ -376,17 +411,39 @@ static int32_t extractDataAndRspForDbStbSubscribe(STQ* pTq, STqHandle* pHandle, TQ_ERR_GO_TO_END(tqTaosxScanLog(pTq, pHandle, submit, &taosxRsp, &totalRows, pRequest)); + if (pHandle->fetchMeta == ONLY_META && taosArrayGetSize(taosxRsp.createTableReq) > 0){ + int32_t len = 0; + void *pBuf = NULL; + code = buildCreateTbBatchReqBinary(&taosxRsp, &pBuf, &len); + if (code == 0){ + code = buildBatchMeta(&btMetaRsp, TDMT_VND_CREATE_TABLE, len, pBuf); + } + taosMemoryFree(pBuf); + taosArrayDestroyP(taosxRsp.createTableReq, NULL); + taosxRsp.createTableReq = NULL; + fetchVer++; + if (code != 0){ + goto END; + } + totalMetaRows++; + if ((taosArrayGetSize(btMetaRsp.batchMetaReq) >= tmqRowSize) || + (taosGetTimestampMs() - st > pRequest->timeout) || + (!pRequest->enableBatchMeta && !pRequest->useSnapshot)) { + SEND_BATCH_META_RSP + } + continue; + } + if ((pRequest->rawData == 0 && totalRows >= pRequest->minPollRows) || (taosGetTimestampMs() - st > pRequest->timeout) || (pRequest->rawData != 0 && (taosArrayGetSize(taosxRsp.blockData) > pRequest->minPollRows || terrno == TSDB_CODE_TMQ_RAW_DATA_SPLIT))) { -// tqDebug("start to send rsp, block num:%d, totalRows:%d, createTableNum:%d, terrno:%d", -// (int)taosArrayGetSize(taosxRsp.blockData), totalRows, taosxRsp.createTableNum, terrno); - tqOffsetResetToLog(&taosxRsp.rspOffset, terrno == TSDB_CODE_TMQ_RAW_DATA_SPLIT ? fetchVer : fetchVer + 1); - code = tqSendDataRsp(pHandle, pMsg, pRequest, &taosxRsp, - POLL_RSP_TYPE(pRequest, taosxRsp), vgId); - if (terrno == TSDB_CODE_TMQ_RAW_DATA_SPLIT){terrno = 0;} - goto END; + if (terrno == TSDB_CODE_TMQ_RAW_DATA_SPLIT){ + terrno = 0; + } else{ + fetchVer++; + } + SEND_DATA_RSP } else { fetchVer++; } diff --git a/source/dnode/vnode/src/tqCommon/tqCommon.c b/source/dnode/vnode/src/tqCommon/tqCommon.c index 197716ba79..0c4a3932b7 100644 --- a/source/dnode/vnode/src/tqCommon/tqCommon.c +++ b/source/dnode/vnode/src/tqCommon/tqCommon.c @@ -268,13 +268,13 @@ int32_t tqStreamTaskProcessUpdateReq(SStreamMeta* pMeta, SMsgCb* cb, SRpcMsg* pM // stream do update the nodeEp info, write it into stream meta. if (updated) { tqDebug("s-task:%s vgId:%d save task after update epset, and stop task", idstr, vgId); - code = streamMetaSaveTaskInMeta(pMeta, pTask); + code = streamMetaSaveTask(pMeta, pTask); if (code) { tqError("s-task:%s vgId:%d failed to save task, code:%s", idstr, vgId, tstrerror(code)); } if (pHTask != NULL) { - code = streamMetaSaveTaskInMeta(pMeta, pHTask); + code = streamMetaSaveTask(pMeta, pHTask); if (code) { tqError("s-task:%s vgId:%d failed to save related history task, code:%s", idstr, vgId, tstrerror(code)); } @@ -751,8 +751,6 @@ int32_t tqStreamTaskProcessDropReq(SStreamMeta* pMeta, char* msg, int32_t msgLen } streamMetaWUnLock(pMeta); - tqDebug("vgId:%d process drop task:0x%x completed", vgId, pReq->taskId); - return 0; // always return success } @@ -867,9 +865,6 @@ int32_t tqStreamTaskProcessRunReq(SStreamMeta* pMeta, SRpcMsg* pMsg, bool isLead } else if (type == STREAM_EXEC_T_ADD_FAILED_TASK) { code = streamMetaAddFailedTask(pMeta, req.streamId, req.taskId); return code; - } else if (type == STREAM_EXEC_T_STOP_ONE_TASK) { - code = streamMetaStopOneTask(pMeta, req.streamId, req.taskId); - return code; } else if (type == STREAM_EXEC_T_RESUME_TASK) { // task resume to run after idle for a while SStreamTask* pTask = NULL; code = streamMetaAcquireTask(pMeta, req.streamId, req.taskId, &pTask); diff --git a/source/dnode/vnode/src/vnd/vnodeSvr.c b/source/dnode/vnode/src/vnd/vnodeSvr.c index 40cee195ea..abaa61744d 100644 --- a/source/dnode/vnode/src/vnd/vnodeSvr.c +++ b/source/dnode/vnode/src/vnd/vnodeSvr.c @@ -934,7 +934,9 @@ int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) { int32_t vnodeProcessStreamMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) { vTrace("vgId:%d, msg:%p in stream queue is processing", pVnode->config.vgId, pMsg); - if (!syncIsReadyForRead(pVnode->sync)) { + if ((pMsg->msgType == TDMT_SCH_FETCH || pMsg->msgType == TDMT_VND_TABLE_META || pMsg->msgType == TDMT_VND_TABLE_CFG || + pMsg->msgType == TDMT_VND_BATCH_META) && + !syncIsReadyForRead(pVnode->sync)) { vnodeRedirectRpcMsg(pVnode, pMsg, terrno); return 0; } @@ -946,6 +948,8 @@ int32_t vnodeProcessStreamMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) return tqProcessTaskRetrieveReq(pVnode->pTq, pMsg); case TDMT_STREAM_RETRIEVE_RSP: return tqProcessTaskRetrieveRsp(pVnode->pTq, pMsg); + case TDMT_VND_STREAM_SCAN_HISTORY: + return tqProcessTaskScanHistory(pVnode->pTq, pMsg); case TDMT_VND_GET_STREAM_PROGRESS: return tqStreamProgressRetrieveReq(pVnode->pTq, pMsg); default: @@ -992,22 +996,6 @@ int32_t vnodeProcessStreamCtrlMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pIn } } -int32_t vnodeProcessStreamLongExecMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) { - vTrace("vgId:%d, msg:%p in stream long exec queue is processing", pVnode->config.vgId, pMsg); - if (!syncIsReadyForRead(pVnode->sync)) { - vnodeRedirectRpcMsg(pVnode, pMsg, terrno); - return 0; - } - - switch (pMsg->msgType) { - case TDMT_VND_STREAM_SCAN_HISTORY: - return tqProcessTaskScanHistory(pVnode->pTq, pMsg); - default: - vError("unknown msg type:%d in stream long exec queue", pMsg->msgType); - return TSDB_CODE_APP_ERROR; - } -} - void smaHandleRes(void *pVnode, int64_t smaId, const SArray *data) { int32_t code = tdProcessTSmaInsert(((SVnode *)pVnode)->pSma, smaId, (const char *)data); if (code) { diff --git a/source/libs/executor/inc/executil.h b/source/libs/executor/inc/executil.h index 9e36a29476..726ab6eaea 100644 --- a/source/libs/executor/inc/executil.h +++ b/source/libs/executor/inc/executil.h @@ -42,6 +42,7 @@ typedef struct SGroupResInfo { int32_t index; // rows consumed in func:doCopyToSDataBlockXX int32_t iter; // relate to index-1, last consumed data's slot id in hash table void* dataPos; // relate to index-1, last consumed data's position, in the nodelist of cur slot + int32_t delIndex; // rows consumed in func:doBuildDeleteDataBlock SArray* pRows; // SArray char* pBuf; bool freeItem; diff --git a/source/libs/executor/inc/executorInt.h b/source/libs/executor/inc/executorInt.h index e7bc1f67e1..2ad625e309 100644 --- a/source/libs/executor/inc/executorInt.h +++ b/source/libs/executor/inc/executorInt.h @@ -686,6 +686,54 @@ typedef struct SResultWindowInfo { bool isOutput; } SResultWindowInfo; +typedef struct SSessionAggOperatorInfo { + SOptrBasicInfo binfo; + SAggSupporter aggSup; + SExprSupp scalarSupp; // supporter for perform scalar function + SGroupResInfo groupResInfo; + SWindowRowsSup winSup; + bool reptScan; // next round scan + int64_t gap; // session window gap + int32_t tsSlotId; // primary timestamp slot id + STimeWindowAggSupp twAggSup; + struct SOperatorInfo* pOperator; + bool cleanGroupResInfo; +} SSessionAggOperatorInfo; + +typedef struct SStateWindowOperatorInfo { + SOptrBasicInfo binfo; + SAggSupporter aggSup; + SExprSupp scalarSup; + SGroupResInfo groupResInfo; + SWindowRowsSup winSup; + SColumn stateCol; // start row index + bool hasKey; + SStateKeys stateKey; + int32_t tsSlotId; // primary timestamp column slot id + STimeWindowAggSupp twAggSup; + struct SOperatorInfo* pOperator; + bool cleanGroupResInfo; + int64_t trueForLimit; +} SStateWindowOperatorInfo; + + +typedef struct SEventWindowOperatorInfo { + SOptrBasicInfo binfo; + SAggSupporter aggSup; + SExprSupp scalarSup; + SWindowRowsSup winSup; + int32_t tsSlotId; // primary timestamp column slot id + STimeWindowAggSupp twAggSup; + uint64_t groupId; // current group id, used to identify the data block from different groups + SFilterInfo* pStartCondInfo; + SFilterInfo* pEndCondInfo; + bool inWindow; + SResultRow* pRow; + SSDataBlock* pPreDataBlock; + struct SOperatorInfo* pOperator; + int64_t trueForLimit; +} SEventWindowOperatorInfo; + typedef struct SStreamSessionAggOperatorInfo { SOptrBasicInfo binfo; SSteamOpBasicInfo basic; @@ -746,6 +794,7 @@ typedef struct SStreamStateAggOperatorInfo { SSHashObj* pPkDeleted; bool destHasPrimaryKey; struct SOperatorInfo* pOperator; + int64_t trueForLimit; } SStreamStateAggOperatorInfo; typedef struct SStreamEventAggOperatorInfo { @@ -778,6 +827,7 @@ typedef struct SStreamEventAggOperatorInfo { struct SOperatorInfo* pOperator; SNodeList* pStartCondCols; SNodeList* pEndCondCols; + int64_t trueForLimit; } SStreamEventAggOperatorInfo; typedef struct SStreamCountAggOperatorInfo { @@ -1052,7 +1102,8 @@ int32_t saveSessionOutputBuf(SStreamAggSupporter* pAggSup, SResultWindowInfo* pW int32_t saveResult(SResultWindowInfo winInfo, SSHashObj* pStUpdated); int32_t saveDeleteRes(SSHashObj* pStDelete, SSessionKey key); void removeSessionResult(SStreamAggSupporter* pAggSup, SSHashObj* pHashMap, SSHashObj* pResMap, SSessionKey* pKey); -void doBuildDeleteDataBlock(struct SOperatorInfo* pOp, SSHashObj* pStDeleted, SSDataBlock* pBlock, void** Ite); +void doBuildDeleteDataBlock(struct SOperatorInfo* pOp, SSHashObj* pStDeleted, SSDataBlock* pBlock, void** Ite, + SGroupResInfo* pGroupResInfo); void doBuildSessionResult(struct SOperatorInfo* pOperator, void* pState, SGroupResInfo* pGroupResInfo, SSDataBlock* pBlock, SArray* pSessionKeys); int32_t getSessionWindowInfoByKey(SStreamAggSupporter* pAggSup, SSessionKey* pKey, SResultWindowInfo* pWinInfo); @@ -1109,6 +1160,7 @@ int32_t getNextQualifiedWindow(SInterval* pInterval, STimeWindow* pNext, SDataBl int32_t extractQualifiedTupleByFilterResult(SSDataBlock* pBlock, const SColumnInfoData* p, int32_t status); bool getIgoreNullRes(SExprSupp* pExprSup); bool checkNullRow(SExprSupp* pExprSup, SSDataBlock* pSrcBlock, int32_t index, bool ignoreNull); +int64_t getMinWindowSize(struct SOperatorInfo* pOperator); #ifdef __cplusplus } diff --git a/source/libs/executor/src/aggregateoperator.c b/source/libs/executor/src/aggregateoperator.c index 5713726501..1b8e8298b4 100644 --- a/source/libs/executor/src/aggregateoperator.c +++ b/source/libs/executor/src/aggregateoperator.c @@ -15,6 +15,7 @@ #include "filter.h" #include "function.h" +#include "nodes.h" #include "os.h" #include "querynodes.h" #include "tfill.h" diff --git a/source/libs/executor/src/eventwindowoperator.c b/source/libs/executor/src/eventwindowoperator.c index e68a91d97d..925a4bd7ff 100644 --- a/source/libs/executor/src/eventwindowoperator.c +++ b/source/libs/executor/src/eventwindowoperator.c @@ -24,22 +24,6 @@ #include "tdatablock.h" #include "ttime.h" -typedef struct SEventWindowOperatorInfo { - SOptrBasicInfo binfo; - SAggSupporter aggSup; - SExprSupp scalarSup; - SWindowRowsSup winSup; - int32_t tsSlotId; // primary timestamp column slot id - STimeWindowAggSupp twAggSup; - uint64_t groupId; // current group id, used to identify the data block from different groups - SFilterInfo* pStartCondInfo; - SFilterInfo* pEndCondInfo; - bool inWindow; - SResultRow* pRow; - SSDataBlock* pPreDataBlock; - SOperatorInfo* pOperator; -} SEventWindowOperatorInfo; - static int32_t eventWindowAggregateNext(SOperatorInfo* pOperator, SSDataBlock** pRes); static void destroyEWindowOperatorInfo(void* param); static int32_t eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* pInfo, SSDataBlock* pBlock); @@ -114,8 +98,9 @@ int32_t createEventwindowOperatorInfo(SOperatorInfo* downstream, SPhysiNode* phy pInfo->tsSlotId = tsSlotId; pInfo->pPreDataBlock = NULL; pInfo->pOperator = pOperator; + pInfo->trueForLimit = pEventWindowNode->trueForLimit; - setOperatorInfo(pOperator, "EventWindowOperator", QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE, true, OP_NOT_OPENED, pInfo, + setOperatorInfo(pOperator, "EventWindowOperator", QUERY_NODE_PHYSICAL_PLAN_MERGE_EVENT, true, OP_NOT_OPENED, pInfo, pTaskInfo); pOperator->fpSet = createOperatorFpSet(optrDummyOpenFn, eventWindowAggregateNext, NULL, destroyEWindowOperatorInfo, optrDefaultBufFn, NULL, optrDefaultGetNextExtFn, NULL); @@ -297,6 +282,7 @@ int32_t eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* p TSKEY* tsList = (TSKEY*)pColInfoData->pData; SWindowRowsSup* pRowSup = &pInfo->winSup; int32_t rowIndex = 0; + int64_t minWindowSize = getMinWindowSize(pOperator); pRowSup->numOfRows = 0; if (pInfo->groupId == 0) { @@ -341,18 +327,23 @@ int32_t eventWindowAggImpl(SOperatorInfo* pOperator, SEventWindowOperatorInfo* p QUERY_CHECK_CODE(code, lino, _return); doUpdateNumOfRows(pSup->pCtx, pInfo->pRow, pSup->numOfExprs, pSup->rowEntryInfoOffset); - // check buffer size - if (pRes->info.rows + pInfo->pRow->numOfRows >= pRes->info.capacity) { - int32_t newSize = pRes->info.rows + pInfo->pRow->numOfRows; - code = blockDataEnsureCapacity(pRes, newSize); + if (pRowSup->win.ekey - pRowSup->win.skey < minWindowSize) { + qDebug("skip small window, groupId: %" PRId64 ", windowSize: %" PRId64 ", minWindowSize: %" PRId64, + pInfo->groupId, pRowSup->win.ekey - pRowSup->win.skey, minWindowSize); + } else { + // check buffer size + if (pRes->info.rows + pInfo->pRow->numOfRows >= pRes->info.capacity) { + int32_t newSize = pRes->info.rows + pInfo->pRow->numOfRows; + code = blockDataEnsureCapacity(pRes, newSize); + QUERY_CHECK_CODE(code, lino, _return); + } + + code = copyResultrowToDataBlock(pSup->pExprInfo, pSup->numOfExprs, pInfo->pRow, pSup->pCtx, pRes, + pSup->rowEntryInfoOffset, pTaskInfo); QUERY_CHECK_CODE(code, lino, _return); + + pRes->info.rows += pInfo->pRow->numOfRows; } - - code = copyResultrowToDataBlock(pSup->pExprInfo, pSup->numOfExprs, pInfo->pRow, pSup->pCtx, pRes, - pSup->rowEntryInfoOffset, pTaskInfo); - QUERY_CHECK_CODE(code, lino, _return); - - pRes->info.rows += pInfo->pRow->numOfRows; pInfo->pRow->numOfRows = 0; pInfo->inWindow = false; diff --git a/source/libs/executor/src/executil.c b/source/libs/executor/src/executil.c index 147d62d245..7bff5550df 100644 --- a/source/libs/executor/src/executil.c +++ b/source/libs/executor/src/executil.c @@ -18,6 +18,8 @@ #include "index.h" #include "os.h" #include "query.h" +#include "querynodes.h" +#include "tarray.h" #include "tdatablock.h" #include "thash.h" #include "tmsg.h" @@ -212,6 +214,7 @@ void cleanupGroupResInfo(SGroupResInfo* pGroupResInfo) { pGroupResInfo->pRows = NULL; } pGroupResInfo->index = 0; + pGroupResInfo->delIndex = 0; } int32_t resultrowComparAsc(const void* p1, const void* p2) { @@ -303,6 +306,7 @@ void initMultiResInfoFromArrayList(SGroupResInfo* pGroupResInfo, SArray* pArrayL pGroupResInfo->freeItem = true; pGroupResInfo->pRows = pArrayList; pGroupResInfo->index = 0; + pGroupResInfo->delIndex = 0; } bool hasRemainResults(SGroupResInfo* pGroupResInfo) { @@ -1956,6 +1960,7 @@ int32_t createExprFromOneNode(SExprInfo* pExp, SNode* pNode, int16_t slotId) { QUERY_CHECK_CODE(code, lino, _end); } } + pExp->pExpr->_function.bindExprID = ((SExprNode*)pNode)->bindExprID; } else if (type == QUERY_NODE_OPERATOR) { pExp->pExpr->nodeType = QUERY_NODE_OPERATOR; SOperatorNode* pOpNode = (SOperatorNode*)pNode; @@ -1993,7 +1998,7 @@ int32_t createExprFromOneNode(SExprInfo* pExp, SNode* pNode, int16_t slotId) { code = TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; QUERY_CHECK_CODE(code, lino, _end); } - + pExp->pExpr->relatedTo = ((SExprNode*)pNode)->relatedTo; _end: if (code != TSDB_CODE_SUCCESS) { qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); @@ -2074,42 +2079,78 @@ int32_t createExprInfo(SNodeList* pNodeList, SNodeList* pGroupKeys, SExprInfo** return code; } +static void deleteSubsidiareCtx(void* pData) { + SSubsidiaryResInfo* pCtx = (SSubsidiaryResInfo*)pData; + if (pCtx->pCtx) { + taosMemoryFreeClear(pCtx->pCtx); + } +} + // set the output buffer for the selectivity + tag query static int32_t setSelectValueColumnInfo(SqlFunctionCtx* pCtx, int32_t numOfOutput) { int32_t num = 0; int32_t code = TSDB_CODE_SUCCESS; int32_t lino = 0; - SqlFunctionCtx* p = NULL; - SqlFunctionCtx** pValCtx = taosMemoryCalloc(numOfOutput, POINTER_BYTES); - if (pValCtx == NULL) { - return terrno; + SArray* pValCtxArray = NULL; + for (int32_t i = numOfOutput - 1; i > 0; --i) { // select Func is at the end of the list + int32_t funcIdx = pCtx[i].pExpr->pExpr->_function.bindExprID; + if (funcIdx > 0) { + if (pValCtxArray == NULL) { + // the end of the list is the select function of biggest index + pValCtxArray = taosArrayInit_s(sizeof(SSubsidiaryResInfo*), funcIdx); + if (pValCtxArray == NULL) { + return terrno; + } + } + if (funcIdx > pValCtxArray->size) { + qError("funcIdx:%d is out of range", funcIdx); + taosArrayDestroyP(pValCtxArray, deleteSubsidiareCtx); + return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR; + } + SSubsidiaryResInfo* pSubsidiary = &pCtx[i].subsidiaries; + pSubsidiary->pCtx = taosMemoryCalloc(numOfOutput, POINTER_BYTES); + if (pSubsidiary->pCtx == NULL) { + taosArrayDestroyP(pValCtxArray, deleteSubsidiareCtx); + return terrno; + } + pSubsidiary->num = 0; + taosArraySet(pValCtxArray, funcIdx - 1, &pSubsidiary); + } } - SHashObj* pSelectFuncs = taosHashInit(8, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK); - QUERY_CHECK_NULL(pSelectFuncs, code, lino, _end, terrno); + SqlFunctionCtx* p = NULL; + SqlFunctionCtx** pValCtx = NULL; + if (pValCtxArray == NULL) { + pValCtx = taosMemoryCalloc(numOfOutput, POINTER_BYTES); + if (pValCtx == NULL) { + QUERY_CHECK_CODE(terrno, lino, _end); + } + } for (int32_t i = 0; i < numOfOutput; ++i) { const char* pName = pCtx[i].pExpr->pExpr->_function.functionName; - if ((strcmp(pName, "_select_value") == 0) || (strcmp(pName, "_group_key") == 0) || - (strcmp(pName, "_group_const_value") == 0)) { - pValCtx[num++] = &pCtx[i]; - } else if (fmIsSelectFunc(pCtx[i].functionId)) { - void* data = taosHashGet(pSelectFuncs, pName, strlen(pName)); - if (taosHashGetSize(pSelectFuncs) != 0 && data == NULL) { - p = NULL; - break; + if ((strcmp(pName, "_select_value") == 0)) { + if (pValCtxArray == NULL) { + pValCtx[num++] = &pCtx[i]; } else { - int32_t tempRes = taosHashPut(pSelectFuncs, pName, strlen(pName), &num, sizeof(num)); - if (tempRes != TSDB_CODE_SUCCESS && tempRes != TSDB_CODE_DUP_KEY) { - code = tempRes; - QUERY_CHECK_CODE(code, lino, _end); + int32_t bindFuncIndex = pCtx[i].pExpr->pExpr->relatedTo; // start from index 1; + if (bindFuncIndex > 0) { // 0 is default index related to the select function + bindFuncIndex -= 1; } - p = &pCtx[i]; + SSubsidiaryResInfo** pSubsidiary = taosArrayGet(pValCtxArray, bindFuncIndex); + if(pSubsidiary == NULL) { + QUERY_CHECK_CODE(TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR, lino, _end); + } + (*pSubsidiary)->pCtx[(*pSubsidiary)->num] = &pCtx[i]; + (*pSubsidiary)->num++; } + } else if (fmIsSelectFunc(pCtx[i].functionId)) { + if (pValCtxArray == NULL) { + p = &pCtx[i]; + } } } - taosHashCleanup(pSelectFuncs); if (p != NULL) { p->subsidiaries.pCtx = pValCtx; @@ -2120,9 +2161,11 @@ static int32_t setSelectValueColumnInfo(SqlFunctionCtx* pCtx, int32_t numOfOutpu _end: if (code != TSDB_CODE_SUCCESS) { + taosArrayDestroyP(pValCtxArray, deleteSubsidiareCtx); taosMemoryFreeClear(pValCtx); - taosHashCleanup(pSelectFuncs); qError("%s failed at line %d since %s", __func__, lino, tstrerror(code)); + } else { + taosArrayDestroy(pValCtxArray); } return code; } diff --git a/source/libs/executor/src/executor.c b/source/libs/executor/src/executor.c index 7131001f7a..efe0792a76 100644 --- a/source/libs/executor/src/executor.c +++ b/source/libs/executor/src/executor.c @@ -995,43 +995,26 @@ int32_t qAsyncKillTask(qTaskInfo_t qinfo, int32_t rspCode) { return TSDB_CODE_SUCCESS; } -int32_t qKillTask(qTaskInfo_t tinfo, int32_t rspCode, int64_t waitDuration) { - int64_t st = taosGetTimestampMs(); +int32_t qKillTask(qTaskInfo_t tinfo, int32_t rspCode) { SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo; if (pTaskInfo == NULL) { return TSDB_CODE_QRY_INVALID_QHANDLE; } - if (waitDuration > 0) { - qDebug("%s sync killed execTask, and waiting for %.2fs", GET_TASKID(pTaskInfo), waitDuration/1000.0); - } else { - qDebug("%s async killed execTask", GET_TASKID(pTaskInfo)); - } - + qDebug("%s sync killed execTask", GET_TASKID(pTaskInfo)); setTaskKilled(pTaskInfo, TSDB_CODE_TSC_QUERY_KILLED); - if (waitDuration > 0) { - while (1) { - taosWLockLatch(&pTaskInfo->lock); - if (qTaskIsExecuting(pTaskInfo)) { // let's wait for 100 ms and try again - taosWUnLockLatch(&pTaskInfo->lock); - - taosMsleep(200); - - int64_t d = taosGetTimestampMs() - st; - if (d >= waitDuration && waitDuration >= 0) { - qWarn("%s waiting more than %.2fs, not wait anymore", GET_TASKID(pTaskInfo), waitDuration / 1000.0); - return TSDB_CODE_SUCCESS; - } - } else { // not running now - pTaskInfo->code = rspCode; - taosWUnLockLatch(&pTaskInfo->lock); - return TSDB_CODE_SUCCESS; - } + while (1) { + taosWLockLatch(&pTaskInfo->lock); + if (qTaskIsExecuting(pTaskInfo)) { // let's wait for 100 ms and try again + taosWUnLockLatch(&pTaskInfo->lock); + taosMsleep(100); + } else { // not running now + pTaskInfo->code = rspCode; + taosWUnLockLatch(&pTaskInfo->lock); + return TSDB_CODE_SUCCESS; } } - - return TSDB_CODE_SUCCESS; } bool qTaskIsExecuting(qTaskInfo_t qinfo) { diff --git a/source/libs/executor/src/executorInt.c b/source/libs/executor/src/executorInt.c index 51e7cd896a..a7b1b296ac 100644 --- a/source/libs/executor/src/executorInt.c +++ b/source/libs/executor/src/executorInt.c @@ -80,7 +80,8 @@ static void doApplyScalarCalculation(SOperatorInfo* pOperator, SSDataBlock* p static int32_t doSetInputDataBlock(SExprSupp* pExprSup, SSDataBlock* pBlock, int32_t order, int32_t scanFlag, bool createDummyCol); static void doCopyToSDataBlock(SExecTaskInfo* pTaskInfo, SSDataBlock* pBlock, SExprSupp* pSup, SDiskbasedBuf* pBuf, - SGroupResInfo* pGroupResInfo, int32_t threshold, bool ignoreGroup); + SGroupResInfo* pGroupResInfo, int32_t threshold, bool ignoreGroup, + int64_t minWindowSize); SResultRow* getNewResultRow(SDiskbasedBuf* pResultBuf, int32_t* currentPageId, int32_t interBufSize) { SFilePage* pData = NULL; @@ -846,7 +847,7 @@ _end: } void doCopyToSDataBlock(SExecTaskInfo* pTaskInfo, SSDataBlock* pBlock, SExprSupp* pSup, SDiskbasedBuf* pBuf, - SGroupResInfo* pGroupResInfo, int32_t threshold, bool ignoreGroup) { + SGroupResInfo* pGroupResInfo, int32_t threshold, bool ignoreGroup, int64_t minWindowSize) { int32_t code = TSDB_CODE_SUCCESS; int32_t lino = 0; SExprInfo* pExprInfo = pSup->pExprInfo; @@ -874,6 +875,14 @@ void doCopyToSDataBlock(SExecTaskInfo* pTaskInfo, SSDataBlock* pBlock, SExprSupp releaseBufPage(pBuf, page); continue; } + // skip the window which is less than the windowMinSize + if (pRow->win.ekey - pRow->win.skey < minWindowSize) { + qDebug("skip small window, groupId: %" PRId64 ", windowSize: %" PRId64 ", minWindowSize: %" PRId64, pPos->groupId, + pRow->win.ekey - pRow->win.skey, minWindowSize); + pGroupResInfo->index += 1; + releaseBufPage(pBuf, page); + continue; + } if (!ignoreGroup) { if (pBlock->info.id.groupId == 0) { @@ -937,11 +946,11 @@ void doBuildResultDatablock(SOperatorInfo* pOperator, SOptrBasicInfo* pbInfo, SG pBlock->info.id.groupId = 0; if (!pbInfo->mergeResultBlock) { doCopyToSDataBlock(pTaskInfo, pBlock, &pOperator->exprSupp, pBuf, pGroupResInfo, pOperator->resultInfo.threshold, - false); + false, getMinWindowSize(pOperator)); } else { while (hasRemainResults(pGroupResInfo)) { doCopyToSDataBlock(pTaskInfo, pBlock, &pOperator->exprSupp, pBuf, pGroupResInfo, pOperator->resultInfo.threshold, - true); + true, getMinWindowSize(pOperator)); if (pBlock->info.rows >= pOperator->resultInfo.threshold) { break; } diff --git a/source/libs/executor/src/projectoperator.c b/source/libs/executor/src/projectoperator.c index 0aab1511a4..cb91bae691 100644 --- a/source/libs/executor/src/projectoperator.c +++ b/source/libs/executor/src/projectoperator.c @@ -42,7 +42,9 @@ typedef struct SIndefOperatorInfo { } SIndefOperatorInfo; static int32_t doGenerateSourceData(SOperatorInfo* pOperator); +static SSDataBlock* doProjectOperation1(SOperatorInfo* pOperator); static int32_t doProjectOperation(SOperatorInfo* pOperator, SSDataBlock** pResBlock); +static SSDataBlock* doApplyIndefinitFunction1(SOperatorInfo* pOperator); static int32_t doApplyIndefinitFunction(SOperatorInfo* pOperator, SSDataBlock** pResBlock); static int32_t setRowTsColumnOutputInfo(SqlFunctionCtx* pCtx, int32_t numOfCols, SArray** pResList); static int32_t setFunctionResultOutput(SOperatorInfo* pOperator, SOptrBasicInfo* pInfo, SAggSupporter* pSup, @@ -555,6 +557,12 @@ static void doHandleDataBlock(SOperatorInfo* pOperator, SSDataBlock* pBlock, SOp } } +SSDataBlock* doApplyIndefinitFunction1(SOperatorInfo* pOperator) { + SSDataBlock* pResBlock = NULL; + pOperator->pTaskInfo->code = doApplyIndefinitFunction(pOperator, &pResBlock); + return pResBlock; +} + int32_t doApplyIndefinitFunction(SOperatorInfo* pOperator, SSDataBlock** pResBlock) { QRY_PARAM_CHECK(pResBlock); SIndefOperatorInfo* pIndefInfo = pOperator->info; diff --git a/source/libs/executor/src/streamcountwindowoperator.c b/source/libs/executor/src/streamcountwindowoperator.c index c33abb3d89..63ff2fa92b 100644 --- a/source/libs/executor/src/streamcountwindowoperator.c +++ b/source/libs/executor/src/streamcountwindowoperator.c @@ -395,7 +395,7 @@ static int32_t buildCountResult(SOperatorInfo* pOperator, SSDataBlock** ppRes) { STaskNotifyEventStat* pNotifyEventStat = pTaskInfo->streamInfo.pNotifyEventStat; bool addNotifyEvent = false; addNotifyEvent = BIT_FLAG_TEST_MASK(pTaskInfo->streamInfo.eventTypes, SNOTIFY_EVENT_WINDOW_CLOSE); - doBuildDeleteDataBlock(pOperator, pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); + doBuildDeleteDataBlock(pOperator, pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator, &pInfo->groupResInfo); if (pInfo->pDelRes->info.rows > 0) { printDataBlock(pInfo->pDelRes, getStreamOpName(pOperator->operatorType), GET_TASKID(pTaskInfo)); if (addNotifyEvent) { diff --git a/source/libs/executor/src/streameventwindowoperator.c b/source/libs/executor/src/streameventwindowoperator.c index ab2aa600bb..d258eb08ff 100644 --- a/source/libs/executor/src/streameventwindowoperator.c +++ b/source/libs/executor/src/streameventwindowoperator.c @@ -616,8 +616,8 @@ static int32_t buildEventResult(SOperatorInfo* pOperator, SSDataBlock** ppRes) { SStreamNotifyEventSupp* pNotifySup = &pInfo->basic.notifyEventSup; STaskNotifyEventStat* pNotifyEventStat = pTaskInfo->streamInfo.pNotifyEventStat; bool addNotifyEvent = false; - addNotifyEvent = BIT_FLAG_TEST_MASK(pTaskInfo->streamInfo.eventTypes, SNOTIFY_EVENT_WINDOW_CLOSE); - doBuildDeleteDataBlock(pOperator, pInfo->pSeDeleted, pInfo->pDelRes, &pInfo->pDelIterator); + addNotifyEvent = BIT_FLAG_TEST_MASK(pTaskInfo->streamInfo.eventTypes, SNOTIFY_EVENT_WINDOW_CLOSE); + doBuildDeleteDataBlock(pOperator, pInfo->pSeDeleted, pInfo->pDelRes, &pInfo->pDelIterator, &pInfo->groupResInfo); if (pInfo->pDelRes->info.rows > 0) { printDataBlock(pInfo->pDelRes, getStreamOpName(pOperator->operatorType), GET_TASKID(pTaskInfo)); if (addNotifyEvent) { @@ -1075,6 +1075,8 @@ int32_t createStreamEventAggOperatorInfo(SOperatorInfo* downstream, SPhysiNode* code = nodesCollectColumnsFromNode((SNode*)pEventNode->pEndCond, NULL, COLLECT_COL_TYPE_ALL, &pInfo->pEndCondCols); QUERY_CHECK_CODE(code, lino, _error); + pInfo->trueForLimit = pEventNode->trueForLimit; + *pOptrInfo = pOperator; return TSDB_CODE_SUCCESS; diff --git a/source/libs/executor/src/streamexecutorInt.c b/source/libs/executor/src/streamexecutorInt.c index 635de21b6e..a928521aeb 100644 --- a/source/libs/executor/src/streamexecutorInt.c +++ b/source/libs/executor/src/streamexecutorInt.c @@ -22,13 +22,13 @@ #define NOTIFY_EVENT_NAME_CACHE_LIMIT_MB 16 typedef struct SStreamNotifyEvent { - uint64_t gid; - int64_t eventType; + uint64_t gid; + int64_t eventType; STimeWindow win; - cJSON* pJson; + cJSON* pJson; } SStreamNotifyEvent; -#define NOTIFY_EVENT_KEY_SIZE \ +#define NOTIFY_EVENT_KEY_SIZE \ ((sizeof(((struct SStreamNotifyEvent*)0)->gid) + sizeof(((struct SStreamNotifyEvent*)0)->eventType)) + \ sizeof(((struct SStreamNotifyEvent*)0)->win.skey)) diff --git a/source/libs/executor/src/streamtimewindowoperator.c b/source/libs/executor/src/streamtimewindowoperator.c index d36e38f450..747133379f 100644 --- a/source/libs/executor/src/streamtimewindowoperator.c +++ b/source/libs/executor/src/streamtimewindowoperator.c @@ -485,6 +485,7 @@ void clearGroupResInfo(SGroupResInfo* pGroupResInfo) { taosArrayDestroy(pGroupResInfo->pRows); pGroupResInfo->pRows = NULL; pGroupResInfo->index = 0; + pGroupResInfo->delIndex = 0; } void destroyStreamFinalIntervalOperatorInfo(void* param) { @@ -2887,14 +2888,86 @@ inline int32_t sessionKeyCompareAsc(const void* pKey1, const void* pKey2) { return 0; } -void doBuildDeleteDataBlock(SOperatorInfo* pOp, SSHashObj* pStDeleted, SSDataBlock* pBlock, void** Ite) { +static int32_t appendToDeleteDataBlock(SOperatorInfo* pOp, SSDataBlock *pBlock, SSessionKey *pKey) { int32_t code = TSDB_CODE_SUCCESS; int32_t lino = 0; - SStorageAPI* pAPI = &pOp->pTaskInfo->storageAPI; SExecTaskInfo* pTaskInfo = pOp->pTaskInfo; + QUERY_CHECK_NULL(pBlock, code, lino, _end, TSDB_CODE_INVALID_PARA); + QUERY_CHECK_NULL(pKey, code, lino, _end, TSDB_CODE_INVALID_PARA); + + SColumnInfoData* pStartTsCol = taosArrayGet(pBlock->pDataBlock, START_TS_COLUMN_INDEX); + code = colDataSetVal(pStartTsCol, pBlock->info.rows, (const char*)&pKey->win.skey, false); + QUERY_CHECK_CODE(code, lino, _end); + + SColumnInfoData* pEndTsCol = taosArrayGet(pBlock->pDataBlock, END_TS_COLUMN_INDEX); + code = colDataSetVal(pEndTsCol, pBlock->info.rows, (const char*)&pKey->win.skey, false); + QUERY_CHECK_CODE(code, lino, _end); + + SColumnInfoData* pUidCol = taosArrayGet(pBlock->pDataBlock, UID_COLUMN_INDEX); + colDataSetNULL(pUidCol, pBlock->info.rows); + + SColumnInfoData* pGpCol = taosArrayGet(pBlock->pDataBlock, GROUPID_COLUMN_INDEX); + code = colDataSetVal(pGpCol, pBlock->info.rows, (const char*)&pKey->groupId, false); + QUERY_CHECK_CODE(code, lino, _end); + + SColumnInfoData* pCalStCol = taosArrayGet(pBlock->pDataBlock, CALCULATE_START_TS_COLUMN_INDEX); + colDataSetNULL(pCalStCol, pBlock->info.rows); + + SColumnInfoData* pCalEdCol = taosArrayGet(pBlock->pDataBlock, CALCULATE_END_TS_COLUMN_INDEX); + colDataSetNULL(pCalEdCol, pBlock->info.rows); + + SColumnInfoData* pTableCol = taosArrayGet(pBlock->pDataBlock, TABLE_NAME_COLUMN_INDEX); + if (!pTableCol) { + QUERY_CHECK_CODE(code, lino, _end); + } + + void* tbname = NULL; + int32_t winCode = TSDB_CODE_SUCCESS; + SStorageAPI* pAPI = &pOp->pTaskInfo->storageAPI; + code = + pAPI->stateStore.streamStateGetParName(pOp->pTaskInfo->streamInfo.pState, pKey->groupId, &tbname, false, &winCode); + QUERY_CHECK_CODE(code, lino, _end); + + if (winCode != TSDB_CODE_SUCCESS) { + colDataSetNULL(pTableCol, pBlock->info.rows); + } else { + char parTbName[VARSTR_HEADER_SIZE + TSDB_TABLE_NAME_LEN]; + STR_WITH_MAXSIZE_TO_VARSTR(parTbName, tbname, sizeof(parTbName)); + code = colDataSetVal(pTableCol, pBlock->info.rows, (const char*)parTbName, false); + QUERY_CHECK_CODE(code, lino, _end); + pAPI->stateStore.streamStateFreeVal(tbname); + } + pBlock->info.rows += 1; + +_end: + if (code != TSDB_CODE_SUCCESS) { + qError("%s failed at line %d since %s. task:%s", __func__, lino, tstrerror(code), GET_TASKID(pTaskInfo)); + } + return code; +} + +void doBuildDeleteDataBlock(SOperatorInfo* pOp, SSHashObj* pStDeleted, SSDataBlock* pBlock, void** Ite, + SGroupResInfo* pGroupResInfo) { + int32_t code = TSDB_CODE_SUCCESS; + int32_t lino = 0; + SExecTaskInfo* pTaskInfo = pOp->pTaskInfo; + int64_t minWindowSize = getMinWindowSize(pOp); + int32_t numOfRows = getNumOfTotalRes(pGroupResInfo); + blockDataCleanup(pBlock); int32_t size = tSimpleHashGetSize(pStDeleted); + if (minWindowSize > 0) { + // Add the number of windows that are below the minimum width limit. + for (int32_t i = pGroupResInfo->delIndex; i < numOfRows; ++i) { + SResultWindowInfo* pWinInfo = taosArrayGet(pGroupResInfo->pRows, i); + SRowBuffPos* pPos = pWinInfo->pStatePos; + SSessionKey* pKey = (SSessionKey*)pPos->pKey; + if (pKey->win.ekey - pKey->win.skey < minWindowSize) { + size++; + } + } + } if (size == 0) { return; } @@ -2907,48 +2980,21 @@ void doBuildDeleteDataBlock(SOperatorInfo* pOp, SSHashObj* pStDeleted, SSDataBlo break; } SSessionKey* res = tSimpleHashGetKey(*Ite, NULL); - SColumnInfoData* pStartTsCol = taosArrayGet(pBlock->pDataBlock, START_TS_COLUMN_INDEX); - code = colDataSetVal(pStartTsCol, pBlock->info.rows, (const char*)&res->win.skey, false); + code = appendToDeleteDataBlock(pOp, pBlock, res); QUERY_CHECK_CODE(code, lino, _end); + } - SColumnInfoData* pEndTsCol = taosArrayGet(pBlock->pDataBlock, END_TS_COLUMN_INDEX); - code = colDataSetVal(pEndTsCol, pBlock->info.rows, (const char*)&res->win.skey, false); - QUERY_CHECK_CODE(code, lino, _end); - - SColumnInfoData* pUidCol = taosArrayGet(pBlock->pDataBlock, UID_COLUMN_INDEX); - colDataSetNULL(pUidCol, pBlock->info.rows); - - SColumnInfoData* pGpCol = taosArrayGet(pBlock->pDataBlock, GROUPID_COLUMN_INDEX); - code = colDataSetVal(pGpCol, pBlock->info.rows, (const char*)&res->groupId, false); - QUERY_CHECK_CODE(code, lino, _end); - - SColumnInfoData* pCalStCol = taosArrayGet(pBlock->pDataBlock, CALCULATE_START_TS_COLUMN_INDEX); - colDataSetNULL(pCalStCol, pBlock->info.rows); - - SColumnInfoData* pCalEdCol = taosArrayGet(pBlock->pDataBlock, CALCULATE_END_TS_COLUMN_INDEX); - colDataSetNULL(pCalEdCol, pBlock->info.rows); - - SColumnInfoData* pTableCol = taosArrayGet(pBlock->pDataBlock, TABLE_NAME_COLUMN_INDEX); - if (!pTableCol) { - QUERY_CHECK_CODE(code, lino, _end); + if (minWindowSize > 0) { + for (int32_t i = pGroupResInfo->delIndex; i < numOfRows; ++i) { + SResultWindowInfo* pWinInfo = taosArrayGet(pGroupResInfo->pRows, i); + SRowBuffPos* pPos = pWinInfo->pStatePos; + SSessionKey* pKey = (SSessionKey*)pPos->pKey; + if (pKey->win.ekey - pKey->win.skey < minWindowSize) { + code = appendToDeleteDataBlock(pOp, pBlock, pKey); + QUERY_CHECK_CODE(code, lino, _end); + } } - - void* tbname = NULL; - int32_t winCode = TSDB_CODE_SUCCESS; - code = pAPI->stateStore.streamStateGetParName(pOp->pTaskInfo->streamInfo.pState, res->groupId, &tbname, false, - &winCode); - QUERY_CHECK_CODE(code, lino, _end); - - if (winCode != TSDB_CODE_SUCCESS) { - colDataSetNULL(pTableCol, pBlock->info.rows); - } else { - char parTbName[VARSTR_HEADER_SIZE + TSDB_TABLE_NAME_LEN]; - STR_WITH_MAXSIZE_TO_VARSTR(parTbName, tbname, sizeof(parTbName)); - code = colDataSetVal(pTableCol, pBlock->info.rows, (const char*)parTbName, false); - QUERY_CHECK_CODE(code, lino, _end); - pAPI->stateStore.streamStateFreeVal(tbname); - } - pBlock->info.rows += 1; + pGroupResInfo->delIndex = numOfRows; } _end: @@ -3141,6 +3187,7 @@ void initGroupResInfoFromArrayList(SGroupResInfo* pGroupResInfo, SArray* pArrayL pGroupResInfo->index = 0; pGroupResInfo->pBuf = NULL; pGroupResInfo->freeItem = false; + pGroupResInfo->delIndex = 0; } int32_t buildSessionResultDataBlock(SOperatorInfo* pOperator, void* pState, SSDataBlock* pBlock, SExprSupp* pSup, @@ -3153,6 +3200,7 @@ int32_t buildSessionResultDataBlock(SOperatorInfo* pOperator, void* pState, SSDa int32_t numOfExprs = pSup->numOfExprs; int32_t* rowEntryOffset = pSup->rowEntryInfoOffset; SqlFunctionCtx* pCtx = pSup->pCtx; + int64_t minWindowSize = getMinWindowSize(pOperator); int32_t numOfRows = getNumOfTotalRes(pGroupResInfo); @@ -3193,6 +3241,13 @@ int32_t buildSessionResultDataBlock(SOperatorInfo* pOperator, void* pState, SSDa pGroupResInfo->index += 1; continue; } + // skip the window which is less than the windowMinSize + if (pKey->win.ekey - pKey->win.skey < minWindowSize) { + qDebug("skip small window, groupId: %" PRId64 ", windowSize: %" PRId64 ", minWindowSize: %" PRId64, pKey->groupId, + pKey->win.ekey - pKey->win.skey, minWindowSize); + pGroupResInfo->index += 1; + continue; + } if (pBlock->info.rows + pRow->numOfRows > pBlock->info.capacity) { break; @@ -3286,7 +3341,7 @@ static int32_t buildSessionResult(SOperatorInfo* pOperator, SSDataBlock** ppRes) bool addNotifyEvent = false; addNotifyEvent = IS_NORMAL_SESSION_OP(pOperator) && BIT_FLAG_TEST_MASK(pTaskInfo->streamInfo.eventTypes, SNOTIFY_EVENT_WINDOW_CLOSE); - doBuildDeleteDataBlock(pOperator, pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator); + doBuildDeleteDataBlock(pOperator, pInfo->pStDeleted, pInfo->pDelRes, &pInfo->pDelIterator, &pInfo->groupResInfo); if (pInfo->pDelRes->info.rows > 0) { printDataBlock(pInfo->pDelRes, getStreamOpName(pOperator->operatorType), GET_TASKID(pTaskInfo)); if (addNotifyEvent) { @@ -4928,7 +4983,7 @@ static int32_t buildStateResult(SOperatorInfo* pOperator, SSDataBlock** ppRes) { STaskNotifyEventStat* pNotifyEventStat = pTaskInfo->streamInfo.pNotifyEventStat; bool addNotifyEvent = false; addNotifyEvent = BIT_FLAG_TEST_MASK(pTaskInfo->streamInfo.eventTypes, SNOTIFY_EVENT_WINDOW_CLOSE); - doBuildDeleteDataBlock(pOperator, pInfo->pSeDeleted, pInfo->pDelRes, &pInfo->pDelIterator); + doBuildDeleteDataBlock(pOperator, pInfo->pSeDeleted, pInfo->pDelRes, &pInfo->pDelIterator, &pInfo->groupResInfo); if (pInfo->pDelRes->info.rows > 0) { printDataBlock(pInfo->pDelRes, getStreamOpName(pOperator->operatorType), GET_TASKID(pTaskInfo)); if (addNotifyEvent) { @@ -5363,6 +5418,8 @@ int32_t createStreamStateAggOperatorInfo(SOperatorInfo* downstream, SPhysiNode* code = appendDownstream(pOperator, &downstream, 1); QUERY_CHECK_CODE(code, lino, _error); + pInfo->trueForLimit = pStateNode->trueForLimit; + *pOptrInfo = pOperator; return TSDB_CODE_SUCCESS; diff --git a/source/libs/executor/src/timesliceoperator.c b/source/libs/executor/src/timesliceoperator.c index 49fd557fe3..96ad51149c 100644 --- a/source/libs/executor/src/timesliceoperator.c +++ b/source/libs/executor/src/timesliceoperator.c @@ -1390,3 +1390,22 @@ void destroyTimeSliceOperatorInfo(void* param) { } taosMemoryFreeClear(param); } + +int64_t getMinWindowSize(struct SOperatorInfo* pOperator) { + if (pOperator == NULL) { + return 0; + } + + switch (pOperator->operatorType) { + case QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE: + return ((SStateWindowOperatorInfo*)pOperator->info)->trueForLimit; + case QUERY_NODE_PHYSICAL_PLAN_STREAM_STATE: + return ((SStreamStateAggOperatorInfo*)pOperator->info)->trueForLimit; + case QUERY_NODE_PHYSICAL_PLAN_MERGE_EVENT: + return ((SEventWindowOperatorInfo*)pOperator->info)->trueForLimit; + case QUERY_NODE_PHYSICAL_PLAN_STREAM_EVENT: + return ((SStreamEventAggOperatorInfo*)pOperator->info)->trueForLimit; + default: + return 0; + } +} diff --git a/source/libs/executor/src/timewindowoperator.c b/source/libs/executor/src/timewindowoperator.c index 71c71a547e..96a3b02464 100644 --- a/source/libs/executor/src/timewindowoperator.c +++ b/source/libs/executor/src/timewindowoperator.c @@ -27,35 +27,6 @@ #include "tlog.h" #include "ttime.h" -typedef struct SSessionAggOperatorInfo { - SOptrBasicInfo binfo; - SAggSupporter aggSup; - SExprSupp scalarSupp; // supporter for perform scalar function - SGroupResInfo groupResInfo; - SWindowRowsSup winSup; - bool reptScan; // next round scan - int64_t gap; // session window gap - int32_t tsSlotId; // primary timestamp slot id - STimeWindowAggSupp twAggSup; - SOperatorInfo* pOperator; - bool cleanGroupResInfo; -} SSessionAggOperatorInfo; - -typedef struct SStateWindowOperatorInfo { - SOptrBasicInfo binfo; - SAggSupporter aggSup; - SExprSupp scalarSup; - SGroupResInfo groupResInfo; - SWindowRowsSup winSup; - SColumn stateCol; // start row index - bool hasKey; - SStateKeys stateKey; - int32_t tsSlotId; // primary timestamp column slot id - STimeWindowAggSupp twAggSup; - SOperatorInfo* pOperator; - bool cleanGroupResInfo; -} SStateWindowOperatorInfo; - typedef enum SResultTsInterpType { RESULT_ROW_START_INTERP = 1, RESULT_ROW_END_INTERP = 2, @@ -1743,6 +1714,7 @@ int32_t createStatewindowOperatorInfo(SOperatorInfo* downstream, SStateWinodwPhy pInfo->tsSlotId = tsSlotId; pInfo->pOperator = pOperator; pInfo->cleanGroupResInfo = false; + pInfo->trueForLimit = pStateNode->trueForLimit; setOperatorInfo(pOperator, "StateWindowOperator", QUERY_NODE_PHYSICAL_PLAN_MERGE_STATE, true, OP_NOT_OPENED, pInfo, pTaskInfo); pOperator->fpSet = createOperatorFpSet(openStateWindowAggOptr, doStateWindowAggNext, NULL, destroyStateWindowOperatorInfo, diff --git a/source/libs/function/inc/functionMgtInt.h b/source/libs/function/inc/functionMgtInt.h index e10581beb6..d676d4b728 100644 --- a/source/libs/function/inc/functionMgtInt.h +++ b/source/libs/function/inc/functionMgtInt.h @@ -59,6 +59,7 @@ extern "C" { #define FUNC_MGT_COUNT_LIKE_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(30) // funcs that should also return 0 when no rows found #define FUNC_MGT_PROCESS_BY_ROW FUNC_MGT_FUNC_CLASSIFICATION_MASK(31) #define FUNC_MGT_FORECAST_PC_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(32) +#define FUNC_MGT_SELECT_COLS_FUNC FUNC_MGT_FUNC_CLASSIFICATION_MASK(33) #define FUNC_MGT_TEST_MASK(val, mask) (((val) & (mask)) != 0) diff --git a/source/libs/function/src/builtins.c b/source/libs/function/src/builtins.c index b42d739b40..5a6b042cd0 100644 --- a/source/libs/function/src/builtins.c +++ b/source/libs/function/src/builtins.c @@ -1702,6 +1702,10 @@ static int32_t translateOutVarchar(SFunctionNode* pFunc, char* pErrBuf, int32_t return TSDB_CODE_SUCCESS; } +static int32_t invalidColsFunction(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; +} + static int32_t translateHistogramImpl(SFunctionNode* pFunc, char* pErrBuf, int32_t len) { FUNC_ERR_RET(validateParam(pFunc, pErrBuf, len)); int32_t numOfParams = LIST_LENGTH(pFunc->pParameterList); @@ -2737,7 +2741,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .translateFunc = translateOutFirstIn, .dynDataRequiredFunc = firstDynDataReq, .getEnvFunc = getFirstLastFuncEnv, - .initFunc = functionSetup, + .initFunc = firstLastFunctionSetup, .processFunc = firstFunction, .sprocessFunc = firstLastScalarFunction, .finalizeFunc = firstLastFinalize, @@ -4232,7 +4236,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "_group_key", .type = FUNCTION_TYPE_GROUP_KEY, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_KEEP_ORDER_FUNC | FUNC_MGT_SKIP_SCAN_CHECK_FUNC, + .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_KEEP_ORDER_FUNC | FUNC_MGT_SKIP_SCAN_CHECK_FUNC, .translateFunc = translateGroupKey, .getEnvFunc = getGroupKeyFuncEnv, .initFunc = functionSetup, @@ -4952,7 +4956,7 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { { .name = "_group_const_value", .type = FUNCTION_TYPE_GROUP_CONST_VALUE, - .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_SELECT_FUNC | FUNC_MGT_KEEP_ORDER_FUNC, + .classification = FUNC_MGT_AGG_FUNC | FUNC_MGT_KEEP_ORDER_FUNC, .parameters = {.minParamNum = 0, .maxParamNum = 0, .paramInfoPattern = 0, @@ -5647,7 +5651,11 @@ const SBuiltinFuncDefinition funcMgtBuiltins[] = { .paramInfoPattern = 0, .outputParaInfo = {.validDataType = FUNC_PARAM_SUPPORT_VARCHAR_TYPE}}, .translateFunc = translateOutVarchar, - } + }, + { + .name = "cols", + .translateFunc = invalidColsFunction, + }, }; // clang-format on diff --git a/source/libs/function/src/builtinsimpl.c b/source/libs/function/src/builtinsimpl.c index efe16ce662..91be791dff 100644 --- a/source/libs/function/src/builtinsimpl.c +++ b/source/libs/function/src/builtinsimpl.c @@ -912,14 +912,13 @@ int32_t minmaxFunctionFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { if (pEntryInfo->numOfRes > 0) { code = setSelectivityValue(pCtx, pBlock, &pRes->tuplePos, currentRow); } else { - code = setSelectivityValue(pCtx, pBlock, &pRes->nullTuplePos, currentRow); + code = setNullSelectivityValue(pCtx, pBlock, currentRow); } } return code; } -#ifdef BUILD_NO_CALL int32_t setNullSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, int32_t rowIndex) { if (pCtx->subsidiaries.num <= 0) { return TSDB_CODE_SUCCESS; @@ -930,12 +929,14 @@ int32_t setNullSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, int32 int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; SColumnInfoData* pDstCol = taosArrayGet(pBlock->pDataBlock, dstSlotId); + if (NULL == pDstCol) { + return terrno; + } colDataSetNULL(pDstCol, rowIndex); } return TSDB_CODE_SUCCESS; } -#endif int32_t setSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, const STuplePos* pTuplePos, int32_t rowIndex) { if (pCtx->subsidiaries.num <= 0) { @@ -961,20 +962,14 @@ int32_t setSelectivityValue(SqlFunctionCtx* pCtx, SSDataBlock* pBlock, const STu SqlFunctionCtx* pc = pCtx->subsidiaries.pCtx[j]; int32_t dstSlotId = pc->pExpr->base.resSchema.slotId; - // group_key function has its own process function - // do not process there - if (fmIsGroupKeyFunc(pc->functionId)) { - continue; - } - SColumnInfoData* pDstCol = taosArrayGet(pBlock->pDataBlock, dstSlotId); if (NULL == pDstCol) { - return TSDB_CODE_OUT_OF_RANGE; + return terrno; } if (nullList[j]) { colDataSetNULL(pDstCol, rowIndex); } else { - code = colDataSetVal(pDstCol, rowIndex, pStart, false); + code = colDataSetValOrCover(pDstCol, rowIndex, pStart, false); if (TSDB_CODE_SUCCESS != code) { return code; } @@ -2431,8 +2426,6 @@ int32_t firstLastFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResIn } SFirstLastRes* pRes = GET_ROWCELL_INTERBUF(pResInfo); - SInputColumnInfoData* pInput = &pCtx->input; - pRes->nullTupleSaved = false; pRes->nullTuplePos.pageId = -1; return TSDB_CODE_SUCCESS; @@ -2464,10 +2457,10 @@ static int32_t firstlastSaveTupleData(const SSDataBlock* pSrcBlock, int32_t rowI } if (!pInfo->hasResult) { - code = saveTupleData(pCtx, rowIndex, pSrcBlock, noElements ? &pInfo->nullTuplePos : &pInfo->pos); - } else { + code = saveTupleData(pCtx, rowIndex, pSrcBlock, &pInfo->pos); + } else if (!noElements) { code = updateTupleData(pCtx, rowIndex, pSrcBlock, &pInfo->pos); - } + } else { } // dothing return code; } @@ -2538,7 +2531,7 @@ int32_t firstFunction(SqlFunctionCtx* pCtx) { if (pInput->colDataSMAIsSet && (pInput->pColumnDataAgg[0]->numOfNull == pInput->totalRows) && pInputCol->hasNull == true) { // save selectivity value for column consisted of all null values - int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, !pInfo->nullTupleSaved); + int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, true); if (code != TSDB_CODE_SUCCESS) { return code; } @@ -2633,7 +2626,7 @@ int32_t firstFunction(SqlFunctionCtx* pCtx) { if (numOfElems == 0) { // save selectivity value for column consisted of all null values - int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, !pInfo->nullTupleSaved); + int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, true); if (code != TSDB_CODE_SUCCESS) { return code; } @@ -2654,11 +2647,11 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { int32_t type = pInputCol->info.type; int32_t bytes = pInputCol->info.bytes; - pInfo->bytes = bytes; if (IS_NULL_TYPE(type)) { return TSDB_CODE_SUCCESS; } + pInfo->bytes = bytes; SColumnInfoData* pkCol = pInput->pPrimaryKey; pInfo->pkType = -1; @@ -2673,7 +2666,7 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { if (pInput->colDataSMAIsSet && (pInput->pColumnDataAgg[0]->numOfNull == pInput->totalRows) && pInputCol->hasNull == true) { // save selectivity value for column consisted of all null values - int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, !pInfo->nullTupleSaved); + int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, true); if (code != TSDB_CODE_SUCCESS) { return code; } @@ -2768,7 +2761,7 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { if (pResInfo->numOfRes == 0 || pInfo->ts < cts) { char* data = colDataGetData(pInputCol, chosen); - int32_t code = doSaveCurrentVal(pCtx, i, cts, NULL, type, data); + int32_t code = doSaveCurrentVal(pCtx, chosen, cts, NULL, type, data); if (code != TSDB_CODE_SUCCESS) { return code; } @@ -2816,7 +2809,7 @@ int32_t lastFunction(SqlFunctionCtx* pCtx) { // save selectivity value for column consisted of all null values if (numOfElems == 0) { - int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, !pInfo->nullTupleSaved); + int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, true); if (code != TSDB_CODE_SUCCESS) { return code; } @@ -2865,7 +2858,7 @@ static bool firstLastTransferInfoImpl(SFirstLastRes* pInput, SFirstLastRes* pOut static int32_t firstLastTransferInfo(SqlFunctionCtx* pCtx, SFirstLastRes* pInput, SFirstLastRes* pOutput, bool isFirst, int32_t rowIndex) { if (firstLastTransferInfoImpl(pInput, pOutput, isFirst)) { - int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, rowIndex, pCtx, pOutput, pOutput->nullTupleSaved); + int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, rowIndex, pCtx, pOutput, false); if (TSDB_CODE_SUCCESS != code) { return code; } @@ -2914,7 +2907,7 @@ static int32_t firstLastFunctionMergeImpl(SqlFunctionCtx* pCtx, bool isFirstQuer } if (numOfElems == 0) { - int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, !pInfo->nullTupleSaved); + int32_t code = firstlastSaveTupleData(pCtx->pSrcBlock, pInput->startRowIndex, pCtx, pInfo, true); if (code != TSDB_CODE_SUCCESS) { return code; } @@ -2944,7 +2937,7 @@ int32_t firstLastFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { if (pResInfo->isNullRes) { colDataSetNULL(pCol, pBlock->info.rows); - return setSelectivityValue(pCtx, pBlock, &pRes->nullTuplePos, pBlock->info.rows); + return setNullSelectivityValue(pCtx, pBlock, pBlock->info.rows); } code = colDataSetVal(pCol, pBlock->info.rows, pRes->buf, pRes->isNull || pResInfo->isNullRes); if (TSDB_CODE_SUCCESS != code) { @@ -2983,7 +2976,7 @@ int32_t firstLastPartialFinalize(SqlFunctionCtx* pCtx, SSDataBlock* pBlock) { if (pEntryInfo->numOfRes == 0) { colDataSetNULL(pCol, pBlock->info.rows); - code = setSelectivityValue(pCtx, pBlock, &pRes->nullTuplePos, pBlock->info.rows); + code = setNullSelectivityValue(pCtx, pBlock, pBlock->info.rows); } else { code = colDataSetVal(pCol, pBlock->info.rows, res, false); if (TSDB_CODE_SUCCESS != code) { diff --git a/source/libs/function/src/functionMgt.c b/source/libs/function/src/functionMgt.c index 4fb3aab49d..03935bf1a0 100644 --- a/source/libs/function/src/functionMgt.c +++ b/source/libs/function/src/functionMgt.c @@ -173,6 +173,8 @@ bool fmIsScalarFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC bool fmIsVectorFunc(int32_t funcId) { return !fmIsScalarFunc(funcId) && !fmIsPseudoColumnFunc(funcId); } +bool fmIsSelectColsFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC_MGT_SELECT_COLS_FUNC); } + bool fmIsSelectFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC_MGT_SELECT_FUNC); } bool fmIsTimelineFunc(int32_t funcId) { return isSpecificClassifyFunc(funcId, FUNC_MGT_TIMELINE_FUNC); } @@ -441,6 +443,8 @@ int32_t createFunctionWithSrcFunc(const char* pName, const SFunctionNode* pSrcFu return code; } resetOutputChangedFunc(*ppFunc, pSrcFunc); + (*ppFunc)->node.relatedTo = pSrcFunc->node.relatedTo; + (*ppFunc)->node.bindExprID = pSrcFunc->node.bindExprID; return code; } diff --git a/source/libs/nodes/src/nodesCloneFuncs.c b/source/libs/nodes/src/nodesCloneFuncs.c index 0e7a719c2c..fcb39c4284 100644 --- a/source/libs/nodes/src/nodesCloneFuncs.c +++ b/source/libs/nodes/src/nodesCloneFuncs.c @@ -106,6 +106,8 @@ static int32_t exprNodeCopy(const SExprNode* pSrc, SExprNode* pDst) { COPY_SCALAR_FIELD(asParam); COPY_SCALAR_FIELD(asPosition); COPY_SCALAR_FIELD(projIdx); + COPY_SCALAR_FIELD(relatedTo); + COPY_SCALAR_FIELD(bindExprID); return TSDB_CODE_SUCCESS; } @@ -354,6 +356,7 @@ static int32_t limitNodeCopy(const SLimitNode* pSrc, SLimitNode* pDst) { static int32_t stateWindowNodeCopy(const SStateWindowNode* pSrc, SStateWindowNode* pDst) { CLONE_NODE_FIELD(pCol); CLONE_NODE_FIELD(pExpr); + CLONE_NODE_FIELD(pTrueForLimit); return TSDB_CODE_SUCCESS; } @@ -361,6 +364,7 @@ static int32_t eventWindowNodeCopy(const SEventWindowNode* pSrc, SEventWindowNod CLONE_NODE_FIELD(pCol); CLONE_NODE_FIELD(pStartCond); CLONE_NODE_FIELD(pEndCond); + CLONE_NODE_FIELD(pTrueForLimit); return TSDB_CODE_SUCCESS; } @@ -627,6 +631,7 @@ static int32_t logicWindowCopy(const SWindowLogicNode* pSrc, SWindowLogicNode* p CLONE_NODE_FIELD(pStateExpr); CLONE_NODE_FIELD(pStartCond); CLONE_NODE_FIELD(pEndCond); + COPY_SCALAR_FIELD(trueForLimit); COPY_SCALAR_FIELD(triggerType); COPY_SCALAR_FIELD(watermark); COPY_SCALAR_FIELD(deleteMark); diff --git a/source/libs/nodes/src/nodesCodeFuncs.c b/source/libs/nodes/src/nodesCodeFuncs.c index 6966f6a463..e906c79a08 100644 --- a/source/libs/nodes/src/nodesCodeFuncs.c +++ b/source/libs/nodes/src/nodesCodeFuncs.c @@ -3127,6 +3127,7 @@ static int32_t jsonToPhysiSessionWindowNode(const SJson* pJson, void* pObj) { } static const char* jkStateWindowPhysiPlanStateKey = "StateKey"; +static const char* jkStateWindowPhysiPlanTrueForLimit = "TrueForLimit"; static int32_t physiStateWindowNodeToJson(const void* pObj, SJson* pJson) { const SStateWinodwPhysiNode* pNode = (const SStateWinodwPhysiNode*)pObj; @@ -3135,6 +3136,9 @@ static int32_t physiStateWindowNodeToJson(const void* pObj, SJson* pJson) { if (TSDB_CODE_SUCCESS == code) { code = tjsonAddObject(pJson, jkStateWindowPhysiPlanStateKey, nodeToJson, pNode->pStateKey); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkStateWindowPhysiPlanTrueForLimit, pNode->trueForLimit); + } return code; } @@ -3146,12 +3150,16 @@ static int32_t jsonToPhysiStateWindowNode(const SJson* pJson, void* pObj) { if (TSDB_CODE_SUCCESS == code) { code = jsonToNodeObject(pJson, jkStateWindowPhysiPlanStateKey, &pNode->pStateKey); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkStateWindowPhysiPlanTrueForLimit, &pNode->trueForLimit); + } return code; } static const char* jkEventWindowPhysiPlanStartCond = "StartCond"; static const char* jkEventWindowPhysiPlanEndCond = "EndCond"; +static const char* jkEventWindowPhysiPlanTrueForLimit = "TrueForLimit"; static int32_t physiEventWindowNodeToJson(const void* pObj, SJson* pJson) { const SEventWinodwPhysiNode* pNode = (const SEventWinodwPhysiNode*)pObj; @@ -3163,6 +3171,9 @@ static int32_t physiEventWindowNodeToJson(const void* pObj, SJson* pJson) { if (TSDB_CODE_SUCCESS == code) { code = tjsonAddObject(pJson, jkEventWindowPhysiPlanEndCond, nodeToJson, pNode->pEndCond); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddIntegerToObject(pJson, jkEventWindowPhysiPlanTrueForLimit, pNode->trueForLimit); + } return code; } @@ -3177,6 +3188,9 @@ static int32_t jsonToPhysiEventWindowNode(const SJson* pJson, void* pObj) { if (TSDB_CODE_SUCCESS == code) { code = jsonToNodeObject(pJson, jkEventWindowPhysiPlanEndCond, &pNode->pEndCond); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonGetBigIntValue(pJson, jkEventWindowPhysiPlanTrueForLimit, &pNode->trueForLimit); + } return code; } @@ -4960,6 +4974,7 @@ static int32_t jsonToLimitNode(const SJson* pJson, void* pObj) { static const char* jkStateWindowCol = "StateWindowCol"; static const char* jkStateWindowExpr = "StateWindowExpr"; +static const char* jkStateWindowTrueForLimit = "TrueForLimit"; static int32_t stateWindowNodeToJson(const void* pObj, SJson* pJson) { const SStateWindowNode* pNode = (const SStateWindowNode*)pObj; @@ -4967,6 +4982,9 @@ static int32_t stateWindowNodeToJson(const void* pObj, SJson* pJson) { if (TSDB_CODE_SUCCESS == code) { code = tjsonAddObject(pJson, jkStateWindowExpr, nodeToJson, pNode->pExpr); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddObject(pJson, jkStateWindowTrueForLimit, nodeToJson, pNode->pTrueForLimit); + } return code; } @@ -4977,6 +4995,9 @@ static int32_t jsonToStateWindowNode(const SJson* pJson, void* pObj) { if (TSDB_CODE_SUCCESS == code) { code = jsonToNodeObject(pJson, jkStateWindowExpr, (SNode**)&pNode->pExpr); } + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeObject(pJson, jkStateWindowTrueForLimit, (SNode**)&pNode->pTrueForLimit); + } return code; } @@ -5006,6 +5027,7 @@ static int32_t jsonToSessionWindowNode(const SJson* pJson, void* pObj) { static const char* jkEventWindowTsPrimaryKey = "TsPrimaryKey"; static const char* jkEventWindowStartCond = "StartCond"; static const char* jkEventWindowEndCond = "EndCond"; +static const char* jkEventWindowTrueForLimit = "TrueForLimit"; static int32_t eventWindowNodeToJson(const void* pObj, SJson* pJson) { const SEventWindowNode* pNode = (const SEventWindowNode*)pObj; @@ -5017,6 +5039,9 @@ static int32_t eventWindowNodeToJson(const void* pObj, SJson* pJson) { if (TSDB_CODE_SUCCESS == code) { code = tjsonAddObject(pJson, jkEventWindowEndCond, nodeToJson, pNode->pEndCond); } + if (TSDB_CODE_SUCCESS == code) { + code = tjsonAddObject(pJson, jkEventWindowTrueForLimit, nodeToJson, pNode->pTrueForLimit); + } return code; } @@ -5030,6 +5055,9 @@ static int32_t jsonToEventWindowNode(const SJson* pJson, void* pObj) { if (TSDB_CODE_SUCCESS == code) { code = jsonToNodeObject(pJson, jkEventWindowEndCond, &pNode->pEndCond); } + if (TSDB_CODE_SUCCESS == code) { + code = jsonToNodeObject(pJson, jkEventWindowTrueForLimit, &pNode->pTrueForLimit); + } return code; } diff --git a/source/libs/nodes/src/nodesEqualFuncs.c b/source/libs/nodes/src/nodesEqualFuncs.c index 891843761a..fa15c9f2fa 100644 --- a/source/libs/nodes/src/nodesEqualFuncs.c +++ b/source/libs/nodes/src/nodesEqualFuncs.c @@ -13,6 +13,7 @@ * along with this program. If not, see . */ +#include "functionMgt.h" #include "querynodes.h" #define COMPARE_SCALAR_FIELD(fldname) \ @@ -137,6 +138,15 @@ static bool functionNodeEqual(const SFunctionNode* a, const SFunctionNode* b) { COMPARE_SCALAR_FIELD(funcId); COMPARE_STRING_FIELD(functionName); COMPARE_NODE_LIST_FIELD(pParameterList); + if (a->funcType == FUNCTION_TYPE_SELECT_VALUE) { + if ((a->node.relatedTo != b->node.relatedTo)) return false; + } else { + // select cols(cols(first(c0), ts), first(c0) from meters; + if ((a->node.bindExprID != b->node.bindExprID)) { + return false; + } + } + return true; } diff --git a/source/libs/nodes/src/nodesMsgFuncs.c b/source/libs/nodes/src/nodesMsgFuncs.c index 1becd07aba..aef29c6ec6 100644 --- a/source/libs/nodes/src/nodesMsgFuncs.c +++ b/source/libs/nodes/src/nodesMsgFuncs.c @@ -664,11 +664,18 @@ static int32_t msgToDataType(STlvDecoder* pDecoder, void* pObj) { return code; } -enum { EXPR_CODE_RES_TYPE = 1 }; +enum { EXPR_CODE_RES_TYPE = 1, EXPR_CODE_BIND_TUPLE_FUNC_IDX, EXPR_CODE_TUPLE_FUNC_IDX }; static int32_t exprNodeToMsg(const void* pObj, STlvEncoder* pEncoder) { const SExprNode* pNode = (const SExprNode*)pObj; - return tlvEncodeObj(pEncoder, EXPR_CODE_RES_TYPE, dataTypeToMsg, &pNode->resType); + int32_t code = tlvEncodeObj(pEncoder, EXPR_CODE_RES_TYPE, dataTypeToMsg, &pNode->resType); + if (TSDB_CODE_SUCCESS == code) { + code = tlvEncodeI32(pEncoder, EXPR_CODE_BIND_TUPLE_FUNC_IDX, pNode->relatedTo); + } + if (TSDB_CODE_SUCCESS == code) { + code = tlvEncodeI32(pEncoder, EXPR_CODE_TUPLE_FUNC_IDX, pNode->bindExprID); + } + return code; } static int32_t msgToExprNode(STlvDecoder* pDecoder, void* pObj) { @@ -681,6 +688,12 @@ static int32_t msgToExprNode(STlvDecoder* pDecoder, void* pObj) { case EXPR_CODE_RES_TYPE: code = tlvDecodeObjFromTlv(pTlv, msgToDataType, &pNode->resType); break; + case EXPR_CODE_BIND_TUPLE_FUNC_IDX: + code = tlvDecodeI32(pTlv, &pNode->relatedTo); + break; + case EXPR_CODE_TUPLE_FUNC_IDX: + code = tlvDecodeI32(pTlv, &pNode->bindExprID); + break; default: break; } @@ -695,6 +708,12 @@ static int32_t columnNodeInlineToMsg(const void* pObj, STlvEncoder* pEncoder) { const SColumnNode* pNode = (const SColumnNode*)pObj; int32_t code = dataTypeInlineToMsg(&pNode->node.resType, pEncoder); + if (TSDB_CODE_SUCCESS == code) { + code = tlvEncodeValueI32(pEncoder, pNode->node.relatedTo); + } + if (TSDB_CODE_SUCCESS == code) { + code = tlvEncodeValueI32(pEncoder, pNode->node.bindExprID); + } if (TSDB_CODE_SUCCESS == code) { code = tlvEncodeValueU64(pEncoder, pNode->tableId); } @@ -745,6 +764,12 @@ static int32_t msgToColumnNodeInline(STlvDecoder* pDecoder, void* pObj) { SColumnNode* pNode = (SColumnNode*)pObj; int32_t code = msgToDataTypeInline(pDecoder, &pNode->node.resType); + if (TSDB_CODE_SUCCESS == code) { + code = tlvDecodeValueI32(pDecoder, &pNode->node.relatedTo); + } + if (TSDB_CODE_SUCCESS == code) { + code = tlvDecodeValueI32(pDecoder, &pNode->node.bindExprID); + } if (TSDB_CODE_SUCCESS == code) { code = tlvDecodeValueU64(pDecoder, &pNode->tableId); } @@ -3443,7 +3468,7 @@ static int32_t msgToPhysiSessionWindowNode(STlvDecoder* pDecoder, void* pObj) { return code; } -enum { PHY_STATE_CODE_WINDOW = 1, PHY_STATE_CODE_KEY }; +enum { PHY_STATE_CODE_WINDOW = 1, PHY_STATE_CODE_KEY, PHY_STATE_CODE_TRUE_FOR_LIMIT }; static int32_t physiStateWindowNodeToMsg(const void* pObj, STlvEncoder* pEncoder) { const SStateWinodwPhysiNode* pNode = (const SStateWinodwPhysiNode*)pObj; @@ -3452,6 +3477,9 @@ static int32_t physiStateWindowNodeToMsg(const void* pObj, STlvEncoder* pEncoder if (TSDB_CODE_SUCCESS == code) { code = tlvEncodeObj(pEncoder, PHY_STATE_CODE_KEY, nodeToMsg, pNode->pStateKey); } + if (TSDB_CODE_SUCCESS == code) { + code = tlvEncodeI64(pEncoder, PHY_STATE_CODE_TRUE_FOR_LIMIT, pNode->trueForLimit); + } return code; } @@ -3469,6 +3497,9 @@ static int32_t msgToPhysiStateWindowNode(STlvDecoder* pDecoder, void* pObj) { case PHY_STATE_CODE_KEY: code = msgToNodeFromTlv(pTlv, (void**)&pNode->pStateKey); break; + case PHY_STATE_CODE_TRUE_FOR_LIMIT: + code = tlvDecodeI64(pTlv, &pNode->trueForLimit); + break; default: break; } @@ -3477,7 +3508,7 @@ static int32_t msgToPhysiStateWindowNode(STlvDecoder* pDecoder, void* pObj) { return code; } -enum { PHY_EVENT_CODE_WINDOW = 1, PHY_EVENT_CODE_START_COND, PHY_EVENT_CODE_END_COND }; +enum { PHY_EVENT_CODE_WINDOW = 1, PHY_EVENT_CODE_START_COND, PHY_EVENT_CODE_END_COND, PHY_EVENT_CODE_TRUE_FOR_LIMIT }; static int32_t physiEventWindowNodeToMsg(const void* pObj, STlvEncoder* pEncoder) { const SEventWinodwPhysiNode* pNode = (const SEventWinodwPhysiNode*)pObj; @@ -3489,6 +3520,9 @@ static int32_t physiEventWindowNodeToMsg(const void* pObj, STlvEncoder* pEncoder if (TSDB_CODE_SUCCESS == code) { code = tlvEncodeObj(pEncoder, PHY_EVENT_CODE_END_COND, nodeToMsg, pNode->pEndCond); } + if (TSDB_CODE_SUCCESS == code) { + code = tlvEncodeI64(pEncoder, PHY_EVENT_CODE_TRUE_FOR_LIMIT, pNode->trueForLimit); + } return code; } @@ -3509,6 +3543,9 @@ static int32_t msgToPhysiEventWindowNode(STlvDecoder* pDecoder, void* pObj) { case PHY_EVENT_CODE_END_COND: code = msgToNodeFromTlv(pTlv, (void**)&pNode->pEndCond); break; + case PHY_EVENT_CODE_TRUE_FOR_LIMIT: + code = tlvDecodeI64(pTlv, &pNode->trueForLimit); + break; default: break; } diff --git a/source/libs/nodes/src/nodesTraverseFuncs.c b/source/libs/nodes/src/nodesTraverseFuncs.c index f3f7395a37..a269a6db0f 100644 --- a/source/libs/nodes/src/nodesTraverseFuncs.c +++ b/source/libs/nodes/src/nodesTraverseFuncs.c @@ -102,6 +102,9 @@ static EDealRes dispatchExpr(SNode* pNode, ETraversalOrder order, FNodeWalker wa if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { res = walkExpr(pState->pCol, order, walker, pContext); } + if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { + res = walkExpr(pState->pTrueForLimit, order, walker, pContext); + } break; } case QUERY_NODE_SESSION_WINDOW: { @@ -174,6 +177,9 @@ static EDealRes dispatchExpr(SNode* pNode, ETraversalOrder order, FNodeWalker wa if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { res = walkExpr(pEvent->pEndCond, order, walker, pContext); } + if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { + res = walkExpr(pEvent->pTrueForLimit, order, walker, pContext); + } break; } case QUERY_NODE_COUNT_WINDOW: { @@ -313,6 +319,9 @@ static EDealRes rewriteExpr(SNode** pRawNode, ETraversalOrder order, FNodeRewrit if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { res = rewriteExpr(&pState->pCol, order, rewriter, pContext); } + if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { + res = rewriteExpr(&pState->pTrueForLimit, order, rewriter, pContext); + } break; } case QUERY_NODE_SESSION_WINDOW: { @@ -385,6 +394,9 @@ static EDealRes rewriteExpr(SNode** pRawNode, ETraversalOrder order, FNodeRewrit if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { res = rewriteExpr(&pEvent->pEndCond, order, rewriter, pContext); } + if (DEAL_RES_ERROR != res && DEAL_RES_END != res) { + res = rewriteExpr(&pEvent->pTrueForLimit, order, rewriter, pContext); + } break; } case QUERY_NODE_WINDOW_OFFSET: { @@ -475,6 +487,7 @@ void nodesWalkSelectStmtImpl(SSelectStmt* pSelect, ESqlClause clause, FNodeWalke nodesWalkExprs(pSelect->pOrderByList, walker, pContext); case SQL_CLAUSE_ORDER_BY: nodesWalkExprs(pSelect->pProjectionList, walker, pContext); + nodesWalkExprs(pSelect->pProjectionBindList, walker, pContext); default: break; } @@ -515,6 +528,7 @@ void nodesRewriteSelectStmt(SSelectStmt* pSelect, ESqlClause clause, FNodeRewrit nodesRewriteExprs(pSelect->pOrderByList, rewriter, pContext); case SQL_CLAUSE_ORDER_BY: nodesRewriteExprs(pSelect->pProjectionList, rewriter, pContext); + nodesRewriteExprs(pSelect->pProjectionBindList, rewriter, pContext); default: break; } diff --git a/source/libs/nodes/src/nodesUtilFuncs.c b/source/libs/nodes/src/nodesUtilFuncs.c index 9473e75642..874d6df811 100644 --- a/source/libs/nodes/src/nodesUtilFuncs.c +++ b/source/libs/nodes/src/nodesUtilFuncs.c @@ -15,6 +15,7 @@ #include "cmdnodes.h" #include "functionMgt.h" +#include "nodes.h" #include "nodesUtil.h" #include "plannodes.h" #include "querynodes.h" @@ -1125,6 +1126,7 @@ void nodesDestroyNode(SNode* pNode) { SStateWindowNode* pState = (SStateWindowNode*)pNode; nodesDestroyNode(pState->pCol); nodesDestroyNode(pState->pExpr); + nodesDestroyNode(pState->pTrueForLimit); break; } case QUERY_NODE_SESSION_WINDOW: { @@ -1239,6 +1241,7 @@ void nodesDestroyNode(SNode* pNode) { nodesDestroyNode(pEvent->pCol); nodesDestroyNode(pEvent->pStartCond); nodesDestroyNode(pEvent->pEndCond); + nodesDestroyNode(pEvent->pTrueForLimit); break; } case QUERY_NODE_COUNT_WINDOW: { @@ -1292,6 +1295,7 @@ void nodesDestroyNode(SNode* pNode) { case QUERY_NODE_SELECT_STMT: { SSelectStmt* pStmt = (SSelectStmt*)pNode; nodesDestroyList(pStmt->pProjectionList); + nodesDestroyList(pStmt->pProjectionBindList); nodesDestroyNode(pStmt->pFromTable); nodesDestroyNode(pStmt->pWhere); nodesDestroyList(pStmt->pPartitionByList); @@ -3261,3 +3265,12 @@ int32_t nodesListDeduplicate(SNodeList** ppList) { } return code; } + +void rewriteExprAliasName(SExprNode* pNode, int64_t num) { + (void)tsnprintf(pNode->aliasName, TSDB_COL_NAME_LEN, "expr_%x", num); + return; +} + +bool isRelatedToOtherExpr(SExprNode* pExpr) { + return pExpr->relatedTo != 0; +} diff --git a/source/libs/nodes/test/nodesCloneTest.cpp b/source/libs/nodes/test/nodesCloneTest.cpp index ea0201f2ef..6ac6c72f48 100644 --- a/source/libs/nodes/test/nodesCloneTest.cpp +++ b/source/libs/nodes/test/nodesCloneTest.cpp @@ -91,6 +91,7 @@ TEST_F(NodesCloneTest, stateWindow) { SStateWindowNode* pDstNode = (SStateWindowNode*)pDst; ASSERT_EQ(nodeType(pSrcNode->pCol), nodeType(pDstNode->pCol)); ASSERT_EQ(nodeType(pSrcNode->pExpr), nodeType(pDstNode->pExpr)); + ASSERT_EQ(nodeType(pSrcNode->pTrueForLimit), nodeType(pDstNode->pTrueForLimit)); }); std::unique_ptr srcNode(nullptr, nodesDestroyNode); @@ -102,6 +103,7 @@ TEST_F(NodesCloneTest, stateWindow) { SStateWindowNode* pNode = (SStateWindowNode*)srcNode.get(); code = nodesMakeNode(QUERY_NODE_COLUMN, &pNode->pCol); code = nodesMakeNode(QUERY_NODE_OPERATOR, &pNode->pExpr); + code = nodesMakeNode(QUERY_NODE_VALUE, &pNode->pTrueForLimit); return srcNode.get(); }()); } diff --git a/source/libs/parser/inc/parAst.h b/source/libs/parser/inc/parAst.h index d99dfc977a..65274f85e1 100644 --- a/source/libs/parser/inc/parAst.h +++ b/source/libs/parser/inc/parAst.h @@ -155,8 +155,8 @@ SNode* createViewNode(SAstCreateContext* pCxt, SToken* pDbName, SToken* pVie SNode* createLimitNode(SAstCreateContext* pCxt, SNode* pLimit, SNode* pOffset); SNode* createOrderByExprNode(SAstCreateContext* pCxt, SNode* pExpr, EOrder order, ENullOrder nullOrder); SNode* createSessionWindowNode(SAstCreateContext* pCxt, SNode* pCol, SNode* pGap); -SNode* createStateWindowNode(SAstCreateContext* pCxt, SNode* pExpr); -SNode* createEventWindowNode(SAstCreateContext* pCxt, SNode* pStartCond, SNode* pEndCond); +SNode* createStateWindowNode(SAstCreateContext* pCxt, SNode* pExpr, SNode *pTrueForLimit); +SNode* createEventWindowNode(SAstCreateContext* pCxt, SNode* pStartCond, SNode* pEndCond, SNode *pTrueForLimit); SNode* createCountWindowNode(SAstCreateContext* pCxt, const SToken* pCountToken, const SToken* pSlidingToken); SNode* createAnomalyWindowNode(SAstCreateContext* pCxt, SNode* pExpr, const SToken* pFuncOpt); SNode* createIntervalWindowNode(SAstCreateContext* pCxt, SNode* pInterval, SNode* pOffset, SNode* pSliding, @@ -335,6 +335,7 @@ SNode* createDropTSMAStmt(SAstCreateContext* pCxt, bool ignoreNotExists, SNode* SNode* createShowCreateTSMAStmt(SAstCreateContext* pCxt, SNode* pRealTable); SNode* createShowTSMASStmt(SAstCreateContext* pCxt, SNode* dbName); SNode* createShowDiskUsageStmt(SAstCreateContext* pCxt, SNode* dbName, ENodeType type); +SNodeList* createColsFuncParamNodeList(SAstCreateContext* pCxt, SNode* pFuncNode, SNodeList* pNodeList, SToken* pAlias); #ifdef __cplusplus } diff --git a/source/libs/parser/inc/sql.y b/source/libs/parser/inc/sql.y index f7dcd1b4ce..42d2e95d24 100755 --- a/source/libs/parser/inc/sql.y +++ b/source/libs/parser/inc/sql.y @@ -1283,6 +1283,7 @@ pseudo_column(A) ::= IROWTS_ORIGIN(B). function_expression(A) ::= function_name(B) NK_LP expression_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); } function_expression(A) ::= star_func(B) NK_LP star_func_para_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); } +function_expression(A) ::= cols_func(B) NK_LP cols_func_para_list(C) NK_RP(D). { A = createRawExprNodeExt(pCxt, &B, &D, createFunctionNode(pCxt, &B, C)); } function_expression(A) ::= CAST(B) NK_LP expr_or_subquery(C) AS type_name(D) NK_RP(E). { A = createRawExprNodeExt(pCxt, &B, &E, createCastFunctionNode(pCxt, releaseRawExprNode(pCxt, C), D)); } function_expression(A) ::= @@ -1345,6 +1346,23 @@ star_func(A) ::= FIRST(B). star_func(A) ::= LAST(B). { A = B; } star_func(A) ::= LAST_ROW(B). { A = B; } +%type cols_func { SToken } +%destructor cols_func { } +cols_func(A) ::= COLS(B). { A = B; } + +%type cols_func_para_list { SNodeList* } +%destructor cols_func_para_list { nodesDestroyList($$); } +cols_func_para_list(A) ::= function_expression(B) NK_COMMA cols_func_expression_list(C). { A = createColsFuncParamNodeList(pCxt, B, C, NULL); } + +cols_func_expression(A) ::= expr_or_subquery(B). { A = releaseRawExprNode(pCxt, B); } +cols_func_expression(A) ::= expr_or_subquery(B) column_alias(C). { A = setProjectionAlias(pCxt, releaseRawExprNode(pCxt, B), &C);} +cols_func_expression(A) ::= expr_or_subquery(B) AS column_alias(C). { A = setProjectionAlias(pCxt, releaseRawExprNode(pCxt, B), &C);} + +%type cols_func_expression_list { SNodeList* } +%destructor cols_func_expression_list { nodesDestroyList($$); } +cols_func_expression_list(A) ::= cols_func_expression(B). { A = createNodeList(pCxt, B); } +cols_func_expression_list(A) ::= cols_func_expression_list(B) NK_COMMA cols_func_expression(C). { A = addNodeToList(pCxt, B, C); } + %type star_func_para_list { SNodeList* } %destructor star_func_para_list { nodesDestroyList($$); } star_func_para_list(A) ::= NK_STAR(B). { A = createNodeList(pCxt, createColumnNode(pCxt, NULL, &B)); } @@ -1628,7 +1646,8 @@ partition_item(A) ::= expr_or_subquery(B) AS column_alias(C). twindow_clause_opt(A) ::= . { A = NULL; } twindow_clause_opt(A) ::= SESSION NK_LP column_reference(B) NK_COMMA interval_sliding_duration_literal(C) NK_RP. { A = createSessionWindowNode(pCxt, releaseRawExprNode(pCxt, B), releaseRawExprNode(pCxt, C)); } -twindow_clause_opt(A) ::= STATE_WINDOW NK_LP expr_or_subquery(B) NK_RP. { A = createStateWindowNode(pCxt, releaseRawExprNode(pCxt, B)); } +twindow_clause_opt(A) ::= + STATE_WINDOW NK_LP expr_or_subquery(B) NK_RP true_for_opt(C). { A = createStateWindowNode(pCxt, releaseRawExprNode(pCxt, B), C); } twindow_clause_opt(A) ::= INTERVAL NK_LP interval_sliding_duration_literal(B) NK_RP sliding_opt(C) fill_opt(D). { A = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, B), NULL, C, D); } twindow_clause_opt(A) ::= @@ -1637,9 +1656,9 @@ twindow_clause_opt(A) ::= sliding_opt(D) fill_opt(E). { A = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, B), releaseRawExprNode(pCxt, C), D, E); } twindow_clause_opt(A) ::= INTERVAL NK_LP interval_sliding_duration_literal(B) NK_COMMA - AUTO(C) NK_RP sliding_opt(D) fill_opt(E). { A = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, B), createDurationValueNode(pCxt, &C), D, E); } -twindow_clause_opt(A) ::= - EVENT_WINDOW START WITH search_condition(B) END WITH search_condition(C). { A = createEventWindowNode(pCxt, B, C); } + AUTO(C) NK_RP sliding_opt(D) fill_opt(E). { A = createIntervalWindowNode(pCxt, releaseRawExprNode(pCxt, B), createDurationValueNode(pCxt, &C), D, E); } +twindow_clause_opt(A) ::= EVENT_WINDOW START WITH search_condition(B) + END WITH search_condition(C) true_for_opt(D). { A = createEventWindowNode(pCxt, B, C, D); } twindow_clause_opt(A) ::= COUNT_WINDOW NK_LP NK_INTEGER(B) NK_RP. { A = createCountWindowNode(pCxt, &B, &B); } twindow_clause_opt(A) ::= @@ -1717,6 +1736,9 @@ range_opt(A) ::= every_opt(A) ::= . { A = NULL; } every_opt(A) ::= EVERY NK_LP duration_literal(B) NK_RP. { A = releaseRawExprNode(pCxt, B); } +true_for_opt(A) ::= . { A = NULL; } +true_for_opt(A) ::= TRUE_FOR NK_LP interval_sliding_duration_literal(B) NK_RP. { A = releaseRawExprNode(pCxt, B); } + /************************************************ query_expression ****************************************************/ query_expression(A) ::= query_simple(B) order_by_clause_opt(C) slimit_clause_opt(D) limit_clause_opt(E). { diff --git a/source/libs/parser/src/parAstCreater.c b/source/libs/parser/src/parAstCreater.c index ec0b38a5e7..341fc7e603 100644 --- a/source/libs/parser/src/parAstCreater.c +++ b/source/libs/parser/src/parAstCreater.c @@ -16,6 +16,7 @@ #include #include +#include "nodes.h" #include "parAst.h" #include "parUtil.h" #include "tglobal.h" @@ -356,6 +357,33 @@ SToken getTokenFromRawExprNode(SAstCreateContext* pCxt, SNode* pNode) { return t; } +SNodeList* createColsFuncParamNodeList(SAstCreateContext* pCxt, SNode* pNode, SNodeList* pNodeList, SToken* pAlias) { + CHECK_PARSER_STATUS(pCxt); + if (NULL == pNode || QUERY_NODE_RAW_EXPR != nodeType(pNode)) { + pCxt->errCode = TSDB_CODE_PAR_SYNTAX_ERROR; + } + CHECK_PARSER_STATUS(pCxt); + SRawExprNode* pRawExpr = (SRawExprNode*)pNode; + SNode* pFuncNode = pRawExpr->pNode; + if(pFuncNode->type != QUERY_NODE_FUNCTION) { + pCxt->errCode = TSDB_CODE_PAR_SYNTAX_ERROR; + } + CHECK_PARSER_STATUS(pCxt); + SNodeList* list = NULL; + pCxt->errCode = nodesMakeList(&list); + CHECK_MAKE_NODE(list); + pCxt->errCode = nodesListAppend(list, pFuncNode); + CHECK_PARSER_STATUS(pCxt); + pCxt->errCode = nodesListAppendList(list, pNodeList); + CHECK_PARSER_STATUS(pCxt); + return list; + + _err: + nodesDestroyNode(pFuncNode); + nodesDestroyList(pNodeList); + return NULL; +} + SNodeList* createNodeList(SAstCreateContext* pCxt, SNode* pNode) { CHECK_PARSER_STATUS(pCxt); SNodeList* list = NULL; @@ -1332,7 +1360,7 @@ _err: return NULL; } -SNode* createStateWindowNode(SAstCreateContext* pCxt, SNode* pExpr) { +SNode* createStateWindowNode(SAstCreateContext* pCxt, SNode* pExpr, SNode* pTrueForLimit) { SStateWindowNode* state = NULL; CHECK_PARSER_STATUS(pCxt); pCxt->errCode = nodesMakeNode(QUERY_NODE_STATE_WINDOW, (SNode**)&state); @@ -1340,14 +1368,16 @@ SNode* createStateWindowNode(SAstCreateContext* pCxt, SNode* pExpr) { state->pCol = createPrimaryKeyCol(pCxt, NULL); CHECK_MAKE_NODE(state->pCol); state->pExpr = pExpr; + state->pTrueForLimit = pTrueForLimit; return (SNode*)state; _err: nodesDestroyNode((SNode*)state); nodesDestroyNode(pExpr); + nodesDestroyNode(pTrueForLimit); return NULL; } -SNode* createEventWindowNode(SAstCreateContext* pCxt, SNode* pStartCond, SNode* pEndCond) { +SNode* createEventWindowNode(SAstCreateContext* pCxt, SNode* pStartCond, SNode* pEndCond, SNode* pTrueForLimit) { SEventWindowNode* pEvent = NULL; CHECK_PARSER_STATUS(pCxt); pCxt->errCode = nodesMakeNode(QUERY_NODE_EVENT_WINDOW, (SNode**)&pEvent); @@ -1356,11 +1386,13 @@ SNode* createEventWindowNode(SAstCreateContext* pCxt, SNode* pStartCond, SNode* CHECK_MAKE_NODE(pEvent->pCol); pEvent->pStartCond = pStartCond; pEvent->pEndCond = pEndCond; + pEvent->pTrueForLimit = pTrueForLimit; return (SNode*)pEvent; _err: nodesDestroyNode((SNode*)pEvent); nodesDestroyNode(pStartCond); nodesDestroyNode(pEndCond); + nodesDestroyNode(pTrueForLimit); return NULL; } diff --git a/source/libs/parser/src/parTokenizer.c b/source/libs/parser/src/parTokenizer.c index 3b08d403dc..eb949d5206 100644 --- a/source/libs/parser/src/parTokenizer.c +++ b/source/libs/parser/src/parTokenizer.c @@ -355,10 +355,12 @@ static SKeyword keywordTable[] = { {"FORCE_WINDOW_CLOSE", TK_FORCE_WINDOW_CLOSE}, {"DISK_INFO", TK_DISK_INFO}, {"AUTO", TK_AUTO}, + {"COLS", TK_COLS}, {"NOTIFY", TK_NOTIFY}, {"ON_FAILURE", TK_ON_FAILURE}, {"NOTIFY_HISTORY", TK_NOTIFY_HISTORY}, {"REGEXP", TK_REGEXP}, + {"TRUE_FOR", TK_TRUE_FOR} }; // clang-format on diff --git a/source/libs/parser/src/parTranslater.c b/source/libs/parser/src/parTranslater.c index dac6446402..2fdba9bad9 100755 --- a/source/libs/parser/src/parTranslater.c +++ b/source/libs/parser/src/parTranslater.c @@ -13,8 +13,13 @@ * along with this program. If not, see . */ +#include "nodes.h" #include "parInt.h" #include "parTranslater.h" +#include +#include "query.h" +#include "querynodes.h" +#include "taoserror.h" #include "tdatablock.h" #include "catalog.h" @@ -1099,6 +1104,15 @@ static bool isForecastPseudoColumnFunc(const SNode* pNode) { return (QUERY_NODE_FUNCTION == nodeType(pNode) && fmIsForecastPseudoColumnFunc(((SFunctionNode*)pNode)->funcId)); } +static bool isColsFunctionResult(const SNode* pNode) { + return ((nodesIsExprNode(pNode)) && (isRelatedToOtherExpr((SExprNode*)pNode))); +} + +static bool isInvalidColsBindFunction(const SFunctionNode* pFunc) { + return (pFunc->node.bindExprID != 0 && (!fmIsSelectFunc(pFunc->funcId) || fmIsMultiRowsFunc(pFunc->funcId) || + fmIsIndefiniteRowsFunc(pFunc->funcId))); +} + #ifdef BUILD_NO_CALL static bool isTimelineFunc(const SNode* pNode) { return (QUERY_NODE_FUNCTION == nodeType(pNode) && fmIsTimelineFunc(((SFunctionNode*)pNode)->funcId)); @@ -1125,6 +1139,10 @@ static bool isVectorFunc(const SNode* pNode) { return (QUERY_NODE_FUNCTION == nodeType(pNode) && fmIsVectorFunc(((SFunctionNode*)pNode)->funcId)); } +static bool isColsFunc(const SNode* pNode) { + return (QUERY_NODE_FUNCTION == nodeType(pNode) && fmIsSelectColsFunc(((SFunctionNode*)pNode)->funcId)); +} + static bool isDistinctOrderBy(STranslateContext* pCxt) { return (SQL_CLAUSE_ORDER_BY == pCxt->currClause && isSelectStmt(pCxt->pCurrStmt) && ((SSelectStmt*)pCxt->pCurrStmt)->isDistinct); @@ -1557,24 +1575,27 @@ static int32_t findAndSetColumn(STranslateContext* pCxt, SColumnNode** pColRef, STempTableNode* pTempTable = (STempTableNode*)pTable; SNodeList* pProjectList = getProjectList(pTempTable->pSubquery); SNode* pNode; + SExprNode* pFoundExpr = NULL; FOREACH(pNode, pProjectList) { SExprNode* pExpr = (SExprNode*)pNode; if (0 == strcmp(pCol->colName, pExpr->aliasName)) { if (*pFound) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_AMBIGUOUS_COLUMN, pCol->colName); } - code = setColumnInfoByExpr(pTempTable, pExpr, pColRef); - if (TSDB_CODE_SUCCESS != code) { - break; - } + pFoundExpr = pExpr; *pFound = true; } else if (isPrimaryKeyImpl(pNode) && isInternalPrimaryKey(pCol)) { - code = setColumnInfoByExpr(pTempTable, pExpr, pColRef); - if (TSDB_CODE_SUCCESS != code) break; + pFoundExpr = pExpr; pCol->isPrimTs = true; *pFound = true; } } + if (pFoundExpr) { + code = setColumnInfoByExpr(pTempTable, pFoundExpr, pColRef); + if (TSDB_CODE_SUCCESS != code) { + return code; + } + } } return code; } @@ -2331,7 +2352,7 @@ static EDealRes translateOperator(STranslateContext* pCxt, SOperatorNode* pOp) { static EDealRes haveVectorFunction(SNode* pNode, void* pContext) { if (isAggFunc(pNode) || isIndefiniteRowsFunc(pNode) || isWindowPseudoColumnFunc(pNode) || - isInterpPseudoColumnFunc(pNode) || isForecastPseudoColumnFunc(pNode)) { + isInterpPseudoColumnFunc(pNode) || isForecastPseudoColumnFunc(pNode) || isColsFunctionResult(pNode)) { *((bool*)pContext) = true; return DEAL_RES_END; } @@ -2479,9 +2500,9 @@ static int32_t rewriteCountTbname(STranslateContext* pCxt, SFunctionNode* pCount return code; } -static bool hasInvalidFuncNesting(SNodeList* pParameterList) { +static bool hasInvalidFuncNesting(SFunctionNode* pFunc) { bool hasInvalidFunc = false; - nodesWalkExprs(pParameterList, haveVectorFunction, &hasInvalidFunc); + nodesWalkExprs(pFunc->pParameterList, haveVectorFunction, &hasInvalidFunc); return hasInvalidFunc; } @@ -2503,7 +2524,7 @@ static int32_t translateAggFunc(STranslateContext* pCxt, SFunctionNode* pFunc) { if (beforeHaving(pCxt->currClause)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_ILLEGAL_USE_AGG_FUNCTION); } - if (hasInvalidFuncNesting(pFunc->pParameterList)) { + if (hasInvalidFuncNesting(pFunc)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_AGG_FUNC_NESTING); } // The auto-generated COUNT function in the DELETE statement is legal @@ -2547,7 +2568,7 @@ static int32_t translateIndefiniteRowsFunc(STranslateContext* pCxt, SFunctionNod return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_NOT_ALLOWED_FUNC, "%s function is not supported in window query or group query", pFunc->functionName); } - if (hasInvalidFuncNesting(pFunc->pParameterList)) { + if (hasInvalidFuncNesting(pFunc)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_AGG_FUNC_NESTING); } return TSDB_CODE_SUCCESS; @@ -2562,7 +2583,7 @@ static int32_t translateMultiRowsFunc(STranslateContext* pCxt, SFunctionNode* pF ((SSelectStmt*)pCxt->pCurrStmt)->hasMultiRowsFunc) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NOT_ALLOWED_FUNC); } - if (hasInvalidFuncNesting(pFunc->pParameterList)) { + if (hasInvalidFuncNesting(pFunc)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_AGG_FUNC_NESTING); } return TSDB_CODE_SUCCESS; @@ -2593,7 +2614,7 @@ static int32_t translateInterpFunc(STranslateContext* pCxt, SFunctionNode* pFunc return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_NOT_ALLOWED_FUNC, "%s function is not supported in window query or group query", pFunc->functionName); } - if (hasInvalidFuncNesting(pFunc->pParameterList)) { + if (hasInvalidFuncNesting(pFunc)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_AGG_FUNC_NESTING); } return TSDB_CODE_SUCCESS; @@ -2659,7 +2680,7 @@ static int32_t translateForecastFunc(STranslateContext* pCxt, SFunctionNode* pFu return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_NOT_ALLOWED_FUNC, "%s function is not supported in window query or group query", pFunc->functionName); } - if (hasInvalidFuncNesting(pFunc->pParameterList)) { + if (hasInvalidFuncNesting(pFunc)) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_AGG_FUNC_NESTING); } return TSDB_CODE_SUCCESS; @@ -2889,6 +2910,9 @@ static int32_t calcSelectFuncNum(SFunctionNode* pFunc, int32_t currSelectFuncNum if (fmIsCumulativeFunc(pFunc->funcId)) { return currSelectFuncNum > 0 ? currSelectFuncNum : 1; } + if(fmIsSelectColsFunc(pFunc->funcId)) { + return currSelectFuncNum; + } return currSelectFuncNum + ((fmIsMultiResFunc(pFunc->funcId) && !fmIsLastRowFunc(pFunc->funcId)) ? getMultiResFuncNum(pFunc->pParameterList) : 1); @@ -3314,6 +3338,10 @@ static EDealRes translateFunction(STranslateContext* pCxt, SFunctionNode** pFunc pCxt->errCode = TSDB_CODE_PAR_ILLEGAL_USE_AGG_FUNCTION; } } + if (isInvalidColsBindFunction(*pFunc)) { + pCxt->errCode = TSDB_CODE_PAR_INVALID_COLS_SELECTFUNC; + return DEAL_RES_ERROR; + } if (TSDB_CODE_SUCCESS == pCxt->errCode) { pCxt->errCode = translateFunctionImpl(pCxt, pFunc); } @@ -3555,6 +3583,9 @@ static EDealRes rewriteColToSelectValFunc(STranslateContext* pCxt, SNode** pNode tstrncpy(pFunc->functionName, "_select_value", TSDB_FUNC_NAME_LEN); tstrncpy(pFunc->node.aliasName, ((SExprNode*)*pNode)->aliasName, TSDB_COL_NAME_LEN); tstrncpy(pFunc->node.userAlias, ((SExprNode*)*pNode)->userAlias, TSDB_COL_NAME_LEN); + + pFunc->node.relatedTo = ((SExprNode*)*pNode)->relatedTo; + pFunc->node.bindExprID = ((SExprNode*)*pNode)->bindExprID; pCxt->errCode = nodesListMakeAppend(&pFunc->pParameterList, *pNode); if (TSDB_CODE_SUCCESS == pCxt->errCode) { pCxt->errCode = getFuncInfo(pCxt, pFunc); @@ -3827,6 +3858,7 @@ static EDealRes doCheckExprForGroupBy(SNode** pNode, void* pContext) { if (isVectorFunc(*pNode) && !isDistinctOrderBy(pCxt)) { return DEAL_RES_IGNORE_CHILD; } + bool isSingleTable = fromSingleTable(((SSelectStmt*)pCxt->pCurrStmt)->pFromTable); SNode* pGroupNode = NULL; FOREACH(pGroupNode, getGroupByList(pCxt)) { SNode* pActualNode = getGroupByNode(pGroupNode); @@ -3836,10 +3868,13 @@ static EDealRes doCheckExprForGroupBy(SNode** pNode, void* pContext) { if (IsEqualTbNameFuncNode(pSelect, pActualNode, *pNode)) { return rewriteExprToGroupKeyFunc(pCxt, pNode); } - if (isTbnameFuction(pActualNode) && QUERY_NODE_COLUMN == nodeType(*pNode) && + if ((isTbnameFuction(pActualNode) || isSingleTable) && QUERY_NODE_COLUMN == nodeType(*pNode) && ((SColumnNode*)*pNode)->colType == COLUMN_TYPE_TAG) { return rewriteExprToSelectTagFunc(pCxt, pNode); } + if(isSingleTable && isTbnameFuction(*pNode)) { + return rewriteExprToSelectTagFunc(pCxt, pNode); + } } SNode* pPartKey = NULL; bool partionByTbname = hasTbnameFunction(pSelect->pPartitionByList); @@ -3863,17 +3898,29 @@ static EDealRes doCheckExprForGroupBy(SNode** pNode, void* pContext) { } } - if (isScanPseudoColumnFunc(*pNode) || QUERY_NODE_COLUMN == nodeType(*pNode)) { - if (pSelect->selectFuncNum > 1 || (isDistinctOrderBy(pCxt) && pCxt->currClause == SQL_CLAUSE_ORDER_BY)) { + if (isScanPseudoColumnFunc(*pNode)) { + if (((pSelect->selectFuncNum > 1 && pCxt->stableQuery) || + (isDistinctOrderBy(pCxt) && pCxt->currClause == SQL_CLAUSE_ORDER_BY)) && + !isRelatedToOtherExpr((SExprNode*)*pNode)) { return generateDealNodeErrMsg(pCxt, getGroupByErrorCode(pCxt), ((SExprNode*)(*pNode))->userAlias); } + } + + if (QUERY_NODE_COLUMN == nodeType(*pNode)) { + if (((pSelect->selectFuncNum > 1) || (isDistinctOrderBy(pCxt) && pCxt->currClause == SQL_CLAUSE_ORDER_BY)) && + !isRelatedToOtherExpr((SExprNode*)*pNode)) { + return generateDealNodeErrMsg(pCxt, getGroupByErrorCode(pCxt), ((SExprNode*)(*pNode))->userAlias); + } + } + + if (isScanPseudoColumnFunc(*pNode) || QUERY_NODE_COLUMN == nodeType(*pNode)) { if (isWindowJoinStmt(pSelect) && (isWindowJoinProbeTableCol(pSelect, *pNode) || isWindowJoinGroupCol(pSelect, *pNode) || (isWindowJoinSubTbname(pSelect, *pNode)) || isWindowJoinSubTbTag(pSelect, *pNode))) { return rewriteExprToGroupKeyFunc(pCxt, pNode); } - if (pSelect->hasOtherVectorFunc || !pSelect->hasSelectFunc) { + if ((pSelect->hasOtherVectorFunc || !pSelect->hasSelectFunc) && !isRelatedToOtherExpr((SExprNode*)*pNode)) { return generateDealNodeErrMsg(pCxt, getGroupByErrorCode(pCxt), ((SExprNode*)(*pNode))->userAlias); } @@ -3920,6 +3967,7 @@ static int32_t rewriteColsToSelectValFunc(STranslateContext* pCxt, SSelectStmt* typedef struct CheckAggColCoexistCxt { STranslateContext* pTranslateCxt; bool existCol; + bool hasColFunc; SNodeList* pColList; } CheckAggColCoexistCxt; @@ -3928,6 +3976,10 @@ static EDealRes doCheckAggColCoexist(SNode** pNode, void* pContext) { if (isVectorFunc(*pNode)) { return DEAL_RES_IGNORE_CHILD; } + if(isColsFunctionResult(*pNode)) { + pCxt->hasColFunc = true; + } + SNode* pPartKey = NULL; bool partionByTbname = false; if (fromSingleTable(((SSelectStmt*)pCxt->pTranslateCxt->pCurrStmt)->pFromTable) || @@ -3947,7 +3999,8 @@ static EDealRes doCheckAggColCoexist(SNode** pNode, void* pContext) { ((QUERY_NODE_COLUMN == nodeType(*pNode) && ((SColumnNode*)*pNode)->colType == COLUMN_TYPE_TAG))) { return rewriteExprToSelectTagFunc(pCxt->pTranslateCxt, pNode); } - if (isScanPseudoColumnFunc(*pNode) || QUERY_NODE_COLUMN == nodeType(*pNode)) { + if ((isScanPseudoColumnFunc(*pNode) || QUERY_NODE_COLUMN == nodeType(*pNode)) && + ((!nodesIsExprNode(*pNode) || !isRelatedToOtherExpr((SExprNode*)*pNode)))) { pCxt->existCol = true; } return DEAL_RES_CONTINUE; @@ -4006,7 +4059,7 @@ static int32_t checkAggColCoexist(STranslateContext* pCxt, SSelectStmt* pSelect) if (!pSelect->onlyHasKeepOrderFunc) { pSelect->timeLineResMode = TIME_LINE_NONE; } - CheckAggColCoexistCxt cxt = {.pTranslateCxt = pCxt, .existCol = false}; + CheckAggColCoexistCxt cxt = {.pTranslateCxt = pCxt, .existCol = false, .hasColFunc = false}; nodesRewriteExprs(pSelect->pProjectionList, doCheckAggColCoexist, &cxt); if (!pSelect->isDistinct) { nodesRewriteExprs(pSelect->pOrderByList, doCheckAggColCoexist, &cxt); @@ -4018,6 +4071,9 @@ static int32_t checkAggColCoexist(STranslateContext* pCxt, SSelectStmt* pSelect) if (cxt.existCol) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_NOT_SINGLE_GROUP); } + if (cxt.hasColFunc) { + return rewriteColsToSelectValFunc(pCxt, pSelect); + } return TSDB_CODE_SUCCESS; } @@ -4060,7 +4116,7 @@ static int32_t checkWinJoinAggColCoexist(STranslateContext* pCxt, SSelectStmt* p } static int32_t checkHavingGroupBy(STranslateContext* pCxt, SSelectStmt* pSelect) { - int32_t code = TSDB_CODE_SUCCESS; + int32_t code = TSDB_CODE_SUCCESS; if (NULL == getGroupByList(pCxt) && NULL == pSelect->pPartitionByList && NULL == pSelect->pWindow && !isWindowJoinStmt(pSelect)) { return code; @@ -5425,9 +5481,38 @@ static int32_t translateClausePosition(STranslateContext* pCxt, SNodeList* pProj return TSDB_CODE_SUCCESS; } +static int32_t rewriteColsFunction(STranslateContext* pCxt, SNodeList** nodeList, SNodeList** selectFuncList); +static int32_t rewriteHavingColsNode(STranslateContext* pCxt, SNode** pNode, SNodeList** selectFuncList); + +static int32_t prepareColumnExpansion(STranslateContext* pCxt, ESqlClause clause, SSelectStmt* pSelect) { + int32_t code = TSDB_CODE_SUCCESS; + int32_t len = LIST_LENGTH(pSelect->pProjectionBindList); + if (clause == SQL_CLAUSE_SELECT) { + code = rewriteColsFunction(pCxt, &pSelect->pProjectionList, &pSelect->pProjectionBindList); + } else if (clause == SQL_CLAUSE_HAVING) { + code = rewriteHavingColsNode(pCxt, &pSelect->pHaving, &pSelect->pProjectionBindList); + } else if (clause == SQL_CLAUSE_ORDER_BY) { + code = rewriteColsFunction(pCxt, &pSelect->pOrderByList, &pSelect->pProjectionBindList); + } else { + code = + generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_WRONG_VALUE_TYPE, "Invalid clause for column expansion"); + } + if (TSDB_CODE_SUCCESS == code && LIST_LENGTH(pSelect->pProjectionBindList) > len) { + code = translateExprList(pCxt, pSelect->pProjectionBindList); + } + if (pSelect->pProjectionBindList != NULL) { + pSelect->hasAggFuncs = true; + } + return code; +} + static int32_t translateOrderBy(STranslateContext* pCxt, SSelectStmt* pSelect) { bool other; - int32_t code = translateClausePosition(pCxt, pSelect->pProjectionList, pSelect->pOrderByList, &other); + int32_t code = prepareColumnExpansion(pCxt, SQL_CLAUSE_ORDER_BY, pSelect); + if (TSDB_CODE_SUCCESS != code) { + return code; + } + code = translateClausePosition(pCxt, pSelect->pProjectionList, pSelect->pOrderByList, &other); if (TSDB_CODE_SUCCESS == code) { if (0 == LIST_LENGTH(pSelect->pOrderByList)) { NODES_DESTORY_LIST(pSelect->pOrderByList); @@ -5528,7 +5613,7 @@ static int32_t rewriteProjectAlias(SNodeList* pProjectionList) { if ('\0' == pExpr->userAlias[0]) { tstrncpy(pExpr->userAlias, pExpr->aliasName, TSDB_COL_NAME_LEN); } - snprintf(pExpr->aliasName, TSDB_COL_NAME_LEN,"#expr_%d", no++); + rewriteExprAliasName(pExpr, no++); } return TSDB_CODE_SUCCESS; } @@ -5557,14 +5642,14 @@ static int32_t checkProjectAlias(STranslateContext* pCxt, SNodeList* pProjection } static int32_t translateProjectionList(STranslateContext* pCxt, SSelectStmt* pSelect) { + SNode* pNode; + int32_t projIdx = 1; + FOREACH(pNode, pSelect->pProjectionList) { ((SExprNode*)pNode)->projIdx = projIdx++; } + if (!pSelect->isSubquery) { return rewriteProjectAlias(pSelect->pProjectionList); - } else { - SNode* pNode; - int32_t projIdx = 1; - FOREACH(pNode, pSelect->pProjectionList) { ((SExprNode*)pNode)->projIdx = projIdx++; } - return TSDB_CODE_SUCCESS; } + return TSDB_CODE_SUCCESS; } typedef struct SReplaceGroupByAliasCxt { @@ -5646,7 +5731,10 @@ static int32_t translatePartitionByList(STranslateContext* pCxt, SSelectStmt* pS static int32_t translateSelectList(STranslateContext* pCxt, SSelectStmt* pSelect) { pCxt->currClause = SQL_CLAUSE_SELECT; - int32_t code = translateExprList(pCxt, pSelect->pProjectionList); + int32_t code = prepareColumnExpansion(pCxt, SQL_CLAUSE_SELECT, pSelect); + if (TSDB_CODE_SUCCESS == code) { + code = translateExprList(pCxt, pSelect->pProjectionList); + } if (TSDB_CODE_SUCCESS == code) { code = translateStar(pCxt, pSelect); } @@ -5670,10 +5758,18 @@ static int32_t translateSelectList(STranslateContext* pCxt, SSelectStmt* pSelect } static int32_t translateHaving(STranslateContext* pCxt, SSelectStmt* pSelect) { + int32_t code = TSDB_CODE_SUCCESS; if (NULL == pSelect->pGroupByList && NULL == pSelect->pPartitionByList && NULL == pSelect->pWindow && !isWindowJoinStmt(pSelect) && NULL != pSelect->pHaving) { return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_GROUPBY_LACK_EXPRESSION); } + pCxt->currClause = SQL_CLAUSE_HAVING; + if (NULL != pSelect->pHaving) { + code = prepareColumnExpansion(pCxt, SQL_CLAUSE_HAVING, pSelect); + if (TSDB_CODE_SUCCESS != code) { + return code; + } + } if (isWindowJoinStmt(pSelect)) { if (NULL != pSelect->pHaving) { bool hasFunc = false; @@ -5683,9 +5779,7 @@ static int32_t translateHaving(STranslateContext* pCxt, SSelectStmt* pSelect) { } } } - pCxt->currClause = SQL_CLAUSE_HAVING; - int32_t code = translateExpr(pCxt, &pSelect->pHaving); - return code; + return translateExpr(pCxt, &pSelect->pHaving); } static int32_t translateGroupBy(STranslateContext* pCxt, SSelectStmt* pSelect) { @@ -6043,6 +6137,20 @@ static int32_t checkStateWindowForStream(STranslateContext* pCxt, SSelectStmt* p return TSDB_CODE_SUCCESS; } +static int32_t checkTrueForLimit(STranslateContext *pCxt, SNode *pNode) { + SValueNode *pTrueForLimit = (SValueNode *)pNode; + if (pTrueForLimit == NULL) { + return TSDB_CODE_SUCCESS; + } + if (pTrueForLimit->datum.i < 0) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_TRUE_FOR_NEGATIVE); + } + if (IS_CALENDAR_TIME_DURATION(pTrueForLimit->unit)) { + return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_TRUE_FOR_UNIT); + } + return TSDB_CODE_SUCCESS; +} + static int32_t translateStateWindow(STranslateContext* pCxt, SSelectStmt* pSelect) { if (QUERY_NODE_TEMP_TABLE == nodeType(pSelect->pFromTable) && !isGlobalTimeLineQuery(((STempTableNode*)pSelect->pFromTable)->pSubquery)) { @@ -6055,6 +6163,9 @@ static int32_t translateStateWindow(STranslateContext* pCxt, SSelectStmt* pSelec if (TSDB_CODE_SUCCESS == code) { code = checkStateWindowForStream(pCxt, pSelect); } + if (TSDB_CODE_SUCCESS == code) { + code = checkTrueForLimit(pCxt, pState->pTrueForLimit); + } return code; } @@ -6081,7 +6192,7 @@ static int32_t translateEventWindow(STranslateContext* pCxt, SSelectStmt* pSelec return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_TIMELINE_QUERY, "EVENT_WINDOW requires valid time series input"); } - return TSDB_CODE_SUCCESS; + return checkTrueForLimit(pCxt, ((SEventWindowNode*)pSelect->pWindow)->pTrueForLimit); } static int32_t translateCountWindow(STranslateContext* pCxt, SSelectStmt* pSelect) { @@ -7310,6 +7421,307 @@ static int32_t translateSelectWithoutFrom(STranslateContext* pCxt, SSelectStmt* pCxt->dual = true; return translateExprList(pCxt, pSelect->pProjectionList); } +typedef struct SCheckColsFuncCxt { + bool hasColsFunc; + SNodeList** selectFuncList; + int32_t status; +} SCheckColsFuncCxt; + +static bool isColsFuncByName(SFunctionNode* pFunc) { + if (strcasecmp(pFunc->functionName, "cols") != 0) { + return false; + } + return true; +} + +static bool isMultiColsFuncNode(SNode* pNode) { + if (QUERY_NODE_FUNCTION == nodeType(pNode)) { + SFunctionNode* pFunc = (SFunctionNode*)pNode; + if (isColsFuncByName(pFunc)) { + if (pFunc->pParameterList->length > 2) { + return true; + } + } + } + return false; +} + +typedef struct SBindTupleFuncCxt { + SNode* root; + int32_t bindExprID; +} SBindTupleFuncCxt; + +static EDealRes pushDownBindSelectFunc(SNode** pNode, void* pContext) { + SBindTupleFuncCxt* pCxt = pContext; + if (nodesIsExprNode(*pNode)) { + SExprNode* pExpr = (SExprNode*)*pNode; + pExpr->relatedTo = pCxt->bindExprID; + if (nodeType(*pNode) != QUERY_NODE_COLUMN) { + return DEAL_RES_CONTINUE; + } + + if (*pNode != pCxt->root) { + int len = strlen(pExpr->aliasName); + if (len + TSDB_COL_NAME_EXLEN >= TSDB_COL_NAME_LEN) { + char buffer[TSDB_COL_NAME_EXLEN + TSDB_COL_NAME_LEN + 1] = {0}; + (void)tsnprintf(buffer, sizeof(buffer), "%s.%d", pExpr->aliasName, pExpr->relatedTo); + uint64_t hashVal = MurmurHash3_64(buffer, TSDB_COL_NAME_EXLEN + TSDB_COL_NAME_LEN + 1); + (void)tsnprintf(pExpr->aliasName, TSDB_COL_NAME_EXLEN, "%" PRIu64, hashVal); + } else { + (void)tsnprintf(pExpr->aliasName + len, TSDB_COL_NAME_EXLEN, ".%d", pExpr->relatedTo); + } + } + } + return DEAL_RES_CONTINUE; +} + +static int32_t getSelectFuncIndex(SNodeList* FuncNodeList, SNode* pSelectFunc) { + SNode* pNode = NULL; + int32_t selectFuncIndex = 0; + FOREACH(pNode, FuncNodeList) { + ++selectFuncIndex; + if (nodesEqualNode(pNode, pSelectFunc)) { + return selectFuncIndex; + } + } + return 0; +} + +static EDealRes checkHasColsFunc(SNode** pNode, void* pContext){ + if (QUERY_NODE_FUNCTION == nodeType(*pNode)) { + SFunctionNode* pFunc = (SFunctionNode*)*pNode; + if (isColsFuncByName(pFunc)) { + *(bool*)pContext = true; + return DEAL_RES_END; + } + } + return DEAL_RES_CONTINUE; +} + +static int32_t checkMultColsFuncParam(SNodeList* pParameterList) { + if (!pParameterList || pParameterList->length < 2) { + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + } + int32_t index = 0; + SNode* pNode = NULL; + FOREACH(pNode, pParameterList) { + if (index == 0) { // the first parameter is select function + if (QUERY_NODE_FUNCTION != nodeType(pNode) || isColsFuncByName((SFunctionNode*)pNode)) { + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + } + SFunctionNode* pFunc = (SFunctionNode*)pNode; + // pFunc->funcId is zero at here, so need to check at * step + // if(!fmIsSelectFunc(pFunc->funcId)) { + // return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + // } + SNode* pTmpNode = NULL; + FOREACH(pTmpNode, pFunc->pParameterList) { + bool hasColsFunc = false; + nodesRewriteExpr(&pTmpNode, checkHasColsFunc, (void*)&hasColsFunc); + if (hasColsFunc) { + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + } + } + } else { + bool hasColsFunc = false; + nodesRewriteExpr(&pNode, checkHasColsFunc, &hasColsFunc); + if (hasColsFunc) { + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + } + } + ++index; + } + return TSDB_CODE_SUCCESS; +} + +static EDealRes rewriteSingleColsFunc(SNode** pNode, void* pContext) { + int32_t code = TSDB_CODE_SUCCESS; + if (QUERY_NODE_FUNCTION != nodeType(*pNode)) { + return DEAL_RES_CONTINUE; + } + SCheckColsFuncCxt* pCxt = pContext; + SFunctionNode* pFunc = (SFunctionNode*)*pNode; + if (isColsFuncByName(pFunc)) { + if(pFunc->pParameterList->length > 2) { + pCxt->status = TSDB_CODE_PAR_INVALID_COLS_SELECTFUNC; + return DEAL_RES_ERROR; + } + SNode* pSelectFunc = nodesListGetNode(pFunc->pParameterList, 0); + SNode* pExpr = nodesListGetNode(pFunc->pParameterList, 1); + if (nodeType(pSelectFunc) != QUERY_NODE_FUNCTION || isColsFuncByName((SFunctionNode*)pSelectFunc)) { + pCxt->status = TSDB_CODE_PAR_INVALID_COLS_SELECTFUNC; + parserError("%s Invalid cols function, the first parameter must be a select function", __func__); + return DEAL_RES_ERROR; + } + if (pFunc->node.asAlias) { + if (((SExprNode*)pExpr)->asAlias) { + pCxt->status = TSDB_CODE_INVALID_COLS_ALIAS; + parserError("%s Invalid using alias for cols function", __func__); + return DEAL_RES_ERROR; + } else { + ((SExprNode*)pExpr)->asAlias = true; + tstrncpy(((SExprNode*)pExpr)->userAlias, pFunc->node.userAlias, TSDB_COL_NAME_LEN); + } + } + if(*pCxt->selectFuncList == NULL) { + code = nodesMakeList(pCxt->selectFuncList); + if (NULL == *pCxt->selectFuncList) { + pCxt->status = code; + return DEAL_RES_ERROR; + } + } + int32_t selectFuncCount = (*pCxt->selectFuncList)->length; + int32_t selectFuncIndex = getSelectFuncIndex(*pCxt->selectFuncList, pSelectFunc); + if (selectFuncIndex == 0) { + ++selectFuncCount; + selectFuncIndex = selectFuncCount; + SNode* pNewNode = NULL; + code = nodesCloneNode(pSelectFunc, &pNewNode); + if(code) goto _end; + ((SExprNode*)pNewNode)->bindExprID = selectFuncIndex; + code = nodesListMakeStrictAppend(pCxt->selectFuncList, pNewNode); + if(code) goto _end; + } + + SNode* pNewNode = NULL; + code = nodesCloneNode(pExpr, &pNewNode); + if(code) goto _end; + if (nodesIsExprNode(pNewNode)) { + SBindTupleFuncCxt pCxt = {pNewNode, selectFuncIndex}; + nodesRewriteExpr(&pNewNode, pushDownBindSelectFunc, &pCxt); + } else { + pCxt->status = TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + parserError("%s Invalid cols function, the first parameter must be a select function", __func__); + return DEAL_RES_ERROR; + } + nodesDestroyNode(*pNode); + *pNode = pNewNode; + } + return DEAL_RES_CONTINUE; +_end: + pCxt->status = code; + return DEAL_RES_ERROR; +} + +static int32_t rewriteHavingColsNode(STranslateContext* pCxt, SNode** pNode, SNodeList** selectFuncList) { + int32_t code = TSDB_CODE_SUCCESS; + if(!pNode || *pNode == NULL) return code; + if (isMultiColsFuncNode(*pNode)) { + parserWarn("%s Invalid using multi cols func in having.", __func__); + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + } else { + SCheckColsFuncCxt pSelectFuncCxt = {false, selectFuncList, TSDB_CODE_SUCCESS}; + nodesRewriteExpr(pNode, rewriteSingleColsFunc, &pSelectFuncCxt); + if (pSelectFuncCxt.status != TSDB_CODE_SUCCESS) { + return pSelectFuncCxt.status; + } + } + return code; +} + +static int32_t rewriteColsFunction(STranslateContext* pCxt, SNodeList** nodeList, SNodeList** selectFuncList) { + int32_t code = TSDB_CODE_SUCCESS; + bool needRewrite = false; + SNode** pNode = NULL; + FOREACH_FOR_REWRITE(pNode, *nodeList) { + if (isMultiColsFuncNode(*pNode)) { + code = checkMultColsFuncParam(((SFunctionNode*)*pNode)->pParameterList); + if (TSDB_CODE_SUCCESS != code) { + return code; + } + needRewrite = true; + } else { + SCheckColsFuncCxt pSelectFuncCxt = {false, selectFuncList, TSDB_CODE_SUCCESS}; + nodesRewriteExpr(pNode, rewriteSingleColsFunc, &pSelectFuncCxt); + if (pSelectFuncCxt.status != TSDB_CODE_SUCCESS) { + return pSelectFuncCxt.status; + } + } + } + + SNodeList* pNewNodeList = NULL; + SNode* pNewNode = NULL; + if (needRewrite) { + if (pCxt->createStream) { + return TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + } + code = nodesMakeList(&pNewNodeList); + if (NULL == pNewNodeList) { + return code; + } + if (*selectFuncList == NULL) { + code = nodesMakeList(selectFuncList); + if (NULL == *selectFuncList) { + nodesDestroyList(pNewNodeList); + return code; + } + } + + int32_t nums = 0; + int32_t selectFuncCount = (*selectFuncList)->length; + SNode* pTmpNode = NULL; + FOREACH(pTmpNode, *nodeList) { + if (isMultiColsFuncNode(pTmpNode)) { + SFunctionNode* pFunc = (SFunctionNode*)pTmpNode; + if(pFunc->node.asAlias) { + code = TSDB_CODE_INVALID_COLS_ALIAS; + parserError("%s Invalid using alias for cols function", __func__); + goto _end; + } + + SNode* pSelectFunc = nodesListGetNode(pFunc->pParameterList, 0); + if (nodeType(pSelectFunc) != QUERY_NODE_FUNCTION) { + code = TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + parserError("%s Invalid cols function, the first parameter must be a select function", __func__); + goto _end; + } + int32_t selectFuncIndex = getSelectFuncIndex(*selectFuncList, pSelectFunc); + if (selectFuncIndex == 0) { + ++selectFuncCount; + selectFuncIndex = selectFuncCount; + code = nodesCloneNode(pSelectFunc, &pNewNode); + if(TSDB_CODE_SUCCESS != code) goto _end; + ((SExprNode*)pNewNode)->bindExprID = selectFuncIndex; + code = nodesListMakeStrictAppend(selectFuncList, pNewNode); + if(TSDB_CODE_SUCCESS != code) goto _end; + } + // start from index 1, because the first parameter is select function which needn't to output. + for (int i = 1; i < pFunc->pParameterList->length; ++i) { + SNode* pExpr = nodesListGetNode(pFunc->pParameterList, i); + + code = nodesCloneNode(pExpr, &pNewNode); + if(TSDB_CODE_SUCCESS != code) goto _end; + if (nodesIsExprNode(pNewNode)) { + SBindTupleFuncCxt pCxt = {pNewNode, selectFuncIndex}; + nodesRewriteExpr(&pNewNode, pushDownBindSelectFunc, &pCxt); + } else { + code = TSDB_CODE_PAR_INVALID_COLS_FUNCTION; + parserError("%s Invalid cols function, the first parameter must be a select function", __func__); + goto _end; + } + if (TSDB_CODE_SUCCESS != code) goto _end; + code = nodesListMakeStrictAppend(&pNewNodeList, pNewNode); + if (TSDB_CODE_SUCCESS != code) goto _end; + } + continue; + } + + code = nodesCloneNode(pTmpNode, &pNewNode); + if (TSDB_CODE_SUCCESS != code) goto _end; + code = nodesListMakeStrictAppend(&pNewNodeList, pNewNode); + if (TSDB_CODE_SUCCESS != code) goto _end; + } + nodesDestroyList(*nodeList); + *nodeList = pNewNodeList; + return TSDB_CODE_SUCCESS; + } + _end: + if (TSDB_CODE_SUCCESS != code) { + nodesDestroyNode(pNewNode); + nodesDestroyList(pNewNodeList); + } + return code; +} static int32_t translateSelectFrom(STranslateContext* pCxt, SSelectStmt* pSelect) { pCxt->pCurrStmt = (SNode*)pSelect; diff --git a/source/libs/parser/src/parUtil.c b/source/libs/parser/src/parUtil.c index 0cda428487..a5758a17c3 100644 --- a/source/libs/parser/src/parUtil.c +++ b/source/libs/parser/src/parUtil.c @@ -227,6 +227,10 @@ static char* getSyntaxErrFormat(int32_t errCode) { return "Some functions cannot appear in the select list at the same time"; case TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR: return "Syntax error in regular expression"; + case TSDB_CODE_PAR_TRUE_FOR_NEGATIVE: + return "True_for duration cannot be negative"; + case TSDB_CODE_PAR_TRUE_FOR_UNIT: + return "Cannot use 'year' or 'month' as true_for duration"; default: return "Unknown error"; } diff --git a/source/libs/planner/src/planLogicCreater.c b/source/libs/planner/src/planLogicCreater.c index 3f064f2b66..98b2ef341e 100644 --- a/source/libs/planner/src/planLogicCreater.c +++ b/source/libs/planner/src/planLogicCreater.c @@ -123,6 +123,7 @@ static EDealRes doRewriteExpr(SNode** pNode, void* pContext) { tstrncpy(pCol->node.userAlias, ((SExprNode*)pExpr)->userAlias, TSDB_COL_NAME_LEN); tstrncpy(pCol->colName, ((SExprNode*)pExpr)->aliasName, TSDB_COL_NAME_LEN); pCol->node.projIdx = ((SExprNode*)(*pNode))->projIdx; + pCol->node.relatedTo = ((SExprNode*)(*pNode))->relatedTo; if (QUERY_NODE_FUNCTION == nodeType(pExpr)) { setColumnInfo((SFunctionNode*)pExpr, pCol, pCxt->isPartitionBy); } @@ -150,7 +151,7 @@ static EDealRes doNameExpr(SNode* pNode, void* pContext) { case QUERY_NODE_LOGIC_CONDITION: case QUERY_NODE_FUNCTION: { if ('\0' == ((SExprNode*)pNode)->aliasName[0]) { - snprintf(((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN, "#expr_%p", pNode); + rewriteExprAliasName((SExprNode*)pNode, (int64_t)pNode); } return DEAL_RES_IGNORE_CHILD; } @@ -757,6 +758,7 @@ static SColumnNode* createColumnByExpr(const char* pStmtName, SExprNode* pExpr) if (NULL != pStmtName) { snprintf(pCol->tableAlias, sizeof(pCol->tableAlias), "%s", pStmtName); } + pCol->node.relatedTo = pExpr->relatedTo; return pCol; } @@ -1159,6 +1161,9 @@ static int32_t createWindowLogicNodeByState(SLogicPlanContext* pCxt, SStateWindo nodesDestroyNode((SNode*)pWindow); return code; } + if (pState->pTrueForLimit) { + pWindow->trueForLimit = ((SValueNode*)pState->pTrueForLimit)->datum.i; + } // rewrite the expression in subsequent clauses code = rewriteExprForSelect(pWindow->pStateExpr, pSelect, SQL_CLAUSE_WINDOW); if (TSDB_CODE_SUCCESS == code) { @@ -1272,6 +1277,9 @@ static int32_t createWindowLogicNodeByEvent(SLogicPlanContext* pCxt, SEventWindo nodesDestroyNode((SNode*)pWindow); return TSDB_CODE_OUT_OF_MEMORY; } + if (pEvent->pTrueForLimit) { + pWindow->trueForLimit = ((SValueNode*)pEvent->pTrueForLimit)->datum.i; + } return createWindowLogicNodeFinalize(pCxt, pSelect, pWindow, pLogicNode); } diff --git a/source/libs/planner/src/planOptimizer.c b/source/libs/planner/src/planOptimizer.c index e8a8e6889a..367d0f4b4f 100644 --- a/source/libs/planner/src/planOptimizer.c +++ b/source/libs/planner/src/planOptimizer.c @@ -3050,7 +3050,7 @@ static int32_t smaIndexOptCreateSmaCols(SNodeList* pFuncs, uint64_t tableId, SNo } SExprNode exprNode; exprNode.resType = ((SExprNode*)pWsNode)->resType; - snprintf(exprNode.aliasName, TSDB_COL_NAME_LEN, "#expr_%d", index + 1); + rewriteExprAliasName(&exprNode, index + 1); SColumnNode* pkNode = NULL; code = smaIndexOptCreateSmaCol((SNode*)&exprNode, tableId, PRIMARYKEY_TIMESTAMP_COL_ID, &pkNode); if (TSDB_CODE_SUCCESS != code) { diff --git a/source/libs/planner/src/planPhysiCreater.c b/source/libs/planner/src/planPhysiCreater.c index 316699bba1..e12ec77bff 100644 --- a/source/libs/planner/src/planPhysiCreater.c +++ b/source/libs/planner/src/planPhysiCreater.c @@ -13,6 +13,7 @@ * along with this program. If not, see . */ +#include "nodes.h" #include "planInt.h" #include "catalog.h" @@ -30,98 +31,82 @@ typedef struct SSlotIndex { SArray* pSlotIdsInfo; // duplicate name slot } SSlotIndex; +enum { + SLOT_KEY_TYPE_ALL = 1, + SLOT_KEY_TYPE_COLNAME = 2, +}; + +static int32_t getSlotKeyHelper(SNode* pNode, const char* pPreName, const char* name, char** ppKey, int32_t callocLen, + int32_t* pLen, uint16_t extraBufLen, int8_t slotKeyType) { + int32_t code = 0; + *ppKey = taosMemoryCalloc(1, callocLen); + if (!*ppKey) { + return terrno; + } + if (slotKeyType == SLOT_KEY_TYPE_ALL) { + TAOS_STRNCAT(*ppKey, pPreName, TSDB_TABLE_NAME_LEN); + TAOS_STRNCAT(*ppKey, ".", 2); + TAOS_STRNCAT(*ppKey, name, TSDB_COL_NAME_LEN); + *pLen = taosHashBinary(*ppKey, strlen(*ppKey)); + } else { + TAOS_STRNCAT(*ppKey, name, TSDB_COL_NAME_LEN); + *pLen = strlen(*ppKey); + } + + return code; +} + static int32_t getSlotKey(SNode* pNode, const char* pStmtName, char** ppKey, int32_t* pLen, uint16_t extraBufLen) { int32_t code = 0; + int32_t callocLen = 0; if (QUERY_NODE_COLUMN == nodeType(pNode)) { SColumnNode* pCol = (SColumnNode*)pNode; if (NULL != pStmtName) { if ('\0' != pStmtName[0]) { - *ppKey = taosMemoryCalloc(1, TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pStmtName, TSDB_TABLE_NAME_LEN); - TAOS_STRNCAT(*ppKey, ".", 2); - TAOS_STRNCAT(*ppKey, pCol->node.aliasName, TSDB_COL_NAME_LEN); - *pLen = taosHashBinary(*ppKey, strlen(*ppKey)); - return code; + callocLen = TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pStmtName, pCol->node.aliasName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_ALL); } else { - *ppKey = taosMemoryCalloc(1, TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pCol->node.aliasName, TSDB_COL_NAME_LEN); - *pLen = strlen(*ppKey); - return code; + callocLen = TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pStmtName, pCol->node.aliasName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_COLNAME); } } if ('\0' == pCol->tableAlias[0]) { - *ppKey = taosMemoryCalloc(1, TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pCol->colName, TSDB_COL_NAME_LEN); - *pLen = strlen(*ppKey); - return code; + callocLen = TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pStmtName, pCol->colName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_COLNAME); } - - *ppKey = taosMemoryCalloc(1, TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pCol->tableAlias, TSDB_TABLE_NAME_LEN); - TAOS_STRNCAT(*ppKey, ".", 2); - TAOS_STRNCAT(*ppKey, pCol->colName, TSDB_COL_NAME_LEN); - *pLen = taosHashBinary(*ppKey, strlen(*ppKey)); - return code; + callocLen = TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pCol->tableAlias, pCol->colName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_ALL); } else if (QUERY_NODE_FUNCTION == nodeType(pNode)) { SFunctionNode* pFunc = (SFunctionNode*)pNode; if (FUNCTION_TYPE_TBNAME == pFunc->funcType) { SValueNode* pVal = (SValueNode*)nodesListGetNode(pFunc->pParameterList, 0); if (pVal) { if (NULL != pStmtName && '\0' != pStmtName[0]) { - *ppKey = taosMemoryCalloc(1, TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pStmtName, TSDB_TABLE_NAME_LEN); - TAOS_STRNCAT(*ppKey, ".", 2); - TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN); - *pLen = taosHashBinary(*ppKey, strlen(*ppKey)); - return code; + callocLen = TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pStmtName, ((SExprNode*)pNode)->aliasName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_ALL); } int32_t literalLen = strlen(pVal->literal); - *ppKey = taosMemoryCalloc(1, literalLen + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pVal->literal, literalLen); - TAOS_STRNCAT(*ppKey, ".", 2); - TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN); - *pLen = taosHashBinary(*ppKey, strlen(*ppKey)); - return code; + callocLen = literalLen + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pVal->literal, ((SExprNode*)pNode)->aliasName, ppKey, callocLen, pLen, + extraBufLen, SLOT_KEY_TYPE_ALL); } } } if (NULL != pStmtName && '\0' != pStmtName[0]) { - *ppKey = taosMemoryCalloc(1, TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, pStmtName, TSDB_TABLE_NAME_LEN); - TAOS_STRNCAT(*ppKey, ".", 2); - TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN); - *pLen = taosHashBinary(*ppKey, strlen(*ppKey)); - return code; + callocLen = TSDB_TABLE_NAME_LEN + 1 + TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pStmtName, ((SExprNode*)pNode)->aliasName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_ALL); } - *ppKey = taosMemoryCalloc(1, TSDB_COL_NAME_LEN + 1 + extraBufLen); - if (!*ppKey) { - return terrno; - } - TAOS_STRNCAT(*ppKey, ((SExprNode*)pNode)->aliasName, TSDB_COL_NAME_LEN); - *pLen = strlen(*ppKey); + callocLen = TSDB_COL_NAME_LEN + 1 + extraBufLen; + return getSlotKeyHelper(pNode, pStmtName, ((SExprNode*)pNode)->aliasName, ppKey, callocLen, pLen, extraBufLen, + SLOT_KEY_TYPE_COLNAME); return code; } @@ -2328,6 +2313,8 @@ static int32_t createStateWindowPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pC // } } + pState->trueForLimit = pWindowLogicNode->trueForLimit; + if (TSDB_CODE_SUCCESS == code) { code = createWindowPhysiNodeFinalize(pCxt, pChildren, &pState->window, pWindowLogicNode); } @@ -2358,6 +2345,7 @@ static int32_t createEventWindowPhysiNode(SPhysiPlanContext* pCxt, SNodeList* pC if (TSDB_CODE_SUCCESS == code) { code = setNodeSlotId(pCxt, pChildTupe->dataBlockId, -1, pWindowLogicNode->pEndCond, &pEvent->pEndCond); } + pEvent->trueForLimit = pWindowLogicNode->trueForLimit; if (TSDB_CODE_SUCCESS == code) { code = createWindowPhysiNodeFinalize(pCxt, pChildren, &pEvent->window, pWindowLogicNode); } diff --git a/source/libs/planner/src/planUtil.c b/source/libs/planner/src/planUtil.c index 1cc8c93d29..a31b132cae 100644 --- a/source/libs/planner/src/planUtil.c +++ b/source/libs/planner/src/planUtil.c @@ -79,6 +79,7 @@ static EDealRes doCreateColumn(SNode* pNode, void* pContext) { } } } + pCol->node.relatedTo = pExpr->relatedTo; return (TSDB_CODE_SUCCESS == nodesListStrictAppend(pCxt->pList, (SNode*)pCol) ? DEAL_RES_IGNORE_CHILD : DEAL_RES_ERROR); } diff --git a/source/libs/scalar/src/sclfunc.c b/source/libs/scalar/src/sclfunc.c index ce431c0b18..f3b56da372 100644 --- a/source/libs/scalar/src/sclfunc.c +++ b/source/libs/scalar/src/sclfunc.c @@ -4402,3 +4402,4 @@ int32_t uniqueScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarPara int32_t modeScalarFunction(SScalarParam *pInput, int32_t inputNum, SScalarParam *pOutput) { return selectScalarFunction(pInput, inputNum, pOutput); } + diff --git a/source/libs/stream/src/streamCheckpoint.c b/source/libs/stream/src/streamCheckpoint.c index a43cdd0b85..c53e1a19a3 100644 --- a/source/libs/stream/src/streamCheckpoint.c +++ b/source/libs/stream/src/streamCheckpoint.c @@ -595,68 +595,71 @@ void streamTaskClearCheckInfo(SStreamTask* pTask, bool clearChkpReadyMsg) { pTask->id.idStr, pInfo->failedId, pTask->chkInfo.checkpointId); } -int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SVUpdateCheckpointInfoReq* pReq) { +// The checkpointInfo can be updated in the following three cases: +// 1. follower tasks; 2. leader task with status of TASK_STATUS__CK; 3. restore not completed +static int32_t doUpdateCheckpointInfoCheck(SStreamTask* pTask, bool restored, SVUpdateCheckpointInfoReq* pReq, + bool* pContinue) { SStreamMeta* pMeta = pTask->pMeta; int32_t vgId = pMeta->vgId; int32_t code = 0; const char* id = pTask->id.idStr; SCheckpointInfo* pInfo = &pTask->chkInfo; - streamMutexLock(&pTask->lock); + *pContinue = true; // not update the checkpoint info if the checkpointId is less than the failed checkpointId if (pReq->checkpointId < pInfo->pActiveInfo->failedId) { stWarn("s-task:%s vgId:%d not update the checkpoint-info, since update checkpointId:%" PRId64 - " is less than the failed checkpointId:%" PRId64 ", discard the update info", + " is less than the failed checkpointId:%" PRId64 ", discard", id, vgId, pReq->checkpointId, pInfo->pActiveInfo->failedId); - streamMutexUnlock(&pTask->lock); - // always return true + *pContinue = false; return TSDB_CODE_SUCCESS; } + // it's an expired checkpointInfo update msg, we still try to drop the required drop fill-history task. if (pReq->checkpointId <= pInfo->checkpointId) { stDebug("s-task:%s vgId:%d latest checkpointId:%" PRId64 " Ver:%" PRId64 " no need to update checkpoint info, updated checkpointId:%" PRId64 " Ver:%" PRId64 " transId:%d ignored", id, vgId, pInfo->checkpointId, pInfo->checkpointVer, pReq->checkpointId, pReq->checkpointVer, pReq->transId); - streamMutexUnlock(&pTask->lock); - { // destroy the related fill-history tasks - // drop task should not in the meta-lock, and drop the related fill-history task now - if (pReq->dropRelHTask) { - code = streamMetaUnregisterTask(pMeta, pReq->hStreamId, pReq->hTaskId); - int32_t numOfTasks = streamMetaGetNumOfTasks(pMeta); - stDebug("s-task:%s vgId:%d related fill-history task:0x%x dropped in update checkpointInfo, remain tasks:%d", - id, vgId, pReq->taskId, numOfTasks); - } + { // destroy the related fill-history tasks + if (pReq->dropRelHTask) { + code = streamMetaUnregisterTask(pMeta, pReq->hStreamId, pReq->hTaskId); - if (pReq->dropRelHTask) { - code = streamMetaCommit(pMeta); - } - } + int32_t numOfTasks = streamMetaGetNumOfTasks(pMeta); + stDebug("s-task:%s vgId:%d related fill-history task:0x%x dropped in update checkpointInfo, remain tasks:%d", + id, vgId, pReq->taskId, numOfTasks); + //todo: task may not exist, commit anyway, optimize this later + code = streamMetaCommit(pMeta); + } + } + + *pContinue = false; // always return true return TSDB_CODE_SUCCESS; } - SStreamTaskState pStatus = streamTaskGetStatus(pTask); + SStreamTaskState status = streamTaskGetStatus(pTask); if (!restored) { // during restore procedure, do update checkpoint-info stDebug("s-task:%s vgId:%d status:%s update the checkpoint-info during restore, checkpointId:%" PRId64 "->%" PRId64 " checkpointVer:%" PRId64 "->%" PRId64 " checkpointTs:%" PRId64 "->%" PRId64, - id, vgId, pStatus.name, pInfo->checkpointId, pReq->checkpointId, pInfo->checkpointVer, pReq->checkpointVer, + id, vgId, status.name, pInfo->checkpointId, pReq->checkpointId, pInfo->checkpointVer, pReq->checkpointVer, pInfo->checkpointTime, pReq->checkpointTs); } else { // not in restore status, must be in checkpoint status - if ((pStatus.state == TASK_STATUS__CK) || (pMeta->role == NODE_ROLE_FOLLOWER)) { - stDebug("s-task:%s vgId:%d status:%s role:%d start to update the checkpoint-info, checkpointId:%" PRId64 "->%" PRId64 - " checkpointVer:%" PRId64 "->%" PRId64 " checkpointTs:%" PRId64 "->%" PRId64, - id, vgId, pStatus.name, pMeta->role, pInfo->checkpointId, pReq->checkpointId, pInfo->checkpointVer, + if (((status.state == TASK_STATUS__CK) && (pMeta->role == NODE_ROLE_LEADER)) || + (pMeta->role == NODE_ROLE_FOLLOWER)) { + stDebug("s-task:%s vgId:%d status:%s role:%d start to update the checkpoint-info, checkpointId:%" PRId64 + "->%" PRId64 " checkpointVer:%" PRId64 "->%" PRId64 " checkpointTs:%" PRId64 "->%" PRId64, + id, vgId, status.name, pMeta->role, pInfo->checkpointId, pReq->checkpointId, pInfo->checkpointVer, pReq->checkpointVer, pInfo->checkpointTime, pReq->checkpointTs); } else { stDebug("s-task:%s vgId:%d status:%s NOT update the checkpoint-info, checkpointId:%" PRId64 "->%" PRId64 " checkpointVer:%" PRId64 "->%" PRId64, - id, vgId, pStatus.name, pInfo->checkpointId, pReq->checkpointId, pInfo->checkpointVer, + id, vgId, status.name, pInfo->checkpointId, pReq->checkpointId, pInfo->checkpointVer, pReq->checkpointVer); } } @@ -665,14 +668,48 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV pInfo->processedVer <= pReq->checkpointVer); if (!valid) { - stFatal("s-task:%s invalid checkpointId update info recv, current checkpointId:%" PRId64 " checkpointVer:%" PRId64 - " processedVer:%" PRId64 " req checkpointId:%" PRId64 " checkpointVer:%" PRId64 " discard it", - id, pInfo->checkpointId, pInfo->checkpointVer, pInfo->processedVer, pReq->checkpointId, - pReq->checkpointVer); - streamMutexUnlock(&pTask->lock); - return TSDB_CODE_STREAM_INTERNAL_ERROR; + // invalid update checkpoint info for leader, since the processedVer is greater than the checkpointVer + // It is possible for follower tasks that the processedVer is greater than the checkpointVer, and the processed info + // in follower tasks will be discarded, since the leader/follower switch happens before the checkpoint of the + // processedVer being generated. + if (pMeta->role == NODE_ROLE_LEADER) { + + stFatal("s-task:%s checkpointId update info recv, current checkpointId:%" PRId64 " checkpointVer:%" PRId64 + " processedVer:%" PRId64 " req checkpointId:%" PRId64 " checkpointVer:%" PRId64 " discard it", + id, pInfo->checkpointId, pInfo->checkpointVer, pInfo->processedVer, pReq->checkpointId, + pReq->checkpointVer); + + *pContinue = false; + return TSDB_CODE_STREAM_INTERNAL_ERROR; + } else { + stInfo("s-task:%s vgId:%d follower recv checkpointId update info, current checkpointId:%" PRId64 + " checkpointVer:%" PRId64 " processedVer:%" PRId64 " req checkpointId:%" PRId64 " checkpointVer:%" PRId64, + id, pMeta->vgId, pInfo->checkpointId, pInfo->checkpointVer, pInfo->processedVer, pReq->checkpointId, + pReq->checkpointVer); + } } + return TSDB_CODE_SUCCESS; +} + +int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SVUpdateCheckpointInfoReq* pReq) { + SStreamMeta* pMeta = pTask->pMeta; + int32_t vgId = pMeta->vgId; + int32_t code = 0; + const char* id = pTask->id.idStr; + SCheckpointInfo* pInfo = &pTask->chkInfo; + bool continueUpdate = true; + + streamMutexLock(&pTask->lock); + code = doUpdateCheckpointInfoCheck(pTask, restored, pReq, &continueUpdate); + + if (!continueUpdate) { + streamMutexUnlock(&pTask->lock); + return code; + } + + SStreamTaskState pStatus = streamTaskGetStatus(pTask); + // update only it is in checkpoint status, or during restore procedure. if ((pStatus.state == TASK_STATUS__CK) || (!restored) || (pMeta->role == NODE_ROLE_FOLLOWER)) { pInfo->checkpointId = pReq->checkpointId; @@ -697,7 +734,7 @@ int32_t streamTaskUpdateTaskCheckpointInfo(SStreamTask* pTask, bool restored, SV pTask->status.taskStatus = TASK_STATUS__READY; - code = streamMetaSaveTaskInMeta(pMeta, pTask); + code = streamMetaSaveTask(pMeta, pTask); streamMutexUnlock(&pTask->lock); if (code != TSDB_CODE_SUCCESS) { @@ -1537,14 +1574,6 @@ int32_t deleteCheckpointFile(const char* id, const char* name) { int32_t streamTaskSendNegotiateChkptIdMsg(SStreamTask* pTask) { streamMutexLock(&pTask->lock); ETaskStatus p = streamTaskGetStatus(pTask).state; - // if (pInfo->alreadySendChkptId == true) { - // stDebug("s-task:%s already start to consensus-checkpointId, not start again before it completed", id); - // streamMutexUnlock(&pTask->lock); - // return TSDB_CODE_SUCCESS; - // } else { - // pInfo->alreadySendChkptId = true; - // } - // streamTaskSetReqConsenChkptId(pTask, taosGetTimestampMs()); streamMutexUnlock(&pTask->lock); diff --git a/source/libs/stream/src/streamExec.c b/source/libs/stream/src/streamExec.c index 4e9e236507..ee34648a47 100644 --- a/source/libs/stream/src/streamExec.c +++ b/source/libs/stream/src/streamExec.c @@ -875,7 +875,7 @@ static int32_t doStreamExecTask(SStreamTask* pTask) { } double el = (taosGetTimestampMs() - st) / 1000.0; - if (el > 2.0) { // elapsed more than 5 sec, not occupy the CPU anymore + if (el > 5.0) { // elapsed more than 5 sec, not occupy the CPU anymore stDebug("s-task:%s occupy more than 5.0s, release the exec threads and idle for 500ms", id); streamTaskSetIdleInfo(pTask, 500); return code; diff --git a/source/libs/stream/src/streamMeta.c b/source/libs/stream/src/streamMeta.c index bb15eb1e6a..7482c6229b 100644 --- a/source/libs/stream/src/streamMeta.c +++ b/source/libs/stream/src/streamMeta.c @@ -633,7 +633,7 @@ void streamMetaCloseImpl(void* arg) { } // todo let's check the status for each task -int32_t streamMetaSaveTaskInMeta(SStreamMeta* pMeta, SStreamTask* pTask) { +int32_t streamMetaSaveTask(SStreamMeta* pMeta, SStreamTask* pTask) { int32_t vgId = pTask->pMeta->vgId; void* buf = NULL; int32_t len; @@ -683,7 +683,7 @@ int32_t streamMetaSaveTaskInMeta(SStreamMeta* pMeta, SStreamTask* pTask) { return code; } -int32_t streamMetaRemoveTaskInMeta(SStreamMeta* pMeta, STaskId* pTaskId) { +int32_t streamMetaRemoveTask(SStreamMeta* pMeta, STaskId* pTaskId) { int64_t key[2] = {pTaskId->streamId, pTaskId->taskId}; int32_t code = tdbTbDelete(pMeta->pTaskDb, key, STREAM_TASK_KEY_LEN, pMeta->txn); if (code != 0) { @@ -706,7 +706,7 @@ int32_t streamMetaRegisterTask(SStreamMeta* pMeta, int64_t ver, SStreamTask* pTa void* p = taosHashGet(pMeta->pTasksMap, &id, sizeof(id)); if (p != NULL) { - stDebug("s-task:0x%" PRIx64 " already exist in meta, no need to register", id.taskId); + stDebug("s-task:%" PRIx64 " already exist in meta, no need to register", id.taskId); tFreeStreamTask(pTask); return code; } @@ -736,7 +736,7 @@ int32_t streamMetaRegisterTask(SStreamMeta* pMeta, int64_t ver, SStreamTask* pTa return code; } - if ((code = streamMetaSaveTaskInMeta(pMeta, pTask)) != 0) { + if ((code = streamMetaSaveTask(pMeta, pTask)) != 0) { int32_t unused = taosHashRemove(pMeta->pTasksMap, &id, sizeof(id)); void* pUnused = taosArrayPop(pMeta->pTaskList); @@ -886,8 +886,6 @@ static void doRemoveIdFromList(SArray* pTaskList, int32_t num, SStreamTaskId* id static int32_t streamTaskSendTransSuccessMsg(SStreamTask* pTask, void* param) { int32_t code = 0; - int32_t waitingDuration = 5000; - if (pTask->info.taskLevel == TASK_LEVEL__SOURCE) { code = streamTaskSendCheckpointSourceRsp(pTask); if (code) { @@ -898,7 +896,7 @@ static int32_t streamTaskSendTransSuccessMsg(SStreamTask* pTask, void* param) { // let's kill the query procedure within stream, to end it ASAP. if (pTask->info.taskLevel != TASK_LEVEL__SINK && pTask->exec.pExecutor != NULL) { - code = qKillTask(pTask->exec.pExecutor, TSDB_CODE_SUCCESS, -1); + code = qKillTask(pTask->exec.pExecutor, TSDB_CODE_SUCCESS); if (code != TSDB_CODE_SUCCESS) { stError("s-task:%s failed to kill task related query handle, code:%s", pTask->id.idStr, tstrerror(code)); } @@ -935,7 +933,7 @@ int32_t streamMetaUnregisterTask(SStreamMeta* pMeta, int64_t streamId, int32_t t code = taosHashRemove(pMeta->pTasksMap, &id, sizeof(id)); doRemoveIdFromList(pMeta->pTaskList, (int32_t)taosArrayGetSize(pMeta->pTaskList), &pTask->id); - code = streamMetaRemoveTaskInMeta(pMeta, &id); + code = streamMetaRemoveTask(pMeta, &id); if (code) { stError("vgId:%d failed to remove task:0x%" PRIx64 ", code:%s", pMeta->vgId, id.taskId, tstrerror(code)); } @@ -966,32 +964,6 @@ int32_t streamMetaUnregisterTask(SStreamMeta* pMeta, int64_t streamId, int32_t t return 0; } -int32_t streamMetaStopOneTask(SStreamMeta* pMeta, int64_t streamId, int32_t taskId) { - SStreamTask* pTask = NULL; - int32_t code = 0; - int32_t vgId = pMeta->vgId; - int32_t numOfTasks = 0; - - streamMetaWLock(pMeta); - -// code = streamMetaUnregisterTask(pMeta, streamId, taskId); -// numOfTasks = streamMetaGetNumOfTasks(pMeta); -// if (code) { -// stError("vgId:%d failed to drop task:0x%x, code:%s", vgId, taskId, tstrerror(code)); -// } -// -// code = streamMetaCommit(pMeta); -// if (code) { -// stError("vgId:%d failed to commit after drop task:0x%x, code:%s", vgId, taskId, tstrerror(code)); -// } else { -// stDebug("s-task:0x%"PRIx64"-0x%x vgId:%d dropped, remain tasks:%d", streamId, taskId, pMeta->vgId, numOfTasks); -// } - - streamMetaWUnLock(pMeta); - - return code; -} - int32_t streamMetaBegin(SStreamMeta* pMeta) { streamMetaWLock(pMeta); int32_t code = tdbBegin(pMeta->db, &pMeta->txn, tdbDefaultMalloc, tdbDefaultFree, NULL, @@ -1215,7 +1187,7 @@ void streamMetaLoadAllTasks(SStreamMeta* pMeta) { if (taosArrayGetSize(pRecycleList) > 0) { for (int32_t i = 0; i < taosArrayGetSize(pRecycleList); ++i) { STaskId* pId = taosArrayGet(pRecycleList, i); - code = streamMetaRemoveTaskInMeta(pMeta, pId); + code = streamMetaRemoveTask(pMeta, pId); if (code) { stError("s-task:0x%" PRIx64 " failed to remove task, code:%s", pId->taskId, tstrerror(code)); } diff --git a/source/libs/stream/src/streamStartHistory.c b/source/libs/stream/src/streamStartHistory.c index f8b1b5ecbc..54a8929123 100644 --- a/source/libs/stream/src/streamStartHistory.c +++ b/source/libs/stream/src/streamStartHistory.c @@ -76,7 +76,7 @@ int32_t streamStartScanHistoryAsync(SStreamTask* pTask, int8_t igUntreated) { memcpy(serializedReq, &req, len); SRpcMsg rpcMsg = {.contLen = len, .pCont = serializedReq, .msgType = TDMT_VND_STREAM_SCAN_HISTORY}; - return tmsgPutToQueue(pTask->pMsgCb, STREAM_LONG_EXEC_QUEUE, &rpcMsg); + return tmsgPutToQueue(pTask->pMsgCb, STREAM_QUEUE, &rpcMsg); } void streamExecScanHistoryInFuture(SStreamTask* pTask, int32_t idleDuration) { diff --git a/source/libs/stream/src/streamStartTask.c b/source/libs/stream/src/streamStartTask.c index d9ca506849..c40d5ef928 100644 --- a/source/libs/stream/src/streamStartTask.c +++ b/source/libs/stream/src/streamStartTask.c @@ -45,6 +45,10 @@ int32_t streamMetaStartAllTasks(SStreamMeta* pMeta) { if (numOfTasks == 0) { stInfo("vgId:%d no tasks exist, quit from consensus checkpointId", pMeta->vgId); + + streamMetaWLock(pMeta); + streamMetaResetStartInfo(&pMeta->startInfo, vgId); + streamMetaWUnLock(pMeta); return TSDB_CODE_SUCCESS; } @@ -447,6 +451,7 @@ int32_t streamMetaStopAllTasks(SStreamMeta* pMeta) { continue; } + int64_t refId = pTask->id.refId; int32_t ret = streamTaskStop(pTask); if (ret) { stError("s-task:0x%x failed to stop task, code:%s", pTaskId->taskId, tstrerror(ret)); diff --git a/source/libs/stream/src/streamTask.c b/source/libs/stream/src/streamTask.c index 378aaa27d0..7209b6434f 100644 --- a/source/libs/stream/src/streamTask.c +++ b/source/libs/stream/src/streamTask.c @@ -710,7 +710,7 @@ int32_t streamTaskStop(SStreamTask* pTask) { } if (pTask->info.taskLevel != TASK_LEVEL__SINK && pTask->exec.pExecutor != NULL) { - code = qKillTask(pTask->exec.pExecutor, TSDB_CODE_SUCCESS, 5000); + code = qKillTask(pTask->exec.pExecutor, TSDB_CODE_SUCCESS); if (code != TSDB_CODE_SUCCESS) { stError("s-task:%s failed to kill task related query handle, code:%s", id, tstrerror(code)); } @@ -869,7 +869,7 @@ int32_t streamTaskClearHTaskAttr(SStreamTask* pTask, int32_t resetRelHalt) { pStreamTask->status.taskStatus = TASK_STATUS__READY; } - code = streamMetaSaveTaskInMeta(pMeta, pStreamTask); + code = streamMetaSaveTask(pMeta, pStreamTask); streamMutexUnlock(&(pStreamTask->lock)); streamMetaReleaseTask(pMeta, pStreamTask); @@ -1034,7 +1034,7 @@ static int32_t taskPauseCallback(SStreamTask* pTask, void* param) { // in case of fill-history task, stop the tsdb file scan operation. if (pTask->info.fillHistory == 1) { void* pExecutor = pTask->exec.pExecutor; - code = qKillTask(pExecutor, TSDB_CODE_SUCCESS, 10000); + code = qKillTask(pExecutor, TSDB_CODE_SUCCESS); } stDebug("vgId:%d s-task:%s set pause flag and pause task", pMeta->vgId, pTask->id.idStr); @@ -1296,8 +1296,6 @@ const char* streamTaskGetExecType(int32_t type) { return "resume-task-from-idle"; case STREAM_EXEC_T_ADD_FAILED_TASK: return "record-start-failed-task"; - case STREAM_EXEC_T_STOP_ONE_TASK: - return "stop-one-task"; case 0: return "exec-all-tasks"; default: diff --git a/source/util/src/terror.c b/source/util/src/terror.c index f34b00bec5..98396d7016 100644 --- a/source/util/src/terror.c +++ b/source/util/src/terror.c @@ -564,6 +564,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_GRANT_MULTI_STORAGE_EXPIRED, "License expired for m TAOS_DEFINE_ERROR(TSDB_CODE_GRANT_OBJECT_STROAGE_EXPIRED, "License expired for object storage function") TAOS_DEFINE_ERROR(TSDB_CODE_GRANT_DUAL_REPLICA_HA_EXPIRED,"License expired for dual-replica HA function") TAOS_DEFINE_ERROR(TSDB_CODE_GRANT_DB_ENCRYPTION_EXPIRED, "License expired for database encryption function") +TAOS_DEFINE_ERROR(TSDB_CODE_GRANT_TD_GPT_EXPIRED, "License expired for TDgpt function") // sync TAOS_DEFINE_ERROR(TSDB_CODE_SYN_TIMEOUT, "Sync timeout") @@ -750,8 +751,15 @@ TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_ANOMALY_WIN_TYPE, "ANOMALY_WINDOW only TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_ANOMALY_WIN_COL, "ANOMALY_WINDOW not support on tag column") TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_ANOMALY_WIN_OPT, "ANOMALY_WINDOW option should include algo field") TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_FORECAST_CLAUSE, "Invalid forecast clause") -TAOS_DEFINE_ERROR(TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR, "Syntax error in regular expression") +TAOS_DEFINE_ERROR(TSDB_CODE_PAR_REGULAR_EXPRESSION_ERROR, "Syntax error in regular expression") TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_VGID_LIST, "Invalid vgid list") +TAOS_DEFINE_ERROR(TSDB_CODE_PAR_TRUE_FOR_NEGATIVE, "True_for duration cannot be negative") +TAOS_DEFINE_ERROR(TSDB_CODE_PAR_TRUE_FOR_UNIT, "Cannot use 'year' or 'month' as true_for duration") +TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_COLS_FUNCTION, "Invalid cols function") +TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INVALID_COLS_SELECTFUNC, "cols function's first param must be a select function that output a single row") +TAOS_DEFINE_ERROR(TSDB_CODE_INVALID_MULITI_COLS_FUNC, "Improper use of cols function with multiple output columns") +TAOS_DEFINE_ERROR(TSDB_CODE_INVALID_COLS_ALIAS, "Invalid using alias for cols function") + TAOS_DEFINE_ERROR(TSDB_CODE_PAR_INTERNAL_ERROR, "Parser internal error") //planner diff --git a/source/util/src/tworker.c b/source/util/src/tworker.c index 469f98fcf0..dbd8cb159e 100644 --- a/source/util/src/tworker.c +++ b/source/util/src/tworker.c @@ -256,7 +256,7 @@ static void *tAutoQWorkerThreadFp(SQueueWorker *worker) { return NULL; } -STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem fp, int32_t minNum) { +STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem fp) { int32_t code; STaosQueue *queue; @@ -280,10 +280,7 @@ STaosQueue *tAutoQWorkerAllocQueue(SAutoQWorkerPool *pool, void *ahandle, FItem int32_t queueNum = taosGetQueueNumber(pool->qset); int32_t curWorkerNum = taosArrayGetSize(pool->workers); int32_t dstWorkerNum = ceilf(queueNum * pool->ratio); - - if (dstWorkerNum < minNum) { - dstWorkerNum = minNum; - } + if (dstWorkerNum < 2) dstWorkerNum = 2; // spawn a thread to process queue while (curWorkerNum < dstWorkerNum) { diff --git a/source/util/test/errorCodeTable.ini b/source/util/test/errorCodeTable.ini index f67c8ab834..f66948504d 100644 --- a/source/util/test/errorCodeTable.ini +++ b/source/util/test/errorCodeTable.ini @@ -400,6 +400,7 @@ TSDB_CODE_GRANT_MULTI_STORAGE_EXPIRED = 0x8000082A TSDB_CODE_GRANT_OBJECT_STROAGE_EXPIRED = 0x8000082B TSDB_CODE_GRANT_DUAL_REPLICA_HA_EXPIRED = 0x8000082C TSDB_CODE_GRANT_DB_ENCRYPTION_EXPIRED = 0x8000082D +TSDB_CODE_GRANT_TD_GPT_EXPIRED = 0x8000082E TSDB_CODE_SYN_TIMEOUT = 0x80000903 TSDB_CODE_SYN_MISMATCHED_SIGNATURE = 0x80000907 TSDB_CODE_SYN_NOT_LEADER = 0x8000090C @@ -554,6 +555,8 @@ TSDB_CODE_PAR_COL_PK_TYPE = 0x80002679 TSDB_CODE_PAR_INVALID_PK_OP = 0x8000267A TSDB_CODE_PAR_PRIMARY_KEY_IS_NULL = 0x8000267B TSDB_CODE_PAR_PRIMARY_KEY_IS_NONE = 0x8000267C +TSDB_CODE_PAR_TRUE_FOR_NEGATIVE = 0x80002687 +TSDB_CODE_PAR_TRUE_FOR_UNIT = 0x80002688 TSDB_CODE_PAR_INTERNAL_ERROR = 0x800026FF TSDB_CODE_PLAN_INTERNAL_ERROR = 0x80002700 TSDB_CODE_PLAN_EXPECTED_TS_EQUAL = 0x80002701 diff --git a/tests/army/whole/checkErrorCode.py b/tests/army/whole/checkErrorCode.py index 1a14269f54..d66ae56e46 100644 --- a/tests/army/whole/checkErrorCode.py +++ b/tests/army/whole/checkErrorCode.py @@ -50,7 +50,7 @@ ignoreCodes = [ '0x80000734', '0x80000735', '0x80000736', '0x80000737', '0x80000738', '0x8000080E', '0x8000080F', '0x80000810', '0x80000811', '0x80000812', '0x80000813', '0x80000814', '0x80000815', '0x80000816', '0x80000817', '0x80000818', '0x80000819', '0x80000820', '0x80000821', '0x80000822', '0x80000823', '0x80000824', '0x80000825', '0x80000826', '0x80000827', '0x80000828', '0x80000829', '0x8000082A', '0x8000082B', '0x8000082C', - '0x8000082D', '0x80000907', '0x80000919', '0x8000091A', '0x8000091B', '0x8000091C', '0x8000091D', '0x8000091E', '0x8000091F', '0x80000A00', + '0x8000082D', '0x8000082E', '0x80000907', '0x80000919', '0x8000091A', '0x8000091B', '0x8000091C', '0x8000091D', '0x8000091E', '0x8000091F', '0x80000A00', '0x80000A01', '0x80000A03', '0x80000A06', '0x80000A07', '0x80000A08', '0x80000A09', '0x80000A0A', '0x80000A0B', '0x80000A0E', '0x80002206', '0x80002207', '0x80002406', '0x80002407', '0x80002503', '0x80002506', '0x80002507', '0x8000261B', '0x80002653', '0x80002668', '0x80002669', '0x8000266A', '0x8000266B', '0x8000266C', '0x8000266D', '0x8000266E', '0x8000266F', '0x80002670', '0x80002671', '0x80002672', '0x80002673', diff --git a/tests/ci/func.txt b/tests/ci/func.txt index c724568537..45d4fb1c11 100644 --- a/tests/ci/func.txt +++ b/tests/ci/func.txt @@ -79,7 +79,7 @@ (void)streamMetaAddFailedTask (void)streamMetaAddTaskLaunchResult (void)streamMetaCommit -(void)streamMetaRemoveTaskInMeta +(void)streamMetaRemoveTask (void)streamMetaSendHbHelper (void)streamMetaStartAllTasks (void)streamMetaStartOneTask diff --git a/tests/parallel_test/cases.task b/tests/parallel_test/cases.task index 50f5879a14..890e152554 100644 --- a/tests/parallel_test/cases.task +++ b/tests/parallel_test/cases.task @@ -470,8 +470,6 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_taosx.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts5466.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_td33504.py -,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts-5473.py -,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts-5776.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts5906.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/td-32187.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/td-33225.py @@ -1260,6 +1258,10 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/odbc.py ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/fill_with_group.py ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/state_window.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cols_function.py +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cols_function.py -Q 2 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cols_function.py -Q 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/cols_function.py -Q 4 ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-21561.py -Q 4 ,,y,system-test,./pytest.sh python3 ./test.py -f 99-TDcase/TD-20582.py ,,n,system-test,python3 ./test.py -f eco-system/meta/database/keep_time_offset.py @@ -1268,6 +1270,7 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/operator.py -Q 3 ,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/operator.py -Q 4 ,,y,system-test,./pytest.sh python3 ./test.py -f eco-system/manager/schema_change.py -N 3 -M 3 +,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/test_window_true_for.py #tsim test ,,y,script,./test.sh -f tsim/query/timeline.sim diff --git a/tests/parallel_test/cases_tdengine.task b/tests/parallel_test/cases_tdengine.task index e52fe68957..18c50322ad 100644 --- a/tests/parallel_test/cases_tdengine.task +++ b/tests/parallel_test/cases_tdengine.task @@ -298,7 +298,7 @@ ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/dataFromTsdbNWal-multiCtb.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_taosx.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts5466.py -,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts-5473.py +,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_c_test.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/td-32187.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/td-33225.py ,,y,system-test,./pytest.sh python3 ./test.py -f 7-tmq/tmq_ts4563.py diff --git a/tests/pytest/util/sql.py b/tests/pytest/util/sql.py index 8d247a690a..3c79c22bee 100644 --- a/tests/pytest/util/sql.py +++ b/tests/pytest/util/sql.py @@ -671,6 +671,15 @@ class TDSql: caller = inspect.getframeinfo(inspect.stack()[1][0]) args = (caller.filename, caller.lineno, self.sql, col_name_list, expect_col_name_list) tdLog.exit("%s(%d) failed: sql:%s, col_name_list:%s != expect_col_name_list:%s" % args) + + def checkResColNameList(self, expect_col_name_list): + col_name_list = [] + col_type_list = [] + for query_col in self.cursor.description: + col_name_list.append(query_col[0]) + col_type_list.append(query_col[1]) + + self.checkColNameList(col_name_list, expect_col_name_list) def __check_equal(self, elm, expect_elm): if elm == expect_elm: diff --git a/tests/pytest/util/tserror.py b/tests/pytest/util/tserror.py index 0d88d78e80..ba9b78e8eb 100644 --- a/tests/pytest/util/tserror.py +++ b/tests/pytest/util/tserror.py @@ -15,3 +15,5 @@ TSDB_CODE_UDF_FUNC_EXEC_FAILURE = (TAOS_DEF_ERROR_CODE | 0x290A) TSDB_CODE_TSC_INTERNAL_ERROR = (TAOS_DEF_ERROR_CODE | 0x02FF) TSDB_CODE_PAR_SYNTAX_ERROR = (TAOS_DEF_ERROR_CODE | 0x2600) + +TSDB_CODE_PAR_INVALID_COLS_FUNCTION = (TAOS_DEF_ERROR_CODE | 0x2689) diff --git a/tests/script/tsim/parser/having_child.sim b/tests/script/tsim/parser/having_child.sim index db9a25365e..093877c3dd 100644 --- a/tests/script/tsim/parser/having_child.sim +++ b/tests/script/tsim/parser/having_child.sim @@ -1294,7 +1294,10 @@ sql_error select avg(f1), spread(f1), spread(f2), spread(tb1.f1) from tb1 group sql_error select avg(f1), spread(f1), spread(f2), spread(tb1.f1) from tb1 group by f1 having spread(f1) > id1 and sum(f1); -sql_error select avg(f1), spread(f1), spread(f2), spread(tb1.f1) from tb1 group by f1 having spread(f1) > id1 and sum(f1) > 1; +sql select avg(f1), spread(f1), spread(f2), spread(tb1.f1) from tb1 group by f1 having spread(f1) > id1 and sum(f1) > 1; +if $rows != 0 then + return -1 +endi sql select avg(f1), spread(f1), spread(f2), spread(tb1.f1) from tb1 group by f1 having spread(f1) > 2 and sum(f1) > 1 order by f1; if $rows != 0 then diff --git a/tests/script/tsim/stream/basic1.sim b/tests/script/tsim/stream/basic1.sim index 0d89f528d1..9d31b1e073 100644 --- a/tests/script/tsim/stream/basic1.sim +++ b/tests/script/tsim/stream/basic1.sim @@ -809,7 +809,6 @@ sql create stable st(ts timestamp, a int, b int , c int) tags(ta int,tb int,tc i sql create table ts1 using st tags(1,1,1); sql create stream streams5 trigger at_once IGNORE EXPIRED 0 IGNORE UPDATE 0 into streamt5 as select count(*), _wstart, _wend, max(a) from ts1 interval(10s) ; sql create stream streams6 trigger at_once IGNORE EXPIRED 0 IGNORE UPDATE 0 into streamt6 as select count(*), _wstart, _wend, max(a), _wstart as ts from ts1 interval(10s) ; - run tsim/stream/checkTaskStatus.sim sql_error create stream streams7 trigger at_once into streamt7 as select _wstart, count(*), _wstart, _wend, max(a) from ts1 interval(10s) ; @@ -833,14 +832,14 @@ if $loop_count == 10 then endi if $rows != 1 then - print =====rows=$rows + print ===== streamt5: rows=$rows goto loop170 endi sql select * from streamt6; if $rows != 1 then - print =====rows=$rows + print ===== streamt6: rows=$rows goto loop170 endi diff --git a/tests/script/tsim/view/create_drop_view.sim b/tests/script/tsim/view/create_drop_view.sim index ca829f7456..7ec6a04345 100644 --- a/tests/script/tsim/view/create_drop_view.sim +++ b/tests/script/tsim/view/create_drop_view.sim @@ -1,3 +1,4 @@ +## \brief Test create view and drop view functions sql connect sql use testa; diff --git a/tests/system-test/0-others/grant.py b/tests/system-test/0-others/grant.py index 25af6eb842..038992b65a 100644 --- a/tests/system-test/0-others/grant.py +++ b/tests/system-test/0-others/grant.py @@ -237,7 +237,7 @@ class TDTestCase: tdLog.info(f"expireTime: {expireTime}, serviceTime: {serviceTime}") tdSql.checkEqual(True, abs(expireTime - serviceTime - 864000) < 15) tdSql.query(f'show grants full;') - nGrantItems = 31 + nGrantItems = 32 tdSql.checkEqual(len(tdSql.queryResult), nGrantItems) tdSql.checkEqual(tdSql.queryResult[0][2], serviceTimeStr) for i in range(1, nGrantItems): diff --git a/tests/system-test/0-others/information_schema.py b/tests/system-test/0-others/information_schema.py index 96cd596a20..63f0a236c3 100644 --- a/tests/system-test/0-others/information_schema.py +++ b/tests/system-test/0-others/information_schema.py @@ -283,6 +283,7 @@ class TDTestCase: 'active_active':'Active-Active', 'dual_replica':'Dual-Replica HA', 'db_encryption':'Database Encryption', + 'tdgpt':'TDgpt', 'opc_da':'OPC_DA', 'opc_ua':'OPC_UA', 'pi':'Pi', diff --git a/tests/system-test/2-query/cols_function.py b/tests/system-test/2-query/cols_function.py new file mode 100644 index 0000000000..bacab811ea --- /dev/null +++ b/tests/system-test/2-query/cols_function.py @@ -0,0 +1,1331 @@ +import random +import string +from util.log import * +from util.cases import * +from util.sql import * +from util.common import * +from util.tserror import * +import numpy as np + + +class TDTestCase: + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + + self.dbname = 'test' + + def condition_check(self, condition, row, col, expected_value): + if condition: + tdSql.checkData(row, col, expected_value) + + def create_test_data(self): + tdSql.execute(f'create database if not exists {self.dbname};') + tdSql.execute(f'use {self.dbname}') + tdSql.execute(f'drop table if exists {self.dbname}.meters') + + # tdLog.info("create test data") + # tdLog.info("taosBenchmark -y -t 10 -n 100 -b INT,FLOAT,NCHAR,BOOL") + # os.system("taosBenchmark -y -t 10 -n 100 -b INT,FLOAT,NCHAR,BOOL") + + tdSql.execute(f'create table {self.dbname}.meters (ts timestamp, c0 int, c1 float, c2 nchar(30), c3 bool) tags (t1 nchar(30))') + tdSql.execute(f'create table {self.dbname}.d0 using {self.dbname}.meters tags("st1")') + tdSql.execute(f'create table {self.dbname}.d1 using {self.dbname}.meters tags("st2")') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929000, 1, 1, "c2", true)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929001, 2, 2, "bbbbbbbbb1", false)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929002, 2, 2, "bbbbbbbbb1", false)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929003, 3, 3, "a2", true)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929004, 4, 4, "bbbbbbbbb2", false)') + + tdSql.execute(f'insert into {self.dbname}.d1 values(1734574929000, 1, 1, "c2", true)') + + tdSql.execute(f'use {self.dbname}') + tdSql.execute(f'Create table {self.dbname}.normal_table (ts timestamp, c0 int, c1 float, c2 nchar(30), c3 bool)') + tdSql.execute(f'insert into {self.dbname}.normal_table (select * from {self.dbname}.d0)') + + def one_cols_1output_test(self): + tdLog.info("one_cols_1output_test") + tdSql.query(f'select cols(last(ts), ts) from {self.dbname}.meters') + tdSql.checkResColNameList(['ts']) + tdSql.query(f'select cols(last(ts), ts) as t1 from {self.dbname}.meters') + tdSql.checkResColNameList(['t1']) + tdSql.query(f'select cols(last(ts), ts as t1) from {self.dbname}.meters') + tdSql.checkResColNameList(['t1']) + tdSql.query(f'select cols(last(ts), c0) from {self.dbname}.meters') + tdSql.checkResColNameList(['c0']) + tdSql.query(f'select cols(last(ts), c1) from {self.dbname}.meters group by tbname') + tdSql.checkResColNameList(['c1']) + + + tdSql.query(f'select cols(last(ts+1), ts) as t1 from {self.dbname}.meters') + tdSql.checkResColNameList(['t1']) + tdSql.query(f'select cols(last(ts+1), ts+2 as t1) from {self.dbname}.meters') + tdSql.checkResColNameList(['t1']) + tdSql.query(f'select cols(last(ts+1), c0+10) from {self.dbname}.meters') + tdSql.checkResColNameList(['c0+10']) + + def one_cols_multi_output_with_group_test(self, from_table = 'test.meters', isTmpTable = False): + select_t1 = ["", ", t1", ", t1 as tag1"] + for t1 in select_t1: + tags_count = 0 if t1 == "" else 1 + tdLog.info("one_cols_1output_test_with_group") + tdSql.query(f'select cols(last(c1), ts) {t1} from {from_table} group by tbname') + tdSql.checkRows(2) + tdSql.checkCols(1 + tags_count) + tdSql.query(f'select cols(last(c1), ts) {t1} from {from_table} group by tbname order by tbname') + tdSql.checkCols(1 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(1, 0, 1734574929000) + tdSql.query(f'select cols(last(c1), ts) {t1} from {from_table} group by tbname order by ts') + tdSql.checkCols(1 + tags_count) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(1, 0, 1734574929004) + tdSql.query(f'select cols(last(c1), ts), tbname {t1} from {from_table} group by tbname order by tbname') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 'd0') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 'd1') + tdSql.query(f'select cols(last(c1), ts), tbname, t1 from {from_table} group by tbname order by tbname') + tdSql.checkRows(2) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 'd0') + tdSql.checkData(0, 2, 'st1') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 'd1') + tdSql.checkData(1, 2, 'st2') + tdSql.query(f'select cols(last(c1), ts), tbname, t1 from {from_table} group by tbname order by t1') + tdSql.checkResColNameList(['ts', 'tbname', 't1']) + tdSql.checkRows(2) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 'd0') + tdSql.checkData(0, 2, 'st1') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 'd1') + tdSql.checkData(1, 2, 'st2') + tdSql.query(f'select cols(last(ts), ts, c0), count(1) {t1} from {from_table} group by t1 order by t1') + tdSql.checkRows(2) + tdSql.checkCols(3 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 5) + self.condition_check(t1!="", 0, 3, 'st1') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 1) + tdSql.checkData(1, 2, 1) + self.condition_check(t1!="", 1, 3, 'st2') + + tdSql.query(f'select cols(last(ts), ts, c0), sum(c0) {t1} from {from_table} group by t1 order by t1') + tdSql.checkRows(2) + tdSql.checkCols(3 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 12) + self.condition_check(t1!="", 0, 3, 'st1') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 1) + tdSql.checkData(1, 2, 1) + self.condition_check(t1!="", 1, 3, 'st2') + + tdSql.error(f'select cols(last(ts), ts, c0), count(1), t1 from {from_table} group by t1 order by tbname') + + if t1 != "" and isTmpTable: + # Not a GROUP BY expression + tdSql.error(f'select cols(last(ts), ts, c0), count(1) {t1} from {from_table} group by tbname order by tbname') + tdSql.error(f'select cols(last(ts), ts, c0), count(1), tbname {t1} from {from_table} group by tbname order by tbname') + tdSql.error(f'select cols(max(c0), ts, c0), count(1), tbname {t1} from {from_table} group by tbname order by tbname') + tdSql.error(f'select cols(last(c1), ts), count(1) {t1} from {from_table} group by tbname') + continue + tdSql.query(f'select cols(last(ts), ts, c0), count(1) {t1} from {from_table} group by tbname order by tbname') + tdSql.checkRows(2) + tdSql.checkCols(3 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 5) + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 1) + tdSql.checkData(1, 2, 1) + tdSql.query(f'select cols(last(ts), ts, c0), count(1), tbname {t1} from {from_table} group by tbname order by tbname') + tdSql.checkRows(2) + tdSql.checkCols(4 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 5) + tdSql.checkData(0, 3, 'd0') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 1) + tdSql.checkData(1, 2, 1) + tdSql.checkData(1, 3, 'd1') + tdSql.query(f'select cols(max(c0), ts, c0), count(1), tbname {t1} from {from_table} group by tbname order by tbname') + tdSql.checkRows(2) + tdSql.checkCols(4 + tags_count) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 5) + tdSql.checkData(0, 3, 'd0') + self.condition_check(t1!="", 0, 4, 'st1') + tdSql.checkData(1, 0, 1734574929000) + tdSql.checkData(1, 1, 1) + tdSql.checkData(1, 2, 1) + tdSql.checkData(1, 3, 'd1') + self.condition_check(t1!="", 1, 4, 'st2') + + tdSql.query(f'select cols(last(c1), ts), count(1) {t1} from {from_table} group by tbname') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + + def one_cols_multi_output_test(self, from_table = 'test.meters'): + tdLog.info(f"one_cols_1output_test {from_table}") + tdSql.query(f'select cols(last(ts), ts, c0) from {from_table}') + tdSql.checkResColNameList(['ts', 'c0']) + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.query(f'select cols(last(ts), ts as time, c0 cc) from {from_table}') + tdSql.checkResColNameList(['time', 'cc']) + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.query(f'select cols(last(ts), ts as t123456789t123456789t123456789t123456789t123456789t123456789t123456789, c0 cc) from {from_table}') + tdSql.checkResColNameList(['t123456789t123456789t123456789t123456789t123456789t123456789t123', 'cc']) + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.query(f'select cols(last(ts), c0, c1, c2, c3) from {from_table}') + tdSql.checkResColNameList(['c0', 'c1', 'c2', 'c3']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 4) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 'bbbbbbbbb2') + tdSql.checkData(0, 3, False) + tdSql.query(f'select cols(last(ts), c0, t1) from {from_table}') + tdSql.checkResColNameList(['c0', 't1']) + tdSql.checkRows(1) + tdSql.checkData(0, 0, 4) + tdSql.checkData(0, 1, 'st1') + tdSql.query(f'select cols(max(c0), ts) from {from_table}') + tdSql.checkResColNameList(['ts']) + tdSql.checkCols(1) + tdSql.checkData(0, 0, 1734574929004) + tdSql.query(f'select cols(min(c1), ts, c0) from {from_table}') + tdSql.checkResColNameList(['ts', 'c0']) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + + tdSql.query(f'select cols(max(c0), ts, c0), count(1) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 6) + tdSql.query(f'select cols(last(ts), ts, c0), count(1) from {self.dbname}.d0') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 5) + tdSql.query(f'select cols(last(ts), ts, c0), count(1) from {self.dbname}.d1') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1) + tdSql.query(f'select cols(last(ts), ts, c0), count(1) from {self.dbname}.normal_table') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 5) + tdSql.query(f'select cols(first(ts), ts, c0), count(1) from {self.dbname}.normal_table') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 5) + tdSql.query(f'select cols(min(c0), ts, c0), count(1) from {self.dbname}.normal_table') + tdSql.checkResColNameList(['ts', 'c0', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 5) + + tdSql.query(f'select cols(last(ts), ts, c0), avg(c0) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 2.1666666666666665) + + tdSql.query(f'select cols(last(ts), ts, c0), count(1) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 6) + + tdSql.query(f'select cols(last(ts), ts, c0), sum(c0) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 13) + + tdSql.query(f'select count(1), cols(last(ts), ts, c0), min(c0) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 6) + tdSql.checkData(0, 1, 1734574929004) + tdSql.checkData(0, 2, 4) + tdSql.checkData(0, 3, 1) + + tdSql.query(f'select count(1), cols(last_row(ts), ts, c0), min(c0) from test.meters') + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 6) + tdSql.checkData(0, 1, 1734574929004) + tdSql.checkData(0, 2, 4) + tdSql.checkData(0, 3, 1) + + tdSql.query(f'select count(1), cols(last_row(ts), ts, c0), last_row(c1), last_row(c3) from test.meters') + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 6) + tdSql.checkData(0, 1, 1734574929004) + tdSql.checkData(0, 2, 4) + tdSql.checkData(0, 3, 4) + tdSql.checkData(0, 4, False) + + + tdSql.query(f'select cols(last(ts), ts, c0), count(1) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 6) + + tdSql.query(f'select cols(max(c0), ts, c0), count(1) from {from_table}') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 6) + + tdSql.query(f'select cols(last(ts), ts as time, c0 cc), count(1) from {from_table}') + tdSql.checkResColNameList(['time', 'cc', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 6) + + tdSql.query(f'select cols(max(c1), ts as time, c0 cc), count(1) from {from_table}') + tdSql.checkResColNameList(['time', 'cc', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 6) + + tdSql.query(f'select cols(last(ts), c0, c1, c2, c3), count(1) from {from_table}') + tdSql.checkResColNameList(['c0', 'c1', 'c2', 'c3', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 4) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 'bbbbbbbbb2') + tdSql.checkData(0, 3, False) + tdSql.checkData(0, 4, 6) + + tdSql.query(f'select cols(max(c0), ts), count(1) from {from_table}') + tdSql.query(f'select cols(min(c1), ts, c0), count(1) from {from_table}') + tdSql.query(f'select count(1), cols(max(c0), ts) from {from_table}') + tdSql.query(f'select max(c0), cols(max(c0), ts) from {from_table}') + tdSql.query(f'select max(c1), cols(max(c0), ts) from {from_table}') + + def multi_cols_output_test(self, from_table = 'test.meters', isTmpTable = False): + tdLog.info("multi_cols_output_test") + tdSql.query(f'select cols(last(c0), ts, c1), cols(first(c0), ts, c1), count(1) from {self.dbname}.meters') + tdSql.checkResColNameList(['ts', 'c1', 'ts', 'c1', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 6) + tdSql.query(f'select cols(last(c0),ts lts, c1 lc1), cols(first(c0), ts fts, c1 as fc1), count(1) from test.meters') + tdSql.checkResColNameList(['lts', 'lc1', 'fts', 'fc1', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 6) + tdSql.query(f'select cols(max(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), count(1) from {self.dbname}.meters') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 6) + tdSql.query(f'select cols(max(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), count(1) from {self.dbname}.meters where c0 < 4') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929003) + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 5) + tdSql.query(f'select cols(max(c0), ts as t123456789t123456789t123456789t123456789t123456789t123456789t123456789, c1 as c11), cols(first(c0), \ + ts as t123456789t123456789t123456789t123456789t123456789t123456789t123456789, c1 c21), count(1) from {self.dbname}.meters') + tdSql.checkResColNameList(['t123456789t123456789t123456789t123456789t123456789t123456789t123', 'c11', \ + 't123456789t123456789t123456789t123456789t123456789t123456789t123', 'c21', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 6) + tdSql.query(f'select cols(max(c0), ts as t123456789t123456789t123456789t123456789t123456789t123456789t123456789, c1 as c11), cols(first(c0), \ + ts as t123456789t123456789t123456789t123456789t123456789t123456789t123456789, c1 c21), count(1) from {self.dbname}.meters where c0 < 4') + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929003) + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 5) + tdSql.query(f'select cols(max(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), count(1) from test.meters where c0 < 4 group by tbname order by t1') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'count(1)']) + tdSql.checkRows(2) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 3) + tdSql.checkData(1, 2, 1734574929000) + tdSql.checkData(1, 3, 1) + tdSql.checkData(1, 4, 4) + + tdSql.query(f'select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), count(1) from test.meters where c0 < 4 group by tbname order by t1') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'count(1)']) + tdSql.checkRows(2) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 3) + tdSql.checkData(1, 2, 1734574929000) + tdSql.checkData(1, 3, 1) + tdSql.checkData(1, 4, 4) + + tdSql.query(f'select cols(last_row(c0), ts as t1, c1 as c11), cols(min(c0), ts as t2, c1 c21), count(1) from test.meters where c0 < 4 group by tbname order by t1') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'count(1)']) + tdSql.checkRows(2) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 3) + tdSql.checkData(1, 2, 1734574929000) + tdSql.checkData(1, 3, 1) + tdSql.checkData(1, 4, 4) + + tdSql.query(f'select cols(last_row(c0), ts as t1, c1 as c11), cols(mode(c0), ts as t2, c1 c21), count(1) from test.meters where c0 < 4 group by tbname order by t1') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'count(1)']) + tdSql.checkRows(2) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 3) + #tdSql.checkData(1, 2, 1734574929000) # mode(c0) is return a random ts of same c0 + tdSql.checkData(1, 3, 2) + tdSql.checkData(1, 4, 4) + + # fixed: has same select function outof cols func + tdSql.query(f'select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4 group by tbname order by t1') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'first(c0)']) + tdSql.checkRows(2) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 3) + tdSql.checkData(1, 2, 1734574929000) + tdSql.checkData(1, 3, 1) + tdSql.checkData(1, 4, 1) + + tdSql.query(f'select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), first(c0), cols(last(c0), ts, c1) from test.meters where c0 < 4 group by tbname order by t1;') + tdSql.checkResColNameList(['t1', 'c11', 't2', 'c21', 'first(c0)', 'ts', 'c1']) + tdSql.checkRows(2) + tdSql.checkCols(7) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(0, 5, 1734574929000) + tdSql.checkData(0, 6, 1) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 3) + tdSql.checkData(1, 2, 1734574929000) + tdSql.checkData(1, 3, 1) + tdSql.checkData(1, 4, 1) + tdSql.checkData(1, 5, 1734574929003) + tdSql.checkData(1, 6, 3) + + # sub query has cols func + tdSql.query(f'select c11 from (select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(1) + tdSql.checkData(0, 0, 3) + tdSql.query(f'select c11, c21 from (select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 3) + tdSql.checkData(0, 1, 1) + tdSql.query(f'select c1, c21 from (select cols(last_row(c0), ts as t1, c1), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 3) + tdSql.checkData(0, 1, 1) + tdSql.query(f'select * from (select cols(last_row(c0), ts as t1, c1), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.error(f'select c1 from (select cols(last_row(c0), ts as t1, c1), cols(first(c0), ts as t2, c1), first(c0) from test.meters where c0 < 4)') + + tdSql.query(f'select c11, _rowts from (select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 3) + tdSql.query(f'select c11, c21, _rowts from (select cols(last_row(c0), ts as t1, c1 as c11), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 3) + tdSql.checkData(0, 1, 1) + tdSql.query(f'select c1, c21, _rowts from (select cols(last_row(c0), ts as t1, c1), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 3) + tdSql.checkData(0, 1, 1) + tdSql.query(f'select *, _rowts from (select cols(last_row(c0), ts as t1, c1), cols(first(c0), ts as t2, c1 c21), first(c0) from test.meters where c0 < 4)') + tdSql.checkRows(1) + tdSql.checkCols(6) + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.error(f'select c1, _rowts from (select cols(last_row(c0), ts as t1, c1), cols(first(c0), ts as t2, c1), first(c0) from test.meters where c0 < 4)') + + + # cols on system table + tdSql.query(f'select cols(max(vgroup_id), uid) from information_schema.ins_tables') + tdSql.checkRows(1) + tdSql.checkCols(1) + tdSql.query(f'select cols(max(vgroup_id), uid, `ttl`, create_time) from information_schema.ins_tables') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.query(f'select cols(max(vgroup_id), uid as uidname) from information_schema.ins_tables') + tdSql.checkRows(1) + tdSql.checkCols(1) + tdSql.error(f'select cols(last(vgroup_id), uid, `ttl`, create_time) from information_schema.ins_tables') + tdSql.error(f'select cols(first(vgroup_id), uid, `ttl`, create_time) from information_schema.ins_tables') + + def funcSupperTableTest(self): + tdSql.execute('create database if not exists db;') + tdSql.execute('use db') + tdSql.execute(f'drop table if exists db.st') + + tdSql.execute('create table db.st (ts timestamp, c0 int, c1 float, c2 nchar(30), c3 bool) tags (t1 nchar(30))') + tdSql.execute('create table db.st_1 using db.st tags("st1")') + tdSql.execute('create table db.st_2 using db.st tags("st1")') + tdSql.execute('insert into db.st_1 values(1734574929000, 1, 1, "c2", true)') + tdSql.execute('insert into db.st_1 values(1734574929001, 2, 2, "bbbbbbbbb1", false)') + tdSql.execute('insert into db.st_1 values(1734574929002, 3, 3, "a2", true)') + tdSql.execute('insert into db.st_1 values(1734574929004, 4, 4, "bbbbbbbbb2", false)') + + tdSql.query(f'select cols(last(c0), ts, c1, c2, c3), cols(first(c0), ts, c1, c2, c3) from db.st') + tdSql.checkRows(1) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, '4.0') + tdSql.checkData(0, 2, 'bbbbbbbbb2') + tdSql.checkData(0, 3, False) + tdSql.checkData(0, 4, 1734574929000) + tdSql.checkData(0, 5, '1.0') + tdSql.checkData(0, 6, 'c2') + tdSql.checkData(0, 7, True) + + tdSql.execute(f'drop table if exists db.st') + + + def funcNestTest(self): + tdSql.execute('create database db;') + tdSql.execute('use db') + tdSql.execute(f'drop table if exists db.d1') + + tdSql.execute('create table db.d1 (ts timestamp, c0 int, c1 float, c2 nchar(30), c3 bool)') + tdSql.execute('insert into db.d1 values(1734574929000, 1, 1.1, "a", true)') + tdSql.execute('insert into db.d1 values(1734574930000, 2, 2.2, "bbbbbbbbb", false)') + + groupby = ["", "group by tbname order ts", "group by tbname order t1", "group by tbname order ts"] + + tdSql.query(f'select cols(last(c0), ts, c2), cols(first(c0), ts, c2) from db.d1') + tdSql.checkResColNameList(['ts', 'c2', 'ts', 'c2']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574930000) + tdSql.checkData(0, 1, 'bbbbbbbbb') + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 'a') + tdSql.query(f'select cols(last(c0), ts, c1, c2, c3), cols(first(c0), ts, c1, c2, c3) from db.d1') + tdSql.checkResColNameList(['ts', 'c1', 'c2', 'c3', 'ts', 'c1', 'c2', 'c3']) + tdSql.checkRows(1) + tdSql.checkCols(8) + tdSql.checkData(0, 0, 1734574930000) + tdSql.checkData(0, 1, 2.2) + tdSql.checkData(0, 2, 'bbbbbbbbb') + tdSql.checkData(0, 3, False) + tdSql.checkData(0, 4, 1734574929000) + tdSql.checkData(0, 5, 1.1) + tdSql.checkData(0, 6, 'a') + tdSql.checkData(0, 7, True) + + tdSql.query(f'select cols(last(ts), c1), cols(first(ts), c1) from db.d1') + tdSql.checkResColNameList(['c1', 'c1']) + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 2.2) + tdSql.checkData(0, 1, 1.1) + + tdSql.query(f'select cols(first(ts), c0, c1), cols(first(ts), c0, c1) from db.d1') + tdSql.checkResColNameList(['c0', 'c1', 'c0', 'c1']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1.1) + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 1.1) + + tdSql.query(f'select cols(first(ts), c0, c1), cols(first(ts+1), c0, c1) from db.d1') + tdSql.checkResColNameList(['c0', 'c1', 'c0', 'c1']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1.1) + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 1.1) + + tdSql.query(f'select cols(first(ts), c0, c1), cols(first(ts), c0+1, c1+2) from db.d1') + tdSql.checkResColNameList(['c0', 'c1', 'c0+1', 'c1+2']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1.1) + tdSql.checkData(0, 2, 2) + tdSql.checkData(0, 3, 3.1) + + tdSql.query(f'select cols(first(c0), ts, length(c2)), cols(last(c0), ts, length(c2)) from db.d1') + tdSql.checkResColNameList(['ts', 'length(c2)', 'ts', 'length(c2)']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574930000) + tdSql.checkData(0, 3, 36) + tdSql.query(f'select cols(first(c0), ts, length(c2)), cols(last(c0), ts, length(c2) + 2) from db.d1') + tdSql.checkResColNameList(['ts', 'length(c2)', 'ts', 'length(c2) + 2']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574930000) + tdSql.checkData(0, 3, 38) + + tdSql.query(f'select cols(first(c0), ts, c2), cols(last(c0), ts, length(c2) + 2) from db.d1') + tdSql.checkResColNameList(['ts', 'c2', 'ts', 'length(c2) + 2']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 'a') + tdSql.checkData(0, 2, 1734574930000) + tdSql.checkData(0, 3, 38) + + tdSql.query(f'select cols(min(c0), ts, c2), cols(last(c0), ts, length(c2) + 2) from db.d1') + tdSql.checkResColNameList(['ts', 'c2', 'ts', 'length(c2) + 2']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 'a') + tdSql.checkData(0, 2, 1734574930000) + tdSql.checkData(0, 3, 38) + + tdSql.query(f'select cols(min(c0), ts, c2), cols(first(c0), ts, length(c2) + 2) from db.d1') + tdSql.checkResColNameList(['ts', 'c2', 'ts', 'length(c2) + 2']) + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 'a') + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 6) + + def orderby_test(self, from_table = 'test.meters', col_name='c0', isTmpTable = False): + select_t1 = ["", ", t1", ", t1 as tag1"] + for t1 in select_t1: + if t1 != "" and isTmpTable: + # Not a GROUP BY expression + tdSql.error(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2)') + tdSql.error(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2) desc') + tdSql.error(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name})') + tdSql.error(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + tdSql.error(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2)') + tdSql.error(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2) desc') + tdSql.error(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name})') + tdSql.error(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + tdSql.error(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2)') + tdSql.error(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2) desc') + tdSql.error(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name})') + tdSql.error(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + continue + tdSql.query(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2)') + tdSql.checkRows(2) + tags_count = 0 if t1 == "" else 1 + tdLog.debug(f'tags_count: {tags_count}') + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 5) + tdSql.checkData(0, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 0, 2, 'st1') + tdSql.checkData(1, 0, 1) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st2') + + tdSql.query(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2) desc') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name})') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by 1') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by 2') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 5) + tdSql.checkData(0, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 0, 2, 'st1') + tdSql.checkData(1, 0, 1) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st2') + + tdSql.query(f'select count(1), cols(last({col_name}),c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2)') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 5) + tdSql.checkData(0, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 0, 2, 'st1') + tdSql.checkData(1, 0, 1) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st2') + + tdSql.query(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2) desc') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name})') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), last(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'bbbbbbbbb2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2)') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 5) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st1') + tdSql.checkData(1, 0, 1) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st2') + + tdSql.query(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), c2) desc') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name})') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select count(1), max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 0, 5) + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st1') + + tdSql.query(f'select _rowts, max(c2) {t1} from {from_table} group by tbname order by cols(last({col_name}), {col_name}), cols(last({col_name}), c1)') + tdSql.checkRows(2) + tdSql.checkCols(2 + tags_count) + tdSql.checkData(0, 1, 'c2') + self.condition_check(t1 != "", 0, 2, 'st2') + tdSql.checkData(1, 1, 'c2') + self.condition_check(t1 != "", 1, 2, 'st1') + + def parse_test(self): + tdLog.info("parse test") + + #** error sql **# + tdSql.error(f'select cols(ts) from {self.dbname}.meters group by tbname') + tdSql.error(f'select cols(ts) from {self.dbname}.meters') + tdSql.error(f'select last(cols(ts)) from {self.dbname}.meters') + tdSql.error(f'select last(cols(ts, ts)) from {self.dbname}.meters') + tdSql.error(f'select last(cols(ts, ts), ts) from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), ts as t1) as t1 from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), ts, c0) t1 from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), ts t1) tt from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), c0 cc0, c1 cc1) cc from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), c0 as cc0) as cc from {self.dbname}.meters') + tdSql.error(f'select cols(ts) + 1 from {self.dbname}.meters group by tbname') + tdSql.error(f'select last(cols(ts)+1) from {self.dbname}.meters') + tdSql.error(f'select last(cols(ts+1, ts)) from {self.dbname}.meters') + tdSql.error(f'select last(cols(ts, ts), ts+1) from {self.dbname}.meters') + tdSql.error(f'select last(cols(last(ts+1), ts+1), ts) from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), ts+1 as t1) as t1 from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts+1), ts, c0) t1 from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), ts t1) tt from {self.dbname}.meters') + tdSql.error(f'select cols(first(ts+1), c0+2 cc0, c1 cc1) cc from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts)+1, c0+2 as cc0) as cc from {self.dbname}.meters') + tdSql.error(f'select cols(ABS(c0), c1) from {self.dbname}.meters group by tbname') + tdSql.error(f'select cols(top(c0, 5), c1) from {self.dbname}.meters') + tdSql.error(f'select cols(tail(c0, 5),c1) from {self.dbname}.meters') + tdSql.error(f'select cols(BOTTOM(c0, 5),c1) from {self.dbname}.meters') + tdSql.error(f'select cols(tail(ts, 5),c1) from {self.dbname}.meters') + tdSql.error(f'select cols(UNIQUE(ts),c1) from {self.dbname}.meters') + tdSql.error(f'select cols(sample(ts, 5),c1) from {self.dbname}.meters') + + tdSql.error(f'select cols(last(ts)+1, ts) from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts)+10, c1+10) from {self.dbname}.meters group by tbname') + + tdSql.error(f'select cols(cols(last(ts), c0), c0) as cc from {self.dbname}.meters') + tdSql.error(f'select cols(last(ts), cols(last(ts), c0), c0) as cc from {self.dbname}.meters') + + # Aggregate functions do not support nesting + tdSql.error(f'select count(1), cols(last_row(ts), ts, first(c0)), last_row(c1) from {self.dbname}.meters') + + # Not a GROUP BY expression + tdSql.error(f'select count(1), cols(last(c0),c0) from test.meters group by tbname order by c3 desc') + + def subquery_test(self): + tdSql.query(f'select count(1), cols(last(c0),c0) from (select * from test.d0)') + tdSql.query(f'select count(1), cols(last(c0),c0) from (select *, tbname from test.meters) group by tbname') + + tdLog.info("subquery_test: orderby_test from meters") + self.orderby_test("test.meters", "c0", False) + tdLog.info("subquery_test: orderby_test from (select *, tbname from meters)") + self.orderby_test("(select *, tbname from test.meters)", "c0", True) + tdLog.info("subquery_test: one_cols_multi_output_with_group_test from meters") + self.one_cols_multi_output_with_group_test("test.meters", False) + tdLog.info("subquery_test: one_cols_multi_output_with_group_test from (select *, tbname from meters)") + self.one_cols_multi_output_with_group_test("(select *, tbname from test.meters)", True) + + self.one_cols_multi_output_test("test.meters") + self.one_cols_multi_output_test("(select *, tbname from test.meters)") + + def window_test(self): + tdSql.query(f'select tbname, _wstart,_wend, max(c0), max(c1), cols( max(c0), c1) from test.meters partition \ + by tbname count_window(2) order by tbname') + tdSql.checkRows(4) + tdSql.checkCols(6) + tdSql.checkData(0, 0, 'd0') + tdSql.checkData(3, 0, 'd1') + tdSql.checkData(3, 5, 1) + tdSql.query(f'select _wstart,_wend, max(c0), max(c1), cols( max(c0), c1) from test.d1 count_window(2);') + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 4, 1) + tdSql.query(f'select _wstart,_wend, max(c0), max(c1), cols( max(c0), c1) from test.normal_table count_window(2);') + tdSql.checkRows(3) + tdSql.checkCols(5) + tdSql.checkData(0, 4, 2) + + tdSql.query(f'select c0, cols(first(ts), c0, c1) from test.meters state_window(c0) order by c0') + tdSql.checkRows(4) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1) + tdSql.checkData(1, 0, 2) + tdSql.checkData(1, 1, 2) + tdSql.checkData(1, 2, 2) + tdSql.checkData(2, 0, 3) + + tdSql.query(f'select c0, cols(first(ts), c0, c1), tbname from test.meters state_window(c0)') + tdSql.checkRows(4) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 'd0') + tdSql.checkData(1, 0, 2) + tdSql.checkData(1, 1, 2) + tdSql.checkData(1, 2, 2) + tdSql.checkData(2, 0, 3) + + tdSql.query(f'select c0, cols(first(ts), c0, c1), t1 from test.meters state_window(c0)') + tdSql.checkRows(4) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 'st1') + tdSql.checkData(1, 0, 2) + tdSql.checkData(1, 1, 2) + tdSql.checkData(1, 2, 2) + tdSql.checkData(2, 0, 3) + + tdSql.error(f'select tbname, c0, cols(first(ts), c0, c1), cols(last(ts), c0) from test.meters state_window(c0)') + + def window_test2(self): + tdSql.query(f'select cols(first(ts), c0, c1), count(1) from test.meters session(ts, 3)') + tdSql.checkRows(2) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 6) + tdSql.checkData(1, 0, 0) + tdSql.checkData(1, 1, None) + tdSql.checkData(1, 2, 5) + + tdSql.query(f'select cols(max(c0), c1), count(1) from test.meters session(ts, 3)') + tdSql.checkRows(2) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 4) + tdSql.checkData(0, 1, 6) + tdSql.checkData(1, 0, None) + tdSql.checkData(1, 1, 5) + + tdSql.query(f'select cols(first(ts), c0, c1), first(ts), count(1) from test.meters session(ts, 3)') + tdSql.checkRows(2) + tdSql.checkCols(4) + + tdSql.query(f'select cols(first(ts), c0, c1), first(ts), count(1) from test.meters event_window start with c3=true end with c3!=true') + tdSql.checkRows(2) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 1) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 3) + tdSql.checkData(1, 0, 3) + tdSql.checkData(1, 1, 3) + tdSql.checkData(1, 2, 1734574929003) + tdSql.checkData(1, 3, 2) + + tdSql.query(f'select first(ts), count(1) from test.meters event_window start with c3=true end with c3!=true') + tdSql.checkRows(2) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 1734574929000) + tdSql.checkData(0, 1, 3) + tdSql.checkData(1, 0, 1734574929003) + tdSql.checkData(1, 1, 2) + + + def join_test(self): + tdSql.query(f'select cols(last(a.ts), a.c0) from test.d0 a join test.d1 b on a.ts = b.ts;') + tdSql.checkRows(1) + tdSql.checkCols(1) + tdSql.checkData(0, 0, 1) + tdSql.query(f'select cols(last(a.ts), a.c0) from test.d0 a join test.d1 b on a.ts = b.ts and a.c0 = b.c0;') + tdSql.checkRows(1) + tdSql.checkCols(1) + tdSql.checkData(0, 0, 1) + tdSql.query(f'select cols(last(a.ts), a.c0) from test.d0 a join test.d1 b on a.ts = b.ts and a.c0 > b.c0;') + tdSql.checkRows(0) + tdSql.query(f'select tbname, ts, c0 from (select cols(last(a.ts), a.tbname, a.ts, a.c0) from test.d0 a join test.d1 b on a.ts = b.ts and a.c0=b.c0)') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 'd0') + tdSql.checkData(0, 1, 1734574929000) + tdSql.checkData(0, 2, 1) + tdSql.query(f'select tbname, ts, c0 from (select cols(first(a.ts), a.tbname, a.ts, a.c0) from test.d0 a join test.d1 b on a.ts = b.ts and a.c0=b.c0)') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 'd0') + tdSql.checkData(0, 1, 1734574929000) + tdSql.checkData(0, 2, 1) + tdSql.error(f'select tbname, ts, c0 from (select cols(first(a.ts), a.tbname, a.ts, a.c0), cols(last(a.ts), b.tbname, b.ts, b.c0) from test.d0 a join test.d1 b on a.ts = b.ts and a.c0=b.c0)') + tdSql.query(f'select tbname, ts, c0 from (select cols(first(a.ts), a.tbname, a.ts, a.c0), cols(last(a.ts), b.tbname tbname1, b.ts ts2, b.c0 c02) from test.d0 a join test.d1 b on a.ts = b.ts and a.c0=b.c0)') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 'd0') + tdSql.checkData(0, 1, 1734574929000) + tdSql.checkData(0, 2, 1) + + def stream_cols_test(self): + tdSql.error(f'CREATE STREAM last_col_s1 INTO last_col1 AS SELECT cols(last(ts), ts, c0) FROM meters PARTITION BY tbname INTERVAL(1s) SLIDING(1s);', TSDB_CODE_PAR_INVALID_COLS_FUNCTION) + tdSql.query(f'CREATE STREAM last_col_s INTO last_col AS SELECT last(ts), c0 FROM meters PARTITION BY tbname INTERVAL(1s) SLIDING(1s);') + + def include_null_test(self): + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929010, 0, NULL, NULL, NULL)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929011, NULL, 1, NULL, NULL)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929012, NULL, NULL, 2, NULL)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929013, NULL, NULL, NULL, false)') + tdSql.execute(f'insert into {self.dbname}.d0 values(1734574929014, NULL, NULL, NULL, NULL)') + + tdSql.query(f'select cols(last(c0), ts), cols(last(c1), ts), cols(last(c2), ts), cols(last(c3), ts) from {self.dbname}.d0') + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 1734574929010) + tdSql.checkData(0, 1, 1734574929011) + tdSql.checkData(0, 2, 1734574929012) + tdSql.checkData(0, 3, 1734574929013) + + def long_column_name_test(self): + tdSql.execute(f'use {self.dbname}') + tdSql.execute(f'create table {self.dbname}.long_col_test (ts timestamp, longcolumntestlongcolumntestlongcolumntestlongcolumntest88888888 int, c1 float, c2 nchar(30), c3 bool) tags (t1 nchar(30))') + tdSql.execute(f'create table {self.dbname}.dl0 using {self.dbname}.long_col_test tags("st1")') + tdSql.execute(f'create table {self.dbname}.dl1 using {self.dbname}.long_col_test tags("st2")') + tdSql.execute(f'insert into {self.dbname}.dl0 values(1734574929000, 1, 1, "c2", true)') + tdSql.execute(f'insert into {self.dbname}.dl0 values(1734574929001, 2, 2, "bbbbbbbbb1", false)') + tdSql.execute(f'insert into {self.dbname}.dl0 values(1734574929002, 2, 2, "bbbbbbbbb1", false)') + tdSql.execute(f'insert into {self.dbname}.dl0 values(1734574929003, 3, 3, "a2", true)') + tdSql.execute(f'insert into {self.dbname}.dl0 values(1734574929004, 4, 4, "bbbbbbbbb2", false)') + + tdSql.execute(f'insert into {self.dbname}.dl1 values(1734574929000, 1, 1, "c2", true)') + + tdSql.query(f'select cols(last(longcolumntestlongcolumntestlongcolumntestlongcolumntest88888888),ts lts, c1 lc1), cols(first(longcolumntestlongcolumntestlongcolumntestlongcolumntest88888888), ts fts, c1 as fc1), count(1) from test.long_col_test') + tdSql.checkResColNameList(['lts', 'lc1', 'fts', 'fc1', 'count(1)']) + tdSql.checkRows(1) + tdSql.checkCols(5) + tdSql.checkData(0, 0, 1734574929004) + tdSql.checkData(0, 1, 4) + tdSql.checkData(0, 2, 1734574929000) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 6) + + tdLog.info("long_column_name_test subquery_test: orderby_test from meters") + self.orderby_test("test.long_col_test", "longcolumntestlongcolumntestlongcolumntestlongcolumntest88888888", False) + tdLog.info("long_column_name_test subquery_test: orderby_test from (select *, tbname from meters)") + self.orderby_test("(select *, tbname from test.long_col_test)", "longcolumntestlongcolumntestlongcolumntestlongcolumntest88888888", True) + tdLog.info("long_column_name_test subquery_test: one_cols_multi_output_with_group_test from meters") + + def test_in_interval(self): + dbname = "db1" + tdSql.execute(f"drop database if exists {dbname} ") + tdSql.execute(f"create database {dbname} vgroups 6") + tdSql.execute(f"use {dbname}") + + tdSql.execute(f" create stable {dbname}.sta (ts timestamp, f1 int, f2 binary(10), f3 bool) tags(t1 int, t2 bool, t3 binary(10));") + tdSql.execute(f" create table {dbname}.tba1 using sta tags(0, false, '0');") + tdSql.execute(f" create table {dbname}.tba2 using sta tags(1, true, '1');") + tdSql.execute(f" create table {dbname}.tba3 using sta tags(null, null, '');") + tdSql.execute(f" create table {dbname}.tba4 using sta tags(1, false, null);") + tdSql.execute(f" create table {dbname}.tba5 using sta tags(3, true, 'aa');") + + tdSql.execute(f"insert into {dbname}.tba1 values ('2022-09-26 15:15:01', 0, 'a', false);") + tdSql.execute(f"insert into {dbname}.tba1 values ('2022-09-26 15:15:02', 1, '0', true);") + tdSql.execute(f"insert into {dbname}.tba1 values ('2022-09-26 15:15:03', 5, '5', false);") + tdSql.execute(f"insert into {dbname}.tba1 values ('2022-09-26 15:15:04', null, null, null);") + + tdSql.execute(f"insert into {dbname}.tba2 values ('2022-09-27 15:15:01', 0, 'a', false);") + tdSql.execute(f"insert into {dbname}.tba2 values ('2022-09-27 15:15:02', 1, '0', true);") + tdSql.execute(f"insert into {dbname}.tba2 values ('2022-09-27 15:15:03', 5, '5', false);") + tdSql.execute(f"insert into {dbname}.tba2 values ('2022-09-27 15:15:04', null, null, null);") + + tdSql.execute(f"insert into {dbname}.tba3 values ('2022-09-28 15:15:01', 0, 'a', false);") + tdSql.execute(f"insert into {dbname}.tba3 values ('2022-09-28 15:15:02', 1, '0', true);") + tdSql.execute(f"insert into {dbname}.tba3 values ('2022-09-28 15:15:03', 5, '5', false);") + tdSql.execute(f"insert into {dbname}.tba3 values ('2022-09-28 15:15:04', null, null, null);") + + tdSql.execute(f"insert into {dbname}.tba4 values ('2022-09-29 15:15:01', 0, 'a', false);") + tdSql.execute(f"insert into {dbname}.tba4 values ('2022-09-29 15:15:02', 1, '0', true);") + tdSql.execute(f"insert into {dbname}.tba4 values ('2022-09-29 15:15:03', 5, '5', false);") + tdSql.execute(f"insert into {dbname}.tba4 values ('2022-09-29 15:15:04', null, null, null);") + + tdSql.execute(f"insert into {dbname}.tba5 values ('2022-09-30 15:15:01', 0, 'a', false);") + tdSql.execute(f"insert into {dbname}.tba5 values ('2022-09-30 15:15:02', 1, '0', true);") + tdSql.execute(f"insert into {dbname}.tba5 values ('2022-09-30 15:15:03', 5, '5', false);") + tdSql.execute(f"insert into {dbname}.tba5 values ('2022-09-30 15:15:04', null, null, null);") + + tdSql.query(f'select tbname, cols(last(ts), ts as ts1), cols(last(f1), ts as ts2) from tba5 interval(1s);') + tdSql.checkRows(4) + tdSql.checkCols(3) + tdSql.checkData(0, 1, "2022-09-30 15:15:01") + tdSql.checkData(0, 2, "2022-09-30 15:15:01") + tdSql.checkData(1, 1, "2022-09-30 15:15:02") + tdSql.checkData(1, 2, "2022-09-30 15:15:02") + tdSql.checkData(2, 1, "2022-09-30 15:15:03") + tdSql.checkData(2, 2, "2022-09-30 15:15:03") + tdSql.checkData(3, 1, "2022-09-30 15:15:04") + tdSql.checkData(3, 2, None) + + tdSql.query(f'select tbname, last(f1), ts from tba5 interval(1s)') + tdSql.checkRows(3) + tdSql.checkCols(3) + tdSql.checkData(0, 1, 0) + tdSql.checkData(0, 2, "2022-09-30 15:15:01") + tdSql.checkData(1, 1, 1) + tdSql.checkData(1, 2, "2022-09-30 15:15:02") + tdSql.checkData(2, 1, 5) + tdSql.checkData(2, 2, "2022-09-30 15:15:03") + + tdSql.query(f'select tbname, last(ts) from tba5 interval(1s)') + tdSql.checkData(0, 1, "2022-09-30 15:15:01") + tdSql.checkData(1, 1, "2022-09-30 15:15:02") + tdSql.checkData(2, 1, "2022-09-30 15:15:03") + tdSql.checkData(3, 1, "2022-09-30 15:15:04") + + tdSql.query(f'select tbname, cols(last(ts), ts as ts1), cols(last(f1), ts as ts2) from {dbname}.sta group by tbname;') + tdSql.checkRows(5) + tdSql.checkCols(3) + tdSql.query(f'select tbname, cols(last(ts), ts as ts1), cols(last(f1), ts as ts2) from {dbname}.sta group by tbname order by ts1;') + tdSql.checkRows(5) + tdSql.checkCols(3) + tdSql.checkData(0, 1, "2022-09-26 15:15:04") + tdSql.checkData(0, 2, "2022-09-26 15:15:03") + tdSql.checkData(1, 1, "2022-09-27 15:15:04") + tdSql.checkData(1, 2, "2022-09-27 15:15:03") + tdSql.checkData(2, 1, "2022-09-28 15:15:04") + tdSql.checkData(2, 2, "2022-09-28 15:15:03") + tdSql.checkData(3, 1, "2022-09-29 15:15:04") + tdSql.checkData(3, 2, "2022-09-29 15:15:03") + tdSql.checkData(4, 1, "2022-09-30 15:15:04") + tdSql.checkData(4, 2, "2022-09-30 15:15:03") + + tdSql.query(f'select t1, cols(last(ts), ts as ts1), cols(last(f1), ts as ts2) from {dbname}.sta group by t1;') + tdSql.checkRows(4) + tdSql.checkCols(3) + tdSql.query(f'select t1, cols(last(ts), ts as ts1), cols(last(f1), ts as ts2) from {dbname}.sta group by t1 order by ts1;') + tdSql.checkRows(4) + tdSql.checkCols(3) + tdSql.checkData(0, 1, "2022-09-26 15:15:04") + tdSql.checkData(0, 2, "2022-09-26 15:15:03") + tdSql.checkData(1, 1, "2022-09-28 15:15:04") + tdSql.checkData(1, 2, "2022-09-28 15:15:03") + tdSql.checkData(2, 1, "2022-09-29 15:15:04") + tdSql.checkData(2, 2, "2022-09-29 15:15:03") + tdSql.checkData(3, 1, "2022-09-30 15:15:04") + tdSql.checkData(3, 2, "2022-09-30 15:15:03") + + def having_test(self, table_name, is_subquery): + tdLog.info("having_test") + t1 = f"from {table_name} " + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(ts), ts) > 1734574929000') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 'd0') + tdSql.checkData(0, 1, 1734574929014) + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(ts), ts) = 1734574929000') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 'd1') + tdSql.checkData(0, 1, 1734574929000) + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(ts), ts) < 1734574929000') + tdSql.checkRows(0) + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(ts), ts) != 1734574929000') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 'd0') + tdSql.checkData(0, 1, 1734574929014) + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(ts), ts) >= 1734574929000') + tdSql.checkRows(2) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 'd1') + tdSql.checkData(0, 1, 1734574929000) + tdSql.checkData(1, 0, 'd0') + tdSql.checkData(1, 1, 1734574929014) + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(ts), ts) <= 1734574929000') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 'd1') + tdSql.checkData(0, 1, 1734574929000) + tdSql.query(f'select tbname, cols(last(ts), ts) {t1} group by tbname having cols(last(c0), ts) between 1734574929000 and 1734574929014') + tdSql.checkRows(2) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 'd1') + tdSql.checkData(0, 1, 1734574929000) + tdSql.checkData(1, 0, 'd0') + tdSql.checkData(1, 1, 1734574929014) + + def test_null2(self): + dbname = "test_null2" + tdSql.execute(f"drop database if exists {dbname}") + tdSql.execute(f"create database test_null2 vgroups 5") + tdSql.execute(f"use test_null2") + tdSql.execute(f"create stable {dbname}.stb_null1 (ts timestamp, c0 int, c1 int, c2 nchar(30), c3 bool) tags (t1 nchar(30))") + tdSql.execute(f"create table {dbname}.sub_null_1 using {dbname}.stb_null1 tags('st1')") + tdSql.execute(f"create table {dbname}.sub_null_2 using {dbname}.stb_null1 tags('st2')") + + tdSql.execute(f"insert into {dbname}.sub_null_1 values(1734574929000, 1, null, null, null)") + tdSql.execute(f"insert into {dbname}.sub_null_1 values(1734574929001, 2, null, null, null)") + + tdSql.execute(f"insert into {dbname}.sub_null_2 values(1734574929000, 21, null, null, null)") + tdSql.execute(f"insert into {dbname}.sub_null_2 values(1734574929001, 22, null, null, null)") + + tdSql.query(f'select cols(last(ts), ts), cols(last(c2), ts) from {dbname}.stb_null1') + tdSql.checkRows(1) + tdSql.checkCols(2) + tdSql.checkData(0, 0, 1734574929001) + tdSql.checkData(0, 1, None) + + tdSql.query(f'select tbname, cols(last(ts), c0), cols(last(c2), c0) from {dbname}.stb_null1 group by tbname order by tbname') + tdSql.checkRows(2) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 'sub_null_1') + tdSql.checkData(0, 1, 2) + tdSql.checkData(0, 2, None) + tdSql.checkData(1, 0, 'sub_null_2') + tdSql.checkData(1, 1, 22) + tdSql.checkData(1, 2, None) + + tdSql.query(f'select tbname, t1, cols(last(ts), c0), cols(last(c2), c0) from {dbname}.sub_null_1') + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 'sub_null_1') + tdSql.checkData(0, 1, "st1") + tdSql.checkData(0, 2, 2) + tdSql.checkData(0, 3, None) + + tdSql.query(f'select tbname, t1, cols(last(ts), c0), cols(last(c2), ts) from {dbname}.sub_null_2') + tdSql.checkRows(1) + tdSql.checkCols(4) + tdSql.checkData(0, 0, 'sub_null_2') + tdSql.checkData(0, 1, "st2") + tdSql.checkData(0, 2, 22) + tdSql.checkData(0, 3, None) + + tdSql.query(f'select tbname, cols(last(ts), c0), cols(last(c2), c0) from {dbname}.sub_null_1') + tdSql.checkRows(1) + tdSql.checkCols(3) + tdSql.checkData(0, 0, 'sub_null_1') + tdSql.checkData(0, 1, 2) + tdSql.checkData(0, 2, None) + + tdSql.error(f'select tbname, cols(last(ts), c0), cols(last(c2), c0) from {dbname}.stb_null1') + tdSql.error(f'select t1, cols(last(ts), c0), cols(last(c2), c0) from {dbname}.stb_null1') + + def run(self): + self.funcNestTest() + self.funcSupperTableTest() + self.create_test_data() + self.parse_test() + self.one_cols_1output_test() + self.multi_cols_output_test() + self.subquery_test() + self.window_test() + self.join_test() + self.stream_cols_test() + self.test_in_interval() + self.include_null_test() + self.long_column_name_test() + + self.having_test("test.meters", False) + self.having_test("(select tbname, * from test.meters)", True) + self.test_null2() + self.window_test2() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/system-test/2-query/last_cache_scan.py b/tests/system-test/2-query/last_cache_scan.py index 049fe60f16..097e3903eb 100644 --- a/tests/system-test/2-query/last_cache_scan.py +++ b/tests/system-test/2-query/last_cache_scan.py @@ -295,7 +295,7 @@ class TDTestCase: tdSql.query("select last(ts) from meters partition by tbname") tdSql.query("select last(ts) from meters partition by t1") sql_template = 'select %s from meters partition by tbname' - select_items = ["ts, last(c10), c10, ts", "ts, ts, last(c10), c10, tbname", "last(c10), c10, ts"] + select_items = ["ts, last(c10), c10, ts", "ts, ts, last(c10), c10, tbname", "last(c10), c10, ts", "ts, ts, last(c10), c10, t1" ] has_last_row_scan_res = [1,1,1] sqls = self.format_sqls(sql_template, select_items) self.explain_and_check_res(sqls, has_last_row_scan_res) @@ -312,6 +312,13 @@ class TDTestCase: tdSql.checkData(0,1, '2018-11-25 19:30:00.000') tdSql.checkData(0,2, '2018-11-25 19:30:01.000') tdSql.checkData(0,3, '2018-11-25 19:30:01.000') + + tdSql.query(sqls[3], queryTimes=1) + tdSql.checkRows(10) + tdSql.checkData(0,0, '2018-11-25 19:30:00.000') + tdSql.checkData(0,1, '2018-11-25 19:30:00.000') + tdSql.checkData(0,2, '2018-11-25 19:30:01.000') + tdSql.checkData(0,3, '2018-11-25 19:30:01.000') sql_template = 'select %s from meters partition by t1' select_items = ["ts, last(c10), c10, ts", "ts, ts, last(c10), c10, t1", "last(c10), c10, ts"] diff --git a/tests/system-test/2-query/test_window_true_for.py b/tests/system-test/2-query/test_window_true_for.py new file mode 100644 index 0000000000..2db1ac6531 --- /dev/null +++ b/tests/system-test/2-query/test_window_true_for.py @@ -0,0 +1,488 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import sys +import time + +import taos +from util.log import * +from util.cases import * +from util.sql import * + +class TDTestCase: + # init + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), True) + + def create_objects(self): + tdSql.execute("drop database if exists test", show=True) + tdSql.execute("create database test keep 36500 precision 'ms'", show=True) + tdSql.execute("use test", show=True) + tdSql.execute("create stable st (ts timestamp, c1 int) tags (gid int)", show=True) + tdSql.execute("create table ct_0 using st(gid) tags (0)") + tdSql.execute("create table ct_1 using st(gid) tags (1)") + + tdSql.execute(f'''create stream s_event_1 into d_event_1 as + select _wstart, _wend, count(*) from ct_0 + event_window start with c1 > 0 end with c1 < 0 true_for(3s);''', show=True) + tdSql.execute(f'''create stream s_event_2 ignore update 0 ignore expired 0 into d_event_2 as + select _wstart, _wend, count(*) from ct_0 + event_window start with c1 > 0 end with c1 < 0 true_for(3s);''', show=True) + + tdSql.execute(f'''create stream s_event_3 into d_event_3 as + select _wstart, _wend, count(*) from ct_0 + event_window start with c1 > 0 end with c1 < 0 true_for(2999);''', show=True) + tdSql.execute(f'''create stream s_event_4 ignore update 0 ignore expired 0 into d_event_4 as + select _wstart, _wend, count(*) from ct_0 + event_window start with c1 > 0 end with c1 < 0 true_for(2999);''', show=True) + + tdSql.execute(f'''create stream s_event_5 into d_event_5 as + select _wstart, _wend, count(*) from ct_0 + event_window start with c1 > 0 end with c1 < 0 true_for('3001a');''', show=True) + tdSql.execute(f'''create stream s_event_6 ignore update 0 ignore expired 0 into d_event_6 as + select _wstart, _wend, count(*) from ct_0 + event_window start with c1 > 0 end with c1 < 0 true_for('3001a');''', show=True) + + tdSql.execute(f'''create stream s_state_1 into d_state_1 as + select _wstart, _wend, count(*) from ct_1 + state_window(c1) true_for (3s);''', show=True) + tdSql.execute(f'''create stream s_state_2 ignore update 0 ignore expired 0 into d_state_2 as + select _wstart, _wend, count(*) from ct_1 + state_window(c1) true_for (3s);''', show=True) + + tdSql.execute(f'''create stream s_state_3 into d_state_3 as + select _wstart, _wend, count(*) from ct_1 + state_window(c1) true_for (2999);''', show=True) + tdSql.execute(f'''create stream s_state_4 ignore update 0 ignore expired 0 into d_state_4 as + select _wstart, _wend, count(*) from ct_1 + state_window(c1) true_for (2999);''', show=True) + + tdSql.execute(f'''create stream s_state_5 into d_state_5 as + select _wstart, _wend, count(*) from ct_1 + state_window(c1) true_for ('3001a');''', show=True) + tdSql.execute(f'''create stream s_state_6 ignore update 0 ignore expired 0 into d_state_6 as + select _wstart, _wend, count(*) from ct_1 + state_window(c1) true_for ('3001a');''', show=True) + # Wait for the stream tasks to be ready + for i in range(50): + tdLog.info(f"i={i} wait for stream tasks ready ...") + time.sleep(1) + rows = tdSql.query("select * from information_schema.ins_stream_tasks where status <> 'ready';") + if rows == 0: + break + + def insert_data(self): + tdSql.execute(f'''insert into ct_0 values + ('2025-01-01 00:00:00.000', -1), + ('2025-01-01 00:00:01.000', 1), + ('2025-01-01 00:00:02.000', -1), + ('2025-01-01 00:00:03.000', 1), + ('2025-01-01 00:00:04.000', 1), + ('2025-01-01 00:00:05.000', -1), + ('2025-01-01 00:00:06.000', 1), + ('2025-01-01 00:00:07.000', 1), + ('2025-01-01 00:00:08.000', 1), + ('2025-01-01 00:00:08.999', -1), + ('2025-01-01 00:00:10.000', 1), + ('2025-01-01 00:00:11.000', 1), + ('2025-01-01 00:00:12.000', 1), + ('2025-01-01 00:00:13.000', -1), + ('2025-01-01 00:00:14.000', 1), + ('2025-01-01 00:00:15.000', 1), + ('2025-01-01 00:00:16.000', 1), + ('2025-01-01 00:00:17.001', -1), + ('2025-01-01 00:00:18.000', 1), + ('2025-01-01 00:00:19.000', 1), + ('2025-01-01 00:00:20.000', 1), + ('2025-01-01 00:00:21.000', 1), + ('2025-01-01 00:00:22.000', -1), + ('2025-01-01 00:00:23.000', -1), + ('2025-01-01 00:00:24.000', 1), + ('2025-01-01 00:00:25.000', 1), + ('2025-01-01 00:00:26.000', 1), + ('2025-01-01 00:00:27.000', 1), + ('2025-01-01 00:00:28.000', 1), + ('2025-01-01 00:00:29.000', 1), + ('2025-01-01 00:00:30.000', -1), + ('2025-01-01 00:00:31.000', 0);''', show=True) + tdSql.execute(f'''insert into ct_1 values + ('2025-01-01 00:00:00.000', 0), + ('2025-01-01 00:00:01.000', 1), + ('2025-01-01 00:00:02.000', 1), + ('2025-01-01 00:00:03.000', 2), + ('2025-01-01 00:00:04.000', 2), + ('2025-01-01 00:00:05.000', 2), + ('2025-01-01 00:00:06.000', 3), + ('2025-01-01 00:00:07.000', 3), + ('2025-01-01 00:00:08.000', 3), + ('2025-01-01 00:00:08.999', 3), + ('2025-01-01 00:00:10.000', 4), + ('2025-01-01 00:00:11.000', 4), + ('2025-01-01 00:00:12.000', 4), + ('2025-01-01 00:00:13.000', 4), + ('2025-01-01 00:00:14.000', 5), + ('2025-01-01 00:00:15.000', 5), + ('2025-01-01 00:00:16.000', 5), + ('2025-01-01 00:00:17.001', 5), + ('2025-01-01 00:00:18.000', 6), + ('2025-01-01 00:00:19.000', 6), + ('2025-01-01 00:00:20.000', 6), + ('2025-01-01 00:00:21.000', 6), + ('2025-01-01 00:00:22.000', 6), + ('2025-01-01 00:00:23.000', 0), + ('2025-01-01 00:00:24.000', 7), + ('2025-01-01 00:00:25.000', 7), + ('2025-01-01 00:00:26.000', 7), + ('2025-01-01 00:00:27.000', 7), + ('2025-01-01 00:00:28.000', 7), + ('2025-01-01 00:00:29.000', 7), + ('2025-01-01 00:00:30.000', 7), + ('2025-01-01 00:00:31.000', 0);''', show=True) + tdLog.info("wait for all stream tasks to be ready ...") + time.sleep(10) + + def update_data(self): + tdSql.execute(f'''insert into ct_0 values + ('2025-01-01 00:00:00.000', 1), + ('2025-01-01 00:00:22.000', 1), + ('2025-01-01 00:00:28.000', -1);''', show=True) + tdSql.execute(f'''insert into ct_1 values + ('2025-01-01 00:00:00.000', 1), + ('2025-01-01 00:00:23.000', 6), + ('2025-01-01 00:00:29.000', 8), + ('2025-01-01 00:00:30.000', 8);''', show=True) + tdLog.info("wait for all stream tasks to be ready ...") + time.sleep(5) + + def check_result(self): + tdSql.query("select * from d_event_1", show=True) + tdSql.checkRows(4) + tdSql.checkData(0, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:22.000') + tdSql.checkData(2, 2, 5) + tdSql.checkData(3, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:30.000') + tdSql.checkData(3, 2, 7) + + tdSql.query("select * from d_event_2", show=True) + tdSql.checkRows(4) + tdSql.checkData(0, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(2, 2, 6) + tdSql.checkData(3, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(3, 2, 5) + + tdSql.query("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(3s);", show=True) + tdSql.checkRows(4) + tdSql.checkData(0, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(2, 2, 6) + tdSql.checkData(3, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(3, 2, 5) + + tdSql.query("select * from d_event_3", show=True) + tdSql.checkRows(5) + tdSql.checkData(0, 0, '2025-01-01 00:00:06.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:08.999') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(2, 2, 4) + tdSql.checkData(3, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:22.000') + tdSql.checkData(3, 2, 5) + tdSql.checkData(4, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(4, 1, '2025-01-01 00:00:30.000') + tdSql.checkData(4, 2, 7) + + tdSql.query("select * from d_event_4", show=True) + tdSql.checkRows(5) + tdSql.checkData(0, 0, '2025-01-01 00:00:06.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:08.999') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(2, 2, 4) + tdSql.checkData(3, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(3, 2, 6) + tdSql.checkData(4, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(4, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(4, 2, 5) + + tdSql.query("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(2999);", show=True) + tdSql.checkRows(5) + tdSql.checkData(0, 0, '2025-01-01 00:00:06.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:08.999') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(2, 2, 4) + tdSql.checkData(3, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(3, 2, 6) + tdSql.checkData(4, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(4, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(4, 2, 5) + + tdSql.query("select * from d_event_5", show=True) + tdSql.checkRows(3) + tdSql.checkData(0, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:22.000') + tdSql.checkData(1, 2, 5) + tdSql.checkData(2, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:30.000') + tdSql.checkData(2, 2, 7) + + tdSql.query("select * from d_event_6", show=True) + tdSql.checkRows(3) + tdSql.checkData(0, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(1, 2, 6) + tdSql.checkData(2, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(2, 2, 5) + + tdSql.query("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for('3001a');", show=True) + tdSql.checkRows(3) + tdSql.checkData(0, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(1, 2, 6) + tdSql.checkData(2, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(2, 2, 5) + + tdSql.query("select * from d_state_1", show=True) + tdSql.checkRows(4) + tdSql.checkData(0, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:22.000') + tdSql.checkData(2, 2, 5) + tdSql.checkData(3, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:30.000') + tdSql.checkData(3, 2, 7) + + tdSql.query("select * from d_state_2", show=True) + tdSql.checkRows(4) + tdSql.checkData(0, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(2, 2, 6) + tdSql.checkData(3, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(3, 2, 5) + + tdSql.query("select _wstart, _wend, count(*) from ct_1 state_window(c1) true_for(3s);", show=True) + tdSql.checkRows(4) + tdSql.checkData(0, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(2, 2, 6) + tdSql.checkData(3, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(3, 2, 5) + + tdSql.query("select * from d_state_3", show=True) + tdSql.checkRows(5) + tdSql.checkData(0, 0, '2025-01-01 00:00:06.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:08.999') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(2, 2, 4) + tdSql.checkData(3, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:22.000') + tdSql.checkData(3, 2, 5) + tdSql.checkData(4, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(4, 1, '2025-01-01 00:00:30.000') + tdSql.checkData(4, 2, 7) + + tdSql.query("select * from d_state_4", show=True) + tdSql.checkRows(5) + tdSql.checkData(0, 0, '2025-01-01 00:00:06.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:08.999') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(2, 2, 4) + tdSql.checkData(3, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(3, 2, 6) + tdSql.checkData(4, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(4, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(4, 2, 5) + + tdSql.query("select _wstart, _wend, count(*) from ct_1 state_window(c1) true_for(2999);", show=True) + tdSql.checkRows(5) + tdSql.checkData(0, 0, '2025-01-01 00:00:06.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:08.999') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:10.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:13.000') + tdSql.checkData(1, 2, 4) + tdSql.checkData(2, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(2, 2, 4) + tdSql.checkData(3, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(3, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(3, 2, 6) + tdSql.checkData(4, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(4, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(4, 2, 5) + + tdSql.query("select * from d_state_5", show=True) + tdSql.checkRows(3) + tdSql.checkData(0, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:22.000') + tdSql.checkData(1, 2, 5) + tdSql.checkData(2, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:30.000') + tdSql.checkData(2, 2, 7) + + tdSql.query("select * from d_state_6", show=True) + tdSql.checkRows(3) + tdSql.checkData(0, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(1, 2, 6) + tdSql.checkData(2, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(2, 2, 5) + + tdSql.query("select _wstart, _wend, count(*) from ct_1 state_window(c1) true_for('3001a');", show=True) + tdSql.checkRows(3) + tdSql.checkData(0, 0, '2025-01-01 00:00:14.000') + tdSql.checkData(0, 1, '2025-01-01 00:00:17.001') + tdSql.checkData(0, 2, 4) + tdSql.checkData(1, 0, '2025-01-01 00:00:18.000') + tdSql.checkData(1, 1, '2025-01-01 00:00:23.000') + tdSql.checkData(1, 2, 6) + tdSql.checkData(2, 0, '2025-01-01 00:00:24.000') + tdSql.checkData(2, 1, '2025-01-01 00:00:28.000') + tdSql.checkData(2, 2, 5) + + def test_abnormal_query(self): + tdLog.info("test abnormal window true_for limit") + tdSql.error("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(3n);") + tdSql.error("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(3y);") + tdSql.error("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(-1);") + tdSql.error("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(-1a);") + tdSql.error("select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for('-1a');") + tdSql.error("create stream s_ab into dst as select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(3n);") + tdSql.error("create stream s_ab into dst as select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(3y);") + tdSql.error("create stream s_ab into dst as select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(-1);") + tdSql.error("create stream s_ab into dst as select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for(-1a);") + tdSql.error("create stream s_ab into dst as select _wstart, _wend, count(*) from ct_0 event_window start with c1 > 0 end with c1 < 0 true_for('-1a');") + + def test_window_true_for_limit(self): + """ Test the functionality of the true_for window function. + + This test covers: + 1. Both batch query and stream computing scenarios. + 2. Two types of windows: event_window and state_window. + 3. Parameter types for true_for: numeric and string. + 4. Boundary value tests. + 5. Error case tests. + + Since: v3.3.6.0 + + Labels: true_for, state_window, event_window + + Jira: TS-5470 + + History: + - 2025-02-21 Kuang Jinqing Created + """ + self.create_objects() + self.insert_data() + self.update_data() + self.check_result() + self.test_abnormal_query() + + # run + def run(self): + self.test_window_true_for_limit() + + # stop + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/system-test/7-tmq/tmq_ts-5473.py b/tests/system-test/7-tmq/tmq_c_test.py similarity index 81% rename from tests/system-test/7-tmq/tmq_ts-5473.py rename to tests/system-test/7-tmq/tmq_c_test.py index ad08fa559c..a2ed4aa708 100644 --- a/tests/system-test/7-tmq/tmq_ts-5473.py +++ b/tests/system-test/7-tmq/tmq_c_test.py @@ -29,6 +29,14 @@ class TDTestCase: tdLog.info(cmdStr) os.system(cmdStr) + cmdStr = '%s/build/bin/tmq_ts5776'%(buildPath) + tdLog.info(cmdStr) + os.system(cmdStr) + + cmdStr = '%s/build/bin/tmq_td33798'%(buildPath) + tdLog.info(cmdStr) + os.system(cmdStr) + return def stop(self): diff --git a/tests/system-test/7-tmq/tmq_ts-5776.py b/tests/system-test/7-tmq/tmq_ts-5776.py deleted file mode 100644 index 738d69701f..0000000000 --- a/tests/system-test/7-tmq/tmq_ts-5776.py +++ /dev/null @@ -1,39 +0,0 @@ - -import taos -import sys -import time -import socket -import os -import threading - -from util.log import * -from util.sql import * -from util.cases import * -from util.dnodes import * -from util.common import * -from taos.tmq import * -sys.path.append("./7-tmq") -from tmqCommon import * - -class TDTestCase: - updatecfgDict = {'debugFlag': 135, 'asynclog': 0} - def init(self, conn, logSql, replicaVar=1): - self.replicaVar = int(replicaVar) - tdLog.debug(f"start to excute {__file__}") - tdSql.init(conn.cursor()) - #tdSql.init(conn.cursor(), logSql) # output sql.txt file - - def run(self): - buildPath = tdCom.getBuildPath() - cmdStr = '%s/build/bin/tmq_ts5776'%(buildPath) - tdLog.info(cmdStr) - os.system(cmdStr) - - return - - def stop(self): - tdSql.close() - tdLog.success(f"{__file__} successfully executed") - -tdCases.addLinux(__file__, TDTestCase()) -tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/case_list_docs/data_write/stmt.md b/tests/test_new/case_list_docs/data_write/stmt.md new file mode 100644 index 0000000000..76b57498e9 --- /dev/null +++ b/tests/test_new/case_list_docs/data_write/stmt.md @@ -0,0 +1 @@ +::: data_write.stmt.test_stmt_set_tbname_tag \ No newline at end of file diff --git a/tests/test_new/case_list_docs/high_availability/2_replica.md b/tests/test_new/case_list_docs/high_availability/2_replica.md new file mode 100644 index 0000000000..9e3d7f3858 --- /dev/null +++ b/tests/test_new/case_list_docs/high_availability/2_replica.md @@ -0,0 +1 @@ +::: high_availability.2_replica.test_replica2 \ No newline at end of file diff --git a/tests/test_new/case_list_docs/index.md b/tests/test_new/case_list_docs/index.md new file mode 100644 index 0000000000..c63acebd67 --- /dev/null +++ b/tests/test_new/case_list_docs/index.md @@ -0,0 +1,205 @@ +# 用例编写规范 +## 1. 背景 +在 TDengine 的测试中,长期存在以下两个问题: +### 问题一:测试用例缺乏整理 +在 TDengine 的 CI 测试中,执行了 1,600+ 个测试用例。但随着产品迭代,这些用例覆盖了哪些功能,没有覆盖哪些功能,缺乏文档整理。构建和维护一个详尽的测试用例描述文档,将有助于我们识别缺失的用例,并进行针对性地补充。但使用传统的方法,用文档或测试用例管理系统来维护测试用例,存在以下两个弊端: + +1. 维护成本高 +2. 文档和代码分离,导致文档容易“生锈” +### 问题二:多个测试框架并存 +由于历史原因,TDengine 的 CI 测试用例是使用不同的语言、框架 (包括 TSIM, System Test, ARMY 等) 编写的,分布在 TDengine/tests 目录中的不同的子目录下,用例的管理稍显混乱。为了执行“全量测试”,需要使用不同的框架,分别执行多个不同子目录下的用例才能完成。每个框架编写的用例,编写方式和执行方式都存在差异,也很难生成统一风格的测试报告。## 2. 变更历史 + +## 2. 变更历史 +| 日期 | 版本 | 负责人 |主要修改内容| +|:-----------|:-----------|:-------------| :--- | +| 2025/2/12 | 0.1 | @冯超 @霍宏 @王旭 | 初始文档| +| 2025/2/17 | 0.2 | @霍宏 @王旭 | 修改用例编写规范 | + +## 3. 解决方案 +为了解决以上问题,平台部经和研发负责人的讨论,达成一致意见,研发部需要进行以下调整: + +1. 平台部负责对现有的测试框架进行整合和优化,以后所有用例的编写,均使用统一的语言、框架,主要改动如下: + - 统一使用 Python 进行功能用例的编写 + - 引入 pytest 作为 test runner, pytest 是 Python 生态中应用非常广泛,可以方便地控制用例的执行粒度(执行一个用例、一组用例、按 marker 执行用例等),除此以外,它还提供了丰富的插件,例如:与测试报告相关的 allure-pytest 等 + - 将 system-test, TestNG 等现有框架的核心库进行整合,以简化测试用例的编写 +2. 产品研发人员,编写用例时,需要以统一的格式为用例添加描述信息等 metadata, 平台部将使用工具,以自动化的方式,从测试用例代码自动生成测试用例文档,这也符合我们公司推行的“一切代码化”的要求。 +3. 工具平台部将使用工具 mkdocs + mkdocstrings 提取以上信息,生成用例文档,并将其部署至 Github Pages 或内网 +## 4. 目标和范围 +本规范适用于以下范围: + +- 仓库 + - TDengine: /tests/test_new (暂定名,为了和当前的 /tests 目录下存量用例进行区分) + - taosX: /tests/e2e +- 测试类型:端到端的功能测试 +- 编程语言:Python + +其它仓库或其它类型的测试,基于不同框架/语言实现,暂时不对用例规范做强制要求,可仅作为参考。 +## 5. 用例编写规范 +### 5.1 用例目录 +对用例目录的要求如下: + +1. 新增用例统一存放至 tests/test_new +2. 用例采用两级目录的形式组织,详见:附录1 +### 5.2 用例文件 +对用例文件的要求如下: + +1. 用例 Python 文件命名应以 test_开头,例如:test_join.py +5.3 用例规范 +对用例文件内容的要求如下: + +1. 用例 Python 文件中,需要定义一个 Class,以 Test 开头命名,建议与文件名一致,例如:TestJoin +2. 一个用例 Python 文件可以包含多个用例,以 test class 下的 test method 的形式存在,test method 的命名需要以 test_ 开头,例如:test_join() +3. Class 中通常定义 init(),run(), stop()等方法,用于用例初始化数据、用例执行、用例清理环境等操作;这里,主要是为了与当前的 system-test 框架兼容,保证大家使用新规范编写的用例,可以用当前的框架执行,待新框架的整合完成后,将不存在这种要求,平台将进行统一的替换。 +4. run()方法中,需要调用该类中定义的所有 test methods, 不涉及其他逻辑 +5. 在每个 test method 中,以标准的 Python docstring 的形式,添加用例描述信息,包括多个字段,每个字段之间用空行间隔,字段及其要求,如下所示: + | 字段名称|字段描述|是否必填| +|:--------| ---- | :---: | +| |用例的一句话描述,仅支持一行|是| +| |用例的详细描述,支持多行|是| +| Since | 用例开始支持的 TDengine 版本,新 Feature 必填 | 是 | +| Lables | 用例标签,多个标签用英文逗号分隔,标签采用 snake case, 即全部小写,多个单词用 _ 连接|否| +| Jira | Jira ID, 多个用英文逗号分隔 | 否 | +| History |用例变更历史|否| + +### 5.4 其它要求、建议 + +1. 建议相同环境/数据配置的用例放在同一文件中执行,提高运行效率 +2. 用例中的调试信息,应使用框架提供的 log 打印,例如:util.log.tdLog,而不要使用 print() 函数 +### 5.5 用例模板 +```Python +# tests/test_new/xxx/xxx/test_xxx.py +import ... + +class TestXxxx: + def init(self, args=value ...): + tdLog.debug("start to execute %s" % __file__) + ... + + + def user_defined_function(self, args=value ...): + ... + + + def test_template(self): + """用例目标,必填,用一行简短总结 + <空行> + 用例详细描述,必填,允许多行 + <空行> + Since: 用例开始支持的TDengine版本,新增用例必填 + <空行> + Labels: 筛选标签,选填,多个标签用英文逗号分隔 + <空行> + Jira: 相关jira任务id,选填 + <空行> + History: 用例变更历史,选填,每行一次变更信息 + - 日期1 变更人1 变更原因1 + - 日期2 变更人2 变更原因2 + """ + ... # test case code + + + def test_demo(self): + """测试超级表插入各种数据类型 + + 使用多种数据类型创建超级表,向超级表插入数据, + 包括:常规数据,空数据,边界值等,插入均执行成功 + + Since: v3.3.0.0 + + Labels: stable, data_type + + Jira: TD-12345, TS-1234 + + History: + - 2024-2-6 Feng Chao Created + - 2024-2-7 Huo Hong updated for feature TD-23456 + """ + ... # test case code + + + def run(self): + self.test_template() + self.test_demo() + ...# case function list + + + def stop(self): + ... + tdLog.success("%s successfully executed" % __file__) +``` +说明: + +- 在 Docstring 中,每一部分之间需要用空行分隔,否则会出现格式错误 + +## 6. 附录 +### 6.1 附录1:用例目录结构 +新增用例在 TDengine/tests/test_new 目录下,采用二级目录的形式组织,以下仅作为建议,大家可以提出 comments: + +``` +test_new/ +├── data_write/ +│ ├── csv/ +│ ├── sql_statement/ +│ ├── stmt/ +│ ├── stmt2/ +│ └── schemaless/ +├── metadata/ +│ ├── child_table/ +│ ├── data_type/ +│ ├── db/ +│ ├── dnode/ +│ ├── mnode/ +│ ├── naming_rule/ +│ ├── qnode/ +│ ├── reqular_table/ +│ ├── snode/ +│ ├── super_table/ +│ ├── system_table/ +│ ├── tag_index/ +│ └── time_precision/ +├── high_availability/ +│ ├── 2_replica/ +│ ├── 3_replica/ +│ ├── active_active/ +│ ├── cluster_expansion_reduction/ +│ └── replica_change/ +├── operation/ +│ ├── balance/ +│ ├── configuration/ +│ ├── redistribute/ +│ ├── restore/ +│ ├── slow_query/ +│ ├── split/ +│ ├── transaction/ +│ └── upgrade/ +├── query/ +│ ├── case_when/ +│ ├── escape_character/ +│ ├── function/ +│ ├── having/ +│ ├── hint/ +│ ├── index/ +│ ├── join/ +│ ├── nested/ +│ ├── operator/ +│ ├── pseudo_column/ +│ ├── sql_syntax/ +│ ├── union/ +│ ├── view/ +│ └── window/ +├── security/ +│ ├── audit/ +│ ├── authorization/ +│ ├── permission/ +│ └── encryption/ +├── storage/ +│ ├── compress/ +│ ├── multilevel/ +│ ├── sma/ +│ ├── tsma/ +│ └── s3/ +├── stream/ +├── tdgpt/ +├── tmq/ +└── udf/ +``` \ No newline at end of file diff --git a/tests/test_new/case_list_docs/metadata/time_precision.md b/tests/test_new/case_list_docs/metadata/time_precision.md new file mode 100644 index 0000000000..05fa26041f --- /dev/null +++ b/tests/test_new/case_list_docs/metadata/time_precision.md @@ -0,0 +1 @@ +::: metadata.time_precision.test_precision_ns \ No newline at end of file diff --git a/tests/test_new/case_list_docs/query/function.md b/tests/test_new/case_list_docs/query/function.md new file mode 100644 index 0000000000..b548b34e6b --- /dev/null +++ b/tests/test_new/case_list_docs/query/function.md @@ -0,0 +1 @@ +::: query.function.test_sin \ No newline at end of file diff --git a/tests/test_new/case_list_docs/query/hint.md b/tests/test_new/case_list_docs/query/hint.md new file mode 100644 index 0000000000..a719bdb1c7 --- /dev/null +++ b/tests/test_new/case_list_docs/query/hint.md @@ -0,0 +1 @@ +::: query.hint.test_hint \ No newline at end of file diff --git a/tests/test_new/case_list_docs/query/join.md b/tests/test_new/case_list_docs/query/join.md new file mode 100644 index 0000000000..d474667d93 --- /dev/null +++ b/tests/test_new/case_list_docs/query/join.md @@ -0,0 +1 @@ +::: query.join.test_join \ No newline at end of file diff --git a/tests/test_new/data_write/sql_statement/test_insert_stb.py b/tests/test_new/data_write/sql_statement/test_insert_stb.py new file mode 100644 index 0000000000..5ed5860e01 --- /dev/null +++ b/tests/test_new/data_write/sql_statement/test_insert_stb.py @@ -0,0 +1,223 @@ +import sys + +sys.path.append("../tests/pytest") +from util.log import * +from util.cases import * +from util.sql import * +from util.dnodes import tdDnodes +from math import inf +import taos + + +class TestInsertStb: + def caseDescription(self): + ''' + case1: [TS-3932] insert into stb + ''' + return + + def init(self, conn, logSql, replicaVer=1): + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor(), True) + self.conn = conn + + + def restartTaosd(self, index=1, dbname="db"): + tdDnodes.stop(index) + tdDnodes.startWithoutSleep(index) + tdSql.execute(f"use insert_stb") + + + def test_normal(self): + """测试超级表基本插入操作 + + 插入语句使用using、dbname.stable、tbname等关键字插入数据,插入单表数据,单表多条数据,多表多条数据,带标签插入等语句,执行成功 + + Since: v3.3.0.0 + + Labels: stable + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + + print("running {}".format("normal")) + tdSql.execute("drop database if exists insert_stb") + tdSql.execute("create database if not exists insert_stb") + tdSql.execute('use insert_stb') + tdSql.execute('create database d1') + + tdSql.execute('create database d2') + + tdSql.execute('use d1;') + + tdSql.execute('create table st(ts timestamp, f int) tags(t int);') + + tdSql.execute("insert into ct1 using st tags(1) values('2021-04-19 00:00:00', 1);") + + tdSql.execute("insert into ct2 using st tags(2) values('2021-04-19 00:00:01', 2);") + + tdSql.execute("insert into ct1 values('2021-04-19 00:00:02', 2);") + + tdSql.execute('use d2;') + + tdSql.execute('create table st(ts timestamp, f int) tags(t int);') + + tdSql.execute("insert into ct1 using st tags(1) values('2021-04-19 00:00:00', 1);") + + tdSql.execute("insert into ct2 using st tags(2) values('2021-04-19 00:00:01', 2);") + + tdSql.execute('create database db1 vgroups 1;') + + tdSql.execute('create table db1.stb (ts timestamp, c1 int, c2 int) tags(t1 int, t2 int);') + + tdSql.execute('use d1;') + + tdSql.execute("insert into st (tbname, ts, f, t) values('ct3', '2021-04-19 08:00:03', 3, 3);") + + tdSql.execute("insert into d1.st (tbname, ts, f) values('ct6', '2021-04-19 08:00:04', 6);") + + tdSql.execute("insert into d1.st (tbname, ts, f) values('ct6', '2021-04-19 08:00:05', 7)('ct8', '2021-04-19 08:00:06', 8);") + + tdSql.execute("insert into d1.st (tbname, ts, f, t) values('ct6', '2021-04-19 08:00:07', 9, 9)('ct8', '2021-04-19 08:00:08', 10, 10);") + + tdSql.execute("insert into d1.st (tbname, ts, f, t) values('ct6', '2021-04-19 08:00:09', 9, 9)('ct8', '2021-04-19 08:00:10', 10, 10) d2.st (tbname, ts, f, t) values('ct6', '2021-04-19 08:00:11', 9, 9)('ct8', '2021-04-19 08:00:12', 10, 10);") + + tdSql.query('select * from d1.st order by ts;') + tdSql.checkRows(11) + tdSql.checkData(0, 0, datetime.datetime(2021, 4, 19, 0, 0)) + + + def test_insert_stb(self): + """测试超级表插入各种数据类型 + + 使用多种数据类型创建超级表,向超级表插入数据,包括常规数据,空数据,边界值等,插入均执行成功 + + Since: v3.3.0.0 + + Labels: stable, dataType + + Jira: TD-12345,TS-1234 + + History: + 2024-2-6 Feng Chao Created + + """ + + print("running {}".format('insert_stb')) + self.conn.select_db('insert_stb') + tdSql.execute('create table stb1 (ts timestamp, c1 bool, c2 tinyint, c3 smallint, c4 int, c5 bigint, c6 float, c7 double, c8 binary(10), c9 nchar(10), c10 tinyint unsigned, c11 smallint unsigned, c12 int unsigned, c13 bigint unsigned) TAGS(t1 int, t2 binary(10), t3 double);') + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values (\'2021-11-11 09:00:00\',true,1,1,1,1,1,1,"123","1234",1,1,1,1, 1, \'1\', 1.0, \'tb1\');') + + tdSql.execute("insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values ('2021-11-11 09:00:01',true,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL,NULL, 2, '2', 2.0, 'tb1');") + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values (\'2021-11-11 09:00:02\',true,2,NULL,2,NULL,2,NULL,"234",NULL,2,NULL,2,NULL, 2, \'2\', 2.0, \'tb2\');') + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values (\'2021-11-11 09:00:03\',false,NULL,3,NULL,3,NULL,3,NULL,"3456",NULL,3,NULL,3, 3, \'3\', 3.0, \'tb3\');') + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values (\'2021-11-11 09:00:04\',true,4,4,4,4,4,4,"456","4567",4,4,4,4, 4, \'4.0\', 4.0, \'tb4\');') + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values (\'2021-11-11 09:00:05\',true,127,32767,2147483647,9223372036854775807,3.402823466e+38,1.79769e+308,"567","5678",254,65534,4294967294,9223372036854775807, 5, \'5\', 5, \'max\' );') + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,t1,t2,t3,tbname) values (\'2021-11-11 09:00:06\',true,-127,-32767,-2147483647,-9223372036854775807,-3.402823466e+38,-1.79769e+308,"678","6789",0,0,0,0, 6, \'6\', 6, \'min\');') + + tdSql.execute('insert into stb1(ts,c1,c2,c3,c4,c5,c6,c7,c8,c9,c10,c11,c12,c13,tbname,t1,t2,t3) values (\'2021-11-11 09:00:07\',true,-127,-32767,-2147483647,-9223372036854775807,-3.402823466e+38,-1.79769e+308,"678","6789",0,0,0,0, \'min\', 6, \'6\', 6);') + + tdSql.query('select tbname,* from stb1 order by ts;') + tdSql.checkRows(8) + tdSql.checkData(0, 0, 'tb1') + tdSql.checkData(0, 1, datetime.datetime(2021, 11, 11, 9, 0)) + tdSql.checkData(0, 2, True) + + def test_stmt_error(self): + """测试超级表插入stmt数据失败 + + 创建参数绑定对象,tag设置为填充对象,绑定参数后插入预期失败 + + Since: v3.3.0.0 + + Labels: stable, negative + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + conn = self.conn + conn.select_db('insert_stb') + conn.execute('create table stb9(ts timestamp, f int) tags (t int)') + try: + stmt = conn.statement("insert into stb9(tbname, f, t) values('ctb91', 1, ?)") + params = taos.new_bind_params(1) + params[0].int(1) + stmt.bind_param(params) + stmt.execute() + result = stmt.use_result() + except Exception as err: + print(str(err)) + + def test_consecutive_seq(self): + """测试超级表连续插入 + + 向超级表连续插入多条数据,插入均执行成功 + + Since: v3.3.0.0 + + Labels: stable + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + print("running {}".format("consecutive_seq")) + tdSql.execute("drop database if exists insert_stb3") + tdSql.execute("create database if not exists insert_stb3") + tdSql.execute('use insert_stb3') + tdSql.execute('create table st (ts timestamp, ti tinyint, si smallint, i int, bi bigint, f float, d double, b binary(10)) tags(t1 int, t2 float, t3 binary(10))') + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct0', 0, 0.000000, 'childtable', 1546300800000, 0, 0, 0, 0, 0.000000, 0.000000, 'hello') ('ct0', 0, 0.000000, 'childtable', 1546300800001, 1, 1, 1, 1, 1.000000, 2.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct1', 1, 1.000000, 'childtable', 1546301800000, 64, 16960, 1000000, 1000000, 1000000.000000, 2000000.000000, 'hello') ('ct1', 1, 1.000000, 'childtable', 1546301800001, 65, 16961, 1000001, 1000001, 1000001.000000, 2000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct2', 2, 2.000000, 'childtable', 1546302800000, -128, -31616, 2000000, 2000000, 2000000.000000, 4000000.000000, 'hello') ('ct2', 2, 2.000000, 'childtable', 1546302800001, -127, -31615, 2000001, 2000001, 2000001.000000, 4000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct3', 3, 3.000000, 'childtable', 1546303800000, -64, -14656, 3000000, 3000000, 3000000.000000, 6000000.000000, 'hello') ('ct3', 3, 3.000000, 'childtable', 1546303800001, -63, -14655, 3000001, 3000001, 3000001.000000, 6000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct4', 4, 4.000000, 'childtable', 1546304800000, 0, 2304, 4000000, 4000000, 4000000.000000, 8000000.000000, 'hello') ('ct4', 4, 4.000000, 'childtable', 1546304800001, 1, 2305, 4000001, 4000001, 4000001.000000, 8000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct5', 5, 5.000000, 'childtable', 1546305800000, 64, 19264, 5000000, 5000000, 5000000.000000, 10000000.000000, 'hello') ('ct5', 5, 5.000000, 'childtable', 1546305800001, 65, 19265, 5000001, 5000001, 5000001.000000, 10000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct6', 6, 6.000000, 'childtable', 1546306800000, -128, -29312, 6000000, 6000000, 6000000.000000, 12000000.000000, 'hello') ('ct6', 6, 6.000000, 'childtable', 1546306800001, -127, -29311, 6000001, 6000001, 6000001.000000, 12000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct7', 7, 7.000000, 'childtable', 1546307800000, -64, -12352, 7000000, 7000000, 7000000.000000, 14000000.000000, 'hello') ('ct7', 7, 7.000000, 'childtable', 1546307800001, -63, -12351, 7000001, 7000001, 7000001.000000, 14000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct8', 8, 8.000000, 'childtable', 1546308800000, 0, 4608, 8000000, 8000000, 8000000.000000, 16000000.000000, 'hello') ('ct8', 8, 8.000000, 'childtable', 1546308800001, 1, 4609, 8000001, 8000001, 8000001.000000, 16000002.000000, 'hello')") + + tdSql.execute("insert into st(tbname, t1, t2, t3, ts, ti, si, i, bi, f, d, b) values ('ct9', 9, 9.000000, 'childtable', 1546309800000, 64, 21568, 9000000, 9000000, 9000000.000000, 18000000.000000, 'hello') ('ct9', 9, 9.000000, 'childtable', 1546309800001, 65, 21569, 9000001, 9000001, 9000001.000000, 18000002.000000, 'hello')") + + tdSql.query('select * from st order by ts') + tdSql.checkRows(20) + tdSql.checkData(0, 0, datetime.datetime(2019, 1, 1, 8, 0)) + + tdSql.execute('drop database insert_stb3') + + def run(self): + self.test_normal() + self.test_insert_stb() + self.test_stmt_error() + self.test_consecutive_seq() + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TestInsertStb()) +tdCases.addLinux(__file__, TestInsertStb()) diff --git a/tests/test_new/data_write/stmt/test_stmt_set_tbname_tag.py b/tests/test_new/data_write/stmt/test_stmt_set_tbname_tag.py new file mode 100644 index 0000000000..88f222b3c7 --- /dev/null +++ b/tests/test_new/data_write/stmt/test_stmt_set_tbname_tag.py @@ -0,0 +1,276 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import sys +import os +import threading as thd +import multiprocessing as mp +from numpy.lib.function_base import insert +import taos +from taos import * +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np +import datetime as dt +from datetime import datetime +from ctypes import * +import time +# constant define +WAITS = 5 # wait seconds + +class TestStmtSetTbnameTag: + # + # --------------- main frame ------------------- + def caseDescription(self): + ''' + limit and offset keyword function test cases; + case1: limit offset base function test + case2: offset return valid + ''' + return + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files or "taosd.exe" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root)-len("/build/bin")] + break + return buildPath + + # init + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + # tdSql.prepare() + # self.create_tables(); + self.ts = 1500000000000 + + # stop + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + + # --------------- case ------------------- + + + def newcon(self,host,cfg): + user = "root" + password = "taosdata" + port =6030 + con=taos.connect(host=host, user=user, password=password, config=cfg ,port=port) + tdLog.debug(con) + return con + + def stmtExe(self,conn,sql,bindStat): + queryStat=conn.statement("%s"%sql) + queryStat.bind_param(bindStat) + queryStat.execute() + result=queryStat.use_result() + rows=result.fetch_all() + return rows + + def test_stmt_set_tbname_tag(self,conn): + """测试参数绑定tbname和tag + + 创建参数绑定对象,绑定tbname和多种数据类型tag,插入数据成功 + + Since: v3.3.0.0 + + Labels: stmt, + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + dbname = "stmt_tag" + stablename = 'log' + try: + conn.execute("drop database if exists %s" % dbname) + conn.execute("create database if not exists %s PRECISION 'us' " % dbname) + conn.select_db(dbname) + conn.execute("create table if not exists %s(ts timestamp, bo bool, nil tinyint, ti tinyint, si smallint, ii int,\ + bi bigint, tu tinyint unsigned, su smallint unsigned, iu int unsigned, bu bigint unsigned, \ + ff float, dd double, bb binary(100), nn nchar(100), tt timestamp , vc varchar(100)) tags (t1 timestamp, t2 bool,\ + t3 tinyint, t4 tinyint, t5 smallint, t6 int, t7 bigint, t8 tinyint unsigned, t9 smallint unsigned, \ + t10 int unsigned, t11 bigint unsigned, t12 float, t13 double, t14 binary(100), t15 nchar(100), t16 timestamp)"%stablename) + + stmt = conn.statement("insert into ? using log tags (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?) \ + values (?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?,?)") + tags = new_bind_params(16) + tags[0].timestamp(1626861392589123, PrecisionEnum.Microseconds) + tags[1].bool(True) + tags[2].bool(False) + tags[3].tinyint(2) + tags[4].smallint(3) + tags[5].int(4) + tags[6].bigint(5) + tags[7].tinyint_unsigned(6) + tags[8].smallint_unsigned(7) + tags[9].int_unsigned(8) + tags[10].bigint_unsigned(9) + tags[11].float(10.1) + tags[12].double(10.11) + tags[13].binary("hello") + tags[14].nchar("stmt") + tags[15].timestamp(1626861392589, PrecisionEnum.Milliseconds) + stmt.set_tbname_tags("tb1", tags) + params = new_multi_binds(17) + params[0].timestamp((1626861392589111, 1626861392590111, 1626861392591111)) + params[1].bool((True, None, False)) + params[2].tinyint([-128, -128, None]) # -128 is tinyint null + params[3].tinyint([0, 127, None]) + params[4].smallint([3, None, 2]) + params[5].int([3, 4, None]) + params[6].bigint([3, 4, None]) + params[7].tinyint_unsigned([3, 4, None]) + params[8].smallint_unsigned([3, 4, None]) + params[9].int_unsigned([3, 4, None]) + params[10].bigint_unsigned([3, 4, 5]) + params[11].float([3, None, 1]) + params[12].double([3, None, 1.2]) + params[13].binary(["abc", "dddafadfadfadfadfa", None]) + params[14].nchar(["涛思数据", None, "a long string with 中文?字符"]) + params[15].timestamp([None, None, 1626861392591]) + params[16].binary(["涛思数据16", None, None]) + + stmt.bind_param_batch(params) + stmt.execute() + + assert stmt.affected_rows == 3 + + #query all + queryparam=new_bind_params(1) + queryparam[0].int(10) + rows=self.stmtExe(conn,"select * from log where bu < ?",queryparam) + tdLog.debug("assert 1st case %s"%rows) + assert str(rows[0][0]) == "2021-07-21 17:56:32.589111" + assert rows[0][10] == 3 , '1st case is failed' + assert rows[1][10] == 4 , '1st case is failed' + + #query: Numeric Functions + queryparam=new_bind_params(2) + queryparam[0].int(5) + queryparam[1].int(5) + rows=self.stmtExe(conn,"select abs(?) from log where bu < ?",queryparam) + tdLog.debug("assert 2nd case %s"%rows) + assert rows[0][0] == 5 , '2nd case is failed' + assert rows[1][0] == 5 , '2nd case is failed' + + + #query: Numeric Functions and escapes + queryparam=new_bind_params(1) + queryparam[0].int(5) + rows=self.stmtExe(conn,"select abs(?) from log where nn= 'a? long string with 中文字符'",queryparam) + tdLog.debug("assert 3rd case %s"%rows) + assert rows == [] , '3rd case is failed' + + #query: string Functions + queryparam=new_bind_params(1) + queryparam[0].binary('中文字符') + rows=self.stmtExe(conn,"select CHAR_LENGTH(?) from log ",queryparam) + tdLog.debug("assert 4th case %s"%rows) + assert rows[0][0] == 4, '4th case is failed' + assert rows[1][0] == 4, '4th case is failed' + + queryparam=new_bind_params(1) + queryparam[0].binary('123') + rows=self.stmtExe(conn,"select CHAR_LENGTH(?) from log ",queryparam) + tdLog.debug("assert 4th case %s"%rows) + assert rows[0][0] == 3, '4th.1 case is failed' + assert rows[1][0] == 3, '4th.1 case is failed' + + #query: conversion Functions + queryparam=new_bind_params(1) + queryparam[0].binary('1232a') + rows=self.stmtExe(conn,"select cast( ? as bigint) from log",queryparam) + tdLog.debug("assert 5th case %s"%rows) + assert rows[0][0] == 1232, '5th.1 case is failed' + assert rows[1][0] == 1232, '5th.1 case is failed' + + querystmt4=conn.statement("select cast( ? as binary(10)) from log ") + queryparam=new_bind_params(1) + queryparam[0].int(123) + rows=self.stmtExe(conn,"select cast( ? as bigint) from log",queryparam) + tdLog.debug("assert 6th case %s"%rows) + assert rows[0][0] == 123, '6th.1 case is failed' + assert rows[1][0] == 123, '6th.1 case is failed' + + #query: datatime Functions + queryparam=new_bind_params(1) + queryparam[0].timestamp(1626861392591112) + rows=self.stmtExe(conn,"select timediff('2021-07-21 17:56:32.590111',?,1a) from log",queryparam) + tdLog.debug("assert 7th case %s"%rows) + assert rows[0][0] == -1, '7th case is failed' + assert rows[1][0] == -1, '7th case is failed' + + #query: aggregate Functions + queryparam=new_bind_params(1) + queryparam[0].int(123) + rows=self.stmtExe(conn,"select count(?) from log ",queryparam) + tdLog.debug("assert 8th case %s"%rows) + assert rows[0][0] == 3, ' 8th case is failed' + + #query: selector Functions 9 + queryparam=new_bind_params(1) + queryparam[0].int(2) + rows=self.stmtExe(conn,"select bottom(bu,?) from log group by bu order by bu desc ; ",queryparam) + tdLog.debug("assert 9th case %s"%rows) + assert rows[1][0] == 4, ' 9 case is failed' + assert rows[2][0] == 3, ' 9 case is failed' + + # #query: time-series specific Functions 10 + + querystmt=conn.statement(" select twa(?) from log; ") + queryparam=new_bind_params(1) + queryparam[0].int(15) + rows=self.stmtExe(conn," select twa(?) from log; ",queryparam) + tdLog.debug("assert 10th case %s"%rows) + assert rows[0][0] == 15, ' 10th case is failed' + + + # conn.execute("drop database if exists %s" % dbname) + conn.close() + tdLog.success("%s successfully executed" % __file__) + + except Exception as err: + # conn.execute("drop database if exists %s" % dbname) + conn.close() + raise err + + def run(self): + buildPath = self.getBuildPath() + config = buildPath+ "../sim/dnode1/cfg/" + host="localhost" + connectstmt=self.newcon(host,config) + self.test_stmt_set_tbname_tag(connectstmt) + + return + + +# add case with filename +# +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/test_new/high_availability/2_replica/test_replica2.py b/tests/test_new/high_availability/2_replica/test_replica2.py new file mode 100644 index 0000000000..3124435cb7 --- /dev/null +++ b/tests/test_new/high_availability/2_replica/test_replica2.py @@ -0,0 +1,66 @@ +################################################################### + # Copyright (c) 2016 by TAOS Technologies, Inc. + # All rights reserved. + # + # This file is proprietary and confidential to TAOS Technologies. + # No part of this file may be reproduced, stored, transmitted, + # disclosed or used in any form or by any means other than as + # expressly provided by the written permission from Jianhui Tao + # +################################################################### +from util.cases import * +from util.sql import * +from util.dnodes import * +from util.log import * + +class TestReplica2: + def init(self, conn, logSql, replicaVar=1): + tdLog.debug(f"start to init {__file__}") + self.replicaVar = int(replicaVar) + tdSql.init(conn.cursor(), logSql) + + def test_replica2(self): + """测试双副本 + + 创建双副本db,查看vgroups,停止dnode2和3,校验assigned + + Since: v3.3.0.0 + + Labels: 3nodes, replica + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.run() + def run(self): + tdSql.execute('CREATE DATABASE db vgroups 1 replica 2;') + + time.sleep(1) + + tdSql.query("show db.vgroups;") + + if(tdSql.queryResult[0][4] == "follower") and (tdSql.queryResult[0][6] == "leader"): + tdLog.info("stop dnode2") + sc.dnodeStop(2) + + if(tdSql.queryResult[0][6] == "follower") and (tdSql.queryResult[0][4] == "leader"): + tdLog.info("stop dnode 3") + sc.dnodeStop(3) + + tdLog.info("wait 10 seconds") + time.sleep(10) + + tdSql.query("show db.vgroups;") + + if(tdSql.queryResult[0][4] != "assigned") and (tdSql.queryResult[0][6] != "assigned"): + tdLog.exit("failed to set aasigned") + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/test_new/metadata/time_precision/test_precision_ns.py b/tests/test_new/metadata/time_precision/test_precision_ns.py new file mode 100644 index 0000000000..f45058a035 --- /dev/null +++ b/tests/test_new/metadata/time_precision/test_precision_ns.py @@ -0,0 +1,381 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import sys +import random +import time +import platform +import taos +from util.log import * +from util.cases import * +from util.sql import * + +class TestPrecisionNs: + + # get col value and total max min ... + def getColsValue(self, i, j): + # c1 value + if random.randint(1, 10) == 5: + c1 = None + else: + c1 = 1 + + # c2 value + if j % 3200 == 0: + c2 = 8764231 + elif random.randint(1, 10) == 5: + c2 = None + else: + c2 = random.randint(-87654297, 98765321) + + + value = f"({self.ts}, " + + # c1 + if c1 is None: + value += "null," + else: + self.c1Cnt += 1 + value += f"{c1}," + # c2 + if c2 is None: + value += "null," + else: + value += f"{c2}," + # total count + self.c2Cnt += 1 + # max + if self.c2Max is None: + self.c2Max = c2 + else: + if c2 > self.c2Max: + self.c2Max = c2 + # min + if self.c2Min is None: + self.c2Min = c2 + else: + if c2 < self.c2Min: + self.c2Min = c2 + # sum + if self.c2Sum is None: + self.c2Sum = c2 + else: + self.c2Sum += c2 + + # c3 same with ts + value += f"{self.ts})" + + # move next + self.ts += 1 + + return value + + # insert data + def insertData(self): + tdLog.info("insert data ....") + sqls = "" + for i in range(self.childCnt): + # insert child table + values = "" + pre_insert = f"insert into t{i} values " + for j in range(self.childRow): + if values == "": + values = self.getColsValue(i, j) + else: + values += "," + self.getColsValue(i, j) + + # batch insert + if j % self.batchSize == 0 and values != "": + sql = pre_insert + values + tdSql.execute(sql) + values = "" + # append last + if values != "": + sql = pre_insert + values + tdSql.execute(sql) + values = "" + + sql = "flush database db;" + tdLog.info(sql) + tdSql.execute(sql) + # insert finished + tdLog.info(f"insert data successfully.\n" + f" inserted child table = {self.childCnt}\n" + f" inserted child rows = {self.childRow}\n" + f" total inserted rows = {self.childCnt*self.childRow}\n") + return + + + # prepareEnv + def prepareEnv(self): + # init + self.ts = 1680000000000*1000*1000 + self.childCnt = 5 + self.childRow = 10000 + self.batchSize = 5000 + + # total + self.c1Cnt = 0 + self.c2Cnt = 0 + self.c2Max = None + self.c2Min = None + self.c2Sum = None + + # create database db + sql = f"create database db vgroups 2 precision 'ns' " + tdLog.info(sql) + tdSql.execute(sql) + sql = f"use db" + tdSql.execute(sql) + + # create super talbe st + sql = f"create table st(ts timestamp, c1 int, c2 bigint, ts1 timestamp) tags(area int)" + tdLog.info(sql) + tdSql.execute(sql) + + # create child table + for i in range(self.childCnt): + sql = f"create table t{i} using st tags({i}) " + tdSql.execute(sql) + + # create stream + if platform.system().lower() != 'windows': + sql = "create stream ma into sta as select count(ts) from st interval(100b)" + tdLog.info(sql) + tdSql.execute(sql) + + # insert data + self.insertData() + + # check data correct + def checkExpect(self, sql, expectVal): + tdSql.query(sql) + rowCnt = tdSql.getRows() + for i in range(rowCnt): + val = tdSql.getData(i,0) + if val != expectVal: + tdLog.exit(f"Not expect . query={val} expect={expectVal} i={i} sql={sql}") + return False + + tdLog.info(f"check expect ok. sql={sql} expect ={expectVal} rowCnt={rowCnt}") + return True + + + def test_time_macro(self): + """测试time函数使用ns时间戳 + + 创建数据库指定precision为ns,查询语句使用ns时间戳查询、timetruncate函数使用ns时间戳查询、timediff函数使用ns时间戳查询均返回正确结果 + + Since: v3.3.0.0 + + Labels: precision + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.checkTimeMacro() + + # check time macro + def checkTimeMacro(self): + # 2 week + val = 2 + nsval = -val*7*24*60*60*1000*1000*1000 + expectVal = self.childCnt * self.childRow + sql = f"select count(ts) from st where timediff(ts - {val}w, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + + # 20 day + val = 20 + nsval = -val*24*60*60*1000*1000*1000 + uint = "d" + sql = f"select count(ts) from st where timediff(ts - {val}{uint}, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + + # 30 hour + val = 30 + nsval = -val*60*60*1000*1000*1000 + uint = "h" + sql = f"select count(ts) from st where timediff(ts - {val}{uint}, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + + # 90 minutes + val = 90 + nsval = -val*60*1000*1000*1000 + uint = "m" + sql = f"select count(ts) from st where timediff(ts - {val}{uint}, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + # 2s + val = 2 + nsval = -val*1000*1000*1000 + uint = "s" + sql = f"select count(ts) from st where timediff(ts - {val}{uint}, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + # 20a + val = 5 + nsval = -val*1000*1000 + uint = "a" + sql = f"select count(ts) from st where timediff(ts - {val}{uint}, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + # 300u + val = 300 + nsval = -val*1000 + uint = "u" + sql = f"select count(ts) from st where timediff(ts - {val}{uint}, ts1) = {nsval} " + self.checkExpect(sql, expectVal) + # 8b + val = 8 + sql = f"select timediff(ts1, ts - {val}b) from st " + self.checkExpect(sql, val) + + # timetruncate check + sql = '''select ts,timetruncate(ts,1u), + timetruncate(ts,1b), + timetruncate(ts,1m), + timetruncate(ts,1h), + timetruncate(ts,1w) + from t0 order by ts desc limit 1;''' + tdSql.query(sql) + tdSql.checkData(0,1, "2023-03-28 18:40:00.000009000") + tdSql.checkData(0,2, "2023-03-28 18:40:00.000009999") + tdSql.checkData(0,3, "2023-03-28 18:40:00.000000000") + tdSql.checkData(0,4, "2023-03-28 18:00:00.000000000") + tdSql.checkData(0,5, "2023-03-23 00:00:00.000000000") + + # timediff + sql = '''select ts,timediff(ts,ts+1b,1b), + timediff(ts,ts+1u,1u), + timediff(ts,ts+1a,1a), + timediff(ts,ts+1s,1s), + timediff(ts,ts+1m,1m), + timediff(ts,ts+1h,1h), + timediff(ts,ts+1d,1d), + timediff(ts,ts+1w,1w) + from t0 order by ts desc limit 1;''' + tdSql.query(sql) + tdSql.checkData(0,1, -1) + tdSql.checkData(0,2, -1) + tdSql.checkData(0,3, -1) + tdSql.checkData(0,4, -1) + tdSql.checkData(0,5, -1) + tdSql.checkData(0,6, -1) + tdSql.checkData(0,7, -1) + tdSql.checkData(0,8, -1) + + # init + def init(self, conn, logSql, replicaVar=1): + seed = time.time() % 10000 + random.seed(seed) + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), True) + + + def test_where(self): + """测试where使用ns时间戳 + + 创建数据库指定precision为ns,查询语句中where使用ns时间戳查询均返回正确结果 + + Since: v3.3.0.0 + + Labels: precision + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.checkWhere() + + + # where + def checkWhere(self): + cnt = 300 + start = self.ts - cnt + sql = f"select count(ts) from st where ts >= {start} and ts <= {self.ts}" + self.checkExpect(sql, cnt) + + for i in range(50): + cnt = random.randint(1,40000) + base = 2000 + start = self.ts - cnt - base + end = self.ts - base + sql = f"select count(ts) from st where ts >= {start} and ts < {end}" + self.checkExpect(sql, cnt) + + + def test_stream(self): + """测试steam使用ns时间戳 + + 创建数据库指定precision为ns,创建stream,查询流计算表的字段、时间戳返回正确结果 + + Since: v3.3.0.0 + + Labels: precision + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.checkStream() + + # stream + def checkStream(self): + allRows = self.childCnt * self.childRow + # ensure write data is expected + sql = "select count(*) from (select diff(ts) as a from (select ts from st order by ts asc)) where a=1;" + self.checkExpect(sql, allRows - 1) + + # stream count is ok + sql =f"select count(*) from sta" + cnt = int(allRows / 100) - 1 # last window is not close, so need reduce one + self.checkExpect(sql, cnt) + + # check fields + sql =f"select count(*) from sta where `count(ts)` != 100" + self.checkExpect(sql, 0) + + # check timestamp + sql =f"select count(*) from (select diff(`_wstart`) from sta)" + self.checkExpect(sql, cnt - 1) + sql =f"select count(*) from (select diff(`_wstart`) as a from sta) where a != 100" + self.checkExpect(sql, 0) + + # run + def run(self): + # prepare env + self.prepareEnv() + + # time macro like 1w 1d 1h 1m 1s 1a 1u 1b + self.checkTimeMacro() + + # check where + self.checkWhere() + + # check stream + if platform.system().lower() != 'windows': + self.checkStream() + + # stop + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/mkdocs.yml b/tests/test_new/mkdocs.yml new file mode 100644 index 0000000000..3193b0ff38 --- /dev/null +++ b/tests/test_new/mkdocs.yml @@ -0,0 +1,25 @@ +site_name: TDengine Case List +docs_dir: case_list_docs +theme: + name: "material" + +markdown_extensions: +- attr_list +- toc: + permalink: true + separator: "-" + baselevel: 1 + toc_depth: 4 +plugins: +- search +- mkdocstrings: + custom_templates: templates + handlers: + python: + options: + heading_level: 2 + show_root_heading: true + show_bases: false + show_source: false + show_object_full_path: false + filters: [ "^test_", "^Test" ] diff --git a/tests/test_new/operation/split/test_split_vgroup.py b/tests/test_new/operation/split/test_split_vgroup.py new file mode 100644 index 0000000000..79ba2f8513 --- /dev/null +++ b/tests/test_new/operation/split/test_split_vgroup.py @@ -0,0 +1,481 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import sys +import random +import time +import copy +import string +import platform +import taos +from util.log import * +from util.cases import * +from util.sql import * + +class TestSplitVGroup: + + # random string + def random_string(self, count): + letters = string.ascii_letters + return ''.join(random.choice(letters) for i in range(count)) + + # get col value and total max min ... + def getColsValue(self, i, j): + # c1 value + if random.randint(1, 10) == 5: + c1 = None + else: + c1 = 1 + + # c2 value + if j % 3200 == 0: + c2 = 8764231 + elif random.randint(1, 10) == 5: + c2 = None + else: + c2 = random.randint(-87654297, 98765321) + + + value = f"({self.ts}, " + + # c1 + if c1 is None: + value += "null," + else: + self.c1Cnt += 1 + value += f"{c1}," + # c2 + if c2 is None: + value += "null," + else: + value += f"{c2}," + # total count + self.c2Cnt += 1 + # max + if self.c2Max is None: + self.c2Max = c2 + else: + if c2 > self.c2Max: + self.c2Max = c2 + # min + if self.c2Min is None: + self.c2Min = c2 + else: + if c2 < self.c2Min: + self.c2Min = c2 + # sum + if self.c2Sum is None: + self.c2Sum = c2 + else: + self.c2Sum += c2 + + # c3 same with ts + value += f"{self.ts})" + + # move next + self.ts += 1 + + return value + + # insert data + def insertData(self): + tdLog.info("insert data ....") + sqls = "" + for i in range(self.childCnt): + # insert child table + values = "" + pre_insert = f"insert into @db_name.t{i} values " + for j in range(self.childRow): + if values == "": + values = self.getColsValue(i, j) + else: + values += "," + self.getColsValue(i, j) + + # batch insert + if j % self.batchSize == 0 and values != "": + sql = pre_insert + values + self.exeDouble(sql) + values = "" + # append last + if values != "": + sql = pre_insert + values + self.exeDouble(sql) + values = "" + + # insert nomal talbe + for i in range(20): + self.ts += 1000 + name = self.random_string(20) + sql = f"insert into @db_name.ta values({self.ts}, {i}, {self.ts%100000}, '{name}', false)" + self.exeDouble(sql) + + # insert finished + tdLog.info(f"insert data successfully.\n" + f" inserted child table = {self.childCnt}\n" + f" inserted child rows = {self.childRow}\n" + f" total inserted rows = {self.childCnt*self.childRow}\n") + return + + def exeDouble(self, sql): + # dbname replace + sql1 = sql.replace("@db_name", self.db1) + + if len(sql1) > 100: + tdLog.info(sql1[:100]) + else: + tdLog.info(sql1) + tdSql.execute(sql1) + + sql2 = sql.replace("@db_name", self.db2) + if len(sql2) > 100: + tdLog.info(sql2[:100]) + else: + tdLog.info(sql2) + tdSql.execute(sql2) + + + # prepareEnv + def prepareEnv(self): + # init + self.ts = 1680000000000 + self.childCnt = 4 + self.childRow = 10000 + self.batchSize = 50000 + self.vgroups1 = 1 + self.vgroups2 = 1 + self.db1 = "db1" + self.db2 = "db2" + + # total + self.c1Cnt = 0 + self.c2Cnt = 0 + self.c2Max = None + self.c2Min = None + self.c2Sum = None + + # create database db wal_retention_period 0 + sql = f"create database @db_name vgroups {self.vgroups1} replica {self.replicaVar} wal_retention_period 0 wal_retention_size 1" + self.exeDouble(sql) + + # create super talbe st + sql = f"create table @db_name.st(ts timestamp, c1 int, c2 bigint, ts1 timestamp) tags(area int)" + self.exeDouble(sql) + + # create child table + for i in range(self.childCnt): + sql = f"create table @db_name.t{i} using @db_name.st tags({i}) " + self.exeDouble(sql) + + # create normal table + sql = f"create table @db_name.ta(ts timestamp, c1 int, c2 bigint, c3 binary(32), c4 bool)" + self.exeDouble(sql) + + # insert data + self.insertData() + + # update + self.ts = 1680000000000 + 20000 + self.childRow = 1000 + + + # delete data + sql = "delete from @db_name.st where ts > 1680000019000 and ts < 1680000062000" + self.exeDouble(sql) + sql = "delete from @db_name.st where ts > 1680000099000 and ts < 1680000170000" + self.exeDouble(sql) + + # check data correct + def checkExpect(self, sql, expectVal): + tdSql.query(sql) + rowCnt = tdSql.getRows() + for i in range(rowCnt): + val = tdSql.getData(i,0) + if val != expectVal: + tdLog.exit(f"Not expect . query={val} expect={expectVal} i={i} sql={sql}") + return False + + tdLog.info(f"check expect ok. sql={sql} expect ={expectVal} rowCnt={rowCnt}") + return True + + # init + def init(self, conn, logSql, replicaVar=1): + seed = time.time() % 10000 + random.seed(seed) + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), True) + + # check query result same + def queryDouble(self, sql): + # sql + sql1 = sql.replace('@db_name', self.db1) + tdLog.info(sql1) + start1 = time.time() + rows1 = tdSql.query(sql1,queryTimes=2) + spend1 = time.time() - start1 + res1 = copy.deepcopy(tdSql.queryResult) + + sql2 = sql.replace('@db_name', self.db2) + tdLog.info(sql2) + start2 = time.time() + tdSql.query(sql2,queryTimes=2) + spend2 = time.time() - start2 + res2 = tdSql.queryResult + + rowlen1 = len(res1) + rowlen2 = len(res2) + errCnt = 0 + + if rowlen1 != rowlen2: + tdLog.exit(f"both row count not equal. rowlen1={rowlen1} rowlen2={rowlen2} ") + return False + + for i in range(rowlen1): + row1 = res1[i] + row2 = res2[i] + collen1 = len(row1) + collen2 = len(row2) + if collen1 != collen2: + tdLog.exit(f"both col count not equal. collen1={collen1} collen2={collen2}") + return False + for j in range(collen1): + if row1[j] != row2[j]: + tdLog.info(f"error both column value not equal. row={i} col={j} col1={row1[j]} col2={row2[j]} .") + errCnt += 1 + + if errCnt > 0: + tdLog.exit(f" db2 column value different with db2. different count ={errCnt} ") + + # warning performance + diff = (spend2 - spend1)*100/spend1 + tdLog.info("spend1=%.6fs spend2=%.6fs diff=%.1f%%"%(spend1, spend2, diff)) + if spend2 > spend1 and diff > 20: + tdLog.info("warning: the diff for performance after spliting is over 20%") + + return True + + + # check result + def checkResult(self): + # check vgroupid + sql = f"select vgroup_id from information_schema.ins_vgroups where db_name='{self.db2}'" + tdSql.query(sql,queryTimes=2) + tdSql.checkRows(self.vgroups2) + + # check child table count same + sql = "select table_name from information_schema.ins_tables where db_name='@db_name' order by table_name" + self.queryDouble(sql) + + # check row value is ok + sql = "select * from @db_name.st order by ts, tbname" + self.queryDouble(sql) + + # where + sql = "select *,tbname from @db_name.st where c1 < 1000 order by ts, tbname" + self.queryDouble(sql) + + # max + sql = "select max(c1) from @db_name.st" + self.queryDouble(sql) + + # min + sql = "select min(c2) from @db_name.st" + self.queryDouble(sql) + + # sum + sql = "select sum(c1) from @db_name.st" + self.queryDouble(sql) + + # normal table + + # count + sql = "select count(*) from @db_name.ta" + self.queryDouble(sql) + + # all rows + sql = "select * from @db_name.ta" + self.queryDouble(sql) + + # sum + sql = "select sum(c1) from @db_name.ta" + self.queryDouble(sql) + + + # get vgroup list + def getVGroup(self, db_name): + vgidList = [] + sql = f"select vgroup_id from information_schema.ins_vgroups where db_name='{db_name}'" + res = tdSql.getResult(sql) + rows = len(res) + for i in range(rows): + vgidList.append(res[i][0]) + + return vgidList; + + # split vgroup on db2 + def splitVGroup(self, db_name): + vgids = self.getVGroup(db_name) + selid = random.choice(vgids) + sql = f"split vgroup {selid}" + tdLog.info(sql) + tdSql.execute(sql) + + # wait end + seconds = 300 + for i in range(seconds): + sql ="show transactions;" + rows = tdSql.query(sql) + if rows == 0: + tdLog.info("split vgroup finished.") + return True + #tdLog.info(f"i={i} wait split vgroup ...") + time.sleep(1) + + tdLog.exit(f"split vgroup transaction is not finished after executing {seconds}s") + return False + + # split error + def expectSplitError(self, dbName): + vgids = self.getVGroup(dbName) + selid = random.choice(vgids) + sql = f"split vgroup {selid}" + tdLog.info(sql) + tdSql.error(sql) + + # expect split ok + def expectSplitOk(self, dbName): + # split vgroup + vgList1 = self.getVGroup(dbName) + self.splitVGroup(dbName) + vgList2 = self.getVGroup(dbName) + vgNum1 = len(vgList1) + 1 + vgNum2 = len(vgList2) + if vgNum1 != vgNum2: + tdLog.exit(f" vglist len={vgNum1} is not same for expect {vgNum2}") + return + + # split empty database + def splitEmptyDB(self): + dbName = "emptydb" + vgNum = 2 + # create database + sql = f"create database {dbName} vgroups {vgNum} replica {self.replicaVar }" + tdLog.info(sql) + tdSql.execute(sql) + + # split vgroup + self.expectSplitOk(dbName) + + + # forbid + def checkForbid(self): + # stream + if platform.system().lower() != 'windows': + tdLog.info("check forbid split having stream...") + tdSql.execute("create database streamdb;") + tdSql.execute("use streamdb;") + tdSql.execute("create table ta(ts timestamp, age int);") + tdSql.execute("create stream ma into sta as select count(*) from ta interval(1s);") + self.expectSplitError("streamdb") + tdSql.execute("drop stream ma;") + self.expectSplitOk("streamdb") + + # topic + tdLog.info("check forbid split having topic...") + tdSql.execute("create database topicdb wal_retention_period 10;") + tdSql.execute("use topicdb;") + tdSql.execute("create table ta(ts timestamp, age int);") + tdSql.execute("create topic toa as select * from ta;") + + #self.expectSplitError("topicdb") + tdSql.execute("drop topic toa;") + self.expectSplitOk("topicdb") + + # compact and check db2 + def compactAndCheck(self): + tdLog.info("compact db2 and check result ...") + # compact + tdSql.execute(f"compact database {self.db2};") + # check result + self.checkResult() + + + def test_run(self): + """测试split vgroup + + 创建两个数据库,写入多条同样的数据并落盘内存中的数据,在db2上执行split vgroup,相同数据重写写入,比较两个数据库数据一致 + + Since: v3.0.6.0 + + Labels: vgroup + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + # run + def run(self): + # prepare env + self.prepareEnv() + tdLog.info("generate at least two stt files of the same fileset (e.g. v4f1944) for db2 and db1 ") + for dbname in [self.db2, self.db1]: + tdSql.execute(f'insert into {dbname}.t1 values("2023-03-28 10:40:00.010",103,103,"2023-03-28 18:41:39.999") ;') + tdSql.execute(f'flush database {dbname}') + tdSql.execute(f'insert into {dbname}.t1 values("2023-03-28 10:40:00.100",103,103,"2023-03-28 18:41:39.999") ;') + tdSql.execute(f'flush database {dbname}') + tdSql.execute(f'insert into {dbname}.t1 values("2023-03-28 10:40:00.100",103,103,"2023-03-28 18:41:39.999") ;') + tdSql.execute(f'flush database {dbname}') + tdLog.info("check db1 and db2 same after creating ...") + + self.checkResult() + + for i in range(3): + # split vgroup on db2 + start = time.time() + self.splitVGroup(self.db2) + end = time.time() + self.vgroups2 += 1 + + # insert the same data per tables into splited vgroups + tdLog.info("insert the same data per tables into splited vgroups(3,4)") + for dbname in [self.db2, self.db1]: + for tableid in range(self.childCnt): + tdSql.execute(f'insert into {dbname}.t{tableid} values("2023-03-28 10:40:00.100",103,103,"2023-03-28 18:41:39.999") ;') + tdSql.execute(f'flush database {dbname}') + tdSql.execute(f'insert into {dbname}.ta values("2023-03-28 10:40:00.100",103,103,"2023-03-28 18:41:39.999",0);') + tdSql.execute(f'flush database {dbname}') + + # check two db query result same + self.checkResult() + spend = "%.3f"%(end-start) + tdLog.info(f"split vgroup i={i} passed. spend = {spend}s") + + # split empty db + self.splitEmptyDB() + + # check topic and stream forib + self.checkForbid() + + # compact database + self.compactAndCheck() + + # stop + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/pytest.sh b/tests/test_new/pytest.sh new file mode 100755 index 0000000000..060717c20e --- /dev/null +++ b/tests/test_new/pytest.sh @@ -0,0 +1,114 @@ +#!/bin/bash + +################################################## +# +# Do simulation test +# +################################################## + +set +e +#set -x +if [[ "$OSTYPE" == "darwin"* ]]; then + TD_OS="Darwin" +else + OS=$(cat /etc/*-release | grep "^NAME=" | cut -d= -f2) + len=$(echo ${#OS}) + len=$((len - 2)) + TD_OS=$(echo -ne ${OS:1:${len}} | cut -d" " -f1) +fi + +UNAME_BIN=$(which uname) +OS_TYPE=$($UNAME_BIN) + +cd . + +# Get responsible directories +CODE_DIR=$(dirname $0) +CODE_DIR=$(pwd) + +IN_TDINTERNAL="community" +if [[ "$CODE_DIR" == *"$IN_TDINTERNAL"* ]]; then + cd ../../.. +else + cd ../../ +fi + +TOP_DIR=$(pwd) +TAOSD_DIR=$(find . -name "taosd" | grep bin | head -n1) + +cut_opt="-f " + +if [[ "$TAOSD_DIR" == *"$IN_TDINTERNAL"* ]]; then + BIN_DIR=$(find . -name "taosd" | grep bin | head -n1 | cut -d '/' ${cut_opt}2,3) +else + BIN_DIR=$(find . -name "taosd" | grep bin | head -n1 | cut -d '/' ${cut_opt}2) +fi + +declare -x BUILD_DIR=$TOP_DIR/$BIN_DIR +declare -x SIM_DIR=$TOP_DIR/sim +PROGRAM=$BUILD_DIR/build/bin/tsim +PRG_DIR=$SIM_DIR/tsim +ASAN_DIR=$SIM_DIR/asan + +chmod -R 777 $PRG_DIR +echo "------------------------------------------------------------------------" +echo "Start TDengine Testing Case ..." +echo "BUILD_DIR: $BUILD_DIR" +echo "SIM_DIR : $SIM_DIR" +echo "CODE_DIR : $CODE_DIR" +echo "ASAN_DIR : $ASAN_DIR" + +# prevent delete / folder or /usr/bin +if [ ${#SIM_DIR} -lt 10 ]; then + echo "len(SIM_DIR) < 10 , danger so exit. SIM_DIR=$SIM_DIR" + exit 1 +fi + +rm -rf "${SIM_DIR:?}"/* + +mkdir -p $PRG_DIR +mkdir -p $ASAN_DIR + +cd "$CODE_DIR" || exit +ulimit -n 600000 +ulimit -c unlimited + +#sudo sysctl -w kernel.core_pattern=$TOP_DIR/core.%p.%e + +echo "ExcuteCmd:" $* + +if [[ "$TD_OS" == "Alpine" ]]; then + "$@" +else + AsanFile=$ASAN_DIR/psim.info + echo "AsanFile:" "$AsanFile" + + unset LD_PRELOAD + #export LD_PRELOAD=libasan.so.5 + #export LD_PRELOAD=$(gcc -print-file-name=libasan.so) + export LD_PRELOAD="$(realpath "$(gcc -print-file-name=libasan.so)") $(realpath "$(gcc -print-file-name=libstdc++.so)")" + echo "Preload AsanSo:" $? + + $* -a 2> $AsanFile + cat $AsanFile + unset LD_PRELOAD + for ((i = 1; i <= 20; i++)); do + AsanFileLen=$(cat $AsanFile | wc -l) + echo "AsanFileLen:" $AsanFileLen + if [ $AsanFileLen -gt 10 ]; then + break + fi + sleep 1 + done + # check case successful + AsanFileSuccessLen=$(grep -w "successfully executed" $AsanFile | wc -l) + echo "AsanFileSuccessLen:" $AsanFileSuccessLen + + if [ $AsanFileSuccessLen -gt 0 ]; then + echo "Execute script successfully and check asan" + $CODE_DIR/../script/sh/checkAsan.sh + else + echo "Execute script failure" + exit 1 + fi +fi diff --git a/tests/test_new/query/function/test_sin.py b/tests/test_new/query/function/test_sin.py new file mode 100644 index 0000000000..48a539734e --- /dev/null +++ b/tests/test_new/query/function/test_sin.py @@ -0,0 +1,628 @@ +import taos +import sys +import datetime +import inspect +import math +from util.log import * +from util.sql import * +from util.cases import * + + +class TestSin: + + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor()) + + def prepare_datas(self, dbname="db"): + tdSql.execute( + f'''create table {dbname}.stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t1 int) + ''' + ) + + tdSql.execute( + f''' + create table {dbname}.t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(4): + tdSql.execute(f'create table {dbname}.ct{i+1} using {dbname}.stb1 tags ( {i+1} )') + + for i in range(9): + tdSql.execute( + f"insert into {dbname}.ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into {dbname}.ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute(f"insert into {dbname}.ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute(f"insert into {dbname}.ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute(f"insert into {dbname}.ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute(f"insert into {dbname}.ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute(f"insert into {dbname}.ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute(f"insert into {dbname}.ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute(f"insert into {dbname}.ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into {dbname}.t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + def check_result_auto_sin(self ,origin_query , pow_query): + + pow_result = tdSql.getResult(pow_query) + + origin_result = tdSql.getResult(origin_query) + + auto_result =[] + + for row in origin_result: + row_check = [] + for elem in row: + if elem == None: + elem = None + else: + elem = math.sin(elem) + row_check.append(elem) + auto_result.append(row_check) + tdSql.query(pow_query) + for row_index , row in enumerate(pow_result): + for col_index , elem in enumerate(row): + tdSql.checkData(row_index ,col_index ,auto_result[row_index][col_index]) + + + def test_errors(self, dbname="db"): + """测试sin()函数error + + 执行非法select语句包含sin()函数,返回预期错误 + + Since: v3.3.0.0 + + Labels: sin, negative + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + error_sql_lists = [ + f"select sin from {dbname}.t1", + # f"select sin(-+--+c1 ) from {dbname}.t1", + # f"select +-sin(c1) from {dbname}.t1", + # f"select ++-sin(c1) from {dbname}.t1", + # f"select ++--sin(c1) from {dbname}.t1", + # f"select - -sin(c1)*0 from {dbname}.t1", + # f"select sin(tbname+1) from {dbname}.t1 ", + f"select sin(123--123)==1 from {dbname}.t1", + f"select sin(c1) as 'd1' from {dbname}.t1", + f"select sin(c1 ,c2) from {dbname}.t1", + f"select sin(c1 ,NULL ) from {dbname}.t1", + f"select sin(,) from {dbname}.t1;", + f"select sin(sin(c1) ab from {dbname}.t1)", + f"select sin(c1 ) as int from {dbname}.t1", + f"select sin from {dbname}.stb1", + # f"select sin(-+--+c1) from {dbname}.stb1", + # f"select +-sin(c1) from {dbname}.stb1", + # f"select ++-sin(c1) from {dbname}.stb1", + # f"select ++--sin(c1) from {dbname}.stb1", + # f"select - -sin(c1)*0 from {dbname}.stb1", + # f"select sin(tbname+1) from {dbname}.stb1 ", + f"select sin(123--123)==1 from {dbname}.stb1", + f"select sin(c1) as 'd1' from {dbname}.stb1", + f"select sin(c1 ,c2 ) from {dbname}.stb1", + f"select sin(c1 ,NULL) from {dbname}.stb1", + f"select sin(,) from {dbname}.stb1;", + f"select sin(sin(c1) ab from {dbname}.stb1)", + f"select sin(c1) as int from {dbname}.stb1" + ] + for error_sql in error_sql_lists: + tdSql.error(error_sql) + + + def test_types(self): + """测试sin()函数数据类型 + + select语句包含sin()函数,参数传入不同数据类型,不支持的类型返回预期错误,支持的类型返回执行成功 + + Since: v3.3.0.0 + + Labels: sin, dataType + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + - 2024-2-7 Hong updated for feature TD-23456 + + """ + self.support_types() + def support_types(self, dbname="db"): + type_error_sql_lists = [ + f"select sin(ts) from {dbname}.t1" , + f"select sin(c7) from {dbname}.t1", + f"select sin(c8) from {dbname}.t1", + f"select sin(c9) from {dbname}.t1", + f"select sin(ts) from {dbname}.ct1" , + f"select sin(c7) from {dbname}.ct1", + f"select sin(c8) from {dbname}.ct1", + f"select sin(c9) from {dbname}.ct1", + f"select sin(ts) from {dbname}.ct3" , + f"select sin(c7) from {dbname}.ct3", + f"select sin(c8) from {dbname}.ct3", + f"select sin(c9) from {dbname}.ct3", + f"select sin(ts) from {dbname}.ct4" , + f"select sin(c7) from {dbname}.ct4", + f"select sin(c8) from {dbname}.ct4", + f"select sin(c9) from {dbname}.ct4", + f"select sin(ts) from {dbname}.stb1" , + f"select sin(c7) from {dbname}.stb1", + f"select sin(c8) from {dbname}.stb1", + f"select sin(c9) from {dbname}.stb1" , + + f"select sin(ts) from {dbname}.stbbb1" , + f"select sin(c7) from {dbname}.stbbb1", + + f"select sin(ts) from {dbname}.tbname", + f"select sin(c9) from {dbname}.tbname" + + ] + + for type_sql in type_error_sql_lists: + tdSql.error(type_sql) + + + type_sql_lists = [ + f"select sin(c1) from {dbname}.t1", + f"select sin(c2) from {dbname}.t1", + f"select sin(c3) from {dbname}.t1", + f"select sin(c4) from {dbname}.t1", + f"select sin(c5) from {dbname}.t1", + f"select sin(c6) from {dbname}.t1", + + f"select sin(c1) from {dbname}.ct1", + f"select sin(c2) from {dbname}.ct1", + f"select sin(c3) from {dbname}.ct1", + f"select sin(c4) from {dbname}.ct1", + f"select sin(c5) from {dbname}.ct1", + f"select sin(c6) from {dbname}.ct1", + + f"select sin(c1) from {dbname}.ct3", + f"select sin(c2) from {dbname}.ct3", + f"select sin(c3) from {dbname}.ct3", + f"select sin(c4) from {dbname}.ct3", + f"select sin(c5) from {dbname}.ct3", + f"select sin(c6) from {dbname}.ct3", + + f"select sin(c1) from {dbname}.stb1", + f"select sin(c2) from {dbname}.stb1", + f"select sin(c3) from {dbname}.stb1", + f"select sin(c4) from {dbname}.stb1", + f"select sin(c5) from {dbname}.stb1", + f"select sin(c6) from {dbname}.stb1", + + f"select sin(c6) as alisb from {dbname}.stb1", + f"select sin(c6) alisb from {dbname}.stb1", + ] + + for type_sql in type_sql_lists: + tdSql.query(type_sql) + + def test_basic_sin_function(self): + """测试sin()函数基础功能 + + 使用包含sin()函数的select语句,查询空表、子表、普通表、不存在表;嵌套sin函数查询均返回成功,与聚合函数一起查询返回失败 + + Since: v3.3.0.0 + + Labels: sin + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.basic_sin_function() + + def basic_sin_function(self, dbname="db"): + + # basic query + tdSql.query(f"select c1 from {dbname}.ct3") + tdSql.checkRows(0) + tdSql.query(f"select c1 from {dbname}.t1") + tdSql.checkRows(12) + tdSql.query(f"select c1 from {dbname}.stb1") + tdSql.checkRows(25) + + # used for empty table , ct3 is empty + tdSql.query(f"select sin(c1) from {dbname}.ct3") + tdSql.checkRows(0) + tdSql.query(f"select sin(c2) from {dbname}.ct3") + tdSql.checkRows(0) + tdSql.query(f"select sin(c3) from {dbname}.ct3") + tdSql.checkRows(0) + tdSql.query(f"select sin(c4) from {dbname}.ct3") + tdSql.checkRows(0) + tdSql.query(f"select sin(c5) from {dbname}.ct3") + tdSql.checkRows(0) + tdSql.query(f"select sin(c6) from {dbname}.ct3") + tdSql.checkRows(0) + + + # # used for regular table + tdSql.query(f"select sin(c1) from {dbname}.t1") + tdSql.checkData(0, 0, None) + tdSql.checkData(1 , 0, 0.841470985) + tdSql.checkData(3 , 0, 0.141120008) + tdSql.checkData(5 , 0, None) + + tdSql.query(f"select c1, c2, c3 , c4, c5 from {dbname}.t1") + tdSql.checkData(1, 4, 1.11000) + tdSql.checkData(3, 3, 33) + tdSql.checkData(5, 4, None) + + tdSql.query(f"select ts,c1, c2, c3 , c4, c5 from {dbname}.t1") + tdSql.checkData(1, 5, 1.11000) + tdSql.checkData(3, 4, 33) + tdSql.checkData(5, 5, None) + + self.check_result_auto_sin( f"select abs(c1), abs(c2), abs(c3) , abs(c4), abs(c5) from {dbname}.t1", f"select sin(abs(c1)), sin(abs(c2)) ,sin(abs(c3)), sin(abs(c4)), sin(abs(c5)) from {dbname}.t1") + + # used for sub table + tdSql.query(f"select c2 ,sin(c2) from {dbname}.ct1") + tdSql.checkData(0, 1, -0.220708349) + tdSql.checkData(1 , 1, -0.556921845) + tdSql.checkData(3 , 1, -0.798311364) + tdSql.checkData(4 , 1, 0.000000000) + + tdSql.query(f"select c1, c5 ,sin(c5) from {dbname}.ct4") + tdSql.checkData(0 , 2, None) + tdSql.checkData(1 , 2, 0.518228108) + tdSql.checkData(2 , 2, 0.996475613) + tdSql.checkData(3 , 2, 0.367960369) + tdSql.checkData(5 , 2, None) + + self.check_result_auto_sin( f"select c1, c2, c3 , c4, c5 from {dbname}.ct1", f"select sin(c1), sin(c2) ,sin(c3), sin(c4), sin(c5) from {dbname}.ct1") + + # nest query for sin functions + tdSql.query(f"select c4 , sin(c4) ,sin(sin(c4)) , sin(sin(sin(c4))) from {dbname}.ct1;") + tdSql.checkData(0 , 0 , 88) + tdSql.checkData(0 , 1 , 0.035398303) + tdSql.checkData(0 , 2 , 0.035390911) + tdSql.checkData(0 , 3 , 0.035383523) + + tdSql.checkData(1 , 0 , 77) + tdSql.checkData(1 , 1 , 0.999520159) + tdSql.checkData(1 , 2 , 0.841211629) + tdSql.checkData(1 , 3 , 0.745451290) + + tdSql.checkData(11 , 0 , -99) + tdSql.checkData(11 , 1 , 0.999206834) + tdSql.checkData(11 , 2 , 0.841042171) + tdSql.checkData(11 , 3 , 0.745338326) + + # used for stable table + + tdSql.query(f"select sin(c1) from {dbname}.stb1") + tdSql.checkRows(25) + + + # used for not exists table + tdSql.error(f"select sin(c1) from {dbname}.stbbb1") + tdSql.error(f"select sin(c1) from {dbname}.tbname") + tdSql.error(f"select sin(c1) from {dbname}.ct5") + + # mix with common col + tdSql.query(f"select c1, sin(c1) from {dbname}.ct1") + tdSql.query(f"select c2, sin(c2) from {dbname}.ct4") + + + # mix with common functions + tdSql.query(f"select c1, sin(c1),sin(c1), sin(sin(c1)) from {dbname}.ct4 ") + tdSql.checkData(0 , 0 ,None) + tdSql.checkData(0 , 1 ,None) + tdSql.checkData(0 , 2 ,None) + tdSql.checkData(0 , 3 ,None) + + tdSql.checkData(3 , 0 , 6) + tdSql.checkData(3 , 1 ,-0.279415498) + tdSql.checkData(3 , 2 ,-0.279415498) + tdSql.checkData(3 , 3 ,-0.275793863) + + tdSql.query(f"select c1, sin(c1),c5, floor(c5) from {dbname}.stb1 ") + + # # mix with agg functions , not support + tdSql.error(f"select c1, sin(c1),c5, count(c5) from {dbname}.stb1 ") + tdSql.error(f"select c1, sin(c1),c5, count(c5) from {dbname}.ct1 ") + tdSql.error(f"select sin(c1), count(c5) from {dbname}.stb1 ") + tdSql.error(f"select sin(c1), count(c5) from {dbname}.ct1 ") + tdSql.error(f"select c1, count(c5) from {dbname}.ct1 ") + tdSql.error(f"select c1, count(c5) from {dbname}.stb1 ") + + # agg functions mix with agg functions + + tdSql.query(f"select max(c5), count(c5) from {dbname}.stb1") + tdSql.query(f"select max(c5), count(c5) from {dbname}.ct1") + + + # # bug fix for compute + tdSql.query(f"select c1, sin(c1) -0 ,sin(c1-4)-0 from {dbname}.ct4 ") + tdSql.checkData(0, 0, None) + tdSql.checkData(0, 1, None) + tdSql.checkData(0, 2, None) + tdSql.checkData(1, 0, 8) + tdSql.checkData(1, 1, 0.989358247) + tdSql.checkData(1, 2, -0.756802495) + + tdSql.query(f"select c1, sin(c1) -0 ,sin(c1-0.1)-0.1 from {dbname}.ct4") + tdSql.checkData(0, 0, None) + tdSql.checkData(0, 1, None) + tdSql.checkData(0, 2, None) + tdSql.checkData(1, 0, 8) + tdSql.checkData(1, 1, 0.989358247) + tdSql.checkData(1, 2, 0.898941342) + + tdSql.query(f"select c1, sin(c1), c2, sin(c2), c3, sin(c3) from {dbname}.ct1") + + def test_big_number(self, dbname="db"): + """测试sin()函数大数参数 + + 使用包含sin()函数的select语句查询,参数为几十位的double参数,返回结果正确 + + Since: v3.3.0.0 + + Labels: sin + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + + tdSql.query(f"select c1, sin(100000000) from {dbname}.ct1") # bigint to double data overflow + tdSql.checkData(4, 1, math.sin(100000000)) + + + tdSql.query(f"select c1, sin(10000000000000) from {dbname}.ct1") # bigint to double data overflow + tdSql.checkData(4, 1, math.sin(10000000000000)) + + tdSql.query(f"select c1, sin(10000000000000000000000000) from {dbname}.ct1") # bigint to double data overflow + tdSql.query(f"select c1, sin(10000000000000000000000000.0) from {dbname}.ct1") # 10000000000000000000000000.0 is a double value + tdSql.checkData(1, 1, math.sin(10000000000000000000000000.0)) + + tdSql.query(f"select c1, sin(10000000000000000000000000000000000) from {dbname}.ct1") # bigint to double data overflow + tdSql.query(f"select c1, sin(10000000000000000000000000000000000.0) from {dbname}.ct1") # 10000000000000000000000000.0 is a double value + tdSql.checkData(4, 1, math.sin(10000000000000000000000000000000000.0)) + + tdSql.query(f"select c1, sin(10000000000000000000000000000000000000000) from {dbname}.ct1") # bigint to double data overflow + tdSql.query(f"select c1, sin(10000000000000000000000000000000000000000.0) from {dbname}.ct1") # 10000000000000000000000000.0 is a double value + + tdSql.checkData(4, 1, math.sin(10000000000000000000000000000000000000000.0)) + + tdSql.query(f"select c1, sin(10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000) from {dbname}.ct1") # bigint to double data overflow + + + def test_func_filter(self): + """测试sin()函数filter + + 使用包含sin()函数的where语句查询,返回结果正确 + + Since: v3.3.0.0 + + Labels: sin + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.abs_func_filter() + def abs_func_filter(self, dbname="db"): + tdSql.query(f"select c1, abs(c1) -0 ,ceil(c1-0.1)-0 ,floor(c1+0.1)-0.1 ,ceil(sin(c1)-0.5) from {dbname}.ct4 where c1>5 ") + tdSql.checkRows(3) + tdSql.checkData(0,0,8) + tdSql.checkData(0,1,8.000000000) + tdSql.checkData(0,2,8.000000000) + tdSql.checkData(0,3,7.900000000) + tdSql.checkData(0,4,1.000000000) + + tdSql.query(f"select c1, abs(c1) -0 ,ceil(c1-0.1)-0 ,floor(c1+0.1)-0.1 ,ceil(sin(c1)-0.5) from {dbname}.ct4 where c1=5 ") + tdSql.checkRows(1) + tdSql.checkData(0,0,5) + tdSql.checkData(0,1,5.000000000) + tdSql.checkData(0,2,5.000000000) + tdSql.checkData(0,3,4.900000000) + tdSql.checkData(0,4,-1.000000000) + + tdSql.query(f"select c1,c2 , abs(c1) -0 ,ceil(c1-0.1)-0 ,floor(c1+0.1)-0.1 ,ceil(sin(c1)-0.5) from {dbname}.ct4 where c1=sin(c1) limit 1 ") + tdSql.checkRows(1) + tdSql.checkData(0,0,0) + tdSql.checkData(0,1,0) + tdSql.checkData(0,2,0.000000000) + tdSql.checkData(0,3,0.000000000) + tdSql.checkData(0,4,-0.100000000) + tdSql.checkData(0,5,0.000000000) + + + def test_basic_sin_function(self): + """测试sin()函数边界值 + + 使用包含sin()函数的select语句,参数为不同数字类型的边界值,查询均返回成功 + + Since: v3.3.0.0 + + Labels: sin + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.check_boundary_values() + def check_boundary_values(self, dbname="testdb"): + + PI=3.1415926 + + tdSql.execute(f"drop database if exists {dbname}") + tdSql.execute(f"create database if not exists {dbname}") + tdSql.execute( + f"create table {dbname}.stb_bound (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(32),c9 nchar(32), c10 timestamp) tags (t1 int);" + ) + tdSql.execute(f'create table {dbname}.sub1_bound using {dbname}.stb_bound tags ( 1 )') + tdSql.execute( + f"insert into {dbname}.sub1_bound values ( now()-10s, 2147483647, 9223372036854775807, 32767, 127, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )" + ) + tdSql.execute( + f"insert into {dbname}.sub1_bound values ( now()-5s, -2147483647, -9223372036854775807, -32767, -127, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )" + ) + tdSql.execute( + f"insert into {dbname}.sub1_bound values ( now(), 2147483646, 9223372036854775806, 32766, 126, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )" + ) + tdSql.execute( + f"insert into {dbname}.sub1_bound values ( now()+5s, -2147483646, -9223372036854775806, -32766, -126, -3.40E+38, -1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )" + ) + tdSql.error( + f"insert into {dbname}.sub1_bound values ( now()+10s, 2147483648, 9223372036854775808, 32768, 128, 3.40E+38, 1.7e+308, True, 'binary_tb1', 'nchar_tb1', now() )" + ) + self.check_result_auto_sin( f"select abs(c1), abs(c2), abs(c3) , abs(c4) from {dbname}.sub1_bound ", f"select sin(abs(c1)), sin(abs(c2)) ,sin(abs(c3)), sin(abs(c4)) from {dbname}.sub1_bound") + + self.check_result_auto_sin( f"select c1, c2, c3 , c3, c2 ,c1 from {dbname}.sub1_bound ", f"select sin(c1), sin(c2) ,sin(c3), sin(c3), sin(c2) ,sin(c1) from {dbname}.sub1_bound") + + self.check_result_auto_sin(f"select abs(abs(abs(abs(abs(abs(abs(abs(abs(c1))))))))) nest_col_func from {dbname}.sub1_bound" , f"select sin(abs(c1)) from {dbname}.sub1_bound" ) + + # check basic elem for table per row + tdSql.query(f"select sin(abs(c1)) ,sin(abs(c2)) , sin(abs(c3)) , sin(abs(c4)), sin(abs(c5)), sin(abs(c6)) from {dbname}.sub1_bound ") + tdSql.checkData(0,0,math.sin(2147483647)) + tdSql.checkData(0,1,math.sin(9223372036854775807)) + tdSql.checkData(0,2,math.sin(32767)) + tdSql.checkData(0,3,math.sin(127)) + tdSql.checkData(0,4,math.sin(339999995214436424907732413799364296704.00000)) + tdSql.checkData(1,0,math.sin(2147483647)) + tdSql.checkData(1,1,math.sin(9223372036854775807)) + tdSql.checkData(1,2,math.sin(32767)) + tdSql.checkData(1,3,math.sin(127)) + tdSql.checkData(1,4,math.sin(339999995214436424907732413799364296704.00000)) + tdSql.checkData(3,0,math.sin(2147483646)) + tdSql.checkData(3,1,math.sin(9223372036854775806)) + tdSql.checkData(3,2,math.sin(32766)) + tdSql.checkData(3,3,math.sin(126)) + tdSql.checkData(3,4,math.sin(339999995214436424907732413799364296704.00000)) + + # check + - * / in functions + tdSql.query(f"select sin(abs(c1+1)) ,sin(abs(c2)) , sin(abs(c3*1)) , sin(abs(c4/2)), sin(abs(c5))/2, sin(abs(c6)) from {dbname}.sub1_bound ") + tdSql.checkData(0,0,math.sin(2147483648.000000000)) + tdSql.checkData(0,1,math.sin(9223372036854775807)) + tdSql.checkData(0,2,math.sin(32767.000000000)) + tdSql.checkData(0,3,math.sin(63.500000000)) + + tdSql.execute(f"create stable {dbname}.st (ts timestamp, num1 float, num2 double) tags (t1 int);") + tdSql.execute(f'create table {dbname}.tb1 using {dbname}.st tags (1)') + tdSql.execute(f'create table {dbname}.tb2 using {dbname}.st tags (2)') + tdSql.execute(f'create table {dbname}.tb3 using {dbname}.st tags (3)') + tdSql.execute(f'insert into {dbname}.tb1 values (now()-40s, {PI/2}, {PI/2})') + tdSql.execute(f'insert into {dbname}.tb1 values (now()-30s, {PI}, {PI})') + tdSql.execute(f'insert into {dbname}.tb1 values (now()-20s, {PI*1.5}, {PI*1.5})') + tdSql.execute(f'insert into {dbname}.tb1 values (now()-10s, {PI*2}, {PI*2})') + tdSql.execute(f'insert into {dbname}.tb1 values (now(), {PI*2.5}, {PI*2.5})') + + tdSql.execute(f'insert into {dbname}.tb2 values (now()-40s, {PI/2}, {PI/2})') + tdSql.execute(f'insert into {dbname}.tb2 values (now()-30s, {PI}, {PI})') + tdSql.execute(f'insert into {dbname}.tb2 values (now()-20s, {PI*1.5}, {PI*1.5})') + tdSql.execute(f'insert into {dbname}.tb2 values (now()-10s, {PI*2}, {PI*2})') + tdSql.execute(f'insert into {dbname}.tb2 values (now(), {PI*2.5}, {PI*2.5})') + + self.check_result_auto_sin(f"select num1,num2 from {dbname}.tb3;" , f"select sin(num1),sin(num2) from {dbname}.tb3") + + def test_super_table(self): + """测试sin()函数查询超表 + + Description: 使用包含sin()函数的select语句,查询超级表返回成功 + + Since: v3.3.0.0 + + Labels: sin + + Jira: TD-12345,TS-1234 + + Update history: + 2024-2-6 Feng Chao Created + + """ + self.support_super_table_test() + def support_super_table_test(self, dbname="db"): + self.check_result_auto_sin( f"select c5 from {dbname}.stb1 order by ts " , f"select sin(c5) from {dbname}.stb1 order by ts" ) + self.check_result_auto_sin( f"select c5 from {dbname}.stb1 order by tbname " , f"select sin(c5) from {dbname}.stb1 order by tbname" ) + self.check_result_auto_sin( f"select c5 from {dbname}.stb1 where c1 > 0 order by tbname " , f"select sin(c5) from {dbname}.stb1 where c1 > 0 order by tbname" ) + self.check_result_auto_sin( f"select c5 from {dbname}.stb1 where c1 > 0 order by tbname " , f"select sin(c5) from {dbname}.stb1 where c1 > 0 order by tbname" ) + + self.check_result_auto_sin( f"select t1,c5 from {dbname}.stb1 order by ts " , f"select sin(t1), sin(c5) from {dbname}.stb1 order by ts" ) + self.check_result_auto_sin( f"select t1,c5 from {dbname}.stb1 order by tbname " , f"select sin(t1) ,sin(c5) from {dbname}.stb1 order by tbname" ) + self.check_result_auto_sin( f"select t1,c5 from {dbname}.stb1 where c1 > 0 order by tbname " , f"select sin(t1) ,sin(c5) from {dbname}.stb1 where c1 > 0 order by tbname" ) + self.check_result_auto_sin( f"select t1,c5 from {dbname}.stb1 where c1 > 0 order by tbname " , f"select sin(t1) , sin(c5) from {dbname}.stb1 where c1 > 0 order by tbname" ) + + def run(self): # sourcery skip: extract-duplicate-method, remove-redundant-fstring + tdSql.prepare() + + tdLog.printNoPrefix("==========step1:create table ==============") + + self.prepare_datas() + + tdLog.printNoPrefix("==========step2:test errors ==============") + + self.test_errors() + + tdLog.printNoPrefix("==========step3:support types ============") + + self.support_types() + + tdLog.printNoPrefix("==========step4: sin basic query ============") + + self.basic_sin_function() + + tdLog.printNoPrefix("==========step5: sin filter query ============") + + self.abs_func_filter() + + tdLog.printNoPrefix("==========step6: big number sin query ============") + + self.test_big_number() + + + tdLog.printNoPrefix("==========step7: sin boundary query ============") + + self.check_boundary_values() + + + tdLog.printNoPrefix("==========step8: check sin result of stable query ============") + + self.support_super_table_test() + + + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/query/hint/test_hint.py b/tests/test_new/query/hint/test_hint.py new file mode 100644 index 0000000000..9e958ef810 --- /dev/null +++ b/tests/test_new/query/hint/test_hint.py @@ -0,0 +1,104 @@ +from wsgiref.headers import tspecials +from util.log import * +from util.cases import * +from util.sql import * +import numpy as np + + +class TestHint: + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + + self.rowNum = 10 + self.batchNum = 5 + self.ts = 1537146000000 + + def test_hint(self): + """测试hint查询 + + 执行支持的hint查询,返回结果正确 + + Since: v3.3.0.0 + + Labels: stable + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.run() + + def run(self): + dbname = "db" + tdSql.prepare() + + tdSql.execute(f'''create table sta(ts timestamp, col1 int, col2 bigint) tags(tg1 int, tg2 binary(20))''') + tdSql.execute(f"create table sta1 using sta tags(1, 'a')") + tdSql.execute(f"create table sta2 using sta tags(2, 'b')") + tdSql.execute(f"create table sta3 using sta tags(3, 'c')") + tdSql.execute(f"create table sta4 using sta tags(4, 'a')") + tdSql.execute(f"insert into sta1 values(1537146000001, 11, 110)") + tdSql.execute(f"insert into sta1 values(1537146000002, 12, 120)") + tdSql.execute(f"insert into sta1 values(1537146000003, 13, 130)") + tdSql.execute(f"insert into sta2 values(1537146000001, 21, 210)") + tdSql.execute(f"insert into sta2 values(1537146000002, 22, 220)") + tdSql.execute(f"insert into sta2 values(1537146000003, 23, 230)") + tdSql.execute(f"insert into sta3 values(1537146000001, 31, 310)") + tdSql.execute(f"insert into sta3 values(1537146000002, 32, 320)") + tdSql.execute(f"insert into sta3 values(1537146000003, 33, 330)") + tdSql.execute(f"insert into sta4 values(1537146000001, 41, 410)") + tdSql.execute(f"insert into sta4 values(1537146000002, 42, 420)") + tdSql.execute(f"insert into sta4 values(1537146000003, 43, 430)") + + tdSql.execute(f'''create table stb(ts timestamp, col1 int, col2 bigint) tags(tg1 int, tg2 binary(20))''') + tdSql.execute(f"create table stb1 using stb tags(1, 'a')") + tdSql.execute(f"create table stb2 using stb tags(2, 'b')") + tdSql.execute(f"create table stb3 using stb tags(3, 'c')") + tdSql.execute(f"create table stb4 using stb tags(4, 'a')") + tdSql.execute(f"insert into stb1 values(1537146000001, 911, 9110)") + tdSql.execute(f"insert into stb1 values(1537146000002, 912, 9120)") + tdSql.execute(f"insert into stb1 values(1537146000003, 913, 9130)") + tdSql.execute(f"insert into stb2 values(1537146000001, 921, 9210)") + tdSql.execute(f"insert into stb2 values(1537146000002, 922, 9220)") + tdSql.execute(f"insert into stb2 values(1537146000003, 923, 9230)") + tdSql.execute(f"insert into stb3 values(1537146000001, 931, 9310)") + tdSql.execute(f"insert into stb3 values(1537146000002, 932, 9320)") + tdSql.execute(f"insert into stb3 values(1537146000003, 933, 9330)") + tdSql.execute(f"insert into stb4 values(1537146000001, 941, 9410)") + tdSql.execute(f"insert into stb4 values(1537146000002, 942, 9420)") + tdSql.execute(f"insert into stb4 values(1537146000003, 943, 9430)") + + tdSql.query(f"select /*+ batch_scan() */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+ no_batch_scan() */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+ batch_scan(a) */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+ batch_scan(a,) */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+ a,a */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+*/ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+ batch_scan(),no_batch_scan() */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + tdSql.query(f"select /*+ no_batch_scan() batch_scan() */ count(*) from sta a, stb b where a.tg1=b.tg1 and a.ts=b.ts and b.tg2 > 'a' interval(1a);") + tdSql.checkRows(3) + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/test_new/query/join/test_join.py b/tests/test_new/query/join/test_join.py new file mode 100644 index 0000000000..dfc9156c2d --- /dev/null +++ b/tests/test_new/query/join/test_join.py @@ -0,0 +1,550 @@ +import datetime + +from dataclasses import dataclass, field +from typing import List, Any, Tuple +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * + +PRIMARY_COL = "ts" + +INT_COL = "c_int" +BINT_COL = "c_bint" +SINT_COL = "c_sint" +TINT_COL = "c_tint" +FLOAT_COL = "c_float" +DOUBLE_COL = "c_double" +BOOL_COL = "c_bool" +TINT_UN_COL = "c_utint" +SINT_UN_COL = "c_usint" +BINT_UN_COL = "c_ubint" +INT_UN_COL = "c_uint" +BINARY_COL = "c_binary" +NCHAR_COL = "c_nchar" +TS_COL = "c_ts" + +NUM_COL = [INT_COL, BINT_COL, SINT_COL, TINT_COL, FLOAT_COL, DOUBLE_COL, ] +CHAR_COL = [BINARY_COL, NCHAR_COL, ] +BOOLEAN_COL = [BOOL_COL, ] +TS_TYPE_COL = [TS_COL, ] + +INT_TAG = "t_int" + +ALL_COL = [PRIMARY_COL, INT_COL, BINT_COL, SINT_COL, TINT_COL, FLOAT_COL, DOUBLE_COL, BINARY_COL, NCHAR_COL, BOOL_COL, TS_COL] +TAG_COL = [INT_TAG] +# insert data args: +TIME_STEP = 10000 +NOW = int(datetime.datetime.timestamp(datetime.datetime.now()) * 1000) + +# init db/table +DBNAME = "db" +STBNAME = f"{DBNAME}.stb1" +CTBNAME = f"{DBNAME}.ct1" +NTBNAME = f"{DBNAME}.nt1" + +@dataclass +class DataSet: + ts_data : List[int] = field(default_factory=list) + int_data : List[int] = field(default_factory=list) + bint_data : List[int] = field(default_factory=list) + sint_data : List[int] = field(default_factory=list) + tint_data : List[int] = field(default_factory=list) + int_un_data : List[int] = field(default_factory=list) + bint_un_data: List[int] = field(default_factory=list) + sint_un_data: List[int] = field(default_factory=list) + tint_un_data: List[int] = field(default_factory=list) + float_data : List[float] = field(default_factory=list) + double_data : List[float] = field(default_factory=list) + bool_data : List[int] = field(default_factory=list) + binary_data : List[str] = field(default_factory=list) + nchar_data : List[str] = field(default_factory=list) + + +class TestJoin: + + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), True) + + def __query_condition(self,tbname): + query_condition = [] + for char_col in CHAR_COL: + query_condition.extend( + ( + f"{tbname}.{char_col}", + # f"upper( {tbname}.{char_col} )", + ) + ) + query_condition.extend( f"cast( {tbname}.{un_char_col} as binary(16) ) " for un_char_col in NUM_COL) + for num_col in NUM_COL: + query_condition.extend( + ( + f"sin( {tbname}.{num_col} )", + ) + ) + query_condition.extend( f"{tbname}.{num_col} + {tbname}.{num_col_1} " for num_col_1 in NUM_COL ) + + query_condition.append(''' "test1234!@#$%^&*():'>= 0" + if col in CHAR_COL: + return f" where lower( {tbname}.{col} ) like 'bina%' or lower( {tbname}.{col} ) like '_cha%' " + if col in BOOLEAN_COL: + return f" where {tbname}.{col} in (false, true) " + if col in TS_TYPE_COL or col in PRIMARY_COL: + return f" where cast( {tbname}.{col} as binary(16) ) is not null " + + return "" + + def __group_condition(self, col, having = None): + if isinstance(col, str): + if col.startswith("count"): + col = col[6:-1] + elif col.startswith("max"): + col = col[4:-1] + elif col.startswith("sum"): + col = col[4:-1] + elif col.startswith("min"): + col = col[4:-1] + return f" group by {col} having {having}" if having else f" group by {col} " + + def __gen_sql(self, select_clause, from_clause, where_condition="", group_condition=""): + if isinstance(select_clause, str) and "on" not in from_clause and select_clause.split(".")[0] != from_clause.split(".")[0]: + return + return f"select {select_clause} from {from_clause} {where_condition} {group_condition}" + + @property + def __join_tblist(self, dbname=DBNAME): + return [ + # ["ct1", "ct2"], + [f"{dbname}.ct1", f"{dbname}.ct4"], + [f"{dbname}.ct1", f"{dbname}.nt1"], + # ["ct2", "ct4"], + # ["ct2", "nt1"], + # ["ct4", "nt1"], + # ["ct1", "ct2", "ct4"], + # ["ct1", "ct2", "nt1"], + # ["ct1", "ct4", "nt1"], + # ["ct2", "ct4", "nt1"], + # ["ct1", "ct2", "ct4", "nt1"], + ] + + @property + def __sqls_list(self): + sqls = [] + __join_tblist = self.__join_tblist + for join_tblist in __join_tblist: + alias_tb = "tb1" + # for join_tb in join_tblist: + select_claus_list = self.__query_condition(alias_tb) + for select_claus in select_claus_list: + group_claus = self.__group_condition( col=select_claus) + where_claus = self.__where_condition( query_conditon=select_claus ) + having_claus = self.__group_condition( col=select_claus, having=f"{select_claus} is not null" ) + sqls.extend( + ( + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, alias_tb1=alias_tb), where_claus, group_claus), + self.__gen_sql(select_claus, self.__join_condition(join_tblist, alias_tb1=alias_tb), where_claus, having_claus), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, alias_tb1=alias_tb), where_claus), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, alias_tb1=alias_tb), group_claus), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, alias_tb1=alias_tb), having_claus), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, alias_tb1=alias_tb)), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, INNER=True, alias_tb1=alias_tb), where_claus, group_claus), + self.__gen_sql(select_claus, self.__join_condition(join_tblist, INNER=True, alias_tb1=alias_tb), where_claus, having_claus), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, INNER=True, alias_tb1=alias_tb), where_claus, ), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, INNER=True, alias_tb1=alias_tb), having_claus ), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, INNER=True, alias_tb1=alias_tb), group_claus ), + # self.__gen_sql(select_claus, self.__join_condition(join_tblist, INNER=True, alias_tb1=alias_tb) ), + ) + ) + return list(filter(None, sqls)) + + + def test_join_check(self): + """测试join关键字合法查询 + + join语句select查询,与多种where语句、having语句、group语句组合查询,查询结果返回正确 + + Since: v3.3.0.0 + + Labels: join + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.__join_check() + + def __join_check(self,): + tdLog.printNoPrefix("==========current sql condition check , must return query ok==========") + for i in range(len(self.__sqls_list)): + tdSql.query(self.__sqls_list[i]) + # if i % 10 == 0 : + # tdLog.success(f"{i} sql is already executed success !") + + def __join_check_old(self, tblist, checkrows, join_flag=True): + query_conditions = self.__query_condition(tblist[0]) + join_condition = self.__join_condition(tb_list=tblist) if join_flag else " " + for condition in query_conditions: + where_condition = self.__where_condition(col=condition, tbname=tblist[0]) + group_having = self.__group_condition(col=condition, having=f"{condition} is not null " ) + group_no_having= self.__group_condition(col=condition ) + groups = ["", group_having, group_no_having] + for group_condition in groups: + if where_condition: + sql = f" select {condition} from {tblist[0]},{tblist[1]} where {join_condition} and {where_condition} {group_condition} " + else: + sql = f" select {condition} from {tblist[0]},{tblist[1]} where {join_condition} {group_condition} " + + if not join_flag : + tdSql.error(sql=sql) + break + if len(tblist) == 2: + if "ct1" in tblist or "nt1" in tblist: + self.__join_current(sql, checkrows) + elif where_condition or "not null" in group_condition: + self.__join_current(sql, checkrows + 2 ) + elif group_condition: + self.__join_current(sql, checkrows + 3 ) + else: + self.__join_current(sql, checkrows + 5 ) + if len(tblist) > 2 or len(tblist) < 1: + tdSql.error(sql=sql) + + def __join_current(self, sql, checkrows): + tdSql.query(sql=sql) + # tdSql.checkRows(checkrows) + + def test_error(self): + """测试join关键字非法查询 + + join语句select查询,与不支持的where语句、having语句、group语句组合查询,查询结果返回期望错误 + + Since: v3.3.0.0 + + Labels: join, negative + + Jira: TD-12345,TS-1234 + + History: + - 2024-2-6 Feng Chao Created + + """ + self.__test_error() + + def __test_error(self, dbname=DBNAME): + # sourcery skip: extract-duplicate-method, move-assign-in-block + tdLog.printNoPrefix("==========err sql condition check , must return error==========") + err_list_1 = [f"{dbname}.ct1", f"{dbname}.ct2", f"{dbname}.ct4"] + err_list_2 = [f"{dbname}.ct1", f"{dbname}.ct2", f"{dbname}.nt1"] + err_list_3 = [f"{dbname}.ct1", f"{dbname}.ct4", f"{dbname}.nt1"] + err_list_4 = [f"{dbname}.ct2", f"{dbname}.ct4", f"{dbname}.nt1"] + err_list_5 = [f"{dbname}.ct1", f"{dbname}.ct2", f"{dbname}.ct4", f"{dbname}.nt1"] + self.__join_check_old(err_list_1, -1) + tdLog.printNoPrefix(f"==========err sql condition check in {err_list_1} over==========") + self.__join_check_old(err_list_2, -1) + tdLog.printNoPrefix(f"==========err sql condition check in {err_list_2} over==========") + self.__join_check_old(err_list_3, -1) + tdLog.printNoPrefix(f"==========err sql condition check in {err_list_3} over==========") + self.__join_check_old(err_list_4, -1) + tdLog.printNoPrefix(f"==========err sql condition check in {err_list_4} over==========") + self.__join_check_old(err_list_5, -1) + tdLog.printNoPrefix(f"==========err sql condition check in {err_list_5} over==========") + self.__join_check_old(["ct2", "ct4"], -1, join_flag=False) + tdLog.printNoPrefix("==========err sql condition check in has no join condition over==========") + + tdSql.error( f"select c1, c2 from {dbname}.ct2, {dbname}.ct4 where ct2.{PRIMARY_COL}=ct4.{PRIMARY_COL}" ) + tdSql.error( f"select ct2.c1, ct2.c2 from {dbname}.ct2 as ct2, {dbname}.ct4 as ct4 where ct2.{INT_COL}=ct4.{INT_COL}" ) + tdSql.error( f"select ct2.c1, ct2.c2 from {dbname}.ct2 as ct2, {dbname}.ct4 as ct4 where ct2.{TS_COL}=ct4.{TS_COL}" ) + tdSql.error( f"select ct2.c1, ct2.c2 from {dbname}.ct2 as ct2, {dbname}.ct4 as ct4 where ct2.{PRIMARY_COL}=ct4.{TS_COL}" ) + tdSql.error( f"select ct2.c1, ct1.c2 from {dbname}.ct2 as ct2, {dbname}.ct4 as ct4 where ct2.{PRIMARY_COL}=ct4.{PRIMARY_COL}" ) + tdSql.error( f"select ct2.c1, ct4.c2 from {dbname}.ct2 as ct2, {dbname}.ct4 as ct4 where ct2.{PRIMARY_COL}=ct4.{PRIMARY_COL} and c1 is not null " ) + tdSql.error( f"select ct2.c1, ct4.c2 from {dbname}.ct2 as ct2, {dbname}.ct4 as ct4 where ct2.{PRIMARY_COL}=ct4.{PRIMARY_COL} and ct1.c1 is not null " ) + + + tbname = [f"{dbname}.ct1", f"{dbname}.ct2", f"{dbname}.ct4", f"{dbname}.nt1"] + + # for tb in tbname: + # for errsql in self.__join_err_check(tb): + # tdSql.error(sql=errsql) + # tdLog.printNoPrefix(f"==========err sql condition check in {tb} over==========") + + + def all_test(self): + self.__join_check() + self.__test_error() + + + def __create_tb(self, stb="stb1", ctb_num=20, ntbnum=1, dbname=DBNAME): + create_stb_sql = f'''create table {dbname}.{stb}( + ts timestamp, {INT_COL} int, {BINT_COL} bigint, {SINT_COL} smallint, {TINT_COL} tinyint, + {FLOAT_COL} float, {DOUBLE_COL} double, {BOOL_COL} bool, + {BINARY_COL} binary(16), {NCHAR_COL} nchar(32), {TS_COL} timestamp, + {TINT_UN_COL} tinyint unsigned, {SINT_UN_COL} smallint unsigned, + {INT_UN_COL} int unsigned, {BINT_UN_COL} bigint unsigned + ) tags ({INT_TAG} int) + ''' + for i in range(ntbnum): + + create_ntb_sql = f'''create table {dbname}.nt{i+1}( + ts timestamp, {INT_COL} int, {BINT_COL} bigint, {SINT_COL} smallint, {TINT_COL} tinyint, + {FLOAT_COL} float, {DOUBLE_COL} double, {BOOL_COL} bool, + {BINARY_COL} binary(16), {NCHAR_COL} nchar(32), {TS_COL} timestamp, + {TINT_UN_COL} tinyint unsigned, {SINT_UN_COL} smallint unsigned, + {INT_UN_COL} int unsigned, {BINT_UN_COL} bigint unsigned + ) + ''' + tdSql.execute(create_stb_sql) + tdSql.execute(create_ntb_sql) + + for i in range(ctb_num): + tdSql.execute(f'create table {dbname}.ct{i+1} using {dbname}.{stb} tags ( {i+1} )') + + def __data_set(self, rows): + data_set = DataSet() + + for i in range(rows): + data_set.ts_data.append(NOW + 1 * (rows - i)) + data_set.int_data.append(rows - i) + data_set.bint_data.append(11111 * (rows - i)) + data_set.sint_data.append(111 * (rows - i) % 32767) + data_set.tint_data.append(11 * (rows - i) % 127) + data_set.int_un_data.append(rows - i) + data_set.bint_un_data.append(11111 * (rows - i)) + data_set.sint_un_data.append(111 * (rows - i) % 32767) + data_set.tint_un_data.append(11 * (rows - i) % 127) + data_set.float_data.append(1.11 * (rows - i)) + data_set.double_data.append(1100.0011 * (rows - i)) + data_set.bool_data.append((rows - i) % 2) + data_set.binary_data.append(f'binary{(rows - i)}') + data_set.nchar_data.append(f'nchar_测试_{(rows - i)}') + + return data_set + + def __insert_data(self, dbname=DBNAME): + tdLog.printNoPrefix("==========step: start inser data into tables now.....") + data = self.__data_set(rows=self.rows) + + # now_time = int(datetime.datetime.timestamp(datetime.datetime.now()) * 1000) + null_data = '''null, null, null, null, null, null, null, null, null, null, null, null, null, null''' + zero_data = "0, 0, 0, 0, 0, 0, 0, 'binary_0', 'nchar_0', 0, 0, 0, 0, 0" + + for i in range(self.rows): + row_data = f''' + {data.int_data[i]}, {data.bint_data[i]}, {data.sint_data[i]}, {data.tint_data[i]}, {data.float_data[i]}, {data.double_data[i]}, + {data.bool_data[i]}, '{data.binary_data[i]}', '{data.nchar_data[i]}', {data.ts_data[i]}, {data.tint_un_data[i]}, + {data.sint_un_data[i]}, {data.int_un_data[i]}, {data.bint_un_data[i]} + ''' + neg_row_data = f''' + {-1 * data.int_data[i]}, {-1 * data.bint_data[i]}, {-1 * data.sint_data[i]}, {-1 * data.tint_data[i]}, {-1 * data.float_data[i]}, {-1 * data.double_data[i]}, + {data.bool_data[i]}, '{data.binary_data[i]}', '{data.nchar_data[i]}', {data.ts_data[i]}, {1 * data.tint_un_data[i]}, + {1 * data.sint_un_data[i]}, {1 * data.int_un_data[i]}, {1 * data.bint_un_data[i]} + ''' + + tdSql.execute( f"insert into {dbname}.ct1 values ( {NOW - i * TIME_STEP}, {row_data} )" ) + tdSql.execute( f"insert into {dbname}.ct2 values ( {NOW - i * int(TIME_STEP * 0.6)}, {neg_row_data} )" ) + tdSql.execute( f"insert into {dbname}.ct4 values ( {NOW - i * int(TIME_STEP * 0.8) }, {row_data} )" ) + tdSql.execute( f"insert into {dbname}.nt1 values ( {NOW - i * int(TIME_STEP * 1.2)}, {row_data} )" ) + + tdSql.execute( f"insert into {dbname}.ct2 values ( {NOW + int(TIME_STEP * 0.6)}, {null_data} )" ) + tdSql.execute( f"insert into {dbname}.ct2 values ( {NOW - (self.rows + 1) * int(TIME_STEP * 0.6)}, {null_data} )" ) + tdSql.execute( f"insert into {dbname}.ct2 values ( {NOW - self.rows * int(TIME_STEP * 0.29) }, {null_data} )" ) + + tdSql.execute( f"insert into {dbname}.ct4 values ( {NOW + int(TIME_STEP * 0.8)}, {null_data} )" ) + tdSql.execute( f"insert into {dbname}.ct4 values ( {NOW - (self.rows + 1) * int(TIME_STEP * 0.8)}, {null_data} )" ) + tdSql.execute( f"insert into {dbname}.ct4 values ( {NOW - self.rows * int(TIME_STEP * 0.39)}, {null_data} )" ) + + tdSql.execute( f"insert into {dbname}.nt1 values ( {NOW + int(TIME_STEP * 1.2)}, {null_data} )" ) + tdSql.execute( f"insert into {dbname}.nt1 values ( {NOW - (self.rows + 1) * int(TIME_STEP * 1.2)}, {null_data} )" ) + tdSql.execute( f"insert into {dbname}.nt1 values ( {NOW - self.rows * int(TIME_STEP * 0.59)}, {null_data} )" ) + + def ts5863(self, dbname=DBNAME): + tdSql.execute(f"CREATE STABLE {dbname}.`st_quality` (`ts` TIMESTAMP, `quality` INT, `val` NCHAR(64), `rts` TIMESTAMP) \ + TAGS (`cx` VARCHAR(10), `gyd` VARCHAR(10), `gx` VARCHAR(10), `lx` VARCHAR(10)) SMA(`ts`,`quality`,`val`)") + + tdSql.execute(f"create table {dbname}.st_q1 using {dbname}.st_quality tags ('cx', 'gyd', 'gx1', 'lx1')") + + sql1 = f"select t.val as batch_no, a.tbname as sample_point_code, min(cast(a.val as double)) as `min`, \ + max(cast(a.val as double)) as `max`, avg(cast(a.val as double)) as `avg` from {dbname}.st_quality t \ + left join {dbname}.st_quality a on a.ts=t.ts and a.cx=t.cx and a.gyd=t.gyd \ + where t.ts >= 1734574900000 and t.ts <= 1734575000000 \ + and t.tbname = 'st_q1' \ + and a.tbname in ('st_q2', 'st_q3') \ + group by t.val, a.tbname" + tdSql.query(sql1) + tdSql.checkRows(0) + + tdSql.execute(f"create table {dbname}.st_q2 using {dbname}.st_quality tags ('cx2', 'gyd2', 'gx2', 'lx2')") + tdSql.execute(f"create table {dbname}.st_q3 using {dbname}.st_quality tags ('cx', 'gyd', 'gx3', 'lx3')") + tdSql.execute(f"create table {dbname}.st_q4 using {dbname}.st_quality tags ('cx', 'gyd', 'gx4', 'lx4')") + + tdSql.query(sql1) + tdSql.checkRows(0) + + tdSql.execute(f"insert into {dbname}.st_q1 values (1734574900000, 1, '1', 1734574900000)") + tdSql.query(sql1) + tdSql.checkRows(0) + tdSql.execute(f"insert into {dbname}.st_q2 values (1734574900000, 1, '1', 1734574900000)") + tdSql.query(sql1) + tdSql.checkRows(0) + tdSql.execute(f"insert into {dbname}.st_q3 values (1734574900000, 1, '1', 1734574900000)") + tdSql.query(sql1) + tdSql.checkRows(1) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'st_q3') + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + + tdSql.execute(f"insert into {dbname}.st_q1 values (1734574900001, 2, '2', 1734574900000)") + tdSql.execute(f"insert into {dbname}.st_q3 values (1734574900001, 2, '2', 1734574900000)") + sql2 = f"select t.val as batch_no, a.tbname as sample_point_code, min(cast(a.val as double)) as `min`, \ + max(cast(a.val as double)) as `max`, avg(cast(a.val as double)) as `avg` from {dbname}.st_quality t \ + left join {dbname}.st_quality a on a.ts=t.ts and a.cx=t.cx and a.gyd=t.gyd \ + where t.ts >= 1734574900000 and t.ts <= 1734575000000 \ + and t.tbname = 'st_q1' \ + and a.tbname in ('st_q2', 'st_q3') \ + group by t.val, a.tbname order by batch_no" + tdSql.query(sql2) + tdSql.checkRows(2) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'st_q3') + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 2) + tdSql.checkData(1, 1, 'st_q3') + tdSql.checkData(1, 2, 2) + tdSql.checkData(1, 3, 2) + tdSql.checkData(1, 4, 2) + sql3 = f"select min(cast(a.val as double)) as `min` from {dbname}.st_quality t left join {dbname}.st_quality \ + a on a.ts=t.ts and a.cx=t.cx where t.tbname = 'st_q3' and a.tbname in ('st_q3', 'st_q2')" + tdSql.execute(f"insert into {dbname}.st_q1 values (1734574900002, 2, '2', 1734574900000)") + tdSql.execute(f"insert into {dbname}.st_q4 values (1734574900002, 2, '2', 1734574900000)") + tdSql.execute(f"insert into {dbname}.st_q1 values (1734574900003, 3, '3', 1734574900000)") + tdSql.execute(f"insert into {dbname}.st_q3 values (1734574900003, 3, '3', 1734574900000)") + tdSql.query(sql3) + tdSql.checkRows(1) + tdSql.checkData(0, 0, 1) + sql3 = f"select min(cast(a.val as double)) as `min`, max(cast(a.val as double)) as `max`, avg(cast(a.val as double)) as `avg` \ + from {dbname}.st_quality t left join {dbname}.st_quality a \ + on a.ts=t.ts and a.cx=t.cx where t.tbname = 'st_q3' and a.tbname in ('st_q3', 'st_q2')" + tdSql.query(sql3) + tdSql.checkRows(1) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 3) + tdSql.checkData(0, 2, 2) + tdSql.query(sql1) + tdSql.checkRows(3) + tdSql.query(sql2) + tdSql.checkRows(3) + tdSql.checkData(0, 0, 1) + tdSql.checkData(0, 1, 'st_q3') + tdSql.checkData(0, 2, 1) + tdSql.checkData(0, 3, 1) + tdSql.checkData(0, 4, 1) + tdSql.checkData(1, 0, 2) + tdSql.checkData(1, 1, 'st_q3') + tdSql.checkData(1, 2, 2) + tdSql.checkData(1, 3, 2) + tdSql.checkData(1, 4, 2) + tdSql.checkData(2, 0, 3) + tdSql.checkData(2, 1, 'st_q3') + tdSql.checkData(2, 2, 3) + tdSql.checkData(2, 3, 3) + tdSql.checkData(2, 4, 3) + + def run(self): + tdSql.prepare() + + tdLog.printNoPrefix("==========step1:create table") + self.__create_tb(dbname=DBNAME) + + tdLog.printNoPrefix("==========step2:insert data") + self.rows = 10 + self.__insert_data(dbname=DBNAME) + + tdLog.printNoPrefix("==========step3:all check") + tdSql.query(f"select count(*) from {DBNAME}.ct1") + tdSql.checkData(0, 0, self.rows) + self.all_test() + + tdLog.printNoPrefix("==========step4:cross db check") + dbname1 = "db1" + tdSql.execute(f"create database {dbname1} duration 172800m") + tdSql.execute(f"use {dbname1}") + self.__create_tb(dbname=dbname1) + self.__insert_data(dbname=dbname1) + + tdSql.query("select ct1.c_int from db.ct1 as ct1 join db1.ct1 as cy1 on ct1.ts=cy1.ts") + tdSql.checkRows(self.rows) + tdSql.query("select ct1.c_int from db.stb1 as ct1 join db1.ct1 as cy1 on ct1.ts=cy1.ts") + tdSql.checkRows(self.rows + int(self.rows * 0.6 //3)+ int(self.rows * 0.8 // 4)) + tdSql.query("select ct1.c_int from db.nt1 as ct1 join db1.nt1 as cy1 on ct1.ts=cy1.ts") + tdSql.checkRows(self.rows + 3) + tdSql.query("select ct1.c_int from db.stb1 as ct1 join db1.stb1 as cy1 on ct1.ts=cy1.ts") + tdSql.checkRows(50) + + tdSql.query("select count(*) from db.ct1") + tdSql.checkData(0, 0, self.rows) + tdSql.query("select count(*) from db1.ct1") + tdSql.checkData(0, 0, self.rows) + + self.all_test() + tdSql.query("select count(*) from db.ct1") + tdSql.checkData(0, 0, self.rows) + tdSql.query("select count(*) from db1.ct1") + tdSql.checkData(0, 0, self.rows) + + tdSql.execute(f"flush database {DBNAME}") + tdSql.execute(f"flush database {dbname1}") + # tdDnodes.stop(1) + # tdDnodes.start(1) + + tdSql.execute("use db") + tdSql.query("select count(*) from db.ct1") + tdSql.checkData(0, 0, self.rows) + tdSql.query("select count(*) from db1.ct1") + tdSql.checkData(0, 0, self.rows) + + tdLog.printNoPrefix("==========step4:after wal, all check again ") + self.all_test() + tdSql.query("select count(*) from db.ct1") + tdSql.checkData(0, 0, self.rows) + self.ts5863(dbname=dbname1) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/query/nested/test_nested_query.py b/tests/test_new/query/nested/test_nested_query.py new file mode 100755 index 0000000000..251a12cad6 --- /dev/null +++ b/tests/test_new/query/nested/test_nested_query.py @@ -0,0 +1,6202 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import random +import os +import time +import taos +import subprocess +from faker import Faker +from util.log import tdLog +from util.cases import tdCases +from util.sql import tdSql +from util.dnodes import tdDnodes +from util.dnodes import * + +class TestNestedQuery: + updatecfgDict = {'maxSQLLength':1048576,'debugFlag': 131 ,"cDebugFlag":131,"uDebugFlag":131 ,"rpcDebugFlag":131 , "tmrDebugFlag":131 , + "jniDebugFlag":131 ,"simDebugFlag":131,"dDebugFlag":131, "dDebugFlag":131,"vDebugFlag":131,"mDebugFlag":131,"qDebugFlag":131, + "wDebugFlag":131,"sDebugFlag":131,"tsdbDebugFlag":131,"tqDebugFlag":131 ,"fsDebugFlag":131 ,"fnDebugFlag":131} + + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor(), logSql) + + self.testcasePath = os.path.split(__file__)[0] + self.testcaseFilename = os.path.split(__file__)[-1] + os.system("rm -rf %s/%s.sql" % (self.testcasePath,self.testcaseFilename)) + + self.num = 10 + self.fornum = 15 + + self.db_nest = "nest" + self.dropandcreateDB_random("%s" %self.db_nest, 1) + + # regular column select + #q_select= ['ts' , '*' , 'q_int', 'q_bigint' , 'q_bigint' , 'q_smallint' , 'q_tinyint' , 'q_bool' , 'q_binary' , 'q_nchar' ,'q_float' , 'q_double' ,'q_ts '] + self.q_select= ['q_int', 'q_bigint' , 'q_bigint' , 'q_smallint' , 'q_tinyint' , 'q_bool' , 'q_binary' , 'q_nchar' ,'q_float' , 'q_double' ,'q_ts ', 'q_int_null ', 'q_bigint_null ' , 'q_bigint_null ' , 'q_smallint_null ' , 'q_tinyint_null ' , 'q_bool_null ' , 'q_binary_null ' , 'q_nchar_null ' ,'q_float_null ' , 'q_double_null ' ,'q_ts_null '] + + # tag column select + #t_select= ['*' , 'loc' ,'t_int', 't_bigint' , 't_bigint' , 't_smallint' , 't_tinyint' , 't_bool' , 't_binary' , 't_nchar' ,'t_float' , 't_double' ,'t_ts '] + self.t_select= ['loc','t_int', 't_bigint' , 't_bigint' , 't_smallint' , 't_tinyint' , 't_bool' , 't_binary' , 't_nchar' ,'t_float' , 't_double' ,'t_ts '] + + # regular and tag column select + self.qt_select= self.q_select + self.t_select + + # distinct regular column select + self.dq_select= ['distinct q_int', 'distinct q_bigint' , 'distinct q_smallint' , 'distinct q_tinyint' , + 'distinct q_bool' , 'distinct q_binary' , 'distinct q_nchar' ,'distinct q_float' , 'distinct q_double' ,'distinct q_ts '] + + # distinct tag column select + self.dt_select= ['distinct loc', 'distinct t_int', 'distinct t_bigint' , 'distinct t_smallint' , 'distinct t_tinyint' , + 'distinct t_bool' , 'distinct t_binary' , 'distinct t_nchar' ,'distinct t_float' , 'distinct t_double' ,'distinct t_ts '] + + # distinct regular and tag column select + self.dqt_select= self.dq_select + self.dt_select + + # special column select + self.s_r_select= ['_c0', '_rowts' , '_C0' ] + self.s_s_select= ['tbname' , '_rowts' , '_c0', '_C0' ] + self.unionall_or_union= [ ' union ' , ' union all ' ] + + # regular column where + self.q_where = ['ts < now +1s','q_bigint >= -9223372036854775807 and q_bigint <= 9223372036854775807', 'q_int <= 2147483647 and q_int >= -2147483647', + 'q_smallint >= -32767 and q_smallint <= 32767','q_tinyint >= -127 and q_tinyint <= 127','q_float >= -1.7E308 and q_float <= 1.7E308', + 'q_double >= -1.7E308 and q_double <= 1.7E308', 'q_binary like \'binary%\' or q_binary = \'0\' ' , 'q_nchar like \'nchar%\' or q_nchar = \'0\' ' , + 'q_bool = true or q_bool = false' , 'q_bool in (0 , 1)' , 'q_bool in ( true , false)' , 'q_bool = 0 or q_bool = 1', + 'q_bigint between -9223372036854775807 and 9223372036854775807',' q_int between -2147483647 and 2147483647','q_smallint between -32767 and 32767', + 'q_bigint not between 9223372036854775807 and -9223372036854775807','q_int not between 2147483647 and -2147483647','q_smallint not between 32767 and -32767', + 'q_tinyint between -127 and 127 ','q_float >= -3.4E38 ','q_float <= 3.4E38 ','q_double >= -1.7E308 ', + 'q_double <= 1.7E308 ','q_float between -3.4E38 and 3.4E38 ','q_double between -1.7E308 and 1.7E308 ' ,'q_float not between 3.4E38 and -3.4E38 ','q_double not between 1.7E308 and -1.7E308 ', + 'q_float is not null ' ,'q_double is not null ' ,'q_binary match \'binary\' ','q_binary nmatch \'binarynchar\' ','q_nchar match \'nchar\' ','q_nchar nmatch \'binarynchar\' ', + 'q_binary like \'binary%\' ','(q_binary like \'binary%\' or q_nchar = \'0\' or q_binary = \'binary_\' ) ','q_nchar like \'nchar%\' ','(q_nchar like \'nchar%\' or q_binary = \'0\' or q_nchar = \'nchar_\' ) ',] + #TD-6201 ,'q_bool between 0 and 1' + + # regular column where for test union,join + self.q_u_where = ['t1.ts < now +1s' , 't2.ts < now +1s','t1.q_bigint >= -9223372036854775807 and t1.q_bigint <= 9223372036854775807 and t2.q_bigint >= -9223372036854775807 and t2.q_bigint <= 9223372036854775807', + 't1.q_int <= 2147483647 and t1.q_int >= -2147483647 and t2.q_int <= 2147483647 and t2.q_int >= -2147483647', + 't1.q_smallint >= -32767 and t1.q_smallint <= 32767 and t2.q_smallint >= -32767 and t2.q_smallint <= 32767', + 't1.q_tinyint >= -127 and t1.q_tinyint <= 127 and t2.q_tinyint >= -127 and t2.q_tinyint <= 127', + 't1.q_float >= - 1.7E308 and t1.q_float <= 1.7E308 and t2.q_float >= - 1.7E308 and t2.q_float <= 1.7E308', + 't1.q_double >= - 1.7E308 and t1.q_double <= 1.7E308 and t2.q_double >= - 1.7E308 and t2.q_double <= 1.7E308', + 't1.q_binary like \'binary%\' and t2.q_binary like \'binary%\' ' , + 't1.q_nchar like \'nchar%\' and t2.q_nchar like \'nchar%\' ' , + 't1.q_bool in (0 , 1) and t2.q_bool in (0 , 1)' , 't1.q_bool in ( true , false) and t2.q_bool in ( true , false)' , + 't1.q_bigint between -9223372036854775807 and 9223372036854775807 and t2.q_bigint between -9223372036854775807 and 9223372036854775807', + 't1.q_int between -2147483647 and 2147483647 and t2.q_int between -2147483647 and 2147483647', + 't1.q_smallint between -32767 and 32767 and t2.q_smallint between -32767 and 32767', + 't1.q_tinyint between -127 and 127 and t2.q_tinyint between -127 and 127 ','t1.q_float between -1.7E308 and 1.7E308 and t2.q_float between -1.7E308 and 1.7E308', + 't1.q_double between -1.7E308 and 1.7E308 and t2.q_double between -1.7E308 and 1.7E308', + 't1.q_bigint not between 9223372036854775807 and -9223372036854775807 and t2.q_bigint not between 9223372036854775807 and -9223372036854775807', + 't1.q_int not between 2147483647 and -2147483647 and t2.q_int not between 2147483647 and -2147483647', + 't1.q_smallint not between 32767 and -32767 and t2.q_smallint not between 32767 and -32767', + 't1.q_tinyint not between 127 and -127 and t2.q_tinyint not between 127 and -127 ','t1.q_float not between -1.7E308 and -1.7E308 and t2.q_float not between 1.7E308 and -1.7E308', + 't1.q_double not between 1.7E308 and -1.7E308 and t2.q_double not between 1.7E308 and -1.7E308'] + #TD-6201 ,'t1.q_bool between 0 and 1 or t2.q_bool between 0 and 1'] + #'t1.q_bool = true and t1.q_bool = false and t2.q_bool = true and t2.q_bool = false' , 't1.q_bool = 0 and t1.q_bool = 1 and t2.q_bool = 0 and t2.q_bool = 1' , + + self.q_u_or_where = ['(t1.q_binary like \'binary%\' or t1.q_binary = \'0\' or t2.q_binary like \'binary%\' or t2.q_binary = \'0\' )' , + '(t1.q_nchar like \'nchar%\' or t1.q_nchar = \'0\' or t2.q_nchar like \'nchar%\' or t2.q_nchar = \'0\' )' , '(t1.q_bool = true or t1.q_bool = false or t2.q_bool = true or t2.q_bool = false)' , + '(t1.q_bool in (0 , 1) or t2.q_bool in (0 , 1))' , '(t1.q_bool in ( true , false) or t2.q_bool in ( true , false))' , '(t1.q_bool = 0 or t1.q_bool = 1 or t2.q_bool = 0 or t2.q_bool = 1)' , + '(t1.q_bigint between -9223372036854775807 and 9223372036854775807 or t2.q_bigint between -9223372036854775807 and 9223372036854775807)', + '(t1.q_int between -2147483647 and 2147483647 or t2.q_int between -2147483647 and 2147483647)', + '(t1.q_smallint between -32767 and 32767 or t2.q_smallint between -32767 and 32767)', + '(t1.q_tinyint between -127 and 127 or t2.q_tinyint between -127 and 127 )','(t1.q_float between -1.7E308 and 1.7E308 or t2.q_float between -1.7E308 and 1.7E308)', + '(t1.q_double between -1.7E308 and 1.7E308 or t2.q_double between -1.7E308 and 1.7E308)'] + + # tag column where + self.t_where = ['ts < now +1s','t_bigint >= -9223372036854775807 and t_bigint <= 9223372036854775807','t_int <= 2147483647 and t_int >= -2147483647', + 't_smallint >= -32767 and t_smallint <= 32767','q_tinyint >= -127 and t_tinyint <= 127','t_float >= -1.7E308 and t_float <= 1.7E308', + 't_double >= -1.7E308 and t_double <= 1.7E308', 't_binary like \'binary%\' or t_binary = \'0\' ' , 't_nchar like \'nchar%\' or t_nchar = \'0\'' , + 't_bool = true or t_bool = false' , 't_bool in (0 , 1)' , 't_bool in ( true , false)' , 't_bool = 0 or t_bool = 1', + 't_bigint between -9223372036854775807 and 9223372036854775807',' t_int between -2147483647 and 2147483647','t_smallint between -32767 and 32767', + 't_tinyint between -127 and 127 ','t_float between -1.7E308 and 1.7E308','t_double between -1.7E308 and 1.7E308', + 't_binary match \'binary\' ','t_binary nmatch \'binarynchar\' ','t_nchar match \'nchar\' ','t_nchar nmatch \'binarynchar\' ', + 't_binary like \'binary%\' ','t_nchar like \'nchar%\' ','(t_binary like \'binary%\' or t_nchar = \'0\' ) ','(t_nchar like \'nchar%\' or t_binary = \'0\' ) ',] + #TD-6201,'t_bool between 0 and 1' + + # tag column where for test union,join | this is not support + self.t_u_where = ['t1.ts < now +1s' , 't2.ts < now +1s','t1.t_bigint >= -9223372036854775807 and t1.t_bigint <= 9223372036854775807 and t2.t_bigint >= -9223372036854775807 and t2.t_bigint <= 9223372036854775807', + 't1.t_int <= 2147483647 and t1.t_int >= -2147483647 and t2.t_int <= 2147483647 and t2.t_int >= -2147483647', + 't1.t_smallint >= -32767 and t1.t_smallint <= 32767 and t2.t_smallint >= -32767 and t2.t_smallint <= 32767', + 't1.t_tinyint >= -127 and t1.t_tinyint <= 127 and t2.t_tinyint >= -127 and t2.t_tinyint <= 127', + 't1.t_float >= -1.7E308 and t1.t_float <= 1.7E308 and t2.t_float >= -1.7E308 and t2.t_float <= 1.7E308', + 't1.t_double >= -1.7E308 and t1.t_double <= 1.7E308 and t2.t_double >= -1.7E308 and t2.t_double <= 1.7E308', + '(t1.t_binary like \'binary%\' or t1.t_binary = \'0\' or t2.t_binary like \'binary%\' or t2.t_binary = \'0\') ' , + '(t1.t_nchar like \'nchar%\' or t1.t_nchar = \'0\' or t2.t_nchar like \'nchar%\' or t2.t_nchar = \'0\' )' , '(t1.t_bool = true or t1.t_bool = false or t2.t_bool = true or t2.t_bool = false)' , + 't1.t_bool in (0 , 1) and t2.t_bool in (0 , 1)' , 't1.t_bool in ( true , false) and t2.t_bool in ( true , false)' , '(t1.t_bool = 0 or t1.t_bool = 1 or t2.t_bool = 0 or t2.t_bool = 1)', + 't1.t_bigint between -9223372036854775807 and 9223372036854775807 and t2.t_bigint between -9223372036854775807 and 9223372036854775807', + 't1.t_int between -2147483647 and 2147483647 and t2.t_int between -2147483647 and 2147483647', + 't1.t_smallint between -32767 and 32767 and t2.t_smallint between -32767 and 32767', + '(t1.t_tinyint between -127 and 127 and t2.t_tinyint between -127 and 127) ','t1.t_float between -1.7E308 and 1.7E308 and t2.t_float between -1.7E308 and 1.7E308', + '(t1.t_double between -1.7E308 and 1.7E308 and t2.t_double between -1.7E308 and 1.7E308)'] + #TD-6201,'t1.t_bool between 0 and 1 or t2.q_bool between 0 and 1'] + + self.t_u_or_where = ['(t1.t_binary like \'binary%\' or t1.t_binary = \'0\' or t2.t_binary like \'binary%\' or t2.t_binary = \'0\' )' , + '(t1.t_nchar like \'nchar%\' or t1.t_nchar = \'0\' or t2.t_nchar like \'nchar%\' or t2.t_nchar = \'0\' )' , '(t1.t_bool = true or t1.t_bool = false or t2.t_bool = true or t2.t_bool = false)' , + '(t1.t_bool in (0 , 1) or t2.t_bool in (0 , 1))' , '(t1.t_bool in ( true , false) or t2.t_bool in ( true , false))' , '(t1.t_bool = 0 or t1.t_bool = 1 or t2.t_bool = 0 or t2.t_bool = 1)', + '(t1.t_bigint between -9223372036854775807 and 9223372036854775807 or t2.t_bigint between -9223372036854775807 and 9223372036854775807)', + '(t1.t_int between -2147483647 and 2147483647 or t2.t_int between -2147483647 and 2147483647)', + '(t1.t_smallint between -32767 and 32767 or t2.t_smallint between -32767 and 32767)', + '(t1.t_tinyint between -127 and 127 or t2.t_tinyint between -127 and 127 )','(t1.t_float between -1.7E308 and 1.7E308 or t2.t_float between -1.7E308 and 1.7E308)', + '(t1.t_double between -1.7E308 and 1.7E308 or t2.t_double between -1.7E308 and 1.7E308)'] + + # self.t_u_where = ['t1.ts < now +1s'] # 超级表tag不支持,暂时注掉 + # self.t_u_or_where = ['(t1.q_bool in (0 , 1))'] #超级表tag不支持,暂时注掉 + + # regular and tag column where + self.qt_where = self.q_where + self.t_where + #self.qt_where = self.q_where #超级表tag不支持,暂时注掉 + self.qt_u_where = self.q_u_where + self.t_u_where + # now,qt_u_or_where is not support + self.qt_u_or_where = self.q_u_or_where + self.t_u_or_where + + # tag column where for test super join | this is support , 't1.t_bool = t2.t_bool ' ??? + self.t_join_where = ['t1.t_bigint = t2.t_bigint ', 't1.t_int = t2.t_int ', 't1.t_smallint = t2.t_smallint ', 't1.t_tinyint = t2.t_tinyint ', + 't1.t_float = t2.t_float ', 't1.t_double = t2.t_double ', 't1.t_binary = t2.t_binary ' , 't1.t_nchar = t2.t_nchar ' ] + #self.t_join_where = ['t1.ts = t2.ts'] # 超级表tag不支持,暂时注掉 + + # session && fill + self.session_where = ['session(ts,10a)' , 'session(ts,10s)', 'session(ts,10m)' , 'session(ts,10h)','session(ts,10d)' , 'session(ts,10w)'] + self.session_u_where = ['session(t1.ts,10a)' , 'session(t1.ts,10s)', 'session(t1.ts,10m)' , 'session(t1.ts,10h)','session(t1.ts,10d)' , 'session(t1.ts,10w)', + 'session(t2.ts,10a)' , 'session(t2.ts,10s)', 'session(t2.ts,10m)' , 'session(t2.ts,10h)','session(t2.ts,10d)' , 'session(t2.ts,10w)'] + + self.fill_where = ['FILL(NONE)','FILL(PREV)','FILL(NULL)','FILL(LINEAR)','FILL(NEXT)','FILL(VALUE, 1.23)'] + + self.state_window = ['STATE_WINDOW(q_tinyint)','STATE_WINDOW(q_bigint)','STATE_WINDOW(q_int)','STATE_WINDOW(q_bool)','STATE_WINDOW(q_smallint)'] + self.state_u_window = ['STATE_WINDOW(t1.q_tinyint)','STATE_WINDOW(t1.q_bigint)','STATE_WINDOW(t1.q_int)','STATE_WINDOW(t1.q_bool)','STATE_WINDOW(t1.q_smallint)', + 'STATE_WINDOW(t2.q_tinyint)','STATE_WINDOW(t2.q_bigint)','STATE_WINDOW(t2.q_int)','STATE_WINDOW(t2.q_bool)','STATE_WINDOW(t2.q_smallint)'] + + # order by where + self.order_where = ['order by ts' , 'order by ts asc'] + self.order_u_where = ['order by t1.ts' , 'order by t1.ts asc' , 'order by t2.ts' , 'order by t2.ts asc'] + self.order_desc_where = ['order by ts' , 'order by ts asc' , 'order by ts desc' ] + self.orders_desc_where = ['order by ts' , 'order by ts asc' , 'order by ts desc' , 'order by ts,loc' , 'order by ts,loc asc' , 'order by ts,loc desc'] + + self.group_where = ['group by tbname , loc' , 'group by tbname', 'group by tbname, t_bigint', 'group by tbname,t_int', 'group by tbname, t_smallint', 'group by tbname,t_tinyint', + 'group by tbname,t_float', 'group by tbname,t_double' , 'group by tbname,t_binary', 'group by tbname,t_nchar', 'group by tbname,t_bool' ,'group by tbname ,loc ,t_bigint', + 'group by tbname,t_binary ,t_nchar ,t_bool' , 'group by tbname,t_int ,t_smallint ,t_tinyint' , 'group by tbname,t_float ,t_double ' , + 'PARTITION BY tbname , loc' , 'PARTITION BY tbname', 'PARTITION BY tbname, t_bigint', 'PARTITION BY tbname,t_int', 'PARTITION BY tbname, t_smallint', 'PARTITION BY tbname,t_tinyint', + 'PARTITION BY tbname,t_float', 'PARTITION BY tbname,t_double' , 'PARTITION BY tbname,t_binary', 'PARTITION BY tbname,t_nchar', 'PARTITION BY tbname,t_bool' ,'PARTITION BY tbname ,loc ,t_bigint', + 'PARTITION BY tbname,t_binary ,t_nchar ,t_bool' , 'PARTITION BY tbname,t_int ,t_smallint ,t_tinyint' , 'PARTITION BY tbname,t_float ,t_double '] + self.group_where_j = ['group by t1.loc' , 'group by t1.t_bigint', 'group by t1.t_int', 'group by t1.t_smallint', 'group by t1.t_tinyint', + 'group by t1.t_float', 'group by t1.t_double' , 'group by t1.t_binary', 'group by t1.t_nchar', 'group by t1.t_bool' ,'group by t1.loc ,t1.t_bigint', + 'group by t1.t_binary ,t1.t_nchar ,t1.t_bool' , 'group by t1.t_int ,t1.t_smallint ,t1.t_tinyint' , 'group by t1.t_float ,t1.t_double ' , + 'PARTITION BY t1.loc' , 'PARTITION by t1.t_bigint', 'PARTITION by t1.t_int', 'PARTITION by t1.t_smallint', 'PARTITION by t1.t_tinyint', + 'PARTITION by t1.t_float', 'PARTITION by t1.t_double' , 'PARTITION by t1.t_binary', 'PARTITION by t1.t_nchar', 'PARTITION by t1.t_bool' ,'PARTITION BY t1.loc ,t1.t_bigint', + 'PARTITION by t1.t_binary ,t1.t_nchar ,t1.t_bool' , 'PARTITION by t1.t_int ,t1.t_smallint ,t1.t_tinyint' , 'PARTITION by t1.t_float ,t1.t_double ', + 'group by t2.loc' , 'group by t2.t_bigint', 'group by t2.t_int', 'group by t2.t_smallint', 'group by t2.t_tinyint', + 'group by t2.t_float', 'group by t2.t_double' , 'group by t2.t_binary', 'group by t2.t_nchar', 'group by t2.t_bool' ,'group by t2.loc ,t2.t_bigint', + 'group by t2.t_binary ,t2.t_nchar ,t2.t_bool' , 'group by t2.t_int ,t2.t_smallint ,t2.t_tinyint' , 'group by t2.t_float ,t2.t_double ' , + 'PARTITION BY t2.loc' , 'PARTITION by t2.t_bigint', 'PARTITION by t2.t_int', 'PARTITION by t2.t_smallint', 'PARTITION by t2.t_tinyint', + 'PARTITION by t2.t_float', 'PARTITION by t2.t_double' , 'PARTITION by t2.t_binary', 'PARTITION by t2.t_nchar', 'PARTITION by t2.t_bool' ,'PARTITION BY t2.loc ,t2.t_bigint', + 'PARTITION by t2.t_binary ,t2.t_nchar ,t2.t_bool' , 'PARTITION by t2.t_int ,t2.t_smallint ,t2.t_tinyint' , 'PARTITION by t2.t_float ,t2.t_double '] + + self.group_only_where = ['group by tbname , loc' , 'group by tbname', 'group by tbname, t_bigint', 'group by tbname,t_int', 'group by tbname, t_smallint', 'group by tbname,t_tinyint', + 'group by tbname,t_float', 'group by tbname,t_double' , 'group by tbname,t_binary', 'group by tbname,t_nchar', 'group by tbname,t_bool' ,'group by tbname ,loc ,t_bigint', + 'group by tbname,t_binary ,t_nchar ,t_bool' , 'group by tbname,t_int ,t_smallint ,t_tinyint' , 'group by tbname,t_float ,t_double ' ] + self.group_only_where_j = ['group by t1.loc' , 'group by t1.t_bigint', 'group by t1.t_int', 'group by t1.t_smallint', 'group by t1.t_tinyint', + 'group by t1.t_float', 'group by t1.t_double' , 'group by t1.t_binary', 'group by t1.t_nchar', 'group by t1.t_bool' ,'group by t1.loc ,t1.t_bigint', + 'group by t1.t_binary ,t1.t_nchar ,t1.t_bool' , 'group by t1.t_int ,t1.t_smallint ,t1.t_tinyint' , 'group by t1.t_float ,t1.t_double ' , + 'group by t2.loc' , 'group by t2.t_bigint', 'group by t2.t_int', 'group by t2.t_smallint', 'group by t2.t_tinyint', + 'group by t2.t_float', 'group by t2.t_double' , 'group by t2.t_binary', 'group by t2.t_nchar', 'group by t2.t_bool' ,'group by t2.loc ,t2.t_bigint', + 'group by t2.t_binary ,t2.t_nchar ,t2.t_bool' , 'group by t2.t_int ,t2.t_smallint ,t2.t_tinyint' , 'group by t2.t_float ,t2.t_double ' ] + + self.partiton_where = ['PARTITION BY tbname , loc' , 'PARTITION BY tbname', 'PARTITION BY tbname, t_bigint', 'PARTITION BY tbname,t_int', 'PARTITION BY tbname, t_smallint', 'PARTITION BY tbname,t_tinyint', + 'PARTITION BY tbname,t_float', 'PARTITION BY tbname,t_double' , 'PARTITION BY tbname,t_binary', 'PARTITION BY tbname,t_nchar', 'PARTITION BY tbname,t_bool' ,'PARTITION BY tbname ,loc ,t_bigint', + 'PARTITION BY tbname,t_binary ,t_nchar ,t_bool' , 'PARTITION BY tbname,t_int ,t_smallint ,t_tinyint' , 'PARTITION BY tbname,t_float ,t_double '] + self.partiton_where_j = ['PARTITION BY t1.loc' , 'PARTITION by t1.t_bigint', 'PARTITION by t1.t_int', 'PARTITION by t1.t_smallint', 'PARTITION by t1.t_tinyint', + 'PARTITION by t1.t_float', 'PARTITION by t1.t_double' , 'PARTITION by t1.t_binary', 'PARTITION by t1.t_nchar', 'PARTITION by t1.t_bool' ,'PARTITION BY t1.loc ,t1.t_bigint', + 'PARTITION by t1.t_binary ,t1.t_nchar ,t1.t_bool' , 'PARTITION by t1.t_int ,t1.t_smallint ,t1.t_tinyint' , 'PARTITION by t1.t_float ,t1.t_double ', + 'PARTITION BY t2.loc' , 'PARTITION by t2.t_bigint', 'PARTITION by t2.t_int', 'PARTITION by t2.t_smallint', 'PARTITION by t2.t_tinyint', + 'PARTITION by t2.t_float', 'PARTITION by t2.t_double' , 'PARTITION by t2.t_binary', 'PARTITION by t2.t_nchar', 'PARTITION by t2.t_bool' ,'PARTITION BY t2.loc ,t2.t_bigint', + 'PARTITION by t2.t_binary ,t2.t_nchar ,t2.t_bool' , 'PARTITION by t2.t_int ,t2.t_smallint ,t2.t_tinyint' , 'PARTITION by t2.t_float ,t2.t_double '] + + + self.group_where_regular = ['group by tbname ' , 'group by tbname', 'group by tbname, q_bigint', 'group by tbname,q_int', 'group by tbname, q_smallint', 'group by tbname,q_tinyint', + 'group by tbname,q_float', 'group by tbname,q_double' , 'group by tbname,q_binary', 'group by tbname,q_nchar', 'group by tbname,q_bool' ,'group by tbname ,q_bigint', + 'group by tbname,q_binary ,q_nchar ,q_bool' , 'group by tbname,q_int ,q_smallint ,q_tinyint' , 'group by tbname,q_float ,q_double ' , + 'PARTITION BY tbname ' , 'PARTITION BY tbname', 'PARTITION BY tbname, q_bigint', 'PARTITION BY tbname,q_int', 'PARTITION BY tbname, q_smallint', 'PARTITION BY tbname,q_tinyint', + 'PARTITION BY tbname,q_float', 'PARTITION BY tbname,q_double' , 'PARTITION BY tbname,q_binary', 'PARTITION BY tbname,q_nchar', 'PARTITION BY tbname,q_bool' ,'PARTITION BY tbname ,q_bigint', + 'PARTITION BY tbname,q_binary ,q_nchar ,q_bool' , 'PARTITION BY tbname,q_int ,q_smallint ,q_tinyint' , 'PARTITION BY tbname,q_float ,q_double '] + self.group_where_regular_j = ['group by t1.q_bigint', 'group by t1.q_int', 'group by t1.q_smallint', 'group by t1.q_tinyint', + 'group by t1.q_float', 'group by t1.q_double' , 'group by t1.q_binary', 'group by t1.q_nchar', 'group by t1.q_bool' ,'group by t1.q_bigint', + 'group by t1.q_binary ,t1.q_nchar ,t1.q_bool' , 'group by t1.q_int ,t1.q_smallint ,t1.q_tinyint' , 'group by t1.q_float ,t1.q_double ' , + 'PARTITION by t1.q_bigint', 'PARTITION by t1.q_int', 'PARTITION by t1.q_smallint', 'PARTITION by t1.q_tinyint', + 'PARTITION by t1.q_float', 'PARTITION by t1.q_double' , 'PARTITION by t1.q_binary', 'PARTITION by t1.q_nchar', 'PARTITION by t1.q_bool' ,'PARTITION BY t1.q_bigint', + 'PARTITION by t1.q_binary ,t1.q_nchar ,t1.q_bool' , 'PARTITION by t1.q_int ,t1.q_smallint ,t1.q_tinyint' , 'PARTITION by t1.q_float ,t1.q_double ', + 'group by t2.q_bigint', 'group by t2.q_int', 'group by t2.q_smallint', 'group by t2.q_tinyint', + 'group by t2.q_float', 'group by t2.q_double' , 'group by t2.q_binary', 'group by t2.q_nchar', 'group by t2.q_bool' ,'group by t2.q_bigint', + 'group by t2.q_binary ,t2.q_nchar ,t2.q_bool' , 'group by t2.q_int ,t2.q_smallint ,t2.q_tinyint' , 'group by t2.q_float ,t2.q_double ' , + 'PARTITION by t2.q_bigint', 'PARTITION by t2.q_int', 'PARTITION by t2.q_smallint', 'PARTITION by t2.q_tinyint', + 'PARTITION by t2.q_float', 'PARTITION by t2.q_double' , 'PARTITION by t2.q_binary', 'PARTITION by t2.q_nchar', 'PARTITION by t2.q_bool' ,'PARTITION BY t2.q_bigint', + 'PARTITION by t2.q_binary ,t2.q_nchar ,t2.q_bool' , 'PARTITION by t2.q_int ,t2.q_smallint ,t2.q_tinyint' , 'PARTITION by t2.q_float ,t2.q_double '] + + self.partiton_where_regular = ['PARTITION BY tbname ' , 'PARTITION BY tbname', 'PARTITION BY tbname, q_bigint', 'PARTITION BY tbname,q_int', 'PARTITION BY tbname, q_smallint', 'PARTITION BY tbname,q_tinyint', + 'PARTITION BY tbname,q_float', 'PARTITION BY tbname,q_double' , 'PARTITION BY tbname,q_binary', 'PARTITION BY tbname,q_nchar', 'PARTITION BY tbname,q_bool' ,'PARTITION BY tbname ,q_bigint', + 'PARTITION BY tbname,q_binary ,q_nchar ,q_bool' , 'PARTITION BY tbname,q_int ,q_smallint ,q_tinyint' , 'PARTITION BY tbname,q_float ,q_double '] + self.partiton_where_regular_j = ['PARTITION by t1.q_bigint', 'PARTITION by t1.q_int', 'PARTITION by t1.q_smallint', 'PARTITION by t1.q_tinyint', + 'PARTITION by t1.q_float', 'PARTITION by t1.q_double' , 'PARTITION by t1.q_binary', 'PARTITION by t1.q_nchar', 'PARTITION by t1.q_bool' ,'PARTITION BY t1.q_bigint', + 'PARTITION by t1.q_binary ,t1.q_nchar ,t1.q_bool' , 'PARTITION by t1.q_int ,t1.q_smallint ,t1.q_tinyint' , 'PARTITION by t1.q_float ,t1.q_double ', + 'PARTITION by t2.q_bigint', 'PARTITION by t2.q_int', 'PARTITION by t2.q_smallint', 'PARTITION by t2.q_tinyint', + 'PARTITION by t2.q_float', 'PARTITION by t2.q_double' , 'PARTITION by t2.q_binary', 'PARTITION by t2.q_nchar', 'PARTITION by t2.q_bool' ,'PARTITION BY t2.q_bigint', + 'PARTITION by t2.q_binary ,t2.q_nchar ,t2.q_bool' , 'PARTITION by t2.q_int ,t2.q_smallint ,t2.q_tinyint' , 'PARTITION by t2.q_float ,t2.q_double '] + + self.having_support = ['having count(q_int) > 0','having count(q_bigint) > 0','having count(q_smallint) > 0','having count(q_tinyint) > 0','having count(q_float) > 0','having count(q_double) > 0','having count(q_bool) > 0', + 'having avg(q_int) > 0','having avg(q_bigint) > 0','having avg(q_smallint) > 0','having avg(q_tinyint) > 0','having avg(q_float) > 0','having avg(q_double) > 0', + 'having sum(q_int) > 0','having sum(q_bigint) > 0','having sum(q_smallint) > 0','having sum(q_tinyint) > 0','having sum(q_float) > 0','having sum(q_double) > 0', + 'having STDDEV(q_int) > 0','having STDDEV(q_bigint) > 0','having STDDEV(q_smallint) > 0','having STDDEV(q_tinyint) > 0','having STDDEV(q_float) > 0','having STDDEV(q_double) > 0', + 'having TWA(q_int) > 0','having TWA(q_bigint) > 0','having TWA(q_smallint) > 0','having TWA(q_tinyint) > 0','having TWA(q_float) > 0','having TWA(q_double) > 0', + 'having IRATE(q_int) > 0','having IRATE(q_bigint) > 0','having IRATE(q_smallint) > 0','having IRATE(q_tinyint) > 0','having IRATE(q_float) > 0','having IRATE(q_double) > 0', + 'having MIN(q_int) > 0','having MIN(q_bigint) > 0','having MIN(q_smallint) > 0','having MIN(q_tinyint) > 0','having MIN(q_float) > 0','having MIN(q_double) > 0', + 'having MAX(q_int) > 0','having MAX(q_bigint) > 0','having MAX(q_smallint) > 0','having MAX(q_tinyint) > 0','having MAX(q_float) > 0','having MAX(q_double) > 0', + 'having FIRST(q_int) > 0','having FIRST(q_bigint) > 0','having FIRST(q_smallint) > 0','having FIRST(q_tinyint) > 0','having FIRST(q_float) > 0','having FIRST(q_double) > 0', + 'having LAST(q_int) > 0','having LAST(q_bigint) > 0','having LAST(q_smallint) > 0','having LAST(q_tinyint) > 0','having LAST(q_float) > 0','having LAST(q_double) > 0', + 'having APERCENTILE(q_int,10) > 0','having APERCENTILE(q_bigint,10) > 0','having APERCENTILE(q_smallint,10) > 0','having APERCENTILE(q_tinyint,10) > 0','having APERCENTILE(q_float,10) > 0','having APERCENTILE(q_double,10) > 0', + 'having count(q_int_null) > 0','having count(q_bigint_null) > 0','having count(q_smallint_null) > 0','having count(q_tinyint_null) > 0','having count(q_float_null) > 0','having count(q_double_null) > 0','having count(q_bool_null) > 0', + 'having avg(q_int_null) > 0','having avg(q_bigint_null) > 0','having avg(q_smallint_null) > 0','having avg(q_tinyint_null) > 0','having avg(q_float_null) > 0','having avg(q_double_null) > 0', + 'having sum(q_int_null) > 0','having sum(q_bigint_null) > 0','having sum(q_smallint_null) > 0','having sum(q_tinyint_null) > 0','having sum(q_float_null) > 0','having sum(q_double_null) > 0', + 'having STDDEV(q_int_null) > 0','having STDDEV(q_bigint_null) > 0','having STDDEV(q_smallint_null) > 0','having STDDEV(q_tinyint_null) > 0','having STDDEV(q_float_null) > 0','having STDDEV(q_double_null) > 0', + 'having TWA(q_int_null) > 0','having TWA(q_bigint_null) > 0','having TWA(q_smallint_null) > 0','having TWA(q_tinyint_null) > 0','having TWA(q_float_null) > 0','having TWA(q_double_null) > 0', + 'having IRATE(q_int_null) > 0','having IRATE(q_bigint_null) > 0','having IRATE(q_smallint_null) > 0','having IRATE(q_tinyint_null) > 0','having IRATE(q_float_null) > 0','having IRATE(q_double_null) > 0', + 'having MIN(q_int_null) > 0','having MIN(q_bigint_null) > 0','having MIN(q_smallint_null) > 0','having MIN(q_tinyint_null) > 0','having MIN(q_float_null) > 0','having MIN(q_double_null) > 0', + 'having MAX(q_int_null) > 0','having MAX(q_bigint_null) > 0','having MAX(q_smallint_null) > 0','having MAX(q_tinyint_null) > 0','having MAX(q_float_null) > 0','having MAX(q_double_null) > 0', + 'having FIRST(q_int_null) > 0','having FIRST(q_bigint_null) > 0','having FIRST(q_smallint_null) > 0','having FIRST(q_tinyint_null) > 0','having FIRST(q_float_null) > 0','having FIRST(q_double_null) > 0', + 'having LAST(q_int_null) > 0','having LAST(q_bigint_null) > 0','having LAST(q_smallint_null) > 0','having LAST(q_tinyint_null) > 0','having LAST(q_float_null) > 0','having LAST(q_double_null) > 0', + 'having APERCENTILE(q_int_null,10) > 0','having APERCENTILE(q_bigint_null,10) > 0','having APERCENTILE(q_smallint_null,10) > 0','having APERCENTILE(q_tinyint_null,10) > 0','having APERCENTILE(q_float_null,10) > 0','having APERCENTILE(q_double_null,10) > 0'] + self.having_not_support = ['having TOP(q_int,10) > 0','having TOP(q_bigint,10) > 0','having TOP(q_smallint,10) > 0','having TOP(q_tinyint,10) > 0','having TOP(q_float,10) > 0','having TOP(q_double,10) > 0','having TOP(q_bool,10) > 0', + 'having BOTTOM(q_int,10) > 0','having BOTTOM(q_bigint,10) > 0','having BOTTOM(q_smallint,10) > 0','having BOTTOM(q_tinyint,10) > 0','having BOTTOM(q_float,10) > 0','having BOTTOM(q_double,10) > 0','having BOTTOM(q_bool,10) > 0', + 'having LEASTSQUARES(q_int) > 0','having LEASTSQUARES(q_bigint) > 0','having LEASTSQUARES(q_smallint) > 0','having LEASTSQUARES(q_tinyint) > 0','having LEASTSQUARES(q_float) > 0','having LEASTSQUARES(q_double) > 0','having LEASTSQUARES(q_bool) > 0', + 'having FIRST(q_bool) > 0','having IRATE(q_bool) > 0','having PERCENTILE(q_bool,10) > 0','having avg(q_bool) > 0','having LAST_ROW(q_bool) > 0','having sum(q_bool) > 0','having STDDEV(q_bool) > 0','having APERCENTILE(q_bool,10) > 0','having TWA(q_bool) > 0','having LAST(q_bool) > 0', + 'having PERCENTILE(q_int,10) > 0','having PERCENTILE(q_bigint,10) > 0','having PERCENTILE(q_smallint,10) > 0','having PERCENTILE(q_tinyint,10) > 0','having PERCENTILE(q_float,10) > 0','having PERCENTILE(q_double,10) > 0', + 'having TOP(q_int_null,10) > 0','having TOP(q_bigint_null,10) > 0','having TOP(q_smallint_null,10) > 0','having TOP(q_tinyint_null,10) > 0','having TOP(q_float_null,10) > 0','having TOP(q_double_null,10) > 0','having TOP(q_bool_null,10) > 0', + 'having BOTTOM(q_int_null,10) > 0','having BOTTOM(q_bigint_null,10) > 0','having BOTTOM(q_smallint_null,10) > 0','having BOTTOM(q_tinyint_null,10) > 0','having BOTTOM(q_float_null,10) > 0','having BOTTOM(q_double_null,10) > 0','having BOTTOM(q_bool_null,10) > 0', + 'having LEASTSQUARES(q_int_null) > 0','having LEASTSQUARES(q_bigint_null) > 0','having LEASTSQUARES(q_smallint_null) > 0','having LEASTSQUARES(q_tinyint_null) > 0','having LEASTSQUARES(q_float_null) > 0','having LEASTSQUARES(q_double_null) > 0','having LEASTSQUARES(q_bool_null) > 0', + 'having FIRST(q_bool_null) > 0','having IRATE(q_bool_null) > 0','having PERCENTILE(q_bool_null,10) > 0','having avg(q_bool_null) > 0','having LAST_ROW(q_bool_null) > 0','having sum(q_bool_null) > 0','having STDDEV(q_bool_null) > 0','having APERCENTILE(q_bool_null,10) > 0','having TWA(q_bool_null) > 0','having LAST(q_bool_null) > 0', + 'having PERCENTILE(q_int_null,10) > 0','having PERCENTILE(q_bigint_null,10) > 0','having PERCENTILE(q_smallint_null,10) > 0','having PERCENTILE(q_tinyint_null,10) > 0','having PERCENTILE(q_float_null,10) > 0','having PERCENTILE(q_double_null,10) > 0'] + self.having_tagnot_support = ['having LAST_ROW(q_int) > 0','having LAST_ROW(q_bigint) > 0','having LAST_ROW(q_smallint) > 0','having LAST_ROW(q_tinyint) > 0','having LAST_ROW(q_float) > 0','having LAST_ROW(q_double) > 0', + 'having LAST_ROW(q_int_null) > 0','having LAST_ROW(q_bigint_null) > 0','having LAST_ROW(q_smallint_null) > 0','having LAST_ROW(q_tinyint_null) > 0','having LAST_ROW(q_float_null) > 0','having LAST_ROW(q_double_null) > 0'] + + self.having_support_j = ['having count(t1.q_int) > 0','having count(t1.q_bigint) > 0','having count(t1.q_smallint) > 0','having count(t1.q_tinyint) > 0','having count(t1.q_float) > 0','having count(t1.q_double) > 0','having count(t1.q_bool) > 0', + 'having avg(t1.q_int) > 0','having avg(t1.q_bigint) > 0','having avg(t1.q_smallint) > 0','having avg(t1.q_tinyint) > 0','having avg(t1.q_float) > 0','having avg(t1.q_double) > 0', + 'having sum(t1.q_int) > 0','having sum(t1.q_bigint) > 0','having sum(t1.q_smallint) > 0','having sum(t1.q_tinyint) > 0','having sum(t1.q_float) > 0','having sum(t1.q_double) > 0', + 'having STDDEV(t1.q_int) > 0','having STDDEV(t1.q_bigint) > 0','having STDDEV(t1.q_smallint) > 0','having STDDEV(t1.q_tinyint) > 0','having STDDEV(t1.q_float) > 0','having STDDEV(t1.q_double) > 0', + 'having TWA(t1.q_int) > 0','having TWA(t1.q_bigint) > 0','having TWA(t1.q_smallint) > 0','having TWA(t1.q_tinyint) > 0','having TWA(t1.q_float) > 0','having TWA(t1.q_double) > 0', + 'having IRATE(t1.q_int) > 0','having IRATE(t1.q_bigint) > 0','having IRATE(t1.q_smallint) > 0','having IRATE(t1.q_tinyint) > 0','having IRATE(t1.q_float) > 0','having IRATE(t1.q_double) > 0', + 'having MIN(t1.q_int) > 0','having MIN(t1.q_bigint) > 0','having MIN(t1.q_smallint) > 0','having MIN(t1.q_tinyint) > 0','having MIN(t1.q_float) > 0','having MIN(t1.q_double) > 0', + 'having MAX(t1.q_int) > 0','having MAX(t1.q_bigint) > 0','having MAX(t1.q_smallint) > 0','having MAX(t1.q_tinyint) > 0','having MAX(t1.q_float) > 0','having MAX(t1.q_double) > 0', + 'having FIRST(t1.q_int) > 0','having FIRST(t1.q_bigint) > 0','having FIRST(t1.q_smallint) > 0','having FIRST(t1.q_tinyint) > 0','having FIRST(t1.q_float) > 0','having FIRST(t1.q_double) > 0', + 'having LAST(t1.q_int) > 0','having LAST(t1.q_bigint) > 0','having LAST(t1.q_smallint) > 0','having LAST(t1.q_tinyint) > 0','having LAST(t1.q_float) > 0','having LAST(t1.q_double) > 0', + 'having APERCENTILE(t1.q_int,10) > 0','having APERCENTILE(t1.q_bigint,10) > 0','having APERCENTILE(t1.q_smallint,10) > 0','having APERCENTILE(t1.q_tinyint,10) > 0','having APERCENTILE(t1.q_float,10) > 0','having APERCENTILE(t1.q_double,10) > 0'] + + # limit offset where + self.limit_where = ['limit 1 offset 1' , 'limit 1' , 'limit 2 offset 1' , 'limit 2', 'limit 12 offset 1' , 'limit 20', 'limit 20 offset 10' , 'limit 200'] + self.limit1_where = ['limit 1 offset 1' , 'limit 1' ] + self.limit_u_where = ['limit 100 offset 10' , 'limit 50' , 'limit 100' , 'limit 10' ] + + # slimit soffset where + self.slimit_where = ['slimit 1 soffset 1' , 'slimit 1' , 'slimit 2 soffset 1' , 'slimit 2'] + self.slimit1_where = ['slimit 2 soffset 1' , 'slimit 1' ] + + # aggregate function include [all:count(*)\avg\sum\stddev ||regualr:twa\irate\leastsquares ||group by tbname:twa\irate\] + # select function include [all: min\max\first(*)\last(*)\top\bottom\apercentile\last_row(*)(not with interval)\interp(*)(FILL) ||regualr: percentile] + # calculation function include [all:spread\+-*/ ||regualr:diff\derivative ||group by tbname:diff\derivative\] + # **_ns_** express is not support stable, therefore, separated from regular tables + # calc_select_all calc_select_regular calc_select_in_ts calc_select_fill calc_select_not_interval + # calc_aggregate_all calc_aggregate_regular calc_aggregate_groupbytbname + # calc_calculate_all calc_calculate_regular calc_calculate_groupbytbname + + # calc_select_all calc_select_regular calc_select_in_ts calc_select_fill calc_select_not_interval + # select function include [all: min\max\first(*)\last(*)\top\bottom\apercentile\last_row(*)(not with interval)\interp(*)(FILL) ||regualr: percentile] + + self.calc_select_all = ['bottom(q_int,20)' , 'bottom(q_bigint,20)' , 'bottom(q_smallint,20)' , 'bottom(q_tinyint,20)' ,'bottom(q_float,20)' , 'bottom(q_double,20)' , + 'top(q_int,20)' , 'top(q_bigint,20)' , 'top(q_smallint,20)' ,'top(q_tinyint,20)' ,'top(q_float,20)' ,'top(q_double,20)' , + 'first(q_int)' , 'first(q_bigint)' , 'first(q_smallint)' , 'first(q_tinyint)' , 'first(q_float)' ,'first(q_double)' ,'first(q_binary)' ,'first(q_nchar)' ,'first(q_bool)' ,'first(q_ts)' , + 'last(q_int)' , 'last(q_bigint)' , 'last(q_smallint)' , 'last(q_tinyint)' , 'last(q_float)' ,'last(q_double)' , 'last(q_binary)' ,'last(q_nchar)' ,'last(q_bool)' ,'last(q_ts)' , + 'min(q_int)' , 'min(q_bigint)' , 'min(q_smallint)' , 'min(q_tinyint)' , 'min(q_float)' ,'min(q_double)' , + 'max(q_int)' , 'max(q_bigint)' , 'max(q_smallint)' , 'max(q_tinyint)' ,'max(q_float)' ,'max(q_double)' , + 'apercentile(q_int,20)' , 'apercentile(q_bigint,20)' ,'apercentile(q_smallint,20)' ,'apercentile(q_tinyint,20)' ,'apercentile(q_float,20)' ,'apercentile(q_double,20)' , + 'last_row(q_int)' , 'last_row(q_bigint)' , 'last_row(q_smallint)' , 'last_row(q_tinyint)' , 'last_row(q_float)' , + 'last_row(q_double)' , 'last_row(q_bool)' ,'last_row(q_binary)' ,'last_row(q_nchar)' ,'last_row(q_ts)', + 'bottom(q_int_null,20)' , 'bottom(q_bigint_null,20)' , 'bottom(q_smallint_null,20)' , 'bottom(q_tinyint_null,20)' ,'bottom(q_float_null,20)' , 'bottom(q_double_null,20)' , + 'top(q_int_null,20)' , 'top(q_bigint_null,20)' , 'top(q_smallint_null,20)' ,'top(q_tinyint_null,20)' ,'top(q_float_null,20)' ,'top(q_double_null,20)' , + 'first(q_int_null)' , 'first(q_bigint_null)' , 'first(q_smallint_null)' , 'first(q_tinyint_null)' , 'first(q_float_null)' ,'first(q_double_null)' ,'first(q_binary_null)' ,'first(q_nchar_null)' ,'first(q_bool_null)' ,'first(q_ts_null)' , + 'last(q_int_null)' , 'last(q_bigint_null)' , 'last(q_smallint_null)' , 'last(q_tinyint_null)' , 'last(q_float_null)' ,'last(q_double_null)' , 'last(q_binary_null)' ,'last(q_nchar_null)' ,'last(q_bool_null)' ,'last(q_ts_null)' , + 'min(q_int_null)' , 'min(q_bigint_null)' , 'min(q_smallint_null)' , 'min(q_tinyint_null)' , 'min(q_float_null)' ,'min(q_double_null)' , + 'max(q_int_null)' , 'max(q_bigint_null)' , 'max(q_smallint_null)' , 'max(q_tinyint_null)' ,'max(q_float_null)' ,'max(q_double_null)' , + 'last_row(q_int_null)' , 'last_row(q_bigint_null)' , 'last_row(q_smallint_null)' , 'last_row(q_tinyint_null)' , 'last_row(q_float_null)' , + 'last_row(q_double_null)' , 'last_row(q_bool_null)' ,'last_row(q_binary_null)' ,'last_row(q_nchar_null)' ,'last_row(q_ts_null)', + 'apercentile(q_int_null,20)' , 'apercentile(q_bigint_null,20)' ,'apercentile(q_smallint_null,20)' ,'apercentile(q_tinyint_null,20)' ,'apercentile(q_float_null,20)' ,'apercentile(q_double_null,20)' ,] + + self.calc_select_in_ts = ['bottom(q_int,20)' , 'bottom(q_bigint,20)' , 'bottom(q_smallint,20)' , 'bottom(q_tinyint,20)' ,'bottom(q_float,20)' , 'bottom(q_double,20)' , + 'top(q_int,20)' , 'top(q_bigint,20)' , 'top(q_smallint,20)' ,'top(q_tinyint,20)' ,'top(q_float,20)' ,'top(q_double,20)' , + 'bottom(q_int_null,20)' , 'bottom(q_bigint_null,20)' , 'bottom(q_smallint_null,20)' , 'bottom(q_tinyint_null,20)' ,'bottom(q_float_null,20)' , 'bottom(q_double_null,20)' , + 'top(q_int_null,20)' , 'top(q_bigint_null,20)' , 'top(q_smallint_null,20)' ,'top(q_tinyint_null,20)' ,'top(q_float_null,20)' ,'top(q_double_null,20)' , + 'first(q_int)' , 'first(q_bigint)' , 'first(q_smallint)' , 'first(q_tinyint)' , 'first(q_float)' ,'first(q_double)' ,'first(q_binary)' ,'first(q_nchar)' ,'first(q_bool)' ,'first(q_ts)' , + 'last(q_int)' , 'last(q_bigint)' , 'last(q_smallint)' , 'last(q_tinyint)' , 'last(q_float)' ,'last(q_double)' , 'last(q_binary)' ,'last(q_nchar)' ,'last(q_bool)' ,'last(q_ts)' , + 'first(q_int_null)' , 'first(q_bigint_null)' , 'first(q_smallint_null)' , 'first(q_tinyint_null)' , 'first(q_float_null)' ,'first(q_double_null)' ,'first(q_binary_null)' ,'first(q_nchar_null)' ,'first(q_bool_null)' ,'first(q_ts_null)' , + 'last(q_int_null)' , 'last(q_bigint_null)' , 'last(q_smallint_null)' , 'last(q_tinyint_null)' , 'last(q_float_null)' ,'last(q_double_null)' , 'last(q_binary_null)' ,'last(q_nchar_null)' ,'last(q_bool_null)' ,'last(q_ts_null)' ] + + self.calc_select_in = ['min(q_int)' , 'min(q_bigint)' , 'min(q_smallint)' , 'min(q_tinyint)' , 'min(q_float)' ,'min(q_double)' , + 'max(q_int)' , 'max(q_bigint)' , 'max(q_smallint)' , 'max(q_tinyint)' ,'max(q_float)' ,'max(q_double)' , + 'apercentile(q_int,20)' , 'apercentile(q_bigint,20)' ,'apercentile(q_smallint,20)' ,'apercentile(q_tinyint,20)' ,'apercentile(q_float,20)' ,'apercentile(q_double,20)' , + 'last_row(q_int)' , 'last_row(q_bigint)' , 'last_row(q_smallint)' , 'last_row(q_tinyint)' , 'last_row(q_float)' , + 'last_row(q_double)' , 'last_row(q_bool)' ,'last_row(q_binary)' ,'last_row(q_nchar)' ,'last_row(q_ts)', + 'min(q_int_null)' , 'min(q_bigint_null)' , 'min(q_smallint_null)' , 'min(q_tinyint_null)' , 'min(q_float_null)' ,'min(q_double_null)' , + 'max(q_int_null)' , 'max(q_bigint_null)' , 'max(q_smallint_null)' , 'max(q_tinyint_null)' ,'max(q_float_null)' ,'max(q_double_null)' , + 'apercentile(q_int_null,20)' , 'apercentile(q_bigint_null,20)' ,'apercentile(q_smallint_null,20)' ,'apercentile(q_tinyint_null,20)' ,'apercentile(q_float_null,20)' ,'apercentile(q_double_null,20)' , + 'last_row(q_int_null)' , 'last_row(q_bigint_null)' , 'last_row(q_smallint_null)' , 'last_row(q_tinyint_null)' , 'last_row(q_float_null)' , + 'last_row(q_double_null)' , 'last_row(q_bool_null)' ,'last_row(q_binary_null)' ,'last_row(q_nchar_null)' ,'last_row(q_ts_null)'] + + self.calc_select_not_support_ts = ['first(q_int)' , 'first(q_bigint)' , 'first(q_smallint)' , 'first(q_tinyint)' , 'first(q_float)' ,'first(q_double)' ,'first(q_binary)' ,'first(q_nchar)' ,'first(q_bool)' ,'first(q_ts)' , + 'last(q_int)' , 'last(q_bigint)' , 'last(q_smallint)' , 'last(q_tinyint)' , 'last(q_float)' ,'last(q_double)' , 'last(q_binary)' ,'last(q_nchar)' ,'last(q_bool)' ,'last(q_ts)' , + 'last_row(q_int)' , 'last_row(q_bigint)' , 'last_row(q_smallint)' , 'last_row(q_tinyint)' , 'last_row(q_float)' , + 'last_row(q_double)' , 'last_row(q_bool)' ,'last_row(q_binary)' ,'last_row(q_nchar)' ,'last_row(q_ts)', + 'apercentile(q_int,20)' , 'apercentile(q_bigint,20)' ,'apercentile(q_smallint,20)' ,'apercentile(q_tinyint,20)' ,'apercentile(q_float,20)' ,'apercentile(q_double,20)', + 'first(q_int_null)' , 'first(q_bigint_null)' , 'first(q_smallint_null)' , 'first(q_tinyint_null)' , 'first(q_float_null)' ,'first(q_double_null)' ,'first(q_binary_null)' ,'first(q_nchar_null)' ,'first(q_bool_null)' ,'first(q_ts_null)' , + 'last(q_int_null)' , 'last(q_bigint_null)' , 'last(q_smallint_null)' , 'last(q_tinyint_null)' , 'last(q_float_null)' ,'last(q_double_null)' , 'last(q_binary_null)' ,'last(q_nchar_null)' ,'last(q_bool_null)' ,'last(q_ts_null)' , + 'last_row(q_int_null)' , 'last_row(q_bigint_null)' , 'last_row(q_smallint_null)' , 'last_row(q_tinyint_null)' , 'last_row(q_float_null)' , + 'last_row(q_double_null)' , 'last_row(q_bool_null)' ,'last_row(q_binary_null)' ,'last_row(q_nchar_null)' ,'last_row(q_ts_null)', + 'apercentile(q_int_null,20)' , 'apercentile(q_bigint_null,20)' ,'apercentile(q_smallint_null,20)' ,'apercentile(q_tinyint_null,20)' ,'apercentile(q_float_null,20)' ,'apercentile(q_double_null,20)'] + + self.calc_select_support_ts = ['bottom(q_int,20)' , 'bottom(q_bigint,20)' , 'bottom(q_smallint,20)' , 'bottom(q_tinyint,20)' ,'bottom(q_float,20)' , 'bottom(q_double,20)' , + 'top(q_int,20)' , 'top(q_bigint,20)' , 'top(q_smallint,20)' ,'top(q_tinyint,20)' ,'top(q_float,20)' ,'top(q_double,20)' , + 'bottom(q_int_null,20)' , 'bottom(q_bigint_null,20)' , 'bottom(q_smallint_null,20)' , 'bottom(q_tinyint_null,20)' ,'bottom(q_float_null,20)' , 'bottom(q_double_null,20)' , + 'top(q_int_null,20)' , 'top(q_bigint_null,20)' , 'top(q_smallint_null,20)' ,'top(q_tinyint_null,20)' ,'top(q_float_null,20)' ,'top(q_double_null,20)' , + 'min(q_int)' , 'min(q_bigint)' , 'min(q_smallint)' , 'min(q_tinyint)' , 'min(q_float)' ,'min(q_double)' , + 'max(q_int)' , 'max(q_bigint)' , 'max(q_smallint)' , 'max(q_tinyint)' ,'max(q_float)' ,'max(q_double)' , + 'min(q_int_null)' , 'min(q_bigint_null)' , 'min(q_smallint_null)' , 'min(q_tinyint_null)' , 'min(q_float_null)' ,'min(q_double_null)' , + 'max(q_int_null)' , 'max(q_bigint_null)' , 'max(q_smallint_null)' , 'max(q_tinyint_null)' ,'max(q_float_null)' ,'max(q_double_null)'] + + self.calc_select_regular = [ 'PERCENTILE(q_int,10)' ,'PERCENTILE(q_bigint,20)' , 'PERCENTILE(q_smallint,30)' ,'PERCENTILE(q_tinyint,40)' ,'PERCENTILE(q_float,50)' ,'PERCENTILE(q_double,60)', + 'PERCENTILE(q_int_null,10)' ,'PERCENTILE(q_bigint_null,20)' , 'PERCENTILE(q_smallint_null,30)' ,'PERCENTILE(q_tinyint_null,40)' ,'PERCENTILE(q_float_null,50)' ,'PERCENTILE(q_double_null,60)'] + + + self.calc_select_fill = ['INTERP(q_int)' ,'INTERP(q_bigint)' ,'INTERP(q_smallint)' ,'INTERP(q_tinyint)', 'INTERP(q_float)' ,'INTERP(q_double)'] + self.interp_where = ['ts = now' , 'ts = \'2020-09-13 20:26:40.000\'' , 'ts = \'2020-09-13 20:26:40.009\'' ,'tbname in (\'table_1\') and ts = now' ,'tbname in (\'table_0\' ,\'table_1\',\'table_2\',\'table_3\',\'table_4\',\'table_5\') and ts = \'2020-09-13 20:26:40.000\'','tbname like \'table%\' and ts = \'2020-09-13 20:26:40.002\''] + + #two table join + self.calc_select_in_ts_j = ['bottom(t1.q_int,20)' , 'bottom(t1.q_bigint,20)' , 'bottom(t1.q_smallint,20)' , 'bottom(t1.q_tinyint,20)' ,'bottom(t1.q_float,20)' , 'bottom(t1.q_double,20)' , + 'top(t1.q_int,20)' , 'top(t1.q_bigint,20)' , 'top(t1.q_smallint,20)' ,'top(t1.q_tinyint,20)' ,'top(t1.q_float,20)' ,'top(t1.q_double,20)' , + 'first(t1.q_int)' , 'first(t1.q_bigint)' , 'first(t1.q_smallint)' , 'first(t1.q_tinyint)' , 'first(t1.q_float)' ,'first(t1.q_double)' ,'first(t1.q_binary)' ,'first(t1.q_nchar)' ,'first(t1.q_bool)' ,'first(t1.q_ts)' , + 'last(t1.q_int)' , 'last(t1.q_bigint)' , 'last(t1.q_smallint)' , 'last(t1.q_tinyint)' , 'last(t1.q_float)' ,'last(t1.q_double)' , 'last(t1.q_binary)' ,'last(t1.q_nchar)' ,'last(t1.q_bool)' ,'last(t1.q_ts)' , + 'bottom(t2.q_int,20)' , 'bottom(t2.q_bigint,20)' , 'bottom(t2.q_smallint,20)' , 'bottom(t2.q_tinyint,20)' ,'bottom(t2.q_float,20)' , 'bottom(t2.q_double,20)' , + 'top(t2.q_int,20)' , 'top(t2.q_bigint,20)' , 'top(t2.q_smallint,20)' ,'top(t2.q_tinyint,20)' ,'top(t2.q_float,20)' ,'top(t2.q_double,20)' , + 'first(t2.q_int)' , 'first(t2.q_bigint)' , 'first(t2.q_smallint)' , 'first(t2.q_tinyint)' , 'first(t2.q_float)' ,'first(t2.q_double)' ,'first(t2.q_binary)' ,'first(t2.q_nchar)' ,'first(t2.q_bool)' ,'first(t2.q_ts)' , + 'last(t2.q_int)' , 'last(t2.q_bigint)' , 'last(t2.q_smallint)' , 'last(t2.q_tinyint)' , 'last(t2.q_float)' ,'last(t2.q_double)' , 'last(t2.q_binary)' ,'last(t2.q_nchar)' ,'last(t2.q_bool)' ,'last(t2.q_ts)', + 'bottom(t1.q_int_null,20)' , 'bottom(t1.q_bigint_null,20)' , 'bottom(t1.q_smallint_null,20)' , 'bottom(t1.q_tinyint_null,20)' ,'bottom(t1.q_float_null,20)' , 'bottom(t1.q_double_null,20)' , + 'top(t1.q_int_null,20)' , 'top(t1.q_bigint_null,20)' , 'top(t1.q_smallint_null,20)' ,'top(t1.q_tinyint_null,20)' ,'top(t1.q_float_null,20)' ,'top(t1.q_double_null,20)' , + 'first(t1.q_int_null)' , 'first(t1.q_bigint_null)' , 'first(t1.q_smallint_null)' , 'first(t1.q_tinyint_null)' , 'first(t1.q_float_null)' ,'first(t1.q_double_null)' ,'first(t1.q_binary_null)' ,'first(t1.q_nchar_null))' ,'first(t1.q_bool_null)' ,'first(t1.q_ts_null)' , + 'last(t1.q_int_null)' , 'last(t1.q_bigint_null)' , 'last(t1.q_smallint_null)' , 'last(t1.q_tinyint_null)' , 'last(t1.q_float_null)' ,'last(t1.q_double_null)' , 'last(t1.q_binary_null)' ,'last(t1.q_nchar_null))' ,'last(t1.q_bool_null)' ,'last(t1.q_ts_null)' , + 'bottom(t2.q_int_null,20)' , 'bottom(t2.q_bigint_null,20)' , 'bottom(t2.q_smallint_null,20)' , 'bottom(t2.q_tinyint_null,20)' ,'bottom(t2.q_float_null,20)' , 'bottom(t2.q_double_null,20)' , + 'top(t2.q_int_null,20)' , 'top(t2.q_bigint_null,20)' , 'top(t2.q_smallint_null,20)' ,'top(t2.q_tinyint_null,20)' ,'top(t2.q_float_null,20)' ,'top(t2.q_double_null,20)' , + 'first(t2.q_int_null)' , 'first(t2.q_bigint_null)' , 'first(t2.q_smallint_null)' , 'first(t2.q_tinyint_null)' , 'first(t2.q_float_null)' ,'first(t2.q_double_null)' ,'first(t2.q_binary_null)' ,'first(t2.q_nchar_null))' ,'first(t2.q_bool_null)' ,'first(t2.q_ts_null)' , + 'last(t2.q_int_null)' , 'last(t2.q_bigint_null)' , 'last(t2.q_smallint_null)' , 'last(t2.q_tinyint_null)' , 'last(t2.q_float_null)' ,'last(t2.q_double_null)' , 'last(t2.q_binary_null)' ,'last(t2.q_nchar_null))' ,'last(t2.q_bool_null)' ,'last(t2.q_ts_null)'] + + self.calc_select_in_support_ts_j = ['bottom(t1.q_int,20)' , 'bottom(t1.q_bigint,20)' , 'bottom(t1.q_smallint,20)' , 'bottom(t1.q_tinyint,20)' ,'bottom(t1.q_float,20)' , 'bottom(t1.q_double,20)' , + 'top(t1.q_int,20)' , 'top(t1.q_bigint,20)' , 'top(t1.q_smallint,20)' ,'top(t1.q_tinyint,20)' ,'top(t1.q_float,20)' ,'top(t1.q_double,20)' , + 'min(t1.q_int)' , 'min(t1.q_bigint)' , 'min(t1.q_smallint)' , 'min(t1.q_tinyint)' , 'min(t1.q_float)' ,'min(t1.q_double)' , + 'max(t1.q_int)' , 'max(t1.q_bigint)' , 'max(t1.q_smallint)' , 'max(t1.q_tinyint)' ,'max(t1.q_float)' ,'max(t1.q_double)' , + 'bottom(t2.q_int,20)' , 'bottom(t2.q_bigint,20)' , 'bottom(t2.q_smallint,20)' , 'bottom(t2.q_tinyint,20)' ,'bottom(t2.q_float,20)' , 'bottom(t2.q_double,20)' , + 'top(t2.q_int,20)' , 'top(t2.q_bigint,20)' , 'top(t2.q_smallint,20)' ,'top(t2.q_tinyint,20)' ,'top(t2.q_float,20)' ,'top(t2.q_double,20)' , + 'min(t2.q_int)' , 'min(t2.q_bigint)' , 'min(t2.q_smallint)' , 'min(t2.q_tinyint)' , 'min(t2.q_float)' ,'min(t2.q_double)' , + 'max(t2.q_int)' , 'max(t2.q_bigint)' , 'max(t2.q_smallint)' , 'max(t2.q_tinyint)' ,'max(t2.q_float)' ,'max(t2.q_double)' , + 'bottom(t1.q_int_null,20)' , 'bottom(t1.q_bigint_null,20)' , 'bottom(t1.q_smallint_null,20)' , 'bottom(t1.q_tinyint_null,20)' ,'bottom(t1.q_float_null,20)' , 'bottom(t1.q_double_null,20)' , + 'top(t1.q_int_null,20)' , 'top(t1.q_bigint_null,20)' , 'top(t1.q_smallint_null,20)' ,'top(t1.q_tinyint_null,20)' ,'top(t1.q_float_null,20)' ,'top(t1.q_double_null,20)' , + 'bottom(t2.q_int_null,20)' , 'bottom(t2.q_bigint_null,20)' , 'bottom(t2.q_smallint_null,20)' , 'bottom(t2.q_tinyint_null,20)' ,'bottom(t2.q_float_null,20)' , 'bottom(t2.q_double_null,20)' , + 'top(t2.q_int_null,20)' , 'top(t2.q_bigint_null,20)' , 'top(t2.q_smallint_null,20)' ,'top(t2.q_tinyint_null,20)' ,'top(t2.q_float_null,20)' ,'top(t2.q_double_null,20)' , + 'min(t1.q_int_null)' , 'min(t1.q_bigint_null)' , 'min(t1.q_smallint_null)' , 'min(t1.q_tinyint_null)' , 'min(t1.q_float_null)' ,'min(t1.q_double_null)' , + 'max(t1.q_int_null)' , 'max(t1.q_bigint_null)' , 'max(t1.q_smallint_null)' , 'max(t1.q_tinyint_null)' ,'max(t1.q_float_null)' ,'max(t1.q_double_null)' , + 'min(t2.q_int_null)' , 'min(t2.q_bigint_null)' , 'min(t2.q_smallint_null)' , 'min(t2.q_tinyint_null)' , 'min(t2.q_float_null)' ,'min(t2.q_double_null)' , + 'max(t2.q_int_null)' , 'max(t2.q_bigint_null)' , 'max(t2.q_smallint_null)' , 'max(t2.q_tinyint_null)' ,'max(t2.q_float_null)' ,'max(t2.q_double_null)' ] + + self.calc_select_in_not_support_ts_j = ['apercentile(t1.q_int,20)' , 'apercentile(t1.q_bigint,20)' ,'apercentile(t1.q_smallint,20)' ,'apercentile(t1.q_tinyint,20)' ,'apercentile(t1.q_float,20)' ,'apercentile(t1.q_double,20)' , + 'apercentile(t1.q_int_null,20)' , 'apercentile(t1.q_bigint_null,20)' ,'apercentile(t1.q_smallint_null,20)' ,'apercentile(t1.q_tinyint_null,20)' ,'apercentile(t1.q_float_null,20)' ,'apercentile(t1.q_double_null,20)' , + 'apercentile(t2.q_int,20)' , 'apercentile(t2.q_bigint,20)' ,'apercentile(t2.q_smallint,20)' ,'apercentile(t2.q_tinyint,20)' ,'apercentile(t2.q_float,20)' ,'apercentile(t2.q_double,20)' , + 'apercentile(t2.q_int_null,20)' , 'apercentile(t2.q_bigint_null,20)' ,'apercentile(t2.q_smallint_null,20)' ,'apercentile(t2.q_tinyint_null,20)' ,'apercentile(t2.q_float_null,20)' ,'apercentile(t2.q_double_null,20)' ] + + self.calc_select_in_j = ['min(t1.q_int)' , 'min(t1.q_bigint)' , 'min(t1.q_smallint)' , 'min(t1.q_tinyint)' , 'min(t1.q_float)' ,'min(t1.q_double)' , + 'max(t1.q_int)' , 'max(t1.q_bigint)' , 'max(t1.q_smallint)' , 'max(t1.q_tinyint)' ,'max(t1.q_float)' ,'max(t1.q_double)' , + 'apercentile(t1.q_int,20)' , 'apercentile(t1.q_bigint,20)' ,'apercentile(t1.q_smallint,20)' ,'apercentile(t1.q_tinyint,20)' ,'apercentile(t1.q_float,20)' ,'apercentile(t1.q_double,20)' , + 'min(t1.q_int_null)' , 'min(t1.q_bigint_null)' , 'min(t1.q_smallint_null)' , 'min(t1.q_tinyint_null)' , 'min(t1.q_float_null)' ,'min(t1.q_double_null)' , + 'max(t1.q_int_null)' , 'max(t1.q_bigint_null)' , 'max(t1.q_smallint_null)' , 'max(t1.q_tinyint_null)' ,'max(t1.q_float_null)' ,'max(t1.q_double_null)' , + 'apercentile(t1.q_int_null,20)' , 'apercentile(t1.q_bigint_null,20)' ,'apercentile(t1.q_smallint_null,20)' ,'apercentile(t1.q_tinyint_null,20)' ,'apercentile(t1.q_float_null,20)' ,'apercentile(t1.q_double_null,20)' , + 'last_row(t1.q_int)' , 'last_row(t1.q_bigint)' , 'last_row(t1.q_smallint)' , 'last_row(t1.q_tinyint)' , 'last_row(t1.q_float)' , + 'last_row(t1.q_double)' , 'last_row(t1.q_bool)' ,'last_row(t1.q_binary)' ,'last_row(t1.q_nchar)' ,'last_row(t1.q_ts)' , + 'min(t2.q_int)' , 'min(t2.q_bigint)' , 'min(t2.q_smallint)' , 'min(t2.q_tinyint)' , 'min(t2.q_float)' ,'min(t2.q_double)' , + 'max(t2.q_int)' , 'max(t2.q_bigint)' , 'max(t2.q_smallint)' , 'max(t2.q_tinyint)' ,'max(t2.q_float)' ,'max(t2.q_double)' , + 'last_row(t1.q_int_null)' , 'last_row(t1.q_bigint_null)' , 'last_row(t1.q_smallint_null)' , 'last_row(t1.q_tinyint_null)' , 'last_row(t1.q_float_null)' , + 'last_row(t1.q_double_null)' , 'last_row(t1.q_bool_null)' ,'last_row(t1.q_binary_null)' ,'last_row(t1.q_nchar_null)' ,'last_row(t1.q_ts_null)' , + 'min(t2.q_int_null)' , 'min(t2.q_bigint_null)' , 'min(t2.q_smallint_null)' , 'min(t2.q_tinyint_null)' , 'min(t2.q_float_null)' ,'min(t2.q_double_null)' , + 'max(t2.q_int_null)' , 'max(t2.q_bigint_null)' , 'max(t2.q_smallint_null)' , 'max(t2.q_tinyint_null)' ,'max(t2.q_float_null)' ,'max(t2.q_double_null)' , + 'apercentile(t2.q_int,20)' , 'apercentile(t2.q_bigint,20)' ,'apercentile(t2.q_smallint,20)' ,'apercentile(t2.q_tinyint,20)' ,'apercentile(t2.q_float,20)' ,'apercentile(t2.q_double,20)' , + 'apercentile(t2.q_int_null,20)' , 'apercentile(t2.q_bigint_null,20)' ,'apercentile(t2.q_smallint_null,20)' ,'apercentile(t2.q_tinyint_null,20)' ,'apercentile(t2.q_float_null,20)' ,'apercentile(t2.q_double_null,20)' , + 'last_row(t2.q_int)' , 'last_row(t2.q_bigint)' , 'last_row(t2.q_smallint)' , 'last_row(t2.q_tinyint)' , 'last_row(t2.q_float)' , + 'last_row(t2.q_double)' , 'last_row(t2.q_bool)' ,'last_row(t2.q_binary)' ,'last_row(t2.q_nchar)' ,'last_row(t2.q_ts)', + 'last_row(t2.q_int_null)' , 'last_row(t2.q_bigint_null)' , 'last_row(t2.q_smallint_null)' , 'last_row(t2.q_tinyint_null)' , 'last_row(t2.q_float_null)' , + 'last_row(t2.q_double_null)' , 'last_row(t2.q_bool_null)' ,'last_row(t2.q_binary_null)' ,'last_row(t2.q_nchar_null)' ,'last_row(t2.q_ts_null)'] + self.calc_select_all_j = self.calc_select_in_ts_j + self.calc_select_in_j + + self.calc_select_regular_j = [ 'PERCENTILE(t1.q_int,10)' ,'PERCENTILE(t1.q_bigint,20)' , 'PERCENTILE(t1.q_smallint,30)' ,'PERCENTILE(t1.q_tinyint,40)' ,'PERCENTILE(t1.q_float,50)' ,'PERCENTILE(t1.q_double,60)' , + 'PERCENTILE(t2.q_int,10)' ,'PERCENTILE(t2.q_bigint,20)' , 'PERCENTILE(t2.q_smallint,30)' ,'PERCENTILE(t2.q_tinyint,40)' ,'PERCENTILE(t2.q_float,50)' ,'PERCENTILE(t2.q_double,60)', + 'PERCENTILE(t1.q_int_null,10)' ,'PERCENTILE(t1.q_bigint_null,20)' , 'PERCENTILE(t1.q_smallint_null,30)' ,'PERCENTILE(t1.q_tinyint_null,40)' ,'PERCENTILE(t1.q_float_null,50)' ,'PERCENTILE(t1.q_double_null,60)' , + 'PERCENTILE(t2.q_int_null,10)' ,'PERCENTILE(t2.q_bigint_null,20)' , 'PERCENTILE(t2.q_smallint_null,30)' ,'PERCENTILE(t2.q_tinyint_null,40)' ,'PERCENTILE(t2.q_float_null,50)' ,'PERCENTILE(t2.q_double_null,60)'] + + + self.calc_select_fill_j = ['INTERP(t1.q_int)' ,'INTERP(t1.q_bigint)' ,'INTERP(t1.q_smallint)' ,'INTERP(t1.q_tinyint)', 'INTERP(t1.q_float)' ,'INTERP(t1.q_double)' , + 'INTERP(t2.q_int)' ,'INTERP(t2.q_bigint)' ,'INTERP(t2.q_smallint)' ,'INTERP(t2.q_tinyint)', 'INTERP(t2.q_float)' ,'INTERP(t2.q_double)'] + self.interp_where_j = ['t1.ts = now' , 't1.ts = \'2020-09-13 20:26:40.000\'' , 't1.ts = \'2020-09-13 20:26:40.009\'' ,'t2.ts = now' , 't2.ts = \'2020-09-13 20:26:40.000\'' , 't2.ts = \'2020-09-13 20:26:40.009\'' , + 't1.tbname in (\'table_1\') and t1.ts = now' ,'t1.tbname in (\'table_0\' ,\'table_1\',\'table_2\',\'table_3\',\'table_4\',\'table_5\') and t1.ts = \'2020-09-13 20:26:40.000\'','t1.tbname like \'table%\' and t1.ts = \'2020-09-13 20:26:40.002\'', + 't2.tbname in (\'table_1\') and t2.ts = now' ,'t2.tbname in (\'table_0\' ,\'table_1\',\'table_2\',\'table_3\',\'table_4\',\'table_5\') and t2.ts = \'2020-09-13 20:26:40.000\'','t2.tbname like \'table%\' and t2.ts = \'2020-09-13 20:26:40.002\''] + + # calc_aggregate_all calc_aggregate_regular calc_aggregate_groupbytbname APERCENTILE\PERCENTILE + # aggregate function include [all:count(*)\avg\sum\stddev ||regualr:twa\irate\leastsquares ||group by tbname:twa\irate\] + self.calc_aggregate_all = ['count(*)' , 'count(q_int)' ,'count(q_bigint)' , 'count(q_smallint)' ,'count(q_tinyint)' ,'count(q_float)' , + 'count(q_double)' ,'count(q_binary)' ,'count(q_nchar)' ,'count(q_bool)' ,'count(q_ts)' , + 'avg(q_int)' ,'avg(q_bigint)' , 'avg(q_smallint)' ,'avg(q_tinyint)' ,'avg(q_float)' ,'avg(q_double)' , + 'sum(q_int)' ,'sum(q_bigint)' , 'sum(q_smallint)' ,'sum(q_tinyint)' ,'sum(q_float)' ,'sum(q_double)' , + 'STDDEV(q_int)' ,'STDDEV(q_bigint)' , 'STDDEV(q_smallint)' ,'STDDEV(q_tinyint)' ,'STDDEV(q_float)' ,'STDDEV(q_double)', + 'APERCENTILE(q_int,10)' ,'APERCENTILE(q_bigint,20)' , 'APERCENTILE(q_smallint,30)' ,'APERCENTILE(q_tinyint,40)' ,'APERCENTILE(q_float,50)' ,'APERCENTILE(q_double,60)', + 'count(q_int_null)' ,'count(q_bigint_null)' , 'count(q_smallint_null)' ,'count(q_tinyint_null)' ,'count(q_float_null)' , + 'count(q_double_null)' ,'count(q_binary_null)' ,'count(q_nchar_null)' ,'count(q_bool_null)' ,'count(q_ts_null)' , + 'avg(q_int_null)' ,'avg(q_bigint_null)' , 'avg(q_smallint_null)' ,'avg(q_tinyint_null)' ,'avg(q_float_null)' ,'avg(q_double_null)' , + 'sum(q_int_null)' ,'sum(q_bigint_null)' , 'sum(q_smallint_null)' ,'sum(q_tinyint_null)' ,'sum(q_float_null)' ,'sum(q_double_null)' , + 'STDDEV(q_int_null)' ,'STDDEV(q_bigint_null)' , 'STDDEV(q_smallint_null)' ,'STDDEV(q_tinyint_null)' ,'STDDEV(q_float_null)' ,'STDDEV(q_double_null)', + 'APERCENTILE(q_int_null,10)' ,'APERCENTILE(q_bigint_null,20)' , 'APERCENTILE(q_smallint_null,30)' ,'APERCENTILE(q_tinyint_null,40)' ,'APERCENTILE(q_float_null,50)' ,'APERCENTILE(q_double_null,60)'] + + self.calc_aggregate_regular = ['twa(q_int)' ,'twa(q_bigint)' , 'twa(q_smallint)' ,'twa(q_tinyint)' ,'twa (q_float)' ,'twa(q_double)' , + 'IRATE(q_int)' ,'IRATE(q_bigint)' , 'IRATE(q_smallint)' ,'IRATE(q_tinyint)' ,'IRATE (q_float)' ,'IRATE(q_double)' , + 'twa(q_int_null)' ,'twa(q_bigint_null)' , 'twa(q_smallint_null)' ,'twa(q_tinyint_null)' ,'twa (q_float_null)' ,'twa(q_double_null)' , + 'IRATE(q_int_null)' ,'IRATE(q_bigint_null)' , 'IRATE(q_smallint_null)' ,'IRATE(q_tinyint_null)' ,'IRATE (q_float_null)' ,'IRATE(q_double_null)' , + 'LEASTSQUARES(q_int,15,3)' , 'LEASTSQUARES(q_bigint,10,1)' , 'LEASTSQUARES(q_smallint,20,3)' ,'LEASTSQUARES(q_tinyint,10,4)' ,'LEASTSQUARES(q_float,6,4)' ,'LEASTSQUARES(q_double,3,1)' , + 'PERCENTILE(q_int,10)' ,'PERCENTILE(q_bigint,20)' , 'PERCENTILE(q_smallint,30)' ,'PERCENTILE(q_tinyint,40)' ,'PERCENTILE(q_float,50)' ,'PERCENTILE(q_double,60)', + 'LEASTSQUARES(q_int_null,15,3)' , 'LEASTSQUARES(q_bigint_null,10,1)' , 'LEASTSQUARES(q_smallint_null,20,3)' ,'LEASTSQUARES(q_tinyint_null,10,4)' ,'LEASTSQUARES(q_float_null,6,4)' ,'LEASTSQUARES(q_double_null,3,1)' , + 'PERCENTILE(q_int_null,10)' ,'PERCENTILE(q_bigint_null,20)' , 'PERCENTILE(q_smallint_null,30)' ,'PERCENTILE(q_tinyint_null,40)' ,'PERCENTILE(q_float_null,50)' ,'PERCENTILE(q_double_null,60)'] + + self.calc_aggregate_groupbytbname = ['twa(q_int)' ,'twa(q_bigint)' , 'twa(q_smallint)' ,'twa(q_tinyint)' ,'twa (q_float)' ,'twa(q_double)' , + 'IRATE(q_int)' ,'IRATE(q_bigint)' , 'IRATE(q_smallint)' ,'IRATE(q_tinyint)' ,'IRATE (q_float)' ,'IRATE(q_double)', + 'twa(q_int_null)' ,'twa(q_bigint_null)' , 'twa(q_smallint_null)' ,'twa(q_tinyint_null)' ,'twa (q_float_null)' ,'twa(q_double_null)' , + 'IRATE(q_int_null)' ,'IRATE(q_bigint_null)' , 'IRATE(q_smallint_null)' ,'IRATE(q_tinyint_null)' ,'IRATE (q_float_null)' ,'IRATE(q_double_null)'] + + #two table join + self.calc_aggregate_all_j = ['count(t1.*)' , 'count(t1.q_int)' ,'count(t1.q_bigint)' , 'count(t1.q_smallint)' ,'count(t1.q_tinyint)' ,'count(t1.q_float)' , + 'count(t1.q_double)' ,'count(t1.q_binary)' ,'count(t1.q_nchar)' ,'count(t1.q_bool)' ,'count(t1.q_ts)' , + 'avg(t1.q_int)' ,'avg(t1.q_bigint)' , 'avg(t1.q_smallint)' ,'avg(t1.q_tinyint)' ,'avg(t1.q_float)' ,'avg(t1.q_double)' , + 'sum(t1.q_int)' ,'sum(t1.q_bigint)' , 'sum(t1.q_smallint)' ,'sum(t1.q_tinyint)' ,'sum(t1.q_float)' ,'sum(t1.q_double)' , + 'STDDEV(t1.q_int)' ,'STDDEV(t1.q_bigint)' , 'STDDEV(t1.q_smallint)' ,'STDDEV(t1.q_tinyint)' ,'STDDEV(t1.q_float)' ,'STDDEV(t1.q_double)', + 'APERCENTILE(t1.q_int,10)' ,'APERCENTILE(t1.q_bigint,20)' , 'APERCENTILE(t1.q_smallint,30)' ,'APERCENTILE(t1.q_tinyint,40)' ,'APERCENTILE(t1.q_float,50)' ,'APERCENTILE(t1.q_double,60)' , + 'count(t1.q_int_null)' ,'count(t1.q_bigint_null)' , 'count(t1.q_smallint_null)' ,'count(t1.q_tinyint_null)' ,'count(t1.q_float_null)' , + 'count(t1.q_double_null)' ,'count(t1.q_binary_null)' ,'count(t1.q_nchar_null)' ,'count(t1.q_bool_null)' ,'count(t1.q_ts_null)' , + 'avg(t1.q_int_null)' ,'avg(t1.q_bigint_null)' , 'avg(t1.q_smallint_null)' ,'avg(t1.q_tinyint_null)' ,'avg(t1.q_float_null)' ,'avg(t1.q_double_null)' , + 'sum(t1.q_int_null)' ,'sum(t1.q_bigint_null)' , 'sum(t1.q_smallint_null)' ,'sum(t1.q_tinyint_null)' ,'sum(t1.q_float_null)' ,'sum(t1.q_double_null)' , + 'STDDEV(t1.q_int_null)' ,'STDDEV(t1.q_bigint_null)' , 'STDDEV(t1.q_smallint_null)' ,'STDDEV(t1.q_tinyint_null)' ,'STDDEV(t1.q_float_null)' ,'STDDEV(t1.q_double_null)', + 'APERCENTILE(t1.q_int_null,10)' ,'APERCENTILE(t1.q_bigint_null,20)' , 'APERCENTILE(t1.q_smallint_null,30)' ,'APERCENTILE(t1.q_tinyint_null,40)' ,'APERCENTILE(t1.q_float_null,50)' ,'APERCENTILE(t1.q_double,60)' , + 'count(t2.*)' , 'count(t2.q_int)' ,'count(t2.q_bigint)' , 'count(t2.q_smallint)' ,'count(t2.q_tinyint)' ,'count(t2.q_float)' , + 'count(t2.q_double)' ,'count(t2.q_binary)' ,'count(t2.q_nchar)' ,'count(t2.q_bool)' ,'count(t2.q_ts)' , + 'avg(t2.q_int)' ,'avg(t2.q_bigint)' , 'avg(t2.q_smallint)' ,'avg(t2.q_tinyint)' ,'avg(t2.q_float)' ,'avg(t2.q_double)' , + 'sum(t2.q_int)' ,'sum(t2.q_bigint)' , 'sum(t2.q_smallint)' ,'sum(t2.q_tinyint)' ,'sum(t2.q_float)' ,'sum(t2.q_double)' , + 'STDDEV(t2.q_int)' ,'STDDEV(t2.q_bigint)' , 'STDDEV(t2.q_smallint)' ,'STDDEV(t2.q_tinyint)' ,'STDDEV(t2.q_float)' ,'STDDEV(t2.q_double)', + 'APERCENTILE(t2.q_int,10)' ,'APERCENTILE(t2.q_bigint,20)' , 'APERCENTILE(t2.q_smallint,30)' ,'APERCENTILE(t2.q_tinyint,40)' ,'APERCENTILE(t2.q_float,50)' ,'APERCENTILE(t2.q_double,60)', + 'count(t2.q_int_null)' ,'count(t2.q_bigint_null)' , 'count(t2.q_smallint_null)' ,'count(t2.q_tinyint_null)' ,'count(t2.q_float_null)' , + 'count(t2.q_double_null)' ,'count(t2.q_binary_null)' ,'count(t2.q_nchar_null)' ,'count(t2.q_bool_null)' ,'count(t2.q_ts_null)' , + 'avg(t2.q_int_null)' ,'avg(t2.q_bigint_null)' , 'avg(t2.q_smallint_null)' ,'avg(t2.q_tinyint_null)' ,'avg(t2.q_float_null)' ,'avg(t2.q_double_null)' , + 'sum(t2.q_int_null)' ,'sum(t2.q_bigint_null)' , 'sum(t2.q_smallint_null)' ,'sum(t2.q_tinyint_null)' ,'sum(t2.q_float_null)' ,'sum(t2.q_double_null)' , + 'STDDEV(t2.q_int_null)' ,'STDDEV(t2.q_bigint_null)' , 'STDDEV(t2.q_smallint_null)' ,'STDDEV(t2.q_tinyint_null)' ,'STDDEV(t2.q_float_null)' ,'STDDEV(t2.q_double_null)', + 'APERCENTILE(t2.q_int_null,10)' ,'APERCENTILE(t2.q_bigint_null,20)' , 'APERCENTILE(t2.q_smallint_null,30)' ,'APERCENTILE(t2.q_tinyint_null,40)' ,'APERCENTILE(t2.q_float_null,50)' ,'APERCENTILE(t2.q_double,60)'] + + self.calc_aggregate_regular_j = ['twa(t1.q_int)' ,'twa(t1.q_bigint)' , 'twa(t1.q_smallint)' ,'twa(t1.q_tinyint)' ,'twa (t1.q_float)' ,'twa(t1.q_double)' , + 'IRATE(t1.q_int)' ,'IRATE(t1.q_bigint)' , 'IRATE(t1.q_smallint)' ,'IRATE(t1.q_tinyint)' ,'IRATE (t1.q_float)' ,'IRATE(t1.q_double)' , + 'LEASTSQUARES(t1.q_int,15,3)' , 'LEASTSQUARES(t1.q_bigint,10,1)' , 'LEASTSQUARES(t1.q_smallint,20,3)' ,'LEASTSQUARES(t1.q_tinyint,10,4)' ,'LEASTSQUARES(t1.q_float,6,4)' ,'LEASTSQUARES(t1.q_double,3,1)' , + 'twa(t2.q_int)' ,'twa(t2.q_bigint)' , 'twa(t2.q_smallint)' ,'twa(t2.q_tinyint)' ,'twa (t2.q_float)' ,'twa(t2.q_double)' , + 'IRATE(t2.q_int)' ,'IRATE(t2.q_bigint)' , 'IRATE(t2.q_smallint)' ,'IRATE(t2.q_tinyint)' ,'IRATE (t2.q_float)' ,'IRATE(t2.q_double)', + 'LEASTSQUARES(t2.q_int,15,3)' , 'LEASTSQUARES(t2.q_bigint,10,1)' , 'LEASTSQUARES(t2.q_smallint,20,3)' ,'LEASTSQUARES(t2.q_tinyint,10,4)' ,'LEASTSQUARES(t2.q_float,6,4)' ,'LEASTSQUARES(t2.q_double,3,1)' , + 'twa(t1.q_int_null)' ,'twa(t1.q_bigint_null)' , 'twa(t1.q_smallint_null)' ,'twa(t1.q_tinyint_null)' ,'twa (t1.q_float_null)' ,'twa(t1.q_double_null)' , + 'IRATE(t1.q_int_null)' ,'IRATE(t1.q_bigint_null)' , 'IRATE(t1.q_smallint_null)' ,'IRATE(t1.q_tinyint_null)' ,'IRATE (t1.q_float_null)' ,'IRATE(t1.q_double_null)' , + 'LEASTSQUARES(t1.q_int_null,15,3)' , 'LEASTSQUARES(t1.q_bigint_null,10,1)' , 'LEASTSQUARES(t1.q_smallint_null,20,3)' ,'LEASTSQUARES(t1.q_tinyint_null,10,4)' ,'LEASTSQUARES(t1.q_float_null,6,4)' ,'LEASTSQUARES(t1.q_double_null,3,1)' , + 'twa(t2.q_int_null)' ,'twa(t2.q_bigint_null)' , 'twa(t2.q_smallint_null)' ,'twa(t2.q_tinyint_null)' ,'twa (t2.q_float_null)' ,'twa(t2.q_double_null)' , + 'IRATE(t2.q_int_null)' ,'IRATE(t2.q_bigint_null)' , 'IRATE(t2.q_smallint_null)' ,'IRATE(t2.q_tinyint_null)' ,'IRATE (t2.q_float_null)' ,'IRATE(t2.q_double_null)', + 'LEASTSQUARES(t2.q_int_null,15,3)' , 'LEASTSQUARES(t2.q_bigint_null,10,1)' , 'LEASTSQUARES(t2.q_smallint_null,20,3)' ,'LEASTSQUARES(t2.q_tinyint_null,10,4)' ,'LEASTSQUARES(t2.q_float_null,6,4)' ,'LEASTSQUARES(t2.q_double_null,3,1)' ] + + self.calc_aggregate_groupbytbname_j = ['twa(t1.q_int)' ,'twa(t1.q_bigint)' , 'twa(t1.q_smallint)' ,'twa(t1.q_tinyint)' ,'twa (t1.q_float)' ,'twa(t1.q_double)' , + 'IRATE(t1.q_int)' ,'IRATE(t1.q_bigint)' , 'IRATE(t1.q_smallint)' ,'IRATE(t1.q_tinyint)' ,'IRATE (t1.q_float)' ,'IRATE(t1.q_double)' , + 'twa(t2.q_int)' ,'twa(t2.q_bigint)' , 'twa(t2.q_smallint)' ,'twa(t2.q_tinyint)' ,'twa (t2.q_float)' ,'twa(t2.q_double)' , + 'IRATE(t2.q_int)' ,'IRATE(t2.q_bigint)' , 'IRATE(t2.q_smallint)' ,'IRATE(t2.q_tinyint)' ,'IRATE (t2.q_float)' ,'IRATE(t2.q_double)' , + 'twa(t1.q_int_null)' ,'twa(t1.q_bigint_null)' , 'twa(t1.q_smallint_null)' ,'twa(t1.q_tinyint_null)' ,'twa (t1.q_float_null)' ,'twa(t1.q_double_null)' , + 'IRATE(t1.q_int_null)' ,'IRATE(t1.q_bigint_null)' , 'IRATE(t1.q_smallint_null)' ,'IRATE(t1.q_tinyint_null)' ,'IRATE (t1.q_float_null)' ,'IRATE(t1.q_double_null)' , + 'twa(t2.q_int_null)' ,'twa(t2.q_bigint_null)' , 'twa(t2.q_smallint_null)' ,'twa(t2.q_tinyint_null)' ,'twa (t2.q_float_null)' ,'twa(t2.q_double_null)' , + 'IRATE(t2.q_int_null)' ,'IRATE(t2.q_bigint_null)' , 'IRATE(t2.q_smallint_null)' ,'IRATE(t2.q_tinyint_null)' ,'IRATE (t2.q_float_null)' ,'IRATE(t2.q_double_null)' ] + + # calc_calculate_all calc_calculate_regular calc_calculate_groupbytbname + # calculation function include [all:spread\+-*/ ||regualr:diff\derivative ||group by tbname:diff\derivative\] + self.calc_calculate_all = ['SPREAD(ts)' , 'SPREAD(q_ts)' , 'SPREAD(q_int)' ,'SPREAD(q_bigint)' , 'SPREAD(q_smallint)' ,'SPREAD(q_tinyint)' ,'SPREAD(q_float)' ,'SPREAD(q_double)' , + '(SPREAD(q_int) + SPREAD(q_bigint))' , '(SPREAD(q_smallint) - SPREAD(q_float))', '(SPREAD(q_double) * SPREAD(q_tinyint))' , '(SPREAD(q_double) / SPREAD(q_float))', + 'SPREAD(q_ts_null)' , 'SPREAD(q_int_null)' ,'SPREAD(q_bigint_null)' , 'SPREAD(q_smallint_null)' ,'SPREAD(q_tinyint_null)' ,'SPREAD(q_float_null)' ,'SPREAD(q_double_null)' , + '(SPREAD(q_int_null) + SPREAD(q_bigint_null))' , '(SPREAD(q_smallint_null) - SPREAD(q_float_null))', '(SPREAD(q_double_null) * SPREAD(q_tinyint_null))' , '(SPREAD(q_double_null) / SPREAD(q_float_null))'] + self.calc_calculate_regular = ['DIFF(q_int)' ,'DIFF(q_bigint)' , 'DIFF(q_smallint)' ,'DIFF(q_tinyint)' ,'DIFF(q_float)' ,'DIFF(q_double)' , + 'DIFF(q_int,0)' ,'DIFF(q_bigint,0)' , 'DIFF(q_smallint,0)' ,'DIFF(q_tinyint,0)' ,'DIFF(q_float,0)' ,'DIFF(q_double,0)' , + 'DIFF(q_int,1)' ,'DIFF(q_bigint,1)' , 'DIFF(q_smallint,1)' ,'DIFF(q_tinyint,1)' ,'DIFF(q_float,1)' ,'DIFF(q_double,1)' , + 'DERIVATIVE(q_int,15s,0)' , 'DERIVATIVE(q_bigint,10s,1)' , 'DERIVATIVE(q_smallint,20s,0)' ,'DERIVATIVE(q_tinyint,10s,1)' ,'DERIVATIVE(q_float,6s,0)' ,'DERIVATIVE(q_double,3s,1)', + 'DIFF(q_int_null)' ,'DIFF(q_bigint_null)' , 'DIFF(q_smallint_null)' ,'DIFF(q_tinyint_null)' ,'DIFF(q_float_null)' ,'DIFF(q_double_null)' , + 'DIFF(q_int_null,0)' ,'DIFF(q_bigint_null,0)' , 'DIFF(q_smallint_null,0)' ,'DIFF(q_tinyint_null,0)' ,'DIFF(q_float_null,0)' ,'DIFF(q_double_null,0)' , + 'DIFF(q_int_null,1)' ,'DIFF(q_bigint_null,1)' , 'DIFF(q_smallint_null,1)' ,'DIFF(q_tinyint_null,1)' ,'DIFF(q_float_null,1)' ,'DIFF(q_double_null,1)' , + 'DERIVATIVE(q_int_null,15s,0)' , 'DERIVATIVE(q_bigint_null,10s,1)' , 'DERIVATIVE(q_smallint_null,20s,0)' ,'DERIVATIVE(q_tinyint_null,10s,1)' ,'DERIVATIVE(q_float_null,6s,0)' ,'DERIVATIVE(q_double_null,3s,1)'] + self.calc_calculate_groupbytbname = self.calc_calculate_regular + + #two table join + self.calc_calculate_all_j = ['SPREAD(t1.ts)' , 'SPREAD(t1.q_ts)' , 'SPREAD(t1.q_int)' ,'SPREAD(t1.q_bigint)' , 'SPREAD(t1.q_smallint)' ,'SPREAD(t1.q_tinyint)' ,'SPREAD(t1.q_float)' ,'SPREAD(t1.q_double)' , + 'SPREAD(t2.ts)' , 'SPREAD(t2.q_ts)' , 'SPREAD(t2.q_int)' ,'SPREAD(t2.q_bigint)' , 'SPREAD(t2.q_smallint)' ,'SPREAD(t2.q_tinyint)' ,'SPREAD(t2.q_float)' ,'SPREAD(t2.q_double)' , + '(SPREAD(t1.q_int) + SPREAD(t1.q_bigint))' , '(SPREAD(t1.q_tinyint) - SPREAD(t1.q_float))', '(SPREAD(t1.q_double) * SPREAD(t1.q_tinyint))' , '(SPREAD(t1.q_double) / SPREAD(t1.q_tinyint))', + '(SPREAD(t2.q_int) + SPREAD(t2.q_bigint))' , '(SPREAD(t2.q_smallint) - SPREAD(t2.q_float))', '(SPREAD(t2.q_double) * SPREAD(t2.q_tinyint))' , '(SPREAD(t2.q_double) / SPREAD(t2.q_tinyint))', + '(SPREAD(t1.q_int) + SPREAD(t1.q_smallint))' , '(SPREAD(t2.q_smallint) - SPREAD(t2.q_float))', '(SPREAD(t1.q_double) * SPREAD(t1.q_tinyint))' , '(SPREAD(t1.q_double) / SPREAD(t1.q_float))', + 'SPREAD(t1.q_ts_null)' , 'SPREAD(t1.q_int_null)' ,'SPREAD(t1.q_bigint_null)' , 'SPREAD(t1.q_smallint_null)' ,'SPREAD(t1.q_tinyint_null)' ,'SPREAD(t1.q_float_null)' ,'SPREAD(t1.q_double_null)' , + 'SPREAD(t2.q_ts_null)' , 'SPREAD(t2.q_int_null)' ,'SPREAD(t2.q_bigint_null)' , 'SPREAD(t2.q_smallint_null)' ,'SPREAD(t2.q_tinyint_null)' ,'SPREAD(t2.q_float_null)' ,'SPREAD(t2.q_double_null)' , + '(SPREAD(t1.q_int_null) + SPREAD(t1.q_bigint_null))' , '(SPREAD(t1.q_tinyint_null) - SPREAD(t1.q_float_null))', '(SPREAD(t1.q_double_null) * SPREAD(t1.q_tinyint_null))' , '(SPREAD(t1.q_double_null) / SPREAD(t1.q_tinyint_null))', + '(SPREAD(t2.q_int_null) + SPREAD(t2.q_bigint_null))' , '(SPREAD(t2.q_smallint_null) - SPREAD(t2.q_float_null))', '(SPREAD(t2.q_double_null) * SPREAD(t2.q_tinyint_null))' , '(SPREAD(t2.q_double_null) / SPREAD(t2.q_tinyint_null))', + '(SPREAD(t1.q_int_null) + SPREAD(t1.q_smallint_null))' , '(SPREAD(t2.q_smallint_null) - SPREAD(t2.q_float_null))', '(SPREAD(t1.q_double_null) * SPREAD(t1.q_tinyint_null))' , '(SPREAD(t1.q_double_null) / SPREAD(t1.q_float_null))'] + self.calc_calculate_regular_j = ['DIFF(t1.q_int)' ,'DIFF(t1.q_bigint)' , 'DIFF(t1.q_smallint)' ,'DIFF(t1.q_tinyint)' ,'DIFF(t1.q_float)' ,'DIFF(t1.q_double)' , + 'DIFF(t1.q_int,0)' ,'DIFF(t1.q_bigint,0)' , 'DIFF(t1.q_smallint,0)' ,'DIFF(t1.q_tinyint,0)' ,'DIFF(t1.q_float,0)' ,'DIFF(t1.q_double,0)' , + 'DIFF(t1.q_int,1)' ,'DIFF(t1.q_bigint,1)' , 'DIFF(t1.q_smallint,1)' ,'DIFF(t1.q_tinyint,1)' ,'DIFF(t1.q_float,1)' ,'DIFF(t1.q_double,1)' , + 'DERIVATIVE(t1.q_int,15s,0)' , 'DERIVATIVE(t1.q_bigint,10s,1)' , 'DERIVATIVE(t1.q_smallint,20s,0)' ,'DERIVATIVE(t1.q_tinyint,10s,1)' ,'DERIVATIVE(t1.q_float,6s,0)' ,'DERIVATIVE(t1.q_double,3s,1)' , + 'DIFF(t2.q_int)' ,'DIFF(t2.q_bigint)' , 'DIFF(t2.q_smallint)' ,'DIFF(t2.q_tinyint)' ,'DIFF(t2.q_float)' ,'DIFF(t2.q_double)' , + 'DIFF(t2.q_int,0)' ,'DIFF(t2.q_bigint,0)' , 'DIFF(t2.q_smallint,0)' ,'DIFF(t2.q_tinyint,0)' ,'DIFF(t2.q_float,0)' ,'DIFF(t2.q_double,0)' , + 'DIFF(t2.q_int,1)' ,'DIFF(t2.q_bigint,1)' , 'DIFF(t2.q_smallint,1)' ,'DIFF(t2.q_tinyint,1)' ,'DIFF(t2.q_float,1)' ,'DIFF(t2.q_double,1)' , + 'DERIVATIVE(t2.q_int,15s,0)' , 'DERIVATIVE(t2.q_bigint,10s,1)' , 'DERIVATIVE(t2.q_smallint,20s,0)' ,'DERIVATIVE(t2.q_tinyint,10s,1)' ,'DERIVATIVE(t2.q_float,6s,0)' ,'DERIVATIVE(t2.q_double,3s,1)' , + 'DIFF(t1.q_int_null)' ,'DIFF(t1.q_bigint_null)' , 'DIFF(t1.q_smallint_null)' ,'DIFF(t1.q_tinyint_null)' ,'DIFF(t1.q_float_null)' ,'DIFF(t1.q_double_null)' , + 'DIFF(t1.q_int_null,0)' ,'DIFF(t1.q_bigint_null,0)' , 'DIFF(t1.q_smallint_null,0)' ,'DIFF(t1.q_tinyint_null,0)' ,'DIFF(t1.q_float_null,0)' ,'DIFF(t1.q_double_null,0)' , + 'DIFF(t1.q_int_null,1)' ,'DIFF(t1.q_bigint_null,1)' , 'DIFF(t1.q_smallint_null,1)' ,'DIFF(t1.q_tinyint_null,1)' ,'DIFF(t1.q_float_null,1)' ,'DIFF(t1.q_double_null,1)' , + 'DERIVATIVE(t1.q_int_null,15s,0)' , 'DERIVATIVE(t1.q_bigint_null,10s,1)' , 'DERIVATIVE(t1.q_smallint_null,20s,0)' ,'DERIVATIVE(t1.q_tinyint_null,10s,1)' ,'DERIVATIVE(t1.q_float_null,6s,0)' ,'DERIVATIVE(t1.q_double_null,3s,1)' , + 'DIFF(t2.q_int_null)' ,'DIFF(t2.q_bigint_null)' , 'DIFF(t2.q_smallint_null)' ,'DIFF(t2.q_tinyint_null)' ,'DIFF(t2.q_float_null)' ,'DIFF(t2.q_double_null)' , + 'DIFF(t2.q_int_null,0)' ,'DIFF(t2.q_bigint_null,0)' , 'DIFF(t2.q_smallint_null,0)' ,'DIFF(t2.q_tinyint_null,0)' ,'DIFF(t2.q_float_null,0)' ,'DIFF(t2.q_double_null,0)' , + 'DIFF(t2.q_int_null,1)' ,'DIFF(t2.q_bigint_null,1)' , 'DIFF(t2.q_smallint_null,1)' ,'DIFF(t2.q_tinyint_null,1)' ,'DIFF(t2.q_float_null,1)' ,'DIFF(t2.q_double_null,1)' , + 'DERIVATIVE(t2.q_int_null,15s,0)' , 'DERIVATIVE(t2.q_bigint_null,10s,1)' , 'DERIVATIVE(t2.q_smallint_null,20s,0)' ,'DERIVATIVE(t2.q_tinyint_null,10s,1)' ,'DERIVATIVE(t2.q_float_null,6s,0)' ,'DERIVATIVE(t2.q_double_null,3s,1)'] + self.calc_calculate_groupbytbname_j = self.calc_calculate_regular_j + + #inter && calc_aggregate_all\calc_aggregate_regular\calc_select_all + self.interval_sliding = ['interval(4w) sliding(1w) ','interval(1w) sliding(1d) ','interval(1d) sliding(1h) ' , + 'interval(1h) sliding(1m) ','interval(1m) sliding(1s) ','interval(1s) sliding(10a) ', + 'interval(1y) ','interval(1n) ','interval(1w) ','interval(1d) ','interval(1h) ','interval(1m) ','interval(1s) ' ,'interval(10a)', + 'interval(1y,1n) ','interval(1n,1w) ','interval(1w,1d) ','interval(1d,1h) ','interval(1h,1m) ','interval(1m,1s) ','interval(1s,10a) ' ,'interval(100a,30a)'] + + self.conn1 = taos.connect(host="127.0.0.1", user="root", password="taosdata", config="/etc/taos/") + self.cur1 = self.conn1.cursor() + print(self.cur1) + self.cur1.execute("use %s ;" %self.db_nest) + sql = 'select * from stable_1 limit 5;' + self.cur1.execute(sql) + + + def data_matrix_equal(self, sql1,row1_s,row1_e,col1_s,col1_e, sql2,row2_s,row2_e,col2_s,col2_e): + # ----row1_start----col1_start---- + # - - - - 是一个矩阵内的数据相等- - - + # - - - - - - - - - - - - - - - - + # ----row1_end------col1_end------ + self.sql1 = sql1 + list1 =[] + tdSql.query(sql1) + for i1 in range(row1_s-1,row1_e): + #print("iiii=%d"%i1) + for j1 in range(col1_s-1,col1_e): + #print("jjjj=%d"%j1) + #print("data=%s" %(tdSql.getData(i1,j1))) + list1.append(tdSql.getData(i1,j1)) + print("=====list1-------list1---=%s" %set(list1)) + + tdSql.execute("reset query cache;") + self.sql2 = sql2 + list2 =[] + tdSql.query(sql2) + for i2 in range(row2_s-1,row2_e): + #print("iiii222=%d"%i2) + for j2 in range(col2_s-1,col2_e): + #print("jjjj222=%d"%j2) + #print("data=%s" %(tdSql.getData(i2,j2))) + list2.append(tdSql.getData(i2,j2)) + print("=====list2-------list2---=%s" %set(list2)) + + if (list1 == list2) and len(list2)>0: + # print(("=====matrix===sql1.list1:'%s',sql2.list2:'%s'") %(list1,list2)) + tdLog.info(("===matrix===sql1:'%s' matrix_result = sql2:'%s' matrix_result") %(sql1,sql2)) + elif (set(list2)).issubset(set(list1)): + # 解决不同子表排列结果乱序 + # print(("=====list_issubset==matrix2in1-true===sql1.list1:'%s',sql2.list2:'%s'") %(list1,list2)) + tdLog.info(("===matrix_issubset===sql1:'%s' matrix_set_result = sql2:'%s' matrix_set_result") %(sql1,sql2)) + #elif abs(float(str(list1).replace("]","").replace("[","").replace("e+","")) - float(str(list2).replace("]","").replace("[","").replace("e+",""))) <= 0.0001: + elif abs(float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace("e+","").replace(", ","").replace("(","").replace(")","").replace("-","")) - float(str(list2).replace("datetime.datetime","").replace("]","").replace("[","").replace("e+","").replace(", ","").replace("(","").replace(")","").replace("-",""))) <= 0.0001: + print(("=====matrix_abs+e+===sql1.list1:'%s',sql2.list2:'%s'") %(list1,list2)) + print(("=====matrix_abs+e+replace_after===sql1.list1:'%s',sql2.list2:'%s'") %(float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace("e+","").replace(", ","").replace("(","").replace(")","").replace("-","")),float(str(list2).replace("datetime.datetime","").replace("]","").replace("[","").replace("e+","").replace(", ","").replace("(","").replace(")","").replace("-","")))) + tdLog.info(("===matrix_abs+e+===sql1:'%s' matrix_result = sql2:'%s' matrix_result") %(sql1,sql2)) + elif abs(float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-","")) - float(str(list2).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-",""))) <= 0.1: + #{datetime.datetime(2021, 8, 27, 1, 46, 40), -441.46841430664057}replace + print(("=====matrix_abs+replace===sql1.list1:'%s',sql2.list2:'%s'") %(list1,list2)) + print(("=====matrix_abs+replace_after===sql1.list1:'%s',sql2.list2:'%s'") %(float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-","")),float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-","")))) + tdLog.info(("===matrix_abs+replace===sql1:'%s' matrix_result = sql2:'%s' matrix_result") %(sql1,sql2)) + elif abs(float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-","")) - float(str(list2).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-",""))) <= 0.5: + print(("=====matrix_abs===sql1.list1:'%s',sql2.list2:'%s'") %(list1,list2)) + print(("=====matrix_abs===sql1.list1:'%s',sql2.list2:'%s'") %(float(str(list1).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-","")),float(str(list2).replace("datetime.datetime","").replace("]","").replace("[","").replace(", ","").replace("(","").replace(")","").replace("-","")))) + tdLog.info(("===matrix_abs======sql1:'%s' matrix_result = sql2:'%s' matrix_result") %(sql1,sql2)) + else: + print(("=====matrix_error===sql1.list1:'%s',sql2.list2:'%s'") %(list1,list2)) + tdLog.info(("sql1:'%s' matrix_result != sql2:'%s' matrix_result") %(sql1,sql2)) + return tdSql.checkEqual(list1,list2) + + def restartDnodes(self): + pass + # tdDnodes.stop(1) + # tdDnodes.start(1) + + def dropandcreateDB_random(self,database,n): + ts = 1630000000000 + num_random = 100 + fake = Faker('zh_CN') + tdSql.execute('''drop database if exists %s ;''' %database) + tdSql.execute('''create database %s keep 36500;'''%database) + tdSql.execute('''use %s;'''%database) + + tdSql.execute('''create stable stable_1 (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) \ + tags(loc nchar(100) , t_int int , t_bigint bigint , t_smallint smallint , t_tinyint tinyint, t_bool bool , t_binary binary(100) , t_nchar nchar(100) ,t_float float , t_double double , t_ts timestamp);''') + tdSql.execute('''create stable stable_2 (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) \ + tags(loc nchar(100) , t_int int , t_bigint bigint , t_smallint smallint , t_tinyint tinyint, t_bool bool , t_binary binary(100) , t_nchar nchar(100) ,t_float float , t_double double , t_ts timestamp);''') + + tdSql.execute('''create stable stable_null_data (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) \ + tags(loc nchar(100) , t_int int , t_bigint bigint , t_smallint smallint , t_tinyint tinyint, t_bool bool , t_binary binary(100) , t_nchar nchar(100) ,t_float float , t_double double , t_ts timestamp);''') + + tdSql.execute('''create stable stable_null_childtable (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) \ + tags(loc nchar(100) , t_int int , t_bigint bigint , t_smallint smallint , t_tinyint tinyint, t_bool bool , t_binary binary(100) , t_nchar nchar(100) ,t_float float , t_double double , t_ts timestamp);''') + + #tdSql.execute('''create table stable_1_1 using stable_1 tags('stable_1_1', '0' , '0' , '0' , '0' , 0 , 'binary1' , 'nchar1' , '0' , '0' ,'0') ;''') + tdSql.execute('''create table stable_1_1 using stable_1 tags('stable_1_1', '%d' , '%d', '%d' , '%d' , 0 , 'binary1.%s' , 'nchar1.%s' , '%f', '%f' ,'%d') ;''' + %(fake.random_int(min=-2147483647, max=2147483647, step=1), fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1), + fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) , + fake.pystr() ,fake.pystr() ,fake.pyfloat(),fake.pyfloat(),fake.random_int(min=-2147483647, max=2147483647, step=1))) + tdSql.execute('''create table stable_1_2 using stable_1 tags('stable_1_2', '2147483647' , '9223372036854775807' , '32767' , '127' , 1 , 'binary2' , 'nchar2' , '2' , '22' , \'1999-09-09 09:09:09.090\') ;''') + tdSql.execute('''create table stable_1_3 using stable_1 tags('stable_1_3', '-2147483647' , '-9223372036854775807' , '-32767' , '-127' , false , 'binary3' , 'nchar3nchar3' , '-3.3' , '-33.33' , \'2099-09-09 09:09:09.090\') ;''') + #tdSql.execute('''create table stable_1_4 using stable_1 tags('stable_1_4', '0' , '0' , '0' , '0' , 0 , '0' , '0' , '0' , '0' ,'0') ;''') + tdSql.execute('''create table stable_1_4 using stable_1 tags('stable_1_4', '%d' , '%d', '%d' , '%d' , 1 , 'binary1.%s' , 'nchar1.%s' , '%f', '%f' ,'%d') ;''' + %(fake.random_int(min=-2147483647, max=2147483647, step=1), fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1), + fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) , + fake.pystr() ,fake.pystr() ,fake.pyfloat(),fake.pyfloat(),fake.random_int(min=-2147483647, max=2147483647, step=1))) + + # tdSql.execute('''create table stable_2_1 using stable_2 tags('stable_2_1' , '0' , '0' , '0' , '0' , 0 , 'binary21' , 'nchar21' , '0' , '0' ,'0') ;''') + # tdSql.execute('''create table stable_2_2 using stable_2 tags('stable_2_2' , '0' , '0' , '0' , '0' , 0 , '0' , '0' , '0' , '0' ,'0') ;''') + + # tdSql.execute('''create table stable_null_data_1 using stable_null_data tags('stable_null_data_1', '0' , '0' , '0' , '0' , 0 , '0' , '0' , '0' , '0' ,'0') ;''') + + tdSql.execute('''create table stable_2_1 using stable_2 tags('stable_2_1' , '0' , '0' , '0' , '0' , 0 , 'binary21' , 'nchar21' , '0' , '0' ,\'2099-09-09 09:09:09.090\') ;''') + tdSql.execute('''create table stable_2_2 using stable_2 tags('stable_2_2' , '%d' , '%d', '%d' , '%d' , 0 , 'binary2.%s' , 'nchar2.%s' , '%f', '%f' ,'%d') ;''' + %(fake.random_int(min=-2147483647, max=2147483647, step=1), fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1), + fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) , + fake.pystr() ,fake.pystr() ,fake.pyfloat(),fake.pyfloat(),fake.random_int(min=-2147483647, max=2147483647, step=1))) + + tdSql.execute('''create table stable_null_data_1 using stable_null_data tags('stable_null_data_1', '%d' , '%d', '%d' , '%d' , 1 , 'binary1.%s' , 'nchar1.%s' , '%f', '%f' ,'%d') ;''' + %(fake.random_int(min=-2147483647, max=2147483647, step=1), fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1), + fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) , + fake.pystr() ,fake.pystr() ,fake.pyfloat(),fake.pyfloat(),fake.random_int(min=-2147483647, max=2147483647, step=1))) + + #regular table + tdSql.execute('''create table regular_table_1 \ + (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) ;''') + tdSql.execute('''create table regular_table_2 \ + (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) ;''') + tdSql.execute('''create table regular_table_3 \ + (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) ;''') + + tdSql.execute('''create table regular_table_null \ + (ts timestamp , q_int int , q_bigint bigint , q_smallint smallint , q_tinyint tinyint , q_float float , q_double double , q_bool bool , q_binary binary(100) , q_nchar nchar(100) , q_ts timestamp , \ + q_binary1 binary(100) , q_nchar1 nchar(100) ,q_binary2 binary(100) , q_nchar2 nchar(100) ,q_binary3 binary(100) , q_nchar3 nchar(100) ,q_binary4 binary(100) , q_nchar4 nchar(100) ,\ + q_binary5 binary(100) , q_nchar5 nchar(100) ,q_binary6 binary(100) , q_nchar6 nchar(100) ,q_binary7 binary(100) , q_nchar7 nchar(100) ,q_binary8 binary(100) , q_nchar8 nchar(100) ,\ + q_int_null int , q_bigint_null bigint , q_smallint_null smallint , q_tinyint_null tinyint, q_float_null float , q_double_null double , q_bool_null bool , q_binary_null binary(20) , q_nchar_null nchar(20) , q_ts_null timestamp) ;''') + + + for i in range(1, num_random*n + 1): + tdSql.execute('''insert into stable_1_1 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double , q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000, fake.random_int(min=-2147483647, max=2147483647, step=1), + fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1), + fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + tdSql.execute('''insert into regular_table_1 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000, fake.random_int(min=-2147483647, max=2147483647, step=1) , + fake.random_int(min=-9223372036854775807, max=9223372036854775807, step=1) , + fake.random_int(min=-32767, max=32767, step=1) , fake.random_int(min=-127, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + + tdSql.execute('''insert into stable_1_2 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8)\ + values(%d, %d, %d, %d, %d, %f, %f, 1, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000*60*60*2, fake.random_int(min=0, max=2147483647, step=1), + fake.random_int(min=0, max=9223372036854775807, step=1), + fake.random_int(min=0, max=32767, step=1) , fake.random_int(min=0, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + tdSql.execute('''insert into regular_table_2 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 1, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000, fake.random_int(min=0, max=2147483647, step=1), + fake.random_int(min=0, max=9223372036854775807, step=1), + fake.random_int(min=0, max=32767, step=1) , fake.random_int(min=0, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + + tdSql.execute('''insert into stable_1_2 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 1, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000*60*60*2 +1, fake.random_int(min=-2147483647, max=0, step=1), + fake.random_int(min=-9223372036854775807, max=0, step=1), + fake.random_int(min=-32767, max=0, step=1) , fake.random_int(min=-127, max=0, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i +1, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + tdSql.execute('''insert into regular_table_2 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 1, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000 +1, fake.random_int(min=-2147483647, max=0, step=1), + fake.random_int(min=-9223372036854775807, max=0, step=1), + fake.random_int(min=-32767, max=0, step=1) , fake.random_int(min=-127, max=0, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i +1, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + + tdSql.execute('''insert into stable_2_1 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000*60*60*4, fake.random_int(min=-0, max=2147483647, step=1), + fake.random_int(min=-0, max=9223372036854775807, step=1), + fake.random_int(min=-0, max=32767, step=1) , fake.random_int(min=-0, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + + tdSql.execute('''insert into stable_2_1 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000*60*60*4 +1, fake.random_int(min=-0, max=2147483647, step=1), + fake.random_int(min=-0, max=9223372036854775807, step=1), + fake.random_int(min=-0, max=32767, step=1) , fake.random_int(min=-0, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + + tdSql.execute('''insert into stable_2_1 (ts , q_int , q_bigint , q_smallint , q_tinyint , q_float , q_double, q_bool , q_binary , q_nchar, q_ts,\ + q_binary1 , q_nchar1 , q_binary2 , q_nchar2 , q_binary3 , q_nchar3 , q_binary4 , q_nchar4 , q_binary5 , q_nchar5 , q_binary6 , q_nchar6 , q_binary7 , q_nchar7, q_binary8 , q_nchar8) \ + values(%d, %d, %d, %d, %d, %f, %f, 0, 'binary.%s', 'nchar.%s', %d, 'binary1.%s', 'nchar1.%s', 'binary2.%s', 'nchar2.%s', 'binary3.%s', 'nchar3.%s', \ + 'binary4.%s', 'nchar4.%s', 'binary5.%s', 'nchar5.%s', 'binary6.%s', 'nchar6.%s', 'binary7.%s', 'nchar7.%s', 'binary8.%s', 'nchar8.%s') ;''' + % (ts + i*1000*60*60*4 +10, fake.random_int(min=-0, max=2147483647, step=1), + fake.random_int(min=-0, max=9223372036854775807, step=1), + fake.random_int(min=-0, max=32767, step=1) , fake.random_int(min=-0, max=127, step=1) , + fake.pyfloat() , fake.pyfloat() , fake.pystr() , fake.pystr() , ts + i, fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , + fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr() , fake.pystr())) + + tdSql.query("select count(*) from stable_1;") + tdSql.checkData(0,0,3*num_random*n) + tdSql.query("select count(*) from regular_table_1;") + tdSql.checkData(0,0,num_random*n) + + def explain_sql(self,sql): + # #执行sql解析 + sql = "explain " + sql + tdLog.info(sql) + tdSql.query(sql) + #pass + + def data_check(self,sql,mark='mark') : + tdLog.info("========mark==%s==="% mark); + try: + tdSql.query(sql,queryTimes=1) + self.explain_sql(sql) + except: + tdLog.info("sql is not support :=====%s; " %sql) + tdSql.error(sql) + + + def math_nest(self,mathlist): + + print("==========%s===start=============" %mathlist) + os.system("rm -rf %s/%s.sql" % (self.testcasePath,self.testcaseFilename)) + + self.dropandcreateDB_random("%s" %self.db_nest, 1) + + if (mathlist == ['ABS','SQRT']) or (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['FLOOR','CEIL','ROUND']) \ + or (mathlist == ['CSUM']) : + math_functions = mathlist + fun_fix_column = ['(q_bigint)','(q_smallint)','(q_tinyint)','(q_int)','(q_float)','(q_double)','(q_bigint_null)','(q_smallint_null)','(q_tinyint_null)','(q_int_null)','(q_float_null)','(q_double_null)'] + fun_column_1 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_2 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_j = ['(t1.q_bigint)','(t1.q_smallint)','(t1.q_tinyint)','(t1.q_int)','(t1.q_float)','(t1.q_double)','(t1.q_bigint_null)','(t1.q_smallint_null)','(t1.q_tinyint_null)','(t1.q_int_null)','(t1.q_float_null)','(t1.q_double_null)', + '(t2.q_bigint)','(t2.q_smallint)','(t2.q_tinyint)','(t2.q_int)','(t2.q_float)','(t2.q_double)','(t2.q_bigint_null)','(t2.q_smallint_null)','(t2.q_tinyint_null)','(t2.q_int_null)','(t2.q_float_null)','(t2.q_double_null)'] + fun_column_join_1 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_join_2 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + elif (mathlist == ['UNIQUE']) or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['MODE']) : + math_functions = mathlist + fun_fix_column = ['(q_bigint)','(q_smallint)','(q_tinyint)','(q_int)','(q_float)','(q_double)','(q_binary)','(q_nchar)','(q_bool)','(q_ts)', + '(q_bigint_null)','(q_smallint_null)','(q_tinyint_null)','(q_int_null)','(q_float_null)','(q_double_null)','(q_binary_null)','(q_nchar_null)','(q_bool_null)','(q_ts_null)'] + fun_column_1 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_2 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_j = ['(t1.q_bigint)','(t1.q_smallint)','(t1.q_tinyint)','(t1.q_int)','(t1.q_float)','(t1.q_double)','(t1.q_bigint_null)','(t1.q_smallint_null)','(t1.q_tinyint_null)','(t1.q_int_null)','(t1.q_float_null)','(t1.q_double_null)','(t1.q_ts)','(t1.q_ts_null)', + '(t2.q_bigint)','(t2.q_smallint)','(t2.q_tinyint)','(t2.q_int)','(t2.q_float)','(t2.q_double)','(t2.q_bigint_null)','(t2.q_smallint_null)','(t2.q_tinyint_null)','(t2.q_int_null)','(t2.q_float_null)','(t2.q_double_null)','(t2.q_ts)','(t2.q_ts_null)'] + fun_column_join_1 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_join_2 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + elif (mathlist == ['TAIL']): + math_functions = mathlist + num = random.randint(1, 100) + offset_rows = random.randint(0, 100) + fun_fix_column = ['(q_bigint,num)','(q_smallint,num)','(q_tinyint,num)','(q_int,num)','(q_float,num)','(q_double,num)','(q_binary,num)','(q_nchar,num)','(q_bool,num)','(q_ts,num)', + '(q_bigint_null,num)','(q_smallint_null,num)','(q_tinyint_null,num)','(q_int_null,num)','(q_float_null,num)','(q_double_null,num)','(q_binary_null,num)','(q_nchar_null,num)','(q_bool_null,num)','(q_ts_null,num)', + '(q_bigint,num,offset_rows)','(q_smallint,num,offset_rows)','(q_tinyint,num,offset_rows)','(q_int,num,offset_rows)','(q_float,num,offset_rows)','(q_double,num,offset_rows)','(q_binary,num,offset_rows)','(q_nchar,num,offset_rows)','(q_bool,num,offset_rows)','(q_ts,num,offset_rows)', + '(q_bigint_null,num,offset_rows)','(q_smallint_null,num,offset_rows)','(q_tinyint_null,num,offset_rows)','(q_int_null,num,offset_rows)','(q_float_null,num,offset_rows)','(q_double_null,num,offset_rows)','(q_binary_null,num,offset_rows)','(q_nchar_null,num,offset_rows)','(q_bool_null,num,offset_rows)','(q_ts_null,num,offset_rows)'] + fun_column_1 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)).replace("offset_rows",str(offset_rows)) + fun_column_2 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)).replace("offset_rows",str(offset_rows)) + + fun_fix_column_j = ['(t1.q_bigint,num)','(t1.q_smallint,num)','(t1.q_tinyint,num)','(t1.q_int,num)','(t1.q_float,num)','(t1.q_double,num)','(t1.q_binary,num)','(t1.q_nchar,num)','(t1.q_bool,num)','(t1.q_ts,num)', + '(t1.q_bigint_null,num)','(t1.q_smallint_null,num)','(t1.q_tinyint_null,num)','(t1.q_int_null,num)','(t1.q_float_null,num)','(t1.q_double_null,num)','(t1.q_binary_null,num)','(t1.q_nchar_null,num)','(t1.q_bool_null,num)','(t1.q_ts_null,num)', + '(t2.q_bigint,num)','(t2.q_smallint,num)','(t2.q_tinyint,num)','(t2.q_int,num)','(t2.q_float,num)','(t2.q_double,num)','(t2.q_binary,num)','(t2.q_nchar,num)','(t2.q_bool,num)','(t2.q_ts,num)', + '(t2.q_bigint_null,num)','(t2.q_smallint_null,num)','(t2.q_tinyint_null,num)','(t2.q_int_null,num)','(t2.q_float_null,num)','(t2.q_double_null,num)','(t2.q_binary_null,num)','(t2.q_nchar_null,num)','(t2.q_bool_null,num)','(t2.q_ts_null,num)', + '(t1.q_bigint,num,offset_rows)','(t1.q_smallint,num,offset_rows)','(t1.q_tinyint,num,offset_rows)','(t1.q_int,num,offset_rows)','(t1.q_float,num,offset_rows)','(t1.q_double,num,offset_rows)','(t1.q_binary,num,offset_rows)','(t1.q_nchar,num,offset_rows)','(t1.q_bool,num,offset_rows)','(t1.q_ts,num,offset_rows)', + '(t1.q_bigint_null,num,offset_rows)','(t1.q_smallint_null,num,offset_rows)','(t1.q_tinyint_null,num,offset_rows)','(t1.q_int_null,num,offset_rows)','(t1.q_float_null,num,offset_rows)','(t1.q_double_null,num,offset_rows)','(t1.q_binary_null,num,offset_rows)','(t1.q_nchar_null,num,offset_rows)','(t1.q_bool_null,num,offset_rows)','(t1.q_ts_null,num,offset_rows)', + '(t2.q_bigint,num,offset_rows)','(t2.q_smallint,num,offset_rows)','(t2.q_tinyint,num,offset_rows)','(t2.q_int,num,offset_rows)','(t2.q_float,num,offset_rows)','(t2.q_double,num,offset_rows)','(t2.q_binary,num,offset_rows)','(t2.q_nchar,num,offset_rows)','(t2.q_bool,num,offset_rows)','(t2.q_ts,num,offset_rows)', + '(t2.q_bigint_null,num,offset_rows)','(t2.q_smallint_null,num,offset_rows)','(t2.q_tinyint_null,num,offset_rows)','(t2.q_int_null,num,offset_rows)','(t2.q_float_null,num,offset_rows)','(t2.q_double_null,num,offset_rows)','(t2.q_binary_null,num,offset_rows)','(t2.q_nchar_null,num,offset_rows)','(t2.q_bool_null,num,offset_rows)','(t2.q_ts_null,num,offset_rows)'] + fun_column_join_1 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)).replace("offset_rows",str(offset_rows)) + fun_column_join_2 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)).replace("offset_rows",str(offset_rows)) + + elif (mathlist == ['POW','LOG']) or (mathlist == ['MAVG']) or (mathlist == ['SAMPLE']) : + math_functions = mathlist + num = random.randint(1, 1000) + fun_fix_column = ['(q_bigint,num)','(q_smallint,num)','(q_tinyint,num)','(q_int,num)','(q_float,num)','(q_double,num)', + '(q_bigint_null,num)','(q_smallint_null,num)','(q_tinyint_null,num)','(q_int_null,num)','(q_float_null,num)','(q_double_null,num)'] + fun_column_1 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)) + fun_column_2 = random.sample(math_functions,1)+random.sample(fun_fix_column,1) + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)) + + fun_fix_column_j = ['(t1.q_bigint,num)','(t1.q_smallint,num)','(t1.q_tinyint,num)','(t1.q_int,num)','(t1.q_float,num)','(t1.q_double,num)', + '(t1.q_bigint_null,num)','(t1.q_smallint_null,num)','(t1.q_tinyint_null,num)','(t1.q_int_null,num)','(t1.q_float_null,num)','(t1.q_double_null,num)', + '(t2.q_bigint,num)','(t2.q_smallint,num)','(t2.q_tinyint,num)','(t2.q_int,num)','(t2.q_float,num)','(t2.q_double,num)', + '(t2.q_bigint_null,num)','(t2.q_smallint_null,num)','(t2.q_tinyint_null,num)','(t2.q_int_null,num)','(t2.q_float_null,num)','(t2.q_double_null,num)'] + fun_column_join_1 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)) + fun_column_join_2 = random.sample(math_functions,1)+random.sample(fun_fix_column_j,1) + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",str(num)) + + elif (mathlist == ['statecount','stateduration']): + math_functions = mathlist + num = random.randint(-1000, 1000) + + operator = ['LT' , 'GT' ,'GE','NE','EQ'] + oper = str(random.sample(operator,1)).replace("[","").replace("]","")#.replace("'","") + + fun_fix_column = ['(q_bigint,oper,num,time)','(q_smallint,oper,num,time)','(q_tinyint,oper,num,time)','(q_int,oper,num,time)','(q_float,oper,num,time)','(q_double,oper,num,time)', + '(q_bigint_null,oper,num,time)','(q_smallint_null,oper,num,time)','(q_tinyint_null,oper,num,time)','(q_int_null,oper,num,time)','(q_float_null,oper,num,time)','(q_double_null,oper,num,time)'] + + hanshu_select1 = random.sample(math_functions,1) + fun_column_1 = random.sample(hanshu_select1,1)+random.sample(fun_fix_column,1) + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + + if str(hanshu_select1).replace("[","").replace("]","").replace("'","") == 'statecount': + math_fun_1 = math_fun_1.replace("oper","%s" %oper).replace(",time","").replace("num",str(num)) + elif str(hanshu_select1).replace("[","").replace("]","").replace("'","") == 'stateduration': + timeunit = ['1s' , '1m' ,'1h'] + time = str(random.sample(timeunit,1)).replace("[","").replace("]","").replace("'","") + math_fun_1 = math_fun_1.replace("oper","%s" %oper).replace("time","%s" %time).replace("num",str(num)) + + hanshu_select2 = random.sample(math_functions,1) + fun_column_2 = random.sample(hanshu_select2,1)+random.sample(fun_fix_column,1) + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + if str(hanshu_select2).replace("[","").replace("]","").replace("'","") == 'statecount': + math_fun_2 = math_fun_2.replace("oper","%s" %oper).replace(",time","").replace("num",str(num)) + elif str(hanshu_select2).replace("[","").replace("]","").replace("'","") == 'stateduration': + timeunit = ['1s' , '1m' ,'1h'] + time = str(random.sample(timeunit,1)).replace("[","").replace("]","").replace("'","") + math_fun_2 = math_fun_2.replace("oper","%s" %oper).replace("time","%s" %time).replace("num",str(num)) + + fun_fix_column_j = ['(t1.q_bigint,oper,num,time)','(t1.q_smallint,oper,num,time)','(t1.q_tinyint,oper,num,time)','(t1.q_int,oper,num,time)','(t1.q_float,oper,num,time)','(t1.q_double,oper,num,time)', + '(t1.q_bigint_null,oper,num,time)','(t1.q_smallint_null,oper,num,time)','(t1.q_tinyint_null,oper,num,time)','(t1.q_int_null,oper,num,time)','(t1.q_float_null,oper,num,time)','(t1.q_double_null,oper,num,time)', + '(t2.q_bigint,oper,num,time)','(t2.q_smallint,oper,num,time)','(t2.q_tinyint,oper,num,time)','(t2.q_int,oper,num,time)','(t2.q_float,oper,num,time)','(t2.q_double,oper,num,time)', + '(t2.q_bigint_null,oper,num,time)','(t2.q_smallint_null,oper,num,time)','(t2.q_tinyint_null,oper,num,time)','(t2.q_int_null,oper,num,time)','(t2.q_float_null,oper,num,time)','(t2.q_double_null,oper,num,time)'] + + hanshu_select_join_1 = random.sample(math_functions,1) + fun_column_join_1 = random.sample(hanshu_select_join_1,1)+random.sample(fun_fix_column_j,1) + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + + if str(hanshu_select_join_1).replace("[","").replace("]","").replace("'","") == 'statecount': + math_fun_join_1 = math_fun_join_1.replace("oper","%s" %oper).replace(",time","").replace("num",str(num)) + elif str(hanshu_select_join_1).replace("[","").replace("]","").replace("'","") == 'stateduration': + timeunit = ['1s' , '1m' ,'1h'] + time = str(random.sample(timeunit,1)).replace("[","").replace("]","").replace("'","") + math_fun_join_1 = math_fun_join_1.replace("oper","%s" %oper).replace("time","%s" %time).replace("num",str(num)) + + hanshu_select_join_2 = random.sample(math_functions,1) + fun_column_join_2 = random.sample(hanshu_select_join_2,1)+random.sample(fun_fix_column_j,1) + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + if str(hanshu_select_join_2).replace("[","").replace("]","").replace("'","") == 'statecount': + math_fun_join_2 = math_fun_join_2.replace("oper","%s" %oper).replace(",time","").replace("num",str(num)) + elif str(hanshu_select_join_2).replace("[","").replace("]","").replace("'","") == 'stateduration': + timeunit = ['1s' , '1m' ,'1h'] + time = str(random.sample(timeunit,1)).replace("[","").replace("]","").replace("'","") + math_fun_join_2 = math_fun_join_2.replace("oper","%s" %oper).replace("time","%s" %time).replace("num",str(num)) + + elif(mathlist == ['HISTOGRAM']) : + math_functions = mathlist + fun_fix_column = ['(q_bigint','(q_smallint','(q_tinyint','(q_int','(q_float','(q_double','(q_bigint_null','(q_smallint_null','(q_tinyint_null','(q_int_null','(q_float_null','(q_double_null'] + + fun_fix_column_j = ['(t1.q_bigint','(t1.q_smallint','(t1.q_tinyint','(t1.q_int','(t1.q_float','(t1.q_double','(t1.q_bigint_null','(t1.q_smallint_null','(t1.q_tinyint_null','(t1.q_int_null','(t1.q_float_null','(t1.q_double_null', + '(t2.q_bigint','(t2.q_smallint','(t2.q_tinyint','(t2.q_int','(t2.q_float','(t2.q_double','(t2.q_bigint_null','(t2.q_smallint_null','(t2.q_tinyint_null','(t2.q_int_null','(t2.q_float_null','(t2.q_double_null'] + + normalized = random.randint(0, 1) + + i = random.randint(1,3) + if i == 1: + bin_type = 'user_input' + bin_description = {-11111119395555977777} #9一会转译成, + fun_column_1 = [math_functions , random.sample(fun_fix_column,1), ',',"'%s'" %bin_type, ',',"'%s'" % bin_description, ',', "%d" %normalized,')'] + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("{","[").replace("}","]").replace("9",",") + + fun_column_2 = [math_functions , random.sample(fun_fix_column,1), ',',"'%s'" %bin_type, ',',"'%s'" % bin_description, ',', "%d" %normalized,')'] + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("{","[").replace("}","]").replace("9",",") + + fun_column_join_1 = [math_functions , random.sample(fun_fix_column_j,1), ',',"'%s'" %bin_type, ',',"'%s'" % bin_description, ',', "%d" %normalized,')'] + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("{","[").replace("}","]").replace("9",",") + + fun_column_join_2 = [math_functions , random.sample(fun_fix_column_j,1), ',',"'%s'" %bin_type, ',',"'%s'" % bin_description, ',', "%d" %normalized,')'] + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("{","[").replace("}","]").replace("9",",") + + elif i == 2: + bin_type = 'linear_bin' + true_false = random.randint(10, 11) + bin_description = {"ZstartZ": -333339, "ZwidthZ":559, "ZcountZ":59, "ZinfinityZ":'%d' %true_false} #Z一会转译成" ,9一会转译成 , + fun_column_1 = [math_functions , random.sample(fun_fix_column,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + fun_column_2 = [math_functions , random.sample(fun_fix_column,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + fun_column_join_1 = [math_functions , random.sample(fun_fix_column_j,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + fun_column_join_2 = [math_functions , random.sample(fun_fix_column_j,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + elif i == 3: + bin_type = 'log_bin' + true_false = random.randint(10, 11) + bin_description = {"ZstartZ": -333339, "ZfactorZ":559, "ZcountZ":59, "ZinfinityZ":'%d' %true_false} #Z一会转译成" ,9一会转译成 , + fun_column_1 = [math_functions , random.sample(fun_fix_column,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + fun_column_2 = [math_functions , random.sample(fun_fix_column,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + fun_column_join_1 = [math_functions , random.sample(fun_fix_column_j,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + fun_column_join_2 = [math_functions , random.sample(fun_fix_column_j,1), ',',"'%s'" %bin_type, ',','%s' % bin_description, ',', "%d" %normalized,')'] + math_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("9",",").replace("Z","\"").replace("10","false").replace("11","true").replace("\"{","'{").replace("}\"","}'") + + tdSql.query("select 1-1 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']): + sql = "select ts1 , floor(asct1) from ( select " + sql += "%s as asct1, " % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts1 from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE']) or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-2 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts , abs(asct1) from ( select " + sql += "%s as asct1, " % math_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s )" % random.choice(self.order_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select ts , asct2 from ( select " + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE']) or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']): + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += "from regular_table_1 where " + sql += "%s )" % random.choice(self.q_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select count(asct2) from ( select " + sql += "%s as asct2 " % math_fun_2 + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-3 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts , min(asct1) from ( select " + sql += "%s as asct1, ts ," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s " % random.choice(self.q_select) + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2, ts ," % math_fun_2 + sql += "%s as asct1, " % math_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s " % random.choice(self.q_select) + sql += " from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MODE']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1," % math_fun_1 + sql += "%s as asct2 " % math_fun_2 + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2 ," % math_fun_2 + sql += "%s as asct1 " % math_fun_1 + sql += " from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['statecount','stateduration']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['TAIL']) or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['SAMPLE']): + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2 " % math_fun_2 + sql += " from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += " order by asct1 asc " + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-4 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1,ts2 ,timediff(ts1,ts2), asct1 from ( select t1.ts as ts1," + sql += "%s as asct0, " % math_fun_join_1 + sql += "%s as asct1, " % math_fun_join_2 + sql += "%s as asct2, " % math_fun_join_1 + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct22, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_join_2 + sql += "from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-5 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts ," + sql += "%s, " % math_fun_1 + sql += "%s as asct1, " % random.choice(self.q_select) + sql += "%s as asct2, " % random.choice(self.q_select) + sql += "%s " % math_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select " + sql += "%s " % math_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-6 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1 ,timediff(ts1,ts2), max(asct1) from ( select t1.ts,t1.ts as ts1," + sql += "%s as asct0, " % math_fun_join_1 + sql += "%s as asct1, " % math_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t2.%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % math_fun_join_1 + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_join_2 + sql += "from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-7 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1,ts2 , abs(asct1) from ( select " + sql += "%s as asct1, ts as ts1," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts as ts2 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE']) or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += "from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-8 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1,floor(asct1),ts2 " + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s as asct1, ts as ts1," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts as ts2 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) " + sql += "from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-9 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1,ts2 ,timediff(ts1,ts2) , max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % math_fun_join_1 + sql += "%s as asct1, " % math_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct21, " % random.choice(self.q_select) + sql += "t2.%s as asct22, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['CSUM']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_join_2 + sql += "from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + elif (mathlist == ['TAIL']) or (mathlist == ['SAMPLE']) or (mathlist == ['UNIQUE']) or (mathlist == ['HISTOGRAM']) or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['MODE']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_join_2 + sql += "from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + self.restartDnodes() + tdSql.query("select 1-10 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts , min(asct1) from ( select " + sql += "%s as asct1 ," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select ts , max(asct2) from ( select " + sql += "%s as asct1 ," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select count(asct2) from ( select " + sql += "%s as asct2 " % math_fun_2 + sql += "from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #3 inter union not support + tdSql.query("select 1-11 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select min(asct1), max(asct2) from ( select " + sql += "%s as asct1, ts ," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as t2ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1, ts as t1ts," % math_fun_1 + sql += "%s as asct2, " % math_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as t2ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct2 " % math_fun_2 + sql += " from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-12 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1,ts2 ,timediff(ts1,ts2), max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % math_fun_join_1 + sql += "%s as asct1, " % math_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct21, " % random.choice(self.q_select) + sql += "t2.%s as asct111, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_join_2 + sql += " from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-13 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts ," + sql += "%s as asct11, " % math_fun_1 + sql += "%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % random.choice(self.q_select) + sql += "%s as asct14, " % math_fun_2 + sql += "%s as asct15 " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select " + sql += "%s " % math_fun_2 + sql += "%s " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-14 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select avg(asct1),count(asct2) from ( select " + sql += "%s as asct1, " % math_fun_1 + sql += "%s as asct2" % math_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE'])or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_1 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-15 as math_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (mathlist == ['SIN','COS','TAN','ASIN','ACOS','ATAN']) or (mathlist == ['ABS','SQRT']) \ + or (mathlist == ['POW','LOG']) or (mathlist == ['FLOOR','CEIL','ROUND']) : + sql = "select ts1,ts ,timediff(ts1,ts), max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % math_fun_join_1 + sql += "%s as asct1, " % math_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (mathlist == ['MAVG']) or (mathlist == ['SAMPLE']) or (mathlist == ['TAIL']) or (mathlist == ['CSUM']) or (mathlist == ['HISTOGRAM']) \ + or (mathlist == ['HYPERLOGLOG']) or (mathlist == ['UNIQUE']) or (mathlist == ['MODE']) or (mathlist == ['statecount','stateduration']) : + sql = "select count(asct1) from ( select " + sql += "%s as asct1 " % math_fun_join_2 + sql += "from stable_1 t1, stable_2 t2 where t1.ts = t2.ts " + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #taos -f sql + # startTime_taosf = time.time() + print("taos -f %s sql start!" %mathlist) + taos_cmd1 = "taos -f %s/%s.sql" % (self.testcasePath,self.testcaseFilename) + #_ = subprocess.check_output(taos_cmd1, shell=True).decode("utf-8") + _ = subprocess.check_output(taos_cmd1, shell=True) + print("taos -f %s sql over!" %mathlist) + # endTime_taosf = time.time() + # print("taos_f total time %ds" % (endTime_taos_f - startTime_taos_f)) + + print("=========%s====over=============" %mathlist) + + + def str_nest(self,strlist): + + print("==========%s===start=============" %strlist) + os.system("rm -rf %s/%s.sql" % (self.testcasePath,self.testcaseFilename)) + + self.dropandcreateDB_random("%s" %self.db_nest, 1) + + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['LENGTH','CHAR_LENGTH']) \ + or (strlist == ['']): + str_functions = strlist + fun_fix_column = ['(q_nchar)','(q_binary)','(q_nchar_null)','(q_binary_null)'] + fun_column_1 = random.sample(str_functions,1)+random.sample(fun_fix_column,1) + str_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_2 = random.sample(str_functions,1)+random.sample(fun_fix_column,1) + str_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_j = ['(t1.q_nchar)','(t1.q_binary)','(t1.q_nchar_null)','(t1.q_binary_null)', + '(t2.q_nchar)','(t2.q_binary)','(t2.q_nchar_null)','(t2.q_binary_null)'] + fun_column_join_1 = random.sample(str_functions,1)+random.sample(fun_fix_column_j,1) + str_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_join_2 = random.sample(str_functions,1)+random.sample(fun_fix_column_j,1) + str_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_s = ['(q_nchar)','(q_binary)','(q_nchar_null)','(q_binary_null)','(loc)','(tbname)'] + fun_column_s_1 = random.sample(str_functions,1)+random.sample(fun_fix_column_s,1) + str_fun_s_1 = str(fun_column_s_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_s_2 = random.sample(str_functions,1)+random.sample(fun_fix_column_s,1) + str_fun_s_2 = str(fun_column_s_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_s_j = ['(t1.q_nchar)','(t1.q_binary)','(t1.q_nchar_null)','(t1.q_binary_null)','(t1.loc)','(t1.tbname)', + '(t2.q_nchar)','(t2.q_binary)','(t2.q_nchar_null)','(t2.q_binary_null)','(t2.loc)','(t2.tbname)'] + fun_column_join_s_1 = random.sample(str_functions,1)+random.sample(fun_fix_column_j,1) + str_fun_join_s_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_join_s_2 = random.sample(str_functions,1)+random.sample(fun_fix_column_j,1) + str_fun_join_s_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + elif (strlist == ['SUBSTR']) : + str_functions = strlist + pos = random.randint(1, 20) + sub_len = random.randint(1, 10) + fun_fix_column = ['(q_nchar,pos)','(q_binary,pos)','(q_nchar_null,pos)','(q_binary_null,pos)', + '(q_nchar,pos,sub_len)','(q_binary,pos,sub_len)','(q_nchar_null,pos,sub_len)','(q_binary_null,pos,sub_len)',] + fun_column_1 = random.sample(str_functions,1)+random.sample(fun_fix_column,1) + str_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + fun_column_2 = random.sample(str_functions,1)+random.sample(fun_fix_column,1) + str_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + + fun_fix_column_j = ['(t1.q_nchar,pos)','(t1.q_binary,pos)','(t1.q_nchar_null,pos)','(t1.q_binary_null,pos)', + '(t1.q_nchar,pos,sub_len)','(t1.q_binary,pos,sub_len)','(t1.q_nchar_null,pos,sub_len)','(t1.q_binary_null,pos,sub_len)', + '(t2.q_nchar,pos)','(t2.q_binary,pos)','(t2.q_nchar_null,pos)','(t2.q_binary_null,pos)', + '(t2.q_nchar,pos,sub_len)','(t2.q_binary,pos,sub_len)','(t2.q_nchar_null,pos,sub_len)','(t2.q_binary_null,pos,sub_len)'] + fun_column_join_1 = random.sample(str_functions,1)+random.sample(fun_fix_column_j,1) + str_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + fun_column_join_2 = random.sample(str_functions,1)+random.sample(fun_fix_column_j,1) + str_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + + fun_fix_column_s = ['(q_nchar,pos)','(q_binary,pos)','(q_nchar_null,pos)','(q_binary_null,pos)','(loc,pos)', + '(q_nchar,pos,sub_len)','(q_binary,pos,sub_len)','(q_nchar_null,pos,sub_len)','(q_binary_null,pos,sub_len)','(loc,pos,sub_len)',] + fun_column_s_1 = random.sample(str_functions,1)+random.sample(fun_fix_column_s,1) + str_fun_s_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + fun_column_s_2 = random.sample(str_functions,1)+random.sample(fun_fix_column_s,1) + str_fun_s_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + + fun_fix_column_s_j = ['(t1.q_nchar,pos)','(t1.q_binary,pos)','(t1.q_nchar_null,pos)','(t1.q_binary_null,pos)','(t1.loc,pos)', + '(t1.q_nchar,pos,sub_len)','(t1.q_binary,pos,sub_len)','(t1.q_nchar_null,pos,sub_len)','(t1.q_binary_null,pos,sub_len)','(t1.loc,pos,sub_len)', + '(t2.q_nchar,pos)','(t2.q_binary,pos)','(t2.q_nchar_null,pos)','(t2.q_binary_null,pos)','(t2.loc,pos)', + '(t2.q_nchar,pos,sub_len)','(t2.q_binary,pos,sub_len)','(t2.q_nchar_null,pos,sub_len)','(t2.q_binary_null,pos,sub_len)','(t2.loc,pos,sub_len)'] + fun_column_join_s_1 = random.sample(str_functions,1)+random.sample(fun_fix_column_s_j,1) + str_fun_join_s_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + fun_column_join_s_2 = random.sample(str_functions,1)+random.sample(fun_fix_column_s_j,1) + str_fun_join_s_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("pos",str(pos)).replace("sub_len",str(sub_len)) + + elif (strlist == ['CONCAT']) : + str_functions = strlist + i = random.randint(2,4) + fun_fix_column = ['q_nchar','q_nchar1','q_nchar2','q_nchar3','q_nchar4','q_nchar5','q_nchar6','q_nchar7','q_nchar8','q_nchar_null', + 'q_binary','q_binary1','q_binary2','q_binary3','q_binary4','q_binary5','q_binary6','q_binary7','q_binary8','q_binary_null'] + + column1 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+column1+')' + str_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","") + + column2 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+column2+')' + str_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_j = ['(t1.q_nchar)','(t1.q_nchar1)','(t1.q_nchar2)','(t1.q_nchar3)','(t1.q_nchar4)','(t1.q_nchar5)','(t1.q_nchar6)','(t1.q_nchar7)','(t1.q_nchar8)','(t1.q_nchar_null)', + '(t2.q_nchar)','(t2.q_nchar1)','(t2.q_nchar2)','(t2.q_nchar3)','(t2.q_nchar4)','(t2.q_nchar5)','(t2.q_nchar6)','(t2.q_nchar7)','(t2.q_nchar8)','(t2.q_nchar_null)', + '(t1.q_binary)','(t1.q_binary1)','(t1.q_binary2)','(t1.q_binary3)','(t1.q_binary4)','(t1.q_binary5)','(t1.q_binary6)','(t1.q_binary7)','(t1.q_binary8)','(t1.q_binary_null)', + '(t2.q_binary)','(t2.q_binary1)','(t2.q_binary2)','(t2.q_binary3)','(t2.q_binary4)','(t2.q_binary5)','(t2.q_binary6)','(t2.q_binary7)','(t2.q_binary8)','(t2.q_binary_null)'] + + column_j1 = str(random.sample(fun_fix_column_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+column_j1+')' + str_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","") + + column_j2 = str(random.sample(fun_fix_column_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+column_j2+')' + str_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_s = ['q_nchar','q_nchar1','q_nchar2','q_nchar3','q_nchar4','q_nchar5','q_nchar6','q_nchar7','q_nchar8','loc','q_nchar_null', + 'q_binary','q_binary1','q_binary2','q_binary3','q_binary4','q_binary5','q_binary6','q_binary7','q_binary8','q_binary_null'] + + column_s1 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_s_1 = str(random.sample(str_functions,1))+'('+column_s1+')' + str_fun_s_1 = str(fun_column_s_1).replace("[","").replace("]","").replace("'","") + + column_s2 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_s_2 = str(random.sample(str_functions,1))+'('+column_s2+')' + str_fun_s_2 = str(fun_column_s_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_s_j = ['(t1.q_nchar)','(t1.q_nchar1)','(t1.q_nchar2)','(t1.q_nchar3)','(t1.q_nchar4)','(t1.q_nchar5)','(t1.q_nchar6)','(t1.q_nchar7)','(t1.q_nchar8)','(t1.q_nchar_null)','(t1.loc)', + '(t2.q_nchar)','(t2.q_nchar1)','(t2.q_nchar2)','(t2.q_nchar3)','(t2.q_nchar4)','(t2.q_nchar5)','(t2.q_nchar6)','(t2.q_nchar7)','(t2.q_nchar8)','(t2.q_nchar_null)','(t2.loc)', + '(t1.q_binary)','(t1.q_binary1)','(t1.q_binary2)','(t1.q_binary3)','(t1.q_binary4)','(t1.q_binary5)','(t1.q_binary6)','(t1.q_binary7)','(t1.q_binary8)','(t1.q_binary_null)', + '(t2.q_binary)','(t2.q_binary1)','(t2.q_binary2)','(t2.q_binary3)','(t2.q_binary4)','(t2.q_binary5)','(t2.q_binary6)','(t2.q_binary7)','(t2.q_binary8)','(t2.q_binary_null)'] + + column_j_s1 = str(random.sample(fun_fix_column_s_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_s_1 = str(random.sample(str_functions,1))+'('+column_j_s1+')' + str_fun_join_s_1 = str(fun_column_join_s_1).replace("[","").replace("]","").replace("'","") + + column_j_s2 = str(random.sample(fun_fix_column_s_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_s_2 = str(random.sample(str_functions,1))+'('+column_j_s2+')' + str_fun_join_s_2 = str(fun_column_join_s_2).replace("[","").replace("]","").replace("'","") + + elif (strlist == ['CONCAT_WS']): + str_functions = strlist + i = random.randint(2,4) + fun_fix_column = ['q_nchar','q_nchar1','q_nchar2','q_nchar3','q_nchar4','q_nchar5','q_nchar6','q_nchar7','q_nchar8','q_nchar_null', + 'q_binary','q_binary1','q_binary2','q_binary3','q_binary4','q_binary5','q_binary6','q_binary7','q_binary8','q_binary_null'] + + separators = ['',' ','abc','123','!','@','#','$','%','^','&','*','(',')','-','_','+','=','{', + '[','}',']','|',';',':',',','.','<','>','?','/','~','`','taos','涛思'] + separator = str(random.sample(separators,i)).replace("[","").replace("]","") + + column1 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column1+')' + str_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","") + + column2 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column2+')' + str_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_j = ['(t1.q_nchar)','(t1.q_nchar1)','(t1.q_nchar2)','(t1.q_nchar3)','(t1.q_nchar4)','(t1.q_nchar5)','(t1.q_nchar6)','(t1.q_nchar7)','(t1.q_nchar8)','(t1.q_nchar_null)', + '(t2.q_nchar)','(t2.q_nchar1)','(t2.q_nchar2)','(t2.q_nchar3)','(t2.q_nchar4)','(t2.q_nchar5)','(t2.q_nchar6)','(t2.q_nchar7)','(t2.q_nchar8)','(t2.q_nchar_null)', + '(t1.q_binary)','(t1.q_binary1)','(t1.q_binary2)','(t1.q_binary3)','(t1.q_binary4)','(t1.q_binary5)','(t1.q_binary6)','(t1.q_binary7)','(t1.q_binary8)','(t1.q_binary_null)', + '(t2.q_binary)','(t2.q_binary1)','(t2.q_binary2)','(t2.q_binary3)','(t2.q_binary4)','(t2.q_binary5)','(t2.q_binary6)','(t2.q_binary7)','(t2.q_binary8)','(t2.q_binary_null)'] + + column_j1 = str(random.sample(fun_fix_column_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column_j1+')' + str_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","") + + column_j2 = str(random.sample(fun_fix_column_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column_j2+')' + str_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_s = ['q_nchar','q_nchar1','q_nchar2','q_nchar3','q_nchar4','q_nchar5','q_nchar6','q_nchar7','q_nchar8','loc','q_nchar_null', + 'q_binary','q_binary1','q_binary2','q_binary3','q_binary4','q_binary5','q_binary6','q_binary7','q_binary8','q_binary_null'] + + column_s1 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_s_1 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column_s1+')' + str_fun_s_1 = str(fun_column_s_1).replace("[","").replace("]","").replace("'","") + + column_s2 = str(random.sample(fun_fix_column,i)).replace("[","").replace("]","").replace("'","") + fun_column_s_2 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column_s2+')' + str_fun_s_2 = str(fun_column_s_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_s_j = ['(t1.q_nchar)','(t1.q_nchar1)','(t1.q_nchar2)','(t1.q_nchar3)','(t1.q_nchar4)','(t1.q_nchar5)','(t1.q_nchar6)','(t1.q_nchar7)','(t1.q_nchar8)','(t1.q_nchar_null)','(t1.loc)', + '(t2.q_nchar)','(t2.q_nchar1)','(t2.q_nchar2)','(t2.q_nchar3)','(t2.q_nchar4)','(t2.q_nchar5)','(t2.q_nchar6)','(t2.q_nchar7)','(t2.q_nchar8)','(t2.q_nchar_null)','(t2.loc)', + '(t1.q_binary)','(t1.q_binary1)','(t1.q_binary2)','(t1.q_binary3)','(t1.q_binary4)','(t1.q_binary5)','(t1.q_binary6)','(t1.q_binary7)','(t1.q_binary8)','(t1.q_binary_null)', + '(t2.q_binary)','(t2.q_binary1)','(t2.q_binary2)','(t2.q_binary3)','(t2.q_binary4)','(t2.q_binary5)','(t2.q_binary6)','(t2.q_binary7)','(t2.q_binary8)','(t2.q_binary_null)'] + + column_j_s1 = str(random.sample(fun_fix_column_s_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_s_1 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column_j_s1+')' + str_fun_join_s_1 = str(fun_column_join_s_1).replace("[","").replace("]","").replace("'","") + + column_j_s2 = str(random.sample(fun_fix_column_s_j,i)).replace("[","").replace("]","").replace("'","") + fun_column_join_s_2 = str(random.sample(str_functions,1))+'('+'\"'+separator+'\",'+column_j_s2+')' + str_fun_join_s_2 = str(fun_column_join_s_2).replace("[","").replace("]","").replace("'","") + + + tdSql.query("select 1-1 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']) : + sql = "select t1s , LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1, " % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as t1s from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1, " % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-2 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']) : + sql = "select ts , asct1 from ( select " + sql += "%s as asct1, " % str_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s )" % random.choice(self.order_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select ts , asct2 from ( select " + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + #sql += "%s " % random.choice(having_support) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1) from ( select " + sql += "%s as asct1, " % str_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s )" % random.choice(self.order_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select sum(asct2), min(asct2) from ( select " + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-3 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts , LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2 ," % str_fun_2 + sql += "%s as asct1, " % str_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + #tdSql.query(sql) #'unexpected end of data' + # self.cur1.execute(sql) + # self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2 ," % str_fun_2 + sql += "%s as asct1, " % str_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-4 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_1 + sql += "%s as asct1, " % str_fun_join_2 + sql += "%s, " % str_fun_join_1 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t2.%s as asct12, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_1 + sql += "%s as asct1, " % str_fun_join_2 + sql += "%s, " % str_fun_join_1 + sql += "t1.%s as asct21, " % random.choice(self.q_select) + sql += "t2.%s as asct22, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-5 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts ," + sql += "%s, " % str_fun_1 + sql += "%s as asct21, " % random.choice(self.q_select) + sql += "%s as asct22, " % random.choice(self.q_select) + sql += "%s " % str_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select ts ," + sql += "%s, " % str_fun_1 + sql += "%s as asct22, " % random.choice(self.q_select) + sql += "%s as asct21, " % random.choice(self.q_select) + sql += "%s " % str_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-6 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts1,ts ,timediff(ts1,ts), LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_1 + sql += "%s as asct1, " % str_fun_join_2 + sql += "t1.%s as asct22, " % random.choice(self.q_select) + sql += "t2.%s as asct21, " % random.choice(self.q_select) + sql += "%s, " % str_fun_join_1 + sql += "t2.ts from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_1 + sql += "%s as asct1, " % str_fun_join_2 + sql += "t1.%s as asct22, " % random.choice(self.q_select) + sql += "t2.%s as asct21, " % random.choice(self.q_select) + sql += "%s, " % str_fun_join_1 + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-7 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select t1s ,ts1, LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1, ts as t1s," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts as ts1 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1, ts as ts1," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts as t1s from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-8 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts1,st1, LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) " + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s as asct1, ts as st1," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts as ts1 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) " + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s as asct1, ts as ts1," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts as st1 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-9 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_s_1 + sql += "%s as asct1, " % str_fun_join_s_2 + sql += "t1.%s as asct21, " % random.choice(self.q_select) + sql += "t1.%s as asct22, " % random.choice(self.q_select) + sql += "t2.%s as asct23, " % random.choice(self.q_select) + sql += "t2.%s as asct24, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_s_1 + sql += "%s as asct1, " % str_fun_join_s_2 + sql += "t1.%s as asct21, " % random.choice(self.q_select) + sql += "t1.%s as asct22, " % random.choice(self.q_select) + sql += "t2.%s as asct23, " % random.choice(self.q_select) + sql += "t2.%s as asct24, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + self.restartDnodes() + tdSql.query("select 1-10 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts , LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select ts , LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #3 inter union not support + tdSql.query("select 1-11 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts , LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1, ts ," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s " % random.choice(self.q_select) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1, ts ," % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s " % random.choice(self.q_select) + sql += " from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1 ," % str_fun_s_1 + sql += "%s as asct2, " % str_fun_s_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1 ," % str_fun_1 + sql += "%s as asct2, " % str_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-12 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_s_1 + sql += "%s as asct1, " % str_fun_join_s_2 + sql += "t1.%s as asct21, " % random.choice(self.q_select) + sql += "t1.%s as asct22, " % random.choice(self.q_select) + sql += "t2.%s as asct23, " % random.choice(self.q_select) + sql += "t2.%s as asct24, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_s_1 + sql += "%s as asct1, " % str_fun_join_s_2 + sql += "t1.%s as asct21, " % random.choice(self.q_select) + sql += "t1.%s as asct22, " % random.choice(self.q_select) + sql += "t2.%s as asct23, " % random.choice(self.q_select) + sql += "t2.%s as asct24, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-13 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts ," + sql += "%s as asct10, " % str_fun_1 + sql += "%s as asct1, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % str_fun_2 + sql += "%s as asct14 " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select ts ," + sql += "%s as asct1, " % str_fun_1 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % str_fun_2 + sql += "%s as asct14 " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-14 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select " + sql += "%s as asct1, " % str_fun_s_1 + sql += "%s as asct2" % str_fun_s_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select " + sql += "%s as asct1, " % str_fun_s_1 + sql += "%s as asct2" % str_fun_s_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-15 as str_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (strlist == ['LTRIM','RTRIM','LOWER','UPPER']) or (strlist == ['SUBSTR']) or (strlist == ['CONCAT']) or (strlist == ['CONCAT_WS']): + sql = "select ts,ts2 ,timediff(ts,ts2), LTRIM(asct1), LOWER(asct1), RTRIM(asct2), UPPER(asct2) from ( select t1.ts ," + sql += "%s as asct2, " % str_fun_join_s_1 + sql += "%s as asct1, " % str_fun_join_s_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (strlist == ['LENGTH','CHAR_LENGTH']): + sql = "select sum(asct1), min(asct1), max(asct2), avg(asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % str_fun_join_s_1 + sql += "%s as asct1, " % str_fun_join_s_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14 " % random.choice(self.q_select) + sql += "from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #taos -f sql + startTime_taos_f = time.time() + print("taos -f %s sql start!" %strlist) + taos_cmd1 = "taos -f %s/%s.sql" % (self.testcasePath,self.testcaseFilename) + #_ = subprocess.check_output(taos_cmd1, shell=True).decode("utf-8") + _ = subprocess.check_output(taos_cmd1, shell=True) + print("taos -f %s sql over!" %strlist) + endTime_taos_f = time.time() + print("taos_f total time %ds" % (endTime_taos_f - startTime_taos_f)) + + print("=========%s====over=============" %strlist) + + def time_nest(self,timelist): + + print("==========%s===start=============" %timelist) + os.system("rm -rf %s/%s.sql" % (self.testcasePath,self.testcaseFilename)) + + self.dropandcreateDB_random("%s" %self.db_nest, 1) + + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMEZONE']): + time_functions = timelist + fun_fix_column = ['()'] + fun_column_1 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_2 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_j = ['()'] + fun_column_join_1 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_join_2 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + elif (timelist == ['TIMETRUNCATE']): + time_functions = timelist + + t = time.time() + t_to_s = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(t)) + fun_fix_column = ['q_ts','ts','_c0','_C0','_rowts','1600000000000','1600000000000000','1600000000000000000', + '%d' %t, '%d000' %t, '%d000000' %t,'t_to_s'] + + timeunits = ['1a' ,'1s', '1m' ,'1h', '1d'] + timeunit = str(random.sample(timeunits,1)).replace("[","").replace("]","").replace("'","") + + column_1 = ['(%s,timeutil)'%(random.sample(fun_fix_column,1))] + fun_column_1 = random.sample(time_functions,1)+random.sample(column_1,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_1 = str(time_fun_1).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s) + + column_2 = ['(%s,timeutil)'%(random.sample(fun_fix_column,1))] + fun_column_2 = random.sample(time_functions,1)+random.sample(column_2,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_2 = str(time_fun_2).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s) + + + fun_fix_column_j = ['(t1.q_ts)','(t1.ts)', '(t2.q_ts)','(t2.ts)','(1600000000000)','(1600000000000000)','(1600000000000000000)', + '(%d)' %t, '(%d000)' %t, '(%d000000)' %t,'t_to_s'] + + column_j1 = ['(%s,timeutil)'%(random.sample(fun_fix_column_j,1))] + fun_column_join_1 = random.sample(time_functions,1)+random.sample(column_j1,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_join_1 = str(time_fun_join_1).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s) + + column_j2 = ['(%s,timeutil)'%(random.sample(fun_fix_column_j,1))] + fun_column_join_2 = random.sample(time_functions,1)+random.sample(column_j2,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_join_2 = str(time_fun_join_2).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s) + + elif (timelist == ['TO_ISO8601']): + time_functions = timelist + + t = time.time() + fun_fix_column = ['(now())','(ts)','(q_ts)','(_rowts)','(_c0)','(_C0)', + '(1600000000000)','(1600000000000000)','(1600000000000000000)', + '(%d)' %t, '(%d000)' %t, '(%d000000)' %t] + + fun_column_1 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_column_2 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_j = ['(t1.q_ts)','(t1.ts)', '(t2.q_ts)','(t2.ts)','(1600000000000)','(1600000000000000)','(1600000000000000000)','(now())', + '(%d)' %t, '(%d000)' %t, '(%d000000)' %t] + + fun_column_join_1 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_column_join_2 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + elif (timelist == ['TO_UNIXTIMESTAMP']): + time_functions = timelist + + t = time.time() + t_to_s = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(t)) + fun_fix_column = ['(q_nchar)','(q_nchar1)','(q_nchar2)','(q_nchar3)','(q_nchar4)','(q_nchar_null)','(q_binary)','(q_binary5)','(q_binary6)','(q_binary7)','(q_binary8)','(q_binary_null)','(t_to_s)'] + + fun_column_1 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("t_to_s","'t_to_s'") + time_fun_1 = str(time_fun_1).replace("t_to_s","%s" %t_to_s) + + fun_column_2 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("t_to_s","'t_to_s'") + time_fun_2 = str(time_fun_2).replace("t_to_s","%s" %t_to_s) + + fun_fix_column_j = ['(t1.q_nchar)','(t1.q_binary)', '(t2.q_nchar)','(t2.q_binary)','(t_to_s)'] + + fun_column_join_1 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("t_to_s","'t_to_s'") + time_fun_join_1 = str(time_fun_join_1).replace("t_to_s","%s" %t_to_s) + + fun_column_join_2 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("t_to_s","'t_to_s'") + time_fun_join_2 = str(time_fun_join_2).replace("t_to_s","%s" %t_to_s) + + elif (timelist == ['TIMEDIFF_1']): + time_functions = timelist + + t = time.time() + t_to_s = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(t)) + timeunits = [ '1a' ,'1s', '1m' ,'1h', '1d'] + timeunit = str(random.sample(timeunits,1)).replace("[","").replace("]","").replace("'","") + + fun_fix_column = ['q_ts','ts','_c0','_C0','_rowts','1600000000000','1600000000000000','1600000000000000000', + '%d' %t, '%d000' %t, '%d000000' %t,'t_to_s'] + + column_1,column_2 = random.sample(fun_fix_column,1),random.sample(fun_fix_column,1) + column_12 = ['(%s,%s,timeutil)'%(column_1,column_2)] + fun_column_1 = random.sample(time_functions,1)+random.sample(column_12,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_1 = str(time_fun_1).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s).replace("_1","") + + column_3,column_4 = random.sample(fun_fix_column,1),random.sample(fun_fix_column,1) + column_34 = ['(%s,%s,timeutil)'%(column_3,column_4)] + fun_column_2 = random.sample(time_functions,1)+random.sample(column_34,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_2 = str(time_fun_2).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s).replace("_1","") + + fun_fix_column_j = ['(t1.q_ts)','(t1.ts)', '(t2.q_ts)','(t2.ts)','1600000000000','1600000000000000','1600000000000000000', + '%d' %t, '%d000' %t, '%d000000' %t,'t_to_s'] + + column_j1,column_j2 = random.sample(fun_fix_column_j,1),random.sample(fun_fix_column_j,1) + column_j12 = ['(%s,%s,timeutil)'%(column_j1,column_j2)] + fun_column_join_1 = random.sample(time_functions,1)+random.sample(column_j12,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_join_1 = str(time_fun_join_1).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s).replace("_1","") + + column_j3,column_j4 = random.sample(fun_fix_column_j,1),random.sample(fun_fix_column_j,1) + column_j34 = ['(%s,%s,timeutil)'%(column_j3,column_j4)] + fun_column_join_2 = random.sample(time_functions,1)+random.sample(column_j34,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_join_2 = str(time_fun_join_2).replace("timeutil","%s" %timeunit).replace("t_to_s","%s" %t_to_s).replace("_1","") + + elif (timelist == ['TIMEDIFF_2']): + time_functions = timelist + + t = time.time() + t_to_s = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(t)) + + fun_fix_column = ['q_ts','ts','_c0','_C0','_rowts','1600000000000','1600000000000000','1600000000000000000', + '%d' %t, '%d000' %t, '%d000000' %t,'t_to_s'] + + column_1,column_2 = random.sample(fun_fix_column,1),random.sample(fun_fix_column,1) + column_12 = ['(%s,%s)'%(column_1,column_2)] + fun_column_1 = random.sample(time_functions,1)+random.sample(column_12,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_1 = str(time_fun_1).replace("t_to_s","%s" %t_to_s).replace("_2","") + + column_3,column_4 = random.sample(fun_fix_column,1),random.sample(fun_fix_column,1) + column_34 = ['(%s,%s)'%(column_3,column_4)] + fun_column_2 = random.sample(time_functions,1)+random.sample(column_34,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_2 = str(time_fun_2).replace("t_to_s","%s" %t_to_s).replace("_2","") + + fun_fix_column_j = ['(t1.q_ts)','(t1.ts)', '(t2.q_ts)','(t2.ts)','1600000000000','1600000000000000','1600000000000000000', + '%d' %t, '%d000' %t, '%d000000' %t,'t_to_s'] + + column_j1,column_j2 = random.sample(fun_fix_column_j,1),random.sample(fun_fix_column_j,1) + column_j12 = ['(%s,%s)'%(column_j1,column_j2)] + fun_column_join_1 = random.sample(time_functions,1)+random.sample(column_j12,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_join_1 = str(time_fun_join_1).replace("t_to_s","%s" %t_to_s).replace("_2","") + + column_j3,column_j4 = random.sample(fun_fix_column_j,1),random.sample(fun_fix_column_j,1) + column_j34 = ['(%s,%s)'%(column_j3,column_j4)] + fun_column_join_2 = random.sample(time_functions,1)+random.sample(column_j34,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("\"","").replace("t_to_s","'t_to_s'") + time_fun_join_2 = str(time_fun_join_2).replace("t_to_s","%s" %t_to_s).replace("_2","") + + elif (timelist == ['ELAPSED']): + time_functions = timelist + + fun_fix_column = ['(ts)','(_c0)','(_C0)','(_rowts)','(ts,time_unit)','(_c0,time_unit)','(_C0,time_unit)','(_rowts,time_unit)'] + + time_units = ['1s','1m','1h','1d','1a'] + time_unit1 = str(random.sample(time_units,1)).replace("[","").replace("]","").replace("'","") + time_unit2 = str(random.sample(time_units,1)).replace("[","").replace("]","").replace("'","") + + fun_column_1 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("time_unit","%s" %time_unit1) + + fun_column_2 = random.sample(time_functions,1)+random.sample(fun_fix_column,1) + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("time_unit","%s" %time_unit2) + + + fun_fix_column_j = ['(t1.ts)', '(t2.ts)','(t1.ts,time_unit)','(t1.ts,time_unit)','(t2.ts,time_unit)','(t2.ts,time_unit)'] + + fun_column_join_1 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("time_unit","%s" %time_unit1) + + fun_column_join_2 = random.sample(time_functions,1)+random.sample(fun_fix_column_j,1) + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("time_unit","%s" %time_unit2) + + + elif (timelist == ['CAST']) : + str_functions = timelist + #下面的4个是全的,这个只是1个 + i = random.randint(1,4) + if i ==1: + print('===========cast_1===========') + fun_fix_column = ['q_bool','q_bool_null','q_bigint','q_bigint_null','q_smallint','q_smallint_null', + 'q_tinyint','q_tinyint_null','q_int','q_int_null','q_float','q_float_null','q_double','q_double_null'] + type_names = ['BIGINT','BINARY(100)','TIMESTAMP','NCHAR(100)','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_j = ['t1.q_bool','t1.q_bool_null','t1.q_bigint','t1.q_bigint_null','t1.q_smallint','t1.q_smallint_null', + 't1.q_tinyint','t1.q_tinyint_null','t1.q_int','t1.q_int_null','t1.q_float','t1.q_float_null','t1.q_double','t1.q_double_null', + 't2.q_bool','t2.q_bool_null','t2.q_bigint','t2.q_bigint_null','t2.q_smallint','t2.q_smallint_null', + 't2.q_tinyint','t2.q_tinyint_null','t2.q_int','t2.q_int_null','t2.q_float','t2.q_float_null','t2.q_double','t2.q_double_null'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","") + + elif i==2: + print('===========cast_2===========') + fun_fix_column = ['q_binary','q_binary_null','q_binary1','q_binary2','q_binary3','q_binary4'] + type_names = ['BIGINT','BINARY(100)','NCHAR(100)','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_j = ['t1.q_binary','t1.q_binary_null','t1.q_binary1','t1.q_binary2','t1.q_binary3','t1.q_binary4', + 't2.q_binary','t2.q_binary_null','t2.q_binary1','t2.q_binary2','t2.q_binary3','t2.q_binary4'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","") + + elif i==3: + print('===========cast_3===========') + fun_fix_column = ['q_nchar','q_nchar_null','q_nchar5','q_nchar6','q_nchar7','q_nchar8'] + type_names = ['BIGINT','NCHAR(100)','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_j = ['t1.q_nchar','t1.q_nchar_null','t1.q_nchar5','t1.q_nchar6','t1.q_nchar7','t1.q_nchar8', + 't2.q_nchar','t2.q_nchar_null','t2.q_nchar5','t2.q_nchar6','t2.q_nchar7','t2.q_nchar8'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","") + + elif i==4: + print('===========cast_4===========') + fun_fix_column = ['q_ts','q_ts_null','_C0','_c0','ts','_rowts'] + type_names = ['BIGINT','TIMESTAMP','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","") + + fun_fix_column_j = ['t1.q_ts','t1.q_ts_null','t1.ts','t2.q_ts','t2.q_ts_null','t2.ts'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","") + + elif (timelist == ['CAST_1']) : + str_functions = timelist + + print('===========cast_1===========') + fun_fix_column = ['q_bool','q_bool_null','q_bigint','q_bigint_null','q_smallint','q_smallint_null', + 'q_tinyint','q_tinyint_null','q_int','q_int_null','q_float','q_float_null','q_double','q_double_null'] + type_names = ['BIGINT','BINARY(100)','TIMESTAMP','NCHAR(100)','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace("_1","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace("_1","") + + fun_fix_column_j = ['t1.q_bool','t1.q_bool_null','t1.q_bigint','t1.q_bigint_null','t1.q_smallint','t1.q_smallint_null', + 't1.q_tinyint','t1.q_tinyint_null','t1.q_int','t1.q_int_null','t1.q_float','t1.q_float_null','t1.q_double','t1.q_double_null', + 't2.q_bool','t2.q_bool_null','t2.q_bigint','t2.q_bigint_null','t2.q_smallint','t2.q_smallint_null', + 't2.q_tinyint','t2.q_tinyint_null','t2.q_int','t2.q_int_null','t2.q_float','t2.q_float_null','t2.q_double','t2.q_double_null'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace("_1","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace("_1","") + + elif (timelist == ['CAST_2']) : + str_functions = timelist + print('===========cast_2===========') + fun_fix_column = ['q_binary','q_binary_null','q_binary1','q_binary2','q_binary3','q_binary4'] + type_names = ['BIGINT','BINARY(100)','NCHAR(100)','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace("_2","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace("_2","") + + fun_fix_column_j = ['t1.q_binary','t1.q_binary_null','t1.q_binary1','t1.q_binary2','t1.q_binary3','t1.q_binary4', + 't2.q_binary','t2.q_binary_null','t2.q_binary1','t2.q_binary2','t2.q_binary3','t2.q_binary4'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace("_2","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace("_2","") + + elif (timelist == ['CAST_3']) : + str_functions = timelist + print('===========cast_3===========') + fun_fix_column = ['q_nchar','q_nchar_null','q_nchar5','q_nchar6','q_nchar7','q_nchar8'] + type_names = ['BIGINT','NCHAR(100)','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace("_3","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace("_3","") + + fun_fix_column_j = ['t1.q_nchar','t1.q_nchar_null','t1.q_nchar5','t1.q_nchar6','t1.q_nchar7','t1.q_nchar8', + 't2.q_nchar','t2.q_nchar_null','t2.q_nchar5','t2.q_nchar6','t2.q_nchar7','t2.q_nchar8'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace("_3","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace("_3","") + + elif (timelist == ['CAST_4']) : + str_functions = timelist + print('===========cast_4===========') + fun_fix_column = ['q_ts','q_ts_null','_C0','_c0','ts','_rowts'] + type_names = ['BIGINT','TIMESTAMP','BIGINT UNSIGNED'] + + type_name1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name1+')' + time_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace("_4","") + + type_name2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column,1))+' AS '+type_name2+')' + time_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace("_4","") + + fun_fix_column_j = ['t1.q_ts','t1.q_ts_null','t1.ts','t2.q_ts','t2.q_ts_null','t2.ts'] + + type_name_j1 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_1 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j1+')' + time_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace("_4","") + + type_name_j2 = str(random.sample(type_names,1)).replace("[","").replace("]","").replace("'","") + fun_column_join_2 = str(random.sample(str_functions,1))+'('+str(random.sample(fun_fix_column_j,1))+' AS '+type_name_j2+')' + time_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace("_4","") + + tdSql.query("select 1-1 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1 , timediff(asct1,now) from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts1 from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) \ + or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts2 , asct1,now(),today(),timezone() from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts2 from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select max(asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += "from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-2 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts , timediff(asct1,now),now(),today(),timezone() from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s )" % random.choice(self.order_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select ts , timediff(asct2,now),now(),today(),timezone() from ( select " + sql += "%s as asct2, " % time_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + #sql += "%s " % random.choice(having_support) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts , (asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s )" % random.choice(self.order_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select ts , asct2,now(),today(),timezone() from ( select " + sql += "%s as asct2, " % time_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select min(asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1 " % time_fun_1 + sql += " from regular_table_1 where " + sql += "%s )" % random.choice(self.q_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select avg(asct2),now(),today(),timezone() from ( select " + sql += "%s as asct2 " % time_fun_2 + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-3 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts , timediff(asct1,now) from ( select " + sql += "%s as asct1, ts ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s " % random.choice(self.q_select) + sql += "from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2, ts ," % time_fun_2 + sql += "%s as asct1, " % time_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s " % random.choice(self.q_select) + sql += "from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += " order by asct1 desc " + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts , (asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1, ts ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s " % random.choice(self.q_select) + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2, ts ," % time_fun_2 + sql += "%s as asct1, " % time_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s " % random.choice(self.q_select) + sql += "from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += " order by asct1 desc " + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select abs(asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1," % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += "from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2," % time_fun_2 + sql += "%s as asct1 " % time_fun_1 + sql += "from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += " order by asct1 asc " + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-4 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), timediff(asct1,now) from ( select t1.ts as ts1," + sql += "%s as asct11, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "%s as asct12, " % time_fun_join_1 + sql += "t1.%s as asct111, " % random.choice(self.q_select) + sql += "t2.%s as asct121, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), (asct1) from ( select t1.ts as ts1," + sql += "%s as asct10, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "%s as asct11, " % time_fun_join_1 + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select floor(asct1) from ( select " + sql += "%s as asct10, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "%s as asct11" % time_fun_join_1 + sql += " from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-5 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['ELAPSED']) : + sql = "select now(),today(),timezone(), " + sql += "%s, " % time_fun_1 + sql += "%s " % time_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + else: + sql = "select ts ,now(),today(),timezone(), " + sql += "%s as asct11, " % time_fun_1 + sql += "%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % random.choice(self.q_select) + sql += "%s as asct14 " % time_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-6 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1,ts ,timediff(ts1,ts), timediff(asct1,now) from ( select t1.ts as ts1," + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t2.%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % time_fun_join_1 + sql += "t2.ts from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts1,ts ,timediff(ts1,ts), (asct1) from ( select t1.ts as ts1," + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t2.%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % time_fun_join_1 + sql += "t2.ts from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select (asct1)*111 from ( select " + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "%s as asct122 " % time_fun_join_1 + sql += " from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-7 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1,m1 , timediff(asct1,now) from ( select " + sql += "%s as asct1, ts as m1," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.t_select) + sql += "ts as ts1 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select tm1,tm2 , (asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1, ts as tm1," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.t_select) + sql += "ts as tm2 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select (asct1)/asct2 ,now(),today(),timezone() from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += "from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-8 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select tm1,tm2 , timediff(asct1,now) " + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s as asct1, ts as tm1," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.t_select) + sql += "ts as tm2 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts1,ts2 , (asct1),now(),today(),timezone() " + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s as asct1, ts as ts1," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.t_select) + sql += "ts as ts2 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select floor(abs(asct1)),now(),today(),timezone() " + sql += "from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += "from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-9 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), timediff(asct1,now) from ( select t1.ts as ts1," + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), asct1 from ( select t1.ts as ts1," + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select min(asct1*110) from ( select " + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1 " % time_fun_join_2 + sql += "from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + self.restartDnodes() + tdSql.query("select 1-10 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts , timediff(asct1,now) from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select ts , timediff(asct1,now) from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts , (asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select ts , (asct2),now(),today(),timezone() from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select abs(asct1),now(),today(),timezone() from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select max(asct2),now(),today(),timezone() from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += "from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #3 inter union not support + tdSql.query("select 1-11 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts , timediff(asct1,now), timediff(now,asct2) from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts , asct1,now(),now(),asct2 from ( select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2, " % time_fun_2 + sql += "%s as asct121, " % random.choice(self.s_r_select) + sql += "%s as asct122, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select asct1+asct2,now(),today(),timezone() from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1 ," % time_fun_1 + sql += "%s as asct2 " % time_fun_2 + sql += " from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "order by asct1 " + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-12 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), timediff(asct1,now) from ( select t1.ts as ts1," + sql += "%s, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts1,ts2 ,timediff(ts1,ts2), asct1,now() from ( select t1.ts as ts1," + sql += "%s, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts as ts2 from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select min(floor(asct1)),now() from ( select " + sql += "%s as asct121, " % time_fun_join_1 + sql += "%s as asct1 " % time_fun_join_2 + sql += " from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-13 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts , timediff(%s,now)," % time_fun_2 + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % time_fun_2 + sql += "%s as asct122 " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts ,now(),today(),timezone(), " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct11, " % random.choice(self.q_select) + sql += "%s as asct12, " % random.choice(self.q_select) + sql += "%s as asct13, " % time_fun_2 + sql += "%s as asct122 " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select now(),today(),timezone(), " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct12 " % time_fun_2 + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-14 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts , timediff(asct1,now),timediff(now,asct2) from ( select ts ts ," + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2" % time_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts , (asct1),now(),(now()),asct2 from ( select ts ts ," + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2" % time_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select (asct1)*asct2,now(),(now()) from ( select " + sql += "%s as asct1, " % time_fun_1 + sql += "%s as asct2" % time_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-15 as time_nest from stable_1 limit 1;") + for i in range(self.fornum): + if (timelist == ['NOW','TODAY']) or (timelist == ['TIMETRUNCATE']) or (timelist == ['TO_ISO8601'])\ + or (timelist == ['TO_UNIXTIMESTAMP']) or (timelist == ['TIMEDIFF_1']) or (timelist == ['TIMEDIFF_2']): + sql = "select ts1,ts ,timediff(ts1,ts), timediff(asct1,now),timediff(now,asct2) from ( select t1.ts as ts1," + sql += "%s as asct2, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['TIMEZONE']) or (timelist == ['CAST']) or (timelist == ['CAST_1']) or (timelist == ['CAST_2']) or (timelist == ['CAST_3']) or (timelist == ['CAST_4']): + sql = "select ts1,ts ,timediff(ts1,ts), asct1,(now()),(now()),asct2 ,now(),today(),timezone() from ( select t1.ts as ts1," + sql += "%s as asct2, " % time_fun_join_1 + sql += "%s as asct1, " % time_fun_join_2 + sql += "t1.%s as asct11, " % random.choice(self.q_select) + sql += "t1.%s as asct12, " % random.choice(self.q_select) + sql += "t2.%s as asct13, " % random.choice(self.q_select) + sql += "t2.%s as asct14, " % random.choice(self.q_select) + sql += "t2.ts from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + elif (timelist == ['ELAPSED']) : + sql = "select asct1,(now()),(now()),asct2 ,now(),today(),timezone() from ( select " + sql += "%s as asct2, " % time_fun_join_1 + sql += "%s as asct1 " % time_fun_join_2 + sql += "from stable_1 t1, stable_2 t2 where t1.ts = t2.ts " + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #taos -f sql + startTime_taos_f = time.time() + print("taos -f %s sql start!" %timelist) + taos_cmd1 = "taos -f %s/%s.sql" % (self.testcasePath,self.testcaseFilename) + #_ = subprocess.check_output(taos_cmd1, shell=True).decode("utf-8") + _ = subprocess.check_output(taos_cmd1, shell=True) + print("taos -f %s sql over!" %timelist) + endTime_taos_f = time.time() + print("taos_f total time %ds" % (endTime_taos_f - startTime_taos_f)) + + print("=========%s====over=============" %timelist) + + def base_nest(self,baselist): + + print("==========%s===start=============" %baselist) + os.system("rm -rf %s/%s.sql" % (self.testcasePath,self.testcaseFilename)) + + self.dropandcreateDB_random("%s" %self.db_nest, 1) + + if (baselist == ['A']) or (baselist == ['S']) or (baselist == ['F']) \ + or (baselist == ['C']): + base_functions = baselist + fun_fix_column = ['(q_bigint)','(q_smallint)','(q_tinyint)','(q_int)','(q_float)','(q_double)','(q_bigint_null)','(q_smallint_null)','(q_tinyint_null)','(q_int_null)','(q_float_null)','(q_double_null)'] + fun_column_1 = random.sample(base_functions,1)+random.sample(fun_fix_column,1) + base_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_2 = random.sample(base_functions,1)+random.sample(fun_fix_column,1) + base_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + fun_fix_column_j = ['(t1.q_bigint)','(t1.q_smallint)','(t1.q_tinyint)','(t1.q_int)','(t1.q_float)','(t1.q_double)','(t1.q_bigint_null)','(t1.q_smallint_null)','(t1.q_tinyint_null)','(t1.q_int_null)','(t1.q_float_null)','(t1.q_double_null)', + '(t2.q_bigint)','(t2.q_smallint)','(t2.q_tinyint)','(t2.q_int)','(t2.q_float)','(t2.q_double)','(t2.q_bigint_null)','(t2.q_smallint_null)','(t2.q_tinyint_null)','(t2.q_int_null)','(t2.q_float_null)','(t2.q_double_null)'] + fun_column_join_1 = random.sample(base_functions,1)+random.sample(fun_fix_column_j,1) + base_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","") + fun_column_join_2 = random.sample(base_functions,1)+random.sample(fun_fix_column_j,1) + base_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","") + + elif (baselist == ['P']) or (baselist == ['M']) or (baselist == ['S'])or (baselist == ['T']): + base_functions = baselist + num = random.randint(0, 1000) + fun_fix_column = ['(q_bigint,num)','(q_smallint,num)','(q_tinyint,num)','(q_int,num)','(q_float,num)','(q_double,num)', + '(q_bigint_null,num)','(q_smallint_null,num)','(q_tinyint_null,num)','(q_int_null,num)','(q_float_null,num)','(q_double_null,num)'] + fun_column_1 = random.sample(base_functions,1)+random.sample(fun_fix_column,1) + base_fun_1 = str(fun_column_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",base(num)) + fun_column_2 = random.sample(base_functions,1)+random.sample(fun_fix_column,1) + base_fun_2 = str(fun_column_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",base(num)) + + fun_fix_column_j = ['(t1.q_bigint,num)','(t1.q_smallint,num)','(t1.q_tinyint,num)','(t1.q_int,num)','(t1.q_float,num)','(t1.q_double,num)', + '(t1.q_bigint_null,num)','(t1.q_smallint_null,num)','(t1.q_tinyint_null,num)','(t1.q_int_null,num)','(t1.q_float_null,num)','(t1.q_double_null,num)', + '(t2.q_bigint,num)','(t2.q_smallint,num)','(t2.q_tinyint,num)','(t2.q_int,num)','(t2.q_float,num)','(t2.q_double,num)', + '(t2.q_bigint_null,num)','(t2.q_smallint_null,num)','(t2.q_tinyint_null,num)','(t2.q_int_null,num)','(t2.q_float_null,num)','(t2.q_double_null,num)'] + fun_column_join_1 = random.sample(base_functions,1)+random.sample(fun_fix_column_j,1) + base_fun_join_1 = str(fun_column_join_1).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",base(num)) + fun_column_join_2 = random.sample(base_functions,1)+random.sample(fun_fix_column_j,1) + base_fun_join_2 = str(fun_column_join_2).replace("[","").replace("]","").replace("'","").replace(", ","").replace("num",base(num)) + + tdSql.query("select 1-1 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , floor(asct1) from ( select " + sql += "%s as asct1, " % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + + tdSql.query("select 1-2 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , abs(asct1) from ( select " + sql += "%s as asct1, " % base_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s )" % random.choice(self.order_where) + sql += "%s " % random.choice(self.unionall_or_union) + sql += "select ts , asct2 from ( select " + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + #sql += "%s " % random.choice(having_support) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + #tdSql.query(sql) + #self.cur1.execute(sql) + + tdSql.query("select 1-3 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , min(asct1) from ( select " + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s select " % random.choice(self.unionall_or_union) + sql += "%s as asct2, ts ," % base_fun_2 + sql += "%s as asct1, " % base_fun_1 + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + #tdSql.query(sql) + #self.cur1.execute(sql) + + tdSql.query("select 1-4 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts1,ts2 ,timediff(ts1,ts2), asct1 from ( select t1.ts as ts1," + sql += "%s, " % base_fun_join_1 + sql += "%s as asct1, " % base_fun_join_2 + sql += "%s, " % base_fun_join_1 + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.ts from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + + tdSql.query("select 1-5 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts ," + sql += "%s, " % base_fun_1 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.q_select) + sql += "%s " % base_fun_2 + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + + tdSql.query("select 1-6 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % base_fun_join_1 + sql += "%s as asct1, " % base_fun_join_2 + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "%s, " % base_fun_join_1 + sql += "t2.ts from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s )" % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + + tdSql.query("select 1-7 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , abs(asct1) from ( select " + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + + tdSql.query("select 1-8 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts,floor(asct1) " + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + + tdSql.query("select 1-9 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % base_fun_join_1 + sql += "%s as asct1, " % base_fun_join_2 + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.ts from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "and %s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + + self.restartDnodes() + tdSql.query("select 1-10 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , min(asct1) from ( select " + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") %s " % random.choice(self.unionall_or_union) + sql += "select ts , max(asct2) from ( select " + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + #tdSql.query(sql) + #self.cur1.execute(sql) + + #3 inter union not support + tdSql.query("select 1-11 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , min(asct1), max(asct2) from ( select " + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + #sql += "%s " % random.choice(limit1_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s as asct1, ts ," % base_fun_1 + sql += "%s as asct2, " % base_fun_2 + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + #TD-15837 tdSql.query(sql) + # self.cur1.execute(sql) + + tdSql.query("select 1-12 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % base_fun_join_1 + sql += "%s as asct1, " % base_fun_join_2 + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.ts from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "and %s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + + tdSql.query("select 1-13 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts ," + sql += "%s, " % base_fun_1 + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.q_select) + sql += "%s " % base_fun_2 + sql += "%s " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + + tdSql.query("select 1-14 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select avg(asct1),count(asct2) from ( select " + sql += "%s as asct1, " % base_fun_1 + sql += "%s as asct2" % base_fun_2 + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ) ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + + tdSql.query("select 1-15 as base_nest from stable_1 limit 1;") + for i in range(self.fornum): + sql = "select ts , max(asct1) from ( select t1.ts as ts1," + sql += "%s, " % base_fun_join_1 + sql += "%s as asct1, " % base_fun_join_2 + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.%s " % random.choice(self.q_select) + sql += "from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + + #taos -f sql + startTime_taos_f = time.time() + print("taos -f %s sql start!" %baselist) + taos_cmd1 = "taos -f %s/%s.sql" % (self.testcasePath,self.testcaseFilename) + _ = subprocess.check_output(taos_cmd1, shell=True).decode("utf-8") + print("taos -f %s sql over!" %baselist) + endTime_taos_f = time.time() + print("taos_f total time %ds" % (endTime_taos_f - startTime_taos_f)) + + print("=========%s====over=============" %baselist) + + def function_before_26(self): + + print('=====================2.6 old function start ===========') + os.system("rm -rf %s/%s.sql" % (self.testcasePath,self.testcaseFilename)) + + self.dropandcreateDB_random("%s" %self.db_nest, 1) + + #1 select * from (select column form regular_table where <\>\in\and\or order by) + tdSql.query("select 1-1 from stable_1;") + for i in range(self.fornum): + sql = "select tas from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as tas from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql,queryTimes=1) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + #1 outer union not support + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 1-2 from stable_1;") + for i in range(self.fornum): + #sql = "select ts , * from ( select " + sql = "select t1s from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as t1s from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") union " + sql += "select t2s from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as t2s from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 1-2 from stable_1;") + for i in range(self.fornum): + sql = "select ts from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") union all " + sql += "select ts from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(200) + self.cur1.execute(sql) + self.explain_sql(sql) + + #1 inter union not support + tdSql.query("select 1-3 from stable_1;") + for i in range(self.fornum): + #sql = "select ts , * from ( select " + sql = "select ts from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "" + sql += " union all select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 1-3 from stable_1;") + for i in range(self.fornum): + sql = "select ts from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " union all select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from regular_table_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + + #join:select * from (select column form regular_table1,regular_table2 where t1.ts=t2.ts and <\>\in\and\or order by) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 1-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select t1.ts as t1ts," + sql += "t1.%s as t11, " % random.choice(self.q_select) + sql += "t1.%s as t12, " % random.choice(self.q_select) + sql += "t2.%s as t21, " % random.choice(self.q_select) + sql += "t2.%s as t22, " % random.choice(self.q_select) + sql += "t2.ts as t2ts from regular_table_1 t1 , regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + + #2 select column from (select * form regular_table ) where <\>\in\and\or order by + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 2-1 from stable_1;") + for i in range(self.fornum): + sql = "select ts ," + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s " % random.choice(self.q_select) + sql += " from ( select * from regular_table_1 ) where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + #join: select column from (select column form regular_table1,regular_table2 )where t1.ts=t2.ts and <\>\in\and\or order by + #cross join not supported yet + tdSql.query("select 2-2 from stable_1;") + for i in range(self.fornum): + sql = "select ts , * from ( select t1.ts ," + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t1.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.%s, " % random.choice(self.q_select) + sql += "t2.ts from regular_table_1 t1 , regular_table_2 t2 ) where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.order_u_where) + #sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + #3 select * from (select column\tag form stable where <\>\in\and\or order by ) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 3-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + tdSql.query("select 3-1 from stable_1;") + for i in range(self.fornum): + sql = "select ts, " + sql += "%s " % random.choice(self.s_r_select) + sql += "from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.q_select) + sql += "%s, " % random.choice(self.t_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + + # select ts,* from (select column\tag form stable1,stable2 where t1.ts = t2.ts and <\>\in\and\or order by ) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 3-2 from stable_1;") + for i in range(self.fornum): + #sql = "select ts , * from ( select t1.ts as t1ts , " + sql = "select t1ts , t2ts from ( select t1.ts as t1ts , " + sql += "t1.%s as t11, " % random.choice(self.t_select) + sql += "t1.%s as t12, " % random.choice(self.q_select) + sql += "t2.%s as t13, " % random.choice(self.t_select) + sql += "t2.%s as t14, " % random.choice(self.q_select) + sql += "t2.ts as t2ts from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #3 outer union not support + self.restartDnodes() + tdSql.query("select 3-3 from stable_1;") + for i in range(self.fornum): + #sql = "select ts , * from ( select " + sql = "select ts1 from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts1 from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") union " + sql += "select ts2 from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts2 from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(500) + self.cur1.execute(sql) + self.explain_sql(sql) + for i in range(self.fornum): + sql = "select ts1 from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts1 from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ") union all " + sql += "select ts2 from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts2 from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(600) + self.cur1.execute(sql) + self.explain_sql(sql) + + #3 inter union not support + tdSql.query("select 3-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += " %s " % random.choice(self.unionall_or_union) + sql += " select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts from stable_2 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += ")" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #join:select * from (select column form stable1,stable2 where t1.ts=t2.ts and <\>\in\and\or order by) + tdSql.query("select 3-5 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select t1.ts as t1ts," + sql += "t1.%s as t11, " % random.choice(self.q_select) + sql += "t1.%s as t12, " % random.choice(self.q_select) + sql += "t2.%s as t21, " % random.choice(self.q_select) + sql += "t2.%s as t22, " % random.choice(self.q_select) + sql += "t2.ts as t2ts from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 3-6 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select t1.ts as t1ts ," + sql += "t1.%s as t11, " % random.choice(self.q_select) + sql += "t1.%s as t12, " % random.choice(self.q_select) + sql += "t2.%s as t21, " % random.choice(self.q_select) + sql += "t2.%s as t22, " % random.choice(self.q_select) + sql += "t2.ts as t2ts from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += ");" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(100) + self.cur1.execute(sql) + self.explain_sql(sql) + + #4 select column from (select * form stable where <\>\in\and\or order by ) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 4-1 from stable_1;") + for i in range(self.fornum): + sql = "select ts , " + sql += "%s as t11, " % random.choice(self.q_select) + sql += "%s as t12, " % random.choice(self.q_select) + sql += "%s " % random.choice(self.t_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(300) + self.cur1.execute(sql) + self.explain_sql(sql) + + #5 select distinct column\tag from (select * form stable where <\>\in\and\or order by limit offset ) + tdSql.query("select 5-1 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.dqt_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #5-1 select distinct column\tag from (select calc form stable where <\>\in\and\or order by limit offset ) + tdSql.query("select 5-2 from stable_1;") + for i in range(self.fornum): + sql = "select distinct c5_1 " + sql += " from ( select " + sql += "%s " % random.choice(self.calc_select_in_ts) + sql += " as c5_1 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #6-error select * from (select distinct(tag) form stable where <\>\in\and\or order by limit ) + tdSql.query("select 6-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.dt_select) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + tdSql.query("select 6-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.dt_select) + sql += " from stable_1 where " + sql += "%s ) ;" % random.choice(self.qt_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #7-error select * from (select distinct(tag) form stable where <\>\in\and\or order by limit ) + tdSql.query("select 7-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.dq_select) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[0] , self.limit_where[1]] ) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + tdSql.query("select 7-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.dq_select) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice([self.limit_where[0] , self.limit_where[1]] ) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + tdSql.checkRows(1) + self.cur1.execute(sql) + self.explain_sql(sql) + + #calc_select,TWA/Diff/Derivative/Irate are not allowed to apply to super table directly + #8 select * from (select ts,calc form ragular_table where <\>\in\and\or order by ) + + # dcDB = self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 8-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select ts ," + sql += "%s " % random.choice(self.calc_select_support_ts) + sql += "from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + tdSql.query("select 8-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_select_not_support_ts) + sql += "from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_select_in_ts) + sql += "from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 8-2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select t1.ts, " + sql += "%s " % random.choice(self.calc_select_in_support_ts_j) + sql += "from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_select_in_not_support_ts_j) + sql += "from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #9 select * from (select ts,calc form stable where <\>\in\and\or order by ) + # self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 9-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_select_not_support_ts) + sql += "from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + tdSql.query("select 9-2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select ts ," + sql += "%s " % random.choice(self.calc_select_support_ts) + sql += "from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 9-3 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_select_in_not_support_ts_j) + sql += "from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + tdSql.query("select 9-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select t1.ts," + sql += "%s " % random.choice(self.calc_select_in_support_ts_j) + sql += "from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += " and %s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #10 select calc from (select * form regualr_table where <\>\in\and\or order by ) + tdSql.query("select 10-1 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_select_in_ts) + sql += "as calc10_1 from ( select * from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #10-1 select calc from (select * form regualr_table where <\>\in\and\or order by ) + # rsDn = self.restartDnodes() + # self.dropandcreateDB_random("%s" %db, 1) + # rsDn = self.restartDnodes() + tdSql.query("select 10-2 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_select_all) + sql += "as calc10_2 from ( select * from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #10-2 select calc from (select * form regualr_tables where <\>\in\and\or order by ) + tdSql.query("select 10-3 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "count(*) as calc10_3 " + sql += " from ( select t1.ts as t11, t2.ts as t22 from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += " and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 10-4 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s as calc10_4 " % random.choice(self.calc_select_all) + sql += " from ( select * from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += " and %s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + #11 select calc from (select * form stable where <\>\in\and\or order by limit ) + tdSql.query("select 11-1 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_select_in_ts) + sql += "as calc11_1 from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #11-1 select calc from (select * form stable where <\>\in\and\or order by limit ) + tdSql.query("select 11-2 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_select_all) + sql += "as calc11_1 from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #11-2 select calc from (select * form stables where <\>\in\and\or order by limit ) + tdSql.query("select 11-3 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_select_all) + sql += "as calc11_1 from ( select * from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + tdSql.query("select 11-4 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_select_all) + sql += "as calc11_1 from ( select * from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + #12 select calc-diff from (select * form regualr_table where <\>\in\and\or order by limit ) + ##self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 12-1 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_calculate_regular) + sql += " from ( select * from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 12-2 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_calculate_regular) + sql += " from ( select * from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + tdSql.query("select 12-2.2 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_calculate_regular) + sql += " from ( select * from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + #12-1 select calc-diff from (select * form stable where <\>\in\and\or order by limit ) + tdSql.query("select 12-3 from stable_1;") + self.restartDnodes() + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_calculate_regular) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += ") " + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 12-4 from stable_1;") + #join query does not support group by + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_calculate_regular_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts " + sql += "%s " % random.choice(self.partiton_where_j) + sql += ") " + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 12-5 from stable_1;") + #join query does not support group by + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_calculate_regular_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += ") " + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += " ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + + #13 select calc-diff as diffns from (select * form stable where <\>\in\and\or order by limit ) + tdSql.query("select 13-1 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_calculate_regular) + sql += " as calc13_1 from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.orders_desc_where) + sql += "%s " % random.choice([self.limit_where[2] , self.limit_where[3]] ) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #14 select * from (select calc_aggregate_alls as agg from stable where <\>\in\and\or group by order by slimit soffset ) + tdSql.query("select 14-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc14_1, " % random.choice(self.calc_aggregate_all) + sql += "%s as calc14_2, " % random.choice(self.calc_aggregate_all) + sql += "%s " % random.choice(self.calc_aggregate_all) + sql += " as calc14_3 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.group_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + # error group by in out query + tdSql.query("select 14-2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc14_1, " % random.choice(self.calc_aggregate_all) + sql += "%s as calc14_2, " % random.choice(self.calc_aggregate_all) + sql += "%s " % random.choice(self.calc_aggregate_all) + sql += " as calc14_3 from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.partiton_where_regular) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #14-2 select * from (select calc_aggregate_all_js as agg from stables where <\>\in\and\or group by order by slimit soffset ) + tdSql.query("select 14-3 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc14_1, " % random.choice(self.calc_aggregate_all_j) + sql += "%s as calc14_2, " % random.choice(self.calc_aggregate_all_j) + sql += "%s " % random.choice(self.calc_aggregate_all_j) + sql += " as calc14_3 from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 14-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc14_1, " % random.choice(self.calc_aggregate_all_j) + sql += "%s as calc14_2, " % random.choice(self.calc_aggregate_all_j) + sql += "%s " % random.choice(self.calc_aggregate_all_j) + sql += " as calc14_3 from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.partiton_where_j) + sql += "%s " % random.choice(self.slimit1_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #15 select * from (select calc_aggregate_regulars as agg from regular_table where <\>\in\and\or order by slimit soffset ) + tdSql.query("select 15-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_1, " % random.choice(self.calc_aggregate_regular) + sql += "%s as calc15_2, " % random.choice(self.calc_aggregate_regular) + sql += "%s " % random.choice(self.calc_aggregate_regular) + sql += " as calc15_3 from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.group_where_regular) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + self.data_check(sql,mark='15-1') + + tdSql.query("select 15-2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_2 " % random.choice(self.calc_aggregate_regular_j) + sql += "from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.group_where_regular_j) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + self.data_check(sql,mark='15-2') + + tdSql.query("select 15-2.2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_1, " % random.choice(self.calc_aggregate_regular_j) + sql += "%s as calc15_2, " % random.choice(self.calc_aggregate_regular_j) + sql += "%s " % random.choice(self.calc_aggregate_regular_j) + sql += " as calc15_3 from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.group_where_regular_j) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + sql += "%s ;" % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + self.data_check(sql,mark='15-2.2') + + self.restartDnodes() + tdSql.query("select 15-3 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_1, " % random.choice(self.calc_aggregate_groupbytbname) + sql += "%s as calc15_2, " % random.choice(self.calc_aggregate_groupbytbname) + sql += "%s " % random.choice(self.calc_aggregate_groupbytbname) + sql += " as calc15_3 from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.group_only_where) + sql += "%s " % random.choice(self.having_support) + sql += ") " + sql += "order by calc15_1 " + sql += "%s " % random.choice(self.limit_where) + tdLog.info(sql) + tdLog.info(len(sql)) + self.data_check(sql,mark='15-3') + + tdSql.query("select 15-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_1, " % random.choice(self.calc_aggregate_groupbytbname_j) + sql += "%s as calc15_2, " % random.choice(self.calc_aggregate_groupbytbname_j) + sql += "%s " % random.choice(self.calc_aggregate_groupbytbname_j) + sql += " as calc15_3 from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.group_only_where_j) + sql += "%s " % random.choice(self.having_support_j) + sql += ") " + sql += "order by calc15_1 " + sql += "%s " % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + self.data_check(sql,mark='15-4') + + tdSql.query("select 15-4.2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_1, " % random.choice(self.calc_aggregate_groupbytbname_j) + sql += "%s as calc15_2, " % random.choice(self.calc_aggregate_groupbytbname_j) + sql += "%s " % random.choice(self.calc_aggregate_groupbytbname_j) + sql += " as calc15_3 from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.group_where_j) + sql += ") " + sql += "order by calc15_1 " + sql += "%s " % random.choice(self.limit_u_where) + tdLog.info(sql) + tdLog.info(len(sql)) + self.data_check(sql,mark='15-4.2') + + tdSql.query("select 15-5 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc15_1, " % random.choice(self.calc_aggregate_groupbytbname) + sql += "%s as calc15_2, " % random.choice(self.calc_aggregate_groupbytbname) + sql += "%s " % random.choice(self.calc_aggregate_groupbytbname) + sql += " as calc15_3 from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.group_where) + sql += ") " + sql += "order by calc15_1 " + sql += "%s " % random.choice(self.limit_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.data_check(sql,mark='15-5') + + #16 select * from (select calc_aggregate_regulars as agg from regular_table where <\>\in\and\or order by limit offset ) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 16-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_0 , " % random.choice(self.calc_calculate_all) + sql += "%s as calc16_1 , " % random.choice(self.calc_aggregate_all) + sql += "%s as calc16_2 " % random.choice(self.calc_select_in) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.group_where) + sql += ") " + sql += "order by calc16_0 " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_0 " % random.choice(self.calc_calculate_all_j) + sql += ", %s as calc16_1 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += ") " + sql += "order by calc16_0 " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-2.2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_0 " % random.choice(self.calc_calculate_all_j) + sql += ", %s as calc16_1 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += ") " + sql += "order by calc16_0 " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-3 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 " % random.choice(self.calc_calculate_regular) + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "limit 2 ) " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 " % random.choice(self.calc_calculate_regular_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "limit 2 ) " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-4.2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 " % random.choice(self.calc_calculate_regular_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "limit 2 ) " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-5 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 , " % random.choice(self.calc_calculate_all) + sql += "%s as calc16_2 , " % random.choice(self.calc_calculate_all) + sql += "%s as calc16_3 " % random.choice(self.calc_calculate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.group_where) + sql += ") " + sql += "order by calc16_1 " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-6 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 " % random.choice(self.calc_calculate_groupbytbname) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "limit 2 ) " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-7 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 " % random.choice(self.calc_calculate_groupbytbname_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts " + sql += "limit 2 ) " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 16-8 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s as calc16_1 " % random.choice(self.calc_calculate_groupbytbname_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "limit 2 ) " + sql += "%s " % random.choice(self.limit1_where) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #17 select apercentile from (select calc_aggregate_alls form regualr_table or stable where <\>\in\and\or interval_sliding group by having order by limit offset )interval_sliding + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 17-1 from stable_1;") + for i in range(self.fornum): + #this is having_support , but tag-select cannot mix with last_row,other select can + sql = "select apercentile(cal17_0, %d)/10 ,apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_0 , " % random.choice(self.calc_calculate_all) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-2 from stable_1;") + for i in range(self.fornum): + #this is having_support , but tag-select cannot mix with last_row,other select can + sql = "select apercentile(cal17_0, %d)/10 ,apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_0 , " % random.choice(self.calc_calculate_all_j) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-2.2 from stable_1;") + for i in range(self.fornum): + #this is having_support , but tag-select cannot mix with last_row,other select can + sql = "select apercentile(cal17_0, %d)/10 ,apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_0 , " % random.choice(self.calc_calculate_all_j) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + self.restartDnodes() + tdSql.query("select 17-3 from stable_1;") + for i in range(self.fornum): + #this is having_tagnot_support , because tag-select cannot mix with last_row... + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-4 from stable_1;") + for i in range(self.fornum): + #this is having_tagnot_support , because tag-select cannot mix with last_row... + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-4.2 from stable_1;") + for i in range(self.fornum): + #this is having_tagnot_support , because tag-select cannot mix with last_row... + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-5 from stable_1;") + for i in range(self.fornum): + #having_not_support + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.partiton_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-6 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-7 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1_1 t1, stable_1_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-7.2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1_1 t1, stable_1_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + self.restartDnodes() + tdSql.query("select 17-8 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all) + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-9 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 17-10 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal17_1, %d)/1000 ,apercentile(cal17_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal17_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal17_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.interval_sliding) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #18 select apercentile from (select calc_aggregate_alls form regualr_table or stable where <\>\in\and\or session order by limit )interval_sliding + tdSql.query("select 18-1 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all) + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.session_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 18-2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.session_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 18-2.2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.session_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + self.restartDnodes() + tdSql.query("select 18-3 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.session_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 18-4 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.session_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 18-4.2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.session_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 18-5 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.session_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 18-6 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.session_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + tdSql.query("select 18-7 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal18_1, %d)/1000 ,apercentile(cal18_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal18_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal18_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1, stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.session_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #19 select apercentile from (select calc_aggregate_alls form regualr_table or stable where <\>\in\and\or session order by limit )interval_sliding + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 19-1 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all) + sql += " from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.state_window) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.state_u_window) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-2.2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.state_u_window) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-3 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.state_window) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-4 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1_1 t1, stable_1_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-4.2 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1_1 t1, stable_1_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_or_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-5 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += "%s " % random.choice(self.state_window) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit1_where) + sql += ") " + sql += "%s " % random.choice(self.interval_sliding) + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + tdSql.query("select 19-6 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.q_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + tdSql.query("select 19-7 from stable_1;") + for i in range(self.fornum): + sql = "select apercentile(cal19_1, %d)/1000 ,apercentile(cal19_2, %d)*10+%d from ( select " %(random.randint(0,100) , random.randint(0,100) ,random.randint(-1000,1000)) + sql += "%s as cal19_1 ," % random.choice(self.calc_aggregate_all_j) + sql += "%s as cal19_2 " % random.choice(self.calc_aggregate_all_j) + sql += " from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #20 select * from (select calc_select_fills form regualr_table or stable where <\>\in\and\or fill_where group by order by limit offset ) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 20-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill) + sql += "%s ," % random.choice(self.calc_select_fill) + sql += "%s " % random.choice(self.calc_select_fill) + sql += " from stable_1 where " + sql += "%s " % random.choice(self.interp_where) + sql += "%s " % random.choice(self.fill_where) + sql += "%s " % random.choice(self.group_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + #interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + rsDn = self.restartDnodes() + tdSql.query("select 20-2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill_j) + sql += "%s ," % random.choice(self.calc_select_fill_j) + sql += "%s " % random.choice(self.calc_select_fill_j) + sql += " from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s and " % random.choice(self.t_join_where) + sql += "%s " % random.choice(self.interp_where_j) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + #interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + tdSql.query("select 20-2.2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill_j) + sql += "%s ," % random.choice(self.calc_select_fill_j) + sql += "%s " % random.choice(self.calc_select_fill_j) + sql += " from stable_1 t1 , stable_2 t2 where t1.ts = t2.ts and " + sql += "%s and " % random.choice(self.qt_u_or_where) + sql += "%s " % random.choice(self.interp_where_j) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + #interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + tdSql.query("select 20-3 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill) + sql += "%s ," % random.choice(self.calc_select_fill) + sql += "%s " % random.choice(self.calc_select_fill) + sql += " from stable_1 where " + sql += "%s " % self.interp_where[2] + sql += "%s " % random.choice(self.fill_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + #interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + tdSql.query("select 20-4 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill_j) + sql += "%s ," % random.choice(self.calc_select_fill_j) + sql += "%s " % random.choice(self.calc_select_fill_j) + sql += " from stable_1 t1, table_1 t2 where t1.ts = t2.ts and " + #sql += "%s and " % random.choice(self.t_join_where) + sql += "%s " % self.interp_where_j[random.randint(0,5)] + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + #interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + tdSql.query("select 20-4.2 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill_j) + sql += "%s ," % random.choice(self.calc_select_fill_j) + sql += "%s " % random.choice(self.calc_select_fill_j) + sql += " from stable_1 t1, stable_1_1 t2 where t1.ts = t2.ts and " + sql += "%s and " % random.choice(self.qt_u_or_where) + sql += "%s " % self.interp_where_j[random.randint(0,5)] + sql += "%s " % random.choice(self.fill_where) + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + ##interp不支持 tdSql.error(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + tdSql.query("select 20-5 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill) + sql += "%s ," % random.choice(self.calc_select_fill) + sql += "%s " % random.choice(self.calc_select_fill) + sql += " from regular_table_1 where " + sql += "%s " % self.interp_where[1] + sql += "%s " % random.choice(self.fill_where) + sql += "%s " % random.choice(self.order_where) + sql += "%s " % random.choice(self.limit_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + ##interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + tdSql.query("select 20-6 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s , " % random.choice(self.calc_select_fill_j) + sql += "%s ," % random.choice(self.calc_select_fill_j) + sql += "%s " % random.choice(self.calc_select_fill_j) + sql += " from regular_table_1 t1, regular_table_2 t2 where t1.ts = t2.ts and " + #sql += "%s " % random.choice(self.interp_where_j) + sql += "%s " % self.interp_where_j[random.randint(0,5)] + sql += "%s " % random.choice(self.order_u_where) + sql += "%s " % random.choice(self.limit_u_where) + sql += ") " + tdLog.info(sql) + tdLog.info(len(sql)) + ##interp不支持 tdSql.query(sql) + #self.cur1.execute(sql) + #self.explain_sql(sql) + + #1 select * from (select * from (select * form regular_table where <\>\in\and\or order by limit )) + tdSql.query("select 1-1 from stable_1;") + for i in range(self.fornum): + # sql_start = "select * from ( " + # sql_end = ")" + for_num = random.randint(1, 15); + sql = "select * from (" * for_num + sql += "select * from ( select * from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ttt from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ")) " + sql += ")" * for_num + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + sql2 = "select * from ( select * from ( select " + sql2 += "%s, " % random.choice(self.s_r_select) + sql2 += "%s, " % random.choice(self.q_select) + sql2 += "ts as tin from regular_table_1 where " + sql2 += "%s " % random.choice(self.q_where) + sql2 += ")) " + tdLog.info(sql2) + tdLog.info(len(sql2)) + tdSql.query(sql2) + self.cur1.execute(sql2) + self.explain_sql(sql2) + + self.data_matrix_equal('%s' %sql ,1,10,1,1,'%s' %sql2 ,1,10,1,1) + self.data_matrix_equal('%s' %sql ,1,10,1,1,'%s' %sql ,1,10,3,3) + self.data_matrix_equal('%s' %sql ,1,10,3,3,'%s' %sql2 ,1,10,3,3) + + tdLog.info("=====1-1==over=========") + + for i in range(self.fornum): + for_num = random.randint(1, 15); + sql = "select ts2 from (" * for_num + sql += "select * from ( select * from ( select " + sql += "%s, " % random.choice(self.s_r_select) + sql += "%s, " % random.choice(self.q_select) + sql += "ts as ts2 from regular_table_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ")) " + sql += ")" * for_num + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + sql2 = "select * from ( select * from ( select " + sql2 += "%s, " % random.choice(self.s_r_select) + sql2 += "%s, " % random.choice(self.q_select) + sql2 += "ts as tt from regular_table_1 where " + sql2 += "%s " % random.choice(self.q_where) + sql2 += ")) " + tdLog.info(sql2) + tdLog.info(len(sql2)) + tdSql.query(sql2) + self.cur1.execute(sql2) + self.explain_sql(sql2) + + self.data_matrix_equal('%s' %sql ,1,10,1,1,'%s' %sql2 ,1,10,1,1) + tdLog.info("=====1-2==over=========") + + #2 select * from (select * from (select * form stable where <\>\in\and\or order by limit )) + tdSql.query("select 2-1 from stable_1;") + for i in range(self.fornum): + for_num = random.randint(1, 15); + sql = "select * from (" * for_num + sql += "select * from ( select * from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.qt_select) + sql += "ts as tss from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ")) " + sql += ")" * for_num + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + sql2 = "select * from ( select * from ( select " + sql2 += "%s, " % random.choice(self.s_s_select) + sql2 += "%s, " % random.choice(self.qt_select) + sql2 += "ts as tst from stable_1 where " + sql2 += "%s " % random.choice(self.q_where) + sql2 += ")) " + tdLog.info(sql2) + tdLog.info(len(sql2)) + tdSql.query(sql2) + self.cur1.execute(sql2) + self.explain_sql(sql2) + + self.data_matrix_equal('%s' %sql ,1,10,3,3,'%s' %sql2 ,1,10,3,3) + + tdLog.info("=====2-1==over=========") + + for i in range(self.fornum): + for_num = random.randint(1, 15); + sql = "select tsn from (" * for_num + sql += "select * from ( select * from ( select " + sql += "%s, " % random.choice(self.s_s_select) + sql += "%s, " % random.choice(self.qt_select) + sql += "ts as tsn from stable_1 where " + sql += "%s " % random.choice(self.q_where) + sql += ")) " + sql += ")" * for_num + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + sql2 = "select ts1 from ( select * from ( select " + sql2 += "%s, " % random.choice(self.s_s_select) + sql2 += "%s, " % random.choice(self.qt_select) + sql2 += "ts as ts1 from stable_1 where " + sql2 += "%s " % random.choice(self.q_where) + sql2 += ")) " + tdLog.info(sql2) + tdLog.info(len(sql2)) + tdSql.query(sql2) + self.cur1.execute(sql2) + self.explain_sql(sql2) + + self.data_matrix_equal('%s' %sql ,1,10,1,1,'%s' %sql2 ,1,10,1,1) + tdLog.info("=====2-2==over=========") + + #3 select ts ,calc from (select * form stable where <\>\in\and\or order by limit ) + #self.dropandcreateDB_random("%s" %db, 1) + tdSql.query("select 3-1 from stable_1;") + for i in range(self.fornum): + sql = "select " + sql += "%s " % random.choice(self.calc_calculate_regular) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.orders_desc_where) + sql += "%s " % random.choice(self.limit_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #4 select * from (select calc form stable where <\>\in\and\or order by limit ) + tdSql.query("select 4-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select " + sql += "%s " % random.choice(self.calc_select_in_ts) + sql += "from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + #sql += "%s " % random.choice(self.order_desc_where) + sql += "%s " % random.choice(self.limit_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #5 select ts ,tbname from (select * form stable where <\>\in\and\or order by limit ) + tdSql.query("select 5-1 from stable_1;") + for i in range(self.fornum): + sql = "select ts , tbname , " + sql += "%s ," % random.choice(self.calc_calculate_regular) + sql += "%s ," % random.choice(self.dqt_select) + sql += "%s " % random.choice(self.qt_select) + sql += " from ( select * from stable_1 where " + sql += "%s " % random.choice(self.qt_where) + sql += "%s " % random.choice(self.orders_desc_where) + sql += "%s " % random.choice(self.limit_where) + sql += ") ;" + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.error(sql) + + #6 + tdSql.query("select 6-1 from stable_1;") + for i in range(self.fornum): + sql = "select count(*) from (select avg(q_int)/1000 from stable_1); " + tdLog.info(sql) + tdLog.info(len(sql)) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #special sql + tdSql.query("select 7-1 from stable_1;") + for i in range(self.fornum): + sql = "select * from ( select _block_dist() from stable_1);" + tdSql.error(sql) + sql = "select _block_dist() from (select * from stable_1);" + tdSql.error(sql) + sql = "select * from (select database());" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + sql = "select * from (select client_version());" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + sql = "select * from (select client_version() as version);" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + sql = "select * from (select server_version());" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + sql = "select * from (select server_version() as version);" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + sql = "select * from (select server_status());" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + sql = "select * from (select server_status() as status);" + tdLog.info(sql) + tdSql.query(sql) + self.cur1.execute(sql) + self.explain_sql(sql) + + #taos -f sql + startTime_taos_f = time.time() + print("taos -f sql start!") + taos_cmd1 = "taos -f %s/%s.sql" % (self.testcasePath,self.testcaseFilename) + #_ = subprocess.check_output(taos_cmd1, shell=True).decode("utf-8") + _ = subprocess.check_output(taos_cmd1, shell=True) + print("taos -f sql over!") + endTime_taos_f = time.time() + print("taos_f total time %ds" % (endTime_taos_f - startTime_taos_f)) + + print('=====================2.6 old function end ===========') + + + + def run(self): + tdSql.prepare() + + startTime = time.time() + + #self.function_before_26() + + self.math_nest(['UNIQUE']) + self.math_nest(['MODE']) + self.math_nest(['SAMPLE']) + + # self.math_nest(['ABS','SQRT']) + # self.math_nest(['SIN','COS','TAN','ASIN','ACOS','ATAN']) + # self.math_nest(['POW','LOG']) + # self.math_nest(['FLOOR','CEIL','ROUND']) + # self.math_nest(['MAVG']) + # self.math_nest(['HYPERLOGLOG']) + # self.math_nest(['TAIL']) + self.math_nest(['CSUM']) + self.math_nest(['statecount','stateduration']) + self.math_nest(['HISTOGRAM']) + + # self.str_nest(['LTRIM','RTRIM','LOWER','UPPER']) + # self.str_nest(['LENGTH','CHAR_LENGTH']) + # self.str_nest(['SUBSTR']) + # self.str_nest(['CONCAT']) + # self.str_nest(['CONCAT_WS']) + # self.time_nest(['CAST']) #放到time里起来弄 + # self.time_nest(['CAST_1']) + # self.time_nest(['CAST_2']) + # self.time_nest(['CAST_3']) + # self.time_nest(['CAST_4']) + + + + # self.time_nest(['NOW','TODAY']) + # self.time_nest(['TIMEZONE']) + # self.time_nest(['TIMETRUNCATE']) + # self.time_nest(['TO_ISO8601']) + # self.time_nest(['TO_UNIXTIMESTAMP']) + # self.time_nest(['ELAPSED']) + self.time_nest(['TIMEDIFF_1']) + self.time_nest(['TIMEDIFF_2']) + + + endTime = time.time() + print("total time %ds" % (endTime - startTime)) + + + + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) diff --git a/tests/test_new/query/union/union.py b/tests/test_new/query/union/union.py new file mode 100644 index 0000000000..380b7879c4 --- /dev/null +++ b/tests/test_new/query/union/union.py @@ -0,0 +1,474 @@ +import datetime + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * + +PRIMARY_COL = "ts" + +INT_COL = "c1" +BINT_COL = "c2" +SINT_COL = "c3" +TINT_COL = "c4" +FLOAT_COL = "c5" +DOUBLE_COL = "c6" +BOOL_COL = "c7" + +BINARY_COL = "c8" +NCHAR_COL = "c9" +TS_COL = "c10" + +NUM_COL = [ INT_COL, BINT_COL, SINT_COL, TINT_COL, FLOAT_COL, DOUBLE_COL, ] +CHAR_COL = [ BINARY_COL, NCHAR_COL, ] +BOOLEAN_COL = [ BOOL_COL, ] +TS_TYPE_COL = [ TS_COL, ] + +class TDTestCase: + + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor()) + + def __query_condition(self,tbname): + query_condition = [] + for char_col in CHAR_COL: + query_condition.extend( + ( + f"count( {tbname}.{char_col} )", + f"cast( {tbname}.{char_col} as nchar(3) )", + ) + ) + + for num_col in NUM_COL: + query_condition.extend( + ( + f"log( {tbname}.{num_col}, {tbname}.{num_col})", + ) + ) + + query_condition.extend( + ( + ''' "test12" ''', + # 1010, + ) + ) + + return query_condition + + def __join_condition(self, tb_list, filter=PRIMARY_COL, INNER=False): + table_reference = tb_list[0] + join_condition = f'{table_reference} {table_reference.split(".")[-1]}' + join = "inner join" if INNER else "join" + for i in range(len(tb_list[1:])): + join_condition += f" {join} {tb_list[i+1]} {tb_list[i+1].split('.')[-1]} on {table_reference.split('.')[-1]}.{filter}={tb_list[i+1].split('.')[-1]}.{filter}" + + return join_condition + + def __where_condition(self, col=None, tbname=None, query_conditon=None): + if query_conditon and isinstance(query_conditon, str): + if query_conditon.startswith("count"): + query_conditon = query_conditon[6:-1] + elif query_conditon.startswith("max"): + query_conditon = query_conditon[4:-1] + elif query_conditon.startswith("sum"): + query_conditon = query_conditon[4:-1] + elif query_conditon.startswith("min"): + query_conditon = query_conditon[4:-1] + + if query_conditon: + return f" where {query_conditon} is not null" + if col in NUM_COL: + return f" where abs( {tbname}.{col} ) >= 0" + if col in CHAR_COL: + return f" where lower( {tbname}.{col} ) like 'bina%' or lower( {tbname}.{col} ) like '_cha%' " + if col in BOOLEAN_COL: + return f" where {tbname}.{col} in (false, true) " + if col in TS_TYPE_COL or col in PRIMARY_COL: + return f" where cast( {tbname}.{col} as binary(16) ) is not null " + + return "" + + def __group_condition(self, col, having = None): + if isinstance(col, str): + if col.startswith("count"): + col = col[6:-1] + elif col.startswith("max"): + col = col[4:-1] + elif col.startswith("sum"): + col = col[4:-1] + elif col.startswith("min"): + col = col[4:-1] + return f" group by {col} having {having}" if having else f" group by {col} " + + def __single_sql(self, select_clause, from_clause, where_condition="", group_condition=""): + if isinstance(select_clause, str) and "on" not in from_clause and select_clause.split(".")[0] != from_clause.split(".")[0]: + return + return f"select {select_clause} from {from_clause} {where_condition} {group_condition}" + + @property + def __join_tblist(self, dbname="db"): + return [ + [f"{dbname}.ct1", f"{dbname}.t1"], + [f"{dbname}.ct4", f"{dbname}.t1"], + # ["ct1", "ct2", "ct4"], + # ["ct1", "ct2", "t1"], + # ["ct1", "ct4", "t1"], + # ["ct2", "ct4", "t1"], + # ["ct1", "ct2", "ct4", "t1"], + ] + + @property + def __tb_list(self, dbname="db"): + return [ + f"{dbname}.ct1", + f"{dbname}.ct4", + ] + + def sql_list(self): + sqls = [] + __join_tblist = self.__join_tblist + for join_tblist in __join_tblist: + for join_tb in join_tblist: + join_tb_name = join_tb.split(".")[-1] + select_claus_list = self.__query_condition(join_tb_name) + for select_claus in select_claus_list: + group_claus = self.__group_condition( col=select_claus) + where_claus = self.__where_condition(query_conditon=select_claus) + having_claus = self.__group_condition( col=select_claus, having=f"{select_claus} is not null") + sqls.extend( + ( + self.__single_sql(select_claus, self.__join_condition(join_tblist, INNER=True), where_claus, having_claus), + ) + ) + __no_join_tblist = self.__tb_list + for tb in __no_join_tblist: + tb_name = join_tb.split(".")[-1] + select_claus_list = self.__query_condition(tb_name) + for select_claus in select_claus_list: + group_claus = self.__group_condition(col=select_claus) + where_claus = self.__where_condition(query_conditon=select_claus) + having_claus = self.__group_condition(col=select_claus, having=f"{select_claus} is not null") + sqls.extend( + ( + self.__single_sql(select_claus, tb, where_claus, having_claus), + ) + ) + + # return filter(None, sqls) + return list(filter(None, sqls)) + + def __get_type(self, col): + if tdSql.cursor.istype(col, "BOOL"): + return "BOOL" + if tdSql.cursor.istype(col, "INT"): + return "INT" + if tdSql.cursor.istype(col, "BIGINT"): + return "BIGINT" + if tdSql.cursor.istype(col, "TINYINT"): + return "TINYINT" + if tdSql.cursor.istype(col, "SMALLINT"): + return "SMALLINT" + if tdSql.cursor.istype(col, "FLOAT"): + return "FLOAT" + if tdSql.cursor.istype(col, "DOUBLE"): + return "DOUBLE" + if tdSql.cursor.istype(col, "BINARY"): + return "BINARY" + if tdSql.cursor.istype(col, "NCHAR"): + return "NCHAR" + if tdSql.cursor.istype(col, "TIMESTAMP"): + return "TIMESTAMP" + if tdSql.cursor.istype(col, "JSON"): + return "JSON" + if tdSql.cursor.istype(col, "TINYINT UNSIGNED"): + return "TINYINT UNSIGNED" + if tdSql.cursor.istype(col, "SMALLINT UNSIGNED"): + return "SMALLINT UNSIGNED" + if tdSql.cursor.istype(col, "INT UNSIGNED"): + return "INT UNSIGNED" + if tdSql.cursor.istype(col, "BIGINT UNSIGNED"): + return "BIGINT UNSIGNED" + + def union_check(self, dbname = "db"): + sqls = self.sql_list() + for i in range(len(sqls)): + tdSql.query(sqls[i]) + res1_type = self.__get_type(0) + # if i % 5 == 0: + # tdLog.success(f"{i} : sql is already executing!") + for j in range(len(sqls[i:])): + tdSql.query(sqls[j+i]) + order_union_type = False + rev_order_type = False + all_union_type = False + res2_type = self.__get_type(0) + + if res2_type == res1_type: + all_union_type = True + elif res1_type in ( "BIGINT" , "NCHAR" ) and res2_type in ("BIGINT" , "NCHAR"): + all_union_type = True + elif res1_type in ("BIGINT", "NCHAR"): + order_union_type = True + elif res2_type in ("BIGINT", "NCHAR"): + rev_order_type = True + elif res1_type == "TIMESAMP" and res2_type not in ("BINARY", "NCHAR"): + order_union_type = True + elif res2_type == "TIMESAMP" and res1_type not in ("BINARY", "NCHAR"): + rev_order_type = True + elif res1_type == "BINARY" and res2_type != "NCHAR": + order_union_type = True + elif res2_type == "BINARY" and res1_type != "NCHAR": + rev_order_type = True + + if all_union_type: + tdSql.execute(f"{sqls[i]} union {sqls[j+i]}") + tdSql.execute(f"{sqls[j+i]} union all {sqls[i]}") + elif order_union_type: + tdSql.execute(f"{sqls[i]} union all {sqls[j+i]}") + elif rev_order_type: + tdSql.execute(f"{sqls[j+i]} union {sqls[i]}") + else: + tdSql.error(f"{sqls[i]} union {sqls[j+i]}") + + # check union with timeline function + tdSql.query(f"select first(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1 order by ts)") + tdSql.checkRows(1) + tdSql.checkData(0, 0, 9) + tdSql.query(f"select last(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1 order by ts desc)") + tdSql.checkRows(1) + tdSql.checkData(0, 0, 2147450880) + tdSql.query(f"select irate(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1 order by ts)") + tdSql.checkRows(1) + tdSql.checkData(0, 0, 9.102222222222222) + tdSql.query(f"select elapsed(ts) from (select * from {dbname}.t1 union select * from {dbname}.t1 order by ts)") + tdSql.checkRows(1) + tdSql.checkData(0, 0, 46800000.000000000000000) + tdSql.query(f"select diff(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1 order by ts)") + tdSql.checkRows(14) + tdSql.query(f"select derivative(c1, 1s, 0) from (select * from {dbname}.t1 union select * from {dbname}.t1 order by ts)") + tdSql.checkRows(11) + tdSql.query(f"select count(*) from {dbname}.t1 as a join {dbname}.t1 as b on a.ts = b.ts and a.ts is null") + tdSql.checkRows(1) + tdSql.checkData(0, 0, 0) + + tdSql.query(f"select first(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1)") + tdSql.query(f"select last(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1)") + tdSql.error(f"select irate(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1)") + tdSql.error(f"select elapsed(ts) from (select * from {dbname}.t1 union select * from {dbname}.t1)") + tdSql.error(f"select diff(c1) from (select * from {dbname}.t1 union select * from {dbname}.t1)") + tdSql.error(f"select derivative(c1, 1s, 0) from (select * from {dbname}.t1 union select * from {dbname}.t1)") + + + def __test_error(self, dbname="db"): + + tdSql.error( f"show {dbname}.tables union show {dbname}.tables" ) + tdSql.error( f"create table {dbname}.errtb1 union all create table {dbname}.errtb2" ) + tdSql.error( f"drop table {dbname}.ct1 union all drop table {dbname}.ct3" ) + tdSql.error( f"select c1 from {dbname}.ct1 union all drop table {dbname}.ct3" ) + tdSql.error( f"select c1 from {dbname}.ct1 union all '' " ) + tdSql.error( f" '' union all select c1 from{dbname}. ct1 " ) + + def all_test(self): + self.__test_error() + self.union_check() + + def __create_tb(self, dbname="db"): + + tdLog.printNoPrefix("==========step1:create table") + create_stb_sql = f'''create table {dbname}.stb1( + ts timestamp, {INT_COL} int, {BINT_COL} bigint, {SINT_COL} smallint, {TINT_COL} tinyint, + {FLOAT_COL} float, {DOUBLE_COL} double, {BOOL_COL} bool, + {BINARY_COL} binary(16), {NCHAR_COL} nchar(32), {TS_COL} timestamp + ) tags (tag1 int) + ''' + create_ntb_sql = f'''create table {dbname}.t1( + ts timestamp, {INT_COL} int, {BINT_COL} bigint, {SINT_COL} smallint, {TINT_COL} tinyint, + {FLOAT_COL} float, {DOUBLE_COL} double, {BOOL_COL} bool, + {BINARY_COL} binary(16), {NCHAR_COL} nchar(32), {TS_COL} timestamp + ) + ''' + tdSql.execute(create_stb_sql) + tdSql.execute(create_ntb_sql) + + for i in range(4): + tdSql.execute(f'create table {dbname}.ct{i+1} using {dbname}.stb1 tags ( {i+1} )') + + def __insert_data(self, rows, dbname="db"): + now_time = int(datetime.datetime.timestamp(datetime.datetime.now()) * 1000) + for i in range(rows): + tdSql.execute( + f"insert into {dbname}.ct1 values ( { now_time - i * 1000 }, {i}, {11111 * i}, {111 * i % 32767 }, {11 * i % 127}, {1.11*i}, {1100.0011*i}, {i%2}, 'binary{i}', 'nchar_测试_{i}', { now_time + 1 * i } )" + ) + tdSql.execute( + f"insert into {dbname}.ct4 values ( { now_time - i * 7776000000 }, {i}, {11111 * i}, {111 * i % 32767 }, {11 * i % 127}, {1.11*i}, {1100.0011*i}, {i%2}, 'binary{i}', 'nchar_测试_{i}', { now_time + 1 * i } )" + ) + tdSql.execute( + f"insert into {dbname}.ct2 values ( { now_time - i * 7776000000 }, {-i}, {-11111 * i}, {-111 * i % 32767 }, {-11 * i % 127}, {-1.11*i}, {-1100.0011*i}, {i%2}, 'binary{i}', 'nchar_测试_{i}', { now_time + 1 * i } )" + ) + tdSql.execute( + f'''insert into {dbname}.ct1 values + ( { now_time - rows * 5 }, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar_测试_0', { now_time + 8 } ) + ( { now_time + 10000 }, { rows }, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar_测试_9', { now_time + 9 } ) + ''' + ) + + tdSql.execute( + f'''insert into {dbname}.ct4 values + ( { now_time - rows * 7776000000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time - rows * 3888000000 + 10800000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time + 7776000000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( + { now_time + 5184000000}, {pow(2,31)-pow(2,15)}, {pow(2,63)-pow(2,30)}, 32767, 127, + { 3.3 * pow(10,38) }, { 1.3 * pow(10,308) }, { rows % 2 }, "binary_limit-1", "nchar_测试_limit-1", { now_time - 86400000} + ) + ( + { now_time + 2592000000 }, {pow(2,31)-pow(2,16)}, {pow(2,63)-pow(2,31)}, 32766, 126, + { 3.2 * pow(10,38) }, { 1.2 * pow(10,308) }, { (rows-1) % 2 }, "binary_limit-2", "nchar_测试_limit-2", { now_time - 172800000} + ) + ''' + ) + + tdSql.execute( + f'''insert into {dbname}.ct2 values + ( { now_time - rows * 7776000000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time - rows * 3888000000 + 10800000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time + 7776000000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( + { now_time + 5184000000 }, { -1 * pow(2,31) + pow(2,15) }, { -1 * pow(2,63) + pow(2,30) }, -32766, -126, + { -1 * 3.2 * pow(10,38) }, { -1.2 * pow(10,308) }, { rows % 2 }, "binary_limit-1", "nchar_测试_limit-1", { now_time - 86400000 } + ) + ( + { now_time + 2592000000 }, { -1 * pow(2,31) + pow(2,16) }, { -1 * pow(2,63) + pow(2,31) }, -32767, -127, + { - 3.3 * pow(10,38) }, { -1.3 * pow(10,308) }, { (rows-1) % 2 }, "binary_limit-2", "nchar_测试_limit-2", { now_time - 172800000 } + ) + ''' + ) + + for i in range(rows): + insert_data = f'''insert into {dbname}.t1 values + ( { now_time - i * 3600000 }, {i}, {i * 11111}, { i % 32767 }, { i % 127}, { i * 1.11111 }, { i * 1000.1111 }, { i % 2}, + "binary_{i}", "nchar_测试_{i}", { now_time - 1000 * i } ) + ''' + tdSql.execute(insert_data) + tdSql.execute( + f'''insert into {dbname}.t1 values + ( { now_time + 10800000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time - (( rows // 2 ) * 60 + 30) * 60000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time - rows * 3600000 }, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( { now_time + 7200000 }, { pow(2,31) - pow(2,15) }, { pow(2,63) - pow(2,30) }, 32767, 127, + { 3.3 * pow(10,38) }, { 1.3 * pow(10,308) }, { rows % 2 }, + "binary_limit-1", "nchar_测试_limit-1", { now_time - 86400000 } + ) + ( + { now_time + 3600000 } , { pow(2,31) - pow(2,16) }, { pow(2,63) - pow(2,31) }, 32766, 126, + { 3.2 * pow(10,38) }, { 1.2 * pow(10,308) }, { (rows-1) % 2 }, + "binary_limit-2", "nchar_测试_limit-2", { now_time - 172800000 } + ) + ''' + ) + + def test_TS_5630(self): + sql = "CREATE DATABASE `ep_iot` BUFFER 256 CACHESIZE 20 CACHEMODEL 'both' COMP 2 DURATION 14400m WAL_FSYNC_PERIOD 3000 MAXROWS 4096 MINROWS 100 STT_TRIGGER 2 KEEP 5256000m,5256000m,5256000m PAGES 256 PAGESIZE 4 PRECISION 'ms' REPLICA 1 WAL_LEVEL 1 VGROUPS 3 SINGLE_STABLE 0 TABLE_PREFIX 0 TABLE_SUFFIX 0 TSDB_PAGESIZE 4 WAL_RETENTION_PERIOD 3600 WAL_RETENTION_SIZE 0" + tdSql.execute(sql, queryTimes=1) + tdLog.info("database ep_iot created") + sql = "CREATE STABLE `ep_iot`.`sldc_dp` (`ts` TIMESTAMP, `data_write_time` TIMESTAMP, `jz1fdgl` DOUBLE, `jz1ssfdfh` DOUBLE, `jz1fdmh` DOUBLE, `jz1gdmh` DOUBLE, `jz1qjrhl` DOUBLE, `jz1zhcydl` DOUBLE, `jz1zkby` DOUBLE, `jz1zzqyl` DOUBLE, `jz1zzqwda` DOUBLE, `jz1zzqwdb` DOUBLE, `jz1zzqll` DOUBLE, `jz1gswd` DOUBLE, `jz1gsll` DOUBLE, `jz1glxl` DOUBLE, `jz1qjrh` DOUBLE, `jz1zhrxl` DOUBLE, `jz1gmjassllfk` DOUBLE, `jz1gmjasslllj` DOUBLE, `jz1gmjbssllfk` DOUBLE, `jz1gmjbsslllj` DOUBLE, `jz1gmjcssllfk` DOUBLE, `jz1gmjcsslllj` DOUBLE, `jz1gmjdssllfk` DOUBLE, `jz1gmjdsslllj` DOUBLE, `jz1gmjessllfk` DOUBLE, `jz1gmjesslllj` DOUBLE, `jz1gmjfssllfk` DOUBLE, `jz1gmjfsslllj` DOUBLE, `jz1zrqwda` DOUBLE, `jz1zrqwdb` DOUBLE, `jz1zrzqyl` DOUBLE, `jz1mmjadl` DOUBLE, `jz1mmjbdl` DOUBLE, `jz1mmjcdl` DOUBLE, `jz1mmjddl` DOUBLE, `jz1mmjedl` DOUBLE, `jz1mmjfdl` DOUBLE, `jz1cyqckwda` DOUBLE, `jz1cyqckwdb` DOUBLE, `jz1njswd` DOUBLE, `jz1nqqxhsckawd` DOUBLE, `jz1nqqxhsckbwd` DOUBLE, `jz1nqqxhsrkawd` DOUBLE, `jz1nqqxhsrkbwd` DOUBLE, `jz1kyqackyqwdsel` DOUBLE, `jz1kyqbckyqwdsel` DOUBLE, `jz1yfjackyqwd` DOUBLE, `jz1yfjbckyqwd` DOUBLE, `jz1trkyqwd` DOUBLE, `jz1trkyqwd1` DOUBLE, `jz1trkyqwd2` DOUBLE, `jz1trkyqwd3` DOUBLE, `jz1tckjyqwd1` DOUBLE, `jz1tckjyqwd2` DOUBLE, `jz1tckyqwd1` DOUBLE, `jz1bya` DOUBLE, `jz1byb` DOUBLE, `jz1pqwda` DOUBLE, `jz1pqwdb` DOUBLE, `jz1gmjadl` DOUBLE, `jz1gmjbdl` DOUBLE, `jz1gmjcdl` DOUBLE, `jz1gmjddl` DOUBLE, `jz1gmjedl` DOUBLE, `jz1gmjfdl` DOUBLE, `jz1yfjadl` DOUBLE, `jz1yfjbdl` DOUBLE, `jz1ycfjadl` DOUBLE, `jz1ycfjbdl` DOUBLE, `jz1sfjadl` DOUBLE, `jz1sfjbdl` DOUBLE, `jz1fdjyggl` DOUBLE, `jz1fdjwggl` DOUBLE, `jz1sjzs` DOUBLE, `jz1zfl` DOUBLE, `jz1ltyl` DOUBLE, `jz1smb` DOUBLE, `jz1rll` DOUBLE, `jz1grd` DOUBLE, `jz1zjwd` DOUBLE, `jz1yl` DOUBLE, `jz1kyqckwd` DOUBLE, `jz1abmfsybrkcy` DOUBLE, `jz1bbmfsybrkcy` DOUBLE, `jz1abjcsdmfytwdzdz` DOUBLE, `jz1bbjcsdmfytwdzdz` DOUBLE, `jz2fdgl` DOUBLE, `jz2ssfdfh` DOUBLE, `jz2fdmh` DOUBLE, `jz2gdmh` DOUBLE, `jz2qjrhl` DOUBLE, `jz2zhcydl` DOUBLE, `jz2zkby` DOUBLE, `jz2zzqyl` DOUBLE, `jz2zzqwda` DOUBLE, `jz2zzqwdb` DOUBLE, `jz2zzqll` DOUBLE, `jz2gswd` DOUBLE, `jz2gsll` DOUBLE, `jz2glxl` DOUBLE, `jz2qjrh` DOUBLE, `jz2zhrxl` DOUBLE, `jz2gmjassllfk` DOUBLE, `jz2gmjasslllj` DOUBLE, `jz2gmjbssllfk` DOUBLE, `jz2gmjbsslllj` DOUBLE, `jz2gmjcssllfk` DOUBLE, `jz2gmjcsslllj` DOUBLE, `jz2gmjdssllfk` DOUBLE, `jz2gmjdsslllj` DOUBLE, `jz2gmjessllfk` DOUBLE, `jz2gmjesslllj` DOUBLE, `jz2gmjfssllfk` DOUBLE, `jz2gmjfsslllj` DOUBLE, `jz2zrqwda` DOUBLE, `jz2zrqwdb` DOUBLE, `jz2zrzqyl` DOUBLE, `jz2mmjadl` DOUBLE, `jz2mmjbdl` DOUBLE, `jz2mmjcdl` DOUBLE, `jz2mmjddl` DOUBLE, `jz2mmjedl` DOUBLE, `jz2mmjfdl` DOUBLE, `jz2cyqckwda` DOUBLE, `jz2cyqckwdb` DOUBLE, `jz2njswd` DOUBLE, `jz2nqqxhsckawd` DOUBLE, `jz2nqqxhsckbwd` DOUBLE, `jz2nqqxhsrkawd` DOUBLE, `jz2nqqxhsrkbwd` DOUBLE, `jz2kyqackyqwdsel` DOUBLE, `jz2kyqbckyqwdsel` DOUBLE, `jz2yfjackyqwd` DOUBLE, `jz2yfjbckyqwd` DOUBLE, `jz2trkyqwd` DOUBLE, `jz2trkyqwd1` DOUBLE, `jz2trkyqwd2` DOUBLE, `jz2trkyqwd3` DOUBLE, `jz2tckjyqwd1` DOUBLE, `jz2tckjyqwd2` DOUBLE, `jz2tckyqwd1` DOUBLE, `jz2bya` DOUBLE, `jz2byb` DOUBLE, `jz2pqwda` DOUBLE, `jz2pqwdb` DOUBLE, `jz2gmjadl` DOUBLE, `jz2gmjbdl` DOUBLE, `jz2gmjcdl` DOUBLE, `jz2gmjddl` DOUBLE, `jz2gmjedl` DOUBLE, `jz2gmjfdl` DOUBLE, `jz2yfjadl` DOUBLE, `jz2yfjbdl` DOUBLE, `jz2ycfjadl` DOUBLE, `jz2ycfjbdl` DOUBLE, `jz2sfjadl` DOUBLE, `jz2sfjbdl` DOUBLE, `jz2fdjyggl` DOUBLE, `jz2fdjwggl` DOUBLE, `jz2sjzs` DOUBLE, `jz2zfl` DOUBLE, `jz2ltyl` DOUBLE, `jz2smb` DOUBLE, `jz2rll` DOUBLE, `jz2grd` DOUBLE, `jz2zjwd` DOUBLE, `jz2yl` DOUBLE, `jz2kyqckwd` DOUBLE, `jz2abmfsybrkcy` DOUBLE, `jz2bbmfsybrkcy` DOUBLE, `jz2abjcsdmfytwdzdz` DOUBLE, `jz2bbjcsdmfytwdzdz` DOUBLE) TAGS (`iot_hub_id` VARCHAR(100), `device_group_code` VARCHAR(100), `device_code` VARCHAR(100))" + tdLog.info("stable ep_iot.sldc_dp created") + tdSql.execute(sql, queryTimes=1) + sql = "insert into ep_iot.sldc_dp_t1 using ep_iot.sldc_dp tags('a','a','a') values(now, now, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9,0,1);" + tdSql.execute(sql, queryTimes=1) + sql = "insert into ep_iot.sldc_dp_t1 using ep_iot.sldc_dp tags('b','b','b') values(now, now, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9,0,1);" + tdSql.execute(sql, queryTimes=1) + sql = "insert into ep_iot.sldc_dp_t1 using ep_iot.sldc_dp tags('c','c','c') values(now, now, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9,0,1);" + tdSql.execute(sql, queryTimes=1) + sql = "insert into ep_iot.sldc_dp_t1 using ep_iot.sldc_dp tags('d','d','d') values(now, now, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9,0,1);" + tdSql.execute(sql, queryTimes=1) + sql = "insert into ep_iot.sldc_dp_t1 using ep_iot.sldc_dp tags('e','e','e') values(now, now, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9, 0,1,2,3,4,5,6,7,8,9,0,1);" + tdSql.execute(sql, queryTimes=1) + sql = "select scdw_code, scdw_name, jzmc, fdgl, jzzt from ((select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组1' as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组2' as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp)) where scdw_code like '%%';" + tdSql.query(sql, queryTimes=1) + tdSql.checkCols(5) + tdSql.checkRows(6) + + sql = "select scdw_name, scdw_code, jzmc, fdgl, jzzt from ((select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组1' as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组2' as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp)) where scdw_code like '%%';" + tdSql.query(sql, queryTimes=1) + tdSql.checkCols(5) + tdSql.checkRows(6) + sql = "select scdw_name, scdw_code, jzzt from ((select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组1' as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组2' as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp)) where scdw_code like '%%';" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(6) + tdSql.checkCols(3) + + sql = "select scdw_code, scdw_name, jzmc, fdgl, jzzt,ts from ((select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组1' as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01072016' as scdw_code, '盛鲁电厂' as scdw_name, '机组2' as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '00103673' as scdw_code, '鲁西电厂' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt, last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组1'as jzmc, last(jz1fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp) union all ( select '01061584' as scdw_code, '富源热电' as scdw_name, '机组2'as jzmc, last(jz2fdjyggl) as fdgl, '填报' as jzzt ,last(ts) as ts from ep_iot.sldc_dp)) where scdw_code like '%%';" + tdSql.query(sql, queryTimes=1) + tdSql.checkCols(6) + tdSql.checkRows(6) + ##tdSql.execute("drop database ep_iot") + + def test_case_for_nodes_match_node(self): + sql = "create table db.nt (ts timestamp, c1 int primary key, c2 int)" + tdSql.execute(sql, queryTimes=1) + sql = 'select diff (ts) from (select * from db.tt union select * from db.tt order by c1, case when ts < now - 1h then ts + 1h else ts end) partition by c1, case when ts < now - 1h then ts + 1h else ts end' + tdSql.error(sql, -2147473917) + + def run(self): + tdSql.prepare() + self.test_TS_5630() + + tdLog.printNoPrefix("==========step1:create table") + self.__create_tb() + + tdLog.printNoPrefix("==========step2:insert data") + self.rows = 10 + self.__insert_data(self.rows) + + tdLog.printNoPrefix("==========step3:all check") + self.all_test() + + tdSql.execute("flush database db") + + tdSql.execute("use db") + + tdLog.printNoPrefix("==========step4:after wal, all check again ") + self.all_test() + self.test_TD_33137() + self.test_case_for_nodes_match_node() + + def test_TD_33137(self): + sql = "select 'asd' union all select 'asdasd'" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(2) + sql = "select db_name `TABLE_CAT`, '' `TABLE_SCHEM`, stable_name `TABLE_NAME`, 'TABLE' `TABLE_TYPE`, table_comment `REMARKS` from information_schema.ins_stables union all select db_name `TABLE_CAT`, '' `TABLE_SCHEM`, table_name `TABLE_NAME`, case when `type`='SYSTEM_TABLE' then 'TABLE' when `type`='NORMAL_TABLE' then 'TABLE' when `type`='CHILD_TABLE' then 'TABLE' else 'UNKNOWN' end `TABLE_TYPE`, table_comment `REMARKS` from information_schema.ins_tables union all select db_name `TABLE_CAT`, '' `TABLE_SCHEM`, view_name `TABLE_NAME`, 'VIEW' `TABLE_TYPE`, NULL `REMARKS` from information_schema.ins_views" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(49) + + sql = "select null union select null" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(1) + tdSql.checkData(0, 0, None) + + sql = "select null union all select null" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(2) + tdSql.checkData(0, 0, None) + tdSql.checkData(1, 0, None) + + sql = "select null union select 1" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(2) + tdSql.checkData(0, 0, None) + tdSql.checkData(1, 0, 1) + + sql = "select null union select 'asd'" + tdSql.query(sql, queryTimes=1) + tdSql.checkRows(2) + tdSql.checkData(0, 0, None) + tdSql.checkData(1, 0, 'asd') + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/query/view/test_view.py b/tests/test_new/query/view/test_view.py new file mode 100644 index 0000000000..059b016c3d --- /dev/null +++ b/tests/test_new/query/view/test_view.py @@ -0,0 +1,602 @@ + +import taos +import os +import sys +import time +from pathlib import Path +sys.path.append(os.path.dirname(Path(__file__).resolve().parent.parent.parent) + "/7-tmq") + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * +from util.common import * +from util.sqlset import * +from tmqCommon import * + +class TDTestCase: + """This test case is used to veirfy the tmq consume data from non marterial view + """ + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor()) + self.setsql = TDSetSql() + + # db info + self.dbname = "view_db" + self.stbname = 'stb' + self.ctbname_list = ["ct1", "ct2"] + self.stable_column_dict = { + 'ts': 'timestamp', + 'col1': 'float', + 'col2': 'int', + } + self.tag_dict = { + 'ctbname': 'binary(10)' + } + + def prepare_data(self, conn=None): + """Create the db and data for test + """ + tdLog.debug("Start to prepare the data") + if not conn: + conn = tdSql + # create datebase + conn.execute(f"create database {self.dbname}") + conn.execute(f"use {self.dbname}") + time.sleep(2) + + # create stable + conn.execute(self.setsql.set_create_stable_sql(self.stbname, self.stable_column_dict, self.tag_dict)) + tdLog.debug("Create stable {} successfully".format(self.stbname)) + + # create child tables + for ctname in self.ctbname_list: + conn.execute(f"create table {ctname} using {self.stbname} tags('{ctname}');") + tdLog.debug("Create child table {} successfully".format(ctname)) + + # insert data into child tables + conn.execute(f"insert into {ctname} values(now, 1.1, 1)(now+1s, 2.2, 2)(now+2s, 3.3, 3)(now+3s, 4.4, 4)(now+4s, 5.5, 5)(now+5s, 6.6, 6)(now+6s, 7.7, 7)(now+7s, 8.8, 8)(now+8s, 9.9, 9)(now+9s, 10.1, 10);)") + tdLog.debug(f"Insert into data to {ctname} successfully") + + def prepare_tmq_data(self, para_dic): + tdLog.debug("Start to prepare the tmq data") + tmqCom.initConsumerTable() + tdCom.create_database(tdSql, para_dic["dbName"], para_dic["dropFlag"], vgroups=para_dic["vgroups"], replica=1) + tdLog.info("create stb") + tdCom.create_stable(tdSql, dbname=para_dic["dbName"], stbname=para_dic["stbName"], column_elm_list=para_dic['colSchema'], tag_elm_list=para_dic['tagSchema']) + tdLog.info("create ctb") + tdCom.create_ctable(tdSql, dbname=para_dic["dbName"], stbname=para_dic["stbName"],tag_elm_list=para_dic['tagSchema'], count=para_dic["ctbNum"], default_ctbname_prefix=para_dic['ctbPrefix']) + tdLog.info("insert data") + tmqCom.insert_data(tdSql, para_dic["dbName"], para_dic["ctbPrefix"], para_dic["ctbNum"], para_dic["rowsPerTbl"], para_dic["batchNum"], para_dic["startTs"]) + tdLog.debug("Finish to prepare the tmq data") + + def check_view_num(self, num): + tdSql.query("show views;") + rows = tdSql.queryRows + assert(rows == num) + tdLog.debug(f"Verify the view number successfully") + + def create_user(self, username, password): + tdSql.execute(f"create user {username} pass '{password}';") + tdSql.execute(f"alter user {username} createdb 1;") + tdLog.debug("Create user {} with password {} successfully".format(username, password)) + + def check_permissions(self, username, db_name, permission_dict, view_name=None): + """ + :param permission_dict: {'db': ["read", "write], 'view': ["read", "write", "alter"]} + """ + tdSql.query("select * from information_schema.ins_user_privileges;") + for item in permission_dict.keys(): + if item == "db": + for permission in permission_dict[item]: + assert((username, permission, db_name, "", "", "") in tdSql.queryResult) + tdLog.debug(f"Verify the {item} {db_name} {permission} permission successfully") + elif item == "view": + for permission in permission_dict[item]: + assert((username, permission, db_name, view_name, "", "view") in tdSql.queryResult) + tdLog.debug(f"Verify the {item} {db_name} {view_name} {permission} permission successfully") + else: + raise Exception(f"Invalid permission type: {item}") + + def test_create_view_from_one_database(self): + """This test case is used to verify the create view from one database + """ + self.prepare_data() + tdSql.execute(f"create view v1 as select * from {self.stbname};") + self.check_view_num(1) + tdSql.error(f'create view v1 as select * from {self.stbname};', expectErrInfo='view already exists in db') + tdSql.error(f'create view db2.v2 as select * from {self.stbname};', expectErrInfo='Fail to get table info, error: Database not exist') + tdSql.error(f'create view v2 as select c2 from {self.stbname};', expectErrInfo='Invalid column name: c2') + tdSql.error(f'create view v2 as select ts, col1 from tt1;', expectErrInfo='Fail to get table info, error: Table does not exist') + + tdSql.execute(f"drop database {self.dbname}") + tdLog.debug("Finish test case 'test_create_view_from_one_database'") + + def test_create_view_from_multi_database(self): + """This test case is used to verify the create view from multi database + """ + self.prepare_data() + tdSql.execute(f"create view v1 as select * from view_db.{self.stbname};") + self.check_view_num(1) + + self.dbname = "view_db2" + self.prepare_data() + tdSql.execute(f"create view v1 as select * from view_db2.{self.stbname};") + tdSql.execute(f"create view v2 as select * from view_db.v1;") + self.check_view_num(2) + + self.dbname = "view_db" + tdSql.execute(f"drop database view_db;") + tdSql.execute(f"drop database view_db2;") + tdLog.debug("Finish test case 'test_create_view_from_multi_database'") + + def test_create_view_name_params(self): + """This test case is used to verify the create view with different view name params + """ + self.prepare_data() + tdSql.execute(f"create view v1 as select * from {self.stbname};") + self.check_view_num(1) + tdSql.error(f"create view v/2 as select * from {self.stbname};", expectErrInfo='syntax error near "/2 as select * from stb;"') + tdSql.execute(f"create view v2 as select ts, col1 from {self.stbname};") + self.check_view_num(2) + view_name_192_characters = "rzuoxoIXilAGgzNjYActiQwgzZK7PZYpDuaOe1lSJMFMVYXaexh1OfMmk3LvJcQbTeXXW7uGJY8IHuweHF73VHgoZgf0waO33YpZiTKfDQbdWtN4YmR2eWjL84ZtkfjM4huCP6lCysbDMj8YNwWksTdUq70LIyNhHp2V8HhhxyYSkREYFLJ1kOE78v61MQT6" + tdSql.execute(f"create view {view_name_192_characters} as select * from {self.stbname};") + self.check_view_num(3) + tdSql.error(f"create view {view_name_192_characters}1 as select * from {self.stbname};", expectErrInfo='Invalid identifier name: rzuoxoixilaggznjyactiqwgzzk7pzypduaoe1lsjmfmvyxaexh1ofmmk3lvjcqbtexxw7ugjy8ihuwehf73vhgozgf0wao33ypzitkfdqbdwtn4ymr2ewjl84ztkfjm4hucp6lcysbdmj8ynwwkstduq70liynhhp2v8hhhxyyskreyflj1koe78v61mqt61 as select * from stb;') + tdSql.execute(f"drop database {self.dbname}") + tdLog.debug("Finish test case 'test_create_view_name_params'") + + def test_create_view_query(self): + """This test case is used to verify the create view with different data type in query + """ + self.prepare_data() + # add different data type table + tdSql.execute(f"create table tb (ts timestamp, c1 int, c2 int unsigned, c3 bigint, c4 bigint unsigned, c5 float, c6 double, c7 binary(16), c8 smallint, c9 smallint unsigned, c10 tinyint, c11 tinyint unsigned, c12 bool, c13 varchar(16), c14 nchar(8), c15 geometry(21), c16 varbinary(16));") + tdSql.execute(f"create view v1 as select ts, c1, c2, c3, c4, c5, c6, c7, c8, c9, c10, c11, c12, c13, c14, c15, c16 from tb;") + # check data type in create view sql + tdSql.query("desc v1;") + res = tdSql.queryResult + data_type_list = [res[index][1] for index in range(len(res))] + tdLog.debug(data_type_list) + assert('TIMESTAMP' in data_type_list and 'INT' in data_type_list and 'INT UNSIGNED' in data_type_list and 'BIGINT' in data_type_list and 'BIGINT UNSIGNED' in data_type_list and 'FLOAT' in data_type_list and 'DOUBLE' in data_type_list and 'VARCHAR' in data_type_list and 'SMALLINT' in data_type_list and 'SMALLINT UNSIGNED' in data_type_list and 'TINYINT' in data_type_list and 'TINYINT UNSIGNED' in data_type_list and 'BOOL' in data_type_list and 'VARCHAR' in data_type_list and 'NCHAR' in data_type_list and 'GEOMETRY' in data_type_list and 'VARBINARY' in data_type_list) + tdSql.execute("create view v2 as select * from tb where c1 >5 and c7 like '%ab%';") + self.check_view_num(2) + tdSql.error("create view v3 as select * from tb where c1 like '%ab%';", expectErrInfo='Invalid operation') + tdSql.execute("create view v3 as select first(ts), sum(c1) from tb group by c2 having avg(c4) > 0;") + tdSql.execute("create view v4 as select _wstart,sum(c6) from tb interval(10s);") + tdSql.execute("create view v5 as select * from tb join v2 on tb.ts = v2.ts;") + tdSql.execute("create view v6 as select * from (select ts, c1, c2 from (select * from v2));") + self.check_view_num(6) + for v in ['v1', 'v2', 'v3', 'v4', 'v5', 'v6']: + tdSql.execute(f"drop view {v};") + tdSql.execute(f"drop database {self.dbname}") + tdLog.debug("Finish test case 'test_create_view_query'") + + def test_show_view(self): + """This test case is used to verify the show view + """ + self.prepare_data() + tdSql.execute(f"create view v1 as select * from {self.ctbname_list[0]};") + + # query from show sql + tdSql.query("show views;") + res = tdSql.queryResult + assert(res[0][0] == 'v1' and res[0][1] == 'view_db' and res[0][2] == 'root' and res[0][4] == 'NORMAL' and res[0][5] == 'select * from ct1;') + + # show create sql + tdSql.query("show create view v1;") + res = tdSql.queryResult + assert(res[0][1] == 'CREATE VIEW `view_db`.`v1` AS select * from ct1;') + + # query from desc results + tdSql.query("desc view_db.v1;") + res = tdSql.queryResult + assert(res[0][1] == 'TIMESTAMP' and res[1][1] == 'FLOAT' and res[2][1] == 'INT') + + # query from system table + tdSql.query("select * from information_schema.ins_views;") + res = tdSql.queryResult + assert(res[0][0] == 'v1' and res[0][1] == 'view_db' and res[0][2] == 'root' and res[0][4] == 'NORMAL' and res[0][5] == 'select * from ct1;') + tdSql.error("show db3.views;", expectErrInfo='Database not exist') + tdSql.error("desc viewx;", expectErrInfo='Table does not exist') + tdSql.error(f"show create view {self.dbname}.viewx;", expectErrInfo='view not exists in db') + tdSql.execute(f"drop database {self.dbname}") + tdSql.error("show views;", expectErrInfo='Database not exist') + tdLog.debug("Finish test case 'test_show_view'") + + def test_drop_view(self): + """This test case is used to verify the drop view + """ + self.prepare_data() + self.dbname = "view_db2" + self.prepare_data() + tdSql.execute("create view view_db.v1 as select * from view_db.stb;") + tdSql.execute("create view view_db2.v1 as select * from view_db2.stb;") + # delete view without database name + tdSql.execute("drop view v1;") + # delete view with database name + tdSql.execute("drop view view_db.v1;") + # delete non exist view + tdSql.error("drop view view_db.v11;", expectErrInfo='view not exists in db') + tdSql.execute("drop database view_db") + tdSql.execute("drop database view_db2;") + self.dbname = "view_db" + tdLog.debug("Finish test case 'test_drop_view'") + + def test_view_permission_db_all_view_all(self): + """This test case is used to verify the view permission with db all and view all, + the time sleep to wait the permission take effect + """ + self.prepare_data() + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + # grant all db permission to user + tdSql.execute("grant all on view_db.* to view_test;") + + conn = taos.connect(user=username, password=password) + conn.execute(f"use {self.dbname};") + conn.execute("create view v1 as select * from stb;") + res = conn.query("show views;") + assert(len(res.fetch_all()) == 1) + tdLog.debug(f"Verify the show view permission of user '{username}' with db all and view all successfully") + self.check_permissions("view_test", "view_db", {"db": ["read", "write"], "view": ["read", "write", "alter"]}, "v1") + tdLog.debug(f"Verify the view permission from system table successfully") + time.sleep(2) + conn.execute("drop view v1;") + tdSql.execute("revoke all on view_db.* from view_test;") + tdSql.execute(f"drop database {self.dbname};") + time.sleep(1) + + # prepare data by user 'view_test' + self.prepare_data(conn) + + conn.execute("create view v1 as select * from stb;") + res = conn.query("show views;") + assert(len(res.fetch_all()) == 1) + tdLog.debug(f"Verify the view permission of user '{username}' with db all and view all successfully") + self.check_permissions("view_test", "view_db", {"db": ["read", "write"], "view": ["read", "write", "alter"]}, "v1") + tdLog.debug(f"Verify the view permission from system table successfully") + time.sleep(2) + conn.execute("drop view v1;") + tdSql.execute("revoke all on view_db.* from view_test;") + tdSql.execute("revoke all on view_db.v1 from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_all_view_all'") + + def test_view_permission_db_write_view_all(self): + """This test case is used to verify the view permission with db write and view all + """ + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + conn = taos.connect(user=username, password=password) + self.prepare_data(conn) + conn.execute("create view v1 as select * from stb;") + tdSql.execute("revoke read on view_db.* from view_test;") + self.check_permissions("view_test", "view_db", {"db": ["write"], "view": ["read", "write", "alter"]}, "v1") + # create view permission error + try: + conn.execute("create view v2 as select * from v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + # query from view permission error + try: + conn.query("select * from v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + # view query permission + res = conn.query("show views;") + assert(len(res.fetch_all()) == 1) + time.sleep(2) + conn.execute("drop view v1;") + tdSql.execute("revoke write on view_db.* from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_write_view_all'") + + def test_view_permission_db_write_view_read(self): + """This test case is used to verify the view permission with db write and view read + """ + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + conn = taos.connect(user=username, password=password) + self.prepare_data() + + tdSql.execute("create view v1 as select * from stb;") + tdSql.execute("grant write on view_db.* to view_test;") + tdSql.execute("grant read on view_db.v1 to view_test;") + + conn.execute(f"use {self.dbname};") + time.sleep(2) + res = conn.query("select * from v1;") + assert(len(res.fetch_all()) == 20) + + conn.execute("create view v2 as select * from v1;") + # create view from super table of database + try: + conn.execute("create view v3 as select * from stb;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + time.sleep(2) + conn.execute("drop view v2;") + try: + conn.execute("drop view v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + tdSql.execute("revoke read on view_db.v1 from view_test;") + tdSql.execute("revoke write on view_db.* from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_write_view_read'") + + def test_view_permission_db_write_view_alter(self): + """This test case is used to verify the view permission with db write and view alter + """ + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + conn = taos.connect(user=username, password=password) + self.prepare_data() + + tdSql.execute("create view v1 as select * from stb;") + tdSql.execute("grant write on view_db.* to view_test;") + tdSql.execute("grant alter on view_db.v1 to view_test;") + try: + conn.execute(f"use {self.dbname};") + conn.execute("select * from v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + time.sleep(2) + conn.execute("drop view v1;") + tdSql.execute("revoke write on view_db.* from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_write_view_alter'") + + def test_view_permission_db_read_view_all(self): + """This test case is used to verify the view permission with db read and view all + """ + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + conn = taos.connect(user=username, password=password) + self.prepare_data() + + tdSql.execute("create view v1 as select * from stb;") + tdSql.execute("grant read on view_db.* to view_test;") + tdSql.execute("grant all on view_db.v1 to view_test;") + try: + conn.execute(f"use {self.dbname};") + conn.execute("create view v2 as select * from v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + time.sleep(2) + res = conn.query("select * from v1;") + assert(len(res.fetch_all()) == 20) + conn.execute("drop view v1;") + tdSql.execute("revoke read on view_db.* from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_read_view_all'") + + def test_view_permission_db_read_view_alter(self): + """This test case is used to verify the view permission with db read and view alter + """ + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + conn = taos.connect(user=username, password=password) + self.prepare_data() + + tdSql.execute("create view v1 as select * from stb;") + tdSql.execute("grant read on view_db.* to view_test;") + tdSql.execute("grant alter on view_db.v1 to view_test;") + try: + conn.execute(f"use {self.dbname};") + conn.execute("select * from v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + + time.sleep(2) + conn.execute("drop view v1;") + tdSql.execute("revoke read on view_db.* from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_read_view_alter'") + + def test_view_permission_db_read_view_read(self): + """This test case is used to verify the view permission with db read and view read + """ + username = "view_test" + password = "test123@#$" + self.create_user(username, password) + conn = taos.connect(user=username, password=password) + self.prepare_data() + + tdSql.execute("create view v1 as select * from stb;") + tdSql.execute("grant read on view_db.* to view_test;") + tdSql.execute("grant read on view_db.v1 to view_test;") + conn.execute(f"use {self.dbname};") + time.sleep(2) + res = conn.query("select * from v1;") + assert(len(res.fetch_all()) == 20) + try: + conn.execute("drop view v1;") + except Exception as ex: + assert("[0x2644]: Permission denied or target object not exist" in str(ex)) + tdSql.execute("revoke read on view_db.* from view_test;") + tdSql.execute("revoke read on view_db.v1 from view_test;") + tdSql.execute(f"drop database {self.dbname}") + tdSql.execute("drop user view_test;") + tdLog.debug("Finish test case 'test_view_permission_db_read_view_read'") + + def test_query_from_view(self): + """This test case is used to verify the query from view + """ + self.prepare_data() + view_name_list = [] + + # common query from super table + tdSql.execute(f"create view v1 as select * from {self.stbname};") + tdSql.query(f"select * from v1;") + rows = tdSql.queryRows + assert(rows == 20) + view_name_list.append("v1") + tdLog.debug("Verify the query from super table successfully") + + # common query from child table + tdSql.execute(f"create view v2 as select * from {self.ctbname_list[0]};") + tdSql.query(f"select * from v2;") + rows = tdSql.queryRows + assert(rows == 10) + view_name_list.append("v2") + tdLog.debug("Verify the query from child table successfully") + + # join query + tdSql.execute(f"create view v3 as select * from {self.stbname} join {self.ctbname_list[1]} on {self.ctbname_list[1]}.ts = {self.stbname}.ts;") + tdSql.query(f"select * from v3;") + rows = tdSql.queryRows + assert(rows == 10) + view_name_list.append("v3") + tdLog.debug("Verify the join query successfully") + + # group by query + tdSql.execute(f"create view v4 as select count(*) from {self.stbname} group by tbname;") + tdSql.query(f"select * from v4;") + rows = tdSql.queryRows + assert(rows == 2) + res = tdSql.queryResult + assert(res[0][0] == 10) + view_name_list.append("v4") + tdLog.debug("Verify the group by query successfully") + + # partition by query + tdSql.execute(f"create view v5 as select sum(col1) from {self.stbname} where col2 > 4 partition by tbname interval(3s);") + tdSql.query(f"select * from v5;") + rows = tdSql.queryRows + assert(rows >= 4) + view_name_list.append("v5") + tdLog.debug("Verify the partition by query successfully") + + # query from nested view + tdSql.execute(f"create view v6 as select * from v5;") + tdSql.query(f"select * from v6;") + rows = tdSql.queryRows + assert(rows >= 4) + view_name_list.append("v6") + tdLog.debug("Verify the query from nested view successfully") + + # delete view + for view in view_name_list: + tdSql.execute(f"drop view {view};") + tdLog.debug(f"Drop view {view} successfully") + tdSql.execute(f"drop database {self.dbname}") + tdLog.debug("Finish test case 'test_query_from_view'") + + def test_tmq_from_view(self): + """This test case is used to verify the tmq consume data from view + """ + # params for db + paraDict = {'dbName': 'view_db', + 'dropFlag': 1, + 'event': '', + 'vgroups': 4, + 'stbName': 'stb', + 'colPrefix': 'c', + 'tagPrefix': 't', + 'colSchema': [{'type': 'INT', 'count':1}, {'type': 'binary', 'len':20, 'count':1}], + 'tagSchema': [{'type': 'INT', 'count':1}, {'type': 'binary', 'len':20, 'count':1}], + 'ctbPrefix': 'ctb', + 'ctbNum': 1, + 'rowsPerTbl': 10000, + 'batchNum': 10, + 'startTs': 1640966400000, # 2022-01-01 00:00:00.000 + 'pollDelay': 10, + 'showMsg': 1, + 'showRow': 1} + # topic info + topic_name_list = ['topic1'] + view_name_list = ['view1'] + expectRowsList = [] + + self.prepare_tmq_data(paraDict) + + # init consume info, and start tmq_sim, then check consume result + tmqCom.initConsumerTable() + queryString = "select * from %s.%s"%(paraDict['dbName'], paraDict['stbName']) + tdSql.execute(f"create view {view_name_list[0]} as {queryString}") + sqlString = "create topic %s as %s" %(topic_name_list[0], "select * from %s"%view_name_list[0]) + tdLog.info("create topic sql: %s"%sqlString) + tdSql.execute(sqlString) + tdSql.query(queryString) + expectRowsList.append(tdSql.getRows()) + + consumerId = 1 + topicList = topic_name_list[0] + expectrowcnt = paraDict["rowsPerTbl"] * paraDict["ctbNum"] + keyList = 'group.id:cgrp1, enable.auto.commit:false, auto.commit.interval.ms:6000, auto.offset.reset:earliest' + ifcheckdata = 1 + ifManualCommit = 1 + tmqCom.insertConsumerInfo(consumerId, expectrowcnt, topicList, keyList, ifcheckdata, ifManualCommit) + + tdLog.info("start consume processor") + tmqCom.startTmqSimProcess(paraDict['pollDelay'], paraDict["dbName"], paraDict['showMsg'], paraDict['showRow']) + + tdLog.info("wait the consume result") + expectRows = 1 + resultList = tmqCom.selectConsumeResult(expectRows) + if expectRowsList[0] != resultList[0]: + tdLog.info("expect consume rows: %d, act consume rows: %d"%(expectRowsList[0], resultList[0])) + tdLog.exit("1 tmq consume rows error!") + + tmqCom.checkFileContent(consumerId, queryString) + + time.sleep(10) + for i in range(len(topic_name_list)): + tdSql.query("drop topic %s"%topic_name_list[i]) + for i in range(len(view_name_list)): + tdSql.query("drop view %s"%view_name_list[i]) + + # drop database + tdSql.execute(f"drop database {paraDict['dbName']}") + tdSql.execute("drop database cdb;") + tdLog.debug("Finish test case 'test_tmq_from_view'") + def test_TD_33390(self): + tdSql.execute('create database test') + tdSql.execute('create table test.nt(ts timestamp, c1 int)') + for i in range(0, 200): + tdSql.execute(f'create view test.view{i} as select * from test.nt') + tdSql.query("show test.views") + + for i in range(0, 200): + tdSql.execute(f'drop view test.view{i}') + + def run(self): + self.test_TD_33390() + self.test_create_view_from_one_database() + self.test_create_view_from_multi_database() + self.test_create_view_name_params() + self.test_create_view_query() + self.test_show_view() + self.test_drop_view() + self.test_view_permission_db_all_view_all() + self.test_view_permission_db_write_view_all() + self.test_view_permission_db_write_view_read() + self.test_view_permission_db_write_view_alter() + self.test_view_permission_db_read_view_all() + self.test_view_permission_db_read_view_alter() + self.test_view_permission_db_read_view_read() + self.test_query_from_view() + self.test_tmq_from_view() + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/security/permission/user_privilege.py b/tests/test_new/security/permission/user_privilege.py new file mode 100644 index 0000000000..809881589a --- /dev/null +++ b/tests/test_new/security/permission/user_privilege.py @@ -0,0 +1,155 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import taos +from taos.tmq import * +from util.cases import * +from util.common import * +from util.log import * +from util.sql import * +from util.sqlset import * + + +class TDTestCase: + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor()) + self.setsql = TDSetSql() + self.stbname = 'stb' + self.user_name = 'test' + self.binary_length = 20 # the length of binary for column_dict + self.nchar_length = 20 # the length of nchar for column_dict + self.dbnames = ['db1', 'db2'] + self.column_dict = { + 'ts': 'timestamp', + 'col1': 'float', + 'col2': 'int', + 'col3': 'float', + } + + self.tag_dict = { + 't1': 'int', + 't2': f'binary({self.binary_length})' + } + + self.tag_list = [ + f'1, "Beijing"', + f'2, "Shanghai"', + f'3, "Guangzhou"', + f'4, "Shenzhen"' + ] + + self.values_list = [ + f'now, 9.1, 200, 0.3' + ] + + self.tbnum = 4 + self.stbnum_grant = 200 + + def create_user(self): + tdSql.execute(f'create user {self.user_name} pass "test123@#$"') + tdSql.execute(f'grant read on {self.dbnames[0]}.{self.stbname} with t2 = "Beijing" to {self.user_name}') + tdSql.execute(f'grant write on {self.dbnames[1]}.{self.stbname} with t1 = 2 to {self.user_name}') + + def prepare_data(self): + for db in self.dbnames: + tdSql.execute(f"create database {db}") + tdSql.execute(f"use {db}") + tdSql.execute(self.setsql.set_create_stable_sql(self.stbname, self.column_dict, self.tag_dict)) + for i in range(self.tbnum): + tdSql.execute(f'create table {self.stbname}_{i} using {self.stbname} tags({self.tag_list[i]})') + for j in self.values_list: + tdSql.execute(f'insert into {self.stbname}_{i} values({j})') + for i in range(self.stbnum_grant): + tdSql.execute(f'create table {self.stbname}_grant_{i} (ts timestamp, c0 int) tags(t0 int)') + + def user_read_privilege_check(self, dbname): + testconn = taos.connect(user='test', password='test123@#$') + expectErrNotOccured = False + + try: + sql = f"select count(*) from {dbname}.stb where t2 = 'Beijing'" + res = testconn.query(sql) + data = res.fetch_all() + count = data[0][0] + except BaseException: + expectErrNotOccured = True + + if expectErrNotOccured: + caller = inspect.getframeinfo(inspect.stack()[1][0]) + tdLog.exit(f"{caller.filename}({caller.lineno}) failed: sql:{sql}, expect error not occured") + elif count != 1: + tdLog.exit(f"{sql}, expect result doesn't match") + pass + + def user_write_privilege_check(self, dbname): + testconn = taos.connect(user='test', password='test123@#$') + expectErrNotOccured = False + + try: + sql = f"insert into {dbname}.stb_1 values(now, 1.1, 200, 0.3)" + testconn.execute(sql) + except BaseException: + expectErrNotOccured = True + + if expectErrNotOccured: + caller = inspect.getframeinfo(inspect.stack()[1][0]) + tdLog.exit(f"{caller.filename}({caller.lineno}) failed: sql:{sql}, expect error not occured") + else: + pass + + def user_privilege_error_check(self): + testconn = taos.connect(user='test', password='test123@#$') + expectErrNotOccured = False + + sql_list = [f"alter talbe {self.dbnames[0]}.stb_1 set t2 = 'Wuhan'", + f"insert into {self.dbnames[0]}.stb_1 values(now, 1.1, 200, 0.3)", + f"drop table {self.dbnames[0]}.stb_1", + f"select count(*) from {self.dbnames[1]}.stb"] + + for sql in sql_list: + try: + res = testconn.execute(sql) + except BaseException: + expectErrNotOccured = True + + if expectErrNotOccured: + pass + else: + caller = inspect.getframeinfo(inspect.stack()[1][0]) + tdLog.exit(f"{caller.filename}({caller.lineno}) failed: sql:{sql}, expect error not occured") + pass + + def user_privilege_grant_check(self): + for db in self.dbnames: + tdSql.execute(f"use {db}") + for i in range(self.stbnum_grant): + tdSql.execute(f'grant read on {db}.{self.stbname}_grant_{i} to {self.user_name}') + tdSql.execute(f'grant write on {db}.{self.stbname}_grant_{i} to {self.user_name}') + + def run(self): + self.prepare_data() + self.create_user() + self.user_read_privilege_check(self.dbnames[0]) + self.user_write_privilege_check(self.dbnames[1]) + self.user_privilege_error_check() + self.user_privilege_grant_check() + + def stop(self): + tdSql.close() + tdLog.success("%s successfully executed" % __file__) + + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/test_new/storage/compress/compressBasic.py b/tests/test_new/storage/compress/compressBasic.py new file mode 100644 index 0000000000..446cb920fb --- /dev/null +++ b/tests/test_new/storage/compress/compressBasic.py @@ -0,0 +1,388 @@ +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### + +# -*- coding: utf-8 -*- + +import sys +import time +import random + +import taos +import frame +import frame.etool + + +from frame.log import * +from frame.cases import * +from frame.sql import * +from frame.caseBase import * +from frame import * +from frame.autogen import * + + +class TDTestCase(TBase): + updatecfgDict = { + "compressMsgSize" : "100", + } + # compress + compresses = ["lz4","zlib","zstd","disabled","xz"] + + compressDefaultDict = {}; + compressDefaultDict["BOOL"] = "zstd" + compressDefaultDict["TINYINT"] = "zlib" + compressDefaultDict["SMALLINT"] = "zlib" + compressDefaultDict["INT"] = "lz4" + compressDefaultDict["BIGINT"] = "lz4" + compressDefaultDict["FLOAT"] = "lz4" + compressDefaultDict["DOUBLE"] = "lz4" + compressDefaultDict["VARCHAR"] = "zstd" + compressDefaultDict["TIMESTAMP"] = "lz4" + compressDefaultDict["NCHAR"] = "zstd" + compressDefaultDict["TINYINT UNSIGNED"] = "zlib" + compressDefaultDict["SMALLINT UNSIGNED"] = "zlib" + compressDefaultDict["INT UNSIGNED"] = "lz4" + compressDefaultDict["BIGINT UNSIGNED"] = "lz4" + compressDefaultDict["NCHAR"] = "zstd" + compressDefaultDict["BLOB"] = "lz4" + compressDefaultDict["VARBINARY"] = "zstd" + + # level + levels = ["high","medium","low"] + + # default compress + defCompress = "lz4" + # default level + defLevel = "medium" + + # datatype 17 + dtypes = [ "tinyint","tinyint unsigned","smallint","smallint unsigned","int","int unsigned", + "bigint","bigint unsigned","timestamp","bool","float","double","binary(16)","nchar(16)", + "varchar(16)","varbinary(16)"] + + # encode + encodes = [ + [["tinyint","tinyint unsigned","smallint","smallint unsigned","int","int unsigned","bigint","bigint unsigned"], ["simple8B"]], + [["timestamp","bigint","bigint unsigned"], ["Delta-i"]], + [["bool"], ["Bit-packing"]], + [["float","double"], ["Delta-d"]] + ] + + + def combineValid(self, datatype, encode, compress): + if datatype != "float" and datatype != "double": + if compress == "tsz": + return False + return True + + def genAllSqls(self, stbName, max): + + c = 0 # column number + t = 0 # table number + + sqls = [] + sql = "" + + # loop append sqls + for lines in self.encodes: + for datatype in lines[0]: + for encode in lines[1]: + for compress in self.compresses: + for level in self.levels: + if sql == "": + # first + sql = f"create table {self.db}.st{t} (ts timestamp" + else: + if self.combineValid(datatype, encode, compress): + sql += f", c{c} {datatype} ENCODE '{encode}' COMPRESS '{compress}' LEVEL '{level}'" + c += 1 + + if c >= max: + # append sqls + sql += f") tags(groupid int) " + sqls.append(sql) + # reset + sql = "" + c = 0 + t += 1 + + # break loop + if c > 0: + # append sqls + sql += f") tags(groupid int) " + sqls.append(sql) + + return sqls + + # check error create + def errorCreate(self): + sqls = [ + f"create table terr(ts timestamp, c0 int ENCODE 'simple8B' COMPRESS 'tsz' LEVEL 'high') ", + f"create table terr(ts timestamp, bi bigint encode 'bit-packing') tags (area int);" + f"create table terr(ts timestamp, ic int encode 'delta-d') tags (area int);" + ] + tdSql.errors(sqls) + + for dtype in self.dtypes: + # encode + sql = f"create table terr(ts timestamp, c0 {dtype} ENCODE 'abc') " + tdSql.error(sql) + # compress + sql = f"create table terr(ts timestamp, c0 {dtype} COMPRESS 'def') " + tdSql.error(sql) + # level + sql = f"create table terr(ts timestamp, c0 {dtype} LEVEL 'hig') " + tdSql.error(sql) + + # tsz check + if dtype != "float" and dtype != "double": + sql = f"create table terr(ts timestamp, c0 {dtype} COMPRESS 'tsz') " + tdSql.error(sql) + + # default value correct + def defaultCorrect(self): + # get default encode compress level + sql = f"describe {self.db}.{self.stb}" + tdSql.query(sql) + + # see AutoGen.types + defEncodes = [ "delta-i","delta-i","simple8b","simple8b","simple8b","simple8b","simple8b","simple8b", + "simple8b","simple8b","delta-d","delta-d","bit-packing", + "disabled","disabled","disabled","disabled"] + + count = tdSql.getRows() + for i in range(count): + node = tdSql.getData(i, 3) + if node == "TAG": + break + # check + tdLog.info(f"check default encode {tdSql.getData(i, 1)}") + #tdLog.info(f"check default encode compressDefaultDict[tdSql.getData(i, 2)]") + defaultValue = self.compressDefaultDict[tdSql.getData(i, 1)] + if defaultValue == None: + defaultValue = self.defCompress + tdLog.info(f"check default compress {tdSql.getData(i, 1)} {defaultValue}") + tdSql.checkData(i, 5, defaultValue) + tdSql.checkData(i, 6, self.defLevel) + + # geometry encode is disabled + sql = f"create table {self.db}.ta(ts timestamp, pos geometry(64)) " + tdSql.execute(sql) + sql = f"describe {self.db}.ta" + tdSql.query(sql) + tdSql.checkData(1, 4, "disabled") + + tdLog.info("check default encode compress and level successfully.") + + def checkDataDesc(self, tbname, row, col, value): + sql = f"describe {tbname}" + tdSql.query(sql) + tdSql.checkData(row, col, value) + + + def writeData(self, count): + self.autoGen.insert_data(count, True) + + # alter encode compress level + def checkAlter(self): + tbname = f"{self.db}.{self.stb}" + # alter encode 4 + comp = "delta-i" + sql = f"alter table {tbname} modify column c7 ENCODE '{comp}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, 8, 4, comp) + self.writeData(1000) + sql = f"alter table {tbname} modify column c8 ENCODE '{comp}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, 9, 4, comp) + self.writeData(1000) + + # alter compress 5 + comps = self.compresses[2:] + comps.append(self.compresses[0]) # add lz4 + for comp in comps: + for i in range(self.colCnt - 1): + self.writeData(1000) + + # alter float(c9) double(c10) to tsz + comp = "tsz" + sql = f"alter table {tbname} modify column c9 COMPRESS '{comp}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, 10, 5, comp) + self.writeData(10000) + sql = f"alter table {tbname} modify column c10 COMPRESS '{comp}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, 11, 5, comp) + self.writeData(10000) + + # alter level 6 + for level in self.levels: + for i in range(self.colCnt - 1): + col = f"c{i}" + sql = f"alter table {tbname} modify column {col} LEVEL '{level}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, i + 1, 6, level) + self.writeData(1000) + + # modify two combine + + + i = 9 + encode = "delta-d" + compress = "zlib" + sql = f"alter table {tbname} modify column c{i} ENCODE '{encode}' COMPRESS '{compress}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, i + 1, 4, encode) + self.checkDataDesc(tbname, i + 1, 5, compress) + + i = 10 + encode = "delta-d" + level = "high" + sql = f"alter table {tbname} modify column c{i} ENCODE '{encode}' LEVEL '{level}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, i + 1, 4, encode) + self.checkDataDesc(tbname, i + 1, 6, level) + + i = 2 + compress = "zlib" + level = "high" + sql = f"alter table {tbname} modify column c{i} COMPRESS '{compress}' LEVEL '{level}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, i + 1, 5, compress) + self.checkDataDesc(tbname, i + 1, 6, level) + + # modify three combine + i = 7 + encode = "simple8b" + compress = "zstd" + level = "medium" + sql = f"alter table {tbname} modify column c{i} ENCODE '{encode}' COMPRESS '{compress}' LEVEL '{level}';" + tdSql.execute(sql, show=True) + self.checkDataDesc(tbname, i + 1, 4, encode) + self.checkDataDesc(tbname, i + 1, 5, compress) + self.checkDataDesc(tbname, i + 1, 6, level) + + # alter error + sqls = [ + "alter table nodb.nostb modify column ts LEVEL 'high';", + "alter table db.stb modify column ts encode 'simple8b';", + "alter table db.stb modify column c1 compress 'errorcompress';", + "alter table db.stb modify column c2 level 'errlevel';", + "alter table db.errstb modify column c3 compress 'xz';" + ] + tdSql.errors(sqls) + + # add column + def checkAddColumn(self): + c = 0 + tbname = f"{self.db}.tbadd" + sql = f"create table {tbname}(ts timestamp, c0 int) tags(area int);" + tdSql.execute(sql) + + # loop append sqls + for lines in self.encodes: + for datatype in lines[0]: + for encode in lines[1]: + for compress in self.compresses: + for level in self.levels: + if self.combineValid(datatype, encode, compress): + sql = f"alter table {tbname} add column col{c} {datatype} ENCODE '{encode}' COMPRESS '{compress}' LEVEL '{level}';" + tdSql.execute(sql, 3, True) + c += 1 + + # alter error + sqls = [ + f"alter table {tbname} add column a1 int ONLYOPTION", + f"alter table {tbname} add column a1 int 'simple8b';", + f"alter table {tbname} add column a1 int WRONG 'simple8b';", + f"alter table {tbname} add column a1 int 123456789 'simple8b';", + f"alter table {tbname} add column a1 int WRONGANDVERYLONG 'simple8b';", + f"alter table {tbname} add column a1 int ENCODE 'veryveryveryveryveryverylong';", + f"alter table {tbname} add column a1 int ENCODE 'simple8bAA';", + f"alter table {tbname} add column a2 int COMPRESS 'AABB';", + f"alter table {tbname} add column a3 bigint LEVEL 'high1';", + f"alter table {tbname} add column a4 BINARY(12) ENCODE 'simple8b' LEVEL 'high2';", + f"alter table {tbname} add column a5 VARCHAR(16) ENCODE 'simple8b' COMPRESS 'gzip' LEVEL 'high3';" + ] + tdSql.errors(sqls) + + def validCreate(self): + sqls = self.genAllSqls(self.stb, 50) + tdSql.executes(sqls, show=True) + + # sql syntax + def checkSqlSyntax(self): + + # create tables positive + self.validCreate() + + # create table negtive + self.errorCreate() + + # check default value corrent + self.defaultCorrect() + + # check alter and write + self.checkAlter() + + # check add column + self.checkAddColumn() + + def checkCorrect(self): + # check data correct + tbname = f"{self.db}.{self.stb}" + # count + sql = f"select count(*) from {tbname}" + count = tdSql.getFirstValue(sql) + step = 100000 + offset = 0 + + while offset < count: + sql = f"select * from {tbname} limit {step} offset {offset}" + tdLog.info(sql) + tdSql.query(sql) + self.autoGen.dataCorrect(tdSql.res, tdSql.getRows(), step) + offset += step + tdLog.info(f"check data correct rows={offset}") + + tdLog.info(F"check {tbname} rows {count} data correct successfully.") + + + # run + def run(self): + tdLog.debug(f"start to excute {__file__}") + + # create db and stable + self.autoGen = AutoGen(step = 10, genDataMode = "fillts") + self.autoGen.create_db(self.db, 2, 3) + tdSql.execute(f"use {self.db}") + self.colCnt = 17 + self.autoGen.create_stable(self.stb, 5, self.colCnt, 32, 32) + self.childCnt = 4 + self.autoGen.create_child(self.stb, "d", self.childCnt) + self.autoGen.insert_data(1000) + + # sql syntax + self.checkSqlSyntax() + + # operateor + self.writeData(1000) + self.flushDb() + self.writeData(1000) + + # check corrent + self.checkCorrect() + + tdLog.success(f"{__file__} successfully executed") + + + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/stream/at_once_interval.py b/tests/test_new/stream/at_once_interval.py new file mode 100644 index 0000000000..eb581e84c4 --- /dev/null +++ b/tests/test_new/stream/at_once_interval.py @@ -0,0 +1,235 @@ +import sys +import threading +from util.log import * +from util.sql import * +from util.cases import * +from util.common import * + +class TDTestCase: + updatecfgDict = {'debugFlag': 135, 'asynclog': 0} + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug("start to execute %s" % __file__) + tdSql.init(conn.cursor(), logSql) + self.tdCom = tdCom + + def at_once_interval(self, interval, partition="tbname", delete=False, fill_value=None, fill_history_value=None, case_when=None): + tdLog.info(f"*** testing stream at_once+interval: interval: {interval}, partition: {partition}, fill_history: {fill_history_value}, fill: {fill_value}, delete: {delete}, case_when: {case_when} ***") + col_value_type = "Incremental" if partition=="c1" else "random" + custom_col_index = 1 if partition=="c1" else None + self.tdCom.custom_col_val = 0 + self.delete = delete + self.tdCom.case_name = sys._getframe().f_code.co_name + self.tdCom.prepare_data(interval=interval, fill_history_value=fill_history_value, custom_col_index=custom_col_index, col_value_type=col_value_type) + self.stb_name = self.tdCom.stb_name.replace(f"{self.tdCom.dbname}.", "") + self.ctb_name = self.tdCom.ctb_name.replace(f"{self.tdCom.dbname}.", "") + self.tb_name = self.tdCom.tb_name.replace(f"{self.tdCom.dbname}.", "") + self.stb_stream_des_table = f'{self.stb_name}{self.tdCom.des_table_suffix}' + self.ctb_stream_des_table = f'{self.ctb_name}{self.tdCom.des_table_suffix}' + self.tb_stream_des_table = f'{self.tb_name}{self.tdCom.des_table_suffix}' + if partition == "tbname": + if case_when: + stream_case_when_partition = case_when + else: + stream_case_when_partition = self.tdCom.partition_tbname_alias + + partition_elm_alias = self.tdCom.partition_tbname_alias + elif partition == "c1": + if case_when: + stream_case_when_partition = case_when + else: + stream_case_when_partition = self.tdCom.partition_col_alias + partition_elm_alias = self.tdCom.partition_col_alias + elif partition == "abs(c1)": + partition_elm_alias = self.tdCom.partition_expression_alias + elif partition is None: + partition_elm_alias = '"no_partition"' + else: + partition_elm_alias = self.tdCom.partition_tag_alias + if partition == "tbname" or partition is None: + if case_when: + stb_subtable_value = f'concat(concat("{self.stb_name}_{self.tdCom.subtable_prefix}", {stream_case_when_partition}), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + ctb_subtable_value = f'concat(concat("{self.ctb_name}_{self.tdCom.subtable_prefix}", {stream_case_when_partition}), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + tb_subtable_value = f'concat(concat("{self.tb_name}_{self.tdCom.subtable_prefix}", {stream_case_when_partition}), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + else: + stb_subtable_value = f'concat(concat("{self.stb_name}_{self.tdCom.subtable_prefix}", {partition_elm_alias}), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + ctb_subtable_value = f'concat(concat("{self.ctb_name}_{self.tdCom.subtable_prefix}", {partition_elm_alias}), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + tb_subtable_value = f'concat(concat("{self.tb_name}_{self.tdCom.subtable_prefix}", {partition_elm_alias}), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + else: + stb_subtable_value = f'concat(concat("{self.stb_name}_{self.tdCom.subtable_prefix}", cast(cast(abs(cast({partition_elm_alias} as int)) as bigint) as varchar(100))), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + ctb_subtable_value = f'concat(concat("{self.ctb_name}_{self.tdCom.subtable_prefix}", cast(cast(abs(cast({partition_elm_alias} as int)) as bigint) as varchar(100))), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + tb_subtable_value = f'concat(concat("{self.tb_name}_{self.tdCom.subtable_prefix}", cast(cast(abs(cast({partition_elm_alias} as int)) as bigint) as varchar(100))), "{self.tdCom.subtable_suffix}")' if self.tdCom.subtable else None + if partition: + partition_elm = f'partition by {partition} {partition_elm_alias}' + else: + partition_elm = "" + if fill_value: + if "value" in fill_value.lower(): + fill_value='VALUE,1,2,3,4,5,6,7,8,9,10,11,1,2,3,4,5,6,7,8,9,10,11' + self.tdCom.create_stream(stream_name=f'{self.stb_name}{self.tdCom.stream_suffix}', des_table=self.stb_stream_des_table, source_sql=f'select _wstart AS wstart, {self.tdCom.stb_source_select_str} from {self.stb_name} {partition_elm} interval({self.tdCom.dataDict["interval"]}s)', trigger_mode="at_once", subtable_value=stb_subtable_value, fill_value=fill_value, fill_history_value=fill_history_value) + self.tdCom.create_stream(stream_name=f'{self.ctb_name}{self.tdCom.stream_suffix}', des_table=self.ctb_stream_des_table, source_sql=f'select _wstart AS wstart, {self.tdCom.stb_source_select_str} from {self.ctb_name} {partition_elm} interval({self.tdCom.dataDict["interval"]}s)', trigger_mode="at_once", subtable_value=ctb_subtable_value, fill_value=fill_value, fill_history_value=fill_history_value) + if fill_value: + if "value" in fill_value.lower(): + fill_value='VALUE,1,2,3,4,5,6,7,8,9,10,11' + self.tdCom.create_stream(stream_name=f'{self.tb_name}{self.tdCom.stream_suffix}', des_table=self.tb_stream_des_table, source_sql=f'select _wstart AS wstart, {self.tdCom.tb_source_select_str} from {self.tb_name} {partition_elm} interval({self.tdCom.dataDict["interval"]}s)', trigger_mode="at_once", subtable_value=tb_subtable_value, fill_value=fill_value, fill_history_value=fill_history_value) + start_time = self.tdCom.date_time + + time.sleep(1) + + for i in range(self.tdCom.range_count): + ts_value = str(self.tdCom.date_time+self.tdCom.dataDict["interval"])+f'+{i*10}s' + ts_cast_delete_value = self.tdCom.time_cast(ts_value) + self.tdCom.sinsert_rows(tbname=self.tdCom.ctb_name, ts_value=ts_value, custom_col_index=custom_col_index, col_value_type=col_value_type) + if i%2 == 0: + self.tdCom.sinsert_rows(tbname=self.tdCom.ctb_name, ts_value=ts_value, custom_col_index=custom_col_index, col_value_type=col_value_type) + if self.delete and i%2 != 0: + self.tdCom.sdelete_rows(tbname=self.tdCom.ctb_name, start_ts=ts_cast_delete_value) + self.tdCom.date_time += 1 + self.tdCom.sinsert_rows(tbname=self.tdCom.tb_name, ts_value=ts_value, custom_col_index=custom_col_index, col_value_type=col_value_type) + if i%2 == 0: + self.tdCom.sinsert_rows(tbname=self.tdCom.tb_name, ts_value=ts_value, custom_col_index=custom_col_index, col_value_type=col_value_type) + if self.delete and i%2 != 0: + self.tdCom.sdelete_rows(tbname=self.tdCom.tb_name, start_ts=ts_cast_delete_value) + self.tdCom.date_time += 1 + if partition: + partition_elm = f'partition by {partition}' + else: + partition_elm = "" + + if not fill_value: + for tbname in [self.stb_name, self.ctb_name, self.tb_name]: + if tbname != self.tb_name: + self.tdCom.check_query_data(f'select wstart, {self.tdCom.stb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart', f'select _wstart AS wstart, {self.tdCom.stb_source_select_str} from {tbname} {partition_elm} interval({self.tdCom.dataDict["interval"]}s) order by wstart', sorted=True) + else: + self.tdCom.check_query_data(f'select wstart, {self.tdCom.tb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart', f'select _wstart AS wstart, {self.tdCom.tb_source_select_str} from {tbname} {partition_elm} interval({self.tdCom.dataDict["interval"]}s) order by wstart', sorted=True) + + if self.tdCom.subtable: + for tname in [self.stb_name, self.ctb_name]: + group_id = self.tdCom.get_group_id_from_stb(f'{tname}_output') + tdSql.query(f'select * from {self.ctb_name}') + ptn_counter = 0 + for c1_value in tdSql.queryResult: + if partition == "c1": + tbname = self.tdCom.get_subtable_wait(f'{tname}_{self.tdCom.subtable_prefix}{abs(c1_value[1])}{self.tdCom.subtable_suffix}') + tdSql.query(f'select count(*) from `{tbname}`') + elif partition is None: + tbname = self.tdCom.get_subtable_wait(f'{tname}_{self.tdCom.subtable_prefix}no_partition{self.tdCom.subtable_suffix}') + tdSql.query(f'select count(*) from `{tbname}`') + elif partition == "abs(c1)": + abs_c1_value = abs(c1_value[1]) + tbname = self.tdCom.get_subtable_wait(f'{tname}_{self.tdCom.subtable_prefix}{abs_c1_value}{self.tdCom.subtable_suffix}') + tdSql.query(f'select count(*) from `{tbname}`') + elif partition == "tbname" and ptn_counter == 0: + tbname = self.tdCom.get_subtable_wait(f'{tname}_{self.tdCom.subtable_prefix}{self.ctb_name}{self.tdCom.subtable_suffix}_{tname}_output_{group_id}') + tdSql.query(f'select count(*) from `{tbname}`') + ptn_counter += 1 + tdSql.checkEqual(tdSql.queryResult[0][0] > 0, True) + group_id = self.tdCom.get_group_id_from_stb(f'{self.tb_name}_output') + tdSql.query(f'select * from {self.tb_name}') + ptn_counter = 0 + for c1_value in tdSql.queryResult: + if partition == "c1": + tbname = self.tdCom.get_subtable_wait(f'{self.tb_name}_{self.tdCom.subtable_prefix}{abs(c1_value[1])}{self.tdCom.subtable_suffix}') + tdSql.query(f'select count(*) from `{tbname}`') + elif partition is None: + tbname = self.tdCom.get_subtable_wait(f'{self.tb_name}_{self.tdCom.subtable_prefix}no_partition{self.tdCom.subtable_suffix}') + tdSql.query(f'select count(*) from `{tbname}`') + elif partition == "abs(c1)": + abs_c1_value = abs(c1_value[1]) + tbname = self.tdCom.get_subtable_wait(f'{self.tb_name}_{self.tdCom.subtable_prefix}{abs_c1_value}{self.tdCom.subtable_suffix}') + tdSql.query(f'select count(*) from `{tbname}`') + elif partition == "tbname" and ptn_counter == 0: + tbname = self.tdCom.get_subtable_wait(f'{self.tb_name}_{self.tdCom.subtable_prefix}{self.tb_name}{self.tdCom.subtable_suffix}_{self.tb_name}_output_{group_id}') + tdSql.query(f'select count(*) from `{tbname}`') + ptn_counter += 1 + + tdSql.checkEqual(tdSql.queryResult[0][0] > 0, True) + if fill_value: + end_date_time = self.tdCom.date_time + final_range_count = self.tdCom.range_count + history_ts = str(start_time)+f'-{self.tdCom.dataDict["interval"]*(final_range_count+2)}s' + start_ts = self.tdCom.time_cast(history_ts, "-") + future_ts = str(end_date_time)+f'+{self.tdCom.dataDict["interval"]*(final_range_count+2)}s' + end_ts = self.tdCom.time_cast(future_ts) + self.tdCom.sinsert_rows(tbname=self.ctb_name, ts_value=history_ts) + self.tdCom.sinsert_rows(tbname=self.tb_name, ts_value=history_ts) + self.tdCom.sinsert_rows(tbname=self.ctb_name, ts_value=future_ts) + self.tdCom.sinsert_rows(tbname=self.tb_name, ts_value=future_ts) + self.tdCom.date_time = start_time + # update + history_ts = str(start_time)+f'-{self.tdCom.dataDict["interval"]*(final_range_count+2)}s' + start_ts = self.tdCom.time_cast(history_ts, "-") + future_ts = str(end_date_time)+f'+{self.tdCom.dataDict["interval"]*(final_range_count+2)}s' + end_ts = self.tdCom.time_cast(future_ts) + self.tdCom.sinsert_rows(tbname=self.ctb_name, ts_value=history_ts) + self.tdCom.sinsert_rows(tbname=self.tb_name, ts_value=history_ts) + self.tdCom.sinsert_rows(tbname=self.ctb_name, ts_value=future_ts) + self.tdCom.sinsert_rows(tbname=self.tb_name, ts_value=future_ts) + self.tdCom.date_time = start_time + for i in range(self.tdCom.range_count): + ts_value = str(self.tdCom.date_time+self.tdCom.dataDict["interval"])+f'+{i*10}s' + ts_cast_delete_value = self.tdCom.time_cast(ts_value) + self.tdCom.sinsert_rows(tbname=self.ctb_name, ts_value=ts_value) + self.tdCom.date_time += 1 + self.tdCom.sinsert_rows(tbname=self.tb_name, ts_value=ts_value) + self.tdCom.date_time += 1 + if self.delete: + self.tdCom.sdelete_rows(tbname=self.ctb_name, start_ts=self.tdCom.time_cast(start_time), end_ts=ts_cast_delete_value) + self.tdCom.sdelete_rows(tbname=self.tb_name, start_ts=self.tdCom.time_cast(start_time), end_ts=ts_cast_delete_value) + for tbname in [self.stb_name, self.ctb_name, self.tb_name]: + if tbname != self.tb_name: + if "value" in fill_value.lower(): + fill_value='VALUE,1,2,3,6,7,8,9,10,11,1,2,3,4,5,6,7,8,9,10,11' + if partition == "tbname": + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_stb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart', f'select _wstart AS wstart, {self.tdCom.fill_stb_source_select_str} from {tbname} where ts >= {start_ts} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart', fill_value=fill_value) + else: + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_stb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} where `min(c1)` is not Null order by wstart,`min(c1)`', f'select * from (select _wstart AS wstart, {self.tdCom.fill_stb_source_select_str} from {tbname} where ts >= {start_ts} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart) where `min(c1)` is not Null order by wstart,`min(c1)`', fill_value=fill_value) + else: + if "value" in fill_value.lower(): + fill_value='VALUE,1,2,3,6,7,8,9,10,11' + if partition == "tbname": + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_tb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart', f'select _wstart AS wstart, {self.tdCom.fill_tb_source_select_str} from {tbname} where ts >= {start_ts} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart', fill_value=fill_value) + else: + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_tb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} where `min(c1)` is not Null order by wstart,`min(c1)`', f'select * from (select _wstart AS wstart, {self.tdCom.fill_tb_source_select_str} from {tbname} where ts >= {start_ts} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart) where `min(c1)` is not Null order by wstart,`min(c1)`', fill_value=fill_value) + + if self.delete: + self.tdCom.sdelete_rows(tbname=self.ctb_name, start_ts=start_ts, end_ts=ts_cast_delete_value) + self.tdCom.sdelete_rows(tbname=self.tb_name, start_ts=start_ts, end_ts=ts_cast_delete_value) + for tbname in [self.stb_name, self.ctb_name, self.tb_name]: + if tbname != self.tb_name: + if "value" in fill_value.lower(): + fill_value='VALUE,1,2,3,6,7,8,9,10,11,1,2,3,4,5,6,7,8,9,10,11' + if partition == "tbname": + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_stb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart', f'select _wstart AS wstart, {self.tdCom.fill_stb_source_select_str} from {tbname} where ts >= {start_ts.replace("-", "+")} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart', fill_value=fill_value) + else: + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_stb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart,`min(c1)`', f'select * from (select _wstart AS wstart, {self.tdCom.fill_stb_source_select_str} from {tbname} where ts >= {start_ts} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart) where `min(c1)` is not Null order by wstart,`min(c1)`', fill_value=fill_value) + + else: + if "value" in fill_value.lower(): + fill_value='VALUE,1,2,3,6,7,8,9,10,11' + if partition == "tbname": + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_tb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart', f'select _wstart AS wstart, {self.tdCom.fill_tb_source_select_str} from {tbname} where ts >= {start_ts.replace("-", "+")} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart', fill_value=fill_value) + else: + self.tdCom.check_query_data(f'select wstart, {self.tdCom.fill_tb_output_select_str} from {tbname}{self.tdCom.des_table_suffix} order by wstart,`min(c1)`', f'select * from (select _wstart AS wstart, {self.tdCom.fill_tb_source_select_str} from {tbname} where ts >= {start_ts} and ts <= {end_ts} partition by {partition} interval({self.tdCom.dataDict["interval"]}s) fill ({fill_value}) order by wstart) where `min(c1)` is not Null order by wstart,`min(c1)`', fill_value=fill_value) + + + def run(self): + self.at_once_interval(interval=random.randint(10, 15), partition="tbname", delete=True) + self.at_once_interval(interval=random.randint(10, 15), partition="c1", delete=True) + self.at_once_interval(interval=random.randint(10, 15), partition="abs(c1)", delete=True) + self.at_once_interval(interval=random.randint(10, 15), partition=None, delete=True) + self.at_once_interval(interval=random.randint(10, 15), partition=self.tdCom.stream_case_when_tbname, case_when=f'case when {self.tdCom.stream_case_when_tbname} = tbname then {self.tdCom.partition_tbname_alias} else tbname end') + self.at_once_interval(interval=random.randint(10, 15), partition="tbname", fill_history_value=1, fill_value="NULL") + for fill_value in ["NULL", "PREV", "NEXT", "LINEAR", "VALUE,1,2,3,4,5,6,7,8,9,10,11,1,2,3,4,5,6,7,8,9,10,11"]: + # for fill_value in ["PREV", "NEXT", "LINEAR", "VALUE,1,2,3,4,5,6,7,8,9,10,11,1,2,3,4,5,6,7,8,9,10,11"]: + self.at_once_interval(interval=random.randint(10, 15), partition="tbname", fill_value=fill_value) + self.at_once_interval(interval=random.randint(10, 15), partition="tbname", fill_value=fill_value, delete=True) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +event = threading.Event() + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) \ No newline at end of file diff --git a/tests/test_new/tdgpt/test_gpt.py b/tests/test_new/tdgpt/test_gpt.py new file mode 100644 index 0000000000..16ab6bb22f --- /dev/null +++ b/tests/test_new/tdgpt/test_gpt.py @@ -0,0 +1,67 @@ +from util.log import * +from util.cases import * +from util.sql import * +from util.common import * +import taos + + + +class TDTestCase: + clientCfgDict = {'debugFlag': 135} + updatecfgDict = { + "debugFlag" : "135", + "queryBufferSize" : 10240, + 'clientCfg' : clientCfgDict + } + + def init(self, conn, logSql, replicaVal=1): + self.replicaVar = int(replicaVal) + tdLog.debug(f"start to excute {__file__}") + self.conn = conn + tdSql.init(conn.cursor(), False) + self.passwd = {'root':'taosdata', + 'test':'test'} + + def prepare_anode_data(self): + tdSql.execute(f"create anode '127.0.0.1:6090'") + tdSql.execute(f"create database db_gpt") + tdSql.execute(f"create table if not exists db_gpt.stb (ts timestamp, c1 int, c2 float, c3 double) tags (t1 int unsigned);") + tdSql.execute(f"create table db_gpt.ct1 using db_gpt.stb tags(1000);") + tdSql.execute(f"insert into db_gpt.ct1(ts, c1) values(now-1a, 5)(now+1a, 14)(now+2a, 15)(now+3a, 15)(now+4a, 14);") + tdSql.execute(f"insert into db_gpt.ct1(ts, c1) values(now+5a, 19)(now+6a, 17)(now+7a, 16)(now+8a, 20)(now+9a, 22);") + tdSql.execute(f"insert into db_gpt.ct1(ts, c1) values(now+10a, 8)(now+11a, 21)(now+12a, 28)(now+13a, 11)(now+14a, 9);") + tdSql.execute(f"insert into db_gpt.ct1(ts, c1) values(now+15a, 29)(now+16a, 40);") + + + def test_forecast(self): + """ + Test forecast + """ + tdLog.info(f"Test forecast") + tdSql.query(f"SELECT _frowts, FORECAST(c1, \"algo=arima,alpha=95,period=10,start_p=1,max_p=5,start_q=1,max_q=5,d=1\") from db_gpt.ct1 ;") + tdSql.checkRows(10) + + def test_anomaly_window(self): + """ + Test anomaly window + """ + tdLog.info(f"Test anomaly window") + tdSql.query(f"SELECT _wstart, _wend, SUM(c1) FROM db_gpt.ct1 ANOMALY_WINDOW(c1, \"algo=iqr\");") + tdSql.checkData(0,2,40) + + + def run(self): + self.prepare_anode_data() + self.test_forecast() + self.test_anomaly_window() + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addWindows(__file__, TDTestCase()) +tdCases.addLinux(__file__, TDTestCase()) + + + + diff --git a/tests/test_new/templates/python/material/function.html b/tests/test_new/templates/python/material/function.html new file mode 100644 index 0000000000..d3a9dbb193 --- /dev/null +++ b/tests/test_new/templates/python/material/function.html @@ -0,0 +1,11 @@ +{% extends "function.html.jinja" %} + +{% block logs scoped %} + {{ super() }} + {# TODO: Switch to a warning after some time. #} + {{ log.info( + "DeprecationWarning: Extending '_base/function.html' is deprecated, extend '_base/function.html.jinja' instead. " ~ + "After some time, this message will be logged as a warning, causing strict builds to fail.", + once=True, + ) }} +{% endblock logs %} \ No newline at end of file diff --git a/tests/test_new/templates/python/material/function.html.jinja b/tests/test_new/templates/python/material/function.html.jinja new file mode 100644 index 0000000000..1146007823 --- /dev/null +++ b/tests/test_new/templates/python/material/function.html.jinja @@ -0,0 +1,154 @@ +{#- Template for Python functions. + +This template renders a Python function or method. + +Context: + function (griffe.Function): The function to render. + root (bool): Whether this is the root object, injected with `:::` in a Markdown page. + heading_level (int): The HTML heading level to use. + config (dict): The configuration options. +-#} + +{% block logs scoped %} + {#- Logging block. + + This block can be used to log debug messages, deprecation messages, warnings, etc. + -#} + {{ log.debug("Rendering " + function.path) }} +{% endblock logs %} + +{% import "language"|get_template as lang with context %} +{#- Language module providing the `t` translation method. -#} + +
+ {% with obj = function, html_id = function.path %} + + {% if root %} + {% set show_full_path = config.show_root_full_path %} + {% set root_members = True %} + {% elif root_members %} + {% set show_full_path = config.show_root_members_full_path or config.show_object_full_path %} + {% set root_members = False %} + {% else %} + {% set show_full_path = config.show_object_full_path %} + {% endif %} + + {% set function_name = function.path if show_full_path else function.name %} + {#- Brief or full function name depending on configuration. -#} + {% set symbol_type = "method" if function.parent.is_class else "function" %} + {#- Symbol type: method when parent is a class, function otherwise. -#} + + {% if not root or config.show_root_heading %} + {% filter heading( + heading_level, + role="function", + id=html_id, + class="doc doc-heading", + toc_label=((' ')|safe if config.show_symbol_type_toc else '') + function.docstring.value.split("\n", 1)[0] if function.docstring.value.split("\n", 1)[0] else function.name, + ) %} + + {% block heading scoped %} + {#- Heading block. + + This block renders the heading for the function. + -#} + {% if config.show_symbol_type_heading %}{% endif %} + {% if config.separate_signature %} + {{ config.heading if config.heading and root else function_name }} + {% else %} + {%+ filter highlight(language="python", inline=True) %} + {{ function_name }}{% include "signature"|get_template with context %} + {% endfilter %} + {% endif %} + {% endblock heading %} + + {% block labels scoped %} + {#- Labels block. + + This block renders the labels for the function. + -#} + {% with labels = function.labels %} + {% include "labels"|get_template with context %} + {% endwith %} + {% endblock labels %} + + {% endfilter %} + + {% block signature scoped %} + {#- Signature block. + + This block renders the signature for the function, + as well as its overloaded signatures if any. + -#} + {% if function.overloads %} +
+ {% for overload in function.overloads %} + {% filter format_signature(overload, config.line_length, annotations=True, crossrefs=config.signature_crossrefs) %} + {{ overload.name }} + {% endfilter %} + {% endfor %} +
+ {% endif %} + {% if config.separate_signature %} + {% filter format_signature(function, config.line_length, crossrefs=config.signature_crossrefs) %} + {{ function.name }} + {% endfilter %} + {% endif %} + {% endblock signature %} + + {% else %} + + {% if config.show_root_toc_entry %} + {% filter heading( + heading_level, + role="function", + id=html_id, + toc_label=((' ')|safe if config.show_symbol_type_toc else '') + (config.toc_label if config.toc_label and root else function.name), + hidden=True, + ) %} + {% endfilter %} + {% endif %} + {% set heading_level = heading_level - 1 %} + {% endif %} + +
+ {% block contents scoped %} + {#- Contents block. + + This block renders the contents of the function. + It contains other blocks that users can override. + Overriding the contents block allows to rearrange the order of the blocks. + -#} + {% block docstring scoped %} + {#- Docstring block. + + This block renders the docstring for the function. + -#} + {% with docstring_sections = function.docstring.parsed %} + {% include "docstring"|get_template with context %} + {% endwith %} + {% endblock docstring %} + + {% block source scoped %} + {#- Source block. + + This block renders the source code for the function. + -#} + {% if config.show_source and function.source %} +
+ {{ lang.t("Source code in") }} + {%- if function.relative_filepath.is_absolute() -%} + {{ function.relative_package_filepath }} + {%- else -%} + {{ function.relative_filepath }} + {%- endif -%} + + {{ function.source|highlight(language="python", linestart=function.lineno or 0, linenums=True) }} +
+ {% endif %} + {% endblock source %} + {% endblock contents %} +
+ + {% endwith %} +
diff --git a/tests/test_new/test.py b/tests/test_new/test.py new file mode 100644 index 0000000000..ab1bdc21d3 --- /dev/null +++ b/tests/test_new/test.py @@ -0,0 +1,704 @@ +#!/usr/bin/python +################################################################### +# Copyright (c) 2016 by TAOS Technologies, Inc. +# All rights reserved. +# +# This file is proprietary and confidential to TAOS Technologies. +# No part of this file may be reproduced, stored, transmitted, +# disclosed or used in any form or by any means other than as +# expressly provided by the written permission from Jianhui Tao +# +################################################################### +# install pip +# pip install src/connector/python/ + +# -*- coding: utf-8 -*- +import os +import sys +import getopt +import subprocess +import time +import base64 +import json +import platform +import socket +import threading +import importlib +print(f"Python version: {sys.version}") +print(f"Version info: {sys.version_info}") + +import toml +sys.path.append("../pytest") +from util.log import * +from util.dnodes import * +from util.cases import * +from util.cluster import * +from util.taosadapter import * + +import taos +import taosrest +import taosws + +def checkRunTimeError(): + import win32gui + timeCount = 0 + while 1: + time.sleep(1) + timeCount = timeCount + 1 + print("checkRunTimeError",timeCount) + if (timeCount>1200): + print("stop the test.") + os.system("TASKKILL /F /IM taosd.exe") + os.system("TASKKILL /F /IM taos.exe") + os.system("TASKKILL /F /IM tmq_sim.exe") + os.system("TASKKILL /F /IM mintty.exe") + os.system("TASKKILL /F /IM python.exe") + quit(0) + hwnd = win32gui.FindWindow(None, "Microsoft Visual C++ Runtime Library") + if hwnd: + os.system("TASKKILL /F /IM taosd.exe") + +# +# run case on previous cluster +# +def runOnPreviousCluster(host, config, fileName): + print("enter run on previeous") + + # load case module + sep = "/" + if platform.system().lower() == 'windows': + sep = os.sep + moduleName = fileName.replace(".py", "").replace(sep, ".") + uModule = importlib.import_module(moduleName) + case = uModule.TDTestCase() + + # create conn + conn = taos.connect(host, config) + + # run case + case.init(conn, False) + try: + case.run() + except Exception as e: + tdLog.notice(repr(e)) + tdLog.exit("%s failed" % (fileName)) + # stop + case.stop() + + +if __name__ == "__main__": + + # + # analysis paramaters + # + fileName = "all" + deployPath = "" + masterIp = "" + testCluster = False + valgrind = 0 + killValgrind = 1 + logSql = True + stop = 0 + restart = False + dnodeNums = 1 + mnodeNums = 0 + updateCfgDict = {} + adapter_cfg_dict = {} + execCmd = "" + queryPolicy = 1 + createDnodeNums = 1 + restful = False + websocket = False + replicaVar = 1 + asan = False + independentMnode = False + previousCluster = False + crashGen = False + opts, args = getopt.gnu_getopt(sys.argv[1:], 'f:p:m:l:scghrd:k:e:N:M:Q:C:RWD:n:i:aP:G', [ + 'file=', 'path=', 'master', 'logSql', 'stop', 'cluster', 'valgrind', 'help', 'restart', 'updateCfgDict', 'killv', 'execCmd','dnodeNums','mnodeNums','queryPolicy','createDnodeNums','restful','websocket','adaptercfgupdate','replicaVar','independentMnode','previous',"crashGen"]) + for key, value in opts: + if key in ['-h', '--help']: + tdLog.printNoPrefix( + 'A collection of test cases written using Python') + tdLog.printNoPrefix('-f Name of test case file written by Python') + tdLog.printNoPrefix('-p Deploy Path for Simulator') + tdLog.printNoPrefix('-m Master Ip for Simulator') + tdLog.printNoPrefix('-l logSql Flag') + tdLog.printNoPrefix('-s stop All dnodes') + tdLog.printNoPrefix('-c Test Cluster Flag') + tdLog.printNoPrefix('-g valgrind Test Flag') + tdLog.printNoPrefix('-r taosd restart test') + tdLog.printNoPrefix('-d update cfg dict, base64 json str') + tdLog.printNoPrefix('-k not kill valgrind processer') + tdLog.printNoPrefix('-e eval str to run') + tdLog.printNoPrefix('-N start dnodes numbers in clusters') + tdLog.printNoPrefix('-M create mnode numbers in clusters') + tdLog.printNoPrefix('-Q set queryPolicy in one dnode') + tdLog.printNoPrefix('-C create Dnode Numbers in one cluster') + tdLog.printNoPrefix('-R restful realization form') + tdLog.printNoPrefix('-W websocket connection') + tdLog.printNoPrefix('-D taosadapter update cfg dict ') + tdLog.printNoPrefix('-n the number of replicas') + tdLog.printNoPrefix('-i independentMnode Mnode') + tdLog.printNoPrefix('-a address sanitizer mode') + tdLog.printNoPrefix('-P run case with [P]revious cluster, do not create new cluster to run case.') + tdLog.printNoPrefix('-G crashGen mode') + + sys.exit(0) + + if key in ['-r', '--restart']: + restart = True + + if key in ['-f', '--file']: + fileName = value + + if key in ['-p', '--path']: + deployPath = value + + if key in ['-m', '--master']: + masterIp = value + + if key in ['-l', '--logSql']: + if (value.upper() == "TRUE"): + logSql = True + elif (value.upper() == "FALSE"): + logSql = False + else: + tdLog.printNoPrefix("logSql value %s is invalid" % logSql) + sys.exit(0) + + if key in ['-c', '--cluster']: + testCluster = True + + if key in ['-g', '--valgrind']: + valgrind = 1 + + if key in ['-s', '--stop']: + stop = 1 + + if key in ['-d', '--updateCfgDict']: + try: + updateCfgDict = eval(base64.b64decode(value.encode()).decode()) + except: + print('updateCfgDict convert fail.') + sys.exit(0) + + if key in ['-k', '--killValgrind']: + killValgrind = 1 + + if key in ['-e', '--execCmd']: + try: + execCmd = base64.b64decode(value.encode()).decode() + except: + print('execCmd run fail.') + sys.exit(0) + + if key in ['-N', '--dnodeNums']: + dnodeNums = value + + if key in ['-M', '--mnodeNums']: + mnodeNums = value + + if key in ['-Q', '--queryPolicy']: + queryPolicy = value + + if key in ['-C', '--createDnodeNums']: + createDnodeNums = value + + if key in ['-i', '--independentMnode']: + independentMnode = value + + if key in ['-R', '--restful']: + restful = True + + if key in ['-W', '--websocket']: + websocket = True + + if key in ['-a', '--asan']: + asan = True + + if key in ['-D', '--adaptercfgupdate']: + try: + adaptercfgupdate = eval(base64.b64decode(value.encode()).decode()) + except: + print('adapter cfg update convert fail.') + sys.exit(0) + + if key in ['-n', '--replicaVar']: + replicaVar = value + + if key in ['-P', '--previous']: + previousCluster = True + + if key in ['-G', '--crashGen']: + crashGen = True + + + # + # do exeCmd command + # + if not execCmd == "": + if restful or websocket: + tAdapter.init(deployPath) + else: + tdDnodes.init(deployPath) + print(execCmd) + exec(execCmd) + quit() + + # + # do stop option + # + if (stop != 0): + if (valgrind == 0): + toBeKilled = "taosd" + else: + toBeKilled = "valgrind.bin" + + killCmd = "ps -ef|grep -w %s| grep -v grep | awk '{print $2}' | xargs kill -TERM > /dev/null 2>&1" % toBeKilled + + psCmd = "ps -ef|grep -w %s| grep -v grep | awk '{print $2}'" % toBeKilled + processID = subprocess.check_output(psCmd, shell=True) + + while(processID): + os.system(killCmd) + time.sleep(1) + processID = subprocess.check_output(psCmd, shell=True) + + for port in range(6030, 6041): + usePortPID = "lsof -i tcp:%d | grep LISTEN | awk '{print $2}'" % port + processID = subprocess.check_output(usePortPID, shell=True) + + if processID: + killCmd = "kill -TERM %s" % processID + os.system(killCmd) + fuserCmd = "fuser -k -n tcp %d" % port + os.system(fuserCmd) + if valgrind: + time.sleep(2) + + if restful or websocket: + toBeKilled = "taosadapter" + + # killCmd = "ps -ef|grep -w %s| grep -v grep | awk '{print $2}' | xargs kill -TERM > /dev/null 2>&1" % toBeKilled + killCmd = f"pkill {toBeKilled}" + + psCmd = "ps -ef|grep -w %s| grep -v grep | awk '{print $2}'" % toBeKilled + # psCmd = f"pgrep {toBeKilled}" + processID = subprocess.check_output(psCmd, shell=True) + + while(processID): + os.system(killCmd) + time.sleep(1) + processID = subprocess.check_output(psCmd, shell=True) + + port = 6041 + usePortPID = f"lsof -i tcp:{port} | grep LISTEN | awk '{{print $2}}'" + processID = subprocess.check_output(usePortPID, shell=True) + + if processID: + killCmd = f"kill -TERM {processID}" + os.system(killCmd) + fuserCmd = f"fuser -k -n tcp {port}" + os.system(fuserCmd) + + tdLog.info('stop taosadapter') + + tdLog.info('stop All dnodes') + + # + # get hostname + # + if masterIp == "": + host = socket.gethostname() + else: + try: + config = eval(masterIp) + host = config["host"] + except Exception as r: + host = masterIp + tdLog.info("Procedures for tdengine deployed in %s" % (host)) + + # + # do previousCluster option + # + if previousCluster: + tdDnodes.init(deployPath, masterIp) + runOnPreviousCluster(host, tdDnodes.getSimCfgPath(), fileName) + tdLog.info("run on previous cluster end.") + quit() + + # + # windows run + # + if platform.system().lower() == 'windows': + fileName = fileName.replace("/", os.sep) + if (masterIp == "" and not fileName == "0-others\\udf_create.py"): + threading.Thread(target=checkRunTimeError,daemon=True).start() + tdLog.info("Procedures for testing self-deployment") + tdDnodes.init(deployPath, masterIp) + tdDnodes.setTestCluster(testCluster) + tdDnodes.setValgrind(valgrind) + tdDnodes.stopAll() + key_word = 'tdCases.addWindows' + is_test_framework = 0 + try: + if key_word in open(fileName, encoding='UTF-8').read(): + is_test_framework = 1 + except Exception as r: + print(r) + updateCfgDictStr = '' + # adapter_cfg_dict_str = '' + if is_test_framework: + moduleName = fileName.replace(".py", "").replace(os.sep, ".") + uModule = importlib.import_module(moduleName) + try: + ucase = uModule.TDTestCase() + if ((json.dumps(updateCfgDict) == '{}') and hasattr(ucase, 'updatecfgDict')): + updateCfgDict = ucase.updatecfgDict + updateCfgDictStr = "-d %s"%base64.b64encode(json.dumps(updateCfgDict).encode()).decode() + if ((json.dumps(adapter_cfg_dict) == '{}') and hasattr(ucase, 'taosadapter_cfg_dict')): + adapter_cfg_dict = ucase.taosadapter_cfg_dict + # adapter_cfg_dict_str = f"-D {base64.b64encode(toml.dumps(adapter_cfg_dict).encode()).decode()}" + except Exception as r: + print(r) + else: + pass + # if restful: + tAdapter.init(deployPath, masterIp) + tAdapter.stop(force_kill=True) + + if dnodeNums == 1 : + tdDnodes.deploy(1,updateCfgDict) + tdDnodes.start(1) + tdCases.logSql(logSql) + if restful or websocket: + tAdapter.deploy(adapter_cfg_dict) + tAdapter.start() + + if queryPolicy != 1: + queryPolicy=int(queryPolicy) + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host,config=tdDnodes.getSimCfgPath()) + + cursor = conn.cursor() + cursor.execute("create qnode on dnode 1") + cursor.execute(f'alter local "queryPolicy" "{queryPolicy}"') + cursor.execute("show local variables") + res = cursor.fetchall() + for i in range(cursor.rowcount): + if res[i][0] == "queryPolicy" : + if int(res[i][1]) == int(queryPolicy): + tdLog.info(f'alter queryPolicy to {queryPolicy} successfully') + cursor.close() + else: + tdLog.debug(res) + tdLog.exit(f"alter queryPolicy to {queryPolicy} failed") + else : + tdLog.debug("create an cluster with %s nodes and make %s dnode as independent mnode"%(dnodeNums,mnodeNums)) + dnodeslist = cluster.configure_cluster(dnodeNums=dnodeNums, mnodeNums=mnodeNums, independentMnode=independentMnode) + tdDnodes = ClusterDnodes(dnodeslist) + tdDnodes.init(deployPath, masterIp) + tdDnodes.setTestCluster(testCluster) + tdDnodes.setValgrind(valgrind) + tdDnodes.stopAll() + for dnode in tdDnodes.dnodes: + tdDnodes.deploy(dnode.index, updateCfgDict) + for dnode in tdDnodes.dnodes: + tdDnodes.starttaosd(dnode.index) + tdCases.logSql(logSql) + + if restful or websocket: + tAdapter.deploy(adapter_cfg_dict) + tAdapter.start() + + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host,config=tdDnodes.getSimCfgPath()) + # tdLog.info(tdDnodes.getSimCfgPath(),host) + if createDnodeNums == 1: + createDnodeNums=dnodeNums + else: + createDnodeNums=createDnodeNums + cluster.create_dnode(conn,createDnodeNums) + cluster.create_mnode(conn,mnodeNums) + try: + if cluster.check_dnode(conn) : + print("check dnode ready") + except Exception as r: + print(r) + if queryPolicy != 1: + queryPolicy=int(queryPolicy) + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host,config=tdDnodes.getSimCfgPath()) + + cursor = conn.cursor() + cursor.execute("create qnode on dnode 1") + cursor.execute(f'alter local "queryPolicy" "{queryPolicy}"') + cursor.execute("show local variables") + res = cursor.fetchall() + for i in range(cursor.rowcount): + if res[i][0] == "queryPolicy" : + if int(res[i][1]) == int(queryPolicy): + tdLog.info(f'alter queryPolicy to {queryPolicy} successfully') + cursor.close() + else: + tdLog.debug(res) + tdLog.exit(f"alter queryPolicy to {queryPolicy} failed") + + if ucase is not None and hasattr(ucase, 'noConn') and ucase.noConn == True: + conn = None + else: + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + + if testCluster: + tdLog.info("Procedures for testing cluster") + if fileName == "all": + tdCases.runAllCluster() + else: + tdCases.runOneCluster(fileName) + else: + tdLog.info("Procedures for testing self-deployment") + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + + if fileName == "all": + tdCases.runAllWindows(conn) + else: + tdCases.runOneWindows(conn, fileName, replicaVar) + + if restart: + if fileName == "all": + tdLog.info("not need to query ") + else: + sp = fileName.rsplit(".", 1) + if len(sp) == 2 and sp[1] == "py": + tdDnodes.stopAll() + tdDnodes.start(1) + time.sleep(1) + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + tdLog.info("Procedures for tdengine deployed in %s" % (host)) + tdLog.info("query test after taosd restart") + tdCases.runOneWindows(conn, sp[0] + "_" + "restart.py", replicaVar) + else: + tdLog.info("not need to query") + else: + tdDnodes.setKillValgrind(killValgrind) + tdDnodes.init(deployPath, masterIp) + tdDnodes.setTestCluster(testCluster) + tdDnodes.setValgrind(valgrind) + tdDnodes.setAsan(asan) + tdDnodes.stopAll() + is_test_framework = 0 + key_word = 'tdCases.addLinux' + try: + if key_word in open(fileName).read(): + is_test_framework = 1 + except: + pass + if is_test_framework: + moduleName = fileName.replace(".py", "").replace("/", ".") + uModule = importlib.import_module(moduleName) + try: + ucase = uModule.TDTestCase() + if (json.dumps(updateCfgDict) == '{}'): + updateCfgDict = ucase.updatecfgDict + if (json.dumps(adapter_cfg_dict) == '{}'): + adapter_cfg_dict = ucase.taosadapter_cfg_dict + except: + pass + + if restful or websocket: + tAdapter.init(deployPath, masterIp) + tAdapter.stop(force_kill=True) + + if dnodeNums == 1 : + # dnode is one + tdDnodes.deploy(1,updateCfgDict) + tdDnodes.start(1) + tdCases.logSql(logSql) + + if restful or websocket: + tAdapter.deploy(adapter_cfg_dict) + tAdapter.start() + + if queryPolicy != 1: + queryPolicy=int(queryPolicy) + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + # tdSql.init(conn.cursor()) + # tdSql.execute("create qnode on dnode 1") + # tdSql.execute('alter local "queryPolicy" "%d"'%queryPolicy) + # tdSql.query("show local variables;") + # for i in range(tdSql.queryRows): + # if tdSql.queryResult[i][0] == "queryPolicy" : + # if int(tdSql.queryResult[i][1]) == int(queryPolicy): + # tdLog.info('alter queryPolicy to %d successfully'%queryPolicy) + # else : + # tdLog.debug(tdSql.queryResult) + # tdLog.exit("alter queryPolicy to %d failed"%queryPolicy) + + cursor = conn.cursor() + cursor.execute("create qnode on dnode 1") + cursor.execute(f'alter local "queryPolicy" "{queryPolicy}"') + cursor.execute("show local variables") + res = cursor.fetchall() + for i in range(cursor.rowcount): + if res[i][0] == "queryPolicy" : + if int(res[i][1]) == int(queryPolicy): + tdLog.info(f'alter queryPolicy to {queryPolicy} successfully') + cursor.close() + else: + tdLog.debug(res) + tdLog.exit(f"alter queryPolicy to {queryPolicy} failed") + + else : + # dnode > 1 cluster + tdLog.debug("create an cluster with %s nodes and make %s dnode as independent mnode"%(dnodeNums,mnodeNums)) + print(independentMnode,"independentMnode valuse") + dnodeslist = cluster.configure_cluster(dnodeNums=dnodeNums, mnodeNums=mnodeNums, independentMnode=independentMnode) + tdDnodes = ClusterDnodes(dnodeslist) + tdDnodes.init(deployPath, masterIp) + tdDnodes.setTestCluster(testCluster) + tdDnodes.setValgrind(valgrind) + tdDnodes.setAsan(asan) + tdDnodes.stopAll() + for dnode in tdDnodes.dnodes: + tdDnodes.deploy(dnode.index,updateCfgDict) + for dnode in tdDnodes.dnodes: + tdDnodes.starttaosd(dnode.index) + tdCases.logSql(logSql) + + if restful or websocket: + tAdapter.deploy(adapter_cfg_dict) + tAdapter.start() + + # create taos connect + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + print(tdDnodes.getSimCfgPath(),host) + if createDnodeNums == 1: + createDnodeNums=dnodeNums + else: + createDnodeNums=createDnodeNums + cluster.create_dnode(conn,createDnodeNums) + cluster.create_mnode(conn,mnodeNums) + + try: + if cluster.check_dnode(conn) : + print("check dnode ready") + except Exception as r: + print(r) + + # do queryPolicy option + if queryPolicy != 1: + queryPolicy=int(queryPolicy) + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + + cursor = conn.cursor() + cursor.execute("create qnode on dnode 1") + cursor.execute(f'alter local "queryPolicy" "{queryPolicy}"') + cursor.execute("show local variables") + res = cursor.fetchall() + for i in range(cursor.rowcount): + if res[i][0] == "queryPolicy" : + if int(res[i][1]) == int(queryPolicy): + tdLog.info(f'alter queryPolicy to {queryPolicy} successfully') + cursor.close() + else: + tdLog.debug(res) + tdLog.exit(f"alter queryPolicy to {queryPolicy} failed") + + + # run case + if testCluster: + tdLog.info("Procedures for testing cluster") + if fileName == "all": + tdCases.runAllCluster() + else: + tdCases.runOneCluster(fileName) + else: + tdLog.info("Procedures for testing self-deployment") + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + + if fileName == "all": + tdCases.runAllLinux(conn) + else: + tdCases.runOneLinux(conn, fileName, replicaVar) + + # do restart option + if restart: + if fileName == "all": + tdLog.info("not need to query ") + else: + sp = fileName.rsplit(".", 1) + if len(sp) == 2 and sp[1] == "py": + tdDnodes.stopAll() + tdDnodes.start(1) + time.sleep(1) + if restful: + conn = taosrest.connect(url=f"http://{host}:6041",timezone="utc") + elif websocket: + conn = taosws.connect(f"taosws://root:taosdata@{host}:6041") + else: + conn = taos.connect(host=f"{host}", config=tdDnodes.getSimCfgPath()) + tdLog.info("Procedures for tdengine deployed in %s" % (host)) + tdLog.info("query test after taosd restart") + tdCases.runOneLinux(conn, sp[0] + "_" + "restart.py", replicaVar) + else: + tdLog.info("not need to query") + + # close for end + if conn is not None: + conn.close() + if asan: + # tdDnodes.StopAllSigint() + tdLog.info("Address sanitizer mode finished") + else: + if not crashGen: + tdDnodes.stopAll() + tdLog.info("stop all td process finished") + sys.exit(0) diff --git a/tests/test_new/tmq/subscribeDb.py b/tests/test_new/tmq/subscribeDb.py new file mode 100644 index 0000000000..631813e7ab --- /dev/null +++ b/tests/test_new/tmq/subscribeDb.py @@ -0,0 +1,505 @@ + +import taos +import sys +import time +import socket +import os +import threading + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * + +class TDTestCase: + hostname = socket.gethostname() + # rpcDebugFlagVal = '143' + #clientCfgDict = {'serverPort': '', 'firstEp': '', 'secondEp':'', 'rpcDebugFlag':'135', 'fqdn':''} + #clientCfgDict["rpcDebugFlag"] = rpcDebugFlagVal + #updatecfgDict = {'clientCfg': {}, 'serverPort': '', 'firstEp': '', 'secondEp':'', 'rpcDebugFlag':'135', 'fqdn':''} + # updatecfgDict["rpcDebugFlag"] = rpcDebugFlagVal + #print ("===================: ", updatecfgDict) + + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor()) + #tdSql.init(conn.cursor(), logSql) # output sql.txt file + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files or "taosd.exe" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def newcur(self,cfg,host,port): + user = "root" + password = "taosdata" + con=taos.connect(host=host, user=user, password=password, config=cfg ,port=port) + cur=con.cursor() + print(cur) + return cur + + def initConsumerTable(self,cdbName='cdb'): + tdLog.info("create consume database, and consume info table, and consume result table") + tdSql.query("create database if not exists %s vgroups 1 wal_retention_period 3600"%(cdbName)) + tdSql.query("drop table if exists %s.consumeinfo "%(cdbName)) + tdSql.query("drop table if exists %s.consumeresult "%(cdbName)) + + tdSql.query("create table %s.consumeinfo (ts timestamp, consumerid int, topiclist binary(1024), keylist binary(1024), expectmsgcnt bigint, ifcheckdata int, ifmanualcommit int)"%cdbName) + tdSql.query("create table %s.consumeresult (ts timestamp, consumerid int, consummsgcnt bigint, consumrowcnt bigint, checkresult int)"%cdbName) + + def insertConsumerInfo(self,consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifmanualcommit,cdbName='cdb'): + sql = "insert into %s.consumeinfo values "%cdbName + sql += "(now + %ds, %d, '%s', '%s', %d, %d, %d)"%(consumerId, consumerId, topicList, keyList, expectrowcnt, ifcheckdata, ifmanualcommit) + tdLog.info("consume info sql: %s"%sql) + tdSql.query(sql) + + def selectConsumeResult(self,expectRows,cdbName='cdb'): + resultList=[] + while 1: + tdSql.query("select * from %s.consumeresult"%cdbName) + #tdLog.info("row: %d, %l64d, %l64d"%(tdSql.getData(0, 1),tdSql.getData(0, 2),tdSql.getData(0, 3)) + if tdSql.getRows() == expectRows: + break + else: + time.sleep(5) + + for i in range(expectRows): + tdLog.info ("consume id: %d, consume msgs: %d, consume rows: %d"%(tdSql.getData(i , 1), tdSql.getData(i , 2), tdSql.getData(i , 3))) + resultList.append(tdSql.getData(i , 3)) + + return resultList + + def startTmqSimProcess(self,buildPath,cfgPath,pollDelay,dbName,showMsg=1,showRow=1,cdbName='cdb',valgrind=0): + if valgrind == 1: + logFile = cfgPath + '/../log/valgrind-tmq.log' + shellCmd = 'nohup valgrind --log-file=' + logFile + shellCmd += '--tool=memcheck --leak-check=full --show-reachable=no --track-origins=yes --show-leak-kinds=all --num-callers=20 -v --workaround-gcc296-bugs=yes ' + + if (platform.system().lower() == 'windows'): + shellCmd = 'mintty -h never -w hide ' + buildPath + '\\build\\bin\\tmq_sim.exe -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> nul 2>&1 &" + else: + shellCmd = 'nohup ' + buildPath + '/build/bin/tmq_sim -c ' + cfgPath + shellCmd += " -y %d -d %s -g %d -r %d -w %s "%(pollDelay, dbName, showMsg, showRow, cdbName) + shellCmd += "> /dev/null 2>&1 &" + tdLog.info(shellCmd) + os.system(shellCmd) + + def create_tables(self,tsql, dbName,vgroups,stbName,ctbNum): + tsql.execute("create database if not exists %s vgroups %d wal_retention_period 3600"%(dbName, vgroups)) + tsql.execute("use %s" %dbName) + tsql.execute("create table if not exists %s (ts timestamp, c1 bigint, c2 binary(16)) tags(t1 int)"%stbName) + pre_create = "create table" + sql = pre_create + #tdLog.debug("doing create one stable %s and %d child table in %s ..." %(stbname, count ,dbname)) + for i in range(ctbNum): + sql += " %s_%d using %s tags(%d)"%(stbName,i,stbName,i+1) + if (i > 0) and (i%100 == 0): + tsql.execute(sql) + sql = pre_create + if sql != pre_create: + tsql.execute(sql) + + event.set() + tdLog.debug("complete to create database[%s], stable[%s] and %d child tables" %(dbName, stbName, ctbNum)) + return + + def insert_data(self,tsql,dbName,stbName,ctbNum,rowsPerTbl,batchNum,startTs): + tdLog.debug("start to insert data ............") + tsql.execute("use %s" %dbName) + pre_insert = "insert into " + sql = pre_insert + + t = time.time() + startTs = int(round(t * 1000)) + #tdLog.debug("doing insert data into stable:%s rows:%d ..."%(stbName, allRows)) + for i in range(ctbNum): + sql += " %s_%d values "%(stbName,i) + for j in range(rowsPerTbl): + sql += "(%d, %d, 'tmqrow_%d') "%(startTs + j, j, j) + if (j > 0) and ((j%batchNum == 0) or (j == rowsPerTbl - 1)): + tsql.execute(sql) + if j < rowsPerTbl - 1: + sql = "insert into %s_%d values " %(stbName,i) + else: + sql = "insert into " + #end sql + if sql != pre_insert: + #print("insert sql:%s"%sql) + tsql.execute(sql) + tdLog.debug("insert data ............ [OK]") + return + + def prepareEnv(self, **parameterDict): + print ("input parameters:") + print (parameterDict) + # create new connector for my thread + tsql=self.newcur(parameterDict['cfg'], 'localhost', 6030) + self.create_tables(tsql,\ + parameterDict["dbName"],\ + parameterDict["vgroups"],\ + parameterDict["stbName"],\ + parameterDict["ctbNum"]) + + self.insert_data(tsql,\ + parameterDict["dbName"],\ + parameterDict["stbName"],\ + parameterDict["ctbNum"],\ + parameterDict["rowsPerTbl"],\ + parameterDict["batchNum"],\ + parameterDict["startTs"]) + return + + def tmqCase1(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 1: Produce while one consume to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db1', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 5000, \ + 'batchNum': 100, \ + 'replica': self.replicaVar, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d replica %d wal_retention_period 3600" %(parameterDict['dbName'], parameterDict['vgroups'], parameterDict['replica'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 0 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 100 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + # wait for data ready + prepareEnvThread.join() + + tdLog.info("1-insert process end, and start to check consume result") + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.info("creat the same topic name , and start to consume") + self.initConsumerTable() + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 0 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + tdLog.info("start consume processor") + pollDelay = 20 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 1 end ...... ") + + def tmqCase2(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 2: Produce while two consumers to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db2', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 5000, \ + 'batchNum': 100, \ + 'replica': self.replicaVar, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d replica %d wal_retention_period 3600" %(parameterDict['dbName'], parameterDict['vgroups'], parameterDict['replica'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 1 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + consumerId = 1 + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 20 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + # wait for data ready + prepareEnvThread.join() + + tdLog.info("2-insert process end, and start to check consume result") + expectRows = 2 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + if not (totalConsumeRows >= expectrowcnt): + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 2 end ...... ") + + def tmqCase2a(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 2a: Produce while two consumers to subscribe one db, inclue 1 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db2a', \ + 'vgroups': 4, \ + 'stbName': 'stb1', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 5000, \ + 'batchNum': 100, \ + 'replica': self.replicaVar, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d wal_retention_period 3600" %(parameterDict['dbName'], parameterDict['vgroups'])) + tdSql.execute("create table if not exists %s.%s (ts timestamp, c1 bigint, c2 binary(16)) tags(t1 int)"%(parameterDict['dbName'], parameterDict['stbName'])) + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 1 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + consumerId = 1 + keyList = 'group.id:cgrp2,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + tdLog.info("start consume processor") + pollDelay = 100 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + # wait for data ready + prepareEnvThread.join() + + tdLog.info("3-insert process end, and start to check consume result") + expectRows = 2 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt * 2: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt*2)) + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 2a end ...... ") + + def tmqCase3(self, cfgPath, buildPath): + tdLog.printNoPrefix("======== test case 3: Produce while one consumers to subscribe one db, include 2 stb") + tdLog.info("step 1: create database, stb, ctb and insert data") + # create and start thread + parameterDict = {'cfg': '', \ + 'dbName': 'db3', \ + 'vgroups': 4, \ + 'stbName': 'stb', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 5000, \ + 'batchNum': 100, \ + 'replica': self.replicaVar, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + self.initConsumerTable() + + tdSql.execute("create database if not exists %s vgroups %d replica %d wal_retention_period 3600" %(parameterDict['dbName'], parameterDict['vgroups'], parameterDict['replica'])) + + prepareEnvThread = threading.Thread(target=self.prepareEnv, kwargs=parameterDict) + prepareEnvThread.start() + + parameterDict2 = {'cfg': '', \ + 'dbName': 'db3', \ + 'vgroups': 4, \ + 'stbName': 'stb2', \ + 'ctbNum': 10, \ + 'rowsPerTbl': 5000, \ + 'batchNum': 100, \ + 'startTs': 1640966400000} # 2022-01-01 00:00:00.000 + parameterDict['cfg'] = cfgPath + + prepareEnvThread2 = threading.Thread(target=self.prepareEnv, kwargs=parameterDict2) + prepareEnvThread2.start() + + tdLog.info("create topics from db") + topicName1 = 'topic_db1' + + tdSql.execute("create topic %s as database %s" %(topicName1, parameterDict['dbName'])) + + consumerId = 0 + expectrowcnt = parameterDict["rowsPerTbl"] * parameterDict["ctbNum"] + parameterDict2["rowsPerTbl"] * parameterDict2["ctbNum"] + topicList = topicName1 + ifcheckdata = 0 + ifManualCommit = 0 + keyList = 'group.id:cgrp1,\ + enable.auto.commit:false,\ + auto.commit.interval.ms:6000,\ + auto.offset.reset:earliest' + self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + # consumerId = 1 + # self.insertConsumerInfo(consumerId, expectrowcnt,topicList,keyList,ifcheckdata,ifManualCommit) + + event.wait() + + tdLog.info("start consume processor") + pollDelay = 100 + showMsg = 1 + showRow = 1 + self.startTmqSimProcess(buildPath,cfgPath,pollDelay,parameterDict["dbName"],showMsg, showRow) + + # wait for data ready + prepareEnvThread.join() + prepareEnvThread2.join() + + tdLog.info("4-insert process end, and start to check consume result") + expectRows = 1 + resultList = self.selectConsumeResult(expectRows) + totalConsumeRows = 0 + for i in range(expectRows): + totalConsumeRows += resultList[i] + + if totalConsumeRows != expectrowcnt: + tdLog.info("act consume rows: %d, expect consume rows: %d"%(totalConsumeRows, expectrowcnt)) + tdLog.exit("tmq consume rows error!") + + tdSql.query("drop topic %s"%topicName1) + + tdLog.printNoPrefix("======== test case 3 end ...... ") + + def run(self): + tdSql.prepare() + + buildPath = self.getBuildPath() + if (buildPath == ""): + tdLog.exit("taosd not found!") + else: + tdLog.info("taosd found in %s" % buildPath) + cfgPath = buildPath + "/../sim/psim/cfg" + tdLog.info("cfgPath: %s" % cfgPath) + + self.tmqCase1(cfgPath, buildPath) + self.tmqCase2(cfgPath, buildPath) + self.tmqCase2a(cfgPath, buildPath) + self.tmqCase3(cfgPath, buildPath) + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +event = threading.Event() + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/tests/test_new/udf/udf_create.py b/tests/test_new/udf/udf_create.py new file mode 100644 index 0000000000..9038d99ff9 --- /dev/null +++ b/tests/test_new/udf/udf_create.py @@ -0,0 +1,703 @@ +from distutils.log import error +import taos +import sys +import time +import os + +from util.log import * +from util.sql import * +from util.cases import * +from util.dnodes import * +import subprocess +if (platform.system().lower() == 'windows'): + import win32gui +import threading + +class TDTestCase: + + def init(self, conn, logSql, replicaVar=1): + self.replicaVar = int(replicaVar) + tdLog.debug(f"start to excute {__file__}") + tdSql.init(conn.cursor(), logSql) + + def getBuildPath(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + + for root, dirs, files in os.walk(projPath): + if ("taosd" in files or "taosd.exe" in files): + rootRealPath = os.path.dirname(os.path.realpath(root)) + if ("packaging" not in rootRealPath): + buildPath = root[:len(root) - len("/build/bin")] + break + return buildPath + + def prepare_udf_so(self): + selfPath = os.path.dirname(os.path.realpath(__file__)) + + if ("community" in selfPath): + projPath = selfPath[:selfPath.find("community")] + else: + projPath = selfPath[:selfPath.find("tests")] + print(projPath) + + if platform.system().lower() == 'windows': + self.libudf1 = subprocess.Popen('(for /r %s %%i in ("udf1.d*") do @echo %%i)|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf1_dup = subprocess.Popen('(for /r %s %%i in ("udf1_dup.d*") do @echo %%i)|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf2 = subprocess.Popen('(for /r %s %%i in ("udf2.d*") do @echo %%i)|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf2_dup = subprocess.Popen('(for /r %s %%i in ("udf2_dup.d*") do @echo %%i)|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + if (not tdDnodes.dnodes[0].remoteIP == ""): + tdDnodes.dnodes[0].remote_conn.get(tdDnodes.dnodes[0].config["path"]+'/debug/build/lib/libudf1.so',projPath+"\\debug\\build\\lib\\") + tdDnodes.dnodes[0].remote_conn.get(tdDnodes.dnodes[0].config["path"]+'/debug/build/lib/libudf1_dup.so',projPath+"\\debug\\build\\lib\\") + tdDnodes.dnodes[0].remote_conn.get(tdDnodes.dnodes[0].config["path"]+'/debug/build/lib/libudf2.so',projPath+"\\debug\\build\\lib\\") + tdDnodes.dnodes[0].remote_conn.get(tdDnodes.dnodes[0].config["path"]+'/debug/build/lib/libudf2_dup.so',projPath+"\\debug\\build\\lib\\") + self.libudf1 = self.libudf1.replace('udf1.dll','libudf1.so') + self.libudf1_dup = self.libudf1_dup.replace('udf1_dup.dll','libudf1_dup.so') + self.libudf2 = self.libudf2.replace('udf2.dll','libudf2.so') + self.libudf2_dup = self.libudf2_dup.replace('udf2_dup.dll','libudf2_dup.so') + else: + self.libudf1 = subprocess.Popen('find %s -name "libudf1.so"|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf1_dup = subprocess.Popen('find %s -name "libudf1_dup.so"|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf2 = subprocess.Popen('find %s -name "libudf2.so"|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf2_dup = subprocess.Popen('find %s -name "libudf2_dup.so"|grep lib|head -n1'%projPath , shell=True, stdout=subprocess.PIPE,stderr=subprocess.STDOUT).stdout.read().decode("utf-8") + self.libudf1 = self.libudf1.replace('\r','').replace('\n','') + self.libudf1_dup = self.libudf1_dup.replace('\r','').replace('\n','') + self.libudf2 = self.libudf2.replace('\r','').replace('\n','') + self.libudf2_dup = self.libudf2_dup.replace('\r','').replace('\n','') + + + def prepare_data(self): + + tdSql.execute("drop database if exists db ") + tdSql.execute("create database if not exists db duration 100") + tdSql.execute("use db") + tdSql.execute( + '''create table stb1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + tags (t1 int) + ''' + ) + + tdSql.execute( + ''' + create table t1 + (ts timestamp, c1 int, c2 bigint, c3 smallint, c4 tinyint, c5 float, c6 double, c7 bool, c8 binary(16),c9 nchar(32), c10 timestamp) + ''' + ) + for i in range(4): + tdSql.execute(f'create table ct{i+1} using stb1 tags ( {i+1} )') + + for i in range(9): + tdSql.execute( + f"insert into ct1 values ( now()-{i*10}s, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute( + f"insert into ct4 values ( now()-{i*90}d, {1*i}, {11111*i}, {111*i}, {11*i}, {1.11*i}, {11.11*i}, {i%2}, 'binary{i}', 'nchar{i}', now()+{1*i}a )" + ) + tdSql.execute("insert into ct1 values (now()-45s, 0, 0, 0, 0, 0, 0, 0, 'binary0', 'nchar0', now()+8a )") + tdSql.execute("insert into ct1 values (now()+10s, 9, -99999, -999, -99, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+15s, 9, -99999, -999, -99, -9.99, NULL, 1, 'binary9', 'nchar9', now()+9a )") + tdSql.execute("insert into ct1 values (now()+20s, 9, -99999, -999, NULL, -9.99, -99.99, 1, 'binary9', 'nchar9', now()+9a )") + + tdSql.execute("insert into ct4 values (now()-810d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()-400d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + tdSql.execute("insert into ct4 values (now()+90d, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) ") + + tdSql.execute( + f'''insert into t1 values + ( '2020-04-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2020-10-21 01:01:01.000', 1, 11111, 111, 11, 1.11, 11.11, 1, "binary1", "nchar1", now()+1a ) + ( '2020-12-31 01:01:01.000', 2, 22222, 222, 22, 2.22, 22.22, 0, "binary2", "nchar2", now()+2a ) + ( '2021-01-01 01:01:06.000', 3, 33333, 333, 33, 3.33, 33.33, 0, "binary3", "nchar3", now()+3a ) + ( '2021-05-07 01:01:10.000', 4, 44444, 444, 44, 4.44, 44.44, 1, "binary4", "nchar4", now()+4a ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ( '2021-09-30 01:01:16.000', 5, 55555, 555, 55, 5.55, 55.55, 0, "binary5", "nchar5", now()+5a ) + ( '2022-02-01 01:01:20.000', 6, 66666, 666, 66, 6.66, 66.66, 1, "binary6", "nchar6", now()+6a ) + ( '2022-10-28 01:01:26.000', 7, 00000, 000, 00, 0.00, 00.00, 1, "binary7", "nchar7", "1970-01-01 08:00:00.000" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -888, -88, -8.88, -88.88, 0, "binary8", "nchar8", "1969-01-01 01:00:00.000" ) + ( '2022-12-31 01:01:36.000', 9, -99999999999999999, -999, -99, -9.99, -999999999999999999999.99, 1, "binary9", "nchar9", "1900-01-01 00:00:00.000" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL ) + ''' + ) + + tdSql.execute("create table tb (ts timestamp , num1 int , num2 int, num3 double , num4 binary(30))") + tdSql.execute( + f'''insert into tb values + ( '2020-04-21 01:01:01.000', NULL, 1, 1, "binary1" ) + ( '2020-10-21 01:01:01.000', 1, 1, 1.11, "binary1" ) + ( '2020-12-31 01:01:01.000', 2, 22222, 22, "binary1" ) + ( '2021-01-01 01:01:06.000', 3, 33333, 33, "binary1" ) + ( '2021-05-07 01:01:10.000', 4, 44444, 44, "binary1" ) + ( '2021-07-21 01:01:01.000', NULL, NULL, NULL, "binary1" ) + ( '2021-09-30 01:01:16.000', 5, 55555, 55, "binary1" ) + ( '2022-02-01 01:01:20.000', 6, 66666, 66, "binary1" ) + ( '2022-10-28 01:01:26.000', 0, 00000, 00, "binary1" ) + ( '2022-12-01 01:01:30.000', 8, -88888, -88, "binary1" ) + ( '2022-12-31 01:01:36.000', 9, -9999999, -99, "binary1" ) + ( '2023-02-21 01:01:01.000', NULL, NULL, NULL, "binary1" ) + ''' + ) + + # udf functions with join + ts_start = 1652517451000 + tdSql.execute("create stable st (ts timestamp , c1 int , c2 int ,c3 double ,c4 double ) tags(ind int)") + tdSql.execute("create table sub1 using st tags(1)") + tdSql.execute("create table sub2 using st tags(2)") + + for i in range(10): + ts = ts_start + i *1000 + tdSql.execute(" insert into sub1 values({} , {},{},{},{})".format(ts,i ,i*10,i*100.0,i*1000.0)) + tdSql.execute(" insert into sub2 values({} , {},{},{},{})".format(ts,i ,i*10,i*100.0,i*1000.0)) + + + def create_udf_function(self): + + for i in range(5): + # create scalar functions + tdSql.execute("create function udf1 as '%s' outputtype int;"%self.libudf1) + + # create aggregate functions + + tdSql.execute("create aggregate function udf2 as '%s' outputtype double bufSize 8;"%self.libudf2) + + functions = tdSql.getResult("show functions") + function_nums = len(functions) + if function_nums == 2: + tdLog.info("create two udf functions success ") + + # drop functions + + tdSql.execute("drop function udf1") + tdSql.execute("drop function udf2") + + functions = tdSql.getResult("show functions") + for function in functions: + if "udf1" in function[0] or "udf2" in function[0]: + tdLog.info("drop udf functions failed ") + tdLog.exit("drop udf functions failed") + + tdLog.info("drop two udf functions success ") + + # create scalar functions + tdSql.execute("create function udf1 as '%s' outputtype int;"%self.libudf1) + tdSql.execute("create function udf1_dup as '%s' outputtype int;"%self.libudf1_dup) + + # create aggregate functions + + tdSql.execute("create aggregate function udf2 as '%s' outputtype double bufSize 8;"%self.libudf2) + tdSql.execute("create aggregate function udf2_dup as '%s' outputtype double bufSize 8;"%self.libudf2_dup) + + functions = tdSql.getResult("show functions") + function_nums = len(functions) + if function_nums == 2: + tdLog.info("create two udf functions success ") + + def basic_udf_query(self): + # create tsma of udf + tdSql.error("create tsma tsma_udf on db.tb function(udf1(num1)) interval(10m);") # DB error: Not buildin function (0.001656s) + tdSql.error("create tsma tsma_udf on db.stb1 function(udf1(c1)) interval(10m);") # DB error: Not buildin function (0.001656s) + # scalar functions + + # udf1_dup + tdSql.query("select udf1(num1) ,udf1_dup(num1) from tb") + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,2) + tdSql.checkData(2,0,1) + tdSql.checkData(2,1,2) + + tdSql.execute("use db ") + tdSql.query("select num1 , udf1(num1) ,num2 ,udf1(num2),num3 ,udf1(num3),num4 ,udf1(num4) from tb") + tdSql.checkData(0,0,None) + tdSql.checkData(0,1,None) + tdSql.checkData(0,2,1) + tdSql.checkData(0,3,1) + tdSql.checkData(0,4,1.000000000) + tdSql.checkData(0,5,1) + tdSql.checkData(0,6,"binary1") + tdSql.checkData(0,7,1) + + tdSql.checkData(3,0,3) + tdSql.checkData(3,1,1) + tdSql.checkData(3,2,33333) + tdSql.checkData(3,3,1) + tdSql.checkData(3,4,33.000000000) + tdSql.checkData(3,5,1) + tdSql.checkData(3,6,"binary1") + tdSql.checkData(3,7,1) + + tdSql.checkData(11,0,None) + tdSql.checkData(11,1,None) + tdSql.checkData(11,2,None) + tdSql.checkData(11,3,None) + tdSql.checkData(11,4,None) + tdSql.checkData(11,5,None) + tdSql.checkData(11,6,"binary1") + tdSql.checkData(11,7,1) + + tdSql.query("select c1 , udf1(c1) ,c2 ,udf1(c2), c3 ,udf1(c3), c4 ,udf1(c4) from stb1 order by c1") + tdSql.checkData(0,0,None) + tdSql.checkData(0,1,None) + tdSql.checkData(0,2,None) + tdSql.checkData(0,3,None) + tdSql.checkData(0,4,None) + tdSql.checkData(0,5,None) + tdSql.checkData(0,6,None) + tdSql.checkData(0,7,None) + + tdSql.checkData(20,0,8) + tdSql.checkData(20,1,1) + tdSql.checkData(20,2,88888) + tdSql.checkData(20,3,1) + tdSql.checkData(20,4,888) + tdSql.checkData(20,5,1) + tdSql.checkData(20,6,88) + tdSql.checkData(20,7,1) + + + # aggregate functions + tdSql.query("select udf2(num1) ,udf2_dup(num1) from tb") + val = tdSql.queryResult[0][0] + 100 + tdSql.checkData(0,1,val) + + tdSql.query("select udf2(num1) ,udf2(num2), udf2(num3) from tb") + tdSql.checkData(0,0,15.362291496) + tdSql.checkData(0,1,10000949.553189287) + tdSql.checkData(0,2,168.633425216) + + # Arithmetic compute + tdSql.query("select udf2(num1)+100 ,udf2(num2)-100, udf2(num3)*100 ,udf2(num3)/100 from tb") + tdSql.checkData(0,0,115.362291496) + tdSql.checkData(0,1,10000849.553189287) + tdSql.checkData(0,2,16863.342521576) + tdSql.checkData(0,3,1.686334252) + + tdSql.query("select udf2(c1) ,udf2(c6) from stb1 ") + tdSql.checkData(0,0,25.514701644) + tdSql.checkData(0,1,265.247614504) + + tdSql.query("select udf2(c1)+100 ,udf2(c6)-100 ,udf2(c1)*100 ,udf2(c6)/100 from stb1 ") + tdSql.checkData(0,0,125.514701644) + tdSql.checkData(0,1,165.247614504) + tdSql.checkData(0,2,2551.470164435) + tdSql.checkData(0,3,2.652476145) + + # # bug for crash when query sub table + tdSql.query("select udf2(c1+100) ,udf2(c6-100) ,udf2(c1*100) ,udf2(c6/100) from ct1") + tdSql.checkData(0,0,378.215547010) + tdSql.checkData(0,1,353.808067460) + tdSql.checkData(0,2,2114.237451187) + tdSql.checkData(0,3,2.125468151) + + tdSql.query("select udf2(c1+100) ,udf2(c6-100) ,udf2(c1*100) ,udf2(c6/100) from stb1 ") + tdSql.checkData(0,0,490.358032462) + tdSql.checkData(0,1,400.460106627) + tdSql.checkData(0,2,2551.470164435) + tdSql.checkData(0,3,2.652476145) + + + # regular table with aggregate functions + + tdSql.error("select udf1(num1) , count(num1) from tb;") + tdSql.error("select udf1(num1) , avg(num1) from tb;") + tdSql.error("select udf1(num1) , twa(num1) from tb;") + tdSql.error("select udf1(num1) , irate(num1) from tb;") + tdSql.error("select udf1(num1) , sum(num1) from tb;") + tdSql.error("select udf1(num1) , stddev(num1) from tb;") + tdSql.error("select udf1(num1) , HYPERLOGLOG(num1) from tb;") + # stable + tdSql.error("select udf1(c1) , count(c1) from stb1;") + tdSql.error("select udf1(c1) , avg(c1) from stb1;") + tdSql.error("select udf1(c1) , twa(c1) from stb1;") + tdSql.error("select udf1(c1) , irate(c1) from stb1;") + tdSql.error("select udf1(c1) , sum(c1) from stb1;") + tdSql.error("select udf1(c1) , stddev(c1) from stb1;") + tdSql.error("select udf1(c1) , HYPERLOGLOG(c1) from stb1;") + + # regular table with select functions + + tdSql.query("select udf1(num1) , max(num1) from tb;") + tdSql.checkRows(1) + tdSql.query("select floor(num1) , max(num1) from tb;") + tdSql.checkRows(1) + tdSql.query("select udf1(num1) , min(num1) from tb;") + tdSql.checkRows(1) + tdSql.query("select ceil(num1) , min(num1) from tb;") + tdSql.checkRows(1) + tdSql.query("select udf1(num1) , first(num1) from tb;") + + tdSql.query("select abs(num1) , first(num1) from tb;") + + tdSql.query("select udf1(num1) , last(num1) from tb;") + + tdSql.query("select round(num1) , last(num1) from tb;") + + tdSql.query("select udf1(num1) , top(num1,1) from tb;") + tdSql.checkRows(1) + tdSql.query("select udf1(num1) , bottom(num1,1) from tb;") + tdSql.checkRows(1) + # tdSql.query("select udf1(num1) , last_row(num1) from tb;") + # tdSql.checkRows(1) + + # tdSql.query("select round(num1) , last_row(num1) from tb;") + # tdSql.checkRows(1) + + + # stable + tdSql.query("select udf1(c1) , max(c1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select abs(c1) , max(c1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select udf1(c1) , min(c1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select floor(c1) , min(c1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select udf1(c1) , first(c1) from stb1;") + + tdSql.query("select udf1(c1) , last(c1) from stb1;") + + tdSql.query("select udf1(c1) , top(c1 ,1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select abs(c1) , top(c1 ,1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select udf1(c1) , bottom(c1,1) from stb1;") + tdSql.checkRows(1) + tdSql.query("select ceil(c1) , bottom(c1,1) from stb1;") + tdSql.checkRows(1) + + # tdSql.query("select udf1(c1) , last_row(c1) from stb1;") + # tdSql.checkRows(1) + # tdSql.query("select ceil(c1) , last_row(c1) from stb1;") + # tdSql.checkRows(1) + + # regular table with compute functions + + tdSql.query("select udf1(num1) , abs(num1) from tb;") + tdSql.checkRows(12) + tdSql.query("select floor(num1) , abs(num1) from tb;") + tdSql.checkRows(12) + + # # bug need fix + + #tdSql.query("select udf1(num1) , csum(num1) from tb;") + #tdSql.checkRows(9) + #tdSql.query("select ceil(num1) , csum(num1) from tb;") + #tdSql.checkRows(9) + #tdSql.query("select udf1(c1) , csum(c1) from stb1;") + #tdSql.checkRows(22) + #tdSql.query("select floor(c1) , csum(c1) from stb1;") + #tdSql.checkRows(22) + + # stable with compute functions + tdSql.query("select udf1(c1) , abs(c1) from stb1;") + tdSql.checkRows(25) + tdSql.query("select abs(c1) , ceil(c1) from stb1;") + tdSql.checkRows(25) + + # nest query + tdSql.query("select abs(udf1(c1)) , abs(ceil(c1)) from stb1 order by ts;") + tdSql.checkRows(25) + tdSql.checkData(0,0,None) + tdSql.checkData(0,1,None) + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,8) + + tdSql.query("select abs(udf1(c1)) , abs(ceil(c1)) from ct1 order by ts;") + tdSql.checkRows(13) + tdSql.checkData(0,0,1) + tdSql.checkData(0,1,8) + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,7) + + # bug fix for crash + # order by udf function result + for _ in range(50): + tdSql.query("select udf2(c1) from stb1 group by 1-udf1(c1)") + print(tdSql.queryResult) + + # udf functions with filter + + tdSql.query("select abs(udf1(c1)) , abs(ceil(c1)) from stb1 where c1 is null order by ts;") + tdSql.checkRows(3) + tdSql.checkData(0,0,None) + tdSql.checkData(0,1,None) + + tdSql.query("select c1 ,udf1(c1) , c6 ,udf1(c6) from stb1 where c1 > 8 order by ts") + tdSql.checkRows(3) + tdSql.checkData(0,0,9) + tdSql.checkData(0,1,1) + tdSql.checkData(0,2,-99.990000000) + tdSql.checkData(0,3,1) + + tdSql.query("select sub1.c1, sub2.c2 from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,0) + tdSql.checkData(0,1,0) + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,10) + + tdSql.query("select udf1(sub1.c1), udf1(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,1) + tdSql.checkData(0,1,1) + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,1) + + tdSql.query("select sub1.c1 , udf1(sub1.c1), sub2.c2 ,udf1(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,0) + tdSql.checkData(0,1,1) + tdSql.checkData(0,2,0) + tdSql.checkData(0,3,1) + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,1) + tdSql.checkData(1,2,10) + tdSql.checkData(1,3,1) + + tdSql.query("select udf2(sub1.c1), udf2(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,16.881943016) + tdSql.checkData(0,1,168.819430161) + tdSql.error("select sub1.c1 , udf2(sub1.c1), sub2.c2 ,udf2(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + + # udf functions with group by + tdSql.query("select udf1(c1) from ct1 group by c1") + tdSql.checkRows(10) + tdSql.query("select udf1(c1) from stb1 group by c1") + tdSql.checkRows(11) + tdSql.query("select c1,c2, udf1(c1,c2) from ct1 group by c1,c2") + tdSql.checkRows(10) + tdSql.query("select c1,c2, udf1(c1,c2) from stb1 group by c1,c2") + tdSql.checkRows(11) + + tdSql.query("select udf2(c1) from ct1 group by c1") + tdSql.checkRows(10) + tdSql.query("select udf2(c1) from stb1 group by c1") + tdSql.checkRows(11) + tdSql.query("select c1,c2, udf2(c1,c6) from ct1 group by c1,c2") + tdSql.checkRows(10) + tdSql.query("select c1,c2, udf2(c1,c6) from stb1 group by c1,c2") + tdSql.checkRows(11) + tdSql.query("select udf2(c1) from stb1 group by udf1(c1)") + tdSql.checkRows(2) + tdSql.query("select udf2(c1) from stb1 group by floor(c1)") + tdSql.checkRows(11) + + # udf mix with order by + tdSql.query("select udf2(c1) from stb1 group by floor(c1) order by udf2(c1)") + tdSql.checkRows(11) + + + def multi_cols_udf(self): + tdSql.query("select num1,num2,num3,udf1(num1,num2,num3) from tb") + tdSql.checkData(0,0,None) + tdSql.checkData(0,1,1) + tdSql.checkData(0,2,1.000000000) + tdSql.checkData(0,3,None) + tdSql.checkData(1,0,1) + tdSql.checkData(1,1,1) + tdSql.checkData(1,2,1.110000000) + tdSql.checkData(1,3,88) + + tdSql.query("select c1,c6,udf1(c1,c6) from stb1 order by ts") + tdSql.checkData(1,0,8) + tdSql.checkData(1,1,88.880000000) + tdSql.checkData(1,2,88) + + tdSql.query("select abs(udf1(c1,c6,c1,c6)) , abs(ceil(c1)) from stb1 where c1 is not null order by ts;") + tdSql.checkRows(22) + + tdSql.query("select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,169.661427555) + tdSql.checkData(0,1,169.661427555) + + def try_query_sql(self): + udf1_sqls = [ + "select num1 , udf1(num1) ,num2 ,udf1(num2),num3 ,udf1(num3),num4 ,udf1(num4) from tb" , + "select c1 , udf1(c1) ,c2 ,udf1(c2), c3 ,udf1(c3), c4 ,udf1(c4) from stb1 order by c1" , + "select udf1(num1) , max(num1) from tb;" , + "select udf1(num1) , min(num1) from tb;" , + #"select udf1(num1) , top(num1,1) from tb;" , + #"select udf1(num1) , bottom(num1,1) from tb;" , + "select udf1(c1) , max(c1) from stb1;" , + "select udf1(c1) , min(c1) from stb1;" , + #"select udf1(c1) , top(c1 ,1) from stb1;" , + #"select udf1(c1) , bottom(c1,1) from stb1;" , + "select udf1(num1) , abs(num1) from tb;" , + #"select udf1(num1) , csum(num1) from tb;" , + #"select udf1(c1) , csum(c1) from stb1;" , + "select udf1(c1) , abs(c1) from stb1;" , + "select abs(udf1(c1)) , abs(ceil(c1)) from stb1 order by ts;" , + "select abs(udf1(c1)) , abs(ceil(c1)) from ct1 order by ts;" , + "select abs(udf1(c1)) , abs(ceil(c1)) from stb1 where c1 is null order by ts;" , + "select c1 ,udf1(c1) , c6 ,udf1(c6) from stb1 where c1 > 8 order by ts" , + "select udf1(sub1.c1), udf1(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null" , + "select sub1.c1 , udf1(sub1.c1), sub2.c2 ,udf1(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null" , + "select udf1(c1) from ct1 group by c1" , + "select udf1(c1) from stb1 group by c1" , + "select c1,c2, udf1(c1,c2) from ct1 group by c1,c2" , + "select c1,c2, udf1(c1,c2) from stb1 group by c1,c2" , + "select num1,num2,num3,udf1(num1,num2,num3) from tb" , + "select c1,c6,udf1(c1,c6) from stb1 order by ts" , + "select abs(udf1(c1,c6,c1,c6)) , abs(ceil(c1)) from stb1 where c1 is not null order by ts;" + ] + udf2_sqls = ["select udf2(sub1.c1), udf2(sub2.c2) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null" , + "select udf2(c1) from stb1 group by 1-udf1(c1)" , + "select udf2(num1) ,udf2(num2), udf2(num3) from tb" , + "select udf2(num1)+100 ,udf2(num2)-100, udf2(num3)*100 ,udf2(num3)/100 from tb" , + "select udf2(c1) ,udf2(c6) from stb1 " , + "select udf2(c1)+100 ,udf2(c6)-100 ,udf2(c1)*100 ,udf2(c6)/100 from stb1 " , + "select udf2(c1+100) ,udf2(c6-100) ,udf2(c1*100) ,udf2(c6/100) from ct1" , + "select udf2(c1+100) ,udf2(c6-100) ,udf2(c1*100) ,udf2(c6/100) from stb1 " , + "select udf2(c1) from ct1 group by c1" , + "select udf2(c1) from stb1 group by c1" , + "select c1,c2, udf2(c1,c6) from ct1 group by c1,c2" , + "select c1,c2, udf2(c1,c6) from stb1 group by c1,c2" , + "select udf2(c1) from stb1 group by udf1(c1)" , + "select udf2(c1) from stb1 group by floor(c1)" , + "select udf2(c1) from stb1 group by floor(c1) order by udf2(c1)" , + + "select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null" , + "select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null" , + "select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null" , + "select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null"] + + return udf1_sqls ,udf2_sqls + + + def checkRunTimeError(self): + if (platform.system().lower() == 'windows' and tdDnodes.dnodes[0].remoteIP == ""): + while 1: + time.sleep(1) + hwnd = win32gui.FindWindow(None, "Microsoft Visual C++ Runtime Library") + if hwnd: + os.system("TASKKILL /F /IM udfd.exe") + + def unexpected_create(self): + if (platform.system().lower() == 'windows' and tdDnodes.dnodes[0].remoteIP == ""): + checkErrorThread = threading.Thread(target=self.checkRunTimeError,daemon=True) + checkErrorThread.start() + + tdLog.info(" create function with out bufsize ") + tdSql.query("drop function udf1 ") + tdSql.query("drop function udf2 ") + + # create function without buffer + tdSql.execute("create function udf1 as '%s' outputtype int"%self.libudf1) + tdSql.execute("create aggregate function udf2 as '%s' outputtype double"%self.libudf2) + udf1_sqls ,udf2_sqls = self.try_query_sql() + + for scalar_sql in udf1_sqls: + tdSql.query(scalar_sql) + for aggregate_sql in udf2_sqls: + tdSql.error(aggregate_sql) + + # create function without aggregate + + tdLog.info(" create function with out aggregate ") + tdSql.query("drop function udf1 ") + tdSql.query("drop function udf2 ") + + # create function without buffer + tdSql.execute("create aggregate function udf1 as '%s' outputtype int bufSize 8 "%self.libudf1) + tdSql.execute("create function udf2 as '%s' outputtype double "%self.libudf2) + udf1_sqls ,udf2_sqls = self.try_query_sql() + + for scalar_sql in udf1_sqls: + tdSql.error(scalar_sql) + for aggregate_sql in udf2_sqls: + tdSql.error(aggregate_sql) + + tdSql.execute(" create function db as '%s' outputtype int "%self.libudf1) + tdSql.execute(" create aggregate function test as '%s' outputtype int bufSize 8 "%self.libudf1) + tdSql.error(" select db(c1) from stb1 ") + tdSql.error(" select db(c1,c6), db(c6) from stb1 ") + tdSql.error(" select db(num1,num2), db(num1) from tb ") + tdSql.error(" select test(c1) from stb1 ") + tdSql.error(" select test(c1,c6), test(c6) from stb1 ") + tdSql.error(" select test(num1,num2), test(num1) from tb ") + + + + def loop_kill_udfd(self): + + buildPath = self.getBuildPath() + if (buildPath == ""): + tdLog.exit("taosd not found!") + else: + tdLog.info("taosd found in %s" % buildPath) + + cfgPath = buildPath + "/../sim/dnode1/cfg" + udfdPath = buildPath +'/build/bin/udfd' + + for i in range(3): + + tdLog.info(" loop restart udfd %d_th" % i) + + tdSql.query("select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,169.661427555) + tdSql.checkData(0,1,169.661427555) + # stop udfd cmds + get_processID = "ps -ef | grep -w udfd | grep -v grep| grep -v defunct | awk '{print $2}'" + processID = subprocess.check_output(get_processID, shell=True).decode("utf-8") + stop_udfd = " kill -9 %s" % processID + os.system(stop_udfd) + + time.sleep(2) + + tdSql.query("select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,169.661427555) + tdSql.checkData(0,1,169.661427555) + + # # start udfd cmds + # start_udfd = "nohup " + udfdPath +'-c' +cfgPath +" > /dev/null 2>&1 &" + # tdLog.info("start udfd : %s " % start_udfd) + + def test_function_name(self): + tdLog.info(" create function name is not build_in functions ") + tdSql.execute(" drop function udf1 ") + tdSql.execute(" drop function udf2 ") + tdSql.error("create function max as '%s' outputtype int"%self.libudf1) + tdSql.error("create aggregate function sum as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create function max as '%s' outputtype int"%self.libudf1) + tdSql.error("create aggregate function sum as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function tbname as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function function as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function stable as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function union as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function 123 as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function 123db as '%s' outputtype double bufSize 8"%self.libudf2) + tdSql.error("create aggregate function mnode as '%s' outputtype double bufSize 8"%self.libudf2) + + def restart_taosd_query_udf(self): + + self.create_udf_function() + + for i in range(5): + tdLog.info(" this is %d_th restart taosd " %i) + tdSql.execute("use db ") + tdSql.query("select count(*) from stb1") + tdSql.checkRows(1) + tdSql.query("select udf2(sub1.c1 ,sub1.c2), udf2(sub2.c2 ,sub2.c1) from sub1, sub2 where sub1.ts=sub2.ts and sub1.c1 is not null") + tdSql.checkData(0,0,169.661427555) + tdSql.checkData(0,1,169.661427555) + tdDnodes.stop(1) + tdDnodes.start(1) + time.sleep(2) + + + def run(self): # sourcery skip: extract-duplicate-method, remove-redundant-fstring + + print(" env is ok for all ") + self.prepare_udf_so() + self.prepare_data() + self.create_udf_function() + self.basic_udf_query() + self.unexpected_create() + + + def stop(self): + tdSql.close() + tdLog.success(f"{__file__} successfully executed") + +tdCases.addLinux(__file__, TDTestCase()) +tdCases.addWindows(__file__, TDTestCase()) diff --git a/utils/test/c/CMakeLists.txt b/utils/test/c/CMakeLists.txt index d73f91aad3..1b2716b8e5 100644 --- a/utils/test/c/CMakeLists.txt +++ b/utils/test/c/CMakeLists.txt @@ -8,6 +8,7 @@ add_executable(tmq_td32526 tmq_td32526.c) add_executable(tmq_td32187 tmq_td32187.c) add_executable(tmq_ts5776 tmq_ts5776.c) add_executable(tmq_td32471 tmq_td32471.c) +add_executable(tmq_td33798 tmq_td33798.c) add_executable(tmq_write_raw_test tmq_write_raw_test.c) add_executable(write_raw_block_test write_raw_block_test.c) add_executable(sml_test sml_test.c) @@ -81,6 +82,13 @@ target_link_libraries( PUBLIC common PUBLIC os ) +target_link_libraries( + tmq_td33798 + PUBLIC ${TAOS_LIB} + PUBLIC util + PUBLIC common + PUBLIC os +) target_link_libraries( tmq_td32526 PUBLIC ${TAOS_LIB} diff --git a/utils/test/c/tmq_td33798.c b/utils/test/c/tmq_td33798.c new file mode 100644 index 0000000000..a029b17b61 --- /dev/null +++ b/utils/test/c/tmq_td33798.c @@ -0,0 +1,312 @@ +/* + * Copyright (c) 2019 TAOS Data, Inc. + * + * This program is free software: you can use, redistribute, and/or modify + * it under the terms of the GNU Affero General Public License, version 3 + * or later ("AGPL"), as published by the Free Software Foundation. + * + * This program is distributed in the hope that it will be useful, but WITHOUT + * ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or + * FITNESS FOR A PARTICULAR PURPOSE. + * + * You should have received a copy of the GNU Affero General Public License + * along with this program. If not, see . + */ + +#include +#include +#include +#include +#include +#include "cJSON.h" +#include "taos.h" +#include "tmsg.h" +#include "types.h" + +bool batchMeta = false; +int32_t consumeIndex = 0; +static TAOS* use_db() { + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return NULL; + } + + TAOS_RES* pRes = taos_query(pConn, "use db_taosx"); + if (taos_errno(pRes) != 0) { + printf("error in use db_taosx, reason:%s\n", taos_errstr(pRes)); + return NULL; + } + taos_free_result(pRes); + return pConn; +} + +void checkBatchMeta(TAOS_RES* msg){ + char* result = tmq_get_json_meta(msg); + printf("meta result: %s\n", result); + switch (consumeIndex) { + case 0: + ASSERT(strcmp(result, "{\"tmq_meta_version\":\"1.0\",\"metas\":[{\"type\":\"create\",\"tableType\":\"super\",\"tableName\":\"st1\",\"columns\":[{\"name\":\"ts\",\"type\":9,\"isPrimarykey\":false,\"encode\":\"delta-i\",\"compress\":\"lz4\",\"level\":\"medium\"},{\"name\":\"c1\",\"type\":4,\"isPrimarykey\":false,\"encode\":\"simple8b\",\"compress\":\"lz4\",\"level\":\"medium\"},{\"name\":\"c2\",\"type\":6,\"isPrimarykey\":false,\"encode\":\"delta-d\",\"compress\":\"lz4\",\"level\":\"medium\"},{\"name\":\"c3\",\"type\":8,\"length\":16,\"isPrimarykey\":false,\"encode\":\"disabled\",\"compress\":\"zstd\",\"level\":\"medium\"}],\"tags\":[{\"name\":\"t1\",\"type\":4},{\"name\":\"t3\",\"type\":10,\"length\":8},{\"name\":\"t4\",\"type\":1}]},{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct0\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]},{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct1\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]},{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct11\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]},{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct10\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]},{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct2\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[{\"tableName\":\"ct2\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}]},{\"tableName\":\"ct3\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}]}]}]}") == 0); + break; + default: + ASSERT(0); + break; + } + + tmq_free_json_meta(result); +} + +void checkNonBatchMeta(TAOS_RES* msg){ + char* result = tmq_get_json_meta(msg); + printf("meta result: %s\n", result); + switch (consumeIndex) { + case 0: + ASSERT(strcmp(result, "{\"type\":\"create\",\"tableType\":\"super\",\"tableName\":\"st1\",\"columns\":[{\"name\":\"ts\",\"type\":9,\"isPrimarykey\":false,\"encode\":\"delta-i\",\"compress\":\"lz4\",\"level\":\"medium\"},{\"name\":\"c1\",\"type\":4,\"isPrimarykey\":false,\"encode\":\"simple8b\",\"compress\":\"lz4\",\"level\":\"medium\"},{\"name\":\"c2\",\"type\":6,\"isPrimarykey\":false,\"encode\":\"delta-d\",\"compress\":\"lz4\",\"level\":\"medium\"},{\"name\":\"c3\",\"type\":8,\"length\":16,\"isPrimarykey\":false,\"encode\":\"disabled\",\"compress\":\"zstd\",\"level\":\"medium\"}],\"tags\":[{\"name\":\"t1\",\"type\":4},{\"name\":\"t3\",\"type\":10,\"length\":8},{\"name\":\"t4\",\"type\":1}]}") == 0); + break; + case 1: + ASSERT(strcmp(result, "{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct0\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]}") == 0); + break; + case 2: + ASSERT(strcmp(result, "{\"tmq_meta_version\":\"1.0\",\"metas\":[{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct1\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]}]}") == 0); + break; + case 3: + ASSERT(strcmp(result, "{\"tmq_meta_version\":\"1.0\",\"metas\":[{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct11\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]}]}") == 0); + break; + case 4: + ASSERT(strcmp(result, "{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct10\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[]}") == 0); + break; + case 5: + ASSERT(strcmp(result, "{\"tmq_meta_version\":\"1.0\",\"metas\":[{\"type\":\"create\",\"tableType\":\"child\",\"tableName\":\"ct2\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}],\"createList\":[{\"tableName\":\"ct2\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}]},{\"tableName\":\"ct3\",\"using\":\"st1\",\"tagNum\":3,\"tags\":[{\"name\":\"t1\",\"type\":4,\"value\":1000},{\"name\":\"t3\",\"type\":10,\"value\":\"\\\"ttt\\\"\"},{\"name\":\"t4\",\"type\":1,\"value\":1}]}]}]}") == 0); + break; + default: + ASSERT(0); + break; + } + + tmq_free_json_meta(result); +} + +static void msg_process(TAOS_RES* msg) { + printf("-----------topic-------------: %s\n", tmq_get_topic_name(msg)); + printf("db: %s\n", tmq_get_db_name(msg)); + printf("vg: %d\n", tmq_get_vgroup_id(msg)); + TAOS* pConn = use_db(); + ASSERT (tmq_get_res_type(msg) == TMQ_RES_TABLE_META); + if (batchMeta){ + checkBatchMeta(msg); + } else { + checkNonBatchMeta(msg); + } + taos_close(pConn); +} + +int buildDatabase(TAOS* pConn, TAOS_RES* pRes) { + pRes = taos_query(pConn, + "create stable if not exists st1 (ts timestamp, c1 int, c2 float, c3 binary(16)) tags(t1 int, t3 " + "nchar(8), t4 bool)"); + if (taos_errno(pRes) != 0) { + printf("failed to create super table st1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table if not exists ct0 using st1 tags(1000, \"ttt\", true)"); + if (taos_errno(pRes) != 0) { + printf("failed to create child table tu1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "insert into ct1 using st1 tags(1000, \"ttt\", true) values(1626006833400, 1, 2, 'a')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ct0, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "insert into ct1 values(1626006833600, 3, 4, 'b')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ct1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "insert into ct11 using st1 tags(1000, \"ttt\", true) values(1626006833400, 1, 2, 'a')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ct0, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create table if not exists ct10 using st1 tags(1000, \"ttt\", true)"); + if (taos_errno(pRes) != 0) { + printf("failed to create child table tu1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + taosSsleep(1); + pRes = taos_query(pConn, "insert into ct1 using st1 tags(1000, \"ttt\", true) values(1626006833400, 1, 2, 'a') ct2 using st1 tags(1000, \"ttt\", true) values(1626006833400, 1, 2, 'a') ct3 using st1 tags(1000, \"ttt\", true) values(1626006833400, 1, 2, 'a')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ct0, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + + pRes = taos_query(pConn, "insert into ct1 values(1626006833600, 3, 4, 'b')"); + if (taos_errno(pRes) != 0) { + printf("failed to insert into ct1, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + return 0; +} + +int32_t init_env() { + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return -1; + } + + TAOS_RES* pRes = taos_query(pConn, "drop database if exists db_taosx"); + if (taos_errno(pRes) != 0) { + printf("error in drop db_taosx, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create database if not exists db_taosx vgroups 1 wal_retention_period 3600"); + if (taos_errno(pRes) != 0) { + printf("error in create db_taosx, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "drop topic if exists topic_db"); + if (taos_errno(pRes) != 0) { + printf("error in drop topic, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "drop database if exists abc1"); + if (taos_errno(pRes) != 0) { + printf("error in drop db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "create database if not exists abc1 vgroups 1 wal_retention_period 3600"); + if (taos_errno(pRes) != 0) { + printf("error in create db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + pRes = taos_query(pConn, "use abc1"); + if (taos_errno(pRes) != 0) { + printf("error in use db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + buildDatabase(pConn, pRes); + + taos_close(pConn); + return 0; +} + +int32_t create_topic() { + printf("create topic\n"); + TAOS_RES* pRes; + TAOS* pConn = taos_connect("localhost", "root", "taosdata", NULL, 0); + if (pConn == NULL) { + return -1; + } + + pRes = taos_query(pConn, "create topic topic_db only meta as database abc1"); + if (taos_errno(pRes) != 0) { + printf("failed to create topic topic_db, reason:%s\n", taos_errstr(pRes)); + return -1; + } + taos_free_result(pRes); + + taos_close(pConn); + return 0; +} + +void tmq_commit_cb_print(tmq_t* tmq, int32_t code, void* param) { + printf("commit %d tmq %p param %p\n", code, tmq, param); +} + +tmq_t* build_consumer() { + tmq_conf_t* conf = tmq_conf_new(); + tmq_conf_set(conf, "group.id", batchMeta ? "batch" : "nonbatch"); + tmq_conf_set(conf, "client.id", "my app 1"); + tmq_conf_set(conf, "td.connect.user", "root"); + tmq_conf_set(conf, "td.connect.pass", "taosdata"); + tmq_conf_set(conf, "msg.with.table.name", "true"); + tmq_conf_set(conf, "enable.auto.commit", "true"); + tmq_conf_set(conf, "auto.offset.reset", "earliest"); + tmq_conf_set(conf, "msg.consume.excluded", "1"); + if (batchMeta) { + tmq_conf_set(conf, "msg.enable.batchmeta", "1"); + } + + tmq_conf_set_auto_commit_cb(conf, tmq_commit_cb_print, NULL); + tmq_t* tmq = tmq_consumer_new(conf, NULL, 0); + assert(tmq); + tmq_conf_destroy(conf); + return tmq; +} + +tmq_list_t* build_topic_list() { + tmq_list_t* topic_list = tmq_list_new(); + tmq_list_append(topic_list, "topic_db"); + return topic_list; +} + +void basic_consume_loop(tmq_t* tmq, tmq_list_t* topics) { + int32_t code; + + if ((code = tmq_subscribe(tmq, topics))) { + fprintf(stderr, "%% Failed to start consuming topics: %s\n", tmq_err2str(code)); + printf("subscribe err\n"); + return; + } + while (1) { + TAOS_RES* tmqmessage = tmq_consumer_poll(tmq, 5000); + if (tmqmessage) { + msg_process(tmqmessage); + consumeIndex++; + taos_free_result(tmqmessage); + } else { + break; + } + } + + code = tmq_consumer_close(tmq); + if (code) + fprintf(stderr, "%% Failed to close consumer: %s\n", tmq_err2str(code)); + else + fprintf(stderr, "%% Consumer closed\n"); +} + +int main(int argc, char* argv[]) { + if (init_env() < 0) { + return -1; + } + create_topic(); + + tmq_list_t* topic_list = build_topic_list(); + tmq_t* tmq = build_consumer(); + basic_consume_loop(tmq, topic_list); + + batchMeta = true; + consumeIndex = 0; + tmq = build_consumer(); + basic_consume_loop(tmq, topic_list); + + tmq_list_destroy(topic_list); +}