xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch 'develop' into test/TD-4652

This commit is contained in:
bryanchang0603 2021-06-17 14:27:34 +08:00
parent ee74391f99 4066fb7c50
commit 516f427655
87 changed files with 3869 additions and 2431 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
cmake/install.inc
View File

@ -32,7 +32,7 @@ ELSEIF (TD_WINDOWS)
#INSTALL(TARGETS taos RUNTIME DESTINATION driver)
#INSTALL(TARGETS shell RUNTIME DESTINATION .)
IF (TD_MVN_INSTALLED)
INSTALL(FILES ${LIBRARY_OUTPUT_PATH}/taos-jdbcdriver-2.0.30.jar DESTINATION connector/jdbc)
INSTALL(FILES ${LIBRARY_OUTPUT_PATH}/taos-jdbcdriver-2.0.31.jar DESTINATION connector/jdbc)
ENDIF ()
ELSEIF (TD_DARWIN)
SET(TD_MAKE_INSTALL_SH "${TD_COMMUNITY_DIR}/packaging/tools/make_install.sh")

2
documentation20/cn/00.index/docs.md
View File

@ -81,7 +81,7 @@ TDengine是一个高效的存储、查询、分析时序大数据的平台，专
## [与其他工具的连接](/connections)
* [Grafana](/connections#grafana)：获取并可视化保存在TDengine的数据
* [Matlab](/connections#matlab)：通过配置Matlab的JDBC数据源访问保存在TDengine的数据
* [MATLAB](/connections#matlab)：通过配置MATLAB的JDBC数据源访问保存在TDengine的数据
* [R](/connections#r)：通过配置R的JDBC数据源访问保存在TDengine的数据
* [IDEA Database](https://www.taosdata.com/blog/2020/08/27/1767.html)：通过IDEA 数据库管理工具可视化使用 TDengine

4
documentation20/cn/01.evaluation/docs.md
View File

@ -9,8 +9,8 @@ TDengine的模块之一是时序数据库。但除此之外，为减少研发的
* __10倍以上的性能提升__：定义了创新的数据存储结构，单核每秒能处理至少2万次请求，插入数百万个数据点，读出一千万以上数据点，比现有通用数据库快十倍以上。
* __硬件或云服务成本降至1/5__：由于超强性能，计算资源不到通用大数据方案的1/5；通过列式存储和先进的压缩算法，存储空间不到通用数据库的1/10。
* __全栈时序数据处理引擎__：将数据库、消息队列、缓存、流式计算等功能融为一体，应用无需再集成Kafka/Redis/HBase/Spark/HDFS等软件，大幅降低应用开发和维护的复杂度成本。
* __强大的分析功能__：无论是十年前还是一秒钟前的数据，指定时间范围即可查询。数据可在时间轴上或多个设备上进行聚合。即席查询可通过Shell, Python, R, Matlab随时进行。
* __与第三方工具无缝连接__：不用一行代码，即可与Telegraf, Grafana, EMQ, HiveMQ, Prometheus, Matlab, R等集成。后续将支持OPC, Hadoop, Spark等, BI工具也将无缝连接。
* __强大的分析功能__：无论是十年前还是一秒钟前的数据，指定时间范围即可查询。数据可在时间轴上或多个设备上进行聚合。即席查询可通过Shell, Python, R, MATLAB随时进行。
* __与第三方工具无缝连接__：不用一行代码，即可与Telegraf, Grafana, EMQ, HiveMQ, Prometheus, MATLAB, R等集成。后续将支持OPC, Hadoop, Spark等, BI工具也将无缝连接。
* __零运维成本、零学习成本__：安装集群简单快捷，无需分库分表，实时备份。类似标准SQL，支持RESTful, 支持Python/Java/C/C++/C#/Go/Node.js, 与MySQL相似，零学习成本。
采用TDengine，可将典型的物联网、车联网、工业互联网大数据平台的总拥有成本大幅降低。但需要指出的是，因充分利用了物联网时序数据的特点，它无法用来处理网络爬虫、微博、微信、电商、ERP、CRM等通用型数据。

2
documentation20/cn/08.connector/docs.md
View File

@ -56,7 +56,7 @@ TDengine提供了丰富的应用程序开发接口，其中包括C/C++、Java、
​ *taos.tar.gz*：应用驱动安装包
​ *driver*：TDengine应用驱动driver
​ *connector*: 各种编程语言连接器（go/grafanaplugin/nodejs/python/JDBC）
​ *examples*: 各种编程语言的示例程序(c/C#/go/JDBC/matlab/python/R)
​ *examples*: 各种编程语言的示例程序(c/C#/go/JDBC/MATLAB/python/R)
运行install_client.sh进行安装

14
documentation20/cn/09.connections/docs.md
View File

@ -75,17 +75,17 @@ sudo cp -rf /usr/local/taos/connector/grafanaplugin /var/lib/grafana/plugins/tde
![img](page://images/connections/import_dashboard2.jpg)
## <a class="anchor" id="matlab"></a>Matlab
## <a class="anchor" id="matlab"></a>MATLAB
MatLab可以通过安装包内提供的JDBC Driver直接连接到TDengine获取数据到本地工作空间。
MATLAB可以通过安装包内提供的JDBC Driver直接连接到TDengine获取数据到本地工作空间。
### MatLab的JDBC接口适配
### MATLAB的JDBC接口适配
MatLab的适配有下面几个步骤，下面以Windows10上适配MatLab2017a为例：
MATLAB的适配有下面几个步骤，下面以Windows10上适配MATLAB2017a为例：
- 将TDengine安装包内的驱动程序JDBCDriver-1.0.0-dist.jar拷贝到${matlab_root}\MATLAB\R2017a\java\jar\toolbox
- 将TDengine安装包内的taos.lib文件拷贝至${matlab_ root _dir}\MATLAB\R2017a\lib\win64
- 将新添加的驱动jar包加入MatLab的classpath。在${matlab_ root _dir}\MATLAB\R2017a\toolbox\local\classpath.txt文件中添加下面一行
- 将新添加的驱动jar包加入MATLAB的classpath。在${matlab_ root _dir}\MATLAB\R2017a\toolbox\local\classpath.txt文件中添加下面一行
​
```
$matlabroot/java/jar/toolbox/JDBCDriver-1.0.0-dist.jar
@ -96,9 +96,9 @@ $matlabroot/java/jar/toolbox/JDBCDriver-1.0.0-dist.jar
C:\Windows\System32
```
### 在MatLab中连接TDengine获取数据
### 在MATLAB中连接TDengine获取数据
在成功进行了上述配置后，打开MatLab。
在成功进行了上述配置后，打开MATLAB。
- 创建一个连接：

31
documentation20/cn/11.administrator/docs.md
View File

@ -116,20 +116,22 @@ taosd -C
**注意：**对于端口，TDengine会使用从serverPort起13个连续的TCP和UDP端口号，请务必在防火墙打开。因此如果是缺省配置，需要打开从6030到6042共13个端口，而且必须TCP和UDP都打开。（详细的端口情况请参见 [TDengine 2.0 端口说明](https://www.taosdata.com/cn/documentation/faq#port)）
不同应用场景的数据往往具有不同的数据特征，比如保留天数、副本数、采集频次、记录大小、采集点的数量、压缩等都可完全不同。为获得在存储上的最高效率，TDengine提供如下存储相关的系统配置参数：
不同应用场景的数据往往具有不同的数据特征，比如保留天数、副本数、采集频次、记录大小、采集点的数量、压缩等都可完全不同。为获得在存储上的最高效率，TDengine提供如下存储相关的系统配置参数（既可以作为 create database 指令的参数，也可以写在 taos.cfg 配置文件中用来设定创建新数据库时所采用的默认值）：
- days：一个数据文件存储数据的时间跨度，单位为天，默认值：10。
- keep：数据库中数据保留的天数，单位为天，默认值：3650。（可通过 alter database 修改）
- minRows：文件块中记录的最小条数，单位为条，默认值：100。
- maxRows：文件块中记录的最大条数，单位为条，默认值：4096。
- comp：文件压缩标志位，0：关闭；1：一阶段压缩；2：两阶段压缩。默认值：2。（可通过 alter database 修改）
- walLevel：WAL级别。1：写wal，但不执行fsync；2：写wal, 而且执行fsync。默认值：1。
- fsync：当wal设置为2时，执行fsync的周期。设置为0，表示每次写入，立即执行fsync。单位为毫秒，默认值：3000。
- cache：内存块的大小，单位为兆字节（MB），默认值：16。
- days：一个数据文件存储数据的时间跨度。单位为天，默认值：10。
- keep：数据库中数据保留的天数。单位为天，默认值：3650。（可通过 alter database 修改）
- minRows：文件块中记录的最小条数。单位为条，默认值：100。
- maxRows：文件块中记录的最大条数。单位为条，默认值：4096。
- comp：文件压缩标志位。0：关闭；1：一阶段压缩；2：两阶段压缩。默认值：2。（可通过 alter database 修改）
- wal：WAL级别。1：写wal，但不执行fsync；2：写wal, 而且执行fsync。默认值：1。（在 taos.cfg 中参数名需要写作 walLevel）（可通过 alter database 修改）
- fsync：当wal设置为2时，执行fsync的周期。设置为0，表示每次写入，立即执行fsync。单位为毫秒，默认值：3000。（可通过 alter database 修改）
- cache：内存块的大小。单位为兆字节（MB），默认值：16。
- blocks：每个VNODE（TSDB）中有多少cache大小的内存块。因此一个VNODE的用的内存大小粗略为（cache * blocks）。单位为块，默认值：4。（可通过 alter database 修改）
- replica：副本个数，取值范围：1-3。单位为个，默认值：1。（可通过 alter database 修改）
- precision：时间戳精度标识，ms表示毫秒，us表示微秒。默认值：ms。
- cacheLast：是否在内存中缓存子表的最近数据，0：关闭；1：缓存子表最近一行数据；2：缓存子表每一列的最近的非NULL值，3：同时打开缓存最近行和列功能，默认值：0。（可通过 alter database 修改）（从 2.0.11 版本开始支持此参数）
- replica：副本个数。取值范围：1-3，单位为个，默认值：1。（可通过 alter database 修改）
- quorum：多副本环境下指令执行的确认数要求。取值范围：1、2，单位为个，默认值：1。（可通过 alter database 修改）
- precision：时间戳精度标识。ms表示毫秒，us表示微秒，默认值：ms。（2.1.2.0 版本之前、2.0.20.7 版本之前在 taos.cfg 文件中不支持此参数。）
- cacheLast：是否在内存中缓存子表的最近数据。0：关闭；1：缓存子表最近一行数据；2：缓存子表每一列的最近的非NULL值；3：同时打开缓存最近行和列功能。默认值：0。（可通过 alter database 修改）（从 2.1.2.0 版本开始此参数支持 0～3 的取值范围，在此之前取值只能是 [0, 1]；而 2.0.11.0 之前的版本在 SQL 指令中不支持此参数。）（2.1.2.0 版本之前、2.0.20.7 版本之前在 taos.cfg 文件中不支持此参数。）
- update：是否允许更新。0：不允许；1：允许。默认值：0。（可通过 alter database 修改）
对于一个应用场景，可能有多种数据特征的数据并存，最佳的设计是将具有相同数据特征的表放在一个库里，这样一个应用有多个库，而每个库可以配置不同的存储参数，从而保证系统有最优的性能。TDengine允许应用在创建库时指定上述存储参数，如果指定，该参数就将覆盖对应的系统配置参数。举例，有下述SQL：
@ -142,7 +144,6 @@ taosd -C
TDengine集群中加入一个新的dnode时，涉及集群相关的一些参数必须与已有集群的配置相同，否则不能成功加入到集群中。会进行校验的参数如下：
- numOfMnodes：系统中管理节点个数。默认值：3。
- balance：是否启动负载均衡。0：否，1：是。默认值：1。
- mnodeEqualVnodeNum: 一个mnode等同于vnode消耗的个数。默认值：4。
- offlineThreshold: dnode离线阈值，超过该时间将导致该dnode从集群中删除。单位为秒，默认值：86400*10（即10天）。
- statusInterval: dnode向mnode报告状态时长。单位为秒，默认值：1。
@ -150,6 +151,10 @@ TDengine集群中加入一个新的dnode时，涉及集群相关的一些参数
- maxVgroupsPerDb: 每个数据库中能够使用的最大vgroup个数。
- arbitrator: 系统中裁决器的end point，缺省为空。
- timezone、locale、charset 的配置见客户端配置。（2.0.20.0 及以上的版本里，集群中加入新节点已不要求 locale 和 charset 参数取值一致）
- balance：是否启用负载均衡。0：否，1：是。默认值：1。
- flowctrl：是否启用非阻塞流控。0：否，1：是。默认值：1。
- slaveQuery：是否启用 slave vnode 参与查询。0：否，1：是。默认值：1。
- adjustMaster：是否启用 vnode master 负载均衡。0：否，1：是。默认值：1。
为方便调试，可通过SQL语句临时调整每个dnode的日志配置，系统重启后会失效：

31
documentation20/cn/12.taos-sql/docs.md
View File

@ -126,9 +126,25 @@ TDengine 缺省的时间戳是毫秒精度，但通过在 CREATE DATABASE 时传
```mysql
ALTER DATABASE db_name CACHELAST 0;
```
CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11.0 版本开始支持。从 2.1.1.0 版本开始，修改此参数后无需重启服务器即可生效。）
CACHELAST 参数控制是否在内存中缓存子表的最近数据。缺省值为 0，取值范围 [0, 1, 2, 3]。其中 0 表示不缓存，1 表示缓存子表最近一行数据，2 表示缓存子表每一列的最近的非 NULL 值，3 表示同时打开缓存最近行和列功能。（从 2.0.11.0 版本开始支持参数值 [0, 1]，从 2.1.2.0 版本开始支持参数值 [0, 1, 2, 3]。）
说明：缓存最近行，将显著改善 LAST_ROW 函数的性能表现；缓存每列的最近非 NULL 值，将显著改善无特殊影响（WHERE、ORDER BY、GROUP BY、INTERVAL）下的 LAST 函数的性能表现。
**Tips**: 以上所有参数修改后都可以用show databases来确认是否修改成功。
```mysql
ALTER DATABASE db_name WAL 1;
```
WAL 参数控制 WAL 日志的落盘方式。缺省值为 1，取值范围为 [1, 2]。1 表示写 WAL，但不执行 fsync；2 表示写 WAL，而且执行 fsync。
```mysql
ALTER DATABASE db_name FSYNC 3000;
```
FSYNC 参数控制执行 fsync 操作的周期。缺省值为 3000，单位是毫秒，取值范围为 [0, 180000]。如果设置为 0，表示每次写入，立即执行 fsync。该设置项主要用于调节 WAL 参数设为 2 时的系统行为。
```mysql
ALTER DATABASE db_name UPDATE 0;
```
UPDATE 参数控制是否允许更新数据。缺省值为 0，取值范围为 [0, 1]。0 表示会直接丢弃后写入的相同时间戳的数据；1 表示会使用后写入的数据覆盖已有的相同时间戳的数据。
**Tips**: 以上所有参数修改后都可以用show databases来确认是否修改成功。另外，从 2.1.1.0 版本开始，修改这些参数后无需重启服务器即可生效。
- **显示系统所有数据库**
@ -681,9 +697,10 @@ Query OK, 1 row(s) in set (0.001091s)
| % | match with any char sequences | **`binary`** **`nchar`** |
| _ | match with a single char | **`binary`** **`nchar`** |
1. 同时进行多个字段的范围过滤，需要使用关键词 AND 来连接不同的查询条件，暂不支持 OR 连接的不同列之间的查询过滤条件。
2. 针对单一字段的过滤，如果是时间过滤条件，则一条语句中只支持设定一个；但针对其他的（普通）列或标签列，则可以使用 `OR` 关键字进行组合条件的查询过滤。例如：((value > 20 AND value < 30) OR (value < 12)) 。
3. 从 2.0.17 版本开始，条件过滤开始支持 BETWEEN AND 语法，例如 `WHERE col2 BETWEEN 1.5 AND 3.25` 表示查询条件为“1.5 ≤ col2 ≤ 3.25”。
1. <> 算子也可以写为 != ，请注意，这个算子不能用于数据表第一列的 timestamp 字段。
2. 同时进行多个字段的范围过滤，需要使用关键词 AND 来连接不同的查询条件，暂不支持 OR 连接的不同列之间的查询过滤条件。
3. 针对单一字段的过滤，如果是时间过滤条件，则一条语句中只支持设定一个；但针对其他的（普通）列或标签列，则可以使用 `OR` 关键字进行组合条件的查询过滤。例如：((value > 20 AND value < 30) OR (value < 12)) 。
4. 从 2.0.17 版本开始，条件过滤开始支持 BETWEEN AND 语法，例如 `WHERE col2 BETWEEN 1.5 AND 3.25` 表示查询条件为“1.5 ≤ col2 ≤ 3.25”。
<!--
<a class="anchor" id="having"></a>
@ -1178,9 +1195,9 @@ TDengine支持针对数据的聚合查询。提供支持的聚合和选择函数
应用字段：不能应用在timestamp、binary、nchar、bool类型字段。
适用于：**表**。
适用于：**表、（超级表）**。
说明：输出结果行数是范围内总行数减一，第一行没有结果输出。
说明：输出结果行数是范围内总行数减一，第一行没有结果输出。从 2.1.3.0 版本开始，DIFF 函数可以在由 GROUP BY 划分出单独时间线的情况下用于超级表（也即 GROUP BY tbname）。
示例：
```mysql

4
documentation20/en/00.index/docs.md
View File

@ -29,7 +29,7 @@ TDengine is a highly efficient platform to store, query, and analyze time-series
## [Data Modeling](/model)
- [Create a Library](/model#create-db): create a library for all data collection points with similar features
- [Create a Database](/model#create-db): create a database for all data collection points with similar features
- [Create a Super Table(STable)](/model#create-stable): create a STable for all data collection points with the same type
- [Create a Table](/model#create-table): use STable as the template, to create a table for each data collecting point
@ -82,7 +82,7 @@ TDengine is a highly efficient platform to store, query, and analyze time-series
## [Connections with Other Tools](/connections)
- [Grafana](/connections#grafana): query the data saved in TDengine and provide visualization
- [Matlab](/connections#matlab): access data stored in TDengine server via JDBC configured within Matlab
- [MATLAB](/connections#matlab): access data stored in TDengine server via JDBC configured within MATLAB
- [R](/connections#r): access data stored in TDengine server via JDBC configured within R
- [IDEA Database](https://www.taosdata.com/blog/2020/08/27/1767.html): use TDengine visually through IDEA Database Management Tool

4
documentation20/en/01.evaluation/docs.md
View File

@ -9,8 +9,8 @@ One of the modules of TDengine is the time-series database. However, in addition
- **Performance improvement over 10 times**: An innovative data storage structure is defined, with each single core can process at least 20,000 requests per second, insert millions of data points, and read more than 10 million data points, which is more than 10 times faster than other existing general database.
- **Reduce the cost of hardware or cloud services to 1/5**: Due to its ultra-performance, TDengine’s computing resources consumption is less than 1/5 of other common Big Data solutions; through columnar storage and advanced compression algorithms, the storage consumption is less than 1/10 of other general databases.
- **Full-stack time-series data processing engine**: Integrate database, message queue, cache, stream computing, and other functions, and the applications do not need to integrate with software such as Kafka/Redis/HBase/Spark/HDFS, thus greatly reducing the complexity cost of application development and maintenance.
- **Powerful analysis functions**: Data from ten years ago or one second ago, can all be queried based on a specified time range. Data can be aggregated on a timeline or multiple devices. Ad-hoc queries can be made at any time through Shell, Python, R, and Matlab.
- **Seamless connection with third-party tools**: Integration with Telegraf, Grafana, EMQ, HiveMQ, Prometheus, Matlab, R, etc. without even one single line of code. OPC, Hadoop, Spark, etc. will be supported in the future, and more BI tools will be seamlessly connected to.
- **Powerful analysis functions**: Data from ten years ago or one second ago, can all be queried based on a specified time range. Data can be aggregated on a timeline or multiple devices. Ad-hoc queries can be made at any time through Shell, Python, R, and MATLAB.
- **Seamless connection with third-party tools**: Integration with Telegraf, Grafana, EMQ, HiveMQ, Prometheus, MATLAB, R, etc. without even one single line of code. OPC, Hadoop, Spark, etc. will be supported in the future, and more BI tools will be seamlessly connected to.
- **Zero operation cost & zero learning cost**: Installing clusters is simple and quick, with real-time backup built-in, and no need to split libraries or tables. Similar to standard SQL, TDengine can support RESTful, Python/Java/C/C + +/C#/Go/Node.js, and similar to MySQL with zero learning cost.
With TDengine, the total cost of ownership of typical IoT, Internet of Vehicles, and Industrial Internet Big Data platforms can be greatly reduced. However, it should be pointed out that due to making full use of the characteristics of IoT time-series data, TDengine cannot be used to process general data from web crawlers, microblogs, WeChat, e-commerce, ERP, CRM, and other sources.

2
documentation20/en/08.connector/docs.md
View File

@ -58,7 +58,7 @@ After extracting the package, you will see the following files (directories) in
*connector*: Connectors for various programming languages (go/grafanaplugin/nodejs/python/JDBC)
*Examples*: Sample programs for various programming languages (C/C #/go/JDBC/matlab/python/R)
*Examples*: Sample programs for various programming languages (C/C #/go/JDBC/MATLAB/python/R)
Run install_client.sh to install.

14
documentation20/en/09.connections/docs.md
View File

@ -74,17 +74,17 @@ You can see as follows after Dashboard imported.
![img](page://images/connections/import_dashboard2.jpg)
## <a class="anchor" id="matlab"></a> Matlab
## <a class="anchor" id="matlab"></a> MATLAB
MatLab can access data to the local workspace by connecting directly to the TDengine via the JDBC Driver provided in the installation package.
MATLAB can access data to the local workspace by connecting directly to the TDengine via the JDBC Driver provided in the installation package.
### JDBC Interface Adaptation of MatLab
### JDBC Interface Adaptation of MATLAB
Several steps are required to adapt Matlab to TDengine. Taking adapting Matlab2017a on Windows10 as an example:
Several steps are required to adapt MATLAB to TDengine. Taking adapting MATLAB2017a on Windows10 as an example:
- Copy the file JDBCDriver-1.0.0-dist.ja*r* in TDengine package to the directory ${matlab_root}\MATLAB\R2017a\java\jar\toolbox
- Copy the file taos.lib in TDengine package to ${matlab root dir}\MATLAB\R2017a\lib\win64
- Add the .jar package just copied to the Matlab classpath. Append the line below as the end of the file of ${matlab root dir}\MATLAB\R2017a\toolbox\local\classpath.txt
- Add the .jar package just copied to the MATLAB classpath. Append the line below as the end of the file of ${matlab root dir}\MATLAB\R2017a\toolbox\local\classpath.txt
- ```
$matlabroot/java/jar/toolbox/JDBCDriver-1.0.0-dist.jar
```
@ -94,9 +94,9 @@ Several steps are required to adapt Matlab to TDengine. Taking adapting Matlab20
C:\Windows\System32
```
- ### Connect to TDengine in MatLab to get data
- ### Connect to TDengine in MATLAB to get data
After the above configured successfully, open MatLab.
After the above configured successfully, open MATLAB.
- Create a connection:

12
src/balance/src/bnMain.c
View File

@ -220,10 +220,6 @@ int32_t bnAllocVnodes(SVgObj *pVgroup) {
}
static bool bnCheckVgroupReady(SVgObj *pVgroup, SVnodeGid *pRmVnode) {
if (pVgroup->lbTime + 5 * tsStatusInterval > tsAccessSquence) {
return false;
}
int32_t rmVnodeVer = 0;
for (int32_t i = 0; i < pVgroup->numOfVnodes; ++i) {
SVnodeGid *pVnode = pVgroup->vnodeGid + i;
@ -405,7 +401,7 @@ void bnReset() {
if (pDnode == NULL) break;
// while master change, should reset dnode to offline
mInfo("dnode:%d set access:%d to 0", pDnode->dnodeId, pDnode->lastAccess);
mInfo("dnode:%d set access:%" PRId64 " to 0", pDnode->dnodeId, pDnode->lastAccess);
pDnode->lastAccess = 0;
if (pDnode->status != TAOS_DN_STATUS_DROPPING) {
pDnode->status = TAOS_DN_STATUS_OFFLINE;
@ -499,7 +495,7 @@ static bool bnMontiorDropping() {
if (dnodeIsMasterEp(pDnode->dnodeEp)) continue;
if (mnodeGetDnodesNum() <= 1) continue;
mLInfo("dnode:%d, set to removing state for it offline:%d seconds", pDnode->dnodeId,
mLInfo("dnode:%d, set to removing state for it offline:%" PRId64 " seconds", pDnode->dnodeId,
tsAccessSquence - pDnode->lastAccess);
pDnode->status = TAOS_DN_STATUS_DROPPING;
@ -574,8 +570,8 @@ void bnCheckStatus() {
if (pDnode->status != TAOS_DN_STATUS_DROPPING && pDnode->status != TAOS_DN_STATUS_OFFLINE) {
pDnode->status = TAOS_DN_STATUS_OFFLINE;
pDnode->offlineReason = TAOS_DN_OFF_STATUS_MSG_TIMEOUT;
mInfo("dnode:%d, set to offline state, access seq:%d last seq:%d laststat:%d", pDnode->dnodeId, tsAccessSquence,
pDnode->lastAccess, pDnode->status);
mInfo("dnode:%d, set to offline state, access seq:%" PRId64 " last seq:%" PRId64 " laststat:%d", pDnode->dnodeId,
tsAccessSquence, pDnode->lastAccess, pDnode->status);
bnSetVgroupOffline(pDnode);
bnStartTimer(3000);
}

9
src/balance/src/bnThread.c
View File

@ -102,12 +102,12 @@ static void bnProcessTimer(void *handle, void *tmrId) {
if (tsBnThread.stop) return;
tsBnThread.timer = NULL;
tsAccessSquence++;
bnStartTimer(-1);
bnCheckStatus();
if (handle == NULL) {
++tsAccessSquence;
if (tsAccessSquence % tsBalanceInterval == 0) {
mDebug("balance function is scheduled by timer");
bnPostSignal();
@ -122,8 +122,7 @@ static void bnProcessTimer(void *handle, void *tmrId) {
void bnStartTimer(int32_t mseconds) {
if (tsBnThread.stop) return;
bool updateSoon = (mseconds != -1);
if (updateSoon) {
if (mseconds != -1) {
mTrace("balance function will be called after %d ms", mseconds);
taosTmrReset(bnProcessTimer, mseconds, (void *)(int64_t)mseconds, tsMnodeTmr, &tsBnThread.timer);
} else {
@ -132,5 +131,5 @@ void bnStartTimer(int32_t mseconds) {
}
void bnNotify() {
bnStartTimer(500);
bnStartTimer(500);
}

1
src/client/inc/tscUtil.h
View File

@ -123,7 +123,6 @@ int32_t tscGetDataBlockFromList(SHashObj* pHashList, int64_t id, int32_t size, i
*/
bool tscIsPointInterpQuery(SQueryInfo* pQueryInfo);
bool tscIsTWAQuery(SQueryInfo* pQueryInfo);
bool tscIsDiffQuery(SQueryInfo* pQueryInfo);
bool tscIsSessionWindowQuery(SQueryInfo* pQueryInfo);
bool tscIsSecondStageQuery(SQueryInfo* pQueryInfo);
bool tsIsArithmeticQueryOnAggResult(SQueryInfo* pQueryInfo);

17
src/client/src/tscPrepare.c
View File

@ -951,7 +951,10 @@ static int insertStmtBindParamBatch(STscStmt* stmt, TAOS_MULTI_BIND* bind, int c
}
}
assert(colIdx == -1 || (colIdx >= 0 && colIdx < pBlock->numOfParams));
if (!(colIdx == -1 || (colIdx >= 0 && colIdx < pBlock->numOfParams))) {
tscError("0x%"PRIx64" invalid colIdx:%d", pStmt->pSql->self, colIdx);
return invalidOperationMsg(tscGetErrorMsgPayload(&stmt->pSql->cmd), "invalid param colIdx");
}
uint32_t totalDataSize = sizeof(SSubmitBlk) + (pCmd->batchSize + rowNum) * pBlock->rowSize;
if (totalDataSize > pBlock->nAllocSize) {
@ -1462,6 +1465,11 @@ int taos_stmt_prepare(TAOS_STMT* stmt, const char* sql, unsigned long length) {
STMT_RET(TSDB_CODE_TSC_DISCONNECTED);
}
if (sql == NULL) {
tscError("sql is NULL");
STMT_RET(invalidOperationMsg(tscGetErrorMsgPayload(&pStmt->pSql->cmd), "sql is NULL"));
}
if (pStmt->last != STMT_INIT) {
tscError("prepare status error, last:%d", pStmt->last);
STMT_RET(invalidOperationMsg(tscGetErrorMsgPayload(&pStmt->pSql->cmd), "prepare status error"));
@ -1528,13 +1536,14 @@ int taos_stmt_prepare(TAOS_STMT* stmt, const char* sql, unsigned long length) {
int taos_stmt_set_tbname_tags(TAOS_STMT* stmt, const char* name, TAOS_BIND* tags) {
STscStmt* pStmt = (STscStmt*)stmt;
SSqlObj* pSql = pStmt->pSql;
SSqlCmd* pCmd = &pSql->cmd;
if (stmt == NULL || pStmt->pSql == NULL || pStmt->taos == NULL) {
STMT_RET(TSDB_CODE_TSC_DISCONNECTED);
}
SSqlObj* pSql = pStmt->pSql;
SSqlCmd* pCmd = &pSql->cmd;
if (name == NULL) {
tscError("0x%"PRIx64" name is NULL", pSql->self);
STMT_RET(invalidOperationMsg(tscGetErrorMsgPayload(&pStmt->pSql->cmd), "name is NULL"));
@ -1735,7 +1744,7 @@ int taos_stmt_bind_single_param_batch(TAOS_STMT* stmt, TAOS_MULTI_BIND* bind, in
STMT_RET(TSDB_CODE_TSC_DISCONNECTED);
}
if (bind == NULL || bind->num <= 0 || bind->num > INT16_MAX) {
if (bind == NULL || bind->num <= 0 || bind->num > INT16_MAX || colIdx < 0) {
tscError("0x%"PRIx64" invalid parameter", pStmt->pSql->self);
STMT_RET(invalidOperationMsg(tscGetErrorMsgPayload(&pStmt->pSql->cmd), "invalid bind param"));
}

67
src/client/src/tscSQLParser.c
View File

@ -111,6 +111,7 @@ static int32_t validateDNodeConfig(SMiscInfo* pOptions);
static int32_t validateLocalConfig(SMiscInfo* pOptions);
static int32_t validateColumnName(char* name);
static int32_t setKillInfo(SSqlObj* pSql, struct SSqlInfo* pInfo, int32_t killType);
static int32_t setCompactVnodeInfo(SSqlObj* pSql, struct SSqlInfo* pInfo);
static bool validateOneTags(SSqlCmd* pCmd, TAOS_FIELD* pTagField);
static bool hasTimestampForPointInterpQuery(SQueryInfo* pQueryInfo);
@ -150,12 +151,9 @@ int16_t getNewResColId(SSqlCmd* pCmd) {
// formate "type | size | value"
bool serializeExprListToVariant(SArray* pList, tVariant **dst, int16_t colType) {
bool ret = false;
if (!pList || pList->size <= 0) {
if (!pList || pList->size <= 0 || colType < 0) {
return ret;
}
if (colType == TSDB_DATA_TYPE_DOUBLE || colType == TSDB_DATA_TYPE_FLOAT) {
return ret;
}
tSqlExprItem* item = (tSqlExprItem *)taosArrayGet(pList, 0);
int32_t firstTokenType = item->pNode->token.type;
@ -509,7 +507,7 @@ int32_t tscValidateSqlInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
break;
}
case TSDB_SQL_CREATE_DNODE: {
const char* msg = "invalid host name (ip address)";
@ -805,7 +803,13 @@ int32_t tscValidateSqlInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
}
break;
}
case TSDB_SQL_COMPACT_VNODE:{
const char* msg = "invalid compact";
if (setCompactVnodeInfo(pSql, pInfo) != TSDB_CODE_SUCCESS) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg);
}
break;
}
default:
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), "not support sql expression");
}
@ -1791,7 +1795,6 @@ int32_t validateSelectNodeList(SSqlCmd* pCmd, SQueryInfo* pQueryInfo, SArray* pS
assert(pSelNodeList != NULL && pCmd != NULL);
const char* msg1 = "too many items in selection clause";
const char* msg2 = "functions or others can not be mixed up";
const char* msg3 = "not support query expression";
const char* msg4 = "only support distinct one tag";
@ -2720,6 +2723,8 @@ void getColumnName(tSqlExprItem* pItem, char* resultFieldName, char* rawName, in
strncpy(rawName, pItem->pNode->token.z, len);
if (pItem->aliasName != NULL) {
int32_t aliasNameLen = (int32_t) strlen(pItem->aliasName);
len = (aliasNameLen < nameLength)? aliasNameLen:nameLength;
strncpy(resultFieldName, pItem->aliasName, len);
} else {
strncpy(resultFieldName, rawName, len);
@ -2966,7 +2971,12 @@ int32_t setKillInfo(SSqlObj* pSql, struct SSqlInfo* pInfo, int32_t killType) {
return TSDB_CODE_SUCCESS;
}
static int32_t setCompactVnodeInfo(SSqlObj* pSql, struct SSqlInfo* pInfo) {
SSqlCmd* pCmd = &pSql->cmd;
pCmd->command = pInfo->type;
return TSDB_CODE_SUCCESS;
}
bool validateIpAddress(const char* ip, size_t size) {
char tmp[128] = {0}; // buffer to build null-terminated string
assert(size < 128);
@ -3066,7 +3076,7 @@ bool hasUnsupportFunctionsForSTableQuery(SSqlCmd* pCmd, SQueryInfo* pQueryInfo)
}
}
if (tscIsTWAQuery(pQueryInfo) || tscIsDiffQuery(pQueryInfo)) {
if (tscIsTWAQuery(pQueryInfo) || tscIsDiffDerivQuery(pQueryInfo)) {
if (pQueryInfo->groupbyExpr.numOfGroupCols == 0) {
invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg1);
return true;
@ -4562,11 +4572,7 @@ static int32_t validateTagCondExpr(SSqlCmd* pCmd, tExprNode *p) {
free(tmp);
} else {
double tmp;
if (p->_node.optr == TSDB_RELATION_IN) {
retVal = validateParamOfRelationIn(vVariant, schemaType);
} else {
retVal = tVariantDump(vVariant, (char*)&tmp, schemaType, false);
}
retVal = tVariantDump(vVariant, (char*)&tmp, schemaType, false);
}
if (retVal != TSDB_CODE_SUCCESS) {
@ -4888,7 +4894,10 @@ int32_t getTimeRange(STimeWindow* win, tSqlExpr* pRight, int32_t optr, int16_t t
win->skey = val;
} else if (optr == TK_EQ) {
win->ekey = win->skey = val;
} else if (optr == TK_NE) {
return TSDB_CODE_TSC_INVALID_OPERATION;
}
return TSDB_CODE_SUCCESS;
}
@ -7728,6 +7737,7 @@ int32_t validateSqlNode(SSqlObj* pSql, SSqlNode* pSqlNode, SQueryInfo* pQueryInf
const char* msg4 = "interval query not supported, since the result of sub query not include valid timestamp column";
const char* msg5 = "only tag query not compatible with normal column filter";
const char* msg6 = "not support stddev/percentile in outer query yet";
const char* msg7 = "drivative requires timestamp column exists in subquery";
int32_t code = TSDB_CODE_SUCCESS;
@ -7777,11 +7787,27 @@ int32_t validateSqlNode(SSqlObj* pSql, SSqlNode* pSqlNode, SQueryInfo* pQueryInf
}
}
// todo derivate funtion requires ts column exists in subquery
STableMeta* pTableMeta = tscGetMetaInfo(pQueryInfo, 0)->pTableMeta;
SSchema* pSchema = tscGetTableColumnSchema(pTableMeta, 0);
if (tscNumOfExprs(pQueryInfo) > 1) {
SExprInfo* pExpr = tscExprGet(pQueryInfo, 1);
if (pExpr->base.functionId == TSDB_FUNC_DERIVATIVE && pSchema->type != TSDB_DATA_TYPE_TIMESTAMP) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), msg7);
}
}
// validate the query filter condition info
if (pSqlNode->pWhere != NULL) {
if (validateWhereNode(pQueryInfo, &pSqlNode->pWhere, pSql) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_TSC_INVALID_OPERATION;
}
if (pTableMeta->tableInfo.precision == TSDB_TIME_PRECISION_MILLI) {
pQueryInfo->window.skey = pQueryInfo->window.skey / 1000;
pQueryInfo->window.ekey = pQueryInfo->window.ekey / 1000;
}
}
// validate the interval info
@ -7802,7 +7828,6 @@ int32_t validateSqlNode(SSqlObj* pSql, SSqlNode* pSqlNode, SQueryInfo* pQueryInf
}
// set order by info
STableMeta* pTableMeta = tscGetMetaInfo(pQueryInfo, 0)->pTableMeta;
if (validateOrderbyNode(pCmd, pQueryInfo, pSqlNode, tscGetTableSchema(pTableMeta)) != TSDB_CODE_SUCCESS) {
return TSDB_CODE_TSC_INVALID_OPERATION;
}
@ -7953,7 +7978,6 @@ int32_t validateSqlNode(SSqlObj* pSql, SSqlNode* pSqlNode, SQueryInfo* pQueryInf
pQueryInfo->arithmeticOnAgg = tsIsArithmeticQueryOnAggResult(pQueryInfo);
pQueryInfo->orderProjectQuery = tscOrderedProjectionQueryOnSTable(pQueryInfo, 0);
// pQueryInfo->diffQuery = tscIsDiffQuery(pQueryInfo);
SExprInfo** p = NULL;
int32_t numOfExpr = 0;
@ -8065,18 +8089,23 @@ int32_t exprTreeFromSqlExpr(SSqlCmd* pCmd, tExprNode **pExpr, const tSqlExpr* pS
return TSDB_CODE_SUCCESS;
} else if (pSqlExpr->tokenId == TK_SET) {
int32_t type = -1;
int32_t colType = -1;
STableMeta* pTableMeta = tscGetMetaInfo(pQueryInfo, 0)->pTableMeta;
if (pCols != NULL) {
SColIndex* idx = taosArrayGet(pCols, 0);
SSchema* pSchema = tscGetTableColumnSchema(pTableMeta, idx->colIndex);
if (pSchema != NULL) {
type = pSchema->type;
colType = pSchema->type;
}
}
tVariant *pVal;
if (serializeExprListToVariant(pSqlExpr->pParam, &pVal, type) == false) {
if (colType >= TSDB_DATA_TYPE_TINYINT && colType <= TSDB_DATA_TYPE_BIGINT) {
colType = TSDB_DATA_TYPE_BIGINT;
} else if (colType == TSDB_DATA_TYPE_FLOAT || colType == TSDB_DATA_TYPE_DOUBLE) {
colType = TSDB_DATA_TYPE_DOUBLE;
}
if (serializeExprListToVariant(pSqlExpr->pParam, &pVal, colType) == false) {
return invalidOperationMsg(tscGetErrorMsgPayload(pCmd), "not support filter expression");
}
*pExpr = calloc(1, sizeof(tExprNode));

85
src/client/src/tscServer.c
View File

@ -47,6 +47,31 @@ static int32_t getWaitingTimeInterval(int32_t count) {
return initial * ((2u)<<(count - 2));
}
static int32_t vgIdCompare(const void *lhs, const void *rhs) {
int32_t left = *(int32_t *)lhs;
int32_t right = *(int32_t *)rhs;
if (left == right) {
return 0;
} else {
return left > right ? 1 : -1;
}
}
static int32_t removeDupVgid(int32_t *src, int32_t sz) {
if (src == NULL || sz <= 0) {
return 0;
}
qsort(src, sz, sizeof(src[0]), vgIdCompare);
int32_t ret = 1;
for (int i = 1; i < sz; i++) {
if (src[i] != src[i - 1]) {
src[ret++] = src[i];
}
}
return ret;
}
static void tscSetDnodeEpSet(SRpcEpSet* pEpSet, SVgroupInfo* pVgroupInfo) {
assert(pEpSet != NULL && pVgroupInfo != NULL && pVgroupInfo->numOfEps > 0);
@ -1515,6 +1540,60 @@ int tscAlterDbMsg(SSqlObj *pSql, SSqlInfo *pInfo) {
return TSDB_CODE_SUCCESS;
}
int tscBuildCompactMsg(SSqlObj *pSql, SSqlInfo *pInfo) {
if (pInfo->list == NULL || taosArrayGetSize(pInfo->list) <= 0) {
return TSDB_CODE_TSC_INVALID_OPERATION;
}
STscObj *pObj = pSql->pTscObj;
SSqlCmd *pCmd = &pSql->cmd;
SArray *pList = pInfo->list;
int32_t size = (int32_t)taosArrayGetSize(pList);
int32_t *result = malloc(sizeof(int32_t) * size);
if (result == NULL) {
return TSDB_CODE_TSC_OUT_OF_MEMORY;
}
for (int32_t i = 0; i < size; i++) {
tSqlExprItem* pSub = taosArrayGet(pList, i);
tVariant* pVar = &pSub->pNode->value;
if (pVar->nType >= TSDB_DATA_TYPE_TINYINT && pVar->nType <= TSDB_DATA_TYPE_BIGINT) {
result[i] = (int32_t)(pVar->i64);
} else {
free(result);
return TSDB_CODE_TSC_INVALID_OPERATION;
}
}
int count = removeDupVgid(result, size);
pCmd->payloadLen = sizeof(SCompactMsg) + count * sizeof(int32_t);
pCmd->msgType = TSDB_MSG_TYPE_CM_COMPACT_VNODE;
if (TSDB_CODE_SUCCESS != tscAllocPayload(pCmd, pCmd->payloadLen)) {
tscError("0x%"PRIx64" failed to malloc for query msg", pSql->self);
free(result);
return TSDB_CODE_TSC_OUT_OF_MEMORY;
}
SCompactMsg *pCompactMsg = (SCompactMsg *)pCmd->payload;
STableMetaInfo *pTableMetaInfo = tscGetTableMetaInfoFromCmd(pCmd, 0);
if (tNameIsEmpty(&pTableMetaInfo->name)) {
pthread_mutex_lock(&pObj->mutex);
tstrncpy(pCompactMsg->db, pObj->db, sizeof(pCompactMsg->db));
pthread_mutex_unlock(&pObj->mutex);
} else {
tNameGetFullDbName(&pTableMetaInfo->name, pCompactMsg->db);
}
pCompactMsg->numOfVgroup = htons(count);
for (int32_t i = 0; i < count; i++) {
pCompactMsg->vgid[i] = htons(result[i]);
}
free(result);
return TSDB_CODE_SUCCESS;
}
int tscBuildRetrieveFromMgmtMsg(SSqlObj *pSql, SSqlInfo *pInfo) {
SSqlCmd *pCmd = &pSql->cmd;
@ -2268,6 +2347,10 @@ int tscProcessAlterDbMsgRsp(SSqlObj *pSql) {
UNUSED(pSql);
return 0;
}
int tscProcessCompactRsp(SSqlObj *pSql) {
UNUSED(pSql);
return TSDB_CODE_SUCCESS;
}
int tscProcessShowCreateRsp(SSqlObj *pSql) {
return tscLocalResultCommonBuilder(pSql, 1);
@ -2654,6 +2737,7 @@ void tscInitMsgsFp() {
tscBuildMsg[TSDB_SQL_ALTER_TABLE] = tscBuildAlterTableMsg;
tscBuildMsg[TSDB_SQL_UPDATE_TAGS_VAL] = tscBuildUpdateTagMsg;
tscBuildMsg[TSDB_SQL_ALTER_DB] = tscAlterDbMsg;
tscBuildMsg[TSDB_SQL_COMPACT_VNODE] = tscBuildCompactMsg;
tscBuildMsg[TSDB_SQL_CONNECT] = tscBuildConnectMsg;
tscBuildMsg[TSDB_SQL_USE_DB] = tscBuildUseDbMsg;
@ -2694,6 +2778,7 @@ void tscInitMsgsFp() {
tscProcessMsgRsp[TSDB_SQL_ALTER_TABLE] = tscProcessAlterTableMsgRsp;
tscProcessMsgRsp[TSDB_SQL_ALTER_DB] = tscProcessAlterDbMsgRsp;
tscProcessMsgRsp[TSDB_SQL_COMPACT_VNODE] = tscProcessCompactRsp;
tscProcessMsgRsp[TSDB_SQL_SHOW_CREATE_TABLE] = tscProcessShowCreateRsp;
tscProcessMsgRsp[TSDB_SQL_SHOW_CREATE_STABLE] = tscProcessShowCreateRsp;

7
src/client/src/tscSubquery.c
View File

@ -3152,6 +3152,13 @@ int32_t tscHandleMultivnodeInsert(SSqlObj *pSql) {
// it is the failure retry insert
if (pSql->pSubs != NULL) {
int32_t blockNum = (int32_t)taosArrayGetSize(pCmd->insertParam.pDataBlocks);
if (pSql->subState.numOfSub != blockNum) {
tscError("0x%"PRIx64" sub num:%d is not same with data block num:%d", pSql->self, pSql->subState.numOfSub, blockNum);
pRes->code = TSDB_CODE_TSC_APP_ERROR;
return pRes->code;
}
for(int32_t i = 0; i < pSql->subState.numOfSub; ++i) {
SSqlObj* pSub = pSql->pSubs[i];
SInsertSupporter* pSup = calloc(1, sizeof(SInsertSupporter));

12
src/client/src/tscSystem.c
View File

@ -288,16 +288,24 @@ static int taos_options_imp(TSDB_OPTION option, const char *pStr) {
if (strlen(tsLocale) == 0) { // locale does not set yet
char* defaultLocale = setlocale(LC_CTYPE, "");
// The locale of the current OS does not be set correctly, so the default locale cannot be acquired.
// The launch of current system will abort soon.
if (defaultLocale == NULL) {
tscError("failed to get default locale, please set the correct locale in current OS");
return -1;
}
tstrncpy(tsLocale, defaultLocale, TSDB_LOCALE_LEN);
}
// set the user specified locale
char *locale = setlocale(LC_CTYPE, pStr);
if (locale != NULL) {
if (locale != NULL) { // failed to set the user specified locale
tscInfo("locale set, prev locale:%s, new locale:%s", tsLocale, locale);
cfg->cfgStatus = TAOS_CFG_CSTATUS_OPTION;
} else { // set the user-specified localed failed, use default LC_CTYPE as current locale
} else { // set the user specified locale failed, use default LC_CTYPE as current locale
locale = setlocale(LC_CTYPE, tsLocale);
tscInfo("failed to set locale:%s, current locale:%s", pStr, tsLocale);
}

22
src/client/src/tscUtil.c
View File

@ -460,24 +460,6 @@ bool tscIsTWAQuery(SQueryInfo* pQueryInfo) {
return false;
}
bool tscIsDiffQuery(SQueryInfo* pQueryInfo) {
size_t num = tscNumOfExprs(pQueryInfo);
for(int32_t i = 0; i < num; ++i) {
SExprInfo* pExpr = tscExprGet(pQueryInfo, i);
int32_t f = pExpr->base.functionId;
if (pExpr == NULL || f == TSDB_FUNC_TS_DUMMY) {
continue;
}
if (f == TSDB_FUNC_DIFF || f == TSDB_FUNC_DERIVATIVE) {
return true;
}
}
return false;
}
bool tscIsSessionWindowQuery(SQueryInfo* pQueryInfo) {
return pQueryInfo->sessionWindow.gap > 0;
}
@ -516,7 +498,7 @@ bool isSimpleAggregateRv(SQueryInfo* pQueryInfo) {
return false;
}
if (tscIsDiffQuery(pQueryInfo)) {
if (tscIsDiffDerivQuery(pQueryInfo)) {
return false;
}
@ -4260,7 +4242,7 @@ int32_t tscCreateQueryFromQueryInfo(SQueryInfo* pQueryInfo, SQueryAttr* pQueryAt
pQueryAttr->hasTagResults = hasTagValOutput(pQueryInfo);
pQueryAttr->stabledev = isStabledev(pQueryInfo);
pQueryAttr->tsCompQuery = isTsCompQuery(pQueryInfo);
pQueryAttr->diffQuery = tscIsDiffQuery(pQueryInfo);
pQueryAttr->diffQuery = tscIsDiffDerivQuery(pQueryInfo);
pQueryAttr->simpleAgg = isSimpleAggregateRv(pQueryInfo);
pQueryAttr->needReverseScan = tscNeedReverseScan(pQueryInfo);
pQueryAttr->stableQuery = QUERY_IS_STABLE_QUERY(pQueryInfo->type);

7
src/client/tests/CMakeLists.txt
View File

@ -7,7 +7,12 @@ FIND_LIBRARY(LIB_GTEST_STATIC_DIR libgtest.a /usr/lib/ /usr/local/lib)
IF (HEADER_GTEST_INCLUDE_DIR AND LIB_GTEST_STATIC_DIR)
MESSAGE(STATUS "gTest library found, build unit test")
INCLUDE_DIRECTORIES(${HEADER_GTEST_INCLUDE_DIR})
# GoogleTest requires at least C++11
SET(CMAKE_CXX_STANDARD 11)
INCLUDE_DIRECTORIES(/usr/include /usr/local/include)
LINK_DIRECTORIES(/usr/lib /usr/local/lib)
AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)
ADD_EXECUTABLE(cliTest ${SOURCE_LIST})

2
src/common/inc/tcmdtype.h
View File

@ -51,6 +51,7 @@ enum {
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_ALTER_ACCT, "alter-acct" )
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_ALTER_TABLE, "alter-table" )
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_ALTER_DB, "alter-db" )
TSDB_DEFINE_SQL_TYPE(TSDB_SQL_SYNC_DB_REPLICA, "sync db-replica")
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_CREATE_MNODE, "create-mnode" )
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_DROP_MNODE, "drop-mnode" )
@ -63,6 +64,7 @@ enum {
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_KILL_QUERY, "kill-query" )
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_KILL_STREAM, "kill-stream" )
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_KILL_CONNECTION, "kill-connection" )
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_COMPACT_VNODE, "compact-vnode" )
// SQL below is for read operation
TSDB_DEFINE_SQL_TYPE( TSDB_SQL_READ, "read" )

3
src/common/src/texpr.c
View File

@ -470,6 +470,9 @@ void buildFilterSetFromBinary(void **q, const char *buf, int32_t len) {
SBufferReader br = tbufInitReader(buf, len, false);
uint32_t type = tbufReadUint32(&br);
SHashObj *pObj = taosHashInit(256, taosGetDefaultHashFunction(type), true, false);
taosHashSetEqualFp(pObj, taosGetDefaultEqualFunction(type));
int dummy = -1;
int32_t sz = tbufReadInt32(&br);
for (int32_t i = 0; i < sz; i++) {

3
src/connector/jdbc/CMakeLists.txt
View File

@ -8,9 +8,8 @@ IF (TD_MVN_INSTALLED)
ADD_CUSTOM_COMMAND(OUTPUT ${JDBC_CMD_NAME}
POST_BUILD
COMMAND mvn -Dmaven.test.skip=true install -f ${CMAKE_CURRENT_SOURCE_DIR}/pom.xml
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/target/taos-jdbcdriver-2.0.30.jar ${LIBRARY_OUTPUT_PATH}
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/target/taos-jdbcdriver-2.0.31.jar ${LIBRARY_OUTPUT_PATH}
COMMAND mvn -Dmaven.test.skip=true clean -f ${CMAKE_CURRENT_SOURCE_DIR}/pom.xml
COMMENT "build jdbc driver")
ADD_CUSTOM_TARGET(${JDBC_TARGET_NAME} ALL WORKING_DIRECTORY ${EXECUTABLE_OUTPUT_PATH} DEPENDS ${JDBC_CMD_NAME})
ENDIF ()

2
src/connector/jdbc/deploy-pom.xml
View File

@ -5,7 +5,7 @@
<groupId>com.taosdata.jdbc</groupId>
<artifactId>taos-jdbcdriver</artifactId>
<version>2.0.30</version>
<version>2.0.31</version>
<packaging>jar</packaging>
<name>JDBCDriver</name>

3
src/connector/jdbc/pom.xml
View File

@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.taosdata.jdbc</groupId>
<artifactId>taos-jdbcdriver</artifactId>
<version>2.0.30</version>
<version>2.0.31</version>
<packaging>jar</packaging>
<name>JDBCDriver</name>
<url>https://github.com/taosdata/TDengine/tree/master/src/connector/jdbc</url>
@ -123,6 +123,7 @@
<exclude>**/InvalidResultSetPointerTest.java</exclude>
<exclude>**/RestfulConnectionTest.java</exclude>
<exclude>**/TD4144Test.java</exclude>
<exclude>**/ConnectMultiTaosdByRestfulWithDifferentTokenTest.java</exclude>
</excludes>
<testFailureIgnore>true</testFailureIgnore>
</configuration>

9
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulConnection.java
View File

@ -17,16 +17,18 @@ public class RestfulConnection extends AbstractConnection {
private final int port;
private final String url;
private volatile String database;
private final String token;
/******************************************************/
private boolean isClosed;
private final DatabaseMetaData metadata;
public RestfulConnection(String host, String port, Properties props, String database, String url) {
public RestfulConnection(String host, String port, Properties props, String database, String url, String token) {
super(props);
this.host = host;
this.port = Integer.parseInt(port);
this.database = database;
this.url = url;
this.token = token;
this.metadata = new RestfulDatabaseMetaData(url, props.getProperty(TSDBDriver.PROPERTY_KEY_USER), this);
}
@ -66,6 +68,7 @@ public class RestfulConnection extends AbstractConnection {
return this.metadata;
}
// getters
public String getHost() {
return host;
}
@ -81,4 +84,8 @@ public class RestfulConnection extends AbstractConnection {
public String getUrl() {
return url;
}
public String getToken() {
return token;
}
}

12
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulDriver.java
View File

@ -38,15 +38,11 @@ public class RestfulDriver extends AbstractDriver {
String port = props.getProperty(TSDBDriver.PROPERTY_KEY_PORT, "6041");
String database = props.containsKey(TSDBDriver.PROPERTY_KEY_DBNAME) ? props.getProperty(TSDBDriver.PROPERTY_KEY_DBNAME) : null;
String loginUrl = "http://" + props.getProperty(TSDBDriver.PROPERTY_KEY_HOST) + ":"
+ props.getProperty(TSDBDriver.PROPERTY_KEY_PORT) + "/rest/login/"
+ props.getProperty(TSDBDriver.PROPERTY_KEY_USER) + "/"
+ props.getProperty(TSDBDriver.PROPERTY_KEY_PASSWORD) + "";
String loginUrl = "http://" + host + ":" + port + "/rest/login/" + props.getProperty(TSDBDriver.PROPERTY_KEY_USER) + "/" + props.getProperty(TSDBDriver.PROPERTY_KEY_PASSWORD) + "";
try {
String user = URLEncoder.encode(props.getProperty(TSDBDriver.PROPERTY_KEY_USER), "UTF-8");
String password = URLEncoder.encode(props.getProperty(TSDBDriver.PROPERTY_KEY_PASSWORD), "UTF-8");
loginUrl = "http://" + props.getProperty(TSDBDriver.PROPERTY_KEY_HOST) + ":"
+ props.getProperty(TSDBDriver.PROPERTY_KEY_PORT) + "/rest/login/" + user + "/" + password + "";
loginUrl = "http://" + props.getProperty(TSDBDriver.PROPERTY_KEY_HOST) + ":" + props.getProperty(TSDBDriver.PROPERTY_KEY_PORT) + "/rest/login/" + user + "/" + password + "";
} catch (UnsupportedEncodingException e) {
e.printStackTrace();
}
@ -55,12 +51,12 @@ public class RestfulDriver extends AbstractDriver {
JSONObject jsonResult = JSON.parseObject(result);
String status = jsonResult.getString("status");
String token = jsonResult.getString("desc");
HttpClientPoolUtil.token = token;
if (!status.equals("succ")) {
throw new SQLException(jsonResult.getString("desc"));
}
RestfulConnection conn = new RestfulConnection(host, port, props, database, url);
RestfulConnection conn = new RestfulConnection(host, port, props, database, url, token);
if (database != null && !database.trim().replaceAll("\\s", "").isEmpty()) {
Statement stmt = conn.createStatement();
stmt.execute("use " + database);

6
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulStatement.java
View File

@ -83,7 +83,7 @@ public class RestfulStatement extends AbstractStatement {
}
if (SqlSyntaxValidator.isUseSql(sql)) {
HttpClientPoolUtil.execute(url, sql);
HttpClientPoolUtil.execute(url, sql, this.conn.getToken());
this.database = sql.trim().replace("use", "").trim();
this.conn.setCatalog(this.database);
result = false;
@ -116,7 +116,7 @@ public class RestfulStatement extends AbstractStatement {
if ("UTC".equalsIgnoreCase(timestampFormat))
url = "http://" + conn.getHost() + ":" + conn.getPort() + "/rest/sqlutc";
String result = HttpClientPoolUtil.execute(url, sql);
String result = HttpClientPoolUtil.execute(url, sql, this.conn.getToken());
JSONObject resultJson = JSON.parseObject(result);
if (resultJson.getString("status").equals("error")) {
throw TSDBError.createSQLException(resultJson.getInteger("code"), resultJson.getString("desc"));
@ -130,7 +130,7 @@ public class RestfulStatement extends AbstractStatement {
if (!SqlSyntaxValidator.isValidForExecuteUpdate(sql))
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE_UPDATE, "not a valid sql for executeUpdate: " + sql);
String result = HttpClientPoolUtil.execute(url, sql);
String result = HttpClientPoolUtil.execute(url, sql, this.conn.getToken());
JSONObject jsonObject = JSON.parseObject(result);
if (jsonObject.getString("status").equals("error")) {
throw TSDBError.createSQLException(jsonObject.getInteger("code"), jsonObject.getString("desc"));

44
src/connector/jdbc/src/main/java/com/taosdata/jdbc/utils/HttpClientPoolUtil.java
View File

@ -20,39 +20,29 @@ import java.nio.charset.Charset;
public class HttpClientPoolUtil {
public static PoolingHttpClientConnectionManager cm = null;
public static CloseableHttpClient httpClient = null;
public static String token = "cm9vdDp0YW9zZGF0YQ==";
/**
* 默认content 类型
*/
private static final String DEFAULT_CONTENT_TYPE = "application/json";
/**
* 默认请求超时时间30s
*/
private static final int DEFAULT_TIME_OUT = 15000;
private static final int count = 32;
private static final int totalCount = 1000;
private static final int Http_Default_Keep_Time = 15000;
/**
* 初始化连接池
*/
private static final String DEFAULT_CONTENT_TYPE = "application/json";
private static final String DEFAULT_TOKEN = "cm9vdDp0YW9zZGF0YQ==";
private static final int DEFAULT_TIME_OUT = 15000;
private static final int DEFAULT_MAX_PER_ROUTE = 32;
private static final int DEFAULT_MAX_TOTAL = 1000;
private static final int DEFAULT_HTTP_KEEP_TIME = 15000;
private static PoolingHttpClientConnectionManager connectionManager;
private static CloseableHttpClient httpClient;
private static synchronized void initPools() {
if (httpClient == null) {
cm = new PoolingHttpClientConnectionManager();
cm.setDefaultMaxPerRoute(count);
cm.setMaxTotal(totalCount);
httpClient = HttpClients.custom().setKeepAliveStrategy(defaultStrategy).setConnectionManager(cm).build();
connectionManager = new PoolingHttpClientConnectionManager();
connectionManager.setDefaultMaxPerRoute(DEFAULT_MAX_PER_ROUTE);
connectionManager.setMaxTotal(DEFAULT_MAX_TOTAL);
httpClient = HttpClients.custom().setKeepAliveStrategy(DEFAULT_KEEP_ALIVE_STRATEGY).setConnectionManager(connectionManager).build();
}
}
/**
* Http connection keepAlive 设置
*/
private static ConnectionKeepAliveStrategy defaultStrategy = (response, context) -> {
private static ConnectionKeepAliveStrategy DEFAULT_KEEP_ALIVE_STRATEGY = (response, context) -> {
HeaderElementIterator it = new BasicHeaderElementIterator(response.headerIterator(HTTP.CONN_KEEP_ALIVE));
int keepTime = Http_Default_Keep_Time * 1000;
int keepTime = DEFAULT_HTTP_KEEP_TIME * 1000;
while (it.hasNext()) {
HeaderElement headerElement = it.nextElement();
String param = headerElement.getName();
@ -76,7 +66,7 @@ public class HttpClientPoolUtil {
* @param data 请求数据
* @return responseBody
*/
public static String execute(String uri, String data) {
public static String execute(String uri, String data, String token) {
long startTime = System.currentTimeMillis();
HttpEntity httpEntity = null;
HttpEntityEnclosingRequestBase method = null;

4
src/connector/jdbc/src/test/java/com/taosdata/jdbc/cases/BatchInsertTest.java
View File

@ -28,9 +28,7 @@ public class BatchInsertTest {
@Before
public void before() {
try {
Class.forName("com.taosdata.jdbc.TSDBDriver");
Properties properties = new Properties();
properties.setProperty(TSDBDriver.PROPERTY_KEY_HOST, host);
properties.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_LOCALE, "en_US.UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8");
@ -44,7 +42,7 @@ public class BatchInsertTest {
String createTableSql = "create table " + stbName + "(ts timestamp, f1 int, f2 int, f3 int) tags(areaid int, loc binary(20))";
statement.executeUpdate(createTableSql);
// create tables
for(int i = 0; i < numOfTables; i++) {
for (int i = 0; i < numOfTables; i++) {
String loc = i % 2 == 0 ? "beijing" : "shanghai";
String createSubTalbesSql = "create table " + tablePrefix + i + " using " + stbName + " tags(" + i + ", '" + loc + "')";
statement.executeUpdate(createSubTalbesSql);

59
src/connector/jdbc/src/test/java/com/taosdata/jdbc/cases/ConnectMultiTaosdByRestfulWithDifferentTokenTest.java Normal file
View File

@ -0,0 +1,59 @@
package com.taosdata.jdbc.cases;
import com.taosdata.jdbc.TSDBDriver;
import org.junit.Before;
import org.junit.Test;
import java.sql.*;
import java.util.Properties;
public class ConnectMultiTaosdByRestfulWithDifferentTokenTest {
private static String host1 = "192.168.17.156";
private static String user1 = "root";
private static String password1 = "tqueue";
private Connection conn1;
private static String host2 = "192.168.17.82";
private static String user2 = "root";
private static String password2 = "taosdata";
private Connection conn2;
@Test
public void test() {
//when
executeSelectStatus(conn1);
executeSelectStatus(conn2);
executeSelectStatus(conn1);
}
private void executeSelectStatus(Connection connection) {
try (Statement stmt = connection.createStatement()) {
ResultSet rs = stmt.executeQuery("select server_status()");
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
for (int i = 1; i <= meta.getColumnCount(); i++) {
System.out.println(meta.getColumnLabel(i) + ": " + rs.getString(i));
}
}
} catch (SQLException e) {
e.printStackTrace();
}
}
@Before
public void before() {
Properties properties = new Properties();
properties.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_LOCALE, "en_US.UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8");
String url1 = "jdbc:TAOS-RS://" + host1 + ":6041/?user=" + user1 + "&password=" + password1;
String url2 = "jdbc:TAOS-RS://" + host2 + ":6041/?user=" + user2 + "&password=" + password2;
try {
conn1 = DriverManager.getConnection(url1, properties);
conn2 = DriverManager.getConnection(url2, properties);
} catch (SQLException e) {
e.printStackTrace();
}
}
}

3
src/dnode/src/dnodePeer.c
View File

@ -47,7 +47,8 @@ int32_t dnodeInitServer() {
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_MD_DROP_VNODE] = dnodeDispatchToVMgmtQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_MD_ALTER_STREAM] = dnodeDispatchToVMgmtQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_MD_CONFIG_DNODE] = dnodeDispatchToVMgmtQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_MD_CREATE_MNODE] = dnodeDispatchToVMgmtQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_MD_CREATE_MNODE] = dnodeDispatchToVMgmtQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_MD_COMPACT_VNODE] = dnodeDispatchToVMgmtQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_DM_CONFIG_TABLE] = dnodeDispatchToMPeerQueue;
dnodeProcessReqMsgFp[TSDB_MSG_TYPE_DM_CONFIG_VNODE] = dnodeDispatchToMPeerQueue;

1
src/dnode/src/dnodeShell.c
View File

@ -61,6 +61,7 @@ int32_t dnodeInitShell() {
dnodeProcessShellMsgFp[TSDB_MSG_TYPE_CM_KILL_STREAM] = dnodeDispatchToMWriteQueue;
dnodeProcessShellMsgFp[TSDB_MSG_TYPE_CM_KILL_CONN] = dnodeDispatchToMWriteQueue;
dnodeProcessShellMsgFp[TSDB_MSG_TYPE_CM_CONFIG_DNODE]= dnodeDispatchToMWriteQueue;
dnodeProcessShellMsgFp[TSDB_MSG_TYPE_CM_COMPACT_VNODE]= dnodeDispatchToMWriteQueue;
// the following message shall be treated as mnode query
dnodeProcessShellMsgFp[TSDB_MSG_TYPE_CM_HEARTBEAT] = dnodeDispatchToMReadQueue;

10
src/dnode/src/dnodeVMgmt.c
View File

@ -31,6 +31,7 @@ static void * dnodeProcessMgmtQueue(void *param);
static int32_t dnodeProcessCreateVnodeMsg(SRpcMsg *pMsg);
static int32_t dnodeProcessAlterVnodeMsg(SRpcMsg *pMsg);
static int32_t dnodeProcessSyncVnodeMsg(SRpcMsg *pMsg);
static int32_t dnodeProcessCompactVnodeMsg(SRpcMsg *pMsg);
static int32_t dnodeProcessDropVnodeMsg(SRpcMsg *pMsg);
static int32_t dnodeProcessAlterStreamMsg(SRpcMsg *pMsg);
static int32_t dnodeProcessConfigDnodeMsg(SRpcMsg *pMsg);
@ -40,7 +41,8 @@ static int32_t (*dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MAX])(SRpcMsg *pMsg);
int32_t dnodeInitVMgmt() {
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_CREATE_VNODE] = dnodeProcessCreateVnodeMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_ALTER_VNODE] = dnodeProcessAlterVnodeMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_SYNC_VNODE] = dnodeProcessSyncVnodeMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_SYNC_VNODE] = dnodeProcessSyncVnodeMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_COMPACT_VNODE]= dnodeProcessCompactVnodeMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_DROP_VNODE] = dnodeProcessDropVnodeMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_ALTER_STREAM] = dnodeProcessAlterStreamMsg;
dnodeProcessMgmtMsgFp[TSDB_MSG_TYPE_MD_CONFIG_DNODE] = dnodeProcessConfigDnodeMsg;
@ -187,6 +189,12 @@ static int32_t dnodeProcessSyncVnodeMsg(SRpcMsg *rpcMsg) {
return vnodeSync(pSyncVnode->vgId);
}
static int32_t dnodeProcessCompactVnodeMsg(SRpcMsg *rpcMsg) {
SCompactVnodeMsg *pCompactVnode = rpcMsg->pCont;
pCompactVnode->vgId = htonl(pCompactVnode->vgId);
return vnodeCompact(pCompactVnode->vgId);
}
static int32_t dnodeProcessDropVnodeMsg(SRpcMsg *rpcMsg) {
SDropVnodeMsg *pDrop = rpcMsg->pCont;
pDrop->vgId = htonl(pDrop->vgId);

2
src/inc/taosdef.h
View File

@ -166,7 +166,7 @@ do { \
#define TSDB_BINARY_OP_REMAINDER 34
#define IS_RELATION_OPTR(op) (((op) >= TSDB_RELATION_LESS) && ((op) <= TSDB_RELATION_IN))
#define IS_RELATION_OPTR(op) (((op) >= TSDB_RELATION_LESS) && ((op) < TSDB_RELATION_IN))
#define IS_ARITHMETIC_OPTR(op) (((op) >= TSDB_BINARY_OP_ADD) && ((op) <= TSDB_BINARY_OP_REMAINDER))
#define TS_PATH_DELIMITER_LEN 1

13
src/inc/taosmsg.h
View File

@ -61,9 +61,11 @@ TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_MD_CONFIG_DNODE, "config-dnode" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_MD_ALTER_VNODE, "alter-vnode" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_MD_SYNC_VNODE, "sync-vnode" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_MD_CREATE_MNODE, "create-mnode" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_MD_COMPACT_VNODE, "compact-vnode" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_DUMMY6, "dummy6" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_DUMMY7, "dummy7" )
// message from client to mnode
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_CONNECT, "connect" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_CREATE_ACCT, "create-acct" )
@ -84,6 +86,7 @@ TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_DROP_TABLE, "drop-table" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_ALTER_TABLE, "alter-table" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_TABLE_META, "table-meta" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_STABLE_VGROUP, "stable-vgroup" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_COMPACT_VNODE, "compact-vnode" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_TABLES_META, "multiTable-meta" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_ALTER_STREAM, "alter-stream" )
TAOS_DEFINE_MESSAGE_TYPE( TSDB_MSG_TYPE_CM_SHOW, "show" )
@ -393,7 +396,7 @@ typedef struct {
typedef struct {
int32_t vgId;
} SDropVnodeMsg, SSyncVnodeMsg;
} SDropVnodeMsg, SSyncVnodeMsg, SCompactVnodeMsg;
typedef struct SColIndex {
int16_t colId; // column id
@ -539,7 +542,7 @@ typedef struct {
uint8_t status;
uint8_t role;
uint8_t replica;
uint8_t reserved;
uint8_t compact;
} SVnodeLoad;
typedef struct {
@ -781,6 +784,12 @@ typedef struct {
char payload[];
} SShowMsg;
typedef struct {
char db[TSDB_ACCT_ID_LEN + TSDB_DB_NAME_LEN];
int32_t numOfVgroup;
int32_t vgid[];
} SCompactMsg;
typedef struct SShowRsp {
uint64_t qhandle;
STableMetaMsg tableMeta;

227
src/inc/ttokendef.h
View File

@ -16,7 +16,6 @@
#ifndef TDENGINE_TTOKENDEF_H
#define TDENGINE_TTOKENDEF_H
#define TK_ID 1
#define TK_BOOL 2
#define TK_TINYINT 3
@ -95,121 +94,117 @@
#define TK_PASS 76
#define TK_PRIVILEGE 77
#define TK_LOCAL 78
#define TK_IF 79
#define TK_EXISTS 80
#define TK_PPS 81
#define TK_TSERIES 82
#define TK_DBS 83
#define TK_STORAGE 84
#define TK_QTIME 85
#define TK_CONNS 86
#define TK_STATE 87
#define TK_COMMA 88
#define TK_KEEP 89
#define TK_CACHE 90
#define TK_REPLICA 91
#define TK_QUORUM 92
#define TK_DAYS 93
#define TK_MINROWS 94
#define TK_MAXROWS 95
#define TK_BLOCKS 96
#define TK_CTIME 97
#define TK_WAL 98
#define TK_FSYNC 99
#define TK_COMP 100
#define TK_PRECISION 101
#define TK_UPDATE 102
#define TK_CACHELAST 103
#define TK_PARTITIONS 104
#define TK_LP 105
#define TK_RP 106
#define TK_UNSIGNED 107
#define TK_TAGS 108
#define TK_USING 109
#define TK_AS 110
#define TK_NULL 111
#define TK_SELECT 112
#define TK_UNION 113
#define TK_ALL 114
#define TK_DISTINCT 115
#define TK_FROM 116
#define TK_VARIABLE 117
#define TK_INTERVAL 118
#define TK_SESSION 119
#define TK_STATE_WINDOW 120
#define TK_FILL 121
#define TK_SLIDING 122
#define TK_ORDER 123
#define TK_BY 124
#define TK_ASC 125
#define TK_DESC 126
#define TK_GROUP 127
#define TK_HAVING 128
#define TK_LIMIT 129
#define TK_OFFSET 130
#define TK_SLIMIT 131
#define TK_SOFFSET 132
#define TK_WHERE 133
#define TK_NOW 134
#define TK_RESET 135
#define TK_QUERY 136
#define TK_SYNCDB 137
#define TK_ADD 138
#define TK_COLUMN 139
#define TK_MODIFY 140
#define TK_TAG 141
#define TK_CHANGE 142
#define TK_SET 143
#define TK_KILL 144
#define TK_CONNECTION 145
#define TK_STREAM 146
#define TK_COLON 147
#define TK_ABORT 148
#define TK_AFTER 149
#define TK_ATTACH 150
#define TK_BEFORE 151
#define TK_BEGIN 152
#define TK_CASCADE 153
#define TK_CLUSTER 154
#define TK_CONFLICT 155
#define TK_COPY 156
#define TK_DEFERRED 157
#define TK_DELIMITERS 158
#define TK_DETACH 159
#define TK_EACH 160
#define TK_END 161
#define TK_EXPLAIN 162
#define TK_FAIL 163
#define TK_FOR 164
#define TK_IGNORE 165
#define TK_IMMEDIATE 166
#define TK_INITIALLY 167
#define TK_INSTEAD 168
#define TK_MATCH 169
#define TK_KEY 170
#define TK_OF 171
#define TK_RAISE 172
#define TK_REPLACE 173
#define TK_RESTRICT 174
#define TK_ROW 175
#define TK_STATEMENT 176
#define TK_TRIGGER 177
#define TK_VIEW 178
#define TK_SEMI 179
#define TK_NONE 180
#define TK_PREV 181
#define TK_LINEAR 182
#define TK_IMPORT 183
#define TK_TBNAME 184
#define TK_JOIN 185
#define TK_INSERT 186
#define TK_INTO 187
#define TK_VALUES 188
#define TK_COMPACT 79
#define TK_LP 80
#define TK_RP 81
#define TK_IF 82
#define TK_EXISTS 83
#define TK_PPS 84
#define TK_TSERIES 85
#define TK_DBS 86
#define TK_STORAGE 87
#define TK_QTIME 88
#define TK_CONNS 89
#define TK_STATE 90
#define TK_COMMA 91
#define TK_KEEP 92
#define TK_CACHE 93
#define TK_REPLICA 94
#define TK_QUORUM 95
#define TK_DAYS 96
#define TK_MINROWS 97
#define TK_MAXROWS 98
#define TK_BLOCKS 99
#define TK_CTIME 100
#define TK_WAL 101
#define TK_FSYNC 102
#define TK_COMP 103
#define TK_PRECISION 104
#define TK_UPDATE 105
#define TK_CACHELAST 106
#define TK_PARTITIONS 107
#define TK_UNSIGNED 108
#define TK_TAGS 109
#define TK_USING 110
#define TK_AS 111
#define TK_NULL 112
#define TK_SELECT 113
#define TK_UNION 114
#define TK_ALL 115
#define TK_DISTINCT 116
#define TK_FROM 117
#define TK_VARIABLE 118
#define TK_INTERVAL 119
#define TK_SESSION 120
#define TK_STATE_WINDOW 121
#define TK_FILL 122
#define TK_SLIDING 123
#define TK_ORDER 124
#define TK_BY 125
#define TK_ASC 126
#define TK_DESC 127
#define TK_GROUP 128
#define TK_HAVING 129
#define TK_LIMIT 130
#define TK_OFFSET 131
#define TK_SLIMIT 132
#define TK_SOFFSET 133
#define TK_WHERE 134
#define TK_NOW 135
#define TK_RESET 136
#define TK_QUERY 137
#define TK_SYNCDB 138
#define TK_ADD 139
#define TK_COLUMN 140
#define TK_MODIFY 141
#define TK_TAG 142
#define TK_CHANGE 143
#define TK_SET 144
#define TK_KILL 145
#define TK_CONNECTION 146
#define TK_STREAM 147
#define TK_COLON 148
#define TK_ABORT 149
#define TK_AFTER 150
#define TK_ATTACH 151
#define TK_BEFORE 152
#define TK_BEGIN 153
#define TK_CASCADE 154
#define TK_CLUSTER 155
#define TK_CONFLICT 156
#define TK_COPY 157
#define TK_DEFERRED 158
#define TK_DELIMITERS 159
#define TK_DETACH 160
#define TK_EACH 161
#define TK_END 162
#define TK_EXPLAIN 163
#define TK_FAIL 164
#define TK_FOR 165
#define TK_IGNORE 166
#define TK_IMMEDIATE 167
#define TK_INITIALLY 168
#define TK_INSTEAD 169
#define TK_MATCH 170
#define TK_KEY 171
#define TK_OF 172
#define TK_RAISE 173
#define TK_REPLACE 174
#define TK_RESTRICT 175
#define TK_ROW 176
#define TK_STATEMENT 177
#define TK_TRIGGER 178
#define TK_VIEW 179
#define TK_SEMI 180
#define TK_NONE 181
#define TK_PREV 182
#define TK_LINEAR 183
#define TK_IMPORT 184
#define TK_TBNAME 185
#define TK_JOIN 186
#define TK_INSERT 187
#define TK_INTO 188
#define TK_VALUES 189
#define TK_SPACE 300

1
src/inc/vnode.h
View File

@ -62,6 +62,7 @@ int32_t vnodeOpen(int32_t vgId);
int32_t vnodeAlter(void *pVnode, SCreateVnodeMsg *pVnodeCfg);
int32_t vnodeSync(int32_t vgId);
int32_t vnodeClose(int32_t vgId);
int32_t vnodeCompact(int32_t vgId);
// vnodeMgmt
int32_t vnodeInitMgmt();

4
src/mnode/inc/mnodeDef.h
View File

@ -48,9 +48,8 @@ typedef struct SDnodeObj {
int32_t dnodeId;
int32_t openVnodes;
int64_t createdTime;
int32_t resever0; // from dnode status msg, config information
int64_t lastAccess;
int32_t customScore; // config by user
uint32_t lastAccess;
uint16_t numOfCores; // from dnode status msg
uint16_t dnodePort;
char dnodeFqdn[TSDB_FQDN_LEN];
@ -145,6 +144,7 @@ typedef struct SVgObj {
int8_t reserved0[4];
SVnodeGid vnodeGid[TSDB_MAX_REPLICA];
int32_t vgCfgVersion;
int8_t compact;
int8_t reserved1[8];
int8_t updateEnd[4];
int32_t refCount;

2
src/mnode/inc/mnodeDnode.h
View File

@ -78,7 +78,7 @@ void mnodeUpdateDnode(SDnodeObj *pDnode);
int32_t mnodeDropDnode(SDnodeObj *pDnode, void *pMsg);
int32_t mnodeCompactDnodes();
extern int32_t tsAccessSquence;
extern int64_t tsAccessSquence;
#ifdef __cplusplus
}

1
src/mnode/inc/mnodeVgroup.h
View File

@ -51,6 +51,7 @@ void mnodeSendDropVnodeMsg(int32_t vgId, SRpcEpSet *epSet, void *ahandle);
void mnodeSendCreateVgroupMsg(SVgObj *pVgroup, void *ahandle);
void mnodeSendAlterVgroupMsg(SVgObj *pVgroup,SMnodeMsg *pMsg);
void mnodeSendSyncVgroupMsg(SVgObj *pVgroup);
void mnodeSendCompactVgroupMsg(SVgObj *pVgroup);
SRpcEpSet mnodeGetEpSetFromVgroup(SVgObj *pVgroup);
SRpcEpSet mnodeGetEpSetFromIp(char *ep);

56
src/mnode/src/mnodeDb.c
View File

@ -53,6 +53,7 @@ static int32_t mnodeRetrieveDbs(SShowObj *pShow, char *data, int32_t rows, void
static int32_t mnodeProcessCreateDbMsg(SMnodeMsg *pMsg);
static int32_t mnodeProcessDropDbMsg(SMnodeMsg *pMsg);
static int32_t mnodeProcessSyncDbMsg(SMnodeMsg *pMsg);
static int32_t mnodeProcessCompactMsg(SMnodeMsg *pMsg);
int32_t mnodeProcessAlterDbMsg(SMnodeMsg *pMsg);
#ifndef _TOPIC
@ -200,10 +201,12 @@ int32_t mnodeInitDbs() {
mnodeAddWriteMsgHandle(TSDB_MSG_TYPE_CM_ALTER_DB, mnodeProcessAlterDbMsg);
mnodeAddWriteMsgHandle(TSDB_MSG_TYPE_CM_DROP_DB, mnodeProcessDropDbMsg);
mnodeAddWriteMsgHandle(TSDB_MSG_TYPE_CM_SYNC_DB, mnodeProcessSyncDbMsg);
mnodeAddWriteMsgHandle(TSDB_MSG_TYPE_CM_COMPACT_VNODE, mnodeProcessCompactMsg);
mnodeAddShowMetaHandle(TSDB_MGMT_TABLE_DB, mnodeGetDbMeta);
mnodeAddShowRetrieveHandle(TSDB_MGMT_TABLE_DB, mnodeRetrieveDbs);
mnodeAddShowFreeIterHandle(TSDB_MGMT_TABLE_DB, mnodeCancelGetNextDb);
mDebug("table:dbs table is created");
return tpInit();
}
@ -1267,7 +1270,7 @@ static int32_t mnodeProcessDropDbMsg(SMnodeMsg *pMsg) {
static int32_t mnodeSyncDb(SDbObj *pDb, SMnodeMsg *pMsg) {
void *pIter = NULL;
SVgObj *pVgroup = NULL;
while (1) {
while (1) {
pIter = mnodeGetNextVgroup(pIter, &pVgroup);
if (pVgroup == NULL) break;
if (pVgroup->pDb == pDb) {
@ -1281,6 +1284,43 @@ static int32_t mnodeSyncDb(SDbObj *pDb, SMnodeMsg *pMsg) {
return TSDB_CODE_SUCCESS;
}
static int32_t mnodeCompact(SDbObj *pDb, SCompactMsg *pCompactMsg) {
int32_t count = ntohs(pCompactMsg->numOfVgroup);
int32_t *buf = malloc(sizeof(int32_t) * count);
if (buf == NULL) {
return TSDB_CODE_MND_OUT_OF_MEMORY;
}
for (int32_t i = 0; i < count; i++) {
buf[i] = ntohs(pCompactMsg->vgid[i]);
}
// copy from mnodeSyncDb, so ugly
for (int32_t i = 0; i < count; i++) {
SVgObj *pVgroup = NULL;
void *pIter = NULL;
bool valid = false;
while (1) {
pIter = mnodeGetNextVgroup(pIter, &pVgroup);
if (pVgroup == NULL) break;
if (pVgroup->pDb == pDb && pVgroup->vgId == buf[i]) {
mnodeSendCompactVgroupMsg(pVgroup);
mnodeDecVgroupRef(pVgroup);
valid = true;
break;
}
mnodeDecVgroupRef(pVgroup);
}
if (valid == false) {
mLError("db:%s, cannot find valid vgId: %d", pDb->name, buf[i]);
}
}
free(buf);
mLInfo("db:%s, trigger compact", pDb->name);
return TSDB_CODE_SUCCESS;
}
static int32_t mnodeProcessSyncDbMsg(SMnodeMsg *pMsg) {
SSyncDbMsg *pSyncDb = pMsg->rpcMsg.pCont;
mDebug("db:%s, syncdb is received from thandle:%p, ignore:%d", pSyncDb->db, pMsg->rpcMsg.handle, pSyncDb->ignoreNotExists);
@ -1303,6 +1343,20 @@ static int32_t mnodeProcessSyncDbMsg(SMnodeMsg *pMsg) {
return mnodeSyncDb(pMsg->pDb, pMsg);
}
static int32_t mnodeProcessCompactMsg(SMnodeMsg *pMsg) {
SCompactMsg *pCompact = pMsg->rpcMsg.pCont;
mDebug("db:%s, compact is received from thandle:%p", pCompact->db, pMsg->rpcMsg.handle);
if (pMsg->pDb == NULL) pMsg->pDb = mnodeGetDb(pCompact->db);
if (pMsg->pDb == NULL) return TSDB_CODE_MND_DB_NOT_SELECTED;
if (pMsg->pDb->status != TSDB_DB_STATUS_READY) {
mError("db:%s, status:%d, in dropping, ignore compact request", pCompact->db, pMsg->pDb->status);
return TSDB_CODE_MND_DB_IN_DROPPING;
}
return mnodeCompact(pMsg->pDb, pCompact);
}
void mnodeDropAllDbs(SAcctObj *pAcct) {
int32_t numOfDbs = 0;

12
src/mnode/src/mnodeDnode.c
View File

@ -39,8 +39,8 @@
#include "mnodePeer.h"
#include "mnodeCluster.h"
int32_t tsAccessSquence = 0;
int64_t tsDnodeRid = -1;
int64_t tsAccessSquence = 0;
int64_t tsDnodeRid = -1;
static void * tsDnodeSdb = NULL;
static int32_t tsDnodeUpdateSize = 0;
extern void * tsMnodeSdb;
@ -567,7 +567,7 @@ static int32_t mnodeProcessDnodeStatusMsg(SMnodeMsg *pMsg) {
mnodeGetClusterId());
return TSDB_CODE_MND_INVALID_CLUSTER_ID;
} else {
mTrace("dnode:%d, status received, access times %d openVnodes:%d:%d", pDnode->dnodeId, pDnode->lastAccess,
mTrace("dnode:%d, status received, access times %" PRId64 " openVnodes:%d:%d", pDnode->dnodeId, pDnode->lastAccess,
htons(pStatus->openVnodes), pDnode->openVnodes);
}
}
@ -629,9 +629,9 @@ static int32_t mnodeProcessDnodeStatusMsg(SMnodeMsg *pMsg) {
bnNotify();
}
if (!tsEnableBalance) {
int32_t numOfMnodes = mnodeGetMnodesNum();
if (numOfMnodes < tsNumOfMnodes) bnNotify();
int32_t numOfMnodes = mnodeGetMnodesNum();
if (numOfMnodes < tsNumOfMnodes && numOfMnodes < mnodeGetOnlineDnodesNum() && !pDnode->isMgmt) {
bnNotify();
}
if (openVnodes != pDnode->openVnodes) {

17
src/mnode/src/mnodeSdb.c
View File

@ -670,10 +670,17 @@ static int32_t sdbProcessWrite(void *wparam, void *hparam, int32_t qtype, void *
pTable->name, actStr[action], sdbGetKeyStr(pTable, pHead->cont), qtype, pHead->version, tsSdbMgmt.version);
return TSDB_CODE_SUCCESS;
} else if (pHead->version != tsSdbMgmt.version + 1) {
pthread_mutex_unlock(&tsSdbMgmt.mutex);
sdbError("vgId:1, sdb:%s, failed to restore %s key:%s from source(%d), hver:%" PRIu64 " too large, mver:%" PRIu64,
pTable->name, actStr[action], sdbGetKeyStr(pTable, pHead->cont), qtype, pHead->version, tsSdbMgmt.version);
return TSDB_CODE_SYN_INVALID_VERSION;
if (qtype != TAOS_QTYPE_WAL) {
pthread_mutex_unlock(&tsSdbMgmt.mutex);
sdbError(
"vgId:1, sdb:%s, failed to restore %s key:%s from source(%d), hver:%" PRIu64 " too large, mver:%" PRIu64,
pTable->name, actStr[action], sdbGetKeyStr(pTable, pHead->cont), qtype, pHead->version, tsSdbMgmt.version);
return TSDB_CODE_SYN_INVALID_VERSION;
} else {
// If cksum is wrong when recovering wal, use this code
tsSdbMgmt.version = pHead->version;
}
} else {
tsSdbMgmt.version = pHead->version;
}
@ -1195,4 +1202,4 @@ int32_t mnodeCompactWal() {
sdbInfo("vgId:1, compact mnode wal success");
return 0;
}
}

4
src/mnode/src/mnodeTable.c
View File

@ -1475,6 +1475,7 @@ static int32_t mnodeChangeSuperTableColumn(SMnodeMsg *pMsg) {
SSchema *schema = (SSchema *) (pStable->schema + col);
ASSERT(schema->type == TSDB_DATA_TYPE_BINARY || schema->type == TSDB_DATA_TYPE_NCHAR);
schema->bytes = pAlter->schema[0].bytes;
pStable->sversion++;
mInfo("msg:%p, app:%p stable %s, start to modify column %s len to %d", pMsg, pMsg->rpcMsg.ahandle, pStable->info.tableId,
name, schema->bytes);
@ -1501,9 +1502,10 @@ static int32_t mnodeChangeSuperTableTag(SMnodeMsg *pMsg) {
}
// update
SSchema *schema = (SSchema *) (pStable->schema + col);
SSchema *schema = (SSchema *) (pStable->schema + col + pStable->numOfColumns);
ASSERT(schema->type == TSDB_DATA_TYPE_BINARY || schema->type == TSDB_DATA_TYPE_NCHAR);
schema->bytes = pAlter->schema[0].bytes;
pStable->tversion++;
mInfo("msg:%p, app:%p stable %s, start to modify tag len %s to %d", pMsg, pMsg->rpcMsg.ahandle, pStable->info.tableId,
name, schema->bytes);

45
src/mnode/src/mnodeVgroup.c
View File

@ -60,6 +60,7 @@ static int32_t mnodeGetVgroupMeta(STableMetaMsg *pMeta, SShowObj *pShow, void *p
static int32_t mnodeRetrieveVgroups(SShowObj *pShow, char *data, int32_t rows, void *pConn);
static void mnodeProcessCreateVnodeRsp(SRpcMsg *rpcMsg);
static void mnodeProcessAlterVnodeRsp(SRpcMsg *rpcMsg);
static void mnodeProcessCompactVnodeRsp(SRpcMsg *rpcMsg);
static void mnodeProcessDropVnodeRsp(SRpcMsg *rpcMsg);
static int32_t mnodeProcessVnodeCfgMsg(SMnodeMsg *pMsg) ;
static void mnodeSendDropVgroupMsg(SVgObj *pVgroup, void *ahandle);
@ -236,6 +237,7 @@ int32_t mnodeInitVgroups() {
mnodeAddShowFreeIterHandle(TSDB_MGMT_TABLE_VGROUP, mnodeCancelGetNextVgroup);
mnodeAddPeerRspHandle(TSDB_MSG_TYPE_MD_CREATE_VNODE_RSP, mnodeProcessCreateVnodeRsp);
mnodeAddPeerRspHandle(TSDB_MSG_TYPE_MD_ALTER_VNODE_RSP, mnodeProcessAlterVnodeRsp);
mnodeAddPeerRspHandle(TSDB_MSG_TYPE_MD_COMPACT_VNODE_RSP, mnodeProcessCompactVnodeRsp);
mnodeAddPeerRspHandle(TSDB_MSG_TYPE_MD_DROP_VNODE_RSP, mnodeProcessDropVnodeRsp);
mnodeAddPeerMsgHandle(TSDB_MSG_TYPE_DM_CONFIG_VNODE, mnodeProcessVnodeCfgMsg);
@ -350,6 +352,7 @@ void mnodeUpdateVgroupStatus(SVgObj *pVgroup, SDnodeObj *pDnode, SVnodeLoad *pVl
pVgroup->pDb->dbCfgVersion, pVgroup->vgCfgVersion, pVgroup->numOfVnodes);
mnodeSendAlterVgroupMsg(pVgroup,NULL);
}
pVgroup->compact = pVload->compact;
}
static int32_t mnodeAllocVgroupIdPool(SVgObj *pInputVgroup) {
@ -717,6 +720,13 @@ static int32_t mnodeGetVgroupMeta(STableMetaMsg *pMeta, SShowObj *pShow, void *p
cols++;
}
pShow->bytes[cols] = 4;
pSchema[cols].type = TSDB_DATA_TYPE_INT;
strcpy(pSchema[cols].name, "compacting");
pSchema[cols].bytes = htons(pShow->bytes[cols]);
cols++;
pMeta->numOfColumns = htons(cols);
pShow->numOfColumns = cols;
@ -820,7 +830,11 @@ static int32_t mnodeRetrieveVgroups(SShowObj *pShow, char *data, int32_t rows, v
STR_WITH_MAXSIZE_TO_VARSTR(pWrite, role, pShow->bytes[cols]);
cols++;
}
pWrite = data + pShow->offset[cols] * rows + pShow->bytes[cols] * numOfRows;
*(int8_t *)pWrite = pVgroup->compact;
cols++;
mnodeDecVgroupRef(pVgroup);
numOfRows++;
}
@ -979,6 +993,7 @@ static SSyncVnodeMsg *mnodeBuildSyncVnodeMsg(int32_t vgId) {
return pSyncVnode;
}
static void mnodeSendSyncVnodeMsg(SVgObj *pVgroup, SRpcEpSet *epSet) {
SSyncVnodeMsg *pSyncVnode = mnodeBuildSyncVnodeMsg(pVgroup->vgId);
SRpcMsg rpcMsg = {
@ -991,6 +1006,18 @@ static void mnodeSendSyncVnodeMsg(SVgObj *pVgroup, SRpcEpSet *epSet) {
dnodeSendMsgToDnode(epSet, &rpcMsg);
}
static void mnodeSendCompactVnodeMsg(SVgObj *pVgroup, SRpcEpSet *epSet) {
SCompactVnodeMsg *pCompactVnode = mnodeBuildSyncVnodeMsg(pVgroup->vgId);
SRpcMsg rpcMsg = {
.ahandle = NULL,
.pCont = pCompactVnode,
.contLen = pCompactVnode ? sizeof(SCompactVnodeMsg) : 0,
.code = 0,
.msgType = TSDB_MSG_TYPE_MD_COMPACT_VNODE
};
dnodeSendMsgToDnode(epSet, &rpcMsg);
}
void mnodeSendSyncVgroupMsg(SVgObj *pVgroup) {
mDebug("vgId:%d, send sync all vnodes msg, numOfVnodes:%d db:%s", pVgroup->vgId, pVgroup->numOfVnodes,
@ -1004,6 +1031,17 @@ void mnodeSendSyncVgroupMsg(SVgObj *pVgroup) {
}
}
void mnodeSendCompactVgroupMsg(SVgObj *pVgroup) {
mDebug("vgId:%d, send compact all vnodes msg, numOfVnodes:%d db:%s", pVgroup->vgId, pVgroup->numOfVnodes, pVgroup->dbName);
for (int32_t i = 0; i < pVgroup->numOfVnodes; ++i) {
//if (pVgroup->vnodeGid[i].role != TAOS_SYNC_ROLE_SLAVE) continue; //TODO(yihaoDeng): compact slave or not ?
SRpcEpSet epSet = mnodeGetEpSetFromIp(pVgroup->vnodeGid[i].pDnode->dnodeEp);
mDebug("vgId:%d, index:%d, send compact vnode msg to dnode %s", pVgroup->vgId, i,
pVgroup->vnodeGid[i].pDnode->dnodeEp);
mnodeSendCompactVnodeMsg(pVgroup, &epSet);
}
}
static void mnodeSendCreateVnodeMsg(SVgObj *pVgroup, SRpcEpSet *epSet, void *ahandle) {
SCreateVnodeMsg *pCreate = mnodeBuildVnodeMsg(pVgroup);
SRpcMsg rpcMsg = {
@ -1050,6 +1088,9 @@ static void mnodeProcessAlterVnodeRsp(SRpcMsg *rpcMsg) {
dnodeSendRpcMWriteRsp(mnodeMsg, code);
}
}
static void mnodeProcessCompactVnodeRsp(SRpcMsg *rpcMsg) {
mDebug("compact vnode rsp received");
}
static void mnodeProcessCreateVnodeRsp(SRpcMsg *rpcMsg) {
if (rpcMsg->ahandle == NULL) return;
@ -1346,4 +1387,4 @@ int32_t mnodeCompactVgroups() {
mInfo("end to compact vgroups table...");
return 0;
}
}

7
src/os/tests/CMakeLists.txt
View File

@ -7,7 +7,12 @@ FIND_LIBRARY(LIB_GTEST_STATIC_DIR libgtest.a /usr/lib/ /usr/local/lib)
IF (HEADER_GTEST_INCLUDE_DIR AND LIB_GTEST_STATIC_DIR)
MESSAGE(STATUS "gTest library found, build unit test")
INCLUDE_DIRECTORIES(${HEADER_GTEST_INCLUDE_DIR})
# GoogleTest requires at least C++11
SET(CMAKE_CXX_STANDARD 11)
INCLUDE_DIRECTORIES(/usr/include /usr/local/include)
LINK_DIRECTORIES(/usr/lib /usr/local/lib)
AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)
ADD_EXECUTABLE(osTest ${SOURCE_LIST})

64
src/plugins/http/inc/httpParser.h
View File

@ -19,6 +19,70 @@
#define HTTP_MAX_URL 5 // http url stack size
#define HTTP_CODE_CONTINUE 100
#define HTTP_CODE_SWITCHING_PROTOCOL 101
#define HTTP_CODE_PROCESSING 102
#define HTTP_CODE_EARLY_HINTS 103
#define HTTP_CODE_OK 200
#define HTTP_CODE_CREATED 201
#define HTTP_CODE_ACCEPTED 202
#define HTTP_CODE_NON_AUTHORITATIVE_INFO 203
#define HTTP_CODE_NO_CONTENT 204
#define HTTP_CODE_RESET_CONTENT 205
#define HTTP_CODE_PARTIAL_CONTENT 206
#define HTTP_CODE_MULTI_STATUS 207
#define HTTP_CODE_ALREADY_REPORTED 208
#define HTTP_CODE_IM_USED 226
#define HTTP_CODE_MULTIPLE_CHOICE 300
#define HTTP_CODE_MOVED_PERMANENTLY 301
#define HTTP_CODE_FOUND 302
#define HTTP_CODE_SEE_OTHER 303
#define HTTP_CODE_NOT_MODIFIED 304
#define HTTP_CODE_USE_PROXY 305
#define HTTP_CODE_UNUSED 306
#define HTTP_CODE_TEMPORARY_REDIRECT 307
#define HTTP_CODE_PERMANENT_REDIRECT 308
#define HTTP_CODE_BAD_REQUEST 400
#define HTTP_CODE_UNAUTHORIZED 401
#define HTTP_CODE_PAYMENT_REQUIRED 402
#define HTTP_CODE_FORBIDDEN 403
#define HTTP_CODE_NOT_FOUND 404
#define HTTP_CODE_METHOD_NOT_ALLOWED 405
#define HTTP_CODE_NOT_ACCEPTABLE 406
#define HTTP_CODE_PROXY_AUTH_REQUIRED 407
#define HTTP_CODE_REQUEST_TIMEOUT 408
#define HTTP_CODE_CONFLICT 409
#define HTTP_CODE_GONE 410
#define HTTP_CODE_LENGTH_REQUIRED 411
#define HTTP_CODE_PRECONDITION_FAILED 412
#define HTTP_CODE_PAYLOAD_TOO_LARGE 413
#define HTTP_CODE_URI_TOO_LARGE 414
#define HTTP_CODE_UNSUPPORTED_MEDIA_TYPE 415
#define HTTP_CODE_RANGE_NOT_SATISFIABLE 416
#define HTTP_CODE_EXPECTATION_FAILED 417
#define HTTP_CODE_IM_A_TEAPOT 418
#define HTTP_CODE_MISDIRECTED_REQUEST 421
#define HTTP_CODE_UNPROCESSABLE_ENTITY 422
#define HTTP_CODE_LOCKED 423
#define HTTP_CODE_FAILED_DEPENDENCY 424
#define HTTP_CODE_TOO_EARLY 425
#define HTTP_CODE_UPGRADE_REQUIRED 426
#define HTTP_CODE_PRECONDITION_REQUIRED 428
#define HTTP_CODE_TOO_MANY_REQUESTS 429
#define HTTP_CODE_REQ_HDR_FIELDS_TOO_LARGE 431
#define HTTP_CODE_UNAVAIL_4_LEGAL_REASONS 451
#define HTTP_CODE_INTERNAL_SERVER_ERROR 500
#define HTTP_CODE_NOT_IMPLEMENTED 501
#define HTTP_CODE_BAD_GATEWAY 502
#define HTTP_CODE_SERVICE_UNAVAILABLE 503
#define HTTP_CODE_GATEWAY_TIMEOUT 504
#define HTTP_CODE_HTTP_VER_NOT_SUPPORTED 505
#define HTTP_CODE_VARIANT_ALSO_NEGOTIATES 506
#define HTTP_CODE_INSUFFICIENT_STORAGE 507
#define HTTP_CODE_LOOP_DETECTED 508
#define HTTP_CODE_NOT_EXTENDED 510
#define HTTP_CODE_NETWORK_AUTH_REQUIRED 511
typedef enum HTTP_PARSER_STATE {
HTTP_PARSER_BEGIN,
HTTP_PARSER_REQUEST_OR_RESPONSE,

10
src/plugins/http/src/httpGcJson.c
View File

@ -133,6 +133,16 @@ bool gcBuildQueryJson(HttpContext *pContext, HttpSqlCmd *cmd, TAOS_RES *result,
int32_t len;
len = snprintf(target, HTTP_GC_TARGET_SIZE, "%s{", aliasBuffer);
for (int32_t i = dataFields + 1; i < num_fields; i++) {
if (row[i] == NULL) {
len += snprintf(target + len, HTTP_GC_TARGET_SIZE - len, "%s:nil", fields[i].name);
if (i < num_fields - 1) {
len += snprintf(target + len, HTTP_GC_TARGET_SIZE - len, ", ");
}
continue;
}
switch (fields[i].type) {
case TSDB_DATA_TYPE_BOOL:
case TSDB_DATA_TYPE_TINYINT:

194
src/plugins/http/src/httpParser.c
View File

@ -25,70 +25,70 @@
static void httpOnData(ehttp_gzip_t *gzip, void *arg, const char *buf, int32_t len);
static HttpStatus httpStatusCodes[] = {
{100, "Continue"},
{101, "Switching Protocol"},
{102, "Processing (WebDAV)"},
{103, "Early Hints"},
{200, "OK"},
{201, "Created"},
{202, "Accepted"},
{203, "Non-Authoritative Information"},
{204, "No Content"},
{205, "Reset Content"},
{206, "Partial Content"},
{207, "Multi-Status (WebDAV)"},
{208, "Already Reported (WebDAV)"},
{226, "IM Used (HTTP Delta encoding)"},
{300, "Multiple Choice"},
{301, "Moved Permanently"},
{302, "Found"},
{303, "See Other"},
{304, "Not Modified"},
{305, "Use Proxy"},
{306, "unused"},
{307, "Temporary Redirect"},
{308, "Permanent Redirect"},
{400, "Bad Request"},
{401, "Unauthorized"},
{402, "Payment Required"},
{403, "Forbidden"},
{404, "Not Found"},
{405, "Method Not Allowed"},
{406, "Not Acceptable"},
{407, "Proxy Authentication Required"},
{408, "Request Timeout"},
{409, "Conflict"},
{410, "Gone"},
{411, "Length Required"},
{412, "Precondition Failed"},
{413, "Payload Too Large"},
{414, "URI Too Long"},
{415, "Unsupported Media Type"},
{416, "Range Not Satisfiable"},
{417, "Expectation Failed"},
{418, "I'm a teapot"},
{421, "Misdirected Request"},
{422, "Unprocessable Entity (WebDAV)"},
{423, "Locked (WebDAV)"},
{424, "Failed Dependency (WebDAV)"},
{425, "Too Early"},
{426, "Upgrade Required"},
{428, "Precondition Required"},
{429, "Too Many Requests"},
{431, "Request Header Fields Too Large"},
{451, "Unavailable For Legal Reasons"},
{500, "Internal Server Error"},
{501, "Not Implemented"},
{502, "Bad Gateway"},
{503, "Service Unavailable"},
{504, "Gateway Timeout"},
{505, "HTTP Version Not Supported"},
{506, "Variant Also Negotiates"},
{507, "Insufficient Storage"},
{508, "Loop Detected (WebDAV)"},
{510, "Not Extended"},
{511, "Network Authentication Required"},
{0, NULL}
{HTTP_CODE_CONTINUE, "Continue"},
{HTTP_CODE_SWITCHING_PROTOCOL, "Switching Protocol"},
{HTTP_CODE_PROCESSING, "Processing (WebDAV)"},
{HTTP_CODE_EARLY_HINTS, "Early Hints"},
{HTTP_CODE_OK, "OK"},
{HTTP_CODE_CREATED, "Created"},
{HTTP_CODE_ACCEPTED, "Accepted"},
{HTTP_CODE_NON_AUTHORITATIVE_INFO, "Non-Authoritative Information"},
{HTTP_CODE_NO_CONTENT, "No Content"},
{HTTP_CODE_RESET_CONTENT, "Reset Content"},
{HTTP_CODE_PARTIAL_CONTENT, "Partial Content"},
{HTTP_CODE_MULTI_STATUS, "Multi-Status (WebDAV)"},
{HTTP_CODE_ALREADY_REPORTED, "Already Reported (WebDAV)"},
{HTTP_CODE_IM_USED, "IM Used (HTTP Delta encoding)"},
{HTTP_CODE_MULTIPLE_CHOICE, "Multiple Choice"},
{HTTP_CODE_MOVED_PERMANENTLY, "Moved Permanently"},
{HTTP_CODE_FOUND, "Found"},
{HTTP_CODE_SEE_OTHER, "See Other"},
{HTTP_CODE_NOT_MODIFIED, "Not Modified"},
{HTTP_CODE_USE_PROXY, "Use Proxy"},
{HTTP_CODE_UNUSED, "unused"},
{HTTP_CODE_TEMPORARY_REDIRECT, "Temporary Redirect"},
{HTTP_CODE_PERMANENT_REDIRECT, "Permanent Redirect"},
{HTTP_CODE_BAD_REQUEST, "Bad Request"},
{HTTP_CODE_UNAUTHORIZED, "Unauthorized"},
{HTTP_CODE_PAYMENT_REQUIRED, "Payment Required"},
{HTTP_CODE_FORBIDDEN, "Forbidden"},
{HTTP_CODE_NOT_FOUND, "Not Found"},
{HTTP_CODE_METHOD_NOT_ALLOWED, "Method Not Allowed"},
{HTTP_CODE_NOT_ACCEPTABLE, "Not Acceptable"},
{HTTP_CODE_PROXY_AUTH_REQUIRED, "Proxy Authentication Required"},
{HTTP_CODE_REQUEST_TIMEOUT, "Request Timeout"},
{HTTP_CODE_CONFLICT, "Conflict"},
{HTTP_CODE_GONE, "Gone"},
{HTTP_CODE_LENGTH_REQUIRED, "Length Required"},
{HTTP_CODE_PRECONDITION_FAILED, "Precondition Failed"},
{HTTP_CODE_PAYLOAD_TOO_LARGE, "Payload Too Large"},
{HTTP_CODE_URI_TOO_LARGE, "URI Too Long"},
{HTTP_CODE_UNSUPPORTED_MEDIA_TYPE, "Unsupported Media Type"},
{HTTP_CODE_RANGE_NOT_SATISFIABLE, "Range Not Satisfiable"},
{HTTP_CODE_EXPECTATION_FAILED, "Expectation Failed"},
{HTTP_CODE_IM_A_TEAPOT, "I'm a teapot"},
{HTTP_CODE_MISDIRECTED_REQUEST, "Misdirected Request"},
{HTTP_CODE_UNPROCESSABLE_ENTITY, "Unprocessable Entity (WebDAV)"},
{HTTP_CODE_LOCKED, "Locked (WebDAV)"},
{HTTP_CODE_FAILED_DEPENDENCY, "Failed Dependency (WebDAV)"},
{HTTP_CODE_TOO_EARLY, "Too Early"},
{HTTP_CODE_UPGRADE_REQUIRED, "Upgrade Required"},
{HTTP_CODE_PRECONDITION_REQUIRED, "Precondition Required"},
{HTTP_CODE_TOO_MANY_REQUESTS, "Too Many Requests"},
{HTTP_CODE_REQ_HDR_FIELDS_TOO_LARGE,"Request Header Fields Too Large"},
{HTTP_CODE_UNAVAIL_4_LEGAL_REASONS, "Unavailable For Legal Reasons"},
{HTTP_CODE_INTERNAL_SERVER_ERROR, "Internal Server Error"},
{HTTP_CODE_NOT_IMPLEMENTED, "Not Implemented"},
{HTTP_CODE_BAD_GATEWAY, "Bad Gateway"},
{HTTP_CODE_SERVICE_UNAVAILABLE, "Service Unavailable"},
{HTTP_CODE_GATEWAY_TIMEOUT, "Gateway Timeout"},
{HTTP_CODE_HTTP_VER_NOT_SUPPORTED, "HTTP Version Not Supported"},
{HTTP_CODE_VARIANT_ALSO_NEGOTIATES, "Variant Also Negotiates"},
{HTTP_CODE_INSUFFICIENT_STORAGE, "Insufficient Storage"},
{HTTP_CODE_LOOP_DETECTED, "Loop Detected (WebDAV)"},
{HTTP_CODE_NOT_EXTENDED, "Not Extended"},
{HTTP_CODE_NETWORK_AUTH_REQUIRED, "Network Authentication Required"},
{0, NULL}
};
char *httpGetStatusDesc(int32_t statusCode) {
@ -552,7 +552,7 @@ static int32_t httpParserOnBegin(HttpParser *parser, HTTP_PARSER_STATE state, co
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
break;
}
httpPopStack(parser);
@ -561,7 +561,7 @@ static int32_t httpParserOnBegin(HttpParser *parser, HTTP_PARSER_STATE state, co
}
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
} while (0);
return ok;
}
@ -599,7 +599,7 @@ static int32_t httpParserOnRquestOrResponse(HttpParser *parser, HTTP_PARSER_STAT
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
} while (0);
return ok;
}
@ -612,7 +612,7 @@ static int32_t httpParserOnMethod(HttpParser *parser, HTTP_PARSER_STATE state, c
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
break;
}
break;
@ -621,7 +621,7 @@ static int32_t httpParserOnMethod(HttpParser *parser, HTTP_PARSER_STATE state, c
if (!parser->method) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_METHOD_FAILED);
break;
} else {
httpTrace("context:%p, fd:%d, httpMethod:%s", pContext, pContext->fd, parser->method);
@ -641,7 +641,7 @@ static int32_t httpParserOnTarget(HttpParser *parser, HTTP_PARSER_STATE state, c
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_TARGET_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_TARGET_FAILED);
break;
}
break;
@ -650,7 +650,7 @@ static int32_t httpParserOnTarget(HttpParser *parser, HTTP_PARSER_STATE state, c
if (!parser->target) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_TARGET_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_TARGET_FAILED);
break;
}
httpClearString(&parser->str);
@ -670,13 +670,13 @@ static int32_t httpParserOnVersion(HttpParser *parser, HTTP_PARSER_STATE state,
if (prefix[parser->str.pos] != c) {
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
break;
}
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
break;
}
break;
@ -685,14 +685,14 @@ static int32_t httpParserOnVersion(HttpParser *parser, HTTP_PARSER_STATE state,
if (c != '0' && c != '1' && c != '2') {
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
break;
}
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
break;
}
@ -709,7 +709,7 @@ static int32_t httpParserOnVersion(HttpParser *parser, HTTP_PARSER_STATE state,
if (!parser->version) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_VERSION_FAILED);
break;
}
@ -739,7 +739,7 @@ static int32_t httpParserOnSp(HttpParser *parser, HTTP_PARSER_STATE state, const
}
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_SP_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_SP_FAILED);
} while (0);
return ok;
}
@ -752,7 +752,7 @@ static int32_t httpParserOnStatusCode(HttpParser *parser, HTTP_PARSER_STATE stat
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_STATUS_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_STATUS_FAILED);
break;
}
if (parser->str.pos < 3) break;
@ -764,7 +764,7 @@ static int32_t httpParserOnStatusCode(HttpParser *parser, HTTP_PARSER_STATE stat
}
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_STATUS_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_STATUS_FAILED);
} while (0);
return ok;
}
@ -778,7 +778,7 @@ static int32_t httpParserOnReasonPhrase(HttpParser *parser, HTTP_PARSER_STATE st
if (!parser->reasonPhrase) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_PHRASE_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_PHRASE_FAILED);
break;
}
ok = httpOnStatusLine(parser, parser->statusCode, parser->reasonPhrase);
@ -790,7 +790,7 @@ static int32_t httpParserOnReasonPhrase(HttpParser *parser, HTTP_PARSER_STATE st
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_PHRASE_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_PHRASE_FAILED);
break;
}
} while (0);
@ -802,7 +802,7 @@ static int32_t httpParserPostProcess(HttpParser *parser) {
if (parser->gzip) {
if (ehttp_gzip_finish(parser->gzip)) {
httpError("context:%p, fd:%d, gzip failed", pContext, pContext->fd);
httpOnError(parser, 507, TSDB_CODE_HTTP_FINISH_GZIP_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_FINISH_GZIP_FAILED);
return -1;
}
}
@ -819,13 +819,13 @@ static int32_t httpParserOnCrlf(HttpParser *parser, HTTP_PARSER_STATE state, con
if (s[parser->str.pos] != c) {
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_CRLF_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_CRLF_FAILED);
break;
}
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_CRLF_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_CRLF_FAILED);
break;
}
if (parser->str.pos == len) {
@ -862,7 +862,7 @@ static int32_t httpParserOnHeader(HttpParser *parser, HTTP_PARSER_STATE state, c
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_HEADER_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_HEADER_FAILED);
break;
}
httpPushStack(parser, HTTP_PARSER_CRLF);
@ -873,7 +873,7 @@ static int32_t httpParserOnHeader(HttpParser *parser, HTTP_PARSER_STATE state, c
}
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_HEADER_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_HEADER_FAILED);
} while (0);
return ok;
}
@ -886,7 +886,7 @@ static int32_t httpParserOnHeaderKey(HttpParser *parser, HTTP_PARSER_STATE state
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED);
break;
}
break;
@ -896,7 +896,7 @@ static int32_t httpParserOnHeaderKey(HttpParser *parser, HTTP_PARSER_STATE state
if (!parser->key) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED);
break;
}
httpClearString(&parser->str);
@ -905,7 +905,7 @@ static int32_t httpParserOnHeaderKey(HttpParser *parser, HTTP_PARSER_STATE state
}
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED);
} while (0);
return ok;
}
@ -919,7 +919,7 @@ static int32_t httpParserOnHeaderVal(HttpParser *parser, HTTP_PARSER_STATE state
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
parser->parseCode = TSDB_CODE_HTTP_PARSE_HEADER_VAL_FAILED;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_HEADER_VAL_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_HEADER_VAL_FAILED);
break;
}
break;
@ -948,7 +948,7 @@ static int32_t httpParserOnChunkSize(HttpParser *parser, HTTP_PARSER_STATE state
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED);
break;
}
break;
@ -982,7 +982,7 @@ static int32_t httpParserOnChunkSize(HttpParser *parser, HTTP_PARSER_STATE state
}
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 400, TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED);
httpOnError(parser, HTTP_CODE_BAD_REQUEST, TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED);
} while (0);
return ok;
}
@ -994,7 +994,7 @@ static int32_t httpParserOnChunk(HttpParser *parser, HTTP_PARSER_STATE state, co
if (httpAppendString(&parser->str, &c, 1)) {
httpError("context:%p, fd:%d, parser state:%d, char:[%c]%02x, oom", pContext, pContext->fd, state, c, c);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_CHUNK_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_CHUNK_FAILED);
break;
}
++parser->receivedSize;
@ -1005,7 +1005,7 @@ static int32_t httpParserOnChunk(HttpParser *parser, HTTP_PARSER_STATE state, co
if (ehttp_gzip_write(parser->gzip, parser->str.str, parser->str.pos)) {
httpError("context:%p, fd:%d, gzip failed", pContext, pContext->fd);
ok = -1;
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_CHUNK_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_CHUNK_FAILED);
break;
}
} else {
@ -1027,7 +1027,7 @@ static int32_t httpParserOnEnd(HttpParser *parser, HTTP_PARSER_STATE state, cons
do {
ok = -1;
httpError("context:%p, fd:%d, parser state:%d, unexpected char:[%c]%02x", pContext, pContext->fd, state, c, c);
httpOnError(parser, 507, TSDB_CODE_HTTP_PARSE_END_FAILED);
httpOnError(parser, HTTP_CODE_INSUFFICIENT_STORAGE, TSDB_CODE_HTTP_PARSE_END_FAILED);
} while (0);
return ok;
}
@ -1104,7 +1104,7 @@ static int32_t httpParseChar(HttpParser *parser, const char c, int32_t *again) {
ok = -1;
httpError("context:%p, fd:%d, unknown parse state:%d", pContext, pContext->fd, state);
httpOnError(parser, 500, TSDB_CODE_HTTP_PARSE_INVALID_STATE);
httpOnError(parser, HTTP_CODE_INTERNAL_SERVER_ERROR, TSDB_CODE_HTTP_PARSE_INVALID_STATE);
} while (0);
if (ok == -1) {
@ -1115,7 +1115,7 @@ static int32_t httpParseChar(HttpParser *parser, const char c, int32_t *again) {
if (ok == -2) {
ok = -1;
httpError("context:%p, fd:%d, failed to parse, invalid state", pContext, pContext->fd);
httpOnError(parser, 500, TSDB_CODE_HTTP_PARSE_ERROR_STATE);
httpOnError(parser, HTTP_CODE_INTERNAL_SERVER_ERROR, TSDB_CODE_HTTP_PARSE_ERROR_STATE);
}
return ok;

78
src/plugins/http/src/httpResp.c
View File

@ -67,81 +67,81 @@ static void httpSendErrorRespImp(HttpContext *pContext, int32_t httpCode, char *
}
void httpSendErrorResp(HttpContext *pContext, int32_t errNo) {
int32_t httpCode = 500;
int32_t httpCode = HTTP_CODE_INTERNAL_SERVER_ERROR;
if (errNo == TSDB_CODE_SUCCESS)
httpCode = 200;
httpCode = HTTP_CODE_OK;
else if (errNo == TSDB_CODE_HTTP_SERVER_OFFLINE)
httpCode = 404;
httpCode = HTTP_CODE_NOT_FOUND;
else if (errNo == TSDB_CODE_HTTP_UNSUPPORT_URL)
httpCode = 404;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_INVALID_URL)
httpCode = 404;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_NO_ENOUGH_MEMORY)
httpCode = 507;
httpCode = HTTP_CODE_INSUFFICIENT_STORAGE;
else if (errNo == TSDB_CODE_HTTP_REQUSET_TOO_BIG)
httpCode = 413;
httpCode = HTTP_CODE_PAYLOAD_TOO_LARGE;
else if (errNo == TSDB_CODE_HTTP_NO_AUTH_INFO)
httpCode = 401;
httpCode = HTTP_CODE_UNAUTHORIZED;
else if (errNo == TSDB_CODE_HTTP_NO_MSG_INPUT)
httpCode = 400;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_NO_SQL_INPUT)
httpCode = 400;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_NO_EXEC_USEDB)
httpCode = 400;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_SESSION_FULL)
httpCode = 421;
httpCode = HTTP_CODE_INTERNAL_SERVER_ERROR;
else if (errNo == TSDB_CODE_HTTP_GEN_TAOSD_TOKEN_ERR)
httpCode = 507;
httpCode = HTTP_CODE_INTERNAL_SERVER_ERROR;
else if (errNo == TSDB_CODE_HTTP_INVALID_MULTI_REQUEST)
httpCode = 400;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_CREATE_GZIP_FAILED)
httpCode = 507;
httpCode = HTTP_CODE_INTERNAL_SERVER_ERROR;
else if (errNo == TSDB_CODE_HTTP_FINISH_GZIP_FAILED)
httpCode = 507;
httpCode = HTTP_CODE_INTERNAL_SERVER_ERROR;
else if (errNo == TSDB_CODE_HTTP_INVALID_VERSION)
httpCode = 406;
httpCode = HTTP_CODE_HTTP_VER_NOT_SUPPORTED;
else if (errNo == TSDB_CODE_HTTP_INVALID_CONTENT_LENGTH)
httpCode = 406;
httpCode = HTTP_CODE_LENGTH_REQUIRED;
else if (errNo == TSDB_CODE_HTTP_INVALID_AUTH_TYPE)
httpCode = 406;
httpCode = HTTP_CODE_UNAUTHORIZED;
else if (errNo == TSDB_CODE_HTTP_INVALID_AUTH_FORMAT)
httpCode = 406;
httpCode = HTTP_CODE_UNAUTHORIZED;
else if (errNo == TSDB_CODE_HTTP_INVALID_BASIC_AUTH)
httpCode = 406;
httpCode = HTTP_CODE_UNAUTHORIZED;
else if (errNo == TSDB_CODE_HTTP_INVALID_TAOSD_AUTH)
httpCode = 406;
httpCode = HTTP_CODE_UNAUTHORIZED;
else if (errNo == TSDB_CODE_HTTP_PARSE_METHOD_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_TARGET_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_VERSION_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_SP_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_STATUS_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_PHRASE_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_CRLF_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_HEADER_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_HEADER_KEY_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_HEADER_VAL_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_CHUNK_SIZE_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_CHUNK_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_END_FAILED)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_INVALID_STATE)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else if (errNo == TSDB_CODE_HTTP_PARSE_ERROR_STATE)
httpCode = 406;
httpCode = HTTP_CODE_BAD_REQUEST;
else
httpCode = 400;
httpCode = HTTP_CODE_BAD_REQUEST;
if (pContext->parser && pContext->parser->httpCode != 0) {
httpCode = pContext->parser->httpCode;
@ -152,7 +152,7 @@ void httpSendErrorResp(HttpContext *pContext, int32_t errNo) {
}
void httpSendTaosdInvalidSqlErrorResp(HttpContext *pContext, char *errMsg) {
int32_t httpCode = 400;
int32_t httpCode = HTTP_CODE_BAD_REQUEST;
char temp[512] = {0};
int32_t len = sprintf(temp, "invalid SQL: %s", errMsg);

2
src/query/inc/qSqlparser.h
View File

@ -312,6 +312,8 @@ void setCreateUserSql(SSqlInfo *pInfo, SStrToken *pName, SStrToken *pPasswd);
void setKillSql(SSqlInfo *pInfo, int32_t type, SStrToken *ip);
void setAlterUserSql(SSqlInfo *pInfo, int16_t type, SStrToken *pName, SStrToken* pPwd, SStrToken *pPrivilege);
void setCompactVnodeSql(SSqlInfo *pInfo, int32_t type, SArray *pParam);
void setDefaultCreateDbOption(SCreateDbInfo *pDBInfo);
void setDefaultCreateTopicOption(SCreateDbInfo *pDBInfo);

4
src/query/inc/sql.y
View File

@ -174,6 +174,10 @@ cmd ::= ALTER TOPIC ids(X) alter_topic_optr(Y). { SStrToken t = {0}; setCreateD
cmd ::= ALTER ACCOUNT ids(X) acct_optr(Z). { setCreateAcctSql(pInfo, TSDB_SQL_ALTER_ACCT, &X, NULL, &Z);}
cmd ::= ALTER ACCOUNT ids(X) PASS ids(Y) acct_optr(Z). { setCreateAcctSql(pInfo, TSDB_SQL_ALTER_ACCT, &X, &Y, &Z);}
////////////////////////////// COMPACT STATEMENT //////////////////////////////////////////////
cmd ::= COMPACT VNODES IN LP exprlist(Y) RP. { setCompactVnodeSql(pInfo, TSDB_SQL_COMPACT_VNODE, Y);}
// An IDENTIFIER can be a generic identifier, or one of several keywords.
// Any non-standard keyword can also be an identifier.
// And "ids" is an identifer-or-string.

12
src/query/src/qAggMain.c
View File

@ -3662,7 +3662,7 @@ static void diff_function(SQLFunctionCtx *pCtx) {
if (pCtx->param[1].nType != INITIAL_VALUE_NOT_ASSIGNED) { // initial value is not set yet
*pOutput = (int32_t)(pData[i] - pCtx->param[1].i64); // direct previous may be null
*pTimestamp = tsList[i];
*pTimestamp = (tsList != NULL)? tsList[i]:0;
pOutput += 1;
pTimestamp += 1;
}
@ -3684,7 +3684,7 @@ static void diff_function(SQLFunctionCtx *pCtx) {
if (pCtx->param[1].nType != INITIAL_VALUE_NOT_ASSIGNED) { // initial value is not set yet
*pOutput = pData[i] - pCtx->param[1].i64; // direct previous may be null
*pTimestamp = tsList[i];
*pTimestamp = (tsList != NULL)? tsList[i]:0;
pOutput += 1;
pTimestamp += 1;
}
@ -3706,7 +3706,7 @@ static void diff_function(SQLFunctionCtx *pCtx) {
if (pCtx->param[1].nType != INITIAL_VALUE_NOT_ASSIGNED) { // initial value is not set yet
*pOutput = pData[i] - pCtx->param[1].dKey; // direct previous may be null
*pTimestamp = tsList[i];
*pTimestamp = (tsList != NULL)? tsList[i]:0;
pOutput += 1;
pTimestamp += 1;
}
@ -3728,7 +3728,7 @@ static void diff_function(SQLFunctionCtx *pCtx) {
if (pCtx->param[1].nType != INITIAL_VALUE_NOT_ASSIGNED) { // initial value is not set yet
*pOutput = (float)(pData[i] - pCtx->param[1].dKey); // direct previous may be null
*pTimestamp = tsList[i];
*pTimestamp = (tsList != NULL)? tsList[i]:0;
pOutput += 1;
pTimestamp += 1;
}
@ -3750,7 +3750,7 @@ static void diff_function(SQLFunctionCtx *pCtx) {
if (pCtx->param[1].nType != INITIAL_VALUE_NOT_ASSIGNED) { // initial value is not set yet
*pOutput = (int16_t)(pData[i] - pCtx->param[1].i64); // direct previous may be null
*pTimestamp = tsList[i];
*pTimestamp = (tsList != NULL)? tsList[i]:0;
pOutput += 1;
pTimestamp += 1;
}
@ -3773,7 +3773,7 @@ static void diff_function(SQLFunctionCtx *pCtx) {
if (pCtx->param[1].nType != INITIAL_VALUE_NOT_ASSIGNED) { // initial value is not set yet
*pOutput = (int8_t)(pData[i] - pCtx->param[1].i64); // direct previous may be null
*pTimestamp = tsList[i];
*pTimestamp = (tsList != NULL)? tsList[i]:0;
pOutput += 1;
pTimestamp += 1;
}

27
src/query/src/qExecutor.c
View File

@ -2282,7 +2282,7 @@ static bool doFilterByBlockStatistics(SQueryRuntimeEnv* pRuntimeEnv, SDataStatis
if (pDataStatis == NULL || pQueryAttr->numOfFilterCols == 0) {
return true;
}
bool ret = true;
for (int32_t k = 0; k < pQueryAttr->numOfFilterCols; ++k) {
SSingleColumnFilterInfo *pFilterInfo = &pQueryAttr->pFilterInfo[k];
@ -2319,26 +2319,34 @@ static bool doFilterByBlockStatistics(SQueryRuntimeEnv* pRuntimeEnv, SDataStatis
}
SDataStatis* pDataBlockst = &pDataStatis[index];
if (pFilterInfo->info.type == TSDB_DATA_TYPE_FLOAT) {
float minval = (float)(*(double *)(&pDataBlockst->min));
float maxval = (float)(*(double *)(&pDataBlockst->max));
for (int32_t i = 0; i < pFilterInfo->numOfFilters; ++i) {
if (pFilterInfo->pFilters[i].fp(&pFilterInfo->pFilters[i], (char *)&minval, (char *)&maxval, TSDB_DATA_TYPE_FLOAT)) {
return true;
if (pFilterInfo->pFilters[i].filterInfo.lowerRelOptr == TSDB_RELATION_IN) {
continue;
}
ret &= pFilterInfo->pFilters[i].fp(&pFilterInfo->pFilters[i], (char *)&minval, (char *)&maxval, TSDB_DATA_TYPE_FLOAT);
if (ret == false) {
return false;
}
}
} else {
for (int32_t i = 0; i < pFilterInfo->numOfFilters; ++i) {
if (pFilterInfo->pFilters[i].fp(&pFilterInfo->pFilters[i], (char *)&pDataBlockst->min, (char *)&pDataBlockst->max, pFilterInfo->info.type)) {
return true;
if (pFilterInfo->pFilters[i].filterInfo.lowerRelOptr == TSDB_RELATION_IN) {
continue;
}
ret &= pFilterInfo->pFilters[i].fp(&pFilterInfo->pFilters[i], (char *)&pDataBlockst->min, (char *)&pDataBlockst->max, pFilterInfo->info.type);
if (ret == false) {
return false;
}
}
}
}
return false;
return ret;
}
static bool overlapWithTimeWindow(SQueryAttr* pQueryAttr, SDataBlockInfo* pBlockInfo) {
@ -7519,7 +7527,8 @@ static void doSetTagValueToResultBuf(char* output, const char* val, int16_t type
if (IS_VAR_DATA_TYPE(type)) {
// Binary data overflows for sort of unknown reasons. Let trim the overflow data
if (varDataTLen(val) > bytes) {
int32_t len = bytes - VARSTR_HEADER_SIZE; // remain available space
int32_t maxLen = bytes - VARSTR_HEADER_SIZE;
int32_t len = (varDataLen(val) > maxLen)? maxLen:varDataLen(val);
memcpy(varDataVal(output), varDataVal(val), len);
varDataSetLen(output, len);
} else {

2
src/query/src/qFilterfunc.c
View File

@ -262,7 +262,7 @@ bool inOperator(SColumnFilterElem *pFilter, const char* minval, const char* maxv
return NULL != taosHashGet((SHashObj *)pFilter->q, (char *)&minv, sizeof(minv));
}
return true;
} else if (type == TSDB_DATA_TYPE_DOUBLE || type == TSDB_DATA_TYPE_DOUBLE) {
} else if (type == TSDB_DATA_TYPE_DOUBLE || type == TSDB_DATA_TYPE_FLOAT) {
double v;
GET_TYPED_DATA(v, double, type, minval);
return NULL != taosHashGet((SHashObj *)pFilter->q, (char *)&v, sizeof(v));

4
src/query/src/qSqlParser.c
View File

@ -1067,6 +1067,10 @@ void setCreateAcctSql(SSqlInfo *pInfo, int32_t type, SStrToken *pName, SStrToken
pInfo->pMiscInfo->user.passwd = *pPwd;
}
}
void setCompactVnodeSql(SSqlInfo *pInfo, int32_t type, SArray *pParam) {
pInfo->type = type;
pInfo->list = pParam;
}
void setCreateUserSql(SSqlInfo *pInfo, SStrToken *pName, SStrToken *pPasswd) {
pInfo->type = TSDB_SQL_CREATE_USER;

29
src/query/src/qTsbuf.c
View File

@ -812,34 +812,31 @@ int32_t tsBufMerge(STSBuf* pDestBuf, const STSBuf* pSrcBuf) {
int64_t offset = getDataStartOffset();
int32_t size = (int32_t)pSrcBuf->fileSize - (int32_t)offset;
int64_t rc = taosFSendFile(pDestBuf->f, pSrcBuf->f, &offset, size);
int64_t written = taosFSendFile(pDestBuf->f, pSrcBuf->f, &offset, size);
if (rc == -1) {
// tscError("failed to merge tsBuf from:%s to %s, reason:%s\n", pSrcBuf->path, pDestBuf->path, strerror(errno));
return -1;
}
if (rc != size) {
// tscError("failed to merge tsBuf from:%s to %s, reason:%s\n", pSrcBuf->path, pDestBuf->path, strerror(errno));
if (written == -1 || written != size) {
return -1;
}
pDestBuf->numOfTotal += pSrcBuf->numOfTotal;
int32_t oldSize = pDestBuf->fileSize;
// file meta data may be cached, close and reopen the file for accurate file size.
fclose(pDestBuf->f);
pDestBuf->f = fopen(pDestBuf->path, "rb+");
if (pDestBuf->f == NULL) {
return -1;
}
struct stat fileStat;
if (fstat(fileno(pDestBuf->f), &fileStat) != 0) {
return -1;
}
pDestBuf->fileSize = (uint32_t)fileStat.st_size;
assert(pDestBuf->fileSize == oldSize + size);
// tscDebug("tsBuf merge success, %p, path:%s, fd:%d, file size:%d, numOfGroups:%d, autoDelete:%d", pDestBuf,
// pDestBuf->path, fileno(pDestBuf->f), pDestBuf->fileSize, pDestBuf->numOfGroups, pDestBuf->autoDelete);
return 0;
}

3049
src/query/src/sql.c
View File

File diff suppressed because it is too large Load Diff

8
src/query/tests/CMakeLists.txt
View File

@ -7,11 +7,17 @@ FIND_LIBRARY(LIB_GTEST_STATIC_DIR libgtest.a /usr/lib/ /usr/local/lib)
IF (HEADER_GTEST_INCLUDE_DIR AND LIB_GTEST_STATIC_DIR)
MESSAGE(STATUS "gTest library found, build unit test")
# GoogleTest requires at least C++11
SET(CMAKE_CXX_STANDARD 11)
INCLUDE_DIRECTORIES(/usr/include /usr/local/include)
LINK_DIRECTORIES(/usr/lib /usr/local/lib)
INCLUDE_DIRECTORIES(${HEADER_GTEST_INCLUDE_DIR})
AUX_SOURCE_DIRECTORY(${CMAKE_CURRENT_SOURCE_DIR} SOURCE_LIST)
ADD_EXECUTABLE(queryTest ${SOURCE_LIST})
TARGET_LINK_LIBRARIES(queryTest taos query gtest pthread gcov)
TARGET_LINK_LIBRARIES(queryTest taos query gtest pthread)
ENDIF()
SET_SOURCE_FILES_PROPERTIES(./astTest.cpp PROPERTIES COMPILE_FLAGS -w)

693
src/tsdb/src/tsdbCompact.c
View File

@ -14,8 +14,6 @@
*/
#include "tsdbint.h"
#ifndef _TSDB_PLUGINS
typedef struct {
STable * pTable;
SBlockIdx * pBlkIdx;
@ -120,408 +118,407 @@ static int tsdbCompactMeta(STsdbRepo *pRepo) {
return 0;
}
static int tsdbCompactTSData(STsdbRepo *pRepo) {
SCompactH compactH;
SDFileSet *pSet = NULL;
static int tsdbCompactTSData(STsdbRepo *pRepo) {
SCompactH compactH;
SDFileSet *pSet = NULL;
tsdbDebug("vgId:%d start to compact TS data", REPO_ID(pRepo));
tsdbDebug("vgId:%d start to compact TS data", REPO_ID(pRepo));
// If no file, just return 0;
if (taosArrayGetSize(REPO_FS(pRepo)->cstatus->df) <= 0) {
tsdbDebug("vgId:%d no TS data file to compact, compact over", REPO_ID(pRepo));
// If no file, just return 0;
if (taosArrayGetSize(REPO_FS(pRepo)->cstatus->df) <= 0) {
tsdbDebug("vgId:%d no TS data file to compact, compact over", REPO_ID(pRepo));
return 0;
}
if (tsdbInitCompactH(&compactH, pRepo) < 0) {
return -1;
}
while ((pSet = tsdbFSIterNext(&(compactH.fsIter)))) {
// Remove those expired files
if (pSet->fid < compactH.rtn.minFid) {
tsdbInfo("vgId:%d FSET %d on level %d disk id %d expires, remove it", REPO_ID(pRepo), pSet->fid,
TSDB_FSET_LEVEL(pSet), TSDB_FSET_ID(pSet));
continue;
}
if (TSDB_FSET_LEVEL(pSet) == TFS_MAX_LEVEL) {
tsdbDebug("vgId:%d FSET %d on level %d, should not compact", REPO_ID(pRepo), pSet->fid, TFS_MAX_LEVEL);
tsdbUpdateDFileSet(REPO_FS(pRepo), pSet);
continue;
}
if (tsdbCompactFSet(&compactH, pSet) < 0) {
tsdbDestroyCompactH(&compactH);
tsdbError("vgId:%d failed to compact FSET %d since %s", REPO_ID(pRepo), pSet->fid, tstrerror(terrno));
return -1;
}
}
tsdbDestroyCompactH(&compactH);
tsdbDebug("vgId:%d compact TS data over", REPO_ID(pRepo));
return 0;
}
if (tsdbInitCompactH(&compactH, pRepo) < 0) {
return -1;
}
static int tsdbCompactFSet(SCompactH *pComph, SDFileSet *pSet) {
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
SDiskID did;
while ((pSet = tsdbFSIterNext(&(compactH.fsIter)))) {
// Remove those expired files
if (pSet->fid < compactH.rtn.minFid) {
tsdbInfo("vgId:%d FSET %d on level %d disk id %d expires, remove it", REPO_ID(pRepo), pSet->fid,
TSDB_FSET_LEVEL(pSet), TSDB_FSET_ID(pSet));
continue;
}
tsdbDebug("vgId:%d start to compact FSET %d on level %d id %d", REPO_ID(pRepo), pSet->fid, TSDB_FSET_LEVEL(pSet),
TSDB_FSET_ID(pSet));
if (TSDB_FSET_LEVEL(pSet) == TFS_MAX_LEVEL) {
tsdbDebug("vgId:%d FSET %d on level %d, should not compact", REPO_ID(pRepo), pSet->fid, TFS_MAX_LEVEL);
tsdbUpdateDFileSet(REPO_FS(pRepo), pSet);
continue;
}
if (tsdbCompactFSet(&compactH, pSet) < 0) {
tsdbDestroyCompactH(&compactH);
tsdbError("vgId:%d failed to compact FSET %d since %s", REPO_ID(pRepo), pSet->fid, tstrerror(terrno));
return -1;
}
}
tsdbDestroyCompactH(&compactH);
tsdbDebug("vgId:%d compact TS data over", REPO_ID(pRepo));
return 0;
}
static int tsdbCompactFSet(SCompactH *pComph, SDFileSet *pSet) {
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
SDiskID did;
tsdbDebug("vgId:%d start to compact FSET %d on level %d id %d", REPO_ID(pRepo), pSet->fid, TSDB_FSET_LEVEL(pSet),
TSDB_FSET_ID(pSet));
if (tsdbCompactFSetInit(pComph, pSet) < 0) {
return -1;
}
if (!tsdbShouldCompact(pComph)) {
tsdbDebug("vgId:%d no need to compact FSET %d", REPO_ID(pRepo), pSet->fid);
if (tsdbApplyRtnOnFSet(TSDB_COMPACT_REPO(pComph), pSet, &(pComph->rtn)) < 0) {
tsdbCompactFSetEnd(pComph);
return -1;
}
} else {
// Create new fset as compacted fset
tfsAllocDisk(tsdbGetFidLevel(pSet->fid, &(pComph->rtn)), &(did.level), &(did.id));
if (did.level == TFS_UNDECIDED_LEVEL) {
terrno = TSDB_CODE_TDB_NO_AVAIL_DISK;
tsdbError("vgId:%d failed to compact FSET %d since %s", REPO_ID(pRepo), pSet->fid, tstrerror(terrno));
tsdbCompactFSetEnd(pComph);
if (tsdbCompactFSetInit(pComph, pSet) < 0) {
return -1;
}
tsdbInitDFileSet(TSDB_COMPACT_WSET(pComph), did, REPO_ID(pRepo), TSDB_FSET_FID(pSet),
FS_TXN_VERSION(REPO_FS(pRepo)));
if (tsdbCreateDFileSet(TSDB_COMPACT_WSET(pComph), true) < 0) {
tsdbError("vgId:%d failed to compact FSET %d since %s", REPO_ID(pRepo), pSet->fid, tstrerror(terrno));
tsdbCompactFSetEnd(pComph);
return -1;
}
if (!tsdbShouldCompact(pComph)) {
tsdbDebug("vgId:%d no need to compact FSET %d", REPO_ID(pRepo), pSet->fid);
if (tsdbApplyRtnOnFSet(TSDB_COMPACT_REPO(pComph), pSet, &(pComph->rtn)) < 0) {
tsdbCompactFSetEnd(pComph);
return -1;
}
} else {
// Create new fset as compacted fset
tfsAllocDisk(tsdbGetFidLevel(pSet->fid, &(pComph->rtn)), &(did.level), &(did.id));
if (did.level == TFS_UNDECIDED_LEVEL) {
terrno = TSDB_CODE_TDB_NO_AVAIL_DISK;
tsdbError("vgId:%d failed to compact FSET %d since %s", REPO_ID(pRepo), pSet->fid, tstrerror(terrno));
tsdbCompactFSetEnd(pComph);
return -1;
}
tsdbInitDFileSet(TSDB_COMPACT_WSET(pComph), did, REPO_ID(pRepo), TSDB_FSET_FID(pSet),
FS_TXN_VERSION(REPO_FS(pRepo)));
if (tsdbCreateDFileSet(TSDB_COMPACT_WSET(pComph), true) < 0) {
tsdbError("vgId:%d failed to compact FSET %d since %s", REPO_ID(pRepo), pSet->fid, tstrerror(terrno));
tsdbCompactFSetEnd(pComph);
return -1;
}
if (tsdbCompactFSetImpl(pComph) < 0) {
tsdbCloseDFileSet(TSDB_COMPACT_WSET(pComph));
tsdbRemoveDFileSet(TSDB_COMPACT_WSET(pComph));
tsdbCompactFSetEnd(pComph);
return -1;
}
if (tsdbCompactFSetImpl(pComph) < 0) {
tsdbCloseDFileSet(TSDB_COMPACT_WSET(pComph));
tsdbRemoveDFileSet(TSDB_COMPACT_WSET(pComph));
tsdbCompactFSetEnd(pComph);
tsdbUpdateDFileSet(REPO_FS(pRepo), TSDB_COMPACT_WSET(pComph));
tsdbDebug("vgId:%d FSET %d compact over", REPO_ID(pRepo), pSet->fid);
}
tsdbCompactFSetEnd(pComph);
return 0;
}
static bool tsdbShouldCompact(SCompactH *pComph) {
STsdbRepo * pRepo = TSDB_COMPACT_REPO(pComph);
STsdbCfg * pCfg = REPO_CFG(pRepo);
SReadH * pReadh = &(pComph->readh);
STableCompactH *pTh;
SBlock * pBlock;
int defaultRows = TSDB_DEFAULT_BLOCK_ROWS(pCfg->maxRowsPerFileBlock);
SDFile * pDataF = TSDB_READ_DATA_FILE(pReadh);
SDFile * pLastF = TSDB_READ_LAST_FILE(pReadh);
int tblocks = 0; // total blocks
int nSubBlocks = 0; // # of blocks with sub-blocks
int nSmallBlocks = 0; // # of blocks with rows < defaultRows
int64_t tsize = 0;
for (size_t i = 0; i < taosArrayGetSize(pComph->tbArray); i++) {
pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, i);
if (pTh->pTable == NULL || pTh->pBlkIdx == NULL) continue;
for (size_t bidx = 0; bidx < pTh->pBlkIdx->numOfBlocks; bidx++) {
tblocks++;
pBlock = pTh->pInfo->blocks + bidx;
if (pBlock->numOfRows < defaultRows) {
nSmallBlocks++;
}
if (pBlock->numOfSubBlocks > 1) {
nSubBlocks++;
for (int k = 0; k < pBlock->numOfSubBlocks; k++) {
SBlock *iBlock = ((SBlock *)POINTER_SHIFT(pTh->pInfo, pBlock->offset)) + k;
tsize = tsize + iBlock->len;
}
} else if (pBlock->numOfSubBlocks == 1) {
tsize += pBlock->len;
} else {
ASSERT(0);
}
}
}
return (((nSubBlocks * 1.0 / tblocks) > 0.33) || ((nSmallBlocks * 1.0 / tblocks) > 0.33) ||
(tsize * 1.0 / (pDataF->info.size + pLastF->info.size - 2 * TSDB_FILE_HEAD_SIZE) < 0.85));
}
static int tsdbInitCompactH(SCompactH *pComph, STsdbRepo *pRepo) {
STsdbCfg *pCfg = REPO_CFG(pRepo);
memset(pComph, 0, sizeof(*pComph));
TSDB_FSET_SET_CLOSED(TSDB_COMPACT_WSET(pComph));
tsdbGetRtnSnap(pRepo, &(pComph->rtn));
tsdbFSIterInit(&(pComph->fsIter), REPO_FS(pRepo), TSDB_FS_ITER_FORWARD);
if (tsdbInitReadH(&(pComph->readh), pRepo) < 0) {
return -1;
}
if (tsdbInitCompTbArray(pComph) < 0) {
tsdbDestroyCompactH(pComph);
return -1;
}
pComph->aBlkIdx = taosArrayInit(1024, sizeof(SBlockIdx));
if (pComph->aBlkIdx == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbDestroyCompactH(pComph);
return -1;
}
pComph->aSupBlk = taosArrayInit(1024, sizeof(SBlock));
if (pComph->aSupBlk == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbDestroyCompactH(pComph);
return -1;
}
pComph->pDataCols = tdNewDataCols(0, 0, pCfg->maxRowsPerFileBlock);
if (pComph->pDataCols == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbDestroyCompactH(pComph);
return -1;
}
return 0;
}
static void tsdbDestroyCompactH(SCompactH *pComph) {
pComph->pDataCols = tdFreeDataCols(pComph->pDataCols);
pComph->aSupBlk = taosArrayDestroy(pComph->aSupBlk);
pComph->aBlkIdx = taosArrayDestroy(pComph->aBlkIdx);
tsdbDestroyCompTbArray(pComph);
tsdbDestroyReadH(&(pComph->readh));
tsdbCloseDFileSet(TSDB_COMPACT_WSET(pComph));
tsdbUpdateDFileSet(REPO_FS(pRepo), TSDB_COMPACT_WSET(pComph));
tsdbDebug("vgId:%d FSET %d compact over", REPO_ID(pRepo), pSet->fid);
}
tsdbCompactFSetEnd(pComph);
return 0;
}
static int tsdbInitCompTbArray(SCompactH *pComph) { // Init pComp->tbArray
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
STsdbMeta *pMeta = pRepo->tsdbMeta;
static bool tsdbShouldCompact(SCompactH *pComph) {
STsdbRepo * pRepo = TSDB_COMPACT_REPO(pComph);
STsdbCfg * pCfg = REPO_CFG(pRepo);
SReadH * pReadh = &(pComph->readh);
STableCompactH *pTh;
SBlock * pBlock;
int defaultRows = TSDB_DEFAULT_BLOCK_ROWS(pCfg->maxRowsPerFileBlock);
SDFile * pDataF = TSDB_READ_DATA_FILE(pReadh);
SDFile * pLastF = TSDB_READ_LAST_FILE(pReadh);
if (tsdbRLockRepoMeta(pRepo) < 0) return -1;
int tblocks = 0; // total blocks
int nSubBlocks = 0; // # of blocks with sub-blocks
int nSmallBlocks = 0; // # of blocks with rows < defaultRows
int64_t tsize = 0;
for (size_t i = 0; i < taosArrayGetSize(pComph->tbArray); i++) {
pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, i);
if (pTh->pTable == NULL || pTh->pBlkIdx == NULL) continue;
for (size_t bidx = 0; bidx < pTh->pBlkIdx->numOfBlocks; bidx++) {
tblocks++;
pBlock = pTh->pInfo->blocks + bidx;
if (pBlock->numOfRows < defaultRows) {
nSmallBlocks++;
}
if (pBlock->numOfSubBlocks > 1) {
nSubBlocks++;
for (int k = 0; k < pBlock->numOfSubBlocks; k++) {
SBlock *iBlock = ((SBlock *)POINTER_SHIFT(pTh->pInfo, pBlock->offset)) + k;
tsize = tsize + iBlock->len;
}
} else if (pBlock->numOfSubBlocks == 1) {
tsize += pBlock->len;
} else {
ASSERT(0);
}
}
}
return (((nSubBlocks * 1.0 / tblocks) > 0.33) || ((nSmallBlocks * 1.0 / tblocks) > 0.33) ||
(tsize * 1.0 / (pDataF->info.size + pLastF->info.size - 2 * TSDB_FILE_HEAD_SIZE) < 0.85));
}
static int tsdbInitCompactH(SCompactH *pComph, STsdbRepo *pRepo) {
STsdbCfg *pCfg = REPO_CFG(pRepo);
memset(pComph, 0, sizeof(*pComph));
TSDB_FSET_SET_CLOSED(TSDB_COMPACT_WSET(pComph));
tsdbGetRtnSnap(pRepo, &(pComph->rtn));
tsdbFSIterInit(&(pComph->fsIter), REPO_FS(pRepo), TSDB_FS_ITER_FORWARD);
if (tsdbInitReadH(&(pComph->readh), pRepo) < 0) {
return -1;
}
if (tsdbInitCompTbArray(pComph) < 0) {
tsdbDestroyCompactH(pComph);
return -1;
}
pComph->aBlkIdx = taosArrayInit(1024, sizeof(SBlockIdx));
if (pComph->aBlkIdx == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbDestroyCompactH(pComph);
return -1;
}
pComph->aSupBlk = taosArrayInit(1024, sizeof(SBlock));
if (pComph->aSupBlk == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbDestroyCompactH(pComph);
return -1;
}
pComph->pDataCols = tdNewDataCols(0, 0, pCfg->maxRowsPerFileBlock);
if (pComph->pDataCols == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbDestroyCompactH(pComph);
return -1;
}
return 0;
}
static void tsdbDestroyCompactH(SCompactH *pComph) {
pComph->pDataCols = tdFreeDataCols(pComph->pDataCols);
pComph->aSupBlk = taosArrayDestroy(pComph->aSupBlk);
pComph->aBlkIdx = taosArrayDestroy(pComph->aBlkIdx);
tsdbDestroyCompTbArray(pComph);
tsdbDestroyReadH(&(pComph->readh));
tsdbCloseDFileSet(TSDB_COMPACT_WSET(pComph));
}
static int tsdbInitCompTbArray(SCompactH *pComph) { // Init pComp->tbArray
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
STsdbMeta *pMeta = pRepo->tsdbMeta;
if (tsdbRLockRepoMeta(pRepo) < 0) return -1;
pComph->tbArray = taosArrayInit(pMeta->maxTables, sizeof(STableCompactH));
if (pComph->tbArray == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbUnlockRepoMeta(pRepo);
return -1;
}
// Note here must start from 0
for (int i = 0; i < pMeta->maxTables; i++) {
STableCompactH ch = {0};
if (pMeta->tables[i] != NULL) {
tsdbRefTable(pMeta->tables[i]);
ch.pTable = pMeta->tables[i];
}
if (taosArrayPush(pComph->tbArray, &ch) == NULL) {
pComph->tbArray = taosArrayInit(pMeta->maxTables, sizeof(STableCompactH));
if (pComph->tbArray == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbUnlockRepoMeta(pRepo);
return -1;
}
}
if (tsdbUnlockRepoMeta(pRepo) < 0) return -1;
return 0;
}
// Note here must start from 0
for (int i = 0; i < pMeta->maxTables; i++) {
STableCompactH ch = {0};
if (pMeta->tables[i] != NULL) {
tsdbRefTable(pMeta->tables[i]);
ch.pTable = pMeta->tables[i];
}
static void tsdbDestroyCompTbArray(SCompactH *pComph) {
STableCompactH *pTh;
if (pComph->tbArray == NULL) return;
for (size_t i = 0; i < taosArrayGetSize(pComph->tbArray); i++) {
pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, i);
if (pTh->pTable) {
tsdbUnRefTable(pTh->pTable);
if (taosArrayPush(pComph->tbArray, &ch) == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
tsdbUnlockRepoMeta(pRepo);
return -1;
}
}
pTh->pInfo = taosTZfree(pTh->pInfo);
if (tsdbUnlockRepoMeta(pRepo) < 0) return -1;
return 0;
}
pComph->tbArray = taosArrayDestroy(pComph->tbArray);
}
static void tsdbDestroyCompTbArray(SCompactH *pComph) {
STableCompactH *pTh;
static int tsdbCacheFSetIndex(SCompactH *pComph) {
SReadH *pReadH = &(pComph->readh);
if (pComph->tbArray == NULL) return;
if (tsdbLoadBlockIdx(pReadH) < 0) {
return -1;
for (size_t i = 0; i < taosArrayGetSize(pComph->tbArray); i++) {
pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, i);
if (pTh->pTable) {
tsdbUnRefTable(pTh->pTable);
}
pTh->pInfo = taosTZfree(pTh->pInfo);
}
pComph->tbArray = taosArrayDestroy(pComph->tbArray);
}
for (int tid = 1; tid < taosArrayGetSize(pComph->tbArray); tid++) {
STableCompactH *pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, tid);
pTh->pBlkIdx = NULL;
static int tsdbCacheFSetIndex(SCompactH *pComph) {
SReadH *pReadH = &(pComph->readh);
if (pTh->pTable == NULL) continue;
if (tsdbSetReadTable(pReadH, pTh->pTable) < 0) {
if (tsdbLoadBlockIdx(pReadH) < 0) {
return -1;
}
if (pReadH->pBlkIdx == NULL) continue;
pTh->bindex = *(pReadH->pBlkIdx);
pTh->pBlkIdx = &(pTh->bindex);
for (int tid = 1; tid < taosArrayGetSize(pComph->tbArray); tid++) {
STableCompactH *pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, tid);
pTh->pBlkIdx = NULL;
if (tsdbMakeRoom((void **)(&(pTh->pInfo)), pTh->pBlkIdx->len) < 0) {
return -1;
}
if (tsdbLoadBlockInfo(pReadH, (void *)(pTh->pInfo)) < 0) {
return -1;
}
}
return 0;
}
static int tsdbCompactFSetInit(SCompactH *pComph, SDFileSet *pSet) {
taosArrayClear(pComph->aBlkIdx);
taosArrayClear(pComph->aSupBlk);
if (tsdbSetAndOpenReadFSet(&(pComph->readh), pSet) < 0) {
return -1;
}
if (tsdbCacheFSetIndex(pComph) < 0) {
tsdbCloseAndUnsetFSet(&(pComph->readh));
return -1;
}
return 0;
}
static void tsdbCompactFSetEnd(SCompactH *pComph) { tsdbCloseAndUnsetFSet(&(pComph->readh)); }
static int tsdbCompactFSetImpl(SCompactH *pComph) {
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
STsdbCfg * pCfg = REPO_CFG(pRepo);
SReadH * pReadh = &(pComph->readh);
SBlockIdx blkIdx;
void ** ppBuf = &(TSDB_COMPACT_BUF(pComph));
void ** ppCBuf = &(TSDB_COMPACT_COMP_BUF(pComph));
int defaultRows = TSDB_DEFAULT_BLOCK_ROWS(pCfg->maxRowsPerFileBlock);
taosArrayClear(pComph->aBlkIdx);
for (int tid = 1; tid < taosArrayGetSize(pComph->tbArray); tid++) {
STableCompactH *pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, tid);
STSchema * pSchema;
if (pTh->pTable == NULL || pTh->pBlkIdx == NULL) continue;
pSchema = tsdbGetTableSchemaImpl(pTh->pTable, true, true, -1);
taosArrayClear(pComph->aSupBlk);
if ((tdInitDataCols(pComph->pDataCols, pSchema) < 0) || (tdInitDataCols(pReadh->pDCols[0], pSchema) < 0) ||
(tdInitDataCols(pReadh->pDCols[1], pSchema) < 0)) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
return -1;
}
tdFreeSchema(pSchema);
// Loop to compact each block data
for (int i = 0; i < pTh->pBlkIdx->numOfBlocks; i++) {
SBlock *pBlock = pTh->pInfo->blocks + i;
// Load the block data
if (tsdbLoadBlockData(pReadh, pBlock, pTh->pInfo) < 0) {
if (pTh->pTable == NULL) continue;
if (tsdbSetReadTable(pReadH, pTh->pTable) < 0) {
return -1;
}
// Merge pComph->pDataCols and pReadh->pDCols[0] and write data to file
if (pComph->pDataCols->numOfRows == 0 && pBlock->numOfRows >= defaultRows) {
if (tsdbWriteBlockToRightFile(pComph, pTh->pTable, pReadh->pDCols[0], ppBuf, ppCBuf) < 0) {
return -1;
}
} else {
int ridx = 0;
if (pReadH->pBlkIdx == NULL) continue;
pTh->bindex = *(pReadH->pBlkIdx);
pTh->pBlkIdx = &(pTh->bindex);
while (true) {
if (pReadh->pDCols[0]->numOfRows - ridx == 0) break;
int rowsToMerge = MIN(pReadh->pDCols[0]->numOfRows - ridx, defaultRows - pComph->pDataCols->numOfRows);
if (tsdbMakeRoom((void **)(&(pTh->pInfo)), pTh->pBlkIdx->len) < 0) {
return -1;
}
tdMergeDataCols(pComph->pDataCols, pReadh->pDCols[0], rowsToMerge, &ridx);
if (pComph->pDataCols->numOfRows < defaultRows) {
break;
}
if (tsdbWriteBlockToRightFile(pComph, pTh->pTable, pComph->pDataCols, ppBuf, ppCBuf) < 0) {
return -1;
}
tdResetDataCols(pComph->pDataCols);
}
if (tsdbLoadBlockInfo(pReadH, (void *)(pTh->pInfo)) < 0) {
return -1;
}
}
if (pComph->pDataCols->numOfRows > 0 &&
tsdbWriteBlockToRightFile(pComph, pTh->pTable, pComph->pDataCols, ppBuf, ppCBuf) < 0) {
return 0;
}
static int tsdbCompactFSetInit(SCompactH *pComph, SDFileSet *pSet) {
taosArrayClear(pComph->aBlkIdx);
taosArrayClear(pComph->aSupBlk);
if (tsdbSetAndOpenReadFSet(&(pComph->readh), pSet) < 0) {
return -1;
}
if (tsdbWriteBlockInfoImpl(TSDB_COMPACT_HEAD_FILE(pComph), pTh->pTable, pComph->aSupBlk, NULL, ppBuf, &blkIdx) <
0) {
if (tsdbCacheFSetIndex(pComph) < 0) {
tsdbCloseAndUnsetFSet(&(pComph->readh));
return -1;
}
if ((blkIdx.numOfBlocks > 0) && (taosArrayPush(pComph->aBlkIdx, (void *)(&blkIdx)) == NULL)) {
return 0;
}
static void tsdbCompactFSetEnd(SCompactH *pComph) { tsdbCloseAndUnsetFSet(&(pComph->readh)); }
static int tsdbCompactFSetImpl(SCompactH *pComph) {
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
STsdbCfg * pCfg = REPO_CFG(pRepo);
SReadH * pReadh = &(pComph->readh);
SBlockIdx blkIdx;
void ** ppBuf = &(TSDB_COMPACT_BUF(pComph));
void ** ppCBuf = &(TSDB_COMPACT_COMP_BUF(pComph));
int defaultRows = TSDB_DEFAULT_BLOCK_ROWS(pCfg->maxRowsPerFileBlock);
taosArrayClear(pComph->aBlkIdx);
for (int tid = 1; tid < taosArrayGetSize(pComph->tbArray); tid++) {
STableCompactH *pTh = (STableCompactH *)taosArrayGet(pComph->tbArray, tid);
STSchema * pSchema;
if (pTh->pTable == NULL || pTh->pBlkIdx == NULL) continue;
pSchema = tsdbGetTableSchemaImpl(pTh->pTable, true, true, -1);
taosArrayClear(pComph->aSupBlk);
if ((tdInitDataCols(pComph->pDataCols, pSchema) < 0) || (tdInitDataCols(pReadh->pDCols[0], pSchema) < 0) ||
(tdInitDataCols(pReadh->pDCols[1], pSchema) < 0)) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
return -1;
}
tdFreeSchema(pSchema);
// Loop to compact each block data
for (int i = 0; i < pTh->pBlkIdx->numOfBlocks; i++) {
SBlock *pBlock = pTh->pInfo->blocks + i;
// Load the block data
if (tsdbLoadBlockData(pReadh, pBlock, pTh->pInfo) < 0) {
return -1;
}
// Merge pComph->pDataCols and pReadh->pDCols[0] and write data to file
if (pComph->pDataCols->numOfRows == 0 && pBlock->numOfRows >= defaultRows) {
if (tsdbWriteBlockToRightFile(pComph, pTh->pTable, pReadh->pDCols[0], ppBuf, ppCBuf) < 0) {
return -1;
}
} else {
int ridx = 0;
while (true) {
if (pReadh->pDCols[0]->numOfRows - ridx == 0) break;
int rowsToMerge = MIN(pReadh->pDCols[0]->numOfRows - ridx, defaultRows - pComph->pDataCols->numOfRows);
tdMergeDataCols(pComph->pDataCols, pReadh->pDCols[0], rowsToMerge, &ridx);
if (pComph->pDataCols->numOfRows < defaultRows) {
break;
}
if (tsdbWriteBlockToRightFile(pComph, pTh->pTable, pComph->pDataCols, ppBuf, ppCBuf) < 0) {
return -1;
}
tdResetDataCols(pComph->pDataCols);
}
}
}
if (pComph->pDataCols->numOfRows > 0 &&
tsdbWriteBlockToRightFile(pComph, pTh->pTable, pComph->pDataCols, ppBuf, ppCBuf) < 0) {
return -1;
}
if (tsdbWriteBlockInfoImpl(TSDB_COMPACT_HEAD_FILE(pComph), pTh->pTable, pComph->aSupBlk, NULL, ppBuf, &blkIdx) <
0) {
return -1;
}
if ((blkIdx.numOfBlocks > 0) && (taosArrayPush(pComph->aBlkIdx, (void *)(&blkIdx)) == NULL)) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
return -1;
}
}
if (tsdbWriteBlockIdx(TSDB_COMPACT_HEAD_FILE(pComph), pComph->aBlkIdx, ppBuf) < 0) {
return -1;
}
return 0;
}
static int tsdbWriteBlockToRightFile(SCompactH *pComph, STable *pTable, SDataCols *pDataCols, void **ppBuf,
void **ppCBuf) {
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
STsdbCfg * pCfg = REPO_CFG(pRepo);
SDFile * pDFile;
bool isLast;
SBlock block;
ASSERT(pDataCols->numOfRows > 0);
if (pDataCols->numOfRows < pCfg->minRowsPerFileBlock) {
pDFile = TSDB_COMPACT_LAST_FILE(pComph);
isLast = true;
} else {
pDFile = TSDB_COMPACT_DATA_FILE(pComph);
isLast = false;
}
if (tsdbWriteBlockImpl(pRepo, pTable, pDFile, pDataCols, &block, isLast, true, ppBuf, ppCBuf) < 0) {
return -1;
}
if (taosArrayPush(pComph->aSupBlk, (void *)(&block)) == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
return -1;
}
}
if (tsdbWriteBlockIdx(TSDB_COMPACT_HEAD_FILE(pComph), pComph->aBlkIdx, ppBuf) < 0) {
return -1;
}
return 0;
return 0;
}
static int tsdbWriteBlockToRightFile(SCompactH *pComph, STable *pTable, SDataCols *pDataCols, void **ppBuf,
void **ppCBuf) {
STsdbRepo *pRepo = TSDB_COMPACT_REPO(pComph);
STsdbCfg * pCfg = REPO_CFG(pRepo);
SDFile * pDFile;
bool isLast;
SBlock block;
ASSERT(pDataCols->numOfRows > 0);
if (pDataCols->numOfRows < pCfg->minRowsPerFileBlock) {
pDFile = TSDB_COMPACT_LAST_FILE(pComph);
isLast = true;
} else {
pDFile = TSDB_COMPACT_DATA_FILE(pComph);
isLast = false;
}
if (tsdbWriteBlockImpl(pRepo, pTable, pDFile, pDataCols, &block, isLast, true, ppBuf, ppCBuf) < 0) {
return -1;
}
if (taosArrayPush(pComph->aSupBlk, (void *)(&block)) == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
return -1;
}
return 0;
}
#endif

6
src/tsdb/src/tsdbMeta.c
View File

@ -148,7 +148,9 @@ int tsdbCreateTable(STsdbRepo *repo, STableCfg *pCfg) {
return 0;
_err:
tsdbFreeTable(super);
if (newSuper) {
tsdbFreeTable(super);
}
tsdbFreeTable(table);
return -1;
}
@ -211,7 +213,7 @@ void *tsdbGetTableTagVal(const void* pTable, int32_t colId, int16_t type, int16_
}
char *val = tdGetKVRowValOfCol(((STable*)pTable)->tagVal, colId);
assert(type == pCol->type && bytes == pCol->bytes);
assert(type == pCol->type && bytes >= pCol->bytes);
// if (val != NULL && IS_VAR_DATA_TYPE(type)) {
// assert(varDataLen(val) < pCol->bytes);

10
src/util/inc/hash.h
View File

@ -61,6 +61,7 @@ typedef struct SHashObj {
size_t size; // number of elements in hash table
_hash_fn_t hashFp; // hash function
_hash_free_fn_t freeFp; // hash node free callback function
_equal_fn_t equalFp; // equal function
SRWLatch lock; // read-write spin lock
SHashLockTypeE type; // lock type
@ -78,6 +79,15 @@ typedef struct SHashObj {
*/
SHashObj *taosHashInit(size_t capacity, _hash_fn_t fn, bool update, SHashLockTypeE type);
/**
* set equal func of the hash table
* @param pHashObj
* @param equalFp
* @return
*/
void taosHashSetEqualFp(SHashObj *pHashObj, _equal_fn_t fp);
/**
* return the size of hash table
* @param pHashObj

9
src/util/inc/hashfunc.h
View File

@ -20,6 +20,8 @@
typedef uint32_t (*_hash_fn_t)(const char *, uint32_t);
typedef int32_t (*_equal_fn_t)(const void *a, const void *b, size_t sz);
/**
* murmur hash algorithm
* @key usually string
@ -36,9 +38,14 @@ uint32_t MurmurHash3_32(const char *key, uint32_t len);
* @return
*/
uint32_t taosIntHash_32(const char *key, uint32_t len);
uint32_t taosIntHash_64(const char *key, uint32_t len);
int32_t taosFloatEqual(const void *a, const void *b, size_t sz);
int32_t taosDoubleEqual(const void *a,const void *b, size_t sz);
_hash_fn_t taosGetDefaultHashFunction(int32_t type);
_equal_fn_t taosGetDefaultEqualFunction(int32_t type);
#endif //TDENGINE_HASHUTIL_H

7
src/util/inc/tcompare.h
View File

@ -27,6 +27,13 @@ extern "C" {
#define TSDB_PATTERN_NOWILDCARDMATCH 2
#define TSDB_PATTERN_STRING_MAX_LEN 20
#define FLT_COMPAR_TOL_FACTOR 4
#define FLT_EQUAL(_x, _y) (fabs((_x) - (_y)) <= (FLT_COMPAR_TOL_FACTOR * FLT_EPSILON))
#define FLT_GREATER(_x, _y) (!FLT_EQUAL((_x), (_y)) && ((_x) > (_y)))
#define FLT_LESS(_x, _y) (!FLT_EQUAL((_x), (_y)) && ((_x) < (_y)))
#define FLT_GREATEREQUAL(_x, _y) (FLT_EQUAL((_x), (_y)) || ((_x) > (_y)))
#define FLT_LESSEQUAL(_x, _y) (FLT_EQUAL((_x), (_y)) || ((_x) < (_y)))
#define PATTERN_COMPARE_INFO_INITIALIZER { '%', '_' }
typedef struct SPatternCompareInfo {

20
src/util/src/hash.c
View File

@ -73,10 +73,10 @@ static FORCE_INLINE int32_t taosHashCapacity(int32_t length) {
return i;
}
static FORCE_INLINE SHashNode *doSearchInEntryList(SHashEntry *pe, const void *key, size_t keyLen, uint32_t hashVal) {
static FORCE_INLINE SHashNode *doSearchInEntryList(SHashObj *pHashObj, SHashEntry *pe, const void *key, size_t keyLen, uint32_t hashVal) {
SHashNode *pNode = pe->next;
while (pNode) {
if ((pNode->keyLen == keyLen) && (memcmp(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) && pNode->removed == 0) {
if ((pNode->keyLen == keyLen) && ((*(pHashObj->equalFp))(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) && pNode->removed == 0) {
assert(pNode->hashVal == hashVal);
break;
}
@ -165,8 +165,8 @@ SHashObj *taosHashInit(size_t capacity, _hash_fn_t fn, bool update, SHashLockTyp
// the max slots is not defined by user
pHashObj->capacity = taosHashCapacity((int32_t)capacity);
assert((pHashObj->capacity & (pHashObj->capacity - 1)) == 0);
pHashObj->hashFp = fn;
pHashObj->equalFp = memcmp;
pHashObj->hashFp = fn;
pHashObj->type = type;
pHashObj->enableUpdate = update;
@ -189,6 +189,12 @@ SHashObj *taosHashInit(size_t capacity, _hash_fn_t fn, bool update, SHashLockTyp
return pHashObj;
}
void taosHashSetEqualFp(SHashObj *pHashObj, _equal_fn_t fp) {
if (pHashObj != NULL && fp != NULL) {
pHashObj->equalFp = fp;
}
}
int32_t taosHashGetSize(const SHashObj *pHashObj) { return (int32_t)((pHashObj == NULL) ? 0 : pHashObj->size); }
int32_t taosHashPut(SHashObj *pHashObj, const void *key, size_t keyLen, void *data, size_t size) {
@ -223,7 +229,7 @@ int32_t taosHashPut(SHashObj *pHashObj, const void *key, size_t keyLen, void *da
SHashNode* prev = NULL;
while (pNode) {
if ((pNode->keyLen == keyLen) && (memcmp(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) && pNode->removed == 0) {
if ((pNode->keyLen == keyLen) && ((*(pHashObj->equalFp))(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) && pNode->removed == 0) {
assert(pNode->hashVal == hashVal);
break;
}
@ -306,7 +312,7 @@ void* taosHashGetClone(SHashObj *pHashObj, const void *key, size_t keyLen, void
assert(pe->next == NULL);
}
SHashNode *pNode = doSearchInEntryList(pe, key, keyLen, hashVal);
SHashNode *pNode = doSearchInEntryList(pHashObj, pe, key, keyLen, hashVal);
if (pNode != NULL) {
if (fp != NULL) {
fp(GET_HASH_NODE_DATA(pNode));
@ -362,7 +368,7 @@ int32_t taosHashRemoveWithData(SHashObj *pHashObj, const void *key, size_t keyLe
SHashNode *prevNode = NULL;
while (pNode) {
if ((pNode->keyLen == keyLen) && (memcmp(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) && pNode->removed == 0)
if ((pNode->keyLen == keyLen) && ((*(pHashObj->equalFp))(GET_HASH_NODE_KEY(pNode), key, keyLen) == 0) && pNode->removed == 0)
break;
prevNode = pNode;

18
src/util/src/tcompare.c
View File

@ -90,13 +90,10 @@ int32_t compareFloatVal(const void *pLeft, const void *pRight) {
if (isnan(p2)) {
return 1;
}
float ret = p1 - p2;
if (fabs(ret) < FLT_EPSILON) {
if (FLT_EQUAL(p1, p2)) {
return 0;
} else {
return ret > 0? 1 : -1;
}
}
return FLT_GREATER(p1, p2) ? 1: -1;
}
int32_t compareDoubleVal(const void *pLeft, const void *pRight) {
@ -114,13 +111,10 @@ int32_t compareDoubleVal(const void *pLeft, const void *pRight) {
if (isnan(p2)) {
return 1;
}
double ret = p1 - p2;
if (fabs(ret) < FLT_EPSILON) {
if (FLT_EQUAL(p1, p2)) {
return 0;
} else {
return ret > 0? 1 : -1;
}
}
return FLT_GREATER(p1, p2) ? 1: -1;
}
int32_t compareLenPrefixedStr(const void *pLeft, const void *pRight) {

70
src/util/src/thashutil.c
View File

@ -16,17 +16,20 @@
#include "os.h"
#include "hashfunc.h"
#include "tutil.h"
#include "tcompare.h"
#define ROTL32(x, r) ((x) << (r) | (x) >> (32u - (r)))
#define DLT (FLT_COMPAR_TOL_FACTOR * FLT_EPSILON)
#define BASE 1000
#define FMIX32(h) \
do { \
(h) ^= (h) >> 16; \
(h) *= 0x85ebca6b; \
(h) ^= (h) >> 13; \
(h) *= 0xc2b2ae35; \
(h) ^= (h) >> 16; \
} while (0)
(h) ^= (h) >> 16; } while (0)
uint32_t MurmurHash3_32(const char *key, uint32_t len) {
const uint8_t *data = (const uint8_t *)key;
@ -78,7 +81,38 @@ uint32_t MurmurHash3_32(const char *key, uint32_t len) {
uint32_t taosIntHash_32(const char *key, uint32_t UNUSED_PARAM(len)) { return *(uint32_t *)key; }
uint32_t taosIntHash_16(const char *key, uint32_t UNUSED_PARAM(len)) { return *(uint16_t *)key; }
uint32_t taosIntHash_8(const char *key, uint32_t UNUSED_PARAM(len)) { return *(uint8_t *)key; }
uint32_t taosFloatHash(const char *key, uint32_t UNUSED_PARAM(len)) {
float f = GET_FLOAT_VAL(key);
if (isnan(f)) {
return 0x7fc00000;
}
if (FLT_EQUAL(f, 0.0)) {
return 0;
}
if (fabs(f) < FLT_MAX/BASE - DLT) {
int t = (int)(round(BASE * (f + DLT)));
return (uint32_t)t;
} else {
return 0x7fc00000;
}
}
uint32_t taosDoubleHash(const char *key, uint32_t UNUSED_PARAM(len)) {
double f = GET_DOUBLE_VAL(key);
if (isnan(f)) {
return 0x7fc00000;
}
if (FLT_EQUAL(f, 0.0)) {
return 0;
}
if (fabs(f) < DBL_MAX/BASE - DLT) {
int t = (int)(round(BASE * (f + DLT)));
return (uint32_t)t;
} else {
return 0x7fc00000;
}
}
uint32_t taosIntHash_64(const char *key, uint32_t UNUSED_PARAM(len)) {
uint64_t val = *(uint64_t *)key;
@ -92,13 +126,35 @@ _hash_fn_t taosGetDefaultHashFunction(int32_t type) {
_hash_fn_t fn = NULL;
switch(type) {
case TSDB_DATA_TYPE_TIMESTAMP:
case TSDB_DATA_TYPE_BIGINT: fn = taosIntHash_64;break;
case TSDB_DATA_TYPE_BINARY: fn = MurmurHash3_32;break;
case TSDB_DATA_TYPE_INT: fn = taosIntHash_32; break;
case TSDB_DATA_TYPE_BIGINT: fn = taosIntHash_64;break;
case TSDB_DATA_TYPE_BINARY: fn = MurmurHash3_32;break;
case TSDB_DATA_TYPE_NCHAR: fn = MurmurHash3_32;break;
case TSDB_DATA_TYPE_INT: fn = taosIntHash_32; break;
case TSDB_DATA_TYPE_SMALLINT: fn = taosIntHash_16; break;
case TSDB_DATA_TYPE_TINYINT: fn = taosIntHash_8; break;
case TSDB_DATA_TYPE_TINYINT: fn = taosIntHash_8; break;
case TSDB_DATA_TYPE_FLOAT: fn = taosFloatHash; break;
case TSDB_DATA_TYPE_DOUBLE: fn = taosDoubleHash; break;
default: fn = taosIntHash_32;break;
}
return fn;
}
}
int32_t taosFloatEqual(const void *a, const void *b, size_t UNUSED_PARAM(sz)) {
return getComparFunc(TSDB_DATA_TYPE_FLOAT, -1)(a, b);
}
int32_t taosDoubleEqual(const void *a, const void *b, size_t UNUSED_PARAM(sz)) {
return getComparFunc(TSDB_DATA_TYPE_DOUBLE, -1)(a, b);
}
_equal_fn_t taosGetDefaultEqualFunction(int32_t type) {
_equal_fn_t fn = NULL;
switch (type) {
case TSDB_DATA_TYPE_FLOAT: fn = taosFloatEqual; break;
case TSDB_DATA_TYPE_DOUBLE: fn = taosDoubleEqual; break;
default: fn = memcmp; break;
}
return fn;
}

1
src/util/src/ttokenizer.c
View File

@ -219,6 +219,7 @@ static SKeyword keywordTable[] = {
{"PARTITIONS", TK_PARTITIONS},
{"TOPIC", TK_TOPIC},
{"TOPICS", TK_TOPICS},
{"COMPACT", TK_COMPACT},
{"MODIFY", TK_MODIFY}
};

14
src/vnode/src/vnodeMain.c
View File

@ -114,6 +114,7 @@ int32_t vnodeSync(int32_t vgId) {
return TSDB_CODE_SUCCESS;
}
int32_t vnodeDrop(int32_t vgId) {
SVnodeObj *pVnode = vnodeAcquire(vgId);
if (pVnode == NULL) {
@ -133,6 +134,19 @@ int32_t vnodeDrop(int32_t vgId) {
return TSDB_CODE_SUCCESS;
}
int32_t vnodeCompact(int32_t vgId) {
void *pVnode = vnodeAcquire(vgId);
if (pVnode != NULL) {
vDebug("vgId:%d, compact vnode msg is received", vgId);
//not care success or not
tsdbCompact(((SVnodeObj*)pVnode)->tsdb);
vnodeRelease(pVnode);
} else {
vInfo("vgId:%d, vnode not exist, can't compact it", vgId);
return TSDB_CODE_VND_INVALID_VGROUP_ID;
}
return TSDB_CODE_SUCCESS;
}
static int32_t vnodeAlterImp(SVnodeObj *pVnode, SCreateVnodeMsg *pVnodeCfg) {
STsdbCfg tsdbCfg = pVnode->tsdbCfg;

1
src/vnode/src/vnodeMgmt.c
View File

@ -148,6 +148,7 @@ static void vnodeBuildVloadMsg(SVnodeObj *pVnode, SStatusMsg *pStatus) {
pLoad->status = pVnode->status;
pLoad->role = pVnode->role;
pLoad->replica = pVnode->syncCfg.replica;
pLoad->compact = (pVnode->tsdb != NULL) && tsdbInCompact(pVnode->tsdb) ? 1 : 0;
}
int32_t vnodeGetVnodeList(int32_t vnodeList[], int32_t *numOfVnodes) {

2
src/wal/src/walWrite.c
View File

@ -346,7 +346,7 @@ static int32_t walRestoreWalFile(SWal *pWal, void *pVnode, FWalWrite writeFp, ch
}
#if defined(WAL_CHECKSUM_WHOLE)
if (pHead->sver == 0 && !walValidateChecksum(pHead)) {
if ((pHead->sver == 0 && !walValidateChecksum(pHead)) || pHead->sver < 0 || pHead->sver > 1) {
wError("vgId:%d, file:%s, wal head cksum is messed up, hver:%" PRIu64 " len:%d offset:%" PRId64, pWal->vgId, name,
pHead->version, pHead->len, offset);
code = walSkipCorruptedRecord(pWal, pHead, tfd, &offset);

20
tests/pytest/alter/alter_keep.py
View File

@ -15,6 +15,7 @@ import sys
from util.log import *
from util.cases import *
from util.sql import *
import time
class TDTestCase:
@ -129,9 +130,6 @@ class TDTestCase:
tdSql.prepare()
##TODO: need to wait for TD-4445 to implement the following
## tests
## preset the keep
tdSql.prepare()
@ -176,20 +174,22 @@ class TDTestCase:
tdSql.error('insert into tb values (now-15d, 10)')
tdSql.query('select * from tb')
tdSql.checkRows(rowNum)
tdLog.notice('testing keep will be altered if sudden change from small to big')
tdSql.execute('alter database db keep 14,14,14')
tdSql.execute('alter database db keep 15,15,15')
tdSql.execute('insert into tb values (now-15d, 10)')
tdSql.query('select * from tb')
tdSql.checkRows(rowNum + 1)
for i in range(30):
tdSql.execute('alter database db keep 14,14,14')
tdSql.execute('alter database db keep 16,16,16')
tdSql.execute('insert into tb values (now-15d, 10)')
tdSql.query('select * from tb')
rowNum += 1
tdSql.checkRows(rowNum )
tdLog.notice('testing keep will be altered if sudden change from big to small')
tdSql.execute('alter database db keep 16,16,16')
tdSql.execute('alter database db keep 14,14,14')
tdSql.error('insert into tb values (now-15d, 10)')
tdSql.query('select * from tb')
tdSql.checkRows(rowNum + 1)
tdSql.checkRows(rowNum)

28
tests/pytest/client/change_time_1_1.py
View File

@ -19,6 +19,7 @@ from util.pathFinding import *
from util.dnodes import tdDnodes
from datetime import datetime
import subprocess
import time
##TODO: this is now automatic, but not sure if this will run through jenkins
class TDTestCase:
@ -34,24 +35,36 @@ class TDTestCase:
## change system time to 2020/10/20
os.system('sudo timedatectl set-ntp off')
tdLog.sleep(10)
os.system('sudo timedatectl set-time 2020-10-20')
#run taosdemo to insert data. one row per second from 2020/10/11 to 2020/10/20
#11 data files should be generated
#vnode at TDinternal/community/sim/dnode1/data/vnode
os.system(f"{binPath}taosdemo -f tools/taosdemoAllTest/manual_change_time_1_1_A.json")
commandArray = ['ls', '-l', f'{TDenginePath}/sim/dnode1/data/vnode/vnode2/tsdb/data']
result = subprocess.run(commandArray, stdout=subprocess.PIPE).stdout.decode('utf-8')
print(result.count('data'))
try:
os.system(f"{binPath}taosdemo -f tools/taosdemoAllTest/manual_change_time_1_1_A.json")
commandArray = ['ls', '-l', f'{TDenginePath}/sim/dnode1/data/vnode/vnode2/tsdb/data']
result = subprocess.run(commandArray, stdout=subprocess.PIPE).stdout.decode('utf-8')
except BaseException:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
if result.count('data') != 11:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
tdLog.exit('wrong number of files')
else:
tdLog.debug("data file number correct")
#move 5 days ahead to 2020/10/25. 4 oldest files should be removed during the new write
#leaving 7 data files.
os.system ('timedatectl set-time 2020-10-25')
os.system(f"{binPath}taosdemo -f tools/taosdemoAllTest/manual_change_time_1_1_B.json")
try:
os.system ('timedatectl set-time 2020-10-25')
os.system(f"{binPath}taosdemo -f tools/taosdemoAllTest/manual_change_time_1_1_B.json")
except BaseException:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
commandArray = ['ls', '-l', f'{TDenginePath}/sim/dnode1/data/vnode/vnode2/tsdb/data']
result = subprocess.run(commandArray, stdout=subprocess.PIPE).stdout.decode('utf-8')
print(result.count('data'))
@ -61,9 +74,12 @@ class TDTestCase:
tdLog.debug("data file number correct")
tdSql.query('select first(ts) from stb_0')
tdSql.checkData(0,0,datetime(2020,10,14,8,0,0,0)) #check the last data in the database
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
def stop(self):
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
tdSql.close()
tdLog.success("alter block manual check finish")

46
tests/pytest/client/change_time_1_2.py
View File

@ -34,34 +34,52 @@ class TDTestCase:
## change system time to 2020/10/20
os.system ('timedatectl set-ntp off')
tdLog.sleep(10)
os.system ('timedatectl set-time 2020-10-20')
#run taosdemo to insert data. one row per second from 2020/10/11 to 2020/10/20
#11 data files should be generated
#vnode at TDinternal/community/sim/dnode1/data/vnode
os.system(f"{binPath}taosdemo -f tools/taosdemoAllTest/manual_change_time_1_1_A.json")
commandArray = ['ls', '-l', f'{TDenginePath}/sim/dnode1/data/vnode/vnode2/tsdb/data']
result = subprocess.run(commandArray, stdout=subprocess.PIPE).stdout.decode('utf-8')
print(result.count('data'))
try:
os.system(f"{binPath}taosdemo -f tools/taosdemoAllTest/manual_change_time_1_1_A.json")
commandArray = ['ls', '-l', f'{TDenginePath}/sim/dnode1/data/vnode/vnode2/tsdb/data']
result = subprocess.run(commandArray, stdout=subprocess.PIPE).stdout.decode('utf-8')
except BaseException:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
if result.count('data') != 11:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
tdLog.exit('wrong number of files')
else:
tdLog.debug("data file number correct")
tdSql.query('select first(ts) from stb_0') #check the last data in the database
tdSql.checkData(0,0,datetime(2020,10,11,0,0,0,0))
os.system ('timedatectl set-time 2020-10-25')
try:
tdSql.query('select first(ts) from stb_0') #check the last data in the database
tdSql.checkData(0,0,datetime(2020,10,11,0,0,0,0))
except BaseException:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
#moves 5 days ahead to 2020/10/25 and restart taosd
#4 oldest data file should be removed from tsdb/data
#7 data file should be found
#vnode at TDinternal/community/sim/dnode1/data/vnode
os.system ('timedatectl set-time 2020-10-25')
tdDnodes.stop(1)
tdDnodes.start(1)
tdSql.query('select first(ts) from stb_0')
tdSql.checkData(0,0,datetime(2020,10,14,8,0,0,0)) #check the last data in the database
try:
os.system ('timedatectl set-time 2020-10-25')
tdDnodes.stop(1)
tdDnodes.start(1)
tdSql.query('select first(ts) from stb_0')
tdSql.checkData(0,0,datetime(2020,10,14,8,0,0,0)) #check the last data in the database
except BaseException:
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
commandArray = ['ls', '-l', f'{TDenginePath}/sim/dnode1/data/vnode/vnode2/tsdb/data']
result = subprocess.run(commandArray, stdout=subprocess.PIPE).stdout.decode('utf-8')
print(result.count('data'))
@ -69,8 +87,8 @@ class TDTestCase:
tdLog.exit('wrong number of files')
else:
tdLog.debug("data file number correct")
os.system('sudo timedatectl set-ntp on')
tdLog.sleep(10)
def stop(self):
os.system('sudo timedatectl set-ntp on')

6
tests/pytest/fulltest.sh
View File

@ -21,6 +21,7 @@ python3 insert/retentionpolicy.py
python3 ./test.py -f insert/alterTableAndInsert.py
python3 ./test.py -f insert/insertIntoTwoTables.py
python3 ./test.py -f insert/before_1970.py
python3 ./test.py -f insert/special_character_show.py
python3 bug2265.py
python3 ./test.py -f insert/bug3654.py
python3 ./test.py -f insert/insertDynamicColBeforeVal.py
@ -256,8 +257,8 @@ python3 ./test.py -f client/client.py
python3 ./test.py -f client/version.py
python3 ./test.py -f client/alterDatabase.py
python3 ./test.py -f client/noConnectionErrorTest.py
python3 test.py -f client/change_time_1_1.py
python3 test.py -f client/change_time_1_2.py
# python3 test.py -f client/change_time_1_1.py
# python3 test.py -f client/change_time_1_2.py
# Misc
python3 testCompress.py
@ -318,6 +319,7 @@ python3 ./test.py -f account/account_create.py
python3 ./test.py -f alter/alter_table.py
python3 ./test.py -f query/queryGroupbySort.py
python3 ./test.py -f functions/function_stateWindow.py
python3 ./test.py -f functions/function_derivative.py
python3 ./test.py -f insert/unsignedInt.py
python3 ./test.py -f insert/unsignedBigint.py

139
tests/pytest/functions/function_derivative.py Normal file
View File

@ -0,0 +1,139 @@
###################################################################
# Copyright (c) 2016 by TAOS Technologies, Inc.
# All rights reserved.
#
# This file is proprietary and confidential to TAOS Technologies.
# No part of this file may be reproduced, stored, transmitted,
# disclosed or used in any form or by any means other than as
# expressly provided by the written permission from Jianhui Tao
#
###################################################################
# -*- coding: utf-8 -*-
import sys
import taos
from util.log import *
from util.cases import *
from util.sql import *
import numpy as np
class TDTestCase:
def init(self, conn, logSql):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor())
self.rowNum = 10
self.ts = 1537146000000
def insertAndCheckData(self):
types = ["tinyint", "tinyint unsigned", "smallint", "smallint unsigned", "int", "int unsigned", "bigint", "bigint unsigned", "float", "double", "bool", "binary(20)", "nchar(20)"]
for type in types:
print("============== create table using %s type ================" % type)
tdSql.execute("drop table if exists stb")
tdSql.execute("create table stb(ts timestamp, col %s) tags (id int)" % type)
tdSql.execute("create table tb1 using stb tags(1)")
tdSql.execute("create table tb2 using stb tags(2)")
if type == "tinyint" or type == "smallint" or type == "int" or type == "bigint":
tdSql.execute("insert into tb1 values(%d, 1)(%d, 11)(%d, 21)" % (self.ts, self.ts + 10000, self.ts + 20000))
tdSql.execute("insert into tb1 values(%d, -1)(%d, -11)(%d, -21)" % (self.ts + 30000, self.ts + 40000, self.ts + 50000))
tdSql.execute("insert into tb2 values(%d, 10)(%d, 20)(%d, 30)" % (self.ts + 60000, self.ts + 70000, self.ts + 80000))
tdSql.execute("insert into tb2 values(%d, -10)(%d, -20)(%d, -30)" % (self.ts + 90000, self.ts + 1000000, self.ts + 1100000))
tdSql.execute("insert into tb3 using stb tags(3) values(%d, 10)" % (self.ts + 1200000))
tdSql.query("select derivative(col, 1s, 1) from stb group by tbname")
tdSql.checkRows(4)
tdSql.query("select derivative(col, 10s, 1) from stb group by tbname")
tdSql.checkRows(4)
tdSql.query("select derivative(col, 10s, 0) from stb group by tbname")
tdSql.checkRows(10)
tdSql.error("select derivative(col, 10s, 0) from tb1 group by tbname")
tdSql.query("select derivative(col, 10s, 1) from tb1")
tdSql.checkRows(2)
tdSql.query("select derivative(col, 10s, 0) from tb1")
tdSql.checkRows(5)
tdSql.query("select derivative(col, 10s, 1) from tb2")
tdSql.checkRows(2)
tdSql.query("select derivative(col, 10s, 0) from tb2")
tdSql.checkRows(5)
tdSql.query("select derivative(col, 10s, 0) from tb3")
tdSql.checkRows(0)
elif type == "tinyint unsigned" or type == "smallint unsigned" or type == "int unsigned" or type == "bigint unsigned":
tdSql.execute("insert into tb1 values(%d, 1)(%d, 11)(%d, 21)" % (self.ts, self.ts + 10000, self.ts + 20000))
tdSql.execute("insert into tb2 values(%d, 10)(%d, 20)(%d, 30)" % (self.ts + 60000, self.ts + 70000, self.ts + 80000))
tdSql.error("select derivative(col, 1s, 1) from tb1")
tdSql.error("select derivative(col, 10s, 0) from tb1")
tdSql.error("select derivative(col, 999ms, 0) from tb1")
tdSql.error("select derivative(col, 1s, 1) from tb2")
tdSql.error("select derivative(col, 10s, 0) from tb2")
tdSql.error("select derivative(col, 999ms, 0) from tb2")
elif type == "float" or type == "double":
tdSql.execute("insert into tb1 values(%d, 1.0)(%d, 11.0)(%d, 21.0)" % (self.ts, self.ts + 10000, self.ts + 20000))
tdSql.execute("insert into tb2 values(%d, 3.0)(%d, 4.0)(%d, 5.0)" % (self.ts + 60000, self.ts + 70000, self.ts + 80000))
tdSql.query("select derivative(col, 10s, 1) from tb1")
tdSql.checkRows(2)
tdSql.query("select derivative(col, 10s, 0) from tb1")
tdSql.checkRows(2)
tdSql.query("select derivative(col, 10s, 1) from tb2")
tdSql.checkRows(2)
tdSql.query("select derivative(col, 10s, 0) from tb2")
tdSql.checkRows(2)
elif type == "bool":
tdSql.execute("insert into tb1 values(%d, true)(%d, false)(%d, true)" % (self.ts, self.ts + 10000, self.ts + 20000))
tdSql.execute("insert into tb2 values(%d, false)(%d, true)(%d, true)" % (self.ts + 60000, self.ts + 70000, self.ts + 80000))
tdSql.error("select derivative(col, 1s, 1) from tb1")
tdSql.error("select derivative(col, 10s, 0) from tb1")
tdSql.error("select derivative(col, 999ms, 0) from tb1")
tdSql.error("select derivative(col, 1s, 1) from tb2")
tdSql.error("select derivative(col, 10s, 0) from tb2")
tdSql.error("select derivative(col, 999ms, 0) from tb2")
else:
tdSql.execute("insert into tb1 values(%d, 'test01')(%d, 'test01')(%d, 'test01')" % (self.ts, self.ts + 10000, self.ts + 20000))
tdSql.execute("insert into tb2 values(%d, 'test01')(%d, 'test01')(%d, 'test01')" % (self.ts + 60000, self.ts + 70000, self.ts + 80000))
tdSql.error("select derivative(col, 1s, 1) from tb1")
tdSql.error("select derivative(col, 10s, 0) from tb1")
tdSql.error("select derivative(col, 999ms, 0) from tb1")
tdSql.error("select derivative(col, 1s, 1) from tb2")
tdSql.error("select derivative(col, 10s, 0) from tb2")
tdSql.error("select derivative(col, 999ms, 0) from tb2")
tdSql.error("select derivative(col, 10s, 1) from stb")
tdSql.error("select derivative(col, 10s, 1) from stb group by col")
tdSql.error("select derivative(col, 10s, 1) from stb group by id")
tdSql.error("select derivative(col, 999ms, 1) from stb group by id")
tdSql.error("select derivative(col, 10s, 2) from stb group by id")
def run(self):
tdSql.prepare()
self.insertAndCheckData()
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
tdCases.addWindows(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())

40
tests/pytest/insert/restfulInsert.py
View File

@ -40,12 +40,13 @@ class RestfulInsert:
if tableID + i >= self.numOfTables : break
name = 'beijing' if (tableID + i) % 2 == 0 else 'shanghai'
data = "create table if not exists %s.%s%d using %s.meters tags(%d, '%s')" % (self.dbname, self.tableNamePerfix, tableID + i, self.dbname, tableID + i, name)
response = requests.post(self.url, data, headers = self.header)
if response.status_code != 200:
print(response.content)
try:
response = requests.post(self.url, data, headers = self.header)
if response.status_code != 200:
print(response.content)
except Exception as e:
print(e)
def insertData(self, threadID):
print("thread %d started" % threadID)
tablesPerThread = int (self.numOfTables / self.numOfThreads)
@ -90,10 +91,16 @@ class RestfulInsert:
if len(data) > 1024*1024 :
print ('batch size is larger than 1M')
exit(-1)
response = requests.post(self.url, data, headers = self.header)
if response.status_code != 200:
print(response.content)
try:
startTime = time.time()
response = requests.post(self.url, data, headers = self.header)
endTime = time.time()
if response.status_code != 200:
print(response.content)
else:
print("inserted %d records, %d seconds" % (bloop, endTime - startTime))
except Exception as e:
print(e)
def insertUnlimitedData(self, threadID):
print("thread %d started" % threadID)
@ -119,10 +126,17 @@ class RestfulInsert:
else:
random.shuffle(values)
for k in range(len(values)):
data += values[k]
response = requests.post(self.url, data, headers = self.header)
if response.status_code != 200:
print(response.content)
data += values[k]
try:
startTime = time.time()
response = requests.post(self.url, data, headers = self.header)
endTime = time.time()
if response.status_code != 200:
print(response.content)
else:
print("inserted %d records, %d seconds" % (self.batchSize, endTime - startTime))
except Exception as e:
print(e)
def run(self):
data = "create database if not exists %s" % self.dbname

59
tests/pytest/insert/special_character_show.py Normal file
View File

@ -0,0 +1,59 @@
###################################################################
# Copyright (c) 2016 by TAOS Technologies, Inc.
# All rights reserved.
#
# This file is proprietary and confidential to TAOS Technologies.
# No part of this file may be reproduced, stored, transmitted,
# disclosed or used in any form or by any means other than as
# expressly provided by the written permission from Jianhui Tao
#
###################################################################
# -*- coding: utf-8 -*-
import sys
from util.log import *
from util.cases import *
from util.sql import *
class TDTestCase:
def init(self, conn, logSql):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor(), logSql)
def run(self):
tdSql.prepare()
# test case for https://jira.taosdata.com:18080/browse/TD-4584
#1
tdLog.info('=============== step1,create stable')
tdLog.info('create table stb1 (ts timestamp, value double) tags (bin binary(128))')
tdSql.execute('create table stb1 (ts timestamp, value double) tags (bin binary(128))')
tdLog.info('=============== step2,create table增加了转义字符')
tdLog.info('create table tb1 using stb1 tags("abc\\"def")')
#增加了转义字符\
tdSql.execute('create table tb1 using stb1 tags("abc\\"def")')
tdLog.info('=============== step3,insert data')
tdLog.info('insert into tb1 values(now,1.0)')
tdSql.execute('insert into tb1 values(now,1.0)')
tdLog.info('=============== step4,select table')
tdLog.info('select * from stb1 ')
tdSql.query('select * from stb1 ')
tdLog.info('=============== step5,check data')
tdSql.checkData(0,2,'abc"def')
def stop(self):
tdSql.close()
tdLog.success("%s successfully executed" % __file__)
tdCases.addWindows(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())

4
tests/pytest/query/filter.py
View File

@ -91,8 +91,8 @@ class TDTestCase:
tdSql.query("select * from db.st where name = 1231231")
tdSql.checkRows(0)
# <> for timestamp type
tdSql.query("select * from db.st where ts <> '2020-05-13 10:00:00.002'")
# <> for timestamp type not supported on primary timestamp
tdSql.error("select * from db.st where ts <> '2020-05-13 10:00:00.002'")
# tdSql.checkRows(4)
# <> for numeric type

4
tests/pytest/query/querySecondtscolumnTowherenow.py
View File

@ -58,8 +58,8 @@ class TDTestCase:
ts_len4 = len(tdSql.cursor.fetchall())
tdSql.execute("select * from t2ts1 where ts = now")
ts_len5 = len(tdSql.cursor.fetchall())
tdSql.execute("select * from t2ts1 where ts <> now")
ts_len6 = len(tdSql.cursor.fetchall())
# tdSql.execute("select * from t2ts1 where ts <> now")
ts_len6 = 3
tdSql.execute("select * from t2ts1 where ts between 0 and now")
ts_len7 = len(tdSql.cursor.fetchall())
tdSql.execute("select * from t2ts1 where ts between now and now+100d")

46
tests/pytest/smoketest.sh
View File

@ -2,38 +2,38 @@
ulimit -c unlimited
# insert
python3.8 ./test.py $1 -f insert/basic.py
python3.8 ./test.py $1 -s && sleep 1
python3.8 ./test.py $1 -f insert/bigint.py
python3.8 ./test.py $1 -s && sleep 1
python3.8 ./test.py $1 -f insert/nchar.py
python3.8 ./test.py $1 -s && sleep 1
python3.8 ./test.py $1 -f insert/multi.py
python3.8 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f insert/basic.py
python3 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f insert/bigint.py
python3 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f insert/nchar.py
python3 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f insert/multi.py
python3 ./test.py $1 -s && sleep 1
# table
python3.8 ./test.py $1 -f table/column_name.py
python3.8 ./test.py $1 -s && sleep 1
python3.8 ./test.py $1 -f table/column_num.py
python3.8 ./test.py $1 -s && sleep 1
python3.8 ./test.py $1 -f table/db_table.py
python3.8 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f table/column_name.py
python3 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f table/column_num.py
python3 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f table/db_table.py
python3 ./test.py $1 -s && sleep 1
# import
python3.8 ./test.py $1 -f import_merge/importDataLastSub.py
python3.8 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f import_merge/importDataLastSub.py
python3 ./test.py $1 -s && sleep 1
#tag
python3.8 ./test.py $1 -f tag_lite/filter.py
python3.8 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f tag_lite/filter.py
python3 ./test.py $1 -s && sleep 1
#query
python3.8 ./test.py $1 -f query/filter.py
python3.8 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f query/filter.py
python3 ./test.py $1 -s && sleep 1
# client
python3.8 ./test.py $1 -f client/client.py
python3.8 ./test.py $1 -s && sleep 1
python3 ./test.py $1 -f client/client.py
python3 ./test.py $1 -s && sleep 1
# connector
python3.8 ./test.py $1 -f connector/lua.py
python3 ./test.py $1 -f connector/lua.py

649
tests/script/api/stmtBatchTest.c
View File

@ -4406,6 +4406,654 @@ static void SpecifyColumnBatchCase_autoCreateTbl(TAOS *taos) {
}
/*=======================*/
/*
char * taos_stmt_errstr(TAOS_STMT *stmt) 用于在其他stmt API 返回错误（返回错误码或空指针）时获取错误信息，返回的错误信息可能是以下3种的一个：
1. 当stmt或sqlobj为空时的terrno对于的错误消息;
2. （可有的）详细错误消息；
3. 返回的错误码对于的错误消息；
*/
static int stmt_specifyCol_bind_error_case_001(TAOS_STMT *stmt, int tableNum, int rowsOfPerColum, int bingNum, int lenOfBinaryDef, int lenOfBinaryAct, int columnNum) {
sampleValue* v = (sampleValue *)calloc(1, sizeof(sampleValue));
int totalRowsPerTbl = rowsOfPerColum * bingNum;
v->ts = (int64_t *)malloc(sizeof(int64_t) * (size_t)(totalRowsPerTbl * tableNum));
v->br = (char *)malloc(sizeof(int64_t) * (size_t)(totalRowsPerTbl * lenOfBinaryDef));
v->nr = (char *)malloc(sizeof(int64_t) * (size_t)(totalRowsPerTbl * lenOfBinaryDef));
int *lb = (int *)malloc(MAX_ROWS_OF_PER_COLUMN * sizeof(int));
TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * (size_t)(bingNum * columnNum * (tableNum+1) * rowsOfPerColum));
char* is_null = malloc(sizeof(char) * MAX_ROWS_OF_PER_COLUMN);
char* no_null = malloc(sizeof(char) * MAX_ROWS_OF_PER_COLUMN);
int64_t tts = 1591060628000;
for (int i = 0; i < rowsOfPerColum; ++i) {
lb[i] = lenOfBinaryAct;
no_null[i] = 0;
is_null[i] = (i % 10 == 2) ? 1 : 0;
v->b[i] = (int8_t)(i % 2);
v->v1[i] = (int8_t)((i+1) % 2);
v->v2[i] = (int16_t)i;
v->v4[i] = (int32_t)(i+1);
v->v8[i] = (int64_t)(i+2);
v->f4[i] = (float)(i+3);
v->f8[i] = (double)(i+4);
char tbuf[MAX_BINARY_DEF_LEN];
memset(tbuf, 0, MAX_BINARY_DEF_LEN);
sprintf(tbuf, "binary-%d-0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789", i%10);
memcpy(v->br + i*lenOfBinaryDef, tbuf, (size_t)lenOfBinaryAct);
memset(tbuf, 0, MAX_BINARY_DEF_LEN);
sprintf(tbuf, "nchar-%d-0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789", i%10);
memcpy(v->nr + i*lenOfBinaryDef, tbuf, (size_t)lenOfBinaryAct);
v->ts2[i] = tts + i;
}
int i = 0;
for (int j = 0; j < bingNum * tableNum; j++) {
params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
params[i+0].buffer_length = sizeof(int64_t);
params[i+0].buffer = &v->ts[j*rowsOfPerColum];
params[i+0].length = NULL;
params[i+0].is_null = no_null;
params[i+0].num = rowsOfPerColum;
params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
params[i+1].buffer_length = sizeof(int8_t);
params[i+1].buffer = v->b;
params[i+1].length = NULL;
params[i+1].is_null = is_null;
params[i+1].num = rowsOfPerColum;
params[i+2].buffer_type = TSDB_DATA_TYPE_INT;
params[i+2].buffer_length = sizeof(int32_t);
params[i+2].buffer = v->v4;
params[i+2].length = NULL;
params[i+2].is_null = is_null;
params[i+2].num = rowsOfPerColum;
params[i+3].buffer_type = TSDB_DATA_TYPE_FLOAT;
params[i+3].buffer_length = sizeof(float);
params[i+3].buffer = v->f4;
params[i+3].length = NULL;
params[i+3].is_null = is_null;
params[i+3].num = rowsOfPerColum;
params[i+4].buffer_type = TSDB_DATA_TYPE_BINARY;
params[i+4].buffer_length = (uintptr_t)lenOfBinaryDef;
params[i+4].buffer = v->br;
params[i+4].length = lb;
params[i+4].is_null = is_null;
params[i+4].num = rowsOfPerColum;
i+=columnNum;
}
//int64_t tts = 1591060628000;
for (int i = 0; i < totalRowsPerTbl * tableNum; ++i) {
v->ts[i] = tts + i;
}
unsigned long long starttime = getCurrentTime();
// create table m%d (ts timestamp, b bool, v1 tinyint, v2 smallint, v4 int, v8 bigint, f4 float, f8 double, br binary(%d), nr nchar(%d), ts2 timestamp)
char *sql = "insert into m0 (ts,b,v4,f4,br) values(?,?,?,?,?)";
int code;
//TAOS_STMT *stmtNull = NULL;
//int code = taos_stmt_prepare(stmtNull, sql, 0);
//if (code != 0){
// printf("failed to execute taos_stmt_prepare. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmtNull));
// return -1;
//}
//char* sqlNull = NULL;
//code = taos_stmt_prepare(stmt, sqlNull, 0);
//if (code != 0){
// printf("failed to execute taos_stmt_prepare. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmt));
//}
code = taos_stmt_prepare(stmt, sql, 0);
if (code != 0){
printf("failed to execute taos_stmt_prepare. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmt));
return -1;
}
int id = 0;
for (int l = 0; l < bingNum; l++) {
for (int zz = 0; zz < tableNum; zz++) {
//char buf[32];
//sprintf(buf, "m%d", zz);
//code = taos_stmt_set_tbname(stmt, buf);
//if (code != 0){
// printf("failed to execute taos_stmt_set_tbname. code:0x%x[%s]\n", code, tstrerror(code));
// return -1;
//}
//for (int col=0; col < columnNum + 1000; ++col) {
for (int col=0; col < columnNum; ++col) {
if (col >= columnNum) col = 0;
//code = taos_stmt_bind_single_param_batch(NULL, params + id, col);
//code = taos_stmt_bind_single_param_batch(stmt, NULL, col);
//code = taos_stmt_bind_single_param_batch(stmt, params + id, columnNum);
code = taos_stmt_bind_single_param_batch(stmt, params + id, col);
if (code != 0){
printf("failed to execute taos_stmt_bind_single_param_batch. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmt));
return -1;
}
id++;
}
//code = taos_stmt_add_batch(NULL);
code = taos_stmt_add_batch(stmt);
if (code != 0) {
printf("failed to execute taos_stmt_add_batch. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmt));
return -1;
}
//code = taos_stmt_add_batch(stmt);
//if (code != 0) {
// printf("failed to execute taos_stmt_add_batch second. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmt));
// return -1;
//}
}
code = taos_stmt_execute(stmt);
if (code != 0) {
printf("failed to execute taos_stmt_execute. code:0x%x[%s]\n", code, tstrerror(code));
return -1;
}
}
unsigned long long endtime = getCurrentTime();
unsigned long long totalRows = (uint32_t)(totalRowsPerTbl * tableNum);
printf("insert total %d records, used %u seconds, avg:%u useconds per record\n", totalRows, (endtime-starttime)/1000000UL, (endtime-starttime)/totalRows);
free(v->ts);
free(v->br);
free(v->nr);
free(v);
free(lb);
free(params);
free(is_null);
free(no_null);
return 0;
}
static int stmt_specifyCol_bind_error_case_002(TAOS_STMT *stmt, int tableNum, int rowsOfPerColum, int bingNum, int lenOfBinaryDef, int lenOfBinaryAct, int columnNum) {
sampleValue* v = (sampleValue *)calloc(1, sizeof(sampleValue));
int totalRowsPerTbl = rowsOfPerColum * bingNum;
v->ts = (int64_t *)malloc(sizeof(int64_t) * (size_t)(totalRowsPerTbl * tableNum));
v->br = (char *)malloc(sizeof(int64_t) * (size_t)(totalRowsPerTbl * lenOfBinaryDef));
v->nr = (char *)malloc(sizeof(int64_t) * (size_t)(totalRowsPerTbl * lenOfBinaryDef));
int *lb = (int *)malloc(MAX_ROWS_OF_PER_COLUMN * sizeof(int));
TAOS_MULTI_BIND *params = calloc(1, sizeof(TAOS_MULTI_BIND) * (size_t)(bingNum * columnNum * (tableNum+1) * rowsOfPerColum));
char* is_null = malloc(sizeof(char) * MAX_ROWS_OF_PER_COLUMN);
char* no_null = malloc(sizeof(char) * MAX_ROWS_OF_PER_COLUMN);
int64_t tts = 1591060628000;
for (int i = 0; i < rowsOfPerColum; ++i) {
lb[i] = lenOfBinaryAct;
no_null[i] = 0;
is_null[i] = (i % 10 == 2) ? 1 : 0;
v->b[i] = (int8_t)(i % 2);
v->v1[i] = (int8_t)((i+1) % 2);
v->v2[i] = (int16_t)i;
v->v4[i] = (int32_t)(i+1);
v->v8[i] = (int64_t)(i+2);
v->f4[i] = (float)(i+3);
v->f8[i] = (double)(i+4);
char tbuf[MAX_BINARY_DEF_LEN];
memset(tbuf, 0, MAX_BINARY_DEF_LEN);
sprintf(tbuf, "binary-%d", i%10);
memcpy(v->br + i*lenOfBinaryDef, tbuf, (size_t)lenOfBinaryAct);
memset(tbuf, 0, MAX_BINARY_DEF_LEN);
sprintf(tbuf, "nchar-%d", i%10);
memcpy(v->nr + i*lenOfBinaryDef, tbuf, (size_t)lenOfBinaryAct);
v->ts2[i] = tts + i;
}
int i = 0;
for (int j = 0; j < bingNum * tableNum; j++) {
params[i+0].buffer_type = TSDB_DATA_TYPE_TIMESTAMP;
params[i+0].buffer_length = sizeof(int64_t);
params[i+0].buffer = &v->ts[j*rowsOfPerColum];
params[i+0].length = NULL;
params[i+0].is_null = no_null;
params[i+0].num = rowsOfPerColum;
params[i+1].buffer_type = TSDB_DATA_TYPE_BOOL;
params[i+1].buffer_length = sizeof(int8_t);
params[i+1].buffer = v->b;
params[i+1].length = NULL;
params[i+1].is_null = is_null;
params[i+1].num = rowsOfPerColum;
params[i+2].buffer_type = TSDB_DATA_TYPE_INT;
params[i+2].buffer_length = sizeof(int32_t);
params[i+2].buffer = v->v4;
params[i+2].length = NULL;
params[i+2].is_null = is_null;
params[i+2].num = rowsOfPerColum;
params[i+3].buffer_type = TSDB_DATA_TYPE_FLOAT;
params[i+3].buffer_length = sizeof(float);
params[i+3].buffer = v->f4;
params[i+3].length = NULL;
params[i+3].is_null = is_null;
params[i+3].num = rowsOfPerColum;
params[i+4].buffer_type = TSDB_DATA_TYPE_BINARY;
params[i+4].buffer_length = (uintptr_t)lenOfBinaryDef;
params[i+4].buffer = v->br;
params[i+4].length = lb;
params[i+4].is_null = is_null;
params[i+4].num = rowsOfPerColum;
i+=columnNum;
}
//int64_t tts = 1591060628000;
for (int i = 0; i < totalRowsPerTbl * tableNum; ++i) {
v->ts[i] = tts + i;
}
unsigned long long starttime = getCurrentTime();
// create table m%d (ts timestamp, b bool, v1 tinyint, v2 smallint, v4 int, v8 bigint, f4 float, f8 double, br binary(%d), nr nchar(%d), ts2 timestamp)
char *sql = "insert into ? (ts,b,v4,f4,br) values(?,?,?,?,?)";
int code = taos_stmt_prepare(stmt, sql, 0);
if (code != 0){
printf("failed to execute taos_stmt_prepare. code:0x%x[%s]\n", code, tstrerror(code));
return -1;
}
int id = 0;
for (int l = 0; l < bingNum; l++) {
for (int zz = 0; zz < tableNum; zz++) {
char buf[32];
sprintf(buf, "m%d", zz);
//stmt=NULL;
code = taos_stmt_set_tbname(stmt, buf);
if (code != 0){
printf("failed to execute taos_stmt_set_tbname. code:0x%x[%s][%s]\n", code, tstrerror(code), taos_stmt_errstr(stmt));
return -1;
}
for (int col=0; col < columnNum; ++col) {
code = taos_stmt_bind_single_param_batch(stmt, params + id, col);
if (code != 0){
printf("failed to execute taos_stmt_bind_single_param_batch. code:0x%x[%s]\n", code, tstrerror(code));
return -1;
}
id++;
}
code = taos_stmt_add_batch(stmt);
if (code != 0) {
printf("failed to execute taos_stmt_add_batch. code:0x%x[%s]\n", code, tstrerror(code));
return -1;
}
}
code = taos_stmt_execute(stmt);
if (code != 0) {
printf("failed to execute taos_stmt_execute. code:0x%x[%s]\n", code, tstrerror(code));
return -1;
}
}
unsigned long long endtime = getCurrentTime();
unsigned long long totalRows = (uint32_t)(totalRowsPerTbl * tableNum);
printf("insert total %d records, used %u seconds, avg:%u useconds per record\n", totalRows, (endtime-starttime)/1000000UL, (endtime-starttime)/totalRows);
free(v->ts);
free(v->br);
free(v->nr);
free(v);
free(lb);
free(params);
free(is_null);
free(no_null);
return 0;
}
static void SpecifyColumnBatchErrorCase(TAOS *taos) {
TAOS_STMT *stmt = NULL;
int tableNum;
int lenOfBinaryDef;
int rowsOfPerColum;
int bingNum;
int lenOfBinaryAct;
int columnNum;
int totalRowsPerTbl;
//=======================================================================//
//=============================== single table ==========================//
//========== case 1: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 1;
bingNum = 1;
lenOfBinaryDef = 40;
lenOfBinaryAct = 8;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db1");
stmt_specifyCol_bind_error_case_001(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 1 check result end\n\n");
}
#endif
//========== case 2: ======================//
#if 1
{
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 5;
bingNum = 1;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 15;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db2");
stmt_specifyCol_bind_error_case_002(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
//checkResult(taos, "m1", 0, totalRowsPerTbl);
//checkResult(taos, "m2", 0, totalRowsPerTbl);
//checkResult(taos, "m3", 0, totalRowsPerTbl);
//checkResult(taos, "m4", 0, totalRowsPerTbl);
//checkResult(taos, "m5", 0, totalRowsPerTbl);
//checkResult(taos, "m6", 0, totalRowsPerTbl);
//checkResult(taos, "m7", 0, totalRowsPerTbl);
//checkResult(taos, "m8", 0, totalRowsPerTbl);
//checkResult(taos, "m9", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 2 check result end\n\n");
}
#endif
//========== case 2-1: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 32767;
bingNum = 1;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 15;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db2_1");
stmt_specifyCol_bind_case_001(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
//checkResult(taos, "m1", 0, totalRowsPerTbl);
//checkResult(taos, "m2", 0, totalRowsPerTbl);
//checkResult(taos, "m3", 0, totalRowsPerTbl);
//checkResult(taos, "m4", 0, totalRowsPerTbl);
//checkResult(taos, "m5", 0, totalRowsPerTbl);
//checkResult(taos, "m6", 0, totalRowsPerTbl);
//checkResult(taos, "m7", 0, totalRowsPerTbl);
//checkResult(taos, "m8", 0, totalRowsPerTbl);
//checkResult(taos, "m9", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 2-1 check result end\n\n");
}
#endif
//========== case 2-2: ======================//
#if 0
{
printf("====case 2-2 error test start\n");
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 32768;
bingNum = 1;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 15;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db2_2");
stmt_specifyCol_bind_case_001(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
//checkResult(taos, "m1", 0, totalRowsPerTbl);
//checkResult(taos, "m2", 0, totalRowsPerTbl);
//checkResult(taos, "m3", 0, totalRowsPerTbl);
//checkResult(taos, "m4", 0, totalRowsPerTbl);
//checkResult(taos, "m5", 0, totalRowsPerTbl);
//checkResult(taos, "m6", 0, totalRowsPerTbl);
//checkResult(taos, "m7", 0, totalRowsPerTbl);
//checkResult(taos, "m8", 0, totalRowsPerTbl);
//checkResult(taos, "m9", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("====case 2-2 check result end\n\n");
}
#endif
//========== case 3: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 1;
bingNum = 5;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 20;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db3");
stmt_specifyCol_bind_case_001(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
//checkResult(taos, "m1", 0, totalRowsPerTbl);
//checkResult(taos, "m2", 0, totalRowsPerTbl);
//checkResult(taos, "m3", 0, totalRowsPerTbl);
//checkResult(taos, "m4", 0, totalRowsPerTbl);
//checkResult(taos, "m5", 0, totalRowsPerTbl);
//checkResult(taos, "m6", 0, totalRowsPerTbl);
//checkResult(taos, "m7", 0, totalRowsPerTbl);
//checkResult(taos, "m8", 0, totalRowsPerTbl);
//checkResult(taos, "m9", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 3 check result end\n\n");
}
#endif
//========== case 4: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 5;
bingNum = 5;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 33;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db4");
stmt_specifyCol_bind_case_001(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
//checkResult(taos, "m1", 0, totalRowsPerTbl);
//checkResult(taos, "m2", 0, totalRowsPerTbl);
//checkResult(taos, "m3", 0, totalRowsPerTbl);
//checkResult(taos, "m4", 0, totalRowsPerTbl);
//checkResult(taos, "m5", 0, totalRowsPerTbl);
//checkResult(taos, "m6", 0, totalRowsPerTbl);
//checkResult(taos, "m7", 0, totalRowsPerTbl);
//checkResult(taos, "m8", 0, totalRowsPerTbl);
//checkResult(taos, "m9", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 4 check result end\n\n");
}
#endif
//=======================================================================//
//=============================== multi tables ==========================//
//========== case 5: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 5;
rowsOfPerColum = 1;
bingNum = 1;
lenOfBinaryDef = 40;
lenOfBinaryAct = 16;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db5");
stmt_specifyCol_bind_case_002(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
checkResult(taos, "m1", 0, totalRowsPerTbl);
checkResult(taos, "m2", 0, totalRowsPerTbl);
checkResult(taos, "m3", 0, totalRowsPerTbl);
checkResult(taos, "m4", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 5 check result end\n\n");
}
#endif
//========== case 6: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 5;
rowsOfPerColum = 5;
bingNum = 1;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 20;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db6");
stmt_specifyCol_bind_case_002(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
checkResult(taos, "m1", 0, totalRowsPerTbl);
checkResult(taos, "m2", 0, totalRowsPerTbl);
checkResult(taos, "m3", 0, totalRowsPerTbl);
checkResult(taos, "m4", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 6 check result end\n\n");
}
#endif
//========== case 7: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 5;
rowsOfPerColum = 1;
bingNum = 5;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 33;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db7");
stmt_specifyCol_bind_case_002(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
checkResult(taos, "m1", 0, totalRowsPerTbl);
checkResult(taos, "m2", 0, totalRowsPerTbl);
checkResult(taos, "m3", 0, totalRowsPerTbl);
checkResult(taos, "m4", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 7 check result end\n\n");
}
#endif
//========== case 8: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 5;
rowsOfPerColum = 5;
bingNum = 5;
lenOfBinaryDef = 1000;
lenOfBinaryAct = 40;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db8");
stmt_specifyCol_bind_case_002(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
checkResult(taos, "m1", 0, totalRowsPerTbl);
checkResult(taos, "m2", 0, totalRowsPerTbl);
checkResult(taos, "m3", 0, totalRowsPerTbl);
checkResult(taos, "m4", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 8 check result end\n\n");
}
#endif
//=======================================================================//
//=============================== multi-rows to single table ==========================//
//========== case 9: ======================//
#if 0
{
stmt = taos_stmt_init(taos);
tableNum = 1;
rowsOfPerColum = 23740;
bingNum = 1;
lenOfBinaryDef = 40;
lenOfBinaryAct = 8;
columnNum = 5;
prepareVcolumn(taos, 1, tableNum, lenOfBinaryDef, "db9");
stmt_specifyCol_bind_case_001(stmt, tableNum, rowsOfPerColum, bingNum, lenOfBinaryDef, lenOfBinaryAct, columnNum);
totalRowsPerTbl = rowsOfPerColum * bingNum;
checkResult(taos, "m0", 0, totalRowsPerTbl);
taos_stmt_close(stmt);
printf("case 9 check result end\n\n");
}
#endif
return ;
}
int main(int argc, char *argv[])
{
TAOS *taos;
@ -4472,6 +5120,7 @@ int main(int argc, char *argv[])
//runCase_long(taos);
//SpecifyColumnBatchCase(taos);
SpecifyColumnBatchCase_autoCreateTbl(taos);
//SpecifyColumnBatchErrorCase(taos);
return 0;
}

6
tests/script/general/parser/function.sim
View File

@ -1080,6 +1080,8 @@ sql insert into t1 values('2020-1-1 1:1:10', 20000);
sql_error select derivative(k, 1s, 0) from m1;
sql_error select derivative(k, 1s, 0) from m1 group by a;
sql_error select derivative(f1, 1s, 0) from (select k from t1);
sql select derivative(k, 1s, 0) from m1 group by tbname
if $rows != 12 then
return -1
@ -1119,4 +1121,6 @@ endi
if $data92 != t1 then
return -1
endi
endi
sql select derivative(test_column_alias_name, 1s, 0) from (select avg(k) test_column_alias_name from t1 interval(1s));

14
tests/script/general/parser/nestquery.sim
View File

@ -273,4 +273,18 @@ if $data03 != @20-09-15 00:00:00.000@ then
return -1
endi
sql_error select derivative(val, 1s, 0) from (select c1 val from nest_tb0);
sql select diff(val) from (select c1 val from nest_tb0);
if $rows != 9999 then
return -1
endi
if $data00 != @70-01-01 08:00:00.000@ then
return -1
endi
if $data01 != 1 then
return -1
endi
system sh/exec.sh -n dnode1 -s stop -x SIGINT

4
tests/script/general/parser/timestamp_query.sim
View File

@ -24,6 +24,8 @@ $tsu = $tsu + $ts0
print ==================>issue #3481, normal column not allowed,
sql_error select ts,c1,min(c2) from ts_stb0
print ==================>issue #4681, not equal operator on primary timestamp not allowed
sql_error select * from ts_stb0 where ts <> $ts0
##### select from supertable
$tb = $tbPrefix . 0
@ -51,4 +53,4 @@ sql select first(c1), last(c1), (1537325400 - 1537146000)/(5*60) v from $tb wher
if $data13 != 598.000000000 then
print expect 598.000000000, actual $data03
return -1
endi
endi