Merge branch '3.0' into feature/stream
This commit is contained in:
commit
414673db57
|
@ -202,6 +202,10 @@ The configuration parameters in the URL are as follows.
|
||||||
- batchfetch: true: pull the result set in batch when executing the query; false: pull the result set row by row. The default value is false. batchfetch uses HTTP for data transfer. The JDBC REST connection supports bulk data pulling function in taos-jdbcdriver-2.0.38 and TDengine 2.4.0.12 and later versions. taos-jdbcdriver and TDengine transfer data via WebSocket connection. Compared with HTTP, WebSocket enables JDBC REST connection to support large data volume querying and improve query performance.
|
- batchfetch: true: pull the result set in batch when executing the query; false: pull the result set row by row. The default value is false. batchfetch uses HTTP for data transfer. The JDBC REST connection supports bulk data pulling function in taos-jdbcdriver-2.0.38 and TDengine 2.4.0.12 and later versions. taos-jdbcdriver and TDengine transfer data via WebSocket connection. Compared with HTTP, WebSocket enables JDBC REST connection to support large data volume querying and improve query performance.
|
||||||
- charset: specify the charset to parse the string, this parameter is valid only when set batchfetch to true.
|
- charset: specify the charset to parse the string, this parameter is valid only when set batchfetch to true.
|
||||||
- batchErrorIgnore: true: when executing executeBatch of Statement, if one SQL execution fails in the middle, continue to execute the following SQL. false: no longer execute any statement after the failed SQL. The default value is: false.
|
- batchErrorIgnore: true: when executing executeBatch of Statement, if one SQL execution fails in the middle, continue to execute the following SQL. false: no longer execute any statement after the failed SQL. The default value is: false.
|
||||||
|
- httpConnectTimeout: REST connection timeout in milliseconds, the default value is 5000 ms.
|
||||||
|
- httpSocketTimeout: socket timeout in milliseconds, the default value is 5000 ms. It only takes effect when batchfetch is false.
|
||||||
|
- messageWaitTimeout: message transmission timeout in milliseconds, the default value is 3000 ms. It only takes effect when batchfetch is true.
|
||||||
|
- useSSL: connecting Securely Using SSL. true: using SSL conneciton, false: not using SSL connection.
|
||||||
|
|
||||||
**Note**: Some configuration items (e.g., locale, timezone) do not work in the REST connection.
|
**Note**: Some configuration items (e.g., locale, timezone) do not work in the REST connection.
|
||||||
|
|
||||||
|
@ -257,14 +261,18 @@ In the above example, a connection is established to `taosdemo.com`, port is 603
|
||||||
|
|
||||||
The configuration parameters in properties are as follows.
|
The configuration parameters in properties are as follows.
|
||||||
|
|
||||||
- TSDBDriver.PROPERTY_KEY_USER: Login TDengine user name, default value 'root'.
|
- TSDBDriver.PROPERTY_KEY_USER: login TDengine user name, default value 'root'.
|
||||||
- TSDBDriver.PROPERTY_KEY_PASSWORD: user login password, default value 'taosdata'.
|
- TSDBDriver.PROPERTY_KEY_PASSWORD: user login password, default value 'taosdata'.
|
||||||
- TSDBDriver.PROPERTY_KEY_BATCH_LOAD: true: pull the result set in batch when executing query; false: pull the result set row by row. The default value is: false.
|
- TSDBDriver.PROPERTY_KEY_BATCH_LOAD: true: pull the result set in batch when executing query; false: pull the result set row by row. The default value is: false.
|
||||||
- TSDBDriver.PROPERTY_KEY_BATCH_ERROR_IGNORE: true: when executing executeBatch of Statement, if there is a SQL execution failure in the middle, continue to execute the following sq. false: no longer execute any statement after the failed SQL. The default value is: false.
|
- TSDBDriver.PROPERTY_KEY_BATCH_ERROR_IGNORE: true: when executing executeBatch of Statement, if there is a SQL execution failure in the middle, continue to execute the following sq. false: no longer execute any statement after the failed SQL. The default value is: false.
|
||||||
- TSDBDriver.PROPERTY_KEY_CONFIG_DIR: Only works when using JDBC native connection. Client configuration file directory path, default value `/etc/taos` on Linux OS, default value `C:/TDengine/cfg` on Windows OS.
|
- TSDBDriver.PROPERTY_KEY_CONFIG_DIR: only works when using JDBC native connection. Client configuration file directory path, default value `/etc/taos` on Linux OS, default value `C:/TDengine/cfg` on Windows OS.
|
||||||
- TSDBDriver.PROPERTY_KEY_CHARSET: In the character set used by the client, the default value is the system character set.
|
- TSDBDriver.PROPERTY_KEY_CHARSET: In the character set used by the client, the default value is the system character set.
|
||||||
- TSDBDriver.PROPERTY_KEY_LOCALE: this only takes effect when using JDBC native connection. Client language environment, the default value is system current locale.
|
- TSDBDriver.PROPERTY_KEY_LOCALE: this only takes effect when using JDBC native connection. Client language environment, the default value is system current locale.
|
||||||
- TSDBDriver.PROPERTY_KEY_TIME_ZONE: only takes effect when using JDBC native connection. In the time zone used by the client, the default value is the system's current time zone.
|
- TSDBDriver.PROPERTY_KEY_TIME_ZONE: only takes effect when using JDBC native connection. In the time zone used by the client, the default value is the system's current time zone.
|
||||||
|
- TSDBDriver.HTTP_CONNECT_TIMEOUT: REST connection timeout in milliseconds, the default value is 5000 ms. It only takes effect when using JDBC REST connection.
|
||||||
|
- TSDBDriver.HTTP_SOCKET_TIMEOUT: socket timeout in milliseconds, the default value is 5000 ms. It only takes effect when using JDBC REST connection and batchfetch is false.
|
||||||
|
- TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: message transmission timeout in milliseconds, the default value is 3000 ms. It only takes effect when using JDBC REST connection and batchfetch is true.
|
||||||
|
- TSDBDriver.PROPERTY_KEY_USE_SSL: connecting Securely Using SSL. true: using SSL conneciton, false: not using SSL connection. It only takes effect when using using JDBC REST connection.
|
||||||
For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](/reference/config/#Client-Only).
|
For JDBC native connections, you can specify other parameters, such as log level, SQL length, etc., by specifying URL and Properties. For more detailed configuration, please refer to [Client Configuration](/reference/config/#Client-Only).
|
||||||
|
|
||||||
### Priority of configuration parameters
|
### Priority of configuration parameters
|
||||||
|
@ -813,10 +821,11 @@ Please refer to: [JDBC example](https://github.com/taosdata/TDengine/tree/develo
|
||||||
## Recent update logs
|
## Recent update logs
|
||||||
|
|
||||||
| taos-jdbcdriver version | major changes |
|
| taos-jdbcdriver version | major changes |
|
||||||
| :---------------------: | :------------------------------------------: |
|
| :---------------------: | :--------------------------------------------: |
|
||||||
|
| 2.0.39 - 2.0.40 | Add REST connection/request timeout parameters |
|
||||||
| 2.0.38 | JDBC REST connections add bulk pull function |
|
| 2.0.38 | JDBC REST connections add bulk pull function |
|
||||||
| 2.0.37 | Added support for json tags |
|
| 2.0.37 | Support json tags |
|
||||||
| 2.0.36 | Add support for schemaless writing |
|
| 2.0.36 | Support schemaless writing |
|
||||||
|
|
||||||
## Frequently Asked Questions
|
## Frequently Asked Questions
|
||||||
|
|
||||||
|
|
|
@ -201,6 +201,10 @@ url 中的配置参数如下:
|
||||||
- batchfetch: true:在执行查询时批量拉取结果集;false:逐行拉取结果集。默认值为:false。逐行拉取结果集使用 HTTP 方式进行数据传输。从 taos-jdbcdriver-2.0.38 和 TDengine 2.4.0.12 版本开始,JDBC REST 连接增加批量拉取数据功能。taos-jdbcdriver 与 TDengine 之间通过 WebSocket 连接进行数据传输。相较于 HTTP,WebSocket 可以使 JDBC REST 连接支持大数据量查询,并提升查询性能。
|
- batchfetch: true:在执行查询时批量拉取结果集;false:逐行拉取结果集。默认值为:false。逐行拉取结果集使用 HTTP 方式进行数据传输。从 taos-jdbcdriver-2.0.38 和 TDengine 2.4.0.12 版本开始,JDBC REST 连接增加批量拉取数据功能。taos-jdbcdriver 与 TDengine 之间通过 WebSocket 连接进行数据传输。相较于 HTTP,WebSocket 可以使 JDBC REST 连接支持大数据量查询,并提升查询性能。
|
||||||
- charset: 当开启批量拉取数据时,指定解析字符串数据的字符集。
|
- charset: 当开启批量拉取数据时,指定解析字符串数据的字符集。
|
||||||
- batchErrorIgnore:true:在执行 Statement 的 executeBatch 时,如果中间有一条 SQL 执行失败,继续执行下面的 SQL 了。false:不再执行失败 SQL 后的任何语句。默认值为:false。
|
- batchErrorIgnore:true:在执行 Statement 的 executeBatch 时,如果中间有一条 SQL 执行失败,继续执行下面的 SQL 了。false:不再执行失败 SQL 后的任何语句。默认值为:false。
|
||||||
|
- httpConnectTimeout: 连接超时时间,单位 ms, 默认值为 5000。
|
||||||
|
- httpSocketTimeout: socket 超时时间,单位 ms,默认值为 5000。仅在 batchfetch 设置为 false 时生效。
|
||||||
|
- messageWaitTimeout: 消息超时时间, 单位 ms, 默认值为 3000。 仅在 batchfetch 设置为 true 时生效。
|
||||||
|
- useSSL: 连接中是否使用 SSL。
|
||||||
|
|
||||||
**注意**:部分配置项(比如:locale、timezone)在 REST 连接中不生效。
|
**注意**:部分配置项(比如:locale、timezone)在 REST 连接中不生效。
|
||||||
|
|
||||||
|
@ -264,7 +268,11 @@ properties 中的配置参数如下:
|
||||||
- TSDBDriver.PROPERTY_KEY_CHARSET:客户端使用的字符集,默认值为系统字符集。
|
- TSDBDriver.PROPERTY_KEY_CHARSET:客户端使用的字符集,默认值为系统字符集。
|
||||||
- TSDBDriver.PROPERTY_KEY_LOCALE:仅在使用 JDBC 原生连接时生效。 客户端语言环境,默认值系统当前 locale。
|
- TSDBDriver.PROPERTY_KEY_LOCALE:仅在使用 JDBC 原生连接时生效。 客户端语言环境,默认值系统当前 locale。
|
||||||
- TSDBDriver.PROPERTY_KEY_TIME_ZONE:仅在使用 JDBC 原生连接时生效。 客户端使用的时区,默认值为系统当前时区。
|
- TSDBDriver.PROPERTY_KEY_TIME_ZONE:仅在使用 JDBC 原生连接时生效。 客户端使用的时区,默认值为系统当前时区。
|
||||||
- 此外对 JDBC 原生连接,通过指定 URL 和 Properties 还可以指定其他参数,比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](/reference/config/#仅客户端适用)。
|
- TSDBDriver.HTTP_CONNECT_TIMEOUT: 连接超时时间,单位 ms, 默认值为 5000。仅在 REST 连接时生效。
|
||||||
|
- TSDBDriver.HTTP_SOCKET_TIMEOUT: socket 超时时间,单位 ms,默认值为 5000。仅在 REST 连接且 batchfetch 设置为 false 时生效。
|
||||||
|
- TSDBDriver.PROPERTY_KEY_MESSAGE_WAIT_TIMEOUT: 消息超时时间, 单位 ms, 默认值为 3000。 仅在 REST 连接且 batchfetch 设置为 true 时生效。
|
||||||
|
- TSDBDriver.PROPERTY_KEY_USE_SSL: 连接中是否使用 SSL。仅在 REST 连接时生效。
|
||||||
|
此外对 JDBC 原生连接,通过指定 URL 和 Properties 还可以指定其他参数,比如日志级别、SQL 长度等。更多详细配置请参考[客户端配置](/reference/config/#仅客户端适用)。
|
||||||
|
|
||||||
### 配置参数的优先级
|
### 配置参数的优先级
|
||||||
|
|
||||||
|
@ -809,6 +817,7 @@ Query OK, 1 row(s) in set (0.000141s)
|
||||||
|
|
||||||
| taos-jdbcdriver 版本 | 主要变化 |
|
| taos-jdbcdriver 版本 | 主要变化 |
|
||||||
| :------------------: | :----------------------------: |
|
| :------------------: | :----------------------------: |
|
||||||
|
| 2.0.39 - 2.0.40 | 增加 REST 连接/请求 超时设置 |
|
||||||
| 2.0.38 | JDBC REST 连接增加批量拉取功能 |
|
| 2.0.38 | JDBC REST 连接增加批量拉取功能 |
|
||||||
| 2.0.37 | 增加对 json tag 支持 |
|
| 2.0.37 | 增加对 json tag 支持 |
|
||||||
| 2.0.36 | 增加对 schemaless 写入支持 |
|
| 2.0.36 | 增加对 schemaless 写入支持 |
|
||||||
|
|
|
@ -28,8 +28,9 @@ static void msg_process(TAOS_RES* msg) {
|
||||||
printf("db: %s\n", tmq_get_db_name(msg));
|
printf("db: %s\n", tmq_get_db_name(msg));
|
||||||
printf("vg: %d\n", tmq_get_vgroup_id(msg));
|
printf("vg: %d\n", tmq_get_vgroup_id(msg));
|
||||||
if (tmq_get_res_type(msg) == TMQ_RES_TABLE_META) {
|
if (tmq_get_res_type(msg) == TMQ_RES_TABLE_META) {
|
||||||
tmq_raw_data* raw = tmq_get_raw_meta(msg);
|
tmq_raw_data raw = {0};
|
||||||
if (raw) {
|
int32_t code = tmq_get_raw_meta(msg, &raw);
|
||||||
|
if (code == 0) {
|
||||||
TAOS* pConn = taos_connect("192.168.1.86", "root", "taosdata", NULL, 0);
|
TAOS* pConn = taos_connect("192.168.1.86", "root", "taosdata", NULL, 0);
|
||||||
if (pConn == NULL) {
|
if (pConn == NULL) {
|
||||||
return;
|
return;
|
||||||
|
@ -53,7 +54,6 @@ static void msg_process(TAOS_RES* msg) {
|
||||||
printf("write raw data: %s\n", tmq_err2str(ret));
|
printf("write raw data: %s\n", tmq_err2str(ret));
|
||||||
taos_close(pConn);
|
taos_close(pConn);
|
||||||
}
|
}
|
||||||
tmq_free_raw_meta(raw);
|
|
||||||
char* result = tmq_get_json_meta(msg);
|
char* result = tmq_get_json_meta(msg);
|
||||||
if (result) {
|
if (result) {
|
||||||
printf("meta result: %s\n", result);
|
printf("meta result: %s\n", result);
|
||||||
|
|
|
@ -259,13 +259,17 @@ enum tmq_res_t {
|
||||||
TMQ_RES_TABLE_META = 2,
|
TMQ_RES_TABLE_META = 2,
|
||||||
};
|
};
|
||||||
|
|
||||||
|
typedef struct tmq_raw_data{
|
||||||
|
void* raw_meta;
|
||||||
|
uint32_t raw_meta_len;
|
||||||
|
uint16_t raw_meta_type;
|
||||||
|
} tmq_raw_data;
|
||||||
|
|
||||||
typedef enum tmq_res_t tmq_res_t;
|
typedef enum tmq_res_t tmq_res_t;
|
||||||
typedef struct tmq_raw_data tmq_raw_data;
|
|
||||||
|
|
||||||
DLL_EXPORT tmq_res_t tmq_get_res_type(TAOS_RES *res);
|
DLL_EXPORT tmq_res_t tmq_get_res_type(TAOS_RES *res);
|
||||||
DLL_EXPORT tmq_raw_data *tmq_get_raw_meta(TAOS_RES *res);
|
DLL_EXPORT int32_t tmq_get_raw_meta(TAOS_RES *res, tmq_raw_data *raw_meta);
|
||||||
DLL_EXPORT int32_t taos_write_raw_meta(TAOS *taos, tmq_raw_data *raw_meta);
|
DLL_EXPORT int32_t taos_write_raw_meta(TAOS *taos, tmq_raw_data raw_meta);
|
||||||
DLL_EXPORT void tmq_free_raw_meta(tmq_raw_data *rawMeta);
|
|
||||||
DLL_EXPORT char *tmq_get_json_meta(TAOS_RES *res); // Returning null means error. Returned result need to be freed by tmq_free_json_meta
|
DLL_EXPORT char *tmq_get_json_meta(TAOS_RES *res); // Returning null means error. Returned result need to be freed by tmq_free_json_meta
|
||||||
DLL_EXPORT void tmq_free_json_meta(char* jsonMeta);
|
DLL_EXPORT void tmq_free_json_meta(char* jsonMeta);
|
||||||
DLL_EXPORT const char *tmq_get_topic_name(TAOS_RES *res);
|
DLL_EXPORT const char *tmq_get_topic_name(TAOS_RES *res);
|
||||||
|
|
|
@ -40,6 +40,7 @@ typedef struct SReadHandle {
|
||||||
bool initMetaReader;
|
bool initMetaReader;
|
||||||
bool initTableReader;
|
bool initTableReader;
|
||||||
bool initTqReader;
|
bool initTqReader;
|
||||||
|
int32_t numOfVgroups;
|
||||||
} SReadHandle;
|
} SReadHandle;
|
||||||
|
|
||||||
// in queue mode, data streams are seperated by msg
|
// in queue mode, data streams are seperated by msg
|
||||||
|
@ -63,7 +64,7 @@ qTaskInfo_t qCreateStreamExecTaskInfo(void* msg, SReadHandle* readers);
|
||||||
* @param SReadHandle
|
* @param SReadHandle
|
||||||
* @return
|
* @return
|
||||||
*/
|
*/
|
||||||
qTaskInfo_t qCreateQueueExecTaskInfo(void* msg, SReadHandle* readers);
|
qTaskInfo_t qCreateQueueExecTaskInfo(void* msg, SReadHandle* readers, int32_t* numOfCols);
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Set the input data block for the stream scan.
|
* Set the input data block for the stream scan.
|
||||||
|
|
|
@ -162,9 +162,12 @@ typedef struct SRequestConnInfo {
|
||||||
SEpSet mgmtEps;
|
SEpSet mgmtEps;
|
||||||
} SRequestConnInfo;
|
} SRequestConnInfo;
|
||||||
|
|
||||||
|
typedef void (*__freeFunc)(void *param);
|
||||||
|
|
||||||
typedef struct SMsgSendInfo {
|
typedef struct SMsgSendInfo {
|
||||||
__async_send_cb_fn_t fp; // async callback function
|
__async_send_cb_fn_t fp; // async callback function
|
||||||
STargetInfo target; // for update epset
|
STargetInfo target; // for update epset
|
||||||
|
__freeFunc paramFreeFp;
|
||||||
void* param;
|
void* param;
|
||||||
uint64_t requestId;
|
uint64_t requestId;
|
||||||
uint64_t requestObjRefId;
|
uint64_t requestObjRefId;
|
||||||
|
@ -188,6 +191,8 @@ int32_t cleanupTaskQueue();
|
||||||
*/
|
*/
|
||||||
int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code);
|
int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code);
|
||||||
|
|
||||||
|
void destroySendMsgInfo(SMsgSendInfo* pMsgBody);
|
||||||
|
|
||||||
int32_t asyncSendMsgToServerExt(void* pTransporter, SEpSet* epSet, int64_t* pTransporterId, const SMsgSendInfo* pInfo,
|
int32_t asyncSendMsgToServerExt(void* pTransporter, SEpSet* epSet, int64_t* pTransporterId, const SMsgSendInfo* pInfo,
|
||||||
bool persistHandle, void* ctx);
|
bool persistHandle, void* ctx);
|
||||||
|
|
||||||
|
|
|
@ -269,6 +269,7 @@ typedef struct SStreamTask {
|
||||||
int64_t startVer;
|
int64_t startVer;
|
||||||
int64_t checkpointVer;
|
int64_t checkpointVer;
|
||||||
int64_t processedVer;
|
int64_t processedVer;
|
||||||
|
int32_t numOfVgroups;
|
||||||
|
|
||||||
// children info
|
// children info
|
||||||
SArray* childEpInfo; // SArray<SStreamChildEpInfo*>
|
SArray* childEpInfo; // SArray<SStreamChildEpInfo*>
|
||||||
|
|
|
@ -26,7 +26,7 @@ extern "C" {
|
||||||
|
|
||||||
extern bool gRaftDetailLog;
|
extern bool gRaftDetailLog;
|
||||||
|
|
||||||
#define SYNC_RESP_TTL_MS 5000
|
#define SYNC_RESP_TTL_MS 10000
|
||||||
|
|
||||||
#define SYNC_MAX_BATCH_SIZE 500
|
#define SYNC_MAX_BATCH_SIZE 500
|
||||||
#define SYNC_INDEX_BEGIN 0
|
#define SYNC_INDEX_BEGIN 0
|
||||||
|
|
|
@ -73,7 +73,7 @@ cp ${compile_dir}/../include/client/taos.h ${pkg_dir}${install_home_pat
|
||||||
cp ${compile_dir}/../include/common/taosdef.h ${pkg_dir}${install_home_path}/include
|
cp ${compile_dir}/../include/common/taosdef.h ${pkg_dir}${install_home_path}/include
|
||||||
cp ${compile_dir}/../include/util/taoserror.h ${pkg_dir}${install_home_path}/include
|
cp ${compile_dir}/../include/util/taoserror.h ${pkg_dir}${install_home_path}/include
|
||||||
cp ${compile_dir}/../include/libs/function/taosudf.h ${pkg_dir}${install_home_path}/include
|
cp ${compile_dir}/../include/libs/function/taosudf.h ${pkg_dir}${install_home_path}/include
|
||||||
cp ${compile_dir}/../src/inc/taosws.h ${pkg_dir}${install_home_path}/include ||:
|
cp ${compile_dir}/build/include/taosws.h ${pkg_dir}${install_home_path}/include ||:
|
||||||
cp -r ${top_dir}/examples/* ${pkg_dir}${install_home_path}/examples
|
cp -r ${top_dir}/examples/* ${pkg_dir}${install_home_path}/examples
|
||||||
#cp -r ${top_dir}/src/connector/python ${pkg_dir}${install_home_path}/connector
|
#cp -r ${top_dir}/src/connector/python ${pkg_dir}${install_home_path}/connector
|
||||||
#cp -r ${top_dir}/src/connector/go ${pkg_dir}${install_home_path}/connector
|
#cp -r ${top_dir}/src/connector/go ${pkg_dir}${install_home_path}/connector
|
||||||
|
|
|
@ -80,7 +80,7 @@ cp %{_compiledir}/../include/client/taos.h %{buildroot}%{homepath}/incl
|
||||||
cp %{_compiledir}/../include/common/taosdef.h %{buildroot}%{homepath}/include
|
cp %{_compiledir}/../include/common/taosdef.h %{buildroot}%{homepath}/include
|
||||||
cp %{_compiledir}/../include/util/taoserror.h %{buildroot}%{homepath}/include
|
cp %{_compiledir}/../include/util/taoserror.h %{buildroot}%{homepath}/include
|
||||||
cp %{_compiledir}/../include/libs/function/taosudf.h %{buildroot}%{homepath}/include
|
cp %{_compiledir}/../include/libs/function/taosudf.h %{buildroot}%{homepath}/include
|
||||||
cp %{_compiledir}/../src/inc/taosws.h %{buildroot}%{homepath}/include ||:
|
cp %{_compiledir}/build/include/taosws.h %{buildroot}%{homepath}/include ||:
|
||||||
#cp -r %{_compiledir}/../src/connector/python %{buildroot}%{homepath}/connector
|
#cp -r %{_compiledir}/../src/connector/python %{buildroot}%{homepath}/connector
|
||||||
#cp -r %{_compiledir}/../src/connector/go %{buildroot}%{homepath}/connector
|
#cp -r %{_compiledir}/../src/connector/go %{buildroot}%{homepath}/connector
|
||||||
#cp -r %{_compiledir}/../src/connector/nodejs %{buildroot}%{homepath}/connector
|
#cp -r %{_compiledir}/../src/connector/nodejs %{buildroot}%{homepath}/connector
|
||||||
|
|
|
@ -229,13 +229,13 @@ function install_lib() {
|
||||||
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
|
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
|
||||||
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
|
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
|
||||||
|
|
||||||
${csudo}ln -s ${lib_link_dir}/libtaosws.so ${lib_link_dir}/libtaosws.so || :
|
[ -f ${install_main_dir}/driver/libtaosws.so ] && ${csudo}ln -s ${install_main_dir}/driver/libtaosws.so ${lib_link_dir}/libtaosws.so || :
|
||||||
|
|
||||||
if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.so ]]; then
|
if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.so ]]; then
|
||||||
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
|
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
|
||||||
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
|
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
|
||||||
|
|
||||||
${csudo}ln -s ${lib64_link_dir}/libtaosws.so ${lib64_link_dir}/libtaosws.so || :
|
${csudo}ln -s ${install_main_dir}/libtaosws.so ${lib64_link_dir}/libtaosws.so || :
|
||||||
fi
|
fi
|
||||||
|
|
||||||
${csudo}ldconfig
|
${csudo}ldconfig
|
||||||
|
|
|
@ -116,6 +116,7 @@ function install_bin() {
|
||||||
|
|
||||||
function clean_lib() {
|
function clean_lib() {
|
||||||
sudo rm -f /usr/lib/libtaos.* || :
|
sudo rm -f /usr/lib/libtaos.* || :
|
||||||
|
sudo rm -f /usr/lib/libtaosws.* || :
|
||||||
sudo rm -rf ${lib_dir} || :
|
sudo rm -rf ${lib_dir} || :
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -123,6 +124,9 @@ function install_lib() {
|
||||||
# Remove links
|
# Remove links
|
||||||
${csudo}rm -f ${lib_link_dir}/libtaos.* || :
|
${csudo}rm -f ${lib_link_dir}/libtaos.* || :
|
||||||
${csudo}rm -f ${lib64_link_dir}/libtaos.* || :
|
${csudo}rm -f ${lib64_link_dir}/libtaos.* || :
|
||||||
|
|
||||||
|
${csudo}rm -f ${lib_link_dir}/libtaosws.* || :
|
||||||
|
${csudo}rm -f ${lib64_link_dir}/libtaosws.* || :
|
||||||
#${csudo}rm -rf ${v15_java_app_dir} || :
|
#${csudo}rm -rf ${v15_java_app_dir} || :
|
||||||
|
|
||||||
${csudo}cp -rf ${script_dir}/driver/* ${install_main_dir}/driver && ${csudo}chmod 777 ${install_main_dir}/driver/*
|
${csudo}cp -rf ${script_dir}/driver/* ${install_main_dir}/driver && ${csudo}chmod 777 ${install_main_dir}/driver/*
|
||||||
|
@ -131,13 +135,19 @@ function install_lib() {
|
||||||
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
|
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.so.1
|
||||||
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
|
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
|
||||||
|
|
||||||
|
[ -f ${install_main_dir}/driver/libtaosws.so ] && ${csudo}ln -s ${install_main_dir}/driver/libtaosws.so ${lib_link_dir}/libtaosws.so
|
||||||
|
|
||||||
if [ -d "${lib64_link_dir}" ]; then
|
if [ -d "${lib64_link_dir}" ]; then
|
||||||
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
|
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
|
||||||
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
|
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
|
||||||
|
|
||||||
|
[ -f ${install_main_dir}/driver/libtaosws.so ] && ${csudo}ln -s ${install_main_dir}/driver/libtaosws.so ${lib64_link_dir}/libtaosws.so || :
|
||||||
fi
|
fi
|
||||||
else
|
else
|
||||||
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.1.dylib
|
${csudo}ln -s ${install_main_dir}/driver/libtaos.* ${lib_link_dir}/libtaos.1.dylib
|
||||||
${csudo}ln -s ${lib_link_dir}/libtaos.1.dylib ${lib_link_dir}/libtaos.dylib
|
${csudo}ln -s ${lib_link_dir}/libtaos.1.dylib ${lib_link_dir}/libtaos.dylib
|
||||||
|
|
||||||
|
[ -f ${install_main_dir}/driver/libtaosws.dylib ] && ${csudo}ln -s ${install_main_dir}/driver/libtaosws.dylib ${lib_link_dir}/libtaosws.dylib
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [ "$osType" != "Darwin" ]; then
|
if [ "$osType" != "Darwin" ]; then
|
||||||
|
@ -154,6 +164,8 @@ function install_header() {
|
||||||
${csudo}ln -s ${install_main_dir}/include/taosdef.h ${inc_link_dir}/taosdef.h
|
${csudo}ln -s ${install_main_dir}/include/taosdef.h ${inc_link_dir}/taosdef.h
|
||||||
${csudo}ln -s ${install_main_dir}/include/taoserror.h ${inc_link_dir}/taoserror.h
|
${csudo}ln -s ${install_main_dir}/include/taoserror.h ${inc_link_dir}/taoserror.h
|
||||||
${csudo}ln -s ${install_main_dir}/include/taosudf.h ${inc_link_dir}/taosudf.h
|
${csudo}ln -s ${install_main_dir}/include/taosudf.h ${inc_link_dir}/taosudf.h
|
||||||
|
|
||||||
|
[ -f ${install_main_dir}/include/taosws.h ] && ${csudo}ln -s ${install_main_dir}/include/taosws.h ${inc_link_dir}/taos.h
|
||||||
}
|
}
|
||||||
|
|
||||||
function install_jemalloc() {
|
function install_jemalloc() {
|
||||||
|
|
|
@ -57,12 +57,16 @@ if [ "$osType" != "Darwin" ]; then
|
||||||
${script_dir}/get_client.sh"
|
${script_dir}/get_client.sh"
|
||||||
fi
|
fi
|
||||||
lib_files="${build_dir}/lib/libtaos.so.${version}"
|
lib_files="${build_dir}/lib/libtaos.so.${version}"
|
||||||
|
wslib_files="${build_dir}/lib/libtaosws.so"
|
||||||
else
|
else
|
||||||
bin_files="${build_dir}/bin/${clientName} ${script_dir}/remove_client.sh"
|
bin_files="${build_dir}/bin/${clientName} ${script_dir}/remove_client.sh"
|
||||||
lib_files="${build_dir}/lib/libtaos.${version}.dylib"
|
lib_files="${build_dir}/lib/libtaos.${version}.dylib"
|
||||||
|
wslib_files="${build_dir}/lib/libtaosws.dylib"
|
||||||
fi
|
fi
|
||||||
|
|
||||||
header_files="${code_dir}/include/client/taos.h ${code_dir}/include/common/taosdef.h ${code_dir}/include/util/taoserror.h ${code_dir}/include/libs/function/taosudf.h"
|
header_files="${code_dir}/include/client/taos.h ${code_dir}/include/common/taosdef.h ${code_dir}/include/util/taoserror.h ${code_dir}/include/libs/function/taosudf.h"
|
||||||
|
wsheader_files="${build_dir}/include/taosws.h"
|
||||||
|
|
||||||
if [ "$dbName" != "taos" ]; then
|
if [ "$dbName" != "taos" ]; then
|
||||||
cfg_dir="${top_dir}/../enterprise/packaging/cfg"
|
cfg_dir="${top_dir}/../enterprise/packaging/cfg"
|
||||||
else
|
else
|
||||||
|
@ -74,6 +78,8 @@ install_files="${script_dir}/install_client.sh"
|
||||||
# make directories.
|
# make directories.
|
||||||
mkdir -p ${install_dir}
|
mkdir -p ${install_dir}
|
||||||
mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
|
mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
|
||||||
|
[ -f ${wsheader_files} ] && cp ${wsheader_files} ${install_dir}/inc
|
||||||
|
|
||||||
mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/${configFile} ${install_dir}/cfg/${configFile}
|
mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/${configFile} ${install_dir}/cfg/${configFile}
|
||||||
mkdir -p ${install_dir}/bin && cp ${bin_files} ${install_dir}/bin && chmod a+x ${install_dir}/bin/*
|
mkdir -p ${install_dir}/bin && cp ${bin_files} ${install_dir}/bin && chmod a+x ${install_dir}/bin/*
|
||||||
|
|
||||||
|
@ -187,6 +193,7 @@ cp ${lib_files} ${install_dir}/driver
|
||||||
# Copy connector
|
# Copy connector
|
||||||
connector_dir="${code_dir}/connector"
|
connector_dir="${code_dir}/connector"
|
||||||
mkdir -p ${install_dir}/connector
|
mkdir -p ${install_dir}/connector
|
||||||
|
[ -f ${wslib_files} ] && cp ${wslib_files} ${install_dir}/driver
|
||||||
|
|
||||||
if [[ "$pagMode" != "lite" ]] && [[ "$cpuType" != "aarch32" ]]; then
|
if [[ "$pagMode" != "lite" ]] && [[ "$cpuType" != "aarch32" ]]; then
|
||||||
if [ "$osType" != "Darwin" ]; then
|
if [ "$osType" != "Darwin" ]; then
|
||||||
|
|
|
@ -96,7 +96,7 @@ lib_files="${build_dir}/lib/libtaos.so.${version}"
|
||||||
wslib_files="${build_dir}/lib/libtaosws.so."
|
wslib_files="${build_dir}/lib/libtaosws.so."
|
||||||
header_files="${code_dir}/include/client/taos.h ${code_dir}/include/common/taosdef.h ${code_dir}/include/util/taoserror.h ${code_dir}/include/libs/function/taosudf.h"
|
header_files="${code_dir}/include/client/taos.h ${code_dir}/include/common/taosdef.h ${code_dir}/include/util/taoserror.h ${code_dir}/include/libs/function/taosudf.h"
|
||||||
|
|
||||||
wsheader_files="${code_dir}/inc/taosws.h"
|
wsheader_files="${build_dir}/include/taosws.h"
|
||||||
|
|
||||||
if [ "$dbName" != "taos" ]; then
|
if [ "$dbName" != "taos" ]; then
|
||||||
cfg_dir="${top_dir}/../enterprise/packaging/cfg"
|
cfg_dir="${top_dir}/../enterprise/packaging/cfg"
|
||||||
|
@ -114,7 +114,7 @@ init_file_rpm=${script_dir}/../rpm/taosd
|
||||||
mkdir -p ${install_dir}
|
mkdir -p ${install_dir}
|
||||||
mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
|
mkdir -p ${install_dir}/inc && cp ${header_files} ${install_dir}/inc
|
||||||
|
|
||||||
${wsheader_files} ${install_dir}/inc || :
|
[ -f ${wsheader_files} ] && cp ${wsheader_files} ${install_dir}/inc || :
|
||||||
|
|
||||||
mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/${configFile} ${install_dir}/cfg/${configFile}
|
mkdir -p ${install_dir}/cfg && cp ${cfg_dir}/${configFile} ${install_dir}/cfg/${configFile}
|
||||||
|
|
||||||
|
|
|
@ -82,22 +82,33 @@ function kill_taosd() {
|
||||||
|
|
||||||
function install_include() {
|
function install_include() {
|
||||||
${csudo}rm -f ${inc_link_dir}/taos.h ${inc_link_dir}/taosdef.h ${inc_link_dir}/taoserror.h ${inc_link_dir}/taosudf.h || :
|
${csudo}rm -f ${inc_link_dir}/taos.h ${inc_link_dir}/taosdef.h ${inc_link_dir}/taoserror.h ${inc_link_dir}/taosudf.h || :
|
||||||
|
${csudo}rm -f ${inc_link_dir}/taosws.h
|
||||||
|
|
||||||
${csudo}ln -s ${inc_dir}/taos.h ${inc_link_dir}/taos.h
|
${csudo}ln -s ${inc_dir}/taos.h ${inc_link_dir}/taos.h
|
||||||
${csudo}ln -s ${inc_dir}/taosdef.h ${inc_link_dir}/taosdef.h
|
${csudo}ln -s ${inc_dir}/taosdef.h ${inc_link_dir}/taosdef.h
|
||||||
${csudo}ln -s ${inc_dir}/taoserror.h ${inc_link_dir}/taoserror.h
|
${csudo}ln -s ${inc_dir}/taoserror.h ${inc_link_dir}/taoserror.h
|
||||||
${csudo}ln -s ${inc_dir}/taosudf.h ${inc_link_dir}/taosudf.h
|
${csudo}ln -s ${inc_dir}/taosudf.h ${inc_link_dir}/taosudf.h
|
||||||
|
|
||||||
|
[ -f ${inc_dir}/taosws.h ] && ${csudo}ln -s ${inc_dir}/taosudf.h ${inc_link_dir}/taosudf.h ||:
|
||||||
}
|
}
|
||||||
|
|
||||||
function install_lib() {
|
function install_lib() {
|
||||||
${csudo}rm -f ${lib_link_dir}/libtaos* || :
|
${csudo}rm -f ${lib_link_dir}/libtaos* || :
|
||||||
${csudo}rm -f ${lib64_link_dir}/libtaos* || :
|
${csudo}rm -f ${lib64_link_dir}/libtaos* || :
|
||||||
|
|
||||||
|
${csudo}rm -f ${lib_link_dir}/libtaosws* || :
|
||||||
|
${csudo}rm -f ${lib64_link_dir}/libtaosws* || :
|
||||||
|
|
||||||
${csudo}ln -s ${lib_dir}/libtaos.* ${lib_link_dir}/libtaos.so.1
|
${csudo}ln -s ${lib_dir}/libtaos.* ${lib_link_dir}/libtaos.so.1
|
||||||
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
|
${csudo}ln -s ${lib_link_dir}/libtaos.so.1 ${lib_link_dir}/libtaos.so
|
||||||
|
|
||||||
|
[ -f ${lib_dir}/libtaosws.so ]${csudo}ln -s ${lib_dir}/libtaosws.so ${lib_link_dir}/libtaosws.so
|
||||||
|
|
||||||
if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.so ]]; then
|
if [[ -d ${lib64_link_dir} && ! -e ${lib64_link_dir}/libtaos.so ]]; then
|
||||||
${csudo}ln -s ${lib_dir}/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
|
${csudo}ln -s ${lib_dir}/libtaos.* ${lib64_link_dir}/libtaos.so.1 || :
|
||||||
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
|
${csudo}ln -s ${lib64_link_dir}/libtaos.so.1 ${lib64_link_dir}/libtaos.so || :
|
||||||
|
|
||||||
|
[ -ff ${lib_dir}/libtaosws.so ] && ${csudo}ln -s ${lib_dir}/libtaosws.so ${lib64_link_dir}/libtaosws.so || :
|
||||||
fi
|
fi
|
||||||
|
|
||||||
${csudo}ldconfig
|
${csudo}ldconfig
|
||||||
|
|
|
@ -286,13 +286,10 @@ static int32_t hbAsyncCallBack(void *param, SDataBuf *pMsg, int32_t code) {
|
||||||
if (pInst == NULL || NULL == *pInst) {
|
if (pInst == NULL || NULL == *pInst) {
|
||||||
taosThreadMutexUnlock(&appInfo.mutex);
|
taosThreadMutexUnlock(&appInfo.mutex);
|
||||||
tscError("cluster not exist, key:%s", key);
|
tscError("cluster not exist, key:%s", key);
|
||||||
taosMemoryFreeClear(param);
|
|
||||||
tFreeClientHbBatchRsp(&pRsp);
|
tFreeClientHbBatchRsp(&pRsp);
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
|
||||||
taosMemoryFreeClear(param);
|
|
||||||
|
|
||||||
if (code != 0) {
|
if (code != 0) {
|
||||||
(*pInst)->onlineDnodes = ((*pInst)->totalDnodes ? 0 : -1);
|
(*pInst)->onlineDnodes = ((*pInst)->totalDnodes ? 0 : -1);
|
||||||
tscDebug("hb rsp error %s, update server status %d/%d", tstrerror(code), (*pInst)->onlineDnodes, (*pInst)->totalDnodes);
|
tscDebug("hb rsp error %s, update server status %d/%d", tstrerror(code), (*pInst)->onlineDnodes, (*pInst)->totalDnodes);
|
||||||
|
@ -716,6 +713,7 @@ static void *hbThreadFunc(void *param) {
|
||||||
pInfo->msgInfo.len = tlen;
|
pInfo->msgInfo.len = tlen;
|
||||||
pInfo->msgType = TDMT_MND_HEARTBEAT;
|
pInfo->msgType = TDMT_MND_HEARTBEAT;
|
||||||
pInfo->param = strdup(pAppHbMgr->key);
|
pInfo->param = strdup(pAppHbMgr->key);
|
||||||
|
pInfo->paramFreeFp = taosMemoryFree;
|
||||||
pInfo->requestId = generateRequestId();
|
pInfo->requestId = generateRequestId();
|
||||||
pInfo->requestObjRefId = 0;
|
pInfo->requestObjRefId = 0;
|
||||||
|
|
||||||
|
|
|
@ -29,7 +29,6 @@
|
||||||
|
|
||||||
static int32_t initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* pEpSet);
|
static int32_t initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* pEpSet);
|
||||||
static SMsgSendInfo* buildConnectMsg(SRequestObj* pRequest);
|
static SMsgSendInfo* buildConnectMsg(SRequestObj* pRequest);
|
||||||
static void destroySendMsgInfo(SMsgSendInfo* pMsgBody);
|
|
||||||
|
|
||||||
static bool stringLengthCheck(const char* str, size_t maxsize) {
|
static bool stringLengthCheck(const char* str, size_t maxsize) {
|
||||||
if (str == NULL) {
|
if (str == NULL) {
|
||||||
|
@ -1215,13 +1214,6 @@ static SMsgSendInfo* buildConnectMsg(SRequestObj* pRequest) {
|
||||||
return pMsgSendInfo;
|
return pMsgSendInfo;
|
||||||
}
|
}
|
||||||
|
|
||||||
static void destroySendMsgInfo(SMsgSendInfo* pMsgBody) {
|
|
||||||
assert(pMsgBody != NULL);
|
|
||||||
taosMemoryFreeClear(pMsgBody->target.dbFName);
|
|
||||||
taosMemoryFreeClear(pMsgBody->msgInfo.pData);
|
|
||||||
taosMemoryFreeClear(pMsgBody);
|
|
||||||
}
|
|
||||||
|
|
||||||
void updateTargetEpSet(SMsgSendInfo* pSendInfo, STscObj* pTscObj, SRpcMsg* pMsg, SEpSet* pEpSet) {
|
void updateTargetEpSet(SMsgSendInfo* pSendInfo, STscObj* pTscObj, SRpcMsg* pMsg, SEpSet* pEpSet) {
|
||||||
if (NULL == pEpSet) {
|
if (NULL == pEpSet) {
|
||||||
return;
|
return;
|
||||||
|
@ -2101,8 +2093,9 @@ TAOS_RES* taosQueryImpl(TAOS* taos, const char* sql, bool validateOnly) {
|
||||||
|
|
||||||
taosAsyncQueryImpl(*(int64_t*)taos, sql, syncQueryFn, param, validateOnly);
|
taosAsyncQueryImpl(*(int64_t*)taos, sql, syncQueryFn, param, validateOnly);
|
||||||
tsem_wait(¶m->sem);
|
tsem_wait(¶m->sem);
|
||||||
|
if (param->pRequest != NULL) {
|
||||||
param->pRequest->syncQuery = true;
|
param->pRequest->syncQuery = true;
|
||||||
|
}
|
||||||
return param->pRequest;
|
return param->pRequest;
|
||||||
#else
|
#else
|
||||||
size_t sqlLen = strlen(sql);
|
size_t sqlLen = strlen(sql);
|
||||||
|
|
|
@ -255,6 +255,8 @@ int32_t processDropDbRsp(void* param, SDataBuf* pMsg, int32_t code) {
|
||||||
catalogRemoveDB(pCatalog, dropdbRsp.db, dropdbRsp.uid);
|
catalogRemoveDB(pCatalog, dropdbRsp.db, dropdbRsp.uid);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
taosMemoryFree(pMsg->pData);
|
||||||
|
|
||||||
if (pRequest->body.queryFp != NULL) {
|
if (pRequest->body.queryFp != NULL) {
|
||||||
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
|
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
|
||||||
} else {
|
} else {
|
||||||
|
@ -278,6 +280,8 @@ int32_t processAlterStbRsp(void* param, SDataBuf* pMsg, int32_t code) {
|
||||||
pRequest->body.resInfo.execRes.res = alterRsp.pMeta;
|
pRequest->body.resInfo.execRes.res = alterRsp.pMeta;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
taosMemoryFree(pMsg->pData);
|
||||||
|
|
||||||
if (pRequest->body.queryFp != NULL) {
|
if (pRequest->body.queryFp != NULL) {
|
||||||
SExecResult* pRes = &pRequest->body.resInfo.execRes;
|
SExecResult* pRes = &pRequest->body.resInfo.execRes;
|
||||||
|
|
||||||
|
@ -387,6 +391,8 @@ int32_t processShowVariablesRsp(void* param, SDataBuf* pMsg, int32_t code) {
|
||||||
tFreeSShowVariablesRsp(&rsp);
|
tFreeSShowVariablesRsp(&rsp);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
taosMemoryFree(pMsg->pData);
|
||||||
|
|
||||||
if (pRequest->body.queryFp != NULL) {
|
if (pRequest->body.queryFp != NULL) {
|
||||||
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
|
pRequest->body.queryFp(pRequest->body.param, pRequest, code);
|
||||||
} else {
|
} else {
|
||||||
|
|
|
@ -268,7 +268,7 @@ static int32_t smlGenerateSchemaAction(SSchema *colField, SHashObj *colHash, SSm
|
||||||
*actionNeeded = true;
|
*actionNeeded = true;
|
||||||
}
|
}
|
||||||
if (*actionNeeded) {
|
if (*actionNeeded) {
|
||||||
uDebug("SML:0x%" PRIx64 " generate schema action. column name: %s, action: %d", info->id, colField->name,
|
uDebug("SML:0x%" PRIx64 " generate schema action. kv->name: %s, action: %d", info->id, kv->key,
|
||||||
action->action);
|
action->action);
|
||||||
}
|
}
|
||||||
return 0;
|
return 0;
|
||||||
|
@ -436,6 +436,7 @@ static int32_t smlProcessSchemaAction(SSmlHandle *info, SSchema *schemaField, SH
|
||||||
SSchemaAction *action, bool isTag) {
|
SSchemaAction *action, bool isTag) {
|
||||||
int32_t code = TSDB_CODE_SUCCESS;
|
int32_t code = TSDB_CODE_SUCCESS;
|
||||||
for (int j = 0; j < taosArrayGetSize(cols); ++j) {
|
for (int j = 0; j < taosArrayGetSize(cols); ++j) {
|
||||||
|
if(j == 0 && !isTag) continue;
|
||||||
SSmlKv *kv = (SSmlKv *)taosArrayGetP(cols, j);
|
SSmlKv *kv = (SSmlKv *)taosArrayGetP(cols, j);
|
||||||
bool actionNeeded = false;
|
bool actionNeeded = false;
|
||||||
code = smlGenerateSchemaAction(schemaField, schemaHash, kv, isTag, action, &actionNeeded, info);
|
code = smlGenerateSchemaAction(schemaField, schemaHash, kv, isTag, action, &actionNeeded, info);
|
||||||
|
@ -452,18 +453,25 @@ static int32_t smlProcessSchemaAction(SSmlHandle *info, SSchema *schemaField, SH
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
|
||||||
static int32_t smlCheckMeta(SSchema *schema, int32_t length, SArray *cols) {
|
static int32_t smlCheckMeta(SSchema *schema, int32_t length, SArray *cols, bool isTag) {
|
||||||
SHashObj *hashTmp = taosHashInit(length, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
|
SHashObj *hashTmp = taosHashInit(length, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), true, HASH_NO_LOCK);
|
||||||
for (uint16_t i = 0; i < length; i++) {
|
int32_t i = 0;
|
||||||
|
for ( ;i < length; i++) {
|
||||||
taosHashPut(hashTmp, schema[i].name, strlen(schema[i].name), &i, SHORT_BYTES);
|
taosHashPut(hashTmp, schema[i].name, strlen(schema[i].name), &i, SHORT_BYTES);
|
||||||
}
|
}
|
||||||
|
|
||||||
for (int32_t i = 0; i < taosArrayGetSize(cols); i++) {
|
if (isTag){
|
||||||
|
i = 0;
|
||||||
|
} else {
|
||||||
|
i = 1;
|
||||||
|
}
|
||||||
|
for (; i < taosArrayGetSize(cols); i++) {
|
||||||
SSmlKv *kv = (SSmlKv *)taosArrayGetP(cols, i);
|
SSmlKv *kv = (SSmlKv *)taosArrayGetP(cols, i);
|
||||||
if (taosHashGet(hashTmp, kv->key, kv->keyLen) == NULL) {
|
if (taosHashGet(hashTmp, kv->key, kv->keyLen) == NULL) {
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
taosHashCleanup(hashTmp);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -523,7 +531,7 @@ static int32_t smlModifyDBSchemas(SSmlHandle *info) {
|
||||||
}
|
}
|
||||||
|
|
||||||
taosHashClear(hashTmp);
|
taosHashClear(hashTmp);
|
||||||
for (uint16_t i = 0; i < pTableMeta->tableInfo.numOfColumns; i++) {
|
for (uint16_t i = 1; i < pTableMeta->tableInfo.numOfColumns; i++) {
|
||||||
taosHashPut(hashTmp, pTableMeta->schema[i].name, strlen(pTableMeta->schema[i].name), &i, SHORT_BYTES);
|
taosHashPut(hashTmp, pTableMeta->schema[i].name, strlen(pTableMeta->schema[i].name), &i, SHORT_BYTES);
|
||||||
}
|
}
|
||||||
code = smlProcessSchemaAction(info, pTableMeta->schema, hashTmp, sTableData->cols, &schemaAction, false);
|
code = smlProcessSchemaAction(info, pTableMeta->schema, hashTmp, sTableData->cols, &schemaAction, false);
|
||||||
|
@ -551,12 +559,12 @@ static int32_t smlModifyDBSchemas(SSmlHandle *info) {
|
||||||
|
|
||||||
if (needCheckMeta) {
|
if (needCheckMeta) {
|
||||||
code = smlCheckMeta(&(pTableMeta->schema[pTableMeta->tableInfo.numOfColumns]), pTableMeta->tableInfo.numOfTags,
|
code = smlCheckMeta(&(pTableMeta->schema[pTableMeta->tableInfo.numOfColumns]), pTableMeta->tableInfo.numOfTags,
|
||||||
sTableData->tags);
|
sTableData->tags, true);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
uError("SML:0x%" PRIx64 " check tag failed. super table name %s", info->id, (char *)superTable);
|
uError("SML:0x%" PRIx64 " check tag failed. super table name %s", info->id, (char *)superTable);
|
||||||
goto end;
|
goto end;
|
||||||
}
|
}
|
||||||
code = smlCheckMeta(&(pTableMeta->schema[0]), pTableMeta->tableInfo.numOfColumns, sTableData->cols);
|
code = smlCheckMeta(&(pTableMeta->schema[0]), pTableMeta->tableInfo.numOfColumns, sTableData->cols, false);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
uError("SML:0x%" PRIx64 " check cols failed. super table name %s", info->id, (char *)superTable);
|
uError("SML:0x%" PRIx64 " check cols failed. super table name %s", info->id, (char *)superTable);
|
||||||
goto end;
|
goto end;
|
||||||
|
@ -832,6 +840,7 @@ static int64_t smlParseOpenTsdbTime(SSmlHandle *info, const char *data, int32_t
|
||||||
static int32_t smlParseTS(SSmlHandle *info, const char *data, int32_t len, SArray *cols) {
|
static int32_t smlParseTS(SSmlHandle *info, const char *data, int32_t len, SArray *cols) {
|
||||||
int64_t ts = 0;
|
int64_t ts = 0;
|
||||||
if (info->protocol == TSDB_SML_LINE_PROTOCOL) {
|
if (info->protocol == TSDB_SML_LINE_PROTOCOL) {
|
||||||
|
// uError("SML:data:%s,len:%d", data, len);
|
||||||
ts = smlParseInfluxTime(info, data, len);
|
ts = smlParseInfluxTime(info, data, len);
|
||||||
} else if (info->protocol == TSDB_SML_TELNET_PROTOCOL) {
|
} else if (info->protocol == TSDB_SML_TELNET_PROTOCOL) {
|
||||||
ts = smlParseOpenTsdbTime(info, data, len);
|
ts = smlParseOpenTsdbTime(info, data, len);
|
||||||
|
@ -1208,6 +1217,9 @@ static int32_t smlParseCols(const char *data, int32_t len, SArray *cols, char *c
|
||||||
kv->value = value;
|
kv->value = value;
|
||||||
kv->length = valueLen;
|
kv->length = valueLen;
|
||||||
if (isTag) {
|
if (isTag) {
|
||||||
|
if(valueLen > (TSDB_MAX_NCHAR_LEN - VARSTR_HEADER_SIZE) / TSDB_NCHAR_SIZE){
|
||||||
|
return TSDB_CODE_PAR_INVALID_VAR_COLUMN_LEN;
|
||||||
|
}
|
||||||
kv->type = TSDB_DATA_TYPE_NCHAR;
|
kv->type = TSDB_DATA_TYPE_NCHAR;
|
||||||
} else {
|
} else {
|
||||||
int32_t ret = smlParseValue(kv, msg);
|
int32_t ret = smlParseValue(kv, msg);
|
||||||
|
@ -2031,6 +2043,8 @@ static int32_t smlParseJSONString(SSmlHandle *info, cJSON *root, SSmlTableInfo *
|
||||||
|
|
||||||
static int32_t smlParseInfluxLine(SSmlHandle *info, const char *sql) {
|
static int32_t smlParseInfluxLine(SSmlHandle *info, const char *sql) {
|
||||||
SSmlLineInfo elements = {0};
|
SSmlLineInfo elements = {0};
|
||||||
|
uDebug("SML:0x%" PRIx64 " smlParseInfluxLine sql:%s, hello", info->id, sql);
|
||||||
|
|
||||||
int ret = smlParseInfluxString(sql, &elements, &info->msgBuf);
|
int ret = smlParseInfluxString(sql, &elements, &info->msgBuf);
|
||||||
if (ret != TSDB_CODE_SUCCESS) {
|
if (ret != TSDB_CODE_SUCCESS) {
|
||||||
uError("SML:0x%" PRIx64 " smlParseInfluxLine failed", info->id);
|
uError("SML:0x%" PRIx64 " smlParseInfluxLine failed", info->id);
|
||||||
|
|
|
@ -113,12 +113,6 @@ struct tmq_t {
|
||||||
tsem_t rspSem;
|
tsem_t rspSem;
|
||||||
};
|
};
|
||||||
|
|
||||||
struct tmq_raw_data {
|
|
||||||
void* raw_meta;
|
|
||||||
int32_t raw_meta_len;
|
|
||||||
int16_t raw_meta_type;
|
|
||||||
};
|
|
||||||
|
|
||||||
enum {
|
enum {
|
||||||
TMQ_VG_STATUS__IDLE = 0,
|
TMQ_VG_STATUS__IDLE = 0,
|
||||||
TMQ_VG_STATUS__WAIT,
|
TMQ_VG_STATUS__WAIT,
|
||||||
|
@ -510,6 +504,7 @@ static int32_t tmqSendCommitReq(tmq_t* tmq, SMqClientVg* pVg, SMqClientTopic* pT
|
||||||
pMsgSendInfo->requestId = generateRequestId();
|
pMsgSendInfo->requestId = generateRequestId();
|
||||||
pMsgSendInfo->requestObjRefId = 0;
|
pMsgSendInfo->requestObjRefId = 0;
|
||||||
pMsgSendInfo->param = pParam;
|
pMsgSendInfo->param = pParam;
|
||||||
|
pMsgSendInfo->paramFreeFp = taosMemoryFree;
|
||||||
pMsgSendInfo->fp = tmqCommitCb2;
|
pMsgSendInfo->fp = tmqCommitCb2;
|
||||||
pMsgSendInfo->msgType = TDMT_VND_MQ_COMMIT_OFFSET;
|
pMsgSendInfo->msgType = TDMT_VND_MQ_COMMIT_OFFSET;
|
||||||
// send msg
|
// send msg
|
||||||
|
@ -1918,16 +1913,15 @@ const char* tmq_get_table_name(TAOS_RES* res) {
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
tmq_raw_data* tmq_get_raw_meta(TAOS_RES* res) {
|
int32_t tmq_get_raw_meta(TAOS_RES* res, tmq_raw_data *raw) {
|
||||||
if (TD_RES_TMQ_META(res)) {
|
if (TD_RES_TMQ_META(res) && raw) {
|
||||||
tmq_raw_data* raw = taosMemoryCalloc(1, sizeof(tmq_raw_data));
|
|
||||||
SMqMetaRspObj* pMetaRspObj = (SMqMetaRspObj*)res;
|
SMqMetaRspObj* pMetaRspObj = (SMqMetaRspObj*)res;
|
||||||
raw->raw_meta = pMetaRspObj->metaRsp.metaRsp;
|
raw->raw_meta = pMetaRspObj->metaRsp.metaRsp;
|
||||||
raw->raw_meta_len = pMetaRspObj->metaRsp.metaRspLen;
|
raw->raw_meta_len = pMetaRspObj->metaRsp.metaRspLen;
|
||||||
raw->raw_meta_type = pMetaRspObj->metaRsp.resMsgType;
|
raw->raw_meta_type = pMetaRspObj->metaRsp.resMsgType;
|
||||||
return raw;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
return NULL;
|
return TSDB_CODE_INVALID_PARA;
|
||||||
}
|
}
|
||||||
|
|
||||||
static char* buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id,
|
static char* buildCreateTableJson(SSchemaWrapper* schemaRow, SSchemaWrapper* schemaTag, char* name, int64_t id,
|
||||||
|
@ -2199,17 +2193,19 @@ static char* buildCreateCTableJson(STag* pTag, char* sname, char* name, SArray*
|
||||||
cJSON* ttype = cJSON_CreateNumber(pTagVal->type);
|
cJSON* ttype = cJSON_CreateNumber(pTagVal->type);
|
||||||
cJSON_AddItemToObject(tag, "type", ttype);
|
cJSON_AddItemToObject(tag, "type", ttype);
|
||||||
|
|
||||||
char* buf = NULL;
|
cJSON* tvalue = NULL;
|
||||||
if (IS_VAR_DATA_TYPE(pTagVal->type)) {
|
if (IS_VAR_DATA_TYPE(pTagVal->type)) {
|
||||||
buf = taosMemoryCalloc(pTagVal->nData + 1, 1);
|
char* buf = taosMemoryCalloc(pTagVal->nData + 3, 1);
|
||||||
|
if(!buf) goto end;
|
||||||
dataConverToStr(buf, pTagVal->type, pTagVal->pData, pTagVal->nData, NULL);
|
dataConverToStr(buf, pTagVal->type, pTagVal->pData, pTagVal->nData, NULL);
|
||||||
|
tvalue = cJSON_CreateString(buf);
|
||||||
|
taosMemoryFree(buf);
|
||||||
} else {
|
} else {
|
||||||
buf = taosMemoryCalloc(32, 1);
|
double val = 0;
|
||||||
dataConverToStr(buf, pTagVal->type, &pTagVal->i64, tDataTypes[pTagVal->type].bytes, NULL);
|
GET_TYPED_DATA(val, double, pTagVal->type, &pTagVal->i64);
|
||||||
|
tvalue = cJSON_CreateNumber(val);
|
||||||
}
|
}
|
||||||
|
|
||||||
cJSON* tvalue = cJSON_CreateString(buf);
|
|
||||||
taosMemoryFree(buf);
|
|
||||||
cJSON_AddItemToObject(tag, "value", tvalue);
|
cJSON_AddItemToObject(tag, "value", tvalue);
|
||||||
cJSON_AddItemToArray(tags, tag);
|
cJSON_AddItemToArray(tags, tag);
|
||||||
}
|
}
|
||||||
|
@ -2935,23 +2931,23 @@ end:
|
||||||
return code;
|
return code;
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t taos_write_raw_meta(TAOS* taos, tmq_raw_data* raw_meta) {
|
int32_t taos_write_raw_meta(TAOS *taos, tmq_raw_data raw_meta){
|
||||||
if (!taos || !raw_meta) {
|
if (!taos) {
|
||||||
return TSDB_CODE_INVALID_PARA;
|
return TSDB_CODE_INVALID_PARA;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (raw_meta->raw_meta_type == TDMT_VND_CREATE_STB) {
|
if(raw_meta.raw_meta_type == TDMT_VND_CREATE_STB) {
|
||||||
return taosCreateStb(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
|
return taosCreateStb(taos, raw_meta.raw_meta, raw_meta.raw_meta_len);
|
||||||
} else if (raw_meta->raw_meta_type == TDMT_VND_ALTER_STB) {
|
}else if(raw_meta.raw_meta_type == TDMT_VND_ALTER_STB){
|
||||||
return taosCreateStb(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
|
return taosCreateStb(taos, raw_meta.raw_meta, raw_meta.raw_meta_len);
|
||||||
} else if (raw_meta->raw_meta_type == TDMT_VND_DROP_STB) {
|
}else if(raw_meta.raw_meta_type == TDMT_VND_DROP_STB){
|
||||||
return taosDropStb(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
|
return taosDropStb(taos, raw_meta.raw_meta, raw_meta.raw_meta_len);
|
||||||
} else if (raw_meta->raw_meta_type == TDMT_VND_CREATE_TABLE) {
|
}else if(raw_meta.raw_meta_type == TDMT_VND_CREATE_TABLE){
|
||||||
return taosCreateTable(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
|
return taosCreateTable(taos, raw_meta.raw_meta, raw_meta.raw_meta_len);
|
||||||
} else if (raw_meta->raw_meta_type == TDMT_VND_ALTER_TABLE) {
|
}else if(raw_meta.raw_meta_type == TDMT_VND_ALTER_TABLE){
|
||||||
return taosAlterTable(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
|
return taosAlterTable(taos, raw_meta.raw_meta, raw_meta.raw_meta_len);
|
||||||
} else if (raw_meta->raw_meta_type == TDMT_VND_DROP_TABLE) {
|
}else if(raw_meta.raw_meta_type == TDMT_VND_DROP_TABLE){
|
||||||
return taosDropTable(taos, raw_meta->raw_meta, raw_meta->raw_meta_len);
|
return taosDropTable(taos, raw_meta.raw_meta, raw_meta.raw_meta_len);
|
||||||
}
|
}
|
||||||
return TSDB_CODE_INVALID_PARA;
|
return TSDB_CODE_INVALID_PARA;
|
||||||
}
|
}
|
||||||
|
|
File diff suppressed because one or more lines are too long
|
@ -556,7 +556,7 @@ static void taosSetSystemCfg(SConfig *pCfg) {
|
||||||
osSetSystemLocale(locale, charset);
|
osSetSystemLocale(locale, charset);
|
||||||
|
|
||||||
bool enableCore = cfgGetItem(pCfg, "enableCoreFile")->bval;
|
bool enableCore = cfgGetItem(pCfg, "enableCoreFile")->bval;
|
||||||
taosSetConsoleEcho(enableCore);
|
taosSetCoreDump(enableCore);
|
||||||
|
|
||||||
// todo
|
// todo
|
||||||
tsVersion = 30000000;
|
tsVersion = 30000000;
|
||||||
|
@ -675,7 +675,7 @@ int32_t taosSetCfg(SConfig *pCfg, char *name) {
|
||||||
case 'e': {
|
case 'e': {
|
||||||
if (strcasecmp("enableCoreFile", name) == 0) {
|
if (strcasecmp("enableCoreFile", name) == 0) {
|
||||||
bool enableCore = cfgGetItem(pCfg, "enableCoreFile")->bval;
|
bool enableCore = cfgGetItem(pCfg, "enableCoreFile")->bval;
|
||||||
taosSetConsoleEcho(enableCore);
|
taosSetCoreDump(enableCore);
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
|
@ -49,9 +49,9 @@ static void dmProcessStatusRsp(SDnodeMgmt *pMgmt, SRpcMsg *pRsp) {
|
||||||
dmUpdateEps(pMgmt->pData, statusRsp.pDnodeEps);
|
dmUpdateEps(pMgmt->pData, statusRsp.pDnodeEps);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
rpcFreeCont(pRsp->pCont);
|
|
||||||
tFreeSStatusRsp(&statusRsp);
|
tFreeSStatusRsp(&statusRsp);
|
||||||
}
|
}
|
||||||
|
rpcFreeCont(pRsp->pCont);
|
||||||
}
|
}
|
||||||
|
|
||||||
void dmSendStatusReq(SDnodeMgmt *pMgmt) {
|
void dmSendStatusReq(SDnodeMgmt *pMgmt) {
|
||||||
|
|
|
@ -35,6 +35,46 @@
|
||||||
extern "C" {
|
extern "C" {
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
|
typedef enum {
|
||||||
|
MND_OPER_CONNECT = 1,
|
||||||
|
MND_OPER_CREATE_ACCT,
|
||||||
|
MND_OPER_DROP_ACCT,
|
||||||
|
MND_OPER_ALTER_ACCT,
|
||||||
|
MND_OPER_CREATE_USER,
|
||||||
|
MND_OPER_DROP_USER,
|
||||||
|
MND_OPER_ALTER_USER,
|
||||||
|
MND_OPER_CREATE_BNODE,
|
||||||
|
MND_OPER_DROP_BNODE,
|
||||||
|
MND_OPER_CREATE_DNODE,
|
||||||
|
MND_OPER_DROP_DNODE,
|
||||||
|
MND_OPER_CONFIG_DNODE,
|
||||||
|
MND_OPER_CREATE_MNODE,
|
||||||
|
MND_OPER_DROP_MNODE,
|
||||||
|
MND_OPER_CREATE_QNODE,
|
||||||
|
MND_OPER_DROP_QNODE,
|
||||||
|
MND_OPER_CREATE_SNODE,
|
||||||
|
MND_OPER_DROP_SNODE,
|
||||||
|
MND_OPER_REDISTRIBUTE_VGROUP,
|
||||||
|
MND_OPER_MERGE_VGROUP,
|
||||||
|
MND_OPER_SPLIT_VGROUP,
|
||||||
|
MND_OPER_BALANCE_VGROUP,
|
||||||
|
MND_OPER_CREATE_FUNC,
|
||||||
|
MND_OPER_DROP_FUNC,
|
||||||
|
MND_OPER_KILL_TRANS,
|
||||||
|
MND_OPER_KILL_CONN,
|
||||||
|
MND_OPER_KILL_QUERY,
|
||||||
|
MND_OPER_CREATE_DB,
|
||||||
|
MND_OPER_ALTER_DB,
|
||||||
|
MND_OPER_DROP_DB,
|
||||||
|
MND_OPER_COMPACT_DB,
|
||||||
|
MND_OPER_TRIM_DB,
|
||||||
|
MND_OPER_USE_DB,
|
||||||
|
MND_OPER_WRITE_DB,
|
||||||
|
MND_OPER_READ_DB,
|
||||||
|
MND_OPER_READ_OR_WRITE_DB,
|
||||||
|
MND_OPER_SHOW_VARIBALES,
|
||||||
|
} EOperType;
|
||||||
|
|
||||||
typedef enum {
|
typedef enum {
|
||||||
MND_AUTH_ACCT_START = 0,
|
MND_AUTH_ACCT_START = 0,
|
||||||
MND_AUTH_ACCT_USER,
|
MND_AUTH_ACCT_USER,
|
||||||
|
@ -109,9 +149,9 @@ typedef struct {
|
||||||
ETrnPolicy policy;
|
ETrnPolicy policy;
|
||||||
ETrnConflct conflict;
|
ETrnConflct conflict;
|
||||||
ETrnExec exec;
|
ETrnExec exec;
|
||||||
|
EOperType oper;
|
||||||
int32_t code;
|
int32_t code;
|
||||||
int32_t failedTimes;
|
int32_t failedTimes;
|
||||||
SRpcHandleInfo rpcInfo;
|
|
||||||
void* rpcRsp;
|
void* rpcRsp;
|
||||||
int32_t rpcRspLen;
|
int32_t rpcRspLen;
|
||||||
int32_t redoActionPos;
|
int32_t redoActionPos;
|
||||||
|
@ -130,6 +170,7 @@ typedef struct {
|
||||||
int32_t stopFunc;
|
int32_t stopFunc;
|
||||||
int32_t paramLen;
|
int32_t paramLen;
|
||||||
void* param;
|
void* param;
|
||||||
|
SArray* pRpcArray;
|
||||||
} STrans;
|
} STrans;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
|
|
|
@ -22,46 +22,6 @@
|
||||||
extern "C" {
|
extern "C" {
|
||||||
#endif
|
#endif
|
||||||
|
|
||||||
typedef enum {
|
|
||||||
MND_OPER_CONNECT = 1,
|
|
||||||
MND_OPER_CREATE_ACCT,
|
|
||||||
MND_OPER_DROP_ACCT,
|
|
||||||
MND_OPER_ALTER_ACCT,
|
|
||||||
MND_OPER_CREATE_USER,
|
|
||||||
MND_OPER_DROP_USER,
|
|
||||||
MND_OPER_ALTER_USER,
|
|
||||||
MND_OPER_CREATE_BNODE,
|
|
||||||
MND_OPER_DROP_BNODE,
|
|
||||||
MND_OPER_CREATE_DNODE,
|
|
||||||
MND_OPER_DROP_DNODE,
|
|
||||||
MND_OPER_CONFIG_DNODE,
|
|
||||||
MND_OPER_CREATE_MNODE,
|
|
||||||
MND_OPER_DROP_MNODE,
|
|
||||||
MND_OPER_CREATE_QNODE,
|
|
||||||
MND_OPER_DROP_QNODE,
|
|
||||||
MND_OPER_CREATE_SNODE,
|
|
||||||
MND_OPER_DROP_SNODE,
|
|
||||||
MND_OPER_REDISTRIBUTE_VGROUP,
|
|
||||||
MND_OPER_MERGE_VGROUP,
|
|
||||||
MND_OPER_SPLIT_VGROUP,
|
|
||||||
MND_OPER_BALANCE_VGROUP,
|
|
||||||
MND_OPER_CREATE_FUNC,
|
|
||||||
MND_OPER_DROP_FUNC,
|
|
||||||
MND_OPER_KILL_TRANS,
|
|
||||||
MND_OPER_KILL_CONN,
|
|
||||||
MND_OPER_KILL_QUERY,
|
|
||||||
MND_OPER_CREATE_DB,
|
|
||||||
MND_OPER_ALTER_DB,
|
|
||||||
MND_OPER_DROP_DB,
|
|
||||||
MND_OPER_COMPACT_DB,
|
|
||||||
MND_OPER_TRIM_DB,
|
|
||||||
MND_OPER_USE_DB,
|
|
||||||
MND_OPER_WRITE_DB,
|
|
||||||
MND_OPER_READ_DB,
|
|
||||||
MND_OPER_READ_OR_WRITE_DB,
|
|
||||||
MND_OPER_SHOW_VARIBALES,
|
|
||||||
} EOperType;
|
|
||||||
|
|
||||||
int32_t mndInitPrivilege(SMnode *pMnode);
|
int32_t mndInitPrivilege(SMnode *pMnode);
|
||||||
void mndCleanupPrivilege(SMnode *pMnode);
|
void mndCleanupPrivilege(SMnode *pMnode);
|
||||||
|
|
||||||
|
|
|
@ -73,12 +73,14 @@ void mndTransSetRpcRsp(STrans *pTrans, void *pCont, int32_t contLen);
|
||||||
void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *param, int32_t paramLen);
|
void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *param, int32_t paramLen);
|
||||||
void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2);
|
void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2);
|
||||||
void mndTransSetSerial(STrans *pTrans);
|
void mndTransSetSerial(STrans *pTrans);
|
||||||
|
void mndTransSetOper(STrans *pTrans, EOperType oper);
|
||||||
|
|
||||||
int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans);
|
int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans);
|
||||||
int32_t mndTransProcessRsp(SRpcMsg *pRsp);
|
int32_t mndTransProcessRsp(SRpcMsg *pRsp);
|
||||||
void mndTransPullup(SMnode *pMnode);
|
void mndTransPullup(SMnode *pMnode);
|
||||||
int32_t mndKillTrans(SMnode *pMnode, STrans *pTrans);
|
int32_t mndKillTrans(SMnode *pMnode, STrans *pTrans);
|
||||||
void mndTransExecute(SMnode *pMnode, STrans *pTrans);
|
void mndTransExecute(SMnode *pMnode, STrans *pTrans);
|
||||||
|
int32_t mndSetRpcInfoForDbTrans(SMnode *pMnode, SRpcMsg *pMsg, EOperType oper, const char *dbname);
|
||||||
|
|
||||||
#ifdef __cplusplus
|
#ifdef __cplusplus
|
||||||
}
|
}
|
||||||
|
|
|
@ -487,6 +487,7 @@ static int32_t mndCreateDb(SMnode *pMnode, SRpcMsg *pReq, SCreateDbReq *pCreate,
|
||||||
mDebug("trans:%d, used to create db:%s", pTrans->id, pCreate->db);
|
mDebug("trans:%d, used to create db:%s", pTrans->id, pCreate->db);
|
||||||
|
|
||||||
mndTransSetDbName(pTrans, dbObj.name, NULL);
|
mndTransSetDbName(pTrans, dbObj.name, NULL);
|
||||||
|
mndTransSetOper(pTrans, MND_OPER_CREATE_DB);
|
||||||
if (mndSetCreateDbRedoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
|
if (mndSetCreateDbRedoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
|
||||||
if (mndSetCreateDbUndoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
|
if (mndSetCreateDbUndoLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
|
||||||
if (mndSetCreateDbCommitLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
|
if (mndSetCreateDbCommitLogs(pMnode, pTrans, &dbObj, pVgroups) != 0) goto _OVER;
|
||||||
|
@ -534,6 +535,14 @@ static int32_t mndProcessCreateDbReq(SRpcMsg *pReq) {
|
||||||
terrno = TSDB_CODE_MND_DB_ALREADY_EXIST;
|
terrno = TSDB_CODE_MND_DB_ALREADY_EXIST;
|
||||||
goto _OVER;
|
goto _OVER;
|
||||||
}
|
}
|
||||||
|
} else if (terrno == TSDB_CODE_SDB_OBJ_CREATING) {
|
||||||
|
if (mndSetRpcInfoForDbTrans(pMnode, pReq, MND_OPER_CREATE_DB, createReq.db) == 0) {
|
||||||
|
mDebug("db:%s, is creating and response after trans finished", createReq.db);
|
||||||
|
code = TSDB_CODE_ACTION_IN_PROGRESS;
|
||||||
|
goto _OVER;
|
||||||
|
} else {
|
||||||
|
goto _OVER;
|
||||||
|
}
|
||||||
} else if (terrno != TSDB_CODE_MND_DB_NOT_EXIST) {
|
} else if (terrno != TSDB_CODE_MND_DB_NOT_EXIST) {
|
||||||
goto _OVER;
|
goto _OVER;
|
||||||
}
|
}
|
||||||
|
|
|
@ -383,6 +383,11 @@ int32_t mndScheduleStream(SMnode* pMnode, SStreamObj* pStream) {
|
||||||
// exec
|
// exec
|
||||||
pInnerTask->execType = TASK_EXEC__PIPE;
|
pInnerTask->execType = TASK_EXEC__PIPE;
|
||||||
|
|
||||||
|
SDbObj* pSourceDb = mndAcquireDb(pMnode, pStream->sourceDb);
|
||||||
|
ASSERT(pDbObj != NULL);
|
||||||
|
sdbRelease(pSdb, pSourceDb);
|
||||||
|
pInnerTask->numOfVgroups = pSourceDb->cfg.numOfVgroups;
|
||||||
|
|
||||||
if (tsSchedStreamToSnode) {
|
if (tsSchedStreamToSnode) {
|
||||||
SSnodeObj* pSnode = mndSchedFetchOneSnode(pMnode);
|
SSnodeObj* pSnode = mndSchedFetchOneSnode(pMnode);
|
||||||
if (pSnode == NULL) {
|
if (pSnode == NULL) {
|
||||||
|
|
|
@ -144,9 +144,11 @@ int32_t mndSnapshotStartWrite(struct SSyncFSM *pFsm, void *pParam, void **ppWrit
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t mndSnapshotStopWrite(struct SSyncFSM *pFsm, void *pWriter, bool isApply, SSnapshot *pSnapshot) {
|
int32_t mndSnapshotStopWrite(struct SSyncFSM *pFsm, void *pWriter, bool isApply, SSnapshot *pSnapshot) {
|
||||||
mInfo("stop to apply snapshot to sdb, apply:%d", isApply);
|
mInfo("stop to apply snapshot to sdb, apply:%d, index:%" PRId64 " term:%" PRIu64 " config:%" PRId64, isApply,
|
||||||
|
pSnapshot->lastApplyIndex, pSnapshot->lastApplyTerm, pSnapshot->lastApplyIndex);
|
||||||
SMnode *pMnode = pFsm->data;
|
SMnode *pMnode = pFsm->data;
|
||||||
return sdbStopWrite(pMnode->pSdb, pWriter, isApply);
|
return sdbStopWrite(pMnode->pSdb, pWriter, isApply, pSnapshot->lastApplyIndex, pSnapshot->lastApplyTerm,
|
||||||
|
pSnapshot->lastConfigIndex);
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t mndSnapshotDoWrite(struct SSyncFSM *pFsm, void *pWriter, void *pBuf, int32_t len) {
|
int32_t mndSnapshotDoWrite(struct SSyncFSM *pFsm, void *pWriter, void *pBuf, int32_t len) {
|
||||||
|
@ -157,7 +159,7 @@ int32_t mndSnapshotDoWrite(struct SSyncFSM *pFsm, void *pWriter, void *pBuf, int
|
||||||
void mndLeaderTransfer(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) {
|
void mndLeaderTransfer(struct SSyncFSM *pFsm, const SRpcMsg *pMsg, SFsmCbMeta cbMeta) {
|
||||||
SMnode *pMnode = pFsm->data;
|
SMnode *pMnode = pFsm->data;
|
||||||
atomic_store_8(&(pMnode->syncMgmt.leaderTransferFinish), 1);
|
atomic_store_8(&(pMnode->syncMgmt.leaderTransferFinish), 1);
|
||||||
mDebug("vgId:1, mnd leader transfer finish");
|
mDebug("vgId:1, mnode leader transfer finish");
|
||||||
}
|
}
|
||||||
|
|
||||||
SSyncFSM *mndSyncMakeFsm(SMnode *pMnode) {
|
SSyncFSM *mndSyncMakeFsm(SMnode *pMnode) {
|
||||||
|
|
|
@ -122,6 +122,10 @@ static SSdbRaw *mndTransActionEncode(STrans *pTrans) {
|
||||||
SDB_SET_INT8(pRaw, dataPos, pTrans->policy, _OVER)
|
SDB_SET_INT8(pRaw, dataPos, pTrans->policy, _OVER)
|
||||||
SDB_SET_INT8(pRaw, dataPos, pTrans->conflict, _OVER)
|
SDB_SET_INT8(pRaw, dataPos, pTrans->conflict, _OVER)
|
||||||
SDB_SET_INT8(pRaw, dataPos, pTrans->exec, _OVER)
|
SDB_SET_INT8(pRaw, dataPos, pTrans->exec, _OVER)
|
||||||
|
SDB_SET_INT8(pRaw, dataPos, pTrans->oper, _OVER)
|
||||||
|
SDB_SET_INT8(pRaw, dataPos, 0, _OVER)
|
||||||
|
SDB_SET_INT8(pRaw, dataPos, 0, _OVER)
|
||||||
|
SDB_SET_INT8(pRaw, dataPos, 0, _OVER)
|
||||||
SDB_SET_INT64(pRaw, dataPos, pTrans->createdTime, _OVER)
|
SDB_SET_INT64(pRaw, dataPos, pTrans->createdTime, _OVER)
|
||||||
SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER)
|
SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER)
|
||||||
SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER)
|
SDB_SET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER)
|
||||||
|
@ -269,15 +273,22 @@ static SSdbRow *mndTransActionDecode(SSdbRaw *pRaw) {
|
||||||
int8_t policy = 0;
|
int8_t policy = 0;
|
||||||
int8_t conflict = 0;
|
int8_t conflict = 0;
|
||||||
int8_t exec = 0;
|
int8_t exec = 0;
|
||||||
|
int8_t oper = 0;
|
||||||
|
int8_t reserved = 0;
|
||||||
int8_t actionType = 0;
|
int8_t actionType = 0;
|
||||||
SDB_GET_INT8(pRaw, dataPos, &stage, _OVER)
|
SDB_GET_INT8(pRaw, dataPos, &stage, _OVER)
|
||||||
SDB_GET_INT8(pRaw, dataPos, &policy, _OVER)
|
SDB_GET_INT8(pRaw, dataPos, &policy, _OVER)
|
||||||
SDB_GET_INT8(pRaw, dataPos, &conflict, _OVER)
|
SDB_GET_INT8(pRaw, dataPos, &conflict, _OVER)
|
||||||
SDB_GET_INT8(pRaw, dataPos, &exec, _OVER)
|
SDB_GET_INT8(pRaw, dataPos, &exec, _OVER)
|
||||||
|
SDB_GET_INT8(pRaw, dataPos, &oper, _OVER)
|
||||||
|
SDB_GET_INT8(pRaw, dataPos, &reserved, _OVER)
|
||||||
|
SDB_GET_INT8(pRaw, dataPos, &reserved, _OVER)
|
||||||
|
SDB_GET_INT8(pRaw, dataPos, &reserved, _OVER)
|
||||||
pTrans->stage = stage;
|
pTrans->stage = stage;
|
||||||
pTrans->policy = policy;
|
pTrans->policy = policy;
|
||||||
pTrans->conflict = conflict;
|
pTrans->conflict = conflict;
|
||||||
pTrans->exec = exec;
|
pTrans->exec = exec;
|
||||||
|
pTrans->oper = oper;
|
||||||
SDB_GET_INT64(pRaw, dataPos, &pTrans->createdTime, _OVER)
|
SDB_GET_INT64(pRaw, dataPos, &pTrans->createdTime, _OVER)
|
||||||
SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER)
|
SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname1, TSDB_DB_FNAME_LEN, _OVER)
|
||||||
SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER)
|
SDB_GET_BINARY(pRaw, dataPos, pTrans->dbname2, TSDB_DB_FNAME_LEN, _OVER)
|
||||||
|
@ -495,6 +506,10 @@ static void mndTransDropData(STrans *pTrans) {
|
||||||
mndTransDropActions(pTrans->commitActions);
|
mndTransDropActions(pTrans->commitActions);
|
||||||
pTrans->commitActions = NULL;
|
pTrans->commitActions = NULL;
|
||||||
}
|
}
|
||||||
|
if (pTrans->pRpcArray != NULL) {
|
||||||
|
taosArrayDestroy(pTrans->pRpcArray);
|
||||||
|
pTrans->pRpcArray = NULL;
|
||||||
|
}
|
||||||
if (pTrans->rpcRsp != NULL) {
|
if (pTrans->rpcRsp != NULL) {
|
||||||
taosMemoryFree(pTrans->rpcRsp);
|
taosMemoryFree(pTrans->rpcRsp);
|
||||||
pTrans->rpcRsp = NULL;
|
pTrans->rpcRsp = NULL;
|
||||||
|
@ -585,14 +600,18 @@ STrans *mndTransCreate(SMnode *pMnode, ETrnPolicy policy, ETrnConflct conflict,
|
||||||
pTrans->redoActions = taosArrayInit(TRANS_ARRAY_SIZE, sizeof(STransAction));
|
pTrans->redoActions = taosArrayInit(TRANS_ARRAY_SIZE, sizeof(STransAction));
|
||||||
pTrans->undoActions = taosArrayInit(TRANS_ARRAY_SIZE, sizeof(STransAction));
|
pTrans->undoActions = taosArrayInit(TRANS_ARRAY_SIZE, sizeof(STransAction));
|
||||||
pTrans->commitActions = taosArrayInit(TRANS_ARRAY_SIZE, sizeof(STransAction));
|
pTrans->commitActions = taosArrayInit(TRANS_ARRAY_SIZE, sizeof(STransAction));
|
||||||
|
pTrans->pRpcArray = taosArrayInit(1, sizeof(SRpcHandleInfo));
|
||||||
|
|
||||||
if (pTrans->redoActions == NULL || pTrans->undoActions == NULL || pTrans->commitActions == NULL) {
|
if (pTrans->redoActions == NULL || pTrans->undoActions == NULL || pTrans->commitActions == NULL ||
|
||||||
|
pTrans->pRpcArray == NULL) {
|
||||||
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
terrno = TSDB_CODE_OUT_OF_MEMORY;
|
||||||
mError("failed to create transaction since %s", terrstr());
|
mError("failed to create transaction since %s", terrstr());
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pReq != NULL) pTrans->rpcInfo = pReq->info;
|
if (pReq != NULL) {
|
||||||
|
taosArrayPush(pTrans->pRpcArray, &pReq->info);
|
||||||
|
}
|
||||||
mTrace("trans:%d, local object is created, data:%p", pTrans->id, pTrans);
|
mTrace("trans:%d, local object is created, data:%p", pTrans->id, pTrans);
|
||||||
return pTrans;
|
return pTrans;
|
||||||
}
|
}
|
||||||
|
@ -677,6 +696,31 @@ void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *
|
||||||
pTrans->paramLen = paramLen;
|
pTrans->paramLen = paramLen;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
int32_t mndSetRpcInfoForDbTrans(SMnode *pMnode, SRpcMsg *pMsg, EOperType oper, const char *dbname) {
|
||||||
|
STrans *pTrans = NULL;
|
||||||
|
void *pIter = NULL;
|
||||||
|
int32_t code = -1;
|
||||||
|
|
||||||
|
while (1) {
|
||||||
|
pIter = sdbFetch(pMnode->pSdb, SDB_TRANS, pIter, (void **)&pTrans);
|
||||||
|
if (pIter == NULL) break;
|
||||||
|
|
||||||
|
if (pTrans->oper == oper) {
|
||||||
|
if (strcasecmp(dbname, pTrans->dbname1) == 0) {
|
||||||
|
mDebug("trans:%d, db:%s oper:%d matched with input", pTrans->id, dbname, oper);
|
||||||
|
if (taosArrayPush(pTrans->pRpcArray, &pMsg->info) != NULL) {
|
||||||
|
code = 0;
|
||||||
|
}
|
||||||
|
sdbRelease(pMnode->pSdb, pTrans);
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
sdbRelease(pMnode->pSdb, pTrans);
|
||||||
|
}
|
||||||
|
return code;
|
||||||
|
}
|
||||||
|
|
||||||
void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2) {
|
void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2) {
|
||||||
if (dbname1 != NULL) {
|
if (dbname1 != NULL) {
|
||||||
memcpy(pTrans->dbname1, dbname1, TSDB_DB_FNAME_LEN);
|
memcpy(pTrans->dbname1, dbname1, TSDB_DB_FNAME_LEN);
|
||||||
|
@ -688,6 +732,8 @@ void mndTransSetDbName(STrans *pTrans, const char *dbname1, const char *dbname2)
|
||||||
|
|
||||||
void mndTransSetSerial(STrans *pTrans) { pTrans->exec = TRN_EXEC_SERIAL; }
|
void mndTransSetSerial(STrans *pTrans) { pTrans->exec = TRN_EXEC_SERIAL; }
|
||||||
|
|
||||||
|
void mndTransSetOper(STrans *pTrans, EOperType oper) { pTrans->oper = oper; }
|
||||||
|
|
||||||
static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) {
|
static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) {
|
||||||
SSdbRaw *pRaw = mndTransActionEncode(pTrans);
|
SSdbRaw *pRaw = mndTransActionEncode(pTrans);
|
||||||
if (pRaw == NULL) {
|
if (pRaw == NULL) {
|
||||||
|
@ -711,7 +757,7 @@ static int32_t mndTransSync(SMnode *pMnode, STrans *pTrans) {
|
||||||
|
|
||||||
static bool mndCheckDbConflict(const char *db, STrans *pTrans) {
|
static bool mndCheckDbConflict(const char *db, STrans *pTrans) {
|
||||||
if (db[0] == 0) return false;
|
if (db[0] == 0) return false;
|
||||||
if (strcmp(db, pTrans->dbname1) == 0 || strcmp(db, pTrans->dbname2) == 0) return true;
|
if (strcasecmp(db, pTrans->dbname1) == 0 || strcasecmp(db, pTrans->dbname2) == 0) return true;
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -784,9 +830,10 @@ int32_t mndTransPrepare(SMnode *pMnode, STrans *pTrans) {
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
|
||||||
pNew->rpcInfo = pTrans->rpcInfo;
|
pNew->pRpcArray = pTrans->pRpcArray;
|
||||||
pNew->rpcRsp = pTrans->rpcRsp;
|
pNew->rpcRsp = pTrans->rpcRsp;
|
||||||
pNew->rpcRspLen = pTrans->rpcRspLen;
|
pNew->rpcRspLen = pTrans->rpcRspLen;
|
||||||
|
pTrans->pRpcArray = NULL;
|
||||||
pTrans->rpcRsp = NULL;
|
pTrans->rpcRsp = NULL;
|
||||||
pTrans->rpcRspLen = 0;
|
pTrans->rpcRspLen = 0;
|
||||||
|
|
||||||
|
@ -835,13 +882,20 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if (sendRsp && pTrans->rpcInfo.handle != NULL) {
|
if (!sendRsp) return;
|
||||||
|
|
||||||
|
int32_t size = taosArrayGetSize(pTrans->pRpcArray);
|
||||||
|
if (size <= 0) return;
|
||||||
|
|
||||||
|
for (int32_t i = 0; i < size; ++i) {
|
||||||
|
SRpcHandleInfo *pInfo = taosArrayGet(pTrans->pRpcArray, i);
|
||||||
|
if (pInfo->handle != NULL) {
|
||||||
mDebug("trans:%d, send rsp, code:0x%x stage:%s app:%p", pTrans->id, code, mndTransStr(pTrans->stage),
|
mDebug("trans:%d, send rsp, code:0x%x stage:%s app:%p", pTrans->id, code, mndTransStr(pTrans->stage),
|
||||||
pTrans->rpcInfo.ahandle);
|
pInfo->ahandle);
|
||||||
if (code == TSDB_CODE_RPC_NETWORK_UNAVAIL) {
|
if (code == TSDB_CODE_RPC_NETWORK_UNAVAIL) {
|
||||||
code = TSDB_CODE_MND_TRANS_NETWORK_UNAVAILL;
|
code = TSDB_CODE_MND_TRANS_NETWORK_UNAVAILL;
|
||||||
}
|
}
|
||||||
SRpcMsg rspMsg = {.code = code, .info = pTrans->rpcInfo};
|
SRpcMsg rspMsg = {.code = code, .info = *pInfo};
|
||||||
|
|
||||||
if (pTrans->rpcRspLen != 0) {
|
if (pTrans->rpcRspLen != 0) {
|
||||||
void *rpcCont = rpcMallocCont(pTrans->rpcRspLen);
|
void *rpcCont = rpcMallocCont(pTrans->rpcRspLen);
|
||||||
|
@ -850,15 +904,13 @@ static void mndTransSendRpcRsp(SMnode *pMnode, STrans *pTrans) {
|
||||||
rspMsg.pCont = rpcCont;
|
rspMsg.pCont = rpcCont;
|
||||||
rspMsg.contLen = pTrans->rpcRspLen;
|
rspMsg.contLen = pTrans->rpcRspLen;
|
||||||
}
|
}
|
||||||
taosMemoryFree(pTrans->rpcRsp);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
tmsgSendRsp(&rspMsg);
|
tmsgSendRsp(&rspMsg);
|
||||||
pTrans->rpcInfo.handle = NULL;
|
|
||||||
pTrans->rpcRsp = NULL;
|
|
||||||
pTrans->rpcRspLen = 0;
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
taosArrayClear(pTrans->pRpcArray);
|
||||||
|
}
|
||||||
|
|
||||||
int32_t mndTransProcessRsp(SRpcMsg *pRsp) {
|
int32_t mndTransProcessRsp(SRpcMsg *pRsp) {
|
||||||
SMnode *pMnode = pRsp->info.node;
|
SMnode *pMnode = pRsp->info.node;
|
||||||
|
|
|
@ -925,7 +925,7 @@ TEST_F(MndTestSdb, 01_Read_Str) {
|
||||||
}
|
}
|
||||||
|
|
||||||
sdbStopRead(pSdb, pReader);
|
sdbStopRead(pSdb, pReader);
|
||||||
sdbStopWrite(pSdb, pWritter, true);
|
sdbStopWrite(pSdb, pWritter, true, -1, -1, -1);
|
||||||
}
|
}
|
||||||
|
|
||||||
ASSERT_EQ(sdbGetSize(pSdb, SDB_CONSUMER), 1);
|
ASSERT_EQ(sdbGetSize(pSdb, SDB_CONSUMER), 1);
|
||||||
|
|
|
@ -394,7 +394,7 @@ int32_t sdbStopRead(SSdb *pSdb, SSdbIter *pIter);
|
||||||
int32_t sdbDoRead(SSdb *pSdb, SSdbIter *pIter, void **ppBuf, int32_t *len);
|
int32_t sdbDoRead(SSdb *pSdb, SSdbIter *pIter, void **ppBuf, int32_t *len);
|
||||||
|
|
||||||
int32_t sdbStartWrite(SSdb *pSdb, SSdbIter **ppIter);
|
int32_t sdbStartWrite(SSdb *pSdb, SSdbIter **ppIter);
|
||||||
int32_t sdbStopWrite(SSdb *pSdb, SSdbIter *pIter, bool isApply);
|
int32_t sdbStopWrite(SSdb *pSdb, SSdbIter *pIter, bool isApply, int64_t index, int64_t term, int64_t config);
|
||||||
int32_t sdbDoWrite(SSdb *pSdb, SSdbIter *pIter, void *pBuf, int32_t len);
|
int32_t sdbDoWrite(SSdb *pSdb, SSdbIter *pIter, void *pBuf, int32_t len);
|
||||||
|
|
||||||
const char *sdbTableName(ESdbType type);
|
const char *sdbTableName(ESdbType type);
|
||||||
|
|
|
@ -613,7 +613,7 @@ int32_t sdbStartWrite(SSdb *pSdb, SSdbIter **ppIter) {
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t sdbStopWrite(SSdb *pSdb, SSdbIter *pIter, bool isApply) {
|
int32_t sdbStopWrite(SSdb *pSdb, SSdbIter *pIter, bool isApply, int64_t index, int64_t term, int64_t config) {
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
|
|
||||||
if (!isApply) {
|
if (!isApply) {
|
||||||
|
@ -641,6 +641,19 @@ int32_t sdbStopWrite(SSdb *pSdb, SSdbIter *pIter, bool isApply) {
|
||||||
return -1;
|
return -1;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
if (config > 0) {
|
||||||
|
ASSERT(pSdb->commitConfig == config);
|
||||||
|
pSdb->commitConfig = config;
|
||||||
|
}
|
||||||
|
if (term > 0) {
|
||||||
|
ASSERT(pSdb->commitTerm == term);
|
||||||
|
pSdb->commitTerm = term;
|
||||||
|
}
|
||||||
|
if (index > 0) {
|
||||||
|
ASSERT(pSdb->commitIndex == index);
|
||||||
|
pSdb->commitIndex = index;
|
||||||
|
}
|
||||||
|
|
||||||
mDebug("sdbiter:%p, successfully applyed to sdb", pIter);
|
mDebug("sdbiter:%p, successfully applyed to sdb", pIter);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
|
@ -153,23 +153,22 @@ static int32_t sdbInsertRow(SSdb *pSdb, SHashObj *hash, SSdbRaw *pRaw, SSdbRow *
|
||||||
return terrno;
|
return terrno;
|
||||||
}
|
}
|
||||||
|
|
||||||
taosThreadRwlockUnlock(pLock);
|
|
||||||
|
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
SdbInsertFp insertFp = pSdb->insertFps[pRow->type];
|
SdbInsertFp insertFp = pSdb->insertFps[pRow->type];
|
||||||
if (insertFp != NULL) {
|
if (insertFp != NULL) {
|
||||||
code = (*insertFp)(pSdb, pRow->pObj);
|
code = (*insertFp)(pSdb, pRow->pObj);
|
||||||
if (code != 0) {
|
if (code != 0) {
|
||||||
code = terrno;
|
code = terrno;
|
||||||
taosThreadRwlockWrlock(pLock);
|
|
||||||
taosHashRemove(hash, pRow->pObj, keySize);
|
taosHashRemove(hash, pRow->pObj, keySize);
|
||||||
taosThreadRwlockUnlock(pLock);
|
|
||||||
sdbFreeRow(pSdb, pRow, false);
|
sdbFreeRow(pSdb, pRow, false);
|
||||||
terrno = code;
|
terrno = code;
|
||||||
|
taosThreadRwlockUnlock(pLock);
|
||||||
return terrno;
|
return terrno;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
taosThreadRwlockUnlock(pLock);
|
||||||
|
|
||||||
if (pSdb->keyTypes[pRow->type] == SDB_KEY_INT32) {
|
if (pSdb->keyTypes[pRow->type] == SDB_KEY_INT32) {
|
||||||
pSdb->maxId[pRow->type] = TMAX(pSdb->maxId[pRow->type], *((int32_t *)pRow->pObj));
|
pSdb->maxId[pRow->type] = TMAX(pSdb->maxId[pRow->type], *((int32_t *)pRow->pObj));
|
||||||
}
|
}
|
||||||
|
@ -194,7 +193,6 @@ static int32_t sdbUpdateRow(SSdb *pSdb, SHashObj *hash, SSdbRaw *pRaw, SSdbRow *
|
||||||
SSdbRow *pOldRow = *ppOldRow;
|
SSdbRow *pOldRow = *ppOldRow;
|
||||||
pOldRow->status = pRaw->status;
|
pOldRow->status = pRaw->status;
|
||||||
sdbPrintOper(pSdb, pOldRow, "update");
|
sdbPrintOper(pSdb, pOldRow, "update");
|
||||||
taosThreadRwlockUnlock(pLock);
|
|
||||||
|
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
SdbUpdateFp updateFp = pSdb->updateFps[pNewRow->type];
|
SdbUpdateFp updateFp = pSdb->updateFps[pNewRow->type];
|
||||||
|
@ -202,6 +200,7 @@ static int32_t sdbUpdateRow(SSdb *pSdb, SHashObj *hash, SSdbRaw *pRaw, SSdbRow *
|
||||||
code = (*updateFp)(pSdb, pOldRow->pObj, pNewRow->pObj);
|
code = (*updateFp)(pSdb, pOldRow->pObj, pNewRow->pObj);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
taosThreadRwlockUnlock(pLock);
|
||||||
sdbFreeRow(pSdb, pNewRow, false);
|
sdbFreeRow(pSdb, pNewRow, false);
|
||||||
|
|
||||||
pSdb->tableVer[pOldRow->type]++;
|
pSdb->tableVer[pOldRow->type]++;
|
||||||
|
|
|
@ -88,7 +88,7 @@ typedef struct {
|
||||||
STqExecTb execTb;
|
STqExecTb execTb;
|
||||||
STqExecDb execDb;
|
STqExecDb execDb;
|
||||||
};
|
};
|
||||||
|
int32_t numOfCols; // number of out pout column, temporarily used
|
||||||
} STqExecHandle;
|
} STqExecHandle;
|
||||||
|
|
||||||
typedef struct {
|
typedef struct {
|
||||||
|
|
|
@ -174,6 +174,10 @@ void tsdbCalcColDataSMA(SColData *pColData, SColumnDataAgg *pColAgg);
|
||||||
int32_t tsdbMemTableCreate(STsdb *pTsdb, SMemTable **ppMemTable);
|
int32_t tsdbMemTableCreate(STsdb *pTsdb, SMemTable **ppMemTable);
|
||||||
void tsdbMemTableDestroy(SMemTable *pMemTable);
|
void tsdbMemTableDestroy(SMemTable *pMemTable);
|
||||||
void tsdbGetTbDataFromMemTable(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, STbData **ppTbData);
|
void tsdbGetTbDataFromMemTable(SMemTable *pMemTable, tb_uid_t suid, tb_uid_t uid, STbData **ppTbData);
|
||||||
|
void tsdbRefMemTable(SMemTable *pMemTable);
|
||||||
|
void tsdbUnrefMemTable(SMemTable *pMemTable);
|
||||||
|
int32_t tsdbTakeMemSnapshot(STsdb *pTsdb, SMemTable **ppMem, SMemTable **ppIMem);
|
||||||
|
void tsdbUntakeMemSnapshot(STsdb *pTsdb, SMemTable *pMem, SMemTable *pIMem);
|
||||||
// STbDataIter
|
// STbDataIter
|
||||||
int32_t tsdbTbDataIterCreate(STbData *pTbData, TSDBKEY *pFrom, int8_t backward, STbDataIter **ppIter);
|
int32_t tsdbTbDataIterCreate(STbData *pTbData, TSDBKEY *pFrom, int8_t backward, STbDataIter **ppIter);
|
||||||
void *tsdbTbDataIterDestroy(STbDataIter *pIter);
|
void *tsdbTbDataIterDestroy(STbDataIter *pIter);
|
||||||
|
@ -186,6 +190,7 @@ int32_t tsdbGetNRowsInTbData(STbData *pTbData);
|
||||||
typedef enum { TSDB_HEAD_FILE = 0, TSDB_DATA_FILE, TSDB_LAST_FILE, TSDB_SMA_FILE } EDataFileT;
|
typedef enum { TSDB_HEAD_FILE = 0, TSDB_DATA_FILE, TSDB_LAST_FILE, TSDB_SMA_FILE } EDataFileT;
|
||||||
void tsdbDataFileName(STsdb *pTsdb, SDFileSet *pDFileSet, EDataFileT ftype, char fname[]);
|
void tsdbDataFileName(STsdb *pTsdb, SDFileSet *pDFileSet, EDataFileT ftype, char fname[]);
|
||||||
bool tsdbFileIsSame(SDFileSet *pDFileSet1, SDFileSet *pDFileSet2, EDataFileT ftype);
|
bool tsdbFileIsSame(SDFileSet *pDFileSet1, SDFileSet *pDFileSet2, EDataFileT ftype);
|
||||||
|
bool tsdbDelFileIsSame(SDelFile *pDelFile1, SDelFile *pDelFile2);
|
||||||
int32_t tsdbUpdateDFileHdr(TdFilePtr pFD, SDFileSet *pSet, EDataFileT ftype);
|
int32_t tsdbUpdateDFileHdr(TdFilePtr pFD, SDFileSet *pSet, EDataFileT ftype);
|
||||||
int32_t tsdbDFileRollback(STsdb *pTsdb, SDFileSet *pSet, EDataFileT ftype);
|
int32_t tsdbDFileRollback(STsdb *pTsdb, SDFileSet *pSet, EDataFileT ftype);
|
||||||
int32_t tPutDataFileHdr(uint8_t *p, SDFileSet *pSet, EDataFileT ftype);
|
int32_t tPutDataFileHdr(uint8_t *p, SDFileSet *pSet, EDataFileT ftype);
|
||||||
|
@ -274,23 +279,14 @@ typedef struct {
|
||||||
struct STsdb {
|
struct STsdb {
|
||||||
char *path;
|
char *path;
|
||||||
SVnode *pVnode;
|
SVnode *pVnode;
|
||||||
TdThreadMutex mutex;
|
|
||||||
bool repoLocked;
|
|
||||||
STsdbKeepCfg keepCfg;
|
STsdbKeepCfg keepCfg;
|
||||||
|
TdThreadRwlock rwLock;
|
||||||
SMemTable *mem;
|
SMemTable *mem;
|
||||||
SMemTable *imem;
|
SMemTable *imem;
|
||||||
SRtn rtn;
|
STsdbFS *pFS;
|
||||||
STsdbFS *fs;
|
|
||||||
SLRUCache *lruCache;
|
SLRUCache *lruCache;
|
||||||
};
|
};
|
||||||
|
|
||||||
struct STable {
|
|
||||||
uint64_t suid;
|
|
||||||
uint64_t uid;
|
|
||||||
STSchema *pSchema; // latest schema
|
|
||||||
STSchema *pCacheSchema; // cached cache
|
|
||||||
};
|
|
||||||
|
|
||||||
struct TSDBKEY {
|
struct TSDBKEY {
|
||||||
int64_t version;
|
int64_t version;
|
||||||
TSKEY ts;
|
TSKEY ts;
|
||||||
|
@ -331,7 +327,8 @@ struct STbData {
|
||||||
struct SMemTable {
|
struct SMemTable {
|
||||||
SRWLatch latch;
|
SRWLatch latch;
|
||||||
STsdb *pTsdb;
|
STsdb *pTsdb;
|
||||||
int32_t nRef;
|
SVBufPool *pPool;
|
||||||
|
volatile int32_t nRef;
|
||||||
TSKEY minKey;
|
TSKEY minKey;
|
||||||
TSKEY maxKey;
|
TSKEY maxKey;
|
||||||
int64_t minVersion;
|
int64_t minVersion;
|
||||||
|
@ -341,9 +338,6 @@ struct SMemTable {
|
||||||
SArray *aTbData; // SArray<STbData*>
|
SArray *aTbData; // SArray<STbData*>
|
||||||
};
|
};
|
||||||
|
|
||||||
int tsdbLockRepo(STsdb *pTsdb);
|
|
||||||
int tsdbUnlockRepo(STsdb *pTsdb);
|
|
||||||
|
|
||||||
struct TSDBROW {
|
struct TSDBROW {
|
||||||
int8_t type; // 0 for row from tsRow, 1 for row from block data
|
int8_t type; // 0 for row from tsRow, 1 for row from block data
|
||||||
union {
|
union {
|
||||||
|
|
|
@ -63,7 +63,8 @@ struct SVBufPoolNode {
|
||||||
|
|
||||||
struct SVBufPool {
|
struct SVBufPool {
|
||||||
SVBufPool* next;
|
SVBufPool* next;
|
||||||
int64_t nRef;
|
SVnode* pVnode;
|
||||||
|
volatile int32_t nRef;
|
||||||
int64_t size;
|
int64_t size;
|
||||||
uint8_t* ptr;
|
uint8_t* ptr;
|
||||||
SVBufPoolNode* pTail;
|
SVBufPoolNode* pTail;
|
||||||
|
|
|
@ -77,6 +77,8 @@ typedef struct SSnapDataHdr SSnapDataHdr;
|
||||||
// vnd.h
|
// vnd.h
|
||||||
void* vnodeBufPoolMalloc(SVBufPool* pPool, int size);
|
void* vnodeBufPoolMalloc(SVBufPool* pPool, int size);
|
||||||
void vnodeBufPoolFree(SVBufPool* pPool, void* p);
|
void vnodeBufPoolFree(SVBufPool* pPool, void* p);
|
||||||
|
void vnodeBufPoolRef(SVBufPool* pPool);
|
||||||
|
void vnodeBufPoolUnRef(SVBufPool* pPool);
|
||||||
|
|
||||||
// meta
|
// meta
|
||||||
typedef struct SMCtbCursor SMCtbCursor;
|
typedef struct SMCtbCursor SMCtbCursor;
|
||||||
|
@ -252,10 +254,10 @@ struct SVnode {
|
||||||
SVState state;
|
SVState state;
|
||||||
STfs* pTfs;
|
STfs* pTfs;
|
||||||
SMsgCb msgCb;
|
SMsgCb msgCb;
|
||||||
|
TdThreadMutex mutex;
|
||||||
|
TdThreadCond poolNotEmpty;
|
||||||
SVBufPool* pPool;
|
SVBufPool* pPool;
|
||||||
SVBufPool* inUse;
|
SVBufPool* inUse;
|
||||||
SVBufPool* onCommit;
|
|
||||||
SVBufPool* onRecycle;
|
|
||||||
SMeta* pMeta;
|
SMeta* pMeta;
|
||||||
SSma* pSma;
|
SSma* pSma;
|
||||||
STsdb* pTsdb;
|
STsdb* pTsdb;
|
||||||
|
@ -273,8 +275,8 @@ struct SVnode {
|
||||||
|
|
||||||
#define VND_TSDB(vnd) ((vnd)->pTsdb)
|
#define VND_TSDB(vnd) ((vnd)->pTsdb)
|
||||||
#define VND_RSMA0(vnd) ((vnd)->pTsdb)
|
#define VND_RSMA0(vnd) ((vnd)->pTsdb)
|
||||||
#define VND_RSMA1(vnd) ((vnd)->pSma->pRSmaTsdb1)
|
#define VND_RSMA1(vnd) ((vnd)->pSma->pRSmaTsdb[TSDB_RETENTION_L0])
|
||||||
#define VND_RSMA2(vnd) ((vnd)->pSma->pRSmaTsdb2)
|
#define VND_RSMA2(vnd) ((vnd)->pSma->pRSmaTsdb[TSDB_RETENTION_L1])
|
||||||
#define VND_RETENTIONS(vnd) (&(vnd)->config.tsdbCfg.retentions)
|
#define VND_RETENTIONS(vnd) (&(vnd)->config.tsdbCfg.retentions)
|
||||||
#define VND_IS_RSMA(v) ((v)->config.isRsma == 1)
|
#define VND_IS_RSMA(v) ((v)->config.isRsma == 1)
|
||||||
#define VND_IS_TSMA(v) ((v)->config.isTsma == 1)
|
#define VND_IS_TSMA(v) ((v)->config.isTsma == 1)
|
||||||
|
@ -289,8 +291,7 @@ struct SSma {
|
||||||
bool locked;
|
bool locked;
|
||||||
TdThreadMutex mutex;
|
TdThreadMutex mutex;
|
||||||
SVnode* pVnode;
|
SVnode* pVnode;
|
||||||
STsdb* pRSmaTsdb1;
|
STsdb* pRSmaTsdb[TSDB_RETENTION_L2];
|
||||||
STsdb* pRSmaTsdb2;
|
|
||||||
void* pTSmaEnv;
|
void* pTSmaEnv;
|
||||||
void* pRSmaEnv;
|
void* pRSmaEnv;
|
||||||
};
|
};
|
||||||
|
@ -305,8 +306,8 @@ struct SSma {
|
||||||
#define SMA_TSMA_ENV(s) ((s)->pTSmaEnv)
|
#define SMA_TSMA_ENV(s) ((s)->pTSmaEnv)
|
||||||
#define SMA_RSMA_ENV(s) ((s)->pRSmaEnv)
|
#define SMA_RSMA_ENV(s) ((s)->pRSmaEnv)
|
||||||
#define SMA_RSMA_TSDB0(s) ((s)->pVnode->pTsdb)
|
#define SMA_RSMA_TSDB0(s) ((s)->pVnode->pTsdb)
|
||||||
#define SMA_RSMA_TSDB1(s) ((s)->pRSmaTsdb1)
|
#define SMA_RSMA_TSDB1(s) ((s)->pRSmaTsdb[TSDB_RETENTION_L0])
|
||||||
#define SMA_RSMA_TSDB2(s) ((s)->pRSmaTsdb2)
|
#define SMA_RSMA_TSDB2(s) ((s)->pRSmaTsdb[TSDB_RETENTION_L1])
|
||||||
|
|
||||||
// sma
|
// sma
|
||||||
void smaHandleRes(void* pVnode, int64_t smaId, const SArray* data);
|
void smaHandleRes(void* pVnode, int64_t smaId, const SArray* data);
|
||||||
|
|
|
@ -371,6 +371,13 @@ int metaCreateTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq) {
|
||||||
}
|
}
|
||||||
metaReaderClear(&mr);
|
metaReaderClear(&mr);
|
||||||
|
|
||||||
|
if (pReq->type == TSDB_CHILD_TABLE) {
|
||||||
|
tb_uid_t suid = metaGetTableEntryUidByName(pMeta, pReq->ctb.name);
|
||||||
|
if (suid == 0) {
|
||||||
|
terrno = TSDB_CODE_PAR_TABLE_NOT_EXIST;
|
||||||
|
return -1;
|
||||||
|
}
|
||||||
|
}
|
||||||
// build SMetaEntry
|
// build SMetaEntry
|
||||||
me.version = version;
|
me.version = version;
|
||||||
me.type = pReq->type;
|
me.type = pReq->type;
|
||||||
|
|
|
@ -123,11 +123,9 @@ int32_t smaOpen(SVnode *pVnode) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// restore the rsma
|
// restore the rsma
|
||||||
#if 1
|
|
||||||
if (rsmaRestore(pSma) < 0) {
|
if (rsmaRestore(pSma) < 0) {
|
||||||
goto _err;
|
goto _err;
|
||||||
}
|
}
|
||||||
#endif
|
|
||||||
}
|
}
|
||||||
|
|
||||||
return 0;
|
return 0;
|
||||||
|
|
|
@ -605,7 +605,7 @@ static int32_t tdRSmaFetchAndSubmitResult(SRSmaInfoItem *pItem, STSchema *pTSche
|
||||||
snprintf(flag, 10, "level %" PRIi8, pItem->level);
|
snprintf(flag, 10, "level %" PRIi8, pItem->level);
|
||||||
blockDebugShowDataBlocks(pResult, flag);
|
blockDebugShowDataBlocks(pResult, flag);
|
||||||
#endif
|
#endif
|
||||||
STsdb *sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb1 : pSma->pRSmaTsdb2);
|
STsdb *sinkTsdb = (pItem->level == TSDB_RETENTION_L1 ? pSma->pRSmaTsdb[0] : pSma->pRSmaTsdb[1]);
|
||||||
SSubmitReq *pReq = NULL;
|
SSubmitReq *pReq = NULL;
|
||||||
// TODO: the schema update should be handled
|
// TODO: the schema update should be handled
|
||||||
if (buildSubmitReqFromDataBlock(&pReq, pResult, pTSchema, SMA_VID(pSma), suid) < 0) {
|
if (buildSubmitReqFromDataBlock(&pReq, pResult, pTSchema, SMA_VID(pSma), suid) < 0) {
|
||||||
|
@ -949,13 +949,8 @@ _err:
|
||||||
* @return int32_t
|
* @return int32_t
|
||||||
*/
|
*/
|
||||||
static int32_t tdRSmaRestoreTSDataReload(SSma *pSma, int64_t committed) {
|
static int32_t tdRSmaRestoreTSDataReload(SSma *pSma, int64_t committed) {
|
||||||
// TODO
|
// NOTHING TODO: the data would be restored from the unified WAL replay procedure
|
||||||
smaDebug("vgId:%d, rsma restore from %" PRIi64 ", ts data reload success", SMA_VID(pSma), committed);
|
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
_err:
|
|
||||||
smaError("vgId:%d, rsma restore from %" PRIi64 ", ts data reload failed since %s", SMA_VID(pSma), committed,
|
|
||||||
terrstr());
|
|
||||||
return TSDB_CODE_FAILED;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t tdProcessRSmaRestoreImpl(SSma *pSma) {
|
int32_t tdProcessRSmaRestoreImpl(SSma *pSma) {
|
||||||
|
|
|
@ -506,7 +506,7 @@ int32_t tqProcessVgChangeReq(STQ* pTq, char* msg, int32_t msgLen) {
|
||||||
.initTqReader = true,
|
.initTqReader = true,
|
||||||
.version = ver,
|
.version = ver,
|
||||||
};
|
};
|
||||||
pHandle->execHandle.execCol.task[i] = qCreateQueueExecTaskInfo(pHandle->execHandle.execCol.qmsg, &handle);
|
pHandle->execHandle.execCol.task[i] = qCreateQueueExecTaskInfo(pHandle->execHandle.execCol.qmsg, &handle, &pHandle->execHandle.numOfCols);
|
||||||
ASSERT(pHandle->execHandle.execCol.task[i]);
|
ASSERT(pHandle->execHandle.execCol.task[i]);
|
||||||
void* scanner = NULL;
|
void* scanner = NULL;
|
||||||
qExtractStreamScanner(pHandle->execHandle.execCol.task[i], &scanner);
|
qExtractStreamScanner(pHandle->execHandle.execCol.task[i], &scanner);
|
||||||
|
@ -591,7 +591,11 @@ int32_t tqProcessTaskDeployReq(STQ* pTq, char* msg, int32_t msgLen) {
|
||||||
};
|
};
|
||||||
pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, &handle);
|
pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, &handle);
|
||||||
} else {
|
} else {
|
||||||
pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, NULL);
|
SReadHandle mgHandle = {
|
||||||
|
.vnode = NULL,
|
||||||
|
.numOfVgroups = pTask->numOfVgroups,
|
||||||
|
};
|
||||||
|
pTask->exec.executor = qCreateStreamExecTaskInfo(pTask->exec.qmsg, &mgHandle);
|
||||||
}
|
}
|
||||||
ASSERT(pTask->exec.executor);
|
ASSERT(pTask->exec.executor);
|
||||||
}
|
}
|
||||||
|
|
|
@ -15,7 +15,7 @@
|
||||||
|
|
||||||
#include "tq.h"
|
#include "tq.h"
|
||||||
|
|
||||||
static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp) {
|
static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp, int32_t numOfCols) {
|
||||||
int32_t dataStrLen = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock);
|
int32_t dataStrLen = sizeof(SRetrieveTableRsp) + blockGetEncodeSize(pBlock);
|
||||||
void* buf = taosMemoryCalloc(1, dataStrLen);
|
void* buf = taosMemoryCalloc(1, dataStrLen);
|
||||||
if (buf == NULL) return -1;
|
if (buf == NULL) return -1;
|
||||||
|
@ -29,7 +29,7 @@ static int32_t tqAddBlockDataToRsp(const SSDataBlock* pBlock, SMqDataRsp* pRsp)
|
||||||
|
|
||||||
// TODO enable compress
|
// TODO enable compress
|
||||||
int32_t actualLen = 0;
|
int32_t actualLen = 0;
|
||||||
blockEncode(pBlock, pRetrieve->data, &actualLen, taosArrayGetSize(pBlock->pDataBlock), false);
|
blockEncode(pBlock, pRetrieve->data, &actualLen, numOfCols, false);
|
||||||
actualLen += sizeof(SRetrieveTableRsp);
|
actualLen += sizeof(SRetrieveTableRsp);
|
||||||
ASSERT(actualLen <= dataStrLen);
|
ASSERT(actualLen <= dataStrLen);
|
||||||
taosArrayPush(pRsp->blockDataLen, &actualLen);
|
taosArrayPush(pRsp->blockDataLen, &actualLen);
|
||||||
|
@ -97,7 +97,7 @@ int64_t tqScan(STQ* pTq, const STqHandle* pHandle, SMqDataRsp* pRsp, STqOffsetVa
|
||||||
pRsp->withTbName = 0;
|
pRsp->withTbName = 0;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
tqAddBlockDataToRsp(pDataBlock, pRsp);
|
tqAddBlockDataToRsp(pDataBlock, pRsp, pExec->numOfCols);
|
||||||
pRsp->blockNum++;
|
pRsp->blockNum++;
|
||||||
if (pOffset->type == TMQ_OFFSET__LOG) {
|
if (pOffset->type == TMQ_OFFSET__LOG) {
|
||||||
continue;
|
continue;
|
||||||
|
@ -202,7 +202,7 @@ int32_t tqLogScanExec(STQ* pTq, STqExecHandle* pExec, SSubmitReq* pReq, SMqDataR
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
tqAddBlockDataToRsp(&block, pRsp);
|
tqAddBlockDataToRsp(&block, pRsp, taosArrayGetSize(block.pDataBlock));
|
||||||
tqAddBlockSchemaToRsp(pExec, workerId, pRsp);
|
tqAddBlockSchemaToRsp(pExec, workerId, pRsp);
|
||||||
pRsp->blockNum++;
|
pRsp->blockNum++;
|
||||||
}
|
}
|
||||||
|
@ -221,7 +221,7 @@ int32_t tqLogScanExec(STQ* pTq, STqExecHandle* pExec, SSubmitReq* pReq, SMqDataR
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
tqAddBlockDataToRsp(&block, pRsp);
|
tqAddBlockDataToRsp(&block, pRsp, taosArrayGetSize(block.pDataBlock));
|
||||||
tqAddBlockSchemaToRsp(pExec, workerId, pRsp);
|
tqAddBlockSchemaToRsp(pExec, workerId, pRsp);
|
||||||
pRsp->blockNum++;
|
pRsp->blockNum++;
|
||||||
}
|
}
|
||||||
|
|
|
@ -92,7 +92,8 @@ int32_t tqMetaOpen(STQ* pTq) {
|
||||||
.initTqReader = true,
|
.initTqReader = true,
|
||||||
.version = handle.snapshotVer,
|
.version = handle.snapshotVer,
|
||||||
};
|
};
|
||||||
handle.execHandle.execCol.task[i] = qCreateQueueExecTaskInfo(handle.execHandle.execCol.qmsg, &reader);
|
|
||||||
|
handle.execHandle.execCol.task[i] = qCreateQueueExecTaskInfo(handle.execHandle.execCol.qmsg, &reader, &handle.execHandle.numOfCols);
|
||||||
ASSERT(handle.execHandle.execCol.task[i]);
|
ASSERT(handle.execHandle.execCol.task[i]);
|
||||||
void* scanner = NULL;
|
void* scanner = NULL;
|
||||||
qExtractStreamScanner(handle.execHandle.execCol.task[i], &scanner);
|
qExtractStreamScanner(handle.execHandle.execCol.task[i], &scanner);
|
||||||
|
|
|
@ -49,8 +49,11 @@ SSubmitReq* tdBlockToSubmit(const SArray* pBlocks, const STSchema* pTSchema, boo
|
||||||
}
|
}
|
||||||
|
|
||||||
SVCreateTbReq createTbReq = {0};
|
SVCreateTbReq createTbReq = {0};
|
||||||
|
SName name = {0};
|
||||||
|
tNameFromString(&name, stbFullName, T_NAME_ACCT | T_NAME_DB | T_NAME_TABLE);
|
||||||
|
|
||||||
createTbReq.name = buildCtbNameByGroupId(stbFullName, pDataBlock->info.groupId);
|
createTbReq.name = buildCtbNameByGroupId(stbFullName, pDataBlock->info.groupId);
|
||||||
createTbReq.ctb.name = strdup(stbFullName);
|
createTbReq.ctb.name = strdup((char*)tNameGetTableName(&name)); // strdup(stbFullName);
|
||||||
createTbReq.flags = 0;
|
createTbReq.flags = 0;
|
||||||
createTbReq.type = TSDB_CHILD_TABLE;
|
createTbReq.type = TSDB_CHILD_TABLE;
|
||||||
createTbReq.ctb.suid = suid;
|
createTbReq.ctb.suid = suid;
|
||||||
|
|
|
@ -464,7 +464,7 @@ static int32_t getNextRowFromFS(void *iter, TSDBROW **ppRow) {
|
||||||
|
|
||||||
switch (state->state) {
|
switch (state->state) {
|
||||||
case SFSNEXTROW_FS:
|
case SFSNEXTROW_FS:
|
||||||
state->aDFileSet = state->pTsdb->fs->cState->aDFileSet;
|
state->aDFileSet = state->pTsdb->pFS->cState->aDFileSet;
|
||||||
state->nFileSet = taosArrayGetSize(state->aDFileSet);
|
state->nFileSet = taosArrayGetSize(state->aDFileSet);
|
||||||
state->iFileSet = state->nFileSet;
|
state->iFileSet = state->nFileSet;
|
||||||
|
|
||||||
|
@ -793,6 +793,9 @@ typedef struct {
|
||||||
TSDBROW memRow, imemRow, fsRow;
|
TSDBROW memRow, imemRow, fsRow;
|
||||||
|
|
||||||
TsdbNextRowState input[3];
|
TsdbNextRowState input[3];
|
||||||
|
SMemTable *pMemTable;
|
||||||
|
SMemTable *pIMemTable;
|
||||||
|
STsdb *pTsdb;
|
||||||
} CacheNextRowIter;
|
} CacheNextRowIter;
|
||||||
|
|
||||||
static int32_t nextRowIterOpen(CacheNextRowIter *pIter, tb_uid_t uid, STsdb *pTsdb) {
|
static int32_t nextRowIterOpen(CacheNextRowIter *pIter, tb_uid_t uid, STsdb *pTsdb) {
|
||||||
|
@ -800,21 +803,25 @@ static int32_t nextRowIterOpen(CacheNextRowIter *pIter, tb_uid_t uid, STsdb *pTs
|
||||||
|
|
||||||
tb_uid_t suid = getTableSuidByUid(uid, pTsdb);
|
tb_uid_t suid = getTableSuidByUid(uid, pTsdb);
|
||||||
|
|
||||||
|
tsdbTakeMemSnapshot(pTsdb, &pIter->pMemTable, &pIter->pIMemTable);
|
||||||
|
|
||||||
STbData *pMem = NULL;
|
STbData *pMem = NULL;
|
||||||
if (pTsdb->mem) {
|
if (pIter->pMemTable) {
|
||||||
tsdbGetTbDataFromMemTable(pTsdb->mem, suid, uid, &pMem);
|
tsdbGetTbDataFromMemTable(pIter->pMemTable, suid, uid, &pMem);
|
||||||
}
|
}
|
||||||
|
|
||||||
STbData *pIMem = NULL;
|
STbData *pIMem = NULL;
|
||||||
if (pTsdb->imem) {
|
if (pIter->pIMemTable) {
|
||||||
tsdbGetTbDataFromMemTable(pTsdb->imem, suid, uid, &pIMem);
|
tsdbGetTbDataFromMemTable(pIter->pIMemTable, suid, uid, &pIMem);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pIter->pTsdb = pTsdb;
|
||||||
|
|
||||||
pIter->pSkyline = taosArrayInit(32, sizeof(TSDBKEY));
|
pIter->pSkyline = taosArrayInit(32, sizeof(TSDBKEY));
|
||||||
|
|
||||||
SDelIdx delIdx;
|
SDelIdx delIdx;
|
||||||
|
|
||||||
SDelFile *pDelFile = tsdbFSStateGetDelFile(pTsdb->fs->cState);
|
SDelFile *pDelFile = tsdbFSStateGetDelFile(pTsdb->pFS->cState);
|
||||||
if (pDelFile) {
|
if (pDelFile) {
|
||||||
SDelFReader *pDelFReader;
|
SDelFReader *pDelFReader;
|
||||||
|
|
||||||
|
@ -878,6 +885,8 @@ static int32_t nextRowIterClose(CacheNextRowIter *pIter) {
|
||||||
taosArrayDestroy(pIter->pSkyline);
|
taosArrayDestroy(pIter->pSkyline);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
tsdbUntakeMemSnapshot(pIter->pTsdb, pIter->pMemTable, pIter->pIMemTable);
|
||||||
|
|
||||||
return code;
|
return code;
|
||||||
_err:
|
_err:
|
||||||
return code;
|
return code;
|
||||||
|
@ -1189,7 +1198,7 @@ static int32_t mergeLastRow(tb_uid_t uid, STsdb *pTsdb, bool *dup, STSRow **ppRo
|
||||||
|
|
||||||
SDelIdx delIdx;
|
SDelIdx delIdx;
|
||||||
|
|
||||||
SDelFile *pDelFile = tsdbFSStateGetDelFile(pTsdb->fs->cState);
|
SDelFile *pDelFile = tsdbFSStateGetDelFile(pTsdb->pFS->cState);
|
||||||
if (pDelFile) {
|
if (pDelFile) {
|
||||||
SDelFReader *pDelFReader;
|
SDelFReader *pDelFReader;
|
||||||
|
|
||||||
|
@ -1377,7 +1386,7 @@ static int32_t mergeLast(tb_uid_t uid, STsdb *pTsdb, SArray **ppLastArray) {
|
||||||
|
|
||||||
SDelIdx delIdx;
|
SDelIdx delIdx;
|
||||||
|
|
||||||
SDelFile *pDelFile = tsdbFSStateGetDelFile(pTsdb->fs->cState);
|
SDelFile *pDelFile = tsdbFSStateGetDelFile(pTsdb->pFS->cState);
|
||||||
if (pDelFile) {
|
if (pDelFile) {
|
||||||
SDelFReader *pDelFReader;
|
SDelFReader *pDelFReader;
|
||||||
|
|
||||||
|
|
|
@ -64,9 +64,26 @@ int32_t tsdbBegin(STsdb *pTsdb) {
|
||||||
|
|
||||||
if (!pTsdb) return code;
|
if (!pTsdb) return code;
|
||||||
|
|
||||||
code = tsdbMemTableCreate(pTsdb, &pTsdb->mem);
|
SMemTable *pMemTable;
|
||||||
|
code = tsdbMemTableCreate(pTsdb, &pMemTable);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
|
// lock
|
||||||
|
code = taosThreadRwlockWrlock(&pTsdb->rwLock);
|
||||||
|
if (code) {
|
||||||
|
code = TAOS_SYSTEM_ERROR(code);
|
||||||
|
goto _err;
|
||||||
|
}
|
||||||
|
|
||||||
|
pTsdb->mem = pMemTable;
|
||||||
|
|
||||||
|
// unlock
|
||||||
|
code = taosThreadRwlockUnlock(&pTsdb->rwLock);
|
||||||
|
if (code) {
|
||||||
|
code = TAOS_SYSTEM_ERROR(code);
|
||||||
|
goto _err;
|
||||||
|
}
|
||||||
|
|
||||||
return code;
|
return code;
|
||||||
|
|
||||||
_err:
|
_err:
|
||||||
|
@ -83,9 +100,11 @@ int32_t tsdbCommit(STsdb *pTsdb) {
|
||||||
|
|
||||||
// check
|
// check
|
||||||
if (pMemTable->nRow == 0 && pMemTable->nDel == 0) {
|
if (pMemTable->nRow == 0 && pMemTable->nDel == 0) {
|
||||||
// TODO: lock?
|
taosThreadRwlockWrlock(&pTsdb->rwLock);
|
||||||
pTsdb->mem = NULL;
|
pTsdb->mem = NULL;
|
||||||
tsdbMemTableDestroy(pMemTable);
|
taosThreadRwlockUnlock(&pTsdb->rwLock);
|
||||||
|
|
||||||
|
tsdbUnrefMemTable(pMemTable);
|
||||||
goto _exit;
|
goto _exit;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -139,7 +158,7 @@ static int32_t tsdbCommitDelStart(SCommitter *pCommitter) {
|
||||||
goto _err;
|
goto _err;
|
||||||
}
|
}
|
||||||
|
|
||||||
SDelFile *pDelFileR = pTsdb->fs->nState->pDelFile;
|
SDelFile *pDelFileR = pTsdb->pFS->nState->pDelFile;
|
||||||
if (pDelFileR) {
|
if (pDelFileR) {
|
||||||
code = tsdbDelFReaderOpen(&pCommitter->pDelFReader, pDelFileR, pTsdb, NULL);
|
code = tsdbDelFReaderOpen(&pCommitter->pDelFReader, pDelFileR, pTsdb, NULL);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
@ -228,7 +247,7 @@ static int32_t tsdbCommitDelEnd(SCommitter *pCommitter) {
|
||||||
code = tsdbUpdateDelFileHdr(pCommitter->pDelFWriter);
|
code = tsdbUpdateDelFileHdr(pCommitter->pDelFWriter);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbFSStateUpsertDelFile(pTsdb->fs->nState, &pCommitter->pDelFWriter->fDel);
|
code = tsdbFSStateUpsertDelFile(pTsdb->pFS->nState, &pCommitter->pDelFWriter->fDel);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbDelFWriterClose(&pCommitter->pDelFWriter, 1);
|
code = tsdbDelFWriterClose(&pCommitter->pDelFWriter, 1);
|
||||||
|
@ -263,7 +282,7 @@ static int32_t tsdbCommitFileDataStart(SCommitter *pCommitter) {
|
||||||
taosArrayClear(pCommitter->aBlockIdx);
|
taosArrayClear(pCommitter->aBlockIdx);
|
||||||
tMapDataReset(&pCommitter->oBlockMap);
|
tMapDataReset(&pCommitter->oBlockMap);
|
||||||
tBlockDataReset(&pCommitter->oBlockData);
|
tBlockDataReset(&pCommitter->oBlockData);
|
||||||
pRSet = tsdbFSStateGetDFileSet(pTsdb->fs->nState, pCommitter->commitFid, TD_EQ);
|
pRSet = tsdbFSStateGetDFileSet(pTsdb->pFS->nState, pCommitter->commitFid, TD_EQ);
|
||||||
if (pRSet) {
|
if (pRSet) {
|
||||||
code = tsdbDataFReaderOpen(&pCommitter->pReader, pTsdb, pRSet);
|
code = tsdbDataFReaderOpen(&pCommitter->pReader, pTsdb, pRSet);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
@ -836,7 +855,7 @@ static int32_t tsdbCommitFileDataEnd(SCommitter *pCommitter) {
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
// upsert SDFileSet
|
// upsert SDFileSet
|
||||||
code = tsdbFSStateUpsertDFileSet(pCommitter->pTsdb->fs->nState, tsdbDataFWriterGetWSet(pCommitter->pWriter));
|
code = tsdbFSStateUpsertDFileSet(pCommitter->pTsdb->pFS->nState, tsdbDataFWriterGetWSet(pCommitter->pWriter));
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
// close and sync
|
// close and sync
|
||||||
|
@ -941,10 +960,10 @@ static int32_t tsdbStartCommit(STsdb *pTsdb, SCommitter *pCommitter) {
|
||||||
memset(pCommitter, 0, sizeof(*pCommitter));
|
memset(pCommitter, 0, sizeof(*pCommitter));
|
||||||
ASSERT(pTsdb->mem && pTsdb->imem == NULL);
|
ASSERT(pTsdb->mem && pTsdb->imem == NULL);
|
||||||
|
|
||||||
// lock();
|
taosThreadRwlockWrlock(&pTsdb->rwLock);
|
||||||
pTsdb->imem = pTsdb->mem;
|
pTsdb->imem = pTsdb->mem;
|
||||||
pTsdb->mem = NULL;
|
pTsdb->mem = NULL;
|
||||||
// unlock();
|
taosThreadRwlockUnlock(&pTsdb->rwLock);
|
||||||
|
|
||||||
pCommitter->pTsdb = pTsdb;
|
pCommitter->pTsdb = pTsdb;
|
||||||
pCommitter->commitID = pTsdb->pVnode->state.commitID;
|
pCommitter->commitID = pTsdb->pVnode->state.commitID;
|
||||||
|
@ -954,7 +973,7 @@ static int32_t tsdbStartCommit(STsdb *pTsdb, SCommitter *pCommitter) {
|
||||||
pCommitter->maxRow = pTsdb->pVnode->config.tsdbCfg.maxRows;
|
pCommitter->maxRow = pTsdb->pVnode->config.tsdbCfg.maxRows;
|
||||||
pCommitter->cmprAlg = pTsdb->pVnode->config.tsdbCfg.compression;
|
pCommitter->cmprAlg = pTsdb->pVnode->config.tsdbCfg.compression;
|
||||||
|
|
||||||
code = tsdbFSBegin(pTsdb->fs);
|
code = tsdbFSBegin(pTsdb->pFS);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
return code;
|
return code;
|
||||||
|
@ -1135,13 +1154,16 @@ static int32_t tsdbEndCommit(SCommitter *pCommitter, int32_t eno) {
|
||||||
SMemTable *pMemTable = pTsdb->imem;
|
SMemTable *pMemTable = pTsdb->imem;
|
||||||
|
|
||||||
if (eno == 0) {
|
if (eno == 0) {
|
||||||
code = tsdbFSCommit(pTsdb->fs);
|
code = tsdbFSCommit(pTsdb->pFS);
|
||||||
} else {
|
} else {
|
||||||
code = tsdbFSRollback(pTsdb->fs);
|
code = tsdbFSRollback(pTsdb->pFS);
|
||||||
}
|
}
|
||||||
|
|
||||||
tsdbMemTableDestroy(pMemTable);
|
taosThreadRwlockWrlock(&pTsdb->rwLock);
|
||||||
pTsdb->imem = NULL;
|
pTsdb->imem = NULL;
|
||||||
|
taosThreadRwlockUnlock(&pTsdb->rwLock);
|
||||||
|
|
||||||
|
tsdbUnrefMemTable(pMemTable);
|
||||||
|
|
||||||
tsdbInfo("vgId:%d tsdb end commit", TD_VID(pTsdb->pVnode));
|
tsdbInfo("vgId:%d tsdb end commit", TD_VID(pTsdb->pVnode));
|
||||||
return code;
|
return code;
|
||||||
|
|
|
@ -245,7 +245,7 @@ static int32_t tsdbApplyDelFileChange(STsdbFS *pFS, SDelFile *pFrom, SDelFile *p
|
||||||
char fname[TSDB_FILENAME_LEN];
|
char fname[TSDB_FILENAME_LEN];
|
||||||
|
|
||||||
if (pFrom && pTo) {
|
if (pFrom && pTo) {
|
||||||
if (pFrom != pTo) {
|
if (!tsdbDelFileIsSame(pFrom, pTo)) {
|
||||||
tsdbDelFileName(pFS->pTsdb, pFrom, fname);
|
tsdbDelFileName(pFS->pTsdb, pFrom, fname);
|
||||||
if (taosRemoveFile(fname) < 0) {
|
if (taosRemoveFile(fname) < 0) {
|
||||||
code = TAOS_SYSTEM_ERROR(errno);
|
code = TAOS_SYSTEM_ERROR(errno);
|
||||||
|
|
|
@ -140,6 +140,8 @@ bool tsdbFileIsSame(SDFileSet *pDFileSet1, SDFileSet *pDFileSet2, EDataFileT fty
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool tsdbDelFileIsSame(SDelFile *pDelFile1, SDelFile *pDelFile2) { return pDelFile1->commitID == pDelFile2->commitID; }
|
||||||
|
|
||||||
int32_t tsdbUpdateDFileHdr(TdFilePtr pFD, SDFileSet *pSet, EDataFileT ftype) {
|
int32_t tsdbUpdateDFileHdr(TdFilePtr pFD, SDFileSet *pSet, EDataFileT ftype) {
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
int64_t n;
|
int64_t n;
|
||||||
|
|
|
@ -41,6 +41,7 @@ int32_t tsdbMemTableCreate(STsdb *pTsdb, SMemTable **ppMemTable) {
|
||||||
}
|
}
|
||||||
taosInitRWLatch(&pMemTable->latch);
|
taosInitRWLatch(&pMemTable->latch);
|
||||||
pMemTable->pTsdb = pTsdb;
|
pMemTable->pTsdb = pTsdb;
|
||||||
|
pMemTable->pPool = pTsdb->pVnode->inUse;
|
||||||
pMemTable->nRef = 1;
|
pMemTable->nRef = 1;
|
||||||
pMemTable->minKey = TSKEY_MAX;
|
pMemTable->minKey = TSKEY_MAX;
|
||||||
pMemTable->maxKey = TSKEY_MIN;
|
pMemTable->maxKey = TSKEY_MIN;
|
||||||
|
@ -54,6 +55,7 @@ int32_t tsdbMemTableCreate(STsdb *pTsdb, SMemTable **ppMemTable) {
|
||||||
taosMemoryFree(pMemTable);
|
taosMemoryFree(pMemTable);
|
||||||
goto _err;
|
goto _err;
|
||||||
}
|
}
|
||||||
|
vnodeBufPoolRef(pMemTable->pPool);
|
||||||
|
|
||||||
*ppMemTable = pMemTable;
|
*ppMemTable = pMemTable;
|
||||||
return code;
|
return code;
|
||||||
|
@ -65,6 +67,7 @@ _err:
|
||||||
|
|
||||||
void tsdbMemTableDestroy(SMemTable *pMemTable) {
|
void tsdbMemTableDestroy(SMemTable *pMemTable) {
|
||||||
if (pMemTable) {
|
if (pMemTable) {
|
||||||
|
vnodeBufPoolUnRef(pMemTable->pPool);
|
||||||
taosArrayDestroy(pMemTable->aTbData);
|
taosArrayDestroy(pMemTable->aTbData);
|
||||||
taosMemoryFree(pMemTable);
|
taosMemoryFree(pMemTable);
|
||||||
}
|
}
|
||||||
|
@ -590,3 +593,58 @@ _err:
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t tsdbGetNRowsInTbData(STbData *pTbData) { return pTbData->sl.size; }
|
int32_t tsdbGetNRowsInTbData(STbData *pTbData) { return pTbData->sl.size; }
|
||||||
|
|
||||||
|
void tsdbRefMemTable(SMemTable *pMemTable) {
|
||||||
|
int32_t nRef = atomic_fetch_add_32(&pMemTable->nRef, 1);
|
||||||
|
ASSERT(nRef > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
void tsdbUnrefMemTable(SMemTable *pMemTable) {
|
||||||
|
int32_t nRef = atomic_sub_fetch_32(&pMemTable->nRef, 1);
|
||||||
|
if (nRef == 0) {
|
||||||
|
tsdbMemTableDestroy(pMemTable);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
int32_t tsdbTakeMemSnapshot(STsdb *pTsdb, SMemTable **ppMem, SMemTable **ppIMem) {
|
||||||
|
int32_t code = 0;
|
||||||
|
|
||||||
|
// lock
|
||||||
|
code = taosThreadRwlockRdlock(&pTsdb->rwLock);
|
||||||
|
if (code) {
|
||||||
|
code = TAOS_SYSTEM_ERROR(code);
|
||||||
|
goto _exit;
|
||||||
|
}
|
||||||
|
|
||||||
|
// take snapshot
|
||||||
|
*ppMem = pTsdb->mem;
|
||||||
|
*ppIMem = pTsdb->imem;
|
||||||
|
|
||||||
|
if (*ppMem) {
|
||||||
|
tsdbRefMemTable(*ppMem);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (*ppIMem) {
|
||||||
|
tsdbRefMemTable(*ppIMem);
|
||||||
|
}
|
||||||
|
|
||||||
|
// unlock
|
||||||
|
code = taosThreadRwlockUnlock(&pTsdb->rwLock);
|
||||||
|
if (code) {
|
||||||
|
code = TAOS_SYSTEM_ERROR(code);
|
||||||
|
goto _exit;
|
||||||
|
}
|
||||||
|
|
||||||
|
_exit:
|
||||||
|
return code;
|
||||||
|
}
|
||||||
|
|
||||||
|
void tsdbUntakeMemSnapshot(STsdb *pTsdb, SMemTable *pMem, SMemTable *pIMem) {
|
||||||
|
if (pMem) {
|
||||||
|
tsdbUnrefMemTable(pMem);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (pIMem) {
|
||||||
|
tsdbUnrefMemTable(pIMem);
|
||||||
|
}
|
||||||
|
}
|
|
@ -54,8 +54,7 @@ int tsdbOpen(SVnode *pVnode, STsdb **ppTsdb, const char *dir, STsdbKeepCfg *pKee
|
||||||
sprintf(pTsdb->path, "%s%s%s", pVnode->path, TD_DIRSEP, dir);
|
sprintf(pTsdb->path, "%s%s%s", pVnode->path, TD_DIRSEP, dir);
|
||||||
taosRealPath(pTsdb->path, NULL, slen);
|
taosRealPath(pTsdb->path, NULL, slen);
|
||||||
pTsdb->pVnode = pVnode;
|
pTsdb->pVnode = pVnode;
|
||||||
pTsdb->repoLocked = false;
|
taosThreadRwlockInit(&pTsdb->rwLock, NULL);
|
||||||
taosThreadMutexInit(&pTsdb->mutex, NULL);
|
|
||||||
if (!pKeepCfg) {
|
if (!pKeepCfg) {
|
||||||
tsdbSetKeepCfg(&pTsdb->keepCfg, &pVnode->config.tsdbCfg);
|
tsdbSetKeepCfg(&pTsdb->keepCfg, &pVnode->config.tsdbCfg);
|
||||||
} else {
|
} else {
|
||||||
|
@ -67,7 +66,7 @@ int tsdbOpen(SVnode *pVnode, STsdb **ppTsdb, const char *dir, STsdbKeepCfg *pKee
|
||||||
tfsMkdir(pVnode->pTfs, pTsdb->path);
|
tfsMkdir(pVnode->pTfs, pTsdb->path);
|
||||||
|
|
||||||
// open tsdb
|
// open tsdb
|
||||||
if (tsdbFSOpen(pTsdb, &pTsdb->fs) < 0) {
|
if (tsdbFSOpen(pTsdb, &pTsdb->pFS) < 0) {
|
||||||
goto _err;
|
goto _err;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -88,33 +87,10 @@ _err:
|
||||||
|
|
||||||
int tsdbClose(STsdb **pTsdb) {
|
int tsdbClose(STsdb **pTsdb) {
|
||||||
if (*pTsdb) {
|
if (*pTsdb) {
|
||||||
taosThreadMutexDestroy(&(*pTsdb)->mutex);
|
taosThreadRwlockDestroy(&(*pTsdb)->rwLock);
|
||||||
tsdbFSClose((*pTsdb)->fs);
|
tsdbFSClose((*pTsdb)->pFS);
|
||||||
tsdbCloseCache((*pTsdb)->lruCache);
|
tsdbCloseCache((*pTsdb)->lruCache);
|
||||||
taosMemoryFreeClear(*pTsdb);
|
taosMemoryFreeClear(*pTsdb);
|
||||||
}
|
}
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
int tsdbLockRepo(STsdb *pTsdb) {
|
|
||||||
int code = taosThreadMutexLock(&pTsdb->mutex);
|
|
||||||
if (code != 0) {
|
|
||||||
tsdbError("vgId:%d, failed to lock tsdb since %s", TD_VID(pTsdb->pVnode), strerror(errno));
|
|
||||||
terrno = TAOS_SYSTEM_ERROR(code);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
pTsdb->repoLocked = true;
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
||||||
int tsdbUnlockRepo(STsdb *pTsdb) {
|
|
||||||
// ASSERT(IS_REPO_LOCKED(pTsdb));
|
|
||||||
pTsdb->repoLocked = false;
|
|
||||||
int code = taosThreadMutexUnlock(&pTsdb->mutex);
|
|
||||||
if (code != 0) {
|
|
||||||
tsdbError("vgId:%d, failed to unlock tsdb since %s", TD_VID(pTsdb->pVnode), strerror(errno));
|
|
||||||
terrno = TAOS_SYSTEM_ERROR(code);
|
|
||||||
return -1;
|
|
||||||
}
|
|
||||||
return 0;
|
|
||||||
}
|
|
||||||
|
|
|
@ -118,6 +118,8 @@ struct STsdbReader {
|
||||||
char* idStr; // query info handle, for debug purpose
|
char* idStr; // query info handle, for debug purpose
|
||||||
int32_t type; // query type: 1. retrieve all data blocks, 2. retrieve direct prev|next rows
|
int32_t type; // query type: 1. retrieve all data blocks, 2. retrieve direct prev|next rows
|
||||||
SBlockLoadSuppInfo suppInfo;
|
SBlockLoadSuppInfo suppInfo;
|
||||||
|
SMemTable* pMem;
|
||||||
|
SMemTable* pIMem;
|
||||||
|
|
||||||
SIOCostSummary cost;
|
SIOCostSummary cost;
|
||||||
STSchema* pSchema;
|
STSchema* pSchema;
|
||||||
|
@ -1453,38 +1455,71 @@ static bool keyOverlapFileBlock(TSDBKEY key, SBlock* pBlock, SVersionRange* pVer
|
||||||
(pBlock->minVersion <= pVerRange->maxVer);
|
(pBlock->minVersion <= pVerRange->maxVer);
|
||||||
}
|
}
|
||||||
|
|
||||||
static bool overlapWithDelSkyline(STableBlockScanInfo* pBlockScanInfo, const SBlock* pBlock, int32_t order) {
|
|
||||||
if (pBlockScanInfo->delSkyline == NULL) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
TSDBKEY* pFirst = taosArrayGet(pBlockScanInfo->delSkyline, 0);
|
static bool doCheckforDatablockOverlap(STableBlockScanInfo* pBlockScanInfo, const SBlock* pBlock) {
|
||||||
TSDBKEY* pLast = taosArrayGetLast(pBlockScanInfo->delSkyline);
|
|
||||||
|
|
||||||
// ts is not overlap
|
|
||||||
if (pBlock->minKey.ts > pLast->ts || pBlock->maxKey.ts < pFirst->ts) {
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
|
|
||||||
int32_t step = ASCENDING_TRAVERSE(order) ? 1 : -1;
|
|
||||||
|
|
||||||
// version is not overlap
|
|
||||||
size_t num = taosArrayGetSize(pBlockScanInfo->delSkyline);
|
size_t num = taosArrayGetSize(pBlockScanInfo->delSkyline);
|
||||||
for (int32_t i = pBlockScanInfo->fileDelIndex; i < num; i += step) {
|
|
||||||
|
for (int32_t i = pBlockScanInfo->fileDelIndex; i < num; i += 1) {
|
||||||
TSDBKEY* p = taosArrayGet(pBlockScanInfo->delSkyline, i);
|
TSDBKEY* p = taosArrayGet(pBlockScanInfo->delSkyline, i);
|
||||||
if (p->ts >= pBlock->minKey.ts && p->ts <= pBlock->maxKey.ts) {
|
if (p->ts >= pBlock->minKey.ts && p->ts <= pBlock->maxKey.ts) {
|
||||||
if (p->version >= pBlock->minVersion) {
|
if (p->version >= pBlock->minVersion) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
} else if (p->ts > pBlock->maxKey.ts) {
|
} else if (p->ts < pBlock->minKey.ts) { // p->ts < pBlock->minKey.ts
|
||||||
|
if (p->version >= pBlock->minVersion) {
|
||||||
|
if (i < num - 1) {
|
||||||
|
TSDBKEY* pnext = taosArrayGet(pBlockScanInfo->delSkyline, i + 1);
|
||||||
|
if (i + 1 == num - 1) { // pnext is the last point
|
||||||
|
if (pnext->ts >= pBlock->minKey.ts) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
if (pnext->ts >= pBlock->minKey.ts && pnext->version >= pBlock->minVersion) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else { // it must be the last point
|
||||||
|
ASSERT(p->version == 0);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else { // (p->ts > pBlock->maxKey.ts) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
ASSERT(0);
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
static bool overlapWithDelSkyline(STableBlockScanInfo* pBlockScanInfo, const SBlock* pBlock, int32_t order) {
|
||||||
|
if (pBlockScanInfo->delSkyline == NULL) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ts is not overlap
|
||||||
|
TSDBKEY* pFirst = taosArrayGet(pBlockScanInfo->delSkyline, 0);
|
||||||
|
TSDBKEY* pLast = taosArrayGetLast(pBlockScanInfo->delSkyline);
|
||||||
|
if (pBlock->minKey.ts > pLast->ts || pBlock->maxKey.ts < pFirst->ts) {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
// version is not overlap
|
||||||
|
if (ASCENDING_TRAVERSE(order)) {
|
||||||
|
return doCheckforDatablockOverlap(pBlockScanInfo, pBlock);
|
||||||
|
} else {
|
||||||
|
int32_t index = pBlockScanInfo->fileDelIndex;
|
||||||
|
while(1) {
|
||||||
|
TSDBKEY* p = taosArrayGet(pBlockScanInfo->delSkyline, index);
|
||||||
|
if (p->ts > pBlock->minKey.ts && index > 0) {
|
||||||
|
index -= 1;
|
||||||
|
} else { // find the first point that is smaller than the minKey.ts of dataBlock.
|
||||||
|
break;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return doCheckforDatablockOverlap(pBlockScanInfo, pBlock);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// 1. the version of all rows should be less than the endVersion
|
// 1. the version of all rows should be less than the endVersion
|
||||||
// 2. current block should not overlap with next neighbor block
|
// 2. current block should not overlap with next neighbor block
|
||||||
// 3. current timestamp should not be overlap with each other
|
// 3. current timestamp should not be overlap with each other
|
||||||
|
@ -1847,8 +1882,8 @@ static int32_t initMemDataIterator(STableBlockScanInfo* pBlockScanInfo, STsdbRea
|
||||||
int32_t backward = (!ASCENDING_TRAVERSE(pReader->order));
|
int32_t backward = (!ASCENDING_TRAVERSE(pReader->order));
|
||||||
|
|
||||||
STbData* d = NULL;
|
STbData* d = NULL;
|
||||||
if (pReader->pTsdb->mem != NULL) {
|
if (pReader->pMem != NULL) {
|
||||||
tsdbGetTbDataFromMemTable(pReader->pTsdb->mem, pReader->suid, pBlockScanInfo->uid, &d);
|
tsdbGetTbDataFromMemTable(pReader->pMem, pReader->suid, pBlockScanInfo->uid, &d);
|
||||||
if (d != NULL) {
|
if (d != NULL) {
|
||||||
code = tsdbTbDataIterCreate(d, &startKey, backward, &pBlockScanInfo->iter.iter);
|
code = tsdbTbDataIterCreate(d, &startKey, backward, &pBlockScanInfo->iter.iter);
|
||||||
if (code == TSDB_CODE_SUCCESS) {
|
if (code == TSDB_CODE_SUCCESS) {
|
||||||
|
@ -1868,8 +1903,8 @@ static int32_t initMemDataIterator(STableBlockScanInfo* pBlockScanInfo, STsdbRea
|
||||||
}
|
}
|
||||||
|
|
||||||
STbData* di = NULL;
|
STbData* di = NULL;
|
||||||
if (pReader->pTsdb->imem != NULL) {
|
if (pReader->pIMem != NULL) {
|
||||||
tsdbGetTbDataFromMemTable(pReader->pTsdb->imem, pReader->suid, pBlockScanInfo->uid, &di);
|
tsdbGetTbDataFromMemTable(pReader->pIMem, pReader->suid, pBlockScanInfo->uid, &di);
|
||||||
if (di != NULL) {
|
if (di != NULL) {
|
||||||
code = tsdbTbDataIterCreate(di, &startKey, backward, &pBlockScanInfo->iiter.iter);
|
code = tsdbTbDataIterCreate(di, &startKey, backward, &pBlockScanInfo->iiter.iter);
|
||||||
if (code == TSDB_CODE_SUCCESS) {
|
if (code == TSDB_CODE_SUCCESS) {
|
||||||
|
@ -1905,7 +1940,7 @@ int32_t initDelSkylineIterator(STableBlockScanInfo* pBlockScanInfo, STsdbReader*
|
||||||
|
|
||||||
SArray* pDelData = taosArrayInit(4, sizeof(SDelData));
|
SArray* pDelData = taosArrayInit(4, sizeof(SDelData));
|
||||||
|
|
||||||
SDelFile* pDelFile = tsdbFSStateGetDelFile(pTsdb->fs->cState);
|
SDelFile* pDelFile = tsdbFSStateGetDelFile(pTsdb->pFS->cState);
|
||||||
if (pDelFile) {
|
if (pDelFile) {
|
||||||
SDelFReader* pDelFReader = NULL;
|
SDelFReader* pDelFReader = NULL;
|
||||||
code = tsdbDelFReaderOpen(&pDelFReader, pDelFile, pTsdb, NULL);
|
code = tsdbDelFReaderOpen(&pDelFReader, pDelFile, pTsdb, NULL);
|
||||||
|
@ -1926,11 +1961,13 @@ int32_t initDelSkylineIterator(STableBlockScanInfo* pBlockScanInfo, STsdbReader*
|
||||||
SDelIdx idx = {.suid = pReader->suid, .uid = pBlockScanInfo->uid};
|
SDelIdx idx = {.suid = pReader->suid, .uid = pBlockScanInfo->uid};
|
||||||
SDelIdx* pIdx = taosArraySearch(aDelIdx, &idx, tCmprDelIdx, TD_EQ);
|
SDelIdx* pIdx = taosArraySearch(aDelIdx, &idx, tCmprDelIdx, TD_EQ);
|
||||||
|
|
||||||
|
if (pIdx != NULL) {
|
||||||
code = tsdbReadDelData(pDelFReader, pIdx, pDelData, NULL);
|
code = tsdbReadDelData(pDelFReader, pIdx, pDelData, NULL);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
goto _err;
|
goto _err;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
SDelData* p = NULL;
|
SDelData* p = NULL;
|
||||||
if (pMemTbData != NULL) {
|
if (pMemTbData != NULL) {
|
||||||
|
@ -2795,7 +2832,7 @@ int32_t tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, SArray* pTabl
|
||||||
|
|
||||||
SDataBlockIter* pBlockIter = &pReader->status.blockIter;
|
SDataBlockIter* pBlockIter = &pReader->status.blockIter;
|
||||||
|
|
||||||
STsdbFSState* pFState = pReader->pTsdb->fs->cState;
|
STsdbFSState* pFState = pReader->pTsdb->pFS->cState;
|
||||||
initFilesetIterator(&pReader->status.fileIter, pFState, pReader->order, pReader->idStr);
|
initFilesetIterator(&pReader->status.fileIter, pFState, pReader->order, pReader->idStr);
|
||||||
resetDataBlockIterator(&pReader->status.blockIter, pReader->order);
|
resetDataBlockIterator(&pReader->status.blockIter, pReader->order);
|
||||||
|
|
||||||
|
@ -2809,6 +2846,8 @@ int32_t tsdbReaderOpen(SVnode* pVnode, SQueryTableDataCond* pCond, SArray* pTabl
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
tsdbTakeMemSnapshot(pReader->pTsdb, &pReader->pMem, &pReader->pIMem);
|
||||||
|
|
||||||
tsdbDebug("%p total numOfTable:%d in this query %s", pReader, numOfTables, pReader->idStr);
|
tsdbDebug("%p total numOfTable:%d in this query %s", pReader, numOfTables, pReader->idStr);
|
||||||
return code;
|
return code;
|
||||||
|
|
||||||
|
@ -2824,6 +2863,8 @@ void tsdbReaderClose(STsdbReader* pReader) {
|
||||||
|
|
||||||
SBlockLoadSuppInfo* pSupInfo = &pReader->suppInfo;
|
SBlockLoadSuppInfo* pSupInfo = &pReader->suppInfo;
|
||||||
|
|
||||||
|
tsdbUntakeMemSnapshot(pReader->pTsdb, pReader->pMem, pReader->pIMem);
|
||||||
|
|
||||||
taosMemoryFreeClear(pSupInfo->plist);
|
taosMemoryFreeClear(pSupInfo->plist);
|
||||||
taosMemoryFree(pSupInfo->colIds);
|
taosMemoryFree(pSupInfo->colIds);
|
||||||
|
|
||||||
|
@ -3042,7 +3083,7 @@ int32_t tsdbReaderReset(STsdbReader* pReader, SQueryTableDataCond* pCond) {
|
||||||
|
|
||||||
tsdbDataFReaderClose(&pReader->pFileReader);
|
tsdbDataFReaderClose(&pReader->pFileReader);
|
||||||
|
|
||||||
STsdbFSState* pFState = pReader->pTsdb->fs->cState;
|
STsdbFSState* pFState = pReader->pTsdb->pFS->cState;
|
||||||
initFilesetIterator(&pReader->status.fileIter, pFState, pReader->order, pReader->idStr);
|
initFilesetIterator(&pReader->status.fileIter, pFState, pReader->order, pReader->idStr);
|
||||||
resetDataBlockIterator(&pReader->status.blockIter, pReader->order);
|
resetDataBlockIterator(&pReader->status.blockIter, pReader->order);
|
||||||
resetDataBlockScanInfo(pReader->status.pTableMap);
|
resetDataBlockScanInfo(pReader->status.pTableMap);
|
||||||
|
|
|
@ -246,7 +246,7 @@ int32_t tsdbDelFReaderOpen(SDelFReader **ppReader, SDelFile *pFile, STsdb *pTsdb
|
||||||
|
|
||||||
tsdbDelFileName(pTsdb, pFile, fname);
|
tsdbDelFileName(pTsdb, pFile, fname);
|
||||||
pDelFReader->pReadH = taosOpenFile(fname, TD_FILE_READ);
|
pDelFReader->pReadH = taosOpenFile(fname, TD_FILE_READ);
|
||||||
if (pDelFReader == NULL) {
|
if (pDelFReader->pReadH == NULL) {
|
||||||
code = TAOS_SYSTEM_ERROR(errno);
|
code = TAOS_SYSTEM_ERROR(errno);
|
||||||
taosMemoryFree(pDelFReader);
|
taosMemoryFree(pDelFReader);
|
||||||
goto _err;
|
goto _err;
|
||||||
|
|
|
@ -20,10 +20,10 @@ static int32_t tsdbDoRetentionImpl(STsdb *pTsdb, int64_t now, int8_t try, int8_t
|
||||||
STsdbFSState *pState;
|
STsdbFSState *pState;
|
||||||
|
|
||||||
if (try) {
|
if (try) {
|
||||||
pState = pTsdb->fs->cState;
|
pState = pTsdb->pFS->cState;
|
||||||
*canDo = 0;
|
*canDo = 0;
|
||||||
} else {
|
} else {
|
||||||
pState = pTsdb->fs->nState;
|
pState = pTsdb->pFS->nState;
|
||||||
}
|
}
|
||||||
|
|
||||||
for (int32_t iSet = 0; iSet < taosArrayGetSize(pState->aDFileSet); iSet++) {
|
for (int32_t iSet = 0; iSet < taosArrayGetSize(pState->aDFileSet); iSet++) {
|
||||||
|
@ -83,7 +83,7 @@ int32_t tsdbDoRetention(STsdb *pTsdb, int64_t now) {
|
||||||
if (!canDo) goto _exit;
|
if (!canDo) goto _exit;
|
||||||
|
|
||||||
// begin
|
// begin
|
||||||
code = tsdbFSBegin(pTsdb->fs);
|
code = tsdbFSBegin(pTsdb->pFS);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
// do retention
|
// do retention
|
||||||
|
@ -91,7 +91,7 @@ int32_t tsdbDoRetention(STsdb *pTsdb, int64_t now) {
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
// commit
|
// commit
|
||||||
code = tsdbFSCommit(pTsdb->fs);
|
code = tsdbFSCommit(pTsdb->pFS);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
_exit:
|
_exit:
|
||||||
|
@ -99,6 +99,6 @@ _exit:
|
||||||
|
|
||||||
_err:
|
_err:
|
||||||
tsdbError("vgId:%d tsdb do retention failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
|
tsdbError("vgId:%d tsdb do retention failed since %s", TD_VID(pTsdb->pVnode), tstrerror(code));
|
||||||
tsdbFSRollback(pTsdb->fs);
|
tsdbFSRollback(pTsdb->pFS);
|
||||||
return code;
|
return code;
|
||||||
}
|
}
|
|
@ -45,7 +45,7 @@ static int32_t tsdbSnapReadData(STsdbSnapReader* pReader, uint8_t** ppData) {
|
||||||
|
|
||||||
while (true) {
|
while (true) {
|
||||||
if (pReader->pDataFReader == NULL) {
|
if (pReader->pDataFReader == NULL) {
|
||||||
SDFileSet* pSet = tsdbFSStateGetDFileSet(pTsdb->fs->cState, pReader->fid, TD_GT);
|
SDFileSet* pSet = tsdbFSStateGetDFileSet(pTsdb->pFS->cState, pReader->fid, TD_GT);
|
||||||
|
|
||||||
if (pSet == NULL) goto _exit;
|
if (pSet == NULL) goto _exit;
|
||||||
|
|
||||||
|
@ -159,7 +159,7 @@ _err:
|
||||||
static int32_t tsdbSnapReadDel(STsdbSnapReader* pReader, uint8_t** ppData) {
|
static int32_t tsdbSnapReadDel(STsdbSnapReader* pReader, uint8_t** ppData) {
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
STsdb* pTsdb = pReader->pTsdb;
|
STsdb* pTsdb = pReader->pTsdb;
|
||||||
SDelFile* pDelFile = pTsdb->fs->cState->pDelFile;
|
SDelFile* pDelFile = pTsdb->pFS->cState->pDelFile;
|
||||||
|
|
||||||
if (pReader->pDelFReader == NULL) {
|
if (pReader->pDelFReader == NULL) {
|
||||||
if (pDelFile == NULL) {
|
if (pDelFile == NULL) {
|
||||||
|
@ -798,7 +798,7 @@ static int32_t tsdbSnapWriteDataEnd(STsdbSnapWriter* pWriter) {
|
||||||
code = tsdbWriteBlockIdx(pWriter->pDataFWriter, pWriter->aBlockIdxW, NULL);
|
code = tsdbWriteBlockIdx(pWriter->pDataFWriter, pWriter->aBlockIdxW, NULL);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbFSStateUpsertDFileSet(pTsdb->fs->nState, tsdbDataFWriterGetWSet(pWriter->pDataFWriter));
|
code = tsdbFSStateUpsertDFileSet(pTsdb->pFS->nState, tsdbDataFWriterGetWSet(pWriter->pDataFWriter));
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbDataFWriterClose(&pWriter->pDataFWriter, 1);
|
code = tsdbDataFWriterClose(&pWriter->pDataFWriter, 1);
|
||||||
|
@ -843,7 +843,7 @@ static int32_t tsdbSnapWriteData(STsdbSnapWriter* pWriter, uint8_t* pData, uint3
|
||||||
pWriter->fid = fid;
|
pWriter->fid = fid;
|
||||||
|
|
||||||
// read
|
// read
|
||||||
SDFileSet* pSet = tsdbFSStateGetDFileSet(pTsdb->fs->nState, fid, TD_EQ);
|
SDFileSet* pSet = tsdbFSStateGetDFileSet(pTsdb->pFS->nState, fid, TD_EQ);
|
||||||
if (pSet) {
|
if (pSet) {
|
||||||
code = tsdbDataFReaderOpen(&pWriter->pDataFReader, pTsdb, pSet);
|
code = tsdbDataFReaderOpen(&pWriter->pDataFReader, pTsdb, pSet);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
@ -907,7 +907,7 @@ static int32_t tsdbSnapWriteDel(STsdbSnapWriter* pWriter, uint8_t* pData, uint32
|
||||||
STsdb* pTsdb = pWriter->pTsdb;
|
STsdb* pTsdb = pWriter->pTsdb;
|
||||||
|
|
||||||
if (pWriter->pDelFWriter == NULL) {
|
if (pWriter->pDelFWriter == NULL) {
|
||||||
SDelFile* pDelFile = tsdbFSStateGetDelFile(pTsdb->fs->nState);
|
SDelFile* pDelFile = tsdbFSStateGetDelFile(pTsdb->pFS->nState);
|
||||||
|
|
||||||
// reader
|
// reader
|
||||||
if (pDelFile) {
|
if (pDelFile) {
|
||||||
|
@ -1017,7 +1017,7 @@ static int32_t tsdbSnapWriteDelEnd(STsdbSnapWriter* pWriter) {
|
||||||
code = tsdbUpdateDelFileHdr(pWriter->pDelFWriter);
|
code = tsdbUpdateDelFileHdr(pWriter->pDelFWriter);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbFSStateUpsertDelFile(pTsdb->fs->nState, &pWriter->pDelFWriter->fDel);
|
code = tsdbFSStateUpsertDelFile(pTsdb->pFS->nState, &pWriter->pDelFWriter->fDel);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbDelFWriterClose(&pWriter->pDelFWriter, 1);
|
code = tsdbDelFWriterClose(&pWriter->pDelFWriter, 1);
|
||||||
|
@ -1096,7 +1096,7 @@ int32_t tsdbSnapWriterOpen(STsdb* pTsdb, int64_t sver, int64_t ever, STsdbSnapWr
|
||||||
goto _err;
|
goto _err;
|
||||||
}
|
}
|
||||||
|
|
||||||
code = tsdbFSBegin(pTsdb->fs);
|
code = tsdbFSBegin(pTsdb->pFS);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
*ppWriter = pWriter;
|
*ppWriter = pWriter;
|
||||||
|
@ -1113,7 +1113,7 @@ int32_t tsdbSnapWriterClose(STsdbSnapWriter** ppWriter, int8_t rollback) {
|
||||||
STsdbSnapWriter* pWriter = *ppWriter;
|
STsdbSnapWriter* pWriter = *ppWriter;
|
||||||
|
|
||||||
if (rollback) {
|
if (rollback) {
|
||||||
code = tsdbFSRollback(pWriter->pTsdb->fs);
|
code = tsdbFSRollback(pWriter->pTsdb->pFS);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
} else {
|
} else {
|
||||||
code = tsdbSnapWriteDataEnd(pWriter);
|
code = tsdbSnapWriteDataEnd(pWriter);
|
||||||
|
@ -1122,7 +1122,7 @@ int32_t tsdbSnapWriterClose(STsdbSnapWriter** ppWriter, int8_t rollback) {
|
||||||
code = tsdbSnapWriteDelEnd(pWriter);
|
code = tsdbSnapWriteDelEnd(pWriter);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
|
|
||||||
code = tsdbFSCommit(pWriter->pTsdb->fs);
|
code = tsdbFSCommit(pWriter->pTsdb->pFS);
|
||||||
if (code) goto _err;
|
if (code) goto _err;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -17,7 +17,7 @@
|
||||||
|
|
||||||
/* ------------------------ STRUCTURES ------------------------ */
|
/* ------------------------ STRUCTURES ------------------------ */
|
||||||
|
|
||||||
static int vnodeBufPoolCreate(int64_t size, SVBufPool **ppPool);
|
static int vnodeBufPoolCreate(SVnode *pVnode, int64_t size, SVBufPool **ppPool);
|
||||||
static int vnodeBufPoolDestroy(SVBufPool *pPool);
|
static int vnodeBufPoolDestroy(SVBufPool *pPool);
|
||||||
|
|
||||||
int vnodeOpenBufPool(SVnode *pVnode, int64_t size) {
|
int vnodeOpenBufPool(SVnode *pVnode, int64_t size) {
|
||||||
|
@ -28,7 +28,7 @@ int vnodeOpenBufPool(SVnode *pVnode, int64_t size) {
|
||||||
|
|
||||||
for (int i = 0; i < 3; i++) {
|
for (int i = 0; i < 3; i++) {
|
||||||
// create pool
|
// create pool
|
||||||
ret = vnodeBufPoolCreate(size, &pPool);
|
ret = vnodeBufPoolCreate(pVnode, size, &pPool);
|
||||||
if (ret < 0) {
|
if (ret < 0) {
|
||||||
vError("vgId:%d, failed to open vnode buffer pool since %s", TD_VID(pVnode), tstrerror(terrno));
|
vError("vgId:%d, failed to open vnode buffer pool since %s", TD_VID(pVnode), tstrerror(terrno));
|
||||||
vnodeCloseBufPool(pVnode);
|
vnodeCloseBufPool(pVnode);
|
||||||
|
@ -120,7 +120,7 @@ void vnodeBufPoolFree(SVBufPool *pPool, void *p) {
|
||||||
}
|
}
|
||||||
|
|
||||||
// STATIC METHODS -------------------
|
// STATIC METHODS -------------------
|
||||||
static int vnodeBufPoolCreate(int64_t size, SVBufPool **ppPool) {
|
static int vnodeBufPoolCreate(SVnode *pVnode, int64_t size, SVBufPool **ppPool) {
|
||||||
SVBufPool *pPool;
|
SVBufPool *pPool;
|
||||||
|
|
||||||
pPool = taosMemoryMalloc(sizeof(SVBufPool) + size);
|
pPool = taosMemoryMalloc(sizeof(SVBufPool) + size);
|
||||||
|
@ -130,6 +130,7 @@ static int vnodeBufPoolCreate(int64_t size, SVBufPool **ppPool) {
|
||||||
}
|
}
|
||||||
|
|
||||||
pPool->next = NULL;
|
pPool->next = NULL;
|
||||||
|
pPool->pVnode = pVnode;
|
||||||
pPool->nRef = 0;
|
pPool->nRef = 0;
|
||||||
pPool->size = 0;
|
pPool->size = 0;
|
||||||
pPool->ptr = pPool->node.data;
|
pPool->ptr = pPool->node.data;
|
||||||
|
@ -147,3 +148,25 @@ static int vnodeBufPoolDestroy(SVBufPool *pPool) {
|
||||||
taosMemoryFree(pPool);
|
taosMemoryFree(pPool);
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void vnodeBufPoolRef(SVBufPool *pPool) {
|
||||||
|
int32_t nRef = atomic_fetch_add_32(&pPool->nRef, 1);
|
||||||
|
ASSERT(nRef > 0);
|
||||||
|
}
|
||||||
|
|
||||||
|
void vnodeBufPoolUnRef(SVBufPool *pPool) {
|
||||||
|
int32_t nRef = atomic_sub_fetch_32(&pPool->nRef, 1);
|
||||||
|
if (nRef == 0) {
|
||||||
|
SVnode *pVnode = pPool->pVnode;
|
||||||
|
|
||||||
|
vnodeBufPoolReset(pPool);
|
||||||
|
|
||||||
|
taosThreadMutexLock(&pVnode->mutex);
|
||||||
|
|
||||||
|
pPool->next = pVnode->pPool;
|
||||||
|
pVnode->pPool = pPool;
|
||||||
|
taosThreadCondSignal(&pVnode->poolNotEmpty);
|
||||||
|
|
||||||
|
taosThreadMutexUnlock(&pVnode->mutex);
|
||||||
|
}
|
||||||
|
}
|
|
@ -27,18 +27,18 @@ static void vnodeWaitCommit(SVnode *pVnode);
|
||||||
|
|
||||||
int vnodeBegin(SVnode *pVnode) {
|
int vnodeBegin(SVnode *pVnode) {
|
||||||
// alloc buffer pool
|
// alloc buffer pool
|
||||||
/* pthread_mutex_lock(); */
|
taosThreadMutexLock(&pVnode->mutex);
|
||||||
|
|
||||||
while (pVnode->pPool == NULL) {
|
while (pVnode->pPool == NULL) {
|
||||||
/* pthread_cond_wait(); */
|
taosThreadCondWait(&pVnode->poolNotEmpty, &pVnode->mutex);
|
||||||
}
|
}
|
||||||
|
|
||||||
pVnode->inUse = pVnode->pPool;
|
pVnode->inUse = pVnode->pPool;
|
||||||
|
pVnode->inUse->nRef = 1;
|
||||||
pVnode->pPool = pVnode->inUse->next;
|
pVnode->pPool = pVnode->inUse->next;
|
||||||
pVnode->inUse->next = NULL;
|
pVnode->inUse->next = NULL;
|
||||||
/* ref pVnode->inUse buffer pool */
|
|
||||||
|
|
||||||
/* pthread_mutex_unlock(); */
|
taosThreadMutexUnlock(&pVnode->mutex);
|
||||||
|
|
||||||
pVnode->state.commitID++;
|
pVnode->state.commitID++;
|
||||||
// begin meta
|
// begin meta
|
||||||
|
@ -217,7 +217,7 @@ int vnodeCommit(SVnode *pVnode) {
|
||||||
vInfo("vgId:%d, start to commit, commit ID:%" PRId64 " version:%" PRId64, TD_VID(pVnode), pVnode->state.commitID,
|
vInfo("vgId:%d, start to commit, commit ID:%" PRId64 " version:%" PRId64, TD_VID(pVnode), pVnode->state.commitID,
|
||||||
pVnode->state.applied);
|
pVnode->state.applied);
|
||||||
|
|
||||||
pVnode->onCommit = pVnode->inUse;
|
vnodeBufPoolUnRef(pVnode->inUse);
|
||||||
pVnode->inUse = NULL;
|
pVnode->inUse = NULL;
|
||||||
|
|
||||||
// save info
|
// save info
|
||||||
|
@ -284,10 +284,6 @@ int vnodeCommit(SVnode *pVnode) {
|
||||||
|
|
||||||
// apply the commit (TODO)
|
// apply the commit (TODO)
|
||||||
walEndSnapshot(pVnode->pWal);
|
walEndSnapshot(pVnode->pWal);
|
||||||
vnodeBufPoolReset(pVnode->onCommit);
|
|
||||||
pVnode->onCommit->next = pVnode->pPool;
|
|
||||||
pVnode->pPool = pVnode->onCommit;
|
|
||||||
pVnode->onCommit = NULL;
|
|
||||||
|
|
||||||
vInfo("vgId:%d, commit over", TD_VID(pVnode));
|
vInfo("vgId:%d, commit over", TD_VID(pVnode));
|
||||||
|
|
||||||
|
|
|
@ -89,6 +89,8 @@ SVnode *vnodeOpen(const char *path, STfs *pTfs, SMsgCb msgCb) {
|
||||||
|
|
||||||
tsem_init(&pVnode->syncSem, 0, 0);
|
tsem_init(&pVnode->syncSem, 0, 0);
|
||||||
tsem_init(&(pVnode->canCommit), 0, 1);
|
tsem_init(&(pVnode->canCommit), 0, 1);
|
||||||
|
taosThreadMutexInit(&pVnode->mutex, NULL);
|
||||||
|
taosThreadCondInit(&pVnode->poolNotEmpty, NULL);
|
||||||
|
|
||||||
// open buffer pool
|
// open buffer pool
|
||||||
if (vnodeOpenBufPool(pVnode, pVnode->config.isHeap ? 0 : pVnode->config.szBuf / 3) < 0) {
|
if (vnodeOpenBufPool(pVnode, pVnode->config.isHeap ? 0 : pVnode->config.szBuf / 3) < 0) {
|
||||||
|
@ -195,6 +197,8 @@ void vnodeClose(SVnode *pVnode) {
|
||||||
// destroy handle
|
// destroy handle
|
||||||
tsem_destroy(&(pVnode->canCommit));
|
tsem_destroy(&(pVnode->canCommit));
|
||||||
tsem_destroy(&pVnode->syncSem);
|
tsem_destroy(&pVnode->syncSem);
|
||||||
|
taosThreadCondDestroy(&pVnode->poolNotEmpty);
|
||||||
|
taosThreadMutexDestroy(&pVnode->mutex);
|
||||||
taosMemoryFree(pVnode);
|
taosMemoryFree(pVnode);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
@ -27,6 +27,8 @@ struct SVSnapReader {
|
||||||
// tsdb
|
// tsdb
|
||||||
int8_t tsdbDone;
|
int8_t tsdbDone;
|
||||||
STsdbSnapReader *pTsdbReader;
|
STsdbSnapReader *pTsdbReader;
|
||||||
|
// rsma
|
||||||
|
int8_t rsmaDone[TSDB_RETENTION_L2];
|
||||||
};
|
};
|
||||||
|
|
||||||
int32_t vnodeSnapReaderOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapReader **ppReader) {
|
int32_t vnodeSnapReaderOpen(SVnode *pVnode, int64_t sver, int64_t ever, SVSnapReader **ppReader) {
|
||||||
|
@ -115,6 +117,42 @@ int32_t vnodeSnapRead(SVSnapReader *pReader, uint8_t **ppData, uint32_t *nData)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// RSMA ==============
|
||||||
|
#if 0
|
||||||
|
if (VND_IS_RSMA(pReader->pVnode)) {
|
||||||
|
// RSMA1/RSMA2
|
||||||
|
for (int32_t i = 0; i < TSDB_RETENTION_L2; ++i) {
|
||||||
|
if (!pReader->rsmaDone[i]) {
|
||||||
|
if (!pReader->pVnode->pSma->pRSmaTsdb[i]) {
|
||||||
|
// no valid tsdb
|
||||||
|
pReader->rsmaDone[i] = 1;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (pReader->pTsdbReader == NULL) {
|
||||||
|
code = tsdbSnapReaderOpen(pReader->pVnode->pSma->pRSmaTsdb[i], pReader->sver, pReader->ever,
|
||||||
|
&pReader->pTsdbReader);
|
||||||
|
if (code) goto _err;
|
||||||
|
}
|
||||||
|
|
||||||
|
code = tsdbSnapRead(pReader->pTsdbReader, ppData);
|
||||||
|
if (code) {
|
||||||
|
goto _err;
|
||||||
|
} else {
|
||||||
|
if (*ppData) {
|
||||||
|
goto _exit;
|
||||||
|
} else {
|
||||||
|
pReader->tsdbDone = 1;
|
||||||
|
code = tsdbSnapReaderClose(&pReader->pTsdbReader);
|
||||||
|
if (code) goto _err;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// QTaskInfoFile
|
||||||
|
// TODO ...
|
||||||
|
}
|
||||||
|
#endif
|
||||||
|
|
||||||
*ppData = NULL;
|
*ppData = NULL;
|
||||||
*nData = 0;
|
*nData = 0;
|
||||||
|
|
||||||
|
|
|
@ -532,6 +532,14 @@ typedef struct SCtgOperation {
|
||||||
} \
|
} \
|
||||||
} while (0)
|
} while (0)
|
||||||
|
|
||||||
|
#define CTG_API_JENTER() do { \
|
||||||
|
CTG_API_DEBUG("CTG API enter %s", __FUNCTION__); \
|
||||||
|
CTG_LOCK(CTG_READ, &gCtgMgmt.lock); \
|
||||||
|
if (atomic_load_8((int8_t*)&gCtgMgmt.exit)) { \
|
||||||
|
CTG_ERR_JRET(TSDB_CODE_CTG_OUT_OF_SERVICE); \
|
||||||
|
} \
|
||||||
|
} while (0)
|
||||||
|
|
||||||
|
|
||||||
#define CTG_API_LEAVE_NOLOCK(c) do { \
|
#define CTG_API_LEAVE_NOLOCK(c) do { \
|
||||||
int32_t __code = c; \
|
int32_t __code = c; \
|
||||||
|
|
|
@ -244,10 +244,11 @@ int32_t ctgProcessRspMsg(void* out, int32_t reqType, char* msg, int32_t msgSize,
|
||||||
int32_t ctgHandleMsgCallback(void *param, SDataBuf *pMsg, int32_t rspCode) {
|
int32_t ctgHandleMsgCallback(void *param, SDataBuf *pMsg, int32_t rspCode) {
|
||||||
SCtgTaskCallbackParam* cbParam = (SCtgTaskCallbackParam*)param;
|
SCtgTaskCallbackParam* cbParam = (SCtgTaskCallbackParam*)param;
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
|
SCtgJob* pJob = NULL;
|
||||||
|
|
||||||
CTG_API_ENTER();
|
CTG_API_JENTER();
|
||||||
|
|
||||||
SCtgJob* pJob = taosAcquireRef(gCtgMgmt.jobPool, cbParam->refId);
|
pJob = taosAcquireRef(gCtgMgmt.jobPool, cbParam->refId);
|
||||||
if (NULL == pJob) {
|
if (NULL == pJob) {
|
||||||
qDebug("ctg job refId 0x%" PRIx64 " already dropped", cbParam->refId);
|
qDebug("ctg job refId 0x%" PRIx64 " already dropped", cbParam->refId);
|
||||||
goto _return;
|
goto _return;
|
||||||
|
@ -267,8 +268,6 @@ _return:
|
||||||
taosReleaseRef(gCtgMgmt.jobPool, cbParam->refId);
|
taosReleaseRef(gCtgMgmt.jobPool, cbParam->refId);
|
||||||
}
|
}
|
||||||
|
|
||||||
taosMemoryFree(param);
|
|
||||||
|
|
||||||
CTG_API_LEAVE(code);
|
CTG_API_LEAVE(code);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -293,6 +292,7 @@ int32_t ctgMakeMsgSendInfo(SCtgTask* pTask, int32_t msgType, SMsgSendInfo **pMsg
|
||||||
param->taskId = pTask->taskId;
|
param->taskId = pTask->taskId;
|
||||||
|
|
||||||
msgSendInfo->param = param;
|
msgSendInfo->param = param;
|
||||||
|
msgSendInfo->paramFreeFp = taosMemoryFree;
|
||||||
msgSendInfo->fp = ctgHandleMsgCallback;
|
msgSendInfo->fp = ctgHandleMsgCallback;
|
||||||
|
|
||||||
*pMsgSendInfo = msgSendInfo;
|
*pMsgSendInfo = msgSendInfo;
|
||||||
|
|
|
@ -159,6 +159,13 @@ typedef struct {
|
||||||
int64_t recoverEndVer;
|
int64_t recoverEndVer;
|
||||||
} SStreamTaskInfo;
|
} SStreamTaskInfo;
|
||||||
|
|
||||||
|
typedef struct {
|
||||||
|
char* tablename;
|
||||||
|
char* dbname;
|
||||||
|
int32_t tversion;
|
||||||
|
SSchemaWrapper* sw;
|
||||||
|
} SSchemaInfo;
|
||||||
|
|
||||||
typedef struct SExecTaskInfo {
|
typedef struct SExecTaskInfo {
|
||||||
STaskIdInfo id;
|
STaskIdInfo id;
|
||||||
uint32_t status;
|
uint32_t status;
|
||||||
|
@ -168,18 +175,12 @@ typedef struct SExecTaskInfo {
|
||||||
int32_t code;
|
int32_t code;
|
||||||
|
|
||||||
SStreamTaskInfo streamInfo;
|
SStreamTaskInfo streamInfo;
|
||||||
|
SSchemaInfo schemaInfo;
|
||||||
struct {
|
|
||||||
char* tablename;
|
|
||||||
char* dbname;
|
|
||||||
int32_t tversion;
|
|
||||||
SSchemaWrapper* sw;
|
|
||||||
} schemaVer;
|
|
||||||
|
|
||||||
STableListInfo tableqinfoList; // this is a table list
|
STableListInfo tableqinfoList; // this is a table list
|
||||||
const char* sql; // query sql string
|
const char* sql; // query sql string
|
||||||
jmp_buf env; // jump to this position when error happens.
|
jmp_buf env; // jump to this position when error happens.
|
||||||
EOPTR_EXEC_MODEL execModel; // operator execution model [batch model|stream model]
|
EOPTR_EXEC_MODEL execModel; // operator execution model [batch model|stream model]
|
||||||
|
SSubplan* pSubplan;
|
||||||
struct SOperatorInfo* pRoot;
|
struct SOperatorInfo* pRoot;
|
||||||
} SExecTaskInfo;
|
} SExecTaskInfo;
|
||||||
|
|
||||||
|
|
|
@ -227,14 +227,14 @@ int32_t extractTargetSlotId(const SArray* pColMatchInfo, SExecTaskInfo* pTaskInf
|
||||||
|
|
||||||
for (int32_t i = 0; i < numOfCols; ++i) {
|
for (int32_t i = 0; i < numOfCols; ++i) {
|
||||||
SColMatchInfo* pColMatch = taosArrayGet(pColMatchInfo, i);
|
SColMatchInfo* pColMatch = taosArrayGet(pColMatchInfo, i);
|
||||||
for (int32_t j = 0; j < pTaskInfo->schemaVer.sw->nCols; ++j) {
|
for (int32_t j = 0; j < pTaskInfo->schemaInfo.sw->nCols; ++j) {
|
||||||
if (pColMatch->colId == pTaskInfo->schemaVer.sw->pSchema[j].colId &&
|
if (pColMatch->colId == pTaskInfo->schemaInfo.sw->pSchema[j].colId &&
|
||||||
pColMatch->colId == PRIMARYKEY_TIMESTAMP_COL_ID) {
|
pColMatch->colId == PRIMARYKEY_TIMESTAMP_COL_ID) {
|
||||||
(*pSlotIds)[pColMatch->targetSlotId] = -1;
|
(*pSlotIds)[pColMatch->targetSlotId] = -1;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pColMatch->colId == pTaskInfo->schemaVer.sw->pSchema[j].colId) {
|
if (pColMatch->colId == pTaskInfo->schemaInfo.sw->pSchema[j].colId) {
|
||||||
(*pSlotIds)[pColMatch->targetSlotId] = j;
|
(*pSlotIds)[pColMatch->targetSlotId] = j;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
|
|
@ -67,15 +67,15 @@ static bool needCompress(const SSDataBlock* pData, int32_t numOfCols) {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// clang-format off
|
||||||
// data format:
|
// data format:
|
||||||
// +----------------+--------------+----------+--------------------------------------------+--------------------------------------+-------------+-----------+-------------+-----------+
|
// +----------------+--------------+-----------------+--------------------------------------------+------------------------------------+-------------+-----------+-------------+-----------+
|
||||||
// |SDataCacheEntry | total length | group id | col1_schema | col2_schema | col3_schema ...| column#1 length, column#2
|
// |SDataCacheEntry | total length | group id | col1_schema | col2_schema | col3_schema... | column#1 length, column#2 length...| col1 bitmap | col1 data | col2 bitmap | col2 data | .... | | (4 bytes) |(8 bytes)
|
||||||
// length ... | col1 bitmap | col1 data | col2 bitmap | col2 data | .... | | (4 bytes) |(8 bytes)
|
// | |sizeof(int32) |sizeof(uint64_t) |(sizeof(int16_t)+sizeof(int32_t))*numOfCols | sizeof(int32_t) * numOfCols | actual size | |
|
||||||
// |(sizeof(int16_t)+sizeof(int32_t))*numOfCols | sizeof(int32_t) * numOfCols | actual size | |
|
// +----------------+--------------+-----------------+--------------------------------------------+------------------------------------+-------------+-----------+-------------+-----------+
|
||||||
// actual size | |
|
|
||||||
// +----------------+--------------+----------+--------------------------------------------+--------------------------------------+-------------+-----------+-------------+-----------+
|
|
||||||
// The length of bitmap is decided by number of rows of this data block, and the length of each column data is
|
// The length of bitmap is decided by number of rows of this data block, and the length of each column data is
|
||||||
// recorded in the first segment, next to the struct header
|
// recorded in the first segment, next to the struct header
|
||||||
|
// clang-format on
|
||||||
static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pInput, SDataDispatchBuf* pBuf) {
|
static void toDataCacheEntry(SDataDispatchHandle* pHandle, const SInputData* pInput, SDataDispatchBuf* pBuf) {
|
||||||
int32_t numOfCols = 0;
|
int32_t numOfCols = 0;
|
||||||
SNode* pNode;
|
SNode* pNode;
|
||||||
|
|
|
@ -91,7 +91,6 @@ _return:
|
||||||
tsem_post(&pInserter->ready);
|
tsem_post(&pInserter->ready);
|
||||||
|
|
||||||
taosMemoryFree(pMsg->pData);
|
taosMemoryFree(pMsg->pData);
|
||||||
taosMemoryFree(param);
|
|
||||||
|
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
@ -110,6 +109,7 @@ static int32_t sendSubmitRequest(SDataInserterHandle* pInserter, SSubmitReq* pMs
|
||||||
pParam->pInserter = pInserter;
|
pParam->pInserter = pInserter;
|
||||||
|
|
||||||
pMsgSendInfo->param = pParam;
|
pMsgSendInfo->param = pParam;
|
||||||
|
pMsgSendInfo->paramFreeFp = taosMemoryFree;
|
||||||
pMsgSendInfo->msgInfo.pData = pMsg;
|
pMsgSendInfo->msgInfo.pData = pMsg;
|
||||||
pMsgSendInfo->msgInfo.len = ntohl(pMsg->length);
|
pMsgSendInfo->msgInfo.len = ntohl(pMsg->length);
|
||||||
pMsgSendInfo->msgType = TDMT_VND_SUBMIT;
|
pMsgSendInfo->msgType = TDMT_VND_SUBMIT;
|
||||||
|
|
|
@ -115,27 +115,40 @@ int32_t qSetMultiStreamInput(qTaskInfo_t tinfo, const void* pBlocks, size_t numO
|
||||||
return code;
|
return code;
|
||||||
}
|
}
|
||||||
|
|
||||||
qTaskInfo_t qCreateQueueExecTaskInfo(void* msg, SReadHandle* readers) {
|
qTaskInfo_t qCreateQueueExecTaskInfo(void* msg, SReadHandle* readers, int32_t* numOfCols) {
|
||||||
if (msg == NULL) {
|
if (msg == NULL) {
|
||||||
// TODO create raw scan
|
// TODO create raw scan
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
struct SSubplan* plan = NULL;
|
struct SSubplan* pPlan = NULL;
|
||||||
int32_t code = qStringToSubplan(msg, &plan);
|
int32_t code = qStringToSubplan(msg, &pPlan);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
terrno = code;
|
terrno = code;
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
qTaskInfo_t pTaskInfo = NULL;
|
qTaskInfo_t pTaskInfo = NULL;
|
||||||
code = qCreateExecTask(readers, 0, 0, plan, &pTaskInfo, NULL, NULL, OPTR_EXEC_MODEL_QUEUE);
|
code = qCreateExecTask(readers, 0, 0, pPlan, &pTaskInfo, NULL, NULL, OPTR_EXEC_MODEL_QUEUE);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
// TODO: destroy SSubplan & pTaskInfo
|
nodesDestroyNode((SNode*)pPlan);
|
||||||
|
qDestroyTask(pTaskInfo);
|
||||||
terrno = code;
|
terrno = code;
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// extract the number of output columns
|
||||||
|
SDataBlockDescNode* pDescNode = pPlan->pNode->pOutputDataBlockDesc;
|
||||||
|
*numOfCols = 0;
|
||||||
|
|
||||||
|
SNode* pNode;
|
||||||
|
FOREACH(pNode, pDescNode->pSlots) {
|
||||||
|
SSlotDescNode* pSlotDesc = (SSlotDescNode*)pNode;
|
||||||
|
if (pSlotDesc->output) {
|
||||||
|
++(*numOfCols);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
return pTaskInfo;
|
return pTaskInfo;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -146,17 +159,18 @@ qTaskInfo_t qCreateStreamExecTaskInfo(void* msg, SReadHandle* readers) {
|
||||||
|
|
||||||
/*qDebugL("stream task string %s", (const char*)msg);*/
|
/*qDebugL("stream task string %s", (const char*)msg);*/
|
||||||
|
|
||||||
struct SSubplan* plan = NULL;
|
struct SSubplan* pPlan = NULL;
|
||||||
int32_t code = qStringToSubplan(msg, &plan);
|
int32_t code = qStringToSubplan(msg, &pPlan);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
terrno = code;
|
terrno = code;
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
qTaskInfo_t pTaskInfo = NULL;
|
qTaskInfo_t pTaskInfo = NULL;
|
||||||
code = qCreateExecTask(readers, 0, 0, plan, &pTaskInfo, NULL, NULL, OPTR_EXEC_MODEL_STREAM);
|
code = qCreateExecTask(readers, 0, 0, pPlan, &pTaskInfo, NULL, NULL, OPTR_EXEC_MODEL_STREAM);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
// TODO: destroy SSubplan & pTaskInfo
|
nodesDestroyNode((SNode*)pPlan);
|
||||||
|
qDestroyTask(pTaskInfo);
|
||||||
terrno = code;
|
terrno = code;
|
||||||
return NULL;
|
return NULL;
|
||||||
}
|
}
|
||||||
|
@ -239,19 +253,19 @@ int32_t qGetQueryTableSchemaVersion(qTaskInfo_t tinfo, char* dbName, char* table
|
||||||
ASSERT(tinfo != NULL && dbName != NULL && tableName != NULL);
|
ASSERT(tinfo != NULL && dbName != NULL && tableName != NULL);
|
||||||
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
|
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
|
||||||
|
|
||||||
if (pTaskInfo->schemaVer.sw == NULL) {
|
if (pTaskInfo->schemaInfo.sw == NULL) {
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
|
||||||
*sversion = pTaskInfo->schemaVer.sw->version;
|
*sversion = pTaskInfo->schemaInfo.sw->version;
|
||||||
*tversion = pTaskInfo->schemaVer.tversion;
|
*tversion = pTaskInfo->schemaInfo.tversion;
|
||||||
if (pTaskInfo->schemaVer.dbname) {
|
if (pTaskInfo->schemaInfo.dbname) {
|
||||||
strcpy(dbName, pTaskInfo->schemaVer.dbname);
|
strcpy(dbName, pTaskInfo->schemaInfo.dbname);
|
||||||
} else {
|
} else {
|
||||||
dbName[0] = 0;
|
dbName[0] = 0;
|
||||||
}
|
}
|
||||||
if (pTaskInfo->schemaVer.tablename) {
|
if (pTaskInfo->schemaInfo.tablename) {
|
||||||
strcpy(tableName, pTaskInfo->schemaVer.tablename);
|
strcpy(tableName, pTaskInfo->schemaInfo.tablename);
|
||||||
} else {
|
} else {
|
||||||
tableName[0] = 0;
|
tableName[0] = 0;
|
||||||
}
|
}
|
||||||
|
|
|
@ -39,6 +39,7 @@ int32_t qCreateExecTask(SReadHandle* readHandle, int32_t vgId, uint64_t taskId,
|
||||||
|
|
||||||
taosThreadOnce(&initPoolOnce, initRefPool);
|
taosThreadOnce(&initPoolOnce, initRefPool);
|
||||||
atexit(cleanupRefPool);
|
atexit(cleanupRefPool);
|
||||||
|
|
||||||
int32_t code = createExecTaskInfoImpl(pSubplan, pTask, readHandle, taskId, sql, model);
|
int32_t code = createExecTaskInfoImpl(pSubplan, pTask, readHandle, taskId, sql, model);
|
||||||
if (code != TSDB_CODE_SUCCESS) {
|
if (code != TSDB_CODE_SUCCESS) {
|
||||||
goto _error;
|
goto _error;
|
||||||
|
|
|
@ -1337,7 +1337,9 @@ void doFilter(const SNode* pFilterNode, SSDataBlock* pBlock) {
|
||||||
if (pFilterNode == NULL) {
|
if (pFilterNode == NULL) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
if (pBlock->info.rows == 0) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
SFilterInfo* filter = NULL;
|
SFilterInfo* filter = NULL;
|
||||||
|
|
||||||
// todo move to the initialization function
|
// todo move to the initialization function
|
||||||
|
@ -1994,16 +1996,9 @@ int32_t loadRemoteDataCallback(void* param, SDataBuf* pMsg, int32_t code) {
|
||||||
tsem_post(&pExchangeInfo->ready);
|
tsem_post(&pExchangeInfo->ready);
|
||||||
taosReleaseRef(exchangeObjRefPool, pWrapper->exchangeId);
|
taosReleaseRef(exchangeObjRefPool, pWrapper->exchangeId);
|
||||||
|
|
||||||
taosMemoryFree(pWrapper);
|
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
|
||||||
static void destroySendMsgInfo(SMsgSendInfo* pMsgBody) {
|
|
||||||
assert(pMsgBody != NULL);
|
|
||||||
taosMemoryFreeClear(pMsgBody->msgInfo.pData);
|
|
||||||
taosMemoryFreeClear(pMsgBody);
|
|
||||||
}
|
|
||||||
|
|
||||||
void qProcessRspMsg(void* parent, SRpcMsg* pMsg, SEpSet* pEpSet) {
|
void qProcessRspMsg(void* parent, SRpcMsg* pMsg, SEpSet* pEpSet) {
|
||||||
SMsgSendInfo* pSendInfo = (SMsgSendInfo*)pMsg->info.ahandle;
|
SMsgSendInfo* pSendInfo = (SMsgSendInfo*)pMsg->info.ahandle;
|
||||||
assert(pMsg->info.ahandle != NULL);
|
assert(pMsg->info.ahandle != NULL);
|
||||||
|
@ -2063,6 +2058,7 @@ static int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInf
|
||||||
pWrapper->sourceIndex = sourceIndex;
|
pWrapper->sourceIndex = sourceIndex;
|
||||||
|
|
||||||
pMsgSendInfo->param = pWrapper;
|
pMsgSendInfo->param = pWrapper;
|
||||||
|
pMsgSendInfo->paramFreeFp = taosMemoryFree;
|
||||||
pMsgSendInfo->msgInfo.pData = pMsg;
|
pMsgSendInfo->msgInfo.pData = pMsg;
|
||||||
pMsgSendInfo->msgInfo.len = sizeof(SResFetchReq);
|
pMsgSendInfo->msgInfo.len = sizeof(SResFetchReq);
|
||||||
pMsgSendInfo->msgType = pSource->fetchMsgType;
|
pMsgSendInfo->msgType = pSource->fetchMsgType;
|
||||||
|
@ -4127,7 +4123,7 @@ static SExecTaskInfo* createExecTaskInfo(uint64_t queryId, uint64_t taskId, EOPT
|
||||||
SExecTaskInfo* pTaskInfo = taosMemoryCalloc(1, sizeof(SExecTaskInfo));
|
SExecTaskInfo* pTaskInfo = taosMemoryCalloc(1, sizeof(SExecTaskInfo));
|
||||||
setTaskStatus(pTaskInfo, TASK_NOT_COMPLETED);
|
setTaskStatus(pTaskInfo, TASK_NOT_COMPLETED);
|
||||||
|
|
||||||
pTaskInfo->schemaVer.dbname = strdup(dbFName);
|
pTaskInfo->schemaInfo.dbname = strdup(dbFName);
|
||||||
pTaskInfo->cost.created = taosGetTimestampMs();
|
pTaskInfo->cost.created = taosGetTimestampMs();
|
||||||
pTaskInfo->id.queryId = queryId;
|
pTaskInfo->id.queryId = queryId;
|
||||||
pTaskInfo->execModel = model;
|
pTaskInfo->execModel = model;
|
||||||
|
@ -4153,35 +4149,35 @@ int32_t extractTableSchemaInfo(SReadHandle* pHandle, uint64_t uid, SExecTaskInfo
|
||||||
return terrno;
|
return terrno;
|
||||||
}
|
}
|
||||||
|
|
||||||
pTaskInfo->schemaVer.tablename = strdup(mr.me.name);
|
pTaskInfo->schemaInfo.tablename = strdup(mr.me.name);
|
||||||
|
|
||||||
if (mr.me.type == TSDB_SUPER_TABLE) {
|
if (mr.me.type == TSDB_SUPER_TABLE) {
|
||||||
pTaskInfo->schemaVer.sw = tCloneSSchemaWrapper(&mr.me.stbEntry.schemaRow);
|
pTaskInfo->schemaInfo.sw = tCloneSSchemaWrapper(&mr.me.stbEntry.schemaRow);
|
||||||
pTaskInfo->schemaVer.tversion = mr.me.stbEntry.schemaTag.version;
|
pTaskInfo->schemaInfo.tversion = mr.me.stbEntry.schemaTag.version;
|
||||||
} else if (mr.me.type == TSDB_CHILD_TABLE) {
|
} else if (mr.me.type == TSDB_CHILD_TABLE) {
|
||||||
tDecoderClear(&mr.coder);
|
tDecoderClear(&mr.coder);
|
||||||
|
|
||||||
tb_uid_t suid = mr.me.ctbEntry.suid;
|
tb_uid_t suid = mr.me.ctbEntry.suid;
|
||||||
metaGetTableEntryByUid(&mr, suid);
|
metaGetTableEntryByUid(&mr, suid);
|
||||||
pTaskInfo->schemaVer.sw = tCloneSSchemaWrapper(&mr.me.stbEntry.schemaRow);
|
pTaskInfo->schemaInfo.sw = tCloneSSchemaWrapper(&mr.me.stbEntry.schemaRow);
|
||||||
pTaskInfo->schemaVer.tversion = mr.me.stbEntry.schemaTag.version;
|
pTaskInfo->schemaInfo.tversion = mr.me.stbEntry.schemaTag.version;
|
||||||
} else {
|
} else {
|
||||||
pTaskInfo->schemaVer.sw = tCloneSSchemaWrapper(&mr.me.ntbEntry.schemaRow);
|
pTaskInfo->schemaInfo.sw = tCloneSSchemaWrapper(&mr.me.ntbEntry.schemaRow);
|
||||||
}
|
}
|
||||||
|
|
||||||
metaReaderClear(&mr);
|
metaReaderClear(&mr);
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
|
||||||
static void cleanupTableSchemaInfo(SExecTaskInfo* pTaskInfo) {
|
static void cleanupTableSchemaInfo(SSchemaInfo* pSchemaInfo) {
|
||||||
taosMemoryFreeClear(pTaskInfo->schemaVer.dbname);
|
taosMemoryFreeClear(pSchemaInfo->dbname);
|
||||||
if (pTaskInfo->schemaVer.sw == NULL) {
|
if (pSchemaInfo->sw == NULL) {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
|
|
||||||
taosMemoryFree(pTaskInfo->schemaVer.sw->pSchema);
|
taosMemoryFree(pSchemaInfo->tablename);
|
||||||
taosMemoryFree(pTaskInfo->schemaVer.sw);
|
taosMemoryFree(pSchemaInfo->sw->pSchema);
|
||||||
taosMemoryFree(pTaskInfo->schemaVer.tablename);
|
taosMemoryFree(pSchemaInfo->sw);
|
||||||
}
|
}
|
||||||
|
|
||||||
static int32_t sortTableGroup(STableListInfo* pTableListInfo, int32_t groupNum) {
|
static int32_t sortTableGroup(STableListInfo* pTableListInfo, int32_t groupNum) {
|
||||||
|
@ -4424,7 +4420,7 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo
|
||||||
.calTrigger = pTableScanNode->triggerType,
|
.calTrigger = pTableScanNode->triggerType,
|
||||||
.maxTs = INT64_MIN,
|
.maxTs = INT64_MIN,
|
||||||
};
|
};
|
||||||
if (pHandle) {
|
if (pHandle->vnode) {
|
||||||
int32_t code = createScanTableListInfo(&pTableScanNode->scan, pTableScanNode->pGroupTags,
|
int32_t code = createScanTableListInfo(&pTableScanNode->scan, pTableScanNode->pGroupTags,
|
||||||
pTableScanNode->groupSort, pHandle, pTableListInfo, pTagCond, pTagIndexCond, GET_TASKID(pTaskInfo));
|
pTableScanNode->groupSort, pHandle, pTableListInfo, pTagCond, pTagIndexCond, GET_TASKID(pTaskInfo));
|
||||||
if (code) {
|
if (code) {
|
||||||
|
@ -4590,7 +4586,7 @@ SOperatorInfo* createOperatorTree(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo
|
||||||
int32_t children = 0;
|
int32_t children = 0;
|
||||||
pOptr = createStreamFinalIntervalOperatorInfo(ops[0], pPhyNode, pTaskInfo, children);
|
pOptr = createStreamFinalIntervalOperatorInfo(ops[0], pPhyNode, pTaskInfo, children);
|
||||||
} else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL == type) {
|
} else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_FINAL_INTERVAL == type) {
|
||||||
int32_t children = 1;
|
int32_t children = pHandle->numOfVgroups;
|
||||||
pOptr = createStreamFinalIntervalOperatorInfo(ops[0], pPhyNode, pTaskInfo, children);
|
pOptr = createStreamFinalIntervalOperatorInfo(ops[0], pPhyNode, pTaskInfo, children);
|
||||||
} else if (QUERY_NODE_PHYSICAL_PLAN_SORT == type) {
|
} else if (QUERY_NODE_PHYSICAL_PLAN_SORT == type) {
|
||||||
pOptr = createSortOperatorInfo(ops[0], (SSortPhysiNode*)pPhyNode, pTaskInfo);
|
pOptr = createSortOperatorInfo(ops[0], (SSortPhysiNode*)pPhyNode, pTaskInfo);
|
||||||
|
@ -4939,6 +4935,7 @@ int32_t createExecTaskInfoImpl(SSubplan* pPlan, SExecTaskInfo** pTaskInfo, SRead
|
||||||
}
|
}
|
||||||
|
|
||||||
(*pTaskInfo)->sql = sql;
|
(*pTaskInfo)->sql = sql;
|
||||||
|
(*pTaskInfo)->pSubplan = pPlan;
|
||||||
(*pTaskInfo)->pRoot = createOperatorTree(pPlan->pNode, *pTaskInfo, pHandle, &(*pTaskInfo)->tableqinfoList, pPlan->pTagCond, pPlan->pTagIndexCond, pPlan->user);
|
(*pTaskInfo)->pRoot = createOperatorTree(pPlan->pNode, *pTaskInfo, pHandle, &(*pTaskInfo)->tableqinfoList, pPlan->pTagCond, pPlan->pTagIndexCond, pPlan->user);
|
||||||
|
|
||||||
if (NULL == (*pTaskInfo)->pRoot) {
|
if (NULL == (*pTaskInfo)->pRoot) {
|
||||||
|
@ -4977,7 +4974,9 @@ void doDestroyTask(SExecTaskInfo* pTaskInfo) {
|
||||||
|
|
||||||
doDestroyTableList(&pTaskInfo->tableqinfoList);
|
doDestroyTableList(&pTaskInfo->tableqinfoList);
|
||||||
destroyOperatorInfo(pTaskInfo->pRoot);
|
destroyOperatorInfo(pTaskInfo->pRoot);
|
||||||
cleanupTableSchemaInfo(pTaskInfo);
|
cleanupTableSchemaInfo(&pTaskInfo->schemaInfo);
|
||||||
|
|
||||||
|
nodesDestroyNode((SNode*)pTaskInfo->pSubplan);
|
||||||
|
|
||||||
taosMemoryFreeClear(pTaskInfo->sql);
|
taosMemoryFreeClear(pTaskInfo->sql);
|
||||||
taosMemoryFreeClear(pTaskInfo->id.str);
|
taosMemoryFreeClear(pTaskInfo->id.str);
|
||||||
|
|
|
@ -1196,7 +1196,6 @@ static int32_t setBlockIntoRes(SStreamScanInfo* pInfo, const SSDataBlock* pBlock
|
||||||
if (pResCol->info.colId == pColMatchInfo->colId) {
|
if (pResCol->info.colId == pColMatchInfo->colId) {
|
||||||
SColumnInfoData* pDst = taosArrayGet(pInfo->pRes->pDataBlock, pColMatchInfo->targetSlotId);
|
SColumnInfoData* pDst = taosArrayGet(pInfo->pRes->pDataBlock, pColMatchInfo->targetSlotId);
|
||||||
colDataAssign(pDst, pResCol, pBlock->info.rows, &pInfo->pRes->info);
|
colDataAssign(pDst, pResCol, pBlock->info.rows, &pInfo->pRes->info);
|
||||||
// taosArraySet(pInfo->pRes->pDataBlock, pColMatchInfo->targetSlotId, pResCol);
|
|
||||||
colExists = true;
|
colExists = true;
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
@ -1460,6 +1459,7 @@ static SSDataBlock* doStreamScan(SOperatorInfo* pOperator) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
qDebug("scan rows: %d", pBlockInfo->rows);
|
qDebug("scan rows: %d", pBlockInfo->rows);
|
||||||
return (pBlockInfo->rows == 0) ? NULL : pInfo->pRes;
|
return (pBlockInfo->rows == 0) ? NULL : pInfo->pRes;
|
||||||
|
|
||||||
|
@ -1533,7 +1533,6 @@ SOperatorInfo* createStreamScanOperatorInfo(SReadHandle* pHandle, STableScanPhys
|
||||||
}
|
}
|
||||||
|
|
||||||
SScanPhysiNode* pScanPhyNode = &pTableScanNode->scan;
|
SScanPhysiNode* pScanPhyNode = &pTableScanNode->scan;
|
||||||
|
|
||||||
SDataBlockDescNode* pDescNode = pScanPhyNode->node.pOutputDataBlockDesc;
|
SDataBlockDescNode* pDescNode = pScanPhyNode->node.pOutputDataBlockDesc;
|
||||||
|
|
||||||
pInfo->pTagCond = pTagCond;
|
pInfo->pTagCond = pTagCond;
|
||||||
|
@ -1564,7 +1563,7 @@ SOperatorInfo* createStreamScanOperatorInfo(SReadHandle* pHandle, STableScanPhys
|
||||||
goto _error;
|
goto _error;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pHandle) {
|
if (pHandle->vnode) {
|
||||||
SOperatorInfo* pTableScanOp = createTableScanOperatorInfo(pTableScanNode, pHandle, pTaskInfo);
|
SOperatorInfo* pTableScanOp = createTableScanOperatorInfo(pTableScanNode, pHandle, pTaskInfo);
|
||||||
STableScanInfo* pTSInfo = (STableScanInfo*)pTableScanOp->info;
|
STableScanInfo* pTSInfo = (STableScanInfo*)pTableScanOp->info;
|
||||||
if (pHandle->version > 0) {
|
if (pHandle->version > 0) {
|
||||||
|
|
|
@ -1543,6 +1543,7 @@ void destroyStreamFinalIntervalOperatorInfo(void* param, int32_t numOfOutput) {
|
||||||
taosArrayDestroy(pInfo->pPullWins);
|
taosArrayDestroy(pInfo->pPullWins);
|
||||||
blockDataDestroy(pInfo->pPullDataRes);
|
blockDataDestroy(pInfo->pPullDataRes);
|
||||||
taosArrayDestroy(pInfo->pRecycledPages);
|
taosArrayDestroy(pInfo->pRecycledPages);
|
||||||
|
blockDataDestroy(pInfo->pUpdateRes);
|
||||||
|
|
||||||
if (pInfo->pChildren) {
|
if (pInfo->pChildren) {
|
||||||
int32_t size = taosArrayGetSize(pInfo->pChildren);
|
int32_t size = taosArrayGetSize(pInfo->pChildren);
|
||||||
|
|
|
@ -1646,9 +1646,9 @@ static int32_t translateCast(SFunctionNode* pFunc, char* pErrBuf, int32_t len) {
|
||||||
if (IS_VAR_DATA_TYPE(para2Type)) {
|
if (IS_VAR_DATA_TYPE(para2Type)) {
|
||||||
para2Bytes -= VARSTR_HEADER_SIZE;
|
para2Bytes -= VARSTR_HEADER_SIZE;
|
||||||
}
|
}
|
||||||
if (para2Bytes <= 0 || para2Bytes > 1000) { // cast dst var type length limits to 1000
|
if (para2Bytes <= 0 || para2Bytes > 4096) { // cast dst var type length limits to 4096 bytes
|
||||||
return buildFuncErrMsg(pErrBuf, len, TSDB_CODE_FUNC_FUNTION_ERROR,
|
return buildFuncErrMsg(pErrBuf, len, TSDB_CODE_FUNC_FUNTION_ERROR,
|
||||||
"CAST function converted length should be in range [0, 1000]");
|
"CAST function converted length should be in range [0, 4096] bytes");
|
||||||
}
|
}
|
||||||
|
|
||||||
// add database precision as param
|
// add database precision as param
|
||||||
|
|
|
@ -5098,12 +5098,7 @@ bool modeFunctionSetup(SqlFunctionCtx* pCtx, SResultRowEntryInfo* pResInfo) {
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
static void doModeAdd(SModeInfo* pInfo, char* data, bool isNull) {
|
static void doModeAdd(SModeInfo* pInfo, char* data) {
|
||||||
// ignore null elements
|
|
||||||
if (isNull) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
int32_t hashKeyBytes = IS_VAR_DATA_TYPE(pInfo->colType) ? varDataTLen(data) : pInfo->colBytes;
|
int32_t hashKeyBytes = IS_VAR_DATA_TYPE(pInfo->colType) ? varDataTLen(data) : pInfo->colBytes;
|
||||||
SModeItem** pHashItem = taosHashGet(pInfo->pHash, data, hashKeyBytes);
|
SModeItem** pHashItem = taosHashGet(pInfo->pHash, data, hashKeyBytes);
|
||||||
if (pHashItem == NULL) {
|
if (pHashItem == NULL) {
|
||||||
|
@ -5128,10 +5123,16 @@ int32_t modeFunction(SqlFunctionCtx* pCtx) {
|
||||||
SColumnInfoData* pInputCol = pInput->pData[0];
|
SColumnInfoData* pInputCol = pInput->pData[0];
|
||||||
SColumnInfoData* pOutput = (SColumnInfoData*)pCtx->pOutput;
|
SColumnInfoData* pOutput = (SColumnInfoData*)pCtx->pOutput;
|
||||||
|
|
||||||
|
int32_t numOfElems = 0;
|
||||||
int32_t startOffset = pCtx->offset;
|
int32_t startOffset = pCtx->offset;
|
||||||
for (int32_t i = pInput->startRowIndex; i < pInput->numOfRows + pInput->startRowIndex; ++i) {
|
for (int32_t i = pInput->startRowIndex; i < pInput->numOfRows + pInput->startRowIndex; ++i) {
|
||||||
char* data = colDataGetData(pInputCol, i);
|
char* data = colDataGetData(pInputCol, i);
|
||||||
doModeAdd(pInfo, data, colDataIsNull_s(pInputCol, i));
|
if (colDataIsNull_s(pInputCol, i)) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
|
||||||
|
numOfElems++;
|
||||||
|
doModeAdd(pInfo, data);
|
||||||
|
|
||||||
if (sizeof(SModeInfo) + pInfo->numOfPoints * (sizeof(SModeItem) + pInfo->colBytes) >= MODE_MAX_RESULT_SIZE) {
|
if (sizeof(SModeInfo) + pInfo->numOfPoints * (sizeof(SModeItem) + pInfo->colBytes) >= MODE_MAX_RESULT_SIZE) {
|
||||||
taosHashCleanup(pInfo->pHash);
|
taosHashCleanup(pInfo->pHash);
|
||||||
|
@ -5139,7 +5140,7 @@ int32_t modeFunction(SqlFunctionCtx* pCtx) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
SET_VAL(pResInfo, 1, 1);
|
SET_VAL(pResInfo, numOfElems, 1);
|
||||||
|
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
@ -5660,8 +5661,13 @@ int32_t derivativeFunction(SqlFunctionCtx* pCtx) {
|
||||||
} else {
|
} else {
|
||||||
double r = ((v - pDerivInfo->prevValue) * pDerivInfo->tsWindow) / (tsList[i] - pDerivInfo->prevTs);
|
double r = ((v - pDerivInfo->prevValue) * pDerivInfo->tsWindow) / (tsList[i] - pDerivInfo->prevTs);
|
||||||
if (pDerivInfo->ignoreNegative && r < 0) {
|
if (pDerivInfo->ignoreNegative && r < 0) {
|
||||||
|
} else {
|
||||||
|
if (isinf(r) || isnan(r)) {
|
||||||
|
colDataAppendNULL(pOutput, pos);
|
||||||
} else {
|
} else {
|
||||||
colDataAppend(pOutput, pos, (const char*)&r, false);
|
colDataAppend(pOutput, pos, (const char*)&r, false);
|
||||||
|
}
|
||||||
|
|
||||||
if (pTsOutput != NULL) {
|
if (pTsOutput != NULL) {
|
||||||
colDataAppendInt64(pTsOutput, pos, &tsList[i]);
|
colDataAppendInt64(pTsOutput, pos, &tsList[i]);
|
||||||
}
|
}
|
||||||
|
@ -5687,8 +5693,13 @@ int32_t derivativeFunction(SqlFunctionCtx* pCtx) {
|
||||||
} else {
|
} else {
|
||||||
double r = ((pDerivInfo->prevValue - v) * pDerivInfo->tsWindow) / (pDerivInfo->prevTs - tsList[i]);
|
double r = ((pDerivInfo->prevValue - v) * pDerivInfo->tsWindow) / (pDerivInfo->prevTs - tsList[i]);
|
||||||
if (pDerivInfo->ignoreNegative && r < 0) {
|
if (pDerivInfo->ignoreNegative && r < 0) {
|
||||||
|
} else {
|
||||||
|
if (isinf(r) || isnan(r)) {
|
||||||
|
colDataAppendNULL(pOutput, pos);
|
||||||
} else {
|
} else {
|
||||||
colDataAppend(pOutput, pos, (const char*)&r, false);
|
colDataAppend(pOutput, pos, (const char*)&r, false);
|
||||||
|
}
|
||||||
|
|
||||||
if (pTsOutput != NULL) {
|
if (pTsOutput != NULL) {
|
||||||
colDataAppendInt64(pTsOutput, pos, &pDerivInfo->prevTs);
|
colDataAppendInt64(pTsOutput, pos, &pDerivInfo->prevTs);
|
||||||
}
|
}
|
||||||
|
|
|
@ -1923,15 +1923,18 @@ int32_t nodesPartitionCond(SNode** pCondition, SNode** pPrimaryKeyCond, SNode**
|
||||||
return partitionLogicCond(pCondition, pPrimaryKeyCond, pTagIndexCond, pTagCond, pOtherCond);
|
return partitionLogicCond(pCondition, pPrimaryKeyCond, pTagIndexCond, pTagCond, pOtherCond);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
bool needOutput = false;
|
||||||
switch (classifyCondition(*pCondition)) {
|
switch (classifyCondition(*pCondition)) {
|
||||||
case COND_TYPE_PRIMARY_KEY:
|
case COND_TYPE_PRIMARY_KEY:
|
||||||
if (NULL != pPrimaryKeyCond) {
|
if (NULL != pPrimaryKeyCond) {
|
||||||
*pPrimaryKeyCond = *pCondition;
|
*pPrimaryKeyCond = *pCondition;
|
||||||
|
needOutput = true;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case COND_TYPE_TAG_INDEX:
|
case COND_TYPE_TAG_INDEX:
|
||||||
if (NULL != pTagIndexCond) {
|
if (NULL != pTagIndexCond) {
|
||||||
*pTagIndexCond = *pCondition;
|
*pTagIndexCond = *pCondition;
|
||||||
|
needOutput = true;
|
||||||
}
|
}
|
||||||
if (NULL != pTagCond) {
|
if (NULL != pTagCond) {
|
||||||
SNode* pTempCond = *pCondition;
|
SNode* pTempCond = *pCondition;
|
||||||
|
@ -1942,21 +1945,26 @@ int32_t nodesPartitionCond(SNode** pCondition, SNode** pPrimaryKeyCond, SNode**
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
*pTagCond = pTempCond;
|
*pTagCond = pTempCond;
|
||||||
|
needOutput = true;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case COND_TYPE_TAG:
|
case COND_TYPE_TAG:
|
||||||
if (NULL != pTagCond) {
|
if (NULL != pTagCond) {
|
||||||
*pTagCond = *pCondition;
|
*pTagCond = *pCondition;
|
||||||
|
needOutput = true;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
case COND_TYPE_NORMAL:
|
case COND_TYPE_NORMAL:
|
||||||
default:
|
default:
|
||||||
if (NULL != pOtherCond) {
|
if (NULL != pOtherCond) {
|
||||||
*pOtherCond = *pCondition;
|
*pOtherCond = *pCondition;
|
||||||
|
needOutput = true;
|
||||||
}
|
}
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
|
if (needOutput) {
|
||||||
*pCondition = NULL;
|
*pCondition = NULL;
|
||||||
|
}
|
||||||
|
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
|
|
@ -2161,7 +2161,7 @@ static void smlDestroyTableHandle(void* pHandle) {
|
||||||
tdDestroySVCreateTbReq(&handle->createTblReq);
|
tdDestroySVCreateTbReq(&handle->createTblReq);
|
||||||
}
|
}
|
||||||
|
|
||||||
static int32_t smlBoundColumnData(SArray* cols, SParsedDataColInfo* pColList, SSchema* pSchema) {
|
static int32_t smlBoundColumnData(SArray* cols, SParsedDataColInfo* pColList, SSchema* pSchema, bool isTag) {
|
||||||
col_id_t nCols = pColList->numOfCols;
|
col_id_t nCols = pColList->numOfCols;
|
||||||
|
|
||||||
pColList->numOfBound = 0;
|
pColList->numOfBound = 0;
|
||||||
|
@ -2177,7 +2177,8 @@ static int32_t smlBoundColumnData(SArray* cols, SParsedDataColInfo* pColList, SS
|
||||||
SSmlKv* kv = taosArrayGetP(cols, i);
|
SSmlKv* kv = taosArrayGetP(cols, i);
|
||||||
SToken sToken = {.n = kv->keyLen, .z = (char*)kv->key};
|
SToken sToken = {.n = kv->keyLen, .z = (char*)kv->key};
|
||||||
col_id_t t = lastColIdx + 1;
|
col_id_t t = lastColIdx + 1;
|
||||||
col_id_t index = findCol(&sToken, t, nCols, pSchema);
|
col_id_t index = ((t == 0 && !isTag) ? 0 : findCol(&sToken, t, nCols, pSchema));
|
||||||
|
uDebug("SML, index:%d, t:%d, ncols:%d, kv->name:%s", index, t, nCols, kv->key);
|
||||||
if (index < 0 && t > 0) {
|
if (index < 0 && t > 0) {
|
||||||
index = findCol(&sToken, 0, t, pSchema);
|
index = findCol(&sToken, 0, t, pSchema);
|
||||||
isOrdered = false;
|
isOrdered = false;
|
||||||
|
@ -2312,7 +2313,7 @@ int32_t smlBindData(void* handle, SArray* tags, SArray* colsSchema, SArray* cols
|
||||||
smlDestroyTableHandle(&smlHandle->tableExecHandle); // free for each table
|
smlDestroyTableHandle(&smlHandle->tableExecHandle); // free for each table
|
||||||
SSchema* pTagsSchema = getTableTagSchema(pTableMeta);
|
SSchema* pTagsSchema = getTableTagSchema(pTableMeta);
|
||||||
setBoundColumnInfo(&smlHandle->tableExecHandle.tags, pTagsSchema, getNumOfTags(pTableMeta));
|
setBoundColumnInfo(&smlHandle->tableExecHandle.tags, pTagsSchema, getNumOfTags(pTableMeta));
|
||||||
int ret = smlBoundColumnData(tags, &smlHandle->tableExecHandle.tags, pTagsSchema);
|
int ret = smlBoundColumnData(tags, &smlHandle->tableExecHandle.tags, pTagsSchema, true);
|
||||||
if (ret != TSDB_CODE_SUCCESS) {
|
if (ret != TSDB_CODE_SUCCESS) {
|
||||||
buildInvalidOperationMsg(&pBuf, "bound tags error");
|
buildInvalidOperationMsg(&pBuf, "bound tags error");
|
||||||
return ret;
|
return ret;
|
||||||
|
@ -2343,7 +2344,7 @@ int32_t smlBindData(void* handle, SArray* tags, SArray* colsSchema, SArray* cols
|
||||||
|
|
||||||
SSchema* pSchema = getTableColumnSchema(pTableMeta);
|
SSchema* pSchema = getTableColumnSchema(pTableMeta);
|
||||||
|
|
||||||
ret = smlBoundColumnData(colsSchema, &pDataBlock->boundColumnInfo, pSchema);
|
ret = smlBoundColumnData(colsSchema, &pDataBlock->boundColumnInfo, pSchema, false);
|
||||||
if (ret != TSDB_CODE_SUCCESS) {
|
if (ret != TSDB_CODE_SUCCESS) {
|
||||||
buildInvalidOperationMsg(&pBuf, "bound cols error");
|
buildInvalidOperationMsg(&pBuf, "bound cols error");
|
||||||
return ret;
|
return ret;
|
||||||
|
@ -2384,7 +2385,7 @@ int32_t smlBindData(void* handle, SArray* tags, SArray* colsSchema, SArray* cols
|
||||||
if (format) {
|
if (format) {
|
||||||
if (j < rowDataSize) {
|
if (j < rowDataSize) {
|
||||||
kv = taosArrayGetP(rowData, j);
|
kv = taosArrayGetP(rowData, j);
|
||||||
if (rowDataSize != spd->numOfBound &&
|
if (rowDataSize != spd->numOfBound && j != 0 &&
|
||||||
(kv->keyLen != strlen(pColSchema->name) || strncmp(kv->key, pColSchema->name, kv->keyLen) != 0)) {
|
(kv->keyLen != strlen(pColSchema->name) || strncmp(kv->key, pColSchema->name, kv->keyLen) != 0)) {
|
||||||
kv = NULL;
|
kv = NULL;
|
||||||
} else {
|
} else {
|
||||||
|
@ -2401,7 +2402,9 @@ int32_t smlBindData(void* handle, SArray* tags, SArray* colsSchema, SArray* cols
|
||||||
} else {
|
} else {
|
||||||
int32_t colLen = kv->length;
|
int32_t colLen = kv->length;
|
||||||
if (pColSchema->type == TSDB_DATA_TYPE_TIMESTAMP) {
|
if (pColSchema->type == TSDB_DATA_TYPE_TIMESTAMP) {
|
||||||
|
// uError("SML:data before:%ld, precision:%d", kv->i, pTableMeta->tableInfo.precision);
|
||||||
kv->i = convertTimePrecision(kv->i, TSDB_TIME_PRECISION_NANO, pTableMeta->tableInfo.precision);
|
kv->i = convertTimePrecision(kv->i, TSDB_TIME_PRECISION_NANO, pTableMeta->tableInfo.precision);
|
||||||
|
// uError("SML:data after:%ld, precision:%d", kv->i, pTableMeta->tableInfo.precision);
|
||||||
}
|
}
|
||||||
|
|
||||||
if (IS_VAR_DATA_TYPE(kv->type)) {
|
if (IS_VAR_DATA_TYPE(kv->type)) {
|
||||||
|
|
|
@ -138,6 +138,16 @@ int32_t taosAsyncExec(__async_exec_fn_t execFn, void* execParam, int32_t* code)
|
||||||
return 0;
|
return 0;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void destroySendMsgInfo(SMsgSendInfo* pMsgBody) {
|
||||||
|
assert(pMsgBody != NULL);
|
||||||
|
taosMemoryFreeClear(pMsgBody->target.dbFName);
|
||||||
|
taosMemoryFreeClear(pMsgBody->msgInfo.pData);
|
||||||
|
if (pMsgBody->paramFreeFp) {
|
||||||
|
(*pMsgBody->paramFreeFp)(pMsgBody->param);
|
||||||
|
}
|
||||||
|
taosMemoryFreeClear(pMsgBody);
|
||||||
|
}
|
||||||
|
|
||||||
int32_t asyncSendMsgToServerExt(void* pTransporter, SEpSet* epSet, int64_t* pTransporterId, const SMsgSendInfo* pInfo,
|
int32_t asyncSendMsgToServerExt(void* pTransporter, SEpSet* epSet, int64_t* pTransporterId, const SMsgSendInfo* pInfo,
|
||||||
bool persistHandle, void* rpcCtx) {
|
bool persistHandle, void* rpcCtx) {
|
||||||
char* pMsg = rpcMallocCont(pInfo->msgInfo.len);
|
char* pMsg = rpcMallocCont(pInfo->msgInfo.len);
|
||||||
|
|
|
@ -150,7 +150,6 @@ typedef struct SQWTaskCtx {
|
||||||
|
|
||||||
void *taskHandle;
|
void *taskHandle;
|
||||||
void *sinkHandle;
|
void *sinkHandle;
|
||||||
SSubplan *plan;
|
|
||||||
STbVerInfo tbInfo;
|
STbVerInfo tbInfo;
|
||||||
} SQWTaskCtx;
|
} SQWTaskCtx;
|
||||||
|
|
||||||
|
|
|
@ -306,11 +306,6 @@ void qwFreeTaskCtx(SQWTaskCtx *ctx) {
|
||||||
dsDestroyDataSinker(ctx->sinkHandle);
|
dsDestroyDataSinker(ctx->sinkHandle);
|
||||||
ctx->sinkHandle = NULL;
|
ctx->sinkHandle = NULL;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (ctx->plan) {
|
|
||||||
nodesDestroyNode((SNode*)ctx->plan);
|
|
||||||
ctx->plan = NULL;
|
|
||||||
}
|
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t qwDropTaskCtx(QW_FPARAMS_DEF) {
|
int32_t qwDropTaskCtx(QW_FPARAMS_DEF) {
|
||||||
|
@ -327,7 +322,6 @@ int32_t qwDropTaskCtx(QW_FPARAMS_DEF) {
|
||||||
|
|
||||||
atomic_store_ptr(&ctx->taskHandle, NULL);
|
atomic_store_ptr(&ctx->taskHandle, NULL);
|
||||||
atomic_store_ptr(&ctx->sinkHandle, NULL);
|
atomic_store_ptr(&ctx->sinkHandle, NULL);
|
||||||
atomic_store_ptr(&ctx->plan, NULL);
|
|
||||||
|
|
||||||
QW_SET_EVENT_PROCESSED(ctx, QW_EVENT_DROP);
|
QW_SET_EVENT_PROCESSED(ctx, QW_EVENT_DROP);
|
||||||
|
|
||||||
|
|
|
@ -522,8 +522,6 @@ int32_t qwProcessQuery(QW_FPARAMS_DEF, SQWMsg *qwMsg, const char* sql) {
|
||||||
QW_ERR_JRET(code);
|
QW_ERR_JRET(code);
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx->plan = plan;
|
|
||||||
|
|
||||||
code = qCreateExecTask(qwMsg->node, mgmt->nodeId, tId, plan, &pTaskInfo, &sinkHandle, sql, OPTR_EXEC_MODEL_BATCH);
|
code = qCreateExecTask(qwMsg->node, mgmt->nodeId, tId, plan, &pTaskInfo, &sinkHandle, sql, OPTR_EXEC_MODEL_BATCH);
|
||||||
if (code) {
|
if (code) {
|
||||||
QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code));
|
QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code));
|
||||||
|
@ -928,8 +926,6 @@ int32_t qwProcessDelete(QW_FPARAMS_DEF, SQWMsg *qwMsg, SDeleteRes *pRes) {
|
||||||
QW_ERR_JRET(code);
|
QW_ERR_JRET(code);
|
||||||
}
|
}
|
||||||
|
|
||||||
ctx.plan = plan;
|
|
||||||
|
|
||||||
code = qCreateExecTask(qwMsg->node, mgmt->nodeId, tId, plan, &pTaskInfo, &sinkHandle, NULL, OPTR_EXEC_MODEL_BATCH);
|
code = qCreateExecTask(qwMsg->node, mgmt->nodeId, tId, plan, &pTaskInfo, &sinkHandle, NULL, OPTR_EXEC_MODEL_BATCH);
|
||||||
if (code) {
|
if (code) {
|
||||||
QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code));
|
QW_TASK_ELOG("qCreateExecTask failed, code:%x - %s", code, tstrerror(code));
|
||||||
|
|
|
@ -277,7 +277,7 @@ typedef struct SSchJob {
|
||||||
bool fetched;
|
bool fetched;
|
||||||
int32_t resNumOfRows;
|
int32_t resNumOfRows;
|
||||||
SSchResInfo userRes;
|
SSchResInfo userRes;
|
||||||
const char *sql;
|
char *sql;
|
||||||
SQueryProfileSummary summary;
|
SQueryProfileSummary summary;
|
||||||
} SSchJob;
|
} SSchJob;
|
||||||
|
|
||||||
|
@ -461,7 +461,6 @@ int32_t schJobFetchRows(SSchJob *pJob);
|
||||||
int32_t schJobFetchRowsA(SSchJob *pJob);
|
int32_t schJobFetchRowsA(SSchJob *pJob);
|
||||||
int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, void *handle, int32_t execId);
|
int32_t schUpdateTaskHandle(SSchJob *pJob, SSchTask *pTask, bool dropExecNode, void *handle, int32_t execId);
|
||||||
int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId* pEpId, SArray* pStatusList);
|
int32_t schProcessOnTaskStatusRsp(SQueryNodeEpId* pEpId, SArray* pStatusList);
|
||||||
void schFreeSMsgSendInfo(SMsgSendInfo *msgSendInfo);
|
|
||||||
char* schGetOpStr(SCH_OP_TYPE type);
|
char* schGetOpStr(SCH_OP_TYPE type);
|
||||||
int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync);
|
int32_t schBeginOperation(SSchJob *pJob, SCH_OP_TYPE type, bool sync);
|
||||||
int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq);
|
int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq);
|
||||||
|
|
|
@ -675,6 +675,7 @@ void schFreeJobImpl(void *job) {
|
||||||
|
|
||||||
taosMemoryFreeClear(pJob->userRes.execRes);
|
taosMemoryFreeClear(pJob->userRes.execRes);
|
||||||
taosMemoryFreeClear(pJob->fetchRes);
|
taosMemoryFreeClear(pJob->fetchRes);
|
||||||
|
taosMemoryFreeClear(pJob->sql);
|
||||||
taosMemoryFree(pJob);
|
taosMemoryFree(pJob);
|
||||||
|
|
||||||
int32_t jobNum = atomic_sub_fetch_32(&schMgmt.jobNum, 1);
|
int32_t jobNum = atomic_sub_fetch_32(&schMgmt.jobNum, 1);
|
||||||
|
@ -718,7 +719,9 @@ int32_t schInitJob(int64_t *pJobId, SSchedulerReq *pReq) {
|
||||||
|
|
||||||
pJob->attr.explainMode = pReq->pDag->explainInfo.mode;
|
pJob->attr.explainMode = pReq->pDag->explainInfo.mode;
|
||||||
pJob->conn = *pReq->pConn;
|
pJob->conn = *pReq->pConn;
|
||||||
pJob->sql = pReq->sql;
|
if (pReq->sql) {
|
||||||
|
pJob->sql = strdup(pReq->sql);
|
||||||
|
}
|
||||||
pJob->pDag = pReq->pDag;
|
pJob->pDag = pReq->pDag;
|
||||||
pJob->chkKillFp = pReq->chkKillFp;
|
pJob->chkKillFp = pReq->chkKillFp;
|
||||||
pJob->chkKillParam = pReq->chkKillParam;
|
pJob->chkKillParam = pReq->chkKillParam;
|
||||||
|
|
|
@ -386,7 +386,6 @@ int32_t schHandleCallback(void *param, SDataBuf *pMsg, int32_t rspCode) {
|
||||||
schProcessOnCbEnd(pJob, pTask, code);
|
schProcessOnCbEnd(pJob, pTask, code);
|
||||||
|
|
||||||
taosMemoryFreeClear(pMsg->pData);
|
taosMemoryFreeClear(pMsg->pData);
|
||||||
taosMemoryFreeClear(param);
|
|
||||||
|
|
||||||
qDebug("end to handle rsp msg, type:%s, handle:%p, code:%s", TMSG_INFO(pMsg->msgType), pMsg->handle,
|
qDebug("end to handle rsp msg, type:%s, handle:%p, code:%s", TMSG_INFO(pMsg->msgType), pMsg->handle,
|
||||||
tstrerror(rspCode));
|
tstrerror(rspCode));
|
||||||
|
@ -398,7 +397,6 @@ int32_t schHandleDropCallback(void *param, SDataBuf *pMsg, int32_t code) {
|
||||||
SSchTaskCallbackParam *pParam = (SSchTaskCallbackParam *)param;
|
SSchTaskCallbackParam *pParam = (SSchTaskCallbackParam *)param;
|
||||||
qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 " drop task rsp received, code:0x%x", pParam->queryId, pParam->taskId,
|
qDebug("QID:0x%" PRIx64 ",TID:0x%" PRIx64 " drop task rsp received, code:0x%x", pParam->queryId, pParam->taskId,
|
||||||
code);
|
code);
|
||||||
taosMemoryFreeClear(param);
|
|
||||||
return TSDB_CODE_SUCCESS;
|
return TSDB_CODE_SUCCESS;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -447,8 +445,8 @@ int32_t schHandleHbCallback(void *param, SDataBuf *pMsg, int32_t code) {
|
||||||
SCH_ERR_JRET(schProcessOnTaskStatusRsp(&rsp.epId, rsp.taskStatus));
|
SCH_ERR_JRET(schProcessOnTaskStatusRsp(&rsp.epId, rsp.taskStatus));
|
||||||
|
|
||||||
_return:
|
_return:
|
||||||
|
|
||||||
tFreeSSchedulerHbRsp(&rsp);
|
tFreeSSchedulerHbRsp(&rsp);
|
||||||
taosMemoryFree(param);
|
|
||||||
taosMemoryFree(pMsg->pData);
|
taosMemoryFree(pMsg->pData);
|
||||||
SCH_RET(code);
|
SCH_RET(code);
|
||||||
}
|
}
|
||||||
|
@ -514,7 +512,9 @@ int32_t schGenerateCallBackInfo(SSchJob *pJob, SSchTask *pTask, void *msg, uint3
|
||||||
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
|
SCH_ERR_RET(TSDB_CODE_QRY_OUT_OF_MEMORY);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
msgSendInfo->paramFreeFp = taosMemoryFree;
|
||||||
SCH_ERR_JRET(schMakeCallbackParam(pJob, pTask, msgType, isHb, trans, &msgSendInfo->param));
|
SCH_ERR_JRET(schMakeCallbackParam(pJob, pTask, msgType, isHb, trans, &msgSendInfo->param));
|
||||||
|
|
||||||
SCH_ERR_JRET(schGetCallbackFp(msgType, &msgSendInfo->fp));
|
SCH_ERR_JRET(schGetCallbackFp(msgType, &msgSendInfo->fp));
|
||||||
|
|
||||||
if (pJob) {
|
if (pJob) {
|
||||||
|
@ -535,7 +535,7 @@ int32_t schGenerateCallBackInfo(SSchJob *pJob, SSchTask *pTask, void *msg, uint3
|
||||||
|
|
||||||
_return:
|
_return:
|
||||||
|
|
||||||
schFreeSMsgSendInfo(msgSendInfo);
|
destroySendMsgInfo(msgSendInfo);
|
||||||
|
|
||||||
SCH_RET(code);
|
SCH_RET(code);
|
||||||
}
|
}
|
||||||
|
@ -676,6 +676,7 @@ int32_t schMakeHbRpcCtx(SSchJob *pJob, SSchTask *pTask, SRpcCtx *pCtx) {
|
||||||
param->pTrans = pJob->conn.pTrans;
|
param->pTrans = pJob->conn.pTrans;
|
||||||
|
|
||||||
pMsgSendInfo->param = param;
|
pMsgSendInfo->param = param;
|
||||||
|
pMsgSendInfo->paramFreeFp = taosMemoryFree;
|
||||||
pMsgSendInfo->fp = fp;
|
pMsgSendInfo->fp = fp;
|
||||||
|
|
||||||
SRpcCtxVal ctxVal = {.val = pMsgSendInfo, .clone = schCloneSMsgSendInfo};
|
SRpcCtxVal ctxVal = {.val = pMsgSendInfo, .clone = schCloneSMsgSendInfo};
|
||||||
|
@ -795,6 +796,7 @@ int32_t schCloneSMsgSendInfo(void *src, void **dst) {
|
||||||
pDst->param = NULL;
|
pDst->param = NULL;
|
||||||
|
|
||||||
SCH_ERR_JRET(schCloneCallbackParam(pSrc->param, (SSchCallbackParamHeader **)&pDst->param));
|
SCH_ERR_JRET(schCloneCallbackParam(pSrc->param, (SSchCallbackParamHeader **)&pDst->param));
|
||||||
|
pDst->paramFreeFp = taosMemoryFree;
|
||||||
|
|
||||||
*dst = pDst;
|
*dst = pDst;
|
||||||
|
|
||||||
|
@ -861,8 +863,7 @@ _return:
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pMsgSendInfo) {
|
if (pMsgSendInfo) {
|
||||||
taosMemoryFreeClear(pMsgSendInfo->param);
|
destroySendMsgInfo(pMsgSendInfo);
|
||||||
taosMemoryFreeClear(pMsgSendInfo);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
SCH_RET(code);
|
SCH_RET(code);
|
||||||
|
|
|
@ -50,6 +50,12 @@ char* schGetOpStr(SCH_OP_TYPE type) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
void schFreeHbTrans(SSchHbTrans *pTrans) {
|
||||||
|
rpcReleaseHandle(pTrans->trans.pHandle, TAOS_CONN_CLIENT);
|
||||||
|
|
||||||
|
schFreeRpcCtx(&pTrans->rpcCtx);
|
||||||
|
}
|
||||||
|
|
||||||
void schCleanClusterHb(void* pTrans) {
|
void schCleanClusterHb(void* pTrans) {
|
||||||
SCH_LOCK(SCH_WRITE, &schMgmt.hbLock);
|
SCH_LOCK(SCH_WRITE, &schMgmt.hbLock);
|
||||||
|
|
||||||
|
@ -57,7 +63,7 @@ void schCleanClusterHb(void* pTrans) {
|
||||||
while (hb) {
|
while (hb) {
|
||||||
if (hb->trans.pTrans == pTrans) {
|
if (hb->trans.pTrans == pTrans) {
|
||||||
SQueryNodeEpId* pEpId = taosHashGetKey(hb, NULL);
|
SQueryNodeEpId* pEpId = taosHashGetKey(hb, NULL);
|
||||||
rpcReleaseHandle(hb->trans.pHandle, TAOS_CONN_CLIENT);
|
schFreeHbTrans(hb);
|
||||||
taosHashRemove(schMgmt.hbConnections, pEpId, sizeof(SQueryNodeEpId));
|
taosHashRemove(schMgmt.hbConnections, pEpId, sizeof(SQueryNodeEpId));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -68,8 +74,6 @@ void schCleanClusterHb(void* pTrans) {
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t schRemoveHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *epId) {
|
int32_t schRemoveHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *epId) {
|
||||||
return TSDB_CODE_SUCCESS; // TODO ENABLE IT WHEN RPC IS READY
|
|
||||||
|
|
||||||
int32_t code = 0;
|
int32_t code = 0;
|
||||||
|
|
||||||
SCH_LOCK(SCH_WRITE, &schMgmt.hbLock);
|
SCH_LOCK(SCH_WRITE, &schMgmt.hbLock);
|
||||||
|
@ -82,7 +86,7 @@ int32_t schRemoveHbConnection(SSchJob *pJob, SSchTask *pTask, SQueryNodeEpId *ep
|
||||||
|
|
||||||
int64_t taskNum = atomic_load_64(&hb->taskNum);
|
int64_t taskNum = atomic_load_64(&hb->taskNum);
|
||||||
if (taskNum <= 0) {
|
if (taskNum <= 0) {
|
||||||
rpcReleaseHandle(hb->trans.pHandle, TAOS_CONN_CLIENT);
|
schFreeHbTrans(hb);
|
||||||
taosHashRemove(schMgmt.hbConnections, epId, sizeof(SQueryNodeEpId));
|
taosHashRemove(schMgmt.hbConnections, epId, sizeof(SQueryNodeEpId));
|
||||||
}
|
}
|
||||||
SCH_UNLOCK(SCH_WRITE, &schMgmt.hbLock);
|
SCH_UNLOCK(SCH_WRITE, &schMgmt.hbLock);
|
||||||
|
@ -265,9 +269,7 @@ void schFreeRpcCtxVal(const void *arg) {
|
||||||
}
|
}
|
||||||
|
|
||||||
SMsgSendInfo *pMsgSendInfo = (SMsgSendInfo *)arg;
|
SMsgSendInfo *pMsgSendInfo = (SMsgSendInfo *)arg;
|
||||||
taosMemoryFreeClear(pMsgSendInfo->param);
|
destroySendMsgInfo(pMsgSendInfo);
|
||||||
taosMemoryFreeClear(pMsgSendInfo->msgInfo.pData);
|
|
||||||
taosMemoryFreeClear(pMsgSendInfo);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
void schFreeRpcCtx(SRpcCtx *pCtx) {
|
void schFreeRpcCtx(SRpcCtx *pCtx) {
|
||||||
|
@ -290,15 +292,6 @@ void schFreeRpcCtx(SRpcCtx *pCtx) {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
void schFreeSMsgSendInfo(SMsgSendInfo *msgSendInfo) {
|
|
||||||
if (NULL == msgSendInfo) {
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
|
|
||||||
taosMemoryFree(msgSendInfo->param);
|
|
||||||
taosMemoryFree(msgSendInfo);
|
|
||||||
}
|
|
||||||
|
|
||||||
int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTask) {
|
int32_t schGetTaskFromList(SHashObj *pTaskList, uint64_t taskId, SSchTask **pTask) {
|
||||||
int32_t s = taosHashGetSize(pTaskList);
|
int32_t s = taosHashGetSize(pTaskList);
|
||||||
if (s <= 0) {
|
if (s <= 0) {
|
||||||
|
|
|
@ -64,6 +64,7 @@ int32_t tEncodeSStreamTask(SEncoder* pEncoder, const SStreamTask* pTask) {
|
||||||
if (tEncodeI32(pEncoder, pTask->selfChildId) < 0) return -1;
|
if (tEncodeI32(pEncoder, pTask->selfChildId) < 0) return -1;
|
||||||
if (tEncodeI32(pEncoder, pTask->nodeId) < 0) return -1;
|
if (tEncodeI32(pEncoder, pTask->nodeId) < 0) return -1;
|
||||||
if (tEncodeSEpSet(pEncoder, &pTask->epSet) < 0) return -1;
|
if (tEncodeSEpSet(pEncoder, &pTask->epSet) < 0) return -1;
|
||||||
|
if (tEncodeI32(pEncoder, pTask->numOfVgroups) < 0) return -1;
|
||||||
|
|
||||||
int32_t epSz = taosArrayGetSize(pTask->childEpInfo);
|
int32_t epSz = taosArrayGetSize(pTask->childEpInfo);
|
||||||
if (tEncodeI32(pEncoder, epSz) < 0) return -1;
|
if (tEncodeI32(pEncoder, epSz) < 0) return -1;
|
||||||
|
@ -118,6 +119,7 @@ int32_t tDecodeSStreamTask(SDecoder* pDecoder, SStreamTask* pTask) {
|
||||||
if (tDecodeI32(pDecoder, &pTask->selfChildId) < 0) return -1;
|
if (tDecodeI32(pDecoder, &pTask->selfChildId) < 0) return -1;
|
||||||
if (tDecodeI32(pDecoder, &pTask->nodeId) < 0) return -1;
|
if (tDecodeI32(pDecoder, &pTask->nodeId) < 0) return -1;
|
||||||
if (tDecodeSEpSet(pDecoder, &pTask->epSet) < 0) return -1;
|
if (tDecodeSEpSet(pDecoder, &pTask->epSet) < 0) return -1;
|
||||||
|
if (tDecodeI32(pDecoder, &pTask->numOfVgroups) < 0) return -1;
|
||||||
|
|
||||||
int32_t epSz;
|
int32_t epSz;
|
||||||
if (tDecodeI32(pDecoder, &epSz) < 0) return -1;
|
if (tDecodeI32(pDecoder, &epSz) < 0) return -1;
|
||||||
|
|
|
@ -125,7 +125,7 @@ int32_t syncNodeRequestVote(SSyncNode* pSyncNode, const SRaftId* destRaftId, con
|
||||||
syncUtilU642Addr(destRaftId->addr, host, sizeof(host), &port);
|
syncUtilU642Addr(destRaftId->addr, host, sizeof(host), &port);
|
||||||
sDebug("vgId:%d, send sync-request-vote to %s:%d, {term:%" PRIu64 ", last-index:%" PRId64 ", last-term:%" PRIu64
|
sDebug("vgId:%d, send sync-request-vote to %s:%d, {term:%" PRIu64 ", last-index:%" PRId64 ", last-term:%" PRIu64
|
||||||
"}",
|
"}",
|
||||||
pSyncNode->vgId, host, port, pMsg->term, pMsg->lastLogTerm, pMsg->lastLogIndex);
|
pSyncNode->vgId, host, port, pMsg->term, pMsg->lastLogIndex, pMsg->lastLogTerm);
|
||||||
} while (0);
|
} while (0);
|
||||||
|
|
||||||
SRpcMsg rpcMsg;
|
SRpcMsg rpcMsg;
|
||||||
|
|
|
@ -1539,20 +1539,21 @@ void syncNodeEventLog(const SSyncNode* pSyncNode, char* str) {
|
||||||
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
||||||
snprintf(logBuf, sizeof(logBuf),
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
||||||
", snapshot:%" PRId64
|
", snapshot:%" PRId64 ", snapshot-term:%" PRIu64
|
||||||
", standby:%d, "
|
", standby:%d, "
|
||||||
"strategy:%d, batch:%d, "
|
"strategy:%d, batch:%d, "
|
||||||
"replica-num:%d, "
|
"replica-num:%d, "
|
||||||
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
||||||
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
||||||
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex,
|
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
|
||||||
pSyncNode->pRaftCfg->isStandBy, pSyncNode->pRaftCfg->snapshotStrategy, pSyncNode->pRaftCfg->batchSize,
|
pSyncNode->pRaftCfg->isStandBy, pSyncNode->pRaftCfg->snapshotStrategy, pSyncNode->pRaftCfg->batchSize,
|
||||||
pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing,
|
pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing,
|
||||||
pSyncNode->restoreFinish, printStr);
|
pSyncNode->restoreFinish, printStr);
|
||||||
} else {
|
} else {
|
||||||
snprintf(logBuf, sizeof(logBuf), "%s", str);
|
snprintf(logBuf, sizeof(logBuf), "%s", str);
|
||||||
}
|
}
|
||||||
sDebug("%s", logBuf);
|
// sDebug("%s", logBuf);
|
||||||
|
sInfo("%s", logBuf);
|
||||||
|
|
||||||
} else {
|
} else {
|
||||||
int len = 256 + userStrLen;
|
int len = 256 + userStrLen;
|
||||||
|
@ -1560,20 +1561,21 @@ void syncNodeEventLog(const SSyncNode* pSyncNode, char* str) {
|
||||||
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
||||||
snprintf(s, len,
|
snprintf(s, len,
|
||||||
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
||||||
", snapshot:%" PRId64
|
", snapshot:%" PRId64 ", snapshot-term:%" PRIu64
|
||||||
", standby:%d, "
|
", standby:%d, "
|
||||||
"strategy:%d, batch:%d, "
|
"strategy:%d, batch:%d, "
|
||||||
"replica-num:%d, "
|
"replica-num:%d, "
|
||||||
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
||||||
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
||||||
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex,
|
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
|
||||||
pSyncNode->pRaftCfg->isStandBy, pSyncNode->pRaftCfg->snapshotStrategy, pSyncNode->pRaftCfg->batchSize,
|
pSyncNode->pRaftCfg->isStandBy, pSyncNode->pRaftCfg->snapshotStrategy, pSyncNode->pRaftCfg->batchSize,
|
||||||
pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing,
|
pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing,
|
||||||
pSyncNode->restoreFinish, printStr);
|
pSyncNode->restoreFinish, printStr);
|
||||||
} else {
|
} else {
|
||||||
snprintf(s, len, "%s", str);
|
snprintf(s, len, "%s", str);
|
||||||
}
|
}
|
||||||
sDebug("%s", s);
|
// sDebug("%s", s);
|
||||||
|
sInfo("%s", s);
|
||||||
taosMemoryFree(s);
|
taosMemoryFree(s);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -1606,14 +1608,16 @@ void syncNodeErrorLog(const SSyncNode* pSyncNode, char* str) {
|
||||||
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
||||||
snprintf(logBuf, sizeof(logBuf),
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
||||||
", snapshot:%" PRId64
|
", snapshot:%" PRId64 ", snapshot-term:%" PRIu64
|
||||||
", standby:%d, "
|
", standby:%d, "
|
||||||
|
"strategy:%d, batch:%d, "
|
||||||
"replica-num:%d, "
|
"replica-num:%d, "
|
||||||
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
||||||
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
||||||
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex,
|
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
|
||||||
pSyncNode->pRaftCfg->isStandBy, pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex,
|
pSyncNode->pRaftCfg->isStandBy, pSyncNode->pRaftCfg->snapshotStrategy, pSyncNode->pRaftCfg->batchSize,
|
||||||
pSyncNode->changing, pSyncNode->restoreFinish, printStr);
|
pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing,
|
||||||
|
pSyncNode->restoreFinish, printStr);
|
||||||
} else {
|
} else {
|
||||||
snprintf(logBuf, sizeof(logBuf), "%s", str);
|
snprintf(logBuf, sizeof(logBuf), "%s", str);
|
||||||
}
|
}
|
||||||
|
@ -1625,14 +1629,16 @@ void syncNodeErrorLog(const SSyncNode* pSyncNode, char* str) {
|
||||||
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
if (pSyncNode != NULL && pSyncNode->pRaftCfg != NULL && pSyncNode->pRaftStore != NULL) {
|
||||||
snprintf(s, len,
|
snprintf(s, len,
|
||||||
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
"vgId:%d, sync %s %s, term:%" PRIu64 ", commit:%" PRId64 ", first:%" PRId64 ", last:%" PRId64
|
||||||
", snapshot:%" PRId64
|
", snapshot:%" PRId64 ", snapshot-term:%" PRIu64
|
||||||
", standby:%d, "
|
", standby:%d, "
|
||||||
|
"strategy:%d, batch:%d, "
|
||||||
"replica-num:%d, "
|
"replica-num:%d, "
|
||||||
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
"lconfig:%" PRId64 ", changing:%d, restore:%d, %s",
|
||||||
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
pSyncNode->vgId, syncUtilState2String(pSyncNode->state), str, pSyncNode->pRaftStore->currentTerm,
|
||||||
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex,
|
pSyncNode->commitIndex, logBeginIndex, logLastIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
|
||||||
pSyncNode->pRaftCfg->isStandBy, pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex,
|
pSyncNode->pRaftCfg->isStandBy, pSyncNode->pRaftCfg->snapshotStrategy, pSyncNode->pRaftCfg->batchSize,
|
||||||
pSyncNode->changing, pSyncNode->restoreFinish, printStr);
|
pSyncNode->replicaNum, pSyncNode->pRaftCfg->lastConfigIndex, pSyncNode->changing,
|
||||||
|
pSyncNode->restoreFinish, printStr);
|
||||||
} else {
|
} else {
|
||||||
snprintf(s, len, "%s", str);
|
snprintf(s, len, "%s", str);
|
||||||
}
|
}
|
||||||
|
|
|
@ -313,7 +313,7 @@ static int32_t raftLogGetEntry(struct SSyncLogStore* pLogStore, SyncIndex index,
|
||||||
snprintf(logBuf, sizeof(logBuf), "wal read error, index:%" PRId64 ", err:%d %X, msg:%s, syserr:%d, sysmsg:%s",
|
snprintf(logBuf, sizeof(logBuf), "wal read error, index:%" PRId64 ", err:%d %X, msg:%s, syserr:%d, sysmsg:%s",
|
||||||
index, err, err, errStr, sysErr, sysErrStr);
|
index, err, err, errStr, sysErr, sysErrStr);
|
||||||
if (terrno == TSDB_CODE_WAL_LOG_NOT_EXIST) {
|
if (terrno == TSDB_CODE_WAL_LOG_NOT_EXIST) {
|
||||||
syncNodeEventLog(pData->pSyncNode, logBuf);
|
// syncNodeEventLog(pData->pSyncNode, logBuf);
|
||||||
} else {
|
} else {
|
||||||
syncNodeErrorLog(pData->pSyncNode, logBuf);
|
syncNodeErrorLog(pData->pSyncNode, logBuf);
|
||||||
}
|
}
|
||||||
|
@ -499,7 +499,7 @@ SSyncRaftEntry* logStoreGetEntry(SSyncLogStore* pLogStore, SyncIndex index) {
|
||||||
snprintf(logBuf, sizeof(logBuf), "wal read error, index:%" PRId64 ", err:%d %X, msg:%s, syserr:%d, sysmsg:%s",
|
snprintf(logBuf, sizeof(logBuf), "wal read error, index:%" PRId64 ", err:%d %X, msg:%s, syserr:%d, sysmsg:%s",
|
||||||
index, err, err, errStr, sysErr, sysErrStr);
|
index, err, err, errStr, sysErr, sysErrStr);
|
||||||
if (terrno == TSDB_CODE_WAL_LOG_NOT_EXIST) {
|
if (terrno == TSDB_CODE_WAL_LOG_NOT_EXIST) {
|
||||||
syncNodeEventLog(pData->pSyncNode, logBuf);
|
// syncNodeEventLog(pData->pSyncNode, logBuf);
|
||||||
} else {
|
} else {
|
||||||
syncNodeErrorLog(pData->pSyncNode, logBuf);
|
syncNodeErrorLog(pData->pSyncNode, logBuf);
|
||||||
}
|
}
|
||||||
|
|
|
@ -159,16 +159,53 @@ static bool syncNodeOnRequestVoteLogOK(SSyncNode* pSyncNode, SyncRequestVote* pM
|
||||||
SyncIndex myLastIndex = syncNodeGetLastIndex(pSyncNode);
|
SyncIndex myLastIndex = syncNodeGetLastIndex(pSyncNode);
|
||||||
|
|
||||||
if (myLastTerm == SYNC_TERM_INVALID) {
|
if (myLastTerm == SYNC_TERM_INVALID) {
|
||||||
|
do {
|
||||||
|
char logBuf[128];
|
||||||
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
|
"logok:0, {my-lterm:%" PRIu64 ", my-lindex:%" PRId64 ", recv-lterm:%" PRIu64 ", recv-lindex:%" PRId64
|
||||||
|
", recv-term:%" PRIu64 "}",
|
||||||
|
myLastTerm, myLastIndex, pMsg->lastLogTerm, pMsg->lastLogIndex, pMsg->term);
|
||||||
|
syncNodeEventLog(pSyncNode, logBuf);
|
||||||
|
} while (0);
|
||||||
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pMsg->lastLogTerm > myLastTerm) {
|
if (pMsg->lastLogTerm > myLastTerm) {
|
||||||
|
do {
|
||||||
|
char logBuf[128];
|
||||||
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
|
"logok:1, {my-lterm:%" PRIu64 ", my-lindex:%" PRId64 ", recv-lterm:%" PRIu64 ", recv-lindex:%" PRId64
|
||||||
|
", recv-term:%" PRIu64 "}",
|
||||||
|
myLastTerm, myLastIndex, pMsg->lastLogTerm, pMsg->lastLogIndex, pMsg->term);
|
||||||
|
syncNodeEventLog(pSyncNode, logBuf);
|
||||||
|
} while (0);
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (pMsg->lastLogTerm == myLastTerm && pMsg->lastLogIndex >= myLastIndex) {
|
if (pMsg->lastLogTerm == myLastTerm && pMsg->lastLogIndex >= myLastIndex) {
|
||||||
|
do {
|
||||||
|
char logBuf[128];
|
||||||
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
|
"logok:1, {my-lterm:%" PRIu64 ", my-lindex:%" PRId64 ", recv-lterm:%" PRIu64 ", recv-lindex:%" PRId64
|
||||||
|
", recv-term:%" PRIu64 "}",
|
||||||
|
myLastTerm, myLastIndex, pMsg->lastLogTerm, pMsg->lastLogIndex, pMsg->term);
|
||||||
|
syncNodeEventLog(pSyncNode, logBuf);
|
||||||
|
} while (0);
|
||||||
|
|
||||||
return true;
|
return true;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
do {
|
||||||
|
char logBuf[128];
|
||||||
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
|
"logok:0, {my-lterm:%" PRIu64 ", my-lindex:%" PRId64 ", recv-lterm:%" PRIu64 ", recv-lindex:%" PRId64
|
||||||
|
", recv-term:%" PRIu64 "}",
|
||||||
|
myLastTerm, myLastIndex, pMsg->lastLogTerm, pMsg->lastLogIndex, pMsg->term);
|
||||||
|
syncNodeEventLog(pSyncNode, logBuf);
|
||||||
|
} while (0);
|
||||||
|
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
@ -224,8 +261,8 @@ int32_t syncNodeOnRequestVoteSnapshotCb(SSyncNode* ths, SyncRequestVote* pMsg) {
|
||||||
uint16_t port;
|
uint16_t port;
|
||||||
syncUtilU642Addr(pMsg->srcId.addr, host, sizeof(host), &port);
|
syncUtilU642Addr(pMsg->srcId.addr, host, sizeof(host), &port);
|
||||||
snprintf(logBuf, sizeof(logBuf),
|
snprintf(logBuf, sizeof(logBuf),
|
||||||
"recv sync-request-vote from %s:%d, term:%" PRIu64 ", lindex:%" PRId64 ", lterm:%" PRIu64
|
"recv sync-request-vote from %s:%d, {term:%" PRIu64 ", lindex:%" PRId64 ", lterm:%" PRIu64
|
||||||
", reply-grant:%d",
|
", reply-grant:%d}",
|
||||||
host, port, pMsg->term, pMsg->lastLogIndex, pMsg->lastLogTerm, pReply->voteGranted);
|
host, port, pMsg->term, pMsg->lastLogIndex, pMsg->lastLogTerm, pReply->voteGranted);
|
||||||
syncNodeEventLog(ths, logBuf);
|
syncNodeEventLog(ths, logBuf);
|
||||||
} while (0);
|
} while (0);
|
||||||
|
|
|
@ -120,15 +120,18 @@ void syncRespClean(SSyncRespMgr *pObj) {
|
||||||
void syncRespCleanByTTL(SSyncRespMgr *pObj, int64_t ttl) {
|
void syncRespCleanByTTL(SSyncRespMgr *pObj, int64_t ttl) {
|
||||||
SRespStub *pStub = (SRespStub *)taosHashIterate(pObj->pRespHash, NULL);
|
SRespStub *pStub = (SRespStub *)taosHashIterate(pObj->pRespHash, NULL);
|
||||||
int cnt = 0;
|
int cnt = 0;
|
||||||
|
int sum = 0;
|
||||||
SSyncNode *pSyncNode = pObj->data;
|
SSyncNode *pSyncNode = pObj->data;
|
||||||
|
|
||||||
SArray *delIndexArray = taosArrayInit(0, sizeof(uint64_t));
|
SArray *delIndexArray = taosArrayInit(0, sizeof(uint64_t));
|
||||||
ASSERT(delIndexArray != NULL);
|
ASSERT(delIndexArray != NULL);
|
||||||
|
sDebug("vgId:%d, resp mgr begin clean by ttl", pSyncNode->vgId);
|
||||||
|
|
||||||
while (pStub) {
|
while (pStub) {
|
||||||
size_t len;
|
size_t len;
|
||||||
void *key = taosHashGetKey(pStub, &len);
|
void *key = taosHashGetKey(pStub, &len);
|
||||||
uint64_t *pSeqNum = (uint64_t *)key;
|
uint64_t *pSeqNum = (uint64_t *)key;
|
||||||
|
sum++;
|
||||||
|
|
||||||
int64_t nowMS = taosGetTimestampMs();
|
int64_t nowMS = taosGetTimestampMs();
|
||||||
if (nowMS - pStub->createTime > ttl) {
|
if (nowMS - pStub->createTime > ttl) {
|
||||||
|
@ -155,7 +158,7 @@ void syncRespCleanByTTL(SSyncRespMgr *pObj, int64_t ttl) {
|
||||||
}
|
}
|
||||||
|
|
||||||
int32_t arraySize = taosArrayGetSize(delIndexArray);
|
int32_t arraySize = taosArrayGetSize(delIndexArray);
|
||||||
sDebug("vgId:%d, resp mgr clean by ttl, cnt:%d, array-size:%d", pSyncNode->vgId, cnt, arraySize);
|
sDebug("vgId:%d, resp mgr end clean by ttl, sum:%d, cnt:%d, array-size:%d", pSyncNode->vgId, sum, cnt, arraySize);
|
||||||
|
|
||||||
for (int32_t i = 0; i < arraySize; ++i) {
|
for (int32_t i = 0; i < arraySize; ++i) {
|
||||||
uint64_t *pSeqNum = taosArrayGet(delIndexArray, i);
|
uint64_t *pSeqNum = taosArrayGet(delIndexArray, i);
|
||||||
|
|
|
@ -30,6 +30,7 @@ import argparse
|
||||||
import sys
|
import sys
|
||||||
import os
|
import os
|
||||||
import io
|
import io
|
||||||
|
import datetime
|
||||||
import signal
|
import signal
|
||||||
import traceback
|
import traceback
|
||||||
import requests
|
import requests
|
||||||
|
@ -1107,14 +1108,20 @@ class Database:
|
||||||
# TODO: fix the error as result of above: "tsdb timestamp is out of range"
|
# TODO: fix the error as result of above: "tsdb timestamp is out of range"
|
||||||
@classmethod
|
@classmethod
|
||||||
def setupLastTick(cls):
|
def setupLastTick(cls):
|
||||||
t1 = datetime.datetime(2020, 6, 1)
|
# start time will be auto generated , start at 10 years ago local time
|
||||||
|
local_time = datetime.datetime.utcnow().strftime('%Y-%m-%d %H:%M:%S.%f')[:-16]
|
||||||
|
local_epoch_time = [int(i) for i in local_time.split("-")]
|
||||||
|
#local_epoch_time will be such as : [2022, 7, 18]
|
||||||
|
|
||||||
|
t1 = datetime.datetime(local_epoch_time[0]-5, local_epoch_time[1], local_epoch_time[2])
|
||||||
t2 = datetime.datetime.now()
|
t2 = datetime.datetime.now()
|
||||||
# maybe a very large number, takes 69 years to exceed Python int range
|
# maybe a very large number, takes 69 years to exceed Python int range
|
||||||
elSec = int(t2.timestamp() - t1.timestamp())
|
elSec = int(t2.timestamp() - t1.timestamp())
|
||||||
elSec2 = (elSec % (8 * 12 * 30 * 24 * 60 * 60 / 500)) * \
|
elSec2 = (elSec % (8 * 12 * 30 * 24 * 60 * 60 / 500)) * \
|
||||||
500 # a number representing seconds within 10 years
|
500 # a number representing seconds within 10 years
|
||||||
# print("elSec = {}".format(elSec))
|
# print("elSec = {}".format(elSec))
|
||||||
t3 = datetime.datetime(2012, 1, 1) # default "keep" is 10 years
|
|
||||||
|
t3 = datetime.datetime(local_epoch_time[0]-10, local_epoch_time[1], local_epoch_time[2]) # default "keep" is 10 years
|
||||||
t4 = datetime.datetime.fromtimestamp(
|
t4 = datetime.datetime.fromtimestamp(
|
||||||
t3.timestamp() + elSec2) # see explanation above
|
t3.timestamp() + elSec2) # see explanation above
|
||||||
Logging.debug("Setting up TICKS to start from: {}".format(t4))
|
Logging.debug("Setting up TICKS to start from: {}".format(t4))
|
||||||
|
|
|
@ -489,9 +489,12 @@ class TDDnode:
|
||||||
processID = subprocess.check_output(
|
processID = subprocess.check_output(
|
||||||
psCmd, shell=True).decode("utf-8")
|
psCmd, shell=True).decode("utf-8")
|
||||||
|
|
||||||
|
onlyKillOnceWindows = 0
|
||||||
while(processID):
|
while(processID):
|
||||||
|
if not platform.system().lower() == 'windows' or (onlyKillOnceWindows == 0 and platform.system().lower() == 'windows'):
|
||||||
killCmd = "kill -INT %s > /dev/null 2>&1" % processID
|
killCmd = "kill -INT %s > /dev/null 2>&1" % processID
|
||||||
os.system(killCmd)
|
os.system(killCmd)
|
||||||
|
onlyKillOnceWindows = 1
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
processID = subprocess.check_output(
|
processID = subprocess.check_output(
|
||||||
psCmd, shell=True).decode("utf-8")
|
psCmd, shell=True).decode("utf-8")
|
||||||
|
@ -524,9 +527,12 @@ class TDDnode:
|
||||||
processID = subprocess.check_output(
|
processID = subprocess.check_output(
|
||||||
psCmd, shell=True).decode("utf-8")
|
psCmd, shell=True).decode("utf-8")
|
||||||
|
|
||||||
|
onlyKillOnceWindows = 0
|
||||||
while(processID):
|
while(processID):
|
||||||
|
if not platform.system().lower() == 'windows' or (onlyKillOnceWindows == 0 and platform.system().lower() == 'windows'):
|
||||||
killCmd = "kill -INT %s > /dev/null 2>&1" % processID
|
killCmd = "kill -INT %s > /dev/null 2>&1" % processID
|
||||||
os.system(killCmd)
|
os.system(killCmd)
|
||||||
|
onlyKillOnceWindows = 1
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
processID = subprocess.check_output(
|
processID = subprocess.check_output(
|
||||||
psCmd, shell=True).decode("utf-8")
|
psCmd, shell=True).decode("utf-8")
|
||||||
|
@ -550,9 +556,12 @@ class TDDnode:
|
||||||
processID = subprocess.check_output(
|
processID = subprocess.check_output(
|
||||||
psCmd, shell=True).decode("utf-8")
|
psCmd, shell=True).decode("utf-8")
|
||||||
|
|
||||||
|
onlyKillOnceWindows = 0
|
||||||
while(processID):
|
while(processID):
|
||||||
|
if not platform.system().lower() == 'windows' or (onlyKillOnceWindows == 0 and platform.system().lower() == 'windows'):
|
||||||
killCmd = "kill -KILL %s > /dev/null 2>&1" % processID
|
killCmd = "kill -KILL %s > /dev/null 2>&1" % processID
|
||||||
os.system(killCmd)
|
os.system(killCmd)
|
||||||
|
onlyKillOnceWindows = 1
|
||||||
time.sleep(1)
|
time.sleep(1)
|
||||||
processID = subprocess.check_output(
|
processID = subprocess.check_output(
|
||||||
psCmd, shell=True).decode("utf-8")
|
psCmd, shell=True).decode("utf-8")
|
||||||
|
|
|
@ -56,6 +56,7 @@
|
||||||
# unsupport ./test.sh -f tsim/dnode/redistribute_vgroup_replica3_v2.sim
|
# unsupport ./test.sh -f tsim/dnode/redistribute_vgroup_replica3_v2.sim
|
||||||
# unsupport ./test.sh -f tsim/dnode/redistribute_vgroup_replica3_v3.sim
|
# unsupport ./test.sh -f tsim/dnode/redistribute_vgroup_replica3_v3.sim
|
||||||
# unsupport ./test.sh -f tsim/dnode/vnode_clean.sim
|
# unsupport ./test.sh -f tsim/dnode/vnode_clean.sim
|
||||||
|
./test.sh -f tsim/dnode/use_dropped_dnode.sim
|
||||||
|
|
||||||
# ---- import
|
# ---- import
|
||||||
./test.sh -f tsim/import/basic.sim
|
./test.sh -f tsim/import/basic.sim
|
||||||
|
|
|
@ -1,18 +1,125 @@
|
||||||
system sh/stop_dnodes.sh
|
system sh/stop_dnodes.sh
|
||||||
system sh/deploy.sh -n dnode1 -i 1
|
system sh/deploy.sh -n dnode1 -i 1
|
||||||
system sh/cfg.sh -n dnode1 -c telemetryReporting -v 1
|
system sh/deploy.sh -n dnode2 -i 2
|
||||||
system sh/cfg.sh -n dnode1 -c telemetryInterval -v 1
|
system sh/deploy.sh -n dnode3 -i 3
|
||||||
system sh/cfg.sh -n dnode1 -c telemetryServer -v localhost
|
system sh/deploy.sh -n dnode4 -i 4
|
||||||
system sh/cfg.sh -n dnode1 -c telemetryPort -v 80
|
|
||||||
|
|
||||||
return
|
|
||||||
system sh/exec.sh -n dnode1 -s start
|
system sh/exec.sh -n dnode1 -s start
|
||||||
|
system sh/exec.sh -n dnode2 -s start
|
||||||
|
system sh/exec.sh -n dnode3 -s start
|
||||||
|
system sh/exec.sh -n dnode4 -s start
|
||||||
sql connect
|
sql connect
|
||||||
|
|
||||||
sql create database db
|
print =============== step1: create dnodes
|
||||||
sql create table db.tb (ts timestamp, i int)
|
sql create dnode $hostname port 7200
|
||||||
sql insert into db.tb values(now, 1)
|
sql create dnode $hostname port 7300
|
||||||
|
sql create dnode $hostname port 7400
|
||||||
|
|
||||||
|
$x = 0
|
||||||
|
step1:
|
||||||
|
$x = $x + 1
|
||||||
|
sleep 1000
|
||||||
|
if $x == 10 then
|
||||||
|
print ====> dnode not ready!
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
sql show dnodes
|
||||||
|
print ===> rows: $rows
|
||||||
|
print ===> $data00 $data01 $data02 $data03 $data04 $data05
|
||||||
|
print ===> $data10 $data11 $data12 $data13 $data14 $data15
|
||||||
|
print ===> $data20 $data21 $data22 $data23 $data24 $data25
|
||||||
|
print ===> $data30 $data31 $data32 $data33 $data24 $data35
|
||||||
|
if $rows != 4 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(1)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
if $data(2)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
if $data(3)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
if $data(4)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
|
||||||
|
print =============== step2: create database
|
||||||
|
sql create database db vgroups 1 replica 3
|
||||||
|
sql show databases
|
||||||
|
if $rows != 3 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(db)[4] != 3 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
|
||||||
|
sql show dnodes
|
||||||
|
if $data(2)[2] != 1 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(3)[2] != 1 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(4)[2] != 1 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
|
||||||
|
# vnodes
|
||||||
|
sql show dnodes
|
||||||
|
if $data(2)[2] != 1 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(3)[2] != 1 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(4)[2] != 1 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
|
||||||
|
# v1_dnode
|
||||||
|
$hasleader = 0
|
||||||
|
$x = 0
|
||||||
|
step2:
|
||||||
|
$x = $x + 1
|
||||||
|
sleep 1000
|
||||||
|
if $x == 20 then
|
||||||
|
print ====> dnode not ready!
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
sql show db.vgroups
|
||||||
|
print ===> $data00 $data01 $data02 $data03 $data04 $data05 $data06 $data07 $data08
|
||||||
|
if $data(2)[3] != 2 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(2)[5] != 3 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(2)[7] != 4 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(2)[4] == leader then
|
||||||
|
$hasleader = 1
|
||||||
|
endi
|
||||||
|
if $data(2)[6] == leader then
|
||||||
|
$hasleader = 1
|
||||||
|
endi
|
||||||
|
if $data(2)[8] == leader then
|
||||||
|
$hasleader = 1
|
||||||
|
endi
|
||||||
|
if $hasleader != 1 then
|
||||||
|
goto step2
|
||||||
|
endi
|
||||||
|
|
||||||
|
sql use db;
|
||||||
|
sql create table stb (ts timestamp, c int) tags (t int);
|
||||||
|
sql create table t0 using stb tags (0);
|
||||||
|
sql insert into t0 values(now, 1);
|
||||||
|
sql show db.stables;
|
||||||
|
sql show db.tables;
|
||||||
|
sql show db.vgroups;
|
||||||
|
|
||||||
|
return
|
||||||
print ======== start back
|
print ======== start back
|
||||||
run_back tmp/back.sim
|
run_back tmp/back.sim
|
||||||
|
|
||||||
|
|
|
@ -6,32 +6,21 @@ system sh/cfg.sh -n dnode1 -c monitorInterval -v 1
|
||||||
system sh/cfg.sh -n dnode1 -c monitorComp -v 1
|
system sh/cfg.sh -n dnode1 -c monitorComp -v 1
|
||||||
#system sh/cfg.sh -n dnode1 -c supportVnodes -v 128
|
#system sh/cfg.sh -n dnode1 -c supportVnodes -v 128
|
||||||
|
|
||||||
|
#system sh/cfg.sh -n dnode1 -c telemetryReporting -v 1
|
||||||
|
#system sh/cfg.sh -n dnode1 -c telemetryInterval -v 1
|
||||||
|
#system sh/cfg.sh -n dnode1 -c telemetryServer -v localhost
|
||||||
|
#system sh/cfg.sh -n dnode1 -c telemetryPort -v 80
|
||||||
|
|
||||||
system sh/exec.sh -n dnode1 -s start
|
system sh/exec.sh -n dnode1 -s start
|
||||||
sql connect
|
sql connect
|
||||||
|
|
||||||
print =============== show dnodes
|
print =============== show dnodes
|
||||||
sleep 2000
|
|
||||||
sql create database db vgroups 2;
|
sql create database db vgroups 2;
|
||||||
sql use db;
|
sql use db;
|
||||||
sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 binary(16)) comment "abd";
|
sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 binary(16)) comment "abd";
|
||||||
sleep 2000
|
|
||||||
|
|
||||||
print =============== create drop qnode 1
|
print =============== create drop qnode 1
|
||||||
sql create qnode on dnode 1
|
sql create qnode on dnode 1
|
||||||
sql create snode on dnode 1
|
sql create snode on dnode 1
|
||||||
sql create bnode on dnode 1
|
sql create bnode on dnode 1
|
||||||
|
|
||||||
return
|
|
||||||
print =============== restart
|
|
||||||
system sh/exec.sh -n dnode1 -s stop -x SIGINT
|
|
||||||
system sh/exec.sh -n dnode1 -s start
|
|
||||||
|
|
||||||
|
|
||||||
return
|
|
||||||
system sh/deploy.sh -n dnode2 -i 2
|
|
||||||
system sh/exec.sh -n dnode2 -s start
|
|
||||||
system sh/exec.sh -n dnode2 -s stop -x SIGINT
|
|
||||||
system sh/exec.sh -n dnode2 -s start
|
|
||||||
|
|
||||||
system sh/exec.sh -n dnode1 -s stop -x SIGINT
|
|
||||||
system sh/exec.sh -n dnode2 -s stop -x SIGINT
|
|
||||||
|
|
|
@ -111,15 +111,6 @@ if $hasleader != 1 then
|
||||||
goto step2
|
goto step2
|
||||||
endi
|
endi
|
||||||
|
|
||||||
# sql use db;
|
|
||||||
# sql create table stb (ts timestamp, c int) tags (t int);
|
|
||||||
# sql create table t0 using stb tags (0);
|
|
||||||
# sql insert into t0 values(now, 1);
|
|
||||||
# sql show db.stables;
|
|
||||||
# sql show db.tables;
|
|
||||||
# sql show db.vgroups;
|
|
||||||
return
|
|
||||||
|
|
||||||
sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 binary(16)) comment "abd"
|
sql create table db.stb (ts timestamp, c1 int, c2 binary(4)) tags(t1 int, t2 binary(16)) comment "abd"
|
||||||
sql create table db.ctb using db.stb tags(101, "102")
|
sql create table db.ctb using db.stb tags(101, "102")
|
||||||
sql insert into db.ctb values(now, 1, "2")
|
sql insert into db.ctb values(now, 1, "2")
|
||||||
|
|
|
@ -0,0 +1,133 @@
|
||||||
|
system sh/stop_dnodes.sh
|
||||||
|
system sh/deploy.sh -n dnode1 -i 1
|
||||||
|
system sh/deploy.sh -n dnode2 -i 2
|
||||||
|
system sh/deploy.sh -n dnode3 -i 3
|
||||||
|
system sh/exec.sh -n dnode1 -s start
|
||||||
|
system sh/exec.sh -n dnode2 -s start
|
||||||
|
system sh/exec.sh -n dnode3 -s start
|
||||||
|
sql connect
|
||||||
|
|
||||||
|
print =============== step1 create dnode2
|
||||||
|
sql create dnode $hostname port 7200
|
||||||
|
sql create dnode $hostname port 7300
|
||||||
|
|
||||||
|
$x = 0
|
||||||
|
step1:
|
||||||
|
$ = $x + 1
|
||||||
|
sleep 1000
|
||||||
|
if $x == 10 then
|
||||||
|
print ====> dnode not online!
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
sql show dnodes
|
||||||
|
print ===> $data00 $data01 $data02 $data03 $data04 $data05
|
||||||
|
print ===> $data10 $data11 $data12 $data13 $data14 $data15
|
||||||
|
print ===> $data20 $data21 $data22 $data23 $data24 $data25
|
||||||
|
if $rows != 3 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(1)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
if $data(2)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
if $data(3)[4] != ready then
|
||||||
|
goto step1
|
||||||
|
endi
|
||||||
|
|
||||||
|
print =============== step2 drop dnode 3
|
||||||
|
sql drop dnode 3
|
||||||
|
sql create dnode $hostname port 7300
|
||||||
|
|
||||||
|
$x = 0
|
||||||
|
step2:
|
||||||
|
$ = $x + 1
|
||||||
|
sleep 1000
|
||||||
|
if $x == 10 then
|
||||||
|
print ====> dnode not online!
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
sql show dnodes
|
||||||
|
print ===> $data00 $data01 $data02 $data03 $data04 $data05
|
||||||
|
print ===> $data10 $data11 $data12 $data13 $data14 $data15
|
||||||
|
print ===> $data20 $data21 $data22 $data23 $data24 $data25
|
||||||
|
print ===> $data30 $data31 $data32 $data33 $data34 $data35
|
||||||
|
if $rows != 3 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(1)[4] != ready then
|
||||||
|
goto step2
|
||||||
|
endi
|
||||||
|
if $data(2)[4] != ready then
|
||||||
|
goto step2
|
||||||
|
endi
|
||||||
|
if $data(3)[4] != null then
|
||||||
|
goto step2
|
||||||
|
endi
|
||||||
|
if $data(4)[4] != offline then
|
||||||
|
goto step2
|
||||||
|
endi
|
||||||
|
|
||||||
|
print =============== step3: create dnode 4
|
||||||
|
sleep 1000
|
||||||
|
system sh/exec.sh -n dnode3 -s stop -x SIGINT
|
||||||
|
system sh/deploy.sh -n dnode3 -i 3
|
||||||
|
system sh/exec.sh -n dnode3 -s start
|
||||||
|
|
||||||
|
$x = 0
|
||||||
|
step3:
|
||||||
|
$ = $x + 1
|
||||||
|
sleep 1000
|
||||||
|
if $x == 10 then
|
||||||
|
print ====> dnode not online!
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
sql show dnodes
|
||||||
|
print ===> $data00 $data01 $data02 $data03 $data04 $data05
|
||||||
|
print ===> $data10 $data11 $data12 $data13 $data14 $data15
|
||||||
|
print ===> $data20 $data21 $data22 $data23 $data24 $data25
|
||||||
|
print ===> $data30 $data31 $data32 $data33 $data34 $data35
|
||||||
|
if $rows != 3 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(1)[4] != ready then
|
||||||
|
goto step3
|
||||||
|
endi
|
||||||
|
if $data(2)[4] != ready then
|
||||||
|
goto step3
|
||||||
|
endi
|
||||||
|
if $data(3)[4] != null then
|
||||||
|
goto step3
|
||||||
|
endi
|
||||||
|
if $data(4)[4] != ready then
|
||||||
|
goto step3
|
||||||
|
endi
|
||||||
|
|
||||||
|
print =============== step4: create mnode 4
|
||||||
|
sql create mnode on dnode 4
|
||||||
|
|
||||||
|
$x = 0
|
||||||
|
step4:
|
||||||
|
$ = $x + 1
|
||||||
|
sleep 1000
|
||||||
|
if $x == 10 then
|
||||||
|
print ====> dnode not online!
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
sql show mnodes
|
||||||
|
print ===> $data00 $data01 $data02 $data03 $data04 $data05
|
||||||
|
print ===> $data10 $data11 $data12 $data13 $data14 $data15
|
||||||
|
if $rows != 2 then
|
||||||
|
return -1
|
||||||
|
endi
|
||||||
|
if $data(1)[2] != leader then
|
||||||
|
goto step4
|
||||||
|
endi
|
||||||
|
if $data(4)[2] != follower then
|
||||||
|
goto step4
|
||||||
|
endi
|
||||||
|
|
||||||
|
system sh/exec.sh -n dnode1 -s stop -x SIGINT
|
||||||
|
system sh/exec.sh -n dnode2 -s stop -x SIGINT
|
||||||
|
system sh/exec.sh -n dnode3 -s stop -x SIGINT
|
|
@ -152,7 +152,7 @@ endi
|
||||||
|
|
||||||
system_content sh/checkValgrind.sh -n dnode2
|
system_content sh/checkValgrind.sh -n dnode2
|
||||||
print cmd return result ----> [ $system_content ]
|
print cmd return result ----> [ $system_content ]
|
||||||
if $system_content > 4 then
|
if $system_content > 0 then
|
||||||
return -1
|
return -1
|
||||||
endi
|
endi
|
||||||
|
|
||||||
|
|
|
@ -56,6 +56,16 @@ class TDTestCase:
|
||||||
}
|
}
|
||||||
return numbers.get(value, 'other')
|
return numbers.get(value, 'other')
|
||||||
|
|
||||||
|
def getCacheModelNum(self,str):
|
||||||
|
numbers = {
|
||||||
|
"none" : 0,
|
||||||
|
"last_row" : 1,
|
||||||
|
"last_value" : 2,
|
||||||
|
"both" : 3
|
||||||
|
|
||||||
|
}
|
||||||
|
return numbers.get(str, 'other')
|
||||||
|
|
||||||
def prepare_datas(self):
|
def prepare_datas(self):
|
||||||
for i in range(4):
|
for i in range(4):
|
||||||
str = self.getCacheModelStr(i)
|
str = self.getCacheModelStr(i)
|
||||||
|
@ -69,7 +79,7 @@ class TDTestCase:
|
||||||
tdSql.execute(" insert into tb1 values(now , %d, %f)" %(k,k*10) )
|
tdSql.execute(" insert into tb1 values(now , %d, %f)" %(k,k*10) )
|
||||||
tdSql.execute(" insert into tb2 values(now , %d, %f)" %(k,k*10) )
|
tdSql.execute(" insert into tb2 values(now , %d, %f)" %(k,k*10) )
|
||||||
|
|
||||||
def check_cache_last_sets(self):
|
def check_cachemodel_sets(self):
|
||||||
|
|
||||||
|
|
||||||
# check cache_last value for database
|
# check cache_last value for database
|
||||||
|
@ -84,52 +94,54 @@ class TDTestCase:
|
||||||
# print(cache_last_value)
|
# print(cache_last_value)
|
||||||
if dbname in ["information_schema" , "performance_schema"]:
|
if dbname in ["information_schema" , "performance_schema"]:
|
||||||
continue
|
continue
|
||||||
cache_lasts[dbname]=cache_last_value
|
cache_lasts[dbname]=self.getCacheModelNum(cache_last_value)
|
||||||
|
|
||||||
|
|
||||||
# cache_last_set value
|
# cache_last_set value
|
||||||
for k , v in cache_lasts.items():
|
for k , v in cache_lasts.items():
|
||||||
|
|
||||||
if k=="testdb_"+str(v):
|
if k=="testdb_"+str(self.getCacheModelStr(v)):
|
||||||
tdLog.info(" database %s cache_last value check pass, value is %s "%(k,v) )
|
tdLog.info(" database %s cache_last value check pass, value is %s "%(k,self.getCacheModelStr(v)) )
|
||||||
else:
|
else:
|
||||||
tdLog.exit(" database %s cache_last value check fail, value is %s "%(k,v) )
|
tdLog.exit(" database %s cache_last value check fail, value is %s "%(k,self.getCacheModelStr(v)) )
|
||||||
|
|
||||||
# # check storage layer implementation
|
# # check storage layer implementation
|
||||||
|
|
||||||
|
|
||||||
# buildPath = self.getBuildPath()
|
buildPath = self.getBuildPath()
|
||||||
# if (buildPath == ""):
|
if (buildPath == ""):
|
||||||
# tdLog.exit("taosd not found!")
|
tdLog.exit("taosd not found!")
|
||||||
# else:
|
else:
|
||||||
# tdLog.info("taosd found in %s" % buildPath)
|
tdLog.info("taosd found in %s" % buildPath)
|
||||||
# dataPath = buildPath + "/../sim/dnode1/data"
|
dataPath = buildPath + "/../sim/dnode1/data"
|
||||||
# abs_vnodePath = os.path.abspath(dataPath)+"/vnode/"
|
abs_vnodePath = os.path.abspath(dataPath)+"/vnode/"
|
||||||
# tdLog.info("abs_vnodePath: %s" % abs_vnodePath)
|
tdLog.info("abs_vnodePath: %s" % abs_vnodePath)
|
||||||
|
|
||||||
# tdSql.query(" show dnodes ")
|
tdSql.query(" show dnodes ")
|
||||||
# dnode_id = tdSql.queryResult[0][0]
|
dnode_id = tdSql.queryResult[0][0]
|
||||||
|
|
||||||
# for dbname in cache_lasts.keys():
|
for dbname in cache_lasts.keys():
|
||||||
# print(dbname)
|
# print(dbname)
|
||||||
# tdSql.execute(" use %s" % dbname)
|
tdSql.execute(" use %s" % dbname)
|
||||||
# tdSql.query(" show vgroups ")
|
tdSql.query(" show vgroups ")
|
||||||
# vgroups_infos = tdSql.queryResult
|
vgroups_infos = tdSql.queryResult
|
||||||
# for vgroup_info in vgroups_infos:
|
for vgroup_info in vgroups_infos:
|
||||||
# vnode_json = abs_vnodePath + "/vnode" +f"{vgroup_info[0]}/" + "vnode.json"
|
vnode_json = abs_vnodePath + "/vnode" +f"{vgroup_info[0]}/" + "vnode.json"
|
||||||
# vnode_info_of_db = f"cat {vnode_json}"
|
vnode_info_of_db = f"cat {vnode_json}"
|
||||||
# vnode_info = subprocess.check_output(vnode_info_of_db, shell=True).decode("utf-8")
|
vnode_info = subprocess.check_output(vnode_info_of_db, shell=True).decode("utf-8")
|
||||||
# infoDict = json.loads(vnode_info)
|
infoDict = json.loads(vnode_info)
|
||||||
# vnode_json_of_dbname = f"{dnode_id}."+ dbname
|
vnode_json_of_dbname = f"{dnode_id}."+ dbname
|
||||||
# config = infoDict["config"]
|
config = infoDict["config"]
|
||||||
# if infoDict["config"]["dbname"] == vnode_json_of_dbname:
|
if infoDict["config"]["dbname"] == vnode_json_of_dbname:
|
||||||
# if "cachelast" in infoDict["config"]:
|
if "cacheLast" in infoDict["config"]:
|
||||||
# if int(infoDict["config"]["cachelast"]) != cache_lasts[dbname]:
|
if int(infoDict["config"]["cacheLast"]) != cache_lasts[dbname]:
|
||||||
# tdLog.exit("cachelast value is error in vnode.json of vnode%d "%(vgroup_info[0]))
|
tdLog.exit("cachemodel value is error in vnode.json of vnode%d "%(vgroup_info[0]))
|
||||||
# else:
|
else:
|
||||||
# tdLog.exit("cachelast not found in vnode.json of vnode%d "%(vgroup_info[0]))
|
tdLog.info("cachemodel value is success in vnode.json of vnode%d "%(vgroup_info[0]))
|
||||||
|
else:
|
||||||
|
tdLog.exit("cacheLast not found in vnode.json of vnode%d "%(vgroup_info[0]))
|
||||||
|
|
||||||
def restart_check_cache_last_sets(self):
|
def restart_check_cachemodel_sets(self):
|
||||||
|
|
||||||
for i in range(3):
|
for i in range(3):
|
||||||
tdSql.query("show dnodes")
|
tdSql.query("show dnodes")
|
||||||
|
@ -137,14 +149,14 @@ class TDTestCase:
|
||||||
tdDnodes.stop(index)
|
tdDnodes.stop(index)
|
||||||
tdDnodes.start(index)
|
tdDnodes.start(index)
|
||||||
time.sleep(3)
|
time.sleep(3)
|
||||||
self.check_cache_last_sets()
|
self.check_cachemodel_sets()
|
||||||
|
|
||||||
|
|
||||||
def run(self): # sourcery skip: extract-duplicate-method, remove-redundant-fstring
|
def run(self): # sourcery skip: extract-duplicate-method, remove-redundant-fstring
|
||||||
self.illegal_params()
|
self.illegal_params()
|
||||||
self.prepare_datas()
|
self.prepare_datas()
|
||||||
self.check_cache_last_sets()
|
self.check_cachemodel_sets()
|
||||||
self.restart_check_cache_last_sets()
|
self.restart_check_cachemodel_sets()
|
||||||
|
|
||||||
def stop(self):
|
def stop(self):
|
||||||
tdSql.close()
|
tdSql.close()
|
|
@ -24,7 +24,7 @@ class TDTestCase:
|
||||||
tdLog.debug("start to execute %s" % __file__)
|
tdLog.debug("start to execute %s" % __file__)
|
||||||
tdSql.init(conn.cursor())
|
tdSql.init(conn.cursor())
|
||||||
self.dbname = 'db'
|
self.dbname = 'db'
|
||||||
self.delaytime = 10
|
self.delaytime = 3
|
||||||
def get_database_info(self):
|
def get_database_info(self):
|
||||||
tdSql.query('select database()')
|
tdSql.query('select database()')
|
||||||
tdSql.checkData(0,0,None)
|
tdSql.checkData(0,0,None)
|
||||||
|
@ -42,12 +42,13 @@ class TDTestCase:
|
||||||
tdSql.checkData(0,0,version_info)
|
tdSql.checkData(0,0,version_info)
|
||||||
|
|
||||||
def get_server_status(self):
|
def get_server_status(self):
|
||||||
|
sleep(self.delaytime)
|
||||||
tdSql.query('select server_status()')
|
tdSql.query('select server_status()')
|
||||||
tdSql.checkData(0,0,1)
|
tdSql.checkData(0,0,1)
|
||||||
#!for bug
|
#!for bug
|
||||||
# tdDnodes.stoptaosd(1)
|
tdDnodes.stoptaosd(1)
|
||||||
# sleep(self.delaytime)
|
sleep(self.delaytime)
|
||||||
# tdSql.error('select server_status()')
|
tdSql.error('select server_status()')
|
||||||
|
|
||||||
def run(self):
|
def run(self):
|
||||||
self.get_database_info()
|
self.get_database_info()
|
||||||
|
|
|
@ -0,0 +1,190 @@
|
||||||
|
###################################################################
|
||||||
|
# Copyright (c) 2016 by TAOS Technologies, Inc.
|
||||||
|
# All rights reserved.
|
||||||
|
#
|
||||||
|
# This file is proprietary and confidential to TAOS Technologies.
|
||||||
|
# No part of this file may be reproduced, stored, transmitted,
|
||||||
|
# disclosed or used in any form or by any means other than as
|
||||||
|
# expressly provided by the written permission from Jianhui Tao
|
||||||
|
#
|
||||||
|
###################################################################
|
||||||
|
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import random
|
||||||
|
import string
|
||||||
|
|
||||||
|
from numpy import logspace
|
||||||
|
from util import constant
|
||||||
|
from util.log import *
|
||||||
|
from util.cases import *
|
||||||
|
from util.sql import *
|
||||||
|
from util.common import *
|
||||||
|
from util.sqlset import TDSetSql
|
||||||
|
|
||||||
|
class TDTestCase:
|
||||||
|
def init(self, conn, logSql):
|
||||||
|
tdLog.debug("start to execute %s" % __file__)
|
||||||
|
tdSql.init(conn.cursor(),logSql)
|
||||||
|
self.dbname = 'db_test'
|
||||||
|
self.setsql = TDSetSql()
|
||||||
|
self.ntbname = 'ntb'
|
||||||
|
self.rowNum = 10
|
||||||
|
self.tbnum = 20
|
||||||
|
self.ts = 1537146000000
|
||||||
|
self.binary_str = 'taosdata'
|
||||||
|
self.nchar_str = '涛思数据'
|
||||||
|
self.str_length = 20
|
||||||
|
self.column_dict = {
|
||||||
|
'col1': 'tinyint',
|
||||||
|
'col2': 'smallint',
|
||||||
|
'col3': 'int',
|
||||||
|
'col4': 'bigint',
|
||||||
|
'col5': 'tinyint unsigned',
|
||||||
|
'col6': 'smallint unsigned',
|
||||||
|
'col7': 'int unsigned',
|
||||||
|
'col8': 'bigint unsigned',
|
||||||
|
'col9': 'float',
|
||||||
|
'col10': 'double',
|
||||||
|
'col11': 'bool',
|
||||||
|
'col12': f'binary({self.str_length})',
|
||||||
|
'col13': f'nchar({self.str_length})',
|
||||||
|
|
||||||
|
}
|
||||||
|
self.tinyint_val = random.randint(constant.TINYINT_MIN,constant.TINYINT_MAX)
|
||||||
|
self.smallint_val = random.randint(constant.SMALLINT_MIN,constant.SMALLINT_MAX)
|
||||||
|
self.int_val = random.randint(constant.INT_MIN,constant.INT_MAX)
|
||||||
|
self.bigint_val = random.randint(constant.BIGINT_MIN,constant.BIGINT_MAX)
|
||||||
|
self.untingint_val = random.randint(constant.TINYINT_UN_MIN,constant.TINYINT_UN_MAX)
|
||||||
|
self.unsmallint_val = random.randint(constant.SMALLINT_UN_MIN,constant.SMALLINT_UN_MAX)
|
||||||
|
self.unint_val = random.randint(constant.INT_UN_MIN,constant.INT_MAX)
|
||||||
|
self.unbigint_val = random.randint(constant.BIGINT_UN_MIN,constant.BIGINT_UN_MAX)
|
||||||
|
self.float_val = random.uniform(constant.FLOAT_MIN,constant.FLOAT_MAX)
|
||||||
|
self.double_val = random.uniform(constant.DOUBLE_MIN*(1E-300),constant.DOUBLE_MAX*(1E-300))
|
||||||
|
self.bool_val = random.randint(0,100)%2
|
||||||
|
self.binary_val = tdCom.getLongName(random.randint(0,self.str_length))
|
||||||
|
self.nchar_val = tdCom.getLongName(random.randint(0,self.str_length))
|
||||||
|
self.base_data = {
|
||||||
|
'tinyint':self.tinyint_val,
|
||||||
|
'smallint':self.smallint_val,
|
||||||
|
'int':self.int_val,
|
||||||
|
'bigint':self.bigint_val,
|
||||||
|
'tinyint unsigned':self.untingint_val,
|
||||||
|
'smallint unsigned':self.unsmallint_val,
|
||||||
|
'int unsigned':self.unint_val,
|
||||||
|
'bigint unsigned':self.unbigint_val,
|
||||||
|
'bool':self.bool_val,
|
||||||
|
'float':self.float_val,
|
||||||
|
'double':self.double_val,
|
||||||
|
'binary':self.binary_val,
|
||||||
|
'nchar':self.nchar_val
|
||||||
|
}
|
||||||
|
def insert_base_data(self,col_type,tbname,rows,base_data):
|
||||||
|
for i in range(rows):
|
||||||
|
if col_type.lower() == 'tinyint':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["tinyint"]})')
|
||||||
|
elif col_type.lower() == 'smallint':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["smallint"]})')
|
||||||
|
elif col_type.lower() == 'int':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["int"]})')
|
||||||
|
elif col_type.lower() == 'bigint':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["bigint"]})')
|
||||||
|
elif col_type.lower() == 'tinyint unsigned':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["tinyint unsigned"]})')
|
||||||
|
elif col_type.lower() == 'smallint unsigned':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["smallint unsigned"]})')
|
||||||
|
elif col_type.lower() == 'int unsigned':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["int unsigned"]})')
|
||||||
|
elif col_type.lower() == 'bigint unsigned':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["bigint unsigned"]})')
|
||||||
|
elif col_type.lower() == 'bool':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["bool"]})')
|
||||||
|
elif col_type.lower() == 'float':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["float"]})')
|
||||||
|
elif col_type.lower() == 'double':
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts+i},{base_data["double"]})')
|
||||||
|
elif 'binary' in col_type.lower():
|
||||||
|
tdSql.execute(f'''insert into {tbname} values({self.ts+i},"{base_data['binary']}")''')
|
||||||
|
elif 'nchar' in col_type.lower():
|
||||||
|
tdSql.execute(f'''insert into {tbname} values({self.ts+i},"{base_data['nchar']}")''')
|
||||||
|
|
||||||
|
def delete_all_data(self,tbname,col_type,row_num,base_data,dbname):
|
||||||
|
tdSql.execute(f'delete from {tbname}')
|
||||||
|
tdSql.execute(f'flush database {dbname}')
|
||||||
|
tdSql.execute('reset query cache')
|
||||||
|
tdSql.query(f'select * from {tbname}')
|
||||||
|
tdSql.checkRows(0)
|
||||||
|
self.insert_base_data(col_type,tbname,row_num,base_data)
|
||||||
|
tdSql.execute(f'flush database {dbname}')
|
||||||
|
tdSql.execute('reset query cache')
|
||||||
|
tdSql.query(f'select * from {tbname}')
|
||||||
|
tdSql.checkRows(row_num)
|
||||||
|
def delete_one_row(self,tbname,column_type,column_name,base_data,dbname):
|
||||||
|
tdSql.execute(f'delete from {tbname} where ts={self.ts}')
|
||||||
|
tdSql.execute(f'flush database {dbname}')
|
||||||
|
tdSql.execute('reset query cache')
|
||||||
|
tdSql.query(f'select {column_name} from {tbname}')
|
||||||
|
tdSql.checkRows(self.rowNum-1)
|
||||||
|
tdSql.query(f'select {column_name} from {tbname} where ts={self.ts}')
|
||||||
|
tdSql.checkRows(0)
|
||||||
|
if 'binary' in column_type.lower():
|
||||||
|
tdSql.execute(f'''insert into {tbname} values({self.ts},"{base_data['binary']}")''')
|
||||||
|
elif 'nchar' in column_type.lower():
|
||||||
|
tdSql.execute(f'''insert into {tbname} values({self.ts},"{base_data['nchar']}")''')
|
||||||
|
else:
|
||||||
|
tdSql.execute(f'insert into {tbname} values({self.ts},{base_data[column_type]})')
|
||||||
|
tdSql.query(f'select {column_name} from {tbname} where ts={self.ts}')
|
||||||
|
if column_type.lower() == 'float' or column_type.lower() == 'double':
|
||||||
|
if abs(tdSql.queryResult[0][0] - base_data[column_type]) / base_data[column_type] <= 0.0001:
|
||||||
|
tdSql.checkEqual(tdSql.queryResult[0][0],tdSql.queryResult[0][0])
|
||||||
|
else:
|
||||||
|
tdLog.exit(f'{column_type} data check failure')
|
||||||
|
elif 'binary' in column_type.lower():
|
||||||
|
tdSql.checkEqual(tdSql.queryResult[0][0],base_data['binary'])
|
||||||
|
elif 'nchar' in column_type.lower():
|
||||||
|
tdSql.checkEqual(tdSql.queryResult[0][0],base_data['nchar'])
|
||||||
|
else:
|
||||||
|
tdSql.checkEqual(tdSql.queryResult[0][0],base_data[column_type])
|
||||||
|
|
||||||
|
def delete_rows(self):
|
||||||
|
|
||||||
|
|
||||||
|
pass
|
||||||
|
def delete_error(self,tbname,column_name,column_type,base_data):
|
||||||
|
for error_list in ['',f'ts = {self.ts} and',f'ts = {self.ts} or']:
|
||||||
|
if 'binary' in column_type.lower():
|
||||||
|
tdSql.error(f'''delete from {tbname} where {error_list} {column_name} ="{base_data['binary']}"''')
|
||||||
|
elif 'nchar' in column_type.lower():
|
||||||
|
tdSql.error(f'''delete from {tbname} where {error_list} {column_name} ="{base_data['nchar']}"''')
|
||||||
|
else:
|
||||||
|
tdSql.error('delete from {tbname} where {error_list} {column_name} = {base_data[column_type]}')
|
||||||
|
|
||||||
|
def delete_data_ntb(self):
|
||||||
|
tdSql.execute(f'create database if not exists {self.dbname}')
|
||||||
|
tdSql.execute(f'use {self.dbname}')
|
||||||
|
for col_name,col_type in self.column_dict.items():
|
||||||
|
tdSql.execute(f'create table {self.ntbname} (ts timestamp,{col_name} {col_type})')
|
||||||
|
self.insert_base_data(col_type,self.ntbname,self.rowNum,self.base_data)
|
||||||
|
self.delete_one_row(self.ntbname,col_type,col_name,self.base_data,self.dbname)
|
||||||
|
self.delete_all_data(self.ntbname,col_type,self.rowNum,self.base_data,self.dbname)
|
||||||
|
self.delete_error(self.ntbname,col_name,col_type,self.base_data)
|
||||||
|
for i in range(self.rowNum):
|
||||||
|
tdSql.execute(f'delete from {self.ntbname} where ts>{self.ts+i}')
|
||||||
|
tdSql.execute(f'flush database {self.dbname}')
|
||||||
|
tdSql.execute('reset query cache')
|
||||||
|
tdSql.query(f'select {col_name} from {self.ntbname}')
|
||||||
|
tdSql.checkRows(i+1)
|
||||||
|
self.insert_base_data(col_type,self.ntbname,self.rowNum,self.base_data)
|
||||||
|
|
||||||
|
tdSql.execute(f'drop table {self.ntbname}')
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
self.delete_data_ntb()
|
||||||
|
|
||||||
|
|
||||||
|
def stop(self):
|
||||||
|
tdSql.close()
|
||||||
|
tdLog.success("%s successfully executed" % __file__)
|
||||||
|
|
||||||
|
tdCases.addWindows(__file__, TDTestCase())
|
||||||
|
tdCases.addLinux(__file__, TDTestCase())
|
|
@ -0,0 +1,49 @@
|
||||||
|
# -*- coding: utf-8 -*-
|
||||||
|
|
||||||
|
import sys
|
||||||
|
from util.log import *
|
||||||
|
from util.cases import *
|
||||||
|
from util.sql import *
|
||||||
|
import threading
|
||||||
|
|
||||||
|
|
||||||
|
class TDTestCase:
|
||||||
|
def init(self, conn, logSql):
|
||||||
|
tdLog.debug("start to execute %s" % __file__)
|
||||||
|
tdSql.init(conn.cursor(), logSql)
|
||||||
|
|
||||||
|
def genMultiThreadSeq(self, sql_list):
|
||||||
|
tlist = list()
|
||||||
|
for insert_sql in sql_list:
|
||||||
|
t = threading.Thread(target=tdSql.execute, args=(insert_sql,))
|
||||||
|
tlist.append(t)
|
||||||
|
return tlist
|
||||||
|
|
||||||
|
def multiThreadRun(self, tlist):
|
||||||
|
for t in tlist:
|
||||||
|
t.start()
|
||||||
|
for t in tlist:
|
||||||
|
t.join()
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
tdSql.prepare()
|
||||||
|
tdSql.execute('create database if not exists test;')
|
||||||
|
tdSql.execute('create table test.stb (ts timestamp, c11 int, c12 float ) TAGS(t11 int, t12 int );')
|
||||||
|
tdSql.execute('create table test.tb using test.stb TAGS (1, 1);')
|
||||||
|
sql_list = list()
|
||||||
|
for i in range(5):
|
||||||
|
sql = f'insert into test.tb values (now-{i}m, {i}, {i});'
|
||||||
|
sql_list.append(sql)
|
||||||
|
sql_list.append(f'drop database test;')
|
||||||
|
tlist = self.genMultiThreadSeq(sql_list)
|
||||||
|
self.multiThreadRun(tlist)
|
||||||
|
tdSql.query(f'show databases')
|
||||||
|
|
||||||
|
|
||||||
|
def stop(self):
|
||||||
|
tdSql.close()
|
||||||
|
tdLog.success("%s successfully executed" % __file__)
|
||||||
|
|
||||||
|
|
||||||
|
tdCases.addWindows(__file__, TDTestCase())
|
||||||
|
tdCases.addLinux(__file__, TDTestCase())
|
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue