xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch 'develop' into feature/TD-3295

This commit is contained in:
dapan1121 2021-03-19 14:43:00 +08:00
parent 647a0e9919 31e5c26bdd
commit a4eb005385
142 changed files with 5810 additions and 4190 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
Jenkinsfile vendored
View File

@ -42,15 +42,15 @@ def pre_test(){
cd ${WKC}
git checkout develop
git reset --hard HEAD~10 >/dev/null
git pull
git pull >/dev/null
git fetch origin +refs/pull/${CHANGE_ID}/merge
git checkout -qf FETCH_HEAD
git --no-pager diff --name-only FETCH_HEAD $(git merge-base FETCH_HEAD develop)|grep -v -E '.*md|//src//connector|Jenkinsfile' || exit 0
git --no-pager diff --name-only FETCH_HEAD $(git merge-base FETCH_HEAD develop)|grep -v -E '.*md|//src//connector|Jenkinsfile'
find ${WKC}/tests/pytest -name \'*\'.sql -exec rm -rf {} \\;
cd ${WK}
git reset --hard HEAD~10
git checkout develop
git pull
git pull >/dev/null
cd ${WK}
export TZ=Asia/Harbin
date

39
README.md
View File

@ -160,39 +160,42 @@ mkdir debug && cd debug
cmake .. && cmake --build .
```
# Quick Run
# Quick Run
To quickly start a TDengine server after building, run the command below in terminal:
```bash
./build/bin/taosd -c test/cfg
```
In another terminal, use the TDengine shell to connect the server:
```bash
./build/bin/taos -c test/cfg
```
option "-c test/cfg" specifies the system configuration file directory.
# Installing
After building successfully, TDengine can be installed by:
```bash
make install
sudo make install
```
Users can find more information about directories installed on the system in the [directory and files](https://www.taosdata.com/en/documentation/administrator/#Directory-and-Files) section. It should be noted that installing from source code does not configure service management for TDengine.
Users can find more information about directories installed on the system in the [directory and files](https://www.taosdata.com/en/documentation/administrator/#Directory-and-Files) section. Since version 2.0, installing from source code will also configure service management for TDengine.
Users can also choose to [install from packages](https://www.taosdata.com/en/getting-started/#Install-from-Package) for it.
To start the service after installation, in a terminal, use:
```cmd
taosd
```bash
sudo systemctl start taosd
```
Then users can use the [TDengine shell](https://www.taosdata.com/en/getting-started/#TDengine-Shell) to connect the TDengine server. In a terminal, use:
```cmd
```bash
taos
```
If TDengine shell connects the server successfully, welcome messages and version info are printed. Otherwise, an error message is shown.
## Quick Run
If you don't want to run TDengine as a service, you can run it in current shell. For example, to quickly start a TDengine server after building, run the command below in terminal:
```bash
./build/bin/taosd -c test/cfg
```
In another terminal, use the TDengine shell to connect the server:
```bash
./build/bin/taos -c test/cfg
```
option "-c test/cfg" specifies the system configuration file directory.
# Try TDengine
It is easy to run SQL commands from TDengine shell which is the same as other SQL databases.
```sql

2
cmake/install.inc
View File

@ -32,7 +32,7 @@ ELSEIF (TD_WINDOWS)
#INSTALL(TARGETS taos RUNTIME DESTINATION driver)
#INSTALL(TARGETS shell RUNTIME DESTINATION .)
IF (TD_MVN_INSTALLED)
INSTALL(FILES ${LIBRARY_OUTPUT_PATH}/taos-jdbcdriver-2.0.21-dist.jar DESTINATION connector/jdbc)
INSTALL(FILES ${LIBRARY_OUTPUT_PATH}/taos-jdbcdriver-2.0.22-dist.jar DESTINATION connector/jdbc)
ENDIF ()
ELSEIF (TD_DARWIN)
SET(TD_MAKE_INSTALL_SH "${TD_COMMUNITY_DIR}/packaging/tools/make_install.sh")

2
documentation20/cn/03.architecture/docs.md
View File

@ -145,7 +145,7 @@ TDengine 建议用数据采集点的名字(如上表中的D1001)来做表名。
在TDengine的设计里，**表用来代表一个具体的数据采集点，超级表用来代表一组相同类型的数据采集点集合**。当为某个具体数据采集点创建表时，用户使用超级表的定义做模板，同时指定该具体采集点（表）的标签值。与传统的关系型数据库相比，表（一个数据采集点）是带有静态标签的，而且这些标签可以事后增加、删除、修改。**一张超级表包含有多张表，这些表具有相同的时序数据schema，但带有不同的标签值**。
当对多个具有相同数据类型的数据采集点进行聚合操作时，TDengine将先把满足标签过滤条件的表从超级表的中查找出来，然后再扫描这些表的时序数据，进行聚合操作，这样能将需要扫描的数据集大幅减少，从而大幅提高聚合计算的性能。
当对多个具有相同数据类型的数据采集点进行聚合操作时，TDengine会先把满足标签过滤条件的表从超级表中找出来，然后再扫描这些表的时序数据，进行聚合操作，这样需要扫描的数据集会大幅减少，从而显著提高聚合计算的性能。
## <a class="anchor" id="cluster"></a>集群与基本逻辑单元

2
documentation20/cn/11.administrator/docs.md
View File

@ -14,6 +14,8 @@ Memory Size = maxVgroupsPerDb * (blocks * cache + 10MB) + numOfTables * (tagSize
示例：假设是 4 核机器，cache 是缺省大小 16M, blocks 是缺省值 6，假设有 10 万张表，标签总长度是 256 字节，则总的内存需求为：4 \* (16 \* 6 + 10) + 100000 \* (0.25 + 0.5) / 1000 = 499M。
注意：从这个公式计算得到的内存容量，应理解为系统的“必要需求”，而不是“内存总数”。在实际运行的生产系统中，由于操作系统缓存、资源管理调度等方面的需要，内存规划应当在计算结果的基础上保留一定冗余，以维持系统状态和系统性能的稳定性。
实际运行的系统往往会根据数据特点的不同，将数据存放在不同的 DB 里。因此做规划时，也需要考虑。
如果内存充裕，可以加大 Blocks 的配置，这样更多数据将保存在内存里，提高查询速度。

3
documentation20/cn/12.taos-sql/docs.md
View File

@ -125,7 +125,7 @@ TDengine缺省的时间戳是毫秒精度，但通过修改配置参数enableMic
```mysql
ALTER DATABASE db_name CACHELAST 0;
```
CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11 版本开始支持）
CACHELAST 参数控制是否在内存中缓存数据子表的 last_row。缺省值为 0，取值范围 [0, 1]。其中 0 表示不启用、1 表示启用。（从 2.0.11 版本开始支持，修改后需要重启服务器生效。）
**Tips**: 以上所有参数修改后都可以用show databases来确认是否修改成功。
@ -332,6 +332,7 @@ TDengine缺省的时间戳是毫秒精度，但通过修改配置参数enableMic
INSERT INTO tb_name VALUES (field1_value1, ...) (field1_value2, ...) ...;
```
向表tb_name中插入多条记录
**注意**：在使用“插入多条记录”方式写入数据时，不能把第一列的时间戳取值都设为now，否则会导致语句中的多条记录使用相同的时间戳，于是就可能出现相互覆盖以致这些数据行无法全部被正确保存。
- **按指定的列插入多条记录**
```mysql

10
documentation20/cn/13.faq/docs.md
View File

@ -16,13 +16,13 @@
## 1. TDengine2.0之前的版本升级到2.0及以上的版本应该注意什么？☆☆☆
2.0版本在之前版本的基础上，进行了完全的重构，配置文件和数据文件是不兼容的。在升级之前务必进行如下操作：
2.0版在之前版本的基础上，进行了完全的重构，配置文件和数据文件是不兼容的。在升级之前务必进行如下操作：
1. 删除配置文件，执行 <code> sudo rm -rf /etc/taos/taos.cfg </code>
2. 删除日志文件，执行 <code> sudo rm -rf /var/log/taos/ </code>
3. 确保数据已经不再需要的前提下，删除数据文件，执行 <code> sudo rm -rf /var/lib/taos/ </code>
1. 删除配置文件，执行 `sudo rm -rf /etc/taos/taos.cfg`
2. 删除日志文件，执行 `sudo rm -rf /var/log/taos/`
3. 确保数据已经不再需要的前提下，删除数据文件，执行 `sudo rm -rf /var/lib/taos/`
4. 安装最新稳定版本的 TDengine
5. 如果数据需要迁移数据或者数据文件损坏，请联系涛思数据官方技术支持团队，进行协助解决
5. 如果需要迁移数据或者数据文件损坏，请联系涛思数据官方技术支持团队，进行协助解决
## 2. Windows平台下JDBCDriver找不到动态链接库，怎么办？

4
packaging/tools/remove.sh
View File

@ -213,10 +213,10 @@ fi
if echo $osinfo | grep -qwi "ubuntu" ; then
# echo "this is ubuntu system"
${csudo} rm -f /var/lib/dpkg/info/tdengine* || :
${csudo} dpkg --force-all -P tdengine || :
elif echo $osinfo | grep -qwi "debian" ; then
# echo "this is debian system"
${csudo} rm -f /var/lib/dpkg/info/tdengine* || :
${csudo} dpkg --force-all -P tdengine || :
elif echo $osinfo | grep -qwi "centos" ; then
# echo "this is centos system"
${csudo} rpm -e --noscripts tdengine || :

2
src/balance/src/bnMain.c
View File

@ -425,7 +425,7 @@ static bool bnMonitorVgroups() {
while (1) {
pIter = mnodeGetNextVgroup(pIter, &pVgroup);
if (pVgroup == NULL) break;
if (pVgroup == NULL || pVgroup->pDb == NULL) break;
int32_t dbReplica = pVgroup->pDb->cfg.replications;
int32_t vgReplica = pVgroup->numOfVnodes;

4
src/client/src/TSDBJNIConnector.c
View File

@ -481,15 +481,19 @@ JNIEXPORT jint JNICALL Java_com_taosdata_jdbc_TSDBJNIConnector_fetchRowImp(JNIEn
case TSDB_DATA_TYPE_BOOL:
(*env)->CallVoidMethod(env, rowobj, g_rowdataSetBooleanFp, i, (jboolean)(*((char *)row[i]) == 1));
break;
case TSDB_DATA_TYPE_UTINYINT:
case TSDB_DATA_TYPE_TINYINT:
(*env)->CallVoidMethod(env, rowobj, g_rowdataSetByteFp, i, (jbyte) * ((int8_t *)row[i]));
break;
case TSDB_DATA_TYPE_USMALLINT:
case TSDB_DATA_TYPE_SMALLINT:
(*env)->CallVoidMethod(env, rowobj, g_rowdataSetShortFp, i, (jshort) * ((int16_t *)row[i]));
break;
case TSDB_DATA_TYPE_UINT:
case TSDB_DATA_TYPE_INT:
(*env)->CallVoidMethod(env, rowobj, g_rowdataSetIntFp, i, (jint) * (int32_t *)row[i]);
break;
case TSDB_DATA_TYPE_UBIGINT:
case TSDB_DATA_TYPE_BIGINT:
(*env)->CallVoidMethod(env, rowobj, g_rowdataSetLongFp, i, (jlong) * ((int64_t *)row[i]));
break;

7
src/client/src/tscAsync.c
View File

@ -281,7 +281,7 @@ void tscQueueAsyncError(void(*fp), void *param, int32_t code) {
}
static void tscAsyncResultCallback(SSchedMsg *pMsg) {
SSqlObj* pSql = pMsg->ahandle;
SSqlObj* pSql = (SSqlObj*)taosAcquireRef(tscObjRef, (int64_t)pMsg->ahandle);
if (pSql == NULL || pSql->signature != pSql) {
tscDebug("%p SqlObj is freed, not add into queue async res", pSql);
return;
@ -292,23 +292,26 @@ static void tscAsyncResultCallback(SSchedMsg *pMsg) {
SSqlRes *pRes = &pSql->res;
if (pSql->fp == NULL || pSql->fetchFp == NULL){
taosReleaseRef(tscObjRef, pSql->self);
return;
}
pSql->fp = pSql->fetchFp;
(*pSql->fp)(pSql->param, pSql, pRes->code);
taosReleaseRef(tscObjRef, pSql->self);
}
void tscAsyncResultOnError(SSqlObj* pSql) {
SSchedMsg schedMsg = {0};
schedMsg.fp = tscAsyncResultCallback;
schedMsg.ahandle = pSql;
schedMsg.ahandle = (void *)pSql->self;
schedMsg.thandle = (void *)1;
schedMsg.msg = 0;
taosScheduleTask(tscQhandle, &schedMsg);
}
int tscSendMsgToServer(SSqlObj *pSql);
void tscTableMetaCallBack(void *param, TAOS_RES *res, int code) {

2
src/client/src/tscParseInsert.c
View File

@ -394,7 +394,7 @@ static int32_t tsCheckTimestamp(STableDataBlocks *pDataBlocks, const char *start
TSKEY k = *(TSKEY *)start;
if (k == 0) {
if (k == INT64_MIN) {
if (pDataBlocks->tsSource == TSDB_USE_CLI_TS) {
return -1;
} else if (pDataBlocks->tsSource == -1) {

88
src/client/src/tscSQLParser.c
View File

@ -2223,7 +2223,7 @@ int32_t addExprAndResultField(SSqlCmd* pCmd, SQueryInfo* pQueryInfo, int32_t col
bool multiColOutput = taosArrayGetSize(pItem->pNode->pParam) > 1;
setResultColName(name, pItem, cvtFunc.originFuncId, &pParamElem->pNode->colInfo, multiColOutput);
if (setExprInfoForFunctions(pCmd, pQueryInfo, pSchema, cvtFunc, name, colIndex + i, &index, finalResult) != 0) {
if (setExprInfoForFunctions(pCmd, pQueryInfo, pSchema, cvtFunc, name, colIndex++, &index, finalResult) != 0) {
return TSDB_CODE_TSC_INVALID_SQL;
}
}
@ -3092,18 +3092,26 @@ static int32_t doExtractColumnFilterInfo(SSqlCmd* pCmd, SQueryInfo* pQueryInfo,
}
int32_t retVal = TSDB_CODE_SUCCESS;
int32_t bufLen = 0;
if (IS_NUMERIC_TYPE(pRight->value.nType)) {
bufLen = 60;
} else {
bufLen = pRight->value.nLen + 1;
}
if (pExpr->tokenId == TK_LE || pExpr->tokenId == TK_LT) {
retVal = tVariantDump(&pRight->value, (char*)&pColumnFilter->upperBndd, colType, false);
// TK_GT,TK_GE,TK_EQ,TK_NE are based on the pColumn->lowerBndd
} else if (colType == TSDB_DATA_TYPE_BINARY) {
pColumnFilter->pz = (int64_t)calloc(1, pRight->value.nLen + TSDB_NCHAR_SIZE);
pColumnFilter->pz = (int64_t)calloc(1, bufLen * TSDB_NCHAR_SIZE);
pColumnFilter->len = pRight->value.nLen;
retVal = tVariantDump(&pRight->value, (char*)pColumnFilter->pz, colType, false);
} else if (colType == TSDB_DATA_TYPE_NCHAR) {
// pRight->value.nLen + 1 is larger than the actual nchar string length
pColumnFilter->pz = (int64_t)calloc(1, (pRight->value.nLen + 1) * TSDB_NCHAR_SIZE);
pColumnFilter->pz = (int64_t)calloc(1, bufLen * TSDB_NCHAR_SIZE);
retVal = tVariantDump(&pRight->value, (char*)pColumnFilter->pz, colType, false);
size_t len = twcslen((wchar_t*)pColumnFilter->pz);
pColumnFilter->len = len * TSDB_NCHAR_SIZE;
@ -4230,13 +4238,21 @@ static int32_t validateTagCondExpr(SSqlCmd* pCmd, tExprNode *p) {
}
int32_t retVal = TSDB_CODE_SUCCESS;
int32_t bufLen = 0;
if (IS_NUMERIC_TYPE(vVariant->nType)) {
bufLen = 60; // The maximum length of string that a number is converted to.
} else {
bufLen = vVariant->nLen + 1;
}
if (schemaType == TSDB_DATA_TYPE_BINARY) {
char *tmp = calloc(1, vVariant->nLen + TSDB_NCHAR_SIZE);
char *tmp = calloc(1, bufLen * TSDB_NCHAR_SIZE);
retVal = tVariantDump(vVariant, tmp, schemaType, false);
free(tmp);
} else if (schemaType == TSDB_DATA_TYPE_NCHAR) {
// pRight->value.nLen + 1 is larger than the actual nchar string length
char *tmp = calloc(1, (vVariant->nLen + 1) * TSDB_NCHAR_SIZE);
char *tmp = calloc(1, bufLen * TSDB_NCHAR_SIZE);
retVal = tVariantDump(vVariant, tmp, schemaType, false);
free(tmp);
} else {
@ -6568,13 +6584,13 @@ int32_t doCheckForStream(SSqlObj* pSql, SSqlInfo* pInfo) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg1);
}
SArray* pSrcMeterName = pInfo->pCreateTableInfo->pSelect->from;
if (pSrcMeterName == NULL || taosArrayGetSize(pSrcMeterName) == 0) {
SFromInfo* pFromInfo = pInfo->pCreateTableInfo->pSelect->from;
if (pFromInfo == NULL || taosArrayGetSize(pFromInfo->tableList) == 0) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg6);
}
tVariantListItem* p1 = taosArrayGet(pSrcMeterName, 0);
SStrToken srcToken = {.z = p1->pVar.pz, .n = p1->pVar.nLen, .type = TK_STRING};
STableNamePair* p1 = taosArrayGet(pFromInfo->tableList, 0);
SStrToken srcToken = {.z = p1->name.z, .n = p1->name.n, .type = TK_STRING};
if (tscValidateName(&srcToken) != TSDB_CODE_SUCCESS) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg1);
}
@ -6692,7 +6708,7 @@ static int32_t checkQueryRangeForFill(SSqlCmd* pCmd, SQueryInfo* pQueryInfo) {
}
int32_t doValidateSqlNode(SSqlObj* pSql, SQuerySqlNode* pQuerySqlNode, int32_t index) {
assert(pQuerySqlNode != NULL && (pQuerySqlNode->from == NULL || taosArrayGetSize(pQuerySqlNode->from) > 0));
assert(pQuerySqlNode != NULL && (pQuerySqlNode->from == NULL || taosArrayGetSize(pQuerySqlNode->from->tableList) > 0));
const char* msg0 = "invalid table name";
const char* msg1 = "point interpolation query needs timestamp";
@ -6701,6 +6717,7 @@ int32_t doValidateSqlNode(SSqlObj* pSql, SQuerySqlNode* pQuerySqlNode, int32_t i
const char* msg5 = "too many columns in selection clause";
const char* msg6 = "too many tables in from clause";
const char* msg7 = "invalid table alias name";
const char* msg8 = "alias name too long";
int32_t code = TSDB_CODE_SUCCESS;
@ -6732,7 +6749,7 @@ int32_t doValidateSqlNode(SSqlObj* pSql, SQuerySqlNode* pQuerySqlNode, int32_t i
return doLocalQueryProcess(pCmd, pQueryInfo, pQuerySqlNode);
}
size_t fromSize = taosArrayGetSize(pQuerySqlNode->from);
size_t fromSize = taosArrayGetSize(pQuerySqlNode->from->tableList);
if (fromSize > TSDB_MAX_JOIN_TABLE_NUM * 2) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg6);
}
@ -6740,59 +6757,60 @@ int32_t doValidateSqlNode(SSqlObj* pSql, SQuerySqlNode* pQuerySqlNode, int32_t i
pQueryInfo->command = TSDB_SQL_SELECT;
// set all query tables, which are maybe more than one.
for (int32_t i = 0; i < fromSize; ) {
tVariantListItem* item = taosArrayGet(pQuerySqlNode->from, i);
tVariant* pTableItem = &item->pVar;
for (int32_t i = 0; i < fromSize; ++i) {
STableNamePair* item = taosArrayGet(pQuerySqlNode->from->tableList, i);
SStrToken* pTableItem = &item->name;
if (pTableItem->nType != TSDB_DATA_TYPE_BINARY) {
if (pTableItem->type != TSDB_DATA_TYPE_BINARY) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg0);
}
pTableItem->nLen = strdequote(pTableItem->pz);
tscDequoteAndTrimToken(pTableItem);
SStrToken tableName = {.z = pTableItem->pz, .n = pTableItem->nLen, .type = TK_STRING};
SStrToken tableName = {.z = pTableItem->z, .n = pTableItem->n, .type = TK_STRING};
if (tscValidateName(&tableName) != TSDB_CODE_SUCCESS) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg0);
}
if (pQueryInfo->numOfTables <= i/2) { // more than one table
if (pQueryInfo->numOfTables <= i) { // more than one table
tscAddEmptyMetaInfo(pQueryInfo);
}
STableMetaInfo* pTableMetaInfo1 = tscGetMetaInfo(pQueryInfo, i/2);
SStrToken t = {.type = TSDB_DATA_TYPE_BINARY, .n = pTableItem->nLen, .z = pTableItem->pz};
code = tscSetTableFullName(pTableMetaInfo1, &t, pSql);
STableMetaInfo* pTableMetaInfo1 = tscGetMetaInfo(pQueryInfo, i);
code = tscSetTableFullName(pTableMetaInfo1, pTableItem, pSql);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
tVariantListItem* p1 = taosArrayGet(pQuerySqlNode->from, i + 1);
if (p1->pVar.nType != TSDB_DATA_TYPE_BINARY) {
SStrToken* aliasName = &item->aliasName;
if (TPARSER_HAS_TOKEN(*aliasName)) {
if (aliasName->type != TSDB_DATA_TYPE_BINARY) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg7);
}
SStrToken aliasName = {.z = p1->pVar.pz, .n = p1->pVar.nLen, .type = TK_STRING};
if (tscValidateName(&aliasName) != TSDB_CODE_SUCCESS) {
tscDequoteAndTrimToken(aliasName);
SStrToken aliasName1 = {.z = aliasName->z, .n = aliasName->n, .type = TK_STRING};
if (tscValidateName(&aliasName1) != TSDB_CODE_SUCCESS) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg7);
}
// has no table alias name
if (memcmp(pTableItem->pz, p1->pVar.pz, p1->pVar.nLen) == 0) {
strncpy(pTableMetaInfo1->aliasName, tNameGetTableName(&pTableMetaInfo1->name), tListLen(pTableMetaInfo->aliasName));
if (aliasName1.n >= TSDB_TABLE_NAME_LEN) {
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), msg8);
}
strncpy(pTableMetaInfo1->aliasName, aliasName1.z, aliasName1.n);
} else {
tstrncpy(pTableMetaInfo1->aliasName, p1->pVar.pz, sizeof(pTableMetaInfo1->aliasName));
strncpy(pTableMetaInfo1->aliasName, tNameGetTableName(&pTableMetaInfo1->name), tListLen(pTableMetaInfo1->aliasName));
}
code = tscGetTableMeta(pSql, pTableMetaInfo1);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
i += 2;
}
assert(pQueryInfo->numOfTables == taosArrayGetSize(pQuerySqlNode->from) / 2);
assert(pQueryInfo->numOfTables == taosArrayGetSize(pQuerySqlNode->from->tableList));
bool isSTable = false;
if (UTIL_TABLE_IS_SUPER_TABLE(pTableMetaInfo)) {
@ -6826,12 +6844,12 @@ int32_t doValidateSqlNode(SSqlObj* pSql, SQuerySqlNode* pQuerySqlNode, int32_t i
pQueryInfo->window.ekey = pQueryInfo->window.ekey / 1000;
}
} else { // set the time rang
if (taosArrayGetSize(pQuerySqlNode->from) > 2) { // it is a join query, no wher clause is not allowed.
if (taosArrayGetSize(pQuerySqlNode->from->tableList) > 1) { // it is a join query, no where clause is not allowed.
return invalidSqlErrMsg(tscGetErrorMsgPayload(pCmd), "condition missing for join query ");
}
}
int32_t joinQuery = (pQuerySqlNode->from != NULL && taosArrayGetSize(pQuerySqlNode->from) > 2);
int32_t joinQuery = (pQuerySqlNode->from != NULL && taosArrayGetSize(pQuerySqlNode->from->tableList) > 1);
int32_t timeWindowQuery =
(TPARSER_HAS_TOKEN(pQuerySqlNode->interval.interval) || TPARSER_HAS_TOKEN(pQuerySqlNode->sessionVal.gap));

6
src/client/src/tscServer.c
View File

@ -520,7 +520,7 @@ int tscBuildFetchMsg(SSqlObj *pSql, SSqlInfo *pInfo) {
assert(pVgroupInfo->vgroups[vgIndex].vgId > 0 && vgIndex < pTableMetaInfo->vgroupList->numOfVgroups);
pRetrieveMsg->header.vgId = htonl(pVgroupInfo->vgroups[vgIndex].vgId);
tscDebug("%p build fetch msg from vgId:%d, vgIndex:%d", pSql, pVgroupInfo->vgroups[vgIndex].vgId, vgIndex);
tscDebug("%p build fetch msg from vgId:%d, vgIndex:%d, qhandle:%" PRIX64, pSql, pVgroupInfo->vgroups[vgIndex].vgId, vgIndex, pSql->res.qhandle);
} else {
int32_t numOfVgroups = (int32_t)taosArrayGetSize(pTableMetaInfo->pVgroupTables);
assert(vgIndex >= 0 && vgIndex < numOfVgroups);
@ -528,12 +528,12 @@ int tscBuildFetchMsg(SSqlObj *pSql, SSqlInfo *pInfo) {
SVgroupTableInfo* pTableIdList = taosArrayGet(pTableMetaInfo->pVgroupTables, vgIndex);
pRetrieveMsg->header.vgId = htonl(pTableIdList->vgInfo.vgId);
tscDebug("%p build fetch msg from vgId:%d, vgIndex:%d", pSql, pTableIdList->vgInfo.vgId, vgIndex);
tscDebug("%p build fetch msg from vgId:%d, vgIndex:%d, qhandle:%" PRIX64, pSql, pTableIdList->vgInfo.vgId, vgIndex, pSql->res.qhandle);
}
} else {
STableMeta* pTableMeta = pTableMetaInfo->pTableMeta;
pRetrieveMsg->header.vgId = htonl(pTableMeta->vgId);
tscDebug("%p build fetch msg from only one vgroup, vgId:%d", pSql, pTableMeta->vgId);
tscDebug("%p build fetch msg from only one vgroup, vgId:%d, qhandle:%" PRIX64, pSql, pTableMeta->vgId, pSql->res.qhandle);
}
pSql->cmd.payloadLen = sizeof(SRetrieveTableMsg);

25
src/common/src/texpr.c
View File

@ -41,41 +41,46 @@ static uint8_t UNUSED_FUNC isQueryOnPrimaryKey(const char *primaryColumnName, co
static void reverseCopy(char* dest, const char* src, int16_t type, int32_t numOfRows) {
switch(type) {
case TSDB_DATA_TYPE_TINYINT: {
case TSDB_DATA_TYPE_TINYINT:
case TSDB_DATA_TYPE_UTINYINT:{
int8_t* p = (int8_t*) dest;
int8_t* pSrc = (int8_t*) src;
for(int32_t i = 0; i < numOfRows; ++i) {
p[i] = pSrc[numOfRows - i - 1];
}
break;
return;
}
case TSDB_DATA_TYPE_SMALLINT: {
case TSDB_DATA_TYPE_SMALLINT:
case TSDB_DATA_TYPE_USMALLINT:{
int16_t* p = (int16_t*) dest;
int16_t* pSrc = (int16_t*) src;
for(int32_t i = 0; i < numOfRows; ++i) {
p[i] = pSrc[numOfRows - i - 1];
}
break;
return;
}
case TSDB_DATA_TYPE_INT: {
case TSDB_DATA_TYPE_INT:
case TSDB_DATA_TYPE_UINT: {
int32_t* p = (int32_t*) dest;
int32_t* pSrc = (int32_t*) src;
for(int32_t i = 0; i < numOfRows; ++i) {
p[i] = pSrc[numOfRows - i - 1];
}
break;
return;
}
case TSDB_DATA_TYPE_BIGINT: {
case TSDB_DATA_TYPE_BIGINT:
case TSDB_DATA_TYPE_UBIGINT: {
int64_t* p = (int64_t*) dest;
int64_t* pSrc = (int64_t*) src;
for(int32_t i = 0; i < numOfRows; ++i) {
p[i] = pSrc[numOfRows - i - 1];
}
break;
return;
}
case TSDB_DATA_TYPE_FLOAT: {
float* p = (float*) dest;
@ -84,7 +89,7 @@ static void reverseCopy(char* dest, const char* src, int16_t type, int32_t numOf
for(int32_t i = 0; i < numOfRows; ++i) {
p[i] = pSrc[numOfRows - i - 1];
}
break;
return;
}
case TSDB_DATA_TYPE_DOUBLE: {
double* p = (double*) dest;
@ -93,7 +98,7 @@ static void reverseCopy(char* dest, const char* src, int16_t type, int32_t numOf
for(int32_t i = 0; i < numOfRows; ++i) {
p[i] = pSrc[numOfRows - i - 1];
}
break;
return;
}
default: assert(0);
}

2
src/connector/jdbc/CMakeLists.txt
View File

@ -8,7 +8,7 @@ IF (TD_MVN_INSTALLED)
ADD_CUSTOM_COMMAND(OUTPUT ${JDBC_CMD_NAME}
POST_BUILD
COMMAND mvn -Dmaven.test.skip=true install -f ${CMAKE_CURRENT_SOURCE_DIR}/pom.xml
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/target/taos-jdbcdriver-2.0.21-dist.jar ${LIBRARY_OUTPUT_PATH}
COMMAND ${CMAKE_COMMAND} -E copy ${CMAKE_CURRENT_SOURCE_DIR}/target/taos-jdbcdriver-2.0.22-dist.jar ${LIBRARY_OUTPUT_PATH}
COMMAND mvn -Dmaven.test.skip=true clean -f ${CMAKE_CURRENT_SOURCE_DIR}/pom.xml
COMMENT "build jdbc driver")
ADD_CUSTOM_TARGET(${JDBC_TARGET_NAME} ALL WORKING_DIRECTORY ${EXECUTABLE_OUTPUT_PATH} DEPENDS ${JDBC_CMD_NAME})

2
src/connector/jdbc/deploy-pom.xml
View File

@ -5,7 +5,7 @@
<groupId>com.taosdata.jdbc</groupId>
<artifactId>taos-jdbcdriver</artifactId>
<version>2.0.21</version>
<version>2.0.22</version>
<packaging>jar</packaging>
<name>JDBCDriver</name>

4
src/connector/jdbc/pom.xml
View File

@ -3,7 +3,7 @@
<modelVersion>4.0.0</modelVersion>
<groupId>com.taosdata.jdbc</groupId>
<artifactId>taos-jdbcdriver</artifactId>
<version>2.0.21</version>
<version>2.0.22</version>
<packaging>jar</packaging>
<name>JDBCDriver</name>
<url>https://github.com/taosdata/TDengine/tree/master/src/connector/jdbc</url>
@ -102,6 +102,8 @@
<include>**/*Test.java</include>
</includes>
<excludes>
<exclude>**/TSDBJNIConnectorTest.java</exclude>
<exclude>**/UnsignedNumberJniTest.java</exclude>
<exclude>**/DatetimeBefore1970Test.java</exclude>
<exclude>**/AppMemoryLeakTest.java</exclude>
<exclude>**/AuthenticationTest.java</exclude>

7
src/connector/jdbc/src/main/java/com/taosdata/jdbc/AbstractDriver.java
View File

@ -12,7 +12,7 @@ public abstract class AbstractDriver implements Driver {
hostProp.required = false;
hostProp.description = "Hostname";
DriverPropertyInfo portProp = new DriverPropertyInfo(TSDBDriver.PROPERTY_KEY_PORT, info.getProperty(TSDBDriver.PROPERTY_KEY_PORT, TSDBConstants.DEFAULT_PORT));
DriverPropertyInfo portProp = new DriverPropertyInfo(TSDBDriver.PROPERTY_KEY_PORT, info.getProperty(TSDBDriver.PROPERTY_KEY_PORT));
portProp.required = false;
portProp.description = "Port";
@ -40,11 +40,11 @@ public abstract class AbstractDriver implements Driver {
protected Properties parseURL(String url, Properties defaults) {
Properties urlProps = (defaults != null) ? defaults : new Properties();
// parse properties
// parse properties in url
int beginningOfSlashes = url.indexOf("//");
int index = url.indexOf("?");
if (index != -1) {
String paramString = url.substring(index + 1, url.length());
String paramString = url.substring(index + 1);
url = url.substring(0, index);
StringTokenizer queryParams = new StringTokenizer(paramString, "&");
while (queryParams.hasMoreElements()) {
@ -68,6 +68,7 @@ public abstract class AbstractDriver implements Driver {
String dbProductName = url.substring(0, beginningOfSlashes);
dbProductName = dbProductName.substring(dbProductName.indexOf(":") + 1);
dbProductName = dbProductName.substring(0, dbProductName.indexOf(":"));
urlProps.setProperty(TSDBDriver.PROPERTY_KEY_PRODUCT_NAME,dbProductName);
// parse dbname
url = url.substring(beginningOfSlashes + 2);
int indexOfSlash = url.indexOf("/");

10
src/connector/jdbc/src/main/java/com/taosdata/jdbc/AbstractResultSet.java
View File

@ -29,12 +29,7 @@ public abstract class AbstractResultSet extends WrapperImpl implements ResultSet
public abstract boolean getBoolean(int columnIndex) throws SQLException;
@Override
public byte getByte(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNSUPPORTED_METHOD);
}
public abstract byte getByte(int columnIndex) throws SQLException;
@Override
public abstract short getShort(int columnIndex) throws SQLException;
@ -1205,6 +1200,7 @@ public abstract class AbstractResultSet extends WrapperImpl implements ResultSet
public <T> T getObject(String columnLabel, Class<T> type) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_STATEMENT_CLOSED);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNSUPPORTED_METHOD); }
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNSUPPORTED_METHOD);
}
}

7
src/connector/jdbc/src/main/java/com/taosdata/jdbc/SavedPreparedStatement.java
View File

@ -122,8 +122,7 @@ public class SavedPreparedStatement {
initPreparedParam = initDefaultParam(tableName, middleParamSize, valueListSize);
} else {
// not match
throw new SQLException(TSDBConstants.WrapErrMsg("the sql is not complete!"));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_INVALID_SQL);
}
}
@ -189,7 +188,7 @@ public class SavedPreparedStatement {
String errorMsg = String.format("the parameterIndex %s out of the range [1, %s]", parameterIndex, paramSize);
if (parameterIndex < 1 || parameterIndex > paramSize) {
throw new SQLException(TSDBConstants.WrapErrMsg(errorMsg));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE,errorMsg);
}
this.isAddBatch = false; //set isAddBatch to false
@ -212,7 +211,7 @@ public class SavedPreparedStatement {
return;
}
throw new SQLException(TSDBConstants.WrapErrMsg(errorMsg));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE,errorMsg);
}
public void addBatch() {

132
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBConstants.java
View File

@ -16,16 +16,11 @@ package com.taosdata.jdbc;
import java.sql.SQLException;
import java.sql.Types;
import java.util.HashMap;
import java.util.Map;
public abstract class TSDBConstants {
public static final String DEFAULT_PORT = "6200";
public static Map<Integer, String> DATATYPE_MAP = null;
public static final long JNI_NULL_POINTER = 0L;
// JNI_ERROR_NUMBER
public static final int JNI_SUCCESS = 0;
public static final int JNI_TDENGINE_ERROR = -1;
public static final int JNI_CONNECTION_NULL = -2;
@ -34,8 +29,7 @@ public abstract class TSDBConstants {
public static final int JNI_SQL_NULL = -5;
public static final int JNI_FETCH_END = -6;
public static final int JNI_OUT_OF_MEMORY = -7;
public static final int TSDB_DATA_TYPE_NULL = 0;
// TSDB Data Types
public static final int TSDB_DATA_TYPE_BOOL = 1;
public static final int TSDB_DATA_TYPE_TINYINT = 2;
public static final int TSDB_DATA_TYPE_SMALLINT = 3;
@ -46,46 +40,36 @@ public abstract class TSDBConstants {
public static final int TSDB_DATA_TYPE_BINARY = 8;
public static final int TSDB_DATA_TYPE_TIMESTAMP = 9;
public static final int TSDB_DATA_TYPE_NCHAR = 10;
// nchar field's max length
/*
系统增加新的无符号数据类型，分别是：
unsigned tinyint， 数值范围：0-254, NULL 为255
unsigned smallint，数值范围： 0-65534， NULL 为65535
unsigned int，数值范围：0-4294967294，NULL 为4294967295u
unsigned bigint，数值范围：0-18446744073709551614u，NULL 为18446744073709551615u。
example:
create table tb(ts timestamp, a tinyint unsigned, b smallint unsigned, c int unsigned, d bigint unsigned);
*/
public static final int TSDB_DATA_TYPE_UTINYINT = 11; //unsigned tinyint
public static final int TSDB_DATA_TYPE_USMALLINT = 12; //unsigned smallint
public static final int TSDB_DATA_TYPE_UINT = 13; //unsigned int
public static final int TSDB_DATA_TYPE_UBIGINT = 14; //unsigned bigint
// nchar column max length
public static final int maxFieldSize = 16 * 1024;
public static String WrapErrMsg(String msg) {
return "TDengine Error: " + msg;
}
public static String FixErrMsg(int code) {
switch (code) {
case JNI_TDENGINE_ERROR:
return WrapErrMsg("internal error of database!");
case JNI_CONNECTION_NULL:
return WrapErrMsg("invalid tdengine connection!");
case JNI_RESULT_SET_NULL:
return WrapErrMsg("invalid resultset pointer!");
case JNI_NUM_OF_FIELDS_0:
return WrapErrMsg("invalid num of fields!");
case JNI_SQL_NULL:
return WrapErrMsg("can't execute empty sql!");
case JNI_FETCH_END:
return WrapErrMsg("fetch to the end of resultset");
default:
break;
}
return WrapErrMsg("unkown error!");
}
public static int taosType2JdbcType(int taosType) throws SQLException {
switch (taosType) {
case TSDBConstants.TSDB_DATA_TYPE_NULL:
return Types.NULL;
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return Types.BOOLEAN;
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
case TSDBConstants.TSDB_DATA_TYPE_UTINYINT:
return Types.TINYINT;
case TSDBConstants.TSDB_DATA_TYPE_USMALLINT:
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return Types.SMALLINT;
case TSDBConstants.TSDB_DATA_TYPE_UINT:
case TSDBConstants.TSDB_DATA_TYPE_INT:
return Types.INTEGER;
case TSDBConstants.TSDB_DATA_TYPE_UBIGINT:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return Types.BIGINT;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
@ -99,13 +83,42 @@ public abstract class TSDBConstants {
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
return Types.NCHAR;
}
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN_SQL_TYPE_IN_TDENGINE);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN_TAOS_TYPE_IN_TDENGINE);
}
public static String taosType2JdbcTypeName(int taosType) throws SQLException {
switch (taosType){
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return "BOOL";
case TSDBConstants.TSDB_DATA_TYPE_UTINYINT:
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return "TINYINT";
case TSDBConstants.TSDB_DATA_TYPE_USMALLINT:
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return "SMALLINT";
case TSDBConstants.TSDB_DATA_TYPE_UINT:
case TSDBConstants.TSDB_DATA_TYPE_INT:
return "INT";
case TSDBConstants.TSDB_DATA_TYPE_UBIGINT:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return "BIGINT";
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return "FLOAT";
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return "DOUBLE";
case TSDBConstants.TSDB_DATA_TYPE_BINARY:
return "BINARY";
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
return "TIMESTAMP";
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
return "NCHAR";
default:
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN_TAOS_TYPE_IN_TDENGINE);
}
}
public static int jdbcType2TaosType(int jdbcType) throws SQLException {
switch (jdbcType){
case Types.NULL:
return TSDBConstants.TSDB_DATA_TYPE_NULL;
case Types.BOOLEAN:
return TSDBConstants.TSDB_DATA_TYPE_BOOL;
case Types.TINYINT:
@ -130,22 +143,31 @@ public abstract class TSDBConstants {
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN_SQL_TYPE_IN_TDENGINE);
}
static {
DATATYPE_MAP = new HashMap<>();
DATATYPE_MAP.put(0, "NULL");
DATATYPE_MAP.put(1, "BOOL");
DATATYPE_MAP.put(2, "TINYINT");
DATATYPE_MAP.put(3, "SMALLINT");
DATATYPE_MAP.put(4, "INT");
DATATYPE_MAP.put(5, "BIGINT");
DATATYPE_MAP.put(6, "FLOAT");
DATATYPE_MAP.put(7, "DOUBLE");
DATATYPE_MAP.put(8, "BINARY");
DATATYPE_MAP.put(9, "TIMESTAMP");
DATATYPE_MAP.put(10, "NCHAR");
public static String jdbcType2TaosTypeName(int jdbcType) throws SQLException {
switch (jdbcType){
case Types.BOOLEAN:
return "BOOL";
case Types.TINYINT:
return "TINYINT";
case Types.SMALLINT:
return "SMALLINT";
case Types.INTEGER:
return "INT";
case Types.BIGINT:
return "BIGINT";
case Types.FLOAT:
return "FLOAT";
case Types.DOUBLE:
return "DOUBLE";
case Types.BINARY:
return "BINARY";
case Types.TIMESTAMP:
return "TIMESTAMP";
case Types.NCHAR:
return "NCHAR";
default:
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_UNKNOWN_SQL_TYPE_IN_TDENGINE);
}
}
public static String jdbcType2TaosTypeName(int type) throws SQLException {
return DATATYPE_MAP.get(jdbcType2TaosType(type));
}
}

28
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBDriver.java
View File

@ -44,6 +44,10 @@ public class TSDBDriver extends AbstractDriver {
private static final String URL_PREFIX = "jdbc:TAOS://";
/**
* PRODUCT_NAME
*/
public static final String PROPERTY_KEY_PRODUCT_NAME = "productName";
/**
* Key used to retrieve the host value from the properties instance passed to
* the driver.
@ -96,38 +100,34 @@ public class TSDBDriver extends AbstractDriver {
static {
try {
java.sql.DriverManager.registerDriver(new TSDBDriver());
} catch (SQLException E) {
throw new RuntimeException(TSDBConstants.WrapErrMsg("can't register tdengine jdbc driver!"));
} catch (SQLException e) {
throw TSDBError.createRuntimeException(TSDBErrorNumbers.ERROR_CANNOT_REGISTER_JNI_DRIVER, e);
}
}
public Connection connect(String url, Properties info) throws SQLException {
if (url == null)
throw new SQLException(TSDBConstants.WrapErrMsg("url is not set!"));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_URL_NOT_SET);
if (!acceptsURL(url))
return null;
Properties props = null;
if ((props = parseURL(url, info)) == null) {
Properties props = parseURL(url, info);
if (props == null) {
return null;
}
try {
TSDBJNIConnector.init((String) props.get(PROPERTY_KEY_CONFIG_DIR), (String) props.get(PROPERTY_KEY_LOCALE),
(String) props.get(PROPERTY_KEY_CHARSET), (String) props.get(PROPERTY_KEY_TIME_ZONE));
Connection newConn = new TSDBConnection(props, this.dbMetaData);
return newConn;
return new TSDBConnection(props, this.dbMetaData);
} catch (SQLWarning sqlWarning) {
sqlWarning.printStackTrace();
Connection newConn = new TSDBConnection(props, this.dbMetaData);
return newConn;
return new TSDBConnection(props, this.dbMetaData);
} catch (SQLException sqlEx) {
throw sqlEx;
} catch (Exception ex) {
SQLException sqlEx = new SQLException("SQLException:" + ex.toString());
sqlEx.initCause(ex);
throw sqlEx;
throw new SQLException("SQLException:" + ex.toString(), ex);
}
}
@ -139,8 +139,8 @@ public class TSDBDriver extends AbstractDriver {
*/
public boolean acceptsURL(String url) throws SQLException {
if (url == null)
throw new SQLException(TSDBConstants.WrapErrMsg("url is null"));
return (url != null && url.length() > 0 && url.trim().length() > 0) && (url.startsWith(URL_PREFIX) || url.startsWith(URL_PREFIX1));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_URL_NOT_SET);
return url.length() > 0 && url.trim().length() > 0 && (url.startsWith(URL_PREFIX) || url.startsWith(URL_PREFIX1));
}
public DriverPropertyInfo[] getPropertyInfo(String url, Properties info) throws SQLException {

24
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBError.java
View File

@ -3,6 +3,7 @@ package com.taosdata.jdbc;
import java.sql.SQLClientInfoException;
import java.sql.SQLException;
import java.sql.SQLFeatureNotSupportedException;
import java.sql.SQLWarning;
import java.util.HashMap;
import java.util.Map;
@ -18,18 +19,25 @@ public class TSDBError {
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_BATCH_IS_EMPTY, "Batch is empty!");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_WITH_EXECUTEQUERY, "Can not issue data manipulation statements with executeQuery()");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_WITH_EXECUTEUPDATE, "Can not issue SELECT via executeUpdate()");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE_QUERY, "not a valid sql for executeQuery: (?)");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE_QUERY, "invalid sql for executeQuery: (?)");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_DATABASE_NOT_SPECIFIED_OR_AVAILABLE, "Database not specified or available");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE_UPDATE, "not a valid sql for executeUpdate: (?)");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE, "not a valid sql for execute: (?)");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE_UPDATE, "invalid sql for executeUpdate: (?)");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_FOR_EXECUTE, "invalid sql for execute: (?)");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "parameter index out of range");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_SQLCLIENT_EXCEPTION_ON_CONNECTION_CLOSED, "connection already closed");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_UNKNOWN_SQL_TYPE_IN_TDENGINE, "unknown sql type in tdengine");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_CANNOT_REGISTER_JNI_DRIVER, "can't register JDBC-JNI driver");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_CANNOT_REGISTER_RESTFUL_DRIVER, "can't register JDBC-RESTful driver");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_URL_NOT_SET, "url is not set");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_INVALID_SQL, "invalid sql");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE, "numeric value out of range");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_UNKNOWN_TAOS_TYPE_IN_TDENGINE, "unknown taos type in tdengine");
/**************************************************/
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_UNKNOWN, "unknown error");
/**************************************************/
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_SUBSCRIBE_FAILED, "failed to create subscription");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_UNSUPPORTED_ENCODING, "Unsupported encoding");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_JNI_TDENGINE_ERROR, "internal error of database");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL, "JNI connection is NULL");
TSDBErrorMap.put(TSDBErrorNumbers.ERROR_JNI_RESULT_SET_NULL, "JNI result set is NULL");
@ -65,4 +73,12 @@ public class TSDBError {
return new SQLException("TDengine ERROR (" + Integer.toHexString(errorCode) + "): " + message, "", errorCode);
}
public static RuntimeException createRuntimeException(int errorCode, Throwable t) {
String message = TSDBErrorMap.get(errorCode);
return new RuntimeException("ERROR (" + Integer.toHexString(errorCode) + "): " + message, t);
}
public static SQLWarning createSQLWarning(String message) {
return new SQLWarning(message);
}
}

15
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBErrorNumbers.java
View File

@ -19,12 +19,17 @@ public class TSDBErrorNumbers {
public static final int ERROR_PARAMETER_INDEX_OUT_RANGE = 0x230d; // parameter index out of range
public static final int ERROR_SQLCLIENT_EXCEPTION_ON_CONNECTION_CLOSED = 0x230e; // connection already closed
public static final int ERROR_UNKNOWN_SQL_TYPE_IN_TDENGINE = 0x230f; //unknown sql type in tdengine
public static final int ERROR_CANNOT_REGISTER_JNI_DRIVER = 0x2310; // can't register JDBC-JNI driver
public static final int ERROR_CANNOT_REGISTER_RESTFUL_DRIVER = 0x2311; // can't register JDBC-RESTful driver
public static final int ERROR_URL_NOT_SET = 0x2312; // url is not set
public static final int ERROR_INVALID_SQL = 0x2313; // invalid sql
public static final int ERROR_NUMERIC_VALUE_OUT_OF_RANGE = 0x2314; // numeric value out of range
public static final int ERROR_UNKNOWN_TAOS_TYPE_IN_TDENGINE = 0x2315; //unknown taos type in tdengine
public static final int ERROR_UNKNOWN = 0x2350; //unknown error
public static final int ERROR_SUBSCRIBE_FAILED = 0x2351; // failed to create subscription
public static final int ERROR_UNSUPPORTED_ENCODING = 0x2352; // Unsupported encoding
public static final int ERROR_JNI_TDENGINE_ERROR = 0x2353; // internal error of database
public static final int ERROR_JNI_CONNECTION_NULL = 0x2354; // JNI connection is NULL
public static final int ERROR_JNI_RESULT_SET_NULL = 0x2355; // invalid JNI result set
@ -51,11 +56,16 @@ public class TSDBErrorNumbers {
errorNumbers.add(ERROR_PARAMETER_INDEX_OUT_RANGE);
errorNumbers.add(ERROR_SQLCLIENT_EXCEPTION_ON_CONNECTION_CLOSED);
errorNumbers.add(ERROR_UNKNOWN_SQL_TYPE_IN_TDENGINE);
errorNumbers.add(ERROR_CANNOT_REGISTER_JNI_DRIVER);
errorNumbers.add(ERROR_CANNOT_REGISTER_RESTFUL_DRIVER);
errorNumbers.add(ERROR_URL_NOT_SET);
errorNumbers.add(ERROR_INVALID_SQL);
errorNumbers.add(ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
errorNumbers.add(ERROR_UNKNOWN_TAOS_TYPE_IN_TDENGINE);
/*****************************************************/
errorNumbers.add(ERROR_SUBSCRIBE_FAILED);
errorNumbers.add(ERROR_UNSUPPORTED_ENCODING);
errorNumbers.add(ERROR_JNI_TDENGINE_ERROR);
errorNumbers.add(ERROR_JNI_CONNECTION_NULL);
errorNumbers.add(ERROR_JNI_RESULT_SET_NULL);
@ -63,7 +73,6 @@ public class TSDBErrorNumbers {
errorNumbers.add(ERROR_JNI_SQL_NULL);
errorNumbers.add(ERROR_JNI_FETCH_END);
errorNumbers.add(ERROR_JNI_OUT_OF_MEMORY);
}
private TSDBErrorNumbers() {

17
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBJNIConnector.java
View File

@ -1,18 +1,19 @@
/**
* *************************************************************************
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* <p>
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* <p>
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* <p>
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
**************************************************************************** */
* ***************************************************************************
*/
package com.taosdata.jdbc;
import com.taosdata.jdbc.utils.TaosInfo;
@ -23,7 +24,7 @@ import java.util.List;
/**
* JNI connector
* */
*/
public class TSDBJNIConnector {
private static volatile Boolean isInitialized = false;
@ -72,13 +73,13 @@ public class TSDBJNIConnector {
if (!isInitialized) {
initImp(configDir);
if (setOptions(0, locale) < 0) {
throw new SQLWarning(TSDBConstants.WrapErrMsg("Failed to set locale: " + locale + ". System default will be used."));
throw TSDBError.createSQLWarning("Failed to set locale: " + locale + ". System default will be used.");
}
if (setOptions(1, charset) < 0) {
throw new SQLWarning(TSDBConstants.WrapErrMsg("Failed to set charset: " + charset + ". System default will be used."));
throw TSDBError.createSQLWarning("Failed to set charset: " + charset + ". System default will be used.");
}
if (setOptions(2, timezone) < 0) {
throw new SQLWarning(TSDBConstants.WrapErrMsg("Failed to set timezone: " + timezone + ". System default will be used."));
throw TSDBError.createSQLWarning("Failed to set timezone: " + timezone + ". System default will be used.");
}
isInitialized = true;
TaosGlobalConfig.setCharset(getTsCharset());

85
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBResultSet.java
View File

@ -27,8 +27,8 @@ public class TSDBResultSet extends AbstractResultSet implements ResultSet {
private final TSDBResultSetRowData rowData;
private final TSDBResultSetBlockData blockData;
private boolean batchFetch = false;
private boolean lastWasNull = false;
private boolean batchFetch;
private boolean lastWasNull;
private boolean isClosed;
public void setBatchFetch(boolean batchFetch) {
@ -86,7 +86,6 @@ public class TSDBResultSet extends AbstractResultSet implements ResultSet {
if (rowData != null) {
this.rowData.clear();
}
int code = this.jniConnector.fetchRow(this.resultSetPointer, this.rowData);
if (code == TSDBConstants.JNI_CONNECTION_NULL) {
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL);
@ -124,30 +123,27 @@ public class TSDBResultSet extends AbstractResultSet implements ResultSet {
String res = null;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.getString(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getString(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
return this.blockData.getString(colIndex);
}
}
public boolean getBoolean(int columnIndex) throws SQLException {
boolean res = false;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.getBoolean(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getBoolean(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
} else {
return this.blockData.getBoolean(colIndex);
}
return res;
}
@ -155,91 +151,84 @@ public class TSDBResultSet extends AbstractResultSet implements ResultSet {
byte res = 0;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return (byte) this.blockData.getInt(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = (byte) this.rowData.getInt(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
return (byte) this.blockData.getInt(colIndex);
}
}
public short getShort(int columnIndex) throws SQLException {
short res = 0;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return (short) this.blockData.getInt(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = (short) this.rowData.getInt(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
return (short) this.blockData.getInt(colIndex);
}
}
public int getInt(int columnIndex) throws SQLException {
int res = 0;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.getInt(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getInt(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
return this.blockData.getInt(colIndex);
}
}
public long getLong(int columnIndex) throws SQLException {
long res = 0L;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.getLong(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getLong(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
return this.blockData.getLong(colIndex);
}
}
public float getFloat(int columnIndex) throws SQLException {
float res = 0;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getFloat(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
if (this.getBatchFetch())
return (float) this.blockData.getDouble(colIndex);
}
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull)
res = this.rowData.getFloat(colIndex, this.columnMetaDataList.get(colIndex).getColType());
return res;
}
public double getDouble(int columnIndex) throws SQLException {
double res = 0;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.getDouble(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getDouble(colIndex, this.columnMetaDataList.get(colIndex).getColType());
}
return res;
} else {
return this.blockData.getDouble(colIndex);
}
}
@Deprecated
@ -255,15 +244,14 @@ public class TSDBResultSet extends AbstractResultSet implements ResultSet {
Timestamp res = null;
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.getTimestamp(columnIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
if (!lastWasNull) {
res = this.rowData.getTimestamp(colIndex);
}
return res;
} else {
return this.blockData.getTimestamp(columnIndex);
}
}
public ResultSetMetaData getMetaData() throws SQLException {
@ -274,12 +262,11 @@ public class TSDBResultSet extends AbstractResultSet implements ResultSet {
public Object getObject(int columnIndex) throws SQLException {
int colIndex = getTrueColumnIndex(columnIndex);
if (!this.getBatchFetch()) {
if (this.getBatchFetch())
return this.blockData.get(colIndex);
this.lastWasNull = this.rowData.wasNull(colIndex);
return this.rowData.get(colIndex);
} else {
return this.blockData.get(colIndex);
}
}
@Override

4
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBResultSetBlockData.java
View File

@ -100,24 +100,28 @@ public class TSDBResultSetBlockData {
this.colData.set(col, buf);
break;
}
case TSDBConstants.TSDB_DATA_TYPE_UTINYINT:
case TSDBConstants.TSDB_DATA_TYPE_TINYINT: {
ByteBuffer buf = ByteBuffer.wrap(value, 0, length);
buf.order(ByteOrder.LITTLE_ENDIAN);
this.colData.set(col, buf);
break;
}
case TSDBConstants.TSDB_DATA_TYPE_USMALLINT:
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT: {
ByteBuffer buf = ByteBuffer.wrap(value, 0, length);
ShortBuffer sb = buf.order(ByteOrder.LITTLE_ENDIAN).asShortBuffer();
this.colData.set(col, sb);
break;
}
case TSDBConstants.TSDB_DATA_TYPE_UINT:
case TSDBConstants.TSDB_DATA_TYPE_INT: {
ByteBuffer buf = ByteBuffer.wrap(value, 0, length);
IntBuffer ib = buf.order(ByteOrder.LITTLE_ENDIAN).asIntBuffer();
this.colData.set(col, ib);
break;
}
case TSDBConstants.TSDB_DATA_TYPE_UBIGINT:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT: {
ByteBuffer buf = ByteBuffer.wrap(value, 0, length);
LongBuffer lb = buf.order(ByteOrder.LITTLE_ENDIAN).asLongBuffer();

26
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBResultSetMetaData.java
View File

@ -126,34 +126,12 @@ public class TSDBResultSetMetaData extends WrapperImpl implements ResultSetMetaD
public int getColumnType(int column) throws SQLException {
ColumnMetaData meta = this.colMetaDataList.get(column - 1);
switch (meta.getColType()) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return Types.BOOLEAN;
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return java.sql.Types.TINYINT;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return java.sql.Types.SMALLINT;
case TSDBConstants.TSDB_DATA_TYPE_INT:
return java.sql.Types.INTEGER;
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return java.sql.Types.BIGINT;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return java.sql.Types.FLOAT;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return java.sql.Types.DOUBLE;
case TSDBConstants.TSDB_DATA_TYPE_BINARY:
return Types.BINARY;
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
return java.sql.Types.TIMESTAMP;
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
return Types.NCHAR;
}
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_INVALID_VARIABLE);
return TSDBConstants.taosType2JdbcType(meta.getColType());
}
public String getColumnTypeName(int column) throws SQLException {
ColumnMetaData meta = this.colMetaDataList.get(column - 1);
return TSDBConstants.DATATYPE_MAP.get(meta.getColType());
return TSDBConstants.taosType2JdbcTypeName(meta.getColType());
}
public boolean isReadOnly(int column) throws SQLException {

192
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBResultSetRowData.java
View File

@ -14,6 +14,7 @@
*****************************************************************************/
package com.taosdata.jdbc;
import java.math.BigDecimal;
import java.sql.SQLException;
import java.sql.Timestamp;
import java.util.ArrayList;
@ -55,14 +56,21 @@ public class TSDBResultSetRowData {
Object obj = data.get(col);
switch (srcType) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL: return (Boolean) obj;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT: return ((Float) obj) == 1.0? Boolean.TRUE:Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE: return ((Double) obj) == 1.0? Boolean.TRUE:Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_TINYINT: return ((Byte) obj) == 1? Boolean.TRUE:Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:return ((Short)obj) == 1? Boolean.TRUE:Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_INT: return ((Integer)obj) == 1? Boolean.TRUE:Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return (Boolean) obj;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return ((Float) obj) == 1.0 ? Boolean.TRUE : Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return ((Double) obj) == 1.0 ? Boolean.TRUE : Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return ((Byte) obj) == 1 ? Boolean.TRUE : Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return ((Short) obj) == 1 ? Boolean.TRUE : Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_INT:
return ((Integer) obj) == 1 ? Boolean.TRUE : Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT: return ((Long) obj) == 1L? Boolean.TRUE:Boolean.FALSE;
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return ((Long) obj) == 1L ? Boolean.TRUE : Boolean.FALSE;
}
return Boolean.TRUE;
@ -84,16 +92,48 @@ public class TSDBResultSetRowData {
Object obj = data.get(col);
switch (srcType) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL: return Boolean.TRUE.equals(obj)? 1:0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT: return ((Float) obj).intValue();
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE: return ((Double)obj).intValue();
case TSDBConstants.TSDB_DATA_TYPE_TINYINT: return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT: return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return Boolean.TRUE.equals(obj) ? 1 : 0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return ((Float) obj).intValue();
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return ((Double) obj).intValue();
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT:
return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT: return ((Long) obj).intValue();
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return ((Long) obj).intValue();
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
case TSDBConstants.TSDB_DATA_TYPE_BINARY: return Integer.parseInt((String) obj);
case TSDBConstants.TSDB_DATA_TYPE_BINARY:
return Integer.parseInt((String) obj);
case TSDBConstants.TSDB_DATA_TYPE_UTINYINT: {
Byte value = (byte) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
case TSDBConstants.TSDB_DATA_TYPE_USMALLINT: {
short value = (short) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
case TSDBConstants.TSDB_DATA_TYPE_UINT: {
int value = (int) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
case TSDBConstants.TSDB_DATA_TYPE_UBIGINT: {
long value = (long) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return new Long(value).intValue();
}
}
return 0;
@ -107,16 +147,48 @@ public class TSDBResultSetRowData {
Object obj = data.get(col);
switch (srcType) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL: return Boolean.TRUE.equals(obj)? 1:0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT: return ((Float) obj).longValue();
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE: return ((Double) obj).longValue();
case TSDBConstants.TSDB_DATA_TYPE_TINYINT: return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT: return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return Boolean.TRUE.equals(obj) ? 1 : 0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return ((Float) obj).longValue();
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return ((Double) obj).longValue();
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT:
return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT: return (Long) obj;
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return (Long) obj;
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
case TSDBConstants.TSDB_DATA_TYPE_BINARY: return Long.parseLong((String) obj);
case TSDBConstants.TSDB_DATA_TYPE_BINARY:
return Long.parseLong((String) obj);
case TSDBConstants.TSDB_DATA_TYPE_UTINYINT: {
Byte value = (byte) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
case TSDBConstants.TSDB_DATA_TYPE_USMALLINT: {
short value = (short) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
case TSDBConstants.TSDB_DATA_TYPE_UINT: {
int value = (int) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
case TSDBConstants.TSDB_DATA_TYPE_UBIGINT: {
long value = (long) obj;
if (value < 0)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE);
return value;
}
}
return 0;
@ -130,14 +202,21 @@ public class TSDBResultSetRowData {
Object obj = data.get(col);
switch (srcType) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL: return Boolean.TRUE.equals(obj)? 1:0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT: return (Float) obj;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE: return ((Double) obj).floatValue();
case TSDBConstants.TSDB_DATA_TYPE_TINYINT: return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT: return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT: return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return Boolean.TRUE.equals(obj) ? 1 : 0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return (Float) obj;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return ((Double) obj).floatValue();
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT:
return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT: return (Long) obj;
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return (Long) obj;
}
return 0;
@ -151,14 +230,21 @@ public class TSDBResultSetRowData {
Object obj = data.get(col);
switch (srcType) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL: return Boolean.TRUE.equals(obj)? 1:0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT: return (Float) obj;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE: return (Double) obj;
case TSDBConstants.TSDB_DATA_TYPE_TINYINT: return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT: return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return Boolean.TRUE.equals(obj) ? 1 : 0;
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return (Float) obj;
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return (Double) obj;
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return (Byte) obj;
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return (Short) obj;
case TSDBConstants.TSDB_DATA_TYPE_INT:
return (Integer) obj;
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT: return (Long) obj;
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return (Long) obj;
}
return 0;
@ -178,14 +264,42 @@ public class TSDBResultSetRowData {
/**
* The original type may not be a string type, but will be converted to by calling this method
*
* @param col column index
* @return
* @throws SQLException
*/
public String getString(int col, int srcType) throws SQLException {
if (srcType == TSDBConstants.TSDB_DATA_TYPE_BINARY || srcType == TSDBConstants.TSDB_DATA_TYPE_NCHAR) {
switch (srcType) {
case TSDBConstants.TSDB_DATA_TYPE_BINARY:
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
return (String) data.get(col);
} else {
case TSDBConstants.TSDB_DATA_TYPE_UTINYINT: {
Byte value = new Byte(String.valueOf(data.get(col)));
if (value >= 0)
return value.toString();
return Integer.toString(value & 0xff);
}
case TSDBConstants.TSDB_DATA_TYPE_USMALLINT: {
Short value = new Short(String.valueOf(data.get(col)));
if (value >= 0)
return value.toString();
return Integer.toString(value & 0xffff);
}
case TSDBConstants.TSDB_DATA_TYPE_UINT: {
Integer value = new Integer(String.valueOf(data.get(col)));
if (value >= 0)
return value.toString();
return Long.toString(value & 0xffffffffl);
}
case TSDBConstants.TSDB_DATA_TYPE_UBIGINT: {
Long value = new Long(String.valueOf(data.get(col)));
if (value >= 0)
return value.toString();
long lowValue = value & 0x7fffffffffffffffL;
return BigDecimal.valueOf(lowValue).add(BigDecimal.valueOf(Long.MAX_VALUE)).add(BigDecimal.valueOf(1)).toString();
}
default:
return String.valueOf(data.get(col));
}
}

1
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBStatement.java
View File

@ -51,7 +51,6 @@ public class TSDBStatement extends AbstractStatement {
this.connector.freeResultSet(pSql);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_INVALID_WITH_EXECUTEQUERY);
}
TSDBResultSet res = new TSDBResultSet(this, this.connector, pSql);
res.setBatchFetch(this.connection.getBatchFetch());
return res;

22
src/connector/jdbc/src/main/java/com/taosdata/jdbc/TSDBSubscribe.java
View File

@ -21,27 +21,23 @@ public class TSDBSubscribe {
private final long id;
TSDBSubscribe(TSDBJNIConnector connecter, long id) throws SQLException {
if (null != connecter) {
if (connecter == null)
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL);
this.connecter = connecter;
this.id = id;
} else {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_CONNECTION_NULL));
}
}
/**
* consume
*
*/
public TSDBResultSet consume() throws SQLException {
if (this.connecter.isClosed()) {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_CONNECTION_NULL));
}
if (this.connecter.isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL);
long resultSetPointer = this.connecter.consume(this.id);
if (resultSetPointer == TSDBConstants.JNI_CONNECTION_NULL) {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_CONNECTION_NULL));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL);
} else if (resultSetPointer == TSDBConstants.JNI_NULL_POINTER) {
return null;
} else {
@ -56,9 +52,9 @@ public class TSDBSubscribe {
* @throws SQLException
*/
public void close(boolean keepProgress) throws SQLException {
if (this.connecter.isClosed()) {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_CONNECTION_NULL));
}
if (this.connecter.isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL);
this.connecter.unsubscribe(this.id, keepProgress);
}
}

1
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulConnection.java
View File

@ -27,7 +27,6 @@ public class RestfulConnection extends AbstractConnection {
public Statement createStatement() throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_CONNECTION_CLOSED);
;
return new RestfulStatement(this, database);
}

23
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulDriver.java
View File

@ -2,9 +2,7 @@ package com.taosdata.jdbc.rs;
import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONObject;
import com.taosdata.jdbc.AbstractDriver;
import com.taosdata.jdbc.TSDBConstants;
import com.taosdata.jdbc.TSDBDriver;
import com.taosdata.jdbc.*;
import com.taosdata.jdbc.utils.HttpClientPoolUtil;
import java.io.UnsupportedEncodingException;
@ -21,15 +19,16 @@ public class RestfulDriver extends AbstractDriver {
try {
DriverManager.registerDriver(new RestfulDriver());
} catch (SQLException e) {
throw new RuntimeException(TSDBConstants.WrapErrMsg("can not register Restful JDBC driver"), e);
throw TSDBError.createRuntimeException(TSDBErrorNumbers.ERROR_URL_NOT_SET, e);
}
}
@Override
public Connection connect(String url, Properties info) throws SQLException {
// throw SQLException if url is null
if (url == null)
throw new SQLException(TSDBConstants.WrapErrMsg("url is not set!"));
if (url == null || url.trim().isEmpty() || url.trim().replaceAll("\\s", "").isEmpty())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_URL_NOT_SET);
// return null if url is not be accepted
if (!acceptsURL(url))
return null;
@ -61,14 +60,20 @@ public class RestfulDriver extends AbstractDriver {
throw new SQLException(jsonResult.getString("desc"));
}
return new RestfulConnection(host, port, props, database, url);
RestfulConnection conn = new RestfulConnection(host, port, props, database, url);
if (database != null && !database.trim().replaceAll("\\s", "").isEmpty()) {
Statement stmt = conn.createStatement();
stmt.execute("use " + database);
stmt.close();
}
return conn;
}
@Override
public boolean acceptsURL(String url) throws SQLException {
if (url == null)
throw new SQLException(TSDBConstants.WrapErrMsg("url is null"));
return (url != null && url.length() > 0 && url.trim().length() > 0) && url.startsWith(URL_PREFIX);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_URL_NOT_SET);
return url.length() > 0 && url.trim().length() > 0 && url.startsWith(URL_PREFIX);
}
@Override

190
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulResultSet.java
View File

@ -9,7 +9,6 @@ import com.taosdata.jdbc.TSDBErrorNumbers;
import java.sql.*;
import java.util.ArrayList;
import java.util.List;
public class RestfulResultSet extends AbstractResultSet implements ResultSet {
private volatile boolean isClosed;
@ -17,8 +16,9 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
private final String database;
private final Statement statement;
// private final JSONObject resultJson;
// data
private ArrayList<ArrayList<Object>> resultSet;
private final ArrayList<ArrayList<Object>> resultSet;
// meta
private ArrayList<String> columnNames;
private ArrayList<Field> columns;
@ -32,6 +32,8 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
public RestfulResultSet(String database, Statement statement, JSONObject resultJson) throws SQLException {
this.database = database;
this.statement = statement;
// this.resultJson = resultJson;
// column metadata
JSONArray columnMeta = resultJson.getJSONArray("column_meta");
columnNames = new ArrayList<>();
@ -39,10 +41,11 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
for (int colIndex = 0; colIndex < columnMeta.size(); colIndex++) {
JSONArray col = columnMeta.getJSONArray(colIndex);
String col_name = col.getString(0);
int col_type = TSDBConstants.taosType2JdbcType(col.getInteger(1));
int taos_type = col.getInteger(1);
int col_type = TSDBConstants.taosType2JdbcType(taos_type);
int col_length = col.getInteger(2);
columnNames.add(col_name);
columns.add(new Field(col_name, col_type, col_length, ""));
columns.add(new Field(col_name, col_type, col_length, "", taos_type));
}
this.metaData = new RestfulResultSetMetaData(this.database, columns, this);
@ -53,105 +56,50 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
ArrayList row = new ArrayList();
JSONArray jsonRow = data.getJSONArray(rowIndex);
for (int colIndex = 0; colIndex < jsonRow.size(); colIndex++) {
row.add(parseColumnData(jsonRow, colIndex, columns.get(colIndex).type));
row.add(parseColumnData(jsonRow, colIndex, columns.get(colIndex).taos_type));
}
resultSet.add(row);
}
/*
int columnIndex = 0;
for (; columnIndex < data.size(); columnIndex++) {
ArrayList oneRow = new ArrayList<>();
JSONArray one = data.getJSONArray(columnIndex);
for (int j = 0; j < one.size(); j++) {
oneRow.add(one.getString(j));
}
resultSet.add(oneRow);
}
// column only names
JSONArray head = resultJson.getJSONArray("head");
for (int i = 0; i < head.size(); i++) {
String name = head.getString(i);
columnNames.add(name);
columns.add(new Field(name, "", 0, ""));
}
this.metaData = new RestfulResultSetMetaData(this.database, columns, this);
*/
}
private Object parseColumnData(JSONArray row, int colIndex, int sqlType) {
switch (sqlType) {
case Types.NULL:
return null;
case Types.BOOLEAN:
private Object parseColumnData(JSONArray row, int colIndex, int taosType) {
switch (taosType) {
case TSDBConstants.TSDB_DATA_TYPE_BOOL:
return row.getBoolean(colIndex);
case Types.TINYINT:
case Types.SMALLINT:
case TSDBConstants.TSDB_DATA_TYPE_TINYINT:
return row.getByte(colIndex);
case TSDBConstants.TSDB_DATA_TYPE_SMALLINT:
return row.getShort(colIndex);
case Types.INTEGER:
case TSDBConstants.TSDB_DATA_TYPE_INT:
return row.getInteger(colIndex);
case Types.BIGINT:
case TSDBConstants.TSDB_DATA_TYPE_BIGINT:
return row.getBigInteger(colIndex);
case Types.FLOAT:
case TSDBConstants.TSDB_DATA_TYPE_FLOAT:
return row.getFloat(colIndex);
case Types.DOUBLE:
case TSDBConstants.TSDB_DATA_TYPE_DOUBLE:
return row.getDouble(colIndex);
case Types.TIMESTAMP:
case TSDBConstants.TSDB_DATA_TYPE_TIMESTAMP:
return new Timestamp(row.getDate(colIndex).getTime());
case Types.BINARY:
case Types.NCHAR:
case TSDBConstants.TSDB_DATA_TYPE_BINARY:
case TSDBConstants.TSDB_DATA_TYPE_NCHAR:
default:
return row.getString(colIndex);
}
}
// /**
// * 由多个resultSet的JSON构造结果集
// *
// * @param resultJson: 包含data信息的结果集，有sql返回的结果集
// * @param fieldJson: 包含多个（最多2个）meta信息的结果集，有describe xxx
// **/
// public RestfulResultSet(String database, Statement statement, JSONObject resultJson, List<JSONObject> fieldJson) throws SQLException {
// this(database, statement, resultJson);
// ArrayList<Field> newColumns = new ArrayList<>();
//
// for (Field column : columns) {
// Field field = findField(column.name, fieldJson);
// if (field != null) {
// newColumns.add(field);
// } else {
// newColumns.add(column);
// }
// }
// this.columns = newColumns;
// this.metaData = new RestfulResultSetMetaData(this.database, this.columns, this);
// }
// public Field findField(String columnName, List<JSONObject> fieldJsonList) {
// for (JSONObject fieldJSON : fieldJsonList) {
// JSONArray fieldDataJson = fieldJSON.getJSONArray("data");
// for (int i = 0; i < fieldDataJson.size(); i++) {
// JSONArray field = fieldDataJson.getJSONArray(i);
// if (columnName.equalsIgnoreCase(field.getString(0))) {
// return new Field(field.getString(0), field.getString(1), field.getInteger(2), field.getString(3));
// }
// }
// }
// return null;
// }
public class Field {
String name;
int type;
int length;
String note;
int taos_type;
public Field(String name, int type, int length, String note) {
public Field(String name, int type, int length, String note, int taos_type) {
this.name = name;
this.type = type;
this.length = length;
this.note = note;
this.taos_type = taos_type;
}
}
@ -184,53 +132,119 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
public String getString(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size()) {
throw new SQLException(TSDBConstants.WrapErrMsg("Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size()));
}
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
return resultSet.get(pos).get(columnIndex).toString();
Object value = resultSet.get(pos).get(columnIndex);
return value == null ? null : value.toString();
}
@Override
public boolean getBoolean(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
int result = getInt(columnIndex);
return result == 0 ? false : true;
}
@Override
public byte getByte(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
Object value = resultSet.get(pos).get(columnIndex);
if (value == null)
return 0;
long valueAsLong = Long.parseLong(value.toString());
if (valueAsLong == Byte.MIN_VALUE)
return 0;
if (valueAsLong < Byte.MIN_VALUE || valueAsLong > Byte.MAX_VALUE)
throwRangeException(value.toString(), columnIndex, Types.TINYINT);
return (byte) valueAsLong;
}
private void throwRangeException(String valueAsString, int columnIndex, int jdbcType) throws SQLException {
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_NUMERIC_VALUE_OUT_OF_RANGE,
"'" + valueAsString + "' in column '" + columnIndex + "' is outside valid range for the jdbcType " + TSDBConstants.jdbcType2TaosTypeName(jdbcType));
}
@Override
public short getShort(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
return Short.parseShort(resultSet.get(pos).get(columnIndex).toString());
Object value = resultSet.get(pos).get(columnIndex);
if (value == null)
return 0;
long valueAsLong = Long.parseLong(value.toString());
if (valueAsLong == Short.MIN_VALUE)
return 0;
if (valueAsLong < Short.MIN_VALUE || valueAsLong > Short.MAX_VALUE)
throwRangeException(value.toString(), columnIndex, Types.SMALLINT);
return (short) valueAsLong;
}
@Override
public int getInt(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
return Integer.parseInt(resultSet.get(pos).get(columnIndex).toString());
Object value = resultSet.get(pos).get(columnIndex);
if (value == null)
return 0;
long valueAsLong = Long.parseLong(value.toString());
if (valueAsLong == Integer.MIN_VALUE)
return 0;
if (valueAsLong < Integer.MIN_VALUE || valueAsLong > Integer.MAX_VALUE)
throwRangeException(value.toString(), columnIndex, Types.INTEGER);
return (int) valueAsLong;
}
@Override
public long getLong(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
return Long.parseLong(resultSet.get(pos).get(columnIndex).toString());
Object value = resultSet.get(pos).get(columnIndex);
if (value == null)
return 0;
long valueAsLong = 0;
try {
valueAsLong = Long.parseLong(value.toString());
if (valueAsLong == Long.MIN_VALUE)
return 0;
} catch (NumberFormatException e) {
throwRangeException(value.toString(), columnIndex, Types.BIGINT);
}
return valueAsLong;
}
@Override
public float getFloat(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
return Float.parseFloat(resultSet.get(pos).get(columnIndex).toString());
}
@ -239,6 +253,8 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
public double getDouble(int columnIndex) throws SQLException {
if (isClosed())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_RESULTSET_CLOSED);
if (columnIndex > resultSet.get(pos).size())
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE, "Column Index out of range, " + columnIndex + " > " + resultSet.get(pos).size());
columnIndex = getTrueColumnIndex(columnIndex);
return Double.parseDouble(resultSet.get(pos).get(columnIndex).toString());
@ -246,12 +262,14 @@ public class RestfulResultSet extends AbstractResultSet implements ResultSet {
private int getTrueColumnIndex(int columnIndex) throws SQLException {
if (columnIndex < 1) {
throw new SQLException("Column Index out of range, " + columnIndex + " < 1");
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE
, "Column Index out of range, " + columnIndex + " < 1");
}
int numOfCols = resultSet.get(pos).size();
if (columnIndex > numOfCols) {
throw new SQLException("Column Index out of range, " + columnIndex + " > " + numOfCols);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_PARAMETER_INDEX_OUT_RANGE
, "Column Index out of range, " + columnIndex + " > " + numOfCols);
}
return columnIndex - 1;

25
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulResultSetMetaData.java
View File

@ -1,6 +1,7 @@
package com.taosdata.jdbc.rs;
import com.taosdata.jdbc.TSDBConstants;
import com.taosdata.jdbc.WrapperImpl;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
@ -8,7 +9,7 @@ import java.sql.Timestamp;
import java.sql.Types;
import java.util.ArrayList;
public class RestfulResultSetMetaData implements ResultSetMetaData {
public class RestfulResultSetMetaData extends WrapperImpl implements ResultSetMetaData {
private final String database;
private ArrayList<RestfulResultSet.Field> fields;
@ -20,6 +21,10 @@ public class RestfulResultSetMetaData implements ResultSetMetaData {
this.resultSet = resultSet;
}
public ArrayList<RestfulResultSet.Field> getFields() {
return fields;
}
@Override
public int getColumnCount() throws SQLException {
return fields.size();
@ -134,8 +139,8 @@ public class RestfulResultSetMetaData implements ResultSetMetaData {
@Override
public String getColumnTypeName(int column) throws SQLException {
int type = fields.get(column - 1).type;
return TSDBConstants.jdbcType2TaosTypeName(type);
int taos_type = fields.get(column - 1).taos_type;
return TSDBConstants.taosType2JdbcTypeName(taos_type);
}
@Override
@ -180,18 +185,4 @@ public class RestfulResultSetMetaData implements ResultSetMetaData {
return columnClassName;
}
@Override
public <T> T unwrap(Class<T> iface) throws SQLException {
try {
return iface.cast(this);
} catch (ClassCastException cce) {
throw new SQLException("Unable to unwrap to " + iface.toString());
}
}
@Override
public boolean isWrapperFor(Class<?> iface) throws SQLException {
return iface.isInstance(this);
}
}

72
src/connector/jdbc/src/main/java/com/taosdata/jdbc/rs/RestfulStatement.java
View File

@ -4,17 +4,14 @@ import com.alibaba.fastjson.JSON;
import com.alibaba.fastjson.JSONArray;
import com.alibaba.fastjson.JSONObject;
import com.taosdata.jdbc.AbstractStatement;
import com.taosdata.jdbc.TSDBConstants;
import com.taosdata.jdbc.TSDBError;
import com.taosdata.jdbc.TSDBErrorNumbers;
import com.taosdata.jdbc.utils.HttpClientPoolUtil;
import com.taosdata.jdbc.utils.SqlSyntaxValidator;
import java.sql.*;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.SQLException;
public class RestfulStatement extends AbstractStatement {
@ -30,39 +27,6 @@ public class RestfulStatement extends AbstractStatement {
this.database = database;
}
protected String[] parseTableIdentifier(String sql) {
sql = sql.trim().toLowerCase();
String[] ret = null;
if (sql.contains("where"))
sql = sql.substring(0, sql.indexOf("where"));
if (sql.contains("interval"))
sql = sql.substring(0, sql.indexOf("interval"));
if (sql.contains("fill"))
sql = sql.substring(0, sql.indexOf("fill"));
if (sql.contains("sliding"))
sql = sql.substring(0, sql.indexOf("sliding"));
if (sql.contains("group by"))
sql = sql.substring(0, sql.indexOf("group by"));
if (sql.contains("order by"))
sql = sql.substring(0, sql.indexOf("order by"));
if (sql.contains("slimit"))
sql = sql.substring(0, sql.indexOf("slimit"));
if (sql.contains("limit"))
sql = sql.substring(0, sql.indexOf("limit"));
// parse
if (sql.contains("from")) {
sql = sql.substring(sql.indexOf("from") + 4).trim();
return Arrays.asList(sql.split(",")).stream()
.map(tableIdentifier -> {
tableIdentifier = tableIdentifier.trim();
if (tableIdentifier.contains(" "))
tableIdentifier = tableIdentifier.substring(0, tableIdentifier.indexOf(" "));
return tableIdentifier;
}).collect(Collectors.joining(",")).split(",");
}
return ret;
}
@Override
public ResultSet executeQuery(String sql) throws SQLException {
if (isClosed())
@ -75,9 +39,8 @@ public class RestfulStatement extends AbstractStatement {
return executeOneQuery(url, sql);
}
// if (this.database == null || this.database.isEmpty())
// throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_DATABASE_NOT_SPECIFIED_OR_AVAILABLE);
HttpClientPoolUtil.execute(url, "use " + this.database);
// if (this.database != null && !this.database.trim().replaceAll("\\s","").isEmpty())
// HttpClientPoolUtil.execute(url, "use " + this.database);
return executeOneQuery(url, sql);
}
@ -93,10 +56,8 @@ public class RestfulStatement extends AbstractStatement {
return executeOneUpdate(url, sql);
}
// if (this.database == null || this.database.isEmpty())
// throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_DATABASE_NOT_SPECIFIED_OR_AVAILABLE);
HttpClientPoolUtil.execute(url, "use " + this.database);
// if (this.database != null && !this.database.trim().replaceAll("\\s", "").isEmpty())
// HttpClientPoolUtil.execute(url, "use " + this.database);
return executeOneUpdate(url, sql);
}
@ -148,24 +109,9 @@ public class RestfulStatement extends AbstractStatement {
String result = HttpClientPoolUtil.execute(url, sql);
JSONObject resultJson = JSON.parseObject(result);
if (resultJson.getString("status").equals("error")) {
throw new SQLException(TSDBConstants.WrapErrMsg("SQL execution error: " + resultJson.getString("desc") + "\n" + "error code: " + resultJson.getString("code")));
throw TSDBError.createSQLException(resultJson.getInteger("code"), resultJson.getString("desc"));
}
// parse table name from sql
// String[] tableIdentifiers = parseTableIdentifier(sql);
// if (tableIdentifiers != null) {
// List<JSONObject> fieldJsonList = new ArrayList<>();
// for (String tableIdentifier : tableIdentifiers) {
// String fields = HttpClientPoolUtil.execute(url, "DESCRIBE " + tableIdentifier);
// JSONObject fieldJson = JSON.parseObject(fields);
// if (fieldJson.getString("status").equals("error")) {
// throw new SQLException(TSDBConstants.WrapErrMsg("SQL execution error: " + fieldJson.getString("desc") + "\n" + "error code: " + fieldJson.getString("code")));
// }
// fieldJsonList.add(fieldJson);
// }
// this.resultSet = new RestfulResultSet(database, this, resultJson, fieldJsonList);
// } else {
this.resultSet = new RestfulResultSet(database, this, resultJson);
// }
this.affectedRows = 0;
return resultSet;
}
@ -177,7 +123,7 @@ public class RestfulStatement extends AbstractStatement {
String result = HttpClientPoolUtil.execute(url, sql);
JSONObject jsonObject = JSON.parseObject(result);
if (jsonObject.getString("status").equals("error")) {
throw new SQLException(TSDBConstants.WrapErrMsg("SQL execution error: " + jsonObject.getString("desc") + "\n" + "error code: " + jsonObject.getString("code")));
throw TSDBError.createSQLException(jsonObject.getInteger("code"), jsonObject.getString("desc"));
}
this.resultSet = null;
this.affectedRows = checkJsonResultSet(jsonObject);

12
src/connector/jdbc/src/test/java/com/taosdata/jdbc/TSDBJNIConnectorTest.java
View File

@ -15,12 +15,12 @@ public class TSDBJNIConnectorTest {
public void test() {
try {
// init
TSDBJNIConnector.init(null, null, null, null);
TSDBJNIConnector.init("/etc/taos/taos.cfg", null, null, null);
// connect
TSDBJNIConnector connector = new TSDBJNIConnector();
connector.connect("127.0.0.1", 6030, null, "root", "taosdata");
connector.connect("127.0.0.1", 6030, "unsign_jni", "root", "taosdata");
// executeQuery
long pSql = connector.executeQuery("show variables");
long pSql = connector.executeQuery("select * from unsign_jni.us_table");
if (connector.isUpdateQuery(pSql)) {
connector.freeResultSet(pSql);
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_INVALID_WITH_EXECUTEQUERY);
@ -29,13 +29,13 @@ public class TSDBJNIConnectorTest {
List<ColumnMetaData> columnMetaDataList = new ArrayList<>();
int code = connector.getSchemaMetaData(pSql, columnMetaDataList);
if (code == TSDBConstants.JNI_CONNECTION_NULL) {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_CONNECTION_NULL));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_CONNECTION_NULL);
}
if (code == TSDBConstants.JNI_RESULT_SET_NULL) {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_RESULT_SET_NULL));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_RESULT_SET_NULL);
}
if (code == TSDBConstants.JNI_NUM_OF_FIELDS_0) {
throw new SQLException(TSDBConstants.FixErrMsg(TSDBConstants.JNI_NUM_OF_FIELDS_0));
throw TSDBError.createSQLException(TSDBErrorNumbers.ERROR_JNI_NUM_OF_FIELDS_0);
}
int columnSize = columnMetaDataList.size();
// print metadata

92
src/connector/jdbc/src/test/java/com/taosdata/jdbc/cases/InsertDbwithoutUseDbTest.java Normal file
View File

@ -0,0 +1,92 @@
package com.taosdata.jdbc.cases;
import org.junit.Assert;
import org.junit.BeforeClass;
import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;
import java.sql.*;
import java.util.Properties;
import java.util.Random;
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class InsertDbwithoutUseDbTest {
private static String host = "127.0.0.1";
// private static String host = "master";
private static Properties properties;
private static Random random = new Random(System.currentTimeMillis());
@Test
public void case001() throws ClassNotFoundException, SQLException {
// prepare schema
Class.forName("com.taosdata.jdbc.TSDBDriver");
String url = "jdbc:TAOS://127.0.0.1:6030/?user=root&password=taosdata";
Connection conn = DriverManager.getConnection(url, properties);
try (Statement stmt = conn.createStatement()) {
stmt.execute("drop database if exists inWithoutDb");
stmt.execute("create database if not exists inWithoutDb");
stmt.execute("create table inWithoutDb.weather(ts timestamp, f1 int)");
}
conn.close();
// execute insert
url = "jdbc:TAOS://127.0.0.1:6030/inWithoutDb?user=root&password=taosdata";
conn = DriverManager.getConnection(url, properties);
try (Statement stmt = conn.createStatement()) {
int affectedRow = stmt.executeUpdate("insert into weather(ts, f1) values(now," + random.nextInt(100) + ")");
Assert.assertEquals(1, affectedRow);
boolean flag = stmt.execute("insert into weather(ts, f1) values(now + 10s," + random.nextInt(100) + ")");
Assert.assertEquals(false, flag);
ResultSet rs = stmt.executeQuery("select count(*) from weather");
rs.next();
int count = rs.getInt("count(*)");
Assert.assertEquals(2, count);
} catch (SQLException e) {
e.printStackTrace();
}
conn.close();
}
@Test
public void case002() throws ClassNotFoundException, SQLException {
// prepare the schema
Class.forName("com.taosdata.jdbc.rs.RestfulDriver");
final String url = "jdbc:TAOS-RS://" + host + ":6041/inWithoutDb?user=root&password=taosdata";
Connection conn = DriverManager.getConnection(url, properties);
try (Statement stmt = conn.createStatement()) {
stmt.execute("drop database if exists inWithoutDb");
stmt.execute("create database if not exists inWithoutDb");
stmt.execute("create table inWithoutDb.weather(ts timestamp, f1 int)");
}
conn.close();
// execute
conn = DriverManager.getConnection(url, properties);
try (Statement stmt = conn.createStatement()) {
int affectedRow = stmt.executeUpdate("insert into weather(ts, f1) values(now," + random.nextInt(100) + ")");
Assert.assertEquals(1, affectedRow);
boolean flag = stmt.execute("insert into weather(ts, f1) values(now + 10s," + random.nextInt(100) + ")");
Assert.assertEquals(false, flag);
ResultSet rs = stmt.executeQuery("select count(*) from weather");
rs.next();
int count = rs.getInt("count(*)");
Assert.assertEquals(2, count);
} catch (SQLException e) {
e.printStackTrace();
}
}
@BeforeClass
public static void beforeClass() {
properties = new Properties();
properties.setProperty("charset", "UTF-8");
properties.setProperty("locale", "en_US.UTF-8");
properties.setProperty("timezone", "UTC-8");
}
}

191
src/connector/jdbc/src/test/java/com/taosdata/jdbc/cases/UnsignedNumberJniTest.java Normal file
View File

@ -0,0 +1,191 @@
package com.taosdata.jdbc.cases;
import com.taosdata.jdbc.TSDBDriver;
import org.junit.*;
import org.junit.runners.MethodSorters;
import java.sql.*;
import java.util.Properties;
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class UnsignedNumberJniTest {
private static final String host = "127.0.0.1";
private static Connection conn;
@Test
public void testCase001() {
try (Statement stmt = conn.createStatement()) {
ResultSet rs = stmt.executeQuery("select * from us_table");
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
for (int i = 1; i <= meta.getColumnCount(); i++) {
System.out.print(meta.getColumnLabel(i) + ": " + rs.getString(i) + "\t");
}
System.out.println();
Assert.assertEquals("127", rs.getString(2));
Assert.assertEquals("32767", rs.getString(3));
Assert.assertEquals("2147483647", rs.getString(4));
Assert.assertEquals("9223372036854775807", rs.getString(5));
}
} catch (SQLException e) {
e.printStackTrace();
}
}
@Test
public void testCase002() {
try (Statement stmt = conn.createStatement()) {
ResultSet rs = stmt.executeQuery("select * from us_table");
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
Assert.assertEquals(127, rs.getByte(2));
Assert.assertEquals(32767, rs.getShort(3));
Assert.assertEquals(2147483647, rs.getInt(4));
Assert.assertEquals(9223372036854775807l, rs.getLong(5));
}
} catch (SQLException e) {
e.printStackTrace();
}
}
@Test(expected = SQLException.class)
public void testCase003() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 127, 32767,2147483647, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
Assert.assertEquals(127, rs.getByte(2));
Assert.assertEquals(32767, rs.getShort(3));
Assert.assertEquals(2147483647, rs.getInt(4));
}
}
}
@Test(expected = SQLException.class)
public void testCase004() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 127, 32767,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
Assert.assertEquals(127, rs.getByte(2));
Assert.assertEquals(32767, rs.getShort(3));
}
}
}
@Test(expected = SQLException.class)
public void testCase005() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 127, 65534,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
Assert.assertEquals(127, rs.getByte(2));
}
}
}
@Test(expected = SQLException.class)
public void testCase006() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 254, 65534,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
}
}
}
@Test
public void testCase007() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 254, 65534,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
for (int i = 1; i <= meta.getColumnCount(); i++) {
System.out.print(meta.getColumnLabel(i) + ": " + rs.getString(i) + "\t");
}
System.out.println();
Assert.assertEquals("254", rs.getString(2));
Assert.assertEquals("65534", rs.getString(3));
Assert.assertEquals("4294967294", rs.getString(4));
Assert.assertEquals("18446744073709551614", rs.getString(5));
}
}
}
@BeforeClass
public static void beforeClass() {
Properties properties = new Properties();
properties.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_LOCALE, "en_US.UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8");
try {
Class.forName("com.taosdata.jdbc.TSDBDriver");
final String url = "jdbc:TAOS://" + host + ":6030/?user=root&password=taosdata";
conn = DriverManager.getConnection(url, properties);
Statement stmt = conn.createStatement();
stmt.execute("drop database if exists unsign_jni");
stmt.execute("create database if not exists unsign_jni");
stmt.execute("use unsign_jni");
stmt.execute("create table us_table(ts timestamp, f1 tinyint unsigned, f2 smallint unsigned, f3 int unsigned, f4 bigint unsigned)");
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(now, 127, 32767,2147483647, 9223372036854775807)");
stmt.close();
} catch (ClassNotFoundException | SQLException e) {
e.printStackTrace();
}
}
@AfterClass
public static void afterClass() {
try {
if (conn != null)
conn.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}

177
src/connector/jdbc/src/test/java/com/taosdata/jdbc/cases/UnsignedNumberRestfulTest.java Normal file
View File

@ -0,0 +1,177 @@
package com.taosdata.jdbc.cases;
import com.taosdata.jdbc.TSDBDriver;
import org.junit.*;
import org.junit.runners.MethodSorters;
import java.sql.*;
import java.util.Properties;
@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class UnsignedNumberRestfulTest {
private static final String host = "127.0.0.1";
// private static final String host = "master";
private static Connection conn;
@Test
public void testCase001() {
try (Statement stmt = conn.createStatement()) {
ResultSet rs = stmt.executeQuery("select * from us_table");
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
for (int i = 1; i <= meta.getColumnCount(); i++) {
System.out.print(meta.getColumnLabel(i) + ": " + rs.getString(i) + "\t");
}
System.out.println();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
@Test
public void testCase002() {
try (Statement stmt = conn.createStatement()) {
ResultSet rs = stmt.executeQuery("select * from us_table");
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
}
} catch (SQLException e) {
e.printStackTrace();
}
}
@Test(expected = SQLException.class)
public void testCase003() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 127, 32767,2147483647, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
}
}
}
@Test(expected = SQLException.class)
public void testCase004() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 127, 32767,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
}
}
}
@Test(expected = SQLException.class)
public void testCase005() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 127, 65534,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
}
}
}
@Test(expected = SQLException.class)
public void testCase006() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 254, 65534,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
System.out.print(meta.getColumnLabel(1) + ": " + rs.getTimestamp(1) + "\t");
System.out.print(meta.getColumnLabel(2) + ": " + rs.getByte(2) + "\t");
System.out.print(meta.getColumnLabel(3) + ": " + rs.getShort(3) + "\t");
System.out.print(meta.getColumnLabel(4) + ": " + rs.getInt(4) + "\t");
System.out.print(meta.getColumnLabel(5) + ": " + rs.getLong(5) + "\t");
System.out.println();
}
}
}
@Test
public void testCase007() throws SQLException {
try (Statement stmt = conn.createStatement()) {
long now = System.currentTimeMillis();
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(" + now + ", 254, 65534,4294967294, 18446744073709551614)");
ResultSet rs = stmt.executeQuery("select * from us_table where ts = " + now);
ResultSetMetaData meta = rs.getMetaData();
while (rs.next()) {
for (int i = 1; i <= meta.getColumnCount(); i++) {
System.out.print(meta.getColumnLabel(i) + ": " + rs.getString(i) + "\t");
}
System.out.println();
Assert.assertEquals("254", rs.getString(2));
Assert.assertEquals("65534", rs.getString(3));
Assert.assertEquals("4294967294", rs.getString(4));
Assert.assertEquals("18446744073709551614", rs.getString(5));
}
}
}
@BeforeClass
public static void beforeClass() {
Properties properties = new Properties();
properties.setProperty(TSDBDriver.PROPERTY_KEY_CHARSET, "UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_LOCALE, "en_US.UTF-8");
properties.setProperty(TSDBDriver.PROPERTY_KEY_TIME_ZONE, "UTC-8");
try {
Class.forName("com.taosdata.jdbc.rs.RestfulDriver");
final String url = "jdbc:TAOS-RS://" + host + ":6041/?user=root&password=taosdata";
conn = DriverManager.getConnection(url, properties);
Statement stmt = conn.createStatement();
stmt.execute("drop database if exists unsign_restful");
stmt.execute("create database if not exists unsign_restful");
stmt.execute("use unsign_restful");
stmt.execute("create table us_table(ts timestamp, f1 tinyint unsigned, f2 smallint unsigned, f3 int unsigned, f4 bigint unsigned)");
stmt.executeUpdate("insert into us_table(ts,f1,f2,f3,f4) values(now, 127, 32767,2147483647, 9223372036854775807)");
stmt.close();
} catch (ClassNotFoundException | SQLException e) {
e.printStackTrace();
}
}
@AfterClass
public static void afterClass() {
try {
if (conn != null)
conn.close();
} catch (SQLException e) {
e.printStackTrace();
}
}
}

7
src/connector/jdbc/src/test/java/com/taosdata/jdbc/rs/RestfulResultSetTest.java
View File

@ -40,9 +40,12 @@ public class RestfulResultSetTest {
Assert.assertEquals(true, f9);
}
@Test(expected = SQLFeatureNotSupportedException.class)
@Test
public void getByte() throws SQLException {
rs.getByte(1);
byte f8 = rs.getByte("f8");
Assert.assertEquals(10, f8);
f8 = rs.getByte(8);
Assert.assertEquals(10, f8);
}
@Test

2
src/dnode/src/dnodeVWrite.c
View File

@ -222,7 +222,7 @@ static void *dnodeProcessVWriteQueue(void *wparam) {
dnodeSendRpcVWriteRsp(pVnode, pWrite, pWrite->code);
} else {
if (qtype == TAOS_QTYPE_FWD) {
vnodeConfirmForward(pVnode, pWrite->pHead.version, 0);
vnodeConfirmForward(pVnode, pWrite->pHead.version, 0, pWrite->pHead.msgType != TSDB_MSG_TYPE_SUBMIT);
}
if (pWrite->rspRet.rsp) {
rpcFreeCont(pWrite->rspRet.rsp);

5
src/inc/query.h
View File

@ -28,7 +28,7 @@ typedef void* qinfo_t;
* @param qinfo
* @return
*/
int32_t qCreateQueryInfo(void* tsdb, int32_t vgId, SQueryTableMsg* pQueryTableMsg, qinfo_t* qinfo);
int32_t qCreateQueryInfo(void* tsdb, int32_t vgId, SQueryTableMsg* pQueryTableMsg, qinfo_t* qinfo, uint64_t *qId);
/**
@ -88,9 +88,10 @@ void* qOpenQueryMgmt(int32_t vgId);
void qQueryMgmtNotifyClosed(void* pExecutor);
void qQueryMgmtReOpen(void *pExecutor);
void qCleanupQueryMgmt(void* pExecutor);
void** qRegisterQInfo(void* pMgmt, uint64_t qInfo);
void** qRegisterQInfo(void* pMgmt, uint64_t qId, uint64_t qInfo);
void** qAcquireQInfo(void* pMgmt, uint64_t key);
void** qReleaseQInfo(void* pMgmt, void* pQInfo, bool freeHandle);
bool checkQIdEqual(void *qHandle, uint64_t qId);
#ifdef __cplusplus
}

2
src/inc/tp.h
View File

@ -22,7 +22,7 @@ extern "C" {
int32_t tpInit();
void tpCleanUp();
void tpUpdateTs(int32_t *seq, void *pMsg);
void tpUpdateTs(int32_t vgId, int64_t *seq, void *pMsg);
#ifdef __cplusplus
}

28
src/inc/tsdb.h
View File

@ -25,6 +25,8 @@
#include "tdataformat.h"
#include "tname.h"
#include "hash.h"
#include "tlockfree.h"
#include "tlist.h"
#ifdef __cplusplus
extern "C" {
@ -172,10 +174,32 @@ typedef struct STsdbQueryCond {
int32_t type; // data block load type:
} STsdbQueryCond;
typedef struct STableData STableData;
typedef struct {
T_REF_DECLARE()
SRWLatch latch;
TSKEY keyFirst;
TSKEY keyLast;
int64_t numOfRows;
int32_t maxTables;
STableData **tData;
SList * actList;
SList * extraBuffList;
SList * bufBlockList;
int64_t pointsAdd; // TODO
int64_t storageAdd; // TODO
} SMemTable;
typedef struct {
SMemTable* mem;
SMemTable* imem;
SMemTable mtable;
SMemTable* omem;
} SMemSnapshot;
typedef struct SMemRef {
int32_t ref;
void * mem;
void * imem;
SMemSnapshot snapshot;
} SMemRef;
typedef struct SDataBlockInfo {

8
src/inc/tsync.h
View File

@ -79,6 +79,9 @@ typedef void (*FStopSyncFile)(int32_t vgId, uint64_t fversion);
// get file version
typedef int32_t (*FGetVersion)(int32_t vgId, uint64_t *fver, uint64_t *vver);
// reset version
typedef int32_t (*FResetVersion)(int32_t vgId, uint64_t fver);
typedef int32_t (*FSendFile)(void *tsdb, SOCKET socketFd);
typedef int32_t (*FRecvFile)(void *tsdb, SOCKET socketFd);
@ -96,6 +99,7 @@ typedef struct {
FStartSyncFile startSyncFileFp;
FStopSyncFile stopSyncFileFp;
FGetVersion getVersionFp;
FResetVersion resetVersionFp;
FSendFile sendFileFp;
FRecvFile recvFileFp;
} SSyncInfo;
@ -108,8 +112,8 @@ void syncCleanUp();
int64_t syncStart(const SSyncInfo *);
void syncStop(int64_t rid);
int32_t syncReconfig(int64_t rid, const SSyncCfg *);
int32_t syncForwardToPeer(int64_t rid, void *pHead, void *mhandle, int32_t qtype);
void syncConfirmForward(int64_t rid, uint64_t version, int32_t code);
int32_t syncForwardToPeer(int64_t rid, void *pHead, void *mhandle, int32_t qtype, bool force);
void syncConfirmForward(int64_t rid, uint64_t version, int32_t code, bool force);
void syncRecover(int64_t rid); // recover from other nodes:
int32_t syncGetNodesRole(int64_t rid, SNodesRole *);

12
src/inc/ttokendef.h
View File

@ -197,13 +197,11 @@
#define TK_PREV 178
#define TK_LINEAR 179
#define TK_IMPORT 180
#define TK_METRIC 181
#define TK_TBNAME 182
#define TK_JOIN 183
#define TK_METRICS 184
#define TK_INSERT 185
#define TK_INTO 186
#define TK_VALUES 187
#define TK_TBNAME 181
#define TK_JOIN 182
#define TK_INSERT 183
#define TK_INTO 184
#define TK_VALUES 185
#define TK_SPACE 300

1
src/inc/twal.h
View File

@ -65,6 +65,7 @@ void walFsync(twalh, bool forceFsync);
int32_t walRestore(twalh, void *pVnode, FWalWrite writeFp);
int32_t walGetWalFile(twalh, char *fileName, int64_t *fileId);
uint64_t walGetVersion(twalh);
void walResetVersion(twalh, uint64_t newVer);
#ifdef __cplusplus
}

2
src/inc/vnode.h
View File

@ -78,7 +78,7 @@ void vnodeFreeFromWQueue(void *pVnode, SVWriteMsg *pWrite);
int32_t vnodeProcessWrite(void *pVnode, void *pHead, int32_t qtype, void *pRspRet);
// vnodeSync
void vnodeConfirmForward(void *pVnode, uint64_t version, int32_t code);
void vnodeConfirmForward(void *pVnode, uint64_t version, int32_t code, bool force);
// vnodeRead
int32_t vnodeWriteToRQueue(void *pVnode, void *pCont, int32_t contLen, int8_t qtype, void *rparam);

1
src/kit/shell/CMakeLists.txt
View File

@ -36,6 +36,7 @@ ELSEIF (TD_DARWIN)
LIST(APPEND SRC ./src/shellDarwin.c)
LIST(APPEND SRC ./src/shellCommand.c)
LIST(APPEND SRC ./src/shellImport.c)
LIST(APPEND SRC ./src/shellCheck.c)
ADD_EXECUTABLE(shell ${SRC})
# linking with dylib
TARGET_LINK_LIBRARIES(shell taos)

2
src/kit/shell/inc/shell.h
View File

@ -51,6 +51,7 @@ typedef struct SShellArguments {
char file[TSDB_FILENAME_LEN];
char dir[TSDB_FILENAME_LEN];
int threadNum;
int check;
char* commands;
int abort;
int port;
@ -71,6 +72,7 @@ void read_history();
void write_history();
void source_file(TAOS* con, char* fptr);
void source_dir(TAOS* con, SShellArguments* args);
void shellCheck(TAOS* con, SShellArguments* args);
void get_history_path(char* history);
void cleanup_handler(void* arg);
void exitShell();

199
src/kit/shell/src/shellCheck.c Normal file
View File

@ -0,0 +1,199 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#define _GNU_SOURCE
#define _XOPEN_SOURCE
#define _DEFAULT_SOURCE
#include "os.h"
#include "shell.h"
#include "shellCommand.h"
#include "tglobal.h"
#include "tutil.h"
#define SHELL_SQL_LEN 1024
static int32_t tbNum = 0;
static int32_t tbMallocNum = 0;
static char ** tbNames = NULL;
static int32_t checkedNum = 0;
static int32_t errorNum = 0;
typedef struct {
pthread_t threadID;
int threadIndex;
int totalThreads;
void * taos;
char * db;
} ShellThreadObj;
static int32_t shellUseDb(TAOS *con, char *db) {
if (db == NULL) {
fprintf(stdout, "no dbname input\n");
return -1;
}
char sql[SHELL_SQL_LEN] = {0};
snprintf(sql, SHELL_SQL_LEN, "use %s", db);
TAOS_RES *pSql = taos_query(con, sql);
int32_t code = taos_errno(pSql);
if (code != 0) {
fprintf(stdout, "failed to execute sql:%s since %s", sql, taos_errstr(pSql));
}
taos_free_result(pSql);
return code;
}
static int32_t shellShowTables(TAOS *con, char *db) {
char sql[SHELL_SQL_LEN] = {0};
snprintf(sql, SHELL_SQL_LEN, "show %s.tables", db);
TAOS_RES *pSql = taos_query(con, sql);
int32_t code = taos_errno(pSql);
if (code != 0) {
fprintf(stdout, "failed to execute sql:%s since %s\n", sql, taos_errstr(pSql));
} else {
TAOS_ROW row;
while ((row = taos_fetch_row(pSql))) {
int32_t tbIndex = tbNum++;
if (tbMallocNum < tbNum) {
tbMallocNum = (tbMallocNum * 2 + 1);
tbNames = realloc(tbNames, tbMallocNum * sizeof(char *));
if (tbNames == NULL) {
fprintf(stdout, "failed to malloc tablenames, num:%d\n", tbMallocNum);
code = TSDB_CODE_TSC_OUT_OF_MEMORY;
break;
}
}
tbNames[tbIndex] = malloc(TSDB_TABLE_NAME_LEN);
strncpy(tbNames[tbIndex], (const char *)row[0], TSDB_TABLE_NAME_LEN);
if (tbIndex % 100000 == 0 && tbIndex != 0) {
fprintf(stdout, "%d tablenames fetched\n", tbIndex);
}
}
}
taos_free_result(pSql);
fprintf(stdout, "total %d tablenames fetched, over\n", tbNum);
return code;
}
static void shellFreeTbnames() {
for (int32_t i = 0; i < tbNum; ++i) {
free(tbNames[i]);
}
free(tbNames);
}
static void *shellCheckThreadFp(void *arg) {
ShellThreadObj *pThread = (ShellThreadObj *)arg;
int32_t interval = tbNum / pThread->totalThreads + 1;
int32_t start = pThread->threadIndex * interval;
int32_t end = (pThread->threadIndex + 1) * interval;
if (end > tbNum) end = tbNum + 1;
char file[32] = {0};
snprintf(file, 32, "tb%d.txt", pThread->threadIndex);
FILE *fp = fopen(file, "w");
if (!fp) {
fprintf(stdout, "failed to open %s, reason:%s", file, strerror(errno));
return NULL;
}
char sql[SHELL_SQL_LEN];
for (int32_t t = start; t < end; ++t) {
char *tbname = tbNames[t];
if (tbname == NULL) break;
snprintf(sql, SHELL_SQL_LEN, "select * from %s limit 1", tbname);
TAOS_RES *pSql = taos_query(pThread->taos, sql);
int32_t code = taos_errno(pSql);
if (code != 0) {
int32_t len = snprintf(sql, SHELL_SQL_LEN, "drop table %s.%s;\n", pThread->db, tbname);
fwrite(sql, 1, len, fp);
atomic_add_fetch_32(&errorNum, 1);
}
int32_t cnum = atomic_add_fetch_32(&checkedNum, 1);
if (cnum % 5000 == 0 && cnum != 0) {
fprintf(stdout, "%d tables checked\n", cnum);
}
taos_free_result(pSql);
}
fsync(fileno(fp));
fclose(fp);
return NULL;
}
static void shellRunCheckThreads(TAOS *con, SShellArguments *args) {
pthread_attr_t thattr;
ShellThreadObj *threadObj = (ShellThreadObj *)calloc(args->threadNum, sizeof(ShellThreadObj));
for (int t = 0; t < args->threadNum; ++t) {
ShellThreadObj *pThread = threadObj + t;
pThread->threadIndex = t;
pThread->totalThreads = args->threadNum;
pThread->taos = con;
pThread->db = args->database;
pthread_attr_init(&thattr);
pthread_attr_setdetachstate(&thattr, PTHREAD_CREATE_JOINABLE);
if (pthread_create(&(pThread->threadID), &thattr, shellCheckThreadFp, (void *)pThread) != 0) {
fprintf(stderr, "ERROR: thread:%d failed to start\n", pThread->threadIndex);
exit(0);
}
}
for (int t = 0; t < args->threadNum; ++t) {
pthread_join(threadObj[t].threadID, NULL);
}
for (int t = 0; t < args->threadNum; ++t) {
taos_close(threadObj[t].taos);
}
free(threadObj);
}
void shellCheck(TAOS *con, SShellArguments *args) {
int64_t start = taosGetTimestampMs();
if (shellUseDb(con, args->database) != 0) {
shellFreeTbnames();
return;
}
if (shellShowTables(con, args->database) != 0) {
shellFreeTbnames();
return;
}
fprintf(stdout, "total %d tables will be checked by %d threads\n", tbNum, args->threadNum);
shellRunCheckThreads(con, args);
int64_t end = taosGetTimestampMs();
fprintf(stdout, "total %d tables checked, failed:%d, time spent %.2f seconds\n", checkedNum, errorNum,
(end - start) / 1000.0);
}

8
src/kit/shell/src/shellEngine.c
View File

@ -121,6 +121,12 @@ TAOS *shellInit(SShellArguments *args) {
taos_close(con);
exit(EXIT_SUCCESS);
}
if (args->check != 0) {
shellCheck(con, args);
taos_close(con);
exit(EXIT_SUCCESS);
}
#endif
return con;
@ -412,7 +418,7 @@ static char* formatTimestamp(char* buf, int64_t val, int precision) {
#ifdef WINDOWS
if (tt < 0) tt = 0;
#endif
if (tt < 0 && ms != 0) {
if (tt <= 0 && ms < 0) {
tt--;
if (precision == TSDB_TIME_PRECISION_MICRO) {
ms += 1000000;

4
src/kit/shell/src/shellLinux.c
View File

@ -45,6 +45,7 @@ static struct argp_option options[] = {
{"file", 'f', "FILE", 0, "Script to run without enter the shell."},
{"directory", 'D', "DIRECTORY", 0, "Use multi-thread to import all SQL files in the directory separately."},
{"thread", 'T', "THREADNUM", 0, "Number of threads when using multi-thread to import data."},
{"check", 'k', "CHECK", 0, "Check tables."},
{"database", 'd', "DATABASE", 0, "Database to use when connecting to the server."},
{"timezone", 't', "TIMEZONE", 0, "Time zone of the shell, default is local."},
{"netrole", 'n', "NETROLE", 0, "Net role when network connectivity test, default is startup, options: client|server|rpc|startup|sync."},
@ -130,6 +131,9 @@ static error_t parse_opt(int key, char *arg, struct argp_state *state) {
return -1;
}
break;
case 'k':
arguments->check = atoi(arg);
break;
case 'd':
arguments->database = arg;
break;

57
src/kit/taosdemo/insert-interlace.json Normal file
View File

@ -0,0 +1,57 @@
{
"filetype": "insert",
"cfgdir": "/etc/taos",
"host": "127.0.0.1",
"port": 6030,
"user": "root",
"password": "taosdata",
"thread_count": 4,
"thread_count_create_tbl": 4,
"result_file": "./insert_res.txt",
"confirm_parameter_prompt": "no",
"insert_interval": 1000,
"num_of_records_per_req": 100,
"max_sql_len": 1024000,
"databases": [{
"dbinfo": {
"name": "db",
"drop": "yes",
"replica": 1,
"days": 10,
"cache": 16,
"blocks": 8,
"precision": "ms",
"keep": 365,
"minRows": 100,
"maxRows": 4096,
"comp":2,
"walLevel":1,
"cachelast":0,
"quorum":1,
"fsync":3000,
"update": 0
},
"super_tables": [{
"name": "stb",
"child_table_exists":"no",
"childtable_count": 100,
"childtable_prefix": "stb_",
"auto_create_table": "no",
"data_source": "rand",
"insert_mode": "taosc",
"insert_rows": 1000,
"multi_thread_write_one_tbl": "no",
"rows_per_tbl": 20,
"max_sql_len": 1024000,
"disorder_ratio": 0,
"disorder_range": 1000,
"timestamp_step": 1,
"start_timestamp": "2020-10-01 00:00:00.000",
"sample_format": "csv",
"sample_file": "./sample.csv",
"tags_file": "",
"columns": [{"type": "INT"}],
"tags": [{"type": "TINYINT", "count":1}]
}]
}]
}

3
src/kit/taosdemo/insert.json
View File

@ -41,8 +41,7 @@
"insert_mode": "taosc",
"insert_rows": 100000,
"multi_thread_write_one_tbl": "no",
"number_of_tbl_in_one_sql": 0,
"rows_per_tbl": 100,
"rows_per_tbl": 0,
"max_sql_len": 1024000,
"disorder_ratio": 0,
"disorder_range": 1000,

1332
src/kit/taosdemo/taosdemo.c
View File

File diff suppressed because it is too large Load Diff

2
src/mnode/inc/mnodeVgroup.h
View File

@ -44,7 +44,7 @@ void mnodeDropVgroup(SVgObj *pVgroup, void *ahandle);
void mnodeAlterVgroup(SVgObj *pVgroup, void *ahandle);
int32_t mnodeGetAvailableVgroup(struct SMnodeMsg *pMsg, SVgObj **pVgroup, int32_t *sid);
void mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable);
int32_t mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable, bool needCheck);
void mnodeRemoveTableFromVgroup(SVgObj *pVgroup, SCTableObj *pTable);
void mnodeSendDropVnodeMsg(int32_t vgId, SRpcEpSet *epSet, void *ahandle);
void mnodeSendCreateVgroupMsg(SVgObj *pVgroup, void *ahandle);

2
src/mnode/src/mnodeDb.c
View File

@ -55,7 +55,7 @@ int32_t mnodeProcessAlterDbMsg(SMnodeMsg *pMsg);
#ifndef _TOPIC
int32_t tpInit() { return 0; }
void tpCleanUp() {}
void tpUpdateTs(int32_t *seq, void *pMsg) {}
void tpUpdateTs(int32_t vgId, int64_t *seq, void *pMsg) {}
#endif
static void mnodeDestroyDb(SDbObj *pDb) {

8
src/mnode/src/mnodeSdb.c
View File

@ -552,7 +552,7 @@ static int32_t sdbInsertHash(SSdbTable *pTable, SSdbRow *pRow) {
int32_t code = (*pTable->fpInsert)(pRow);
if (code != TSDB_CODE_SUCCESS) {
sdbError("vgId:1, sdb:%s, failed to insert key:%s to hash, remove it", pTable->name,
sdbError("vgId:1, sdb:%s, failed to perform insert action for key:%s, remove it", pTable->name,
sdbGetRowStr(pTable, pRow->pObj));
sdbDeleteHash(pTable, pRow);
}
@ -680,7 +680,7 @@ static int32_t sdbProcessWrite(void *wparam, void *hparam, int32_t qtype, void *
if (pRow != NULL) {
// forward to peers
pRow->processedCount = 0;
int32_t syncCode = syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC);
int32_t syncCode = syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC, false);
if (syncCode <= 0) pRow->processedCount = 1;
if (syncCode < 0) {
@ -700,7 +700,7 @@ static int32_t sdbProcessWrite(void *wparam, void *hparam, int32_t qtype, void *
actStr[action], sdbGetKeyStr(pTable, pHead->cont), pHead->version);
// even it is WAL/FWD, it shall be called to update version in sync
syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC);
syncForwardToPeer(tsSdbMgmt.sync, pHead, pRow, TAOS_QTYPE_RPC, false);
// from wal or forward msg, row not created, should add into hash
if (action == SDB_ACTION_INSERT) {
@ -1119,7 +1119,7 @@ static void *sdbWorkerFp(void *pWorker) {
sdbConfirmForward(1, pRow, pRow->code);
} else {
if (qtype == TAOS_QTYPE_FWD) {
syncConfirmForward(tsSdbMgmt.sync, pRow->pHead.version, pRow->code);
syncConfirmForward(tsSdbMgmt.sync, pRow->pHead.version, pRow->code, false);
}
sdbFreeFromQueue(pRow);
}

41
src/mnode/src/mnodeTable.c
View File

@ -108,10 +108,12 @@ static int32_t mnodeChildTableActionDestroy(SSdbRow *pRow) {
static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
SCTableObj *pTable = pRow->pObj;
int32_t code = 0;
SVgObj *pVgroup = mnodeGetVgroup(pTable->vgId);
if (pVgroup == NULL) {
mError("ctable:%s, not in vgId:%d", pTable->info.tableId, pTable->vgId);
code = -1;
}
SDbObj *pDb = NULL;
@ -119,6 +121,7 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
pDb = mnodeGetDb(pVgroup->dbName);
if (pDb == NULL) {
mError("ctable:%s, vgId:%d not in db:%s", pTable->info.tableId, pVgroup->vgId, pVgroup->dbName);
code = -1;
}
}
@ -127,6 +130,7 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
pAcct = mnodeGetAcct(pDb->acct);
if (pAcct == NULL) {
mError("ctable:%s, acct:%s not exists", pTable->info.tableId, pDb->acct);
code = -1;
}
}
@ -139,6 +143,7 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
if (pAcct) pAcct->acctInfo.numOfTimeSeries += (pTable->superTable->numOfColumns - 1);
} else {
mError("table:%s:%p, correspond stable not found suid:%" PRIu64, pTable->info.tableId, pTable, pTable->suid);
code = -1;
}
} else {
grantAdd(TSDB_GRANT_TIMESERIES, pTable->numOfColumns - 1);
@ -146,18 +151,31 @@ static int32_t mnodeChildTableActionInsert(SSdbRow *pRow) {
}
if (pDb) mnodeAddTableIntoDb(pDb);
if (pVgroup) mnodeAddTableIntoVgroup(pVgroup, pTable);
if (pVgroup) {
if (mnodeAddTableIntoVgroup(pVgroup, pTable, pRow->pMsg == NULL) != 0) {
mError("table:%s, vgId:%d tid:%d, failed to perform insert action, uid:%" PRIu64 " suid:%" PRIu64,
pTable->info.tableId, pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
code = -1;
}
}
mnodeDecVgroupRef(pVgroup);
mnodeDecDbRef(pDb);
mnodeDecAcctRef(pAcct);
return TSDB_CODE_SUCCESS;
if (code == 0) {
mTrace("table:%s, vgId:%d tid:%d, perform insert action, uid:%" PRIu64 " suid:%" PRIu64, pTable->info.tableId,
pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
}
return code;
}
static int32_t mnodeChildTableActionDelete(SSdbRow *pRow) {
SCTableObj *pTable = pRow->pObj;
if (pTable->vgId == 0) {
mError("table:%s, vgId:%d tid:%d, failed to perform delete action, uid:%" PRIu64 " suid:%" PRIu64,
pTable->info.tableId, pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
return TSDB_CODE_MND_VGROUP_NOT_EXIST;
}
@ -188,6 +206,8 @@ static int32_t mnodeChildTableActionDelete(SSdbRow *pRow) {
mnodeDecDbRef(pDb);
mnodeDecAcctRef(pAcct);
mTrace("table:%s, vgId:%d tid:%d, perform delete action, uid:%" PRIu64 " suid:%" PRIu64, pTable->info.tableId,
pTable->vgId, pTable->tid, pTable->uid, pTable->suid);
return TSDB_CODE_SUCCESS;
}
@ -399,13 +419,13 @@ static void mnodeAddTableIntoStable(SSTableObj *pStable, SCTableObj *pCtable) {
if (pStable->vgHash == NULL) {
pStable->vgHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT), true, HASH_ENTRY_LOCK);
mDebug("table:%s, create hash:%p", pStable->info.tableId, pStable->vgHash);
mDebug("stable:%s, create vgId hash:%p", pStable->info.tableId, pStable->vgHash);
}
if (pStable->vgHash != NULL) {
if (taosHashGet(pStable->vgHash, &pCtable->vgId, sizeof(pCtable->vgId)) == NULL) {
taosHashPut(pStable->vgHash, &pCtable->vgId, sizeof(pCtable->vgId), &pCtable->vgId, sizeof(pCtable->vgId));
mDebug("table:%s, vgId:%d is put into stable hash:%p, sizeOfVgList:%d", pStable->info.tableId, pCtable->vgId,
mDebug("stable:%s, vgId:%d is put into stable vgId hash:%p, sizeOfVgList:%d", pStable->info.tableId, pCtable->vgId,
pStable->vgHash, taosHashGetSize(pStable->vgHash));
}
}
@ -450,6 +470,8 @@ static int32_t mnodeSuperTableActionInsert(SSdbRow *pRow) {
mnodeDecDbRef(pDb);
taosHashPut(tsSTableUidHash, &pStable->uid, sizeof(int64_t), &pStable, sizeof(int64_t));
mTrace("stable:%s, perform insert action, uid:%" PRIu64, pStable->info.tableId, pStable->uid);
return TSDB_CODE_SUCCESS;
}
@ -463,6 +485,8 @@ static int32_t mnodeSuperTableActionDelete(SSdbRow *pRow) {
mnodeDecDbRef(pDb);
taosHashRemove(tsSTableUidHash, &pStable->uid, sizeof(int64_t));
mTrace("stable:%s, perform delete action, uid:%" PRIu64, pStable->info.tableId, pStable->uid);
return TSDB_CODE_SUCCESS;
}
@ -1889,8 +1913,7 @@ static int32_t mnodeDoCreateChildTableCb(SMnodeMsg *pMsg, int32_t code) {
if (pMsg->pBatchMasterMsg) {
++pMsg->pBatchMasterMsg->successed;
if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received
>= pMsg->pBatchMasterMsg->expected) {
if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received >= pMsg->pBatchMasterMsg->expected) {
dnodeSendRpcMWriteRsp(pMsg->pBatchMasterMsg, code);
}
@ -1911,8 +1934,7 @@ static int32_t mnodeDoCreateChildTableCb(SMnodeMsg *pMsg, int32_t code) {
if (pMsg->pBatchMasterMsg) {
++pMsg->pBatchMasterMsg->received;
pMsg->pBatchMasterMsg->code = code;
if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received
>= pMsg->pBatchMasterMsg->expected) {
if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received >= pMsg->pBatchMasterMsg->expected) {
dnodeSendRpcMWriteRsp(pMsg->pBatchMasterMsg, code);
}
@ -2662,8 +2684,7 @@ static void mnodeProcessCreateChildTableRsp(SRpcMsg *rpcMsg) {
if (pMsg->pBatchMasterMsg) {
++pMsg->pBatchMasterMsg->successed;
if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received
>= pMsg->pBatchMasterMsg->expected) {
if (pMsg->pBatchMasterMsg->successed + pMsg->pBatchMasterMsg->received >= pMsg->pBatchMasterMsg->expected) {
dnodeSendRpcMWriteRsp(pMsg->pBatchMasterMsg, rpcMsg->code);
}

17
src/mnode/src/mnodeVgroup.c
View File

@ -443,6 +443,7 @@ int32_t mnodeGetAvailableVgroup(SMnodeMsg *pMsg, SVgObj **ppVgroup, int32_t *pSi
mDebug("msg:%p, app:%p db:%s, no enough sid in vgId:%d", pMsg, pMsg->rpcMsg.ahandle, pDb->name, pVgroup->vgId);
continue;
}
mTrace("vgId:%d, alloc tid:%d", pVgroup->vgId, sid);
*pSid = sid;
*ppVgroup = pVgroup;
@ -507,6 +508,7 @@ int32_t mnodeGetAvailableVgroup(SMnodeMsg *pMsg, SVgObj **ppVgroup, int32_t *pSi
pDb->vgListIndex = 0;
pthread_mutex_unlock(&pDb->mutex);
mTrace("vgId:%d, alloc tid:%d", pVgroup->vgId, sid);
return TSDB_CODE_SUCCESS;
}
@ -832,26 +834,37 @@ static int32_t mnodeRetrieveVgroups(SShowObj *pShow, char *data, int32_t rows, v
return numOfRows;
}
void mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable) {
int32_t mnodeAddTableIntoVgroup(SVgObj *pVgroup, SCTableObj *pTable, bool needCheck) {
int32_t idPoolSize = taosIdPoolMaxSize(pVgroup->idPool);
if (pTable->tid > idPoolSize) {
mnodeAllocVgroupIdPool(pVgroup);
}
if (pTable->tid >= 1) {
taosIdPoolMarkStatus(pVgroup->idPool, pTable->tid);
if (taosIdPoolMarkStatus(pVgroup->idPool, pTable->tid) || !needCheck) {
pVgroup->numOfTables++;
mTrace("table:%s, vgId:%d tid:%d, mark tid used, uid:%" PRIu64, pTable->info.tableId, pTable->vgId, pTable->tid,
pTable->uid);
// The create vgroup message may be received later than the create table message
// and the writing order in sdb is therefore uncertain
// which will cause the reference count of the vgroup to be incorrect when restarting
// mnodeIncVgroupRef(pVgroup);
} else {
mError("table:%s, vgId:%d tid:%d, failed to mark tid, uid:%" PRIu64, pTable->info.tableId, pTable->vgId,
pTable->tid, pTable->uid);
return -1;
}
}
return 0;
}
void mnodeRemoveTableFromVgroup(SVgObj *pVgroup, SCTableObj *pTable) {
if (pTable->tid >= 1) {
taosFreeId(pVgroup->idPool, pTable->tid);
pVgroup->numOfTables--;
mTrace("table:%s, vgId:%d tid:%d, put tid back uid:%" PRIu64, pTable->info.tableId, pTable->vgId, pTable->tid,
pTable->uid);
// The create vgroup message may be received later than the create table message
// and the writing order in sdb is therefore uncertain
// which will cause the reference count of the vgroup to be incorrect when restarting

4
src/plugins/http/src/httpParser.c
View File

@ -110,7 +110,7 @@ static void httpCleanupString(HttpString *str) {
static int32_t httpAppendString(HttpString *str, const char *s, int32_t len) {
if (str->size == 0) {
str->pos = 0;
str->size = 64;
str->size = len + 1;
str->str = malloc(str->size);
} else if (str->pos + len + 1 >= str->size) {
str->size += len;
@ -715,10 +715,12 @@ static int32_t httpParserOnVersion(HttpParser *parser, HTTP_PARSER_STATE state,
if (parser->method) {
ok = httpOnRequestLine(parser, parser->method, parser->target, parser->version);
/*
if (parser->target) {
free(parser->target);
parser->target = NULL;
}
*/
}
httpClearString(&parser->str);

3
src/query/inc/qExecutor.h
View File

@ -308,6 +308,7 @@ enum {
typedef struct SQInfo {
void* signature;
uint64_t qId;
int32_t code; // error code to returned to client
int64_t owner; // if it is in execution
@ -429,7 +430,7 @@ int32_t createIndirectQueryFuncExprFromMsg(SQueryTableMsg *pQueryMsg, int32_t nu
SSqlGroupbyExpr *createGroupbyExprFromMsg(SQueryTableMsg *pQueryMsg, SColIndex *pColIndex, int32_t *code);
SQInfo *createQInfoImpl(SQueryTableMsg *pQueryMsg, SSqlGroupbyExpr *pGroupbyExpr, SExprInfo *pExprs,
SExprInfo *pSecExprs, STableGroupInfo *pTableGroupInfo, SColumnInfo* pTagCols, bool stableQuery, char* sql);
SExprInfo *pSecExprs, STableGroupInfo *pTableGroupInfo, SColumnInfo* pTagCols, bool stableQuery, char* sql, uint64_t *qId);
int32_t initQInfo(SQueryTableMsg *pQueryMsg, void *tsdb, int32_t vgId, SQInfo *pQInfo, SQueryParam* param, bool isSTable);
void freeColumnFilterInfo(SColumnFilterInfo* pFilter, int32_t numOfFilters);

38
src/query/inc/qSqlparser.h
View File

@ -39,6 +39,11 @@ enum SQL_NODE_TYPE {
SQL_NODE_EXPR = 4,
};
enum SQL_NODE_FROM_TYPE {
SQL_NODE_FROM_SUBQUERY = 1,
SQL_NODE_FROM_NAMELIST = 2,
};
extern char tTokenTypeSwitcher[13];
#define toTSDBType(x) \
@ -78,9 +83,11 @@ typedef struct SSessionWindowVal {
SStrToken gap;
} SSessionWindowVal;
struct SFromInfo;
typedef struct SQuerySqlNode {
struct SArray *pSelectList; // select clause
SArray *from; // from clause SArray<SQuerySqlNode>
struct SFromInfo *from; // from clause SArray<SQuerySqlNode>
struct tSqlExpr *pWhere; // where clause [optional]
SArray *pGroupby; // groupby clause, only for tags[optional], SArray<tVariantListItem>
SArray *pSortOrder; // orderby [optional], SArray<tVariantListItem>
@ -93,6 +100,24 @@ typedef struct SQuerySqlNode {
SStrToken sqlstr; // sql string in select clause
} SQuerySqlNode;
typedef struct STableNamePair {
SStrToken name;
SStrToken aliasName;
} STableNamePair;
typedef struct SSubclauseInfo { // "UNION" multiple select sub-clause
SQuerySqlNode **pClause;
int32_t numOfClause;
} SSubclauseInfo;
typedef struct SFromInfo {
int32_t type; // nested query|table name list
union {
SSubclauseInfo *pNode;
SArray *tableList; // SArray<STableNamePair>
};
} SFromInfo;
typedef struct SCreatedTableInfo {
SStrToken name; // table name token
SStrToken stableName; // super table name token , for using clause
@ -188,11 +213,6 @@ typedef struct SMiscInfo {
};
} SMiscInfo;
typedef struct SSubclauseInfo { // "UNION" multiple select sub-clause
SQuerySqlNode **pClause;
int32_t numOfClause;
} SSubclauseInfo;
typedef struct SSqlInfo {
int32_t type;
bool valid;
@ -233,6 +253,10 @@ SArray *tVariantListAppend(SArray *pList, tVariant *pVar, uint8_t sortOrder);
SArray *tVariantListInsert(SArray *pList, tVariant *pVar, uint8_t sortOrder, int32_t index);
SArray *tVariantListAppendToken(SArray *pList, SStrToken *pAliasToken, uint8_t sortOrder);
SFromInfo *setTableNameList(SFromInfo* pFromInfo, SStrToken *pName, SStrToken* pAlias);
SFromInfo *setSubquery(SFromInfo* pFromInfo, SQuerySqlNode *pSqlNode);
void *destroyFromInfo(SFromInfo* pFromInfo);
// sql expr leaf node
tSqlExpr *tSqlExprCreateIdValue(SStrToken *pToken, int32_t optrType);
tSqlExpr *tSqlExprCreateFunction(SArray *pParam, SStrToken *pFuncToken, SStrToken *endToken, int32_t optType);
@ -246,7 +270,7 @@ void tSqlExprDestroy(tSqlExpr *pExpr);
SArray *tSqlExprListAppend(SArray *pList, tSqlExpr *pNode, SStrToken *pDistinct, SStrToken *pToken);
void tSqlExprListDestroy(SArray *pList);
SQuerySqlNode *tSetQuerySqlNode(SStrToken *pSelectToken, SArray *pSelectList, SArray *pFrom, tSqlExpr *pWhere,
SQuerySqlNode *tSetQuerySqlNode(SStrToken *pSelectToken, SArray *pSelectList, SFromInfo *pFrom, tSqlExpr *pWhere,
SArray *pGroupby, SArray *pSortOrder, SIntervalVal *pInterval, SSessionWindowVal *ps,
SStrToken *pSliding, SArray *pFill, SLimitVal *pLimit, SLimitVal *pgLimit);

34
src/query/inc/sql.y
View File

@ -456,20 +456,20 @@ select(A) ::= SELECT(T) selcollist(W) from(X) where_opt(Y) interval_opt(K) sessi
A = tSetQuerySqlNode(&T, W, X, Y, P, Z, &K, &H, &S, F, &L, &G);
}
select(A) ::= LP select(B) RP. {A = B;}
%type union {SSubclauseInfo*}
%destructor union {destroyAllSelectClause($$);}
union(Y) ::= select(X). { Y = setSubclause(NULL, X); }
union(Y) ::= LP union(X) RP. { Y = X; }
union(Y) ::= union(Z) UNION ALL select(X). { Y = appendSelectClause(Z, X); }
union(Y) ::= union(Z) UNION ALL LP select(X) RP. { Y = appendSelectClause(Z, X); }
cmd ::= union(X). { setSqlInfo(pInfo, X, NULL, TSDB_SQL_SELECT); }
// Support for the SQL exprssion without from & where subclauses, e.g.,
// select current_database(),
// select server_version(), select client_version(),
// select server_state();
// select current_database()
// select server_version()
// select client_version()
// select server_state()
select(A) ::= SELECT(T) selcollist(W). {
A = tSetQuerySqlNode(&T, W, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL);
}
@ -495,7 +495,6 @@ selcollist(A) ::= sclp(P) STAR. {
// An option "AS <id>" phrase that can follow one of the expressions that
// define the result set, or one of the tables in the FROM clause.
//
%type as {SStrToken}
as(X) ::= AS ids(Y). { X = Y; }
as(X) ::= ids(Y). { X = Y; }
@ -506,39 +505,36 @@ distinct(X) ::= DISTINCT(Y). { X = Y; }
distinct(X) ::= . { X.n = 0;}
// A complete FROM clause.
%type from {SArray*}
// current not support query from no-table
%type from {SFromInfo*}
from(A) ::= FROM tablelist(X). {A = X;}
from(A) ::= FROM LP union(Y) RP. {A = Y;}
%type tablelist {SArray*}
tablelist(A) ::= ids(X) cpxName(Y). {
toTSDBType(X.type);
X.n += Y.n;
A = tVariantListAppendToken(NULL, &X, -1);
A = tVariantListAppendToken(A, &X, -1); // table alias name
A = setTableNameList(NULL, &X, NULL);
}
tablelist(A) ::= ids(X) cpxName(Y) ids(Z). {
toTSDBType(X.type);
toTSDBType(Z.type);
X.n += Y.n;
A = tVariantListAppendToken(NULL, &X, -1);
A = tVariantListAppendToken(A, &Z, -1);
A = setTableNameList(NULL, &X, &Z);
}
tablelist(A) ::= tablelist(Y) COMMA ids(X) cpxName(Z). {
toTSDBType(X.type);
X.n += Z.n;
A = tVariantListAppendToken(Y, &X, -1);
A = tVariantListAppendToken(A, &X, -1);
A = setTableNameList(Y, &X, NULL);
}
tablelist(A) ::= tablelist(Y) COMMA ids(X) cpxName(Z) ids(F). {
toTSDBType(X.type);
toTSDBType(F.type);
X.n += Z.n;
A = tVariantListAppendToken(Y, &X, -1);
A = tVariantListAppendToken(A, &F, -1);
A = setTableNameList(Y, &X, &F);
}
// The value of interval should be the form of "number+[a,s,m,h,d,n,y]" or "now"
@ -842,6 +838,4 @@ cmd ::= KILL QUERY INTEGER(X) COLON(Z) INTEGER(Y). {X.n += (Z.n + Y.n); s
%fallback ID ABORT AFTER ASC ATTACH BEFORE BEGIN CASCADE CLUSTER CONFLICT COPY DATABASE DEFERRED
DELIMITERS DESC DETACH EACH END EXPLAIN FAIL FOR GLOB IGNORE IMMEDIATE INITIALLY INSTEAD
LIKE MATCH KEY OF OFFSET RAISE REPLACE RESTRICT ROW STATEMENT TRIGGER VIEW ALL
NOW IPTOKEN SEMI NONE PREV LINEAR IMPORT
METRIC TBNAME JOIN METRICS STABLE NULL INSERT INTO VALUES.
NOW IPTOKEN SEMI NONE PREV LINEAR IMPORT TBNAME JOIN STABLE NULL INSERT INTO VALUES.

41
src/query/src/qExecutor.c
View File

@ -98,6 +98,9 @@ static UNUSED_FUNC void* u_realloc(void* p, size_t __size) {
#define GET_NUM_OF_TABLEGROUP(q) taosArrayGetSize((q)->tableqinfoGroupInfo.pGroupList)
#define QUERY_IS_INTERVAL_QUERY(_q) ((_q)->interval.interval > 0)
uint64_t queryHandleId = 0;
int32_t getMaximumIdleDurationSec() {
return tsShellActivityTimer * 2;
}
@ -1837,7 +1840,7 @@ static void doFreeQueryHandle(SQueryRuntimeEnv* pRuntimeEnv) {
pRuntimeEnv->pQueryHandle = NULL;
SMemRef* pMemRef = &pQuery->memRef;
assert(pMemRef->ref == 0 && pMemRef->imem == NULL && pMemRef->mem == NULL);
assert(pMemRef->ref == 0 && pMemRef->snapshot.imem == NULL && pMemRef->snapshot.mem == NULL);
}
static void teardownQueryRuntimeEnv(SQueryRuntimeEnv *pRuntimeEnv) {
@ -2366,13 +2369,13 @@ static int32_t doTSJoinFilter(SQueryRuntimeEnv *pRuntimeEnv, TSKEY key, bool asc
}
void filterRowsInDataBlock(SQueryRuntimeEnv* pRuntimeEnv, SSingleColumnFilterInfo* pFilterInfo, int32_t numOfFilterCols,
SSDataBlock* pBlock, STSBuf* pTsBuf, bool ascQuery) {
SSDataBlock* pBlock, bool ascQuery) {
int32_t numOfRows = pBlock->info.rows;
int8_t *p = calloc(numOfRows, sizeof(int8_t));
bool all = true;
if (pTsBuf != NULL) {
if (pRuntimeEnv->pTsBuf != NULL) {
SColumnInfoData* pColInfoData = taosArrayGet(pBlock->pDataBlock, 0);
TSKEY* k = (TSKEY*) pColInfoData->pData;
@ -2394,6 +2397,7 @@ void filterRowsInDataBlock(SQueryRuntimeEnv* pRuntimeEnv, SSingleColumnFilterInf
}
}
// save the cursor status
pRuntimeEnv->pQuery->current->cur = tsBufGetCursor(pRuntimeEnv->pTsBuf);
} else {
for (int32_t i = 0; i < numOfRows; ++i) {
@ -2655,8 +2659,7 @@ int32_t loadDataBlockOnDemand(SQueryRuntimeEnv* pRuntimeEnv, STableScanInfo* pTa
doSetFilterColumnInfo(pQuery, pBlock);
if (pQuery->numOfFilterCols > 0 || pRuntimeEnv->pTsBuf != NULL) {
filterRowsInDataBlock(pRuntimeEnv, pQuery->pFilterInfo, pQuery->numOfFilterCols, pBlock, pRuntimeEnv->pTsBuf,
ascQuery);
filterRowsInDataBlock(pRuntimeEnv, pQuery->pFilterInfo, pQuery->numOfFilterCols, pBlock, ascQuery);
}
}
@ -2747,7 +2750,10 @@ static void doSetTagValueInParam(void* pTable, int32_t tagColId, tVariant *tag,
}
if (type == TSDB_DATA_TYPE_BINARY || type == TSDB_DATA_TYPE_NCHAR) {
tVariantCreateFromBinary(tag, varDataVal(val), varDataLen(val), type);
int32_t maxLen = bytes - VARSTR_HEADER_SIZE;
int32_t len = (varDataLen(val) > maxLen)? maxLen:varDataLen(val);
tVariantCreateFromBinary(tag, varDataVal(val), len, type);
//tVariantCreateFromBinary(tag, varDataVal(val), varDataLen(val), type);
} else {
tVariantCreateFromBinary(tag, val, bytes, type);
}
@ -4170,8 +4176,10 @@ static SSDataBlock* doTableScan(void* param) {
assert(ret);
}
if (pResultRowInfo->size > 0) {
pResultRowInfo->curIndex = 0;
pResultRowInfo->prevSKey = pResultRowInfo->pResult[0]->win.skey;
}
qDebug("QInfo:%p start to repeat scan data blocks due to query func required, qrange:%" PRId64 "-%" PRId64,
pRuntimeEnv->qinfo, cond.twindow.skey, cond.twindow.ekey);
@ -6106,9 +6114,13 @@ void setResultBufSize(SQuery* pQuery, SRspResultInfo* pResultInfo) {
pResultInfo->total = 0;
}
FORCE_INLINE bool checkQIdEqual(void *qHandle, uint64_t qId) {
return ((SQInfo *)qHandle)->qId == qId;
}
SQInfo* createQInfoImpl(SQueryTableMsg* pQueryMsg, SSqlGroupbyExpr* pGroupbyExpr, SExprInfo* pExprs,
SExprInfo* pSecExprs, STableGroupInfo* pTableGroupInfo, SColumnInfo* pTagCols, bool stableQuery,
char* sql) {
char* sql, uint64_t *qId) {
int16_t numOfCols = pQueryMsg->numOfCols;
int16_t numOfOutput = pQueryMsg->numOfOutput;
@ -6249,7 +6261,9 @@ SQInfo* createQInfoImpl(SQueryTableMsg* pQueryMsg, SSqlGroupbyExpr* pGroupbyExpr
// todo refactor
pQInfo->query.queryBlockDist = (numOfOutput == 1 && pExprs[0].base.colInfo.colId == TSDB_BLOCK_DIST_COLUMN_INDEX);
qDebug("qmsg:%p QInfo:%p created", pQueryMsg, pQInfo);
pQInfo->qId = atomic_add_fetch_64(&queryHandleId, 1);
*qId = pQInfo->qId;
qDebug("qmsg:%p QInfo:%" PRIu64 "-%p created", pQueryMsg, pQInfo->qId, pQInfo);
return pQInfo;
_cleanup_qinfo:
@ -6539,8 +6553,15 @@ static void doSetTagValueToResultBuf(char* output, const char* val, int16_t type
return;
}
if (type == TSDB_DATA_TYPE_BINARY || type == TSDB_DATA_TYPE_NCHAR) {
memcpy(output, val, varDataTLen(val));
if (IS_VAR_DATA_TYPE(type)) {
// Binary data overflows for sort of unknown reasons. Let trim the overflow data
if (varDataTLen(val) > bytes) {
int32_t len = bytes - VARSTR_HEADER_SIZE; // remain available space
memcpy(varDataVal(output), varDataVal(val), len);
varDataSetLen(output, len);
} else {
varDataCopy(output, val);
}
} else {
memcpy(output, val, bytes);
}

56
src/query/src/qSqlParser.c
View File

@ -443,6 +443,52 @@ SArray *tVariantListInsert(SArray *pList, tVariant *pVar, uint8_t sortOrder, int
return pList;
}
SFromInfo *setTableNameList(SFromInfo* pFromInfo, SStrToken *pName, SStrToken* pAlias) {
if (pFromInfo == NULL) {
pFromInfo = calloc(1, sizeof(SFromInfo));
pFromInfo->tableList = taosArrayInit(4, sizeof(STableNamePair));
}
pFromInfo->type = SQL_NODE_FROM_NAMELIST;
STableNamePair p = {.name = *pName};
if (pAlias != NULL) {
p.aliasName = *pAlias;
} else {
TPARSER_SET_NONE_TOKEN(p.aliasName);
}
taosArrayPush(pFromInfo->tableList, &p);
return pFromInfo;
}
SFromInfo *setSubquery(SFromInfo* pFromInfo, SQuerySqlNode* pSqlNode) {
if (pFromInfo == NULL) {
pFromInfo = calloc(1, sizeof(SFromInfo));
}
pFromInfo->type = SQL_NODE_FROM_SUBQUERY;
pFromInfo->pNode->pClause[pFromInfo->pNode->numOfClause - 1] = pSqlNode;
return pFromInfo;
}
void* destroyFromInfo(SFromInfo* pFromInfo) {
if (pFromInfo == NULL) {
return NULL;
}
if (pFromInfo->type == SQL_NODE_FROM_NAMELIST) {
taosArrayDestroy(pFromInfo->tableList);
} else {
destroyAllSelectClause(pFromInfo->pNode);
}
tfree(pFromInfo);
return NULL;
}
void tSetDbName(SStrToken *pCpxName, SStrToken *pDb) {
pCpxName->type = pDb->type;
pCpxName->z = pDb->z;
@ -582,9 +628,10 @@ void tSetColumnType(TAOS_FIELD *pField, SStrToken *type) {
/*
* extract the select info out of sql string
*/
SQuerySqlNode *tSetQuerySqlNode(SStrToken *pSelectToken, SArray *pSelectList, SArray *pFrom, tSqlExpr *pWhere,
SArray *pGroupby, SArray *pSortOrder, SIntervalVal *pInterval, SSessionWindowVal *pSession,
SStrToken *pSliding, SArray *pFill, SLimitVal *pLimit, SLimitVal *psLimit) {
SQuerySqlNode *tSetQuerySqlNode(SStrToken *pSelectToken, SArray *pSelectList, SFromInfo *pFrom, tSqlExpr *pWhere,
SArray *pGroupby, SArray *pSortOrder, SIntervalVal *pInterval,
SSessionWindowVal *pSession, SStrToken *pSliding, SArray *pFill, SLimitVal *pLimit,
SLimitVal *psLimit) {
assert(pSelectList != NULL);
SQuerySqlNode *pSqlNode = calloc(1, sizeof(SQuerySqlNode));
@ -668,8 +715,7 @@ void destroyQuerySqlNode(SQuerySqlNode *pQuerySql) {
taosArrayDestroyEx(pQuerySql->pGroupby, freeVariant);
pQuerySql->pGroupby = NULL;
taosArrayDestroyEx(pQuerySql->from, freeVariant);
pQuerySql->from = NULL;
pQuerySql->from = destroyFromInfo(pQuerySql->from);
taosArrayDestroyEx(pQuerySql->fillType, freeVariant);
pQuerySql->fillType = NULL;

2
src/query/src/qTokenizer.c
View File

@ -206,10 +206,8 @@ static SKeyword keywordTable[] = {
{"PREV", TK_PREV},
{"LINEAR", TK_LINEAR},
{"IMPORT", TK_IMPORT},
{"METRIC", TK_METRIC},
{"TBNAME", TK_TBNAME},
{"JOIN", TK_JOIN},
{"METRICS", TK_METRICS},
{"STABLE", TK_STABLE},
{"FILE", TK_FILE},
{"VNODES", TK_VNODES},

13
src/query/src/queryMain.c
View File

@ -68,7 +68,7 @@ void freeParam(SQueryParam *param) {
tfree(param->prevResult);
}
int32_t qCreateQueryInfo(void* tsdb, int32_t vgId, SQueryTableMsg* pQueryMsg, qinfo_t* pQInfo) {
int32_t qCreateQueryInfo(void* tsdb, int32_t vgId, SQueryTableMsg* pQueryMsg, qinfo_t* pQInfo, uint64_t *qId) {
assert(pQueryMsg != NULL && tsdb != NULL);
int32_t code = TSDB_CODE_SUCCESS;
@ -158,7 +158,7 @@ int32_t qCreateQueryInfo(void* tsdb, int32_t vgId, SQueryTableMsg* pQueryMsg, qi
goto _over;
}
(*pQInfo) = createQInfoImpl(pQueryMsg, param.pGroupbyExpr, param.pExprs, param.pSecExprs, &tableGroupInfo, param.pTagColumnInfo, isSTableQuery, param.sql);
(*pQInfo) = createQInfoImpl(pQueryMsg, param.pGroupbyExpr, param.pExprs, param.pSecExprs, &tableGroupInfo, param.pTagColumnInfo, isSTableQuery, param.sql, qId);
param.sql = NULL;
param.pExprs = NULL;
@ -430,7 +430,7 @@ void qQueryMgmtNotifyClosed(void* pQMgmt) {
}
SQueryMgmt* pQueryMgmt = pQMgmt;
qDebug("vgId:%d, set querymgmt closed, wait for all queries cancelled", pQueryMgmt->vgId);
qInfo("vgId:%d, set querymgmt closed, wait for all queries cancelled", pQueryMgmt->vgId);
pthread_mutex_lock(&pQueryMgmt->lock);
pQueryMgmt->closed = true;
@ -445,7 +445,7 @@ void qQueryMgmtReOpen(void *pQMgmt) {
}
SQueryMgmt *pQueryMgmt = pQMgmt;
qDebug("vgId:%d, set querymgmt reopen", pQueryMgmt->vgId);
qInfo("vgId:%d, set querymgmt reopen", pQueryMgmt->vgId);
pthread_mutex_lock(&pQueryMgmt->lock);
pQueryMgmt->closed = false;
@ -472,7 +472,7 @@ void qCleanupQueryMgmt(void* pQMgmt) {
qDebug("vgId:%d, queryMgmt cleanup completed", vgId);
}
void** qRegisterQInfo(void* pMgmt, uint64_t qInfo) {
void** qRegisterQInfo(void* pMgmt, uint64_t qId, uint64_t qInfo) {
if (pMgmt == NULL) {
terrno = TSDB_CODE_VND_INVALID_VGROUP_ID;
return NULL;
@ -492,8 +492,7 @@ void** qRegisterQInfo(void* pMgmt, uint64_t qInfo) {
terrno = TSDB_CODE_VND_INVALID_VGROUP_ID;
return NULL;
} else {
TSDB_CACHE_PTR_TYPE handleVal = (TSDB_CACHE_PTR_TYPE) qInfo;
void** handle = taosCachePut(pQueryMgmt->qinfoPool, &handleVal, sizeof(TSDB_CACHE_PTR_TYPE), &qInfo, sizeof(TSDB_CACHE_PTR_TYPE),
void** handle = taosCachePut(pQueryMgmt->qinfoPool, &qId, sizeof(qId), &qInfo, sizeof(TSDB_CACHE_PTR_TYPE),
(getMaximumIdleDurationSec()*1000));
pthread_mutex_unlock(&pQueryMgmt->lock);

1733
src/query/src/sql.c
View File

File diff suppressed because it is too large Load Diff

1
src/sync/inc/syncInt.h
View File

@ -117,6 +117,7 @@ typedef struct SSyncNode {
FStartSyncFile startSyncFileFp;
FStopSyncFile stopSyncFileFp;
FGetVersion getVersionFp;
FResetVersion resetVersionFp;
FSendFile sendFileFp;
FRecvFile recvFileFp;
pthread_mutex_t mutex;

15
src/sync/src/syncMain.c
View File

@ -56,7 +56,7 @@ static void syncMonitorNodeRole(void *param, void *tmrId);
static void syncProcessFwdAck(SSyncNode *pNode, SFwdInfo *pFwdInfo, int32_t code);
static int32_t syncSaveFwdInfo(SSyncNode *pNode, uint64_t version, void *mhandle);
static void syncRestartPeer(SSyncPeer *pPeer);
static int32_t syncForwardToPeerImpl(SSyncNode *pNode, void *data, void *mhandle, int32_t qtyp);
static int32_t syncForwardToPeerImpl(SSyncNode *pNode, void *data, void *mhandle, int32_t qtype, bool force);
static SSyncPeer *syncAddPeer(SSyncNode *pNode, const SNodeInfo *pInfo);
static void syncStartCheckPeerConn(SSyncPeer *pPeer);
@ -182,6 +182,7 @@ int64_t syncStart(const SSyncInfo *pInfo) {
pNode->startSyncFileFp = pInfo->startSyncFileFp;
pNode->stopSyncFileFp = pInfo->stopSyncFileFp;
pNode->getVersionFp = pInfo->getVersionFp;
pNode->resetVersionFp = pInfo->resetVersionFp;
pNode->sendFileFp = pInfo->sendFileFp;
pNode->recvFileFp = pInfo->recvFileFp;
@ -377,24 +378,24 @@ int32_t syncReconfig(int64_t rid, const SSyncCfg *pNewCfg) {
return 0;
}
int32_t syncForwardToPeer(int64_t rid, void *data, void *mhandle, int32_t qtype) {
int32_t syncForwardToPeer(int64_t rid, void *data, void *mhandle, int32_t qtype, bool force) {
if (rid <= 0) return 0;
SSyncNode *pNode = syncAcquireNode(rid);
if (pNode == NULL) return 0;
int32_t code = syncForwardToPeerImpl(pNode, data, mhandle, qtype);
int32_t code = syncForwardToPeerImpl(pNode, data, mhandle, qtype, force);
syncReleaseNode(pNode);
return code;
}
void syncConfirmForward(int64_t rid, uint64_t version, int32_t code) {
void syncConfirmForward(int64_t rid, uint64_t version, int32_t code, bool force) {
SSyncNode *pNode = syncAcquireNode(rid);
if (pNode == NULL) return;
SSyncPeer *pPeer = pNode->pMaster;
if (pPeer && pNode->quorum > 1) {
if (pPeer && (pNode->quorum > 1 || force)) {
SFwdRsp rsp;
syncBuildSyncFwdRsp(&rsp, pNode->vgId, version, code);
@ -1413,7 +1414,7 @@ static void syncMonitorFwdInfos(void *param, void *tmrId) {
syncReleaseNode(pNode);
}
static int32_t syncForwardToPeerImpl(SSyncNode *pNode, void *data, void *mhandle, int32_t qtype) {
static int32_t syncForwardToPeerImpl(SSyncNode *pNode, void *data, void *mhandle, int32_t qtype, bool force) {
SSyncPeer *pPeer;
SSyncHead *pSyncHead;
SWalHead * pWalHead = data;
@ -1457,7 +1458,7 @@ static int32_t syncForwardToPeerImpl(SSyncNode *pNode, void *data, void *mhandle
if (pPeer == NULL || pPeer->peerFd < 0) continue;
if (pPeer->role != TAOS_SYNC_ROLE_SLAVE && pPeer->sstatus != TAOS_SYNC_STATUS_CACHE) continue;
if (pNode->quorum > 1 && code == 0) {
if ((pNode->quorum > 1 || force) && code == 0) {
code = syncSaveFwdInfo(pNode, pWalHead->version, mhandle);
if (code >= 0) code = 1;
}

1
src/sync/src/syncRestore.c
View File

@ -238,6 +238,7 @@ static int32_t syncRestoreDataStepByStep(SSyncPeer *pPeer) {
(*pNode->stopSyncFileFp)(pNode->vgId, fversion);
nodeVersion = fversion;
if (pNode->resetVersionFp) (*pNode->resetVersionFp)(pNode->vgId, fversion);
sInfo("%s, start to restore wal, fver:%" PRIu64, pPeer->id, nodeVersion);
uint64_t wver = 0;

46
src/sync/src/syncRetrieve.c
View File

@ -29,7 +29,7 @@ static int32_t syncGetWalVersion(SSyncNode *pNode, SSyncPeer *pPeer) {
uint64_t fver, wver;
int32_t code = (*pNode->getVersionFp)(pNode->vgId, &fver, &wver);
if (code != 0) {
sDebug("%s, vnode is commiting while retrieve, last wver:%" PRIu64, pPeer->id, pPeer->lastWalVer);
sInfo("%s, vnode is commiting while retrieve, last wver:%" PRIu64, pPeer->id, pPeer->lastWalVer);
return -1;
}
@ -41,12 +41,12 @@ static bool syncIsWalModified(SSyncNode *pNode, SSyncPeer *pPeer) {
uint64_t fver, wver;
int32_t code = (*pNode->getVersionFp)(pNode->vgId, &fver, &wver);
if (code != 0) {
sDebug("%s, vnode is commiting while retrieve, last wver:%" PRIu64, pPeer->id, pPeer->lastWalVer);
sInfo("%s, vnode is commiting while retrieve, last wver:%" PRIu64, pPeer->id, pPeer->lastWalVer);
return true;
}
if (wver != pPeer->lastWalVer) {
sDebug("%s, wal is modified while retrieve, wver:%" PRIu64 ", last:%" PRIu64, pPeer->id, wver, pPeer->lastWalVer);
sInfo("%s, wal is modified while retrieve, wver:%" PRIu64 ", last:%" PRIu64, pPeer->id, wver, pPeer->lastWalVer);
return true;
}
@ -57,7 +57,7 @@ static int32_t syncGetFileVersion(SSyncNode *pNode, SSyncPeer *pPeer) {
uint64_t fver, wver;
int32_t code = (*pNode->getVersionFp)(pNode->vgId, &fver, &wver);
if (code != 0) {
sDebug("%s, vnode is commiting while get fver for retrieve, last fver:%" PRIu64, pPeer->id, pPeer->lastFileVer);
sInfo("%s, vnode is commiting while get fver for retrieve, last fver:%" PRIu64, pPeer->id, pPeer->lastFileVer);
return -1;
}
@ -69,13 +69,13 @@ static bool syncAreFilesModified(SSyncNode *pNode, SSyncPeer *pPeer) {
uint64_t fver, wver;
int32_t code = (*pNode->getVersionFp)(pNode->vgId, &fver, &wver);
if (code != 0) {
sDebug("%s, vnode is commiting while retrieve, last fver:%" PRIu64, pPeer->id, pPeer->lastFileVer);
sInfo("%s, vnode is commiting while retrieve, last fver:%" PRIu64, pPeer->id, pPeer->lastFileVer);
pPeer->fileChanged = 1;
return true;
}
if (fver != pPeer->lastFileVer) {
sDebug("%s, files are modified while retrieve, fver:%" PRIu64 ", last:%" PRIu64, pPeer->id, fver, pPeer->lastFileVer);
sInfo("%s, files are modified while retrieve, fver:%" PRIu64 ", last:%" PRIu64, pPeer->id, fver, pPeer->lastFileVer);
pPeer->fileChanged = 1;
return true;
}
@ -143,13 +143,13 @@ static int32_t syncReadOneWalRecord(int32_t sfd, SWalHead *pHead) {
}
if (ret == 0) {
sDebug("sfd:%d, read to the end of file, ret:%d", sfd, ret);
sInfo("sfd:%d, read to the end of file, ret:%d", sfd, ret);
return 0;
}
if (ret != sizeof(SWalHead)) {
// file is not at end yet, it shall be reloaded
sDebug("sfd:%d, a partial wal head is read out, ret:%d", sfd, ret);
sInfo("sfd:%d, a partial wal head is read out, ret:%d", sfd, ret);
return 0;
}
@ -163,7 +163,7 @@ static int32_t syncReadOneWalRecord(int32_t sfd, SWalHead *pHead) {
if (ret != pHead->len) {
// file is not at end yet, it shall be reloaded
sDebug("sfd:%d, a partial wal conetnt is read out, ret:%d", sfd, ret);
sInfo("sfd:%d, a partial wal conetnt is read out, ret:%d", sfd, ret);
return 0;
}
@ -184,7 +184,7 @@ static int64_t syncRetrieveLastWal(SSyncPeer *pPeer, char *name, uint64_t fversi
return -1;
}
sDebug("%s, retrieve last wal:%s, offset:%" PRId64 " fver:%" PRIu64, pPeer->id, name, offset, fversion);
sInfo("%s, retrieve last wal:%s, offset:%" PRId64 " fver:%" PRIu64, pPeer->id, name, offset, fversion);
SWalHead *pHead = malloc(SYNC_MAX_SIZE);
int64_t bytes = 0;
@ -198,7 +198,7 @@ static int64_t syncRetrieveLastWal(SSyncPeer *pPeer, char *name, uint64_t fversi
if (code == 0) {
code = bytes;
sDebug("%s, read to the end of wal, bytes:%" PRId64, pPeer->id, bytes);
sInfo("%s, read to the end of wal, bytes:%" PRId64, pPeer->id, bytes);
break;
}
@ -217,7 +217,7 @@ static int64_t syncRetrieveLastWal(SSyncPeer *pPeer, char *name, uint64_t fversi
if (pHead->version >= fversion && fversion > 0) {
code = 0;
sDebug("%s, retrieve wal finished, hver:%" PRIu64 " fver:%" PRIu64, pPeer->id, pHead->version, fversion);
sInfo("%s, retrieve wal finished, hver:%" PRIu64 " fver:%" PRIu64, pPeer->id, pHead->version, fversion);
break;
}
}
@ -237,7 +237,7 @@ static int64_t syncProcessLastWal(SSyncPeer *pPeer, char *wname, int64_t index)
// get full path to wal file
snprintf(fname, sizeof(fname), "%s/%s", pNode->path, wname);
sDebug("%s, start to retrieve last wal:%s", pPeer->id, fname);
sInfo("%s, start to retrieve last wal:%s", pPeer->id, fname);
while (1) {
if (syncAreFilesModified(pNode, pPeer)) return -1;
@ -245,7 +245,7 @@ static int64_t syncProcessLastWal(SSyncPeer *pPeer, char *wname, int64_t index)
int64_t bytes = syncRetrieveLastWal(pPeer, fname, fversion, offset);
if (bytes < 0) {
sDebug("%s, failed to retrieve last wal, bytes:%" PRId64, pPeer->id, bytes);
sInfo("%s, failed to retrieve last wal, bytes:%" PRId64, pPeer->id, bytes);
return bytes;
}
@ -257,13 +257,13 @@ static int64_t syncProcessLastWal(SSyncPeer *pPeer, char *wname, int64_t index)
if (fversion == 0) {
pPeer->sstatus = TAOS_SYNC_STATUS_CACHE; // start to forward pkt
fversion = nodeVersion; // must read data to fversion
sDebug("%s, set sstatus:%s and fver:%" PRIu64, pPeer->id, syncStatus[pPeer->sstatus], fversion);
sInfo("%s, set sstatus:%s and fver:%" PRIu64, pPeer->id, syncStatus[pPeer->sstatus], fversion);
}
}
// if all data up to fversion is read out, it is over
if (pPeer->sversion >= fversion && fversion > 0) {
sDebug("%s, data up to fver:%" PRIu64 " has been read out, bytes:%" PRId64 " sver:%" PRIu64, pPeer->id, fversion, bytes,
sInfo("%s, data up to fver:%" PRIu64 " has been read out, bytes:%" PRId64 " sver:%" PRIu64, pPeer->id, fversion, bytes,
pPeer->sversion);
return 0;
}
@ -277,7 +277,7 @@ static int64_t syncProcessLastWal(SSyncPeer *pPeer, char *wname, int64_t index)
// if bytes > 0, file is updated, or fversion is not reached but file still open, read again
once = 1;
offset += bytes;
sDebug("%s, continue retrieve last wal, bytes:%" PRId64 " offset:%" PRId64 " sver:%" PRIu64 " fver:%" PRIu64, pPeer->id,
sInfo("%s, continue retrieve last wal, bytes:%" PRId64 " offset:%" PRId64 " sver:%" PRIu64 " fver:%" PRIu64, pPeer->id,
bytes, offset, pPeer->sversion, fversion);
}
@ -303,7 +303,7 @@ static int64_t syncRetrieveWal(SSyncPeer *pPeer) {
if (wname[0] == 0) { // no wal file
code = 0;
sDebug("%s, no wal file anymore", pPeer->id);
sInfo("%s, no wal file anymore", pPeer->id);
break;
}
@ -320,12 +320,12 @@ static int64_t syncRetrieveWal(SSyncPeer *pPeer) {
struct stat fstat;
if (stat(fname, &fstat) < 0) {
code = -1;
sDebug("%s, failed to stat wal:%s for retrieve since %s, code:0x%" PRIx64, pPeer->id, fname, strerror(errno), code);
sInfo("%s, failed to stat wal:%s for retrieve since %s, code:0x%" PRIx64, pPeer->id, fname, strerror(errno), code);
break;
}
size = fstat.st_size;
sDebug("%s, retrieve wal:%s size:%d", pPeer->id, fname, size);
sInfo("%s, retrieve wal:%s size:%d", pPeer->id, fname, size);
int32_t sfd = open(fname, O_RDONLY | O_BINARY);
if (sfd < 0) {
@ -334,7 +334,7 @@ static int64_t syncRetrieveWal(SSyncPeer *pPeer) {
break;
}
code = (int32_t)taosSendFile(pPeer->syncFd, sfd, NULL, size);
code = taosSendFile(pPeer->syncFd, sfd, NULL, size);
close(sfd);
if (code < 0) {
sError("%s, failed to send wal:%s for retrieve since %s, code:0x%" PRIx64, pPeer->id, fname, strerror(errno), code);
@ -374,7 +374,7 @@ static int32_t syncRetrieveFirstPkt(SSyncPeer *pPeer) {
sError("%s, failed to send sync-data msg since %s, tranId:%u", pPeer->id, strerror(errno), msg.tranId);
return -1;
}
sDebug("%s, send sync-data msg to peer, tranId:%u", pPeer->id, msg.tranId);
sInfo("%s, send sync-data msg to peer, tranId:%u", pPeer->id, msg.tranId);
SSyncRsp rsp;
if (taosReadMsg(pPeer->syncFd, &rsp, sizeof(SSyncRsp)) != sizeof(SSyncRsp)) {
@ -382,7 +382,7 @@ static int32_t syncRetrieveFirstPkt(SSyncPeer *pPeer) {
return -1;
}
sDebug("%s, recv sync-data rsp from peer, tranId:%u rsp-tranId:%u", pPeer->id, msg.tranId, rsp.tranId);
sInfo("%s, recv sync-data rsp from peer, tranId:%u rsp-tranId:%u", pPeer->id, msg.tranId, rsp.tranId);
return 0;
}

23
src/tsdb/inc/tsdbMemTable.h
View File

@ -31,29 +31,14 @@ typedef struct {
SSkipListIterator *pIter;
} SCommitIter;
typedef struct {
struct STableData {
uint64_t uid;
TSKEY keyFirst;
TSKEY keyLast;
int64_t numOfRows;
SSkipList* pData;
T_REF_DECLARE()
} STableData;
typedef struct {
T_REF_DECLARE()
SRWLatch latch;
TSKEY keyFirst;
TSKEY keyLast;
int64_t numOfRows;
int32_t maxTables;
STableData** tData;
SList* actList;
SList* extraBuffList;
SList* bufBlockList;
int64_t pointsAdd; // TODO
int64_t storageAdd; // TODO
} SMemTable;
};
enum { TSDB_UPDATE_META, TSDB_DROP_META };
@ -77,8 +62,8 @@ typedef struct {
int tsdbRefMemTable(STsdbRepo* pRepo, SMemTable* pMemTable);
int tsdbUnRefMemTable(STsdbRepo* pRepo, SMemTable* pMemTable);
int tsdbTakeMemSnapshot(STsdbRepo* pRepo, SMemTable** pMem, SMemTable** pIMem, SArray* pATable);
void tsdbUnTakeMemSnapShot(STsdbRepo* pRepo, SMemTable* pMem, SMemTable* pIMem);
int tsdbTakeMemSnapshot(STsdbRepo* pRepo, SMemSnapshot* pSnapshot, SArray* pATable);
void tsdbUnTakeMemSnapShot(STsdbRepo* pRepo, SMemSnapshot* pSnapshot);
void* tsdbAllocBytes(STsdbRepo* pRepo, int bytes);
int tsdbAsyncCommit(STsdbRepo* pRepo);
int tsdbLoadDataFromCache(STable* pTable, SSkipListIterator* pIter, TSKEY maxKey, int maxRowsToRead, SDataCols* pCols,

92
src/tsdb/src/tsdbMemTable.c
View File

@ -124,88 +124,80 @@ int tsdbUnRefMemTable(STsdbRepo *pRepo, SMemTable *pMemTable) {
return 0;
}
int tsdbTakeMemSnapshot(STsdbRepo *pRepo, SMemTable **pMem, SMemTable **pIMem, SArray *pATable) {
SMemTable *tmem;
int tsdbTakeMemSnapshot(STsdbRepo *pRepo, SMemSnapshot *pSnapshot, SArray *pATable) {
memset(pSnapshot, 0, sizeof(*pSnapshot));
// Get snap object
if (tsdbLockRepo(pRepo) < 0) return -1;
tmem = pRepo->mem;
*pIMem = pRepo->imem;
tsdbRefMemTable(pRepo, tmem);
tsdbRefMemTable(pRepo, *pIMem);
pSnapshot->omem = pRepo->mem;
pSnapshot->imem = pRepo->imem;
tsdbRefMemTable(pRepo, pRepo->mem);
tsdbRefMemTable(pRepo, pRepo->imem);
if (tsdbUnlockRepo(pRepo) < 0) return -1;
// Copy mem objects and ref needed STableData
if (tmem) {
taosRLockLatch(&(tmem->latch));
if (pSnapshot->omem) {
taosRLockLatch(&(pSnapshot->omem->latch));
*pMem = (SMemTable *)calloc(1, sizeof(**pMem));
if (*pMem == NULL) {
pSnapshot->mem = &(pSnapshot->mtable);
pSnapshot->mem->tData = (STableData **)calloc(pSnapshot->omem->maxTables, sizeof(STableData *));
if (pSnapshot->mem->tData == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
taosRUnLockLatch(&(tmem->latch));
tsdbUnRefMemTable(pRepo, tmem);
tsdbUnRefMemTable(pRepo, *pIMem);
*pMem = NULL;
*pIMem = NULL;
taosRUnLockLatch(&(pSnapshot->omem->latch));
tsdbUnRefMemTable(pRepo, pSnapshot->omem);
tsdbUnRefMemTable(pRepo, pSnapshot->imem);
pSnapshot->mem = NULL;
pSnapshot->imem = NULL;
pSnapshot->omem = NULL;
return -1;
}
(*pMem)->tData = (STableData **)calloc(tmem->maxTables, sizeof(STableData *));
if ((*pMem)->tData == NULL) {
terrno = TSDB_CODE_TDB_OUT_OF_MEMORY;
taosRUnLockLatch(&(tmem->latch));
free(*pMem);
tsdbUnRefMemTable(pRepo, tmem);
tsdbUnRefMemTable(pRepo, *pIMem);
*pMem = NULL;
*pIMem = NULL;
return -1;
}
(*pMem)->keyFirst = tmem->keyFirst;
(*pMem)->keyLast = tmem->keyLast;
(*pMem)->numOfRows = tmem->numOfRows;
(*pMem)->maxTables = tmem->maxTables;
pSnapshot->mem->keyFirst = pSnapshot->omem->keyFirst;
pSnapshot->mem->keyLast = pSnapshot->omem->keyLast;
pSnapshot->mem->numOfRows = pSnapshot->omem->numOfRows;
pSnapshot->mem->maxTables = pSnapshot->omem->maxTables;
for (size_t i = 0; i < taosArrayGetSize(pATable); i++) {
STable * pTable = *(STable **)taosArrayGet(pATable, i);
int32_t tid = TABLE_TID(pTable);
STableData *pTableData = (tid < tmem->maxTables) ? tmem->tData[tid] : NULL;
STableData *pTableData = (tid < pSnapshot->omem->maxTables) ? pSnapshot->omem->tData[tid] : NULL;
if ((pTableData == NULL) || (TABLE_UID(pTable) != pTableData->uid)) continue;
(*pMem)->tData[tid] = tmem->tData[tid];
T_REF_INC(tmem->tData[tid]);
pSnapshot->mem->tData[tid] = pTableData;
T_REF_INC(pTableData);
}
taosRUnLockLatch(&(tmem->latch));
taosRUnLockLatch(&(pSnapshot->omem->latch));
}
tsdbUnRefMemTable(pRepo, tmem);
tsdbDebug("vgId:%d take memory snapshot, pMem %p pIMem %p", REPO_ID(pRepo), *pMem, *pIMem);
tsdbDebug("vgId:%d take memory snapshot, pMem %p pIMem %p", REPO_ID(pRepo), pSnapshot->omem, pSnapshot->imem);
return 0;
}
void tsdbUnTakeMemSnapShot(STsdbRepo *pRepo, SMemTable *pMem, SMemTable *pIMem) {
tsdbDebug("vgId:%d untake memory snapshot, pMem %p pIMem %p", REPO_ID(pRepo), pMem, pIMem);
void tsdbUnTakeMemSnapShot(STsdbRepo *pRepo, SMemSnapshot *pSnapshot) {
tsdbDebug("vgId:%d untake memory snapshot, pMem %p pIMem %p", REPO_ID(pRepo), pSnapshot->omem, pSnapshot->imem);
if (pMem != NULL) {
for (size_t i = 0; i < pMem->maxTables; i++) {
STableData *pTableData = pMem->tData[i];
if (pSnapshot->mem) {
ASSERT(pSnapshot->omem != NULL);
for (size_t i = 0; i < pSnapshot->mem->maxTables; i++) {
STableData *pTableData = pSnapshot->mem->tData[i];
if (pTableData) {
tsdbFreeTableData(pTableData);
}
}
free(pMem->tData);
free(pMem);
tfree(pSnapshot->mem->tData);
tsdbUnRefMemTable(pRepo, pSnapshot->omem);
}
if (pIMem != NULL) {
tsdbUnRefMemTable(pRepo, pIMem);
}
tsdbUnRefMemTable(pRepo, pSnapshot->imem);
pSnapshot->mem = NULL;
pSnapshot->imem = NULL;
pSnapshot->omem = NULL;
}
void *tsdbAllocBytes(STsdbRepo *pRepo, int bytes) {

6
src/tsdb/src/tsdbMeta.c
View File

@ -213,9 +213,9 @@ void *tsdbGetTableTagVal(const void* pTable, int32_t colId, int16_t type, int16_
char *val = tdGetKVRowValOfCol(((STable*)pTable)->tagVal, colId);
assert(type == pCol->type && bytes == pCol->bytes);
if (val != NULL && IS_VAR_DATA_TYPE(type)) {
assert(varDataLen(val) < pCol->bytes);
}
// if (val != NULL && IS_VAR_DATA_TYPE(type)) {
// assert(varDataLen(val) < pCol->bytes);
// }
return val;
}

32
src/tsdb/src/tsdbRead.c
View File

@ -194,7 +194,7 @@ static void tsdbMayTakeMemSnapshot(STsdbQueryHandle* pQueryHandle, SArray* psTab
SMemRef* pMemRef = pQueryHandle->pMemRef;
if (pQueryHandle->pMemRef->ref++ == 0) {
tsdbTakeMemSnapshot(pQueryHandle->pTsdb, (SMemTable**)&(pMemRef->mem), (SMemTable**)&(pMemRef->imem), psTable);
tsdbTakeMemSnapshot(pQueryHandle->pTsdb, &(pMemRef->snapshot), psTable);
}
taosArrayDestroy(psTable);
@ -208,9 +208,7 @@ static void tsdbMayUnTakeMemSnapshot(STsdbQueryHandle* pQueryHandle) {
}
if (--pMemRef->ref == 0) {
tsdbUnTakeMemSnapShot(pQueryHandle->pTsdb, pMemRef->mem, pMemRef->imem);
pMemRef->mem = NULL;
pMemRef->imem = NULL;
tsdbUnTakeMemSnapShot(pQueryHandle->pTsdb, &(pMemRef->snapshot));
}
pQueryHandle->pMemRef = NULL;
@ -231,8 +229,8 @@ int64_t tsdbGetNumOfRowsInMemTable(TsdbQueryHandleT* pHandle) {
STableData* pMem = NULL;
STableData* pIMem = NULL;
SMemTable *pMemT = (SMemTable *)(pMemRef->mem);
SMemTable *pIMemT = (SMemTable *)(pMemRef->imem);
SMemTable* pMemT = pMemRef->snapshot.mem;
SMemTable* pIMemT = pMemRef->snapshot.imem;
if (pMemT && pCheckInfo->tableId.tid < pMemT->maxTables) {
pMem = pMemT->tData[pCheckInfo->tableId.tid];
@ -605,7 +603,7 @@ static bool initTableMemIterator(STsdbQueryHandle* pHandle, STableCheckInfo* pCh
int32_t order = pHandle->order;
// no data in buffer, abort
if (pHandle->pMemRef->mem == NULL && pHandle->pMemRef->imem == NULL) {
if (pHandle->pMemRef->snapshot.mem == NULL && pHandle->pMemRef->snapshot.imem == NULL) {
return false;
}
@ -614,8 +612,8 @@ static bool initTableMemIterator(STsdbQueryHandle* pHandle, STableCheckInfo* pCh
STableData* pMem = NULL;
STableData* pIMem = NULL;
SMemTable* pMemT = pHandle->pMemRef->mem;
SMemTable* pIMemT = pHandle->pMemRef->imem;
SMemTable* pMemT = pHandle->pMemRef->snapshot.mem;
SMemTable* pIMemT = pHandle->pMemRef->snapshot.imem;
if (pMemT && pCheckInfo->tableId.tid < pMemT->maxTables) {
pMem = pMemT->tData[pCheckInfo->tableId.tid];
@ -1021,6 +1019,14 @@ static int32_t doLoadFileDataBlock(STsdbQueryHandle* pQueryHandle, SBlock* pBloc
pBlock->numOfRows = pCols->numOfRows;
// Convert from TKEY to TSKEY for primary timestamp column if current block has timestamp before 1970-01-01T00:00:00Z
if(pBlock->keyFirst < 0 && colIds[0] == PRIMARYKEY_TIMESTAMP_COL_INDEX) {
int64_t* src = pCols->cols[0].pData;
for(int32_t i = 0; i < pBlock->numOfRows; ++i) {
src[i] = tdGetKey(src[i]);
}
}
int64_t elapsedTime = (taosGetTimestampUs() - st);
pQueryHandle->cost.blockLoadTime += elapsedTime;
@ -1277,13 +1283,7 @@ int32_t doCopyRowsFromFileBlock(STsdbQueryHandle* pQueryHandle, int32_t capacity
}
if (pColInfo->info.colId == src->colId) {
if (pColInfo->info.type == TSDB_DATA_TYPE_TIMESTAMP) {
for (int32_t n = 0; n < num; n++) {
TKEY tkey = *(TKEY *)((char*)src->pData + bytes * start + n * sizeof(TKEY));
*(TSKEY *)(pData + n * sizeof(TSKEY)) = tdGetKey(tkey);
}
} else if (pColInfo->info.type != TSDB_DATA_TYPE_BINARY && pColInfo->info.type != TSDB_DATA_TYPE_NCHAR) {
if (pColInfo->info.type != TSDB_DATA_TYPE_BINARY && pColInfo->info.type != TSDB_DATA_TYPE_NCHAR) {
memmove(pData, (char*)src->pData + bytes * start, bytes * num);
} else { // handle the var-string
char* dst = pData;

44
src/tsdb/src/tsdbSync.c
View File

@ -152,14 +152,14 @@ static int32_t tsdbSyncSendMeta(SSyncH *pSynch) {
return -1;
}
int32_t writeLen = (int32_t)mf.info.size;
tsdbInfo("vgId:%d, metafile:%s will be sent, size:%d", REPO_ID(pRepo), mf.f.aname, writeLen);
int64_t writeLen = mf.info.size;
tsdbInfo("vgId:%d, metafile:%s will be sent, size:%" PRId64, REPO_ID(pRepo), mf.f.aname, writeLen);
int32_t ret = (int32_t)taosSendFile(pSynch->socketFd, TSDB_FILE_FD(&mf), 0, writeLen);
int64_t ret = taosSendFile(pSynch->socketFd, TSDB_FILE_FD(&mf), 0, writeLen);
if (ret != writeLen) {
terrno = TAOS_SYSTEM_ERROR(errno);
tsdbError("vgId:%d, failed to send metafile since %s, ret:%d writeLen:%d", REPO_ID(pRepo), tstrerror(terrno), ret,
writeLen);
tsdbError("vgId:%d, failed to send metafile since %s, ret:%" PRId64 " writeLen:%" PRId64, REPO_ID(pRepo),
tstrerror(terrno), ret, writeLen);
tsdbCloseMFile(&mf);
return -1;
}
@ -217,18 +217,18 @@ static int32_t tsdbSyncRecvMeta(SSyncH *pSynch) {
tsdbInfo("vgId:%d, metafile:%s is created", REPO_ID(pRepo), mf.f.aname);
int32_t readLen = (int32_t)pSynch->pmf->info.size;
int32_t ret = taosCopyFds(pSynch->socketFd, TSDB_FILE_FD(&mf), readLen);
int64_t readLen = pSynch->pmf->info.size;
int64_t ret = taosCopyFds(pSynch->socketFd, TSDB_FILE_FD(&mf), readLen);
if (ret != readLen) {
terrno = TAOS_SYSTEM_ERROR(errno);
tsdbError("vgId:%d, failed to recv metafile since %s, ret:%d readLen:%d", REPO_ID(pRepo), tstrerror(terrno), ret,
readLen);
tsdbError("vgId:%d, failed to recv metafile since %s, ret:%" PRId64 " readLen:%" PRId64, REPO_ID(pRepo),
tstrerror(terrno), ret, readLen);
tsdbCloseMFile(&mf);
tsdbRemoveMFile(&mf);
return -1;
}
tsdbInfo("vgId:%d, metafile is received, size:%d", REPO_ID(pRepo), readLen);
tsdbInfo("vgId:%d, metafile is received, size:%" PRId64, REPO_ID(pRepo), readLen);
mf.info = pSynch->pmf->info;
tsdbCloseMFile(&mf);
@ -463,12 +463,12 @@ static int32_t tsdbSyncRecvDFileSetArray(SSyncH *pSynch) {
tsdbInfo("vgId:%d, file:%s will be received, osize:%" PRIu64 " rsize:%" PRIu64, REPO_ID(pRepo),
pDFile->f.aname, pDFile->info.size, pRDFile->info.size);
int32_t writeLen = (int32_t)pRDFile->info.size;
int32_t ret = taosCopyFds(pSynch->socketFd, pDFile->fd, writeLen);
int64_t writeLen = pRDFile->info.size;
int64_t ret = taosCopyFds(pSynch->socketFd, pDFile->fd, writeLen);
if (ret != writeLen) {
terrno = TAOS_SYSTEM_ERROR(errno);
tsdbError("vgId:%d, failed to recv file:%s since %s, ret:%d writeLen:%d", REPO_ID(pRepo), pDFile->f.aname,
tstrerror(terrno), ret, writeLen);
tsdbError("vgId:%d, failed to recv file:%s since %s, ret:%" PRId64 " writeLen:%" PRId64, REPO_ID(pRepo),
pDFile->f.aname, tstrerror(terrno), ret, writeLen);
tsdbCloseDFileSet(&fset);
tsdbRemoveDFileSet(&fset);
return -1;
@ -476,7 +476,7 @@ static int32_t tsdbSyncRecvDFileSetArray(SSyncH *pSynch) {
// Update new file info
pDFile->info = pRDFile->info;
tsdbInfo("vgId:%d, file:%s is received, size:%d", REPO_ID(pRepo), pDFile->f.aname, writeLen);
tsdbInfo("vgId:%d, file:%s is received, size:%" PRId64, REPO_ID(pRepo), pDFile->f.aname, writeLen);
}
tsdbCloseDFileSet(&fset);
@ -575,14 +575,14 @@ static int32_t tsdbSyncSendDFileSet(SSyncH *pSynch, SDFileSet *pSet) {
return -1;
}
int32_t writeLen = (int32_t)df.info.size;
tsdbInfo("vgId:%d, file:%s will be sent, size:%d", REPO_ID(pRepo), df.f.aname, writeLen);
int64_t writeLen = df.info.size;
tsdbInfo("vgId:%d, file:%s will be sent, size:%" PRId64, REPO_ID(pRepo), df.f.aname, writeLen);
int32_t ret = (int32_t)taosSendFile(pSynch->socketFd, TSDB_FILE_FD(&df), 0, writeLen);
int64_t ret = taosSendFile(pSynch->socketFd, TSDB_FILE_FD(&df), 0, writeLen);
if (ret != writeLen) {
terrno = TAOS_SYSTEM_ERROR(errno);
tsdbError("vgId:%d, failed to send file:%s since %s, ret:%d writeLen:%d", REPO_ID(pRepo), df.f.aname,
tstrerror(terrno), ret, writeLen);
tsdbError("vgId:%d, failed to send file:%s since %s, ret:%" PRId64 " writeLen:%" PRId64, REPO_ID(pRepo),
df.f.aname, tstrerror(terrno), ret, writeLen);
tsdbCloseDFile(&df);
return -1;
}
@ -677,13 +677,13 @@ static int32_t tsdbRecvDFileSetInfo(SSyncH *pSynch) {
static int tsdbReload(STsdbRepo *pRepo, bool isMfChanged) {
// TODO: may need to stop and restart stream
if (isMfChanged) {
// if (isMfChanged) {
tsdbCloseMeta(pRepo);
tsdbFreeMeta(pRepo->tsdbMeta);
pRepo->tsdbMeta = tsdbNewMeta(REPO_CFG(pRepo));
tsdbOpenMeta(pRepo);
tsdbLoadMetaCache(pRepo, true);
}
// }
tsdbUnRefMemTable(pRepo, pRepo->mem);
tsdbUnRefMemTable(pRepo, pRepo->imem);

2
src/util/inc/tidpool.h
View File

@ -34,7 +34,7 @@ void taosIdPoolCleanUp(void *handle);
int taosIdPoolNumOfUsed(void *handle);
void taosIdPoolMarkStatus(void *handle, int id);
bool taosIdPoolMarkStatus(void *handle, int id);
#ifdef __cplusplus
}

2
src/util/inc/tsocket.h
View File

@ -28,7 +28,7 @@ int32_t taosReadn(SOCKET sock, char *buffer, int32_t len);
int32_t taosWriteMsg(SOCKET fd, void *ptr, int32_t nbytes);
int32_t taosReadMsg(SOCKET fd, void *ptr, int32_t nbytes);
int32_t taosNonblockwrite(SOCKET fd, char *ptr, int32_t nbytes);
int32_t taosCopyFds(SOCKET sfd, int32_t dfd, int64_t len);
int64_t taosCopyFds(SOCKET sfd, int32_t dfd, int64_t len);
int32_t taosSetNonblocking(SOCKET sock, int32_t on);
SOCKET taosOpenUdpSocket(uint32_t localIp, uint16_t localPort);

21
src/util/src/tidpool.c
View File

@ -104,10 +104,16 @@ void taosIdPoolCleanUp(void *handle) {
int taosIdPoolNumOfUsed(void *handle) {
id_pool_t *pIdPool = handle;
return pIdPool->maxId - pIdPool->numOfFree;
pthread_mutex_lock(&pIdPool->mutex);
int ret = pIdPool->maxId - pIdPool->numOfFree;
pthread_mutex_unlock(&pIdPool->mutex);
return ret;
}
void taosIdPoolMarkStatus(void *handle, int id) {
bool taosIdPoolMarkStatus(void *handle, int id) {
bool ret = false;
id_pool_t *pIdPool = handle;
pthread_mutex_lock(&pIdPool->mutex);
@ -115,9 +121,13 @@ void taosIdPoolMarkStatus(void *handle, int id) {
if (!pIdPool->freeList[slot]) {
pIdPool->freeList[slot] = true;
pIdPool->numOfFree--;
ret = true;
} else {
ret = false;
}
pthread_mutex_unlock(&pIdPool->mutex);
return ret;
}
int taosUpdateIdPool(id_pool_t *handle, int maxId) {
@ -148,5 +158,10 @@ int taosUpdateIdPool(id_pool_t *handle, int maxId) {
int taosIdPoolMaxSize(void *handle) {
id_pool_t *pIdPool = (id_pool_t *)handle;
return pIdPool->maxId;
pthread_mutex_lock(&pIdPool->mutex);
int ret = pIdPool->maxId;
pthread_mutex_unlock(&pIdPool->mutex);
return ret;
}

20
src/util/src/tsocket.c
View File

@ -465,36 +465,36 @@ void tinet_ntoa(char *ipstr, uint32_t ip) {
#define COPY_SIZE 32768
// sendfile shall be used
int32_t taosCopyFds(SOCKET sfd, int32_t dfd, int64_t len) {
int64_t taosCopyFds(SOCKET sfd, int32_t dfd, int64_t len) {
int64_t leftLen;
int32_t readLen, writeLen;
int64_t readLen, writeLen;
char temp[COPY_SIZE];
leftLen = len;
while (leftLen > 0) {
if (leftLen < COPY_SIZE)
readLen = (int32_t)leftLen;
readLen = leftLen;
else
readLen = COPY_SIZE; // 4K
int32_t retLen = taosReadMsg(sfd, temp, (int32_t)readLen);
int64_t retLen = taosReadMsg(sfd, temp, (int32_t)readLen);
if (readLen != retLen) {
uError("read error, readLen:%d retLen:%d len:%" PRId64 " leftLen:%" PRId64 ", reason:%s", readLen, retLen, len,
leftLen, strerror(errno));
uError("read error, readLen:%" PRId64 " retLen:%" PRId64 " len:%" PRId64 " leftLen:%" PRId64 ", reason:%s",
readLen, retLen, len, leftLen, strerror(errno));
return -1;
}
writeLen = taosWriteMsg(dfd, temp, readLen);
writeLen = taosWriteMsg(dfd, temp, (int32_t)readLen);
if (readLen != writeLen) {
uError("copy error, readLen:%d writeLen:%d len:%" PRId64 " leftLen:%" PRId64 ", reason:%s", readLen, writeLen,
len, leftLen, strerror(errno));
uError("copy error, readLen:%" PRId64 " writeLen:%" PRId64 " len:%" PRId64 " leftLen:%" PRId64 ", reason:%s",
readLen, writeLen, len, leftLen, strerror(errno));
return -1;
}
leftLen -= readLen;
}
return (int32_t)len;
return len;
}

2
src/vnode/inc/vnodeInt.h
View File

@ -40,7 +40,7 @@ typedef struct {
int32_t queuedWMsg;
int32_t queuedRMsg;
int32_t flowctrlLevel;
int32_t sequence; // for topic
int64_t sequence; // for topic
int8_t status;
int8_t role;
int8_t accessState;

3
src/vnode/inc/vnodeSync.h
View File

@ -30,8 +30,9 @@ void vnodeStopSyncFile(int32_t vgId, uint64_t fversion);
void vnodeConfirmForard(int32_t vgId, void *wparam, int32_t code);
int32_t vnodeWriteToCache(int32_t vgId, void *wparam, int32_t qtype, void *rparam);
int32_t vnodeGetVersion(int32_t vgId, uint64_t *fver, uint64_t *wver);
int32_t vnodeResetVersion(int32_t vgId, uint64_t fver);
void vnodeConfirmForward(void *pVnode, uint64_t version, int32_t code);
void vnodeConfirmForward(void *pVnode, uint64_t version, int32_t code, bool force);
#ifdef __cplusplus
}

67
src/vnode/src/vnodeMain.c
View File

@ -110,53 +110,95 @@ int32_t vnodeDrop(int32_t vgId) {
}
static int32_t vnodeAlterImp(SVnodeObj *pVnode, SCreateVnodeMsg *pVnodeCfg) {
STsdbCfg tsdbCfg = pVnode->tsdbCfg;
SSyncCfg syncCfg = pVnode->syncCfg;
int32_t dbCfgVersion = pVnode->dbCfgVersion;
int32_t vgCfgVersion = pVnode->vgCfgVersion;
int32_t code = vnodeWriteCfg(pVnodeCfg);
if (code != TSDB_CODE_SUCCESS) {
pVnode->dbCfgVersion = dbCfgVersion;
pVnode->vgCfgVersion = vgCfgVersion;
pVnode->syncCfg = syncCfg;
pVnode->tsdbCfg = tsdbCfg;
return code;
}
code = vnodeReadCfg(pVnode);
if (code != TSDB_CODE_SUCCESS) {
pVnode->dbCfgVersion = dbCfgVersion;
pVnode->vgCfgVersion = vgCfgVersion;
pVnode->syncCfg = syncCfg;
pVnode->tsdbCfg = tsdbCfg;
return code;
}
code = walAlter(pVnode->wal, &pVnode->walCfg);
if (code != TSDB_CODE_SUCCESS) {
pVnode->dbCfgVersion = dbCfgVersion;
pVnode->vgCfgVersion = vgCfgVersion;
pVnode->syncCfg = syncCfg;
pVnode->tsdbCfg = tsdbCfg;
return code;
}
bool tsdbCfgChanged = (memcmp(&tsdbCfg, &pVnode->tsdbCfg, sizeof(STsdbCfg)) != 0);
bool syncCfgChanged = (memcmp(&syncCfg, &pVnode->syncCfg, sizeof(SSyncCfg)) != 0);
vDebug("vgId:%d, tsdbchanged:%d syncchanged:%d while alter vnode", pVnode->vgId, tsdbCfgChanged, syncCfgChanged);
if (/*tsdbCfgChanged || */syncCfgChanged) {
// vnode in non-ready state and still needs to return success instead of TSDB_CODE_VND_INVALID_STATUS
// dbCfgVersion can be corrected by status msg
if (!vnodeSetUpdatingStatus(pVnode)) {
vDebug("vgId:%d, vnode is not ready, do alter operation later", pVnode->vgId);
pVnode->dbCfgVersion = dbCfgVersion;
pVnode->vgCfgVersion = vgCfgVersion;
pVnode->syncCfg = syncCfg;
pVnode->tsdbCfg = tsdbCfg;
return TSDB_CODE_SUCCESS;
}
code = syncReconfig(pVnode->sync, &pVnode->syncCfg);
if (code != TSDB_CODE_SUCCESS) {
pVnode->dbCfgVersion = dbCfgVersion;
pVnode->vgCfgVersion = vgCfgVersion;
pVnode->syncCfg = syncCfg;
pVnode->tsdbCfg = tsdbCfg;
vnodeSetReadyStatus(pVnode);
return code;
}
if (pVnode->tsdb) {
code = tsdbConfigRepo(pVnode->tsdb, &pVnode->tsdbCfg);
if (code != TSDB_CODE_SUCCESS) {
pVnode->dbCfgVersion = dbCfgVersion;
pVnode->vgCfgVersion = vgCfgVersion;
pVnode->syncCfg = syncCfg;
pVnode->tsdbCfg = tsdbCfg;
vnodeSetReadyStatus(pVnode);
return code;
}
}
vnodeSetReadyStatus(pVnode);
}
return 0;
}
int32_t vnodeAlter(void *vparam, SCreateVnodeMsg *pVnodeCfg) {
SVnodeObj *pVnode = vparam;
if (pVnode->dbCfgVersion == pVnodeCfg->cfg.dbCfgVersion && pVnode->vgCfgVersion == pVnodeCfg->cfg.vgCfgVersion) {
vDebug("vgId:%d, dbCfgVersion:%d and vgCfgVersion:%d not change", pVnode->vgId, pVnode->dbCfgVersion,
pVnode->vgCfgVersion);
return TSDB_CODE_SUCCESS;
}
// vnode in non-ready state and still needs to return success instead of TSDB_CODE_VND_INVALID_STATUS
// dbCfgVersion can be corrected by status msg
if (!vnodeSetUpdatingStatus(pVnode)) {
vDebug("vgId:%d, vnode is not ready, do alter operation later", pVnode->vgId);
vDebug("vgId:%d, current dbCfgVersion:%d vgCfgVersion:%d, input dbCfgVersion:%d vgCfgVersion:%d", pVnode->vgId,
pVnode->dbCfgVersion, pVnode->vgCfgVersion, pVnodeCfg->cfg.dbCfgVersion, pVnodeCfg->cfg.vgCfgVersion);
if (pVnode->dbCfgVersion == pVnodeCfg->cfg.dbCfgVersion && pVnode->vgCfgVersion == pVnodeCfg->cfg.vgCfgVersion) {
vDebug("vgId:%d, cfg not change", pVnode->vgId);
return TSDB_CODE_SUCCESS;
}
int32_t code = vnodeAlterImp(pVnode, pVnodeCfg);
vnodeSetReadyStatus(pVnode);
if (code != 0) {
vError("vgId:%d, failed to alter vnode, code:0x%x", pVnode->vgId, code);
@ -305,6 +347,7 @@ int32_t vnodeOpen(int32_t vgId) {
syncInfo.startSyncFileFp = vnodeStartSyncFile;
syncInfo.stopSyncFileFp = vnodeStopSyncFile;
syncInfo.getVersionFp = vnodeGetVersion;
syncInfo.resetVersionFp = vnodeResetVersion;
syncInfo.sendFileFp = tsdbSyncSend;
syncInfo.recvFileFp = tsdbSyncRecv;
syncInfo.pTsdb = pVnode->tsdb;
@ -439,7 +482,7 @@ static int32_t vnodeProcessTsdbStatus(void *arg, int32_t status, int32_t eno) {
if (status == TSDB_STATUS_COMMIT_START) {
pVnode->isCommiting = 1;
pVnode->cversion = pVnode->version;
vDebug("vgId:%d, start commit, fver:%" PRIu64 " vver:%" PRIu64, pVnode->vgId, pVnode->fversion, pVnode->version);
vInfo("vgId:%d, start commit, fver:%" PRIu64 " vver:%" PRIu64, pVnode->vgId, pVnode->fversion, pVnode->version);
if (!vnodeInInitStatus(pVnode)) {
return walRenew(pVnode->wal);
}
@ -450,7 +493,7 @@ static int32_t vnodeProcessTsdbStatus(void *arg, int32_t status, int32_t eno) {
pVnode->isCommiting = 0;
pVnode->isFull = 0;
pVnode->fversion = pVnode->cversion;
vDebug("vgId:%d, commit over, fver:%" PRIu64 " vver:%" PRIu64, pVnode->vgId, pVnode->fversion, pVnode->version);
vInfo("vgId:%d, commit over, fver:%" PRIu64 " vver:%" PRIu64, pVnode->vgId, pVnode->fversion, pVnode->version);
if (!vnodeInInitStatus(pVnode)) {
walRemoveOneOldFile(pVnode->wal);
}

2
src/vnode/src/vnodeMgmt.c
View File

@ -128,7 +128,7 @@ static void vnodeBuildVloadMsg(SVnodeObj *pVnode, SStatusMsg *pStatus) {
int64_t compStorage = 0;
int64_t pointsWritten = 0;
if (!vnodeInReadyStatus(pVnode)) return;
if (vnodeInClosingStatus(pVnode)) return;
if (pStatus->openVnodes >= TSDB_MAX_VNODES) return;
if (pVnode->tsdb) {

23
src/vnode/src/vnodeRead.c
View File

@ -247,7 +247,8 @@ static int32_t vnodeProcessQueryMsg(SVnodeObj *pVnode, SVReadMsg *pRead) {
if (contLen != 0) {
qinfo_t pQInfo = NULL;
code = qCreateQueryInfo(pVnode->tsdb, pVnode->vgId, pQueryTableMsg, &pQInfo);
uint64_t qId = 0;
code = qCreateQueryInfo(pVnode->tsdb, pVnode->vgId, pQueryTableMsg, &pQInfo, &qId);
SQueryTableRsp *pRsp = (SQueryTableRsp *)rpcMallocCont(sizeof(SQueryTableRsp));
pRsp->code = code;
@ -259,22 +260,22 @@ static int32_t vnodeProcessQueryMsg(SVnodeObj *pVnode, SVReadMsg *pRead) {
// current connect is broken
if (code == TSDB_CODE_SUCCESS) {
handle = qRegisterQInfo(pVnode->qMgmt, (uint64_t)pQInfo);
handle = qRegisterQInfo(pVnode->qMgmt, qId, (uint64_t)pQInfo);
if (handle == NULL) { // failed to register qhandle
pRsp->code = terrno;
terrno = 0;
vError("vgId:%d, QInfo:%p register qhandle failed, return to app, code:%s", pVnode->vgId, (void *)pQInfo,
vError("vgId:%d, QInfo:%"PRIu64 "-%p register qhandle failed, return to app, code:%s", pVnode->vgId, qId, (void *)pQInfo,
tstrerror(pRsp->code));
qDestroyQueryInfo(pQInfo); // destroy it directly
return pRsp->code;
} else {
assert(*handle == pQInfo);
pRsp->qhandle = htobe64((uint64_t)pQInfo);
pRsp->qhandle = htobe64(qId);
}
if (handle != NULL &&
vnodeNotifyCurrentQhandle(pRead->rpcHandle, *handle, pVnode->vgId) != TSDB_CODE_SUCCESS) {
vError("vgId:%d, QInfo:%p, query discarded since link is broken, %p", pVnode->vgId, *handle,
vError("vgId:%d, QInfo:%"PRIu64 "-%p, query discarded since link is broken, %p", pVnode->vgId, qId, *handle,
pRead->rpcHandle);
pRsp->code = TSDB_CODE_RPC_NETWORK_UNAVAIL;
qReleaseQInfo(pVnode->qMgmt, (void **)&handle, true);
@ -285,7 +286,7 @@ static int32_t vnodeProcessQueryMsg(SVnodeObj *pVnode, SVReadMsg *pRead) {
}
if (handle != NULL) {
vTrace("vgId:%d, QInfo:%p, dnode query msg disposed, create qhandle and returns to app", vgId, *handle);
vTrace("vgId:%d, QInfo:%"PRIu64 "-%p, dnode query msg disposed, create qhandle and returns to app", vgId, qId, *handle);
code = vnodePutItemIntoReadQueue(pVnode, handle, pRead->rpcHandle);
if (code != TSDB_CODE_SUCCESS) {
pRsp->code = code;
@ -349,7 +350,7 @@ static int32_t vnodeProcessFetchMsg(SVnodeObj *pVnode, SVReadMsg *pRead) {
pRetrieve->free = htons(pRetrieve->free);
pRetrieve->qhandle = htobe64(pRetrieve->qhandle);
vTrace("vgId:%d, QInfo:%p, retrieve msg is disposed, free:%d, conn:%p", pVnode->vgId, (void *)pRetrieve->qhandle,
vTrace("vgId:%d, QInfo:%" PRIu64 ", retrieve msg is disposed, free:%d, conn:%p", pVnode->vgId, pRetrieve->qhandle,
pRetrieve->free, pRead->rpcHandle);
memset(pRet, 0, sizeof(SRspRet));
@ -360,19 +361,19 @@ static int32_t vnodeProcessFetchMsg(SVnodeObj *pVnode, SVReadMsg *pRead) {
if (handle == NULL) {
code = terrno;
terrno = TSDB_CODE_SUCCESS;
} else if ((*handle) != (void *)pRetrieve->qhandle) {
} else if (!checkQIdEqual(*handle, pRetrieve->qhandle)) {
code = TSDB_CODE_QRY_INVALID_QHANDLE;
}
if (code != TSDB_CODE_SUCCESS) {
vError("vgId:%d, invalid handle in retrieving result, code:%s, QInfo:%p", pVnode->vgId, tstrerror(code), (void *)pRetrieve->qhandle);
vError("vgId:%d, invalid handle in retrieving result, code:%s, QInfo:%" PRIu64, pVnode->vgId, tstrerror(code), pRetrieve->qhandle);
vnodeBuildNoResultQueryRsp(pRet);
return code;
}
// kill current query and free corresponding resources.
if (pRetrieve->free == 1) {
vDebug("vgId:%d, QInfo:%p, retrieve msg received to kill query and free qhandle", pVnode->vgId, *handle);
vWarn("vgId:%d, QInfo:%"PRIu64 "-%p, retrieve msg received to kill query and free qhandle", pVnode->vgId, pRetrieve->qhandle, *handle);
qKillQuery(*handle);
qReleaseQInfo(pVnode->qMgmt, (void **)&handle, true);
@ -383,7 +384,7 @@ static int32_t vnodeProcessFetchMsg(SVnodeObj *pVnode, SVReadMsg *pRead) {
// register the qhandle to connect to quit query immediate if connection is broken
if (vnodeNotifyCurrentQhandle(pRead->rpcHandle, *handle, pVnode->vgId) != TSDB_CODE_SUCCESS) {
vError("vgId:%d, QInfo:%p, retrieve discarded since link is broken, %p", pVnode->vgId, *handle, pRead->rpcHandle);
vError("vgId:%d, QInfo:%"PRIu64 "-%p, retrieve discarded since link is broken, %p", pVnode->vgId, pRetrieve->qhandle, *handle, pRead->rpcHandle);
code = TSDB_CODE_RPC_NETWORK_UNAVAIL;
qKillQuery(*handle);
qReleaseQInfo(pVnode->qMgmt, (void **)&handle, true);

14
src/vnode/src/vnodeStatus.c
View File

@ -109,6 +109,8 @@ bool vnodeSetResetStatus(SVnodeObj* pVnode) {
taosMsleep(1);
}
vInfo("vgId:%d, set to reset status", pVnode->vgId);
// release local resources only after cutting off outside connections
qQueryMgmtNotifyClosed(pVnode->qMgmt);
vnodeWaitReadCompleted(pVnode);
@ -153,6 +155,18 @@ bool vnodeInReadyOrUpdatingStatus(SVnodeObj* pVnode) {
return in;
}
bool vnodeInClosingStatus(SVnodeObj* pVnode) {
bool in = false;
pthread_mutex_lock(&pVnode->statusMutex);
if (pVnode->status == TAOS_VN_STATUS_CLOSING) {
in = true;
}
pthread_mutex_unlock(&pVnode->statusMutex);
return in;
}
bool vnodeInResetStatus(SVnodeObj* pVnode) {
bool in = false;
pthread_mutex_lock(&pVnode->statusMutex);

26
src/vnode/src/vnodeSync.c
View File

@ -91,7 +91,7 @@ void vnodeStartSyncFile(int32_t vgId) {
return;
}
vDebug("vgId:%d, datafile will be synced", vgId);
vInfo("vgId:%d, datafile will be synced", vgId);
vnodeSetResetStatus(pVnode);
vnodeRelease(pVnode);
@ -147,7 +147,7 @@ int32_t vnodeGetVersion(int32_t vgId, uint64_t *fver, uint64_t *wver) {
int32_t code = 0;
if (pVnode->isCommiting) {
vDebug("vgId:%d, vnode is commiting while get version", vgId);
vInfo("vgId:%d, vnode is commiting while get version", vgId);
code = -1;
} else {
*fver = pVnode->fversion;
@ -158,7 +158,23 @@ int32_t vnodeGetVersion(int32_t vgId, uint64_t *fver, uint64_t *wver) {
return code;
}
void vnodeConfirmForward(void *vparam, uint64_t version, int32_t code) {
SVnodeObj *pVnode = vparam;
syncConfirmForward(pVnode->sync, version, code);
int32_t vnodeResetVersion(int32_t vgId, uint64_t fver) {
SVnodeObj *pVnode = vnodeAcquire(vgId);
if (pVnode == NULL) {
vError("vgId:%d, vnode not found while reset version", vgId);
return -1;
}
pVnode->fversion = fver;
pVnode->version = fver;
walResetVersion(pVnode->wal, fver);
vInfo("vgId:%d, version reset to %" PRIu64, vgId, fver);
vnodeRelease(pVnode);
return 0;
}
void vnodeConfirmForward(void *vparam, uint64_t version, int32_t code, bool force) {
SVnodeObj *pVnode = vparam;
syncConfirmForward(pVnode->sync, version, code, force);
}

5
src/vnode/src/vnodeWrite.c
View File

@ -89,7 +89,8 @@ int32_t vnodeProcessWrite(void *vparam, void *wparam, int32_t qtype, void *rpara
// forward to peers, even it is WAL/FWD, it shall be called to update version in sync
int32_t syncCode = 0;
syncCode = syncForwardToPeer(pVnode->sync, pHead, pWrite, qtype);
bool force = (pWrite == NULL ? false : pWrite->pHead.msgType != TSDB_MSG_TYPE_SUBMIT);
syncCode = syncForwardToPeer(pVnode->sync, pHead, pWrite, qtype, force);
if (syncCode < 0) return syncCode;
// write into WAL
@ -141,7 +142,7 @@ static int32_t vnodeProcessSubmitMsg(SVnodeObj *pVnode, void *pCont, SRspRet *pR
vTrace("vgId:%d, submit msg is processed", pVnode->vgId);
if (pVnode->dbType == TSDB_DB_TYPE_TOPIC && pVnode->role == TAOS_SYNC_ROLE_MASTER) {
tpUpdateTs(&pVnode->sequence, pCont);
tpUpdateTs(pVnode->vgId, &pVnode->sequence, pCont);
}
// save insert result into item

13
src/wal/src/walWrite.c
View File

@ -446,3 +446,16 @@ uint64_t walGetVersion(twalh param) {
return pWal->version;
}
// Wal version in slave (dnode1) must be reset.
// Because after the data file is recovered from peer (dnode2), the new file version in dnode1 may become smaller than origin.
// Some new wal record cannot be written to the wal file in dnode1 for wal version not reset, then fversion and the record in wal file may inconsistent,
// At this time, if dnode2 down, dnode1 switched to master. After dnode2 start and restore data from dnode1, data loss will occur
void walResetVersion(twalh param, uint64_t newVer) {
SWal *pWal = param;
if (pWal == 0) return;
wInfo("vgId:%d, version reset from %" PRIu64 " to %" PRIu64, pWal->vgId, pWal->version, newVer);
pWal->version = newVer;
}

29
tests/pytest/cluster/clusterEnvSetup/basic.py
View File

@ -18,13 +18,15 @@ import argparse
class BuildDockerCluser:
def __init__(self, hostName, user, password, configDir, numOfNodes, clusterVersion):
def __init__(self, hostName, user, password, configDir, numOfNodes, clusterVersion, dockerDir, removeFlag):
self.hostName = hostName
self.user = user
self.password = password
self.configDir = configDir
self.numOfNodes = numOfNodes
self.clusterVersion = clusterVersion
self.dockerDir = dockerDir
self.removeFlag = removeFlag
def getConnection(self):
self.conn = taos.connect(
@ -46,7 +48,10 @@ class BuildDockerCluser:
if self.numOfNodes < 2 or self.numOfNodes > 5:
print("the number of nodes must be between 2 and 5")
exit(0)
os.system("./buildClusterEnv.sh -n %d -v %s" % (self.numOfNodes, self.clusterVersion))
print("remove Flag value %s" % self.removeFlag)
if self.removeFlag == False:
os.system("./cleanClusterEnv.sh -d %s" % self.dockerDir)
os.system("./buildClusterEnv.sh -n %d -v %s -d %s" % (self.numOfNodes, self.clusterVersion, self.dockerDir))
self.getConnection()
self.createDondes()
self.startArbitrator()
@ -91,10 +96,24 @@ parser.add_argument(
'-v',
'--version',
action='store',
default='2.0.14.1',
default='2.0.17.1',
type=str,
help='the version of the cluster to be build, Default is 2.0.14.1')
help='the version of the cluster to be build, Default is 2.0.17.1')
parser.add_argument(
'-d',
'--docker-dir',
action='store',
default='/data',
type=str,
help='the data dir for docker, default is /data')
parser.add_argument(
'--flag',
action='store_true',
help='remove docker containers flag, default: True')
args = parser.parse_args()
cluster = BuildDockerCluser(args.host, args.user, args.password, args.config_dir, args.num_of_nodes, args.version)
cluster = BuildDockerCluser(args.host, args.user, args.password, args.config_dir, args.num_of_nodes, args.version, args.docker_dir, args.flag)
cluster.run()
# usage 1: python3 basic.py -n 2 --flag (flag is True)
# usage 2: python3 basic.py -n 2 (flag should be False when it is not specified)

41
tests/pytest/cluster/clusterEnvSetup/buildClusterEnv.sh
View File

@ -1,18 +1,19 @@
#!/bin/bash
echo "Executing buildClusterEnv.sh"
DOCKER_DIR=/data
CURR_DIR=`pwd`
if [ $# != 4 ]; then
if [ $# != 6 ]; then
echo "argument list need input : "
echo " -n numOfNodes"
echo " -v version"
echo " -d docker dir"
exit 1
fi
NUM_OF_NODES=
VERSION=
while getopts "n:v:" arg
DOCKER_DIR=
while getopts "n:v:d:" arg
do
case $arg in
n)
@ -21,6 +22,9 @@ do
v)
VERSION=$OPTARG
;;
d)
DOCKER_DIR=$OPTARG
;;
?)
echo "unkonwn argument"
;;
@ -30,30 +34,29 @@ done
function addTaoscfg {
for i in {1..5}
do
touch /data/node$i/cfg/taos.cfg
echo 'firstEp tdnode1:6030' > /data/node$i/cfg/taos.cfg
echo 'fqdn tdnode'$i >> /data/node$i/cfg/taos.cfg
echo 'arbitrator tdnode1:6042' >> /data/node$i/cfg/taos.cfg
touch $DOCKER_DIR/node$i/cfg/taos.cfg
echo 'firstEp tdnode1:6030' > $DOCKER_DIR/node$i/cfg/taos.cfg
echo 'fqdn tdnode'$i >> $DOCKER_DIR/node$i/cfg/taos.cfg
echo 'arbitrator tdnode1:6042' >> $DOCKER_DIR/node$i/cfg/taos.cfg
done
}
function createDIR {
for i in {1..5}
do
mkdir -p /data/node$i/data
mkdir -p /data/node$i/log
mkdir -p /data/node$i/cfg
mkdir -p /data/node$i/core
mkdir -p $DOCKER_DIR/node$i/data
mkdir -p $DOCKER_DIR/node$i/log
mkdir -p $DOCKER_DIR/node$i/cfg
mkdir -p $DOCKER_DIR/node$i/core
done
}
function cleanEnv {
echo "Clean up docker environment"
for i in {1..5}
do
echo /data/node$i/data/*
rm -rf /data/node$i/data/*
echo /data/node$i/log/*
rm -rf /data/node$i/log/*
rm -rf $DOCKER_DIR/node$i/data/*
rm -rf $DOCKER_DIR/node$i/log/*
done
}
@ -98,19 +101,19 @@ function clusterUp {
if [ $NUM_OF_NODES -eq 2 ]; then
echo "create 2 dnodes"
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION docker-compose up -d
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz TARBITRATORPKG=TDengine-arbitrator-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION DATADIR=$DOCKER_DIR docker-compose up -d
fi
if [ $NUM_OF_NODES -eq 3 ]; then
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION docker-compose -f docker-compose.yml -f node3.yml up -d
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz TARBITRATORPKG=TDengine-arbitrator-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION DATADIR=$DOCKER_DIR docker-compose -f docker-compose.yml -f node3.yml up -d
fi
if [ $NUM_OF_NODES -eq 4 ]; then
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION docker-compose -f docker-compose.yml -f node3.yml -f node4.yml up -d
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz TARBITRATORPKG=TDengine-arbitrator-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION DATADIR=$DOCKER_DIR docker-compose -f docker-compose.yml -f node3.yml -f node4.yml up -d
fi
if [ $NUM_OF_NODES -eq 5 ]; then
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION docker-compose -f docker-compose.yml -f node3.yml -f node4.yml -f node5.yml up -d
PACKAGE=TDengine-server-$VERSION-Linux-x64.tar.gz TARBITRATORPKG=TDengine-arbitrator-$VERSION-Linux-x64.tar.gz DIR=TDengine-server-$VERSION DIR2=TDengine-arbitrator-$VERSION VERSION=$VERSION DATADIR=$DOCKER_DIR docker-compose -f docker-compose.yml -f node3.yml -f node4.yml -f node5.yml up -d
fi
echo "docker compose finish"

39
tests/pytest/cluster/clusterEnvSetup/cleanClusterEnv.sh Executable file
View File

@ -0,0 +1,39 @@
#!/bin/bash
echo "Executing cleanClusterEnv.sh"
CURR_DIR=`pwd`
if [ $# != 2 ]; then
echo "argument list need input : "
echo " -d docker dir"
exit 1
fi
DOCKER_DIR=
while getopts "d:" arg
do
case $arg in
d)
DOCKER_DIR=$OPTARG
;;
?)
echo "unkonwn argument"
;;
esac
done
function removeDockerContainers {
cd $DOCKER_DIR
docker-compose down --remove-orphans
}
function cleanEnv {
echo "Clean up docker environment"
for i in {1..5}
do
rm -rf $DOCKER_DIR/node$i/data/*
rm -rf $DOCKER_DIR/node$i/log/*
done
}
removeDockerContainers
cleanEnv

20
tests/pytest/cluster/clusterEnvSetup/docker-compose.yml
View File

@ -9,6 +9,7 @@ services:
- TARBITRATORPKG=${TARBITRATORPKG}
- EXTRACTDIR=${DIR}
- EXTRACTDIR2=${DIR2}
- DATADIR=${DATADIR}
image: 'tdengine:${VERSION}'
container_name: 'tdnode1'
cap_add:
@ -32,19 +33,19 @@ services:
volumes:
# bind data directory
- type: bind
source: /data/node1/data
source: ${DATADIR}/node1/data
target: /var/lib/taos
# bind log directory
- type: bind
source: /data/node1/log
source: ${DATADIR}/node1/log
target: /var/log/taos
# bind configuration
- type: bind
source: /data/node1/cfg
source: ${DATADIR}/node1/cfg
target: /etc/taos
# bind core dump path
- type: bind
source: /data/node1/core
source: ${DATADIR}/node1/core
target: /coredump
- type: bind
source: /data
@ -61,6 +62,7 @@ services:
args:
- PACKAGE=${PACKAGE}
- EXTRACTDIR=${DIR}
- DATADIR=${DATADIR}
image: 'tdengine:${VERSION}'
container_name: 'tdnode2'
cap_add:
@ -84,22 +86,22 @@ services:
volumes:
# bind data directory
- type: bind
source: /data/node2/data
source: ${DATADIR}/node2/data
target: /var/lib/taos
# bind log directory
- type: bind
source: /data/node2/log
source: ${DATADIR}/node2/log
target: /var/log/taos
# bind configuration
- type: bind
source: /data/node2/cfg
source: ${DATADIR}/node2/cfg
target: /etc/taos
# bind core dump path
- type: bind
source: /data/node2/core
source: ${DATADIR}/node2/core
target: /coredump
- type: bind
source: /data
source: ${DATADIR}
target: /root
hostname: tdnode2
networks:

11
tests/pytest/cluster/clusterEnvSetup/node3.yml
View File

@ -7,6 +7,7 @@ services:
args:
- PACKAGE=${PACKAGE}
- EXTRACTDIR=${DIR}
- DATADIR=${DATADIR}
image: 'tdengine:${VERSION}'
container_name: 'tdnode3'
cap_add:
@ -30,22 +31,22 @@ services:
volumes:
# bind data directory
- type: bind
source: /data/node3/data
source: ${DATADIR}/node3/data
target: /var/lib/taos
# bind log directory
- type: bind
source: /data/node3/log
source: ${DATADIR}/node3/log
target: /var/log/taos
# bind configuration
- type: bind
source: /data/node3/cfg
source: ${DATADIR}/node3/cfg
target: /etc/taos
# bind core dump path
- type: bind
source: /data/node3/core
source: ${DATADIR}/node3/core
target: /coredump
- type: bind
source: /data
source: ${DATADIR}
target: /root
hostname: tdnode3
networks:

11
tests/pytest/cluster/clusterEnvSetup/node4.yml
View File

@ -7,6 +7,7 @@ services:
args:
- PACKAGE=${PACKAGE}
- EXTRACTDIR=${DIR}
- DATADIR=${DATADIR}
image: 'tdengine:${VERSION}'
container_name: 'tdnode4'
cap_add:
@ -30,22 +31,22 @@ services:
volumes:
# bind data directory
- type: bind
source: /data/node4/data
source: ${DATADIR}/node4/data
target: /var/lib/taos
# bind log directory
- type: bind
source: /data/node4/log
source: ${DATADIR}/node4/log
target: /var/log/taos
# bind configuration
- type: bind
source: /data/node4/cfg
source: ${DATADIR}/node4/cfg
target: /etc/taos
# bind core dump path
- type: bind
source: /data/node4/core
source: ${DATADIR}/node4/core
target: /coredump
- type: bind
source: /data
source: ${DATADIR}
target: /root
hostname: tdnode4
networks:

Some files were not shown because too many files have changed in this diff Show More