xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

other: merge 3.0

This commit is contained in:
Haojun Liao 2024-03-22 17:57:25 +08:00
parent 9f492a8962 635a0e7b85
commit ef6ea16d64
104 changed files with 7892 additions and 4760 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
docs/en/08-client-libraries/50-odbc.mdx
View File

@ -49,7 +49,7 @@ TDengine ODBC driver supports two kinds of connections to TDengine cluster, nati
4.2 [Connection Type]: required field, we choose "WebSocket"
4.3 [URL]: required field, the URL for the ODBC data source, for example, `http://localhost:6041` is the URL for a local TDengine cluster, `https://gw.cloud.taosdata.com?token=your_token` is the URL for a TDengine cloud service.
4.3 [URL]: required field, the URL for the ODBC data source, for example, `http://localhost:6041` is the URL for a local TDengine cluster.
4.4 [Database]: optional field, the default database to access

6
docs/en/10-deployment/02-docker.md
View File

@ -128,7 +128,7 @@ If you want to start your application in a container, you need to add the corres
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y wget
ENV TDENGINE_VERSION=3.0.0.0
RUN wget -c https://www.taosdata.com/assets-download/3.0/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
RUN wget -c https://tdengine.com/assets-download/3.0/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
&& tar xvf TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
&& cd TDengine-client-${TDENGINE_VERSION} \
&& ./install_client.sh \
@ -229,7 +229,7 @@ Here is the full Dockerfile:
```docker
FROM golang:1.17.6-buster as builder
ENV TDENGINE_VERSION=3.0.0.0
RUN wget -c https://www.taosdata.com/assets-download/3.0/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
RUN wget -c https://tdengine.com/assets-download/3.0/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
&& tar xvf TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
&& cd TDengine-client-${TDENGINE_VERSION} \
&& ./install_client.sh \
@ -245,7 +245,7 @@ RUN go build
FROM ubuntu:20.04
RUN apt-get update && apt-get install -y wget
ENV TDENGINE_VERSION=3.0.0.0
RUN wget -c https://www.taosdata.com/assets-download/3.0/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
RUN wget -c https://tdengine.com/assets-download/3.0/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
&& tar xvf TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
&& cd TDengine-client-${TDENGINE_VERSION} \
&& ./install_client.sh \

2
docs/en/10-deployment/03-k8s.md
View File

@ -8,7 +8,7 @@ description: This document describes how to deploy TDengine on Kubernetes.
As a time series database for Cloud Native architecture design, TDengine supports Kubernetes deployment. Firstly we introduce how to use YAML files to create a highly available TDengine cluster from scratch step by step for production usage, and highlight the common operations of TDengine in Kubernetes environment.
To meet [high availability ](https://docs.taosdata.com/tdinternal/high-availability/)requirements, clusters need to meet the following requirements:
To meet [high availability](../../tdinternal/high-availability/) requirements, clusters need to meet the following requirements:
- 3 or more dnodes: multiple vnodes in the same vgroup of TDengine are not allowed to be distributed in one dnode at the same time, so if you create a database with 3 replicas, the number of dnodes is greater than or equal to 3
- 3 mnodes: mnode is responsible for the management of the entire TDengine cluster. The default number of mnode in TDengine cluster is only one. If the dnode where the mnode located is dropped, the entire cluster is unavailable.

2
docs/en/14-reference/06-taosdump.md
View File

@ -19,7 +19,7 @@ Users should not use taosdump to back up raw data, environment settings, hardwar
There are two ways to install taosdump:
- Install the taosTools official installer. Please find taosTools from [Release History](https://docs.taosdata.com/releases/tools/) page and download and install it.
- Install the taosTools official installer. Please find taosTools from [Release History](../../releases/tools/) page and download and install it.
- Compile taos-tools separately and install it. Please refer to the [taos-tools](https://github.com/taosdata/taos-tools) repository for details.

4
docs/en/20-third-party/75-powerbi.md vendored
View File

@ -19,7 +19,7 @@ With TDengine ODBC driver, PowerBI can access time series data stored in TDengin
## Install Driver
Depending on your TDengine server version, download appropriate version of TDengine client package from TDengine website [Download Link](https://docs.taosdata.com/get-started/package/), or TDengine explorer if you are using a local TDengine cluster. Install the TDengine client package on same Windows machine where PowerBI is running.
Depending on your TDengine server version, download appropriate version of TDengine client package from TDengine website [Download Link](../../get-started/package/), or TDengine explorer if you are using a local TDengine cluster. Install the TDengine client package on same Windows machine where PowerBI is running.
### Configure Data Source
@ -38,7 +38,7 @@ To better use Power BI to analyze the data stored in TDengine, you need to under
1. Dimention: it's normally category (text) data to describe such information as device, collection point, model. In the supertable template of TDengine, we use tag columns to store the dimention information. You can use SQL like `select distinct tbname, tag1, tag2 from supertable` to get dimentions.
2. Metric: quantitive (numeric) fileds that can be calculated, like SUM, AVERAGE, MINIMUM. If the collecting frequency is 1 second, then there are 31,536,000 records in one year, it will be too low efficient to import so big data into Power BI. In TDengine, you can use data partition query, window partition query, in combination with pseudo columns related to window, to import downsampled data into Power BI. For more details, please refer to [TDengine Specialized Queries](https://docs.taosdata.com/taos-sql/distinguished/)。
2. Metric: quantitive (numeric) fileds that can be calculated, like SUM, AVERAGE, MINIMUM. If the collecting frequency is 1 second, then there are 31,536,000 records in one year, it will be too low efficient to import so big data into Power BI. In TDengine, you can use data partition query, window partition query, in combination with pseudo columns related to window, to import downsampled data into Power BI. For more details, please refer to [TDengine Specialized Queries](../../taos-sql/distinguished/)。
- Window partition query: for example, thermal meters collect one data per second, but you need to query the average temperature every 10 minutes, you can use window subclause to get the downsampling data you need. The corresponding SQL is like `select tbname, _wstart date，avg(temperature) temp from table interval(10m)`, in which _wstart is a pseudo column indicting the start time of a widow, 10m is the duration of the window, `avg(temperature)` indicates the aggregate value inside a window.

2
docs/en/25-application/_03-immigrate.md
View File

@ -184,8 +184,6 @@ TDengine supports the standard JDBC 3.0 interface for manipulating databases, bu
To facilitate historical data migration, we provide a plug-in for the data synchronization tool DataX, which can automatically write data into TDengine.The automatic data migration of DataX can only support the data migration process of a single value model.
For the specific usage of DataX and how to use DataX to write data to TDengine, please refer to [DataX-based TDengine Data Migration Tool](https://www.taosdata.com/engineering/16401.html).
After migrating via DataX, we found that we can significantly improve the efficiency of migrating historical data by starting multiple processes and migrating numerous metrics simultaneously. The following are some records of the migration process. We provide these as a reference for application migration.
| Number of datax instances (number of concurrent processes) | Migration record speed (pieces/second) |

2
docs/en/28-releases/02-tools.md
View File

@ -6,7 +6,7 @@ description: This document provides download links for all released versions of
taosTools installation packages can be downloaded at the following links:
For other historical version installers, please visit [here](https://www.taosdata.com/all-downloads).
For other historical version installers, please visit [here](https://tdengine.com/downloads/historical/).
import Release from "/components/ReleaseV3";

1
include/common/systable.h
View File

@ -29,6 +29,7 @@ extern "C" {
#define TSDB_INS_TABLE_QNODES "ins_qnodes"
#define TSDB_INS_TABLE_BNODES "ins_bnodes" // no longer used
#define TSDB_INS_TABLE_SNODES "ins_snodes"
#define TSDB_INS_TABLE_ARBGROUPS "ins_arbgroups"
#define TSDB_INS_TABLE_CLUSTER "ins_cluster"
#define TSDB_INS_TABLE_DATABASES "ins_databases"
#define TSDB_INS_TABLE_FUNCTIONS "ins_functions"

5
include/common/tglobal.h
View File

@ -76,6 +76,11 @@ extern int32_t tsHeartbeatInterval;
extern int32_t tsHeartbeatTimeout;
extern int32_t tsSnapReplMaxWaitN;
// arbitrator
extern int32_t tsArbHeartBeatIntervalSec;
extern int32_t tsArbCheckSyncIntervalSec;
extern int32_t tsArbSetAssignedTimeoutSec;
// vnode
extern int64_t tsVndCommitMaxIntervalMs;

7
include/common/tgrant.h
View File

@ -61,9 +61,6 @@ typedef enum {
int32_t grantCheck(EGrantType grant);
int32_t grantCheckExpire(EGrantType grant);
char* tGetMachineId();
#ifdef TD_UNIQ_GRANT
int32_t grantCheckLE(EGrantType grant);
#endif
// #ifndef GRANTS_CFG
#ifdef TD_ENTERPRISE
@ -76,7 +73,7 @@ int32_t grantCheckLE(EGrantType grant);
{.name = "state", .bytes = 9 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "timeseries", .bytes = 21 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "dnodes", .bytes = 10 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "cpu_cores", .bytes = 10 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "cpu_cores", .bytes = 13 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
}
#else
#define GRANTS_SCHEMA \
@ -88,7 +85,7 @@ int32_t grantCheckLE(EGrantType grant);
{.name = "state", .bytes = 9 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "timeseries", .bytes = 21 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "dnodes", .bytes = 10 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "cpu_cores", .bytes = 10 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
{.name = "cpu_cores", .bytes = 13 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true}, \
}
#endif
// #define GRANT_CFG_ADD

98
include/common/tmsg.h
View File

@ -150,6 +150,7 @@ typedef enum _mgmt_table {
TSDB_MGMT_TABLE_GRANTS_FULL,
TSDB_MGMT_TABLE_GRANTS_LOGS,
TSDB_MGMT_TABLE_MACHINES,
TSDB_MGMT_TABLE_ARBGROUP,
TSDB_MGMT_TABLE_MAX,
} EShowType;
@ -336,6 +337,7 @@ typedef enum ENodeType {
QUERY_NODE_SHOW_QNODES_STMT,
QUERY_NODE_SHOW_SNODES_STMT,
QUERY_NODE_SHOW_BNODES_STMT,
QUERY_NODE_SHOW_ARBGROUPS_STMT,
QUERY_NODE_SHOW_CLUSTER_STMT,
QUERY_NODE_SHOW_DATABASES_STMT,
QUERY_NODE_SHOW_FUNCTIONS_STMT,
@ -1154,6 +1156,7 @@ typedef struct {
int32_t tsdbPageSize;
int32_t sqlLen;
char* sql;
int8_t withArbitrator;
} SCreateDbReq;
int32_t tSerializeSCreateDbReq(void* buf, int32_t bufLen, SCreateDbReq* pReq);
@ -1182,6 +1185,7 @@ typedef struct {
int32_t walRetentionSize;
int32_t sqlLen;
char* sql;
int8_t withArbitrator;
} SAlterDbReq;
int32_t tSerializeSAlterDbReq(void* buf, int32_t bufLen, SAlterDbReq* pReq);
@ -1304,6 +1308,7 @@ typedef struct {
SArray* pRetensions;
int8_t schemaless;
int16_t sstTrigger;
int8_t withArbitrator;
} SDbCfgRsp;
typedef SDbCfgRsp SDbCfgInfo;
@ -2147,6 +2152,99 @@ typedef struct {
int32_t tSerializeSDCreateMnodeReq(void* buf, int32_t bufLen, SDCreateMnodeReq* pReq);
int32_t tDeserializeSDCreateMnodeReq(void* buf, int32_t bufLen, SDCreateMnodeReq* pReq);
typedef struct {
int32_t vgId;
int32_t hbSeq;
} SVArbHbReqMember;
typedef struct {
int32_t dnodeId;
char* arbToken;
int64_t arbTerm;
SArray* hbMembers; // SVArbHbReqMember
} SVArbHeartBeatReq;
int32_t tSerializeSVArbHeartBeatReq(void* buf, int32_t bufLen, SVArbHeartBeatReq* pReq);
int32_t tDeserializeSVArbHeartBeatReq(void* buf, int32_t bufLen, SVArbHeartBeatReq* pReq);
void tFreeSVArbHeartBeatReq(SVArbHeartBeatReq* pReq);
typedef struct {
int32_t vgId;
char memberToken[TSDB_ARB_TOKEN_SIZE];
int32_t hbSeq;
} SVArbHbRspMember;
typedef struct {
char arbToken[TSDB_ARB_TOKEN_SIZE];
int32_t dnodeId;
SArray* hbMembers; // SVArbHbRspMember
} SVArbHeartBeatRsp;
int32_t tSerializeSVArbHeartBeatRsp(void* buf, int32_t bufLen, SVArbHeartBeatRsp* pRsp);
int32_t tDeserializeSVArbHeartBeatRsp(void* buf, int32_t bufLen, SVArbHeartBeatRsp* pRsp);
void tFreeSVArbHeartBeatRsp(SVArbHeartBeatRsp* pRsp);
typedef struct {
char* arbToken;
int64_t arbTerm;
char* member0Token;
char* member1Token;
} SVArbCheckSyncReq;
int32_t tSerializeSVArbCheckSyncReq(void* buf, int32_t bufLen, SVArbCheckSyncReq* pReq);
int32_t tDeserializeSVArbCheckSyncReq(void* buf, int32_t bufLen, SVArbCheckSyncReq* pReq);
void tFreeSVArbCheckSyncReq(SVArbCheckSyncReq* pRsp);
typedef struct {
char* arbToken;
char* member0Token;
char* member1Token;
int32_t vgId;
int32_t errCode;
} SVArbCheckSyncRsp;
int32_t tSerializeSVArbCheckSyncRsp(void* buf, int32_t bufLen, SVArbCheckSyncRsp* pRsp);
int32_t tDeserializeSVArbCheckSyncRsp(void* buf, int32_t bufLen, SVArbCheckSyncRsp* pRsp);
void tFreeSVArbCheckSyncRsp(SVArbCheckSyncRsp* pRsp);
typedef struct {
char* arbToken;
int64_t arbTerm;
char* memberToken;
} SVArbSetAssignedLeaderReq;
int32_t tSerializeSVArbSetAssignedLeaderReq(void* buf, int32_t bufLen, SVArbSetAssignedLeaderReq* pReq);
int32_t tDeserializeSVArbSetAssignedLeaderReq(void* buf, int32_t bufLen, SVArbSetAssignedLeaderReq* pReq);
void tFreeSVArbSetAssignedLeaderReq(SVArbSetAssignedLeaderReq* pReq);
typedef struct {
char* arbToken;
char* memberToken;
int32_t vgId;
} SVArbSetAssignedLeaderRsp;
int32_t tSerializeSVArbSetAssignedLeaderRsp(void* buf, int32_t bufLen, SVArbSetAssignedLeaderRsp* pRsp);
int32_t tDeserializeSVArbSetAssignedLeaderRsp(void* buf, int32_t bufLen, SVArbSetAssignedLeaderRsp* pRsp);
void tFreeSVArbSetAssignedLeaderRsp(SVArbSetAssignedLeaderRsp* pRsp);
typedef struct {
int32_t dnodeId;
char* token;
} SMArbUpdateGroupReqMember;
typedef struct {
int32_t vgId;
int64_t dbUid;
SMArbUpdateGroupReqMember members[2];
int8_t isSync;
SMArbUpdateGroupReqMember assignedLeader;
int64_t version;
} SMArbUpdateGroupReq;
int32_t tSerializeSMArbUpdateGroupReq(void* buf, int32_t bufLen, SMArbUpdateGroupReq* pReq);
int32_t tDeserializeSMArbUpdateGroupReq(void* buf, int32_t bufLen, SMArbUpdateGroupReq* pReq);
void tFreeSMArbUpdateGroupReq(SMArbUpdateGroupReq* pReq);
typedef struct {
char queryStrId[TSDB_QUERY_ID_LEN];
} SKillQueryReq;

3
include/common/tmsgcb.h
View File

@ -36,9 +36,11 @@ typedef enum {
SYNC_QUEUE,
SYNC_RD_QUEUE,
STREAM_QUEUE,
ARB_QUEUE,
QUEUE_MAX,
} EQueueType;
typedef void (*GetDnodeEp)(void* pData, int32_t dnodeId, char* pEp, char* pFqdn, uint16_t* pPort);
typedef bool (*UpdateDnodeInfoFp)(void* pData, int32_t* dnodeId, int64_t* clusterId, char* fqdn, uint16_t* port);
typedef int32_t (*PutToQueueFp)(void* pMgmt, EQueueType qtype, SRpcMsg* pMsg);
typedef int32_t (*GetQueueSizeFp)(void* pMgmt, int32_t vgId, EQueueType qtype);
@ -64,6 +66,7 @@ typedef struct {
ReleaseHandleFp releaseHandleFp;
ReportStartup reportStartupFp;
UpdateDnodeInfoFp updateDnodeInfoFp;
GetDnodeEp getDnodeEpFp;
} SMsgCb;
void tmsgSetDefault(const SMsgCb* msgcb);

10
include/common/tmsgdef.h
View File

@ -272,6 +272,8 @@
TD_DEF_MSG_TYPE(TDMT_VND_DISABLE_WRITE, "vnode-disable-write", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_QUERY_COMPACT_PROGRESS, "vnode-query-compact-progress", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_KILL_COMPACT, "kill-compact", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_ARB_HEARTBEAT, "vnode-arb-hb", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_ARB_CHECK_SYNC, "vnode-arb-check-sync", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_MAX_MSG, "vnd-max", NULL, NULL)
TD_CLOSE_MSG_SEG(TDMT_END_VND_MSG)
@ -341,6 +343,7 @@
TD_DEF_MSG_TYPE(TDMT_SYNC_PREP_SNAPSHOT_REPLY, "sync-prep-snapshot-reply", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_SYNC_UNUSED_CODE, "sync-unused", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_SYNC_FORCE_FOLLOWER, "sync-force-become-follower", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_SYNC_SET_ASSIGNED_LEADER, "sync-set-assigned-leader", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_SYNC_MAX_MSG, "sync-max", NULL, NULL)
TD_CLOSE_MSG_SEG(TDMT_END_SYNC_MSG)
@ -367,6 +370,13 @@
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_MAX_MSG, "vnd-tmq-max", NULL, NULL)
TD_CLOSE_MSG_SEG(TDMT_END_TMQ_MSG)
TD_NEW_MSG_SEG(TDMT_MND_ARB_MSG) //9 << 8
TD_DEF_MSG_TYPE(TDMT_MND_ARB_HEARTBEAT_TIMER, "mnd-arb-hb-tmr", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_ARB_CHECK_SYNC_TIMER, "mnd-arb-check-sync-tmr", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_ARB_UPDATE_GROUP, "mnd-arb-update-group", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_ARB_MAX_MSG, "mnd-arb-max", NULL, NULL)
TD_CLOSE_MSG_SEG(TDMT_END_ARB_MSG)
TD_NEW_MSG_SEG(TDMT_MAX_MSG) // msg end mark
TD_CLOSE_MSG_SEG(TDMT_END_MAX_MSG)

401
include/common/ttokendef.h
View File

@ -170,206 +170,207 @@
#define TK_STABLES 151
#define TK_MNODES 152
#define TK_QNODES 153
#define TK_FUNCTIONS 154
#define TK_INDEXES 155
#define TK_ACCOUNTS 156
#define TK_APPS 157
#define TK_CONNECTIONS 158
#define TK_LICENCES 159
#define TK_GRANTS 160
#define TK_FULL 161
#define TK_LOGS 162
#define TK_MACHINES 163
#define TK_QUERIES 164
#define TK_SCORES 165
#define TK_TOPICS 166
#define TK_VARIABLES 167
#define TK_BNODES 168
#define TK_SNODES 169
#define TK_TRANSACTIONS 170
#define TK_DISTRIBUTED 171
#define TK_CONSUMERS 172
#define TK_SUBSCRIPTIONS 173
#define TK_VNODES 174
#define TK_ALIVE 175
#define TK_VIEWS 176
#define TK_VIEW 177
#define TK_COMPACTS 178
#define TK_NORMAL 179
#define TK_CHILD 180
#define TK_LIKE 181
#define TK_TBNAME 182
#define TK_QTAGS 183
#define TK_AS 184
#define TK_SYSTEM 185
#define TK_INDEX 186
#define TK_FUNCTION 187
#define TK_INTERVAL 188
#define TK_COUNT 189
#define TK_LAST_ROW 190
#define TK_META 191
#define TK_ONLY 192
#define TK_TOPIC 193
#define TK_CONSUMER 194
#define TK_GROUP 195
#define TK_DESC 196
#define TK_DESCRIBE 197
#define TK_RESET 198
#define TK_QUERY 199
#define TK_CACHE 200
#define TK_EXPLAIN 201
#define TK_ANALYZE 202
#define TK_VERBOSE 203
#define TK_NK_BOOL 204
#define TK_RATIO 205
#define TK_NK_FLOAT 206
#define TK_OUTPUTTYPE 207
#define TK_AGGREGATE 208
#define TK_BUFSIZE 209
#define TK_LANGUAGE 210
#define TK_REPLACE 211
#define TK_STREAM 212
#define TK_INTO 213
#define TK_PAUSE 214
#define TK_RESUME 215
#define TK_TRIGGER 216
#define TK_AT_ONCE 217
#define TK_WINDOW_CLOSE 218
#define TK_IGNORE 219
#define TK_EXPIRED 220
#define TK_FILL_HISTORY 221
#define TK_UPDATE 222
#define TK_SUBTABLE 223
#define TK_UNTREATED 224
#define TK_KILL 225
#define TK_CONNECTION 226
#define TK_TRANSACTION 227
#define TK_BALANCE 228
#define TK_VGROUP 229
#define TK_LEADER 230
#define TK_MERGE 231
#define TK_REDISTRIBUTE 232
#define TK_SPLIT 233
#define TK_DELETE 234
#define TK_INSERT 235
#define TK_NK_BIN 236
#define TK_NK_HEX 237
#define TK_NULL 238
#define TK_NK_QUESTION 239
#define TK_NK_ALIAS 240
#define TK_NK_ARROW 241
#define TK_ROWTS 242
#define TK_QSTART 243
#define TK_QEND 244
#define TK_QDURATION 245
#define TK_WSTART 246
#define TK_WEND 247
#define TK_WDURATION 248
#define TK_IROWTS 249
#define TK_ISFILLED 250
#define TK_CAST 251
#define TK_NOW 252
#define TK_TODAY 253
#define TK_TIMEZONE 254
#define TK_CLIENT_VERSION 255
#define TK_SERVER_VERSION 256
#define TK_SERVER_STATUS 257
#define TK_CURRENT_USER 258
#define TK_CASE 259
#define TK_WHEN 260
#define TK_THEN 261
#define TK_ELSE 262
#define TK_BETWEEN 263
#define TK_IS 264
#define TK_NK_LT 265
#define TK_NK_GT 266
#define TK_NK_LE 267
#define TK_NK_GE 268
#define TK_NK_NE 269
#define TK_MATCH 270
#define TK_NMATCH 271
#define TK_CONTAINS 272
#define TK_IN 273
#define TK_JOIN 274
#define TK_INNER 275
#define TK_SELECT 276
#define TK_NK_HINT 277
#define TK_DISTINCT 278
#define TK_WHERE 279
#define TK_PARTITION 280
#define TK_BY 281
#define TK_SESSION 282
#define TK_STATE_WINDOW 283
#define TK_EVENT_WINDOW 284
#define TK_COUNT_WINDOW 285
#define TK_SLIDING 286
#define TK_FILL 287
#define TK_VALUE 288
#define TK_VALUE_F 289
#define TK_NONE 290
#define TK_PREV 291
#define TK_NULL_F 292
#define TK_LINEAR 293
#define TK_NEXT 294
#define TK_HAVING 295
#define TK_RANGE 296
#define TK_EVERY 297
#define TK_ORDER 298
#define TK_SLIMIT 299
#define TK_SOFFSET 300
#define TK_LIMIT 301
#define TK_OFFSET 302
#define TK_ASC 303
#define TK_NULLS 304
#define TK_ABORT 305
#define TK_AFTER 306
#define TK_ATTACH 307
#define TK_BEFORE 308
#define TK_BEGIN 309
#define TK_BITAND 310
#define TK_BITNOT 311
#define TK_BITOR 312
#define TK_BLOCKS 313
#define TK_CHANGE 314
#define TK_COMMA 315
#define TK_CONCAT 316
#define TK_CONFLICT 317
#define TK_COPY 318
#define TK_DEFERRED 319
#define TK_DELIMITERS 320
#define TK_DETACH 321
#define TK_DIVIDE 322
#define TK_DOT 323
#define TK_EACH 324
#define TK_FAIL 325
#define TK_FILE 326
#define TK_FOR 327
#define TK_GLOB 328
#define TK_ID 329
#define TK_IMMEDIATE 330
#define TK_IMPORT 331
#define TK_INITIALLY 332
#define TK_INSTEAD 333
#define TK_ISNULL 334
#define TK_MODULES 335
#define TK_NK_BITNOT 336
#define TK_NK_SEMI 337
#define TK_NOTNULL 338
#define TK_OF 339
#define TK_PLUS 340
#define TK_PRIVILEGE 341
#define TK_RAISE 342
#define TK_RESTRICT 343
#define TK_ROW 344
#define TK_SEMI 345
#define TK_STAR 346
#define TK_STATEMENT 347
#define TK_STRICT 348
#define TK_STRING 349
#define TK_TIMES 350
#define TK_VALUES 351
#define TK_VARIABLE 352
#define TK_WAL 353
#define TK_ARBGROUPS 154
#define TK_FUNCTIONS 155
#define TK_INDEXES 156
#define TK_ACCOUNTS 157
#define TK_APPS 158
#define TK_CONNECTIONS 159
#define TK_LICENCES 160
#define TK_GRANTS 161
#define TK_FULL 162
#define TK_LOGS 163
#define TK_MACHINES 164
#define TK_QUERIES 165
#define TK_SCORES 166
#define TK_TOPICS 167
#define TK_VARIABLES 168
#define TK_BNODES 169
#define TK_SNODES 170
#define TK_TRANSACTIONS 171
#define TK_DISTRIBUTED 172
#define TK_CONSUMERS 173
#define TK_SUBSCRIPTIONS 174
#define TK_VNODES 175
#define TK_ALIVE 176
#define TK_VIEWS 177
#define TK_VIEW 178
#define TK_COMPACTS 179
#define TK_NORMAL 180
#define TK_CHILD 181
#define TK_LIKE 182
#define TK_TBNAME 183
#define TK_QTAGS 184
#define TK_AS 185
#define TK_SYSTEM 186
#define TK_INDEX 187
#define TK_FUNCTION 188
#define TK_INTERVAL 189
#define TK_COUNT 190
#define TK_LAST_ROW 191
#define TK_META 192
#define TK_ONLY 193
#define TK_TOPIC 194
#define TK_CONSUMER 195
#define TK_GROUP 196
#define TK_DESC 197
#define TK_DESCRIBE 198
#define TK_RESET 199
#define TK_QUERY 200
#define TK_CACHE 201
#define TK_EXPLAIN 202
#define TK_ANALYZE 203
#define TK_VERBOSE 204
#define TK_NK_BOOL 205
#define TK_RATIO 206
#define TK_NK_FLOAT 207
#define TK_OUTPUTTYPE 208
#define TK_AGGREGATE 209
#define TK_BUFSIZE 210
#define TK_LANGUAGE 211
#define TK_REPLACE 212
#define TK_STREAM 213
#define TK_INTO 214
#define TK_PAUSE 215
#define TK_RESUME 216
#define TK_TRIGGER 217
#define TK_AT_ONCE 218
#define TK_WINDOW_CLOSE 219
#define TK_IGNORE 220
#define TK_EXPIRED 221
#define TK_FILL_HISTORY 222
#define TK_UPDATE 223
#define TK_SUBTABLE 224
#define TK_UNTREATED 225
#define TK_KILL 226
#define TK_CONNECTION 227
#define TK_TRANSACTION 228
#define TK_BALANCE 229
#define TK_VGROUP 230
#define TK_LEADER 231
#define TK_MERGE 232
#define TK_REDISTRIBUTE 233
#define TK_SPLIT 234
#define TK_DELETE 235
#define TK_INSERT 236
#define TK_NK_BIN 237
#define TK_NK_HEX 238
#define TK_NULL 239
#define TK_NK_QUESTION 240
#define TK_NK_ALIAS 241
#define TK_NK_ARROW 242
#define TK_ROWTS 243
#define TK_QSTART 244
#define TK_QEND 245
#define TK_QDURATION 246
#define TK_WSTART 247
#define TK_WEND 248
#define TK_WDURATION 249
#define TK_IROWTS 250
#define TK_ISFILLED 251
#define TK_CAST 252
#define TK_NOW 253
#define TK_TODAY 254
#define TK_TIMEZONE 255
#define TK_CLIENT_VERSION 256
#define TK_SERVER_VERSION 257
#define TK_SERVER_STATUS 258
#define TK_CURRENT_USER 259
#define TK_CASE 260
#define TK_WHEN 261
#define TK_THEN 262
#define TK_ELSE 263
#define TK_BETWEEN 264
#define TK_IS 265
#define TK_NK_LT 266
#define TK_NK_GT 267
#define TK_NK_LE 268
#define TK_NK_GE 269
#define TK_NK_NE 270
#define TK_MATCH 271
#define TK_NMATCH 272
#define TK_CONTAINS 273
#define TK_IN 274
#define TK_JOIN 275
#define TK_INNER 276
#define TK_SELECT 277
#define TK_NK_HINT 278
#define TK_DISTINCT 279
#define TK_WHERE 280
#define TK_PARTITION 281
#define TK_BY 282
#define TK_SESSION 283
#define TK_STATE_WINDOW 284
#define TK_EVENT_WINDOW 285
#define TK_COUNT_WINDOW 286
#define TK_SLIDING 287
#define TK_FILL 288
#define TK_VALUE 289
#define TK_VALUE_F 290
#define TK_NONE 291
#define TK_PREV 292
#define TK_NULL_F 293
#define TK_LINEAR 294
#define TK_NEXT 295
#define TK_HAVING 296
#define TK_RANGE 297
#define TK_EVERY 298
#define TK_ORDER 299
#define TK_SLIMIT 300
#define TK_SOFFSET 301
#define TK_LIMIT 302
#define TK_OFFSET 303
#define TK_ASC 304
#define TK_NULLS 305
#define TK_ABORT 306
#define TK_AFTER 307
#define TK_ATTACH 308
#define TK_BEFORE 309
#define TK_BEGIN 310
#define TK_BITAND 311
#define TK_BITNOT 312
#define TK_BITOR 313
#define TK_BLOCKS 314
#define TK_CHANGE 315
#define TK_COMMA 316
#define TK_CONCAT 317
#define TK_CONFLICT 318
#define TK_COPY 319
#define TK_DEFERRED 320
#define TK_DELIMITERS 321
#define TK_DETACH 322
#define TK_DIVIDE 323
#define TK_DOT 324
#define TK_EACH 325
#define TK_FAIL 326
#define TK_FILE 327
#define TK_FOR 328
#define TK_GLOB 329
#define TK_ID 330
#define TK_IMMEDIATE 331
#define TK_IMPORT 332
#define TK_INITIALLY 333
#define TK_INSTEAD 334
#define TK_ISNULL 335
#define TK_MODULES 336
#define TK_NK_BITNOT 337
#define TK_NK_SEMI 338
#define TK_NOTNULL 339
#define TK_OF 340
#define TK_PLUS 341
#define TK_PRIVILEGE 342
#define TK_RAISE 343
#define TK_RESTRICT 344
#define TK_ROW 345
#define TK_SEMI 346
#define TK_STAR 347
#define TK_STATEMENT 348
#define TK_STRICT 349
#define TK_STRING 350
#define TK_TIMES 351
#define TK_VALUES 352
#define TK_VARIABLE 353
#define TK_WAL 354
#define TK_NK_SPACE 600

3
include/dnode/mnode/mnode.h
View File

@ -75,6 +75,9 @@ void mndStop(SMnode *pMnode);
int32_t mndIsCatchUp(SMnode *pMnode);
ESyncRole mndGetRole(SMnode *pMnode);
int64_t mndGetTerm(SMnode *pMnode);
int32_t mndGetArbToken(SMnode *pMnode, char *outToken);
/**
* @brief Get mnode monitor info.

12
include/libs/executor/storageapi.h
View File

@ -238,12 +238,12 @@ typedef struct SStoreSnapshotFn {
} SStoreSnapshotFn;
typedef struct SStoreMeta {
SMTbCursor* (*openTableMetaCursor)(void* pVnode); // metaOpenTbCursor
void (*closeTableMetaCursor)(SMTbCursor* pTbCur); // metaCloseTbCursor
void (*pauseTableMetaCursor)(SMTbCursor* pTbCur); // metaPauseTbCursor
void (*resumeTableMetaCursor)(SMTbCursor* pTbCur, int8_t first); // metaResumeTbCursor
int32_t (*cursorNext)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorNext
int32_t (*cursorPrev)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorPrev
SMTbCursor* (*openTableMetaCursor)(void* pVnode); // metaOpenTbCursor
void (*closeTableMetaCursor)(SMTbCursor* pTbCur); // metaCloseTbCursor
void (*pauseTableMetaCursor)(SMTbCursor* pTbCur); // metaPauseTbCursor
void (*resumeTableMetaCursor)(SMTbCursor* pTbCur, int8_t first, int8_t move); // metaResumeTbCursor
int32_t (*cursorNext)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorNext
int32_t (*cursorPrev)(SMTbCursor* pTbCur, ETableType jumpTableType); // metaTbCursorPrev
int32_t (*getTableTags)(void* pVnode, uint64_t suid, SArray* uidList);
int32_t (*getTableTagsByUid)(void* pVnode, int64_t suid, SArray* uidList);

1
include/libs/nodes/cmdnodes.h
View File

@ -100,6 +100,7 @@ typedef struct SDatabaseOptions {
int32_t sstTrigger;
int32_t tablePrefix;
int32_t tableSuffix;
int8_t withArbitrator;
} SDatabaseOptions;
typedef struct SCreateDatabaseStmt {

8
include/libs/sync/sync.h
View File

@ -80,6 +80,7 @@ typedef enum {
TAOS_SYNC_STATE_LEADER = 102,
TAOS_SYNC_STATE_ERROR = 103,
TAOS_SYNC_STATE_LEARNER = 104,
TAOS_SYNC_STATE_ASSIGNED_LEADER = 105,
} ESyncState;
typedef enum {
@ -184,6 +185,7 @@ typedef struct SSyncFSM {
void (*FpBecomeLeaderCb)(const struct SSyncFSM* pFsm);
void (*FpBecomeFollowerCb)(const struct SSyncFSM* pFsm);
void (*FpBecomeLearnerCb)(const struct SSyncFSM* pFsm);
void (*FpBecomeAssignedLeaderCb)(const struct SSyncFSM* pFsm);
int32_t (*FpGetSnapshot)(const struct SSyncFSM* pFsm, SSnapshot* pSnapshot, void* pReaderParam, void** ppReader);
int32_t (*FpGetSnapshotInfo)(const struct SSyncFSM* pFsm, SSnapshot* pSnapshot);
@ -273,6 +275,7 @@ int32_t syncPropose(int64_t rid, SRpcMsg* pMsg, bool isWeak, int64_t* seq);
int32_t syncCheckMember(int64_t rid);
int32_t syncIsCatchUp(int64_t rid);
ESyncRole syncGetRole(int64_t rid);
int64_t syncGetTerm(int64_t rid);
int32_t syncProcessMsg(int64_t rid, SRpcMsg* pMsg);
int32_t syncReconfig(int64_t rid, SSyncCfg* pCfg);
int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex);
@ -284,8 +287,13 @@ bool syncSnapshotSending(int64_t rid);
bool syncSnapshotRecving(int64_t rid);
int32_t syncSendTimeoutRsp(int64_t rid, int64_t seq);
int32_t syncForceBecomeFollower(SSyncNode* ths, const SRpcMsg* pRpcMsg);
int32_t syncBecomeAssignedLeader(SSyncNode* ths, SRpcMsg* pRpcMsg);
int32_t syncUpdateArbTerm(int64_t rid, SyncTerm arbTerm);
SSyncState syncGetState(int64_t rid);
int32_t syncGetArbToken(int64_t rid, char* outToken);
int32_t syncGetAssignedLogSynced(int64_t rid);
void syncGetRetryEpSet(int64_t rid, SEpSet* pEpSet);
const char* syncStr(ESyncState state);

4
include/util/taoserror.h
View File

@ -323,6 +323,9 @@ int32_t* taosGetErrno();
#define TSDB_CODE_MND_BNODE_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x03A7)
#define TSDB_CODE_MND_TOO_FEW_MNODES TAOS_DEF_ERROR_CODE(0, 0x03A8)
#define TSDB_CODE_MND_TOO_MANY_MNODES TAOS_DEF_ERROR_CODE(0, 0x03A9)
#define TSDB_CODE_MND_ARBGROUP_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x03AA)
#define TSDB_CODE_MND_ARBGROUP_NOT_EXIST TAOS_DEF_ERROR_CODE(0, 0x03AB)
#define TSDB_CODE_MND_ARB_TOKEN_MISMATCH TAOS_DEF_ERROR_CODE(0, 0x03AC)
// mnode-dnode-part2
#define TSDB_CODE_MND_TOO_MANY_DNODES TAOS_DEF_ERROR_CODE(0, 0x03B0)
@ -472,6 +475,7 @@ int32_t* taosGetErrno();
#define TSDB_CODE_VND_ALREADY_IS_VOTER TAOS_DEF_ERROR_CODE(0, 0x0533) // internal
#define TSDB_CODE_VND_DIR_ALREADY_EXIST TAOS_DEF_ERROR_CODE(0, 0x0534)
#define TSDB_CODE_VND_META_DATA_UNSAFE_DELETE TAOS_DEF_ERROR_CODE(0, 0x0535)
#define TSDB_CODE_VND_ARB_NOT_SYNCED TAOS_DEF_ERROR_CODE(0, 0x0536) // internal
// tsdb
#define TSDB_CODE_TDB_INVALID_TABLE_ID TAOS_DEF_ERROR_CODE(0, 0x0600)

23
include/util/tdef.h
View File

@ -279,6 +279,9 @@ typedef enum ELogicConditionType {
#define TSDB_SHOW_SUBQUERY_LEN 1000
#define TSDB_LOG_VAR_LEN 32
#define TSDB_ARB_GROUP_MEMBER_NUM 2
#define TSDB_ARB_TOKEN_SIZE 32
#define TSDB_TRANS_STAGE_LEN 12
#define TSDB_TRANS_TYPE_LEN 16
#define TSDB_TRANS_ERROR_LEN 512
@ -425,6 +428,10 @@ typedef enum ELogicConditionType {
#define TSDB_DB_MIN_WAL_SEGMENT_SIZE 0
#define TSDB_DEFAULT_DB_WAL_SEGMENT_SIZE 0
#define TSDB_DEFAULT_DB_WITH_ARBITRATOR 0
#define TSDB_MIN_DB_WITH_ARBITRATOR 0
#define TSDB_MAX_DB_WITH_ARBITRATOR 1
#define TSDB_MIN_ROLLUP_MAX_DELAY 1 // unit millisecond
#define TSDB_MAX_ROLLUP_MAX_DELAY (15 * 60 * 1000)
#define TSDB_MIN_ROLLUP_WATERMARK 0 // unit millisecond
@ -523,12 +530,12 @@ enum {
SND_WORKER_TYPE__UNIQUE,
};
#define DEFAULT_HANDLE 0
#define MNODE_HANDLE 1
#define QNODE_HANDLE -1
#define SNODE_HANDLE -2
#define VNODE_HANDLE -3
#define CLIENT_HANDLE -5
#define DEFAULT_HANDLE 0
#define MNODE_HANDLE 1
#define QNODE_HANDLE -1
#define SNODE_HANDLE -2
#define VNODE_HANDLE -3
#define CLIENT_HANDLE -5
#define TSDB_CONFIG_OPTION_LEN 32
#define TSDB_CONFIG_VALUE_LEN 64
@ -548,8 +555,8 @@ enum {
// sort page size by default
#define DEFAULT_PAGESIZE 4096
#define VNODE_TIMEOUT_SEC 60
#define MNODE_TIMEOUT_SEC 60
#define VNODE_TIMEOUT_SEC 60
#define MNODE_TIMEOUT_SEC 60
#define MONITOR_TABLENAME_LEN 200
#define MONITOR_TAG_NAME_LEN 100

27
source/common/src/systable.c
View File

@ -68,6 +68,16 @@ static const SSysDbTableSchema snodesSchema[] = {
{.name = "create_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP, .sysInfo = true},
};
static const SSysDbTableSchema arbGroupsSchema[] = {
{.name = "db_name", .bytes = SYSTABLE_SCH_DB_NAME_LEN, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
{.name = "vgroup_id", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = true},
{.name = "v1_dnode", .bytes = 2, .type = TSDB_DATA_TYPE_SMALLINT, .sysInfo = true},
{.name = "v2_dnode", .bytes = 2, .type = TSDB_DATA_TYPE_SMALLINT, .sysInfo = true},
{.name = "is_sync", .bytes = 2, .type = TSDB_DATA_TYPE_SMALLINT, .sysInfo = true},
{.name = "assigned_dnode", .bytes = 2, .type = TSDB_DATA_TYPE_SMALLINT, .sysInfo = true},
{.name = "assigned_token", .bytes = TSDB_ARB_TOKEN_SIZE + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
};
static const SSysDbTableSchema clusterSchema[] = {
{.name = "id", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT, .sysInfo = true},
{.name = "name", .bytes = TSDB_CLUSTER_ID_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
@ -107,6 +117,7 @@ static const SSysDbTableSchema userDBSchema[] = {
{.name = "table_suffix", .bytes = 2, .type = TSDB_DATA_TYPE_SMALLINT, .sysInfo = true},
{.name = "tsdb_pagesize", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = true},
{.name = "keep_time_offset", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false},
{.name = "with_arbitrator", .bytes = 1, .type = TSDB_DATA_TYPE_TINYINT, .sysInfo = true},
};
static const SSysDbTableSchema userFuncSchema[] = {
@ -348,21 +359,24 @@ static const SSysDbTableSchema userCompactsDetailSchema[] = {
{.name = "start_time", .bytes = 8, .type = TSDB_DATA_TYPE_TIMESTAMP, .sysInfo = false},
};
static const SSysDbTableSchema useGrantsFullSchema[] = {
static const SSysDbTableSchema userGrantsFullSchema[] = {
{.name = "grant_name", .bytes = 32 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
{.name = "display_name", .bytes = 256 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
{.name = "expire", .bytes = 32 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
{.name = "limits", .bytes = 512 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
};
static const SSysDbTableSchema useGrantsLogsSchema[] = {
static const SSysDbTableSchema userGrantsLogsSchema[] = {
{.name = "state", .bytes = 1536 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
{.name = "active", .bytes = 512 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
{.name = "machine", .bytes = TSDB_GRANT_LOG_COL_LEN + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
};
static const SSysDbTableSchema useMachinesSchema[] = {
static const SSysDbTableSchema userMachinesSchema[] = {
{.name = "id", .bytes = TSDB_CLUSTER_ID_LEN + 1 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
#ifndef TD_UNIQ_GRANT
{.name = "dnode_num", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = true},
#endif
{.name = "machine", .bytes = 7552 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = true},
};
@ -395,9 +409,10 @@ static const SSysTableMeta infosMeta[] = {
{TSDB_INS_TABLE_VIEWS, userViewsSchema, tListLen(userViewsSchema), false},
{TSDB_INS_TABLE_COMPACTS, userCompactsSchema, tListLen(userCompactsSchema), false},
{TSDB_INS_TABLE_COMPACT_DETAILS, userCompactsDetailSchema, tListLen(userCompactsDetailSchema), false},
{TSDB_INS_TABLE_GRANTS_FULL, useGrantsFullSchema, tListLen(useGrantsFullSchema), true},
{TSDB_INS_TABLE_GRANTS_LOGS, useGrantsLogsSchema, tListLen(useGrantsLogsSchema), true},
{TSDB_INS_TABLE_MACHINES, useMachinesSchema, tListLen(useMachinesSchema), true},
{TSDB_INS_TABLE_GRANTS_FULL, userGrantsFullSchema, tListLen(userGrantsFullSchema), true},
{TSDB_INS_TABLE_GRANTS_LOGS, userGrantsLogsSchema, tListLen(userGrantsLogsSchema), true},
{TSDB_INS_TABLE_MACHINES, userMachinesSchema, tListLen(userMachinesSchema), true},
{TSDB_INS_TABLE_ARBGROUPS, arbGroupsSchema, tListLen(arbGroupsSchema), true},
};
static const SSysDbTableSchema connectionsSchema[] = {

19
source/common/src/tglobal.c
View File

@ -82,6 +82,11 @@ int32_t tsMndGrantMode = 0;
bool tsMndSkipGrant = false;
bool tsEnableWhiteList = false; // ip white list cfg
// arbitrator
int32_t tsArbHeartBeatIntervalSec = 5;
int32_t tsArbCheckSyncIntervalSec = 10;
int32_t tsArbSetAssignedTimeoutSec = 30;
// dnode
int64_t tsDndStart = 0;
int64_t tsDndStartOsUptime = 0;
@ -672,6 +677,16 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
(TSDB_SYNC_SNAP_BUFFER_SIZE >> 2), CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0)
return -1;
if (cfgAddInt32(pCfg, "arbHeartBeatIntervalSec", tsArbHeartBeatIntervalSec, 1, 60 * 24 * 2, CFG_SCOPE_SERVER,
CFG_DYN_NONE) != 0)
return -1;
if (cfgAddInt32(pCfg, "arbCheckSyncIntervalSec", tsArbCheckSyncIntervalSec, 1, 60 * 24 * 2, CFG_SCOPE_SERVER,
CFG_DYN_NONE) != 0)
return -1;
if (cfgAddInt32(pCfg, "arbSetAssignedTimeoutSec", tsArbSetAssignedTimeoutSec, 1, 60 * 24 * 2, CFG_SCOPE_SERVER,
CFG_DYN_NONE) != 0)
return -1;
if (cfgAddInt64(pCfg, "mndSdbWriteDelta", tsMndSdbWriteDelta, 20, 10000, CFG_SCOPE_SERVER, CFG_DYN_ENT_SERVER) != 0)
return -1;
if (cfgAddInt64(pCfg, "mndLogRetention", tsMndLogRetention, 500, 10000, CFG_SCOPE_SERVER, CFG_DYN_NONE) != 0)
@ -1187,6 +1202,10 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
tsHeartbeatTimeout = cfgGetItem(pCfg, "syncHeartbeatTimeout")->i32;
tsSnapReplMaxWaitN = cfgGetItem(pCfg, "syncSnapReplMaxWaitN")->i32;
tsArbHeartBeatIntervalSec = cfgGetItem(pCfg, "arbHeartBeatIntervalSec")->i32;
tsArbCheckSyncIntervalSec = cfgGetItem(pCfg, "arbCheckSyncIntervalSec")->i32;
tsArbSetAssignedTimeoutSec = cfgGetItem(pCfg, "arbSetAssignedTimeoutSec")->i32;
tsMndSdbWriteDelta = cfgGetItem(pCfg, "mndSdbWriteDelta")->i64;
tsMndLogRetention = cfgGetItem(pCfg, "mndLogRetention")->i64;
tsMndSkipGrant = cfgGetItem(pCfg, "skipGrant")->bval;

6
source/common/src/tgrant.c
View File

@ -21,11 +21,5 @@
int32_t grantCheck(EGrantType grant) { return TSDB_CODE_SUCCESS; }
int32_t grantCheckExpire(EGrantType grant) { return TSDB_CODE_SUCCESS; }
#ifdef TD_UNIQ_GRANT
int32_t grantCheckLE(EGrantType grant) { return TSDB_CODE_SUCCESS; }
#endif
#else
#ifdef TD_UNIQ_GRANT
int32_t grantCheckExpire(EGrantType grant) { return TSDB_CODE_SUCCESS; }
#endif
#endif

358
source/common/src/tmsg.c
View File

@ -3024,6 +3024,9 @@ int32_t tSerializeSCreateDbReq(void *buf, int32_t bufLen, SCreateDbReq *pReq) {
if (tEncodeI32(&encoder, pReq->keepTimeOffset) < 0) return -1;
ENCODESQL();
if (tEncodeI32(&encoder, pReq->withArbitrator) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
@ -3093,8 +3096,12 @@ int32_t tDeserializeSCreateDbReq(void *buf, int32_t bufLen, SCreateDbReq *pReq)
DECODESQL();
tEndDecode(&decoder);
pReq->withArbitrator = TSDB_DEFAULT_DB_WITH_ARBITRATOR;
if (!tDecodeIsEnd(&decoder)) {
if (tDecodeI8(&decoder, &pReq->withArbitrator) < 0) return -1;
}
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
@ -3133,6 +3140,7 @@ int32_t tSerializeSAlterDbReq(void *buf, int32_t bufLen, SAlterDbReq *pReq) {
if (tEncodeI32(&encoder, pReq->walRetentionSize) < 0) return -1;
if (tEncodeI32(&encoder, pReq->keepTimeOffset) < 0) return -1;
ENCODESQL();
if (tEncodeI32(&encoder, pReq->withArbitrator) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
@ -3182,6 +3190,10 @@ int32_t tDeserializeSAlterDbReq(void *buf, int32_t bufLen, SAlterDbReq *pReq) {
}
DECODESQL();
pReq->withArbitrator = TSDB_DEFAULT_DB_WITH_ARBITRATOR;
if (!tDecodeIsEnd(&decoder)) {
if (tDecodeI8(&decoder, &pReq->withArbitrator) < 0) return -1;
}
tEndDecode(&decoder);
tDecoderClear(&decoder);
@ -3969,6 +3981,7 @@ int32_t tSerializeSDbCfgRspImpl(SEncoder *encoder, const SDbCfgRsp *pRsp) {
if (tEncodeI8(encoder, pRsp->schemaless) < 0) return -1;
if (tEncodeI16(encoder, pRsp->sstTrigger) < 0) return -1;
if (tEncodeI32(encoder, pRsp->keepTimeOffset) < 0) return -1;
if (tEncodeI8(encoder, pRsp->withArbitrator) < 0) return -1;
return 0;
}
@ -4041,6 +4054,10 @@ int32_t tDeserializeSDbCfgRspImpl(SDecoder *decoder, SDbCfgRsp *pRsp) {
if (!tDecodeIsEnd(decoder)) {
if (tDecodeI32(decoder, &pRsp->keepTimeOffset) < 0) return -1;
}
pRsp->withArbitrator = TSDB_DEFAULT_DB_WITH_ARBITRATOR;
if (!tDecodeIsEnd(decoder)) {
if (tDecodeI8(decoder, &pRsp->withArbitrator) < 0) return -1;
}
return 0;
}
@ -5916,6 +5933,345 @@ int32_t tDeserializeSDCreateMnodeReq(void *buf, int32_t bufLen, SDCreateMnodeReq
return 0;
}
int32_t tSerializeSVArbHeartBeatReq(void *buf, int32_t bufLen, SVArbHeartBeatReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeI32(&encoder, pReq->dnodeId) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->arbToken) < 0) return -1;
if (tEncodeI64(&encoder, pReq->arbTerm) < 0) return -1;
int32_t size = taosArrayGetSize(pReq->hbMembers);
if (tEncodeI32(&encoder, size) < 0) return -1;
for (int32_t i = 0; i < size; i++) {
SVArbHbReqMember *pMember = taosArrayGet(pReq->hbMembers, i);
if (tEncodeI32(&encoder, pMember->vgId) < 0) return -1;
if (tEncodeI32(&encoder, pMember->hbSeq) < 0) return -1;
}
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVArbHeartBeatReq(void *buf, int32_t bufLen, SVArbHeartBeatReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeI32(&decoder, &pReq->dnodeId) < 0) return -1;
pReq->arbToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->arbToken) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->arbTerm) < 0) return -1;
pReq->hbMembers = taosArrayInit(16, sizeof(SVArbHbReqMember));
int32_t size = 0;
if (tDecodeI32(&decoder, &size) < 0) return -1;
for (int32_t i = 0; i < size; i++) {
SVArbHbReqMember member = {0};
if (tDecodeI32(&decoder, &member.vgId) < 0) return -1;
if (tDecodeI32(&decoder, &member.hbSeq) < 0) return -1;
taosArrayPush(pReq->hbMembers, &member);
}
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSVArbHeartBeatReq(SVArbHeartBeatReq *pReq) {
if (!pReq) return;
taosMemoryFree(pReq->arbToken);
taosArrayDestroy(pReq->hbMembers);
}
int32_t tSerializeSVArbHeartBeatRsp(void *buf, int32_t bufLen, SVArbHeartBeatRsp *pRsp) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->arbToken) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->dnodeId) < 0) return -1;
int32_t sz = taosArrayGetSize(pRsp->hbMembers);
if (tEncodeI32(&encoder, sz) < 0) return -1;
for (int32_t i = 0; i < sz; i++) {
SVArbHbRspMember *pMember = taosArrayGet(pRsp->hbMembers, i);
if (tEncodeI32(&encoder, pMember->vgId) < 0) return -1;
if (tEncodeI32(&encoder, pMember->hbSeq) < 0) return -1;
if (tEncodeCStr(&encoder, pMember->memberToken) < 0) return -1;
}
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVArbHeartBeatRsp(void *buf, int32_t bufLen, SVArbHeartBeatRsp *pRsp) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeCStrTo(&decoder, pRsp->arbToken) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->dnodeId) < 0) return -1;
int32_t sz = 0;
if (tDecodeI32(&decoder, &sz) < 0) return -1;
pRsp->hbMembers = taosArrayInit(sz, sizeof(SVArbHbRspMember));
for (int32_t i = 0; i < sz; i++) {
SVArbHbRspMember hbMember = {0};
if (tDecodeI32(&decoder, &hbMember.vgId) < 0) return -1;
if (tDecodeI32(&decoder, &hbMember.hbSeq) < 0) return -1;
if (tDecodeCStrTo(&decoder, hbMember.memberToken) < 0) return -1;
taosArrayPush(pRsp->hbMembers, &hbMember);
}
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSVArbHeartBeatRsp(SVArbHeartBeatRsp *pRsp) {
if (NULL == pRsp) {
return;
}
taosArrayDestroy(pRsp->hbMembers);
}
int32_t tSerializeSVArbCheckSyncReq(void *buf, int32_t bufLen, SVArbCheckSyncReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->arbToken) < 0) return -1;
if (tEncodeI64(&encoder, pReq->arbTerm) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->member0Token) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->member1Token) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVArbCheckSyncReq(void *buf, int32_t bufLen, SVArbCheckSyncReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
pReq->arbToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->arbToken) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->arbTerm) < 0) return -1;
pReq->member0Token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->member0Token) < 0) return -1;
pReq->member1Token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->member1Token) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSVArbCheckSyncReq(SVArbCheckSyncReq *pRsp) {
if (NULL == pRsp) {
return;
}
taosMemoryFreeClear(pRsp->arbToken);
taosMemoryFreeClear(pRsp->member0Token);
taosMemoryFreeClear(pRsp->member1Token);
}
int32_t tSerializeSVArbCheckSyncRsp(void *buf, int32_t bufLen, SVArbCheckSyncRsp *pRsp) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->arbToken) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->member0Token) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->member1Token) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->vgId) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->errCode) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVArbCheckSyncRsp(void *buf, int32_t bufLen, SVArbCheckSyncRsp *pRsp) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
pRsp->arbToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pRsp->arbToken) < 0) return -1;
pRsp->member0Token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pRsp->member0Token) < 0) return -1;
pRsp->member1Token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pRsp->member1Token) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->vgId) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->errCode) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSVArbCheckSyncRsp(SVArbCheckSyncRsp *pRsp) {
if (NULL == pRsp) {
return;
}
taosMemoryFreeClear(pRsp->arbToken);
taosMemoryFreeClear(pRsp->member0Token);
taosMemoryFreeClear(pRsp->member1Token);
}
int32_t tSerializeSVArbSetAssignedLeaderReq(void *buf, int32_t bufLen, SVArbSetAssignedLeaderReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->arbToken) < 0) return -1;
if (tEncodeI64(&encoder, pReq->arbTerm) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->memberToken) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVArbSetAssignedLeaderReq(void *buf, int32_t bufLen, SVArbSetAssignedLeaderReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
pReq->arbToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->arbToken) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->arbTerm) < 0) return -1;
pReq->memberToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->memberToken) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSVArbSetAssignedLeaderReq(SVArbSetAssignedLeaderReq *pReq) {
if (NULL == pReq) {
return;
}
taosMemoryFreeClear(pReq->arbToken);
taosMemoryFreeClear(pReq->memberToken);
}
int32_t tSerializeSVArbSetAssignedLeaderRsp(void *buf, int32_t bufLen, SVArbSetAssignedLeaderRsp *pRsp) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->arbToken) < 0) return -1;
if (tEncodeCStr(&encoder, pRsp->memberToken) < 0) return -1;
if (tEncodeI32(&encoder, pRsp->vgId) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVArbSetAssignedLeaderRsp(void *buf, int32_t bufLen, SVArbSetAssignedLeaderRsp *pRsp) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
pRsp->arbToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pRsp->arbToken) < 0) return -1;
pRsp->memberToken = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pRsp->memberToken) < 0) return -1;
if (tDecodeI32(&decoder, &pRsp->vgId) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSVArbSetAssignedLeaderRsp(SVArbSetAssignedLeaderRsp *pRsp) {
if (NULL == pRsp) {
return;
}
taosMemoryFreeClear(pRsp->arbToken);
taosMemoryFreeClear(pRsp->memberToken);
}
int32_t tSerializeSMArbUpdateGroupReq(void *buf, int32_t bufLen, SMArbUpdateGroupReq *pReq) {
SEncoder encoder = {0};
tEncoderInit(&encoder, buf, bufLen);
if (tStartEncode(&encoder) < 0) return -1;
if (tEncodeI32(&encoder, pReq->vgId) < 0) return -1;
if (tEncodeI64(&encoder, pReq->dbUid) < 0) return -1;
for (int i = 0; i < TSDB_ARB_GROUP_MEMBER_NUM; i++) {
if (tEncodeI32(&encoder, pReq->members[i].dnodeId) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->members[i].token) < 0) return -1;
}
if (tEncodeI8(&encoder, pReq->isSync) < 0) return -1;
if (tEncodeI32(&encoder, pReq->assignedLeader.dnodeId) < 0) return -1;
if (tEncodeCStr(&encoder, pReq->assignedLeader.token) < 0) return -1;
if (tEncodeI64(&encoder, pReq->version) < 0) return -1;
tEndEncode(&encoder);
int32_t tlen = encoder.pos;
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSMArbUpdateGroupReq(void *buf, int32_t bufLen, SMArbUpdateGroupReq *pReq) {
SDecoder decoder = {0};
tDecoderInit(&decoder, buf, bufLen);
if (tStartDecode(&decoder) < 0) return -1;
if (tDecodeI32(&decoder, &pReq->vgId) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->dbUid) < 0) return -1;
for (int i = 0; i < TSDB_ARB_GROUP_MEMBER_NUM; i++) {
if (tDecodeI32(&decoder, &pReq->members[i].dnodeId) < 0) return -1;
pReq->members[i].token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->members[i].token) < 0) return -1;
}
if (tDecodeI8(&decoder, &pReq->isSync) < 0) return -1;
if (tDecodeI32(&decoder, &pReq->assignedLeader.dnodeId) < 0) return -1;
pReq->assignedLeader.token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE);
if (tDecodeCStrTo(&decoder, pReq->assignedLeader.token) < 0) return -1;
if (tDecodeI64(&decoder, &pReq->version) < 0) return -1;
tEndDecode(&decoder);
tDecoderClear(&decoder);
return 0;
}
void tFreeSMArbUpdateGroupReq(SMArbUpdateGroupReq *pReq) {
if (NULL == pReq) {
return;
}
for (int i = 0; i < 2; i++) {
taosMemoryFreeClear(pReq->members[i].token);
}
taosMemoryFreeClear(pReq->assignedLeader.token);
}
// int32_t tSerializeSAuthReq(void *buf, int32_t bufLen, SAuthReq *pReq) {
// SEncoder encoder = {0};
// tEncoderInit(&encoder, buf, bufLen);

10
source/dnode/mgmt/mgmt_dnode/src/dmHandle.c
View File

@ -90,16 +90,6 @@ static void dmProcessStatusRsp(SDnodeMgmt *pMgmt, SRpcMsg *pRsp) {
rpcFreeCont(pRsp->pCont);
}
void dmEpSetToStr(char *buf, int32_t len, SEpSet *epSet) {
int32_t n = 0;
n += snprintf(buf + n, len - n, "%s", "{");
for (int i = 0; i < epSet->numOfEps; i++) {
n += snprintf(buf + n, len - n, "%s:%d%s", epSet->eps[i].fqdn, epSet->eps[i].port,
(i + 1 < epSet->numOfEps ? ", " : ""));
}
n += snprintf(buf + n, len - n, "%s", "}");
}
void dmSendStatusReq(SDnodeMgmt *pMgmt) {
SStatusReq req = {0};

2
source/dnode/mgmt/mgmt_mnode/inc/mmInt.h
View File

@ -33,6 +33,7 @@ typedef struct SMnodeMgmt {
SSingleWorker fetchWorker;
SSingleWorker readWorker;
SSingleWorker writeWorker;
SSingleWorker arbWorker;
SSingleWorker syncWorker;
SSingleWorker syncRdWorker;
bool stopped;
@ -53,6 +54,7 @@ int32_t mmProcessDropReq(const SMgmtInputOpt *pInput, SRpcMsg *pMsg);
int32_t mmStartWorker(SMnodeMgmt *pMgmt);
void mmStopWorker(SMnodeMgmt *pMgmt);
int32_t mmPutMsgToWriteQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t mmPutMsgToArbQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t mmPutMsgToSyncQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t mmPutMsgToSyncRdQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t mmPutMsgToReadQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg);

3
source/dnode/mgmt/mgmt_mnode/src/mmHandle.c
View File

@ -235,6 +235,9 @@ SArray *mmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_VND_CREATE_INDEX_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_INDEX_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DISABLE_WRITE_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ARB_HEARTBEAT_RSP, mmPutMsgToArbQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ARB_CHECK_SYNC_RSP, mmPutMsgToArbQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SYNC_SET_ASSIGNED_LEADER_RSP, mmPutMsgToArbQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SYNC_TIMEOUT_ELECTION, mmPutMsgToSyncQueue, 1) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SYNC_CLIENT_REQUEST, mmPutMsgToSyncQueue, 1) == NULL) goto _OVER;

20
source/dnode/mgmt/mgmt_mnode/src/mmWorker.c
View File

@ -107,6 +107,10 @@ int32_t mmPutMsgToWriteQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) {
return mmPutMsgToWorker(pMgmt, &pMgmt->writeWorker, pMsg);
}
int32_t mmPutMsgToArbQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) {
return mmPutMsgToWorker(pMgmt, &pMgmt->arbWorker, pMsg);
}
int32_t mmPutMsgToSyncQueue(SMnodeMgmt *pMgmt, SRpcMsg *pMsg) {
return mmPutMsgToWorker(pMgmt, &pMgmt->syncWorker, pMsg);
}
@ -153,6 +157,9 @@ int32_t mmPutMsgToQueue(SMnodeMgmt *pMgmt, EQueueType qtype, SRpcMsg *pRpc) {
case READ_QUEUE:
pWorker = &pMgmt->readWorker;
break;
case ARB_QUEUE:
pWorker = &pMgmt->arbWorker;
break;
case SYNC_QUEUE:
pWorker = &pMgmt->syncWorker;
break;
@ -252,6 +259,18 @@ int32_t mmStartWorker(SMnodeMgmt *pMgmt) {
return -1;
}
SSingleWorkerCfg arbCfg = {
.min = 1,
.max = 1,
.name = "mnode-arb",
.fp = (FItem)mmProcessRpcMsg,
.param = pMgmt,
};
if (tSingleWorkerInit(&pMgmt->arbWorker, &arbCfg) != 0) {
dError("failed to start mnode mnode-arb worker since %s", terrstr());
return -1;
}
dDebug("mnode workers are initialized");
return 0;
}
@ -263,6 +282,7 @@ void mmStopWorker(SMnodeMgmt *pMgmt) {
tSingleWorkerCleanup(&pMgmt->fetchWorker);
tSingleWorkerCleanup(&pMgmt->readWorker);
tSingleWorkerCleanup(&pMgmt->writeWorker);
tSingleWorkerCleanup(&pMgmt->arbWorker);
tSingleWorkerCleanup(&pMgmt->syncWorker);
tSingleWorkerCleanup(&pMgmt->syncRdWorker);
dDebug("mnode workers are closed");

1
source/dnode/mgmt/mgmt_vnode/inc/vmInt.h
View File

@ -101,6 +101,7 @@ int32_t vmProcessDisableVnodeWriteReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t vmProcessAlterHashRangeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t vmProcessAlterVnodeTypeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t vmProcessCheckLearnCatchupReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg);
int32_t vmProcessArbHeartBeatReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg);
// vmFile.c
int32_t vmGetVnodeListFromFile(SVnodeMgmt *pMgmt, SWrapperCfg **ppCfgs, int32_t *numOfVnodes);

91
source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
View File

@ -88,7 +88,9 @@ void vmGetMonitorInfo(SVnodeMgmt *pMgmt, SMonVmInfo *pInfo) {
numOfInsertSuccessReqs += pLoad->numOfInsertSuccessReqs;
numOfBatchInsertReqs += pLoad->numOfBatchInsertReqs;
numOfBatchInsertSuccessReqs += pLoad->numOfBatchInsertSuccessReqs;
if (pLoad->syncState == TAOS_SYNC_STATE_LEADER) masterNum++;
if (pLoad->syncState == TAOS_SYNC_STATE_LEADER || pLoad->syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
masterNum++;
}
totalVnodes++;
}
@ -770,6 +772,89 @@ int32_t vmProcessDropVnodeReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
return 0;
}
int32_t vmProcessArbHeartBeatReq(SVnodeMgmt *pMgmt, SRpcMsg *pMsg) {
SVArbHeartBeatReq arbHbReq = {0};
SVArbHeartBeatRsp arbHbRsp = {0};
if (tDeserializeSVArbHeartBeatReq(pMsg->pCont, pMsg->contLen, &arbHbReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
return -1;
}
if (arbHbReq.dnodeId != pMgmt->pData->dnodeId) {
terrno = TSDB_CODE_INVALID_MSG;
dError("dnodeId:%d not matched with local dnode", arbHbReq.dnodeId);
goto _OVER;
}
if (strlen(arbHbReq.arbToken) == 0) {
terrno = TSDB_CODE_INVALID_MSG;
dError("dnodeId:%d arbToken is empty", arbHbReq.dnodeId);
goto _OVER;
}
size_t size = taosArrayGetSize(arbHbReq.hbMembers);
arbHbRsp.dnodeId = pMgmt->pData->dnodeId;
strncpy(arbHbRsp.arbToken, arbHbReq.arbToken, TSDB_ARB_TOKEN_SIZE);
arbHbRsp.hbMembers = taosArrayInit(size, sizeof(SVArbHbRspMember));
for (int32_t i = 0; i < size; i++) {
SVArbHbReqMember *pReqMember = taosArrayGet(arbHbReq.hbMembers, i);
SVnodeObj *pVnode = vmAcquireVnode(pMgmt, pReqMember->vgId);
if (pVnode == NULL) {
dError("dnodeId:%d vgId:%d not found failed to process arb hb req", arbHbReq.dnodeId, pReqMember->vgId);
continue;
}
SVArbHbRspMember rspMember = {0};
rspMember.vgId = pReqMember->vgId;
rspMember.hbSeq = pReqMember->hbSeq;
if (vnodeGetArbToken(pVnode->pImpl, rspMember.memberToken) != 0) {
dError("dnodeId:%d vgId:%d failed to get arb token", arbHbReq.dnodeId, pReqMember->vgId);
vmReleaseVnode(pMgmt, pVnode);
continue;
}
if (vnodeUpdateArbTerm(pVnode->pImpl, arbHbReq.arbTerm) != 0) {
dError("dnodeId:%d vgId:%d failed to update arb term", arbHbReq.dnodeId, pReqMember->vgId);
vmReleaseVnode(pMgmt, pVnode);
continue;
}
taosArrayPush(arbHbRsp.hbMembers, &rspMember);
vmReleaseVnode(pMgmt, pVnode);
}
SRpcMsg rspMsg = {.info = pMsg->info};
int32_t rspLen = tSerializeSVArbHeartBeatRsp(NULL, 0, &arbHbRsp);
if (rspLen < 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _OVER;
}
void *pRsp = rpcMallocCont(rspLen);
if (pRsp == NULL) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _OVER;
}
if (tSerializeSVArbHeartBeatRsp(pRsp, rspLen, &arbHbRsp) <= 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
rpcFreeCont(pRsp);
goto _OVER;
}
pMsg->info.rsp = pRsp;
pMsg->info.rspLen = rspLen;
terrno = TSDB_CODE_SUCCESS;
_OVER:
tFreeSVArbHeartBeatReq(&arbHbReq);
tFreeSVArbHeartBeatRsp(&arbHbRsp);
return terrno == TSDB_CODE_SUCCESS ? 0 : -1;
}
SArray *vmGetMsgHandles() {
int32_t code = -1;
SArray *pArray = taosArrayInit(32, sizeof(SMgmtHandle));
@ -873,6 +958,10 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_SYNC_SNAPSHOT_RSP, vmPutMsgToSyncRdQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SYNC_PREP_SNAPSHOT_REPLY, vmPutMsgToSyncRdQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ARB_HEARTBEAT, vmPutMsgToMgmtQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_ARB_CHECK_SYNC, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_SYNC_SET_ASSIGNED_LEADER, vmPutMsgToSyncQueue, 0) == NULL) goto _OVER;
code = 0;
_OVER:

3
source/dnode/mgmt/mgmt_vnode/src/vmWorker.c
View File

@ -56,6 +56,9 @@ static void vmProcessMgmtQueue(SQueueInfo *pInfo, SRpcMsg *pMsg) {
case TDMT_DND_CHECK_VNODE_LEARNER_CATCHUP:
code = vmProcessCheckLearnCatchupReq(pMgmt, pMsg);
break;
case TDMT_VND_ARB_HEARTBEAT:
code = vmProcessArbHeartBeatReq(pMgmt, pMsg);
break;
default:
terrno = TSDB_CODE_MSG_NOT_PROCESSED;
dGError("msg:%p, not processed in vnode-mgmt queue", pMsg);

1
source/dnode/mgmt/node_mgmt/src/dmTransport.c
View File

@ -538,6 +538,7 @@ SMsgCb dmGetMsgcb(SDnode *pDnode) {
.releaseHandleFp = dmReleaseHandle,
.reportStartupFp = dmReportStartup,
.updateDnodeInfoFp = dmUpdateDnodeInfo,
.getDnodeEpFp = dmGetDnodeEp,
.data = &pDnode->data,
};
return msgCb;

2
source/dnode/mgmt/node_util/inc/dmUtil.h
View File

@ -184,11 +184,13 @@ int32_t dmReadEps(SDnodeData *pData);
int32_t dmWriteEps(SDnodeData *pData);
void dmUpdateEps(SDnodeData *pData, SArray *pDnodeEps);
void dmGetMnodeEpSet(SDnodeData *pData, SEpSet *pEpSet);
void dmEpSetToStr(char *buf, int32_t len, SEpSet *epSet);
void dmRotateMnodeEpSet(SDnodeData *pData);
void dmGetMnodeEpSetForRedirect(SDnodeData *pData, SRpcMsg *pMsg, SEpSet *pEpSet);
void dmSetMnodeEpSet(SDnodeData *pData, SEpSet *pEpSet);
bool dmUpdateDnodeInfo(void *pData, int32_t *dnodeId, int64_t *clusterId, char *fqdn, uint16_t *port);
void dmRemoveDnodePairs(SDnodeData *pData);
void dmGetDnodeEp(void *pData, int32_t dnodeId, char *pEp, char *pFqdn, uint16_t *pPort);
#ifdef __cplusplus
}

13
source/dnode/mgmt/node_util/src/dmEps.c
View File

@ -31,7 +31,8 @@ static bool dmIsEpChanged(SDnodeData *pData, int32_t dnodeId, const char *ep)
static void dmResetEps(SDnodeData *pData, SArray *dnodeEps);
static int32_t dmReadDnodePairs(SDnodeData *pData);
static void dmGetDnodeEp(SDnodeData *pData, int32_t dnodeId, char *pEp, char *pFqdn, uint16_t *pPort) {
void dmGetDnodeEp(void *data, int32_t dnodeId, char *pEp, char *pFqdn, uint16_t *pPort) {
SDnodeData *pData = data;
taosThreadRwlockRdlock(&pData->lock);
SDnodeEp *pDnodeEp = taosHashGet(pData->dnodeHash, &dnodeId, sizeof(int32_t));
@ -335,6 +336,16 @@ void dmGetMnodeEpSet(SDnodeData *pData, SEpSet *pEpSet) {
taosThreadRwlockUnlock(&pData->lock);
}
void dmEpSetToStr(char *buf, int32_t len, SEpSet *epSet) {
int32_t n = 0;
n += snprintf(buf + n, len - n, "%s", "{");
for (int i = 0; i < epSet->numOfEps; i++) {
n += snprintf(buf + n, len - n, "%s:%d%s", epSet->eps[i].fqdn, epSet->eps[i].port,
(i + 1 < epSet->numOfEps ? ", " : ""));
}
n += snprintf(buf + n, len - n, "%s", "}");
}
static FORCE_INLINE void dmSwapEps(SEp *epLhs, SEp *epRhs) {
SEp epTmp;

53
source/dnode/mnode/impl/inc/mndArbGroup.h Normal file
View File

@ -0,0 +1,53 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef _TD_MND_ARBGROUP_H_
#define _TD_MND_ARBGROUP_H_
#include "mndInt.h"
#ifdef __cplusplus
extern "C" {
#endif
int32_t mndInitArbGroup(SMnode *pMnode);
void mndCleanupArbGroup(SMnode *pMnode);
SArbGroup *mndAcquireArbGroup(SMnode *pMnode, int32_t vgId);
void mndReleaseArbGroup(SMnode *pMnode, SArbGroup *pObj);
SSdbRaw *mndArbGroupActionEncode(SArbGroup *pGroup);
SSdbRow *mndArbGroupActionDecode(SSdbRaw *pRaw);
void mndArbGroupInitFromVgObj(SVgObj *pVgObj, SArbGroup *outGroup);
int32_t mndSetCreateArbGroupRedoLogs(STrans *pTrans, SArbGroup *pGroup);
int32_t mndSetCreateArbGroupUndoLogs(STrans *pTrans, SArbGroup *pGroup);
int32_t mndSetCreateArbGroupCommitLogs(STrans *pTrans, SArbGroup *pGroup);
int32_t mndSetDropArbGroupCommitLogs(STrans *pTrans, SArbGroup *pGroup);
bool mndUpdateArbGroupByHeartBeat(SArbGroup *pGroup, SVArbHbRspMember *pRspMember, int64_t nowMs, int32_t dnodeId,
SArbGroup *pNewGroup);
bool mndUpdateArbGroupByCheckSync(SArbGroup *pGroup, int32_t vgId, char *member0Token, char *member1Token,
bool newIsSync, SArbGroup *pNewGroup);
bool mndUpdateArbGroupBySetAssignedLeader(SArbGroup *pGroup, int32_t vgId, char *memberToken, int32_t errcode,
SArbGroup *pNewGroup);
int32_t mndGetArbGroupSize(SMnode *pMnode);
#ifdef __cplusplus
}
#endif
#endif /*_TD_MND_ARBGROUP_H_*/

39
source/dnode/mnode/impl/inc/mndDef.h
View File

@ -104,6 +104,7 @@ typedef enum {
TRN_CONFLICT_DB_INSIDE = 3,
TRN_CONFLICT_TOPIC = 4,
TRN_CONFLICT_TOPIC_INSIDE = 5,
TRN_CONFLICT_ARBGROUP = 6,
} ETrnConflct;
typedef enum {
@ -177,6 +178,7 @@ typedef struct {
tmsg_t originRpcType;
char dbname[TSDB_TABLE_FNAME_LEN];
char stbname[TSDB_TABLE_FNAME_LEN];
int32_t arbGroupId;
int32_t startFunc;
int32_t stopFunc;
int32_t paramLen;
@ -215,8 +217,6 @@ typedef struct {
uint16_t port;
char fqdn[TSDB_FQDN_LEN];
char ep[TSDB_EP_LEN];
char active[TSDB_ACTIVE_KEY_LEN];
char connActive[TSDB_CONN_ACTIVE_KEY_LEN];
char machineId[TSDB_MACHINE_ID_LEN + 1];
} SDnodeObj;
@ -248,6 +248,40 @@ typedef struct {
SDnodeObj* pDnode;
} SSnodeObj;
typedef struct {
int32_t dnodeId;
char token[TSDB_ARB_TOKEN_SIZE];
} SArbAssignedLeader;
typedef struct {
int32_t dnodeId;
} SArbMemberInfo;
typedef struct {
int32_t nextHbSeq;
int32_t responsedHbSeq;
char token[TSDB_ARB_TOKEN_SIZE];
int64_t lastHbMs;
} SArbMemberState;
typedef struct {
SArbMemberInfo info;
SArbMemberState state;
} SArbGroupMember;
typedef struct {
int32_t vgId;
int64_t dbUid;
SArbGroupMember members[TSDB_ARB_GROUP_MEMBER_NUM];
int8_t isSync;
SArbAssignedLeader assignedLeader;
int64_t version;
// following fields will not be duplicated
bool mutexInited;
TdThreadMutex mutex;
} SArbGroup;
typedef struct {
int32_t maxUsers;
int32_t maxDbs;
@ -344,6 +378,7 @@ typedef struct {
int32_t walRollPeriod;
int64_t walRetentionSize;
int64_t walSegmentSize;
int8_t withArbitrator;
} SDbCfg;
typedef struct {

1
source/dnode/mnode/impl/inc/mndDnode.h
View File

@ -27,6 +27,7 @@ void mndCleanupDnode(SMnode *pMnode);
SDnodeObj *mndAcquireDnode(SMnode *pMnode, int32_t dnodeId);
void mndReleaseDnode(SMnode *pMnode, SDnodeObj *pDnode);
SEpSet mndGetDnodeEpset(SDnodeObj *pDnode);
SEpSet mndGetDnodeEpsetById(SMnode *pMnode, int32_t dnodeId);
int32_t mndGetDnodeSize(SMnode *pMnode);
bool mndIsDnodeOnline(SDnodeObj *pDnode, int64_t curMs);
void mndGetDnodeData(SMnode *pMnode, SArray *pDnodeInfo);

2
source/dnode/mnode/impl/inc/mndStream.h
View File

@ -107,7 +107,7 @@ SArray *mndTakeVgroupSnapshot(SMnode *pMnode, bool *allReady);
void mndKillTransImpl(SMnode *pMnode, int32_t transId, const char *pDbName);
int32_t setTransAction(STrans *pTrans, void *pCont, int32_t contLen, int32_t msgType, const SEpSet *pEpset,
int32_t retryCode);
STrans *doCreateTrans(SMnode *pMnode, SStreamObj *pStream, SRpcMsg *pReq, const char *name, const char *pMsg);
STrans *doCreateTrans(SMnode *pMnode, SStreamObj *pStream, SRpcMsg *pReq, ETrnConflct conflict, const char *name, const char *pMsg);
int32_t mndPersistTransLog(SStreamObj *pStream, STrans *pTrans, int32_t status);
SSdbRaw *mndStreamActionEncode(SStreamObj *pStream);
void killAllCheckpointTrans(SMnode *pMnode, SVgroupChangeInfo *pChangeInfo);

1
source/dnode/mnode/impl/inc/mndTrans.h
View File

@ -78,6 +78,7 @@ int32_t mndTransAppendUndoAction(STrans *pTrans, STransAction *pAction);
void mndTransSetRpcRsp(STrans *pTrans, void *pCont, int32_t contLen);
void mndTransSetCb(STrans *pTrans, ETrnFunc startFunc, ETrnFunc stopFunc, void *param, int32_t paramLen);
void mndTransSetDbName(STrans *pTrans, const char *dbname, const char *stbname);
void mndTransSetArbGroupId(STrans *pTrans, int32_t groupId);
void mndTransSetSerial(STrans *pTrans);
void mndTransSetParallel(STrans *pTrans);
void mndTransSetOper(STrans *pTrans, EOperType oper);

7
source/dnode/mnode/impl/inc/mndVgroup.h
View File

@ -29,6 +29,7 @@ void mndReleaseVgroup(SMnode *pMnode, SVgObj *pVgroup);
SSdbRaw *mndVgroupActionEncode(SVgObj *pVgroup);
SSdbRow *mndVgroupActionDecode(SSdbRaw *pRaw);
SEpSet mndGetVgroupEpset(SMnode *pMnode, const SVgObj *pVgroup);
SEpSet mndGetVgroupEpsetById(SMnode *pMnode, int32_t vgId);
int32_t mndGetVnodesNum(SMnode *pMnode, int32_t dnodeId);
void mndSortVnodeGid(SVgObj *pVgroup);
int64_t mndGetVnodesMemory(SMnode *pMnode, int32_t dnodeId);
@ -44,7 +45,7 @@ int32_t mndAddAlterVnodeAction(SMnode *, STrans *pTrans, SDbObj *pDb, SVgObj *pV
int32_t mndAddDropVnodeAction(SMnode *, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroup, SVnodeGid *pVgid, bool isRedo);
int32_t mndSetMoveVgroupsInfoToTrans(SMnode *, STrans *pTrans, int32_t dropDnodeId, bool force, bool unsafe);
int32_t mndBuildAlterVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj *pOldDb, SDbObj *pNewDb, SVgObj *pVgroup,
SArray *pArray);
SArray *pArray, SVgObj* pNewVgroup);
int32_t mndBuildCompactVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj *pDb, SVgObj *pVgroup, int64_t compactTs,
STimeWindow tw);
int32_t mndBuildRaftAlterVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj *pOldDb, SDbObj *pNewDb, SVgObj *pVgroup,
@ -54,8 +55,8 @@ void *mndBuildCreateVnodeReq(SMnode *, SDnodeObj *pDnode, SDbObj *pDb, SVgObj *p
void *mndBuildDropVnodeReq(SMnode *, SDnodeObj *pDnode, SDbObj *pDb, SVgObj *pVgroup, int32_t *pContLen);
bool mndVgroupInDb(SVgObj *pVgroup, int64_t dbUid);
bool mndVgroupInDnode(SVgObj *pVgroup, int32_t dnodeId);
int32_t mndBuildRestoreAlterVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj *db, SVgObj *pVgroup,
SDnodeObj *pDnode);
int32_t mndBuildRestoreAlterVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj *db, SVgObj *pVgroup,
SDnodeObj *pDnode);
int32_t mndSplitVgroup(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SVgObj *pVgroup);

1096
source/dnode/mnode/impl/src/mndArbGroup.c Normal file
View File

File diff suppressed because it is too large Load Diff

204
source/dnode/mnode/impl/src/mndDb.c
View File

@ -15,6 +15,8 @@
#define _DEFAULT_SOURCE
#include "mndDb.h"
#include "audit.h"
#include "mndArbGroup.h"
#include "mndCluster.h"
#include "mndDnode.h"
#include "mndIndex.h"
@ -30,12 +32,11 @@
#include "mndVgroup.h"
#include "mndView.h"
#include "systable.h"
#include "tjson.h"
#include "thttp.h"
#include "audit.h"
#include "tjson.h"
#define DB_VER_NUMBER 1
#define DB_RESERVE_SIZE 42
#define DB_RESERVE_SIZE 41
static SSdbRow *mndDbActionDecode(SSdbRaw *pRaw);
static int32_t mndDbActionInsert(SSdb *pSdb, SDbObj *pDb);
@ -43,14 +44,14 @@ static int32_t mndDbActionDelete(SSdb *pSdb, SDbObj *pDb);
static int32_t mndDbActionUpdate(SSdb *pSdb, SDbObj *pOld, SDbObj *pNew);
static int32_t mndNewDbActionValidate(SMnode *pMnode, STrans *pTrans, SSdbRaw *pRaw);
static int32_t mndProcessCreateDbReq(SRpcMsg *pReq);
static int32_t mndProcessAlterDbReq(SRpcMsg *pReq);
static int32_t mndProcessDropDbReq(SRpcMsg *pReq);
static int32_t mndProcessUseDbReq(SRpcMsg *pReq);
static int32_t mndProcessTrimDbReq(SRpcMsg *pReq);
static int32_t mndRetrieveDbs(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rowsCapacity);
static void mndCancelGetNextDb(SMnode *pMnode, void *pIter);
static int32_t mndProcessGetDbCfgReq(SRpcMsg *pReq);
static int32_t mndProcessCreateDbReq(SRpcMsg *pReq);
static int32_t mndProcessAlterDbReq(SRpcMsg *pReq);
static int32_t mndProcessDropDbReq(SRpcMsg *pReq);
static int32_t mndProcessUseDbReq(SRpcMsg *pReq);
static int32_t mndProcessTrimDbReq(SRpcMsg *pReq);
static int32_t mndRetrieveDbs(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rowsCapacity);
static void mndCancelGetNextDb(SMnode *pMnode, void *pIter);
static int32_t mndProcessGetDbCfgReq(SRpcMsg *pReq);
#ifndef TD_ENTERPRISE
int32_t mndProcessCompactDbReq(SRpcMsg *pReq) { return TSDB_CODE_OPS_NOT_SUPPORT; }
@ -139,6 +140,7 @@ SSdbRaw *mndDbActionEncode(SDbObj *pDb) {
SDB_SET_INT32(pRaw, dataPos, pDb->cfg.tsdbPageSize, _OVER)
SDB_SET_INT64(pRaw, dataPos, pDb->compactStartTime, _OVER)
SDB_SET_INT32(pRaw, dataPos, pDb->cfg.keepTimeOffset, _OVER)
SDB_SET_INT8(pRaw, dataPos, pDb->cfg.withArbitrator, _OVER)
SDB_SET_RESERVE(pRaw, dataPos, DB_RESERVE_SIZE, _OVER)
SDB_SET_DATALEN(pRaw, dataPos, _OVER)
@ -230,6 +232,7 @@ static SSdbRow *mndDbActionDecode(SSdbRaw *pRaw) {
SDB_GET_INT32(pRaw, dataPos, &pDb->cfg.tsdbPageSize, _OVER)
SDB_GET_INT64(pRaw, dataPos, &pDb->compactStartTime, _OVER)
SDB_GET_INT32(pRaw, dataPos, &pDb->cfg.keepTimeOffset, _OVER)
SDB_GET_INT8(pRaw, dataPos, &pDb->cfg.withArbitrator, _OVER)
SDB_GET_RESERVE(pRaw, dataPos, DB_RESERVE_SIZE, _OVER)
taosInitRWLatch(&pDb->lock);
@ -319,6 +322,7 @@ static int32_t mndDbActionUpdate(SSdb *pSdb, SDbObj *pOld, SDbObj *pNew) {
pOld->cfg.minRows = pNew->cfg.minRows;
pOld->cfg.maxRows = pNew->cfg.maxRows;
pOld->cfg.tsdbPageSize = pNew->cfg.tsdbPageSize;
pOld->cfg.withArbitrator = pNew->cfg.withArbitrator;
pOld->compactStartTime = pNew->compactStartTime;
taosWUnLockLatch(&pOld->lock);
return 0;
@ -393,7 +397,11 @@ static int32_t mndCheckDbCfg(SMnode *pMnode, SDbCfg *pCfg) {
if (pCfg->precision < TSDB_MIN_PRECISION && pCfg->precision > TSDB_MAX_PRECISION) return -1;
if (pCfg->compression < TSDB_MIN_COMP_LEVEL || pCfg->compression > TSDB_MAX_COMP_LEVEL) return -1;
if (pCfg->replications < TSDB_MIN_DB_REPLICA || pCfg->replications > TSDB_MAX_DB_REPLICA) return -1;
#ifdef TD_ENTERPRISE
if ((pCfg->replications == 2) ^ (pCfg->withArbitrator == TSDB_MAX_DB_WITH_ARBITRATOR)) return -1;
#else
if (pCfg->replications != 1 && pCfg->replications != 3) return -1;
#endif
if (pCfg->strict < TSDB_DB_STRICT_OFF || pCfg->strict > TSDB_DB_STRICT_ON) return -1;
if (pCfg->schemaless < TSDB_DB_SCHEMALESS_OFF || pCfg->schemaless > TSDB_DB_SCHEMALESS_ON) return -1;
if (pCfg->cacheLast < TSDB_CACHE_MODEL_NONE || pCfg->cacheLast > TSDB_CACHE_MODEL_BOTH) return -1;
@ -418,33 +426,48 @@ static int32_t mndCheckDbCfg(SMnode *pMnode, SDbCfg *pCfg) {
return terrno;
}
static int32_t mndCheckInChangeDbCfg(SMnode *pMnode, SDbCfg *pCfg) {
static int32_t mndCheckInChangeDbCfg(SMnode *pMnode, SDbCfg *pOldCfg, SDbCfg *pNewCfg) {
terrno = TSDB_CODE_MND_INVALID_DB_OPTION;
if (pCfg->buffer < TSDB_MIN_BUFFER_PER_VNODE || pCfg->buffer > TSDB_MAX_BUFFER_PER_VNODE) return -1;
if (pCfg->pages < TSDB_MIN_PAGES_PER_VNODE || pCfg->pages > TSDB_MAX_PAGES_PER_VNODE) return -1;
if (pCfg->pageSize < TSDB_MIN_PAGESIZE_PER_VNODE || pCfg->pageSize > TSDB_MAX_PAGESIZE_PER_VNODE) return -1;
if (pCfg->daysPerFile < TSDB_MIN_DAYS_PER_FILE || pCfg->daysPerFile > TSDB_MAX_DAYS_PER_FILE) return -1;
if (pCfg->daysToKeep0 < TSDB_MIN_KEEP || pCfg->daysToKeep0 > TSDB_MAX_KEEP) return -1;
if (pCfg->daysToKeep1 < TSDB_MIN_KEEP || pCfg->daysToKeep1 > TSDB_MAX_KEEP) return -1;
if (pCfg->daysToKeep2 < TSDB_MIN_KEEP || pCfg->daysToKeep2 > TSDB_MAX_KEEP) return -1;
if (pCfg->daysToKeep0 < pCfg->daysPerFile) return -1;
if (pCfg->daysToKeep0 > pCfg->daysToKeep1) return -1;
if (pCfg->daysToKeep1 > pCfg->daysToKeep2) return -1;
if (pCfg->keepTimeOffset < TSDB_MIN_KEEP_TIME_OFFSET || pCfg->keepTimeOffset > TSDB_MAX_KEEP_TIME_OFFSET) return -1;
if (pCfg->walFsyncPeriod < TSDB_MIN_FSYNC_PERIOD || pCfg->walFsyncPeriod > TSDB_MAX_FSYNC_PERIOD) return -1;
if (pCfg->walLevel < TSDB_MIN_WAL_LEVEL || pCfg->walLevel > TSDB_MAX_WAL_LEVEL) return -1;
if (pCfg->cacheLast < TSDB_CACHE_MODEL_NONE || pCfg->cacheLast > TSDB_CACHE_MODEL_BOTH) return -1;
if (pCfg->cacheLastSize < TSDB_MIN_DB_CACHE_SIZE || pCfg->cacheLastSize > TSDB_MAX_DB_CACHE_SIZE) return -1;
if (pCfg->replications < TSDB_MIN_DB_REPLICA || pCfg->replications > TSDB_MAX_DB_REPLICA) return -1;
if (pCfg->replications != 1 && pCfg->replications != 3) return -1;
if (pCfg->sstTrigger < TSDB_MIN_STT_TRIGGER || pCfg->sstTrigger > TSDB_MAX_STT_TRIGGER) return -1;
if (pCfg->minRows < TSDB_MIN_MINROWS_FBLOCK || pCfg->minRows > TSDB_MAX_MINROWS_FBLOCK) return -1;
if (pCfg->maxRows < TSDB_MIN_MAXROWS_FBLOCK || pCfg->maxRows > TSDB_MAX_MAXROWS_FBLOCK) return -1;
if (pCfg->minRows > pCfg->maxRows) return -1;
if (pCfg->walRetentionPeriod < TSDB_DB_MIN_WAL_RETENTION_PERIOD) return -1;
if (pCfg->walRetentionSize < TSDB_DB_MIN_WAL_RETENTION_SIZE) return -1;
if (pCfg->strict < TSDB_DB_STRICT_OFF || pCfg->strict > TSDB_DB_STRICT_ON) return -1;
if (pCfg->replications > mndGetDnodeSize(pMnode)) {
if (pNewCfg->buffer < TSDB_MIN_BUFFER_PER_VNODE || pNewCfg->buffer > TSDB_MAX_BUFFER_PER_VNODE) return -1;
if (pNewCfg->pages < TSDB_MIN_PAGES_PER_VNODE || pNewCfg->pages > TSDB_MAX_PAGES_PER_VNODE) return -1;
if (pNewCfg->pageSize < TSDB_MIN_PAGESIZE_PER_VNODE || pNewCfg->pageSize > TSDB_MAX_PAGESIZE_PER_VNODE) return -1;
if (pNewCfg->daysPerFile < TSDB_MIN_DAYS_PER_FILE || pNewCfg->daysPerFile > TSDB_MAX_DAYS_PER_FILE) return -1;
if (pNewCfg->daysToKeep0 < TSDB_MIN_KEEP || pNewCfg->daysToKeep0 > TSDB_MAX_KEEP) return -1;
if (pNewCfg->daysToKeep1 < TSDB_MIN_KEEP || pNewCfg->daysToKeep1 > TSDB_MAX_KEEP) return -1;
if (pNewCfg->daysToKeep2 < TSDB_MIN_KEEP || pNewCfg->daysToKeep2 > TSDB_MAX_KEEP) return -1;
if (pNewCfg->daysToKeep0 < pNewCfg->daysPerFile) return -1;
if (pNewCfg->daysToKeep0 > pNewCfg->daysToKeep1) return -1;
if (pNewCfg->daysToKeep1 > pNewCfg->daysToKeep2) return -1;
if (pNewCfg->keepTimeOffset < TSDB_MIN_KEEP_TIME_OFFSET || pNewCfg->keepTimeOffset > TSDB_MAX_KEEP_TIME_OFFSET)
return -1;
if (pNewCfg->walFsyncPeriod < TSDB_MIN_FSYNC_PERIOD || pNewCfg->walFsyncPeriod > TSDB_MAX_FSYNC_PERIOD) return -1;
if (pNewCfg->walLevel < TSDB_MIN_WAL_LEVEL || pNewCfg->walLevel > TSDB_MAX_WAL_LEVEL) return -1;
if (pNewCfg->cacheLast < TSDB_CACHE_MODEL_NONE || pNewCfg->cacheLast > TSDB_CACHE_MODEL_BOTH) return -1;
if (pNewCfg->cacheLastSize < TSDB_MIN_DB_CACHE_SIZE || pNewCfg->cacheLastSize > TSDB_MAX_DB_CACHE_SIZE) return -1;
if (pNewCfg->replications < TSDB_MIN_DB_REPLICA || pNewCfg->replications > TSDB_MAX_DB_REPLICA) return -1;
#ifdef TD_ENTERPRISE
if ((pNewCfg->replications == 2) ^ (pNewCfg->withArbitrator == TSDB_MAX_DB_WITH_ARBITRATOR)) return -1;
if (pNewCfg->replications == 2 && pNewCfg->withArbitrator == TSDB_MAX_DB_WITH_ARBITRATOR) {
if (pOldCfg->replications != 1 && pOldCfg->replications != 2) {
terrno = TSDB_CODE_OPS_NOT_SUPPORT;
return -1;
}
}
if (pNewCfg->replications != 2 && pOldCfg->replications == 2) {
terrno = TSDB_CODE_OPS_NOT_SUPPORT;
return -1;
}
#else
if (pNewCfg->replications != 1 && pNewCfg->replications != 3) return -1;
#endif
if (pNewCfg->sstTrigger < TSDB_MIN_STT_TRIGGER || pNewCfg->sstTrigger > TSDB_MAX_STT_TRIGGER) return -1;
if (pNewCfg->minRows < TSDB_MIN_MINROWS_FBLOCK || pNewCfg->minRows > TSDB_MAX_MINROWS_FBLOCK) return -1;
if (pNewCfg->maxRows < TSDB_MIN_MAXROWS_FBLOCK || pNewCfg->maxRows > TSDB_MAX_MAXROWS_FBLOCK) return -1;
if (pNewCfg->minRows > pNewCfg->maxRows) return -1;
if (pNewCfg->walRetentionPeriod < TSDB_DB_MIN_WAL_RETENTION_PERIOD) return -1;
if (pNewCfg->walRetentionSize < TSDB_DB_MIN_WAL_RETENTION_SIZE) return -1;
if (pNewCfg->strict < TSDB_DB_STRICT_OFF || pNewCfg->strict > TSDB_DB_STRICT_ON) return -1;
if (pNewCfg->replications > mndGetDnodeSize(pMnode)) {
terrno = TSDB_CODE_MND_NO_ENOUGH_DNODES;
return -1;
}
@ -484,6 +507,7 @@ static void mndSetDefaultDbCfg(SDbCfg *pCfg) {
if (pCfg->walSegmentSize < 0) pCfg->walSegmentSize = TSDB_DEFAULT_DB_WAL_SEGMENT_SIZE;
if (pCfg->sstTrigger <= 0) pCfg->sstTrigger = TSDB_DEFAULT_SST_TRIGGER;
if (pCfg->tsdbPageSize <= 0) pCfg->tsdbPageSize = TSDB_DEFAULT_TSDB_PAGESIZE;
if (pCfg->withArbitrator < 0) pCfg->withArbitrator = TSDB_DEFAULT_DB_WITH_ARBITRATOR;
}
static int32_t mndSetCreateDbPrepareAction(SMnode *pMnode, STrans *pTrans, SDbObj *pDb) {
@ -515,6 +539,15 @@ static int32_t mndSetCreateDbRedoLogs(SMnode *pMnode, STrans *pTrans, SDbObj *pD
if (sdbSetRawStatus(pVgRaw, SDB_STATUS_UPDATE) != 0) return -1;
}
if (pDb->cfg.withArbitrator) {
for (int32_t v = 0; v < pDb->cfg.numOfVgroups; ++v) {
SVgObj *pVgObj = pVgroups + v;
SArbGroup arbGroup = {0};
mndArbGroupInitFromVgObj(pVgObj, &arbGroup);
if (mndSetCreateArbGroupRedoLogs(pTrans, &arbGroup) != 0) return -1;
}
}
return 0;
}
@ -531,6 +564,15 @@ static int32_t mndSetCreateDbUndoLogs(SMnode *pMnode, STrans *pTrans, SDbObj *pD
if (sdbSetRawStatus(pVgRaw, SDB_STATUS_DROPPED) != 0) return -1;
}
if (pDb->cfg.withArbitrator) {
for (int32_t v = 0; v < pDb->cfg.numOfVgroups; ++v) {
SVgObj *pVgObj = pVgroups + v;
SArbGroup arbGroup = {0};
mndArbGroupInitFromVgObj(pVgObj, &arbGroup);
if (mndSetCreateArbGroupUndoLogs(pTrans, &arbGroup) != 0) return -1;
}
}
return 0;
}
@ -548,6 +590,15 @@ static int32_t mndSetCreateDbCommitLogs(SMnode *pMnode, STrans *pTrans, SDbObj *
if (sdbSetRawStatus(pVgRaw, SDB_STATUS_READY) != 0) return -1;
}
if (pDb->cfg.withArbitrator) {
for (int32_t v = 0; v < pDb->cfg.numOfVgroups; ++v) {
SVgObj *pVgObj = pVgroups + v;
SArbGroup arbGroup = {0};
mndArbGroupInitFromVgObj(pVgObj, &arbGroup);
if (mndSetCreateArbGroupCommitLogs(pTrans, &arbGroup) != 0) return -1;
}
}
if (pUserDuped) {
SSdbRaw *pUserRaw = mndUserActionEncode(pUserDuped);
if (pUserRaw == NULL) return -1;
@ -629,6 +680,7 @@ static int32_t mndCreateDb(SMnode *pMnode, SRpcMsg *pReq, SCreateDbReq *pCreate,
.hashPrefix = pCreate->hashPrefix,
.hashSuffix = pCreate->hashSuffix,
.tsdbPageSize = pCreate->tsdbPageSize,
.withArbitrator = pCreate->withArbitrator,
};
dbObj.cfg.numOfRetensions = pCreate->numOfRetensions;
@ -698,17 +750,17 @@ _OVER:
return code;
}
static void mndBuildAuditDetailInt32(char* detail, char* tmp, char* format, int32_t para){
if(para > 0){
if(strlen(detail) > 0) strcat(detail, ", ");
static void mndBuildAuditDetailInt32(char *detail, char *tmp, char *format, int32_t para) {
if (para > 0) {
if (strlen(detail) > 0) strcat(detail, ", ");
sprintf(tmp, format, para);
strcat(detail, tmp);
}
}
static void mndBuildAuditDetailInt64(char* detail, char* tmp, char* format, int64_t para){
if(para > 0){
if(strlen(detail) > 0) strcat(detail, ", ");
static void mndBuildAuditDetailInt64(char *detail, char *tmp, char *format, int64_t para) {
if (para > 0) {
if (strlen(detail) > 0) strcat(detail, ", ");
sprintf(tmp, format, para);
strcat(detail, tmp);
}
@ -893,6 +945,12 @@ static int32_t mndSetDbCfgFromAlterDbReq(SDbObj *pDb, SAlterDbReq *pAlter) {
terrno = 0;
}
if (pAlter->withArbitrator >= TSDB_MIN_DB_WITH_ARBITRATOR && pAlter->withArbitrator != pDb->cfg.withArbitrator) {
pDb->cfg.withArbitrator = pAlter->withArbitrator;
pDb->vgVersion++;
terrno = 0;
}
return terrno;
}
@ -931,12 +989,24 @@ static int32_t mndSetAlterDbRedoActions(SMnode *pMnode, STrans *pTrans, SDbObj *
if (pIter == NULL) break;
if (mndVgroupInDb(pVgroup, pNewDb->uid)) {
if (mndBuildAlterVgroupAction(pMnode, pTrans, pOldDb, pNewDb, pVgroup, pArray) != 0) {
SVgObj newVgroup = {0};
if (mndBuildAlterVgroupAction(pMnode, pTrans, pOldDb, pNewDb, pVgroup, pArray, &newVgroup) != 0) {
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pVgroup);
taosArrayDestroy(pArray);
return -1;
}
if (pNewDb->cfg.withArbitrator != pOldDb->cfg.withArbitrator) {
if (pNewDb->cfg.withArbitrator) {
SArbGroup arbGroup = {0};
mndArbGroupInitFromVgObj(&newVgroup, &arbGroup);
if (mndSetCreateArbGroupCommitLogs(pTrans, &arbGroup) != 0) return -1;
} else {
SArbGroup arbGroup = {0};
mndArbGroupInitFromVgObj(pVgroup, &arbGroup);
if (mndSetDropArbGroupCommitLogs(pTrans, &arbGroup) != 0) return -1;
}
}
}
sdbRelease(pSdb, pVgroup);
@ -1013,7 +1083,7 @@ static int32_t mndProcessAlterDbReq(SRpcMsg *pReq) {
goto _OVER;
}
code = mndCheckInChangeDbCfg(pMnode, &dbObj.cfg);
code = mndCheckInChangeDbCfg(pMnode, &pDb->cfg, &dbObj.cfg);
if (code != 0) goto _OVER;
dbObj.cfgVersion++;
@ -1081,6 +1151,7 @@ static void mndDumpDbCfgInfo(SDbCfgRsp *cfgRsp, SDbObj *pDb) {
cfgRsp->pRetensions = taosArrayDup(pDb->cfg.pRetensions, NULL);
cfgRsp->schemaless = pDb->cfg.schemaless;
cfgRsp->sstTrigger = pDb->cfg.sstTrigger;
cfgRsp->withArbitrator = pDb->cfg.withArbitrator;
}
static int32_t mndProcessGetDbCfgReq(SRpcMsg *pReq) {
@ -1150,6 +1221,22 @@ static int32_t mndSetDropDbCommitLogs(SMnode *pMnode, STrans *pTrans, SDbObj *pD
SSdb *pSdb = pMnode->pSdb;
void *pIter = NULL;
while (1) {
SArbGroup *pArbGroup = NULL;
pIter = sdbFetch(pSdb, SDB_ARBGROUP, pIter, (void **)&pArbGroup);
if (pIter == NULL) break;
if (pArbGroup->dbUid == pDb->uid) {
if (mndSetDropArbGroupCommitLogs(pTrans,pArbGroup) != 0) {
sdbCancelFetch(pSdb, pIter);
sdbRelease(pSdb, pArbGroup);
return -1;
}
}
sdbRelease(pSdb, pArbGroup);
}
while (1) {
SVgObj *pVgroup = NULL;
pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void **)&pVgroup);
@ -1277,7 +1364,7 @@ static int32_t mndDropDb(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb) {
if (mndDropStreamByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
#ifdef TD_ENTERPRISE
if (mndDropViewByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
#endif
#endif
if (mndDropSmasByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
if (mndDropIdxsByDb(pMnode, pTrans, pDb) != 0) goto _OVER;
if (mndSetDropDbRedoActions(pMnode, pTrans, pDb) != 0) goto _OVER;
@ -1390,7 +1477,7 @@ static void mndBuildDBVgroupInfo(SDbObj *pDb, SMnode *pMnode, SArray *pVgList) {
pEp->port = pDnode->port;
}
mndReleaseDnode(pMnode, pDnode);
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER) {
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER || pVgid->syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
vgInfo.epSet.inUse = gid;
}
}
@ -1562,21 +1649,19 @@ int32_t mndValidateDbInfo(SMnode *pMnode, SDbCacheInfo *pDbs, int32_t numOfDbs,
int32_t numOfTable = mndGetDBTableNum(pDb, pMnode);
if (pDbCacheInfo->vgVersion >= pDb->vgVersion &&
pDbCacheInfo->cfgVersion >= pDb->cfgVersion &&
numOfTable == pDbCacheInfo->numOfTable &&
pDbCacheInfo->stateTs == pDb->stateTs) {
if (pDbCacheInfo->vgVersion >= pDb->vgVersion && pDbCacheInfo->cfgVersion >= pDb->cfgVersion &&
numOfTable == pDbCacheInfo->numOfTable && pDbCacheInfo->stateTs == pDb->stateTs) {
mTrace("db:%s, valid dbinfo, vgVersion:%d cfgVersion:%d stateTs:%" PRId64
" numOfTables:%d, not changed vgVersion:%d cfgVersion:%d stateTs:%" PRId64 " numOfTables:%d",
pDbCacheInfo->dbFName, pDbCacheInfo->vgVersion, pDbCacheInfo->cfgVersion, pDbCacheInfo->stateTs, pDbCacheInfo->numOfTable,
pDb->vgVersion, pDb->cfgVersion, pDb->stateTs, numOfTable);
pDbCacheInfo->dbFName, pDbCacheInfo->vgVersion, pDbCacheInfo->cfgVersion, pDbCacheInfo->stateTs,
pDbCacheInfo->numOfTable, pDb->vgVersion, pDb->cfgVersion, pDb->stateTs, numOfTable);
mndReleaseDb(pMnode, pDb);
continue;
} else {
mInfo("db:%s, valid dbinfo, vgVersion:%d cfgVersion:%d stateTs:%" PRId64
" numOfTables:%d, changed to vgVersion:%d cfgVersion:%d stateTs:%" PRId64 " numOfTables:%d",
pDbCacheInfo->dbFName, pDbCacheInfo->vgVersion, pDbCacheInfo->cfgVersion, pDbCacheInfo->stateTs, pDbCacheInfo->numOfTable,
pDb->vgVersion, pDb->cfgVersion, pDb->stateTs, numOfTable);
pDbCacheInfo->dbFName, pDbCacheInfo->vgVersion, pDbCacheInfo->cfgVersion, pDbCacheInfo->stateTs,
pDbCacheInfo->numOfTable, pDb->vgVersion, pDb->cfgVersion, pDb->stateTs, numOfTable);
}
if (pDbCacheInfo->cfgVersion < pDb->cfgVersion) {
@ -1584,8 +1669,7 @@ int32_t mndValidateDbInfo(SMnode *pMnode, SDbCacheInfo *pDbs, int32_t numOfDbs,
mndDumpDbCfgInfo(rsp.cfgRsp, pDb);
}
if (pDbCacheInfo->vgVersion < pDb->vgVersion ||
numOfTable != pDbCacheInfo->numOfTable ||
if (pDbCacheInfo->vgVersion < pDb->vgVersion || numOfTable != pDbCacheInfo->numOfTable ||
pDbCacheInfo->stateTs != pDb->stateTs) {
rsp.useDbRsp = taosMemoryCalloc(1, sizeof(SUseDbRsp));
rsp.useDbRsp->pVgroupInfos = taosArrayInit(pDb->cfg.numOfVgroups, sizeof(SVgroupInfo));
@ -1801,7 +1885,8 @@ bool mndIsDbReady(SMnode *pMnode, SDbObj *pDb) {
if (pVgroup->dbUid == pDb->uid && pVgroup->replica > 1) {
bool hasLeader = false;
for (int32_t i = 0; i < pVgroup->replica; ++i) {
if (pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_LEADER) {
if (pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_LEADER ||
pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
hasLeader = true;
}
}
@ -1991,6 +2076,9 @@ static void mndDumpDbInfoData(SMnode *pMnode, SSDataBlock *pBlock, SDbObj *pDb,
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataSetVal(pColInfo, rows, (const char *)&pDb->cfg.keepTimeOffset, false);
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
colDataSetVal(pColInfo, rows, (const char *)&pDb->cfg.withArbitrator, false);
}
taosMemoryFree(buf);

66
source/dnode/mnode/impl/src/mndDnode.c
View File

@ -33,7 +33,7 @@
#include "taos_monitor.h"
#define TSDB_DNODE_VER_NUMBER 2
#define TSDB_DNODE_RESERVE_SIZE 64
#define TSDB_DNODE_RESERVE_SIZE 40
static const char *offlineReason[] = {
"",
@ -183,11 +183,10 @@ static SSdbRaw *mndDnodeActionEncode(SDnodeObj *pDnode) {
SDB_SET_INT64(pRaw, dataPos, pDnode->updateTime, _OVER)
SDB_SET_INT16(pRaw, dataPos, pDnode->port, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pDnode->fqdn, TSDB_FQDN_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pDnode->machineId, TSDB_MACHINE_ID_LEN, _OVER)
SDB_SET_RESERVE(pRaw, dataPos, TSDB_DNODE_RESERVE_SIZE, _OVER)
SDB_SET_INT16(pRaw, dataPos, TSDB_ACTIVE_KEY_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pDnode->active, TSDB_ACTIVE_KEY_LEN, _OVER)
SDB_SET_INT16(pRaw, dataPos, TSDB_CONN_ACTIVE_KEY_LEN, _OVER)
SDB_SET_BINARY(pRaw, dataPos, pDnode->connActive, TSDB_CONN_ACTIVE_KEY_LEN, _OVER)
SDB_SET_INT16(pRaw, dataPos, 0, _OVER) // forward/backward compatible
SDB_SET_INT16(pRaw, dataPos, 0, _OVER) // forward/backward compatible
SDB_SET_DATALEN(pRaw, dataPos, _OVER);
terrno = 0;
@ -227,13 +226,14 @@ static SSdbRow *mndDnodeActionDecode(SSdbRaw *pRaw) {
SDB_GET_INT64(pRaw, dataPos, &pDnode->updateTime, _OVER)
SDB_GET_INT16(pRaw, dataPos, &pDnode->port, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pDnode->fqdn, TSDB_FQDN_LEN, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pDnode->machineId, TSDB_MACHINE_ID_LEN, _OVER)
SDB_GET_RESERVE(pRaw, dataPos, TSDB_DNODE_RESERVE_SIZE, _OVER)
if (sver > 1) {
int16_t keyLen = 0;
SDB_GET_INT16(pRaw, dataPos, &keyLen, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pDnode->active, keyLen, _OVER)
SDB_GET_BINARY(pRaw, dataPos, NULL, keyLen, _OVER)
SDB_GET_INT16(pRaw, dataPos, &keyLen, _OVER)
SDB_GET_BINARY(pRaw, dataPos, pDnode->connActive, keyLen, _OVER)
SDB_GET_BINARY(pRaw, dataPos, NULL, keyLen, _OVER)
}
terrno = 0;
@ -271,12 +271,7 @@ static int32_t mndDnodeActionUpdate(SSdb *pSdb, SDnodeObj *pOld, SDnodeObj *pNew
mTrace("dnode:%d, perform update action, old row:%p new row:%p", pOld->id, pOld, pNew);
pOld->updateTime = pNew->updateTime;
#ifdef TD_ENTERPRISE
if (strncmp(pOld->active, pNew->active, TSDB_ACTIVE_KEY_LEN) != 0) {
strncpy(pOld->active, pNew->active, TSDB_ACTIVE_KEY_LEN);
}
if (strncmp(pOld->connActive, pNew->connActive, TSDB_CONN_ACTIVE_KEY_LEN) != 0) {
strncpy(pOld->connActive, pNew->connActive, TSDB_CONN_ACTIVE_KEY_LEN);
}
tstrncpy(pOld->machineId, pNew->machineId, TSDB_MACHINE_ID_LEN + 1);
#endif
return 0;
}
@ -311,6 +306,17 @@ SEpSet mndGetDnodeEpset(SDnodeObj *pDnode) {
return epSet;
}
SEpSet mndGetDnodeEpsetById(SMnode *pMnode, int32_t dnodeId) {
SEpSet epSet = {0};
SDnodeObj *pDnode = mndAcquireDnode(pMnode, dnodeId);
if (!pDnode) return epSet;
epSet = mndGetDnodeEpset(pDnode);
mndReleaseDnode(pMnode, pDnode);
return epSet;
}
static SDnodeObj *mndAcquireDnodeByEp(SMnode *pMnode, char *pEpStr) {
SSdb *pSdb = pMnode->pSdb;
@ -705,6 +711,35 @@ _OVER:
*/
}
static int32_t mndUpdateDnodeObj(SMnode *pMnode, SDnodeObj *pDnode) {
int32_t code = 0;
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_ROLLBACK, TRN_CONFLICT_NOTHING, NULL, "update-dnode-obj");
if (pTrans == NULL) {
code = terrno;
goto _exit;
}
pDnode->updateTime = taosGetTimestampMs();
SSdbRaw *pCommitRaw = mndDnodeActionEncode(pDnode);
if (pCommitRaw == NULL || mndTransAppendCommitlog(pTrans, pCommitRaw) != 0) {
mError("trans:%d, failed to append commit log since %s", pTrans->id, terrstr());
code = terrno;
goto _exit;
}
(void)sdbSetRawStatus(pCommitRaw, SDB_STATUS_READY);
if (mndTransPrepare(pMnode, pTrans) != 0) {
mError("trans:%d, failed to prepare since %s", pTrans->id, terrstr());
code = terrno;
goto _exit;
}
_exit:
mndTransDrop(pTrans);
return code;
}
static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
SStatusReq statusReq = {0};
@ -776,7 +811,7 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
SVgObj *pVgroup = mndAcquireVgroup(pMnode, pVload->vgId);
if (pVgroup != NULL) {
if (pVload->syncState == TAOS_SYNC_STATE_LEADER) {
if (pVload->syncState == TAOS_SYNC_STATE_LEADER || pVload->syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
pVgroup->cacheUsage = pVload->cacheUsage;
pVgroup->numOfCachedTables = pVload->numOfCachedTables;
pVgroup->numOfTables = pVload->numOfTables;
@ -874,8 +909,9 @@ static int32_t mndProcessStatusReq(SRpcMsg *pReq) {
pDnode->numOfDiskCfg = statusReq.numOfDiskCfg;
pDnode->memAvail = statusReq.memAvail;
pDnode->memTotal = statusReq.memTotal;
if (pDnode->machineId[0] == 0 && statusReq.machineId[0] != 0) {
if (memcmp(pDnode->machineId, statusReq.machineId, TSDB_MACHINE_ID_LEN) != 0) {
tstrncpy(pDnode->machineId, statusReq.machineId, TSDB_MACHINE_ID_LEN + 1);
mndUpdateDnodeObj(pMnode, pDnode);
}
SStatusRsp statusRsp = {0};

42
source/dnode/mnode/impl/src/mndMain.c
View File

@ -15,6 +15,7 @@
#define _DEFAULT_SOURCE
#include "mndAcct.h"
#include "mndArbGroup.h"
#include "mndCluster.h"
#include "mndCompact.h"
#include "mndCompactDetail.h"
@ -140,6 +141,22 @@ static void mndPullupTrimDb(SMnode *pMnode) {
tmsgPutToQueue(&pMnode->msgCb, WRITE_QUEUE, &rpcMsg);
}
static int32_t mndPullupArbHeartbeat(SMnode *pMnode) {
mTrace("pullup arb hb");
int32_t contLen = 0;
void *pReq = mndBuildTimerMsg(&contLen);
SRpcMsg rpcMsg = {.msgType = TDMT_MND_ARB_HEARTBEAT_TIMER, .pCont = pReq, .contLen = contLen, .info.noResp = 1};
return tmsgPutToQueue(&pMnode->msgCb, ARB_QUEUE, &rpcMsg);
}
static int32_t mndPullupArbCheckSync(SMnode *pMnode) {
mTrace("pullup arb sync");
int32_t contLen = 0;
void *pReq = mndBuildTimerMsg(&contLen);
SRpcMsg rpcMsg = {.msgType = TDMT_MND_ARB_CHECK_SYNC_TIMER, .pCont = pReq, .contLen = contLen, .info.noResp = 1};
return tmsgPutToQueue(&pMnode->msgCb, ARB_QUEUE, &rpcMsg);
}
static void mndCalMqRebalance(SMnode *pMnode) {
int32_t contLen = 0;
void *pReq = mndBuildTimerMsg(&contLen);
@ -391,6 +408,18 @@ static void *mndThreadFp(void *param) {
continue;
}
mndDoTimerPullupTask(pMnode, sec);
if (sec % (tsArbHeartBeatIntervalSec) == 0) {
if (mndPullupArbHeartbeat(pMnode) != 0) {
mError("failed to pullup arb heartbeat, since:%s", terrstr());
}
}
if (sec % (tsArbCheckSyncIntervalSec) == 0) {
if (mndPullupArbCheckSync(pMnode) != 0) {
mError("failed to pullup arb check sync, since:%s", terrstr());
}
}
}
return NULL;
@ -513,6 +542,7 @@ static int32_t mndInitSteps(SMnode *pMnode) {
if (mndAllocStep(pMnode, "mnode-mnode", mndInitMnode, mndCleanupMnode) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-qnode", mndInitQnode, mndCleanupQnode) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-snode", mndInitSnode, mndCleanupSnode) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-arbgroup", mndInitArbGroup, mndCleanupArbGroup) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-dnode", mndInitDnode, mndCleanupDnode) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-user", mndInitUser, mndCleanupUser) != 0) return -1;
if (mndAllocStep(pMnode, "mnode-grant", mndInitGrant, mndCleanupGrant) != 0) return -1;
@ -694,6 +724,13 @@ ESyncRole mndGetRole(SMnode *pMnode) {
return syncGetRole(rid);
}
int64_t mndGetTerm(SMnode *pMnode) {
int64_t rid = pMnode->syncMgmt.sync;
return syncGetTerm(rid);
}
int32_t mndGetArbToken(SMnode *pMnode, char *outToken) { return syncGetArbToken(pMnode->syncMgmt.sync, outToken); }
void mndStop(SMnode *pMnode) {
mndSetStop(pMnode);
mndSyncStop(pMnode);
@ -768,7 +805,8 @@ _OVER:
pMsg->msgType == TDMT_MND_TRIM_DB_TIMER || pMsg->msgType == TDMT_MND_UPTIME_TIMER ||
pMsg->msgType == TDMT_MND_COMPACT_TIMER || pMsg->msgType == TDMT_MND_NODECHECK_TIMER ||
pMsg->msgType == TDMT_MND_GRANT_HB_TIMER || pMsg->msgType == TDMT_MND_STREAM_CHECKPOINT_CANDIDITATE ||
pMsg->msgType == TDMT_MND_STREAM_CHECKPOINT_TIMER || pMsg->msgType == TDMT_MND_STREAM_REQ_CHKPT) {
pMsg->msgType == TDMT_MND_STREAM_CHECKPOINT_TIMER || pMsg->msgType == TDMT_MND_STREAM_REQ_CHKPT ||
pMsg->msgType == TDMT_MND_ARB_HEARTBEAT_TIMER || pMsg->msgType == TDMT_MND_ARB_CHECK_SYNC_TIMER) {
mTrace("timer not process since mnode restored:%d stopped:%d, sync restored:%d role:%s ", pMnode->restored,
pMnode->stopped, state.restored, syncStr(state.state));
return -1;
@ -951,7 +989,7 @@ int32_t mndGetMonitorInfo(SMnode *pMnode, SMonClusterInfo *pClusterInfo, SMonVgr
pVnDesc->dnode_id = pVgid->dnodeId;
tstrncpy(pVnDesc->vnode_role, syncStr(pVgid->syncState), sizeof(pVnDesc->vnode_role));
pVnDesc->syncState = pVgid->syncState;
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER) {
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER || pVgid->syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
tstrncpy(desc.status, "ready", sizeof(desc.status));
pClusterInfo->vgroups_alive++;
}

4
source/dnode/mnode/impl/src/mndQnode.c
View File

@ -189,7 +189,7 @@ int32_t mndSetCreateQnodeCommitLogs(STrans *pTrans, SQnodeObj *pObj) {
return 0;
}
bool mndQnodeInDnode(SQnodeObj *pQnode, int32_t dnodeId) {
bool mndQnodeInDnode(SQnodeObj *pQnode, int32_t dnodeId) {
return pQnode->pDnode->id == dnodeId;
}
@ -434,7 +434,7 @@ _OVER:
}
mndReleaseQnode(pMnode, pObj);
tFreeSMCreateQnodeReq(&dropReq);
tFreeSDDropQnodeReq(&dropReq);
return code;
}

2
source/dnode/mnode/impl/src/mndShow.c
View File

@ -67,6 +67,8 @@ static int32_t convertToRetrieveType(char *name, int32_t len) {
type = TSDB_MGMT_TABLE_QNODE;
} else if (strncasecmp(name, TSDB_INS_TABLE_SNODES, len) == 0) {
type = TSDB_MGMT_TABLE_SNODE;
} else if (strncasecmp(name, TSDB_INS_TABLE_ARBGROUPS, len) == 0) {
type = TSDB_MGMT_TABLE_ARBGROUP;
} else if (strncasecmp(name, TSDB_INS_TABLE_CLUSTER, len) == 0) {
type = TSDB_MGMT_TABLE_CLUSTER;
} else if (strncasecmp(name, TSDB_INS_TABLE_DATABASES, len) == 0) {

35
source/dnode/mnode/impl/src/mndStream.c
View File

@ -513,6 +513,7 @@ static int32_t mndCreateStbForStream(SMnode *pMnode, STrans *pTrans, const SStre
createReq.numOfColumns = pStream->outputSchema.nCols;
createReq.numOfTags = 1; // group id
createReq.pColumns = taosArrayInit_s(sizeof(SField), createReq.numOfColumns);
// build fields
for (int32_t i = 0; i < createReq.numOfColumns; i++) {
SField *pField = taosArrayGet(createReq.pColumns, i);
@ -586,12 +587,15 @@ static int32_t mndCreateStbForStream(SMnode *pMnode, STrans *pTrans, const SStre
mndFreeStb(&stbObj);
mndReleaseStb(pMnode, pStb);
mndReleaseDb(pMnode, pDb);
mDebug("stream:%s create dst stable:%s, cols:%d", pStream->name, pStream->targetSTbName, pStream->outputSchema.nCols);
return 0;
_OVER:
tFreeSMCreateStbReq(&createReq);
mndReleaseStb(pMnode, pStb);
mndReleaseDb(pMnode, pDb);
mDebug("stream:%s failed to create dst stable:%s, code:%s", pStream->name, pStream->targetSTbName, tstrerror(terrno));
return -1;
}
@ -647,7 +651,7 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
terrno = TSDB_CODE_MND_INVALID_PLATFORM;
goto _OVER;
#endif
mInfo("stream:%s, start to create, sql:%s", createReq.name, createReq.sql);
mInfo("stream:%s, start to create stream, sql:%s", createReq.name, createReq.sql);
if (mndCheckCreateStreamReq(&createReq) != 0) {
mError("stream:%s, failed to create since %s", createReq.name, terrstr());
@ -684,17 +688,20 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
goto _OVER;
}
STrans *pTrans = doCreateTrans(pMnode, &streamObj, pReq, MND_STREAM_CREATE_NAME, "create stream tasks on dnodes");
STrans *pTrans = doCreateTrans(pMnode, &streamObj, pReq, TRN_CONFLICT_DB, MND_STREAM_CREATE_NAME, "create stream tasks on dnodes");
if (pTrans == NULL) {
goto _OVER;
}
// create stb for stream
if (createReq.createStb == STREAM_CREATE_STABLE_TRUE &&
mndCreateStbForStream(pMnode, pTrans, &streamObj, pReq->info.conn.user) < 0) {
mError("trans:%d, failed to create stb for stream %s since %s", pTrans->id, createReq.name, terrstr());
mndTransDrop(pTrans);
goto _OVER;
if (createReq.createStb == STREAM_CREATE_STABLE_TRUE) {
if (mndCreateStbForStream(pMnode, pTrans, &streamObj, pReq->info.conn.user) < 0) {
mError("trans:%d, failed to create stb for stream %s since %s", pTrans->id, createReq.name, terrstr());
mndTransDrop(pTrans);
goto _OVER;
}
} else {
mDebug("stream:%s no need create stable", createReq.name);
}
// schedule stream task for stream obj
@ -724,7 +731,7 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
// add into buffer firstly
// to make sure when the hb from vnode arrived, the newly created tasks have been in the task map already.
taosThreadMutexLock(&execInfo.lock);
mDebug("stream stream:%s tasks register into node list", createReq.name);
mDebug("stream stream:%s start to register tasks into task_node_list", createReq.name);
saveStreamTasksInfo(&streamObj, &execInfo);
taosThreadMutexUnlock(&execInfo.lock);
@ -890,7 +897,7 @@ static int32_t mndProcessStreamCheckpointTrans(SMnode *pMnode, SStreamObj *pStre
return -1;
}
STrans *pTrans = doCreateTrans(pMnode, pStream, NULL, MND_STREAM_CHECKPOINT_NAME, "gen checkpoint for stream");
STrans *pTrans = doCreateTrans(pMnode, pStream, NULL, TRN_CONFLICT_NOTHING, MND_STREAM_CHECKPOINT_NAME, "gen checkpoint for stream");
if (pTrans == NULL) {
mError("failed to checkpoint of stream name%s, checkpointId: %" PRId64 ", reason:%s", pStream->name, checkpointId,
tstrerror(TSDB_CODE_MND_TRANS_CONFLICT));
@ -1143,7 +1150,7 @@ static int32_t mndProcessDropStreamReq(SRpcMsg *pReq) {
return -1;
}
STrans *pTrans = doCreateTrans(pMnode, pStream, pReq, MND_STREAM_DROP_NAME, "drop stream");
STrans *pTrans = doCreateTrans(pMnode, pStream, pReq, TRN_CONFLICT_NOTHING, MND_STREAM_DROP_NAME, "drop stream");
if (pTrans == NULL) {
mError("stream:%s, failed to drop since %s", dropReq.name, terrstr());
sdbRelease(pMnode->pSdb, pStream);
@ -1573,7 +1580,7 @@ static int32_t mndProcessPauseStreamReq(SRpcMsg *pReq) {
return -1;
}
STrans *pTrans = doCreateTrans(pMnode, pStream, pReq, MND_STREAM_PAUSE_NAME, "pause the stream");
STrans *pTrans = doCreateTrans(pMnode, pStream, pReq, TRN_CONFLICT_NOTHING, MND_STREAM_PAUSE_NAME, "pause the stream");
if (pTrans == NULL) {
mError("stream:%s failed to pause stream since %s", pauseReq.name, terrstr());
sdbRelease(pMnode->pSdb, pStream);
@ -1662,7 +1669,7 @@ static int32_t mndProcessResumeStreamReq(SRpcMsg *pReq) {
return -1;
}
STrans *pTrans = doCreateTrans(pMnode, pStream, pReq, MND_STREAM_RESUME_NAME, "resume the stream");
STrans *pTrans = doCreateTrans(pMnode, pStream, pReq, TRN_CONFLICT_NOTHING, MND_STREAM_RESUME_NAME, "resume the stream");
if (pTrans == NULL) {
mError("stream:%s, failed to resume stream since %s", resumeReq.name, terrstr());
sdbRelease(pMnode->pSdb, pStream);
@ -1794,7 +1801,7 @@ static int32_t mndProcessVgroupChange(SMnode *pMnode, SVgroupChangeInfo *pChange
// here create only one trans
if (pTrans == NULL) {
pTrans = doCreateTrans(pMnode, pStream, NULL, MND_STREAM_TASK_UPDATE_NAME, "update task epsets");
pTrans = doCreateTrans(pMnode, pStream, NULL, TRN_CONFLICT_NOTHING, MND_STREAM_TASK_UPDATE_NAME, "update task epsets");
if (pTrans == NULL) {
sdbRelease(pSdb, pStream);
sdbCancelFetch(pSdb, pIter);

4
source/dnode/mnode/impl/src/mndStreamHb.c
View File

@ -66,7 +66,7 @@ static void addIntoCheckpointList(SArray* pList, const SFailedCheckpointInfo* pI
}
int32_t mndCreateStreamResetStatusTrans(SMnode *pMnode, SStreamObj *pStream) {
STrans *pTrans = doCreateTrans(pMnode, pStream, NULL, MND_STREAM_TASK_RESET_NAME, " reset from failed checkpoint");
STrans *pTrans = doCreateTrans(pMnode, pStream, NULL, TRN_CONFLICT_NOTHING, MND_STREAM_TASK_RESET_NAME, " reset from failed checkpoint");
if (pTrans == NULL) {
return terrno;
}
@ -156,7 +156,7 @@ static int32_t mndDropOrphanTasks(SMnode* pMnode, SArray* pList) {
}
SStreamObj dummyObj = {.uid = pTask->streamId, .sourceDb = "", .targetSTbName = ""};
STrans* pTrans = doCreateTrans(pMnode, &dummyObj, NULL, MND_STREAM_DROP_NAME, "drop stream");
STrans* pTrans = doCreateTrans(pMnode, &dummyObj, NULL, TRN_CONFLICT_NOTHING, MND_STREAM_DROP_NAME, "drop stream");
if (pTrans == NULL) {
mError("failed to create trans to drop orphan tasks since %s", terrstr());
return -1;

4
source/dnode/mnode/impl/src/mndStreamTrans.c
View File

@ -161,8 +161,8 @@ int32_t mndAddtoCheckpointWaitingList(SStreamObj* pStream, int64_t checkpointId)
return TSDB_CODE_SUCCESS;
}
STrans *doCreateTrans(SMnode *pMnode, SStreamObj *pStream, SRpcMsg *pReq, const char *name, const char *pMsg) {
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_NOTHING, pReq, name);
STrans *doCreateTrans(SMnode *pMnode, SStreamObj *pStream, SRpcMsg *pReq, ETrnConflct conflict, const char *name, const char *pMsg) {
STrans *pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, conflict, pReq, name);
if (pTrans == NULL) {
mError("failed to build trans:%s, reason: %s", name, tstrerror(TSDB_CODE_OUT_OF_MEMORY));
terrno = TSDB_CODE_OUT_OF_MEMORY;

1
source/dnode/mnode/impl/src/mndSync.c
View File

@ -409,6 +409,7 @@ SSyncFSM *mndSyncMakeFsm(SMnode *pMnode) {
pFsm->FpApplyQueueItems = mndApplyQueueItems;
pFsm->FpReConfigCb = NULL;
pFsm->FpBecomeLeaderCb = mndBecomeLeader;
pFsm->FpBecomeAssignedLeaderCb = NULL;
pFsm->FpBecomeFollowerCb = mndBecomeFollower;
pFsm->FpBecomeLearnerCb = mndBecomeLearner;
pFsm->FpGetSnapshot = mndSyncGetSnapshot;

14
source/dnode/mnode/impl/src/mndTrans.c
View File

@ -733,6 +733,8 @@ void mndTransSetDbName(STrans *pTrans, const char *dbname, const char *stbname)
}
}
void mndTransSetArbGroupId(STrans *pTrans, int32_t groupId) { pTrans->arbGroupId = groupId; }
void mndTransSetSerial(STrans *pTrans) { pTrans->exec = TRN_EXEC_SERIAL; }
void mndTransSetParallel(STrans *pTrans) { pTrans->exec = TRN_EXEC_PARALLEL; }
@ -814,6 +816,12 @@ static bool mndCheckTransConflict(SMnode *pMnode, STrans *pNew) {
conflict = true;
}
}
if (pNew->conflict == TRN_CONFLICT_ARBGROUP) {
if (pTrans->conflict == TRN_CONFLICT_GLOBAL) conflict = true;
if (pTrans->conflict == TRN_CONFLICT_ARBGROUP) {
if (pNew->arbGroupId == pTrans->arbGroupId) conflict = true;
}
}
if (conflict) {
mError("trans:%d, db:%s stb:%s type:%d, can't execute since conflict with trans:%d db:%s stb:%s type:%d",
@ -1255,9 +1263,9 @@ static int32_t mndTransExecuteActions(SMnode *pMnode, STrans *pTrans, SArray *pA
for (int32_t action = 0; action < numOfActions; ++action) {
STransAction *pAction = taosArrayGet(pArray, action);
mDebug("trans:%d, %s:%d Sent:%d, Received:%d, errCode:0x%x, acceptableCode:0x%x, retryCode:0x%x", pTrans->id,
mndTransStr(pAction->stage), pAction->id, pAction->msgSent, pAction->msgReceived, pAction->errCode,
pAction->acceptableCode, pAction->retryCode);
mDebug("trans:%d, %s:%d Sent:%d, Received:%d, errCode:0x%x, acceptableCode:0x%x, retryCode:0x%x",
pTrans->id, mndTransStr(pAction->stage), pAction->id, pAction->msgSent, pAction->msgReceived,
pAction->errCode, pAction->acceptableCode, pAction->retryCode);
if (pAction->msgSent) {
if (pAction->msgReceived) {
if (pAction->errCode != 0 && pAction->errCode != pAction->acceptableCode) {

152
source/dnode/mnode/impl/src/mndVgroup.c
View File

@ -16,6 +16,7 @@
#define _DEFAULT_SOURCE
#include "mndVgroup.h"
#include "audit.h"
#include "mndArbGroup.h"
#include "mndDb.h"
#include "mndDnode.h"
#include "mndMnode.h"
@ -351,7 +352,7 @@ void *mndBuildCreateVnodeReq(SMnode *pMnode, SDnodeObj *pDnode, SDbObj *pDb, SVg
mInfo(
"vgId:%d, build create vnode req, replica:%d selfIndex:%d learnerReplica:%d learnerSelfIndex:%d strict:%d "
"changeVersion:%d",
createReq.vgId, createReq.replica, createReq.selfIndex, createReq.learnerReplica, createReq.learnerReplica,
createReq.vgId, createReq.replica, createReq.selfIndex, createReq.learnerReplica, createReq.learnerSelfIndex,
createReq.strict, createReq.changeVersion);
for (int32_t i = 0; i < createReq.replica; ++i) {
mInfo("vgId:%d, replica:%d ep:%s:%u", createReq.vgId, i, createReq.replicas[i].fqdn, createReq.replicas[i].port);
@ -738,10 +739,6 @@ void mndSortVnodeGid(SVgObj *pVgroup) {
}
static int32_t mndGetAvailableDnode(SMnode *pMnode, SDbObj *pDb, SVgObj *pVgroup, SArray *pArray) {
SSdb *pSdb = pMnode->pSdb;
int32_t allocedVnodes = 0;
void *pIter = NULL;
mDebug("start to sort %d dnodes", (int32_t)taosArrayGetSize(pArray));
taosArraySort(pArray, (__compar_fn_t)mndCompareDnodeVnodes);
for (int32_t i = 0; i < (int32_t)taosArrayGetSize(pArray); ++i) {
@ -883,7 +880,7 @@ SEpSet mndGetVgroupEpset(SMnode *pMnode, const SVgObj *pVgroup) {
SDnodeObj *pDnode = mndAcquireDnode(pMnode, pVgid->dnodeId);
if (pDnode == NULL) continue;
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER) {
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER || pVgid->syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
epset.inUse = epset.numOfEps;
}
@ -895,6 +892,29 @@ SEpSet mndGetVgroupEpset(SMnode *pMnode, const SVgObj *pVgroup) {
return epset;
}
SEpSet mndGetVgroupEpsetById(SMnode *pMnode, int32_t vgId) {
SEpSet epset = {0};
SVgObj * pVgroup = mndAcquireVgroup(pMnode, vgId);
if (!pVgroup) return epset;
for (int32_t v = 0; v < pVgroup->replica; ++v) {
const SVnodeGid *pVgid = &pVgroup->vnodeGid[v];
SDnodeObj *pDnode = mndAcquireDnode(pMnode, pVgid->dnodeId);
if (pDnode == NULL) continue;
if (pVgid->syncState == TAOS_SYNC_STATE_LEADER || pVgid->syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
epset.inUse = epset.numOfEps;
}
addEpIntoEpSet(&epset, pDnode->fqdn, pDnode->port);
mndReleaseDnode(pMnode, pDnode);
}
mndReleaseVgroup(pMnode, pVgroup);
return epset;
}
static int32_t mndRetrieveVgroups(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock, int32_t rows) {
SMnode *pMnode = pReq->info.node;
SSdb *pSdb = pMnode->pSdb;
@ -958,7 +978,8 @@ static int32_t mndRetrieveVgroups(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *p
strcpy(role, "dropping");
} else if (online) {
char *star = "";
if (pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_LEADER) {
if (pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_LEADER ||
pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
if (!pVgroup->vnodeGid[i].syncRestore && !pVgroup->vnodeGid[i].syncCanRead) {
star = "**";
} else if (!pVgroup->vnodeGid[i].syncRestore && pVgroup->vnodeGid[i].syncCanRead) {
@ -2288,7 +2309,7 @@ int32_t mndAddVgroupBalanceToTrans(SMnode *pMnode, SVgObj *pVgroup, STrans *pTra
return -1;
}
int32_t dnodeId = pVgroup->vnodeGid[0].dnodeId;
int32_t dnodeId = 0;
for (int i = 0; i < replica; i++) {
if (pVgroup->vnodeGid[i].syncState == TAOS_SYNC_STATE_LEADER) {
@ -2389,81 +2410,102 @@ static int32_t mndCheckDnodeMemory(SMnode *pMnode, SDbObj *pOldDb, SDbObj *pNewD
}
int32_t mndBuildAlterVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj *pOldDb, SDbObj *pNewDb, SVgObj *pVgroup,
SArray *pArray) {
SVgObj newVgroup = {0};
memcpy(&newVgroup, pVgroup, sizeof(SVgObj));
SArray *pArray, SVgObj *pNewVgroup) {
memcpy(pNewVgroup, pVgroup, sizeof(SVgObj));
if (pVgroup->replica <= 0 || pVgroup->replica == pNewDb->cfg.replications) {
if (mndAddAlterVnodeConfigAction(pMnode, pTrans, pNewDb, pVgroup) != 0) return -1;
if (mndCheckDnodeMemory(pMnode, pOldDb, pNewDb, &newVgroup, pVgroup, pArray) != 0) return -1;
if (mndCheckDnodeMemory(pMnode, pOldDb, pNewDb, pNewVgroup, pVgroup, pArray) != 0) return -1;
return 0;
}
mndTransSetSerial(pTrans);
if (newVgroup.replica == 1 && pNewDb->cfg.replications == 3) {
if (pNewVgroup->replica == 1 && pNewDb->cfg.replications == 3) {
mInfo("db:%s, vgId:%d, will add 2 vnodes, vn:0 dnode:%d", pVgroup->dbName, pVgroup->vgId,
pVgroup->vnodeGid[0].dnodeId);
// add second
if (mndAddVnodeToVgroup(pMnode, pTrans, &newVgroup, pArray) != 0) return -1;
if (mndAddVnodeToVgroup(pMnode, pTrans, pNewVgroup, pArray) != 0) return -1;
// learner stage
newVgroup.vnodeGid[0].nodeRole = TAOS_SYNC_ROLE_VOTER;
newVgroup.vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_LEARNER;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[0].dnodeId) != 0)
pNewVgroup->vnodeGid[0].nodeRole = TAOS_SYNC_ROLE_VOTER;
pNewVgroup->vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_LEARNER;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddCreateVnodeAction(pMnode, pTrans, pNewDb, &newVgroup, &newVgroup.vnodeGid[1]) != 0) return -1;
if (mndAddCreateVnodeAction(pMnode, pTrans, pNewDb, pNewVgroup, &pNewVgroup->vnodeGid[1]) != 0) return -1;
// follower stage
newVgroup.vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_VOTER;
if (mndAddAlterVnodeTypeAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[1].dnodeId) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[0].dnodeId) != 0)
pNewVgroup->vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_VOTER;
if (mndAddAlterVnodeTypeAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[1].dnodeId) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, &newVgroup) != 0) return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, pNewVgroup) != 0) return -1;
// add third
if (mndAddVnodeToVgroup(pMnode, pTrans, &newVgroup, pArray) != 0) return -1;
if (mndAddVnodeToVgroup(pMnode, pTrans, pNewVgroup, pArray) != 0) return -1;
newVgroup.vnodeGid[0].nodeRole = TAOS_SYNC_ROLE_VOTER;
newVgroup.vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_VOTER;
newVgroup.vnodeGid[2].nodeRole = TAOS_SYNC_ROLE_VOTER;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[0].dnodeId) != 0)
pNewVgroup->vnodeGid[0].nodeRole = TAOS_SYNC_ROLE_VOTER;
pNewVgroup->vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_VOTER;
pNewVgroup->vnodeGid[2].nodeRole = TAOS_SYNC_ROLE_VOTER;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[1].dnodeId) != 0)
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[1].dnodeId) != 0)
return -1;
if (mndAddCreateVnodeAction(pMnode, pTrans, pNewDb, &newVgroup, &newVgroup.vnodeGid[2]) != 0) return -1;
if (mndAddCreateVnodeAction(pMnode, pTrans, pNewDb, pNewVgroup, &pNewVgroup->vnodeGid[2]) != 0) return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, &newVgroup) != 0) return -1;
} else if (newVgroup.replica == 3 && pNewDb->cfg.replications == 1) {
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, pNewVgroup) != 0) return -1;
} else if (pNewVgroup->replica == 3 && pNewDb->cfg.replications == 1) {
mInfo("db:%s, vgId:%d, will remove 2 vnodes, vn:0 dnode:%d vn:1 dnode:%d vn:2 dnode:%d", pVgroup->dbName,
pVgroup->vgId, pVgroup->vnodeGid[0].dnodeId, pVgroup->vnodeGid[1].dnodeId, pVgroup->vnodeGid[2].dnodeId);
SVnodeGid del1 = {0};
SVnodeGid del2 = {0};
if (mndRemoveVnodeFromVgroup(pMnode, pTrans, &newVgroup, pArray, &del1) != 0) return -1;
if (mndAddDropVnodeAction(pMnode, pTrans, pNewDb, &newVgroup, &del1, true) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[0].dnodeId) != 0)
if (mndRemoveVnodeFromVgroup(pMnode, pTrans, pNewVgroup, pArray, &del1) != 0) return -1;
if (mndAddDropVnodeAction(pMnode, pTrans, pNewDb, pNewVgroup, &del1, true) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[1].dnodeId) != 0)
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[1].dnodeId) != 0)
return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, &newVgroup) != 0) return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, pNewVgroup) != 0) return -1;
if (mndRemoveVnodeFromVgroup(pMnode, pTrans, &newVgroup, pArray, &del2) != 0) return -1;
if (mndAddDropVnodeAction(pMnode, pTrans, pNewDb, &newVgroup, &del2, true) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, &newVgroup, newVgroup.vnodeGid[0].dnodeId) != 0)
if (mndRemoveVnodeFromVgroup(pMnode, pTrans, pNewVgroup, pArray, &del2) != 0) return -1;
if (mndAddDropVnodeAction(pMnode, pTrans, pNewDb, pNewVgroup, &del2, true) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, &newVgroup) != 0) return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, pNewVgroup) != 0) return -1;
} else if (pNewVgroup->replica == 1 && pNewDb->cfg.replications == 2) {
mInfo("db:%s, vgId:%d, will add 1 vnode, vn:0 dnode:%d", pVgroup->dbName, pVgroup->vgId,
pVgroup->vnodeGid[0].dnodeId);
// add second
if (mndAddVnodeToVgroup(pMnode, pTrans, pNewVgroup, pArray) != 0) return -1;
// learner stage
pNewVgroup->vnodeGid[0].nodeRole = TAOS_SYNC_ROLE_VOTER;
pNewVgroup->vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_LEARNER;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddCreateVnodeAction(pMnode, pTrans, pNewDb, pNewVgroup, &pNewVgroup->vnodeGid[1]) != 0) return -1;
// follower stage
pNewVgroup->vnodeGid[1].nodeRole = TAOS_SYNC_ROLE_VOTER;
if (mndAddAlterVnodeTypeAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[1].dnodeId) != 0) return -1;
if (mndAddAlterVnodeReplicaAction(pMnode, pTrans, pNewDb, pNewVgroup, pNewVgroup->vnodeGid[0].dnodeId) != 0)
return -1;
if (mndAddAlterVnodeConfirmAction(pMnode, pTrans, pNewDb, pNewVgroup) != 0) return -1;
} else {
return -1;
}
mndSortVnodeGid(&newVgroup);
mndSortVnodeGid(pNewVgroup);
{
SSdbRaw *pVgRaw = mndVgroupActionEncode(&newVgroup);
SSdbRaw *pVgRaw = mndVgroupActionEncode(pNewVgroup);
if (pVgRaw == NULL) return -1;
if (mndTransAppendCommitlog(pTrans, pVgRaw) != 0) {
sdbFreeRaw(pVgRaw);
@ -2622,7 +2664,7 @@ int32_t mndBuildRestoreAlterVgroupAction(SMnode *pMnode, STrans *pTrans, SDbObj
}
}
if (mndAddCreateVnodeAction(pMnode, pTrans, db, &newVgroup, &newVgroup.vnodeGid[selected]) != 0) return -1;
} else if (newVgroup.replica == 3) {
} else if (newVgroup.replica == 2 || newVgroup.replica == 3) {
for (int i = 0; i < newVgroup.replica; i++) {
if (newVgroup.vnodeGid[i].dnodeId == pDnode->id) {
newVgroup.vnodeGid[i].nodeRole = TAOS_SYNC_ROLE_LEARNER;
@ -2716,6 +2758,12 @@ int32_t mndSplitVgroup(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SVgObj *pVgro
}
#endif
if (pDb->cfg.withArbitrator) {
code = TSDB_CODE_OPS_NOT_SUPPORT;
mError("vgId:%d, db:%s, with arbitrator, split vgroup not allowed", pVgroup->vgId, pVgroup->dbName);
goto _OVER;
}
pTrans = mndTransCreate(pMnode, TRN_POLICY_RETRY, TRN_CONFLICT_DB, pReq, "split-vgroup");
if (pTrans == NULL) goto _OVER;
mndTransSetSerial(pTrans);
@ -2813,13 +2861,15 @@ int32_t mndSplitVgroup(SMnode *pMnode, SRpcMsg *pReq, SDbObj *pDb, SVgObj *pVgro
// adjust vgroup replica
if (pDb->cfg.replications != newVg1.replica) {
if (mndBuildAlterVgroupAction(pMnode, pTrans, pDb, pDb, &newVg1, pArray) != 0) goto _OVER;
SVgObj tmpGroup = {0};
if (mndBuildAlterVgroupAction(pMnode, pTrans, pDb, pDb, &newVg1, pArray, &tmpGroup) != 0) goto _OVER;
} else {
if (mndAddVgStatusAction(pTrans, &newVg1, SDB_STATUS_READY, TRN_STAGE_COMMIT_ACTION) < 0) goto _OVER;
}
if (pDb->cfg.replications != newVg2.replica) {
if (mndBuildAlterVgroupAction(pMnode, pTrans, pDb, pDb, &newVg2, pArray) != 0) goto _OVER;
SVgObj tmpGroup = {0};
if (mndBuildAlterVgroupAction(pMnode, pTrans, pDb, pDb, &newVg2, pArray, &tmpGroup) != 0) goto _OVER;
} else {
if (mndAddVgStatusAction(pTrans, &newVg2, SDB_STATUS_READY, TRN_STAGE_COMMIT_ACTION) < 0) goto _OVER;
}
@ -2907,10 +2957,22 @@ static int32_t mndBalanceVgroupBetweenDnode(SMnode *pMnode, STrans *pTrans, SDno
}
SDbObj *pDb = mndAcquireDb(pMnode, pVgroup->dbName);
if (pDb == NULL) {
mError("vgId:%d, balance vgroup can't find db obj dbName:%s", pVgroup->vgId, pVgroup->dbName);
goto _OUT;
}
if (pDb->cfg.withArbitrator) {
mInfo("vgId:%d, db:%s, with arbitrator, balance vgroup not allowed", pVgroup->vgId, pVgroup->dbName);
goto _OUT;
}
code = mndSetBalanceVgroupInfoToTrans(pMnode, pTrans, pDb, pVgroup, pSrc, pDst);
if (code == 0) {
code = taosHashPut(pBalancedVgroups, &pVgroup->vgId, sizeof(int32_t), &pVgroup->vgId, sizeof(int32_t));
}
_OUT:
mndReleaseDb(pMnode, pDb);
sdbRelease(pSdb, pVgroup);
sdbCancelFetch(pSdb, pIter);

1
source/dnode/mnode/impl/test/CMakeLists.txt
View File

@ -1,6 +1,7 @@
enable_testing()
add_subdirectory(acct)
add_subdirectory(arbgroup)
#add_subdirectory(db)
#add_subdirectory(dnode)
add_subdirectory(func)

13
source/dnode/mnode/impl/test/arbgroup/CMakeLists.txt Normal file
View File

@ -0,0 +1,13 @@
SET(CMAKE_CXX_STANDARD 11)
aux_source_directory(. MNODE_ARBGROUP_TEST_SRC)
add_executable(arbgroupTest ${MNODE_ARBGROUP_TEST_SRC})
target_link_libraries(
arbgroupTest
PRIVATE dnode nodes planner gtest qcom
)
add_test(
NAME arbgroupTest
COMMAND arbgroupTest
)

286
source/dnode/mnode/impl/test/arbgroup/arbgroup.cpp Normal file
View File

@ -0,0 +1,286 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
// #include <iostream>
#include <string>
#include <gtest/gtest.h>
// #include "nodes.h"
// #include "planner.h"
#pragma GCC diagnostic push
#pragma GCC diagnostic ignored "-Wwrite-strings"
#pragma GCC diagnostic ignored "-Wunused-function"
#pragma GCC diagnostic ignored "-Wunused-variable"
#pragma GCC diagnostic ignored "-Wsign-compare"
// #include <libs/transport/trpc.h>
#include "../../inc/mndArbGroup.h"
namespace {
void generateArbToken(int32_t nodeId, int32_t vgId, char* buf) {
memset(buf, 0, TSDB_ARB_TOKEN_SIZE);
int32_t randVal = taosSafeRand() % 1000;
int64_t currentMs = taosGetTimestampMs();
snprintf(buf, TSDB_ARB_TOKEN_SIZE, "d%d#g%d#%" PRId64 "#%d", nodeId, vgId, currentMs, randVal);
}
} // namespace
class ArbgroupTest : public testing::Test {
protected:
static void SetUpTestSuite() {
std::cout << "setup env for arbgroupTest suite" << std::endl;
}
static void TearDownTestSuite() { std::cout << "tearDown env for arbgroupTest suite" << std::endl; }
virtual void SetUp() override {}
virtual void TearDown() override {}
};
int main(int argc, char** argv) {
testing::InitGoogleTest(&argc, argv);
return RUN_ALL_TESTS();
}
TEST_F(ArbgroupTest, 01_encode_decode_sdb) {
SArbGroup group = {0};
group.vgId = 5;
group.dbUid = 1234;
group.members[0].info.dnodeId = 1;
generateArbToken(1, 5, group.members[0].state.token);
group.members[1].info.dnodeId = 2;
generateArbToken(2, 5, group.members[1].state.token);
group.isSync = 1;
group.assignedLeader.dnodeId = 1;
generateArbToken(1, 5, group.assignedLeader.token);
group.version = 2234;
// --------------------------------------------------------------------------------
SSdbRaw* pRaw = mndArbGroupActionEncode(&group);
ASSERT_NE(pRaw, nullptr);
SSdbRow* pRow = mndArbGroupActionDecode(pRaw);
ASSERT_NE(pRow, nullptr);
SArbGroup* pNewGroup = (SArbGroup*)sdbGetRowObj(pRow);
EXPECT_EQ(group.vgId, pNewGroup->vgId);
EXPECT_EQ(group.dbUid, pNewGroup->dbUid);
EXPECT_EQ(group.members[0].info.dnodeId, pNewGroup->members[0].info.dnodeId);
EXPECT_EQ(group.members[1].info.dnodeId, pNewGroup->members[1].info.dnodeId);
EXPECT_EQ(group.isSync, pNewGroup->isSync);
EXPECT_EQ(group.assignedLeader.dnodeId, pNewGroup->assignedLeader.dnodeId);
EXPECT_EQ(std::string(group.members[0].state.token), std::string(pNewGroup->members[0].state.token));
EXPECT_EQ(std::string(group.members[1].state.token), std::string(pNewGroup->members[1].state.token));
EXPECT_EQ(std::string(group.assignedLeader.token), std::string(pNewGroup->assignedLeader.token));
EXPECT_EQ(group.version, pNewGroup->version);
taosMemoryFree(pRow);
taosMemoryFree(pRaw);
}
TEST_F(ArbgroupTest, 02_process_heart_beat_rsp) {
const int32_t dnodeId = 1;
const int32_t vgId = 5;
SArbGroup group = {0};
group.vgId = vgId;
group.dbUid = 1234;
group.members[0].info.dnodeId = dnodeId;
generateArbToken(dnodeId, vgId, group.members[0].state.token);
group.members[0].state.lastHbMs = 1000;
group.members[0].state.responsedHbSeq = 100;
group.members[0].state.nextHbSeq = 102;
group.members[1].info.dnodeId = 2;
generateArbToken(2, vgId, group.members[1].state.token);
group.isSync = 1;
group.assignedLeader.dnodeId = dnodeId;
strncpy(group.assignedLeader.token, group.members[0].state.token, TSDB_ARB_TOKEN_SIZE);
taosThreadMutexInit(&group.mutex, NULL);
// --------------------------------------------------------------------------------
{ // expired hb => skip
SVArbHbRspMember rspMember = {0};
rspMember.vgId = vgId;
rspMember.hbSeq = group.members[0].state.responsedHbSeq - 1;
strncpy(rspMember.memberToken, group.members[0].state.token, TSDB_ARB_TOKEN_SIZE);
int32_t nowMs = group.members[0].state.lastHbMs + 10;
SArbGroup newGroup = {0};
bool updateToken = mndUpdateArbGroupByHeartBeat(&group, &rspMember, nowMs, dnodeId, &newGroup);
EXPECT_FALSE(updateToken);
EXPECT_NE(group.members[0].state.responsedHbSeq, rspMember.hbSeq);
EXPECT_NE(group.members[0].state.lastHbMs, nowMs);
}
{ // old token
SVArbHbRspMember rspMember = {0};
rspMember.vgId = vgId;
rspMember.hbSeq = group.members[0].state.responsedHbSeq + 1;
strncpy(rspMember.memberToken, group.members[0].state.token, TSDB_ARB_TOKEN_SIZE);
int32_t nowMs = group.members[0].state.lastHbMs + 10;
SArbGroup newGroup = {0};
bool updateToken = mndUpdateArbGroupByHeartBeat(&group, &rspMember, nowMs, dnodeId, &newGroup);
EXPECT_FALSE(updateToken);
EXPECT_EQ(group.members[0].state.responsedHbSeq, rspMember.hbSeq);
EXPECT_EQ(group.members[0].state.lastHbMs, nowMs);
}
{ // new token
SVArbHbRspMember rspMember = {0};
rspMember.vgId = vgId;
rspMember.hbSeq = group.members[0].state.responsedHbSeq + 1;
generateArbToken(dnodeId, vgId, rspMember.memberToken);
int32_t nowMs = group.members[0].state.lastHbMs + 10;
SArbGroup newGroup = {0};
bool updateToken = mndUpdateArbGroupByHeartBeat(&group, &rspMember, nowMs, dnodeId, &newGroup);
EXPECT_TRUE(updateToken);
EXPECT_EQ(group.members[0].state.responsedHbSeq, rspMember.hbSeq);
EXPECT_EQ(group.members[0].state.lastHbMs, nowMs);
EXPECT_EQ(std::string(newGroup.members[0].state.token), std::string(rspMember.memberToken));
EXPECT_FALSE(newGroup.isSync);
EXPECT_EQ(newGroup.assignedLeader.dnodeId, 0);
EXPECT_EQ(std::string(newGroup.assignedLeader.token).size(), 0);
}
taosThreadMutexDestroy(&group.mutex);
}
TEST_F(ArbgroupTest, 03_process_check_sync_rsp) {
const int32_t dnodeId = 1;
const int32_t vgId = 5;
SArbGroup group = {0};
group.vgId = vgId;
group.dbUid = 1234;
group.members[0].info.dnodeId = dnodeId;
generateArbToken(dnodeId, vgId, group.members[0].state.token);
group.members[0].state.lastHbMs = 1000;
group.members[0].state.responsedHbSeq = 100;
group.members[0].state.nextHbSeq = 102;
group.members[1].info.dnodeId = 2;
generateArbToken(2, vgId, group.members[1].state.token);
group.isSync = 0;
taosThreadMutexInit(&group.mutex, NULL);
// --------------------------------------------------------------------------------
{ // token mismatch => skip
char member0Token[TSDB_ARB_TOKEN_SIZE] = {0};
strncpy(member0Token, group.members[0].state.token, TSDB_ARB_TOKEN_SIZE);
char member1Token[TSDB_ARB_TOKEN_SIZE] = {0};
generateArbToken(2, 5, member1Token);
bool newIsSync = false;
SArbGroup newGroup = {0};
bool updateIsSync = mndUpdateArbGroupByCheckSync(&group, vgId, member0Token, member1Token, newIsSync, &newGroup);
EXPECT_FALSE(updateIsSync);
}
{ // newIsSync
char member0Token[TSDB_ARB_TOKEN_SIZE] = {0};
strncpy(member0Token, group.members[0].state.token, TSDB_ARB_TOKEN_SIZE);
char member1Token[TSDB_ARB_TOKEN_SIZE] = {0};
strncpy(member1Token, group.members[1].state.token, TSDB_ARB_TOKEN_SIZE);
bool newIsSync = true;
SArbGroup newGroup = {0};
bool updateIsSync = mndUpdateArbGroupByCheckSync(&group, vgId, member0Token, member1Token, newIsSync, &newGroup);
EXPECT_TRUE(updateIsSync);
EXPECT_TRUE(newGroup.isSync);
}
taosThreadMutexDestroy(&group.mutex);
}
TEST_F(ArbgroupTest, 04_process_set_assigned_leader){
const int32_t dnodeId = 1;
const int32_t vgId = 5;
SArbGroup group = {0};
group.vgId = vgId;
group.dbUid = 1234;
group.members[0].info.dnodeId = dnodeId;
generateArbToken(dnodeId, vgId, group.members[0].state.token);
group.members[0].state.lastHbMs = 1000;
group.members[0].state.responsedHbSeq = 100;
group.members[0].state.nextHbSeq = 102;
group.members[1].info.dnodeId = 2;
generateArbToken(2, vgId, group.members[1].state.token);
group.isSync = 1;
group.assignedLeader.dnodeId = dnodeId;
strncpy(group.assignedLeader.token, group.members[0].state.token, TSDB_ARB_TOKEN_SIZE);
taosThreadMutexInit(&group.mutex, NULL);
// --------------------------------------------------------------------------------
{ // token mismatch => skip
char memberToken[TSDB_ARB_TOKEN_SIZE] = {0};
generateArbToken(dnodeId, vgId, memberToken);
int32_t errcode = TSDB_CODE_SUCCESS;
SArbGroup newGroup = {0};
bool updateAssigned = mndUpdateArbGroupBySetAssignedLeader(&group, vgId, memberToken, errcode, &newGroup);
EXPECT_FALSE(updateAssigned);
}
{ // errcode != TSDB_CODE_SUCCESS
char memberToken[TSDB_ARB_TOKEN_SIZE] = {0};
strncpy(memberToken, group.assignedLeader.token, TSDB_ARB_TOKEN_SIZE);
int32_t errcode = TSDB_CODE_MND_ARB_TOKEN_MISMATCH;
SArbGroup newGroup = {0};
bool updateAssigned = mndUpdateArbGroupBySetAssignedLeader(&group, vgId, memberToken, errcode, &newGroup);
EXPECT_FALSE(updateAssigned);
}
{ // errcode == TSDB_CODE_SUCCESS
char memberToken[TSDB_ARB_TOKEN_SIZE] = {0};
strncpy(memberToken, group.assignedLeader.token, TSDB_ARB_TOKEN_SIZE);
int32_t errcode = TSDB_CODE_SUCCESS;
SArbGroup newGroup = {0};
bool updateAssigned = mndUpdateArbGroupBySetAssignedLeader(&group, vgId, memberToken, errcode, &newGroup);
EXPECT_TRUE(updateAssigned);
EXPECT_FALSE(newGroup.isSync);
}
taosThreadMutexDestroy(&group.mutex);
}
#pragma GCC diagnostic pop

3
source/dnode/mnode/sdb/inc/sdb.h
View File

@ -153,7 +153,8 @@ typedef enum {
SDB_COMPACT = 24,
SDB_COMPACT_DETAIL = 25,
SDB_GRANT = 26, // grant log
SDB_MAX = 27
SDB_ARBGROUP = 27,
SDB_MAX = 28
} ESdbType;
typedef struct SSdbRaw {

4
source/dnode/mnode/sdb/src/sdbHash.c
View File

@ -70,7 +70,9 @@ const char *sdbTableName(ESdbType type) {
return "compact";
case SDB_COMPACT_DETAIL:
return "compact_detail";
default:
case SDB_ARBGROUP:
return "arb_group";
default:
return "undefine";
}
}

3
source/dnode/vnode/inc/vnode.h
View File

@ -76,6 +76,9 @@ int32_t vnodeGetTableList(void *pVnode, int8_t type, SArray *pList);
int32_t vnodeGetAllTableList(SVnode *pVnode, uint64_t uid, SArray *list);
int32_t vnodeIsCatchUp(SVnode *pVnode);
ESyncRole vnodeGetRole(SVnode *pVnode);
int32_t vnodeGetArbToken(SVnode *pVnode, char *outToken);
int32_t vnodeUpdateArbTerm(SVnode *pVnode, int64_t arbTerm);
int32_t vnodeGetCtbIdList(void *pVnode, int64_t suid, SArray *list);
int32_t vnodeGetCtbIdListByFilter(SVnode *pVnode, int64_t suid, SArray *list, bool (*filter)(void *arg), void *arg);

2
source/dnode/vnode/src/inc/vnodeInt.h
View File

@ -136,7 +136,7 @@ typedef SVCreateTSmaReq SSmaCfg;
SMTbCursor* metaOpenTbCursor(void* pVnode);
void metaCloseTbCursor(SMTbCursor* pTbCur);
void metaPauseTbCursor(SMTbCursor* pTbCur);
void metaResumeTbCursor(SMTbCursor* pTbCur, int8_t first);
void metaResumeTbCursor(SMTbCursor* pTbCur, int8_t first, int8_t move);
int32_t metaTbCursorNext(SMTbCursor* pTbCur, ETableType jumpTableType);
int32_t metaTbCursorPrev(SMTbCursor* pTbCur, ETableType jumpTableType);

18
source/dnode/vnode/src/meta/metaOpen.c
View File

@ -246,32 +246,20 @@ int metaAlterCache(SMeta *pMeta, int32_t nPage) {
}
int32_t metaRLock(SMeta *pMeta) {
int32_t ret = 0;
metaTrace("meta rlock %p", &pMeta->lock);
ret = taosThreadRwlockRdlock(&pMeta->lock);
int32_t ret = taosThreadRwlockRdlock(&pMeta->lock);
return ret;
}
int32_t metaWLock(SMeta *pMeta) {
int32_t ret = 0;
metaTrace("meta wlock %p", &pMeta->lock);
ret = taosThreadRwlockWrlock(&pMeta->lock);
int32_t ret = taosThreadRwlockWrlock(&pMeta->lock);
return ret;
}
int32_t metaULock(SMeta *pMeta) {
int32_t ret = 0;
metaTrace("meta ulock %p", &pMeta->lock);
ret = taosThreadRwlockUnlock(&pMeta->lock);
int32_t ret = taosThreadRwlockUnlock(&pMeta->lock);
return ret;
}

10
source/dnode/vnode/src/meta/metaQuery.c
View File

@ -248,7 +248,7 @@ SMTbCursor *metaOpenTbCursor(void *pVnode) {
// tdbTbcMoveToFirst((TBC *)pTbCur->pDbc);
pTbCur->pMeta = pVnodeObj->pMeta;
pTbCur->paused = 1;
metaResumeTbCursor(pTbCur, 1);
metaResumeTbCursor(pTbCur, 1, 0);
return pTbCur;
}
@ -273,7 +273,7 @@ void metaPauseTbCursor(SMTbCursor *pTbCur) {
pTbCur->paused = 1;
}
}
void metaResumeTbCursor(SMTbCursor *pTbCur, int8_t first) {
void metaResumeTbCursor(SMTbCursor *pTbCur, int8_t first, int8_t move) {
if (pTbCur->paused) {
metaReaderDoInit(&pTbCur->mr, pTbCur->pMeta, META_READER_LOCK);
@ -282,9 +282,11 @@ void metaResumeTbCursor(SMTbCursor *pTbCur, int8_t first) {
if (first) {
tdbTbcMoveToFirst((TBC *)pTbCur->pDbc);
} else {
int c = 0;
int c = 1;
tdbTbcMoveTo(pTbCur->pDbc, pTbCur->pKey, pTbCur->kLen, &c);
if (c < 0) {
if (c == 0) {
if (move) tdbTbcMoveToNext(pTbCur->pDbc);
} else if (c < 0) {
tdbTbcMoveToPrev(pTbCur->pDbc);
} else {
tdbTbcMoveToNext(pTbCur->pDbc);

2
source/dnode/vnode/src/tsdb/tsdbMergeTree.c
View File

@ -437,7 +437,7 @@ static int32_t doLoadSttFilesBlk(SSttBlockLoadInfo *pBlockLoadInfo, SLDataIter *
code = loadTombFn(pReader1, pIter->pReader, pIter->pBlockLoadInfo);
double el = (taosGetTimestampUs() - st) / 1000.0;
tsdbDebug("load the stt file info completed, elapsed time:%.2fms, %s", el, idStr);
tsdbDebug("load the stt file blk info completed, elapsed time:%.2fms, %s", el, idStr);
return code;
}

4
source/dnode/vnode/src/tsdb/tsdbRead2.c
View File

@ -2168,8 +2168,8 @@ static bool initSttBlockReader(SSttBlockReader* pSttBlockReader, STableBlockScan
pScanInfo->sttKeyInfo.nextProcKey =
ASCENDING_TRAVERSE(pReader->info.order) ? pScanInfo->sttWindow.skey : pScanInfo->sttWindow.ekey;
hasData = (pScanInfo->sttKeyInfo.status == STT_FILE_HAS_DATA);
} else { // not clean stt blocks
INIT_TIMEWINDOW(&pScanInfo->sttWindow); // reset the time window
} else { // not clean stt blocks
INIT_TIMEWINDOW(&pScanInfo->sttWindow); //reset the time window
pScanInfo->sttBlockReturned = false;
hasData = nextRowFromSttBlocks(pSttBlockReader, pScanInfo, pReader->suppInfo.pkSrcSlot, &pReader->info.verRange);
}

7
source/dnode/vnode/src/vnd/vnodeOpen.c
View File

@ -493,7 +493,7 @@ SVnode *vnodeOpen(const char *path, int32_t diskPrimary, STfs *pTfs, SMsgCb msgC
VNODE_METRIC_TAG_NAME_DNODE_ID, VNODE_METRIC_TAG_NAME_DNODE_EP,
VNODE_METRIC_TAG_NAME_VGROUP_ID, VNODE_METRIC_TAG_NAME_USERNAME,
VNODE_METRIC_TAG_NAME_RESULT};
counter = taos_counter_new(VNODE_METRIC_SQL_COUNT, "counter for insert sql",
counter = taos_counter_new(VNODE_METRIC_SQL_COUNT, "counter for insert sql",
label_count, sample_labels);
vInfo("vgId:%d, new metric:%p",TD_VID(pVnode), counter);
if(taos_collector_registry_register_metric(counter) == 1){
@ -503,7 +503,7 @@ SVnode *vnodeOpen(const char *path, int32_t diskPrimary, STfs *pTfs, SMsgCb msgC
}
}
pVnode->monitor.insertCounter = counter;
vInfo("vgId:%d, succeed to set metric:%p",TD_VID(pVnode), counter);
vInfo("vgId:%d, succeed to set metric:%p",TD_VID(pVnode), counter);
}
return pVnode;
@ -560,6 +560,9 @@ int32_t vnodeIsCatchUp(SVnode *pVnode) { return syncIsCatchUp(pVnode->sync); }
ESyncRole vnodeGetRole(SVnode *pVnode) { return syncGetRole(pVnode->sync); }
int32_t vnodeUpdateArbTerm(SVnode *pVnode, int64_t arbTerm) { return syncUpdateArbTerm(pVnode->sync, arbTerm); }
int32_t vnodeGetArbToken(SVnode *pVnode, char *outToken) { return syncGetArbToken(pVnode->sync, outToken); }
void vnodeStop(SVnode *pVnode) {}
int64_t vnodeGetSyncHandle(SVnode *pVnode) { return pVnode->sync; }

2
source/dnode/vnode/src/vnd/vnodeQuery.c
View File

@ -406,7 +406,7 @@ int32_t vnodeGetLoad(SVnode *pVnode, SVnodeLoad *pLoad) {
int32_t vnodeGetLoadLite(SVnode *pVnode, SVnodeLoadLite *pLoad) {
SSyncState syncState = syncGetState(pVnode->sync);
if (syncState.state == TAOS_SYNC_STATE_LEADER) {
if (syncState.state == TAOS_SYNC_STATE_LEADER || syncState.state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
pLoad->vgId = TD_VID(pVnode);
pLoad->nTimeSeries = metaGetTimeSeriesNum(pVnode->pMeta, 1);
return 0;

146
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -31,20 +31,24 @@ static int32_t vnodeProcessCreateTbReq(SVnode *pVnode, int64_t ver, void *pReq,
static int32_t vnodeProcessAlterTbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessDropTbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
SRpcMsg *pOriginRpc);
static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
SRpcMsg *pOriginalMsg);
static int32_t vnodeProcessCreateTSmaReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessAlterConfirmReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessAlterConfigReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessDropTtlTbReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessTrimReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
SRpcMsg *pOriginalMsg);
static int32_t vnodeProcessBatchDeleteReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessCreateIndexReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessDropIndexReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessCompactVnodeReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessConfigChangeReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodeProcessArbCheckSyncReq(SVnode *pVnode, void *pReq, int32_t len, SRpcMsg *pRsp);
static int32_t vnodePreCheckAssignedLogSyncd(SVnode *pVnode, char *member0Token, char *member1Token);
static int32_t vnodeCheckAssignedLogSyncd(SVnode *pVnode, char *member0Token, char *member1Token);
extern int32_t vnodeProcessKillCompactReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp);
extern int32_t vnodeQueryCompactProgress(SVnode *pVnode, SRpcMsg *pMsg);
@ -451,6 +455,21 @@ static int32_t vnodePreProcessBatchDeleteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
return code;
}
static int32_t vnodePreProcessArbCheckSyncMsg(SVnode *pVnode, SRpcMsg *pMsg) {
SVArbCheckSyncReq syncReq = {0};
if (tDeserializeSVArbCheckSyncReq((char *)pMsg->pCont + sizeof(SMsgHead), pMsg->contLen - sizeof(SMsgHead),
&syncReq) != 0) {
return TSDB_CODE_INVALID_MSG;
}
(void)vnodePreCheckAssignedLogSyncd(pVnode, syncReq.member0Token, syncReq.member1Token);
int32_t code = terrno;
tFreeSVArbCheckSyncReq(&syncReq);
return code;
}
int32_t vnodePreProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
@ -473,6 +492,9 @@ int32_t vnodePreProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg) {
case TDMT_VND_BATCH_DEL: {
code = vnodePreProcessBatchDeleteMsg(pVnode, pMsg);
} break;
case TDMT_VND_ARB_CHECK_SYNC: {
code = vnodePreProcessArbCheckSyncMsg(pVnode, pMsg);
} break;
default:
break;
}
@ -643,6 +665,10 @@ int32_t vnodeProcessWriteMsg(SVnode *pVnode, SRpcMsg *pMsg, int64_t ver, SRpcMsg
vnodeProcessKillCompactReq(pVnode, ver, pReq, len, pRsp);
break;
#endif
/* ARB */
case TDMT_VND_ARB_CHECK_SYNC:
vnodeProcessArbCheckSyncReq(pVnode, pReq, len, pRsp);
break;
default:
vError("vgId:%d, unprocessed msg, %d", TD_VID(pVnode), pMsg->msgType);
return -1;
@ -1489,7 +1515,7 @@ static int32_t vnodeRebuildSubmitReqMsg(SSubmitReq2 *pSubmitReq, void **ppMsg) {
return code;
}
static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
static int32_t vnodeProcessSubmitReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
SRpcMsg *pOriginalMsg) {
int32_t code = 0;
terrno = 0;
@ -1710,8 +1736,8 @@ _exit:
atomic_add_fetch_64(&pVnode->statis.nBatchInsert, 1);
if(tsEnableMonitor && pSubmitRsp->affectedRows > 0 && strlen(pOriginalMsg->info.conn.user) > 0){
const char *sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT_AFFECTED_ROWS, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
const char *sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT_AFFECTED_ROWS, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
pOriginalMsg->info.conn.user, "Success"};
taos_counter_add(pVnode->monitor.insertCounter, pSubmitRsp->affectedRows, sample_labels);
}
@ -1725,14 +1751,14 @@ _exit:
atomic_add_fetch_64(&pVnode->statis.nBatchInsertSuccess, 1);
code = tdProcessRSmaSubmit(pVnode->pSma, ver, pSubmitReq, pReq, len);
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
pOriginalMsg->info.conn.user, "Success"};
taos_counter_inc(pVnode->monitor.insertCounter, batch_sample_labels);
}
else{
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_INSERT, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
pOriginalMsg->info.conn.user, "Failed"};
taos_counter_inc(pVnode->monitor.insertCounter, batch_sample_labels);
}
@ -1992,7 +2018,7 @@ static int32_t vnodeProcessBatchDeleteReq(SVnode *pVnode, int64_t ver, void *pRe
return 0;
}
static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp,
SRpcMsg *pOriginalMsg) {
int32_t code = 0;
SDecoder *pCoder = &(SDecoder){0};
@ -2038,14 +2064,14 @@ static int32_t vnodeProcessDeleteReq(SVnode *pVnode, int64_t ver, void *pReq, in
_err:
/*
if(code == TSDB_CODE_SUCCESS){
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_DELETE, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_DELETE, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
pOriginalMsg->info.conn.user, "Success"};
taos_counter_inc(pVnode->monitor.insertCounter, batch_sample_labels);
}
else{
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_DELETE, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
const char *batch_sample_labels[] = {VNODE_METRIC_TAG_VALUE_DELETE, pVnode->monitor.strClusterId,
pVnode->monitor.strDnodeId, tsLocalEp, pVnode->monitor.strVgId,
pOriginalMsg->info.conn.user, "Failed"};
taos_counter_inc(pVnode->monitor.insertCounter, batch_sample_labels);
}
@ -2119,6 +2145,98 @@ static int32_t vnodeProcessConfigChangeReq(SVnode *pVnode, int64_t ver, void *pR
return 0;
}
static int32_t vnodePreCheckAssignedLogSyncd(SVnode *pVnode, char *member0Token, char *member1Token) {
SSyncState syncState = syncGetState(pVnode->sync);
if (syncState.state != TAOS_SYNC_STATE_LEADER) {
terrno = TSDB_CODE_SYN_NOT_LEADER;
return -1;
}
char token[TSDB_ARB_TOKEN_SIZE] = {0};
if (vnodeGetArbToken(pVnode, token) != 0) {
terrno = TSDB_CODE_NOT_FOUND;
return -1;
}
if (strncmp(token, member0Token, TSDB_ARB_TOKEN_SIZE) != 0 &&
strncmp(token, member1Token, TSDB_ARB_TOKEN_SIZE) != 0) {
terrno = TSDB_CODE_MND_ARB_TOKEN_MISMATCH;
return -1;
}
terrno = TSDB_CODE_SUCCESS;
return 0;
}
static int32_t vnodeCheckAssignedLogSyncd(SVnode *pVnode, char *member0Token, char *member1Token) {
int32_t code = vnodePreCheckAssignedLogSyncd(pVnode, member0Token, member1Token);
if (code != 0) {
return code;
}
return syncGetAssignedLogSynced(pVnode->sync);
}
static int32_t vnodeProcessArbCheckSyncReq(SVnode *pVnode, void *pReq, int32_t len, SRpcMsg *pRsp) {
int32_t code = 0;
SVArbCheckSyncReq syncReq = {0};
if (tDeserializeSVArbCheckSyncReq(pReq, len, &syncReq) != 0) {
terrno = TSDB_CODE_INVALID_MSG;
return -1;
}
pRsp->msgType = TDMT_VND_ARB_CHECK_SYNC_RSP;
pRsp->code = TSDB_CODE_SUCCESS;
pRsp->pCont = NULL;
pRsp->contLen = 0;
SVArbCheckSyncRsp syncRsp = {0};
syncRsp.arbToken = syncReq.arbToken;
syncRsp.member0Token = syncReq.member0Token;
syncRsp.member1Token = syncReq.member1Token;
syncRsp.vgId = TD_VID(pVnode);
(void)vnodeCheckAssignedLogSyncd(pVnode, syncReq.member0Token, syncReq.member1Token);
syncRsp.errCode = terrno;
if (vnodeUpdateArbTerm(pVnode, syncReq.arbTerm) != 0) {
vError("vgId:%d, failed to update arb term", TD_VID(pVnode));
code = -1;
goto _OVER;
}
int32_t contLen = tSerializeSVArbCheckSyncRsp(NULL, 0, &syncRsp);
if (contLen <= 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
code = -1;
goto _OVER;
}
void *pHead = rpcMallocCont(contLen);
if (!pHead) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
code = -1;
goto _OVER;
}
if (tSerializeSVArbCheckSyncRsp(pHead, contLen, &syncRsp) <= 0) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
rpcFreeCont(pHead);
code = -1;
goto _OVER;
}
pRsp->pCont = pHead;
pRsp->contLen = contLen;
terrno = TSDB_CODE_SUCCESS;
_OVER:
tFreeSVArbCheckSyncReq(&syncReq);
return code;
}
#ifndef TD_ENTERPRISE
int32_t vnodeProcessCompactVnodeReqImpl(SVnode *pVnode, int64_t ver, void *pReq, int32_t len, SRpcMsg *pRsp) {
return 0;

9
source/dnode/vnode/src/vnd/vnodeSync.c
View File

@ -638,6 +638,14 @@ static void vnodeBecomeLeader(const SSyncFSM *pFsm) {
}
}
static void vnodeBecomeAssignedLeader(const SSyncFSM* pFsm) {
SVnode *pVnode = pFsm->data;
vDebug("vgId:%d, become assigned leader", pVnode->config.vgId);
if (pVnode->pTq) {
tqUpdateNodeStage(pVnode->pTq, true);
}
}
static bool vnodeApplyQueueEmpty(const SSyncFSM *pFsm) {
SVnode *pVnode = pFsm->data;
@ -674,6 +682,7 @@ static SSyncFSM *vnodeSyncMakeFsm(SVnode *pVnode) {
pFsm->FpApplyQueueEmptyCb = vnodeApplyQueueEmpty;
pFsm->FpApplyQueueItems = vnodeApplyQueueItems;
pFsm->FpBecomeLeaderCb = vnodeBecomeLeader;
pFsm->FpBecomeAssignedLeaderCb = vnodeBecomeAssignedLeader;
pFsm->FpBecomeFollowerCb = vnodeBecomeFollower;
pFsm->FpBecomeLearnerCb = vnodeBecomeLearner;
pFsm->FpReConfigCb = NULL;

203
source/libs/executor/src/sysscanoperator.c
View File

@ -433,95 +433,113 @@ static bool sysTableIsCondOnOneTable(SNode* pCond, char* condTable) {
return false;
}
static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SStorageAPI* pAPI = &pTaskInfo->storageAPI;
static SSDataBlock* doOptimizeTableNameFilter(SOperatorInfo* pOperator, SSDataBlock* dataBlock, char* dbname) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SStorageAPI* pAPI = &pTaskInfo->storageAPI;
SSysTableScanInfo* pInfo = pOperator->info;
if (pOperator->status == OP_EXEC_DONE) {
int32_t numOfRows = 0;
char tableName[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(tableName, pInfo->req.filterTb);
SMetaReader smrTable = {0};
pAPI->metaReaderFn.initReader(&smrTable, pInfo->readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
int32_t code = pAPI->metaReaderFn.getTableEntryByName(&smrTable, pInfo->req.filterTb);
if (code != TSDB_CODE_SUCCESS) {
// terrno has been set by pAPI->metaReaderFn.getTableEntryByName, therefore, return directly
pAPI->metaReaderFn.clearReader(&smrTable);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
blockDataCleanup(pInfo->pRes);
int32_t numOfRows = 0;
if (smrTable.me.type == TSDB_SUPER_TABLE) {
pAPI->metaReaderFn.clearReader(&smrTable);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
SSDataBlock* dataBlock = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_COLS);
blockDataEnsureCapacity(dataBlock, pOperator->resultInfo.capacity);
if (smrTable.me.type == TSDB_CHILD_TABLE) {
int64_t suid = smrTable.me.ctbEntry.suid;
pAPI->metaReaderFn.clearReader(&smrTable);
pAPI->metaReaderFn.initReader(&smrTable, pInfo->readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
code = pAPI->metaReaderFn.getTableEntryByUid(&smrTable, suid);
if (code != TSDB_CODE_SUCCESS) {
// terrno has been set by pAPI->metaReaderFn.getTableEntryByName, therefore, return directly
pAPI->metaReaderFn.clearReader(&smrTable);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
}
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
SSchemaWrapper* schemaRow = NULL;
if (smrTable.me.type == TSDB_SUPER_TABLE) {
schemaRow = &smrTable.me.stbEntry.schemaRow;
STR_TO_VARSTR(typeName, "CHILD_TABLE");
} else if (smrTable.me.type == TSDB_NORMAL_TABLE) {
schemaRow = &smrTable.me.ntbEntry.schemaRow;
STR_TO_VARSTR(typeName, "NORMAL_TABLE");
}
sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, dataBlock, tableName, schemaRow, typeName);
pAPI->metaReaderFn.clearReader(&smrTable);
if (numOfRows > 0) {
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, dataBlock, pOperator->exprSupp.pFilterInfo);
numOfRows = 0;
}
pInfo->loadInfo.totalRows += pInfo->pRes->info.rows;
setOperatorCompleted(pOperator);
qDebug("get cols success, total rows:%" PRIu64 ", current:%" PRId64 " %s", pInfo->loadInfo.totalRows,
pInfo->pRes->info.rows, GET_TASKID(pTaskInfo));
return (pInfo->pRes->info.rows == 0) ? NULL : pInfo->pRes;
}
int32_t doExtractDbName(char* dbname, SSysTableScanInfo* pInfo, SStorageAPI* pAPI) {
SName sn = {0};
const char* db = NULL;
int32_t vgId = 0;
pAPI->metaFn.getBasicInfo(pInfo->readHandle.vnode, &db, &vgId, NULL, NULL);
SName sn = {0};
char dbname[TSDB_DB_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
tNameFromString(&sn, db, T_NAME_ACCT | T_NAME_DB);
tNameGetDbName(&sn, varDataVal(dbname));
varDataSetLen(dbname, strlen(varDataVal(dbname)));
// optimize when sql like where table_name='tablename' and xxx.
if (pInfo->req.filterTb[0]) {
char tableName[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
STR_TO_VARSTR(tableName, pInfo->req.filterTb);
return TSDB_CODE_SUCCESS;
}
SMetaReader smrTable = {0};
pAPI->metaReaderFn.initReader(&smrTable, pInfo->readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
int32_t code = pAPI->metaReaderFn.getTableEntryByName(&smrTable, pInfo->req.filterTb);
if (code != TSDB_CODE_SUCCESS) {
// terrno has been set by pAPI->metaReaderFn.getTableEntryByName, therefore, return directly
pAPI->metaReaderFn.clearReader(&smrTable);
blockDataDestroy(dataBlock);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SStorageAPI* pAPI = &pTaskInfo->storageAPI;
SSysTableScanInfo* pInfo = pOperator->info;
int32_t numOfRows = 0;
int32_t ret = 0;
char dbname[TSDB_DB_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
SSDataBlock* pDataBlock = NULL;
if (smrTable.me.type == TSDB_SUPER_TABLE) {
pAPI->metaReaderFn.clearReader(&smrTable);
blockDataDestroy(dataBlock);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
if (smrTable.me.type == TSDB_CHILD_TABLE) {
int64_t suid = smrTable.me.ctbEntry.suid;
pAPI->metaReaderFn.clearReader(&smrTable);
pAPI->metaReaderFn.initReader(&smrTable, pInfo->readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
code = pAPI->metaReaderFn.getTableEntryByUid(&smrTable, suid);
if (code != TSDB_CODE_SUCCESS) {
// terrno has been set by pAPI->metaReaderFn.getTableEntryByName, therefore, return directly
pAPI->metaReaderFn.clearReader(&smrTable);
blockDataDestroy(dataBlock);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
}
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
SSchemaWrapper* schemaRow = NULL;
if (smrTable.me.type == TSDB_SUPER_TABLE) {
schemaRow = &smrTable.me.stbEntry.schemaRow;
STR_TO_VARSTR(typeName, "CHILD_TABLE");
} else if (smrTable.me.type == TSDB_NORMAL_TABLE) {
schemaRow = &smrTable.me.ntbEntry.schemaRow;
STR_TO_VARSTR(typeName, "NORMAL_TABLE");
}
sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, dataBlock, tableName, schemaRow, typeName);
pAPI->metaReaderFn.clearReader(&smrTable);
if (numOfRows > 0) {
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, dataBlock, pOperator->exprSupp.pFilterInfo);
numOfRows = 0;
}
blockDataDestroy(dataBlock);
pInfo->loadInfo.totalRows += pInfo->pRes->info.rows;
setOperatorCompleted(pOperator);
return (pInfo->pRes->info.rows == 0) ? NULL : pInfo->pRes;
if (pOperator->status == OP_EXEC_DONE) {
return NULL;
}
blockDataCleanup(pInfo->pRes);
pDataBlock = buildInfoSchemaTableMetaBlock(TSDB_INS_TABLE_COLS);
blockDataEnsureCapacity(pDataBlock, pOperator->resultInfo.capacity);
doExtractDbName(dbname, pInfo, pAPI);
// optimize when sql like where table_name='tablename' and xxx.
if (pInfo->req.filterTb[0]) {
SSDataBlock* p = doOptimizeTableNameFilter(pOperator, pDataBlock, dbname);
blockDataDestroy(pDataBlock);
return p;
}
int32_t ret = 0;
if (pInfo->pCur == NULL) {
pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode);
} else {
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 0);
}
if (pInfo->pSchema == NULL) {
@ -532,26 +550,20 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
if (!pInfo->pCur || !pInfo->pSchema) {
terrno = TSDB_CODE_OUT_OF_MEMORY;
qError("sysTableScanUserCols failed since %s", terrstr(terrno));
blockDataDestroy(dataBlock);
blockDataDestroy(pDataBlock);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
int32_t restore = pInfo->restore;
pInfo->restore = false;
while (restore || ((ret = pAPI->metaFn.cursorNext(pInfo->pCur, TSDB_TABLE_MAX)) == 0)) {
if (restore) {
restore = false;
}
while (((ret = pAPI->metaFn.cursorNext(pInfo->pCur, TSDB_TABLE_MAX)) == 0)) {
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
char tableName[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
SSchemaWrapper* schemaRow = NULL;
if (pInfo->pCur->mr.me.type == TSDB_SUPER_TABLE) {
qDebug("sysTableScanUserCols cursor get super table");
qDebug("sysTableScanUserCols cursor get super table, %s", GET_TASKID(pTaskInfo));
void* schema = taosHashGet(pInfo->pSchema, &pInfo->pCur->mr.me.uid, sizeof(int64_t));
if (schema == NULL) {
SSchemaWrapper* schemaWrapper = tCloneSSchemaWrapper(&pInfo->pCur->mr.me.stbEntry.schemaRow);
@ -559,7 +571,8 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
}
continue;
} else if (pInfo->pCur->mr.me.type == TSDB_CHILD_TABLE) {
qDebug("sysTableScanUserCols cursor get child table");
qDebug("sysTableScanUserCols cursor get child table, %s", GET_TASKID(pTaskInfo));
STR_TO_VARSTR(typeName, "CHILD_TABLE");
STR_TO_VARSTR(tableName, pInfo->pCur->mr.me.name);
int64_t suid = pInfo->pCur->mr.me.ctbEntry.suid;
@ -568,13 +581,14 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
schemaRow = *(SSchemaWrapper**)schema;
} else {
SMetaReader smrSuperTable = {0};
pAPI->metaReaderFn.initReader(&smrSuperTable, pInfo->readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
pAPI->metaReaderFn.initReader(&smrSuperTable, pInfo->readHandle.vnode, META_READER_NOLOCK, &pAPI->metaFn);
int code = pAPI->metaReaderFn.getTableEntryByUid(&smrSuperTable, suid);
if (code != TSDB_CODE_SUCCESS) {
// terrno has been set by pAPI->metaReaderFn.getTableEntryByName, therefore, return directly
qError("sysTableScanUserCols get meta by suid:%" PRId64 " error, code:%d", suid, code);
qError("sysTableScanUserCols get meta by suid:%" PRId64 " error, code:%d, %s", suid, code, GET_TASKID(pTaskInfo));
pAPI->metaReaderFn.clearReader(&smrSuperTable);
blockDataDestroy(dataBlock);
blockDataDestroy(pDataBlock);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
@ -584,34 +598,35 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
pAPI->metaReaderFn.clearReader(&smrSuperTable);
}
} else if (pInfo->pCur->mr.me.type == TSDB_NORMAL_TABLE) {
qDebug("sysTableScanUserCols cursor get normal table");
qDebug("sysTableScanUserCols cursor get normal table, %s", GET_TASKID(pTaskInfo));
schemaRow = &pInfo->pCur->mr.me.ntbEntry.schemaRow;
STR_TO_VARSTR(typeName, "NORMAL_TABLE");
STR_TO_VARSTR(tableName, pInfo->pCur->mr.me.name);
} else {
qDebug("sysTableScanUserCols cursor get invalid table");
qDebug("sysTableScanUserCols cursor get invalid table, %s", GET_TASKID(pTaskInfo));
continue;
}
if ((numOfRows + schemaRow->nCols) > pOperator->resultInfo.capacity) {
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, dataBlock, pOperator->exprSupp.pFilterInfo);
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, pDataBlock, pOperator->exprSupp.pFilterInfo);
numOfRows = 0;
pInfo->restore = true;
if (pInfo->pRes->info.rows > 0) {
pAPI->metaFn.pauseTableMetaCursor(pInfo->pCur);
break;
}
} else {
sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, dataBlock, tableName, schemaRow, typeName);
sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, pDataBlock, tableName, schemaRow, typeName);
}
}
if (numOfRows > 0) {
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, dataBlock, pOperator->exprSupp.pFilterInfo);
pAPI->metaFn.pauseTableMetaCursor(pInfo->pCur);
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, pDataBlock, pOperator->exprSupp.pFilterInfo);
numOfRows = 0;
}
blockDataDestroy(dataBlock);
blockDataDestroy(pDataBlock);
if (ret != 0) {
pAPI->metaFn.closeTableMetaCursor(pInfo->pCur);
pInfo->pCur = NULL;
@ -619,8 +634,7 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
}
pInfo->loadInfo.totalRows += pInfo->pRes->info.rows;
qDebug("sysTableScanUserCols get cols success, rows:%" PRIu64, pInfo->loadInfo.totalRows);
qDebug("get cols success, rows:%" PRIu64 " %s", pInfo->loadInfo.totalRows, GET_TASKID(pTaskInfo));
return (pInfo->pRes->info.rows == 0) ? NULL : pInfo->pRes;
}
@ -702,6 +716,8 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
int32_t ret = 0;
if (pInfo->pCur == NULL) {
pInfo->pCur = pAPI->metaFn.openTableMetaCursor(pInfo->readHandle.vnode);
} else {
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 0);
}
bool blockFull = false;
@ -727,7 +743,6 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
}
if ((smrSuperTable.me.stbEntry.schemaTag.nCols + numOfRows) > pOperator->resultInfo.capacity) {
pAPI->metaFn.cursorPrev(pInfo->pCur, TSDB_TABLE_MAX);
blockFull = true;
} else {
sysTableUserTagsFillOneTableTags(pInfo, &smrSuperTable, &pInfo->pCur->mr, dbname, tableName, &numOfRows,
@ -741,6 +756,7 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
numOfRows = 0;
if (pInfo->pRes->info.rows > 0) {
pAPI->metaFn.pauseTableMetaCursor(pInfo->pCur);
break;
}
@ -749,6 +765,7 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
}
if (numOfRows > 0) {
pAPI->metaFn.pauseTableMetaCursor(pInfo->pCur);
relocateAndFilterSysTagsScanResult(pInfo, numOfRows, dataBlock, pOperator->exprSupp.pFilterInfo);
numOfRows = 0;
}
@ -1329,7 +1346,7 @@ static SSDataBlock* sysTableBuildUserTables(SOperatorInfo* pOperator) {
firstMetaCursor = 1;
}
if (!firstMetaCursor) {
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0);
pAPI->metaFn.resumeTableMetaCursor(pInfo->pCur, 0, 1);
}
blockDataCleanup(pInfo->pRes);

22
source/libs/nodes/src/nodesCodeFuncs.c
View File

@ -209,6 +209,8 @@ const char* nodesNodeName(ENodeType type) {
return "ShowSnodesStmt";
case QUERY_NODE_SHOW_BNODES_STMT:
return "ShowBnodesStmt";
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
return "ShowArbGroupsStmt";
case QUERY_NODE_SHOW_CLUSTER_STMT:
return "ShowClusterStmt";
case QUERY_NODE_SHOW_DATABASES_STMT:
@ -266,7 +268,7 @@ const char* nodesNodeName(ENodeType type) {
case QUERY_NODE_SHOW_COMPACTS_STMT:
return "ShowCompactsStmt";
case QUERY_NODE_SHOW_COMPACT_DETAILS_STMT:
return "ShowCompactDetailsStmt";
return "ShowCompactDetailsStmt";
case QUERY_NODE_SHOW_GRANTS_FULL_STMT:
return "ShowGrantsFullStmt";
case QUERY_NODE_SHOW_GRANTS_LOGS_STMT:
@ -3178,16 +3180,16 @@ static int32_t physiGroupCacheNodeToJson(const void* pObj, SJson* pJson) {
int32_t code = physicPlanNodeToJson(pObj, pJson);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkGroupCachePhysiPlanGrpColsMayBeNull, pNode->grpColsMayBeNull);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkGroupCachePhysiPlanGroupByUid, pNode->grpByUid);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkGroupCachePhysiPlanGlobalGroup, pNode->globalGrp);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkGroupCachePhysiPlanBatchFetch, pNode->batchFetch);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkGroupCachePhysiPlanGroupCols, pNode->pGroupCols);
}
@ -6684,6 +6686,10 @@ static int32_t showQnodesStmtToJson(const void* pObj, SJson* pJson) { return sho
static int32_t jsonToShowQnodesStmt(const SJson* pJson, void* pObj) { return jsonToShowStmt(pJson, pObj); }
static int32_t showArbGroupsStmtToJson(const void* pObj, SJson* pJson) { return showStmtToJson(pObj, pJson); }
static int32_t jsonToShowArbGroupsStmt(const SJson* pJson, void* pObj) { return jsonToShowStmt(pJson, pObj); }
static int32_t showClusterStmtToJson(const void* pObj, SJson* pJson) { return showStmtToJson(pObj, pJson); }
static int32_t jsonToShowClusterStmt(const SJson* pJson, void* pObj) { return jsonToShowStmt(pJson, pObj); }
@ -7209,6 +7215,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
return showMnodesStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_QNODES_STMT:
return showQnodesStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
return showArbGroupsStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_CLUSTER_STMT:
return showClusterStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_DATABASES_STMT:
@ -7546,6 +7554,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToShowMnodesStmt(pJson, pObj);
case QUERY_NODE_SHOW_QNODES_STMT:
return jsonToShowQnodesStmt(pJson, pObj);
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
return jsonToShowArbGroupsStmt(pJson, pObj);
case QUERY_NODE_SHOW_CLUSTER_STMT:
return jsonToShowClusterStmt(pJson, pObj);
case QUERY_NODE_SHOW_DATABASES_STMT:
@ -7610,7 +7620,7 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToRestoreQnodeStmt(pJson, pObj);
case QUERY_NODE_RESTORE_MNODE_STMT:
return jsonToRestoreMnodeStmt(pJson, pObj);
case QUERY_NODE_RESTORE_VNODE_STMT:
case QUERY_NODE_RESTORE_VNODE_STMT:
return jsonToRestoreVnodeStmt(pJson, pObj);
case QUERY_NODE_LOGIC_PLAN_SCAN:
return jsonToLogicScanNode(pJson, pObj);

8
source/libs/nodes/src/nodesUtilFuncs.c
View File

@ -419,6 +419,7 @@ SNode* nodesMakeNode(ENodeType type) {
case QUERY_NODE_SHOW_QNODES_STMT:
case QUERY_NODE_SHOW_SNODES_STMT:
case QUERY_NODE_SHOW_BNODES_STMT:
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
case QUERY_NODE_SHOW_CLUSTER_STMT:
case QUERY_NODE_SHOW_DATABASES_STMT:
case QUERY_NODE_SHOW_FUNCTIONS_STMT:
@ -1072,6 +1073,7 @@ void nodesDestroyNode(SNode* pNode) {
case QUERY_NODE_SHOW_QNODES_STMT:
case QUERY_NODE_SHOW_SNODES_STMT:
case QUERY_NODE_SHOW_BNODES_STMT:
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
case QUERY_NODE_SHOW_CLUSTER_STMT:
case QUERY_NODE_SHOW_DATABASES_STMT:
case QUERY_NODE_SHOW_FUNCTIONS_STMT:
@ -1363,7 +1365,7 @@ void nodesDestroyNode(SNode* pNode) {
nodesDestroyList(pPhyNode->pOnRight);
nodesDestroyNode(pPhyNode->pFilterConditions);
nodesDestroyList(pPhyNode->pTargets);
nodesDestroyNode(pPhyNode->pPrimKeyCond);
nodesDestroyNode(pPhyNode->pColEqCond);
nodesDestroyNode(pPhyNode->pTagEqCond);
@ -1703,9 +1705,9 @@ void nodesListInsertListAfterPos(SNodeList* pTarget, SListCell* pPos, SNodeList*
pSrc->pTail->pNext = pPos->pNext;
pPos->pNext = pSrc->pHead;
pTarget->length += pSrc->length;
nodesFree(pSrc);
nodesFree(pSrc);
}
SNode* nodesListGetNode(SNodeList* pList, int32_t index) {

2
source/libs/parser/inc/parAst.h
View File

@ -196,7 +196,7 @@ SNode* setShowKind(SAstCreateContext* pCxt, SNode* pStmt, EShowKind showKind);
SNode* createShowStmt(SAstCreateContext* pCxt, ENodeType type);
SNode* createShowStmtWithCond(SAstCreateContext* pCxt, ENodeType type, SNode* pDbName, SNode* pTbName,
EOperatorType tableCondType);
SNode* createShowTablesStmt(SAstCreateContext* pCxt, SShowTablesOption option, SNode* pTbName, EOperatorType tableCondType);
SNode* createShowTablesStmt(SAstCreateContext* pCxt, SShowTablesOption option, SNode* pTbName, EOperatorType tableCondType);
SNode* createShowCreateDatabaseStmt(SAstCreateContext* pCxt, SToken* pDbName);
SNode* createShowAliveStmt(SAstCreateContext* pCxt, SNode* pDbName, ENodeType type);
SNode* createShowCreateTableStmt(SAstCreateContext* pCxt, ENodeType type, SNode* pRealTable);

3
source/libs/parser/inc/sql.y
View File

@ -479,6 +479,7 @@ cmd ::= SHOW db_name_cond_opt(A) VGROUPS.
cmd ::= SHOW MNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MNODES_STMT); }
//cmd ::= SHOW MODULES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_MODULES_STMT); }
cmd ::= SHOW QNODES. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_QNODES_STMT); }
cmd ::= SHOW ARBGROUPS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_ARBGROUPS_STMT); }
cmd ::= SHOW FUNCTIONS. { pCxt->pRootNode = createShowStmt(pCxt, QUERY_NODE_SHOW_FUNCTIONS_STMT); }
cmd ::= SHOW INDEXES FROM table_name_cond(A) from_db_opt(B). { pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_INDEXES_STMT, B, A, OP_TYPE_EQUAL); }
cmd ::= SHOW INDEXES FROM db_name(B) NK_DOT table_name(A). { pCxt->pRootNode = createShowStmtWithCond(pCxt, QUERY_NODE_SHOW_INDEXES_STMT, createIdentifierValueNode(pCxt, &B), createIdentifierValueNode(pCxt, &A), OP_TYPE_EQUAL); }
@ -530,7 +531,7 @@ table_kind_db_name_cond_opt(A) ::= db_name(C) NK_DOT.
table_kind_db_name_cond_opt(A) ::= table_kind(B) db_name(C) NK_DOT. { A.kind = B; A.dbName = C; }
%type table_kind { EShowKind }
%destructor table_kind { }
%destructor table_kind { }
table_kind(A) ::= NORMAL. { A = SHOW_KIND_TABLES_NORMAL; }
table_kind(A) ::= CHILD. { A = SHOW_KIND_TABLES_CHILD; }

11
source/libs/parser/src/parAstCreater.c
View File

@ -288,7 +288,7 @@ SNode* releaseRawExprNode(SAstCreateContext* pCxt, SNode* pNode) {
} else if (pRawExpr->isPseudoColumn) {
// all pseudo column are translate to function with same name
strcpy(pExpr->userAlias, ((SFunctionNode*)pExpr)->functionName);
strcpy(pExpr->aliasName, ((SFunctionNode*)pExpr)->functionName);
strcpy(pExpr->aliasName, ((SFunctionNode*)pExpr)->functionName);
} else {
int32_t len = TMIN(sizeof(pExpr->aliasName) - 1, pRawExpr->n);
@ -1189,11 +1189,11 @@ SNode* createSelectStmt(SAstCreateContext* pCxt, bool isDistinct, SNodeList* pPr
return select;
}
SNode* setSelectStmtTagMode(SAstCreateContext* pCxt, SNode* pStmt, bool bSelectTags) {
SNode* setSelectStmtTagMode(SAstCreateContext* pCxt, SNode* pStmt, bool bSelectTags) {
if (pStmt && QUERY_NODE_SELECT_STMT == nodeType(pStmt)) {
if (pCxt->pQueryCxt->biMode) {
((SSelectStmt*)pStmt)->tagScan = true;
} else {
} else {
((SSelectStmt*)pStmt)->tagScan = bSelectTags;
}
}
@ -1263,6 +1263,7 @@ SNode* createDefaultDatabaseOptions(SAstCreateContext* pCxt) {
pOptions->sstTrigger = TSDB_DEFAULT_SST_TRIGGER;
pOptions->tablePrefix = TSDB_DEFAULT_HASH_PREFIX;
pOptions->tableSuffix = TSDB_DEFAULT_HASH_SUFFIX;
pOptions->withArbitrator = TSDB_DEFAULT_DB_WITH_ARBITRATOR;
return (SNode*)pOptions;
}
@ -1298,6 +1299,7 @@ SNode* createAlterDatabaseOptions(SAstCreateContext* pCxt) {
pOptions->sstTrigger = -1;
pOptions->tablePrefix = -1;
pOptions->tableSuffix = -1;
pOptions->withArbitrator = -1;
return (SNode*)pOptions;
}
@ -1353,6 +1355,7 @@ static SNode* setDatabaseOptionImpl(SAstCreateContext* pCxt, SNode* pOptions, ED
break;
case DB_OPTION_REPLICA:
pDbOptions->replica = taosStr2Int8(((SToken*)pVal)->z, NULL, 10);
pDbOptions->withArbitrator = (pDbOptions->replica == 2);
if (!alter) {
updateWalOptionsDefault(pDbOptions);
}
@ -1783,7 +1786,7 @@ SNode* createShowCompactsStmt(SAstCreateContext* pCxt, ENodeType type) {
SNode* setShowKind(SAstCreateContext* pCxt, SNode* pStmt, EShowKind showKind) {
if (pStmt == NULL) {
return NULL;
}
}
SShowStmt* pShow = (SShowStmt*)pStmt;
pShow->showKind = showKind;
return pStmt;

14
source/libs/parser/src/parAstParser.c
View File

@ -170,7 +170,7 @@ static int32_t collectMetaKeyFromRealTableImpl(SCollectMetaKeyCxt* pCxt, const c
code = reserveUserAuthInCache(pCxt->pParseCxt->acctId, pCxt->pParseCxt->pEffectiveUser, pDb, pTable, authType,
pCxt->pMetaCache);
}
#endif
#endif
if (TSDB_CODE_SUCCESS == code) {
code = reserveDbVgInfoInCache(pCxt->pParseCxt->acctId, pDb, pCxt->pMetaCache);
}
@ -459,6 +459,14 @@ static int32_t collectMetaKeyFromShowBnodes(SCollectMetaKeyCxt* pCxt, SShowStmt*
return TSDB_CODE_SUCCESS;
}
static int32_t collectMetaKeyFromShowArbGroups(SCollectMetaKeyCxt* pCxt, SShowStmt* pStmt) {
if (pCxt->pParseCxt->enableSysInfo) {
return reserveTableMetaInCache(pCxt->pParseCxt->acctId, TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_ARBGROUPS,
pCxt->pMetaCache);
}
return TSDB_CODE_SUCCESS;
}
static int32_t collectMetaKeyFromShowCluster(SCollectMetaKeyCxt* pCxt, SShowStmt* pStmt) {
if (pCxt->pParseCxt->enableSysInfo) {
return reserveTableMetaInCache(pCxt->pParseCxt->acctId, TSDB_INFORMATION_SCHEMA_DB, TSDB_INS_TABLE_CLUSTER,
@ -668,7 +676,7 @@ static int32_t collectMetaKeyFromShowCreateView(SCollectMetaKeyCxt* pCxt, SShowC
if (TSDB_CODE_SUCCESS == code) {
code = reserveTableMetaInCache(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->viewName, pCxt->pMetaCache);
}
return code;
}
@ -806,6 +814,8 @@ static int32_t collectMetaKeyFromQuery(SCollectMetaKeyCxt* pCxt, SNode* pStmt) {
return collectMetaKeyFromShowSnodes(pCxt, (SShowStmt*)pStmt);
case QUERY_NODE_SHOW_BNODES_STMT:
return collectMetaKeyFromShowBnodes(pCxt, (SShowStmt*)pStmt);
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
return collectMetaKeyFromShowArbGroups(pCxt, (SShowStmt*)pStmt);
case QUERY_NODE_SHOW_CLUSTER_STMT:
return collectMetaKeyFromShowCluster(pCxt, (SShowStmt*)pStmt);
case QUERY_NODE_SHOW_DATABASES_STMT:

5
source/libs/parser/src/parAuthenticator.c
View File

@ -41,7 +41,7 @@ static void setUserAuthInfo(SParseContext* pCxt, const char* pDbName, const char
} else {
snprintf(pAuth->user, sizeof(pAuth->user), "%s", pCxt->pUser);
}
if (NULL == pTabName) {
tNameSetDbName(&pAuth->tbName, pCxt->acctId, pDbName, strlen(pDbName));
} else {
@ -173,7 +173,7 @@ static EDealRes authSelectImpl(SNode* pNode, void* pContext) {
isView = true;
}
taosMemoryFree(pTableMeta);
#endif
#endif
if (!isView) {
pAuthCxt->errCode = checkAuth(pAuthCxt, pTable->dbName, pTable->tableName, AUTH_TYPE_READ, &pTagCond);
if (TSDB_CODE_SUCCESS != pAuthCxt->errCode && NULL != pAuthCxt->pParseCxt->pEffectiveUser) {
@ -355,6 +355,7 @@ static int32_t authQuery(SAuthCxt* pCxt, SNode* pStmt) {
case QUERY_NODE_SHOW_GRANTS_FULL_STMT:
case QUERY_NODE_SHOW_GRANTS_LOGS_STMT:
case QUERY_NODE_SHOW_CLUSTER_MACHINES_STMT:
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
return !pCxt->pParseCxt->enableSysInfo ? TSDB_CODE_PAR_PERMISSION_DENIED : TSDB_CODE_SUCCESS;
case QUERY_NODE_SHOW_TABLES_STMT:
case QUERY_NODE_SHOW_STABLES_STMT:

2
source/libs/parser/src/parInsertSql.c
View File

@ -969,7 +969,7 @@ static int32_t parseTagsClauseImpl(SInsertParseContext* pCxt, SVnodeModifyOpStmt
}
_exit:
for (int32_t i = 0; i < TARRAY_SIZE(pTagVals); ++i) {
for (int32_t i = 0; i < taosArrayGetSize(pTagVals); ++i) {
STagVal* p = (STagVal*)TARRAY_GET_ELEM(pTagVals, i);
if (IS_VAR_DATA_TYPE(p->type)) {
taosMemoryFreeClear(p->pData);

1
source/libs/parser/src/parTokenizer.c
View File

@ -306,6 +306,7 @@ static SKeyword keywordTable[] = {
{"ALIVE", TK_ALIVE},
{"VARBINARY", TK_VARBINARY},
{"KEEP_TIME_OFFSET", TK_KEEP_TIME_OFFSET},
{"ARBGROUPS", TK_ARBGROUPS},
};
// clang-format on

98
source/libs/parser/src/parTranslater.c
View File

@ -100,6 +100,13 @@ static const SSysTableShowAdapter sysTableShowAdapter[] = {
.numOfShowCols = 1,
.pShowCols = {"*"}
},
{
.showType = QUERY_NODE_SHOW_ARBGROUPS_STMT,
.pDbName = TSDB_INFORMATION_SCHEMA_DB,
.pTableName = TSDB_INS_TABLE_ARBGROUPS,
.numOfShowCols = 1,
.pShowCols = {"*"}
},
{
.showType = QUERY_NODE_SHOW_CLUSTER_STMT,
.pDbName = TSDB_INFORMATION_SCHEMA_DB,
@ -263,7 +270,7 @@ static const SSysTableShowAdapter sysTableShowAdapter[] = {
.pTableName = TSDB_INS_TABLE_COMPACTS,
.numOfShowCols = 1,
.pShowCols = {"*"}
},
},
{ .showType = QUERY_NODE_SHOW_COMPACT_DETAILS_STMT,
.pDbName = TSDB_INFORMATION_SCHEMA_DB,
.pTableName = TSDB_INS_TABLE_COMPACT_DETAILS,
@ -380,7 +387,7 @@ static int32_t getViewMetaImpl(SParseContext* pParCxt, SParseMetaCache* pMetaCac
#endif
int32_t code = TSDB_CODE_SUCCESS;
if (pParCxt->async) {
code = getViewMetaFromCache(pMetaCache, pName, pMeta);
} else {
@ -390,7 +397,7 @@ static int32_t getViewMetaImpl(SParseContext* pParCxt, SParseMetaCache* pMetaCac
.mgmtEps = pParCxt->mgmtEpSet};
code = catalogGetViewMeta(pParCxt->pCatalog, &conn, pName, pMeta);
}
if (TSDB_CODE_SUCCESS != code && TSDB_CODE_PAR_TABLE_NOT_EXIST != code) {
parserError("0x%" PRIx64 " catalogGetViewMeta error, code:%s, dbName:%s, viewName:%s", pParCxt->requestId,
tstrerror(code), pName->dbname, pName->tname);
@ -401,7 +408,7 @@ static int32_t getViewMetaImpl(SParseContext* pParCxt, SParseMetaCache* pMetaCac
int32_t getTargetMetaImpl(SParseContext* pParCxt, SParseMetaCache* pMetaCache, const SName* pName, STableMeta** pMeta, bool couldBeView) {
int32_t code = TSDB_CODE_SUCCESS;
if (pParCxt->async) {
code = getTableMetaFromCache(pMetaCache, pName, pMeta);
#ifdef TD_ENTERPRISE
@ -420,7 +427,7 @@ int32_t getTargetMetaImpl(SParseContext* pParCxt, SParseMetaCache* pMetaCache, c
.mgmtEps = pParCxt->mgmtEpSet};
code = catalogGetTableMeta(pParCxt->pCatalog, &conn, pName, pMeta);
}
if (TSDB_CODE_SUCCESS != code && TSDB_CODE_PAR_TABLE_NOT_EXIST != code) {
parserError("0x%" PRIx64 " catalogGetTableMeta error, code:%s, dbName:%s, tbName:%s", pParCxt->requestId,
tstrerror(code), pName->dbname, pName->tname);
@ -960,7 +967,7 @@ static void setColumnInfoByExpr(STempTableNode* pTable, SExprNode* pExpr, SColum
assNode.pPlace = (SNode**)pColRef;
assNode.pAssociationNode = (SNode*)*pColRef;
taosArrayPush(pExpr->pAssociation, &assNode);
strcpy(pCol->tableAlias, pTable->table.tableAlias);
pCol->colId = isPrimaryKey(pTable, (SNode*)pExpr) ? PRIMARYKEY_TIMESTAMP_COL_ID : 0;
strcpy(pCol->colName, pExpr->aliasName);
@ -1177,8 +1184,8 @@ static SNode* biMakeTbnameProjectAstNode(char* funcName, char* tableAlias) {
if (valNode != NULL) {
nodesListMakeAppend(&tbNameFunc->pParameterList, (SNode*)valNode);
}
snprintf(tbNameFunc->node.userAlias, sizeof(tbNameFunc->node.userAlias),
(tableAlias)? "%s.tbname" : "%stbname",
snprintf(tbNameFunc->node.userAlias, sizeof(tbNameFunc->node.userAlias),
(tableAlias)? "%s.tbname" : "%stbname",
(tableAlias)? tableAlias : "");
strncpy(tbNameFunc->node.aliasName, tbNameFunc->functionName, TSDB_COL_NAME_LEN);
@ -1190,13 +1197,13 @@ static SNode* biMakeTbnameProjectAstNode(char* funcName, char* tableAlias) {
nodesListMakeAppend(&multiResFunc->pParameterList, (SNode*)tbNameFunc);
if (tsKeepColumnName) {
snprintf(multiResFunc->node.userAlias, sizeof(tbNameFunc->node.userAlias),
(tableAlias)? "%s.tbname" : "%stbname",
snprintf(multiResFunc->node.userAlias, sizeof(tbNameFunc->node.userAlias),
(tableAlias)? "%s.tbname" : "%stbname",
(tableAlias)? tableAlias : "");
strcpy(multiResFunc->node.aliasName, tbNameFunc->functionName);
} else {
snprintf(multiResFunc->node.userAlias, sizeof(multiResFunc->node.userAlias),
tableAlias? "%s(%s.tbname)" : "%s(%stbname)", funcName,
snprintf(multiResFunc->node.userAlias, sizeof(multiResFunc->node.userAlias),
tableAlias? "%s(%s.tbname)" : "%s(%stbname)", funcName,
tableAlias? tableAlias: "");
biMakeAliasNameInMD5(multiResFunc->node.userAlias, strlen(multiResFunc->node.userAlias), multiResFunc->node.aliasName);
}
@ -1209,7 +1216,7 @@ static int32_t biRewriteSelectFuncParamStar(STranslateContext* pCxt, SSelectStmt
SNodeList* pTbnameNodeList = nodesMakeList();
SFunctionNode* pFunc = (SFunctionNode*)pNode;
if (strcasecmp(pFunc->functionName, "last") == 0 ||
if (strcasecmp(pFunc->functionName, "last") == 0 ||
strcasecmp(pFunc->functionName, "last_row") == 0 ||
strcasecmp(pFunc->functionName, "first") == 0) {
SNodeList* pParams = pFunc->pParameterList;
@ -1259,8 +1266,8 @@ int32_t biRewriteSelectStar(STranslateContext* pCxt, SSelectStmt* pSelect) {
size_t n = taosArrayGetSize(pTables);
for (int32_t i = 0; i < n; ++i) {
STableNode* pTable = taosArrayGetP(pTables, i);
if (nodeType(pTable) == QUERY_NODE_REAL_TABLE &&
((SRealTableNode*)pTable)->pMeta != NULL &&
if (nodeType(pTable) == QUERY_NODE_REAL_TABLE &&
((SRealTableNode*)pTable)->pMeta != NULL &&
((SRealTableNode*)pTable)->pMeta->tableType == TSDB_SUPER_TABLE) {
SNode* pTbnameNode = biMakeTbnameProjectAstNode(NULL, NULL);
nodesListAppend(pTbnameNodeList, pTbnameNode);
@ -1273,9 +1280,9 @@ int32_t biRewriteSelectStar(STranslateContext* pCxt, SSelectStmt* pSelect) {
char* pTableAlias = ((SColumnNode*)pNode)->tableAlias;
STableNode* pTable = NULL;
int32_t code = findTable(pCxt, pTableAlias, &pTable);
if (TSDB_CODE_SUCCESS == code &&
if (TSDB_CODE_SUCCESS == code &&
nodeType(pTable) == QUERY_NODE_REAL_TABLE &&
((SRealTableNode*)pTable)->pMeta != NULL &&
((SRealTableNode*)pTable)->pMeta != NULL &&
((SRealTableNode*)pTable)->pMeta->tableType == TSDB_SUPER_TABLE) {
SNode* pTbnameNode = biMakeTbnameProjectAstNode(NULL, pTableAlias);
nodesListAppend(pTbnameNodeList, pTbnameNode);
@ -1967,7 +1974,7 @@ static int32_t translateInterpPseudoColumnFunc(STranslateContext* pCxt, SNode**
return code;
}
translateColumn(pCxt, (SColumnNode**)ppNode);
return pCxt->errCode;
return pCxt->errCode;
}
return TSDB_CODE_SUCCESS;
}
@ -2272,7 +2279,7 @@ static int32_t replacePsedudoColumnFuncWithColumn(STranslateContext* pCxt, SNode
nodesDestroyNode(*ppNode);
*ppNode = (SNode*)pCol;
return TSDB_CODE_SUCCESS;
return TSDB_CODE_SUCCESS;
}
}
pCol->node.resType = pOldExpr->resType;
@ -2296,14 +2303,14 @@ static int32_t rewriteToColumnAndRetranslate(STranslateContext* pCxt, SNode** pp
return generateSyntaxErrMsg(&pCxt->msgBuf, errCode);
} else {
return TSDB_CODE_SUCCESS;
}
}
}
static int32_t translateWindowPseudoColumnFunc(STranslateContext* pCxt, SNode** ppNode, bool* pRewriteToColumn) {
SFunctionNode* pFunc = (SFunctionNode*)(*ppNode);
if (!fmIsWindowPseudoColumnFunc(pFunc->funcId)) {
return TSDB_CODE_SUCCESS;
}
}
if (!isSelectStmt(pCxt->pCurrStmt)) {
return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_WINDOW_PC);
}
@ -2337,7 +2344,7 @@ static int32_t translateScanPseudoColumnFunc(STranslateContext* pCxt, SNode** pp
pCxt->errCode = findTable(pCxt, pVal->literal, &pTable);
if (TSDB_CODE_SUCCESS != pCxt->errCode || (NULL == pTable)) {
return generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_TBNAME);
}
}
if (nodeType(pTable) != QUERY_NODE_REAL_TABLE) {
*pRewriteToColumn = true;
return rewriteToColumnAndRetranslate(pCxt, ppNode, TSDB_CODE_PAR_INVALID_TBNAME);
@ -2832,7 +2839,7 @@ static int32_t checkAggColCoexist(STranslateContext* pCxt, SSelectStmt* pSelect)
if (!pSelect->isDistinct) {
nodesRewriteExprs(pSelect->pOrderByList, doCheckAggColCoexist, &cxt);
}
if (((!cxt.existCol && 0 < pSelect->selectFuncNum) || (cxt.existCol && 1 == pSelect->selectFuncNum) )
if (((!cxt.existCol && 0 < pSelect->selectFuncNum) || (cxt.existCol && 1 == pSelect->selectFuncNum) )
&& !pSelect->hasOtherVectorFunc) {
return rewriteColsToSelectValFunc(pCxt, pSelect);
}
@ -4377,7 +4384,7 @@ static bool findEqCondTbNameInOperatorNode(STranslateContext* pCxt, SNode* pWher
} else {
code = findTable(pCxt, pTableAlias, &pTable);
}
if (code == TSDB_CODE_SUCCESS && nodeType(pTable) == QUERY_NODE_REAL_TABLE &&
if (code == TSDB_CODE_SUCCESS && nodeType(pTable) == QUERY_NODE_REAL_TABLE &&
((SRealTableNode*)pTable)->pMeta && ((SRealTableNode*)pTable)->pMeta->tableType == TSDB_SUPER_TABLE) {
pInfo->pRealTable = (SRealTableNode*)pTable;
return true;
@ -4413,7 +4420,7 @@ static void findEqualCondTbnameInLogicCondAnd(STranslateContext* pCxt, SNode* pW
}
}
}
//TODO: logic cond
//TODO: logic cond
}
}
@ -4533,7 +4540,7 @@ static int32_t setEqualTbnameTableVgroups(STranslateContext* pCxt, SSelectStmt*
pInfo->pRealTable->pVgroupList = vgsInfo;
} else {
taosMemoryFree(vgsInfo);
}
}
}
return TSDB_CODE_SUCCESS;
}
@ -5111,7 +5118,7 @@ static int32_t translateInsertTable(STranslateContext* pCxt, SNode** pTable) {
if (TSDB_CODE_SUCCESS == code && TSDB_CHILD_TABLE != ((SRealTableNode*)*pTable)->pMeta->tableType &&
TSDB_NORMAL_TABLE != ((SRealTableNode*)*pTable)->pMeta->tableType) {
code = buildInvalidOperationMsg(&pCxt->msgBuf, "insert data into super table is not supported");
}
return code;
}
@ -5208,6 +5215,7 @@ static int32_t buildCreateDbReq(STranslateContext* pCxt, SCreateDatabaseStmt* pS
pReq->tsdbPageSize = pStmt->pOptions->tsdbPageSize;
pReq->keepTimeOffset = pStmt->pOptions->keepTimeOffset;
pReq->ignoreExist = pStmt->ignoreExists;
pReq->withArbitrator = pStmt->pOptions->withArbitrator;
return buildCreateDbRetentions(pStmt->pOptions->pRetentions, pReq);
}
@ -5445,7 +5453,7 @@ static int32_t checkDbRetentionsOption(STranslateContext* pCxt, SNodeList* pRete
"Invalid option retentions(freq/keep): %s should larger than %s", pKeep->literal,
pFreq->literal);
}
if (NULL != pPrevFreq && pPrevFreq->datum.i >= pFreq->datum.i) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_DB_OPTION,
"Invalid option retentions(freq): %s should larger than %s", pFreq->literal,
@ -5509,6 +5517,12 @@ static int32_t checkOptionsDependency(STranslateContext* pCxt, const char* pDbNa
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_DB_OPTION,
"Invalid duration value, should be keep2 >= keep1 >= keep0 >= 3 * duration");
}
if ((pOptions->replica == 2) ^ (pOptions->withArbitrator == TSDB_MAX_DB_WITH_ARBITRATOR)) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_DB_OPTION,
"Invalid database option, with_arbitrator should be used with replica 2");
}
return TSDB_CODE_SUCCESS;
}
@ -5561,7 +5575,7 @@ static int32_t checkDatabaseOptions(STranslateContext* pCxt, const char* pDbName
TSDB_MAX_TSDB_PAGESIZE);
}
if (TSDB_CODE_SUCCESS == code) {
code = checkDbEnumOption(pCxt, "replications", pOptions->replica, TSDB_MIN_DB_REPLICA, TSDB_MAX_DB_REPLICA);
code = checkDbRangeOption(pCxt, "replications", pOptions->replica, TSDB_MIN_DB_REPLICA, TSDB_MAX_DB_REPLICA);
}
if (TSDB_CODE_SUCCESS == code) {
code = checkDbStrictOption(pCxt, pOptions);
@ -5600,6 +5614,10 @@ static int32_t checkDatabaseOptions(STranslateContext* pCxt, const char* pDbName
if (TSDB_CODE_SUCCESS == code) {
code = checkDbRangeOption(pCxt, "sstTrigger", pOptions->sstTrigger, TSDB_MIN_STT_TRIGGER, TSDB_MAX_STT_TRIGGER);
}
if (TSDB_CODE_SUCCESS == code) {
code = checkDbEnumOption(pCxt, "withArbitrator", pOptions->withArbitrator, TSDB_MIN_DB_WITH_ARBITRATOR,
TSDB_MAX_DB_WITH_ARBITRATOR);
}
if (TSDB_CODE_SUCCESS == code) {
code = checkDbTbPrefixSuffixOptions(pCxt, pOptions->tablePrefix, pOptions->tableSuffix);
}
@ -5812,6 +5830,7 @@ static void buildAlterDbReq(STranslateContext* pCxt, SAlterDatabaseStmt* pStmt,
pReq->minRows = pStmt->pOptions->minRowsPerBlock;
pReq->walRetentionPeriod = pStmt->pOptions->walRetentionPeriod;
pReq->walRetentionSize = pStmt->pOptions->walRetentionSize;
pReq->withArbitrator = pStmt->pOptions->withArbitrator;
return;
}
@ -7566,7 +7585,7 @@ static int32_t translateDescribe(STranslateContext* pCxt, SDescribeStmt* pStmt)
taosMemoryFree(res.schemaRes.pSchema);
}
}
#endif
#endif
return code;
}
@ -7667,8 +7686,8 @@ static int32_t checkCreateStream(STranslateContext* pCxt, SCreateStreamStmt* pSt
if (TSDB_VIEW_TABLE == tableType) {
return generateSyntaxErrMsgExt(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_STREAM_QUERY, "Unsupported stream query");
}
#endif
#endif
return TSDB_CODE_SUCCESS;
}
@ -8671,7 +8690,7 @@ static int32_t translateCreateView(STranslateContext* pCxt, SCreateViewStmt* pSt
tNameGetFullDbName(&name, dbFName);
int32_t code = validateCreateView(pCxt, pStmt);
if (TSDB_CODE_SUCCESS == code) {
if (TSDB_CODE_SUCCESS == code) {
code = (*pCxt->pParseCxt->parseSqlFp)(pCxt->pParseCxt->parseSqlParam, pStmt->dbName, pStmt->pQuerySql, false, NULL, &res);
}
if (TSDB_CODE_SUCCESS == code) {
@ -8694,7 +8713,7 @@ static int32_t translateCreateView(STranslateContext* pCxt, SCreateViewStmt* pSt
if (TSDB_CODE_SUCCESS == code) {
code = buildCmdMsg(pCxt, TDMT_MND_CREATE_VIEW, (FSerializeFunc)tSerializeSCMCreateViewReq, &pStmt->createReq);
}
tFreeSCMCreateViewReq(&pStmt->createReq);
return code;
}
@ -8722,7 +8741,7 @@ static int32_t translateDropView(STranslateContext* pCxt, SDropViewStmt* pStmt)
if (TSDB_CODE_SUCCESS != code) {
return code;
}
return buildCmdMsg(pCxt, TDMT_MND_DROP_VIEW, (FSerializeFunc)tSerializeSCMDropViewReq, &dropReq);
}
@ -8915,7 +8934,7 @@ static int32_t translateRevoke(STranslateContext* pCxt, SRevokeStmt* pStmt) {
taosMemoryFree(pTableMeta);
}
#endif
strcpy(req.user, pStmt->userName);
sprintf(req.objname, "%d.%s", pCxt->pParseCxt->acctId, pStmt->objName);
sprintf(req.tabName, "%s", pStmt->tabName);
@ -9024,7 +9043,7 @@ static int32_t translateShowCreateTable(STranslateContext* pCxt, SShowCreateTabl
static int32_t translateShowCreateView(STranslateContext* pCxt, SShowCreateViewStmt* pStmt) {
#ifndef TD_ENTERPRISE
return TSDB_CODE_OPS_NOT_SUPPORT;
#else
#else
SName name;
toName(pCxt->pParseCxt->acctId, pStmt->dbName, pStmt->viewName, &name);
return getViewMetaFromMetaCache(pCxt, &name, (SViewMeta**)&pStmt->pViewMeta);
@ -9439,7 +9458,7 @@ int32_t extractResultSchema(const SNode* pRoot, int32_t* numOfCols, SSchema** pS
case QUERY_NODE_SHOW_VARIABLES_STMT:
return extractShowVariablesResultSchema(numOfCols, pSchema);
case QUERY_NODE_COMPACT_DATABASE_STMT:
return extractCompactDbResultSchema(numOfCols, pSchema);
return extractCompactDbResultSchema(numOfCols, pSchema);
default:
break;
}
@ -10926,7 +10945,7 @@ static int32_t rewriteShowCompactDetailsStmt(STranslateContext* pCxt, SQuery* pQ
nodesDestroyNode(pQuery->pRoot);
pQuery->pRoot = (SNode*)pStmt;
}
return code;
return code;
}
static int32_t rewriteQuery(STranslateContext* pCxt, SQuery* pQuery) {
@ -10960,6 +10979,7 @@ static int32_t rewriteQuery(STranslateContext* pCxt, SQuery* pQuery) {
case QUERY_NODE_SHOW_GRANTS_FULL_STMT:
case QUERY_NODE_SHOW_GRANTS_LOGS_STMT:
case QUERY_NODE_SHOW_CLUSTER_MACHINES_STMT:
case QUERY_NODE_SHOW_ARBGROUPS_STMT:
code = rewriteShow(pCxt, pQuery);
break;
case QUERY_NODE_SHOW_VGROUPS_STMT:

8042
source/libs/parser/src/sql.c
View File

File diff suppressed because it is too large Load Diff

2
source/libs/parser/test/parAlterToBalanceTest.cpp
View File

@ -337,7 +337,7 @@ TEST_F(ParserInitialATest, alterDatabaseSemanticCheck) {
run("ALTER DATABASE test PAGES 63", TSDB_CODE_PAR_INVALID_DB_OPTION);
run("ALTER DATABASE test WAL_LEVEL 0", TSDB_CODE_PAR_INVALID_DB_OPTION);
run("ALTER DATABASE test WAL_LEVEL 3", TSDB_CODE_PAR_INVALID_DB_OPTION);
run("ALTER DATABASE test REPLICA 2", TSDB_CODE_PAR_INVALID_DB_OPTION);
//run("ALTER DATABASE test REPLICA 2", TSDB_CODE_PAR_INVALID_DB_OPTION);
run("ALTER DATABASE test STT_TRIGGER 0", TSDB_CODE_PAR_INVALID_DB_OPTION);
run("ALTER DATABASE test STT_TRIGGER 17", TSDB_CODE_PAR_INVALID_DB_OPTION);
// Regardless of the specific sentence

7
source/libs/planner/src/planOptimizer.c
View File

@ -2096,6 +2096,13 @@ static bool eliminateProjOptMayBeOptimized(SLogicNode* pNode) {
TSDB_SUPER_TABLE == ((SScanLogicNode*)nodesListGetNode(pNode->pChildren, 0))->tableType)) {
return false;
}
if (QUERY_NODE_LOGIC_PLAN_DYN_QUERY_CTRL == nodeType(nodesListGetNode(pNode->pChildren, 0))) {
SLogicNode* pChild = (SLogicNode*)nodesListGetNode(pNode->pChildren, 0);
if(LIST_LENGTH(pChild->pTargets) != LIST_LENGTH(pNode->pTargets)) {
return false;
}
}
SProjectLogicNode* pProjectNode = (SProjectLogicNode*)pNode;
if (NULL != pProjectNode->node.pLimit || NULL != pProjectNode->node.pSlimit ||

6
source/libs/stream/src/streamMeta.c
View File

@ -1128,7 +1128,11 @@ static int32_t metaHeartbeatToMnodeImpl(SStreamMeta* pMeta) {
}
if ((*pTask)->exec.pWalReader != NULL) {
entry.processedVer = (*pTask)->chkInfo.nextProcessVer - 1;
entry.processedVer = walReaderGetCurrentVer((*pTask)->exec.pWalReader) - 1;
if (entry.processedVer < 0) {
entry.processedVer = (*pTask)->chkInfo.processedVer;
}
walReaderValidVersionRange((*pTask)->exec.pWalReader, &entry.verStart, &entry.verEnd);
}

2
source/libs/sync/inc/syncCommit.h
View File

@ -53,6 +53,8 @@ bool syncNodeAgreedUpon(SSyncNode* pNode, SyncIndex index);
int64_t syncNodeUpdateCommitIndex(SSyncNode* ths, SyncIndex commitIndex);
int64_t syncNodeCheckCommitIndex(SSyncNode* ths, SyncIndex indexLikely);
int64_t syncNodeUpdateAssignedCommitIndex(SSyncNode* ths, SyncIndex assignedCommitIndex);
#ifdef __cplusplus
}
#endif

10
source/libs/sync/inc/syncInt.h
View File

@ -160,6 +160,13 @@ typedef struct SSyncNode {
SSyncLogStore* pLogStore;
SyncIndex commitIndex;
// assigned leader log vars
SyncIndex assignedCommitIndex;
SyncTerm arbTerm;
TdThreadMutex arbTokenMutex;
char arbToken[TSDB_ARB_TOKEN_SIZE];
// timer ms init
int32_t pingBaseLine;
int32_t electBaseLine;
@ -219,6 +226,7 @@ typedef struct SSyncNode {
int32_t electNum;
int32_t becomeLeaderNum;
int32_t becomeAssignedLeaderNum;
int32_t configChangeNum;
int32_t hbSlowNum;
int32_t hbrSlowNum;
@ -282,10 +290,12 @@ void syncNodeStepDown(SSyncNode* pSyncNode, SyncTerm newTerm);
void syncNodeBecomeFollower(SSyncNode* pSyncNode, const char* debugStr);
void syncNodeBecomeLearner(SSyncNode* pSyncNode, const char* debugStr);
void syncNodeBecomeLeader(SSyncNode* pSyncNode, const char* debugStr);
void syncNodeBecomeAssignedLeader(SSyncNode* pSyncNode);
void syncNodeCandidate2Leader(SSyncNode* pSyncNode);
void syncNodeFollower2Candidate(SSyncNode* pSyncNode);
void syncNodeLeader2Follower(SSyncNode* pSyncNode);
void syncNodeCandidate2Follower(SSyncNode* pSyncNode);
int32_t syncNodeAssignedLeader2Leader(SSyncNode* pSyncNode);
// raft vote --------------
void syncNodeVoteForTerm(SSyncNode* pSyncNode, SyncTerm term, SRaftId* pRaftId);

2
source/libs/sync/inc/syncUtil.h
View File

@ -78,6 +78,8 @@ void syncUtilMsgHtoN(void* msg);
bool syncUtilUserPreCommit(tmsg_t msgType);
bool syncUtilUserRollback(tmsg_t msgType);
void syncUtilGenerateArbToken(int32_t nodeId, int32_t groupId, char* buf);
void syncPrintNodeLog(const char* flags, ELogLevel level, int32_t dflag, SSyncNode* pNode, const char* format, ...);
void syncPrintSnapshotSenderLog(const char* flags, ELogLevel level, int32_t dflag, SSyncSnapshotSender* pSender,
const char* format, ...);

24
source/libs/sync/src/syncAppendEntriesReply.c
View File

@ -55,7 +55,7 @@ int32_t syncNodeOnAppendEntriesReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
return 0;
}
if (ths->state == TAOS_SYNC_STATE_LEADER) {
if (ths->state == TAOS_SYNC_STATE_LEADER || ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
if (pMsg->term > raftStoreGetTerm(ths)) {
syncLogRecvAppendEntriesReply(ths, pMsg, "error term");
syncNodeStepDown(ths, pMsg->term);
@ -76,7 +76,27 @@ int32_t syncNodeOnAppendEntriesReply(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
// commit if needed
SyncIndex indexLikely = TMIN(pMsg->matchIndex, ths->pLogBuf->matchIndex);
SyncIndex commitIndex = syncNodeCheckCommitIndex(ths, indexLikely);
(void)syncLogBufferCommit(ths->pLogBuf, ths, commitIndex);
if (ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
if (commitIndex >= ths->assignedCommitIndex) {
terrno = TSDB_CODE_SUCCESS;
raftStoreNextTerm(ths);
if (terrno != TSDB_CODE_SUCCESS) {
sError("vgId:%d, failed to update term, reason:%s", ths->vgId, tstrerror(terrno));
return -1;
}
if (syncNodeAssignedLeader2Leader(ths) != 0) {
sError("vgId:%d, failed to change state from assigned leader to leader", ths->vgId);
return -1;
}
taosThreadMutexLock(&ths->arbTokenMutex);
syncUtilGenerateArbToken(ths->myNodeInfo.nodeId, ths->vgId, ths->arbToken);
sInfo("vgId:%d, assigned leader to leader, arbToken:%s", ths->vgId, ths->arbToken);
taosThreadMutexUnlock(&ths->arbTokenMutex);
}
} else {
(void)syncLogBufferCommit(ths->pLogBuf, ths, commitIndex);
}
}
// replicate log

8
source/libs/sync/src/syncCommit.c
View File

@ -88,3 +88,11 @@ int64_t syncNodeCheckCommitIndex(SSyncNode* ths, SyncIndex indexLikely) {
}
return ths->commitIndex;
}
int64_t syncNodeUpdateAssignedCommitIndex(SSyncNode* ths, SyncIndex assignedCommitIndex) {
SyncIndex lastVer = ths->pLogStore->syncLogLastIndex(ths->pLogStore);
assignedCommitIndex = TMAX(assignedCommitIndex, ths->assignedCommitIndex);
ths->assignedCommitIndex = TMIN(assignedCommitIndex, lastVer);
ths->pLogStore->syncLogUpdateCommitIndex(ths->pLogStore, ths->assignedCommitIndex);
return ths->commitIndex;
}

270
source/libs/sync/src/syncMain.c
View File

@ -173,7 +173,7 @@ int32_t syncReconfig(int64_t rid, SSyncCfg* pNewCfg) {
syncNodeUpdateNewConfigIndex(pSyncNode, pNewCfg);
syncNodeDoConfigChange(pSyncNode, pNewCfg, pNewCfg->lastIndex);
if (pSyncNode->state == TAOS_SYNC_STATE_LEADER) {
if (pSyncNode->state == TAOS_SYNC_STATE_LEADER || pSyncNode->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
syncNodeStopHeartbeatTimer(pSyncNode);
for (int32_t i = 0; i < TSDB_MAX_REPLICA + TSDB_MAX_LEARNER_REPLICA; ++i) {
@ -235,6 +235,9 @@ int32_t syncProcessMsg(int64_t rid, SRpcMsg* pMsg) {
case TDMT_SYNC_FORCE_FOLLOWER:
code = syncForceBecomeFollower(pSyncNode, pMsg);
break;
case TDMT_SYNC_SET_ASSIGNED_LEADER:
code = syncBecomeAssignedLeader(pSyncNode, pMsg);
break;
default:
terrno = TSDB_CODE_MSG_NOT_PROCESSED;
code = -1;
@ -271,6 +274,86 @@ int32_t syncForceBecomeFollower(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
return 0;
}
int32_t syncBecomeAssignedLeader(SSyncNode* ths, SRpcMsg* pRpcMsg) {
int32_t ret = -1;
SVArbSetAssignedLeaderReq req = {0};
if (tDeserializeSVArbSetAssignedLeaderReq((char*)pRpcMsg->pCont + sizeof(SMsgHead), pRpcMsg->contLen, &req) != 0) {
sError("vgId:%d, failed to deserialize SVArbSetAssignedLeaderReq", ths->vgId);
terrno = TSDB_CODE_INVALID_MSG;
return -1;
}
int32_t errcode = TSDB_CODE_MND_ARB_TOKEN_MISMATCH;
if (ths->arbTerm > req.arbTerm) {
sInfo("vgId:%d, skip to set assigned leader, msg with lower term, local:%" PRId64 "msg:%" PRId64, ths->vgId,
ths->arbTerm, req.arbTerm);
goto _OVER;
}
ths->arbTerm = TMAX(req.arbTerm, ths->arbTerm);
if (strncmp(req.memberToken, ths->arbToken, TSDB_ARB_TOKEN_SIZE) == 0) {
if (ths->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) {
raftStoreNextTerm(ths);
if (terrno != TSDB_CODE_SUCCESS) {
sError("vgId:%d, failed to set next term since:%s", ths->vgId, terrstr());
goto _OVER;
}
syncNodeBecomeAssignedLeader(ths);
if (syncNodeAppendNoop(ths) < 0) {
sError("vgId:%d, assigned leader failed to append noop entry since %s", ths->vgId, terrstr());
}
}
errcode = TSDB_CODE_SUCCESS;
} else {
sInfo("vgId:%d, skip to set assigned leader, token mismatch, local:%s, msg:%s", ths->vgId, ths->arbToken,
req.memberToken);
goto _OVER;
}
SVArbSetAssignedLeaderRsp rsp = {0};
rsp.arbToken = req.arbToken;
rsp.memberToken = req.memberToken;
rsp.vgId = ths->vgId;
int32_t contLen = tSerializeSVArbSetAssignedLeaderRsp(NULL, 0, &rsp);
if (contLen <= 0) {
sError("vgId:%d, failed to serialize SVArbSetAssignedLeaderRsp", ths->vgId);
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _OVER;
}
void* pHead = rpcMallocCont(contLen);
if (!pHead) {
sError("vgId:%d, failed to malloc memory for SVArbSetAssignedLeaderRsp", ths->vgId);
terrno = TSDB_CODE_OUT_OF_MEMORY;
goto _OVER;
}
if (tSerializeSVArbSetAssignedLeaderRsp(pHead, contLen, &rsp) <= 0) {
sError("vgId:%d, failed to serialize SVArbSetAssignedLeaderRsp", ths->vgId);
terrno = TSDB_CODE_OUT_OF_MEMORY;
rpcFreeCont(pHead);
goto _OVER;
}
SRpcMsg rspMsg = {
.code = errcode,
.pCont = pHead,
.contLen = contLen,
.info = pRpcMsg->info,
};
tmsgSendRsp(&rspMsg);
ret = 0;
_OVER:
tFreeSVArbSetAssignedLeaderReq(&req);
return ret;
}
int32_t syncSendTimeoutRsp(int64_t rid, int64_t seq) {
SSyncNode* pNode = syncNodeAcquire(rid);
if (pNode == NULL) return -1;
@ -342,7 +425,7 @@ int32_t syncBeginSnapshot(int64_t rid, int64_t lastApplyIndex) {
if (pSyncNode->totalReplicaNum > 1) {
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER && pSyncNode->state != TAOS_SYNC_STATE_FOLLOWER &&
pSyncNode->state != TAOS_SYNC_STATE_LEARNER) {
pSyncNode->state != TAOS_SYNC_STATE_LEARNER && pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) {
sNTrace(pSyncNode, "new-snapshot-index:%" PRId64 " candidate or unknown state, do not delete wal",
lastApplyIndex);
syncNodeRelease(pSyncNode);
@ -434,7 +517,7 @@ bool syncNodeIsReadyForRead(SSyncNode* pSyncNode) {
return false;
}
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER && pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) {
terrno = TSDB_CODE_SYN_NOT_LEADER;
return false;
}
@ -562,6 +645,55 @@ SSyncState syncGetState(int64_t rid) {
return state;
}
int32_t syncGetArbToken(int64_t rid, char* outToken) {
SSyncNode* pSyncNode = syncNodeAcquire(rid);
if (pSyncNode == NULL) {
terrno = TSDB_CODE_NOT_FOUND;
return -1;
}
memset(outToken, 0, TSDB_ARB_TOKEN_SIZE);
taosThreadMutexLock(&pSyncNode->arbTokenMutex);
strncpy(outToken, pSyncNode->arbToken, TSDB_ARB_TOKEN_SIZE);
taosThreadMutexUnlock(&pSyncNode->arbTokenMutex);
terrno = TSDB_CODE_SUCCESS;
syncNodeRelease(pSyncNode);
return 0;
}
int32_t syncGetAssignedLogSynced(int64_t rid) {
SSyncNode* pSyncNode = syncNodeAcquire(rid);
if (pSyncNode == NULL) {
terrno = TSDB_CODE_NOT_FOUND;
return -1;
}
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
terrno = TSDB_CODE_VND_ARB_NOT_SYNCED;
syncNodeRelease(pSyncNode);
return 0;
}
bool isSync = pSyncNode->commitIndex >= pSyncNode->assignedCommitIndex;
terrno = (isSync ? TSDB_CODE_SUCCESS : TSDB_CODE_VND_ARB_NOT_SYNCED);
syncNodeRelease(pSyncNode);
return 0;
}
int32_t syncUpdateArbTerm(int64_t rid, SyncTerm arbTerm) {
SSyncNode* pSyncNode = syncNodeAcquire(rid);
if (pSyncNode == NULL) {
terrno = TSDB_CODE_NOT_FOUND;
return -1;
}
pSyncNode->arbTerm = TMAX(arbTerm, pSyncNode->arbTerm);
syncNodeRelease(pSyncNode);
return 0;
}
SyncIndex syncNodeGetSnapshotConfigIndex(SSyncNode* pSyncNode, SyncIndex snapshotLastApplyIndex) {
ASSERT(pSyncNode->raftCfg.configIndexCount >= 1);
SyncIndex lastIndex = (pSyncNode->raftCfg.configIndexArr)[0];
@ -668,8 +800,21 @@ ESyncRole syncGetRole(int64_t rid) {
return role;
}
int64_t syncGetTerm(int64_t rid) {
SSyncNode* pSyncNode = syncNodeAcquire(rid);
if (pSyncNode == NULL) {
sError("sync Node Acquire error since %d", errno);
return -1;
}
int64_t term = raftStoreGetTerm(pSyncNode);
syncNodeRelease(pSyncNode);
return term;
}
int32_t syncNodePropose(SSyncNode* pSyncNode, SRpcMsg* pMsg, bool isWeak, int64_t* seq) {
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER && pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) {
terrno = TSDB_CODE_SYN_NOT_LEADER;
sNWarn(pSyncNode, "sync propose not leader, type:%s", TMSG_INFO(pMsg->msgType));
return -1;
@ -683,7 +828,7 @@ int32_t syncNodePropose(SSyncNode* pSyncNode, SRpcMsg* pMsg, bool isWeak, int64_
}
// heartbeat timeout
if (syncNodeHeartbeatReplyTimeout(pSyncNode)) {
if (pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER && syncNodeHeartbeatReplyTimeout(pSyncNode)) {
terrno = TSDB_CODE_SYN_PROPOSE_NOT_READY;
sNError(pSyncNode, "failed to sync propose since heartbeat timeout, type:%s, last:%" PRId64 ", cmt:%" PRId64,
TMSG_INFO(pMsg->msgType), syncNodeGetLastIndex(pSyncNode), pSyncNode->commitIndex);
@ -916,6 +1061,11 @@ SSyncNode* syncNodeOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion) {
goto _error;
}
pSyncNode->arbTerm = -1;
taosThreadMutexInit(&pSyncNode->arbTokenMutex, NULL);
syncUtilGenerateArbToken(pSyncNode->myNodeInfo.nodeId, pSyncInfo->vgId, pSyncNode->arbToken);
sInfo("vgId:%d, arb token:%s", pSyncNode->vgId, pSyncNode->arbToken);
// init peersNum, peers, peersId
pSyncNode->peersNum = pSyncNode->raftCfg.cfg.totalReplicaNum - 1;
int32_t j = 0;
@ -1064,9 +1214,9 @@ SSyncNode* syncNodeOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion) {
pSyncNode->heartbeatTimerMS = pSyncNode->hbBaseLine;
atomic_store_64(&pSyncNode->heartbeatTimerLogicClock, 0);
atomic_store_64(&pSyncNode->heartbeatTimerLogicClockUser, 0);
#ifdef BUILD_NO_CALL
#ifdef BUILD_NO_CALL
pSyncNode->FpHeartbeatTimerCB = syncNodeEqHeartbeatTimer;
#endif
#endif
pSyncNode->heartbeatTimerCounter = 0;
// init peer heartbeat timer
@ -1138,6 +1288,7 @@ SSyncNode* syncNodeOpen(SSyncInfo* pSyncInfo, int32_t vnodeVersion) {
pSyncNode->isStart = true;
pSyncNode->electNum = 0;
pSyncNode->becomeLeaderNum = 0;
pSyncNode->becomeAssignedLeaderNum = 0;
pSyncNode->configChangeNum = 0;
pSyncNode->hbSlowNum = 0;
pSyncNode->hbrSlowNum = 0;
@ -1305,6 +1456,8 @@ void syncNodeClose(SSyncNode* pSyncNode) {
syncLogBufferDestroy(pSyncNode->pLogBuf);
pSyncNode->pLogBuf = NULL;
taosThreadMutexDestroy(&pSyncNode->arbTokenMutex);
for (int32_t i = 0; i < TSDB_MAX_REPLICA + TSDB_MAX_LEARNER_REPLICA; ++i) {
if (pSyncNode->senders[i] != NULL) {
sDebug("vgId:%d, snapshot sender destroy while close, data:%p", pSyncNode->vgId, pSyncNode->senders[i]);
@ -1906,6 +2059,66 @@ void syncNodeBecomeLeader(SSyncNode* pSyncNode, const char* debugStr) {
sNInfo(pSyncNode, "become leader %s", debugStr);
}
void syncNodeBecomeAssignedLeader(SSyncNode* pSyncNode) {
pSyncNode->becomeAssignedLeaderNum++;
pSyncNode->hbrSlowNum = 0;
// reset restoreFinish
//pSyncNode->restoreFinish = false;
// state change
pSyncNode->state = TAOS_SYNC_STATE_ASSIGNED_LEADER;
pSyncNode->roleTimeMs = taosGetTimestampMs();
// set leader cache
pSyncNode->leaderCache = pSyncNode->myRaftId;
for (int32_t i = 0; i < pSyncNode->pNextIndex->replicaNum; ++i) {
SyncIndex lastIndex;
SyncTerm lastTerm;
int32_t code = syncNodeGetLastIndexTerm(pSyncNode, &lastIndex, &lastTerm);
ASSERT(code == 0);
pSyncNode->pNextIndex->index[i] = lastIndex + 1;
}
for (int32_t i = 0; i < pSyncNode->pMatchIndex->replicaNum; ++i) {
// maybe overwrite myself, no harm
// just do it!
pSyncNode->pMatchIndex->index[i] = SYNC_INDEX_INVALID;
}
// init peer mgr
syncNodePeerStateInit(pSyncNode);
// close receiver
if (snapshotReceiverIsStart(pSyncNode->pNewNodeReceiver)) {
snapshotReceiverStop(pSyncNode->pNewNodeReceiver);
}
// stop elect timer
syncNodeStopElectTimer(pSyncNode);
// start heartbeat timer
syncNodeStartHeartbeatTimer(pSyncNode);
// send heartbeat right now
syncNodeHeartbeatPeers(pSyncNode);
// call back
if (pSyncNode->pFsm != NULL && pSyncNode->pFsm->FpBecomeAssignedLeaderCb != NULL) {
pSyncNode->pFsm->FpBecomeAssignedLeaderCb(pSyncNode->pFsm);
}
// min match index
pSyncNode->minMatchIndex = SYNC_INDEX_INVALID;
// reset log buffer
syncLogBufferReset(pSyncNode->pLogBuf, pSyncNode);
// trace log
sNInfo(pSyncNode, "become assigned leader");
}
void syncNodeCandidate2Leader(SSyncNode* pSyncNode) {
ASSERT(pSyncNode->state == TAOS_SYNC_STATE_CANDIDATE);
bool granted = voteGrantedMajority(pSyncNode->pVotesGranted);
@ -1972,6 +2185,24 @@ void syncNodeCandidate2Follower(SSyncNode* pSyncNode) {
}
#endif
int32_t syncNodeAssignedLeader2Leader(SSyncNode* pSyncNode) {
ASSERT(pSyncNode->state == TAOS_SYNC_STATE_ASSIGNED_LEADER);
syncNodeBecomeLeader(pSyncNode, "assigned leader to leader");
sNTrace(pSyncNode, "assigned leader to leader");
int32_t ret = syncNodeAppendNoop(pSyncNode);
if (ret < 0) {
sError("vgId:%d, failed to append noop entry since %s", pSyncNode->vgId, terrstr());
}
SyncIndex lastIndex = pSyncNode->pLogStore->syncLogLastIndex(pSyncNode->pLogStore);
sInfo("vgId:%d, become leader from assigned leader. term:%" PRId64 ", commit index:%" PRId64
"assigned commit index:%" PRId64 ", last index:%" PRId64,
pSyncNode->vgId, raftStoreGetTerm(pSyncNode), pSyncNode->commitIndex, pSyncNode->assignedCommitIndex, lastIndex);
return 0;
}
// just called by syncNodeVoteForSelf
// need assert
void syncNodeVoteForTerm(SSyncNode* pSyncNode, SyncTerm term, SRaftId* pRaftId) {
@ -2291,7 +2522,7 @@ static void syncNodeEqPeerHeartbeatTimer(void* param, void* tmrId) {
return;
}
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER && pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) {
syncNodeRelease(pSyncNode);
syncHbTimerDataRelease(pData);
sError("vgId:%d, hb timer sync node not leader", pSyncNode->vgId);
@ -2426,7 +2657,8 @@ int32_t syncNodeCheckChangeConfig(SSyncNode* ths, SSyncRaftEntry* pEntry) {
SSyncCfg cfg = {0};
syncBuildConfigFromReq(&req, &cfg);
if (cfg.totalReplicaNum >= 1 && ths->state == TAOS_SYNC_STATE_LEADER) {
if (cfg.totalReplicaNum >= 1 &&
(ths->state == TAOS_SYNC_STATE_LEADER || ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER)) {
bool incfg = false;
for (int32_t j = 0; j < cfg.totalReplicaNum; ++j) {
if (strcmp(ths->myNodeInfo.nodeFqdn, cfg.nodeInfo[j].nodeFqdn) == 0 &&
@ -2934,6 +3166,16 @@ _out:;
ths->vgId, pEntry->index, pEntry->term, ths->pLogBuf->startIndex, ths->pLogBuf->commitIndex,
ths->pLogBuf->matchIndex, ths->pLogBuf->endIndex);
if (code == 0 && ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
(void)syncNodeUpdateAssignedCommitIndex(ths, matchIndex);
if (ths->fsmState != SYNC_FSM_STATE_INCOMPLETE &&
syncLogBufferCommit(ths->pLogBuf, ths, ths->assignedCommitIndex) < 0) {
sError("vgId:%d, failed to commit until commitIndex:%" PRId64 "", ths->vgId, ths->commitIndex);
code = -1;
}
}
// multi replica
if (ths->replicaNum > 1) {
return code;
@ -3080,7 +3322,8 @@ int32_t syncNodeOnHeartbeat(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
currentTerm = pMsg->term;
}
if (pMsg->term == currentTerm && ths->state != TAOS_SYNC_STATE_LEADER) {
if (pMsg->term == currentTerm &&
(ths->state != TAOS_SYNC_STATE_LEADER && ths->state != TAOS_SYNC_STATE_ASSIGNED_LEADER)) {
syncIndexMgrSetRecvTime(ths->pNextIndex, &(pMsg->srcId), tsMs);
resetElect = true;
@ -3109,7 +3352,8 @@ int32_t syncNodeOnHeartbeat(SSyncNode* ths, const SRpcMsg* pRpcMsg) {
}
}
if (pMsg->term >= currentTerm && ths->state == TAOS_SYNC_STATE_LEADER) {
if (pMsg->term >= currentTerm &&
(ths->state == TAOS_SYNC_STATE_LEADER || ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER)) {
SRpcMsg rpcMsgLocalCmd = {0};
(void)syncBuildLocalCmd(&rpcMsgLocalCmd, ths->vgId);
@ -3239,7 +3483,7 @@ int32_t syncNodeOnClientRequest(SSyncNode* ths, SRpcMsg* pMsg, SyncIndex* pRetIn
return -1;
}
if (ths->state == TAOS_SYNC_STATE_LEADER) {
if (ths->state == TAOS_SYNC_STATE_LEADER || ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) {
if (pRetIndex) {
(*pRetIndex) = index;
}
@ -3288,6 +3532,8 @@ const char* syncStr(ESyncState state) {
return "offline";
case TAOS_SYNC_STATE_LEARNER:
return "learner";
case TAOS_SYNC_STATE_ASSIGNED_LEADER:
return "assigned leader";
default:
return "unknown";
}

18
source/libs/sync/src/syncPipeline.c
View File

@ -484,7 +484,7 @@ int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* p
taosMsleep(1);
goto _out;
}
if(pEntry->originalRpcType == TDMT_SYNC_CONFIG_CHANGE){
if(pNode->pLogBuf->commitIndex == pEntry->index -1){
sInfo("vgId:%d, to change config at %s. "
@ -492,7 +492,7 @@ int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* p
"node, restore:%d, commitIndex:%" PRId64 ", "
"cond: (pre entry index:%" PRId64 "== buf commit index:%" PRId64 ")",
pNode->vgId, str,
pEntry->index, pEntry->term,
pEntry->index, pEntry->term,
pNode->restoreFinish, pNode->commitIndex,
pEntry->index - 1, pNode->pLogBuf->commitIndex);
if(syncNodeChangeConfig(pNode, pEntry, str) != 0){
@ -507,9 +507,9 @@ int64_t syncLogBufferProceed(SSyncLogBuffer* pBuf, SSyncNode* pNode, SyncTerm* p
"node, commitIndex:%" PRId64 ", pBuf: [%" PRId64 " %" PRId64 " %" PRId64 ", %" PRId64 "), "
"cond:( pre entry index:%" PRId64" != buf commit index:%" PRId64 ")",
pNode->vgId, str,
pEntry->index, pEntry->term,
pEntry->index, pEntry->term,
pNode->commitIndex, pNode->pLogBuf->startIndex, pNode->pLogBuf->commitIndex,
pNode->pLogBuf->matchIndex, pNode->pLogBuf->endIndex,
pNode->pLogBuf->matchIndex, pNode->pLogBuf->endIndex,
pEntry->index - 1, pNode->pLogBuf->commitIndex);
}
}
@ -653,8 +653,8 @@ int32_t syncLogBufferCommit(SSyncLogBuffer* pBuf, SSyncNode* pNode, int64_t comm
"current entry, index:%" PRId64 ", term:%" PRId64", "
"node, role:%d, current term:%" PRId64 ", restore:%d, "
"cond, next entry index:%" PRId64 ", msgType:%s",
vgId,
pEntry->index, pEntry->term,
vgId,
pEntry->index, pEntry->term,
role, currentTerm, pNode->restoreFinish,
pNextEntry->index, TMSG_INFO(pNextEntry->originalRpcType));
@ -685,12 +685,12 @@ int32_t syncLogBufferCommit(SSyncLogBuffer* pBuf, SSyncNode* pNode, int64_t comm
syncEntryDestroy(pNextEntry);
pNextEntry = NULL;
}
}
}
if (!inBuf) {
syncEntryDestroy(pEntry);
pEntry = NULL;
}
}
}
// recycle

3
source/libs/sync/src/syncReplication.c
View File

@ -64,7 +64,8 @@ int32_t syncNodeReplicate(SSyncNode* pNode) {
}
int32_t syncNodeReplicateWithoutLock(SSyncNode* pNode) {
if (pNode->state != TAOS_SYNC_STATE_LEADER || pNode->raftCfg.cfg.totalReplicaNum == 1) {
if ((pNode->state != TAOS_SYNC_STATE_LEADER && pNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) ||
pNode->raftCfg.cfg.totalReplicaNum == 1) {
return -1;
}
for (int32_t i = 0; i < pNode->totalReplicaNum; i++) {

2
source/libs/sync/src/syncSnapshot.c
View File

@ -1242,7 +1242,7 @@ int32_t syncNodeOnSnapshotRsp(SSyncNode *pSyncNode, SRpcMsg *pRpcMsg) {
goto _ERROR;
}
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER) {
if (pSyncNode->state != TAOS_SYNC_STATE_LEADER && pSyncNode->state != TAOS_SYNC_STATE_ASSIGNED_LEADER) {
sSError(pSender, "snapshot sender not leader");
terrno = TSDB_CODE_SYN_NOT_LEADER;
goto _ERROR;

3
source/libs/sync/src/syncTimeout.c
View File

@ -77,7 +77,8 @@ static int32_t syncNodeTimerRoutine(SSyncNode* ths) {
for (int i = 0; i < ths->peersNum; ++i) {
SSyncSnapshotSender* pSender = syncNodeGetSnapshotSender(ths, &(ths->peersId[i]));
if (pSender != NULL) {
if (ths->isStart && ths->state == TAOS_SYNC_STATE_LEADER && pSender->start) {
if (ths->isStart && (ths->state == TAOS_SYNC_STATE_LEADER || ths->state == TAOS_SYNC_STATE_ASSIGNED_LEADER) &&
pSender->start) {
int64_t elapsedMs = timeNow - pSender->lastSendTime;
if (elapsedMs < SYNC_SNAP_RESEND_MS) {
continue;

40
source/libs/sync/src/syncUtil.c
View File

@ -44,7 +44,7 @@ void syncUtilNodeInfo2EpSet(const SNodeInfo* pInfo, SEpSet* pEpSet) {
bool syncUtilNodeInfo2RaftId(const SNodeInfo* pInfo, SyncGroupId vgId, SRaftId* raftId) {
uint32_t ipv4 = 0xFFFFFFFF;
sDebug("vgId:%d, start to resolve sync addr fqdn in %d seconds, "
"dnode:%d cluster:%" PRId64 " fqdn:%s port:%u ",
"dnode:%d cluster:%" PRId64 " fqdn:%s port:%u ",
vgId, tsResolveFQDNRetryTime,
pInfo->nodeId, pInfo->clusterId, pInfo->nodeFqdn, pInfo->nodePort);
for(int i = 0; i < tsResolveFQDNRetryTime; i++){
@ -57,7 +57,7 @@ bool syncUtilNodeInfo2RaftId(const SNodeInfo* pInfo, SyncGroupId vgId, SRaftId*
break;
}
}
if (ipv4 == 0xFFFFFFFF || ipv4 == 1) {
sError("failed to resolve ipv4 addr, fqdn:%s", pInfo->nodeFqdn);
terrno = TSDB_CODE_TSC_INVALID_FQDN;
@ -103,6 +103,13 @@ bool syncUtilUserPreCommit(tmsg_t msgType) { return msgType != TDMT_SYNC_NOOP &&
bool syncUtilUserRollback(tmsg_t msgType) { return msgType != TDMT_SYNC_NOOP && msgType != TDMT_SYNC_LEADER_TRANSFER; }
void syncUtilGenerateArbToken(int32_t nodeId, int32_t groupId, char* buf) {
memset(buf, 0, TSDB_ARB_TOKEN_SIZE);
int32_t randVal = taosSafeRand() % 1000;
int64_t currentMs = taosGetTimestampMs();
snprintf(buf, TSDB_ARB_TOKEN_SIZE, "d%d#g%d#%" PRId64 "#%d", nodeId, groupId, currentMs, randVal);
}
// for leader
static void syncHearbeatReplyTime2Str(SSyncNode* pSyncNode, char* buf, int32_t bufLen) {
int32_t len = 0;
@ -221,19 +228,22 @@ void syncPrintNodeLog(const char* flags, ELogLevel level, int32_t dflag, SSyncNo
SyncIndex appliedIndex = pNode->pFsm->FpAppliedIndexCb(pNode->pFsm);
if (pNode != NULL) {
taosPrintLog(flags, level, dflag,
"vgId:%d, %s, sync:%s, term:%" PRIu64 ", commit-index:%" PRId64 ", applied-index:%" PRId64
", first-ver:%" PRId64 ", last-ver:%" PRId64 ", min:%" PRId64 ", snap:%" PRId64 ", snap-term:%" PRIu64
", elect-times:%d, as-leader-times:%d, cfg-ch-times:%d, hb-slow:%d, hbr-slow:%d, "
"aq-items:%d, snaping:%" PRId64 ", replicas:%d, last-cfg:%" PRId64
", chging:%d, restore:%d, quorum:%d, elect-lc-timer:%" PRId64 ", hb:%" PRId64
", buffer:%s, repl-mgrs:%s, members:%s, hb:%s, hb-reply:%s",
pNode->vgId, eventLog, syncStr(pNode->state), currentTerm, pNode->commitIndex, appliedIndex,
logBeginIndex, logLastIndex, pNode->minMatchIndex, snapshot.lastApplyIndex, snapshot.lastApplyTerm,
pNode->electNum, pNode->becomeLeaderNum, pNode->configChangeNum, pNode->hbSlowNum, pNode->hbrSlowNum,
aqItems, pNode->snapshottingIndex, pNode->replicaNum, pNode->raftCfg.lastConfigIndex, pNode->changing,
pNode->restoreFinish, syncNodeDynamicQuorum(pNode), pNode->electTimerLogicClock,
pNode->heartbeatTimerLogicClockUser, bufferStatesStr, replMgrStatesStr, cfgStr, hbTimeStr, hbrTimeStr);
taosPrintLog(
flags, level, dflag,
"vgId:%d, %s, sync:%s, term:%" PRIu64 ", commit-index:%" PRId64 ", assigned-index:%" PRId64
", applied-index:%" PRId64 ", first-ver:%" PRId64 ", last-ver:%" PRId64 ", min:%" PRId64 ", snap:%" PRId64
", snap-term:%" PRIu64
", elect-times:%d, as-leader-times:%d, as-assigned-leader-times:%d, cfg-ch-times:%d, hb-slow:%d, hbr-slow:%d, "
"aq-items:%d, snaping:%" PRId64 ", replicas:%d, last-cfg:%" PRId64
", chging:%d, restore:%d, quorum:%d, elect-lc-timer:%" PRId64 ", hb:%" PRId64
", buffer:%s, repl-mgrs:%s, members:%s, hb:%s, hb-reply:%s",
pNode->vgId, eventLog, syncStr(pNode->state), currentTerm, pNode->commitIndex, pNode->assignedCommitIndex,
appliedIndex, logBeginIndex, logLastIndex, pNode->minMatchIndex, snapshot.lastApplyIndex,
snapshot.lastApplyTerm, pNode->electNum, pNode->becomeLeaderNum, pNode->becomeAssignedLeaderNum,
pNode->configChangeNum, pNode->hbSlowNum, pNode->hbrSlowNum, aqItems, pNode->snapshottingIndex,
pNode->replicaNum, pNode->raftCfg.lastConfigIndex, pNode->changing, pNode->restoreFinish,
syncNodeDynamicQuorum(pNode), pNode->electTimerLogicClock, pNode->heartbeatTimerLogicClockUser, bufferStatesStr,
replMgrStatesStr, cfgStr, hbTimeStr, hbrTimeStr);
}
}

6
source/util/src/terror.c
View File

@ -254,6 +254,9 @@ TAOS_DEFINE_ERROR(TSDB_CODE_MND_SNODE_ALREADY_EXIST, "Snode can only be cre
TAOS_DEFINE_ERROR(TSDB_CODE_MND_SNODE_NOT_EXIST, "Snode not there")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOO_FEW_MNODES, "The replica of mnode cannot less than 1")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOO_MANY_MNODES, "The replica of mnode cannot exceed 3")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_ARBGROUP_ALREADY_EXIST, "Arbitrator group already exists")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_ARBGROUP_NOT_EXIST, "Arbitrator group not there")
TAOS_DEFINE_ERROR(TSDB_CODE_MND_ARB_TOKEN_MISMATCH, "Arbitrator token mismatch")
// mnode-dnode-part2
TAOS_DEFINE_ERROR(TSDB_CODE_MND_TOO_MANY_DNODES, "Too many dnodes")
@ -368,6 +371,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_VND_NOT_CATCH_UP, "Vnode didn't catch up
TAOS_DEFINE_ERROR(TSDB_CODE_VND_ALREADY_IS_VOTER, "Vnode already is a voter")
TAOS_DEFINE_ERROR(TSDB_CODE_VND_DIR_ALREADY_EXIST, "Vnode directory already exist")
TAOS_DEFINE_ERROR(TSDB_CODE_VND_META_DATA_UNSAFE_DELETE, "Single replica vnode data will lost permanently after this operation, if you make sure this, please use drop dnode <id> unsafe to execute")
TAOS_DEFINE_ERROR(TSDB_CODE_VND_ARB_NOT_SYNCED, "Vgroup peer is not synced")
// tsdb
TAOS_DEFINE_ERROR(TSDB_CODE_TDB_INVALID_TABLE_ID, "Invalid table ID")
@ -709,7 +713,7 @@ TAOS_DEFINE_ERROR(TSDB_CODE_TMQ_REPLAY_NOT_SUPPORT, "Replay is disabled
TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_TASK_NOT_EXIST, "Stream task not exist")
TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_EXEC_CANCELLED, "Stream task exec cancelled")
TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_INVALID_STATETRANS, "Invalid task state to handle event")
TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_TASK_IVLD_STATUS,"Invalid task status to proceed")
TAOS_DEFINE_ERROR(TSDB_CODE_STREAM_TASK_IVLD_STATUS, "Invalid task status to proceed")
// TDLite
TAOS_DEFINE_ERROR(TSDB_CODE_TDLITE_IVLD_OPEN_FLAGS, "Invalid TDLite open flags")

11
source/util/src/tlog.c
View File

@ -41,6 +41,12 @@
#define LOG_BUF_SIZE(x) ((x)->buffSize)
#define LOG_BUF_MUTEX(x) ((x)->buffMutex)
#ifdef TD_ENTERPRISE
#define LOG_EDITION_FLG ("E")
#else
#define LOG_EDITION_FLG ("C")
#endif
typedef struct {
char *buffer;
int32_t buffStart;
@ -490,8 +496,9 @@ static inline int32_t taosBuildLogHead(char *buffer, const char *flags) {
time_t curTime = timeSecs.tv_sec;
ptm = taosLocalTime(&curTime, &Tm, NULL);
return sprintf(buffer, "%02d/%02d %02d:%02d:%02d.%06d %08" PRId64 " %s", ptm->tm_mon + 1, ptm->tm_mday, ptm->tm_hour,
ptm->tm_min, ptm->tm_sec, (int32_t)timeSecs.tv_usec, taosGetSelfPthreadId(), flags);
return sprintf(buffer, "%02d/%02d %02d:%02d:%02d.%06d %08" PRId64 " %s %s", ptm->tm_mon + 1, ptm->tm_mday,
ptm->tm_hour, ptm->tm_min, ptm->tm_sec, (int32_t)timeSecs.tv_usec, taosGetSelfPthreadId(),
LOG_EDITION_FLG, flags);
}
static inline void taosPrintLogImp(ELogLevel level, int32_t dflag, const char *buffer, int32_t len) {

24
tests/develop-test/2-query/table_count_scan.py
View File

@ -1,4 +1,4 @@
import sys
import sys
from util.log import *
from util.cases import *
from util.sql import *
@ -9,14 +9,14 @@ class TDTestCase:
def caseDescription(self):
'''
case1<shenglian zhou>: [TD-21890] table count scan test case
'''
'''
return
def init(self, conn, logSql, replicaVer=1):
tdLog.debug("start to execute %s" % __file__)
tdSql.init(conn.cursor(), True)
self._conn = conn
def restartTaosd(self, index=1, dbname="db"):
tdDnodes.stop(index)
tdDnodes.startWithoutSleep(index)
@ -65,7 +65,7 @@ class TDTestCase:
tdSql.query('select count(*),db_name, stable_name from information_schema.ins_tables group by db_name, stable_name;')
tdSql.checkRows(3)
tdSql.checkData(0, 0, 29)
tdSql.checkData(0, 0, 30)
tdSql.checkData(0, 1, 'information_schema')
tdSql.checkData(0, 2, None)
tdSql.checkData(1, 0, 3)
@ -77,7 +77,7 @@ class TDTestCase:
tdSql.query('select count(1) v,db_name, stable_name from information_schema.ins_tables group by db_name, stable_name order by v desc;')
tdSql.checkRows(3)
tdSql.checkData(0, 0, 29)
tdSql.checkData(0, 0, 30)
tdSql.checkData(0, 1, 'information_schema')
tdSql.checkData(0, 2, None)
tdSql.checkData(1, 0, 5)
@ -93,7 +93,7 @@ class TDTestCase:
tdSql.checkData(1, 1, 'performance_schema')
tdSql.checkData(0, 0, 3)
tdSql.checkData(0, 1, 'tbl_count')
tdSql.checkData(2, 0, 29)
tdSql.checkData(2, 0, 30)
tdSql.checkData(2, 1, 'information_schema')
tdSql.query("select count(*) from information_schema.ins_tables where db_name='tbl_count'")
@ -106,7 +106,7 @@ class TDTestCase:
tdSql.query('select count(*) from information_schema.ins_tables')
tdSql.checkRows(1)
tdSql.checkData(0, 0, 37)
tdSql.checkData(0, 0, 38)
tdSql.execute('create table stba (ts timestamp, c1 bool, c2 tinyint, c3 smallint, c4 int, c5 bigint, c6 float, c7 double, c8 binary(10), c9 nchar(10), c10 tinyint unsigned, c11 smallint unsigned, c12 int unsigned, c13 bigint unsigned) TAGS(t1 int, t2 binary(10), t3 double);')
@ -189,7 +189,7 @@ class TDTestCase:
tdSql.checkData(2, 0, 5)
tdSql.checkData(2, 1, 'performance_schema')
tdSql.checkData(2, 2, None)
tdSql.checkData(3, 0, 29)
tdSql.checkData(3, 0, 30)
tdSql.checkData(3, 1, 'information_schema')
tdSql.checkData(3, 2, None)
@ -204,7 +204,7 @@ class TDTestCase:
tdSql.checkData(2, 0, 5)
tdSql.checkData(2, 1, 'performance_schema')
tdSql.checkData(2, 2, None)
tdSql.checkData(3, 0, 29)
tdSql.checkData(3, 0, 30)
tdSql.checkData(3, 1, 'information_schema')
tdSql.checkData(3, 2, None)
@ -215,7 +215,7 @@ class TDTestCase:
tdSql.checkData(0, 1, 'tbl_count')
tdSql.checkData(1, 0, 5)
tdSql.checkData(1, 1, 'performance_schema')
tdSql.checkData(2, 0, 29)
tdSql.checkData(2, 0, 30)
tdSql.checkData(2, 1, 'information_schema')
tdSql.query("select count(*) from information_schema.ins_tables where db_name='tbl_count'")
@ -228,7 +228,7 @@ class TDTestCase:
tdSql.query('select count(*) from information_schema.ins_tables')
tdSql.checkRows(1)
tdSql.checkData(0, 0, 38)
tdSql.checkData(0, 0, 39)
tdSql.execute('drop database tbl_count')

1
tests/parallel_test/cases.task
View File

@ -591,6 +591,7 @@
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/ts-4348-td-27939.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/backslash_g.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/test_ts4467.py
,,y,system-test,./pytest.sh python3 ./test.py -f 2-query/geometry.py
,,n,system-test,python3 ./test.py -f 2-query/queryQnode.py
,,y,system-test,./pytest.sh python3 ./test.py -f 6-cluster/5dnode1mnode.py

11
tests/pytest/util/sql.py
View File

@ -115,13 +115,17 @@ class TDSql:
else:
tdLog.info("sql:%s, check passed, no ErrInfo occurred" % (sql))
def error(self, sql, expectedErrno = None, expectErrInfo = None, fullMatched = True):
def error(self, sql, expectedErrno = None, expectErrInfo = None, fullMatched = True, show = False):
caller = inspect.getframeinfo(inspect.stack()[1][0])
expectErrNotOccured = True
if show:
tdLog.info("sql:%s" % (sql))
try:
self.cursor.execute(sql)
except BaseException as e:
tdLog.info("err:%s" % (e))
expectErrNotOccured = False
self.errno = e.errno
error_info = repr(e)
@ -161,7 +165,10 @@ class TDSql:
return self.error_info
def query(self, sql, row_tag=None, queryTimes=10, count_expected_res=None):
def query(self, sql, row_tag=None, queryTimes=10, count_expected_res=None, show = False):
if show:
tdLog.info("sql:%s" % (sql))
self.sql = sql
i=1
while i <= queryTimes:

2
tests/script/tsim/db/create_all_options.sim
View File

@ -362,7 +362,7 @@ if $data4_db != 1 then
return -1
endi
sql drop database db
sql_error create database db REPLICA 2
#sql_error create database db REPLICA 2
sql_error create database db REPLICA 0
sql_error create database db REPLICA -1
sql_error create database db REPLICA 4

4
tests/script/tsim/query/sys_tbname.sim
View File

@ -52,13 +52,13 @@ if $data00 != @ins_stables@ then
endi
sql select * from information_schema.ins_tables where table_name='';
if $rows != 0 then
if $rows != 0 then
return -1
endi
sql select tbname from information_schema.ins_tables;
print $rows $data00
if $rows != 38 then
if $rows != 39 then
return -1
endi
if $data00 != @ins_tables@ then

Some files were not shown because too many files have changed in this diff Show More