Merge branch 'main' into docs/sangshuduo/refine-get-started
This commit is contained in:
Shuduo Sang
commit
d766adcfc2
|
|
@ -18,7 +18,7 @@ The full package of TDengine includes the TDengine Server (`taosd`), TDengine Cl
|
|||
|
||||
The standard server installation package includes `taos`, `taosd`, `taosAdapter`, `taosBenchmark`, and sample code. You can also download the Lite package that includes only `taosd` and the C/C++ connector.
|
||||
|
||||
The TDengine Community Edition is released as Deb and RPM packages. The Deb package can be installed on Debian, Ubuntu, and derivative systems. The RPM package can be installed on CentOS, RHEL, SUSE, and derivative systems. A .tar.gz package is also provided for enterprise customers, and you can install TDengine over `apt-get` as well. The .tar.tz package includes `taosdump` and the TDinsight installation script. If you want to use these utilities with the Deb or RPM package, download and install taosTools separately. TDengine can also be installed on x64 Windows and x64/m1 macOS.
|
||||
TDengine OSS is released as Deb and RPM packages. The Deb package can be installed on Debian, Ubuntu, and derivative systems. The RPM package can be installed on CentOS, RHEL, SUSE, and derivative systems. A .tar.gz package is also provided for enterprise customers, and you can install TDengine over `apt-get` as well. The .tar.tz package includes `taosdump` and the TDinsight installation script. If you want to use these utilities with the Deb or RPM package, download and install taosTools separately. TDengine can also be installed on x64 Windows and x64/m1 macOS.
|
||||
|
||||
## Operating environment requirements
|
||||
In the Linux system, the minimum requirements for the operating environment are as follows:
|
||||
|
|
|
|||
|
|
@ -21,17 +21,6 @@ import {useCurrentSidebarCategory} from '@docusaurus/theme-common';
|
|||
<DocCardList items={useCurrentSidebarCategory().items}/>
|
||||
```
|
||||
|
||||
## Study TDengine Knowledge Map
|
||||
|
||||
The TDengine Knowledge Map covers the various knowledge points of TDengine, revealing the invocation relationships and data flow between various conceptual entities. Learning and understanding the TDengine Knowledge Map will help you quickly master the TDengine knowledge system.
|
||||
|
||||
<figure>
|
||||
<center>
|
||||
<a href="pathname:///img/tdengine-map.svg" target="_blank"><img src="/img/tdengine-map.svg" width="80%" /></a>
|
||||
<figcaption>Diagram 1. TDengine Knowledge Map</figcaption>
|
||||
</center>
|
||||
</figure>
|
||||
|
||||
## Join TDengine Community
|
||||
|
||||
<table width="100%">
|
||||
|
|
|
|||
|
|
@ -4,23 +4,31 @@ sidebar_label: Kubernetes
|
|||
description: This document describes how to deploy TDengine on Kubernetes.
|
||||
---
|
||||
|
||||
TDengine is a cloud-native time-series database that can be deployed on Kubernetes. This document gives a step-by-step description of how you can use YAML files to create a TDengine cluster and introduces common operations for TDengine in a Kubernetes environment.
|
||||
## Overview
|
||||
|
||||
As a time series database for Cloud Native architecture design, TDengine supports Kubernetes deployment. Firstly we introduce how to use YAML files to create a highly available TDengine cluster from scratch step by step for production usage, and highlight the common operations of TDengine in Kubernetes environment.
|
||||
|
||||
To meet [high availability ](https://docs.taosdata.com/tdinternal/high-availability/)requirements, clusters need to meet the following requirements:
|
||||
|
||||
- 3 or more dnodes: multiple vnodes in the same vgroup of TDengine are not allowed to be distributed in one dnode at the same time, so if you create a database with 3 replicas, the number of dnodes is greater than or equal to 3
|
||||
- 3 mnodes: mnode is responsible for the management of the entire TDengine cluster. The default number of mnode in TDengine cluster is only one. If the dnode where the mnode located is dropped, the entire cluster is unavailable.
|
||||
- Database 3 replicas: The TDengine replica configuration is the database level, so 3 replicas for the database must need three dnodes in the cluster. If any one dnode is offline, does not affect the normal usage of the whole cluster. **If the number of offline** **dnodes** **is 2, then the cluster is not available,** **because** ** the cluster can not complete the election based on RAFT** **.** (Enterprise version: in the disaster recovery scenario, any node data file is damaged, can be restored by pulling up the dnode again)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before deploying TDengine on Kubernetes, perform the following:
|
||||
|
||||
* Current steps are compatible with Kubernetes v1.5 and later version.
|
||||
* Install and configure minikube, kubectl, and helm.
|
||||
* Install and deploy Kubernetes and ensure that it can be accessed and used normally. Update any container registries or other services as necessary.
|
||||
- This article applies Kubernetes 1.19 and above
|
||||
- This article uses the **kubectl** tool to install and deploy, please install the corresponding software in advance
|
||||
- Kubernetes have been installed and deployed and can access or update the necessary container repositories or other services
|
||||
|
||||
You can download the configuration files in this document from [GitHub](https://github.com/taosdata/TDengine-Operator/tree/3.0/src/tdengine).
|
||||
|
||||
## Configure the service
|
||||
|
||||
Create a service configuration file named `taosd-service.yaml`. Record the value of `metadata.name` (in this example, `taos`) for use in the next step. Add the ports required by TDengine:
|
||||
Create a service configuration file named `taosd-service.yaml`. Record the value of `metadata.name` (in this example, `taos`) for use in the next step. And then add the ports required by TDengine and record the value of the selector label "app" (in this example, `tdengine`) for use in the next step:
|
||||
|
||||
```yaml
|
||||
```YAML
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
|
@ -31,10 +39,10 @@ metadata:
|
|||
spec:
|
||||
ports:
|
||||
- name: tcp6030
|
||||
- protocol: "TCP"
|
||||
protocol: "TCP"
|
||||
port: 6030
|
||||
- name: tcp6041
|
||||
- protocol: "TCP"
|
||||
protocol: "TCP"
|
||||
port: 6041
|
||||
selector:
|
||||
app: "tdengine"
|
||||
|
|
@ -42,10 +50,11 @@ spec:
|
|||
|
||||
## Configure the service as StatefulSet
|
||||
|
||||
Configure the TDengine service as a StatefulSet.
|
||||
Create the `tdengine.yaml` file and set `replicas` to 3. In this example, the region is set to Asia/Shanghai and 10 GB of standard storage are allocated per node. You can change the configuration based on your environment and business requirements.
|
||||
According to Kubernetes instructions for various deployments, we will use StatefulSet as the deployment resource type of TDengine. Create the file `tdengine.yaml `, where replicas defines the number of cluster nodes as 3. The node time zone is China (Asia/Shanghai), and each node is allocated 5G standard storage (refer to the [Storage Classes ](https://kubernetes.io/docs/concepts/storage/storage-classes/)configuration storage class). You can also modify accordingly according to the actual situation.
|
||||
|
||||
```yaml
|
||||
Please pay special attention to the startupProbe configuration. If dnode's Pod drops for a period of time and then restart, the newly launched dnode Pod will be temporarily unavailable. The reason is the startupProbe configuration is too small, Kubernetes will know that the Pod is in an abnormal state and try to restart it, then the dnode's Pod will restart frequently and never return to the normal status. Refer to [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
|
||||
|
||||
```YAML
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
|
|
@ -69,14 +78,14 @@ spec:
|
|||
spec:
|
||||
containers:
|
||||
- name: "tdengine"
|
||||
image: "tdengine/tdengine:3.0.0.0"
|
||||
image: "tdengine/tdengine:3.0.7.1"
|
||||
imagePullPolicy: "IfNotPresent"
|
||||
ports:
|
||||
- name: tcp6030
|
||||
- protocol: "TCP"
|
||||
protocol: "TCP"
|
||||
containerPort: 6030
|
||||
- name: tcp6041
|
||||
- protocol: "TCP"
|
||||
protocol: "TCP"
|
||||
containerPort: 6041
|
||||
env:
|
||||
# POD_NAME for FQDN config
|
||||
|
|
@ -102,12 +111,18 @@ spec:
|
|||
# Must set if you want a cluster.
|
||||
- name: TAOS_FIRST_EP
|
||||
value: "$(STS_NAME)-0.$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local:$(TAOS_SERVER_PORT)"
|
||||
# TAOS_FQDN should always be set in k8s env.
|
||||
# TAOS_FQND should always be set in k8s env.
|
||||
- name: TAOS_FQDN
|
||||
value: "$(POD_NAME).$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local"
|
||||
volumeMounts:
|
||||
- name: taosdata
|
||||
mountPath: /var/lib/taos
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- taos-check
|
||||
failureThreshold: 360
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
|
|
@ -129,266 +144,401 @@ spec:
|
|||
storageClassName: "standard"
|
||||
resources:
|
||||
requests:
|
||||
storage: "10Gi"
|
||||
storage: "5Gi"
|
||||
```
|
||||
|
||||
## Use kubectl to deploy TDengine
|
||||
|
||||
Run the following commands:
|
||||
First create the corresponding namespace, and then execute the following command in sequence :
|
||||
|
||||
```bash
|
||||
kubectl apply -f taosd-service.yaml
|
||||
kubectl apply -f tdengine.yaml
|
||||
```Bash
|
||||
kubectl apply -f taosd-service.yaml -n tdengine-test
|
||||
kubectl apply -f tdengine.yaml -n tdengine-test
|
||||
```
|
||||
|
||||
The preceding configuration generates a TDengine cluster with three nodes in which dnodes are automatically configured. You can run the `show dnodes` command to query the nodes in the cluster:
|
||||
The above configuration will generate a three-node TDengine cluster, dnode is automatically configured, you can use the **show dnodes** command to view the nodes of the current cluster:
|
||||
|
||||
```bash
|
||||
kubectl exec -i -t tdengine-0 -- taos -s "show dnodes"
|
||||
kubectl exec -i -t tdengine-1 -- taos -s "show dnodes"
|
||||
kubectl exec -i -t tdengine-2 -- taos -s "show dnodes"
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show dnodes"
|
||||
kubectl exec -it tdengine-2 -n tdengine-test -- taos -s "show dnodes"
|
||||
```
|
||||
|
||||
The output is as follows:
|
||||
|
||||
```
|
||||
```Bash
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||||
============================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||||
Query OK, 3 rows in database (0.003655s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 0 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-19 17:54:18.469 | | | |
|
||||
2 | tdengine-1.ta... | 0 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-19 17:54:38.698 | | | |
|
||||
3 | tdengine-2.ta... | 0 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-19 17:55:02.039 | | | |
|
||||
Query OK, 3 row(s) in set (0.001853s)
|
||||
```
|
||||
|
||||
View the current mnode
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show mnodes\G"
|
||||
taos> show mnodes\G
|
||||
*************************** 1.row ***************************
|
||||
id: 1
|
||||
endpoint: tdengine-0.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: leader
|
||||
status: ready
|
||||
create_time: 2023-07-19 17:54:18.559
|
||||
reboot_time: 2023-07-19 17:54:19.520
|
||||
Query OK, 1 row(s) in set (0.001282s)
|
||||
```
|
||||
|
||||
## Create mnode
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "create mnode on dnode 2"
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "create mnode on dnode 3"
|
||||
```
|
||||
|
||||
View mnode
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show mnodes\G"
|
||||
|
||||
taos> show mnodes\G
|
||||
*************************** 1.row ***************************
|
||||
id: 1
|
||||
endpoint: tdengine-0.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: leader
|
||||
status: ready
|
||||
create_time: 2023-07-19 17:54:18.559
|
||||
reboot_time: 2023-07-20 09:19:36.060
|
||||
*************************** 2.row ***************************
|
||||
id: 2
|
||||
endpoint: tdengine-1.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: follower
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:05.600
|
||||
reboot_time: 2023-07-20 09:22:12.838
|
||||
*************************** 3.row ***************************
|
||||
id: 3
|
||||
endpoint: tdengine-2.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: follower
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:20.042
|
||||
reboot_time: 2023-07-20 09:22:23.271
|
||||
Query OK, 3 row(s) in set (0.003108s)
|
||||
```
|
||||
|
||||
## Enable port forwarding
|
||||
|
||||
The kubectl port forwarding feature allows applications to access the TDengine cluster running on Kubernetes.
|
||||
Kubectl port forwarding enables applications to access TDengine clusters running in Kubernetes environments.
|
||||
|
||||
```
|
||||
kubectl port-forward tdengine-0 6041:6041 &
|
||||
```bash
|
||||
kubectl port-forward -n tdengine-test tdengine-0 6041:6041 &
|
||||
```
|
||||
|
||||
Use curl to verify that the TDengine REST API is working on port 6041:
|
||||
Use **curl** to verify that the TDengine REST API is working on port 6041:
|
||||
|
||||
```
|
||||
$ curl -u root:taosdata -d "show databases" 127.0.0.1:6041/rest/sql
|
||||
Handling connection for 6041
|
||||
{"code":0,"column_meta":[["name","VARCHAR",64],["create_time","TIMESTAMP",8],["vgroups","SMALLINT",2],["ntables","BIGINT",8],["replica","TINYINT",1],["strict","VARCHAR",4],["duration","VARCHAR",10],["keep","VARCHAR",32],["buffer","INT",4],["pagesize","INT",4],["pages","INT",4],["minrows","INT",4],["maxrows","INT",4],["comp","TINYINT",1],["precision","VARCHAR",2],["status","VARCHAR",10],["retention","VARCHAR",60],["single_stable","BOOL",1],["cachemodel","VARCHAR",11],["cachesize","INT",4],["wal_level","TINYINT",1],["wal_fsync_period","INT",4],["wal_retention_period","INT",4],["wal_retention_size","BIGINT",8],["wal_roll_period","INT",4],["wal_segment_size","BIGINT",8]],"data":[["information_schema",null,null,16,null,null,null,null,null,null,null,null,null,null,null,"ready",null,null,null,null,null,null,null,null,null,null],["performance_schema",null,null,10,null,null,null,null,null,null,null,null,null,null,null,"ready",null,null,null,null,null,null,null,null,null,null]],"rows":2}
|
||||
```bash
|
||||
curl -u root:taosdata -d "show databases" 127.0.0.1:6041/rest/sql
|
||||
{"code":0,"column_meta":[["name","VARCHAR",64]],"data":[["information_schema"],["performance_schema"],["test"],["test1"]],"rows":4}
|
||||
```
|
||||
|
||||
## Enable the dashboard for visualization
|
||||
## Test cluster
|
||||
|
||||
The minikube dashboard command enables visualized cluster management.
|
||||
### Data preparation
|
||||
|
||||
```
|
||||
$ minikube dashboard
|
||||
* Verifying dashboard health ...
|
||||
* Launching proxy ...
|
||||
* Verifying proxy health ...
|
||||
* Opening http://127.0.0.1:46617/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...
|
||||
http://127.0.0.1:46617/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
|
||||
#### taosBenchmark
|
||||
|
||||
Create a 3 replicas database with taosBenchmark, write 100 million data at the same time, and view the data at the same time
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taosBenchmark -I stmt -d test -n 10000 -t 10000 -a 3
|
||||
|
||||
# query data
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "select count(*) from test.meters;"
|
||||
|
||||
taos> select count(*) from test.meters;
|
||||
count(*) |
|
||||
========================
|
||||
100000000 |
|
||||
Query OK, 1 row(s) in set (0.103537s)
|
||||
```
|
||||
|
||||
In some public clouds, minikube cannot be remotely accessed if it is bound to 127.0.0.1. In this case, use the kubectl proxy command to map the port to 0.0.0.0. Then, you can access the dashboard by using a web browser to open the dashboard URL above on the public IP address and port of the virtual machine.
|
||||
View vnode distribution by showing dnodes
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 8 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-19 17:54:18.469 | | | |
|
||||
2 | tdengine-1.ta... | 8 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-19 17:54:38.698 | | | |
|
||||
3 | tdengine-2.ta... | 8 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-19 17:55:02.039 | | | |
|
||||
Query OK, 3 row(s) in set (0.001357s)
|
||||
```
|
||||
$ kubectl proxy --accept-hosts='^.*$' --address='0.0.0.0'
|
||||
|
||||
View xnode distribution by showing vgroup
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show test.vgroups"
|
||||
|
||||
taos> show test.vgroups
|
||||
vgroup_id | db_name | tables | v1_dnode | v1_status | v2_dnode | v2_status | v3_dnode | v3_status | v4_dnode | v4_status | cacheload | cacheelements | tsma |
|
||||
==============================================================================================================================================================================================
|
||||
2 | test | 1267 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
3 | test | 1215 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
4 | test | 1215 | 1 | leader | 2 | follower | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
5 | test | 1307 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
6 | test | 1245 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
7 | test | 1275 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
8 | test | 1231 | 1 | leader | 2 | follower | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
9 | test | 1245 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
Query OK, 8 row(s) in set (0.001488s)
|
||||
```
|
||||
|
||||
#### Manually created
|
||||
|
||||
Common a three-copy test1, and create a table, write 2 pieces of data
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- \
|
||||
taos -s \
|
||||
"create database if not exists test1 replica 3;
|
||||
use test1;
|
||||
create table if not exists t1(ts timestamp, n int);
|
||||
insert into t1 values(now, 1)(now+1s, 2);"
|
||||
```
|
||||
|
||||
View xnode distribution by showing test1.vgroup
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show test1.vgroups"
|
||||
|
||||
taos> show test1.vgroups
|
||||
vgroup_id | db_name | tables | v1_dnode | v1_status | v2_dnode | v2_status | v3_dnode | v3_status | v4_dnode | v4_status | cacheload | cacheelements | tsma |
|
||||
==============================================================================================================================================================================================
|
||||
10 | test1 | 1 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
11 | test1 | 0 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
Query OK, 2 row(s) in set (0.001489s)
|
||||
```
|
||||
|
||||
### Test fault tolerance
|
||||
|
||||
The dnode where the mnode leader is located is disconnected, dnode1
|
||||
|
||||
```Bash
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 0/1 ErrImagePull 2 (2s ago) 20m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (6m48s ago) 20m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 21m 10.244.1.223 node85 <none> <none>
|
||||
```
|
||||
|
||||
At this time, the cluster mnode has a re-election, and the monde on dnode1 becomes the leader.
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show mnodes\G"
|
||||
Welcome to the TDengine Command Line Interface, Client Version:3.0.7.1.202307190706
|
||||
Copyright (c) 2022 by TDengine, all rights reserved.
|
||||
|
||||
taos> show mnodes\G
|
||||
*************************** 1.row ***************************
|
||||
id: 1
|
||||
endpoint: tdengine-0.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: offline
|
||||
status: offline
|
||||
create_time: 2023-07-19 17:54:18.559
|
||||
reboot_time: 1970-01-01 08:00:00.000
|
||||
*************************** 2.row ***************************
|
||||
id: 2
|
||||
endpoint: tdengine-1.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: leader
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:05.600
|
||||
reboot_time: 2023-07-20 09:32:00.227
|
||||
*************************** 3.row ***************************
|
||||
id: 3
|
||||
endpoint: tdengine-2.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: follower
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:20.042
|
||||
reboot_time: 2023-07-20 09:32:00.026
|
||||
Query OK, 3 row(s) in set (0.001513s)
|
||||
```
|
||||
|
||||
Cluster can read and write normally
|
||||
|
||||
```Bash
|
||||
# insert
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "insert into test1.t1 values(now, 1)(now+1s, 2);"
|
||||
|
||||
taos> insert into test1.t1 values(now, 1)(now+1s, 2);
|
||||
Insert OK, 2 row(s) affected (0.002098s)
|
||||
|
||||
# select
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "select *from test1.t1"
|
||||
|
||||
taos> select *from test1.t1
|
||||
ts | n |
|
||||
========================================
|
||||
2023-07-19 18:04:58.104 | 1 |
|
||||
2023-07-19 18:04:59.104 | 2 |
|
||||
2023-07-19 18:06:00.303 | 1 |
|
||||
2023-07-19 18:06:01.303 | 2 |
|
||||
Query OK, 4 row(s) in set (0.001994s)
|
||||
```
|
||||
|
||||
Similarly, as for the non-leader mnode dropped, read and write can of course be normal, here will not do too much display .
|
||||
|
||||
## Scaling Out Your Cluster
|
||||
|
||||
TDengine clusters can scale automatically:
|
||||
TDengine cluster supports automatic expansion:
|
||||
|
||||
```bash
|
||||
```Bash
|
||||
kubectl scale statefulsets tdengine --replicas=4
|
||||
```
|
||||
|
||||
The preceding command increases the number of replicas to 4. After running this command, query the pod status:
|
||||
The parameter `--replica = 4 `in the above command line indicates that you want to expand the TDengine cluster to 4 nodes. After execution, first check the status of the Pod:
|
||||
|
||||
```bash
|
||||
kubectl get pods -l app=tdengine
|
||||
```Bash
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
```
|
||||
|
||||
The output is as follows:
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 161m
|
||||
tdengine-1 1/1 Running 0 161m
|
||||
tdengine-2 1/1 Running 0 32m
|
||||
tdengine-3 1/1 Running 0 32m
|
||||
```Plain
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 1/1 Running 4 (6h26m ago) 6h53m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (6h39m ago) 6h53m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 5h16m 10.244.1.224 node85 <none> <none>
|
||||
tdengine-3 1/1 Running 0 3m24s 10.244.2.76 node86 <none> <none>
|
||||
```
|
||||
|
||||
The status of all pods is Running. Once the pod status changes to Ready, you can check the dnode status:
|
||||
At this time, the state of the POD is still Running, and the dnode state in the TDengine cluster can only be seen after the Pod status is `ready `:
|
||||
|
||||
```bash
|
||||
kubectl exec -i -t tdengine-3 -- taos -s "show dnodes"
|
||||
```Bash
|
||||
kubectl exec -it tdengine-3 -n tdengine-test -- taos -s "show dnodes"
|
||||
```
|
||||
|
||||
The following output shows that the TDengine cluster has been expanded to 4 replicas:
|
||||
The dnode list of the expanded four-node TDengine cluster:
|
||||
|
||||
```
|
||||
```Plain
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||||
============================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||||
4 | tdengine-3.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:33:16.039 | |
|
||||
Query OK, 4 rows in database (0.008377s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
4 | tdengine-3.ta... | 0 | 16 | ready | 2023-07-20 16:01:44.007 | 2023-07-20 16:01:44.889 | | | |
|
||||
Query OK, 4 row(s) in set (0.003628s)
|
||||
```
|
||||
|
||||
## Scaling In Your Cluster
|
||||
|
||||
When you scale in a TDengine cluster, your data is migrated to different nodes. You must run the drop dnodes command in TDengine to remove dnodes before scaling in your Kubernetes environment.
|
||||
Since the TDengine cluster will migrate data between nodes during volume expansion and contraction, using the **kubectl** command to reduce the volume requires first using the "drop dnodes" command ( **If there are 3 replicas of db in the cluster, the number of dnodes after reduction must also be greater than or equal to 3, otherwise the drop dnode operation will be aborted** ), the node deletion is completed before Kubernetes cluster reduction.
|
||||
|
||||
Note: In a Kubernetes StatefulSet service, the newest pods are always removed first. For this reason, when you scale in your TDengine cluster, ensure that you drop the newest dnodes.
|
||||
Note: Since Kubernetes Pods in the Statefulset can only be removed in reverse order of creation, the TDengine drop dnode also needs to be removed in reverse order of creation, otherwise the Pod will be in an error state.
|
||||
|
||||
```
|
||||
$ kubectl exec -i -t tdengine-0 -- taos -s "drop dnode 4"
|
||||
```
|
||||
|
||||
```bash
|
||||
$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "drop dnode 4"
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||||
============================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||||
Query OK, 3 rows in database (0.004861s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
Query OK, 3 row(s) in set (0.003324s)
|
||||
```
|
||||
|
||||
Verify that the dnode have been successfully removed by running the `kubectl exec -i -t tdengine-0 -- taos -s "show dnodes"` command. Then run the following command to remove the pod:
|
||||
After confirming that the removal is successful (use kubectl exec -i -t tdengine-0 --taos -s "show dnodes" to view and confirm the dnode list), use the kubectl command to remove the Pod:
|
||||
|
||||
```
|
||||
kubectl scale statefulsets tdengine --replicas=3
|
||||
```Plain
|
||||
kubectl scale statefulsets tdengine --replicas=3 -n tdengine-test
|
||||
```
|
||||
|
||||
The newest pod in the deployment is removed. Run the `kubectl get pods -l app=tdengine` command to query the pod status:
|
||||
The last Pod will be deleted. Use the command kubectl get pods -l app = tdengine to check the Pod status:
|
||||
|
||||
```
|
||||
$ kubectl get pods -l app=tdengine
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 4m7s
|
||||
tdengine-1 1/1 Running 0 3m55s
|
||||
tdengine-2 1/1 Running 0 2m28s
|
||||
```Plain
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 1/1 Running 4 (6h55m ago) 7h22m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (7h9m ago) 7h23m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 5h45m 10.244.1.224 node85 <none> <none>
|
||||
```
|
||||
|
||||
After the pod has been removed, manually delete the PersistentVolumeClaim (PVC). Otherwise, future scale-outs will attempt to use existing data.
|
||||
After the Pod is deleted, the PVC needs to be deleted manually, otherwise the previous data will continue to be used for the next expansion, resulting in the inability to join the cluster normally.
|
||||
|
||||
```bash
|
||||
$ kubectl delete pvc taosdata-tdengine-3
|
||||
```Bash
|
||||
kubectl delete pvc aosdata-tdengine-3 -n tdengine-test
|
||||
```
|
||||
|
||||
Your cluster has now been safely scaled in, and you can scale it out again as necessary.
|
||||
The cluster state at this time is safe and can be scaled up again if needed.
|
||||
|
||||
```bash
|
||||
$ kubectl scale statefulsets tdengine --replicas=4
|
||||
```Bash
|
||||
kubectl scale statefulsets tdengine --replicas=4 -n tdengine-test
|
||||
statefulset.apps/tdengine scaled
|
||||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl get pods -l app=tdengine
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 35m
|
||||
tdengine-1 1/1 Running 0 34m
|
||||
tdengine-2 1/1 Running 0 12m
|
||||
tdengine-3 0/1 ContainerCreating 0 4s
|
||||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl get pods -l app=tdengine
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 35m
|
||||
tdengine-1 1/1 Running 0 34m
|
||||
tdengine-2 1/1 Running 0 12m
|
||||
tdengine-3 0/1 Running 0 7s
|
||||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||||
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 1/1 Running 4 (6h59m ago) 7h27m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (7h13m ago) 7h27m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 5h49m 10.244.1.224 node85 <none> <none>
|
||||
tdengine-3 1/1 Running 0 20s 10.244.2.77 node86 <none> <none>
|
||||
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 17:38:49.012 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 4 | ready | 2022-07-25 17:39:01.517 | |
|
||||
5 | tdengine-2.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 18:01:36.479 | |
|
||||
6 | tdengine-3.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 18:13:54.411 | |
|
||||
Query OK, 4 row(s) in set (0.001348s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
5 | tdengine-3.ta... | 0 | 16 | ready | 2023-07-20 16:31:34.092 | 2023-07-20 16:38:17.419 | | | |
|
||||
Query OK, 4 row(s) in set (0.003881s)
|
||||
```
|
||||
|
||||
## Remove a TDengine Cluster
|
||||
|
||||
To fully remove a TDengine cluster, you must delete its statefulset, svc, configmap, and pvc entries:
|
||||
> **When deleting the PVC, you need to pay attention to the pv persistentVolumeReclaimPolicy policy. It is recommended to change to Delete, so that the PV will be automatically cleaned up when the PVC is deleted, and the underlying CSI storage resources will be cleaned up at the same time. If the policy of deleting the PVC to automatically clean up the PV is not configured, and then after deleting the pvc, when manually cleaning up the PV, the CSI storage resources corresponding to the PV may not be released.**
|
||||
|
||||
```bash
|
||||
kubectl delete statefulset -l app=tdengine
|
||||
kubectl delete svc -l app=tdengine
|
||||
kubectl delete pvc -l app=tdengine
|
||||
kubectl delete configmap taoscfg
|
||||
Complete removal of TDengine cluster, need to clean up statefulset, svc, configmap, pvc respectively.
|
||||
|
||||
```Bash
|
||||
kubectl delete statefulset -l app=tdengine -n tdengine-test
|
||||
kubectl delete svc -l app=tdengine -n tdengine-test
|
||||
kubectl delete pvc -l app=tdengine -n tdengine-test
|
||||
kubectl delete configmap taoscfg -n tdengine-test
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Error 1
|
||||
|
||||
If you remove a pod without first running `drop dnode`, some TDengine nodes will go offline.
|
||||
No "drop dnode" is directly reduced. Since the TDengine has not deleted the node, the reduced pod causes some nodes in the TDengine cluster to be offline.
|
||||
|
||||
```
|
||||
$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||||
```Plain
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 17:38:49.012 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 4 | ready | 2022-07-25 17:39:01.517 | |
|
||||
5 | tdengine-2.taosd.default.sv... | 0 | 4 | offline | 2022-07-25 18:01:36.479 | status msg timeout |
|
||||
6 | tdengine-3.taosd.default.sv... | 0 | 4 | offline | 2022-07-25 18:13:54.411 | status msg timeout |
|
||||
Query OK, 4 row(s) in set (0.001323s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
5 | tdengine-3.ta... | 0 | 16 | offline | 2023-07-20 16:31:34.092 | 2023-07-20 16:38:17.419 | status msg timeout | | |
|
||||
Query OK, 4 row(s) in set (0.003862s)
|
||||
```
|
||||
|
||||
### Error 2
|
||||
## Finally
|
||||
|
||||
If the number of nodes after a scale-in is less than the value of the replica parameter, the cluster will go down:
|
||||
For the high availability and high reliability of TDengine in a Kubernetes environment, hardware damage and disaster recovery are divided into two levels:
|
||||
|
||||
Create a database with replica set to 2 and add data.
|
||||
1. The disaster recovery capability of the underlying distributed Block Storage, the multi-copy of Block Storage, the current popular distributed Block Storage such as Ceph, has the multi-copy capability, extending the storage copy to different racks, cabinets, computer rooms, Data center (or directly use the Block Storage service provided by Public Cloud vendors)
|
||||
2. TDengine disaster recovery, in TDengine Enterprise, itself has when a dnode permanently offline (TCE-metal disk damage, data sorting loss), re-pull a blank dnode to restore the original dnode work.
|
||||
|
||||
```bash
|
||||
kubectl exec -i -t tdengine-0 -- \
|
||||
taos -s \
|
||||
"create database if not exists test replica 2;
|
||||
use test;
|
||||
create table if not exists t1(ts timestamp, n int);
|
||||
insert into t1 values(now, 1)(now+1s, 2);"
|
||||
Finally, welcome to [TDengine Cloud ](https://cloud.tdengine.com/)to experience the one-stop fully managed TDengine Cloud as a Service.
|
||||
|
||||
|
||||
```
|
||||
|
||||
Scale in to one node:
|
||||
|
||||
```bash
|
||||
kubectl scale statefulsets tdengine --replicas=1
|
||||
|
||||
```
|
||||
|
||||
In the TDengine CLI, you can see that no database operations succeed:
|
||||
|
||||
```
|
||||
taos> show dnodes;
|
||||
id | end_point | vnodes | cores | status | role | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 2 | 40 | ready | any | 2021-06-01 15:55:52.562 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 40 | offline | any | 2021-06-01 15:56:07.212 | status msg timeout |
|
||||
Query OK, 2 row(s) in set (0.000845s)
|
||||
|
||||
taos> show dnodes;
|
||||
id | end_point | vnodes | cores | status | role | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 2 | 40 | ready | any | 2021-06-01 15:55:52.562 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 40 | offline | any | 2021-06-01 15:56:07.212 | status msg timeout |
|
||||
Query OK, 2 row(s) in set (0.000837s)
|
||||
|
||||
taos> use test;
|
||||
Database changed.
|
||||
|
||||
taos> insert into t1 values(now, 3);
|
||||
|
||||
DB error: Unable to resolve FQDN (0.013874s)
|
||||
|
||||
```
|
||||
> TDengine Cloud is a minimalist fully managed time series data processing Cloud as a Service platform developed based on the open source time series database TDengine. In addition to high-performance time series database, it also has system functions such as caching, subscription and stream computing, and provides convenient and secure data sharing, as well as numerous enterprise-level functions. It allows enterprises in the fields of Internet of Things, Industrial Internet, Finance, IT operation and maintenance monitoring to significantly reduce labor costs and operating costs in the management of time series data.
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ database_option: {
|
|||
- WAL_FSYNC_PERIOD: specifies the interval (in milliseconds) at which data is written from the WAL to disk. This parameter takes effect only when the WAL parameter is set to 2. The default value is 3000. Enter a value between 0 and 180000. The value 0 indicates that incoming data is immediately written to disk.
|
||||
- MAXROWS: specifies the maximum number of rows recorded in a block. The default value is 4096.
|
||||
- MINROWS: specifies the minimum number of rows recorded in a block. The default value is 100.
|
||||
- KEEP: specifies the time for which data is retained. Enter a value between 1 and 365000. The default value is 3650. The value of the KEEP parameter must be greater than or equal to the value of the DURATION parameter. TDengine automatically deletes data that is older than the value of the KEEP parameter. You can use m (minutes), h (hours), and d (days) as the unit, for example KEEP 100h or KEEP 10d. If you do not include a unit, d is used by default. The Enterprise Edition supports [Tiered Storage](https://docs.tdengine.com/tdinternal/arch/#tiered-storage) function, thus multiple KEEP values (comma separated and up to 3 values supported, and meet keep 0 <= keep 1 <= keep 2, e.g. KEEP 100h,100d,3650d) are supported; the Community Edition does not support Tiered Storage function (although multiple keep values are configured, they do not take effect, only the maximum keep value is used as KEEP).
|
||||
- KEEP: specifies the time for which data is retained. Enter a value between 1 and 365000. The default value is 3650. The value of the KEEP parameter must be greater than or equal to the value of the DURATION parameter. TDengine automatically deletes data that is older than the value of the KEEP parameter. You can use m (minutes), h (hours), and d (days) as the unit, for example KEEP 100h or KEEP 10d. If you do not include a unit, d is used by default. TDengine Enterprise supports [Tiered Storage](https://docs.tdengine.com/tdinternal/arch/#tiered-storage) function, thus multiple KEEP values (comma separated and up to 3 values supported, and meet keep 0 <= keep 1 <= keep 2, e.g. KEEP 100h,100d,3650d) are supported; TDengine OSS does not support Tiered Storage function (although multiple keep values are configured, they do not take effect, only the maximum keep value is used as KEEP).
|
||||
- PAGES: specifies the number of pages in the metadata storage engine cache on each vnode. Enter a value greater than or equal to 64. The default value is 256. The space occupied by metadata storage on each vnode is equal to the product of the values of the PAGESIZE and PAGES parameters. The space occupied by default is 1 MB.
|
||||
- PAGESIZE: specifies the size (in KB) of each page in the metadata storage engine cache on each vnode. The default value is 4. Enter a value between 1 and 16384.
|
||||
- PRECISION: specifies the precision at which a database records timestamps. Enter ms for milliseconds, us for microseconds, or ns for nanoseconds. The default value is ms.
|
||||
|
|
|
|||
|
|
@ -214,19 +214,6 @@ The data of tdinsight dashboard is stored in `log` database (default. You can ch
|
|||
|dnode\_ep|NCHAR|TAG|dnode endpoint|
|
||||
|cluster\_id|NCHAR|TAG|cluster id|
|
||||
|
||||
### logs table
|
||||
|
||||
`logs` table contains login information records.
|
||||
|
||||
|field|type|is\_tag|comment|
|
||||
|:----|:---|:-----|:------|
|
||||
|ts|TIMESTAMP||timestamp|
|
||||
|level|VARCHAR||log level|
|
||||
|content|NCHAR||log content|
|
||||
|dnode\_id|INT|TAG|dnode id|
|
||||
|dnode\_ep|NCHAR|TAG|dnode endpoint|
|
||||
|cluster\_id|NCHAR|TAG|cluster id|
|
||||
|
||||
### log\_summary table
|
||||
|
||||
`log_summary` table contains log summary information records.
|
||||
|
|
|
|||
|
|
@ -648,12 +648,12 @@ stmt.execute()?;
|
|||
//stmt.execute()?;
|
||||
```
|
||||
|
||||
For a working example, see [GitHub](https://github.com/taosdata/taos-connector-rust/blob/main/examples/bind.rs).
|
||||
For a working example, see [GitHub](https://github.com/taosdata/taos-connector-rust/blob/main/taos/examples/bind.rs).
|
||||
|
||||
|
||||
For information about other structure APIs, see the [Rust documentation](https://docs.rs/taos).
|
||||
|
||||
[taos]: https://github.com/taosdata/rust-connector-taos
|
||||
[taos]: https://github.com/taosdata/taos-connector-rust
|
||||
[r2d2]: https://crates.io/crates/r2d2
|
||||
[TaosBuilder]: https://docs.rs/taos/latest/taos/struct.TaosBuilder.html
|
||||
[TaosCfg]: https://docs.rs/taos/latest/taos/struct.TaosCfg.html
|
||||
|
|
|
|||
|
|
@ -1007,13 +1007,12 @@ consumer.close()
|
|||
### Other sample programs
|
||||
|
||||
| Example program links | Example program content |
|
||||
| ------------------------------------------------------------------------------------------------------------- | ------------------- ---- |
|
||||
| [bind_multi.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/bind-multi.py) | parameter binding,
|
||||
bind multiple rows at once |
|
||||
| [bind_row.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/bind-row.py) | bind_row.py
|
||||
|-----------------------|-------------------------|
|
||||
| [bind_multi.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/bind-multi.py) | parameter binding, bind multiple rows at once |
|
||||
| [bind_row.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/bind-row.py) | parameter binding, bind one row at once |
|
||||
| [insert_lines.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/insert-lines.py) | InfluxDB line protocol writing |
|
||||
| [json_tag.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/json-tag.py) | Use JSON type tags |
|
||||
| [tmq.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/tmq.py) | TMQ subscription |
|
||||
| [tmq_consumer.py](https://github.com/taosdata/taos-connector-python/blob/main/examples/tmq_consumer.py) | TMQ subscription |
|
||||
|
||||
## Other notes
|
||||
|
||||
|
|
|
|||
|
|
@ -364,6 +364,7 @@ The configuration parameters for specifying super table tag columns and data col
|
|||
- **min**: The minimum value of the column/label of the data type. The generated value will equal or large than the minimum value.
|
||||
|
||||
- **max**: The maximum value of the column/label of the data type. The generated value will less than the maximum value.
|
||||
- **fun**: This column of data is filled with functions. Currently, only the sin and cos functions are supported. The input parameter is the timestamp and converted to an angle value. The conversion formula is: angle x=input time column ts value % 360. At the same time, it supports coefficient adjustment and random fluctuation factor adjustment, presented in a fixed format expression, such as fun="10\*sin(x)+100\*random(5)", where x represents the angle, ranging from 0 to 360 degrees, and the growth step size is consistent with the time column step size. 10 represents the coefficient of multiplication, 100 represents the coefficient of addition or subtraction, and 5 represents the fluctuation range within a random range of 5%. The currently supported data types are int, bigint, float, and double. Note: The expression is fixed and cannot be reversed.
|
||||
|
||||
- **values**: The value field of the nchar/binary column/label, which will be chosen randomly from the values.
|
||||
|
||||
|
|
|
|||
|
|
@ -6,7 +6,7 @@ description: This document provides download links for all released versions of
|
|||
|
||||
TDengine 3.x installation packages can be downloaded at the following links:
|
||||
|
||||
For TDengine 2.x installation packages by version, please visit [here](https://www.taosdata.com/all-downloads).
|
||||
For TDengine 2.x installation packages by version, please visit [here](https://tdengine.com/downloads/historical/).
|
||||
|
||||
import Release from "/components/ReleaseV3";
|
||||
|
||||
|
|
|
|||
|
|
@ -4,23 +4,31 @@ title: 在 Kubernetes 上部署 TDengine 集群
|
|||
description: 利用 Kubernetes 部署 TDengine 集群的详细指南
|
||||
---
|
||||
|
||||
作为面向云原生架构设计的时序数据库，TDengine 支持 Kubernetes 部署。这里介绍如何使用 YAML 文件一步一步从头创建一个 TDengine 集群，并重点介绍 Kubernetes 环境下 TDengine 的常用操作。
|
||||
## 概述
|
||||
|
||||
作为面向云原生架构设计的时序数据库，TDengine 本身就支持 Kubernetes 部署。这里介绍如何使用 YAML 文件从头一步一步创建一个可用于生产使用的高可用 TDengine 集群，并重点介绍 Kubernetes 环境下 TDengine 的常用操作。
|
||||
|
||||
为了满足[高可用](https://docs.taosdata.com/tdinternal/high-availability/)的需求，集群需要满足如下要求：
|
||||
|
||||
- 3个及以上 dnode ：TDengine 的同一个 vgroup 中的多个 vnode ，不允许同时分布在一个 dnode ，所以如果创建3副本的数据库，则 dnode 数大于等于3
|
||||
- 3个 mnode ：mnode 负责整个集群的管理工作，TDengine 默认是一个 mnode。如果这个 mnode 所在的 dnode 掉线，则整个集群不可用。
|
||||
- 数据库的3副本：TDengine 的副本配置是数据库级别，所以数据库3副本可满足在3个 dnode 的集群中，任意一个 dnode 下线，都不影响集群的正常使用。**如果下线** **dnode** **个数为2时，此时集群不可用，****因为****RAFT无法完成选举****。**（企业版：在灾难恢复场景，任一节点数据文件损坏，都可以通过重新拉起dnode进行恢复）
|
||||
|
||||
## 前置条件
|
||||
|
||||
要使用 Kubernetes 部署管理 TDengine 集群，需要做好如下准备工作。
|
||||
|
||||
* 本文适用 Kubernetes v1.5 以上版本
|
||||
* 本文和下一章使用 minikube、kubectl 和 helm 等工具进行安装部署，请提前安装好相应软件
|
||||
* Kubernetes 已经安装部署并能正常访问使用或更新必要的容器仓库或其他服务
|
||||
- 本文适用 Kubernetes v1.19 以上版本
|
||||
- 本文使用 kubectl 工具进行安装部署，请提前安装好相应软件
|
||||
- Kubernetes 已经安装部署并能正常访问使用或更新必要的容器仓库或其他服务
|
||||
|
||||
以下配置文件也可以从 [GitHub 仓库](https://github.com/taosdata/TDengine-Operator/tree/3.0/src/tdengine) 下载。
|
||||
|
||||
## 配置 Service 服务
|
||||
|
||||
创建一个 Service 配置文件：`taosd-service.yaml`，服务名称 `metadata.name` (此处为 "taosd") 将在下一步中使用到。添加 TDengine 所用到的端口：
|
||||
创建一个 Service 配置文件：`taosd-service.yaml`，服务名称 `metadata.name` (此处为 "taosd") 将在下一步中使用到。首先添加 TDengine 所用到的端口，然后在选择器设置确定的标签 app (此处为 “tdengine”)。
|
||||
|
||||
```yaml
|
||||
```YAML
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
|
|
@ -42,10 +50,11 @@ spec:
|
|||
|
||||
## 有状态服务 StatefulSet
|
||||
|
||||
根据 Kubernetes 对各类部署的说明，我们将使用 StatefulSet 作为 TDengine 的服务类型。
|
||||
创建文件 `tdengine.yaml`，其中 replicas 定义集群节点的数量为 3。节点时区为中国（Asia/Shanghai），每个节点分配 10G 标准（standard）存储。你也可以根据实际情况进行相应修改。
|
||||
根据 Kubernetes 对各类部署的说明，我们将使用 StatefulSet 作为 TDengine 的部署资源类型。 创建文件 `tdengine.yaml`，其中 replicas 定义集群节点的数量为 3。节点时区为中国（Asia/Shanghai），每个节点分配 5G 标准（standard）存储（参考[Storage Classes](https://kubernetes.io/docs/concepts/storage/storage-classes/) 配置 storage class ）。你也可以根据实际情况进行相应修改。
|
||||
|
||||
```yaml
|
||||
请特别注意startupProbe的配置，在 dnode 的 Pod 掉线一段时间后，再重新启动，这个时候新上线的 dnode 会短暂不可用。如果startupProbe配置过小，Kubernetes 会认为该 Pod 处于不正常的状态，并尝试重启该 Pod，该 dnode 的 Pod 会频繁重启，始终无法恢复到正常状态。参考 [Configure Liveness, Readiness and Startup Probes](https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/)
|
||||
|
||||
```YAML
|
||||
---
|
||||
apiVersion: apps/v1
|
||||
kind: StatefulSet
|
||||
|
|
@ -69,7 +78,7 @@ spec:
|
|||
spec:
|
||||
containers:
|
||||
- name: "tdengine"
|
||||
image: "tdengine/tdengine:3.0.0.0"
|
||||
image: "tdengine/tdengine:3.0.7.1"
|
||||
imagePullPolicy: "IfNotPresent"
|
||||
ports:
|
||||
- name: tcp6030
|
||||
|
|
@ -108,6 +117,12 @@ spec:
|
|||
volumeMounts:
|
||||
- name: taosdata
|
||||
mountPath: /var/lib/taos
|
||||
startupProbe:
|
||||
exec:
|
||||
command:
|
||||
- taos-check
|
||||
failureThreshold: 360
|
||||
periodSeconds: 10
|
||||
readinessProbe:
|
||||
exec:
|
||||
command:
|
||||
|
|
@ -129,199 +144,373 @@ spec:
|
|||
storageClassName: "standard"
|
||||
resources:
|
||||
requests:
|
||||
storage: "10Gi"
|
||||
storage: "5Gi"
|
||||
```
|
||||
|
||||
## 使用 kubectl 命令部署 TDengine 集群
|
||||
|
||||
顺序执行以下命令。
|
||||
首先创建对应的 namespace，然后顺序执行以下命令：
|
||||
|
||||
```bash
|
||||
kubectl apply -f taosd-service.yaml
|
||||
kubectl apply -f tdengine.yaml
|
||||
```Bash
|
||||
kubectl apply -f taosd-service.yaml -n tdengine-test
|
||||
kubectl apply -f tdengine.yaml -n tdengine-test
|
||||
```
|
||||
|
||||
上面的配置将生成一个三节点的 TDengine 集群，dnode 为自动配置，可以使用 show dnodes 命令查看当前集群的节点：
|
||||
|
||||
```bash
|
||||
kubectl exec -i -t tdengine-0 -- taos -s "show dnodes"
|
||||
kubectl exec -i -t tdengine-1 -- taos -s "show dnodes"
|
||||
kubectl exec -i -t tdengine-2 -- taos -s "show dnodes"
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show dnodes"
|
||||
kubectl exec -it tdengine-2 -n tdengine-test -- taos -s "show dnodes"
|
||||
```
|
||||
|
||||
输出如下：
|
||||
|
||||
```
|
||||
```Bash
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||||
============================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||||
Query OK, 3 rows in database (0.003655s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 0 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-19 17:54:18.469 | | | |
|
||||
2 | tdengine-1.ta... | 0 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-19 17:54:38.698 | | | |
|
||||
3 | tdengine-2.ta... | 0 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-19 17:55:02.039 | | | |
|
||||
Query OK, 3 row(s) in set (0.001853s)
|
||||
```
|
||||
|
||||
查看当前mnode
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show mnodes\G"
|
||||
taos> show mnodes\G
|
||||
*************************** 1.row ***************************
|
||||
id: 1
|
||||
endpoint: tdengine-0.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: leader
|
||||
status: ready
|
||||
create_time: 2023-07-19 17:54:18.559
|
||||
reboot_time: 2023-07-19 17:54:19.520
|
||||
Query OK, 1 row(s) in set (0.001282s)
|
||||
```
|
||||
|
||||
## 创建mnode
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "create mnode on dnode 2"
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "create mnode on dnode 3"
|
||||
```
|
||||
|
||||
查看mnode
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show mnodes\G"
|
||||
|
||||
taos> show mnodes\G
|
||||
*************************** 1.row ***************************
|
||||
id: 1
|
||||
endpoint: tdengine-0.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: leader
|
||||
status: ready
|
||||
create_time: 2023-07-19 17:54:18.559
|
||||
reboot_time: 2023-07-20 09:19:36.060
|
||||
*************************** 2.row ***************************
|
||||
id: 2
|
||||
endpoint: tdengine-1.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: follower
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:05.600
|
||||
reboot_time: 2023-07-20 09:22:12.838
|
||||
*************************** 3.row ***************************
|
||||
id: 3
|
||||
endpoint: tdengine-2.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: follower
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:20.042
|
||||
reboot_time: 2023-07-20 09:22:23.271
|
||||
Query OK, 3 row(s) in set (0.003108s)
|
||||
```
|
||||
|
||||
## 使能端口转发
|
||||
|
||||
利用 kubectl 端口转发功能可以使应用可以访问 Kubernetes 环境运行的 TDengine 集群。
|
||||
|
||||
```
|
||||
kubectl port-forward tdengine-0 6041:6041 &
|
||||
```Plain
|
||||
kubectl port-forward -n tdengine-test tdengine-0 6041:6041 &
|
||||
```
|
||||
|
||||
使用 curl 命令验证 TDengine REST API 使用的 6041 接口。
|
||||
|
||||
```
|
||||
$ curl -u root:taosdata -d "show databases" 127.0.0.1:6041/rest/sql
|
||||
Handling connection for 6041
|
||||
{"code":0,"column_meta":[["name","VARCHAR",64],["create_time","TIMESTAMP",8],["vgroups","SMALLINT",2],["ntables","BIGINT",8],["replica","TINYINT",1],["strict","VARCHAR",4],["duration","VARCHAR",10],["keep","VARCHAR",32],["buffer","INT",4],["pagesize","INT",4],["pages","INT",4],["minrows","INT",4],["maxrows","INT",4],["comp","TINYINT",1],["precision","VARCHAR",2],["status","VARCHAR",10],["retention","VARCHAR",60],["single_stable","BOOL",1],["cachemodel","VARCHAR",11],["cachesize","INT",4],["wal_level","TINYINT",1],["wal_fsync_period","INT",4],["wal_retention_period","INT",4],["wal_retention_size","BIGINT",8],["wal_roll_period","INT",4],["wal_segment_size","BIGINT",8]],"data":[["information_schema",null,null,16,null,null,null,null,null,null,null,null,null,null,null,"ready",null,null,null,null,null,null,null,null,null,null],["performance_schema",null,null,10,null,null,null,null,null,null,null,null,null,null,null,"ready",null,null,null,null,null,null,null,null,null,null]],"rows":2}
|
||||
```Plain
|
||||
curl -u root:taosdata -d "show databases" 127.0.0.1:6041/rest/sql
|
||||
{"code":0,"column_meta":[["name","VARCHAR",64]],"data":[["information_schema"],["performance_schema"],["test"],["test1"]],"rows":4}
|
||||
```
|
||||
|
||||
## 使用 dashboard 进行图形化管理
|
||||
## 集群测试
|
||||
|
||||
minikube 提供 dashboard 命令支持图形化管理界面。
|
||||
### 数据准备
|
||||
|
||||
```
|
||||
$ minikube dashboard
|
||||
* Verifying dashboard health ...
|
||||
* Launching proxy ...
|
||||
* Verifying proxy health ...
|
||||
* Opening http://127.0.0.1:46617/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...
|
||||
http://127.0.0.1:46617/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
|
||||
#### taosBenchmark
|
||||
|
||||
通过taosBenchmark 创建一个3副本的数据库，同时写入1亿条数据，同时查看数据
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taosBenchmark -I stmt -d test -n 10000 -t 10000 -a 3
|
||||
|
||||
# query data
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "select count(*) from test.meters;"
|
||||
|
||||
taos> select count(*) from test.meters;
|
||||
count(*) |
|
||||
========================
|
||||
100000000 |
|
||||
Query OK, 1 row(s) in set (0.103537s)
|
||||
```
|
||||
|
||||
对于某些公有云环境，minikube 绑定在 127.0.0.1 IP 地址上无法通过远程访问，需要使用 kubectl proxy 命令将端口映射到 0.0.0.0 IP 地址上，再通过浏览器访问虚拟机公网 IP 和端口以及相同的 dashboard URL 路径即可远程访问 dashboard。
|
||||
查看vnode分布，通过show dnodes
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 8 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-19 17:54:18.469 | | | |
|
||||
2 | tdengine-1.ta... | 8 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-19 17:54:38.698 | | | |
|
||||
3 | tdengine-2.ta... | 8 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-19 17:55:02.039 | | | |
|
||||
Query OK, 3 row(s) in set (0.001357s)
|
||||
```
|
||||
$ kubectl proxy --accept-hosts='^.*$' --address='0.0.0.0'
|
||||
|
||||
通过show vgroup 查看 vnode 分布情况
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show test.vgroups"
|
||||
|
||||
taos> show test.vgroups
|
||||
vgroup_id | db_name | tables | v1_dnode | v1_status | v2_dnode | v2_status | v3_dnode | v3_status | v4_dnode | v4_status | cacheload | cacheelements | tsma |
|
||||
==============================================================================================================================================================================================
|
||||
2 | test | 1267 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
3 | test | 1215 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
4 | test | 1215 | 1 | leader | 2 | follower | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
5 | test | 1307 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
6 | test | 1245 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
7 | test | 1275 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
8 | test | 1231 | 1 | leader | 2 | follower | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
9 | test | 1245 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
Query OK, 8 row(s) in set (0.001488s)
|
||||
```
|
||||
|
||||
#### 手工创建
|
||||
|
||||
常见一个三副本的test1，并创建一张表，写入2条数据
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- \
|
||||
taos -s \
|
||||
"create database if not exists test1 replica 3;
|
||||
use test1;
|
||||
create table if not exists t1(ts timestamp, n int);
|
||||
insert into t1 values(now, 1)(now+1s, 2);"
|
||||
```
|
||||
|
||||
通过show test1.vgroup 查看xnode分布情况
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show test1.vgroups"
|
||||
|
||||
taos> show test1.vgroups
|
||||
vgroup_id | db_name | tables | v1_dnode | v1_status | v2_dnode | v2_status | v3_dnode | v3_status | v4_dnode | v4_status | cacheload | cacheelements | tsma |
|
||||
==============================================================================================================================================================================================
|
||||
10 | test1 | 1 | 1 | follower | 2 | follower | 3 | leader | NULL | NULL | 0 | 0 | 0 |
|
||||
11 | test1 | 0 | 1 | follower | 2 | leader | 3 | follower | NULL | NULL | 0 | 0 | 0 |
|
||||
Query OK, 2 row(s) in set (0.001489s)
|
||||
```
|
||||
|
||||
### 容错测试
|
||||
|
||||
Mnode leader 所在的 dnode 掉线，dnode1
|
||||
|
||||
```Bash
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 0/1 ErrImagePull 2 (2s ago) 20m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (6m48s ago) 20m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 21m 10.244.1.223 node85 <none> <none>
|
||||
```
|
||||
|
||||
此时集群mnode发生重新选举，dnode1上的monde 成为leader
|
||||
|
||||
```Bash
|
||||
kubectl exec -it tdengine-1 -n tdengine-test -- taos -s "show mnodes\G"
|
||||
Welcome to the TDengine Command Line Interface, Client Version:3.0.7.1.202307190706
|
||||
Copyright (c) 2022 by TDengine, all rights reserved.
|
||||
|
||||
taos> show mnodes\G
|
||||
*************************** 1.row ***************************
|
||||
id: 1
|
||||
endpoint: tdengine-0.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: offline
|
||||
status: offline
|
||||
create_time: 2023-07-19 17:54:18.559
|
||||
reboot_time: 1970-01-01 08:00:00.000
|
||||
*************************** 2.row ***************************
|
||||
id: 2
|
||||
endpoint: tdengine-1.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: leader
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:05.600
|
||||
reboot_time: 2023-07-20 09:32:00.227
|
||||
*************************** 3.row ***************************
|
||||
id: 3
|
||||
endpoint: tdengine-2.taosd.tdengine-test.svc.cluster.local:6030
|
||||
role: follower
|
||||
status: ready
|
||||
create_time: 2023-07-20 09:22:20.042
|
||||
reboot_time: 2023-07-20 09:32:00.026
|
||||
Query OK, 3 row(s) in set (0.001513s)
|
||||
```
|
||||
|
||||
集群可以正常读写
|
||||
|
||||
```Bash
|
||||
# insert
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "insert into test1.t1 values(now, 1)(now+1s, 2);"
|
||||
|
||||
taos> insert into test1.t1 values(now, 1)(now+1s, 2);
|
||||
Insert OK, 2 row(s) affected (0.002098s)
|
||||
|
||||
# select
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "select *from test1.t1"
|
||||
|
||||
taos> select *from test1.t1
|
||||
ts | n |
|
||||
========================================
|
||||
2023-07-19 18:04:58.104 | 1 |
|
||||
2023-07-19 18:04:59.104 | 2 |
|
||||
2023-07-19 18:06:00.303 | 1 |
|
||||
2023-07-19 18:06:01.303 | 2 |
|
||||
Query OK, 4 row(s) in set (0.001994s)
|
||||
```
|
||||
|
||||
同理，至于非leader得mnode掉线，读写当然可以正常进行，这里就不做过多的展示。
|
||||
|
||||
## 集群扩容
|
||||
|
||||
TDengine 集群支持自动扩容：
|
||||
|
||||
```bash
|
||||
```Bash
|
||||
kubectl scale statefulsets tdengine --replicas=4
|
||||
```
|
||||
|
||||
上面命令行中参数 `--replica=4` 表示要将 TDengine 集群扩容到 4 个节点，执行后首先检查 POD 的状态：
|
||||
|
||||
```bash
|
||||
kubectl get pods -l app=tdengine
|
||||
```Bash
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
```
|
||||
|
||||
输出如下：
|
||||
|
||||
```
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 161m
|
||||
tdengine-1 1/1 Running 0 161m
|
||||
tdengine-2 1/1 Running 0 32m
|
||||
tdengine-3 1/1 Running 0 32m
|
||||
```Plain
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 1/1 Running 4 (6h26m ago) 6h53m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (6h39m ago) 6h53m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 5h16m 10.244.1.224 node85 <none> <none>
|
||||
tdengine-3 1/1 Running 0 3m24s 10.244.2.76 node86 <none> <none>
|
||||
```
|
||||
|
||||
此时 POD 的状态仍然是 Running，TDengine 集群中的 dnode 状态要等 POD 状态为 `ready` 之后才能看到：
|
||||
此时 Pod 的状态仍然是 Running，TDengine 集群中的 dnode 状态要等 Pod 状态为 `ready` 之后才能看到：
|
||||
|
||||
```bash
|
||||
kubectl exec -i -t tdengine-3 -- taos -s "show dnodes"
|
||||
```Bash
|
||||
kubectl exec -it tdengine-3 -n tdengine-test -- taos -s "show dnodes"
|
||||
```
|
||||
|
||||
扩容后的四节点 TDengine 集群的 dnode 列表:
|
||||
|
||||
```
|
||||
```Plain
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||||
============================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||||
4 | tdengine-3.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:33:16.039 | |
|
||||
Query OK, 4 rows in database (0.008377s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
4 | tdengine-3.ta... | 0 | 16 | ready | 2023-07-20 16:01:44.007 | 2023-07-20 16:01:44.889 | | | |
|
||||
Query OK, 4 row(s) in set (0.003628s)
|
||||
```
|
||||
|
||||
## 集群缩容
|
||||
|
||||
由于 TDengine 集群在扩缩容时会对数据进行节点间迁移，使用 kubectl 命令进行缩容需要首先使用 "drop dnodes" 命令，节点删除完成后再进行 Kubernetes 集群缩容。
|
||||
由于 TDengine 集群在扩缩容时会对数据进行节点间迁移，使用 kubectl 命令进行缩容需要首先使用 "drop dnodes" 命令（**如果集群中存在3副本的db，那么缩容后的** **dnode** **个数也要必须大于等于3，否则drop dnode操作会被中止**），然后再节点删除完成后再进行 Kubernetes 集群缩容。
|
||||
|
||||
注意：由于 Kubernetes Statefulset 中 Pod 的只能按创建顺序逆序移除，所以 TDengine drop dnode 也需要按照创建顺序逆序移除，否则会导致 Pod 处于错误状态。
|
||||
|
||||
```
|
||||
$ kubectl exec -i -t tdengine-0 -- taos -s "drop dnode 4"
|
||||
```
|
||||
|
||||
```bash
|
||||
$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||||
```Bash
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "drop dnode 4"
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||||
============================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||||
Query OK, 3 rows in database (0.004861s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
Query OK, 3 row(s) in set (0.003324s)
|
||||
```
|
||||
|
||||
确认移除成功后（使用 kubectl exec -i -t tdengine-0 -- taos -s "show dnodes" 查看和确认 dnode 列表），使用 kubectl 命令移除 POD：
|
||||
|
||||
```
|
||||
kubectl scale statefulsets tdengine --replicas=3
|
||||
```Plain
|
||||
kubectl scale statefulsets tdengine --replicas=3 -n tdengine-test
|
||||
```
|
||||
|
||||
最后一个 POD 将会被删除。使用命令 kubectl get pods -l app=tdengine 查看POD状态：
|
||||
|
||||
```
|
||||
$ kubectl get pods -l app=tdengine
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 4m7s
|
||||
tdengine-1 1/1 Running 0 3m55s
|
||||
tdengine-2 1/1 Running 0 2m28s
|
||||
```Plain
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 1/1 Running 4 (6h55m ago) 7h22m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (7h9m ago) 7h23m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 5h45m 10.244.1.224 node85 <none> <none>
|
||||
```
|
||||
|
||||
POD删除后，需要手动删除PVC，否则下次扩容时会继续使用以前的数据导致无法正常加入集群。
|
||||
|
||||
```bash
|
||||
$ kubectl delete pvc taosdata-tdengine-3
|
||||
```Bash
|
||||
kubectl delete pvc aosdata-tdengine-3 -n tdengine-test
|
||||
```
|
||||
|
||||
此时的集群状态是安全的，需要时还可以再次进行扩容：
|
||||
|
||||
```bash
|
||||
$ kubectl scale statefulsets tdengine --replicas=4
|
||||
```Bash
|
||||
kubectl scale statefulsets tdengine --replicas=4 -n tdengine-test
|
||||
statefulset.apps/tdengine scaled
|
||||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl get pods -l app=tdengine
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 35m
|
||||
tdengine-1 1/1 Running 0 34m
|
||||
tdengine-2 1/1 Running 0 12m
|
||||
tdengine-3 0/1 ContainerCreating 0 4s
|
||||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl get pods -l app=tdengine
|
||||
NAME READY STATUS RESTARTS AGE
|
||||
tdengine-0 1/1 Running 0 35m
|
||||
tdengine-1 1/1 Running 0 34m
|
||||
tdengine-2 1/1 Running 0 12m
|
||||
tdengine-3 0/1 Running 0 7s
|
||||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||||
|
||||
kubectl get pod -l app=tdengine -n tdengine-test -o wide
|
||||
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
|
||||
tdengine-0 1/1 Running 4 (6h59m ago) 7h27m 10.244.2.75 node86 <none> <none>
|
||||
tdengine-1 1/1 Running 1 (7h13m ago) 7h27m 10.244.0.59 node84 <none> <none>
|
||||
tdengine-2 1/1 Running 0 5h49m 10.244.1.224 node85 <none> <none>
|
||||
tdengine-3 1/1 Running 0 20s 10.244.2.77 node86 <none> <none>
|
||||
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 17:38:49.012 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 4 | ready | 2022-07-25 17:39:01.517 | |
|
||||
5 | tdengine-2.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 18:01:36.479 | |
|
||||
6 | tdengine-3.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 18:13:54.411 | |
|
||||
Query OK, 4 row(s) in set (0.001348s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
5 | tdengine-3.ta... | 0 | 16 | ready | 2023-07-20 16:31:34.092 | 2023-07-20 16:38:17.419 | | | |
|
||||
Query OK, 4 row(s) in set (0.003881s)
|
||||
```
|
||||
|
||||
## 清理 TDengine 集群
|
||||
|
||||
> **删除pvc时需要注意下pv persistentVolumeReclaimPolicy策略，建议改为Delete，这样在删除pvc时才会自动清理pv，同时会清理底层的csi存储资源，如果没有配置删除pvc自动清理pv的策略，再删除pvc后，在手动清理pv时，pv对应的csi存储资源可能不会被释放。**
|
||||
|
||||
完整移除 TDengine 集群，需要分别清理 statefulset、svc、configmap、pvc。
|
||||
|
||||
```bash
|
||||
kubectl delete statefulset -l app=tdengine
|
||||
kubectl delete svc -l app=tdengine
|
||||
kubectl delete pvc -l app=tdengine
|
||||
kubectl delete configmap taoscfg
|
||||
|
||||
```Bash
|
||||
kubectl delete statefulset -l app=tdengine -n tdengine-test
|
||||
kubectl delete svc -l app=tdengine -n tdengine-test
|
||||
kubectl delete pvc -l app=tdengine -n tdengine-test
|
||||
kubectl delete configmap taoscfg -n tdengine-test
|
||||
```
|
||||
|
||||
## 常见错误
|
||||
|
|
@ -330,65 +519,26 @@ kubectl delete configmap taoscfg
|
|||
|
||||
未进行 "drop dnode" 直接进行缩容，由于 TDengine 尚未删除节点，缩容 pod 导致 TDengine 集群中部分节点处于 offline 状态。
|
||||
|
||||
```
|
||||
$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||||
```Plain
|
||||
kubectl exec -it tdengine-0 -n tdengine-test -- taos -s "show dnodes"
|
||||
|
||||
taos> show dnodes
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 17:38:49.012 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 4 | ready | 2022-07-25 17:39:01.517 | |
|
||||
5 | tdengine-2.taosd.default.sv... | 0 | 4 | offline | 2022-07-25 18:01:36.479 | status msg timeout |
|
||||
6 | tdengine-3.taosd.default.sv... | 0 | 4 | offline | 2022-07-25 18:13:54.411 | status msg timeout |
|
||||
Query OK, 4 row(s) in set (0.001323s)
|
||||
id | endpoint | vnodes | support_vnodes | status | create_time | reboot_time | note | active_code | c_active_code |
|
||||
=============================================================================================================================================================================================================================================
|
||||
1 | tdengine-0.ta... | 10 | 16 | ready | 2023-07-19 17:54:18.552 | 2023-07-20 09:39:04.297 | | | |
|
||||
2 | tdengine-1.ta... | 10 | 16 | ready | 2023-07-19 17:54:37.828 | 2023-07-20 09:28:24.240 | | | |
|
||||
3 | tdengine-2.ta... | 10 | 16 | ready | 2023-07-19 17:55:01.141 | 2023-07-20 10:48:43.445 | | | |
|
||||
5 | tdengine-3.ta... | 0 | 16 | offline | 2023-07-20 16:31:34.092 | 2023-07-20 16:38:17.419 | status msg timeout | | |
|
||||
Query OK, 4 row(s) in set (0.003862s)
|
||||
```
|
||||
|
||||
### 错误二
|
||||
## 最后
|
||||
|
||||
TDengine 集群会持有 replica 参数，如果缩容后的节点数小于这个值，集群将无法使用：
|
||||
对于在 Kubernetes 环境下 TDengine 的高可用和高可靠来说，对于硬件损坏、灾难恢复，分为两个层面来讲：
|
||||
|
||||
创建一个库使用 replica 参数为 2，插入部分数据：
|
||||
1. 底层的分布式块存储具备的灾难恢复能力，块存储的多副本，当下流行的分布式块存储如 Ceph，就具备多副本能力，将存储副本扩展到不同的机架、机柜、机房、数据中心（或者直接使用公有云厂商提供的块存储服务）
|
||||
2. TDengine的灾难恢复，在 TDengine Enterprise 中，本身具备了当一个 dnode 永久下线（物理机磁盘损坏，数据分拣丢失）后，重新拉起一个空白的dnode来恢复原dnode的工作。
|
||||
|
||||
```bash
|
||||
kubectl exec -i -t tdengine-0 -- \
|
||||
taos -s \
|
||||
"create database if not exists test replica 2;
|
||||
use test;
|
||||
create table if not exists t1(ts timestamp, n int);
|
||||
insert into t1 values(now, 1)(now+1s, 2);"
|
||||
最后，欢迎使用[TDengine Cloud](https://cloud.taosdata.com/)，来体验一站式全托管的TDengine云服务。
|
||||
|
||||
|
||||
```
|
||||
|
||||
缩容到单节点：
|
||||
|
||||
```bash
|
||||
kubectl scale statefulsets tdengine --replicas=1
|
||||
|
||||
```
|
||||
|
||||
在 TDengine CLI 中的所有数据库操作将无法成功。
|
||||
|
||||
```
|
||||
taos> show dnodes;
|
||||
id | end_point | vnodes | cores | status | role | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 2 | 40 | ready | any | 2021-06-01 15:55:52.562 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 40 | offline | any | 2021-06-01 15:56:07.212 | status msg timeout |
|
||||
Query OK, 2 row(s) in set (0.000845s)
|
||||
|
||||
taos> show dnodes;
|
||||
id | end_point | vnodes | cores | status | role | create_time | offline reason |
|
||||
======================================================================================================================================
|
||||
1 | tdengine-0.taosd.default.sv... | 2 | 40 | ready | any | 2021-06-01 15:55:52.562 | |
|
||||
2 | tdengine-1.taosd.default.sv... | 1 | 40 | offline | any | 2021-06-01 15:56:07.212 | status msg timeout |
|
||||
Query OK, 2 row(s) in set (0.000837s)
|
||||
|
||||
taos> use test;
|
||||
Database changed.
|
||||
|
||||
taos> insert into t1 values(now, 3);
|
||||
|
||||
DB error: Unable to resolve FQDN (0.013874s)
|
||||
|
||||
```
|
||||
> TDengine Cloud 是一个极简的全托管时序数据处理云服务平台，它是基于开源的时序数据库 TDengine 而开发的。除高性能的时序数据库之外，它还具有缓存、订阅和流计算等系统功能，而且提供了便利而又安全的数据分享、以及众多的企业级功能。它可以让物联网、工业互联网、金融、IT 运维监控等领域企业在时序数据的管理上大幅降低人力成本和运营成本。
|
||||
|
|
|
|||
|
|
@ -85,7 +85,7 @@ create database if not exists db vgroups 10 buffer 10
|
|||
|
||||
```
|
||||
|
||||
以上示例创建了一个有 10 个 vgroup 名为 db 的数据库， 其中每个 vnode 分配也 10MB 的写入缓存
|
||||
以上示例创建了一个有 10 个 vgroup 名为 db 的数据库， 其中每个 vnode 分配 10MB 的写入缓存
|
||||
|
||||
### 使用数据库
|
||||
|
||||
|
|
|
|||
|
|
@ -44,11 +44,10 @@ table_option: {
|
|||
1. 表的第一个字段必须是 TIMESTAMP，并且系统自动将其设为主键；
|
||||
2. 表名最大长度为 192；
|
||||
3. 表的每行长度不能超过 48KB（从 3.0.5.0 版本开始为 64KB）;（注意：每个 BINARY/NCHAR 类型的列还会额外占用 2 个字节的存储位置）
|
||||
4. 子表名只能由字母、数字和下划线组成，且不能以数字开头，不区分大小写
|
||||
4. 表名，超级表名，以及子表名只能由字母、数字和下划线组成，且不能以数字开头，不区分大小写
|
||||
5. 使用数据类型 binary 或 nchar，需指定其最长的字节数，如 binary(20)，表示 20 字节；
|
||||
6. 为了兼容支持更多形式的表名，TDengine 引入新的转义符 "\`"，可以让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查。但是同样具有长度限制要求。使用转义字符以后，不再对转义字符中的内容进行大小写统一。
|
||||
6. 为了兼容支持更多形式的表名，TDengine 引入新的转义符 "\`"。 如果不加转义符，表名会被默认转换成小组；加上转义符可以保留表名中的大小写属性。
|
||||
例如：\`aBc\` 和 \`abc\` 是不同的表名，但是 abc 和 aBc 是相同的表名。
|
||||
需要注意的是转义字符中的内容必须是可打印字符。
|
||||
|
||||
**参数说明**
|
||||
|
||||
|
|
|
|||
|
|
@ -10,11 +10,9 @@ description: 合法字符集和命名中的限制规则
|
|||
2. 允许英文字符或下划线开头，不允许以数字开头
|
||||
3. 不区分大小写
|
||||
4. 转义后表（列）名规则：
|
||||
为了兼容支持更多形式的表（列）名，TDengine 引入新的转义符 "`"。可用让表名与关键词不冲突，同时不受限于上述表名称合法性约束检查
|
||||
转义后的表（列）名同样受到长度限制要求，且长度计算的时候不计算转义符。使用转义字符以后，不再对转义字符中的内容进行大小写统一
|
||||
为了兼容支持更多形式的表（列）名，TDengine 引入新的转义符 "`"。使用转义字符以后，不再对转义字符中的内容进行大小写统一，即可以保留用户指定表名中的大小写属性。
|
||||
|
||||
例如：\`aBc\` 和 \`abc\` 是不同的表（列）名，但是 abc 和 aBc 是相同的表（列）名。
|
||||
需要注意的是转义字符中的内容必须是可打印字符。
|
||||
|
||||
## 密码合法字符集
|
||||
|
||||
|
|
@ -48,13 +46,13 @@ description: 合法字符集和命名中的限制规则
|
|||
|
||||
### 转义后表（列）名规则：
|
||||
|
||||
为了兼容支持更多形式的表（列）名，TDengine 引入新的转义符 "`"，可以避免表名与关键词的冲突，同时不受限于上述表名合法性约束检查，转义符不计入表名的长度。
|
||||
为了兼容支持更多形式的表（列）名，TDengine 引入新的转义符 "`"，可以避免表名与关键词的冲突，转义符不计入表名的长度。
|
||||
转义后的表（列）名同样受到长度限制要求，且长度计算的时候不计算转义符。使用转义字符以后，不再对转义字符中的内容进行大小写统一。
|
||||
|
||||
例如：
|
||||
\`aBc\` 和 \`abc\` 是不同的表（列）名，但是 abc 和 aBc 是相同的表（列）名。
|
||||
|
||||
:::note
|
||||
转义字符中的内容必须是可打印字符。
|
||||
转义字符中的内容必须符合命名规则中的字符约束。
|
||||
|
||||
:::
|
||||
|
|
|
|||
|
|
@ -362,6 +362,8 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
|
|||
|
||||
- **max** : 数据类型的 列/标签 的最大值。生成的值将小于最小值。
|
||||
|
||||
- **fun** : 此列数据以函数填充，目前只支持 sin 和 cos 两函数，输入参数为时间戳换算成角度值，换算公式： 角度 x = 输入的时间列ts值 % 360。同时支持系数调节，随机波动因子调节，以固定格式的表达式展现，如 fun=“10\*sin(x)+100\*random(5)” , x 表示角度，取值 0 ~ 360度，增长步长与时间列步长一致。10 表示乘的系数，100 表示加或减的系数，5 表示波动幅度在 5% 的随机范围内。目前支持的数据类型为 int, bigint, float, double 四种数据类型。注意：表达式为固定模式，不可前后颠倒。
|
||||
|
||||
- **values** : nchar/binary 列/标签的值域，将从值中随机选择。
|
||||
|
||||
- **sma**: 将该列加入 SMA 中，值为 "yes" 或者 "no"，默认为 "no"。
|
||||
|
|
|
|||
|
|
@ -210,19 +210,6 @@ TDinsight dashboard 数据来源于 log 库（存放监控数据的默认db，
|
|||
|dnode\_ep|NCHAR|TAG|dnode endpoint|
|
||||
|cluster\_id|NCHAR|TAG|cluster id|
|
||||
|
||||
### logs 表
|
||||
|
||||
`logs` 表记录登录信息。
|
||||
|
||||
|field|type|is\_tag|comment|
|
||||
|:----|:---|:-----|:------|
|
||||
|ts|TIMESTAMP||timestamp|
|
||||
|level|VARCHAR||log level|
|
||||
|content|NCHAR||log content，长度不超过1024字节|
|
||||
|dnode\_id|INT|TAG|dnode id|
|
||||
|dnode\_ep|NCHAR|TAG|dnode endpoint|
|
||||
|cluster\_id|NCHAR|TAG|cluster id|
|
||||
|
||||
### log\_summary 表
|
||||
|
||||
`log_summary` 记录日志统计信息。
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@
|
|||
<dependency>
|
||||
<groupId>com.taosdata.jdbc</groupId>
|
||||
<artifactId>taos-jdbcdriver</artifactId>
|
||||
<version>3.0.0</version>
|
||||
<version>3.2.4</version>
|
||||
</dependency>
|
||||
|
||||
<dependency>
|
||||
|
|
|
|||
|
|
@ -288,7 +288,7 @@ DLL_EXPORT int32_t tmq_consumer_close(tmq_t *tmq);
|
|||
DLL_EXPORT int32_t tmq_commit_sync(tmq_t *tmq, const TAOS_RES *msg);
|
||||
DLL_EXPORT void tmq_commit_async(tmq_t *tmq, const TAOS_RES *msg, tmq_commit_cb *cb, void *param);
|
||||
DLL_EXPORT int32_t tmq_commit_offset_sync(tmq_t *tmq, const char *pTopicName, int32_t vgId, int64_t offset);
|
||||
DLL_EXPORT int32_t tmq_commit_offset_async(tmq_t *tmq, const char *pTopicName, int32_t vgId, int64_t offset, tmq_commit_cb *cb, void *param);
|
||||
DLL_EXPORT void tmq_commit_offset_async(tmq_t *tmq, const char *pTopicName, int32_t vgId, int64_t offset, tmq_commit_cb *cb, void *param);
|
||||
DLL_EXPORT int32_t tmq_get_topic_assignment(tmq_t *tmq, const char *pTopicName, tmq_topic_assignment **assignment,
|
||||
int32_t *numOfAssignment);
|
||||
DLL_EXPORT void tmq_free_assignment(tmq_topic_assignment* pAssignment);
|
||||
|
|
|
|||
|
|
@ -312,7 +312,7 @@ enum {
|
|||
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_CONSUME, "vnode-tmq-consume", SMqPollReq, SMqDataBlkRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_CONSUME_PUSH, "vnode-tmq-consume-push", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_VG_WALINFO, "vnode-tmq-vg-walinfo", SMqPollReq, SMqDataBlkRsp)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_VG_COMMITTEDINFO, "vnode-tmq-committed-walinfo", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_VG_COMMITTEDINFO, "vnode-tmq-committedinfo", NULL, NULL)
|
||||
TD_DEF_MSG_TYPE(TDMT_VND_TMQ_MAX_MSG, "vnd-tmq-max", NULL, NULL)
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -1297,13 +1297,19 @@ int initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* pEpSe
|
|||
return -1;
|
||||
}
|
||||
|
||||
int32_t code = taosGetFqdnPortFromEp(firstEp, &mgmtEpSet->eps[0]);
|
||||
int32_t code = taosGetFqdnPortFromEp(firstEp, &mgmtEpSet->eps[mgmtEpSet->numOfEps]);
|
||||
if (code != TSDB_CODE_SUCCESS) {
|
||||
terrno = TSDB_CODE_TSC_INVALID_FQDN;
|
||||
return terrno;
|
||||
}
|
||||
|
||||
mgmtEpSet->numOfEps++;
|
||||
uint32_t addr = taosGetIpv4FromFqdn(mgmtEpSet->eps[mgmtEpSet->numOfEps].fqdn);
|
||||
if (addr == 0xffffffff) {
|
||||
tscError("failed to resolve firstEp fqdn: %s, code:%s", mgmtEpSet->eps[mgmtEpSet->numOfEps].fqdn,
|
||||
tstrerror(TSDB_CODE_TSC_INVALID_FQDN));
|
||||
memset(&(mgmtEpSet->eps[mgmtEpSet->numOfEps]), 0, sizeof(mgmtEpSet->eps[mgmtEpSet->numOfEps]));
|
||||
} else {
|
||||
mgmtEpSet->numOfEps++;
|
||||
}
|
||||
}
|
||||
|
||||
if (secondEp && secondEp[0] != 0) {
|
||||
|
|
@ -1313,12 +1319,19 @@ int initEpSetFromCfg(const char* firstEp, const char* secondEp, SCorEpSet* pEpSe
|
|||
}
|
||||
|
||||
taosGetFqdnPortFromEp(secondEp, &mgmtEpSet->eps[mgmtEpSet->numOfEps]);
|
||||
mgmtEpSet->numOfEps++;
|
||||
uint32_t addr = taosGetIpv4FromFqdn(mgmtEpSet->eps[mgmtEpSet->numOfEps].fqdn);
|
||||
if (addr == 0xffffffff) {
|
||||
tscError("failed to resolve secondEp fqdn: %s, code:%s", mgmtEpSet->eps[mgmtEpSet->numOfEps].fqdn,
|
||||
tstrerror(TSDB_CODE_TSC_INVALID_FQDN));
|
||||
memset(&(mgmtEpSet->eps[mgmtEpSet->numOfEps]), 0, sizeof(mgmtEpSet->eps[mgmtEpSet->numOfEps]));
|
||||
} else {
|
||||
mgmtEpSet->numOfEps++;
|
||||
}
|
||||
}
|
||||
|
||||
if (mgmtEpSet->numOfEps == 0) {
|
||||
terrno = TSDB_CODE_TSC_INVALID_FQDN;
|
||||
return -1;
|
||||
terrno = TSDB_CODE_RPC_NETWORK_UNAVAIL;
|
||||
return TSDB_CODE_RPC_NETWORK_UNAVAIL;
|
||||
}
|
||||
|
||||
return 0;
|
||||
|
|
|
|||
|
|
@ -99,13 +99,20 @@ int32_t processConnectRsp(void* param, SDataBuf* pMsg, int32_t code) {
|
|||
goto End;
|
||||
}
|
||||
|
||||
int updateEpSet = 1;
|
||||
if (connectRsp.dnodeNum == 1) {
|
||||
SEpSet srcEpSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp);
|
||||
SEpSet dstEpSet = connectRsp.epSet;
|
||||
rpcSetDefaultAddr(pTscObj->pAppInfo->pTransporter, srcEpSet.eps[srcEpSet.inUse].fqdn,
|
||||
dstEpSet.eps[dstEpSet.inUse].fqdn);
|
||||
} else if (connectRsp.dnodeNum > 1 && !isEpsetEqual(&pTscObj->pAppInfo->mgmtEp.epSet, &connectRsp.epSet)) {
|
||||
SEpSet* pOrig = &pTscObj->pAppInfo->mgmtEp.epSet;
|
||||
if (srcEpSet.numOfEps == 1) {
|
||||
rpcSetDefaultAddr(pTscObj->pAppInfo->pTransporter, srcEpSet.eps[srcEpSet.inUse].fqdn,
|
||||
dstEpSet.eps[dstEpSet.inUse].fqdn);
|
||||
updateEpSet = 0;
|
||||
}
|
||||
}
|
||||
if (updateEpSet == 1 && !isEpsetEqual(&pTscObj->pAppInfo->mgmtEp.epSet, &connectRsp.epSet)) {
|
||||
SEpSet corEpSet = getEpSet_s(&pTscObj->pAppInfo->mgmtEp);
|
||||
|
||||
SEpSet* pOrig = &corEpSet;
|
||||
SEp* pOrigEp = &pOrig->eps[pOrig->inUse];
|
||||
SEp* pNewEp = &connectRsp.epSet.eps[connectRsp.epSet.inUse];
|
||||
tscDebug("mnode epset updated from %d/%d=>%s:%d to %d/%d=>%s:%d in connRsp", pOrig->inUse, pOrig->numOfEps,
|
||||
|
|
|
|||
|
|
@ -523,9 +523,7 @@ static int32_t doSendCommitMsg(tmq_t* tmq, int32_t vgId, SEpSet* epSet, STqOffse
|
|||
|
||||
|
||||
int64_t transporterId = 0;
|
||||
asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, epSet, &transporterId, pMsgSendInfo);
|
||||
|
||||
return TSDB_CODE_SUCCESS;
|
||||
return asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, epSet, &transporterId, pMsgSendInfo);
|
||||
}
|
||||
|
||||
static SMqClientTopic* getTopicByName(tmq_t* tmq, const char* pTopicName) {
|
||||
|
|
@ -546,7 +544,6 @@ static SMqClientTopic* getTopicByName(tmq_t* tmq, const char* pTopicName) {
|
|||
static SMqCommitCbParamSet* prepareCommitCbParamSet(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* userParam, int32_t rspNum){
|
||||
SMqCommitCbParamSet* pParamSet = taosMemoryCalloc(1, sizeof(SMqCommitCbParamSet));
|
||||
if (pParamSet == NULL) {
|
||||
pCommitFp(tmq, TSDB_CODE_OUT_OF_MEMORY, userParam);
|
||||
return NULL;
|
||||
}
|
||||
|
||||
|
|
@ -715,7 +712,9 @@ static void asyncCommitAllOffsets(tmq_t* tmq, tmq_commit_cb* pCommitFp, void* us
|
|||
|
||||
end:
|
||||
taosMemoryFree(pParamSet);
|
||||
pCommitFp(tmq, code, userParam);
|
||||
if(pCommitFp != NULL) {
|
||||
pCommitFp(tmq, code, userParam);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
|
|
@ -1860,8 +1859,8 @@ static int32_t tmqHandleNoPollRsp(tmq_t* tmq, SMqRspWrapper* rspWrapper, bool* p
|
|||
static void updateVgInfo(SMqClientVg* pVg, STqOffsetVal* reqOffset, STqOffsetVal* rspOffset, int64_t sver, int64_t ever, int64_t consumerId){
|
||||
if (!pVg->seekUpdated) {
|
||||
tscDebug("consumer:0x%" PRIx64" local offset is update, since seekupdate not set", consumerId);
|
||||
pVg->offsetInfo.beginOffset = *reqOffset;
|
||||
pVg->offsetInfo.endOffset = *rspOffset;
|
||||
if(reqOffset->type != 0) pVg->offsetInfo.beginOffset = *reqOffset;
|
||||
if(rspOffset->type != 0) pVg->offsetInfo.endOffset = *rspOffset;
|
||||
} else {
|
||||
tscDebug("consumer:0x%" PRIx64" local offset is NOT update, since seekupdate is set", consumerId);
|
||||
}
|
||||
|
|
@ -2307,6 +2306,9 @@ const char* tmq_get_table_name(TAOS_RES* res) {
|
|||
void tmq_commit_async(tmq_t* tmq, const TAOS_RES* pRes, tmq_commit_cb* cb, void* param) {
|
||||
if (tmq == NULL) {
|
||||
tscError("invalid tmq handle, null");
|
||||
if(cb != NULL) {
|
||||
cb(tmq, TSDB_CODE_INVALID_PARA, param);
|
||||
}
|
||||
return;
|
||||
}
|
||||
if (pRes == NULL) { // here needs to commit all offsets.
|
||||
|
|
@ -2410,15 +2412,17 @@ int32_t tmq_commit_offset_sync(tmq_t *tmq, const char *pTopicName, int32_t vgId,
|
|||
tsem_destroy(&pInfo->sem);
|
||||
taosMemoryFree(pInfo);
|
||||
|
||||
tscInfo("consumer:0x%" PRIx64 " send seek to vgId:%d, offset:%" PRId64" code:%s", tmq->consumerId, vgId, offset, tstrerror(code));
|
||||
tscInfo("consumer:0x%" PRIx64 " sync send seek to vgId:%d, offset:%" PRId64" code:%s", tmq->consumerId, vgId, offset, tstrerror(code));
|
||||
|
||||
return code;
|
||||
}
|
||||
|
||||
int32_t tmq_commit_offset_async(tmq_t *tmq, const char *pTopicName, int32_t vgId, int64_t offset, tmq_commit_cb *cb, void *param){
|
||||
void tmq_commit_offset_async(tmq_t *tmq, const char *pTopicName, int32_t vgId, int64_t offset, tmq_commit_cb *cb, void *param){
|
||||
int32_t code = 0;
|
||||
if (tmq == NULL || pTopicName == NULL) {
|
||||
tscError("invalid tmq handle, null");
|
||||
return TSDB_CODE_INVALID_PARA;
|
||||
code = TSDB_CODE_INVALID_PARA;
|
||||
goto end;
|
||||
}
|
||||
|
||||
int32_t accId = tmq->pTscObj->acctId;
|
||||
|
|
@ -2427,17 +2431,17 @@ int32_t tmq_commit_offset_async(tmq_t *tmq, const char *pTopicName, int32_t vgId
|
|||
|
||||
taosWLockLatch(&tmq->lock);
|
||||
SMqClientVg* pVg = NULL;
|
||||
int32_t code = getClientVg(tmq, tname, vgId, &pVg);
|
||||
code = getClientVg(tmq, tname, vgId, &pVg);
|
||||
if(code != 0){
|
||||
taosWUnLockLatch(&tmq->lock);
|
||||
return code;
|
||||
goto end;
|
||||
}
|
||||
|
||||
SVgOffsetInfo* pOffsetInfo = &pVg->offsetInfo;
|
||||
code = checkWalRange(pOffsetInfo, offset);
|
||||
if (code != 0) {
|
||||
taosWUnLockLatch(&tmq->lock);
|
||||
return code;
|
||||
goto end;
|
||||
}
|
||||
taosWUnLockLatch(&tmq->lock);
|
||||
|
||||
|
|
@ -2445,9 +2449,12 @@ int32_t tmq_commit_offset_async(tmq_t *tmq, const char *pTopicName, int32_t vgId
|
|||
|
||||
code = asyncCommitOffset(tmq, tname, vgId, &offsetVal, cb, param);
|
||||
|
||||
tscInfo("consumer:0x%" PRIx64 " send seek to vgId:%d, offset:%" PRId64" code:%s", tmq->consumerId, vgId, offset, tstrerror(code));
|
||||
tscInfo("consumer:0x%" PRIx64 " async send seek to vgId:%d, offset:%" PRId64" code:%s", tmq->consumerId, vgId, offset, tstrerror(code));
|
||||
|
||||
return code;
|
||||
end:
|
||||
if(code != 0 && cb != NULL){
|
||||
cb(tmq, code, param);
|
||||
}
|
||||
}
|
||||
|
||||
void updateEpCallbackFn(tmq_t* pTmq, int32_t code, SDataBuf* pDataBuf, void* param) {
|
||||
|
|
@ -2832,6 +2839,7 @@ int64_t tmq_position(tmq_t *tmq, const char *pTopicName, int32_t vgId){
|
|||
tscError("consumer:0x%" PRIx64 " offset type:%d can not be reach here", tmq->consumerId, type);
|
||||
}
|
||||
|
||||
tscInfo("consumer:0x%" PRIx64 " tmq_position vgId:%d position:%" PRId64, tmq->consumerId, vgId, position);
|
||||
return position;
|
||||
}
|
||||
|
||||
|
|
@ -2871,12 +2879,16 @@ int64_t tmq_committed(tmq_t *tmq, const char *pTopicName, int32_t vgId){
|
|||
if(pOffsetInfo->committedOffset.type == TMQ_OFFSET__LOG){
|
||||
committed = pOffsetInfo->committedOffset.version;
|
||||
taosWUnLockLatch(&tmq->lock);
|
||||
return committed;
|
||||
goto end;
|
||||
}
|
||||
SEpSet epSet = pVg->epSet;
|
||||
taosWUnLockLatch(&tmq->lock);
|
||||
|
||||
return getCommittedFromServer(tmq, tname, vgId, &epSet);
|
||||
committed = getCommittedFromServer(tmq, tname, vgId, &epSet);
|
||||
|
||||
end:
|
||||
tscInfo("consumer:0x%" PRIx64 " tmq_committed vgId:%d committed:%" PRId64, tmq->consumerId, vgId, committed);
|
||||
return committed;
|
||||
}
|
||||
|
||||
int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_assignment** assignment,
|
||||
|
|
@ -2897,7 +2909,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a
|
|||
taosWLockLatch(&tmq->lock);
|
||||
SMqClientTopic* pTopic = getTopicByName(tmq, tname);
|
||||
if (pTopic == NULL) {
|
||||
code = TSDB_CODE_INVALID_PARA;
|
||||
code = TSDB_CODE_TMQ_INVALID_TOPIC;
|
||||
goto end;
|
||||
}
|
||||
|
||||
|
|
@ -3040,7 +3052,7 @@ int32_t tmq_get_topic_assignment(tmq_t* tmq, const char* pTopicName, tmq_topic_a
|
|||
}
|
||||
|
||||
SVgOffsetInfo* pOffsetInfo = &pClientVg->offsetInfo;
|
||||
tscInfo("vgId:%d offset is update to:%"PRId64, p->vgId, p->currentOffset);
|
||||
tscInfo("consumer:0x%" PRIx64 " %s vgId:%d offset is update to:%"PRId64, tmq->consumerId, pTopic->topicName, p->vgId, p->currentOffset);
|
||||
|
||||
pOffsetInfo->walVerBegin = p->begin;
|
||||
pOffsetInfo->walVerEnd = p->end;
|
||||
|
|
@ -3078,6 +3090,7 @@ static int32_t tmqSeekCb(void* param, SDataBuf* pMsg, int32_t code) {
|
|||
return 0;
|
||||
}
|
||||
|
||||
// seek interface have to send msg to server to cancel push handle if needed, because consumer may be in wait status if there is no data to poll
|
||||
int32_t tmq_offset_seek(tmq_t* tmq, const char* pTopicName, int32_t vgId, int64_t offset) {
|
||||
if (tmq == NULL || pTopicName == NULL) {
|
||||
tscError("invalid tmq handle, null");
|
||||
|
|
@ -3163,8 +3176,6 @@ int32_t tmq_offset_seek(tmq_t* tmq, const char* pTopicName, int32_t vgId, int64_
|
|||
sendInfo->msgType = TDMT_VND_TMQ_SEEK;
|
||||
|
||||
int64_t transporterId = 0;
|
||||
tscInfo("consumer:0x%" PRIx64 " %s send seek info vgId:%d, epoch %d" PRIx64,
|
||||
tmq->consumerId, tname, vgId, tmq->epoch);
|
||||
asyncSendMsgToServer(tmq->pTscObj->pAppInfo->pTransporter, &epSet, &transporterId, sendInfo);
|
||||
|
||||
tsem_wait(&pParam->sem);
|
||||
|
|
|
|||
|
|
@ -7207,11 +7207,6 @@ bool tOffsetEqual(const STqOffsetVal *pLeft, const STqOffsetVal *pRight) {
|
|||
return pLeft->uid == pRight->uid && pLeft->ts == pRight->ts;
|
||||
} else if (pLeft->type == TMQ_OFFSET__SNAPSHOT_META) {
|
||||
return pLeft->uid == pRight->uid;
|
||||
} else {
|
||||
ASSERT(0);
|
||||
/*ASSERT(pLeft->type == TMQ_OFFSET__RESET_NONE || pLeft->type == TMQ_OFFSET__RESET_EARLIEST ||*/
|
||||
/*pLeft->type == TMQ_OFFSET__RESET_LATEST);*/
|
||||
/*return true;*/
|
||||
}
|
||||
}
|
||||
return false;
|
||||
|
|
|
|||
|
|
@ -94,7 +94,7 @@ void mndDropConsumerFromSdb(SMnode *pMnode, int64_t consumerId){
|
|||
|
||||
bool mndRebTryStart() {
|
||||
int32_t old = atomic_val_compare_exchange_32(&mqRebInExecCnt, 0, 1);
|
||||
mDebug("tq timer, rebalance counter old val:%d", old);
|
||||
mInfo("tq timer, rebalance counter old val:%d", old);
|
||||
return old == 0;
|
||||
}
|
||||
|
||||
|
|
@ -116,7 +116,7 @@ void mndRebCntDec() {
|
|||
int32_t newVal = val - 1;
|
||||
int32_t oldVal = atomic_val_compare_exchange_32(&mqRebInExecCnt, val, newVal);
|
||||
if (oldVal == val) {
|
||||
mDebug("rebalance trans end, rebalance counter:%d", newVal);
|
||||
mInfo("rebalance trans end, rebalance counter:%d", newVal);
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
|
@ -281,7 +281,7 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) {
|
|||
|
||||
// rebalance cannot be parallel
|
||||
if (!mndRebTryStart()) {
|
||||
mDebug("mq rebalance already in progress, do nothing");
|
||||
mInfo("mq rebalance already in progress, do nothing");
|
||||
return 0;
|
||||
}
|
||||
|
||||
|
|
@ -312,7 +312,7 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) {
|
|||
int32_t hbStatus = atomic_add_fetch_32(&pConsumer->hbStatus, 1);
|
||||
int32_t status = atomic_load_32(&pConsumer->status);
|
||||
|
||||
mDebug("check for consumer:0x%" PRIx64 " status:%d(%s), sub-time:%" PRId64 ", createTime:%" PRId64 ", hbstatus:%d",
|
||||
mInfo("check for consumer:0x%" PRIx64 " status:%d(%s), sub-time:%" PRId64 ", createTime:%" PRId64 ", hbstatus:%d",
|
||||
pConsumer->consumerId, status, mndConsumerStatusName(status), pConsumer->subscribeTime, pConsumer->createTime,
|
||||
hbStatus);
|
||||
|
||||
|
|
@ -362,7 +362,7 @@ static int32_t mndProcessMqTimerMsg(SRpcMsg *pMsg) {
|
|||
}
|
||||
|
||||
if (taosHashGetSize(pRebMsg->rebSubHash) != 0) {
|
||||
mInfo("mq rebalance will be triggered");
|
||||
mInfo("mq rebalance will be triggered");
|
||||
SRpcMsg rpcMsg = {
|
||||
.msgType = TDMT_MND_TMQ_DO_REBALANCE,
|
||||
.pCont = pRebMsg,
|
||||
|
|
@ -416,7 +416,7 @@ static int32_t mndProcessMqHbReq(SRpcMsg *pMsg) {
|
|||
|
||||
for(int i = 0; i < taosArrayGetSize(req.topics); i++){
|
||||
TopicOffsetRows* data = taosArrayGet(req.topics, i);
|
||||
mDebug("heartbeat report offset rows.%s:%s", pConsumer->cgroup, data->topicName);
|
||||
mInfo("heartbeat report offset rows.%s:%s", pConsumer->cgroup, data->topicName);
|
||||
|
||||
SMqSubscribeObj *pSub = mndAcquireSubscribe(pMnode, pConsumer->cgroup, data->topicName);
|
||||
if(pSub == NULL){
|
||||
|
|
@ -515,7 +515,7 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
|
|||
char *topic = taosArrayGetP(pConsumer->currentTopics, i);
|
||||
SMqSubscribeObj *pSub = mndAcquireSubscribe(pMnode, pConsumer->cgroup, topic);
|
||||
// txn guarantees pSub is created
|
||||
|
||||
if(pSub == NULL) continue;
|
||||
taosRLockLatch(&pSub->lock);
|
||||
|
||||
SMqSubTopicEp topicEp = {0};
|
||||
|
|
@ -523,6 +523,11 @@ static int32_t mndProcessAskEpReq(SRpcMsg *pMsg) {
|
|||
|
||||
// 2.1 fetch topic schema
|
||||
SMqTopicObj *pTopic = mndAcquireTopic(pMnode, topic);
|
||||
if(pTopic == NULL) {
|
||||
taosRUnLockLatch(&pSub->lock);
|
||||
mndReleaseSubscribe(pMnode, pSub);
|
||||
continue;
|
||||
}
|
||||
taosRLockLatch(&pTopic->lock);
|
||||
tstrncpy(topicEp.db, pTopic->db, TSDB_DB_FNAME_LEN);
|
||||
topicEp.schema.nCols = pTopic->schema.nCols;
|
||||
|
|
@ -1104,13 +1109,13 @@ static int32_t mndRetrieveConsumer(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *
|
|||
}
|
||||
|
||||
if (taosArrayGetSize(pConsumer->assignedTopics) == 0) {
|
||||
mDebug("showing consumer:0x%" PRIx64 " no assigned topic, skip", pConsumer->consumerId);
|
||||
mInfo("showing consumer:0x%" PRIx64 " no assigned topic, skip", pConsumer->consumerId);
|
||||
sdbRelease(pSdb, pConsumer);
|
||||
continue;
|
||||
}
|
||||
|
||||
taosRLockLatch(&pConsumer->lock);
|
||||
mDebug("showing consumer:0x%" PRIx64, pConsumer->consumerId);
|
||||
mInfo("showing consumer:0x%" PRIx64, pConsumer->consumerId);
|
||||
|
||||
int32_t topicSz = taosArrayGetSize(pConsumer->assignedTopics);
|
||||
bool hasTopic = true;
|
||||
|
|
|
|||
|
|
@ -1207,7 +1207,7 @@ int32_t mndRetrieveSubscribe(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock
|
|||
int32_t numOfRows = 0;
|
||||
SMqSubscribeObj *pSub = NULL;
|
||||
|
||||
mDebug("mnd show subscriptions begin");
|
||||
mInfo("mnd show subscriptions begin");
|
||||
|
||||
while (numOfRows < rowsCapacity) {
|
||||
pShow->pIter = sdbFetch(pSdb, SDB_SUBSCRIBE, pShow->pIter, (void **)&pSub);
|
||||
|
|
@ -1247,7 +1247,7 @@ int32_t mndRetrieveSubscribe(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBlock
|
|||
sdbRelease(pSdb, pSub);
|
||||
}
|
||||
|
||||
mDebug("mnd end show subscriptions");
|
||||
mInfo("mnd end show subscriptions");
|
||||
|
||||
pShow->numOfRows += numOfRows;
|
||||
return numOfRows;
|
||||
|
|
|
|||
|
|
@ -703,7 +703,7 @@ int32_t tqProcessDeleteSubReq(STQ* pTq, int64_t sversion, char* msg, int32_t msg
|
|||
SMqVDeleteReq* pReq = (SMqVDeleteReq*)msg;
|
||||
int32_t vgId = TD_VID(pTq->pVnode);
|
||||
|
||||
tqDebug("vgId:%d, tq process delete sub req %s", vgId, pReq->subKey);
|
||||
tqInfo("vgId:%d, tq process delete sub req %s", vgId, pReq->subKey);
|
||||
int32_t code = 0;
|
||||
|
||||
taosWLockLatch(&pTq->lock);
|
||||
|
|
@ -784,7 +784,7 @@ int32_t tqProcessSubscribeReq(STQ* pTq, int64_t sversion, char* msg, int32_t msg
|
|||
return -1;
|
||||
}
|
||||
|
||||
tqDebug("vgId:%d, tq process sub req:%s, Id:0x%" PRIx64 " -> Id:0x%" PRIx64, pTq->pVnode->config.vgId, req.subKey,
|
||||
tqInfo("vgId:%d, tq process sub req:%s, Id:0x%" PRIx64 " -> Id:0x%" PRIx64, pTq->pVnode->config.vgId, req.subKey,
|
||||
req.oldConsumerId, req.newConsumerId);
|
||||
|
||||
STqHandle* pHandle = NULL;
|
||||
|
|
|
|||
|
|
@ -344,9 +344,11 @@ int32_t tqExtractDataForMq(STQ* pTq, STqHandle* pHandle, const SMqPollReq* pRequ
|
|||
if (blockReturned) {
|
||||
return 0;
|
||||
}
|
||||
} else { // use the consumer specified offset
|
||||
} else if(reqOffset.type != 0){ // use the consumer specified offset
|
||||
// the offset value can not be monotonious increase??
|
||||
offset = reqOffset;
|
||||
} else {
|
||||
return TSDB_CODE_TMQ_INVALID_MSG;
|
||||
}
|
||||
|
||||
// this is a normal subscribe requirement
|
||||
|
|
|
|||
|
|
@ -547,7 +547,7 @@ void monSendReport() {
|
|||
monGenGrantJson(pMonitor);
|
||||
monGenDnodeJson(pMonitor);
|
||||
monGenDiskJson(pMonitor);
|
||||
monGenLogJson(pMonitor);
|
||||
//monGenLogJson(pMonitor); // TS-3691
|
||||
|
||||
char *pCont = tjsonToString(pMonitor->pJson);
|
||||
// uDebugL("report cont:%s\n", pCont);
|
||||
|
|
|
|||
|
|
@ -70,17 +70,18 @@ int32_t walNextValidMsg(SWalReader *pReader) {
|
|||
int64_t fetchVer = pReader->curVersion;
|
||||
int64_t lastVer = walGetLastVer(pReader->pWal);
|
||||
int64_t committedVer = walGetCommittedVer(pReader->pWal);
|
||||
int64_t appliedVer = walGetAppliedVer(pReader->pWal);
|
||||
// int64_t appliedVer = walGetAppliedVer(pReader->pWal);
|
||||
|
||||
if(appliedVer < committedVer){ // wait apply ver equal to commit ver, otherwise may lost data when consume data [TD-24010]
|
||||
wDebug("vgId:%d, wal apply ver:%"PRId64" smaller than commit ver:%"PRId64, pReader->pWal->cfg.vgId, appliedVer, committedVer);
|
||||
}
|
||||
// if(appliedVer < committedVer){ // wait apply ver equal to commit ver, otherwise may lost data when consume data [TD-24010]
|
||||
// wDebug("vgId:%d, wal apply ver:%"PRId64" smaller than commit ver:%"PRId64, pReader->pWal->cfg.vgId, appliedVer, committedVer);
|
||||
// }
|
||||
|
||||
int64_t endVer = TMIN(appliedVer, committedVer);
|
||||
// int64_t endVer = TMIN(appliedVer, committedVer);
|
||||
int64_t endVer = committedVer;
|
||||
|
||||
wDebug("vgId:%d, wal start to fetch, index:%" PRId64 ", last index:%" PRId64 " commit index:%" PRId64
|
||||
", applied index:%" PRId64", end index:%" PRId64,
|
||||
pReader->pWal->cfg.vgId, fetchVer, lastVer, committedVer, appliedVer, endVer);
|
||||
", end index:%" PRId64,
|
||||
pReader->pWal->cfg.vgId, fetchVer, lastVer, committedVer, endVer);
|
||||
|
||||
if (fetchVer > endVer){
|
||||
terrno = TSDB_CODE_WAL_LOG_NOT_EXIST;
|
||||
|
|
@ -370,9 +371,9 @@ int32_t walFetchHead(SWalReader *pRead, int64_t ver, SWalCkHead *pHead) {
|
|||
pRead->pWal->vers.appliedVer);
|
||||
|
||||
// TODO: valid ver
|
||||
if (ver > pRead->pWal->vers.appliedVer) {
|
||||
return -1;
|
||||
}
|
||||
// if (ver > pRead->pWal->vers.appliedVer) {
|
||||
// return -1;
|
||||
// }
|
||||
|
||||
if (pRead->curVersion != ver) {
|
||||
code = walReaderSeekVer(pRead, ver);
|
||||
|
|
|
|||
|
|
@ -186,33 +186,6 @@ class RequestHandlerImpl(http.server.BaseHTTPRequestHandler):
|
|||
tdLog.exit("total is null!")
|
||||
|
||||
|
||||
# log_infos ====================================
|
||||
|
||||
if "log_infos" not in infoDict or infoDict["log_infos"]== None:
|
||||
tdLog.exit("log_infos is null!")
|
||||
|
||||
if "logs" not in infoDict["log_infos"] or len(infoDict["log_infos"]["logs"]) < 8:#!= 10:
|
||||
tdLog.exit("logs is null!")
|
||||
|
||||
if "ts" not in infoDict["log_infos"]["logs"][0] or len(infoDict["log_infos"]["logs"][0]["ts"]) <= 10:
|
||||
tdLog.exit("ts is null!")
|
||||
|
||||
if "level" not in infoDict["log_infos"]["logs"][0] or infoDict["log_infos"]["logs"][0]["level"] not in ["error" ,"info" , "debug" ,"trace"]:
|
||||
tdLog.exit("level is null!")
|
||||
|
||||
if "content" not in infoDict["log_infos"]["logs"][0] or len(infoDict["log_infos"]["logs"][0]["ts"]) <= 1:
|
||||
tdLog.exit("content is null!")
|
||||
|
||||
if "summary" not in infoDict["log_infos"] or len(infoDict["log_infos"]["summary"])!= 4:
|
||||
tdLog.exit("summary is null!")
|
||||
|
||||
|
||||
if "total" not in infoDict["log_infos"]["summary"][0] or infoDict["log_infos"]["summary"][0]["total"] < 0 :
|
||||
tdLog.exit("total is null!")
|
||||
|
||||
if "level" not in infoDict["log_infos"]["summary"][0] or infoDict["log_infos"]["summary"][0]["level"] not in ["error" ,"info" , "debug" ,"trace"]:
|
||||
tdLog.exit("level is null!")
|
||||
|
||||
def do_GET(self):
|
||||
"""
|
||||
process GET request
|
||||
|
|
@ -315,4 +288,3 @@ class TDTestCase:
|
|||
|
||||
tdCases.addLinux(__file__, TDTestCase())
|
||||
tdCases.addWindows(__file__, TDTestCase())
|
||||
|
||||
|
|
|
|||
Loading…
Reference in New Issue