395 lines
15 KiB
Markdown
395 lines
15 KiB
Markdown
---
|
||
sidebar_label: Kubernetes
|
||
title: 在 Kubernetes 上部署 TDengine 集群
|
||
description: 利用 Kubernetes 部署 TDengine 集群的详细指南
|
||
---
|
||
|
||
作为面向云原生架构设计的时序数据库，TDengine 支持 Kubernetes 部署。这里介绍如何使用 YAML 文件一步一步从头创建一个 TDengine 集群，并重点介绍 Kubernetes 环境下 TDengine 的常用操作。
|
||
|
||
## 前置条件
|
||
|
||
要使用 Kubernetes 部署管理 TDengine 集群，需要做好如下准备工作。
|
||
|
||
* 本文适用 Kubernetes v1.5 以上版本
|
||
* 本文和下一章使用 minikube、kubectl 和 helm 等工具进行安装部署，请提前安装好相应软件
|
||
* Kubernetes 已经安装部署并能正常访问使用或更新必要的容器仓库或其他服务
|
||
|
||
以下配置文件也可以从 [GitHub 仓库](https://github.com/taosdata/TDengine-Operator/tree/3.0/src/tdengine) 下载。
|
||
|
||
## 配置 Service 服务
|
||
|
||
创建一个 Service 配置文件：`taosd-service.yaml`，服务名称 `metadata.name` (此处为 "taosd") 将在下一步中使用到。添加 TDengine 所用到的端口：
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: v1
|
||
kind: Service
|
||
metadata:
|
||
name: "taosd"
|
||
labels:
|
||
app: "tdengine"
|
||
spec:
|
||
ports:
|
||
- name: tcp6030
|
||
protocol: "TCP"
|
||
port: 6030
|
||
- name: tcp6041
|
||
protocol: "TCP"
|
||
port: 6041
|
||
selector:
|
||
app: "tdengine"
|
||
```
|
||
|
||
## 有状态服务 StatefulSet
|
||
|
||
根据 Kubernetes 对各类部署的说明，我们将使用 StatefulSet 作为 TDengine 的服务类型。
|
||
创建文件 `tdengine.yaml`，其中 replicas 定义集群节点的数量为 3。节点时区为中国（Asia/Shanghai），每个节点分配 10G 标准（standard）存储。你也可以根据实际情况进行相应修改。
|
||
|
||
```yaml
|
||
---
|
||
apiVersion: apps/v1
|
||
kind: StatefulSet
|
||
metadata:
|
||
name: "tdengine"
|
||
labels:
|
||
app: "tdengine"
|
||
spec:
|
||
serviceName: "taosd"
|
||
replicas: 3
|
||
updateStrategy:
|
||
type: RollingUpdate
|
||
selector:
|
||
matchLabels:
|
||
app: "tdengine"
|
||
template:
|
||
metadata:
|
||
name: "tdengine"
|
||
labels:
|
||
app: "tdengine"
|
||
spec:
|
||
containers:
|
||
- name: "tdengine"
|
||
image: "tdengine/tdengine:3.0.0.0"
|
||
imagePullPolicy: "IfNotPresent"
|
||
ports:
|
||
- name: tcp6030
|
||
protocol: "TCP"
|
||
containerPort: 6030
|
||
- name: tcp6041
|
||
protocol: "TCP"
|
||
containerPort: 6041
|
||
env:
|
||
# POD_NAME for FQDN config
|
||
- name: POD_NAME
|
||
valueFrom:
|
||
fieldRef:
|
||
fieldPath: metadata.name
|
||
# SERVICE_NAME and NAMESPACE for fqdn resolve
|
||
- name: SERVICE_NAME
|
||
value: "taosd"
|
||
- name: STS_NAME
|
||
value: "tdengine"
|
||
- name: STS_NAMESPACE
|
||
valueFrom:
|
||
fieldRef:
|
||
fieldPath: metadata.namespace
|
||
# TZ for timezone settings, we recommend to always set it.
|
||
- name: TZ
|
||
value: "Asia/Shanghai"
|
||
# TAOS_ prefix will configured in taos.cfg, strip prefix and camelCase.
|
||
- name: TAOS_SERVER_PORT
|
||
value: "6030"
|
||
# Must set if you want a cluster.
|
||
- name: TAOS_FIRST_EP
|
||
value: "$(STS_NAME)-0.$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local:$(TAOS_SERVER_PORT)"
|
||
# TAOS_FQND should always be setted in k8s env.
|
||
- name: TAOS_FQDN
|
||
value: "$(POD_NAME).$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local"
|
||
volumeMounts:
|
||
- name: taosdata
|
||
mountPath: /var/lib/taos
|
||
readinessProbe:
|
||
exec:
|
||
command:
|
||
- taos-check
|
||
initialDelaySeconds: 5
|
||
timeoutSeconds: 5000
|
||
livenessProbe:
|
||
exec:
|
||
command:
|
||
- taos-check
|
||
initialDelaySeconds: 15
|
||
periodSeconds: 20
|
||
volumeClaimTemplates:
|
||
- metadata:
|
||
name: taosdata
|
||
spec:
|
||
accessModes:
|
||
- "ReadWriteOnce"
|
||
storageClassName: "standard"
|
||
resources:
|
||
requests:
|
||
storage: "10Gi"
|
||
```
|
||
|
||
## 使用 kubectl 命令部署 TDengine 集群
|
||
|
||
顺序执行以下命令。
|
||
|
||
```bash
|
||
kubectl apply -f taosd-service.yaml
|
||
kubectl apply -f tdengine.yaml
|
||
```
|
||
|
||
上面的配置将生成一个三节点的 TDengine 集群，dnode 为自动配置，可以使用 show dnodes 命令查看当前集群的节点：
|
||
|
||
```bash
|
||
kubectl exec -i -t tdengine-0 -- taos -s "show dnodes"
|
||
kubectl exec -i -t tdengine-1 -- taos -s "show dnodes"
|
||
kubectl exec -i -t tdengine-2 -- taos -s "show dnodes"
|
||
```
|
||
|
||
输出如下：
|
||
|
||
```
|
||
taos> show dnodes
|
||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||
============================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||
Query OK, 3 rows in database (0.003655s)
|
||
```
|
||
|
||
## 使能端口转发
|
||
|
||
利用 kubectl 端口转发功能可以使应用可以访问 Kubernetes 环境运行的 TDengine 集群。
|
||
|
||
```
|
||
kubectl port-forward tdengine-0 6041:6041 &
|
||
```
|
||
|
||
使用 curl 命令验证 TDengine REST API 使用的 6041 接口。
|
||
|
||
```
|
||
$ curl -u root:taosdata -d "show databases" 127.0.0.1:6041/rest/sql
|
||
Handling connection for 6041
|
||
{"code":0,"column_meta":[["name","VARCHAR",64],["create_time","TIMESTAMP",8],["vgroups","SMALLINT",2],["ntables","BIGINT",8],["replica","TINYINT",1],["strict","VARCHAR",4],["duration","VARCHAR",10],["keep","VARCHAR",32],["buffer","INT",4],["pagesize","INT",4],["pages","INT",4],["minrows","INT",4],["maxrows","INT",4],["comp","TINYINT",1],["precision","VARCHAR",2],["status","VARCHAR",10],["retention","VARCHAR",60],["single_stable","BOOL",1],["cachemodel","VARCHAR",11],["cachesize","INT",4],["wal_level","TINYINT",1],["wal_fsync_period","INT",4],["wal_retention_period","INT",4],["wal_retention_size","BIGINT",8],["wal_roll_period","INT",4],["wal_segment_size","BIGINT",8]],"data":[["information_schema",null,null,16,null,null,null,null,null,null,null,null,null,null,null,"ready",null,null,null,null,null,null,null,null,null,null],["performance_schema",null,null,10,null,null,null,null,null,null,null,null,null,null,null,"ready",null,null,null,null,null,null,null,null,null,null]],"rows":2}
|
||
```
|
||
|
||
## 使用 dashboard 进行图形化管理
|
||
|
||
minikube 提供 dashboard 命令支持图形化管理界面。
|
||
|
||
```
|
||
$ minikube dashboard
|
||
* Verifying dashboard health ...
|
||
* Launching proxy ...
|
||
* Verifying proxy health ...
|
||
* Opening http://127.0.0.1:46617/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/ in your default browser...
|
||
http://127.0.0.1:46617/api/v1/namespaces/kubernetes-dashboard/services/http:kubernetes-dashboard:/proxy/
|
||
```
|
||
|
||
对于某些公有云环境，minikube 绑定在 127.0.0.1 IP 地址上无法通过远程访问，需要使用 kubectl proxy 命令将端口映射到 0.0.0.0 IP 地址上，再通过浏览器访问虚拟机公网 IP 和端口以及相同的 dashboard URL 路径即可远程访问 dashboard。
|
||
|
||
```
|
||
$ kubectl proxy --accept-hosts='^.*$' --address='0.0.0.0'
|
||
```
|
||
|
||
## 集群扩容
|
||
|
||
TDengine 集群支持自动扩容：
|
||
|
||
```bash
|
||
kubectl scale statefulsets tdengine --replicas=4
|
||
```
|
||
|
||
上面命令行中参数 `--replica=4` 表示要将 TDengine 集群扩容到 4 个节点，执行后首先检查 POD 的状态：
|
||
|
||
```bash
|
||
kubectl get pods -l app=tdengine
|
||
```
|
||
|
||
输出如下：
|
||
|
||
```
|
||
NAME READY STATUS RESTARTS AGE
|
||
tdengine-0 1/1 Running 0 161m
|
||
tdengine-1 1/1 Running 0 161m
|
||
tdengine-2 1/1 Running 0 32m
|
||
tdengine-3 1/1 Running 0 32m
|
||
```
|
||
|
||
此时 POD 的状态仍然是 Running，TDengine 集群中的 dnode 状态要等 POD 状态为 `ready` 之后才能看到：
|
||
|
||
```bash
|
||
kubectl exec -i -t tdengine-3 -- taos -s "show dnodes"
|
||
```
|
||
|
||
扩容后的四节点 TDengine 集群的 dnode 列表:
|
||
|
||
```
|
||
taos> show dnodes
|
||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||
============================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||
4 | tdengine-3.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:33:16.039 | |
|
||
Query OK, 4 rows in database (0.008377s)
|
||
```
|
||
|
||
## 集群缩容
|
||
|
||
由于 TDengine 集群在扩缩容时会对数据进行节点间迁移，使用 kubectl 命令进行缩容需要首先使用 "drop dnodes" 命令，节点删除完成后再进行 Kubernetes 集群缩容。
|
||
|
||
注意：由于 Kubernetes Statefulset 中 Pod 的只能按创建顺序逆序移除，所以 TDengine drop dnode 也需要按照创建顺序逆序移除，否则会导致 Pod 处于错误状态。
|
||
|
||
```
|
||
$ kubectl exec -i -t tdengine-0 -- taos -s "drop dnode 4"
|
||
```
|
||
|
||
```bash
|
||
$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||
|
||
taos> show dnodes
|
||
id | endpoint | vnodes | support_vnodes | status | create_time | note |
|
||
============================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:14:57.285 | |
|
||
2 | tdengine-1.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:11.302 | |
|
||
3 | tdengine-2.taosd.default.sv... | 0 | 256 | ready | 2022-08-10 13:15:23.290 | |
|
||
Query OK, 3 rows in database (0.004861s)
|
||
```
|
||
|
||
确认移除成功后（使用 kubectl exec -i -t tdengine-0 -- taos -s "show dnodes" 查看和确认 dnode 列表），使用 kubectl 命令移除 POD：
|
||
|
||
```
|
||
kubectl scale statefulsets tdengine --replicas=3
|
||
```
|
||
|
||
最后一个 POD 将会被删除。使用命令 kubectl get pods -l app=tdengine 查看POD状态：
|
||
|
||
```
|
||
$ kubectl get pods -l app=tdengine
|
||
NAME READY STATUS RESTARTS AGE
|
||
tdengine-0 1/1 Running 0 4m7s
|
||
tdengine-1 1/1 Running 0 3m55s
|
||
tdengine-2 1/1 Running 0 2m28s
|
||
```
|
||
|
||
POD删除后，需要手动删除PVC，否则下次扩容时会继续使用以前的数据导致无法正常加入集群。
|
||
|
||
```bash
|
||
$ kubectl delete pvc taosdata-tdengine-3
|
||
```
|
||
|
||
此时的集群状态是安全的，需要时还可以再次进行扩容：
|
||
|
||
```bash
|
||
$ kubectl scale statefulsets tdengine --replicas=4
|
||
statefulset.apps/tdengine scaled
|
||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl get pods -l app=tdengine
|
||
NAME READY STATUS RESTARTS AGE
|
||
tdengine-0 1/1 Running 0 35m
|
||
tdengine-1 1/1 Running 0 34m
|
||
tdengine-2 1/1 Running 0 12m
|
||
tdengine-3 0/1 ContainerCreating 0 4s
|
||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl get pods -l app=tdengine
|
||
NAME READY STATUS RESTARTS AGE
|
||
tdengine-0 1/1 Running 0 35m
|
||
tdengine-1 1/1 Running 0 34m
|
||
tdengine-2 1/1 Running 0 12m
|
||
tdengine-3 0/1 Running 0 7s
|
||
it@k8s-2:~/TDengine-Operator/src/tdengine$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||
|
||
taos> show dnodes
|
||
id | endpoint | vnodes | support_vnodes | status | create_time | offline reason |
|
||
======================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 17:38:49.012 | |
|
||
2 | tdengine-1.taosd.default.sv... | 1 | 4 | ready | 2022-07-25 17:39:01.517 | |
|
||
5 | tdengine-2.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 18:01:36.479 | |
|
||
6 | tdengine-3.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 18:13:54.411 | |
|
||
Query OK, 4 row(s) in set (0.001348s)
|
||
```
|
||
|
||
## 清理 TDengine 集群
|
||
|
||
完整移除 TDengine 集群，需要分别清理 statefulset、svc、configmap、pvc。
|
||
|
||
```bash
|
||
kubectl delete statefulset -l app=tdengine
|
||
kubectl delete svc -l app=tdengine
|
||
kubectl delete pvc -l app=tdengine
|
||
kubectl delete configmap taoscfg
|
||
|
||
```
|
||
|
||
## 常见错误
|
||
|
||
### 错误一
|
||
|
||
未进行 "drop dnode" 直接进行缩容，由于 TDengine 尚未删除节点，缩容 pod 导致 TDengine 集群中部分节点处于 offline 状态。
|
||
|
||
```
|
||
$ kubectl exec -it tdengine-0 -- taos -s "show dnodes"
|
||
|
||
taos> show dnodes
|
||
id | endpoint | vnodes | support_vnodes | status | create_time | offline reason |
|
||
======================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 0 | 4 | ready | 2022-07-25 17:38:49.012 | |
|
||
2 | tdengine-1.taosd.default.sv... | 1 | 4 | ready | 2022-07-25 17:39:01.517 | |
|
||
5 | tdengine-2.taosd.default.sv... | 0 | 4 | offline | 2022-07-25 18:01:36.479 | status msg timeout |
|
||
6 | tdengine-3.taosd.default.sv... | 0 | 4 | offline | 2022-07-25 18:13:54.411 | status msg timeout |
|
||
Query OK, 4 row(s) in set (0.001323s)
|
||
```
|
||
|
||
### 错误二
|
||
|
||
TDengine 集群会持有 replica 参数，如果缩容后的节点数小于这个值，集群将无法使用：
|
||
|
||
创建一个库使用 replica 参数为 2，插入部分数据：
|
||
|
||
```bash
|
||
kubectl exec -i -t tdengine-0 -- \
|
||
taos -s \
|
||
"create database if not exists test replica 2;
|
||
use test;
|
||
create table if not exists t1(ts timestamp, n int);
|
||
insert into t1 values(now, 1)(now+1s, 2);"
|
||
|
||
|
||
```
|
||
|
||
缩容到单节点：
|
||
|
||
```bash
|
||
kubectl scale statefulsets tdengine --replicas=1
|
||
|
||
```
|
||
|
||
在 TDengine CLI 中的所有数据库操作将无法成功。
|
||
|
||
```
|
||
taos> show dnodes;
|
||
id | end_point | vnodes | cores | status | role | create_time | offline reason |
|
||
======================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 2 | 40 | ready | any | 2021-06-01 15:55:52.562 | |
|
||
2 | tdengine-1.taosd.default.sv... | 1 | 40 | offline | any | 2021-06-01 15:56:07.212 | status msg timeout |
|
||
Query OK, 2 row(s) in set (0.000845s)
|
||
|
||
taos> show dnodes;
|
||
id | end_point | vnodes | cores | status | role | create_time | offline reason |
|
||
======================================================================================================================================
|
||
1 | tdengine-0.taosd.default.sv... | 2 | 40 | ready | any | 2021-06-01 15:55:52.562 | |
|
||
2 | tdengine-1.taosd.default.sv... | 1 | 40 | offline | any | 2021-06-01 15:56:07.212 | status msg timeout |
|
||
Query OK, 2 row(s) in set (0.000837s)
|
||
|
||
taos> use test;
|
||
Database changed.
|
||
|
||
taos> insert into t1 values(now, 3);
|
||
|
||
DB error: Unable to resolve FQDN (0.013874s)
|
||
|
||
```
|