12 KiB
| title | description |
|---|---|
| TDengine Monitoring | This document describes how to monitor your TDengine cluster. |
After TDengine is started, it automatically writes monitoring data including CPU, memory and disk usage, bandwidth, number of requests, disk I/O speed, slow queries, into a designated database at a predefined interval through taosKeeper. Additionally, some important system operations, like logon, create user, drop database, and alerts and warnings generated in TDengine are written into the log database too. A system operator can view the data in log database from TDengine CLI or from a web console.
The collection of the monitoring information is enabled by default, but can be disabled by parameter monitor in the configuration file.
TDinsight
TDinsight is a complete solution which uses the monitoring database log mentioned previously, and Grafana, to monitor a TDengine cluster.
A script TDinsight.sh is provided to deploy TDinsight automatically.
Download TDinsight.sh with the below command:
wget https://github.com/taosdata/grafanaplugin/raw/master/dashboards/TDinsight.sh
chmod +x TDinsight.sh
Prepare:
-
TDengine Server
- The URL of REST service: for example
http://localhost:6041if TDengine is deployed locally - User name and password
- The URL of REST service: for example
-
Grafana Alert Notification
You can use below command to setup Grafana alert notification.
An existing Grafana Notification Channel can be specified with parameter -E, the notifier uid of the channel can be obtained by curl -u admin:admin localhost:3000/api/alert-notifications |jq
```bash
./TDinsight.sh -a http://localhost:6041 -u root -p taosdata -E <notifier uid>
```
Launch TDinsight.sh with the command above and restart Grafana, then open Dashboard http://localhost:3000/d/tdinsight.
log database
The data of tdinsight dashboard is stored in log database (default. You can change it in taoskeeper's config file. For more infrmation, please reference to taoskeeper document). The taoskeeper will create log database on taoskeeper startup.
taosd_cluster_basic table
taosd_cluster_basic table contains cluster basic information.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| first_ep | VARCHAR | first ep of cluster | |
| first_ep_dnode_id | INT | dnode id or first_ep | |
| cluster_version | VARCHAR | tdengine version. such as: 3.0.4.0 | |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_cluster_info table
taosd_cluster_info table contains cluster information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| cluster_uptime | DOUBLE | seconds of master's uptime | |
| dbs_total | DOUBLE | total number of databases in cluster | |
| tbs_total | DOUBLE | total number of tables in cluster | |
| stbs_total | DOUBLE | total number of stables in cluster | |
| dnodes_total | DOUBLE | total number of dnodes in cluster | |
| dnodes_alive | DOUBLE | total number of dnodes in ready state | |
| mnodes_total | DOUBLE | total number of mnodes in cluster | |
| mnodes_alive | DOUBLE | total number of mnodes in ready state | |
| vgroups_total | DOUBLE | total number of vgroups in cluster | |
| vgroups_alive | DOUBLE | total number of vgroups in ready state | |
| vnodes_total | DOUBLE | total number of vnode in cluster | |
| vnodes_alive | DOUBLE | total number of vnode in ready state | |
| connections_total | DOUBLE | total number of connections to cluster | |
| topics_total | DOUBLE | total number of topics in cluster | |
| streams_total | DOUBLE | total number of streams in cluster | |
| grants_expire_time | DOUBLE | time until grants expire in seconds | |
| grants_timeseries_used | DOUBLE | timeseries used | |
| grants_timeseries_total | DOUBLE | total timeseries | |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_vgroups_info table
taosd_vgroups_info table contains vgroups information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| tables_num | DOUBLE | number of tables per vgroup | |
| status | DOUBLE | status, value range:unsynced = 0, ready = 1 | |
| vgroup_id | VARCHAR | TAG | vgroup id |
| database_name | VARCHAR | TAG | database for the vgroup |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_dnodes_info table
taosd_dnodes_info table contains dnodes information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| uptime | DOUBLE | dnode uptime in seconds |
|
| cpu_engine | DOUBLE | cpu usage of tdengine. read from /proc/<taosd_pid>/stat |
|
| cpu_system | DOUBLE | cpu usage of server. read from /proc/stat |
|
| cpu_cores | DOUBLE | cpu cores of server | |
| mem_engine | DOUBLE | memory usage of tdengine. read from /proc/<taosd_pid>/status |
|
| mem_free | DOUBLE | available memory on the server in KB |
|
| mem_total | DOUBLE | total memory of server in KB |
|
| disk_used | DOUBLE | usage of data dir in bytes |
|
| disk_total | DOUBLE | the capacity of data dir in bytes |
|
| system_net_in | DOUBLE | network throughput rate in byte/s. read from /proc/net/dev |
|
| system_net_out | DOUBLE | network throughput rate in byte/s. read from /proc/net/dev |
|
| io_read | DOUBLE | io throughput rate in byte/s. read from /proc/<taosd_pid>/io |
|
| io_write | DOUBLE | io throughput rate in byte/s. read from /proc/<taosd_pid>/io |
|
| io_read_disk | DOUBLE | io throughput rate of disk in byte/s. read from /proc/<taosd_pid>/io |
|
| io_write_disk | DOUBLE | io throughput rate of disk in byte/s. read from /proc/<taosd_pid>/io |
|
| vnodes_num | DOUBLE | number of vnodes per dnode | |
| masters | DOUBLE | number of master vnodes | |
| has_mnode | DOUBLE | if the dnode has mnode, value range:include=1, not_include=0 | |
| has_qnode | DOUBLE | if the dnode has qnode, value range:include=1, not_include=0 | |
| has_snode | DOUBLE | if the dnode has snode, value range:include=1, not_include=0 | |
| has_bnode | DOUBLE | if the dnode has bnode, value range:include=1, not_include=0 | |
| error_log_count | DOUBLE | error count | |
| info_log_count | DOUBLE | info count | |
| debug_log_count | DOUBLE | debug count | |
| trace_log_count | DOUBLE | trace count | |
| dnode_id | VARCHAR | TAG | dnode id |
| dnode_ep | VARCHAR | TAG | dnode endpoint |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_dnodes_status table
taosd_dnodes_status table contains dnodes information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| status | DOUBLE | dnode status, value range:ready=1，offline =0 | |
| dnode_id | VARCHAR | TAG | dnode id |
| dnode_ep | VARCHAR | TAG | dnode endpoint |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_dnodes_log_dir table
log_dir table contains log directory information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| avail | DOUBLE | available space for log directory in bytes |
|
| used | DOUBLE | used space for data directory in bytes |
|
| total | DOUBLE | total space for data directory in bytes |
|
| name | VARCHAR | TAG | log directory. default is /var/log/taos/ |
| dnode_id | VARCHAR | TAG | dnode id |
| dnode_ep | VARCHAR | TAG | dnode endpoint |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_dnodes_data_dir table
taosd_dnodes_data_dir table contains data directory information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| avail | DOUBLE | available space for data directory in bytes |
|
| used | DOUBLE | used space for data directory in bytes |
|
| total | DOUBLE | total space for data directory in bytes |
|
| level | VARCHAR | TAG | level for multi-level storage |
| name | VARCHAR | TAG | data directory. default is /var/lib/taos |
| dnode_id | VARCHAR | TAG | dnode id |
| dnode_ep | VARCHAR | TAG | dnode endpoint |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_mnodes_info table
taosd_mnodes_info table contains mnode information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| role | DOUBLE | the role of mnode. value range:offline = 0,follower = 100,candidate = 101,leader = 102,error = 103,learner = 104 | |
| mnode_id | VARCHAR | TAG | master node id |
| mnode_ep | VARCHAR | TAG | master node endpoint |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_vnodes_role table
taosd_vnodes_role table contains vnode role information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| role | DOUBLE | role. value range:offline = 0,follower = 100,candidate = 101,leader = 102,error = 103,learner = 104 | |
| vgroup_id | VARCHAR | TAG | vgroup id |
| database_name | VARCHAR | TAG | database for the vgroup |
| dnode_id | VARCHAR | TAG | dnode id |
| cluster_id | VARCHAR | TAG | cluster id |
taosd_sql_req table
taosd_sql_req tables contains taosd sql records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| count | DOUBLE | sql count | |
| result | VARCHAR | TAG | sql execution result，value range: Success, Failed |
| username | VARCHAR | TAG | user name who executed the sql |
| sql_type | VARCHAR | TAG | sql type，value range:inserted_rows |
| dnode_id | VARCHAR | TAG | dnode id |
| dnode_ep | VARCHAR | TAG | dnode endpoint |
| vgroup_id | VARCHAR | TAG | dnode id |
| cluster_id | VARCHAR | TAG | cluster id |
taos_sql_req 表
taos_sql_req tables contains taos sql records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| count | DOUBLE | sql count | |
| result | VARCHAR | TAG | sql execution result，value range: Success, Failed |
| username | VARCHAR | TAG | user name who executed the sql |
| sql_type | VARCHAR | TAG | sql type，value range:select, insert，delete |
| cluster_id | VARCHAR | TAG | cluster id |
taos_slow_sql 表
taos_slow_sql ables contains taos slow sql records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| count | DOUBLE | sql count | |
| result | VARCHAR | TAG | sql execution result，value range: Success, Failed |
| username | VARCHAR | TAG | user name who executed the sql |
| duration | VARCHAR | TAG | sql execution duration，value range:3-10s,10-100s,100-1000s,1000s- |
| cluster_id | VARCHAR | TAG | cluster id |
keeper_monitor table
keeper_monitor table contains keeper monitor information records.
| field | type | is_tag | comment |
|---|---|---|---|
| ts | TIMESTAMP | timestamp | |
| cpu | FLOAT | cpu usage | |
| mem | FLOAT | memory usage | |
| identify | NCHAR | TAG |
taosadapter_restful_http_request_total table
taosadapter_restful_http_request_total table contains taosadapter rest request information record. The timestamp column of this table is _ts.
| field | type | is_tag | comment |
|---|---|---|---|
| _ts | TIMESTAMP | timestamp | |
| gauge | DOUBLE | metric value | |
| client_ip | NCHAR | TAG | client ip |
| endpoint | NCHAR | TAG | taosadpater endpoint |
| request_method | NCHAR | TAG | request method |
| request_uri | NCHAR | TAG | request uri |
| status_code | NCHAR | TAG | status code |
taosadapter_restful_http_request_fail table
taosadapter_restful_http_request_fail table contains taosadapter failed rest request information record. The timestamp column of this table is _ts.
| field | type | is_tag | comment |
|---|---|---|---|
| _ts | TIMESTAMP | timestamp | |
| gauge | DOUBLE | metric value | |
| client_ip | NCHAR | TAG | client ip |
| endpoint | NCHAR | TAG | taosadpater endpoint |
| request_method | NCHAR | TAG | request method |
| request_uri | NCHAR | TAG | request uri |
| status_code | NCHAR | TAG | status code |
taosadapter_restful_http_request_in_flight table
taosadapter_restful_http_request_in_flight table contains taosadapter rest request information record in real time. The timestamp column of this table is _ts.
| field | type | is_tag | comment |
|---|---|---|---|
| _ts | TIMESTAMP | timestamp | |
| gauge | DOUBLE | metric value | |
| endpoint | NCHAR | TAG | taosadpater endpoint |
taosadapter_restful_http_request_summary_milliseconds table
taosadapter_restful_http_request_summary_milliseconds table contains the summary or rest information record. The timestamp column of this table is _ts.
| field | type | is_tag | comment |
|---|---|---|---|
| _ts | TIMESTAMP | timestamp | |
| count | DOUBLE | ||
| sum | DOUBLE | ||
| 0.5 | DOUBLE | ||
| 0.9 | DOUBLE | ||
| 0.99 | DOUBLE | ||
| 0.1 | DOUBLE | ||
| 0.2 | DOUBLE | ||
| endpoint | NCHAR | TAG | taosadpater endpoint |
| request_method | NCHAR | TAG | request method |
| request_uri | NCHAR | TAG | request uri |
taosadapter_system_mem_percent table
taosadapter_system_mem_percent table contains taosadapter memory usage information. The timestamp of this table is _ts.
| field | type | is_tag | comment |
|---|---|---|---|
| _ts | TIMESTAMP | timestamp | |
| gauge | DOUBLE | metric value | |
| endpoint | NCHAR | TAG | taosadpater endpoint |
taosadapter_system_cpu_percent table
taosadapter_system_cpu_percent table contains taosadapter cup usage information. The timestamp of this table is _ts.
| field | type | is_tag | comment |
|---|---|---|---|
| _ts | TIMESTAMP | timestamp | |
| gauge | DOUBLE | mertic value | |
| endpoint | NCHAR | TAG | taosadpater endpoint |