homework-jianmu/2.0/documentation20/en/00.index/docs.md

TDengine Documentation

TDengine is a highly efficient platform to store, query, and analyze time-series data. It is specially designed and optimized for IoT, Internet of Vehicles, Industrial IoT, IT Infrastructure and Application Monitoring, etc. It works like a relational database, such as MySQL, but you are strongly encouraged to read through the following documentation before you experience it, especially the Data Modeling sections. In addition to this document, you should also download and read the technology white paper. For the older TDengine version 1.6 documentation, please click here.

TDengine Introduction

• TDengine Introduction and Features
• TDengine Use Scenes
• TDengine Performance Metrics and Verification

Getting Started

• Quick Install: install via source code/package / Docker within seconds
• Quick Launch: start / stop TDengine quickly with systemctl
• Command-line : an easy way to access TDengine server
• Experience Lightning Speed: running a demo, inserting/querying data to experience faster speed
• List of Supported Platforms: a list of platforms supported by TDengine server and client
• Deploy to Kubernetes：a detailed guide for TDengine deployment in Kubernetes environment

Overall Architecture

• Data Model: relational database model, but one table for one data collection point with static tags
• Cluster and Primary Logical Unit: Take advantage of NoSQL architecture, high availability and horizontal scalability
• Storage Model and Data Partitioning/Sharding: tag data is separated from time-series data, sharded by vnodes and partitioned by time
• Data Writing and Replication Process: records received are written to WAL, cached, with acknowledgement sent back to client, while supporting data replications
• Caching and Persistence: latest records are cached in memory, but are written in columnar format with an ultra-high compression ratio
• Data Query: support various SQL functions, downsampling, interpolation, and multi-table aggregation

Data Modeling

• Create a Database: create a database for all data collection points with similar data characteristics
• Create a Super Table(STable): create a STable for all data collection points with the same type
• Create a Table: use STable as the template to create a table for each data collecting point

Efficient Data Ingestion

• Data Writing via SQL: write one or multiple records into one or multiple tables via SQL insert command
• Data Writing via Prometheus: Configure Prometheus to write data directly without any code
• Data Writing via Telegraf: Configure Telegraf to write collected data directly without any code
• Data Writing via EMQ X: Configure EMQ X to write MQTT data directly without any code
• Data Writing via HiveMQ Broker: Configure HiveMQ to write MQTT data directly without any code

Efficient Data Querying

• Major Features: support various standard query functions, setting filter conditions, and querying per time segment
• Multi-table Aggregation: use STable and set tag filter conditions to perform efficient aggregation
• Downsampling: aggregate data in successive time windows, support interpolation

TAOS SQL

• Data Types: support timestamp, int, float, nchar, bool, and other types
• Database Management: add, drop, check databases
• Table Management: add, drop, check, alter tables
• STable Management: add, drop, check, alter STables
• Tag Management: add, drop, alter tags
• Inserting Records: write single/multiple records a table, multiple records across tables, and historical data
• Data Query: support time segment, value filtering, sorting, manual paging of query results, etc
• SQL Function: support various aggregation functions, selection functions, and calculation functions, such as avg, min, diff, etc
• Cutting and Aggregation: aggregate and reduce the dimension after cutting table data by time segment
• Boundary Restrictions: restrictions for the library, table, SQL, and others
• Error Code: TDengine 2.0 error codes and corresponding decimal codes

Advanced Features

• Continuous Query: Based on sliding windows, the data stream is automatically queried and calculated at regular intervals
• Data Publisher/Subscriber: subscribe to the newly arrived data like a typical messaging system
• Cache: the newly arrived data of each device/table will always be cached
• Alarm Monitoring: automatically monitor out-of-threshold data, and actively push it based-on configuration rules

Connector

• C/C++ Connector: primary method to connect to TDengine server through libtaos client library
• Java Connector(JDBC): driver for connecting to the server from Java applications using the JDBC API
• Python Connector: driver for connecting to TDengine server from Python applications
• RESTful Connector: a simple way to interact with TDengine via HTTP
• Go Connector: driver for connecting to TDengine server from Go applications
• Node.js Connector: driver for connecting to TDengine server from Node.js applications
• C# Connector: driver for connecting to TDengine server from C# applications
• Windows Client: compile your own Windows client, which is required by various connectors on the Windows environment

Connections with Other Tools

• Grafana: query the data saved in TDengine and provide visualization
• MATLAB: access data stored in TDengine server via JDBC configured within MATLAB
• R: access data stored in TDengine server via JDBC configured within R
• IDEA Database: use TDengine visually through IDEA Database Management Tool

Installation and Management of TDengine Cluster

• Preparation: important steps before deploying TDengine for production usage
• Create the First Node: just follow the steps in quick start
• Create Subsequent Nodes: configure taos.cfg for new nodes to add more to the existing cluster
• Node Management: add, delete, and check nodes in the cluster
• High-availability of Vnode: implement high-availability of Vnode through replicas
• Mnode Management: mnodes are created automatically without any manual intervention
• Load Balancing: automatically performed once the number of nodes or load changes
• Offline Node Processing: any node that offline for more than a certain period will be removed from the cluster
• Arbitrator: used in the case of an even number of replicas to prevent split-brain

TDengine Operation and Maintenance

• Capacity Planning: Estimating hardware resources based on scenarios
• Fault Tolerance and Disaster Recovery: set the correct WAL and number of data replicas
• System Configuration: port, cache size, file block size, and other system configurations
• User Management: add/delete TDengine users, modify user password
• Import Data: import data into TDengine from either script or CSV file
• Export Data: export data either from TDengine shell or from the taosdump tool
• System Monitor: monitor the system connections, queries, streaming calculation, logs, and events
• File Directory Structure: directories where TDengine data files and configuration files located
• Parameter Limitss and Reserved Keywords: TDengine’s list of parameter limits and reserved keywords

Performance: TDengine vs Others

• Performance: TDengine vs OpenTSDB
• Performance: TDengine vs Cassandra
• Performance: TDengine vs InfluxDB
• Performance Test Reports of TDengine vs InfluxDB/OpenTSDB/Cassandra/MySQL/ClickHouse

More on IoT Big Data

• Characteristics of IoT and Industry Internet Big Data
• Features and Functions of IoT Big Data platforms
• Why don’t General Big Data Platforms Fit IoT Scenarios?
• Why TDengine is the best choice for IoT, Internet of Vehicles, and Industry Internet Big Data platforms?
• Technical Blog: More technical analysis and architecture design articles

FAQ

• FAQ: Common questions and answers