Merge branch '3.0' into main
This commit is contained in:
Shengliang Guan
commit
f0b51ac04f
|
|
@ -2,7 +2,7 @@
|
|||
# taos-tools
|
||||
ExternalProject_Add(taos-tools
|
||||
GIT_REPOSITORY https://github.com/taosdata/taos-tools.git
|
||||
GIT_TAG d237772
|
||||
GIT_TAG e8bfca6
|
||||
SOURCE_DIR "${TD_SOURCE_DIR}/tools/taos-tools"
|
||||
BINARY_DIR ""
|
||||
#BUILD_IN_SOURCE TRUE
|
||||
|
|
|
|||
|
|
@ -5,7 +5,7 @@ slug: /
|
|||
---
|
||||
|
||||
|
||||
TDengine is an open source, cloud native time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. This document is the TDengine user manual. It introduces the basic, as well as novel concepts, in TDengine, and also talks in detail about installation, features, SQL, APIs, operation, maintenance, kernel design and other topics. It’s written mainly for architects, developers and system administrators.
|
||||
TDengine is an [open source](https://tdengine.com/tdengine/open-source-time-series-database/), [cloud native](https://tdengine.com/tdengine/cloud-native-time-series-database/) time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. It enables efficient, real-time data ingestion, processing, and monitoring of TB and even PB scale data per day, generated by billions of sensors and data collectors. This document is the TDengine user manual. It introduces the basic, as well as novel concepts, in TDengine, and also talks in detail about installation, features, SQL, APIs, operation, maintenance, kernel design and other topics. It’s written mainly for architects, developers and system administrators.
|
||||
|
||||
To get an overview of TDengine, such as a feature list, benchmarks, and competitive advantages, please browse through the [Introduction](./intro) section.
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ title: Introduction
|
|||
toc_max_heading_level: 2
|
||||
---
|
||||
|
||||
TDengine is an open source, high-performance, cloud native time-series database optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. Its code, including its cluster feature is open source under GNU AGPL v3.0. Besides the database engine, it provides [caching](../develop/cache), [stream processing](../develop/stream), [data subscription](../develop/tmq) and other functionalities to reduce the system complexity and cost of development and operation.
|
||||
TDengine is an open source, high-performance, cloud native [time-series database](https://tdengine.com/tsdb/) optimized for Internet of Things (IoT), Connected Cars, and Industrial IoT. Its code, including its cluster feature is open source under GNU AGPL v3.0. Besides the database engine, it provides [caching](/develop/cache), [stream processing](/develop/stream), [data subscription](/develop/tmq) and other functionalities to reduce the system complexity and cost of development and operation.
|
||||
|
||||
This section introduces the major features, competitive advantages, typical use-cases and benchmarks to help you get a high level overview of TDengine.
|
||||
|
||||
|
|
@ -11,21 +11,33 @@ This section introduces the major features, competitive advantages, typical use-
|
|||
|
||||
The major features are listed below:
|
||||
|
||||
1. While TDengine supports [using SQL to insert](/develop/insert-data/sql-writing), it also supports [Schemaless writing](/reference/schemaless/) just like NoSQL databases. TDengine also supports standard protocols like [InfluxDB LINE](/develop/insert-data/influxdb-line)，[OpenTSDB Telnet](/develop/insert-data/opentsdb-telnet), [OpenTSDB JSON ](/develop/insert-data/opentsdb-json) among others.
|
||||
2. TDengine supports seamless integration with third-party data collection agents like [Telegraf](/third-party/telegraf)，[Prometheus](/third-party/prometheus)，[StatsD](/third-party/statsd)，[collectd](/third-party/collectd)，[icinga2](/third-party/icinga2), [TCollector](/third-party/tcollector), [EMQX](/third-party/emq-broker), [HiveMQ](/third-party/hive-mq-broker). These agents can write data into TDengine with simple configuration and without a single line of code.
|
||||
3. Support for [all kinds of queries](/develop/query-data), including aggregation, nested query, downsampling, interpolation and others.
|
||||
4. Support for [user defined functions](/develop/udf).
|
||||
5. Support for [caching](/develop/cache). TDengine always saves the last data point in cache, so Redis is not needed in some scenarios.
|
||||
6. Support for [continuous query](../develop/stream).
|
||||
7. Support for [data subscription](../develop/tmq with the capability to specify filter conditions.
|
||||
8. Support for [cluster](../deployment/), with the capability of increasing processing power by adding more nodes. High availability is supported by replication.
|
||||
9. Provides an interactive [command-line interface](/reference/taos-shell) for management, maintenance and ad-hoc queries.
|
||||
10. Provides many ways to [import](/operation/import) and [export](/operation/export) data.
|
||||
11. Provides [monitoring](/operation/monitor) on running instances of TDengine.
|
||||
12. Provides [connectors](/reference/connector/) for [C/C++](/reference/connector/cpp), [Java](/reference/connector/java), [Python](/reference/connector/python), [Go](/reference/connector/go), [Rust](/reference/connector/rust), [Node.js](/reference/connector/node) and other programming languages.
|
||||
13. Provides a [REST API](/reference/rest-api/).
|
||||
14. Supports seamless integration with [Grafana](/third-party/grafana) for visualization.
|
||||
15. Supports seamless integration with Google Data Studio.
|
||||
1. Insert data
|
||||
* supports [using SQL to insert](/develop/insert-data/sql-writing).
|
||||
* supports [schemaless writing](/reference/schemaless/) just like NoSQL databases. It also supports standard protocols like [InfluxDB LINE](/develop/insert-data/influxdb-line)，[OpenTSDB Telnet](/develop/insert-data/opentsdb-telnet), [OpenTSDB JSON ](/develop/insert-data/opentsdb-json) among others.
|
||||
* supports seamless integration with third-party tools like [Telegraf](/third-party/telegraf/), [Prometheus](/third-party/prometheus/), [collectd](/third-party/collectd/), [StatsD](/third-party/statsd/), [TCollector](/third-party/tcollector/) and [icinga2/](/third-party/icinga2/), they can write data into TDengine with simple configuration and without a single line of code.
|
||||
2. Query data
|
||||
* supports standard [SQL](/taos-sql/), including nested query.
|
||||
* supports [time series specific functions](/taos-sql/function/#time-series-extensions) and [time series specific queries](/taos-sql/distinguished), like downsampling, interpolation, cumulated sum, time weighted average, state window, session window and many others.
|
||||
* supports [user defined functions](/taos-sql/udf).
|
||||
3. [Caching](/develop/cache/): TDengine always saves the last data point in cache, so Redis is not needed for time-series data processing.
|
||||
4. [Stream Processing](/develop/stream/): not only is the continuous query is supported, but TDengine also supports even driven stream processing, so Flink or spark is not needed for time-series daata processing.
|
||||
5. [Data Dubscription](/develop/tmq/): application can subscribe a table or a set of tables. API is the same as Kafka, but you can specify filter conditions.
|
||||
6. Visualization
|
||||
* supports seamless integration with [Grafana](/third-party/grafana/) for visualization.
|
||||
* supports seamless integration with Google Data Studio.
|
||||
7. Cluster
|
||||
* supports [cluster](/deployment/) with the capability of increasing processing power by adding more nodes.
|
||||
* supports [deployment on Kubernetes](/deployment/k8s/)
|
||||
* supports high availability via data replication.
|
||||
8. Administration
|
||||
* provides [monitoring](/operation/monitor) on running instances of TDengine.
|
||||
* provides many ways to [import](/operation/import) and [export](/operation/export) data.
|
||||
9. Tools
|
||||
* provides an interactive [command-line interface](/reference/taos-shell) for management, maintenance and ad-hoc queries.
|
||||
* provides a tool [taosBenchmark](/reference/taosbenchmark/) for testing the performance of TDengine.
|
||||
10. Programming
|
||||
* provides [connectors](/reference/connector/) for [C/C++](/reference/connector/cpp), [Java](/reference/connector/java), [Python](/reference/connector/python), [Go](/reference/connector/go), [Rust](/reference/connector/rust), [Node.js](/reference/connector/node) and other programming languages.
|
||||
* provides a [REST API](/reference/rest-api/).
|
||||
|
||||
For more details on features, please read through the entire documentation.
|
||||
|
||||
|
|
@ -33,17 +45,18 @@ For more details on features, please read through the entire documentation.
|
|||
|
||||
By making full use of [characteristics of time series data](https://tdengine.com/tsdb/characteristics-of-time-series-data/), TDengine differentiates itself from other time series databases, with the following advantages.
|
||||
|
||||
- **High-Performance**: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
|
||||
- **[High-Performance](https://tdengine.com/tdengine/high-performance-time-series-database/)**: TDengine is the only time-series database to solve the high cardinality issue to support billions of data collection points while out performing other time-series databases for data ingestion, querying and data compression.
|
||||
|
||||
- **Simplified Solution**: Through built-in caching, stream processing and data subscription features, TDengine provides a simplified solution for time-series data processing. It reduces system design complexity and operation costs significantly.
|
||||
- **[Simplified Solution](https://tdengine.com/tdengine/simplified-time-series-data-solution/)**: Through built-in caching, stream processing and data subscription features, TDengine provides a simplified solution for time-series data processing. It reduces system design complexity and operation costs significantly.
|
||||
|
||||
- **Cloud Native**: Through native distributed design, sharding and partitioning, separation of compute and storage, RAFT, support for kubernetes deployment and full observability, TDengine is a cloud native Time-Series Database and can be deployed on public, private or hybrid clouds.
|
||||
- **[Cloud Native](https://tdengine.com/tdengine/cloud-native-time-series-database/)**: Through native distributed design, sharding and partitioning, separation of compute and storage, RAFT, support for kubernetes deployment and full observability, TDengine is a cloud native Time-Series Database and can be deployed on public, private or hybrid clouds.
|
||||
|
||||
- **Ease of Use**: For administrators, TDengine significantly reduces the effort to deploy and maintain. For developers, it provides a simple interface, simplified solution and seamless integrations for third party tools. For data users, it gives easy data access.
|
||||
- **[Ease of Use](https://tdengine.com/tdengine/easy-time-series-data-platform/)**: For administrators, TDengine significantly reduces the effort to[
|
||||
](https://tdengine.com/tdengine/easy-time-series-data-platform/) deploy and maintain. For developers, it provides a simple interface, simplified solution and seamless integrations for third party tools. For data users, it gives easy data access.
|
||||
|
||||
- **Easy Data Analytics**: Through super tables, storage and compute separation, data partitioning by time interval, pre-computation and other means, TDengine makes it easy to explore, format, and get access to data in a highly efficient way.
|
||||
- **[Easy Data Analytics](https://tdengine.com/tdengine/time-series-data-analytics-made-easy/)**: Through super tables, storage and compute separation, data partitioning by time interval, pre-computation and other means, TDengine makes it easy to explore, format, and get access to data in a highly efficient way.
|
||||
|
||||
- **Open Source**: TDengine’s core modules, including cluster feature, are all available under open source licenses. It has gathered 18.8k stars on GitHub. There is an active developer community, and over 139k running instances worldwide.
|
||||
- **[Open Source](https://tdengine.com/tdengine/open-source-time-series-database/)**: TDengine’s core modules, including cluster feature, are all available under open source licenses. It has gathered over 19k stars on GitHub. There is an active developer community, and over 140k running instances worldwide.
|
||||
|
||||
With TDengine, the total cost of ownership of your time-series data platform can be greatly reduced. 1: With its superior performance, the computing and storage resources are reduced significantly；2: With SQL support, it can be seamlessly integrated with many third party tools, and learning costs/migration costs are reduced significantly；3: With its simplified solution and nearly zero management, the operation and maintenance costs are reduced significantly.
|
||||
|
||||
|
|
@ -96,14 +109,13 @@ As a high-performance, scalable and SQL supported time-series database, TDengine
|
|||
| **System Maintenance Requirements** | **Not Applicable** | **Might Be Applicable** | **Very Applicable** | **Description** |
|
||||
| ------------------------------------------------- | ------------------ | ----------------------- | ------------------- | ------------------------------------------------------------ |
|
||||
| Native high-reliability | | | √ | TDengine has a very robust, reliable and easily configurable system architecture to simplify routine operation. Human errors and accidents are eliminated to the greatest extent, with a streamlined experience for operators. |
|
||||
| Minimize learning and maintenance costs | | | √ | In addition to being easily configurable, standard SQL support and the Taos shell for ad hoc queries makes maintenance simpler, allows reuse and reduces learning costs.|
|
||||
| Minimize learning and maintenance costs | | | √ | In addition to being easily configurable, standard SQL support and the TDengine CLI for ad hoc queries makes maintenance simpler, allows reuse and reduces learning costs.|
|
||||
| Abundant talent supply | √ | | | Given the above, and given the extensive training and professional services provided by TDengine, it is easy to migrate from existing solutions or create a new and lasting solution based on TDengine.|
|
||||
|
||||
## Comparison with other databases
|
||||
|
||||
- [Writing Performance Comparison of TDengine and InfluxDB ](https://tdengine.com/2022/02/23/4975.html)
|
||||
- [Query Performance Comparison of TDengine and InfluxDB](https://tdengine.com/2022/02/24/5120.html)
|
||||
- [TDengine vs InfluxDB、OpenTSDB、Cassandra、MySQL、ClickHouse](https://www.tdengine.com/downloads/TDengine_Testing_Report_en.pdf)
|
||||
- [TDengine vs OpenTSDB](https://tdengine.com/2019/09/12/710.html)
|
||||
- [TDengine vs Cassandra](https://tdengine.com/2019/09/12/708.html)
|
||||
- [TDengine vs InfluxDB](https://tdengine.com/2019/09/12/706.html)
|
||||
|
|
|
|||
|
|
@ -104,15 +104,15 @@ Each row contains the device ID, time stamp, collected metrics (current, voltage
|
|||
|
||||
## Metric
|
||||
|
||||
Metric refers to the physical quantity collected by sensors, equipment or other types of data collection devices, such as current, voltage, temperature, pressure, GPS position, etc., which change with time, and the data type can be integer, float, Boolean, or strings. As time goes by, the amount of collected metric data stored increases.
|
||||
Metric refers to the physical quantity collected by sensors, equipment or other types of data collection devices, such as current, voltage, temperature, pressure, GPS position, etc., which change with time, and the data type can be integer, float, Boolean, or strings. As time goes by, the amount of collected metric data stored increases. In the smart meters example, current, voltage and phase are the metrics.
|
||||
|
||||
## Label/Tag
|
||||
|
||||
Label/Tag refers to the static properties of sensors, equipment or other types of data collection devices, which do not change with time, such as device model, color, fixed location of the device, etc. The data type can be any type. Although static, TDengine allows users to add, delete or update tag values at any time. Unlike the collected metric data, the amount of tag data stored does not change over time.
|
||||
Label/Tag refers to the static properties of sensors, equipment or other types of data collection devices, which do not change with time, such as device model, color, fixed location of the device, etc. The data type can be any type. Although static, TDengine allows users to add, delete or update tag values at any time. Unlike the collected metric data, the amount of tag data stored does not change over time. In the meters example, `location` and `groupid` are the tags.
|
||||
|
||||
## Data Collection Point
|
||||
|
||||
Data Collection Point (DCP) refers to hardware or software that collects metrics based on preset time periods or triggered by events. A data collection point can collect one or multiple metrics, but these metrics are collected at the same time and have the same time stamp. For some complex equipment, there are often multiple data collection points, and the sampling rate of each collection point may be different, and fully independent. For example, for a car, there could be a data collection point to collect GPS position metrics, a data collection point to collect engine status metrics, and a data collection point to collect the environment metrics inside the car. So in this example the car would have three data collection points.
|
||||
Data Collection Point (DCP) refers to hardware or software that collects metrics based on preset time periods or triggered by events. A data collection point can collect one or multiple metrics, but these metrics are collected at the same time and have the same time stamp. For some complex equipment, there are often multiple data collection points, and the sampling rate of each collection point may be different, and fully independent. For example, for a car, there could be a data collection point to collect GPS position metrics, a data collection point to collect engine status metrics, and a data collection point to collect the environment metrics inside the car. So in this example the car would have three data collection points. In the smart meters example, d1001, d1002, d1003, and d1004 are the data collection points.
|
||||
|
||||
## Table
|
||||
|
||||
|
|
@ -137,7 +137,7 @@ The design of one table for one data collection point will require a huge number
|
|||
|
||||
STable is a template for a type of data collection point. A STable contains a set of data collection points (tables) that have the same schema or data structure, but with different static attributes (tags). To describe a STable, in addition to defining the table structure of the metrics, it is also necessary to define the schema of its tags. The data type of tags can be int, float, string, and there can be multiple tags, which can be added, deleted, or modified afterward. If the whole system has N different types of data collection points, N STables need to be established.
|
||||
|
||||
In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**.
|
||||
In the design of TDengine, **a table is used to represent a specific data collection point, and STable is used to represent a set of data collection points of the same type**. In the smart meters example, we can create a super table named `meters`.
|
||||
|
||||
## Subtable
|
||||
|
||||
|
|
@ -156,7 +156,9 @@ The relationship between a STable and the subtables created based on this STable
|
|||
|
||||
Queries can be executed on both a table (subtable) and a STable. For a query on a STable, TDengine will treat the data in all its subtables as a whole data set for processing. TDengine will first find the subtables that meet the tag filter conditions, then scan the time-series data of these subtables to perform aggregation operation, which reduces the number of data sets to be scanned which in turn greatly improves the performance of data aggregation across multiple DCPs. In essence, querying a supertable is a very efficient aggregate query on multiple DCPs of the same type.
|
||||
|
||||
In TDengine, it is recommended to use a subtable instead of a regular table for a DCP.
|
||||
In TDengine, it is recommended to use a subtable instead of a regular table for a DCP. In the smart meters example, we can create subtables like d1001, d1002, d1003, and d1004 under super table meters.
|
||||
|
||||
To better understand the data model using metri, tags, super table and subtable, please refer to the diagram below which demonstrates the data model of the smart meters example. 
|
||||
|
||||
## Database
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
|
|
@ -31,17 +31,6 @@ You can now access TDengine or run other Linux commands.
|
|||
|
||||
Note: For information about installing docker, see the [official documentation](https://docs.docker.com/get-docker/).
|
||||
|
||||
## Open the TDengine CLI
|
||||
|
||||
On the container, run the following command to open the TDengine CLI:
|
||||
|
||||
```
|
||||
$ taos
|
||||
|
||||
taos>
|
||||
|
||||
```
|
||||
|
||||
## Insert Data into TDengine
|
||||
|
||||
You can use the `taosBenchmark` tool included with TDengine to write test data into your deployment.
|
||||
|
|
@ -53,45 +42,57 @@ To do so, run the following command:
|
|||
|
||||
```
|
||||
|
||||
This command creates the `meters` supertable in the `test` database. In the `meters` supertable, it then creates 10,000 subtables named `d0` to `d9999`. Each table has 10,000 rows and each row has four columns: `ts`, `current`, `voltage`, and `phase`. The timestamps of the data in these columns range from 2017-07-14 10:40:00 000 to 2017-07-14 10:40:09 999. Each table is randomly assigned a `groupId` tag from 1 to ten and a `location` tag of either `California.SanFrancisco` or `California.SanDiego`.
|
||||
This command creates the `meters` supertable in the `test` database. In the `meters` supertable, it then creates 10,000 subtables named `d0` to `d9999`. Each table has 10,000 rows and each row has four columns: `ts`, `current`, `voltage`, and `phase`. The timestamps of the data in these columns range from 2017-07-14 10:40:00 000 to 2017-07-14 10:40:09 999. Each table is randomly assigned a `groupId` tag from 1 to 10 and a `location` tag of either `Campbell`, `Cupertino`, `Los Angeles`, `Mountain View`, `Palo Alto`, `San Diego`, `San Francisco`, `San Jose`, `Santa Clara` or `Sunnyvale`.
|
||||
|
||||
The `taosBenchmark` command creates a deployment with 100 million data points that you can use for testing purposes. The time required depends on the hardware specifications of the local system.
|
||||
|
||||
You can customize the test deployment that taosBenchmark creates by specifying command-line parameters. For information about command-line parameters, run the `taosBenchmark --help` command. For more information about taosBenchmark, see [taosBenchmark](/reference/taosbenchmark).
|
||||
|
||||
## Open the TDengine CLI
|
||||
|
||||
On the container, run the following command to open the TDengine CLI:
|
||||
|
||||
```
|
||||
$ taos
|
||||
|
||||
taos>
|
||||
|
||||
```
|
||||
|
||||
## Query Data in TDengine
|
||||
|
||||
After using taosBenchmark to create your test deployment, you can run queries in the TDengine CLI to test its performance. For example:
|
||||
|
||||
Query the number of rows in the `meters` supertable:
|
||||
From the TDengine CLI query the number of rows in the `meters` supertable:
|
||||
|
||||
```sql
|
||||
taos> select count(*) from test.meters;
|
||||
select count(*) from test.meters;
|
||||
```
|
||||
|
||||
Query the average, maximum, and minimum values of all 100 million rows of data:
|
||||
|
||||
```sql
|
||||
taos> select avg(current), max(voltage), min(phase) from test.meters;
|
||||
select avg(current), max(voltage), min(phase) from test.meters;
|
||||
```
|
||||
|
||||
Query the number of rows whose `location` tag is `California.SanFrancisco`:
|
||||
Query the number of rows whose `location` tag is `San Francisco`:
|
||||
|
||||
```sql
|
||||
taos> select count(*) from test.meters where location="San Francisco";
|
||||
select count(*) from test.meters where location="San Francisco";
|
||||
```
|
||||
|
||||
Query the average, maximum, and minimum values of all rows whose `groupId` tag is `10`:
|
||||
|
||||
```sql
|
||||
taos> select avg(current), max(voltage), min(phase) from test.meters where groupId=10;
|
||||
select avg(current), max(voltage), min(phase) from test.meters where groupId=10;
|
||||
```
|
||||
|
||||
Query the average, maximum, and minimum values for table `d10` in 10 second intervals:
|
||||
Query the average, maximum, and minimum values for table `d10` in 1 second intervals:
|
||||
|
||||
```sql
|
||||
taos> select avg(current), max(voltage), min(phase) from test.d10 interval(10s);
|
||||
select first(ts), avg(current), max(voltage), min(phase) from test.d10 interval(1s);
|
||||
```
|
||||
In the query above you are selecting the first timestamp (ts) in the interval, another way of selecting this would be _wstart which will give the start of the time window. For more information about windowed queries, see [Time-Series Extensions](../../taos-sql/distinguished/).
|
||||
|
||||
## Additional Information
|
||||
|
||||
|
|
|
|||
|
|
@ -67,13 +67,6 @@ Users will be prompted to enter some configuration information when install.sh i
|
|||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Windows" value="windows">
|
||||
|
||||
1. Download the Windows installation package.
|
||||
<PkgListV3 type={3}/>
|
||||
2. Run the downloaded package to install TDengine.
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="apt-get" label="apt-get">
|
||||
You can use `apt-get` to install TDengine from the official package repository.
|
||||
|
||||
|
|
@ -102,6 +95,15 @@ sudo apt-get install tdengine
|
|||
:::tip
|
||||
This installation method is supported only for Debian and Ubuntu.
|
||||
::::
|
||||
</TabItem>
|
||||
<TabItem label="Windows" value="windows">
|
||||
|
||||
Note: TDengine only supports Windows Server 2016/2019 and windows 10/11 system versions on the windows platform.
|
||||
|
||||
1. Download the Windows installation package.
|
||||
<PkgListV3 type={3}/>
|
||||
2. Run the downloaded package to install TDengine.
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
|
@ -172,6 +174,20 @@ After the installation is complete, run `C:\TDengine\taosd.exe` to start TDengin
|
|||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
## Test data insert performance
|
||||
|
||||
After your TDengine Server is running normally, you can run the taosBenchmark utility to test its performance:
|
||||
|
||||
```bash
|
||||
taosBenchmark
|
||||
```
|
||||
|
||||
This command creates the `meters` supertable in the `test` database. In the `meters` supertable, it then creates 10,000 subtables named `d0` to `d9999`. Each table has 10,000 rows and each row has four columns: `ts`, `current`, `voltage`, and `phase`. The timestamps of the data in these columns range from 2017-07-14 10:40:00 000 to 2017-07-14 10:40:09 999. Each table is randomly assigned a `groupId` tag from 1 to 10 and a `location` tag of either `Campbell`, `Cupertino`, `Los Angeles`, `Mountain View`, `Palo Alto`, `San Diego`, `San Francisco`, `San Jose`, `Santa Clara` or `Sunnyvale`.
|
||||
|
||||
The `taosBenchmark` command creates a deployment with 100 million data points that you can use for testing purposes. The time required to create the deployment depends on your hardware. On most modern servers, the deployment is created in less than a minute.
|
||||
|
||||
You can customize the test deployment that taosBenchmark creates by specifying command-line parameters. For information about command-line parameters, run the `taosBenchmark --help` command. For more information about taosBenchmark, see [taosBenchmark](../../reference/taosbenchmark).
|
||||
|
||||
## Command Line Interface
|
||||
|
||||
You can use the TDengine CLI to monitor your TDengine deployment and execute ad hoc queries. To open the CLI, run the following command:
|
||||
|
|
@ -203,51 +219,38 @@ Query OK, 2 row(s) in set (0.003128s)
|
|||
```
|
||||
|
||||
You can also can monitor the deployment status, add and remove user accounts, and manage running instances. You can run the TDengine CLI on either Linux or Windows machines. For more information, see [TDengine CLI](../../reference/taos-shell/).
|
||||
|
||||
## Test data insert performance
|
||||
|
||||
After your TDengine Server is running normally, you can run the taosBenchmark utility to test its performance:
|
||||
|
||||
```bash
|
||||
taosBenchmark
|
||||
```
|
||||
|
||||
This command creates the `meters` supertable in the `test` database. In the `meters` supertable, it then creates 10,000 subtables named `d0` to `d9999`. Each table has 10,000 rows and each row has four columns: `ts`, `current`, `voltage`, and `phase`. The timestamps of the data in these columns range from 2017-07-14 10:40:00 000 to 2017-07-14 10:40:09 999. Each table is randomly assigned a `groupId` tag from 1 to ten and a `location` tag of either `California.SanFrancisco` or `California.LosAngeles`.
|
||||
|
||||
The `taosBenchmark` command creates a deployment with 100 million data points that you can use for testing purposes. The time required to create the deployment depends on your hardware. On most modern servers, the deployment is created in less than a minute.
|
||||
|
||||
You can customize the test deployment that taosBenchmark creates by specifying command-line parameters. For information about command-line parameters, run the `taosBenchmark --help` command. For more information about taosBenchmark, see [taosBenchmark](../../reference/taosbenchmark).
|
||||
|
||||
|
||||
## Test data query performance
|
||||
|
||||
After using taosBenchmark to create your test deployment, you can run queries in the TDengine CLI to test its performance:
|
||||
|
||||
Query the number of rows in the `meters` supertable:
|
||||
From the TDengine CLI query the number of rows in the `meters` supertable:
|
||||
|
||||
```sql
|
||||
taos> select count(*) from test.meters;
|
||||
select count(*) from test.meters;
|
||||
```
|
||||
|
||||
Query the average, maximum, and minimum values of all 100 million rows of data:
|
||||
|
||||
```sql
|
||||
taos> select avg(current), max(voltage), min(phase) from test.meters;
|
||||
select avg(current), max(voltage), min(phase) from test.meters;
|
||||
```
|
||||
|
||||
Query the number of rows whose `location` tag is `California.SanFrancisco`:
|
||||
Query the number of rows whose `location` tag is `San Francisco`:
|
||||
|
||||
```sql
|
||||
taos> select count(*) from test.meters where location="California.SanFrancisco";
|
||||
select count(*) from test.meters where location="San Francisco";
|
||||
```
|
||||
|
||||
Query the average, maximum, and minimum values of all rows whose `groupId` tag is `10`:
|
||||
|
||||
```sql
|
||||
taos> select avg(current), max(voltage), min(phase) from test.meters where groupId=10;
|
||||
select avg(current), max(voltage), min(phase) from test.meters where groupId=10;
|
||||
```
|
||||
|
||||
Query the average, maximum, and minimum values for table `d10` in 10 second intervals:
|
||||
Query the average, maximum, and minimum values for table `d10` in 1 second intervals:
|
||||
|
||||
```sql
|
||||
taos> select avg(current), max(voltage), min(phase) from test.d10 interval(10s);
|
||||
select first(ts), avg(current), max(voltage), min(phase) from test.d10 interval(1s);
|
||||
```
|
||||
In the query above you are selecting the first timestamp (ts) in the interval, another way of selecting this would be _wstart which will give the start of the time window. For more information about windowed queries, see [Time-Series Extensions](../../taos-sql/distinguished/).
|
||||
|
|
|
|||
|
|
@ -0,0 +1,444 @@
|
|||
---
|
||||
sidebar_label: High Performance Writing
|
||||
title: High Performance Writing
|
||||
---
|
||||
|
||||
import Tabs from "@theme/Tabs";
|
||||
import TabItem from "@theme/TabItem";
|
||||
|
||||
This chapter introduces how to write data into TDengine with high throughput.
|
||||
|
||||
## How to achieve high performance data writing
|
||||
|
||||
To achieve high performance writing, there are a few aspects to consider. In the following sections we will describe these important factors in achieving high performance writing.
|
||||
|
||||
### Application Program
|
||||
|
||||
From the perspective of application program, you need to consider:
|
||||
|
||||
1. The data size of each single write, also known as batch size. Generally speaking, higher batch size generates better writing performance. However, once the batch size is over a specific value, you will not get any additional benefit anymore. When using SQL to write into TDengine, it's better to put as much as possible data in single SQL. The maximum SQL length supported by TDengine is 1,048,576 bytes, i.e. 1 MB. It can be configured by parameter `maxSQLLength` on client side, and the default value is 65,480.
|
||||
|
||||
2. The number of concurrent connections. Normally more connections can get better result. However, once the number of connections exceeds the processing ability of the server side, the performance may downgrade.
|
||||
|
||||
3. The distribution of data to be written across tables or sub-tables. Writing to single table in one batch is more efficient than writing to multiple tables in one batch.
|
||||
|
||||
4. Data Writing Protocol.
|
||||
- Prameter binding mode is more efficient than SQL because it doesn't have the cost of parsing SQL.
|
||||
- Writing to known existing tables is more efficient than wirting to uncertain tables in automatic creating mode because the later needs to check whether the table exists or not before actually writing data into it
|
||||
- Writing in SQL is more efficient than writing in schemaless mode because schemaless writing creats table automatically and may alter table schema
|
||||
|
||||
Application programs need to take care of the above factors and try to take advantage of them. The application progam should write to single table in each write batch. The batch size needs to be tuned to a proper value on a specific system. The number of concurrent connections needs to be tuned to a proper value too to achieve the best writing throughput.
|
||||
|
||||
### Data Source
|
||||
|
||||
Application programs need to read data from data source then write into TDengine. If you meet one or more of below situations, you need to setup message queues between the threads for reading from data source and the threads for writing into TDengine.
|
||||
|
||||
1. There are multiple data sources, the data generation speed of each data source is much slower than the speed of single writing thread. In this case, the purpose of message queues is to consolidate the data from multiple data sources together to increase the batch size of single write.
|
||||
2. The speed of data generation from single data source is much higher than the speed of single writing thread. The purpose of message queue in this case is to provide buffer so that data is not lost and multiple writing threads can get data from the buffer.
|
||||
3. The data for single table are from multiple data source. In this case the purpose of message queues is to combine the data for single table together to improve the write efficiency.
|
||||
|
||||
If the data source is Kafka, then the appication program is a consumer of Kafka, you can benefit from some kafka features to achieve high performance writing:
|
||||
|
||||
1. Put the data for a table in single partition of single topic so that it's easier to put the data for each table together and write in batch
|
||||
2. Subscribe multiple topics to accumulate data together.
|
||||
3. Add more consumers to gain more concurrency and throughput.
|
||||
4. Incrase the size of single fetch to increase the size of write batch.
|
||||
|
||||
### Tune TDengine
|
||||
|
||||
TDengine is a distributed and high performance time series database, there are also some ways to tune TDengine to get better writing performance.
|
||||
|
||||
1. Set proper number of `vgroups` according to available CPU cores. Normally, we recommend 2 \* number_of_cores as a starting point. If the verification result shows this is not enough to utilize CPU resources, you can use a higher value.
|
||||
2. Set proper `minTablesPerVnode`, `tableIncStepPerVnode`, and `maxVgroupsPerDb` according to the number of tables so that tables are distributed even across vgroups. The purpose is to balance the workload among all vnodes so that system resources can be utilized better to get higher performance.
|
||||
|
||||
For more performance tuning parameters, please refer to [Configuration Parameters](../../../reference/config).
|
||||
|
||||
## Sample Programs
|
||||
|
||||
This section will introduce the sample programs to demonstrate how to write into TDengine with high performance.
|
||||
|
||||
### Scenario
|
||||
|
||||
Below are the scenario for the sample programs of high performance wrting.
|
||||
|
||||
- Application program reads data from data source, the sample program simulates a data source by generating data
|
||||
- The speed of single writing thread is much slower than the speed of generating data, so the program starts multiple writing threads while each thread establish a connection to TDengine and each thread has a message queue of fixed size.
|
||||
- Application program maps the received data to different writing threads based on table name to make sure all the data for each table is always processed by a specific writing thread.
|
||||
- Each writing thread writes the received data into TDengine once the message queue becomes empty or the read data meets a threshold.
|
||||
|
||||
|
||||
|
||||
### Sample Programs
|
||||
|
||||
The sample programs listed in this section are based on the scenario described previously. If your scenarios is different, please try to adjust the code based on the principles described in this chapter.
|
||||
|
||||
The sample programs assume the source data is for all the different sub tables in same super table (meters). The super table has been created before the sample program starts to writing data. Sub tables are created automatically according to received data. If there are multiple super tables in your case, please try to adjust the part of creating table automatically.
|
||||
|
||||
<Tabs defaultValue="java" groupId="lang">
|
||||
<TabItem label="Java" value="java">
|
||||
|
||||
**Program Inventory**
|
||||
|
||||
| Class | Description |
|
||||
| ---------------- | ----------------------------------------------------------------------------------------------------- |
|
||||
| FastWriteExample | Main Program |
|
||||
| ReadTask | Read data from simulated data source and put into a queue according to the hash value of table name |
|
||||
| WriteTask | Read data from Queue, compose a wirte batch and write into TDengine |
|
||||
| MockDataSource | Generate data for some sub tables of super table meters |
|
||||
| SQLWriter | WriteTask uses this class to compose SQL, create table automatically, check SQL length and write data |
|
||||
| StmtWriter | Write in Parameter binding mode (Not finished yet) |
|
||||
| DataBaseMonitor | Calculate the writing speed and output on console every 10 seconds |
|
||||
|
||||
Below is the list of complete code of the classes in above table and more detailed description.
|
||||
|
||||
<details>
|
||||
<summary>FastWriteExample</summary>
|
||||
The main Program is responsible for:
|
||||
|
||||
1. Create message queues
|
||||
2. Start writing threads
|
||||
3. Start reading threads
|
||||
4. Otuput writing speed every 10 seconds
|
||||
|
||||
The main program provides 4 parameters for tuning：
|
||||
|
||||
1. The number of reading threads, default value is 1
|
||||
2. The number of writing threads, default alue is 2
|
||||
3. The total number of tables in the generated data, default value is 1000. These tables are distributed evenly across all writing threads. If the number of tables is very big, it will cost much time to firstly create these tables.
|
||||
4. The batch size of single write, default value is 3,000
|
||||
|
||||
The capacity of message queue also impacts performance and can be tuned by modifying program. Normally it's always better to have a larger message queue. A larger message queue means lower possibility of being blocked when enqueueing and higher throughput. But a larger message queue consumes more memory space. The default value used in the sample programs is already big enoug.
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/FastWriteExample.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>ReadTask</summary>
|
||||
|
||||
ReadTask reads data from data source. Each ReadTask is associated with a simulated data source, each data source generates data for a group of specific tables, and the data of any table is only generated from a single specific data source.
|
||||
|
||||
ReadTask puts data in message queue in blocking mode. That means, the putting operation is blocked if the message queue is full.
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/ReadTask.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>WriteTask</summary>
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/WriteTask.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>MockDataSource</summary>
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/MockDataSource.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>SQLWriter</summary>
|
||||
|
||||
SQLWriter class encapsulates the logic of composing SQL and writing data. Please be noted that the tables have not been created before writing, but are created automatically when catching the exception of table doesn't exist. For other exceptions caught, the SQL which caused the exception are logged for you to debug.
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/SQLWriter.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>DataBaseMonitor</summary>
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/DataBaseMonitor.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
**Steps to Launch**
|
||||
|
||||
<details>
|
||||
<summary>Launch Java Sample Program</summary>
|
||||
|
||||
You need to set environment variable `TDENGINE_JDBC_URL` before launching the program. If TDengine Server is setup on localhost, then the default value for user name, password and port can be used, like below:
|
||||
|
||||
```
|
||||
TDENGINE_JDBC_URL="jdbc:TAOS://localhost:6030?user=root&password=taosdata"
|
||||
```
|
||||
|
||||
**Launch in IDE**
|
||||
|
||||
1. Clone TDengine repolitory
|
||||
```
|
||||
git clone git@github.com:taosdata/TDengine.git --depth 1
|
||||
```
|
||||
2. Use IDE to open `docs/examples/java` directory
|
||||
3. Configure environment variable `TDENGINE_JDBC_URL`, you can also configure it before launching the IDE, if so you can skip this step.
|
||||
4. Run class `com.taos.example.highvolume.FastWriteExample`
|
||||
|
||||
**Launch on server**
|
||||
|
||||
If you want to launch the sample program on a remote server, please follow below steps:
|
||||
|
||||
1. Package the sample programs. Execute below command under directory `TDengine/docs/examples/java` ：
|
||||
```
|
||||
mvn package
|
||||
```
|
||||
2. Create `examples/java` directory on the server
|
||||
```
|
||||
mkdir -p examples/java
|
||||
```
|
||||
3. Copy dependencies (below commands assume you are working on a local Windows host and try to launch on a remote Linux host)
|
||||
- Copy dependent packages
|
||||
```
|
||||
scp -r .\target\lib <user>@<host>:~/examples/java
|
||||
```
|
||||
- Copy the jar of sample programs
|
||||
```
|
||||
scp -r .\target\javaexample-1.0.jar <user>@<host>:~/examples/java
|
||||
```
|
||||
4. Configure environment variable
|
||||
Edit `~/.bash_profile` or `~/.bashrc` and add below:
|
||||
|
||||
```
|
||||
export TDENGINE_JDBC_URL="jdbc:TAOS://localhost:6030?user=root&password=taosdata"
|
||||
```
|
||||
|
||||
If your TDengine server is not deployed on localhost or doesn't use default port, you need to change the above URL to correct value in your environment.
|
||||
|
||||
5. Launch the sample program
|
||||
|
||||
```
|
||||
java -classpath lib/*:javaexample-1.0.jar com.taos.example.highvolume.FastWriteExample <read_thread_count> <white_thread_count> <total_table_count> <max_batch_size>
|
||||
```
|
||||
|
||||
6. The sample program doesn't exit unless you press <kbd>CTRL</kbd> + <kbd>C</kbd> to terminate it.
|
||||
Below is the output of running on a server of 16 cores, 64GB memory and SSD hard disk.
|
||||
|
||||
```
|
||||
root@vm85$ java -classpath lib/*:javaexample-1.0.jar com.taos.example.highvolume.FastWriteExample 2 12
|
||||
18:56:35.896 [main] INFO c.t.e.highvolume.FastWriteExample - readTaskCount=2, writeTaskCount=12 tableCount=1000 maxBatchSize=3000
|
||||
18:56:36.011 [WriteThread-0] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.015 [WriteThread-0] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.021 [WriteThread-1] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.022 [WriteThread-1] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.031 [WriteThread-2] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.032 [WriteThread-2] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.041 [WriteThread-3] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.042 [WriteThread-3] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.093 [WriteThread-4] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.094 [WriteThread-4] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.099 [WriteThread-5] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.100 [WriteThread-5] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.100 [WriteThread-6] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.101 [WriteThread-6] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.103 [WriteThread-7] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.104 [WriteThread-7] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.105 [WriteThread-8] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.107 [WriteThread-8] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.108 [WriteThread-9] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.109 [WriteThread-9] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.156 [WriteThread-10] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.157 [WriteThread-11] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.158 [WriteThread-10] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.158 [ReadThread-0] INFO com.taos.example.highvolume.ReadTask - started
|
||||
18:56:36.158 [ReadThread-1] INFO com.taos.example.highvolume.ReadTask - started
|
||||
18:56:36.158 [WriteThread-11] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:46.369 [main] INFO c.t.e.highvolume.FastWriteExample - count=18554448 speed=1855444
|
||||
18:56:56.946 [main] INFO c.t.e.highvolume.FastWriteExample - count=39059660 speed=2050521
|
||||
18:57:07.322 [main] INFO c.t.e.highvolume.FastWriteExample - count=59403604 speed=2034394
|
||||
18:57:18.032 [main] INFO c.t.e.highvolume.FastWriteExample - count=80262938 speed=2085933
|
||||
18:57:28.432 [main] INFO c.t.e.highvolume.FastWriteExample - count=101139906 speed=2087696
|
||||
18:57:38.921 [main] INFO c.t.e.highvolume.FastWriteExample - count=121807202 speed=2066729
|
||||
18:57:49.375 [main] INFO c.t.e.highvolume.FastWriteExample - count=142952417 speed=2114521
|
||||
18:58:00.689 [main] INFO c.t.e.highvolume.FastWriteExample - count=163650306 speed=2069788
|
||||
18:58:11.646 [main] INFO c.t.e.highvolume.FastWriteExample - count=185019808 speed=2136950
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
</TabItem>
|
||||
<TabItem label="Python" value="python">
|
||||
|
||||
**Program Inventory**
|
||||
|
||||
Sample programs in Python uses multi-process and cross-process message queues.
|
||||
|
||||
| Function/CLass | Description |
|
||||
| ---------------------------- | --------------------------------------------------------------------------- |
|
||||
| main Function | Program entry point, create child processes and message queues |
|
||||
| run_monitor_process Function | Create database, super table, calculate writing speed and output to console |
|
||||
| run_read_task Function | Read data and distribute to message queues |
|
||||
| MockDataSource Class | Simulate data source, return next 1,000 rows of each table |
|
||||
| run_write_task Function | Read as much as possible data from message queue and write in batch |
|
||||
| SQLWriter Class | Write in SQL and create table utomatically |
|
||||
| StmtWriter Class | Write in parameter binding mode (not finished yet) |
|
||||
|
||||
<details>
|
||||
<summary>main function</summary>
|
||||
|
||||
`main` function is responsible for creating message queues and fork child processes, there are 3 kinds of child processes:
|
||||
|
||||
1. Monitoring process, initializes database and calculating writing speed
|
||||
2. Reading process (n), reads data from data source
|
||||
3. Writing process (m), wirtes data into TDengine
|
||||
|
||||
`main` function provides 5 parameters:
|
||||
|
||||
1. The number of reading tasks, default value is 1
|
||||
2. The number of writing tasks, default value is 1
|
||||
3. The number of tables, default value is 1,000
|
||||
4. The capacity of message queue, default value is 1,000,000 bytes
|
||||
5. The batch size in single write, default value is 3000
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:main}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>run_monitor_process</summary>
|
||||
|
||||
Monitoring process initilizes database and monitoring writing speed.
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:monitor}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>run_read_task function</summary>
|
||||
|
||||
Reading process reads data from other data system and distributes to the message queue allocated for it.
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:read}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>MockDataSource</summary>
|
||||
|
||||
Below is the simulated data source, we assume table name exists in each generated data.
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/mockdatasource.py}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>run_write_task function</summary>
|
||||
|
||||
Writing process tries to read as much as possible data from message queue and writes in batch.
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:write}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
SQLWriter class encapsulates the logic of composing SQL and writing data. Please be noted that the tables have not been created before writing, but are created automatically when catching the exception of table doesn't exist. For other exceptions caught, the SQL which caused the exception are logged for you to debug. This class also checks the SQL length, if the SQL length is closed to `maxSQLLength` the SQL will be executed immediately. To improve writing efficiency, it's better to increase `maxSQLLength` properly.
|
||||
|
||||
<summary>SQLWriter</summary>
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/sql_writer.py}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
**Steps to Launch**
|
||||
|
||||
<details>
|
||||
|
||||
<summary>Launch Sample Program in Python</summary>
|
||||
|
||||
1. Prerequisities
|
||||
|
||||
- TDengine client driver has been installed
|
||||
- Python3 has been installed, the the version >= 3.8
|
||||
- TDengine Python connector `taospy` has been installed
|
||||
|
||||
2. Install faster-fifo to replace python builtin multiprocessing.Queue
|
||||
|
||||
```
|
||||
pip3 install faster-fifo
|
||||
```
|
||||
|
||||
3. Click the "Copy" in the above sample programs to copy `fast_write_example.py` 、 `sql_writer.py` and `mockdatasource.py`.
|
||||
|
||||
4. Execute the program
|
||||
|
||||
```
|
||||
python3 fast_write_example.py <READ_TASK_COUNT> <WRITE_TASK_COUNT> <TABLE_COUNT> <QUEUE_SIZE> <MAX_BATCH_SIZE>
|
||||
```
|
||||
|
||||
Below is the output of running on a server of 16 cores, 64GB memory and SSD hard disk.
|
||||
|
||||
```
|
||||
root@vm85$ python3 fast_write_example.py 8 8
|
||||
2022-07-14 19:13:45,869 [root] - READ_TASK_COUNT=8, WRITE_TASK_COUNT=8, TABLE_COUNT=1000, QUEUE_SIZE=1000000, MAX_BATCH_SIZE=3000
|
||||
2022-07-14 19:13:48,882 [root] - WriteTask-0 started with pid 718347
|
||||
2022-07-14 19:13:48,883 [root] - WriteTask-1 started with pid 718348
|
||||
2022-07-14 19:13:48,884 [root] - WriteTask-2 started with pid 718349
|
||||
2022-07-14 19:13:48,884 [root] - WriteTask-3 started with pid 718350
|
||||
2022-07-14 19:13:48,885 [root] - WriteTask-4 started with pid 718351
|
||||
2022-07-14 19:13:48,885 [root] - WriteTask-5 started with pid 718352
|
||||
2022-07-14 19:13:48,886 [root] - WriteTask-6 started with pid 718353
|
||||
2022-07-14 19:13:48,886 [root] - WriteTask-7 started with pid 718354
|
||||
2022-07-14 19:13:48,887 [root] - ReadTask-0 started with pid 718355
|
||||
2022-07-14 19:13:48,888 [root] - ReadTask-1 started with pid 718356
|
||||
2022-07-14 19:13:48,889 [root] - ReadTask-2 started with pid 718357
|
||||
2022-07-14 19:13:48,889 [root] - ReadTask-3 started with pid 718358
|
||||
2022-07-14 19:13:48,890 [root] - ReadTask-4 started with pid 718359
|
||||
2022-07-14 19:13:48,891 [root] - ReadTask-5 started with pid 718361
|
||||
2022-07-14 19:13:48,892 [root] - ReadTask-6 started with pid 718364
|
||||
2022-07-14 19:13:48,893 [root] - ReadTask-7 started with pid 718365
|
||||
2022-07-14 19:13:56,042 [DataBaseMonitor] - count=6676310 speed=667631.0
|
||||
2022-07-14 19:14:06,196 [DataBaseMonitor] - count=20004310 speed=1332800.0
|
||||
2022-07-14 19:14:16,366 [DataBaseMonitor] - count=32290310 speed=1228600.0
|
||||
2022-07-14 19:14:26,527 [DataBaseMonitor] - count=44438310 speed=1214800.0
|
||||
2022-07-14 19:14:36,673 [DataBaseMonitor] - count=56608310 speed=1217000.0
|
||||
2022-07-14 19:14:46,834 [DataBaseMonitor] - count=68757310 speed=1214900.0
|
||||
2022-07-14 19:14:57,280 [DataBaseMonitor] - count=80992310 speed=1223500.0
|
||||
2022-07-14 19:15:07,689 [DataBaseMonitor] - count=93805310 speed=1281300.0
|
||||
2022-07-14 19:15:18,020 [DataBaseMonitor] - count=106111310 speed=1230600.0
|
||||
2022-07-14 19:15:28,356 [DataBaseMonitor] - count=118394310 speed=1228300.0
|
||||
2022-07-14 19:15:38,690 [DataBaseMonitor] - count=130742310 speed=1234800.0
|
||||
2022-07-14 19:15:49,000 [DataBaseMonitor] - count=143051310 speed=1230900.0
|
||||
2022-07-14 19:15:59,323 [DataBaseMonitor] - count=155276310 speed=1222500.0
|
||||
2022-07-14 19:16:09,649 [DataBaseMonitor] - count=167603310 speed=1232700.0
|
||||
2022-07-14 19:16:19,995 [DataBaseMonitor] - count=179976310 speed=1237300.0
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
:::note
|
||||
Don't establish connection to TDengine in the parent process if using Python connector in multi-process way, otherwise all the connections in child processes are blocked always. This is a known issue.
|
||||
|
||||
:::
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 7.1 KiB |
|
|
@ -16,7 +16,7 @@ import CDemo from "./_sub_c.mdx";
|
|||
|
||||
TDengine provides data subscription and consumption interfaces similar to message queue products. These interfaces make it easier for applications to obtain data written to TDengine either in real time and to process data in the order that events occurred. This simplifies your time-series data processing systems and reduces your costs because it is no longer necessary to deploy a message queue product such as Kafka.
|
||||
|
||||
To use TDengine data subscription, you define topics like in Kafka. However, a topic in TDengine is based on query conditions for an existing supertable, standard table, or subtable - in other words, a SELECT statement. You can use SQL to filter data by tag, table name, column, or expression and then perform a scalar function or user-defined function on the data. Aggregate functions are not supported. This gives TDengine data subscription more flexibility than similar products. The granularity of data can be controlled on demand by applications, while filtering and preprocessing are handled by TDengine instead of the application layer. This implementation reduces the amount of data transmitted and the complexity of applications.
|
||||
To use TDengine data subscription, you define topics like in Kafka. However, a topic in TDengine is based on query conditions for an existing supertable, table, or subtable - in other words, a SELECT statement. You can use SQL to filter data by tag, table name, column, or expression and then perform a scalar function or user-defined function on the data. Aggregate functions are not supported. This gives TDengine data subscription more flexibility than similar products. The granularity of data can be controlled on demand by applications, while filtering and preprocessing are handled by TDengine instead of the application layer. This implementation reduces the amount of data transmitted and the complexity of applications.
|
||||
|
||||
By subscribing to a topic, a consumer can obtain the latest data in that topic in real time. Multiple consumers can be formed into a consumer group that consumes messages together. Consumer groups enable faster speed through multi-threaded, distributed data consumption. Note that consumers in different groups that are subscribed to the same topic do not consume messages together. A single consumer can subscribe to multiple topics. If the data in a supertable is sharded across multiple vnodes, consumer groups can consume it much more efficiently than single consumers. TDengine also includes an acknowledgement mechanism that ensures at-least-once delivery in complicated environments where machines may crash or restart.
|
||||
|
||||
|
|
|
|||
|
|
@ -20,11 +20,11 @@ In theory, larger cache sizes are always better. However, at a certain point, it
|
|||
|
||||
## Read Cache
|
||||
|
||||
When you create a database, you can configure whether the latest data from every subtable is cached. To do so, set the *cachelast* parameter as follows:
|
||||
- 0: Caching is disabled.
|
||||
- 1: The latest row of data in each subtable is cached. This option significantly improves the performance of the `LAST_ROW` function
|
||||
- 2: The latest non-null value in each column of each subtable is cached. This option significantly improves the performance of the `LAST` function in normal situations, such as WHERE, ORDER BY, GROUP BY, and INTERVAL statements.
|
||||
- 3: Rows and columns are both cached. This option is equivalent to simultaneously enabling options 1 and 2.
|
||||
When you create a database, you can configure whether the latest data from every subtable is cached. To do so, set the *cachemodel* parameter as follows:
|
||||
- none: Caching is disabled.
|
||||
- last_row: The latest row of data in each subtable is cached. This option significantly improves the performance of the `LAST_ROW` function
|
||||
- last_value: The latest non-null value in each column of each subtable is cached. This option significantly improves the performance of the `LAST` function in normal situations, such as WHERE, ORDER BY, GROUP BY, and INTERVAL statements.
|
||||
- both: Rows and columns are both cached. This option is equivalent to simultaneously enabling option last_row and last_value.
|
||||
|
||||
## Metadata Cache
|
||||
|
||||
|
|
|
|||
|
|
@ -114,7 +114,9 @@ The above process can be repeated to add more dnodes in the cluster.
|
|||
|
||||
Any node that is in the cluster and online can be the firstEp of new nodes.
|
||||
Nodes use the firstEp parameter only when joining a cluster for the first time. After a node has joined the cluster, it stores the latest mnode in its end point list and no longer makes use of firstEp.
|
||||
However, firstEp is used by clients that connect to the cluster. For example, if you run `taos shell` without arguments, it connects to the firstEp by default.
|
||||
|
||||
However, firstEp is used by clients that connect to the cluster. For example, if you run TDengine CLI `taos` without arguments, it connects to the firstEp by default.
|
||||
|
||||
Two dnodes that are launched without a firstEp value operate independently of each other. It is not possible to add one dnode to the other dnode and form a cluster. It is also not possible to form two independent clusters into a new cluster.
|
||||
|
||||
:::
|
||||
|
|
|
|||
|
|
@ -9,6 +9,7 @@ TDengine is a cloud-native time-series database that can be deployed on Kubernet
|
|||
|
||||
Before deploying TDengine on Kubernetes, perform the following:
|
||||
|
||||
* Current steps are compatible with Kubernetes v1.5 and later version.
|
||||
* Install and configure minikube, kubectl, and helm.
|
||||
* Install and deploy Kubernetes and ensure that it can be accessed and used normally. Update any container registries or other services as necessary.
|
||||
|
||||
|
|
@ -100,7 +101,7 @@ spec:
|
|||
# Must set if you want a cluster.
|
||||
- name: TAOS_FIRST_EP
|
||||
value: "$(STS_NAME)-0.$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local:$(TAOS_SERVER_PORT)"
|
||||
# TAOS_FQND should always be setted in k8s env.
|
||||
# TAOS_FQDN should always be set in k8s env.
|
||||
- name: TAOS_FQDN
|
||||
value: "$(POD_NAME).$(SERVICE_NAME).$(STS_NAMESPACE).svc.cluster.local"
|
||||
volumeMounts:
|
||||
|
|
|
|||
|
|
@ -170,71 +170,21 @@ taoscfg:
|
|||
# number of replications, for cluster only
|
||||
TAOS_REPLICA: "1"
|
||||
|
||||
|
||||
# number of days per DB file
|
||||
# TAOS_DAYS: "10"
|
||||
|
||||
# number of days to keep DB file, default is 10 years.
|
||||
#TAOS_KEEP: "3650"
|
||||
|
||||
# cache block size (Mbyte)
|
||||
#TAOS_CACHE: "16"
|
||||
|
||||
# number of cache blocks per vnode
|
||||
#TAOS_BLOCKS: "6"
|
||||
|
||||
# minimum rows of records in file block
|
||||
#TAOS_MIN_ROWS: "100"
|
||||
|
||||
# maximum rows of records in file block
|
||||
#TAOS_MAX_ROWS: "4096"
|
||||
|
||||
#
|
||||
# TAOS_NUM_OF_THREADS_PER_CORE: number of threads per CPU core
|
||||
#TAOS_NUM_OF_THREADS_PER_CORE: "1.0"
|
||||
# TAOS_NUM_OF_RPC_THREADS: number of threads for RPC
|
||||
#TAOS_NUM_OF_RPC_THREADS: "2"
|
||||
|
||||
|
||||
#
|
||||
# TAOS_NUM_OF_COMMIT_THREADS: number of threads to commit cache data
|
||||
#TAOS_NUM_OF_COMMIT_THREADS: "4"
|
||||
|
||||
#
|
||||
# TAOS_RATIO_OF_QUERY_CORES:
|
||||
# the proportion of total CPU cores available for query processing
|
||||
# 2.0: the query threads will be set to double of the CPU cores.
|
||||
# 1.0: all CPU cores are available for query processing [default].
|
||||
# 0.5: only half of the CPU cores are available for query.
|
||||
# 0.0: only one core available.
|
||||
#TAOS_RATIO_OF_QUERY_CORES: "1.0"
|
||||
|
||||
#
|
||||
# TAOS_KEEP_COLUMN_NAME:
|
||||
# the last_row/first/last aggregator will not change the original column name in the result fields
|
||||
#TAOS_KEEP_COLUMN_NAME: "0"
|
||||
|
||||
# enable/disable backuping vnode directory when removing vnode
|
||||
#TAOS_VNODE_BAK: "1"
|
||||
|
||||
# enable/disable installation / usage report
|
||||
#TAOS_TELEMETRY_REPORTING: "1"
|
||||
|
||||
# enable/disable load balancing
|
||||
#TAOS_BALANCE: "1"
|
||||
|
||||
# max timer control blocks
|
||||
#TAOS_MAX_TMR_CTRL: "512"
|
||||
|
||||
# time interval of system monitor, seconds
|
||||
#TAOS_MONITOR_INTERVAL: "30"
|
||||
|
||||
# number of seconds allowed for a dnode to be offline, for cluster only
|
||||
#TAOS_OFFLINE_THRESHOLD: "8640000"
|
||||
|
||||
# RPC re-try timer, millisecond
|
||||
#TAOS_RPC_TIMER: "1000"
|
||||
|
||||
# RPC maximum time for ack, seconds.
|
||||
#TAOS_RPC_MAX_TIME: "600"
|
||||
|
||||
# time interval of dnode status reporting to mnode, seconds, for cluster only
|
||||
#TAOS_STATUS_INTERVAL: "1"
|
||||
|
||||
|
|
@ -245,37 +195,7 @@ taoscfg:
|
|||
#TAOS_MIN_SLIDING_TIME: "10"
|
||||
|
||||
# minimum time window, milli-second
|
||||
#TAOS_MIN_INTERVAL_TIME: "10"
|
||||
|
||||
# maximum delay before launching a stream computation, milli-second
|
||||
#TAOS_MAX_STREAM_COMP_DELAY: "20000"
|
||||
|
||||
# maximum delay before launching a stream computation for the first time, milli-second
|
||||
#TAOS_MAX_FIRST_STREAM_COMP_DELAY: "10000"
|
||||
|
||||
# retry delay when a stream computation fails, milli-second
|
||||
#TAOS_RETRY_STREAM_COMP_DELAY: "10"
|
||||
|
||||
# the delayed time for launching a stream computation, from 0.1(default, 10% of whole computing time window) to 0.9
|
||||
#TAOS_STREAM_COMP_DELAY_RATIO: "0.1"
|
||||
|
||||
# max number of vgroups per db, 0 means configured automatically
|
||||
#TAOS_MAX_VGROUPS_PER_DB: "0"
|
||||
|
||||
# max number of tables per vnode
|
||||
#TAOS_MAX_TABLES_PER_VNODE: "1000000"
|
||||
|
||||
# the number of acknowledgments required for successful data writing
|
||||
#TAOS_QUORUM: "1"
|
||||
|
||||
# enable/disable compression
|
||||
#TAOS_COMP: "2"
|
||||
|
||||
# write ahead log (WAL) level, 0: no wal; 1: write wal, but no fysnc; 2: write wal, and call fsync
|
||||
#TAOS_WAL_LEVEL: "1"
|
||||
|
||||
# if walLevel is set to 2, the cycle of fsync being executed, if set to 0, fsync is called right away
|
||||
#TAOS_FSYNC: "3000"
|
||||
#TAOS_MIN_INTERVAL_TIME: "1"
|
||||
|
||||
# the compressed rpc message, option:
|
||||
# -1 (no compression)
|
||||
|
|
@ -283,17 +203,8 @@ taoscfg:
|
|||
# > 0 (rpc message body which larger than this value will be compressed)
|
||||
#TAOS_COMPRESS_MSG_SIZE: "-1"
|
||||
|
||||
# max length of an SQL
|
||||
#TAOS_MAX_SQL_LENGTH: "1048576"
|
||||
|
||||
# the maximum number of records allowed for super table time sorting
|
||||
#TAOS_MAX_NUM_OF_ORDERED_RES: "100000"
|
||||
|
||||
# max number of connections allowed in dnode
|
||||
#TAOS_MAX_SHELL_CONNS: "5000"
|
||||
|
||||
# max number of connections allowed in client
|
||||
#TAOS_MAX_CONNECTIONS: "5000"
|
||||
#TAOS_MAX_SHELL_CONNS: "50000"
|
||||
|
||||
# stop writing logs when the disk size of the log folder is less than this value
|
||||
#TAOS_MINIMAL_LOG_DIR_G_B: "0.1"
|
||||
|
|
@ -313,21 +224,8 @@ taoscfg:
|
|||
# enable/disable system monitor
|
||||
#TAOS_MONITOR: "1"
|
||||
|
||||
# enable/disable recording the SQL statements via restful interface
|
||||
#TAOS_HTTP_ENABLE_RECORD_SQL: "0"
|
||||
|
||||
# number of threads used to process http requests
|
||||
#TAOS_HTTP_MAX_THREADS: "2"
|
||||
|
||||
# maximum number of rows returned by the restful interface
|
||||
#TAOS_RESTFUL_ROW_LIMIT: "10240"
|
||||
|
||||
# The following parameter is used to limit the maximum number of lines in log files.
|
||||
# max number of lines per log filters
|
||||
# numOfLogLines 10000000
|
||||
|
||||
# enable/disable async log
|
||||
#TAOS_ASYNC_LOG: "0"
|
||||
#TAOS_ASYNC_LOG: "1"
|
||||
|
||||
#
|
||||
# time of keeping log files, days
|
||||
|
|
@ -344,25 +242,8 @@ taoscfg:
|
|||
# debug flag for all log type, take effect when non-zero value\
|
||||
#TAOS_DEBUG_FLAG: "143"
|
||||
|
||||
# enable/disable recording the SQL in taos client
|
||||
#TAOS_ENABLE_RECORD_SQL: "0"
|
||||
|
||||
# generate core file when service crash
|
||||
#TAOS_ENABLE_CORE_FILE: "1"
|
||||
|
||||
# maximum display width of binary and nchar fields in the shell. The parts exceeding this limit will be hidden
|
||||
#TAOS_MAX_BINARY_DISPLAY_WIDTH: "30"
|
||||
|
||||
# enable/disable stream (continuous query)
|
||||
#TAOS_STREAM: "1"
|
||||
|
||||
# in retrieve blocking model, only in 50% query threads will be used in query processing in dnode
|
||||
#TAOS_RETRIEVE_BLOCKING_MODEL: "0"
|
||||
|
||||
# the maximum allowed query buffer size in MB during query processing for each data node
|
||||
# -1 no limit (default)
|
||||
# 0 no query allowed, queries are disabled
|
||||
#TAOS_QUERY_BUFFER_SIZE: "-1"
|
||||
```
|
||||
|
||||
## Scaling Out
|
||||
|
|
|
|||
|
|
@ -11,7 +11,7 @@ When using TDengine to store and query data, the most important part of the data
|
|||
- The format must be `YYYY-MM-DD HH:mm:ss.MS`, the default time precision is millisecond (ms), for example `2017-08-12 18:25:58.128`
|
||||
- Internal function `now` can be used to get the current timestamp on the client side
|
||||
- The current timestamp of the client side is applied when `now` is used to insert data
|
||||
- Epoch Time：timestamp can also be a long integer number, which means the number of seconds, milliseconds or nanoseconds, depending on the time precision, from 1970-01-01 00:00:00.000 (UTC/GMT)
|
||||
- Epoch Time：timestamp can also be a long integer number, which means the number of seconds, milliseconds or nanoseconds, depending on the time precision, from UTC 1970-01-01 00:00:00.
|
||||
- Add/subtract operations can be carried out on timestamps. For example `now-2h` means 2 hours prior to the time at which query is executed. The units of time in operations can be b(nanosecond), u(microsecond), a(millisecond), s(second), m(minute), h(hour), d(day), or w(week). So `select * from t1 where ts > now-2w and ts <= now-1w` means the data between two weeks ago and one week ago. The time unit can also be n (calendar month) or y (calendar year) when specifying the time window for down sampling operations.
|
||||
|
||||
Time precision in TDengine can be set by the `PRECISION` parameter when executing `CREATE DATABASE`. The default time precision is millisecond. In the statement below, the precision is set to nanonseconds.
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ table_option: {
|
|||
3. MAX_DELAY: specifies the maximum latency for pushing computation results. The default value is 15 minutes or the value of the INTERVAL parameter, whichever is smaller. Enter a value between 0 and 15 minutes in milliseconds, seconds, or minutes. You can enter multiple values separated by commas (,). Note: Retain the default value if possible. Configuring a small MAX_DELAY may cause results to be frequently pushed, affecting storage and query performance. This parameter applies only to supertables and takes effect only when the RETENTIONS parameter has been specified for the database.
|
||||
4. ROLLUP: specifies aggregate functions to roll up. Rolling up a function provides downsampled results based on multiple axes. This parameter applies only to supertables and takes effect only when the RETENTIONS parameter has been specified for the database. You can specify only one function to roll up. The rollup takes effect on all columns except TS. Enter one of the following values: avg, sum, min, max, last, or first.
|
||||
5. SMA: specifies functions on which to enable small materialized aggregates (SMA). SMA is user-defined precomputation of aggregates based on data blocks. Enter one of the following values: max, min, or sum This parameter can be used with supertables and standard tables.
|
||||
6. TTL: specifies the time to live (TTL) for the table. If the period specified by the TTL parameter elapses without any data being written to the table, TDengine will automatically delete the table. Note: The system may not delete the table at the exact moment that the TTL expires. Enter a value in days. The default value is 0. Note: The TTL parameter has a higher priority than the KEEP parameter. If a table is marked for deletion because the TTL has expired, it will be deleted even if the time specified by the KEEP parameter has not elapsed. This parameter can be used with standard tables and subtables.
|
||||
6. TTL: specifies the time to live (TTL) for the table. If TTL is specified when creatinga table, after the time period for which the table has been existing is over TTL, TDengine will automatically delete the table. Please be noted that the system may not delete the table at the exact moment that the TTL expires but guarantee there is such a system and finally the table will be deleted. The unit of TTL is in days. The default value is 0, i.e. never expire.
|
||||
|
||||
## Create Subtables
|
||||
|
||||
|
|
|
|||
|
|
@ -171,8 +171,8 @@ The \_QSTART and \_QEND pseudocolumns contain the beginning and end of the time
|
|||
|
||||
The \_QSTART and \_QEND pseudocolumns cannot be used in a WHERE clause.
|
||||
|
||||
**\_WSTART, \_WEND, and \_DURATION**
|
||||
\_WSTART, \_WEND, and \_WDURATION pseudocolumns
|
||||
**\_WSTART, \_WEND, and \_WDURATION**
|
||||
|
||||
The \_WSTART, \_WEND, and \_WDURATION pseudocolumns indicate the beginning, end, and duration of a window.
|
||||
|
||||
These pseudocolumns can be used only in time window-based aggregations and must occur after the aggregation clause.
|
||||
|
|
|
|||
|
|
@ -846,7 +846,7 @@ SELECT FIRST(field_name) FROM { tb_name | stb_name } [WHERE clause];
|
|||
### INTERP
|
||||
|
||||
```sql
|
||||
SELECT INTERP(field_name) FROM { tb_name | stb_name } [WHERE where_condition] [ RANGE(timestamp1,timestamp2) ] [EVERY(interval)] [FILL ({ VALUE | PREV | NULL | LINEAR | NEXT})];
|
||||
SELECT INTERP(field_name) FROM { tb_name | stb_name } [WHERE where_condition] RANGE(timestamp1,timestamp2) EVERY(interval) FILL({ VALUE | PREV | NULL | LINEAR | NEXT});
|
||||
```
|
||||
|
||||
**Description**: The value that matches the specified timestamp range is returned, if existing; or an interpolation value is returned.
|
||||
|
|
@ -861,11 +861,10 @@ SELECT INTERP(field_name) FROM { tb_name | stb_name } [WHERE where_condition] [
|
|||
|
||||
- `INTERP` is used to get the value that matches the specified time slice from a column. If no such value exists an interpolation value will be returned based on `FILL` parameter.
|
||||
- The input data of `INTERP` is the value of the specified column and a `where` clause can be used to filter the original data. If no `where` condition is specified then all original data is the input.
|
||||
- The output time range of `INTERP` is specified by `RANGE(timestamp1,timestamp2)` parameter, with timestamp1<=timestamp2. timestamp1 is the starting point of the output time range and must be specified. timestamp2 is the ending point of the output time range and must be specified. If `RANGE` is not specified, then the timestamp of the first row that matches the filter condition is treated as timestamp1, the timestamp of the last row that matches the filter condition is treated as timestamp2.
|
||||
- The number of rows in the result set of `INTERP` is determined by the parameter `EVERY`. Starting from timestamp1, one interpolation is performed for every time interval specified `EVERY` parameter. If `EVERY` parameter is not used, the time windows will be considered as no ending timestamp, i.e. there is only one time window from timestamp1.
|
||||
- Interpolation is performed based on `FILL` parameter. No interpolation is performed if `FILL` is not used, that means either the original data that matches is returned or nothing is returned.
|
||||
- `INTERP` can only be used to interpolate in single timeline. So it must be used with `group by tbname` when it's used on a STable. It can't be used with `GROUP BY` when it's used in the inner query of a nested query.
|
||||
- The result of `INTERP` is not influenced by `ORDER BY TIMESTAMP`, which impacts the output order only..
|
||||
- The output time range of `INTERP` is specified by `RANGE(timestamp1,timestamp2)` parameter, with timestamp1<=timestamp2. timestamp1 is the starting point of the output time range and must be specified. timestamp2 is the ending point of the output time range and must be specified.
|
||||
- The number of rows in the result set of `INTERP` is determined by the parameter `EVERY`. Starting from timestamp1, one interpolation is performed for every time interval specified `EVERY` parameter.
|
||||
- Interpolation is performed based on `FILL` parameter.
|
||||
- `INTERP` can only be used to interpolate in single timeline. So it must be used with `partition by tbname` when it's used on a STable.
|
||||
|
||||
### LAST
|
||||
|
||||
|
|
@ -1140,7 +1139,7 @@ SELECT STATECOUNT(field_name, oper, val) FROM { tb_name | stb_name } [WHERE clau
|
|||
|
||||
**Applicable parameter values**:
|
||||
|
||||
- oper : Can be one of `LT` (lower than), `GT` (greater than), `LE` (lower than or equal to), `GE` (greater than or equal to), `NE` (not equal to), `EQ` (equal to), the value is case insensitive
|
||||
- oper : Can be one of `'LT'` (lower than), `'GT'` (greater than), `'LE'` (lower than or equal to), `'GE'` (greater than or equal to), `'NE'` (not equal to), `'EQ'` (equal to), the value is case insensitive, the value must be in quotes.
|
||||
- val ： Numeric types
|
||||
|
||||
**Return value type**: Integer
|
||||
|
|
@ -1167,7 +1166,7 @@ SELECT stateDuration(field_name, oper, val, unit) FROM { tb_name | stb_name } [W
|
|||
|
||||
**Applicable parameter values**:
|
||||
|
||||
- oper : Can be one of `LT` (lower than), `GT` (greater than), `LE` (lower than or equal to), `GE` (greater than or equal to), `NE` (not equal to), `EQ` (equal to), the value is case insensitive
|
||||
- oper : Can be one of `'LT'` (lower than), `'GT'` (greater than), `'LE'` (lower than or equal to), `'GE'` (greater than or equal to), `'NE'` (not equal to), `'EQ'` (equal to), the value is case insensitive, the value must be in quotes.
|
||||
- val ： Numeric types
|
||||
- unit: The unit of time interval. Enter one of the following options: 1b (nanoseconds), 1u (microseconds), 1a (milliseconds), 1s (seconds), 1m (minutes), 1h (hours), 1d (days), or 1w (weeks) If you do not enter a unit of time, the precision of the current database is used by default.
|
||||
|
||||
|
|
@ -1232,7 +1231,7 @@ SELECT SERVER_VERSION();
|
|||
### SERVER_STATUS
|
||||
|
||||
```sql
|
||||
SELECT SERVER_VERSION();
|
||||
SELECT SERVER_STATUS();
|
||||
```
|
||||
|
||||
**Description**: The server status.
|
||||
|
|
|
|||
|
|
@ -58,6 +58,15 @@ The following restrictions apply:
|
|||
- The window clause cannot be used with a GROUP BY clause.
|
||||
- `WHERE` clause can be used to specify the starting and ending time and other filter conditions
|
||||
|
||||
|
||||
### Window Pseudocolumns
|
||||
|
||||
**\_WSTART, \_WEND, and \_WDURATION**
|
||||
|
||||
The \_WSTART, \_WEND, and \_WDURATION pseudocolumns indicate the beginning, end, and duration of a window.
|
||||
|
||||
These pseudocolumns occur after the aggregation clause.
|
||||
|
||||
### FILL Clause
|
||||
|
||||
`FILL` clause is used to specify how to fill when there is data missing in any window, including:
|
||||
|
|
|
|||
|
|
@ -30,7 +30,7 @@ The following characters cannot occur in a password: single quotation marks ('),
|
|||
- Maximum number of columns is 4096. There must be at least 2 columns, and the first column must be timestamp.
|
||||
- The maximum length of a tag name is 64 bytes
|
||||
- Maximum number of tags is 128. There must be at least 1 tag. The total length of tag values cannot exceed 16 KB.
|
||||
- Maximum length of single SQL statement is 1 MB (1048576 bytes). It can be configured in the parameter `maxSQLLength` in the client side, the applicable range is [65480, 1048576].
|
||||
- Maximum length of single SQL statement is 1 MB (1048576 bytes).
|
||||
- At most 4096 columns can be returned by `SELECT`. Functions in the query statement constitute columns. An error is returned if the limit is exceeded.
|
||||
- Maximum numbers of databases, STables, tables are dependent only on the system resources.
|
||||
- The number of replicas can only be 1 or 3.
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ sidebar_label: SHOW Statement
|
|||
title: SHOW Statement for Metadata
|
||||
---
|
||||
|
||||
In addition to running SELECT statements on INFORMATION_SCHEMA, you can also use SHOW to obtain system metadata, information, and status.
|
||||
`SHOW` command can be used to get brief system information. To get details about metatadata, information, and status in the system, please use `select` to query the tables in database `INFORMATION_SCHEMA`.
|
||||
|
||||
## SHOW ACCOUNTS
|
||||
|
||||
|
|
@ -194,7 +194,7 @@ Shows information about streams in the system.
|
|||
SHOW SUBSCRIPTIONS;
|
||||
```
|
||||
|
||||
Shows all subscriptions in the current database.
|
||||
Shows all subscriptions in the system.
|
||||
|
||||
## SHOW TABLES
|
||||
|
||||
|
|
|
|||
|
|
@ -15,9 +15,41 @@ About details of installing TDenine, please refer to [Installation Guide](../../
|
|||
## Uninstall
|
||||
|
||||
<Tabs>
|
||||
<TabItem label="apt-get uninstall" value="aptremove">
|
||||
<TabItem label="Uninstall apt-get" value="aptremove">
|
||||
|
||||
TBD
|
||||
Apt-get package of TDengine can be uninstalled as below:
|
||||
|
||||
```bash
|
||||
$ sudo apt-get remove tdengine
|
||||
Reading package lists... Done
|
||||
Building dependency tree
|
||||
Reading state information... Done
|
||||
The following packages will be REMOVED:
|
||||
tdengine
|
||||
0 upgraded, 0 newly installed, 1 to remove and 18 not upgraded.
|
||||
After this operation, 68.3 MB disk space will be freed.
|
||||
Do you want to continue? [Y/n] y
|
||||
(Reading database ... 135625 files and directories currently installed.)
|
||||
Removing tdengine (3.0.0.0) ...
|
||||
TDengine is removed successfully!
|
||||
|
||||
```
|
||||
|
||||
Apt-get package of taosTools can be uninstalled as below:
|
||||
|
||||
```
|
||||
$ sudo apt remove taostools
|
||||
Reading package lists... Done
|
||||
Building dependency tree
|
||||
Reading state information... Done
|
||||
The following packages will be REMOVED:
|
||||
taostools
|
||||
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
|
||||
After this operation, 68.3 MB disk space will be freed.
|
||||
Do you want to continue? [Y/n]
|
||||
(Reading database ... 147973 files and directories currently installed.)
|
||||
Removing taostools (2.1.2) ...
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem label="Uninstall Deb" value="debuninst">
|
||||
|
|
@ -32,6 +64,14 @@ TDengine is removed successfully!
|
|||
|
||||
```
|
||||
|
||||
Deb package of taosTools can be uninstalled as below:
|
||||
|
||||
```
|
||||
$ sudo dpkg -r taostools
|
||||
(Reading database ... 147973 files and directories currently installed.)
|
||||
Removing taostools (2.1.2) ...
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Uninstall RPM" value="rpmuninst">
|
||||
|
|
@ -43,6 +83,13 @@ $ sudo rpm -e tdengine
|
|||
TDengine is removed successfully!
|
||||
```
|
||||
|
||||
RPM package of taosTools can be uninstalled as below:
|
||||
|
||||
```
|
||||
sudo rpm -e taostools
|
||||
taosToole is removed successfully!
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Uninstall tar.gz" value="taruninst">
|
||||
|
|
@ -54,6 +101,15 @@ $ rmtaos
|
|||
TDengine is removed successfully!
|
||||
```
|
||||
|
||||
tar.gz package of taosTools can be uninstalled as below:
|
||||
|
||||
```
|
||||
$ rmtaostools
|
||||
Start to uninstall taos tools ...
|
||||
|
||||
taos tools is uninstalled successfully!
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
<TabItem label="Windows uninstall" value="windows">
|
||||
Run C:\TDengine\unins000.exe to uninstall TDengine on a Windows system.
|
||||
|
|
|
|||
|
|
@ -27,4 +27,4 @@ The number of dnodes in a TDengine cluster must NOT be lower than the number of
|
|||
|
||||
As long as the dnodes of a TDengine cluster are deployed on different physical machines and the replica number is higher than 1, high availability can be achieved without any other assistance. For disaster recovery, dnodes of a TDengine cluster should be deployed in geographically different data centers.
|
||||
|
||||
Alternatively, you can use taosX to synchronize the data from one TDengine cluster to another cluster in a remote location. For more information, see [taosX](../../reference/taosX).
|
||||
Alternatively, you can use taosX to synchronize the data from one TDengine cluster to another cluster in a remote location. However, taosX is only available in TDengine enterprise version, for more information please contact tdengine.com.
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ One difference from the native connector is that the REST interface is stateless
|
|||
|
||||
## Installation
|
||||
|
||||
The REST interface does not rely on any TDengine native library, so the client application does not need to install any TDengine libraries. The client application's development language only needs to support the HTTP protocol.
|
||||
The REST interface does not rely on any TDengine native library, so the client application does not need to install any TDengine libraries. The client application's development language only needs to support the HTTP protocol. The REST interface is provided by [taosAdapter](../taosadapter), to use REST interface you need to make sure `taosAdapter` is running properly.
|
||||
|
||||
## Verification
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: C/C++
|
||||
title: C/C++ Connector
|
||||
---
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 2
|
||||
sidebar_label: Java
|
||||
title: TDengine Java Connector
|
||||
description: The TDengine Java Connector is implemented on the standard JDBC API and provides native and REST connectors.
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 4
|
||||
sidebar_label: Go
|
||||
title: TDengine Go Connector
|
||||
---
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 5
|
||||
sidebar_label: Rust
|
||||
title: TDengine Rust Connector
|
||||
---
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
sidebar_position: 3
|
||||
sidebar_label: Python
|
||||
title: TDengine Python Connector
|
||||
description: "taospy is the official Python connector for TDengine. taospy provides a rich API that makes it easy for Python applications to use TDengine. tasopy wraps both the native and REST interfaces of TDengine, corresponding to the two submodules of tasopy: taos and taosrest. In addition to wrapping the native and REST interfaces, taospy also provides a programming interface that conforms to the Python Data Access Specification (PEP 249), making it easy to integrate taospy with many third-party tools, such as SQLAlchemy and pandas."
|
||||
|
|
@ -150,10 +149,19 @@ If the test is successful, it will output the server version information, e.g.
|
|||
|
||||
```json
|
||||
{
|
||||
"status": "succ",
|
||||
"head": ["server_version()"],
|
||||
"column_meta": [["server_version()", 8, 8]],
|
||||
"data": [["2.4.0.16"]],
|
||||
"code": 0,
|
||||
"column_meta": [
|
||||
[
|
||||
"server_version()",
|
||||
"VARCHAR",
|
||||
7
|
||||
]
|
||||
],
|
||||
"data": [
|
||||
[
|
||||
"3.0.0.0"
|
||||
]
|
||||
],
|
||||
"rows": 1
|
||||
}
|
||||
```
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 6
|
||||
sidebar_label: Node.js
|
||||
title: TDengine Node.js Connector
|
||||
---
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 7
|
||||
sidebar_label: C#
|
||||
title: C# Connector
|
||||
---
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: PHP
|
||||
title: PHP Connector
|
||||
---
|
||||
|
|
@ -329,4 +329,4 @@ In TDengine server 2.2.x.x or earlier, the TDengine server process (taosd) conta
|
|||
| 3 | telegrafUseFieldNum | See the taosAdapter telegraf configuration method | |
|
||||
| 4 | restfulRowLimit | restfulRowLimit | Embedded httpd outputs 10240 rows of data by default, the maximum allowed is 102400. taosAdapter also provides restfulRowLimit but it is not limited by default. You can configure it according to the actual scenario.
|
||||
| 5 | httpDebugFlag | Not applicable | httpdDebugFlag does not work for taosAdapter |
|
||||
| 6 | httpDBNameMandatory | N/A | taosAdapter requires the database name to be specified in the URL |
|
||||
| 6 | httpDBNameMandatory | N/A | taosAdapter requires the database name to be specified in the URL |
|
||||
|
|
|
|||
|
|
@ -72,7 +72,7 @@ Next, ensure the hostname "tdengine" is resolvable in `/etc/hosts`.
|
|||
echo 127.0.0.1 tdengine |sudo tee -a /etc/hosts
|
||||
```
|
||||
|
||||
Finally, the TDengine service can be accessed from the taos shell or any connector with "tdengine" as the server address.
|
||||
Finally, the TDengine service can be accessed from the TDengine CLI or any connector with "tdengine" as the server address.
|
||||
|
||||
```shell
|
||||
taos -h tdengine -P 6030
|
||||
|
|
@ -115,7 +115,7 @@ If you want to start your application in a container, you need to add the corres
|
|||
```docker
|
||||
FROM ubuntu:20.04
|
||||
RUN apt-get update && apt-get install -y wget
|
||||
ENV TDENGINE_VERSION=2.4.0.0
|
||||
ENV TDENGINE_VERSION=3.0.0.0
|
||||
RUN wget -c https://www.taosdata.com/assets-download/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
|
||||
&& tar xvf TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
|
||||
&& cd TDengine-client-${TDENGINE_VERSION} \
|
||||
|
|
@ -216,7 +216,7 @@ Here is the full Dockerfile:
|
|||
|
||||
```docker
|
||||
FROM golang:1.17.6-buster as builder
|
||||
ENV TDENGINE_VERSION=2.4.0.0
|
||||
ENV TDENGINE_VERSION=3.0.0.0
|
||||
RUN wget -c https://www.taosdata.com/assets-download/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
|
||||
&& tar xvf TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
|
||||
&& cd TDengine-client-${TDENGINE_VERSION} \
|
||||
|
|
@ -232,7 +232,7 @@ RUN go build
|
|||
|
||||
FROM ubuntu:20.04
|
||||
RUN apt-get update && apt-get install -y wget
|
||||
ENV TDENGINE_VERSION=2.4.0.0
|
||||
ENV TDENGINE_VERSION=3.0.0.0
|
||||
RUN wget -c https://www.taosdata.com/assets-download/TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
|
||||
&& tar xvf TDengine-client-${TDENGINE_VERSION}-Linux-x64.tar.gz \
|
||||
&& cd TDengine-client-${TDENGINE_VERSION} \
|
||||
|
|
@ -320,7 +320,7 @@ password: taosdata
|
|||
2. Start the cluster
|
||||
|
||||
```shell
|
||||
$ VERSION=2.4.0.0 docker-compose up -d
|
||||
$ VERSION=3.0.0.0 docker-compose up -d
|
||||
Creating network "test_default" with the default driver
|
||||
Creating volume "test_taosdata-td1" with default driver
|
||||
Creating volume "test_taoslog-td1" with default driver
|
||||
|
|
@ -457,7 +457,7 @@ If you want to deploy a container-based TDengine cluster on multiple hosts, you
|
|||
The docker-compose file can refer to the previous section. Here is the command to start TDengine with docker swarm:
|
||||
|
||||
```shell
|
||||
$ VERSION=2.4.0 docker stack deploy -c docker-compose.yml taos
|
||||
$ VERSION=3.0.0.0 docker stack deploy -c docker-compose.yml taos
|
||||
Creating network taos_inter
|
||||
Creating network taos_api
|
||||
Creating service taos_arbitrator
|
||||
|
|
@ -473,20 +473,20 @@ Checking status:
|
|||
$ docker stack ps taos
|
||||
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
|
||||
79ni8temw59n taos_nginx.1 nginx:latest TM1701 Running Running about a minute ago
|
||||
3e94u72msiyg taos_adapter.1 tdengine/tdengine:2.4.0 TM1702 Running Running 56 seconds ago
|
||||
100amjkwzsc6 taos_td-2.1 tdengine/tdengine:2.4.0 TM1703 Running Running about a minute ago
|
||||
pkjehr2vvaaa taos_td-1.1 tdengine/tdengine:2.4.0 TM1704 Running Running 2 minutes ago
|
||||
tpzvgpsr1qkt taos_arbitrator.1 tdengine/tdengine:2.4.0 TM1705 Running Running 2 minutes ago
|
||||
rvss3g5yg6fa taos_adapter.2 tdengine/tdengine:2.4.0 TM1706 Running Running 56 seconds ago
|
||||
i2augxamfllf taos_adapter.3 tdengine/tdengine:2.4.0 TM1707 Running Running 56 seconds ago
|
||||
lmjyhzccpvpg taos_adapter.4 tdengine/tdengine:2.4.0 TM1708 Running Running 56 seconds ago
|
||||
3e94u72msiyg taos_adapter.1 tdengine/tdengine:3.0.0.0 TM1702 Running Running 56 seconds ago
|
||||
100amjkwzsc6 taos_td-2.1 tdengine/tdengine:3.0.0.0 TM1703 Running Running about a minute ago
|
||||
pkjehr2vvaaa taos_td-1.1 tdengine/tdengine:3.0.0.0 TM1704 Running Running 2 minutes ago
|
||||
tpzvgpsr1qkt taos_arbitrator.1 tdengine/tdengine:3.0.0.0 TM1705 Running Running 2 minutes ago
|
||||
rvss3g5yg6fa taos_adapter.2 tdengine/tdengine:3.0.0.0 TM1706 Running Running 56 seconds ago
|
||||
i2augxamfllf taos_adapter.3 tdengine/tdengine:3.0.0.0 TM1707 Running Running 56 seconds ago
|
||||
lmjyhzccpvpg taos_adapter.4 tdengine/tdengine:3.0.0.0 TM1708 Running Running 56 seconds ago
|
||||
$ docker service ls
|
||||
ID NAME MODE REPLICAS IMAGE PORTS
|
||||
561t4lu6nfw6 taos_adapter replicated 4/4 tdengine/tdengine:2.4.0
|
||||
3hk5ct3q90sm taos_arbitrator replicated 1/1 tdengine/tdengine:2.4.0
|
||||
561t4lu6nfw6 taos_adapter replicated 4/4 tdengine/tdengine:3.0.0.0
|
||||
3hk5ct3q90sm taos_arbitrator replicated 1/1 tdengine/tdengine:3.0.0.0
|
||||
d8qr52envqzu taos_nginx replicated 1/1 nginx:latest *:6041->6041/tcp, *:6044->6044/udp
|
||||
2isssfvjk747 taos_td-1 replicated 1/1 tdengine/tdengine:2.4.0
|
||||
9pzw7u02ichv taos_td-2 replicated 1/1 tdengine/tdengine:2.4.0
|
||||
2isssfvjk747 taos_td-1 replicated 1/1 tdengine/tdengine:3.0.0.0
|
||||
9pzw7u02ichv taos_td-2 replicated 1/1 tdengine/tdengine:3.0.0.0
|
||||
```
|
||||
|
||||
From the above output, you can see two dnodes, two taosAdapters, and one Nginx reverse proxy service.
|
||||
|
|
@ -502,5 +502,5 @@ verify: Service converged
|
|||
|
||||
$ docker service ls -f name=taos_adapter
|
||||
ID NAME MODE REPLICAS IMAGE PORTS
|
||||
561t4lu6nfw6 taos_adapter replicated 1/1 tdengine/tdengine:2.4.0
|
||||
561t4lu6nfw6 taos_adapter replicated 1/1 tdengine/tdengine:3.0.0.0
|
||||
```
|
||||
|
|
|
|||
|
|
@ -82,13 +82,12 @@ The parameters described in this document by the effect that they have on the sy
|
|||
| Protocol | Default Port | Description | How to configure |
|
||||
| :------- | :----------- | :----------------------------------------------- | :--------------------------------------------------------------------------------------------- |
|
||||
| TCP | 6030 | Communication between client and server. In a multi-node cluster, communication between nodes. serverPort |
|
||||
| TCP | 6041 | REST connection between client and server | refer to [taosAdapter](../taosadapter/) |
|
||||
| TCP | 6041 | REST connection between client and server | Prior to 2.4.0.0: serverPort+11; After 2.4.0.0 refer to [taosAdapter](/reference/taosadapter/) |
|
||||
| TCP | 6043 | Service Port of TaosKeeper | The parameter of TaosKeeper |
|
||||
| TCP | 6044 | Data access port for StatsD | Configurable through taosAdapter parameters.
|
||||
| UDP | 6045 | Data access for statsd | Configurable through taosAdapter parameters.
|
||||
| TCP | 6060 | Port of Monitoring Service in Enterprise version | |
|
||||
|
||||
|
||||
### maxShellConns
|
||||
|
||||
| Attribute | Description |
|
||||
|
|
@ -345,7 +344,7 @@ The charset that takes effect is UTF-8.
|
|||
| Attribute | Description |
|
||||
| -------- | --------------------------------- |
|
||||
| Applicable | Server and Client |
|
||||
| Meaning | The interval for taos shell to send heartbeat to mnode |
|
||||
| Meaning | The interval for TDengine CLI to send heartbeat to mnode |
|
||||
| Unit | second |
|
||||
| Value Range | 1-120 |
|
||||
| Default Value | 3 |
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 6.9 KiB After Width: | Height: | Size: 26 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 6.3 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 41 KiB |
Binary file not shown.
|
After Width: | Height: | Size: 259 KiB |
|
|
@ -34,7 +34,7 @@ Please refer to the [official documentation](https://grafana.com/grafana/downloa
|
|||
|
||||
### TDengine
|
||||
|
||||
Download the latest TDengine-server 2.4.0.x or above from the [Downloads](http://taosdata.com/cn/all-downloads/) page on the TAOSData website and install it.
|
||||
Download the latest TDengine-server from the [Downloads](http://tdengine.com/en/all-downloads/) page on the TAOSData website and install it.
|
||||
|
||||
## Data Connection Setup
|
||||
|
||||
|
|
@ -79,5 +79,5 @@ Click on the plus icon on the left and select `Import` to get the data from `htt
|
|||
|
||||
## Wrap-up
|
||||
|
||||
The above demonstrates how to quickly build a IT DevOps visualization system. Thanks to the new schemaless protocol parsing feature in TDengine version 2.4.0.0 and ability to integrate easily with a large software ecosystem, users can build an efficient and easy-to-use IT DevOps visualization system in just a few minutes.
|
||||
The above demonstrates how to quickly build a IT DevOps visualization system. Thanks to the schemaless protocol parsing feature in TDengine and ability to integrate easily with a large software ecosystem, users can build an efficient and easy-to-use IT DevOps visualization system in just a few minutes.
|
||||
Please refer to the official documentation and product implementation cases for other features.
|
||||
|
|
|
|||
|
|
@ -37,7 +37,7 @@ Please refer to the [official documentation](https://grafana.com/grafana/downloa
|
|||
|
||||
### Install TDengine
|
||||
|
||||
Download the latest TDengine-server 2.4.0.x or above from the [Downloads](http://taosdata.com/cn/all-downloads/) page on the TAOSData website and install it.
|
||||
Download the latest TDengine-server from the [Downloads](http://tdengine.com/en/all-downloads/) page on the TAOSData website and install it.
|
||||
|
||||
## Data Connection Setup
|
||||
|
||||
|
|
@ -99,6 +99,6 @@ Download the dashboard json from `https://github.com/taosdata/grafanaplugin/blob
|
|||
|
||||
## Wrap-up
|
||||
|
||||
TDengine, as an emerging time-series big data platform, has the advantages of high performance, high reliability, easy management and easy maintenance. Thanks to the new schemaless protocol parsing feature in TDengine version 2.4.0.0 and ability to integrate easily with a large software ecosystem, users can build an efficient and easy-to-use IT DevOps visualization system, or adapt an existing system, in just a few minutes.
|
||||
TDengine, as an emerging time-series big data platform, has the advantages of high performance, high reliability, easy management and easy maintenance. Thanks to the new schemaless protocol parsing feature in TDengine and ability to integrate easily with a large software ecosystem, users can build an efficient and easy-to-use IT DevOps visualization system, or adapt an existing system, in just a few minutes.
|
||||
|
||||
For TDengine's powerful data writing and querying performance and other features, please refer to the official documentation and successful product implementation cases.
|
||||
|
|
|
|||
Binary file not shown.
|
Before Width: | Height: | Size: 38 KiB After Width: | Height: | Size: 16 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 30 KiB After Width: | Height: | Size: 15 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 41 KiB After Width: | Height: | Size: 15 KiB |
|
|
@ -16,14 +16,14 @@ public class RestInsertExample {
|
|||
|
||||
private static List<String> getRawData() {
|
||||
return Arrays.asList(
|
||||
"d1001,2018-10-03 14:38:05.000,10.30000,219,0.31000,California.SanFrancisco,2",
|
||||
"d1001,2018-10-03 14:38:15.000,12.60000,218,0.33000,California.SanFrancisco,2",
|
||||
"d1001,2018-10-03 14:38:16.800,12.30000,221,0.31000,California.SanFrancisco,2",
|
||||
"d1002,2018-10-03 14:38:16.650,10.30000,218,0.25000,California.SanFrancisco,3",
|
||||
"d1003,2018-10-03 14:38:05.500,11.80000,221,0.28000,California.LosAngeles,2",
|
||||
"d1003,2018-10-03 14:38:16.600,13.40000,223,0.29000,California.LosAngeles,2",
|
||||
"d1004,2018-10-03 14:38:05.000,10.80000,223,0.29000,California.LosAngeles,3",
|
||||
"d1004,2018-10-03 14:38:06.500,11.50000,221,0.35000,California.LosAngeles,3"
|
||||
"d1001,2018-10-03 14:38:05.000,10.30000,219,0.31000,'California.SanFrancisco',2",
|
||||
"d1001,2018-10-03 14:38:15.000,12.60000,218,0.33000,'California.SanFrancisco',2",
|
||||
"d1001,2018-10-03 14:38:16.800,12.30000,221,0.31000,'California.SanFrancisco',2",
|
||||
"d1002,2018-10-03 14:38:16.650,10.30000,218,0.25000,'California.SanFrancisco',3",
|
||||
"d1003,2018-10-03 14:38:05.500,11.80000,221,0.28000,'California.LosAngeles',2",
|
||||
"d1003,2018-10-03 14:38:16.600,13.40000,223,0.29000,'California.LosAngeles',2",
|
||||
"d1004,2018-10-03 14:38:05.000,10.80000,223,0.29000,'California.LosAngeles',3",
|
||||
"d1004,2018-10-03 14:38:06.500,11.50000,221,0.35000,'California.LosAngeles',3"
|
||||
);
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -57,7 +57,7 @@ public class SubscribeDemo {
|
|||
properties.setProperty(TMQConstants.ENABLE_AUTO_COMMIT, "true");
|
||||
properties.setProperty(TMQConstants.GROUP_ID, "test");
|
||||
properties.setProperty(TMQConstants.VALUE_DESERIALIZER,
|
||||
"com.taosdata.jdbc.MetersDeserializer");
|
||||
"com.taos.example.MetersDeserializer");
|
||||
|
||||
// poll data
|
||||
try (TaosConsumer<Meters> consumer = new TaosConsumer<>(properties)) {
|
||||
|
|
|
|||
|
|
@ -0,0 +1,63 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
import java.sql.*;
|
||||
|
||||
/**
|
||||
* Prepare target database.
|
||||
* Count total records in database periodically so that we can estimate the writing speed.
|
||||
*/
|
||||
public class DataBaseMonitor {
|
||||
private Connection conn;
|
||||
private Statement stmt;
|
||||
|
||||
public DataBaseMonitor init() throws SQLException {
|
||||
if (conn == null) {
|
||||
String jdbcURL = System.getenv("TDENGINE_JDBC_URL");
|
||||
conn = DriverManager.getConnection(jdbcURL);
|
||||
stmt = conn.createStatement();
|
||||
}
|
||||
return this;
|
||||
}
|
||||
|
||||
public void close() {
|
||||
try {
|
||||
stmt.close();
|
||||
} catch (SQLException e) {
|
||||
}
|
||||
try {
|
||||
conn.close();
|
||||
} catch (SQLException e) {
|
||||
}
|
||||
}
|
||||
|
||||
public void prepareDatabase() throws SQLException {
|
||||
stmt.execute("DROP DATABASE IF EXISTS test");
|
||||
stmt.execute("CREATE DATABASE test");
|
||||
stmt.execute("CREATE STABLE test.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) TAGS (location BINARY(64), groupId INT)");
|
||||
}
|
||||
|
||||
public Long count() throws SQLException {
|
||||
if (!stmt.isClosed()) {
|
||||
ResultSet result = stmt.executeQuery("SELECT count(*) from test.meters");
|
||||
result.next();
|
||||
return result.getLong(1);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* show test.stables;
|
||||
*
|
||||
* name | created_time | columns | tags | tables |
|
||||
* ============================================================================================
|
||||
* meters | 2022-07-20 08:39:30.902 | 4 | 2 | 620000 |
|
||||
*/
|
||||
public Long getTableCount() throws SQLException {
|
||||
if (!stmt.isClosed()) {
|
||||
ResultSet result = stmt.executeQuery("show test.stables");
|
||||
result.next();
|
||||
return result.getLong(5);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,70 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.sql.*;
|
||||
import java.util.ArrayList;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.ArrayBlockingQueue;
|
||||
import java.util.concurrent.BlockingQueue;
|
||||
|
||||
|
||||
public class FastWriteExample {
|
||||
final static Logger logger = LoggerFactory.getLogger(FastWriteExample.class);
|
||||
|
||||
final static int taskQueueCapacity = 1000000;
|
||||
final static List<BlockingQueue<String>> taskQueues = new ArrayList<>();
|
||||
final static List<ReadTask> readTasks = new ArrayList<>();
|
||||
final static List<WriteTask> writeTasks = new ArrayList<>();
|
||||
final static DataBaseMonitor databaseMonitor = new DataBaseMonitor();
|
||||
|
||||
public static void stopAll() {
|
||||
logger.info("shutting down");
|
||||
readTasks.forEach(task -> task.stop());
|
||||
writeTasks.forEach(task -> task.stop());
|
||||
databaseMonitor.close();
|
||||
}
|
||||
|
||||
public static void main(String[] args) throws InterruptedException, SQLException {
|
||||
int readTaskCount = args.length > 0 ? Integer.parseInt(args[0]) : 1;
|
||||
int writeTaskCount = args.length > 1 ? Integer.parseInt(args[1]) : 3;
|
||||
int tableCount = args.length > 2 ? Integer.parseInt(args[2]) : 1000;
|
||||
int maxBatchSize = args.length > 3 ? Integer.parseInt(args[3]) : 3000;
|
||||
|
||||
logger.info("readTaskCount={}, writeTaskCount={} tableCount={} maxBatchSize={}",
|
||||
readTaskCount, writeTaskCount, tableCount, maxBatchSize);
|
||||
|
||||
databaseMonitor.init().prepareDatabase();
|
||||
|
||||
// Create task queues, whiting tasks and start writing threads.
|
||||
for (int i = 0; i < writeTaskCount; ++i) {
|
||||
BlockingQueue<String> queue = new ArrayBlockingQueue<>(taskQueueCapacity);
|
||||
taskQueues.add(queue);
|
||||
WriteTask task = new WriteTask(queue, maxBatchSize);
|
||||
Thread t = new Thread(task);
|
||||
t.setName("WriteThread-" + i);
|
||||
t.start();
|
||||
}
|
||||
|
||||
// create reading tasks and start reading threads
|
||||
int tableCountPerTask = tableCount / readTaskCount;
|
||||
for (int i = 0; i < readTaskCount; ++i) {
|
||||
ReadTask task = new ReadTask(i, taskQueues, tableCountPerTask);
|
||||
Thread t = new Thread(task);
|
||||
t.setName("ReadThread-" + i);
|
||||
t.start();
|
||||
}
|
||||
|
||||
Runtime.getRuntime().addShutdownHook(new Thread(FastWriteExample::stopAll));
|
||||
|
||||
long lastCount = 0;
|
||||
while (true) {
|
||||
Thread.sleep(10000);
|
||||
long numberOfTable = databaseMonitor.getTableCount();
|
||||
long count = databaseMonitor.count();
|
||||
logger.info("numberOfTable={} count={} speed={}", numberOfTable, count, (count - lastCount) / 10);
|
||||
lastCount = count;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,53 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
import java.util.Iterator;
|
||||
|
||||
/**
|
||||
* Generate test data
|
||||
*/
|
||||
class MockDataSource implements Iterator {
|
||||
private String tbNamePrefix;
|
||||
private int tableCount;
|
||||
private long maxRowsPerTable = 1000000000L;
|
||||
|
||||
// 100 milliseconds between two neighbouring rows.
|
||||
long startMs = System.currentTimeMillis() - maxRowsPerTable * 100;
|
||||
private int currentRow = 0;
|
||||
private int currentTbId = -1;
|
||||
|
||||
// mock values
|
||||
String[] location = {"LosAngeles", "SanDiego", "Hollywood", "Compton", "San Francisco"};
|
||||
float[] current = {8.8f, 10.7f, 9.9f, 8.9f, 9.4f};
|
||||
int[] voltage = {119, 116, 111, 113, 118};
|
||||
float[] phase = {0.32f, 0.34f, 0.33f, 0.329f, 0.141f};
|
||||
|
||||
public MockDataSource(String tbNamePrefix, int tableCount) {
|
||||
this.tbNamePrefix = tbNamePrefix;
|
||||
this.tableCount = tableCount;
|
||||
}
|
||||
|
||||
@Override
|
||||
public boolean hasNext() {
|
||||
currentTbId += 1;
|
||||
if (currentTbId == tableCount) {
|
||||
currentTbId = 0;
|
||||
currentRow += 1;
|
||||
}
|
||||
return currentRow < maxRowsPerTable;
|
||||
}
|
||||
|
||||
@Override
|
||||
public String next() {
|
||||
long ts = startMs + 100 * currentRow;
|
||||
int groupId = currentTbId % 5 == 0 ? currentTbId / 5 : currentTbId / 5 + 1;
|
||||
StringBuilder sb = new StringBuilder(tbNamePrefix + "_" + currentTbId + ","); // tbName
|
||||
sb.append(ts).append(','); // ts
|
||||
sb.append(current[currentRow % 5]).append(','); // current
|
||||
sb.append(voltage[currentRow % 5]).append(','); // voltage
|
||||
sb.append(phase[currentRow % 5]).append(','); // phase
|
||||
sb.append(location[currentRow % 5]).append(','); // location
|
||||
sb.append(groupId); // groupID
|
||||
|
||||
return sb.toString();
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.Iterator;
|
||||
import java.util.List;
|
||||
import java.util.concurrent.BlockingQueue;
|
||||
|
||||
class ReadTask implements Runnable {
|
||||
private final static Logger logger = LoggerFactory.getLogger(ReadTask.class);
|
||||
private final int taskId;
|
||||
private final List<BlockingQueue<String>> taskQueues;
|
||||
private final int queueCount;
|
||||
private final int tableCount;
|
||||
private boolean active = true;
|
||||
|
||||
public ReadTask(int readTaskId, List<BlockingQueue<String>> queues, int tableCount) {
|
||||
this.taskId = readTaskId;
|
||||
this.taskQueues = queues;
|
||||
this.queueCount = queues.size();
|
||||
this.tableCount = tableCount;
|
||||
}
|
||||
|
||||
/**
|
||||
* Assign data received to different queues.
|
||||
* Here we use the suffix number in table name.
|
||||
* You are expected to define your own rule in practice.
|
||||
*
|
||||
* @param line record received
|
||||
* @return which queue to use
|
||||
*/
|
||||
public int getQueueId(String line) {
|
||||
String tbName = line.substring(0, line.indexOf(',')); // For example: tb1_101
|
||||
String suffixNumber = tbName.split("_")[1];
|
||||
return Integer.parseInt(suffixNumber) % this.queueCount;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() {
|
||||
logger.info("started");
|
||||
Iterator<String> it = new MockDataSource("tb" + this.taskId, tableCount);
|
||||
try {
|
||||
while (it.hasNext() && active) {
|
||||
String line = it.next();
|
||||
int queueId = getQueueId(line);
|
||||
taskQueues.get(queueId).put(line);
|
||||
}
|
||||
} catch (Exception e) {
|
||||
logger.error("Read Task Error", e);
|
||||
}
|
||||
}
|
||||
|
||||
public void stop() {
|
||||
logger.info("stop");
|
||||
this.active = false;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,205 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.sql.*;
|
||||
import java.util.HashMap;
|
||||
import java.util.Map;
|
||||
|
||||
/**
|
||||
* A helper class encapsulate the logic of writing using SQL.
|
||||
* <p>
|
||||
* The main interfaces are two methods:
|
||||
* <ol>
|
||||
* <li>{@link SQLWriter#processLine}, which receive raw lines from WriteTask and group them by table names.</li>
|
||||
* <li>{@link SQLWriter#flush}, which assemble INSERT statement and execute it.</li>
|
||||
* </ol>
|
||||
* <p>
|
||||
* There is a technical skill worth mentioning: we create table as needed when "table does not exist" error occur instead of creating table automatically using syntax "INSET INTO tb USING stb".
|
||||
* This ensure that checking table existence is a one-time-only operation.
|
||||
* </p>
|
||||
*
|
||||
* </p>
|
||||
*/
|
||||
public class SQLWriter {
|
||||
final static Logger logger = LoggerFactory.getLogger(SQLWriter.class);
|
||||
|
||||
private Connection conn;
|
||||
private Statement stmt;
|
||||
|
||||
/**
|
||||
* current number of buffered records
|
||||
*/
|
||||
private int bufferedCount = 0;
|
||||
/**
|
||||
* Maximum number of buffered records.
|
||||
* Flush action will be triggered if bufferedCount reached this value,
|
||||
*/
|
||||
private int maxBatchSize;
|
||||
|
||||
|
||||
/**
|
||||
* Maximum SQL length.
|
||||
*/
|
||||
private int maxSQLLength;
|
||||
|
||||
/**
|
||||
* Map from table name to column values. For example:
|
||||
* "tb001" -> "(1648432611249,2.1,114,0.09) (1648432611250,2.2,135,0.2)"
|
||||
*/
|
||||
private Map<String, String> tbValues = new HashMap<>();
|
||||
|
||||
/**
|
||||
* Map from table name to tag values in the same order as creating stable.
|
||||
* Used for creating table.
|
||||
*/
|
||||
private Map<String, String> tbTags = new HashMap<>();
|
||||
|
||||
public SQLWriter(int maxBatchSize) {
|
||||
this.maxBatchSize = maxBatchSize;
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Get Database Connection
|
||||
*
|
||||
* @return Connection
|
||||
* @throws SQLException
|
||||
*/
|
||||
private static Connection getConnection() throws SQLException {
|
||||
String jdbcURL = System.getenv("TDENGINE_JDBC_URL");
|
||||
return DriverManager.getConnection(jdbcURL);
|
||||
}
|
||||
|
||||
/**
|
||||
* Create Connection and Statement
|
||||
*
|
||||
* @throws SQLException
|
||||
*/
|
||||
public void init() throws SQLException {
|
||||
conn = getConnection();
|
||||
stmt = conn.createStatement();
|
||||
stmt.execute("use test");
|
||||
ResultSet rs = stmt.executeQuery("show variables");
|
||||
while (rs.next()) {
|
||||
String configName = rs.getString(1);
|
||||
if ("maxSQLLength".equals(configName)) {
|
||||
maxSQLLength = Integer.parseInt(rs.getString(2));
|
||||
logger.info("maxSQLLength={}", maxSQLLength);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Convert raw data to SQL fragments, group them by table name and cache them in a HashMap.
|
||||
* Trigger writing when number of buffered records reached maxBachSize.
|
||||
*
|
||||
* @param line raw data get from task queue in format: tbName,ts,current,voltage,phase,location,groupId
|
||||
*/
|
||||
public void processLine(String line) throws SQLException {
|
||||
bufferedCount += 1;
|
||||
int firstComma = line.indexOf(',');
|
||||
String tbName = line.substring(0, firstComma);
|
||||
int lastComma = line.lastIndexOf(',');
|
||||
int secondLastComma = line.lastIndexOf(',', lastComma - 1);
|
||||
String value = "(" + line.substring(firstComma + 1, secondLastComma) + ") ";
|
||||
if (tbValues.containsKey(tbName)) {
|
||||
tbValues.put(tbName, tbValues.get(tbName) + value);
|
||||
} else {
|
||||
tbValues.put(tbName, value);
|
||||
}
|
||||
if (!tbTags.containsKey(tbName)) {
|
||||
String location = line.substring(secondLastComma + 1, lastComma);
|
||||
String groupId = line.substring(lastComma + 1);
|
||||
String tagValues = "('" + location + "'," + groupId + ')';
|
||||
tbTags.put(tbName, tagValues);
|
||||
}
|
||||
if (bufferedCount == maxBatchSize) {
|
||||
flush();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
/**
|
||||
* Assemble INSERT statement using buffered SQL fragments in Map {@link SQLWriter#tbValues} and execute it.
|
||||
* In case of "Table does not exit" exception, create all tables in the sql and retry the sql.
|
||||
*/
|
||||
public void flush() throws SQLException {
|
||||
StringBuilder sb = new StringBuilder("INSERT INTO ");
|
||||
for (Map.Entry<String, String> entry : tbValues.entrySet()) {
|
||||
String tableName = entry.getKey();
|
||||
String values = entry.getValue();
|
||||
String q = tableName + " values " + values + " ";
|
||||
if (sb.length() + q.length() > maxSQLLength) {
|
||||
executeSQL(sb.toString());
|
||||
logger.warn("increase maxSQLLength or decrease maxBatchSize to gain better performance");
|
||||
sb = new StringBuilder("INSERT INTO ");
|
||||
}
|
||||
sb.append(q);
|
||||
}
|
||||
executeSQL(sb.toString());
|
||||
tbValues.clear();
|
||||
bufferedCount = 0;
|
||||
}
|
||||
|
||||
private void executeSQL(String sql) throws SQLException {
|
||||
try {
|
||||
stmt.executeUpdate(sql);
|
||||
} catch (SQLException e) {
|
||||
// convert to error code defined in taoserror.h
|
||||
int errorCode = e.getErrorCode() & 0xffff;
|
||||
if (errorCode == 0x362 || errorCode == 0x218) {
|
||||
// Table does not exist
|
||||
createTables();
|
||||
executeSQL(sql);
|
||||
} else {
|
||||
logger.error("Execute SQL: {}", sql);
|
||||
throw e;
|
||||
}
|
||||
} catch (Throwable throwable) {
|
||||
logger.error("Execute SQL: {}", sql);
|
||||
throw throwable;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Create tables in batch using syntax:
|
||||
* <p>
|
||||
* CREATE TABLE [IF NOT EXISTS] tb_name1 USING stb_name TAGS (tag_value1, ...) [IF NOT EXISTS] tb_name2 USING stb_name TAGS (tag_value2, ...) ...;
|
||||
* </p>
|
||||
*/
|
||||
private void createTables() throws SQLException {
|
||||
StringBuilder sb = new StringBuilder("CREATE TABLE ");
|
||||
for (String tbName : tbValues.keySet()) {
|
||||
String tagValues = tbTags.get(tbName);
|
||||
sb.append("IF NOT EXISTS ").append(tbName).append(" USING meters TAGS ").append(tagValues).append(" ");
|
||||
}
|
||||
String sql = sb.toString();
|
||||
try {
|
||||
stmt.executeUpdate(sql);
|
||||
} catch (Throwable throwable) {
|
||||
logger.error("Execute SQL: {}", sql);
|
||||
throw throwable;
|
||||
}
|
||||
}
|
||||
|
||||
public boolean hasBufferedValues() {
|
||||
return bufferedCount > 0;
|
||||
}
|
||||
|
||||
public int getBufferedCount() {
|
||||
return bufferedCount;
|
||||
}
|
||||
|
||||
public void close() {
|
||||
try {
|
||||
stmt.close();
|
||||
} catch (SQLException e) {
|
||||
}
|
||||
try {
|
||||
conn.close();
|
||||
} catch (SQLException e) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,4 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
public class StmtWriter {
|
||||
}
|
||||
|
|
@ -0,0 +1,58 @@
|
|||
package com.taos.example.highvolume;
|
||||
|
||||
import org.slf4j.Logger;
|
||||
import org.slf4j.LoggerFactory;
|
||||
|
||||
import java.util.concurrent.BlockingQueue;
|
||||
|
||||
class WriteTask implements Runnable {
|
||||
private final static Logger logger = LoggerFactory.getLogger(WriteTask.class);
|
||||
private final int maxBatchSize;
|
||||
|
||||
// the queue from which this writing task get raw data.
|
||||
private final BlockingQueue<String> queue;
|
||||
|
||||
// A flag indicate whether to continue.
|
||||
private boolean active = true;
|
||||
|
||||
public WriteTask(BlockingQueue<String> taskQueue, int maxBatchSize) {
|
||||
this.queue = taskQueue;
|
||||
this.maxBatchSize = maxBatchSize;
|
||||
}
|
||||
|
||||
@Override
|
||||
public void run() {
|
||||
logger.info("started");
|
||||
String line = null; // data getting from the queue just now.
|
||||
SQLWriter writer = new SQLWriter(maxBatchSize);
|
||||
try {
|
||||
writer.init();
|
||||
while (active) {
|
||||
line = queue.poll();
|
||||
if (line != null) {
|
||||
// parse raw data and buffer the data.
|
||||
writer.processLine(line);
|
||||
} else if (writer.hasBufferedValues()) {
|
||||
// write data immediately if no more data in the queue
|
||||
writer.flush();
|
||||
} else {
|
||||
// sleep a while to avoid high CPU usage if no more data in the queue and no buffered records, .
|
||||
Thread.sleep(100);
|
||||
}
|
||||
}
|
||||
if (writer.hasBufferedValues()) {
|
||||
writer.flush();
|
||||
}
|
||||
} catch (Exception e) {
|
||||
String msg = String.format("line=%s, bufferedCount=%s", line, writer.getBufferedCount());
|
||||
logger.error(msg, e);
|
||||
} finally {
|
||||
writer.close();
|
||||
}
|
||||
}
|
||||
|
||||
public void stop() {
|
||||
logger.info("stop");
|
||||
this.active = false;
|
||||
}
|
||||
}
|
||||
|
|
@ -23,16 +23,16 @@ public class TestAll {
|
|||
String jdbcUrl = "jdbc:TAOS://localhost:6030?user=root&password=taosdata";
|
||||
try (Connection conn = DriverManager.getConnection(jdbcUrl)) {
|
||||
try (Statement stmt = conn.createStatement()) {
|
||||
String sql = "INSERT INTO power.d1001 USING power.meters TAGS(California.SanFrancisco, 2) VALUES('2018-10-03 14:38:05.000',10.30000,219,0.31000)\n" +
|
||||
" power.d1001 USING power.meters TAGS(California.SanFrancisco, 2) VALUES('2018-10-03 15:38:15.000',12.60000,218,0.33000)\n" +
|
||||
" power.d1001 USING power.meters TAGS(California.SanFrancisco, 2) VALUES('2018-10-03 15:38:16.800',12.30000,221,0.31000)\n" +
|
||||
" power.d1002 USING power.meters TAGS(California.SanFrancisco, 3) VALUES('2018-10-03 15:38:16.650',10.30000,218,0.25000)\n" +
|
||||
" power.d1003 USING power.meters TAGS(California.LosAngeles, 2) VALUES('2018-10-03 15:38:05.500',11.80000,221,0.28000)\n" +
|
||||
" power.d1003 USING power.meters TAGS(California.LosAngeles, 2) VALUES('2018-10-03 15:38:16.600',13.40000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS(California.LosAngeles, 3) VALUES('2018-10-03 15:38:05.000',10.80000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS(California.LosAngeles, 3) VALUES('2018-10-03 15:38:06.000',10.80000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS(California.LosAngeles, 3) VALUES('2018-10-03 15:38:07.000',10.80000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS(California.LosAngeles, 3) VALUES('2018-10-03 15:38:08.500',11.50000,221,0.35000)";
|
||||
String sql = "INSERT INTO power.d1001 USING power.meters TAGS('California.SanFrancisco', 2) VALUES('2018-10-03 14:38:05.000',10.30000,219,0.31000)\n" +
|
||||
" power.d1001 USING power.meters TAGS('California.SanFrancisco', 2) VALUES('2018-10-03 15:38:15.000',12.60000,218,0.33000)\n" +
|
||||
" power.d1001 USING power.meters TAGS('California.SanFrancisco', 2) VALUES('2018-10-03 15:38:16.800',12.30000,221,0.31000)\n" +
|
||||
" power.d1002 USING power.meters TAGS('California.SanFrancisco', 3) VALUES('2018-10-03 15:38:16.650',10.30000,218,0.25000)\n" +
|
||||
" power.d1003 USING power.meters TAGS('California.LosAngeles', 2) VALUES('2018-10-03 15:38:05.500',11.80000,221,0.28000)\n" +
|
||||
" power.d1003 USING power.meters TAGS('California.LosAngeles', 2) VALUES('2018-10-03 15:38:16.600',13.40000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS('California.LosAngeles', 3) VALUES('2018-10-03 15:38:05.000',10.80000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS('California.LosAngeles', 3) VALUES('2018-10-03 15:38:06.000',10.80000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS('California.LosAngeles', 3) VALUES('2018-10-03 15:38:07.000',10.80000,223,0.29000)\n" +
|
||||
" power.d1004 USING power.meters TAGS('California.LosAngeles', 3) VALUES('2018-10-03 15:38:08.500',11.50000,221,0.35000)";
|
||||
|
||||
stmt.execute(sql);
|
||||
}
|
||||
|
|
|
|||
|
|
@ -11,10 +11,10 @@ conn: taos.TaosConnection = taos.connect(host="localhost",
|
|||
server_version = conn.server_info
|
||||
print("server_version", server_version)
|
||||
client_version = conn.client_info
|
||||
print("client_version", client_version) # 2.4.0.16
|
||||
print("client_version", client_version) # 3.0.0.0
|
||||
|
||||
conn.close()
|
||||
|
||||
# possible output:
|
||||
# 2.4.0.16
|
||||
# 2.4.0.16
|
||||
# 3.0.0.0
|
||||
# 3.0.0.0
|
||||
|
|
|
|||
|
|
@ -0,0 +1,180 @@
|
|||
# install dependencies:
|
||||
# recommend python >= 3.8
|
||||
# pip3 install faster-fifo
|
||||
#
|
||||
|
||||
import logging
|
||||
import math
|
||||
import sys
|
||||
import time
|
||||
import os
|
||||
from multiprocessing import Process
|
||||
from faster_fifo import Queue
|
||||
from mockdatasource import MockDataSource
|
||||
from queue import Empty
|
||||
from typing import List
|
||||
|
||||
logging.basicConfig(stream=sys.stdout, level=logging.DEBUG, format="%(asctime)s [%(name)s] - %(message)s")
|
||||
|
||||
READ_TASK_COUNT = 1
|
||||
WRITE_TASK_COUNT = 1
|
||||
TABLE_COUNT = 1000
|
||||
QUEUE_SIZE = 1000000
|
||||
MAX_BATCH_SIZE = 3000
|
||||
|
||||
read_processes = []
|
||||
write_processes = []
|
||||
|
||||
|
||||
def get_connection():
|
||||
"""
|
||||
If variable TDENGINE_FIRST_EP is provided then it will be used. If not, firstEP in /etc/taos/taos.cfg will be used.
|
||||
You can also override the default username and password by supply variable TDENGINE_USER and TDENGINE_PASSWORD
|
||||
"""
|
||||
import taos
|
||||
firstEP = os.environ.get("TDENGINE_FIRST_EP")
|
||||
if firstEP:
|
||||
host, port = firstEP.split(":")
|
||||
else:
|
||||
host, port = None, 0
|
||||
user = os.environ.get("TDENGINE_USER", "root")
|
||||
password = os.environ.get("TDENGINE_PASSWORD", "taosdata")
|
||||
return taos.connect(host=host, port=int(port), user=user, password=password)
|
||||
|
||||
|
||||
# ANCHOR: read
|
||||
|
||||
def run_read_task(task_id: int, task_queues: List[Queue]):
|
||||
table_count_per_task = TABLE_COUNT // READ_TASK_COUNT
|
||||
data_source = MockDataSource(f"tb{task_id}", table_count_per_task)
|
||||
try:
|
||||
for batch in data_source:
|
||||
for table_id, rows in batch:
|
||||
# hash data to different queue
|
||||
i = table_id % len(task_queues)
|
||||
# block putting forever when the queue is full
|
||||
task_queues[i].put_many(rows, block=True, timeout=-1)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
|
||||
|
||||
# ANCHOR_END: read
|
||||
|
||||
# ANCHOR: write
|
||||
def run_write_task(task_id: int, queue: Queue):
|
||||
from sql_writer import SQLWriter
|
||||
log = logging.getLogger(f"WriteTask-{task_id}")
|
||||
writer = SQLWriter(get_connection)
|
||||
lines = None
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
# get as many as possible
|
||||
lines = queue.get_many(block=False, max_messages_to_get=MAX_BATCH_SIZE)
|
||||
writer.process_lines(lines)
|
||||
except Empty:
|
||||
time.sleep(0.01)
|
||||
except KeyboardInterrupt:
|
||||
pass
|
||||
except BaseException as e:
|
||||
log.debug(f"lines={lines}")
|
||||
raise e
|
||||
|
||||
|
||||
# ANCHOR_END: write
|
||||
|
||||
def set_global_config():
|
||||
argc = len(sys.argv)
|
||||
if argc > 1:
|
||||
global READ_TASK_COUNT
|
||||
READ_TASK_COUNT = int(sys.argv[1])
|
||||
if argc > 2:
|
||||
global WRITE_TASK_COUNT
|
||||
WRITE_TASK_COUNT = int(sys.argv[2])
|
||||
if argc > 3:
|
||||
global TABLE_COUNT
|
||||
TABLE_COUNT = int(sys.argv[3])
|
||||
if argc > 4:
|
||||
global QUEUE_SIZE
|
||||
QUEUE_SIZE = int(sys.argv[4])
|
||||
if argc > 5:
|
||||
global MAX_BATCH_SIZE
|
||||
MAX_BATCH_SIZE = int(sys.argv[5])
|
||||
|
||||
|
||||
# ANCHOR: monitor
|
||||
def run_monitor_process():
|
||||
log = logging.getLogger("DataBaseMonitor")
|
||||
conn = get_connection()
|
||||
conn.execute("DROP DATABASE IF EXISTS test")
|
||||
conn.execute("CREATE DATABASE test")
|
||||
conn.execute("CREATE STABLE test.meters (ts TIMESTAMP, current FLOAT, voltage INT, phase FLOAT) "
|
||||
"TAGS (location BINARY(64), groupId INT)")
|
||||
|
||||
def get_count():
|
||||
res = conn.query("SELECT count(*) FROM test.meters")
|
||||
rows = res.fetch_all()
|
||||
return rows[0][0] if rows else 0
|
||||
|
||||
last_count = 0
|
||||
while True:
|
||||
time.sleep(10)
|
||||
count = get_count()
|
||||
log.info(f"count={count} speed={(count - last_count) / 10}")
|
||||
last_count = count
|
||||
|
||||
|
||||
# ANCHOR_END: monitor
|
||||
# ANCHOR: main
|
||||
def main():
|
||||
set_global_config()
|
||||
logging.info(f"READ_TASK_COUNT={READ_TASK_COUNT}, WRITE_TASK_COUNT={WRITE_TASK_COUNT}, "
|
||||
f"TABLE_COUNT={TABLE_COUNT}, QUEUE_SIZE={QUEUE_SIZE}, MAX_BATCH_SIZE={MAX_BATCH_SIZE}")
|
||||
|
||||
monitor_process = Process(target=run_monitor_process)
|
||||
monitor_process.start()
|
||||
time.sleep(3) # waiting for database ready.
|
||||
|
||||
task_queues: List[Queue] = []
|
||||
# create task queues
|
||||
for i in range(WRITE_TASK_COUNT):
|
||||
queue = Queue(max_size_bytes=QUEUE_SIZE)
|
||||
task_queues.append(queue)
|
||||
|
||||
# create write processes
|
||||
for i in range(WRITE_TASK_COUNT):
|
||||
p = Process(target=run_write_task, args=(i, task_queues[i]))
|
||||
p.start()
|
||||
logging.debug(f"WriteTask-{i} started with pid {p.pid}")
|
||||
write_processes.append(p)
|
||||
|
||||
# create read processes
|
||||
for i in range(READ_TASK_COUNT):
|
||||
queues = assign_queues(i, task_queues)
|
||||
p = Process(target=run_read_task, args=(i, queues))
|
||||
p.start()
|
||||
logging.debug(f"ReadTask-{i} started with pid {p.pid}")
|
||||
read_processes.append(p)
|
||||
|
||||
try:
|
||||
monitor_process.join()
|
||||
except KeyboardInterrupt:
|
||||
monitor_process.terminate()
|
||||
[p.terminate() for p in read_processes]
|
||||
[p.terminate() for p in write_processes]
|
||||
[q.close() for q in task_queues]
|
||||
|
||||
|
||||
def assign_queues(read_task_id, task_queues):
|
||||
"""
|
||||
Compute target queues for a specific read task.
|
||||
"""
|
||||
ratio = WRITE_TASK_COUNT / READ_TASK_COUNT
|
||||
from_index = math.floor(read_task_id * ratio)
|
||||
end_index = math.ceil((read_task_id + 1) * ratio)
|
||||
return task_queues[from_index:end_index]
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
# ANCHOR_END: main
|
||||
|
|
@ -0,0 +1,49 @@
|
|||
import time
|
||||
|
||||
|
||||
class MockDataSource:
|
||||
samples = [
|
||||
"8.8,119,0.32,LosAngeles,0",
|
||||
"10.7,116,0.34,SanDiego,1",
|
||||
"9.9,111,0.33,Hollywood,2",
|
||||
"8.9,113,0.329,Compton,3",
|
||||
"9.4,118,0.141,San Francisco,4"
|
||||
]
|
||||
|
||||
def __init__(self, tb_name_prefix, table_count):
|
||||
self.table_name_prefix = tb_name_prefix + "_"
|
||||
self.table_count = table_count
|
||||
self.max_rows = 10000000
|
||||
self.current_ts = round(time.time() * 1000) - self.max_rows * 100
|
||||
# [(tableId, tableName, values),]
|
||||
self.data = self._init_data()
|
||||
|
||||
def _init_data(self):
|
||||
lines = self.samples * (self.table_count // 5 + 1)
|
||||
data = []
|
||||
for i in range(self.table_count):
|
||||
table_name = self.table_name_prefix + str(i)
|
||||
data.append((i, table_name, lines[i])) # tableId, row
|
||||
return data
|
||||
|
||||
def __iter__(self):
|
||||
self.row = 0
|
||||
return self
|
||||
|
||||
def __next__(self):
|
||||
"""
|
||||
next 1000 rows for each table.
|
||||
return: {tableId:[row,...]}
|
||||
"""
|
||||
# generate 1000 timestamps
|
||||
ts = []
|
||||
for _ in range(1000):
|
||||
self.current_ts += 100
|
||||
ts.append(str(self.current_ts))
|
||||
# add timestamp to each row
|
||||
# [(tableId, ["tableName,ts,current,voltage,phase,location,groupId"])]
|
||||
result = []
|
||||
for table_id, table_name, values in self.data:
|
||||
rows = [table_name + ',' + t + ',' + values for t in ts]
|
||||
result.append((table_id, rows))
|
||||
return result
|
||||
|
|
@ -0,0 +1,90 @@
|
|||
import logging
|
||||
import taos
|
||||
|
||||
|
||||
class SQLWriter:
|
||||
log = logging.getLogger("SQLWriter")
|
||||
|
||||
def __init__(self, get_connection_func):
|
||||
self._tb_values = {}
|
||||
self._tb_tags = {}
|
||||
self._conn = get_connection_func()
|
||||
self._max_sql_length = self.get_max_sql_length()
|
||||
self._conn.execute("USE test")
|
||||
|
||||
def get_max_sql_length(self):
|
||||
rows = self._conn.query("SHOW variables").fetch_all()
|
||||
for r in rows:
|
||||
name = r[0]
|
||||
if name == "maxSQLLength":
|
||||
return int(r[1])
|
||||
return 1024 * 1024
|
||||
|
||||
def process_lines(self, lines: str):
|
||||
"""
|
||||
:param lines: [[tbName,ts,current,voltage,phase,location,groupId]]
|
||||
"""
|
||||
for line in lines:
|
||||
ps = line.split(",")
|
||||
table_name = ps[0]
|
||||
value = '(' + ",".join(ps[1:-2]) + ') '
|
||||
if table_name in self._tb_values:
|
||||
self._tb_values[table_name] += value
|
||||
else:
|
||||
self._tb_values[table_name] = value
|
||||
|
||||
if table_name not in self._tb_tags:
|
||||
location = ps[-2]
|
||||
group_id = ps[-1]
|
||||
tag_value = f"('{location}',{group_id})"
|
||||
self._tb_tags[table_name] = tag_value
|
||||
self.flush()
|
||||
|
||||
def flush(self):
|
||||
"""
|
||||
Assemble INSERT statement and execute it.
|
||||
When the sql length grows close to MAX_SQL_LENGTH, the sql will be executed immediately, and a new INSERT statement will be created.
|
||||
In case of "Table does not exit" exception, tables in the sql will be created and the sql will be re-executed.
|
||||
"""
|
||||
sql = "INSERT INTO "
|
||||
sql_len = len(sql)
|
||||
buf = []
|
||||
for tb_name, values in self._tb_values.items():
|
||||
q = tb_name + " VALUES " + values
|
||||
if sql_len + len(q) >= self._max_sql_length:
|
||||
sql += " ".join(buf)
|
||||
self.execute_sql(sql)
|
||||
sql = "INSERT INTO "
|
||||
sql_len = len(sql)
|
||||
buf = []
|
||||
buf.append(q)
|
||||
sql_len += len(q)
|
||||
sql += " ".join(buf)
|
||||
self.execute_sql(sql)
|
||||
self._tb_values.clear()
|
||||
|
||||
def execute_sql(self, sql):
|
||||
try:
|
||||
self._conn.execute(sql)
|
||||
except taos.Error as e:
|
||||
error_code = e.errno & 0xffff
|
||||
# Table does not exit
|
||||
if error_code == 9731:
|
||||
self.create_tables()
|
||||
else:
|
||||
self.log.error("Execute SQL: %s", sql)
|
||||
raise e
|
||||
except BaseException as baseException:
|
||||
self.log.error("Execute SQL: %s", sql)
|
||||
raise baseException
|
||||
|
||||
def create_tables(self):
|
||||
sql = "CREATE TABLE "
|
||||
for tb in self._tb_values.keys():
|
||||
tag_values = self._tb_tags[tb]
|
||||
sql += "IF NOT EXISTS " + tb + " USING meters TAGS " + tag_values + " "
|
||||
try:
|
||||
self._conn.execute(sql)
|
||||
except BaseException as e:
|
||||
self.log.error("Execute SQL: %s", sql)
|
||||
raise e
|
||||
|
|
@ -1,5 +1,6 @@
|
|||
---
|
||||
title: 产品简介
|
||||
description: 简要介绍 TDengine 的主要功能
|
||||
toc_max_heading_level: 2
|
||||
---
|
||||
|
||||
|
|
@ -22,8 +23,8 @@ TDengine的主要功能如下：
|
|||
9. 提供[命令行程序](../reference/taos-shell)，便于管理集群，检查系统状态，做即席查询
|
||||
10. 提供多种数据的[导入](../operation/import)、[导出](../operation/export)
|
||||
11. 支持对[TDengine 集群本身的监控](../operation/monitor)
|
||||
12. 提供 [C/C++](../reference/connector/cpp), [Java](../reference/connector/java), [Python](../reference/connector/python), [Go](../reference/connector/go), [Rust](../reference/connector/rust), [Node.js](../reference/connector/node) 等多种编程语言的[连接器](../reference/connector/)
|
||||
13. 支持 [REST 接口](../reference/rest-api/)
|
||||
12. 提供各种语言的[连接器](../connector): 如 C/C++, Java, Go, Node.JS, Rust, Python, C# 等
|
||||
13. 支持 [REST 接口](../connector/rest-api/)
|
||||
14. 支持与[ Grafana 无缝集成](../third-party/grafana)
|
||||
15. 支持与 Google Data Studio 无缝集成
|
||||
16. 支持 [Kubernetes 部署](../deployment/k8s)
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
sidebar_label: 基本概念
|
||||
title: 数据模型和基本概念
|
||||
description: TDengine 的数据模型和基本概念
|
||||
---
|
||||
|
||||
为了便于解释基本概念，便于撰写示例程序，整个 TDengine 文档以智能电表作为典型时序数据场景。假设每个智能电表采集电流、电压、相位三个量，有多个智能电表，每个电表有位置 location 和分组 group ID 的静态属性. 其采集的数据类似如下的表格：
|
||||
|
|
@ -104,15 +106,15 @@ title: 数据模型和基本概念
|
|||
|
||||
## 采集量 (Metric)
|
||||
|
||||
采集量是指传感器、设备或其他类型采集点采集的物理量，比如电流、电压、温度、压力、GPS 位置等，是随时间变化的，数据类型可以是整型、浮点型、布尔型，也可是字符串。随着时间的推移，存储的采集量的数据量越来越大。
|
||||
采集量是指传感器、设备或其他类型采集点采集的物理量，比如电流、电压、温度、压力、GPS 位置等，是随时间变化的，数据类型可以是整型、浮点型、布尔型，也可是字符串。随着时间的推移，存储的采集量的数据量越来越大。智能电表示例中的电流、电压、相位就是采集量。
|
||||
|
||||
## 标签 (Label/Tag)
|
||||
|
||||
标签是指传感器、设备或其他类型采集点的静态属性，不是随时间变化的，比如设备型号、颜色、设备的所在地等，数据类型可以是任何类型。虽然是静态的，但 TDengine 容许用户修改、删除或增加标签值。与采集量不一样的是，随时间的推移，存储的标签的数据量不会有什么变化。
|
||||
标签是指传感器、设备或其他类型采集点的静态属性，不是随时间变化的，比如设备型号、颜色、设备的所在地等，数据类型可以是任何类型。虽然是静态的，但 TDengine 容许用户修改、删除或增加标签值。与采集量不一样的是，随时间的推移，存储的标签的数据量不会有什么变化。智能电表示例中的location与groupId就是标签。
|
||||
|
||||
## 数据采集点 (Data Collection Point)
|
||||
|
||||
数据采集点是指按照预设时间周期或受事件触发采集物理量的硬件或软件。一个数据采集点可以采集一个或多个采集量，**但这些采集量都是同一时刻采集的，具有相同的时间戳**。对于复杂的设备，往往有多个数据采集点，每个数据采集点采集的周期都可能不一样，而且完全独立，不同步。比如对于一台汽车，有数据采集点专门采集 GPS 位置，有数据采集点专门采集发动机状态，有数据采集点专门采集车内的环境，这样一台汽车就有三个数据采集点。
|
||||
数据采集点是指按照预设时间周期或受事件触发采集物理量的硬件或软件。一个数据采集点可以采集一个或多个采集量，**但这些采集量都是同一时刻采集的，具有相同的时间戳**。对于复杂的设备，往往有多个数据采集点，每个数据采集点采集的周期都可能不一样，而且完全独立，不同步。比如对于一台汽车，有数据采集点专门采集 GPS 位置，有数据采集点专门采集发动机状态，有数据采集点专门采集车内的环境，这样一台汽车就有三个数据采集点。智能电表示例中的d1001, d1002, d1003, d1004等就是数据采集点。
|
||||
|
||||
## 表 (Table)
|
||||
|
||||
|
|
@ -131,13 +133,14 @@ TDengine 建议用数据采集点的名字（如上表中的 D1001）来做表
|
|||
|
||||
对于复杂的设备，比如汽车，它有多个数据采集点，那么就需要为一台汽车建立多张表。
|
||||
|
||||
|
||||
## 超级表 (STable)
|
||||
|
||||
由于一个数据采集点一张表，导致表的数量巨增，难以管理，而且应用经常需要做采集点之间的聚合操作，聚合的操作也变得复杂起来。为解决这个问题，TDengine 引入超级表（Super Table，简称为 STable）的概念。
|
||||
|
||||
超级表是指某一特定类型的数据采集点的集合。同一类型的数据采集点，其表的结构是完全一样的，但每个表（数据采集点）的静态属性（标签）是不一样的。描述一个超级表（某一特定类型的数据采集点的集合），除需要定义采集量的表结构之外，还需要定义其标签的 schema，标签的数据类型可以是整数、浮点数、字符串，标签可以有多个，可以事后增加、删除或修改。如果整个系统有 N 个不同类型的数据采集点，就需要建立 N 个超级表。
|
||||
|
||||
在 TDengine 的设计里，**表用来代表一个具体的数据采集点，超级表用来代表一组相同类型的数据采集点集合**。
|
||||
在 TDengine 的设计里，**表用来代表一个具体的数据采集点，超级表用来代表一组相同类型的数据采集点集合**。智能电表示例中，我们可以创建一个超级表meters.
|
||||
|
||||
## 子表 (Subtable)
|
||||
|
||||
|
|
@ -156,7 +159,9 @@ TDengine 建议用数据采集点的名字（如上表中的 D1001）来做表
|
|||
|
||||
查询既可以在表上进行，也可以在超级表上进行。针对超级表的查询，TDengine 将把所有子表中的数据视为一个整体数据集进行处理，会先把满足标签过滤条件的表从超级表中找出来，然后再扫描这些表的时序数据，进行聚合操作，这样需要扫描的数据集会大幅减少，从而显著提高查询的性能。本质上，TDengine 通过对超级表查询的支持，实现了多个同类数据采集点的高效聚合。
|
||||
|
||||
TDengine系统建议给一个数据采集点建表，需要通过超级表建表，而不是建普通表。
|
||||
TDengine系统建议给一个数据采集点建表，需要通过超级表建表，而不是建普通表。在智能电表的示例中，我们可以通过超级表meters创建子表d1001, d1002, d1003, d1004等。
|
||||
|
||||
为了更好地理解超级与子表的关系，可以参考下面关于智能电表数据模型的示意图。 
|
||||
|
||||
## 库 (database)
|
||||
|
||||
|
|
|
|||
Binary file not shown.
|
After Width: | Height: | Size: 33 KiB |
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
sidebar_label: Docker
|
||||
title: 通过 Docker 快速体验 TDengine
|
||||
description: 使用 Docker 快速体验 TDengine 的高效写入和查询
|
||||
---
|
||||
|
||||
本节首先介绍如何通过 Docker 快速体验 TDengine，然后介绍如何在 Docker 环境下体验 TDengine 的写入和查询功能。如果你不熟悉 Docker，请使用[安装包的方式快速体验](../../get-started/package/)。如果您希望为 TDengine 贡献代码或对内部技术实现感兴趣，请参考 [TDengine GitHub 主页](https://github.com/taosdata/TDengine) 下载源码构建和安装.
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
sidebar_label: 安装包
|
||||
title: 使用安装包立即开始
|
||||
description: 使用安装包快速体验 TDengine
|
||||
---
|
||||
|
||||
import Tabs from "@theme/Tabs";
|
||||
|
|
@ -9,7 +10,7 @@ import PkgListV3 from "/components/PkgListV3";
|
|||
|
||||
您可以[用 Docker 立即体验](../../get-started/docker/) TDengine。如果您希望对 TDengine 贡献代码或对内部实现感兴趣，请参考我们的 [TDengine GitHub 主页](https://github.com/taosdata/TDengine) 下载源码构建和安装.
|
||||
|
||||
TDengine 完整的软件包包括服务端（taosd）、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、应用驱动（taosc）、命令行程序 (CLI，taos) 和一些工具软件。目前 taosAdapter 仅在 Linux 系统上安装和运行，后续将支持 Windows、macOS 等系统。TDengine 除了提供多种语言的连接器之外，还通过 [taosAdapter](../../reference/taosadapter/) 提供 [RESTful 接口](../../reference/rest-api/)。
|
||||
TDengine 完整的软件包包括服务端（taosd）、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、应用驱动（taosc）、命令行程序 (CLI，taos) 和一些工具软件。目前 taosAdapter 仅在 Linux 系统上安装和运行，后续将支持 Windows、macOS 等系统。TDengine 除了提供多种语言的连接器之外，还通过 [taosAdapter](../../reference/taosadapter/) 提供 [RESTful 接口](../../connector/rest-api/)。
|
||||
|
||||
为方便使用，标准的服务端安装包包含了 taosd、taosAdapter、taosc、taos、taosdump、taosBenchmark、TDinsight 安装脚本和示例代码；如果您只需要用到服务端程序和客户端连接的 C/C++ 语言支持，也可以仅下载 lite 版本的安装包。
|
||||
|
||||
|
|
@ -67,13 +68,6 @@ install.sh 安装脚本在执行过程中，会通过命令行交互界面询问
|
|||
|
||||
</TabItem>
|
||||
|
||||
<TabItem label="Windows 安装" value="windows">
|
||||
|
||||
1. 从列表中下载获得 exe 安装程序；
|
||||
<PkgListV3 type={3}/>
|
||||
2. 运行可执行程序来安装 TDengine。
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="apt-get" label="apt-get">
|
||||
可以使用 apt-get 工具从官方仓库安装。
|
||||
|
||||
|
|
@ -102,11 +96,20 @@ sudo apt-get install tdengine
|
|||
:::tip
|
||||
apt-get 方式只适用于 Debian 或 Ubuntu 系统
|
||||
::::
|
||||
</TabItem>
|
||||
<TabItem label="Windows 安装" value="windows">
|
||||
|
||||
注意：目前 TDengine 在 Windows 平台上只支持 Windows server 2016/2019 和 Windows 10/11 系统版本。
|
||||
|
||||
1. 从列表中下载获得 exe 安装程序；
|
||||
<PkgListV3 type={3}/>
|
||||
2. 运行可执行程序来安装 TDengine。
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
:::info
|
||||
下载其他组件、最新 Beta 版及之前版本的安装包，请点击[发布历史页面](../../releases)
|
||||
下载其他组件、最新 Beta 版及之前版本的安装包，请点击[发布历史页面](../../releases/tdengine)
|
||||
:::
|
||||
|
||||
:::note
|
||||
|
|
|
|||
|
|
@ -1,17 +0,0 @@
|
|||
import PkgList from "/components/PkgList";
|
||||
|
||||
TDengine 的安装非常简单，从下载到安装成功仅仅只要几秒钟。
|
||||
|
||||
为方便使用，从 2.4.0.10 开始，标准的服务端安装包包含了 taos、taosd、taosAdapter、taosdump、taosBenchmark、TDinsight 安装脚本和示例代码；如果您只需要用到服务端程序和客户端连接的 C/C++ 语言支持，也可以仅下载 lite 版本的安装包。
|
||||
|
||||
在安装包格式上，我们提供 tar.gz, rpm 和 deb 格式，为企业客户提供 tar.gz 格式安装包，以方便在特定操作系统上使用。需要注意的是，rpm 和 deb 包不含 taosdump、taosBenchmark 和 TDinsight 安装脚本，这些工具需要通过安装 taosTool 包获得。
|
||||
|
||||
发布版本包括稳定版和 Beta 版，Beta 版含有更多新功能。正式上线或测试建议安装稳定版。您可以根据需要选择下载：
|
||||
|
||||
<PkgList type={0}/>
|
||||
|
||||
具体的安装方法，请参见[安装包的安装和卸载](/operation/pkg-install)。
|
||||
|
||||
下载其他组件、最新 Beta 版及之前版本的安装包，请点击[这里](https://www.taosdata.com/all-downloads)
|
||||
|
||||
查看 Release Notes, 请点击[这里](https://github.com/taosdata/TDengine/releases)
|
||||
|
|
@ -3,7 +3,7 @@ title: 立即开始
|
|||
description: '快速设置 TDengine 环境并体验其高效写入和查询'
|
||||
---
|
||||
|
||||
TDengine 完整的软件包包括服务端（taosd）、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、应用驱动（taosc）、命令行程序 (CLI，taos) 和一些工具软件。TDengine 除了提供多种语言的连接器之外，还通过 [taosAdapter](/reference/taosadapter) 提供 [RESTful 接口](/reference/rest-api)。
|
||||
TDengine 完整的软件包包括服务端（taosd）、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、应用驱动（taosc）、命令行程序 (CLI，taos) 和一些工具软件。TDengine 除了提供多种语言的连接器之外，还通过 [taosAdapter](../reference/taosadapter) 提供 [RESTful 接口](../connector/rest-api)。
|
||||
|
||||
本章主要介绍如何利用 Docker 或者安装包快速设置 TDengine 环境并体验其高效写入和查询。
|
||||
|
||||
|
|
|
|||
|
|
@ -12,4 +12,4 @@
|
|||
{{#include docs/examples/java/src/main/java/com/taos/example/WSConnectExample.java:main}}
|
||||
```
|
||||
|
||||
更多连接参数配置，参考[Java 连接器](/reference/connector/java)
|
||||
更多连接参数配置，参考[Java 连接器](../../connector/java)
|
||||
|
|
|
|||
|
|
@ -1,6 +1,6 @@
|
|||
---
|
||||
title: 建立连接
|
||||
description: "本节介绍如何使用连接器建立与 TDengine 的连接，给出连接器安装、连接的简单说明。"
|
||||
description: 使用连接器建立与 TDengine 的连接，以及连接器的安装和连接
|
||||
---
|
||||
|
||||
import Tabs from "@theme/Tabs";
|
||||
|
|
@ -14,10 +14,10 @@ import ConnCSNative from "./_connect_cs.mdx";
|
|||
import ConnC from "./_connect_c.mdx";
|
||||
import ConnR from "./_connect_r.mdx";
|
||||
import ConnPHP from "./_connect_php.mdx";
|
||||
import InstallOnWindows from "../../14-reference/03-connector/_linux_install.mdx";
|
||||
import InstallOnLinux from "../../14-reference/03-connector/_windows_install.mdx";
|
||||
import VerifyLinux from "../../14-reference/03-connector/_verify_linux.mdx";
|
||||
import VerifyWindows from "../../14-reference/03-connector/_verify_windows.mdx";
|
||||
import InstallOnWindows from "../../08-connector/_linux_install.mdx";
|
||||
import InstallOnLinux from "../../08-connector/_windows_install.mdx";
|
||||
import VerifyLinux from "../../08-connector/_verify_linux.mdx";
|
||||
import VerifyWindows from "../../08-connector/_verify_windows.mdx";
|
||||
|
||||
TDengine 提供了丰富的应用程序开发接口，为了便于用户快速开发自己的应用，TDengine 支持了多种编程语言的连接器，其中官方连接器包括支持 C/C++、Java、Python、Go、Node.js、C#、Rust、Lua（社区贡献）和 PHP （社区贡献）的连接器。这些连接器支持使用原生接口（taosc）和 REST 接口（部分语言暂不支持）连接 TDengine 集群。社区开发者也贡献了多个非官方连接器，例如 ADO.NET 连接器、Lua 连接器和 PHP 连接器。
|
||||
|
||||
|
|
@ -33,7 +33,7 @@ TDengine 提供了丰富的应用程序开发接口，为了便于用户快速
|
|||
关键不同点在于：
|
||||
|
||||
1. 使用 REST 连接，用户无需安装客户端驱动程序 taosc，具有跨平台易用的优势，但性能要下降 30%左右。
|
||||
2. 使用原生连接可以体验 TDengine 的全部功能，如[参数绑定接口](/reference/connector/cpp#参数绑定-api)、[订阅](/reference/connector/cpp#订阅和消费-api)等等。
|
||||
2. 使用原生连接可以体验 TDengine 的全部功能，如[参数绑定接口](../../connector/cpp/#参数绑定-api)、[订阅](../../connector/cpp/#订阅和消费-api)等等。
|
||||
|
||||
## 安装客户端驱动 taosc
|
||||
|
||||
|
|
@ -223,7 +223,7 @@ phpize && ./configure && make -j && make install
|
|||
**手动指定 TDengine 目录：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/2.4.0.0 && make -j && make install
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/3.0.0.0 && make -j && make install
|
||||
```
|
||||
|
||||
> `--with-tdengine-dir=` 后跟上 TDengine 目录。
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
sidebar_label: 数据建模
|
||||
title: TDengine 数据建模
|
||||
description: TDengine 中如何建立数据模型
|
||||
---
|
||||
|
||||
TDengine 采用类关系型数据模型，需要建库、建表。因此对于一个具体的应用场景，需要考虑库、超级表和普通表的设计。本节不讨论细致的语法规则，只介绍概念。
|
||||
|
|
|
|||
|
|
@ -23,7 +23,7 @@ import PhpStmt from "./_php_stmt.mdx";
|
|||
|
||||
## SQL 写入简介
|
||||
|
||||
应用通过连接器执行 INSERT 语句来插入数据，用户还可以通过 TAOS Shell，手动输入 INSERT 语句插入数据。
|
||||
应用通过连接器执行 INSERT 语句来插入数据，用户还可以通过 TDengine CLI，手动输入 INSERT 语句插入数据。
|
||||
|
||||
### 一次写入一条
|
||||
下面这条 INSERT 就将一条记录写入到表 d1001 中：
|
||||
|
|
|
|||
|
|
@ -0,0 +1,440 @@
|
|||
import Tabs from "@theme/Tabs";
|
||||
import TabItem from "@theme/TabItem";
|
||||
|
||||
# 高效写入
|
||||
|
||||
本节介绍如何高效地向 TDengine 写入数据。
|
||||
|
||||
## 高效写入原理 {#principle}
|
||||
|
||||
### 客户端程序的角度 {#application-view}
|
||||
|
||||
从客户端程序的角度来说，高效写入数据要考虑以下几个因素：
|
||||
|
||||
1. 单次写入的数据量。一般来讲，每批次写入的数据量越大越高效（但超过一定阈值其优势会消失）。使用 SQL 写入 TDengine 时，尽量在一条 SQL 中拼接更多数据。目前，TDengine 支持的一条 SQL 的最大长度为 1,048,576（1M）个字符。可通过配置客户端参数 maxSQLLength（默认值为 65480）进行修改。
|
||||
2. 并发连接数。一般来讲，同时写入数据的并发连接数越多写入越高效（但超过一定阈值反而会下降，取决于服务端处理能力）。
|
||||
3. 数据在不同表（或子表）之间的分布，即要写入数据的相邻性。一般来说，每批次只向同一张表（或子表）写入数据比向多张表（或子表）写入数据要更高效；
|
||||
4. 写入方式。一般来讲：
|
||||
- 参数绑定写入比 SQL 写入更高效。因参数绑定方式避免了 SQL 解析。（但增加了 C 接口的调用次数，对于连接器也有性能损耗）。
|
||||
- SQL 写入不自动建表比自动建表更高效。因自动建表要频繁检查表是否存在
|
||||
- SQL 写入比无模式写入更高效。因无模式写入会自动建表且支持动态更改表结构
|
||||
|
||||
客户端程序要充分且恰当地利用以上几个因素。在单次写入中尽量只向同一张表（或子表）写入数据，每批次写入的数据量经过测试和调优设定为一个最适合当前系统处理能力的数值，并发写入的连接数同样经过测试和调优后设定为一个最适合当前系统处理能力的数值，以实现在当前系统中的最佳写入速度。
|
||||
|
||||
### 数据源的角度 {#datasource-view}
|
||||
|
||||
客户端程序通常需要从数据源读数据再写入 TDengine。从数据源角度来说，以下几种情况需要在读线程和写线程之间增加队列：
|
||||
|
||||
1. 有多个数据源，单个数据源生成数据的速度远小于单线程写入的速度，但数据量整体比较大。此时队列的作用是把多个数据源的数据汇聚到一起，增加单次写入的数据量。
|
||||
2. 单个数据源生成数据的速度远大于单线程写入的速度。此时队列的作用是增加写入的并发度。
|
||||
3. 单张表的数据分散在多个数据源。此时队列的作用是将同一张表的数据提前汇聚到一起，提高写入时数据的相邻性。
|
||||
|
||||
如果写应用的数据源是 Kafka, 写应用本身即 Kafka 的消费者，则可利用 Kafka 的特性实现高效写入。比如：
|
||||
|
||||
1. 将同一张表的数据写到同一个 Topic 的同一个 Partition，增加数据的相邻性
|
||||
2. 通过订阅多个 Topic 实现数据汇聚
|
||||
3. 通过增加 Consumer 线程数增加写入的并发度
|
||||
4. 通过增加每次 fetch 的最大数据量来增加单次写入的最大数据量
|
||||
|
||||
### 服务器配置的角度 {#setting-view}
|
||||
|
||||
从服务器配置的角度来说，也有很多优化写入性能的方法。
|
||||
|
||||
如果总表数不多(远小于核数乘以1000), 且无论怎么调节客户端程序，taosd 进程的 CPU 使用率都很低，那么很可能是因为表在各个 vgroup 分布不均。比如：数据库总表数是 1000 且 minTablesPerVnode 设置的也是 1000，那么所有的表都会分布在 1 个 vgroup 上。此时如果将 minTablesPerVnode 和 tablelncStepPerVnode 都设置成 100， 则可将表分布至 10 个 vgroup。（假设 maxVgroupsPerDb 大于等于 10）。
|
||||
|
||||
如果总表数比较大（比如大于500万），适当增加 maxVgroupsPerDb 也能显著提高建表的速度。maxVgroupsPerDb 默认值为 0， 自动配置为 CPU 的核数。 如果表的数量巨大，也建议调节 maxTablesPerVnode 参数，以免超过单个 vnode 建表的上限。
|
||||
|
||||
更多调优参数，请参考 [配置参考](../../../reference/config)部分。
|
||||
|
||||
## 高效写入示例 {#sample-code}
|
||||
|
||||
### 场景设计 {#scenario}
|
||||
|
||||
下面的示例程序展示了如何高效写入数据，场景设计如下：
|
||||
|
||||
- TDengine 客户端程序从其它数据源不断读入数据，在示例程序中采用生成模拟数据的方式来模拟读取数据源
|
||||
- 单个连接向 TDengine 写入的速度无法与读数据的速度相匹配，因此客户端程序启动多个线程，每个线程都建立了与 TDengine 的连接，每个线程都有一个独占的固定大小的消息队列
|
||||
- 客户端程序将接收到的数据根据所属的表名（或子表名）HASH 到不同的线程，即写入该线程所对应的消息队列，以此确保属于某个表（或子表）的数据一定会被一个固定的线程处理
|
||||
- 各个子线程在将所关联的消息队列中的数据读空后或者读取数据量达到一个预定的阈值后将该批数据写入 TDengine，并继续处理后面接收到的数据
|
||||
|
||||
|
||||
|
||||
### 示例代码 {#code}
|
||||
|
||||
这一部分是针对以上场景的示例代码。对于其它场景高效写入原理相同，不过代码需要适当修改。
|
||||
|
||||
本示例代码假设源数据属于同一张超级表(meters)的不同子表。程序在开始写入数据之前已经在 test 库创建了这个超级表。对于子表，将根据收到的数据，由应用程序自动创建。如果实际场景是多个超级表，只需修改写任务自动建表的代码。
|
||||
|
||||
<Tabs defaultValue="java" groupId="lang">
|
||||
<TabItem label="Java" value="java">
|
||||
|
||||
**程序清单**
|
||||
|
||||
| 类名 | 功能说明 |
|
||||
| ---------------- | --------------------------------------------------------------------------- |
|
||||
| FastWriteExample | 主程序 |
|
||||
| ReadTask | 从模拟源中读取数据，将表名经过 hash 后得到 Queue 的 index，写入对应的 Queue |
|
||||
| WriteTask | 从 Queue 中获取数据，组成一个 Batch，写入 TDengine |
|
||||
| MockDataSource | 模拟生成一定数量 meters 子表的数据 |
|
||||
| SQLWriter | WriteTask 依赖这个类完成 SQL 拼接、自动建表、 SQL 写入、SQL 长度检查 |
|
||||
| StmtWriter | 实现参数绑定方式批量写入（暂未完成） |
|
||||
| DataBaseMonitor | 统计写入速度，并每隔 10 秒把当前写入速度打印到控制台 |
|
||||
|
||||
|
||||
以下是各类的完整代码和更详细的功能说明。
|
||||
|
||||
<details>
|
||||
<summary>FastWriteExample</summary>
|
||||
主程序负责：
|
||||
|
||||
1. 创建消息队列
|
||||
2. 启动写线程
|
||||
3. 启动读线程
|
||||
4. 每隔 10 秒统计一次写入速度
|
||||
|
||||
主程序默认暴露了 4 个参数，每次启动程序都可调节，用于测试和调优：
|
||||
|
||||
1. 读线程个数。默认为 1。
|
||||
2. 写线程个数。默认为 3。
|
||||
3. 模拟生成的总表数。默认为 1000。将会平分给各个读线程。如果总表数较大，建表需要花费较长，开始统计的写入速度可能较慢。
|
||||
4. 每批最多写入记录数量。默认为 3000。
|
||||
|
||||
队列容量(taskQueueCapacity)也是与性能有关的参数，可通过修改程序调节。一般来讲，队列容量越大，入队被阻塞的概率越小，队列的吞吐量越大，但是内存占用也会越大。 示例程序默认值已经设置地足够大。
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/FastWriteExample.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>ReadTask</summary>
|
||||
|
||||
读任务负责从数据源读数据。每个读任务都关联了一个模拟数据源。每个模拟数据源可生成一点数量表的数据。不同的模拟数据源生成不同表的数据。
|
||||
|
||||
读任务采用阻塞的方式写消息队列。也就是说，一旦队列满了，写操作就会阻塞。
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/ReadTask.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>WriteTask</summary>
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/WriteTask.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>MockDataSource</summary>
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/MockDataSource.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>SQLWriter</summary>
|
||||
|
||||
SQLWriter 类封装了拼 SQL 和写数据的逻辑。注意，所有的表都没有提前创建，而是在 catch 到表不存在异常的时候，再以超级表为模板批量建表，然后重新执行 INSERT 语句。对于其它异常，这里简单地记录当时执行的 SQL 语句到日志中，你也可以记录更多线索到日志，已便排查错误和故障恢复。
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/SQLWriter.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>DataBaseMonitor</summary>
|
||||
|
||||
```java
|
||||
{{#include docs/examples/java/src/main/java/com/taos/example/highvolume/DataBaseMonitor.java}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
**执行步骤**
|
||||
|
||||
<details>
|
||||
<summary>执行 Java 示例程序</summary>
|
||||
|
||||
执行程序前需配置环境变量 `TDENGINE_JDBC_URL`。如果 TDengine Server 部署在本机，且用户名、密码和端口都是默认值，那么可配置：
|
||||
|
||||
```
|
||||
TDENGINE_JDBC_URL="jdbc:TAOS://localhost:6030?user=root&password=taosdata"
|
||||
```
|
||||
|
||||
**本地集成开发环境执行示例程序**
|
||||
|
||||
1. clone TDengine 仓库
|
||||
```
|
||||
git clone git@github.com:taosdata/TDengine.git --depth 1
|
||||
```
|
||||
2. 用集成开发环境打开 `docs/examples/java` 目录。
|
||||
3. 在开发环境中配置环境变量 `TDENGINE_JDBC_URL`。如果已配置了全局的环境变量 `TDENGINE_JDBC_URL` 可跳过这一步。
|
||||
4. 运行类 `com.taos.example.highvolume.FastWriteExample`。
|
||||
|
||||
**远程服务器上执行示例程序**
|
||||
|
||||
若要在服务器上执行示例程序，可按照下面的步骤操作：
|
||||
|
||||
1. 打包示例代码。在目录 TDengine/docs/examples/java 下执行：
|
||||
```
|
||||
mvn package
|
||||
```
|
||||
2. 远程服务器上创建 examples 目录：
|
||||
```
|
||||
mkdir -p examples/java
|
||||
```
|
||||
3. 复制依赖到服务器指定目录：
|
||||
- 复制依赖包，只用复制一次
|
||||
```
|
||||
scp -r .\target\lib <user>@<host>:~/examples/java
|
||||
```
|
||||
- 复制本程序的 jar 包，每次更新代码都需要复制
|
||||
```
|
||||
scp -r .\target\javaexample-1.0.jar <user>@<host>:~/examples/java
|
||||
```
|
||||
4. 配置环境变量。
|
||||
编辑 `~/.bash_profile` 或 `~/.bashrc` 添加如下内容例如：
|
||||
|
||||
```
|
||||
export TDENGINE_JDBC_URL="jdbc:TAOS://localhost:6030?user=root&password=taosdata"
|
||||
```
|
||||
|
||||
以上使用的是本地部署 TDengine Server 时默认的 JDBC URL。你需要根据自己的实际情况更改。
|
||||
|
||||
5. 用 java 命令启动示例程序，命令模板：
|
||||
|
||||
```
|
||||
java -classpath lib/*:javaexample-1.0.jar com.taos.example.highvolume.FastWriteExample <read_thread_count> <white_thread_count> <total_table_count> <max_batch_size>
|
||||
```
|
||||
|
||||
6. 结束测试程序。测试程序不会自动结束，在获取到当前配置下稳定的写入速度后，按 <kbd>CTRL</kbd> + <kbd>C</kbd> 结束程序。
|
||||
下面是一次实际运行的日志输出，机器配置 16核 + 64G + 固态硬盘。
|
||||
|
||||
```
|
||||
root@vm85$ java -classpath lib/*:javaexample-1.0.jar com.taos.example.highvolume.FastWriteExample 2 12
|
||||
18:56:35.896 [main] INFO c.t.e.highvolume.FastWriteExample - readTaskCount=2, writeTaskCount=12 tableCount=1000 maxBatchSize=3000
|
||||
18:56:36.011 [WriteThread-0] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.015 [WriteThread-0] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.021 [WriteThread-1] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.022 [WriteThread-1] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.031 [WriteThread-2] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.032 [WriteThread-2] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.041 [WriteThread-3] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.042 [WriteThread-3] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.093 [WriteThread-4] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.094 [WriteThread-4] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.099 [WriteThread-5] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.100 [WriteThread-5] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.100 [WriteThread-6] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.101 [WriteThread-6] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.103 [WriteThread-7] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.104 [WriteThread-7] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.105 [WriteThread-8] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.107 [WriteThread-8] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.108 [WriteThread-9] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.109 [WriteThread-9] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.156 [WriteThread-10] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.157 [WriteThread-11] INFO c.taos.example.highvolume.WriteTask - started
|
||||
18:56:36.158 [WriteThread-10] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:36.158 [ReadThread-0] INFO com.taos.example.highvolume.ReadTask - started
|
||||
18:56:36.158 [ReadThread-1] INFO com.taos.example.highvolume.ReadTask - started
|
||||
18:56:36.158 [WriteThread-11] INFO c.taos.example.highvolume.SQLWriter - maxSQLLength=1048576
|
||||
18:56:46.369 [main] INFO c.t.e.highvolume.FastWriteExample - count=18554448 speed=1855444
|
||||
18:56:56.946 [main] INFO c.t.e.highvolume.FastWriteExample - count=39059660 speed=2050521
|
||||
18:57:07.322 [main] INFO c.t.e.highvolume.FastWriteExample - count=59403604 speed=2034394
|
||||
18:57:18.032 [main] INFO c.t.e.highvolume.FastWriteExample - count=80262938 speed=2085933
|
||||
18:57:28.432 [main] INFO c.t.e.highvolume.FastWriteExample - count=101139906 speed=2087696
|
||||
18:57:38.921 [main] INFO c.t.e.highvolume.FastWriteExample - count=121807202 speed=2066729
|
||||
18:57:49.375 [main] INFO c.t.e.highvolume.FastWriteExample - count=142952417 speed=2114521
|
||||
18:58:00.689 [main] INFO c.t.e.highvolume.FastWriteExample - count=163650306 speed=2069788
|
||||
18:58:11.646 [main] INFO c.t.e.highvolume.FastWriteExample - count=185019808 speed=2136950
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
</TabItem>
|
||||
<TabItem label="Python" value="python">
|
||||
|
||||
**程序清单**
|
||||
|
||||
Python 示例程序中采用了多进程的架构，并使用了跨进程的消息队列。
|
||||
|
||||
| 函数或类 | 功能说明 |
|
||||
| ------------------------ | -------------------------------------------------------------------- |
|
||||
| main 函数 | 程序入口， 创建各个子进程和消息队列 |
|
||||
| run_monitor_process 函数 | 创建数据库，超级表，统计写入速度并定时打印到控制台 |
|
||||
| run_read_task 函数 | 读进程主要逻辑，负责从其它数据系统读数据，并分发数据到为之分配的队列 |
|
||||
| MockDataSource 类 | 模拟数据源, 实现迭代器接口，每次批量返回每张表的接下来 1000 条数据 |
|
||||
| run_write_task 函数 | 写进程主要逻辑。每次从队列中取出尽量多的数据，并批量写入 |
|
||||
| SQLWriter类 | SQL 写入和自动建表 |
|
||||
| StmtWriter 类 | 实现参数绑定方式批量写入（暂未完成） |
|
||||
|
||||
|
||||
<details>
|
||||
<summary>main 函数</summary>
|
||||
|
||||
main 函数负责创建消息队列和启动子进程，子进程有 3 类：
|
||||
|
||||
1. 1 个监控进程，负责数据库初始化和统计写入速度
|
||||
2. n 个读进程，负责从其它数据系统读数据
|
||||
3. m 个写进程，负责写数据库
|
||||
|
||||
main 函数可以接收 5 个启动参数，依次是：
|
||||
|
||||
1. 读任务（进程）数, 默认为 1
|
||||
2. 写任务（进程）数, 默认为 1
|
||||
3. 模拟生成的总表数，默认为 1000
|
||||
4. 队列大小（单位字节），默认为 1000000
|
||||
5. 每批最多写入记录数量， 默认为 3000
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:main}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>run_monitor_process</summary>
|
||||
|
||||
监控进程负责初始化数据库，并监控当前的写入速度。
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:monitor}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>run_read_task 函数</summary>
|
||||
|
||||
读进程，负责从其它数据系统读数据，并分发数据到为之分配的队列。
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:read}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
<summary>MockDataSource</summary>
|
||||
|
||||
以下是模拟数据源的实现，我们假设数据源生成的每一条数据都带有目标表名信息。实际中你可能需要一定的规则确定目标表名。
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/mockdatasource.py}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
<summary>run_write_task 函数</summary>
|
||||
|
||||
写进程每次从队列中取出尽量多的数据，并批量写入。
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/fast_write_example.py:write}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
<details>
|
||||
|
||||
SQLWriter 类封装了拼 SQL 和写数据的逻辑。所有的表都没有提前创建，而是在发生表不存在错误的时候，再以超级表为模板批量建表，然后重新执行 INSERT 语句。对于其它错误会记录当时执行的 SQL， 以便排查错误和故障恢复。这个类也对 SQL 是否超过最大长度限制做了检查，如果接近 SQL 最大长度限制（maxSQLLength），将会立即执行 SQL。为了减少 SQL 此时，建议将 maxSQLLength 适当调大。
|
||||
|
||||
<summary>SQLWriter</summary>
|
||||
|
||||
```python
|
||||
{{#include docs/examples/python/sql_writer.py}}
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
**执行步骤**
|
||||
|
||||
<details>
|
||||
|
||||
<summary>执行 Python 示例程序</summary>
|
||||
|
||||
1. 前提条件
|
||||
|
||||
- 已安装 TDengine 客户端驱动
|
||||
- 已安装 Python3， 推荐版本 >= 3.8
|
||||
- 已安装 taospy
|
||||
|
||||
2. 安装 faster-fifo 代替 python 内置的 multiprocessing.Queue
|
||||
|
||||
```
|
||||
pip3 install faster-fifo
|
||||
```
|
||||
|
||||
3. 点击上面的“查看源码”链接复制 `fast_write_example.py` 、 `sql_writer.py` 和 `mockdatasource.py` 三个文件。
|
||||
|
||||
4. 执行示例程序
|
||||
|
||||
```
|
||||
python3 fast_write_example.py <READ_TASK_COUNT> <WRITE_TASK_COUNT> <TABLE_COUNT> <QUEUE_SIZE> <MAX_BATCH_SIZE>
|
||||
```
|
||||
|
||||
下面是一次实际运行的输出, 机器配置 16核 + 64G + 固态硬盘。
|
||||
|
||||
```
|
||||
root@vm85$ python3 fast_write_example.py 8 8
|
||||
2022-07-14 19:13:45,869 [root] - READ_TASK_COUNT=8, WRITE_TASK_COUNT=8, TABLE_COUNT=1000, QUEUE_SIZE=1000000, MAX_BATCH_SIZE=3000
|
||||
2022-07-14 19:13:48,882 [root] - WriteTask-0 started with pid 718347
|
||||
2022-07-14 19:13:48,883 [root] - WriteTask-1 started with pid 718348
|
||||
2022-07-14 19:13:48,884 [root] - WriteTask-2 started with pid 718349
|
||||
2022-07-14 19:13:48,884 [root] - WriteTask-3 started with pid 718350
|
||||
2022-07-14 19:13:48,885 [root] - WriteTask-4 started with pid 718351
|
||||
2022-07-14 19:13:48,885 [root] - WriteTask-5 started with pid 718352
|
||||
2022-07-14 19:13:48,886 [root] - WriteTask-6 started with pid 718353
|
||||
2022-07-14 19:13:48,886 [root] - WriteTask-7 started with pid 718354
|
||||
2022-07-14 19:13:48,887 [root] - ReadTask-0 started with pid 718355
|
||||
2022-07-14 19:13:48,888 [root] - ReadTask-1 started with pid 718356
|
||||
2022-07-14 19:13:48,889 [root] - ReadTask-2 started with pid 718357
|
||||
2022-07-14 19:13:48,889 [root] - ReadTask-3 started with pid 718358
|
||||
2022-07-14 19:13:48,890 [root] - ReadTask-4 started with pid 718359
|
||||
2022-07-14 19:13:48,891 [root] - ReadTask-5 started with pid 718361
|
||||
2022-07-14 19:13:48,892 [root] - ReadTask-6 started with pid 718364
|
||||
2022-07-14 19:13:48,893 [root] - ReadTask-7 started with pid 718365
|
||||
2022-07-14 19:13:56,042 [DataBaseMonitor] - count=6676310 speed=667631.0
|
||||
2022-07-14 19:14:06,196 [DataBaseMonitor] - count=20004310 speed=1332800.0
|
||||
2022-07-14 19:14:16,366 [DataBaseMonitor] - count=32290310 speed=1228600.0
|
||||
2022-07-14 19:14:26,527 [DataBaseMonitor] - count=44438310 speed=1214800.0
|
||||
2022-07-14 19:14:36,673 [DataBaseMonitor] - count=56608310 speed=1217000.0
|
||||
2022-07-14 19:14:46,834 [DataBaseMonitor] - count=68757310 speed=1214900.0
|
||||
2022-07-14 19:14:57,280 [DataBaseMonitor] - count=80992310 speed=1223500.0
|
||||
2022-07-14 19:15:07,689 [DataBaseMonitor] - count=93805310 speed=1281300.0
|
||||
2022-07-14 19:15:18,020 [DataBaseMonitor] - count=106111310 speed=1230600.0
|
||||
2022-07-14 19:15:28,356 [DataBaseMonitor] - count=118394310 speed=1228300.0
|
||||
2022-07-14 19:15:38,690 [DataBaseMonitor] - count=130742310 speed=1234800.0
|
||||
2022-07-14 19:15:49,000 [DataBaseMonitor] - count=143051310 speed=1230900.0
|
||||
2022-07-14 19:15:59,323 [DataBaseMonitor] - count=155276310 speed=1222500.0
|
||||
2022-07-14 19:16:09,649 [DataBaseMonitor] - count=167603310 speed=1232700.0
|
||||
2022-07-14 19:16:19,995 [DataBaseMonitor] - count=179976310 speed=1237300.0
|
||||
```
|
||||
|
||||
</details>
|
||||
|
||||
:::note
|
||||
使用 Python 连接器多进程连接 TDengine 的时候，有一个限制：不能在父进程中建立连接，所有连接只能在子进程中创建。
|
||||
如果在父进程中创建连接，子进程再创建连接就会一直阻塞。这是个已知问题。
|
||||
|
||||
:::
|
||||
|
||||
</TabItem>
|
||||
</Tabs>
|
||||
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 7.1 KiB |
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
sidebar_label: 写入数据
|
||||
title: 写入数据
|
||||
description: TDengine 的各种写入方式
|
||||
---
|
||||
|
||||
TDengine 支持多种写入协议，包括 SQL，InfluxDB Line 协议, OpenTSDB Telnet 协议，OpenTSDB JSON 格式协议。数据可以单条插入，也可以批量插入，可以插入一个数据采集点的数据，也可以同时插入多个数据采集点的数据。同时，TDengine 支持多线程插入，支持时间乱序数据插入，也支持历史数据插入。InfluxDB Line 协议、OpenTSDB Telnet 协议和 OpenTSDB JSON 格式协议是 TDengine 支持的三种无模式写入协议。使用无模式方式写入无需提前创建超级表和子表，并且引擎能自适用数据对表结构做调整。
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
---
|
||||
sidebar_label: 查询数据
|
||||
title: 查询数据
|
||||
description: "主要查询功能，通过连接器执行同步查询和异步查询"
|
||||
---
|
||||
|
|
@ -51,7 +52,7 @@ Query OK, 2 row(s) in set (0.001100s)
|
|||
|
||||
### 示例一
|
||||
|
||||
在 TAOS Shell，查找加利福尼亚州所有智能电表采集的电压平均值，并按照 location 分组。
|
||||
在 TDengine CLI，查找加利福尼亚州所有智能电表采集的电压平均值，并按照 location 分组。
|
||||
|
||||
```
|
||||
taos> SELECT AVG(voltage), location FROM meters GROUP BY location;
|
||||
|
|
@ -64,7 +65,7 @@ Query OK, 2 rows in database (0.005995s)
|
|||
|
||||
### 示例二
|
||||
|
||||
在 TAOS shell, 查找 groupId 为 2 的所有智能电表的记录条数，电流的最大值。
|
||||
在 TDengine CLI, 查找 groupId 为 2 的所有智能电表的记录条数，电流的最大值。
|
||||
|
||||
```
|
||||
taos> SELECT count(*), max(current) FROM meters where groupId = 2;
|
||||
|
|
|
|||
|
|
@ -64,7 +64,7 @@ DLL_EXPORT void tmq_conf_destroy(tmq_conf_t *conf);
|
|||
DLL_EXPORT void tmq_conf_set_auto_commit_cb(tmq_conf_t *conf, tmq_commit_cb *cb, void *param);
|
||||
```
|
||||
|
||||
这些 API 的文档请见 [C/C++ Connector](/reference/connector/cpp)，下面介绍一下它们的具体用法（超级表和子表结构请参考“数据建模”一节），完整的示例代码请见下面 C 语言的示例代码。
|
||||
这些 API 的文档请见 [C/C++ Connector](../../connector/cpp)，下面介绍一下它们的具体用法（超级表和子表结构请参考“数据建模”一节），完整的示例代码请见下面 C 语言的示例代码。
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="java" label="Java">
|
||||
|
|
|
|||
|
|
@ -20,11 +20,11 @@ create database db0 vgroups 100 buffer 16MB
|
|||
|
||||
## 读缓存
|
||||
|
||||
在创建数据库时可以选择是否缓存该数据库中每个子表的最新数据。由参数 cachelast 设置，分为三种情况：
|
||||
- 0: 不缓存
|
||||
- 1: 缓存子表最近一行数据，这将显著改善 last_row 函数的性能
|
||||
- 2: 缓存子表每一列最近的非 NULL 值，这将显著改善无特殊影响（比如 WHERE, ORDER BY, GROUP BY, INTERVAL）时的 last 函数的性能
|
||||
- 3: 同时缓存行和列，即等同于上述 cachelast 值为 1 或 2 时的行为同时生效
|
||||
在创建数据库时可以选择是否缓存该数据库中每个子表的最新数据。由参数 cachemodel 设置，分为四种情况：
|
||||
- none: 不缓存
|
||||
- last_row: 缓存子表最近一行数据，这将显著改善 last_row 函数的性能
|
||||
- last_value: 缓存子表每一列最近的非 NULL 值，这将显著改善无特殊影响（比如 WHERE, ORDER BY, GROUP BY, INTERVAL）时的 last 函数的性能
|
||||
- both: 同时缓存最近的行和列，即等同于上述 cachemodel 值为 last_row 和 last_value 的行为同时生效
|
||||
|
||||
## 元数据缓存
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
title: 开发指南
|
||||
sidebar_label: 开发指南
|
||||
description: 让开发者能够快速上手的指南
|
||||
---
|
||||
|
||||
开发一个应用，如果你准备采用TDengine作为时序数据处理的工具，那么有如下几个事情要做：
|
||||
|
|
@ -12,7 +14,7 @@ title: 开发指南
|
|||
7. 在很多场景下(如车辆管理)，应用需要获取每个数据采集点的最新状态，那么建议你采用TDengine的cache功能，而不用单独部署Redis等缓存软件。
|
||||
8. 如果你发现TDengine的函数无法满足你的要求，那么你可以使用用户自定义函数来解决问题。
|
||||
|
||||
本部分内容就是按照上述的顺序组织的。为便于理解，TDengine为每个功能为每个支持的编程语言都提供了示例代码。如果你希望深入了解SQL的使用，需要查看[SQL手册](/taos-sql/)。如果想更深入地了解各连接器的使用，请阅读[连接器参考指南](/reference/connector/)。如果还希望想将TDengine与第三方系统集成起来，比如Grafana, 请参考[第三方工具](/third-party/)。
|
||||
本部分内容就是按照上述的顺序组织的。为便于理解，TDengine为每个功能为每个支持的编程语言都提供了示例代码。如果你希望深入了解SQL的使用，需要查看[SQL手册](/taos-sql/)。如果想更深入地了解各连接器的使用，请阅读[连接器参考指南](../connector/)。如果还希望想将TDengine与第三方系统集成起来，比如Grafana, 请参考[第三方工具](../third-party/)。
|
||||
|
||||
如果在开发过程中遇到任何问题，请点击每个页面下方的["反馈问题"](https://github.com/taosdata/TDengine/issues/new/choose), 在GitHub上直接递交issue。
|
||||
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
title: REST API
|
||||
sidebar_label: REST API
|
||||
description: 详细介绍 TDengine 提供的 RESTful API.
|
||||
---
|
||||
|
||||
为支持各种不同类型平台的开发，TDengine 提供符合 REST 设计标准的 API，即 REST API。为最大程度降低学习成本，不同于其他数据库 REST API 的设计方法，TDengine 直接通过 HTTP POST 请求 BODY 中包含的 SQL 语句来操作数据库，仅需要一个 URL。REST 连接器的使用参见 [视频教程](https://www.taosdata.com/blog/2020/11/11/1965.html)。
|
||||
|
|
@ -10,7 +12,7 @@ title: REST API
|
|||
|
||||
## 安装
|
||||
|
||||
RESTful 接口不依赖于任何 TDengine 的库，因此客户端不需要安装任何 TDengine 的库，只要客户端的开发语言支持 HTTP 协议即可。
|
||||
RESTful 接口不依赖于任何 TDengine 的库，因此客户端不需要安装任何 TDengine 的库，只要客户端的开发语言支持 HTTP 协议即可。TDengine 的 RESTful API 由 [taosAdapter](../../reference/taosadapter) 提供，在使用 RESTful API 之前需要确保 `taosAdapter` 正常运行。
|
||||
|
||||
## 验证
|
||||
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: C/C++
|
||||
title: C/C++ Connector
|
||||
---
|
||||
|
|
@ -22,7 +21,7 @@ TDengine 客户端驱动的动态库位于：
|
|||
|
||||
## 支持的平台
|
||||
|
||||
请参考[支持的平台列表](/reference/connector#支持的平台)
|
||||
请参考[支持的平台列表](../#支持的平台)
|
||||
|
||||
## 支持的版本
|
||||
|
||||
|
|
@ -30,7 +29,7 @@ TDengine 客户端驱动的版本号与 TDengine 服务端的版本号是一一
|
|||
|
||||
## 安装步骤
|
||||
|
||||
TDengine 客户端驱动的安装请参考 [安装指南](/reference/connector#安装步骤)
|
||||
TDengine 客户端驱动的安装请参考 [安装指南](../#安装步骤)
|
||||
|
||||
## 建立连接
|
||||
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 2
|
||||
sidebar_label: Java
|
||||
title: TDengine Java Connector
|
||||
description: TDengine Java 连接器基于标准 JDBC API 实现, 并提供原生连接与 REST连接两种连接器。
|
||||
|
|
@ -35,7 +34,7 @@ REST 连接支持所有能运行 Java 的平台。
|
|||
|
||||
## 版本支持
|
||||
|
||||
请参考[版本支持列表](/reference/connector#版本支持)
|
||||
请参考[版本支持列表](../#版本支持)
|
||||
|
||||
## TDengine DataType 和 Java DataType
|
||||
|
||||
|
|
@ -64,7 +63,7 @@ TDengine 目前支持时间戳、数字、字符、布尔类型，与 Java 对
|
|||
使用 Java Connector 连接数据库前，需要具备以下条件：
|
||||
|
||||
- 已安装 Java 1.8 或以上版本运行时环境和 Maven 3.6 或以上版本
|
||||
- 已安装 TDengine 客户端驱动（使用原生连接必须安装，使用 REST 连接无需安装），具体步骤请参考[安装客户端驱动](/reference/connector#安装客户端驱动)
|
||||
- 已安装 TDengine 客户端驱动（使用原生连接必须安装，使用 REST 连接无需安装），具体步骤请参考[安装客户端驱动](../#安装客户端驱动)
|
||||
|
||||
### 安装连接器
|
||||
|
||||
|
|
@ -630,7 +629,7 @@ public void setNString(int columnIndex, ArrayList<String> list, int size) throws
|
|||
|
||||
### 无模式写入
|
||||
|
||||
TDengine 支持无模式写入功能。无模式写入兼容 InfluxDB 的 行协议（Line Protocol）、OpenTSDB 的 telnet 行协议和 OpenTSDB 的 JSON 格式协议。详情请参见[无模式写入](../../schemaless)。
|
||||
TDengine 支持无模式写入功能。无模式写入兼容 InfluxDB 的 行协议（Line Protocol）、OpenTSDB 的 telnet 行协议和 OpenTSDB 的 JSON 格式协议。详情请参见[无模式写入](../../reference/schemaless/)。
|
||||
|
||||
**注意**：
|
||||
|
||||
|
|
@ -9,11 +9,11 @@ import Tabs from '@theme/Tabs';
|
|||
import TabItem from '@theme/TabItem';
|
||||
|
||||
import Preparition from "./_preparition.mdx"
|
||||
import GoInsert from "../../07-develop/03-insert-data/_go_sql.mdx"
|
||||
import GoInfluxLine from "../../07-develop/03-insert-data/_go_line.mdx"
|
||||
import GoOpenTSDBTelnet from "../../07-develop/03-insert-data/_go_opts_telnet.mdx"
|
||||
import GoOpenTSDBJson from "../../07-develop/03-insert-data/_go_opts_json.mdx"
|
||||
import GoQuery from "../../07-develop/04-query-data/_go.mdx"
|
||||
import GoInsert from "../07-develop/03-insert-data/_go_sql.mdx"
|
||||
import GoInfluxLine from "../07-develop/03-insert-data/_go_line.mdx"
|
||||
import GoOpenTSDBTelnet from "../07-develop/03-insert-data/_go_opts_telnet.mdx"
|
||||
import GoOpenTSDBJson from "../07-develop/03-insert-data/_go_opts_json.mdx"
|
||||
import GoQuery from "../07-develop/04-query-data/_go.mdx"
|
||||
|
||||
`driver-go` 是 TDengine 的官方 Go 语言连接器，实现了 Go 语言[ database/sql ](https://golang.org/pkg/database/sql/) 包的接口。Go 开发人员可以通过它开发存取 TDengine 集群数据的应用软件。
|
||||
|
||||
|
|
@ -30,7 +30,7 @@ REST 连接支持所有能运行 Go 的平台。
|
|||
|
||||
## 版本支持
|
||||
|
||||
请参考[版本支持列表](/reference/connector#版本支持)
|
||||
请参考[版本支持列表](../#版本支持)
|
||||
|
||||
## 支持的功能特性
|
||||
|
||||
|
|
@ -56,7 +56,7 @@ REST 连接支持所有能运行 Go 的平台。
|
|||
### 安装前准备
|
||||
|
||||
* 安装 Go 开发环境（Go 1.14 及以上，GCC 4.8.5 及以上）
|
||||
* 如果使用原生连接器，请安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](/reference/connector#安装客户端驱动)
|
||||
* 如果使用原生连接器，请安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](../#安装客户端驱动)
|
||||
|
||||
配置好环境变量，检查命令：
|
||||
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 5
|
||||
sidebar_label: Rust
|
||||
title: TDengine Rust Connector
|
||||
---
|
||||
|
|
@ -9,9 +8,9 @@ import Tabs from '@theme/Tabs';
|
|||
import TabItem from '@theme/TabItem';
|
||||
|
||||
import Preparition from "./_preparition.mdx"
|
||||
import RustInsert from "../../07-develop/03-insert-data/_rust_sql.mdx"
|
||||
import RustBind from "../../07-develop/03-insert-data/_rust_stmt.mdx"
|
||||
import RustQuery from "../../07-develop/04-query-data/_rust.mdx"
|
||||
import RustInsert from "../07-develop/03-insert-data/_rust_sql.mdx"
|
||||
import RustBind from "../07-develop/03-insert-data/_rust_stmt.mdx"
|
||||
import RustQuery from "../07-develop/04-query-data/_rust.mdx"
|
||||
|
||||
[](https://crates.io/crates/taos)  [](https://docs.rs/taos)
|
||||
|
||||
|
|
@ -28,7 +27,7 @@ Websocket 连接支持所有能运行 Rust 的平台。
|
|||
|
||||
## 版本支持
|
||||
|
||||
请参考[版本支持列表](/reference/connector#版本支持)
|
||||
请参考[版本支持列表](../#版本支持)
|
||||
|
||||
Rust 连接器仍然在快速开发中，1.0 之前无法保证其向后兼容。建议使用 3.0 版本以上的 TDengine，以避免已知问题。
|
||||
|
||||
|
|
@ -37,7 +36,7 @@ Rust 连接器仍然在快速开发中，1.0 之前无法保证其向后兼容
|
|||
### 安装前准备
|
||||
|
||||
* 安装 Rust 开发工具链
|
||||
* 如果使用原生连接，请安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](/reference/connector#安装客户端驱动)
|
||||
* 如果使用原生连接，请安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](../#安装客户端驱动)
|
||||
|
||||
### 添加 taos 依赖
|
||||
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
sidebar_position: 3
|
||||
sidebar_label: Python
|
||||
title: TDengine Python Connector
|
||||
description: "taospy 是 TDengine 的官方 Python 连接器。taospy 提供了丰富的 API， 使得 Python 应用可以很方便地使用 TDengine。tasopy 对 TDengine 的原生接口和 REST 接口都进行了封装， 分别对应 tasopy 的两个子模块：tasos 和 taosrest。除了对原生接口和 REST 接口的封装，taospy 还提供了符合 Python 数据访问规范(PEP 249)的编程接口。这使得 taospy 和很多第三方工具集成变得简单，比如 SQLAlchemy 和 pandas"
|
||||
|
|
@ -8,7 +7,7 @@ description: "taospy 是 TDengine 的官方 Python 连接器。taospy 提供了
|
|||
import Tabs from "@theme/Tabs";
|
||||
import TabItem from "@theme/TabItem";
|
||||
|
||||
`taospy` 是 TDengine 的官方 Python 连接器。`taospy` 提供了丰富的 API， 使得 Python 应用可以很方便地使用 TDengine。`taospy` 对 TDengine 的[原生接口](/reference/connector/cpp)和 [REST 接口](/reference/rest-api)都进行了封装， 分别对应 `taospy` 包的 `taos` 模块 和 `taosrest` 模块。
|
||||
`taospy` 是 TDengine 的官方 Python 连接器。`taospy` 提供了丰富的 API， 使得 Python 应用可以很方便地使用 TDengine。`taospy` 对 TDengine 的[原生接口](../cpp)和 [REST 接口](../rest-api)都进行了封装， 分别对应 `taospy` 包的 `taos` 模块 和 `taosrest` 模块。
|
||||
除了对原生接口和 REST 接口的封装，`taospy` 还提供了符合 [Python 数据访问规范(PEP 249)](https://peps.python.org/pep-0249/) 的编程接口。这使得 `taospy` 和很多第三方工具集成变得简单，比如 [SQLAlchemy](https://www.sqlalchemy.org/) 和 [pandas](https://pandas.pydata.org/)。
|
||||
|
||||
使用客户端驱动提供的原生接口直接与服务端建立的连接的方式下文中称为“原生连接”；使用 taosAdapter 提供的 REST 接口与服务端建立的连接的方式下文中称为“REST 连接”。
|
||||
|
|
@ -17,7 +16,7 @@ Python 连接器的源码托管在 [GitHub](https://github.com/taosdata/taos-con
|
|||
|
||||
## 支持的平台
|
||||
|
||||
- 原生连接[支持的平台](/reference/connector/#支持的平台)和 TDengine 客户端支持的平台一致。
|
||||
- 原生连接[支持的平台](../#支持的平台)和 TDengine 客户端支持的平台一致。
|
||||
- REST 连接支持所有能运行 Python 的平台。
|
||||
|
||||
## 版本选择
|
||||
|
|
@ -150,10 +149,19 @@ curl -u root:taosdata http://<FQDN>:<PORT>/rest/sql -d "select server_version()"
|
|||
|
||||
```json
|
||||
{
|
||||
"status": "succ",
|
||||
"head": ["server_version()"],
|
||||
"column_meta": [["server_version()", 8, 8]],
|
||||
"data": [["2.4.0.16"]],
|
||||
"code": 0,
|
||||
"column_meta": [
|
||||
[
|
||||
"server_version()",
|
||||
"VARCHAR",
|
||||
7
|
||||
]
|
||||
],
|
||||
"data": [
|
||||
[
|
||||
"3.0.0.0"
|
||||
]
|
||||
],
|
||||
"rows": 1
|
||||
}
|
||||
```
|
||||
|
|
@ -266,7 +274,7 @@ TaosCursor 类使用原生连接进行写入、查询操作。在客户端多线
|
|||
|
||||
##### RestClient 类的使用
|
||||
|
||||
`RestClient` 类是对于 [REST API](/reference/rest-api) 的直接封装。它只包含一个 `sql()` 方法用于执行任意 SQL 语句， 并返回执行结果。
|
||||
`RestClient` 类是对于 [REST API](../rest-api) 的直接封装。它只包含一个 `sql()` 方法用于执行任意 SQL 语句， 并返回执行结果。
|
||||
|
||||
```python title="RestClient 的使用"
|
||||
{{#include docs/examples/python/rest_client_example.py}}
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 6
|
||||
sidebar_label: Node.js
|
||||
title: TDengine Node.js Connector
|
||||
---
|
||||
|
|
@ -9,11 +8,11 @@ import Tabs from "@theme/Tabs";
|
|||
import TabItem from "@theme/TabItem";
|
||||
|
||||
import Preparition from "./_preparition.mdx";
|
||||
import NodeInsert from "../../07-develop/03-insert-data/_js_sql.mdx";
|
||||
import NodeInfluxLine from "../../07-develop/03-insert-data/_js_line.mdx";
|
||||
import NodeOpenTSDBTelnet from "../../07-develop/03-insert-data/_js_opts_telnet.mdx";
|
||||
import NodeOpenTSDBJson from "../../07-develop/03-insert-data/_js_opts_json.mdx";
|
||||
import NodeQuery from "../../07-develop/04-query-data/_js.mdx";
|
||||
import NodeInsert from "../07-develop/03-insert-data/_js_sql.mdx";
|
||||
import NodeInfluxLine from "../07-develop/03-insert-data/_js_line.mdx";
|
||||
import NodeOpenTSDBTelnet from "../07-develop/03-insert-data/_js_opts_telnet.mdx";
|
||||
import NodeOpenTSDBJson from "../07-develop/03-insert-data/_js_opts_json.mdx";
|
||||
import NodeQuery from "../07-develop/04-query-data/_js.mdx";
|
||||
|
||||
`@tdengine/client` 和 `@tdengine/rest` 是 TDengine 的官方 Node.js 语言连接器。 Node.js 开发人员可以通过它开发可以存取 TDengine 集群数据的应用软件。注意：从 TDengine 3.0 开始 Node.js 原生连接器的包名由 `td2.0-connector` 改名为 `@tdengine/client` 而 rest 连接器的包名由 `td2.0-rest-connector` 改为 `@tdengine/rest`。并且不与 TDengine 2.x 兼容。
|
||||
|
||||
|
|
@ -28,7 +27,7 @@ REST 连接器支持所有能运行 Node.js 的平台。
|
|||
|
||||
## 版本支持
|
||||
|
||||
请参考[版本支持列表](/reference/connector#版本支持)
|
||||
请参考[版本支持列表](../#版本支持)
|
||||
|
||||
## 支持的功能特性
|
||||
|
||||
|
|
@ -52,7 +51,7 @@ REST 连接器支持所有能运行 Node.js 的平台。
|
|||
### 安装前准备
|
||||
|
||||
- 安装 Node.js 开发环境
|
||||
- 如果使用 REST 连接器，跳过此步。但如果使用原生连接器，请安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](/reference/connector#安装客户端驱动)。我们使用 [node-gyp](https://github.com/nodejs/node-gyp) 和 TDengine 实例进行交互，还需要根据具体操作系统来安装下文提到的一些依赖工具。
|
||||
- 如果使用 REST 连接器，跳过此步。但如果使用原生连接器，请安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](../#安装客户端驱动)。我们使用 [node-gyp](https://github.com/nodejs/node-gyp) 和 TDengine 实例进行交互，还需要根据具体操作系统来安装下文提到的一些依赖工具。
|
||||
|
||||
<Tabs defaultValue="Linux">
|
||||
<TabItem value="Linux" label="Linux 系统安装依赖工具">
|
||||
|
|
@ -1,6 +1,5 @@
|
|||
---
|
||||
toc_max_heading_level: 4
|
||||
sidebar_position: 7
|
||||
sidebar_label: C#
|
||||
title: C# Connector
|
||||
---
|
||||
|
|
@ -9,16 +8,16 @@ import Tabs from '@theme/Tabs';
|
|||
import TabItem from '@theme/TabItem';
|
||||
|
||||
import Preparition from "./_preparition.mdx"
|
||||
import CSInsert from "../../07-develop/03-insert-data/_cs_sql.mdx"
|
||||
import CSInfluxLine from "../../07-develop/03-insert-data/_cs_line.mdx"
|
||||
import CSOpenTSDBTelnet from "../../07-develop/03-insert-data/_cs_opts_telnet.mdx"
|
||||
import CSOpenTSDBJson from "../../07-develop/03-insert-data/_cs_opts_json.mdx"
|
||||
import CSQuery from "../../07-develop/04-query-data/_cs.mdx"
|
||||
import CSAsyncQuery from "../../07-develop/04-query-data/_cs_async.mdx"
|
||||
import CSInsert from "../07-develop/03-insert-data/_cs_sql.mdx"
|
||||
import CSInfluxLine from "../07-develop/03-insert-data/_cs_line.mdx"
|
||||
import CSOpenTSDBTelnet from "../07-develop/03-insert-data/_cs_opts_telnet.mdx"
|
||||
import CSOpenTSDBJson from "../07-develop/03-insert-data/_cs_opts_json.mdx"
|
||||
import CSQuery from "../07-develop/04-query-data/_cs.mdx"
|
||||
import CSAsyncQuery from "../07-develop/04-query-data/_cs_async.mdx"
|
||||
|
||||
`TDengine.Connector` 是 TDengine 提供的 C# 语言连接器。C# 开发人员可以通过它开发存取 TDengine 集群数据的 C# 应用软件。
|
||||
|
||||
`TDengine.Connector` 连接器支持通过 TDengine 客户端驱动（taosc）建立与 TDengine 运行实例的连接，提供数据写入、查询、订阅、schemaless 数据写入、参数绑定接口数据写入等功能 `TDengine.Connector` 目前暂未提供 REST 连接方式，用户可以参考 [REST API](/reference/rest-api/) 文档自行编写。
|
||||
`TDengine.Connector` 连接器支持通过 TDengine 客户端驱动（taosc）建立与 TDengine 运行实例的连接，提供数据写入、查询、订阅、schemaless 数据写入、参数绑定接口数据写入等功能 `TDengine.Connector` 目前暂未提供 REST 连接方式，用户可以参考 [REST API](../rest-api/) 文档自行编写。
|
||||
|
||||
本文介绍如何在 Linux 或 Windows 环境中安装 `TDengine.Connector`，并通过 `TDengine.Connector` 连接 TDengine 集群，进行数据写入、查询等基本操作。
|
||||
|
||||
|
|
@ -32,7 +31,7 @@ import CSAsyncQuery from "../../07-develop/04-query-data/_cs_async.mdx"
|
|||
|
||||
## 版本支持
|
||||
|
||||
请参考[版本支持列表](/reference/connector#版本支持)
|
||||
请参考[版本支持列表](../#版本支持)
|
||||
|
||||
## 支持的功能特性
|
||||
|
||||
|
|
@ -49,7 +48,7 @@ import CSAsyncQuery from "../../07-develop/04-query-data/_cs_async.mdx"
|
|||
|
||||
* 安装 [.NET SDK](https://dotnet.microsoft.com/download)
|
||||
* [Nuget 客户端](https://docs.microsoft.com/en-us/nuget/install-nuget-client-tools) （可选安装）
|
||||
* 安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](/reference/connector#安装客户端驱动)
|
||||
* 安装 TDengine 客户端驱动，具体步骤请参考[安装客户端驱动](../#安装客户端驱动)
|
||||
|
||||
### 使用 dotnet CLI 安装
|
||||
|
||||
|
|
@ -1,5 +1,4 @@
|
|||
---
|
||||
sidebar_position: 1
|
||||
sidebar_label: PHP
|
||||
title: PHP Connector
|
||||
---
|
||||
|
|
@ -38,7 +37,7 @@ TDengine 客户端驱动的版本号与 TDengine 服务端的版本号是一一
|
|||
|
||||
### 安装 TDengine 客户端驱动
|
||||
|
||||
TDengine 客户端驱动的安装请参考 [安装指南](/reference/connector#安装步骤)
|
||||
TDengine 客户端驱动的安装请参考 [安装指南](../#安装步骤)
|
||||
|
||||
### 编译安装 php-tdengine
|
||||
|
||||
|
|
@ -61,7 +60,7 @@ phpize && ./configure && make -j && make install
|
|||
**手动指定 tdengine 目录：**
|
||||
|
||||
```shell
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/2.4.0.0 && make -j && make install
|
||||
phpize && ./configure --with-tdengine-dir=/usr/local/Cellar/tdengine/3.0.0.0 && make -j && make install
|
||||
```
|
||||
|
||||
> `--with-tdengine-dir=` 后跟上 tdengine 目录。
|
||||
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
sidebar_label: 错误码
|
||||
title: TDengine C/C++ 连接器错误码
|
||||
description: C/C++ 连接器的错误码列表和详细说明
|
||||
---
|
||||
|
||||
本文中详细列举了在使用 TDengine C/C++ 连接器时客户端可能得到的错误码以及所要采取的相应动作。其它语言的连接器在使用原生连接方式时也会所得到的返回码返回给连接器的调用者。
|
||||
|
|
@ -4,7 +4,7 @@ import PkgListV3 from "/components/PkgListV3";
|
|||
|
||||
<PkgListV3 type={1} sys="Linux" />
|
||||
|
||||
[所有下载](../../releases)
|
||||
[所有下载](../../releases/tdengine)
|
||||
|
||||
2. 解压缩软件包
|
||||
|
||||
|
|
@ -4,7 +4,8 @@ import PkgListV3 from "/components/PkgListV3";
|
|||
|
||||
<PkgListV3 type={4} sys="Windows" />
|
||||
|
||||
[所有下载](../../releases)
|
||||
[所有下载](../../releases/tdengine)
|
||||
|
||||
2. 执行安装程序，按提示选择默认值，完成安装
|
||||
3. 安装路径
|
||||
|
||||
|
Before Width: | Height: | Size: 25 KiB After Width: | Height: | Size: 25 KiB |
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
sidebar_label: 连接器
|
||||
title: 连接器
|
||||
description: 详细介绍各种语言的连接器及 REST API
|
||||
---
|
||||
|
||||
TDengine 提供了丰富的应用程序开发接口，为了便于用户快速开发自己的应用，TDengine 支持了多种编程语言的连接器，其中官方连接器包括支持 C/C++、Java、Python、Go、Node.js、C# 和 Rust 的连接器。这些连接器支持使用原生接口（taosc）和 REST 接口（部分语言暂不支持）连接 TDengine 集群。社区开发者也贡献了多个非官方连接器，例如 ADO.NET 连接器、Lua 连接器和 PHP 连接器。
|
||||
|
Before Width: | Height: | Size: 81 KiB After Width: | Height: | Size: 81 KiB |
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
sidebar_label: 手动部署
|
||||
title: 集群部署和管理
|
||||
description: 使用命令行工具手动部署 TDengine 集群
|
||||
---
|
||||
|
||||
## 准备工作
|
||||
|
|
@ -70,7 +71,7 @@ serverPort 6030
|
|||
|
||||
## 启动集群
|
||||
|
||||
按照《立即开始》里的步骤，启动第一个数据节点，例如 h1.taosdata.com，然后执行 taos，启动 taos shell，从 shell 里执行命令“SHOW DNODES”，如下所示：
|
||||
按照《立即开始》里的步骤，启动第一个数据节点，例如 h1.taosdata.com，然后执行 taos，启动 TDengine CLI，在其中执行命令 “SHOW DNODES”，如下所示：
|
||||
|
||||
```
|
||||
taos> show dnodes;
|
||||
|
|
@ -114,7 +115,7 @@ SHOW DNODES;
|
|||
|
||||
任何已经加入集群在线的数据节点，都可以作为后续待加入节点的 firstEp。
|
||||
firstEp 这个参数仅仅在该数据节点首次加入集群时有作用，加入集群后，该数据节点会保存最新的 mnode 的 End Point 列表，不再依赖这个参数。
|
||||
接下来，配置文件中的 firstEp 参数就主要在客户端连接的时候使用了，例如 taos shell 如果不加参数，会默认连接由 firstEp 指定的节点。
|
||||
接下来，配置文件中的 firstEp 参数就主要在客户端连接的时候使用了，例如 TDengine CLI 如果不加参数，会默认连接由 firstEp 指定的节点。
|
||||
两个没有配置 firstEp 参数的数据节点 dnode 启动后，会独立运行起来。这个时候，无法将其中一个数据节点加入到另外一个数据节点，形成集群。无法将两个独立的集群合并成为新的集群。
|
||||
|
||||
:::
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
sidebar_label: Kubernetes
|
||||
title: 在 Kubernetes 上部署 TDengine 集群
|
||||
description: 利用 Kubernetes 部署 TDengine 集群的详细指南
|
||||
---
|
||||
|
||||
作为面向云原生架构设计的时序数据库，TDengine 支持 Kubernetes 部署。这里介绍如何使用 YAML 文件一步一步从头创建一个 TDengine 集群，并重点介绍 Kubernetes 环境下 TDengine 的常用操作。
|
||||
|
|
@ -9,6 +10,7 @@ title: 在 Kubernetes 上部署 TDengine 集群
|
|||
|
||||
要使用 Kubernetes 部署管理 TDengine 集群，需要做好如下准备工作。
|
||||
|
||||
* 本文适用 Kubernetes v1.5 以上版本
|
||||
* 本文和下一章使用 minikube、kubectl 和 helm 等工具进行安装部署，请提前安装好相应软件
|
||||
* Kubernetes 已经安装部署并能正常访问使用或更新必要的容器仓库或其他服务
|
||||
|
||||
|
|
@ -365,7 +367,7 @@ kubectl scale statefulsets tdengine --replicas=1
|
|||
|
||||
```
|
||||
|
||||
在 taos shell 中的所有数据库操作将无法成功。
|
||||
在 TDengine CLI 中的所有数据库操作将无法成功。
|
||||
|
||||
```
|
||||
taos> show dnodes;
|
||||
|
|
|
|||
|
|
@ -1,6 +1,7 @@
|
|||
---
|
||||
sidebar_label: Helm
|
||||
title: 使用 Helm 部署 TDengine 集群
|
||||
description: 使用 Helm 部署 TDengine 集群的详细指南
|
||||
---
|
||||
|
||||
Helm 是 Kubernetes 的包管理器，上一节使用 Kubernets 部署 TDengine 集群的操作已经足够简单，但 Helm 依然可以提供更强大的能力。
|
||||
|
|
@ -171,70 +172,19 @@ taoscfg:
|
|||
TAOS_REPLICA: "1"
|
||||
|
||||
|
||||
# number of days per DB file
|
||||
# TAOS_DAYS: "10"
|
||||
|
||||
# number of days to keep DB file, default is 10 years.
|
||||
#TAOS_KEEP: "3650"
|
||||
|
||||
# cache block size (Mbyte)
|
||||
#TAOS_CACHE: "16"
|
||||
|
||||
# number of cache blocks per vnode
|
||||
#TAOS_BLOCKS: "6"
|
||||
|
||||
# minimum rows of records in file block
|
||||
#TAOS_MIN_ROWS: "100"
|
||||
|
||||
# maximum rows of records in file block
|
||||
#TAOS_MAX_ROWS: "4096"
|
||||
|
||||
#
|
||||
# TAOS_NUM_OF_THREADS_PER_CORE: number of threads per CPU core
|
||||
#TAOS_NUM_OF_THREADS_PER_CORE: "1.0"
|
||||
# TAOS_NUM_OF_RPC_THREADS: number of threads for RPC
|
||||
#TAOS_NUM_OF_RPC_THREADS: "2"
|
||||
|
||||
#
|
||||
# TAOS_NUM_OF_COMMIT_THREADS: number of threads to commit cache data
|
||||
#TAOS_NUM_OF_COMMIT_THREADS: "4"
|
||||
|
||||
#
|
||||
# TAOS_RATIO_OF_QUERY_CORES:
|
||||
# the proportion of total CPU cores available for query processing
|
||||
# 2.0: the query threads will be set to double of the CPU cores.
|
||||
# 1.0: all CPU cores are available for query processing [default].
|
||||
# 0.5: only half of the CPU cores are available for query.
|
||||
# 0.0: only one core available.
|
||||
#TAOS_RATIO_OF_QUERY_CORES: "1.0"
|
||||
|
||||
#
|
||||
# TAOS_KEEP_COLUMN_NAME:
|
||||
# the last_row/first/last aggregator will not change the original column name in the result fields
|
||||
#TAOS_KEEP_COLUMN_NAME: "0"
|
||||
|
||||
# enable/disable backuping vnode directory when removing vnode
|
||||
#TAOS_VNODE_BAK: "1"
|
||||
|
||||
# enable/disable installation / usage report
|
||||
#TAOS_TELEMETRY_REPORTING: "1"
|
||||
|
||||
# enable/disable load balancing
|
||||
#TAOS_BALANCE: "1"
|
||||
|
||||
# max timer control blocks
|
||||
#TAOS_MAX_TMR_CTRL: "512"
|
||||
|
||||
# time interval of system monitor, seconds
|
||||
#TAOS_MONITOR_INTERVAL: "30"
|
||||
|
||||
# number of seconds allowed for a dnode to be offline, for cluster only
|
||||
#TAOS_OFFLINE_THRESHOLD: "8640000"
|
||||
|
||||
# RPC re-try timer, millisecond
|
||||
#TAOS_RPC_TIMER: "1000"
|
||||
|
||||
# RPC maximum time for ack, seconds.
|
||||
#TAOS_RPC_MAX_TIME: "600"
|
||||
|
||||
# time interval of dnode status reporting to mnode, seconds, for cluster only
|
||||
#TAOS_STATUS_INTERVAL: "1"
|
||||
|
||||
|
|
@ -245,37 +195,7 @@ taoscfg:
|
|||
#TAOS_MIN_SLIDING_TIME: "10"
|
||||
|
||||
# minimum time window, milli-second
|
||||
#TAOS_MIN_INTERVAL_TIME: "10"
|
||||
|
||||
# maximum delay before launching a stream computation, milli-second
|
||||
#TAOS_MAX_STREAM_COMP_DELAY: "20000"
|
||||
|
||||
# maximum delay before launching a stream computation for the first time, milli-second
|
||||
#TAOS_MAX_FIRST_STREAM_COMP_DELAY: "10000"
|
||||
|
||||
# retry delay when a stream computation fails, milli-second
|
||||
#TAOS_RETRY_STREAM_COMP_DELAY: "10"
|
||||
|
||||
# the delayed time for launching a stream computation, from 0.1(default, 10% of whole computing time window) to 0.9
|
||||
#TAOS_STREAM_COMP_DELAY_RATIO: "0.1"
|
||||
|
||||
# max number of vgroups per db, 0 means configured automatically
|
||||
#TAOS_MAX_VGROUPS_PER_DB: "0"
|
||||
|
||||
# max number of tables per vnode
|
||||
#TAOS_MAX_TABLES_PER_VNODE: "1000000"
|
||||
|
||||
# the number of acknowledgments required for successful data writing
|
||||
#TAOS_QUORUM: "1"
|
||||
|
||||
# enable/disable compression
|
||||
#TAOS_COMP: "2"
|
||||
|
||||
# write ahead log (WAL) level, 0: no wal; 1: write wal, but no fysnc; 2: write wal, and call fsync
|
||||
#TAOS_WAL_LEVEL: "1"
|
||||
|
||||
# if walLevel is set to 2, the cycle of fsync being executed, if set to 0, fsync is called right away
|
||||
#TAOS_FSYNC: "3000"
|
||||
#TAOS_MIN_INTERVAL_TIME: "1"
|
||||
|
||||
# the compressed rpc message, option:
|
||||
# -1 (no compression)
|
||||
|
|
@ -283,17 +203,8 @@ taoscfg:
|
|||
# > 0 (rpc message body which larger than this value will be compressed)
|
||||
#TAOS_COMPRESS_MSG_SIZE: "-1"
|
||||
|
||||
# max length of an SQL
|
||||
#TAOS_MAX_SQL_LENGTH: "1048576"
|
||||
|
||||
# the maximum number of records allowed for super table time sorting
|
||||
#TAOS_MAX_NUM_OF_ORDERED_RES: "100000"
|
||||
|
||||
# max number of connections allowed in dnode
|
||||
#TAOS_MAX_SHELL_CONNS: "5000"
|
||||
|
||||
# max number of connections allowed in client
|
||||
#TAOS_MAX_CONNECTIONS: "5000"
|
||||
#TAOS_MAX_SHELL_CONNS: "50000"
|
||||
|
||||
# stop writing logs when the disk size of the log folder is less than this value
|
||||
#TAOS_MINIMAL_LOG_DIR_G_B: "0.1"
|
||||
|
|
@ -313,21 +224,8 @@ taoscfg:
|
|||
# enable/disable system monitor
|
||||
#TAOS_MONITOR: "1"
|
||||
|
||||
# enable/disable recording the SQL statements via restful interface
|
||||
#TAOS_HTTP_ENABLE_RECORD_SQL: "0"
|
||||
|
||||
# number of threads used to process http requests
|
||||
#TAOS_HTTP_MAX_THREADS: "2"
|
||||
|
||||
# maximum number of rows returned by the restful interface
|
||||
#TAOS_RESTFUL_ROW_LIMIT: "10240"
|
||||
|
||||
# The following parameter is used to limit the maximum number of lines in log files.
|
||||
# max number of lines per log filters
|
||||
# numOfLogLines 10000000
|
||||
|
||||
# enable/disable async log
|
||||
#TAOS_ASYNC_LOG: "0"
|
||||
#TAOS_ASYNC_LOG: "1"
|
||||
|
||||
#
|
||||
# time of keeping log files, days
|
||||
|
|
@ -344,25 +242,8 @@ taoscfg:
|
|||
# debug flag for all log type, take effect when non-zero value\
|
||||
#TAOS_DEBUG_FLAG: "143"
|
||||
|
||||
# enable/disable recording the SQL in taos client
|
||||
#TAOS_ENABLE_RECORD_SQL: "0"
|
||||
|
||||
# generate core file when service crash
|
||||
#TAOS_ENABLE_CORE_FILE: "1"
|
||||
|
||||
# maximum display width of binary and nchar fields in the shell. The parts exceeding this limit will be hidden
|
||||
#TAOS_MAX_BINARY_DISPLAY_WIDTH: "30"
|
||||
|
||||
# enable/disable stream (continuous query)
|
||||
#TAOS_STREAM: "1"
|
||||
|
||||
# in retrieve blocking model, only in 50% query threads will be used in query processing in dnode
|
||||
#TAOS_RETRIEVE_BLOCKING_MODEL: "0"
|
||||
|
||||
# the maximum allowed query buffer size in MB during query processing for each data node
|
||||
# -1 no limit (default)
|
||||
# 0 no query allowed, queries are disabled
|
||||
#TAOS_QUERY_BUFFER_SIZE: "-1"
|
||||
```
|
||||
|
||||
## 扩容
|
||||
|
|
|
|||
|
|
@ -1,5 +1,7 @@
|
|||
---
|
||||
sidebar_label: 部署集群
|
||||
title: 部署集群
|
||||
description: 部署 TDengine 集群的多种方式
|
||||
---
|
||||
|
||||
TDengine 支持集群，提供水平扩展的能力。如果需要获得更高的处理能力，只需要多增加节点即可。TDengine 采用虚拟节点技术，将一个节点虚拟化为多个虚拟节点，以实现负载均衡。同时，TDengine可以将多个节点上的虚拟节点组成虚拟节点组，通过多副本机制，以保证供系统的高可用。TDengine的集群功能完全开源。
|
||||
|
|
|
|||
Some files were not shown because too many files have changed in this diff Show More
Loading…
Reference in New Issue