xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

Merge branch 'main' into merge/mainto3.02

This commit is contained in:
Shengliang Guan 2025-02-07 09:35:15 +08:00
parent 62a872bb49 053c58c85d
commit dd63bf6364
15 changed files with 812 additions and 656 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

387
docs/en/14-reference/01-components/03-taosadapter.md
View File

@ -13,45 +13,108 @@ import Icinga2 from "../../assets/resources/_icinga2.mdx"
import TCollector from "../../assets/resources/_tcollector.mdx"
taosAdapter is a companion tool for TDengine, serving as a bridge and adapter between the TDengine cluster and applications. It provides an easy and efficient way to ingest data directly from data collection agents (such as Telegraf, StatsD, collectd, etc.). It also offers InfluxDB/OpenTSDB compatible data ingestion interfaces, allowing InfluxDB/OpenTSDB applications to be seamlessly ported to TDengine.
The connectors of TDengine in various languages communicate with TDengine through the WebSocket interface, hence the taosAdapter must be installed.
taosAdapter offers the following features:
- RESTful interface
- Compatible with InfluxDB v1 write interface
- Compatible with OpenTSDB JSON and telnet format writing
- Seamless connection to Telegraf
- Seamless connection to collectd
- Seamless connection to StatsD
- Supports Prometheus remote_read and remote_write
- Retrieves the VGroup ID of the virtual node group (VGroup) where the table is located
## taosAdapter Architecture Diagram
The architecture diagram is as follows:
<figure>
<Image img={imgAdapter} alt="taosAdapter architecture"/>
<figcaption>Figure 1. taosAdapter architecture</figcaption>
</figure>
## Deployment Methods for taosAdapter
## Feature List
### Installing taosAdapter
The taosAdapter provides the following features:
- WebSocket Interface:
Supports executing SQL, schemaless writing, parameter binding, and data subscription through the WebSocket protocol.
- Compatible with InfluxDB v1 write interface:
[https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/](https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/)
- Compatible with OpenTSDB JSON and telnet format writing:
- [http://opentsdb.net/docs/build/html/api_http/put.html](http://opentsdb.net/docs/build/html/api_http/put.html)
- [http://opentsdb.net/docs/build/html/api_telnet/put.html](http://opentsdb.net/docs/build/html/api_telnet/put.html)
- collectd data writing:
collectd is a system statistics collection daemon, visit [https://collectd.org/](https://collectd.org/) for more information.
- StatsD data writing:
StatsD is a simple yet powerful daemon for gathering statistics. Visit [https://github.com/statsd/statsd](https://github.com/statsd/statsd) for more information.
- icinga2 OpenTSDB writer data writing:
icinga2 is a software for collecting check results metrics and performance data. Visit [https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer](https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer) for more information.
- TCollector data writing:
TCollector is a client process that collects data from local collectors and pushes it to OpenTSDB. Visit [http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html](http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html) for more information.
- node_exporter data collection and writing:
node_exporter is an exporter of machine metrics. Visit [https://github.com/prometheus/node_exporter](https://github.com/prometheus/node_exporter) for more information.
- Supports Prometheus remote_read and remote_write:
remote_read and remote_write are Prometheus's data read-write separation cluster solutions. Visit [https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis) for more information.
- RESTful API:
[RESTful API](../../client-libraries/rest-api/)
### WebSocket Interface
Through the WebSocket interface of taosAdapter, connectors in various languages can achieve SQL execution, schemaless writing, parameter binding, and data subscription functionalities. Refer to the [Development Guide](../../../developer-guide/connecting-to-tdengine/#websocket-connection) for more details.
### Compatible with InfluxDB v1 write interface
You can use any client that supports the HTTP protocol to write data in InfluxDB compatible format to TDengine by accessing the Restful interface URL `http://<fqdn>:6041/influxdb/v1/write`.
Supported InfluxDB parameters are as follows:
- `db` specifies the database name used by TDengine
- `precision` the time precision used by TDengine
- `u` TDengine username
- `p` TDengine password
- `ttl` the lifespan of automatically created subtables, determined by the TTL parameter of the first data entry in the subtable, which cannot be updated. For more information, please refer to the TTL parameter in the [table creation document](../../sql-manual/manage-tables/).
Note: Currently, InfluxDB's token authentication method is not supported, only Basic authentication and query parameter verification are supported.
Example: `curl --request POST http://127.0.0.1:6041/influxdb/v1/write?db=test --user "root:taosdata" --data-binary "measurement,host=host1 field1=2i,field2=2.0 1577836800000000000"`
### Compatible with OpenTSDB JSON and telnet format writing
You can use any client that supports the HTTP protocol to write data in OpenTSDB compatible format to TDengine by accessing the Restful interface URL `http://<fqdn>:6041/<APIEndPoint>`. EndPoint as follows:
```text
/opentsdb/v1/put/json/<db>
/opentsdb/v1/put/telnet/<db>
```
### collectd data writing
<CollectD />
### StatsD data writing
<StatsD />
### icinga2 OpenTSDB writer data writing
<Icinga2 />
### TCollector data writing
<TCollector />
### node_exporter data collection and writing
An exporter used by Prometheus that exposes hardware and operating system metrics from \*NIX kernels
- Enable configuration of taosAdapter node_exporter.enable
- Set the relevant configuration for node_exporter
- Restart taosAdapter
### Supports Prometheus remote_read and remote_write
<Prometheus />
### RESTful API
You can use any client that supports the HTTP protocol to write data to TDengine or query data from TDengine by accessing the RESTful interface URL `http://<fqdn>:6041/rest/sql`. For details, please refer to the [REST API documentation](../../client-libraries/rest-api/).
## Installation
taosAdapter is part of the TDengine server software. If you are using TDengine server, you do not need any additional steps to install taosAdapter. If you need to deploy taosAdapter separately from the TDengine server, you should install the complete TDengine on that server to install taosAdapter. If you need to compile taosAdapter from source code, you can refer to the [Build taosAdapter](https://github.com/taosdata/taosadapter/blob/3.0/BUILD.md) document.
### Starting/Stopping taosAdapter
After the installation is complete, you can start the taosAdapter service using the command `systemctl start taosadapter`.
On Linux systems, the taosAdapter service is managed by default by systemd. Use the command `systemctl start taosadapter` to start the taosAdapter service. Use the command `systemctl stop taosadapter` to stop the taosAdapter service.
### Removing taosAdapter
Use the command rmtaos to remove the TDengine server software, including taosAdapter.
### Upgrading taosAdapter
taosAdapter and TDengine server need to use the same version. Please upgrade taosAdapter by upgrading the TDengine server.
taosAdapter deployed separately from taosd must be upgraded by upgrading the TDengine server on its server.
## taosAdapter Parameter List
## Configuration
taosAdapter supports configuration through command-line parameters, environment variables, and configuration files. The default configuration file is `/etc/taos/taosadapter.toml`.
@ -80,6 +143,7 @@ Usage of taosAdapter:
--instanceId int instance ID. Env "TAOS_ADAPTER_INSTANCE_ID" (default 32)
--log.compress whether to compress old log. Env "TAOS_ADAPTER_LOG_COMPRESS"
--log.enableRecordHttpSql whether to record http sql. Env "TAOS_ADAPTER_LOG_ENABLE_RECORD_HTTP_SQL"
--log.keepDays uint log retention days, must be a positive integer. Env "TAOS_ADAPTER_LOG_KEEP_DAYS" (default 30)
--log.level string log level (trace debug info warning error). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
--log.path string log path. Env "TAOS_ADAPTER_LOG_PATH" (default "/var/log/taos")
--log.reservedDiskSize string reserved disk size for log dir (KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_RESERVED_DISK_SIZE" (default "1GB")
@ -90,6 +154,8 @@ Usage of taosAdapter:
--log.sqlRotationSize string record sql log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_SIZE" (default "1GB")
--log.sqlRotationTime duration record sql log rotation time. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_TIME" (default 24h0m0s)
--logLevel string log level (trace debug info warning error). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
--maxAsyncConcurrentLimit int The maximum number of concurrent calls allowed for the C asynchronous method. 0 means use CPU core count. Env "TAOS_ADAPTER_MAX_ASYNC_CONCURRENT_LIMIT"
--maxSyncConcurrentLimit int The maximum number of concurrent calls allowed for the C synchronized method. 0 means use CPU core count. Env "TAOS_ADAPTER_MAX_SYNC_CONCURRENT_LIMIT"
--monitor.collectDuration duration Set monitor duration. Env "TAOS_ADAPTER_MONITOR_COLLECT_DURATION" (default 3s)
--monitor.disable Whether to disable monitoring. Env "TAOS_ADAPTER_MONITOR_DISABLE" (default true)
--monitor.identity string The identity of the current instance, or 'hostname:port' if it is empty. Env "TAOS_ADAPTER_MONITOR_IDENTITY"
@ -118,7 +184,7 @@ Usage of taosAdapter:
--opentsdb_telnet.flushInterval duration opentsdb_telnet flush interval (0s means not valid) . Env "TAOS_ADAPTER_OPENTSDB_TELNET_FLUSH_INTERVAL"
--opentsdb_telnet.maxTCPConnections int max tcp connections. Env "TAOS_ADAPTER_OPENTSDB_TELNET_MAX_TCP_CONNECTIONS" (default 250)
--opentsdb_telnet.password string opentsdb_telnet password. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PASSWORD" (default "taosdata")
--opentsdb_telnet.ports ints opentsdb_telnet tcp port. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PORTS" (default [6046,6047,6048,6049])
--opentsdb_telnet.ports ints opentsdb telnet tcp port. Env "TAOS_ADAPTER_OPENTSDB_TELNET_PORTS" (default [6046,6047,6048,6049])
--opentsdb_telnet.tcpKeepAlive enable tcp keep alive. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TCP_KEEP_ALIVE"
--opentsdb_telnet.ttl int opentsdb_telnet data ttl. Env "TAOS_ADAPTER_OPENTSDB_TELNET_TTL"
--opentsdb_telnet.user string opentsdb_telnet user. Env "TAOS_ADAPTER_OPENTSDB_TELNET_USER" (default "root")
@ -131,6 +197,9 @@ Usage of taosAdapter:
--prometheus.enable enable prometheus. Env "TAOS_ADAPTER_PROMETHEUS_ENABLE" (default true)
--restfulRowLimit int restful returns the maximum number of rows (-1 means no limit). Env "TAOS_ADAPTER_RESTFUL_ROW_LIMIT" (default -1)
--smlAutoCreateDB Whether to automatically create db when writing with schemaless. Env "TAOS_ADAPTER_SML_AUTO_CREATE_DB"
--ssl.certFile string ssl cert file path. Env "TAOS_ADAPTER_SSL_CERT_FILE"
--ssl.enable enable ssl. Env "TAOS_ADAPTER_SSL_ENABLE"
--ssl.keyFile string ssl key file path. Env "TAOS_ADAPTER_SSL_KEY_FILE"
--statsd.allowPendingMessages int statsd allow pending messages. Env "TAOS_ADAPTER_STATSD_ALLOW_PENDING_MESSAGES" (default 50000)
--statsd.db string statsd db name. Env "TAOS_ADAPTER_STATSD_DB" (default "statsd")
--statsd.deleteCounters statsd delete counter cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_COUNTERS" (default true)
@ -157,27 +226,44 @@ Usage of taosAdapter:
-V, --version Print the version and exit
```
Note:
When using a browser to make API calls, please set the following Cross-Origin Resource Sharing (CORS) parameters according to the actual situation:
See the example configuration file at [example/config/taosadapter.toml](https://github.com/taosdata/taosadapter/blob/3.0/example/config/taosadapter.toml).
```text
AllowAllOrigins
AllowOrigins
AllowHeaders
ExposeHeaders
AllowCredentials
AllowWebSockets
```
### Cross-Origin Configuration
When making API calls from the browser, please configure the following Cross-Origin Resource Sharing (CORS) parameters based on your actual situation:
- **`cors.allowAllOrigins`**: Whether to allow all origins to access, default is true.
- **`cors.allowOrigins`**: A comma-separated list of origins allowed to access. Multiple origins can be specified.
- **`cors.allowHeaders`**: A comma-separated list of request headers allowed for cross-origin access. Multiple headers can be specified.
- **`cors.exposeHeaders`**: A comma-separated list of response headers exposed for cross-origin access. Multiple headers can be specified.
- **`cors.allowCredentials`**: Whether to allow cross-origin requests to include user credentials, such as cookies, HTTP authentication information, or client SSL certificates.
- **`cors.allowWebSockets`**: Whether to allow WebSockets connections.
If you are not making API calls through a browser, you do not need to worry about these configurations.
The above configurations take effect for the following interfaces:
* RESTful API requests
* WebSocket API requests
* InfluxDB v1 write interface
* OpenTSDB HTTP write interface
For details about the CORS protocol, please refer to: [https://www.w3.org/wiki/CORS_Enabled](https://www.w3.org/wiki/CORS_Enabled) or [https://developer.mozilla.org/docs/Web/HTTP/CORS](https://developer.mozilla.org/docs/Web/HTTP/CORS).
See the example configuration file at [example/config/taosadapter.toml](https://github.com/taosdata/taosadapter/blob/3.0/example/config/taosadapter.toml).
### Connection Pool Configuration
### Connection Pool Parameters Description
taosAdapter uses a connection pool to manage connections to TDengine, improving concurrency performance and resource utilization. The connection pool configuration applies to the following interfaces, and these interfaces share a single connection pool:
When using the RESTful API, the system will manage TDengine connections through a connection pool. The connection pool can be configured with the following parameters:
* RESTful API requests
* InfluxDB v1 write interface
* OpenTSDB JSON and telnet format writing
* Telegraf data writing
* collectd data writing
* StatsD data writing
* node_exporter data collection writing
* Prometheus remote_read and remote_write
The configuration parameters for the connection pool are as follows:
- **`pool.maxConnect`**: The maximum number of connections allowed in the pool, default is twice the number of CPU cores. It is recommended to keep the default setting.
- **`pool.maxIdle`**: The maximum number of idle connections in the pool, default is the same as `pool.maxConnect`. It is recommended to keep the default setting.
@ -185,153 +271,136 @@ When using the RESTful API, the system will manage TDengine connections through
- **`pool.waitTimeout`**: Timeout for obtaining a connection from the pool, default is set to 60 seconds. If a connection is not obtained within the timeout period, HTTP status code 503 will be returned. This parameter is available starting from version 3.3.3.0.
- **`pool.maxWait`**: The maximum number of requests waiting to get a connection in the pool, default is 0, which means no limit. When the number of queued requests exceeds this value, new requests will return HTTP status code 503. This parameter is available starting from version 3.3.3.0.
## Feature List
### HTTP Response Code Configuration
- RESTful API
[RESTful API](../../client-libraries/rest-api/)
- Compatible with InfluxDB v1 write interface
[https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/](https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/)
- Compatible with OpenTSDB JSON and telnet format writing
- [http://opentsdb.net/docs/build/html/api_http/put.html](http://opentsdb.net/docs/build/html/api_http/put.html)
- [http://opentsdb.net/docs/build/html/api_telnet/put.html](http://opentsdb.net/docs/build/html/api_telnet/put.html)
- Seamless connection with collectd.
collectd is a system statistics collection daemon, visit [https://collectd.org/](https://collectd.org/) for more information.
- Seamless connection with StatsD.
StatsD is a simple yet powerful daemon for gathering statistics. Visit [https://github.com/statsd/statsd](https://github.com/statsd/statsd) for more information.
- Seamless connection with icinga2.
icinga2 is a software for collecting check results metrics and performance data. Visit [https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer](https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer) for more information.
- Seamless connection with tcollector.
TCollector is a client process that collects data from local collectors and pushes it to OpenTSDB. Visit [http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html](http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html) for more information.
- Seamless connection with node_exporter.
node_exporter is an exporter of machine metrics. Visit [https://github.com/prometheus/node_exporter](https://github.com/prometheus/node_exporter) for more information.
- Supports Prometheus remote_read and remote_write.
remote_read and remote_write are Prometheus's data read-write separation cluster solutions. Visit [https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis) for more information.
- Get the VGroup ID of the virtual node group (VGroup) where the table is located.
taosAdapter uses the parameter `httpCodeServerError` to set whether to return a non-200 HTTP status code when the C interface returns an error. When set to true, it will return different HTTP status codes based on the error code returned by C. See [HTTP Response Codes](../../client-libraries/rest-api/) for details.
## Interface
This configuration only affects the **RESTful interface**.
### TDengine RESTful Interface
**Parameter Description**
You can use any client that supports the HTTP protocol to write data to TDengine or query data from TDengine by accessing the RESTful interface URL `http://<fqdn>:6041/rest/sql`. For details, please refer to the [REST API documentation](../../client-libraries/rest-api/).
- **`httpCodeServerError`**:
- **When set to `true`**: Map the error code returned by the C interface to the corresponding HTTP status code.
- **When set to `false`**: Regardless of the error returned by the C interface, always return the HTTP status code `200` (default value).
### InfluxDB
### Memory limit configuration
You can use any client that supports the HTTP protocol to write data in InfluxDB compatible format to TDengine by accessing the Restful interface URL `http://<fqdn>:6041/influxdb/v1/write`.
taosAdapter will monitor the memory usage during its operation and adjust it through two thresholds. The valid value range is an integer from 1 to 100, and the unit is the percentage of system physical memory.
Supported InfluxDB parameters are as follows:
This configuration only affects the following interfaces:
- `db` specifies the database name used by TDengine
- `precision` the time precision used by TDengine
- `u` TDengine username
- `p` TDengine password
- `ttl` the lifespan of automatically created subtables, determined by the TTL parameter of the first data entry in the subtable, which cannot be updated. For more information, please refer to the TTL parameter in the [table creation document](../../sql-manual/manage-tables/).
* RESTful interface request
* InfluxDB v1 write interface
* OpenTSDB HTTP write interface
* Prometheus remote_read and remote_write interfaces
Note: Currently, InfluxDB's token authentication method is not supported, only Basic authentication and query parameter verification are supported.
Example: curl --request POST `http://127.0.0.1:6041/influxdb/v1/write?db=test` --user "root:taosdata" --data-binary "measurement,host=host1 field1=2i,field2=2.0 1577836800000000000"
**Parameter Description**
### OpenTSDB
- **`pauseQueryMemoryThreshold`**:
- When memory usage exceeds this threshold, taosAdapter will stop processing query requests.
- Default value: `70` (i.e. 70% of system physical memory).
- **`pauseAllMemoryThreshold`**:
- When memory usage exceeds this threshold, taosAdapter will stop processing all requests (including writes and queries).
- Default value: `80` (i.e. 80% of system physical memory).
You can use any client that supports the HTTP protocol to write data in OpenTSDB compatible format to TDengine by accessing the Restful interface URL `http://<fqdn>:6041/<APIEndPoint>`. EndPoint as follows:
When memory usage falls below the threshold, taosAdapter will automatically resume the corresponding function.
```text
/opentsdb/v1/put/json/<db>
/opentsdb/v1/put/telnet/<db>
```
**HTTP return content:**
### collectd
- **When `pauseQueryMemoryThreshold` is exceeded**:
- HTTP status code: `503`
- Return content: `"query memory exceeds threshold"`
<CollectD />
- **When `pauseAllMemoryThreshold` is exceeded**:
- HTTP status code: `503`
- Return content: `"memory exceeds threshold"`
### StatsD
**Status check interface:**
<StatsD />
The memory status of taosAdapter can be checked through the following interface:
- **Normal status**: `http://<fqdn>:6041/-/ping` returns `code 200`.
- **Memory exceeds threshold**:
- If the memory exceeds `pauseAllMemoryThreshold`, `code 503` is returned.
- If the memory exceeds `pauseQueryMemoryThreshold` and the request parameter contains `action=query`, `code 503` is returned.
### icinga2 OpenTSDB writer
**Related configuration parameters:**
<Icinga2 />
- **`monitor.collectDuration`**: memory monitoring interval, default value is `3s`, environment variable is `TAOS_MONITOR_COLLECT_DURATION`.
- **`monitor.incgroup`**: whether to run in a container (set to `true` for running in a container), default value is `false`, environment variable is `TAOS_MONITOR_INCGROUP`.
- **`monitor.pauseQueryMemoryThreshold`**: memory threshold (percentage) for query request pause, default value is `70`, environment variable is `TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD`.
- **`monitor.pauseAllMemoryThreshold`**: memory threshold (percentage) for query and write request pause, default value is `80`, environment variable is `TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD`.
### TCollector
You can make corresponding adjustments based on the specific project application scenario and operation strategy, and it is recommended to use operation monitoring software to monitor the system memory status in a timely manner. The load balancer can also check the operation status of taosAdapter through this interface.
<TCollector />
### Schemaless write create DB configuration
### node_exporter
Starting from **version 3.0.4.0**, taosAdapter provides the parameter `smlAutoCreateDB` to control whether to automatically create a database (DB) when writing to the schemaless protocol.
An exporter used by Prometheus that exposes hardware and operating system metrics from \*NIX kernels
The `smlAutoCreateDB` parameter only affects the following interfaces:
- Enable configuration of taosAdapter node_exporter.enable
- Set the relevant configuration for node_exporter
- Restart taosAdapter
- InfluxDB v1 write interface
- OpenTSDB JSON and telnet format writing
- Telegraf data writing
- collectd data writing
- StatsD data writing
- node_exporter data writing
### prometheus
**Parameter Description**
<Prometheus />
- **`smlAutoCreateDB`**:
- **When set to `true`**: When writing to the schemaless protocol, if the target database does not exist, taosAdapter will automatically create the database.
- **When set to `false`**: The user needs to manually create the database, otherwise the write will fail (default value).
### Getting the VGroup ID of a table
### Number of results returned configuration
You can send a POST request to the HTTP interface `http://<fqdn>:<port>/rest/sql/<db>/vgid` to get the VGroup ID of a table.
The body should be a JSON array of multiple table names.
taosAdapter provides the parameter `restfulRowLimit` to control the number of results returned by the HTTP interface.
Example: Get the VGroup ID for the database power and tables d_bind_1 and d_bind_2.
The `restfulRowLimit` parameter only affects the return results of the following interfaces:
- RESTful interface
- Prometheus remote_read interface
**Parameter Description**
- **`restfulRowLimit`**:
- **When set to a positive integer**: The number of results returned by the interface will not exceed this value.
- **When set to `-1`**: The number of results returned by the interface is unlimited (default value).
### Log configuration
1. You can set the taosAdapter log output detail level by setting the --log.level parameter or the environment variable TAOS_ADAPTER_LOG_LEVEL. Valid values ​​include: panic, fatal, error, warn, warning, info, debug, and trace.
2. Starting from **3.3.5.0 version**, taosAdapter supports dynamic modification of log level through HTTP interface. Users can dynamically adjust the log level by sending HTTP PUT request to /config interface. The authentication method of this interface is the same as /rest/sql interface, and the configuration item key-value pair in JSON format must be passed in the request body.
The following is an example of setting the log level to debug through the curl command:
```shell
curl --location 'http://127.0.0.1:6041/rest/sql/power/vgid' \
--user 'root:taosdata' \
--data '["d_bind_1","d_bind_2"]'
curl --location --request PUT 'http://127.0.0.1:6041/config' \
-u root:taosdata \
--data '{"log.level": "debug"}'
```
response:
## Service Management
```json
{"code":0,"vgIDs":[153,152]}
```
### Starting/Stopping taosAdapter
## Memory Usage Optimization Methods
On Linux systems, the taosAdapter service is managed by default by systemd. Use the command `systemctl start taosadapter` to start the taosAdapter service. Use the command `systemctl stop taosadapter` to stop the taosAdapter service.
taosAdapter will monitor its memory usage during operation and adjust it through two thresholds. Valid values range from -1 to 100 as a percentage of system physical memory.
### Upgrading taosAdapter
- pauseQueryMemoryThreshold
- pauseAllMemoryThreshold
taosAdapter and TDengine server need to use the same version. Please upgrade taosAdapter by upgrading the TDengine server.
taosAdapter deployed separately from taosd must be upgraded by upgrading the TDengine server on its server.
When the pauseQueryMemoryThreshold threshold is exceeded, it stops processing query requests.
### Removing taosAdapter
HTTP return content:
Use the command rmtaos to remove the TDengine server software, including taosAdapter.
- code 503
- body "query memory exceeds threshold"
## Monitoring Metrics
When the pauseAllMemoryThreshold threshold is exceeded, it stops processing all write and query requests.
Currently, taosAdapter only collects monitoring indicators for RESTful/WebSocket related requests. There are no monitoring indicators for other interfaces.
HTTP return content:
taosAdapter reports monitoring indicators to taosKeeper, which will be written to the monitoring database by taosKeeper. The default is the `log` database, which can be modified in the taoskeeper configuration file. The following is a detailed introduction to these monitoring indicators.
- code 503
- body "memory exceeds threshold"
When memory falls below the threshold, the corresponding functions are resumed.
Status check interface `http://<fqdn>:6041/-/ping`
- Normally returns `code 200`
- No parameters If memory exceeds pauseAllMemoryThreshold, it will return `code 503`
- Request parameter `action=query` If memory exceeds either pauseQueryMemoryThreshold or pauseAllMemoryThreshold, it will return `code 503`
Corresponding configuration parameters
```text
monitor.collectDuration Monitoring interval Environment variable "TAOS_MONITOR_COLLECT_DURATION" (default value 3s)
monitor.incgroup Whether it is running in cgroup (set to true in containers) Environment variable "TAOS_MONITOR_INCGROUP"
monitor.pauseAllMemoryThreshold Memory threshold for stopping inserts and queries Environment variable "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (default value 80)
monitor.pauseQueryMemoryThreshold Memory threshold for stopping queries Environment variable "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (default value 70)
```
You can adjust according to the specific project application scenario and operational strategy, and it is recommended to use operational monitoring software to monitor the system memory status in real time. Load balancers can also check the running status of taosAdapter through this interface.
## taosAdapter Monitoring Metrics
taosAdapter collects monitoring metrics related to REST/WebSocket requests. These monitoring metrics are reported to taosKeeper, which writes them into the monitoring database, by default the `log` database, which can be modified in the taoskeeper configuration file. Below is a detailed introduction to these monitoring metrics.
### adapter_requests table
`adapter_requests` records taosadapter monitoring data.
The `adapter_requests` table records taosAdapter monitoring data, and the fields are as follows:
| field | type | is_tag | comment |
| :--------------- | :----------- | :----- | :---------------------------------------- |
@ -354,32 +423,10 @@ taosAdapter collects monitoring metrics related to REST/WebSocket requests. Thes
| endpoint | VARCHAR | | request endpoint |
| req_type | NCHAR | tag | request type: 0 for REST, 1 for WebSocket |
## Result Return Limit
taosAdapter controls the number of results returned through the parameter `restfulRowLimit`, -1 represents no limit, default is no limit.
## Changes after upgrading httpd to taosAdapter
This parameter controls the return of the following interfaces
- `http://<fqdn>:6041/rest/sql`
- `http://<fqdn>:6041/prometheus/v1/remote_read/:db`
## Configure HTTP Return Codes
taosAdapter uses the parameter `httpCodeServerError` to set whether to return a non-200 HTTP status code when the C interface returns an error. When set to true, it will return different HTTP status codes based on the error code returned by C. See [HTTP Response Codes](../../client-libraries/rest-api/) for details.
## Configure Automatic DB Creation for Schemaless Writes
Starting from version 3.0.4.0, taosAdapter provides the parameter `smlAutoCreateDB` to control whether to automatically create a DB when writing via the schemaless protocol. The default value is false, which does not automatically create a DB, and requires the user to manually create a DB before performing schemaless writes.
## Troubleshooting
You can check the running status of taosAdapter with the command `systemctl status taosadapter`.
You can also adjust the detail level of taosAdapter log output by setting the --logLevel parameter or the environment variable TAOS_ADAPTER_LOG_LEVEL. Valid values include: panic, fatal, error, warn, warning, info, debug, and trace.
## How to Migrate from Older Versions of TDengine to taosAdapter
In TDengine server version 2.2.x.x or earlier, the taosd process included an embedded HTTP service. As mentioned earlier, taosAdapter is a standalone software managed by systemd, having its own process. Moreover, there are some differences in configuration parameters and behaviors between the two, as shown in the table below:
In TDengine server version 2.2.x.x or earlier, the taosd process included an embedded HTTP service(httpd). As mentioned earlier, taosAdapter is a standalone software managed by systemd, having its own process. Moreover, there are some differences in configuration parameters and behaviors between the two, as shown in the table below:
| **#** | **embedded httpd** | **taosAdapter** | **comment** |
| ----- | ------------------- | ---------------------------------------------------------- | ------------------------------------------------------------ |

120
docs/en/assets/resources/_collectd.mdx
View File

@ -1,84 +1,84 @@
### Configuring taosAdapter
#### Configuring taosAdapter
Method to configure taosAdapter to receive collectd data:
- Enable the configuration item in the taosAdapter configuration file (default location is /etc/taos/taosadapter.toml)
```
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
```toml
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
The default database name written by taosAdapter is `collectd`, but you can also modify the dbs item in the taosAdapter configuration file to specify a different name. Fill in user and password with the actual TDengine configuration values. After modifying the configuration file, taosAdapter needs to be restarted.
The default database name written by taosAdapter is `collectd`, but you can also modify the dbs item in the taosAdapter configuration file to specify a different name. Fill in user and password with the actual TDengine configuration values. After modifying the configuration file, taosAdapter needs to be restarted.
- You can also use taosAdapter command line parameters or set environment variables to start, to enable taosAdapter to receive collectd data, for more details please refer to the taosAdapter reference manual.
### Configuring collectd
#### Configuring collectd
collectd uses a plugin mechanism that can write the collected monitoring data to different data storage software in various forms. TDengine supports direct collection plugins and write_tsdb plugins.
#### Configuring to receive direct collection plugin data
1. **Configuring to receive direct collection plugin data**
Modify the related configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).
Modify the related configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).
```text
LoadPlugin network
<Plugin network>
Server "<taosAdapter's host>" "<port for collectd direct>"
</Plugin>
```
```xml
LoadPlugin network
<Plugin network>
Server "<taosAdapter's host>" "<port for collectd direct>"
</Plugin>
```
Where \<taosAdapter's host> should be filled with the domain name or IP address of the server running taosAdapter. \<port for collectd direct> should be filled with the port used by taosAdapter to receive collectd data (default is 6045).
Where \<taosAdapter's host> should be filled with the domain name or IP address of the server running taosAdapter. \<port for collectd direct> should be filled with the port used by taosAdapter to receive collectd data (default is 6045).
Example as follows:
Example as follows:
```text
LoadPlugin network
<Plugin network>
Server "127.0.0.1" "6045"
</Plugin>
```
```xml
LoadPlugin network
<Plugin network>
Server "127.0.0.1" "6045"
</Plugin>
```
#### Configuring write_tsdb plugin data
2. **Configuring write_tsdb plugin data**
Modify the related configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).
Modify the related configuration items in the collectd configuration file (default location /etc/collectd/collectd.conf).
```text
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "<taosAdapter's host>"
Port "<port for collectd write_tsdb plugin>"
...
</Node>
</Plugin>
```
```xml
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "<taosAdapter's host>"
Port "<port for collectd write_tsdb plugin>"
...
</Node>
</Plugin>
```
Where \<taosAdapter's host> should be filled with the domain name or IP address of the server running taosAdapter. \<port for collectd write_tsdb plugin> should be filled with the port used by taosAdapter to receive collectd write_tsdb plugin data (default is 6047).
Where \<taosAdapter's host> should be filled with the domain name or IP address of the server running taosAdapter. \<port for collectd write_tsdb plugin> should be filled with the port used by taosAdapter to receive collectd write_tsdb plugin data (default is 6047).
```text
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "127.0.0.1"
Port "6047"
HostTags "status=production"
StoreRates false
AlwaysAppendDS false
</Node>
</Plugin>
```
```xml
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "127.0.0.1"
Port "6047"
HostTags "status=production"
StoreRates false
AlwaysAppendDS false
</Node>
</Plugin>
```
Then restart collectd:
Then restart collectd:
```
systemctl restart collectd
```
```shell
systemctl restart collectd
```

56
docs/en/assets/resources/_icinga2.mdx
View File

@ -1,43 +1,43 @@
### Configuring taosAdapter
#### Configuring taosAdapter
Method to configure taosAdapter to receive icinga2 data:
- Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
```
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
```toml
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
The default database name written by taosAdapter is `icinga2`, but you can also modify the dbs item in the taosAdapter configuration file to specify a different name. Fill in user and password with the actual TDengine configuration values. taosAdapter needs to be restarted after modifications.
The default database name written by taosAdapter is `icinga2`, but you can also modify the dbs item in the taosAdapter configuration file to specify a different name. Fill in user and password with the actual TDengine configuration values. taosAdapter needs to be restarted after modifications.
- You can also use taosAdapter command line parameters or set environment variables to enable taosAdapter to receive icinga2 data, for more details please refer to the taosAdapter reference manual
### Configuring icinga2
#### Configuring icinga2
- Enable icinga2's opentsdb-writer (reference link https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer)
- Modify the configuration file `/etc/icinga2/features-enabled/opentsdb.conf` filling in \<taosAdapter's host> with the domain name or IP address of the server running taosAdapter, \<port for icinga2> with the corresponding port supported by taosAdapter for receiving icinga2 data (default is 6048)
```
object OpenTsdbWriter "opentsdb" {
host = "<taosAdapter's host>"
port = <port for icinga2>
}
```
```c
object OpenTsdbWriter "opentsdb" {
host = "<taosAdapter's host>"
port = <port for icinga2>
}
```
Example file:
Example file:
```
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 6048
}
```
```c
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 6048
}
```

6
docs/en/assets/resources/_prometheus.mdx
View File

@ -1,18 +1,18 @@
Configuring Prometheus is done by editing the Prometheus configuration file `prometheus.yml` (default location `/etc/prometheus/prometheus.yml`).
### Configure Third-Party Database Address
#### Configure Third-Party Database Address
Set the `remote_read url` and `remote_write url` to point to the domain name or IP address of the server running the taosAdapter service, the REST service port (taosAdapter defaults to 6041), and the name of the database you want to write to in TDengine, ensuring the URLs are formatted as follows:
- remote_read url: `http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_read/<database name>`
- remote_write url: `http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_write/<database name>`
### Configure Basic Authentication
#### Configure Basic Authentication
- username: \<TDengine's username>
- password: \<TDengine's password>
### Example configuration of remote_write and remote_read in the prometheus.yml file
#### Example configuration of remote_write and remote_read in the prometheus.yml file
```yaml
remote_write:

52
docs/en/assets/resources/_statsd.mdx
View File

@ -1,46 +1,46 @@
### Configure taosAdapter
#### Configure taosAdapter
Method to configure taosAdapter to receive StatsD data:
- Enable the configuration item in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
```
...
[statsd]
enable = true
port = 6044
db = "statsd"
user = "root"
password = "taosdata"
worker = 10
gatherInterval = "5s"
protocol = "udp"
maxTCPConnections = 250
tcpKeepAlive = false
allowPendingMessages = 50000
deleteCounters = true
deleteGauges = true
deleteSets = true
deleteTimings = true
...
```
```toml
...
[statsd]
enable = true
port = 6044
db = "statsd"
user = "root"
password = "taosdata"
worker = 10
gatherInterval = "5s"
protocol = "udp"
maxTCPConnections = 250
tcpKeepAlive = false
allowPendingMessages = 50000
deleteCounters = true
deleteGauges = true
deleteSets = true
deleteTimings = true
...
```
The default database name written by taosAdapter is `statsd`, but you can also modify the db item in the taosAdapter configuration file to specify a different name. Fill in the user and password with the actual TDengine configuration values. After modifying the configuration file, taosAdapter needs to be restarted.
The default database name written by taosAdapter is `statsd`, but you can also modify the db item in the taosAdapter configuration file to specify a different name. Fill in the user and password with the actual TDengine configuration values. After modifying the configuration file, taosAdapter needs to be restarted.
- You can also use taosAdapter command line arguments or set environment variables to enable the taosAdapter to receive StatsD data. For more details, please refer to the taosAdapter reference manual.
### Configure StatsD
#### Configure StatsD
To use StatsD, download its [source code](https://github.com/statsd/statsd). Modify its configuration file according to the example file `exampleConfig.js` found in the root directory of the local source code download. Replace \<taosAdapter's host> with the domain name or IP address of the server running taosAdapter, and \<port for StatsD> with the port that taosAdapter uses to receive StatsD data (default is 6044).
```
```text
Add to the backends section "./backends/repeater"
Add to the repeater section { host:'<taosAdapter's host>', port: <port for StatsD>}
```
Example configuration file:
```
```js
{
port: 8125
, backends: ["./backends/repeater"]
@ -50,7 +50,7 @@ port: 8125
After adding the following content, start StatsD (assuming the configuration file is modified to config.js).
```
```shell
npm install
node stats.js config.js &
```

32
docs/en/assets/resources/_tcollector.mdx
View File

@ -1,27 +1,27 @@
### Configuring taosAdapter
#### Configuring taosAdapter
To configure taosAdapter to receive data from TCollector:
- Enable the configuration in the taosAdapter configuration file (default location /etc/taos/taosadapter.toml)
```
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
```toml
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
The default database name that taosAdapter writes to is `tcollector`, but you can specify a different name by modifying the dbs option in the taosAdapter configuration file. Fill in the user and password with the actual values configured in TDengine. After modifying the configuration file, taosAdapter needs to be restarted.
The default database name that taosAdapter writes to is `tcollector`, but you can specify a different name by modifying the dbs option in the taosAdapter configuration file. Fill in the user and password with the actual values configured in TDengine. After modifying the configuration file, taosAdapter needs to be restarted.
- You can also use taosAdapter command line arguments or set environment variables to enable the taosAdapter to receive tcollector data. For more details, please refer to the taosAdapter reference manual.
### Configuring TCollector
#### Configuring TCollector
To use TCollector, download its [source code](https://github.com/OpenTSDB/tcollector). Its configuration options are in its source code. Note: There are significant differences between different versions of TCollector; this only refers to the latest code in the current master branch (git commit: 37ae920).
@ -29,7 +29,7 @@ Modify the contents of `collectors/etc/config.py` and `tcollector.py`. Change th
Example of git diff output for source code modifications:
```
```diff
index e7e7a1c..ec3e23c 100644
--- a/collectors/etc/config.py
+++ b/collectors/etc/config.py

476
docs/zh/14-reference/01-components/03-taosadapter.md
View File

@ -11,42 +11,109 @@ import Icinga2 from "./_icinga2.mdx"
import TCollector from "./_tcollector.mdx"
taosAdapter 是一个 TDengine 的配套工具，是 TDengine 集群和应用程序之间的桥梁和适配器。它提供了一种易于使用和高效的方式来直接从数据收集代理软件（如 Telegraf、StatsD、collectd 等）摄取数据。它还提供了 InfluxDB/OpenTSDB 兼容的数据摄取接口，允许 InfluxDB/OpenTSDB 应用程序无缝移植到 TDengine。
TDengine 的各语言连接器通过 WebSocket 接口与 TDengine 进行通信，因此必须安装 taosAdapter。
taosAdapter 提供以下功能：
- Websocket/RESTful 接口
- 兼容 InfluxDB v1 写接口
- 兼容 OpenTSDB JSON 和 telnet 格式写入
- 无缝连接到 Telegraf
- 无缝连接到 collectd
- 无缝连接到 StatsD
- 支持 Prometheus remote_read 和 remote_write
- 获取 table 所在的虚拟节点组（VGroup）的 VGroup ID
## taosAdapter 架构图
架构图如下：
![TDengine Database taosAdapter Architecture](taosAdapter-architecture.webp)
## taosAdapter 部署方法
## 功能列表
### 安装 taosAdapter
taosAdapter 提供了以下功能：
- WebSocket 接口：
支持通过 WebSocket 协议执行 SQL、无模式数据写入、参数绑定和数据订阅功能。
- 兼容 InfluxDB v1 写接口：
[https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/](https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/)
- 兼容 OpenTSDB JSON 和 telnet 格式写入：
- [http://opentsdb.net/docs/build/html/api_http/put.html](http://opentsdb.net/docs/build/html/api_http/put.html)
- [http://opentsdb.net/docs/build/html/api_telnet/put.html](http://opentsdb.net/docs/build/html/api_telnet/put.html)
- collectd 数据写入：
collectd 是一个系统统计收集守护程序，请访问 [https://collectd.org/](https://collectd.org/) 了解更多信息。
- StatsD 数据写入：
StatsD 是一个简单而强大的统计信息汇总的守护程序。请访问 [https://github.com/statsd/statsd](https://github.com/statsd/statsd) 了解更多信息。
- icinga2 OpenTSDB writer 数据写入：
icinga2 是一个收集检查结果指标和性能数据的软件。请访问 [https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer](https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer) 了解更多信息。
- TCollector 数据写入：
TCollector是一个客户端进程，从本地收集器收集数据，并将数据推送到 OpenTSDB。请访问 [http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html](http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html) 了解更多信息。
- node_exporter 采集写入：
node_export 是一个机器指标的导出器。请访问 [https://github.com/prometheus/node_exporter](https://github.com/prometheus/node_exporter) 了解更多信息。
- Prometheus remote_read 和 remote_write：
remote_read 和 remote_write 是 Prometheus 数据读写分离的集群方案。请访问[https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis) 了解更多信息。
- RESTful 接口：
[RESTful API](../../connector/rest-api)
### WebSocket 接口
各语言连接器通过 taosAdapter 的 WebSocket 接口，能够实现 SQL 执行、无模式写入、参数绑定和数据订阅功能。参考[开发指南](../../../develop/connect/#websocket-连接)。
### 兼容 InfluxDB v1 写接口
您可以使用任何支持 HTTP 协议的客户端访问 Restful 接口地址 `http://<fqdn>:6041/influxdb/v1/write` 来写入 InfluxDB 兼容格式的数据到 TDengine。
支持 InfluxDB 参数如下：
- `db` 指定 TDengine 使用的数据库名
- `precision` TDengine 使用的时间精度
- `u` TDengine 用户名
- `p` TDengine 密码
- `ttl` 自动创建的子表生命周期，以子表的第一条数据的 TTL 参数为准，不可更新。更多信息请参考[创建表文档](../../taos-sql/table/#创建表)的 TTL 参数。
注意： 目前不支持 InfluxDB 的 token 验证方式，仅支持 Basic 验证和查询参数验证。
示例：
```shell
curl --request POST http://127.0.0.1:6041/influxdb/v1/write?db=test --user "root:taosdata" --data-binary "measurement,host=host1 field1=2i,field2=2.0 1577836800000000000"
```
### 兼容 OpenTSDB JSON 和 telnet 格式写入
您可以使用任何支持 HTTP 协议的客户端访问 Restful 接口地址 `http://<fqdn>:6041/<APIEndPoint>` 来写入 OpenTSDB 兼容格式的数据到 TDengine。EndPoint 如下：
```text
/opentsdb/v1/put/json/<db>
/opentsdb/v1/put/telnet/<db>
```
### collectd 数据写入
<CollectD />
### StatsD 数据写入
<StatsD />
### icinga2 OpenTSDB writer 数据写入
<Icinga2 />
### TCollector 数据写入
<TCollector />
### node_exporter 采集写入
Prometheus 使用的由 \*NIX 内核暴露的硬件和操作系统指标的输出器
- 启用 taosAdapter 的配置 node_exporter.enable
- 设置 node_exporter 的相关配置
- 重新启动 taosAdapter
### Prometheus remote_read 和 remote_write
<Prometheus />
### RESTful 接口
您可以使用任何支持 HTTP 协议的客户端通过访问 RESTful 接口地址 `http://<fqdn>:6041/rest/sql` 来写入数据到 TDengine 或从 TDengine 中查询数据。细节请参考[REST API 文档](../../connector/rest-api/)。
## 安装
taosAdapter 是 TDengine 服务端软件 的一部分，如果您使用 TDengine server 您不需要任何额外的步骤来安装 taosAdapter。您可以从[涛思数据官方网站](https://docs.taosdata.com/releases/tdengine/)下载 TDengine server 安装包。如果需要将 taosAdapter 分离部署在 TDengine server 之外的服务器上，则应该在该服务器上安装完整的 TDengine 来安装 taosAdapter。如果您需要使用源代码编译生成 taosAdapter，您可以参考[构建 taosAdapter](https://github.com/taosdata/taosadapter/blob/3.0/BUILD-CN.md)文档。
### 启动/停止 taosAdapter
安装完成后使用命令 `systemctl start taosadapter` 可以启动 taosAdapter 服务。
在 Linux 系统上 taosAdapter 服务默认由 systemd 管理。使用命令 `systemctl start taosadapter` 可以启动 taosAdapter 服务。使用命令 `systemctl stop taosadapter` 可以停止 taosAdapter 服务。
### 移除 taosAdapter
使用命令 rmtaos 可以移除包括 taosAdapter 在内的 TDengine server 软件。
### 升级 taosAdapter
taosAdapter 和 TDengine server 需要使用相同版本。请通过升级 TDengine server 来升级 taosAdapter。
与 taosd 分离部署的 taosAdapter 必须通过升级其所在服务器的 TDengine server 才能得到升级。
## taosAdapter 参数列表
## 配置
taosAdapter 支持通过命令行参数、环境变量和配置文件来进行配置。默认配置文件是 /etc/taos/taosadapter.toml。
@ -75,6 +142,7 @@ Usage of taosAdapter:
--instanceId int instance ID. Env "TAOS_ADAPTER_INSTANCE_ID" (default 32)
--log.compress whether to compress old log. Env "TAOS_ADAPTER_LOG_COMPRESS"
--log.enableRecordHttpSql whether to record http sql. Env "TAOS_ADAPTER_LOG_ENABLE_RECORD_HTTP_SQL"
--log.keepDays uint log retention days, must be a positive integer. Env "TAOS_ADAPTER_LOG_KEEP_DAYS" (default 30)
--log.level string log level (trace debug info warning error). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
--log.path string log path. Env "TAOS_ADAPTER_LOG_PATH" (default "/var/log/taos")
--log.reservedDiskSize string reserved disk size for log dir (KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_RESERVED_DISK_SIZE" (default "1GB")
@ -85,6 +153,8 @@ Usage of taosAdapter:
--log.sqlRotationSize string record sql log rotation size(KB MB GB), must be a positive integer. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_SIZE" (default "1GB")
--log.sqlRotationTime duration record sql log rotation time. Env "TAOS_ADAPTER_LOG_SQL_ROTATION_TIME" (default 24h0m0s)
--logLevel string log level (trace debug info warning error). Env "TAOS_ADAPTER_LOG_LEVEL" (default "info")
--maxAsyncConcurrentLimit int The maximum number of concurrent calls allowed for the C asynchronous method. 0 means use CPU core count. Env "TAOS_ADAPTER_MAX_ASYNC_CONCURRENT_LIMIT"
--maxSyncConcurrentLimit int The maximum number of concurrent calls allowed for the C synchronized method. 0 means use CPU core count. Env "TAOS_ADAPTER_MAX_SYNC_CONCURRENT_LIMIT"
--monitor.collectDuration duration Set monitor duration. Env "TAOS_ADAPTER_MONITOR_COLLECT_DURATION" (default 3s)
--monitor.disable Whether to disable monitoring. Env "TAOS_ADAPTER_MONITOR_DISABLE" (default true)
--monitor.identity string The identity of the current instance, or 'hostname:port' if it is empty. Env "TAOS_ADAPTER_MONITOR_IDENTITY"
@ -126,6 +196,9 @@ Usage of taosAdapter:
--prometheus.enable enable prometheus. Env "TAOS_ADAPTER_PROMETHEUS_ENABLE" (default true)
--restfulRowLimit int restful returns the maximum number of rows (-1 means no limit). Env "TAOS_ADAPTER_RESTFUL_ROW_LIMIT" (default -1)
--smlAutoCreateDB Whether to automatically create db when writing with schemaless. Env "TAOS_ADAPTER_SML_AUTO_CREATE_DB"
--ssl.certFile string ssl cert file path. Env "TAOS_ADAPTER_SSL_CERT_FILE"
--ssl.enable enable ssl. Env "TAOS_ADAPTER_SSL_ENABLE"
--ssl.keyFile string ssl key file path. Env "TAOS_ADAPTER_SSL_KEY_FILE"
--statsd.allowPendingMessages int statsd allow pending messages. Env "TAOS_ADAPTER_STATSD_ALLOW_PENDING_MESSAGES" (default 50000)
--statsd.db string statsd db name. Env "TAOS_ADAPTER_STATSD_DB" (default "statsd")
--statsd.deleteCounters statsd delete counter cache after gather. Env "TAOS_ADAPTER_STATSD_DELETE_COUNTERS" (default true)
@ -152,27 +225,44 @@ Usage of taosAdapter:
-V, --version Print the version and exit
```
备注：
使用浏览器进行接口调用请根据实际情况设置如下跨源资源共享（CORS）参数：
示例配置文件参见 [example/config/taosadapter.toml](https://github.com/taosdata/taosadapter/blob/3.0/example/config/taosadapter.toml)。
```text
AllowAllOrigins
AllowOrigins
AllowHeaders
ExposeHeaders
AllowCredentials
AllowWebSockets
```
### 跨域配置
使用浏览器进行接口调用时，请根据实际情况设置如下跨域（CORS）参数：
- **`cors.allowAllOrigins`**：是否允许所有来源访问，默认为 `true`。
- **`cors.allowOrigins`**：允许跨域访问的来源列表，支持多个来源，以逗号分隔。
- **`cors.allowHeaders`**：允许跨域访问的请求头列表，支持多个请求头，以逗号分隔。
- **`cors.exposeHeaders`**：允许跨域访问的响应头列表，支持多个响应头，以逗号分隔。
- **`cors.allowCredentials`**：是否允许跨域请求包含用户凭证，如 cookies、HTTP 认证信息或客户端 SSL 证书。
- **`cors.allowWebSockets`**：是否允许 WebSockets 连接。
如果不通过浏览器进行接口调用无需关心这几项配置。
以上配置对以下接口生效：
* RESTful 接口请求
* WebSocket 接口请求
* InfluxDB v1 写接口
* OpenTSDB HTTP 写入接口
关于 CORS 协议细节请参考：[https://www.w3.org/wiki/CORS_Enabled](https://www.w3.org/wiki/CORS_Enabled) 或 [https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS](https://developer.mozilla.org/zh-CN/docs/Web/HTTP/CORS)。
示例配置文件参见 [example/config/taosadapter.toml](https://github.com/taosdata/taosadapter/blob/3.0/example/config/taosadapter.toml)。
### 连接池配置
### 连接池参数说明
taosAdapter 使用连接池管理与 TDengine 的连接，以提高并发性能和资源利用率。连接池配置对以下接口生效，且以下接口共享一个连接池：
在使用 RESTful 接口请求时，系统将通过连接池管理 TDengine 连接。连接池可通过以下参数进行配置：
* RESTful 接口请求
* InfluxDB v1 写接口
* OpenTSDB JSON 和 telnet 格式写入
* Telegraf 数据写入
* collectd 数据写入
* StatsD 数据写入
* 采集 node_exporter 数据写入
* Prometheus remote_read 和 remote_write
连接池的配置参数如下：
- **`pool.maxConnect`**：连接池允许的最大连接数，默认值为 2 倍 CPU 核心数。建议保持默认设置。
- **`pool.maxIdle`**：连接池中允许的最大空闲连接数，默认与 `pool.maxConnect` 相同。建议保持默认设置。
@ -180,205 +270,167 @@ AllowWebSockets
- **`pool.waitTimeout`**：从连接池获取连接的超时时间，默认设置为 60 秒。如果在超时时间内未能获取连接，将返回 HTTP 状态码 503。该参数从版本 3.3.3.0 开始提供。
- **`pool.maxWait`**：连接池中等待获取连接的请求数上限，默认值为 0，表示不限制。当排队请求数超过此值时，新的请求将返回 HTTP 状态码 503。该参数从版本 3.3.3.0 开始提供。
## 功能列表
### HTTP 返回码配置
- RESTful 接口
[RESTful API](../../connector/rest-api)
- 兼容 InfluxDB v1 写接口
[https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/](https://docs.influxdata.com/influxdb/v2.0/reference/api/influxdb-1x/write/)
- 兼容 OpenTSDB JSON 和 telnet 格式写入
- [http://opentsdb.net/docs/build/html/api_http/put.html](http://opentsdb.net/docs/build/html/api_http/put.html)
- [http://opentsdb.net/docs/build/html/api_telnet/put.html](http://opentsdb.net/docs/build/html/api_telnet/put.html)
- 与 collectd 无缝连接。
collectd 是一个系统统计收集守护程序，请访问 [https://collectd.org/](https://collectd.org/) 了解更多信息。
- Seamless connection with StatsD。
StatsD 是一个简单而强大的统计信息汇总的守护程序。请访问 [https://github.com/statsd/statsd](https://github.com/statsd/statsd) 了解更多信息。
- 与 icinga2 的无缝连接。
icinga2 是一个收集检查结果指标和性能数据的软件。请访问 [https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer](https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer) 了解更多信息。
- 与 tcollector 无缝连接。
TCollector是一个客户端进程，从本地收集器收集数据，并将数据推送到 OpenTSDB。请访问 [http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html](http://opentsdb.net/docs/build/html/user_guide/utilities/tcollector.html) 了解更多信息。
- 无缝连接 node_exporter。
node_export 是一个机器指标的导出器。请访问 [https://github.com/prometheus/node_exporter](https://github.com/prometheus/node_exporter) 了解更多信息。
- 支持 Prometheus remote_read 和 remote_write。
remote_read 和 remote_write 是 Prometheus 数据读写分离的集群方案。请访问[https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis](https://prometheus.io/blog/2019/10/10/remote-read-meets-streaming/#remote-apis) 了解更多信息。
- 获取 table 所在的虚拟节点组（VGroup）的 VGroup ID。
taosAdapter 通过参数 `httpCodeServerError` 来控制当底层 C 接口返回错误时，是否在 RESTful 接口请求中返回非 200 的 HTTP 状态码。当设置为 `true` 时，taosAdapter 会根据 C 接口返回的错误码映射为相应的 HTTP 状态码。具体映射规则请参考 [HTTP 响应码](../../connector/rest-api/#http-响应码)。
## 接口
该配置只会影响 **RESTful 接口**。
### TDengine RESTful 接口
**参数说明**
您可以使用任何支持 http 协议的客户端通过访问 RESTful 接口地址 `http://<fqdn>:6041/rest/sql` 来写入数据到 TDengine 或从 TDengine 中查询数据。细节请参考[REST API 文档](../../connector/rest-api/)。
- **`httpCodeServerError`**：
- **设置为 `true` 时**：根据 C 接口返回的错误码映射为相应的 HTTP 状态码。
- **设置为 `false` 时**：无论 C 接口返回什么错误，始终返回 HTTP 状态码 `200`（默认值）。
### InfluxDB
您可以使用任何支持 http 协议的客户端访问 Restful 接口地址 `http://<fqdn>:6041/influxdb/v1/write` 来写入 InfluxDB 兼容格式的数据到 TDengine。
### 内存限制配置
支持 InfluxDB 参数如下：
taosAdapter 将监测自身运行过程中内存使用率并通过两个阈值进行调节。有效值范围为 1 到 100 的整数，单位为系统物理内存的百分比。
- `db` 指定 TDengine 使用的数据库名
- `precision` TDengine 使用的时间精度
- `u` TDengine 用户名
- `p` TDengine 密码
- `ttl` 自动创建的子表生命周期，以子表的第一条数据的 TTL 参数为准，不可更新。更多信息请参考[创建表文档](../../taos-sql/table/#创建表)的 TTL 参数。
该配置只会影响以下接口：
注意： 目前不支持 InfluxDB 的 token 验证方式，仅支持 Basic 验证和查询参数验证。
示例： curl --request POST http://127.0.0.1:6041/influxdb/v1/write?db=test --user "root:taosdata" --data-binary "measurement,host=host1 field1=2i,field2=2.0 1577836800000000000"
### OpenTSDB
* RESTful 接口请求
* InfluxDB v1 写接口
* OpenTSDB HTTP 写入接口
* Prometheus remote_read 和 remote_write 接口
您可以使用任何支持 http 协议的客户端访问 Restful 接口地址 `http://<fqdn>:6041/<APIEndPoint>` 来写入 OpenTSDB 兼容格式的数据到 TDengine。EndPoint 如下：
**参数说明**
```text
/opentsdb/v1/put/json/<db>
/opentsdb/v1/put/telnet/<db>
```
- **`pauseQueryMemoryThreshold`**：
- 当内存使用超过此阈值时，taosAdapter 将停止处理查询请求。
- 默认值：`70`（即 70% 的系统物理内存）。
- **`pauseAllMemoryThreshold`**：
- 当内存使用超过此阈值时，taosAdapter 将停止处理所有请求（包括写入和查询）。
- 默认值：`80`（即 80% 的系统物理内存）。
### collectd
当内存使用回落到阈值以下时，taosAdapter 会自动恢复相应功能。
<CollectD />
**HTTP 返回内容：**
### StatsD
- **超过 `pauseQueryMemoryThreshold` 时**：
- HTTP 状态码：`503`
- 返回内容：`"query memory exceeds threshold"`
- **超过 `pauseAllMemoryThreshold` 时**：
- HTTP 状态码：`503`
- 返回内容：`"memory exceeds threshold"`
<StatsD />
**状态检查接口：**
### icinga2 OpenTSDB writer
可以通过以下接口检查 taosAdapter 的内存状态：
- **正常状态**：`http://<fqdn>:6041/-/ping` 返回 `code 200`。
- **内存超过阈值**：
- 如果内存超过 `pauseAllMemoryThreshold`，返回 `code 503`。
- 如果内存超过 `pauseQueryMemoryThreshold`，且请求参数包含 `action=query`，返回 `code 503`。
<Icinga2 />
**相关配置参数：**
### TCollector
<TCollector />
### node_exporter
Prometheus 使用的由 \*NIX 内核暴露的硬件和操作系统指标的输出器
- 启用 taosAdapter 的配置 node_exporter.enable
- 设置 node_exporter 的相关配置
- 重新启动 taosAdapter
### prometheus
<Prometheus />
### 获取 table 的 VGroup ID
可以 POST 请求 http 接口 `http://<fqdn>:<port>/rest/sql/<db>/vgid` 获取 table 的 VGroup ID，body 是多个表名 JSON 数组。
样例：获取数据库为 power，表名为 d_bind_1 和 d_bind_2 的 VGroup ID
```shell
curl --location 'http://127.0.0.1:6041/rest/sql/power/vgid' \
--user 'root:taosdata' \
--data '["d_bind_1","d_bind_2"]'
```
响应：
```json
{"code":0,"vgIDs":[153,152]}
```
## 内存使用优化方法
taosAdapter 将监测自身运行过程中内存使用率并通过两个阈值进行调节。有效值范围为 -1 到 100 的整数，单位为系统物理内存的百分比。
- pauseQueryMemoryThreshold
- pauseAllMemoryThreshold
当超过 pauseQueryMemoryThreshold 阈值时时停止处理查询请求。
http 返回内容：
- code 503
- body "query memory exceeds threshold"
当超过 pauseAllMemoryThreshold 阈值时停止处理所有写入和查询请求。
http 返回内容：
- code 503
- body "memory exceeds threshold"
当内存回落到阈值之下时恢复对应功能。
状态检查接口 `http://<fqdn>:6041/-/ping`
- 正常返回 `code 200`
- 无参数 如果内存超过 pauseAllMemoryThreshold 将返回 `code 503`
- 请求参数 `action=query` 如果内存超过 pauseQueryMemoryThreshold 或 pauseAllMemoryThreshold 将返回 `code 503`
对应配置参数
```text
monitor.collectDuration 监测间隔 环境变量 "TAOS_MONITOR_COLLECT_DURATION" (默认值 3s)
monitor.incgroup 是否是cgroup中运行(容器中运行设置为 true) 环境变量 "TAOS_MONITOR_INCGROUP"
monitor.pauseAllMemoryThreshold 不再进行插入和查询的内存阈值 环境变量 "TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD" (默认值 80)
monitor.pauseQueryMemoryThreshold 不再进行查询的内存阈值 环境变量 "TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD" (默认值 70)
```
- **`monitor.collectDuration`**：内存监控间隔，默认值为 `3s`，环境变量为 `TAOS_MONITOR_COLLECT_DURATION`。
- **`monitor.incgroup`**：是否在容器中运行（容器中运行设置为 `true`），默认值为 `false`，环境变量为 `TAOS_MONITOR_INCGROUP`。
- **`monitor.pauseQueryMemoryThreshold`**：查询请求暂停的内存阈值（百分比），默认值为 `70`，环境变量为 `TAOS_MONITOR_PAUSE_QUERY_MEMORY_THRESHOLD`。
- **`monitor.pauseAllMemoryThreshold`**：查询和写入请求暂停的内存阈值（百分比），默认值为 `80`，环境变量为 `TAOS_MONITOR_PAUSE_ALL_MEMORY_THRESHOLD`。
您可以根据具体项目应用场景和运营策略进行相应调整，并建议使用运营监控软件及时进行系统内存状态监控。负载均衡器也可以通过这个接口检查 taosAdapter 运行状态。
## taosAdapter 监控指标
### 无模式写入创建 DB 配置
taosAdapter 采集 REST/WebSocket 相关请求的监控指标。将监控指标上报给 taosKeeper，这些监控指标会被 taosKeeper 写入监控数据库，默认是 `log` 库，可以在 taoskeeper 配置文件中修改。以下是这些监控指标的详细介绍。
从 **3.0.4.0 版本** 开始，taosAdapter 提供了参数 `smlAutoCreateDB`，用于控制在 schemaless 协议写入时是否自动创建数据库（DB）。
#### adapter\_requests 表
`smlAutoCreateDB` 参数只会影响以下接口：
`adapter_requests` 记录 taosadapter 监控数据。
- InfluxDB v1 写接口
- OpenTSDB JSON 和 telnet 格式写入
- Telegraf 数据写入
- collectd 数据写入
- StatsD 数据写入
- node_exporter 数据写入
| field | type | is\_tag | comment |
| :----------------- | :----------- | :------ | :---------------------------------- |
| ts | TIMESTAMP | | timestamp |
| total | INT UNSIGNED | | 总请求数 |
| query | INT UNSIGNED | | 查询请求数 |
| write | INT UNSIGNED | | 写入请求数 |
| other | INT UNSIGNED | | 其他请求数 |
| in\_process | INT UNSIGNED | | 正在处理请求数 |
| success | INT UNSIGNED | | 成功请求数 |
| fail | INT UNSIGNED | | 失败请求数 |
| query\_success | INT UNSIGNED | | 查询成功请求数 |
| query\_fail | INT UNSIGNED | | 查询失败请求数 |
| write\_success | INT UNSIGNED | | 写入成功请求数 |
| write\_fail | INT UNSIGNED | | 写入失败请求数 |
| other\_success | INT UNSIGNED | | 其他成功请求数 |
| other\_fail | INT UNSIGNED | | 其他失败请求数 |
| query\_in\_process | INT UNSIGNED | | 正在处理查询请求数 |
| write\_in\_process | INT UNSIGNED | | 正在处理写入请求数 |
| endpoint | VARCHAR | | 请求端点 |
**参数说明**
- **`smlAutoCreateDB`**：
- **设置为 `true` 时**：在 schemaless 协议写入时，如果目标数据库不存在，taosAdapter 会自动创建该数据库。
- **设置为 `false` 时**：用户需要手动创建数据库，否则写入会失败（默认值）。
### 结果返回条数配置
taosAdapter 提供了参数 `restfulRowLimit`，用于控制 HTTP 接口返回的结果条数。
`restfulRowLimit` 参数只会影响以下接口的返回结果：
- RESTful 接口
- Prometheus remote_read 接口
**参数说明**
- **`restfulRowLimit`**：
- **设置为正整数时**：接口返回的结果条数将不超过该值。
- **设置为 `-1` 时**：接口返回的结果条数无限制（默认值）。
### 日志配置
1. 可以通过设置 --log.level 参数或者环境变量 TAOS_ADAPTER_LOG_LEVEL 来设置 taosAdapter 日志输出详细程度。有效值包括：panic、fatal、error、warn、warning、info、debug 以及 trace。
2. 从 **3.3.5.0 版本** 开始，taosAdapter 支持通过 HTTP 接口动态修改日志级别。用户可以通过发送 HTTP PUT 请求到 /config 接口，动态调整日志级别。该接口的验证方式与 /rest/sql 接口相同，请求体中需传入 JSON 格式的配置项键值对。
以下是通过 curl 命令将日志级别设置为 debug 的示例：
```shell
curl --location --request PUT 'http://127.0.0.1:6041/config' \
-u root:taosdata \
--data '{"log.level": "debug"}'
```
## 服务管理
### 启动/停止 taosAdapter
在 Linux 系统上 taosAdapter 服务默认由 systemd 管理。使用命令 `systemctl start taosadapter` 可以启动 taosAdapter 服务。使用命令 `systemctl stop taosadapter` 可以停止 taosAdapter 服务。使用命令 `systemctl status taosadapter` 来检查 taosAdapter 运行状态。
### 升级 taosAdapter
taosAdapter 和 TDengine server 需要使用相同版本。请通过升级 TDengine server 来升级 taosAdapter。
与 taosd 分离部署的 taosAdapter 必须通过升级其所在服务器的 TDengine server 才能得到升级。
### 移除 taosAdapter
使用命令 rmtaos 可以移除包括 taosAdapter 在内的 TDengine server 软件。
## 监控指标
taosAdapter 目前仅采集 RESTful/WebSocket 相关请求的监控指标，其他接口暂无监控指标。
taosAdapter 将监控指标上报给 taosKeeper，这些监控指标会被 taosKeeper 写入监控数据库，默认是 `log` 库，可以在 taoskeeper 配置文件中修改。以下是这些监控指标的详细介绍。
`adapter_requests` 表记录 taosAdapter 监控数据，字段如下：
| field | type | is\_tag | comment |
|:-------------------|:-------------|:--------|:----------------------------|
| ts | TIMESTAMP | | timestamp |
| total | INT UNSIGNED | | 总请求数 |
| query | INT UNSIGNED | | 查询请求数 |
| write | INT UNSIGNED | | 写入请求数 |
| other | INT UNSIGNED | | 其他请求数 |
| in\_process | INT UNSIGNED | | 正在处理请求数 |
| success | INT UNSIGNED | | 成功请求数 |
| fail | INT UNSIGNED | | 失败请求数 |
| query\_success | INT UNSIGNED | | 查询成功请求数 |
| query\_fail | INT UNSIGNED | | 查询失败请求数 |
| write\_success | INT UNSIGNED | | 写入成功请求数 |
| write\_fail | INT UNSIGNED | | 写入失败请求数 |
| other\_success | INT UNSIGNED | | 其他成功请求数 |
| other\_fail | INT UNSIGNED | | 其他失败请求数 |
| query\_in\_process | INT UNSIGNED | | 正在处理查询请求数 |
| write\_in\_process | INT UNSIGNED | | 正在处理写入请求数 |
| endpoint | VARCHAR | | 请求端点 |
| req\_type | NCHAR | TAG | 请求类型：0 为 REST，1 为 WebSocket |
## 结果返回条数限制
## httpd 升级为 taosAdapter 的变化
taosAdapter 通过参数 `restfulRowLimit` 来控制结果的返回条数，-1 代表无限制，默认无限制。
在 TDengine server 2.2.x.x 或更早期版本中，taosd 进程包含一个内嵌的 http 服务（httpd）。如前面所述，taosAdapter 是一个使用 systemd 管理的独立软件，拥有自己的进程。并且两者有一些配置参数和行为是不同的，请见下表：
该参数控制以下接口返回
- `http://<fqdn>:6041/rest/sql`
- `http://<fqdn>:6041/prometheus/v1/remote_read/:db`
## 配置 http 返回码
taosAdapter 通过参数 `httpCodeServerError` 来设置当 C 接口返回错误时是否返回非 200 的 http 状态码。当设置为 true 时将根据 C 返回的错误码返回不同 http 状态码。具体见 [HTTP 响应码](../../connector/rest-api/#http-响应码)。
## 配置 schemaless 写入是否自动创建 DB
taosAdapter 从 3.0.4.0 版本开始，提供参数 `smlAutoCreateDB` 来控制在 schemaless 协议写入时是否自动创建 DB。默认值为 false 不自动创建 DB，需要用户手动创建 DB 后进行 schemaless 写入。
## 故障解决
您可以通过命令 `systemctl status taosadapter` 来检查 taosAdapter 运行状态。
您也可以通过设置 --logLevel 参数或者环境变量 TAOS_ADAPTER_LOG_LEVEL 来调节 taosAdapter 日志输出详细程度。有效值包括： panic、fatal、error、warn、warning、info、debug 以及 trace。
## 如何从旧版本 TDengine 迁移到 taosAdapter
在 TDengine server 2.2.x.x 或更早期版本中，taosd 进程包含一个内嵌的 http 服务。如前面所述，taosAdapter 是一个使用 systemd 管理的独立软件，拥有自己的进程。并且两者有一些配置参数和行为是不同的，请见下表：
| **#** | **embedded httpd** | **taosAdapter** | **comment** |
| ----- | ------------------- | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | httpEnableRecordSql | --logLevel=debug | |
| 2 | httpMaxThreads | n/a | taosAdapter 自动管理线程池，无需此参数 |
| **#** | **embedded httpd** | **taosAdapter** | **comment** |
|-------|---------------------|-------------------------------|------------------------------------------------------------------------------------------------|
| 1 | httpEnableRecordSql | --logLevel=debug | |
| 2 | httpMaxThreads | n/a | taosAdapter 自动管理线程池，无需此参数 |
| 3 | telegrafUseFieldNum | 请参考 taosAdapter telegraf 配置方法 |
| 4 | restfulRowLimit | restfulRowLimit | 内嵌 httpd 默认输出 10240 行数据，最大允许值为 102400。taosAdapter 也提供 restfulRowLimit 但是默认不做限制。您可以根据实际场景需求进行配置 |
| 5 | httpDebugFlag | 不适用 | httpdDebugFlag 对 taosAdapter 不起作用 |
| 6 | httpDBNameMandatory | 不适用 | taosAdapter 要求 URL 中必须指定数据库名 |
| 4 | restfulRowLimit | restfulRowLimit | 内嵌 httpd 默认输出 10240 行数据，最大允许值为 102400。taosAdapter 也提供 restfulRowLimit 但是默认不做限制。您可以根据实际场景需求进行配置 |
| 5 | httpDebugFlag | 不适用 | httpdDebugFlag 对 taosAdapter 不起作用 |
| 6 | httpDBNameMandatory | 不适用 | taosAdapter 要求 URL 中必须指定数据库名 |

124
docs/zh/14-reference/01-components/_collectd.mdx
View File

@ -1,85 +1,85 @@
### 配置 taosAdapter
#### 配置 taosAdapter
配置 taosAdapter 接收 collectd 数据的方法：
- 在 taosAdapter 配置文件（默认位置为 /etc/taos/taosadapter.toml）中使能配置项
```
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
```toml
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
其中 taosAdapter 默认写入的数据库名称为 `collectd`，也可以修改 taosAdapter 配置文件 dbs 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过配置文件 taosAdapter 需重新启动。
其中 taosAdapter 默认写入的数据库名称为 `collectd`，也可以修改 taosAdapter 配置文件 dbs 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过配置文件 taosAdapter 需重新启动。
- 也可以使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 collectd 数据功能，具体细节请参考 taosAdapter 的参考手册
- 使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 collectd 数据功能，具体细节请参考 taosAdapter 的参考手册。
#### 配置 collectd
### 配置 collectd
#
collectd 使用插件机制可以以多种形式将采集到的监控数据写入到不同的数据存储软件。TDengine 支持直接采集插件和 write_tsdb 插件。
#### 配置接收直接采集插件数据
1. **配置直接采集插件**
修改 collectd 配置文件（默认位置 /etc/collectd/collectd.conf）相关配置项。
修改 collectd 配置文件（默认位置 /etc/collectd/collectd.conf）相关配置项。
```text
LoadPlugin network
<Plugin network>
Server "<taosAdapter's host>" "<port for collectd direct>"
</Plugin>
```
```xml
LoadPlugin network
<Plugin network>
Server "<taosAdapter's host>" "<port for collectd direct>"
</Plugin>
```
其中 \<taosAdapter's host> 填写运行 taosAdapter 的服务器域名或 IP 地址。\<port for collectd direct> 填写 taosAdapter 用于接收 collectd 数据的端口（默认为 6045）。
其中 \<taosAdapter's host> 填写运行 taosAdapter 的服务器域名或 IP 地址。\<port for collectd direct> 填写 taosAdapter 用于接收 collectd 数据的端口（默认为 6045）。
示例如下：
示例如下：
```text
LoadPlugin network
<Plugin network>
Server "127.0.0.1" "6045"
</Plugin>
```
```xml
LoadPlugin network
<Plugin network>
Server "127.0.0.1" "6045"
</Plugin>
```
#### 配置 write_tsdb 插件数据
2. **配置 write_tsdb 插件**
修改 collectd 配置文件（默认位置 /etc/collectd/collectd.conf）相关配置项。
修改 collectd 配置文件（默认位置 /etc/collectd/collectd.conf）相关配置项。
```text
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "<taosAdapter's host>"
Port "<port for collectd write_tsdb plugin>"
...
</Node>
</Plugin>
```
```xml
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "<taosAdapter's host>"
Port "<port for collectd write_tsdb plugin>"
...
</Node>
</Plugin>
```
其中 \<taosAdapter's host> 填写运行 taosAdapter 的服务器域名或 IP 地址。\<port for collectd write_tsdb plugin> 填写 taosAdapter 用于接收 collectd write_tsdb 插件的数据（默认为 6047）。
其中 \<taosAdapter's host> 填写运行 taosAdapter 的服务器域名或 IP 地址。\<port for collectd write_tsdb plugin> 填写 taosAdapter 用于接收 collectd write_tsdb 插件的数据（默认为 6047）。
```text
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "127.0.0.1"
Port "6047"
HostTags "status=production"
StoreRates false
AlwaysAppendDS false
</Node>
</Plugin>
```
```xml
LoadPlugin write_tsdb
<Plugin write_tsdb>
<Node>
Host "127.0.0.1"
Port "6047"
HostTags "status=production"
StoreRates false
AlwaysAppendDS false
</Node>
</Plugin>
```
然后重启 collectd：
然后重启 collectd：
```
systemctl restart collectd
```
```shell
systemctl restart collectd
```

58
docs/zh/14-reference/01-components/_icinga2.mdx
View File

@ -1,44 +1,44 @@
### 配置 taosAdapter
#### 配置 taosAdapter
配置 taosAdapter 接收 icinga2 数据的方法：
- 在 taosAdapter 配置文件（默认位置 /etc/taos/taosadapter.toml）中使能配置项
```
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
```toml
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
其中 taosAdapter 默认写入的数据库名称为 `icinga2`，也可以修改 taosAdapter 配置文件 dbs 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过 taosAdapter 需重新启动。
其中 taosAdapter 默认写入的数据库名称为 `icinga2`，也可以修改 taosAdapter 配置文件 dbs 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过 taosAdapter 需重新启动。
- 也可以使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 icinga2 数据功能，具体细节请参考 taosAdapter 的参考手册
- 使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 icinga2 数据功能，具体细节请参考 taosAdapter 的参考手册
### 配置 icinga2
#### 配置 icinga2
- 使能 icinga2 的 opentsdb-writer（参考链接 https://icinga.com/docs/icinga-2/latest/doc/14-features/#opentsdb-writer）
- 修改配置文件 `/etc/icinga2/features-enabled/opentsdb.conf` 填写 \<taosAdapter's host> 为运行 taosAdapter 的服务器的域名或 IP 地址，\<port for icinga2> 填写 taosAdapter 支持接收 icinga2 数据的相应端口（默认为 6048）
```
object OpenTsdbWriter "opentsdb" {
host = "<taosAdapter's host>"
port = <port for icinga2>
}
```
```c
object OpenTsdbWriter "opentsdb" {
host = "<taosAdapter's host>"
port = <port for icinga2>
}
```
示例文件：
示例文件：
```
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 6048
}
```
```c
object OpenTsdbWriter "opentsdb" {
host = "127.0.0.1"
port = 6048
}
```

6
docs/zh/14-reference/01-components/_prometheus.mdx
View File

@ -1,18 +1,18 @@
配置 Prometheus 是通过编辑 Prometheus 配置文件 prometheus.yml （默认位置 /etc/prometheus/prometheus.yml）完成的。
### 配置第三方数据库地址
#### 配置第三方数据库地址
将其中的 remote_read url 和 remote_write url 指向运行 taosAdapter 服务的服务器域名或 IP 地址，REST 服务端口（taosAdapter 默认使用 6041），以及希望写入 TDengine 的数据库名称，并确保相应的 URL 形式如下：
- remote_read url : `http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_read/<database name>`
- remote_write url : `http://<taosAdapter's host>:<REST service port>/prometheus/v1/remote_write/<database name>`
### 配置 Basic 验证
#### 配置 Basic 验证
- username： \<TDengine's username>
- password： \<TDengine's password>
### prometheus.yml 文件中 remote_write 和 remote_read 相关部分配置示例
#### prometheus.yml 文件中 remote_write 和 remote_read 相关部分配置示例
```yaml
remote_write:

52
docs/zh/14-reference/01-components/_statsd.mdx
View File

@ -1,35 +1,35 @@
### 配置 taosAdapter
#### 配置 taosAdapter
配置 taosAdapter 接收 StatsD 数据的方法：
- 在 taosAdapter 配置文件（默认位置 /etc/taos/taosadapter.toml）中使能配置项
```
...
[statsd]
enable = true
port = 6044
db = "statsd"
user = "root"
password = "taosdata"
worker = 10
gatherInterval = "5s"
protocol = "udp"
maxTCPConnections = 250
tcpKeepAlive = false
allowPendingMessages = 50000
deleteCounters = true
deleteGauges = true
deleteSets = true
deleteTimings = true
...
```
```toml
...
[statsd]
enable = true
port = 6044
db = "statsd"
user = "root"
password = "taosdata"
worker = 10
gatherInterval = "5s"
protocol = "udp"
maxTCPConnections = 250
tcpKeepAlive = false
allowPendingMessages = 50000
deleteCounters = true
deleteGauges = true
deleteSets = true
deleteTimings = true
...
```
其中 taosAdapter 默认写入的数据库名称为 `statsd`，也可以修改 taosAdapter 配置文件 db 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过配置文件 taosAdapter 需重新启动。
其中 taosAdapter 默认写入的数据库名称为 `statsd`，也可以修改 taosAdapter 配置文件 db 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过配置文件 taosAdapter 需重新启动。
- 也可以使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 StatsD 数据功能，具体细节请参考 taosAdapter 的参考手册
- 使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 StatsD 数据功能，具体细节请参考 taosAdapter 的参考手册
### 配置 StatsD
#### 配置 StatsD
使用 StatsD 需要下载其[源代码](https://github.com/statsd/statsd)。其配置文件请参考其源代码下载到本地的根目录下的示例文件 `exampleConfig.js` 进行修改。其中 \<taosAdapter's host> 填写运行 taosAdapter 的服务器域名或 IP 地址，\<port for StatsD> 请填写 taosAdapter 接收 StatsD 数据的端口（默认为 6044）。
@ -40,7 +40,7 @@ repeater 部分添加 { host:'<taosAdapter's host>', port: <port for StatsD>}
示例配置文件：
```
```js
{
port: 8125
, backends: ["./backends/repeater"]
@ -50,7 +50,7 @@ port: 8125
增加如下内容后启动 StatsD（假设配置文件修改为 config.js）。
```
```shell
npm install
node stats.js config.js &
```

34
docs/zh/14-reference/01-components/_tcollector.mdx
View File

@ -1,28 +1,28 @@
### 配置 taosAdapter
#### 配置 taosAdapter
配置 taosAdapter 接收 TCollector 数据的方法：
- 在 taosAdapter 配置文件（默认位置 /etc/taos/taosadapter.toml）中使能配置项
```
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
```toml
...
[opentsdb_telnet]
enable = true
maxTCPConnections = 250
tcpKeepAlive = false
dbs = ["opentsdb_telnet", "collectd", "icinga2", "tcollector"]
ports = [6046, 6047, 6048, 6049]
user = "root"
password = "taosdata"
...
```
其中 taosAdapter 默认写入的数据库名称为 `tcollector`，也可以修改 taosAdapter 配置文件 dbs 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过配置文件 taosAdapter 需重新启动。
其中 taosAdapter 默认写入的数据库名称为 `tcollector`，也可以修改 taosAdapter 配置文件 dbs 项来指定不同的名称。user 和 password 填写实际 TDengine 配置的值。修改过配置文件 taosAdapter 需重新启动。
- 也可以使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 tcollector 数据功能，具体细节请参考 taosAdapter 的参考手册
- 使用 taosAdapter 命令行参数或设置环境变量启动的方式，使能 taosAdapter 接收 tcollector 数据功能，具体细节请参考 taosAdapter 的参考手册
### 配置 TCollector
#### 配置 TCollector
使用 TCollector 需下载其[源代码](https://github.com/OpenTSDB/tcollector)。其配置项在其源代码中。注意：TCollector 各个版本区别较大，这里仅以当前 master 分支最新代码 (git commit: 37ae920) 为例。
@ -30,7 +30,7 @@ password = "taosdata"
示例为源代码修改内容的 git diff 输出：
```
```diff
index e7e7a1c..ec3e23c 100644
--- a/collectors/etc/config.py
+++ b/collectors/etc/config.py

10
source/libs/parser/src/parTranslater.c
View File

@ -7585,9 +7585,13 @@ static int32_t partitionDeleteWhere(STranslateContext* pCxt, SDeleteStmt* pDelet
}
if (TSDB_CODE_SUCCESS == code) {
bool isStrict = false;
code = getTimeRange(&pPrimaryKeyCond, &pDelete->timeRange, &isStrict);
if (TSDB_CODE_SUCCESS == code && !isStrict) {
code = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_DELETE_WHERE);
if (NULL != pPrimaryKeyCond) {
code = getTimeRange(&pPrimaryKeyCond, &pDelete->timeRange, &isStrict);
if (TSDB_CODE_SUCCESS == code && !isStrict) {
code = generateSyntaxErrMsg(&pCxt->msgBuf, TSDB_CODE_PAR_INVALID_DELETE_WHERE);
}
} else {
pDelete->timeRange = TSWINDOW_INITIALIZER;
}
}
nodesDestroyNode(pPrimaryKeyCond);

42
tests/README.md
View File

@ -9,6 +9,7 @@
- [3.4 Smoke Test](#34-smoke-test)
- [3.5 Chaos Test](#35-chaos-test)
- [3.6 CI Test](#36-ci-test)
- [3.7 TSBS Test](#37-tsbs-test)
# 1. Introduction
@ -231,3 +232,44 @@ cd tests
### 3.6.2 How to add new cases?
Please refer to the [Unit Test](#31-unit-test)、[System Test](#32-system-test) and [Legacy Test](#33-legacy-test) sections for detailed steps to add new test cases, when new cases are added in aboved tests, they will be run automatically by CI test.
## 3.7 TSBS Test
[Time Series Benchmark Suite (TSBS)](https://github.com/timescale/tsbs) is an open-source performance benchmarking platform specifically designed for time-series data processing systems, such as databases. It provides a standardized approach to evaluating the performance of various databases by simulating typical use cases such as IoT and DevOps.
### 3.7.1 How to run tests?
TSBS test can be started locally by running command below. Ensure that your virtual machine supports the AVX instruction set:
```bash
cd /usr/local/src && \
git clone https://github.com/taosdata/tsbs.git && \
cd tsbs && \
git checkout enh/chr-td-33357 && \
cd scripts/tsdbComp && \
./testTsbs.sh
```
### 3.7.2 How to start client and server on different hosts?
By default, both client and server will be started on the local host. To start the client and server on separate hosts, please follow steps below to configure `test.ini` before starting the test:
1. Modify IP and hostname of client and server in `test.ini`:
```ini
clientIP="192.168.0.203" # client ip
clientHost="trd03" # client hostname
serverIP="192.168.0.204" # server ip
serverHost="trd04" # server hostname
```
2. Passwordless ssh login between the client and server is required; otherwise, please set the ssh login password in `test.int`, for example:
```ini
serverPass="taosdata123" # server root password
```
### 3.7.3 Check test results
When the test is done, the result can be found in `/data2/` directory, which can also be configured in `test.ini`.

13
tests/system-test/1-insert/delete_stable.py
View File

@ -200,6 +200,17 @@ class TDTestCase:
tdSql.execute(f'delete from {tbname} where ts between {self.ts+i+1} and {self.ts}')
tdSql.query(f'select {col_name} from {tbname}')
tdSql.checkRows(tb_num*row_num)
for i in range(row_num):
tdSql.execute(f'delete from {tbname} where t1 = 1')
tdSql.execute(f'flush database {dbname}')
tdSql.execute('reset query cache')
tdSql.query(f'select {col_name} from {tbname}')
tdSql.checkRows(0)
for j in range(tb_num):
self.insert_base_data(col_type,f'{tbname}_{j}',row_num,base_data)
tdSql.execute(f'delete from {tbname} where t1 = 0')
tdSql.query(f'select {col_name} from {tbname}')
tdSql.checkRows(tb_num*row_num)
def delete_error(self,tbname,column_name,column_type,base_data):
for error_list in ['',f'ts = {self.ts} and',f'ts = {self.ts} or']:
if 'binary' in column_type.lower():
@ -268,4 +279,4 @@ class TDTestCase:
tdLog.success("%s successfully executed" % __file__)
tdCases.addWindows(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())
tdCases.addLinux(__file__, TDTestCase())