Merge remote-tracking branch 'origin/main' into doc/contrib

This commit is contained in:
Shengliang Guan 2025-02-18 14:56:55 +08:00
commit 63e039e55d
37 changed files with 674 additions and 557 deletions

View File

@ -304,7 +304,7 @@ Not supported
## Querying the Written Data
By running the example code from the previous section, tables will be automatically created in the power database. We can query the data using taos shell or an application. Below is an example of querying the data from the supertable and meters table using taos shell.
By running the example code from the previous section, tables will be automatically created in the power database. We can query the data using TDengine CLI or an application. Below is an example of querying the data from the supertable and meters table using TDengine CLI.
```shell
taos> show power.stables;

View File

@ -10,7 +10,7 @@ TDengine provides data subscription and consumption interfaces similar to those
## Creating Topics
Please use taos shell or refer to the [Execute SQL](../running-sql-statements/) section to execute the SQL for creating topics: `CREATE TOPIC IF NOT EXISTS topic_meters AS SELECT ts, current, voltage, phase, groupid, location FROM meters`
Please use TDengine CLI or refer to the [Execute SQL](../running-sql-statements/) section to execute the SQL for creating topics: `CREATE TOPIC IF NOT EXISTS topic_meters AS SELECT ts, current, voltage, phase, groupid, location FROM meters`
The above SQL will create a subscription named topic_meters. Each record in the messages obtained using this subscription is composed of the columns selected by this query statement `SELECT ts, current, voltage, phase, groupid, location FROM meters`.

View File

@ -87,7 +87,7 @@ Additionally, to accelerate the data processing process, TDengine specifically s
```text
Users can use the performance testing tool taosBenchmark to assess the data compression effect of TDengine. By using the -f option to specify the write configuration file, taosBenchmark can write a specified number of CSV sample data into the specified database parameters and table structure.
After completing the data writing, users can execute the flush database command in the taos shell to force all data to be written to the disk. Then, use the du command of the Linux operating system to get the size of the data folder of the specified vnode. Finally, divide the original data size by the actual storage data size to calculate the real compression ratio.
After completing the data writing, users can execute the flush database command in the TDengine CLI to force all data to be written to the disk. Then, use the du command of the Linux operating system to get the size of the data folder of the specified vnode. Finally, divide the original data size by the actual storage data size to calculate the real compression ratio.
```
The following command can be used to obtain the storage space occupied by TDengine.

View File

@ -55,7 +55,7 @@ For dnodes wishing to join the cluster, it is essential to ensure that the param
### 5. Start
Start the first dnode, such as `h1.tdengine.com`, following the steps mentioned above. Then execute taos in the terminal to start TDengine's CLI program taos, and execute the `show dnodes` command within it to view all dnode information in the current cluster.
Start the first dnode, such as `h1.tdengine.com`, following the steps mentioned above. Then execute taos in the terminal to start TDengine CLI program taos, and execute the `show dnodes` command within it to view all dnode information in the current cluster.
```shell
taos> show dnodes;
@ -68,7 +68,7 @@ You can see that the endpoint of the dnode node that has just started is `h1.tde
### 6. Adding dnode
Follow the steps mentioned earlier, start taosd on each physical node. Each dnode needs to configure the firstEp parameter in the taos.cfg file to the endpoint of the first node of the new cluster, which in this case is `h1.tdengine.com:6030`. On the machine where the first dnode is located, run taos in the terminal, open TDengine's CLI program taos, then log into the TDengine cluster, and execute the following SQL.
Follow the steps mentioned earlier, start taosd on each physical node. Each dnode needs to configure the firstEp parameter in the taos.cfg file to the endpoint of the first node of the new cluster, which in this case is `h1.tdengine.com:6030`. On the machine where the first dnode is located, run taos in the terminal, open TDengine CLI program taos, then log into the TDengine cluster, and execute the following SQL.
```shell
create dnode "h2.tdengine.com:6030"
@ -84,13 +84,13 @@ In the logs, please confirm that the fqdn and port of the output dnode are consi
**Tips**
- Any dnode that has joined the cluster can serve as the firstEp for subsequent nodes to be added. The firstEp parameter only functions when that dnode first joins the cluster. After joining, the dnode will save the latest mnode's endpoint list, and subsequently, it no longer depends on this parameter. The firstEp parameter in the configuration file is mainly used for client connections, and if no parameters are set for TDengine's CLI, it will default to connecting to the node specified by firstEp.
- Any dnode that has joined the cluster can serve as the firstEp for subsequent nodes to be added. The firstEp parameter only functions when that dnode first joins the cluster. After joining, the dnode will save the latest mnode's endpoint list, and subsequently, it no longer depends on this parameter. The firstEp parameter in the configuration file is mainly used for client connections, and if no parameters are set for TDengine CLI, it will default to connecting to the node specified by firstEp.
- Two dnodes that have not configured the firstEp parameter will run independently after starting. At this time, it is not possible to join one dnode to another to form a cluster.
- TDengine does not allow merging two independent clusters into a new cluster.
### 7. Adding mnode
When creating a TDengine cluster, the first dnode automatically becomes the mnode of the cluster, responsible for managing and coordinating the cluster. To achieve high availability of mnode, subsequent dnodes need to manually create mnode. Please note that a cluster can create up to 3 mnodes, and only one mnode can be created on each dnode. When the number of dnodes in the cluster reaches or exceeds 3, you can create mnode for the existing cluster. In the first dnode, first log into TDengine through the CLI program taos, then execute the following SQL.
When creating a TDengine cluster, the first dnode automatically becomes the mnode of the cluster, responsible for managing and coordinating the cluster. To achieve high availability of mnode, subsequent dnodes need to manually create mnode. Please note that a cluster can create up to 3 mnodes, and only one mnode can be created on each dnode. When the number of dnodes in the cluster reaches or exceeds 3, you can create mnode for the existing cluster. In the first dnode, first log into TDengine through TDengine CLI program taos, then execute the following SQL.
```shell
create mnode on dnode <dnodeId>

View File

@ -456,7 +456,7 @@ export POD_NAME=$(kubectl get pods --namespace default \
kubectl --namespace default exec $POD_NAME -- taos -s "show dnodes; show mnodes"
3. Run into taos shell:
3. Run into TDengine CLI:
kubectl --namespace default exec -it $POD_NAME -- taos
```

View File

@ -6,6 +6,9 @@ slug: /tdengine-reference/tools/tdengine-cli
The TDengine command line program (hereinafter referred to as TDengine CLI) is the simplest and most commonly used tool for users to operate and interact with TDengine instances. It requires the installation of either the TDengine Server package or the TDengine Client package before use.
## Get
TDengine CLI is the default installation component in the TDengine server and client installation package. It can be used after installation, refer to [TDengine Installation](../../../get-started/)
## Startup
To enter the TDengine CLI, simply execute `taos` in the terminal.
@ -29,15 +32,83 @@ To exit the TDengine CLI, execute `q`, `quit`, or `exit` and press enter.
taos> quit
```
## Command Line Parameters
### Basic Parameters
You can change the behavior of the TDengine CLI by configuring command line parameters. Below are some commonly used command line parameters:
- -h HOST: The FQDN of the server where the TDengine service is located, default is to connect to the local service.
- -P PORT: Specifies the port number used by the server.
- -u USER: Username to use when connecting.
- -p PASSWORD: Password to use when connecting to the server.
- -?, --help: Prints out all command line parameters.
- -s COMMAND: SQL command executed in non-interactive mode.
Use the `-s` parameter to execute SQL non interactively, and exit after execution. This mode is suitable for use in automated scripts.
For example, connect to the server h1.taos.com with the following command, and execute the SQL specified by `-s`:
```bash
taos -h my-server -s "use db; show tables;"
```
- -c CONFIGDIR: Specify the configuration file directory.
In Linux, the default is `/etc/tao`. The default name of the configuration file in this directory is `taos.cfg`.
Use the `-c` parameter to change the location where the `taosc` client loads the configuration file. For client configuration parameters, refer to [Client Configuration](../../components/taosc).
The following command specifies the `taos.cfg` configuration file under `/root/cfg/` loaded by the `taosc` client.
```bash
taos -c /root/cfg/
```
### Advanced Parameters
- -a AUTHSTR: Authorization information for connecting to the server.
- -A: Calculate authorization information using username and password.
- -B: Set BI tool display mode, after setting, all outputs follow the format of BI tools.
- -C: Print the configuration parameters of `taos.cfg` in the directory specified by -c.
- -d DATABASE: Specifies the database to use when connecting to the server.
- -E dsn: Connect to cloud services or servers providing WebSocket connections using WebSocket DSN.
- -f FILE: Execute SQL script file in non-interactive mode. Each SQL statement in the file must occupy one line.
- -k: Test the running status of the server, 0: unavailable, 1: network ok, 2: service ok, 3: service degraded, 4: exiting.
- -l PKTLEN: Test packet size used during network testing.
- -n NETROLE: Test range during network connection testing, default is `client`, options are `client`, `server`.
- -N PKTNUM: Number of test packets used during network testing.
- -r: Convert time columns to unsigned 64-bit integer type output (i.e., uint64_t in C language).
- -R: Connect to the server using RESTful mode.
- -t: Test the startup status of the server, status same as -k.
- -w DISPLAYWIDTH: Client column display width.
- -z TIMEZONE: Specifies the timezone, default is the local timezone.
- -V: Print the current version number.
## Data Export/Import
### Data Export To a File
- You can use the symbol “>>” to export query results to a file, the syntax is: sql query statement >> 'output file name'; If no path is written for the output file, it will be output to the current directory. For example, `select * from d0 >> '/root/d0.csv';` will output the query results to /root/d0.csv.
### Data Import From a File
- You can use insert into table_name file 'input file name', to import the data file exported in the previous step back into the specified table. For example, `insert into d0 file '/root/d0.csv';` means to import all the data exported above back into the d0 table.
## Execute SQL Scripts
In the TDengine CLI, you can run multiple SQL commands from a script file using the `source` command.
In the TDengine CLI, you can run multiple SQL commands from a script file using the `source` command, multiple SQL statements in the script file can be written in line.
```sql
taos> source <filename>;
```
## Online Modification of Display Character Width
## TDengine CLI Tips
### TAB Key Completion
- Pressing the TAB key when no command is present will list all commands supported by TDengine CLI.
- Pressing the TAB key when preceded by a space will display the first of all possible command words at this position, pressing TAB again will switch to the next one.
- If a string precedes the TAB key, it will search for all command words that match the prefix of this string and display the first one, pressing TAB again will switch to the next one.
- Entering a backslash `\` + TAB key, will automatically complete to the column display mode command word `\G;`.
### Modification of Display Character Width
You can adjust the display character width in the TDengine CLI using the following command:
@ -47,71 +118,16 @@ taos> SET MAX_BINARY_DISPLAY_WIDTH <nn>;
If the displayed content ends with ..., it indicates that the content has been truncated. You can modify the display character width with this command to display the full content.
## Command Line Parameters
### Other
You can change the behavior of the TDengine CLI by configuring command line parameters. Below are some commonly used command line parameters:
- You can use the up and down arrow keys to view previously entered commands.
- In TDengine CLI, use the `alter user` command to change user passwords, the default password is `taosdata`.
- Ctrl+C to stop a query that is in progress.
- Execute `RESET QUERY CACHE` to clear the cache of the local table Schema.
- Batch execute SQL statements.
You can store a series of TDengine CLI commands (ending with a semicolon `;`, each SQL statement on a new line) in a file, and execute the command `source <file-name>` in TDengine CLI to automatically execute all SQL statements in that file.
- -h HOST: The FQDN of the server where the TDengine service is located, default is to connect to the local service
- -P PORT: Specifies the port number used by the server
- -u USER: Username to use when connecting
- -p PASSWORD: Password to use when connecting to the server
- -?, --help: Prints out all command line parameters
There are many other parameters:
- -a AUTHSTR: Authorization information for connecting to the server
- -A: Calculate authorization information using username and password
- -B: Set BI tool display mode, after setting, all outputs follow the format of BI tools
- -c CONFIGDIR: Specify the configuration file directory, default in Linux environment is `/etc/taos`, the default name of the configuration file in this directory is `taos.cfg`
- -C: Print the configuration parameters of `taos.cfg` in the directory specified by -c
- -d DATABASE: Specifies the database to use when connecting to the server
- -E dsn: Connect to cloud services or servers providing WebSocket connections using WebSocket DSN
- -f FILE: Execute SQL script file in non-interactive mode. Each SQL statement in the file must occupy one line
- -k: Test the running status of the server, 0: unavailable, 1: network ok, 2: service ok, 3: service degraded, 4: exiting
- -l PKTLEN: Test packet size used during network testing
- -n NETROLE: Test range during network connection testing, default is `client`, options are `client`, `server`
- -N PKTNUM: Number of test packets used during network testing
- -r: Convert time columns to unsigned 64-bit integer type output (i.e., uint64_t in C language)
- -R: Connect to the server using RESTful mode
- -s COMMAND: SQL command executed in non-interactive mode
- -t: Test the startup status of the server, status same as -k
- -w DISPLAYWIDTH: Client column display width
- -z TIMEZONE: Specifies the timezone, default is the local timezone
- -V: Print the current version number
Example:
```shell
taos -h h1.taos.com -s "use db; show tables;"
```
## Configuration File
You can also control the behavior of the TDengine CLI through parameters set in the configuration file. For available configuration parameters, please refer to [Client Configuration](../../components/taosc)
## Error Code Table
Starting from TDengine version 3.3.4.8, TDengine CLI returns specific error codes in error messages. Users can visit the TDengine official website's error code page to find specific reasons and solutions, see: [Error Code Reference](../../error-codes/)
## TDengine CLI TAB Key Completion
- Pressing the TAB key when no command is present will list all commands supported by TDengine CLI
- Pressing the TAB key when preceded by a space will display the first of all possible command words at this position, pressing TAB again will switch to the next one
- If a string precedes the TAB key, it will search for all command words that match the prefix of this string and display the first one, pressing TAB again will switch to the next one
- Entering a backslash `\` + TAB key, will automatically complete to the column display mode command word `\G;`
## TDengine CLI Tips
- You can use the up and down arrow keys to view previously entered commands
- In TDengine CLI, use the `alter user` command to change user passwords, the default password is `taosdata`
- Ctrl+C to stop a query that is in progress
- Execute `RESET QUERY CACHE` to clear the cache of the local table Schema
- Batch execute SQL statements. You can store a series of TDengine CLI commands (ending with a semicolon `;`, each SQL statement on a new line) in a file, and execute the command `source <file-name>` in TDengine CLI to automatically execute all SQL statements in that file
## TDengine CLI Export Query Results to a File
- You can use the symbol “>>” to export query results to a file, the syntax is: sql query statement >> 'output file name'; If no path is written for the output file, it will be output to the current directory. For example, select * from d0 >> '/root/d0.csv'; will output the query results to /root/d0.csv.
## TDengine CLI Import Data from a File into a Table
- You can use insert into table_name file 'input file name', to import the data file exported in the previous step back into the specified table. For example, insert into d0 file '/root/d0.csv'; means to import all the data exported above back into the d0 table.

View File

@ -6,45 +6,26 @@ slug: /tdengine-reference/tools/taosdump
`taosdump` is a TDengine data backup/recovery tool provided for open source users, and the backed up data files adopt the standard [Apache AVRO](https://avro.apache.org/)
Format, convenient for exchanging data with the external ecosystem.
Taosdump provides multiple data backup and recovery options to meet different data needs, and all supported options can be viewed through --help.
taosdump provides multiple data backup and recovery options to meet different data needs, and all supported options can be viewed through --help.
## Installation
## Get
taosdump is the default installation component in the TDengine installation package, which can be used after installing TDengine. For how to install TDengine, please refer to [TDengine Installation](../../../get-started/)
taosdump is the default installation component in the TDengine server and client installation package. It can be used after installation, refer to [TDengine Installation](../../../get-started/)
## Common Use Cases
## Startup
### taosdump Backup Data
taosdump needs to be run in the command line terminal. It must be run with parameters to indicate backup or restore operations, such as:
``` bash
taosdump -h my-server -D test -o /root/test/
```
The above command means to backup the `test` database on the `my server` machine to the `/root/test/` directory.
1. Backup all databases: specify the `-A` or `--all-databases` parameter;
2. Backup multiple specified databases: use the `-D db1,db2,...` parameter;
3. Backup certain supertables or basic tables in a specified database: use the `dbname stbname1 stbname2 tbname1 tbname2 ...` parameter, note that this input sequence starts with the database name, supports only one database, and the second and subsequent parameters are the names of the supertables or basic tables in that database, separated by spaces;
4. Backup the system log database: TDengine clusters usually include a system database named `log`, which contains data for TDengine's own operation, taosdump does not back up the log database by default. If there is a specific need to back up the log database, you can use the `-a` or `--allow-sys` command line parameter.
5. "Tolerant" mode backup: Versions after taosdump 1.4.1 provide the `-n` and `-L` parameters, used for backing up data without using escape characters and in "tolerant" mode, which can reduce backup data time and space occupied when table names, column names, and label names do not use escape characters. If unsure whether to use `-n` and `-L`, use the default parameters for "strict" mode backup. For an explanation of escape characters, please refer to the [official documentation](../../sql-manual/escape-characters/).
6. If a backup file already exists in the directory specified by the `-o` parameter, to prevent data from being overwritten, taosdump will report an error and exit. Please replace it with another empty directory or clear the original data before backing up.
7. Currently, taosdump does not support data breakpoint backup function. Once the data backup is interrupted, it needs to be started from scratch.
If the backup takes a long time, it is recommended to use the (-S -E options) method to specify the start/end time for segmented backup.
``` bash
taosdump -h my-server -i /root/test/
```
The above command means to restore the previously backed up data files in the `/root/test/` directory to the host named `my server`.
:::tip
- Versions after taosdump 1.4.1 provide the `-I` parameter, used for parsing avro file schema and data, specifying the `-s` parameter will only parse the schema.
- Backups after taosdump 1.4.2 use the `-B` parameter to specify the number of batches, the default value is 16384. If "Error actual dump .. batch .." occurs due to insufficient network speed or disk performance in some environments, you can try adjusting the `-B` parameter to a smaller value.
- taosdump's export does not support interruption recovery, so the correct way to handle an unexpected termination of the process is to delete all related files that have been exported or generated.
- taosdump's import supports interruption recovery, but when the process restarts, you may receive some "table already exists" prompts, which can be ignored.
:::
### taosdump Restore Data
- Restore data files from a specified path: use the `-i` parameter along with the data file path. As mentioned earlier, the same directory should not be used to back up different data sets, nor should the same path be used to back up the same data set multiple times, otherwise, the backup data will cause overwriting or multiple backups.
- taosdump supports data recovery to a new database name with the parameter `-W`, please refer to the command line parameter description for details.
:::tip
taosdump internally uses the TDengine stmt binding API to write restored data, currently using 16384 as a batch for writing. If there are many columns in the backup data, it may cause a "WAL size exceeds limit" error, in which case you can try adjusting the `-B` parameter to a smaller value.
:::
## Detailed Command Line Parameters List
## Command Line Parameters
Below is the detailed command line parameters list for taosdump:
@ -65,8 +46,8 @@ Usage: taosdump [OPTION...] dbname [tbname ...]
-c, --config-dir=CONFIG_DIR Configure directory. Default is /etc/taos
-i, --inpath=INPATH Input file path.
-o, --outpath=OUTPATH Output file path.
-r, --resultFile=RESULTFILE DumpOut/In Result file path and name.
-a, --allow-sys Allow to dump system database
-r, --resultFile=RESULTFILE DumpOut/In Result file path and name.
-a, --allow-sys Allow to dump system database.
-A, --all-databases Dump all databases.
-D, --databases=DATABASES Dump inputted databases. Use comma to separate
databases' name.
@ -98,23 +79,54 @@ Usage: taosdump [OPTION...] dbname [tbname ...]
-n, --no-escape No escape char '`'. Default is using it.
-Q, --dot-replace Replace dot character with underline character in
the table name.(Version 2.5.3)
-T, --thread-num=THREAD_NUM Number of thread for dump in file. Default is
8.
-W, --rename=RENAME-LIST Rename database name with new name during
-T, --thread-num=THREAD_NUM Number of thread for dump in file. Default is 8.
-W, --rename=RENAME-LIST Rename database name with new name during.
importing data. RENAME-LIST:
"db1=newDB1|db2=newDB2" means rename db1 to newDB1
and rename db2 to newDB2 (Version 2.5.4)
and rename db2 to newDB2 (Version 2.5.4).
-k, --retry-count=VALUE Set the number of retry attempts for connection or
query failures
-z, --retry-sleep-ms=VALUE retry interval sleep time, unit ms
-C, --cloud=CLOUD_DSN specify a DSN to access TDengine cloud service
-R, --restful Use RESTful interface to connect TDengine
query failures.
-z, --retry-sleep-ms=VALUE retry interval sleep time, unit ms.
-C, --cloud=CLOUD_DSN specify a DSN to access TDengine cloud service.
-R, --restful Use RESTful interface to connect TDengine.
-t, --timeout=SECONDS The timeout seconds for websocket to interact.
-g, --debug Print debug info.
-?, --help Give this help list
--usage Give a short usage message
-V, --version Print program version
-?, --help Give this help list.
--usage Give a short usage message.
-V, --version Print program version.
Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.
```
## Common Use Cases
### Backup Data
1. Backup all databases: specify the `-A` or `--all-databases` parameter.
2. Backup multiple specified databases: use the `-D db1,db2,...` parameter.
3. Backup certain supertables or basic tables in a specified database: use the `dbname stbname1 stbname2 tbname1 tbname2 ...` parameter, note that this input sequence starts with the database name, supports only one database, and the second and subsequent parameters are the names of the supertables or basic tables in that database, separated by spaces.
4. Backup the system log database: TDengine clusters usually include a system database named `log`, which contains data for TDengine's own operation, taosdump does not back up the log database by default. If there is a specific need to back up the log database, you can use the `-a` or `--allow-sys` command line parameter.
5. "Tolerant" mode backup: Versions after taosdump 1.4.1 provide the `-n` and `-L` parameters, used for backing up data without using escape characters and in "tolerant" mode, which can reduce backup data time and space occupied when table names, column names, and label names do not use escape characters. If unsure whether to use `-n` and `-L`, use the default parameters for "strict" mode backup. For an explanation of escape characters, please refer to the [official documentation](../../sql-manual/escape-characters/)
6. If a backup file already exists in the directory specified by the `-o` parameter, to prevent data from being overwritten, taosdump will report an error and exit. Please replace it with another empty directory or clear the original data before backing up.
7. Currently, taosdump does not support data breakpoint backup function. Once the data backup is interrupted, it needs to be started from scratch.
If the backup takes a long time, it is recommended to use the (-S -E options) method to specify the start/end time for segmented backup.
:::tip
- Versions after taosdump 1.4.1 provide the `-I` parameter, used for parsing avro file schema and data, specifying the `-s` parameter will only parse the schema.
- Backups after taosdump 1.4.2 use the `-B` parameter to specify the number of batches, the default value is 16384. If "Error actual dump .. batch .." occurs due to insufficient network speed or disk performance in some environments, you can try adjusting the `-B` parameter to a smaller value.
- taosdump's export does not support interruption recovery, so the correct way to handle an unexpected termination of the process is to delete all related files that have been exported or generated.
- taosdump's import supports interruption recovery, but when the process restarts, you may receive some "table already exists" prompts, which can be ignored.
:::
### Restore Data
- Restore data files from a specified path: use the `-i` parameter along with the data file path. As mentioned earlier, the same directory should not be used to back up different data sets, nor should the same path be used to back up the same data set multiple times, otherwise, the backup data will cause overwriting or multiple backups.
- taosdump supports data recovery to a new database name with the parameter `-W`, please refer to the command line parameter description for details.
:::tip
taosdump internally uses the TDengine stmt binding API to write restored data, currently using 16384 as a batch for writing. If there are many columns in the backup data, it may cause a "WAL size exceeds limit" error, in which case you can try adjusting the `-B` parameter to a smaller value.
:::

View File

@ -2,30 +2,33 @@
title: taosBenchmark Reference
sidebar_label: taosBenchmark
slug: /tdengine-reference/tools/taosbenchmark
toc_max_heading_level: 4
---
TaosBenchmark is a performance benchmarking tool for TDengine products, providing insertion, query, and subscription performance testing for TDengine products, and outputting performance indicators.
## Installation
## Get
taosBenchmark is the default installation component in the TDengine installation package, which can be used after installing TDengine. For how to install TDengine, please refer to [TDengine Installation](../../../get started/)
taosBenchmark is the default installation component in the TDengine server and client installation package. It can be used after installation, refer to [TDengine Installation](../../../get-started/)
## Operation
## Startup
### Configuration and Operation Methods
taosbenchmark supports three operating modes:
- No Parameter Mode
- Command Line Mode
- JSON Configuration File Mode
taosBbenchmark supports three operating modes:
- No parameter mode
- Command line mode
- JSON configuration file mode
The command-line approach is a subset of the functionality of JSON configuration files, which immediately uses the command line and then the configuration file, with the parameters specified by the command line taking precedence.
The `Command Line Mode` is a subset of the `JSON Configuration File Mode` function. When both are used at the same time, the `Command Line Mode` takes precedence.
**Ensure that the TDengine cluster is running correctly before running taosBenchmark.**
:::tip
Ensure that the TDengine cluster is running correctly before running taosBenchmark.
:::
### Running Without Command Line Arguments
### No Parameter Mode
Execute the following command to quickly experience taosBenchmark performing a write performance test on TDengine based on the default configuration.
```shell
taosBenchmark
```
@ -33,17 +36,18 @@ taosBenchmark
When running without parameters, taosBenchmark defaults to connecting to the TDengine cluster specified in `/etc/taos/taos.cfg `.
After successful connection, a smart meter example database test, super meters, and 10000 sub meters will be created, with 10000 records per sub meter. If the test database already exists, it will be deleted before creating a new one.
### Running Using Command Line Configuration Parameters
### Command Line Mode
When running taosBenchmark using command line parameters and controlling its behavior, the `-f <json file>` parameter cannot be used. All configuration parameters must be specified through the command line. Below is an example of using command line mode to test the write performance of taosBenchmark.
The parameters supported by the command line are those frequently used in the write function. The query and subscription function does not support the command line mode.
For example:
```shell
taosBenchmark -I stmt -n 200 -t 100
```bash
taosBenchmark -d db -t 100 -n 1000 -T 4 -I stmt -y
```
The above command `taosBenchmark` will create a database named `test`, establish a supertable `meters` within it, create 100 subtables in the supertable, and insert 200 records into each subtable using parameter binding.
This command means that using `taosBenchmark` will create a database named `db`, create the default super table `meters`, sub table 100, and use parameter binding (stmt) to write 1000 records for each sub table.
### Running Using a Configuration File
### JSON Configuration File Mode
Running in configuration file mode provides all functions, so parameters can be configured to run in the configuration file.
@ -51,42 +55,8 @@ Running in configuration file mode provides all functions, so parameters can be
taosBenchmark -f <json file>
```
**Below are a few examples of configuration files:**
#### JSON Configuration File Example for Insertion Scenario
<details>
<summary>insert.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/insert.json}}
```
</details>
#### Example JSON Configuration File for Query Scenario
<details>
<summary>query.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/query.json}}
```
</details>
#### Example JSON Configuration File for Subscription Scenario
<details>
<summary>tmq.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/tmq.json}}
```
</details>
## Detailed Explanation of Command Line Parameters
## Command Line Parameters
- **-f/--file \<json file>** :
The JSON configuration file to use, specifying all parameters. This parameter cannot be used simultaneously with other command line parameters. There is no default value.
@ -212,62 +182,7 @@ taosBenchmark -A INT,DOUBLE,NCHAR,BINARY\(16\)
Displays help information and exits. Cannot be used with other parameters.
## Output performance indicators
#### Write indicators
After writing is completed, a summary performance metric will be output in the last two lines in the following format:
``` bash
SUCC: Spent 8.527298 (real 8.117379) seconds to insert rows: 10000000 with 8 thread(s) into test 1172704.41 (real 1231924.74) records/second
SUCC: insert delay, min: 19.6780ms, avg: 64.9390ms, p90: 94.6900ms, p95: 105.1870ms, p99: 130.6660ms, max: 157.0830ms
```
First line write speed statistics:
- Spent: Total write time, in seconds, counting from the start of writing the first data to the end of the last data. This indicates that a total of 8.527298 seconds were spent
- Real: Total write time (calling the engine), excluding the time spent preparing data for the testing framework. Purely counting the time spent on engine calls, The time spent is 8.117379 seconds. If 8.527298-8.117379=0.409919 seconds, it is the time spent preparing data for the testing framework
- Rows: Write the total number of rows, which is 10 million pieces of data
- Threads: The number of threads being written, which is 8 threads writing simultaneously
- Records/second write speed = `total write time` / `total number of rows written`, real in parentheses is the same as before, indicating pure engine write speed
Second line single write delay statistics:
- min: Write minimum delay
- avg: Write normal delay
- p90: Write delay p90 percentile delay number
- p95: Write delay p95 percentile delay number
- p99: Write delay p99 percentile delay number
- max: maximum write delay
Through this series of indicators, the distribution of write request latency can be observed
#### Query indicators
The query performance test mainly outputs the QPS indicator of query request speed, and the output format is as follows:
``` bash
complete query with 3 threads and 10000 query delay avg: 0.002686s min: 0.001182s max: 0.012189s p90: 0.002977s p95: 0.003493s p99: 0.004645s SQL command: select ...
INFO: Spend 26.9530 second completed total queries: 30000, the QPS of all threads: 1113.049
```
- The first line represents the percentile distribution of query execution and query request delay for each of the three threads executing 10000 queries. The SQL command is the test query statement
- The second line indicates that the total query time is 26.9653 seconds, and the query rate per second (QPS) is 1113.049 times/second
- If the `continue_if_fail` option is set to `yes` in the query, the last line will output the number of failed requests and error rate, the format like "error + number of failed requests (error rate)"
- QPS = number of successful requests / time spent (in seconds)
- Error rate = number of failed requests / (number of successful requests + number of failed requests)
#### Subscription metrics
The subscription performance test mainly outputs consumer consumption speed indicators, with the following output format:
``` bash
INFO: consumer id 0 has poll total msgs: 376, period rate: 37.592 msgs/s, total rows: 3760000, period rate: 375924.815 rows/s
INFO: consumer id 1 has poll total msgs: 362, period rate: 36.131 msgs/s, total rows: 3620000, period rate: 361313.504 rows/s
INFO: consumer id 2 has poll total msgs: 364, period rate: 36.378 msgs/s, total rows: 3640000, period rate: 363781.731 rows/s
INFO: consumerId: 0, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 1, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 2, consume msgs: 1000, consume rows: 10000000
INFO: Consumed total msgs: 3000, total rows: 30000000
```
- Lines 1 to 3 real-time output of the current consumption speed of each consumer, msgs/s represents the number of consumption messages, each message contains multiple rows of data, and rows/s represents the consumption speed calculated by rows
- Lines 4 to 6 show the overall statistics of each consumer after the test is completed, including the total number of messages consumed and the total number of lines
- The overall statistics of all consumers in line 7, `msgs` represents how many messages were consumed in total, `rows` represents how many rows of data were consumed in total
## Configuration File Parameters Detailed Explanation
## Configuration File Parameters
### General Configuration Parameters
@ -284,7 +199,7 @@ The parameters listed in this section apply to all functional modes.
- **password** : Password for connecting to the TDengine server, default value is taosdata.
### Configuration Parameters for Insertion Scenarios
### Insertion Configuration Parameters
In insertion scenarios, `filetype` must be set to `insert`. For this parameter and other common parameters, see Common Configuration Parameters.
@ -299,7 +214,7 @@ In insertion scenarios, `filetype` must be set to `insert`. For this parameter a
“continue_if_fail”: “yes”, taosBenchmark warns the user and continues writing
“continue_if_fail”: “smart”, if the child table does not exist upon failure, taosBenchmark will create the child table and continue writing
#### Database Related Configuration Parameters
#### Database Parameters
Parameters related to database creation are configured in the `dbinfo` section of the json configuration file, specific parameters are as follows. Other parameters correspond to those specified in TDengine's `create database`, see [../../taos-sql/database]
@ -307,23 +222,7 @@ Parameters related to database creation are configured in the `dbinfo` section o
- **drop**: Whether to delete the database before insertion, options are "yes" or "no", "no" means do not create. Default is to delete.
#### Stream Computing Related Configuration Parameters
Parameters related to stream computing are configured in the `stream` section of the json configuration file, specific parameters are as follows.
- **stream_name**: Name of the stream computing, mandatory.
- **stream_stb**: Name of the supertable corresponding to the stream computing, mandatory.
- **stream_sql**: SQL statement for the stream computing, mandatory.
- **trigger_mode**: Trigger mode for the stream computing, optional.
- **watermark**: Watermark for the stream computing, optional.
- **drop**: Whether to create stream computing, options are "yes" or "no", "no" means do not create.
#### Supertable Related Configuration Parameters
#### Supertable Parameters
Parameters related to supertable creation are configured in the `super_tables` section of the json configuration file, specific parameters are as follows.
@ -357,7 +256,7 @@ Parameters related to supertable creation are configured in the `super_tables` s
- **childtable_limit** : Effective only when child_table_exists is yes, specifies the limit when getting the subtable list from the supertable.
- **interlace_rows** : Enables interlaced insertion mode and specifies the number of rows to insert into each subtable at a time. Interlaced insertion mode means inserting the number of rows specified by this parameter into each subtable in turn and repeating this process until all subtable data is inserted. The default value is 0, i.e., data is inserted into one subtable before moving to the next subtable.
- **interlace_rows** : Enables interlaced insertion mode and specifies the number of rows to insert into each subtable at a time. Interlaced insertion mode means inserting the number of rows specified by this parameter into each subtable in turn and repeating this process until all subtable data is inserted. The default is 0, i.e., data is inserted into one subtable before moving to the next subtable.
- **insert_interval** : Specifies the insertion interval for interlaced insertion mode, in ms, default value is 0. Only effective when `-B/--interlace-rows` is greater than 0. It means that the data insertion thread will wait for the time interval specified by this value after inserting interlaced records for each subtable before proceeding to the next round of writing.
@ -385,7 +284,7 @@ Parameters related to supertable creation are configured in the `super_tables` s
- **sqls** : Array of strings type, specifies the array of sql to be executed after the supertable is successfully created, the table name specified in sql must be prefixed with the database name, otherwise an unspecified database error will occur
#### Tag and Data Column Configuration Parameters
#### Tag and Data Columns
Specify the configuration parameters for tag and data columns in `super_tables` under `columns` and `tag`.
@ -420,7 +319,7 @@ Specify the configuration parameters for tag and data columns in `super_tables`
- **fillNull**: String type, specifies whether this column randomly inserts NULL values, can be specified as "true" or "false", only effective when generate_row_rule is 2.
#### Insertion Behavior Configuration Parameters
#### Insertion Behavior Parameters
- **thread_count**: The number of threads for inserting data, default is 8.
@ -432,7 +331,7 @@ Specify the configuration parameters for tag and data columns in `super_tables`
- **confirm_parameter_prompt** : A toggle parameter that requires user confirmation after a prompt to continue. The value can be "yes" or "no", by default "no".
- **interlace_rows** : Enables interleaved insertion mode and specifies the number of rows to insert into each subtable at a time. Interleaved insertion mode refers to inserting the specified number of rows into each subtable in sequence and repeating this process until all subtable data has been inserted. The default value is 0, meaning data is inserted into one subtable completely before moving to the next.
- **interlace_rows** : Enables interleaved insertion mode and specifies the number of rows to insert into each subtable at a time. Interleaved insertion mode refers to inserting the specified number of rows into each subtable in sequence and repeating this process until all subtable data has been inserted. The default is 0, meaning data is inserted into one subtable completely before moving to the next.
This parameter can also be configured in `super_tables`; if configured, the settings in `super_tables` take higher priority and override the global settings.
- **insert_interval** :
@ -442,68 +341,84 @@ Specify the configuration parameters for tag and data columns in `super_tables`
- **num_of_records_per_req** :
The number of data rows requested per write to TDengine, default is 30000. If set too high, the TDengine client driver will return corresponding error messages, and this parameter needs to be reduced to meet the writing requirements.
- **prepare_rand** : The number of unique values in the generated random data. If it is 1, it means all data are the same. The default value is 10000.
- **prepare_rand** : The number of unique values in the generated random data. If it is 1, it means all data are the same. The default is 10000.
- **pre_load_tb_meta** : Whether to pre-load the meta data of subtables, values are “yes” or "no". When there are a large number of subtables, turning on this option can improve the writing speed.
### Configuration Parameters for Query Scenarios
### Query Parameters
In query scenarios, `filetype` must be set to `query`.
`query_times` specifies the number of times to run the query, numeric type.
Query scenarios can control the execution of slow query statements by setting `kill_slow_query_threshold` and `kill_slow_query_interval` parameters, where threshold controls that queries exceeding the specified exec_usec time will be killed by taosBenchmark, in seconds; interval controls the sleep time to avoid continuous slow query CPU consumption, in seconds.
For other common parameters, see Common Configuration Parameters.
For other common parameters, see [General Configuration Parameters](#general-configuration-parameters)
#### Configuration Parameters for Executing Specified Query Statements
Configuration parameters for querying specified tables (can specify supertables, subtables, or regular tables) are set in `specified_table_query`.
`General Query`: Each SQL in `sqls` starts `threads` threads to query this SQL, Each thread exits after executing the `query_times` queries, and only after all threads executing this SQL have completed can the next SQL be executed.
The total number of queries(`General Query`) = the number of `sqls` * `query_times` * `threads`
- `Mixed Query` : All SQL statements in `sqls` are divided into `threads` groups, with each thread executing one group. Each SQL statement needs to execute `query_times` queries.
The total number of queries(`Mixed Query`) = the number of `sqls` * `query_times`
#### Specified Query
Configuration parameters for querying specified tables (can specify supertables, subtables, or regular tables) are set in `specified_table_query`.
- **mixed_query** : Query Mode . "yes" is `Mixed Query`, "no" is `General Query`, default is "no".
`General Query`:
Each SQL in `sqls` starts `threads` threads to query this SQL, Each thread exits after executing the `query_times` queries, and only after all threads executing this SQL have completed can the next SQL be executed.
The total number of queries(`General Query`) = the number of `sqls` * `query_times` * `threads`
`Mixed Query`:
All SQL statements in `sqls` are divided into `threads` groups, with each thread executing one group. Each SQL statement needs to execute `query_times` queries.
The total number of queries(`Mixed Query`) = the number of `sqls` * `query_times`
- **query_interval** : Query interval, in millisecond, default is 0.
- **threads** : Number of threads executing the SQL query, default is 1.
- **sqls**:
- **sql**: The SQL command to execute, required.
- **result**: File to save the query results, if not specified, results are not saved.
#### Configuration Parameters for Querying Supertables
#### Supertables
Configuration parameters for querying supertables are set in `super_table_query`.
The thread mode of the super table query is the same as the `Normal Query` mode of the specified query statement described above, except that `sqls` is filled all sub tables.
- **stblname** : The name of the supertable to query, required.
- **query_interval** : Query interval, in seconds, default is 0.
- **threads** : Number of threads executing the SQL query, default is 1.
- **sqls** :
- **sql** : The SQL command to execute, required; for supertable queries, keep "xxxx" in the SQL command, the program will automatically replace it with all subtable names of the supertable.
- **result** : File to save the query results, if not specified, results are not saved.
- **Note**: The maximum number of SQL arrays configured under SQL is 100.
### Configuration Parameters for Subscription Scenarios
### Subscription Parameters
In subscription scenarios, `filetype` must be set to `subscribe`, this parameter and other common parameters see Common Configuration Parameters.
In the subscription scenario, `filetype` must be set to `subscribe`. For details of this parameter and other general parameters, see [General Configuration Parameters](#general-configuration-parameters)
The subscription configuration parameters are set under `tmq_info`. The parameters are as follows:
#### Configuration Parameters for Executing Specified Subscription Statements
- **concurrent**: the number of consumers who consume subscriptions, or the number of concurrent consumers. The default value is 1.
- **create_mode**: create a consumer mode.
Which can be sequential: create in sequence. parallel: It is created at the same time. It is required and has no default value.
- **group_mode**: generate the consumer groupId mode.
Which can take the value share: all consumers can only generate one groupId independent: Each consumer generates an independent groupId. If `group.id` is not set, this item is mandatory and has no default value.
-**poll_delay**: The polling timeout time passed in by calling tmq_consumer_poll.
The unit is milliseconds. A negative number means the default timeout is 1 second.
-**enable.manual.commit**: whether manual submission is allowed.
The value can be true: manual submission is allowed, after consuming messages, manually call tmq_commit_sync to complete the submission. falseDo not submit, default value: false.
-**rows_file**: a file that stores consumption data.
It can be a full path or a relative path with a file name.The actual saved file will be followed by the consumer serial number. For example, rows_file is result, and the actual file name is result_1 (consumer 1) result_2 (consumer 2).
-**expect_rows**: the number of rows and data types expected to be consumed by each consumer.
When the consumption reaches this number, the consumption will exit, and the consumption will continue without setting.
-**topic_list**: specifies the topic list and array type of consumption.
Example of topic list format: `{"name": "topic1", "sql": "select * from test.meters;"}`.
name: Specify the topic name.
sql: Specify the sql statement for creating topic, Ensure that the sql is correct, and the framework will automatically create topic.
Configuration parameters for subscribing to specified tables (can specify supertables, subtables, or regular tables) are set in `specified_table_query`.
For the following parameters, see the description of [Subscription](../../../advanced-features/data-subscription/):
- **client.id**
- **auto.offset.reset**
- **enable.auto.commit**
- **enable.auto.commit**
- **msg.with.table.name**
- **auto.commit.interval.ms**
- **group.id**: If this value is not specified, the groupId will be generated by the rule specified by `group_mode`. If this value is specified, the `group_mode` parameter is ignore.
- **threads/concurrent** : Number of threads executing the SQL, default is 1.
- **sqls** :
- **sql** : The SQL command to execute, required.
### Data Type Comparison Table
### Data Type Writing Comparison Table in Configuration File
| # | **Engine** | **taosBenchmark**
| # | **TDengine** | **taosBenchmark**
| --- | :----------------: | :---------------:
| 1 | TIMESTAMP | timestamp
| 2 | INT | int
@ -525,3 +440,97 @@ Configuration parameters for subscribing to specified tables (can specify supert
| 18 | JSON | json
Note: Data types in the taosBenchmark configuration file must be in lowercase to be recognized.
## Example Of Configuration Files
**Below are a few examples of configuration files:**
#### Insertion Example
<details>
<summary>insert.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/insert.json}}
```
</details>
#### Query Example
<details>
<summary>query.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/query.json}}
```
</details>
#### Subscription Example
<details>
<summary>tmq.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/tmq.json}}
```
</details>
Other json examples see [here](https://github.com/taosdata/TDengine/tree/main/tools/taos-tools/example)
## Output Performance Indicators
#### Write Indicators
After writing is completed, a summary performance metric will be output in the last two lines in the following format:
``` bash
SUCC: Spent 8.527298 (real 8.117379) seconds to insert rows: 10000000 with 8 thread(s) into test 1172704.41 (real 1231924.74) records/second
SUCC: insert delay, min: 19.6780ms, avg: 64.9390ms, p90: 94.6900ms, p95: 105.1870ms, p99: 130.6660ms, max: 157.0830ms
```
First line write speed statistics:
- Spent: Total write time, in seconds, counting from the start of writing the first data to the end of the last data. This indicates that a total of 8.527298 seconds were spent.
- Real: Total write time (calling the engine), excluding the time spent preparing data for the testing framework. Purely counting the time spent on engine calls, The time spent is 8.117379 seconds. If 8.527298-8.117379=0.409919 seconds, it is the time spent preparing data for the testing framework.
- Rows: Write the total number of rows, which is 10 million pieces of data.
- Threads: The number of threads being written, which is 8 threads writing simultaneously.
- Records/second write speed = `total write time` / `total number of rows written`, real in parentheses is the same as before, indicating pure engine write speed.
Second line single write delay statistics:
- min: Write minimum delay.
- avg: Write normal delay.
- p90: Write delay p90 percentile delay number.
- p95: Write delay p95 percentile delay number.
- p99: Write delay p99 percentile delay number.
- max: maximum write delay.
Through this series of indicators, the distribution of write request latency can be observed.
#### Query indicators
The query performance test mainly outputs the QPS indicator of query request speed, and the output format is as follows:
``` bash
complete query with 3 threads and 10000 query delay avg: 0.002686s min: 0.001182s max: 0.012189s p90: 0.002977s p95: 0.003493s p99: 0.004645s SQL command: select ...
INFO: Spend 26.9530 second completed total queries: 30000, the QPS of all threads: 1113.049
```
- The first line represents the percentile distribution of query execution and query request delay for each of the three threads executing 10000 queries. The SQL command is the test query statement.
- The second line indicates that the total query time is 26.9653 seconds, and the query rate per second (QPS) is 1113.049 times/second.
- If the `continue_if_fail` option is set to `yes` in the query, the last line will output the number of failed requests and error rate, the format like "error + number of failed requests (error rate)".
- QPS = number of successful requests / time spent (in seconds)
- Error rate = number of failed requests / (number of successful requests + number of failed requests)
#### Subscription indicators
The subscription performance test mainly outputs consumer consumption speed indicators, with the following output format:
``` bash
INFO: consumer id 0 has poll total msgs: 376, period rate: 37.592 msgs/s, total rows: 3760000, period rate: 375924.815 rows/s
INFO: consumer id 1 has poll total msgs: 362, period rate: 36.131 msgs/s, total rows: 3620000, period rate: 361313.504 rows/s
INFO: consumer id 2 has poll total msgs: 364, period rate: 36.378 msgs/s, total rows: 3640000, period rate: 363781.731 rows/s
INFO: consumerId: 0, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 1, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 2, consume msgs: 1000, consume rows: 10000000
INFO: Consumed total msgs: 3000, total rows: 30000000
```
- Lines 1 to 3 real-time output of the current consumption speed of each consumer, msgs/s represents the number of consumption messages, each message contains multiple rows of data, and rows/s represents the consumption speed calculated by rows.
- Lines 4 to 6 show the overall statistics of each consumer after the test is completed, including the total number of messages consumed and the total number of lines.
- The overall statistics of all consumers in line 7, `msgs` represents how many messages were consumed in total, `rows` represents how many rows of data were consumed in total.

View File

@ -73,7 +73,7 @@ If the client encounters a connection failure, please follow the steps below to
### 5. What to do if you encounter the error "Unable to resolve FQDN"?
This error occurs because the client or data node cannot resolve the FQDN (Fully Qualified Domain Name). For the TAOS Shell or client applications, please check the following:
This error occurs because the client or data node cannot resolve the FQDN (Fully Qualified Domain Name). For the TDengine CLI or client applications, please check the following:
1. Check if the FQDN of the server you are connecting to is correct.
2. If there is a DNS server in the network configuration, check if it is working properly
@ -244,15 +244,15 @@ launchctl limit maxfiles
This prompt indicates that the number of vnodes required for creating the db is not enough, exceeding the upper limit of vnodes in the dnode. By default, a dnode contains twice the number of CPU cores worth of vnodes, which can also be controlled by the supportVnodes parameter in the configuration file.
Normally, increase the supportVnodes parameter in taos.cfg.
### 21 Why can data from a specified time period be queried using taos-CLI on the server, but not on the client machine?
### 21 Why can data from a specified time period be queried using TDengine CLI on the server, but not on the client machine?
This issue is due to the client and server having different time zone settings. Adjusting the client's time zone to match the server's will resolve the issue.
### 22 The table name is confirmed to exist, but returns "table name does not exist" when writing or querying, why?
In TDengine, all names, including database names and table names, are case-sensitive. If these names are not enclosed in backticks (\`) in the program or taos-CLI, even if you input them in uppercase, the engine will convert them to lowercase for use. If the names are enclosed in backticks, the engine will not convert them to lowercase and will use them as is.
In TDengine, all names, including database names and table names, are case-sensitive. If these names are not enclosed in backticks (\`) in the program or TDengine CLI, even if you input them in uppercase, the engine will convert them to lowercase for use. If the names are enclosed in backticks, the engine will not convert them to lowercase and will use them as is.
### 23 How to fully display field content in taos-CLI queries?
### 23 How to fully display field content in TDengine CLI queries?
You can use the \G parameter for vertical display, such as `show databases\G\;` (for ease of input, press TAB after "\" to automatically complete the content).
@ -315,4 +315,4 @@ Problem solving: You should configure the automatic mount of the dataDir directo
Directly querying from child table is fast. The query from super table with TAG filter is designed to meet the convenience of querying. It can filter data from multiple child tables at the same time. If the goal is to pursue performance and the child table has been clearly queried, directly querying from the sub table can achieve higher performance
### 35 How to view data compression ratio indicators?
Currently, TDengine only provides compression ratios based on tables, not databases or the entire system. To view the compression ratios, execute the `SHOW TABLE DISTRIBUTED table_name;` command in the client taos-CLI. The table_name can be a super table, regular table, or subtable. For details [Click Here](https://docs.tdengine.com/tdengine-reference/sql-manual/show-commands/#show-table-distributed)
Currently, TDengine only provides compression ratios based on tables, not databases or the entire system. To view the compression ratios, execute the `SHOW TABLE DISTRIBUTED table_name;` command in the client TDengine CLI. The table_name can be a super table, regular table, or subtable. For details [Click Here](https://docs.tdengine.com/tdengine-reference/sql-manual/show-commands/#show-table-distributed)

View File

@ -56,4 +56,4 @@ slug: /release-history/release-notes/3-3-2-0
13. Upgrading to 3.3.0.0 and enabling `cachemodel` causes incorrect row count returns for `last + group by`
14. `taos-explorer` navigation bar does not display all supertable names (Enterprise Edition only)
15. Querying causes `taosd` to crash when compound primary key VARCHAR length exceeds 125
16. High CPU usage by `taos CLI` and `taosAdapter`
16. High CPU usage by `TDengine CLI` and `taosAdapter`

View File

@ -53,7 +53,7 @@ slug: /release-history/release-notes/3-3-3-0
22. Cursor error in data filling during cache update causes taosd to exit abnormally
23. Random incorrect results of STDDEV function calculation
24. Unable to add offline nodes in multi-tier storage and encryption scenarios
25. taos CLI cannot input passwords longer than 20 bytes
25. TDengine CLI cannot input passwords longer than 20 bytes
26. SQL write error: int data overflow
27. Metadata consistency in scenarios of high query concurrency
28. Attempt to solve the issue where manually clicking the stop button does not stop the task
@ -90,4 +90,4 @@ slug: /release-history/release-notes/3-3-3-0
59. Client memory leak
60. Open-source users unable to modify other database options after upgrading stt_trigger value
61. Incorrect results for NOT IN (NULL) queries
62. taos shell and taosBenchmark unable to successfully connect to cloud service instances
62. TDengine CLI and taosBenchmark unable to successfully connect to cloud service instances

View File

@ -52,13 +52,13 @@ slug: /release-history/release-notes/3-3-4-3
1. fix: memory leak caused by repeated addition and deletion of tables on the Windows platform.
1. fix(stream): check the right return code for concurrent checkpoint trans.
1. fix: the "too many session" problem while perform large concurrent queries.
1. fix: the problem of taos shell crashing in slow query scenarios on the Windows platform.
1. fix: the encrypted database cannot be recovered when opening the dnode log.
1. fix: the problem that taosd cannot be started due to mnode synchronization timeout.
1. fix: the slow sorting of file group data during snapshot synchronization leads to the inability of Vnode to recover.
1. fix: when writing data with escape characters to a varchar field throug line protocol, taosd will crash.
1. fix: metadata file damage caused by incorrect logic processing of error code
1. fix: when a query statement contains multiple nested "not" conditional statements, not setting the scalar mode will lead to query errors.
1. fix: the problem of dnode going offline due to timeout of vnode stat report.
1. fix: taosd failed to start on servers that not support AVX instructions.
1. fix(taosX): handle 0x09xx error codes in migration
2. fix: the problem of TDengine CLI crashing in slow query scenarios on the Windows platform.
3. fix: the encrypted database cannot be recovered when opening the dnode log.
4. fix: the problem that taosd cannot be started due to mnode synchronization timeout.
5. fix: the slow sorting of file group data during snapshot synchronization leads to the inability of Vnode to recover.
6. fix: when writing data with escape characters to a varchar field throug line protocol, taosd will crash.
7. fix: metadata file damage caused by incorrect logic processing of error code
8. fix: when a query statement contains multiple nested "not" conditional statements, not setting the scalar mode will lead to query errors.
9. fix: the problem of dnode going offline due to timeout of vnode stat report.
10. fix: taosd failed to start on servers that not support AVX instructions.
11. fix(taosX): handle 0x09xx error codes in migration

View File

@ -10,7 +10,7 @@ slug: /release-history/release-notes/3-3-5-0
2. feat: refactor taosX incremental backup-restore
3. feat: add stmt2 apis in JDBC via websocket connection
4. feat: add stmt2 api in Rust connector
5. feat: add error codes in error prompts in taos-CLI
5. feat: add error codes in error prompts in TDengine CLI
6. feat: superSet can connect TDengine with python connector
7. feat: configurable grafana dashboards in explorer management
8. feat: add taosX-agent in-memory cache queu capacity option

View File

@ -38,6 +38,6 @@ slug: /release-history/release-notes/3.3.5.2
20. fix: column names were not correctly copied when using SELECT * FROM subqueries
21. fix: when performing max/min function on string type data, the results are inaccurate and taosd will crash
22. fix: stream computing does not support the use of the HAVING clause, but no error is reported during creation
23. fix: the version information displayed by taos shell for the server is inaccurate, such as being unable to correctly distinguish between the community edition and the enterprise edition
23. fix: the version information displayed by TDengine CLI for the server is inaccurate, such as being unable to correctly distinguish between the community edition and the enterprise edition
24. fix: in certain specific query scenarios, when JOIN and CAST are used together, taosd may crash

View File

@ -8,7 +8,7 @@ import Tabs from "@theme/Tabs";
import TabItem from "@theme/TabItem";
import PkgListV3 from "/components/PkgListV3";
TDengine 完整的软件包包括服务端taosd、应用驱动taosc、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、命令行程序CLItaos)和一些工具软件。目前 TDinsight 仅在 Linux 系统上安装和运行,后续将支持 Windows、macOS 等系统。TDengine 除了提供多种语言的连接器之外,还通过 [taosAdapter](../../reference/components/taosadapter/) 提供 [RESTful 接口](../../reference/connector/rest-api/)。
TDengine 完整的软件包包括服务端taosd、应用驱动taosc、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、命令行程序TDengine CLI和一些工具软件。目前 TDinsight 仅在 Linux 系统上安装和运行,后续将支持 Windows、macOS 等系统。TDengine 除了提供多种语言的连接器之外,还通过 [taosAdapter](../../reference/components/taosadapter/) 提供 [RESTful 接口](../../reference/connector/rest-api/)。
为方便使用,标准的服务端安装包包含了 taosd、taosAdapter、taosc、taos、taosdump、taosBenchmark、TDinsight 安装脚本和示例代码;如果您只需要用到服务端程序和客户端连接的 C/C++ 语言支持,也可以仅下载 Lite 版本的安装包。

View File

@ -8,7 +8,7 @@ import xiaot_new from './xiaot-20231007.png'
import channel from './channel.webp'
import official_account from './official-account.webp'
TDengine 完整的软件包包括服务端taosd、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、应用驱动taosc、命令行程序 (taos) 和一些工具软件。TDengine 除了提供多种语言的连接器之外,还通过 [taosAdapter](../reference/components/taosadapter) 提供 [RESTful 接口](../reference/connector/rest-api)。
TDengine 完整的软件包包括服务端taosd、用于与第三方系统对接并提供 RESTful 接口的 taosAdapter、应用驱动taosc、命令行程序 (TDengine CLI) 和一些工具软件。TDengine 除了提供多种语言的连接器之外,还通过 [taosAdapter](../reference/components/taosadapter) 提供 [RESTful 接口](../reference/connector/rest-api)。
本章主要介绍如何快速设置 TDengine 环境并体验其高效写入和查询。

View File

@ -3,6 +3,9 @@ title: 预测算法
description: 预测算法
---
import fc_result from '../pic/fc.png';
import fc_result_figure from '../pic/fc-result.png';
时序数据预测处理以持续一个时间段的时序数据作为输入预测接下来一个连续时间区间内时间序列数据趋势。用户可以指定输出的预测时间序列数据点的数量因此其输出的结果行数不确定。为此TDengine 使用新 SQL 函数 `FORECAST` 提供时序数据预测服务。基础数据(用于预测的历史时间序列数据)是该函数的输入,预测结果是该函数的输出。用户可以通过 `FORECAST` 函数调用 Anode 提供的预测算法提供的服务。
在后续章节中,使用时序数据表`foo`作为示例,介绍预测和异常检测算法的使用方式,`foo` 表的模式如下:
@ -106,3 +109,37 @@ taos> select _flow, _fhigh, _frowts, forecast(i32) from foo;
- PatchTST (Patch Time Series Transformer)
- Temporal Fusion Transformer
- TimesNet
## 算法有效性评估工具
TDgpt 提供预测分析算法有效性评估工具 `analytics_compare`,调用该工具并设置合适的参数,能够使用 TDengine 中的数据作为回测依据,评估不同预测算法或相同的预测算法在不同的参数或训练模型的下的预测有效性。预测有效性的评估使用 `MSE``MAE` 指标作为依据,后续还将增加 `MAPE`指标。
```ini
[forecast]
# 训练数据的周期,每个周期包含多少个输入点
period = 10
# 使用范围内最后 10 条记录作为预测结果
rows = 10
# 训练数据开始时间
start_time = 1949-01-01T00:00:00
# 训练数据结束时间
end_time = 1960-12-01T00:00:00
# 输出结果的起始时间
res_start_time = 1730000000000
# 是否绘制预测结果图
gen_figure = true
```
算法对比分析运行完成以后,生成 fc-results.xlsx 文件,其中包含了调用算法的预测分析误差、执行时间、调用参数等信息。如下图所示:
<img src={fc_result} width="760" alt="预测对比结果" />
如果设置了 `gen_figure` 为 true分析结果中还会有绘制的分析预测结果图如下图所示
<img src={fc_result_figure} width="760" alt="预测对比结果" />

View File

@ -4,6 +4,8 @@ description: 异常检测算法
---
import ad from '../pic/anomaly-detection.png';
import ad_result from '../pic/ad-result.png';
import ad_result_figure from '../pic/ad-result-figure.png';
TDengine 中定义了异常(状态)窗口来提供异常检测服务。异常窗口可以视为一种特殊的**事件窗口Event Window**,即异常检测算法确定的连续异常时间序列数据所在的时间窗口。与普通事件窗口区别在于——时间窗口的起始时间和结束时间均是分析算法识别确定,不是用户给定的表达式进行判定。因此,在 `WHERE` 子句中使用 `ANOMALY_WINDOW` 关键词即可调用时序数据异常检测服务,同时窗口伪列(`_WSTART`, `_WEND`, `_WDURATION`)也能够像其他时间窗口一样用于描述异常窗口的起始时间(`_WSTART`)、结束时间(`_WEND`)、持续时间(`_WDURATION`)。例如:
@ -67,3 +69,38 @@ Query OK, 1 row(s) in set (0.028946s)
### 内置异常检测算法
分析平台内置了6个异常检查模型分为3个类别分别是[基于统计学的算法](./02-statistics-approach.md)、[基于数据密度的算法](./03-data-density.md)、以及[基于机器学习的算法](./04-machine-learning.md)。在不指定异常检测使用的方法的情况下,默认调用 IQR 进行异常检测。
### 异常检测算法有效性比较工具
TDgpt 提供自动化的工具对比不同数据集的不同算法监测有效性针对异常检测算法提供查全率recall和查准率precision两个指标衡量不同算法的有效性。
通过在配置文件中(analysis.ini)设置以下的选项可以调用需要使用的异常检测算法,异常检测算法测试用数据的时间范围、是否生成标注结果的图片、调用的异常检测算法以及相应的参数。
调用异常检测算法比较之前,需要人工手动标注异常监测数据集的结果,即设置[anno_res]选项的数值,第几个数值是异常点,需要标注在数组中,如下测试集中,第 9 个点是异常点,我们就标注异常结果为 [9].
```bash
[ad]
# training data start time
start_time = 2021-01-01T01:01:01
# training data end time
end_time = 2021-01-01T01:01:11
# draw the results or not
gen_figure = true
# annotate the anomaly_detection result
anno_res = [9]
# algorithms list that is involved in the comparion
[ad.algos]
ksigma={"k": 2}
iqr={}
grubbs={}
lof={"algo":"auto", "n_neighbor": 3}
```
对比程序执行完成以后,会自动生成名称为`ad_result.xlsx` 的文件,第一个卡片是算法运行结果(如下图所示),分别包含了算法名称、执行调用参数、查全率、查准率、执行时间 5 个指标。
<img src={ad_result} width="760" alt="异常检测对比结果" />
如果设置了 `gen_figure``true`,比较程序会自动将每个参与比较的算法分析结果采用图片方式呈现出来(如下图所示为 ksigma 的异常检测结果标注)。
<img src={ad_result_figure} width="760" alt="异常检测标注图" />

View File

@ -47,7 +47,7 @@ class _MyAnomalyDetectionService(AbstractAnomalyDetectionService):
return super().set_params(params)
```
将该文件保存在 `./lib/taosanalytics/algo/ad/` 目录下,然后重启 taosanode 服务。在 TDengine 命令行接口 taos 中执行 `SHOW ANODES FULL` 就能够看到新加入的算法,然后就可以通过 SQL 语句调用该算法。
将该文件保存在 `./lib/taosanalytics/algo/ad/` 目录下,然后重启 taosanode 服务。在 TDengine CLI 中执行 `SHOW ANODES FULL` 就能够看到新加入的算法,然后就可以通过 SQL 语句调用该算法。
```SQL
--- 对 col 列进行异常检测,通过指定 algo 参数为 myad 来调用新添加的异常检测类

Binary file not shown.

After

Width:  |  Height:  |  Size: 41 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 20 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 61 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 13 KiB

View File

@ -299,7 +299,7 @@ writer.write(lineDemo, SchemalessProtocolType.LINE, SchemalessTimestampType.NANO
## 查询写入的数据
运行上节的样例代码,会在 power 数据库中自动建表,我们可以通过 taos shell 或者应用程序来查询数据。下面给出用 taos shell 查询超级表和 meters 表数据的样例。
运行上节的样例代码,会在 power 数据库l中自动建表我们可以通过 TDengine CLI 或者应用程序来查询数据。下面给出用 TDengine CLI 查询超级表和 meters 表数据的样例。
```shell
taos> show power.stables;

View File

@ -10,7 +10,7 @@ import TabItem from "@theme/TabItem";
TDengine 提供了类似于消息队列产品的数据订阅和消费接口。在许多场景中,采用 TDengine 的时序大数据平台,无须再集成消息队列产品,从而简化应用程序设计并降低运维成本。本章介绍各语言连接器数据订阅的相关 API 以及使用方法。 数据订阅的基础知识请参考 [数据订阅](../../advanced/subscription/)
## 创建主题
请用 taos shell 或者 参考 [执行 SQL](../sql/) 章节用程序执行创建主题的 SQL`CREATE TOPIC IF NOT EXISTS topic_meters AS SELECT ts, current, voltage, phase, groupid, location FROM meters`
请用 TDengine CLI 或者 参考 [执行 SQL](../sql/) 章节用程序执行创建主题的 SQL`CREATE TOPIC IF NOT EXISTS topic_meters AS SELECT ts, current, voltage, phase, groupid, location FROM meters`
上述 SQL 将创建一个名为 topic_meters 的订阅。使用该订阅所获取的消息中的每条记录都由此查询语句 `SELECT ts, current, voltage, phase, groupid, location FROM meters` 所选择的列组成。

View File

@ -87,7 +87,7 @@ RawDataSize = numOfTables × rowSizePerTable × rowsPerTable
**技巧** 如何估算 TDengine 压缩率
```text
用户可以利用性能测试工具 taosBenchmark 来评估 TDengine 的数据压缩效果。通过使用 -f 选项指定写入配置文件taosBenchmark 可以将指定数量的 CSV 样例数据写入指定的库参数和表结构中。
在完成数据写入后,用户可以在 taos shell 中执行 flush database 命令,将所有数据强制写入硬盘。接着,通过 Linux 操作系统的 du 命令获取指定 vnode 的数据文件夹大小。最后,将原始数据大小除以实际存储的数据大小,即可计算出真实的压缩率。
在完成数据写入后,用户可以在 TDengine CLI 中执行 flush database 命令,将所有数据强制写入硬盘。接着,通过 Linux 操作系统的 du 命令获取指定 vnode 的数据文件夹大小。最后,将原始数据大小除以实际存储的数据大小,即可计算出真实的压缩率。
```
通过如下命令可以获得 TDengine 占用的存储空间。

View File

@ -58,7 +58,7 @@ serverPort 6030
#### 5. 启动
按照前述步骤启动第 1 个 dnode例如 h1.taosdata.com。接着在终端中执行 taos启动 TDengine CLI 程序 taos并在其中执行 show dnodes 命令,以查看当前集群中的所有 dnode 信息。
按照前述步骤启动第 1 个 dnode例如 h1.taosdata.com。接着在终端中执行 taos启动 TDengine CLI 程序 taos并在其中执行 show dnodes 命令,以查看当前集群中的所有 dnode 信息。
```shell
taos> show dnodes;
@ -71,7 +71,7 @@ taos> show dnodes;
#### 6. 添加 dnode
按照前述步骤,在每个物理节点启动 taosd。每个 dnode 都需要在 taos.cfg 文件中将 firstEp 参数配置为新建集群首个节点的 endpoint在本例中是 h1.taosdata.com:6030。在第 1 个 dnode 所在机器,在终端中运行 taos打开 TDengine CLI 程序 taos然后登录TDengine 集群,执行如下 SQL。
按照前述步骤,在每个物理节点启动 taosd。每个 dnode 都需要在 taos.cfg 文件中将 firstEp 参数配置为新建集群首个节点的 endpoint在本例中是 h1.taosdata.com:6030。在第 1 个 dnode 所在机器,在终端中运行 taos打开 TDengine CLI 程序 taos然后登录TDengine 集群,执行如下 SQL。
```shell
create dnode "h2.taosdata.com:6030"
@ -86,13 +86,13 @@ show dnodes;
在日志中,请确认输出的 dnode 的 fqdn 和端口是否与你刚刚尝试添加的 endpoint 一致。如果不一致,请修正为正确的 endpoint。遵循上述步骤你可以持续地将新的 dnode 逐个加入集群,从而扩展集群规模并提高整体性能。确保在添加新节点时遵循正确的流程,这有助于维持集群的稳定性和可靠性。
**Tips**
- 任何已经加入集群的 dnode 都可以作为后续待加入节点的 firstEp。firstEp 参数仅仅在该 dnode 首次加入集群时起作用,加入集群后,该 dnode 会保存最新的 mnode 的 endpoint 列表,后续不再依赖这个参数。之后配置文件中的 firstEp 参数主要用于客户端连接,如果没有为 TDengine CLI 设置参数,则默认连接由 firstEp 指定的节点。
- 任何已经加入集群的 dnode 都可以作为后续待加入节点的 firstEp。firstEp 参数仅仅在该 dnode 首次加入集群时起作用,加入集群后,该 dnode 会保存最新的 mnode 的 endpoint 列表,后续不再依赖这个参数。之后配置文件中的 firstEp 参数主要用于客户端连接,如果没有为 TDengine CLI 设置参数,则默认连接由 firstEp 指定的节点。
- 两个没有配置 firstEp 参数的 dnode 在启动后会独立运行。这时无法将其中一个dnode 加入另外一个 dnode形成集群。
- TDengine 不允许将两个独立的集群合并成新的集群。
#### 7. 添加 mnode
在创建 TDengine 集群时,首个 dnode 将自动成为集群的 mnode负责集群的管理和协调工作。为了实现 mnode 的高可用性,后续添加的 dnode 需要手动创建 mnode。请注意一个集群最多允许创建 3 个 mnode且每个 dnode 上只能创建一个 mnode。当集群中的 dnode 数量达到或超过 3 个时,你可以为现有集群创建 mnode。在第 1个 dnode 中,首先通过 TDengine CLI 程序 taos 登录 TDengine然后执行如下 SQL。
在创建 TDengine 集群时,首个 dnode 将自动成为集群的 mnode负责集群的管理和协调工作。为了实现 mnode 的高可用性,后续添加的 dnode 需要手动创建 mnode。请注意一个集群最多允许创建 3 个 mnode且每个 dnode 上只能创建一个 mnode。当集群中的 dnode 数量达到或超过 3 个时,你可以为现有集群创建 mnode。在第 1个 dnode 中,首先通过 TDengine CLI 程序 taos 登录 TDengine然后执行如下 SQL。
```shell
create mnode on dnode <dnodeId>

View File

@ -203,7 +203,7 @@ toc_max_heading_level: 4
通过 “工具” 页面,用户可以了解如下 TDengine 周边工具的使用方法。
- TDengine CLI。
- taosBenchmark。
- taosDump。
- taosdump。
- TDengine 与 BI 工具的集成,例如 Google Data Studio、Power BI、永洪 BI 等。
- TDengine 与 Grafana、Seeq 的集成。

View File

@ -1,6 +1,6 @@
---
title: TDengine CLI 参考手册
sidebar_label: taos
sidebar_label: TDengine CLI
toc_max_heading_level: 4
---
@ -27,102 +27,103 @@ taos>
```
进入 TDengine CLI 后,可执行各种 SQL 语句,包括插入、查询以及各种管理命令。
退出 TDengine CLI 执行 `q``quit``exit` 回车即可
退出 TDengine CLI 执行 `q``quit``exit` 回车即可
```shell
taos> quit
```
## 命令行参数
### 常用参数
### 基础参数
可通过配置命令行参数来改变 TDengine CLI 的行为。以下为常用的几个命令行参数:
- -h HOST: 要连接的 TDengine 服务端所在服务器的 FQDN, 默认值: 127.0.0.1
- -P PORT: 指定服务端所用端口号默认值6030
- -u USER: 连接时使用的用户名默认值root
- -p PASSWORD: 连接服务端时使用的密码,特殊字符如 `! & ( ) < > ; |` 需使用字符 `\` 进行转义处理, 默认值taosdata
- -?, --help: 打印出所有命令行参数
- -h HOST: 要连接的 TDengine 服务端所在服务器的 FQDN, 默认值: 127.0.0.1 。
- -P PORT: 指定服务端所用端口号默认值6030 。
- -u USER: 连接时使用的用户名默认值root 。
- -p PASSWORD: 连接服务端时使用的密码,特殊字符如 `! & ( ) < > ; |` 需使用字符 `\` 进行转义处理, 默认值taosdata 。
- -?, --help: 打印出所有命令行参数。
- -s COMMAND: 以非交互模式执行的 SQL 命令。
### 更多参数
使用 `-s` 参数可进行非交互式执行 SQL执行完成后退出此模式适合在自动化脚本中使用。
如以下命令连接到服务器 h1.taos.com, 执行 -s 指定的 SQL:
```bash
taos -h my-server -s "use db; show tables;"
```
- -a AUTHSTR: 连接服务端的授权信息
- -A: 通过用户名和密码计算授权信息
- -B: 设置 BI 工具显示模式,设置后所有输出都遵循 BI 工具的格式进行输出
- -c CONFIGDIR: 指定配置文件目录Linux 环境下默认为 `/etc/taos`,该目录下的配置文件默认名称为 `taos.cfg`
- -C: 打印 -c 指定的目录中 `taos.cfg` 的配置参数
- -d DATABASE: 指定连接到服务端时使用的数据库
- -E dsn: 使用 WebSocket DSN 连接云服务或者提供 WebSocket 连接的服务端
- -f FILE: 以非交互模式执行 SQL 脚本文件。文件中一个 SQL 语句只能占一行
- -k: 测试服务端运行状态0: unavailable1: network ok2: service ok3: service degraded4: exiting
- -l PKTLEN: 网络测试时使用的测试包大小
- -n NETROLE: 网络连接测试时的测试范围,默认为 `client`, 可选值为 `client`、`server`
- -N PKTNUM: 网络测试时使用的测试包数量
- -r: 将时间列转化为无符号 64 位整数类型输出(即 C 语言中 uint64_t)
- -R: 使用 RESTful 模式连接服务端
- -s COMMAND: 以非交互模式执行的 SQL 命令
- -t: 测试服务端启动状态,状态同-k
- -w DISPLAYWIDTH: 客户端列显示宽度
- -z TIMEZONE: 指定时区,默认为本地时区
- -V: 打印出当前版本号
- -c CONFIGDIR: 指定配置文件目录。
Linux 环境下默认为 `/etc/taos`,该目录下的配置文件默认名称为 `taos.cfg`
使用 `-c` 参数改变 `taosc` 客户端加载配置文件的位置,客户端配置参数参考 [客户端配置](../../components/taosc) 。
以下命令指定了 `taosc` 客户端加载 `/root/cfg/` 下的 `taos.cfg` 配置文件。
```bash
taos -c /root/cfg/
```
### 非交互式执行
### 高级参数
使用 `-s` 参数可进行非交互式执行 SQL执行完成后退出此模式适合在自动化脚本中使用。
如以下命令连接到服务器 h1.taos.com, 执行 -s 指定的 SQL:
```bash
taos -h h1.taos.com -s "use db; show tables;"
```
- -a AUTHSTR: 连接服务端的授权信息。
- -A: 通过用户名和密码计算授权信息。
- -B: 设置 BI 工具显示模式,设置后所有输出都遵循 BI 工具的格式进行输出。
- -C: 打印 -c 指定的目录中 `taos.cfg` 的配置参数。
- -d DATABASE: 指定连接到服务端时使用的数据库。
- -E dsn: 使用 WebSocket DSN 连接云服务或者提供 WebSocket 连接的服务端。
- -f FILE: 以非交互模式执行 SQL 脚本文件。文件中一个 SQL 语句只能占一行。
- -k: 测试服务端运行状态0: unavailable1: network ok2: service ok3: service degraded4: exiting 。
- -l PKTLEN: 网络测试时使用的测试包大小。
- -n NETROLE: 网络连接测试时的测试范围,默认为 `client`, 可选值为 `client`、`server` 。
- -N PKTNUM: 网络测试时使用的测试包数量。
- -r: 将时间列转化为无符号 64 位整数类型输出(即 C 语言中 uint64_t) 。
- -R: 使用 RESTful 模式连接服务端。
- -t: 测试服务端启动状态,状态同 -k 。
- -w DISPLAYWIDTH: 客户端列显示宽度。
- -z TIMEZONE: 指定时区,默认为本地时区。
- -V: 打印出当前版本号。
### taosc 配置文件
使用 `-c` 参数改变 `taosc` 客户端加载配置文件的位置,客户端配置参数参考 [客户端配置](../../components/taosc)
以下命令指定了 `taosc` 客户端加载 `/root/cfg/` 下的 `taos.cfg` 配置文件
```bash
taos -c /root/cfg/
```
## 数据导出/导入
### 数据导出
- 可以使用符号 “>>” 导出查询结果到某个文件中,语法为: sql 查询语句 >> ‘输出文件名’; 输出文件如果不写路径的话,将输出至当前目录下。如 `select * from d0 >> /root/d0.csv;` 将把查询结果输出到 /root/d0.csv 中。
### 数据导入
- 可以使用 insert into table_name file '输入文件名',把上一步中导出的数据文件再导入到指定表中。如 `insert into d0 file '/root/d0.csv';` 表示把上面导出的数据全部再导致至 d0 表中。
## 执行 SQL 脚本
在 TDengine CLI 里可以通过 `source` 命令来运行脚本文件中的多条 SQL 命令。
在 TDengine CLI 里可以通过 `source` 命令来运行脚本文件中的多条 SQL 命令,脚本文件中多条 SQL 按行书写即可
```sql
taos> source <filename>;
```
## 数据导入/导出
### 导出查询结果
## 使用小技巧
- 可以使用符号 “>>” 导出查询结果到某个文件中,语法为: sql 查询语句 >> ‘输出文件名’; 输出文件如果不写路径的话,将输出至当前目录下。如 select * from d0 >> /root/d0.csv; 将把查询结果输出到 /root/d0.csv 中。
### TAB 键自动补全
### 数据从文件导入
- TAB 键前为空命令状态下按 TAB 键,会列出 TDengine CLI 支持的所有命令。
- TAB 键前为空格状态下按 TAB 键,会显示此位置可以出现的所有命令词的第一个,再次按 TAB 键切为下一个。
- TAB 键前为字符串,会搜索与此字符串前缀匹配的所有可出现命令词,并显示第一个,再次按 TAB 键切为下一个。
- 输入反斜杠 `\` + TAB 键, 会自动补全为列显示模式命令词 `\G;`
- 可以使用 insert into table_name file '输入文件名',把上一步中导出的数据文件再导入到指定表中。如 insert into d0 file '/root/d0.csv'; 表示把上面导出的数据全部再导致至 d0 表中。
## 设置字符类型显示宽度
可以在 TDengine CLI 里使用如下命令调整字符显示宽度
### 设置字符列显示宽度
可以在 TDengine CLI 里使用如下命令调整字符串类型字段列显示宽度,默认显示宽度为 30 个字符。
以下命令设置了显示宽度为 120 个字符:
```sql
taos> SET MAX_BINARY_DISPLAY_WIDTH <nn>;
taos> SET MAX_BINARY_DISPLAY_WIDTH 120;
```
如显示的内容后面以 ... 结尾时,表示该内容已被截断,可通过本命令修改显示字符宽度以显示完整的内容。
## TAB 键自动补全
### 其它
- TAB 键前为空命令状态下按 TAB 键,会列出 TDengine CLI 支持的所有命令
- TAB 键前为空格状态下按 TAB 键,会显示此位置可以出现的所有命令词的第一个,再次按 TAB 键切为下一个
- TAB 键前为字符串,会搜索与此字符串前缀匹配的所有可出现命令词,并显示第一个,再次按 TAB 键切为下一个
- 输入反斜杠 `\` + TAB 键, 会自动补全为列显示模式命令词 `\G;`
## 使用小技巧
- 可以使用上下光标键查看历史输入的指令
- 在 TDengine CLI 中使用 `alter user` 命令可以修改用户密码,缺省密码为 `taosdata`
- Ctrl+C 中止正在进行中的查询
- 执行 `RESET QUERY CACHE` 可清除本地表 Schema 的缓存
- 批量执行 SQL 语句。可以将一系列的 TDengine CLI 命令(以英文 ; 结尾,每个 SQL 语句为一行)按行存放在文件里,在 TDengine CLI 里执行命令 `source <file-name>` 自动执行该文件里所有的 SQL 语句
- 可以使用上下光标键查看历史输入的指令。
- 在 TDengine CLI 中使用 `alter user` 命令可以修改用户密码,缺省密码为 `taosdata`
- Ctrl+C 中止正在进行中的查询。
- 执行 `RESET QUERY CACHE` 可清除本地表 Schema 的缓存。
- 批量执行 SQL 语句。可以将一系列的 TDengine CLI 命令(以英文 ; 结尾,每个 SQL 语句为一行)按行存放在文件里,在 TDengine CLI 里执行命令 `source <file-name>` 自动执行该文件里所有的 SQL 语句。
## 错误代码表
在 TDengine 3.3.4.8 版本后 TDengine CLI 在返回错误信息中返回了具体错误码,用户可到 TDengine 官网错误码页面查找具体原因及解决措施,见:[错误码参考表](https://docs.taosdata.com/reference/error-code/)

View File

@ -8,19 +8,19 @@ taosdump 是为开源用户提供的 TDengine 数据备份/恢复工具,备份
## 工具获取
taosdump 是 TDengine 服务器及客户端安装包中默认安装组件,安装后即可使用,参考 [TDengine 安装](../../../get-started/)
taosdump 是 TDengine 服务器及客户端安装包中默认安装组件,安装后即可使用,参考 [TDengine 安装](../../../get-started/)
## 运行
taosdump 需在命令行终端中运行,运行时必须带参数,指明是备份操作或还原操作,如:
``` bash
taosdump -h dev126 -D test -o /root/test/
taosdump -h my-server -D test -o /root/test/
```
以上命令表示备份主机名为 `dev126` 机器上的 `test` 数据库到 `/root/test/` 目录下
以上命令表示备份主机名为 `my-server` 机器上的 `test` 数据库到 `/root/test/` 目录下
``` bash
taosdump -h dev126 -i /root/test/
taosdump -h my-server -i /root/test/
```
以上命令表示把 `/root/test/` 目录下之前备份的数据文件恢复到主机名为 `dev126` 的主机上
以上命令表示把 `/root/test/` 目录下之前备份的数据文件恢复到主机名为 `my-server` 的主机上。
## 命令行参数
@ -71,28 +71,27 @@ Usage: taosdump [OPTION...] dbname [tbname ...]
restore, please adjust the value to a smaller one
and try. The workable value is related to the
length of the row and type of table schema.
-I, --inspect inspect avro file content and print on screen
-I, --inspect inspect avro file content and print on screen.
-L, --loose-mode Using loose mode if the table name and column name
use letter and number only. Default is NOT.
-n, --no-escape No escape char '`'. Default is using it.
-Q, --dot-replace Repalce dot character with underline character in
the table name.(Version 2.5.3)
-T, --thread-num=THREAD_NUM Number of thread for dump in file. Default is
8.
the table name.(Version 2.5.3).
-T, --thread-num=THREAD_NUM Number of thread for dump in file. Default is 8
-W, --rename=RENAME-LIST Rename database name with new name during
importing data. RENAME-LIST:
"db1=newDB1|db2=newDB2" means rename db1 to newDB1
and rename db2 to newDB2 (Version 2.5.4)
and rename db2 to newDB2 (Version 2.5.4).
-k, --retry-count=VALUE Set the number of retry attempts for connection or
query failures
-z, --retry-sleep-ms=VALUE retry interval sleep time, unit ms
-C, --cloud=CLOUD_DSN specify a DSN to access TDengine cloud service
-R, --restful Use RESTful interface to connect TDengine
query failures.
-z, --retry-sleep-ms=VALUE retry interval sleep time, unit ms.
-C, --cloud=CLOUD_DSN specify a DSN to access TDengine cloud service.
-R, --restful Use RESTful interface to connect TDengine.
-t, --timeout=SECONDS The timeout seconds for websocket to interact.
-g, --debug Print debug info.
-?, --help Give this help list
--usage Give a short usage message
-V, --version Print program version
-?, --help Give this help list.
--usage Give a short usage message.
-V, --version Print program version.
Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.
@ -102,15 +101,15 @@ Report bugs to <support@taosdata.com>.
## 常用使用场景
### taosdump 备份数据
### 备份数据
1. 备份所有数据库:指定 `-A``--all-databases` 参数
2. 备份多个指定数据库:使用 `-D db1,db2,...` 参数
3. 备份指定数据库中某些超级表或普通表:使用 `dbname stbname1 stbname2 tbname1 tbname2 ...` 参数,注意这种输入序列第一个参数为数据库名称,且只支持一个数据库,第二个和之后的参数为该数据库中的超级表或普通表名称,中间以空格分隔
1. 备份所有数据库:指定 `-A``--all-databases` 参数
2. 备份多个指定数据库:使用 `-D db1,db2,...` 参数
3. 备份指定数据库中某些超级表或普通表:使用 `dbname stbname1 stbname2 tbname1 tbname2 ...` 参数,注意这种输入序列第一个参数为数据库名称,且只支持一个数据库,第二个和之后的参数为该数据库中的超级表或普通表名称,中间以空格分隔
4. 备份系统 log 库TDengine 集群通常会包含一个系统数据库,名为 `log`,这个数据库内的数据为 TDengine 自我运行的数据taosdump 默认不会对 log 库进行备份。如果有特定需求对 log 库进行备份,可以使用 `-a``--allow-sys` 命令行参数。
5. “宽容”模式备份taosdump 1.4.1 之后的版本提供 `-n` 参数和 `-L` 参数,用于备份数据时不使用转义字符和“宽容”模式,可以在表名、列名、标签名没使用转义字符的情况下减少备份数据时间和备份数据占用空间。如果不确定符合使用 `-n``-L` 条件时请使用默认参数进行“严格”模式进行备份。转义字符的说明请参考[官方文档](../../taos-sql/escape)。
5. “宽容”模式备份taosdump 1.4.1 之后的版本提供 `-n` 参数和 `-L` 参数,用于备份数据时不使用转义字符和“宽容”模式,可以在表名、列名、标签名没使用转义字符的情况下减少备份数据时间和备份数据占用空间。如果不确定符合使用 `-n``-L` 条件时请使用默认参数进行“严格”模式进行备份。转义字符的说明请参考 [官方文档](../../taos-sql/escape)
6. `-o` 参数指定的目录下如果已存在备份文件为防止数据被覆盖taosdump 会报错并退出,请更换其它空目录或清空原来数据后再备份。
7. 目前 taosdump 不支持数据断点继备功能,一旦数据备份中断,需要从头开始。如果备份需要很长时间,建议使用(-S -E 选项)指定开始/结束时间进行分段备份的方法
7. 目前 taosdump 不支持数据断点继备功能,一旦数据备份中断,需要从头开始。如果备份需要很长时间,建议使用(-S -E 选项)指定开始/结束时间进行分段备份的方法
:::tip
- taosdump 1.4.1 之后的版本提供 `-I` 参数,用于解析 avro 文件 schema 和数据,如果指定 `-s` 参数将只解析 schema。
@ -120,13 +119,13 @@ Report bugs to <support@taosdata.com>.
:::
### taosdump 恢复数据
### 恢复数据
- 恢复指定路径下的数据文件:使用 `-i` 参数加上数据文件所在路径。如前面提及,不应该使用同一个目录备份不同数据集合,也不应该在同一路径多次备份同一数据集,否则备份数据会造成覆盖或多次备份。
- taosdump 支持数据恢复至新数据库名下,参数是 -W, 详细见命令行参数说明。
:::tip
taosdump 内部使用 TDengine stmt binding API 进行恢复数据的写入,为提高数据恢复性能,目前使用 16384 为一次写入批次。如果备份数据中有较多列数据,可能会导致产生 "WAL size exceeds limit" 错误,此时可以通过使用 `-B` 参数调整为一个更小的值进行尝试。
taosdump 内部使用 TDengine stmt binding API 进行恢复数据的写入,为提高数据恢复性能,目前使用 16384 为一次写入批次。如果备份数据中有较多列数据,可能会导致产生 "WAL size exceeds limit" 错误,此时可以通过使用 `-B` 参数调整为一个更小的值进行尝试。
:::

View File

@ -12,18 +12,13 @@ taosBenchmark 是 TDengine 服务器及客户端安装包中默认安装组件
## 运行
### 运行方式
taosBenchmark 支持无参数、命令行、配置文件三种运行模式,`命令行` 为 `配置文件` 功能子集,两者同时使用时,以命令行方式优先。
taosBenchmark 支持三种运行模式:
- 无参数模式
- 命令行模式
- JSON 配置文件模式
`命令行方式``JSON 配置文件方式` 功能子集,两者都使用时,命令行方式优先。
:::tip
在运行 taosBenchmark 之前要确保 TDengine 集群已经在正确运行。
:::
**在运行 taosBenchmark 之前要确保 TDengine 集群已经在正确运行。**
### 无命令行参数运行
### 无参数模式
```bash
taosBenchmark
@ -32,9 +27,9 @@ taosBenchmark
在无参数运行时taosBenchmark 默认连接 `/etc/taos/taos.cfg` 中指定的 TDengine 集群。
连接成功后,会默认创建智能电表示例数据库 test创建超级表 meters, 创建子表 1 万,每子写入数据 1 万条,若 test 库已存在,默认会先删再建。
### 使用命令行参数运行
### 命令行模式
命令行支持的参数为写入功能中使用较为频繁的参数,查询与订阅功能不支持命令行方式
命令行支持的参数为写入功能中使用较为频繁的参数,查询与订阅功能不支持命令行方式
示例:
```bash
taosBenchmark -d db -t 100 -n 1000 -T 4 -I stmt -y
@ -42,51 +37,14 @@ taosBenchmark -d db -t 100 -n 1000 -T 4 -I stmt -y
此命令表示使用 `taosBenchmark` 将创建一个名为 `db` 的数据库,并建立默认超级表 `meters`,子表 100 ,使用参数绑定(stmt)方式为每张子表写入 1000 条记录。
### 使用配置文件运行
### 配置文件模式
配置文件方式运行提供了全部功能,所有命令行参数都可以在配置文件中配置运行
以 JSON 配置文件方式运行提供了全部功能,所有命令行参数都可以在配置文件中配置运行
```bash
taosBenchmark -f <json file>
```
**下面为支持的写入、查询、订阅三大功能的配置文件示例:**
#### 写入场景 JSON 配置文件示例
<details>
<summary>insert.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/insert.json}}
```
</details>
#### 查询场景 JSON 配置文件示例
<details>
<summary>query.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/query.json}}
```
</details>
#### 订阅场景 JSON 配置文件示例
<details>
<summary>tmq.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/tmq.json}}
```
</details>
查看更多 json 配置文件示例可 [点击这里](https://github.com/taosdata/TDengine/tree/main/tools/taos-tools/example)
## 命令行参数
| 命令行参数 | 功能说明 |
| ---------------------------- | ----------------------------------------------- |
@ -129,60 +87,6 @@ taosBenchmark -f <json file>
| -V/--version | 显示版本信息并退出。不能与其它参数混用|
| -?/--help | 显示帮助信息并退出。不能与其它参数混用|
## 输出性能指标
#### 写入指标
写入结束后会在最后两行输出总体性能指标,格式如下:
``` bash
SUCC: Spent 8.527298 (real 8.117379) seconds to insert rows: 10000000 with 8 thread(s) into test 1172704.41 (real 1231924.74) records/second
SUCC: insert delay, min: 19.6780ms, avg: 64.9390ms, p90: 94.6900ms, p95: 105.1870ms, p99: 130.6660ms, max: 157.0830ms
```
第一行写入速度统计:
- Spent: 写入总耗时,单位秒,从开始写入第一个数据开始计时到最后一条数据结束,这里表示共花了 8.527298 秒
- real : 写入总耗时(调用引擎),此耗时已抛去测试框架准备数据时间,纯统计在引擎调用上花费的时间,示例为 8.117379 秒8.527298 - 8.117379 = 0.409919 秒则为测试框架准备数据消耗时间
- rows : 写入总行数,为 1000 万条数据
- threads: 写入线程数,这里是 8 个线程同时写入
- records/second 写入速度 = `写入总耗时`/ `写入总行数` 括号中 `real` 同前,表示纯引擎写入速度
第二行单个写入延时统计:
- min : 写入最小延时
- avg : 写入平时延时
- p90 : 写入延时 p90 百分位上的延时数
- p95 : 写入延时 p95 百分位上的延时数
- p99 : 写入延时 p99 百分位上的延时数
- max : 写入最大延时
通过此系列指标,可观察到写入请求延时分布情况
#### 查询指标
查询性能测试主要输出查询请求速度 QPS 指标, 输出格式如下:
``` bash
complete query with 3 threads and 10000 query delay avg: 0.002686s min: 0.001182s max: 0.012189s p90: 0.002977s p95: 0.003493s p99: 0.004645s SQL command: select ...
INFO: Spend 26.9530 second completed total queries: 30000, the QPS of all threads: 1113.049
```
- 第一行表示 3 个线程每个线程执行 10000 次查询及查询请求延时百分位分布情况,`SQL command` 为测试的查询语句
- 第二行表示查询总耗时为 26.9653 秒,每秒查询率(QPS)为1113.049 次/秒
- 如果在查询中设置了 `continue_if_fail` 选项为 `yes`,在最后一行中会输出失败请求个数及错误率,格式 error + 失败请求个数 (错误率)
- QPS = 成功请求数量 / 花费时间(单位秒)
- 错误率 = 失败请求数量 /(成功请求数量 + 失败请求数量)
#### 订阅指标
订阅性能测试主要输出消费者消费速度指标,输出格式如下:
``` bash
INFO: consumer id 0 has poll total msgs: 376, period rate: 37.592 msgs/s, total rows: 3760000, period rate: 375924.815 rows/s
INFO: consumer id 1 has poll total msgs: 362, period rate: 36.131 msgs/s, total rows: 3620000, period rate: 361313.504 rows/s
INFO: consumer id 2 has poll total msgs: 364, period rate: 36.378 msgs/s, total rows: 3640000, period rate: 363781.731 rows/s
INFO: consumerId: 0, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 1, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 2, consume msgs: 1000, consume rows: 10000000
INFO: Consumed total msgs: 3000, total rows: 30000000
```
- 1 ~ 3 行实时输出每个消费者当前的消费速度,`msgs/s` 表示消费消息个数,每个消息中包含多行数据,`rows/s` 表示按行数统计的消费速度
- 4 ~ 6 行是测试完成后每个消费者总体统计,统计共消费了多少条消息,共计多少行
- 第 7 行所有消费者总体统计,`msgs` 表示共消费了多少条消息, `rows` 表示共消费了多少行数据
## 配置文件参数
### 通用配置参数
@ -192,28 +96,28 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **filetype** : 功能分类,可选值为 `insert`, `query``subscribe`。分别对应插入、查询和订阅功能。每个配置文件中只能指定其中之一。
- **cfgdir** : TDengine 客户端配置文件所在的目录,默认路径是 /etc/taos 。
- **host** : 指定要连接的 TDengine 服务端的 FQDN默认值为 localhost。
- **host** : 指定要连接的 TDengine 服务端的 FQDN默认值为 localhost
- **port** : 要连接的 TDengine 服务器的端口号,默认值为 6030。
- **port** : 要连接的 TDengine 服务器的端口号,默认值为 6030
- **user** : 用于连接 TDengine 服务端的用户名,默认为 root。
- **user** : 用于连接 TDengine 服务端的用户名,默认为 root
- **password** : 用于连接 TDengine 服务端的密码,默认值为 taosdata。
### 插入场景配置参数
### 写入配置参数
入场景下 `filetype` 必须设置为 `insert`,该参数及其它通用参数详见[通用配置参数](#通用配置参数)
入场景下 `filetype` 必须设置为 `insert`,该参数及其它通用参数详见 [通用配置参数](#通用配置参数)
- **keep_trying** : 失败后进行重试的次数,默认不重试。需使用 v3.0.9 以上版本。
- **trying_interval** : 失败重试间隔时间,单位为毫秒,仅在 keep_trying 指定重试后有效。需使用 v3.0.9 以上版本。
- **childtable_from 和 childtable_to** : 指定写入子表范围,开闭区间为 [childtable_from, childtable_to).
- **childtable_from 和 childtable_to** : 指定写入子表范围,开闭区间为 [childtable_from, childtable_to)
 
- **continue_if_fail** : 允许用户定义失败后行为
- **continue_if_fail** : 允许用户定义失败后行为
“continue_if_fail”:  “no”, 失败 taosBenchmark 自动退出,默认行为
“continue_if_fail”: “yes”, 失败 taosBenchmark 警告用户,并继续写入
“continue_if_fail”: “smart”, 如果子表不存在失败taosBenchmark 会建立子表并继续写入
“continue_if_fail”:  “no”, 失败 taosBenchmark 自动退出,默认行为
“continue_if_fail”: “yes”, 失败 taosBenchmark 警告用户,并继续写入
“continue_if_fail”: “smart”, 如果子表不存在失败taosBenchmark 会建立子表并继续写入
#### 数据库相关
@ -221,7 +125,7 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **name** : 数据库名。
- **drop** : 数据库已存在时是否删除,可选项为 "yes" 或 "no", 默认为 “yes”
- **drop** : 数据库已存在时是否删除,可选项为 "yes" 或 "no", 默认为 “yes”
#### 超级表相关
@ -229,13 +133,13 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **name**: 超级表名,必须配置,没有默认值。
- **child_table_exists** : 子表是否已经存在,默认值为 "no",可选值为 "yes" 或 "no"。
- **child_table_exists** : 子表是否已经存在,默认值为 "no",可选值为 "yes" 或 "no"
- **childtable_count** : 子表的数量,默认值为 10。
- **childtable_prefix** : 子表名称的前缀,必选配置项,没有默认值。
- **escape_character** : 超级表和子表名称中是否包含转义字符,默认值为 "no",可选值为 "yes" 或 "no"。
- **escape_character** : 超级表和子表名称中是否包含转义字符,默认值为 "no",可选值为 "yes" 或 "no"
- **auto_create_table** : 仅当 insert_mode 为 taosc, rest, stmt 并且 child_table_exists 为 "no" 时生效,该参数为 "yes" 表示 taosBenchmark 在插入数据时会自动创建不存在的表;为 "no" 则表示先提前建好所有表再进行插入。
@ -247,7 +151,7 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **non_stop_mode** : 指定是否持续写入,若为 "yes" 则 insert_rows 失效,直到 Ctrl + C 停止程序,写入才会停止。默认值为 "no",即写入指定数量的记录后停止。注:即使在持续写入模式下 insert_rows 失效,但其也必须被配置为一个非零正整数。
- **line_protocol** : 使用行协议插入数据,仅当 insert_mode 为 sml 或 sml-rest 时生效,可选项为 line, telnet, json。
- **line_protocol** : 使用行协议插入数据,仅当 insert_mode 为 sml 或 sml-rest 时生效,可选项为 line, telnet, json
- **tcp_transfer** : telnet 模式下的通信协议,仅当 insert_mode 为 sml-rest 并且 line_protocol 为 telnet 时生效。如果不配置,则默认为 http 协议。
@ -267,9 +171,9 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **disorder_range** : 指定乱序数据的时间戳回退范围。所生成的乱序时间戳为非乱序情况下应该使用的时间戳减去这个范围内的一个随机值。仅在 `-O/--disorder` 指定的乱序数据百分比大于 0 时有效。
- **timestamp_step** : 每个子表中插入数据的时间戳步长,单位与数据库的 `precision` 一致,默认值是 1。
- **timestamp_step** : 每个子表中插入数据的时间戳步长,单位与数据库的 `precision` 一致,默认值是 1
- **start_timestamp** : 每个子表的时间戳起始值,默认值是 now。
- **start_timestamp** : 每个子表的时间戳起始值,默认值是 now
- **sample_format** : 样本数据文件的类型,现在只支持 "csv" 。
@ -279,10 +183,10 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **tags_file** : 仅当 insert_mode 为 taosc, rest 的模式下生效。 最终的 tag 的数值与 childtable_count 有关,如果 csv 文件内的 tag 数据行小于给定的子表数量,那么会循环读取 csv 文件数据直到生成 childtable_count 指定的子表数量;否则则只会读取 childtable_count 行 tag 数据。也即最终生成的子表数量为二者取小。
- **primary_key** : 指定超级表是否有复合主键,取值 1 和 0 复合主键列只能是超级表的第二列,指定生成复合主键后要确保第二列符合复合主键的数据类型,否则会报错
- **repeat_ts_min** : 数值类型,复合主键开启情况下指定生成相同时间戳记录的最小个数,生成相同时间戳记录的个数是在范围[repeat_ts_min, repeat_ts_max] 内的随机值, 最小值等于最大值时为固定个数
- **repeat_ts_max** : 数值类型,复合主键开启情况下指定生成相同时间戳记录的最大个数
- **sqls** : 字符串数组类型,指定超级表创建成功后要执行的 sql 数组sql 中指定表名前面要带数据库名,否则会报未指定数据库错误
- **primary_key** : 指定超级表是否有复合主键,取值 1 和 0 复合主键列只能是超级表的第二列,指定生成复合主键后要确保第二列符合复合主键的数据类型,否则会报错
- **repeat_ts_min** : 数值类型,复合主键开启情况下指定生成相同时间戳记录的最小个数,生成相同时间戳记录的个数是在范围[repeat_ts_min, repeat_ts_max] 内的随机值, 最小值等于最大值时为固定个数
- **repeat_ts_max** : 数值类型,复合主键开启情况下指定生成相同时间戳记录的最大个数
- **sqls** : 字符串数组类型,指定超级表创建成功后要执行的 sql 数组sql 中指定表名前面要带数据库名,否则会报未指定数据库错误
#### 标签列与数据列
@ -308,27 +212,27 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **values** : nchar/binary 列/标签的值域,将从值中随机选择。
- **sma**: 将该列加入 SMA 中,值为 "yes" 或者 "no",默认为 "no"。
- **sma**: 将该列加入 SMA 中,值为 "yes" 或者 "no",默认为 "no"
- **encode**: 字符串类型,指定此列两级压缩中的第一级编码算法,详细参见创建超级表
- **encode**: 字符串类型,指定此列两级压缩中的第一级编码算法,详细参见创建超级表
- **compress**: 字符串类型,指定此列两级压缩中的第二级加密算法,详细参见创建超级表
- **compress**: 字符串类型,指定此列两级压缩中的第二级加密算法,详细参见创建超级表
- **level**: 字符串类型,指定此列两级压缩中的第二级加密算法的压缩率高低,详细参见创建超级表
- **level**: 字符串类型,指定此列两级压缩中的第二级加密算法的压缩率高低,详细参见创建超级表
- **gen**: 字符串类型,指定此列生成数据的方式,不指定为随机,若指定为 “order”, 会按自然数顺序增长
- **gen**: 字符串类型,指定此列生成数据的方式,不指定为随机,若指定为 “order”, 会按自然数顺序增长
- **fillNull**: 字符串类型,指定此列是否随机插入 NULL 值,可指定为 “true” 或 "false", 只有当 generate_row_rule 为 2 时有效
- **fillNull**: 字符串类型,指定此列是否随机插入 NULL 值,可指定为 “true” 或 "false", 只有当 generate_row_rule 为 2 时有效
#### 入行为相关
#### 入行为相关
- **thread_count** : 插入数据的线程数量,默认为 8。
**thread_bind_vgroup** : 写入时 vgroup 是否和写入线程绑定,绑定后可提升写入速度, 取值为 "yes" 或 "no",默认值为 “no”, 设置为 “no” 后与原来行为一致。 当设为 “yes” 时,如果 thread_count 大于写入数据库 vgroups 数量, thread_count 自动调整为 vgroups 数量;如果 thread_count 小于 vgroups 数量,写入线程数量不做调整,一个线程写完一个 vgroup 数据后再写下一个,同时保持一个 vgroup 同时只能由一个线程写入的规则。
- **thread_bind_vgroup** : 写入时 vgroup 是否和写入线程绑定,绑定后可提升写入速度, 取值为 "yes" 或 "no",默认值为 “no”, 设置为 “no” 后与原来行为一致。 当设为 “yes” 时,如果 thread_count 大于写入数据库 vgroups 数量, thread_count 自动调整为 vgroups 数量;如果 thread_count 小于 vgroups 数量,写入线程数量不做调整,一个线程写完一个 vgroup 数据后再写下一个,同时保持一个 vgroup 同时只能由一个线程写入的规则。
- **create_table_thread_count** : 建表的线程数量,默认为 8。
- **result_file** : 结果输出文件的路径,默认值为 ./output.txt。
- **result_file** : 结果输出文件的路径,默认值为 ./output.txt
- **confirm_parameter_prompt** : 开关参数,要求用户在提示后确认才能继续, 可取值 "yes" or "no"。默认值为 "no" 。
@ -346,15 +250,15 @@ INFO: Consumed total msgs: 3000, total rows: 30000000
- **pre_load_tb_meta** :是否提前加载子表的 meta 数据,取值为 “yes” or "no"。当子表数量非常多时,打开此选项可提高写入速度。
### 查询场景配置参数
### 查询配置参数
查询场景下 `filetype` 必须设置为 `query`
`query_times` 指定运行查询的次数,数值类型
`query_times` 指定运行查询的次数,数值类型
查询场景可以通过设置 `kill_slow_query_threshold``kill_slow_query_interval` 参数来控制杀掉慢查询语句的执行threshold 控制如果 exec_usec 超过指定时间的查询将被 taosBenchmark 杀掉,单位为秒
查询场景可以通过设置 `kill_slow_query_threshold``kill_slow_query_interval` 参数来控制杀掉慢查询语句的执行threshold 控制如果 exec_usec 超过指定时间的查询将被 taosBenchmark 杀掉,单位为秒
interval 控制休眠时间,避免持续查询慢查询消耗 CPU ,单位为秒。
其它通用参数详见[通用配置参数](#通用配置参数)
其它通用参数详见 [通用配置参数](#通用配置参数)
#### 执行指定查询语句
@ -391,23 +295,35 @@ interval 控制休眠时间,避免持续查询慢查询消耗 CPU ,单位为
- **sqls**
- **sql** : 执行的 SQL 命令,必填;对于超级表的查询 SQL在 SQL 命令中必须保留 "xxxx",会替换为超级下所有子表名后再执行。
- **result** : 保存查询结果的文件,未指定则不保存。
- **限制项** : sqls 下配置 sql 数组最大为 100 个
- **限制项** : sqls 下配置 sql 数组最大为 100 个
### 订阅场景配置参数
### 订阅配置参数
订阅场景下 `filetype` 必须设置为 `subscribe`,该参数及其它通用参数详见[通用配置参数](#通用配置参数)
订阅场景下 `filetype` 必须设置为 `subscribe`,该参数及其它通用参数详见 [通用配置参数](#通用配置参数)
订阅指定表(可以指定超级表、子表或者普通表)的配置参数在 `specified_table_query` 中设置。
订阅配置参数在 `tmq_info` 项下设置,参数如下:
- **threads/concurrent** : 执行 SQL 的线程数,默认为 1。
- **concurrent** : 消费订阅的消费者数量或称并发消费数量默认值1。
- **create_mode** : 创建消费者模式,可取值 sequential顺序创建 parallel并发同时创建必填项无默认值。
- **group_mode** : 生成消费者 groupId 模式,可取值 share所有消费者只生成一个 groupId independent每个消费者生成一个独立的 groupId如果 `group.id` 未设置,此项为必填项,无默认值。
- **poll_delay** : 调用 tmq_consumer_poll 传入的轮询超时时间,单位为毫秒,负数表示默认超时 1 秒。
- **enable.manual.commit** : 是否允许手动提交,可取值 true允许手动提交每次消费完消息后手动调用 tmq_commit_sync 完成提交, false不进行提交默认值: false。
- **rows_file** : 存储消费数据的文件,可以为全路径或相对路径,带文件名。实际保存的文件会在后面加上消费者序号,如 rows_file 为 result, 实际文件名为 result_1消费者 1 result_2消费者 2 ...
- **expect_rows** : 期望每个消费者消费的行数,数据类型,当消费达到这个数,消费会退出,不设置会一直消费。
- **topic_list** : 指定消费的 topic 列表数组类型。topic 列表格式示例: `{"name": "topic1", "sql": "select * from test.meters;"}` name指定 topic 名sql指定创建 topic 的 sql 语句,需保证 sql 正确,框架会自动创建出 topic。
- **sqls**
- **sql** : 执行的 SQL 命令,必填。
以下参数透传订阅属性,参见 [订阅创建参数](../../../develop/tmq/#创建参数) 说明:
- **client.id**
- **auto.offset.reset**
- **enable.auto.commit**
- **enable.auto.commit**
- **msg.with.table.name**
- **auto.commit.interval.ms**
- **group.id** : 若此值不指定,将由 `group_mode` 指定规则生成 groupId若指定此值`group_mode` 参数不再有效。
### 配置文件中数据类型书写对照表
### 数据类型对照表
| # | **引擎** | **taosBenchmark**
| # | **TDengine** | **taosBenchmark**
| --- | :----------------: | :---------------:
| 1 | TIMESTAMP | timestamp
| 2 | INT | int
@ -428,7 +344,96 @@ interval 控制休眠时间,避免持续查询慢查询消耗 CPU ,单位为
| 17 | GEOMETRY | geometry
| 18 | JSON | json
注意taosBenchmark 配置文件中数据类型必须小写方可识别
注意taosBenchmark 配置文件中数据类型必须小写方可识别
## 配置文件示例
**下面为支持的写入、查询、订阅三大功能的配置文件示例:**
### 写入 JSON 示例
<details>
<summary>insert.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/insert.json}}
```
</details>
### 查询 JSON 示例
<details>
<summary>query.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/query.json}}
```
</details>
### 订阅 JSON 示例
<details>
<summary>tmq.json</summary>
```json
{{#include /TDengine/tools/taos-tools/example/tmq.json}}
```
</details>
查看更多 json 配置文件示例可 [点击这里](https://github.com/taosdata/TDengine/tree/main/tools/taos-tools/example)
## 输出性能指标
#### 写入指标
写入结束后会在最后两行输出总体性能指标,格式如下:
``` bash
SUCC: Spent 8.527298 (real 8.117379) seconds to insert rows: 10000000 with 8 thread(s) into test 1172704.41 (real 1231924.74) records/second
SUCC: insert delay, min: 19.6780ms, avg: 64.9390ms, p90: 94.6900ms, p95: 105.1870ms, p99: 130.6660ms, max: 157.0830ms
```
第一行写入速度统计:
- Spent: 写入总耗时,单位秒,从开始写入第一个数据开始计时到最后一条数据结束,这里表示共花了 8.527298 秒。
- real : 写入总耗时(调用引擎),此耗时已抛去测试框架准备数据时间,纯统计在引擎调用上花费的时间,示例为 8.117379 秒8.527298 - 8.117379 = 0.409919 秒则为测试框架准备数据消耗时间
- rows : 写入总行数,为 1000 万条数据。
- threads: 写入线程数,这里是 8 个线程同时写入。
- records/second 写入速度 = `写入总耗时`/ `写入总行数` 括号中 `real` 同前,表示纯引擎写入速度。
第二行单个写入延时统计:
- min : 写入最小延时。
- avg : 写入平时延时。
- p90 : 写入延时 p90 百分位上的延时数。
- p95 : 写入延时 p95 百分位上的延时数。
- p99 : 写入延时 p99 百分位上的延时数。
- max : 写入最大延时。
通过此系列指标,可观察到写入请求延时分布情况。
#### 查询指标
查询性能测试主要输出查询请求速度 QPS 指标, 输出格式如下:
``` bash
complete query with 3 threads and 10000 query delay avg: 0.002686s min: 0.001182s max: 0.012189s p90: 0.002977s p95: 0.003493s p99: 0.004645s SQL command: select ...
INFO: Spend 26.9530 second completed total queries: 30000, the QPS of all threads: 1113.049
```
- 第一行表示 3 个线程每个线程执行 10000 次查询及查询请求延时百分位分布情况,`SQL command` 为测试的查询语句。
- 第二行表示查询总耗时为 26.9653 秒,每秒查询率(QPS)为1113.049 次/秒。
- 如果在查询中设置了 `continue_if_fail` 选项为 `yes`,在最后一行中会输出失败请求个数及错误率,格式 error + 失败请求个数 (错误率)。
- QPS = 成功请求数量 / 花费时间(单位秒)
- 错误率 = 失败请求数量 /(成功请求数量 + 失败请求数量)
#### 订阅指标
订阅性能测试主要输出消费者消费速度指标,输出格式如下:
``` bash
INFO: consumer id 0 has poll total msgs: 376, period rate: 37.592 msgs/s, total rows: 3760000, period rate: 375924.815 rows/s
INFO: consumer id 1 has poll total msgs: 362, period rate: 36.131 msgs/s, total rows: 3620000, period rate: 361313.504 rows/s
INFO: consumer id 2 has poll total msgs: 364, period rate: 36.378 msgs/s, total rows: 3640000, period rate: 363781.731 rows/s
INFO: consumerId: 0, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 1, consume msgs: 1000, consume rows: 10000000
INFO: consumerId: 2, consume msgs: 1000, consume rows: 10000000
INFO: Consumed total msgs: 3000, total rows: 30000000
```
- 1 ~ 3 行实时输出每个消费者当前的消费速度,`msgs/s` 表示消费消息个数,每个消息中包含多行数据,`rows/s` 表示按行数统计的消费速度。
- 4 ~ 6 行是测试完成后每个消费者总体统计,统计共消费了多少条消息,共计多少行。
- 第 7 行所有消费者总体统计,`msgs` 表示共消费了多少条消息, `rows` 表示共消费了多少行数据。

View File

@ -81,7 +81,7 @@ description: 一些常见问题的解决方法汇总
### 5. 遇到错误 Unable to resolve FQDN” 怎么办?
产生这个错误,是由于客户端或数据节点无法解析 FQDN(Fully Qualified Domain Name)导致。对于 TAOS Shell 或客户端应用,请做如下检查:
产生这个错误,是由于客户端或数据节点无法解析 FQDN(Fully Qualified Domain Name)导致。对于 TDengine CLI 或客户端应用,请做如下检查:
1. 请检查连接的服务器的 FQDN 是否正确FQDN 配置参考:[一篇文章说清楚 TDengine 的 FQDN](https://www.taosdata.com/blog/2020/09/11/1824.html)
2. 如果网络配置有 DNS server请检查是否正常工作
@ -247,13 +247,13 @@ launchctl limit maxfiles
该提示是创建 db 的 vnode 数量不够了,需要的 vnode 不能超过了 dnode 中 vnode 的上限。因为系统默认是一个 dnode 中有 CPU 核数两倍的 vnode也可以通过配置文件中的参数 supportVnodes 控制。
正常调大 taos.cfg 中 supportVnodes 参数即可。
### 21 在服务器上的使用 taos-CLI 能查到指定时间段的数据,但在客户端机器上查不到?
### 21 在服务器上的使用 TDengine CLI能查到指定时间段的数据,但在客户端机器上查不到?
这种情况是因为客户端与服务器上设置的时区不一致导致的,调整客户端与服务器的时区一致即可解决。
### 22 表名确认是存在的,但在写入或查询时返回表名不存在,什么原因?
TDengine 中的所有名称,包括数据库名、表名等都是区分大小写的,如果这些名称在程序或 taos-CLI 中没有使用反引号(`)括起来使用,即使你输入的是大写的,引擎也会转化成小写来使用,如果名称前后加上了反引号,引擎就不会再转化成小写,会保持原样来使用。
TDengine 中的所有名称,包括数据库名、表名等都是区分大小写的,如果这些名称在程序或 TDengine CLI中没有使用反引号(`)括起来使用,即使你输入的是大写的,引擎也会转化成小写来使用,如果名称前后加上了反引号,引擎就不会再转化成小写,会保持原样来使用。
### 23 在 taos-CLI 中查询,字段内容不能完全显示出来怎么办?
### 23 在 TDengine CLI中查询,字段内容不能完全显示出来怎么办?
可以使用 \G 参数来竖式显示,如 show databases\G; (为了输入方便,在"\"后加 TAB 键,会自动补全后面的内容)
### 24 使用 taosBenchmark 测试工具写入数据查询很快,为什么我写入的数据查询非常慢?
@ -311,7 +311,7 @@ TDinsight插件中展示的数据是通过taosKeeper和taosAdapter服务收集
直接查子表更快。超级表带 TAG 过滤查询子查数据是为满足查询方便性,同时可对多个子表中数据进行过滤,如果目的是追求性能并已明确查询子表,直接从子表查性能更高
### 35 如何查看数据库的数据压缩率和磁盘占用指标?
TDengine 3.3.5.0 之前的版本,只提供以表为统计单位的压缩率,数据库及整体还未提供,查看命令是在客户端 taos-CLI 中执行 `SHOW TABLE DISTRIBUTED table_name;` 命令table_name 为要查看压缩率的表,可以为超级表、普通表及子表,详细可 [查看此处](https://docs.taosdata.com/reference/taos-sql/show/#show-table-distributed)
TDengine 3.3.5.0 之前的版本,只提供以表为统计单位的压缩率,数据库及整体还未提供,查看命令是在客户端 TDengine CLI 中执行 `SHOW TABLE DISTRIBUTED table_name;` 命令table_name 为要查看压缩率的表,可以为超级表、普通表及子表,详细可 [查看此处](https://docs.taosdata.com/reference/taos-sql/show/#show-table-distributed)
TDengine 3.3.5.0 及以上的版本,还提供了数据库整体压缩率和磁盘空间占用统计。查看数据库整体的数据压缩率和磁盘空间占用的命令为 `SHOW db_name.disk_info;`,查看数据库各个模块的磁盘空间占用的命令为 `SELECT * FROM INFORMATION_SCHEMA.INS_DISK_USAGE WHERE db_name='db_name';`db_name 为要查看的数据库名称。详细可 [查看此处](https://docs.taosdata.com/reference/taos-sql/database/#%E6%9F%A5%E7%9C%8B-db-%E7%9A%84%E7%A3%81%E7%9B%98%E7%A9%BA%E9%97%B4%E5%8D%A0%E7%94%A8)
@ -324,3 +324,4 @@ TDengine 3.3.5.1 之前的版本StartLimitInterval 为 60 秒。若在 60 秒
问题解决:
1通过 systemd 重启 taosd 服务:推荐方法是先执行命令 `systemctl reset-failed taosd.service` 重置失败计数器,然后再通过 `systemctl restart taosd.service` 重启;若需长期调整,可手动修改 /etc/systemd/system/taosd.service 文件,将 StartLimitInterval 调小或将 StartLimitBurst 调大(注:重新安装 taosd 会重置该参数,需要重新修改),执行 `systemctl daemon-reload` 重新加载配置然后再重启。2也可以不通过 systemd 而是通过 taosd 命令直接重启 taosd 服务,此时不受 StartLimitInterval 和 StartLimitBurst 参数限制。

View File

@ -50,4 +50,4 @@ description: 3.3.2.0 版本说明
13. 升级至3.3.0.0开启cachemodel后查询last+group by返回行数不正确
14. taos-explorer导航栏未显示所有超级表名仅企业版
15. 复合主键VARCHAR长度超125遇查询导致taosd异常退出
16. taos CLI和taosAdapter占用CPU过高
16. TDengine CLI 和taosAdapter占用CPU过高

View File

@ -49,7 +49,7 @@ description: 3.3.3.0 版本说明
22. 缓存更新时数据填充的游标错误导致的 taosd 异常退出
23. STDDEV 函数计算结果随机不正确的问题
24. 多级存储以及加密场景下无法添加离线节点
25. taos CLI 无法输入大于20字节长度的密码
25. TDengine CLI 无法输入大于20字节长度的密码
26. 拼接 sql 写入报错: int data overflow
27. 大量查询并发场景下元数据的一致性
28. 尝试解决手动点击停止按钮,任务无法停止的问题
@ -86,4 +86,4 @@ description: 3.3.3.0 版本说明
59. 客户端内存泄漏
60. 开源用户修改stt_trigger值升级后无法修改其他数据库选项
61. NOT IN (NULL) 查询结果不正确
62. taos shell和taosBenchmark不能成功连接云服务实例
62. TDengine CLI 和 taosBenchmark 不能成功连接云服务实例

View File

@ -57,7 +57,7 @@ description: 3.3.4.3 版本说明
1. 修复Windows 平台重复增删表的内存泄漏
1. 修复:无法限制并发拉起 checkpoint 数量导致流计算消耗资源过多
1. 修复:并发查询时的 too many session 问题
1. 修复Windows 平台下 taos shell 在慢查询场景中崩溃的问题
1. 修复Windows 平台下 TDengine CLI 在慢查询场景中崩溃的问题
1. 修复:当打开 dnode日志时加密数据库无法恢复的问题
1. 修复:由于 mnode 同步超时,进而导致 taosd 无法启动的问题
1. 修复:由于在快照同步过程中整理文件组数据的速度过慢,从而导致 Vnode虚拟节点无法恢复的问题

View File

@ -9,7 +9,7 @@ description: 3.3.5.0 版本说明
2. 特性taosX 增量备份与恢复
3. 特性JDBC WebSocket 连接支持 STMT2 接口
4. 特性Rust 连接器支持 STMT2 接口
5. 特性:taos-CLI 中在错误提示中增加错误码
5. 特性:TDengine CLI 中在错误提示中增加错误码
6. 特性Python 连接器对接 SuperSet
7. 特性Explorer 可配置 Grafana Dashboard
8. 特性taosX-agent 支持配置内存缓存队列长度

View File

@ -37,6 +37,6 @@ description: 3.3.5.2 版本说明
20. 修复:在使用 SELECT * FROM 子查询时,列名未能正确复制到外层查询
21. 修复:对字符串类型数据执行 max/min 函数时,结果不准确且 taosd 可能会崩溃
22. 修复:流式计算不支持使用 HAVING 语句,但在创建时未报告错误
23. 修复:taos shell 显示的服务端版本信息不准确,例如无法正确区分社区版和企业版
23. 修复:TDengine CLI 显示的服务端版本信息不准确,例如无法正确区分社区版和企业版
24. 修复:在某些特定的查询场景下,当 JOIN 和 CAST 联合使用时taosd 可能会崩溃