157 lines
6.2 KiB
Markdown
157 lines
6.2 KiB
Markdown
<center><h1>User's Guide to the Crash_Gen Tool</h1></center>
|
|
|
|
# Introduction
|
|
|
|
To effectively test and debug our TDengine product, we have developed a simple tool to
|
|
exercise various functions of the system in a randomized fashion, hoping to expose
|
|
maximum number of problems, hopefully without a pre-determined scenario.
|
|
|
|
# Features
|
|
|
|
This tool can run as a test client with the following features:
|
|
|
|
1. Any number of concurrent threads
|
|
1. Any number of test steps/loops
|
|
1. Auto-create and writing to multiple databases
|
|
1. Ignore specific error codes
|
|
1. Write small or large data blocks
|
|
1. Auto-generate out-of-sequence data, if needed
|
|
1. Verify the result of write operations
|
|
1. Concurrent writing to a shadow database for later data verification
|
|
1. User specified number of replicas to use, against clusters
|
|
|
|
This tool can also use to start a TDengine service, either in stand-alone mode or
|
|
cluster mode. The features include:
|
|
|
|
1. User specified number of D-Nodes to create/use.
|
|
|
|
# Preparation
|
|
|
|
To run this tool, please ensure the followed preparation work is done first.
|
|
|
|
1. Fetch a copy of the TDengine source code, and build it successfully in the `build/`
|
|
directory
|
|
1. Ensure that the system has Python3.8 or above properly installed. We use
|
|
Ubuntu 20.04LTS as our own development environment, and suggest you also use such
|
|
an environment if possible.
|
|
|
|
# Simple Execution as Client Test Tool
|
|
|
|
To run the tool with the simplest method, follow the steps below:
|
|
|
|
1. Open a terminal window, start the `taosd` service in the `build/` directory
|
|
(or however you prefer to start the `taosd` service)
|
|
1. Open another terminal window, go into the `tests/pytest/` directory, and
|
|
run `./crash_gen.sh -p -t 3 -s 10` (change the two parameters here as you wish)
|
|
1. Watch the output to the end and see if you get a `SUCCESS` or `FAILURE`
|
|
|
|
That's it!
|
|
|
|
# Running Server-side Clusters
|
|
|
|
This tool also makes it easy to test/verify the clustering capabilities of TDengine. You
|
|
can start a cluster quite easily with the following command:
|
|
|
|
```
|
|
$ cd tests/pytest/
|
|
$ rm -rf ../../build/cluster_dnode_?; ./crash_gen.sh -e -o 3 # first part optional
|
|
```
|
|
|
|
The `-e` option above tells the tool to start the service, and do not run any tests, while
|
|
the `-o 3` option tells the tool to start 3 DNodes and join them together in a cluster.
|
|
Obviously you can adjust the the number here. The `rm -rf` command line is optional
|
|
to clean up previous cluster data, so that we can start from a clean state with no data
|
|
at all.
|
|
|
|
## Behind the Scenes
|
|
|
|
When the tool runs a cluster, it users a number of directories, each holding the information
|
|
for a single DNode, see:
|
|
|
|
```
|
|
$ ls build/cluster*
|
|
build/cluster_dnode_0:
|
|
cfg data log
|
|
|
|
build/cluster_dnode_1:
|
|
cfg data log
|
|
|
|
build/cluster_dnode_2:
|
|
cfg data log
|
|
```
|
|
|
|
Therefore, when something goes wrong and you want to reset everything with the cluster, simple
|
|
erase all the files:
|
|
|
|
```
|
|
$ rm -rf build/cluster_dnode_*
|
|
```
|
|
|
|
## Addresses and Ports
|
|
|
|
The DNodes in the cluster all binds the the `127.0.0.1` IP address (for now anyway), and
|
|
uses port 6030 for the first DNode, and 6130 for the 2nd one, and so on.
|
|
|
|
## Testing Against a Cluster
|
|
|
|
In a separate terminal window, you can invoke the tool in client mode and test against
|
|
a cluster, such as:
|
|
|
|
```
|
|
$ ./crash_gen.sh -p -t 10 -s 100 -i 3
|
|
```
|
|
|
|
Here the `-i` option tells the tool to always create tables with 3 replicas, and run
|
|
all tests against such tables.
|
|
|
|
# Additional Features
|
|
|
|
The exhaustive features of the tool is available through the `-h` option:
|
|
|
|
```
|
|
$ ./crash_gen.sh -h
|
|
usage: crash_gen_bootstrap.py [-h] [-a] [-b MAX_DBS] [-c CONNECTOR_TYPE] [-d] [-e] [-g IGNORE_ERRORS]
|
|
[-i NUM_REPLICAS] [-k] [-l] [-m] [-n]
|
|
[-o NUM_DNODES] [-p] [-r] [-s MAX_STEPS] [-t NUM_THREADS] [-v] [-w] [-x]
|
|
|
|
TDengine Auto Crash Generator (PLEASE NOTICE the Prerequisites Below)
|
|
---------------------------------------------------------------------
|
|
1. You build TDengine in the top level ./build directory, as described in offical docs
|
|
2. You run the server there before this script: ./build/bin/taosd -c test/cfg
|
|
|
|
optional arguments:
|
|
-h, --help show this help message and exit
|
|
-a, --auto-start-service
|
|
Automatically start/stop the TDengine service (default: false)
|
|
-b MAX_DBS, --max-dbs MAX_DBS
|
|
Maximum number of DBs to keep, set to disable dropping DB. (default: 0)
|
|
-c CONNECTOR_TYPE, --connector-type CONNECTOR_TYPE
|
|
Connector type to use: native, rest, or mixed (default: 10)
|
|
-d, --debug Turn on DEBUG mode for more logging (default: false)
|
|
-e, --run-tdengine Run TDengine service in foreground (default: false)
|
|
-g IGNORE_ERRORS, --ignore-errors IGNORE_ERRORS
|
|
Ignore error codes, comma separated, 0x supported (default: None)
|
|
-i NUM_REPLICAS, --num-replicas NUM_REPLICAS
|
|
Number (fixed) of replicas to use, when testing against clusters. (default: 1)
|
|
-k, --track-memory-leaks
|
|
Use Valgrind tool to track memory leaks (default: false)
|
|
-l, --larger-data Write larger amount of data during write operations (default: false)
|
|
-m, --mix-oos-data Mix out-of-sequence data into the test data stream (default: true)
|
|
-n, --dynamic-db-table-names
|
|
Use non-fixed names for dbs/tables, for -b, useful for multi-instance executions (default: false)
|
|
-o NUM_DNODES, --num-dnodes NUM_DNODES
|
|
Number of Dnodes to initialize, used with -e option. (default: 1)
|
|
-p, --per-thread-db-connection
|
|
Use a single shared db connection (default: false)
|
|
-r, --record-ops Use a pair of always-fsynced fils to record operations performing + performed, for power-off tests (default: false)
|
|
-s MAX_STEPS, --max-steps MAX_STEPS
|
|
Maximum number of steps to run (default: 100)
|
|
-t NUM_THREADS, --num-threads NUM_THREADS
|
|
Number of threads to run (default: 10)
|
|
-v, --verify-data Verify data written in a number of places by reading back (default: false)
|
|
-w, --use-shadow-db Use a shaddow database to verify data integrity (default: false)
|
|
-x, --continue-on-exception
|
|
Continue execution after encountering unexpected/disallowed errors/exceptions (default: false)
|
|
```
|
|
|