xxq250
/
homework-jianmu
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

merge: from 3.0 to 3.3.6 branch (#30200) 

* test: enhance unit test script with output logging and error handling

* ci: update TDgpt workflow to combine the unit test and function test

* fix: correct directory name in tdgpt test workflow

* ci: enhance function test workflow with environment setup and logging

* ci: remove push trigger from TDgpt test workflow

* test: update ctest command to exclude 'taoscTest'

* test case

* Update .github/workflows/tdgpt-test.yml

Co-authored-by: WANG Xu <feici02@outlook.com>

* Update .github/workflows/tdgpt-test.yml

Co-authored-by: WANG Xu <feici02@outlook.com>

* Update .github/workflows/tdgpt-test.yml

Co-authored-by: WANG Xu <feici02@outlook.com>

* ci: fix function returns test issue due to docs_changed.txt not found

* test: update ctest command to include 'taoscTest'

* Update .github/workflows/tdgpt-test.yml

Co-authored-by: WANG Xu <feici02@outlook.com>

* refactor: readability

* feat:[TS-4897] virtual table (#30098)

* feat: [TS-4897] Support create/drop/alter/show/describe vtable

* feat: [TS-4897] Support vtable's query

* feat: [TS-4897] Support create virtual supertable

* feat: [TS-4897] Support explain analyze / where / count(*) and only select ts of vtable.

* feat: [TS-4897] Add create test and fix bugs

* feat: [TS-4897] Add alter/drop test and fix bugs

* feat: [TS-4897] Add describe/show test and fix bugs

* feat: [TS-4897] Add auth test and fix bugs

* feat: [TS-4897] Fix meta/catalog/cache bugs

* feat: [TS-4897] Support select tag from virtual child table

* feat: [TS-4897] Add select test and fix plenty of bugs

* feat: [TS-4897] Add optimize rule for vtable scan / support create vtable cross database / remove enterprise constraint / fix bugs.

* feat: [TS-4897] Fix 'schema is old'

* feat: [TS-4897] Support virtual stable query

* feat: [TS-4897] Add tests and Fix bugs

* feat: [TS-4897] resolve conflict.

* ci: update non_docs_changed.txt to docs_changed.txt to fix tdgpt workflow issue

* fix: compile error in the release build (#30199)

* fix: compile error in the release build

* fix: compile error in the release build

* feat(stream): support stream processing for virtual tables (#30144)

* enh: add client processing

* enh: add mnode vtables processing

* enh: add mnode vtable processing

* enh: add normal child vtable support

* fix: compile issues

* fix: compile issues

* fix: create stream issues

* fix: multi stream scan issue

* fix: remove debug info

* fix: agg task and task level issues

* fix: correct task output type

* fix: split vtablescan from agg

* fix: memory leak issues

* fix: add limitations

* Update 09-error-code.md

* Update 09-error-code.md

* fix: remove usless case

* feat(stream): extract original table data in source scan task

Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.

* feat(stream): multi-way merge using loser tree in virtual merge task

Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing.  Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.

* fix(test): remove useless cases

---------

Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>

* fix(stream): support packaging enterprise edition. (#30189)

---------

Co-authored-by: chenhaoran <haoran920c@163.com>
Co-authored-by: Feng Chao <flyingangel_2008@163.com>
Co-authored-by: facetosea <285808407@qq.com>
Co-authored-by: WANG Xu <feici02@outlook.com>
Co-authored-by: Jing Sima <simondominic9997@outlook.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
Co-authored-by: Haojun Liao <hjxilinx@users.noreply.github.com>
This commit is contained in:
Simon Guan 2025-03-17 11:25:11 +08:00 committed by GitHub
parent d7008894a4 38f84fa1f1
commit 570044adda
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
242 changed files with 322202 additions and 1988 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

51
.github/workflows/tdgpt-ci.yml vendored
View File

@ -1,51 +0,0 @@
name: TDgpt CI
on:
pull_request:
branches:
- '3.0'
paths:
- 'tools/tdgpt/**'
jobs:
build:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.10"]
defaults:
run:
working-directory: ${{ github.workspace }}/tools/tdgpt
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest pylint
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Checking the code with pylint
run: |
pylint $(git ls-files '*.py') --exit-zero
- name: Checking the code with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Run test cases with pytest
run: |
pytest

200
.github/workflows/tdgpt-test.yml vendored Normal file
View File

@ -0,0 +1,200 @@
name: TDgpt Test
on:
pull_request:
branches:
- 'main'
- '3.0'
- '3.3.6'
paths:
- 'tools/tdgpt/**'
- 'source/libs/executor/src/forecastoperator.c'
- 'source/libs/executor/src/anomalywindowoperator.c'
- 'include/common/tanalytics.h'
- 'source/common/src/tanalytics.c'
- 'tests/parallel/tdgpt_cases.task'
- 'tests/script/tsim/analytics'
jobs:
unit-test:
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
python-version: ["3.10"]
defaults:
run:
working-directory: ${{ github.workspace }}/tools/tdgpt
steps:
- uses: actions/checkout@v4
- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v3
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
- name: Install dependencies
run: |
python -m pip install --upgrade pip
python -m pip install flake8 pytest pylint
if [ -f requirements.txt ]; then pip install -r requirements.txt; fi
- name: Checking the code with pylint
run: |
pylint $(git ls-files '*.py') --exit-zero
- name: Checking the code with flake8
run: |
# stop the build if there are Python syntax errors or undefined names
flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
# exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
- name: Run test cases with pytest
run: |
pytest
function-test:
runs-on:
group: CI
labels: [self-hosted, Linux, X64, testing]
env:
CONTAINER_NAME: 'taosd-test'
WKDIR: '/var/lib/jenkins/workspace'
WK: '/var/lib/jenkins/workspace/TDinternal'
WKC: '/var/lib/jenkins/workspace/TDinternal/community'
SOURCE_BRANCH: ${{ github.event.pull_request.head.ref }}
TARGET_BRANCH: ${{ github.event.pull_request.base.ref }}
PR_NUMBER: ${{ github.event.pull_request.number }}
steps:
- name: Output the environment information
run: |
echo "::group::Environment Info"
date
hostname
env
echo "Runner: ${{ runner.name }}"
echo "Workspace: ${{ env.WKDIR }}"
git --version
echo "${{ env.WKDIR }}/restore.sh -p PR-${{ env.PR_NUMBER }} -n ${{ github.run_number }} -c ${{ env.CONTAINER_NAME }}"
echo "::endgroup::"
- name: Prepare repositories
run: |
set -euo pipefail
prepare_environment() {
cd "$1"
git reset --hard
git clean -f
git remote prune origin
git fetch
git checkout "$2"
}
prepare_environment "${{ env.WK }}" "${{ env.TARGET_BRANCH }}"
prepare_environment "${{ env.WKC }}" "${{ env.TARGET_BRANCH }}"
- name: Get latest codes and logs
run: |
cd ${{ env.WKC }}
git remote prune origin
git pull >/dev/null
git log -5
echo "`date "+%Y%m%d-%H%M%S"` TDengineTest/${{ env.PR_NUMBER }}:${{ github.run_number }}:${{ env.TARGET_BRANCH }}" >>${{ env.WKDIR }}/jenkins.log
echo "CHANGE_BRANCH:${{ env.SOURCE_BRANCH }}" >>${{ env.WKDIR }}/jenkins.log
echo "community log: `git log -5`" >>${{ env.WKDIR }}/jenkins.log
git fetch origin +refs/pull/${{ env.PR_NUMBER }}/merge
git checkout -qf FETCH_HEAD
git log -5
echo "community log merged: `git log -5`" >>${{ env.WKDIR }}/jenkins.log
cd ${{ env.WK }}
git pull >/dev/null
git log -5
echo "TDinternal log: `git log -5`" >>${{ env.WKDIR }}/jenkins.log
- name: Update submodule
run: |
cd ${{ env.WKC }}
git submodule update --init --recursive
- name: Detect non-doc files changed
run: |
mkdir -p ${{ env.WKDIR }}/tmp/${{ env.PR_NUMBER }}_${{ github.run_number }}
cd ${{ env.WKC }}
changed_files_non_doc=$(git --no-pager diff --name-only \
FETCH_HEAD \
$(git merge-base FETCH_HEAD ${{ env.TARGET_BRANCH }}) | \
grep -v "^docs/en/" | \
grep -v "^docs/zh/" | \
grep -v ".md$" | \
tr '\n' ' ' || : \
)
echo $changed_files_non_doc > \
${{ env.WKDIR }}/tmp/${{ env.PR_NUMBER }}_${{ github.run_number }}/docs_changed.txt
- name: Check assert testing
run: |
cd ${{ env.WKC }}/tests/parallel_test
./run_check_assert_container.sh -d ${{ env.WKDIR }}
- name: Check void function testing
run: |
cd ${{ env.WKC }}/tests/parallel_test
./run_check_void_container.sh -d ${{ env.WKDIR }}
- name: Build docker container
run: |
date
rm -rf ${{ env.WKC }}/debug
cd ${{ env.WKC }}/tests/parallel_test
time ./container_build.sh -w ${{ env.WKDIR }} -e
- name: Get parameters for testing
id: get_param
run: |
log_server_file="/home/log_server.json"
timeout_cmd=""
extra_param=""
if [ -f "$log_server_file" ]; then
log_server_enabled=$(jq '.enabled' "$log_server_file")
timeout_param=$(jq '.timeout' "$log_server_file")
if [ "$timeout_param" != "null" ] && [ "$timeout_param" != "0" ]; then
timeout_cmd="timeout $timeout_param"
fi
if [ "$log_server_enabled" == "1" ]; then
log_server=$(jq '.server' "$log_server_file" | sed 's/\\\"//g')
if [ "$log_server" != "null" ] && [ "$log_server" != "" ]; then
extra_param="-w $log_server"
fi
fi
fi
echo "timeout_cmd=$timeout_cmd" >> $GITHUB_OUTPUT
echo "extra_param=$extra_param" >> $GITHUB_OUTPUT
- name: Run function returns with a null pointer scan testing
run: |
cd ${{ env.WKC }}/tests/parallel_test
./run_scan_container.sh \
-d ${{ env.WKDIR }} \
-b ${{ env.PR_NUMBER }}_${{ github.run_number }} \
-f ${{ env.WKDIR }}/tmp/${{ env.PR_NUMBER }}_${{ github.run_number }}/docs_changed.txt \
${{ steps.get_param.outputs.extra_param }}
- name: Run tdgpt test cases
run: |
cd ${{ env.WKC }}/tests/parallel_test
export DEFAULT_RETRY_TIME=2
date
timeout 600 time ./run.sh -e \
-m /home/m.json \
-t tdgpt_cases.task \
-b "${{ env.PR_NUMBER }}_${{ github.run_number }}" \
-l ${{ env.WKDIR }}/log \
-o 300 ${{ steps.get_param.outputs.extra_param }}

185
docs/en/14-reference/09-error-code.md
View File

@ -111,6 +111,7 @@ This document details the server error codes that may be encountered when using
| 0x8000030C | Invalid query id | Internal error | Report issue |
| 0x8000030E | Invalid connection id | Internal error | Report issue |
| 0x80000315 | User is disabled | User is unavailable | Grant permissions |
| 0x80000318 | Mnode internal error | Internal error | Report issue |
| 0x80000320 | Object already there | Internal error | Report issue |
| 0x80000322 | Invalid table type | Internal error | Report issue |
| 0x80000323 | Object not there | Internal error | Report issue |
@ -372,90 +373,90 @@ This document details the server error codes that may be encountered when using
## parser
| Error Code | Description | Possible Error Scenarios or Reasons | Suggested Actions for Users |
| ---------- | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ |
| 0x80002600 | syntax error near | SQL syntax error | Check and correct the SQL statement |
| 0x80002601 | Incomplete SQL statement | Incomplete SQL statement | Check and correct the SQL statement |
| 0x80002602 | Invalid column name | Illegal or non-existent column name | Check and correct the SQL statement |
| 0x80002603 | Table does not exist | Table does not exist | Check and confirm the existence of the table in the SQL statement |
| 0x80002604 | Column ambiguously defined | Column (alias) redefined | Check and correct the SQL statement |
| 0x80002605 | Invalid value type | Illegal constant value | Check and correct the SQL statement |
| 0x80002608 | There mustn't be aggregation | Aggregation function used in illegal clause | Check and correct the SQL statement |
| 0x80002609 | ORDER BY item must be the number of a SELECT-list expression | Illegal position specified in Order by | Check and correct the SQL statement |
| 0x8000260A | Not a GROUP BY expression | Illegal group by statement | Check and correct the SQL statement |
| 0x8000260B | Not SELECTed expression | Illegal expression | Check and correct the SQL statement |
| 0x8000260C | Not a single-group group function | Illegal use of column and function | Check and correct the SQL statement |
| 0x8000260D | Tags number not matched | Mismatched number of tag columns | Check and correct the SQL statement |
| 0x8000260E | Invalid tag name | Invalid or non-existent tag name | Check and correct the SQL statement |
| 0x80002610 | Value is too long | Value length exceeds limit | Check and correct the SQL statement or API parameters |
| 0x80002611 | Password too short or empty | Password is empty or less than 8 chars | Use a valid password |
| 0x80002612 | Port should be an integer that is less than 65535 and greater than 0 | Illegal port number | Check and correct the port number |
| 0x80002613 | Endpoint should be in the format of 'fqdn:port' | Incorrect address format | Check and correct the address information |
| 0x80002614 | This statement is no longer supported | Feature has been deprecated | Refer to the feature documentation |
| 0x80002615 | Interval too small | Interval value exceeds the allowed minimum | Change the INTERVAL value |
| 0x80002616 | Database not specified | Database not specified | Specify the database for the current operation |
| 0x80002617 | Invalid identifier name | Illegal or invalid length ID | Check the names of related libraries, tables, columns, TAGs, etc. in the statement |
| 0x80002618 | Corresponding supertable not in this db | Supertable does not exist | Check if the corresponding supertable exists in the database |
| 0x80002619 | Invalid database option | Illegal database option value | Check and correct the database option values |
| 0x8000261A | Invalid table option | Illegal table option value | Check and correct the table option values |
| 0x80002624 | GROUP BY and WINDOW-clause can't be used together | Group by and window cannot be used together | Check and correct the SQL statement |
| 0x80002627 | Aggregate functions do not support nesting | Functions do not support nested use | Check and correct the SQL statement |
| 0x80002628 | Only support STATE_WINDOW on integer/bool/varchar column | Unsupported STATE_WINDOW data type | Check and correct the SQL statement |
| 0x80002629 | Not support STATE_WINDOW on tag column | STATE_WINDOW not supported on tag column | Check and correct the SQL statement |
| 0x8000262A | STATE_WINDOW not support for supertable query | STATE_WINDOW not supported for supertable | Check and correct the SQL statement |
| 0x8000262B | SESSION gap should be fixed time window, and greater than 0 | Illegal SESSION window value | Check and correct the SQL statement |
| 0x8000262C | Only support SESSION on primary timestamp column | Illegal SESSION window column | Check and correct the SQL statement |
| 0x8000262D | Interval offset cannot be negative | Illegal INTERVAL offset value | Check and correct the SQL statement |
| 0x8000262E | Cannot use 'year' as offset when interval is 'month' | Illegal INTERVAL offset unit | Check and correct the SQL statement |
| 0x8000262F | Interval offset should be shorter than interval | Illegal INTERVAL offset value | Check and correct the SQL statement |
| 0x80002630 | Does not support sliding when interval is natural month/year | Illegal sliding unit | Check and correct the SQL statement |
| 0x80002631 | sliding value no larger than the interval value | Illegal sliding value | Check and correct the SQL statement |
| 0x80002632 | sliding value can not less than 1%% of interval value | Illegal sliding value | Check and correct the SQL statement |
| 0x80002633 | Only one tag if there is a json tag | Only single JSON tag column supported | Check and correct the SQL statement |
| 0x80002634 | Query block has incorrect number of result columns | Mismatched number of columns | Check and correct the SQL statement |
| 0x80002635 | Incorrect TIMESTAMP value | Illegal primary timestamp column value | Check and correct the SQL statement |
| 0x80002637 | soffset/offset can not be less than 0 | Illegal soffset/offset value | Check and correct the SQL statement |
| 0x80002638 | slimit/soffset only available for PARTITION/GROUP BY query | slimit/soffset only supported for PARTITION BY/GROUP BY statements | Check and correct the SQL statement |
| 0x80002639 | Invalid topic query | Unsupported TOPIC query | |
| 0x8000263A | Cannot drop supertable in batch | Batch deletion of supertables not supported | Check and correct the SQL statement |
| 0x8000263B | Start(end) time of query range required or time range too large | Window count exceeds limit | Check and correct the SQL statement |
| 0x8000263C | Duplicated column names | Duplicate column names | Check and correct the SQL statement |
| 0x8000263D | Tags length exceeds max length | tag value length exceeds maximum supported range | Check and correct the SQL statement |
| 0x8000263E | Row length exceeds max length | Row length check and correct SQL statement | Check and correct the SQL statement |
| 0x8000263F | Illegal number of columns | Incorrect number of columns | Check and correct the SQL statement |
| 0x80002640 | Too many columns | Number of columns exceeds limit | Check and correct the SQL statement |
| 0x80002641 | First column must be timestamp | The first column must be the primary timestamp column | Check and correct the SQL statement |
| 0x80002642 | Invalid binary/nchar column/tag length | Incorrect length for binary/nchar | Check and correct the SQL statement |
| 0x80002643 | Invalid number of tag columns | Incorrect number of tag columns | Check and correct the SQL statement |
| 0x80002644 | Permission denied | Permission error | Check and confirm user permissions |
| 0x80002645 | Invalid stream query | Illegal stream statement | Check and correct the SQL statement |
| 0x80002646 | Invalid _c0 or_rowts expression | Illegal use of _c0 or_rowts | Check and correct the SQL statement |
| 0x80002647 | Invalid timeline function | Function depends on non-existent primary timestamp | Check and correct the SQL statement |
| 0x80002648 | Invalid password | Password does not meet standards | Check and change the password |
| 0x80002649 | Invalid alter table statement | Illegal modify table statement | Check and correct the SQL statement |
| 0x8000264A | Primary timestamp column cannot be dropped | Primary timestamp column cannot be deleted | Check and correct the SQL statement |
| 0x8000264B | Only binary/nchar column length could be modified, and the length can only be increased, not decreased | Illegal column modification | Check and correct the SQL statement |
| 0x8000264C | Invalid tbname pseudocolumn | Illegal use of tbname column | Check and correct the SQL statement |
| 0x8000264D | Invalid function name | Illegal function name | Check and correct the function name |
| 0x8000264E | Comment too long | Comment length exceeds limit | Check and correct the SQL statement |
| 0x8000264F | Function(s) only allowed in SELECT list, cannot mixed with non scalar functions or columns | Illegal mixing of functions | Check and correct the SQL statement |
| 0x80002650 | Window query not supported, since no valid timestamp column included in the result of subquery | Window query depends on non-existent primary timestamp column | Check and correct the SQL statement |
| 0x80002651 | No columns can be dropped | Essential columns cannot be deleted | Check and correct the SQL statement |
| 0x80002652 | Only tag can be json type | Normal columns do not support JSON type | Check and correct the SQL statement |
| 0x80002655 | The DELETE statement must have a definite time window range | Illegal WHERE condition in DELETE statement | Check and correct the SQL statement |
| 0x80002656 | The REDISTRIBUTE VGROUP statement only support 1 to 3 dnodes | Illegal number of DNODEs specified in REDISTRIBUTE VGROUP | Check and correct the SQL statement |
| 0x80002657 | Fill now allowed | Function does not allow FILL feature | Check and correct the SQL statement |
| 0x80002658 | Invalid windows pc | Illegal use of window pseudocolumn | Check and correct the SQL statement |
| 0x80002659 | Window not allowed | Function cannot be used in window | Check and correct the SQL statement |
| 0x8000265A | Stream not allowed | Function cannot be used in stream computation | Check and correct the SQL statement |
| 0x8000265B | Group by not allowd | Function cannot be used in grouping | Check and correct the SQL statement |
| 0x8000265D | Invalid interp clause | Illegal INTERP or related statement | Check and correct the SQL statement |
| 0x8000265E | Not valid function ion window | Illegal window statement | Check and correct the SQL statement |
| 0x8000265F | Only support single table | Function only supported in single table queries | Check and correct the SQL statement |
| 0x80002660 | Invalid sma index | Illegal creation of SMA statement | Check and correct the SQL statement |
| 0x80002661 | Invalid SELECTed expression | Invalid query statement | Check and correct the SQL statement |
| 0x80002662 | Fail to get table info | Failed to retrieve table metadata information | Preserve the scene and logs, report issue on GitHub |
| 0x80002663 | Not unique table/alias | Table name (alias) conflict | Check and correct the SQL statement |
| Error Code | Description | Possible Error Scenarios or Reasons | Suggested Actions for Users |
|------------| ------------------------------------------------------------ |----------------------------------------------------------------------------| ------------------------------------------------------------ |
| 0x80002600 | syntax error near | SQL syntax error | Check and correct the SQL statement |
| 0x80002601 | Incomplete SQL statement | Incomplete SQL statement | Check and correct the SQL statement |
| 0x80002602 | Invalid column name | Illegal or non-existent column name | Check and correct the SQL statement |
| 0x80002603 | Table does not exist | Table does not exist | Check and confirm the existence of the table in the SQL statement |
| 0x80002604 | Column ambiguously defined | Column (alias) redefined | Check and correct the SQL statement |
| 0x80002605 | Invalid value type | Illegal constant value | Check and correct the SQL statement |
| 0x80002608 | There mustn't be aggregation | Aggregation function used in illegal clause | Check and correct the SQL statement |
| 0x80002609 | ORDER BY item must be the number of a SELECT-list expression | Illegal position specified in Order by | Check and correct the SQL statement |
| 0x8000260A | Not a GROUP BY expression | Illegal group by statement | Check and correct the SQL statement |
| 0x8000260B | Not SELECTed expression | Illegal expression | Check and correct the SQL statement |
| 0x8000260C | Not a single-group group function | Illegal use of column and function | Check and correct the SQL statement |
| 0x8000260D | Tags number not matched | Mismatched number of tag columns | Check and correct the SQL statement |
| 0x8000260E | Invalid tag name | Invalid or non-existent tag name | Check and correct the SQL statement |
| 0x80002610 | Value is too long | Value length exceeds limit | Check and correct the SQL statement or API parameters |
| 0x80002611 | Password too short or empty | Password is empty or less than 8 chars | Use a valid password |
| 0x80002612 | Port should be an integer that is less than 65535 and greater than 0 | Illegal port number | Check and correct the port number |
| 0x80002613 | Endpoint should be in the format of 'fqdn:port' | Incorrect address format | Check and correct the address information |
| 0x80002614 | This statement is no longer supported | Feature has been deprecated | Refer to the feature documentation |
| 0x80002615 | Interval too small | Interval value exceeds the allowed minimum | Change the INTERVAL value |
| 0x80002616 | Database not specified | Database not specified | Specify the database for the current operation |
| 0x80002617 | Invalid identifier name | Illegal or invalid length ID | Check the names of related libraries, tables, columns, TAGs, etc. in the statement |
| 0x80002618 | Corresponding supertable not in this db | Supertable does not exist | Check if the corresponding supertable exists in the database |
| 0x80002619 | Invalid database option | Illegal database option value | Check and correct the database option values |
| 0x8000261A | Invalid table option | Illegal table option value | Check and correct the table option values |
| 0x80002624 | GROUP BY and WINDOW-clause can't be used together | Group by and window cannot be used together | Check and correct the SQL statement |
| 0x80002627 | Aggregate functions do not support nesting | Functions do not support nested use | Check and correct the SQL statement |
| 0x80002628 | Only support STATE_WINDOW on integer/bool/varchar column | Unsupported STATE_WINDOW data type | Check and correct the SQL statement |
| 0x80002629 | Not support STATE_WINDOW on tag column | STATE_WINDOW not supported on tag column | Check and correct the SQL statement |
| 0x8000262A | STATE_WINDOW not support for supertable query | STATE_WINDOW not supported for supertable | Check and correct the SQL statement |
| 0x8000262B | SESSION gap should be fixed time window, and greater than 0 | Illegal SESSION window value | Check and correct the SQL statement |
| 0x8000262C | Only support SESSION on primary timestamp column | Illegal SESSION window column | Check and correct the SQL statement |
| 0x8000262D | Interval offset cannot be negative | Illegal INTERVAL offset value | Check and correct the SQL statement |
| 0x8000262E | Cannot use 'year' as offset when interval is 'month' | Illegal INTERVAL offset unit | Check and correct the SQL statement |
| 0x8000262F | Interval offset should be shorter than interval | Illegal INTERVAL offset value | Check and correct the SQL statement |
| 0x80002630 | Does not support sliding when interval is natural month/year | Illegal sliding unit | Check and correct the SQL statement |
| 0x80002631 | sliding value no larger than the interval value | Illegal sliding value | Check and correct the SQL statement |
| 0x80002632 | sliding value can not less than 1%% of interval value | Illegal sliding value | Check and correct the SQL statement |
| 0x80002633 | Only one tag if there is a json tag | Only single JSON tag column supported | Check and correct the SQL statement |
| 0x80002634 | Query block has incorrect number of result columns | Mismatched number of columns | Check and correct the SQL statement |
| 0x80002635 | Incorrect TIMESTAMP value | Illegal primary timestamp column value | Check and correct the SQL statement |
| 0x80002637 | soffset/offset can not be less than 0 | Illegal soffset/offset value | Check and correct the SQL statement |
| 0x80002638 | slimit/soffset only available for PARTITION/GROUP BY query | slimit/soffset only supported for PARTITION BY/GROUP BY statements | Check and correct the SQL statement |
| 0x80002639 | Invalid topic query | Unsupported TOPIC query | |
| 0x8000263A | Cannot drop supertable in batch | Batch deletion of supertables not supported | Check and correct the SQL statement |
| 0x8000263B | Start(end) time of query range required or time range too large | Window count exceeds limit | Check and correct the SQL statement |
| 0x8000263C | Duplicated column names | Duplicate column names | Check and correct the SQL statement |
| 0x8000263D | Tags length exceeds max length | tag value length exceeds maximum supported range | Check and correct the SQL statement |
| 0x8000263E | Row length exceeds max length | Row length check and correct SQL statement | Check and correct the SQL statement |
| 0x8000263F | Illegal number of columns | Incorrect number of columns | Check and correct the SQL statement |
| 0x80002640 | Too many columns | Number of columns exceeds limit | Check and correct the SQL statement |
| 0x80002641 | First column must be timestamp | The first column must be the primary timestamp column | Check and correct the SQL statement |
| 0x80002642 | Invalid binary/nchar column/tag length | Incorrect length for binary/nchar | Check and correct the SQL statement |
| 0x80002643 | Invalid number of tag columns | Incorrect number of tag columns | Check and correct the SQL statement |
| 0x80002644 | Permission denied | Permission error | Check and confirm user permissions |
| 0x80002645 | Invalid stream query | Illegal stream statement | Check and correct the SQL statement |
| 0x80002646 | Invalid _c0 or_rowts expression | Illegal use of _c0 or_rowts | Check and correct the SQL statement |
| 0x80002647 | Invalid timeline function | Function depends on non-existent primary timestamp | Check and correct the SQL statement |
| 0x80002648 | Invalid password | Password does not meet standards | Check and change the password |
| 0x80002649 | Invalid alter table statement | Illegal modify table statement | Check and correct the SQL statement |
| 0x8000264A | Primary timestamp column cannot be dropped | Primary timestamp column cannot be deleted | Check and correct the SQL statement |
| 0x8000264B | Only binary/nchar column length could be modified, and the length can only be increased, not decreased | Illegal column modification | Check and correct the SQL statement |
| 0x8000264C | Invalid tbname pseudocolumn | Illegal use of tbname column | Check and correct the SQL statement |
| 0x8000264D | Invalid function name | Illegal function name | Check and correct the function name |
| 0x8000264E | Comment too long | Comment length exceeds limit | Check and correct the SQL statement |
| 0x8000264F | Function(s) only allowed in SELECT list, cannot mixed with non scalar functions or columns | Illegal mixing of functions | Check and correct the SQL statement |
| 0x80002650 | Window query not supported, since no valid timestamp column included in the result of subquery | Window query depends on non-existent primary timestamp column | Check and correct the SQL statement |
| 0x80002651 | No columns can be dropped | Essential columns cannot be deleted | Check and correct the SQL statement |
| 0x80002652 | Only tag can be json type | Normal columns do not support JSON type | Check and correct the SQL statement |
| 0x80002655 | The DELETE statement must have a definite time window range | Illegal WHERE condition in DELETE statement | Check and correct the SQL statement |
| 0x80002656 | The REDISTRIBUTE VGROUP statement only support 1 to 3 dnodes | Illegal number of DNODEs specified in REDISTRIBUTE VGROUP | Check and correct the SQL statement |
| 0x80002657 | Fill now allowed | Function does not allow FILL feature | Check and correct the SQL statement |
| 0x80002658 | Invalid windows pc | Illegal use of window pseudocolumn | Check and correct the SQL statement |
| 0x80002659 | Window not allowed | Function cannot be used in window | Check and correct the SQL statement |
| 0x8000265A | Stream not allowed | Function cannot be used in stream computation | Check and correct the SQL statement |
| 0x8000265B | Group by not allowd | Function cannot be used in grouping | Check and correct the SQL statement |
| 0x8000265D | Invalid interp clause | Illegal INTERP or related statement | Check and correct the SQL statement |
| 0x8000265E | Not valid function ion window | Illegal window statement | Check and correct the SQL statement |
| 0x8000265F | Only support single table | Function only supported in single table queries | Check and correct the SQL statement |
| 0x80002660 | Invalid sma index | Illegal creation of SMA statement | Check and correct the SQL statement |
| 0x80002661 | Invalid SELECTed expression | Invalid query statement | Check and correct the SQL statement |
| 0x80002662 | Fail to get table info | Failed to retrieve table metadata information | Preserve the scene and logs, report issue on GitHub |
| 0x80002663 | Not unique table/alias | Table name (alias) conflict | Check and correct the SQL statement |
| 0x80002664 | Join requires valid time-series input | Unsupported JOIN query without primary timestamp column output in subquery | Check and correct the SQL statement |
| 0x80002665 | The _TAGS pseudocolumn can only be used for subtable and supertable queries | Illegal tag column query | Check and correct the SQL statement |
| 0x80002666 | Subquery does not output primary timestamp column | Check and correct the SQL statement | |
@ -466,10 +467,17 @@ This document details the server error codes that may be encountered when using
| 0x8000268A | Cols function's first param must be a select function that output a single row | The first parameter of the cols function should be a selection function | Check and correct the SQL statement |
| 0x8000268B | Invalid using alias for cols function | Illegal cols function alias | Check and correct the SQL statement |
| 0x8000268C | Join primary key col must be timestmap type | Join primary key data type error | Check and correct the SQL statement |
| 0x8000268D | Invalid virtual table's ref column | Create/Update Virtual table using incorrect data source column | Check and correct the SQL statement |
| 0x8000268E | Invalid table type | Incorrect Table type | Check and correct the SQL statement |
| 0x8000268F | Invalid ref column type | Virtual table's column type and data source column's type are different | Check and correct the SQL statement |
| 0x80002690 | Create child table using virtual super table | Create non-virtual child table using virtual super table | Check and correct the SQL statement |
| 0x800026FF | Parser internal error | Internal error in parser | Preserve the scene and logs, report issue on GitHub |
| 0x80002700 | Planner internal error | Internal error in planner | Preserve the scene and logs, report issue on GitHub |
| 0x80002701 | Expect ts equal | JOIN condition validation failed | Preserve the scene and logs, report issue on GitHub |
| 0x80002702 | Cross join not support | CROSS JOIN not supported | Check and correct the SQL statement |
| 0x80002704 | Planner slot key not found | Planner cannot find slotId during making physic plan | Preserve the scene and logs, report issue on GitHub |
| 0x80002705 | Planner invalid table type | Planner get invalid table type | Preserve the scene and logs, report issue on GitHub |
| 0x80002706 | Planner invalid query control plan type | Planner get invalid query control plan type during making physic plan | Preserve the scene and logs, report issue on GitHub |
## function
@ -547,3 +555,12 @@ This document details the server error codes that may be encountered when using
| 0x80004017 | Invalid status, please subscribe topic first | tmq status invalidate | Without calling subscribe, directly poll data |
| 0x80004100 | Stream task not exist | The stream computing task does not exist | Check the server-side error logs |
## virtual table
| Error Code | Description | Possible Error Scenarios or Reasons | Recommended Actions for Users |
|-------------|---------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------|
| 0x80006200 | Virtual table scan internal error | virtual table scan operator internal error, generally does not occur | Check error logs, contact development for handling |
| 0x80006201 | Virtual table scan invalid downstream operator type | The incorrect execution plan generated causes the downstream operator type of the virtual table scan operator to be incorrect. | Check error logs, contact development for handling |
| 0x80006202 | Virtual table prim timestamp column should not has ref | The timestamp primary key column of a virtual table should not have a data source. If it does, this error will occur during subsequent queries on the virtual table. | Check error logs, contact development for handling |
| 0x80006203 | Create virtual child table must use virtual super table | Create virtual child table using non-virtual super table | create virtual child table using virtual super table |

121
docs/zh/14-reference/09-error-code.md
View File

@ -117,6 +117,7 @@ description: TDengine 服务端的错误码列表和详细说明
| 0x8000030C | Invalid query id | 内部错误 | 上报 issue |
| 0x8000030E | Invalid connection id | 内部错误 | 上报 issue |
| 0x80000315 | User is disabled | 该用户不可用 | 赋权 |
| 0x80000318 | Mnode internal error | 内部错误 | 上报 issue |
| 0x80000320 | Object already there | 内部错误 | 上报 issue |
| 0x80000322 | Invalid table type | 内部错误 | 上报 issue |
| 0x80000323 | Object not there | 内部错误 | 上报 issue |
@ -389,63 +390,63 @@ description: TDengine 服务端的错误码列表和详细说明
## parser
| 错误码 | 错误描述 | 可能的出错场景或者可能的原因 | 建议用户采取的措施 |
| ---------- | ------------------------------------------------------------------------------------------------------ | --------------------------------------------- | ------------------------------------- |
| 0x80002600 | syntax error near | SQL 语法错误 | 检查并修正 SQL 语句 |
| 0x80002601 | Incomplete SQL statement | 不完整的 SQL 语句 | 检查并修正 SQL 语句 |
| 0x80002602 | Invalid column name | 不合法或不存在的列名 | 检查并修正 SQL 语句 |
| 0x80002603 | Table does not exist | 表不存在 | 检查并确认SQL语句中的表是否存在 |
| 0x80002604 | Column ambiguously defined | 列名（别名）重复定义 | 检查并修正 SQL 语句 |
| 0x80002605 | Invalid value type | 常量值非法 | 检查并修正 SQL 语句 |
| 0x80002608 | There mustn't be aggregation | 聚合函数出现在非法子句中 | 检查并修正 SQL 语句 |
| 0x80002609 | ORDER BY item must be the number of a SELECT-list expression | Order by 指定的位置不合法 | 检查并修正 SQL 语句 |
| 错误码 | 错误描述 | 可能的出错场景或者可能的原因 | 建议用户采取的措施 |
|------------| ------------------------------------------------------------------------------------------------------ |---------------------------------------------| ------------------------------------- |
| 0x80002600 | syntax error near | SQL 语法错误 | 检查并修正 SQL 语句 |
| 0x80002601 | Incomplete SQL statement | 不完整的 SQL 语句 | 检查并修正 SQL 语句 |
| 0x80002602 | Invalid column name | 不合法或不存在的列名 | 检查并修正 SQL 语句 |
| 0x80002603 | Table does not exist | 表不存在 | 检查并确认SQL语句中的表是否存在 |
| 0x80002604 | Column ambiguously defined | 列名（别名）重复定义 | 检查并修正 SQL 语句 |
| 0x80002605 | Invalid value type | 常量值非法 | 检查并修正 SQL 语句 |
| 0x80002608 | There mustn't be aggregation | 聚合函数出现在非法子句中 | 检查并修正 SQL 语句 |
| 0x80002609 | ORDER BY item must be the number of a SELECT-list expression | Order by 指定的位置不合法 | 检查并修正 SQL 语句 |
| 0x8000260A | Not a GROUP BY expression | 非法 group by 语句 | 检查并修正 SQL 语句 |
| 0x8000260B | Not SELECTed expression | 非法表达式 | 检查并修正 SQL 语句 |
| 0x8000260C | Not a single-group group function | 非法使用列与函数 | 检查并修正 SQL 语句 |
| 0x8000260D | Tags number not matched | tag 列个数不匹配 | 检查并修正 SQL 语句 |
| 0x8000260E | Invalid tag name | 无效或不存在的 tag 名 | 检查并修正 SQL 语句 |
| 0x80002610 | Value is too long | 值长度超出限制 | 检查并修正 SQL 语句或 API 参数 |
| 0x80002611 | Password too short or empty | 密码为空或少于 8 个字符 | 使用合法的密码 |
| 0x80002612 | Port should be an integer that is less than 65535 and greater than 0 | 端口号非法 | 检查并修正端口号 |
| 0x80002613 | Endpoint should be in the format of 'fqdn:port' | 地址格式错误 | 检查并修正地址信息 |
| 0x80002614 | This statement is no longer supported | 功能已经废弃 | 参考功能文档说明 |
| 0x80002615 | Interval too small | interval 值超过允许的最小值 | 更改 INTERVAL 值 |
| 0x80002616 | Database not specified | 未指定数据库 | 指定当前操作的数据库 |
| 0x80002617 | Invalid identifier name | ID 非法或长度不合法 | 检查语句中相关的库、表、列、TAG 等名称 |
| 0x80002618 | Corresponding super table not in this db | 超级表不存在 | 检查库中是否存在对应的超级表 |
| 0x80002619 | Invalid database option | 数据库选项值非法 | 检查并修正数据库选项值 |
| 0x8000261A | Invalid table option | 表选项值非法 | 检查并修正数据表选项值 |
| 0x80002624 | GROUP BY and WINDOW-clause can't be used together | Group by 和窗口不能同时使用 | 检查并修正 SQL 语句 |
| 0x80002627 | Aggregate functions do not support nesting | 函数不支持嵌套使用 | 检查并修正 SQL 语句 |
| 0x80002628 | Only support STATE_WINDOW on integer/bool/varchar column | 不支持的 STATE_WINDOW 数据类型 | 检查并修正 SQL 语句 |
| 0x8000260B | Not SELECTed expression | 非法表达式 | 检查并修正 SQL 语句 |
| 0x8000260C | Not a single-group group function | 非法使用列与函数 | 检查并修正 SQL 语句 |
| 0x8000260D | Tags number not matched | tag 列个数不匹配 | 检查并修正 SQL 语句 |
| 0x8000260E | Invalid tag name | 无效或不存在的 tag 名 | 检查并修正 SQL 语句 |
| 0x80002610 | Value is too long | 值长度超出限制 | 检查并修正 SQL 语句或 API 参数 |
| 0x80002611 | Password too short or empty | 密码为空或少于 8 个字符 | 使用合法的密码 |
| 0x80002612 | Port should be an integer that is less than 65535 and greater than 0 | 端口号非法 | 检查并修正端口号 |
| 0x80002613 | Endpoint should be in the format of 'fqdn:port' | 地址格式错误 | 检查并修正地址信息 |
| 0x80002614 | This statement is no longer supported | 功能已经废弃 | 参考功能文档说明 |
| 0x80002615 | Interval too small | interval 值超过允许的最小值 | 更改 INTERVAL 值 |
| 0x80002616 | Database not specified | 未指定数据库 | 指定当前操作的数据库 |
| 0x80002617 | Invalid identifier name | ID 非法或长度不合法 | 检查语句中相关的库、表、列、TAG 等名称 |
| 0x80002618 | Corresponding super table not in this db | 超级表不存在 | 检查库中是否存在对应的超级表 |
| 0x80002619 | Invalid database option | 数据库选项值非法 | 检查并修正数据库选项值 |
| 0x8000261A | Invalid table option | 表选项值非法 | 检查并修正数据表选项值 |
| 0x80002624 | GROUP BY and WINDOW-clause can't be used together | Group by 和窗口不能同时使用 | 检查并修正 SQL 语句 |
| 0x80002627 | Aggregate functions do not support nesting | 函数不支持嵌套使用 | 检查并修正 SQL 语句 |
| 0x80002628 | Only support STATE_WINDOW on integer/bool/varchar column | 不支持的 STATE_WINDOW 数据类型 | 检查并修正 SQL 语句 |
| 0x80002629 | Not support STATE_WINDOW on tag column | 不支持 TAG 列的 STATE_WINDOW | 检查并修正 SQL 语句 |
| 0x8000262A | STATE_WINDOW not support for super table query | 不支持超级表的 STATE_WINDOW | 检查并修正 SQL 语句 |
| 0x8000262B | SESSION gap should be fixed time window, and greater than 0 | SESSION 窗口值非法 | 检查并修正 SQL 语句 |
| 0x8000262C | Only support SESSION on primary timestamp column | SESSION 窗口列非法 | 检查并修正 SQL 语句 |
| 0x8000262D | Interval offset cannot be negative | INTERVAL offset值非法 | 检查并修正 SQL 语句 |
| 0x8000262E | Cannot use 'year' as offset when interval is 'month' | INTERVAL offset单位非法 | 检查并修正 SQL 语句 |
| 0x8000262F | Interval offset should be shorter than interval | INTERVAL offset值非法 | 检查并修正 SQL 语句 |
| 0x80002630 | Does not support sliding when interval is natural month/year | sliding 单位非法 | 检查并修正 SQL 语句 |
| 0x8000262A | STATE_WINDOW not support for super table query | 不支持超级表的 STATE_WINDOW | 检查并修正 SQL 语句 |
| 0x8000262B | SESSION gap should be fixed time window, and greater than 0 | SESSION 窗口值非法 | 检查并修正 SQL 语句 |
| 0x8000262C | Only support SESSION on primary timestamp column | SESSION 窗口列非法 | 检查并修正 SQL 语句 |
| 0x8000262D | Interval offset cannot be negative | INTERVAL offset值非法 | 检查并修正 SQL 语句 |
| 0x8000262E | Cannot use 'year' as offset when interval is 'month' | INTERVAL offset单位非法 | 检查并修正 SQL 语句 |
| 0x8000262F | Interval offset should be shorter than interval | INTERVAL offset值非法 | 检查并修正 SQL 语句 |
| 0x80002630 | Does not support sliding when interval is natural month/year | sliding 单位非法 | 检查并修正 SQL 语句 |
| 0x80002631 | sliding value no larger than the interval value | sliding 值非法 | 检查并修正 SQL 语句 |
| 0x80002632 | sliding value can not less than 1%% of interval value | sliding 值非法 | 检查并修正 SQL 语句 |
| 0x80002633 | Only one tag if there is a json tag | 只支持单个 JSON TAG 列 | 检查并修正 SQL 语句 |
| 0x80002634 | Query block has incorrect number of result columns | 列个数不匹配 | 检查并修正 SQL 语句 |
| 0x80002635 | Incorrect TIMESTAMP value | 主键时间戳列值非法 | 检查并修正 SQL 语句 |
| 0x80002633 | Only one tag if there is a json tag | 只支持单个 JSON TAG 列 | 检查并修正 SQL 语句 |
| 0x80002634 | Query block has incorrect number of result columns | 列个数不匹配 | 检查并修正 SQL 语句 |
| 0x80002635 | Incorrect TIMESTAMP value | 主键时间戳列值非法 | 检查并修正 SQL 语句 |
| 0x80002637 | soffset/offset can not be less than 0 | soffset/offset 值非法 | 检查并修正 SQL 语句 |
| 0x80002638 | slimit/soffset only available for PARTITION/GROUP BY query | slimit/soffset 只支持 PARTITION BY/GROUP BY 语句 | 检查并修正 SQL 语句 |
| 0x80002639 | Invalid topic query | 不支持的 TOPIC 查询语法 |
| 0x8000263A | Cannot drop super table in batch | 不支持批量删除超级表 | 检查并修正 SQL 语句 |
| 0x8000263B | Start(end) time of query range required or time range too large | 窗口个数超出限制 | 检查并修正 SQL 语句 |
| 0x8000263C | Duplicated column names | 列名称重复 | 检查并修正 SQL 语句 |
| 0x8000263D | Tags length exceeds max length | TAG 值长度超出最大支持范围 | 检查并修正 SQL 语句 |
| 0x8000263E | Row length exceeds max length | 行长度检查并修正 SQL 语句 | 检查并修正 SQL 语句 |
| 0x8000263F | Illegal number of columns | 列个数错误 | 检查并修正 SQL 语句 |
| 0x80002640 | Too many columns | 列个数超出上限 | 检查并修正 SQL 语句 |
| 0x80002641 | First column must be timestamp | 第一列必须是主键时间戳列 | 检查并修正 SQL 语句 |
| 0x80002642 | Invalid binary/nchar column/tag length | binary/nchar 长度错误 | 检查并修正 SQL 语句 |
| 0x80002643 | Invalid number of tag columns | TAG 列个数错误 | 检查并修正 SQL 语句 |
| 0x80002644 | Permission denied | 权限错误 | 检查确认用户是否有相应操作权限 |
| 0x80002645 | Invalid stream query | 非法流语句 | 检查并修正 SQL 语句 |
| 0x80002639 | Invalid topic query | 不支持的 TOPIC 查询语法 |
| 0x8000263A | Cannot drop super table in batch | 不支持批量删除超级表 | 检查并修正 SQL 语句 |
| 0x8000263B | Start(end) time of query range required or time range too large | 窗口个数超出限制 | 检查并修正 SQL 语句 |
| 0x8000263C | Duplicated column names | 列名称重复 | 检查并修正 SQL 语句 |
| 0x8000263D | Tags length exceeds max length | TAG 值长度超出最大支持范围 | 检查并修正 SQL 语句 |
| 0x8000263E | Row length exceeds max length | 行长度检查并修正 SQL 语句 | 检查并修正 SQL 语句 |
| 0x8000263F | Illegal number of columns | 列个数错误 | 检查并修正 SQL 语句 |
| 0x80002640 | Too many columns | 列个数超出上限 | 检查并修正 SQL 语句 |
| 0x80002641 | First column must be timestamp | 第一列必须是主键时间戳列 | 检查并修正 SQL 语句 |
| 0x80002642 | Invalid binary/nchar column/tag length | binary/nchar 长度错误 | 检查并修正 SQL 语句 |
| 0x80002643 | Invalid number of tag columns | TAG 列个数错误 | 检查并修正 SQL 语句 |
| 0x80002644 | Permission denied | 权限错误 | 检查确认用户是否有相应操作权限 |
| 0x80002645 | Invalid stream query | 非法流语句 | 检查并修正 SQL 语句 |
| 0x80002646 | Invalid _c0 or _rowts expression | _c0 或 _rowts 非法使用 | 检查并修正 SQL 语句 |
| 0x80002647 | Invalid timeline function | 函数依赖的主键时间戳不存在 | 检查并修正 SQL 语句 |
| 0x80002648 | Invalid password | 密码不符合规范 | 检查并修改密码 |
@ -483,11 +484,17 @@ description: TDengine 服务端的错误码列表和详细说明
| 0x8000268A | Cols function's first param must be a select function that output a single row | cols 函数第一个参数应该为选择函数 | 检查并修正 SQL 语句 |
| 0x8000268B | Invalid using alias for cols function | cols 函数输出列重命名错误 | 检查并修正 SQL 语句 |
| 0x8000268C | Join primary key col must be timestmap type | 关联查询主键列等值条件类型错误 | 检查并修正 SQL 语句 |
| 0x8000268D | Invalid virtual table's ref column | 创建/更新虚拟表时数据源列不正确 | 检查并修正SQL语句 |
| 0x8000268E | Invalid table type | 表类型不正确 | 检查并修正SQL语句 |
| 0x8000268F | Invalid ref column type | 虚拟表列的数据类型与数据源的数据类型不同 | 检查并修正SQL语句 |
| 0x80002690 | Create child table using virtual super table | 创建非虚拟子表 USING 了虚拟超级表 | 检查并修正SQL语句 |
| 0x800026FF | Parser internal error | 解析器内部错误 | 保留现场和日志，github上报issue |
| 0x80002700 | Planner internal error | 计划期内部错误 | 保留现场和日志，github上报issue |
| 0x80002701 | Expect ts equal | JOIN 条件校验失败 | 保留现场和日志，github上报issue |
| 0x80002702 | Cross join not support | 不支持 CROSS JOIN | 检查并修正 SQL 语句 |
| 0x80002704 | Planner slot key not found | 生成物理计划时查找不到 slotId | 保留现场和日志，github上报issue |
| 0x80002705 | Planner invalid table type | 计划器生成计划时得到了错误的表类型 | 保留现场和日志，github上报issue |
| 0x80002706 | Planner invalid query control plan type | 计划器生成 dynamic query control 计划时得到的类型不正确 | 保留现场和日志，github上报issue |
## function
@ -567,3 +574,13 @@ description: TDengine 服务端的错误码列表和详细说明
| 0x80004017 | Invalid status, please subscribe topic first | 数据订阅状态不对 | 没有调用 subscribe，直接 poll 数据 |
| 0x80004100 | Stream task not exist | 流计算任务不存在 | 具体查看 server 端的错误日志 |
## virtual table
| 错误码 | 错误描述 | 可能的出错场景或者可能的原因 | 建议用户采取的措施 |
|------------|---------------------------------------------------------|------------------------------------------------|------------------------|
| 0x80006200 | Virtual table scan 算子内部错误 | virtual table scan 算子内部逻辑错误，一般不会出现 | 具体查看client端的错误日志提示 |
| 0x80006201 | Virtual table scan invalid downstream operator type | 由于生成的执行计划不对，导致 virtual table scan 算子的下游算子类型不正确 | 保留 explain 执行计划，联系开发处理 |
| 0x80006202 | Virtual table prim timestamp column should not has ref | 虚拟表的时间戳主键列不应该有数据源，如果有，后续查询虚拟表的时候就会出现该错误 | 检查错误日志，联系开发处理 |
| 0x80006203 | Create virtual child table must use virtual super table | 虚拟子表必须建在虚拟超级表下，否则就会出现该错误 | 创建虚拟子表的时候，USING 虚拟超级表 |

4
include/common/taosdef.h
View File

@ -39,7 +39,9 @@ typedef enum {
TSDB_SYSTEM_TABLE = 5,
TSDB_TSMA_TABLE = 6, // time-range-wise sma
TSDB_VIEW_TABLE = 7,
TSDB_TABLE_MAX = 8
TSDB_VIRTUAL_NORMAL_TABLE = 8,
TSDB_VIRTUAL_CHILD_TABLE = 9,
TSDB_TABLE_MAX = 10
} ETableType;
typedef enum {

1
include/common/tcol.h
View File

@ -80,6 +80,7 @@ uint8_t columnEncodeVal(const char* encode);
uint16_t columnCompressVal(const char* compress);
bool withExtSchema(uint8_t tableType);
bool hasRefCol(uint8_t tableType);
bool checkColumnEncode(char encode[TSDB_CL_COMPRESS_OPTION_LEN]);
bool checkColumnEncodeOrSetDefault(uint8_t type, char encode[TSDB_CL_COMPRESS_OPTION_LEN]);
bool checkColumnCompress(char compress[TSDB_CL_COMPRESS_OPTION_LEN]);

2
include/common/tdatablock.h
View File

@ -267,6 +267,8 @@ int32_t createDataBlock(SSDataBlock** pResBlock);
void blockDataDestroy(SSDataBlock* pBlock);
void blockDataFreeRes(SSDataBlock* pBlock);
int32_t createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData, SSDataBlock** pResBlock);
int32_t createOneDataBlockWithColArray(const SSDataBlock* pDataBlock, SArray* pColArray, SSDataBlock** pResBlock);
int32_t createOneDataBlockWithTwoBlock(const SSDataBlock* pDataBlock, const SSDataBlock* pOrgBlock, SSDataBlock** pResBlock);
int32_t createSpecialDataBlock(EStreamType type, SSDataBlock** pBlock);
int32_t blockCopyOneRow(const SSDataBlock* pDataBlock, int32_t rowIdx, SSDataBlock** pResBlock);

3
include/common/tglobal.h
View File

@ -302,6 +302,9 @@ extern int8_t tsS3EpNum;
extern int32_t tsStreamNotifyMessageSize;
extern int32_t tsStreamNotifyFrameSize;
extern bool tsCompareAsStrInGreatest;
extern int32_t tsStreamVirtualMergeMaxDelayMs;
extern int32_t tsStreamVirtualMergeMaxMemKb;
extern int32_t tsStreamVirtualMergeWaitMode;
extern char tsAdapterFqdn[];
extern uint16_t tsAdapterPort;

178
include/common/tmsg.h
View File

@ -182,6 +182,9 @@ typedef enum _mgmt_table {
#define TSDB_ALTER_TABLE_UPDATE_COLUMN_COMPRESS 13
#define TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COMPRESS_OPTION 14
#define TSDB_ALTER_TABLE_UPDATE_MULTI_TAG_VAL 15
#define TSDB_ALTER_TABLE_ALTER_COLUMN_REF 16
#define TSDB_ALTER_TABLE_REMOVE_COLUMN_REF 17
#define TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COLUMN_REF 18
#define TSDB_FILL_NONE 0
#define TSDB_FILL_NULL 1
@ -222,9 +225,11 @@ typedef enum _mgmt_table {
#define TSDB_COL_IS_UD_COL(f) ((f & (~(TSDB_COL_NULL))) == TSDB_COL_UDC)
#define TSDB_COL_REQ_NULL(f) (((f)&TSDB_COL_NULL) != 0)
#define TD_SUPER_TABLE TSDB_SUPER_TABLE
#define TD_CHILD_TABLE TSDB_CHILD_TABLE
#define TD_NORMAL_TABLE TSDB_NORMAL_TABLE
#define TD_SUPER_TABLE TSDB_SUPER_TABLE
#define TD_CHILD_TABLE TSDB_CHILD_TABLE
#define TD_NORMAL_TABLE TSDB_NORMAL_TABLE
#define TD_VIRTUAL_NORMAL_TABLE TSDB_VIRTUAL_NORMAL_TABLE
#define TD_VIRTUAL_CHILD_TABLE TSDB_VIRTUAL_CHILD_TABLE
typedef enum ENodeType {
// Syntax nodes are used in parser and planner module, and some are also used in executor module, such as COLUMN,
@ -270,6 +275,7 @@ typedef enum ENodeType {
QUERY_NODE_ANOMALY_WINDOW,
QUERY_NODE_RANGE_AROUND,
QUERY_NODE_STREAM_NOTIFY_OPTIONS,
QUERY_NODE_VIRTUAL_TABLE,
// Statement nodes are used in parser and planner module.
QUERY_NODE_SET_OPERATOR = 100,
@ -327,6 +333,13 @@ typedef enum ENodeType {
QUERY_NODE_REVOKE_STMT,
QUERY_NODE_ALTER_CLUSTER_STMT,
QUERY_NODE_S3MIGRATE_DATABASE_STMT,
QUERY_NODE_CREATE_TSMA_STMT,
QUERY_NODE_DROP_TSMA_STMT,
QUERY_NODE_CREATE_VIRTUAL_TABLE_STMT,
QUERY_NODE_CREATE_VIRTUAL_SUBTABLE_STMT,
QUERY_NODE_DROP_VIRTUAL_TABLE_STMT,
QUERY_NODE_ALTER_VIRTUAL_TABLE_STMT,
// placeholder for [154, 180]
QUERY_NODE_SHOW_CREATE_VIEW_STMT = 181,
QUERY_NODE_SHOW_CREATE_DATABASE_STMT,
@ -360,6 +373,8 @@ typedef enum ENodeType {
QUERY_NODE_DROP_ANODE_STMT,
QUERY_NODE_UPDATE_ANODE_STMT,
QUERY_NODE_ASSIGN_LEADER_STMT,
QUERY_NODE_SHOW_CREATE_TSMA_STMT,
QUERY_NODE_SHOW_CREATE_VTABLE_STMT,
// show statement nodes
// see 'sysTableShowAdapter', 'SYSTABLE_SHOW_TYPE_OFFSET'
@ -404,11 +419,9 @@ typedef enum ENodeType {
QUERY_NODE_SHOW_ANODES_STMT,
QUERY_NODE_SHOW_ANODES_FULL_STMT,
QUERY_NODE_SHOW_USAGE_STMT,
QUERY_NODE_CREATE_TSMA_STMT,
QUERY_NODE_SHOW_CREATE_TSMA_STMT,
QUERY_NODE_DROP_TSMA_STMT,
QUERY_NODE_SHOW_FILESETS_STMT,
QUERY_NODE_SHOW_TRANSACTION_DETAILS_STMT,
QUERY_NODE_SHOW_VTABLES_STMT,
// logic plan node
QUERY_NODE_LOGIC_PLAN_SCAN = 1000,
@ -429,6 +442,7 @@ typedef enum ENodeType {
QUERY_NODE_LOGIC_PLAN_GROUP_CACHE,
QUERY_NODE_LOGIC_PLAN_DYN_QUERY_CTRL,
QUERY_NODE_LOGIC_PLAN_FORECAST_FUNC,
QUERY_NODE_LOGIC_PLAN_VIRTUAL_TABLE_SCAN,
// physical plan node
QUERY_NODE_PHYSICAL_PLAN_TAG_SCAN = 1100,
@ -493,6 +507,7 @@ typedef enum ENodeType {
QUERY_NODE_PHYSICAL_PLAN_STREAM_CONTINUE_STATE,
QUERY_NODE_PHYSICAL_PLAN_STREAM_CONTINUE_EVENT,
QUERY_NODE_PHYSICAL_PLAN_STREAM_CONTINUE_COUNT,
QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN,
} ENodeType;
typedef struct {
@ -595,6 +610,49 @@ STSRow* tGetSubmitBlkNext(SSubmitBlkIter* pIter);
// for debug
int32_t tPrintFixedSchemaSubmitReq(SSubmitReq* pReq, STSchema* pSchema);
typedef struct {
bool hasRef;
col_id_t id;
char refDbName[TSDB_DB_NAME_LEN];
char refTableName[TSDB_TABLE_NAME_LEN];
char refColName[TSDB_COL_NAME_LEN];
} SColRef;
typedef struct {
int32_t nCols;
int32_t version;
SColRef* pColRef;
} SColRefWrapper;
typedef struct {
int32_t vgId;
SColRef colRef;
} SColRefEx;
typedef struct {
int16_t colId;
char refDbName[TSDB_DB_NAME_LEN];
char refTableName[TSDB_TABLE_NAME_LEN];
char refColName[TSDB_COL_NAME_LEN];
} SRefColInfo;
typedef struct SVCTableRefCols {
uint64_t uid;
int32_t numOfSrcTbls;
int32_t numOfColRefs;
SRefColInfo* refCols;
} SVCTableRefCols;
typedef struct SVCTableMergeInfo {
uint64_t uid;
int32_t numOfSrcTbls;
} SVCTableMergeInfo;
typedef struct {
int32_t nCols;
SColRefEx* pColRefEx;
} SColRefExWrapper;
struct SSchema {
int8_t type;
int8_t flags;
@ -636,6 +694,9 @@ typedef struct {
int8_t sysInfo;
SSchema* pSchemas;
SSchemaExt* pSchemaExt;
int8_t virtualStb;
int32_t numOfColRefs;
SColRef* pColRefs;
} STableMetaRsp;
typedef struct {
@ -734,6 +795,23 @@ typedef struct {
SColCmpr* pColCmpr;
} SColCmprWrapper;
static FORCE_INLINE int32_t tInitDefaultSColRefWrapperByCols(SColRefWrapper* pRef, int32_t nCols) {
if (pRef->pColRef) {
return TSDB_CODE_INVALID_PARA;
}
pRef->pColRef = (SColRef*)taosMemoryCalloc(nCols, sizeof(SColRef));
if (pRef->pColRef == NULL) {
return terrno;
}
pRef->nCols = nCols;
for (int32_t i = 0; i < nCols; i++) {
pRef->pColRef[i].hasRef = false;
pRef->pColRef[i].id = (col_id_t)(i + 1);
}
return 0;
}
static FORCE_INLINE SColCmprWrapper* tCloneSColCmprWrapper(const SColCmprWrapper* pSrcWrapper) {
if (pSrcWrapper->pColCmpr == NULL || pSrcWrapper->nCols == 0) {
terrno = TSDB_CODE_INVALID_PARA;
@ -878,6 +956,29 @@ static FORCE_INLINE int32_t tDecodeSSchemaExt(SDecoder* pDecoder, SSchemaExt* pS
return 0;
}
static FORCE_INLINE int32_t tEncodeSColRef(SEncoder* pEncoder, const SColRef* pColRef) {
TAOS_CHECK_RETURN(tEncodeI8(pEncoder, pColRef->hasRef));
TAOS_CHECK_RETURN(tEncodeI16(pEncoder, pColRef->id));
if (pColRef->hasRef) {
TAOS_CHECK_RETURN(tEncodeCStr(pEncoder, pColRef->refDbName));
TAOS_CHECK_RETURN(tEncodeCStr(pEncoder, pColRef->refTableName));
TAOS_CHECK_RETURN(tEncodeCStr(pEncoder, pColRef->refColName));
}
return 0;
}
static FORCE_INLINE int32_t tDecodeSColRef(SDecoder* pDecoder, SColRef* pColRef) {
TAOS_CHECK_RETURN(tDecodeI8(pDecoder, (int8_t*)&pColRef->hasRef));
TAOS_CHECK_RETURN(tDecodeI16(pDecoder, &pColRef->id));
if (pColRef->hasRef) {
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, pColRef->refDbName));
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, pColRef->refTableName));
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, pColRef->refColName));
}
return 0;
}
static FORCE_INLINE int32_t taosEncodeSSchemaWrapper(void** buf, const SSchemaWrapper* pSW) {
int32_t tlen = 0;
tlen += taosEncodeVariantI32(buf, pSW->nCols);
@ -977,6 +1078,7 @@ typedef struct {
int32_t sqlLen;
char* sql;
int64_t keep;
int8_t virtualStb;
} SMCreateStbReq;
int32_t tSerializeSMCreateStbReq(void* buf, int32_t bufLen, SMCreateStbReq* pReq);
@ -1335,6 +1437,8 @@ typedef struct {
int32_t tagsLen;
char* pTags;
SSchemaExt* pSchemaExt;
int8_t virtualStb;
SColRef* pColRefs;
} STableCfg;
typedef STableCfg STableCfgRsp;
@ -1346,6 +1450,24 @@ int32_t tSerializeSTableCfgRsp(void* buf, int32_t bufLen, STableCfgRsp* pRsp);
int32_t tDeserializeSTableCfgRsp(void* buf, int32_t bufLen, STableCfgRsp* pRsp);
void tFreeSTableCfgRsp(STableCfgRsp* pRsp);
typedef struct {
SMsgHead header;
tb_uid_t suid;
} SVSubTablesReq;
int32_t tSerializeSVSubTablesReq(void *buf, int32_t bufLen, SVSubTablesReq *pReq);
int32_t tDeserializeSVSubTablesReq(void *buf, int32_t bufLen, SVSubTablesReq *pReq);
typedef struct {
int32_t vgId;
SArray* pTables; //SArray<SVCTableRefCols*>
} SVSubTablesRsp;
int32_t tSerializeSVSubTablesRsp(void *buf, int32_t bufLen, SVSubTablesRsp *pRsp);
int32_t tDeserializeSVSubTablesRsp(void *buf, int32_t bufLen, SVSubTablesRsp *pRsp);
void tDestroySVSubTablesRsp(void* rsp);
typedef struct {
char db[TSDB_DB_FNAME_LEN];
int32_t numOfVgroups;
@ -2843,13 +2965,37 @@ typedef struct SOperatorParam {
int32_t downstreamIdx;
void* value;
SArray* pChildren; // SArray<SOperatorParam*>
bool reUse;
} SOperatorParam;
typedef struct SColIdNameKV {
col_id_t colId;
char colName[TSDB_COL_NAME_LEN];
} SColIdNameKV;
typedef struct SColIdPair {
col_id_t vtbColId;
col_id_t orgColId;
} SColIdPair;
typedef struct SOrgTbInfo {
int32_t vgId;
char tbName[TSDB_TABLE_FNAME_LEN];
SArray* colMap; // SArray<SColIdNameKV>
} SOrgTbInfo;
typedef struct STableScanOperatorParam {
bool tableSeq;
SArray* pUidList;
bool tableSeq;
SArray* pUidList;
SOrgTbInfo* pOrgTbInfo;
STimeWindow window;
} STableScanOperatorParam;
typedef struct SVTableScanOperatorParam {
uint64_t uid;
SArray* pOpParamArray; // SArray<SOperatorParam>
} SVTableScanOperatorParam;
typedef struct {
SMsgHead header;
uint64_t sId;
@ -3027,6 +3173,7 @@ typedef struct {
char pWendName[TSDB_COL_NAME_LEN];
char pGroupIdName[TSDB_COL_NAME_LEN];
char pIsWindowFilledName[TSDB_COL_NAME_LEN];
SArray* pVSubTables; // array of SVSubTablesRsp
} SCMCreateStreamReq;
typedef struct STaskNotifyEventStat {
@ -3279,6 +3426,7 @@ typedef struct SVCreateStbReq {
SColCmprWrapper colCmpr;
int64_t keep;
SExtSchema* pExtSchemas;
int8_t virtualStb;
} SVCreateStbReq;
int tEncodeSVCreateStbReq(SEncoder* pCoder, const SVCreateStbReq* pReq);
@ -3320,6 +3468,7 @@ typedef struct SVCreateTbReq {
char* sql;
SColCmprWrapper colCmpr;
SExtSchema* pExtSchemas;
SColRefWrapper colRef; // col reference for virtual table
} SVCreateTbReq;
int tEncodeSVCreateTbReq(SEncoder* pCoder, const SVCreateTbReq* pReq);
@ -3334,16 +3483,17 @@ static FORCE_INLINE void tdDestroySVCreateTbReq(SVCreateTbReq* req) {
taosMemoryFreeClear(req->sql);
taosMemoryFreeClear(req->name);
taosMemoryFreeClear(req->comment);
if (req->type == TSDB_CHILD_TABLE) {
if (req->type == TSDB_CHILD_TABLE || req->type == TSDB_VIRTUAL_CHILD_TABLE) {
taosMemoryFreeClear(req->ctb.pTag);
taosMemoryFreeClear(req->ctb.stbName);
taosArrayDestroy(req->ctb.tagName);
req->ctb.tagName = NULL;
} else if (req->type == TSDB_NORMAL_TABLE) {
} else if (req->type == TSDB_NORMAL_TABLE || req->type == TSDB_VIRTUAL_NORMAL_TABLE) {
taosMemoryFreeClear(req->ntb.schemaRow.pSchema);
}
taosMemoryFreeClear(req->colCmpr.pColCmpr);
taosMemoryFreeClear(req->pExtSchemas);
taosMemoryFreeClear(req->colRef.pColRef);
}
typedef struct {
@ -3391,6 +3541,7 @@ typedef struct {
uint64_t suid; // for tmq in wal format
int64_t uid;
int8_t igNotExists;
int8_t isVirtual;
} SVDropTbReq;
typedef struct {
@ -3430,7 +3581,7 @@ typedef struct SMultiTagUpateVal {
int8_t isNull;
SArray* pTagArray;
} SMultiTagUpateVal;
typedef struct {
typedef struct SVAlterTbReq{
char* tbName;
int8_t action;
char* colName;
@ -3462,6 +3613,11 @@ typedef struct {
SArray* pMultiTag; // TSDB_ALTER_TABLE_ADD_MULTI_TAGS
// for Add column
STypeMod typeMod;
// TSDB_ALTER_TABLE_ALTER_COLUMN_REF
char* refDbName;
char* refTbName;
char* refColName;
// TSDB_ALTER_TABLE_REMOVE_COLUMN_REF
} SVAlterTbReq;
int32_t tEncodeSVAlterTbReq(SEncoder* pEncoder, const SVAlterTbReq* pReq);

3
include/common/tmsgdef.h
View File

@ -132,6 +132,7 @@
TD_DEF_MSG_TYPE(TDMT_MND_RETRIEVE_ANAL_ALGO, "retrieve-anal-algo", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_FAILED_STREAM, "create-stream-failed", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_CHECK_STREAM_TIMER, "check-stream-status", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_GET_DB_INFO, "get-db-info", NULL, NULL)
TD_CLOSE_MSG_SEG(TDMT_DND_MSG)
TD_NEW_MSG_SEG(TDMT_MND_MSG) // 1<<8
@ -265,6 +266,7 @@
TD_DEF_MSG_TYPE(TDMT_MND_CONFIG, "init-config", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_CONFIG_SDB, "config-sdb", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_MND_RESET_STREAM, "reset-stream", NULL, NULL)
// do not add new message type here, since mnode msg overload. you can add new message type after dnode msg
TD_CLOSE_MSG_SEG(TDMT_END_MND_MSG)
TD_NEW_MSG_SEG(TDMT_VND_MSG) // 2<<8
@ -321,6 +323,7 @@
TD_DEF_MSG_TYPE(TDMT_VND_ARB_CHECK_SYNC, "vnode-arb-check-sync", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_FETCH_TTL_EXPIRED_TBS, "vnode-fetch-ttl-expired-tbs", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_TABLE_NAME, "vnode-table-name", NULL, NULL)
TD_DEF_MSG_TYPE(TDMT_VND_VSUBTABLES_META, "vnode-virtual_stables-meta", NULL, NULL)
TD_CLOSE_MSG_SEG(TDMT_VND_MSG)
TD_NEW_MSG_SEG(TDMT_SCH_MSG) // 3<<8

2
include/libs/catalog/catalog.h
View File

@ -98,6 +98,7 @@ typedef struct SCatalogReq {
SArray* pTableTSMAs; // element is STablesReq
SArray* pTSMAs; // element is STablesReq
SArray* pTableName; // element is STablesReq
SArray* pVSubTable; // element is SName
bool qNodeRequired; // valid qnode
bool dNodeRequired; // valid dnode
bool svrVerRequired;
@ -129,6 +130,7 @@ typedef struct SMetaData {
SArray* pView; // pRes = SViewMeta*
SArray* pTableTsmas; // pRes = SArray<STableTSMAInfo*>
SArray* pTsmas; // pRes = SArray<STableTSMAInfo*>
SArray* pVSubTables; // pRes = SVSubTablesRsp
SMetaRes* pSvrVer; // pRes = char*
} SMetaData;

3
include/libs/executor/dataSinkMgt.h
View File

@ -30,6 +30,7 @@ extern "C" {
#define DS_BUF_EMPTY 3
#define DS_FLAG_USE_MEMPOOL (1 << 0)
#define DS_FLAG_PROCESS_ONE_BLOCK (1 << 1)
struct SSDataBlock;
@ -87,7 +88,7 @@ typedef struct SOutputData {
* @param pHandle output
* @return error code
*/
int32_t dsCreateDataSinker(void* pSinkManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle, void* pParam, const char* id);
int32_t dsCreateDataSinker(void* pSinkManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle, void* pParam, const char* id, bool processOneBlock);
int32_t dsDataSinkGetCacheSize(SDataSinkStat* pStat);

2
include/libs/executor/executor.h
View File

@ -168,7 +168,7 @@ int32_t qGetQueryTableSchemaVersion(qTaskInfo_t tinfo, char* dbName, int32_t dbN
* @param handle
* @return
*/
int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bool* hasMore, SLocalFetch* pLocal);
int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bool* hasMore, SLocalFetch* pLocal, bool processOneBlock);
int32_t qExecTask(qTaskInfo_t tinfo, SSDataBlock** pBlock, uint64_t* useconds);

7
include/libs/executor/storageapi.h
View File

@ -85,6 +85,7 @@ typedef struct SMetaEntry {
SColCmprWrapper colCmpr; // col compress alg
SExtSchema* pExtSchemas;
SColRefWrapper colRef; // col reference for virtual table
} SMetaEntry;
typedef struct SMetaReader {
@ -239,8 +240,8 @@ typedef struct SStoreTqReader {
int64_t (*tqGetResultBlockTime)();
int32_t (*tqGetStreamExecProgress)();
void (*tqReaderSetColIdList)();
void (*tqReaderSetQueryTableList)();
int32_t (*tqReaderSetColIdList)();
int32_t (*tqReaderSetQueryTableList)();
void (*tqReaderAddTables)();
void (*tqReaderRemoveTables)();
@ -255,6 +256,8 @@ typedef struct SStoreTqReader {
int32_t (*tqReaderSetSubmitMsg)(); // todo remove it
// bool (*tqReaderNextBlockFilterOut)();
int32_t (*tqReaderSetVtableInfo)();
} SStoreTqReader;
typedef struct SStoreSnapshotFn {

1
include/libs/function/functionMgt.h
View File

@ -279,6 +279,7 @@ bool fmIsForbidSysTableFunc(int32_t funcId);
bool fmIsIntervalInterpoFunc(int32_t funcId);
bool fmIsInterpFunc(int32_t funcId);
bool fmIsLastRowFunc(int32_t funcId);
bool fmIsLastFunc(int32_t funcId);
bool fmIsForecastFunc(int32_t funcId);
bool fmIsNotNullOutputFunc(int32_t funcId);
bool fmIsSelectValueFunc(int32_t funcId);

40
include/libs/nodes/cmdnodes.h
View File

@ -25,10 +25,12 @@ extern "C" {
#define DESCRIBE_RESULT_COLS 4
#define DESCRIBE_RESULT_COLS_COMPRESS 7
#define DESCRIBE_RESULT_COLS_REF 5
#define DESCRIBE_RESULT_FIELD_LEN (TSDB_COL_NAME_LEN - 1 + VARSTR_HEADER_SIZE)
#define DESCRIBE_RESULT_TYPE_LEN (20 + VARSTR_HEADER_SIZE)
#define DESCRIBE_RESULT_NOTE_LEN (16 + VARSTR_HEADER_SIZE)
#define DESCRIBE_RESULT_COPRESS_OPTION_LEN (TSDB_CL_COMPRESS_OPTION_LEN + VARSTR_HEADER_SIZE)
#define DESCRIBE_RESULT_COL_REF_LEN (TSDB_COL_FNAME_LEN + VARSTR_HEADER_SIZE)
#define SHOW_CREATE_DB_RESULT_COLS 2
#define SHOW_CREATE_DB_RESULT_FIELD1_LEN (TSDB_DB_NAME_LEN + VARSTR_HEADER_SIZE)
@ -184,6 +186,7 @@ typedef struct SCompactVgroupsStmt {
typedef struct STableOptions {
ENodeType type;
bool virtualStb;
bool commentNull;
char comment[TSDB_TB_COMMENT_LEN];
SNodeList* pMaxDelay;
@ -210,7 +213,12 @@ typedef struct SColumnOptions {
char compress[TSDB_CL_COMPRESS_OPTION_LEN];
char compressLevel[TSDB_CL_COMPRESS_OPTION_LEN];
bool bPrimaryKey;
bool hasRef;
char refDb[TSDB_DB_NAME_LEN];
char refTable[TSDB_TABLE_NAME_LEN];
char refColumn[TSDB_COL_NAME_LEN];
} SColumnOptions;
typedef struct SColumnDefNode {
ENodeType type;
char colName[TSDB_COL_NAME_LEN];
@ -229,6 +237,27 @@ typedef struct SCreateTableStmt {
STableOptions* pOptions;
} SCreateTableStmt;
typedef struct SCreateVTableStmt {
ENodeType type;
char dbName[TSDB_DB_NAME_LEN];
char tableName[TSDB_TABLE_NAME_LEN];
bool ignoreExists;
SNodeList* pCols;
} SCreateVTableStmt;
typedef struct SCreateVSubTableStmt {
ENodeType type;
char dbName[TSDB_DB_NAME_LEN];
char tableName[TSDB_TABLE_NAME_LEN];
char useDbName[TSDB_DB_NAME_LEN];
char useTableName[TSDB_TABLE_NAME_LEN];
bool ignoreExists;
SNodeList* pSpecificTags;
SNodeList* pValsOfTags;
SNodeList* pSpecificColRefs;
SNodeList* pColRefs;
} SCreateVSubTableStmt;
typedef struct SCreateSubTableClause {
ENodeType type;
char dbName[TSDB_DB_NAME_LEN];
@ -278,6 +307,14 @@ typedef struct SDropSuperTableStmt {
bool withOpt;
} SDropSuperTableStmt;
typedef struct SDropVirtualTableStmt {
ENodeType type;
char dbName[TSDB_DB_NAME_LEN];
char tableName[TSDB_TABLE_NAME_LEN];
bool ignoreNotExists;
bool withOpt;
} SDropVirtualTableStmt;
typedef struct SAlterTableStmt {
ENodeType type;
char dbName[TSDB_DB_NAME_LEN];
@ -290,6 +327,9 @@ typedef struct SAlterTableStmt {
SValueNode* pVal;
SColumnOptions* pColOptions;
SNodeList* pNodeListTagValue;
char refDbName[TSDB_DB_NAME_LEN];
char refTableName[TSDB_TABLE_NAME_LEN];
char refColName[TSDB_COL_NAME_LEN];
} SAlterTableStmt;
typedef struct SAlterTableMultiStmt {

1
include/libs/nodes/nodes.h
View File

@ -176,6 +176,7 @@ void nodesSortList(SNodeList** pList, int32_t (*)(SNode* pNode1, SNode* pNode
void destroyFuncParam(void* pFuncStruct);
int32_t nodesListDeduplicate(SNodeList** pList);
#ifdef __cplusplus
}
#endif

46
include/libs/nodes/plannodes.h
View File

@ -129,6 +129,7 @@ typedef struct SScanLogicNode {
bool smallDataTsSort; // disable row id sort for table merge scan
bool needSplit;
bool noPseudoRefAfterGrp; // no pseudo columns referenced ater group/partition clause
bool virtualStableScan;
} SScanLogicNode;
typedef struct SJoinLogicNode {
@ -170,6 +171,19 @@ typedef struct SJoinLogicNode {
SNode* pRightOnCond; // table onCond filter
} SJoinLogicNode;
typedef struct SVirtualScanLogicNode {
SLogicNode node;
bool scanAllCols;
SNodeList* pScanCols;
SNodeList* pScanPseudoCols;
int8_t tableType;
uint64_t tableId;
uint64_t stableId;
SVgroupsInfo* pVgroupList;
EScanType scanType;
SName tableName;
} SVirtualScanLogicNode;
typedef struct SAggLogicNode {
SLogicNode node;
SNodeList* pGroupKeys;
@ -247,10 +261,17 @@ typedef struct SDynQueryCtrlStbJoin {
bool srcScan[2];
} SDynQueryCtrlStbJoin;
typedef struct SDynQueryCtrlVtbScan {
bool scanAllCols;
uint64_t suid;
SVgroupsInfo* pVgroupList;
} SDynQueryCtrlVtbScan;
typedef struct SDynQueryCtrlLogicNode {
SLogicNode node;
EDynQueryType qType;
SDynQueryCtrlStbJoin stbJoin;
SDynQueryCtrlVtbScan vtbScan;
} SDynQueryCtrlLogicNode;
typedef enum EModifyTableType { MODIFY_TABLE_TYPE_INSERT = 1, MODIFY_TABLE_TYPE_DELETE } EModifyTableType;
@ -413,6 +434,7 @@ typedef struct SLogicSubplan {
int32_t level;
int32_t splitFlag;
int32_t numOfComputeNodes;
bool processOneBlock;
} SLogicSubplan;
typedef struct SQueryLogicPlan {
@ -462,6 +484,7 @@ typedef struct SScanPhysiNode {
int8_t tableType;
SName tableName;
bool groupOrderScan;
bool virtualStableScan;
} SScanPhysiNode;
typedef struct STagScanPhysiNode {
@ -471,6 +494,18 @@ typedef struct STagScanPhysiNode {
typedef SScanPhysiNode SBlockDistScanPhysiNode;
typedef struct SVirtualScanPhysiNode {
SScanPhysiNode scan;
SNodeList* pGroupTags;
bool groupSort;
bool scanAllCols;
SNodeList* pTargets;
SNodeList* pTags;
SNode* pSubtable;
int8_t igExpired;
int8_t igCheckUpdate;
}SVirtualScanPhysiNode;
typedef struct SLastRowScanPhysiNode {
SScanPhysiNode scan;
SNodeList* pGroupTags;
@ -629,11 +664,20 @@ typedef struct SStbJoinDynCtrlBasic {
bool srcScan[2];
} SStbJoinDynCtrlBasic;
typedef struct SVtbScanDynCtrlBasic {
bool scanAllCols;
uint64_t suid;
int32_t accountId;
SEpSet mgmtEpSet;
SNodeList *pScanCols;
} SVtbScanDynCtrlBasic;
typedef struct SDynQueryCtrlPhysiNode {
SPhysiNode node;
EDynQueryType qType;
union {
SStbJoinDynCtrlBasic stbJoin;
SVtbScanDynCtrlBasic vtbScan;
};
} SDynQueryCtrlPhysiNode;
@ -856,11 +900,13 @@ typedef struct SSubplan {
SDataSinkNode* pDataSink; // data of the subplan flow into the datasink
SNode* pTagCond;
SNode* pTagIndexCond;
SSHashObj* pVTables; // for stream virtual tables
bool showRewrite;
bool isView;
bool isAudit;
bool dynamicRowThreshold;
int32_t rowsThreshold;
bool processOneBlock;
} SSubplan;
typedef enum EExplainMode { EXPLAIN_MODE_DISABLE = 1, EXPLAIN_MODE_STATIC, EXPLAIN_MODE_ANALYZE } EExplainMode;

20
include/libs/nodes/querynodes.h
View File

@ -29,7 +29,7 @@ extern "C" {
#define TABLE_TOTAL_COL_NUM(pMeta) ((pMeta)->tableInfo.numOfColumns + (pMeta)->tableInfo.numOfTags)
#define TABLE_META_SIZE(pMeta) \
(NULL == (pMeta) ? 0 : (sizeof(STableMeta) + TABLE_TOTAL_COL_NUM((pMeta)) * sizeof(SSchema)))
(NULL == (pMeta) ? 0 : (sizeof(STableMeta) + TABLE_TOTAL_COL_NUM((pMeta)) * sizeof(SSchema) + (pMeta)->numOfColRefs * sizeof(SColRef)))
#define VGROUPS_INFO_SIZE(pInfo) \
(NULL == (pInfo) ? 0 : (sizeof(SVgroupsInfo) + (pInfo)->numOfVgroups * sizeof(SVgroupInfo)))
@ -94,12 +94,19 @@ typedef struct SColumnNode {
bool isPk;
int32_t projRefIdx;
int32_t resIdx;
bool hasDep;
bool hasRef;
char refDbName[TSDB_DB_NAME_LEN];
char refTableName[TSDB_TABLE_NAME_LEN];
char refColName[TSDB_COL_NAME_LEN];
} SColumnNode;
typedef struct SColumnRefNode {
ENodeType type;
SDataType resType;
char colName[TSDB_COL_NAME_LEN];
char refDbName[TSDB_DB_NAME_LEN];
char refTableName[TSDB_TABLE_NAME_LEN];
char refColName[TSDB_COL_NAME_LEN];
} SColumnRefNode;
typedef struct STargetNode {
@ -238,6 +245,13 @@ typedef struct STempTableNode {
SNode* pSubquery;
} STempTableNode;
typedef struct SVirtualTableNode {
STableNode table; // QUERY_NODE_VIRTUAL_TABLE
struct STableMeta* pMeta;
SVgroupsInfo* pVgroupList;
SNodeList* refTables;
} SVirtualTableNode;
typedef struct SViewNode {
STableNode table; // QUERY_NODE_REAL_TABLE
struct STableMeta* pMeta;
@ -281,6 +295,7 @@ typedef enum EJoinAlgorithm {
typedef enum EDynQueryType {
DYN_QTYPE_STB_HASH = 1,
DYN_QTYPE_VTB_SCAN,
} EDynQueryType;
typedef struct SJoinTableNode {
@ -395,6 +410,7 @@ typedef enum EShowKind {
SHOW_KIND_ALL = 1,
SHOW_KIND_TABLES_NORMAL,
SHOW_KIND_TABLES_CHILD,
SHOW_KIND_TABLES_VIRTUAL,
SHOW_KIND_DATABASES_USER,
SHOW_KIND_DATABASES_SYSTEM
} EShowKind;

27
include/libs/parser/parser.h
View File

@ -24,6 +24,31 @@ extern "C" {
#include "query.h"
#include "querynodes.h"
#define PAR_ERR_RET(c) \
do { \
int32_t _code = c; \
if (_code != TSDB_CODE_SUCCESS) { \
terrno = _code; \
return _code; \
} \
} while (0)
#define PAR_RET(c) \
do { \
int32_t _code = c; \
if (_code != TSDB_CODE_SUCCESS) { \
terrno = _code; \
} \
return _code; \
} while (0)
#define PAR_ERR_JRET(c) \
do { \
code = c; \
if (code != TSDB_CODE_SUCCESS) { \
terrno = code; \
goto _return; \
} \
} while (0)
typedef struct SStmtCallback {
TAOS_STMT* pStmt;
int32_t (*getTbNameFn)(TAOS_STMT*, char**);
@ -113,7 +138,7 @@ bool qParseDbName(const char* pStr, size_t length, char** pDbName);
// for async mode
int32_t qParseSqlSyntax(SParseContext* pCxt, SQuery** pQuery, struct SCatalogReq* pCatalogReq);
int32_t qAnalyseSqlSemantic(SParseContext* pCxt, const struct SCatalogReq* pCatalogReq,
const struct SMetaData* pMetaData, SQuery* pQuery);
struct SMetaData* pMetaData, SQuery* pQuery);
int32_t qContinueParseSql(SParseContext* pCxt, struct SCatalogReq* pCatalogReq, const struct SMetaData* pMetaData,
SQuery* pQuery);
int32_t qContinueParsePostQuery(SParseContext* pCxt, SQuery* pQuery, SSDataBlock* pBlock);

1
include/libs/planner/planner.h
View File

@ -53,6 +53,7 @@ typedef struct SPlanContext {
char pWendName[TSDB_COL_NAME_LEN];
char pGroupIdName[TSDB_COL_NAME_LEN];
char pIsWindowFilledName[TSDB_COL_NAME_LEN];
bool virtualStableQuery;
} SPlanContext;
// Create the physical plan for the query, according to the AST.

85
include/libs/qcom/query.h
View File

@ -105,8 +105,20 @@ typedef struct SCTableMeta {
} SCTableMeta;
#pragma pack(pop)
#pragma pack(push, 1)
typedef struct SVCTableMeta {
uint64_t uid;
uint64_t suid;
int32_t vgId;
int8_t tableType;
int32_t numOfColRefs;
SColRef* colRef;
} SVCTableMeta;
#pragma pack(pop)
#pragma pack(push, 1)
typedef struct STableMeta {
// BEGIN: KEEP THIS PART SAME WITH SVCTableMeta
// BEGIN: KEEP THIS PART SAME WITH SCTableMeta
uint64_t uid;
uint64_t suid;
@ -114,6 +126,10 @@ typedef struct STableMeta {
int8_t tableType;
// END: KEEP THIS PART SAME WITH SCTableMeta
int32_t numOfColRefs;
SColRef* colRef;
// END: KEEP THIS PART SAME WITH SVCTableMeta
// if the table is TSDB_CHILD_TABLE, the following information is acquired from the corresponding super table meta
// info
int32_t sversion;
@ -121,7 +137,8 @@ typedef struct STableMeta {
STableComInfo tableInfo;
SSchemaExt* schemaExt; // There is no additional memory allocation, and the pointer is fixed to the next address of
// the schema content.
SSchema schema[];
int8_t virtualStb;
SSchema schema[];
} STableMeta;
#pragma pack(pop)
@ -147,22 +164,56 @@ typedef struct SDBVgInfo {
SArray* vgArray; // SVgroupInfo
} SDBVgInfo;
typedef struct SVGroupHashInfo {
int32_t vgId;
uint32_t hashBegin;
uint32_t hashEnd;
} SVGroupHashInfo;
typedef struct SDBVgHashInfo {
int16_t hashPrefix;
int16_t hashSuffix;
int8_t hashMethod;
bool vgSorted;
SArray* vgArray; //SArray<SVGroupHashInfo>
} SDBVgHashInfo;
typedef struct SColIdName {
int16_t colId;
char* colName;
} SColIdName;
typedef struct SStreamVBuildCtx {
int64_t lastUid;
SRefColInfo* lastCol;
SSHashObj* lastVg;
SSHashObj* lastVtable;
SArray* lastOtable;
} SStreamVBuildCtx;
typedef struct SUseDbOutput {
char db[TSDB_DB_FNAME_LEN];
uint64_t dbId;
SDBVgInfo* dbVgroup;
} SUseDbOutput;
enum { META_TYPE_NULL_TABLE = 1, META_TYPE_CTABLE, META_TYPE_TABLE, META_TYPE_BOTH_TABLE };
enum { META_TYPE_NULL_TABLE = 1,
META_TYPE_CTABLE,
META_TYPE_VCTABLE,
META_TYPE_TABLE,
META_TYPE_BOTH_TABLE,
META_TYPE_BOTH_VTABLE};
typedef struct STableMetaOutput {
int32_t metaType;
uint64_t dbId;
char dbFName[TSDB_DB_FNAME_LEN];
char ctbName[TSDB_TABLE_NAME_LEN];
char tbName[TSDB_TABLE_NAME_LEN];
SCTableMeta ctbMeta;
STableMeta* tbMeta;
int32_t metaType;
uint64_t dbId;
char dbFName[TSDB_DB_FNAME_LEN];
char ctbName[TSDB_TABLE_NAME_LEN];
char tbName[TSDB_TABLE_NAME_LEN];
SCTableMeta ctbMeta;
SVCTableMeta* vctbMeta;
STableMeta* tbMeta;
} STableMetaOutput;
typedef struct SViewMetaOutput {
@ -331,6 +382,7 @@ bool tIsValidSchema(struct SSchema* pSchema, int32_t numOfCols, int32_
int32_t getAsofJoinReverseOp(EOperatorType op);
int32_t queryCreateCTableMetaFromMsg(STableMetaRsp* msg, SCTableMeta* pMeta);
int32_t queryCreateVCTableMetaFromMsg(STableMetaRsp *msg, SVCTableMeta **pMeta);
int32_t queryCreateTableMetaFromMsg(STableMetaRsp* msg, bool isSuperTable, STableMeta** pMeta);
int32_t queryCreateTableMetaExFromMsg(STableMetaRsp* msg, bool isSuperTable, STableMeta** pMeta);
char* jobTaskStatusStr(int32_t status);
@ -340,6 +392,7 @@ SSchema createSchema(int8_t type, int32_t bytes, col_id_t colId, const char* nam
void destroyQueryExecRes(SExecResult* pRes);
int32_t dataConverToStr(char* str, int64_t capacity, int type, void* buf, int32_t bufSize, int32_t* len);
void parseTagDatatoJson(void* p, char** jsonStr, void *charsetCxt);
int32_t setColRef(SColRef* colRef, col_id_t colId, char* refColName, char* refTableName, char* refDbName);
int32_t cloneTableMeta(STableMeta* pSrc, STableMeta** pDst);
void getColumnTypeFromMeta(STableMeta* pMeta, char* pName, ETableColumnType* pType);
int32_t cloneDbVgInfo(SDBVgInfo* pSrc, SDBVgInfo** pDst);
@ -347,16 +400,22 @@ int32_t cloneSVreateTbReq(SVCreateTbReq* pSrc, SVCreateTbReq** pDst);
void freeVgInfo(SDBVgInfo* vgInfo);
void freeDbCfgInfo(SDbCfgInfo* pInfo);
void tFreeStreamVtbOtbInfo(void* param);
void tFreeStreamVtbVtbInfo(void* param);
void tFreeStreamVtbDbVgInfo(void* param);
extern int32_t (*queryBuildMsg[TDMT_MAX])(void* input, char** msg, int32_t msgSize, int32_t* msgLen,
void* (*mallocFp)(int64_t));
extern int32_t (*queryProcessMsgRsp[TDMT_MAX])(void* output, char* msg, int32_t msgSize);
void* getTaskPoolWorkerCb();
#define SET_META_TYPE_NULL(t) (t) = META_TYPE_NULL_TABLE
#define SET_META_TYPE_CTABLE(t) (t) = META_TYPE_CTABLE
#define SET_META_TYPE_TABLE(t) (t) = META_TYPE_TABLE
#define SET_META_TYPE_BOTH_TABLE(t) (t) = META_TYPE_BOTH_TABLE
#define SET_META_TYPE_NULL(t) (t) = META_TYPE_NULL_TABLE
#define SET_META_TYPE_CTABLE(t) (t) = META_TYPE_CTABLE
#define SET_META_TYPE_VCTABLE(t) (t) = META_TYPE_VCTABLE
#define SET_META_TYPE_TABLE(t) (t) = META_TYPE_TABLE
#define SET_META_TYPE_BOTH_TABLE(t) (t) = META_TYPE_BOTH_TABLE
#define SET_META_TYPE_BOTH_VTABLE(t) (t) = META_TYPE_BOTH_VTABLE
#define NEED_CLIENT_RM_TBLMETA_ERROR(_code) \
((_code) == TSDB_CODE_PAR_TABLE_NOT_EXIST || (_code) == TSDB_CODE_TDB_TABLE_NOT_EXIST || \

13
include/libs/stream/tstream.h
View File

@ -127,6 +127,7 @@ typedef enum {
TASK_LEVEL__AGG,
TASK_LEVEL__SINK,
TASK_LEVEL_SMA,
TASK_LEVEL__MERGE,
} ETASK_LEVEL;
enum {
@ -135,6 +136,7 @@ enum {
TASK_OUTPUT__TABLE,
TASK_OUTPUT__SMA,
TASK_OUTPUT__FETCH,
TASK_OUTPUT__VTABLE_MAP,
};
enum {
@ -211,6 +213,11 @@ typedef struct {
SUseDbRsp dbInfo;
} STaskDispatcherShuffle;
typedef struct {
SArray* taskInfos; // SArray<STaskDispatcherFixed>
SSHashObj* vtableMap;
} STaskDispatcherVtableMap;
typedef struct {
int32_t nodeId;
} SDownstreamTaskEpset;
@ -396,8 +403,9 @@ typedef struct SHistoryTaskInfo {
typedef struct STaskOutputInfo {
union {
STaskDispatcherFixed fixedDispatcher;
STaskDispatcherShuffle shuffleDispatcher;
STaskDispatcherFixed fixedDispatcher;
STaskDispatcherShuffle shuffleDispatcher;
STaskDispatcherVtableMap vtableMapDispatcher;
STaskSinkTb tbSink;
STaskSinkSma smaSink;
@ -466,6 +474,7 @@ struct SStreamTask {
STaskCheckInfo taskCheckInfo;
SNotifyInfo notifyInfo;
STaskNotifyEventStat notifyEventStat;
SArray* pVTables; // for merge task, SArray<SVCTableMergeInfo>
// the followings attributes don't be serialized
SScanhistorySchedInfo schedHistoryInfo;

13
include/util/taoserror.h
View File

@ -246,6 +246,7 @@ int32_t taosGetErrSize();
#define TSDB_CODE_MND_USER_DISABLED TAOS_DEF_ERROR_CODE(0, 0x0315)
#define TSDB_CODE_MND_INVALID_PLATFORM TAOS_DEF_ERROR_CODE(0, 0x0316)
#define TSDB_CODE_MND_RETURN_VALUE_NULL TAOS_DEF_ERROR_CODE(0, 0x0317)
#define TSDB_CODE_MND_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x0318)
// mnode-sdb
#define TSDB_CODE_SDB_OBJ_ALREADY_THERE TAOS_DEF_ERROR_CODE(0, 0x0320) // internal
@ -922,6 +923,10 @@ int32_t taosGetErrSize();
#define TSDB_CODE_PAR_INVALID_COLS_SELECTFUNC TAOS_DEF_ERROR_CODE(0, 0x268A)
#define TSDB_CODE_PAR_INVALID_COLS_ALIAS TAOS_DEF_ERROR_CODE(0, 0x268B)
#define TSDB_CODE_PAR_PRIM_KEY_MUST_BE_TS TAOS_DEF_ERROR_CODE(0, 0x268C)
#define TSDB_CODE_PAR_INVALID_REF_COLUMN TAOS_DEF_ERROR_CODE(0, 0x268D)
#define TSDB_CODE_PAR_INVALID_TABLE_TYPE TAOS_DEF_ERROR_CODE(0, 0x268E)
#define TSDB_CODE_PAR_INVALID_REF_COLUMN_TYPE TAOS_DEF_ERROR_CODE(0, 0x268F)
#define TSDB_CODE_PAR_MISMATCH_STABLE_TYPE TAOS_DEF_ERROR_CODE(0, 0x2690)
#define TSDB_CODE_PAR_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x26FF)
//planner
@ -929,6 +934,9 @@ int32_t taosGetErrSize();
#define TSDB_CODE_PLAN_EXPECTED_TS_EQUAL TAOS_DEF_ERROR_CODE(0, 0x2701)
#define TSDB_CODE_PLAN_NOT_SUPPORT_CROSS_JOIN TAOS_DEF_ERROR_CODE(0, 0x2702)
#define TSDB_CODE_PLAN_NOT_SUPPORT_JOIN_COND TAOS_DEF_ERROR_CODE(0, 0x2703)
#define TSDB_CODE_PLAN_SLOT_NOT_FOUND TAOS_DEF_ERROR_CODE(0, 0x2704)
#define TSDB_CODE_PLAN_INVALID_TABLE_TYPE TAOS_DEF_ERROR_CODE(0, 0x2705)
#define TSDB_CODE_PLAN_INVALID_DYN_CTRL_TYPE TAOS_DEF_ERROR_CODE(0, 0x2706)
//function
#define TSDB_CODE_FUNC_FUNTION_ERROR TAOS_DEF_ERROR_CODE(0, 0x2800)
@ -1056,6 +1064,11 @@ int32_t taosGetErrSize();
#define TSDB_CODE_AUDIT_FAIL_SEND_AUDIT_RECORD TAOS_DEF_ERROR_CODE(0, 0x6101)
#define TSDB_CODE_AUDIT_FAIL_GENERATE_JSON TAOS_DEF_ERROR_CODE(0, 0x6102)
// VTABLE
#define TSDB_CODE_VTABLE_SCAN_INTERNAL_ERROR TAOS_DEF_ERROR_CODE(0, 0x6200)
#define TSDB_CODE_VTABLE_SCAN_INVALID_DOWNSTREAM TAOS_DEF_ERROR_CODE(0, 0x6201)
#define TSDB_CODE_VTABLE_PRIMTS_HAS_REF TAOS_DEF_ERROR_CODE(0, 0x6202)
#define TSDB_CODE_VTABLE_NOT_VIRTUAL_SUPER_TABLE TAOS_DEF_ERROR_CODE(0, 0x6203)
#ifdef __cplusplus
}
#endif

1
include/util/tdef.h
View File

@ -285,6 +285,7 @@ typedef enum ELogicConditionType {
#define TSDB_COL_NAME_LEN 65
#define TSDB_COL_NAME_EXLEN 8
#define TSDB_COL_FNAME_LEN (TSDB_TABLE_NAME_LEN + TSDB_COL_NAME_LEN + TSDB_NAME_DELIMITER_LEN)
#define TSDB_COL_FNAME_EX_LEN (TSDB_DB_NAME_LEN + TSDB_NAME_DELIMITER_LEN + TSDB_TABLE_NAME_LEN + TSDB_NAME_DELIMITER_LEN + TSDB_COL_NAME_LEN)
#define TSDB_MAX_SAVED_SQL_LEN TSDB_MAX_COLUMNS * 64
#define TSDB_MAX_SQL_LEN TSDB_PAYLOAD_SIZE
#define TSDB_MAX_SQL_SHOW_LEN 1024

1
include/util/tlrucache.h
View File

@ -55,6 +55,7 @@ bool taosLRUCacheRef(SLRUCache *cache, LRUHandle *handle);
bool taosLRUCacheRelease(SLRUCache *cache, LRUHandle *handle, bool eraseIfLastRef);
void *taosLRUCacheValue(SLRUCache *cache, LRUHandle *handle);
void taosLRUCacheUpdate(SLRUCache *cache, LRUHandle *handle, void *value);
size_t taosLRUCacheGetUsage(SLRUCache *cache);
size_t taosLRUCacheGetPinnedUsage(SLRUCache *cache);

482
source/common/src/msg/tmsg.c
View File

@ -704,6 +704,8 @@ int32_t tSerializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pReq
ENCODESQL();
TAOS_CHECK_EXIT(tEncodeI8(&encoder, pReq->virtualStb));
tEndEncode(&encoder);
_exit:
@ -818,6 +820,12 @@ int32_t tDeserializeSMCreateStbReq(void *buf, int32_t bufLen, SMCreateStbReq *pR
DECODESQL();
if (!tDecodeIsEnd(&decoder)) {
TAOS_CHECK_EXIT(tDecodeI8(&decoder, &pReq->virtualStb));
} else {
pReq->virtualStb = 0;
}
tEndDecode(&decoder);
_exit:
@ -3954,6 +3962,14 @@ int32_t tSerializeSTableCfgRsp(void *buf, int32_t bufLen, STableCfgRsp *pRsp) {
}
}
TAOS_CHECK_EXIT(tEncodeI8(&encoder, pRsp->virtualStb));
if (hasRefCol(pRsp->tableType)) {
for (int32_t i = 0; i < pRsp->numOfColumns; ++i) {
SColRef *pColRef = &pRsp->pColRefs[i];
TAOS_CHECK_EXIT(tEncodeSColRef(&encoder, pColRef));
}
}
tEndEncode(&encoder);
_exit:
@ -4037,6 +4053,23 @@ int32_t tDeserializeSTableCfgRsp(void *buf, int32_t bufLen, STableCfgRsp *pRsp)
pRsp->pSchemaExt = NULL;
}
}
if (!tDecodeIsEnd(&decoder)) {
TAOS_CHECK_EXIT(tDecodeI8(&decoder, &pRsp->virtualStb));
if (hasRefCol(pRsp->tableType) && pRsp->numOfColumns > 0) {
pRsp->pColRefs = taosMemoryMalloc(sizeof(SColRef) * pRsp->numOfColumns);
if (pRsp->pColRefs == NULL) {
TAOS_CHECK_EXIT(terrno);
}
for (int32_t i = 0; i < pRsp->numOfColumns; ++i) {
SColRef *pColRef = &pRsp->pColRefs[i];
TAOS_CHECK_EXIT(tDecodeSColRef(&decoder, pColRef));
}
} else {
pRsp->pColRefs = NULL;
}
}
tEndDecode(&decoder);
_exit:
@ -4052,6 +4085,7 @@ void tFreeSTableCfgRsp(STableCfgRsp *pRsp) {
taosMemoryFreeClear(pRsp->pComment);
taosMemoryFreeClear(pRsp->pSchemas);
taosMemoryFreeClear(pRsp->pSchemaExt);
taosMemoryFreeClear(pRsp->pColRefs);
taosMemoryFreeClear(pRsp->pTags);
taosArrayDestroy(pRsp->pFuncs);
@ -4494,6 +4528,174 @@ _exit:
return code;
}
int32_t tSerializeSVSubTablesReq(void *buf, int32_t bufLen, SVSubTablesReq *pReq) {
SEncoder encoder = {0};
int32_t code = 0;
int32_t lino;
int32_t tlen;
tEncoderInit(&encoder, buf, bufLen);
TAOS_CHECK_EXIT(tStartEncode(&encoder));
TAOS_CHECK_EXIT(tEncodeI64(&encoder, pReq->suid));
tEndEncode(&encoder);
_exit:
if (code) {
tlen = code;
} else {
tlen = encoder.pos;
}
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVSubTablesReq(void *buf, int32_t bufLen, SVSubTablesReq *pReq) {
SDecoder decoder = {0};
int32_t code = 0;
int32_t lino;
tDecoderInit(&decoder, buf, bufLen);
TAOS_CHECK_EXIT(tStartDecode(&decoder));
TAOS_CHECK_EXIT(tDecodeI64(&decoder, &pReq->suid));
tEndDecode(&decoder);
_exit:
tDecoderClear(&decoder);
return code;
}
int32_t tSerializeSVSubTablesRspImpl(SEncoder* pEncoder, SVSubTablesRsp *pRsp) {
int32_t code = 0;
int32_t lino;
TAOS_CHECK_EXIT(tEncodeI32(pEncoder, pRsp->vgId));
int32_t numOfTables = taosArrayGetSize(pRsp->pTables);
TAOS_CHECK_EXIT(tEncodeI32(pEncoder, numOfTables));
for (int32_t i = 0; i < numOfTables; ++i) {
SVCTableRefCols *pTb = (SVCTableRefCols *)taosArrayGetP(pRsp->pTables, i);
TAOS_CHECK_EXIT(tEncodeU64(pEncoder, pTb->uid));
TAOS_CHECK_EXIT(tEncodeI32(pEncoder, pTb->numOfSrcTbls));
TAOS_CHECK_EXIT(tEncodeI32(pEncoder, pTb->numOfColRefs));
for (int32_t n = 0; n < pTb->numOfColRefs; ++n) {
SRefColInfo* pCol = pTb->refCols + n;
TAOS_CHECK_EXIT(tEncodeI16(pEncoder, pCol->colId));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pCol->refDbName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pCol->refTableName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pCol->refColName));
}
}
_exit:
return code;
}
int32_t tSerializeSVSubTablesRsp(void *buf, int32_t bufLen, SVSubTablesRsp *pRsp) {
SEncoder encoder = {0};
int32_t code = 0;
int32_t lino;
int32_t tlen;
tEncoderInit(&encoder, buf, bufLen);
TAOS_CHECK_EXIT(tStartEncode(&encoder));
TAOS_CHECK_EXIT(tSerializeSVSubTablesRspImpl(&encoder, pRsp));
tEndEncode(&encoder);
_exit:
if (code) {
tlen = code;
} else {
tlen = encoder.pos;
}
tEncoderClear(&encoder);
return tlen;
}
int32_t tDeserializeSVSubTablesRspImpl(SDecoder* pDecoder, SVSubTablesRsp *pRsp) {
int32_t code = 0;
int32_t lino;
SVCTableRefCols tb = {0};
TAOS_CHECK_EXIT(tDecodeI32(pDecoder, &pRsp->vgId));
int32_t numOfTables = 0;
TAOS_CHECK_EXIT(tDecodeI32(pDecoder, &numOfTables));
if (numOfTables > 0) {
pRsp->pTables = taosArrayInit(numOfTables, POINTER_BYTES);
if (NULL == pRsp->pTables) {
code = terrno;
return code;
}
for (int32_t i = 0; i < numOfTables; ++i) {
TAOS_CHECK_EXIT(tDecodeU64(pDecoder, &tb.uid));
TAOS_CHECK_EXIT(tDecodeI32(pDecoder, &tb.numOfSrcTbls));
TAOS_CHECK_EXIT(tDecodeI32(pDecoder, &tb.numOfColRefs));
if (tb.numOfColRefs > 0) {
SVCTableRefCols* pTb = taosMemoryCalloc(1, sizeof(tb) + tb.numOfColRefs * sizeof(SRefColInfo));
if (NULL == pTb) {
code = terrno;
return code;
}
if (NULL == taosArrayPush(pRsp->pTables, &pTb)) {
code = terrno;
taosMemoryFree(pTb);
return code;
}
pTb->uid = tb.uid;
pTb->numOfSrcTbls = tb.numOfSrcTbls;
pTb->numOfColRefs = tb.numOfColRefs;
pTb->refCols = (SRefColInfo*)(pTb + 1);
for (int32_t n = 0; n < tb.numOfColRefs; ++n) {
TAOS_CHECK_EXIT(tDecodeI16(pDecoder, &pTb->refCols[n].colId));
TAOS_CHECK_EXIT(tDecodeCStrTo(pDecoder, pTb->refCols[n].refDbName));
TAOS_CHECK_EXIT(tDecodeCStrTo(pDecoder, pTb->refCols[n].refTableName));
TAOS_CHECK_EXIT(tDecodeCStrTo(pDecoder, pTb->refCols[n].refColName));
}
}
}
}
_exit:
return code;
}
int32_t tDeserializeSVSubTablesRsp(void *buf, int32_t bufLen, SVSubTablesRsp *pRsp) {
SDecoder decoder = {0};
int32_t code = 0;
int32_t lino;
tDecoderInit(&decoder, buf, bufLen);
TAOS_CHECK_EXIT(tStartDecode(&decoder));
TAOS_CHECK_EXIT(tDeserializeSVSubTablesRspImpl(&decoder, pRsp));
tEndDecode(&decoder);
_exit:
tDecoderClear(&decoder);
return code;
}
void tFreeSVCTableRefCols(void *pParam) {
SVCTableRefCols* pCols = *(SVCTableRefCols**)pParam;
if (NULL == pCols) {
return;
}
taosMemoryFree(pCols);
}
void tDestroySVSubTablesRsp(void* rsp) {
if (NULL == rsp) {
return;
}
SVSubTablesRsp *pRsp = (SVSubTablesRsp*)rsp;
taosArrayDestroyEx(pRsp->pTables, tFreeSVCTableRefCols);
}
int32_t tSerializeSQnodeListReq(void *buf, int32_t bufLen, SQnodeListReq *pReq) {
SEncoder encoder = {0};
int32_t code = 0;
@ -6125,6 +6327,15 @@ static int32_t tEncodeSTableMetaRsp(SEncoder *pEncoder, STableMetaRsp *pRsp) {
}
}
TAOS_CHECK_RETURN(tEncodeI8(pEncoder, pRsp->virtualStb));
TAOS_CHECK_RETURN(tEncodeI32(pEncoder, pRsp->numOfColRefs));
if (hasRefCol(pRsp->tableType)) {
for (int32_t i = 0; i < pRsp->numOfColRefs; ++i) {
SColRef *pColRef = &pRsp->pColRefs[i];
TAOS_CHECK_RETURN(tEncodeSColRef(pEncoder, pColRef));
}
}
return 0;
}
@ -6173,6 +6384,23 @@ static int32_t tDecodeSTableMetaRsp(SDecoder *pDecoder, STableMetaRsp *pRsp) {
pRsp->pSchemaExt = NULL;
}
}
if (!tDecodeIsEnd(pDecoder)) {
TAOS_CHECK_RETURN(tDecodeI8(pDecoder, &pRsp->virtualStb));
TAOS_CHECK_RETURN(tDecodeI32(pDecoder, &pRsp->numOfColRefs));
if (hasRefCol(pRsp->tableType) && pRsp->numOfColRefs > 0) {
pRsp->pColRefs = taosMemoryMalloc(sizeof(SColRef) * pRsp->numOfColRefs);
if (pRsp->pColRefs == NULL) {
TAOS_CHECK_RETURN(terrno);
}
for (int32_t i = 0; i < pRsp->numOfColRefs; ++i) {
SColRef *pColRef = &pRsp->pColRefs[i];
TAOS_CHECK_RETURN(tDecodeSColRef(pDecoder, pColRef));
}
} else {
pRsp->pColRefs = NULL;
}
}
return 0;
}
@ -6282,6 +6510,7 @@ int32_t tDeserializeSSTbHbRsp(void *buf, int32_t bufLen, SSTbHbRsp *pRsp) {
if (taosArrayPush(pRsp->pMetaRsp, &tableMetaRsp) == NULL) {
taosMemoryFree(tableMetaRsp.pSchemas);
taosMemoryFree(tableMetaRsp.pSchemaExt);
taosMemoryFree(tableMetaRsp.pColRefs);
TAOS_CHECK_EXIT(terrno);
}
}
@ -6337,6 +6566,7 @@ void tFreeSTableMetaRsp(void *pRsp) {
taosMemoryFreeClear(((STableMetaRsp *)pRsp)->pSchemas);
taosMemoryFreeClear(((STableMetaRsp *)pRsp)->pSchemaExt);
taosMemoryFreeClear(((STableMetaRsp *)pRsp)->pColRefs);
}
void tFreeSTableIndexRsp(void *info) {
@ -8463,9 +8693,9 @@ int32_t tSerializeSMArbUpdateGroupBatchReq(void *buf, int32_t bufLen, SMArbUpdat
SMArbUpdateGroup *pGroup = taosArrayGet(pReq->updateArray, i);
TAOS_CHECK_EXIT(tEncodeI32(&encoder, pGroup->vgId));
TAOS_CHECK_EXIT(tEncodeI64(&encoder, pGroup->dbUid));
for (int i = 0; i < TSDB_ARB_GROUP_MEMBER_NUM; i++) {
TAOS_CHECK_EXIT(tEncodeI32(&encoder, pGroup->members[i].dnodeId));
TAOS_CHECK_EXIT(tEncodeCStr(&encoder, pGroup->members[i].token));
for (int j = 0; j < TSDB_ARB_GROUP_MEMBER_NUM; j++) {
TAOS_CHECK_EXIT(tEncodeI32(&encoder, pGroup->members[j].dnodeId));
TAOS_CHECK_EXIT(tEncodeCStr(&encoder, pGroup->members[j].token));
}
TAOS_CHECK_EXIT(tEncodeI8(&encoder, pGroup->isSync));
TAOS_CHECK_EXIT(tEncodeI32(&encoder, pGroup->assignedLeader.dnodeId));
@ -8515,12 +8745,12 @@ int32_t tDeserializeSMArbUpdateGroupBatchReq(void *buf, int32_t bufLen, SMArbUpd
SMArbUpdateGroup group = {0};
TAOS_CHECK_EXIT(tDecodeI32(&decoder, &group.vgId));
TAOS_CHECK_EXIT(tDecodeI64(&decoder, &group.dbUid));
for (int i = 0; i < TSDB_ARB_GROUP_MEMBER_NUM; i++) {
TAOS_CHECK_EXIT(tDecodeI32(&decoder, &group.members[i].dnodeId));
if ((group.members[i].token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE)) == NULL) {
for (int j = 0; j < TSDB_ARB_GROUP_MEMBER_NUM; j++) {
TAOS_CHECK_EXIT(tDecodeI32(&decoder, &group.members[j].dnodeId));
if ((group.members[j].token = taosMemoryMalloc(TSDB_ARB_TOKEN_SIZE)) == NULL) {
TAOS_CHECK_EXIT(terrno);
}
TAOS_CHECK_EXIT(tDecodeCStrTo(&decoder, group.members[i].token));
TAOS_CHECK_EXIT(tDecodeCStrTo(&decoder, group.members[j].token));
}
TAOS_CHECK_EXIT(tDecodeI8(&decoder, &group.isSync));
TAOS_CHECK_EXIT(tDecodeI32(&decoder, &group.assignedLeader.dnodeId));
@ -8568,8 +8798,8 @@ void tFreeSMArbUpdateGroupBatchReq(SMArbUpdateGroupBatchReq *pReq) {
int32_t sz = taosArrayGetSize(pReq->updateArray);
for (int32_t i = 0; i < sz; i++) {
SMArbUpdateGroup *pGroup = taosArrayGet(pReq->updateArray, i);
for (int i = 0; i < TSDB_ARB_GROUP_MEMBER_NUM; i++) {
taosMemoryFreeClear(pGroup->members[i].token);
for (int j = 0; j < TSDB_ARB_GROUP_MEMBER_NUM; j++) {
taosMemoryFreeClear(pGroup->members[j].token);
}
taosMemoryFreeClear(pGroup->assignedLeader.token);
}
@ -9209,6 +9439,22 @@ int32_t tSerializeSOperatorParam(SEncoder *pEncoder, SOperatorParam *pOpParam) {
int64_t *pUid = taosArrayGet(pScan->pUidList, m);
TAOS_CHECK_RETURN(tEncodeI64(pEncoder, *pUid));
}
if (pScan->pOrgTbInfo) {
TAOS_CHECK_RETURN(tEncodeBool(pEncoder, true));
TAOS_CHECK_RETURN(tEncodeI32(pEncoder, pScan->pOrgTbInfo->vgId));
TAOS_CHECK_RETURN(tEncodeCStr(pEncoder, pScan->pOrgTbInfo->tbName));
int32_t num = taosArrayGetSize(pScan->pOrgTbInfo->colMap);
TAOS_CHECK_RETURN(tEncodeI32(pEncoder, num));
for (int32_t i = 0; i < num; ++i) {
SColIdNameKV *pColKV = taosArrayGet(pScan->pOrgTbInfo->colMap, i);
TAOS_CHECK_RETURN(tEncodeI16(pEncoder, pColKV->colId));
TAOS_CHECK_RETURN(tEncodeCStr(pEncoder, pColKV->colName));
}
} else {
TAOS_CHECK_RETURN(tEncodeBool(pEncoder, false));
}
TAOS_CHECK_RETURN(tEncodeI64(pEncoder, pScan->window.skey));
TAOS_CHECK_RETURN(tEncodeI64(pEncoder, pScan->window.ekey));
break;
}
default:
@ -9222,6 +9468,7 @@ int32_t tSerializeSOperatorParam(SEncoder *pEncoder, SOperatorParam *pOpParam) {
TAOS_CHECK_RETURN(tSerializeSOperatorParam(pEncoder, pChild));
}
TAOS_CHECK_RETURN(tEncodeBool(pEncoder, pOpParam->reUse));
return 0;
}
@ -9253,6 +9500,33 @@ int32_t tDeserializeSOperatorParam(SDecoder *pDecoder, SOperatorParam *pOpParam)
} else {
pScan->pUidList = NULL;
}
bool hasTbInfo = false;
TAOS_CHECK_RETURN(tDecodeBool(pDecoder, &hasTbInfo));
if (hasTbInfo) {
pScan->pOrgTbInfo = taosMemoryMalloc(sizeof(SOrgTbInfo));
if (NULL == pScan->pOrgTbInfo) {
TAOS_CHECK_RETURN(terrno);
}
TAOS_CHECK_RETURN(tDecodeI32(pDecoder, &pScan->pOrgTbInfo->vgId));
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, pScan->pOrgTbInfo->tbName));
int32_t num = 0;
TAOS_CHECK_RETURN(tDecodeI32(pDecoder, &num));
pScan->pOrgTbInfo->colMap = taosArrayInit(num, sizeof(SColIdNameKV));
for (int32_t i = 0; i < num; ++i) {
SColIdNameKV pColKV;
TAOS_CHECK_RETURN(tDecodeI16(pDecoder, (int16_t *)&(pColKV.colId)));
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, pColKV.colName));
if (taosArrayPush(pScan->pOrgTbInfo->colMap, &pColKV) == NULL) {
TAOS_CHECK_RETURN(terrno);
}
}
} else {
pScan->pOrgTbInfo = NULL;
}
TAOS_CHECK_RETURN(tDecodeI64(pDecoder, &pScan->window.skey));
TAOS_CHECK_RETURN(tDecodeI64(pDecoder, &pScan->window.ekey));
pOpParam->value = pScan;
break;
}
@ -9282,6 +9556,12 @@ int32_t tDeserializeSOperatorParam(SDecoder *pDecoder, SOperatorParam *pOpParam)
pOpParam->pChildren = NULL;
}
if (!tDecodeIsEnd(pDecoder)) {
TAOS_CHECK_RETURN(tDecodeBool(pDecoder, &pOpParam->reUse));
} else {
pOpParam->reUse = false;
}
return 0;
}
@ -10184,6 +10464,13 @@ int32_t tSerializeSCMCreateStreamReq(void *buf, int32_t bufLen, const SCMCreateS
TAOS_CHECK_EXIT(tEncodeCStr(&encoder, pReq->pWendName));
TAOS_CHECK_EXIT(tEncodeCStr(&encoder, pReq->pGroupIdName));
TAOS_CHECK_EXIT(tEncodeCStr(&encoder, pReq->pIsWindowFilledName));
int32_t vgNum = taosArrayGetSize(pReq->pVSubTables);
TAOS_CHECK_EXIT(tEncodeI32(&encoder, vgNum));
for (int32_t i = 0; i < vgNum; ++i) {
SVSubTablesRsp* pVgTables = taosArrayGet(pReq->pVSubTables, i);
TAOS_CHECK_EXIT(tSerializeSVSubTablesRspImpl(&encoder, pVgTables));
}
tEndEncode(&encoder);
_exit:
@ -10348,9 +10635,27 @@ int32_t tDeserializeSCMCreateStreamReq(void *buf, int32_t bufLen, SCMCreateStrea
TAOS_CHECK_EXIT(tDecodeCStrTo(&decoder, pReq->pWendName));
TAOS_CHECK_EXIT(tDecodeCStrTo(&decoder, pReq->pGroupIdName));
TAOS_CHECK_EXIT(tDecodeCStrTo(&decoder, pReq->pIsWindowFilledName));
int32_t vgNum = 0;
TAOS_CHECK_EXIT(tDecodeI32(&decoder, &vgNum));
if (vgNum > 0) {
pReq->pVSubTables = taosArrayInit(vgNum, sizeof(SVSubTablesRsp));
if (pReq->pVSubTables == NULL) {
TAOS_CHECK_EXIT(terrno);
}
SVSubTablesRsp vgTables = {0};
for (int32_t i = 0; i < vgNum; ++i) {
vgTables.pTables = NULL;
TAOS_CHECK_EXIT(tDeserializeSVSubTablesRspImpl(&decoder, &vgTables));
if (taosArrayPush(pReq->pVSubTables, &vgTables) == NULL) {
tDestroySVSubTablesRsp(&vgTables);
TAOS_CHECK_EXIT(terrno);
}
}
}
}
tEndDecode(&decoder);
_exit:
tDecoderClear(&decoder);
return code;
@ -10413,6 +10718,7 @@ void tFreeSCMCreateStreamReq(SCMCreateStreamReq *pReq) {
taosArrayDestroy(pReq->pVgroupVerList);
taosArrayDestroy(pReq->pCols);
taosArrayDestroyP(pReq->pNotifyAddrUrls, NULL);
taosArrayDestroyEx(pReq->pVSubTables, tDestroySVSubTablesRsp);
}
int32_t tEncodeSRSmaParam(SEncoder *pCoder, const SRSmaParam *pRSmaParam) {
@ -10449,6 +10755,57 @@ _exit:
return code;
}
int32_t tEncodeSColRefWrapper(SEncoder *pCoder, const SColRefWrapper *pWrapper) {
int32_t code = 0;
int32_t lino;
TAOS_CHECK_EXIT(tEncodeI32v(pCoder, pWrapper->nCols));
TAOS_CHECK_EXIT(tEncodeI32v(pCoder, pWrapper->version));
for (int32_t i = 0; i < pWrapper->nCols; i++) {
SColRef *p = &pWrapper->pColRef[i];
TAOS_CHECK_EXIT(tEncodeI8(pCoder, p->hasRef));
TAOS_CHECK_EXIT(tEncodeI16v(pCoder, p->id));
if (p->hasRef) {
TAOS_CHECK_EXIT(tEncodeCStr(pCoder, p->refDbName));
TAOS_CHECK_EXIT(tEncodeCStr(pCoder, p->refTableName));
TAOS_CHECK_EXIT(tEncodeCStr(pCoder, p->refColName));
}
}
_exit:
return code;
}
int32_t tDecodeSColRefWrapperEx(SDecoder *pDecoder, SColRefWrapper *pWrapper) {
int32_t code = 0;
int32_t lino;
TAOS_CHECK_EXIT(tDecodeI32v(pDecoder, &pWrapper->nCols));
TAOS_CHECK_EXIT(tDecodeI32v(pDecoder, &pWrapper->version));
pWrapper->pColRef = (SColRef *)tDecoderMalloc(pDecoder, pWrapper->nCols * sizeof(SColRef));
if (pWrapper->pColRef == NULL) {
TAOS_CHECK_EXIT(terrno);
}
for (int i = 0; i < pWrapper->nCols; i++) {
SColRef *p = &pWrapper->pColRef[i];
TAOS_CHECK_EXIT(tDecodeI8(pDecoder, (int8_t *)&p->hasRef));
TAOS_CHECK_EXIT(tDecodeI16v(pDecoder, &p->id));
if (p->hasRef) {
TAOS_CHECK_EXIT(tDecodeCStrTo(pDecoder, p->refDbName));
TAOS_CHECK_EXIT(tDecodeCStrTo(pDecoder, p->refTableName));
TAOS_CHECK_EXIT(tDecodeCStrTo(pDecoder, p->refColName));
}
}
_exit:
if (code) {
taosMemoryFree(pWrapper->pColRef);
}
return code;
}
int32_t tEncodeSColCmprWrapper(SEncoder *pCoder, const SColCmprWrapper *pWrapper) {
int32_t code = 0;
int32_t lino;
@ -10558,6 +10915,7 @@ int tEncodeSVCreateStbReq(SEncoder *pCoder, const SVCreateStbReq *pReq) {
} else {
TAOS_CHECK_EXIT(tEncodeI8(pCoder, 0));
}
TAOS_CHECK_EXIT(tEncodeI8(pCoder, pReq->virtualStb));
tEndEncode(pCoder);
_exit:
@ -10603,6 +10961,9 @@ int tDecodeSVCreateStbReq(SDecoder *pCoder, SVCreateStbReq *pReq) {
}
}
}
if (!tDecodeIsEnd(pCoder)) {
TAOS_CHECK_EXIT(tDecodeI8(pCoder, &pReq->virtualStb));
}
tEndDecode(pCoder);
_exit:
@ -10626,7 +10987,7 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) {
TAOS_CHECK_EXIT(tEncodeCStr(pCoder, pReq->comment));
}
if (pReq->type == TSDB_CHILD_TABLE) {
if (pReq->type == TSDB_CHILD_TABLE || pReq->type == TSDB_VIRTUAL_CHILD_TABLE) {
TAOS_CHECK_EXIT(tEncodeCStr(pCoder, pReq->ctb.stbName));
TAOS_CHECK_EXIT(tEncodeU8(pCoder, pReq->ctb.tagNum));
TAOS_CHECK_EXIT(tEncodeI64(pCoder, pReq->ctb.suid));
@ -10637,7 +10998,7 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) {
char *name = taosArrayGet(pReq->ctb.tagName, i);
TAOS_CHECK_EXIT(tEncodeCStr(pCoder, name));
}
} else if (pReq->type == TSDB_NORMAL_TABLE) {
} else if (pReq->type == TSDB_NORMAL_TABLE || pReq->type == TSDB_VIRTUAL_NORMAL_TABLE) {
TAOS_CHECK_EXIT(tEncodeSSchemaWrapper(pCoder, &pReq->ntb.schemaRow));
} else {
return TSDB_CODE_INVALID_MSG;
@ -10651,12 +11012,15 @@ int tEncodeSVCreateTbReq(SEncoder *pCoder, const SVCreateTbReq *pReq) {
// Encode Column Options: encode compress level
if (pReq->type == TSDB_SUPER_TABLE || pReq->type == TSDB_NORMAL_TABLE) {
TAOS_CHECK_EXIT(tEncodeSColCmprWrapper(pCoder, &pReq->colCmpr));
if (pReq->pExtSchemas) {
TAOS_CHECK_EXIT(tEncodeI8(pCoder, 1));
TAOS_CHECK_EXIT(tEncodeSExtSchemas(pCoder, pReq->pExtSchemas, pReq->ntb.schemaRow.nCols));
} else {
TAOS_CHECK_EXIT(tEncodeI8(pCoder, 0));
}
}
if (pReq->type == TSDB_VIRTUAL_NORMAL_TABLE || pReq->type == TSDB_VIRTUAL_CHILD_TABLE) {
TAOS_CHECK_EXIT(tEncodeSColRefWrapper(pCoder, &pReq->colRef));
}
if (pReq->pExtSchemas) {
TAOS_CHECK_EXIT(tEncodeI8(pCoder, 1));
TAOS_CHECK_EXIT(tEncodeSExtSchemas(pCoder, pReq->pExtSchemas, pReq->ntb.schemaRow.nCols));
} else {
TAOS_CHECK_EXIT(tEncodeI8(pCoder, 0));
}
tEndEncode(pCoder);
@ -10685,7 +11049,7 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) {
TAOS_CHECK_EXIT(tDecodeCStrTo(pCoder, pReq->comment));
}
if (pReq->type == TSDB_CHILD_TABLE) {
if (pReq->type == TSDB_CHILD_TABLE || pReq->type == TSDB_VIRTUAL_CHILD_TABLE) {
TAOS_CHECK_EXIT(tDecodeCStr(pCoder, &pReq->ctb.stbName));
TAOS_CHECK_EXIT(tDecodeU8(pCoder, &pReq->ctb.tagNum));
TAOS_CHECK_EXIT(tDecodeI64(pCoder, &pReq->ctb.suid));
@ -10705,7 +11069,7 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) {
TAOS_CHECK_EXIT(terrno);
}
}
} else if (pReq->type == TSDB_NORMAL_TABLE) {
} else if (pReq->type == TSDB_NORMAL_TABLE || pReq->type == TSDB_VIRTUAL_NORMAL_TABLE) {
TAOS_CHECK_EXIT(tDecodeSSchemaWrapperEx(pCoder, &pReq->ntb.schemaRow));
} else {
return TSDB_CODE_INVALID_MSG;
@ -10717,11 +11081,16 @@ int tDecodeSVCreateTbReq(SDecoder *pCoder, SVCreateTbReq *pReq) {
if (pReq->sqlLen > 0) {
TAOS_CHECK_EXIT(tDecodeBinaryAlloc(pCoder, (void **)&pReq->sql, NULL));
}
if (pReq->type == TSDB_NORMAL_TABLE || pReq->type == TSDB_SUPER_TABLE)
if (pReq->type == TSDB_NORMAL_TABLE || pReq->type == TSDB_SUPER_TABLE) {
if (!tDecodeIsEnd(pCoder)) {
TAOS_CHECK_EXIT(tDecodeSColCmprWrapperEx(pCoder, &pReq->colCmpr));
}
} else if (pReq->type == TSDB_VIRTUAL_NORMAL_TABLE || pReq->type == TSDB_VIRTUAL_CHILD_TABLE) {
if (!tDecodeIsEnd(pCoder)) {
TAOS_CHECK_EXIT(tDecodeSColRefWrapperEx(pCoder, &pReq->colRef));
}
}
if (!tDecodeIsEnd(pCoder)) {
int8_t hasExtSchema = 0;
TAOS_CHECK_EXIT(tDecodeI8(pCoder, &hasExtSchema));
@ -10742,25 +11111,18 @@ void tDestroySVCreateTbReq(SVCreateTbReq *pReq, int32_t flags) {
if (flags & TSDB_MSG_FLG_ENCODE) {
// TODO
} else if (flags & TSDB_MSG_FLG_DECODE) {
if (pReq->comment) {
pReq->comment = NULL;
taosMemoryFree(pReq->comment);
}
taosMemoryFreeClear(pReq->comment);
if (pReq->type == TSDB_CHILD_TABLE) {
if (pReq->ctb.tagName) taosArrayDestroy(pReq->ctb.tagName);
} else if (pReq->type == TSDB_NORMAL_TABLE) {
if (pReq->ntb.schemaRow.pSchema) taosMemoryFree(pReq->ntb.schemaRow.pSchema);
if (pReq->type == TSDB_CHILD_TABLE || pReq->type == TSDB_VIRTUAL_CHILD_TABLE) {
taosArrayDestroy(pReq->ctb.tagName);
} else if (pReq->type == TSDB_NORMAL_TABLE || pReq->type == TSDB_VIRTUAL_NORMAL_TABLE) {
taosMemoryFreeClear(pReq->ntb.schemaRow.pSchema);
}
}
if (pReq->colCmpr.pColCmpr) taosMemoryFree(pReq->colCmpr.pColCmpr);
pReq->colCmpr.pColCmpr = NULL;
if (pReq->sql != NULL) {
taosMemoryFree(pReq->sql);
}
pReq->sql = NULL;
taosMemoryFreeClear(pReq->colCmpr.pColCmpr);
taosMemoryFreeClear(pReq->colRef.pColRef);
taosMemoryFreeClear(pReq->sql);
}
int tEncodeSVCreateTbBatchReq(SEncoder *pCoder, const SVCreateTbBatchReq *pReq) {
@ -10803,7 +11165,7 @@ void tDeleteSVCreateTbBatchReq(SVCreateTbBatchReq *pReq) {
SVCreateTbReq *pCreateReq = pReq->pReqs + iReq;
taosMemoryFreeClear(pCreateReq->sql);
taosMemoryFreeClear(pCreateReq->comment);
if (pCreateReq->type == TSDB_CHILD_TABLE) {
if (pCreateReq->type == TSDB_CHILD_TABLE || pCreateReq->type == TSDB_VIRTUAL_CHILD_TABLE) {
taosArrayDestroy(pCreateReq->ctb.tagName);
pCreateReq->ctb.tagName = NULL;
}
@ -10853,6 +11215,7 @@ void tFreeSVCreateTbRsp(void *param) {
if (pRsp->pMeta) {
taosMemoryFree(pRsp->pMeta->pSchemas);
taosMemoryFree(pRsp->pMeta->pSchemaExt);
taosMemoryFree(pRsp->pMeta->pColRefs);
taosMemoryFree(pRsp->pMeta);
}
}
@ -10864,6 +11227,7 @@ static int32_t tEncodeSVDropTbReq(SEncoder *pCoder, const SVDropTbReq *pReq) {
TAOS_CHECK_RETURN(tEncodeU64(pCoder, pReq->suid));
TAOS_CHECK_RETURN(tEncodeI64(pCoder, pReq->uid));
TAOS_CHECK_RETURN(tEncodeI8(pCoder, pReq->igNotExists));
TAOS_CHECK_RETURN(tEncodeI8(pCoder, pReq->isVirtual));
tEndEncode(pCoder);
return 0;
@ -10875,6 +11239,9 @@ static int32_t tDecodeSVDropTbReq(SDecoder *pCoder, SVDropTbReq *pReq) {
TAOS_CHECK_RETURN(tDecodeU64(pCoder, &pReq->suid));
TAOS_CHECK_RETURN(tDecodeI64(pCoder, &pReq->uid));
TAOS_CHECK_RETURN(tDecodeI8(pCoder, &pReq->igNotExists));
if (!tDecodeIsEnd(pCoder)) {
TAOS_CHECK_RETURN(tDecodeI8(pCoder, &pReq->isVirtual));
}
tEndDecode(pCoder);
return 0;
@ -11081,6 +11448,24 @@ int32_t tEncodeSVAlterTbReq(SEncoder *pEncoder, const SVAlterTbReq *pReq) {
TAOS_CHECK_EXIT(tEncodeI32v(pEncoder, pReq->bytes));
TAOS_CHECK_EXIT(tEncodeU32(pEncoder, pReq->compress));
break;
case TSDB_ALTER_TABLE_ALTER_COLUMN_REF:
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->colName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->refDbName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->refTbName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->refColName));
break;
case TSDB_ALTER_TABLE_REMOVE_COLUMN_REF:
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->colName));
break;
case TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COLUMN_REF:
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->colName));
TAOS_CHECK_EXIT(tEncodeI8(pEncoder, pReq->type));
TAOS_CHECK_EXIT(tEncodeI8(pEncoder, pReq->flags));
TAOS_CHECK_EXIT(tEncodeI32v(pEncoder, pReq->bytes));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->refDbName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->refTbName));
TAOS_CHECK_EXIT(tEncodeCStr(pEncoder, pReq->refColName));
break;
default:
break;
}
@ -11171,6 +11556,25 @@ static int32_t tDecodeSVAlterTbReqCommon(SDecoder *pDecoder, SVAlterTbReq *pReq)
TAOS_CHECK_EXIT(tDecodeI8(pDecoder, &pReq->flags));
TAOS_CHECK_EXIT(tDecodeI32v(pDecoder, &pReq->bytes));
TAOS_CHECK_EXIT(tDecodeU32(pDecoder, &pReq->compress));
break;
case TSDB_ALTER_TABLE_ALTER_COLUMN_REF:
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->colName));
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->refDbName));
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->refTbName));
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->refColName));
break;
case TSDB_ALTER_TABLE_REMOVE_COLUMN_REF:
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->colName));
break;
case TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COLUMN_REF:
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->colName));
TAOS_CHECK_EXIT(tDecodeI8(pDecoder, &pReq->type));
TAOS_CHECK_EXIT(tDecodeI8(pDecoder, &pReq->flags));
TAOS_CHECK_EXIT(tDecodeI32v(pDecoder, &pReq->bytes));
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->refDbName));
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->refTbName));
TAOS_CHECK_EXIT(tDecodeCStr(pDecoder, &pReq->refColName));
break;
default:
break;
}
@ -11308,6 +11712,7 @@ void tFreeSMAlterStbRsp(SMAlterStbRsp *pRsp) {
if (pRsp->pMeta) {
taosMemoryFree(pRsp->pMeta->pSchemas);
taosMemoryFree(pRsp->pMeta->pSchemaExt);
taosMemoryFree(pRsp->pMeta->pColRefs);
taosMemoryFree(pRsp->pMeta);
}
}
@ -11359,6 +11764,7 @@ void tFreeSMCreateStbRsp(SMCreateStbRsp *pRsp) {
if (pRsp->pMeta) {
taosMemoryFree(pRsp->pMeta->pSchemas);
taosMemoryFree(pRsp->pMeta->pSchemaExt);
taosMemoryFree(pRsp->pMeta->pColRefs);
taosMemoryFree(pRsp->pMeta);
}
}
@ -13531,4 +13937,4 @@ void tDeleteMqBatchMetaRsp(SMqBatchMetaRsp *pRsp) {
bool hasExtSchema(const SExtSchema *pExtSchema) {
return pExtSchema->typeMod != 0;
}
}

4
source/common/src/systable.c
View File

@ -164,6 +164,7 @@ static const SSysDbTableSchema userStbsSchema[] = {
{.name = "max_delay", .bytes = 64 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = false},
{.name = "rollup", .bytes = 128 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = false},
{.name = "uid", .bytes = 8, .type = TSDB_DATA_TYPE_BIGINT, .sysInfo = false},
{.name = "isvirtual", .bytes = 1, .type = TSDB_DATA_TYPE_BOOL, .sysInfo = false},
};
static const SSysDbTableSchema streamSchema[] = {
@ -247,7 +248,8 @@ static const SSysDbTableSchema userColsSchema[] = {
{.name = "col_length", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false},
{.name = "col_precision", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false},
{.name = "col_scale", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false},
{.name = "col_nullable", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false}
{.name = "col_nullable", .bytes = 4, .type = TSDB_DATA_TYPE_INT, .sysInfo = false},
{.name = "col_source", .bytes = TSDB_COL_FNAME_LEN - 1 + VARSTR_HEADER_SIZE, .type = TSDB_DATA_TYPE_VARCHAR, .sysInfo = false}
};
static const SSysDbTableSchema userTblDistSchema[] = {

4
source/common/src/tcol.c
View File

@ -354,6 +354,10 @@ bool withExtSchema(uint8_t tableType) {
return TSDB_SUPER_TABLE == tableType || TSDB_NORMAL_TABLE == tableType || TSDB_CHILD_TABLE == tableType;
}
bool hasRefCol(uint8_t tableType) {
return TSDB_VIRTUAL_NORMAL_TABLE == tableType || TSDB_VIRTUAL_CHILD_TABLE == tableType;
}
int8_t validColCompressLevel(uint8_t type, uint8_t level) {
if (level == TSDB_COLVAL_LEVEL_DISABLED) return 1;
if (level < TSDB_COLVAL_LEVEL_NOCHANGE || level > TSDB_COLVAL_LEVEL_HIGH) {

93
source/common/src/tdatablock.c
View File

@ -1224,7 +1224,7 @@ size_t blockDataGetRowSize(SSDataBlock* pBlock) {
*/
size_t blockDataGetSerialMetaSize(uint32_t numOfCols) {
// | version | total length | total rows | blankFull | total columns | flag seg| block group id | column schema
// | each column length |
// | each column length
return sizeof(int32_t) + sizeof(int32_t) + sizeof(int32_t) + sizeof(bool) + sizeof(int32_t) + sizeof(int32_t) +
sizeof(uint64_t) + numOfCols * (sizeof(int8_t) + sizeof(int32_t)) + numOfCols * sizeof(int32_t);
}
@ -2112,6 +2112,97 @@ int32_t createOneDataBlock(const SSDataBlock* pDataBlock, bool copyData, SSDataB
return code;
}
int32_t createOneDataBlockWithColArray(const SSDataBlock* pDataBlock, SArray* pColArray, SSDataBlock** pResBlock) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SSDataBlock* pDstBlock = NULL;
QRY_PARAM_CHECK(pResBlock);
QUERY_CHECK_NULL(pDataBlock, code, lino, _return, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_CODE(createDataBlock(&pDstBlock), lino, _return);
pDstBlock->info = pDataBlock->info;
pDstBlock->info.pks[0].pData = NULL;
pDstBlock->info.pks[1].pData = NULL;
pDstBlock->info.rows = 0;
pDstBlock->info.capacity = 0;
pDstBlock->info.rowSize = 0;
pDstBlock->info.id = pDataBlock->info.id;
pDstBlock->info.blankFill = pDataBlock->info.blankFill;
for (int32_t i = 0; i < taosArrayGetSize(pColArray); ++i) {
SColIdPair *pColPair = taosArrayGet(pColArray, i);
QUERY_CHECK_NULL(pColPair, code, lino, _return, terrno);
for (int32_t j = 0; j < taosArrayGetSize(pDataBlock->pDataBlock); ++j) {
SColumnInfoData* p = taosArrayGet(pDataBlock->pDataBlock, j);
if (p == NULL) {
continue;
}
if (p->info.colId == pColPair->vtbColId) {
QUERY_CHECK_CODE(blockDataAppendColInfo(pDstBlock, p), lino, _return);
break;
}
}
}
*pResBlock = pDstBlock;
return code;
_return:
uError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
blockDataDestroy(pDstBlock);
return code;
}
int32_t createOneDataBlockWithTwoBlock(const SSDataBlock* pDataBlock, const SSDataBlock* pOrgBlock, SSDataBlock** pResBlock) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SSDataBlock *pDstBlock = NULL;
QRY_PARAM_CHECK(pResBlock);
QUERY_CHECK_NULL(pDataBlock, code, lino, _return, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pOrgBlock, code, lino, _return, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_CODE(createOneDataBlock(pOrgBlock, false, &pDstBlock), lino, _return);
QUERY_CHECK_CODE(blockDataEnsureCapacity(pDstBlock, pDataBlock->info.rows), lino, _return);
for (int32_t i = 0; i < taosArrayGetSize(pOrgBlock->pDataBlock); ++i) {
SColumnInfoData* pDst = taosArrayGet(pDstBlock->pDataBlock, i);
SColumnInfoData* pSrc = taosArrayGet(pOrgBlock->pDataBlock, i);
QUERY_CHECK_NULL(pDst, code, lino, _return, terrno);
QUERY_CHECK_NULL(pSrc, code, lino, _return, terrno);
bool found = false;
for (int32_t j = 0; j < taosArrayGetSize(pDataBlock->pDataBlock); j++) {
SColumnInfoData *p = taosArrayGet(pDataBlock->pDataBlock, j);
if (p->info.slotId == pSrc->info.slotId) {
QUERY_CHECK_CODE(colDataAssign(pDst, p, (int32_t)pDataBlock->info.rows, &pDataBlock->info), lino, _return);
found = true;
break;
}
}
if (!found) {
colDataSetNNULL(pDst, 0, pDataBlock->info.rows);
}
}
pDstBlock->info.rows = pDataBlock->info.rows;
pDstBlock->info.capacity = pDataBlock->info.rows;
pDstBlock->info.window = pDataBlock->info.window;
*pResBlock = pDstBlock;
return code;
_return:
uError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
blockDataDestroy(pDstBlock);
return code;
}
int32_t createDataBlock(SSDataBlock** pResBlock) {
QRY_PARAM_CHECK(pResBlock);
SSDataBlock* pBlock = taosMemoryCalloc(1, sizeof(SSDataBlock));

17
source/common/src/tglobal.c
View File

@ -390,6 +390,10 @@ void *pTimezoneNameMap = NULL;
int32_t tsStreamNotifyMessageSize = 8 * 1024; // KB, default 8MB
int32_t tsStreamNotifyFrameSize = 256; // KB, default 256KB
int32_t tsStreamVirtualMergeMaxDelayMs = 10 * 1000; // 10s
int32_t tsStreamVirtualMergeMaxMemKb = 16 * 1024; // 16MB
int32_t tsStreamVirtualMergeWaitMode = 0; // 0 wait forever, 1 wait for max delay, 2 wait for max mem
int32_t taosCheckCfgStrValueLen(const char *name, const char *value, int32_t len);
#define TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, pName) \
@ -1011,6 +1015,10 @@ static int32_t taosAddServerCfg(SConfig *pCfg) {
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "adapterPort", tsAdapterPort, 1, 65056, CFG_SCOPE_SERVER, CFG_DYN_SERVER_LAZY, CFG_CATEGORY_LOCAL));
TAOS_CHECK_RETURN(cfgAddString(pCfg, "adapterToken", tsAdapterToken, CFG_SCOPE_SERVER, CFG_DYN_SERVER_LAZY, CFG_CATEGORY_LOCAL));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "streamVirtualMergeMaxDelay", tsStreamVirtualMergeMaxDelayMs, 500, 10 * 60 * 1000, CFG_SCOPE_SERVER, CFG_DYN_NONE,CFG_CATEGORY_LOCAL));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "streamVirtualMergeMaxMem", tsStreamVirtualMergeMaxMemKb, 8 * 1024, 1 * 1024 * 1024, CFG_SCOPE_SERVER, CFG_DYN_NONE,CFG_CATEGORY_LOCAL));
TAOS_CHECK_RETURN(cfgAddInt32(pCfg, "streamVirtualMergeWaitMode", tsStreamVirtualMergeWaitMode, 0, 2, CFG_SCOPE_SERVER, CFG_DYN_NONE,CFG_CATEGORY_LOCAL));
// clang-format on
// GRANT_CFG_ADD;
@ -1928,6 +1936,15 @@ static int32_t taosSetServerCfg(SConfig *pCfg) {
TAOS_CHECK_RETURN(taosCheckCfgStrValueLen(pItem->name, pItem->str, tListLen(tsAdapterToken)));
tstrncpy(tsAdapterToken, pItem->str, tListLen(tsAdapterToken));
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "streamVirtualMergeMaxDelay");
tsStreamVirtualMergeMaxDelayMs = pItem->i32;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "streamVirtualMergeMaxMem");
tsStreamVirtualMergeMaxMemKb = pItem->i32;
TAOS_CHECK_GET_CFG_ITEM(pCfg, pItem, "streamVirtualMergeWaitMode");
tsStreamVirtualMergeWaitMode = pItem->i32;
// GRANT_CFG_GET;
TAOS_RETURN(TSDB_CODE_SUCCESS);
}

1
source/dnode/mgmt/mgmt_mnode/src/mmHandle.c
View File

@ -188,6 +188,7 @@ SArray *mmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_VND_FETCH_TTL_EXPIRED_TBS_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_DROP_TABLE_RSP, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_RESET_STREAM, mmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_GET_DB_INFO, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_RETRIEVE_ANAL_ALGO, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_MND_RETRIEVE_IP_WHITE, mmPutMsgToReadQueue, 0) == NULL) goto _OVER;

1
source/dnode/mgmt/mgmt_vnode/src/vmHandle.c
View File

@ -966,6 +966,7 @@ SArray *vmGetMsgHandles() {
if (dmSetMgmtHandle(pArray, TDMT_VND_ALTER_TABLE, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_UPDATE_TAG_VAL, vmPutMsgToWriteQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_TABLE_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_VSUBTABLES_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_TABLE_CFG, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_BATCH_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;
if (dmSetMgmtHandle(pArray, TDMT_VND_TABLES_META, vmPutMsgToFetchQueue, 0) == NULL) goto _OVER;

1
source/dnode/mgmt/node_mgmt/src/dmTransport.c
View File

@ -164,6 +164,7 @@ static void dmProcessRpcMsg(SDnode *pDnode, SRpcMsg *pRpc, SEpSet *pEpSet) {
case TDMT_SCH_FETCH_RSP:
case TDMT_SCH_MERGE_FETCH_RSP:
case TDMT_VND_SUBMIT_RSP:
case TDMT_MND_GET_DB_INFO_RSP:
code = qWorkerProcessRspMsg(NULL, NULL, pRpc, 0);
return;
case TDMT_MND_STATUS_RSP:

4
source/dnode/mnode/impl/inc/mndDef.h
View File

@ -593,6 +593,7 @@ typedef struct {
SColCmpr* pCmpr;
int64_t keep;
SExtSchema* pExtSchemas;
int8_t virtualStb;
} SStbObj;
typedef struct {
@ -787,6 +788,7 @@ typedef struct SStreamConf {
int64_t watermark;
} SStreamConf;
typedef struct {
char name[TSDB_STREAM_FNAME_LEN];
SRWLatch lock;
@ -838,6 +840,8 @@ typedef struct {
int8_t subTableWithoutMd5;
char reserve[TSDB_RESERVE_VALUE_LEN];
SSHashObj* pVTableMap; // do not serialize
SQueryPlan* pPlan; // do not serialize
} SStreamObj;
typedef struct SStreamSeq {

2
source/dnode/mnode/impl/inc/mndScheduler.h
View File

@ -27,7 +27,7 @@ int32_t mndSchedInitSubEp(SMnode* pMnode, const SMqTopicObj* pTopic, SMqSubscrib
int32_t mndConvertRsmaTask(char** pDst, int32_t* pDstLen, const char* ast, int64_t uid, int8_t triggerType,
int64_t watermark, int64_t deleteMark);
int32_t mndScheduleStream(SMnode* pMnode, SStreamObj* pStream, int64_t skey, SArray* pVerList);
int32_t mndScheduleStream(SMnode* pMnode, SStreamObj* pStream, SCMCreateStreamReq* pCreate);
#ifdef __cplusplus
}

1
source/dnode/mnode/impl/src/mndDb.c
View File

@ -80,6 +80,7 @@ int32_t mndInitDb(SMnode *pMnode) {
mndSetMsgHandle(pMnode, TDMT_MND_TRIM_DB, mndProcessTrimDbReq);
mndSetMsgHandle(pMnode, TDMT_MND_GET_DB_CFG, mndProcessGetDbCfgReq);
mndSetMsgHandle(pMnode, TDMT_MND_S3MIGRATE_DB, mndProcessS3MigrateDbReq);
mndSetMsgHandle(pMnode, TDMT_MND_GET_DB_INFO, mndProcessUseDbReq);
mndAddShowRetrieveHandle(pMnode, TSDB_MGMT_TABLE_DB, mndRetrieveDbs);
mndAddShowFreeIterHandle(pMnode, TSDB_MGMT_TABLE_DB, mndCancelGetNextDb);

6
source/dnode/mnode/impl/src/mndDef.c
View File

@ -234,6 +234,12 @@ void tFreeStreamObj(SStreamObj *pStream) {
if (pStream->tagSchema.nCols > 0) {
taosMemoryFree(pStream->tagSchema.pSchema);
}
qDestroyQueryPlan(pStream->pPlan);
pStream->pPlan = NULL;
tSimpleHashCleanup(pStream->pVTableMap);
pStream->pVTableMap = NULL;
}
#endif

1
source/dnode/mnode/impl/src/mndDump.c
View File

@ -198,6 +198,7 @@ void dumpStb(SSdb *pSdb, SJson *json) {
RETRIEVE_CHECK_GOTO(tjsonAddStringToObject(item, "commentLen", i642str(pObj->commentLen)), pObj, &lino, _OVER);
RETRIEVE_CHECK_GOTO(tjsonAddStringToObject(item, "ast1Len", i642str(pObj->ast1Len)), pObj, &lino, _OVER);
RETRIEVE_CHECK_GOTO(tjsonAddStringToObject(item, "ast2Len", i642str(pObj->ast2Len)), pObj, &lino, _OVER);
RETRIEVE_CHECK_GOTO(tjsonAddStringToObject(item, "virtual", i642str(pObj->virtualStb)), pObj, &lino, _OVER);
RETRIEVE_CHECK_GOTO(tjsonAddStringToObject(item, "numOfColumns", i642str(pObj->numOfColumns)), pObj, &lino, _OVER);
SJson *columns = tjsonAddArrayToObject(item, "columns");

3
source/dnode/mnode/impl/src/mndInfoSchema.c
View File

@ -47,6 +47,7 @@ static int32_t mndInsInitMeta(SHashObj *hash) {
meta.tableType = TSDB_SYSTEM_TABLE;
meta.sversion = 1;
meta.tversion = 1;
meta.virtualStb = false;
size_t size = 0;
const SSysTableMeta *pInfosTableMeta = NULL;
@ -128,6 +129,7 @@ int32_t mndBuildInsTableCfg(SMnode *pMnode, const char *dbFName, const char *tbN
pRsp->numOfTags = pMeta->numOfTags;
pRsp->numOfColumns = pMeta->numOfColumns;
pRsp->tableType = pMeta->tableType;
pRsp->virtualStb = pMeta->virtualStb;
pRsp->pSchemas = taosMemoryCalloc(pMeta->numOfColumns, sizeof(SSchema));
if (pRsp->pSchemas == NULL) {
@ -139,6 +141,7 @@ int32_t mndBuildInsTableCfg(SMnode *pMnode, const char *dbFName, const char *tbN
memcpy(pRsp->pSchemas, pMeta->pSchemas, pMeta->numOfColumns * sizeof(SSchema));
pRsp->pSchemaExt = taosMemoryCalloc(pMeta->numOfColumns, sizeof(SSchemaExt));
pRsp->pColRefs = taosMemCalloc(pMeta->numOfColumns, sizeof(SColRef));
TAOS_RETURN(code);
}

2
source/dnode/mnode/impl/src/mndPerfSchema.c
View File

@ -46,6 +46,7 @@ int32_t mndPerfsInitMeta(SHashObj *hash) {
meta.tableType = TSDB_SYSTEM_TABLE;
meta.sversion = 1;
meta.tversion = 1;
meta.virtualStb = false;
size_t size = 0;
const SSysTableMeta *pSysDbTableMeta = NULL;
@ -113,6 +114,7 @@ int32_t mndBuildPerfsTableCfg(SMnode *pMnode, const char *dbFName, const char *t
pRsp->numOfTags = pMeta->numOfTags;
pRsp->numOfColumns = pMeta->numOfColumns;
pRsp->tableType = pMeta->tableType;
pRsp->virtualStb = pMeta->virtualStb;
pRsp->pSchemas = taosMemoryCalloc(pMeta->numOfColumns, sizeof(SSchema));
if (pRsp->pSchemas == NULL) {

733
source/dnode/mnode/impl/src/mndScheduler.c
View File

@ -371,10 +371,14 @@ static void haltInitialTaskStatus(SStreamTask* pTask, SSubplan* pPlan, bool isFi
}
static int32_t buildSourceTask(SStreamObj* pStream, SEpSet* pEpset, EStreamTaskType type, bool useTriggerParam,
int8_t hasAggTasks, SStreamTask** pTask) {
int8_t hasAggTasks, SStreamTask** pTask, SArray* pSourceTaskList) {
uint64_t uid = 0;
SArray** pTaskList = NULL;
streamGetUidTaskList(pStream, type, &uid, &pTaskList);
if (pSourceTaskList) {
pTaskList = &pSourceTaskList;
} else {
streamGetUidTaskList(pStream, type, &uid, &pTaskList);
}
int32_t trigger = 0;
if (type == STREAM_RECALCUL_TASK) {
@ -440,10 +444,78 @@ static void setHTasksId(SStreamObj* pStream) {
}
}
static int32_t addSourceTaskVTableOutput(SStreamTask* pTask, SSHashObj* pVgTasks, SSHashObj* pVtables) {
int32_t code = 0;
int32_t lino = 0;
int32_t taskNum = tSimpleHashGetSize(pVgTasks);
int32_t tbNum = tSimpleHashGetSize(pVtables);
SSHashObj *pTaskMap = tSimpleHashInit(taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT));
TSDB_CHECK_NULL(pTaskMap, code, lino, _end, terrno);
pTask->outputInfo.type = TASK_OUTPUT__VTABLE_MAP;
STaskDispatcherVtableMap *pDispatcher = &pTask->outputInfo.vtableMapDispatcher;
pDispatcher->taskInfos = taosArrayInit(taskNum, sizeof(STaskDispatcherFixed));
TSDB_CHECK_NULL(pDispatcher->taskInfos, code, lino, _end, terrno);
pDispatcher->vtableMap = tSimpleHashInit(tbNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT));
TSDB_CHECK_NULL(pDispatcher->vtableMap, code, lino, _end, terrno);
int32_t iter = 0, vgId = 0;
uint64_t uid = 0;
STaskDispatcherFixed* pAddr = NULL;
void* p = NULL;
while (NULL != (p = tSimpleHashIterate(pVtables, p, &iter))) {
char* vgUid = tSimpleHashGetKey(p, NULL);
vgId = *(int32_t*)vgUid;
uid = *(uint64_t*)((int32_t*)vgUid + 1);
pAddr = tSimpleHashGet(pVgTasks, &vgId, sizeof(vgId));
if (NULL == pAddr) {
mError("tSimpleHashGet vgId %d not found", vgId);
return code;
}
void* px = tSimpleHashGet(pTaskMap, &pAddr->taskId, sizeof(int32_t));
int32_t idx = 0;
if (px == NULL) {
px = taosArrayPush(pDispatcher->taskInfos, pAddr);
TSDB_CHECK_NULL(px, code, lino, _end, terrno);
idx = taosArrayGetSize(pDispatcher->taskInfos) - 1;
code = tSimpleHashPut(pTaskMap, &pAddr->taskId, sizeof(int32_t), &idx, sizeof(int32_t));
if (code) {
mError("tSimpleHashPut uid to task idx failed, error:%d", code);
return code;
}
} else {
idx = *(int32_t*)px;
}
code = tSimpleHashPut(pDispatcher->vtableMap, &uid, sizeof(int64_t), &idx, sizeof(int32_t));
if (code) {
mError("tSimpleHashPut uid to STaskDispatcherFixed failed, error:%d", code);
return code;
}
mDebug("source task[%s,vg:%d] add vtable output map, vuid %" PRIu64 " => [%d, vg:%d]",
pTask->id.idStr, pTask->info.nodeId, uid, pAddr->taskId, pAddr->nodeId);
}
_end:
if (code != TSDB_CODE_SUCCESS) {
mError("source task[%s,vg:%d] add vtable output map failed, lino:%d, error:%s", pTask->id.idStr, pTask->info.nodeId,
lino, tstrerror(code));
}
if (pTaskMap != NULL) {
tSimpleHashCleanup(pTaskMap);
}
return code;
}
static int32_t doAddSourceTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream, SEpSet* pEpset, int64_t skey,
SArray* pVerList, SVgObj* pVgroup, EStreamTaskType type, bool useTriggerParam, int8_t hasAggTasks) {
SArray* pVerList, SVgObj* pVgroup, EStreamTaskType type, bool useTriggerParam,
int8_t hasAggTasks, SSHashObj* pVgTasks, SArray* pSourceTaskList) {
SStreamTask* pTask = NULL;
int32_t code = buildSourceTask(pStream, pEpset, type, useTriggerParam, hasAggTasks, &pTask);
int32_t code = buildSourceTask(pStream, pEpset, type, useTriggerParam, hasAggTasks, &pTask, pSourceTaskList);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
@ -461,7 +533,13 @@ static int32_t doAddSourceTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStre
return code;
}
return TDB_CODE_SUCCESS;
mTrace("souce task plan:%s", pTask->exec.qmsg);
if (pVgTasks) {
code = addSourceTaskVTableOutput(pTask, pVgTasks, plan->pVTables);
}
return code;
}
static SSubplan* getScanSubPlan(const SQueryPlan* pPlan) {
@ -480,6 +558,66 @@ static SSubplan* getScanSubPlan(const SQueryPlan* pPlan) {
return plan;
}
static int32_t doAddMergeTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream, SEpSet* pEpset, SVgObj* pVgroup,
bool isHistoryTask, bool useTriggerParam, int8_t hasAggTasks, SArray* pVtables) {
SStreamTask* pTask = NULL;
SArray** pTaskList = taosArrayGetLast(pStream->pTaskList);
int32_t code = tNewStreamTask(pStream->uid, TASK_LEVEL__MERGE, pEpset, isHistoryTask, pStream->conf.trigger,
useTriggerParam ? pStream->conf.triggerParam : 0, *pTaskList, pStream->conf.fillHistory,
pStream->subTableWithoutMd5, hasAggTasks, &pTask);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
int32_t vtbNum = taosArrayGetSize(pVtables);
pTask->pVTables = taosArrayInit(vtbNum, sizeof(SVCTableMergeInfo));
if (NULL == pTask->pVTables) {
code = terrno;
mError("taosArrayInit %d SVCTableMergeInfo failed, error:%d", vtbNum, terrno);
return code;
}
SVCTableMergeInfo tbInfo;
for (int32_t i = 0; i < vtbNum; ++i) {
SVCTableRefCols** pTb = taosArrayGet(pVtables, i);
tbInfo.uid = (*pTb)->uid;
tbInfo.numOfSrcTbls = (*pTb)->numOfSrcTbls;
if (NULL == taosArrayPush(pTask->pVTables, &tbInfo)) {
code = terrno;
mError("taosArrayPush SVCTableMergeInfo failed, error:%d", terrno);
return code;
}
mDebug("merge task[%s, vg:%d] add vtable info: vuid %" PRIu64 ", numOfSrcTbls:%d",
pTask->id.idStr, pVgroup->vgId, tbInfo.uid, tbInfo.numOfSrcTbls);
}
code = mndAssignStreamTaskToVgroup(pMnode, pTask, plan, pVgroup);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
return TDB_CODE_SUCCESS;
}
static SSubplan* getVTbScanSubPlan(const SQueryPlan* pPlan) {
int32_t numOfPlanLevel = LIST_LENGTH(pPlan->pSubplans);
SNodeListNode* inner = (SNodeListNode*)nodesListGetNode(pPlan->pSubplans, numOfPlanLevel - 2);
if (LIST_LENGTH(inner->pNodeList) != 1) {
terrno = TSDB_CODE_QRY_INVALID_INPUT;
return NULL;
}
SSubplan* plan = (SSubplan*)nodesListGetNode(inner->pNodeList, 0);
if (plan->subplanType != SUBPLAN_TYPE_MERGE) {
terrno = TSDB_CODE_QRY_INVALID_INPUT;
return NULL;
}
return plan;
}
static SSubplan* getAggSubPlan(const SQueryPlan* pPlan, int index) {
SNodeListNode* inner = (SNodeListNode*)nodesListGetNode(pPlan->pSubplans, index);
if (LIST_LENGTH(inner->pNodeList) != 1) {
@ -495,6 +633,118 @@ static SSubplan* getAggSubPlan(const SQueryPlan* pPlan, int index) {
return plan;
}
static int32_t addVTableMergeTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream, SEpSet* pEpset,
bool useTriggerParam, bool hasAggTasks, SCMCreateStreamReq* pCreate) {
SVgObj* pVgroup = NULL;
int32_t code = TSDB_CODE_SUCCESS;
int32_t vgNum = taosArrayGetSize(pCreate->pVSubTables);
for (int32_t i = 0; i < vgNum; ++i) {
SVSubTablesRsp* pVg = (SVSubTablesRsp*)taosArrayGet(pCreate->pVSubTables, i);
pVgroup = mndAcquireVgroup(pMnode, pVg->vgId);
if (NULL == pVgroup) {
mWarn("vnode %d in pVSubTables not found", pVg->vgId);
continue;
}
code = doAddMergeTask(pMnode, plan, pStream, pEpset, pVgroup, false, useTriggerParam, hasAggTasks, pVg->pTables);
if (code != 0) {
mError("failed to create stream task, code:%s", tstrerror(code));
mndReleaseVgroup(pMnode, pVgroup);
return code;
}
mndReleaseVgroup(pMnode, pVgroup);
}
return code;
}
static int32_t buildMergeTaskHash(SArray* pMergeTaskList, SSHashObj** ppVgTasks) {
STaskDispatcherFixed addr;
int32_t code = 0;
int32_t taskNum = taosArrayGetSize(pMergeTaskList);
*ppVgTasks = tSimpleHashInit(taskNum, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT));
if (NULL == *ppVgTasks) {
code = terrno;
mError("tSimpleHashInit %d failed", taskNum);
return code;
}
for (int32_t i = 0; i < taskNum; ++i) {
SStreamTask* pTask = taosArrayGetP(pMergeTaskList, i);
addr.taskId = pTask->id.taskId;
addr.nodeId = pTask->info.nodeId;
addr.epSet = pTask->info.epSet;
code = tSimpleHashPut(*ppVgTasks, &addr.nodeId, sizeof(addr.nodeId), &addr, sizeof(addr));
if (code) {
mError("tSimpleHashPut %d STaskDispatcherFixed failed", i);
return code;
}
}
return code;
}
static int32_t addVTableSourceTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream, SEpSet* pEpset,
int64_t nextWindowSkey, SArray* pVerList, bool useTriggerParam, bool hasAggTasks,
SCMCreateStreamReq* pCreate, SSHashObj* pVTableMap, SArray* pSourceTaskList,
SArray* pMergeTaskList) {
int32_t code = 0;
SSHashObj* pVgTasks = NULL;
int32_t vgId = 0;
int32_t iter = 0;
SVgObj* pVgroup = NULL;
void* p = NULL;
code = buildMergeTaskHash(pMergeTaskList, &pVgTasks);
if (code) {
tSimpleHashCleanup(pVgTasks);
return code;
}
while (NULL != (p = tSimpleHashIterate(pVTableMap, p, &iter))) {
char* pDbVg = tSimpleHashGetKey(p, NULL);
char* pVgStr = strrchr(pDbVg, '.');
if (NULL == pVgStr) {
mError("Invalid DbVg string: %s", pDbVg);
tSimpleHashCleanup(pVgTasks);
return TSDB_CODE_MND_INTERNAL_ERROR;
}
(void)taosStr2int32(pVgStr + 1, &vgId);
pVgroup = mndAcquireVgroup(pMnode, vgId);
if (NULL == pVgroup) {
mWarn("vnode %d not found", vgId);
continue;
}
plan->pVTables = *(SSHashObj**)p;
*(SSHashObj**)p = NULL;
code = doAddSourceTask(pMnode, plan, pStream, pEpset, nextWindowSkey, pVerList, pVgroup, false, useTriggerParam,
hasAggTasks, pVgTasks, pSourceTaskList);
if (code != 0) {
mError("failed to create stream task, code:%s", tstrerror(code));
mndReleaseVgroup(pMnode, pVgroup);
tSimpleHashCleanup(pVgTasks);
return code;
}
mndReleaseVgroup(pMnode, pVgroup);
}
tSimpleHashCleanup(pVgTasks);
return TSDB_CODE_SUCCESS;
}
static int32_t addSourceTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream, SEpSet* pEpset,
int64_t nextWindowSkey, SArray* pVerList, bool useTriggerParam, bool hasAggTasks) {
void* pIter = NULL;
@ -517,12 +767,11 @@ static int32_t addSourceTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream
}
code = doAddSourceTask(pMnode, plan, pStream, pEpset, nextWindowSkey, pVerList, pVgroup, STREAM_NORMAL_TASK,
useTriggerParam, hasAggTasks);
useTriggerParam, hasAggTasks, NULL, NULL);
if (code != 0) {
mError("failed to create stream task, code:%s", tstrerror(code));
// todo drop the added source tasks.
sdbRelease(pSdb, pVgroup);
mndReleaseVgroup(pMnode, pVgroup);
return code;
}
@ -535,7 +784,7 @@ static int32_t addSourceTask(SMnode* pMnode, SSubplan* plan, SStreamObj* pStream
}
code = doAddSourceTask(pMnode, plan, pStream, pEpset, nextWindowSkey, pVerList, pVgroup, type,
useTriggerParam, hasAggTasks);
useTriggerParam, hasAggTasks, NULL, NULL);
if (code != 0) {
sdbRelease(pSdb, pVgroup);
return code;
@ -696,6 +945,26 @@ static void bindSourceSink(SStreamObj* pStream, SMnode* pMnode, SArray* tasks, b
}
}
static void bindVtableMergeSink(SStreamObj* pStream, SMnode* pMnode, SArray* tasks, bool hasExtraSink) {
int32_t code = 0;
SArray* pSinkTaskList = taosArrayGetP(tasks, SINK_NODE_LEVEL);
SArray* pMergeTaskList = taosArrayGetP(tasks, hasExtraSink ? SINK_NODE_LEVEL + 2 : SINK_NODE_LEVEL + 1);
for (int i = 0; i < taosArrayGetSize(pMergeTaskList); i++) {
SStreamTask* pMergeTask = taosArrayGetP(pMergeTaskList, i);
mDebug("bindVtableMergeSink taskId:%s to sink task list", pMergeTask->id.idStr);
if (hasExtraSink) {
bindTaskToSinkTask(pStream, pMnode, pSinkTaskList, pMergeTask);
} else {
if ((code = mndSetSinkTaskInfo(pStream, pMergeTask)) != 0) {
mError("failed bind task to sink task since %s", tstrerror(code));
}
}
}
}
static void bindTwoLevel(SArray* tasks, int32_t begin, int32_t end) {
int32_t code = 0;
size_t size = taosArrayGetSize(tasks);
@ -719,13 +988,370 @@ static void bindTwoLevel(SArray* tasks, int32_t begin, int32_t end) {
mDebug("bindTwoLevel task list(%d-%d) to taskId:%s", begin, end - 1, (*(pDownTask))->id.idStr);
}
static int32_t doScheduleStream(SStreamObj* pStream, SMnode* pMnode, SQueryPlan* pPlan, SEpSet* pEpset, int64_t skey,
SArray* pVerList) {
int32_t tableHashValueComp(void const* lp, void const* rp) {
uint32_t* key = (uint32_t*)lp;
SVgroupInfo* pVg = (SVgroupInfo*)rp;
if (*key < pVg->hashBegin) {
return -1;
} else if (*key > pVg->hashEnd) {
return 1;
}
return 0;
}
int dbVgInfoComp(const void* lp, const void* rp) {
SVGroupHashInfo* pLeft = (SVGroupHashInfo*)lp;
SVGroupHashInfo* pRight = (SVGroupHashInfo*)rp;
if (pLeft->hashBegin < pRight->hashBegin) {
return -1;
} else if (pLeft->hashBegin > pRight->hashBegin) {
return 1;
}
return 0;
}
int32_t getTableVgId(SDBVgHashInfo* dbInfo, int32_t acctId, char* dbFName, int32_t* vgId, char *tbName) {
int32_t code = 0;
int32_t lino = 0;
SVgroupInfo* vgInfo = NULL;
char tbFullName[TSDB_TABLE_FNAME_LEN];
(void)snprintf(tbFullName, sizeof(tbFullName), "%s.%s", dbFName, tbName);
uint32_t hashValue = taosGetTbHashVal(tbFullName, (uint32_t)strlen(tbFullName), dbInfo->hashMethod,
dbInfo->hashPrefix, dbInfo->hashSuffix);
if (!dbInfo->vgSorted) {
taosArraySort(dbInfo->vgArray, dbVgInfoComp);
dbInfo->vgSorted = true;
}
vgInfo = taosArraySearch(dbInfo->vgArray, &hashValue, tableHashValueComp, TD_EQ);
if (NULL == vgInfo) {
qError("no hash range found for hash value [%u], dbFName:%s, numOfVgId:%d", hashValue, dbFName,
(int32_t)taosArrayGetSize(dbInfo->vgArray));
return TSDB_CODE_INVALID_PARA;
}
*vgId = vgInfo->vgId;
_return:
return code;
}
static void destroyVSubtableVtb(SSHashObj *pVtable) {
int32_t iter = 0;
void* p = NULL;
while (NULL != (p = tSimpleHashIterate(pVtable, p, &iter))) {
taosArrayDestroy(*(SArray**)p);
}
tSimpleHashCleanup(pVtable);
}
static void destroyVSubtableVgHash(SSHashObj *pVg) {
int32_t iter = 0;
SSHashObj** pVtable = NULL;
void* p = NULL;
while (NULL != (p = tSimpleHashIterate(pVg, p, &iter))) {
pVtable = (SSHashObj**)p;
destroyVSubtableVtb(*pVtable);
}
tSimpleHashCleanup(pVg);
}
static void destroyDbVgroupsHash(SSHashObj *pDbVgs) {
int32_t iter = 0;
SDBVgHashInfo* pVg = NULL;
void* p = NULL;
while (NULL != (p = tSimpleHashIterate(pDbVgs, p, &iter))) {
pVg = (SDBVgHashInfo*)p;
taosArrayDestroy(pVg->vgArray);
}
tSimpleHashCleanup(pDbVgs);
}
static int32_t buildDBVgroupsMap(SMnode* pMnode, SSHashObj* pDbVgroup) {
void* pIter = NULL;
SSdb* pSdb = pMnode->pSdb;
int32_t code = TSDB_CODE_SUCCESS;
char key[TSDB_DB_NAME_LEN + 32];
SArray* pTarget = NULL;
SArray* pNew = NULL;
SDbObj* pDb = NULL;
SDBVgHashInfo dbInfo = {0}, *pDbInfo = NULL;
while (1) {
SVgObj* pVgroup = NULL;
pIter = sdbFetch(pSdb, SDB_VGROUP, pIter, (void**)&pVgroup);
if (pIter == NULL) {
break;
}
pDbInfo = (SDBVgHashInfo*)tSimpleHashGet(pDbVgroup, pVgroup->dbName, strlen(pVgroup->dbName) + 1);
if (NULL == pDbInfo) {
pNew = taosArrayInit(20, sizeof(SVGroupHashInfo));
if (NULL == pNew) {
code = terrno;
mError("taosArrayInit SVGroupHashInfo failed, code:%s", tstrerror(terrno));
sdbRelease(pSdb, pVgroup);
return code;
}
pDb = mndAcquireDb(pMnode, pVgroup->dbName);
if (pDb == NULL) {
code = terrno;
mError("mndAcquireDb %s failed, code:%s", pVgroup->dbName, tstrerror(terrno));
sdbRelease(pSdb, pVgroup);
return code;
}
dbInfo.vgSorted = false;
dbInfo.hashMethod = pDb->cfg.hashMethod;
dbInfo.hashPrefix = pDb->cfg.hashPrefix;
dbInfo.hashSuffix = pDb->cfg.hashSuffix;
dbInfo.vgArray = pNew;
mndReleaseDb(pMnode, pDb);
pTarget = pNew;
} else {
pTarget = pDbInfo->vgArray;
}
SVGroupHashInfo vgInfo = {.vgId = pVgroup->vgId, .hashBegin = pVgroup->hashBegin, .hashEnd = pVgroup->hashEnd};
if (NULL == taosArrayPush(pTarget, &vgInfo)) {
code = terrno;
mError("taosArrayPush SVGroupHashInfo failed, code:%s", tstrerror(terrno));
taosArrayDestroy(pNew);
sdbRelease(pSdb, pVgroup);
return code;
}
if (NULL == pDbInfo) {
code = tSimpleHashPut(pDbVgroup, pVgroup->dbName, strlen(pVgroup->dbName) + 1, &dbInfo, sizeof(dbInfo));
if (code != 0) {
mError("tSimpleHashPut SDBVgHashInfo failed, code:%s", tstrerror(code));
taosArrayDestroy(pNew);
sdbRelease(pSdb, pVgroup);
return code;
}
pNew = NULL;
}
sdbRelease(pSdb, pVgroup);
}
return code;
}
static int32_t addVTableToVnode(SSHashObj* pVg, int32_t vvgId, uint64_t vuid, SRefColInfo* pCol, SStreamVBuildCtx* pCtx) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SSHashObj* pNewVtable = NULL;
SArray* pNewOtable = NULL, *pTarOtable = NULL;
SColIdName col;
char vId[sizeof(int32_t) + sizeof(uint64_t)];
*(int32_t*)vId = vvgId;
*(uint64_t*)((int32_t*)vId + 1) = vuid;
pCtx->lastUid = vuid;
SSHashObj** pVtable = (SSHashObj**)tSimpleHashGet(pVg, vId, sizeof(vId));
if (NULL == pVtable) {
pNewVtable = (SSHashObj*)tSimpleHashInit(0, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY));
TSDB_CHECK_NULL(pNewVtable, code, lino, _return, terrno);
pNewOtable = taosArrayInit(4, sizeof(SColIdName));
TSDB_CHECK_NULL(pNewOtable, code, lino, _return, terrno);
tSimpleHashSetFreeFp(pNewVtable, tFreeStreamVtbOtbInfo);
col.colId = pCol->colId;
col.colName = taosStrdup(pCol->refColName);
TSDB_CHECK_NULL(col.colName, code, lino, _return, terrno);
TSDB_CHECK_NULL(taosArrayPush(pNewOtable, &col), code, lino, _return, terrno);
TSDB_CHECK_CODE(tSimpleHashPut(pNewVtable, pCol->refTableName, strlen(pCol->refTableName) + 1, &pNewOtable, POINTER_BYTES), lino, _return);
TSDB_CHECK_CODE(tSimpleHashPut(pVg, vId, sizeof(vId), &pNewVtable, POINTER_BYTES), lino, _return);
pCtx->lastVtable = pNewVtable;
pCtx->lastOtable = pNewOtable;
return code;
}
SArray** pOtable = tSimpleHashGet(*pVtable, pCol->refTableName, strlen(pCol->refTableName) + 1);
if (NULL == pOtable) {
pNewOtable = taosArrayInit(4, sizeof(SColIdName));
TSDB_CHECK_NULL(pNewOtable, code, lino, _return, terrno);
pTarOtable = pNewOtable;
} else {
pTarOtable = *pOtable;
}
col.colId = pCol->colId;
col.colName = taosStrdup(pCol->refColName);
TSDB_CHECK_NULL(col.colName, code, lino, _return, terrno);
TSDB_CHECK_NULL(taosArrayPush(pTarOtable, &col), code, lino, _return, terrno);
if (NULL == pOtable) {
TSDB_CHECK_CODE(tSimpleHashPut(*pVtable, pCol->refTableName, strlen(pCol->refTableName) + 1, &pNewOtable, POINTER_BYTES), lino, _return);
}
pCtx->lastVtable = *pVtable;
pCtx->lastOtable = pTarOtable;
_return:
if (code) {
mError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
static int32_t addVgroupToRes(char* fDBName, int32_t vvgId, uint64_t vuid, SRefColInfo* pCol, SDBVgHashInfo* pDb, SSHashObj* pRes, SStreamVBuildCtx* pCtx) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
int32_t vgId = 0;
char dbVgId[TSDB_DB_NAME_LEN + 32];
SSHashObj *pTarVg = NULL, *pNewVg = NULL;
TSDB_CHECK_CODE(getTableVgId(pDb, 1, fDBName, &vgId, pCol->refColName), lino, _return);
snprintf(dbVgId, sizeof(dbVgId), "%s.%d", pCol->refDbName, vgId);
SSHashObj** pVg = (SSHashObj**)tSimpleHashGet(pRes, dbVgId, strlen(dbVgId) + 1);
if (NULL == pVg) {
pNewVg = tSimpleHashInit(10, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT));
TSDB_CHECK_NULL(pNewVg, code, lino, _return, terrno);
tSimpleHashSetFreeFp(pNewVg, tFreeStreamVtbVtbInfo);
pTarVg = pNewVg;
} else {
pTarVg = *pVg;
}
TSDB_CHECK_CODE(addVTableToVnode(pTarVg, vvgId, vuid, pCol, pCtx), lino, _return);
if (NULL == pVg) {
TSDB_CHECK_CODE(tSimpleHashPut(pRes, dbVgId, strlen(dbVgId) + 1, &pNewVg, POINTER_BYTES), lino, _return);
pNewVg = NULL;
}
pCtx->lastVg = pTarVg;
_return:
if (code) {
mError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
destroyVSubtableVgHash(pNewVg);
return code;
}
static int32_t addRefColToMap(int32_t vvgId, uint64_t vuid, SRefColInfo* pCol, SSHashObj* pDbVgroups, SSHashObj* pRes, SStreamVBuildCtx* pCtx) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
bool isLastVtable = vuid == pCtx->lastUid;
SSHashObj* currOtable = NULL;
SColIdName col;
char fDBName[TSDB_DB_FNAME_LEN];
if (pCtx->lastCol && pCtx->lastCol->refDbName[0] == pCol->refDbName[0] && pCtx->lastCol->refTableName[0] == pCol->refTableName[0] &&
0 == strcmp(pCtx->lastCol->refDbName, pCol->refDbName) && 0 == strcmp(pCtx->lastCol->refTableName, pCol->refTableName)) {
if (isLastVtable) {
col.colId = pCol->colId;
col.colName = taosStrdup(pCol->refColName);
TSDB_CHECK_NULL(col.colName, code, lino, _return, terrno);
TSDB_CHECK_NULL(taosArrayPush(pCtx->lastOtable, &col), code, lino, _return, terrno);
return code;
}
TSDB_CHECK_CODE(addVTableToVnode(pCtx->lastVg, vvgId, vuid, pCol, pCtx), lino, _return);
return code;
}
snprintf(fDBName, sizeof(fDBName), "1.%s", pCol->refDbName);
SDBVgHashInfo* pDb = (SDBVgHashInfo*)tSimpleHashGet(pDbVgroups, fDBName, strlen(fDBName) + 1);
if (NULL == pDb) {
mError("refDb %s does not exist", pCol->refDbName);
code = TSDB_CODE_MND_DB_NOT_EXIST;
goto _return;
}
TSDB_CHECK_CODE(addVgroupToRes(fDBName, vvgId, vuid, pCol, pDb, pRes, pCtx), lino, _return);
pCtx->lastCol = pCol;
_return:
if (code) {
mError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
return code;
}
static int32_t buildVSubtableMap(SMnode* pMnode, SArray* pVSubTables, SSHashObj** ppRes) {
int32_t code = 0;
int32_t lino = 0;
SSHashObj* pDbVgroups = tSimpleHashInit(20, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY));
if (NULL == pDbVgroups) {
mError("tSimpleHashInit failed, error:%s", tstrerror(terrno));
return terrno;
}
TAOS_CHECK_EXIT(buildDBVgroupsMap(pMnode, pDbVgroups));
*ppRes = tSimpleHashInit(100, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY));
if (NULL == *ppRes) {
code = terrno;
mError("tSimpleHashInit failed, error:%s", tstrerror(terrno));
goto _exit;
}
tSimpleHashSetFreeFp(*ppRes, tFreeStreamVtbDbVgInfo);
SStreamVBuildCtx ctx = {0};
int32_t vgNum = taosArrayGetSize(pVSubTables);
for (int32_t i = 0; i < vgNum; ++i) {
SVSubTablesRsp* pVgTbs = taosArrayGet(pVSubTables, i);
int32_t tbNum = taosArrayGetSize(pVgTbs->pTables);
for (int32_t n = 0; n < tbNum; ++n) {
SVCTableRefCols* pTb = (SVCTableRefCols*)taosArrayGetP(pVgTbs->pTables, n);
for (int32_t m = 0; m < pTb->numOfColRefs; ++m) {
SRefColInfo* pCol = pTb->refCols + m;
TAOS_CHECK_EXIT(addRefColToMap(pVgTbs->vgId, pTb->uid, pCol, pDbVgroups, *ppRes, &ctx));
}
}
}
_exit:
if (code) {
mError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
destroyDbVgroupsHash(pDbVgroups);
return code;
}
static int32_t doScheduleStream(SStreamObj* pStream, SMnode* pMnode, SQueryPlan* pPlan, SEpSet* pEpset, SCMCreateStreamReq* pCreate) {
int32_t code = 0;
bool isVTableStream = (NULL != pCreate->pVSubTables);
int64_t skey = pCreate->lastTs;
SArray* pVerList = pCreate->pVgroupVerList;
SSdb* pSdb = pMnode->pSdb;
int32_t numOfPlanLevel = LIST_LENGTH(pPlan->pSubplans);
bool hasExtraSink = false;
bool externalTargetDB = strcmp(pStream->sourceDb, pStream->targetDb) != 0;
SSubplan* plan = NULL;
SDbObj* pDbObj = mndAcquireDb(pMnode, pStream->targetDb);
if (pDbObj == NULL) {
@ -746,7 +1372,16 @@ static int32_t doScheduleStream(SStreamObj* pStream, SMnode* pMnode, SQueryPlan*
return terrno;
}
if (numOfPlanLevel > 1 || externalTargetDB || multiTarget || pStream->fixedSinkVgId) {
if (pCreate->pVSubTables) {
code = buildVSubtableMap(pMnode, pCreate->pVSubTables, &pStream->pVTableMap);
if (TSDB_CODE_SUCCESS != code) {
mError("failed to buildVSubtableMap, code:%s", tstrerror(terrno));
return code;
}
}
if ((numOfPlanLevel > 1 && !isVTableStream) || (numOfPlanLevel > 2 && isVTableStream) || externalTargetDB ||
multiTarget || pStream->fixedSinkVgId) {
// add extra sink
hasExtraSink = true;
code = addSinkTask(pMnode, pStream, pEpset);
@ -757,20 +1392,57 @@ static int32_t doScheduleStream(SStreamObj* pStream, SMnode* pMnode, SQueryPlan*
pStream->totalLevel = numOfPlanLevel + hasExtraSink;
SSubplan* plan = getScanSubPlan(pPlan); // source plan
if (plan == NULL) {
code = TSDB_CODE_MND_RETURN_VALUE_NULL;
if (terrno != 0) code = terrno;
TAOS_RETURN(code);
}
int8_t hasAggTasks = (numOfPlanLevel > 1) ? 1 : 0; // task level is greater than 1, which means agg existing
code = addSourceTask(pMnode, plan, pStream, pEpset, skey, pVerList, (numOfPlanLevel == 1), hasAggTasks);
if (pStream->pVTableMap) {
code = addNewTaskList(pStream);
if (code) {
return code;
}
plan = getVTbScanSubPlan(pPlan);
if (plan == NULL) {
mError("fail to get vtable scan plan");
code = TSDB_CODE_MND_RETURN_VALUE_NULL;
if (terrno != 0) code = terrno;
TAOS_RETURN(code);
}
SArray** pSourceTaskList = taosArrayGetLast(pStream->pTaskList);
code = addNewTaskList(pStream);
if (code) {
return code;
}
code = addVTableMergeTask(pMnode, plan, pStream, pEpset, (numOfPlanLevel == 1), hasAggTasks, pCreate);
if (code) {
return code;
}
plan = getScanSubPlan(pPlan); // source plan
if (plan == NULL) {
code = TSDB_CODE_MND_RETURN_VALUE_NULL;
if (terrno != 0) code = terrno;
TAOS_RETURN(code);
}
SArray** pMergeTaskList = taosArrayGetLast(pStream->pTaskList);
code = addVTableSourceTask(pMnode, plan, pStream, pEpset, skey, pVerList, (numOfPlanLevel == 1), hasAggTasks,
pCreate, pStream->pVTableMap, *pSourceTaskList, *pMergeTaskList);
} else {
plan = getScanSubPlan(pPlan); // source plan
if (plan == NULL) {
code = TSDB_CODE_MND_RETURN_VALUE_NULL;
if (terrno != 0) code = terrno;
TAOS_RETURN(code);
}
code = addSourceTask(pMnode, plan, pStream, pEpset, skey, pVerList, (numOfPlanLevel == 1), hasAggTasks);
}
if (code != TSDB_CODE_SUCCESS) {
return code;
}
if (numOfPlanLevel == 1) {
if ((numOfPlanLevel == 1 && !isVTableStream)) {
bindSourceSink(pStream, pMnode, pStream->pTaskList, hasExtraSink);
if (needHistoryTask(pStream)) {
bindSourceSink(pStream, pMnode, pStream->pHTaskList, hasExtraSink);
@ -778,8 +1450,14 @@ static int32_t doScheduleStream(SStreamObj* pStream, SMnode* pMnode, SQueryPlan*
return TDB_CODE_SUCCESS;
}
if (numOfPlanLevel == 3) {
plan = getAggSubPlan(pPlan, 1); // middle agg plan
if (numOfPlanLevel == 2 && isVTableStream) {
bindVtableMergeSink(pStream, pMnode, pStream->pTaskList, hasExtraSink);
return TDB_CODE_SUCCESS;
}
if ((numOfPlanLevel == 3 && !isVTableStream) || (numOfPlanLevel == 4 && isVTableStream)) {
int32_t idx = isVTableStream ? 2 : 1;
plan = getAggSubPlan(pPlan, idx); // middle agg plan
if (plan == NULL) {
code = TSDB_CODE_MND_RETURN_VALUE_NULL;
if (terrno != 0) code = terrno;
@ -844,10 +1522,10 @@ static int32_t doScheduleStream(SStreamObj* pStream, SMnode* pMnode, SQueryPlan*
TAOS_RETURN(code);
}
int32_t mndScheduleStream(SMnode* pMnode, SStreamObj* pStream, int64_t skey, SArray* pVgVerList) {
int32_t mndScheduleStream(SMnode* pMnode, SStreamObj* pStream, SCMCreateStreamReq* pCreate) {
int32_t code = 0;
SQueryPlan* pPlan = qStringToQueryPlan(pStream->physicalPlan);
if (pPlan == NULL) {
pStream->pPlan = qStringToQueryPlan(pStream->physicalPlan);
if (pStream->pPlan == NULL) {
code = TSDB_CODE_QRY_INVALID_INPUT;
TAOS_RETURN(code);
}
@ -855,8 +1533,7 @@ int32_t mndScheduleStream(SMnode* pMnode, SStreamObj* pStream, int64_t skey, SAr
SEpSet mnodeEpset = {0};
mndGetMnodeEpSet(pMnode, &mnodeEpset);
code = doScheduleStream(pStream, pMnode, pPlan, &mnodeEpset, skey, pVgVerList);
qDestroyQueryPlan(pPlan);
code = doScheduleStream(pStream, pMnode, pStream->pPlan, &mnodeEpset, pCreate);
TAOS_RETURN(code);
}

6
source/dnode/mnode/impl/src/mndSma.c
View File

@ -89,7 +89,7 @@ int32_t mndInitSma(SMnode *pMnode) {
.deleteFp = (SdbDeleteFp)mndSmaActionDelete,
};
mndSetMsgHandle(pMnode, TDMT_MND_CREATE_SMA, mndProcessCreateSmaReq);
// mndSetMsgHandle(pMnode, TDMT_MND_CREATE_SMA, mndProcessCreateSmaReq);
mndSetMsgHandle(pMnode, TDMT_MND_DROP_SMA, mndProcessDropIdxReq);
mndSetMsgHandle(pMnode, TDMT_VND_CREATE_SMA_RSP, mndTransProcessRsp);
mndSetMsgHandle(pMnode, TDMT_VND_DROP_SMA_RSP, mndTransProcessRsp);
@ -590,6 +590,7 @@ static void mndDestroySmaObj(SSmaObj *pSmaObj) {
}
}
#if 0
static int32_t mndCreateSma(SMnode *pMnode, SRpcMsg *pReq, SMCreateSmaReq *pCreate, SDbObj *pDb, SStbObj *pStb,
const char *streamName) {
int32_t code = 0;
@ -760,6 +761,7 @@ _OVER:
mndTransDrop(pTrans);
TAOS_RETURN(code);
}
#endif
static int32_t mndCheckCreateSmaReq(SMCreateSmaReq *pCreate) {
int32_t code = TSDB_CODE_MND_INVALID_SMA_OPTION;
@ -799,6 +801,7 @@ static int32_t mndGetStreamNameFromSmaName(char *streamName, char *smaName) {
return TSDB_CODE_SUCCESS;
}
#if 0
static int32_t mndProcessCreateSmaReq(SRpcMsg *pReq) {
SMnode *pMnode = pReq->info.node;
int32_t code = -1;
@ -889,6 +892,7 @@ _OVER:
TAOS_RETURN(code);
}
#endif
static int32_t mndSetDropSmaRedoLogs(SMnode *pMnode, STrans *pTrans, SSmaObj *pSma) {
int32_t code = 0;

33
source/dnode/mnode/impl/src/mndStb.c
View File

@ -33,7 +33,7 @@
#include "mndVgroup.h"
#include "tname.h"
#define STB_VER_NUMBER 2
#define STB_VER_NUMBER 3
#define STB_RESERVE_SIZE 56
static SSdbRow *mndStbActionDecode(SSdbRaw *pRaw);
@ -194,13 +194,13 @@ SSdbRaw *mndStbActionEncode(SStbObj *pStb) {
}
SDB_SET_INT64(pRaw, dataPos, pStb->keep, _OVER)
if (hasTypeMod) {
for (int32_t i = 0; i < pStb->numOfColumns; ++i) {
SDB_SET_INT32(pRaw, dataPos, pStb->pExtSchemas[i].typeMod, _OVER);
}
}
SDB_SET_INT8(pRaw, dataPos, pStb->virtualStb, _OVER)
SDB_SET_RESERVE(pRaw, dataPos, STB_RESERVE_SIZE, _OVER)
SDB_SET_DATALEN(pRaw, dataPos, _OVER)
@ -313,7 +313,7 @@ static SSdbRow *mndStbActionDecode(SSdbRaw *pRaw) {
}
pStb->pCmpr = taosMemoryCalloc(pStb->numOfColumns, sizeof(SColCmpr));
if (sver < STB_VER_NUMBER) {
if (sver < STB_VER_NUMBER - 1) {
// compatible with old data, setup default compress value
// impl later
for (int i = 0; i < pStb->numOfColumns; i++) {
@ -341,6 +341,12 @@ static SSdbRow *mndStbActionDecode(SSdbRaw *pRaw) {
}
}
if (sver < STB_VER_NUMBER) {
pStb->virtualStb = 0;
} else {
SDB_GET_INT8(pRaw, dataPos, &pStb->virtualStb, _OVER)
}
SDB_GET_RESERVE(pRaw, dataPos, STB_RESERVE_SIZE, _OVER)
terrno = 0;
@ -544,6 +550,7 @@ void *mndBuildVCreateStbReq(SMnode *pMnode, SVgObj *pVgroup, SStbObj *pStb, int3
req.alterOriData = alterOriData;
req.alterOriDataLen = alterOriDataLen;
req.source = pStb->source;
req.virtualStb = pStb->virtualStb;
// todo
req.schemaRow.nCols = pStb->numOfColumns;
req.schemaRow.version = pStb->colVer;
@ -674,6 +681,11 @@ int32_t mndCheckCreateStbReq(SMCreateStbReq *pCreate) {
TAOS_RETURN(code);
}
if (pCreate->virtualStb != 0 && pCreate->virtualStb != 1) {
code = TSDB_CODE_MND_INVALID_STB_OPTION;
TAOS_RETURN(code);
}
if (pCreate->numOfColumns < TSDB_MIN_COLUMNS || pCreate->numOfTags + pCreate->numOfColumns > TSDB_MAX_COLUMNS) {
code = TSDB_CODE_PAR_INVALID_COLUMNS_NUM;
TAOS_RETURN(code);
@ -909,6 +921,7 @@ int32_t mndBuildStbFromReq(SMnode *pMnode, SStbObj *pDst, SMCreateStbReq *pCreat
pDst->pFuncs = pCreate->pFuncs;
pDst->source = pCreate->source;
pDst->keep = pCreate->keep;
pDst->virtualStb = pCreate->virtualStb;
pCreate->pFuncs = NULL;
if (pDst->commentLen > 0) {
@ -2223,6 +2236,7 @@ static int32_t mndSetAlterStbRedoActions2(SMnode *pMnode, STrans *pTrans, SDbObj
TAOS_RETURN(code);
}
static int32_t mndBuildStbSchemaImp(SDbObj *pDb, SStbObj *pStb, const char *tbName, STableMetaRsp *pRsp) {
int32_t code = 0;
taosRLockLatch(&pStb->lock);
@ -2240,7 +2254,8 @@ static int32_t mndBuildStbSchemaImp(SDbObj *pDb, SStbObj *pStb, const char *tbNa
code = terrno;
TAOS_RETURN(code);
}
pRsp->numOfColRefs = 0;
pRsp->pColRefs = NULL;
tstrncpy(pRsp->dbFName, pStb->db, sizeof(pRsp->dbFName));
tstrncpy(pRsp->tbName, tbName, sizeof(pRsp->tbName));
tstrncpy(pRsp->stbName, tbName, sizeof(pRsp->stbName));
@ -2253,6 +2268,7 @@ static int32_t mndBuildStbSchemaImp(SDbObj *pDb, SStbObj *pStb, const char *tbNa
pRsp->tversion = pStb->tagVer;
pRsp->suid = pStb->uid;
pRsp->tuid = pStb->uid;
pRsp->virtualStb = pStb->virtualStb;
for (int32_t i = 0; i < pStb->numOfColumns; ++i) {
SSchema *pSchema = &pRsp->pSchemas[i];
@ -2350,6 +2366,8 @@ static int32_t mndBuildStbCfgImp(SDbObj *pDb, SStbObj *pStb, const char *tbName,
pSchExt->typeMod = pStb->pExtSchemas[i].typeMod;
}
}
pRsp->virtualStb = pStb->virtualStb;
pRsp->pColRefs = NULL;
taosRUnLockLatch(&pStb->lock);
TAOS_RETURN(code);
@ -3493,6 +3511,10 @@ static int32_t mndRetrieveStb(SRpcMsg *pReq, SShowObj *pShow, SSDataBlock *pBloc
RETRIEVE_CHECK_GOTO(colDataSetVal(pColInfo, numOfRows, (const char *)(&pStb->uid), false), pStb, &lino, _ERROR);
}
pColInfo = taosArrayGet(pBlock->pDataBlock, cols++);
if (pColInfo) {
RETRIEVE_CHECK_GOTO(colDataSetVal(pColInfo, numOfRows, (const char *)(&pStb->virtualStb), false), pStb, &lino, _ERROR);
}
numOfRows++;
sdbRelease(pSdb, pStb);
}
@ -3573,7 +3595,7 @@ static int32_t buildDbColsInfoBlock(const SSDataBlock *p, const SSysTableMeta *p
pColInfoData = taosArrayGet(p->pDataBlock, 5);
TAOS_CHECK_GOTO(colDataSetVal(pColInfoData, numOfRows, (const char *)&pm->schema[j].bytes, false), &lino, _OVER);
for (int32_t k = 6; k <= 8; ++k) {
for (int32_t k = 6; k <= 9; ++k) {
pColInfoData = taosArrayGet(p->pDataBlock, k);
colDataSetNULL(pColInfoData, numOfRows);
}
@ -3581,6 +3603,7 @@ static int32_t buildDbColsInfoBlock(const SSDataBlock *p, const SSysTableMeta *p
numOfRows += 1;
}
}
return numOfRows;
_OVER:
mError("failed at %s:%d since %s", __FUNCTION__, lino, tstrerror(code));
return numOfRows;

5
source/dnode/mnode/impl/src/mndStream.c
View File

@ -1001,15 +1001,18 @@ static int32_t mndProcessCreateStreamReq(SRpcMsg *pReq) {
// schedule stream task for stream obj
if (!buildEmptyStream) {
code = mndScheduleStream(pMnode, &streamObj, createReq.lastTs, createReq.pVgroupVerList);
code = mndScheduleStream(pMnode, &streamObj, &createReq);
if (code != TSDB_CODE_SUCCESS && code != TSDB_CODE_ACTION_IN_PROGRESS) {
mError("stream:%s, failed to schedule since %s", createReq.name, tstrerror(code));
mndTransDrop(pTrans);
goto _OVER;
}
// add notify info into all stream tasks
code = addStreamNotifyInfo(&createReq, &streamObj);
if (code != TSDB_CODE_SUCCESS) {
mError("stream:%s failed to add stream notify info since %s", createReq.name, tstrerror(code));
mndTransDrop(pTrans);
goto _OVER;
}

32
source/dnode/vnode/inc/vnode.h
View File

@ -207,6 +207,21 @@ typedef struct SIdInfo {
int32_t index;
} SIdInfo;
typedef struct SVTColInfo {
int32_t vColId; // column id of virtual table
int32_t pColId; // column id of physical table
int64_t pTbUid; // uid of physical table
} SVTColInfo;
typedef struct SVTSourceScanInfo {
SHashObj *pVirtualTables; // source column info of each vtable column. key: vtUid, value: SArray<SVTColInfo>*
SHashObj *pPhysicalTables; // set of vtables for each ptable. Key: ptUid, value: SArray<vtUid>*
SLRUCache *pPhyTblSchemaCache; // cache for physical table schema
int32_t nextVirtualTableIdx; // index in the value of pVirtualTables
uint64_t metaFetch;
uint64_t cacheHit;
} SVTSourceScanInfo;
typedef struct STqReader {
SPackedData msg;
SSubmitReq2 submit;
@ -224,6 +239,7 @@ typedef struct STqReader {
int64_t lastTs;
bool hasPrimaryKey;
SExtSchema *extSchema;
SVTSourceScanInfo vtSourceScanInfo;
} STqReader;
STqReader *tqReaderOpen(SVnode *pVnode);
@ -232,8 +248,8 @@ void tqReaderClose(STqReader *);
bool tqGetTablePrimaryKey(STqReader *pReader);
void tqSetTablePrimaryKey(STqReader *pReader, int64_t uid);
void tqReaderSetColIdList(STqReader *pReader, SArray *pColIdList);
void tqReaderSetTbUidList(STqReader *pReader, const SArray *tbUidList, const char *id);
int32_t tqReaderSetColIdList(STqReader *pReader, SArray *pColIdList, const char *id);
int32_t tqReaderSetTbUidList(STqReader *pReader, const SArray *tbUidList, const char *id);
void tqReaderAddTbUidList(STqReader *pReader, const SArray *pTableUidList);
void tqReaderRemoveTbUidList(STqReader *pReader, const SArray *tbUidList);
@ -255,6 +271,12 @@ int32_t tqRetrieveDataBlock(STqReader *pReader, SSDataBlock **pRes, const char *
int32_t tqRetrieveTaosxBlock(STqReader *pReader, SMqDataRsp* pRsp, SArray *blocks, SArray *schemas, SSubmitTbData **pSubmitTbDataRet, SArray* rawList, int8_t fetchMeta);
int32_t tqGetStreamExecInfo(SVnode *pVnode, int64_t streamId, int64_t *pDelay, bool *fhFinished);
int32_t tqReaderSetVtableInfo(STqReader *pReader, void *vnode, void *pAPI, SSHashObj *pVtableInfos,
SSDataBlock **ppResBlock, const char *idstr);
int32_t tqRetrieveVTableDataBlock(STqReader *pReader, SSDataBlock **pRes, const char *idstr);
bool tqNextVTableSourceBlockImpl(STqReader *pReader, const char *idstr);
bool tqReaderIsQueriedSourceTable(STqReader *pReader, uint64_t uid);
// sma
int32_t smaGetTSmaDays(SVnodeCfg *pCfg, void *pCont, uint32_t contLen, int32_t *days);
@ -298,6 +320,8 @@ typedef struct {
int64_t numOfSTables;
int64_t numOfCTables;
int64_t numOfNTables;
int64_t numOfVTables;
int64_t numOfVCTables;
int64_t numOfReportedTimeSeries;
int64_t numOfNTimeSeries;
int64_t numOfTimeSeries;
@ -348,6 +372,10 @@ struct SVnodeCfg {
#define TABLE_IS_COL_COMPRESSED(FLG) (((FLG) & (TABLE_COL_COMPRESSED)) != 0)
#define TABLE_SET_COL_COMPRESSED(FLG) ((FLG) |= TABLE_COL_COMPRESSED)
#define TABLE_VIRTUAL ((int8_t)0x4)
#define TABLE_IS_VIRTUAL(FLG) (((FLG) & (TABLE_VIRTUAL)) != 0)
#define TABLE_SET_VIRTUAL(FLG) ((FLG) |= TABLE_VIRTUAL)
struct SFileSetReader;
int32_t tsdbFileSetReaderOpen(void *pVnode, struct SFileSetReader **ppReader);
int32_t tsdbFileSetReaderNext(struct SFileSetReader *pReader);

1
source/dnode/vnode/src/inc/meta.h
View File

@ -165,6 +165,7 @@ int32_t metaFilterTableName(void* pVnode, SMetaFltParam* param, SArray* pUids);
int32_t metaFilterTtl(void* pVnode, SMetaFltParam* param, SArray* pUids);
int32_t metaGetColCmpr(SMeta* pMeta, tb_uid_t uid, SHashObj** colCmprObj);
int32_t updataTableColRef(SColRefWrapper *pWp, const SSchema *pSchema, int8_t add, SColRef *pColRef);
#if !defined(META_REFACT) && !defined(TD_ASTRA)
// SMetaDB
int metaOpenDB(SMeta* pMeta);

1
source/dnode/vnode/src/inc/vnd.h
View File

@ -127,6 +127,7 @@ void vnodeQueryClose(SVnode* pVnode);
int32_t vnodeGetTableMeta(SVnode* pVnode, SRpcMsg* pMsg, bool direct);
int vnodeGetTableCfg(SVnode* pVnode, SRpcMsg* pMsg, bool direct);
int32_t vnodeGetBatchMeta(SVnode* pVnode, SRpcMsg* pMsg);
int32_t vnodeGetVSubtablesMeta(SVnode *pVnode, SRpcMsg *pMsg);
int32_t vnodeGetStreamProgress(SVnode* pVnode, SRpcMsg* pMsg, bool direct);
// vnodeCommit.c

134
source/dnode/vnode/src/meta/metaEntry.c
View File

@ -44,6 +44,25 @@ static int32_t metaEncodeExtSchema(SEncoder* pCoder, const SMetaEntry* pME) {
return 0;
}
int meteEncodeColRefEntry(SEncoder *pCoder, const SMetaEntry *pME) {
const SColRefWrapper *pw = &pME->colRef;
TAOS_CHECK_RETURN(tEncodeI32v(pCoder, pw->nCols));
TAOS_CHECK_RETURN(tEncodeI32v(pCoder, pw->version));
uDebug("encode cols:%d", pw->nCols);
for (int32_t i = 0; i < pw->nCols; i++) {
SColRef *p = &pw->pColRef[i];
TAOS_CHECK_RETURN(tEncodeI8(pCoder, p->hasRef));
TAOS_CHECK_RETURN(tEncodeI16v(pCoder, p->id));
if (p->hasRef) {
TAOS_CHECK_RETURN(tEncodeCStr(pCoder, p->refDbName));
TAOS_CHECK_RETURN(tEncodeCStr(pCoder, p->refTableName));
TAOS_CHECK_RETURN(tEncodeCStr(pCoder, p->refColName));
}
}
return 0;
}
static int32_t metaDecodeExtSchemas(SDecoder* pDecoder, SMetaEntry* pME) {
bool hasExtSchema = false;
SSchemaWrapper* pSchWrapper = NULL;
@ -83,8 +102,8 @@ SExtSchema* metaGetSExtSchema(const SMetaEntry *pME) {
hasTypeMods = schemasHasTypeMod(pSchWrapper->pSchema, pSchWrapper->nCols);
if (hasTypeMods) {
SExtSchema* ret = taosMemoryMalloc(sizeof(SExtSchema) * pSchWrapper->nCols);
if (ret != NULL){
SExtSchema *ret = taosMemoryMalloc(sizeof(SExtSchema) * pSchWrapper->nCols);
if (ret != NULL) {
memcpy(ret, pME->pExtSchemas, pSchWrapper->nCols * sizeof(SExtSchema));
}
return ret;
@ -92,6 +111,62 @@ SExtSchema* metaGetSExtSchema(const SMetaEntry *pME) {
return NULL;
}
int meteDecodeColRefEntry(SDecoder *pDecoder, SMetaEntry *pME) {
SColRefWrapper *pWrapper = &pME->colRef;
TAOS_CHECK_RETURN(tDecodeI32v(pDecoder, &pWrapper->nCols));
if (pWrapper->nCols == 0) {
return 0;
}
TAOS_CHECK_RETURN(tDecodeI32v(pDecoder, &pWrapper->version));
uDebug("decode cols:%d", pWrapper->nCols);
pWrapper->pColRef = (SColRef *)tDecoderMalloc(pDecoder, pWrapper->nCols * sizeof(SColRef));
if (pWrapper->pColRef == NULL) {
return terrno;
}
for (int i = 0; i < pWrapper->nCols; i++) {
SColRef *p = &pWrapper->pColRef[i];
TAOS_CHECK_RETURN(tDecodeI8(pDecoder, (int8_t *)&p->hasRef));
TAOS_CHECK_RETURN(tDecodeI16v(pDecoder, &p->id));
if (p->hasRef) {
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, p->refDbName));
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, p->refTableName));
TAOS_CHECK_RETURN(tDecodeCStrTo(pDecoder, p->refColName));
}
}
return 0;
}
static FORCE_INLINE int32_t metatInitDefaultSColRefWrapper(SDecoder *pDecoder, SColRefWrapper *pRef,
SSchemaWrapper *pSchema) {
pRef->nCols = pSchema->nCols;
if ((pRef->pColRef = (SColRef *)tDecoderMalloc(pDecoder, pRef->nCols * sizeof(SColRef))) == NULL) {
return terrno;
}
for (int32_t i = 0; i < pRef->nCols; i++) {
SColRef *pColRef = &pRef->pColRef[i];
SSchema *pColSchema = &pSchema->pSchema[i];
pColRef->id = pColSchema->colId;
pColRef->hasRef = false;
}
return 0;
}
static int32_t metaCloneColRef(const SColRefWrapper*pSrc, SColRefWrapper *pDst) {
if (pSrc->nCols > 0) {
pDst->nCols = pSrc->nCols;
pDst->version = pSrc->version;
pDst->pColRef = (SColRef*)taosMemoryCalloc(pSrc->nCols, sizeof(SColRef));
if (NULL == pDst->pColRef) {
return terrno;
}
memcpy(pDst->pColRef, pSrc->pColRef, pSrc->nCols * sizeof(SColRef));
}
return 0;
}
int meteEncodeColCmprEntry(SEncoder *pCoder, const SMetaEntry *pME) {
const SColCmprWrapper *pw = &pME->colCmpr;
TAOS_CHECK_RETURN(tEncodeI32v(pCoder, pw->nCols));
@ -155,6 +230,12 @@ static int32_t metaCloneColCmpr(const SColCmprWrapper *pSrc, SColCmprWrapper *pD
return 0;
}
static void metaCloneColRefFree(SColRefWrapper *pColRef) {
if (pColRef) {
taosMemoryFreeClear(pColRef->pColRef);
}
}
static void metaCloneColCmprFree(SColCmprWrapper *pCmpr) {
if (pCmpr) {
taosMemoryFreeClear(pCmpr->pColCmpr);
@ -181,7 +262,7 @@ int metaEncodeEntry(SEncoder *pCoder, const SMetaEntry *pME) {
if (TABLE_IS_ROLLUP(pME->flags)) {
TAOS_CHECK_RETURN(tEncodeSRSmaParam(pCoder, &pME->stbEntry.rsmaParam));
}
} else if (pME->type == TSDB_CHILD_TABLE) {
} else if (pME->type == TSDB_CHILD_TABLE || pME->type == TSDB_VIRTUAL_CHILD_TABLE) {
TAOS_CHECK_RETURN(tEncodeI64(pCoder, pME->ctbEntry.btime));
TAOS_CHECK_RETURN(tEncodeI32(pCoder, pME->ctbEntry.ttlDays));
TAOS_CHECK_RETURN(tEncodeI32v(pCoder, pME->ctbEntry.commentLen));
@ -190,7 +271,7 @@ int metaEncodeEntry(SEncoder *pCoder, const SMetaEntry *pME) {
}
TAOS_CHECK_RETURN(tEncodeI64(pCoder, pME->ctbEntry.suid));
TAOS_CHECK_RETURN(tEncodeTag(pCoder, (const STag *)pME->ctbEntry.pTags));
} else if (pME->type == TSDB_NORMAL_TABLE) {
} else if (pME->type == TSDB_NORMAL_TABLE || pME->type == TSDB_VIRTUAL_NORMAL_TABLE) {
TAOS_CHECK_RETURN(tEncodeI64(pCoder, pME->ntbEntry.btime));
TAOS_CHECK_RETURN(tEncodeI32(pCoder, pME->ntbEntry.ttlDays));
TAOS_CHECK_RETURN(tEncodeI32v(pCoder, pME->ntbEntry.commentLen));
@ -205,7 +286,11 @@ int metaEncodeEntry(SEncoder *pCoder, const SMetaEntry *pME) {
metaError("meta/entry: invalide table type: %" PRId8 " encode failed.", pME->type);
return TSDB_CODE_INVALID_PARA;
}
TAOS_CHECK_RETURN(meteEncodeColCmprEntry(pCoder, pME));
if (pME->type == TSDB_VIRTUAL_NORMAL_TABLE || pME->type == TSDB_VIRTUAL_CHILD_TABLE) {
TAOS_CHECK_RETURN(meteEncodeColRefEntry(pCoder, pME));
} else {
TAOS_CHECK_RETURN(meteEncodeColCmprEntry(pCoder, pME));
}
TAOS_CHECK_RETURN(metaEncodeExtSchema(pCoder, pME));
}
if (pME->type == TSDB_SUPER_TABLE) {
@ -237,7 +322,7 @@ int metaDecodeEntryImpl(SDecoder *pCoder, SMetaEntry *pME, bool headerOnly) {
if (TABLE_IS_ROLLUP(pME->flags)) {
TAOS_CHECK_RETURN(tDecodeSRSmaParam(pCoder, &pME->stbEntry.rsmaParam));
}
} else if (pME->type == TSDB_CHILD_TABLE) {
} else if (pME->type == TSDB_CHILD_TABLE || pME->type == TSDB_VIRTUAL_CHILD_TABLE) {
TAOS_CHECK_RETURN(tDecodeI64(pCoder, &pME->ctbEntry.btime));
TAOS_CHECK_RETURN(tDecodeI32(pCoder, &pME->ctbEntry.ttlDays));
TAOS_CHECK_RETURN(tDecodeI32v(pCoder, &pME->ctbEntry.commentLen));
@ -246,7 +331,7 @@ int metaDecodeEntryImpl(SDecoder *pCoder, SMetaEntry *pME, bool headerOnly) {
}
TAOS_CHECK_RETURN(tDecodeI64(pCoder, &pME->ctbEntry.suid));
TAOS_CHECK_RETURN(tDecodeTag(pCoder, (STag **)&pME->ctbEntry.pTags));
} else if (pME->type == TSDB_NORMAL_TABLE) {
} else if (pME->type == TSDB_NORMAL_TABLE || pME->type == TSDB_VIRTUAL_NORMAL_TABLE) {
TAOS_CHECK_RETURN(tDecodeI64(pCoder, &pME->ntbEntry.btime));
TAOS_CHECK_RETURN(tDecodeI32(pCoder, &pME->ntbEntry.ttlDays));
TAOS_CHECK_RETURN(tDecodeI32v(pCoder, &pME->ntbEntry.commentLen));
@ -288,6 +373,16 @@ int metaDecodeEntryImpl(SDecoder *pCoder, SMetaEntry *pME, bool headerOnly) {
TAOS_CHECK_RETURN(metatInitDefaultSColCmprWrapper(pCoder, &pME->colCmpr, &pME->ntbEntry.schemaRow));
}
TABLE_SET_COL_COMPRESSED(pME->flags);
} else if (pME->type == TSDB_VIRTUAL_NORMAL_TABLE || pME->type == TSDB_VIRTUAL_CHILD_TABLE) {
if (!tDecodeIsEnd(pCoder)) {
uDebug("set type: %d, tableName:%s", pME->type, pME->name);
TAOS_CHECK_RETURN(meteDecodeColRefEntry(pCoder, pME));
} else {
uDebug("set default type: %d, tableName:%s", pME->type, pME->name);
if (pME->type == TSDB_VIRTUAL_NORMAL_TABLE) {
TAOS_CHECK_RETURN(metatInitDefaultSColRefWrapper(pCoder, &pME->colRef, &pME->ntbEntry.schemaRow));
}
}
}
if (!tDecodeIsEnd(pCoder)) {
TAOS_CHECK_RETURN(metaDecodeExtSchemas(pCoder, pME));
@ -342,10 +437,10 @@ void metaCloneEntryFree(SMetaEntry **ppEntry) {
if (TSDB_SUPER_TABLE == (*ppEntry)->type) {
metaCloneSchemaFree(&(*ppEntry)->stbEntry.schemaRow);
metaCloneSchemaFree(&(*ppEntry)->stbEntry.schemaTag);
} else if (TSDB_CHILD_TABLE == (*ppEntry)->type) {
} else if (TSDB_CHILD_TABLE == (*ppEntry)->type || TSDB_VIRTUAL_CHILD_TABLE == (*ppEntry)->type) {
taosMemoryFreeClear((*ppEntry)->ctbEntry.comment);
taosMemoryFreeClear((*ppEntry)->ctbEntry.pTags);
} else if (TSDB_NORMAL_TABLE == (*ppEntry)->type) {
} else if (TSDB_NORMAL_TABLE == (*ppEntry)->type || TSDB_VIRTUAL_NORMAL_TABLE == (*ppEntry)->type) {
metaCloneSchemaFree(&(*ppEntry)->ntbEntry.schemaRow);
taosMemoryFreeClear((*ppEntry)->ntbEntry.comment);
} else {
@ -353,6 +448,7 @@ void metaCloneEntryFree(SMetaEntry **ppEntry) {
}
metaCloneColCmprFree(&(*ppEntry)->colCmpr);
taosMemoryFreeClear((*ppEntry)->pExtSchemas);
metaCloneColRefFree(&(*ppEntry)->colRef);
taosMemoryFreeClear(*ppEntry);
return;
@ -402,7 +498,7 @@ int32_t metaCloneEntry(const SMetaEntry *pEntry, SMetaEntry **ppEntry) {
return code;
}
(*ppEntry)->stbEntry.keep = pEntry->stbEntry.keep;
} else if (pEntry->type == TSDB_CHILD_TABLE) {
} else if (pEntry->type == TSDB_CHILD_TABLE || pEntry->type == TSDB_VIRTUAL_CHILD_TABLE) {
(*ppEntry)->ctbEntry.btime = pEntry->ctbEntry.btime;
(*ppEntry)->ctbEntry.ttlDays = pEntry->ctbEntry.ttlDays;
(*ppEntry)->ctbEntry.suid = pEntry->ctbEntry.suid;
@ -428,7 +524,7 @@ int32_t metaCloneEntry(const SMetaEntry *pEntry, SMetaEntry **ppEntry) {
return code;
}
memcpy((*ppEntry)->ctbEntry.pTags, pEntry->ctbEntry.pTags, pTags->len);
} else if (pEntry->type == TSDB_NORMAL_TABLE) {
} else if (pEntry->type == TSDB_NORMAL_TABLE || pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
(*ppEntry)->ntbEntry.btime = pEntry->ntbEntry.btime;
(*ppEntry)->ntbEntry.ttlDays = pEntry->ntbEntry.ttlDays;
(*ppEntry)->ntbEntry.ncid = pEntry->ntbEntry.ncid;
@ -455,10 +551,18 @@ int32_t metaCloneEntry(const SMetaEntry *pEntry, SMetaEntry **ppEntry) {
return TSDB_CODE_INVALID_PARA;
}
code = metaCloneColCmpr(&pEntry->colCmpr, &(*ppEntry)->colCmpr);
if (code) {
metaCloneEntryFree(ppEntry);
return code;
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE || pEntry->type == TSDB_VIRTUAL_CHILD_TABLE) {
code = metaCloneColRef(&pEntry->colRef, &(*ppEntry)->colRef);
if (code) {
metaCloneEntryFree(ppEntry);
return code;
}
} else {
code = metaCloneColCmpr(&pEntry->colCmpr, &(*ppEntry)->colCmpr);
if (code) {
metaCloneEntryFree(ppEntry);
return code;
}
}
if (pEntry->pExtSchemas && pEntry->colCmpr.nCols > 0) {
(*ppEntry)->pExtSchemas = taosMemoryCalloc(pEntry->colCmpr.nCols, sizeof(SExtSchema));

589
source/dnode/vnode/src/meta/metaEntry2.c
View File

@ -218,7 +218,7 @@ static int32_t metaSchemaTableUpsert(SMeta *pMeta, const SMetaHandleParam *pPara
const SSchemaWrapper *pSchema = NULL;
if (pEntry->type == TSDB_SUPER_TABLE) {
pSchema = &pEntry->stbEntry.schemaRow;
} else if (pEntry->type == TSDB_NORMAL_TABLE) {
} else if (pEntry->type == TSDB_NORMAL_TABLE || pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
pSchema = &pEntry->ntbEntry.schemaRow;
} else {
return TSDB_CODE_INVALID_PARA;
@ -370,6 +370,88 @@ static int32_t metaAddOrDropTagIndexOfSuperTable(SMeta *pMeta, const SMetaHandle
return code;
}
static int32_t metaAddOrDropColumnIndexOfVirtualSuperTable(SMeta *pMeta, const SMetaHandleParam *pParam,
const SSchema *pOldColumn, const SSchema *pNewColumn) {
int32_t code = TSDB_CODE_SUCCESS;
const SMetaEntry *pEntry = pParam->pEntry;
const SMetaEntry *pOldEntry = pParam->pOldEntry;
enum { ADD_COLUMN, DROP_COLUMN } action;
if (pOldColumn && pNewColumn) {
return TSDB_CODE_SUCCESS;
} else if (pOldColumn) {
action = DROP_COLUMN;
} else {
action = ADD_COLUMN;
}
// fetch all child tables
SArray *childTables = 0;
code = metaGetChildUidsOfSuperTable(pMeta, pEntry->uid, &childTables);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
// do drop or add index
for (int32_t i = 0; i < taosArrayGetSize(childTables); i++) {
int64_t uid = *(int64_t *)taosArrayGet(childTables, i);
// fetch child entry
SMetaEntry *pChildEntry = NULL;
code = metaFetchEntryByUid(pMeta, uid, &pChildEntry);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
taosArrayDestroy(childTables);
return code;
}
SMetaHandleParam param = {
.pEntry = pChildEntry
};
if (action == ADD_COLUMN) {
code = updataTableColRef(&pChildEntry->colRef, pNewColumn, 1, NULL);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
taosArrayDestroy(childTables);
metaFetchEntryFree(&pChildEntry);
return code;
}
code = metaEntryTableUpdate(pMeta, &param);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
taosArrayDestroy(childTables);
metaFetchEntryFree(&pChildEntry);
return code;
}
} else {
code = updataTableColRef(&pChildEntry->colRef, pOldColumn, 0, NULL);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
taosArrayDestroy(childTables);
metaFetchEntryFree(&pChildEntry);
return code;
}
code = metaEntryTableUpdate(pMeta, &param);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
taosArrayDestroy(childTables);
metaFetchEntryFree(&pChildEntry);
return code;
}
}
metaFetchEntryFree(&pChildEntry);
}
taosArrayDestroy(childTables);
return code;
}
static int32_t metaUpdateSuperTableTagSchema(SMeta *pMeta, const SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
const SMetaEntry *pEntry = pParam->pEntry;
@ -431,6 +513,67 @@ static int32_t metaUpdateSuperTableTagSchema(SMeta *pMeta, const SMetaHandlePara
return code;
}
static int32_t metaUpdateSuperTableRowSchema(SMeta *pMeta, const SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
const SMetaEntry *pEntry = pParam->pEntry;
const SMetaEntry *pOldEntry = pParam->pOldEntry;
const SSchemaWrapper *pNewRowSchema = &pEntry->stbEntry.schemaRow;
const SSchemaWrapper *pOldRowSchema = &pOldEntry->stbEntry.schemaRow;
int32_t iOld = 0, iNew = 0;
for (; iOld < pOldRowSchema->nCols && iNew < pNewRowSchema->nCols;) {
SSchema *pOldColumn = pOldRowSchema->pSchema + iOld;
SSchema *pNewColumn = pNewRowSchema->pSchema + iNew;
if (pOldColumn->colId == pNewColumn->colId) {
code = metaAddOrDropColumnIndexOfVirtualSuperTable(pMeta, pParam, pOldColumn, pNewColumn);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
iOld++;
iNew++;
} else if (pOldColumn->colId < pNewColumn->colId) {
code = metaAddOrDropColumnIndexOfVirtualSuperTable(pMeta, pParam, pOldColumn, NULL);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
iOld++;
} else {
code = metaAddOrDropColumnIndexOfVirtualSuperTable(pMeta, pParam, NULL, pNewColumn);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
iNew++;
}
}
for (; iOld < pOldRowSchema->nCols; iOld++) {
SSchema *pOldColumn = pOldRowSchema->pSchema + iOld;
code = metaAddOrDropColumnIndexOfVirtualSuperTable(pMeta, pParam, pOldColumn, NULL);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
for (; iNew < pNewRowSchema->nCols; iNew++) {
SSchema *pNewColumn = pNewRowSchema->pSchema + iNew;
code = metaAddOrDropColumnIndexOfVirtualSuperTable(pMeta, pParam, NULL, pNewColumn);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
return code;
}
static int32_t metaSchemaTableUpdate(SMeta *pMeta, const SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
@ -441,7 +584,7 @@ static int32_t metaSchemaTableUpdate(SMeta *pMeta, const SMetaHandleParam *pPara
return metaSchemaTableUpsert(pMeta, pParam, META_TABLE_OP_UPDATA);
}
if (pEntry->type == TSDB_NORMAL_TABLE) {
if (pEntry->type == TSDB_NORMAL_TABLE || pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
// check row schema
if (pOldEntry->ntbEntry.schemaRow.version != pEntry->ntbEntry.schemaRow.version) {
return metaSchemaTableUpsert(pMeta, pParam, META_TABLE_OP_UPDATA);
@ -449,7 +592,12 @@ static int32_t metaSchemaTableUpdate(SMeta *pMeta, const SMetaHandleParam *pPara
} else if (pEntry->type == TSDB_SUPER_TABLE) {
// check row schema
if (pOldEntry->stbEntry.schemaRow.version != pEntry->stbEntry.schemaRow.version) {
return metaSchemaTableUpsert(pMeta, pParam, META_TABLE_OP_UPDATA);
if (TABLE_IS_VIRTUAL(pEntry->flags)) {
return metaUpdateSuperTableRowSchema(pMeta, pParam);
} else {
return metaSchemaTableUpsert(pMeta, pParam, META_TABLE_OP_UPDATA);
}
}
// check tag schema
@ -478,10 +626,10 @@ static void metaBuildEntryInfo(const SMetaEntry *pEntry, SMetaInfo *pInfo) {
if (pEntry->type == TSDB_SUPER_TABLE) {
pInfo->suid = pEntry->uid;
pInfo->skmVer = pEntry->stbEntry.schemaRow.version;
} else if (pEntry->type == TSDB_CHILD_TABLE) {
} else if (pEntry->type == TSDB_CHILD_TABLE || pEntry->type == TSDB_VIRTUAL_CHILD_TABLE) {
pInfo->suid = pEntry->ctbEntry.suid;
pInfo->skmVer = 0;
} else if (pEntry->type == TSDB_NORMAL_TABLE) {
} else if (pEntry->type == TSDB_NORMAL_TABLE || pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
pInfo->suid = 0;
pInfo->skmVer = pEntry->ntbEntry.schemaRow.version;
}
@ -877,9 +1025,9 @@ static int32_t metaBtimeIdxUpsert(SMeta *pMeta, const SMetaHandleParam *pParam,
.uid = pEntry->uid,
};
if (TSDB_CHILD_TABLE == pEntry->type) {
if (TSDB_CHILD_TABLE == pEntry->type || TSDB_VIRTUAL_CHILD_TABLE == pEntry->type) {
key.btime = pEntry->ctbEntry.btime;
} else if (TSDB_NORMAL_TABLE == pEntry->type) {
} else if (TSDB_NORMAL_TABLE == pEntry->type || TSDB_VIRTUAL_NORMAL_TABLE == pEntry->type) {
key.btime = pEntry->ntbEntry.btime;
} else {
return TSDB_CODE_INVALID_PARA;
@ -1249,6 +1397,121 @@ static int32_t metaHandleChildTableCreate(SMeta *pMeta, const SMetaEntry *pEntry
return code;
}
static int32_t metaHandleVirtualNormalTableCreateImpl(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaTableOp ops[] = {
{META_ENTRY_TABLE, META_TABLE_OP_INSERT}, //
{META_SCHEMA_TABLE, META_TABLE_OP_UPDATA}, // TODO: need to be insert
{META_UID_IDX, META_TABLE_OP_INSERT}, //
{META_NAME_IDX, META_TABLE_OP_INSERT}, //
{META_BTIME_IDX, META_TABLE_OP_INSERT}, //
};
for (int i = 0; i < sizeof(ops) / sizeof(ops[0]); i++) {
SMetaTableOp *op = &ops[i];
SMetaHandleParam param = {
.pEntry = pEntry,
};
code = metaTableOpFn[op->table][op->op](pMeta, &param);
if (TSDB_CODE_SUCCESS != code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
return code;
}
static int32_t metaHandleVirtualNormalTableCreate(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
// update TDB
metaWLock(pMeta);
code = metaHandleVirtualNormalTableCreateImpl(pMeta, pEntry);
metaULock(pMeta);
// update other stuff
if (TSDB_CODE_SUCCESS == code) {
pMeta->pVnode->config.vndStats.numOfVTables++;
} else {
metaErr(TD_VID(pMeta->pVnode), code);
}
return code;
}
static int32_t metaHandleVirtualChildTableCreateImpl(SMeta *pMeta, const SMetaEntry *pEntry, const SMetaEntry *pSuperEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaTableOp ops[] = {
{META_ENTRY_TABLE, META_TABLE_OP_INSERT}, //
{META_UID_IDX, META_TABLE_OP_INSERT}, //
{META_NAME_IDX, META_TABLE_OP_INSERT}, //
{META_CHILD_IDX, META_TABLE_OP_INSERT}, //
{META_TAG_IDX, META_TABLE_OP_INSERT}, //
{META_BTIME_IDX, META_TABLE_OP_INSERT}, //
};
for (int i = 0; i < sizeof(ops) / sizeof(ops[0]); i++) {
SMetaTableOp *op = &ops[i];
SMetaHandleParam param = {
.pEntry = pEntry,
.pSuperEntry = pSuperEntry,
};
code = metaTableOpFn[op->table][op->op](pMeta, &param);
if (TSDB_CODE_SUCCESS != code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
if (TSDB_CODE_SUCCESS == code) {
metaUpdateStbStats(pMeta, pSuperEntry->uid, 1, 0, -1);
int32_t ret = metaUidCacheClear(pMeta, pSuperEntry->uid);
if (ret < 0) {
metaErr(TD_VID(pMeta->pVnode), ret);
}
ret = metaTbGroupCacheClear(pMeta, pSuperEntry->uid);
if (ret < 0) {
metaErr(TD_VID(pMeta->pVnode), ret);
}
}
return code;
}
static int32_t metaHandleVirtualChildTableCreate(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaEntry *pSuperEntry = NULL;
// get the super table entry
code = metaFetchEntryByUid(pMeta, pEntry->ctbEntry.suid, &pSuperEntry);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
// update TDB
metaWLock(pMeta);
code = metaHandleVirtualChildTableCreateImpl(pMeta, pEntry, pSuperEntry);
metaULock(pMeta);
// update other stuff
if (TSDB_CODE_SUCCESS == code) {
pMeta->pVnode->config.vndStats.numOfVCTables++;
} else {
metaErr(TD_VID(pMeta->pVnode), code);
}
metaFetchEntryFree(&pSuperEntry);
return code;
}
static int32_t metaHandleNormalTableDropImpl(SMeta *pMeta, SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
@ -1439,6 +1702,165 @@ static int32_t metaHandleChildTableDrop(SMeta *pMeta, const SMetaEntry *pEntry,
return code;
}
static int32_t metaHandleVirtualNormalTableDropImpl(SMeta *pMeta, SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaTableOp ops[] = {
{META_ENTRY_TABLE, META_TABLE_OP_DELETE}, //
{META_UID_IDX, META_TABLE_OP_DELETE}, //
{META_NAME_IDX, META_TABLE_OP_DELETE}, //
{META_BTIME_IDX, META_TABLE_OP_DELETE}, //
// {META_SCHEMA_TABLE, META_TABLE_OP_DELETE}, //
};
for (int32_t i = 0; i < sizeof(ops) / sizeof(ops[0]); i++) {
SMetaTableOp *op = &ops[i];
code = metaTableOpFn[op->table][op->op](pMeta, pParam);
if (code) {
const SMetaEntry *pEntry = pParam->pEntry;
metaErr(TD_VID(pMeta->pVnode), code);
}
}
return code;
}
static int32_t metaHandleVirtualNormalTableDrop(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaEntry *pOldEntry = NULL;
// fetch the entry
code = metaFetchEntryByUid(pMeta, pEntry->uid, &pOldEntry);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
SMetaHandleParam param = {
.pEntry = pEntry,
.pOldEntry = pOldEntry,
};
// do the drop
metaWLock(pMeta);
code = metaHandleVirtualNormalTableDropImpl(pMeta, &param);
metaULock(pMeta);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
metaFetchEntryFree(&pOldEntry);
return code;
}
// update other stuff
pMeta->pVnode->config.vndStats.numOfVTables--;
#if 0
if (tbUids) {
if (taosArrayPush(tbUids, &uid) == NULL) {
rc = terrno;
goto _exit;
}
}
#endif
if (!TSDB_CACHE_NO(pMeta->pVnode->config)) {
int32_t ret = tsdbCacheDropTable(pMeta->pVnode->pTsdb, pOldEntry->uid, 0, NULL);
if (ret < 0) {
metaErr(TD_VID(pMeta->pVnode), ret);
}
}
metaFetchEntryFree(&pOldEntry);
return code;
}
static int32_t metaHandleVirtualChildTableDropImpl(SMeta *pMeta, const SMetaHandleParam *pParam, bool superDropped) {
int32_t code = TSDB_CODE_SUCCESS;
const SMetaEntry *pEntry = pParam->pEntry;
const SMetaEntry *pChild = pParam->pOldEntry;
const SMetaEntry *pSuper = pParam->pSuperEntry;
SMetaTableOp ops[] = {
{META_ENTRY_TABLE, META_TABLE_OP_DELETE}, //
{META_UID_IDX, META_TABLE_OP_DELETE}, //
{META_NAME_IDX, META_TABLE_OP_DELETE}, //
{META_CHILD_IDX, META_TABLE_OP_DELETE}, //
{META_TAG_IDX, META_TABLE_OP_DELETE}, //
{META_BTIME_IDX, META_TABLE_OP_DELETE}, //
};
for (int i = 0; i < sizeof(ops) / sizeof(ops[0]); i++) {
SMetaTableOp *op = &ops[i];
if (op->table == META_ENTRY_TABLE && superDropped) {
continue;
}
code = metaTableOpFn[op->table][op->op](pMeta, pParam);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
--pMeta->pVnode->config.vndStats.numOfVCTables;
metaUpdateStbStats(pMeta, pParam->pSuperEntry->uid, -1, 0, -1);
int32_t ret = metaUidCacheClear(pMeta, pSuper->uid);
if (ret < 0) {
metaErr(TD_VID(pMeta->pVnode), ret);
}
ret = metaTbGroupCacheClear(pMeta, pSuper->uid);
if (ret < 0) {
metaErr(TD_VID(pMeta->pVnode), ret);
}
return code;
}
static int32_t metaHandleVirtualChildTableDrop(SMeta *pMeta, const SMetaEntry *pEntry, bool superDropped) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaEntry *pChild = NULL;
SMetaEntry *pSuper = NULL;
// fetch old entry
code = metaFetchEntryByUid(pMeta, pEntry->uid, &pChild);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
// fetch super entry
code = metaFetchEntryByUid(pMeta, pChild->ctbEntry.suid, &pSuper);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
metaFetchEntryFree(&pChild);
return code;
}
SMetaHandleParam param = {
.pEntry = pEntry,
.pOldEntry = pChild,
.pSuperEntry = pSuper,
};
// do the drop
metaWLock(pMeta);
code = metaHandleVirtualChildTableDropImpl(pMeta, &param, superDropped);
metaULock(pMeta);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
metaFetchEntryFree(&pChild);
metaFetchEntryFree(&pSuper);
return code;
}
metaFetchEntryFree(&pChild);
metaFetchEntryFree(&pSuper);
return code;
}
static int32_t metaGetChildUidsOfSuperTable(SMeta *pMeta, tb_uid_t suid, SArray **childList) {
int32_t code = TSDB_CODE_SUCCESS;
void *key = NULL;
@ -1551,6 +1973,65 @@ static int32_t metaHandleNormalTableUpdateImpl(SMeta *pMeta, const SMetaHandlePa
return code;
}
static int32_t metaHandleVirtualNormalTableUpdateImpl(SMeta *pMeta, const SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
const SMetaEntry *pEntry = pParam->pEntry;
SMetaTableOp ops[] = {
{META_ENTRY_TABLE, META_TABLE_OP_UPDATA}, //
{META_SCHEMA_TABLE, META_TABLE_OP_UPDATA}, //
{META_UID_IDX, META_TABLE_OP_UPDATA}, //
};
for (int32_t i = 0; i < sizeof(ops) / sizeof(ops[0]); i++) {
SMetaTableOp *op = &ops[i];
code = metaTableOpFn[op->table][op->op](pMeta, pParam);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
#if 0
if (metaUpdateChangeTime(pMeta, entry.uid, pAlterTbReq->ctimeMs) < 0) {
metaError("vgId:%d, failed to update change time:%s uid:%" PRId64, TD_VID(pMeta->pVnode), entry.name, entry.uid);
}
#endif
return code;
}
static int32_t metaHandleVirtualChildTableUpdateImpl(SMeta *pMeta, const SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
const SMetaEntry *pEntry = pParam->pEntry;
const SMetaEntry *pOldEntry = pParam->pOldEntry;
const SMetaEntry *pSuperEntry = pParam->pSuperEntry;
SMetaTableOp ops[] = {
{META_ENTRY_TABLE, META_TABLE_OP_UPDATA}, //
{META_UID_IDX, META_TABLE_OP_UPDATA}, //
{META_TAG_IDX, META_TABLE_OP_UPDATA}, //
{META_CHILD_IDX, META_TABLE_OP_UPDATA}, //
};
for (int i = 0; i < sizeof(ops) / sizeof(ops[0]); i++) {
SMetaTableOp *op = &ops[i];
code = metaTableOpFn[op->table][op->op](pMeta, pParam);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
}
if (metaUidCacheClear(pMeta, pSuperEntry->uid) < 0) {
metaErr(TD_VID(pMeta->pVnode), code);
}
if (metaTbGroupCacheClear(pMeta, pSuperEntry->uid) < 0) {
metaErr(TD_VID(pMeta->pVnode), code);
}
return code;
}
static int32_t metaHandleChildTableUpdateImpl(SMeta *pMeta, const SMetaHandleParam *pParam) {
int32_t code = TSDB_CODE_SUCCESS;
@ -1809,6 +2290,76 @@ static int32_t metaHandleNormalTableUpdate(SMeta *pMeta, const SMetaEntry *pEntr
return code;
}
static int32_t metaHandleVirtualNormalTableUpdate(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaEntry *pOldEntry = NULL;
// fetch old entry
code = metaFetchEntryByUid(pMeta, pEntry->uid, &pOldEntry);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
// handle update
SMetaHandleParam param = {
.pEntry = pEntry,
.pOldEntry = pOldEntry,
};
metaWLock(pMeta);
code = metaHandleVirtualNormalTableUpdateImpl(pMeta, &param);
metaULock(pMeta);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
metaFetchEntryFree(&pOldEntry);
return code;
}
metaTimeSeriesNotifyCheck(pMeta);
metaFetchEntryFree(&pOldEntry);
return code;
}
static int32_t metaHandleVirtualChildTableUpdate(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SMetaEntry *pOldEntry = NULL;
SMetaEntry *pSuperEntry = NULL;
code = metaFetchEntryByUid(pMeta, pEntry->uid, &pOldEntry);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
return code;
}
code = metaFetchEntryByUid(pMeta, pEntry->ctbEntry.suid, &pSuperEntry);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
metaFetchEntryFree(&pOldEntry);
return code;
}
SMetaHandleParam param = {
.pEntry = pEntry,
.pOldEntry = pOldEntry,
.pSuperEntry = pSuperEntry,
};
metaWLock(pMeta);
code = metaHandleVirtualChildTableUpdateImpl(pMeta, &param);
metaULock(pMeta);
if (code) {
metaErr(TD_VID(pMeta->pVnode), code);
metaFetchEntryFree(&pOldEntry);
metaFetchEntryFree(&pSuperEntry);
return code;
}
metaFetchEntryFree(&pOldEntry);
metaFetchEntryFree(&pSuperEntry);
return code;
}
static int32_t metaHandleSuperTableDrop(SMeta *pMeta, const SMetaEntry *pEntry) {
int32_t code = TSDB_CODE_SUCCESS;
SArray *childList = NULL;
@ -1912,6 +2463,22 @@ int32_t metaHandleEntry2(SMeta *pMeta, const SMetaEntry *pEntry) {
}
break;
}
case TSDB_VIRTUAL_NORMAL_TABLE: {
if (isExist) {
code = metaHandleVirtualNormalTableUpdate(pMeta, pEntry);
} else {
code = metaHandleVirtualNormalTableCreate(pMeta, pEntry);
}
break;
}
case TSDB_VIRTUAL_CHILD_TABLE: {
if (isExist) {
code = metaHandleVirtualChildTableUpdate(pMeta, pEntry);
} else {
code = metaHandleVirtualChildTableCreate(pMeta, pEntry);
}
break;
}
default: {
code = TSDB_CODE_INVALID_PARA;
break;
@ -1931,6 +2498,14 @@ int32_t metaHandleEntry2(SMeta *pMeta, const SMetaEntry *pEntry) {
code = metaHandleNormalTableDrop(pMeta, pEntry);
break;
}
case TSDB_VIRTUAL_NORMAL_TABLE: {
code = metaHandleVirtualNormalTableDrop(pMeta, pEntry);
break;
}
case TSDB_VIRTUAL_CHILD_TABLE: {
code = metaHandleVirtualChildTableDrop(pMeta, pEntry, false);
break;
}
default: {
code = TSDB_CODE_INVALID_PARA;
break;

99
source/dnode/vnode/src/meta/metaTable.c
View File

@ -25,7 +25,8 @@ int32_t metaUpdateTableTagValue(SMeta *pMeta, int64_t version, SVAlterTbReq *pRe
int32_t metaUpdateTableMultiTagValue(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq);
int32_t metaUpdateTableOptions2(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq);
int32_t metaUpdateTableColCompress2(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq);
int32_t metaAlterTableColumnRef(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMetaRsp *pRsp);
int32_t metaRemoveTableColumnRef(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMetaRsp *pRsp);
int32_t metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSchema *pSchema);
int32_t metaDelJsonVarFromIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSchema *pSchema);
@ -103,6 +104,49 @@ int32_t dropTableExtSchema(SMetaEntry *pEntry, int32_t dropColId, int32_t newCol
return 0;
}
int32_t updataTableColRef(SColRefWrapper *pWp, const SSchema *pSchema, int8_t add, SColRef *pColRef) {
int32_t nCols = pWp->nCols;
int32_t ver = pWp->version;
if (add) {
SColRef *p = taosMemoryRealloc(pWp->pColRef, sizeof(SColRef) * (nCols + 1));
if (p == NULL) {
return terrno;
}
pWp->pColRef = p;
SColRef *pCol = p + nCols;
if (NULL == pColRef) {
pCol->hasRef = false;
pCol->id = pSchema->colId;
} else {
pCol->hasRef = pColRef->hasRef;
pCol->id = pSchema->colId;
if (pCol->hasRef) {
tstrncpy(pCol->refDbName, pColRef->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pCol->refTableName, pColRef->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pCol->refColName, pColRef->refColName, TSDB_COL_NAME_LEN);
}
}
pWp->nCols = nCols + 1;
pWp->version = ver;
} else {
for (int32_t i = 0; i < nCols; i++) {
SColRef *pOColRef = &pWp->pColRef[i];
if (pOColRef->id == pSchema->colId) {
int32_t left = (nCols - i - 1) * sizeof(SColRef);
if (left) {
memmove(pWp->pColRef + i, pWp->pColRef + i + 1, left);
}
nCols--;
break;
}
}
pWp->nCols = nCols;
pWp->version = ver;
}
return 0;
}
int metaUpdateMetaRsp(tb_uid_t uid, char *tbName, SSchemaWrapper *pSchema, STableMetaRsp *pMetaRsp) {
pMetaRsp->pSchemas = taosMemoryMalloc(pSchema->nCols * sizeof(SSchema));
if (NULL == pMetaRsp->pSchemas) {
@ -120,12 +164,56 @@ int metaUpdateMetaRsp(tb_uid_t uid, char *tbName, SSchemaWrapper *pSchema, STabl
pMetaRsp->tableType = TSDB_NORMAL_TABLE;
pMetaRsp->sversion = pSchema->version;
pMetaRsp->tuid = uid;
pMetaRsp->virtualStb = false; // super table will never be processed here
memcpy(pMetaRsp->pSchemas, pSchema->pSchema, pSchema->nCols * sizeof(SSchema));
return 0;
}
int32_t metaUpdateVtbMetaRsp(tb_uid_t uid, char *tbName, SSchemaWrapper *pSchema, SColRefWrapper *pRef,
STableMetaRsp *pMetaRsp, int8_t tableType) {
int32_t code = TSDB_CODE_SUCCESS;
if (!pRef) {
return TSDB_CODE_INVALID_PARA;
}
if (pSchema) {
pMetaRsp->pSchemas = taosMemoryMalloc(pSchema->nCols * sizeof(SSchema));
if (NULL == pMetaRsp->pSchemas) {
code = terrno;
goto _return;
}
pMetaRsp->pSchemaExt = taosMemoryMalloc(pSchema->nCols * sizeof(SSchemaExt));
if (pMetaRsp->pSchemaExt == NULL) {
code = terrno;
goto _return;
}
pMetaRsp->numOfColumns = pSchema->nCols;
pMetaRsp->sversion = pSchema->version;
memcpy(pMetaRsp->pSchemas, pSchema->pSchema, pSchema->nCols * sizeof(SSchema));
}
pMetaRsp->pColRefs = taosMemoryMalloc(pRef->nCols * sizeof(SColRef));
if (NULL == pMetaRsp->pColRefs) {
code = terrno;
goto _return;
}
memcpy(pMetaRsp->pColRefs, pRef->pColRef, pRef->nCols * sizeof(SColRef));
tstrncpy(pMetaRsp->tbName, tbName, TSDB_TABLE_NAME_LEN);
pMetaRsp->tuid = uid;
pMetaRsp->tableType = tableType;
pMetaRsp->virtualStb = false; // super table will never be processed here
pMetaRsp->numOfColRefs = pRef->nCols;
return code;
_return:
taosMemoryFreeClear(pMetaRsp->pSchemaExt);
taosMemoryFreeClear(pMetaRsp->pSchemas);
taosMemoryFreeClear(pMetaRsp->pColRefs);
return code;
}
int metaSaveJsonVarToIdx(SMeta *pMeta, const SMetaEntry *pCtbEntry, const SSchema *pSchema) {
int32_t code = 0;
@ -390,7 +478,7 @@ static int32_t metaFilterTableByHash(SMeta *pMeta, SArray *uidList) {
char tbFName[TSDB_TABLE_FNAME_LEN + 1];
snprintf(tbFName, sizeof(tbFName), "%s.%s", pMeta->pVnode->config.dbname, me.name);
tbFName[TSDB_TABLE_FNAME_LEN] = '\0';
int32_t ret = vnodeValidateTableHash(pMeta->pVnode, tbFName);
ret = vnodeValidateTableHash(pMeta->pVnode, tbFName);
if (ret < 0 && terrno == TSDB_CODE_VND_HASH_MISMATCH) {
if (taosArrayPush(uidList, &me.uid) == NULL) {
code = terrno;
@ -531,7 +619,7 @@ static int metaDropTableByUid(SMeta *pMeta, tb_uid_t uid, int *type, tb_uid_t *p
SMetaEntry stbEntry = {0};
tDecoderInit(&tdc, tData, tLen);
int32_t ret = metaDecodeEntry(&tdc, &stbEntry);
ret = metaDecodeEntry(&tdc, &stbEntry);
if (ret < 0) {
tDecoderClear(&tdc);
metaError("vgId:%d, failed to decode child table:%s uid:%" PRId64 " since %s", TD_VID(pMeta->pVnode), e.name,
@ -709,6 +797,7 @@ int metaAlterTable(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMeta
switch (pReq->action) {
case TSDB_ALTER_TABLE_ADD_COLUMN:
case TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COMPRESS_OPTION:
case TSDB_ALTER_TABLE_ADD_COLUMN_WITH_COLUMN_REF:
return metaAddTableColumn(pMeta, version, pReq, pMetaRsp);
case TSDB_ALTER_TABLE_DROP_COLUMN:
return metaDropTableColumn(pMeta, version, pReq, pMetaRsp);
@ -724,6 +813,10 @@ int metaAlterTable(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMeta
return metaUpdateTableOptions2(pMeta, version, pReq);
case TSDB_ALTER_TABLE_UPDATE_COLUMN_COMPRESS:
return metaUpdateTableColCompress2(pMeta, version, pReq);
case TSDB_ALTER_TABLE_ALTER_COLUMN_REF:
return metaAlterTableColumnRef(pMeta, version, pReq, pMetaRsp);
case TSDB_ALTER_TABLE_REMOVE_COLUMN_REF:
return metaRemoveTableColumnRef(pMeta, version, pReq, pMetaRsp);
default:
return terrno = TSDB_CODE_VND_INVALID_TABLE_ACTION;
break;

596
source/dnode/vnode/src/meta/metaTable2.c
View File

@ -17,6 +17,8 @@
extern int32_t metaHandleEntry2(SMeta *pMeta, const SMetaEntry *pEntry);
extern int32_t metaUpdateMetaRsp(tb_uid_t uid, char *tbName, SSchemaWrapper *pSchema, STableMetaRsp *pMetaRsp);
extern int32_t metaUpdateVtbMetaRsp(tb_uid_t uid, char *tbName, SSchemaWrapper *pSchema, SColRefWrapper *pRef,
STableMetaRsp *pMetaRsp, int8_t tableType);
extern int32_t metaFetchEntryByUid(SMeta *pMeta, int64_t uid, SMetaEntry **ppEntry);
extern int32_t metaFetchEntryByName(SMeta *pMeta, const char *name, SMetaEntry **ppEntry);
extern void metaFetchEntryFree(SMetaEntry **ppEntry);
@ -192,8 +194,13 @@ int32_t metaCreateSuperTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq
TABLE_SET_COL_COMPRESSED(entry.flags);
entry.colCmpr = pReq->colCmpr;
}
entry.pExtSchemas = pReq->pExtSchemas;
if (pReq->virtualStb) {
TABLE_SET_VIRTUAL(entry.flags);
}
code = metaHandleEntry2(pMeta, &entry);
if (TSDB_CODE_SUCCESS == code) {
metaInfo("vgId:%d, super table %s suid:%" PRId64 " is created, version:%" PRId64, TD_VID(pMeta->pVnode), pReq->name,
@ -495,6 +502,141 @@ static int32_t metaCreateNormalTable(SMeta *pMeta, int64_t version, SVCreateTbRe
TAOS_RETURN(code);
}
static int32_t metaBuildCreateVirtualNormalTableRsp(SMeta *pMeta, SMetaEntry *pEntry, STableMetaRsp **ppRsp) {
int32_t code = TSDB_CODE_SUCCESS;
if (NULL == ppRsp) {
return code;
}
*ppRsp = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == *ppRsp) {
return terrno;
}
code = metaUpdateVtbMetaRsp(pEntry->uid, pEntry->name, &pEntry->ntbEntry.schemaRow, &pEntry->colRef, *ppRsp, TSDB_VIRTUAL_NORMAL_TABLE);
if (code) {
taosMemoryFreeClear(*ppRsp);
return code;
}
return code;
}
static int32_t metaCreateVirtualNormalTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq, STableMetaRsp **ppRsp) {
// check request
int32_t code = metaCheckCreateNormalTableReq(pMeta, version, pReq);
if (code) {
if (TSDB_CODE_TDB_TABLE_ALREADY_EXIST != code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64 " name:%s", TD_VID(pMeta->pVnode), __func__,
__FILE__, __LINE__, tstrerror(code), version, pReq->name);
}
TAOS_RETURN(code);
}
SMetaEntry entry = {
.version = version,
.type = TSDB_VIRTUAL_NORMAL_TABLE,
.uid = pReq->uid,
.name = pReq->name,
.ntbEntry.btime = pReq->btime,
.ntbEntry.ttlDays = pReq->ttl,
.ntbEntry.commentLen = pReq->commentLen,
.ntbEntry.comment = pReq->comment,
.ntbEntry.schemaRow = pReq->ntb.schemaRow,
.ntbEntry.ncid = pReq->ntb.schemaRow.pSchema[pReq->ntb.schemaRow.nCols - 1].colId + 1,
.colRef = pReq->colRef
};
code = metaBuildCreateVirtualNormalTableRsp(pMeta, &entry, ppRsp);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s", TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__,
tstrerror(code));
}
// handle entry
code = metaHandleEntry2(pMeta, &entry);
if (TSDB_CODE_SUCCESS == code) {
metaInfo("vgId:%d, normal table:%s uid %" PRId64 " is created, version:%" PRId64, TD_VID(pMeta->pVnode), pReq->name,
pReq->uid, version);
} else {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pReq->uid, pReq->name, version);
}
TAOS_RETURN(code);
#if 0
metaTimeSeriesNotifyCheck(pMeta);
#endif
}
static int32_t metaBuildCreateVirtualChildTableRsp(SMeta *pMeta, SMetaEntry *pEntry, STableMetaRsp **ppRsp) {
int32_t code = TSDB_CODE_SUCCESS;
if (NULL == ppRsp) {
return code;
}
*ppRsp = taosMemoryCalloc(1, sizeof(STableMetaRsp));
if (NULL == *ppRsp) {
return terrno;
}
code = metaUpdateVtbMetaRsp(pEntry->uid, pEntry->name, NULL, &pEntry->colRef, *ppRsp, TSDB_VIRTUAL_CHILD_TABLE);
if (code) {
taosMemoryFreeClear(*ppRsp);
return code;
}
(*ppRsp)->suid = pEntry->ctbEntry.suid;
return code;
}
static int32_t metaCreateVirtualChildTable(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq, STableMetaRsp **ppRsp) {
// check request
int32_t code = metaCheckCreateChildTableReq(pMeta, version, pReq);
if (code) {
if (TSDB_CODE_TDB_TABLE_ALREADY_EXIST != code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64 " name:%s", TD_VID(pMeta->pVnode), __func__,
__FILE__, __LINE__, tstrerror(code), version, pReq->name);
}
TAOS_RETURN(code);
}
SMetaEntry entry = {
.version = version,
.type = TSDB_VIRTUAL_CHILD_TABLE,
.uid = pReq->uid,
.name = pReq->name,
.ctbEntry.btime = pReq->btime,
.ctbEntry.ttlDays = pReq->ttl,
.ctbEntry.commentLen = pReq->commentLen,
.ctbEntry.comment = pReq->comment,
.ctbEntry.suid = pReq->ctb.suid,
.ctbEntry.pTags = pReq->ctb.pTag,
.colRef = pReq->colRef
};
code = metaBuildCreateVirtualChildTableRsp(pMeta, &entry, ppRsp);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s", TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__,
tstrerror(code));
}
// handle entry
code = metaHandleEntry2(pMeta, &entry);
if (TSDB_CODE_SUCCESS == code) {
metaInfo("vgId:%d, normal table:%s uid %" PRId64 " is created, version:%" PRId64, TD_VID(pMeta->pVnode), pReq->name,
pReq->uid, version);
} else {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pReq->uid, pReq->name, version);
}
TAOS_RETURN(code);
#if 0
metaTimeSeriesNotifyCheck(pMeta);
#endif
}
// Drop Normal Table
// Alter Normal Table
@ -505,6 +647,10 @@ int32_t metaCreateTable2(SMeta *pMeta, int64_t version, SVCreateTbReq *pReq, STa
code = metaCreateChildTable(pMeta, version, pReq, ppRsp);
} else if (TSDB_NORMAL_TABLE == pReq->type) {
code = metaCreateNormalTable(pMeta, version, pReq, ppRsp);
} else if (TSDB_VIRTUAL_NORMAL_TABLE == pReq->type) {
code = metaCreateVirtualNormalTable(pMeta, version, pReq, ppRsp);
} else if (TSDB_VIRTUAL_CHILD_TABLE == pReq->type) {
code = metaCreateVirtualChildTable(pMeta, version, pReq, ppRsp);
} else {
code = TSDB_CODE_INVALID_MSG;
}
@ -536,10 +682,18 @@ int32_t metaDropTable2(SMeta *pMeta, int64_t version, SVDropTbReq *pReq) {
.uid = pReq->uid,
};
if (pReq->suid == 0) {
entry.type = -TSDB_NORMAL_TABLE;
if (pReq->isVirtual) {
if (pReq->suid == 0) {
entry.type = -TSDB_VIRTUAL_NORMAL_TABLE;
} else {
entry.type = -TSDB_VIRTUAL_CHILD_TABLE;
}
} else {
entry.type = -TSDB_CHILD_TABLE;
if (pReq->suid == 0) {
entry.type = -TSDB_NORMAL_TABLE;
} else {
entry.type = -TSDB_CHILD_TABLE;
}
}
code = metaHandleEntry2(pMeta, &entry);
if (code) {
@ -583,7 +737,7 @@ static int32_t metaCheckAlterTableColumnReq(SMeta *pMeta, int64_t version, SVAlt
code = TSDB_CODE_INTERNAL_ERROR;
TAOS_RETURN(code);
}
if (info.suid != 0) {
if (info.suid != 0 && pReq->action != TSDB_ALTER_TABLE_ALTER_COLUMN_REF && pReq->action != TSDB_ALTER_TABLE_REMOVE_COLUMN_REF) {
metaError("vgId:%d, %s failed at %s:%d since table %s uid %" PRId64 " is not a normal table, version:%" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pReq->tbName, uid, version);
code = TSDB_CODE_VND_INVALID_TABLE_ACTION;
@ -665,18 +819,39 @@ int32_t metaAddTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, ST
pColumn->colId = pEntry->ntbEntry.ncid++;
extSchema.typeMod = pReq->typeMod;
tstrncpy(pColumn->name, pReq->colName, TSDB_COL_NAME_LEN);
uint32_t compress;
if (TSDB_ALTER_TABLE_ADD_COLUMN == pReq->action) {
compress = createDefaultColCmprByType(pColumn->type);
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
SColRef tmpRef;
if (TSDB_ALTER_TABLE_ADD_COLUMN == pReq->action) {
tmpRef.hasRef = false;
tmpRef.id = pColumn->colId;
} else {
tmpRef.hasRef = true;
tmpRef.id = pColumn->colId;
tstrncpy(tmpRef.refDbName, pReq->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(tmpRef.refTableName, pReq->refTbName, TSDB_TABLE_NAME_LEN);
tstrncpy(tmpRef.refColName, pReq->refColName, TSDB_COL_NAME_LEN);
}
code = updataTableColRef(&pEntry->colRef, pColumn, 1, &tmpRef);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64, TD_VID(pMeta->pVnode), __func__, __FILE__,
__LINE__, tstrerror(code), version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(code);
}
} else {
compress = pReq->compress;
}
code = updataTableColCmpr(&pEntry->colCmpr, pColumn, 1, compress);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64, TD_VID(pMeta->pVnode), __func__, __FILE__,
__LINE__, tstrerror(code), version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(code);
uint32_t compress;
if (TSDB_ALTER_TABLE_ADD_COLUMN == pReq->action) {
compress = createDefaultColCmprByType(pColumn->type);
} else {
compress = pReq->compress;
}
code = updataTableColCmpr(&pEntry->colCmpr, pColumn, 1, compress);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64, TD_VID(pMeta->pVnode), __func__, __FILE__,
__LINE__, tstrerror(code), version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(code);
}
}
code = addTableExtSchema(pEntry, pColumn, pSchema->nCols, &extSchema);
if (code) {
@ -698,14 +873,32 @@ int32_t metaAddTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, ST
pEntry->uid, version);
}
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
if (metaUpdateVtbMetaRsp(pEntry->uid, pReq->tbName, pSchema, &pEntry->colRef, pRsp, pEntry->type) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colRef.nCols; i++) {
SColRef *p = &pEntry->colRef.pColRef[i];
pRsp->pColRefs[i].hasRef = p->hasRef;
pRsp->pColRefs[i].id = p->id;
if (p->hasRef) {
tstrncpy(pRsp->pColRefs[i].refDbName, p->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refTableName, p->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refColName, p->refColName, TSDB_COL_NAME_LEN);
}
}
}
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
}
}
}
@ -779,18 +972,34 @@ int32_t metaDropTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, S
}
pSchema->nCols--;
pSchema->version++;
code = updataTableColCmpr(&pEntry->colCmpr, &tColumn, 0, 0);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64, TD_VID(pMeta->pVnode), __func__, __FILE__,
__LINE__, tstrerror(code), version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(code);
}
if (pEntry->colCmpr.nCols != pSchema->nCols) {
metaError("vgId:%d, %s failed at %s:%d since column count mismatch, version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_VND_INVALID_TABLE_ACTION);
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
code = updataTableColRef(&pEntry->colRef, &tColumn, 0, NULL);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64, TD_VID(pMeta->pVnode), __func__, __FILE__,
__LINE__, tstrerror(code), version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(code);
}
if (pEntry->colRef.nCols != pSchema->nCols) {
metaError("vgId:%d, %s failed at %s:%d since column count mismatch, version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_VND_INVALID_TABLE_ACTION);
}
} else {
code = updataTableColCmpr(&pEntry->colCmpr, &tColumn, 0, 0);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, version:%" PRId64, TD_VID(pMeta->pVnode), __func__, __FILE__,
__LINE__, tstrerror(code), version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(code);
}
if (pEntry->colCmpr.nCols != pSchema->nCols) {
metaError("vgId:%d, %s failed at %s:%d since column count mismatch, version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_VND_INVALID_TABLE_ACTION);
}
}
// update column extschema
@ -814,14 +1023,32 @@ int32_t metaDropTableColumn(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, S
}
// build response
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
if (metaUpdateVtbMetaRsp(pEntry->uid, pReq->tbName, pSchema, &pEntry->colRef, pRsp, pEntry->type) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colRef.nCols; i++) {
SColRef *p = &pEntry->colRef.pColRef[i];
pRsp->pColRefs[i].hasRef = p->hasRef;
pRsp->pColRefs[i].id = p->id;
if (p->hasRef) {
tstrncpy(pRsp->pColRefs[i].refDbName, p->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refTableName, p->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refColName, p->refColName, TSDB_COL_NAME_LEN);
}
}
}
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
}
}
}
@ -904,14 +1131,32 @@ int32_t metaAlterTableColumnName(SMeta *pMeta, int64_t version, SVAlterTbReq *pR
}
// build response
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
if (metaUpdateVtbMetaRsp(pEntry->uid, pReq->tbName, pSchema, &pEntry->colRef, pRsp, pEntry->type) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colRef.nCols; i++) {
SColRef *p = &pEntry->colRef.pColRef[i];
pRsp->pColRefs[i].hasRef = p->hasRef;
pRsp->pColRefs[i].id = p->id;
if (p->hasRef) {
tstrncpy(pRsp->pColRefs[i].refDbName, p->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refTableName, p->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refColName, p->refColName, TSDB_COL_NAME_LEN);
}
}
}
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
}
}
}
@ -1005,14 +1250,32 @@ int32_t metaAlterTableColumnBytes(SMeta *pMeta, int64_t version, SVAlterTbReq *p
}
// build response
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
if (pEntry->type == TSDB_VIRTUAL_NORMAL_TABLE) {
if (metaUpdateVtbMetaRsp(pEntry->uid, pReq->tbName, pSchema, &pEntry->colRef, pRsp, pEntry->type) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colRef.nCols; i++) {
SColRef *p = &pEntry->colRef.pColRef[i];
pRsp->pColRefs[i].hasRef = p->hasRef;
pRsp->pColRefs[i].id = p->id;
if (p->hasRef) {
tstrncpy(pRsp->pColRefs[i].refDbName, p->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refTableName, p->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refColName, p->refColName, TSDB_COL_NAME_LEN);
}
}
}
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
if (metaUpdateMetaRsp(pEntry->uid, pReq->tbName, pSchema, pRsp) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colCmpr.nCols; i++) {
SColCmpr *p = &pEntry->colCmpr.pColCmpr[i];
pRsp->pSchemaExt[i].colId = p->id;
pRsp->pSchemaExt[i].compress = p->alg;
}
}
}
@ -1063,7 +1326,7 @@ int32_t metaUpdateTableTagValue(SMeta *pMeta, int64_t version, SVAlterTbReq *pRe
TAOS_RETURN(code);
}
if (pChild->type != TSDB_CHILD_TABLE) {
if (pChild->type != TSDB_CHILD_TABLE && pChild->type != TSDB_VIRTUAL_CHILD_TABLE) {
metaError("vgId:%d, %s failed at %s:%d since table %s is not a child table, version:%" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pReq->tbName, version);
metaFetchEntryFree(&pChild);
@ -1561,6 +1824,224 @@ int32_t metaUpdateTableColCompress2(SMeta *pMeta, int64_t version, SVAlterTbReq
TAOS_RETURN(code);
}
int32_t metaAlterTableColumnRef(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMetaRsp *pRsp) {
int32_t code = TSDB_CODE_SUCCESS;
// check request
code = metaCheckAlterTableColumnReq(pMeta, version, pReq);
if (code) {
TAOS_RETURN(code);
}
if (NULL == pReq->refDbName) {
metaError("vgId:%d, %s failed at %s:%d since invalid ref db name, version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, version);
TAOS_RETURN(TSDB_CODE_INVALID_MSG);
}
if (NULL == pReq->refTbName) {
metaError("vgId:%d, %s failed at %s:%d since invalid ref table name, version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, version);
TAOS_RETURN(TSDB_CODE_INVALID_MSG);
}
if (NULL == pReq->refColName) {
metaError("vgId:%d, %s failed at %s:%d since invalid ref Col name, version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, version);
TAOS_RETURN(TSDB_CODE_INVALID_MSG);
}
// fetch old entry
SMetaEntry *pEntry = NULL;
code = metaFetchEntryByName(pMeta, pReq->tbName, &pEntry);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since table %s not found, version:%" PRId64, TD_VID(pMeta->pVnode), __func__,
__FILE__, __LINE__, pReq->tbName, version);
TAOS_RETURN(code);
}
if (pEntry->version >= version) {
metaError("vgId:%d, %s failed at %s:%d since table %s version %" PRId64 " is not less than %" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pReq->tbName, pEntry->version, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_INVALID_PARA);
}
// fetch super entry
SMetaEntry *pSuper = NULL;
if (pEntry->type == TSDB_VIRTUAL_CHILD_TABLE) {
code = metaFetchEntryByUid(pMeta, pEntry->ctbEntry.suid, &pSuper);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since super table uid %" PRId64 " not found, version:%" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pEntry->ctbEntry.suid, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_INTERNAL_ERROR);
}
}
// search the column to update
SSchemaWrapper *pSchema = pEntry->type == TSDB_VIRTUAL_CHILD_TABLE ? &pSuper->stbEntry.schemaRow : &pEntry->ntbEntry.schemaRow;
SColRef *pColRef = NULL;
int32_t iColumn = 0;
for (int32_t i = 0; i < pSchema->nCols; i++) {
if (strncmp(pSchema->pSchema[i].name, pReq->colName, TSDB_COL_NAME_LEN) == 0) {
pColRef = &pEntry->colRef.pColRef[i];
iColumn = i;
break;
}
}
if (NULL == pColRef) {
metaError("vgId:%d, %s failed at %s:%d since column id %d not found in table %s, version:%" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pReq->colId, pReq->tbName, version);
metaFetchEntryFree(&pEntry);
metaFetchEntryFree(&pSuper);
TAOS_RETURN(TSDB_CODE_VND_COL_NOT_EXISTS);
}
// do update column name
pEntry->version = version;
pColRef->hasRef = true;
pColRef->id = pSchema->pSchema[iColumn].colId;
tstrncpy(pColRef->refDbName, pReq->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pColRef->refTableName, pReq->refTbName, TSDB_TABLE_NAME_LEN);
tstrncpy(pColRef->refColName, pReq->refColName, TSDB_COL_NAME_LEN);
pSchema->version++;
// do handle entry
code = metaHandleEntry2(pMeta, pEntry);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
metaFetchEntryFree(&pEntry);
metaFetchEntryFree(&pSuper);
TAOS_RETURN(code);
} else {
metaInfo("vgId:%d, table %s uid %" PRId64 " is updated, version:%" PRId64, TD_VID(pMeta->pVnode), pReq->tbName,
pEntry->uid, version);
}
// build response
if (metaUpdateVtbMetaRsp(pEntry->uid, pReq->tbName, pSchema, &pEntry->colRef, pRsp, pEntry->type) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colRef.nCols; i++) {
SColRef *p = &pEntry->colRef.pColRef[i];
pRsp->pColRefs[i].hasRef = p->hasRef;
pRsp->pColRefs[i].id = p->id;
if (p->hasRef) {
tstrncpy(pRsp->pColRefs[i].refDbName, p->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refTableName, p->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refColName, p->refColName, TSDB_COL_NAME_LEN);
}
}
}
metaFetchEntryFree(&pEntry);
metaFetchEntryFree(&pSuper);
TAOS_RETURN(code);
}
int32_t metaRemoveTableColumnRef(SMeta *pMeta, int64_t version, SVAlterTbReq *pReq, STableMetaRsp *pRsp) {
int32_t code = TSDB_CODE_SUCCESS;
// check request
code = metaCheckAlterTableColumnReq(pMeta, version, pReq);
if (code) {
TAOS_RETURN(code);
}
// fetch old entry
SMetaEntry *pEntry = NULL;
code = metaFetchEntryByName(pMeta, pReq->tbName, &pEntry);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since table %s not found, version:%" PRId64, TD_VID(pMeta->pVnode), __func__,
__FILE__, __LINE__, pReq->tbName, version);
TAOS_RETURN(code);
}
if (pEntry->version >= version) {
metaError("vgId:%d, %s failed at %s:%d since table %s version %" PRId64 " is not less than %" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pReq->tbName, pEntry->version, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_INVALID_PARA);
}
// fetch super entry
SMetaEntry *pSuper = NULL;
if (pEntry->type == TSDB_VIRTUAL_CHILD_TABLE) {
code = metaFetchEntryByUid(pMeta, pEntry->ctbEntry.suid, &pSuper);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since super table uid %" PRId64 " not found, version:%" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, pEntry->ctbEntry.suid, version);
metaFetchEntryFree(&pEntry);
TAOS_RETURN(TSDB_CODE_INTERNAL_ERROR);
}
}
// search the column to update
SSchemaWrapper *pSchema = pEntry->type == TSDB_VIRTUAL_CHILD_TABLE ? &pSuper->stbEntry.schemaRow : &pEntry->ntbEntry.schemaRow;
SColRef *pColRef = NULL;
int32_t iColumn = 0;
for (int32_t i = 0; i < pSchema->nCols; i++) {
if (strncmp(pSchema->pSchema[i].name, pReq->colName, TSDB_COL_NAME_LEN) == 0) {
pColRef = &pEntry->colRef.pColRef[i];
iColumn = i;
break;
}
}
if (NULL == pColRef) {
metaError("vgId:%d, %s failed at %s:%d since column id %d not found in table %s, version:%" PRId64,
TD_VID(pMeta->pVnode), __func__, __FILE__, __LINE__, iColumn + 1, pReq->tbName, version);
metaFetchEntryFree(&pEntry);
metaFetchEntryFree(&pSuper);
TAOS_RETURN(TSDB_CODE_VND_COL_NOT_EXISTS);
}
// do update column name
pEntry->version = version;
pColRef->hasRef = false;
pColRef->id = pSchema->pSchema[iColumn].colId;
pSchema->version++;
// do handle entry
code = metaHandleEntry2(pMeta, pEntry);
if (code) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
metaFetchEntryFree(&pEntry);
metaFetchEntryFree(&pSuper);
TAOS_RETURN(code);
} else {
metaInfo("vgId:%d, table %s uid %" PRId64 " is updated, version:%" PRId64, TD_VID(pMeta->pVnode), pReq->tbName,
pEntry->uid, version);
}
// build response
if (metaUpdateVtbMetaRsp(pEntry->uid, pReq->tbName, pSchema, &pEntry->colRef, pRsp, pEntry->type) < 0) {
metaError("vgId:%d, %s failed at %s:%d since %s, uid:%" PRId64 " name:%s version:%" PRId64, TD_VID(pMeta->pVnode),
__func__, __FILE__, __LINE__, tstrerror(code), pEntry->uid, pReq->tbName, version);
} else {
for (int32_t i = 0; i < pEntry->colRef.nCols; i++) {
SColRef *p = &pEntry->colRef.pColRef[i];
pRsp->pColRefs[i].hasRef = p->hasRef;
pRsp->pColRefs[i].id = p->id;
if (p->hasRef) {
tstrncpy(pRsp->pColRefs[i].refDbName, p->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refTableName, p->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRsp->pColRefs[i].refColName, p->refColName, TSDB_COL_NAME_LEN);
}
}
}
metaFetchEntryFree(&pEntry);
metaFetchEntryFree(&pSuper);
TAOS_RETURN(code);
}
int32_t metaAddIndexToSuperTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq) {
int32_t code = TSDB_CODE_SUCCESS;
@ -1791,6 +2272,9 @@ int32_t metaAlterSuperTable(SMeta *pMeta, int64_t version, SVCreateStbReq *pReq)
.pExtSchemas = pReq->pExtSchemas,
};
TABLE_SET_COL_COMPRESSED(entry.flags);
if (pReq->virtualStb) {
TABLE_SET_VIRTUAL(entry.flags);
}
// do handle the entry
code = metaHandleEntry2(pMeta, &entry);

2
source/dnode/vnode/src/sma/smaRollup.c
View File

@ -692,7 +692,7 @@ static int32_t tdRSmaExecAndSubmitResult(SSma *pSma, qTaskInfo_t taskInfo, SRSma
while (1) {
uint64_t ts;
bool hasMore = false;
code = qExecTaskOpt(taskInfo, pResList, &ts, &hasMore, NULL);
code = qExecTaskOpt(taskInfo, pResList, &ts, &hasMore, NULL, false);
if (code == TSDB_CODE_QRY_IN_EXEC) {
code = 0;
break;

624
source/dnode/vnode/src/tq/tqRead.c
View File

@ -16,6 +16,8 @@
#include "tmsg.h"
#include "tq.h"
static int32_t tqCollectPhysicalTables(STqReader* pReader, const char* idstr);
bool isValValidForTable(STqHandle* pHandle, SWalCont* pHead) {
if (pHandle == NULL || pHead == NULL) {
return false;
@ -198,7 +200,7 @@ bool isValValidForTable(STqHandle* pHandle, SWalCont* pHead) {
end:
tDecoderClear(&dcoder);
bool tmp = tbSuid == realTbSuid;
tqDebug("%s suid:%"PRId64" realSuid:%"PRId64" return:%d", __FUNCTION__, tbSuid, realTbSuid, tmp);
tqDebug("%s suid:%" PRId64 " realSuid:%" PRId64 " return:%d", __FUNCTION__, tbSuid, realTbSuid, tmp);
return tmp;
}
@ -264,7 +266,8 @@ int32_t tqFetchLog(STQ* pTq, STqHandle* pHandle, int64_t* fetchOffset, uint64_t
END:
*fetchOffset = offset;
tqDebug("vgId:%d, end to fetch wal, code:%d , index:%" PRId64 ", last:%" PRId64 " commit:%" PRId64 ", applied:%" PRId64 ", 0x%" PRIx64,
tqDebug("vgId:%d, end to fetch wal, code:%d , index:%" PRId64 ", last:%" PRId64 " commit:%" PRId64
", applied:%" PRId64 ", 0x%" PRIx64,
vgId, code, offset, lastVer, committedVer, appliedVer, id);
return code;
}
@ -277,7 +280,7 @@ bool tqGetTablePrimaryKey(STqReader* pReader) {
}
void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) {
tqDebug("%s:%p uid:%"PRId64, __FUNCTION__ , pReader, uid);
tqDebug("%s:%p uid:%" PRId64, __FUNCTION__, pReader, uid);
if (pReader == NULL) {
return;
@ -292,7 +295,7 @@ void tqSetTablePrimaryKey(STqReader* pReader, int64_t uid) {
}
STqReader* tqReaderOpen(SVnode* pVnode) {
tqDebug("%s:%p", __FUNCTION__ , pVnode);
tqDebug("%s:%p", __FUNCTION__, pVnode);
if (pVnode == NULL) {
return NULL;
}
@ -324,7 +327,7 @@ STqReader* tqReaderOpen(SVnode* pVnode) {
}
void tqReaderClose(STqReader* pReader) {
tqDebug("%s:%p", __FUNCTION__ , pReader);
tqDebug("%s:%p", __FUNCTION__, pReader);
if (pReader == NULL) return;
// close wal reader
@ -345,6 +348,10 @@ void tqReaderClose(STqReader* pReader) {
blockDataDestroy(pReader->pResBlock);
taosHashCleanup(pReader->tbIdHash);
tDestroySubmitReq(&pReader->submit, TSDB_MSG_FLG_DECODE);
taosHashCleanup(pReader->vtSourceScanInfo.pVirtualTables);
taosHashCleanup(pReader->vtSourceScanInfo.pPhysicalTables);
taosLRUCacheCleanup(pReader->vtSourceScanInfo.pPhyTblSchemaCache);
taosMemoryFree(pReader);
}
@ -393,8 +400,8 @@ int32_t extractMsgFromWal(SWalReader* pReader, void** pItem, int64_t maxVer, con
return code;
}
} else if (pCont->msgType == TDMT_VND_DELETE) {
void* pBody = POINTER_SHIFT(pCont->body, sizeof(SMsgHead));
int32_t len = pCont->bodyLen - sizeof(SMsgHead);
void* pBody = POINTER_SHIFT(pCont->body, sizeof(SMsgHead));
int32_t len = pCont->bodyLen - sizeof(SMsgHead);
EStreamType blockType = STREAM_DELETE_DATA;
code = tqExtractDelDataBlock(pBody, len, ver, (void**)pItem, 0, blockType);
if (code == TSDB_CODE_SUCCESS) {
@ -412,14 +419,14 @@ int32_t extractMsgFromWal(SWalReader* pReader, void** pItem, int64_t maxVer, con
}
} else if (pCont->msgType == TDMT_VND_DROP_TABLE && pReader->cond.scanDropCtb) {
void* pBody = POINTER_SHIFT(pCont->body, sizeof(SMsgHead));
void* pBody = POINTER_SHIFT(pCont->body, sizeof(SMsgHead));
int32_t len = pCont->bodyLen - sizeof(SMsgHead);
code = tqExtractDropCtbDataBlock(pBody, len, ver, (void**)pItem, 0);
if (TSDB_CODE_SUCCESS == code) {
if (!*pItem) {
continue;
} else {
tqDebug("s-task:%s drop ctb msg extract from WAL, len:%d, ver:%"PRId64, id, len, ver);
tqDebug("s-task:%s drop ctb msg extract from WAL, len:%d, ver:%" PRId64, id, len, ver);
}
} else {
terrno = code;
@ -515,13 +522,12 @@ int32_t tqReaderSetSubmitMsg(STqReader* pReader, void* msgStr, int32_t msgLen, i
return code;
}
void tqReaderClearSubmitMsg(STqReader *pReader) {
void tqReaderClearSubmitMsg(STqReader* pReader) {
tDestroySubmitReq(&pReader->submit, TSDB_MSG_FLG_DECODE);
pReader->nextBlk = 0;
pReader->msg.msgStr = NULL;
}
SWalReader* tqGetWalReader(STqReader* pReader) {
if (pReader == NULL) {
return NULL;
@ -564,10 +570,10 @@ bool tqNextBlockImpl(STqReader* pReader, const char* idstr) {
}
tqReaderClearSubmitMsg(pReader);
tqTrace("iterator data block end, total block num:%d, uid:%"PRId64, blockSz, uid);
tqTrace("iterator data block end, total block num:%d, uid:%" PRId64, blockSz, uid);
END:
tqTrace("%s:%d return:%s, uid:%"PRId64, __FUNCTION__, lino, code?"true":"false", uid);
tqTrace("%s:%d return:%s, uid:%" PRId64, __FUNCTION__, lino, code ? "true" : "false", uid);
return code;
}
@ -591,10 +597,10 @@ bool tqNextDataBlockFilterOut(STqReader* pReader, SHashObj* filterOutUids) {
pReader->nextBlk++;
}
tqReaderClearSubmitMsg(pReader);
tqTrace("iterator data block end, total block num:%d, uid:%"PRId64, blockSz, uid);
tqTrace("iterator data block end, total block num:%d, uid:%" PRId64, blockSz, uid);
END:
tqTrace("%s:%d get data:%s, uid:%"PRId64, __FUNCTION__, lino, code?"true":"false", uid);
tqTrace("%s:%d get data:%s, uid:%" PRId64, __FUNCTION__, lino, code ? "true" : "false", uid);
return code;
}
@ -639,15 +645,15 @@ static int32_t buildResSDataBlock(STqReader* pReader, SSchemaWrapper* pSchema, c
}
SSDataBlock* pBlock = pReader->pResBlock;
if (blockDataGetNumOfCols(pBlock) > 0) {
blockDataDestroy(pBlock);
int32_t code = createDataBlock(&pReader->pResBlock);
if (code) {
return code;
}
pBlock = pReader->pResBlock;
blockDataDestroy(pBlock);
int32_t code = createDataBlock(&pReader->pResBlock);
if (code) {
return code;
}
pBlock = pReader->pResBlock;
pBlock->info.id.uid = pReader->cachedSchemaUid;
pBlock->info.version = pReader->msg.ver;
pBlock->info.id.uid = pReader->cachedSchemaUid;
pBlock->info.version = pReader->msg.ver;
}
int32_t numOfCols = taosArrayGetSize(pColIdList);
@ -947,7 +953,8 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData
TQ_NULL_GO_TO_END(pCol);
int32_t numOfRows = pCol->nVal;
int32_t numOfCols = taosArrayGetSize(pCols);
tqTrace("vgId:%d, tqProcessColData start, col num: %d, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols, numOfRows);
tqTrace("vgId:%d, tqProcessColData start, col num: %d, rows:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols,
numOfRows);
for (int32_t i = 0; i < numOfRows; i++) {
bool buildNew = false;
@ -987,7 +994,8 @@ static int32_t tqProcessColData(STqReader* pReader, SSubmitTbData* pSubmitTbData
}
SSDataBlock* pLastBlock = taosArrayGetLast(blocks);
pLastBlock->info.rows = curRow - lastRow;
tqTrace("vgId:%d, tqProcessColData end, col num: %d, rows:%d, block num:%d", pReader->pWalReader->pWal->cfg.vgId, numOfCols, numOfRows, (int)taosArrayGetSize(blocks));
tqTrace("vgId:%d, tqProcessColData end, col num: %d, rows:%d, block num:%d", pReader->pWalReader->pWal->cfg.vgId,
numOfCols, numOfRows, (int)taosArrayGetSize(blocks));
END:
if (code != TSDB_CODE_SUCCESS) {
tqError("vgId:%d, process col data failed, code:%d", pReader->pWalReader->pWal->cfg.vgId, code);
@ -1049,7 +1057,8 @@ int32_t tqProcessRowData(STqReader* pReader, SSubmitTbData* pSubmitTbData, SArra
SSDataBlock* pLastBlock = taosArrayGetLast(blocks);
pLastBlock->info.rows = curRow - lastRow;
tqTrace("vgId:%d, tqProcessRowData end, rows:%d, block num:%d", pReader->pWalReader->pWal->cfg.vgId, numOfRows, (int)taosArrayGetSize(blocks));
tqTrace("vgId:%d, tqProcessRowData end, rows:%d, block num:%d", pReader->pWalReader->pWal->cfg.vgId, numOfRows,
(int)taosArrayGetSize(blocks));
END:
if (code != TSDB_CODE_SUCCESS) {
tqError("vgId:%d, process row data failed, code:%d", pReader->pWalReader->pWal->cfg.vgId, code);
@ -1059,7 +1068,7 @@ END:
return code;
}
static int32_t buildCreateTbInfo(SMqDataRsp* pRsp, SVCreateTbReq* pCreateTbReq){
static int32_t buildCreateTbInfo(SMqDataRsp* pRsp, SVCreateTbReq* pCreateTbReq) {
int32_t code = 0;
int32_t lino = 0;
void* createReq = NULL;
@ -1089,15 +1098,16 @@ static int32_t buildCreateTbInfo(SMqDataRsp* pRsp, SVCreateTbReq* pCreateTbReq){
pRsp->createTableNum++;
tqTrace("build create table info msg success");
END:
if (code != 0){
END:
if (code != 0) {
tqError("%s failed at %d, failed to build create table info msg:%s", __FUNCTION__, lino, tstrerror(code));
taosMemoryFree(createReq);
}
return code;
}
int32_t tqRetrieveTaosxBlock(STqReader* pReader, SMqDataRsp* pRsp, SArray* blocks, SArray* schemas, SSubmitTbData** pSubmitTbDataRet, SArray* rawList, int8_t fetchMeta) {
int32_t tqRetrieveTaosxBlock(STqReader* pReader, SMqDataRsp* pRsp, SArray* blocks, SArray* schemas,
SSubmitTbData** pSubmitTbDataRet, SArray* rawList, int8_t fetchMeta) {
tqTrace("tq reader retrieve data block msg pointer:%p, index:%d", pReader->msg.msgStr, pReader->nextBlk);
SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk);
if (pSubmitTbData == NULL) {
@ -1111,13 +1121,13 @@ int32_t tqRetrieveTaosxBlock(STqReader* pReader, SMqDataRsp* pRsp, SArray* block
if (fetchMeta == ONLY_META) {
if (pSubmitTbData->pCreateTbReq != NULL) {
if (pRsp->createTableReq == NULL){
if (pRsp->createTableReq == NULL) {
pRsp->createTableReq = taosArrayInit(0, POINTER_BYTES);
if (pRsp->createTableReq == NULL){
if (pRsp->createTableReq == NULL) {
return terrno;
}
}
if (taosArrayPush(pRsp->createTableReq, &pSubmitTbData->pCreateTbReq) == NULL){
if (taosArrayPush(pRsp->createTableReq, &pSubmitTbData->pCreateTbReq) == NULL) {
return terrno;
}
pSubmitTbData->pCreateTbReq = NULL;
@ -1145,7 +1155,7 @@ int32_t tqRetrieveTaosxBlock(STqReader* pReader, SMqDataRsp* pRsp, SArray* block
return code;
}
} else if (rawList != NULL) {
if (taosArrayPush(schemas, &pReader->pSchemaWrapper) == NULL){
if (taosArrayPush(schemas, &pReader->pSchemaWrapper) == NULL) {
return terrno;
}
pReader->pSchemaWrapper = NULL;
@ -1159,16 +1169,17 @@ int32_t tqRetrieveTaosxBlock(STqReader* pReader, SMqDataRsp* pRsp, SArray* block
}
}
void tqReaderSetColIdList(STqReader* pReader, SArray* pColIdList) {
if (pReader == NULL){
return;
int32_t tqReaderSetColIdList(STqReader* pReader, SArray* pColIdList, const char* id) {
if (pReader == NULL) {
return TSDB_CODE_SUCCESS;
}
pReader->pColIdList = pColIdList;
return tqCollectPhysicalTables(pReader, id);
}
void tqReaderSetTbUidList(STqReader* pReader, const SArray* tbUidList, const char* id) {
int32_t tqReaderSetTbUidList(STqReader* pReader, const SArray* tbUidList, const char* id) {
if (pReader == NULL || tbUidList == NULL) {
return;
return TSDB_CODE_SUCCESS;
}
if (pReader->tbIdHash) {
taosHashClear(pReader->tbIdHash);
@ -1176,7 +1187,7 @@ void tqReaderSetTbUidList(STqReader* pReader, const SArray* tbUidList, const cha
pReader->tbIdHash = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), true, HASH_ENTRY_LOCK);
if (pReader->tbIdHash == NULL) {
tqError("s-task:%s failed to init hash table", id);
return;
return terrno;
}
}
@ -1189,6 +1200,7 @@ void tqReaderSetTbUidList(STqReader* pReader, const SArray* tbUidList, const cha
}
tqDebug("s-task:%s %d tables are set to be queried target table", id, (int32_t)taosArrayGetSize(tbUidList));
return tqCollectPhysicalTables(pReader, id);
}
void tqReaderAddTbUidList(STqReader* pReader, const SArray* pTableUidList) {
@ -1325,3 +1337,535 @@ int32_t tqUpdateTbUidList(STQ* pTq, const SArray* tbUidList, bool isAdd) {
streamMetaWUnLock(pTq->pStreamMeta);
return 0;
}
static void destroySourceScanTables(void* ptr) {
SArray** pTables = ptr;
if (pTables && *pTables) {
taosArrayDestroy(*pTables);
*pTables = NULL;
}
}
static int32_t compareSVTColInfo(const void* p1, const void* p2) {
SVTColInfo* pCol1 = (SVTColInfo*)p1;
SVTColInfo* pCol2 = (SVTColInfo*)p2;
if (pCol1->vColId == pCol2->vColId) {
return 0;
} else if (pCol1->vColId < pCol2->vColId) {
return -1;
} else {
return 1;
}
}
int32_t tqReaderSetVtableInfo(STqReader* pReader, void* vnode, void* ptr, SSHashObj* pVtableInfos,
SSDataBlock** ppResBlock, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SStorageAPI* pAPI = ptr;
SVTSourceScanInfo* pScanInfo = NULL;
SHashObj* pVirtualTables = NULL;
SMetaReader metaReader = {0};
SVTColInfo colInfo = {0};
SSchemaWrapper* schema = NULL;
TSDB_CHECK_NULL(pReader, code, lino, _end, TSDB_CODE_INVALID_PARA);
TSDB_CHECK_NULL(vnode, code, lino, _end, TSDB_CODE_INVALID_PARA);
TSDB_CHECK_NULL(pAPI, code, lino, _end, TSDB_CODE_INVALID_PARA);
pScanInfo = &pReader->vtSourceScanInfo;
taosHashCleanup(pScanInfo->pVirtualTables);
pScanInfo->pVirtualTables = NULL;
if (tSimpleHashGetSize(pVtableInfos) == 0) {
goto _end;
}
pVirtualTables = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK);
TSDB_CHECK_NULL(pVirtualTables, code, lino, _end, terrno);
taosHashSetFreeFp(pVirtualTables, destroySourceScanTables);
int32_t iter = 0;
void* px = tSimpleHashIterate(pVtableInfos, NULL, &iter);
while (px != NULL) {
int64_t vTbUid = *(int64_t*)tSimpleHashGetKey(px, NULL);
SArray* pColInfos = taosArrayInit(8, sizeof(SVTColInfo));
TSDB_CHECK_NULL(pColInfos, code, lino, _end, terrno);
code = taosHashPut(pVirtualTables, &vTbUid, sizeof(int64_t), &pColInfos, POINTER_BYTES);
TSDB_CHECK_CODE(code, lino, _end);
SSHashObj* pPhysicalTables = *(SSHashObj**)px;
int32_t iterIn = 0;
void* pxIn = tSimpleHashIterate(pPhysicalTables, NULL, &iterIn);
while (pxIn != NULL) {
char* physicalTableName = tSimpleHashGetKey(pxIn, NULL);
pAPI->metaReaderFn.clearReader(&metaReader);
pAPI->metaReaderFn.initReader(&metaReader, vnode, META_READER_LOCK, &pAPI->metaFn);
code = pAPI->metaReaderFn.getTableEntryByName(&metaReader, physicalTableName);
TSDB_CHECK_CODE(code, lino, _end);
pAPI->metaReaderFn.readerReleaseLock(&metaReader);
colInfo.pTbUid = metaReader.me.uid;
switch (metaReader.me.type) {
case TSDB_CHILD_TABLE: {
int64_t suid = metaReader.me.ctbEntry.suid;
pAPI->metaReaderFn.clearReader(&metaReader);
pAPI->metaReaderFn.initReader(&metaReader, vnode, META_READER_LOCK, &pAPI->metaFn);
code = pAPI->metaReaderFn.getTableEntryByUid(&metaReader, suid);
TSDB_CHECK_CODE(code, lino, _end);
pAPI->metaReaderFn.readerReleaseLock(&metaReader);
schema = &metaReader.me.stbEntry.schemaRow;
break;
}
case TSDB_NORMAL_TABLE: {
schema = &metaReader.me.ntbEntry.schemaRow;
break;
}
default: {
tqError("invalid table type: %d", metaReader.me.type);
code = TSDB_CODE_INVALID_PARA;
TSDB_CHECK_CODE(code, lino, _end);
}
}
SArray* pCols = *(SArray**)pxIn;
int32_t ncols = taosArrayGetSize(pCols);
for (int32_t i = 0; i < ncols; ++i) {
SColIdName* pCol = taosArrayGet(pCols, i);
colInfo.vColId = pCol->colId;
for (int32_t j = 0; j < schema->nCols; ++j) {
if (strncmp(pCol->colName, schema->pSchema[j].name, strlen(schema->pSchema[j].name)) == 0) {
colInfo.pColId = schema->pSchema[j].colId;
void* px = taosArrayPush(pColInfos, &colInfo);
TSDB_CHECK_NULL(px, code, lino, _end, terrno);
break;
}
}
}
taosArraySort(pColInfos, compareSVTColInfo);
pxIn = tSimpleHashIterate(pPhysicalTables, pxIn, &iterIn);
}
px = tSimpleHashIterate(pVtableInfos, px, &iter);
}
pScanInfo->pVirtualTables = pVirtualTables;
pVirtualTables = NULL;
// set the result data block
if (pReader->pResBlock) {
blockDataDestroy(pReader->pResBlock);
}
pReader->pResBlock = *ppResBlock;
*ppResBlock = NULL;
// update reader callback for vtable source scan
pAPI->tqReaderFn.tqRetrieveBlock = tqRetrieveVTableDataBlock;
pAPI->tqReaderFn.tqNextBlockImpl = tqNextVTableSourceBlockImpl;
pAPI->tqReaderFn.tqReaderIsQueriedTable = tqReaderIsQueriedSourceTable;
_end:
if (code != TSDB_CODE_SUCCESS) {
tqError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
pAPI->metaReaderFn.clearReader(&metaReader);
if (pVirtualTables != NULL) {
taosHashCleanup(pVirtualTables);
}
return code;
}
static int32_t tqCollectPhysicalTables(STqReader* pReader, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SVTSourceScanInfo* pScanInfo = NULL;
SHashObj* pVirtualTables = NULL;
SHashObj* pPhysicalTables = NULL;
void* pIter = NULL;
void* px = NULL;
TSDB_CHECK_NULL(pReader, code, lino, _end, TSDB_CODE_INVALID_PARA);
pScanInfo = &pReader->vtSourceScanInfo;
taosHashCleanup(pScanInfo->pPhysicalTables);
pScanInfo->pPhysicalTables = NULL;
taosLRUCacheCleanup(pScanInfo->pPhyTblSchemaCache);
pScanInfo->pPhyTblSchemaCache = NULL;
pScanInfo->nextVirtualTableIdx = -1;
pScanInfo->metaFetch = 0;
pScanInfo->cacheHit = 0;
pVirtualTables = pScanInfo->pVirtualTables;
if (taosHashGetSize(pVirtualTables) == 0 || taosHashGetSize(pReader->tbIdHash) == 0 ||
taosArrayGetSize(pReader->pColIdList) == 0) {
goto _end;
}
pPhysicalTables = taosHashInit(64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK);
TSDB_CHECK_NULL(pPhysicalTables, code, lino, _end, terrno);
taosHashSetFreeFp(pPhysicalTables, destroySourceScanTables);
pIter = taosHashIterate(pReader->tbIdHash, NULL);
while (pIter != NULL) {
int64_t vTbUid = *(int64_t*)taosHashGetKey(pIter, NULL);
px = taosHashGet(pVirtualTables, &vTbUid, sizeof(int64_t));
TSDB_CHECK_NULL(px, code, lino, _end, terrno);
SArray* pColInfos = *(SArray**)px;
TSDB_CHECK_NULL(pColInfos, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
// Traverse all required columns and collect corresponding physical tables
int32_t nColInfos = taosArrayGetSize(pColInfos);
int32_t nOutputCols = taosArrayGetSize(pReader->pColIdList);
for (int32_t i = 0, j = 0; i < nColInfos && j < nOutputCols;) {
SVTColInfo* pCol = taosArrayGet(pColInfos, i);
col_id_t colIdNeed = *(col_id_t*)taosArrayGet(pReader->pColIdList, j);
if (pCol->vColId < colIdNeed) {
i++;
} else if (pCol->vColId > colIdNeed) {
j++;
} else {
SArray* pRelatedVTs = NULL;
px = taosHashGet(pPhysicalTables, &pCol->pTbUid, sizeof(int64_t));
if (px == NULL) {
pRelatedVTs = taosArrayInit(8, sizeof(int64_t));
TSDB_CHECK_NULL(pRelatedVTs, code, lino, _end, terrno);
code = taosHashPut(pPhysicalTables, &pCol->pTbUid, sizeof(int64_t), &pRelatedVTs, POINTER_BYTES);
if (code != TSDB_CODE_SUCCESS) {
taosArrayDestroy(pRelatedVTs);
TSDB_CHECK_CODE(code, lino, _end);
}
} else {
pRelatedVTs = *(SArray**)px;
}
if (taosArrayGetSize(pRelatedVTs) == 0 || *(int64_t*)taosArrayGetLast(pRelatedVTs) != vTbUid) {
px = taosArrayPush(pRelatedVTs, &vTbUid);
TSDB_CHECK_NULL(px, code, lino, _end, terrno);
}
i++;
j++;
}
}
pIter = taosHashIterate(pReader->tbIdHash, pIter);
}
pScanInfo->pPhysicalTables = pPhysicalTables;
pPhysicalTables = NULL;
if (taosHashGetSize(pScanInfo->pPhysicalTables) > 0) {
pScanInfo->pPhyTblSchemaCache = taosLRUCacheInit(1024 * 128, -1, .5);
TSDB_CHECK_NULL(pScanInfo->pPhyTblSchemaCache, code, lino, _end, terrno);
}
_end:
if (code != TSDB_CODE_SUCCESS) {
tqError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (pIter != NULL) {
taosHashCancelIterate(pReader->tbIdHash, pIter);
}
if (pPhysicalTables != NULL) {
taosHashCleanup(pPhysicalTables);
}
return code;
}
static void freeTableSchemaCache(const void* key, size_t keyLen, void* value, void* ud) {
if (value) {
SSchemaWrapper** ppSchemaWrapper = value;
tDeleteSchemaWrapper(*ppSchemaWrapper);
*ppSchemaWrapper = NULL;
}
}
int32_t tqRetrieveVTableDataBlock(STqReader* pReader, SSDataBlock** pRes, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SVTSourceScanInfo* pScanInfo = NULL;
SSubmitTbData* pSubmitTbData = NULL;
SSDataBlock* pBlock = NULL;
void* px = NULL;
int64_t vTbUid = 0;
int64_t pTbUid = 0;
LRUHandle* h = NULL;
STSchema* pPhyTblSchema = NULL;
TSDB_CHECK_NULL(pReader, code, lino, _end, TSDB_CODE_INVALID_PARA);
TSDB_CHECK_NULL(pRes, code, lino, _end, TSDB_CODE_INVALID_PARA);
pScanInfo = &pReader->vtSourceScanInfo;
tqDebug("tq reader retrieve vtable data block from %p, nextBlk:%d, vtbIdx:%d, id:%s", pReader->msg.msgStr,
pReader->nextBlk, pScanInfo->nextVirtualTableIdx, idstr);
*pRes = NULL;
pBlock = pReader->pResBlock;
blockDataCleanup(pBlock);
pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk);
TSDB_CHECK_NULL(pSubmitTbData, code, lino, _end, terrno);
pReader->lastTs = pSubmitTbData->ctimeMs;
pTbUid = pSubmitTbData->uid;
px = taosHashGet(pScanInfo->pPhysicalTables, &pTbUid, sizeof(int64_t));
TSDB_CHECK_NULL(px, code, lino, _end, terrno);
SArray* pRelatedVTs = *(SArray**)px;
vTbUid = *(int64_t*)taosArrayGet(pRelatedVTs, pScanInfo->nextVirtualTableIdx);
px = taosHashGet(pScanInfo->pVirtualTables, &vTbUid, sizeof(int64_t));
TSDB_CHECK_NULL(px, code, lino, _end, terrno);
SArray* pColInfos = *(SArray**)px;
TSDB_CHECK_NULL(pColInfos, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
int32_t nColInfos = taosArrayGetSize(pColInfos);
int32_t nOutputCols = taosArrayGetSize(pBlock->pDataBlock);
int32_t numOfRows = 0;
int32_t nInputCols = 0;
if (pSubmitTbData->flags & SUBMIT_REQ_COLUMN_DATA_FORMAT) {
SColData* pCol = taosArrayGet(pSubmitTbData->aCol, 0);
TSDB_CHECK_NULL(pCol, code, lino, _end, terrno);
numOfRows = pCol->nVal;
nInputCols = taosArrayGetSize(pSubmitTbData->aCol);
} else {
// try to get physical table schema from cache
pScanInfo->metaFetch++;
int64_t cacheKey = (pSubmitTbData->suid == 0) ? pTbUid : pSubmitTbData->suid;
SSchemaWrapper* pWrapper = NULL;
h = taosLRUCacheLookup(pScanInfo->pPhyTblSchemaCache, &cacheKey, sizeof(int64_t));
if (h != NULL) {
pWrapper = taosLRUCacheValue(pScanInfo->pPhyTblSchemaCache, h);
TSDB_CHECK_NULL(pWrapper, code, lino, _end, terrno);
}
if (pWrapper != NULL && pWrapper->version != pSubmitTbData->sver) {
// reset outdated schema
tDeleteSchemaWrapper(pWrapper);
pWrapper = NULL;
taosLRUCacheUpdate(pScanInfo->pPhyTblSchemaCache, h, pWrapper);
}
if (pWrapper == NULL) {
// get physical table schema from meta
pWrapper = metaGetTableSchema(pReader->pVnodeMeta, pTbUid, pSubmitTbData->sver, 1, NULL);
if (pWrapper == NULL) {
tqWarn("vgId:%d, cannot found schema wrapper for table: suid:%" PRId64 ", uid:%" PRId64
"version %d, possibly dropped table",
pReader->pWalReader->pWal->cfg.vgId, pSubmitTbData->suid, pTbUid, pSubmitTbData->sver);
TSDB_CHECK_NULL(pWrapper, code, lino, _end, TSDB_CODE_TQ_TABLE_SCHEMA_NOT_FOUND);
}
if (h == NULL) {
// insert schema to cache
code = taosLRUCacheInsert(pScanInfo->pPhyTblSchemaCache, &cacheKey, sizeof(int64_t), pWrapper, POINTER_BYTES,
freeTableSchemaCache, NULL, NULL, TAOS_LRU_PRIORITY_LOW, NULL);
if (code != TSDB_CODE_SUCCESS) {
tDeleteSchemaWrapper(pWrapper);
}
TSDB_CHECK_CODE(code, lino, _end);
} else {
// update schema in cache
taosLRUCacheUpdate(pScanInfo->pPhyTblSchemaCache, h, pWrapper);
}
} else {
pScanInfo->cacheHit++;
}
TSDB_CHECK_NULL(pWrapper, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
pPhyTblSchema = tBuildTSchema(pWrapper->pSchema, pWrapper->nCols, pWrapper->version);
TSDB_CHECK_NULL(pPhyTblSchema, code, lino, _end, terrno);
numOfRows = taosArrayGetSize(pSubmitTbData->aRowP);
nInputCols = pPhyTblSchema->numOfCols;
}
code = blockDataEnsureCapacity(pBlock, numOfRows);
TSDB_CHECK_CODE(code, lino, _end);
// convert one block
for (int32_t i = 0, j = 1; j < nOutputCols;) {
SColumnInfoData* pOutCol = taosArrayGet(pBlock->pDataBlock, j);
TSDB_CHECK_NULL(pOutCol, code, lino, _end, terrno);
if (i >= nColInfos) {
tqInfo("%s has %d column info, but vtable column %d is missing, id: %s", __func__, nColInfos, pOutCol->info.colId,
idstr);
colDataSetNNULL(pOutCol, 0, numOfRows);
j++;
continue;
}
SVTColInfo* pCol = taosArrayGet(pColInfos, i);
TSDB_CHECK_NULL(pCol, code, lino, _end, terrno);
if (pCol->vColId < pOutCol->info.colId) {
i++;
continue;
} else if (pCol->vColId > pOutCol->info.colId) {
tqInfo("%s does not find column info for vtable column %d, closest vtable column is %d, id: %s", __func__,
pOutCol->info.colId, pCol->vColId, idstr);
colDataSetNNULL(pOutCol, 0, numOfRows);
j++;
continue;
}
// copy data from physical table to the result block of virtual table
if (pCol->pTbUid != pTbUid) {
// skip this column since it is from another physical table
} else if (pSubmitTbData->flags & SUBMIT_REQ_COLUMN_DATA_FORMAT) {
// try to find the corresponding column data of physical table
SColData* pColData = NULL;
for (int32_t k = 0; k < nInputCols; ++k) {
pColData = taosArrayGet(pSubmitTbData->aCol, k);
TSDB_CHECK_NULL(pColData, code, lino, _end, terrno);
if (pColData->cid == pCol->pColId) {
break;
}
pColData = NULL;
}
if (pColData == NULL) {
tqError("%s does not find data of physical table %" PRId64 " column %d, virtual table: %" PRId64
" column: %d, id: %s",
__func__, pTbUid, pCol->pColId, vTbUid, pCol->vColId, idstr);
colDataSetNNULL(pOutCol, 0, numOfRows);
i++;
j++;
continue;
}
SColVal colVal = {0};
for (int32_t k = 0; k < pColData->nVal; ++k) {
code = tColDataGetValue(pColData, k, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
code = doSetVal(pOutCol, k, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
}
} else {
SArray* pRows = pSubmitTbData->aRowP;
TSDB_CHECK_NULL(pPhyTblSchema, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
SColVal colVal = {0};
for (int32_t k = 0; k < numOfRows; ++k) {
SRow* pRow = taosArrayGetP(pRows, k);
TSDB_CHECK_NULL(pRow, code, lino, _end, terrno);
for (int32_t l = 0; l < nInputCols; ++l) {
code = tRowGet(pRow, pPhyTblSchema, l, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
if (colVal.cid == pCol->pColId) {
code = doSetVal(pOutCol, k, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
break;
} else if (colVal.cid > pCol->pColId || l == (nInputCols - 1)) {
colDataSetNULL(pOutCol, k);
break;
}
}
}
}
i++;
j++;
}
// enforce to fill the first ts column
if (pSubmitTbData->flags & SUBMIT_REQ_COLUMN_DATA_FORMAT) {
SColumnInfoData* pOutCol = taosArrayGet(pBlock->pDataBlock, 0);
SColData* pColData = taosArrayGet(pSubmitTbData->aCol, 0);
TSDB_CHECK_NULL(pColData, code, lino, _end, terrno);
SColVal colVal = {0};
for (int32_t k = 0; k < pColData->nVal; ++k) {
code = tColDataGetValue(pColData, k, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
code = doSetVal(pOutCol, k, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
}
} else {
SColumnInfoData* pOutCol = taosArrayGet(pBlock->pDataBlock, 0);
SArray* pRows = pSubmitTbData->aRowP;
TSDB_CHECK_NULL(pPhyTblSchema, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
SColVal colVal = {0};
for (int32_t k = 0; k < numOfRows; ++k) {
SRow* pRow = taosArrayGetP(pRows, k);
TSDB_CHECK_NULL(pRow, code, lino, _end, terrno);
code = tRowGet(pRow, pPhyTblSchema, 0, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
code = doSetVal(pOutCol, k, &colVal);
TSDB_CHECK_CODE(code, lino, _end);
}
}
pBlock->info.rows = numOfRows;
pBlock->info.id.uid = vTbUid;
pBlock->info.id.groupId = pTbUid;
pBlock->info.version = pReader->msg.ver;
pScanInfo->nextVirtualTableIdx++;
if (pScanInfo->nextVirtualTableIdx >= taosArrayGetSize(pRelatedVTs)) {
pReader->nextBlk++;
pScanInfo->nextVirtualTableIdx = -1;
}
tqDebug("tq reader will retrieve next vtable data block from %p, nextBlk:%d, vtbIdx:%d, id:%s", pReader->msg.msgStr,
pReader->nextBlk, pScanInfo->nextVirtualTableIdx, idstr);
*pRes = pBlock;
_end:
if (code != TSDB_CODE_SUCCESS) {
tqError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (h != NULL) {
bool bRes = taosLRUCacheRelease(pScanInfo->pPhyTblSchemaCache, h, false);
tqTrace("release LRU cache, res %d, id: %s", bRes, idstr);
}
if (pPhyTblSchema != NULL) {
taosMemoryFreeClear(pPhyTblSchema);
}
return code;
}
bool tqNextVTableSourceBlockImpl(STqReader* pReader, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SVTSourceScanInfo* pScanInfo = NULL;
TSDB_CHECK_NULL(pReader, code, lino, _end, TSDB_CODE_INVALID_PARA);
pScanInfo = &pReader->vtSourceScanInfo;
if (pReader->msg.msgStr == NULL || taosHashGetSize(pScanInfo->pPhysicalTables) == 0) {
return false;
}
if (pScanInfo->nextVirtualTableIdx >= 0) {
// The data still needs to be converted into the virtual table result block
return true;
}
int32_t blockSz = taosArrayGetSize(pReader->submit.aSubmitTbData);
while (pReader->nextBlk < blockSz) {
SSubmitTbData* pSubmitTbData = taosArrayGet(pReader->submit.aSubmitTbData, pReader->nextBlk);
TSDB_CHECK_NULL(pSubmitTbData, code, lino, _end, terrno);
int64_t pTbUid = pSubmitTbData->uid;
void* px = taosHashGet(pScanInfo->pPhysicalTables, &pTbUid, sizeof(int64_t));
if (px != NULL) {
SArray* pRelatedVTs = *(SArray**)px;
if (taosArrayGetSize(pRelatedVTs) > 0) {
pScanInfo->nextVirtualTableIdx = 0;
return true;
}
}
tqTrace("iterator data block in hash jump block, progress:%d/%d, uid:%" PRId64 "", pReader->nextBlk, blockSz,
pTbUid);
pReader->nextBlk++;
}
tqReaderClearSubmitMsg(pReader);
tqTrace("iterator data block end, total block num:%d", blockSz);
_end:
if (code != TSDB_CODE_SUCCESS) {
tqError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
return (code == TSDB_CODE_SUCCESS);
}
bool tqReaderIsQueriedSourceTable(STqReader* pReader, uint64_t uid) {
if (pReader == NULL) {
return false;
}
return taosHashGet(pReader->vtSourceScanInfo.pPhysicalTables, &uid, sizeof(uint64_t)) != NULL;
}

2
source/dnode/vnode/src/vnd/vnodeInitApi.c
View File

@ -150,6 +150,8 @@ void initTqAPI(SStoreTqReader* pTq) {
pTq->tqGetResultBlockTime = tqGetResultBlockTime;
pTq->tqGetStreamExecProgress = tqGetStreamExecInfo;
pTq->tqReaderSetVtableInfo = tqReaderSetVtableInfo;
#endif
}

281
source/dnode/vnode/src/vnd/vnodeQuery.c
View File

@ -77,6 +77,28 @@ void vnodePrintTableMeta(STableMetaRsp *pMeta) {
}
}
int32_t fillTableColRef(SMetaReader *reader, SColRef *pRef, int32_t numOfCol) {
int8_t tblType = reader->me.type;
if (hasRefCol(tblType)) {
SColRefWrapper *p = &(reader->me.colRef);
if (numOfCol != p->nCols) {
vError("fillTableColRef table type:%d, col num:%d, col cmpr num:%d mismatch", tblType, numOfCol, p->nCols);
return TSDB_CODE_APP_ERROR;
}
for (int i = 0; i < p->nCols; i++) {
SColRef *pColRef = &p->pColRef[i];
pRef[i].hasRef = pColRef->hasRef;
pRef[i].id = pColRef->id;
if(pRef[i].hasRef) {
tstrncpy(pRef[i].refDbName, pColRef->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(pRef[i].refTableName, pColRef->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pRef[i].refColName, pColRef->refColName, TSDB_COL_NAME_LEN);
}
}
}
return 0;
}
int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
STableInfoReq infoReq = {0};
STableMetaRsp metaRsp = {0};
@ -139,24 +161,34 @@ int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
metaRsp.vgId = TD_VID(pVnode);
metaRsp.tuid = mer1.me.uid;
if (mer1.me.type == TSDB_SUPER_TABLE) {
tstrncpy(metaRsp.stbName, mer1.me.name, TSDB_TABLE_NAME_LEN);
schema = mer1.me.stbEntry.schemaRow;
schemaTag = mer1.me.stbEntry.schemaTag;
metaRsp.suid = mer1.me.uid;
} else if (mer1.me.type == TSDB_CHILD_TABLE) {
metaReaderDoInit(&mer2, pVnode->pMeta, META_READER_NOLOCK);
if (metaReaderGetTableEntryByUid(&mer2, mer1.me.ctbEntry.suid) < 0) goto _exit2;
switch (mer1.me.type) {
case TSDB_SUPER_TABLE: {
(void)strcpy(metaRsp.stbName, mer1.me.name);
schema = mer1.me.stbEntry.schemaRow;
schemaTag = mer1.me.stbEntry.schemaTag;
metaRsp.suid = mer1.me.uid;
break;
}
case TSDB_CHILD_TABLE:
case TSDB_VIRTUAL_CHILD_TABLE:{
metaReaderDoInit(&mer2, pVnode->pMeta, META_READER_NOLOCK);
if (metaReaderGetTableEntryByUid(&mer2, mer1.me.ctbEntry.suid) < 0) goto _exit2;
tstrncpy(metaRsp.stbName, mer2.me.name, TSDB_TABLE_NAME_LEN);
metaRsp.suid = mer2.me.uid;
schema = mer2.me.stbEntry.schemaRow;
schemaTag = mer2.me.stbEntry.schemaTag;
} else if (mer1.me.type == TSDB_NORMAL_TABLE) {
schema = mer1.me.ntbEntry.schemaRow;
} else {
vError("vnodeGetTableMeta get invalid table type:%d", mer1.me.type);
goto _exit3;
(void)strcpy(metaRsp.stbName, mer2.me.name);
metaRsp.suid = mer2.me.uid;
schema = mer2.me.stbEntry.schemaRow;
schemaTag = mer2.me.stbEntry.schemaTag;
break;
}
case TSDB_NORMAL_TABLE:
case TSDB_VIRTUAL_NORMAL_TABLE: {
schema = mer1.me.ntbEntry.schemaRow;
break;
}
default: {
vError("vnodeGetTableMeta get invalid table type:%d", mer1.me.type);
goto _exit3;
}
}
metaRsp.numOfTags = schemaTag.nCols;
@ -187,6 +219,19 @@ int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
code = TSDB_CODE_OUT_OF_MEMORY;
goto _exit;
}
if (hasRefCol(mer1.me.type)) {
metaRsp.pColRefs = (SColRef*)taosMemoryMalloc(sizeof(SColRef) * metaRsp.numOfColumns);
if (metaRsp.pColRefs) {
code = fillTableColRef(&mer1, metaRsp.pColRefs, metaRsp.numOfColumns);
if (code < 0) {
goto _exit;
}
}
metaRsp.numOfColRefs = metaRsp.numOfColumns;
} else {
metaRsp.pColRefs = NULL;
metaRsp.numOfColRefs = 0;
}
vnodePrintTableMeta(&metaRsp);
@ -215,6 +260,7 @@ int32_t vnodeGetTableMeta(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
}
_exit:
taosMemoryFree(metaRsp.pColRefs);
taosMemoryFree(metaRsp.pSchemas);
taosMemoryFree(metaRsp.pSchemaExt);
_exit2:
@ -286,7 +332,7 @@ int32_t vnodeGetTableCfg(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
if (mer1.me.type == TSDB_SUPER_TABLE) {
code = TSDB_CODE_VND_HASH_MISMATCH;
goto _exit;
} else if (mer1.me.type == TSDB_CHILD_TABLE) {
} else if (mer1.me.type == TSDB_CHILD_TABLE || mer1.me.type == TSDB_VIRTUAL_CHILD_TABLE) {
metaReaderDoInit(&mer2, pVnode->pMeta, META_READER_NOLOCK);
if (metaReaderGetTableEntryByUid(&mer2, mer1.me.ctbEntry.suid) < 0) goto _exit;
@ -310,7 +356,7 @@ int32_t vnodeGetTableCfg(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
goto _exit;
}
(void)memcpy(cfgRsp.pTags, pTag, cfgRsp.tagsLen);
} else if (mer1.me.type == TSDB_NORMAL_TABLE) {
} else if (mer1.me.type == TSDB_NORMAL_TABLE || mer1.me.type == TSDB_VIRTUAL_NORMAL_TABLE) {
schema = mer1.me.ntbEntry.schemaRow;
cfgRsp.ttl = mer1.me.ntbEntry.ttlDays;
cfgRsp.commentLen = mer1.me.ntbEntry.commentLen;
@ -329,10 +375,12 @@ int32_t vnodeGetTableCfg(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
cfgRsp.numOfTags = schemaTag.nCols;
cfgRsp.numOfColumns = schema.nCols;
cfgRsp.virtualStb = false; // vnode don't have super table, so it's always false
cfgRsp.pSchemas = (SSchema *)taosMemoryMalloc(sizeof(SSchema) * (cfgRsp.numOfColumns + cfgRsp.numOfTags));
cfgRsp.pSchemaExt = (SSchemaExt *)taosMemoryMalloc(cfgRsp.numOfColumns * sizeof(SSchemaExt));
cfgRsp.pColRefs = (SColRef *)taosMemoryMalloc(sizeof(SColRef) * cfgRsp.numOfColumns);
if (NULL == cfgRsp.pSchemas || NULL == cfgRsp.pSchemaExt) {
if (NULL == cfgRsp.pSchemas || NULL == cfgRsp.pSchemaExt || NULL == cfgRsp.pColRefs) {
code = terrno;
goto _exit;
}
@ -341,22 +389,36 @@ int32_t vnodeGetTableCfg(SVnode *pVnode, SRpcMsg *pMsg, bool direct) {
(void)memcpy(cfgRsp.pSchemas + schema.nCols, schemaTag.pSchema, sizeof(SSchema) * schemaTag.nCols);
}
// if (useCompress(cfgRsp.tableType)) {
SMetaReader *pReader = mer1.me.type == TSDB_CHILD_TABLE ? &mer2 : &mer1;
SMetaReader *pReader = (mer1.me.type == TSDB_CHILD_TABLE || mer1.me.type == TSDB_VIRTUAL_CHILD_TABLE) ? &mer2 : &mer1;
SColCmprWrapper *pColCmpr = &pReader->me.colCmpr;
SColRefWrapper *pColRef = &mer1.me.colRef;
for (int32_t i = 0; i < cfgRsp.numOfColumns; i++) {
SColCmpr *pCmpr = &pColCmpr->pColCmpr[i];
SSchemaExt *pSchExt = cfgRsp.pSchemaExt + i;
pSchExt->colId = pCmpr->id;
pSchExt->compress = pCmpr->alg;
if (pReader->me.pExtSchemas)
pSchExt->typeMod = pReader->me.pExtSchemas[i].typeMod;
else
pSchExt->typeMod = 0;
if (withExtSchema(cfgRsp.tableType)) {
for (int32_t i = 0; i < cfgRsp.numOfColumns; i++) {
SColCmpr *pCmpr = &pColCmpr->pColCmpr[i];
SSchemaExt *pSchExt = cfgRsp.pSchemaExt + i;
pSchExt->colId = pCmpr->id;
pSchExt->compress = pCmpr->alg;
if (pReader->me.pExtSchemas)
pSchExt->typeMod = pReader->me.pExtSchemas[i].typeMod;
else
pSchExt->typeMod = 0;
}
}
cfgRsp.virtualStb = false;
if (hasRefCol(cfgRsp.tableType)) {
for (int32_t i = 0; i < cfgRsp.numOfColumns; i++) {
SColRef *pRef = &pColRef->pColRef[i];
cfgRsp.pColRefs[i].hasRef = pRef->hasRef;
cfgRsp.pColRefs[i].id = pRef->id;
if (cfgRsp.pColRefs[i].hasRef) {
tstrncpy(cfgRsp.pColRefs[i].refDbName, pRef->refDbName, TSDB_DB_NAME_LEN);
tstrncpy(cfgRsp.pColRefs[i].refTableName, pRef->refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(cfgRsp.pColRefs[i].refColName, pRef->refColName, TSDB_COL_NAME_LEN);
}
}
}
//}
// encode and send response
rspLen = tSerializeSTableCfgRsp(NULL, 0, &cfgRsp);
@ -547,6 +609,159 @@ _exit:
(void)taosThreadRwlockUnlock(&(pVnode)->metaRWLock); \
} while (0)
int32_t vnodeReadVSubtables(SReadHandle* pHandle, int64_t suid, SArray** ppRes) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t line = 0;
SMetaReader mr = {0};
bool readerInit = false;
SVCTableRefCols* pTb = NULL;
int32_t refColsNum = 0;
char tbFName[TSDB_TABLE_FNAME_LEN];
SArray *pList = taosArrayInit(10, sizeof(uint64_t));
QUERY_CHECK_NULL(pList, code, line, _return, terrno);
QUERY_CHECK_CODE(pHandle->api.metaFn.getChildTableList(pHandle->vnode, suid, pList), line, _return);
size_t num = taosArrayGetSize(pList);
*ppRes = taosArrayInit(num, POINTER_BYTES);
QUERY_CHECK_NULL(*ppRes, code, line, _return, terrno);
SSHashObj* pSrcTbls = tSimpleHashInit(10, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY));
QUERY_CHECK_NULL(pSrcTbls, code, line, _return, terrno);
for (int32_t i = 0; i < num; ++i) {
uint64_t* id = taosArrayGet(pList, i);
QUERY_CHECK_NULL(id, code, line, _return, terrno);
pHandle->api.metaReaderFn.initReader(&mr, pHandle->vnode, META_READER_LOCK, &pHandle->api.metaFn);
QUERY_CHECK_CODE(pHandle->api.metaReaderFn.getTableEntryByUid(&mr, *id), line, _return);
readerInit = true;
refColsNum = 0;
for (int32_t j = 0; j < mr.me.colRef.nCols; j++) {
if (mr.me.colRef.pColRef[j].hasRef) {
refColsNum++;
}
}
if (refColsNum <= 0) {
pHandle->api.metaReaderFn.clearReader(&mr);
readerInit = false;
continue;
}
pTb = taosMemoryCalloc(1, refColsNum * sizeof(SRefColInfo) + sizeof(*pTb));
QUERY_CHECK_NULL(pTb, code, line, _return, terrno);
pTb->uid = mr.me.uid;
pTb->numOfColRefs = refColsNum;
pTb->refCols = (SRefColInfo*)(pTb + 1);
refColsNum = 0;
tSimpleHashClear(pSrcTbls);
for (int32_t j = 0; j < mr.me.colRef.nCols; j++) {
if (!mr.me.colRef.pColRef[j].hasRef) {
continue;
}
pTb->refCols[refColsNum].colId = mr.me.colRef.pColRef[j].id;
tstrncpy(pTb->refCols[refColsNum].refColName, mr.me.colRef.pColRef[j].refColName, TSDB_COL_NAME_LEN);
tstrncpy(pTb->refCols[refColsNum].refTableName, mr.me.colRef.pColRef[j].refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pTb->refCols[refColsNum].refDbName, mr.me.colRef.pColRef[j].refDbName, TSDB_DB_NAME_LEN);
snprintf(tbFName, sizeof(tbFName), "%s.%s", pTb->refCols[refColsNum].refDbName, pTb->refCols[refColsNum].refTableName);
if (NULL == tSimpleHashGet(pSrcTbls, tbFName, strlen(tbFName))) {
QUERY_CHECK_CODE(tSimpleHashPut(pSrcTbls, tbFName, strlen(tbFName), &code, sizeof(code)), line, _return);
}
refColsNum++;
}
pTb->numOfSrcTbls = tSimpleHashGetSize(pSrcTbls);
QUERY_CHECK_NULL(taosArrayPush(*ppRes, &pTb), code, line, _return, terrno);
pTb = NULL;
pHandle->api.metaReaderFn.clearReader(&mr);
readerInit = false;
}
_return:
if (readerInit) {
pHandle->api.metaReaderFn.clearReader(&mr);
}
taosArrayDestroy(pList);
taosMemoryFree(pTb);
tSimpleHashCleanup(pSrcTbls);
if (code) {
qError("%s failed since %s", __func__, tstrerror(code));
}
return code;
}
int32_t vnodeGetVSubtablesMeta(SVnode *pVnode, SRpcMsg *pMsg) {
int32_t code = 0;
int32_t rspSize = 0;
SVSubTablesReq req = {0};
SVSubTablesRsp rsp = {0};
SRpcMsg rspMsg = {0};
void *pRsp = NULL;
int32_t line = 0;
if (tDeserializeSVSubTablesReq(pMsg->pCont, pMsg->contLen, &req)) {
code = terrno;
qError("tDeserializeSVSubTablesReq failed");
goto _return;
}
SReadHandle handle = {.vnode = pVnode};
initStorageAPI(&handle.api);
QUERY_CHECK_CODE(vnodeReadVSubtables(&handle, req.suid, &rsp.pTables), line, _return);
rsp.vgId = TD_VID(pVnode);
rspSize = tSerializeSVSubTablesRsp(NULL, 0, &rsp);
if (rspSize < 0) {
code = rspSize;
qError("tSerializeSVSubTablesRsp failed, error:%d", rspSize);
goto _return;
}
pRsp = rpcMallocCont(rspSize);
if (pRsp == NULL) {
code = terrno;
qError("rpcMallocCont %d failed, error:%d", rspSize, terrno);
goto _return;
}
rspSize = tSerializeSVSubTablesRsp(pRsp, rspSize, &rsp);
if (rspSize < 0) {
code = rspSize;
qError("tSerializeSVSubTablesRsp failed, error:%d", rspSize);
goto _return;
}
_return:
rspMsg.info = pMsg->info;
rspMsg.pCont = pRsp;
rspMsg.contLen = rspSize;
rspMsg.code = code;
rspMsg.msgType = pMsg->msgType;
if (code) {
qError("vnd get virtual subtables failed cause of %s", tstrerror(code));
}
tDestroySVSubTablesRsp(&rsp);
tmsgSendRsp(&rspMsg);
return code;
}
int32_t vnodeGetLoad(SVnode *pVnode, SVnodeLoad *pLoad) {
SSyncState state = syncGetState(pVnode->sync);
pLoad->syncAppliedIndex = pVnode->state.applied;

7
source/dnode/vnode/src/vnd/vnodeSvr.c
View File

@ -898,7 +898,7 @@ int32_t vnodeProcessQueryMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
vTrace("vgId:%d, msg:%p in fetch queue is processing", pVnode->config.vgId, pMsg);
if ((pMsg->msgType == TDMT_SCH_FETCH || pMsg->msgType == TDMT_VND_TABLE_META || pMsg->msgType == TDMT_VND_TABLE_CFG ||
pMsg->msgType == TDMT_VND_BATCH_META || pMsg->msgType == TDMT_VND_TABLE_NAME) &&
pMsg->msgType == TDMT_VND_BATCH_META || pMsg->msgType == TDMT_VND_TABLE_NAME || pMsg->msgType == TDMT_VND_VSUBTABLES_META) &&
!syncIsReadyForRead(pVnode->sync)) {
vnodeRedirectRpcMsg(pVnode, pMsg, terrno);
return 0;
@ -925,6 +925,8 @@ int32_t vnodeProcessFetchMsg(SVnode *pVnode, SRpcMsg *pMsg, SQueueInfo *pInfo) {
return vnodeGetTableCfg(pVnode, pMsg, true);
case TDMT_VND_BATCH_META:
return vnodeGetBatchMeta(pVnode, pMsg);
case TDMT_VND_VSUBTABLES_META:
return vnodeGetVSubtablesMeta(pVnode, pMsg);
#ifdef TD_ENTERPRISE
case TDMT_VND_QUERY_COMPACT_PROGRESS:
return vnodeQueryCompactProgress(pVnode, pMsg);
@ -1563,6 +1565,9 @@ _exit:
taosMemoryFree(vMetaRsp.pSchemas);
taosMemoryFree(vMetaRsp.pSchemaExt);
}
if (vMetaRsp.pColRefs) {
taosMemoryFree(vMetaRsp.pColRefs);
}
return 0;
}

31
source/libs/catalog/inc/catalogInt.h
View File

@ -72,6 +72,7 @@ typedef enum {
CTG_CI_SVR_VER,
CTG_CI_VIEW,
CTG_CI_TBL_TSMA,
CTG_CI_VSUB_TBLS,
CTG_CI_MAX_VALUE,
} CTG_CACHE_ITEM;
@ -133,6 +134,7 @@ typedef enum {
CTG_TASK_GET_TB_TSMA,
CTG_TASK_GET_TSMA,
CTG_TASK_GET_TB_NAME,
CTG_TASK_GET_V_SUBTABLES,
} CTG_TASK_TYPE;
typedef enum {
@ -307,6 +309,20 @@ typedef struct SCtgTbTSMACtx {
SArray* pFetches;
} SCtgTbTSMACtx;
typedef struct SCtgVSubTablesCtx {
SArray* pNames;
STableMeta* pMeta;
int32_t vgNum;
bool clonedVgroups;
SArray* pVgroups;
SVSubTablesRsp* pResList;
int32_t resIdx;
} SCtgVSubTablesCtx;
typedef STableIndexRsp STableIndex;
typedef STableTSMAInfo STSMACache;
@ -433,6 +449,7 @@ typedef struct SCtgJob {
int32_t tbTsmaNum;
int32_t tsmaNum; // currently, only 1 is possible
int32_t tbNameNum;
int32_t vsubTbNum;
} SCtgJob;
typedef struct SCtgMsgCtx {
@ -777,10 +794,12 @@ typedef struct SCtgCacheItemInfo {
#define CTG_META_NHIT_INC() CTG_CACHE_NHIT_INC(CTG_CI_OTHERTABLE_META, 1)
#define CTG_IS_META_NULL(type) ((type) == META_TYPE_NULL_TABLE)
#define CTG_IS_META_CTABLE(type) ((type) == META_TYPE_CTABLE)
#define CTG_IS_META_TABLE(type) ((type) == META_TYPE_TABLE)
#define CTG_IS_META_BOTH(type) ((type) == META_TYPE_BOTH_TABLE)
#define CTG_IS_META_NULL(type) ((type) == META_TYPE_NULL_TABLE)
#define CTG_IS_META_CTABLE(type) ((type) == META_TYPE_CTABLE)
#define CTG_IS_META_VCTABLE(type) ((type) == META_TYPE_VCTABLE)
#define CTG_IS_META_TABLE(type) ((type) == META_TYPE_TABLE)
#define CTG_IS_META_BOTH(type) ((type) == META_TYPE_BOTH_TABLE)
#define CTG_IS_META_VBOTH(type) ((type) == META_TYPE_BOTH_VTABLE)
#define CTG_FLAG_STB 0x1
#define CTG_FLAG_NOT_STB 0x2
@ -1074,6 +1093,7 @@ int32_t ctgOpUpdateTbIndex(SCtgCacheOperation* operation);
int32_t ctgOpClearCache(SCtgCacheOperation* operation);
int32_t ctgOpUpdateViewMeta(SCtgCacheOperation* operation);
int32_t ctgReadTbTypeFromCache(SCatalog* pCtg, char* dbFName, char* tableName, int32_t* tbType);
int32_t ctgReadTbTypeSuidFromCache(SCatalog *pCtg, char *dbFName, char *tbName, int32_t *tbType, int64_t* suid);
int32_t ctgGetTbHashVgroupFromCache(SCatalog* pCtg, const SName* pTableName, SVgroupInfo** pVgroup);
int32_t ctgGetViewsFromCache(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgViewsCtx* ctx, int32_t dbIdx,
int32_t* fetchIdx, int32_t baseResIdx, SArray* pList);
@ -1112,6 +1132,7 @@ int32_t ctgLaunchJob(SCtgJob* pJob);
int32_t ctgMakeAsyncRes(SCtgJob* pJob);
int32_t ctgLaunchSubTask(SCtgTask** ppTask, CTG_TASK_TYPE type, ctgSubTaskCbFp fp, void* param);
int32_t ctgGetTbCfgCb(SCtgTask* pTask);
int32_t ctgGetVSubTablesCb(SCtgTask* pTask);
void ctgFreeHandle(SCatalog* pCatalog);
void ctgFreeSViewMeta(SViewMeta* pMeta);
@ -1210,8 +1231,10 @@ bool isCtgTSMACacheOutOfDate(STSMACache* pTsmaCache);
int32_t ctgGetStreamProgressFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, const SName* pTbName,
SVgroupInfo* vgroupInfo, SStreamProgressRsp* out, SCtgTaskReq* tReq,
void* bInput);
int32_t ctgGetVSubTablesFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, int64_t suid, SVgroupInfo* vgroupInfo, SCtgTaskReq* tReq);
int32_t ctgAddTSMAFetch(SArray** pFetchs, int32_t dbIdx, int32_t tbIdx, int32_t* fetchIdx, int32_t resIdx, int32_t flag,
CTG_TSMA_FETCH_TYPE fetchType, const SName* sourceTbName);
int32_t ctgBuildNormalChildVtbList(SCtgVSubTablesCtx* pCtx);
int32_t ctgOpUpdateDbTsmaVersion(SCtgCacheOperation* pOper);
int32_t ctgUpdateDbTsmaVersionEnqueue(SCatalog* pCtg, int32_t tsmaVersion, const char* dbFName, int64_t dbId,
bool syncOper);

39
source/libs/catalog/src/catalog.c
View File

@ -138,7 +138,7 @@ int32_t ctgRefreshTbMeta(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgTbMetaCtx*
taosMemoryFreeClear(output->tbMeta);
CTG_ERR_JRET(ctgGetTbMetaFromMnodeImpl(pCtg, pConn, output->dbFName, output->tbName, output, NULL));
} else if (CTG_IS_META_BOTH(output->metaType)) {
} else if (CTG_IS_META_BOTH(output->metaType) || CTG_IS_META_VBOTH(output->metaType)) {
int32_t exist = 0;
if (!CTG_FLAG_IS_FORCE_UPDATE(ctx->flag)) {
CTG_ERR_JRET(ctgTbMetaExistInCache(pCtg, output->dbFName, output->tbName, &exist));
@ -157,7 +157,11 @@ int32_t ctgRefreshTbMeta(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgTbMetaCtx*
} else {
taosMemoryFreeClear(output->tbMeta);
SET_META_TYPE_CTABLE(output->metaType);
if (CTG_IS_META_BOTH(output->metaType)) {
SET_META_TYPE_CTABLE(output->metaType);
} else {
SET_META_TYPE_VCTABLE(output->metaType);
}
}
}
}
@ -189,6 +193,7 @@ _return:
if (output) {
taosMemoryFreeClear(output->tbMeta);
taosMemoryFreeClear(output->vctbMeta);
taosMemoryFreeClear(output);
}
@ -219,8 +224,30 @@ int32_t ctgGetTbMeta(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgTbMetaCtx* ctx
goto _return;
}
if (CTG_IS_META_VBOTH(output->metaType) || CTG_IS_META_VCTABLE(output->metaType)) {
int32_t colRefSize = output->vctbMeta->numOfColRefs * sizeof(SColRef);
if (output->tbMeta) {
int32_t metaSize = CTG_META_SIZE(output->tbMeta);
int32_t schemaExtSize = 0;
if (withExtSchema(output->tbMeta->tableType) && output->tbMeta->schemaExt) {
schemaExtSize = output->tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
TAOS_MEMCPY(output->tbMeta, output->vctbMeta, sizeof(SVCTableMeta));
output->tbMeta->colRef = (SColRef *)((char *)output->tbMeta + metaSize + schemaExtSize);
TAOS_MEMCPY(output->tbMeta->colRef, output->vctbMeta->colRef, colRefSize);
} else {
TAOS_MEMCPY(output->tbMeta, output->vctbMeta, sizeof(SVCTableMeta) + colRefSize);
output->tbMeta->colRef = (SColRef *)((char *)output->tbMeta + sizeof(SVCTableMeta));
}
output->tbMeta->numOfColRefs = output->vctbMeta->numOfColRefs;
taosMemoryFreeClear(output->vctbMeta);
*pTableMeta = output->tbMeta;
goto _return;
}
if ((!CTG_IS_META_CTABLE(output->metaType)) || output->tbMeta) {
ctgError("invalid metaType:%d", output->metaType);
taosMemoryFreeClear(output->vctbMeta);
taosMemoryFreeClear(output->tbMeta);
CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
@ -228,6 +255,7 @@ int32_t ctgGetTbMeta(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgTbMetaCtx* ctx
// HANDLE ONLY CHILD TABLE META
taosMemoryFreeClear(output->tbMeta);
taosMemoryFreeClear(output->vctbMeta);
SName stbName = *ctx->pName;
TAOS_STRCPY(stbName.tname, output->tbName);
@ -292,6 +320,12 @@ int32_t ctgUpdateTbMeta(SCatalog* pCtg, STableMetaRsp* rspMsg, bool syncOp) {
SET_META_TYPE_CTABLE(output->metaType);
CTG_ERR_JRET(queryCreateCTableMetaFromMsg(rspMsg, &output->ctbMeta));
} else if (TSDB_VIRTUAL_CHILD_TABLE == rspMsg->tableType && NULL == rspMsg->pSchemas) {
TAOS_STRCPY(output->ctbName, rspMsg->tbName);
SET_META_TYPE_VCTABLE(output->metaType);
CTG_ERR_JRET(queryCreateVCTableMetaFromMsg(rspMsg, &output->vctbMeta));
} else {
TAOS_STRCPY(output->tbName, rspMsg->tbName);
@ -310,6 +344,7 @@ _return:
if (output) {
taosMemoryFreeClear(output->tbMeta);
taosMemoryFreeClear(output->vctbMeta);
taosMemoryFreeClear(output);
}

308
source/libs/catalog/src/ctgAsync.c
View File

@ -623,6 +623,29 @@ static int32_t ctgInitGetTbNamesTask(SCtgJob* pJob, int32_t taskId, void* param)
return TSDB_CODE_SUCCESS;
}
int32_t ctgInitGetVSubTablesTask(SCtgJob* pJob, int32_t taskId, void* param) {
SCtgTask task = {0};
task.type = CTG_TASK_GET_V_SUBTABLES;
task.taskId = taskId;
task.pJob = pJob;
SCtgVSubTablesCtx* pTaskCtx = taosMemoryCalloc(1, sizeof(SCtgVSubTablesCtx));
if (NULL == pTaskCtx) {
CTG_ERR_RET(terrno);
}
task.taskCtx = pTaskCtx;
pTaskCtx->pNames = param;
if (NULL == taosArrayPush(pJob->pTasks, &task)) {
ctgFreeTask(&task, true);
CTG_ERR_RET(terrno);
}
return TSDB_CODE_SUCCESS;
}
int32_t ctgHandleForceUpdateView(SCatalog* pCtg, const SCatalogReq* pReq) {
int32_t viewNum = taosArrayGetSize(pReq->pView);
for (int32_t i = 0; i < viewNum; ++i) {
@ -858,9 +881,10 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const
int32_t tbTsmaNum = tsQuerySmaOptimize ? (int32_t)taosArrayGetSize(pReq->pTableTSMAs) : 0;
int32_t tsmaNum = (int32_t)taosArrayGetSize(pReq->pTSMAs);
int32_t tbNameNum = (int32_t)ctgGetTablesReqNum(pReq->pTableName);
int32_t vsubTbNum = (int32_t)taosArrayGetSize(pReq->pVSubTable);
int32_t taskNum = tbMetaNum + dbVgNum + udfNum + tbHashNum + qnodeNum + dnodeNum + svrVerNum + dbCfgNum + indexNum +
userNum + dbInfoNum + tbIndexNum + tbCfgNum + tbTagNum + viewNum + tbTsmaNum + tbNameNum;
userNum + dbInfoNum + tbIndexNum + tbCfgNum + tbTagNum + viewNum + tbTsmaNum + tbNameNum + vsubTbNum;
int32_t taskNumWithSubTasks = tbMetaNum * gCtgAsyncFps[CTG_TASK_GET_TB_META].subTaskFactor + dbVgNum * gCtgAsyncFps[CTG_TASK_GET_DB_VGROUP].subTaskFactor +
udfNum * gCtgAsyncFps[CTG_TASK_GET_UDF].subTaskFactor + tbHashNum * gCtgAsyncFps[CTG_TASK_GET_TB_HASH].subTaskFactor +
qnodeNum * gCtgAsyncFps[CTG_TASK_GET_QNODE].subTaskFactor + dnodeNum * gCtgAsyncFps[CTG_TASK_GET_DNODE].subTaskFactor +
@ -869,7 +893,8 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const
dbInfoNum * gCtgAsyncFps[CTG_TASK_GET_DB_INFO].subTaskFactor + tbIndexNum * gCtgAsyncFps[CTG_TASK_GET_TB_SMA_INDEX].subTaskFactor +
tbCfgNum * gCtgAsyncFps[CTG_TASK_GET_TB_CFG].subTaskFactor + tbTagNum * gCtgAsyncFps[CTG_TASK_GET_TB_TAG].subTaskFactor +
viewNum * gCtgAsyncFps[CTG_TASK_GET_VIEW].subTaskFactor + tbTsmaNum * gCtgAsyncFps[CTG_TASK_GET_TB_TSMA].subTaskFactor +
tsmaNum * gCtgAsyncFps[CTG_TASK_GET_TSMA].subTaskFactor + tbNameNum * gCtgAsyncFps[CTG_TASK_GET_TB_NAME].subTaskFactor;
tsmaNum * gCtgAsyncFps[CTG_TASK_GET_TSMA].subTaskFactor + tbNameNum * gCtgAsyncFps[CTG_TASK_GET_TB_NAME].subTaskFactor +
vsubTbNum * gCtgAsyncFps[CTG_TASK_GET_V_SUBTABLES].subTaskFactor;
*job = taosMemoryCalloc(1, sizeof(SCtgJob));
if (NULL == *job) {
@ -905,6 +930,7 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const
pJob->tbTsmaNum = tbTsmaNum;
pJob->tsmaNum = tsmaNum;
pJob->tbNameNum = tbNameNum;
pJob->vsubTbNum = vsubTbNum;
#if CTG_BATCH_FETCH
pJob->pBatchs =
@ -1050,6 +1076,9 @@ int32_t ctgInitJob(SCatalog* pCtg, SRequestConnInfo* pConn, SCtgJob** job, const
if (svrVerNum) {
CTG_ERR_JRET(ctgInitTask(pJob, CTG_TASK_GET_SVR_VER, NULL, NULL));
}
if (vsubTbNum) {
CTG_ERR_JRET(ctgInitTask(pJob, CTG_TASK_GET_V_SUBTABLES, pReq->pVSubTable, NULL));
}
pJob->refId = taosAddRef(gCtgMgmt.jobPool, pJob);
if (pJob->refId < 0) {
@ -1440,6 +1469,41 @@ int32_t ctgDumpViewsRes(SCtgTask* pTask) {
return TSDB_CODE_SUCCESS;
}
int32_t ctgDumpVSubTablesRes(SCtgTask* pTask) {
if (pTask->subTask) {
return TSDB_CODE_SUCCESS;
}
SCtgVSubTablesCtx* pCtx = (SCtgVSubTablesCtx*)pTask->taskCtx;
SCatalog* pCtg = pTask->pJob->pCtg;
if (NULL == pCtx->pResList) {
return TSDB_CODE_SUCCESS;
}
SCtgJob* pJob = pTask->pJob;
int32_t resVgNum = (TSDB_SUPER_TABLE == pCtx->pMeta->tableType) ? pCtx->vgNum : 1;
pJob->jobRes.pVSubTables = taosArrayInit(resVgNum, sizeof(SVSubTablesRsp));
if (NULL == pJob->jobRes.pVSubTables) {
ctgError("taosArrayInit %d SVSubTablesRsp failed, error:%d", resVgNum, terrno);
return terrno;
}
SVSubTablesRsp* pVg = NULL;
for (int32_t i = 0; i < resVgNum; ++i) {
pVg = (SVSubTablesRsp*)pCtx->pResList + i;
if (NULL == taosArrayPush(pJob->jobRes.pVSubTables, pVg)) {
ctgError("taosArrayPush failed, error:%d", terrno);
return terrno;
}
pVg->pTables = NULL;
}
return TSDB_CODE_SUCCESS;
}
int32_t ctgCallSubCb(SCtgTask* pTask) {
int32_t code = 0;
@ -1638,7 +1702,7 @@ int32_t ctgHandleGetTbMetaRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf
taosMemoryFreeClear(pOut->tbMeta);
CTG_RET(ctgGetTbMetaFromMnode(pCtg, pConn, pName, NULL, tReq));
} else if (CTG_IS_META_BOTH(pOut->metaType)) {
} else if (CTG_IS_META_BOTH(pOut->metaType) || CTG_IS_META_VBOTH(pOut->metaType)) {
int32_t exist = 0;
if (!CTG_FLAG_IS_FORCE_UPDATE(flag)) {
SName stbName = *pName;
@ -1674,6 +1738,28 @@ int32_t ctgHandleGetTbMetaRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf
TAOS_MEMCPY(pOut->tbMeta, &pOut->ctbMeta, sizeof(pOut->ctbMeta));
}
if (CTG_IS_META_VBOTH(pOut->metaType)) {
int32_t colRefSize = pOut->vctbMeta->numOfColRefs * sizeof(SColRef);
if (pOut->tbMeta) {
int32_t metaSize = CTG_META_SIZE(pOut->tbMeta);
int32_t schemaExtSize = 0;
if (withExtSchema(pOut->tbMeta->tableType) && pOut->tbMeta->schemaExt) {
schemaExtSize = pOut->tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
pOut->tbMeta = taosMemoryRealloc(pOut->tbMeta, metaSize + schemaExtSize + colRefSize);
TAOS_MEMCPY(pOut->tbMeta, pOut->vctbMeta, sizeof(SVCTableMeta));
pOut->tbMeta->colRef = (SColRef *)((char *)pOut->tbMeta + metaSize + schemaExtSize);
TAOS_MEMCPY(pOut->tbMeta->colRef, pOut->vctbMeta->colRef, colRefSize);
} else {
pOut->tbMeta = taosMemoryRealloc(pOut->tbMeta, sizeof(STableMeta) + colRefSize);
TAOS_MEMCPY(pOut->tbMeta, pOut->vctbMeta, sizeof(SVCTableMeta));
TAOS_MEMCPY(pOut->tbMeta + sizeof(STableMeta), pOut->vctbMeta + sizeof(SVCTableMeta), colRefSize);
pOut->tbMeta->colRef = (SColRef *)((char *)pOut->tbMeta + sizeof(STableMeta));
}
pOut->tbMeta->numOfColRefs = pOut->vctbMeta->numOfColRefs;
taosMemoryFreeClear(pOut->vctbMeta);
}
/*
else if (CTG_IS_META_CTABLE(pOut->metaType)) {
SName stbName = *pName;
@ -1823,7 +1909,7 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu
taosMemoryFreeClear(pOut->tbMeta);
CTG_RET(ctgGetTbMetaFromMnode(pCtg, pConn, pName, NULL, tReq));
} else if (CTG_IS_META_BOTH(pOut->metaType)) {
} else if (CTG_IS_META_BOTH(pOut->metaType) || CTG_IS_META_VBOTH(pOut->metaType)) {
int32_t exist = 0;
if (!CTG_FLAG_IS_FORCE_UPDATE(flag)) {
SName stbName = *pName;
@ -1865,6 +1951,27 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu
TAOS_MEMCPY(pOut->tbMeta, &pOut->ctbMeta, sizeof(pOut->ctbMeta));
}
if (CTG_IS_META_VBOTH(pOut->metaType)) {
int32_t colRefSize = pOut->vctbMeta->numOfColRefs * sizeof(SColRef);
if (pOut->tbMeta) {
int32_t metaSize = CTG_META_SIZE(pOut->tbMeta);
int32_t schemaExtSize = 0;
if (withExtSchema(pOut->tbMeta->tableType) && pOut->tbMeta->schemaExt) {
schemaExtSize = pOut->tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
pOut->tbMeta = taosMemoryRealloc(pOut->tbMeta, metaSize + schemaExtSize + colRefSize);
TAOS_MEMCPY(pOut->tbMeta, pOut->vctbMeta, sizeof(SVCTableMeta));
pOut->tbMeta->colRef = (SColRef *)((char *)pOut->tbMeta + metaSize + schemaExtSize);
TAOS_MEMCPY(pOut->tbMeta->colRef, pOut->vctbMeta->colRef, colRefSize);
} else {
pOut->tbMeta = taosMemoryRealloc(pOut->tbMeta, sizeof(STableMeta) + colRefSize);
TAOS_MEMCPY(pOut->tbMeta, pOut->vctbMeta, sizeof(SVCTableMeta));
TAOS_MEMCPY(pOut->tbMeta + sizeof(STableMeta), pOut->vctbMeta + sizeof(SVCTableMeta), colRefSize);
pOut->tbMeta->colRef = (SColRef *)((char *)pOut->tbMeta + sizeof(STableMeta));
}
pOut->tbMeta->numOfColRefs = pOut->vctbMeta->numOfColRefs;
taosMemoryFreeClear(pOut->vctbMeta);
}
/*
else if (CTG_IS_META_CTABLE(pOut->metaType)) {
SName stbName = *pName;
@ -1895,6 +2002,7 @@ int32_t ctgHandleGetTbMetasRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBu
pRes->code = 0;
pRes->pRes = pOut->tbMeta;
pOut->tbMeta = NULL;
pOut->vctbMeta = NULL;
if (0 == atomic_sub_fetch_32(&ctx->fetchNum, 1)) {
TSWAP(pTask->res, ctx->pResList);
taskDone = true;
@ -2066,6 +2174,28 @@ static int32_t ctgHandleGetTbNamesRsp(SCtgTaskReq* tReq, int32_t reqType, const
TAOS_MEMCPY(pOut->tbMeta, &pOut->ctbMeta, sizeof(pOut->ctbMeta));
}
if (CTG_IS_META_VBOTH(pOut->metaType)) {
int32_t colRefSize = pOut->vctbMeta->numOfColRefs * sizeof(SColRef);
if (pOut->tbMeta) {
int32_t metaSize = CTG_META_SIZE(pOut->tbMeta);
int32_t schemaExtSize = 0;
if (withExtSchema(pOut->tbMeta->tableType) && pOut->tbMeta->schemaExt) {
schemaExtSize = pOut->tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
pOut->tbMeta = taosMemoryRealloc(pOut->tbMeta, metaSize + schemaExtSize + colRefSize);
TAOS_MEMCPY(pOut->tbMeta, pOut->vctbMeta, sizeof(SVCTableMeta));
pOut->tbMeta->colRef = (SColRef *)((char *)pOut->tbMeta + metaSize + schemaExtSize);
TAOS_MEMCPY(pOut->tbMeta->colRef, pOut->vctbMeta->colRef, colRefSize);
} else {
pOut->tbMeta = taosMemoryRealloc(pOut->tbMeta, sizeof(STableMeta) + colRefSize);
TAOS_MEMCPY(pOut->tbMeta, pOut->vctbMeta, sizeof(SVCTableMeta));
TAOS_MEMCPY(pOut->tbMeta + sizeof(STableMeta), pOut->vctbMeta + sizeof(SVCTableMeta), colRefSize);
pOut->tbMeta->colRef = (SColRef *)((char *)pOut->tbMeta + sizeof(STableMeta));
}
pOut->tbMeta->numOfColRefs = pOut->vctbMeta->numOfColRefs;
taosMemoryFreeClear(pOut->vctbMeta);
}
SMetaRes* pRes = taosArrayGet(ctx->pResList, pFetch->resIdx);
if (NULL == pRes) {
ctgTaskError("fail to get the %dth res in pResList, resNum:%d", pFetch->resIdx,
@ -2076,8 +2206,10 @@ static int32_t ctgHandleGetTbNamesRsp(SCtgTaskReq* tReq, int32_t reqType, const
if (!pRes->pRes) {
pRes->code = 0;
pRes->pRes = pOut->tbMeta;
taosMemoryFreeClear(pOut->vctbMeta);
pOut->tbMeta = NULL;
} else {
taosMemoryFreeClear(pOut->vctbMeta);
taosMemoryFreeClear(pOut->tbMeta);
}
@ -3008,6 +3140,30 @@ _return:
CTG_RET(code);
}
int32_t ctgHandleGetVSubTablesRsp(SCtgTaskReq* tReq, int32_t reqType, const SDataBuf* pMsg, int32_t rspCode) {
int32_t code = 0;
SCtgTask* pTask = tReq->pTask;
int32_t newCode = TSDB_CODE_SUCCESS;
SCtgVSubTablesCtx* pCtx = (SCtgVSubTablesCtx*)pTask->taskCtx;
CTG_ERR_JRET(ctgProcessRspMsg(pCtx->pResList + atomic_fetch_add_32(&pCtx->resIdx, 1), reqType, pMsg->pData, pMsg->len, rspCode, pTask->msgCtx.target));
if (atomic_load_32(&pCtx->resIdx) < pCtx->vgNum) {
CTG_RET(code);
}
_return:
newCode = ctgHandleTaskEnd(pTask, code);
if (newCode && TSDB_CODE_SUCCESS == code) {
code = newCode;
}
CTG_RET(code);
}
int32_t ctgAsyncRefreshTbMeta(SCtgTaskReq* tReq, int32_t flag, SName* pName, int32_t* vgId) {
SCtgTask* pTask = tReq->pTask;
SCatalog* pCtg = pTask->pJob->pCtg;
@ -4065,6 +4221,122 @@ static int32_t ctgLaunchGetTbNamesTask(SCtgTask* pTask) {
return TSDB_CODE_SUCCESS;
}
int32_t ctgLaunchGetVSubTablesTask(SCtgTask* pTask) {
SCatalog* pCtg = pTask->pJob->pCtg;
SRequestConnInfo* pConn = &pTask->pJob->conn;
SCtgVSubTablesCtx* pCtx = (SCtgVSubTablesCtx*)pTask->taskCtx;
SCtgJob* pJob = pTask->pJob;
SName* pName = NULL;
char dbFName[TSDB_DB_FNAME_LEN];
SCtgDBCache* dbCache = NULL;
int32_t code = TSDB_CODE_SUCCESS;
SCtgMsgCtx* pMsgCtx = CTG_GET_TASK_MSGCTX(pTask, -1);
if (NULL == pMsgCtx) {
ctgError("fail to get the %dth pMsgCtx", -1);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
if (NULL == pMsgCtx->pBatchs) {
pMsgCtx->pBatchs = pJob->pBatchs;
}
int32_t tbNum = taosArrayGetSize(pCtx->pNames);
if (tbNum > 1) {
ctgError("only 1 virtual table supported now, tbnum:%d", tbNum);
CTG_ERR_RET(TSDB_CODE_CTG_INVALID_INPUT);
}
pName = taosArrayGet(pCtx->pNames, 0);
if (NULL == pName) {
ctgError("fail to get SName in VSubtables req, num:%d", tbNum);
CTG_ERR_RET(TSDB_CODE_CTG_INVALID_INPUT);
}
if (NULL == pMsgCtx->target) {
pMsgCtx->target = taosMemoryMalloc(TSDB_TABLE_FNAME_LEN);
if (NULL == pMsgCtx->target) {
ctgError("taosMemoryMalloc %d failed", TSDB_TABLE_FNAME_LEN);
CTG_ERR_RET(terrno);
}
tNameExtractFullName(pName, pMsgCtx->target);
}
(void)tNameGetFullDbName(pName, dbFName);
if (NULL == pCtx->pMeta) {
SCtgTbMetaCtx metaCtx = {0};
metaCtx.pName = pName;
CTG_ERR_RET(ctgReadTbMetaFromCache(pCtg, &metaCtx, &pCtx->pMeta));
if (NULL == pCtx->pMeta) {
SCtgTbMetaParam param;
param.pName = pName;
param.flag = 0;
CTG_ERR_RET(ctgLaunchSubTask(&pTask, CTG_TASK_GET_TB_META, ctgGetVSubTablesCb, &param));
return TSDB_CODE_SUCCESS;
}
}
if (TSDB_VIRTUAL_CHILD_TABLE == pCtx->pMeta->tableType || TSDB_VIRTUAL_NORMAL_TABLE == pCtx->pMeta->tableType) {
CTG_ERR_RET(ctgBuildNormalChildVtbList(pCtx));
CTG_ERR_RET(ctgHandleTaskEnd(pTask, 0));
return TSDB_CODE_SUCCESS;
}
if (TSDB_SUPER_TABLE != pCtx->pMeta->tableType || !pCtx->pMeta->virtualStb) {
CTG_ERR_RET(ctgHandleTaskEnd(pTask, 0));
return TSDB_CODE_SUCCESS;
}
if (NULL == pCtx->pVgroups) {
CTG_ERR_RET(ctgAcquireVgInfoFromCache(pCtg, dbFName, &dbCache));
if (NULL == dbCache) {
CTG_ERR_RET(ctgLaunchSubTask(&pTask, CTG_TASK_GET_DB_VGROUP, ctgGetVSubTablesCb, dbFName));
return TSDB_CODE_SUCCESS;
}
pCtx->pVgroups = dbCache->vgCache.vgInfo->vgArray;
}
pCtx->vgNum = taosArrayGetSize(pCtx->pVgroups);
pCtx->pResList = taosMemoryCalloc(pCtx->vgNum, sizeof(SVSubTablesRsp));
if (NULL == pCtx->pResList) {
ctgError("taosMemoryMalloc %d SVSubTablesRsp failed", pCtx->vgNum);
CTG_ERR_JRET(terrno);
}
for (int32_t i = 0; i < pCtx->vgNum; ++i) {
SVgroupInfo* pVg = (SVgroupInfo*)taosArrayGet(pCtx->pVgroups, i);
ctgDebug("will try to get vsubtable from vg %d", pVg->vgId);
SCtgTaskReq tReq;
tReq.pTask = pTask;
tReq.msgIdx = -1;
CTG_ERR_JRET(ctgGetVSubTablesFromVnode(pCtg, pConn, pCtx->pMeta->suid, pVg, &tReq));
}
if (dbCache) {
ctgReleaseVgInfoToCache(pCtg, dbCache);
}
return TSDB_CODE_SUCCESS;
_return:
if (dbCache) {
ctgReleaseVgInfoToCache(pCtg, dbCache);
}
if (CTG_TASK_LAUNCHED == pTask->status) {
int32_t newCode = ctgHandleTaskEnd(pTask, code);
if (newCode && TSDB_CODE_SUCCESS == code) {
code = newCode;
}
}
CTG_RET(code);
}
int32_t ctgDumpTbTSMARes(SCtgTask* pTask) {
if (pTask->subTask) {
return TSDB_CODE_SUCCESS;
@ -4120,6 +4392,33 @@ _return:
CTG_RET(ctgHandleTaskEnd(pTask, pTask->subRes.code));
}
int32_t ctgGetVSubTablesCb(SCtgTask* pTask) {
int32_t code = 0;
CTG_ERR_JRET(pTask->subRes.code);
SCtgVSubTablesCtx* pCtx = (SCtgVSubTablesCtx*)pTask->taskCtx;
if (CTG_TASK_GET_TB_META == pTask->subRes.type) {
TSWAP(pCtx->pMeta, pTask->subRes.res);
} else if (CTG_TASK_GET_DB_VGROUP == pTask->subRes.type) {
SDBVgInfo* pDb = (SDBVgInfo*)pTask->subRes.res;
pCtx->pVgroups = taosArrayDup(pDb->vgArray, NULL);
if (NULL == pCtx->pVgroups) {
CTG_ERR_JRET(terrno);
}
pCtx->clonedVgroups = true;
}
CTG_RET(ctgLaunchGetVSubTablesTask(pTask));
_return:
CTG_RET(ctgHandleTaskEnd(pTask, pTask->subRes.code));
}
int32_t ctgGetTbTagCb(SCtgTask* pTask) {
int32_t code = 0;
@ -4212,6 +4511,7 @@ SCtgAsyncFps gCtgAsyncFps[] = {
{ctgInitGetTbTSMATask, ctgLaunchGetTbTSMATask, ctgHandleGetTbTSMARsp, ctgDumpTbTSMARes, NULL, NULL, 1},
{ctgInitGetTSMATask, ctgLaunchGetTSMATask, ctgHandleGetTSMARsp, ctgDumpTSMARes, NULL, NULL, 1},
{ctgInitGetTbNamesTask, ctgLaunchGetTbNamesTask, ctgHandleGetTbNamesRsp, ctgDumpTbNamesRes, NULL, NULL, 1},
{ctgInitGetVSubTablesTask, ctgLaunchGetVSubTablesTask, ctgHandleGetVSubTablesRsp, ctgDumpVSubTablesRes, NULL, NULL, 2},
};
int32_t ctgMakeAsyncRes(SCtgJob* pJob) {

137
source/libs/catalog/src/ctgCache.c
View File

@ -538,34 +538,55 @@ int32_t ctgCopyTbMeta(SCatalog *pCtg, SCtgTbMetaCtx *ctx, SCtgDBCache **pDb, SCt
ctx->tbInfo.suid = tbMeta->suid;
ctx->tbInfo.tbType = tbMeta->tableType;
if (tbMeta->tableType != TSDB_CHILD_TABLE) {
if (tbMeta->tableType != TSDB_CHILD_TABLE && tbMeta->tableType != TSDB_VIRTUAL_CHILD_TABLE) {
int32_t schemaExtSize = 0;
int32_t colRefSize = 0;
int32_t metaSize = CTG_META_SIZE(tbMeta);
if (tbMeta->schemaExt != NULL) {
if (withExtSchema(tbMeta->tableType) && tbMeta->schemaExt != NULL) {
schemaExtSize = tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
*pTableMeta = taosMemoryCalloc(1, metaSize + schemaExtSize);
if (hasRefCol(tbMeta->tableType) && tbMeta->colRef != NULL) {
colRefSize += tbMeta->tableInfo.numOfColumns * sizeof(SColRef);
}
*pTableMeta = taosMemoryCalloc(1, metaSize + schemaExtSize + colRefSize);
if (NULL == *pTableMeta) {
CTG_ERR_RET(terrno);
}
TAOS_MEMCPY(*pTableMeta, tbMeta, metaSize);
if (tbMeta->schemaExt != NULL) {
if (withExtSchema(tbMeta->tableType) && tbMeta->schemaExt != NULL) {
(*pTableMeta)->schemaExt = (SSchemaExt *)((char *)*pTableMeta + metaSize);
TAOS_MEMCPY((*pTableMeta)->schemaExt, tbMeta->schemaExt, schemaExtSize);
} else {
(*pTableMeta)->schemaExt = NULL;
}
if (hasRefCol(tbMeta->tableType) && tbMeta->colRef) {
(*pTableMeta)->colRef = (SColRef *)((char *)*pTableMeta + metaSize + schemaExtSize);
TAOS_MEMCPY((*pTableMeta)->colRef, tbMeta->colRef, colRefSize);
(*pTableMeta)->numOfColRefs = tbMeta->numOfColRefs;
} else {
(*pTableMeta)->colRef = NULL;
}
ctgDebug("tb:%s, get meta from cache, type:%d, db:%s", ctx->pName->tname, tbMeta->tableType, dbFName);
return TSDB_CODE_SUCCESS;
}
// PROCESS FOR CHILD TABLE
int32_t metaSize = sizeof(SCTableMeta);
*pTableMeta = taosMemoryCalloc(1, metaSize);
int32_t colRefSize = 0;
int32_t numOfColRefs = 0;
SColRef *tmpRef = NULL;
if (hasRefCol(tbMeta->tableType) && tbMeta->colRef != NULL) {
colRefSize += tbMeta->numOfColRefs * sizeof(SColRef);
numOfColRefs = tbMeta->numOfColRefs;
tmpRef = taosMemoryMalloc(colRefSize);
TAOS_MEMCPY(tmpRef, tbMeta->colRef, colRefSize);
}
*pTableMeta = taosMemoryCalloc(1, metaSize + colRefSize);
if (NULL == *pTableMeta) {
taosMemoryFreeClear(tmpRef);
CTG_ERR_RET(terrno);
}
@ -582,6 +603,7 @@ int32_t ctgCopyTbMeta(SCatalog *pCtg, SCtgTbMetaCtx *ctx, SCtgDBCache **pDb, SCt
CTG_ERR_RET(ctgAcquireStbMetaFromCache(dbCache, pCtg, dbFName, ctx->tbInfo.suid, &tbCache));
if (NULL == tbCache) {
taosMemoryFreeClear(tmpRef);
taosMemoryFreeClear(*pTableMeta);
*pDb = NULL;
ctgDebug("stb:0x%" PRIx64 ", meta not in cache", ctx->tbInfo.suid);
@ -594,6 +616,7 @@ int32_t ctgCopyTbMeta(SCatalog *pCtg, SCtgTbMetaCtx *ctx, SCtgDBCache **pDb, SCt
if (stbMeta->suid != ctx->tbInfo.suid) {
ctgError("stb:0x%" PRIx64 ", suid in stbCache mis-match, expected suid:0x%" PRIx64, stbMeta->suid, ctx->tbInfo.suid);
taosMemoryFreeClear(*pTableMeta);
taosMemoryFreeClear(tmpRef);
CTG_ERR_RET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
@ -602,12 +625,13 @@ int32_t ctgCopyTbMeta(SCatalog *pCtg, SCtgTbMetaCtx *ctx, SCtgDBCache **pDb, SCt
if (stbMeta->schemaExt) {
schemaExtSize = stbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
*pTableMeta = taosMemoryRealloc(*pTableMeta, metaSize + schemaExtSize);
*pTableMeta = taosMemoryRealloc(*pTableMeta, metaSize + schemaExtSize + colRefSize);
if (NULL == *pTableMeta) {
taosMemoryFreeClear(tmpRef);
CTG_ERR_RET(terrno);
}
TAOS_MEMCPY(&(*pTableMeta)->sversion, &stbMeta->sversion, metaSize - sizeof(SCTableMeta));
TAOS_MEMCPY(&(*pTableMeta)->numOfColRefs, &stbMeta->numOfColRefs, metaSize - sizeof(SCTableMeta));
if (stbMeta->schemaExt) {
(*pTableMeta)->schemaExt = (SSchemaExt*)((char*)*pTableMeta + metaSize);
TAOS_MEMCPY((*pTableMeta)->schemaExt, stbMeta->schemaExt, schemaExtSize);
@ -615,6 +639,15 @@ int32_t ctgCopyTbMeta(SCatalog *pCtg, SCtgTbMetaCtx *ctx, SCtgDBCache **pDb, SCt
(*pTableMeta)->schemaExt = NULL;
}
if (colRefSize != 0) {
(*pTableMeta)->colRef = (SColRef *)((char *)*pTableMeta + metaSize + schemaExtSize);
(*pTableMeta)->numOfColRefs = numOfColRefs;
TAOS_MEMCPY((*pTableMeta)->colRef, tmpRef, colRefSize);
} else {
(*pTableMeta)->colRef = NULL;
}
taosMemoryFreeClear(tmpRef);
return TSDB_CODE_SUCCESS;
}
@ -743,6 +776,26 @@ int32_t ctgReadTbTypeFromCache(SCatalog *pCtg, char *dbFName, char *tbName, int3
return TSDB_CODE_SUCCESS;
}
int32_t ctgReadTbTypeSuidFromCache(SCatalog *pCtg, char *dbFName, char *tbName, int32_t *tbType, int64_t* suid) {
SCtgDBCache *dbCache = NULL;
SCtgTbCache *tbCache = NULL;
CTG_ERR_RET(ctgAcquireTbMetaFromCache(pCtg, dbFName, tbName, &dbCache, &tbCache));
if (NULL == tbCache) {
ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache);
return TSDB_CODE_SUCCESS;
}
*tbType = tbCache->pMeta->tableType;
*suid = tbCache->pMeta->suid;
ctgReleaseTbMetaToCache(pCtg, dbCache, tbCache);
ctgDebug("Got tb %s tbType %d suid %" PRIu64 " from cache, dbFName:%s", tbName, *tbType, *suid, dbFName);
return TSDB_CODE_SUCCESS;
}
int32_t ctgReadTbIndexFromCache(SCatalog *pCtg, SName *pTableName, SArray **pRes) {
int32_t code = 0;
SCtgDBCache *dbCache = NULL;
@ -1230,6 +1283,7 @@ _return:
if (output) {
taosMemoryFree(output->tbMeta);
taosMemoryFree(output->vctbMeta);
taosMemoryFree(output);
}
@ -2433,12 +2487,14 @@ int32_t ctgOpUpdateTbMeta(SCtgCacheOperation *operation) {
goto _return;
}
if ((!CTG_IS_META_CTABLE(pMeta->metaType)) && NULL == pMeta->tbMeta) {
if ((!CTG_IS_META_CTABLE(pMeta->metaType) && !CTG_IS_META_VCTABLE(pMeta->metaType)) && NULL == pMeta->tbMeta) {
ctgError("no valid tbmeta got from meta rsp, db:%s, tbName:%s", pMeta->dbFName, pMeta->tbName);
CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
if (CTG_IS_META_BOTH(pMeta->metaType) && TSDB_SUPER_TABLE != pMeta->tbMeta->tableType) {
if ((CTG_IS_META_BOTH(pMeta->metaType) ||
CTG_IS_META_VBOTH(pMeta->metaType)) &&
TSDB_SUPER_TABLE != pMeta->tbMeta->tableType) {
ctgError("table type error, expected:%d, actual:%d", TSDB_SUPER_TABLE, pMeta->tbMeta->tableType);
CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
@ -2449,7 +2505,7 @@ int32_t ctgOpUpdateTbMeta(SCtgCacheOperation *operation) {
CTG_ERR_JRET(TSDB_CODE_CTG_INTERNAL_ERROR);
}
if (CTG_IS_META_TABLE(pMeta->metaType) || CTG_IS_META_BOTH(pMeta->metaType)) {
if (CTG_IS_META_TABLE(pMeta->metaType) || CTG_IS_META_BOTH(pMeta->metaType) || CTG_IS_META_VBOTH(pMeta->metaType)) {
code = ctgWriteTbMetaToCache(pCtg, dbCache, pMeta->dbFName, pMeta->dbId, pMeta->tbName, pMeta->tbMeta);
pMeta->tbMeta = NULL;
CTG_ERR_JRET(code);
@ -2465,9 +2521,16 @@ int32_t ctgOpUpdateTbMeta(SCtgCacheOperation *operation) {
(STableMeta *)ctbMeta));
}
if (CTG_IS_META_VCTABLE(pMeta->metaType) || CTG_IS_META_VBOTH(pMeta->metaType)) {
code = ctgWriteTbMetaToCache(pCtg, dbCache, pMeta->dbFName, pMeta->dbId, pMeta->ctbName, (STableMeta *)pMeta->vctbMeta);
pMeta->vctbMeta = NULL;
CTG_ERR_JRET(code);
}
_return:
taosMemoryFreeClear(pMeta->tbMeta);
taosMemoryFreeClear(pMeta->vctbMeta);
taosMemoryFreeClear(pMeta);
taosMemoryFreeClear(msg);
@ -3162,6 +3225,7 @@ void ctgFreeCacheOperationData(SCtgCacheOperation *op) {
case CTG_OP_UPDATE_TB_META: {
SCtgUpdateTbMetaMsg *msg = op->data;
taosMemoryFreeClear(msg->pMeta->tbMeta);
taosMemoryFreeClear(msg->pMeta->vctbMeta);
taosMemoryFreeClear(msg->pMeta);
taosMemoryFreeClear(op->data);
break;
@ -3426,6 +3490,7 @@ int32_t ctgGetTbMetasFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgTbMe
int32_t code = TSDB_CODE_SUCCESS;
uint64_t lastSuid = 0;
STableMeta *lastTableMeta = NULL;
SColRef *tmpRef = NULL;
SName *pName = taosArrayGet(pList, 0);
if (NULL == pName) {
ctgError("fail to get the 0th SName from tableList, tableNum:%d", (int32_t)taosArrayGetSize(pList));
@ -3505,26 +3570,37 @@ int32_t ctgGetTbMetasFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgTbMe
SMetaRes res = {0};
STableMeta *pTableMeta = NULL;
if (tbMeta->tableType != TSDB_CHILD_TABLE) {
if (tbMeta->tableType != TSDB_CHILD_TABLE && tbMeta->tableType != TSDB_VIRTUAL_CHILD_TABLE) {
int32_t schemaExtSize = 0;
int32_t colRefSize = 0;
int32_t metaSize = CTG_META_SIZE(tbMeta);
if (tbMeta->schemaExt != NULL) {
if (withExtSchema(tbMeta->tableType) && tbMeta->schemaExt != NULL) {
schemaExtSize = tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
if (hasRefCol(tbMeta->tableType) && tbMeta->colRef) {
colRefSize = tbMeta->tableInfo.numOfColumns * sizeof(SColRef);
}
pTableMeta = taosMemoryCalloc(1, metaSize + schemaExtSize);
pTableMeta = taosMemoryCalloc(1, metaSize + schemaExtSize + colRefSize);
if (NULL == pTableMeta) {
ctgReleaseTbMetaToCache(pCtg, dbCache, pCache);
CTG_ERR_RET(terrno);
}
TAOS_MEMCPY(pTableMeta, tbMeta, metaSize);
if (tbMeta->schemaExt != NULL) {
if (withExtSchema(tbMeta->tableType) && tbMeta->schemaExt != NULL) {
pTableMeta->schemaExt = (SSchemaExt *)((char *)pTableMeta + metaSize);
TAOS_MEMCPY(pTableMeta->schemaExt, tbMeta->schemaExt, schemaExtSize);
} else {
pTableMeta->schemaExt = NULL;
}
if (hasRefCol(tbMeta->tableType) && tbMeta->colRef) {
pTableMeta->colRef = (SColRef *)((char *)pTableMeta + metaSize + schemaExtSize);
pTableMeta->numOfColRefs = tbMeta->tableInfo.numOfColumns;
TAOS_MEMCPY(pTableMeta->colRef, tbMeta->colRef, colRefSize);
} else {
pTableMeta->colRef = NULL;
}
CTG_UNLOCK(CTG_READ, &pCache->metaLock);
taosHashRelease(dbCache->tbCache, pCache);
@ -3541,7 +3617,7 @@ int32_t ctgGetTbMetasFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgTbMe
// PROCESS FOR CHILD TABLE
if (lastSuid && tbMeta->suid == lastSuid && lastTableMeta) {
if (lastSuid && tbMeta->suid == lastSuid && lastTableMeta && tbMeta->tableType != TSDB_VIRTUAL_CHILD_TABLE) {
code = cloneTableMeta(lastTableMeta, &pTableMeta);
if (code) {
CTG_UNLOCK(CTG_READ, &pCache->metaLock);
@ -3565,12 +3641,23 @@ int32_t ctgGetTbMetasFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgTbMe
}
int32_t metaSize = sizeof(SCTableMeta);
int32_t colRefSize = 0;
int32_t colRefNum = 0;
pTableMeta = taosMemoryCalloc(1, metaSize);
if (NULL == pTableMeta) {
ctgReleaseTbMetaToCache(pCtg, dbCache, pCache);
CTG_ERR_RET(terrno);
}
if (hasRefCol(tbMeta->tableType) && tbMeta->colRef != NULL) {
colRefSize = tbMeta->numOfColRefs * sizeof(SColRef);
colRefNum = tbMeta->numOfColRefs;
taosMemoryFreeClear(tmpRef);
tmpRef = taosMemoryMalloc(colRefSize);
TAOS_MEMCPY(tmpRef, tbMeta->colRef, colRefSize);
}
TAOS_MEMCPY(pTableMeta, tbMeta, metaSize);
CTG_UNLOCK(CTG_READ, &pCache->metaLock);
@ -3651,19 +3738,29 @@ int32_t ctgGetTbMetasFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgTbMe
schemaExtSize = stbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
metaSize = CTG_META_SIZE(stbMeta);
pTableMeta = taosMemoryRealloc(pTableMeta, metaSize + schemaExtSize);
pTableMeta = taosMemoryRealloc(pTableMeta, metaSize + schemaExtSize + colRefSize);
if (NULL == pTableMeta) {
ctgReleaseTbMetaToCache(pCtg, dbCache, pCache);
taosMemoryFreeClear(tmpRef);
CTG_ERR_RET(terrno);
}
TAOS_MEMCPY(&pTableMeta->sversion, &stbMeta->sversion, metaSize + schemaExtSize - sizeof(SCTableMeta));
if (stbMeta->schemaExt != NULL) {
TAOS_MEMCPY(&pTableMeta->numOfColRefs, &stbMeta->numOfColRefs, metaSize + schemaExtSize - sizeof(SCTableMeta));
if (withExtSchema(stbMeta->tableType) && stbMeta->schemaExt != NULL) {
pTableMeta->schemaExt = (SSchemaExt *)((char *)pTableMeta + metaSize);
} else {
pTableMeta->schemaExt = NULL;
}
if (colRefSize != 0) {
pTableMeta->numOfColRefs = colRefNum;
pTableMeta->colRef = (SColRef *)((char *)pTableMeta + metaSize + schemaExtSize);
TAOS_MEMCPY(pTableMeta->colRef, tmpRef, colRefSize);
} else {
pTableMeta->colRef = NULL;
}
taosMemoryFreeClear(tmpRef);
CTG_UNLOCK(CTG_READ, &pCache->metaLock);
taosHashRelease(dbCache->tbCache, pCache);
@ -3679,7 +3776,7 @@ int32_t ctgGetTbMetasFromCache(SCatalog *pCtg, SRequestConnInfo *pConn, SCtgTbMe
}
_return:
taosMemoryFreeClear(tmpRef);
ctgReleaseDBCache(pCtg, dbCache);
return code;

38
source/libs/catalog/src/ctgRemote.c
View File

@ -412,6 +412,17 @@ int32_t ctgProcessRspMsg(void* out, int32_t reqType, char* msg, int32_t msgSize,
}
break;
}
case TDMT_VND_VSUBTABLES_META: {
if (TSDB_CODE_SUCCESS != rspCode) {
CTG_ERR_RET(rspCode);
}
code = queryProcessMsgRsp[TMSG_INDEX(reqType)](out, msg, msgSize);
if (code) {
qError("Process get vnode virtual subtables rsp failed, err: %s, tbFName: %s", tstrerror(code), target);
CTG_ERR_RET(code);
}
break;
}
default:
if (TSDB_CODE_SUCCESS != rspCode) {
qError("get error rsp, error:%s", tstrerror(rspCode));
@ -1828,3 +1839,30 @@ int32_t ctgGetStreamProgressFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, c
return TSDB_CODE_SUCCESS;
}
int32_t ctgGetVSubTablesFromVnode(SCatalog* pCtg, SRequestConnInfo* pConn, int64_t suid, SVgroupInfo* vgroupInfo, SCtgTaskReq* tReq) {
SCtgTask* pTask = tReq ? tReq->pTask : NULL;
void* (*mallocFp)(int64_t) = pTask ? (MallocType)taosMemMalloc : (MallocType)rpcMallocCont;
int32_t reqType = TDMT_VND_VSUBTABLES_META;
SEp* pEp = &vgroupInfo->epSet.eps[vgroupInfo->epSet.inUse];
ctgDebug("try to get vsubtables meta from vnode, vgId:%d, ep num:%d, ep %s:%d, suid:%" PRIu64, vgroupInfo->vgId,
vgroupInfo->epSet.numOfEps, pEp->fqdn, pEp->port, suid);
char* msg = NULL;
int32_t msgLen = 0;
int32_t code = queryBuildMsg[TMSG_INDEX(reqType)](&suid, &msg, 0, &msgLen, mallocFp);
if (code) {
ctgError("Build vnode vsubtables meta msg failed, code:%x, suid:%" PRIu64, code, suid);
CTG_ERR_RET(code);
}
SRequestConnInfo vConn = {.pTrans = pConn->pTrans,
.requestId = pConn->requestId,
.requestObjRefId = pConn->requestObjRefId,
.mgmtEps = vgroupInfo->epSet};
return ctgAddBatch(pCtg, vgroupInfo->vgId, &vConn, tReq, reqType, msg, msgLen);
}

142
source/libs/catalog/src/ctgUtil.c
View File

@ -202,6 +202,9 @@ void ctgFreeSMetaData(SMetaData* pData) {
taosArrayDestroy(pData->pTsmas);
pData->pTsmas = NULL;
taosArrayDestroyEx(pData->pVSubTables, tDestroySVSubTablesRsp);
pData->pVSubTables = NULL;
taosMemoryFreeClear(pData->pSvrVer);
}
@ -591,6 +594,7 @@ void ctgFreeMsgCtx(SCtgMsgCtx* pCtx) {
case TDMT_VND_TABLE_NAME: {
STableMetaOutput* pOut = (STableMetaOutput*)pCtx->out;
taosMemoryFree(pOut->tbMeta);
taosMemoryFree(pOut->vctbMeta);
taosMemoryFreeClear(pCtx->out);
break;
}
@ -656,6 +660,10 @@ void ctgFreeMsgCtx(SCtgMsgCtx* pCtx) {
}
break;
}
case TDMT_VND_VSUBTABLES_META: {
taosMemoryFreeClear(pCtx->target);
break;
}
default:
qError("invalid reqType %d", pCtx->reqType);
break;
@ -676,6 +684,7 @@ void ctgFreeSTableMetaOutput(STableMetaOutput* pOutput) {
}
taosMemoryFree(pOutput->tbMeta);
taosMemoryFree(pOutput->vctbMeta);
taosMemoryFree(pOutput);
}
@ -857,6 +866,9 @@ void ctgFreeTaskRes(CTG_TASK_TYPE type, void** pRes) {
*pRes = NULL; // no need to free it
break;
}
case CTG_TASK_GET_V_SUBTABLES: {
break;
}
default:
qError("invalid task type %d", type);
break;
@ -923,6 +935,9 @@ void ctgFreeSubTaskRes(CTG_TASK_TYPE type, void** pRes) {
taosArrayDestroyEx(*pRes, ctgFreeBatchMeta);
*pRes = NULL;
break;
}
case CTG_TASK_GET_V_SUBTABLES: {
}
default:
qError("invalid task type %d", type);
@ -1049,6 +1064,23 @@ void ctgFreeTaskCtx(SCtgTask* pTask) {
taosMemoryFreeClear(pTask->taskCtx);
break;
}
case CTG_TASK_GET_V_SUBTABLES: {
SCtgVSubTablesCtx* taskCtx = (SCtgVSubTablesCtx*)pTask->taskCtx;
if (taskCtx->clonedVgroups) {
taosArrayDestroy(taskCtx->pVgroups);
taskCtx->pVgroups = NULL;
}
if (taskCtx->pResList) {
for (int32_t i = 0; i < taskCtx->vgNum; ++i) {
SVSubTablesRsp* pVg = taskCtx->pResList + i;
tDestroySVSubTablesRsp(pVg);
}
taosMemoryFreeClear(taskCtx->pResList);
}
taosMemoryFreeClear(taskCtx->pMeta);
taosMemoryFreeClear(pTask->taskCtx);
break;
}
default:
qError("invalid task type %d", pTask->type);
break;
@ -1681,15 +1713,42 @@ int32_t ctgCloneMetaOutput(STableMetaOutput* output, STableMetaOutput** pOutput)
TAOS_MEMCPY(*pOutput, output, sizeof(STableMetaOutput));
if (output->vctbMeta) {
int32_t metaSize = sizeof(SVCTableMeta);
int32_t colRefSize = 0;
if (hasRefCol(output->vctbMeta->tableType) && (*pOutput)->vctbMeta->colRef) {
colRefSize = output->vctbMeta->numOfColRefs * sizeof(SColRef);
}
(*pOutput)->vctbMeta = taosMemoryMalloc(metaSize + colRefSize);
if (NULL == (*pOutput)->vctbMeta) {
qError("malloc %d failed", (int32_t)sizeof(STableMetaOutput));
taosMemoryFreeClear(*pOutput);
CTG_ERR_RET(terrno);
}
TAOS_MEMCPY((*pOutput)->vctbMeta, output->vctbMeta, metaSize);
if (hasRefCol(output->vctbMeta->tableType) && (*pOutput)->vctbMeta->colRef) {
(*pOutput)->vctbMeta->colRef = (SColRef*)((char*)(*pOutput)->vctbMeta + metaSize);
TAOS_MEMCPY((*pOutput)->vctbMeta->colRef, output->vctbMeta->colRef, colRefSize);
} else {
(*pOutput)->vctbMeta->colRef = NULL;
}
}
if (output->tbMeta) {
int32_t metaSize = CTG_META_SIZE(output->tbMeta);
int32_t schemaExtSize = 0;
int32_t colRefSize = 0;
if (withExtSchema(output->tbMeta->tableType) && (*pOutput)->tbMeta->schemaExt) {
schemaExtSize = output->tbMeta->tableInfo.numOfColumns * sizeof(SSchemaExt);
}
if (hasRefCol(output->tbMeta->tableType) && (*pOutput)->tbMeta->colRef) {
colRefSize = output->tbMeta->tableInfo.numOfColumns * sizeof(SColRef);
}
(*pOutput)->tbMeta = taosMemoryMalloc(metaSize + schemaExtSize);
(*pOutput)->tbMeta = taosMemoryMalloc(metaSize + schemaExtSize + colRefSize);
qTrace("tbmeta cloned, size:%d, p:%p", metaSize, (*pOutput)->tbMeta);
if (NULL == (*pOutput)->tbMeta) {
qError("malloc %d failed", (int32_t)sizeof(STableMetaOutput));
taosMemoryFreeClear(*pOutput);
@ -1703,6 +1762,12 @@ int32_t ctgCloneMetaOutput(STableMetaOutput* output, STableMetaOutput** pOutput)
} else {
(*pOutput)->tbMeta->schemaExt = NULL;
}
if (hasRefCol(output->tbMeta->tableType) && (*pOutput)->tbMeta->colRef) {
(*pOutput)->tbMeta->colRef = (SColRef*)((char*)(*pOutput)->tbMeta + metaSize + schemaExtSize);
TAOS_MEMCPY((*pOutput)->tbMeta->colRef, output->tbMeta->colRef, colRefSize);
} else {
(*pOutput)->tbMeta->colRef = NULL;
}
}
return TSDB_CODE_SUCCESS;
@ -1741,7 +1806,7 @@ int32_t ctgCloneTableIndex(SArray* pIndex, SArray** pRes) {
int32_t ctgUpdateSendTargetInfo(SMsgSendInfo* pMsgSendInfo, int32_t msgType, char* dbFName, int32_t vgId) {
if (msgType == TDMT_VND_TABLE_META || msgType == TDMT_VND_TABLE_CFG || msgType == TDMT_VND_BATCH_META ||
msgType == TDMT_VND_TABLE_NAME) {
msgType == TDMT_VND_TABLE_NAME || msgType == TDMT_VND_VSUBTABLES_META || msgType == TDMT_VND_GET_STREAM_PROGRESS) {
pMsgSendInfo->target.type = TARGET_TYPE_VNODE;
pMsgSendInfo->target.vgId = vgId;
pMsgSendInfo->target.dbFName = taosStrdup(dbFName);
@ -2087,12 +2152,12 @@ int32_t ctgChkSetTbAuthRes(SCatalog* pCtg, SCtgAuthReq* req, SCtgAuthRsp* res) {
CTG_ERR_JRET(ctgGetTbMeta(pCtg, req->pConn, &ctx, &pMeta));
}
if (TSDB_SUPER_TABLE == pMeta->tableType || TSDB_NORMAL_TABLE == pMeta->tableType) {
if (TSDB_SUPER_TABLE == pMeta->tableType || TSDB_NORMAL_TABLE == pMeta->tableType || TSDB_VIRTUAL_NORMAL_TABLE == pMeta->tableType) {
res->pRawRes->pass[AUTH_RES_BASIC] = false;
goto _return;
}
if (TSDB_CHILD_TABLE == pMeta->tableType) {
if (TSDB_CHILD_TABLE == pMeta->tableType || TSDB_VIRTUAL_CHILD_TABLE == pMeta->tableType) {
CTG_ERR_JRET(ctgGetCachedStbNameFromSuid(pCtg, dbFName, pMeta->suid, &stbName));
if (NULL == stbName) {
if (req->onlyCache) {
@ -2410,6 +2475,8 @@ FORCE_INLINE uint64_t ctgGetTbMetaCacheSize(STableMeta* pMeta) {
return sizeof(*pMeta) + (pMeta->tableInfo.numOfColumns + pMeta->tableInfo.numOfTags) * sizeof(SSchema);
case TSDB_CHILD_TABLE:
return sizeof(SCTableMeta);
case TSDB_VIRTUAL_CHILD_TABLE:
return sizeof(SVCTableMeta);
default:
return sizeof(*pMeta) + pMeta->tableInfo.numOfColumns * sizeof(SSchema);
}
@ -2775,3 +2842,70 @@ int32_t ctgAddTSMAFetch(SArray** pFetchs, int32_t dbIdx, int32_t tbIdx, int32_t*
return TSDB_CODE_SUCCESS;
}
int32_t ctgBuildNormalChildVtbList(SCtgVSubTablesCtx* pCtx) {
int32_t code = TSDB_CODE_SUCCESS, line = 0;
char tbFName[TSDB_TABLE_FNAME_LEN];
pCtx->pResList = taosMemoryCalloc(1, sizeof(*pCtx->pResList));
QUERY_CHECK_NULL(pCtx->pResList, code, line, _return, terrno);
pCtx->pResList->vgId = pCtx->pMeta->vgId;
pCtx->pResList->pTables = taosArrayInit(1, POINTER_BYTES);
QUERY_CHECK_NULL(pCtx->pResList->pTables, code, line, _return, terrno);
SSHashObj* pSrcTbls = tSimpleHashInit(10, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY));
QUERY_CHECK_NULL(pSrcTbls, code, line, _return, terrno);
int32_t refColsNum = 0;
for (int32_t i = 0; i < pCtx->pMeta->numOfColRefs; ++i) {
if (!pCtx->pMeta->colRef[i].hasRef) {
continue;
}
refColsNum++;
}
SVCTableRefCols* pTb = (SVCTableRefCols*)taosMemoryCalloc(1, refColsNum * sizeof(SRefColInfo) + sizeof(SVCTableRefCols));
QUERY_CHECK_NULL(pTb, code, line, _return, terrno);
pTb->uid = pCtx->pMeta->uid;
pTb->numOfColRefs = refColsNum;
pTb->refCols = (SRefColInfo*)(pTb + 1);
refColsNum = 0;
for (int32_t j = 0; j < pCtx->pMeta->numOfColRefs; j++) {
if (!pCtx->pMeta->colRef[j].hasRef) {
continue;
}
pTb->refCols[refColsNum].colId = pCtx->pMeta->colRef[j].id;
tstrncpy(pTb->refCols[refColsNum].refColName, pCtx->pMeta->colRef[j].refColName, TSDB_COL_NAME_LEN);
tstrncpy(pTb->refCols[refColsNum].refTableName, pCtx->pMeta->colRef[j].refTableName, TSDB_TABLE_NAME_LEN);
tstrncpy(pTb->refCols[refColsNum].refDbName,pCtx->pMeta->colRef[j].refDbName, TSDB_DB_NAME_LEN);
snprintf(tbFName, sizeof(tbFName), "%s.%s", pTb->refCols[refColsNum].refDbName, pTb->refCols[refColsNum].refTableName);
if (NULL == tSimpleHashGet(pSrcTbls, tbFName, strlen(tbFName))) {
QUERY_CHECK_CODE(tSimpleHashPut(pSrcTbls, tbFName, strlen(tbFName), &code, sizeof(code)), line, _return);
}
refColsNum++;
}
pTb->numOfSrcTbls = tSimpleHashGetSize(pSrcTbls);
QUERY_CHECK_NULL(taosArrayPush(pCtx->pResList->pTables, &pTb), code, line, _return, terrno);
pTb = NULL;
_return:
tSimpleHashCleanup(pSrcTbls);
taosMemoryFree(pTb);
if (code) {
qError("%s failed since %s", __func__, tstrerror(code));
}
return code;
}

2
source/libs/catalog/test/catalogTests.cpp
View File

@ -210,6 +210,8 @@ void ctgTestBuildCTableMetaOutput(STableMetaOutput *output) {
output->tbMeta->sversion = ctgTestSVersion;
output->tbMeta->tversion = ctgTestTVersion;
output->vctbMeta = NULL;
SSchema *s = NULL;
s = &output->tbMeta->schema[0];
s->type = TSDB_DATA_TYPE_TIMESTAMP;

1
source/libs/command/inc/commandInt.h
View File

@ -29,6 +29,7 @@ extern "C" {
//newline area
#define EXPLAIN_TAG_SCAN_FORMAT "Tag Scan on %s"
#define EXPLAIN_VIRTUAL_TABLE_SCAN_FORMAT "Virtual Table Scan on %s"
#define EXPLAIN_TBL_SCAN_FORMAT "Table Scan on %s"
#define EXPLAIN_TBL_MERGE_SCAN_FORMAT "Table Merge Scan on %s"
#define EXPLAIN_SYSTBL_SCAN_FORMAT "System Table Scan on %s"

116
source/libs/command/src/command.c
View File

@ -122,7 +122,10 @@ static int32_t buildDescResultDataBlock(SSDataBlock** pOutput) {
infoData = createColumnInfoData(TSDB_DATA_TYPE_VARCHAR, DESCRIBE_RESULT_COPRESS_OPTION_LEN, 7);
code = blockDataAppendColInfo(pBlock, &infoData);
}
if (TSDB_CODE_SUCCESS == code) {
infoData = createColumnInfoData(TSDB_DATA_TYPE_VARCHAR, DESCRIBE_RESULT_COL_REF_LEN, 8);
code = blockDataAppendColInfo(pBlock, &infoData);
}
if (TSDB_CODE_SUCCESS == code) {
*pOutput = pBlock;
} else {
@ -151,12 +154,18 @@ static int32_t setDescResultIntoDataBlock(bool sysInfoUser, SSDataBlock* pBlock,
SColumnInfoData* pCol6 = NULL;
// level
SColumnInfoData* pCol7 = NULL;
// colref
SColumnInfoData* pCol8 = NULL;
if (withExtSchema(pMeta->tableType)) {
pCol5 = taosArrayGet(pBlock->pDataBlock, 4);
pCol6 = taosArrayGet(pBlock->pDataBlock, 5);
pCol7 = taosArrayGet(pBlock->pDataBlock, 6);
}
if (hasRefCol(pMeta->tableType)) {
pCol5 = taosArrayGet(pBlock->pDataBlock, 4);
}
int32_t fillTagCol = 0;
char buf[DESCRIBE_RESULT_FIELD_LEN] = {0};
for (int32_t i = 0; i < numOfRows; ++i) {
@ -207,6 +216,24 @@ static int32_t setDescResultIntoDataBlock(bool sysInfoUser, SSDataBlock* pBlock,
STR_TO_VARSTR(buf, fillTagCol == 0 ? "" : "disabled");
COL_DATA_SET_VAL_AND_CHECK(pCol7, pBlock->info.rows, buf, false);
}
} else if (hasRefCol(pMeta->tableType) && pMeta->colRef) {
if (i < pMeta->numOfColRefs) {
if (pMeta->colRef[i].hasRef) {
char refColName[TSDB_DB_NAME_LEN + TSDB_NAME_DELIMITER_LEN + TSDB_COL_FNAME_LEN] = {0};
strcat(refColName, pMeta->colRef[i].refDbName);
strcat(refColName, ".");
strcat(refColName, pMeta->colRef[i].refTableName);
strcat(refColName, ".");
strcat(refColName, pMeta->colRef[i].refColName);
STR_TO_VARSTR(buf, refColName);
} else {
STR_TO_VARSTR(buf, "");
}
COL_DATA_SET_VAL_AND_CHECK(pCol5, pBlock->info.rows, buf, false);
} else {
STR_TO_VARSTR(buf, "");
COL_DATA_SET_VAL_AND_CHECK(pCol5, pBlock->info.rows, buf, false);
}
}
fillTagCol = 0;
@ -244,8 +271,14 @@ static int32_t execDescribe(bool sysInfoUser, SNode* pStmt, SRetrieveTableRsp**
code = setDescResultIntoDataBlock(sysInfoUser, pBlock, numOfRows, pDesc->pMeta, biMode);
}
if (TSDB_CODE_SUCCESS == code) {
if (pDesc->pMeta && withExtSchema(pDesc->pMeta->tableType) && pDesc->pMeta->schemaExt) {
code = buildRetrieveTableRsp(pBlock, DESCRIBE_RESULT_COLS_COMPRESS, pRsp);
if (pDesc->pMeta) {
if (withExtSchema(pDesc->pMeta->tableType) && pDesc->pMeta->schemaExt) {
code = buildRetrieveTableRsp(pBlock, DESCRIBE_RESULT_COLS_COMPRESS, pRsp);
} else if (hasRefCol(pDesc->pMeta->tableType) && pDesc->pMeta->colRef) {
code = buildRetrieveTableRsp(pBlock, DESCRIBE_RESULT_COLS_REF, pRsp);
} else {
code = buildRetrieveTableRsp(pBlock, DESCRIBE_RESULT_COLS, pRsp);
}
} else {
code = buildRetrieveTableRsp(pBlock, DESCRIBE_RESULT_COLS, pRsp);
}
@ -536,7 +569,8 @@ static int32_t buildCreateViewResultDataBlock(SSDataBlock** pOutput) {
static void appendColumnFields(char* buf, int32_t* len, STableCfg* pCfg) {
for (int32_t i = 0; i < pCfg->numOfColumns; ++i) {
SSchema* pSchema = pCfg->pSchemas + i;
#define LTYPE_LEN (32 + 60) // 60 byte for compress info
SColRef* pRef = pCfg->pColRefs + i;
#define LTYPE_LEN (32 + 60 + TSDB_COL_FNAME_LEN + TSDB_DB_NAME_LEN + 10) // 60 byte for compress info, TSDB_COL_FNAME_LEN + TSDB_DB_NAME_LEN for column ref
char type[LTYPE_LEN];
snprintf(type, LTYPE_LEN, "%s", tDataTypes[pSchema->type].name);
int typeLen = strlen(type);
@ -560,6 +594,15 @@ static void appendColumnFields(char* buf, int32_t* len, STableCfg* pCfg) {
typeLen += tsnprintf(type + typeLen, LTYPE_LEN - typeLen, " LEVEL \'%s\'",
columnLevelStr(COMPRESS_L2_TYPE_LEVEL_U32(pCfg->pSchemaExt[i].compress)));
}
if (hasRefCol(pCfg->tableType) && pCfg->pColRefs && pRef->hasRef) {
typeLen += tsnprintf(type + typeLen, LTYPE_LEN - typeLen, " FROM `%s`", pRef->refDbName);
typeLen += tsnprintf(type + typeLen, LTYPE_LEN - typeLen, ".");
typeLen += tsnprintf(type + typeLen, LTYPE_LEN - typeLen, "`%s`", pRef->refTableName);
typeLen += tsnprintf(type + typeLen, LTYPE_LEN - typeLen, ".");
typeLen += tsnprintf(type + typeLen, LTYPE_LEN - typeLen, "`%s`", pRef->refColName);
}
if (!(pSchema->flags & COL_IS_KEY)) {
*len += tsnprintf(buf + VARSTR_HEADER_SIZE + *len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + *len),
"%s`%s` %s", ((i > 0) ? ", " : ""), pSchema->name, type);
@ -571,6 +614,29 @@ static void appendColumnFields(char* buf, int32_t* len, STableCfg* pCfg) {
}
}
static void appendColRefFields(char* buf, int32_t* len, STableCfg* pCfg) {
for (int32_t i = 1; i < pCfg->numOfColumns; ++i) {
SSchema* pSchema = pCfg->pSchemas + i;
SColRef* pRef = pCfg->pColRefs + i;
char type[TSDB_COL_NAME_LEN + 10 + TSDB_COL_FNAME_LEN + TSDB_DB_NAME_LEN];
int typeLen = 0;
if (hasRefCol(pCfg->tableType) && pCfg->pColRefs && pRef->hasRef) {
typeLen += tsnprintf(type + typeLen, sizeof(type) - typeLen, "FROM `%s`", pRef->refDbName);
typeLen += tsnprintf(type + typeLen, sizeof(type) - typeLen, ".");
typeLen += tsnprintf(type + typeLen, sizeof(type) - typeLen, "`%s`", pRef->refTableName);
typeLen += tsnprintf(type + typeLen, sizeof(type) - typeLen, ".");
typeLen += tsnprintf(type + typeLen, sizeof(type) - typeLen, "`%s`", pRef->refColName);
} else {
continue;
}
*len += tsnprintf(buf + VARSTR_HEADER_SIZE + *len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + *len),
"%s`%s` %s", ((i > 1) ? ", " : ""), pSchema->name, type);
}
}
static void appendTagFields(char* buf, int32_t* len, STableCfg* pCfg) {
for (int32_t i = 0; i < pCfg->numOfTags; ++i) {
SSchema* pSchema = pCfg->pSchemas + pCfg->numOfColumns + i;
@ -749,6 +815,11 @@ static void appendTableOptions(char* buf, int32_t* len, SDbCfgInfo* pDbCfg, STab
*len += tsnprintf(buf + VARSTR_HEADER_SIZE + *len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - VARSTR_HEADER_SIZE, ")");
}
}
if (pCfg->virtualStb) {
*len += tsnprintf(buf + VARSTR_HEADER_SIZE + *len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + *len),
" VIRTUAL %d", pCfg->virtualStb);
}
}
static int32_t setCreateTBResultIntoDataBlock(SSDataBlock* pBlock, SDbCfgInfo* pDbCfg, char* tbName, STableCfg* pCfg,
@ -791,13 +862,33 @@ static int32_t setCreateTBResultIntoDataBlock(SSDataBlock* pBlock, SDbCfgInfo* p
len +=
snprintf(buf2 + VARSTR_HEADER_SIZE + len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + len), ")");
appendTableOptions(buf2, &len, pDbCfg, pCfg);
} else {
} else if (TSDB_NORMAL_TABLE == pCfg->tableType){
len += tsnprintf(buf2 + VARSTR_HEADER_SIZE, SHOW_CREATE_TB_RESULT_FIELD2_LEN - VARSTR_HEADER_SIZE,
"CREATE TABLE `%s` (", tbName);
appendColumnFields(buf2, &len, pCfg);
len +=
snprintf(buf2 + VARSTR_HEADER_SIZE + len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + len), ")");
appendTableOptions(buf2, &len, pDbCfg, pCfg);
} else if (TSDB_VIRTUAL_NORMAL_TABLE == pCfg->tableType) {
len += tsnprintf(buf2 + VARSTR_HEADER_SIZE, SHOW_CREATE_TB_RESULT_FIELD2_LEN - VARSTR_HEADER_SIZE,
"CREATE VTABLE `%s` (", tbName);
appendColumnFields(buf2, &len, pCfg);
len +=
snprintf(buf2 + VARSTR_HEADER_SIZE + len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + len), ")");
} else if (TSDB_VIRTUAL_CHILD_TABLE == pCfg->tableType) {
len += tsnprintf(buf2 + VARSTR_HEADER_SIZE, SHOW_CREATE_TB_RESULT_FIELD2_LEN - VARSTR_HEADER_SIZE,
"CREATE VTABLE `%s` (", tbName);
appendColRefFields(buf2, &len, pCfg);
len +=
snprintf(buf2 + VARSTR_HEADER_SIZE + len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + len),
") USING `%s` (", pCfg->stbName);
appendTagNameFields(buf2, &len, pCfg);
len += tsnprintf(buf2 + VARSTR_HEADER_SIZE + len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + len),
") TAGS (");
code = appendTagValues(buf2, &len, pCfg, charsetCxt);
TAOS_CHECK_ERRNO(code);
len +=
snprintf(buf2 + VARSTR_HEADER_SIZE + len, SHOW_CREATE_TB_RESULT_FIELD2_LEN - (VARSTR_HEADER_SIZE + len), ")");
}
varDataLen(buf2) = (len > 65535) ? 65535 : len;
@ -856,6 +947,19 @@ static int32_t execShowCreateTable(SShowCreateTableStmt* pStmt, SRetrieveTableRs
return code;
}
static int32_t execShowCreateVTable(SShowCreateTableStmt* pStmt, SRetrieveTableRsp** pRsp, void* charsetCxt) {
SSDataBlock* pBlock = NULL;
int32_t code = buildCreateTbResultDataBlock(&pBlock);
if (TSDB_CODE_SUCCESS == code) {
code = setCreateTBResultIntoDataBlock(pBlock, pStmt->pDbCfg, pStmt->tableName, pStmt->pTableCfg, charsetCxt);
}
if (TSDB_CODE_SUCCESS == code) {
code = buildRetrieveTableRsp(pBlock, SHOW_CREATE_TB_RESULT_COLS, pRsp);
}
(void)blockDataDestroy(pBlock);
return code;
}
static int32_t execShowCreateSTable(SShowCreateTableStmt* pStmt, SRetrieveTableRsp** pRsp, void* charsetCxt) {
STableCfg* pCfg = (STableCfg*)pStmt->pTableCfg;
if (TSDB_SUPER_TABLE != pCfg->tableType) {
@ -1109,6 +1213,8 @@ int32_t qExecCommand(int64_t* pConnId, bool sysInfoUser, SNode* pStmt, SRetrieve
return execShowCreateDatabase((SShowCreateDatabaseStmt*)pStmt, pRsp);
case QUERY_NODE_SHOW_CREATE_TABLE_STMT:
return execShowCreateTable((SShowCreateTableStmt*)pStmt, pRsp, charsetCxt);
case QUERY_NODE_SHOW_CREATE_VTABLE_STMT:
return execShowCreateVTable((SShowCreateTableStmt*)pStmt, pRsp, charsetCxt);
case QUERY_NODE_SHOW_CREATE_STABLE_STMT:
return execShowCreateSTable((SShowCreateTableStmt*)pStmt, pRsp, charsetCxt);
case QUERY_NODE_SHOW_CREATE_VIEW_STMT:

46
source/libs/command/src/explain.c
View File

@ -420,6 +420,52 @@ static int32_t qExplainResNodeToRowsImpl(SExplainResNode *pResNode, SExplainCtx
}
break;
}
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN: {
SVirtualScanPhysiNode *pVirtualTableScanNode = (SVirtualScanPhysiNode *)pNode;
EXPLAIN_ROW_NEW(level, EXPLAIN_VIRTUAL_TABLE_SCAN_FORMAT, pVirtualTableScanNode->scan.tableName.tname);
EXPLAIN_ROW_APPEND(EXPLAIN_LEFT_PARENTHESIS_FORMAT);
if (pResNode->pExecInfo) {
QRY_ERR_RET(qExplainBufAppendExecInfo(pResNode->pExecInfo, tbuf, &tlen));
EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
}
if (pVirtualTableScanNode->scan.pScanPseudoCols) {
EXPLAIN_ROW_APPEND(EXPLAIN_PSEUDO_COLUMNS_FORMAT, pVirtualTableScanNode->scan.pScanPseudoCols->length);
EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
}
EXPLAIN_ROW_APPEND(EXPLAIN_WIDTH_FORMAT, pVirtualTableScanNode->scan.node.pOutputDataBlockDesc->totalRowSize);
EXPLAIN_ROW_APPEND(EXPLAIN_RIGHT_PARENTHESIS_FORMAT);
EXPLAIN_ROW_END();
QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level));
if (verbose) {
EXPLAIN_ROW_NEW(level + 1, EXPLAIN_OUTPUT_FORMAT);
EXPLAIN_ROW_APPEND(EXPLAIN_COLUMNS_FORMAT,
nodesGetOutputNumFromSlotList(pVirtualTableScanNode->scan.node.pOutputDataBlockDesc->pSlots));
EXPLAIN_ROW_APPEND(EXPLAIN_BLANK_FORMAT);
EXPLAIN_ROW_APPEND(EXPLAIN_WIDTH_FORMAT, pVirtualTableScanNode->scan.node.pOutputDataBlockDesc->outputRowSize);
EXPLAIN_ROW_APPEND_LIMIT(pVirtualTableScanNode->scan.node.pLimit);
EXPLAIN_ROW_APPEND_SLIMIT(pVirtualTableScanNode->scan.node.pSlimit);
EXPLAIN_ROW_END();
QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1));
if (pResNode->pExecInfo) {
QRY_ERR_RET(qExplainBufAppendVerboseExecInfo(pResNode->pExecInfo, tbuf, &tlen));
if (tlen) {
EXPLAIN_ROW_END();
QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1));
}
}
if (pVirtualTableScanNode->scan.node.pConditions) {
EXPLAIN_ROW_NEW(level + 1, EXPLAIN_FILTER_FORMAT);
QRY_ERR_RET(nodesNodeToSQL(pVirtualTableScanNode->scan.node.pConditions, tbuf + VARSTR_HEADER_SIZE,
TSDB_EXPLAIN_RESULT_ROW_SIZE, &tlen));
EXPLAIN_ROW_END();
QRY_ERR_RET(qExplainResAppendRow(ctx, tbuf, tlen, level + 1));
}
}
break;
}
case QUERY_NODE_PHYSICAL_PLAN_TABLE_SEQ_SCAN:
case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN:
case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN: {

2
source/libs/executor/inc/dataSinkInt.h
View File

@ -52,7 +52,7 @@ typedef struct SDataSinkHandle {
FGetSinkFlags fGetFlags;
} SDataSinkHandle;
int32_t createDataDispatcher(SDataSinkManager* pManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle);
int32_t createDataDispatcher(SDataSinkManager* pManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle, bool processOneBlock);
int32_t createDataDeleter(SDataSinkManager* pManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle,
void* pParam);
int32_t createDataInserter(SDataSinkManager* pManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle,

18
source/libs/executor/inc/dynqueryctrl.h
View File

@ -74,10 +74,28 @@ typedef struct SStbJoinDynCtrlInfo {
SDataBlockDescNode* pOutputDataBlockDesc;
} SStbJoinDynCtrlInfo;
typedef struct SVtbScanDynCtrlInfo {
int32_t acctId;
SUseDbRsp* pRsp;
SUseDbReq req;
tsem_t ready;
SEpSet epSet;
uint64_t suid;
SReadHandle readHandle;
SArray* childTableList;
int32_t lastTableIdx;
SOperatorParam* vtbScanParam;
int32_t readTableIdx;
SHashObj* dbVgInfoMap;
SArray* readColList;
bool scanAllCols;
} SVtbScanDynCtrlInfo;
typedef struct SDynQueryCtrlOperatorInfo {
EDynQueryType qType;
union {
SStbJoinDynCtrlInfo stbJoin;
SVtbScanDynCtrlInfo vtbScan;
};
} SDynQueryCtrlOperatorInfo;

2
source/libs/executor/inc/executil.h
View File

@ -192,6 +192,8 @@ SColumn extractColumnFromColumnNode(SColumnNode* pColNode);
int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysiNode* pTableScanNode,
const SReadHandle* readHandle);
int32_t initQueryTableDataCondWithColArray(SQueryTableDataCond* pCond, SQueryTableDataCond* pOrgCond,
const SReadHandle* readHandle, SArray* colArray);
void cleanupQueryTableDataCond(SQueryTableDataCond* pCond);
int32_t convertFillType(int32_t mode);

34
source/libs/executor/inc/executorInt.h
View File

@ -31,6 +31,7 @@ extern "C" {
#include "executor.h"
#include "planner.h"
#include "scalar.h"
#include "streamVtableMerge.h"
#include "taosdef.h"
#include "tarray.h"
#include "tfill.h"
@ -161,10 +162,13 @@ typedef struct SSortMergeJoinOperatorParam {
} SSortMergeJoinOperatorParam;
typedef struct SExchangeOperatorBasicParam {
int32_t vgId;
int32_t srcOpType;
bool tableSeq;
SArray* uidList;
int32_t vgId;
int32_t srcOpType;
bool tableSeq;
SArray* uidList;
bool isVtbRefScan;
SOrgTbInfo* colMap;
STimeWindow window;
} SExchangeOperatorBasicParam;
typedef struct SExchangeOperatorBatchParam {
@ -253,6 +257,7 @@ typedef struct STableScanBase {
STsdbReader* dataReader;
SFileBlockLoadRecorder readRecorder;
SQueryTableDataCond cond;
SQueryTableDataCond orgCond; // use for virtual super table scan
SAggOptrPushDownInfo pdInfo;
SColMatchInfo matchInfo;
SReadHandle readHandle;
@ -283,6 +288,9 @@ typedef struct STableScanInfo {
bool hasGroupByTag;
bool filesetDelimited;
bool needCountEmptyTable;
SSDataBlock* pOrgBlock;
bool ignoreTag;
bool virtualStableScan;
} STableScanInfo;
typedef enum ESubTableInputType {
@ -538,6 +546,8 @@ typedef struct SStreamScanInfo {
uint64_t numOfExec; // execution times
STqReader* tqReader;
SHashObj* pVtableMergeHandles; // key: vtable uid, value: SStreamVtableMergeHandle
uint64_t groupId;
bool igCheckGroupId;
struct SUpdateInfo* pUpdateInfo;
@ -1190,6 +1200,7 @@ void* decodeSTimeWindowAggSupp(void* buf, STimeWindowAggSupp* pTwAggSup);
void destroyOperatorParamValue(void* pValues);
int32_t mergeOperatorParams(SOperatorParam* pDst, SOperatorParam* pSrc);
int32_t buildTableScanOperatorParam(SOperatorParam** ppRes, SArray* pUidList, int32_t srcOpType, bool tableSeq);
int32_t buildTableScanOperatorParamEx(SOperatorParam** ppRes, SArray* pUidList, int32_t srcOpType, SOrgTbInfo *pMap, bool tableSeq, STimeWindow *window);
void freeExchangeGetBasicOperatorParam(void* pParam);
void freeOperatorParam(SOperatorParam* pParam, SOperatorParamType type);
void freeResetOperatorParams(struct SOperatorInfo* pOperator, SOperatorParamType type, bool allFree);
@ -1216,6 +1227,21 @@ bool getIgoreNullRes(SExprSupp* pExprSup);
bool checkNullRow(SExprSupp* pExprSup, SSDataBlock* pSrcBlock, int32_t index, bool ignoreNull);
int64_t getMinWindowSize(struct SOperatorInfo* pOperator);
void destroyStreamScanOperatorInfo(void* param);
void prepareRangeScan(SStreamScanInfo* pInfo, SSDataBlock* pBlock, int32_t* pRowIndex, bool* pRes);
int32_t setBlockGroupIdByUid(SStreamScanInfo* pInfo, SSDataBlock* pBlock);
bool hasScanRange(SStreamScanInfo* pInfo);
bool isStreamWindow(SStreamScanInfo* pInfo);
int32_t copyGetResultBlock(SSDataBlock* dest, TSKEY start, TSKEY end);
int32_t generateDeleteResultBlock(SStreamScanInfo* pInfo, SSDataBlock* pSrcBlock, SSDataBlock* pDestBlock);
int32_t doCheckUpdate(SStreamScanInfo* pInfo, TSKEY endKey, SSDataBlock* pBlock);
int32_t checkUpdateData(SStreamScanInfo* pInfo, bool invertible, SSDataBlock* pBlock, bool out);
int32_t calBlockTbName(SStreamScanInfo* pInfo, SSDataBlock* pBlock, int32_t rowId);
int32_t setBlockIntoRes(SStreamScanInfo* pInfo, const SSDataBlock* pBlock, STimeWindow* pTimeWindow, bool filter);
int32_t generateScanRange(SStreamScanInfo* pInfo, SSDataBlock* pSrcBlock, SSDataBlock* pDestBlock, EStreamType type);
int32_t doRangeScan(SStreamScanInfo* pInfo, SSDataBlock* pSDB, int32_t tsColIndex, int32_t* pRowIndex,
SSDataBlock** ppRes);
#ifdef __cplusplus
}
#endif

8
source/libs/executor/inc/operator.h
View File

@ -172,7 +172,7 @@ int32_t createGroupCacheOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfD
int32_t createAnomalywindowOperatorInfo(SOperatorInfo* downstream, SPhysiNode* physiNode, SExecTaskInfo* pTaskInfo, SOperatorInfo** pInfo);
int32_t createDynQueryCtrlOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SDynQueryCtrlPhysiNode* pPhyciNode, SExecTaskInfo* pTaskInfo, SOperatorInfo** pInfo);
int32_t createDynQueryCtrlOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream, SDynQueryCtrlPhysiNode* pPhyciNode, SExecTaskInfo* pTaskInfo, SReadHandle* pHandle, SOperatorInfo** pInfo);
int32_t createStreamTimeSliceOperatorInfo(SOperatorInfo* downstream, SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHandle* pHandle, SOperatorInfo** ppOptInfo);
@ -186,6 +186,9 @@ int32_t createStateNonblockOperatorInfo(SOperatorInfo* downstream, SPhysiNode* p
int32_t createEventNonblockOperatorInfo(SOperatorInfo* downstream, SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHandle* pHandle, SOperatorInfo** ppOptInfo);
int32_t createVirtualTableMergeOperatorInfo(SOperatorInfo** pDownstream, SReadHandle* readHandle, STableListInfo* pTableListInfo, int32_t numOfDownstream, SVirtualScanPhysiNode * pJoinNode, SExecTaskInfo* pTaskInfo, SOperatorInfo** pOptrInfo);
int32_t createStreamVtableMergeOperatorInfo(SReadHandle* pHandle, SVirtualScanPhysiNode* pVirtualScanNode, SNode* pTagCond, SExecTaskInfo* pTaskInfo, SOperatorInfo** pInfo);
// clang-format on
SOperatorFpSet createOperatorFpSet(__optr_open_fn_t openFn, __optr_fn_t nextFn, __optr_fn_t cleanup,
@ -205,7 +208,8 @@ SSDataBlock* getNextBlockFromDownstreamRemain(struct SOperatorInfo* pOperator,
int16_t getOperatorResultBlockId(struct SOperatorInfo* pOperator, int32_t idx);
int32_t createOperator(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHandle* pHandle, SNode* pTagCond,
SNode* pTagIndexCond, const char* pUser, const char* dbname, SOperatorInfo** pOptrInfo);
SNode* pTagIndexCond, const char* pUser, const char* dbname, SOperatorInfo** pOptrInfo,
EOPTR_EXEC_MODEL model);
void destroyOperator(SOperatorInfo* pOperator);
void destroyOperatorAndDownstreams(SOperatorInfo* pOperator, SOperatorInfo** stream, int32_t num);

47
source/libs/executor/inc/streamVtableMerge.h Normal file
View File

@ -0,0 +1,47 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef TDENGINE_STREAM_VTABLE_MERGE_H
#define TDENGINE_STREAM_VTABLE_MERGE_H
#ifdef __cplusplus
extern "C" {
#endif
#include "tcommon.h"
#include "tpagedbuf.h"
typedef struct SStreamVtableMergeHandle SStreamVtableMergeHandle;
typedef enum {
SVM_NEXT_NOT_READY = 0,
SVM_NEXT_FOUND = 1,
} SVM_NEXT_RESULT;
int32_t streamVtableMergeCreateHandle(SStreamVtableMergeHandle **ppHandle, int32_t nSrcTbls, int32_t numPageLimit,
SDiskbasedBuf *pBuf, SSDataBlock *pResBlock, const char *idstr);
void streamVtableMergeDestroyHandle(SStreamVtableMergeHandle **ppHandle);
int32_t streamVtableMergeAddBlock(SStreamVtableMergeHandle *pHandle, SSDataBlock *pDataBlock, const char *idstr);
int32_t streamVtableMergeNextTuple(SStreamVtableMergeHandle *pHandle, SSDataBlock *pResBlock, SVM_NEXT_RESULT *pRes,
const char *idstr);
#ifdef __cplusplus
}
#endif
#endif // TDENGINE_STREAM_VTABLE_MERGE_H

16
source/libs/executor/inc/tsort.h
View File

@ -166,6 +166,22 @@ void tsortGetValue(STupleHandle* pVHandle, int32_t colId, void** pVal);
*/
uint64_t tsortGetGroupId(STupleHandle* pVHandle);
void tsortGetBlockInfo(STupleHandle* pVHandle, SDataBlockInfo* pInfo);
/**
* return the SColumnInfoData of columns in the tuple
* @param pVHandle
* @param colId
* @return
*/
void tsortGetColumnInfo(STupleHandle* pVHandle, int32_t colIndex, SColumnInfoData** pColInfo);
/**
* return the number of columns in the tuple
* @param pVHandle
* @return
*/
size_t tsortGetColNum(STupleHandle* pVHandle);
/**
*
* @param pSortHandle

47
source/libs/executor/inc/virtualtablescan.h Normal file
View File

@ -0,0 +1,47 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#ifndef TDENGINE_VIRTUALTABLESCAN_H
#define TDENGINE_VIRTUALTABLESCAN_H
#ifdef __cplusplus
extern "C" {
#endif
#include "executorInt.h"
#include "operator.h"
#define VTS_ERR_RET(c) \
do { \
int32_t _code = (c); \
if (_code != TSDB_CODE_SUCCESS) { \
terrno = _code; \
return _code; \
} \
} while (0)
#define VTS_ERR_JRET(c) \
do { \
code = (c); \
if (code != TSDB_CODE_SUCCESS) { \
terrno = code; \
goto _return; \
} \
} while (0)
#ifdef __cplusplus
}
#endif
#endif // TDENGINE_VIRTUALTABLESCAN_H

7
source/libs/executor/src/dataDispatcher.c
View File

@ -268,7 +268,7 @@ static int32_t putDataBlock(SDataSinkHandle* pHandle, const SInputData* pInput,
QRY_ERR_JRET(taosWriteQitem(pDispatcher->pDataBlocks, pBuf));
int32_t status = updateStatus(pDispatcher);
*pContinue = (status == DS_BUF_LOW || status == DS_BUF_EMPTY);
*pContinue = (status == DS_BUF_LOW || status == DS_BUF_EMPTY) && !(pDispatcher->flags & DS_FLAG_PROCESS_ONE_BLOCK);
return TSDB_CODE_SUCCESS;
_return:
@ -438,7 +438,7 @@ int32_t getOutputColCounts(SDataBlockDescNode* pInputDataBlockDesc) {
return numOfCols;
}
int32_t createDataDispatcher(SDataSinkManager* pManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle) {
int32_t createDataDispatcher(SDataSinkManager* pManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle, bool processOneBlock) {
int32_t code;
SDataSinkNode* pDataSink = *ppDataSink;
code = blockDescNodeCheck(pDataSink->pInputDataBlockDesc);
@ -480,6 +480,9 @@ int32_t createDataDispatcher(SDataSinkManager* pManager, SDataSinkNode** ppDataS
}
dispatcher->flags = DS_FLAG_USE_MEMPOOL;
if (processOneBlock) {
dispatcher->flags |= DS_FLAG_PROCESS_ONE_BLOCK;
}
*pHandle = dispatcher;
return TSDB_CODE_SUCCESS;

4
source/libs/executor/src/dataSinkMgt.c
View File

@ -39,11 +39,11 @@ int32_t dsDataSinkGetCacheSize(SDataSinkStat* pStat) {
return TSDB_CODE_SUCCESS;
}
int32_t dsCreateDataSinker(void* pSinkManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle, void* pParam, const char* id) {
int32_t dsCreateDataSinker(void* pSinkManager, SDataSinkNode** ppDataSink, DataSinkHandle* pHandle, void* pParam, const char* id, bool processOneBlock) {
SDataSinkManager* pManager = pSinkManager;
switch ((int)nodeType(*ppDataSink)) {
case QUERY_NODE_PHYSICAL_PLAN_DISPATCH:
return createDataDispatcher(pManager, ppDataSink, pHandle);
return createDataDispatcher(pManager, ppDataSink, pHandle, processOneBlock);
case QUERY_NODE_PHYSICAL_PLAN_DELETE: {
return createDataDeleter(pManager, ppDataSink, pHandle, pParam);
}

514
source/libs/executor/src/dynqueryctrloperator.c
View File

@ -15,7 +15,6 @@
#include "executorInt.h"
#include "filter.h"
#include "function.h"
#include "nodes.h"
#include "operator.h"
#include "os.h"
@ -28,6 +27,7 @@
#include "tdatablock.h"
#include "thash.h"
#include "tmsg.h"
#include "trpc.h"
#include "ttypes.h"
#include "dynqueryctrl.h"
@ -80,6 +80,35 @@ static void destroyStbJoinDynCtrlInfo(SStbJoinDynCtrlInfo* pStbJoin) {
destroyStbJoinTableList(pStbJoin->ctx.prev.pListHead);
}
void freeUseDbOutput(void* pOutput) {
SUseDbOutput *pOut = *(SUseDbOutput**)pOutput;
if (NULL == pOutput) {
return;
}
if (pOut->dbVgroup) {
freeVgInfo(pOut->dbVgroup);
}
taosMemFree(pOut);
}
static void destroyVtbScanDynCtrlInfo(SVtbScanDynCtrlInfo* pVtbScan) {
if (pVtbScan->childTableList) {
taosArrayDestroy(pVtbScan->childTableList);
}
if (pVtbScan->readColList) {
taosArrayDestroy(pVtbScan->readColList);
}
if (pVtbScan->dbVgInfoMap) {
taosHashSetFreeFp(pVtbScan->dbVgInfoMap, freeUseDbOutput);
taosHashCleanup(pVtbScan->dbVgInfoMap);
}
if (pVtbScan->pRsp) {
tFreeSUsedbRsp(pVtbScan->pRsp);
taosMemoryFreeClear(pVtbScan->pRsp);
}
}
static void destroyDynQueryCtrlOperator(void* param) {
SDynQueryCtrlOperatorInfo* pDyn = (SDynQueryCtrlOperatorInfo*)param;
@ -87,6 +116,9 @@ static void destroyDynQueryCtrlOperator(void* param) {
case DYN_QTYPE_STB_HASH:
destroyStbJoinDynCtrlInfo(&pDyn->stbJoin);
break;
case DYN_QTYPE_VTB_SCAN:
destroyVtbScanDynCtrlInfo(&pDyn->vtbScan);
break;
default:
qError("unsupported dynamic query ctrl type: %d", pDyn->qType);
break;
@ -199,6 +231,7 @@ static int32_t buildGroupCacheOperatorParam(SOperatorParam** ppRes, int32_t down
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_GROUP_CACHE;
(*ppRes)->downstreamIdx = downstreamIdx;
(*ppRes)->value = pGc;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
@ -226,12 +259,13 @@ static int32_t buildGroupCacheNotifyOperatorParam(SOperatorParam** ppRes, int32_
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_GROUP_CACHE;
(*ppRes)->downstreamIdx = downstreamIdx;
(*ppRes)->value = pGc;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
static int32_t buildExchangeOperatorParam(SOperatorParam** ppRes, int32_t downstreamIdx, int32_t* pVgId, int64_t* pUid) {
static int32_t buildExchangeOperatorParam(SOperatorParam** ppRes, int32_t downstreamIdx, const int32_t* pVgId, int64_t* pUid) {
*ppRes = taosMemoryMalloc(sizeof(SOperatorParam));
if (NULL == *ppRes) {
return terrno;
@ -246,7 +280,9 @@ static int32_t buildExchangeOperatorParam(SOperatorParam** ppRes, int32_t downst
pExc->multiParams = false;
pExc->basic.vgId = *pVgId;
pExc->basic.tableSeq = true;
pExc->basic.isVtbRefScan = false;
pExc->basic.srcOpType = QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN;
pExc->basic.colMap = NULL;
pExc->basic.uidList = taosArrayInit(1, sizeof(int64_t));
if (NULL == pExc->basic.uidList) {
taosMemoryFree(pExc);
@ -261,11 +297,11 @@ static int32_t buildExchangeOperatorParam(SOperatorParam** ppRes, int32_t downst
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_EXCHANGE;
(*ppRes)->downstreamIdx = downstreamIdx;
(*ppRes)->value = pExc;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
static int32_t buildBatchExchangeOperatorParam(SOperatorParam** ppRes, int32_t downstreamIdx, SSHashObj* pVg) {
*ppRes = taosMemoryMalloc(sizeof(SOperatorParam));
if (NULL == *ppRes) {
@ -298,7 +334,9 @@ static int32_t buildBatchExchangeOperatorParam(SOperatorParam** ppRes, int32_t d
SArray* pUidList = *(SArray**)p;
basic.vgId = *pVgId;
basic.uidList = pUidList;
basic.colMap = NULL;
basic.tableSeq = false;
basic.isVtbRefScan = false;
QRY_ERR_RET(tSimpleHashPut(pExc->pBatchs, pVgId, sizeof(*pVgId), &basic, sizeof(basic)));
@ -309,10 +347,65 @@ static int32_t buildBatchExchangeOperatorParam(SOperatorParam** ppRes, int32_t d
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_EXCHANGE;
(*ppRes)->downstreamIdx = downstreamIdx;
(*ppRes)->value = pExc;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
static int32_t buildExchangeOperatorParamForVScan(SOperatorParam** ppRes, int32_t downstreamIdx, SOrgTbInfo* pMap) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SExchangeOperatorParam* pExc = NULL;
SExchangeOperatorBasicParam* basic = NULL;
*ppRes = taosMemoryMalloc(sizeof(SOperatorParam));
QUERY_CHECK_NULL(*ppRes, code, lino, _return, terrno);
(*ppRes)->pChildren = NULL;
pExc = taosMemoryMalloc(sizeof(SExchangeOperatorParam));
QUERY_CHECK_NULL(pExc, code, lino, _return, terrno);
pExc->multiParams = false;
basic = &pExc->basic;
basic->srcOpType = QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN;
basic->vgId = pMap->vgId;
basic->tableSeq = false;
basic->isVtbRefScan = true;
basic->colMap = taosMemoryMalloc(sizeof(SOrgTbInfo));
QUERY_CHECK_NULL(basic->colMap, code, lino, _return, terrno);
basic->colMap->vgId = pMap->vgId;
tstrncpy(basic->colMap->tbName, pMap->tbName, TSDB_TABLE_FNAME_LEN);
basic->colMap->colMap = taosArrayDup(pMap->colMap, NULL);
QUERY_CHECK_NULL(basic->colMap->colMap, code, lino, _return, terrno);
basic->uidList = taosArrayInit(1, sizeof(int64_t));
QUERY_CHECK_NULL(basic->uidList, code, lino, _return, terrno);
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_EXCHANGE;
(*ppRes)->downstreamIdx = downstreamIdx;
(*ppRes)->value = pExc;
(*ppRes)->reUse = true;
return TSDB_CODE_SUCCESS;
_return:
qError("failed to build exchange operator param for vscan, code:%d", code);
taosMemoryFreeClear(*ppRes);
if (basic) {
if (basic->colMap) {
taosArrayDestroy(basic->colMap->colMap);
taosMemoryFreeClear(basic->colMap);
}
if (basic->uidList) {
taosArrayDestroy(basic->uidList);
}
taosMemoryFreeClear(basic);
}
taosMemoryFreeClear(pExc);
return code;
}
static int32_t buildMergeJoinOperatorParam(SOperatorParam** ppRes, bool initParam, SOperatorParam** ppChild0, SOperatorParam** ppChild1) {
int32_t code = TSDB_CODE_SUCCESS;
@ -355,11 +448,11 @@ static int32_t buildMergeJoinOperatorParam(SOperatorParam** ppRes, bool initPara
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN;
(*ppRes)->value = pJoin;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
static int32_t buildMergeJoinNotifyOperatorParam(SOperatorParam** ppRes, SOperatorParam* pChild0, SOperatorParam* pChild1) {
int32_t code = TSDB_CODE_SUCCESS;
*ppRes = taosMemoryMalloc(sizeof(SOperatorParam));
@ -395,12 +488,11 @@ static int32_t buildMergeJoinNotifyOperatorParam(SOperatorParam** ppRes, SOperat
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_MERGE_JOIN;
(*ppRes)->value = NULL;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
static int32_t buildBatchTableScanOperatorParam(SOperatorParam** ppRes, int32_t downstreamIdx, SSHashObj* pVg) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t vgNum = tSimpleHashGetSize(pVg);
@ -516,7 +608,6 @@ static int32_t buildSeqStbJoinOperatorParam(SDynQueryCtrlOperatorInfo* pInfo, SS
return code;
}
static void seqJoinLaunchNewRetrieveImpl(SOperatorInfo* pOperator, SSDataBlock** ppRes) {
SDynQueryCtrlOperatorInfo* pInfo = pOperator->info;
SStbJoinDynCtrlInfo* pStbJoin = (SStbJoinDynCtrlInfo*)&pInfo->stbJoin;
@ -918,8 +1009,11 @@ static int32_t seqStableJoinComposeRes(SStbJoinDynCtrlInfo* pStbJoin, SSDataBloc
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
SColumnInfoData colInfo = createColumnInfoData(pSlot->dataType.type, pSlot->dataType.bytes, pSlot->slotId);
colInfoDataEnsureCapacity(&colInfo, pBlock->info.rows, true);
int32_t code = blockDataAppendColInfo(pBlock, &colInfo);
int32_t code = colInfoDataEnsureCapacity(&colInfo, pBlock->info.rows, true);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
code = blockDataAppendColInfo(pBlock, &colInfo);
if (code != TSDB_CODE_SUCCESS) {
return code;
}
@ -964,7 +1058,7 @@ int32_t seqStableJoin(SOperatorInfo* pOperator, SSDataBlock** pRes) {
_return:
if (pOperator->cost.openCost == 0) {
pOperator->cost.openCost = (taosGetTimestampUs() - st) / 1000.0;
pOperator->cost.openCost = (double)(taosGetTimestampUs() - st) / 1000.0;
}
if (code) {
@ -977,6 +1071,354 @@ _return:
return code;
}
static int32_t buildVtbScanOperatorParam(SDynQueryCtrlOperatorInfo* pInfo, SOperatorParam** ppRes) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SVTableScanOperatorParam* pVScan = NULL;
*ppRes = taosMemoryMalloc(sizeof(SOperatorParam));
QUERY_CHECK_NULL(*ppRes, code, lino, _return, terrno);
(*ppRes)->pChildren = taosArrayInit(1, POINTER_BYTES);
QUERY_CHECK_NULL((*ppRes)->pChildren, code, lino, _return, terrno);
pVScan = taosMemoryMalloc(sizeof(SVTableScanOperatorParam));
QUERY_CHECK_NULL(pVScan, code, lino, _return, terrno);
pVScan->pOpParamArray = taosArrayInit(1, POINTER_BYTES);
QUERY_CHECK_NULL(pVScan->pOpParamArray, code, lino, _return, terrno);
(*ppRes)->opType = QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN;
(*ppRes)->downstreamIdx = 0;
(*ppRes)->value = pVScan;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
_return:
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
if (pVScan) {
taosArrayDestroy(pVScan->pOpParamArray);
taosMemoryFreeClear(pVScan);
}
if (*ppRes) {
taosArrayDestroy((*ppRes)->pChildren);
taosMemoryFreeClear(*ppRes);
}
return code;
}
int32_t dynProcessUseDbRsp(void* param, SDataBuf* pMsg, int32_t code) {
int32_t lino = 0;
SOperatorInfo* operator=(SOperatorInfo*) param;
SDynQueryCtrlOperatorInfo* pScanResInfo = (SDynQueryCtrlOperatorInfo*)operator->info;
if (TSDB_CODE_SUCCESS != code) {
operator->pTaskInfo->code = rpcCvtErrCode(code);
if (operator->pTaskInfo->code != code) {
qError("load systable rsp received, error:%s, cvted error:%s", tstrerror(code),
tstrerror(operator->pTaskInfo->code));
} else {
qError("load systable rsp received, error:%s", tstrerror(code));
}
goto _return;
}
pScanResInfo->vtbScan.pRsp = taosMemoryMalloc(sizeof(SUseDbRsp));
QUERY_CHECK_NULL(pScanResInfo->vtbScan.pRsp, code, lino, _return, terrno);
QUERY_CHECK_CODE(tDeserializeSUseDbRsp(pMsg->pData, (int32_t)pMsg->len, pScanResInfo->vtbScan.pRsp), lino, _return);
taosMemoryFreeClear(pMsg->pData);
QUERY_CHECK_CODE(tsem_post(&pScanResInfo->vtbScan.ready), lino, _return);
return TSDB_CODE_SUCCESS;
_return:
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
return code;
}
static int32_t buildDbVgInfoMap(SOperatorInfo* pOperator, SReadHandle* pHandle, SName* name, SExecTaskInfo* pTaskInfo, SUseDbOutput* output) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
char* buf1 = NULL;
SUseDbReq* pReq = NULL;
SDynQueryCtrlOperatorInfo* pScanResInfo = (SDynQueryCtrlOperatorInfo*)pOperator->info;
pReq = taosMemoryMalloc(sizeof(SUseDbReq));
QUERY_CHECK_NULL(pReq, code, lino, _return, terrno);
QUERY_CHECK_CODE(tNameGetFullDbName(name, pReq->db), lino, _return);
int32_t contLen = tSerializeSUseDbReq(NULL, 0, pReq);
buf1 = taosMemoryCalloc(1, contLen);
QUERY_CHECK_NULL(buf1, code, lino, _return, terrno);
int32_t tempRes = tSerializeSUseDbReq(buf1, contLen, pReq);
if (tempRes < 0) {
QUERY_CHECK_CODE(terrno, lino, _return);
}
// send the fetch remote task result request
SMsgSendInfo* pMsgSendInfo = taosMemoryCalloc(1, sizeof(SMsgSendInfo));
QUERY_CHECK_NULL(pMsgSendInfo, code, lino, _return, terrno);
pMsgSendInfo->param = pOperator;
pMsgSendInfo->msgInfo.pData = buf1;
pMsgSendInfo->msgInfo.len = contLen;
pMsgSendInfo->msgType = TDMT_MND_GET_DB_INFO;
pMsgSendInfo->fp = dynProcessUseDbRsp;
pMsgSendInfo->requestId = pTaskInfo->id.queryId;
QUERY_CHECK_CODE(asyncSendMsgToServer(pHandle->pMsgCb->clientRpc, &pScanResInfo->vtbScan.epSet, NULL, pMsgSendInfo), lino, _return);
QUERY_CHECK_CODE(tsem_wait(&pScanResInfo->vtbScan.ready), lino, _return);
QUERY_CHECK_CODE(queryBuildUseDbOutput(output, pScanResInfo->vtbScan.pRsp), lino, _return);
_return:
if (code) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
taosMemoryFree(buf1);
}
taosMemoryFree(pReq);
tFreeSUsedbRsp(pScanResInfo->vtbScan.pRsp);
taosMemoryFreeClear(pScanResInfo->vtbScan.pRsp);
return code;
}
int dynVgInfoComp(const void* lp, const void* rp) {
SVgroupInfo* pLeft = (SVgroupInfo*)lp;
SVgroupInfo* pRight = (SVgroupInfo*)rp;
if (pLeft->hashBegin < pRight->hashBegin) {
return -1;
} else if (pLeft->hashBegin > pRight->hashBegin) {
return 1;
}
return 0;
}
int32_t dynMakeVgArraySortBy(SDBVgInfo* dbInfo, __compar_fn_t sort_func) {
if (NULL == dbInfo) {
return TSDB_CODE_SUCCESS;
}
if (dbInfo->vgHash && NULL == dbInfo->vgArray) {
int32_t vgSize = taosHashGetSize(dbInfo->vgHash);
dbInfo->vgArray = taosArrayInit(vgSize, sizeof(SVgroupInfo));
if (NULL == dbInfo->vgArray) {
return terrno;
}
void* pIter = taosHashIterate(dbInfo->vgHash, NULL);
while (pIter) {
if (NULL == taosArrayPush(dbInfo->vgArray, pIter)) {
taosHashCancelIterate(dbInfo->vgHash, pIter);
return terrno;
}
pIter = taosHashIterate(dbInfo->vgHash, pIter);
}
taosArraySort(dbInfo->vgArray, sort_func);
}
return TSDB_CODE_SUCCESS;
}
int32_t dynHashValueComp(void const* lp, void const* rp) {
uint32_t* key = (uint32_t*)lp;
SVgroupInfo* pVg = (SVgroupInfo*)rp;
if (*key < pVg->hashBegin) {
return -1;
} else if (*key > pVg->hashEnd) {
return 1;
}
return 0;
}
int32_t getVgId(SDBVgInfo* dbInfo, char* dbFName, int32_t* vgId, char *tbName) {
int32_t code = 0;
int32_t lino = 0;
QUERY_CHECK_CODE(dynMakeVgArraySortBy(dbInfo, dynVgInfoComp), lino, _return);
int32_t vgNum = (int32_t)taosArrayGetSize(dbInfo->vgArray);
if (vgNum <= 0) {
qError("db vgroup cache invalid, db:%s, vgroup number:%d", dbFName, vgNum);
QUERY_CHECK_CODE(TSDB_CODE_TSC_DB_NOT_SELECTED, lino, _return);
}
SVgroupInfo* vgInfo = NULL;
char tbFullName[TSDB_TABLE_FNAME_LEN];
(void)snprintf(tbFullName, sizeof(tbFullName), "%s.", dbFName);
int32_t offset = (int32_t)strlen(tbFullName);
(void)snprintf(tbFullName + offset, sizeof(tbFullName) - offset, "%s", tbName);
uint32_t hashValue = taosGetTbHashVal(tbFullName, (int32_t)strlen(tbFullName), dbInfo->hashMethod,
dbInfo->hashPrefix, dbInfo->hashSuffix);
vgInfo = taosArraySearch(dbInfo->vgArray, &hashValue, dynHashValueComp, TD_EQ);
if (NULL == vgInfo) {
qError("no hash range found for hash value [%u], db:%s, numOfVgId:%d", hashValue, dbFName,
(int32_t)taosArrayGetSize(dbInfo->vgArray));
return TSDB_CODE_CTG_INTERNAL_ERROR;
}
*vgId = vgInfo->vgId;
_return:
return code;
}
bool colNeedScan(SOperatorInfo* pOperator, col_id_t colId) {
SDynQueryCtrlOperatorInfo* pInfo = pOperator->info;
SVtbScanDynCtrlInfo* pVtbScan = (SVtbScanDynCtrlInfo*)&pInfo->vtbScan;
SArray * pColList = pVtbScan->readColList;
if (pVtbScan->scanAllCols) {
return true;
}
for (int32_t i = 0; i < taosArrayGetSize(pColList); i++) {
if (colId == *(col_id_t*)taosArrayGet(pColList, i)) {
return true;
}
}
return false;
}
void destroyOrgTbInfo(void *info) {
SOrgTbInfo *pOrgTbInfo = (SOrgTbInfo *)info;
if (pOrgTbInfo) {
taosArrayDestroy(pOrgTbInfo->colMap);
}
}
int32_t vtbScan(SOperatorInfo* pOperator, SSDataBlock** pRes) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t line = 0;
SDynQueryCtrlOperatorInfo* pInfo = pOperator->info;
SVtbScanDynCtrlInfo* pVtbScan = (SVtbScanDynCtrlInfo*)&pInfo->vtbScan;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SReadHandle* pHandle = &pVtbScan->readHandle;
SMetaReader mr = {0};
SHashObj* orgTbVgColMap = NULL;
QRY_PARAM_CHECK(pRes);
if (pOperator->status == OP_EXEC_DONE) {
return code;
}
int64_t st = 0;
if (pOperator->cost.openCost == 0) {
st = taosGetTimestampUs();
}
size_t num = taosArrayGetSize(pVtbScan->childTableList);
if (num == 0) {
setOperatorCompleted(pOperator);
return code;
}
// TODO(smj) : proper hash size
orgTbVgColMap = taosHashInit(num * 64, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK);
QUERY_CHECK_NULL(orgTbVgColMap, code, line, _return, terrno);
taosHashSetFreeFp(orgTbVgColMap, destroyOrgTbInfo);
while (true) {
if (pVtbScan->readTableIdx == pVtbScan->lastTableIdx) {
QUERY_CHECK_CODE(pOperator->pDownstream[0]->fpSet.getNextFn(pOperator->pDownstream[0], pRes), line, _return);
} else {
uint64_t* id = taosArrayGet(pVtbScan->childTableList, pVtbScan->readTableIdx);
QUERY_CHECK_NULL(id, code, line, _return, terrno);
pHandle->api.metaReaderFn.initReader(&mr, pHandle->vnode, META_READER_LOCK, &pHandle->api.metaFn);
QUERY_CHECK_CODE(pHandle->api.metaReaderFn.getTableEntryByUid(&mr, *id), line, _return);
for (int32_t j = 0; j < mr.me.colRef.nCols; j++) {
if (mr.me.colRef.pColRef[j].hasRef && colNeedScan(pOperator, mr.me.colRef.pColRef[j].id)) {
SName name = {0};
toName(pInfo->vtbScan.acctId, mr.me.colRef.pColRef[j].refDbName, "", &name);
SUseDbOutput* output = NULL;
SUseDbOutput** find = (SUseDbOutput**)taosHashGet(pInfo->vtbScan.dbVgInfoMap, name.dbname, strlen(name.dbname));
if (find == NULL) {
output = taosMemoryMalloc(sizeof(SUseDbOutput));
QUERY_CHECK_CODE(buildDbVgInfoMap(pOperator, pHandle, &name, pTaskInfo, output), line, _return);
QUERY_CHECK_CODE(taosHashPut(pInfo->vtbScan.dbVgInfoMap, name.dbname, strlen(name.dbname), &output, sizeof(output)), line, _return);
} else {
output = *find;
}
int32_t vgId = 0;
char dbFname[TSDB_DB_FNAME_LEN] = {0};
QUERY_CHECK_CODE(tNameGetFullDbName(&name, dbFname), line, _return);
QUERY_CHECK_CODE(getVgId(output->dbVgroup, dbFname, &vgId, mr.me.colRef.pColRef[j].refTableName), line, _return);
char orgTbFName[TSDB_TABLE_FNAME_LEN] = {0};
TAOS_STRNCAT(orgTbFName, mr.me.colRef.pColRef[j].refDbName, TSDB_DB_NAME_LEN);
TAOS_STRNCAT(orgTbFName, ".", 2);
TAOS_STRNCAT(orgTbFName, mr.me.colRef.pColRef[j].refTableName, TSDB_TABLE_NAME_LEN);
void *tbVgCol = taosHashGet(orgTbVgColMap, orgTbFName, sizeof(orgTbFName));
if (!tbVgCol) {
SOrgTbInfo map = {0};
map.vgId = vgId;
tstrncpy(map.tbName, orgTbFName, sizeof(map.tbName));
map.colMap = taosArrayInit(10, sizeof(SColIdNameKV));
QUERY_CHECK_NULL(map.colMap, code, line, _return, terrno);
SColIdNameKV colIdNameKV = {0};
colIdNameKV.colId = mr.me.colRef.pColRef[j].id;
tstrncpy(colIdNameKV.colName, mr.me.colRef.pColRef[j].refColName, sizeof(colIdNameKV.colName));
QUERY_CHECK_NULL(taosArrayPush(map.colMap, &colIdNameKV), code, line, _return, terrno);
QUERY_CHECK_CODE(taosHashPut(orgTbVgColMap, orgTbFName, sizeof(orgTbFName), &map, sizeof(map)), line, _return);
} else {
SOrgTbInfo *map = (SOrgTbInfo *)tbVgCol;
SColIdNameKV colIdNameKV = {0};
colIdNameKV.colId = mr.me.colRef.pColRef[j].id;
tstrncpy(colIdNameKV.colName, mr.me.colRef.pColRef[j].refColName, sizeof(colIdNameKV.colName));
QUERY_CHECK_NULL(taosArrayPush(map->colMap, &colIdNameKV), code, line, _return, terrno);
}
}
}
pVtbScan->vtbScanParam = NULL;
QUERY_CHECK_CODE(buildVtbScanOperatorParam(pInfo, &pVtbScan->vtbScanParam), line, _return);
((SVTableScanOperatorParam*)pVtbScan->vtbScanParam->value)->uid = *id;
void* pIter = taosHashIterate(orgTbVgColMap, NULL);
while (pIter != NULL) {
SOrgTbInfo* pMap = (SOrgTbInfo*)pIter;
SOperatorParam* pExchangeParam = NULL;
QUERY_CHECK_CODE(buildExchangeOperatorParamForVScan(&pExchangeParam, 0, pMap), line, _return);
QUERY_CHECK_NULL(taosArrayPush(((SVTableScanOperatorParam*)pVtbScan->vtbScanParam->value)->pOpParamArray, &pExchangeParam), code, line, _return, terrno);
pIter = taosHashIterate(orgTbVgColMap, pIter);
}
pHandle->api.metaReaderFn.clearReader(&mr);
pOperator->pDownstream[0]->status = OP_NOT_OPENED;
QUERY_CHECK_CODE(pOperator->pDownstream[0]->fpSet.getNextExtFn(pOperator->pDownstream[0], pVtbScan->vtbScanParam, pRes), line, _return);
}
if (*pRes) {
pVtbScan->lastTableIdx = pVtbScan->readTableIdx;
break;
} else {
pVtbScan->readTableIdx++;
if (pVtbScan->readTableIdx >= taosArrayGetSize(pVtbScan->childTableList)) {
setOperatorCompleted(pOperator);
break;
}
}
}
_return:
taosHashCleanup(orgTbVgColMap);
if (pOperator->cost.openCost == 0) {
pOperator->cost.openCost = (double)(taosGetTimestampUs() - st) / 1000.0;
}
if (code) {
qError("%s failed since %s", __func__, tstrerror(code));
pOperator->pTaskInfo->code = code;
T_LONG_JMP(pOperator->pTaskInfo->env, code);
}
return code;
}
int32_t initSeqStbJoinTableHash(SStbJoinPrevJoinCtx* pPrev, bool batchFetch) {
if (batchFetch) {
pPrev->leftHash = tSimpleHashInit(20, taosGetDefaultHashFunction(TSDB_DATA_TYPE_INT));
@ -1005,9 +1447,43 @@ int32_t initSeqStbJoinTableHash(SStbJoinPrevJoinCtx* pPrev, bool batchFetch) {
return TSDB_CODE_SUCCESS;
}
static int32_t initVtbScanInfo(SOperatorInfo* pOperator, SDynQueryCtrlOperatorInfo* pInfo, SReadHandle* pHandle,
SDynQueryCtrlPhysiNode* pPhyciNode, SExecTaskInfo* pTaskInfo) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t line = 0;
QUERY_CHECK_CODE(tsem_init(&pInfo->vtbScan.ready, 0, 0), line, _return);
pInfo->vtbScan.scanAllCols = pPhyciNode->vtbScan.scanAllCols;
pInfo->vtbScan.suid = pPhyciNode->vtbScan.suid;
pInfo->vtbScan.epSet = pPhyciNode->vtbScan.mgmtEpSet;
pInfo->vtbScan.acctId = pPhyciNode->vtbScan.accountId;
pInfo->vtbScan.readHandle = *pHandle;
pInfo->vtbScan.readTableIdx = 0;
pInfo->vtbScan.lastTableIdx = -1;
pInfo->vtbScan.readColList = taosArrayInit(LIST_LENGTH(pPhyciNode->vtbScan.pScanCols), sizeof(col_id_t));
QUERY_CHECK_NULL(pInfo->vtbScan.readColList, code, line, _return, terrno);
for (int32_t i = 0; i < LIST_LENGTH(pPhyciNode->vtbScan.pScanCols); ++i) {
SColumnNode* pNode = (SColumnNode*)nodesListGetNode(pPhyciNode->vtbScan.pScanCols, i);
QUERY_CHECK_NULL(pNode, code, line, _return, terrno);
QUERY_CHECK_NULL(taosArrayPush(pInfo->vtbScan.readColList, &pNode->colId), code, line, _return, terrno);
}
pInfo->vtbScan.childTableList = taosArrayInit(10, sizeof(uint64_t));
QUERY_CHECK_CODE(pHandle->api.metaFn.getChildTableList(pHandle->vnode, pInfo->vtbScan.suid, pInfo->vtbScan.childTableList), line, _return);
pInfo->vtbScan.dbVgInfoMap = taosHashInit(taosArrayGetSize(pInfo->vtbScan.childTableList), taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY), false, HASH_ENTRY_LOCK);
QUERY_CHECK_NULL(pInfo->vtbScan.dbVgInfoMap, code, line, _return, terrno);
_return:
return code;
}
int32_t createDynQueryCtrlOperatorInfo(SOperatorInfo** pDownstream, int32_t numOfDownstream,
SDynQueryCtrlPhysiNode* pPhyciNode, SExecTaskInfo* pTaskInfo,
SOperatorInfo** pOptrInfo) {
SReadHandle* pHandle, SOperatorInfo** pOptrInfo) {
QRY_PARAM_CHECK(pOptrInfo);
int32_t code = TSDB_CODE_SUCCESS;
@ -1025,13 +1501,16 @@ int32_t createDynQueryCtrlOperatorInfo(SOperatorInfo** pDownstream, int32_t numO
goto _error;
}
pTaskInfo->dynamicTask = pPhyciNode->node.dynamicOp;
pTaskInfo->dynamicTask = (int8_t)pPhyciNode->node.dynamicOp;
code = appendDownstream(pOperator, pDownstream, numOfDownstream);
if (TSDB_CODE_SUCCESS != code) {
goto _error;
}
setOperatorInfo(pOperator, "DynQueryCtrlOperator", QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL, false, OP_NOT_OPENED,
pInfo, pTaskInfo);
pInfo->qType = pPhyciNode->qType;
switch (pInfo->qType) {
case DYN_QTYPE_STB_HASH:
@ -1043,15 +1522,16 @@ int32_t createDynQueryCtrlOperatorInfo(SOperatorInfo** pDownstream, int32_t numO
}
nextFp = seqStableJoin;
break;
case DYN_QTYPE_VTB_SCAN:
QUERY_CHECK_CODE(initVtbScanInfo(pOperator, pInfo, pHandle, pPhyciNode, pTaskInfo), code, _error);
nextFp = vtbScan;
break;
default:
qError("unsupported dynamic query ctrl type: %d", pInfo->qType);
code = TSDB_CODE_INVALID_PARA;
goto _error;
}
setOperatorInfo(pOperator, "DynQueryCtrlOperator", QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL, false, OP_NOT_OPENED,
pInfo, pTaskInfo);
pOperator->fpSet = createOperatorFpSet(optrDummyOpenFn, nextFp, NULL, destroyDynQueryCtrlOperator, optrDefaultBufFn,
NULL, optrDefaultGetNextExtFn, NULL);

186
source/libs/executor/src/exchangeoperator.c
View File

@ -43,6 +43,9 @@ typedef struct SSourceDataInfo {
bool tableSeq;
char* decompBuf;
int32_t decompBufSize;
SOrgTbInfo* colMap;
bool isVtbRefScan;
STimeWindow window;
} SSourceDataInfo;
static void destroyExchangeOperatorInfo(void* param);
@ -112,7 +115,7 @@ static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeIn
// todo
SLoadRemoteDataInfo* pLoadInfo = &pExchangeInfo->loadInfo;
if (pRsp->numOfRows == 0) {
if (NULL != pDataInfo->pSrcUidList) {
if (NULL != pDataInfo->pSrcUidList && (!pDataInfo->isVtbRefScan)) {
pDataInfo->status = EX_SOURCE_DATA_NOT_READY;
code = doSendFetchDataRequest(pExchangeInfo, pTaskInfo, i);
if (code != TSDB_CODE_SUCCESS) {
@ -156,7 +159,7 @@ static void concurrentlyLoadRemoteDataImpl(SOperatorInfo* pOperator, SExchangeIn
taosMemoryFreeClear(pDataInfo->pRsp);
if (pDataInfo->status != EX_SOURCE_DATA_EXHAUSTED || NULL != pDataInfo->pSrcUidList) {
if ((pDataInfo->status != EX_SOURCE_DATA_EXHAUSTED || NULL != pDataInfo->pSrcUidList) && !pDataInfo->isVtbRefScan) {
pDataInfo->status = EX_SOURCE_DATA_NOT_READY;
code = doSendFetchDataRequest(pExchangeInfo, pTaskInfo, i);
if (code != TSDB_CODE_SUCCESS) {
@ -603,15 +606,67 @@ int32_t buildTableScanOperatorParam(SOperatorParam** ppRes, SArray* pUidList, in
return terrno;
}
pScan->tableSeq = tableSeq;
pScan->pOrgTbInfo = NULL;
pScan->window.skey = INT64_MAX;
pScan->window.ekey = INT64_MIN;
(*ppRes)->opType = srcOpType;
(*ppRes)->downstreamIdx = 0;
(*ppRes)->value = pScan;
(*ppRes)->pChildren = NULL;
(*ppRes)->reUse = false;
return TSDB_CODE_SUCCESS;
}
int32_t buildTableScanOperatorParamEx(SOperatorParam** ppRes, SArray* pUidList, int32_t srcOpType, SOrgTbInfo *pMap, bool tableSeq, STimeWindow *window) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
STableScanOperatorParam* pScan = NULL;
*ppRes = taosMemoryMalloc(sizeof(SOperatorParam));
QUERY_CHECK_NULL(*ppRes, code, lino, _return, terrno);
pScan = taosMemoryMalloc(sizeof(STableScanOperatorParam));
QUERY_CHECK_NULL(pScan, code, lino, _return, terrno);
pScan->pUidList = taosArrayDup(pUidList, NULL);
QUERY_CHECK_NULL(pScan->pUidList, code, lino, _return, terrno);
pScan->pOrgTbInfo = taosMemoryMalloc(sizeof(SOrgTbInfo));
QUERY_CHECK_NULL(pScan->pOrgTbInfo, code, lino, _return, terrno);
pScan->pOrgTbInfo->vgId = pMap->vgId;
tstrncpy(pScan->pOrgTbInfo->tbName, pMap->tbName, TSDB_TABLE_FNAME_LEN);
pScan->pOrgTbInfo->colMap = taosArrayDup(pMap->colMap, NULL);
QUERY_CHECK_NULL(pScan->pOrgTbInfo->colMap, code, lino, _return, terrno);
pScan->tableSeq = tableSeq;
pScan->window.skey = window->skey;
pScan->window.ekey = window->ekey;
(*ppRes)->opType = srcOpType;
(*ppRes)->downstreamIdx = 0;
(*ppRes)->value = pScan;
(*ppRes)->pChildren = NULL;
(*ppRes)->reUse = false;
return code;
_return:
qError("%s failed at %d, failed to build scan operator msg:%s", __FUNCTION__, lino, tstrerror(code));
taosMemoryFreeClear(*ppRes);
if (pScan) {
taosArrayDestroy(pScan->pUidList);
if (pScan->pOrgTbInfo) {
taosArrayDestroy(pScan->pOrgTbInfo->colMap);
taosMemoryFreeClear(pScan->pOrgTbInfo);
}
taosMemoryFree(pScan);
}
return code;
}
int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInfo* pTaskInfo, int32_t sourceIndex) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
@ -654,9 +709,10 @@ int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInfo* pTas
req.taskId = pSource->taskId;
req.queryId = pTaskInfo->id.queryId;
req.execId = pSource->execId;
if (pDataInfo->pSrcUidList) {
int32_t code =
buildTableScanOperatorParam(&req.pOpParam, pDataInfo->pSrcUidList, pDataInfo->srcOpType, pDataInfo->tableSeq);
if (pDataInfo->isVtbRefScan) {
code = buildTableScanOperatorParamEx(&req.pOpParam, pDataInfo->pSrcUidList, pDataInfo->srcOpType, pDataInfo->colMap, pDataInfo->tableSeq, &pDataInfo->window);
taosArrayDestroy(pDataInfo->colMap->colMap);
taosMemoryFreeClear(pDataInfo->colMap);
taosArrayDestroy(pDataInfo->pSrcUidList);
pDataInfo->pSrcUidList = NULL;
if (TSDB_CODE_SUCCESS != code) {
@ -664,6 +720,17 @@ int32_t doSendFetchDataRequest(SExchangeInfo* pExchangeInfo, SExecTaskInfo* pTas
taosMemoryFree(pWrapper);
return pTaskInfo->code;
}
} else {
if (pDataInfo->pSrcUidList) {
code = buildTableScanOperatorParam(&req.pOpParam, pDataInfo->pSrcUidList, pDataInfo->srcOpType, pDataInfo->tableSeq);
taosArrayDestroy(pDataInfo->pSrcUidList);
pDataInfo->pSrcUidList = NULL;
if (TSDB_CODE_SUCCESS != code) {
pTaskInfo->code = code;
taosMemoryFree(pWrapper);
return pTaskInfo->code;
}
}
}
int32_t msgSize = tSerializeSResFetchReq(NULL, 0, &req);
@ -995,7 +1062,11 @@ int32_t seqLoadRemoteData(SOperatorInfo* pOperator) {
pExchangeInfo->current + 1, pDataInfo->totalRows, pLoadInfo->totalRows);
pDataInfo->status = EX_SOURCE_DATA_EXHAUSTED;
pExchangeInfo->current += 1;
if (pDataInfo->isVtbRefScan) {
pExchangeInfo->current = totalSources;
} else {
pExchangeInfo->current += 1;
}
taosMemoryFreeClear(pDataInfo->pRsp);
continue;
}
@ -1014,14 +1085,20 @@ int32_t seqLoadRemoteData(SOperatorInfo* pOperator) {
pExchangeInfo->current + 1, totalSources);
pDataInfo->status = EX_SOURCE_DATA_EXHAUSTED;
pExchangeInfo->current += 1;
if (pDataInfo->isVtbRefScan) {
pExchangeInfo->current = totalSources;
} else {
pExchangeInfo->current += 1;
}
} else {
qDebug("%s fetch msg rsp from vgId:%d, clientId:0x%" PRIx64 " taskId:0x%" PRIx64 " execId:%d numOfRows:%" PRId64
", totalRows:%" PRIu64 ", totalBytes:%" PRIu64,
GET_TASKID(pTaskInfo), pSource->addr.nodeId, pSource->clientId, pSource->taskId, pSource->execId,
pRetrieveRsp->numOfRows, pLoadInfo->totalRows, pLoadInfo->totalSize);
}
if (pExchangeInfo->dynamicOp && pExchangeInfo->seqLoadData) {
taosArrayClear(pExchangeInfo->pSourceDataInfo);
}
updateLoadRemoteInfo(pLoadInfo, pRetrieveRsp->numOfRows, pRetrieveRsp->compLen, startTs, pOperator);
pDataInfo->totalRows += pRetrieveRsp->numOfRows;
@ -1034,6 +1111,15 @@ _error:
return code;
}
void clearVtbScanDataInfo(void* pItem) {
SSourceDataInfo *pInfo = (SSourceDataInfo *)pItem;
if (pInfo->colMap) {
taosArrayDestroy(pInfo->colMap->colMap);
taosMemoryFreeClear(pInfo->colMap);
}
taosArrayDestroy(pInfo->pSrcUidList);
}
int32_t addSingleExchangeSource(SOperatorInfo* pOperator, SExchangeOperatorBasicParam* pBasicParam) {
SExchangeInfo* pExchangeInfo = pOperator->info;
SExchangeSrcIndex* pIdx = tSimpleHashGet(pExchangeInfo->pHashSources, &pBasicParam->vgId, sizeof(pBasicParam->vgId));
@ -1042,40 +1128,90 @@ int32_t addSingleExchangeSource(SOperatorInfo* pOperator, SExchangeOperatorBasic
return TSDB_CODE_INVALID_PARA;
}
if (pIdx->inUseIdx < 0) {
if (pBasicParam->isVtbRefScan) {
SSourceDataInfo dataInfo = {0};
dataInfo.status = EX_SOURCE_DATA_NOT_READY;
dataInfo.taskId = pExchangeInfo->pTaskId;
dataInfo.index = pIdx->srcIdx;
dataInfo.window = pBasicParam->window;
dataInfo.colMap = taosMemoryMalloc(sizeof(SOrgTbInfo));
dataInfo.colMap->vgId = pBasicParam->colMap->vgId;
tstrncpy(dataInfo.colMap->tbName, pBasicParam->colMap->tbName, TSDB_TABLE_FNAME_LEN);
dataInfo.colMap->colMap = taosArrayDup(pBasicParam->colMap->colMap, NULL);
dataInfo.pSrcUidList = taosArrayDup(pBasicParam->uidList, NULL);
if (dataInfo.pSrcUidList == NULL) {
return terrno;
}
dataInfo.isVtbRefScan = pBasicParam->isVtbRefScan;
dataInfo.srcOpType = pBasicParam->srcOpType;
dataInfo.tableSeq = pBasicParam->tableSeq;
taosArrayClearEx(pExchangeInfo->pSourceDataInfo, clearVtbScanDataInfo);
void* tmp = taosArrayPush(pExchangeInfo->pSourceDataInfo, &dataInfo);
if (!tmp) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno));
return terrno;
}
pIdx->inUseIdx = taosArrayGetSize(pExchangeInfo->pSourceDataInfo) - 1;
} else {
SSourceDataInfo* pDataInfo = taosArrayGet(pExchangeInfo->pSourceDataInfo, pIdx->inUseIdx);
if (!pDataInfo) {
return terrno;
}
if (pDataInfo->status == EX_SOURCE_DATA_EXHAUSTED) {
pDataInfo->status = EX_SOURCE_DATA_NOT_READY;
}
pDataInfo->pSrcUidList = taosArrayDup(pBasicParam->uidList, NULL);
if (pDataInfo->pSrcUidList == NULL) {
return terrno;
}
if (pIdx->inUseIdx < 0) {
SSourceDataInfo dataInfo = {0};
dataInfo.status = EX_SOURCE_DATA_NOT_READY;
dataInfo.taskId = pExchangeInfo->pTaskId;
dataInfo.index = pIdx->srcIdx;
if (pBasicParam->isVtbRefScan) {
dataInfo.window = pBasicParam->window;
dataInfo.colMap = taosMemoryMalloc(sizeof(SOrgTbInfo));
dataInfo.colMap->vgId = pBasicParam->colMap->vgId;
tstrncpy(dataInfo.colMap->tbName, pBasicParam->colMap->tbName, TSDB_TABLE_FNAME_LEN);
dataInfo.colMap->colMap = taosArrayDup(pBasicParam->colMap->colMap, NULL);
}
pDataInfo->srcOpType = pBasicParam->srcOpType;
pDataInfo->tableSeq = pBasicParam->tableSeq;
dataInfo.pSrcUidList = taosArrayDup(pBasicParam->uidList, NULL);
if (dataInfo.pSrcUidList == NULL) {
return terrno;
}
dataInfo.isVtbRefScan = pBasicParam->isVtbRefScan;
dataInfo.srcOpType = pBasicParam->srcOpType;
dataInfo.tableSeq = pBasicParam->tableSeq;
void* tmp = taosArrayPush(pExchangeInfo->pSourceDataInfo, &dataInfo);
if (!tmp) {
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno));
return terrno;
}
pIdx->inUseIdx = taosArrayGetSize(pExchangeInfo->pSourceDataInfo) - 1;
} else {
SSourceDataInfo* pDataInfo = taosArrayGet(pExchangeInfo->pSourceDataInfo, pIdx->inUseIdx);
if (!pDataInfo) {
return terrno;
}
if (pDataInfo->status == EX_SOURCE_DATA_EXHAUSTED) {
pDataInfo->status = EX_SOURCE_DATA_NOT_READY;
}
if (pBasicParam->isVtbRefScan) {
pDataInfo->window = pBasicParam->window;
if (!pDataInfo->colMap) {
pDataInfo->colMap = taosMemoryMalloc(sizeof(SOrgTbInfo));
}
pDataInfo->colMap->vgId = pBasicParam->colMap->vgId;
tstrncpy(pDataInfo->colMap->tbName, pBasicParam->colMap->tbName, TSDB_TABLE_FNAME_LEN);
pDataInfo->colMap->colMap = taosArrayDup(pBasicParam->colMap->colMap, NULL);
}
pDataInfo->pSrcUidList = taosArrayDup(pBasicParam->uidList, NULL);
if (pDataInfo->pSrcUidList == NULL) {
return terrno;
}
pDataInfo->isVtbRefScan = pBasicParam->isVtbRefScan;
pDataInfo->srcOpType = pBasicParam->srcOpType;
pDataInfo->tableSeq = pBasicParam->tableSeq;
}
}
return TSDB_CODE_SUCCESS;
@ -1120,6 +1256,10 @@ int32_t prepareLoadRemoteData(SOperatorInfo* pOperator) {
QUERY_CHECK_CODE(code, lino, _end);
}
if (pOperator->status == OP_NOT_OPENED && pExchangeInfo->dynamicOp && pExchangeInfo->seqLoadData) {
pExchangeInfo->current = 0;
}
int64_t st = taosGetTimestampUs();
if (!pExchangeInfo->seqLoadData) {

54
source/libs/executor/src/executil.c
View File

@ -2403,6 +2403,54 @@ int32_t initQueryTableDataCond(SQueryTableDataCond* pCond, const STableScanPhysi
return TSDB_CODE_SUCCESS;
}
int32_t initQueryTableDataCondWithColArray(SQueryTableDataCond* pCond, SQueryTableDataCond* pOrgCond,
const SReadHandle* readHandle, SArray* colArray) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
pCond->order = TSDB_ORDER_ASC;
pCond->numOfCols = (int32_t)taosArrayGetSize(colArray);
pCond->colList = taosMemoryCalloc(pCond->numOfCols, sizeof(SColumnInfo));
QUERY_CHECK_NULL(pCond->colList, code, lino, _return, terrno);
pCond->pSlotList = taosMemoryMalloc(sizeof(int32_t) * pCond->numOfCols);
QUERY_CHECK_NULL(pCond->pSlotList, code, lino, _return, terrno);
pCond->twindows = pOrgCond->twindows;
pCond->type = pOrgCond->type;
pCond->startVersion = -1;
pCond->endVersion = -1;
pCond->skipRollup = true;
pCond->notLoadData = false;
for (int32_t i = 0; i < pCond->numOfCols; ++i) {
SColIdPair* pColPair = taosArrayGet(colArray, i);
QUERY_CHECK_NULL(pColPair, code, lino, _return, terrno);
bool find = false;
for (int32_t j = 0; j < pOrgCond->numOfCols; ++j) {
if (pOrgCond->colList[j].colId == pColPair->vtbColId) {
pCond->colList[i].type = pOrgCond->colList[j].type;
pCond->colList[i].bytes = pOrgCond->colList[j].bytes;
pCond->colList[i].colId = pColPair->orgColId;
pCond->colList[i].pk = pOrgCond->colList[j].pk;
pCond->pSlotList[i] = i;
find = true;
break;
}
}
QUERY_CHECK_CONDITION(find, code, lino, _return, TSDB_CODE_NOT_FOUND);
}
return code;
_return:
qError("%s failed at line %d since %s", __func__, lino, tstrerror(terrno));
taosMemoryFreeClear(pCond->colList);
taosMemoryFreeClear(pCond->pSlotList);
return code;
}
void cleanupQueryTableDataCond(SQueryTableDataCond* pCond) {
taosMemoryFreeClear(pCond->colList);
taosMemoryFreeClear(pCond->pSlotList);
@ -2995,17 +3043,17 @@ char* getStreamOpName(uint16_t opType) {
void printDataBlock(SSDataBlock* pBlock, const char* flag, const char* taskIdStr) {
if (!pBlock) {
qDebug("%s===stream===%s: Block is Null", taskIdStr, flag);
qInfo("%s===stream===%s: Block is Null", taskIdStr, flag);
return;
} else if (pBlock->info.rows == 0) {
qDebug("%s===stream===%s: Block is Empty. block type %d", taskIdStr, flag, pBlock->info.type);
qInfo("%s===stream===%s: Block is Empty. block type %d", taskIdStr, flag, pBlock->info.type);
return;
}
if (qDebugFlag & DEBUG_DEBUG) {
char* pBuf = NULL;
int32_t code = dumpBlockData(pBlock, flag, &pBuf, taskIdStr);
if (code == 0) {
qDebug("%s", pBuf);
qInfo("%s", pBuf);
taosMemoryFree(pBuf);
}
}

9
source/libs/executor/src/executor.c
View File

@ -141,7 +141,8 @@ static int32_t doSetStreamBlock(SOperatorInfo* pOperator, void* input, size_t nu
const char* id) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (pOperator->operatorType != QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN) {
if (pOperator->operatorType != QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN &&
pOperator->operatorType != QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN) {
if (pOperator->numOfDownstream == 0) {
qError("failed to find stream scan operator to set the input data block, %s" PRIx64, id);
return TSDB_CODE_APP_ERROR;
@ -678,7 +679,7 @@ int32_t qCreateExecTask(SReadHandle* readHandle, int32_t vgId, uint64_t taskId,
}
// pSinkParam has been freed during create sinker.
code = dsCreateDataSinker(pSinkManager, readHandle->localExec ? &pSink : &pSubplan->pDataSink, handle, pSinkParam, (*pTask)->id.str);
code = dsCreateDataSinker(pSinkManager, readHandle->localExec ? &pSink : &pSubplan->pDataSink, handle, pSinkParam, (*pTask)->id.str, pSubplan->processOneBlock);
if (code) {
qError("s-task:%s failed to create data sinker, code:%s", (*pTask)->id.str, tstrerror(code));
}
@ -697,7 +698,7 @@ static void freeBlock(void* param) {
blockDataDestroy(pBlock);
}
int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bool* hasMore, SLocalFetch* pLocal) {
int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bool* hasMore, SLocalFetch* pLocal, bool processOneBlock) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SExecTaskInfo* pTaskInfo = (SExecTaskInfo*)tinfo;
@ -792,7 +793,7 @@ int32_t qExecTaskOpt(qTaskInfo_t tinfo, SArray* pResList, uint64_t* useconds, bo
void* tmp = taosArrayPush(pResList, &p);
QUERY_CHECK_NULL(tmp, code, lino, _end, terrno);
if (current >= rowsThreshold) {
if (current >= rowsThreshold || processOneBlock) {
break;
}

27
source/libs/executor/src/executorInt.c
View File

@ -1202,6 +1202,10 @@ void freeOperatorParamImpl(SOperatorParam* pParam, SOperatorParamType type) {
void freeExchangeGetBasicOperatorParam(void* pParam) {
SExchangeOperatorBasicParam* pBasic = (SExchangeOperatorBasicParam*)pParam;
taosArrayDestroy(pBasic->uidList);
if (pBasic->colMap) {
taosArrayDestroy(pBasic->colMap->colMap);
taosMemoryFreeClear(pBasic->colMap);
}
}
void freeExchangeGetOperatorParam(SOperatorParam* pParam) {
@ -1229,13 +1233,31 @@ void freeMergeJoinNotifyOperatorParam(SOperatorParam* pParam) { freeOperatorPara
void freeTableScanGetOperatorParam(SOperatorParam* pParam) {
STableScanOperatorParam* pTableScanParam = (STableScanOperatorParam*)pParam->value;
taosArrayDestroy(pTableScanParam->pUidList);
if (pTableScanParam->pOrgTbInfo) {
taosArrayDestroy(pTableScanParam->pOrgTbInfo->colMap);
taosMemoryFreeClear(pTableScanParam->pOrgTbInfo);
}
freeOperatorParamImpl(pParam, OP_GET_PARAM);
}
void freeTableScanNotifyOperatorParam(SOperatorParam* pParam) { freeOperatorParamImpl(pParam, OP_NOTIFY_PARAM); }
void freeOpParamItem(void* pItem) {
SOperatorParam* pParam = *(SOperatorParam**)pItem;
pParam->reUse = false;
freeOperatorParam(pParam, OP_GET_PARAM);
}
void freeVirtualTableScanGetOperatorParam(SOperatorParam* pParam) {
SVTableScanOperatorParam* pVTableScanParam = (SVTableScanOperatorParam*)pParam->value;
taosArrayDestroyEx(pVTableScanParam->pOpParamArray, freeOpParamItem);
freeOperatorParamImpl(pParam, OP_GET_PARAM);
}
void freeVTableScanNotifyOperatorParam(SOperatorParam* pParam) { freeOperatorParamImpl(pParam, OP_NOTIFY_PARAM); }
void freeOperatorParam(SOperatorParam* pParam, SOperatorParamType type) {
if (NULL == pParam) {
if (NULL == pParam || pParam->reUse) {
return;
}
@ -1252,6 +1274,9 @@ void freeOperatorParam(SOperatorParam* pParam, SOperatorParamType type) {
case QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN:
type == OP_GET_PARAM ? freeTableScanGetOperatorParam(pParam) : freeTableScanNotifyOperatorParam(pParam);
break;
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
type == OP_GET_PARAM ? freeVirtualTableScanGetOperatorParam(pParam) : freeVTableScanNotifyOperatorParam(pParam);
break;
default:
qError("unsupported op %d param, type %d", pParam->opType, type);
break;

1
source/libs/executor/src/groupcacheoperator.c
View File

@ -962,6 +962,7 @@ static int32_t handleGroupCacheRetrievedBlk(struct SOperatorInfo* pOperator, SSD
fakeGcParam.needCache = true;
fakeParam.downstreamIdx = pSession->downstreamIdx;
fakeParam.value = &fakeGcParam;
fakeParam.reUse = false;
code = addNewGroupData(pOperator, &fakeParam, &pGroup, GROUP_CACHE_DEFAULT_VGID, pBlock->info.id.groupId);
if (TSDB_CODE_SUCCESS != code) {
return code;

16
source/libs/executor/src/mergeoperator.c
View File

@ -39,7 +39,8 @@ typedef struct SNonSortMergeInfo {
typedef struct SColsMergeInfo {
SNodeList* pTargets;
uint64_t srcBlkIds[2];
size_t sourceNum;
uint64_t* srcBlkIds;
} SColsMergeInfo;
typedef struct SMultiwayMergeOperatorInfo {
@ -437,7 +438,7 @@ int32_t doColsMerge(SOperatorInfo* pOperator, SSDataBlock** pResBlock) {
qDebug("start to merge columns, %s", GET_TASKID(pTaskInfo));
for (int32_t i = 0; i < 2; ++i) {
for (int32_t i = 0; i < pColsMerge->sourceNum; ++i) {
pBlock = getNextBlockFromDownstream(pOperator, i);
if (pBlock && pBlock->info.rows > 1) {
qError("more than 1 row returned from downstream, rows:%" PRId64, pBlock->info.rows);
@ -453,7 +454,7 @@ int32_t doColsMerge(SOperatorInfo* pOperator, SSDataBlock** pResBlock) {
}
setOperatorCompleted(pOperator);
if (2 == nullBlkNum) {
if (pColsMerge->sourceNum == nullBlkNum) {
return code;
}
@ -464,6 +465,8 @@ int32_t doColsMerge(SOperatorInfo* pOperator, SSDataBlock** pResBlock) {
}
void destroyColsMergeOperatorInfo(void* param) {
SColsMergeInfo* pColsMergeInfo = param;
taosMemoryFreeClear(pColsMergeInfo->srcBlkIds);
}
int32_t getColsMergeExplainExecInfo(SOperatorInfo* pOptr, void** pOptrExplain, uint32_t* len) {
@ -640,8 +643,11 @@ int32_t createMultiwayMergeOperatorInfo(SOperatorInfo** downStreams, size_t numS
TSDB_CHECK_CODE(code, lino, _error);
pColsMerge->pTargets = pMergePhyNode->pTargets;
pColsMerge->srcBlkIds[0] = getOperatorResultBlockId(downStreams[0], 0);
pColsMerge->srcBlkIds[1] = getOperatorResultBlockId(downStreams[1], 0);
pColsMerge->sourceNum = numStreams;
pColsMerge->srcBlkIds = taosMemoryCalloc(numStreams, sizeof(uint64_t));
for (size_t i = 0; i < numStreams; ++i) {
pColsMerge->srcBlkIds[i] = getOperatorResultBlockId(downStreams[i], 0);
}
break;
}
default:

44
source/libs/executor/src/operator.c
View File

@ -280,7 +280,8 @@ int32_t stopTableScanOperator(SOperatorInfo* pOperator, const char* pIdStr, SSto
}
int32_t createOperator(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHandle* pHandle, SNode* pTagCond,
SNode* pTagIndexCond, const char* pUser, const char* dbname, SOperatorInfo** pOptrInfo) {
SNode* pTagIndexCond, const char* pUser, const char* dbname, SOperatorInfo** pOptrInfo,
EOPTR_EXEC_MODEL model) {
QRY_PARAM_CHECK(pOptrInfo);
int32_t code = 0;
@ -512,6 +513,10 @@ int32_t createOperator(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHand
}
} else if (QUERY_NODE_PHYSICAL_PLAN_PROJECT == type) {
code = createProjectOperatorInfo(NULL, (SProjectPhysiNode*)pPhyNode, pTaskInfo, &pOperator);
} else if (QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN == type && model != OPTR_EXEC_MODEL_STREAM) {
code = createVirtualTableMergeOperatorInfo(NULL, pHandle, NULL, 0, (SVirtualScanPhysiNode*)pPhyNode, pTaskInfo, &pOperator);
} else if (QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN == type && model == OPTR_EXEC_MODEL_STREAM) {
code = createStreamVtableMergeOperatorInfo(pHandle, (SVirtualScanPhysiNode*)pPhyNode, pTagCond, pTaskInfo, &pOperator);
} else {
code = TSDB_CODE_INVALID_PARA;
pTaskInfo->code = code;
@ -536,7 +541,7 @@ int32_t createOperator(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHand
for (int32_t i = 0; i < size; ++i) {
SPhysiNode* pChildNode = (SPhysiNode*)nodesListGetNode(pPhyNode->pChildren, i);
code = createOperator(pChildNode, pTaskInfo, pHandle, pTagCond, pTagIndexCond, pUser, dbname, &ops[i]);
code = createOperator(pChildNode, pTaskInfo, pHandle, pTagCond, pTagIndexCond, pUser, dbname, &ops[i], model);
if (ops[i] == NULL || code != 0) {
for (int32_t j = 0; j < i; ++j) {
destroyOperator(ops[j]);
@ -630,7 +635,7 @@ int32_t createOperator(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHand
} else if (QUERY_NODE_PHYSICAL_PLAN_GROUP_CACHE == type) {
code = createGroupCacheOperatorInfo(ops, size, (SGroupCachePhysiNode*)pPhyNode, pTaskInfo, &pOptr);
} else if (QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL == type) {
code = createDynQueryCtrlOperatorInfo(ops, size, (SDynQueryCtrlPhysiNode*)pPhyNode, pTaskInfo, &pOptr);
code = createDynQueryCtrlOperatorInfo(ops, size, (SDynQueryCtrlPhysiNode*)pPhyNode, pTaskInfo, pHandle, &pOptr);
} else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_COUNT == type) {
code = createStreamCountAggOperatorInfo(ops[0], pPhyNode, pTaskInfo, pHandle, &pOptr);
} else if (QUERY_NODE_PHYSICAL_PLAN_MERGE_COUNT == type) {
@ -651,6 +656,39 @@ int32_t createOperator(SPhysiNode* pPhyNode, SExecTaskInfo* pTaskInfo, SReadHand
code = createEventNonblockOperatorInfo(ops[0], pPhyNode, pTaskInfo, pHandle, &pOptr);
} else if (QUERY_NODE_PHYSICAL_PLAN_STREAM_CONTINUE_COUNT == type) {
//todo (liuyao) add
} else if (QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN == type && model != OPTR_EXEC_MODEL_STREAM) {
SVirtualScanPhysiNode* pVirtualTableScanNode = (SVirtualScanPhysiNode*)pPhyNode;
// NOTE: this is an patch to fix the physical plan
if (pVirtualTableScanNode->scan.node.pLimit != NULL) {
pVirtualTableScanNode->groupSort = true;
}
STableListInfo* pTableListInfo = tableListCreate();
if (!pTableListInfo) {
pTaskInfo->code = terrno;
qError("%s failed at line %d since %s", __func__, __LINE__, tstrerror(terrno));
return terrno;
}
code = initQueriedTableSchemaInfo(pHandle, &pVirtualTableScanNode->scan, dbname, pTaskInfo);
if (code) {
pTaskInfo->code = code;
tableListDestroy(pTableListInfo);
return code;
}
code = createScanTableListInfo(&pVirtualTableScanNode->scan, pVirtualTableScanNode->pGroupTags, pVirtualTableScanNode->groupSort,
pHandle, pTableListInfo, pTagCond, pTagIndexCond, pTaskInfo);
if (code) {
pTaskInfo->code = code;
tableListDestroy(pTableListInfo);
qError("failed to createScanTableListInfo, code:%s, %s", tstrerror(code), idstr);
return code;
}
code = createVirtualTableMergeOperatorInfo(ops, pHandle, pTableListInfo, size, (SVirtualScanPhysiNode*)pPhyNode, pTaskInfo, &pOptr);
} else {
code = TSDB_CODE_INVALID_PARA;
pTaskInfo->code = code;

4
source/libs/executor/src/querytask.c
View File

@ -126,7 +126,7 @@ int32_t createExecTaskInfo(SSubplan* pPlan, SExecTaskInfo** pTaskInfo, SReadHand
(*pTaskInfo)->pSubplan = pPlan;
(*pTaskInfo)->pWorkerCb = pHandle->pWorkerCb;
code = createOperator(pPlan->pNode, *pTaskInfo, pHandle, pPlan->pTagCond, pPlan->pTagIndexCond, pPlan->user,
pPlan->dbFName, &((*pTaskInfo)->pRoot));
pPlan->dbFName, &((*pTaskInfo)->pRoot), model);
if (NULL == (*pTaskInfo)->pRoot || code != 0) {
doDestroyTask(*pTaskInfo);
@ -176,7 +176,7 @@ int32_t initQueriedTableSchemaInfo(SReadHandle* pHandle, SScanPhysiNode* pScanNo
if (mr.me.type == TSDB_SUPER_TABLE) {
schemaInfo.sw = tCloneSSchemaWrapper(&mr.me.stbEntry.schemaRow);
schemaInfo.tversion = mr.me.stbEntry.schemaTag.version;
} else if (mr.me.type == TSDB_CHILD_TABLE) {
} else if (mr.me.type == TSDB_CHILD_TABLE || mr.me.type == TSDB_VIRTUAL_CHILD_TABLE) {
tDecoderClear(&mr.coder);
tb_uid_t suid = mr.me.ctbEntry.suid;

321
source/libs/executor/src/scanoperator.c
View File

@ -332,6 +332,10 @@ bool applyLimitOffset(SLimitInfo* pLimitInfo, SSDataBlock* pBlock, SExecTaskInfo
return false;
}
static bool isDynVtbScan(SOperatorInfo* pOperator) {
return pOperator->dynamicTask && ((STableScanInfo*)(pOperator->info))->virtualStableScan;
}
static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableScanInfo, SSDataBlock* pBlock,
uint32_t* status) {
int32_t code = TSDB_CODE_SUCCESS;
@ -474,9 +478,14 @@ static int32_t loadDataBlock(SOperatorInfo* pOperator, STableScanBase* pTableSca
return code;
}
code = doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, pBlock->info.rows);
if (code) {
return code;
if ((pOperator->operatorType == QUERY_NODE_PHYSICAL_PLAN_TABLE_SCAN) && ((STableScanInfo *)pOperator->info)->ignoreTag) {
// do nothing
} else {
// dyn vtb scan do not read tag from origin tables.
code = doSetTagColumnData(pTableScanInfo, pBlock, pTaskInfo, pBlock->info.rows);
if (code) {
return code;
}
}
// restore the previous value
@ -549,7 +558,7 @@ static int32_t createTableCacheVal(const SMetaReader* pMetaReader, STableCachedV
QUERY_CHECK_NULL(pVal->pName, code, lino, _end, terrno);
// only child table has tag value
if (pMetaReader->me.type == TSDB_CHILD_TABLE) {
if (pMetaReader->me.type == TSDB_CHILD_TABLE || pMetaReader->me.type == TSDB_VIRTUAL_CHILD_TABLE) {
STag* pTag = (STag*)pMetaReader->me.ctbEntry.pTags;
pVal->pTags = taosMemoryMalloc(pTag->len);
QUERY_CHECK_NULL(pVal->pTags, code, lino, _end, terrno);
@ -1183,6 +1192,145 @@ static int32_t createTableListInfoFromParam(SOperatorInfo* pOperator) {
return code;
}
static int32_t doInitReader(STableScanInfo* pInfo, SExecTaskInfo* pTaskInfo, SStorageAPI* pAPI, int32_t* pNum,
STableKeyInfo** pList) {
const char* idStr = GET_TASKID(pTaskInfo);
int32_t code = initNextGroupScan(pInfo, pList, pNum);
if (code) {
qError("%s failed to init groupScan Info, code:%s at line:%d", idStr, tstrerror(code), __LINE__);
return code;
}
if (pInfo->base.dataReader != NULL) {
qError("%s tsdb reader should be null", idStr);
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
code = pAPI->tsdReader.tsdReaderOpen(pInfo->base.readHandle.vnode, &pInfo->base.cond, *pList, *pNum, pInfo->pResBlock,
(void**)&pInfo->base.dataReader, idStr, &pInfo->pIgnoreTables);
if (code) {
qError("%s failed to open tsdbReader, code:%s at line:%d", idStr, tstrerror(code), __LINE__);
}
return code;
}
static int32_t createVTableScanInfoFromParam(SOperatorInfo* pOperator) {
int32_t code = 0;
int32_t lino = 0;
STableScanInfo* pInfo = pOperator->info;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
SStorageAPI* pAPI = &pTaskInfo->storageAPI;
STableListInfo* pListInfo = pInfo->base.pTableListInfo;
STableScanOperatorParam* pParam = (STableScanOperatorParam*)pOperator->pOperatorGetParam->value;
SMetaReader orgTable = {0};
SMetaReader superTable = {0};
SSchemaWrapper* schema = NULL;
SArray* pColArray = NULL;
SArray* pBlockColArray = NULL;
int32_t num = 0;
STableKeyInfo* pList = NULL;
cleanupQueryTableDataCond(&pInfo->base.cond);
if (pAPI->tsdReader.tsdReaderClose) {
pAPI->tsdReader.tsdReaderClose(pInfo->base.dataReader);
}
pAPI->metaReaderFn.initReader(&orgTable, pInfo->base.readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
QUERY_CHECK_CODE(pAPI->metaReaderFn.getTableEntryByName(&orgTable, strstr(pParam->pOrgTbInfo->tbName, ".") + 1), lino, _return);
switch (orgTable.me.type) {
case TSDB_CHILD_TABLE:
pAPI->metaReaderFn.initReader(&superTable, pInfo->base.readHandle.vnode, META_READER_LOCK, &pAPI->metaFn);
QUERY_CHECK_CODE(pAPI->metaReaderFn.getTableEntryByUid(&superTable, orgTable.me.ctbEntry.suid), lino, _return);
schema = &superTable.me.stbEntry.schemaRow;
break;
case TSDB_NORMAL_TABLE:
schema = &orgTable.me.ntbEntry.schemaRow;
break;
default:
qError("invalid table type:%d", orgTable.me.type);
return TSDB_CODE_INVALID_PARA;
break;
}
pColArray = taosArrayInit(schema->nCols, sizeof(SColIdPair));
QUERY_CHECK_NULL(pColArray, code, lino, _return, terrno);
pBlockColArray = taosArrayInit(schema->nCols, sizeof(int32_t));
QUERY_CHECK_NULL(pBlockColArray, code, lino, _return, terrno);
// virtual table's origin table scan do not has ts column.
SColIdPair tsPair = {.vtbColId = PRIMARYKEY_TIMESTAMP_COL_ID, .orgColId = PRIMARYKEY_TIMESTAMP_COL_ID};
QUERY_CHECK_NULL(taosArrayPush(pColArray, &tsPair), code, lino, _return, terrno);
for (int32_t i = 0; i < taosArrayGetSize(pParam->pOrgTbInfo->colMap); ++i) {
SColIdNameKV *kv = taosArrayGet(pParam->pOrgTbInfo->colMap, i);
for (int32_t j = 0; j < schema->nCols; j++) {
if (strncmp(kv->colName, schema->pSchema[j].name, strlen(schema->pSchema[j].name)) == 0) {
SColIdPair pPair = {.vtbColId = kv->colId, .orgColId = (col_id_t)(j + 1)};
QUERY_CHECK_NULL(taosArrayPush(pColArray, &pPair), code, lino, _return, terrno);
break;
}
}
}
for (int32_t i = 0; i < taosArrayGetSize(pColArray); i++) {
SColIdPair *pPair = (SColIdPair*)taosArrayGet(pColArray, i);
for (int32_t j = 0; j < taosArrayGetSize(pInfo->base.matchInfo.pList); j++) {
SColMatchItem *pItem = taosArrayGet(pInfo->base.matchInfo.pList, j);
if (pItem->colId == pPair->vtbColId) {
SColIdPair tmpPair = {.orgColId = pPair->orgColId, .vtbColId = pItem->dstSlotId};
QUERY_CHECK_NULL(taosArrayPush(pBlockColArray, &tmpPair), code, lino, _return, terrno);
break;
}
}
}
if (pInfo->pResBlock) {
blockDataDestroy(pInfo->pResBlock);
pInfo->pResBlock = NULL;
}
QUERY_CHECK_CODE(createOneDataBlockWithColArray(pInfo->pOrgBlock, pBlockColArray, &pInfo->pResBlock), lino, _return);
QUERY_CHECK_CODE(initQueryTableDataCondWithColArray(&pInfo->base.cond, &pInfo->base.orgCond, &pInfo->base.readHandle, pColArray), lino, _return);
pInfo->base.cond.twindows.skey = pParam->window.ekey + 1;
pInfo->base.cond.suid = orgTable.me.type == TSDB_CHILD_TABLE ? superTable.me.uid : 0;
pInfo->currentGroupId = 0;
pInfo->base.dataReader = NULL;
pInfo->ignoreTag = true;
pListInfo->oneTableForEachGroup = true;
taosHashClear(pListInfo->map);
taosArrayClear(pListInfo->pTableList);
uint64_t pUid = orgTable.me.uid;
STableKeyInfo info = {.groupId = 0, .uid = pUid};
int32_t tableIdx = 0;
QUERY_CHECK_CODE(taosHashPut(pListInfo->map, &pUid, sizeof(uint64_t), &tableIdx, sizeof(int32_t)), lino, _return);
QUERY_CHECK_NULL(taosArrayPush(pListInfo->pTableList, &info), code, lino, _return, terrno);
qDebug("add dynamic table scan uid:%" PRIu64 ", %s", info.uid, GET_TASKID(pTaskInfo));
pOperator->status = OP_OPENED;
taosRLockLatch(&pTaskInfo->lock);
code = doInitReader(pInfo, pTaskInfo, pAPI, &num, &pList);
taosRUnLockLatch(&pTaskInfo->lock);
QUERY_CHECK_CODE(code, lino, _return);
if (pInfo->pResBlock->info.capacity > pOperator->resultInfo.capacity) {
pOperator->resultInfo.capacity = pInfo->pResBlock->info.capacity;
}
pInfo->currentGroupId = -1;
_return:
if (code) {
qError("%s failed at line %d since %s", __func__, lino, tstrerror(code));
}
taosArrayDestroy(pColArray);
taosArrayDestroy(pBlockColArray);
pAPI->metaReaderFn.clearReader(&superTable);
pAPI->metaReaderFn.clearReader(&orgTable);
return code;
}
static int32_t startNextGroupScan(SOperatorInfo* pOperator, SSDataBlock** pResult) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
@ -1239,29 +1387,6 @@ _end:
return code;
}
static int32_t doInitReader(STableScanInfo* pInfo, SExecTaskInfo* pTaskInfo, SStorageAPI* pAPI, int32_t* pNum,
STableKeyInfo** pList) {
const char* idStr = GET_TASKID(pTaskInfo);
int32_t code = initNextGroupScan(pInfo, pList, pNum);
if (code) {
qError("%s failed to init groupScan Info, code:%s at line:%d", idStr, tstrerror(code), __LINE__);
return code;
}
if (pInfo->base.dataReader != NULL) {
qError("%s tsdb reader should be null", idStr);
return TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR;
}
code = pAPI->tsdReader.tsdReaderOpen(pInfo->base.readHandle.vnode, &pInfo->base.cond, *pList, *pNum, pInfo->pResBlock,
(void**)&pInfo->base.dataReader, idStr, &pInfo->pIgnoreTables);
if (code) {
qError("%s failed to open tsdbReader, code:%s at line:%d", idStr, tstrerror(code), __LINE__);
}
return code;
}
static int32_t groupSeqTableScan(SOperatorInfo* pOperator, SSDataBlock** pResBlock) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
@ -1336,25 +1461,49 @@ int32_t doTableScanNext(SOperatorInfo* pOperator, SSDataBlock** ppRes) {
if (pOperator->pOperatorGetParam) {
pOperator->dynamicTask = true;
code = createTableListInfoFromParam(pOperator);
freeOperatorParam(pOperator->pOperatorGetParam, OP_GET_PARAM);
pOperator->pOperatorGetParam = NULL;
QUERY_CHECK_CODE(code, lino, _end);
if (isDynVtbScan(pOperator)) {
code = createVTableScanInfoFromParam(pOperator);
freeOperatorParam(pOperator->pOperatorGetParam, OP_GET_PARAM);
pOperator->pOperatorGetParam = NULL;
QUERY_CHECK_CODE(code, lino, _end);
if (pOperator->status == OP_EXEC_DONE) {
pInfo->currentGroupId = -1;
pOperator->status = OP_OPENED;
SSDataBlock* result = NULL;
while (true) {
code = startNextGroupScan(pOperator, &result);
QUERY_CHECK_CODE(code, lino, _end);
QUERY_CHECK_CODE(startNextGroupScan(pOperator, &result), lino, _end);
if (result || pOperator->status == OP_EXEC_DONE) {
(*ppRes) = result;
SSDataBlock* res = NULL;
if (result) {
QUERY_CHECK_CODE(createOneDataBlockWithTwoBlock(result, pInfo->pOrgBlock, &res), lino, _end);
pInfo->pResBlock = res;
blockDataDestroy(result);
}
(*ppRes) = res;
return code;
}
}
} else {
code = createTableListInfoFromParam(pOperator);
freeOperatorParam(pOperator->pOperatorGetParam, OP_GET_PARAM);
pOperator->pOperatorGetParam = NULL;
QUERY_CHECK_CODE(code, lino, _end);
if (pOperator->status == OP_EXEC_DONE) {
pInfo->currentGroupId = -1;
pOperator->status = OP_OPENED;
SSDataBlock* result = NULL;
while (true) {
code = startNextGroupScan(pOperator, &result);
QUERY_CHECK_CODE(code, lino, _end);
if (result || pOperator->status == OP_EXEC_DONE) {
(*ppRes) = result;
return code;
}
}
}
}
}
@ -1440,6 +1589,7 @@ static int32_t getTableScannerExecInfo(struct SOperatorInfo* pOptr, void** pOptr
static void destroyTableScanBase(STableScanBase* pBase, TsdReader* pAPI) {
cleanupQueryTableDataCond(&pBase->cond);
cleanupQueryTableDataCond(&pBase->orgCond);
if (pAPI->tsdReaderClose) {
pAPI->tsdReaderClose(pBase->dataReader);
@ -1458,6 +1608,7 @@ static void destroyTableScanBase(STableScanBase* pBase, TsdReader* pAPI) {
static void destroyTableScanOperatorInfo(void* param) {
STableScanInfo* pTableScanInfo = (STableScanInfo*)param;
blockDataDestroy(pTableScanInfo->pResBlock);
blockDataDestroy(pTableScanInfo->pOrgBlock);
taosHashCleanup(pTableScanInfo->pIgnoreTables);
destroyTableScanBase(&pTableScanInfo->base, &pTableScanInfo->base.readerAPI);
taosMemoryFreeClear(param);
@ -1531,6 +1682,14 @@ int32_t createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SReadHa
code = prepareDataBlockBuf(pInfo->pResBlock, &pInfo->base.matchInfo);
QUERY_CHECK_CODE(code, lino, _error);
pInfo->virtualStableScan = pScanNode->virtualStableScan;
if (pScanNode->node.dynamicOp && pScanNode->virtualStableScan) {
TSWAP(pInfo->pOrgBlock, pInfo->pResBlock);
pInfo->pResBlock = NULL;
memcpy(&pInfo->base.orgCond, &pInfo->base.cond, sizeof(SQueryTableDataCond));
memset(&pInfo->base.cond, 0, sizeof(SQueryTableDataCond));
}
code = filterInitFromNode((SNode*)pTableScanNode->scan.node.pConditions, &pOperator->exprSupp.pFilterInfo, 0);
QUERY_CHECK_CODE(code, lino, _error);
@ -1545,6 +1704,7 @@ int32_t createTableScanOperatorInfo(STableScanPhysiNode* pTableScanNode, SReadHa
pOperator->exprSupp.numOfExprs = numOfCols;
pInfo->needCountEmptyTable = tsCountAlwaysReturnValue && pTableScanNode->needCountEmptyTable;
pInfo->ignoreTag = false;
pInfo->base.pTableListInfo = pTableListInfo;
pInfo->base.metaCache.pTableMetaEntryCache = taosLRUCacheInit(1024 * 128, -1, .5);
@ -2684,7 +2844,7 @@ _end:
return code;
}
static int32_t generateDeleteResultBlock(SStreamScanInfo* pInfo, SSDataBlock* pSrcBlock, SSDataBlock* pDestBlock) {
int32_t generateDeleteResultBlock(SStreamScanInfo* pInfo, SSDataBlock* pSrcBlock, SSDataBlock* pDestBlock) {
blockDataCleanup(pDestBlock);
int32_t rows = pSrcBlock->info.rows;
if (rows == 0) {
@ -2700,8 +2860,7 @@ static int32_t generateDeleteResultBlock(SStreamScanInfo* pInfo, SSDataBlock* pS
return generateDeleteResultBlockImpl(pInfo, pSrcBlock, pDestBlock);
}
static int32_t generateScanRange(SStreamScanInfo* pInfo, SSDataBlock* pSrcBlock, SSDataBlock* pDestBlock,
EStreamType type) {
int32_t generateScanRange(SStreamScanInfo* pInfo, SSDataBlock* pSrcBlock, SSDataBlock* pDestBlock, EStreamType type) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (isIntervalWindow(pInfo)) {
@ -2748,7 +2907,7 @@ bool checkExpiredData(SStateStore* pAPI, SUpdateInfo* pUpdateInfo, STimeWindowAg
return isExpired;
}
static int32_t checkUpdateData(SStreamScanInfo* pInfo, bool invertible, SSDataBlock* pBlock, bool out) {
int32_t checkUpdateData(SStreamScanInfo* pInfo, bool invertible, SSDataBlock* pBlock, bool out) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (out) {
@ -2991,14 +3150,14 @@ int32_t colIdComparFn(const void* param1, const void* param2) {
}
}
int32_t setBlockIntoRes(SStreamScanInfo* pInfo, const SSDataBlock* pBlock, STimeWindow* pTimeWindow,
bool filter) {
int32_t setBlockIntoRes(SStreamScanInfo* pInfo, const SSDataBlock* pBlock, STimeWindow* pTimeWindow, bool filter) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SDataBlockInfo* pBlockInfo = &pInfo->pRes->info;
SOperatorInfo* pOperator = pInfo->pStreamScanOp;
SExecTaskInfo* pTaskInfo = pOperator->pTaskInfo;
const char* id = GET_TASKID(pTaskInfo);
SSHashObj* pVtableInfos = pTaskInfo->pSubplan->pVTables;
code = blockDataEnsureCapacity(pInfo->pRes, pBlock->info.rows);
QUERY_CHECK_CODE(code, lino, _end);
@ -3009,7 +3168,12 @@ int32_t setBlockIntoRes(SStreamScanInfo* pInfo, const SSDataBlock* pBlock, STime
pBlockInfo->version = pBlock->info.version;
STableScanInfo* pTableScanInfo = pInfo->pTableScanOp->info;
pBlockInfo->id.groupId = tableListGetTableGroupId(pTableScanInfo->base.pTableListInfo, pBlock->info.id.uid);
if (pVtableInfos == NULL) {
pBlockInfo->id.groupId = tableListGetTableGroupId(pTableScanInfo->base.pTableListInfo, pBlock->info.id.uid);
} else {
// use original table uid as groupId for vtable
pBlockInfo->id.groupId = pBlock->info.id.groupId;
}
SArray* pColList = taosArrayInit(4, sizeof(int32_t));
QUERY_CHECK_NULL(pColList, code, lino, _end, terrno);
@ -3331,7 +3495,7 @@ _end:
return code;
}
static int32_t doCheckUpdate(SStreamScanInfo* pInfo, TSKEY endKey, SSDataBlock* pBlock) {
int32_t doCheckUpdate(SStreamScanInfo* pInfo, TSKEY endKey, SSDataBlock* pBlock) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (pInfo->pUpdateInfo) {
@ -3530,17 +3694,17 @@ _end:
qInfo("%s end at line %d", __func__, lino);
}
static bool hasScanRange(SStreamScanInfo* pInfo) {
bool hasScanRange(SStreamScanInfo* pInfo) {
SStreamAggSupporter* pSup = pInfo->windowSup.pStreamAggSup;
return pSup && pSup->pScanBlock->info.rows > 0 && (isStateWindow(pInfo) || isCountWindow(pInfo));
}
static bool isStreamWindow(SStreamScanInfo* pInfo) {
bool isStreamWindow(SStreamScanInfo* pInfo) {
return isIntervalWindow(pInfo) || isSessionWindow(pInfo) || isStateWindow(pInfo) || isCountWindow(pInfo) ||
isTimeSlice(pInfo);
}
static int32_t copyGetResultBlock(SSDataBlock* dest, TSKEY start, TSKEY end) {
int32_t copyGetResultBlock(SSDataBlock* dest, TSKEY start, TSKEY end) {
int32_t code = blockDataEnsureCapacity(dest, 1);
if (code != TSDB_CODE_SUCCESS) {
return code;
@ -4313,7 +4477,7 @@ _end:
return code;
}
static void destroyStreamScanOperatorInfo(void* param) {
void destroyStreamScanOperatorInfo(void* param) {
if (param == NULL) {
return;
}
@ -4326,6 +4490,10 @@ static void destroyStreamScanOperatorInfo(void* param) {
if (pStreamScan->tqReader != NULL && pStreamScan->readerFn.tqReaderClose != NULL) {
pStreamScan->readerFn.tqReaderClose(pStreamScan->tqReader);
}
if (pStreamScan->pVtableMergeHandles) {
taosHashCleanup(pStreamScan->pVtableMergeHandles);
pStreamScan->pVtableMergeHandles = NULL;
}
if (pStreamScan->matchInfo.pList) {
taosArrayDestroy(pStreamScan->matchInfo.pList);
}
@ -4513,6 +4681,42 @@ _end:
return code;
}
static int32_t createStreamVtableBlock(SColMatchInfo *pMatchInfo, SSDataBlock **ppRes, const char *idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SSDataBlock *pRes = NULL;
QUERY_CHECK_NULL(pMatchInfo, code, lino, _end, TSDB_CODE_INVALID_PARA);
*ppRes = NULL;
code = createDataBlock(&pRes);
QUERY_CHECK_CODE(code, lino, _end);
int32_t numOfOutput = taosArrayGetSize(pMatchInfo->pList);
for (int32_t i = 0; i < numOfOutput; ++i) {
SColMatchItem* pItem = taosArrayGet(pMatchInfo->pList, i);
if (!pItem->needOutput) {
continue;
}
SColumnInfoData colInfo = createColumnInfoData(pItem->dataType.type, pItem->dataType.bytes, pItem->colId);
code = blockDataAppendColInfo(pRes, &colInfo);
QUERY_CHECK_CODE(code, lino, _end);
}
*ppRes = pRes;
pRes = NULL;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (pRes != NULL) {
blockDataDestroy(pRes);
}
return code;
}
static int32_t createStreamNormalScanOperatorInfo(SReadHandle* pHandle, STableScanPhysiNode* pTableScanNode,
SNode* pTagCond, STableListInfo* pTableListInfo,
SExecTaskInfo* pTaskInfo, SOperatorInfo** pOptrInfo) {
@ -4525,6 +4729,7 @@ static int32_t createStreamNormalScanOperatorInfo(SReadHandle* pHandle, STableSc
SOperatorInfo* pOperator = taosMemoryCalloc(1, sizeof(SOperatorInfo));
SStorageAPI* pAPI = &pTaskInfo->storageAPI;
const char* idstr = pTaskInfo->id.str;
SSHashObj* pVtableInfos = pTaskInfo->pSubplan->pVTables;
if (pInfo == NULL || pOperator == NULL) {
code = terrno;
@ -4631,6 +4836,16 @@ static int32_t createStreamNormalScanOperatorInfo(SReadHandle* pHandle, STableSc
QUERY_CHECK_NULL(pInfo->tqReader, code, lino, _error, TSDB_CODE_QRY_EXECUTOR_INTERNAL_ERROR);
}
if (pVtableInfos != NULL) {
// save vtable info into tqReader for vtable source scan
SSDataBlock* pResBlock = NULL;
code = createStreamVtableBlock(&pInfo->matchInfo, &pResBlock, idstr);
QUERY_CHECK_CODE(code, lino, _error);
code = pAPI->tqReaderFn.tqReaderSetVtableInfo(pInfo->tqReader, pHandle->vnode, pAPI, pVtableInfos, &pResBlock,
idstr);
QUERY_CHECK_CODE(code, lino, _error);
}
pInfo->pUpdateInfo = NULL;
pInfo->pTableScanOp = pTableScanOp;
if (pInfo->pTableScanOp->pTaskInfo->streamInfo.pState) {
@ -4648,12 +4863,14 @@ static int32_t createStreamNormalScanOperatorInfo(SReadHandle* pHandle, STableSc
QUERY_CHECK_CODE(code, lino, _error);
// set the extract column id to streamHandle
pAPI->tqReaderFn.tqReaderSetColIdList(pInfo->tqReader, pColIds);
code = pAPI->tqReaderFn.tqReaderSetColIdList(pInfo->tqReader, pColIds, idstr);
QUERY_CHECK_CODE(code, lino, _error);
SArray* tableIdList = NULL;
code = extractTableIdList(((STableScanInfo*)(pInfo->pTableScanOp->info))->base.pTableListInfo, &tableIdList);
QUERY_CHECK_CODE(code, lino, _error);
pAPI->tqReaderFn.tqReaderSetQueryTableList(pInfo->tqReader, tableIdList, idstr);
code = pAPI->tqReaderFn.tqReaderSetQueryTableList(pInfo->tqReader, tableIdList, idstr);
QUERY_CHECK_CODE(code, lino, _error);
taosArrayDestroy(tableIdList);
memcpy(&pTaskInfo->streamInfo.tableCond, &pTSInfo->base.cond, sizeof(SQueryTableDataCond));
} else {

607
source/libs/executor/src/streamVtableMerge.c Normal file
View File

@ -0,0 +1,607 @@
/*
* Copyright (c) 2019 TAOS Data, Inc. <jhtao@taosdata.com>
*
* This program is free software: you can use, redistribute, and/or modify
* it under the terms of the GNU Affero General Public License, version 3
* or later ("AGPL"), as published by the Free Software Foundation.
*
* This program is distributed in the hope that it will be useful, but WITHOUT
* ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or
* FITNESS FOR A PARTICULAR PURPOSE.
*
* You should have received a copy of the GNU Affero General Public License
* along with this program. If not, see <http://www.gnu.org/licenses/>.
*/
#include "streamVtableMerge.h"
#include "query.h"
#include "tdatablock.h"
#include "tglobal.h"
#include "tlosertree.h"
typedef struct SVMBufPageInfo {
int32_t pageId;
int64_t expireTimeMs;
} SVMBufPageInfo;
typedef struct SStreamVtableMergeSource {
SDiskbasedBuf* pBuf; // buffer for storing data
int32_t* pTotalPages; // total pages of all sources in the buffer
SSDataBlock* pInputDataBlock; // data block to be written to the buffer
int64_t currentExpireTimeMs; // expire time of the input data block
SList* pageInfoList; // info of current source's page in the buffer
SSDataBlock* pOutputDataBlock; // data block read from the buffer
int32_t rowIndex; // current row index of the output data block
int64_t latestTs; // latest timestamp of the source
int64_t* pGlobalLatestTs; // global latest timestamp of all sources
} SStreamVtableMergeSource;
typedef struct SStreamVtableMergeHandle {
SDiskbasedBuf* pBuf;
int32_t numOfPages;
int32_t numPageLimit;
int32_t nSrcTbls;
SHashObj* pSources;
SSDataBlock* datablock; // Does not store data, only used to save the schema of input/output data blocks
SMultiwayMergeTreeInfo* pMergeTree;
int64_t globalLatestTs;
} SStreamVtableMergeHandle;
typedef enum StreamVirtualMergeMode {
STREAM_VIRTUAL_MERGE_WAIT_FOREVER,
STREAM_VIRTUAL_MERGE_MAX_DELAY,
STREAM_VIRTUAL_MERGE_MAX_MEMORY,
} StreamVirtualMergeMode;
static void svmSourceDestroy(void* ptr) {
SStreamVtableMergeSource** ppSource = ptr;
if (ppSource == NULL || *ppSource == NULL) {
return;
}
SStreamVtableMergeSource* pSource = *ppSource;
if (pSource->pInputDataBlock) {
blockDataDestroy(pSource->pInputDataBlock);
pSource->pInputDataBlock = NULL;
}
pSource->pageInfoList = tdListFree(pSource->pageInfoList);
if (pSource->pOutputDataBlock) {
blockDataDestroy(pSource->pOutputDataBlock);
pSource->pOutputDataBlock = NULL;
}
taosMemoryFreeClear(*ppSource);
}
static int32_t svmSourceFlushInput(SStreamVtableMergeSource* pSource, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SSDataBlock* pBlock = NULL;
SVMBufPageInfo pageInfo = {0};
void* page = NULL;
QUERY_CHECK_NULL(pSource, code, lino, _end, TSDB_CODE_INVALID_PARA);
// check data block size
pBlock = pSource->pInputDataBlock;
int32_t size = blockDataGetSize(pBlock) + sizeof(int32_t) + taosArrayGetSize(pBlock->pDataBlock) * sizeof(int32_t);
QUERY_CHECK_CONDITION(size <= getBufPageSize(pSource->pBuf), code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
// allocate a new page and write data to it
page = getNewBufPage(pSource->pBuf, &pageInfo.pageId);
QUERY_CHECK_NULL(page, code, lino, _end, terrno);
code = blockDataToBuf(page, pBlock);
QUERY_CHECK_CODE(code, lino, _end);
setBufPageDirty(page, true);
// save page info
pageInfo.expireTimeMs = pSource->currentExpireTimeMs;
code = tdListAppend(pSource->pageInfoList, &pageInfo);
QUERY_CHECK_CODE(code, lino, _end);
releaseBufPage(pSource->pBuf, page);
page = NULL;
blockDataCleanup(pBlock);
(*pSource->pTotalPages)++;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (page != NULL) {
dBufSetBufPageRecycled(pSource->pBuf, page);
}
return code;
}
static int32_t svmSourceAddBlock(SStreamVtableMergeSource* pSource, SSDataBlock* pDataBlock, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
int32_t pageSize = 0;
int32_t holdSize = 0;
SSDataBlock* pInputDataBlock = NULL;
QUERY_CHECK_NULL(pDataBlock, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pSource, code, lino, _end, TSDB_CODE_INVALID_PARA);
pInputDataBlock = pSource->pInputDataBlock;
if (pInputDataBlock == NULL) {
code = createOneDataBlock(pDataBlock, false, &pInputDataBlock);
QUERY_CHECK_CODE(code, lino, _end);
pSource->pInputDataBlock = pInputDataBlock;
}
int32_t start = 0;
int32_t nrows = blockDataGetNumOfRows(pDataBlock);
while (start < nrows) {
int32_t holdSize = blockDataGetSize(pInputDataBlock);
QUERY_CHECK_CONDITION(holdSize < pageSize, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
int32_t stop = 0;
code = blockDataSplitRows(pDataBlock, pDataBlock->info.hasVarCol, start, &stop, pageSize - holdSize);
QUERY_CHECK_CODE(code, lino, _end);
if (stop == start - 1) {
// If pInputDataBlock cannot hold new rows, ignore the error and write pInputDataBlock to the buffer
} else {
// append new rows to pInputDataBlock
if (blockDataGetNumOfRows(pInputDataBlock) == 0) {
// set expires time for the first block
pSource->currentExpireTimeMs = taosGetTimestampMs() + tsStreamVirtualMergeMaxDelayMs;
}
int32_t numOfRows = stop - start + 1;
code = blockDataMergeNRows(pInputDataBlock, pDataBlock, start, numOfRows);
QUERY_CHECK_CODE(code, lino, _end);
}
start = stop + 1;
if (stop == nrows - 1) {
break;
} else {
// pInputDataBlock is full, write it to the buffer
code = svmSourceFlushInput(pSource, idstr);
QUERY_CHECK_CODE(code, lino, _end);
}
}
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
return code;
}
static bool svmSourceIsEmpty(SStreamVtableMergeSource* pSource) { return listNEles(pSource->pageInfoList) == 0; }
static int32_t svmSourceReadBuf(SStreamVtableMergeSource* pSource, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SListNode* pn = NULL;
SVMBufPageInfo* pageInfo = NULL;
void* page = NULL;
QUERY_CHECK_NULL(pSource, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_CONDITION(!svmSourceIsEmpty(pSource), code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pSource->pOutputDataBlock, code, lino, _end, TSDB_CODE_INVALID_PARA);
blockDataCleanup(pSource->pOutputDataBlock);
pn = tdListGetHead(pSource->pageInfoList);
QUERY_CHECK_NULL(pn, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
pageInfo = (SVMBufPageInfo*)pn->data;
QUERY_CHECK_NULL(pageInfo, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
page = getBufPage(pSource->pBuf, pageInfo->pageId);
QUERY_CHECK_NULL(page, code, lino, _end, terrno);
code = blockDataFromBuf(pSource->pOutputDataBlock, page);
QUERY_CHECK_CODE(code, lino, _end);
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (page) {
releaseBufPage(pSource->pBuf, page);
}
return code;
}
static int32_t svmSourceCurrentTs(SStreamVtableMergeSource* pSource, const char* idstr, int64_t* pTs) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SColumnInfoData* tsCol = NULL;
QUERY_CHECK_NULL(pSource, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_CONDITION(!svmSourceIsEmpty(pSource), code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pSource->pOutputDataBlock, code, lino, _end, TSDB_CODE_INVALID_PARA);
if (blockDataGetNumOfRows(pSource->pOutputDataBlock) == 0) {
code = svmSourceReadBuf(pSource, idstr);
QUERY_CHECK_CODE(code, lino, _end);
}
QUERY_CHECK_CONDITION(pSource->rowIndex < blockDataGetNumOfRows(pSource->pOutputDataBlock), code, lino, _end,
TSDB_CODE_INVALID_PARA);
tsCol = taosArrayGet(pSource->pOutputDataBlock->pDataBlock, 0);
QUERY_CHECK_NULL(tsCol, code, lino, _end, terrno);
*pTs = ((int64_t*)tsCol->pData)[pSource->rowIndex];
pSource->latestTs = TMAX(*pTs, pSource->latestTs);
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
return code;
}
static int32_t svmSourceMoveNext(SStreamVtableMergeSource* pSource, const char* idstr, SVM_NEXT_RESULT* pRes) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SListNode* pn = NULL;
void* page = NULL;
int64_t latestTs = 0;
QUERY_CHECK_NULL(pSource, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pSource->pOutputDataBlock, code, lino, _end, TSDB_CODE_INVALID_PARA);
*pRes = SVM_NEXT_NOT_READY;
latestTs = pSource->latestTs;
while (true) {
if (svmSourceIsEmpty(pSource)) {
pSource->rowIndex = 0;
break;
}
QUERY_CHECK_CONDITION(pSource->rowIndex < blockDataGetNumOfRows(pSource->pOutputDataBlock), code, lino, _end,
TSDB_CODE_INVALID_PARA);
pSource->rowIndex++;
if (pSource->rowIndex >= blockDataGetNumOfRows(pSource->pOutputDataBlock)) {
// Pop the page from the list and recycle it
pn = tdListPopHead(pSource->pageInfoList);
QUERY_CHECK_NULL(pn, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
QUERY_CHECK_NULL(pn->data, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
SVMBufPageInfo* pageInfo = (SVMBufPageInfo*)pn->data;
page = getBufPage(pSource->pBuf, pageInfo->pageId);
QUERY_CHECK_NULL(page, code, lino, _end, terrno);
code = dBufSetBufPageRecycled(pSource->pBuf, page);
QUERY_CHECK_CODE(code, lino, _end);
(*pSource->pTotalPages)--;
taosMemoryFreeClear(pn);
pSource->rowIndex = 0;
}
if (svmSourceIsEmpty(pSource)) {
pSource->rowIndex = 0;
break;
}
int64_t ts = 0;
code = svmSourceCurrentTs(pSource, idstr, &ts);
QUERY_CHECK_CODE(code, lino, _end);
if (ts > latestTs && ts >= *pSource->pGlobalLatestTs) {
*pRes = SVM_NEXT_FOUND;
break;
}
}
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
return code;
}
static int32_t svmSourceCompare(const void* pLeft, const void* pRight, void* param) {
int32_t left = *(int32_t*)pLeft;
int32_t right = *(int32_t*)pRight;
int32_t code = TSDB_CODE_SUCCESS;
SArray* pValidSources = param;
SStreamVtableMergeSource* pLeftSource = *(SStreamVtableMergeSource**)taosArrayGet(pValidSources, left);
SStreamVtableMergeSource* pRightSource = *(SStreamVtableMergeSource**)taosArrayGet(pValidSources, right);
int64_t leftTs = 0;
code = svmSourceCurrentTs(pLeftSource, "", &leftTs);
if (code != TSDB_CODE_SUCCESS) {
return -1;
}
int64_t rightTs = 0;
code = svmSourceCurrentTs(pRightSource, "", &rightTs);
if (code != TSDB_CODE_SUCCESS) {
return 1;
}
if (leftTs < rightTs) {
return -1;
} else if (leftTs > rightTs) {
return 1;
} else {
return 0;
}
}
static SStreamVtableMergeSource* svmAddSource(SStreamVtableMergeHandle* pHandle, int64_t uid, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SStreamVtableMergeSource* pSource = NULL;
pSource = taosMemoryCalloc(1, sizeof(SStreamVtableMergeSource));
QUERY_CHECK_NULL(pSource, code, lino, _end, terrno);
pSource->pBuf = pHandle->pBuf;
pSource->pTotalPages = &pHandle->numOfPages;
pSource->pageInfoList = tdListNew(sizeof(SVMBufPageInfo));
QUERY_CHECK_NULL(pSource->pageInfoList, code, lino, _end, terrno);
code = createOneDataBlock(pHandle->datablock, false, &pSource->pOutputDataBlock);
QUERY_CHECK_CODE(code, lino, _end);
pSource->latestTs = INT64_MIN;
pSource->pGlobalLatestTs = &pHandle->globalLatestTs;
code = taosHashPut(pHandle->pSources, &uid, sizeof(uid), &pSource, POINTER_BYTES);
QUERY_CHECK_CODE(code, lino, _end);
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
svmSourceDestroy(&pSource);
pSource = NULL;
terrno = code;
}
return pSource;
}
static void svmDestroyTree(void* ptr) {
SMultiwayMergeTreeInfo** ppTree = ptr;
if (ppTree == NULL || *ppTree == NULL) {
return;
}
SMultiwayMergeTreeInfo* pTree = *ppTree;
if (pTree->param != NULL) {
taosArrayDestroy(pTree->param);
pTree->param = NULL;
}
tMergeTreeDestroy(ppTree);
}
static int32_t svmBuildTree(SStreamVtableMergeHandle* pHandle, SVM_NEXT_RESULT* pRes, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SArray* pReadySources = NULL;
void* pIter = NULL;
void* px = NULL;
int64_t globalExpireTimeMs = INT64_MAX;
QUERY_CHECK_NULL(pHandle, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pRes, code, lino, _end, TSDB_CODE_INVALID_PARA);
*pRes = SVM_NEXT_NOT_READY;
// find all non-empty sources
pReadySources = taosArrayInit(pHandle->nSrcTbls, POINTER_BYTES);
pIter = taosHashIterate(pHandle->pSources, NULL);
while (pIter != NULL) {
SStreamVtableMergeSource* pSource = *(SStreamVtableMergeSource**)pIter;
if (svmSourceIsEmpty(pSource)) {
code = svmSourceFlushInput(pSource, idstr);
QUERY_CHECK_CODE(code, lino, _end);
}
if (!svmSourceIsEmpty(pSource)) {
px = taosArrayPush(pReadySources, &pSource);
QUERY_CHECK_NULL(px, code, lino, _end, terrno);
globalExpireTimeMs = TMIN(globalExpireTimeMs, pSource->currentExpireTimeMs);
}
pIter = taosHashIterate(pHandle->pSources, pIter);
}
if (taosArrayGetSize(pReadySources) == 0) {
// no available sources
goto _end;
}
if (taosArrayGetSize(pReadySources) < pHandle->nSrcTbls) {
// some sources are still empty
switch (tsStreamVirtualMergeWaitMode) {
case STREAM_VIRTUAL_MERGE_WAIT_FOREVER:
goto _end; // wait forever
case STREAM_VIRTUAL_MERGE_MAX_DELAY:
if (taosGetTimestampMs() < globalExpireTimeMs) {
goto _end; // wait for max delay
}
break;
case STREAM_VIRTUAL_MERGE_MAX_MEMORY:
if (pHandle->numOfPages < pHandle->numPageLimit) {
goto _end; // wait for max memory
}
break;
};
}
void* param = NULL;
code = tMergeTreeCreate(&pHandle->pMergeTree, taosArrayGetSize(pReadySources), pReadySources, svmSourceCompare);
QUERY_CHECK_CODE(code, lino, _end);
pReadySources = NULL;
*pRes = SVM_NEXT_FOUND;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (pReadySources != NULL) {
taosArrayDestroy(pReadySources);
}
if (pIter != NULL) {
taosHashCancelIterate(pHandle->pSources, pIter);
}
return code;
}
int32_t streamVtableMergeAddBlock(SStreamVtableMergeHandle* pHandle, SSDataBlock* pDataBlock, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
int64_t pTbUid = 0;
void* px = 0;
SStreamVtableMergeSource* pSource = NULL;
QUERY_CHECK_NULL(pHandle, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pDataBlock, code, lino, _end, TSDB_CODE_INVALID_PARA);
pTbUid = pDataBlock->info.id.uid;
px = taosHashGet(pHandle->pSources, &pTbUid, sizeof(int64_t));
if (px == NULL) {
if (taosHashGetSize(pHandle->pSources) >= pHandle->nSrcTbls) {
qError("Number of source tables exceeded the limit %d, table uid: %" PRId64, pHandle->nSrcTbls, pTbUid);
code = TSDB_CODE_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
// try to allocate a new source
pSource = svmAddSource(pHandle, pTbUid, idstr);
QUERY_CHECK_NULL(pSource, code, lino, _end, terrno);
} else {
pSource = *(SStreamVtableMergeSource**)px;
QUERY_CHECK_NULL(pSource, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
}
code = svmSourceAddBlock(pSource, pDataBlock, idstr);
QUERY_CHECK_CODE(code, lino, _end);
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
return code;
}
int32_t streamVtableMergeNextTuple(SStreamVtableMergeHandle* pHandle, SSDataBlock* pResBlock, SVM_NEXT_RESULT* pRes,
const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
void* px = NULL;
SArray* pReadySources = NULL;
SStreamVtableMergeSource* pSource = NULL;
QUERY_CHECK_NULL(pHandle, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pResBlock, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pRes, code, lino, _end, TSDB_CODE_INVALID_PARA);
*pRes = SVM_NEXT_NOT_READY;
if (pHandle->pMergeTree == NULL) {
SVM_NEXT_RESULT buildRes = SVM_NEXT_NOT_READY;
code = svmBuildTree(pHandle, &buildRes, idstr);
QUERY_CHECK_CODE(code, lino, _end);
if (buildRes == SVM_NEXT_NOT_READY) {
goto _end;
}
}
QUERY_CHECK_NULL(pHandle->pMergeTree, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
int32_t idx = tMergeTreeGetChosenIndex(pHandle->pMergeTree);
pReadySources = pHandle->pMergeTree->param;
px = taosArrayGet(pReadySources, idx);
QUERY_CHECK_NULL(px, code, lino, _end, terrno);
pSource = *(SStreamVtableMergeSource**)px;
code = blockCopyOneRow(pSource->pOutputDataBlock, pSource->rowIndex, &pResBlock);
QUERY_CHECK_CODE(code, lino, _end);
*pRes = SVM_NEXT_FOUND;
pHandle->globalLatestTs = TMAX(pSource->latestTs, pHandle->globalLatestTs);
SVM_NEXT_RESULT nextRes = SVM_NEXT_NOT_READY;
int32_t origNumOfPages = pHandle->numOfPages;
code = svmSourceMoveNext(pSource, idstr, &nextRes);
QUERY_CHECK_CODE(code, lino, _end);
bool needDestroy = false;
if (nextRes == SVM_NEXT_NOT_READY) {
needDestroy = true;
} else if (taosArrayGetSize((SArray*)pHandle->pMergeTree->param) != pHandle->nSrcTbls &&
pHandle->numOfPages != origNumOfPages) {
// The original data for this portion is incomplete. Its merge was forcibly triggered by certain conditions, so we
// must recheck if those conditions are still met.
if (tsStreamVirtualMergeWaitMode == STREAM_VIRTUAL_MERGE_MAX_DELAY) {
int64_t globalExpireTimeMs = INT64_MAX;
for (int32_t i = 0; i < taosArrayGetSize(pReadySources); ++i) {
px = taosArrayGet(pReadySources, i);
QUERY_CHECK_NULL(px, code, lino, _end, TSDB_CODE_INTERNAL_ERROR);
pSource = *(SStreamVtableMergeSource**)px;
globalExpireTimeMs = TMIN(globalExpireTimeMs, pSource->currentExpireTimeMs);
}
needDestroy = taosGetTimestampMs() < globalExpireTimeMs;
} else if (tsStreamVirtualMergeWaitMode == STREAM_VIRTUAL_MERGE_MAX_MEMORY) {
needDestroy = pHandle->numOfPages < pHandle->numPageLimit;
} else {
code = TSDB_CODE_INTERNAL_ERROR;
QUERY_CHECK_CODE(code, lino, _end);
}
}
if (needDestroy) {
svmDestroyTree(&pHandle->pMergeTree);
}
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
return code;
}
int32_t streamVtableMergeCreateHandle(SStreamVtableMergeHandle** ppHandle, int32_t nSrcTbls, int32_t numPageLimit,
SDiskbasedBuf* pBuf, SSDataBlock* pResBlock, const char* idstr) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
SStreamVtableMergeHandle* pHandle = NULL;
QUERY_CHECK_NULL(ppHandle, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_CONDITION(nSrcTbls > 0, code, lino, _end, TSDB_CODE_INVALID_PARA);
QUERY_CHECK_NULL(pBuf, code, lino, _end, TSDB_CODE_INVALID_PARA);
*ppHandle = NULL;
pHandle = taosMemoryCalloc(1, sizeof(SStreamVtableMergeHandle));
QUERY_CHECK_NULL(pHandle, code, lino, _end, terrno);
pHandle->pBuf = pBuf;
pHandle->numPageLimit = numPageLimit;
pHandle->nSrcTbls = nSrcTbls;
pHandle->pSources = taosHashInit(8, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT), false, HASH_NO_LOCK);
QUERY_CHECK_NULL(pHandle->pSources, code, lino, _end, terrno);
taosHashSetFreeFp(pHandle->pSources, svmSourceDestroy);
code = createOneDataBlock(pResBlock, false, &pHandle->datablock);
QUERY_CHECK_CODE(code, lino, _end);
pHandle->globalLatestTs = INT64_MIN;
*ppHandle = pHandle;
pHandle = NULL;
_end:
if (code != TSDB_CODE_SUCCESS) {
qError("%s failed at line %d since %s, id: %s", __func__, lino, tstrerror(code), idstr);
}
if (pHandle != NULL) {
streamVtableMergeDestroyHandle(&pHandle);
}
return code;
}
void streamVtableMergeDestroyHandle(SStreamVtableMergeHandle** ppHandle) {
if (ppHandle == NULL || *ppHandle == NULL) {
return;
}
SStreamVtableMergeHandle* pHandle = *ppHandle;
if (pHandle->pSources != NULL) {
taosHashCleanup(pHandle->pSources);
pHandle->pSources = NULL;
}
svmDestroyTree(&pHandle->pMergeTree);
taosMemoryFreeClear(*ppHandle);
}

279
source/libs/executor/src/sysscanoperator.c
View File

@ -156,7 +156,7 @@ static int32_t sysTableUserTagsFillOneTableTags(const SSysTableScanInfo* pInfo,
static int32_t sysTableUserColsFillOneTableCols(const SSysTableScanInfo* pInfo, const char* dbname, int32_t* pNumOfRows,
const SSDataBlock* dataBlock, char* tName, SSchemaWrapper* schemaRow,
char* tableType);
char* tableType, SColRefWrapper *colRef);
static void relocateAndFilterSysTagsScanResult(SSysTableScanInfo* pInfo, int32_t numOfRows, SSDataBlock* dataBlock,
SFilterInfo* pFilterInfo, SExecTaskInfo* pTaskInfo);
@ -509,15 +509,23 @@ static SSDataBlock* doOptimizeTableNameFilter(SOperatorInfo* pOperator, SSDataBl
char typeName[TSDB_TABLE_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
SSchemaWrapper* schemaRow = NULL;
SColRefWrapper* colRef = NULL;
if (smrTable.me.type == TSDB_SUPER_TABLE) {
schemaRow = &smrTable.me.stbEntry.schemaRow;
STR_TO_VARSTR(typeName, "CHILD_TABLE");
} else if (smrTable.me.type == TSDB_NORMAL_TABLE) {
schemaRow = &smrTable.me.ntbEntry.schemaRow;
STR_TO_VARSTR(typeName, "NORMAL_TABLE");
} else if (smrTable.me.type == TSDB_VIRTUAL_NORMAL_TABLE) {
schemaRow = &smrTable.me.ntbEntry.schemaRow;
colRef = &smrTable.me.colRef;
STR_TO_VARSTR(typeName, "VIRTUAL_NORMAL_TABLE");
} else if (smrTable.me.type == TSDB_VIRTUAL_CHILD_TABLE) {
colRef = &smrTable.me.colRef;
STR_TO_VARSTR(typeName, "VIRTUAL_CHILD_TABLE");
}
code = sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, dataBlock, tableName, schemaRow, typeName);
code = sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, dataBlock, tableName, schemaRow, typeName, colRef);
if (code != TSDB_CODE_SUCCESS) {
pAPI->metaReaderFn.clearReader(&smrTable);
pInfo->loadInfo.totalRows = 0;
@ -623,6 +631,7 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
char tableName[TSDB_TABLE_NAME_LEN + VARSTR_HEADER_SIZE] = {0};
SSchemaWrapper* schemaRow = NULL;
SColRefWrapper* colRef = NULL;
if (pInfo->pCur->mr.me.type == TSDB_SUPER_TABLE) {
qDebug("sysTableScanUserCols cursor get super table, %s", GET_TASKID(pTaskInfo));
@ -684,6 +693,53 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
schemaRow = &pInfo->pCur->mr.me.ntbEntry.schemaRow;
STR_TO_VARSTR(typeName, "NORMAL_TABLE");
STR_TO_VARSTR(tableName, pInfo->pCur->mr.me.name);
} else if (pInfo->pCur->mr.me.type == TSDB_VIRTUAL_NORMAL_TABLE) {
qDebug("sysTableScanUserCols cursor get virtual normal table, %s", GET_TASKID(pTaskInfo));
schemaRow = &pInfo->pCur->mr.me.ntbEntry.schemaRow;
STR_TO_VARSTR(typeName, "VIRTUAL_NORMAL_TABLE");
STR_TO_VARSTR(tableName, pInfo->pCur->mr.me.name);
colRef = &pInfo->pCur->mr.me.colRef;
} else if (pInfo->pCur->mr.me.type == TSDB_VIRTUAL_CHILD_TABLE) {
qDebug("sysTableScanUserCols cursor get virtual child table, %s", GET_TASKID(pTaskInfo));
STR_TO_VARSTR(typeName, "VIRTUAL_CHILD_TABLE");
STR_TO_VARSTR(tableName, pInfo->pCur->mr.me.name);
colRef = &pInfo->pCur->mr.me.colRef;
int64_t suid = pInfo->pCur->mr.me.ctbEntry.suid;
void* schema = taosHashGet(pInfo->pSchema, &pInfo->pCur->mr.me.ctbEntry.suid, sizeof(int64_t));
if (schema != NULL) {
schemaRow = *(SSchemaWrapper**)schema;
} else {
SMetaReader smrSuperTable = {0};
pAPI->metaReaderFn.initReader(&smrSuperTable, pInfo->readHandle.vnode, META_READER_NOLOCK, &pAPI->metaFn);
code = pAPI->metaReaderFn.getTableEntryByUid(&smrSuperTable, suid);
if (code != TSDB_CODE_SUCCESS) {
// terrno has been set by pAPI->metaReaderFn.getTableEntryByName, therefore, return directly
qError("sysTableScanUserCols get meta by suid:%" PRId64 " error, code:%d, %s", suid, code,
GET_TASKID(pTaskInfo));
pAPI->metaReaderFn.clearReader(&smrSuperTable);
blockDataDestroy(pDataBlock);
pInfo->loadInfo.totalRows = 0;
return NULL;
}
SSchemaWrapper* schemaWrapper = tCloneSSchemaWrapper(&smrSuperTable.me.stbEntry.schemaRow);
if (smrSuperTable.me.stbEntry.schemaRow.pSchema) {
if (schemaWrapper == NULL) {
code = terrno;
lino = __LINE__;
pAPI->metaReaderFn.clearReader(&smrSuperTable);
goto _end;
}
}
code = taosHashPut(pInfo->pSchema, &suid, sizeof(int64_t), &schemaWrapper, POINTER_BYTES);
if (code == TSDB_CODE_DUP_KEY) {
code = TSDB_CODE_SUCCESS;
}
schemaRow = schemaWrapper;
pAPI->metaReaderFn.clearReader(&smrSuperTable);
QUERY_CHECK_CODE(code, lino, _end);
}
} else {
qDebug("sysTableScanUserCols cursor get invalid table, %s", GET_TASKID(pTaskInfo));
continue;
@ -699,7 +755,7 @@ static SSDataBlock* sysTableScanUserCols(SOperatorInfo* pOperator) {
}
}
// if pInfo->pRes->info.rows == 0, also need to add the meta to pDataBlock
code = sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, pDataBlock, tableName, schemaRow, typeName);
code = sysTableUserColsFillOneTableCols(pInfo, dbname, &numOfRows, pDataBlock, tableName, schemaRow, typeName, colRef);
QUERY_CHECK_CODE(code, lino, _end);
}
@ -782,7 +838,7 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
return NULL;
}
if (smrChildTable.me.type != TSDB_CHILD_TABLE) {
if (smrChildTable.me.type != TSDB_CHILD_TABLE && smrChildTable.me.type != TSDB_VIRTUAL_CHILD_TABLE) {
pAPI->metaReaderFn.clearReader(&smrChildTable);
blockDataDestroy(dataBlock);
pInfo->loadInfo.totalRows = 0;
@ -828,7 +884,7 @@ static SSDataBlock* sysTableScanUserTags(SOperatorInfo* pOperator) {
}
while ((ret = pAPI->metaFn.cursorNext(pInfo->pCur, TSDB_SUPER_TABLE)) == 0) {
if (pInfo->pCur->mr.me.type != TSDB_CHILD_TABLE) {
if (pInfo->pCur->mr.me.type != TSDB_CHILD_TABLE && pInfo->pCur->mr.me.type != TSDB_VIRTUAL_CHILD_TABLE) {
continue;
}
@ -1181,7 +1237,7 @@ _end:
static int32_t sysTableUserColsFillOneTableCols(const SSysTableScanInfo* pInfo, const char* dbname, int32_t* pNumOfRows,
const SSDataBlock* dataBlock, char* tName, SSchemaWrapper* schemaRow,
char* tableType) {
char* tableType, SColRefWrapper *colRef) {
int32_t code = TSDB_CODE_SUCCESS;
int32_t lino = 0;
if (schemaRow == NULL) {
@ -1252,6 +1308,25 @@ static int32_t sysTableUserColsFillOneTableCols(const SSysTableScanInfo* pInfo,
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, numOfRows);
}
// col data source
pColInfoData = taosArrayGet(dataBlock->pDataBlock, 9);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
if (!colRef || !colRef->pColRef[i].hasRef) {
colDataSetNULL(pColInfoData, numOfRows);
} else {
char refColName[TSDB_DB_NAME_LEN + TSDB_NAME_DELIMITER_LEN + TSDB_COL_FNAME_LEN + VARSTR_HEADER_SIZE] = {0};
char tmpColName[TSDB_DB_NAME_LEN + TSDB_NAME_DELIMITER_LEN + TSDB_COL_FNAME_LEN] = {0};
strcat(tmpColName, colRef->pColRef[i].refDbName);
strcat(tmpColName, ".");
strcat(tmpColName, colRef->pColRef[i].refTableName);
strcat(tmpColName, ".");
strcat(tmpColName, colRef->pColRef[i].refColName);
STR_TO_VARSTR(refColName, tmpColName);
code = colDataSetVal(pColInfoData, numOfRows, (char *)refColName, false);
QUERY_CHECK_CODE(code, lino, _end);
}
++numOfRows;
}
@ -1560,6 +1635,104 @@ static int32_t doSetUserTableMetaInfo(SStoreMetaReader* pMetaReaderFn, SStoreMet
STR_TO_VARSTR(n, "NORMAL_TABLE");
// impl later
} else if (tableType == TSDB_VIRTUAL_NORMAL_TABLE) {
// create time
pColInfoData = taosArrayGet(p->pDataBlock, 2);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, rowIndex, (char*)&pMReader->me.ntbEntry.btime, false);
QUERY_CHECK_CODE(code, lino, _end);
// number of columns
pColInfoData = taosArrayGet(p->pDataBlock, 3);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, rowIndex, (char*)&pMReader->me.ntbEntry.schemaRow.nCols, false);
QUERY_CHECK_CODE(code, lino, _end);
// super table name
pColInfoData = taosArrayGet(p->pDataBlock, 4);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, rowIndex);
// table comment
pColInfoData = taosArrayGet(p->pDataBlock, 8);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, rowIndex);
// uid
pColInfoData = taosArrayGet(p->pDataBlock, 5);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, rowIndex, (char*)&pMReader->me.uid, false);
QUERY_CHECK_CODE(code, lino, _end);
// ttl
pColInfoData = taosArrayGet(p->pDataBlock, 7);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, rowIndex);
STR_TO_VARSTR(n, "VIRTUAL_NORMAL_TABLE");
// impl later
} else if (tableType == TSDB_VIRTUAL_CHILD_TABLE) {
// create time
int64_t ts = pMReader->me.ctbEntry.btime;
pColInfoData = taosArrayGet(p->pDataBlock, 2);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, rowIndex, (char*)&ts, false);
QUERY_CHECK_CODE(code, lino, _end);
SMetaReader mr1 = {0};
pMetaReaderFn->initReader(&mr1, pVnode, META_READER_NOLOCK, pMetaFn);
int64_t suid = pMReader->me.ctbEntry.suid;
code = pMetaReaderFn->getTableEntryByUid(&mr1, suid);
if (code != TSDB_CODE_SUCCESS) {
qError("failed to get super table meta, cname:%s, suid:0x%" PRIx64 ", code:%s, %s", pMReader->me.name, suid,
tstrerror(code), idStr);
pMetaReaderFn->clearReader(&mr1);
QUERY_CHECK_CODE(code, lino, _end);
}
pColInfoData = taosArrayGet(p->pDataBlock, 3);
if (pColInfoData == NULL) {
pMetaReaderFn->clearReader(&mr1);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
}
code = colDataSetVal(pColInfoData, rowIndex, (char*)&mr1.me.stbEntry.schemaRow.nCols, false);
if (code != 0) {
pMetaReaderFn->clearReader(&mr1);
QUERY_CHECK_CODE(code, lino, _end);
}
// super table name
STR_TO_VARSTR(n, mr1.me.name);
pColInfoData = taosArrayGet(p->pDataBlock, 4);
if (pColInfoData == NULL) {
pMetaReaderFn->clearReader(&mr1);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
}
code = colDataSetVal(pColInfoData, rowIndex, n, false);
pMetaReaderFn->clearReader(&mr1);
QUERY_CHECK_CODE(code, lino, _end);
// table comment
pColInfoData = taosArrayGet(p->pDataBlock, 8);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, rowIndex);
// uid
pColInfoData = taosArrayGet(p->pDataBlock, 5);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, rowIndex, (char*)&pMReader->me.uid, false);
QUERY_CHECK_CODE(code, lino, _end);
// ttl
pColInfoData = taosArrayGet(p->pDataBlock, 7);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, rowIndex);
STR_TO_VARSTR(n, "VIRTUAL_CHILD_TABLE");
}
_end:
@ -1866,6 +2039,100 @@ static SSDataBlock* sysTableBuildUserTables(SOperatorInfo* pOperator) {
QUERY_CHECK_CODE(code, lino, _end);
STR_TO_VARSTR(n, "NORMAL_TABLE");
} else if (tableType == TSDB_VIRTUAL_NORMAL_TABLE) {
// create time
pColInfoData = taosArrayGet(p->pDataBlock, 2);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.ntbEntry.btime, false);
QUERY_CHECK_CODE(code, lino, _end);
// number of columns
pColInfoData = taosArrayGet(p->pDataBlock, 3);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.ntbEntry.schemaRow.nCols, false);
QUERY_CHECK_CODE(code, lino, _end);
// super table name
pColInfoData = taosArrayGet(p->pDataBlock, 4);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, numOfRows);
// table comment
pColInfoData = taosArrayGet(p->pDataBlock, 8);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, numOfRows);
// uid
pColInfoData = taosArrayGet(p->pDataBlock, 5);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.uid, false);
QUERY_CHECK_CODE(code, lino, _end);
// ttl
pColInfoData = taosArrayGet(p->pDataBlock, 7);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, numOfRows);
STR_TO_VARSTR(n, "VIRTUAL_NORMAL_TABLE");
} else if (tableType == TSDB_VIRTUAL_CHILD_TABLE) {
// create time
int64_t ts = pInfo->pCur->mr.me.ctbEntry.btime;
pColInfoData = taosArrayGet(p->pDataBlock, 2);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, (char*)&ts, false);
QUERY_CHECK_CODE(code, lino, _end);
SMetaReader mr = {0};
pAPI->metaReaderFn.initReader(&mr, pInfo->readHandle.vnode, META_READER_NOLOCK, &pAPI->metaFn);
uint64_t suid = pInfo->pCur->mr.me.ctbEntry.suid;
code = pAPI->metaReaderFn.getTableEntryByUid(&mr, suid);
if (code != TSDB_CODE_SUCCESS) {
qError("failed to get super table meta, cname:%s, suid:0x%" PRIx64 ", code:%s, %s", pInfo->pCur->mr.me.name,
suid, tstrerror(terrno), GET_TASKID(pTaskInfo));
pAPI->metaReaderFn.clearReader(&mr);
pAPI->metaFn.closeTableMetaCursor(pInfo->pCur);
pInfo->pCur = NULL;
blockDataDestroy(p);
T_LONG_JMP(pTaskInfo->env, terrno);
}
if (isTsmaResSTb(mr.me.name)) {
pAPI->metaReaderFn.clearReader(&mr);
continue;
}
// number of columns
pColInfoData = taosArrayGet(p->pDataBlock, 3);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, (char*)&mr.me.stbEntry.schemaRow.nCols, false);
QUERY_CHECK_CODE(code, lino, _end);
// super table name
STR_TO_VARSTR(n, mr.me.name);
pColInfoData = taosArrayGet(p->pDataBlock, 4);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, n, false);
QUERY_CHECK_CODE(code, lino, _end);
pAPI->metaReaderFn.clearReader(&mr);
// table comment
pColInfoData = taosArrayGet(p->pDataBlock, 8);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, numOfRows);
// uid
pColInfoData = taosArrayGet(p->pDataBlock, 5);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
code = colDataSetVal(pColInfoData, numOfRows, (char*)&pInfo->pCur->mr.me.uid, false);
QUERY_CHECK_CODE(code, lino, _end);
// ttl
pColInfoData = taosArrayGet(p->pDataBlock, 7);
QUERY_CHECK_NULL(pColInfoData, code, lino, _end, terrno);
colDataSetNULL(pColInfoData, numOfRows);
STR_TO_VARSTR(n, "VIRTUAL_CHILD_TABLE");
}
pColInfoData = taosArrayGet(p->pDataBlock, 9);

5
source/libs/executor/src/tsort.c
View File

@ -2929,6 +2929,11 @@ void tsortGetValue(STupleHandle* pVHandle, int32_t colIndex, void** pVal) {
}
}
void tsortGetColumnInfo(STupleHandle* pVHandle, int32_t colIndex, SColumnInfoData** pColInfo) {
*pColInfo = TARRAY_GET_ELEM(pVHandle->pBlock->pDataBlock, colIndex);
}
size_t tsortGetColNum(STupleHandle* pVHandle) { return blockDataGetNumOfCols(pVHandle->pBlock); }
uint64_t tsortGetGroupId(STupleHandle* pVHandle) { return pVHandle->pBlock->info.id.groupId; }
void tsortGetBlockInfo(STupleHandle* pVHandle, SDataBlockInfo* pBlockInfo) { *pBlockInfo = pVHandle->pBlock->info; }

1354
source/libs/executor/src/virtualtablescanoperator.c Normal file
View File

File diff suppressed because it is too large Load Diff

4
source/libs/executor/test/queryPlanTests.cpp
View File

@ -3146,7 +3146,7 @@ void qptExecPlan(SReadHandle* pReadHandle, SNode* pNode, SExecTaskInfo* pTaskInf
case QUERY_NODE_PHYSICAL_PLAN_QUERY_INSERT:
case QUERY_NODE_PHYSICAL_PLAN_DELETE: {
DataSinkHandle handle = NULL;
qptCtx.result.code = dsCreateDataSinker(NULL, (SDataSinkNode**)&pNode, &handle, NULL, NULL);
qptCtx.result.code = dsCreateDataSinker(NULL, (SDataSinkNode**)&pNode, &handle, NULL, NULL, false);
dsDestroyDataSinker(handle);
break;
}
@ -3176,7 +3176,7 @@ void qptExecPlan(SReadHandle* pReadHandle, SNode* pNode, SExecTaskInfo* pTaskInf
qptCtx.result.code = createGroupCacheOperatorInfo(NULL, 0, (SGroupCachePhysiNode*)pNode, pTaskInfo, ppOperaotr);
break;
case QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL:
qptCtx.result.code = createDynQueryCtrlOperatorInfo(NULL, 0, (SDynQueryCtrlPhysiNode*)pNode, pTaskInfo, ppOperaotr);
qptCtx.result.code = createDynQueryCtrlOperatorInfo(NULL, 0, (SDynQueryCtrlPhysiNode*)pNode, pTaskInfo, pReadHandle, ppOperaotr);
break;
case QUERY_NODE_PHYSICAL_PLAN_MERGE_COUNT:
qptCtx.result.code = createCountwindowOperatorInfo(NULL, (SPhysiNode*)pNode, pTaskInfo, ppOperaotr);

2
source/libs/function/src/builtins.c
View File

@ -276,8 +276,6 @@ static SDataType* getSDataTypeFromNode(SNode* pNode) {
if (pNode == NULL) return NULL;
if (nodesIsExprNode(pNode)) {
return &((SExprNode*)pNode)->resType;
} else if (QUERY_NODE_COLUMN_REF == pNode->type) {
return &((SColumnRefNode*)pNode)->resType;
} else {
return NULL;
}

7
source/libs/function/src/functionMgt.c
View File

@ -254,6 +254,13 @@ bool fmIsLastRowFunc(int32_t funcId) {
return FUNCTION_TYPE_LAST_ROW == funcMgtBuiltins[funcId].type;
}
bool fmIsLastFunc(int32_t funcId) {
if (funcId < 0 || funcId >= funcMgtBuiltinsNum) {
return false;
}
return FUNCTION_TYPE_LAST == funcMgtBuiltins[funcId].type;
}
bool fmIsNotNullOutputFunc(int32_t funcId) {
if (funcId < 0 || funcId >= funcMgtBuiltinsNum) {
return false;

49
source/libs/nodes/src/nodesCloneFuncs.c
View File

@ -133,6 +133,11 @@ static int32_t columnNodeCopy(const SColumnNode* pSrc, SColumnNode* pDst) {
COPY_SCALAR_FIELD(numOfPKs);
COPY_SCALAR_FIELD(projRefIdx);
COPY_SCALAR_FIELD(resIdx);
COPY_SCALAR_FIELD(hasDep);
COPY_SCALAR_FIELD(hasRef);
COPY_CHAR_ARRAY_FIELD(refDbName);
COPY_CHAR_ARRAY_FIELD(refTableName);
COPY_CHAR_ARRAY_FIELD(refColName);
return TSDB_CODE_SUCCESS;
}
@ -467,6 +472,13 @@ static int32_t windowOffsetCopy(const SWindowOffsetNode* pSrc, SWindowOffsetNode
return TSDB_CODE_SUCCESS;
}
static int32_t virtualTableNodeCopy(const SVirtualTableNode * pSrc, SVirtualTableNode* pDst) {
COPY_BASE_OBJECT_FIELD(table, tableNodeCopy);
CLONE_OBJECT_FIELD(pMeta, tableMetaClone);
CLONE_OBJECT_FIELD(pVgroupList, vgroupsInfoClone);
CLONE_NODE_LIST_FIELD(refTables);
return TSDB_CODE_SUCCESS;
}
static int32_t logicNodeCopy(const SLogicNode* pSrc, SLogicNode* pDst) {
CLONE_NODE_LIST_FIELD(pTargets);
@ -530,6 +542,21 @@ static int32_t logicScanCopy(const SScanLogicNode* pSrc, SScanLogicNode* pDst) {
COPY_SCALAR_FIELD(smallDataTsSort);
COPY_SCALAR_FIELD(needSplit);
COPY_SCALAR_FIELD(noPseudoRefAfterGrp);
COPY_SCALAR_FIELD(virtualStableScan);
return TSDB_CODE_SUCCESS;
}
static int32_t logicVirtualScanCopy(const SVirtualScanLogicNode * pSrc, SVirtualScanLogicNode* pDst) {
COPY_BASE_OBJECT_FIELD(node, logicNodeCopy);
COPY_SCALAR_FIELD(scanAllCols);
CLONE_NODE_LIST_FIELD(pScanCols);
CLONE_NODE_LIST_FIELD(pScanPseudoCols);
COPY_SCALAR_FIELD(tableType);
COPY_SCALAR_FIELD(tableId);
COPY_SCALAR_FIELD(stableId);
CLONE_OBJECT_FIELD(pVgroupList, vgroupsInfoClone);
COPY_SCALAR_FIELD(scanType);
COPY_OBJECT_FIELD(tableName, sizeof(SName));
return TSDB_CODE_SUCCESS;
}
@ -744,6 +771,9 @@ static int32_t logicDynQueryCtrlCopy(const SDynQueryCtrlLogicNode* pSrc, SDynQue
CLONE_NODE_LIST_FIELD(stbJoin.pVgList);
CLONE_NODE_LIST_FIELD(stbJoin.pUidList);
COPY_OBJECT_FIELD(stbJoin.srcScan, sizeof(pDst->stbJoin.srcScan));
COPY_SCALAR_FIELD(vtbScan.scanAllCols);
COPY_SCALAR_FIELD(vtbScan.suid);
CLONE_OBJECT_FIELD(vtbScan.pVgroupList, vgroupsInfoClone);
return TSDB_CODE_SUCCESS;
}
@ -753,6 +783,7 @@ static int32_t logicSubplanCopy(const SLogicSubplan* pSrc, SLogicSubplan* pDst)
COPY_SCALAR_FIELD(subplanType);
COPY_SCALAR_FIELD(level);
COPY_SCALAR_FIELD(splitFlag);
COPY_SCALAR_FIELD(processOneBlock);
return TSDB_CODE_SUCCESS;
}
@ -779,6 +810,15 @@ static int32_t physiScanCopy(const SScanPhysiNode* pSrc, SScanPhysiNode* pDst) {
return TSDB_CODE_SUCCESS;
}
static int32_t physiVirtualTableScanCopy(const SVirtualScanPhysiNode* pSrc, SVirtualScanPhysiNode* pDst) {
COPY_BASE_OBJECT_FIELD(scan, physiScanCopy);
CLONE_NODE_LIST_FIELD(pGroupTags);
COPY_SCALAR_FIELD(groupSort);
COPY_SCALAR_FIELD(scanAllCols);
CLONE_NODE_LIST_FIELD(pTargets);
return TSDB_CODE_SUCCESS;
}
static int32_t physiTagScanCopy(const STagScanPhysiNode* pSrc, STagScanPhysiNode* pDst) {
COPY_BASE_OBJECT_FIELD(scan, physiScanCopy);
COPY_SCALAR_FIELD(onlyMetaCtbIdx);
@ -1067,6 +1107,9 @@ int32_t nodesCloneNode(const SNode* pNode, SNode** ppNode) {
case QUERY_NODE_WINDOW_OFFSET:
code = windowOffsetCopy((const SWindowOffsetNode*)pNode, (SWindowOffsetNode*)pDst);
break;
case QUERY_NODE_VIRTUAL_TABLE:
code = virtualTableNodeCopy((const SVirtualTableNode*)pNode, (SVirtualTableNode*)pDst);
break;
case QUERY_NODE_SET_OPERATOR:
code = setOperatorCopy((const SSetOperator*)pNode, (SSetOperator*)pDst);
break;
@ -1121,6 +1164,9 @@ int32_t nodesCloneNode(const SNode* pNode, SNode** ppNode) {
case QUERY_NODE_LOGIC_PLAN_DYN_QUERY_CTRL:
code = logicDynQueryCtrlCopy((const SDynQueryCtrlLogicNode*)pNode, (SDynQueryCtrlLogicNode*)pDst);
break;
case QUERY_NODE_LOGIC_PLAN_VIRTUAL_TABLE_SCAN:
code = logicVirtualScanCopy((const SVirtualScanLogicNode *)pNode, (SVirtualScanLogicNode*)pDst);
break;
case QUERY_NODE_LOGIC_SUBPLAN:
code = logicSubplanCopy((const SLogicSubplan*)pNode, (SLogicSubplan*)pDst);
break;
@ -1156,6 +1202,9 @@ int32_t nodesCloneNode(const SNode* pNode, SNode** ppNode) {
case QUERY_NODE_PHYSICAL_PLAN_STREAM_PARTITION:
code = physiPartitionCopy((const SPartitionPhysiNode*)pNode, (SPartitionPhysiNode*)pDst);
break;
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
code = physiVirtualTableScanCopy((const SVirtualScanPhysiNode*)pNode, (SVirtualScanPhysiNode*)pDst);
break;
case QUERY_NODE_PHYSICAL_PLAN_PROJECT:
code = physiProjectCopy((const SProjectPhysiNode*)pNode, (SProjectPhysiNode*)pDst);
break;

678
source/libs/nodes/src/nodesCodeFuncs.c
View File

@ -41,6 +41,8 @@ const char* nodesNodeName(ENodeType type) {
return "Function";
case QUERY_NODE_REAL_TABLE:
return "RealTable";
case QUERY_NODE_VIRTUAL_TABLE:
return "VirtualTable";
case QUERY_NODE_TEMP_TABLE:
return "TempTable";
case QUERY_NODE_JOIN_TABLE:
@ -87,6 +89,8 @@ const char* nodesNodeName(ENodeType type) {
return "StreamOptions";
case QUERY_NODE_LEFT_VALUE:
return "LeftValue";
case QUERY_NODE_COLUMN_REF:
return "ColumnReference";
case QUERY_NODE_WHEN_THEN:
return "WhenThen";
case QUERY_NODE_CASE_WHEN:
@ -123,6 +127,10 @@ const char* nodesNodeName(ENodeType type) {
return "CreateTableStmt";
case QUERY_NODE_CREATE_SUBTABLE_CLAUSE:
return "CreateSubtableClause";
case QUERY_NODE_CREATE_VIRTUAL_TABLE_STMT:
return "CreateVirtualtableStmt";
case QUERY_NODE_CREATE_VIRTUAL_SUBTABLE_STMT:
return "CreateVirtualsubtableStmt";
case QUERY_NODE_CREATE_MULTI_TABLES_STMT:
return "CreateMultiTableStmt";
case QUERY_NODE_DROP_TABLE_CLAUSE:
@ -131,10 +139,14 @@ const char* nodesNodeName(ENodeType type) {
return "DropTableStmt";
case QUERY_NODE_DROP_SUPER_TABLE_STMT:
return "DropSuperTableStmt";
case QUERY_NODE_DROP_VIRTUAL_TABLE_STMT:
return "DropVirtualTableStmt";
case QUERY_NODE_ALTER_TABLE_STMT:
return "AlterTableStmt";
case QUERY_NODE_ALTER_SUPER_TABLE_STMT:
return "AlterSuperTableStmt";
case QUERY_NODE_ALTER_VIRTUAL_TABLE_STMT:
return "AlterVirtualTableStmt";
case QUERY_NODE_CREATE_USER_STMT:
return "CreateUserStmt";
case QUERY_NODE_ALTER_USER_STMT:
@ -202,7 +214,6 @@ const char* nodesNodeName(ENodeType type) {
case QUERY_NODE_ASSIGN_LEADER_STMT:
return "AssignLeaderStmt";
case QUERY_NODE_BALANCE_VGROUP_LEADER_STMT:
return "BalanceVgroupLeaderStmt";
case QUERY_NODE_BALANCE_VGROUP_LEADER_DATABASE_STMT:
return "BalanceVgroupLeaderStmt";
case QUERY_NODE_MERGE_VGROUP_STMT:
@ -253,6 +264,8 @@ const char* nodesNodeName(ENodeType type) {
return "ShowStreamsStmt";
case QUERY_NODE_SHOW_TABLES_STMT:
return "ShowTablesStmt";
case QUERY_NODE_SHOW_VTABLES_STMT:
return "ShowVtablesStmt";
case QUERY_NODE_SHOW_TAGS_STMT:
return "ShowTagsStmt";
case QUERY_NODE_SHOW_USERS_STMT:
@ -284,6 +297,8 @@ const char* nodesNodeName(ENodeType type) {
return "ShowCreateDatabasesStmt";
case QUERY_NODE_SHOW_CREATE_TABLE_STMT:
return "ShowCreateTablesStmt";
case QUERY_NODE_SHOW_CREATE_VTABLE_STMT:
return "ShowCreateVtablesstmt";
case QUERY_NODE_SHOW_CREATE_STABLE_STMT:
return "ShowCreateStablesStmt";
case QUERY_NODE_SHOW_CREATE_VIEW_STMT:
@ -358,6 +373,8 @@ const char* nodesNodeName(ENodeType type) {
return "LogicGroupCache";
case QUERY_NODE_LOGIC_PLAN_DYN_QUERY_CTRL:
return "LogicDynamicQueryCtrl";
case QUERY_NODE_LOGIC_PLAN_VIRTUAL_TABLE_SCAN:
return "LogicVirtualTableScan";
case QUERY_NODE_LOGIC_SUBPLAN:
return "LogicSubplan";
case QUERY_NODE_LOGIC_PLAN:
@ -465,6 +482,8 @@ const char* nodesNodeName(ENodeType type) {
return "PhysiGroupCache";
case QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL:
return "PhysiDynamicQueryCtrl";
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
return "PhysiVirtualTableScan";
case QUERY_NODE_PHYSICAL_SUBPLAN:
return "PhysiSubplan";
case QUERY_NODE_PHYSICAL_PLAN:
@ -620,10 +639,60 @@ static int32_t jsonToSchema(const SJson* pJson, void* pObj) {
return code;
}
static const char* jkRefColHasRef = "HasRef";
static const char* jkRefColColId = "ColId";
static const char* jkRefColDbName = "DbName";
static const char* jkRefColTableName = "TableName";
static const char* jkRefColColName = "ColName";
static int32_t refColToJson(const void* pObj, SJson* pJson) {
const SColRef* pCol = (const SColRef*)pObj;
int32_t code = tjsonAddBoolToObject(pJson, jkRefColHasRef, pCol->hasRef);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkRefColColId, pCol->id);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkRefColDbName, pCol->refDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkRefColTableName, pCol->refTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkRefColColName, pCol->refColName);
}
return code;
}
static int32_t jsonToRefCol(const SJson* pJson, void* pObj) {
SColRef* pCol = (SColRef*)pObj;
int32_t code = tjsonGetBoolValue(pJson, jkRefColHasRef, &pCol->hasRef);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetSmallIntValue(pJson, jkRefColColId, &pCol->id);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkRefColDbName, pCol->refDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkRefColTableName, pCol->refTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkRefColColName, pCol->refColName);
}
return code;
}
static const char* jkTableMetaVgId = "VgId";
static const char* jkTableMetaTableType = "TableType";
static const char* jkTableMetaUid = "Uid";
static const char* jkTableMetaSuid = "Suid";
static const char* jkTableMetaColRefNum = "ColRefNum";
static const char* jkTableMetaRefCols = "RefCols";
static const char* jkTableMetaSversion = "Sversion";
static const char* jkTableMetaTversion = "Tversion";
static const char* jkTableMetaComInfo = "ComInfo";
@ -642,6 +711,13 @@ static int32_t tableMetaToJson(const void* pObj, SJson* pJson) {
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkTableMetaSuid, pNode->suid);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkTableMetaColRefNum, pNode->colRef ? pNode->numOfColRefs : 0);
}
if (TSDB_CODE_SUCCESS == code && pNode->numOfColRefs > 0 && pNode->colRef) {
code = tjsonAddArray(pJson, jkTableMetaRefCols, refColToJson, pNode->colRef, sizeof(SColRef),
pNode->numOfColRefs);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkTableMetaSversion, pNode->sversion);
}
@ -673,6 +749,9 @@ static int32_t jsonToTableMeta(const SJson* pJson, void* pObj) {
if (TSDB_CODE_SUCCESS == code) {
tjsonGetNumberValue(pJson, jkTableMetaSuid, pNode->suid, code);
}
if (TSDB_CODE_SUCCESS == code) {
tjsonGetNumberValue(pJson, jkTableMetaColRefNum, pNode->numOfColRefs, code);
}
if (TSDB_CODE_SUCCESS == code) {
tjsonGetNumberValue(pJson, jkTableMetaSversion, pNode->sversion, code);
}
@ -685,6 +764,10 @@ static int32_t jsonToTableMeta(const SJson* pJson, void* pObj) {
if (TSDB_CODE_SUCCESS == code) {
code = tjsonToArray(pJson, jkTableMetaColSchemas, jsonToSchema, pNode->schema, sizeof(SSchema));
}
if (TSDB_CODE_SUCCESS == code && pNode->numOfColRefs > 0) {
pNode->colRef = (SColRef*)((char*)(pNode + 1) + TABLE_TOTAL_COL_NUM(pNode) * sizeof(SSchema));
code = tjsonToArray(pJson, jkTableMetaRefCols, jsonToRefCol, pNode->colRef, sizeof(SColRef));
}
return code;
}
@ -1867,6 +1950,72 @@ static int32_t jsonToLogicJoinNode(const SJson* pJson, void* pObj) {
return code;
}
static const char* jkVirtualTableScanLogicPlanScanCols = "ScanCols";
static const char* jkVirtualTableScanLogicPlanScanPseudoCols = "ScanPseudoCols";
static const char* jkVirtualTableScanLogicPlanTableType = "TableType";
static const char* jkVirtualTableScanLogicPlanTableId = "TableId";
static const char* jkVirtualTableScanLogicPlanStableId = "StableId";
static const char* jkVirtualTableScanLogicPlanScanType = "ScanType";
static const char* jkVirtualTableScanLogicPlanscanAllCols = "scanAllCols";
static int32_t logicVirtualTableScanNodeToJson(const void* pObj, SJson* pJson) {
const SVirtualScanLogicNode* pNode = (const SVirtualScanLogicNode*)pObj;
int32_t code = logicPlanNodeToJson(pObj, pJson);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkVirtualTableScanLogicPlanscanAllCols, pNode->scanAllCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkVirtualTableScanLogicPlanScanCols, pNode->pScanCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkVirtualTableScanLogicPlanScanPseudoCols, pNode->pScanPseudoCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkVirtualTableScanLogicPlanTableType, pNode->tableType);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkVirtualTableScanLogicPlanTableId, pNode->tableId);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkVirtualTableScanLogicPlanStableId, pNode->stableId);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkVirtualTableScanLogicPlanScanType, pNode->scanType);
}
return code;
}
static int32_t jsonToLogicVirtualTableScanNode(const SJson* pJson, void* pObj) {
SVirtualScanLogicNode* pNode = (SVirtualScanLogicNode *)pObj;
int32_t objSize = 0;
int32_t code = jsonToLogicPlanNode(pJson, pObj);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkVirtualTableScanLogicPlanscanAllCols, &pNode->scanAllCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkVirtualTableScanLogicPlanScanCols, &pNode->pScanCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkVirtualTableScanLogicPlanScanPseudoCols, &pNode->pScanPseudoCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetTinyIntValue(pJson, jkVirtualTableScanLogicPlanTableType, &pNode->tableType);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetUBigIntValue(pJson, jkVirtualTableScanLogicPlanTableId, &pNode->tableId);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetUBigIntValue(pJson, jkVirtualTableScanLogicPlanStableId, &pNode->stableId);
}
if (TSDB_CODE_SUCCESS == code) {
tjsonGetNumberValue(pJson, jkVirtualTableScanLogicPlanScanType, pNode->scanType, code);
}
return code;
}
static const char* jkPhysiPlanOutputDataBlockDesc = "OutputDataBlockDesc";
static const char* jkPhysiPlanConditions = "Conditions";
static const char* jkPhysiPlanChildren = "Children";
@ -2326,6 +2475,54 @@ static int32_t jsonToPhysiTableScanNode(const SJson* pJson, void* pObj) {
}
return code;
}
static const char* jkVirtualTableScanPhysiPlanGroupTags = "GroupTags";
static const char* jkVirtualTableScanPhysiPlanGroupSort = "GroupSort";
static const char* jkVirtualTableScanPhysiPlanscanAllCols= "scanAllCols";
static const char* jkVirtualTableScanPhysiPlanTargets = "Targets";
static int32_t physiVirtualTableScanNodeToJson(const void* pObj, SJson* pJson) {
const SVirtualScanPhysiNode* pNode = (const SVirtualScanPhysiNode*)pObj;
int32_t code = physiScanNodeToJson(pObj, pJson);
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkVirtualTableScanPhysiPlanTargets, pNode->pGroupTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkVirtualTableScanPhysiPlanTargets, pNode->groupSort);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkVirtualTableScanPhysiPlanscanAllCols, pNode->scanAllCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkVirtualTableScanPhysiPlanTargets, pNode->pTargets);
}
return code;
}
static int32_t jsonToPhysiVirtualTableScanNode(const SJson* pJson, void* pObj) {
SVirtualScanPhysiNode* pNode = (SVirtualScanPhysiNode*)pObj;
int32_t code = jsonToPhysicPlanNode(pJson, pObj);
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkVirtualTableScanPhysiPlanGroupTags, &pNode->pGroupTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkVirtualTableScanPhysiPlanGroupSort, &pNode->groupSort);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkVirtualTableScanPhysiPlanscanAllCols, &pNode->scanAllCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkVirtualTableScanPhysiPlanTargets, &pNode->pTargets);
}
return code;
}
static const char* jkSysTableScanPhysiPlanMnodeEpSet = "MnodeEpSet";
static const char* jkSysTableScanPhysiPlanShowRewrite = "ShowRewrite";
@ -3926,6 +4123,110 @@ static int32_t jsonToQueryNodeAddr(const SJson* pJson, void* pObj) {
return code;
}
static const char* jkColOtableName = "colOtableName";
static const char* jkColVtableId = "colVtableId";
static int32_t colIdNameToJson(const void* pObj, SJson* pJson) {
const SColIdName* pCol = (const SColIdName*)pObj;
int32_t code = tjsonAddIntegerToObject(pJson, jkColVtableId, pCol->colId);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkColOtableName, pCol->colName);
}
return code;
}
static const char* jkOtableHashSize = "otbHashSize";
static const char* jkOtableHashKV = "otbHashKeyValue";
static const char* jkOtableHashName = "otbHashName";
static const char* jkOtableHashValue = "otbHashValue";
static int32_t oTableHashToJson(const void* pObj, SJson* pJson) {
const SSHashObj* pHash = (const SSHashObj*)pObj;
int32_t code = tjsonAddIntegerToObject(pJson, jkOtableHashSize, tSimpleHashGetSize(pHash));
if (code) {
return code;
}
SJson* pJsonArray = tjsonAddArrayToObject(pJson, jkOtableHashKV);
if (NULL == pJsonArray) {
return terrno;
}
int32_t iter = 0;
SArray** pCols = NULL;
char* pKey = NULL;
void* p = NULL;
while (NULL != (p = tSimpleHashIterate(pHash, p, &iter))) {
pKey = (char*)tSimpleHashGetKey(p, NULL);
pCols = (SArray**)p;
SJson* pJobj = tjsonCreateObject();
if (pJobj == NULL) {
return terrno;
}
code = tjsonAddStringToObject(pJobj, jkOtableHashName, pKey);
if (code) {
return code;
}
code = tjsonAddArray(pJobj, jkOtableHashValue, colIdNameToJson, TARRAY_GET_ELEM(*pCols, 0), sizeof(SColIdName), taosArrayGetSize(*pCols));
if (code) {
return code;
}
code = tjsonAddItemToArray(pJsonArray, pJobj);
if (code) {
return code;
}
}
return code;
}
static const char* jkVtablesHashSize = "vtbHashSize";
static const char* jkVtablesHashKV = "vtbHashKeyValue";
static const char* jkVtablesVuid = "vtbHashVuid";
static const char* jkVtablesVValue = "vtbHashVValue";
static int32_t vtablesHashToJson(const void* pObj, SJson* pJson) {
const SSHashObj* pHash = (const SSHashObj*)pObj;
int32_t code = tjsonAddIntegerToObject(pJson, jkVtablesHashSize, tSimpleHashGetSize(pHash));
if (code) {
return code;
}
SJson* pJsonArray = tjsonAddArrayToObject(pJson, jkVtablesHashKV);
if (NULL == pJsonArray) {
return terrno;
}
int32_t iter = 0;
SSHashObj** pOtable = NULL;
char* pKey = NULL;
void* p = NULL;
while (NULL != (p = tSimpleHashIterate(pHash, p, &iter))) {
pKey = (char*)tSimpleHashGetKey(p, NULL);
pOtable = (SSHashObj**)p;
SJson* pJobj = tjsonCreateObject();
if (pJobj == NULL) {
return terrno;
}
code = tjsonAddIntegerToObject(pJobj, jkVtablesVuid, *(uint64_t*)((int32_t*)pKey + 1));
if (code) {
return code;
}
code = tjsonAddObject(pJobj, jkVtablesVValue, oTableHashToJson, *pOtable);
if (code) {
return code;
}
code = tjsonAddItemToArray(pJsonArray, pJobj);
if (code) {
return code;
}
}
return code;
}
static const char* jkSubplanId = "Id";
static const char* jkSubplanType = "SubplanType";
static const char* jkSubplanMsgType = "MsgType";
@ -3937,12 +4238,14 @@ static const char* jkSubplanRootNode = "RootNode";
static const char* jkSubplanDataSink = "DataSink";
static const char* jkSubplanTagCond = "TagCond";
static const char* jkSubplanTagIndexCond = "TagIndexCond";
static const char* jkSubplanVTables = "VTables";
static const char* jkSubplanShowRewrite = "ShowRewrite";
static const char* jkSubplanRowsThreshold = "RowThreshold";
static const char* jkSubplanDynamicRowsThreshold = "DyRowThreshold";
static const char* jkSubplanIsView = "IsView";
static const char* jkSubplanIsAudit = "IsAudit";
static int32_t subplanToJson(const void* pObj, SJson* pJson) {
const SSubplan* pNode = (const SSubplan*)pObj;
@ -3977,6 +4280,9 @@ static int32_t subplanToJson(const void* pObj, SJson* pJson) {
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddObject(pJson, jkSubplanTagIndexCond, nodeToJson, pNode->pTagIndexCond);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddObject(pJson, jkSubplanVTables, vtablesHashToJson, pNode->pVTables);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkSubplanShowRewrite, pNode->showRewrite);
}
@ -3996,6 +4302,100 @@ static int32_t subplanToJson(const void* pObj, SJson* pJson) {
return code;
}
static int32_t jsonToOtableCols(const SJson* pJson, void* pObj) {
SArray** pCols = (SArray**)pObj;
SColIdName col;
char colName[TSDB_COL_NAME_LEN];
int32_t code = 0;
int32_t colNum = tjsonGetArraySize(pJson);
*pCols = taosArrayInit(colNum, sizeof(SColIdName));
if (NULL == *pCols) {
return terrno;
}
for (int32_t i = 0; i < colNum; ++i) {
SJson* pCol = tjsonGetArrayItem(pJson, i);
code = tjsonGetSmallIntValue(pCol, jkColVtableId, &col.colId);
if (code < 0) return TSDB_CODE_INVALID_JSON_FORMAT;
code = tjsonGetStringValue(pCol, jkColOtableName, colName);
if (code < 0) return TSDB_CODE_INVALID_JSON_FORMAT;
col.colName = taosStrdup(colName);
if (NULL == col.colName) {
return terrno;
}
if (NULL == taosArrayPush(*pCols, &col)) {
return terrno;
}
}
return code;
}
static int32_t jsonToOtableHash(const SJson* pJson, void* pObj) {
SSHashObj** pHash = (SSHashObj**)pObj;
int32_t hashSize = 0;
int32_t code = tjsonGetIntValue(pJson, jkOtableHashSize, &hashSize);
if (TSDB_CODE_SUCCESS == code && hashSize > 0) {
*pHash = tSimpleHashInit(hashSize, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BINARY));
if (NULL == *pHash) {
return terrno;
}
tSimpleHashSetFreeFp(*pHash, tFreeStreamVtbOtbInfo);
SJson *ovalues = tjsonGetObjectItem(pJson, jkOtableHashKV);
if (ovalues == NULL) return TSDB_CODE_INVALID_JSON_FORMAT;
char tbName[TSDB_TABLE_NAME_LEN];
SArray* pCols = NULL;
for (int32_t d = 0; d < hashSize; ++d) {
SJson *okeyValue = tjsonGetArrayItem(ovalues, d);
if (okeyValue == NULL) return TSDB_CODE_INVALID_JSON_FORMAT;
code = tjsonGetStringValue(okeyValue, jkOtableHashName, tbName);
if (code < 0) return TSDB_CODE_INVALID_JSON_FORMAT;
SJson *ovalue = tjsonGetObjectItem(okeyValue, jkOtableHashValue);
code = jsonToOtableCols(ovalue, &pCols);
if (code < 0) return code;
code = tSimpleHashPut(*pHash, tbName, strlen(tbName) + 1, &pCols, POINTER_BYTES);
if (code < 0) return code;
}
}
return code;
}
static int32_t jsonToVtablesHash(const SJson* pJson, void* pObj) {
SSHashObj** pHash = (SSHashObj**)pObj;
int32_t hashSize = 0;
int32_t code = tjsonGetIntValue(pJson, jkVtablesHashSize, &hashSize);
if (TSDB_CODE_SUCCESS == code && hashSize > 0) {
*pHash = tSimpleHashInit(hashSize, taosGetDefaultHashFunction(TSDB_DATA_TYPE_BIGINT));
if (NULL == *pHash) {
return terrno;
}
tSimpleHashSetFreeFp(*pHash, tFreeStreamVtbVtbInfo);
SJson *vvalues = tjsonGetObjectItem(pJson, jkVtablesHashKV);
if (vvalues == NULL) return TSDB_CODE_INVALID_JSON_FORMAT;
uint64_t vuid = 0;
SSHashObj* pOtable = NULL;
for (int32_t d = 0; d < hashSize; ++d) {
SJson *vkeyValue = tjsonGetArrayItem(vvalues, d);
if (vkeyValue == NULL) return TSDB_CODE_INVALID_JSON_FORMAT;
code = tjsonGetUBigIntValue(vkeyValue, jkVtablesVuid, &vuid);
if (code < 0) return TSDB_CODE_INVALID_JSON_FORMAT;
SJson *vvalue = tjsonGetObjectItem(vkeyValue, jkVtablesVValue);
code = jsonToOtableHash(vvalue, &pOtable);
if (code < 0) return code;
code = tSimpleHashPut(*pHash, &vuid, sizeof(vuid), &pOtable, POINTER_BYTES);
if (code < 0) return code;
}
}
return code;
}
static int32_t jsonToSubplan(const SJson* pJson, void* pObj) {
SSubplan* pNode = (SSubplan*)pObj;
@ -4030,6 +4430,9 @@ static int32_t jsonToSubplan(const SJson* pJson, void* pObj) {
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeObject(pJson, jkSubplanTagIndexCond, (SNode**)&pNode->pTagIndexCond);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonToObject(pJson, jkSubplanVTables, jsonToVtablesHash, &pNode->pVTables);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkSubplanShowRewrite, &pNode->showRewrite);
}
@ -4965,6 +5368,61 @@ static int32_t jsonToJoinTableNode(const SJson* pJson, void* pObj) {
return code;
}
static const char* jkVirtualTableMetaSize = "MetaSize";
static const char* jkVirtuaTableMeta = "Meta";
static const char* jkVirtuaTableVgroupsInfoSize = "VgroupsInfoSize";
static const char* jkVirtuaTableVgroupsInfo = "VgroupsInfo";
static const char* jkVirtuaTableRefTables = "RefTables";
static int32_t virtualTableNodeToJson(const void* pObj, SJson* pJson) {
const SVirtualTableNode* pNode = (const SVirtualTableNode*)pObj;
int32_t code = tableNodeToJson(pObj, pJson);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkVirtualTableMetaSize, TABLE_META_SIZE(pNode->pMeta));
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddObject(pJson, jkVirtuaTableMeta, tableMetaToJson, pNode->pMeta);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddIntegerToObject(pJson, jkVirtuaTableVgroupsInfoSize, VGROUPS_INFO_SIZE(pNode->pVgroupList));
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddObject(pJson, jkVirtuaTableVgroupsInfo, vgroupsInfoToJson, pNode->pVgroupList);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkVirtuaTableRefTables, pNode->refTables);
}
return code;
}
static int32_t jsonToVirtualTableNode(const SJson* pJson, void* pObj) {
SVirtualTableNode* pNode = (SVirtualTableNode*)pObj;
int32_t objSize = 0;
int32_t code = jsonToTableNode(pJson, pObj);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetIntValue(pJson, jkVirtualTableMetaSize, &objSize);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonMakeObject(pJson, jkVirtuaTableMeta, jsonToTableMeta, (void**)&pNode->pMeta, objSize);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetIntValue(pJson, jkVirtuaTableVgroupsInfoSize, &objSize);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonMakeObject(pJson, jkVirtuaTableVgroupsInfo, jsonToVgroupsInfo, (void**)&pNode->pVgroupList, objSize);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkVirtuaTableRefTables, &pNode->refTables);
}
return code;
}
static const char* jkGroupingSetType = "GroupingSetType";
static const char* jkGroupingSetParameter = "Parameters";
@ -5961,6 +6419,42 @@ static int32_t jsonToStreamNotifyOptions(const SJson* pJson, void* pObj) {
if (code == TSDB_CODE_SUCCESS) {
code = tjsonGetBoolValue(pJson, jkStreamNotifyOptionsNotifyHistory, &pNotifyOption->notifyHistory);
}
return code;
}
static const char* jkColumnReferenceColumnName = "ColumnName";
static const char* jkColumnReferenceRefDbName = "RefDbName";
static const char* jkColumnReferenceRefTableName = "RefTableName";
static const char* jkColumnReferenceRefColumnName = "RefColumnName";
static int32_t columnReferenceToJson(const void* pObj, SJson* pJson) {
const SColumnRefNode* pNode = (const SColumnRefNode*)pObj;
int32_t code = tjsonAddStringToObject(pJson, jkColumnReferenceColumnName, pNode->colName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkColumnReferenceRefDbName, pNode->refDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkColumnReferenceRefTableName, pNode->refTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkColumnReferenceRefColumnName, pNode->refColName);
}
return code;
}
static int32_t jsonToColumnReference(const SJson* pJson, void* pObj) {
SColumnRefNode* pNode = (SColumnRefNode*)pObj;
int32_t code = tjsonGetStringValue(pJson, jkColumnReferenceColumnName, pNode->colName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkColumnReferenceRefDbName, pNode->refDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkColumnReferenceRefTableName, pNode->refTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkColumnReferenceRefColumnName, pNode->refColName);
}
return code;
}
@ -6542,6 +7036,117 @@ static int32_t jsonToCreateMultiTablesStmt(const SJson* pJson, void* pObj) {
return jsonToNodeList(pJson, jkCreateMultiTablesStmtSubTables, &pNode->pSubTables);
}
static const char* jkCreateVTableStmtDbName = "DbName";
static const char* jkCreateVTableStmtTableName = "TableName";
static const char* jkCreateVTableStmtIgnoreExists = "IgnoreExists";
static const char* jkCreateVTableStmtCols = "Cols";
static int32_t createVTableStmtToJson(const void* pObj, SJson* pJson) {
const SCreateVTableStmt* pNode = (const SCreateVTableStmt*)pObj;
int32_t code = tjsonAddStringToObject(pJson, jkCreateVTableStmtDbName, pNode->dbName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkCreateVTableStmtTableName, pNode->tableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkCreateVTableStmtIgnoreExists, pNode->ignoreExists);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkCreateVTableStmtCols, pNode->pCols);
}
return code;
}
static int32_t jsonToCreateVTableStmt(const SJson* pJson, void* pObj) {
SCreateVTableStmt* pNode = (SCreateVTableStmt*)pObj;
int32_t code = tjsonGetStringValue(pJson, jkCreateTableStmtDbName, pNode->dbName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkCreateTableStmtTableName, pNode->tableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkCreateTableStmtIgnoreExists, &pNode->ignoreExists);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkCreateTableStmtCols, &pNode->pCols);
}
return code;
}
static const char* jkCreateVSubTableStmtDbName = "DbName";
static const char* jkCreateVSubTableStmtTableName = "TableName";
static const char* jkCreateVSubTableStmtUseDbName = "UseDbName";
static const char* jkCreateVSubTableStmtUseTableName = "UseTableName";
static const char* jkCreateVSubTableStmtIgnoreExists = "IgnoreExists";
static const char* jkCreateVSubTableStmtSpecificTags = "SpecificTags";
static const char* jkCreateVSubTableStmtValsOfTags = "ValsOfTags";
static const char* jkCreateVSubTableStmtSpecificColRefs = "SpecificColRefs";
static const char* jkCreateVSubTableStmtValsOfColRefs = "ValsOfColRefs";
static int32_t createVSubTableStmtToJson(const void* pObj, SJson* pJson) {
const SCreateVSubTableStmt* pNode = (const SCreateVSubTableStmt*)pObj;
int32_t code = tjsonAddStringToObject(pJson, jkCreateVSubTableStmtDbName, pNode->dbName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkCreateVSubTableStmtTableName, pNode->tableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkCreateVSubTableStmtUseDbName, pNode->useDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkCreateVSubTableStmtUseTableName, pNode->useTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkCreateVSubTableStmtIgnoreExists, pNode->ignoreExists);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkCreateVSubTableStmtSpecificTags, pNode->pSpecificTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkCreateVSubTableStmtValsOfTags, pNode->pValsOfTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkCreateVSubTableStmtSpecificColRefs, pNode->pSpecificColRefs);
}
if (TSDB_CODE_SUCCESS == code) {
code = nodeListToJson(pJson, jkCreateVSubTableStmtValsOfColRefs, pNode->pColRefs);
}
return code;
}
static int32_t jsonToCreateVSubTableStmt(const SJson* pJson, void* pObj) {
SCreateVSubTableStmt* pNode = (SCreateVSubTableStmt*)pObj;
int32_t code = tjsonGetStringValue(pJson, jkCreateVSubTableStmtDbName, pNode->dbName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkCreateVSubTableStmtTableName, pNode->tableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkCreateVSubTableStmtUseDbName, pNode->useDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkCreateVSubTableStmtUseTableName, pNode->useTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkCreateVSubTableStmtIgnoreExists, &pNode->ignoreExists);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkCreateVSubTableStmtSpecificTags, &pNode->pSpecificTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkCreateVSubTableStmtValsOfTags, &pNode->pValsOfTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkCreateVSubTableStmtSpecificColRefs, &pNode->pSpecificColRefs);
}
if (TSDB_CODE_SUCCESS == code) {
code = jsonToNodeList(pJson, jkCreateVSubTableStmtValsOfColRefs, &pNode->pColRefs);
}
return code;
}
static const char* jkDropTableClauseDbName = "DbName";
static const char* jkDropTableClauseTableName = "TableName";
static const char* jkDropTableClauseIgnoreNotExists = "IgnoreNotExists";
@ -6625,6 +7230,45 @@ static int32_t jsonToDropStableStmt(const SJson* pJson, void* pObj) {
return code;
}
static const char* jkDropVirtualTableStmtDbName = "DbName";
static const char* jkDropVirtualTableStmtTableName = "TableName";
static const char* jkDropVirtualTableStmtIgnoreNotExists = "IgnoreNotExists";
static const char* jkDropVirtualTableStmtwithOpt = "withOpt";
static int32_t dropVtableStmtToJson(const void* pObj, SJson* pJson) {
const SDropVirtualTableStmt* pNode = (const SDropVirtualTableStmt*)pObj;
int32_t code = tjsonAddStringToObject(pJson, jkDropVirtualTableStmtDbName, pNode->dbName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddStringToObject(pJson, jkDropVirtualTableStmtTableName, pNode->tableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkDropVirtualTableStmtIgnoreNotExists, pNode->ignoreNotExists);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonAddBoolToObject(pJson, jkDropVirtualTableStmtwithOpt, pNode->withOpt);
}
return code;
}
static int32_t jsonToDropVtableStmt(const SJson* pJson, void* pObj) {
SDropVirtualTableStmt* pNode = (SDropVirtualTableStmt*)pObj;
int32_t code = tjsonGetStringValue(pJson, jkDropSuperTableStmtDbName, pNode->dbName);
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetStringValue(pJson, jkDropVirtualTableStmtTableName, pNode->tableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkDropVirtualTableStmtIgnoreNotExists, &pNode->ignoreNotExists);
}
if (TSDB_CODE_SUCCESS == code) {
code = tjsonGetBoolValue(pJson, jkDropVirtualTableStmtwithOpt, &pNode->withOpt);
}
return code;
}
static const char* jkAlterTableStmtDbName = "DbName";
static const char* jkAlterTableStmtTableName = "TableName";
static const char* jkAlterTableStmtAlterType = "AlterType";
@ -8134,6 +8778,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
return tempTableNodeToJson(pObj, pJson);
case QUERY_NODE_JOIN_TABLE:
return joinTableNodeToJson(pObj, pJson);
case QUERY_NODE_VIRTUAL_TABLE:
return virtualTableNodeToJson(pObj, pJson);
case QUERY_NODE_GROUPING_SET:
return groupingSetNodeToJson(pObj, pJson);
case QUERY_NODE_ORDER_BY_EXPR:
@ -8176,6 +8822,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
return streamOptionsToJson(pObj, pJson);
case QUERY_NODE_LEFT_VALUE:
return TSDB_CODE_SUCCESS; // SLeftValueNode has no fields to serialize.
case QUERY_NODE_COLUMN_REF:
return columnReferenceToJson(pObj, pJson);
case QUERY_NODE_WHEN_THEN:
return whenThenNodeToJson(pObj, pJson);
case QUERY_NODE_CASE_WHEN:
@ -8208,6 +8856,10 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
return createTableStmtToJson(pObj, pJson);
case QUERY_NODE_CREATE_SUBTABLE_CLAUSE:
return createSubTableClauseToJson(pObj, pJson);
case QUERY_NODE_CREATE_VIRTUAL_TABLE_STMT:
return createVTableStmtToJson(pObj, pJson);
case QUERY_NODE_CREATE_VIRTUAL_SUBTABLE_STMT:
return createVSubTableStmtToJson(pObj, pJson);
case QUERY_NODE_CREATE_MULTI_TABLES_STMT:
return createMultiTablesStmtToJson(pObj, pJson);
case QUERY_NODE_DROP_TABLE_CLAUSE:
@ -8216,6 +8868,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
return dropTableStmtToJson(pObj, pJson);
case QUERY_NODE_DROP_SUPER_TABLE_STMT:
return dropStableStmtToJson(pObj, pJson);
case QUERY_NODE_DROP_VIRTUAL_TABLE_STMT:
return dropVtableStmtToJson(pObj, pJson);
case QUERY_NODE_ALTER_TABLE_STMT:
return alterTableStmtToJson(pObj, pJson);
case QUERY_NODE_ALTER_SUPER_TABLE_STMT:
@ -8321,6 +8975,7 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
case QUERY_NODE_SHOW_STREAMS_STMT:
return showStreamsStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_TABLES_STMT:
case QUERY_NODE_SHOW_VTABLES_STMT:
return showTablesStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_TAGS_STMT:
return showTagsStmtToJson(pObj, pJson);
@ -8354,6 +9009,7 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
case QUERY_NODE_SHOW_CREATE_DATABASE_STMT:
return showCreateDatabaseStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_CREATE_TABLE_STMT:
case QUERY_NODE_SHOW_CREATE_VTABLE_STMT:
return showCreateTableStmtToJson(pObj, pJson);
case QUERY_NODE_SHOW_CREATE_STABLE_STMT:
return showCreateStableStmtToJson(pObj, pJson);
@ -8375,6 +9031,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
return logicScanNodeToJson(pObj, pJson);
case QUERY_NODE_LOGIC_PLAN_JOIN:
return logicJoinNodeToJson(pObj, pJson);
case QUERY_NODE_LOGIC_PLAN_VIRTUAL_TABLE_SCAN:
return logicVirtualTableScanNodeToJson(pObj, pJson);
case QUERY_NODE_LOGIC_PLAN_AGG:
return logicAggNodeToJson(pObj, pJson);
case QUERY_NODE_LOGIC_PLAN_PROJECT:
@ -8418,6 +9076,8 @@ static int32_t specificNodeToJson(const void* pObj, SJson* pJson) {
case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN:
case QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN:
return physiTableScanNodeToJson(pObj, pJson);
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
return physiVirtualTableScanNodeToJson(pObj, pJson);
case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN:
return physiSysTableScanNodeToJson(pObj, pJson);
case QUERY_NODE_PHYSICAL_PLAN_PROJECT:
@ -8522,6 +9182,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToTempTableNode(pJson, pObj);
case QUERY_NODE_JOIN_TABLE:
return jsonToJoinTableNode(pJson, pObj);
case QUERY_NODE_VIRTUAL_TABLE:
return jsonToVirtualTableNode(pJson, pObj);
case QUERY_NODE_GROUPING_SET:
return jsonToGroupingSetNode(pJson, pObj);
case QUERY_NODE_ORDER_BY_EXPR:
@ -8562,6 +9224,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToStreamOptions(pJson, pObj);
case QUERY_NODE_LEFT_VALUE:
return TSDB_CODE_SUCCESS; // SLeftValueNode has no fields to deserialize.
case QUERY_NODE_COLUMN_REF:
return jsonToColumnReference(pJson, pObj);
case QUERY_NODE_WHEN_THEN:
return jsonToWhenThenNode(pJson, pObj);
case QUERY_NODE_CASE_WHEN:
@ -8594,6 +9258,10 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToCreateTableStmt(pJson, pObj);
case QUERY_NODE_CREATE_SUBTABLE_CLAUSE:
return jsonToCreateSubTableClause(pJson, pObj);
case QUERY_NODE_CREATE_VIRTUAL_TABLE_STMT:
return jsonToCreateVTableStmt(pJson, pObj);
case QUERY_NODE_CREATE_VIRTUAL_SUBTABLE_STMT:
return jsonToCreateVSubTableStmt(pJson, pObj);
case QUERY_NODE_CREATE_MULTI_TABLES_STMT:
return jsonToCreateMultiTablesStmt(pJson, pObj);
case QUERY_NODE_DROP_TABLE_CLAUSE:
@ -8602,6 +9270,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToDropTableStmt(pJson, pObj);
case QUERY_NODE_DROP_SUPER_TABLE_STMT:
return jsonToDropStableStmt(pJson, pObj);
case QUERY_NODE_DROP_VIRTUAL_TABLE_STMT:
return jsonToDropVtableStmt(pJson, pObj);
case QUERY_NODE_ALTER_TABLE_STMT:
return jsonToAlterTableStmt(pJson, pObj);
case QUERY_NODE_ALTER_SUPER_TABLE_STMT:
@ -8701,6 +9371,7 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
case QUERY_NODE_SHOW_STREAMS_STMT:
return jsonToShowStreamsStmt(pJson, pObj);
case QUERY_NODE_SHOW_TABLES_STMT:
case QUERY_NODE_SHOW_VTABLES_STMT:
return jsonToShowTablesStmt(pJson, pObj);
case QUERY_NODE_SHOW_TAGS_STMT:
return jsonToShowTagsStmt(pJson, pObj);
@ -8734,6 +9405,7 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
case QUERY_NODE_SHOW_CREATE_DATABASE_STMT:
return jsonToShowCreateDatabaseStmt(pJson, pObj);
case QUERY_NODE_SHOW_CREATE_TABLE_STMT:
case QUERY_NODE_SHOW_CREATE_VTABLE_STMT:
return jsonToShowCreateTableStmt(pJson, pObj);
case QUERY_NODE_SHOW_CREATE_STABLE_STMT:
return jsonToShowCreateStableStmt(pJson, pObj);
@ -8763,6 +9435,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
return jsonToLogicScanNode(pJson, pObj);
case QUERY_NODE_LOGIC_PLAN_JOIN:
return jsonToLogicJoinNode(pJson, pObj);
case QUERY_NODE_LOGIC_PLAN_VIRTUAL_TABLE_SCAN:
return jsonToLogicVirtualTableScanNode(pJson, pObj);
case QUERY_NODE_LOGIC_PLAN_AGG:
return jsonToLogicAggNode(pJson, pObj);
case QUERY_NODE_LOGIC_PLAN_PROJECT:
@ -8806,6 +9480,8 @@ static int32_t jsonToSpecificNode(const SJson* pJson, void* pObj) {
case QUERY_NODE_PHYSICAL_PLAN_TABLE_MERGE_SCAN:
case QUERY_NODE_PHYSICAL_PLAN_STREAM_SCAN:
return jsonToPhysiTableScanNode(pJson, pObj);
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
return jsonToPhysiVirtualTableScanNode(pJson, pObj);
case QUERY_NODE_PHYSICAL_PLAN_SYSTABLE_SCAN:
return jsonToPhysiSysTableScanNode(pJson, pObj);
case QUERY_NODE_PHYSICAL_PLAN_PROJECT:

1
source/libs/nodes/src/nodesEqualFuncs.c
View File

@ -200,6 +200,7 @@ bool nodesEqualNode(const SNode* a, const SNode* b) {
case QUERY_NODE_GROUPING_SET:
return groupingSetNodeEqual((const SGroupingSetNode*)a, (const SGroupingSetNode*)b);
case QUERY_NODE_REAL_TABLE:
case QUERY_NODE_VIRTUAL_TABLE:
case QUERY_NODE_TEMP_TABLE:
case QUERY_NODE_JOIN_TABLE:
case QUERY_NODE_ORDER_BY_EXPR:

1
source/libs/nodes/src/nodesMatchFuncs.c
View File

@ -168,6 +168,7 @@ bool nodesMatchNode(const SNode* pSub, const SNode* p) {
case QUERY_NODE_REAL_TABLE:
case QUERY_NODE_TEMP_TABLE:
case QUERY_NODE_JOIN_TABLE:
case QUERY_NODE_VIRTUAL_TABLE:
case QUERY_NODE_GROUPING_SET:
case QUERY_NODE_ORDER_BY_EXPR:
case QUERY_NODE_LIMIT:

159
source/libs/nodes/src/nodesMsgFuncs.c
View File

@ -757,7 +757,25 @@ static int32_t columnNodeInlineToMsg(const void* pObj, STlvEncoder* pEncoder) {
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueI16(pEncoder, pNode->numOfPKs);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueBool(pEncoder, pNode->isPrimTs);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueBool(pEncoder, pNode->hasDep);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueBool(pEncoder, pNode->hasRef);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueCStr(pEncoder, pNode->refDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueCStr(pEncoder, pNode->refTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueCStr(pEncoder, pNode->refColName);
}
return code;
}
@ -813,7 +831,25 @@ static int32_t msgToColumnNodeInline(STlvDecoder* pDecoder, void* pObj) {
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueI16(pDecoder, &pNode->numOfPKs);
}
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueBool(pDecoder, &pNode->isPrimTs);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueBool(pDecoder, &pNode->hasDep);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueBool(pDecoder, &pNode->hasRef);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueCStr(pDecoder, pNode->refDbName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueCStr(pDecoder, pNode->refTableName);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueCStr(pDecoder, pNode->refColName);
}
return code;
}
@ -2088,7 +2124,8 @@ enum {
PHY_SCAN_CODE_BASE_SUID,
PHY_SCAN_CODE_BASE_TABLE_TYPE,
PHY_SCAN_CODE_BASE_TABLE_NAME,
PHY_SCAN_CODE_BASE_GROUP_ORDER_SCAN
PHY_SCAN_CODE_BASE_GROUP_ORDER_SCAN,
PHY_SCAN_CODE_BASE_VIRTUAL_STABLE_SCAN
};
static int32_t physiScanNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
@ -2116,6 +2153,9 @@ static int32_t physiScanNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeBool(pEncoder, PHY_SCAN_CODE_BASE_GROUP_ORDER_SCAN, pNode->groupOrderScan);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeBool(pEncoder, PHY_SCAN_CODE_BASE_VIRTUAL_STABLE_SCAN, pNode->virtualStableScan);
}
return code;
}
@ -2151,6 +2191,69 @@ static int32_t msgToPhysiScanNode(STlvDecoder* pDecoder, void* pObj) {
case PHY_SCAN_CODE_BASE_GROUP_ORDER_SCAN:
code = tlvDecodeBool(pTlv, &pNode->groupOrderScan);
break;
case PHY_SCAN_CODE_BASE_VIRTUAL_STABLE_SCAN:
code = tlvDecodeBool(pTlv, &pNode->virtualStableScan);
default:
break;
}
}
return code;
}
enum {
PHY_VIRTUAL_TABLE_SCAN_CODE_SCAN = 1,
PHY_VIRTUAL_TABLE_SCAN_CODE_GROUPTAGS,
PHY_VIRTUAL_TABLE_SCAN_CODE_GROUP_SORT,
PHY_VIRTUAL_TABLE_SCAN_CODE_ONLY_TS,
PHY_VIRTUAL_TABLE_SCAN_CODE_TARGETS,
};
static int32_t physiVirtualTableScanNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
const SVirtualScanPhysiNode* pNode = (const SVirtualScanPhysiNode *)pObj;
int32_t code = tlvEncodeObj(pEncoder, PHY_VIRTUAL_TABLE_SCAN_CODE_SCAN, physiScanNodeToMsg, &pNode->scan);
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeObj(pEncoder, PHY_VIRTUAL_TABLE_SCAN_CODE_GROUPTAGS, nodeListToMsg, pNode->pGroupTags);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeBool(pEncoder, PHY_VIRTUAL_TABLE_SCAN_CODE_GROUP_SORT, pNode->groupSort);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeBool(pEncoder, PHY_VIRTUAL_TABLE_SCAN_CODE_ONLY_TS, pNode->scanAllCols);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeObj(pEncoder, PHY_VIRTUAL_TABLE_SCAN_CODE_TARGETS, nodeListToMsg, pNode->pTargets);
}
return code;
}
static int32_t msgToPhysiVirtualTableScanNode(STlvDecoder* pDecoder, void* pObj) {
SVirtualScanPhysiNode* pNode = (SVirtualScanPhysiNode*)pObj;
int32_t code = TSDB_CODE_SUCCESS;
STlv* pTlv = NULL;
tlvForEach(pDecoder, pTlv, code) {
switch (pTlv->type) {
case PHY_VIRTUAL_TABLE_SCAN_CODE_SCAN:
code = tlvDecodeObjFromTlv(pTlv, msgToPhysiScanNode, &pNode->scan);
break;
case PHY_VIRTUAL_TABLE_SCAN_CODE_GROUPTAGS:
code = msgToNodeListFromTlv(pTlv, (void**)&pNode->pGroupTags);
break;
case PHY_VIRTUAL_TABLE_SCAN_CODE_GROUP_SORT:
code = tlvDecodeBool(pTlv, &pNode->groupSort);
break;
case PHY_VIRTUAL_TABLE_SCAN_CODE_ONLY_TS:
code = tlvDecodeBool(pTlv, &pNode->scanAllCols);
break;
case PHY_VIRTUAL_TABLE_SCAN_CODE_TARGETS:
code = msgToNodeListFromTlv(pTlv, (void**)&pNode->pTargets);
break;
default:
break;
}
@ -4248,6 +4351,11 @@ enum {
PHY_DYN_QUERY_CTRL_CODE_STB_JOIN_UID_SLOT1,
PHY_DYN_QUERY_CTRL_CODE_STB_JOIN_SRC_SCAN0,
PHY_DYN_QUERY_CTRL_CODE_STB_JOIN_SRC_SCAN1,
PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SCAN_ALL_COLS,
PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SUID,
PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_ACCOUNT_ID,
PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_EP_SET,
PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SCAN_COLS,
};
static int32_t physiDynQueryCtrlNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
@ -4281,6 +4389,22 @@ static int32_t physiDynQueryCtrlNodeToMsg(const void* pObj, STlvEncoder* pEncode
}
break;
}
case DYN_QTYPE_VTB_SCAN: {
code = tlvEncodeBool(pEncoder, PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SCAN_ALL_COLS, pNode->vtbScan.scanAllCols);
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeU64(pEncoder, PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SUID, pNode->vtbScan.suid);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeI32(pEncoder, PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_ACCOUNT_ID, pNode->vtbScan.accountId);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeObj(pEncoder, PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_EP_SET, epSetToMsg, &pNode->vtbScan.mgmtEpSet);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeObj(pEncoder, PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SCAN_COLS, nodeListToMsg, pNode->vtbScan.pScanCols);
}
break;
}
default:
return TSDB_CODE_INVALID_PARA;
}
@ -4321,7 +4445,21 @@ static int32_t msgToPhysiDynQueryCtrlNode(STlvDecoder* pDecoder, void* pObj) {
break;
case PHY_DYN_QUERY_CTRL_CODE_STB_JOIN_SRC_SCAN1:
code = tlvDecodeBool(pTlv, &pNode->stbJoin.srcScan[1]);
break;
break;
case PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SCAN_ALL_COLS:
code = tlvDecodeBool(pTlv, &pNode->vtbScan.scanAllCols);
break;
case PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SUID:
code = tlvDecodeU64(pTlv, &pNode->vtbScan.suid);
break;
case PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_ACCOUNT_ID:
code = tlvDecodeI32(pTlv, &pNode->vtbScan.accountId);
break;
case PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_EP_SET:
code = tlvDecodeObjFromTlv(pTlv, msgToEpSet, &pNode->vtbScan.mgmtEpSet);
break;
case PHY_DYN_QUERY_CTRL_CODE_VTB_SCAN_SCAN_COLS:
code = msgToNodeListFromTlv(pTlv, (void**)&pNode->vtbScan.pScanCols);
default:
break;
}
@ -4445,7 +4583,9 @@ static int32_t subplanInlineToMsg(const void* pObj, STlvEncoder* pEncoder) {
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueBool(pEncoder, pNode->isAudit);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvEncodeValueBool(pEncoder, pNode->processOneBlock);
}
return code;
}
@ -4506,6 +4646,9 @@ static int32_t msgToSubplanInline(STlvDecoder* pDecoder, void* pObj) {
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueBool(pDecoder, &pNode->isAudit);
}
if (TSDB_CODE_SUCCESS == code) {
code = tlvDecodeValueBool(pDecoder, &pNode->processOneBlock);
}
return code;
}
@ -4759,6 +4902,9 @@ static int32_t specificNodeToMsg(const void* pObj, STlvEncoder* pEncoder) {
case QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL:
code = physiDynQueryCtrlNodeToMsg(pObj, pEncoder);
break;
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
code = physiVirtualTableScanNodeToMsg(pObj, pEncoder);
break;
case QUERY_NODE_PHYSICAL_SUBPLAN:
code = subplanToMsg(pObj, pEncoder);
break;
@ -4937,6 +5083,9 @@ static int32_t msgToSpecificNode(STlvDecoder* pDecoder, void* pObj) {
case QUERY_NODE_PHYSICAL_PLAN_DYN_QUERY_CTRL:
code = msgToPhysiDynQueryCtrlNode(pDecoder, pObj);
break;
case QUERY_NODE_PHYSICAL_PLAN_VIRTUAL_TABLE_SCAN:
code = msgToPhysiVirtualTableScanNode(pDecoder, pObj);
break;
case QUERY_NODE_PHYSICAL_SUBPLAN:
code = msgToSubplan(pDecoder, pObj);
break;

2
source/libs/nodes/src/nodesTraverseFuncs.c
View File

@ -78,6 +78,7 @@ static EDealRes dispatchExpr(SNode* pNode, ETraversalOrder order, FNodeWalker wa
break;
case QUERY_NODE_REAL_TABLE:
case QUERY_NODE_TEMP_TABLE:
case QUERY_NODE_VIRTUAL_TABLE:
break; // todo
case QUERY_NODE_JOIN_TABLE: {
SJoinTableNode* pJoinTableNode = (SJoinTableNode*)pNode;
@ -295,6 +296,7 @@ static EDealRes rewriteExpr(SNode** pRawNode, ETraversalOrder order, FNodeRewrit
}
case QUERY_NODE_REAL_TABLE:
case QUERY_NODE_TEMP_TABLE:
case QUERY_NODE_VIRTUAL_TABLE:
break; // todo
case QUERY_NODE_JOIN_TABLE: {
SJoinTableNode* pJoinTableNode = (SJoinTableNode*)pNode;

Some files were not shown because too many files have changed in this diff Show More