Commit Graph

2417 Commits

Author SHA1 Message Date
liuyao fac36002c1
ci(stream): add ci and fix issue (#30270)
* ci(stream): add ci and fix issue

* ci: adj test

* ci: add test and fix issue

* ci: adj test

* ci(stream): add test for fill operator and fix issue

* fix: fix issue

* fix: fix issue

* fix: adjust defalut rec value
2025-03-20 10:34:31 +08:00
WANG MINGMING 3fafecb242
feat[TS-6137]: support sliding in force_window_close (#30203)
* feat[TS-6137]: support sliding in force_window_close

* feat[TS-6137]: support sliding in force_window_close

* feat(stream): force window close support interval sliding

* feat[TS-6137]: support sliding in force_window_close

* feat[TS-6137]: support sliding in force_window_close

* feat[TS-6137]: support sliding in force_window_close

---------

Co-authored-by: 54liuyao <54liuyao@163.com>
2025-03-19 10:22:06 +08:00
Jinqing Kuang ac23a4fc55
feat(stream): support stream processing for virtual tables (#30144)
* enh: add client processing

* enh: add mnode vtables processing

* enh: add mnode vtable processing

* enh: add normal child vtable support

* fix: compile issues

* fix: compile issues

* fix: create stream issues

* fix: multi stream scan issue

* fix: remove debug info

* fix: agg task and task level issues

* fix: correct task output type

* fix: split vtablescan from agg

* fix: memory leak issues

* fix: add limitations

* Update 09-error-code.md

* Update 09-error-code.md

* fix: remove usless case

* feat(stream): extract original table data in source scan task

Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.

* feat(stream): multi-way merge using loser tree in virtual merge task

Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing.  Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.

* fix(test): remove useless cases

---------

Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
liuyao 2de6736160
feat(stream): add new trigger continuous_window_close (#30125)
* opt stream build twa result

* opt force window close memory

* feat(stream):optimize new interval and scan operator

* adj log

* opt code

* opt code

* fill history

* fix issue for fill history

* add ci

* feat(stream): add new stream nonblock interval operator

* adjust code

* use new scan operator

* use new scan operator

* add log

* fix issue

* recover stream scan next function

* fix issue

* fix issue

* fix issue

* ignore disorder data

* fix issue

* fix issue for interval sliding

* fix issue

* fix ci issue

* fix ci issue

* fix ci issue

* add semi && final nonblock interval operator

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* refactor(stream): track the msgId for each upstream tasks.

* fix(stream): fix race condition.

* fix(stream): update the task last msgId when putting into input queue succ.

* fix issue

* fix issue

* put recalculate data to rocksdb

* fix issue

* fix issue

* enh(query)[TD-33071]: add support for saving and restoring tsdbReader scan progress

- Implement functionality to save scan progress during tsdbReader operations
- Enable resuming scans from the last saved position

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* add rec interval check

* enh(stream):add recalculate tasks.

* enh(stream): support the re-calculate the tasks.

* fix issue && do refactor

* do refactor

* fix issue

* fix issue

* update backend opt

* add new interface

* add new session operator

* support blob

* add new session operator

* fix issue

* add rec state for rec task

* fix invalid read

* add new session window

* enh(stream): update the stream tasks backend.

* new session operator

* add pull data

* fix(stream): fix error in expand stream backend.

* fix issue

* fix issue

* fix issue

* merge code

* fix issue

* fix(stream): check for null ptr.

* fix(stream): add more check.

* fix issue

* fix issue

* fix issue

* add debug code

* fix issue

* fix issue

* fix issue

* set rec end flag

* fix(stream): opt re-calculate stream tasks.

* fix issue

* fix issue

* add new operator

* enh(stream): dispatch recalculate block to agg tasks.

* fix issue

* fix issue

* fix(stream): adjust filter.

* fix issue

* refactor

* refactor(stream): adjust the recalculate end block.

* fix issue

* fix(stream): set correct create trigger block.

* fix issue

* fix(stream): fix error in build recalculate end block.

* fix(stream): check null ptr.

* add stream client && fix issue

* fix mem leak

* fix(stream): free msg.

* add stream client

* fix(stream): fix error.

* add stream client && fix issue

* add stream client

* refactor(stream): set the recalculate task complete.

* add wend and group_id for session window dest table

* feat(stream): refactor and set the recalcul agg task complete.

* add cfg for adapter

* fix issue

* add state && event operator

* feat(stream): support fill-history task.

* add new fill operator

* fix(stream): set correct backend when updating fill-history task to recalculate task.

* add new fill operator

* fix(stream): s2025-03-06 11:10:31.272
et ops always open in scan tsdb

* fix(stream):set the correct taskType for sink task.

* new fill operator

* adj stream fill operator

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* adj test

* fix issue

* fix(stream): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(stream): fix issue

* fix(stream): fix issue

* fix(stream): fix issue

* fix: ut com error

* fix(stream): fix mem leak and adjust operator type check rule

* fix(stream): fix mem leak and adjust test case

* refactor code

* fix(stream): free items.

* fix(stream): free fix memory leak.

* fix(stream): fix syntax error.

* fix: ignore unexpect block

* fix: adjust op type

---------

Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-14 20:14:01 +08:00
WANG MINGMING 3d053e2c9b
feat(stream): optimize client logic of creating stream if with history (#30059)
* feat:[TS-5617]use last_row cache mode if creating stream in fill_history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* fix: heap use after free

* feat: add log

* fix: ci case error

* fix: compile error in windows

* fix: ci case error

* fix: heap user after free

* fix: memory leak

* fix: ci case error

* fix: ci case error

* fix: ci case error

---------

Co-authored-by: yihaoDeng <yhdeng@taosdata.com>
2025-03-14 13:55:52 +08:00
Simon Guan abaf48ae15 Merge branch '3.0' into merge/mainto3.0 2025-03-10 11:07:48 +08:00
Simon Guan 1170955b95 Merge branch 'main' into merge/mainto3.0 2025-03-10 11:05:52 +08:00
Simon Guan 6c88048f77
Merge pull request #30002 from taosdata/refactor/TD-33926
refactor: do not compile test-related files when BUILD_TEST=OFF
2025-03-06 18:43:46 +08:00
Simon Guan c00d772492
Merge pull request #29649 from taosdata/feat/TS-5927-long-password
feat:[TS-5927]long-password
2025-03-06 17:25:48 +08:00
Simon Guan e321a3be12 refactor: cmake files 2025-03-06 13:49:31 +08:00
Simon Guan 292b0cb317 Merge branch 'main' into merge/mainto3.0 2025-03-05 18:10:31 +08:00
Haojun Liao 24882459e3 chore: merge main 2025-03-05 13:40:44 +08:00
Haojun Liao 04c2ecce69 fix(analytics): revise the test cases. 2025-03-05 13:10:41 +08:00
dongming chen e5fb75797a
Merge branch '3.0' into feat/TS-5927-long-password 2025-03-05 12:15:12 +08:00
Haojun Liao 2c92cca603 test(analytics): add test case and set the unit to be second. 2025-03-04 23:31:02 +08:00
Haojun Liao f563a35b3f fix(stream):check the maximum allowed forecast rows, add one test case, and do some internal refactor. 2025-03-03 17:27:05 +08:00
Simon Guan 6826edf5e0 Merge branch 'main' into merge/mainto3.0 2025-03-03 14:01:43 +08:00
Simon Guan 24fae6adbb
Merge pull request #29900 from taosdata/fix/checkfh
refactor(stream): scan wal is driven by time, instead of insert events.
2025-02-28 13:42:43 +08:00
Haojun Liao d279aaa22c test(stream): sleep a little bit longer. 2025-02-27 11:44:50 +08:00
Shengliang Guan 8fccf318ab
Merge pull request #29902 from taosdata/merge/mainto3.0
merge: from main to 3.0 branch
2025-02-24 15:47:21 +08:00
Shengliang Guan 3a1d06f4d7
Merge pull request #29354 from taosdata/enh/TS-5255/colsFunc2
enh: ts5255/cols func2
2025-02-24 14:19:37 +08:00
Shengliang Guan 3a739beaa5 Merge branch 'main' into merge/mainto3.0 2025-02-22 20:07:40 +08:00
WANG Xu 100136fa4e
Merge pull request #29890 from taosdata/test/new
Test/new
2025-02-21 21:07:40 +08:00
facetosea fa61c13a75 fix: _group_key not select 2025-02-21 00:40:11 +08:00
Shengliang Guan f16ae7abe1
Merge pull request #29809 from taosdata/merge/mainto3.0
merge: from main to 3.0 branch
2025-02-20 10:27:20 +08:00
huohong b1f409be37 new case arch demo 2025-02-20 09:32:15 +08:00
facetosea 00473a9c64 fix: aliasName too long 2025-02-18 13:06:13 +08:00
xiao-77 25a1af2b88 Fix ci problems. 2025-02-18 12:28:07 +08:00
Shengliang Guan 63fc5dea34 Merge branch '3.0' into merge/mainto3.0 2025-02-18 10:29:16 +08:00
dmchen e6e1d3ff1c feat/TS-5927-long-password-fix-case 2025-02-17 16:35:08 +08:00
dmchen c51c64196f retry case 2025-02-17 05:20:35 +00:00
dmchen 9ce37c3ec8 retry case 2025-02-17 04:54:49 +00:00
dmchen 2dcddb024f retry case 2025-02-17 03:26:51 +00:00
dmchen 6dc6a59b9b retry case 2025-02-17 02:53:32 +00:00
xiao-77 9b87d982a4 Fix modify s3 bucket name from global to local. 2025-02-17 09:26:26 +08:00
dmchen 9364bb63f9 retry case 2025-02-17 01:24:55 +00:00
dmchen 49173c6018 retry case 2025-02-14 09:06:59 +00:00
dmchen 1272524f7f retry case 2025-02-14 09:05:31 +00:00
dmchen cb8f132077 feat/TS-5927-long-password-retry-fail-case 2025-02-14 08:43:39 +00:00
dmchen 6d3d2b787a feat/TS-5927-long-password-retry-cases-fail 2025-02-14 08:18:34 +00:00
Shengliang Guan d6bd52ece7
Merge pull request #29750 from taosdata/enh/streamqueue
refactor(stream): add more operation handled in ctrl queue, and limit the stream input q size
2025-02-14 15:02:23 +08:00
dmchen 8f2370f3fa feat/TS-5927-long-password-fix-case 2025-02-14 01:12:09 +00:00
xiao-77 f09dd39f89 Fix ci problems. 2025-02-13 18:24:14 +08:00
xiao-77 d37897a75d Merge branch 'feat/3.0/TS-5584' of https://github.com/taosdata/TDengine into feat/3.0/TS-5584 2025-02-13 14:13:40 +08:00
xiao-77 0cd153aa88 Fix ci problems. 2025-02-13 14:10:24 +08:00
Zhixiao Bao e71c03a90f
Merge branch '3.0' into feat/3.0/TS-5584 2025-02-13 10:17:58 +08:00
xiao-77 540f8dc95d Fix ci problems. 2025-02-13 10:06:47 +08:00
dmchen 48123446d6 feat/TS-5927-long-password-fix-cases 2025-02-13 00:46:08 +00:00
dmchen 36caaf7aab feat/TS-5927-long-password-length 2025-02-12 09:50:56 +00:00
dongming chen 2fd8792c7a
Merge branch '3.0' into feat/TS-5927-long-password 2025-02-12 15:26:41 +08:00