cfcdfdd2e7
fix(stream): update the update node info. ( #30364 )
2025-03-24 10:18:06 +08:00
dfe32225de
chore: merge main.
2025-03-17 14:51:11 +08:00
ab92886820
fix(stream): reduce the consensus checkpoint id trans. ( #30105 )
...
* fix(stream): reduce the consensus checkpoint id trans.
* refactor(stream): add some logs.
* refactor(stream): set the max checkpoint exec time 30min.
* refactor(stream): add checkpoint-consensus trans conflict check.
* refactor(stream): remove unused local variables.
* fix(stream): fix syntax error.
* fix(stream): 1. fix free memory error 2. continue if put result into dst hashmap failed.
* fix issue
* fix issue
* fix(mnd): follower mnode not processes the timer event.
* fix(stream): print correct error msg.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): truncate long subtable name
* fix(stream): add buffer len.
* refactor(stream): update some logs.
* fix issue
* refactor(stream): update some logs.
* refactor(stream): update some logs.
* fix(stream): check return value.
* fix(stream): fix syntax error.
* fix(stream): check return value.
* fix(stream): update the timer check in mnode.
* fix(stream): add restart stage tracking.
* fix(stream): track the start task stage for meta.
* fix(stream): fix error in log.
* refactor(stream): adjust log info.
* fix mem issue
* fix(stream): check the number of required tasks for consensus checkpointId.
* fix(stream): lock the whole start procedure.
* fix(stream): add lock during start all tasks.
* fix(stream): update logs.
* fix(stream): update logs.
* fix(stream): update logs.
* fix(stream): fix dead-lock.
* fix(stream): fix syntax error.
* fix(stream): not drop the scan-history task.
* fix(stream): fix syntax error.
* fix(stream): wait for executor stop before restarting.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): disable some logs.
* fix(stream): reset the start info if no task left.
---------
Co-authored-by: 54liuyao <54liuyao@163.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
2025-03-17 10:20:17 +08:00
ac23a4fc55
feat(stream): support stream processing for virtual tables ( #30144 )
...
* enh: add client processing
* enh: add mnode vtables processing
* enh: add mnode vtable processing
* enh: add normal child vtable support
* fix: compile issues
* fix: compile issues
* fix: create stream issues
* fix: multi stream scan issue
* fix: remove debug info
* fix: agg task and task level issues
* fix: correct task output type
* fix: split vtablescan from agg
* fix: memory leak issues
* fix: add limitations
* Update 09-error-code.md
* Update 09-error-code.md
* fix: remove usless case
* feat(stream): extract original table data in source scan task
Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.
* feat(stream): multi-way merge using loser tree in virtual merge task
Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing. Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.
* fix(test): remove useless cases
---------
Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
2de6736160
feat(stream): add new trigger continuous_window_close ( #30125 )
...
* opt stream build twa result
* opt force window close memory
* feat(stream):optimize new interval and scan operator
* adj log
* opt code
* opt code
* fill history
* fix issue for fill history
* add ci
* feat(stream): add new stream nonblock interval operator
* adjust code
* use new scan operator
* use new scan operator
* add log
* fix issue
* recover stream scan next function
* fix issue
* fix issue
* fix issue
* ignore disorder data
* fix issue
* fix issue for interval sliding
* fix issue
* fix ci issue
* fix ci issue
* fix ci issue
* add semi && final nonblock interval operator
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* refactor(stream): track the msgId for each upstream tasks.
* fix(stream): fix race condition.
* fix(stream): update the task last msgId when putting into input queue succ.
* fix issue
* fix issue
* put recalculate data to rocksdb
* fix issue
* fix issue
* enh(query)[TD-33071]: add support for saving and restoring tsdbReader scan progress
- Implement functionality to save scan progress during tsdbReader operations
- Enable resuming scans from the last saved position
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* add rec interval check
* enh(stream):add recalculate tasks.
* enh(stream): support the re-calculate the tasks.
* fix issue && do refactor
* do refactor
* fix issue
* fix issue
* update backend opt
* add new interface
* add new session operator
* support blob
* add new session operator
* fix issue
* add rec state for rec task
* fix invalid read
* add new session window
* enh(stream): update the stream tasks backend.
* new session operator
* add pull data
* fix(stream): fix error in expand stream backend.
* fix issue
* fix issue
* fix issue
* merge code
* fix issue
* fix(stream): check for null ptr.
* fix(stream): add more check.
* fix issue
* fix issue
* fix issue
* add debug code
* fix issue
* fix issue
* fix issue
* set rec end flag
* fix(stream): opt re-calculate stream tasks.
* fix issue
* fix issue
* add new operator
* enh(stream): dispatch recalculate block to agg tasks.
* fix issue
* fix issue
* fix(stream): adjust filter.
* fix issue
* refactor
* refactor(stream): adjust the recalculate end block.
* fix issue
* fix(stream): set correct create trigger block.
* fix issue
* fix(stream): fix error in build recalculate end block.
* fix(stream): check null ptr.
* add stream client && fix issue
* fix mem leak
* fix(stream): free msg.
* add stream client
* fix(stream): fix error.
* add stream client && fix issue
* add stream client
* refactor(stream): set the recalculate task complete.
* add wend and group_id for session window dest table
* feat(stream): refactor and set the recalcul agg task complete.
* add cfg for adapter
* fix issue
* add state && event operator
* feat(stream): support fill-history task.
* add new fill operator
* fix(stream): set correct backend when updating fill-history task to recalculate task.
* add new fill operator
* fix(stream): s2025-03-06 11:10:31.272
et ops always open in scan tsdb
* fix(stream):set the correct taskType for sink task.
* new fill operator
* adj stream fill operator
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* adj test
* fix issue
* fix(stream): fix issue
* fix(steam): fix issue
* fix(steam): fix issue
* fix(steam): fix issue
* fix(steam): fix issue
* fix(stream): fix issue
* fix(stream): fix issue
* fix(stream): fix issue
* fix: ut com error
* fix(stream): fix mem leak and adjust operator type check rule
* fix(stream): fix mem leak and adjust test case
* refactor code
* fix(stream): free items.
* fix(stream): free fix memory leak.
* fix(stream): fix syntax error.
* fix: ignore unexpect block
* fix: adjust op type
---------
Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-14 20:14:01 +08:00
d24bc139a5
chore: merge 3.0
2025-03-10 18:31:56 +08:00
5cfdfc74d9
fix(stream): remove duplicated code.
2025-03-06 18:47:10 +08:00
95e75196e2
chore: merge main.
2025-03-06 18:45:21 +08:00
16ada1f0d3
Merge branch 'main' into merge/mainto3.0
2025-03-04 11:24:13 +08:00
6826edf5e0
Merge branch 'main' into merge/mainto3.0
2025-03-03 14:01:43 +08:00
c726420081
Merge branch 'main' into fix/droptask
2025-02-28 19:42:35 +08:00
8c3035cf24
fix(stream): avoid lock in timer to avoid sync operation blocks.
2025-02-28 17:47:43 +08:00
e1c4451f81
fix(stream): avoid lock in timer to avoid sync operation blocks.
2025-02-28 17:46:54 +08:00
00e02280b8
fix(stream): adjust the error code position.
2025-02-28 14:01:26 +08:00
67853b4a54
enh(stream): add checkpoint queue for source tasks.
2025-02-28 13:46:40 +08:00
24fae6adbb
Merge pull request #29900 from taosdata/fix/checkfh
...
refactor(stream): scan wal is driven by time, instead of insert events.
2025-02-28 13:42:43 +08:00
bf1f162aa0
fix(stream): drop task in synch model in write thread.
2025-02-27 00:15:36 +08:00
012ee53d35
fix(stream): drop task in synch model in write thread.
2025-02-27 00:15:31 +08:00
c14ba08d7c
refactor(stream): drop task in async ways
2025-02-27 00:12:11 +08:00
9018970135
refactor: do some internal refactor.
2025-02-23 01:39:00 +08:00
034c907ee1
refactor: do some internal refactor.
2025-02-23 01:38:05 +08:00
3a739beaa5
Merge branch 'main' into merge/mainto3.0
2025-02-22 20:07:40 +08:00
ecc667c22b
Revert "refactor(stream): drop task in async ways"
2025-02-21 01:10:18 +08:00
2bee60a41a
Merge branch 'main' into merge/mainto3.0
2025-02-20 18:12:02 +08:00
6766e6db53
refactor(stream): add checkpoint queue to handle chkpt.
2025-02-20 16:13:33 +08:00
4298522792
enh(stream): add checkpoint queue for source tasks.
2025-02-20 16:06:34 +08:00
c4cd6dd62d
fix(stream): drop task in synch model in write thread.
2025-02-17 10:06:09 +08:00
fc2d2672b9
feat(stream)[TS-5469]: add more stream event notifications
...
- Add event notifications for all window types
- Implement window invalidation notifications
- Add statistics for event notification delivery
2025-02-16 11:07:16 +08:00
465e5eabd7
fix(stream): drop task in synch model in write thread.
2025-02-15 23:20:05 +08:00
fd96b096ea
refactor(stream): drop task in async ways
2025-02-13 23:41:06 +08:00
eb5d463490
feat(stream)[TS-5469]. add support for window event notifications in stream processing
...
- Introduce new syntax to specify notification type and destination address
- Collect relevant event information during window computations
- Implement websocket-based notification delivery to the specified address
2025-01-21 17:39:46 +08:00
d2146a2efb
other: merge main
2024-12-17 09:23:13 +08:00
aa12038fc4
fix(stream): record the failure of dispatch msg, and set the update node id.
2024-12-10 17:58:53 +08:00
6db8a25c98
Merge branch 'main' into merge/mainto3.0
2024-11-27 22:31:17 +08:00
96d4a2908e
fix(stream): check failedId before sending checkpoint msg
2024-11-23 02:12:15 +08:00
1c66f22c2b
refactor: move to msg.
2024-11-18 15:00:30 +08:00
f65651f6ef
fix(stream): update the msg encoder.
2024-11-15 00:19:35 +08:00
b2b5a14d9d
fix(stream): reset task add the failed chkptId info.
2024-11-14 17:38:40 +08:00
9a92c136ce
fix drop child table with tsma
2024-11-08 09:47:47 +08:00
ba31bf2c52
Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837
2024-11-06 09:42:50 +08:00
a685c1394a
refactor: do some internal refactor.
2024-10-31 15:20:18 +08:00
c0932d419d
fix(stream): fix error in pause/resume for force_window_close
2024-10-28 18:07:11 +08:00
380f433499
fix(stream): use the refId in stream meta list, in order to avoid access already freed stream tasks.
2024-10-27 15:40:17 +08:00
6a2a5b80a1
Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837
2024-10-18 09:28:18 +08:00
e825d74213
Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837
2024-10-17 17:12:00 +08:00
c8e8cb0697
refactor: do some internal refactor.
2024-10-17 11:48:00 +08:00
dd05353b74
refactor: do some internal refactor.
2024-10-16 22:07:37 +08:00
d13819bbf4
Merge branch '3.0' of https://github.com/taosdata/TDengine into fix/TD-30837
2024-09-30 09:05:46 +08:00
4c98786352
fix(stream): use meta id instead of ptr.
2024-09-29 16:12:46 +08:00
c5eca0486e
Merge remote-tracking branch 'origin/fix/TD-30837' into fix/TD-30837
...
# Conflicts:
# source/dnode/vnode/src/tq/tq.c
# source/libs/stream/inc/streamInt.h
# source/libs/stream/src/streamSched.c
2024-09-19 16:39:54 +08:00
Haojun Liao
Haojun Liao
Jinqing Kuang
liuyao
Simon Guan
wangjiaming0909