Commit Graph

629 Commits

Author SHA1 Message Date
Haojun Liao bff0ad89a4 other: merge 3.0 2024-01-11 09:51:12 +08:00
Haojun Liao 1c562bd535 refactor: enable the restart of stream tasks. 2024-01-11 09:41:40 +08:00
Haojun Liao 9294c8d4ee fix(stream): enable the hb to mnode for stream meta when it becomes the leader. 2024-01-11 09:20:34 +08:00
wangmm0220 df69a89da2 fix:[TD-28185]add pause & resume logic for snode 2024-01-10 18:46:10 +08:00
Haojun Liao 9d0d0d9ddc Merge branch '3.0' into fix/3_liaohj 2024-01-10 09:57:46 +08:00
Haojun Liao dc0c5539a0 fix(stream): remove invalid timer controllers. 2024-01-10 08:48:49 +08:00
Haojun Liao f9d5c0d403 fix(stream): fix error in tq timer. 2024-01-09 09:13:53 +08:00
Haojun Liao 70c1e66959 fix(stream): release task after launch stream tasks. 2024-01-08 18:19:31 +08:00
Yihao Deng 4c5441da7e refactor stream code 2024-01-08 08:38:16 +00:00
Haojun Liao 02f6c971fd fix(stream): check for not ready task when rsp returned. 2024-01-06 19:43:39 +08:00
Haojun Liao 0c4b91dc72 enh(stream): start task async. 2024-01-05 17:55:39 +08:00
Haojun Liao 1912de9b49 enh(stream): scan wal by using timer to trigger it. 2024-01-05 14:49:33 +08:00
Haojun Liao 4df83e92cd fix(stream): fix syntax error. 2024-01-04 19:14:24 +08:00
Haojun Liao 6d37e596ec refactor: do some internal refactor. 2024-01-04 18:47:49 +08:00
Haojun Liao c494252002 fix(stream): record the related fill-history task failure. 2024-01-04 17:44:05 +08:00
Haojun Liao 185fa75376 enh(stream): add task into timer to be idle during exec when the corresponding output queue is full. 2024-01-03 17:59:42 +08:00
wangmm0220 8add4dc177 fix:[TD-28032]concat subtable and hash val as new subtable name to avoid data from multi time line write to one time line 2024-01-02 18:37:29 +08:00
Haojun Liao e7a0044502 Merge branch '3.0' into fix/3_liaohj 2024-01-02 13:02:55 +08:00
Alex Duan 28238fa7dc coverage: no call function with BUILD_NO_CALL MACRO define 2023-12-31 18:29:37 +08:00
Haojun Liao 62cfcefbb0 refactor: remove stream-scan-history event for stream task. 2023-12-26 00:34:04 +08:00
Haojun Liao 60977d56bd refactor: set attribute only on Linux 2023-12-25 12:47:16 +08:00
Haojun Liao 0877e0bfad other: merge 3.0 2023-12-25 09:13:31 +08:00
Haojun Liao cd85dae3fd fix(stream): fix the dead lock. 2023-12-24 23:30:00 +08:00
Haojun Liao 30498a4028 refactor: do some internal refactor. 2023-12-24 02:24:46 +08:00
Haojun Liao 25f798e42f fix(stream): record the start failure tasks 2023-12-23 23:29:30 +08:00
Haojun Liao c61203b55a fix: fix deadlock. 2023-12-23 16:37:21 +08:00
Haojun Liao 8a714db17d fix: fix deadlock. 2023-12-23 15:33:50 +08:00
Haojun Liao 5c1ff76082 fix(stream): set rwlock attr. 2023-12-23 00:58:52 +08:00
Haojun Liao 66bdace9eb fix(stream): check rsp. 2023-12-23 00:09:55 +08:00
Haojun Liao 80604ec8e2 fix(stream): init rwlock. 2023-12-22 23:31:51 +08:00
Haojun Liao b175a4b7b3 refactor(stream): async stop tasks, and do some internal refactor. 2023-12-22 17:56:16 +08:00
Haojun Liao 7ce545bcd4 refactor: refactor the lock type. 2023-12-22 16:14:49 +08:00
Haojun Liao dce0a6c74a fix(stream): remove invalid assert 2023-12-22 15:24:37 +08:00
Haojun Liao e91afa7df2 fix(stream): fix deadlock. 2023-12-22 14:51:04 +08:00
Haojun Liao 90898a16e0 refactor: do some internal refactor. 2023-12-22 13:57:15 +08:00
Haojun Liao c9475060de fix(stream): fix race condition. 2023-12-22 09:34:49 +08:00
Haojun Liao 84583580e2 fix(stream): transfer the checkpoint trans id. 2023-12-21 19:12:49 +08:00
Haojun Liao 2acc8423c5 fix(stream): send msg to mnode before closing all tasks. 2023-12-21 16:00:50 +08:00
Haojun Liao 849aaf8d4f fix(stream): avoid dead lock. 2023-12-20 18:07:53 +08:00
yihaoDeng d6297e26ce init backend mutex 2023-12-20 16:56:12 +08:00
Haojun Liao 1a8583887f fix(stream): remove clear backendcache. 2023-12-20 15:35:43 +08:00
Haojun Liao 17e9e84aa9 Merge branch '3.0' into fix/3_liaohj 2023-12-19 16:43:08 +08:00
Haojun Liao fee198f9ad refactor: do some internal refactor. 2023-12-19 16:21:11 +08:00
Haojun Liao ec3183b6a8 other: merge 3.0 2023-12-18 23:16:37 +08:00
Haojun Liao 612b50c4cd
Merge pull request #24097 from taosdata/fix/TD-27834
fix stream backend crash when transfer
2023-12-18 23:13:08 +08:00
Haojun Liao e734569de0 fix(stream): fix dead lock. 2023-12-18 18:42:57 +08:00
Haojun Liao bba5a4f08b other: merge main into 3.0 2023-12-18 18:26:42 +08:00
Haojun Liao ea6e78cbaa fix(stream): add async call restart, instead of sync wait. 2023-12-18 16:33:44 +08:00
yihaoDeng 0840e6a0d8 fix stream backend crash when transfer 2023-12-18 11:19:15 +08:00
yihaoDeng 547d797a26 fix stream backend crash when transfer 2023-12-16 15:05:27 +08:00
Haojun Liao 81f96603b0 refactor: do some internal refactor. 2023-12-14 09:33:51 +08:00
wangmm0220 f06117dbc9 fix:[TD-27644]heartbeat closed before snode close leading to snode is hanged 2023-12-01 17:32:29 +08:00
Haojun Liao d7a19a9179 fix(stream): fix invalid decode hbmsg and memory leaks caused by invalid hb msg. 2023-12-01 14:45:18 +08:00
Haojun Liao dba58f5e38 fix(stream): fix syntax error. 2023-12-01 11:54:13 +08:00
yihaoDeng 08cc976975 fix crash 2023-11-29 19:57:30 +08:00
yihaoDeng 6416a6153a Merge branch '3.0' into enh/refactorBackend 2023-11-24 16:29:09 +08:00
yihaoDeng 0eba15b4a4 Merge branch '3.0' into enh/refactorBackend 2023-11-22 10:57:41 +08:00
Haojun Liao 785d91c17b
Merge pull request #23661 from taosdata/feat/TD-26060
fix: errors in snode in conditions like fill history/checkpoint/restart
2023-11-21 19:02:38 +08:00
yihaoDeng 939aa7a3cc Merge branch '3.0' into enh/refactorBackend 2023-11-20 17:33:30 +08:00
Haojun Liao 1b4f187ded
Merge pull request #23743 from taosdata/fix/pause_stream
fix(stream): remove invalid assert
2023-11-17 23:48:47 +08:00
Haojun Liao 753716f1e6 fix(stream): remove invalid assert 2023-11-17 16:36:17 +08:00
yihaoDeng 286413abbb fix stream backend convert 2023-11-17 16:14:51 +08:00
Haojun Liao b12e5050ea fix(stream): remove invalid assert 2023-11-17 15:57:46 +08:00
yihaoDeng f5ef2e727f fix stream backend convert 2023-11-16 14:41:04 +08:00
wangmm0220 a16189d2a9 fix:confilct from 3.0 2023-11-15 19:14:27 +08:00
yihaoDeng f4d6cf7c74 Merge branch '3.0' into enh/refactorBackend 2023-11-14 18:12:48 +08:00
Haojun Liao f99e22b91f fix(stream): fix deadlock. 2023-11-13 23:56:42 +08:00
wangmm0220 56381dea90 fix:conflict from 3.0 2023-11-13 18:42:59 +08:00
wangmm0220 1cbe568be7 fix:add logic when snode restart 2023-11-13 18:39:42 +08:00
Haojun Liao aad8a2eb87 fix(stream): fix deadlock. 2023-11-13 18:20:30 +08:00
yihaoDeng 1786f7077d Merge branch '3.0' into enh/refactorBackend 2023-11-13 14:35:23 +08:00
yihaoDeng de7f2a2d1a Merge branch '3.0' into enh/refactorBackend 2023-11-13 11:11:28 +08:00
Haojun Liao 4ff8907b58 fix: fix syntax error. 2023-11-11 23:05:20 +08:00
Haojun Liao 9ee9174608 other: merge 3.0 2023-11-11 15:48:31 +08:00
Haojun Liao 3bacd7516e fix(stream): extact stream nodes list if not initialized. 2023-11-11 15:40:35 +08:00
Haojun Liao 15430f4d50 refactor: 2023-11-11 15:40:33 +08:00
Haojun Liao 5fcef5bd89 refactor:add some logs. 2023-11-11 15:40:32 +08:00
Haojun Liao 7a23df4b1a fix(stream): set the correct updated nodeId. 2023-11-11 15:40:32 +08:00
Haojun Liao af08a189c1 fix(stream): disable concurrently restart stream tasks. 2023-11-11 15:40:10 +08:00
Haojun Liao 08a9244230 fix(stream): extact stream nodes list if not initialized. 2023-11-10 16:24:20 +08:00
Haojun Liao d0307e5876 refactor: 2023-11-10 15:30:21 +08:00
Haojun Liao 0b36081158 refactor:add some logs. 2023-11-10 14:36:18 +08:00
wangmm0220 8a05bad899 fix:snode error 2023-11-10 14:14:22 +08:00
Haojun Liao 1f9a58361d fix(stream): set the correct updated nodeId. 2023-11-10 13:36:35 +08:00
Haojun Liao 9978667c48 fix(stream): disable concurrently restart stream tasks. 2023-11-10 09:24:08 +08:00
yihaoDeng 0669037de0 fix mem leak 2023-11-09 20:37:08 +08:00
yihaoDeng 5f9b922a04 support s3 2023-11-09 20:15:08 +08:00
yihaoDeng 757425bff8 Merge branch '3.0' into enh/refactorBackend 2023-11-09 17:58:49 +08:00
yihaoDeng 3abf0ae4b2 Merge branch 'feat/TD-26060' into enh/refactorBackend 2023-11-08 17:31:16 +08:00
yihaoDeng 92fa1043b8 Merge branch 'enh/refactorBackend' of https://github.com/taosdata/TDengine into enh/refactorBackend 2023-11-08 17:12:32 +08:00
yihaoDeng 76fcc49c6d fix stream transfer error 2023-11-08 17:10:54 +08:00
Haojun Liao 470f244032 Merge remote-tracking branch 'origin/enh/refactorBackend' into enh/refactorBackend
# Conflicts:
#	include/libs/stream/tstream.h
2023-11-08 15:21:21 +08:00
Haojun Liao 99dbbe61d7 refactor(stream): do some internal refactor. 2023-11-08 15:20:47 +08:00
yihaoDeng 7b0891981e merge 3.0 2023-11-08 14:14:01 +08:00
kailixu b9f389a49f Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-08 08:36:35 +08:00
kailixu 6b738884a4 chore: rsma checkpoint 2023-11-07 19:59:05 +08:00
Haojun Liao 7370daa6e0 fix(stream): record the start failure tasks. 2023-11-07 16:39:56 +08:00
Haojun Liao e38e526dda fix(stream):reset the starting flag for the followers. 2023-11-07 16:39:12 +08:00
Haojun Liao 6b1889284b fix(stream): record the start failure tasks. 2023-11-06 15:45:44 +08:00
Haojun Liao 69863febe6 fix(stream):reset the starting flag for the followers. 2023-11-06 13:17:13 +08:00
Haojun Liao d41fd27189 fix(stream): restart tasks in stream threads, instead of write thread. 2023-11-06 11:57:21 +08:00
Haojun Liao 680eb5674e refactor: do some internal refactor. 2023-11-06 09:15:49 +08:00
Haojun Liao e13f003b90 refactor: do some internal refactor. 2023-11-05 01:23:23 +08:00
kailixu 411151d671 fix: buffer overflow/buffer use after free/memory leak 2023-11-04 08:44:26 +08:00
yihaoDeng 24e961976f Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-03 09:31:57 +08:00
Haojun Liao 706f1e4744 fix(stream): fix deadlock in transfer state. 2023-11-02 18:18:04 +08:00
Haojun Liao 39175dc03b fix(stream): fix deadlock in transfer state. 2023-11-02 18:07:36 +08:00
yihaoDeng 7e7a39f077 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-02 10:15:34 +08:00
yihaoDeng b911492270 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-02 09:46:36 +08:00
Haojun Liao dc8a90c864 fix(stream): fix memory leak. 2023-11-02 08:13:11 +08:00
Haojun Liao 79e971385f fix(stream): fix memory leak. 2023-11-02 08:11:51 +08:00
Haojun Liao 6dc468c55f fix(stream): fix dead-lock 2023-11-01 19:38:13 +08:00
Haojun Liao fa7ebf63ea refactor: do some internal refactor. 2023-11-01 19:38:13 +08:00
Haojun Liao 7cf90dde5c refactor: do some internal refactor. 2023-11-01 19:38:13 +08:00
Haojun Liao 00f029e44f refactor: add some logs. 2023-11-01 19:38:12 +08:00
Haojun Liao e7609d8e56 fix(stream): fix dead-lock 2023-11-01 19:09:21 +08:00
Haojun Liao ae25f09c45 refactor: do some internal refactor. 2023-11-01 18:03:44 +08:00
Haojun Liao 246ed4e022 refactor: do some internal refactor. 2023-11-01 17:19:21 +08:00
Haojun Liao 12bc4f32cf refactor: add some logs. 2023-11-01 16:12:15 +08:00
yihaoDeng 4c8edb5545 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-01 16:07:49 +08:00
Haojun Liao 0b18192b23 fix(stream): fix error in finishing event handling. 2023-10-31 16:40:16 +08:00
Haojun Liao 4c0242a58d fix(stream): fix error in finishing event handling. 2023-10-31 16:39:14 +08:00
Haojun Liao 85befa10b9 fix(stream): avoid clear the restart flag . 2023-10-30 09:32:39 +08:00
Haojun Liao c7285c0ede fix(stream): add expired epset node list for stream tasks in hb to mnode. 2023-10-27 23:41:57 +08:00
Haojun Liao 4c267b538f fix(stream): add expired epset node list for stream tasks in hb to mnode. 2023-10-27 23:05:41 +08:00
Haojun Liao afe7fdf672 fix(stream): avoid clear the restart flag . 2023-10-27 14:12:06 +08:00
Haojun Liao 454624516c fix(stream): handle the repeatly issued nodeUpdate msg from dnode. 2023-10-27 09:59:51 +08:00
Haojun Liao 9c9cd08e2e fix(stream): not clear role flag. 2023-10-27 09:59:44 +08:00
Haojun Liao 1ccbb77981 fix(stream): fix dead lock when shutdown all tasks. 2023-10-27 09:59:37 +08:00
Haojun Liao b0ef072556 fix(stream): clear flag for checkpoint. 2023-10-27 09:55:07 +08:00
Haojun Liao e1f6043eed fix(stream): fix bugs caused by refactor sm. 2023-10-27 09:54:57 +08:00
Haojun Liao 1eac7f44c0 fix(stream): fix error in stream. 2023-10-27 09:54:56 +08:00
Haojun Liao 579dd8e6b2 refactor(stream): do some internal refactor. 2023-10-27 09:54:56 +08:00
Haojun Liao 6f3c0d17cc refactor(stream): add state machine to manage the state of stream tasks. 2023-10-27 09:54:49 +08:00
Haojun Liao 596eb073b7 refactor(stream): do some internal refactor. 2023-10-27 09:54:48 +08:00
yihaoDeng 87db93e72e fix stream case error 2023-10-26 11:36:29 +08:00
yihaoDeng db0f6258fe fix stream case error 2023-10-25 19:38:22 +08:00
Haojun Liao 03165d5327 fix(stream): handle the repeatly issued nodeUpdate msg from dnode. 2023-10-25 09:55:02 +08:00
Haojun Liao 5b42204dba fix(stream): not clear role flag. 2023-10-25 00:32:46 +08:00
Haojun Liao 8cde39eebd fix(stream): fix dead lock when shutdown all tasks. 2023-10-24 19:11:24 +08:00
Haojun Liao 059e94e428 fix(stream): clear flag for checkpoint. 2023-10-22 01:25:00 +08:00
yihaoDeng 1a4cfd11a5 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-10-20 15:28:10 +08:00
Haojun Liao 270a847809 fix(stream): fix bugs caused by refactor sm. 2023-10-20 10:23:48 +08:00
Haojun Liao c1cebae6ba fix(stream): fix error in stream. 2023-10-19 16:20:27 +08:00
Haojun Liao 402aefd95d refactor(stream): do some internal refactor. 2023-10-19 15:09:47 +08:00
Haojun Liao f4caeca24a refactor(stream): add state machine to manage the state of stream tasks. 2023-10-19 10:34:20 +08:00
Haojun Liao 54ec679b58 refactor(stream): do some internal refactor. 2023-10-18 15:34:53 +08:00
yihaoDeng 2004c1a346 fix transfer crash 2023-10-17 18:19:52 +08:00
yihaoDeng 77542cc461 fix stream snap deadlock 2023-10-17 11:23:39 +08:00
Haojun Liao 33d9f772be other: merge 3.0 2023-10-13 09:06:13 +08:00
Haojun Liao 84c5359d5b other: merge 3.0 2023-10-13 08:58:26 +08:00
yihaoDeng a56d831a14 refactor backend 2023-10-12 20:16:41 +08:00
yihaoDeng 861d26e356 refactor backend 2023-10-12 15:24:50 +08:00
yihaoDeng 4c94fbb44f refactor backend 2023-10-11 20:26:19 +08:00
yihaoDeng f264ffdcd6 refactor backend 2023-10-11 20:12:07 +08:00
yihaoDeng 83b52a4c1d refactor backend 2023-10-11 20:02:40 +08:00
yihaoDeng 031506aef8 refactor backend 2023-10-11 19:46:35 +08:00
Haojun Liao 05dfadd8f8 fix(stream): set initial version for fill-history task. disable reentry of scan-history task function. 2023-10-10 15:52:48 +08:00
Haojun Liao 5f9b858fc1 other: merge main. 2023-10-10 15:38:46 +08:00
Haojun Liao adf6f65d09 fix(stream): set initial version for fill-history task. disable reentry of scan-history task function. 2023-10-10 14:00:30 +08:00
Haojun Liao 21fb502690 enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks. 2023-10-09 20:10:26 +08:00
Haojun Liao af2c0c95e9 refactor: do some internal refactor. 2023-10-09 20:10:22 +08:00
Haojun Liao 59e284c332 fix(stream): quit from loop when input queue is full. 2023-10-09 20:10:21 +08:00
Haojun Liao 9d5a3b8d78 other: merge stream fix. 2023-10-09 20:10:13 +08:00
Haojun Liao c5997bee5b fix(stream): update the current processing version. 2023-10-09 20:09:23 +08:00
Haojun Liao 28008f8263 enh(stream): add more info for sink task. 2023-10-09 20:09:22 +08:00
Haojun Liao cf14772dd9 enh(stream): optimize the scan wal perf 2023-10-09 20:09:21 +08:00
Haojun Liao 7682c033ad fix(stream): add null check. 2023-10-09 20:09:21 +08:00
Haojun Liao 0ab5d4b342 refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao c295ec322a refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao 583ac1fc45 refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao 83f84d92bc refactor: do some internal refactor (discard the checkpoint source msg during restoring). 2023-10-09 20:09:10 +08:00
Haojun Liao bb4ba54f28 fix(stream): enable new time range for stream task. 2023-10-09 20:09:09 +08:00
Haojun Liao dfe8641c0c refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-10-09 20:08:52 +08:00
Haojun Liao f2848de12a fix(stream): reduce the counter when remove stream tasks. 2023-10-09 20:08:52 +08:00
Haojun Liao 2c0e54f4f7 fix(stream): update the retry interval. 2023-10-09 20:08:13 +08:00
Haojun Liao 0db357401d fix(stream): add new status for meta. 2023-10-09 20:08:12 +08:00
Haojun Liao 3912d80c02 fix(stream): do some internal refactor. 2023-10-09 20:08:11 +08:00
Haojun Liao a32b56f381 fix(stream): fix memory leak. 2023-10-09 20:08:10 +08:00
Haojun Liao 4af7812f0b fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read. 2023-10-09 20:08:10 +08:00
Haojun Liao 1f45bd82d2 other: merge stream fix in main. 2023-10-09 20:07:01 +08:00
Haojun Liao e27111ecd1 enh(stream): add node stage check. 2023-10-09 20:06:09 +08:00
Haojun Liao b95ad74c7f fix(stream): not handle the check msg for follower tasks. 2023-10-09 20:06:04 +08:00
Haojun Liao e6fb9ee46c fix(stream): reset task counter. 2023-10-09 20:06:04 +08:00
Haojun Liao c3a3cf21fb fix(stream): add some logs. 2023-10-09 20:05:39 +08:00
Haojun Liao 9d6199adf7 fix(stream): fix invalid read. 2023-10-09 20:05:37 +08:00
Haojun Liao 0459a4628c fix(stream): fix stream task id error. 2023-10-09 20:05:36 +08:00
Haojun Liao 7ccc273206 other: merge stream fix in main. 2023-10-09 20:05:29 +08:00
Haojun Liao 98ef566eb4 fix(stream): remove fill-history sink task. 2023-10-09 20:04:17 +08:00
Haojun Liao e84eeee6b8 refactor: do some internal refactor. 2023-10-09 20:04:17 +08:00
Haojun Liao c33e521aa1 fix(stream): update some logs. 2023-10-09 20:04:17 +08:00
Haojun Liao 39e0c57323 fix(stream): set the correct guard flag. 2023-10-09 20:04:16 +08:00
Haojun Liao 3d1f29ff21 fix(stream): fix memory leak. 2023-10-09 20:04:16 +08:00
Haojun Liao 6fa54789cb fix(stream): disable follower send hb to mnode. and do some internal refactor. 2023-10-09 19:59:31 +08:00
Haojun Liao 1256eafddb other: merge stream fix. 2023-10-09 19:59:24 +08:00
yihaoDeng 28b546a4ad fix mem leak 2023-10-09 14:25:57 +08:00
yihaoDeng be42143362 Merge branch '3.0' into case/TD-26408-3.0 2023-10-09 14:24:31 +08:00
yihaoDeng 3f9784f475 fix mem leak 2023-10-09 14:07:22 +08:00
Haojun Liao ee6516f3c4 enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks. 2023-10-09 10:13:53 +08:00
yihaoDeng f49a2cbeb0 refact task backend 2023-10-08 18:14:17 +08:00