Commit Graph

605 Commits

Author SHA1 Message Date
Haojun Liao d440da0d8c enh(stream): execute scan history by using time slice 1s. 2023-11-05 01:26:53 +08:00
kailixu 77eadc59dc Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-03 16:26:07 +08:00
yihaoDeng 24e961976f Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-03 09:31:57 +08:00
Haojun Liao 7cf90dde5c refactor: do some internal refactor. 2023-11-01 19:38:13 +08:00
Haojun Liao 00f029e44f refactor: add some logs. 2023-11-01 19:38:12 +08:00
Haojun Liao 246ed4e022 refactor: do some internal refactor. 2023-11-01 17:19:21 +08:00
Haojun Liao 12bc4f32cf refactor: add some logs. 2023-11-01 16:12:15 +08:00
yihaoDeng 4c8edb5545 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-01 16:07:49 +08:00
kailixu 1530fe0c49 Merge branch '3.0' into enh/TD-23769-3.0x 2023-11-01 09:52:16 +08:00
kailixu ad1e6accd4 chore: build checkpoint for rsma 2023-11-01 09:45:58 +08:00
Haojun Liao 0b18192b23 fix(stream): fix error in finishing event handling. 2023-10-31 16:40:16 +08:00
Haojun Liao 4c0242a58d fix(stream): fix error in finishing event handling. 2023-10-31 16:39:14 +08:00
Haojun Liao 86dfc52a6e fix(stream): add event check when handling the end of event. 2023-10-31 10:45:12 +08:00
Haojun Liao 19c5882cbe fix(stream): add event check when handling the end of event. 2023-10-30 10:22:24 +08:00
Haojun Liao c7285c0ede fix(stream): add expired epset node list for stream tasks in hb to mnode. 2023-10-27 23:41:57 +08:00
Haojun Liao 4c267b538f fix(stream): add expired epset node list for stream tasks in hb to mnode. 2023-10-27 23:05:41 +08:00
Haojun Liao 454624516c fix(stream): handle the repeatly issued nodeUpdate msg from dnode. 2023-10-27 09:59:51 +08:00
Haojun Liao 1ccbb77981 fix(stream): fix dead lock when shutdown all tasks. 2023-10-27 09:59:37 +08:00
Haojun Liao b0ef072556 fix(stream): clear flag for checkpoint. 2023-10-27 09:55:07 +08:00
Haojun Liao 091238e7be fix(stream): fix bugs caused by refactor sm. 2023-10-27 09:54:58 +08:00
Haojun Liao e1f6043eed fix(stream): fix bugs caused by refactor sm. 2023-10-27 09:54:57 +08:00
Haojun Liao 1eac7f44c0 fix(stream): fix error in stream. 2023-10-27 09:54:56 +08:00
Haojun Liao 579dd8e6b2 refactor(stream): do some internal refactor. 2023-10-27 09:54:56 +08:00
Haojun Liao 6f3c0d17cc refactor(stream): add state machine to manage the state of stream tasks. 2023-10-27 09:54:49 +08:00
Haojun Liao 596eb073b7 refactor(stream): do some internal refactor. 2023-10-27 09:54:48 +08:00
Haojun Liao 2409a17517 refactor(stream): do some internal refactor. 2023-10-27 09:54:44 +08:00
Haojun Liao 164159f293 enh(stream): add parameter to limit the stream sink task. 2023-10-27 09:54:07 +08:00
yihaoDeng db0f6258fe fix stream case error 2023-10-25 19:38:22 +08:00
Haojun Liao 03165d5327 fix(stream): handle the repeatly issued nodeUpdate msg from dnode. 2023-10-25 09:55:02 +08:00
Haojun Liao 8cde39eebd fix(stream): fix dead lock when shutdown all tasks. 2023-10-24 19:11:24 +08:00
Haojun Liao 059e94e428 fix(stream): clear flag for checkpoint. 2023-10-22 01:25:00 +08:00
yihaoDeng 1a4cfd11a5 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-10-20 15:28:10 +08:00
Haojun Liao 3b8c85f632 fix(stream): fix bugs caused by refactor sm. 2023-10-20 14:19:57 +08:00
Haojun Liao 270a847809 fix(stream): fix bugs caused by refactor sm. 2023-10-20 10:23:48 +08:00
Haojun Liao c1cebae6ba fix(stream): fix error in stream. 2023-10-19 16:20:27 +08:00
Haojun Liao 402aefd95d refactor(stream): do some internal refactor. 2023-10-19 15:09:47 +08:00
Haojun Liao f4caeca24a refactor(stream): add state machine to manage the state of stream tasks. 2023-10-19 10:34:20 +08:00
Haojun Liao 54ec679b58 refactor(stream): do some internal refactor. 2023-10-18 15:34:53 +08:00
Haojun Liao 2904cfd4d5 Merge branch 'main' into refact/streamsm 2023-10-18 10:29:09 +08:00
yihaoDeng 2004c1a346 fix transfer crash 2023-10-17 18:19:52 +08:00
Haojun Liao 7b7ce10475 refactor(stream): do some internal refactor. 2023-10-17 11:23:04 +08:00
Haojun Liao fdb6ec1fa3 enh(stream): add parameter to limit the stream sink task. 2023-10-16 22:52:39 +08:00
yihaoDeng a56d831a14 refactor backend 2023-10-12 20:16:41 +08:00
yihaoDeng f264ffdcd6 refactor backend 2023-10-11 20:12:07 +08:00
yihaoDeng 031506aef8 refactor backend 2023-10-11 19:46:35 +08:00
Haojun Liao 05dfadd8f8 fix(stream): set initial version for fill-history task. disable reentry of scan-history task function. 2023-10-10 15:52:48 +08:00
Haojun Liao adf6f65d09 fix(stream): set initial version for fill-history task. disable reentry of scan-history task function. 2023-10-10 14:00:30 +08:00
Haojun Liao 847baa8b71 fix(stream): fix the pause failure bug. 2023-10-09 20:10:27 +08:00
Haojun Liao 21fb502690 enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks. 2023-10-09 20:10:26 +08:00
Haojun Liao 9a7167aaba fix(stream): sleep for 10ms 2023-10-09 20:10:24 +08:00
Haojun Liao af2c0c95e9 refactor: do some internal refactor. 2023-10-09 20:10:22 +08:00
Haojun Liao 28008f8263 enh(stream): add more info for sink task. 2023-10-09 20:09:22 +08:00
Haojun Liao cf14772dd9 enh(stream): optimize the scan wal perf 2023-10-09 20:09:21 +08:00
Haojun Liao c295ec322a refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao 583ac1fc45 refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao bb4ba54f28 fix(stream): enable new time range for stream task. 2023-10-09 20:09:09 +08:00
Haojun Liao c759d90bf6 enh(stream): optimize the stream status when creating without fill-history option. 2023-10-09 20:09:09 +08:00
Haojun Liao db897fb03a fix(stream): opt perf. 2023-10-09 20:09:09 +08:00
Haojun Liao dfe8641c0c refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-10-09 20:08:52 +08:00
Haojun Liao a248f3d088 refactor: do some internal refactor. 2023-10-09 20:08:12 +08:00
Haojun Liao 0db357401d fix(stream): add new status for meta. 2023-10-09 20:08:12 +08:00
Haojun Liao 3912d80c02 fix(stream): do some internal refactor. 2023-10-09 20:08:11 +08:00
Haojun Liao feb09c8361 fix(stream): fix error in pause scan-history data. 2023-10-09 20:08:11 +08:00
Haojun Liao 4af7812f0b fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read. 2023-10-09 20:08:10 +08:00
Haojun Liao f4f55fcbce fix(stream): record the tasks started info, and do some internal refactor. 2023-10-09 20:08:09 +08:00
Haojun Liao 68b003c804 fix(stream): fix race condition. 2023-10-09 20:08:09 +08:00
Haojun Liao 52763ca2a3 refactor: do some internal refactor. 2023-10-09 20:08:07 +08:00
Haojun Liao c05a8de3e8 fix(stream): rsp when checkpoint source failed. 2023-10-09 20:08:06 +08:00
Haojun Liao 5eee76df65 fix(stream): add msg identifier check. 2023-10-09 20:08:05 +08:00
Haojun Liao e0d19af000 fix(stream): refactor re-try dispatch msg for stream tasks. 2023-10-09 20:08:03 +08:00
Haojun Liao 2b16976902 refactor(stream): refactor send msg. 2023-10-09 20:08:03 +08:00
Haojun Liao 27e00b1f19 fix(stream): add some logs, and remove invalid assert. 2023-10-09 20:08:02 +08:00
Haojun Liao 12687eb287 fix(stream): add check info detailed information. 2023-10-09 20:08:02 +08:00
Haojun Liao ef9efc8a8e refactor: do refactor the sink task. 2023-10-09 20:07:53 +08:00
Haojun Liao 1f45bd82d2 other: merge stream fix in main. 2023-10-09 20:07:01 +08:00
Haojun Liao e27111ecd1 enh(stream): add node stage check. 2023-10-09 20:06:09 +08:00
Haojun Liao 5c0b8ea804 enh(stream): log the checkpoint time. 2023-10-09 20:06:04 +08:00
Haojun Liao b95ad74c7f fix(stream): not handle the check msg for follower tasks. 2023-10-09 20:06:04 +08:00
Haojun Liao 636eccc160 fix(stream): add null check 2023-10-09 20:05:37 +08:00
Haojun Liao 0459a4628c fix(stream): fix stream task id error. 2023-10-09 20:05:36 +08:00
Haojun Liao 7ccc273206 other: merge stream fix in main. 2023-10-09 20:05:29 +08:00
Haojun Liao 98ef566eb4 fix(stream): remove fill-history sink task. 2023-10-09 20:04:17 +08:00
Haojun Liao e84eeee6b8 refactor: do some internal refactor. 2023-10-09 20:04:17 +08:00
Haojun Liao 6fa54789cb fix(stream): disable follower send hb to mnode. and do some internal refactor. 2023-10-09 19:59:31 +08:00
Haojun Liao 1256eafddb other: merge stream fix. 2023-10-09 19:59:24 +08:00
Haojun Liao aaa12ed4f8 refactor: do some internal refactor about the sched status set 2023-10-09 19:56:17 +08:00
Haojun Liao d87db2f829 fix(stream): fix the pause failure bug. 2023-10-09 14:48:11 +08:00
Haojun Liao ee6516f3c4 enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks. 2023-10-09 10:13:53 +08:00
Haojun Liao 23fb26798f fix(stream): sleep for 10ms 2023-10-07 19:47:49 +08:00
Haojun Liao ab17bfdf09 refactor: do some internal refactor. 2023-10-07 19:16:55 +08:00
yihaoDeng c146327da8 refact task backend 2023-10-07 16:48:36 +08:00
yihaoDeng fa70221238 refact task backend 2023-10-07 12:17:55 +08:00
Haojun Liao c20dd002fb enh(stream): add more info for sink task. 2023-10-07 01:14:00 +08:00
Haojun Liao 41ef1a45c1 enh(stream): optimize the scan wal perf 2023-10-04 22:06:46 +08:00
Haojun Liao a543f4ca97 refactor: do some internal refactor. 2023-10-03 01:52:03 +08:00
Haojun Liao 5718158fc0 refactor: do some internal refactor. 2023-10-01 22:27:29 +08:00
Haojun Liao f2d8431619 fix(stream): enable new time range for stream task. 2023-09-28 18:26:52 +08:00
yihaoDeng 54e3ac2c1e refact task backend 2023-09-28 15:46:12 +08:00
Haojun Liao 80ec77cbb6 enh(stream): optimize the stream status when creating without fill-history option. 2023-09-28 11:23:03 +08:00
Haojun Liao 22e485d132 fix(stream): opt perf. 2023-09-28 09:25:23 +08:00
Haojun Liao e13efd5b04 refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-09-27 14:00:48 +08:00
Haojun Liao 890d658aea refactor: do some internal refactor. 2023-09-26 14:11:45 +08:00
Haojun Liao f3f18b0fa3 fix(stream): add new status for meta. 2023-09-26 12:15:12 +08:00
Haojun Liao d214a2f6bd fix(stream): do some internal refactor. 2023-09-26 09:32:57 +08:00
Haojun Liao 6cc3618ed9 fix(stream): fix error in pause scan-history data. 2023-09-26 01:09:55 +08:00
Haojun Liao fcf4cc7169 fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read. 2023-09-25 15:58:24 +08:00
Haojun Liao eb4078b7f3 fix(stream): record the tasks started info, and do some internal refactor. 2023-09-25 09:59:05 +08:00
Haojun Liao c987f61300 fix(stream): fix race condition. 2023-09-25 01:37:27 +08:00
Haojun Liao 4ea737571d refactor: do some internal refactor. 2023-09-24 01:26:51 +08:00
Haojun Liao acd3321494 fix(stream): rsp when checkpoint source failed. 2023-09-23 14:13:33 +08:00
Haojun Liao 5b12e276cf fix(stream): add msg identifier check. 2023-09-22 18:13:18 +08:00
yihaoDeng 11da631bb5 add cvt state 2023-09-22 09:08:42 +00:00
Haojun Liao 619a9f6d08 fix(stream): refactor re-try dispatch msg for stream tasks. 2023-09-22 15:57:55 +08:00
Haojun Liao 898aea5cfb refactor(stream): refactor send msg. 2023-09-22 14:52:18 +08:00
Haojun Liao 87ea745861 fix(stream): add some logs, and remove invalid assert. 2023-09-22 10:53:03 +08:00
Haojun Liao f7bcff862d fix(stream): add check info detailed information. 2023-09-22 10:23:56 +08:00
yihaoDeng 470849982e add cvt func 2023-09-21 09:41:30 +00:00
Haojun Liao 17586af0ce refactor: do refactor the sink task. 2023-09-21 13:32:57 +08:00
Haojun Liao 6633fb971b fix(stream): update logs. 2023-09-20 18:18:31 +08:00
Haojun Liao b4419bda65 enh(stream): add node stage check. 2023-09-20 10:35:01 +08:00
Haojun Liao b7efedf4aa enh(stream): log the checkpoint time. 2023-09-19 09:53:33 +08:00
Haojun Liao e76bab7122 fix(stream): not handle the check msg for follower tasks. 2023-09-19 09:43:14 +08:00
Haojun Liao c33ef4ce88 fix(stream): add null check 2023-09-18 15:14:51 +08:00
Haojun Liao 01144c58a7 fix(stream): fix stream task id error. 2023-09-17 01:19:59 +08:00
Haojun Liao 4846194bbb refactor: do some internal refactor, remove the related fill-history task by sending mesg to replay it in follower node. 2023-09-16 21:37:30 +08:00
Haojun Liao 91e3d70796 fix(stream): remove fill-history sink task. 2023-09-15 17:46:17 +08:00
Haojun Liao 2bfd6e3355 refactor: do some internal refactor. 2023-09-15 09:25:22 +08:00
Haojun Liao f2c3e0a6e8 fix(stream): disable follower send hb to mnode. and do some internal refactor. 2023-09-14 17:22:31 +08:00
Haojun Liao fce5be0eb4 fix(stream): fix dead lock. 2023-09-14 14:09:05 +08:00
Haojun Liao 7305c8dbe1 refactor: do some internal refactor about the sched status set 2023-09-13 14:00:34 +08:00
Haojun Liao 09a15bf340 refactor: do some internal refactor. 2023-09-12 22:07:52 +08:00
Haojun Liao f6a097d96f refactor: do some internal refactor. 2023-09-12 21:58:22 +08:00
Haojun Liao e457d94ac6 refactor: do some internal refactor. 2023-09-12 18:09:02 +08:00
Haojun Liao 85a5c45098 fix(stream): limit the already processed data. 2023-09-10 00:01:25 +08:00
Haojun Liao b317c8ebff fix(stream): disable merge submit blocks. 2023-09-08 12:46:15 +08:00
Haojun Liao b240e91c91 enh(stream): add token bucket to limit the rate of sink operation. 2023-09-07 09:58:48 +08:00
Haojun Liao 8edf86b3bb fix(stream): add some logs. 2023-09-06 19:08:09 +08:00
Haojun Liao 8783ebcb02 enh(stream): add some logs and record the submit package performance. 2023-09-06 14:34:36 +08:00
Haojun Liao aeea699c9c Merge branch '3.0' into fix/3_liaohj 2023-09-04 15:53:39 +08:00
Haojun Liao 037a232bfa refactor: do some internal refactor. 2023-09-01 17:05:36 +08:00
Haojun Liao e816d0efa5
Merge pull request #22669 from taosdata/fix/TD-26069
fix invalid insert
2023-08-31 18:29:25 +08:00
yihaoDeng 1833be028f fix invalid insert 2023-08-31 15:28:31 +08:00
Haojun Liao e2bb64eb18 refactor: do some internal refactor. 2023-08-31 14:23:46 +08:00
Haojun Liao 69f95f9324 other: merge 3.0 2023-08-31 13:56:34 +08:00
Haojun Liao 2285b834b0 refactor: do some internal refactor. 2023-08-31 13:53:12 +08:00
Haojun Liao 3fc3dafca6 refactor: do some internal refactor. 2023-08-31 13:46:48 +08:00
Haojun Liao 3ed78ddadd
Merge pull request #22662 from taosdata/fix/3_liaohj
refactor: do some internal refactor.
2023-08-31 13:46:14 +08:00
Haojun Liao 8645f763b5
Merge pull request #22661 from taosdata/enh/reserve
stream change ver
2023-08-31 13:35:23 +08:00
yihaoDeng 83a5e2be4c stream change ver 2023-08-31 11:42:12 +08:00
Haojun Liao e53b5d4392 refactor: do some internal refactor. 2023-08-31 11:01:53 +08:00
yihaoDeng 05ca71d5de stream change ver 2023-08-31 09:28:43 +08:00
Haojun Liao d99d8875e1 fix(stream): if stream task is running, do not recheck its status. 2023-08-30 17:24:13 +08:00
liuyao 0a72e576bd op stream pause
pause data dispatch

opt stream pause
2023-08-30 14:53:49 +08:00
Haojun Liao 3a7a220d43 fix(stream): fix error found by ci. 2023-08-30 10:14:01 +08:00
Haojun Liao 204fb97f68 fix(stream): fix error in snode, fix memory leaks, fix repeatly create bug in sink task. 2023-08-29 17:34:31 +08:00
Haojun Liao 391103bc5a fix(stream): do dome internal refactor. 2023-08-28 15:54:01 +08:00
Haojun Liao b75ae217f6 fix(stream): notify close for snode. 2023-08-28 09:53:08 +08:00
yihaoDeng 998dac0ffb fix mem leak 2023-08-26 21:54:36 +08:00
yihaoDeng 7d237d4310 fix UAF 2023-08-26 17:37:09 +08:00
Haojun Liao 60d1df0e25
Merge branch '3.0' into enh/triggerCheckPoint2 2023-08-25 15:49:28 +08:00
Haojun Liao ce1a3a379c fix(stream): handle the case when the task not in ready state, and the checkpoint msg arrived. 2023-08-25 10:12:12 +08:00
Haojun Liao 70396aa5a5 fix(stream): wakeup the hb timer every 200ms, to speeup the close db procedure. 2023-08-24 17:54:50 +08:00
Haojun Liao af80d94a88 refactor: do some internal refactor. 2023-08-22 15:43:50 +08:00
Haojun Liao cc8b2263cb fix(stream): fix error in heartbeat from vnode. 2023-08-22 13:37:38 +08:00
Haojun Liao def0877980 fix(stream): add timer to check stream task status. 2023-08-22 10:46:49 +08:00
Haojun Liao b8101afbe2 fix(stream): refactor the tasks restart mechanism. 2023-08-18 22:36:38 +08:00
Haojun Liao 4b10393c81 fix(stream): fix memory leak and do some internal refactor. 2023-08-18 18:02:08 +08:00
Haojun Liao 6746339609 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2
# Conflicts:
#	source/libs/stream/src/streamSnapshot.c
2023-08-17 00:08:44 +08:00
Haojun Liao ea6574b3c7 other: merge 3.0 2023-08-17 00:07:55 +08:00
yihaoDeng ab7e2b3122 Merge branch 'enh/chkpTransfer' into enh/triggerCheckPoint2 2023-08-16 09:00:30 +00:00
Haojun Liao d63758a1c5 fix(stream): fix error in execute task on a tranferred node. 2023-08-16 15:24:24 +08:00
Haojun Liao 39079ab64a fix(stream): load term from sync model to decide if the node is restart or not 2023-08-15 15:52:06 +08:00
Haojun Liao b1d35f43ba enh(stream): using term from sync module to denote the upstream node is restarted/replica-modified/vnode-transferred or not. 2023-08-15 09:25:05 +08:00
Haojun Liao e8c9a019a4 fix(stream): remove the invalid set of scheduler status. 2023-08-14 11:41:24 +08:00
Haojun Liao 0d38f389ab refactor: do some internal refactor. 2023-08-13 16:25:35 +08:00
Haojun Liao afe9b848a5 fix(tsdb): check the 2023-08-12 18:54:39 +08:00
Haojun Liao 09da6c6840 refactor: do some internal refactor. 2023-08-12 18:51:20 +08:00
Haojun Liao 75e6fb0f16 refactor: do some internal refactor. 2023-08-12 18:47:01 +08:00
Haojun Liao 8f9de93cf0 refactor: do some internal refactor. 2023-08-12 18:37:36 +08:00
Haojun Liao ce721a0146 fix(stream): fix error. 2023-08-12 16:32:05 +08:00
Haojun Liao a89ce1a20b fix(stream): transfer state by using data block. 2023-08-12 01:24:07 +08:00
yihaoDeng 6e6eb05d5c Merge branch 'enh/triggerCheckPoint2' into enh/chkpTransfer 2023-08-11 08:03:02 +00:00
Haojun Liao e984affe90 fix(stream): disable the stream in the followers nodes. 2023-08-11 15:58:15 +08:00
yihaoDeng 99be2836b2 Merge branch 'enh/triggerCheckPoint2' into enh/chkpTransfer 2023-08-11 02:50:12 +00:00
Haojun Liao 6a9826cf6e enh(stream): update the stream task epset. 2023-08-11 10:37:25 +08:00
yihaoDeng ba4ce9db02 Merge branch 'enh/triggerCheckPoint2' into enh/chkpTransfer 2023-08-11 02:15:07 +00:00
Haojun Liao f633c7fe2b other: merge 3.0. 2023-08-10 11:56:30 +08:00
yihaoDeng ef247cdb1d support reopen stream state 2023-08-10 01:49:26 +00:00
Haojun Liao 1e8579e8c5 fix(stream): fix other cases. 2023-08-09 16:04:48 +08:00
Haojun Liao 4c92997328 fix(stream): use streamId&taskId to identify the stream task in the stream meta. 2023-08-09 15:52:23 +08:00
Haojun Liao f3c5f20ee2 enh(stream): add timer to check the node changing. 2023-08-09 10:25:16 +08:00
Haojun Liao af3a87560a refactor: do some refactor. 2023-08-08 09:11:41 +08:00
yihaoDeng b6c991f896 support reopen stream state 2023-08-07 08:59:04 +00:00
yihaoDeng eeb97351e8 support reopen stream state 2023-08-07 08:51:01 +00:00
Haojun Liao 225c0e61dc hen(stream): support the rollback and checkdown stream tasks. 2023-08-07 16:13:39 +08:00
Haojun Liao 8a11a0418f other: merge 3.0 2023-08-07 09:19:26 +08:00
yihaoDeng 6dc93a8dec add stream ver 2023-08-04 13:52:20 +08:00
yihaoDeng f9d0874e45 support reopen stream state 2023-08-03 08:24:50 +00:00
Haojun Liao 13543f7daa feature(stream): handle task update in tq module. 2023-08-02 10:33:41 +08:00
Haojun Liao ad4c5916e7 enh(stream): update the stream task. 2023-08-01 18:14:58 +08:00