Commit Graph

1221 Commits

Author SHA1 Message Date
Haojun Liao 17bd6badec refactor(stream): do checkpoint for each task. 2023-11-08 10:47:54 +08:00
Haojun Liao 2674698b36 fix(stream): restart tasks in stream threads, instead of write thread. 2023-11-07 18:51:46 +08:00
Haojun Liao 1301b92844 fix(stream): fix errors in scan-history, introduced by refactor 2023-11-07 16:41:15 +08:00
Haojun Liao 8c2f6bab0e fix(stream): exec scan-history in timer. 2023-11-07 16:40:03 +08:00
Haojun Liao 207b53f3f6 fix(stream): fix errors in scan-history, introduced by refactor 2023-11-07 16:19:09 +08:00
Haojun Liao 4d8548e938 refactor: do some internal refactor. 2023-11-06 23:42:16 +08:00
Haojun Liao d9474016fc fix(stream): exec scan-history in timer. 2023-11-06 18:55:53 +08:00
Haojun Liao d41fd27189 fix(stream): restart tasks in stream threads, instead of write thread. 2023-11-06 11:57:21 +08:00
Haojun Liao 0e44950a37 enh(stream): execute scan history by using time slice 1s. 2023-11-06 09:21:33 +08:00
Haojun Liao 680eb5674e refactor: do some internal refactor. 2023-11-06 09:15:49 +08:00
Haojun Liao d440da0d8c enh(stream): execute scan history by using time slice 1s. 2023-11-05 01:26:53 +08:00
Haojun Liao e13f003b90 refactor: do some internal refactor. 2023-11-05 01:23:23 +08:00
yihaoDeng 24e961976f Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-03 09:31:57 +08:00
Haojun Liao b3d57e4a2d fix(stream): not transfer state if state is not appropriate. 2023-11-02 19:31:35 +08:00
Haojun Liao 7a32b3a209 fix(stream): not transfer state if state is not appropriate. 2023-11-02 19:12:36 +08:00
Haojun Liao c4079332ff fix(stream):update the log level. 2023-11-02 18:18:04 +08:00
Haojun Liao 0a227e807f refactor: set different tq level. 2023-11-02 18:18:04 +08:00
Haojun Liao 48aff868b6 fix(stream):update the log level. 2023-11-02 18:16:47 +08:00
Haojun Liao 3aed8c29ae refactor: set different tq level. 2023-11-02 17:12:20 +08:00
Haojun Liao 95efa07e92 fix(stream): add full unsupported event filtering. 2023-11-02 11:02:39 +08:00
Haojun Liao 351a31302f fix(stream): add full unsupported event filtering. 2023-11-02 09:43:48 +08:00
Haojun Liao 7cf90dde5c refactor: do some internal refactor. 2023-11-01 19:38:13 +08:00
Haojun Liao 00f029e44f refactor: add some logs. 2023-11-01 19:38:12 +08:00
Haojun Liao 9a9c1d689d fix(stream): add lock log. 2023-11-01 19:38:12 +08:00
Haojun Liao 246ed4e022 refactor: do some internal refactor. 2023-11-01 17:19:21 +08:00
Haojun Liao 12bc4f32cf refactor: add some logs. 2023-11-01 16:12:15 +08:00
yihaoDeng 4c8edb5545 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-11-01 16:07:49 +08:00
Haojun Liao 5ff89bc098 fix(stream): add lock log. 2023-11-01 13:19:53 +08:00
Haojun Liao 3e6c89d8eb fix(stream): add unsupport trans. 2023-11-01 11:43:02 +08:00
Haojun Liao 639d5cde47 fix(stream): add unsupport trans. 2023-10-31 23:14:02 +08:00
Haojun Liao 0b18192b23 fix(stream): fix error in finishing event handling. 2023-10-31 16:40:16 +08:00
Haojun Liao 4c0242a58d fix(stream): fix error in finishing event handling. 2023-10-31 16:39:14 +08:00
Haojun Liao 635243f01a refactor: do some internal refactor. 2023-10-31 10:45:12 +08:00
Haojun Liao 6e2240a3d1 fix(stream): fix the deadlock. 2023-10-31 10:45:12 +08:00
Haojun Liao fdf3f210ac refactor: do some internal refactor. 2023-10-30 14:39:24 +08:00
Haojun Liao 3275452bce fix(stream): fix the deadlock. 2023-10-30 14:38:51 +08:00
Haojun Liao c7285c0ede fix(stream): add expired epset node list for stream tasks in hb to mnode. 2023-10-27 23:41:57 +08:00
Haojun Liao 4c267b538f fix(stream): add expired epset node list for stream tasks in hb to mnode. 2023-10-27 23:05:41 +08:00
Haojun Liao dcca6b0868 other: merge 3.0. 2023-10-27 11:41:28 +08:00
Haojun Liao 454624516c fix(stream): handle the repeatly issued nodeUpdate msg from dnode. 2023-10-27 09:59:51 +08:00
Haojun Liao 33dc51cc86 fix(stream): add some logs. 2023-10-27 09:59:44 +08:00
Haojun Liao 1ccbb77981 fix(stream): fix dead lock when shutdown all tasks. 2023-10-27 09:59:37 +08:00
Haojun Liao c99f434706 fix(stream): set checkpoint failure when in pause status. 2023-10-27 09:58:52 +08:00
Haojun Liao 7311f9d2fe fix(sma): fix bugs in sma and do some internal refactor. 2023-10-27 09:58:51 +08:00
Haojun Liao 091238e7be fix(stream): fix bugs caused by refactor sm. 2023-10-27 09:54:58 +08:00
Haojun Liao e1f6043eed fix(stream): fix bugs caused by refactor sm. 2023-10-27 09:54:57 +08:00
Haojun Liao 8729bdac1e fix(stream): fix bugs caused by refactor. 2023-10-27 09:54:57 +08:00
Haojun Liao 1eac7f44c0 fix(stream): fix error in stream. 2023-10-27 09:54:56 +08:00
Haojun Liao 579dd8e6b2 refactor(stream): do some internal refactor. 2023-10-27 09:54:56 +08:00
Haojun Liao 6f3c0d17cc refactor(stream): add state machine to manage the state of stream tasks. 2023-10-27 09:54:49 +08:00
Haojun Liao 596eb073b7 refactor(stream): do some internal refactor. 2023-10-27 09:54:48 +08:00
Haojun Liao 62c58ecd8b fix(stream):add more check for test cases. 2023-10-27 09:54:06 +08:00
yihaoDeng 4c03d372ef fix stream case error 2023-10-26 14:19:34 +08:00
yihaoDeng db0f6258fe fix stream case error 2023-10-25 19:38:22 +08:00
Haojun Liao 03165d5327 fix(stream): handle the repeatly issued nodeUpdate msg from dnode. 2023-10-25 09:55:02 +08:00
Haojun Liao 13d979a2ee fix(stream): add some logs. 2023-10-25 00:03:46 +08:00
Haojun Liao 8cde39eebd fix(stream): fix dead lock when shutdown all tasks. 2023-10-24 19:11:24 +08:00
Haojun Liao e92b276e3e fix(stream): set checkpoint failure when in pause status. 2023-10-24 10:14:29 +08:00
Haojun Liao 7a8b046a35 fix(sma): fix bugs in sma and do some internal refactor. 2023-10-24 09:40:38 +08:00
yihaoDeng 1a4cfd11a5 Merge branch 'enh/new3.0' into enh/refactorBackend 2023-10-20 15:28:10 +08:00
Haojun Liao 3b8c85f632 fix(stream): fix bugs caused by refactor sm. 2023-10-20 14:19:57 +08:00
Haojun Liao 270a847809 fix(stream): fix bugs caused by refactor sm. 2023-10-20 10:23:48 +08:00
Haojun Liao d5059826a8 fix(stream): fix bugs caused by refactor. 2023-10-19 17:59:04 +08:00
Haojun Liao c1cebae6ba fix(stream): fix error in stream. 2023-10-19 16:20:27 +08:00
Haojun Liao 402aefd95d refactor(stream): do some internal refactor. 2023-10-19 15:09:47 +08:00
Benguang Zhao c4e9069a66 fix: set nextProcessedVer properly in tqProcessTaskScanHistory 2023-10-19 11:18:57 +08:00
Haojun Liao f4caeca24a refactor(stream): add state machine to manage the state of stream tasks. 2023-10-19 10:34:20 +08:00
Haojun Liao 54ec679b58 refactor(stream): do some internal refactor. 2023-10-18 15:34:53 +08:00
yihaoDeng 2004c1a346 fix transfer crash 2023-10-17 18:19:52 +08:00
Haojun Liao a1aa2c9e0c fix(stream):add more check for test cases. 2023-10-16 18:47:01 +08:00
wangmm0220 473142fab7 fix:conflicts from 3.0 2023-10-16 11:35:56 +08:00
yihaoDeng 8392c99d36 refactor backend 2023-10-13 14:54:00 +08:00
yihaoDeng a56d831a14 refactor backend 2023-10-12 20:16:41 +08:00
Haojun Liao eb4449999f other: merge 3.0 2023-10-12 10:33:48 +08:00
Haojun Liao adf6f65d09 fix(stream): set initial version for fill-history task. disable reentry of scan-history task function. 2023-10-10 14:00:30 +08:00
Haojun Liao 8266c4ff77 fix(stream): remove invalid clear of rspmsg list. 2023-10-09 15:26:42 +08:00
Haojun Liao ee6516f3c4 enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks. 2023-10-09 10:13:53 +08:00
yihaoDeng bffa6387fe refact task backend 2023-10-08 18:16:45 +08:00
wangmm0220 3f2d890560 feat:[TD-26056] add replay logic 2023-10-08 16:06:15 +08:00
Haojun Liao ab17bfdf09 refactor: do some internal refactor. 2023-10-07 19:16:55 +08:00
yihaoDeng fa70221238 refact task backend 2023-10-07 12:17:55 +08:00
Haojun Liao 5718158fc0 refactor: do some internal refactor. 2023-10-01 22:27:29 +08:00
Haojun Liao ff44539468 fix(stream): drop related fill-history task when dropping stream tasks. 2023-09-28 23:00:29 +08:00
Haojun Liao 31acbc6fe2 refactor: do some internal refactor (discard the checkpoint source msg during restoring). 2023-09-28 22:44:13 +08:00
Haojun Liao f2d8431619 fix(stream): enable new time range for stream task. 2023-09-28 18:26:52 +08:00
yihaoDeng bb265887e6 refact task backend 2023-09-28 17:07:19 +08:00
yihaoDeng 54e3ac2c1e refact task backend 2023-09-28 15:46:12 +08:00
Haojun Liao 22e485d132 fix(stream): opt perf. 2023-09-28 09:25:23 +08:00
Haojun Liao e13efd5b04 refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-09-27 14:00:48 +08:00
Haojun Liao f3f18b0fa3 fix(stream): add new status for meta. 2023-09-26 12:15:12 +08:00
Haojun Liao 6cc3618ed9 fix(stream): fix error in pause scan-history data. 2023-09-26 01:09:55 +08:00
Haojun Liao 966ce4f30e Merge remote-tracking branch 'origin/fix/liaohj' into fix/liaohj 2023-09-25 16:00:02 +08:00
Haojun Liao fcf4cc7169 fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read. 2023-09-25 15:58:24 +08:00
Haojun Liao dbc53069db
Merge branch 'main' into fix/liaohj 2023-09-25 10:41:18 +08:00
Haojun Liao eb4078b7f3 fix(stream): record the tasks started info, and do some internal refactor. 2023-09-25 09:59:05 +08:00
Haojun Liao c987f61300 fix(stream): fix race condition. 2023-09-25 01:37:27 +08:00
Haojun Liao 4ea737571d refactor: do some internal refactor. 2023-09-24 01:26:51 +08:00
Haojun Liao acd3321494 fix(stream): rsp when checkpoint source failed. 2023-09-23 14:13:33 +08:00
Haojun Liao 7d5f76c6f1 fix(stream): disable auto rsp for checkpoint source. 2023-09-23 11:31:14 +08:00
Haojun Liao 5155519559 fix(stream): fix invalid assert. 2023-09-22 23:06:48 +08:00
Haojun Liao cf6c255f22 refactor: do some internal refactor. 2023-09-22 22:50:21 +08:00
Haojun Liao 5b12e276cf fix(stream): add msg identifier check. 2023-09-22 18:13:18 +08:00
yihaoDeng 11da631bb5 add cvt state 2023-09-22 09:08:42 +00:00
Haojun Liao 898aea5cfb refactor(stream): refactor send msg. 2023-09-22 14:52:18 +08:00
Haojun Liao 87ea745861 fix(stream): add some logs, and remove invalid assert. 2023-09-22 10:53:03 +08:00
Haojun Liao f7bcff862d fix(stream): add check info detailed information. 2023-09-22 10:23:56 +08:00
Haojun Liao b1ba716299 refactor: do some internal refactor. 2023-09-22 09:24:36 +08:00
Haojun Liao b075b4438b fix(stream): handle dispatch checkpoint-trigger failure. 2023-09-21 23:15:18 +08:00
Haojun Liao 09accc71d2 log(stream): update log. 2023-09-21 14:01:41 +08:00
wangmm0220 f9c5aa01af fix:enable split if there are topics & fix memory leak 2023-09-21 09:20:34 +08:00
Haojun Liao 2418f509ea fix(stream): avoid invalid read. 2023-09-20 15:47:09 +08:00
Haojun Liao 1d1d9a1b0b fix(stream): unlock when returning. 2023-09-20 15:45:15 +08:00
Haojun Liao c8ab761d2a fix(stream): return if failed. 2023-09-20 15:38:49 +08:00
Haojun Liao 2a781c509f fix(stream): add task status check. 2023-09-20 15:32:15 +08:00
Haojun Liao af487c9048 fix(stream): fix obj. 2023-09-20 14:50:43 +08:00
Haojun Liao 361cde6497 fix(stream): fix deadlock caused by streamTaskHalt function. 2023-09-20 14:49:09 +08:00
Haojun Liao 779c0c5f19
Merge pull request #22970 from taosdata/fix/liaohj
fix(stream): keep the status entry in hash table, instead entry index.
2023-09-20 10:33:05 +08:00
wangmm0220 edb42aa2aa fix:move from main to 3.0 2023-09-19 11:51:52 +08:00
Haojun Liao e76bab7122 fix(stream): not handle the check msg for follower tasks. 2023-09-19 09:43:14 +08:00
Haojun Liao 101161c52e
Merge pull request #22954 from taosdata/fix/TD-26323
fix:core in race condition for pTq->pExecStore & return if poll too l…
2023-09-18 17:41:01 +08:00
Haojun Liao c33ef4ce88 fix(stream): add null check 2023-09-18 15:14:51 +08:00
wangmm0220 c3498fbfe8 fix:core in race condition for pTq->pExecStore & return if poll too long & fix test cases if submit empty 2023-09-18 14:35:07 +08:00
wangmm0220 7869b693f2 add test case for tmq in vnode transform & fix core in race condition for pTq->pExecStore 2023-09-18 14:09:28 +08:00
Haojun Liao 01144c58a7 fix(stream): fix stream task id error. 2023-09-17 01:19:59 +08:00
Haojun Liao 4846194bbb refactor: do some internal refactor, remove the related fill-history task by sending mesg to replay it in follower node. 2023-09-16 21:37:30 +08:00
Haojun Liao 91e3d70796 fix(stream): remove fill-history sink task. 2023-09-15 17:46:17 +08:00
Haojun Liao 2bfd6e3355 refactor: do some internal refactor. 2023-09-15 09:25:22 +08:00
Haojun Liao 117de7ab7d fix(stream): fix error in drop task. 2023-09-15 00:51:13 +08:00
Haojun Liao 73c2782d78 fix(stream): update some logs. 2023-09-14 23:17:23 +08:00
Haojun Liao 5f55f489d8 fix(stream): fix memory leak. 2023-09-14 22:59:11 +08:00
Haojun Liao 27f626769e fix(stream): fix memory leak. 2023-09-14 19:33:05 +08:00
Haojun Liao f2c3e0a6e8 fix(stream): disable follower send hb to mnode. and do some internal refactor. 2023-09-14 17:22:31 +08:00
Haojun Liao fce5be0eb4 fix(stream): fix dead lock. 2023-09-14 14:09:05 +08:00
Haojun Liao 7305c8dbe1 refactor: do some internal refactor about the sched status set 2023-09-13 14:00:34 +08:00
Haojun Liao 60f5056ab7 other: merge 3.0 2023-09-12 22:00:40 +08:00
Haojun Liao e457d94ac6 refactor: do some internal refactor. 2023-09-12 18:09:02 +08:00
Haojun Liao 93159729f6 fix(stream):check error when vnode is closing. 2023-09-12 11:45:51 +08:00
Haojun Liao c729fc989f fix(stream): set the init start timestamp. 2023-09-11 13:49:43 +08:00
Haojun Liao 0980e021e5 fix(stream): set the correct start version for the step 2 stage in fill history. 2023-09-11 13:37:19 +08:00
Haojun Liao 85a5c45098 fix(stream): limit the already processed data. 2023-09-10 00:01:25 +08:00
Haojun Liao 9a30573b8b fix(stream): disable scan wal when halt is set. 2023-09-09 16:12:08 +08:00
Haojun Liao aa8909b267 other: add some logs. 2023-09-09 15:18:54 +08:00
Haojun Liao b317c8ebff fix(stream): disable merge submit blocks. 2023-09-08 12:46:15 +08:00
Haojun Liao c86eeb3938 fix(stream): add task status check when waiting for the table create. 2023-09-06 10:58:33 +08:00
wangmm0220 ef87b9c571 fix:conflicts from 3.0 2023-09-05 16:19:04 +08:00
wangmm0220 4e46ce4c03 fix:wait pHandle idle if vnode receives subscribe msg 2023-09-05 16:04:44 +08:00
Haojun Liao aeea699c9c Merge branch '3.0' into fix/3_liaohj 2023-09-04 15:53:39 +08:00
wangmm0220 fd85d5885a fix:add limit time for consume in 1 seconds 2023-09-01 16:57:49 +08:00
wangmm0220 02ac3eac5a fix:logic error 2023-09-01 16:51:44 +08:00
Haojun Liao 4fee5ba59a fix(stream): even in ck, start stream tasks. 2023-09-01 14:40:30 +08:00
dapan1121 228e6c6acd Merge remote-tracking branch 'origin/main' into 3.0 2023-09-01 10:01:10 +08:00
Haojun Liao e8294ed8dc fix(stream): fix bug in multi-replica vnode redistribute. 2023-09-01 03:10:12 +08:00
Haojun Liao e2bb64eb18 refactor: do some internal refactor. 2023-08-31 14:23:46 +08:00
Haojun Liao 3fc3dafca6 refactor: do some internal refactor. 2023-08-31 13:46:48 +08:00
Haojun Liao e53b5d4392 refactor: do some internal refactor. 2023-08-31 11:01:53 +08:00
wangmm0220 23c4d7eb0a fix:dead lock 2023-08-30 18:10:19 +08:00
Haojun Liao d99d8875e1 fix(stream): if stream task is running, do not recheck its status. 2023-08-30 17:24:13 +08:00
Haojun Liao c5b279f045 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2 2023-08-30 16:01:08 +08:00
Haojun Liao 24b1a050fc other: merge 3.0 2023-08-30 16:00:14 +08:00
liuyao 0a72e576bd op stream pause
pause data dispatch

opt stream pause
2023-08-30 14:53:49 +08:00
dapan1121 f6321d0c4f Merge remote-tracking branch 'origin/main' into fix/m23.1 2023-08-30 11:03:33 +08:00
Haojun Liao 3a7a220d43 fix(stream): fix error found by ci. 2023-08-30 10:14:01 +08:00
Haojun Liao 6e4acddd96
Merge pull request #22603 from taosdata/fix/TD-25962
fix:unregister push manager if delete subscribe to avoid tqProcessPol…
2023-08-29 10:58:49 +08:00
wangmm0220 2e04d513a4 fix:unregister push manager if delete subscribe to avoid tqProcessPollPush 2023-08-28 16:41:28 +08:00
Haojun Liao 05bb164675 fix(stream): release tasks. 2023-08-28 15:56:02 +08:00
Haojun Liao 391103bc5a fix(stream): do dome internal refactor. 2023-08-28 15:54:01 +08:00
Haojun Liao 33672f2f35 fix(stream): release stream tasks. 2023-08-28 15:52:54 +08:00
Haojun Liao b75ae217f6 fix(stream): notify close for snode. 2023-08-28 09:53:08 +08:00
dapan1121 1f40da6fef Merge remote-tracking branch 'origin/main' into 3.0 2023-08-28 09:28:55 +08:00
Haojun Liao 6b327886e9 fix(stream): add null ptr check. 2023-08-27 00:27:04 +08:00
Haojun Liao e013ba6795 fix(stream): fix the counter. 2023-08-25 17:32:34 +08:00
Haojun Liao ab2aad2b2c fix(stream): update the task status even it is in stop status. 2023-08-25 15:32:55 +08:00
Haojun Liao 4c7ad54c9c fix(stream): not restart tasks when in restoring procedure. 2023-08-25 11:28:28 +08:00
Haojun Liao 38bb123701 fix(stream): release task. 2023-08-25 10:33:15 +08:00
Haojun Liao ce1a3a379c fix(stream): handle the case when the task not in ready state, and the checkpoint msg arrived. 2023-08-25 10:12:12 +08:00
Haojun Liao f7ea875af1 fix(stream): add some logs. 2023-08-24 23:27:48 +08:00
Haojun Liao 82e356a9b5 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2
# Conflicts:
#	source/dnode/vnode/src/tq/tq.c
2023-08-24 18:04:44 +08:00
Haojun Liao 70396aa5a5 fix(stream): wakeup the hb timer every 200ms, to speeup the close db procedure. 2023-08-24 17:54:50 +08:00
yihaoDeng 090515c684 refactor code 2023-08-24 17:20:04 +08:00
wangmm0220 b83cc11043 fix:[TD-25651] reset epoch if consumer changed to avoid consumeing no data 2023-08-24 16:38:53 +08:00
wangmm0220 49216ffa54 fix:transaction in tmq 2023-08-24 16:32:15 +08:00
Haojun Liao 230d8c2d80 fix(stream): fix error in stream computing after vnode is transferred to other dnodes. 2023-08-23 17:50:44 +08:00
Haojun Liao b0c5fa8290 fix(stream): reset the task status. 2023-08-23 14:10:41 +08:00
Haojun Liao af80d94a88 refactor: do some internal refactor. 2023-08-22 15:43:50 +08:00
Haojun Liao cc8b2263cb fix(stream): fix error in heartbeat from vnode. 2023-08-22 13:37:38 +08:00
Haojun Liao def0877980 fix(stream): add timer to check stream task status. 2023-08-22 10:46:49 +08:00
Haojun Liao 873c22abb1 fix(stream): reset update stream tasks. 2023-08-21 19:32:27 +08:00
Haojun Liao 8b27f98ee4 refactor: do some internal refactor. 2023-08-21 19:08:27 +08:00
yihaoDeng 387b4d365e refactor checkpoint 2023-08-21 19:07:15 +08:00
Haojun Liao 29bb5854cc fix(stream): fix error in check stream state. 2023-08-21 11:48:55 +08:00
Haojun Liao b8101afbe2 fix(stream): refactor the tasks restart mechanism. 2023-08-18 22:36:38 +08:00
Haojun Liao edbad45772 other: add logs. 2023-08-18 13:45:08 +08:00
Haojun Liao a8b53c2e69 other: add logs. 2023-08-18 10:51:59 +08:00
Haojun Liao 633ecf66e5 other: add logs. 2023-08-18 10:39:32 +08:00
Haojun Liao 9ad8cf48dc other: add logs. 2023-08-18 10:38:24 +08:00
Haojun Liao 87785b1d9c refactor: add some logs. 2023-08-18 10:30:00 +08:00
Haojun Liao ea6574b3c7 other: merge 3.0 2023-08-17 00:07:55 +08:00
yihaoDeng 993ae84398 close backend 2023-08-16 07:17:15 +00:00
Haojun Liao ad9f3cbc43 other: merge main. 2023-08-16 09:14:10 +08:00
Haojun Liao 7628be236d other: merge main. 2023-08-16 09:03:54 +08:00