Commit Graph

463 Commits

Author SHA1 Message Date
Haojun Liao cf14772dd9 enh(stream): optimize the scan wal perf 2023-10-09 20:09:21 +08:00
Haojun Liao 7682c033ad fix(stream): add null check. 2023-10-09 20:09:21 +08:00
Haojun Liao 0ab5d4b342 refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao c295ec322a refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao 583ac1fc45 refactor: do some internal refactor. 2023-10-09 20:09:20 +08:00
Haojun Liao 83f84d92bc refactor: do some internal refactor (discard the checkpoint source msg during restoring). 2023-10-09 20:09:10 +08:00
Haojun Liao bb4ba54f28 fix(stream): enable new time range for stream task. 2023-10-09 20:09:09 +08:00
Haojun Liao dfe8641c0c refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-10-09 20:08:52 +08:00
Haojun Liao f2848de12a fix(stream): reduce the counter when remove stream tasks. 2023-10-09 20:08:52 +08:00
Haojun Liao 2c0e54f4f7 fix(stream): update the retry interval. 2023-10-09 20:08:13 +08:00
Haojun Liao 0db357401d fix(stream): add new status for meta. 2023-10-09 20:08:12 +08:00
Haojun Liao 3912d80c02 fix(stream): do some internal refactor. 2023-10-09 20:08:11 +08:00
Haojun Liao a32b56f381 fix(stream): fix memory leak. 2023-10-09 20:08:10 +08:00
Haojun Liao 4af7812f0b fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read. 2023-10-09 20:08:10 +08:00
Haojun Liao 1f45bd82d2 other: merge stream fix in main. 2023-10-09 20:07:01 +08:00
Haojun Liao e27111ecd1 enh(stream): add node stage check. 2023-10-09 20:06:09 +08:00
Haojun Liao b95ad74c7f fix(stream): not handle the check msg for follower tasks. 2023-10-09 20:06:04 +08:00
Haojun Liao e6fb9ee46c fix(stream): reset task counter. 2023-10-09 20:06:04 +08:00
Haojun Liao c3a3cf21fb fix(stream): add some logs. 2023-10-09 20:05:39 +08:00
Haojun Liao 9d6199adf7 fix(stream): fix invalid read. 2023-10-09 20:05:37 +08:00
Haojun Liao 0459a4628c fix(stream): fix stream task id error. 2023-10-09 20:05:36 +08:00
Haojun Liao 7ccc273206 other: merge stream fix in main. 2023-10-09 20:05:29 +08:00
Haojun Liao 98ef566eb4 fix(stream): remove fill-history sink task. 2023-10-09 20:04:17 +08:00
Haojun Liao e84eeee6b8 refactor: do some internal refactor. 2023-10-09 20:04:17 +08:00
Haojun Liao c33e521aa1 fix(stream): update some logs. 2023-10-09 20:04:17 +08:00
Haojun Liao 39e0c57323 fix(stream): set the correct guard flag. 2023-10-09 20:04:16 +08:00
Haojun Liao 3d1f29ff21 fix(stream): fix memory leak. 2023-10-09 20:04:16 +08:00
Haojun Liao 6fa54789cb fix(stream): disable follower send hb to mnode. and do some internal refactor. 2023-10-09 19:59:31 +08:00
Haojun Liao 1256eafddb other: merge stream fix. 2023-10-09 19:59:24 +08:00
yihaoDeng 28b546a4ad fix mem leak 2023-10-09 14:25:57 +08:00
yihaoDeng be42143362 Merge branch '3.0' into case/TD-26408-3.0 2023-10-09 14:24:31 +08:00
yihaoDeng 3f9784f475 fix mem leak 2023-10-09 14:07:22 +08:00
Haojun Liao ee6516f3c4 enh(stream): add trans to reset task status to avoid being frozen in checkpoint status due to doing checkpoint failure of partial tasks. 2023-10-09 10:13:53 +08:00
yihaoDeng f49a2cbeb0 refact task backend 2023-10-08 18:14:17 +08:00
Haojun Liao ab17bfdf09 refactor: do some internal refactor. 2023-10-07 19:16:55 +08:00
yihaoDeng 97fdcbc60b fix mem leak 2023-10-07 17:48:24 +08:00
yihaoDeng bdc4afec83 refact task backend 2023-10-07 17:17:30 +08:00
Haojun Liao 7ce3d2c200 fix(stream): quit from loop when input queue is full. 2023-10-07 14:54:11 +08:00
yihaoDeng fa70221238 refact task backend 2023-10-07 12:17:55 +08:00
Haojun Liao 115a284ac0 enh(stream): update the checkpoint interval 2023-10-07 09:56:03 +08:00
Haojun Liao 011627d50a fix(stream): update the current processing version. 2023-10-07 09:07:44 +08:00
Haojun Liao c20dd002fb enh(stream): add more info for sink task. 2023-10-07 01:14:00 +08:00
Haojun Liao 41ef1a45c1 enh(stream): optimize the scan wal perf 2023-10-04 22:06:46 +08:00
Haojun Liao 38aff641ee fix(stream): add null check. 2023-10-03 02:39:57 +08:00
Haojun Liao 58f410a9df refactor: do some internal refactor. 2023-10-03 02:37:52 +08:00
Haojun Liao a543f4ca97 refactor: do some internal refactor. 2023-10-03 01:52:03 +08:00
Haojun Liao 5718158fc0 refactor: do some internal refactor. 2023-10-01 22:27:29 +08:00
Haojun Liao 31acbc6fe2 refactor: do some internal refactor (discard the checkpoint source msg during restoring). 2023-09-28 22:44:13 +08:00
Haojun Liao f2d8431619 fix(stream): enable new time range for stream task. 2023-09-28 18:26:52 +08:00
yihaoDeng bb265887e6 refact task backend 2023-09-28 17:07:19 +08:00
yihaoDeng 54e3ac2c1e refact task backend 2023-09-28 15:46:12 +08:00
yihaoDeng d4f87378fd avoid first tag index conflict 2023-09-27 14:44:09 +08:00
Haojun Liao e13efd5b04 refactor: optimize the fill-history task launch policy, and do some other internal refactor. 2023-09-27 14:00:48 +08:00
Haojun Liao 37bf387453 fix(stream): reduce the counter when remove stream tasks. 2023-09-27 09:36:20 +08:00
Haojun Liao 27dcbcb96f fix(stream): update the retry interval. 2023-09-26 23:42:02 +08:00
Haojun Liao f3f18b0fa3 fix(stream): add new status for meta. 2023-09-26 12:15:12 +08:00
Haojun Liao d214a2f6bd fix(stream): do some internal refactor. 2023-09-26 09:32:57 +08:00
Haojun Liao c615c513e8 fix(stream): fix memory leak. 2023-09-25 16:01:17 +08:00
Haojun Liao fcf4cc7169 fix(stream): fix the invalid counter for ready tasks in vnode, fix an invalid read. 2023-09-25 15:58:24 +08:00
yihaoDeng 8890f5c950 add cvt state 2023-09-22 12:51:29 +00:00
yihaoDeng 11da631bb5 add cvt state 2023-09-22 09:08:42 +00:00
yihaoDeng 970b1cb840 add cvt state 2023-09-22 08:38:41 +00:00
yihaoDeng 19e52c364e add cvt state 2023-09-22 08:26:03 +00:00
wangmm0220 803b17a2db fix:memory leak 2023-09-22 09:31:58 +08:00
yihaoDeng 470849982e add cvt func 2023-09-21 09:41:30 +00:00
yihaoDeng ee8c09667e refactor backend 2023-09-20 12:53:58 +00:00
Haojun Liao 6633fb971b fix(stream): update logs. 2023-09-20 18:18:31 +08:00
Haojun Liao b4419bda65 enh(stream): add node stage check. 2023-09-20 10:35:01 +08:00
Haojun Liao e76bab7122 fix(stream): not handle the check msg for follower tasks. 2023-09-19 09:43:14 +08:00
Haojun Liao 9a9fffa577 fix(stream): reset task counter. 2023-09-19 00:52:35 +08:00
Haojun Liao 377fdfacf6 fix(stream): add some logs. 2023-09-18 23:51:08 +08:00
Haojun Liao 301258784a fix(stream): fix invalid read. 2023-09-17 13:59:06 +08:00
Haojun Liao 01144c58a7 fix(stream): fix stream task id error. 2023-09-17 01:19:59 +08:00
Haojun Liao 4846194bbb refactor: do some internal refactor, remove the related fill-history task by sending mesg to replay it in follower node. 2023-09-16 21:37:30 +08:00
Haojun Liao 91e3d70796 fix(stream): remove fill-history sink task. 2023-09-15 17:46:17 +08:00
Haojun Liao 2bfd6e3355 refactor: do some internal refactor. 2023-09-15 09:25:22 +08:00
Haojun Liao 73c2782d78 fix(stream): update some logs. 2023-09-14 23:17:23 +08:00
Haojun Liao 303e8caf0c fix(stream): set the correct guard flag. 2023-09-14 22:44:16 +08:00
Haojun Liao 27f626769e fix(stream): fix memory leak. 2023-09-14 19:33:05 +08:00
Haojun Liao f2c3e0a6e8 fix(stream): disable follower send hb to mnode. and do some internal refactor. 2023-09-14 17:22:31 +08:00
Haojun Liao fce5be0eb4 fix(stream): fix dead lock. 2023-09-14 14:09:05 +08:00
Haojun Liao 60f5056ab7 other: merge 3.0 2023-09-12 22:00:40 +08:00
Haojun Liao 4d33387e80
Merge pull request #22814 from taosdata/fix/fixTransferCrash2
fix merge error
2023-09-12 16:10:07 +08:00
yihaoDeng 3e48f3639c fix transfer crash 2023-09-12 14:46:32 +08:00
yihaoDeng 6ce85f5cae fix transfer crash 2023-09-12 10:44:56 +08:00
Haojun Liao f602aa965f fix(stream): fix coverity scan issues. 2023-09-12 09:16:18 +08:00
Haojun Liao 0441c7a870 Merge branch 'fix/3_liaohj' of github.com:taosdata/tdengine into fix/3_liaohj 2023-09-05 10:40:48 +08:00
Haojun Liao 5b3cb8ec42 fix(stream): fix memory leak. 2023-09-05 09:27:16 +08:00
Haojun Liao 77eed06bcd
Merge branch '3.0' into fix/3_liaohj 2023-09-04 18:51:07 +08:00
Haojun Liao 5869750782 fix(stream): release stream task with max_delay option. 2023-09-04 18:33:33 +08:00
Haojun Liao 325a177170 fix(stream): disable the fill-history task when restart stream tasks. 2023-09-04 17:18:04 +08:00
Minglei Jin 5159bf3e2a fix(stream): fix epset errors 2023-09-01 15:33:23 +08:00
dapan1121 cc17848b78 fix: merge 3.0 to main 2023-09-01 13:24:47 +08:00
Haojun Liao 95da66e3e8 fix(stream): fix error. 2023-08-31 20:50:12 +08:00
Haojun Liao 3fc3dafca6 refactor: do some internal refactor. 2023-08-31 13:46:48 +08:00
Haojun Liao d99d8875e1 fix(stream): if stream task is running, do not recheck its status. 2023-08-30 17:24:13 +08:00
yihaoDeng 57608e7fff fix merge error 2023-08-30 17:12:37 +08:00
Haojun Liao 997d247aa6 fix(stream): remove invalid memset. 2023-08-30 16:58:22 +08:00
Haojun Liao c5b279f045 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2 2023-08-30 16:01:08 +08:00
Haojun Liao 24b1a050fc other: merge 3.0 2023-08-30 16:00:14 +08:00
liuyao 0a72e576bd op stream pause
pause data dispatch

opt stream pause
2023-08-30 14:53:49 +08:00
dapan1121 f6321d0c4f Merge remote-tracking branch 'origin/main' into fix/m23.1 2023-08-30 11:03:33 +08:00
Haojun Liao 3a7a220d43 fix(stream): fix error found by ci. 2023-08-30 10:14:01 +08:00
Haojun Liao 204fb97f68 fix(stream): fix error in snode, fix memory leaks, fix repeatly create bug in sink task. 2023-08-29 17:34:31 +08:00
yihaoDeng 8ed2836a2b refactor code 2023-08-29 09:52:03 +08:00
yihaoDeng a2901411b0 refactor code 2023-08-28 18:52:10 +08:00
yihaoDeng 1687f68156 refactor code 2023-08-28 16:13:27 +08:00
Haojun Liao 391103bc5a fix(stream): do dome internal refactor. 2023-08-28 15:54:01 +08:00
Haojun Liao b75ae217f6 fix(stream): notify close for snode. 2023-08-28 09:53:08 +08:00
yihaoDeng 998dac0ffb fix mem leak 2023-08-26 21:54:36 +08:00
yihaoDeng 7d237d4310 fix UAF 2023-08-26 17:37:09 +08:00
yihaoDeng bed5765013 fix UAF 2023-08-26 16:15:26 +08:00
yihaoDeng 0348ef6e22 fix UAF 2023-08-26 16:13:17 +08:00
Ganlin Zhao af130cc92f Revert "add seperate function for createLogs"
This reverts commit 13d2d72bb6.
2023-08-25 09:55:50 +08:00
Haojun Liao 218e2cc5f3 fix(stream): invoke the timer when not sending hb. 2023-08-24 18:27:09 +08:00
Haojun Liao 82e356a9b5 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2
# Conflicts:
#	source/dnode/vnode/src/tq/tq.c
2023-08-24 18:04:44 +08:00
Haojun Liao 70396aa5a5 fix(stream): wakeup the hb timer every 200ms, to speeup the close db procedure. 2023-08-24 17:54:50 +08:00
yihaoDeng 090515c684 refactor code 2023-08-24 17:20:04 +08:00
Ganlin Zhao 13d2d72bb6 add seperate function for createLogs 2023-08-24 14:56:14 +08:00
Ganlin Zhao 962a0cd403 fix 2023-08-23 23:34:02 +08:00
yihaoDeng 0959758bd5 fix transfer error 2023-08-23 21:43:10 +08:00
Haojun Liao 230d8c2d80 fix(stream): fix error in stream computing after vnode is transferred to other dnodes. 2023-08-23 17:50:44 +08:00
yihaoDeng 12b10dfc76 fix mem leak 2023-08-23 09:08:37 +08:00
yihaoDeng 5fe7563a13 fix mem leak 2023-08-22 21:13:02 +08:00
yihaoDeng ba5c3648db fix mem leak 2023-08-22 21:12:54 +08:00
yihaoDeng ab9490101f Merge branch 'enh/3.0' into enh/triggerCheckPoint2 2023-08-22 18:04:04 +08:00
Haojun Liao af80d94a88 refactor: do some internal refactor. 2023-08-22 15:43:50 +08:00
Haojun Liao e6b49b45d9 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2 2023-08-22 13:37:50 +08:00
Haojun Liao cc8b2263cb fix(stream): fix error in heartbeat from vnode. 2023-08-22 13:37:38 +08:00
yihaoDeng d106b5a87e Merge branch 'enh/triggerCheckPoint2' of https://github.com/taosdata/TDengine into enh/triggerCheckPoint2 2023-08-22 10:48:30 +08:00
yihaoDeng a0fd56c84f fix invalid cmp 2023-08-22 10:48:15 +08:00
Haojun Liao def0877980 fix(stream): add timer to check stream task status. 2023-08-22 10:46:49 +08:00
Haojun Liao 082882b71e fix(stream): reset the upstream stage info. 2023-08-21 19:29:03 +08:00
yihaoDeng 387b4d365e refactor checkpoint 2023-08-21 19:07:15 +08:00
Haojun Liao 29bb5854cc fix(stream): fix error in check stream state. 2023-08-21 11:48:55 +08:00
yihaoDeng 8e4411c144 fix checkpoint error 2023-08-18 21:42:35 +08:00
yihaoDeng 114df7c349 fix transfer error 2023-08-16 11:37:15 +00:00
yihaoDeng ab7e2b3122 Merge branch 'enh/chkpTransfer' into enh/triggerCheckPoint2 2023-08-16 09:00:30 +00:00
Haojun Liao d63758a1c5 fix(stream): fix error in execute task on a tranferred node. 2023-08-16 15:24:24 +08:00
yihaoDeng 993ae84398 close backend 2023-08-16 07:17:15 +00:00
Haojun Liao 39079ab64a fix(stream): load term from sync model to decide if the node is restart or not 2023-08-15 15:52:06 +08:00
Haojun Liao b1d35f43ba enh(stream): using term from sync module to denote the upstream node is restarted/replica-modified/vnode-transferred or not. 2023-08-15 09:25:05 +08:00
yihaoDeng 99be2836b2 Merge branch 'enh/triggerCheckPoint2' into enh/chkpTransfer 2023-08-11 02:50:12 +00:00
Haojun Liao 1e636a2ef9 Merge remote-tracking branch 'origin/enh/triggerCheckPoint2' into enh/triggerCheckPoint2 2023-08-11 10:37:40 +08:00
Haojun Liao 6a9826cf6e enh(stream): update the stream task epset. 2023-08-11 10:37:25 +08:00
yihaoDeng ba4ce9db02 Merge branch 'enh/triggerCheckPoint2' into enh/chkpTransfer 2023-08-11 02:15:07 +00:00
yihaoDeng 32983a11e1 add log and fix crash 2023-08-10 20:34:00 +08:00
Haojun Liao a713b37f35 fix(stream): add more logs. 2023-08-10 14:07:29 +08:00
Haojun Liao f633c7fe2b other: merge 3.0. 2023-08-10 11:56:30 +08:00
yihaoDeng ef247cdb1d support reopen stream state 2023-08-10 01:49:26 +00:00
Haojun Liao c10b323188 other: merge main. 2023-08-10 09:26:33 +08:00
Haojun Liao 0cf8144952 fix(stream): fix the invalid key used by remove data in stream meta hash table. 2023-08-09 22:28:23 +08:00
Haojun Liao d34b9af054 enh(stream): create update stream trans. 2023-08-09 19:38:13 +08:00
Haojun Liao 1e8579e8c5 fix(stream): fix other cases. 2023-08-09 16:04:48 +08:00
Haojun Liao 4c92997328 fix(stream): use streamId&taskId to identify the stream task in the stream meta. 2023-08-09 15:52:23 +08:00
Haojun Liao f3c5f20ee2 enh(stream): add timer to check the node changing. 2023-08-09 10:25:16 +08:00
Haojun Liao 17c6143471 fix(stream): check more status when handling the state transfer. 2023-08-08 15:08:11 +08:00
Haojun Liao a0dcec5890 enh(stream): add function handler for hb message. 2023-08-08 11:23:19 +08:00
Haojun Liao f8ee055641 other: merge 3.0 2023-08-08 00:19:08 +08:00
yihaoDeng 80e78e054d support reopen stream state 2023-08-07 11:39:32 +00:00
yihaoDeng 6c8c575549 support reopen stream state 2023-08-07 10:48:04 +00:00
Haojun Liao aea1690cd1 fix(stream): fix the stream task after checking if it is in the timer activities. 2023-08-07 17:49:36 +08:00
yihaoDeng b6c991f896 support reopen stream state 2023-08-07 08:59:04 +00:00
yihaoDeng eeb97351e8 support reopen stream state 2023-08-07 08:51:01 +00:00
Haojun Liao 225c0e61dc hen(stream): support the rollback and checkdown stream tasks. 2023-08-07 16:13:39 +08:00
yihaoDeng a4da6e9a88 fix mem leak 2023-08-04 15:59:34 +08:00
yihaoDeng 81908be32f add stream ver 2023-08-04 14:02:03 +08:00
yihaoDeng 6dc93a8dec add stream ver 2023-08-04 13:52:20 +08:00
yihaoDeng f9d0874e45 support reopen stream state 2023-08-03 08:24:50 +00:00
Haojun Liao ad4c5916e7 enh(stream): update the stream task. 2023-08-01 18:14:58 +08:00
Haojun Liao cc5ff44604 enh(stream): handle the stream hb. 2023-07-30 19:25:00 +08:00
Haojun Liao 8dfef8768a other: merge 3.0 2023-07-29 22:54:16 +08:00
Haojun Liao 9f943213ed enh(stream): update the hb info. 2023-07-28 14:23:53 +08:00
Haojun Liao 74834aea11 enh(stream): prepare hb to mnode from stream meta. 2023-07-28 13:07:16 +08:00
Haojun Liao 22a5a18fff fix(stream): 1. set correct timewindow after step2. 2. handle the case when the task is failed to be added into the meta store. 2023-07-27 18:57:36 +08:00
Haojun Liao cdffabcdad fix(stream): fix memory leak. 2023-07-26 14:37:04 +08:00
Haojun Liao 063673726a fix(stream): fix syntax error 2023-07-25 18:08:22 +08:00
Haojun Liao 2cc4e0a727 fix(stream): fix syntax error 2023-07-25 17:56:40 +08:00
Haojun Liao 0bdccd2681 refactor: do some internal refactor. 2023-07-25 17:31:54 +08:00
Haojun Liao aed99da6c1 refactor: refactor the stream task starting order. 2023-07-20 12:14:53 +08:00
yihaoDeng 40d59990d8 fix mem leak 2023-07-18 04:01:20 +00:00
Haojun Liao 2f24f776fb fix(stream): fix error in generating checkpoint for state_window. 2023-07-18 09:04:28 +08:00
Haojun Liao b2a905bd91
Merge branch '3.0' into enh/triggerCheckPoint2 2023-07-17 11:50:12 +08:00
yihaoDeng dea2e73d2e add checkpoint 2023-07-17 02:22:21 +00:00
yihaoDeng b4de0892e6 add checkpoint 2023-07-15 09:35:33 +00:00
yihaoDeng e18c9ea672 add checkpoint 2023-07-15 04:26:34 +00:00
yihaoDeng aad669e33c add checkpoint 2023-07-14 14:46:13 +00:00
yihaoDeng 3020e27114 fix mem leak 2023-07-14 10:20:37 +00:00
yihaoDeng 3c4631ef1f add checkpoint 2023-07-14 10:06:24 +00:00
Haojun Liao 2f90a53548 fix(stream): set checkpoint id when expanding task. 2023-07-13 10:16:57 +08:00
yihaoDeng bb72f07a94 rm expire checkpoint 2023-07-12 16:38:46 +08:00
Haojun Liao fed3803739 fix(stream): fix memory leak. 2023-07-11 19:29:52 +08:00
yihaoDeng e547ed3699 rm expire checkpoint 2023-07-11 16:30:14 +08:00
Haojun Liao d7101f7109 enh(stream): support dispatch checkpoint msg. 2023-07-06 18:26:22 +08:00
Haojun Liao 5a9461a69a other: merge 3.0 2023-07-06 11:52:21 +08:00
Haojun Liao f4a3567935 other: merge 3.0 2023-07-06 11:24:30 +08:00
Haojun Liao 03dbcbf055 refactor: do some internal refactor. 2023-07-05 15:55:55 +08:00
yihaoDeng 247d3ac7ee vnode snapshot read 2023-06-30 12:05:05 +00:00
yihaoDeng 7d30a6e27a vnode snapshot read 2023-06-30 11:48:46 +00:00
Haojun Liao 57fcd553f8 refactor: do some internal refactor. 2023-06-28 13:13:13 +08:00